Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
refactor: make SemanticTrack the program skeleton
kashorinf3d319db2250
diff --git a/.agents/skills/hypit/references/playbooks/craft/b-roll.md b/.agents/skills/hypit/references/playbooks/craft/b-roll.mdindex f3e14f28..7f7b4023 100644--- a/.agents/skills/hypit/references/playbooks/craft/b-roll.md+++ b/.agents/skills/hypit/references/playbooks/craft/b-roll.md@@ -76,11 +76,11 @@ only; narration, music, and effects remain explicit audio contributions.## Place B-roll by authored meaning-- Mark the intended spoken range as a Script `Selection`, then use `during={story.selection.NAME}` on- the Media Item and pass `{timing.map}` to the Track.+- Mark the intended spoken range as a Script `Selection`, pass `semantic={speech.semantic}` to the+ Track, then use `during={story.selection.NAME}` on the Media Item.- For a point event, declare a Script `Moment` and use `at={story.moment.NAME}` with an explicit `for`.-- For silent programs or intentionally absolute edits, use `during="program"` or explicit `start` and- `end` expressions in the shared ProgramSpace.+- For intentionally absolute edits, use `during="program"` or explicit `start` and `end` expressions+ in the shared SemanticTrack frame domain.- For a non-contiguous Selection, choose `occurrences="each"` only when the same insert should appearat every occurrence; otherwise author separate Items.- Create J-cuts and L-cuts in the Script boundaries: let narration establish a few words before thediff --git a/.agents/skills/hypit/references/playbooks/craft/captions.md b/.agents/skills/hypit/references/playbooks/craft/captions.mdindex ef4166c4..acb3e487 100644--- a/.agents/skills/hypit/references/playbooks/craft/captions.md+++ b/.agents/skills/hypit/references/playbooks/craft/captions.md@@ -38,7 +38,7 @@ Use one pipeline for the program:```textexact font → caption-fine:Style → caption:Program → caption-ai:Planner+- whisperx:Alignment map+ speech.semantic↓caption-fine:Track```@@ -62,7 +62,7 @@ exact font → caption-fine:Style → caption:Program → caption-ai:Planner<caption-fine:Track id="captions" display={story.caption}correspondence={story.caption.correspondence}- map={timing.map} space={speech.space}+ semantic={speech.semantic}program={caption-program} plan={caption-plan.plan}/>```@@ -88,9 +88,9 @@ captions, omit the Caption components entirely.- `caption-ai:Planner` receives immutable display Atoms and resolved Style runs. It may place Cue cutsonly between complete Atoms and attach fields declared by the Style family.- The Planner does not see audio, rewrite text, select Styles, or invent timestamps.-- `whisperx:Alignment` measures the accepted `speech:Spine` audio and produces the SemanticMap used by- `caption-fine:Track`.-- Recheck alignment whenever the speech audio changes — `production-gates.md` Gate 3 states this for+- Each `whisperx:SemanticTake` measures one accepted normalized Segment take and packages its local+ timing; `speech:Track` assembles those Takes into the SemanticTrack used by `caption-fine:Track`.+- Recompute the affected SemanticTake whenever the speech audio changes — `production-gates.md` Gate 3 states this forevery Track that is timed against it, not only for captions. A Style-only change does not prove thata prior visual review is still valid.diff --git a/.agents/skills/hypit/references/playbooks/craft/overlays.md b/.agents/skills/hypit/references/playbooks/craft/overlays.mdindex a21a0390..3b2e9f0b 100644--- a/.agents/skills/hypit/references/playbooks/craft/overlays.md+++ b/.agents/skills/hypit/references/playbooks/craft/overlays.md@@ -44,7 +44,8 @@ edges are inspected. A PIP is not an ordinary overlay and is not covered by this## Bind overlays to story time-- Use `during={story.selection.NAME}` for a semantic range and pass `{timing.map}` to the Track.+- Pass `semantic={speech.semantic}` to the Track and use `during={story.selection.NAME}` for a+ semantic range.- Use `at={story.moment.NAME}` with an explicit `for` duration for a point event.- Use `during="program"` for persistent overlays and explicit `start`/`end` expressions for deliberateauthor-time placement.diff --git a/.agents/skills/hypit/references/playbooks/craft/persona-and-audio.md b/.agents/skills/hypit/references/playbooks/craft/persona-and-audio.mdindex 60a7246d..6a06a9c6 100644--- a/.agents/skills/hypit/references/playbooks/craft/persona-and-audio.md+++ b/.agents/skills/hypit/references/playbooks/craft/persona-and-audio.md@@ -36,7 +36,7 @@ Connect only the references a component actually needs.- When TTS is the sole narration source, choose the voice and generate or measure the real speechbefore locking visual cut lengths.- Use `estimate:Speech` only for deterministic pre-generation planning. Final caption and semantic- timing comes from accepted audio through `speech:Spine` and `whisperx:Alignment`.+ timing comes from each accepted normalized Take through `whisperx:SemanticTake`.- One continuous voiceover may cover several independently selected visual shots. Do not split TTSmerely because the picture cuts.- For dialogue generated inside speaking Seedance takes, preserve Script Segment order when assembling@@ -44,13 +44,13 @@ Connect only the references a component actually needs.## Choose one path for speech audio-- Put speech-bearing generated or supplied audio in `speech:Spine` as an audio `speech:Take` when it- is the program's authoritative narration. This produces ProgramSpace and speech audio for- `whisperx:Alignment` without inventing a visual clip.-- Do not place the same narration on both Speech Spine and an Audio Track. That duplicates speech in+- Normalize authoritative speech-bearing generated or supplied audio, create its+ `whisperx:SemanticTake`, then put that value in `speech:Track` as a `speech:Take`. An audio-only+ Semantic Take contributes timing and speech audio without inventing a visual clip.+- Do not place the same narration on both Speech Track and an Audio Track. That duplicates speech inthe final mix and creates competing time authority.-## Put non-spine audio on Tracks+## Put non-speech audio on TracksGenerated or supplied audio is a Blob until it is explicitly normalized. Prepare an audio-only sourceas `SynchronizedMedia`, then place it with `audio:Track`:@@ -60,14 +60,14 @@ as `SynchronizedMedia`, then place it with `audio:Track`:<pipeline:Normalize id="music-media" source={music-source}video="none" audio="default" span-authority="audio" frame-rate="30"/>-<audio:Track id="music-track" space={speech.space}>+<audio:Track id="music-track" semantic={speech.semantic}><audio:Clip source={music-media.media} during="program"playback="once" gain="0.18" fade-in="6f" fade-out="12f"/></audio:Track>```- Keep primary speech, music, ambience, and effects as independently inspectable contributions. Add- Speech Spine's audio Track and every selected Audio Track to `film:Film`.+ Speech Track's audio projection and every selected Audio Track to `film:Film`.- Raw Media Track video is visual-only unless `audio="include"` is explicitly authored.- Dialogue wins the mix. Use explicit gain and fades; do not assume automatic ducking, loudnessnormalization, or mastering.diff --git a/.agents/skills/hypit/references/playbooks/craft/pip-overlay.md b/.agents/skills/hypit/references/playbooks/craft/pip-overlay.mdindex 1c941c82..181c135a 100644--- a/.agents/skills/hypit/references/playbooks/craft/pip-overlay.md+++ b/.agents/skills/hypit/references/playbooks/craft/pip-overlay.md@@ -22,7 +22,7 @@ secondary feed, or supporting image stays visible.<space:Frame id="presenter-pip" within={vertical}left="64%" top="52%" right="4%" bottom="8%"/>-<media-track:Track id="presenter-overlay" space={speech.space} canvas={vertical}>+<media-track:Track id="presenter-overlay" semantic={speech.semantic} canvas={vertical}><media-track:Item image={presenter-cutout.image} extent={presenter-extent}frame={presenter-pip} during="program"appearance={studio.media.pip} motion={studio.motion.pip}/>@@ -54,8 +54,8 @@ secondary feed, or supporting image stays visible.## Time and review the PIP-- Enter or exit on a perceptible story event, Script Selection, or Moment. Use semantic timing when a- measured speech map exists and explicit ProgramSpace timing otherwise.+- Enter or exit on a perceptible story event, Script Selection, or Moment. Use the assembled+ SemanticTrack as the timing authority.- Give the audience enough time to recognize both the base evidence and the PIP; avoid rapid togglingthat makes neither readable.- Review edge quality first, then placement on every relevant base shot, then entry/exit motion, audiodiff --git a/.agents/skills/hypit/references/playbooks/craft/production-gates.md b/.agents/skills/hypit/references/playbooks/craft/production-gates.mdindex 4dcc8f62..0c09ddc2 100644--- a/.agents/skills/hypit/references/playbooks/craft/production-gates.md+++ b/.agents/skills/hypit/references/playbooks/craft/production-gates.md@@ -62,14 +62,14 @@ visual QA.## Gate 3: measure and author Tracks-- Build `speech:Spine` only from accepted speech-bearing video or audio Takes, then run one- `whisperx:Alignment` for the authored Narrative.-- Recompute that alignment whenever the speech audio changes. Every Track timed against it — Caption,+- Normalize every accepted speech-bearing video or audio Take, create its Segment-local+ `whisperx:SemanticTake`, then assemble those Takes with `speech:Track`.+- Recompute the affected SemanticTake whenever its speech audio changes. Every Track timed against it — Caption,Media, Typography, Ranking, Deck, Comment, Screen, Audio — is measured against audio that no longerexists otherwise, and the drift is invisible in a still frame. A Style-only change does not prove aprior review still holds.- Review Caption, Media, Typography, Ranking, Deck, Comment, Screen, and Audio Tracks against the- real ProgramSpace and SemanticMap. Check timing, safe zones, occlusion, stacking, audio clarity,+ real SemanticTrack. Check timing, safe zones, occlusion, stacking, audio clarity,and whether each Track contributes unique information.- Treat `check` and `plan` as structural proofs only. They do not prove visual quality, real speechalignment, Provider output quality, or factual correctness.diff --git a/.agents/skills/hypit/references/playbooks/craft/seedance-directing.md b/.agents/skills/hypit/references/playbooks/craft/seedance-directing.mdindex ad5f9dca..6b97e0a9 100644--- a/.agents/skills/hypit/references/playbooks/craft/seedance-directing.md+++ b/.agents/skills/hypit/references/playbooks/craft/seedance-directing.md@@ -65,4 +65,4 @@ than relying on an unstated assumption.physical labels/UI already attached to referenced objects.Review every reference image through `production-gates.md` before calling Seedance, then review the-resulting take before adding it to Speech Spine or Media Track.+resulting take before normalizing it into a Semantic Take or adding it to a Media Track.diff --git a/.agents/skills/hypit/references/playbooks/craft/sfx.md b/.agents/skills/hypit/references/playbooks/craft/sfx.mdindex 4efff62b..5ebb3ea9 100644--- a/.agents/skills/hypit/references/playbooks/craft/sfx.md+++ b/.agents/skills/hypit/references/playbooks/craft/sfx.md@@ -23,15 +23,15 @@ Declare the source, normalize it to audio-only `SynchronizedMedia`, and place it<pipeline:Normalize id="reveal-media" source={reveal-source}video="none" audio="default" span-authority="audio" frame-rate="30"/>-<audio:Track id="effects" space={speech.space}>+<audio:Track id="effects" semantic={speech.semantic}><audio:Clip source={reveal-media.media}- at={story.moment.reveal} for="600ms" map={timing.map}+ at={story.moment.reveal} for="600ms"playback="once" gain="0.45" fade-in="0f" fade-out="3f"/></audio:Track>```- Use `during={story.selection.NAME}` for a range, a Script Moment plus `for` for a point event, or- explicit `start`/`end` timing in ProgramSpace.+ explicit `start`/`end` timing in the SemanticTrack frame domain.- Use `playback="once"` for a discrete effect, loop only for a genuinely repeating texture, andbounded pitch-preserving stretch only when the occupancy must fill an authored range.- Use explicit trim, gain, fade-in, and fade-out. The Audio Track preserves source level and does notdiff --git a/.agents/skills/hypit/references/playbooks/formats/mass-tarot.md b/.agents/skills/hypit/references/playbooks/formats/mass-tarot.mdindex e745a40c..a6a321fa 100644--- a/.agents/skills/hypit/references/playbooks/formats/mass-tarot.md+++ b/.agents/skills/hypit/references/playbooks/formats/mass-tarot.md@@ -11,15 +11,19 @@ ordered reveals and interpretations.`mimo:Preset`, `mimo:VoiceDesign`, or `mimo:VoiceClone`, then use an audio-only Speech Take:```svml-<speech:Spine id="speech" frame-rate="30"+<program:Clock id="clock" frame-rate="30"/>+<pipeline:Normalize id="narration-media" source={narration.audio}+ video="none" audio="default" span-authority="audio" clock={clock}/>+<whisperx:SemanticTake id="narration-semantic" narrative={story}+ segment={story.segment.reading} media={narration-media.media}/>+<speech:Track id="speech"visual-frame={full-frame} visual-appearance={studio.speech.visual} visual-z="0">- <speech:Take audio={narration.audio} segment={story.segment.reading}/>-</speech:Spine>-<whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>+ <speech:Take source={narration-semantic.take}/>+</speech:Track>```- Peer Media Tracks provide every card/table visual while `{speech.audioTrack}` carries narration.-- A visible host may instead use `speaker-v1` speaking takes assembled through the same Spine.+ Peer Media Tracks provide every card/table visual while `{speech.audio}` carries narration.+- A visible host may instead use `speaker-v1` speaking Semantic Takes assembled through the same Track.## Author the card visualsdiff --git a/.agents/skills/hypit/references/playbooks/formats/ranking-listicle.md b/.agents/skills/hypit/references/playbooks/formats/ranking-listicle.mdindex b71cde9f..6cc1bc58 100644--- a/.agents/skills/hypit/references/playbooks/formats/ranking-listicle.md+++ b/.agents/skills/hypit/references/playbooks/formats/ranking-listicle.md@@ -14,12 +14,12 @@ explain why every item belongs where it is placed.## Author the SVML program-1. Build the speech program with accepted speaking or audio-only `speech:Take` inputs, then run- `whisperx:Alignment` so the Selection and Moments resolve against real audio.+1. Normalize each accepted speaking or audio-only source, create its `whisperx:SemanticTake`, then+ assemble those Takes with `speech:Track` so Selections and Moments resolve against real audio.2. Declare exact font bytes and the variant's package-owned Style.3. Choose exactly one primary component: `ranking:TierBoard`, `ranking:Column`, or`ranking:TopThree`.-4. Connect `map={timing.map}`, `space={speech.space}`, the outer `during` Selection, repeated+4. Connect `semantic={speech.semantic}`, the outer `during` Selection, repeated`triggers` Moment, terminal Moment, explicit frame, Style, and the complete ordered item list.5. Add the component's `.visual` output to `film:Film`; add its optional `.audio` output only whenauthored sound is present.diff --git a/.agents/skills/hypit/references/playbooks/formats/scenario-call.md b/.agents/skills/hypit/references/playbooks/formats/scenario-call.mdindex 7f571eba..01c2932b 100644--- a/.agents/skills/hypit/references/playbooks/formats/scenario-call.md+++ b/.agents/skills/hypit/references/playbooks/formats/scenario-call.md@@ -15,8 +15,8 @@ inset. Both tiles remain live while the active speaker changes.direction through `copy:Set`.5. Generate each take with `seedance:ReferenceVideo generate-audio="true"`, the two layout images,and the two ordered voice references.-6. Assemble takes with `speech:Spine`, create one `whisperx:Alignment`, then add the exact-font Caption- chain, Typography, Media, and Audio Tracks.+6. Normalize each take, produce its `whisperx:SemanticTake`, assemble the results with+ `speech:Track`, then add the exact-font Caption chain, Typography, Media, and Audio Tracks.7. Assemble in `film:Film`, render with `render:Video`, and stage review/delivery in `.svrun`.## Preserve call-state truthdiff --git a/.agents/skills/hypit/references/playbooks/formats/street-interview.md b/.agents/skills/hypit/references/playbooks/formats/street-interview.mdindex c11ed09b..61096f60 100644--- a/.agents/skills/hypit/references/playbooks/formats/street-interview.md+++ b/.agents/skills/hypit/references/playbooks/formats/street-interview.md@@ -14,8 +14,8 @@ make the active speaker readable without changing the location or camera topolog`reaction`, and `gesture` in an SVS Recipe.5. For each Script Segment, use `copy:Render` with dialogue plus an optional English action slot,then call `seedance:ReferenceVideo generate-audio="true"` with the scene image and both voices.-6. Assemble the accepted Segment takes with `speech:Spine`, align the complete Script with- `whisperx:Alignment`, then add Caption, Media, Typography, and Audio Tracks as needed.+6. Normalize each accepted Segment take, produce its `whisperx:SemanticTake`, assemble the results+ with `speech:Track`, then add Caption, Media, Typography, and Audio Tracks as needed.7. Assemble peer Tracks in `film:Film`, render with `render:Video`, and use separate `.svrun`Sources with explicit Targets for review and delivery.diff --git a/.agents/skills/hypit/references/playbooks/formats/talking-head.md b/.agents/skills/hypit/references/playbooks/formats/talking-head.mdindex d648a658..ea806bd9 100644--- a/.agents/skills/hypit/references/playbooks/formats/talking-head.md+++ b/.agents/skills/hypit/references/playbooks/formats/talking-head.md@@ -16,8 +16,8 @@ captions, B-roll, and editorial typography as peer Tracks.5. Generate each take with `seedance:ReferenceVideo generate-audio="true"`, one accepted fullpresenter/scene image, and the intended voice reference. Keep reference order identical acrossthe shot group.-6. Assemble accepted takes in Script order with `speech:Spine`, then create one- `whisperx:Alignment` for the complete program.+6. Normalize each accepted take, create one `whisperx:SemanticTake` from that media and its Script+ Segment, then assemble those Semantic Takes in Script order with `speech:Track`.7. Add the exact-font Caption chain when captions are wanted: `caption-fine:Style` →`caption:Program` → `caption-ai:Planner` → `caption-fine:Track`.8. Add B-roll through `media-track:Track`, editorial copy through `typo:Track`, then assemble withdiff --git a/.agents/skills/hypit/references/playbooks/formats/two-person-podcast.md b/.agents/skills/hypit/references/playbooks/formats/two-person-podcast.mdindex 7ac0bef7..c1a5faaf 100644--- a/.agents/skills/hypit/references/playbooks/formats/two-person-podcast.md+++ b/.agents/skills/hypit/references/playbooks/formats/two-person-podcast.md@@ -26,8 +26,8 @@ Draw the room and name camera positions A and B before generating either view.4. Render each Segment with `copy:Render`, Script dialogue, and an optional English action direction.5. Generate each take with `seedance:ReferenceVideo generate-audio="true"`, both final views, andboth voice references in the Kit's declared order.-6. Assemble accepted takes through `speech:Spine`, run `whisperx:Alignment`, then add exact-font- Captions, evidence Media, Typography, and optional Audio Tracks.+6. Normalize each accepted take, produce its `whisperx:SemanticTake`, assemble the results through+ `speech:Track`, then add exact-font Captions, evidence Media, Typography, and optional Audio Tracks.7. Assemble with `film:Film`, render with `render:Video`, and use separate `.svrun` Sources withexplicit Targets for staged review and delivery.diff --git a/.agents/skills/hypit/references/playbooks/formats/voiceover-desk-demo.md b/.agents/skills/hypit/references/playbooks/formats/voiceover-desk-demo.mdindex 6e45d091..c133acc0 100644--- a/.agents/skills/hypit/references/playbooks/formats/voiceover-desk-demo.md+++ b/.agents/skills/hypit/references/playbooks/formats/voiceover-desk-demo.md@@ -11,15 +11,19 @@ screen, product, hand, or proof changes.3. Put the audio directly into an audio-only Speech Take:```svml-<speech:Spine id="speech" frame-rate="30"+<program:Clock id="clock" frame-rate="30"/>+<pipeline:Normalize id="narration-media" source={narration.audio}+ video="none" audio="default" span-authority="audio" clock={clock}/>+<whisperx:SemanticTake id="narration-semantic" narrative={story}+ segment={story.segment.narration} media={narration-media.media}/>+<speech:Track id="speech"visual-frame={full-frame} visual-appearance={studio.speech.visual} visual-z="0">- <speech:Take audio={narration.audio} segment={story.segment.narration}/>-</speech:Spine>-<whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>+ <speech:Take source={narration-semantic.take}/>+</speech:Track>```-4. Use `{speech.space}` for every Media/Typography/Caption/Audio Track and add- `{speech.audioTrack}` directly to `film:Film`.+4. Pass `{speech.semantic}` to every Media/Typography/Caption/Audio Track and add+ `{speech.audio}` directly to `film:Film`.5. Build each visual beat from supplied media or the vendored `broll-v1` Kit. Generated desk/screenshots use `seedance:FrameVideo` or `seedance:ReferenceVideo` with `generate-audio="false"`.6. Place visuals by the measured Script Selections/Moments. One sentence may span several visualdiff --git a/.agents/skills/hypit/references/playbooks/index.md b/.agents/skills/hypit/references/playbooks/index.mdindex 47baaef5..41f3320a 100644--- a/.agents/skills/hypit/references/playbooks/index.md+++ b/.agents/skills/hypit/references/playbooks/index.md@@ -24,9 +24,9 @@ needed for that craft or format.- Keep model invocation explicit with `seedance:TextVideo`, `seedance:FrameVideo`, or`seedance:ReferenceVideo`; keep every media reference, duration, model, resolution, and aspectratio visible in the Author Source.-- Build speech-led programs with `speech:Spine`, measure them with `whisperx:Alignment`, then add+- Build speech-led programs from normalized `whisperx:SemanticTake` values with `speech:Track`, then addpeer Caption, Media, Typography, Ranking, Deck, Comment, Screen, and Audio Tracks.-- Use `speech:Spine` for speech-bearing video or audio Takes. For speech-free formats, select a+- Use `speech:Track` for ordered speech-bearing Semantic Takes. For speech-free formats, select averified ProgramSpace Record in `.svrun`, author explicit timing, and omit WhisperX/Caption work.- Assemble peer Tracks with `film:Film`, render with `render:Video`, and demand outputs through a`.svrun` Target.diff --git a/docs/guide/author-packages.md b/docs/guide/author-packages.mdindex dd4ad376..9dc60a0a 100644--- a/docs/guide/author-packages.md+++ b/docs/guide/author-packages.md@@ -124,13 +124,13 @@ vocabulary: {],children: [ /* accepted child elements */ ],ports: [ /* declared edges */ ],- example: "<mine:Widget id=\"first\" space={speech.space}/>",+ example: "<mine:Widget id=\"first\"/>",notes: [ /* rules a reader would otherwise have to discover by failing */ ],}```Declare a `preview` for every Surface whose result a reader needs to see to understand it. That is-not only Surfaces producing a `VisualTrack`: `@hypit/speech-spine` declares one for a Spine because+not only Surfaces producing a `VisualTrack`: `@hypit/speech-track` declares one for a Spine becauseits shape is easier to see than to describe. Skip it for a Surface with nothing to show, such as onethat only assembles a request.@@ -140,7 +140,7 @@ says what the element draws in every case; the preview shows one honest instanceStudy `packages/media-track/src/manifest.ts` for a complete vocabulary, and`packages/typography-track/`, `packages/caption-fine/`, `packages/deck-track/`,`packages/comment-sticker/`, `packages/screen-overlay/`, `packages/ranking/` and-`packages/speech-spine/` for their `preview/` directories.+`packages/speech-track/` for their `preview/` directories.### Producing the preview imagediff --git a/docs/guide/packages.md b/docs/guide/packages.mdindex e2c524bc..7a4633cb 100644--- a/docs/guide/packages.md+++ b/docs/guide/packages.md@@ -86,11 +86,10 @@ specific Provider deployment.@hypit/mimo-tts three exact Xiaomi MiMo TTS models + author Surfaces@hypit/estimate duration estimation@hypit/speech shared speech products-@hypit/speech-basis generated speech A/V product@hypit/speech-evidence acoustic evidence products-@hypit/semantic-map authored-token timing map@hypit/speech-alignment speech alignment-@hypit/speech-spine ordered speech-take compilation+@hypit/semantic-track continuous semantic program skeleton+@hypit/speech-track ordered speech-take compilation@hypit/whisperx WhisperX component@hypit/caption caption planning and timing@hypit/caption-gemini Gemini caption plannerdiff --git a/docs/quickstart.md b/docs/quickstart.mdindex e1348ddb..ab55568f 100644--- a/docs/quickstart.md+++ b/docs/quickstart.md@@ -192,7 +192,7 @@ Follow the authoring path in order, or jump directly to the part you are changin| [Script](./quickstart/script.md) | Segments, Role Cues, Dual Text, Selections, Moments and text projections || [SVS Stylesheets](./quickstart/styles.md) | Reusable Recipes for captions, media, text and film || [Media & Generation](./quickstart/generation.md) | Images, audio, prompt text and explicit model components |-| [Timing & Assembly](./quickstart/timing.md) | Speech Spine, WhisperX, ProgramSpace and SemanticMap |+| [Timing & Assembly](./quickstart/timing.md) | Speech Track, WhisperX, ProgramSpace and SemanticMap || [Tracks](./quickstart/tracks.md) | Caption, Media, Typography and Audio Tracks || [Film & Rendering](./quickstart/composition.md) | Peer Track composition and explicit rendering || [Run Source & Builds](./quickstart/run.md) | Targets, reuse, Runtime Profiles, Builds and retrieval |diff --git a/docs/quickstart/composition.md b/docs/quickstart/composition.mdindex facb1c73..993bb616 100644--- a/docs/quickstart/composition.md+++ b/docs/quickstart/composition.md@@ -22,9 +22,9 @@ and layers them by stacking order.```svml<space:Canvas id="vertical" width="1080" height="1920"/>-<film:Film id="main" canvas={vertical} space={speech.space} appearance={studio.film.vertical}>+<film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={studio.film.vertical}><film:Track source={speech.visual}/>- <film:Track source={speech.audioTrack}/>+ <film:Track source={speech.audio}/><film:Track source={captions.track}/><film:Track source={product-broll.visual}/><film:Track source={titles.track}/>@@ -35,7 +35,7 @@ and layers them by stacking order.|---|---|---|| `id` | yes | Unique identifier || `canvas` | yes | Explicit CanvasSpace shared with Track layout |-| `space` | yes | ProgramSpace from `speech:Spine` — defines duration and frame rate |+| `space` | yes | ProgramSpace from `speech:Track` — defines duration and frame rate || `appearance` | yes | SVS Film Recipe — the canvas clear color |### film:Track@@ -50,8 +50,8 @@ Common Track sources:| Source | Type | From ||---|---|---|-| `{speech.visual}` | VisualTrack | `speech:Spine` — sparse same-source speech visuals |-| `{speech.audioTrack}` | AudioTrack | `speech:Spine` — synchronized audio |+| `{speech.visual}` | VisualTrack | `speech:Track` — sparse same-source speech visuals |+| `{speech.audio}` | AudioTrack | `speech:Track` — synchronized audio || `{captions.track}` | VisualTrack | a Caption Style-family Track — timed captions || `{cards.visual}` | VisualTrack | `media-track:Track` — media overlays or B-roll || `{titles.track}` | VisualTrack | `text:Track` — text overlays |@@ -82,14 +82,14 @@ absolute stacking key, and paints them onto one canvas.Compiles the Composition into a finished video via the HyperFrames renderer.```svml-<render:Video id="final" composition={main.composition} space={speech.space}/>+<render:Video id="final" composition={main.composition} semantic={speech.semantic}/>```| Attribute | Required | Description ||---|---|---|| `id` | yes | Unique identifier || `composition` | yes | Composition from `film:Film` |-| `space` | yes | ProgramSpace from `speech:Spine` |+| `space` | yes | ProgramSpace from `speech:Track` |The renderer:@@ -118,7 +118,8 @@ The complete data flow from Script to rendered video. This example is based on<import as="wording" from="@hypit/text@1"/><import as="gpt" from="@hypit/gpt-image@1"/><import as="seedance" from="@hypit/seedance@1"/>- <import as="speech" from="@hypit/speech-spine@1"/>+ <import as="pipeline" from="@hypit/media-pipeline@1"/>+ <import as="speech" from="@hypit/speech-track@1"/><import as="whisperx" from="@hypit/whisperx@1"/><import as="caption" from="@hypit/caption@1"/><import as="caption-fine" from="@hypit/caption-fine@1"/>@@ -127,6 +128,7 @@ The complete data flow from Script to rendered video. This example is based on<import as="media-track" from="@hypit/media-track@1"/><import as="text" from="@hypit/typography-track@1"/><import as="space" from="@hypit/spatial@1"/>+ <import as="program" from="@hypit/program-space@1"/><import as="film" from="@hypit/film@1"/><import as="render" from="@hypit/render-hyperframes@1"/><import as="studio" source="./studio.svs"/>@@ -158,6 +160,7 @@ The complete data flow from Script to rendered video. This example is based on</seedance:ReferenceVideo><space:Canvas id="vertical" width="1080" height="1920"/>+ <program:Clock id="clock" frame-rate="30"/><space:Frame id="speech-frame" within={vertical}left="0%" top="0%" right="100%" bottom="100%"/><space:Frame id="title-frame" within={vertical}@@ -165,12 +168,17 @@ The complete data flow from Script to rendered video. This example is based on<space:Frame id="card-frame" within={vertical}left="10%" top="20%" right="90%" bottom="70%"/>- <!-- 3. Timing: assemble spine and align words -->- <speech:Spine id="speech" frame-rate="30"+ <!-- 3. Timing: normalize and align the Segment before assembly -->+ <pipeline:Normalize id="take-media" source={take.video}+ video="primary-moving" audio="default" span-authority="video" clock={clock}/>+ <pipeline:Normalize id="motion-media" source={motion.video}+ video="primary-moving" audio="none" span-authority="video" clock={clock}/>+ <whisperx:SemanticTake id="opening-semantic" narrative={story}+ segment={story.segment.opening} media={take-media.media}/>+ <speech:Track id="speech"visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">- <speech:Take video={take.video} segment={story.segment.opening}/>- </speech:Spine>- <whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>+ <speech:Take source={opening-semantic.take}/>+ </speech:Track><!-- 4. Tracks: captions, Media, text --><fonts:Stack id="caption-font" family="inter" weight="700" style="normal"/>@@ -180,26 +188,24 @@ The complete data flow from Script to rendered video. This example is based ondefault={base-caption}/><caption-ai:Planner id="cue-plan" display={story.caption}program={caption-program} model="gemini-2.5-flash"/>- <caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} map={timing.map}- space={speech.space} plan={cue-plan.plan} program={caption-program}/>+ <caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} semantic={speech.semantic} plan={cue-plan.plan} program={caption-program}/>- <media-track:Track id="cards" map={timing.map}- space={speech.space} canvas={vertical}>- <media-track:Item video={motion.video} during={story.selection.demo}+ <media-track:Track id="cards" semantic={speech.semantic} canvas={vertical}>+ <media-track:Item media={motion-media.media} during={story.selection.demo}frame={card-frame} appearance={studio.media.card} motion={studio.motion.card}/></media-track:Track><text:Style id="title-style" recipe={studio.text.title} font={title-font}/>- <text:Track id="titles" space={speech.space}>+ <text:Track id="titles" semantic={speech.semantic}><text:Area id="meaning" placement={title-frame} style={title-style} during="program">MEANING</text:Area></text:Track><!-- 5. Film: compose all tracks -->- <film:Film id="main" canvas={vertical} space={speech.space}+ <film:Film id="main" canvas={vertical} semantic={speech.semantic}appearance={studio.film.vertical}><film:Track source={speech.visual}/>- <film:Track source={speech.audioTrack}/>+ <film:Track source={speech.audio}/><film:Track source={cards.visual}/><film:Track source={captions.track}/><film:Track source={titles.track}/>@@ -207,7 +213,7 @@ The complete data flow from Script to rendered video. This example is based on<!-- 6. Render: compile to MP4 --><render:Video id="final" composition={main.composition}- space={speech.space}/>+ semantic={speech.semantic}/></svml>```diff --git a/docs/quickstart/generation.md b/docs/quickstart/generation.mdindex bb088a68..a9c03923 100644--- a/docs/quickstart/generation.md+++ b/docs/quickstart/generation.md@@ -374,6 +374,6 @@ A two-take setup with estimated durations feeding explicit Text assembly and See</seedance:ReferenceVideo>```-Each `seedance:ReferenceVideo` produces a `{*.video}` output that feeds into `speech:Spine` in the+Each `seedance:ReferenceVideo` produces a `{*.video}` output that feeds into `speech:Track` in thenext stage. Different takes can use different reference images while sharing the same voice timbreand prompt Recipe.diff --git a/docs/quickstart/preview.md b/docs/quickstart/preview.mdindex 6e7d44e4..1c32debd 100644--- a/docs/quickstart/preview.md+++ b/docs/quickstart/preview.md@@ -148,5 +148,5 @@ Drag the ruler to scrub. `Space` plays and pauses, `←` and `→` step one framHyperFrames renders a silent picture on purpose: programme audio is a separate Track the mediapipeline muxes in at the end. But placing B-roll against speech means hearing the speech, so the-Speech Spine's own material is allowed to sound while the transport is running. Cutaways stay+Speech Track's own material is allowed to sound while the transport is running. Cutaways staysilent, as they are in a build unless they ask otherwise. The speaker button turns it off.diff --git a/docs/quickstart/styles.md b/docs/quickstart/styles.mdindex 337b71d1..a108de07 100644--- a/docs/quickstart/styles.md+++ b/docs/quickstart/styles.md@@ -63,7 +63,7 @@ Referenced by `film:Film` via the `appearance` attribute:```svml<space:Canvas id="vertical" width="1080" height="1920"/>-<film:Film id="main" canvas={vertical} space={speech.space} appearance={studio.film.vertical}>+<film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={studio.film.vertical}>```## Caption Fine@@ -452,5 +452,5 @@ This file is imported once in the `.svml` source and its values are referenced t<caption-fine:Style id="primary-caption" recipe={studio.caption.primary} font={caption-font}/><space:Canvas id="vertical" width="720" height="1280"/>-<film:Film id="main" canvas={vertical} space={speech.space} appearance={studio.film.vertical}>+<film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={studio.film.vertical}>```diff --git a/docs/quickstart/timing.md b/docs/quickstart/timing.mdindex 946126e7..c9c4a6d6 100644--- a/docs/quickstart/timing.md+++ b/docs/quickstart/timing.md@@ -1,219 +1,135 @@---title: Timing & Assembly-description: Speech Spine assembly and WhisperX alignment — connecting generated takes to a unified timeline.+description: Per-take normalization and alignment, followed by SemanticTrack assembly.---# Timing & Assembly-After generation, individual takes must be concatenated into one continuous A/V coordinate space,-and the spoken words must be measured against the actual audio to create a timing map. These two-steps produce the **ProgramSpace** and **SemanticMap** that every downstream component depends on.+Hypit's timing authority is a `SemanticTrack`. Build it in segment-sized pieces:++1. normalize each accepted A/V take into one exact frame domain;+2. align that normalized media with its authored Script Segment to create a self-contained `SemanticTake`;+3. assemble the Semantic Takes in program order with `speech:Track`.++There is no whole-program transcription pass after concatenation. Every Take is already semantic+before it enters the Track.```svml-<import as="speech" from="@hypit/speech-spine@1"/>+<import as="program" from="@hypit/program-space@1"/>+<import as="pipeline" from="@hypit/media-pipeline@1"/><import as="whisperx" from="@hypit/whisperx@1"/>+<import as="speech" from="@hypit/speech-track@1"/><import as="space" from="@hypit/spatial@1"/><import as="studio" source="./studio.svs"/>```-## speech:Spine+## Normalize each take-Concatenates multiple takes into one ordered audio/visual coordinate space. The Spine defines the-program order — the final sequence of Segments in the finished video.+Normalization makes video, audio, duration and frame rate one explicit `SynchronizedMedia` fact.+The Clock is authored once and shared by every Take that will enter the same SemanticTrack.```svml-<space:Canvas id="vertical" width="1080" height="1920"/>-<space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>-<speech:Spine id="speech" frame-rate="30"- visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">- <speech:Take video={hook-take.video} segment={story.segment.hook}/>- <speech:Take video={meeting-take.video} segment={story.segment.meeting}/>- <speech:Take video={evidence-take.video} segment={story.segment.evidence}/>- <speech:Take video={payoff-take.video} segment={story.segment.payoff}/>-</speech:Spine>-```--| Attribute | Required | Description |-|---|---|---|-| `id` | yes | Unique identifier |-| `frame-rate` | yes | Program rate as an integer or rational, such as `30` or `30000/1001` |-| `visual-frame` | yes | Explicit base SpatialFrame for same-source Take visuals |-| `visual-appearance` | yes | SVS Recipe containing only spatial fit properties |-| `visual-z` | yes | Base absolute stacking order for same-source Take visuals |+<program:Clock id="clock" frame-rate="30"/>-### speech:Take--Each `<speech:Take>` child binds one speech-bearing source to a Script Segment:--| Attribute | Required | Description |-|---|---|---|-| `video` | exactly one | Generated/raw A/V Blob — for example `{take.video}` from Seedance |-| `audio` | exactly one | Voice-only Blob; contributes time and master speech audio, but no visual |-| `media` | exactly one | Already prepared `SynchronizedMedia`; bypasses automatic normalization |-| `segment` | yes | Script Segment this take corresponds to — e.g. `{story.segment.hook}` |-| `frame` | visual only | Override the Spine's `visual-frame` for this Take |-| `appearance` | visual only | Override the Spine's `visual-appearance` for this Take |-| `z` | visual only | Override the Spine's `visual-z` for this Take |--`video`, `audio` and `media` are mutually exclusive. The normal AIGC path is `video={take.video}`. The-Speech Surface expands that readable declaration into ordinary Media Pipeline Operations:-inspect the container, select its primary moving video and default audio stream, then normalize-both to the Spine's declared frame rate. A 30 fps source connected to a 60 fps Spine keeps its-duration and is deterministically resampled to a 60 fps frame sequence; it is not played twice as-fast. Use `media=` only when another graph branch has already produced the exact synchronized value-you intend to assemble.--The base is deliberately explicit rather than a hidden full-screen default. A visual Take inherits-all three values unless it overrides them. An audio Take cannot declare visual overrides: while it-is playing, `speech.visual` simply has no Present, so the Film background or peer Tracks remain-visible.--For voiceover timing, use an audio Take and supply the visuals through peer Media Tracks:--```svml-<speech:Spine id="speech" frame-rate="30"- visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">- <speech:Take audio={narration.audio} segment={story.segment.narration}/>-</speech:Spine>+<pipeline:Normalize id="opening-media" source={opening-video.video}+ video="primary-moving" audio="default" span-authority="video" clock={clock}/>+<pipeline:Normalize id="answer-media" source={answer-video.video}+ video="primary-moving" audio="default" span-authority="video" clock={clock}/>```-The fit Recipe is ordinary compile-time SVS data, for example:+Normalization contains no Script meaning and performs no transcription. It only establishes the+media facts that later semantic alignment can trust.-```svs-speech.visual { fit: cover; }-```--The order of `<speech:Take>` children **determines the program order**. The first take starts at-time zero; each subsequent take follows immediately.--### Outputs--The Spine produces four outputs used by downstream components:--| Output | Type | Used by |-|---|---|---|-| `{speech.visual}` | VisualTrack | `film:Film` — sparse same-source Take visuals |-| `{speech.audio}` | Audio | `whisperx:Alignment` — raw audio for word timing |-| `{speech.audioTrack}` | AudioTrack | `film:Film` — the synchronized audio |-| `{speech.space}` | ProgramSpace | Everything — the unified duration and frame domain |--## whisperx:Alignment+## Create one SemanticTake per Segment-Measures word-level timing by running WhisperX speech-to-text alignment on the Spine's audio output.-This produces the **SemanticMap** — the bridge between Script text and physical time.+`whisperx:SemanticTake` measures one normalized Take and aligns the evidence with exactly one+authored Segment:```svml-<whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>+<whisperx:SemanticTake id="opening-semantic" narrative={story}+ segment={story.segment.opening} media={opening-media.media}/>+<whisperx:SemanticTake id="answer-semantic" narrative={story}+ segment={story.segment.answer} media={answer-media.media}/>```-| Attribute | Required | Description |-|---|---|---|-| `id` | yes | Unique identifier |-| `narrative` | yes | The Script component — e.g. `{story}` |-| `audio` | yes | Audio from the Speech Spine — `{speech.audio}` |+Each output contains the normalized media, the Segment identity, every authored word's local frame+window, and all of that Segment's structural anchors. There are two anchors for the Segment and two+for each word. Acoustic evidence is an implementation input to this step; downstream components see+the completed `SemanticTake`, not a second evidence-shaped timing structure.-### Output--| Output | Type | Used by |-|---|---|---|-| `{timing.map}` | CompleteSemanticMap | Caption Style-family Tracks, `media-track:Track`, `text:Track` — timed placement |+## Assemble the SemanticTrack-The SemanticMap maps every authored Script anchor to a time point. It covers all `2M + 2N` identities-(where M = total speech tokens, N = number of Segments). This is how Selections and Moments declared-in the Script become real time ranges and points for downstream visual components.--## ProgramSpace--ProgramSpace is not guessed from a global default. It is produced by `speech:Spine` from the-explicit `frame-rate` and the exact normalized Take durations, then flows to every component that-needs the total program duration and frame domain.+`speech:Track` concatenates already-semantic Takes in document order and projects three aligned+facets from the same items:```svml-<film:Film id="main" canvas={vertical} space={speech.space} ...>-<caption-fine:Track id="captions" ... space={speech.space} .../>-<text:Track id="titles" space={speech.space}>-<render:Video id="final" composition={main.composition} space={speech.space}/>+<space:Canvas id="vertical" width="1080" height="1920"/>+<space:Frame id="speech-frame" within={vertical}+ left="0%" top="0%" right="100%" bottom="100%"/>++<speech:Track id="speech"+ visual-frame={speech-frame}+ visual-appearance={studio.speech.visual}+ visual-z="0">+ <speech:Take source={opening-semantic.take}/>+ <speech:Take source={answer-semantic.take}/>+</speech:Track>```-ProgramSpace owns:--- **Duration** — the total program length-- **Frame rate** — rational frame rate (e.g. 30fps)-- **Frame domain** — exact frame numbers for the entire program--Every component that operates in the time domain takes a `space` attribute pointing to-`{speech.space}`.--### Speech-free programs--A speech-free film still needs an explicit, verified ProgramSpace. Select a previously accepted-ProgramSpace Record in the Run Source with `build-record` and `satisfy`, then connect that named-logical output to Tracks and Film. Use `during="program"` or explicit `start`/`end` windows. Without-speech there is no measured SemanticMap, so omit WhisperX and Caption components and do not use-Selection/Moment timing. See [Reusing results](./run.md#reusing-results) for Run Source syntax.--## SemanticMap--The SemanticMap is the typed bridge between Script text and physical time. When you write-`during={story.selection.demo}` on a Media Item, the component uses the SemanticMap to look up the-exact frame range that Selection covers. Without a SemanticMap, Selections and Moments have no-physical meaning.--A whole Segment needs no synthetic Selection. `during={story.segment.answer}` addresses the-Segment's existing structural start/end anchors directly.--Components that use the map take it via the `map` attribute:--```svml-<media-track:Track id="cards" map={timing.map} ...>-<caption-fine:Track id="captions" ... map={timing.map} .../>-```+| Output | Type | Meaning |+|---|---|---|+| `{speech.semantic}` | SemanticTrack | Global semantic and frame-domain authority |+| `{speech.visual}` | VisualTrack | Same-source pictures, aligned to the semantic items |+| `{speech.audio}` | AudioTrack | Same-source sound, aligned to the semantic items |-The map contains final token windows and semantic anchor points only. It does not propagate-`measured`, `derived` or `estimated` labels. WhisperX evidence and the deterministic M:N aligner are-responsible for using the available recording evidence; downstream Tracks receive one complete map-and do not reinterpret how each point was obtained.+The three facets are projections of the same ordered Takes. They cannot drift independently.+`SemanticTrack` derives global frames by prefix-summing the local Take lengths, and also supplies the+program duration and frame domain required by Film and Render.-## Combination example+## Consume semantic time-The complete timing stage, from generated takes to map and space:+Selections, Moments and whole Segments remain authored Script identities. A downstream component+receives the SemanticTrack once and projects those identities into frames only when it builds its+deterministic Track:```svml-<import as="speech" from="@hypit/speech-spine@1"/>-<import as="whisperx" from="@hypit/whisperx@1"/>-<import as="space" from="@hypit/spatial@1"/>-<import as="studio" source="./studio.svs"/>--<!-- Assemble takes in program order -->-<space:Canvas id="vertical" width="1080" height="1920"/>-<space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>-<speech:Spine id="speech" frame-rate="30"- visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">- <speech:Take video={opening-take.video} segment={story.segment.opening}/>- <speech:Take video={answer-take.video} segment={story.segment.answer}/>-</speech:Spine>--<!-- Measure word timing -->-<whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>--<!-- Downstream components now reference: -->-<!-- {speech.space} — ProgramSpace for duration/frame domain -->-<!-- {speech.visual} — sparse same-source speech VisualTrack -->-<!-- {speech.audioTrack} — AudioTrack for synchronized audio -->-<!-- {timing.map} — SemanticMap for Selection/Moment timing -->+<media-track:Track id="cards" semantic={speech.semantic} canvas={vertical}>+ <media-track:Item image={card.image} extent={card-extent}+ during={story.selection.demo} frame={card-frame}+ appearance={studio.media.card} motion={studio.motion.card}/>+</media-track:Track>++<caption-fine:Track id="captions"+ display={story.caption}+ correspondence={story.caption.correspondence}+ semantic={speech.semantic}+ program={caption-program}+ plan={caption-plan.plan}/>++<film:Film id="main" canvas={vertical}+ semantic={speech.semantic} appearance={studio.film.vertical}>+ <film:Track source={speech.visual}/>+ <film:Track source={speech.audio}/>+ <film:Track source={cards.visual}/>+ <film:Track source={captions.track}/>+</film:Film>++<render:Video id="final"+ composition={main.composition} semantic={speech.semantic}/>```-The data flow:+Use `during={story.segment.answer}` for a whole Segment, a Selection for an authored range, a Moment+for a point event, and `during="program"` for the complete SemanticTrack domain. Components consume+`semantic={speech.semantic}`; they do not receive separate `map` and `space` values.```text-generated video(s) ─────► speech:Spine ──► whisperx:Alignment- │ │- .visual .map (SemanticMap)- .audio │- .audioTrack ▼- .space ──────────► caption-fine:Track- │ media-track:Track- │ text:Track- ▼ film:Film- film:Film render:Video+raw take ─► Normalize ─► SynchronizedMedia ─► SemanticTake ─┐+raw take ─► Normalize ─► SynchronizedMedia ─► SemanticTake ─┤+ ▼+ speech:Track+ ┌───────────┼───────────┐+ ▼ ▼ ▼+ .semantic .visual .audio+ │ │ │+ └──────► Film / Tracks ◄─┘```diff --git a/docs/quickstart/tracks.md b/docs/quickstart/tracks.mdindex 154280fa..308bfef1 100644--- a/docs/quickstart/tracks.md+++ b/docs/quickstart/tracks.md@@ -140,8 +140,7 @@ planner cannot rewrite text, select Styles, see audio or invent time. Its output### caption-fine:Track```svml-<caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} map={timing.map}- space={speech.space} program={caption-program} plan={caption-plan.plan}/>+<caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} semantic={speech.semantic} program={caption-program} plan={caption-plan.plan}/>```The common Caption timing step joins the Plan to the independent SemanticMap. Fine then renders all@@ -176,8 +175,7 @@ Placement is an explicit Spatial Frame edge; appearance and motion remain reusab<space:Frame id="product-frame" within={vertical}left="8%" top="20%" right="92%" bottom="68%"/>-<media-track:Track id="product-broll" map={timing.map}- space={speech.space} canvas={vertical}>+<media-track:Track id="product-broll" semantic={speech.semantic} canvas={vertical}><media-track:Item video={product-motion.video} frame={product-frame}during={story.selection.product-demo}appearance={studio.media.product}@@ -226,7 +224,7 @@ then choose its exact program window and occupancy:<pipeline:Normalize id="music-media" source={music}video="none" audio="default" span-authority="audio" frame-rate="30"/>-<audio:Track id="music-bed" space={speech.space}>+<audio:Track id="music-bed" semantic={speech.semantic}><audio:Clip source={music-media.media} during="program"playback="loop-end" gain="0.28" fade-in="600ms" fade-out="800ms"/></audio:Track>@@ -267,7 +265,7 @@ Container for text items.left="6%" top="6%" right="94%" bottom="16%"/><fonts:Stack id="title-font" family="inter" weight="900" style="normal"/><text:Style id="title-style" recipe={studio.text.title} font={title-font}/>-<text:Track id="titles" space={speech.space}>+<text:Track id="titles" semantic={speech.semantic}><text:Area id="title" placement={title-frame} style={title-style} during="program">EDIT MEANING, NOT TIMELINES</text:Area>@@ -277,7 +275,7 @@ Container for text items.| Attribute | Required | Description ||---|---|---|| `id` | yes | Unique identifier |-| `space` | yes | ProgramSpace from `speech:Spine` |+| `space` | yes | ProgramSpace from `speech:Track` || `map` | no | SemanticMap — needed when items use Selection-based timing |### text:Point, text:Area and text:Path@@ -304,7 +302,7 @@ or a Selection reference for semantic timing:```svml<text:Style id="callout-style" recipe={studio.text.callout} font={title-font}/>-<text:Track id="callout" space={speech.space} map={timing.map}>+<text:Track id="callout" semantic={speech.semantic}><text:Area id="callout-copy" placement={callout-frame}style={callout-style} during={story.selection.callout}>EXACTLY THE RIGHT MOMENT@@ -316,7 +314,7 @@ Graph-produced copy remains visible as an edge:```svml<wording:Value id="headline">EXACTLY THE RIGHT MOMENT</wording:Value>-<text:Track id="callout" space={speech.space}>+<text:Track id="callout" semantic={speech.semantic}><text:Area id="callout-copy" content={headline}placement={callout-frame} style={callout-style} during="program"/></text:Track>@@ -377,7 +375,7 @@ Each variant takes its own, at least one, and ids must be unique within a board.```svml<ranking:ColumnStyle id="board-style" recipe={studio.ranking.board} font={ui-font}/>-<ranking:Column id="board" map={timing.map} space={speech.space} frame={board-frame}+<ranking:Column id="board" semantic={speech.semantic} frame={board-frame}during={story.selection.board} triggers={story.moment.place} terminal={story.moment.done}style={board-style}><ranking:ColumnItem id="row-regen" label="ReGen" icon={icon-regen}/>@@ -425,7 +423,7 @@ give both and it is refused. `size`, `color`, `align`, `block` and `padding` are```svml<space:Frame id="deck-frame" within={vertical} left="44%" top="60%" right="98%" bottom="88%"/>-<deck:DepthStack id="deck" map={timing.map} space={speech.space} canvas={vertical}+<deck:DepthStack id="deck" semantic={speech.semantic} canvas={vertical}frame={deck-frame} appearance={studio.deck.stack} until={story.moment.done}><deck:Card id="card-spatial" source={icon-spatial} extent={square} at={story.moment.deal-one}/><deck:Card id="card-type" source={icon-type} extent={square} at={story.moment.deal-two}/>@@ -450,8 +448,8 @@ empty, each with a required `z` for stacking order and a window that is one of:| Window | Written ||---|---|| The whole programme | `during="program"` |-| A Selection | `during={story.selection.x} map={timing.map}` |-| A Moment, for a length | `at={story.moment.x} for="12f" map={timing.map}` |+| A Selection | `during={story.selection.x}` on an item whose Track has `semantic={speech.semantic}` |+| A Moment, for a length | `at={story.moment.x} for="12f"` on an item whose Track has `semantic={speech.semantic}` || An explicit span | `start="…" end="…"`, optionally against a `selection=` or `moment=` |Anything bound to the Script needs `map`; an explicit span with no Script source must not have one.@@ -465,8 +463,8 @@ required attributes, such as `color` / `intensity` / `attack` / `hold` / `decay`None have defaults: an effect states its whole shape or is refused.```svml-<screen:Track id="effects" space={speech.space} canvas={vertical}>- <screen:Flash during={story.selection.overlay} map={timing.map} z="80"+<screen:Track id="effects" semantic={speech.semantic} canvas={vertical}>+ <screen:Flash during={story.selection.overlay} z="80"color="#ffffff" intensity="0.6" attack="2" hold="2" decay="6"/></screen:Track>```@@ -496,7 +494,7 @@ image, and there is no `z`: stacking order comes from the recipe's `stack-order````svml<comment:Style id="social" recipe={studio.comment} font={ui-font}/>-<comment:Track id="comments" canvas={vertical} space={speech.space} map={timing.map}>+<comment:Track id="comments" canvas={vertical} semantic={speech.semantic}><comment:Sticker id="one" frame={comment-frame} style={social} avatar={viewer-avatar}author="@viewer" meta="Featured" during={story.selection.reaction}>Wait, it pinned the caption to the word, not the second.@@ -529,8 +527,7 @@ All four track families together in one source file:<caption:Program id="caption-program" display={story.caption} default={base-caption}/><caption-ai:Planner id="cue-plan" display={story.caption}program={caption-program} model="gemini-2.5-flash"/>-<caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} map={timing.map}- space={speech.space} plan={cue-plan.plan} program={caption-program}/>+<caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} semantic={speech.semantic} plan={cue-plan.plan} program={caption-program}/><!-- Shared placement is an explicit edge, separate from Text appearance. --><space:Canvas id="vertical" width="1080" height="1920"/>@@ -540,14 +537,14 @@ All four track families together in one source file:left="10%" top="20%" right="90%" bottom="70%"/><!-- Media: one ordinary Item used editorially as B-roll -->-<media-track:Track id="cards" map={timing.map} space={speech.space} canvas={vertical}>+<media-track:Track id="cards" semantic={speech.semantic} canvas={vertical}><media-track:Item video={motion.video} frame={card-frame}during={story.selection.demo} appearance={studio.media.card} motion={studio.motion.card}/></media-track:Track><!-- Text: persistent title overlay --><text:Style id="title-style" recipe={studio.text.title} font={title-font}/>-<text:Track id="titles" space={speech.space}>+<text:Track id="titles" semantic={speech.semantic}><text:Area id="meaning" placement={title-frame} style={title-style} during="program">MEANING</text:Area>@@ -557,15 +554,15 @@ All four track families together in one source file:<media:Audio id="music" src="./assets/music.wav"/><pipeline:Normalize id="music-media" source={music}video="none" audio="default" span-authority="audio" frame-rate="30"/>-<audio:Track id="music-bed" space={speech.space}>+<audio:Track id="music-bed" semantic={speech.semantic}><audio:Clip source={music-media.media} during="program"playback="loop-end" gain="0.28" fade-in="600ms" fade-out="800ms"/></audio:Track><!-- All peer tracks feed into Film -->-<film:Film id="main" canvas={vertical} space={speech.space} appearance={studio.film.vertical}>+<film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={studio.film.vertical}><film:Track source={speech.visual}/>- <film:Track source={speech.audioTrack}/>+ <film:Track source={speech.audio}/><film:Track source={cards.visual}/><film:Track source={captions.track}/><film:Track source={titles.track}/>diff --git a/docs/zh/guide/author-packages.md b/docs/zh/guide/author-packages.mdindex 4a900aed..1563e904 100644--- a/docs/zh/guide/author-packages.md+++ b/docs/zh/guide/author-packages.md@@ -115,16 +115,16 @@ vocabulary: {],children: [ /* 接受的子元素 */ ],ports: [ /* 声明的边 */ ],- example: "<mine:Widget id=\"first\" space={speech.space}/>",+ example: "<mine:Widget id=\"first\"/>",notes: [ /* 读者不看就只能靠报错发现的规则 */ ],}```-凡是读者必须看一眼产出才能理解的 Surface,都要声明 `preview`。这不限于产出 `VisualTrack` 的 Surface:`@hypit/speech-spine` 也为 Spine 声明了预览图,因为它的形态看一眼比读一段描述更快。没有可展示产出的 Surface 可以不声明,例如只负责装配请求的那种。+凡是读者必须看一眼产出才能理解的 Surface,都要声明 `preview`。这不限于产出 `VisualTrack` 的 Surface:`@hypit/speech-track` 也为 Spine 声明了预览图,因为它的形态看一眼比读一段描述更快。没有可展示产出的 Surface 可以不声明,例如只负责装配请求的那种。`appearance` 和 `preview` 回答的是两个不同的问题,谁都替代不了谁:`appearance` 说的是这个元素在所有情况下都会画出什么,预览图给出的是其中一个诚实的实例。-完整词表可研究 `packages/media-track/src/manifest.ts`;`preview/` 目录可参考 `packages/typography-track/`、`packages/caption-fine/`、`packages/deck-track/`、`packages/comment-sticker/`、`packages/screen-overlay/`、`packages/ranking/` 和 `packages/speech-spine/`。+完整词表可研究 `packages/media-track/src/manifest.ts`;`preview/` 目录可参考 `packages/typography-track/`、`packages/caption-fine/`、`packages/deck-track/`、`packages/comment-sticker/`、`packages/screen-overlay/`、`packages/ranking/` 和 `packages/speech-track/`。### 预览图怎么产出diff --git a/docs/zh/guide/packages.md b/docs/zh/guide/packages.mdindex c15adc23..15c89e14 100644--- a/docs/zh/guide/packages.md+++ b/docs/zh/guide/packages.md@@ -81,11 +81,10 @@ Frontend;不存在一个认识全部语法的中央解析器。@hypit/mimo-tts 三个精确 Xiaomi MiMo TTS 模型及作者 Surface@hypit/estimate duration estimation@hypit/speech shared speech products-@hypit/speech-basis generated speech A/V product@hypit/speech-evidence acoustic evidence products-@hypit/semantic-map authored-token timing map@hypit/speech-alignment speech alignment-@hypit/speech-spine ordered speech-take compilation+@hypit/semantic-track continuous semantic program skeleton+@hypit/speech-track ordered speech-take compilation@hypit/whisperx WhisperX component@hypit/caption caption planning and timing@hypit/caption-gemini Gemini caption plannerdiff --git a/docs/zh/quickstart.md b/docs/zh/quickstart.mdindex ad4bc5cd..87cb02b2 100644--- a/docs/zh/quickstart.md+++ b/docs/zh/quickstart.md@@ -59,7 +59,7 @@ npm link## 2. 编译示例-仓库里的示例包含 Script、两次视频生成需求、Speech Spine、WhisperX 对齐、字幕、Media Track、+仓库里的示例包含 Script、两次视频生成需求、Speech Track、WhisperX 对齐、字幕、Media Track、文字、Film 与最终渲染。```bash@@ -187,7 +187,7 @@ Endpoint 配置,但不会执行作者图。| [Script](./quickstart/script.md) | Segment、Role Cue、Dual Text、Selection、Moment 与文字投影 || [SVS 样式表](./quickstart/styles.md) | 字幕、Media、文字与 Film 的可复用 Recipe || [媒体与生成](./quickstart/generation.md) | 图片、音频、Prompt Text 与显式模型组件 |-| [时序与装配](./quickstart/timing.md) | Speech Spine、WhisperX、ProgramSpace 与 SemanticMap |+| [时序与装配](./quickstart/timing.md) | Speech Track、WhisperX、ProgramSpace 与 SemanticMap || [Tracks](./quickstart/tracks.md) | Caption、Media、Typography 与 Audio Track || [Film 与渲染](./quickstart/composition.md) | 平级 Track 合成与显式渲染 || [Run Source 与 Builds](./quickstart/run.md) | Targets、复用、Runtime Profile、Build 与取回结果 |diff --git a/docs/zh/quickstart/composition.md b/docs/zh/quickstart/composition.mdindex 2519e6bd..55322f4b 100644--- a/docs/zh/quickstart/composition.md+++ b/docs/zh/quickstart/composition.md@@ -21,9 +21,9 @@ Composition。然后渲染器将该 Composition 编译为 MP4 视频。```svml<space:Canvas id="vertical" width="1080" height="1920"/>-<film:Film id="main" canvas={vertical} space={speech.space} appearance={studio.film.vertical}>+<film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={studio.film.vertical}><film:Track source={speech.visual}/>- <film:Track source={speech.audioTrack}/>+ <film:Track source={speech.audio}/><film:Track source={captions.track}/><film:Track source={product-broll.visual}/><film:Track source={titles.track}/>@@ -34,7 +34,7 @@ Composition。然后渲染器将该 Composition 编译为 MP4 视频。|---|---|---|| `id` | 是 | 唯一标识符 || `canvas` | 是 | 与 Track 布局共享的显式 CanvasSpace |-| `space` | 是 | 来自 `speech:Spine` 的 ProgramSpace——定义时长和帧率 |+| `space` | 是 | 来自 `speech:Track` 的 ProgramSpace——定义时长和帧率 || `appearance` | 是 | SVS Film Recipe——画布清除颜色 |### film:Track@@ -49,8 +49,8 @@ Composition。然后渲染器将该 Composition 编译为 MP4 视频。| 来源 | 类型 | 来自 ||---|---|---|-| `{speech.visual}` | VisualTrack | `speech:Spine`——稀疏的同源口播视觉 |-| `{speech.audioTrack}` | AudioTrack | `speech:Spine`——同步音频 |+| `{speech.visual}` | VisualTrack | `speech:Track`——稀疏的同源口播视觉 |+| `{speech.audio}` | AudioTrack | `speech:Track`——同步音频 || `{captions.track}` | VisualTrack | Caption 样式族 Track——定时字幕 || `{cards.visual}` | VisualTrack | `media-track:Track`——Media 叠加层或 B-roll || `{titles.track}` | VisualTrack | `text:Track`——文字叠加层 |@@ -79,14 +79,14 @@ Track 是**扁平的**——没有嵌套或分组。Z 轴排序完全由每个 T通过 HyperFrames 渲染器将 Composition 编译为最终视频。```svml-<render:Video id="final" composition={main.composition} space={speech.space}/>+<render:Video id="final" composition={main.composition} semantic={speech.semantic}/>```| 属性 | 必填 | 说明 ||---|---|---|| `id` | 是 | 唯一标识符 || `composition` | 是 | 来自 `film:Film` 的 Composition |-| `space` | 是 | 来自 `speech:Spine` 的 ProgramSpace |+| `space` | 是 | 来自 `speech:Track` 的 ProgramSpace |渲染器:@@ -114,7 +114,8 @@ Build Target,也可以直接接到媒体裁切、音频/帧提取或模型参<import as="wording" from="@hypit/text@1"/><import as="gpt" from="@hypit/gpt-image@1"/><import as="seedance" from="@hypit/seedance@1"/>- <import as="speech" from="@hypit/speech-spine@1"/>+ <import as="pipeline" from="@hypit/media-pipeline@1"/>+ <import as="speech" from="@hypit/speech-track@1"/><import as="whisperx" from="@hypit/whisperx@1"/><import as="caption" from="@hypit/caption@1"/><import as="caption-fine" from="@hypit/caption-fine@1"/>@@ -123,6 +124,7 @@ Build Target,也可以直接接到媒体裁切、音频/帧提取或模型参<import as="media-track" from="@hypit/media-track@1"/><import as="text" from="@hypit/typography-track@1"/><import as="space" from="@hypit/spatial@1"/>+ <import as="program" from="@hypit/program-space@1"/><import as="film" from="@hypit/film@1"/><import as="render" from="@hypit/render-hyperframes@1"/><import as="studio" source="./studio.svs"/>@@ -154,6 +156,7 @@ Build Target,也可以直接接到媒体裁切、音频/帧提取或模型参</seedance:ReferenceVideo><space:Canvas id="vertical" width="1080" height="1920"/>+ <program:Clock id="clock" frame-rate="30"/><space:Frame id="speech-frame" within={vertical}left="0%" top="0%" right="100%" bottom="100%"/><space:Frame id="title-frame" within={vertical}@@ -161,12 +164,17 @@ Build Target,也可以直接接到媒体裁切、音频/帧提取或模型参<space:Frame id="card-frame" within={vertical}left="10%" top="20%" right="90%" bottom="70%"/>- <!-- 3. Timing: assemble spine and align words -->- <speech:Spine id="speech" frame-rate="30"+ <!-- 3. Timing:先归一化并对齐 Segment,再装配 -->+ <pipeline:Normalize id="take-media" source={take.video}+ video="primary-moving" audio="default" span-authority="video" clock={clock}/>+ <pipeline:Normalize id="motion-media" source={motion.video}+ video="primary-moving" audio="none" span-authority="video" clock={clock}/>+ <whisperx:SemanticTake id="opening-semantic" narrative={story}+ segment={story.segment.opening} media={take-media.media}/>+ <speech:Track id="speech"visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">- <speech:Take video={take.video} segment={story.segment.opening}/>- </speech:Spine>- <whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>+ <speech:Take source={opening-semantic.take}/>+ </speech:Track><!-- 4. Tracks: captions, Media, text --><fonts:Stack id="caption-font" family="inter" weight="700" style="normal"/>@@ -176,26 +184,24 @@ Build Target,也可以直接接到媒体裁切、音频/帧提取或模型参default={base-caption}/><caption-ai:Planner id="cue-plan" display={story.caption}program={caption-program} model="gemini-2.5-flash"/>- <caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} map={timing.map}- space={speech.space} plan={cue-plan.plan} program={caption-program}/>+ <caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} semantic={speech.semantic} plan={cue-plan.plan} program={caption-program}/>- <media-track:Track id="cards" map={timing.map}- space={speech.space} canvas={vertical}>- <media-track:Item video={motion.video} during={story.selection.demo}+ <media-track:Track id="cards" semantic={speech.semantic} canvas={vertical}>+ <media-track:Item media={motion-media.media} during={story.selection.demo}frame={card-frame} appearance={studio.media.card} motion={studio.motion.card}/></media-track:Track><text:Style id="title-style" recipe={studio.text.title} font={title-font}/>- <text:Track id="titles" space={speech.space}>+ <text:Track id="titles" semantic={speech.semantic}><text:Area id="meaning" placement={title-frame} style={title-style} during="program">MEANING</text:Area></text:Track><!-- 5. Film: compose all tracks -->- <film:Film id="main" canvas={vertical} space={speech.space}+ <film:Film id="main" canvas={vertical} semantic={speech.semantic}appearance={studio.film.vertical}><film:Track source={speech.visual}/>- <film:Track source={speech.audioTrack}/>+ <film:Track source={speech.audio}/><film:Track source={cards.visual}/><film:Track source={captions.track}/><film:Track source={titles.track}/>@@ -203,7 +209,7 @@ Build Target,也可以直接接到媒体裁切、音频/帧提取或模型参<!-- 6. Render: compile to MP4 --><render:Video id="final" composition={main.composition}- space={speech.space}/>+ semantic={speech.semantic}/></svml>```diff --git a/docs/zh/quickstart/generation.md b/docs/zh/quickstart/generation.mdindex 38621393..d3f47d6b 100644--- a/docs/zh/quickstart/generation.md+++ b/docs/zh/quickstart/generation.md@@ -334,5 +334,5 @@ dialogue/action 由普通 Text 模块组装,结果再像其他生成任务一</seedance:ReferenceVideo>```-每个 `seedance:ReferenceVideo` 产出 `{*.video}`,进入下一阶段的 `speech:Spine`。不同 Take+每个 `seedance:ReferenceVideo` 产出 `{*.video}`,进入下一阶段的 `speech:Track`。不同 Take可以使用不同参考图,同时共享相同的音色与 Prompt Recipe。diff --git a/docs/zh/quickstart/preview.md b/docs/zh/quickstart/preview.mdindex 0588c763..1c91ca50 100644--- a/docs/zh/quickstart/preview.md+++ b/docs/zh/quickstart/preview.md@@ -110,4 +110,4 @@ Segment 包着 Selection,Selection 还能再包 Selection。每一层有自己## 声音-HyperFrames 刻意只渲染无声画面:programme audio 是一条独立的 Track,由 media pipeline 在最后 mux 进去。但把 B-roll 对着语音安放,前提就是能听见那句话,所以走带播放时 Speech Spine 自己的素材允许发声。Cutaway 保持静音——除非显式声明要带音频,这和真实构建里的行为一致。喇叭按钮可以关掉。+HyperFrames 刻意只渲染无声画面:programme audio 是一条独立的 Track,由 media pipeline 在最后 mux 进去。但把 B-roll 对着语音安放,前提就是能听见那句话,所以走带播放时 Speech Track 自己的素材允许发声。Cutaway 保持静音——除非显式声明要带音频,这和真实构建里的行为一致。喇叭按钮可以关掉。diff --git a/docs/zh/quickstart/styles.md b/docs/zh/quickstart/styles.mdindex 4a1f5aa4..8485cfef 100644--- a/docs/zh/quickstart/styles.md+++ b/docs/zh/quickstart/styles.md@@ -59,7 +59,7 @@ film.vertical {```svml<space:Canvas id="vertical" width="1080" height="1920"/>-<film:Film id="main" canvas={vertical} space={speech.space} appearance={studio.film.vertical}>+<film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={studio.film.vertical}>```## Caption Fine@@ -429,5 +429,5 @@ Caption Recipe 不再重复家族、字重或字形。CJK 与 Emoji 即使由多<caption-fine:Style id="primary-caption" recipe={studio.caption.primary} font={caption-font}/><space:Canvas id="vertical" width="720" height="1280"/>-<film:Film id="main" canvas={vertical} space={speech.space} appearance={studio.film.vertical}>+<film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={studio.film.vertical}>```diff --git a/docs/zh/quickstart/timing.md b/docs/zh/quickstart/timing.mdindex 658e0a85..0f6c9aea 100644--- a/docs/zh/quickstart/timing.md+++ b/docs/zh/quickstart/timing.md@@ -1,197 +1,128 @@---title: 时序与装配-description: Speech Spine 装配与 WhisperX 对齐——将生成的片段连接到统一时间线。+description: 逐 Take 归一化与语义对齐,然后装配为 SemanticTrack。---# 时序与装配-生成完成后,各个片段必须拼接成一个连续的音视频坐标空间,并且需要将语音内容与实际音频进行比对,以创建时序映射。这两个步骤产生 **ProgramSpace** 和 **SemanticMap**,所有下游组件均依赖于它们。+Hypit 的时间真相是 `SemanticTrack`。它按 Segment 粒度构建:++1. 把每个已接受的音视频 Take 归一化到同一个精确帧域;+2. 将归一化媒体与对应的 Script Segment 对齐,得到自包含的 `SemanticTake`;+3. 用 `speech:Track` 按节目顺序装配这些 Semantic Take。++系统不再先拼接整段节目、再做一次全局转录。每个 Take 在进入 Track 之前就已经具有语义。```svml-<import as="speech" from="@hypit/speech-spine@1"/>+<import as="program" from="@hypit/program-space@1"/>+<import as="pipeline" from="@hypit/media-pipeline@1"/><import as="whisperx" from="@hypit/whisperx@1"/>+<import as="speech" from="@hypit/speech-track@1"/><import as="space" from="@hypit/spatial@1"/><import as="studio" source="./studio.svs"/>```-## speech:Spine--将多个片段拼接成一个有序的音视频坐标空间。Spine 定义了节目顺序——即最终视频中 Segment 的排列序列。--```svml-<space:Canvas id="vertical" width="1080" height="1920"/>-<space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>-<speech:Spine id="speech" frame-rate="30"- visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">- <speech:Take video={hook-take.video} segment={story.segment.hook}/>- <speech:Take video={meeting-take.video} segment={story.segment.meeting}/>- <speech:Take video={evidence-take.video} segment={story.segment.evidence}/>- <speech:Take video={payoff-take.video} segment={story.segment.payoff}/>-</speech:Spine>-```--| 属性 | 必填 | 描述 |-|---|---|---|-| `id` | 是 | 唯一标识符 |-| `frame-rate` | 是 | 节目帧率,写整数或有理数,例如 `30`、`30000/1001` |-| `visual-frame` | 是 | 同源 Take 视觉的显式基础 SpatialFrame |-| `visual-appearance` | 是 | 只包含空间 fit 属性的 SVS Recipe |-| `visual-z` | 是 | 同源 Take 视觉的基础绝对层级 |--### speech:Take--每个 `<speech:Take>` 子元素将一个带口播的来源绑定到一个 Script Segment:+## 逐 Take 归一化-| 属性 | 必填 | 描述 |-|---|---|---|-| `video` | 三选一 | 生成/原始视频 Blob,例如 Seedance 产出的 `{take.video}` |-| `audio` | 三选一 | 纯语音 Blob;贡献节目时长与主音频,不贡献视觉 |-| `media` | 三选一 | 已准备好的 `SynchronizedMedia`;跳过自动规范化 |-| `segment` | 是 | 此片段对应的 Script Segment——例如 `{story.segment.hook}` |-| `frame` | 仅视觉 | 覆盖本 Take 的基础 `visual-frame` |-| `appearance` | 仅视觉 | 覆盖本 Take 的基础 `visual-appearance` |-| `z` | 仅视觉 | 覆盖本 Take 的基础 `visual-z` |--`video`、`audio` 与 `media` 必须且只能选一个。AIGC 的正常路径就是 `video={take.video}`。Speech-Surface 会把这句易读声明展开为普通 Media Pipeline Operation:检查容器、选择主动态视频流和默认音轨,再按 Spine 声明的帧率规范化。把 30 fps 素材接入 60 fps Spine 时,时长不变,帧序列会被确定性重采样为 60 fps,而不是把视频播放加速一倍。只有上游图已经明确产出所需-`SynchronizedMedia` 时才使用 `media=`。--视觉 Take 默认继承 Spine 上显式写出的三个视觉值,也可以逐段覆盖。纯音频 Take 禁止写视觉覆盖;它播放时 `speech.visual` 就没有 Present,Film 背景或平级 Track 会自然露出,系统不会伪造黑场。--旁白时序使用 audio Take,画面则由对等 Media Track 提供:+归一化把视频、音频、时长和帧率变成一个明确的 `SynchronizedMedia` 事实。同一条+SemanticTrack 内的所有 Take 共享作者显式声明的 Clock。```svml-<speech:Spine id="speech" frame-rate="30"- visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">- <speech:Take audio={narration.audio} segment={story.segment.narration}/>-</speech:Spine>-```--fit Recipe 只是普通编译期 SVS 数据,例如:+<program:Clock id="clock" frame-rate="30"/>-```svs-speech.visual { fit: cover; }+<pipeline:Normalize id="opening-media" source={opening-video.video}+ video="primary-moving" audio="default" span-authority="video" clock={clock}/>+<pipeline:Normalize id="answer-media" source={answer-video.video}+ video="primary-moving" audio="default" span-authority="video" clock={clock}/>```-`<speech:Take>` 子元素的排列顺序**决定了节目顺序**。第一个片段从时间零点开始;后续片段依次紧接。+归一化不包含 Script 语义,也不负责转录;它只建立后续语义对齐可以信任的客观媒体事实。-### 输出+## 每个 Segment 产生一个 SemanticTake-Spine 产生四个输出,供下游组件使用:--| 输出 | 类型 | 使用方 |-|---|---|---|-| `{speech.visual}` | VisualTrack | `film:Film`——稀疏的同源 Take 视觉 |-| `{speech.audio}` | Audio | `whisperx:Alignment`——用于词级时序的原始音频 |-| `{speech.audioTrack}` | AudioTrack | `film:Film`——同步音频 |-| `{speech.space}` | ProgramSpace | 所有组件——统一的时长和帧域 |--## whisperx:Alignment--通过对 Spine 的音频输出运行 WhisperX 语音转文本对齐,测量词级时序。这将产生 **SemanticMap**——连接 Script 文本与物理时间的桥梁。+`whisperx:SemanticTake` 测量一段归一化媒体,并把声学证据与唯一一个作者 Segment 对齐:```svml-<whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>+<whisperx:SemanticTake id="opening-semantic" narrative={story}+ segment={story.segment.opening} media={opening-media.media}/>+<whisperx:SemanticTake id="answer-semantic" narrative={story}+ segment={story.segment.answer} media={answer-media.media}/>```-| 属性 | 必填 | 描述 |-|---|---|---|-| `id` | 是 | 唯一标识符 |-| `narrative` | 是 | Script 组件——例如 `{story}` |-| `audio` | 是 | 来自 Speech Spine 的音频——`{speech.audio}` |--### 输出--| 输出 | 类型 | 使用方 |-|---|---|---|-| `{timing.map}` | CompleteSemanticMap | Caption 样式族 Track、`media-track:Track`、`text:Track`——定时放置 |--SemanticMap 将每个 Script 中标注的锚点映射到一个时间点。它覆盖所有 `2M + 2N` 个标识(其中 M = 语音词元总数,N = Segment 数量)。这就是 Script 中声明的 Selection 和 Moment 如何转化为下游视觉组件所需的实际时间范围和时间点。+每个输出都自带归一化媒体、Segment 身份、每个作者词语的局部帧窗口以及该 Segment 的全部+结构锚点:Segment 有两个锚点,每个词也有两个锚点。声学证据只是这一步的实现输入;下游+组件看到的是完成后的 `SemanticTake`,而不是第二套 evidence 形状的时间结构。-## ProgramSpace+## 装配 SemanticTrack-ProgramSpace 不来自任何全局默认值。`speech:Spine` 用显式 `frame-rate` 和规范化后各 Take-的精确时长生成它,再传给每个需要知道总节目时长和帧域的组件。+`speech:Track` 按文档顺序拼接已经语义化的 Take,并从同一批 item 投影出三个严格对齐的侧面:```svml-<film:Film id="main" canvas={vertical} space={speech.space} ...>-<caption-fine:Track id="captions" ... space={speech.space} .../>-<text:Track id="titles" space={speech.space}>-<render:Video id="final" composition={main.composition} space={speech.space}/>+<space:Canvas id="vertical" width="1080" height="1920"/>+<space:Frame id="speech-frame" within={vertical}+ left="0%" top="0%" right="100%" bottom="100%"/>++<speech:Track id="speech"+ visual-frame={speech-frame}+ visual-appearance={studio.speech.visual}+ visual-z="0">+ <speech:Take source={opening-semantic.take}/>+ <speech:Take source={answer-semantic.take}/>+</speech:Track>```-ProgramSpace 包含:--- **时长**——节目总长度-- **帧率**——有理帧率(例如 30fps)-- **帧域**——整个节目的精确帧编号--每个在时间域中运行的组件都通过 `space` 属性指向 `{speech.space}`。--### 无语音节目--无语音影片仍然需要一份显式且已验证的 ProgramSpace。在 Run Source 中用 `build-record` 与-`satisfy` 选择一份此前已接受的 ProgramSpace Record,再把该具名逻辑输出连接给 Track 与-Film。Track 使用 `during="program"` 或显式 `start`/`end` 窗口。没有语音就没有实测-SemanticMap,因此省略 WhisperX 与 Caption 组件,也不要使用 Selection/Moment 时序。Run-Source 语法见 [复用结果](./run.md#复用结果)。--## SemanticMap--SemanticMap 是连接 Script 文本与物理时间的类型化桥梁。当你在 Media Item 上写 `during={story.selection.demo}` 时,组件会使用 SemanticMap 查找该 Selection 覆盖的精确帧范围。没有 SemanticMap,Selection 和 Moment 就没有物理意义。--整段使用无需伪造 Selection:`during={story.segment.answer}` 直接消费 Segment 已有的首尾结构锚点。--使用该映射的组件通过 `map` 属性接收它:--```svml-<media-track:Track id="cards" map={timing.map} ...>-<caption-fine:Track id="captions" ... map={timing.map} .../>-```+| 输出 | 类型 | 含义 |+|---|---|---|+| `{speech.semantic}` | SemanticTrack | 全局语义与帧域真相 |+| `{speech.visual}` | VisualTrack | 与语义 item 对齐的同源画面 |+| `{speech.audio}` | AudioTrack | 与语义 item 对齐的同源声音 |-Map 只包含最终词窗口和语义锚点,不传播“测量、推导、估算”标签。WhisperX Evidence-与确定性的 M:N 对齐器负责充分使用录音证据;下游 Track 只接收一份完整 Map,不再解释每个点是如何获得的。+三个侧面都是同一批有序 Take 的投影,不能各自漂移。`SemanticTrack` 通过局部 Take 长度的前缀和+得到全局帧位置,同时提供 Film 和 Render 所需的节目时长与帧域。-## 组合示例+## 消费语义时间-完整的时序阶段,从生成的片段到映射和空间:+Selection、Moment 与完整 Segment 始终是 Script 中的作者身份。下游组件只接收一次+SemanticTrack,并在构建确定性 Track 时把这些身份投影成帧:```svml-<import as="speech" from="@hypit/speech-spine@1"/>-<import as="whisperx" from="@hypit/whisperx@1"/>-<import as="space" from="@hypit/spatial@1"/>-<import as="studio" source="./studio.svs"/>--<!-- Assemble takes in program order -->-<space:Canvas id="vertical" width="1080" height="1920"/>-<space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>-<speech:Spine id="speech" frame-rate="30"- visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">- <speech:Take video={opening-take.video} segment={story.segment.opening}/>- <speech:Take video={answer-take.video} segment={story.segment.answer}/>-</speech:Spine>--<!-- Measure word timing -->-<whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>--<!-- Downstream components now reference: -->-<!-- {speech.space} — ProgramSpace for duration/frame domain -->-<!-- {speech.visual} — 稀疏的同源口播 VisualTrack -->-<!-- {speech.audioTrack} — AudioTrack for synchronized audio -->-<!-- {timing.map} — SemanticMap for Selection/Moment timing -->+<media-track:Track id="cards" semantic={speech.semantic} canvas={vertical}>+ <media-track:Item image={card.image} extent={card-extent}+ during={story.selection.demo} frame={card-frame}+ appearance={studio.media.card} motion={studio.motion.card}/>+</media-track:Track>++<caption-fine:Track id="captions"+ display={story.caption}+ correspondence={story.caption.correspondence}+ semantic={speech.semantic}+ program={caption-program}+ plan={caption-plan.plan}/>++<film:Film id="main" canvas={vertical}+ semantic={speech.semantic} appearance={studio.film.vertical}>+ <film:Track source={speech.visual}/>+ <film:Track source={speech.audio}/>+ <film:Track source={cards.visual}/>+ <film:Track source={captions.track}/>+</film:Film>++<render:Video id="final"+ composition={main.composition} semantic={speech.semantic}/>```-数据流向:+整段使用 `during={story.segment.answer}`,作者范围使用 Selection,点事件使用 Moment,完整节目+使用 `during="program"`。组件统一消费 `semantic={speech.semantic}`,不再同时接收分离的 `map`+和 `space`。```text-生成的视频 ─────────────► speech:Spine ──► whisperx:Alignment- │ │- .visual .map (SemanticMap)- .audio │- .audioTrack ▼- .space ──────────► caption-fine:Track- │ media-track:Track- │ text:Track- ▼ film:Film- film:Film render:Video+原始 Take ─► Normalize ─► SynchronizedMedia ─► SemanticTake ─┐+原始 Take ─► Normalize ─► SynchronizedMedia ─► SemanticTake ─┤+ ▼+ speech:Track+ ┌───────────┼───────────┐+ ▼ ▼ ▼+ .semantic .visual .audio+ │ │ │+ └──────► Film / Tracks ◄─┘```diff --git a/docs/zh/quickstart/tracks.md b/docs/zh/quickstart/tracks.mdindex a251b0d3..59d203db 100644--- a/docs/zh/quickstart/tracks.md+++ b/docs/zh/quickstart/tracks.md@@ -100,8 +100,7 @@ Dual Text 右侧。### caption-fine:Track```svml-<caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} map={timing.map}- space={speech.space} program={caption-program} plan={caption-plan.plan}/>+<caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} semantic={speech.semantic} program={caption-program} plan={caption-plan.plan}/>```公共 Caption 先把 Plan 与独立 SemanticMap 拼接,Fine 再把所有默认/覆盖样式渲染成一个普通的对等 `VisualTrack`:`{captions.track}`。@@ -133,8 +132,7 @@ B-roll 是通用 Media Track 的一种剪辑用途,不是独立 Track 家族<space:Frame id="product-frame" within={vertical}left="8%" top="20%" right="92%" bottom="68%"/>-<media-track:Track id="product-broll" map={timing.map}- space={speech.space} canvas={vertical}>+<media-track:Track id="product-broll" semantic={speech.semantic} canvas={vertical}><media-track:Item video={product-motion.video} frame={product-frame}during={story.selection.product-demo}appearance={studio.media.product}@@ -174,7 +172,7 @@ Operation。需要共享或特殊选流时仍可显式写 `<pipeline:Normalize>`<pipeline:Normalize id="music-media" source={music}video="none" audio="default" span-authority="audio" frame-rate="30"/>-<audio:Track id="music-bed" space={speech.space}>+<audio:Track id="music-bed" semantic={speech.semantic}><audio:Clip source={music-media.media} during="program"playback="loop-end" gain="0.28" fade-in="600ms" fade-out="800ms"/></audio:Track>@@ -214,7 +212,7 @@ Track 都会作为独立输入进入 Film。输出 `{music-bed.track}` 是普通left="6%" top="6%" right="94%" bottom="16%"/><fonts:Stack id="title-font" family="inter" weight="900" style="normal"/><text:Style id="title-style" recipe={studio.text.title} font={title-font}/>-<text:Track id="titles" space={speech.space}>+<text:Track id="titles" semantic={speech.semantic}><text:Area id="title" placement={title-frame} style={title-style} during="program">EDIT MEANING, NOT TIMELINES</text:Area>@@ -224,7 +222,7 @@ Track 都会作为独立输入进入 Film。输出 `{music-bed.track}` 是普通| 属性 | 必填 | 描述 ||---|---|---|| `id` | 是 | 唯一标识符 |-| `space` | 是 | 来自 `speech:Spine` 的 ProgramSpace |+| `space` | 是 | 来自 `speech:Track` 的 ProgramSpace || `map` | 否 | SemanticMap——当项目使用基于 Selection 的计时时需要 |### text:Point、text:Area 与 text:Path@@ -250,7 +248,7 @@ Track 都会作为独立输入进入 Film。输出 `{music-bed.track}` 是普通```svml<text:Style id="callout-style" recipe={studio.text.callout} font={title-font}/>-<text:Track id="callout" space={speech.space} map={timing.map}>+<text:Track id="callout" semantic={speech.semantic}><text:Area id="callout-copy" placement={callout-frame}style={callout-style} during={story.selection.callout}>EXACTLY THE RIGHT MOMENT@@ -262,7 +260,7 @@ Track 都会作为独立输入进入 Film。输出 `{music-bed.track}` 是普通```svml<wording:Value id="headline">EXACTLY THE RIGHT MOMENT</wording:Value>-<text:Track id="callout" space={speech.space}>+<text:Track id="callout" semantic={speech.semantic}><text:Area id="callout-copy" content={headline}placement={callout-frame} style={callout-style} during="program"/></text:Track>@@ -315,7 +313,7 @@ Run 时,继续使用内联 `P`/`Span`/`Break`。```svml<ranking:ColumnStyle id="board-style" recipe={studio.ranking.board} font={ui-font}/>-<ranking:Column id="board" map={timing.map} space={speech.space} frame={board-frame}+<ranking:Column id="board" semantic={speech.semantic} frame={board-frame}during={story.selection.board} triggers={story.moment.place} terminal={story.moment.done}style={board-style}><ranking:ColumnItem id="row-regen" label="ReGen" icon={icon-regen}/>@@ -356,7 +354,7 @@ DepthStack 的直接子元素,自闭合,至少一张,按书写顺序发出```svml<space:Frame id="deck-frame" within={vertical} left="44%" top="60%" right="98%" bottom="88%"/>-<deck:DepthStack id="deck" map={timing.map} space={speech.space} canvas={vertical}+<deck:DepthStack id="deck" semantic={speech.semantic} canvas={vertical}frame={deck-frame} appearance={studio.deck.stack} until={story.moment.done}><deck:Card id="card-spatial" source={icon-spatial} extent={square} at={story.moment.deal-one}/><deck:Card id="card-type" source={icon-type} extent={square} at={story.moment.deal-two}/>@@ -378,8 +376,8 @@ DepthStack 的直接子元素,自闭合,至少一张,按书写顺序发出| 时间窗 | 写法 ||---|---|| 整个节目 | `during="program"` |-| 一个 Selection | `during={story.selection.x} map={timing.map}` |-| 一个 Moment,持续一段时长 | `at={story.moment.x} for="12f" map={timing.map}` |+| 一个 Selection | 在带有 `semantic={speech.semantic}` 的 Track 内写 `during={story.selection.x}` |+| 一个 Moment,持续一段时长 | 在带有 `semantic={speech.semantic}` 的 Track 内写 `at={story.moment.x} for="12f"` || 显式区间 | `start="…" end="…"`,可另外指定 `selection=` 或 `moment=` |凡是绑定到 Script 的都需要 `map`;不带 Script 来源的显式区间则不能给 `map`。时长写作 `12f`、`250ms` 或 `1.5s`,`occurrences="each"` 让效果在标记的每一次出现处重复,而不只是第一次。@@ -387,8 +385,8 @@ DepthStack 的直接子元素,自闭合,至少一张,按书写顺序发出可用的效果有十一种——`Flash`、`ColorWash`、`Vignette`、`ScanLines`、`DirectionalMatte`、`WhipVeil`、`GlitchVeil`、`Grain`、`LightLeak`、`Bokeh` 与 `TVStatic`——每种各有自己的必填属性,例如 `Flash` 的 `color` / `intensity` / `attack` / `hold` / `decay`,或 `Vignette` 的 `center-x` / `center-y` / `radius-x` / `radius-y` / `softness` / `color` / `opacity`。它们都没有默认值:一个效果要么把自己的形状说全,要么被拒绝。```svml-<screen:Track id="effects" space={speech.space} canvas={vertical}>- <screen:Flash during={story.selection.overlay} map={timing.map} z="80"+<screen:Track id="effects" semantic={speech.semantic} canvas={vertical}>+ <screen:Flash during={story.selection.overlay} z="80"color="#ffffff" intensity="0.6" attack="2" hold="2" decay="6"/></screen:Track>```@@ -411,7 +409,7 @@ DepthStack 的直接子元素,自闭合,至少一张,按书写顺序发出```svml<comment:Style id="social" recipe={studio.comment} font={ui-font}/>-<comment:Track id="comments" canvas={vertical} space={speech.space} map={timing.map}>+<comment:Track id="comments" canvas={vertical} semantic={speech.semantic}><comment:Sticker id="one" frame={comment-frame} style={social} avatar={viewer-avatar}author="@viewer" meta="Featured" during={story.selection.reaction}>原来它把字幕钉在词上,而不是钉在秒上。@@ -444,8 +442,7 @@ DepthStack 的直接子元素,自闭合,至少一张,按书写顺序发出<caption:Program id="caption-program" display={story.caption} default={base-caption}/><caption-ai:Planner id="cue-plan" display={story.caption}program={caption-program} model="gemini-2.5-flash"/>-<caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} map={timing.map}- space={speech.space} plan={cue-plan.plan} program={caption-program}/>+<caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} semantic={speech.semantic} plan={cue-plan.plan} program={caption-program}/><!-- 共享位置是显式边,与 Media/Text 外观分开。 --><space:Canvas id="vertical" width="1080" height="1920"/>@@ -455,14 +452,14 @@ DepthStack 的直接子元素,自闭合,至少一张,按书写顺序发出left="10%" top="20%" right="90%" bottom="70%"/><!-- Media:Selection 期间显示一个普通 Item -->-<media-track:Track id="cards" map={timing.map} space={speech.space} canvas={vertical}>+<media-track:Track id="cards" semantic={speech.semantic} canvas={vertical}><media-track:Item video={motion.video} frame={card-frame}during={story.selection.demo} appearance={studio.media.card} motion={studio.motion.card}/></media-track:Track><!-- Text: persistent title overlay --><text:Style id="title-style" recipe={studio.text.title} font={title-font}/>-<text:Track id="titles" space={speech.space}>+<text:Track id="titles" semantic={speech.semantic}><text:Area id="meaning" placement={title-frame} style={title-style} during="program">MEANING</text:Area>@@ -472,15 +469,15 @@ DepthStack 的直接子元素,自闭合,至少一张,按书写顺序发出<media:Audio id="music" src="./assets/music.wav"/><pipeline:Normalize id="music-media" source={music}video="none" audio="default" span-authority="audio" frame-rate="30"/>-<audio:Track id="music-bed" space={speech.space}>+<audio:Track id="music-bed" semantic={speech.semantic}><audio:Clip source={music-media.media} during="program"playback="loop-end" gain="0.28" fade-in="600ms" fade-out="800ms"/></audio:Track><!-- 所有对等 Track 都进入 Film -->-<film:Film id="main" canvas={vertical} space={speech.space} appearance={studio.film.vertical}>+<film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={studio.film.vertical}><film:Track source={speech.visual}/>- <film:Track source={speech.audioTrack}/>+ <film:Track source={speech.audio}/><film:Track source={cards.visual}/><film:Track source={captions.track}/><film:Track source={titles.track}/>diff --git a/examples/all-components-preview/main.svml b/examples/all-components-preview/main.svmlindex f4612664..2bba1949 100644--- a/examples/all-components-preview/main.svml+++ b/examples/all-components-preview/main.svml@@ -7,7 +7,7 @@<import as="media" from="@hypit/media@1"/><import as="pipeline" from="@hypit/media-pipeline@1"/><import as="gpt" from="@hypit/gpt-image@1"/>- <import as="speech" from="@hypit/speech-spine@1"/>+ <import as="speech" from="@hypit/speech-track@1"/><import as="whisperx" from="@hypit/whisperx@1"/><import as="media-track" from="@hypit/media-track@1"/><import as="ranking" from="@hypit/ranking@1"/>@@ -83,18 +83,18 @@<whisperx:SemanticTake id="closing-semantic" narrative={story}segment={story.segment.closing} media={closing-media.media}/>- <speech:Spine id="speech" clock={clock}+ <speech:Track id="speech"visual-frame={full} visual-appearance={recipes.speech.visual} visual-z="0"><speech:Take source={opening-semantic.take}/><speech:Take source={closing-semantic.take}/>- </speech:Spine>+ </speech:Track><fonts:Stack id="ui-font" family="inter" weight="700" style="normal"/><!-- A board whose overall lifetime and per-row reveal windows both comedirectly from Selections in the Script. --><ranking:ColumnStyle id="board-style" recipe={recipes.ranking.board} font={ui-font}/>- <ranking:Column id="board" map={speech.semanticMap} space={speech.space} canvas={vertical} frame={board-frame}+ <ranking:Column id="board" semantic={speech.semantic} canvas={vertical} frame={board-frame}during={story.selection.board} style={board-style}><ranking:ColumnItem id="row-spatial" rank="1" label="Spatial" icon={icon-spatial}during={story.selection.spatial}/>@@ -107,15 +107,15 @@</ranking:Column><!-- A card, held for exactly as long as its Selection is spoken. -->- <media-track:Track id="cards" map={speech.semanticMap} space={speech.space} canvas={vertical}>+ <media-track:Track id="cards" semantic={speech.semantic} canvas={vertical}><media-track:Item id="card" image={card-shot} extent={portrait}during={story.selection.title} frame={card-frame}appearance={recipes.media.card} motion={recipes.motion.card}/></media-track:Track><!-- A treatment over the whole picture, placed by a Selection like anything else. -->- <screen:Track id="effects" space={speech.space} canvas={vertical}>- <screen:Flash during={story.selection.overlay} map={speech.semanticMap} z="80"+ <screen:Track id="effects" semantic={speech.semantic} canvas={vertical}>+ <screen:Flash during={story.selection.overlay} z="80"color="#ffffff" intensity="0.6" attack="2" hold="2" decay="6"/></screen:Track>@@ -123,7 +123,7 @@preview was never taught about this Track; it appears because the packageproduces one. --><space:Frame id="deck-frame" within={vertical} left="44%" top="60%" right="98%" bottom="88%"/>- <deck:DepthStack id="deck" map={speech.semanticMap} space={speech.space} canvas={vertical}+ <deck:DepthStack id="deck" semantic={speech.semantic} canvas={vertical}frame={deck-frame} appearance={recipes.deck.stack} until={story.moment.done}><deck:Card id="card-spatial" source={icon-spatial} extent={square} at={story.moment.deal-one}/><deck:Card id="card-type" source={icon-type} extent={square} at={story.moment.deal-two}/>@@ -132,7 +132,7 @@</deck:DepthStack><text:Style id="title-style" recipe={recipes.text.title} font={ui-font}/>- <text:Track id="titles" space={speech.space}>+ <text:Track id="titles" semantic={speech.semantic}><text:Area id="headline" placement={title-frame} style={title-style} during="program">EVERY COMPONENT AT ONCE</text:Area>@@ -144,12 +144,12 @@model="gemini-2.5-flash"/><caption-fine:Track id="captions" display={story.caption}correspondence={story.caption.correspondence}- map={speech.semanticMap} space={speech.space}+ semantic={speech.semantic}plan={cue-plan.plan} program={caption-program}/>- <film:Film id="main" canvas={vertical} space={speech.space} appearance={recipes.film.vertical}>+ <film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={recipes.film.vertical}><film:Track source={speech.visual}/>- <film:Track source={speech.audioTrack}/>+ <film:Track source={speech.audio}/><film:Track source={board.visual}/><film:Track source={cards.visual}/><film:Track source={effects.track}/>@@ -157,5 +157,5 @@<film:Track source={titles.track}/><film:Track source={captions.track}/></film:Film>- <render:Video id="final" composition={main.composition} space={speech.space}/>+ <render:Video id="final" composition={main.composition} semantic={speech.semantic}/></svml>diff --git a/examples/street-interview-preview/README.md b/examples/street-interview-preview/README.mdindex a8ef6b04..2a02f932 100644--- a/examples/street-interview-preview/README.md+++ b/examples/street-interview-preview/README.md@@ -1,43 +1,22 @@# street-interview-preview-A four-scene street interview with three cutaways and captions, read by the-[SVML Playground](../../docs/guide/svml-playground.md) against **real footage and-real speech timings**.+A four-scene street interview with three cutaways and captions, opened in Hypit Studio against+**real footage and real speech timings**.```bash-pnpm svml:playground -- \- --source examples/street-interview-preview/main.svml \- --run examples/street-interview-preview/build.svrun+pnpm studio -- --run examples/street-interview-preview/build.svrun```-Both badges read `measured`. Nothing here has been built: the footage and the-transcript already exist, and the Run Source says so.+The checked-in Run selects the real footage. Its normalized A-roll passes through one+`whisperx:SemanticTake` per Segment, and Speech Track assembles those products into the continuous+`speech.semantic` track used by every timed component.## Where the timings come from-`docs/public/street-interview/transcript.json` is a word-level transcript of the-recording, but a transcript is not a SemanticMap. A map is keyed by *this*-Script's own anchor identities, and pairing loose words with authored tokens is-the aligner's job. `align-transcript.ts` runs the real one — `locateSpeechTiming`-from `@hypit/speech-alignment` — and writes two values:--```bash-node --import tsx examples/street-interview-preview/align-transcript.ts-# Aligned 101 transcript words onto 102 Script tokens.-```--The Run Source then satisfies the two Outputs a timeline needs:--```xml-<value id="aligned-map" type="@hypit/semantic-map@1#CompleteSemanticMap" from="./timing.json"/>-<satisfy output="timing.map" candidate="aligned-map"/>-```--Which timings a Source is read with is an authoring decision, so it is made in-the Run Source rather than guessed by the tool. Drop the two `<satisfy>` lines-and the same Source falls back to the syllable estimate, which lands on 26.0s-against a programme that runs 31.3s — the badge changes to `timing: estimated`-and the cutaways move.+Each normalized A-roll video produces acoustic evidence. WhisperX aligns that evidence against the+corresponding authored Segment and emits a self-contained `SemanticTake`: media, local word windows+and all local anchors travel together. Speech Track preserves their authored order and adds only+the prefix offsets needed to form one global `SemanticTrack`.## Where the picture comes from@@ -58,7 +37,7 @@ away from the whole frame rather than sitting inside it. `motion.cut` fades themin over two frames, which is what a cut looks like; `@receipt` uses `motion.drop`instead, so one of the three is visibly doing something else.-**Captions on the same map.** Cue times come from the aligned map, placed by the+**Captions on the same track.** Cue times come from the SemanticTrack, placed by theScript tokens each Atom corresponds to, so the words appear when they are said.**Three levels of marker.** `<rainbow>` encloses `@fee`, which encloses@@ -69,9 +48,3 @@ or not a Track was hung on it.**Markers that bind nothing.** `@seat` and `@loan` are marked in the Script butno Media Item uses them, so they are coloured in the source and absent from thetimeline. That is a legitimate state, not an error.--## Files--`timing.json` and `space.json` are generated, and committed so the example runs-without ffprobe. Regenerate them whenever the Script text changes — the map is-keyed by the Script's tokens, so editing a word invalidates it.diff --git a/examples/street-interview-preview/main.svml b/examples/street-interview-preview/main.svmlindex e3dfa1fb..34945dcb 100644--- a/examples/street-interview-preview/main.svml+++ b/examples/street-interview-preview/main.svml@@ -5,7 +5,7 @@<import as="wording" from="@hypit/text@1"/><import as="seedance" from="@hypit/seedance@1"/><import as="pipeline" from="@hypit/media-pipeline@1"/>- <import as="speech" from="@hypit/speech-spine@1"/>+ <import as="speech" from="@hypit/speech-track@1"/><import as="whisperx" from="@hypit/whisperx@1"/><import as="media-track" from="@hypit/media-track@1"/><import as="caption" from="@hypit/caption@1"/>@@ -85,13 +85,13 @@<whisperx:SemanticTake id="rainbow-semantic" narrative={story} segment={story.segment.rainbow} media={rainbow-media.media}/><whisperx:SemanticTake id="payment-semantic" narrative={story} segment={story.segment.payment} media={payment-media.media}/>- <speech:Spine id="speech" clock={clock}+ <speech:Track id="speech"visual-frame={speech-frame} visual-appearance={recipes.speech.visual} visual-z="0"><speech:Take source={hook-semantic.take}/><speech:Take source={price-semantic.take}/><speech:Take source={rainbow-semantic.take}/><speech:Take source={payment-semantic.take}/>- </speech:Spine>+ </speech:Track><fonts:Stack id="caption-font" family="inter" weight="700" style="normal"/><caption-fine:Style id="base-caption" recipe={recipes.caption.base} font={caption-font}/>@@ -100,12 +100,12 @@model="gemini-2.5-flash"/><caption-fine:Track id="captions" display={story.caption}correspondence={story.caption.correspondence}- map={speech.semanticMap} space={speech.space}+ semantic={speech.semantic}plan={cue-plan.plan} program={caption-program}/><!-- Three cutaways on one Track. Their Selections do not overlap, so theyshare a single timeline row the way an editor would lay them out. -->- <media-track:Track id="cutaways" map={speech.semanticMap} space={speech.space} canvas={vertical}>+ <media-track:Track id="cutaways" semantic={speech.semantic} canvas={vertical}><media-track:Item id="bags" media={cutaway-bags-media.media} during={story.selection.bags}frame={full} appearance={recipes.media.cutaway} motion={recipes.motion.cut}/><media-track:Item id="fee" media={cutaway-fee-media.media} during={story.selection.fee}@@ -114,11 +114,11 @@frame={full} appearance={recipes.media.cutaway} motion={recipes.motion.drop}/></media-track:Track>- <film:Film id="main" canvas={vertical} space={speech.space} appearance={recipes.film.vertical}>+ <film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={recipes.film.vertical}><film:Track source={speech.visual}/>- <film:Track source={speech.audioTrack}/>+ <film:Track source={speech.audio}/><film:Track source={cutaways.visual}/><film:Track source={captions.track}/></film:Film>- <render:Video id="final" composition={main.composition} space={speech.space}/>+ <render:Video id="final" composition={main.composition} semantic={speech.semantic}/></svml>diff --git a/examples/talking-film-broll-preview/main.svml b/examples/talking-film-broll-preview/main.svmlindex 79395ddf..31f76ead 100644--- a/examples/talking-film-broll-preview/main.svml+++ b/examples/talking-film-broll-preview/main.svml@@ -5,7 +5,7 @@<import as="wording" from="@hypit/text@1"/><import as="seedance" from="@hypit/seedance@1"/><import as="pipeline" from="@hypit/media-pipeline@1"/>- <import as="speech" from="@hypit/speech-spine@1"/>+ <import as="speech" from="@hypit/speech-track@1"/><import as="whisperx" from="@hypit/whisperx@1"/><import as="media-track" from="@hypit/media-track@1"/><import as="space" from="@hypit/spatial@1"/>@@ -43,24 +43,24 @@<whisperx:SemanticTake id="opening-semantic" narrative={story}segment={story.segment.opening} media={take-media.media}/>- <speech:Spine id="speech" clock={clock}+ <speech:Track id="speech"visual-frame={speech-frame} visual-appearance={recipes.speech.visual} visual-z="0"><speech:Take source={opening-semantic.take}/>- </speech:Spine>+ </speech:Track><!-- Two B-roll inserts on one Track. Their Selections do not overlap, so theyshare a single timeline row the way an editor would lay them out. -->- <media-track:Track id="broll" map={speech.semanticMap} space={speech.space} canvas={vertical}>+ <media-track:Track id="broll" semantic={speech.semantic} canvas={vertical}><media-track:Item id="broll-one" media={motion-a-media.media} during={story.selection.claim}frame={card-a} appearance={recipes.media.card} motion={recipes.motion.card}/><media-track:Item id="broll-two" media={motion-b-media.media} during={story.selection.proof}frame={card-b} appearance={recipes.media.wide} motion={recipes.motion.card}/></media-track:Track>- <film:Film id="main" canvas={vertical} space={speech.space} appearance={recipes.film.vertical}>+ <film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={recipes.film.vertical}><film:Track source={speech.visual}/>- <film:Track source={speech.audioTrack}/>+ <film:Track source={speech.audio}/><film:Track source={broll.visual}/></film:Film>- <render:Video id="final" composition={main.composition} space={speech.space}/>+ <render:Video id="final" composition={main.composition} semantic={speech.semantic}/></svml>diff --git a/examples/talking-film-graph-check/README.md b/examples/talking-film-graph-check/README.mdindex 9e9ab167..1ae4e056 100644--- a/examples/talking-film-graph-check/README.md+++ b/examples/talking-film-graph-check/README.md@@ -1,6 +1,6 @@# Talking-film graph check-This fixture compiles the complete author graph: Script, generated-media requests, Speech Spine,+This fixture compiles the complete author graph: Script, generated-media requests, Speech Track,WhisperX alignment, display-only Gemini Caption planning, Media Track, Caption, Text, Film andexplicit HyperFrames rendering. `check` and `plan` do not invoke Seedance, WhisperX, Gemini orHyperFrames.@@ -13,6 +13,6 @@ hypit plan build.svrun```The Gemini planner receives readable indivisible display atoms and resolved Style runs only. Its-generic `CaptionPlan` joins the independent `CompleteSemanticMap` in `@hypit/caption`; neither+generic `CaptionPlan` joins the continuous `SemanticTrack` in `@hypit/caption`; neitherthe planner nor the Vertex Provider has text-rewriting or timing authority.`@hypit/caption-fine` alone owns Recipe interpretation and the concrete visual renderer.diff --git a/examples/talking-film-graph-check/main.svml b/examples/talking-film-graph-check/main.svmlindex 6c827db6..42feef8e 100644--- a/examples/talking-film-graph-check/main.svml+++ b/examples/talking-film-graph-check/main.svml@@ -5,7 +5,7 @@<import as="wording" from="@hypit/text@1"/><import as="seedance" from="@hypit/seedance@1"/><import as="pipeline" from="@hypit/media-pipeline@1"/>- <import as="speech" from="@hypit/speech-spine@1"/>+ <import as="speech" from="@hypit/speech-track@1"/><import as="whisperx" from="@hypit/whisperx@1"/><import as="media-track" from="@hypit/media-track@1"/><import as="caption" from="@hypit/caption@1"/>@@ -43,10 +43,10 @@<whisperx:SemanticTake id="opening-semantic" narrative={story}segment={story.segment.opening} media={take-media.media}/>- <speech:Spine id="speech" clock={clock}+ <speech:Track id="speech"visual-frame={speech-frame} visual-appearance={recipes.speech.visual} visual-z="0"><speech:Take source={opening-semantic.take}/>- </speech:Spine>+ </speech:Track><fonts:Stack id="caption-font" family="inter" weight="700" style="normal"/><fonts:Stack id="title-font" family="inter" weight="900" style="normal"/>@@ -54,26 +54,26 @@<caption:Program id="caption-program" display={story.caption} default={base-caption}/><caption-ai:Planner id="cue-plan" display={story.caption} program={caption-program} model="gemini-2.5-flash"/>- <media-track:Track id="cards" map={speech.semanticMap} space={speech.space} canvas={vertical}>+ <media-track:Track id="cards" semantic={speech.semantic} canvas={vertical}><media-track:Item media={motion-media.media} during={story.selection.demo}frame={card-frame} appearance={recipes.media.card} motion={recipes.motion.card}/></media-track:Track><caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence}- map={speech.semanticMap} space={speech.space}+ semantic={speech.semantic}plan={cue-plan.plan} program={caption-program}/><text:Style id="title-style" recipe={recipes.text.title} font={title-font}/>- <text:Track id="titles" space={speech.space}>+ <text:Track id="titles" semantic={speech.semantic}><text:Area id="meaning" placement={title-frame} style={title-style} during="program">MEANING</text:Area></text:Track>- <film:Film id="main" canvas={vertical} space={speech.space} appearance={recipes.film.vertical}>+ <film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={recipes.film.vertical}><film:Track source={speech.visual}/>- <film:Track source={speech.audioTrack}/>+ <film:Track source={speech.audio}/><film:Track source={cards.visual}/><film:Track source={captions.track}/><film:Track source={titles.track}/></film:Film>- <render:Video id="final" composition={main.composition} space={speech.space}/>+ <render:Video id="final" composition={main.composition} semantic={speech.semantic}/></svml>diff --git a/examples/talking-film-live/README.md b/examples/talking-film-live/README.mdindex 93aeeb8c..365a8398 100644--- a/examples/talking-film-live/README.md+++ b/examples/talking-film-live/README.md@@ -4,7 +4,7 @@ This example deliberately exercises the real path rather than a placeholder path```textScript -> deterministic Estimate -> two Seedance Mini 480p generations- -> media normalization -> Speech Spine -> WhisperX -> SemanticMap+ -> media normalization -> WhisperX SemanticTakes -> Speech Track SemanticTrack-> Vertex Gemini Caption plan -> Caption Track-> Selection-located Text Track -> Film -> HyperFrames -> audio mix -> mux```@@ -71,5 +71,5 @@ Candidates and their explicit Satisfaction edges to another `.svrun`:The second Run compiles a new Build whose two shot outputs explicitly select historical values.It therefore contains no Seedance Operations or KIE Needs. The rebuilt media normalization,-WhisperX, SemanticMap, Tracks, Film and HyperFrames path consumes those values through ordinary+WhisperX, SemanticTrack, Tracks, Film and HyperFrames path consumes those values through ordinarytyped graph edges.diff --git a/examples/talking-film-live/main.svml b/examples/talking-film-live/main.svmlindex 57c9b5e5..4a50879d 100644--- a/examples/talking-film-live/main.svml+++ b/examples/talking-film-live/main.svml@@ -7,7 +7,7 @@<import as="pipeline" from="@hypit/media-pipeline@1"/><import as="estimate" from="@hypit/estimate@1"/><import as="seedance" from="@hypit/seedance@1"/>- <import as="speech" from="@hypit/speech-spine@1"/>+ <import as="speech" from="@hypit/speech-track@1"/><import as="whisperx" from="@hypit/whisperx@1"/><import as="caption" from="@hypit/caption@1"/><import as="caption-fine" from="@hypit/caption-fine@1"/>@@ -100,11 +100,11 @@<whisperx:SemanticTake id="answer-semantic" narrative={story}segment={story.segment.answer} media={answer-media.media}/>- <speech:Spine id="speech" clock={clock}+ <speech:Track id="speech"visual-frame={speech-frame} visual-appearance={recipes.speech.visual} visual-z="0"><speech:Take source={opening-semantic.take}/><speech:Take source={answer-semantic.take}/>- </speech:Spine>+ </speech:Track><fonts:Stack id="caption-font" family="inter" weight="700" style="normal"/><caption-fine:Style id="primary-caption" recipe={recipes.caption.primary} font={caption-font}/>@@ -121,8 +121,7 @@id="captions"display={story.caption}correspondence={story.caption.correspondence}- map={speech.semanticMap}- space={speech.space}+ semantic={speech.semantic}program={caption-program}plan={caption-plan.plan}/>@@ -131,7 +130,7 @@left="8%" top="10%" right="92%" bottom="20%"/><text:Style id="callout-style" recipe={recipes.text.callout} font={caption-font}/>- <text:Track id="callout" space={speech.space} map={speech.semanticMap}>+ <text:Track id="callout" semantic={speech.semantic}><text:Areaid="callout-copy"placement={callout-frame}@@ -142,12 +141,12 @@</text:Area></text:Track>- <film:Film id="main" canvas={vertical} space={speech.space} appearance={recipes.film.vertical}>+ <film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={recipes.film.vertical}><film:Track source={speech.visual}/>- <film:Track source={speech.audioTrack}/>+ <film:Track source={speech.audio}/><film:Track source={captions.track}/><film:Track source={callout.track}/></film:Film>- <render:Video id="final" composition={main.composition} space={speech.space}/>+ <render:Video id="final" composition={main.composition} semantic={speech.semantic}/></svml>diff --git a/examples/talking-head-aroll/README.md b/examples/talking-head-aroll/README.mdindex e420fe6f..edcad8d1 100644--- a/examples/talking-head-aroll/README.md+++ b/examples/talking-head-aroll/README.md@@ -14,9 +14,9 @@ alternate, base.The data-only `seedance-kits/speaker` template and project SVS Recipe feed generic Text rendering.Each resulting Text output connects to an exact `seedance:ReferenceVideo` beside explicit image,audio and duration edges; prompt assembly is therefore part of the graph, not hidden Speaker logic.-The four generated videos are normalized and concatenated into one Speech Spine. One canonical-audio projection goes through WhisperX, direct Script alignment produces the complete SemanticMap,-Gemini plans Cue cuts without seeing timing, and the resulting Caption+The four generated videos are normalized and independently aligned into SemanticTakes, then+concatenated by one Speech Track into a continuous SemanticTrack. Gemini plans Cue cuts without+seeing timing, and the resulting CaptionVisualTrack joins the peer speech visual/audio Tracks in Film. HyperFrames renders and muxes thesingle `final.video` target.diff --git a/examples/talking-head-aroll/main.svml b/examples/talking-head-aroll/main.svmlindex f5e4eec2..a72ae877 100644--- a/examples/talking-head-aroll/main.svml+++ b/examples/talking-head-aroll/main.svml@@ -7,7 +7,7 @@<import as="estimate" from="@hypit/estimate@1"/><import as="seedance" from="@hypit/seedance@1"/><import as="pipeline" from="@hypit/media-pipeline@1"/>- <import as="speech" from="@hypit/speech-spine@1"/>+ <import as="speech" from="@hypit/speech-track@1"/><import as="whisperx" from="@hypit/whisperx@1"/><import as="caption" from="@hypit/caption@1"/><import as="caption-fine" from="@hypit/caption-fine@1"/>@@ -171,13 +171,13 @@<whisperx:SemanticTake id="evidence-semantic" narrative={story} segment={story.segment.evidence} media={evidence-media.media}/><whisperx:SemanticTake id="payoff-semantic" narrative={story} segment={story.segment.payoff} media={payoff-media.media}/>- <speech:Spine id="speech" clock={clock}+ <speech:Track id="speech"visual-frame={speech-frame} visual-appearance={recipes.speech.visual} visual-z="0"><speech:Take source={hook-semantic.take}/><speech:Take source={meeting-semantic.take}/><speech:Take source={evidence-semantic.take}/><speech:Take source={payoff-semantic.take}/>- </speech:Spine>+ </speech:Track><!-- Gemini sees display words and these requirements, never audio or timing. --><fonts:Stack id="caption-font" family="inter" weight="800" style="normal"/>@@ -196,17 +196,16 @@id="captions"display={story.caption}correspondence={story.caption.correspondence}- map={speech.semanticMap}- space={speech.space}+ semantic={speech.semantic}program={caption-program}plan={caption-plan.plan}/>- <film:Film id="main" canvas={vertical} space={speech.space} appearance={recipes.film.vertical}>+ <film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={recipes.film.vertical}><film:Track source={speech.visual}/>- <film:Track source={speech.audioTrack}/>+ <film:Track source={speech.audio}/><film:Track source={captions.track}/></film:Film>- <render:Video id="final" composition={main.composition} space={speech.space}/>+ <render:Video id="final" composition={main.composition} semantic={speech.semantic}/></svml>diff --git a/package.json b/package.jsonindex 557a16ab..60a327dd 100644--- a/package.json+++ b/package.json@@ -82,13 +82,12 @@"@hypit/seedance": "workspace:*","@hypit/seedance-kits": "workspace:*","@hypit/seedream": "workspace:*",- "@hypit/semantic-map": "workspace:*",+ "@hypit/semantic-track": "workspace:*","@hypit/spatial": "workspace:*","@hypit/speech": "workspace:*","@hypit/speech-alignment": "workspace:*",- "@hypit/speech-basis": "workspace:*","@hypit/speech-evidence": "workspace:*",- "@hypit/speech-spine": "workspace:*",+ "@hypit/speech-track": "workspace:*","@hypit/store-sqlite": "workspace:*","@hypit/svs": "workspace:*","@hypit/temporal": "workspace:*",diff --git a/packages/audio-track/package.json b/packages/audio-track/package.jsonindex 57ca95ff..cc6c9805 100644--- a/packages/audio-track/package.json+++ b/packages/audio-track/package.json@@ -18,7 +18,7 @@"@hypit/narrative": "workspace:*","@hypit/program-space": "workspace:*","@hypit/protocol": "workspace:*",- "@hypit/semantic-map": "workspace:*",+ "@hypit/semantic-track": "workspace:*","@hypit/temporal": "workspace:*","@hypit/markup": "workspace:*"},diff --git a/packages/audio-track/src/component.ts b/packages/audio-track/src/component.tsindex c4357ab1..e795bdd7 100644--- a/packages/audio-track/src/component.ts+++ b/packages/audio-track/src/component.ts@@ -1,10 +1,10 @@import type { ComponentPackage } from "@hypit/component-kit";import type { SynchronizedMedia } from "@hypit/media";import type { NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrative";-import type { ProgramSpace } from "@hypit/program-space";import { canonicalize } from "@hypit/protocol";import type { StoredValue } from "@hypit/protocol";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import type { SemanticTrack } from "@hypit/semantic-track";+import { projectSemanticProgramSpace } from "@hypit/semantic-track";import { audioTrackProducers, audioTrackTypes } from "./manifest.js";import { appendMomentAudioItem, appendProgramAudioItem, appendSelectionAudioItem, assertAudioTrackProgram, createAudioTrackSet, finalizeAudioTrack, renderAudioTrack } from "./program.js";@@ -23,26 +23,26 @@ export const audioTrackComponent = {{ producer: audioTrackProducers.createSet, handler: () => ({ outputs: { set: output(createAudioTrackSet()) }, needs: {} }) },{ producer: audioTrackProducers.appendProgram, handler: ({ inputs }) => ({ outputs: { set: base([inline<AudioTrackSet>(inputs.set?.value, "AudioTrackSet"), inline<AudioTrackHeader>(inputs.header?.value, "AudioTrackHeader"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),+ inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"), inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),inline<AudioClipSpec>(inputs.spec?.value, "AudioClipSpec"),]) }, needs: {} }) },{ producer: audioTrackProducers.appendSelection, handler: ({ inputs }) => ({ outputs: { set: output(appendSelectionAudioItem(inline<AudioTrackSet>(inputs.set?.value, "AudioTrackSet"), inline<AudioTrackHeader>(inputs.header?.value, "AudioTrackHeader"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"), inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelection"),+ inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"), inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),+ inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelection"),inline<AudioClipSpec>(inputs.spec?.value, "AudioClipSpec"),)) }, needs: {} }) },{ producer: audioTrackProducers.appendMoment, handler: ({ inputs }) => ({ outputs: { set: output(appendMomentAudioItem(inline<AudioTrackSet>(inputs.set?.value, "AudioTrackSet"), inline<AudioTrackHeader>(inputs.header?.value, "AudioTrackHeader"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"), inline<NarrativeMomentRef>(inputs.moment?.value, "NarrativeMoment"),+ inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"), inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),+ inline<NarrativeMomentRef>(inputs.moment?.value, "NarrativeMoment"),inline<AudioClipSpec>(inputs.spec?.value, "AudioClipSpec"),)) }, needs: {} }) },{ producer: audioTrackProducers.finalize, handler: ({ inputs }) => ({ outputs: { program: output(finalizeAudioTrack(inline<AudioTrackSet>(inputs.set?.value, "AudioTrackSet"), inline<AudioTrackHeader>(inputs.header?.value, "AudioTrackHeader"),)) }, needs: {} }) },{ producer: audioTrackProducers.render, handler: ({ inputs }) => ({ outputs: { track: output(renderAudioTrack(- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<AudioTrackProgram>(inputs.program?.value, "AudioTrackProgram"),+ projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")), inline<AudioTrackProgram>(inputs.program?.value, "AudioTrackProgram"),)) }, needs: {} }) },],validators: [{diff --git a/packages/audio-track/src/fragment.ts b/packages/audio-track/src/fragment.tsindex 500bba81..691048e2 100644--- a/packages/audio-track/src/fragment.ts+++ b/packages/audio-track/src/fragment.ts@@ -3,22 +3,21 @@ import { sealGraphFragment } from "@hypit/elaborator";import type { FragmentOperation } from "@hypit/elaborator";import { mediaTypes } from "@hypit/media";import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { audioTrackProducers, audioTrackTypes } from "./manifest.js";export type AudioTrackFragmentItem =| { readonly kind: "program"; readonly mediaName: string; readonly specName: string }- | { readonly kind: "selection"; readonly mediaName: string; readonly specName: string; readonly mapName: string; readonly sourceName: string }- | { readonly kind: "moment"; readonly mediaName: string; readonly specName: string; readonly mapName: string; readonly sourceName: string };+ | { readonly kind: "selection"; readonly mediaName: string; readonly specName: string; readonly sourceName: string }+ | { readonly kind: "moment"; readonly mediaName: string; readonly specName: string; readonly sourceName: string };const input = (name: string) => ({ kind: "fragment-input" as const, name });const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });export function createAudioTrackFragment(items: readonly AudioTrackFragmentItem[]) {if (items.length === 0) throw new Error("Audio Track Fragment requires at least one Item.");- const inputTypes = new Map<string, (typeof audioTrackTypes.clipSpec | typeof mediaTypes.synchronized | typeof semanticMapTypes.complete | typeof narrativeTypes.selection | typeof narrativeTypes.moment)>();+ const inputTypes = new Map<string, (typeof audioTrackTypes.clipSpec | typeof mediaTypes.synchronized | typeof narrativeTypes.selection | typeof narrativeTypes.moment)>();const operations: FragmentOperation[] = [{ id: "audio:set:empty", producer: audioTrackProducers.createSet, inputs: {}, result: { kind: "output", name: "set" } },];@@ -27,7 +26,6 @@ export function createAudioTrackFragment(items: readonly AudioTrackFragmentItem[inputTypes.set(item.mediaName, mediaTypes.synchronized);inputTypes.set(item.specName, audioTrackTypes.clipSpec);if (item.kind !== "program") {- inputTypes.set(item.mapName, semanticMapTypes.complete);inputTypes.set(item.sourceName, item.kind === "selection" ? narrativeTypes.selection : narrativeTypes.moment);}const id = `audio:set:append:${String(index + 1).padStart(4, "0")}`;@@ -39,10 +37,9 @@ export function createAudioTrackFragment(items: readonly AudioTrackFragmentItem[? audioTrackProducers.appendSelection: audioTrackProducers.appendMoment,inputs: {- set: operation(current), header: input("header"), space: input("space"),+ set: operation(current), header: input("header"), semantic: input("semantic"),media: input(item.mediaName), spec: input(item.specName),...(item.kind === "program" ? {} : {- map: input(item.mapName),[item.kind]: input(item.sourceName),}),},@@ -52,12 +49,12 @@ export function createAudioTrackFragment(items: readonly AudioTrackFragmentItem[});operations.push({ id: "audio:program", producer: audioTrackProducers.finalize, inputs: { set: operation(current), header: input("header") }, result: { kind: "output", name: "program" } },- { id: "audio:track", producer: audioTrackProducers.render, inputs: { space: input("space"), program: operation("audio:program") }, result: { kind: "output", name: "track" } },+ { id: "audio:track", producer: audioTrackProducers.render, inputs: { semantic: input("semantic"), program: operation("audio:program") }, result: { kind: "output", name: "track" } },);return sealGraphFragment({inputs: [{ name: "header", type: audioTrackTypes.header },- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },...[...inputTypes].map(([inputName, type]) => ({ name: inputName, type })),],operations,diff --git a/packages/audio-track/src/manifest.ts b/packages/audio-track/src/manifest.tsindex b94c8c97..0c75b24e 100644--- a/packages/audio-track/src/manifest.ts+++ b/packages/audio-track/src/manifest.ts@@ -1,10 +1,9 @@import { compositionDependency, compositionTypes } from "@hypit/composition";import { mediaDependency, mediaTypes } from "@hypit/media";import { narrativeDependency, narrativeTypes } from "@hypit/narrative";-import { programSpaceDependency, programSpaceTypes } from "@hypit/program-space";import { blobRefObjectSchema } from "@hypit/protocol";import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@hypit/protocol";-import { semanticMapDependency, semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackDependency, semanticTrackTypes } from "@hypit/semantic-track";import { temporalDependency } from "@hypit/temporal";export const audioTrackModuleRef = { name: "@hypit/audio-track", version: "1" } as const;@@ -82,7 +81,7 @@ export const audioTrackProgramSchema: ValueSchema = object({const baseInputs = [{ name: "set", type: audioTrackTypes.set }, { name: "header", type: audioTrackTypes.header },- { name: "space", type: programSpaceTypes.programSpace }, { name: "media", type: mediaTypes.synchronized },+ { name: "semantic", type: semanticTrackTypes.track }, { name: "media", type: mediaTypes.synchronized },{ name: "spec", type: audioTrackTypes.clipSpec },] as const;@@ -93,7 +92,7 @@ export const audioTrackMarkupSurfaces = [{summary: "One Audio Track: explicitly prepared audio Clips placed on a shared ProgramSpace and lowered to one ordinary peer AudioTrack.",attributes: [{ name: "id", kind: "identifier", required: true, summary: "Names this Audio Track and prefixes the identity of every Clip that does not name itself." },- { name: "space", kind: "reference", required: true, accepts: [programSpaceTypes.programSpace], summary: "Fixes the frame and sample domain every Clip window resolves into." },+ { name: "semantic", kind: "reference", required: true, accepts: [semanticTrackTypes.track], summary: "Selects the continuous semantic frame domain every Clip window resolves into." },],children: [{ tag: "Clip", cardinality: "many",@@ -122,9 +121,6 @@ export const audioTrackMarkupSurfaces = [{{ name: "moment", kind: "reference", required: false,accepts: [narrativeTypes.moment],summary: "Binds the Moment that resolves `moment.cue` in a start/end window." },- { name: "map", kind: "reference", required: false,- accepts: [semanticMapTypes.complete],- summary: "Selects the SemanticMap that turns semantic identity into exact time." },{ name: "occurrences", kind: "literal", required: false, values: ["one", "each"],summary: "Decides whether a semantic source contributes one window or every occurrence; defaults to `one`." },{ name: "trim-start", kind: "literal", required: false,@@ -150,7 +146,7 @@ export const audioTrackMarkupSurfaces = [{{ name: "program", type: audioTrackTypes.program, summary: "The resolved sample-exact item list this Track renders from." },{ name: "track", type: compositionTypes.audioTrack, summary: "The rendered AudioTrack that Film composes with its peers." },],- example: `<audio:Track id="music-bed" space={speech.space}>+ example: `<audio:Track id="music-bed" semantic={speech.semantic}><audio:Clip source={music-media.media} during="program"playback="loop-end" gain="0.28" fade-in="600ms" fade-out="800ms"/></audio:Track>`,@@ -158,7 +154,7 @@ export const audioTrackMarkupSurfaces = [{"A Track requires at least one Clip, and neither a Track nor a Clip accepts text content.","A Clip states exactly one window form: `during`, `at` with `for`, or `start` with `end`.","A point expression is `program.start`, `program.end`, `selection.start`, `selection.end` or `moment.cue`, each optionally offset by `+` or `-` and a duration, or a bare duration read as an absolute position.",- "`selection` and `moment` cannot be written together, and `map` is rejected on a start/end window that binds neither.",+ "`selection` and `moment` cannot be written together.","`min-rate` and `max-rate` are rejected unless `playback` is `stretch`.",],},@@ -169,7 +165,7 @@ export const audioTrackManifest: ModuleManifest = {format: "hypit.module@1",name: audioTrackModuleRef.name,version: audioTrackModuleRef.version,- dependencies: [mediaDependency, narrativeDependency, semanticMapDependency, programSpaceDependency, temporalDependency, compositionDependency],+ dependencies: [mediaDependency, narrativeDependency, semanticTrackDependency, temporalDependency, compositionDependency],types: [{ name: audioTrackTypes.header.name },{ name: audioTrackTypes.clipSpec.name },@@ -180,10 +176,10 @@ export const audioTrackManifest: ModuleManifest = {producers: [{ name: audioTrackProducers.createSet.name, inputs: [], outputs: [{ name: "set", type: audioTrackTypes.set }], needs: [] },{ name: audioTrackProducers.appendProgram.name, inputs: [...baseInputs], outputs: [{ name: "set", type: audioTrackTypes.set }], needs: [] },- { name: audioTrackProducers.appendSelection.name, inputs: [...baseInputs, { name: "map", type: semanticMapTypes.complete }, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: audioTrackTypes.set }], needs: [] },- { name: audioTrackProducers.appendMoment.name, inputs: [...baseInputs, { name: "map", type: semanticMapTypes.complete }, { name: "moment", type: narrativeTypes.moment }], outputs: [{ name: "set", type: audioTrackTypes.set }], needs: [] },+ { name: audioTrackProducers.appendSelection.name, inputs: [...baseInputs, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: audioTrackTypes.set }], needs: [] },+ { name: audioTrackProducers.appendMoment.name, inputs: [...baseInputs, { name: "moment", type: narrativeTypes.moment }], outputs: [{ name: "set", type: audioTrackTypes.set }], needs: [] },{ name: audioTrackProducers.finalize.name, inputs: [{ name: "set", type: audioTrackTypes.set }, { name: "header", type: audioTrackTypes.header }], outputs: [{ name: "program", type: audioTrackTypes.program }], needs: [] },- { name: audioTrackProducers.render.name, inputs: [{ name: "space", type: programSpaceTypes.programSpace }, { name: "program", type: audioTrackTypes.program }], outputs: [{ name: "track", type: compositionTypes.audioTrack }], needs: [] },+ { name: audioTrackProducers.render.name, inputs: [{ name: "semantic", type: semanticTrackTypes.track }, { name: "program", type: audioTrackTypes.program }], outputs: [{ name: "track", type: compositionTypes.audioTrack }], needs: [] },],};export const audioTrackDependency = { module: audioTrackModuleRef } as const;diff --git a/packages/audio-track/src/program.ts b/packages/audio-track/src/program.tsindex e935c561..1db3fadf 100644--- a/packages/audio-track/src/program.ts+++ b/packages/audio-track/src/program.ts@@ -10,7 +10,8 @@ import {} from "@hypit/program-space";import type { ProgramSpace } from "@hypit/program-space";import { canonicalize, isDigest } from "@hypit/protocol";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import { projectSemanticProgramSpace } from "@hypit/semantic-track";+import type { SemanticTrack } from "@hypit/semantic-track";import {projectMomentWindows,projectProgramWindow,@@ -108,11 +109,12 @@ function sourceFacts(media: SynchronizedMedia): AudioItemProgram["source"] {function realizedItems(set: AudioTrackSet,header: AudioTrackHeader,- space: ProgramSpace,+ semantic: SemanticTrack,media: SynchronizedMedia,spec: AudioClipSpec,occurrences: readonly ProjectedOccurrence[],): AudioTrackSet {+ const space = projectSemanticProgramSpace(semantic);assertAudioTrackSet(set);assertAudioTrackHeader(header);assertProgramSpaceIdentity(space);@@ -143,14 +145,14 @@ function realizedItems(export function appendProgramAudioItem(set: AudioTrackSet,header: AudioTrackHeader,- space: ProgramSpace,+ semantic: SemanticTrack,media: SynchronizedMedia,spec: AudioClipSpec,): AudioTrackSet {assert(spec.expansion.kind === "one", `Program Audio Clip ${spec.id} must use one occurrence.`);- return realizedItems(set, header, space, media, spec, [projectProgramWindow({+ return realizedItems(set, header, semantic, media, spec, [projectProgramWindow({itemId: spec.id,- space,+ semantic,projection: spec.projection,})]);}@@ -158,28 +160,26 @@ export function appendProgramAudioItem(export function appendSelectionAudioItem(set: AudioTrackSet,header: AudioTrackHeader,- space: ProgramSpace,+ semantic: SemanticTrack,media: SynchronizedMedia,- map: CompleteSemanticMap,selection: NarrativeSelectionRef,spec: AudioClipSpec,): AudioTrackSet {- return realizedItems(set, header, space, media, spec, projectSelectionWindows({- itemId: spec.id, map, selection, space, expansion: spec.expansion, projection: spec.projection,+ return realizedItems(set, header, semantic, media, spec, projectSelectionWindows({+ itemId: spec.id, semantic, selection, expansion: spec.expansion, projection: spec.projection,}));}export function appendMomentAudioItem(set: AudioTrackSet,header: AudioTrackHeader,- space: ProgramSpace,+ semantic: SemanticTrack,media: SynchronizedMedia,- map: CompleteSemanticMap,moment: NarrativeMomentRef,spec: AudioClipSpec,): AudioTrackSet {- return realizedItems(set, header, space, media, spec, projectMomentWindows({- itemId: spec.id, map, moment, space, expansion: spec.expansion, projection: spec.projection,+ return realizedItems(set, header, semantic, media, spec, projectMomentWindows({+ itemId: spec.id, semantic, moment, expansion: spec.expansion, projection: spec.projection,}));}diff --git a/packages/audio-track/src/surface.ts b/packages/audio-track/src/surface.tsindex 0f3bca96..4d5e9412 100644--- a/packages/audio-track/src/surface.ts+++ b/packages/audio-track/src/surface.ts@@ -1,7 +1,6 @@import { mediaTypes } from "@hypit/media";import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import type {StructuredElement,StructuredSurfaceHandler,@@ -128,7 +127,6 @@ type TemporalBinding = {readonly kind: "program" | "selection" | "moment";readonly projection: AudioClipSpec["projection"];readonly source?: SurfaceResolvedReference;- readonly map?: SurfaceResolvedReference;};function temporalBinding(@@ -141,7 +139,6 @@ function temporalBinding(const explicitEnd = optionalText(element, "end");const selection = element.attributes.selection;const moment = element.attributes.moment;- const mapRaw = element.attributes.map;const forms = Number(during !== undefined) + Number(at !== undefined) + Number(explicitStart !== undefined || explicitEnd !== undefined);if (forms !== 1) throw new Error(`${element.name} requires exactly one of during, at/for, or start/end.`);if (during !== undefined) {@@ -150,14 +147,12 @@ function temporalBinding(return { kind: "program", projection: { start: { ref: "program.start" }, end: { ref: "program.end" } } };}const source = resolved(during, `${element.name}.during`, narrativeTypes.selection, resolve);- const map = resolved(mapRaw, `${element.name}.map`, semanticMapTypes.complete, resolve);- return { kind: "selection", source, map, projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } } };+ return { kind: "selection", source, projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } } };}if (at !== undefined) {const source = resolved(at, `${element.name}.at`, narrativeTypes.moment, resolve);- const map = resolved(mapRaw, `${element.name}.map`, semanticMapTypes.complete, resolve);const forDuration = duration(text(element, "for"), `${element.name}.for`);- return { kind: "moment", source, map, projection: {+ return { kind: "moment", source, projection: {start: { ref: "moment.cue" }, end: { ref: "moment.cue", offset: forDuration },} };}@@ -166,16 +161,13 @@ function temporalBinding(if (selection !== undefined) return {kind: "selection",source: resolved(selection, `${element.name}.selection`, narrativeTypes.selection, resolve),- map: resolved(mapRaw, `${element.name}.map`, semanticMapTypes.complete, resolve),projection: { start: point(explicitStart, `${element.name}.start`), end: point(explicitEnd, `${element.name}.end`) },};if (moment !== undefined) return {kind: "moment",source: resolved(moment, `${element.name}.moment`, narrativeTypes.moment, resolve),- map: resolved(mapRaw, `${element.name}.map`, semanticMapTypes.complete, resolve),projection: { start: point(explicitStart, `${element.name}.start`), end: point(explicitEnd, `${element.name}.end`) },};- if (mapRaw !== undefined) throw new Error(`${element.name}.map requires Selection or Moment.`);return {kind: "program",projection: { start: point(explicitStart, `${element.name}.start`), end: point(explicitEnd, `${element.name}.end`) },@@ -183,9 +175,9 @@ function temporalBinding(}export const decodeAudioTrackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- allowed(element, ["id", "space"]);+ allowed(element, ["id", "semantic"]);const id = text(element, "id");- const space = resolved(element.attributes.space, `${element.name}.space`, programSpaceTypes.programSpace, resolveReference);+ const semantic = resolved(element.attributes.semantic, `${element.name}.semantic`, semanticTrackTypes.track, resolveReference);const headerId = `${id}.header`;const records: SurfaceRecordDraft[] = [{id: headerId,@@ -196,7 +188,7 @@ export const decodeAudioTrackSurface: StructuredSurfaceHandler = ({ element, resconst fragmentItems: Parameters<typeof createAudioTrackFragment>[0][number][] = [];const inputs: Record<string, { kind: "record"; id: string } | { kind: "component-output"; component: string; output: string }> = {header: { kind: "record", id: headerId },- space: space.ref,+ semantic: semantic.ref,};let itemIndex = 0;for (const child of element.children) {@@ -207,7 +199,7 @@ export const decodeAudioTrackSurface: StructuredSurfaceHandler = ({ element, resif (!child.name.endsWith(":Clip") && child.name !== "Clip") throw new Error(`${element.name} accepts only Clip children.`);if (child.children.some((node) => node.kind === "element" || node.value.trim())) throw new Error(`${child.name} must be empty.`);allowed(child, [- "id", "source", "during", "at", "for", "start", "end", "selection", "moment", "map", "occurrences",+ "id", "source", "during", "at", "for", "start", "end", "selection", "moment", "occurrences","trim-start", "trim-end", "playback", "min-rate", "max-rate", "gain", "fade-in", "fade-out",]);itemIndex += 1;@@ -246,11 +238,9 @@ export const decodeAudioTrackSurface: StructuredSurfaceHandler = ({ element, resif (binding.kind === "program") {fragmentItems.push({ kind: "program", mediaName, specName });} else {- const mapName = `item-${suffix}-map`;const sourceName = `item-${suffix}-${binding.kind}`;- inputs[mapName] = binding.map!.ref;inputs[sourceName] = binding.source!.ref;- fragmentItems.push({ kind: binding.kind, mediaName, specName, mapName, sourceName });+ fragmentItems.push({ kind: binding.kind, mediaName, specName, sourceName });}}if (fragmentItems.length === 0) throw new Error(`${element.name} requires at least one Clip.`);diff --git a/packages/audio-track/test/audio-track.test.ts b/packages/audio-track/test/audio-track.test.tsindex 29bb002d..7ae36200 100644--- a/packages/audio-track/test/audio-track.test.ts+++ b/packages/audio-track/test/audio-track.test.ts@@ -1,6 +1,7 @@import assert from "node:assert/strict";import test from "node:test";import { fixtureDigest } from "../../../test/fixture-digest.js";+import { semanticTrackFixture } from "../../../test/semantic-track-fixture.js";import {audioTrackManifest,@@ -31,8 +32,7 @@ import { narrativeManifest } from "@hypit/narrative";import { compileAudioProgramPlan } from "@hypit/media-pipeline";import { programSpaceDependency, programSpaceManifest, programSpaceTypes, sealProgramSpace } from "@hypit/program-space";import type { ModuleManifest } from "@hypit/protocol";-import type { CompleteSemanticMap } from "@hypit/semantic-map";-import { semanticMapManifest } from "@hypit/semantic-map";+import { semanticTrackDependency, semanticTrackManifest, semanticTrackTypes } from "@hypit/semantic-track";import { speechEvidenceManifest } from "@hypit/speech-evidence";import { speechManifest } from "@hypit/speech";import { spatialManifest } from "@hypit/spatial";@@ -46,6 +46,10 @@ const space = sealProgramSpace({frameRate: { numerator: 30, denominator: 1 },});const header = sealAudioTrackHeader({ id: "sound" });+const semantic = semanticTrackFixture(space, { anchors: [+ { identity: "a", frame: 30 }, { identity: "b", frame: 60 },+ { identity: "c", frame: 90 }, { identity: "d", frame: 120 },+] });const zero = { unit: "frames" as const, value: 0 };function media(id: string, sampleFrames: number): SynchronizedMedia {@@ -77,7 +81,7 @@ function spec(overrides: Partial<AudioClipSpec> = {}): AudioClipSpec {}function programTrack(source: SynchronizedMedia, clipSpec: AudioClipSpec) {- const set = appendProgramAudioItem(createAudioTrackSet(), header, space, source, clipSpec);+ const set = appendProgramAudioItem(createAudioTrackSet(), header, semantic, source, clipSpec);return renderAudioTrack(space, finalizeAudioTrack(set, header));}@@ -147,16 +151,6 @@ test("trim and fades quantize once into the same sample domain", () => {})), /fade exceeds/u);});-const map: CompleteSemanticMap = {- tokens: [],- anchors: [- { identity: "a", frame: 30 },- { identity: "b", frame: 60 },- { identity: "c", frame: 90 },- { identity: "d", frame: 120 },- ],-};-test("Selection and Moment each expansion creates independent overlapping items", () => {const selection: NarrativeSelectionRef = {@@ -172,12 +166,12 @@ test("Selection and Moment each expansion creates independent overlapping items"occurrences: [{ occurrence: 0, anchorId: "a" }, { occurrence: 1, anchorId: "c" }],};let set: AudioTrackSet = createAudioTrackSet();- set = appendSelectionAudioItem(set, header, space, media("selection", 48_000), map, selection, spec({+ set = appendSelectionAudioItem(set, header, semantic, media("selection", 48_000), selection, spec({id: "selected",projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } },expansion: { kind: "each" },}));- set = appendMomentAudioItem(set, header, space, media("moment", 48_000), map, moment, spec({+ set = appendMomentAudioItem(set, header, semantic, media("moment", 48_000), moment, spec({id: "hit",projection: { start: { ref: "moment.cue" }, end: { ref: "moment.cue", offset: { unit: "seconds", numerator: 1, denominator: 1 } } },expansion: { kind: "each" },@@ -220,11 +214,11 @@ test("one Track with overlaps and two peer Tracks compile to the same deterministest("dynamic Fragment keeps every material and temporal dependency as an explicit input", () => {const fragment = createAudioTrackFragment([{ kind: "program", mediaName: "music", specName: "music-spec" },- { kind: "selection", mediaName: "voice", specName: "voice-spec", mapName: "map", sourceName: "selection" },- { kind: "moment", mediaName: "impact", specName: "impact-spec", mapName: "map", sourceName: "moment" },+ { kind: "selection", mediaName: "voice", specName: "voice-spec", sourceName: "selection" },+ { kind: "moment", mediaName: "impact", specName: "impact-spec", sourceName: "moment" },]);assert.deepEqual(fragment.inputs.map((input) => input.name), [- "header", "impact", "impact-spec", "map", "moment", "music", "music-spec", "selection", "space", "voice", "voice-spec",+ "header", "impact", "impact-spec", "moment", "music", "music-spec", "selection", "semantic", "voice", "voice-spec",]);assert.equal(fragment.exports[1]?.name, "track");});@@ -234,13 +228,13 @@ test("the self-described Audio Surface parses into the same finite Producer grapconst fixtureSurfaceDigest = fixtureDigest("example.audio-inputs/surface@1");const fixtureSurface = {name: "inputs", tag: "Inputs", mode: "structured",- outputs: [mediaTypes.synchronized, programSpaceTypes.programSpace],+ outputs: [mediaTypes.synchronized, semanticTrackTypes.track],} as const;const fixtureManifest: ModuleManifest = {format: "hypit.module@1",name: fixtureModule.name,version: fixtureModule.version,- dependencies: [mediaDependency, programSpaceDependency],+ dependencies: [mediaDependency, semanticTrackDependency],types: [],capabilities: [],producers: [],@@ -253,7 +247,7 @@ test("the self-described Audio Surface parses into the same finite Producer grapspeechManifest,spatialManifest,speechEvidenceManifest,- semanticMapManifest,+ semanticTrackManifest,temporalManifest,visualIrManifest,compositionManifest,@@ -264,7 +258,7 @@ test("the self-described Audio Surface parses into the same finite Producer grapregistry.registerStructured({ module: fixtureModule, declaration: fixtureSurface, handler: ({ element }) => ({records: [{ id: "source", type: mediaTypes.synchronized, value: { kind: "inline", value: media("surface", 48_000) }, range: element.range },- { id: "space", type: programSpaceTypes.programSpace, value: { kind: "inline", value: space }, range: element.range },+ { id: "semantic", type: semanticTrackTypes.track, value: { kind: "inline", value: semantic }, range: element.range },],components: [],fragments: [],@@ -290,7 +284,7 @@ test("the self-described Audio Surface parses into the same finite Producer grap<import as="fixture" from="example.audio-inputs@1"/><import as="audio" from="@hypit/audio-track@1"/><fixture:Inputs/>- <audio:Track id="sound" space={space}>+ <audio:Track id="sound" semantic={semantic}><audio:Clip source={source} during="program" playback="loop-end" gain="0.5" fade-in="2f" fade-out="3f"/></audio:Track></svml>`,diff --git a/packages/caption-fine/README.md b/packages/caption-fine/README.mdindex 9f52f50b..5e147bec 100644--- a/packages/caption-fine/README.md+++ b/packages/caption-fine/README.md@@ -21,10 +21,9 @@ declares no planning fields and has no `important`, random sizing or inferred Woid="captions-track"display={story.caption}correspondence={story.caption.correspondence}- map={timing.map}+ semantic={speech.semantic}program={captions}plan={caption-plan.plan}- space={speech.space}/>```diff --git a/packages/caption-fine/package.json b/packages/caption-fine/package.jsonindex 836eea46..b03a1ad1 100644--- a/packages/caption-fine/package.json+++ b/packages/caption-fine/package.json@@ -19,7 +19,7 @@"@hypit/narrative": "workspace:*","@hypit/program-space": "workspace:*","@hypit/protocol": "workspace:*",- "@hypit/semantic-map": "workspace:*",+ "@hypit/semantic-track": "workspace:*","@hypit/svs": "workspace:*","@hypit/markup": "workspace:*"},diff --git a/packages/caption-fine/src/component.ts b/packages/caption-fine/src/component.tsindex 10453398..2648f2d0 100644--- a/packages/caption-fine/src/component.ts+++ b/packages/caption-fine/src/component.ts@@ -1,7 +1,8 @@import type { CaptionProgram, TimedCaptionProjection } from "@hypit/caption";import type { ComponentPackage } from "@hypit/component-kit";import type { CaptionDisplaySequence } from "@hypit/narrative";-import type { ProgramSpace } from "@hypit/program-space";+import { projectSemanticProgramSpace } from "@hypit/semantic-track";+import type { SemanticTrack } from "@hypit/semantic-track";import type { StoredValue } from "@hypit/protocol";import { canonicalize } from "@hypit/protocol";@@ -21,7 +22,7 @@ export const captionFineComponent = {inline<TimedCaptionProjection>(inputs.caption?.value, "TimedCaptionProjection"),inline<CaptionProgram>(inputs.program?.value, "CaptionProgram"),inline<CaptionDisplaySequence>(inputs.display?.value, "CaptionDisplaySequence"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")),)) } },needs: {},}),diff --git a/packages/caption-fine/src/fragment.ts b/packages/caption-fine/src/fragment.tsindex dde728fa..f49368c8 100644--- a/packages/caption-fine/src/fragment.ts+++ b/packages/caption-fine/src/fragment.ts@@ -2,8 +2,7 @@ import { captionProducers, captionTypes } from "@hypit/caption";import { compositionTypes } from "@hypit/composition";import { sealGraphFragment } from "@hypit/elaborator";import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { captionFineProducers } from "./manifest.js";@@ -15,17 +14,16 @@ export const fineCaptionTrackFragment = sealGraphFragment({inputs: [{ name: "display", type: narrativeTypes.captionDisplay },{ name: "correspondence", type: narrativeTypes.captionCorrespondence },- { name: "map", type: semanticMapTypes.complete },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "plan", type: captionTypes.plan },{ name: "program", type: captionTypes.program },- { name: "space", type: programSpaceTypes.programSpace },],operations: [{id: "caption:temporalize-plan",producer: captionProducers.temporalizePlan,inputs: {- display: input("display"), correspondence: input("correspondence"), map: input("map"),+ display: input("display"), correspondence: input("correspondence"), semantic: input("semantic"),plan: input("plan"), program: input("program"),},result: { kind: "output", name: "caption" },@@ -35,7 +33,7 @@ export const fineCaptionTrackFragment = sealGraphFragment({producer: captionFineProducers.render,inputs: {caption: operation("caption:temporalize-plan"), program: input("program"),- display: input("display"), space: input("space"),+ display: input("display"), semantic: input("semantic"),},result: { kind: "output", name: "track" },},diff --git a/packages/caption-fine/src/manifest.ts b/packages/caption-fine/src/manifest.tsindex 7937f926..427c46e2 100644--- a/packages/caption-fine/src/manifest.ts+++ b/packages/caption-fine/src/manifest.ts@@ -3,9 +3,8 @@ import { readFile } from "node:fs/promises";import { captionManifest, captionModuleRef, captionTypes } from "@hypit/caption";import { compositionDependency, compositionTypes } from "@hypit/composition";import { mediaDependency, mediaTypes } from "@hypit/media";-import { programSpaceDependency, programSpaceTypes } from "@hypit/program-space";import { narrativeDependency, narrativeTypes } from "@hypit/narrative";-import { semanticMapDependency, semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackDependency, semanticTrackTypes } from "@hypit/semantic-track";import { svsRecipeType } from "@hypit/svs";import type { ModuleManifest, ProducerRef } from "@hypit/protocol";@@ -251,7 +250,7 @@ export const captionFineMarkupSurfaces = [{name: "track", tag: "Track", mode: "structured", outputs: [compositionTypes.visualTrack],vocabulary: {- summary: "Joins a planned Caption against the measured SemanticMap and renders it as one ordinary peer VisualTrack.",+ summary: "Joins a planned Caption against the SemanticTrack and renders it as one ordinary peer VisualTrack.",appearance: "One block of caption text wrapped into lines inside a rounded Cue box, placed at a Recipe-chosen point on the Canvas and spanning a fraction of its width. Cues follow the speech one after another, each arriving and leaving with its own motion, and the Words of a Cue either stand there together from its first Frame or uncover as they are spoken. As the speech advances, the Word being spoken, or every Word up to it, is repainted in the active Paint, snapping at the Word boundary or sweeping across the glyphs, and it may take a rule beneath it, a rounded highlight capsule behind it and a brief pop at the moment it becomes the spoken one. That capsule either stands alone on each Atom or grows as one continuous run over everything already read, following the Words across line breaks.\n\nEvery Word of every Cue is drawn in one typeface, the one its Style names: emphasis varies by Word through the active Paint, never by typeface, and lines exist only where the text wraps, so a line cannot be given a face, colour or weight of its own. A caption whose lines are set in different typefaces is outside what this Track can draw.",preview: previewImage("Track.png"),attributes: [@@ -261,14 +260,12 @@ export const captionFineMarkupSurfaces = [summary: "Chooses the display Atoms and Words the Cues are drawn from." },{ name: "correspondence", kind: "reference", required: true, accepts: [narrativeTypes.captionCorrespondence],summary: "Chooses the link from display Words back to the spoken Script." },- { name: "map", kind: "reference", required: true, accepts: [semanticMapTypes.complete],- summary: "Chooses the measured Word windows that give every Cue its time." },+ { name: "semantic", kind: "reference", required: true, accepts: [semanticTrackTypes.track],+ summary: "Chooses the continuous SemanticTrack whose Word anchors give every Cue its time." },{ name: "program", kind: "reference", required: true, accepts: [captionTypes.program],summary: "Chooses the Style assignment that decides which Style each run is rendered in." },{ name: "plan", kind: "reference", required: true, accepts: [captionTypes.plan],summary: "Chooses the Cue cuts the planner produced for this display." },- { name: "space", kind: "reference", required: true, accepts: [programSpaceTypes.programSpace],- summary: "Chooses the ProgramSpace the rendered Track is laid out against." },],ports: [{ name: "track", type: compositionTypes.visualTrack,@@ -278,8 +275,7 @@ export const captionFineMarkupSurfaces = [id="captions"display={story.caption}correspondence={story.caption.correspondence}- map={timing.map}- space={speech.space}+ semantic={speech.semantic}program={caption-program}plan={caption-plan.plan}/>`,@@ -301,8 +297,7 @@ export const captionFineManifest: ModuleManifest = {compositionDependency,mediaDependency,narrativeDependency,- programSpaceDependency,- semanticMapDependency,+ semanticTrackDependency,],types: [],capabilities: [],@@ -312,7 +307,7 @@ export const captionFineManifest: ModuleManifest = {{ name: "caption", type: captionTypes.timedProjection },{ name: "program", type: captionTypes.program },{ name: "display", type: narrativeTypes.captionDisplay },- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },],outputs: [{ name: "track", type: compositionTypes.visualTrack }],needs: [],diff --git a/packages/caption-fine/src/surface.ts b/packages/caption-fine/src/surface.tsindex 90bfdeda..3da1e816 100644--- a/packages/caption-fine/src/surface.ts+++ b/packages/caption-fine/src/surface.ts@@ -2,8 +2,7 @@ import { captionTypes } from "@hypit/caption";import { assertFontArtifactRef, assertFontStackRef, mediaTypes } from "@hypit/media";import type { FontArtifactRef, FontStackRef } from "@hypit/media";import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { svsRecipeType } from "@hypit/svs";import type { SvsRecipe } from "@hypit/svs";import type {@@ -114,7 +113,7 @@ export const decodeFineCaptionStyleSurface: StructuredSurfaceHandler = ({ elemen};export const decodeFineCaptionTrackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- attributes(element, ["id", "display", "correspondence", "map", "program", "plan", "space"]);+ attributes(element, ["id", "display", "correspondence", "semantic", "program", "plan"]);if (element.children.some((child) => child.kind === "element" || child.value.trim())) {throw new Error(`${element.name} does not accept children`);}@@ -126,18 +125,17 @@ export const decodeFineCaptionTrackSurface: StructuredSurfaceHandler = ({ elemennarrativeTypes.captionCorrespondence,resolveReference,);- const map = reference(element, "map", semanticMapTypes.complete, resolveReference);+ const semantic = reference(element, "semantic", semanticTrackTypes.track, resolveReference);const program = reference(element, "program", captionTypes.program, resolveReference);const plan = reference(element, "plan", captionTypes.plan, resolveReference);- const space = reference(element, "space", programSpaceTypes.programSpace, resolveReference);return {records: [],components: [{id,fragment: fineCaptionTrackFragment.id,inputs: {- display: display.ref, correspondence: correspondence.ref, map: map.ref,- program: program.ref, plan: plan.ref, space: space.ref,+ display: display.ref, correspondence: correspondence.ref, semantic: semantic.ref,+ program: program.ref, plan: plan.ref,},outputs: { track: `${id}.track` },range: element.range,diff --git a/packages/caption/README.md b/packages/caption/README.mdindex 11380153..2617f9e4 100644--- a/packages/caption/README.md+++ b/packages/caption/README.md@@ -6,7 +6,7 @@ The package owns:- total Style assignment over an explicit `CaptionDisplaySequence`;- generic `CaptionPlan` validation: Cue boundaries use Atom ids, fields use Word ids;-- the join of `CaptionCorrespondence` and `CompleteSemanticMap` into timed Cues and Atoms.+- the join of `CaptionCorrespondence` and `SemanticTrack` into timed Cues and Atoms.It owns no Script parser, concrete font, box model, field meaning, renderer, LLM, Provider orRuntime policy.diff --git a/packages/caption/package.json b/packages/caption/package.jsonindex 159ce284..3984594b 100644--- a/packages/caption/package.json+++ b/packages/caption/package.json@@ -15,7 +15,7 @@"@hypit/elaborator": "workspace:*","@hypit/narrative": "workspace:*","@hypit/protocol": "workspace:*",- "@hypit/semantic-map": "workspace:*",+ "@hypit/semantic-track": "workspace:*","@hypit/markup": "workspace:*"},"devDependencies": {diff --git a/packages/caption/src/component.ts b/packages/caption/src/component.tsindex b7938141..ffceea17 100644--- a/packages/caption/src/component.ts+++ b/packages/caption/src/component.ts@@ -1,6 +1,6 @@import type { ComponentPackage } from "@hypit/component-kit";import type { CaptionCorrespondence, CaptionDisplaySequence } from "@hypit/narrative";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import type { SemanticTrack } from "@hypit/semantic-track";import type { StoredValue } from "@hypit/protocol";import { canonicalize } from "@hypit/protocol";@@ -33,7 +33,7 @@ export const captionComponent = {value: canonicalize(temporalizeCaptionPlan(inline<CaptionDisplaySequence>(inputs.display?.value, "CaptionDisplaySequence"),inline<CaptionCorrespondence>(inputs.correspondence?.value, "CaptionCorrespondence"),- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"),inline<CaptionProgram>(inputs.program?.value, "CaptionProgram"),inline<CaptionPlan>(inputs.plan?.value, "CaptionPlan"),)),diff --git a/packages/caption/src/fragment.ts b/packages/caption/src/fragment.tsindex 4c6f6138..7018a23f 100644--- a/packages/caption/src/fragment.ts+++ b/packages/caption/src/fragment.ts@@ -1,5 +1,5 @@import { narrativeTypes } from "@hypit/narrative";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { sealGraphFragment } from "@hypit/elaborator";import { captionProducers, captionTypes } from "./manifest.js";@@ -12,7 +12,7 @@ export const plannedCaptionTimingFragment = sealGraphFragment({inputs: [{ name: "display", type: narrativeTypes.captionDisplay },{ name: "correspondence", type: narrativeTypes.captionCorrespondence },- { name: "map", type: semanticMapTypes.complete },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "plan", type: captionTypes.plan },{ name: "program", type: captionTypes.program },],@@ -20,7 +20,7 @@ export const plannedCaptionTimingFragment = sealGraphFragment({id: "temporalize-caption-plan",producer: captionProducers.temporalizePlan,inputs: {- display: input("display"), correspondence: input("correspondence"), map: input("map"),+ display: input("display"), correspondence: input("correspondence"), semantic: input("semantic"),plan: input("plan"), program: input("program"),},result: { kind: "output", name: "caption" },diff --git a/packages/caption/src/manifest.ts b/packages/caption/src/manifest.tsindex e0701b13..d823ab79 100644--- a/packages/caption/src/manifest.ts+++ b/packages/caption/src/manifest.ts@@ -1,5 +1,5 @@import { narrativeDependency, narrativeTypes } from "@hypit/narrative";-import { semanticMapDependency, semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackDependency, semanticTrackTypes } from "@hypit/semantic-track";import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@hypit/protocol";export const captionModuleRef = { name: "@hypit/caption", version: "1" } as const;@@ -150,7 +150,7 @@ export const captionManifest: ModuleManifest = {format: "hypit.module@1",name: captionModuleRef.name,version: captionModuleRef.version,- dependencies: [narrativeDependency, semanticMapDependency],+ dependencies: [narrativeDependency, semanticTrackDependency],types: [{ name: captionTypes.style.name },{ name: captionTypes.program.name },@@ -164,7 +164,7 @@ export const captionManifest: ModuleManifest = {inputs: [{ name: "display", type: narrativeTypes.captionDisplay },{ name: "correspondence", type: narrativeTypes.captionCorrespondence },- { name: "map", type: semanticMapTypes.complete },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "program", type: captionTypes.program },{ name: "plan", type: captionTypes.plan },],diff --git a/packages/caption/src/temporalize.ts b/packages/caption/src/temporalize.tsindex c2ece828..e118b453 100644--- a/packages/caption/src/temporalize.ts+++ b/packages/caption/src/temporalize.ts@@ -1,6 +1,6 @@import type { CaptionCorrespondence, CaptionDisplaySequence } from "@hypit/narrative";-import { tokenFrameSpan } from "@hypit/semantic-map";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import { tokenFrameSpan } from "@hypit/semantic-track";+import type { SemanticTrack } from "@hypit/semantic-track";import { CaptionTimingError } from "./error.js";import { assertCaptionCorrespondence, assertCaptionDisplaySequence } from "./display.js";@@ -47,7 +47,7 @@ export function applyCaptionMute(export function temporalizeCaptionPlan(display: CaptionDisplaySequence,correspondence: CaptionCorrespondence,- map: CompleteSemanticMap,+ semantic: SemanticTrack,program: CaptionProgram,plan: CaptionPlan,): TimedCaptionProjection {@@ -64,11 +64,11 @@ export function temporalizeCaptionPlan(if (atom === undefined || sourceTokenIds === undefined) {throw new CaptionTimingError("CAPTION_ATOM", `Caption Cue ${cue.id} references unknown Atom ${atomId}.`);}- const window = tokenFrameSpan(map, sourceTokenIds);+ const window = tokenFrameSpan(semantic, sourceTokenIds);if (window === undefined) {throw new CaptionTimingError("CAPTION_SPEECH_COVERAGE",- `Caption Atom ${atomId} is absent from the complete speech map.`,+ `Caption Atom ${atomId} is absent from the SemanticTrack.`,);}return {diff --git a/packages/caption/test/caption.test.ts b/packages/caption/test/caption.test.tsindex 3997ce7e..a000e9ea 100644--- a/packages/caption/test/caption.test.ts+++ b/packages/caption/test/caption.test.ts@@ -19,6 +19,7 @@ import type { StoredValue, TypeRef } from "@hypit/protocol";import { sealProgramSpace } from "@hypit/program-space";import { locateAlignedSegmentTiming } from "../../speech-alignment/src/locate.js";import type { AlignmentBasis } from "../../speech-alignment/src/locate.js";+import { materializeSemanticTake } from "../../speech-alignment/src/materialize.js";import { sealAlignedTranscriptEvidence } from "@hypit/speech-evidence";import {captionCorrespondence,@@ -69,7 +70,18 @@ function locate(narrative: Narrative, durationSec: number, segments: readonly TrendFrameExclusive: Math.round(durationSec * 30),}],};- return locateAlignedSegmentTiming(narrative, audioBasis, evidence);+ const timing = locateAlignedSegmentTiming(narrative, audioBasis, evidence);+ const segment = narrative.segments[0]!;+ const take = materializeSemanticTake(narrative, {+ kind: "segment",+ id: segment.id,+ tokenStart: segment.tokenStart,+ tokenEndExclusive: segment.tokenEndExclusive,+ }, {+ timeline: { frameRate: space.frameRate, frameCount: Math.round(durationSec * 30) },+ audio: { artifact: audio },+ }, timing);+ return { id: "caption-semantic", items: [{ take }] };}function style(id: string, fields: readonly CaptionFieldDeclaration[] = []): CaptionStyleIntent {diff --git a/packages/comment-sticker/README.md b/packages/comment-sticker/README.mdindex ece2fc4c..796c4714 100644--- a/packages/comment-sticker/README.md+++ b/packages/comment-sticker/README.md@@ -17,7 +17,7 @@ The author surface keeps the three independent concerns visible:<comment:Style id="social-comment" recipe={styles.comment} font={fonts.ui}/>-<comment:Track id="comments" canvas={video.canvas} space={video.space}>+<comment:Track id="comments" canvas={video.canvas} semantic={speech.semantic}><comment:Stickerid="opening-comment"comment={comment-copy}diff --git a/packages/comment-sticker/package.json b/packages/comment-sticker/package.jsonindex 132f41cc..9692d8bd 100644--- a/packages/comment-sticker/package.json+++ b/packages/comment-sticker/package.json@@ -13,9 +13,9 @@"@hypit/elaborator": "workspace:*","@hypit/media": "workspace:*","@hypit/narrative": "workspace:*",- "@hypit/program-space": "workspace:*","@hypit/protocol": "workspace:*",- "@hypit/semantic-map": "workspace:*",+ "@hypit/program-space": "workspace:*",+ "@hypit/semantic-track": "workspace:*","@hypit/spatial": "workspace:*","@hypit/svs": "workspace:*","@hypit/temporal": "workspace:*",diff --git a/packages/comment-sticker/src/component.ts b/packages/comment-sticker/src/component.tsindex a270b78a..a73a0922 100644--- a/packages/comment-sticker/src/component.ts+++ b/packages/comment-sticker/src/component.ts@@ -1,9 +1,9 @@import type { ComponentPackage, ProducerHandlerContext } from "@hypit/component-kit";import type { NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrative";-import type { ProgramSpace } from "@hypit/program-space";import { canonicalize } from "@hypit/protocol";import type { BlobRef, StoredValue } from "@hypit/protocol";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import type { SemanticTrack } from "@hypit/semantic-track";+import { projectSemanticProgramSpace } from "@hypit/semantic-track";import type { CanvasSpace, SpatialFrame } from "@hypit/spatial";import type { Text } from "@hypit/text";@@ -31,7 +31,7 @@ function common(inputs: Record<string, { readonly value: StoredValue } | undefinheader: inline<CommentStickerHeader>(inputs.header?.value, "CommentStickerHeader"),frame: inline<SpatialFrame>(inputs.frame?.value, "SpatialFrame"),style: inline<CommentStickerStyle>(inputs.style?.value, "CommentStickerStyle"),- space: inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ semantic: inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"),spec: inline<CommentStickerItemSpec>(inputs.spec?.value, "CommentStickerItemSpec"),content: inline<CommentStickerContent>(inputs.content?.value, "CommentStickerContent"),};@@ -69,7 +69,7 @@ export const commentStickerComponent = {handler: ({ inputs }: ProducerHandlerContext) => {const values = common(inputs);return { outputs: { set: output(appendProgramCommentSticker(- values.set, values.header, values.frame, values.style, values.space, values.spec, values.content,+ values.set, values.header, values.frame, values.style, values.semantic, values.spec, values.content,...(avatar ? [inline<BlobRef>(inputs.avatar?.value, "Comment Sticker avatar")] : []),)) }, needs: {} };},@@ -82,8 +82,7 @@ export const commentStickerComponent = {handler: ({ inputs }: ProducerHandlerContext) => {const values = common(inputs);return { outputs: { set: output(appendSelectionCommentSticker(- values.set, values.header, values.frame, values.style, values.space,- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ values.set, values.header, values.frame, values.style, values.semantic,inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelection"), values.spec, values.content,...(avatar ? [inline<BlobRef>(inputs.avatar?.value, "Comment Sticker avatar")] : []),)) }, needs: {} };@@ -97,8 +96,7 @@ export const commentStickerComponent = {handler: ({ inputs }: ProducerHandlerContext) => {const values = common(inputs);return { outputs: { set: output(appendMomentCommentSticker(- values.set, values.header, values.frame, values.style, values.space,- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ values.set, values.header, values.frame, values.style, values.semantic,inline<NarrativeMomentRef>(inputs.moment?.value, "NarrativeMoment"), values.spec, values.content,...(avatar ? [inline<BlobRef>(inputs.avatar?.value, "Comment Sticker avatar")] : []),)) }, needs: {} };@@ -115,7 +113,7 @@ export const commentStickerComponent = {producer: commentStickerProducers.render,handler: ({ inputs }) => ({ outputs: { track: output(renderCommentSticker(inline<CanvasSpace>(inputs.canvas?.value, "CanvasSpace"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")),inline<CommentStickerProgram>(inputs.program?.value, "CommentStickerProgram"),)) }, needs: {} }),},diff --git a/packages/comment-sticker/src/fragment.ts b/packages/comment-sticker/src/fragment.tsindex 9dd64fbf..efc0b600 100644--- a/packages/comment-sticker/src/fragment.ts+++ b/packages/comment-sticker/src/fragment.ts@@ -3,9 +3,8 @@ import { compositionTypes } from "@hypit/composition";import { sealGraphFragment } from "@hypit/elaborator";import type { FragmentOperation } from "@hypit/elaborator";import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";import type { TypeRef } from "@hypit/protocol";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import { textTypes } from "@hypit/text";@@ -13,8 +12,8 @@ import { commentStickerProducers, commentStickerTypes } from "./manifest.js";type TimedItem =| { readonly kind: "program" }- | { readonly kind: "selection"; readonly mapName: string; readonly sourceName: string }- | { readonly kind: "moment"; readonly mapName: string; readonly sourceName: string };+ | { readonly kind: "selection"; readonly sourceName: string }+ | { readonly kind: "moment"; readonly sourceName: string };export type CommentStickerFragmentItem = TimedItem & {readonly specName: string;@@ -60,7 +59,6 @@ export function createCommentStickerFragment(items: readonly CommentStickerFragmif (item.metaName !== undefined) types.set(item.metaName, textTypes.text);if (item.avatarName !== undefined) types.set(item.avatarName, artifactTypes.blob);if (item.kind !== "program") {- types.set(item.mapName, semanticMapTypes.complete);types.set(item.sourceName, item.kind === "selection" ? narrativeTypes.selection : narrativeTypes.moment);}const suffix = String(index + 1).padStart(4, "0");@@ -96,12 +94,11 @@ export function createCommentStickerFragment(items: readonly CommentStickerFragmheader: input("header"),frame: input(item.frameName),style: input(item.styleName),- space: input("space"),+ semantic: input("semantic"),spec: input(item.specName),content,...(item.avatarName === undefined ? {} : { avatar: input(item.avatarName) }),...(item.kind === "program" ? {} : {- map: input(item.mapName),[item.kind]: input(item.sourceName),}),},@@ -119,7 +116,7 @@ export function createCommentStickerFragment(items: readonly CommentStickerFragm{id: "comment:track",producer: commentStickerProducers.render,- inputs: { canvas: input("canvas"), space: input("space"), program: operation("comment:program") },+ inputs: { canvas: input("canvas"), semantic: input("semantic"), program: operation("comment:program") },result: { kind: "output", name: "track" },},);@@ -127,7 +124,7 @@ export function createCommentStickerFragment(items: readonly CommentStickerFragminputs: [{ name: "canvas", type: spatialTypes.canvas },{ name: "header", type: commentStickerTypes.header },- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },...[...types].map(([inputName, type]) => ({ name: inputName, type })),],operations,diff --git a/packages/comment-sticker/src/manifest.ts b/packages/comment-sticker/src/manifest.tsindex a192f7a7..2a2895ab 100644--- a/packages/comment-sticker/src/manifest.ts+++ b/packages/comment-sticker/src/manifest.ts@@ -4,9 +4,8 @@ import { artifactDependency, artifactTypes } from "@hypit/artifact";import { compositionDependency, compositionTypes } from "@hypit/composition";import { fontArtifactSchema, mediaDependency, mediaTypes } from "@hypit/media";import { narrativeDependency, narrativeTypes } from "@hypit/narrative";-import { programSpaceDependency, programSpaceTypes } from "@hypit/program-space";import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@hypit/protocol";-import { semanticMapDependency, semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackDependency, semanticTrackTypes } from "@hypit/semantic-track";import { spatialDependency, spatialFrameSchema, spatialTypes } from "@hypit/spatial";import { svsRecipeType } from "@hypit/svs";import { temporalDependency } from "@hypit/temporal";@@ -145,7 +144,7 @@ export const commentStickerProgramSchema: ValueSchema = object({const appendInputs = [{ name: "set", type: commentStickerTypes.set }, { name: "header", type: commentStickerTypes.header },{ name: "frame", type: spatialTypes.frame }, { name: "style", type: commentStickerTypes.style },- { name: "space", type: programSpaceTypes.programSpace }, { name: "spec", type: commentStickerTypes.itemSpec },+ { name: "semantic", type: semanticTrackTypes.track }, { name: "spec", type: commentStickerTypes.itemSpec },{ name: "content", type: commentStickerTypes.content },] as const;@@ -286,10 +285,8 @@ export const commentStickerMarkupSurfaces = [summary: "Names the Track and prefixes every binding it publishes." },{ name: "canvas", kind: "reference", required: true, accepts: [spatialTypes.canvas],summary: "Chooses the Canvas the cards are laid out on." },- { name: "space", kind: "reference", required: true, accepts: [programSpaceTypes.programSpace],- summary: "Chooses the ProgramSpace whose duration and frame rate every card window is measured against." },- { name: "map", kind: "reference", required: false, accepts: [semanticMapTypes.complete],- summary: "Supplies the SemanticMap that resolves the Selections and Moments its Stickers bind to." },+ { name: "semantic", kind: "reference", required: true, accepts: [semanticTrackTypes.track],+ summary: "Chooses the SemanticTrack that owns the Program frame domain and resolves every temporal binding." },],children: [{ tag: "Sticker", cardinality: "many",@@ -338,7 +335,7 @@ export const commentStickerMarkupSurfaces = [summary: "That Program rendered as one VisualTrack." },],example: `-<comment:Track id="comments" canvas={vertical} space={speech.space} map={timing.map}>+<comment:Track id="comments" canvas={vertical} semantic={speech.semantic}><comment:Sticker id="one" frame={comment-frame} style={social} avatar={viewer-avatar}author="@viewer" meta="Featured" during={story.selection.reaction}>Wait, it pinned the caption to the word, not the second.@@ -347,7 +344,6 @@ export const commentStickerMarkupSurfaces = [`,notes: ["A Track requires at least one Sticker.",- "`map` is refused when no Sticker binds to a Selection or a Moment, rather than ignored.","A Sticker states exactly one temporal form: `during`, `at` with `for`, or `start` with `end`.","With `start` and `end`, `selection` or `moment` binds the point of reference; both together are refused.","A Sticker's copy is either `comment` or the element's own text; stating both is refused, and one of the two is required.",@@ -362,7 +358,7 @@ export const commentStickerManifest: ModuleManifest = {format: "hypit.module@1",name: commentStickerModuleRef.name,version: commentStickerModuleRef.version,- dependencies: [artifactDependency, narrativeDependency, semanticMapDependency, programSpaceDependency, spatialDependency, temporalDependency, mediaDependency, compositionDependency, textDependency],+ dependencies: [artifactDependency, narrativeDependency, semanticTrackDependency, spatialDependency, temporalDependency, mediaDependency, compositionDependency, textDependency],types: [{ name: commentStickerTypes.header.name },{ name: commentStickerTypes.style.name },@@ -387,10 +383,10 @@ export const commentStickerManifest: ModuleManifest = {...([[commentStickerProducers.appendProgram, []],[commentStickerProducers.appendProgramAvatar, [{ name: "avatar", type: artifactTypes.blob }]],- [commentStickerProducers.appendSelection, [{ name: "map", type: semanticMapTypes.complete }, { name: "selection", type: narrativeTypes.selection }]],- [commentStickerProducers.appendSelectionAvatar, [{ name: "map", type: semanticMapTypes.complete }, { name: "selection", type: narrativeTypes.selection }, { name: "avatar", type: artifactTypes.blob }]],- [commentStickerProducers.appendMoment, [{ name: "map", type: semanticMapTypes.complete }, { name: "moment", type: narrativeTypes.moment }]],- [commentStickerProducers.appendMomentAvatar, [{ name: "map", type: semanticMapTypes.complete }, { name: "moment", type: narrativeTypes.moment }, { name: "avatar", type: artifactTypes.blob }]],+ [commentStickerProducers.appendSelection, [{ name: "selection", type: narrativeTypes.selection }]],+ [commentStickerProducers.appendSelectionAvatar, [{ name: "selection", type: narrativeTypes.selection }, { name: "avatar", type: artifactTypes.blob }]],+ [commentStickerProducers.appendMoment, [{ name: "moment", type: narrativeTypes.moment }]],+ [commentStickerProducers.appendMomentAvatar, [{ name: "moment", type: narrativeTypes.moment }, { name: "avatar", type: artifactTypes.blob }]],] as const).map(([producer, extra]) => ({name: producer.name,inputs: [...appendInputs, ...extra],@@ -398,7 +394,7 @@ export const commentStickerManifest: ModuleManifest = {needs: [],})),{ name: commentStickerProducers.finalize.name, inputs: [{ name: "set", type: commentStickerTypes.set }, { name: "header", type: commentStickerTypes.header }], outputs: [{ name: "program", type: commentStickerTypes.program }], needs: [] },- { name: commentStickerProducers.render.name, inputs: [{ name: "canvas", type: spatialTypes.canvas }, { name: "space", type: programSpaceTypes.programSpace }, { name: "program", type: commentStickerTypes.program }], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [] },+ { name: commentStickerProducers.render.name, inputs: [{ name: "canvas", type: spatialTypes.canvas }, { name: "semantic", type: semanticTrackTypes.track }, { name: "program", type: commentStickerTypes.program }], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [] },],};diff --git a/packages/comment-sticker/src/program.ts b/packages/comment-sticker/src/program.tsindex 72c56225..dcfc307c 100644--- a/packages/comment-sticker/src/program.ts+++ b/packages/comment-sticker/src/program.ts@@ -16,7 +16,7 @@ import { assertProgramSpaceIdentity } from "@hypit/program-space";import type { ProgramSpace } from "@hypit/program-space";import { canonicalize, isDigest } from "@hypit/protocol";import type { BlobRef } from "@hypit/protocol";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import type { SemanticTrack } from "@hypit/semantic-track";import { assertCanvasSpace, assertSpatialFrame } from "@hypit/spatial";import type { CanvasSpace, SpatialFrame } from "@hypit/spatial";import {@@ -238,13 +238,13 @@ export function appendProgramCommentSticker(header: CommentStickerHeader,frame: SpatialFrame,style: CommentStickerStyle,- space: ProgramSpace,+ semantic: SemanticTrack,spec: CommentStickerItemSpec,content: CommentStickerContent,avatar?: BlobRef,): CommentStickerSet {assert(spec.expansion.kind === "one", `Program Comment Sticker ${spec.id} must use one occurrence.`);- return realized(set, header, frame, style, spec, content, [projectProgramWindow({ itemId: spec.id, space, projection: spec.projection })], avatar);+ return realized(set, header, frame, style, spec, content, [projectProgramWindow({ itemId: spec.id, semantic, projection: spec.projection })], avatar);}export function appendSelectionCommentSticker(@@ -252,15 +252,14 @@ export function appendSelectionCommentSticker(header: CommentStickerHeader,frame: SpatialFrame,style: CommentStickerStyle,- space: ProgramSpace,- map: CompleteSemanticMap,+ semantic: SemanticTrack,selection: NarrativeSelectionRef,spec: CommentStickerItemSpec,content: CommentStickerContent,avatar?: BlobRef,): CommentStickerSet {return realized(set, header, frame, style, spec, content,- projectSelectionWindows({ itemId: spec.id, map, selection, space, expansion: spec.expansion, projection: spec.projection }), avatar);+ projectSelectionWindows({ itemId: spec.id, semantic, selection, expansion: spec.expansion, projection: spec.projection }), avatar);}export function appendMomentCommentSticker(@@ -268,15 +267,14 @@ export function appendMomentCommentSticker(header: CommentStickerHeader,frame: SpatialFrame,style: CommentStickerStyle,- space: ProgramSpace,- map: CompleteSemanticMap,+ semantic: SemanticTrack,moment: NarrativeMomentRef,spec: CommentStickerItemSpec,content: CommentStickerContent,avatar?: BlobRef,): CommentStickerSet {return realized(set, header, frame, style, spec, content,- projectMomentWindows({ itemId: spec.id, map, moment, space, expansion: spec.expansion, projection: spec.projection }), avatar);+ projectMomentWindows({ itemId: spec.id, semantic, moment, expansion: spec.expansion, projection: spec.projection }), avatar);}export function assertCommentStickerProgram(value: CommentStickerProgram): void {diff --git a/packages/comment-sticker/src/surface.ts b/packages/comment-sticker/src/surface.tsindex 4d8c2b67..ac4b1db3 100644--- a/packages/comment-sticker/src/surface.ts+++ b/packages/comment-sticker/src/surface.ts@@ -2,8 +2,7 @@ import { artifactTypes } from "@hypit/artifact";import { mediaTypes } from "@hypit/media";import type { FontStackRef } from "@hypit/media";import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import { svsRecipeType } from "@hypit/svs";import type { SvsRecipe } from "@hypit/svs";@@ -164,10 +163,10 @@ export const decodeCommentStickerStyleSurface: StructuredSurfaceHandler = ({ ele};export const decodeCommentStickerTrackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- allowed(element, ["id", "canvas", "space", "map"], ["id", "canvas", "space"]);+ allowed(element, ["id", "canvas", "semantic"], ["id", "canvas", "semantic"]);const id = text(element, "id");const canvas = reference(element.attributes.canvas, `${element.name}.canvas`, spatialTypes.canvas, resolveReference);- const space = reference(element.attributes.space, `${element.name}.space`, programSpaceTypes.programSpace, resolveReference);+ const semantic = reference(element.attributes.semantic, `${element.name}.semantic`, semanticTrackTypes.track, resolveReference);const headerId = `${id}.__header`;const records: SurfaceRecordDraft[] = [{id: headerId,@@ -175,9 +174,8 @@ export const decodeCommentStickerTrackSurface: StructuredSurfaceHandler = ({ elevalue: { kind: "inline", value: sealCommentStickerHeader({ id }) },range: element.range,}];- const inputs: Record<string, typeof canvas.ref> = { canvas: canvas.ref, header: { kind: "record", id: headerId }, space: space.ref };+ const inputs: Record<string, typeof canvas.ref> = { canvas: canvas.ref, header: { kind: "record", id: headerId }, semantic: semantic.ref };const items: Parameters<typeof createCommentStickerFragment>[0][number][] = [];- let usesMap = false;for (const child of element.children) {if (child.kind === "text") {if (child.value.trim()) throw new Error(`${element.name} accepts only Sticker children.`);@@ -235,15 +233,12 @@ export const decodeCommentStickerTrackSurface: StructuredSurfaceHandler = ({ eleconst copy = { commentName, ...(authorName === undefined ? {} : { authorName }), ...(headerTextName === undefined ? {} : { headerTextName }), ...(metaName === undefined ? {} : { metaName }) };if (temporal.kind === "program") items.push({ kind: "program", specName, frameName, styleName, ...copy, ...(avatarName === undefined ? {} : { avatarName }) });else {- usesMap = true;const sourceName = `item-${suffix}-${temporal.kind}`;inputs[sourceName] = temporal.source!.ref;- items.push({ kind: temporal.kind, specName, frameName, styleName, ...copy, sourceName, mapName: "map", ...(avatarName === undefined ? {} : { avatarName }) });+ items.push({ kind: temporal.kind, specName, frameName, styleName, ...copy, sourceName, ...(avatarName === undefined ? {} : { avatarName }) });}}if (items.length === 0) throw new Error(`${element.name} requires at least one Sticker.`);- if (usesMap) inputs.map = reference(element.attributes.map, `${element.name}.map`, semanticMapTypes.complete, resolveReference).ref;- else if (element.attributes.map !== undefined) throw new Error(`${element.name}.map is unused.`);const fragment = createCommentStickerFragment(items);return {records,diff --git a/packages/comment-sticker/test/comment-sticker.test.ts b/packages/comment-sticker/test/comment-sticker.test.tsindex bb865af3..d181a148 100644--- a/packages/comment-sticker/test/comment-sticker.test.ts+++ b/packages/comment-sticker/test/comment-sticker.test.ts@@ -1,12 +1,14 @@import assert from "node:assert/strict";import test from "node:test";import { fixtureDigest } from "../../../test/fixture-digest.js";+import { semanticTrackFixture } from "../../../test/semantic-track-fixture.js";import { artifactTypes } from "@hypit/artifact";import { compileHyperframesDocument } from "@hypit/hyperframes";import type { FontArtifactRef, FontStackRef } from "@hypit/media";import { mediaTypes } from "@hypit/media";import { sealProgramSpace } from "@hypit/program-space";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import { svsRecipeType } from "@hypit/svs";import type { SvsRecipe } from "@hypit/svs";@@ -48,6 +50,7 @@ const canvas = { widthPx: 1080, heightPx: 1920,origin: "top-left" as const, xDirection: "right" as const, yDirection: "down" as const, pixelAspect: "square" as const };const space = sealProgramSpace({ durationSec: 3, frameRate: { numerator: 30, denominator: 1 } });const header = sealCommentStickerHeader({ id: "comments" });+const semantic = semanticTrackFixture(space);function item(id: string) {return sealCommentStickerItemSpec({@@ -66,7 +69,7 @@ function content(meta?: string) {}function track(meta?: string) {- const set = appendProgramCommentSticker(createCommentStickerSet(), header, frame, style, space, item("opening"), content(meta));+ const set = appendProgramCommentSticker(createCommentStickerSet(), header, frame, style, semantic, item("opening"), content(meta));return renderCommentSticker(canvas, space, finalizeCommentSticker(set, header));}@@ -114,7 +117,7 @@ test("short Sticker windows compose overlapping enter and exit motion instead of},}, fonts, "compressed");const set = appendProgramCommentSticker(- createCommentStickerSet(), header, frame, compressed, space, item("compressed"), content(),+ createCommentStickerSet(), header, frame, compressed, semantic, item("compressed"), content(),);const rendered = renderCommentSticker(canvas, space, finalizeCommentSticker(set, header));const animation = rendered.presents[0]?.elements.find((element) => element.animation !== undefined)?.animation;@@ -161,7 +164,7 @@ test("Track Surface lowers mixed program and semantic Stickers to a finite expliconst blob = { kind: "blob" as const, digest: fixtureDigest("comment-avatar"), size: 128, mediaType: "image/png" };const refs = new Map<string, SurfaceResolvedReference>([["video.canvas", authored("video.canvas", spatialTypes.canvas, canvas)],- ["video.space", authored("video.space", { module: { name: "@hypit/program-space", version: "1" }, name: "ProgramSpace" }, space)],+ ["video.semantic", authored("video.semantic", semanticTrackTypes.track, semantic)],["layout.comment", authored("layout.comment", spatialTypes.frame, frame)],["social", authored("social", commentStickerTypes.style, style)],["avatar", authored("avatar", artifactTypes.blob, blob)],@@ -169,7 +172,7 @@ test("Track Surface lowers mixed program and semantic Stickers to a finite expli]);const result = await decodeCommentStickerTrackSurface({sourceName: "fixture.svml",- element: parsed(`<comment:Track id="comments" canvas={video.canvas} space={video.space}>+ element: parsed(`<comment:Track id="comments" canvas={video.canvas} semantic={video.semantic}><comment:Sticker id="one" comment={copy} frame={layout.comment} style={social} avatar={avatar} author="@viewer" meta="Featured" during="program"/></comment:Track>`),resolveReference: (path) => refs.get(path),diff --git a/packages/composition/package.json b/packages/composition/package.jsonindex 806fb77c..12f5c4f8 100644--- a/packages/composition/package.json+++ b/packages/composition/package.json@@ -21,7 +21,6 @@"@hypit/artifact": "workspace:*","@hypit/core": "workspace:*","@hypit/narrative": "workspace:*",- "@hypit/semantic-map": "workspace:*","@hypit/spatial": "workspace:*","@hypit/speech": "workspace:*","@hypit/speech-evidence": "workspace:*",diff --git a/packages/composition/test/track.test.ts b/packages/composition/test/track.test.tsindex 25f03310..74cbbb9c 100644--- a/packages/composition/test/track.test.ts+++ b/packages/composition/test/track.test.ts@@ -14,7 +14,6 @@ import { narrativeManifest } from "@hypit/narrative";import { programSpaceManifest, sealProgramSpace } from "@hypit/program-space";import { speechManifest } from "@hypit/speech";import { speechEvidenceManifest } from "@hypit/speech-evidence";-import { semanticMapManifest } from "@hypit/semantic-map";import { spatialManifest } from "@hypit/spatial";import { VISUAL_IR_V1, visualIrManifest } from "@hypit/visual-ir";@@ -31,7 +30,7 @@ import {import type { VisualTrack } from "../src/index.js";const videoContractManifests = [artifactManifest, narrativeManifest, mediaManifest, programSpaceManifest,- speechManifest, speechEvidenceManifest, semanticMapManifest, spatialManifest, visualIrManifest, compositionManifest] as const;+ speechManifest, speechEvidenceManifest, spatialManifest, visualIrManifest, compositionManifest] as const;const font: FontArtifactRef = {sources: [{ artifact: { kind: "blob", digest: fixtureDigest("track:test-font"), size: 1_024, mediaType: "font/woff2" } }],diff --git a/packages/deck-track/README.md b/packages/deck-track/README.mdindex 1eadbc49..23de6b38 100644--- a/packages/deck-track/README.md+++ b/packages/deck-track/README.md@@ -15,8 +15,7 @@ Media layer lowerer, but Deck is not a Media mode and exports only an ordinary p<deck:DepthStackid="proof-stack"- map={timing.map}- space={speech.space}+ semantic={speech.semantic}canvas={vertical}frame={layout.proof-stack}until={story.selection.proof}diff --git a/packages/deck-track/package.json b/packages/deck-track/package.jsonindex f71a1b52..ba43df66 100644--- a/packages/deck-track/package.json+++ b/packages/deck-track/package.json@@ -20,7 +20,7 @@"@hypit/narrative": "workspace:*","@hypit/program-space": "workspace:*","@hypit/protocol": "workspace:*",- "@hypit/semantic-map": "workspace:*",+ "@hypit/semantic-track": "workspace:*","@hypit/spatial": "workspace:*","@hypit/svs": "workspace:*","@hypit/temporal": "workspace:*",diff --git a/packages/deck-track/src/component.ts b/packages/deck-track/src/component.tsindex 2e662458..5b357edb 100644--- a/packages/deck-track/src/component.ts+++ b/packages/deck-track/src/component.ts@@ -1,8 +1,8 @@import type { ComponentPackage, ProducerHandlerContext } from "@hypit/component-kit";-import type { ProgramSpace } from "@hypit/program-space";import { canonicalize } from "@hypit/protocol";import type { StoredValue } from "@hypit/protocol";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import type { SemanticTrack } from "@hypit/semantic-track";+import { projectSemanticProgramSpace } from "@hypit/semantic-track";import type { CanvasSpace, SpatialFrame } from "@hypit/spatial";import type { NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrative";import type { MediaLayerSet } from "@hypit/media-track";@@ -33,7 +33,7 @@ function finalizeInputs(inputs: ProducerHandlerContext["inputs"]) {header: inline<DepthStackHeader>(inputs.header?.value, "DepthStackHeader"),frame: inline<SpatialFrame>(inputs.frame?.value, "SpatialFrame"),spec: inline<DepthStackSpec>(inputs.spec?.value, "DepthStackSpec"),- space: inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ semantic: inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"),};}@@ -57,9 +57,8 @@ export const depthStackComponent = {inline<MediaLayerSet>(inputs.material?.value, "MediaLayerSet"),inline<DepthStackCardLabel>(inputs.label?.value, "DepthStackCardLabel"),inline<DepthStackCardSpec>(inputs.spec?.value, "DepthStackCardSpec"),- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"),inline<NarrativeMomentRef>(inputs.moment?.value, "NarrativeMomentRef"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),)) }, needs: {} }),},{@@ -67,7 +66,7 @@ export const depthStackComponent = {handler: ({ inputs }) => {const value = finalizeInputs(inputs);return { outputs: { program: output(finalizeDepthStackAtProgramEnd(- value.set, value.header, value.frame, value.spec, value.space,+ value.set, value.header, value.frame, value.spec, value.semantic,)) }, needs: {} };},},@@ -77,8 +76,8 @@ export const depthStackComponent = {const value = finalizeInputs(inputs);return { outputs: { program: output(finalizeDepthStackUntilMoment(value.set, value.header, value.frame, value.spec,- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),- inline<NarrativeMomentRef>(inputs.terminal?.value, "NarrativeMomentRef"), value.space,+ value.semantic,+ inline<NarrativeMomentRef>(inputs.terminal?.value, "NarrativeMomentRef"),)) }, needs: {} };},},@@ -89,9 +88,9 @@ export const depthStackComponent = {const value = finalizeInputs(inputs);return { outputs: { program: output(finalizeDepthStackUntilSelection(value.set, value.header, value.frame, value.spec,- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ value.semantic,inline<NarrativeSelectionRef>(inputs.terminal?.value, "NarrativeSelectionRef"),- index === 0 ? "start" : "end", value.space,+ index === 0 ? "start" : "end",)) }, needs: {} };},})),@@ -99,7 +98,7 @@ export const depthStackComponent = {producer: depthStackProducers.render,handler: ({ inputs }) => ({ outputs: { track: output(renderDepthStack(inline<CanvasSpace>(inputs.canvas?.value, "CanvasSpace"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")),inline<DepthStackProgram>(inputs.program?.value, "DepthStackProgram"),)) }, needs: {} }),},diff --git a/packages/deck-track/src/fragment.ts b/packages/deck-track/src/fragment.tsindex ef97f1b7..eb96b020 100644--- a/packages/deck-track/src/fragment.ts+++ b/packages/deck-track/src/fragment.ts@@ -5,8 +5,7 @@ import type { FragmentOperation, GraphFragment } from "@hypit/elaborator";import { mediaTypes } from "@hypit/media";import { mediaTrackProducers, mediaTrackTypes } from "@hypit/media-track";import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import { depthStackProducers, depthStackTypes } from "./manifest.js";@@ -47,8 +46,7 @@ export function createDepthStackFragment({ name: "canvas", type: spatialTypes.canvas },{ name: "frame", type: spatialTypes.frame },{ name: "header", type: depthStackTypes.header },- { name: "map", type: semanticMapTypes.complete },- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "spec", type: depthStackTypes.spec },];const operations: FragmentOperation[] = [{@@ -114,9 +112,8 @@ export function createDepthStackFragment(material: operation(sampleId),label: input(card.labelName),spec: input(card.cardSpecName),- map: input("map"),+ semantic: input("semantic"),moment: input(card.momentName),- space: input("space"),},result: { kind: "output", name: "set" },});@@ -141,9 +138,8 @@ export function createDepthStackFragment(header: input("header"),frame: input("frame"),spec: input("spec"),- space: input("space"),+ semantic: input("semantic"),...(terminal.kind === "program-end" ? {} : {- map: input("map"),terminal: input(terminal.inputName),}),},@@ -152,7 +148,7 @@ export function createDepthStackFragment(operations.push({id: "track",producer: depthStackProducers.render,- inputs: { canvas: input("canvas"), space: input("space"), program: operation("program") },+ inputs: { canvas: input("canvas"), semantic: input("semantic"), program: operation("program") },result: { kind: "output", name: "track" },});return sealGraphFragment({diff --git a/packages/deck-track/src/manifest.ts b/packages/deck-track/src/manifest.tsindex 6a2ff0bd..99ffecef 100644--- a/packages/deck-track/src/manifest.ts+++ b/packages/deck-track/src/manifest.ts@@ -18,9 +18,8 @@ import {mediaTrackTypes,} from "@hypit/media-track";import { narrativeDependency, narrativeTypes } from "@hypit/narrative";-import { programSpaceDependency, programSpaceTypes } from "@hypit/program-space";import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@hypit/protocol";-import { semanticMapDependency, semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackDependency, semanticTrackTypes } from "@hypit/semantic-track";import { spatialDependency, spatialFrameSchema, spatialTypes } from "@hypit/spatial";import { svsRecipeType } from "@hypit/svs";import { temporalDependency } from "@hypit/temporal";@@ -119,7 +118,7 @@ export const depthStackProgramSchema: ValueSchema = object({const finalizeInputs = [{ name: "set", type: depthStackTypes.cardSet }, { name: "header", type: depthStackTypes.header },{ name: "frame", type: spatialTypes.frame }, { name: "spec", type: depthStackTypes.spec },- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },] as const;export const depthStackMarkupSurfaces = [@@ -169,10 +168,8 @@ export const depthStackMarkupSurfaces = [attributes: [{ name: "id", kind: "identifier", required: true,summary: "Names this deck so its Program and Track can be referenced elsewhere in the Source." },- { name: "map", kind: "reference", required: true, accepts: [semanticMapTypes.complete],- summary: "Chooses the SemanticMap that places each Card's Moment in the programme." },- { name: "space", kind: "reference", required: true, accepts: [programSpaceTypes.programSpace],- summary: "Chooses the ProgramSpace the deck is timed against." },+ { name: "semantic", kind: "reference", required: true, accepts: [semanticTrackTypes.track],+ summary: "Chooses the SemanticTrack that owns the frame domain and places each Card's Moment." },{ name: "canvas", kind: "reference", required: true, accepts: [spatialTypes.canvas],summary: "Chooses the Canvas the deck is rendered into." },{ name: "frame", kind: "reference", required: true, accepts: [spatialTypes.frame],@@ -390,8 +387,7 @@ export const depthStackMarkupSurfaces = [],example: `<deck:DepthStackid="proof-stack"- map={timing.map}- space={speech.space}+ semantic={speech.semantic}canvas={vertical}frame={layout.proof-stack}until={story.selection.proof}@@ -413,7 +409,7 @@ export const depthStackMarkupSurfaces = [export const depthStackManifest: ModuleManifest = {format: "hypit.module@1", name: depthStackModuleRef.name, version: depthStackModuleRef.version,- dependencies: [narrativeDependency, semanticMapDependency, programSpaceDependency, spatialDependency, temporalDependency, mediaDependency, mediaTrackDependency, compositionDependency, textDependency],+ dependencies: [narrativeDependency, semanticTrackDependency, spatialDependency, temporalDependency, mediaDependency, mediaTrackDependency, compositionDependency, textDependency],types: [{ name: depthStackTypes.header.name },{ name: depthStackTypes.spec.name },@@ -429,8 +425,7 @@ export const depthStackManifest: ModuleManifest = {{ name: depthStackProducers.appendMomentCard.name, inputs: [{ name: "set", type: depthStackTypes.cardSet }, { name: "material", type: mediaTrackTypes.layerSet },{ name: "label", type: depthStackTypes.cardLabel }, { name: "spec", type: depthStackTypes.cardSpec },- { name: "map", type: semanticMapTypes.complete }, { name: "moment", type: narrativeTypes.moment },- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track }, { name: "moment", type: narrativeTypes.moment },], outputs: [{ name: "set", type: depthStackTypes.cardSet }], needs: [] },{ name: depthStackProducers.finalizeProgramEnd.name, inputs: finalizeInputs, outputs: [{ name: "program", type: depthStackTypes.program }], needs: [] },...([@@ -438,11 +433,11 @@ export const depthStackManifest: ModuleManifest = {[depthStackProducers.finalizeUntilSelectionStart, narrativeTypes.selection],[depthStackProducers.finalizeUntilSelectionEnd, narrativeTypes.selection],] as const).map(([producer, terminalType]) => ({- name: producer.name, inputs: [...finalizeInputs, { name: "map", type: semanticMapTypes.complete }, { name: "terminal", type: terminalType }],+ name: producer.name, inputs: [...finalizeInputs, { name: "terminal", type: terminalType }],outputs: [{ name: "program", type: depthStackTypes.program }], needs: [],})),{ name: depthStackProducers.render.name, inputs: [- { name: "canvas", type: spatialTypes.canvas }, { name: "space", type: programSpaceTypes.programSpace },+ { name: "canvas", type: spatialTypes.canvas }, { name: "semantic", type: semanticTrackTypes.track },{ name: "program", type: depthStackTypes.program },], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [] },],diff --git a/packages/deck-track/src/program.ts b/packages/deck-track/src/program.tsindex ef7672e7..f491af83 100644--- a/packages/deck-track/src/program.ts+++ b/packages/deck-track/src/program.ts@@ -15,7 +15,8 @@ import type { ProgramSpace } from "@hypit/program-space";import { canonicalize } from "@hypit/protocol";import { verifyText } from "@hypit/text";import type { Text } from "@hypit/text";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import type { SemanticTrack } from "@hypit/semantic-track";+import { projectSemanticProgramSpace } from "@hypit/semantic-track";import { assertSpatialFrame } from "@hypit/spatial";import type { SpatialFrame } from "@hypit/spatial";import {@@ -236,30 +237,28 @@ export function appendDepthStackMomentCard(material: DepthStackCardSet["cards"][number]["material"],label: DepthStackCardLabel,spec: DepthStackCardSpec,- map: CompleteSemanticMap,+ semantic: SemanticTrack,moment: NarrativeMomentRef,- space: ProgramSpace,): DepthStackCardSet {- const occurrences = locateMomentOccurrences(map, moment, space);+ const occurrences = locateMomentOccurrences(semantic, moment);assert(occurrences.length === 1,`DepthStack Card ${spec.id} Moment must locate exactly once; received ${occurrences.length}.`);return appendDepthStackCard(set, material, label, spec, occurrences[0]!.cue.frame);}-function terminalMoment(map: CompleteSemanticMap, moment: NarrativeMomentRef, space: ProgramSpace): number {- const occurrences = locateMomentOccurrences(map, moment, space);+function terminalMoment(semantic: SemanticTrack, moment: NarrativeMomentRef): number {+ const occurrences = locateMomentOccurrences(semantic, moment);assert(occurrences.length === 1,`DepthStack terminal Moment must locate exactly once; received ${occurrences.length}.`);return occurrences[0]!.cue.frame;}function terminalSelection(- map: CompleteSemanticMap,+ semantic: SemanticTrack,selection: NarrativeSelectionRef,- space: ProgramSpace,boundary: "start" | "end",): number {- const occurrences = locateSelectionOccurrences(map, selection, space);+ const occurrences = locateSelectionOccurrences(semantic, selection);assert(occurrences.length === 1,`DepthStack terminal Selection must locate exactly once; received ${occurrences.length}.`);return boundary === "start" ? occurrences[0]!.start.frame : occurrences[0]!.end.frame;@@ -271,8 +270,9 @@ export function finalizeDepthStack(frame: SpatialFrame,spec: DepthStackSpec,terminalFrame: number,- space: ProgramSpace,+ semantic: SemanticTrack,): DepthStackProgram {+ const space = projectSemanticProgramSpace(semantic);assertDepthStackCardSet(set);assertDepthStackHeader(header);assertSpatialFrame(frame);@@ -298,9 +298,10 @@ export function finalizeDepthStackAtProgramEnd(header: DepthStackHeader,frame: SpatialFrame,spec: DepthStackSpec,- space: ProgramSpace,+ semantic: SemanticTrack,): DepthStackProgram {- return finalizeDepthStack(set, header, frame, spec, programSpaceFrameCount(space), space);+ const space = projectSemanticProgramSpace(semantic);+ return finalizeDepthStack(set, header, frame, spec, programSpaceFrameCount(space), semantic);}export function finalizeDepthStackUntilMoment(@@ -308,11 +309,10 @@ export function finalizeDepthStackUntilMoment(header: DepthStackHeader,frame: SpatialFrame,spec: DepthStackSpec,- map: CompleteSemanticMap,+ semantic: SemanticTrack,moment: NarrativeMomentRef,- space: ProgramSpace,): DepthStackProgram {- return finalizeDepthStack(set, header, frame, spec, terminalMoment(map, moment, space), space);+ return finalizeDepthStack(set, header, frame, spec, terminalMoment(semantic, moment), semantic);}export function finalizeDepthStackUntilSelection(@@ -320,12 +320,11 @@ export function finalizeDepthStackUntilSelection(header: DepthStackHeader,frame: SpatialFrame,spec: DepthStackSpec,- map: CompleteSemanticMap,+ semantic: SemanticTrack,selection: NarrativeSelectionRef,boundary: "start" | "end",- space: ProgramSpace,): DepthStackProgram {- return finalizeDepthStack(set, header, frame, spec, terminalSelection(map, selection, space, boundary), space);+ return finalizeDepthStack(set, header, frame, spec, terminalSelection(semantic, selection, boundary), semantic);}export function assertDepthStackProgram(program: DepthStackProgram): void {diff --git a/packages/deck-track/src/surface.ts b/packages/deck-track/src/surface.tsindex ee030865..fde11b77 100644--- a/packages/deck-track/src/surface.ts+++ b/packages/deck-track/src/surface.ts@@ -3,9 +3,8 @@ import { mediaTypes } from "@hypit/media";import type { FontStackRef } from "@hypit/media";import { mediaTrackTypes } from "@hypit/media-track";import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";import type { TypeRef } from "@hypit/protocol";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import { svsRecipeType } from "@hypit/svs";import type { SvsRecipe } from "@hypit/svs";@@ -204,10 +203,9 @@ function terminal(}export const decodeDepthStackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- allowed(element, ["id", "map", "space", "canvas", "frame", "appearance", "until", "until-boundary"]);+ allowed(element, ["id", "semantic", "canvas", "frame", "appearance", "until", "until-boundary"]);const id = text(element, "id");- const map = reference(element.attributes.map, `${element.name}.map`, semanticMapTypes.complete, resolveReference);- const space = reference(element.attributes.space, `${element.name}.space`, programSpaceTypes.programSpace, resolveReference);+ const semantic = reference(element.attributes.semantic, `${element.name}.semantic`, semanticTrackTypes.track, resolveReference);const canvas = reference(element.attributes.canvas, `${element.name}.canvas`, spatialTypes.canvas, resolveReference);const frame = reference(element.attributes.frame, `${element.name}.frame`, spatialTypes.frame, resolveReference);const appearance = recipe(element.attributes.appearance, `${element.name}.appearance`, resolveReference);@@ -219,9 +217,9 @@ export const decodeDepthStackSurface: StructuredSurfaceHandler = ({ element, res{ id: headerId, type: depthStackTypes.header, value: { kind: "inline", value: sealDepthStackHeader({ id }) }, range: element.range },{ id: specId, type: depthStackTypes.spec, value: { kind: "inline", value: decodeDepthStackSpec(appearance) }, range: element.range },);- const inputs: Record<string, typeof map.ref> = {- canvas: canvas.ref, frame: frame.ref, header: { kind: "record", id: headerId }, map: map.ref,- space: space.ref, spec: { kind: "record", id: specId },+ const inputs: Record<string, typeof semantic.ref> = {+ canvas: canvas.ref, frame: frame.ref, header: { kind: "record", id: headerId }, semantic: semantic.ref,+ spec: { kind: "record", id: specId },};if (terminalValue.reference !== undefined) inputs.terminal = terminalValue.reference.ref;const cards: DepthStackFragmentCard[] = [];@@ -267,7 +265,7 @@ export const decodeDepthStackSurface: StructuredSurfaceHandler = ({ element, resinputs[sampleSpecName] = { kind: "record", id: sampleId };inputs[cardSpecName] = { kind: "record", id: cardSpecId };inputs[momentName] = moment.ref;- let labelRef: typeof map.ref;+ let labelRef: typeof semantic.ref;if (child.attributes.label === undefined) {const labelId = `${id}.card.${suffix}.label-none`;records.push({ id: labelId, type: depthStackTypes.cardLabel, value: { kind: "inline", value: noDepthStackCardLabel() }, range: child.range });diff --git a/packages/deck-track/test/deck-track.test.ts b/packages/deck-track/test/deck-track.test.tsindex aa3c7219..eaaf6289 100644--- a/packages/deck-track/test/deck-track.test.ts+++ b/packages/deck-track/test/deck-track.test.ts@@ -2,6 +2,7 @@ import assert from "node:assert/strict";import test from "node:test";import { videoContractManifests } from "../../../test/support/video-domain.js";import { fixtureDigest } from "../../../test/fixture-digest.js";+import { semanticTrackFixture } from "../../../test/semantic-track-fixture.js";import { compositionDependency, compositionTypes } from "@hypit/composition";import type { VisualElement, VisualTimedSampling } from "@hypit/composition";@@ -36,9 +37,9 @@ import { mediaTypes } from "@hypit/media";import { mediaPipelineManifest } from "@hypit/media-pipeline";import type { MediaLayerSet } from "@hypit/media-track";import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes, sealProgramSpace } from "@hypit/program-space";+import { sealProgramSpace } from "@hypit/program-space";import type { ModuleManifest } from "@hypit/protocol";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { sealCanvasSpace, sealSpatialFrame, spatialTypes } from "@hypit/spatial";import { svsRecipeType } from "@hypit/svs";import { sealText, textManifest, textTypes } from "@hypit/text";@@ -56,6 +57,7 @@ const space = sealProgramSpace({durationSec: 2,frameRate: { numerator: 30, denominator: 1 },});+const semantic = semanticTrackFixture(space);const canvas = sealCanvasSpace({widthPx: 360,heightPx: 640,@@ -225,7 +227,7 @@ function program(input: {frame,input.spec ?? baseSpec(),input.terminal ?? 60,- space,+ semantic,);}@@ -259,7 +261,7 @@ test("explicit wrapping never aliases one Card into several relative depths", ()test("missing, equal, reversed and terminal-crossing triggers fail in authored order", () => {assert.throws(() => finalizeDepthStack(createDepthStackCardSet(), sealDepthStackHeader({ id: "empty" }),- frame, baseSpec(), 60, space,+ frame, baseSpec(), 60, semantic,), /at least one Card/u);assert.throws(() => program({ triggers: [0, 20, 20] }), /strictly increasing/u);assert.throws(() => program({ triggers: [0, 30, 20] }), /strictly increasing/u);@@ -393,7 +395,7 @@ test("the author Surface keeps every source, trigger, terminal, Frame and optionrecord: { value: { kind: "inline", value: { path, properties } } } as never,});const references = new Map<string, SurfaceResolvedReference>([- ["map", plain("map", semanticMapTypes.complete)], ["space", plain("space", programSpaceTypes.programSpace)],+ ["semantic", plain("semantic", semanticTrackTypes.track)],["canvas", plain("canvas", spatialTypes.canvas)], ["frame", plain("frame", spatialTypes.frame)],["first", plain("first", artifactTypes.blob)], ["first-extent", plain("first-extent", spatialTypes.extent)],["second", plain("second", mediaTypes.synchronized)], ["one", plain("one", narrativeTypes.moment)],@@ -404,7 +406,7 @@ test("the author Surface keeps every source, trigger, terminal, Frame and optionconst result = await decodeDepthStackSurface({sourceName: "deck.svml",element: node("deck:DepthStack", {- id: "proof", map: ref("map"), space: ref("space"), canvas: ref("canvas"), frame: ref("frame"),+ id: "proof", semantic: ref("semantic"), canvas: ref("canvas"), frame: ref("frame"),appearance: ref("deck-style"), until: ref("terminal"),}, [node("deck:Card", { id: "one", source: ref("first"), extent: ref("first-extent"), at: ref("one") }),diff --git a/packages/elaborator/test/fragment.test.ts b/packages/elaborator/test/fragment.test.tsindex 7187fe1c..8e61d574 100644--- a/packages/elaborator/test/fragment.test.ts+++ b/packages/elaborator/test/fragment.test.ts@@ -1,6 +1,6 @@import { videoContractManifests } from "../../../test/support/video-domain.js";import { speechDependency, speechTypes } from "@hypit/speech";-import { compositionTypes } from "@hypit/composition";+import { compositionDependency, compositionTypes } from "@hypit/composition";import { spatialTypes } from "@hypit/spatial";import assert from "node:assert/strict";import test from "node:test";@@ -30,20 +30,18 @@ import type {ProducerRef,TypeRef,} from "@hypit/protocol";-import {- speechBasisManifest,- speechBasisProducers,-} from "@hypit/speech-basis";const testModule = { name: "example.fragment-speech", version: "0.0.0" } as const;const requestType = { module: testModule, name: "Request" } satisfies TypeRef;const generateProducer = { module: testModule, name: "generate" } satisfies ProducerRef;+const projectVisualProducer = { module: testModule, name: "project-visual" } satisfies ProducerRef;+const projectAudioProducer = { module: testModule, name: "project-audio" } satisfies ProducerRef;const manifest: ModuleManifest = {format: "hypit.module@1",name: testModule.name,version: testModule.version,- dependencies: [speechDependency],+ dependencies: [speechDependency, compositionDependency],types: [{ name: requestType.name }],capabilities: [],producers: [{@@ -52,12 +50,22 @@ const manifest: ModuleManifest = {{ name: "request", type: requestType },{ name: "style", type: requestType },],- outputs: [{ name: "take", type: speechTypes.basis }],+ outputs: [{ name: "take", type: speechTypes.semanticTake }],+ needs: [],+ }, {+ name: projectVisualProducer.name,+ inputs: [{ name: "take", type: speechTypes.semanticTake }],+ outputs: [{ name: "visual", type: compositionTypes.visualTrack }],+ needs: [],+ }, {+ name: projectAudioProducer.name,+ inputs: [{ name: "take", type: speechTypes.semanticTake }],+ outputs: [{ name: "audio", type: compositionTypes.audioTrack }],needs: [],}],};-const closure = createResolvedClosure([...videoContractManifests, speechBasisManifest, manifest]);+const closure = createResolvedClosure([...videoContractManifests, manifest]);function program(): LinkedProgram {const rawCanvas = sealRecord({@@ -102,15 +110,21 @@ function speechFragment(): GraphFragment {},{id: "visual",- producer: speechBasisProducers.projectVisual,- inputs: { basis: operation("generate") },+ producer: projectVisualProducer,+ inputs: { take: operation("generate") },result: { kind: "output", name: "visual" },},+ {+ id: "audio",+ producer: projectAudioProducer,+ inputs: { take: operation("generate") },+ result: { kind: "output", name: "audio" },+ },],exports: [{name: "take",- type: speechTypes.basis,+ type: speechTypes.semanticTake,root: operation("generate"),},{@@ -118,6 +132,11 @@ function speechFragment(): GraphFragment {type: compositionTypes.visualTrack,root: operation("visual"),},+ {+ name: "audio",+ type: compositionTypes.audioTrack,+ root: operation("audio"),+ },],});}@@ -166,11 +185,11 @@ test("one FragmentInstance shares its generation Operation across all exports",assert.equal(state.plan.steps.length, 3);const takeRecord = contribution.operations.find((item) => item.producer.name === "generate")?.result.record;assert.equal(- state.plan.steps.find((step) => step.producer.name === "project-audio")?.inputs.basis,+ state.plan.steps.find((step) => step.producer.name === "project-audio")?.inputs.take,takeRecord,);assert.equal(- state.plan.steps.find((step) => step.producer.name === "project-visual")?.inputs.basis,+ state.plan.steps.find((step) => step.producer.name === "project-visual")?.inputs.take,takeRecord,);});diff --git a/packages/film/package.json b/packages/film/package.jsonindex 0bd45e30..305ea3c1 100644--- a/packages/film/package.json+++ b/packages/film/package.json@@ -16,6 +16,7 @@"@hypit/elaborator": "workspace:*","@hypit/program-space": "workspace:*","@hypit/protocol": "workspace:*",+ "@hypit/semantic-track": "workspace:*","@hypit/spatial": "workspace:*","@hypit/svs": "workspace:*","@hypit/markup": "workspace:*"diff --git a/packages/film/src/component.ts b/packages/film/src/component.tsindex 68368e20..01344806 100644--- a/packages/film/src/component.ts+++ b/packages/film/src/component.ts@@ -1,5 +1,6 @@import type { ComponentPackage } from "@hypit/component-kit";-import type { ProgramSpace } from "@hypit/program-space";+import { projectSemanticProgramSpace } from "@hypit/semantic-track";+import type { SemanticTrack } from "@hypit/semantic-track";import type { CanvasSpace } from "@hypit/spatial";import type { AudioTrack, VisualTrack } from "@hypit/composition";import type { StoredValue } from "@hypit/protocol";@@ -28,7 +29,7 @@ export const filmComponent = {handler: ({ inputs }) => ({outputs: { set: { kind: "inline", value: canonicalize(appendFilmVisualTrack(inline<FilmTrackSet>(inputs.set?.value, "FilmTrackSet"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")),inline<VisualTrack>(inputs.track?.value, "VisualTrack"),)) } },needs: {},@@ -39,7 +40,7 @@ export const filmComponent = {handler: ({ inputs }) => ({outputs: { set: { kind: "inline", value: canonicalize(appendFilmAudioTrack(inline<FilmTrackSet>(inputs.set?.value, "FilmTrackSet"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")),inline<AudioTrack>(inputs.track?.value, "AudioTrack"),)) } },needs: {},@@ -51,7 +52,7 @@ export const filmComponent = {outputs: { composition: { kind: "inline", value: canonicalize(compileFilmComposition(inline<FilmProgram>(inputs.program?.value, "FilmProgram"),inline<CanvasSpace>(inputs.canvas?.value, "CanvasSpace"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")),inline<FilmTrackSet>(inputs.set?.value, "FilmTrackSet"),)) } },needs: {},diff --git a/packages/film/src/fragment.ts b/packages/film/src/fragment.tsindex ee7c1013..e3920a91 100644--- a/packages/film/src/fragment.ts+++ b/packages/film/src/fragment.ts@@ -1,4 +1,4 @@-import { programSpaceTypes } from "@hypit/program-space";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import { compositionTypes } from "@hypit/composition";import type { Track } from "@hypit/composition";@@ -12,7 +12,7 @@ const input = (name: string) => ({ kind: "fragment-input" as const, name });const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });function assertTrackInputs(tracks: readonly FilmTrackInput[]): FilmTrackInput[] {- const names = new Set(["program", "canvas", "space"]);+ const names = new Set(["program", "canvas", "semantic"]);return [...tracks].map((track) => ({ name: track.name.trim(), kind: track.kind })).sort((left, right) => left.name.localeCompare(right.name))@@ -43,7 +43,7 @@ export function createFilmAssemblyFragment(options: FilmAssemblyFragmentOptions)operations.push({id,producer: track.kind === "visual" ? filmProducers.appendVisualTrack : filmProducers.appendAudioTrack,- inputs: { set: operation(current), space: input("space"), track: input(track.name) },+ inputs: { set: operation(current), semantic: input("semantic"), track: input(track.name) },result: { kind: "output" as const, name: "set" },});current = id;@@ -54,7 +54,7 @@ export function createFilmAssemblyFragment(options: FilmAssemblyFragmentOptions)inputs: {program: input("program"),canvas: input("canvas"),- space: input("space"),+ semantic: input("semantic"),set: operation(current),},result: { kind: "output" as const, name: "composition" },@@ -63,7 +63,7 @@ export function createFilmAssemblyFragment(options: FilmAssemblyFragmentOptions)inputs: [{ name: "program", type: filmTypes.program },{ name: "canvas", type: spatialTypes.canvas },- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },...tracks.map((track) => ({name: track.name,type: track.kind === "visual" ? compositionTypes.visualTrack : compositionTypes.audioTrack,diff --git a/packages/film/src/manifest.ts b/packages/film/src/manifest.tsindex 9bc47787..93d28e7a 100644--- a/packages/film/src/manifest.ts+++ b/packages/film/src/manifest.ts@@ -1,6 +1,6 @@-import { programSpaceDependency, programSpaceTypes } from "@hypit/program-space";import { audioTrackSchema, compositionDependency, compositionTypes, visualTrackSchema } from "@hypit/composition";import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@hypit/protocol";+import { semanticTrackDependency, semanticTrackTypes } from "@hypit/semantic-track";import { spatialDependency, spatialTypes } from "@hypit/spatial";import { svsManifest, svsModuleRef, svsRecipeType } from "@hypit/svs";@@ -45,7 +45,7 @@ export const filmMarkupSurfaces = [{outputs: [filmTypes.program],vocabulary: {summary:- "Assembles any number of peer VisualTrack and AudioTrack references into one Composition against a Canvas and a ProgramSpace.",+ "Assembles any number of peer VisualTrack and AudioTrack references into one Composition against a Canvas and a SemanticTrack.",appearance:"One flat fill of the entire Canvas, in the single hexadecimal color the Recipe's `background` carries, lying behind everything else in the Frame. It covers the full Canvas width and height, holds that one color from the first Frame to the last, and never moves, fades or changes. Wherever nothing is painted over it, that color is what the Frame shows; the Film puts no mark of its own on top of it.",attributes: [@@ -54,9 +54,9 @@ export const filmMarkupSurfaces = [{{ name: "canvas", kind: "reference", required: true,accepts: [spatialTypes.canvas],summary: "Selects the CanvasSpace that decides the Composition's dimensions." },- { name: "space", kind: "reference", required: true,- accepts: [programSpaceTypes.programSpace],- summary: "Selects the ProgramSpace that decides the Composition's duration and frame rate." },+ { name: "semantic", kind: "reference", required: true,+ accepts: [semanticTrackTypes.track],+ summary: "Selects the SemanticTrack that decides the Composition's duration and frame rate." },{ name: "appearance", kind: "reference", required: true,accepts: [svsRecipeType],summary: "Selects the SVS Recipe that decides the clear color behind every Track.",@@ -79,9 +79,9 @@ export const filmMarkupSurfaces = [{summary: "The assembled Composition, addressed as `<id>.composition`." },],example: [- '<film:Film id="main" canvas={vertical} space={speech.space} appearance={studio.film.vertical}>',+ '<film:Film id="main" canvas={vertical} semantic={speech.semantic} appearance={studio.film.vertical}>'," <film:Track source={speech.visual}/>",- " <film:Track source={speech.audioTrack}/>",+ " <film:Track source={speech.audio}/>"," <film:Track source={captions.track}/>","</film:Film>",].join("\n"),@@ -99,7 +99,7 @@ export const filmManifest: ModuleManifest = {name: filmModuleRef.name,version: filmModuleRef.version,dependencies: [- programSpaceDependency,+ semanticTrackDependency,spatialDependency,compositionDependency,{ module: svsModuleRef },@@ -120,7 +120,7 @@ export const filmManifest: ModuleManifest = {name: filmProducers.appendVisualTrack.name,inputs: [{ name: "set", type: filmTypes.trackSet },- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "track", type: compositionTypes.visualTrack },],outputs: [{ name: "set", type: filmTypes.trackSet }],@@ -130,7 +130,7 @@ export const filmManifest: ModuleManifest = {name: filmProducers.appendAudioTrack.name,inputs: [{ name: "set", type: filmTypes.trackSet },- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "track", type: compositionTypes.audioTrack },],outputs: [{ name: "set", type: filmTypes.trackSet }],@@ -141,7 +141,7 @@ export const filmManifest: ModuleManifest = {inputs: [{ name: "program", type: filmTypes.program },{ name: "canvas", type: spatialTypes.canvas },- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "set", type: filmTypes.trackSet },],outputs: [{ name: "composition", type: compositionTypes.composition }],diff --git a/packages/film/src/surface.ts b/packages/film/src/surface.tsindex a08ade55..41d5ae2a 100644--- a/packages/film/src/surface.ts+++ b/packages/film/src/surface.ts@@ -1,5 +1,4 @@-import { programSpaceTypes } from "@hypit/program-space";-import type { ProgramSpace } from "@hypit/program-space";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import { compositionTypes } from "@hypit/composition";import type { AudioTrack, Track, VisualTrack } from "@hypit/composition";@@ -94,15 +93,15 @@ function trackKind(reference: SurfaceResolvedReference): "visual" | "audio" {}export const decodeFilmSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- exactAttributes(element, ["id", "canvas", "space", "appearance"]);+ exactAttributes(element, ["id", "canvas", "semantic", "appearance"]);const id = stringAttribute(element, "id");const canvas = requiredReference(element, "canvas", resolveReference);if (!sameType(canvas.type, spatialTypes.canvas)) {throw new Error(`${element.name}.canvas must reference CanvasSpace`);}- const space = requiredReference(element, "space", resolveReference);- if (!sameType(space.type, programSpaceTypes.programSpace)) {- throw new Error(`${element.name}.space must reference ProgramSpace`);+ const semantic = requiredReference(element, "semantic", resolveReference);+ if (!sameType(semantic.type, semanticTrackTypes.track)) {+ throw new Error(`${element.name}.semantic must reference SemanticTrack`);}const appearanceReference = requiredReference(element, "appearance", resolveReference);const appearance = filmAppearanceFromRecipe(@@ -139,7 +138,7 @@ export const decodeFilmSurface: StructuredSurfaceHandler = ({ element, resolveReinputs: {program: { kind: "record", id: programId },canvas: canvas.ref,- space: space.ref,+ semantic: semantic.ref,...Object.fromEntries(tracks.map((track) => [track.name, track.source.ref])),},outputs: { composition: `${id}.composition` },diff --git a/packages/film/test/film.test.ts b/packages/film/test/film.test.tsindex 935f5622..41665ec7 100644--- a/packages/film/test/film.test.ts+++ b/packages/film/test/film.test.ts@@ -1,6 +1,8 @@import { compositionComponent, spatialComponent, videoContractManifests } from "../../../test/support/video-domain.js";import { registerTypeValidatorFacets } from "@hypit/component-kit";import { programSpaceTypes, sealProgramSpace } from "@hypit/program-space";+import { projectSemanticProgramSpace, semanticTrackTypes } from "@hypit/semantic-track";+import type { SemanticTrack } from "@hypit/semantic-track";import { compositionTypes, sealAudioTrack, sealVisualTrack } from "@hypit/composition";import type { Composition, Track } from "@hypit/composition";import type { FontArtifactRef } from "@hypit/media";@@ -8,6 +10,7 @@ import { sealCanvasSpace, spatialTypes } from "@hypit/spatial";import assert from "node:assert/strict";import test from "node:test";import { fixtureDigest } from "../../../test/fixture-digest.js";+import { semanticTrackFixture } from "../../../test/semantic-track-fixture.js";import {createResolvedClosure,@@ -36,6 +39,7 @@ const space = sealProgramSpace({durationSec: 4,frameRate: { numerator: 30, denominator: 1 },});+const semantic = semanticTrackFixture(space);function validatorRegistry(): TypeValidatorRegistry {const registry = new TypeValidatorRegistry();@@ -133,6 +137,7 @@ const closure = createResolvedClosure([]);const records = await Promise.all([sealRecord({ id: "space", type: programSpaceTypes.programSpace, value: stored(space) }),+ sealRecord({ id: "semantic", type: semanticTrackTypes.track, value: stored(semantic) }),sealRecord({ id: "canvas", type: spatialTypes.canvas, value: stored(canvas) }),sealRecord({ id: "film-program", type: filmTypes.program, value: stored(filmProgram) }),sealRecord({ id: "text-program", type: typographyTrackTypes.program, value: stored(textProgram) }),@@ -145,7 +150,7 @@ const textInstance = elaborateGraphFragment(linked, typographyTrackFragment, {id: "title",fragment: typographyTrackFragment.id,inputs: {- space: { kind: "record", id: "space" },+ semantic: { kind: "record", id: "semantic" },program: { kind: "record", id: "text-program" },},});@@ -165,7 +170,7 @@ const filmInstance = elaborateGraphFragment(linked, filmFragment, {inputs: {program: { kind: "record", id: "film-program" },canvas: { kind: "record", id: "canvas" },- space: { kind: "record", id: "space" },+ semantic: { kind: "record", id: "semantic" },title: { kind: "logical-output", id: "title.track" },background: { kind: "record", id: "background" },audio: { kind: "record", id: "audio" },@@ -224,7 +229,10 @@ test("Film stops at Composition and Hyperframes remains an ordinary downstream Ftest("the Driver folds peer Tracks, then independently compiles the Composition", async () => {const registry = new ProducerRegistry();registry.registerProducer(typographyTrackProducers.render, ({ inputs }) => ({- outputs: { track: stored(renderTypographyTrack(inline(inputs.space) as typeof space, inline(inputs.program) as typeof textProgram)) },+ outputs: { track: stored(renderTypographyTrack(+ projectSemanticProgramSpace(inline(inputs.semantic) as SemanticTrack),+ inline(inputs.program) as typeof textProgram,+ )) },needs: {},}));registry.registerProducer(filmProducers.createTrackSet, () => ({@@ -234,7 +242,7 @@ test("the Driver folds peer Tracks, then independently compiles the Composition"registry.registerProducer(filmProducers.appendVisualTrack, ({ inputs }) => ({outputs: { set: stored(appendFilmVisualTrack(inline(inputs.set) as never,- inline(inputs.space) as typeof space,+ projectSemanticProgramSpace(inline(inputs.semantic) as SemanticTrack),inline(inputs.track) as never,)) },needs: {},@@ -242,7 +250,7 @@ test("the Driver folds peer Tracks, then independently compiles the Composition"registry.registerProducer(filmProducers.appendAudioTrack, ({ inputs }) => ({outputs: { set: stored(appendFilmAudioTrack(inline(inputs.set) as never,- inline(inputs.space) as typeof space,+ projectSemanticProgramSpace(inline(inputs.semantic) as SemanticTrack),inline(inputs.track) as never,)) },needs: {},@@ -251,7 +259,7 @@ test("the Driver folds peer Tracks, then independently compiles the Composition"outputs: { composition: stored(compileFilmComposition(inline(inputs.program) as never,inline(inputs.canvas) as typeof canvas,- inline(inputs.space) as typeof space,+ projectSemanticProgramSpace(inline(inputs.semantic) as SemanticTrack),inline(inputs.set) as never,)) },needs: {},diff --git a/packages/film/test/surface.test.ts b/packages/film/test/surface.test.tsindex 4d4e4614..168e75a7 100644--- a/packages/film/test/surface.test.ts+++ b/packages/film/test/surface.test.ts@@ -1,11 +1,13 @@import { compositionComponent, spatialComponent, videoContractManifests } from "../../../test/support/video-domain.js";import { registerTypeValidatorFacets } from "@hypit/component-kit";import { programSpaceDependency, programSpaceTypes, sealProgramSpace } from "@hypit/program-space";+import { semanticTrackDependency, semanticTrackTypes } from "@hypit/semantic-track";import { compositionDependency, compositionTypes, sealAudioTrack, sealVisualTrack } from "@hypit/composition";import type { Track } from "@hypit/composition";import assert from "node:assert/strict";import test from "node:test";import { fixtureDigest } from "../../../test/fixture-digest.js";+import { semanticTrackFixture } from "../../../test/semantic-track-fixture.js";import { createResolvedClosure, sealBuildRequest, start } from "@hypit/core";import {@@ -40,14 +42,14 @@ const fixtureModule = { name: "example.film-fixture", version: "1" } as const;const fixtureSurfaceDigest = fixtureDigest("example.film-fixture/inputs-surface@1");const fixtureSurface = {name: "inputs", tag: "Inputs", mode: "structured",- outputs: [programSpaceTypes.programSpace, compositionTypes.visualTrack, compositionTypes.audioTrack],+ outputs: [semanticTrackTypes.track, compositionTypes.visualTrack, compositionTypes.audioTrack],} as const;const fixtureManifest: ModuleManifest = {format: "hypit.module@1",name: fixtureModule.name,version: fixtureModule.version,dependencies: [- programSpaceDependency,+ semanticTrackDependency,compositionDependency,],types: [],@@ -59,6 +61,7 @@ const space = sealProgramSpace({durationSec: 2,frameRate: { numerator: 30, denominator: 1 },});+const semantic = semanticTrackFixture(space);const visual = sealVisualTrack({visualIr: "hypit.visual-ir@1",id: "visual",@@ -102,7 +105,7 @@ async function compileFilm(options: { readonly styles?: string } = {}) {const surfaces = new MarkupSurfaceRegistry();surfaces.registerStructured({ module: fixtureModule, declaration: fixtureSurface, handler: ({ element }) => ({records: [- { id: "space", type: programSpaceTypes.programSpace, value: { kind: "inline", value: space }, range: element.range },+ { id: "semantic", type: semanticTrackTypes.track, value: { kind: "inline", value: semantic }, range: element.range },{ id: "visual", type: compositionTypes.visualTrack, value: { kind: "inline", value: visual }, range: element.range },{ id: "audio", type: compositionTypes.audioTrack, value: { kind: "inline", value: audio }, range: element.range },],@@ -138,7 +141,7 @@ async function compileFilm(options: { readonly styles?: string } = {}) {<import as="studio" source="./recipes.svs"/><fixture:Inputs/><space:Canvas id="vertical" width="1080" height="1920"/>- <film:Film id="main" canvas={vertical} space={space} appearance={studio.film.vertical}>+ <film:Film id="main" canvas={vertical} semantic={semantic} appearance={studio.film.vertical}><film:Track source={visual}/><film:Track source={audio}/></film:Film></svml>`),diff --git a/packages/hyperframes/test/browser-visual.test.ts b/packages/hyperframes/test/browser-visual.test.tsindex f2ce7b60..f680dbfd 100644--- a/packages/hyperframes/test/browser-visual.test.ts+++ b/packages/hyperframes/test/browser-visual.test.ts@@ -5,6 +5,7 @@ import { copyFile, mkdir, mkdtemp, readdir, rm, writeFile } from "node:fs/promisimport os from "node:os";import path from "node:path";import test from "node:test";+import { semanticTrackFixture } from "../../../test/semantic-track-fixture.js";import { deflateSync } from "node:zlib";import { resolveCaptionProgram } from "@hypit/caption";import type { TimedCaptionProjection } from "@hypit/caption";@@ -1300,6 +1301,7 @@ test("Media two-frame sampling, alpha, local motion and handoff survive partitioconst space = sealProgramSpace({durationSec: 1, frameRate: { numerator: 12, denominator: 1 },});+ const semantic = semanticTrackFixture(space);const header = mediaTrack.sealMediaTrackHeader({ id: "browser-media" });const sampleAppearance = { opacity: 1, filter: { blurPx: 0, brightness: 1, contrast: 1, saturation: 1 } };const contentFit = (sizing: "contain" | "cover") => ({@@ -1321,7 +1323,7 @@ test("Media two-frame sampling, alpha, local motion and handoff survive partitio] },}));let set = mediaTrack.appendProgramMediaItem(- mediaTrack.createMediaTrackSet(), header, space, canvas, itemLayers,+ mediaTrack.createMediaTrackSet(), header, semantic, canvas, itemLayers,{ xPx: 0, yPx: 0, widthPx: 80, heightPx: 120 },mediaTrack.sealMediaItemSpec({id: "two-frame",@@ -1437,6 +1439,7 @@ test("DepthStack Deck reflow, exact labels and old-system layout survive partitidurationSec: frames / fps,frameRate: { numerator: fps, denominator: 1 },});+ const semantic = semanticTrackFixture(space);const canvas = sealCanvasSpace({widthPx: width, heightPx: height,origin: "top-left", xDirection: "right", yDirection: "down", pixelAspect: "square",@@ -1533,7 +1536,7 @@ test("DepthStack Deck reflow, exact labels and old-system layout survive partiti{ xPx: 60, yPx: 45, widthPx: 120, heightPx: 90 },spec,frames,- space,+ semantic,);const track = deckTrack.renderDepthStack(canvas, space, program);const document = compileHyperframesDocument(sealComposition({@@ -1617,17 +1620,14 @@ test("all three Ranking components paint frame-pure progressive states under pardurationSec: frames / fps,frameRate: { numerator: fps, denominator: 1 },});- const map = {- tokens: [],- anchors: [+ const semantic = semanticTrackFixture(space, { anchors: [{ identity: "outer-start", frame: 0 },{ identity: "rank-1", frame: 2 },{ identity: "rank-2", frame: 8 },{ identity: "rank-3", frame: 14 },{ identity: "terminal", frame: 20 },{ identity: "outer-end", frame: frames },- ],- };+ ] });const outer = {id: "ranking-window",occurrences: [{ occurrence: 0, startAnchorId: "outer-start", endAnchorId: "outer-end" }],@@ -1643,7 +1643,7 @@ test("all three Ranking components paint frame-pure progressive states under parconst schedule = (header: rankingTrack.RankingHeader, specs: readonly rankingTrack.RankingItemSpec[]) => {let set = rankingTrack.createRankingItemSpecSet(header);for (const spec of specs) set = rankingTrack.appendRankingItemSpec(set, spec);- return rankingTrack.buildRankingSchedule({ header, items: set, map, space, outer, triggers, terminal });+ return rankingTrack.buildRankingSchedule({ header, items: set, semantic, outer, triggers, terminal });};const recipe = (path: string, properties: SvsRecipe["properties"]): SvsRecipe => ({path, properties,@@ -1682,7 +1682,7 @@ test("all three Ranking components paint frame-pure progressive states under parconst columnSchedule = rankingTrack.buildColumnSchedule({header: columnHeader,items: columnSpecSet,- outer: rankingTrack.projectColumnSelectionOuterWindow(map, space, outer),+ outer: rankingTrack.projectColumnSelectionOuterWindow(semantic, outer),candidates: rankingTrack.createColumnWindowCandidateSet(),});const columnCanvas = sealCanvasSpace({@@ -1800,6 +1800,7 @@ test("all Spatial fit modes and equal-point alignments reach exact browser pixelconst space = sealProgramSpace({durationSec: 1, frameRate: { numerator: 1, denominator: 1 },});+ const semantic = semanticTrackFixture(space);const header = mediaTrack.sealMediaTrackHeader({ id: "spatial-browser" });const extent = { widthPx: 40, heightPx: 20 };const appearance = { opacity: 1, filter: { blurPx: 0, brightness: 1, contrast: 1, saturation: 1 } };@@ -1824,7 +1825,7 @@ test("all Spatial fit modes and equal-point alignments reach exact browser pixel}),);set = mediaTrack.appendProgramMediaItem(- set, header, space, canvas, layers, frame,+ set, header, semantic, canvas, layers, frame,mediaTrack.sealMediaItemSpec({id,projection: { start: { ref: "program.start" }, end: { ref: "program.end" } },@@ -1981,6 +1982,7 @@ test("every official Screen Overlay survives real sequential and parallel browsedurationSec: 1,frameRate: { numerator: 12, denominator: 1 },});+ const semantic = semanticTrackFixture(space);const header = sealScreenOverlayHeader({ id: "browser-overlays" });const components: readonly ScreenOverlayComponent[] = [{ kind: "flash", color: "#ffffff", intensity: 0.2, attackFrames: 2, holdFrames: 2, decayFrames: 4 },@@ -1997,7 +1999,7 @@ test("every official Screen Overlay survives real sequential and parallel browse];let set: ScreenOverlaySet = createScreenOverlaySet();components.forEach((content, index) => {- set = appendProgramScreenOverlay(set, header, space, sealScreenOverlayItemSpec({+ set = appendProgramScreenOverlay(set, header, semantic, sealScreenOverlayItemSpec({id: `${content.kind}-${index + 1}`,content,diff --git a/packages/media-execution/src/execute.ts b/packages/media-execution/src/execute.tsindex dce81690..7d6bd909 100644--- a/packages/media-execution/src/execute.ts+++ b/packages/media-execution/src/execute.ts@@ -845,7 +845,7 @@ export async function executeTransformMedia(}}-/** Generic model-reference audio: no Narrative, SpeechBasis or alignment claim is introduced. */+/** Generic model-reference audio: no Narrative, SemanticTake or alignment claim is introduced. */export async function executeExtractAudio(env: MediaExecutionEnvironment,constraints: CanonicalValue,diff --git a/packages/media-pipeline/README.md b/packages/media-pipeline/README.mdindex 6b65bcde..b6871300 100644--- a/packages/media-pipeline/README.md+++ b/packages/media-pipeline/README.md@@ -32,7 +32,7 @@ This package is the shared execution vocabulary. Speech authoring makes the bounalignment produce one `SemanticTake`. Media Track consumes that prepared value; it never owns animplicit visual-only normalization policy.-The frame clock is a separate program fact and can be shared by every Normalize and Speech Spine:+The frame clock is a separate program fact and can be shared by every Normalize and Speech Track:```svml<program:Clock id="clock" frame-rate="30"/>@@ -58,7 +58,7 @@ Three ordinary author operations reuse that same inspection/execution boundary:```Every result is an ordinary `BlobArtifact`. Audio extraction emits a deterministic 48 kHz stereo PCM-WAV but makes no `SpeechBasis`, speaker or alignment claim; it can therefore feed a later model+WAV but makes no `SemanticTake`, speaker or alignment claim; it can therefore feed a later modelreference port directly. Frame extraction supports `first`, `last`, `frame:<index>` and`time:<seconds>`. Transform operations are ordered author meaning and never an arbitrary FFmpeg string.diff --git a/packages/media-pipeline/src/manifest.ts b/packages/media-pipeline/src/manifest.tsindex 18bddc3e..6391f956 100644--- a/packages/media-pipeline/src/manifest.ts+++ b/packages/media-pipeline/src/manifest.ts@@ -233,7 +233,7 @@ export const mediaPipelineMarkupSurfaces = [notes: ["Write exactly one of recipe or the direct video/audio/span-authority attributes, and exactly one of clock or frame-rate.","`primary-moving` excludes attached-picture streams, prefers one declared default and fails closed on an ambiguous container; `stream:<index>` is for a container the author genuinely knows.",- "Selecting embedded audio is a media fact only and makes no SpeechBasis, speaker or alignment claim.",+ "Selecting embedded audio is a media fact only and makes no SemanticTake, speaker or alignment claim.",],},},@@ -308,7 +308,7 @@ export const mediaPipelineMarkupSurfaces = [example: '<media:ExtractAudio id="voice-reference" source={prepared.video} audio="default"/>',notes: ["The element accepts no children and no text content, and the output container, codec, sample rate and channel count are fixed.",- "The result makes no SpeechBasis, speaker or alignment claim, so it can feed a model reference port directly.",+ "The result makes no SemanticTake, speaker or alignment claim, so it can feed a model reference port directly.",],},},diff --git a/packages/media-pipeline/src/selection.ts b/packages/media-pipeline/src/selection.tsindex a602a485..ba9ac100 100644--- a/packages/media-pipeline/src/selection.ts+++ b/packages/media-pipeline/src/selection.ts@@ -85,7 +85,7 @@ function selectedAudio(inspection: MediaInspection, request: MediaSelectionReque/*** Selects container streams only. Choosing an audio stream never claims that it is narrated speech;- * SpeechBasis construction remains a separate semantic Producer with an explicit Narrative edge.+ * SemanticTake construction remains a separate semantic Producer with an explicit Narrative edge.*/export function selectMediaStreams(inspection: MediaInspection,diff --git a/packages/media-track/README.md b/packages/media-track/README.mdindex fd1e9aa9..c939fa46 100644--- a/packages/media-track/README.md+++ b/packages/media-track/README.md@@ -3,7 +3,7 @@Official provider-free Media Item and Sequence authoring package.`Track` accepts explicit ProgramSpace, CanvasSpace and, when semantic points are used, a-CompleteSemanticMap. Each independently timed `Item` receives an explicit SpatialFrame and either+SemanticTrack. Each independently timed `Item` receives an explicit SpatialFrame and eitherone direct source or ordered Paint/sample layers. A visual source is always named by its actualform: `image`, prepared `media`, or compositable `surface`. Moving video must be prepared by anexplicit Media Pipeline `Normalize` Operation before it enters the Track; the Track never guesses a@@ -13,7 +13,7 @@ frame Paint, clipping, borders, shadows, lifecycle motion and sampling motion.Visual and audio are separate deterministic projections. A visual-only Track exports only`VisualTrack`; selecting one source layer's audio or adding explicit enter/exit/handoff sounds also-exports a peer `AudioTrack`. Speech Spine reuses the same lowering laws through a deliberately+exports a peer `AudioTrack`. Speech Track reuses the same lowering laws through a deliberatelyrestricted internal projection.The package lowers only track composition. It does not add Media, B-roll or Provider meaning to Core,diff --git a/packages/media-track/package.json b/packages/media-track/package.jsonindex d2ddd7d8..12d07fcc 100644--- a/packages/media-track/package.json+++ b/packages/media-track/package.json@@ -19,7 +19,7 @@"@hypit/narrative": "workspace:*","@hypit/program-space": "workspace:*","@hypit/protocol": "workspace:*",- "@hypit/semantic-map": "workspace:*",+ "@hypit/semantic-track": "workspace:*","@hypit/spatial": "workspace:*","@hypit/svs": "workspace:*","@hypit/markup": "workspace:*",diff --git a/packages/media-track/src/component.ts b/packages/media-track/src/component.tsindex f899b095..260b6e3b 100644--- a/packages/media-track/src/component.ts+++ b/packages/media-track/src/component.ts@@ -1,10 +1,10 @@import type { ComponentPackage, ProducerHandlerContext } from "@hypit/component-kit";import type { CompositableSurfaceRef, SynchronizedMedia } from "@hypit/media";import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrative";-import type { ProgramSpace } from "@hypit/program-space";import { canonicalize } from "@hypit/protocol";import type { BlobRef, StoredValue } from "@hypit/protocol";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import { projectSemanticProgramSpace } from "@hypit/semantic-track";+import type { SemanticTrack } from "@hypit/semantic-track";import type { CanvasSpace, ContentFit, IntrinsicExtent, SpatialFrame, SpatialPath } from "@hypit/spatial";import {@@ -54,7 +54,7 @@ function itemInputs(inputs: ProducerHandlerContext["inputs"]) {return {set: inline<MediaTrackSet>(inputs.set?.value, "MediaTrackSet"),header: inline<MediaTrackHeader>(inputs.header?.value, "MediaTrackHeader"),- space: inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ semantic: inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"),canvas: inline<CanvasSpace>(inputs.canvas?.value, "CanvasSpace"),layers: inline<MediaLayerSet>(inputs.layers?.value, "MediaLayerSet"),frame: inline<SpatialFrame>(inputs.frame?.value, "SpatialFrame"),@@ -68,8 +68,7 @@ function memberInputs(inputs: ProducerHandlerContext["inputs"]) {members: inline<MediaSequenceMemberSet>(inputs.members?.value, "MediaSequenceMemberSet"),layers: inline<MediaLayerSet>(inputs.layers?.value, "MediaLayerSet"),spec: inline<MediaSequenceMemberSpec>(inputs.spec?.value, "MediaSequenceMemberSpec"),- map: inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),- space: inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ semantic: inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"),};}@@ -77,7 +76,7 @@ function sequenceInputs(inputs: ProducerHandlerContext["inputs"]) {return {set: inline<MediaTrackSet>(inputs.set?.value, "MediaTrackSet"),header: inline<MediaTrackHeader>(inputs.header?.value, "MediaTrackHeader"),- space: inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ semantic: inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"),canvas: inline<CanvasSpace>(inputs.canvas?.value, "CanvasSpace"),members: inline<MediaSequenceMemberSet>(inputs.members?.value, "MediaSequenceMemberSet"),frame: inline<SpatialFrame>(inputs.frame?.value, "SpatialFrame"),@@ -115,24 +114,21 @@ export const mediaTrackComponent = {{ producer: mediaTrackProducers.createSet, handler: () => ({ outputs: { set: output(createMediaTrackSet()) }, needs: {} }) },{ producer: mediaTrackProducers.appendProgramItem, handler: ({ inputs }) => {const value = itemInputs(inputs);- return { outputs: { set: output(appendProgramMediaItem(value.set, value.header, value.space, value.canvas, value.layers, value.frame, value.spec, value.sounds)) }, needs: {} };+ return { outputs: { set: output(appendProgramMediaItem(value.set, value.header, value.semantic, value.canvas, value.layers, value.frame, value.spec, value.sounds)) }, needs: {} };} },{ producer: mediaTrackProducers.appendSelectionItem, handler: ({ inputs }) => {const value = itemInputs(inputs);- return { outputs: { set: output(appendSelectionMediaItem(value.set, value.header, value.space, value.canvas, value.layers, value.frame,- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ return { outputs: { set: output(appendSelectionMediaItem(value.set, value.header, value.semantic, value.canvas, value.layers, value.frame,inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"), value.spec, value.sounds)) }, needs: {} };} },{ producer: mediaTrackProducers.appendSegmentItem, handler: ({ inputs }) => {const value = itemInputs(inputs);- return { outputs: { set: output(appendSegmentMediaItem(value.set, value.header, value.space, value.canvas, value.layers, value.frame,- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ return { outputs: { set: output(appendSegmentMediaItem(value.set, value.header, value.semantic, value.canvas, value.layers, value.frame,inline<NarrativeExcerpt>(inputs.segment?.value, "NarrativeExcerpt"), value.spec, value.sounds)) }, needs: {} };} },{ producer: mediaTrackProducers.appendMomentItem, handler: ({ inputs }) => {const value = itemInputs(inputs);- return { outputs: { set: output(appendMomentMediaItem(value.set, value.header, value.space, value.canvas, value.layers, value.frame,- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ return { outputs: { set: output(appendMomentMediaItem(value.set, value.header, value.semantic, value.canvas, value.layers, value.frame,inline<NarrativeMomentRef>(inputs.moment?.value, "NarrativeMomentRef"), value.spec, value.sounds)) }, needs: {} };} },{ producer: mediaTrackProducers.bindItemClipPath, handler: ({ inputs }) => ({ outputs: { spec: output(bindMediaItemClipPath(@@ -147,63 +143,60 @@ export const mediaTrackComponent = {{ producer: mediaTrackProducers.appendMomentMember, handler: ({ inputs }) => {const value = memberInputs(inputs);return { outputs: { members: output(appendMediaSequenceMomentMember(- value.members, value.layers, value.spec, value.map,- inline<NarrativeMomentRef>(inputs.moment?.value, "NarrativeMomentRef"), value.space,+ value.members, value.layers, value.spec, value.semantic,+ inline<NarrativeMomentRef>(inputs.moment?.value, "NarrativeMomentRef"),)) }, needs: {} };} },{ producer: mediaTrackProducers.appendSelectionStartMember, handler: ({ inputs }) => {const value = memberInputs(inputs);return { outputs: { members: output(appendMediaSequenceSelectionMember(- value.members, value.layers, value.spec, value.map,- inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"), value.space, "start",+ value.members, value.layers, value.spec, value.semantic,+ inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"), "start",)) }, needs: {} };} },{ producer: mediaTrackProducers.appendSelectionEndMember, handler: ({ inputs }) => {const value = memberInputs(inputs);return { outputs: { members: output(appendMediaSequenceSelectionMember(- value.members, value.layers, value.spec, value.map,- inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"), value.space, "end",+ value.members, value.layers, value.spec, value.semantic,+ inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"), "end",)) }, needs: {} };} },{ producer: mediaTrackProducers.appendSequenceProgramEnd, handler: ({ inputs }) => {const value = sequenceInputs(inputs);return { outputs: { set: output(appendMediaSequenceUntilProgramEnd(- value.set, value.header, value.space, value.canvas, value.members, value.frame, value.spec, value.sounds,+ value.set, value.header, value.semantic, value.canvas, value.members, value.frame, value.spec, value.sounds,)) }, needs: {} };} },{ producer: mediaTrackProducers.appendSequenceUntilMoment, handler: ({ inputs }) => {const value = sequenceInputs(inputs);return { outputs: { set: output(appendMediaSequenceUntilMoment(- value.set, value.header, value.space, value.canvas, value.members, value.frame, value.spec, value.sounds,- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ value.set, value.header, value.semantic, value.canvas, value.members, value.frame, value.spec, value.sounds,inline<NarrativeMomentRef>(inputs.moment?.value, "NarrativeMomentRef"),)) }, needs: {} };} },{ producer: mediaTrackProducers.appendSequenceUntilSelectionStart, handler: ({ inputs }) => {const value = sequenceInputs(inputs);return { outputs: { set: output(appendMediaSequenceUntilSelection(- value.set, value.header, value.space, value.canvas, value.members, value.frame, value.spec, value.sounds,- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ value.set, value.header, value.semantic, value.canvas, value.members, value.frame, value.spec, value.sounds,inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"), "start",)) }, needs: {} };} },{ producer: mediaTrackProducers.appendSequenceUntilSelectionEnd, handler: ({ inputs }) => {const value = sequenceInputs(inputs);return { outputs: { set: output(appendMediaSequenceUntilSelection(- value.set, value.header, value.space, value.canvas, value.members, value.frame, value.spec, value.sounds,- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ value.set, value.header, value.semantic, value.canvas, value.members, value.frame, value.spec, value.sounds,inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"), "end",)) }, needs: {} };} },{ producer: mediaTrackProducers.finalize, handler: ({ inputs }) => ({ outputs: { program: output(finalizeMediaTrack(inline<MediaTrackSet>(inputs.set?.value, "MediaTrackSet"), inline<MediaTrackHeader>(inputs.header?.value, "MediaTrackHeader"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")),)) }, needs: {} }) },{ producer: mediaTrackProducers.projectVisual, handler: ({ inputs }) => ({ outputs: { track: output(projectMediaVisualTrack(- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<MediaTrackProgram>(inputs.program?.value, "MediaTrackProgram"),+ projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")), inline<MediaTrackProgram>(inputs.program?.value, "MediaTrackProgram"),)) }, needs: {} }) },{ producer: mediaTrackProducers.projectAudio, handler: ({ inputs }) => ({ outputs: { track: output(projectMediaAudioTrack(- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<MediaTrackProgram>(inputs.program?.value, "MediaTrackProgram"),+ projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")), inline<MediaTrackProgram>(inputs.program?.value, "MediaTrackProgram"),)) }, needs: {} }) },],validators: [{diff --git a/packages/media-track/src/fragment.ts b/packages/media-track/src/fragment.tsindex b28ac9c5..dbdc112d 100644--- a/packages/media-track/src/fragment.ts+++ b/packages/media-track/src/fragment.ts@@ -1,7 +1,7 @@import { artifactTypes } from "@hypit/artifact";import { compositionTypes } from "@hypit/composition";import { sealGraphFragment } from "@hypit/elaborator";-import { programSpaceTypes } from "@hypit/program-space";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import { mediaTrackProducers, mediaTrackTypes } from "./manifest.js";@@ -13,7 +13,7 @@ const operation = (id: string) => ({ kind: "fragment-operation" as const, operatexport const stillMediaTrackFragment = sealGraphFragment({inputs: [{ name: "header", type: mediaTrackTypes.header },- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "canvas", type: spatialTypes.canvas },{ name: "source", type: artifactTypes.blob },{ name: "extent", type: spatialTypes.extent },@@ -30,14 +30,14 @@ export const stillMediaTrackFragment = sealGraphFragment({{ id: "set", producer: mediaTrackProducers.createSet, inputs: {}, result: { kind: "output", name: "set" } },{ id: "sounds", producer: mediaTrackProducers.createSounds, inputs: {}, result: { kind: "output", name: "sounds" } },{ id: "append", producer: mediaTrackProducers.appendProgramItem, inputs: {- set: operation("set"), header: input("header"), space: input("space"), canvas: input("canvas"), layers: operation("sample"),+ set: operation("set"), header: input("header"), semantic: input("semantic"), canvas: input("canvas"), layers: operation("sample"),frame: input("frame"), spec: input("item-spec"), sounds: operation("sounds"),}, result: { kind: "output", name: "set" } },{ id: "finalize", producer: mediaTrackProducers.finalize, inputs: {- set: operation("append"), header: input("header"), space: input("space"),+ set: operation("append"), header: input("header"), semantic: input("semantic"),}, result: { kind: "output", name: "program" } },{ id: "visual", producer: mediaTrackProducers.projectVisual, inputs: {- space: input("space"), program: operation("finalize"),+ semantic: input("semantic"), program: operation("finalize"),}, result: { kind: "output", name: "track" } },],exports: [@@ -47,7 +47,7 @@ export const stillMediaTrackFragment = sealGraphFragment({});export const renderMediaTrackFragment = sealGraphFragment({- inputs: [{ name: "space", type: programSpaceTypes.programSpace }, { name: "program", type: mediaTrackTypes.program }],- operations: [{ id: "visual", producer: mediaTrackProducers.projectVisual, inputs: { space: input("space"), program: input("program") }, result: { kind: "output", name: "track" } }],+ inputs: [{ name: "semantic", type: semanticTrackTypes.track }, { name: "program", type: mediaTrackTypes.program }],+ operations: [{ id: "visual", producer: mediaTrackProducers.projectVisual, inputs: { semantic: input("semantic"), program: input("program") }, result: { kind: "output", name: "track" } }],exports: [{ name: "track", type: compositionTypes.visualTrack, root: operation("visual") }],});diff --git a/packages/media-track/src/manifest.ts b/packages/media-track/src/manifest.tsindex 12e473e9..aab25484 100644--- a/packages/media-track/src/manifest.ts+++ b/packages/media-track/src/manifest.ts@@ -8,9 +8,8 @@ import {mediaTypes,} from "@hypit/media";import { narrativeDependency, narrativeTypes } from "@hypit/narrative";-import { programSpaceDependency, programSpaceTypes } from "@hypit/program-space";import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@hypit/protocol";-import { semanticMapDependency, semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackDependency, semanticTrackTypes } from "@hypit/semantic-track";import {contentFitSchema,intrinsicExtentSchema,@@ -275,18 +274,18 @@ export const mediaTrackHeaderSchema: ValueSchema = object({const itemInputs = [{ name: "set", type: mediaTrackTypes.set }, { name: "header", type: mediaTrackTypes.header },- { name: "space", type: programSpaceTypes.programSpace }, { name: "canvas", type: spatialTypes.canvas },+ { name: "semantic", type: semanticTrackTypes.track }, { name: "canvas", type: spatialTypes.canvas },{ name: "layers", type: mediaTrackTypes.layerSet },{ name: "frame", type: spatialTypes.frame }, { name: "spec", type: mediaTrackTypes.itemSpec },{ name: "sounds", type: mediaTrackTypes.soundSet },] as const;const memberInputs = [{ name: "members", type: mediaTrackTypes.memberSet }, { name: "layers", type: mediaTrackTypes.layerSet },- { name: "spec", type: mediaTrackTypes.memberSpec }, { name: "map", type: semanticMapTypes.complete },+ { name: "spec", type: mediaTrackTypes.memberSpec }, { name: "semantic", type: semanticTrackTypes.track },] as const;const sequenceInputs = [{ name: "set", type: mediaTrackTypes.set }, { name: "header", type: mediaTrackTypes.header },- { name: "space", type: programSpaceTypes.programSpace }, { name: "canvas", type: spatialTypes.canvas },+ { name: "semantic", type: semanticTrackTypes.track }, { name: "canvas", type: spatialTypes.canvas },{ name: "members", type: mediaTrackTypes.memberSet },{ name: "frame", type: spatialTypes.frame }, { name: "spec", type: mediaTrackTypes.sequenceSpec },{ name: "sounds", type: mediaTrackTypes.soundSet },@@ -386,18 +385,16 @@ export const mediaTrackMarkupSurfaces = [{mediaTrackTypes.handoffSpec, mediaTrackTypes.sequenceSpec, mediaTrackTypes.set, mediaTrackTypes.program,compositionTypes.visualTrack, compositionTypes.audioTrack],vocabulary: {- summary: "One Media Track: independently timed Items and replacement Sequences placed on a shared ProgramSpace and Canvas, lowered to one peer VisualTrack and, when audio is authored, one peer AudioTrack.",+ summary: "One Media Track: independently timed Items and replacement Sequences placed on a shared SemanticTrack and Canvas, lowered to one peer VisualTrack and, when audio is authored, one peer AudioTrack.",appearance: "Rectangular pictures, each filling its own Frame exactly where that Frame sits on the Canvas, and nothing besides: the Track draws no chrome, caption or backdrop of its own, and lays nothing out relative to anything else. Inside one Frame the layers stack back to front in the order they are written — flat or gradient Paint fills, then a still image, a video or a Surface scaled in by the fit — clipped to the Frame as a square, a rounded rectangle or an authored Path, optionally ringed by a border and sitting on drop shadows. An Item appears for its own window and leaves at the end of it, entering and exiting on one operator such as a fade, a slide from an edge, a scale, a pop or a wipe, holding a small continuous float, pulse or drift while it is up, and panning, zooming or rotating its picture inside the Frame across the window. A Sequence instead keeps one Frame occupied without a break and replaces the picture inside it at each activation point, the outgoing picture giving way on a cut, a crossfade, a push, a wipe, a cover or a page-turn; where two units overlap, the higher stack order draws in front.",preview: previewImage("Track.png"),attributes: [{ name: "id", kind: "identifier", required: true,summary: "Names this Track and prefixes the identity of every Item, Sequence, layer and sound that does not name itself." },- { name: "space", kind: "reference", required: true, accepts: [programSpaceTypes.programSpace],- summary: "Fixes the frame domain every window resolves into and the consumption policy raw video is normalized against." },+ { name: "semantic", kind: "reference", required: true, accepts: [semanticTrackTypes.track],+ summary: "Selects the continuous SemanticTrack that fixes the frame domain and every semantic anchor." },{ name: "canvas", kind: "reference", required: true, accepts: [spatialTypes.canvas],summary: "Chooses the Canvas every Frame on this Track is measured inside." },- { name: "map", kind: "reference", required: false, accepts: [semanticMapTypes.complete],- summary: "Chooses the SemanticMap that turns semantic identity into exact time." },],children: [{ tag: "Item", cardinality: "many",@@ -479,12 +476,12 @@ export const mediaTrackMarkupSurfaces = [{{ name: "audio", type: compositionTypes.audioTrack,summary: "The rendered sound, published only when a source audio selection or a Sound is authored." },],- example: `<media-track:Track id="cutaways" map={timing.map} space={speech.space} canvas={vertical}>+ example: `<media-track:Track id="cutaways" semantic={speech.semantic} canvas={vertical}><media-track:Item id="bags" video={cutaway-bags.video} during={story.selection.bags}frame={full} appearance={studio.media.cutaway} motion={studio.motion.cut}/></media-track:Track>`,notes: [- "A Track requires at least one Item or Sequence, accepts no text content, and refuses `map` when no child consumes semantic timing.",+ "A Track requires at least one Item or Sequence and accepts no text content.","An Item states exactly one window form: `during`, `at` with `for`, or `start` with `end`; `selection`, `segment` and `moment` bind a start/end window and cannot be written together.","A point expression is `program.start`, `program.end`, `selection.start`, `selection.end`, `segment.start`, `segment.end` or `moment.cue`, each optionally offset by `+` or `-` and a duration, or a bare duration read as an absolute position.","A unit that names a direct source names exactly one of `image`, `video`, `media` or `surface`; `extent` is required with `image` and refused otherwise, and `audio` is only valid with `video`.",@@ -597,8 +594,7 @@ export const mediaTrackManifest: ModuleManifest = {artifactDependency,mediaDependency,narrativeDependency,- semanticMapDependency,- programSpaceDependency,+ semanticTrackDependency,temporalDependency,spatialDependency,compositionDependency,@@ -629,22 +625,22 @@ export const mediaTrackManifest: ModuleManifest = {{ name: mediaTrackProducers.appendSound.name, inputs: [{ name: "sounds", type: mediaTrackTypes.soundSet }, { name: "source", type: mediaTypes.synchronized }, { name: "spec", type: mediaTrackTypes.soundSpec }], outputs: [{ name: "sounds", type: mediaTrackTypes.soundSet }], needs: [] },{ name: mediaTrackProducers.createSet.name, inputs: [], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },{ name: mediaTrackProducers.appendProgramItem.name, inputs: [...itemInputs], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },- { name: mediaTrackProducers.appendSelectionItem.name, inputs: [...itemInputs, { name: "map", type: semanticMapTypes.complete }, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },- { name: mediaTrackProducers.appendSegmentItem.name, inputs: [...itemInputs, { name: "map", type: semanticMapTypes.complete }, { name: "segment", type: narrativeTypes.excerpt }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },- { name: mediaTrackProducers.appendMomentItem.name, inputs: [...itemInputs, { name: "map", type: semanticMapTypes.complete }, { name: "moment", type: narrativeTypes.moment }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },+ { name: mediaTrackProducers.appendSelectionItem.name, inputs: [...itemInputs, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },+ { name: mediaTrackProducers.appendSegmentItem.name, inputs: [...itemInputs, { name: "segment", type: narrativeTypes.excerpt }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },+ { name: mediaTrackProducers.appendMomentItem.name, inputs: [...itemInputs, { name: "moment", type: narrativeTypes.moment }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },{ name: mediaTrackProducers.bindItemClipPath.name, inputs: [{ name: "spec", type: mediaTrackTypes.itemSpec }, { name: "path", type: spatialTypes.path }], outputs: [{ name: "spec", type: mediaTrackTypes.itemSpec }], needs: [] },{ name: mediaTrackProducers.bindSequenceClipPath.name, inputs: [{ name: "spec", type: mediaTrackTypes.sequenceSpec }, { name: "path", type: spatialTypes.path }], outputs: [{ name: "spec", type: mediaTrackTypes.sequenceSpec }], needs: [] },{ name: mediaTrackProducers.createMembers.name, inputs: [], outputs: [{ name: "members", type: mediaTrackTypes.memberSet }], needs: [] },- { name: mediaTrackProducers.appendMomentMember.name, inputs: [...memberInputs, { name: "moment", type: narrativeTypes.moment }, { name: "space", type: programSpaceTypes.programSpace }], outputs: [{ name: "members", type: mediaTrackTypes.memberSet }], needs: [] },- { name: mediaTrackProducers.appendSelectionStartMember.name, inputs: [...memberInputs, { name: "selection", type: narrativeTypes.selection }, { name: "space", type: programSpaceTypes.programSpace }], outputs: [{ name: "members", type: mediaTrackTypes.memberSet }], needs: [] },- { name: mediaTrackProducers.appendSelectionEndMember.name, inputs: [...memberInputs, { name: "selection", type: narrativeTypes.selection }, { name: "space", type: programSpaceTypes.programSpace }], outputs: [{ name: "members", type: mediaTrackTypes.memberSet }], needs: [] },+ { name: mediaTrackProducers.appendMomentMember.name, inputs: [...memberInputs, { name: "moment", type: narrativeTypes.moment }], outputs: [{ name: "members", type: mediaTrackTypes.memberSet }], needs: [] },+ { name: mediaTrackProducers.appendSelectionStartMember.name, inputs: [...memberInputs, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "members", type: mediaTrackTypes.memberSet }], needs: [] },+ { name: mediaTrackProducers.appendSelectionEndMember.name, inputs: [...memberInputs, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "members", type: mediaTrackTypes.memberSet }], needs: [] },{ name: mediaTrackProducers.appendSequenceProgramEnd.name, inputs: sequenceInputs, outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },- { name: mediaTrackProducers.appendSequenceUntilMoment.name, inputs: [...sequenceInputs, { name: "map", type: semanticMapTypes.complete }, { name: "moment", type: narrativeTypes.moment }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },- { name: mediaTrackProducers.appendSequenceUntilSelectionStart.name, inputs: [...sequenceInputs, { name: "map", type: semanticMapTypes.complete }, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },- { name: mediaTrackProducers.appendSequenceUntilSelectionEnd.name, inputs: [...sequenceInputs, { name: "map", type: semanticMapTypes.complete }, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },- { name: mediaTrackProducers.finalize.name, inputs: [{ name: "set", type: mediaTrackTypes.set }, { name: "header", type: mediaTrackTypes.header }, { name: "space", type: programSpaceTypes.programSpace }], outputs: [{ name: "program", type: mediaTrackTypes.program }], needs: [] },- { name: mediaTrackProducers.projectVisual.name, inputs: [{ name: "space", type: programSpaceTypes.programSpace }, { name: "program", type: mediaTrackTypes.program }], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [] },- { name: mediaTrackProducers.projectAudio.name, inputs: [{ name: "space", type: programSpaceTypes.programSpace }, { name: "program", type: mediaTrackTypes.program }], outputs: [{ name: "track", type: compositionTypes.audioTrack }], needs: [] },+ { name: mediaTrackProducers.appendSequenceUntilMoment.name, inputs: [...sequenceInputs, { name: "moment", type: narrativeTypes.moment }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },+ { name: mediaTrackProducers.appendSequenceUntilSelectionStart.name, inputs: [...sequenceInputs, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },+ { name: mediaTrackProducers.appendSequenceUntilSelectionEnd.name, inputs: [...sequenceInputs, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [] },+ { name: mediaTrackProducers.finalize.name, inputs: [{ name: "set", type: mediaTrackTypes.set }, { name: "header", type: mediaTrackTypes.header }, { name: "semantic", type: semanticTrackTypes.track }], outputs: [{ name: "program", type: mediaTrackTypes.program }], needs: [] },+ { name: mediaTrackProducers.projectVisual.name, inputs: [{ name: "semantic", type: semanticTrackTypes.track }, { name: "program", type: mediaTrackTypes.program }], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [] },+ { name: mediaTrackProducers.projectAudio.name, inputs: [{ name: "semantic", type: semanticTrackTypes.track }, { name: "program", type: mediaTrackTypes.program }], outputs: [{ name: "track", type: compositionTypes.audioTrack }], needs: [] },],};diff --git a/packages/media-track/src/program.ts b/packages/media-track/src/program.tsindex 4830e05f..8ad7eb91 100644--- a/packages/media-track/src/program.ts+++ b/packages/media-track/src/program.ts@@ -14,7 +14,8 @@ import {} from "@hypit/program-space";import type { ProgramSpace } from "@hypit/program-space";import { canonicalize, isDigest } from "@hypit/protocol";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import { projectSemanticProgramSpace } from "@hypit/semantic-track";+import type { SemanticTrack } from "@hypit/semantic-track";import { assertCanvasSpace, assertSpatialFrame } from "@hypit/spatial";import type { CanvasSpace } from "@hypit/spatial";import { assertSpatialPath } from "@hypit/spatial";@@ -207,17 +208,18 @@ function realizedItems(export function appendProgramMediaItem(set: MediaTrackSet,header: MediaTrackHeader,- space: ProgramSpace,+ semantic: SemanticTrack,canvas: CanvasSpace,layers: MediaLayerSet,frame: MediaItemProgram["frame"],spec: MediaItemSpec,sounds: MediaSoundSet,): MediaTrackSet {+ const space = projectSemanticProgramSpace(semantic);assert(spec.expansion.kind === "one", `Program Media Item ${spec.id} must use one occurrence.`);return realizedItems(set, header, space, canvas, layers, frame, spec, sounds, [projectProgramWindow({itemId: spec.id,- space,+ semantic,projection: spec.projection,})]);}@@ -225,52 +227,52 @@ export function appendProgramMediaItem(export function appendSelectionMediaItem(set: MediaTrackSet,header: MediaTrackHeader,- space: ProgramSpace,+ semantic: SemanticTrack,canvas: CanvasSpace,layers: MediaLayerSet,frame: MediaItemProgram["frame"],- map: CompleteSemanticMap,selection: NarrativeSelectionRef,spec: MediaItemSpec,sounds: MediaSoundSet,): MediaTrackSet {+ const space = projectSemanticProgramSpace(semantic);return realizedItems(set, header, space, canvas, layers, frame, spec, sounds, projectSelectionWindows({- itemId: spec.id, map, selection, space, expansion: spec.expansion, projection: spec.projection,+ itemId: spec.id, semantic, selection, expansion: spec.expansion, projection: spec.projection,}));}export function appendSegmentMediaItem(set: MediaTrackSet,header: MediaTrackHeader,- space: ProgramSpace,+ semantic: SemanticTrack,canvas: CanvasSpace,layers: MediaLayerSet,frame: MediaItemProgram["frame"],- map: CompleteSemanticMap,segment: NarrativeExcerpt,spec: MediaItemSpec,sounds: MediaSoundSet,): MediaTrackSet {+ const space = projectSemanticProgramSpace(semantic);assert(spec.expansion.kind === "one", `Segment Media Item ${spec.id} must use one occurrence.`);return realizedItems(set, header, space, canvas, layers, frame, spec, sounds, [projectSegmentWindow({- itemId: spec.id, map, segment, space, projection: spec.projection,+ itemId: spec.id, semantic, segment, projection: spec.projection,})]);}export function appendMomentMediaItem(set: MediaTrackSet,header: MediaTrackHeader,- space: ProgramSpace,+ semantic: SemanticTrack,canvas: CanvasSpace,layers: MediaLayerSet,frame: MediaItemProgram["frame"],- map: CompleteSemanticMap,moment: NarrativeMomentRef,spec: MediaItemSpec,sounds: MediaSoundSet,): MediaTrackSet {+ const space = projectSemanticProgramSpace(semantic);return realizedItems(set, header, space, canvas, layers, frame, spec, sounds, projectMomentWindows({- itemId: spec.id, map, moment, space, expansion: spec.expansion, projection: spec.projection,+ itemId: spec.id, semantic, moment, expansion: spec.expansion, projection: spec.projection,}));}@@ -341,30 +343,31 @@ function exactlyOneTerminalFrame(values: readonly number[], label: string): numbexport function appendMediaSequenceUntilProgramEnd(set: MediaTrackSet,header: MediaTrackHeader,- space: ProgramSpace,+ semantic: SemanticTrack,canvas: CanvasSpace,members: MediaSequenceMemberSet,frame: MediaSequenceProgram["frame"],spec: MediaSequenceSpec,sounds: MediaSoundSet,): MediaTrackSet {+ const space = projectSemanticProgramSpace(semantic);return appendMediaSequence(set, header, space, canvas, members, frame, spec, sounds, programSpaceFrameCount(space));}export function appendMediaSequenceUntilMoment(set: MediaTrackSet,header: MediaTrackHeader,- space: ProgramSpace,+ semantic: SemanticTrack,canvas: CanvasSpace,members: MediaSequenceMemberSet,frame: MediaSequenceProgram["frame"],spec: MediaSequenceSpec,sounds: MediaSoundSet,- map: CompleteSemanticMap,moment: NarrativeMomentRef,): MediaTrackSet {+ const space = projectSemanticProgramSpace(semantic);const terminalFrame = exactlyOneTerminalFrame(- locateMomentOccurrences(map, moment, space).map((occurrence) => occurrence.cue.frame),+ locateMomentOccurrences(semantic, moment).map((occurrence) => occurrence.cue.frame),`Media Sequence ${spec.id} terminal Moment`,);return appendMediaSequence(set, header, space, canvas, members, frame, spec, sounds, terminalFrame);@@ -373,18 +376,18 @@ export function appendMediaSequenceUntilMoment(export function appendMediaSequenceUntilSelection(set: MediaTrackSet,header: MediaTrackHeader,- space: ProgramSpace,+ semantic: SemanticTrack,canvas: CanvasSpace,members: MediaSequenceMemberSet,frame: MediaSequenceProgram["frame"],spec: MediaSequenceSpec,sounds: MediaSoundSet,- map: CompleteSemanticMap,selection: NarrativeSelectionRef,boundary: "start" | "end",): MediaTrackSet {+ const space = projectSemanticProgramSpace(semantic);const terminalFrame = exactlyOneTerminalFrame(- locateSelectionOccurrences(map, selection, space).map((occurrence) =>+ locateSelectionOccurrences(semantic, selection).map((occurrence) =>boundary === "start" ? occurrence.start.frame : occurrence.end.frame),`Media Sequence ${spec.id} terminal Selection`,);diff --git a/packages/media-track/src/restricted-speech.ts b/packages/media-track/src/restricted-speech.tsindex 200a3a57..9383dc58 100644--- a/packages/media-track/src/restricted-speech.ts+++ b/packages/media-track/src/restricted-speech.ts@@ -21,7 +21,7 @@ export type RestrictedSpeechVisualClip = {};/**- * Focused reuse point for Speech Spine: one normalized muted take and one+ * Focused reuse point for Speech Track: one normalized muted take and one* explicitly placed foreground layer, without Media Track motion or sequence* semantics.*/diff --git a/packages/media-track/src/sequence.ts b/packages/media-track/src/sequence.tsindex f715d4a2..0b0f11fc 100644--- a/packages/media-track/src/sequence.ts+++ b/packages/media-track/src/sequence.ts@@ -10,7 +10,7 @@ import { canonicalize } from "@hypit/protocol";import { assertCanvasSpace, assertSpatialFrame } from "@hypit/spatial";import type { CanvasSpace } from "@hypit/spatial";import type { NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrative";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import type { SemanticTrack } from "@hypit/semantic-track";import {locateMomentOccurrences,locateSelectionOccurrences,@@ -121,12 +121,11 @@ export function appendMediaSequenceMomentMember(set: MediaSequenceMemberSet,layers: MediaLayerSet,spec: MediaSequenceMemberSpec,- map: CompleteSemanticMap,+ semantic: SemanticTrack,moment: NarrativeMomentRef,- space: ProgramSpace,): MediaSequenceMemberSet {const frame = exactlyOneFrame(- locateMomentOccurrences(map, moment, space).map((occurrence) => occurrence.cue.frame),+ locateMomentOccurrences(semantic, moment).map((occurrence) => occurrence.cue.frame),`Media Sequence member ${spec.id} Moment`,);return appendMediaSequenceMember(set, layers, spec, frame);@@ -136,13 +135,12 @@ export function appendMediaSequenceSelectionMember(set: MediaSequenceMemberSet,layers: MediaLayerSet,spec: MediaSequenceMemberSpec,- map: CompleteSemanticMap,+ semantic: SemanticTrack,selection: NarrativeSelectionRef,- space: ProgramSpace,boundary: "start" | "end",): MediaSequenceMemberSet {const frame = exactlyOneFrame(- locateSelectionOccurrences(map, selection, space).map((occurrence) =>+ locateSelectionOccurrences(semantic, selection).map((occurrence) =>boundary === "start" ? occurrence.start.frame : occurrence.end.frame),`Media Sequence member ${spec.id} Selection`,);diff --git a/packages/media-track/src/surface.ts b/packages/media-track/src/surface.tsindex 71797d58..e9f3166e 100644--- a/packages/media-track/src/surface.ts+++ b/packages/media-track/src/surface.ts@@ -16,8 +16,7 @@ import { sealGraphFragment } from "@hypit/elaborator";import type { FragmentOperation } from "@hypit/elaborator";import { mediaTypes } from "@hypit/media";import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import { svsRecipeType } from "@hypit/svs";import type { SvsRecipe } from "@hypit/svs";@@ -294,7 +293,7 @@ function createMediaTrackSurfaceFragment(inputTypes: readonly { readonly name: sspec = operation(bindId);}const common = {- set: operation(set), header: input("header"), space: input("space"), canvas: input("canvas"),+ set: operation(set), header: input("header"), semantic: input("semantic"), canvas: input("canvas"),layers: operation(layers), frame: input(item.frameName), spec, sounds: operation(sounds),};operations.push({@@ -302,9 +301,7 @@ function createMediaTrackSurfaceFragment(inputTypes: readonly { readonly name: sproducer: item.binding === "program" ? mediaTrackProducers.appendProgramItem: item.binding === "selection" ? mediaTrackProducers.appendSelectionItem: item.binding === "segment" ? mediaTrackProducers.appendSegmentItem : mediaTrackProducers.appendMomentItem,- inputs: item.binding === "program" ? common : {- ...common, map: input("map"), [item.binding]: input(item.sourceName!),- },+ inputs: item.binding === "program" ? common : { ...common, [item.binding]: input(item.sourceName!) },result: { kind: "output", name: "set" },});set = appendId;@@ -322,8 +319,8 @@ function createMediaTrackSurfaceFragment(inputTypes: readonly { readonly name: s: member.binding === "selection-start" ? mediaTrackProducers.appendSelectionStartMember: mediaTrackProducers.appendSelectionEndMember;operations.push({ id: appendId, producer, inputs: {- members: operation(members), layers: operation(layers), spec: input(member.specName), map: input("map"),- space: input("space"), [member.binding === "moment" ? "moment" : "selection"]: input(member.sourceName),+ members: operation(members), layers: operation(layers), spec: input(member.specName), semantic: input("semantic"),+ [member.binding === "moment" ? "moment" : "selection"]: input(member.sourceName),}, result: { kind: "output", name: "members" } });members = appendId;}@@ -338,7 +335,7 @@ function createMediaTrackSurfaceFragment(inputTypes: readonly { readonly name: sspec = operation(bindId);}const common = {- set: operation(set), header: input("header"), space: input("space"), canvas: input("canvas"),+ set: operation(set), header: input("header"), semantic: input("semantic"), canvas: input("canvas"),members: operation(members), frame: input(sequence.frameName), spec, sounds: operation(sounds),};const producer = sequence.terminal === "program-end" ? mediaTrackProducers.appendSequenceProgramEnd@@ -346,15 +343,15 @@ function createMediaTrackSurfaceFragment(inputTypes: readonly { readonly name: s: sequence.terminal === "selection-start" ? mediaTrackProducers.appendSequenceUntilSelectionStart: mediaTrackProducers.appendSequenceUntilSelectionEnd;operations.push({ id: appendId, producer, inputs: sequence.terminal === "program-end" ? common : {- ...common, map: input("map"), [sequence.terminal === "moment" ? "moment" : "selection"]: input(sequence.terminalSourceName!),+ ...common, [sequence.terminal === "moment" ? "moment" : "selection"]: input(sequence.terminalSourceName!),}, result: { kind: "output", name: "set" } });set = appendId;}operations.push(- { id: "track:finalize", producer: mediaTrackProducers.finalize, inputs: { set: operation(set), header: input("header"), space: input("space") }, result: { kind: "output", name: "program" } },- { id: "track:visual", producer: mediaTrackProducers.projectVisual, inputs: { space: input("space"), program: operation("track:finalize") }, result: { kind: "output", name: "track" } },+ { id: "track:finalize", producer: mediaTrackProducers.finalize, inputs: { set: operation(set), header: input("header"), semantic: input("semantic") }, result: { kind: "output", name: "program" } },+ { id: "track:visual", producer: mediaTrackProducers.projectVisual, inputs: { semantic: input("semantic"), program: operation("track:finalize") }, result: { kind: "output", name: "track" } },);- if (audio) operations.push({ id: "track:audio", producer: mediaTrackProducers.projectAudio, inputs: { space: input("space"), program: operation("track:finalize") }, result: { kind: "output", name: "track" } });+ if (audio) operations.push({ id: "track:audio", producer: mediaTrackProducers.projectAudio, inputs: { semantic: input("semantic"), program: operation("track:finalize") }, result: { kind: "output", name: "track" } });return sealGraphFragment({inputs: inputTypes,operations,@@ -627,10 +624,10 @@ function sound(}export const decodeMediaTrackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- allowed(element, ["id", "space", "canvas", "map"]);+ allowed(element, ["id", "semantic", "canvas"]);const trackId = text(element, "id");const state = builder();- state.addReference("space", reference(element.attributes.space, `${element.name}.space`, programSpaceTypes.programSpace, resolveReference));+ state.addReference("semantic", reference(element.attributes.semantic, `${element.name}.semantic`, semanticTrackTypes.track, resolveReference));state.addReference("canvas", reference(element.attributes.canvas, `${element.name}.canvas`, spatialTypes.canvas, resolveReference));const headerId = `${trackId}.header`;state.addRecord("header", headerId, mediaTrackTypes.header,@@ -639,7 +636,6 @@ export const decodeMediaTrackSurface: StructuredSurfaceHandler = ({ element, resconst sequences: FragmentSequence[] = [];let itemIndex = 0;let sequenceIndex = 0;- let usesMap = false;let hasAudio = false;for (const child of element.children) {if (child.kind === "text") {@@ -663,7 +659,6 @@ export const decodeMediaTrackSurface: StructuredSurfaceHandler = ({ element, resconst motionRecipe = child.attributes.motion === undefined ? undefined: recipe(reference(child.attributes.motion, `${child.name}.motion`, svsRecipeType, resolveReference), `${child.name}.motion`);const binding = temporalBinding(child, resolveReference);- if (binding.kind !== "program") usesMap = true;const occurrences = text(child, "occurrences", "one");if (occurrences !== "one" && occurrences !== "each") throw new Error(`${child.name}.occurrences must be one or each.`);const expansion: OccurrenceExpansion = { kind: occurrences };@@ -727,7 +722,6 @@ export const decodeMediaTrackSurface: StructuredSurfaceHandler = ({ element, resconst memberSuffix = suffix(index + 1);allowed(member, ["id", "image", "media", "surface", "extent", "appearance", "at", "boundary", "source-audio", "audio-gain"]);const at = oneOfReferences(member.attributes.at, `${member.name}.at`, [narrativeTypes.moment, narrativeTypes.selection], resolveReference);- usesMap = true;const boundary = optionalText(member, "boundary");const binding = sameType(at.type, narrativeTypes.moment) ? "moment": boundary === "start" ? "selection-start" : boundary === "end" ? "selection-end" : undefined;@@ -769,7 +763,6 @@ export const decodeMediaTrackSurface: StructuredSurfaceHandler = ({ element, resif (child.attributes["until-boundary"] !== undefined) throw new Error(`${child.name}.until-boundary is invalid for program.end.`);} else {terminalReference = oneOfReferences(until, `${child.name}.until`, [narrativeTypes.moment, narrativeTypes.selection], resolveReference);- usesMap = true;if (sameType(terminalReference.type, narrativeTypes.moment)) {if (child.attributes["until-boundary"] !== undefined) throw new Error(`${child.name}.until-boundary is invalid for a Moment.`);terminal = "moment";@@ -790,11 +783,6 @@ export const decodeMediaTrackSurface: StructuredSurfaceHandler = ({ element, res...(clipPathName === undefined ? {} : { clipPathName }), members, sounds: sequenceSounds });}if (items.length === 0 && sequences.length === 0) throw new Error(`${element.name} requires at least one Item or Sequence.`);- if (usesMap) {- state.addReference("map", reference(element.attributes.map, `${element.name}.map`, semanticMapTypes.complete, resolveReference));- } else if (element.attributes.map !== undefined) {- throw new Error(`${element.name}.map is unused because no child consumes semantic timing.`);- }const fragment = createMediaTrackSurfaceFragment(state.inputTypes, items, sequences, hasAudio);return {records: state.records,diff --git a/packages/media-track/test/media-track.test.ts b/packages/media-track/test/media-track.test.tsindex d093e3ac..ecc3e25e 100644--- a/packages/media-track/test/media-track.test.ts+++ b/packages/media-track/test/media-track.test.ts@@ -1,6 +1,7 @@import assert from "node:assert/strict";import test from "node:test";import { fixtureDigest } from "../../../test/fixture-digest.js";+import { semanticTrackFixture } from "../../../test/semantic-track-fixture.js";import { sealComposition } from "@hypit/composition";import type { AudioTrack } from "@hypit/composition";@@ -56,8 +57,7 @@ import type { NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrativeimport { sealProgramSpace } from "@hypit/program-space";import { programSpaceTypes } from "@hypit/program-space";import type { BlobRef } from "@hypit/protocol";-import { semanticMapTypes } from "@hypit/semantic-map";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import { svsRecipeType } from "@hypit/svs";import type { SvsRecipe } from "@hypit/svs";@@ -76,6 +76,17 @@ const canvas = {pixelAspect: "square" as const,};const header = sealMediaTrackHeader({ id: "proof" });+const semantic = semanticTrackFixture(space, {+ segments: [+ { id: "opening", frameCount: 30 },+ { id: "answer", frameCount: 60 },+ { id: "ending", frameCount: 30 },+ ],+ anchors: [+ { identity: "a", frame: 15 }, { identity: "b", frame: 45 },+ { identity: "c", frame: 60 }, { identity: "d", frame: 105 },+ ],+});const source: BlobRef = {kind: "blob",digest: fixtureDigest("media-track:still"),@@ -216,7 +227,7 @@ function sequenceSounds() {test("an Item keeps ordered Paint/sample layers, two-frame fit and frame presentation package-owned", () => {const program = finalizeMediaTrack(appendProgramMediaItem(- createMediaTrackSet(), header, space, canvas, stillLayers(), frame, itemSpec(), createMediaSoundSet(),+ createMediaTrackSet(), header, semantic, canvas, stillLayers(), frame, itemSpec(), createMediaSoundSet(),), header, space);assert.deepEqual(program.items[0]?.span, { startFrame: 0, endFrameExclusive: 120 });assert.deepEqual(program.items[0]?.layers.map((layer) => layer.id), ["backing", "content"]);@@ -250,7 +261,7 @@ test("an owned clip path is an explicit Media input and lowers only over the Ite],});const track = projectMediaVisualTrack(space, finalizeMediaTrack(appendProgramMediaItem(- createMediaTrackSet(), header, space, canvas, stillLayers(), frame, clipped, createMediaSoundSet(),+ createMediaTrackSet(), header, semantic, canvas, stillLayers(), frame, clipped, createMediaSoundSet(),), header, space));const frameElement = track.presents[0]!.elements.find((element) => element.id.endsWith(":frame"));assert.equal(frameElement?.style.find((entry) => entry.name === "clip-path")?.value,@@ -268,7 +279,7 @@ test("self-blur is two explicit samples of one Artifact and Artifact collectionid: "foreground", appearance,}));const track = projectMediaVisualTrack(space, finalizeMediaTrack(appendProgramMediaItem(- createMediaTrackSet(), header, space, canvas, layers, frame, itemSpec(), createMediaSoundSet(),+ createMediaTrackSet(), header, semantic, canvas, layers, frame, itemSpec(), createMediaSoundSet(),), header, space));const document = compileHyperframesDocument(sealComposition({id: "media",@@ -298,7 +309,7 @@ test("timed layer trim/hold, lifecycle and sampling motion remain separate wrapp{ atProgress: 1, zoom: 1.2, offsetX: 20, offsetY: -10, rotationDeg: 2, easing: "ease-out" },] },}));- const program = finalizeMediaTrack(appendProgramMediaItem(createMediaTrackSet(), header, space, canvas, layers, frame, itemSpec({+ const program = finalizeMediaTrack(appendProgramMediaItem(createMediaTrackSet(), header, semantic, canvas, layers, frame, itemSpec({motion: {enter: { operator: "slide", durationFrames: 12, easing: "ease-out", direction: "down", amount: 120 },sustain: [{ operator: "wobble", amount: 2, cycles: 2 }],@@ -344,7 +355,7 @@ test("source audio and edge SFX project separately from the visual Track", () =>sounds = appendMediaSound(sounds, timed("sfx-exit"), sealMediaSoundSpec({id: "exit-sound", trigger: { kind: "exit" }, gain: 0.25,}));- const program = finalizeMediaTrack(appendProgramMediaItem(createMediaTrackSet(), header, space, canvas, layers, frame, itemSpec({+ const program = finalizeMediaTrack(appendProgramMediaItem(createMediaTrackSet(), header, semantic, canvas, layers, frame, itemSpec({sourceAudio: { fromLayer: "video", gain: 0.75 },motion: { sustain: [], exit: { operator: "fade", durationFrames: 10, easing: "ease-in" } },}), sounds), header, space);@@ -363,7 +374,7 @@ test("source audio and edge SFX project separately from the visual Track", () =>});assert.equal(sourceClip?.gain, 0.75);assert.throws(() => projectMediaAudioTrack(space, finalizeMediaTrack(appendProgramMediaItem(- createMediaTrackSet(), header, space, canvas, stillLayers(), frame, itemSpec(), createMediaSoundSet(),+ createMediaTrackSet(), header, semantic, canvas, stillLayers(), frame, itemSpec(), createMediaSoundSet(),), header, space)), /no explicitly authored audio/u);const bgm: AudioTrack = {@@ -417,7 +428,7 @@ test("still and animated typed Surfaces use the same layer law without browser fid: "animated", occupancy: { mode: "loop", align: "start" }, appearance,}));const elements = projectMediaVisualTrack(space, finalizeMediaTrack(appendProgramMediaItem(- createMediaTrackSet(), header, space, canvas, layers, frame, itemSpec(), createMediaSoundSet(),+ createMediaTrackSet(), header, semantic, canvas, layers, frame, itemSpec(), createMediaSoundSet(),), header, space)).presents[0]!.elements;assert.equal(elements.find((element) => element.id === "still")?.kind, "surface");const moving = elements.find((element) => element.id === "animated");@@ -431,15 +442,6 @@ test("still and animated typed Surfaces use the same layer law without browser f});test("Media Items consume the shared one/each temporal algebra without becoming an exclusive lane", () => {- const semanticMap: CompleteSemanticMap = {- tokens: [],- anchors: [- { identity: "a", frame: 15 },- { identity: "b", frame: 45 },- { identity: "c", frame: 60 },- { identity: "d", frame: 105 },- ],- };const selection: NarrativeSelectionRef = {id: "mentions",@@ -448,8 +450,8 @@ test("Media Items consume the shared one/each temporal algebra without becoming{ occurrence: 1, startAnchorId: "c", endAnchorId: "d" },],};- const repeated = appendSelectionMediaItem(createMediaTrackSet(), header, space, canvas, stillLayers(), frame,- semanticMap, selection, itemSpec({+ const repeated = appendSelectionMediaItem(createMediaTrackSet(), header, semantic, canvas, stillLayers(), frame,+ selection, itemSpec({id: "mention",expansion: { kind: "each" },projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } },@@ -461,8 +463,8 @@ test("Media Items consume the shared one/each temporal algebra without becomingconst moment: NarrativeMomentRef = {id: "cue", occurrences: [{ occurrence: 0, anchorId: "b" }],};- const overlapping = appendMomentMediaItem(repeated, header, space, canvas, stillLayers(), frame,- semanticMap, moment, itemSpec({+ const overlapping = appendMomentMediaItem(repeated, header, semantic, canvas, stillLayers(), frame,+ moment, itemSpec({id: "popup",projection: { start: { ref: "moment.cue", offset: { unit: "frames", value: -10 } }, end: { ref: "moment.cue", offset: { unit: "frames", value: 20 } } },stackingOrder: 41,@@ -480,8 +482,8 @@ test("Media Items consume the shared one/each temporal algebra without becomingoccurrences: [selection.occurrences[0]!],};const selectedOnce = appendSelectionMediaItem(- createMediaTrackSet(), header, space, canvas, stillLayers(), frame,- semanticMap, singleSelection, itemSpec({+ createMediaTrackSet(), header, semantic, canvas, stillLayers(), frame,+ singleSelection, itemSpec({id: "selected-once",expansion: { kind: "one" },projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } },@@ -501,8 +503,8 @@ test("Media Items consume the shared one/each temporal algebra without becoming],};const moments = appendMomentMediaItem(- createMediaTrackSet(), header, space, canvas, stillLayers(), frame,- semanticMap, repeatedMoment, itemSpec({+ createMediaTrackSet(), header, semantic, canvas, stillLayers(), frame,+ repeatedMoment, itemSpec({id: "moment-each",expansion: { kind: "each" },projection: {@@ -517,7 +519,7 @@ test("Media Items consume the shared one/each temporal algebra without becoming]);const absolute = appendProgramMediaItem(- createMediaTrackSet(), header, space, canvas, stillLayers(), frame,+ createMediaTrackSet(), header, semantic, canvas, stillLayers(), frame,itemSpec({id: "absolute",projection: {@@ -530,16 +532,8 @@ test("Media Items consume the shared one/each temporal algebra without becoming});test("a Media Item can consume one whole Narrative Segment without a synthetic Selection", () => {- const semanticMap: CompleteSemanticMap = {- tokens: [],- anchors: [- { identity: "segment:answer:start", frame: 30 },- { identity: "segment:answer:end", frame: 90 },- ],- };const result = appendSegmentMediaItem(- createMediaTrackSet(), header, space, canvas, stillLayers(), frame,- semanticMap,+ createMediaTrackSet(), header, semantic, canvas, stillLayers(), frame,{ kind: "segment", id: "answer", tokenStart: 0, tokenEndExclusive: 1 },itemSpec({id: "whole-answer",@@ -587,7 +581,7 @@ test("every documented Media frame and fit remains one ordinary Item instead of{ widthPx: index % 3 === 0 ? 400 : index % 3 === 1 ? 1200 : 800,heightPx: index % 3 === 0 ? 1200 : index % 3 === 1 ? 400 : 800 },localFit, sealMediaSampleLayerSpec({ id: `sample-${index}`, appearance }));- set = appendProgramMediaItem(set, header, space, canvas, layers, frames[index % frames.length]!, itemSpec({+ set = appendProgramMediaItem(set, header, semantic, canvas, layers, frames[index % frames.length]!, itemSpec({id: `fit-${sizing}`, stackingOrder: 100 + index,presentation: { clip: { kind: "none" }, padding: { topPx: 0, rightPx: 0, bottomPx: 0, leftPx: 0 }, shadows: [] },}), createMediaSoundSet());@@ -618,7 +612,7 @@ test("transparent, Paint, self-blur and alternate-source backing are only orderelayers = appendStillMediaLayer(layers, source, extent, fit,sealMediaSampleLayerSpec({ id: "foreground", appearance }));const track = projectMediaVisualTrack(space, finalizeMediaTrack(appendProgramMediaItem(- createMediaTrackSet(), header, space, canvas, layers, frame, itemSpec(), createMediaSoundSet(),+ createMediaTrackSet(), header, semantic, canvas, layers, frame, itemSpec(), createMediaSoundSet(),), header, space));const document = compileHyperframesDocument(sealComposition({id: "layer-matrix", canvas: { width: 1080, height: 1920, clearColor: "#000000" }, tracks: [track],@@ -628,7 +622,7 @@ test("transparent, Paint, self-blur and alternate-source backing are only ordereassert.equal(document.artifacts.length, 2, "repeated samples share bytes while the alternate source remains explicit");const transparent = projectMediaVisualTrack(space, finalizeMediaTrack(appendProgramMediaItem(- createMediaTrackSet(), header, space, canvas,+ createMediaTrackSet(), header, semantic, canvas,appendStillMediaLayer(createMediaLayerSet(), source, extent, fit,sealMediaSampleLayerSpec({ id: "only", appearance })),frame, itemSpec(), createMediaSoundSet(),@@ -640,12 +634,12 @@ test("transparent, Paint, self-blur and alternate-source backing are only orderetest("visual-only normalized video never creates an implicit audio branch", () => {const silent = timedLayers("silent", false);const program = finalizeMediaTrack(appendProgramMediaItem(- createMediaTrackSet(), header, space, canvas, silent, frame, itemSpec(), createMediaSoundSet(),+ createMediaTrackSet(), header, semantic, canvas, silent, frame, itemSpec(), createMediaSoundSet(),), header, space);assert.equal(projectMediaVisualTrack(space, program).presents[0]!.elements.some((element) => element.kind === "video"), true);assert.throws(() => projectMediaAudioTrack(space, program), /no explicitly authored audio projection/u);assert.throws(() => appendProgramMediaItem(- createMediaTrackSet(), header, space, canvas, silent, frame,+ createMediaTrackSet(), header, semantic, canvas, silent, frame,itemSpec({ sourceAudio: { fromLayer: "video", gain: 1 } }), createMediaSoundSet(),), /has no normalized audio/u);});@@ -665,7 +659,7 @@ test("timed visual occupancy resolves every alignment into exact source-frame setest("the graph keeps every source, extent, fit, frame, time and appearance input explicit", () => {assert.deepEqual(stillMediaTrackFragment.inputs.map((input) => input.name), [- "canvas", "extent", "fit", "frame", "header", "item-spec", "sample-spec", "source", "space",+ "canvas", "extent", "fit", "frame", "header", "item-spec", "sample-spec", "semantic", "source",]);});@@ -685,9 +679,8 @@ test("the Media author Surface emits explicit graph edges for layers, semantic tpath, ref: { kind: "record", id: path }, type,});const references = new Map<string, SurfaceResolvedReference>([- ["space", plain("space", programSpaceTypes.programSpace)],["canvas", plain("canvas", spatialTypes.canvas)],- ["map", plain("map", semanticMapTypes.complete)],+ ["semantic", plain("semantic", semanticTrackTypes.track)],["frame", plain("frame", spatialTypes.frame)],["clip-path", plain("clip-path", spatialTypes.path)],["still", plain("still", artifactTypes.blob)],@@ -709,7 +702,7 @@ test("the Media author Surface emits explicit graph edges for layers, semantic t["motion", appearance("motion", { enter: "slide", "enter-frames": 12, "enter-easing": "ease-out", "enter-direction": "up", "enter-origin": "outside-canvas" })],["handoff", appearance("handoff", { operator: "crossfade", "duration-frames": 10, "boundary-ratio": 0.5, audio: "cut" })],]);- const root = node("media:Track", { id: "editorial", space: ref("space"), canvas: ref("canvas"), map: ref("map") }, [+ const root = node("media:Track", { id: "editorial", semantic: ref("semantic"), canvas: ref("canvas") }, [node("media:Item", { id: "still-card", image: ref("still"), extent: ref("extent"), frame: ref("frame"), clip: ref("clip-path"), appearance: ref("still-style"), during: "program" }),node("media:Item", { id: "segment-card", image: ref("still"), extent: ref("extent"), frame: ref("frame"), appearance: ref("still-style"), during: ref("answer-segment") }),node("media:Item", { id: "proof", frame: ref("frame"), appearance: ref("card-style"), motion: ref("motion"), during: ref("selection"), "source-audio": "video", "audio-gain": "0.8" }, [diff --git a/packages/program-space/README.md b/packages/program-space/README.mdindex 3480efe8..4fd6d43a 100644--- a/packages/program-space/README.md+++ b/packages/program-space/README.md@@ -3,5 +3,5 @@The provider-neutral time and frame domain shared by a compiled program.`Clock` is the authored, duration-free frame-rate fact for a video project. It is declared once and-consumed by media normalization and Speech Spine; `ProgramSpace` is the later duration-bearing+consumed by media normalization and Speech Track; `ProgramSpace` is the later duration-bearingdomain assembled from the semantic Takes.diff --git a/packages/ranking/README.md b/packages/ranking/README.mdindex 4376ea1d..c3e9da2b 100644--- a/packages/ranking/README.md+++ b/packages/ranking/README.md@@ -7,7 +7,7 @@ Three independent progressive-ranking author components over one private schedul- `TopThree`The package owns variant-specific Programs and Styles. It consumes explicit Segment/Selection/Moment,-SemanticMap, ProgramSpace, SpatialFrame, font, image and optional sound edges, then lowers to peer+SemanticTrack, SpatialFrame, font, image and optional sound edges, then lowers to peer`VisualTrack` and optional `AudioTrack` values. It adds no Ranking field to Core, Composition orVisual IR.@@ -26,7 +26,7 @@ copy.<copy:Value id="winner">No hidden runtime choice</copy:Value>-<ranking:Column id="priorities" map={speech.semanticMap} space={speech.space} canvas={vertical}+<ranking:Column id="priorities" semantic={speech.semantic} canvas={vertical}frame={layout.ranking} during={story.segment.ranking}style={ranking-style}><ranking:ColumnItem rank="1" label={winner}diff --git a/packages/ranking/package.json b/packages/ranking/package.jsonindex dcd532b8..771e74f1 100644--- a/packages/ranking/package.json+++ b/packages/ranking/package.json@@ -19,7 +19,7 @@"@hypit/narrative": "workspace:*","@hypit/program-space": "workspace:*","@hypit/protocol": "workspace:*",- "@hypit/semantic-map": "workspace:*",+ "@hypit/semantic-track": "workspace:*","@hypit/spatial": "workspace:*","@hypit/svs": "workspace:*","@hypit/temporal": "workspace:*",diff --git a/packages/ranking/src/component.ts b/packages/ranking/src/component.tsindex 61321313..4f92e6ce 100644--- a/packages/ranking/src/component.ts+++ b/packages/ranking/src/component.ts@@ -4,7 +4,8 @@ import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } fromimport type { ProgramSpace } from "@hypit/program-space";import { canonicalize } from "@hypit/protocol";import type { BlobRef, StoredValue } from "@hypit/protocol";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import type { SemanticTrack } from "@hypit/semantic-track";+import { projectSemanticProgramSpace } from "@hypit/semantic-track";import type { CanvasSpace, SpatialFrame } from "@hypit/spatial";import type { Text } from "@hypit/text";@@ -84,7 +85,7 @@ export const rankingComponent = {producer: rankingProducers.schedule,handler: ({ inputs }) => ({ outputs: { schedule: output(buildRankingSchedule({header: inline(inputs.header?.value, "RankingHeader"), items: inline(inputs.items?.value, "RankingItemSpecSet"),- map: inline(inputs.map?.value, "CompleteSemanticMap"), space: inline(inputs.space?.value, "ProgramSpace"),+ semantic: inline(inputs.semantic?.value, "SemanticTrack"),outer: inline(inputs.outer?.value, "NarrativeSelectionRef"), triggers: inline(inputs.triggers?.value, "NarrativeMomentRef"),terminal: inline(inputs.terminal?.value, "NarrativeMomentRef"),})) }, needs: {} }),@@ -92,16 +93,14 @@ export const rankingComponent = {{producer: rankingProducers.projectColumnSelectionOuter,handler: ({ inputs }) => ({ outputs: { outer: output(projectColumnSelectionOuterWindow(- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"),inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"),)) }, needs: {} }),},{producer: rankingProducers.projectColumnSegmentOuter,handler: ({ inputs }) => ({ outputs: { outer: output(projectColumnSegmentOuterWindow(- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"),inline<NarrativeExcerpt>(inputs.segment?.value, "NarrativeExcerpt"),)) }, needs: {} }),},@@ -114,8 +113,7 @@ export const rankingComponent = {handler: ({ inputs }) => ({ outputs: { set: output(appendColumnWindowCandidate(inline<ColumnWindowCandidateSet>(inputs.set?.value, "ColumnWindowCandidateSet"),inline<ColumnItemSpec>(inputs.spec?.value, "ColumnItemSpec"),- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"),inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"),)) }, needs: {} }),},@@ -220,7 +218,7 @@ export const rankingComponent = {{producer: rankingProducers.renderAudio,handler: ({ inputs }) => ({ outputs: { track: output(renderRankingAudio(- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")),inline<RankingSoundEventPlan>(inputs.events?.value, "RankingSoundEventPlan"),inline<RankingSoundStyle>(inputs.style?.value, "RankingSoundStyle"),inline<RankingSoundSet>(inputs.sounds?.value, "RankingSoundSet"),@@ -233,7 +231,7 @@ export const rankingComponent = {] as const).map(([producer, render, label]) => ({producer,handler: ({ inputs }: ProducerHandlerContext) => ({ outputs: { track: output(render(- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline(inputs.program?.value, label) as never,+ projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")), inline(inputs.program?.value, label) as never,)) }, needs: {} }),})),],diff --git a/packages/ranking/src/fragment.ts b/packages/ranking/src/fragment.tsindex df153c1c..b938443e 100644--- a/packages/ranking/src/fragment.ts+++ b/packages/ranking/src/fragment.ts@@ -3,8 +3,7 @@ import { sealGraphFragment } from "@hypit/elaborator";import type { FragmentOperation, GraphFragment } from "@hypit/elaborator";import { mediaTypes } from "@hypit/media";import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import { textTypes } from "@hypit/text";@@ -58,8 +57,7 @@ export function createRankingFragment(const selected = definition(variant);const inputs: Array<GraphFragment["inputs"][number]> = [{ name: "header", type: rankingTypes.header },- { name: "map", type: semanticMapTypes.complete },- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "outer", type: outerKind === "selection" ? narrativeTypes.selection : narrativeTypes.excerpt },...(variant === "column" ? [] : [{ name: "triggers", type: narrativeTypes.moment },@@ -116,7 +114,7 @@ export function createRankingFragment(id: timingId,producer: rankingProducers.appendColumnCandidate,inputs: {- set: candidates, spec: resolvedSpec, map: input("map"), space: input("space"), selection: input(item.timingName),+ set: candidates, spec: resolvedSpec, semantic: input("semantic"), selection: input(item.timingName),},result: { kind: "output", name: "set" },});@@ -127,7 +125,7 @@ export function createRankingFragment(operations.push({id: "outer-window",producer: outerKind === "selection" ? rankingProducers.projectColumnSelectionOuter : rankingProducers.projectColumnSegmentOuter,- inputs: { map: input("map"), space: input("space"), [outerKind]: input("outer") },+ inputs: { semantic: input("semantic"), [outerKind]: input("outer") },result: { kind: "output", name: "outer" },});operations.push({@@ -140,7 +138,7 @@ export function createRankingFragment(id: "schedule",producer: rankingProducers.schedule,inputs: {- header: input("header"), items: specs, map: input("map"), space: input("space"),+ header: input("header"), items: specs, semantic: input("semantic"),outer: input("outer"), triggers: input("triggers"), terminal: input("terminal"),},result: { kind: "output", name: "schedule" },@@ -158,7 +156,7 @@ export function createRankingFragment(operations.push({id: "visual",producer: selected.render,- inputs: { space: input("space"), program: operation("program") },+ inputs: { semantic: input("semantic"), program: operation("program") },result: { kind: "output", name: "track" },});const hasAudio = sound.appearName !== undefined || sound.moveName !== undefined;@@ -187,7 +185,7 @@ export function createRankingFragment(}operations.push({id: "audio", producer: rankingProducers.renderAudio,- inputs: { space: input("space"), events: operation("events"), style: input("sound-style"), sounds },+ inputs: { semantic: input("semantic"), events: operation("events"), style: input("sound-style"), sounds },result: { kind: "output", name: "track" },});}diff --git a/packages/ranking/src/manifest.ts b/packages/ranking/src/manifest.tsindex dbd22498..5c78e35b 100644--- a/packages/ranking/src/manifest.ts+++ b/packages/ranking/src/manifest.ts@@ -4,9 +4,8 @@ import { artifactDependency } from "@hypit/artifact";import { compositionDependency, compositionTypes } from "@hypit/composition";import { mediaDependency, mediaTypes } from "@hypit/media";import { narrativeDependency, narrativeTypes } from "@hypit/narrative";-import { programSpaceDependency, programSpaceTypes } from "@hypit/program-space";import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@hypit/protocol";-import { semanticMapDependency, semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackDependency, semanticTrackTypes } from "@hypit/semantic-track";import { spatialDependency, spatialFrameSchema, spatialTypes } from "@hypit/spatial";import { svsRecipeType } from "@hypit/svs";import { temporalDependency } from "@hypit/temporal";@@ -427,10 +426,8 @@ export const rankingMarkupSurfaces = [attributes: [{ name: "id", kind: "identifier", required: true,summary: "Names this board so its Schedule, Program and Tracks can be referenced elsewhere in the Source." },- { name: "map", kind: "reference", required: true, accepts: [semanticMapTypes.complete],- summary: "Chooses the measured SemanticMap that gives every trigger its frame." },- { name: "space", kind: "reference", required: true, accepts: [programSpaceTypes.programSpace],- summary: "Chooses the ProgramSpace the board is timed and rendered against." },+ { name: "semantic", kind: "reference", required: true, accepts: [semanticTrackTypes.track],+ summary: "Chooses the SemanticTrack that owns the frame domain and gives every trigger its frame." },{ name: "frame", kind: "reference", required: true, accepts: [spatialTypes.frame],summary: "Chooses the Frame the whole board occupies." },{ name: "during", kind: "reference", required: true, accepts: [narrativeTypes.selection],@@ -473,7 +470,7 @@ export const rankingMarkupSurfaces = [summary: "The rendered board sound, published only when a sound is authored." },],example: `<ranking:TierBoardStyle id="tier-style" recipe={studio.ranking.tier} font={ui-font}/>-<ranking:TierBoard id="tiers" map={timing.map} space={speech.space} frame={board-frame}+<ranking:TierBoard id="tiers" semantic={speech.semantic} frame={board-frame}during={story.selection.board} triggers={story.moment.place} terminal={story.moment.done}style={tier-style}><ranking:TierItem id="row-regen" tier="s" icon={icon-regen}/>@@ -494,10 +491,8 @@ export const rankingMarkupSurfaces = [attributes: [{ name: "id", kind: "identifier", required: true,summary: "Names this board so its Schedule, Program and Tracks can be referenced elsewhere in the Source." },- { name: "map", kind: "reference", required: true, accepts: [semanticMapTypes.complete],- summary: "Chooses the measured SemanticMap that gives every trigger its frame." },- { name: "space", kind: "reference", required: true, accepts: [programSpaceTypes.programSpace],- summary: "Chooses the ProgramSpace the board is timed and rendered against." },+ { name: "semantic", kind: "reference", required: true, accepts: [semanticTrackTypes.track],+ summary: "Chooses the SemanticTrack that owns the frame domain and resolves every reveal Selection." },{ name: "canvas", kind: "reference", required: true, accepts: [spatialTypes.canvas],summary: "Chooses the Canvas coordinate space used by the independent reveal stage." },{ name: "frame", kind: "reference", required: true, accepts: [spatialTypes.frame],@@ -542,7 +537,7 @@ export const rankingMarkupSurfaces = [summary: "The rendered board sound, published only when a sound is authored." },],example: `<ranking:ColumnStyle id="board-style" recipe={studio.ranking.board} font={ui-font}/>-<ranking:Column id="board" map={timing.map} space={speech.space} canvas={vertical} frame={board-frame}+<ranking:Column id="board" semantic={speech.semantic} canvas={vertical} frame={board-frame}during={story.segment.ranking}style={board-style}><ranking:ColumnItem id="row-regen" rank="1" label="ReGen" icon={icon-regen} during={story.selection.regen}/>@@ -566,10 +561,8 @@ export const rankingMarkupSurfaces = [attributes: [{ name: "id", kind: "identifier", required: true,summary: "Names this board so its Schedule, Program and Tracks can be referenced elsewhere in the Source." },- { name: "map", kind: "reference", required: true, accepts: [semanticMapTypes.complete],- summary: "Chooses the measured SemanticMap that gives every trigger its frame." },- { name: "space", kind: "reference", required: true, accepts: [programSpaceTypes.programSpace],- summary: "Chooses the ProgramSpace the board is timed and rendered against." },+ { name: "semantic", kind: "reference", required: true, accepts: [semanticTrackTypes.track],+ summary: "Chooses the SemanticTrack that owns the frame domain and gives every trigger its frame." },{ name: "frame", kind: "reference", required: true, accepts: [spatialTypes.frame],summary: "Chooses the Frame the whole board occupies." },{ name: "during", kind: "reference", required: true, accepts: [narrativeTypes.selection],@@ -608,7 +601,7 @@ export const rankingMarkupSurfaces = [summary: "The rendered board sound, published only when a sound is authored." },],example: `<ranking:TopThreeStyle id="podium-style" recipe={studio.ranking.podium} font={ui-font}/>-<ranking:TopThree id="podium" map={timing.map} space={speech.space} frame={board-frame}+<ranking:TopThree id="podium" semantic={speech.semantic} frame={board-frame}during={story.selection.board} triggers={story.moment.place} terminal={story.moment.done}style={podium-style}><ranking:TopThreeItem id="slot-gold" label="ReGen" icon={icon-regen}/>@@ -628,7 +621,7 @@ export const rankingMarkupSurfaces = [export const rankingManifest: ModuleManifest = {format: "hypit.module@1", name: rankingModuleRef.name, version: rankingModuleRef.version,- dependencies: [artifactDependency, mediaDependency, narrativeDependency, semanticMapDependency, programSpaceDependency, spatialDependency, temporalDependency, compositionDependency, textDependency],+ dependencies: [artifactDependency, mediaDependency, narrativeDependency, semanticTrackDependency, spatialDependency, temporalDependency, compositionDependency, textDependency],types: [{ name: rankingTypes.header.name },{ name: rankingTypes.itemSpec.name },@@ -657,15 +650,15 @@ export const rankingManifest: ModuleManifest = {{ name: rankingProducers.appendSpec.name, inputs: [{ name: "set", type: rankingTypes.itemSpecs }, { name: "spec", type: rankingTypes.itemSpec }], outputs: [{ name: "set", type: rankingTypes.itemSpecs }], needs: [] },{ name: rankingProducers.schedule.name, inputs: [{ name: "header", type: rankingTypes.header }, { name: "items", type: rankingTypes.itemSpecs },- { name: "map", type: semanticMapTypes.complete }, { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "outer", type: narrativeTypes.selection }, { name: "triggers", type: narrativeTypes.moment }, { name: "terminal", type: narrativeTypes.moment },], outputs: [{ name: "schedule", type: rankingTypes.schedule }], needs: [] },{ name: rankingProducers.projectColumnSelectionOuter.name, inputs: [- { name: "map", type: semanticMapTypes.complete }, { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "selection", type: narrativeTypes.selection },], outputs: [{ name: "outer", type: rankingTypes.columnOuter }], needs: [] },{ name: rankingProducers.projectColumnSegmentOuter.name, inputs: [- { name: "map", type: semanticMapTypes.complete }, { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "segment", type: narrativeTypes.excerpt },], outputs: [{ name: "outer", type: rankingTypes.columnOuter }], needs: [] },{ name: rankingProducers.createColumnCandidates.name, inputs: [], outputs: [@@ -673,7 +666,7 @@ export const rankingManifest: ModuleManifest = {], needs: [] },{ name: rankingProducers.appendColumnCandidate.name, inputs: [{ name: "set", type: rankingTypes.columnCandidates }, { name: "spec", type: rankingTypes.itemSpec },- { name: "map", type: semanticMapTypes.complete }, { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "selection", type: narrativeTypes.selection },], outputs: [{ name: "set", type: rankingTypes.columnCandidates }], needs: [] },{ name: rankingProducers.columnSchedule.name, inputs: [@@ -710,7 +703,7 @@ export const rankingManifest: ModuleManifest = {{ name: eventProducer.name, inputs: [{ name: "schedule", type: rankingTypes.schedule }, { name: "style", type: styleType }, { name: "specs", type: rankingTypes.itemSpecs },], outputs: [{ name: "events", type: rankingTypes.soundEvents }], needs: [] },- { name: renderProducer.name, inputs: [{ name: "space", type: programSpaceTypes.programSpace }, { name: "program", type: programType }], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [] },+ { name: renderProducer.name, inputs: [{ name: "semantic", type: semanticTrackTypes.track }, { name: "program", type: programType }], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [] },]),{ name: rankingProducers.createSounds.name, inputs: [], outputs: [{ name: "sounds", type: rankingTypes.sounds }], needs: [] },...([@@ -720,7 +713,7 @@ export const rankingManifest: ModuleManifest = {name: producer.name, inputs: [{ name: "sounds", type: rankingTypes.sounds }, { name: "media", type: mediaTypes.synchronized }], outputs: [{ name: "sounds", type: rankingTypes.sounds }], needs: [],})),{ name: rankingProducers.renderAudio.name, inputs: [- { name: "space", type: programSpaceTypes.programSpace }, { name: "events", type: rankingTypes.soundEvents },+ { name: "semantic", type: semanticTrackTypes.track }, { name: "events", type: rankingTypes.soundEvents },{ name: "style", type: rankingTypes.soundStyle }, { name: "sounds", type: rankingTypes.sounds },], outputs: [{ name: "track", type: compositionTypes.audioTrack }], needs: [] },],diff --git a/packages/ranking/src/schedule.ts b/packages/ranking/src/schedule.tsindex 1a38320f..60b0a4e1 100644--- a/packages/ranking/src/schedule.ts+++ b/packages/ranking/src/schedule.ts@@ -16,7 +16,8 @@ import type { ProgramSpace } from "@hypit/program-space";import { canonicalize, isDigest } from "@hypit/protocol";import { verifyText } from "@hypit/text";import type { Text } from "@hypit/text";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import { projectSemanticProgramSpace } from "@hypit/semantic-track";+import type { SemanticTrack } from "@hypit/semantic-track";import { assertCanvasSpace, assertSpatialFrame } from "@hypit/spatial";import type { CanvasSpace } from "@hypit/spatial";import {@@ -292,8 +293,7 @@ export function appendRankingItemSpec(set: RankingItemSpecSet, spec: RankingItemexport function buildRankingSchedule(input: {readonly header: RankingHeader;readonly items: RankingItemSpecSet;- readonly map: CompleteSemanticMap;- readonly space: ProgramSpace;+ readonly semantic: SemanticTrack;readonly outer: NarrativeSelectionRef;readonly triggers: NarrativeMomentRef;readonly terminal: NarrativeMomentRef;@@ -303,12 +303,12 @@ export function buildRankingSchedule(input: {assert(input.items.variant === input.header.variant, "Ranking schedule variant disagrees with its item set.");assert(input.header.variant !== "column", "Column uses item-owned Selection windows, not the triggered Ranking schedule.");assert(input.items.items.length > 0, "Ranking requires at least one Item.");- assertProgramSpaceIdentity(input.space);- const outer = locateSelectionOccurrences(input.map, input.outer, input.space);+ const space = projectSemanticProgramSpace(input.semantic);+ const outer = locateSelectionOccurrences(input.semantic, input.outer);assert(outer.length === 1, `Ranking outer Selection requires exactly one occurrence; received ${outer.length}.`);- const terminal = locateMomentOccurrences(input.map, input.terminal, input.space);+ const terminal = locateMomentOccurrences(input.semantic, input.terminal);assert(terminal.length === 1, `Ranking terminal Moment requires exactly one occurrence; received ${terminal.length}.`);- const triggers = locateMomentOccurrences(input.map, input.triggers, input.space);+ const triggers = locateMomentOccurrences(input.semantic, input.triggers);assert(triggers.length > 0, "Ranking trigger Moment requires at least one occurrence.");assert(triggers.length === input.items.items.length,`Ranking trigger/item cardinality differs: ${triggers.length} triggers for ${input.items.items.length} Items.`);@@ -336,29 +336,27 @@ export function buildRankingSchedule(input: {terminalFrame: resolved.terminalFrame,entries,};- assertRankingSchedule(result, input.space);+ assertRankingSchedule(result, space);return canonicalize(result) as unknown as TriggeredRankingSchedule;}export function projectColumnSelectionOuterWindow(- map: CompleteSemanticMap,- space: ProgramSpace,+ semantic: SemanticTrack,selection: NarrativeSelectionRef,): ColumnOuterWindow {const occurrence = projectSelectionWindows({- itemId: "column-outer", map, selection, space, expansion: { kind: "one" },+ itemId: "column-outer", semantic, selection, expansion: { kind: "one" },projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } },})[0]!;return canonicalize({ span: { ...occurrence.span } }) as unknown as ColumnOuterWindow;}export function projectColumnSegmentOuterWindow(- map: CompleteSemanticMap,- space: ProgramSpace,+ semantic: SemanticTrack,segment: NarrativeExcerpt,): ColumnOuterWindow {const occurrence = projectSegmentWindow({- itemId: "column-outer", map, segment, space,+ itemId: "column-outer", semantic, segment,projection: { start: { ref: "segment.start" }, end: { ref: "segment.end" } },});return canonicalize({ span: { ...occurrence.span } }) as unknown as ColumnOuterWindow;@@ -386,8 +384,7 @@ export function assertColumnWindowCandidateSet(value: ColumnWindowCandidateSet):export function appendColumnWindowCandidate(set: ColumnWindowCandidateSet,spec: ColumnItemSpec,- map: CompleteSemanticMap,- space: ProgramSpace,+ semantic: SemanticTrack,selection: NarrativeSelectionRef,): ColumnWindowCandidateSet {assertColumnWindowCandidateSet(set);@@ -395,7 +392,7 @@ export function appendColumnWindowCandidate(assert(!spec.preset, `Preset Column Item ${spec.id} cannot consume a Selection window.`);assert(!set.entries.some((entry) => entry.itemId === spec.id), `Column Item ${spec.id} already has a Selection window.`);const occurrence = projectSelectionWindows({- itemId: spec.id, map, selection, space, expansion: { kind: "one" },+ itemId: spec.id, semantic, selection, expansion: { kind: "one" },projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } },})[0]!;const result: ColumnWindowCandidateSet = {diff --git a/packages/ranking/src/surface.ts b/packages/ranking/src/surface.tsindex 7f90f145..5b0e3805 100644--- a/packages/ranking/src/surface.ts+++ b/packages/ranking/src/surface.ts@@ -14,8 +14,7 @@ import { sameType, type CanonicalValue, type TypeRef } from "@hypit/protocol";import { mediaTypes } from "@hypit/media";import type { FontArtifactRef, FontStackRef } from "@hypit/media";import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import { svsRecipeType } from "@hypit/svs";import type { SvsRecipe } from "@hypit/svs";@@ -192,15 +191,14 @@ const variantDefinition = {function rankingSurface(variant: RankingVariant): StructuredSurfaceHandler {return ({ element, resolveReference }) => {- const common = ["id", "map", "space", "frame", "during", "triggers", "terminal", "style", "appear-sound", "move-sound"];+ const common = ["id", "semantic", "frame", "during", "triggers", "terminal", "style", "appear-sound", "move-sound"];const attributes = variant === "column"? [...common.filter((name) => name !== "triggers" && name !== "terminal"), "canvas"]: common;allowed(element, attributes);const id = text(element, "id");const selected = variantDefinition[variant];- const map = reference(element.attributes.map, `${element.name}.map`, semanticMapTypes.complete, resolveReference);- const space = reference(element.attributes.space, `${element.name}.space`, programSpaceTypes.programSpace, resolveReference);+ const semantic = reference(element.attributes.semantic, `${element.name}.semantic`, semanticTrackTypes.track, resolveReference);const canvas = variant === "column"? reference(element.attributes.canvas, `${element.name}.canvas`, spatialTypes.canvas, resolveReference): undefined;@@ -221,8 +219,8 @@ function rankingSurface(variant: RankingVariant): StructuredSurfaceHandler {value: { kind: "inline", value: sealRankingHeader({ id, variant }) as unknown as CanonicalValue },range: element.range,});- const inputs: Record<string, typeof map.ref> = {- header: { kind: "record", id: headerId }, map: map.ref, space: space.ref, frame: frame.ref,+ const inputs: Record<string, typeof semantic.ref> = {+ header: { kind: "record", id: headerId }, semantic: semantic.ref, frame: frame.ref,outer: outer.ref, style: style.ref,...(canvas === undefined ? {} : { canvas: canvas.ref }),...(triggers === undefined ? {} : { triggers: triggers.ref }),diff --git a/packages/ranking/test/ranking.test.ts b/packages/ranking/test/ranking.test.tsindex 548dd3d2..cb2d4793 100644--- a/packages/ranking/test/ranking.test.ts+++ b/packages/ranking/test/ranking.test.ts@@ -2,13 +2,13 @@ import assert from "node:assert/strict";import test from "node:test";import { videoContractManifests } from "../../../test/support/video-domain.js";import { fixtureDigest } from "../../../test/fixture-digest.js";+import { semanticTrackFixture } from "../../../test/semantic-track-fixture.js";import { createResolvedClosure } from "@hypit/core";import type { FontArtifactRef, SynchronizedMedia } from "@hypit/media";import { mediaTypes } from "@hypit/media";import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrative";import { sealProgramSpace } from "@hypit/program-space";-import type { CompleteSemanticMap } from "@hypit/semantic-map";import { sealCanvasSpace, sealSpatialFrame } from "@hypit/spatial";import type { SvsRecipe } from "@hypit/svs";@@ -64,8 +64,7 @@ import type {TopThreeItemSpec,} from "@hypit/ranking";import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import { svsRecipeType } from "@hypit/svs";import { sealText, textManifest, textTypes } from "@hypit/text";@@ -87,11 +86,14 @@ const canvas = sealCanvasSpace({const frame = sealSpatialFrame({xPx: 40, yPx: 80, widthPx: 720, heightPx: 560,});-const map: CompleteSemanticMap = {- tokens: [],+const semanticTrack = semanticTrackFixture(space, {+ segments: [+ { id: "opening", frameCount: 10 },+ { id: "ranking", frameCount: 220 },+ { id: "ending", frameCount: 10 },+ ],anchors: [{ identity: "outer-start", frame: 10 },- { identity: "segment:ranking:start", frame: 10 },{ identity: "early-start", frame: 25 },{ identity: "one", frame: 30 },{ identity: "early-end", frame: 65 },@@ -104,9 +106,8 @@ const map: CompleteSemanticMap = {{ identity: "late-end", frame: 160 },{ identity: "terminal", frame: 190 },{ identity: "outer-end", frame: 230 },- { identity: "segment:ranking:end", frame: 230 },],-};+});const outer: NarrativeSelectionRef = {id: "ranking-window",occurrences: [{ occurrence: 0, startAnchorId: "outer-start", endAnchorId: "outer-end" }],@@ -148,7 +149,7 @@ function specs(headerValue: RankingHeader, values: readonly RankingItemSpec[]) {function schedule(headerValue: RankingHeader, values: readonly RankingItemSpec[]): TriggeredRankingSchedule {return buildRankingSchedule({- header: headerValue, items: specs(headerValue, values), map, space, outer,+ header: headerValue, items: specs(headerValue, values), semantic: semanticTrack, outer,triggers: triggers(values.length), terminal,});}@@ -167,14 +168,14 @@ function columnSchedule(owner: RankingHeader,values: readonly ColumnItemSpec[],windows: Readonly<Record<string, NarrativeSelectionRef>>,- outerWindow = projectColumnSegmentOuterWindow(map, space, rankingSegment),+ outerWindow = projectColumnSegmentOuterWindow(semanticTrack, rankingSegment),): ColumnSchedule {let candidates = createColumnWindowCandidateSet();for (const value of values) {if (value.preset) continue;const timing = windows[value.id];if (timing === undefined) throw new Error(`Missing test Selection for ${value.id}.`);- candidates = appendColumnWindowCandidate(candidates, value, map, space, timing);+ candidates = appendColumnWindowCandidate(candidates, value, semanticTrack, timing);}return buildColumnSchedule({ header: owner, items: specs(owner, values), outer: outerWindow, candidates });}@@ -194,15 +195,15 @@ test("RankingSchedule rejects cardinality, outer/terminal ambiguity and authoredconst owner = header("tier-board");const values = [tierSpec("a", "s"), tierSpec("b", "a")];assert.throws(() => buildRankingSchedule({- header: owner, items: specs(owner, values), map, space, outer, triggers: triggers(1), terminal,+ header: owner, items: specs(owner, values), semantic: semanticTrack, outer, triggers: triggers(1), terminal,}), /cardinality/u);assert.throws(() => buildRankingSchedule({- header: owner, items: specs(owner, values), map, space,+ header: owner, items: specs(owner, values), semantic: semanticTrack,outer: { ...outer, occurrences: [...outer.occurrences, { ...outer.occurrences[0]!, occurrence: 1 }] },triggers: triggers(2), terminal,}), /exactly one/u);assert.throws(() => buildRankingSchedule({- header: owner, items: specs(owner, values), map, space, outer,+ header: owner, items: specs(owner, values), semantic: semanticTrack, outer,triggers: { ...triggers(2), occurrences: [triggers(2).occurrences[1]!, triggers(2).occurrences[0]!] }, terminal,}), /strictly increasing/u);});@@ -233,7 +234,7 @@ test("TierBoard owns cumulative direct/stage placement and rejects invalid schedassert.equal(track.presents.find((item) => item.id.endsWith(":item:alpha:settled"))?.span.endFrameExclusive, 230);const tightTerminal = { ...terminal, occurrences: [{ occurrence: 0, anchorId: "two" }] };assert.throws(() => buildRankingSchedule({- header: owner, items: specs(owner, semantic), map, space, outer, triggers: triggers(2), terminal: tightTerminal,+ header: owner, items: specs(owner, semantic), semantic: semanticTrack, outer, triggers: triggers(2), terminal: tightTerminal,}), /outside|strictly increasing/u);});@@ -251,7 +252,7 @@ test("Column projects a Segment outer, keeps rank independent from reveal time,set = appendColumnItem(set, semantic[1]!, image("preset"));set = appendColumnItem(set, semantic[2]!);set = appendColumnItem(set, semantic[3]!, image("overlap"));- const outerWindow = projectColumnSegmentOuterWindow(map, space, rankingSegment);+ const outerWindow = projectColumnSegmentOuterWindow(semanticTrack, rankingSegment);assert.deepEqual(outerWindow.span, { startFrame: 10, endFrameExclusive: 230 });const value = columnSchedule(owner, semantic, {"late-rank-one": late,@@ -358,8 +359,7 @@ test("all three author Surfaces preserve explicit semantic, spatial, font, imagerecord: { value: { kind: "inline", value } } as never,});const references = new Map<string, SurfaceResolvedReference>([- ["map", plain("map", semanticMapTypes.complete)],- ["space", plain("space", programSpaceTypes.programSpace)],+ ["semantic", plain("semantic", semanticTrackTypes.track)],["canvas", plain("canvas", spatialTypes.canvas)],["frame", plain("frame", spatialTypes.frame)],["outer", plain("outer", narrativeTypes.selection)],@@ -395,11 +395,11 @@ test("all three author Surfaces preserve explicit semantic, spatial, font, image}const cases = [[decodeTierBoardSurface, node("ranking:TierBoard", {- id: "tier", map: ref("map"), space: ref("space"), frame: ref("frame"), during: ref("outer"),+ id: "tier", semantic: ref("semantic"), frame: ref("frame"), during: ref("outer"),triggers: ref("triggers"), terminal: ref("terminal"), style: ref("tier-style"),}, [node("ranking:TierItem", { id: "tier-one", tier: "s", entry: "stage", icon: ref("icon-1") })])],[decodeColumnSurface, node("ranking:Column", {- id: "column", map: ref("map"), space: ref("space"), canvas: ref("canvas"), frame: ref("frame"), during: ref("ranking-segment"),+ id: "column", semantic: ref("semantic"), canvas: ref("canvas"), frame: ref("frame"), during: ref("ranking-segment"),style: ref("column-style"),"appear-sound": ref("appear"), "move-sound": ref("move"),}, [@@ -407,7 +407,7 @@ test("all three author Surfaces preserve explicit semantic, spatial, font, imagenode("ranking:ColumnItem", { id: "column-two", rank: "5", preset: "true", label: "Two", icon: ref("icon-2") }),])],[decodeTopThreeSurface, node("ranking:TopThree", {- id: "top", map: ref("map"), space: ref("space"), frame: ref("frame"), during: ref("outer"),+ id: "top", semantic: ref("semantic"), frame: ref("frame"), during: ref("outer"),triggers: ref("triggers"), terminal: ref("terminal"), style: ref("top-style"),}, [node("ranking:TopThreeItem", { label: "First" }), node("ranking:TopThreeItem", { label: "Second", icon: ref("icon-1") })])],] as const;@@ -464,13 +464,13 @@ test("Ranking author Surfaces fail closed on impossible image and sound combinatconst ref = (path: string): MarkupAttributeValue => ({ kind: "reference", path });const plain = (path: string, type: SurfaceResolvedReference["type"]): SurfaceResolvedReference => ({ path, ref: { kind: "record", id: path }, type });const references = new Map<string, SurfaceResolvedReference>([- ["map", plain("map", semanticMapTypes.complete)], ["space", plain("space", programSpaceTypes.programSpace)],+ ["semantic", plain("semantic", semanticTrackTypes.track)],["frame", plain("frame", spatialTypes.frame)], ["outer", plain("outer", narrativeTypes.selection)],["triggers", plain("triggers", narrativeTypes.moment)], ["terminal", plain("terminal", narrativeTypes.moment)],["style", plain("style", rankingTypes.tierStyle)], ["style.sound", plain("style.sound", rankingTypes.soundStyle)],["move", plain("move", mediaTypes.synchronized)],]);- const common = { id: "bad", map: ref("map"), space: ref("space"), frame: ref("frame"), during: ref("outer"), triggers: ref("triggers"), terminal: ref("terminal"), style: ref("style") };+ const common = { id: "bad", semantic: ref("semantic"), frame: ref("frame"), during: ref("outer"), triggers: ref("triggers"), terminal: ref("terminal"), style: ref("style") };const context = (element: StructuredElement) => ({sourceName: "ranking.svml", element, resolveReference: (path: string) => references.get(path),resolveAsset: async () => { throw new Error("no asset resolution expected"); },diff --git a/packages/render-hyperframes/package.json b/packages/render-hyperframes/package.jsonindex f8f31b0c..cc19b262 100644--- a/packages/render-hyperframes/package.json+++ b/packages/render-hyperframes/package.json@@ -20,11 +20,13 @@"@hypit/media-pipeline": "workspace:*","@hypit/program-space": "workspace:*","@hypit/protocol": "workspace:*",+ "@hypit/semantic-track": "workspace:*","@hypit/markup": "workspace:*"},"devDependencies": {"@hypit/core": "workspace:*","@hypit/driver-node": "workspace:*",+ "@hypit/svs": "workspace:*","@hypit/validation": "workspace:*"}}diff --git a/packages/render-hyperframes/src/fragment.ts b/packages/render-hyperframes/src/fragment.tsindex 4647b4ec..772dfea9 100644--- a/packages/render-hyperframes/src/fragment.ts+++ b/packages/render-hyperframes/src/fragment.ts@@ -1,5 +1,5 @@import { artifactTypes } from "@hypit/artifact";-import { programSpaceTypes } from "@hypit/program-space";+import { semanticTrackProducers, semanticTrackTypes } from "@hypit/semantic-track";import { compositionTypes } from "@hypit/composition";import { sealGraphFragment } from "@hypit/elaborator";import { hyperframesProducers } from "@hypit/hyperframes";@@ -15,13 +15,19 @@ const operation = (id: string) => ({ kind: "fragment-operation" as const, operatexport const renderHyperframesFragment = sealGraphFragment({inputs: [{ name: "composition", type: compositionTypes.composition },- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },],operations: [+ {+ id: "project-space",+ producer: semanticTrackProducers.projectProgramSpace,+ inputs: { track: input("semantic") },+ result: { kind: "output", name: "space" },+ },{id: "compile-document",producer: hyperframesProducers.compile,- inputs: { composition: input("composition"), space: input("space") },+ inputs: { composition: input("composition"), space: operation("project-space") },result: { kind: "output", name: "document" },},{@@ -33,7 +39,7 @@ export const renderHyperframesFragment = sealGraphFragment({{id: "compile-audio-program",producer: mediaPipelineProducers.planAudio,- inputs: { composition: input("composition"), space: input("space") },+ inputs: { composition: input("composition"), space: operation("project-space") },result: { kind: "output", name: "plan" },},{diff --git a/packages/render-hyperframes/src/manifest.ts b/packages/render-hyperframes/src/manifest.tsindex 259defe4..9a43914e 100644--- a/packages/render-hyperframes/src/manifest.ts+++ b/packages/render-hyperframes/src/manifest.ts@@ -1,7 +1,7 @@import { artifactDependency, artifactTypes } from "@hypit/artifact";import { mediaDependency, mediaTypes } from "@hypit/media";import { compositionDependency, compositionTypes } from "@hypit/composition";-import { programSpaceTypes } from "@hypit/program-space";+import { semanticTrackDependency, semanticTrackTypes } from "@hypit/semantic-track";import {hyperframesManifest,hyperframesModuleRef,@@ -31,22 +31,22 @@ export const renderHyperframesMarkupSurfaces = [{outputs: [],vocabulary: {summary:- "Renders one Composition on one ProgramSpace into a final video, publishing the muxed result as a BlobArtifact.",+ "Renders one Composition on one SemanticTrack into a final video, publishing the muxed result as a BlobArtifact.",attributes: [{ name: "id", kind: "identifier", required: true,summary: "Names the render component and the final video this element publishes." },{ name: "composition", kind: "reference", required: true,accepts: [compositionTypes.composition],summary: "Selects the Composition this element compiles, renders and muxes." },- { name: "space", kind: "reference", required: true,- accepts: [programSpaceTypes.programSpace],- summary: "Selects the ProgramSpace whose duration and frame rate every rendered Product is bound to." },+ { name: "semantic", kind: "reference", required: true,+ accepts: [semanticTrackTypes.track],+ summary: "Selects the SemanticTrack whose derived duration and frame rate every rendered Product is bound to." },],ports: [{ name: "video", type: artifactTypes.blob,summary: "The final muxed video Artifact, addressed as `<id>.video`." },],- example: '<render:Video id="final" composition={main.composition} space={speech.space}/>',+ example: '<render:Video id="final" composition={main.composition} semantic={speech.semantic}/>',notes: ["All three attributes are required; the element accepts no children and no text content.","The visual render, the audio render and the mux are three separate Needs, each realized by a Provider this package does not choose.",@@ -64,6 +64,7 @@ export const renderHyperframesManifest: ModuleManifest = {artifactDependency,mediaDependency,compositionDependency,+ semanticTrackDependency,{ module: hyperframesModuleRef },{ module: mediaPipelineModuleRef },],diff --git a/packages/render-hyperframes/src/surface.ts b/packages/render-hyperframes/src/surface.tsindex 18982545..bc4a8a69 100644--- a/packages/render-hyperframes/src/surface.ts+++ b/packages/render-hyperframes/src/surface.ts@@ -1,5 +1,4 @@-import { programSpaceTypes } from "@hypit/program-space";-import type { ProgramSpace } from "@hypit/program-space";+import { semanticTrackTypes } from "@hypit/semantic-track";import { compositionTypes } from "@hypit/composition";import type { Composition } from "@hypit/composition";import type {@@ -26,7 +25,7 @@ function referenceAttribute(element: StructuredElement, name: string): string {export const decodeHyperframesRenderSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {const names = Object.keys(element.attributes).sort();- if (names.join(",") !== "composition,id,space") throw new Error(`${element.name} requires exactly composition, id and space`);+ if (names.join(",") !== "composition,id,semantic") throw new Error(`${element.name} requires exactly composition, id and semantic`);if (element.children.some((child) => child.kind === "element" || child.value.trim().length > 0)) {throw new Error(`${element.name} does not accept children`);}@@ -39,20 +38,20 @@ export const decodeHyperframesRenderSurface: StructuredSurfaceHandler = ({ eleme|| composition.type.name !== compositionTypes.composition.name) {throw new Error(`${element.name}.composition must reference Composition`);}- const spacePath = referenceAttribute(element, "space");- const space = resolveReference(spacePath);- if (space === undefined- || space.type.module.name !== programSpaceTypes.programSpace.module.name- || space.type.module.version !== programSpaceTypes.programSpace.module.version- || space.type.name !== programSpaceTypes.programSpace.name) {- throw new Error(`${element.name}.space must reference ProgramSpace`);+ const semanticPath = referenceAttribute(element, "semantic");+ const semantic = resolveReference(semanticPath);+ if (semantic === undefined+ || semantic.type.module.name !== semanticTrackTypes.track.module.name+ || semantic.type.module.version !== semanticTrackTypes.track.module.version+ || semantic.type.name !== semanticTrackTypes.track.name) {+ throw new Error(`${element.name}.semantic must reference SemanticTrack`);}return {records: [],components: [{id,fragment: renderHyperframesFragment.id,- inputs: { composition: composition.ref, space: space.ref },+ inputs: { composition: composition.ref, semantic: semantic.ref },outputs: { video: `${id}.video` },range: element.range,}],diff --git a/packages/render-hyperframes/test/render.test.ts b/packages/render-hyperframes/test/render.test.tsindex 7e577819..10b4a796 100644--- a/packages/render-hyperframes/test/render.test.ts+++ b/packages/render-hyperframes/test/render.test.ts@@ -11,6 +11,7 @@ import type { Composition } from "@hypit/composition";import assert from "node:assert/strict";import test from "node:test";import { fixtureDigest } from "../../../test/fixture-digest.js";+import { semanticTrackFixture } from "../../../test/semantic-track-fixture.js";import {createResolvedClosure,@@ -76,11 +77,18 @@ import {createMarkupAuthorFrontend,MarkupSurfaceRegistry,} from "@hypit/markup";+import {+ semanticTrackComponent,+ semanticTrackDependency,+ semanticTrackTypes,+} from "@hypit/semantic-track";+import { svsManifest } from "@hypit/svs";const space = sealProgramSpace({durationSec: 2,frameRate: { numerator: 30, denominator: 1 },});+const semantic = semanticTrackFixture(space);const composition = sealComposition({id: "render-test",canvas: { width: 1080, height: 1920, clearColor: "#000000" },@@ -99,24 +107,25 @@ const closure = createResolvedClosure([hyperframesManifest,mediaPipelineManifest,renderHyperframesManifest,+ svsManifest,]);const compositionRecord = await admitRecord(closure, sealRecord({id: "composition",type: compositionTypes.composition,value: stored(composition),}), validatorRegistry());-const spaceRecord = await admitRecord(closure, sealRecord({- id: "space",- type: programSpaceTypes.programSpace,- value: stored(space),+const semanticRecord = await admitRecord(closure, sealRecord({+ id: "semantic",+ type: semanticTrackTypes.track,+ value: stored(semantic),}), validatorRegistry());-const linked = link(closure, [compositionRecord, spaceRecord]);+const linked = link(closure, [compositionRecord, semanticRecord]);const instance = elaborateGraphFragment(linked, renderHyperframesFragment, {id: "final",fragment: renderHyperframesFragment.id,inputs: {composition: { kind: "record", id: compositionRecord.id },- space: { kind: "record", id: spaceRecord.id },+ semantic: { kind: "record", id: semanticRecord.id },},});const contribution = bindAuthorFragment(instance, { video: "final.video" });@@ -133,6 +142,7 @@ function producerRegistry(): ProducerRegistry {registerProducerFacets(registry, mediaPipelineComponent.producers);registerProducerFacets(registry, hyperframesComponent.producers);registerProducerFacets(registry, renderHyperframesComponent.producers);+ registerProducerFacets(registry, semanticTrackComponent.producers);return registry;}@@ -147,6 +157,7 @@ function validatorRegistry(): TypeValidatorRegistry {test("HyperFrames rendering is an explicit exact Need after ordinary document compilation", async () => {assert.deepEqual(build().plan.steps.map((step) => step.producer.name).sort(), [+ "project-program-space",hyperframesProducers.compile.name,renderHyperframesProducers.requestVisual.name,mediaPipelineProducers.planAudio.name,@@ -303,13 +314,13 @@ const fixtureModule = { name: "example.composition-fixture", version: "1" } as cconst fixtureSurfaceDigest = fixtureDigest("example.composition-fixture/surface@1");const fixtureSurface = {name: "composition", tag: "Composition", mode: "structured",- outputs: [compositionTypes.composition, programSpaceTypes.programSpace],+ outputs: [compositionTypes.composition, semanticTrackTypes.track],} as const;const fixtureManifest: ModuleManifest = {format: "hypit.module@1",name: fixtureModule.name,version: fixtureModule.version,- dependencies: [compositionDependency, programSpaceDependency],+ dependencies: [compositionDependency, semanticTrackDependency],types: [],capabilities: [],producers: [],@@ -330,12 +341,13 @@ test("the final rendered video is an ordinary BlobArtifact that can feed anothermediaPipelineManifest,renderHyperframesManifest,fixtureManifest,+ svsManifest,]);const surfaces = new MarkupSurfaceRegistry();surfaces.registerStructured({ module: fixtureModule, declaration: fixtureSurface, handler: ({ element }) => ({records: [{ id: "composition", type: compositionTypes.composition, value: stored(composition), range: element.range },- { id: "space", type: programSpaceTypes.programSpace, value: stored(space), range: element.range },+ { id: "semantic", type: semanticTrackTypes.track, value: stored(semantic), range: element.range },],components: [],fragments: [],@@ -365,7 +377,7 @@ test("the final rendered video is an ordinary BlobArtifact that can feed another<import as="render" from="@hypit/render-hyperframes@1"/><import as="media" from="@hypit/media-pipeline@1"/><fixture:Composition/>- <render:Video id="final" composition={composition} space={space}/>+ <render:Video id="final" composition={composition} semantic={semantic}/><media:ExtractFrame id="poster" source={final.video} video="primary-moving" at="last"/></svml>`),closure: sourceClosure,@@ -381,6 +393,7 @@ test("the final rendered video is an ordinary BlobArtifact that can feed anothertargets: [{ output: target.ref.kind === "logical-output" ? target.ref.id : "" }],}));assert.deepEqual(state.plan.steps.map((step) => step.producer.name).sort(), [+ "project-program-space",hyperframesProducers.compile.name,renderHyperframesProducers.requestVisual.name,mediaPipelineProducers.planAudio.name,diff --git a/packages/screen-overlay/package.json b/packages/screen-overlay/package.jsonindex 6da5839f..295e45c5 100644--- a/packages/screen-overlay/package.json+++ b/packages/screen-overlay/package.json@@ -11,9 +11,9 @@"@hypit/composition": "workspace:*","@hypit/elaborator": "workspace:*","@hypit/narrative": "workspace:*",- "@hypit/program-space": "workspace:*","@hypit/protocol": "workspace:*",- "@hypit/semantic-map": "workspace:*",+ "@hypit/program-space": "workspace:*",+ "@hypit/semantic-track": "workspace:*","@hypit/spatial": "workspace:*","@hypit/temporal": "workspace:*","@hypit/markup": "workspace:*"diff --git a/packages/screen-overlay/src/component.ts b/packages/screen-overlay/src/component.tsindex ae7c1f64..dba089ab 100644--- a/packages/screen-overlay/src/component.ts+++ b/packages/screen-overlay/src/component.ts@@ -1,9 +1,9 @@import type { ComponentPackage } from "@hypit/component-kit";import type { NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrative";-import type { ProgramSpace } from "@hypit/program-space";import { canonicalize } from "@hypit/protocol";import type { StoredValue } from "@hypit/protocol";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import type { SemanticTrack } from "@hypit/semantic-track";+import { projectSemanticProgramSpace } from "@hypit/semantic-track";import type { CanvasSpace } from "@hypit/spatial";import { screenOverlayProducers, screenOverlayTypes } from "./manifest.js";import { appendMomentScreenOverlay, appendProgramScreenOverlay, appendSelectionScreenOverlay, assertScreenOverlayProgram, createScreenOverlaySet, finalizeScreenOverlay, renderScreenOverlay } from "./program.js";@@ -13,11 +13,11 @@ const output = (value: unknown) => ({ kind: "inline" as const, value: canonicaliexport const screenOverlayComponent = {producers: [{ producer: screenOverlayProducers.createSet, handler: () => ({ outputs: { set: output(createScreenOverlaySet()) }, needs: {} }) },- { producer: screenOverlayProducers.appendProgram, handler: ({ inputs }) => ({ outputs: { set: output(appendProgramScreenOverlay(inline<ScreenOverlaySet>(inputs.set?.value, "ScreenOverlaySet"), inline<ScreenOverlayHeader>(inputs.header?.value, "ScreenOverlayHeader"), inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<ScreenOverlayItemSpec>(inputs.spec?.value, "ScreenOverlayItemSpec"))) }, needs: {} }) },- { producer: screenOverlayProducers.appendSelection, handler: ({ inputs }) => ({ outputs: { set: output(appendSelectionScreenOverlay(inline<ScreenOverlaySet>(inputs.set?.value, "ScreenOverlaySet"), inline<ScreenOverlayHeader>(inputs.header?.value, "ScreenOverlayHeader"), inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"), inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelection"), inline<ScreenOverlayItemSpec>(inputs.spec?.value, "ScreenOverlayItemSpec"))) }, needs: {} }) },- { producer: screenOverlayProducers.appendMoment, handler: ({ inputs }) => ({ outputs: { set: output(appendMomentScreenOverlay(inline<ScreenOverlaySet>(inputs.set?.value, "ScreenOverlaySet"), inline<ScreenOverlayHeader>(inputs.header?.value, "ScreenOverlayHeader"), inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"), inline<NarrativeMomentRef>(inputs.moment?.value, "NarrativeMoment"), inline<ScreenOverlayItemSpec>(inputs.spec?.value, "ScreenOverlayItemSpec"))) }, needs: {} }) },+ { producer: screenOverlayProducers.appendProgram, handler: ({ inputs }) => ({ outputs: { set: output(appendProgramScreenOverlay(inline<ScreenOverlaySet>(inputs.set?.value, "ScreenOverlaySet"), inline<ScreenOverlayHeader>(inputs.header?.value, "ScreenOverlayHeader"), inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"), inline<ScreenOverlayItemSpec>(inputs.spec?.value, "ScreenOverlayItemSpec"))) }, needs: {} }) },+ { producer: screenOverlayProducers.appendSelection, handler: ({ inputs }) => ({ outputs: { set: output(appendSelectionScreenOverlay(inline<ScreenOverlaySet>(inputs.set?.value, "ScreenOverlaySet"), inline<ScreenOverlayHeader>(inputs.header?.value, "ScreenOverlayHeader"), inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"), inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelection"), inline<ScreenOverlayItemSpec>(inputs.spec?.value, "ScreenOverlayItemSpec"))) }, needs: {} }) },+ { producer: screenOverlayProducers.appendMoment, handler: ({ inputs }) => ({ outputs: { set: output(appendMomentScreenOverlay(inline<ScreenOverlaySet>(inputs.set?.value, "ScreenOverlaySet"), inline<ScreenOverlayHeader>(inputs.header?.value, "ScreenOverlayHeader"), inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"), inline<NarrativeMomentRef>(inputs.moment?.value, "NarrativeMoment"), inline<ScreenOverlayItemSpec>(inputs.spec?.value, "ScreenOverlayItemSpec"))) }, needs: {} }) },{ producer: screenOverlayProducers.finalize, handler: ({ inputs }) => ({ outputs: { program: output(finalizeScreenOverlay(inline<ScreenOverlaySet>(inputs.set?.value, "ScreenOverlaySet"), inline<ScreenOverlayHeader>(inputs.header?.value, "ScreenOverlayHeader"))) }, needs: {} }) },- { producer: screenOverlayProducers.render, handler: ({ inputs }) => ({ outputs: { track: output(renderScreenOverlay(inline<CanvasSpace>(inputs.canvas?.value, "CanvasSpace"), inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<ScreenOverlayProgram>(inputs.program?.value, "ScreenOverlayProgram"))) }, needs: {} }) },+ { producer: screenOverlayProducers.render, handler: ({ inputs }) => ({ outputs: { track: output(renderScreenOverlay(inline<CanvasSpace>(inputs.canvas?.value, "CanvasSpace"), projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")), inline<ScreenOverlayProgram>(inputs.program?.value, "ScreenOverlayProgram"))) }, needs: {} }) },],validators: [{ type: screenOverlayTypes.program, handler: ({ value }) => assertScreenOverlayProgram(inline<ScreenOverlayProgram>(value, "ScreenOverlayProgram")) }],} satisfies ComponentPackage;diff --git a/packages/screen-overlay/src/fragment.ts b/packages/screen-overlay/src/fragment.tsindex bc66f120..cae7b672 100644--- a/packages/screen-overlay/src/fragment.ts+++ b/packages/screen-overlay/src/fragment.ts@@ -2,41 +2,39 @@ import { compositionTypes } from "@hypit/composition";import { sealGraphFragment } from "@hypit/elaborator";import type { FragmentOperation } from "@hypit/elaborator";import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import { screenOverlayProducers, screenOverlayTypes } from "./manifest.js";export type ScreenOverlayFragmentItem =| { readonly kind: "program"; readonly specName: string }- | { readonly kind: "selection"; readonly specName: string; readonly mapName: string; readonly sourceName: string }- | { readonly kind: "moment"; readonly specName: string; readonly mapName: string; readonly sourceName: string };+ | { readonly kind: "selection"; readonly specName: string; readonly sourceName: string }+ | { readonly kind: "moment"; readonly specName: string; readonly sourceName: string };const input = (name: string) => ({ kind: "fragment-input" as const, name });const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });export function createScreenOverlayFragment(items: readonly ScreenOverlayFragmentItem[]) {if (items.length === 0) throw new Error("Screen Overlay Fragment requires at least one Item.");- const types = new Map<string, typeof screenOverlayTypes.itemSpec | typeof semanticMapTypes.complete | typeof narrativeTypes.selection | typeof narrativeTypes.moment>();+ const types = new Map<string, typeof screenOverlayTypes.itemSpec | typeof semanticTrackTypes.track | typeof narrativeTypes.selection | typeof narrativeTypes.moment>();const operations: FragmentOperation[] = [{ id: "overlay:set:empty", producer: screenOverlayProducers.createSet, inputs: {}, result: { kind: "output", name: "set" } }];let current = "overlay:set:empty";items.forEach((item, index) => {types.set(item.specName, screenOverlayTypes.itemSpec);if (item.kind !== "program") {- types.set(item.mapName, semanticMapTypes.complete);types.set(item.sourceName, item.kind === "selection" ? narrativeTypes.selection : narrativeTypes.moment);}const id = `overlay:set:append:${String(index + 1).padStart(4, "0")}`;operations.push({ id, producer: item.kind === "program" ? screenOverlayProducers.appendProgram : item.kind === "selection" ? screenOverlayProducers.appendSelection : screenOverlayProducers.appendMoment,- inputs: { set: operation(current), header: input("header"), space: input("space"), spec: input(item.specName),- ...(item.kind === "program" ? {} : { map: input(item.mapName), [item.kind]: input(item.sourceName) }) }, result: { kind: "output", name: "set" } });+ inputs: { set: operation(current), header: input("header"), semantic: input("semantic"), spec: input(item.specName),+ ...(item.kind === "program" ? {} : { [item.kind]: input(item.sourceName) }) }, result: { kind: "output", name: "set" } });current = id;});operations.push({ id: "overlay:program", producer: screenOverlayProducers.finalize, inputs: { set: operation(current), header: input("header") }, result: { kind: "output", name: "program" } },- { id: "overlay:track", producer: screenOverlayProducers.render, inputs: { canvas: input("canvas"), space: input("space"), program: operation("overlay:program") }, result: { kind: "output", name: "track" } },+ { id: "overlay:track", producer: screenOverlayProducers.render, inputs: { canvas: input("canvas"), semantic: input("semantic"), program: operation("overlay:program") }, result: { kind: "output", name: "track" } },);return sealGraphFragment({ inputs: [{ name: "canvas", type: spatialTypes.canvas }, { name: "header", type: screenOverlayTypes.header },- { name: "space", type: programSpaceTypes.programSpace }, ...[...types].map(([inputName, type]) => ({ name: inputName, type })),+ { name: "semantic", type: semanticTrackTypes.track }, ...[...types].map(([inputName, type]) => ({ name: inputName, type })),], operations, exports: [{ name: "program", type: screenOverlayTypes.program, root: operation("overlay:program") },{ name: "track", type: compositionTypes.visualTrack, root: operation("overlay:track") },diff --git a/packages/screen-overlay/src/manifest.ts b/packages/screen-overlay/src/manifest.tsindex 307452e1..78e8942c 100644--- a/packages/screen-overlay/src/manifest.ts+++ b/packages/screen-overlay/src/manifest.ts@@ -2,9 +2,8 @@ import { readFile } from "node:fs/promises";import { compositionDependency, compositionTypes } from "@hypit/composition";import { narrativeDependency, narrativeTypes } from "@hypit/narrative";-import { programSpaceDependency, programSpaceTypes } from "@hypit/program-space";import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@hypit/protocol";-import { semanticMapDependency, semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackDependency, semanticTrackTypes } from "@hypit/semantic-track";import { spatialDependency, spatialTypes } from "@hypit/spatial";import { temporalDependency } from "@hypit/temporal";@@ -76,7 +75,7 @@ export const screenOverlayHeaderSchema: ValueSchema = object({ id: { schema: strexport const screenOverlayItemSpecSchema: ValueSchema = itemSpec;export const screenOverlaySetSchema: ValueSchema = object({ items: { schema: { kind: "array", items: item } } });export const screenOverlayProgramSchema: ValueSchema = object({ id: { schema: string }, items: { schema: { kind: "array", minItems: 1, items: item } } });-const appendInputs = [{ name: "set", type: screenOverlayTypes.set }, { name: "header", type: screenOverlayTypes.header }, { name: "space", type: programSpaceTypes.programSpace }, { name: "spec", type: screenOverlayTypes.itemSpec }] as const;+const appendInputs = [{ name: "set", type: screenOverlayTypes.set }, { name: "header", type: screenOverlayTypes.header }, { name: "semantic", type: semanticTrackTypes.track }, { name: "spec", type: screenOverlayTypes.itemSpec }] as const;const itemAttributes = [{ name: "id", kind: "identifier", required: false,@@ -97,8 +96,6 @@ const itemAttributes = [summary: "Binds explicit `start` and `end` timing to a Selection." },{ name: "moment", kind: "reference", required: false, accepts: [narrativeTypes.moment],summary: "Binds explicit `start` and `end` timing to a Moment." },- { name: "map", kind: "reference", required: false, accepts: [semanticMapTypes.complete],- summary: "Chooses the semantic map the bound Selection or Moment is located through." },{ name: "occurrences", kind: "literal", required: false, values: ["one", "each"],summary: "Decides whether the item is painted once or at every occurrence of its Selection or Moment; defaults to `one`." },] as const;@@ -114,8 +111,8 @@ export const screenOverlayMarkupSurfaces = [summary: "Names this overlay so its Program and Track can be referenced elsewhere in the Source." },{ name: "canvas", kind: "reference", required: true, accepts: [spatialTypes.canvas],summary: "Chooses the Canvas every item is painted across." },- { name: "space", kind: "reference", required: true, accepts: [programSpaceTypes.programSpace],- summary: "Chooses the ProgramSpace the overlay is timed against." },+ { name: "semantic", kind: "reference", required: true, accepts: [semanticTrackTypes.track],+ summary: "Chooses the SemanticTrack that owns the frame domain and resolves temporal bindings." },],children: [{ tag: "Flash", cardinality: "many",@@ -298,17 +295,17 @@ export const screenOverlayMarkupSurfaces = [{ name: "track", type: compositionTypes.visualTrack,summary: "The rendered overlay, an ordinary peer VisualTrack." },],- example: `<screen:Track id="effects" space={speech.space} canvas={vertical}>- <screen:Flash during={story.selection.overlay} map={timing.map} z="80"+ example: `<screen:Track id="effects" semantic={speech.semantic} canvas={vertical}>+ <screen:Flash during={story.selection.overlay} z="80"color="#ffffff" intensity="0.6" attack="2" hold="2" decay="6"/></screen:Track>`,notes: ["The overlay requires at least one component child, and a component child is empty.","A component child writes exactly one temporal form, `during`, `at`, or `start` with `end`; none of them or more than one is refused.",- "`during` takes the literal `program` for the whole programme, or a Selection reference, which also requires `map`.",- "`at` requires `map` and `for`.",+ "`during` takes the literal `program` for the whole programme, or a Selection reference.",+ "`at` requires `for`.","`start` and `end` each take `program.start`, `program.end`, `selection.start`, `selection.end` or `moment.cue` with an optional `+` or `-` offset, or a bare duration read as an absolute point.",- "Explicit `start` and `end` timing binds a Selection through `selection` or a Moment through `moment`, never both, and either one requires `map`; `map` without a temporal source is refused.",+ "Explicit `start` and `end` timing binds a Selection through `selection` or a Moment through `moment`, never both.","A duration is written as an integral `f` or `ms` count, or as an `s` count that may carry a decimal fraction.",],} },@@ -317,7 +314,7 @@ export const screenOverlayMarkupSurfaces = [export const screenOverlayManifest: ModuleManifest = {format: "hypit.module@1", name: screenOverlayModuleRef.name, version: screenOverlayModuleRef.version,- dependencies: [narrativeDependency, semanticMapDependency, programSpaceDependency, spatialDependency, temporalDependency, compositionDependency],+ dependencies: [narrativeDependency, semanticTrackDependency, spatialDependency, temporalDependency, compositionDependency],types: [{ name: screenOverlayTypes.header.name },{ name: screenOverlayTypes.itemSpec.name },@@ -327,10 +324,10 @@ export const screenOverlayManifest: ModuleManifest = {producers: [{ name: screenOverlayProducers.createSet.name, inputs: [], outputs: [{ name: "set", type: screenOverlayTypes.set }], needs: [] },{ name: screenOverlayProducers.appendProgram.name, inputs: [...appendInputs], outputs: [{ name: "set", type: screenOverlayTypes.set }], needs: [] },- { name: screenOverlayProducers.appendSelection.name, inputs: [...appendInputs, { name: "map", type: semanticMapTypes.complete }, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: screenOverlayTypes.set }], needs: [] },- { name: screenOverlayProducers.appendMoment.name, inputs: [...appendInputs, { name: "map", type: semanticMapTypes.complete }, { name: "moment", type: narrativeTypes.moment }], outputs: [{ name: "set", type: screenOverlayTypes.set }], needs: [] },+ { name: screenOverlayProducers.appendSelection.name, inputs: [...appendInputs, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: screenOverlayTypes.set }], needs: [] },+ { name: screenOverlayProducers.appendMoment.name, inputs: [...appendInputs, { name: "moment", type: narrativeTypes.moment }], outputs: [{ name: "set", type: screenOverlayTypes.set }], needs: [] },{ name: screenOverlayProducers.finalize.name, inputs: [{ name: "set", type: screenOverlayTypes.set }, { name: "header", type: screenOverlayTypes.header }], outputs: [{ name: "program", type: screenOverlayTypes.program }], needs: [] },- { name: screenOverlayProducers.render.name, inputs: [{ name: "canvas", type: spatialTypes.canvas }, { name: "space", type: programSpaceTypes.programSpace }, { name: "program", type: screenOverlayTypes.program }], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [] },+ { name: screenOverlayProducers.render.name, inputs: [{ name: "canvas", type: spatialTypes.canvas }, { name: "semantic", type: semanticTrackTypes.track }, { name: "program", type: screenOverlayTypes.program }], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [] },],};export const screenOverlayDependency = { module: screenOverlayModuleRef } as const;diff --git a/packages/screen-overlay/src/program.ts b/packages/screen-overlay/src/program.tsindex 114d6ca2..9942fe23 100644--- a/packages/screen-overlay/src/program.ts+++ b/packages/screen-overlay/src/program.ts@@ -4,7 +4,7 @@ import type { NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrativeimport { assertProgramSpaceIdentity } from "@hypit/program-space";import type { ProgramSpace } from "@hypit/program-space";import { canonicalize } from "@hypit/protocol";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import type { SemanticTrack } from "@hypit/semantic-track";import { assertCanvasSpace } from "@hypit/spatial";import type { CanvasSpace } from "@hypit/spatial";import { projectMomentWindows, projectProgramWindow, projectSelectionWindows } from "@hypit/temporal";@@ -124,15 +124,15 @@ function realized(additions.forEach((item) => { assert(!ids.has(item.id), `Screen Overlay already contains Item ${item.id}.`); ids.add(item.id); });return { items: [...set.items, ...additions] };}-export function appendProgramScreenOverlay(set: ScreenOverlaySet, header: ScreenOverlayHeader, space: ProgramSpace, spec: ScreenOverlayItemSpec): ScreenOverlaySet {+export function appendProgramScreenOverlay(set: ScreenOverlaySet, header: ScreenOverlayHeader, semantic: SemanticTrack, spec: ScreenOverlayItemSpec): ScreenOverlaySet {assert(spec.expansion.kind === "one", `Program Overlay ${spec.id} must use one occurrence.`);- return realized(set, header, spec, [projectProgramWindow({ itemId: spec.id, space, projection: spec.projection })]);+ return realized(set, header, spec, [projectProgramWindow({ itemId: spec.id, semantic, projection: spec.projection })]);}-export function appendSelectionScreenOverlay(set: ScreenOverlaySet, header: ScreenOverlayHeader, space: ProgramSpace, map: CompleteSemanticMap, selection: NarrativeSelectionRef, spec: ScreenOverlayItemSpec): ScreenOverlaySet {- return realized(set, header, spec, projectSelectionWindows({ itemId: spec.id, map, selection, space, expansion: spec.expansion, projection: spec.projection }));+export function appendSelectionScreenOverlay(set: ScreenOverlaySet, header: ScreenOverlayHeader, semantic: SemanticTrack, selection: NarrativeSelectionRef, spec: ScreenOverlayItemSpec): ScreenOverlaySet {+ return realized(set, header, spec, projectSelectionWindows({ itemId: spec.id, semantic, selection, expansion: spec.expansion, projection: spec.projection }));}-export function appendMomentScreenOverlay(set: ScreenOverlaySet, header: ScreenOverlayHeader, space: ProgramSpace, map: CompleteSemanticMap, moment: NarrativeMomentRef, spec: ScreenOverlayItemSpec): ScreenOverlaySet {- return realized(set, header, spec, projectMomentWindows({ itemId: spec.id, map, moment, space, expansion: spec.expansion, projection: spec.projection }));+export function appendMomentScreenOverlay(set: ScreenOverlaySet, header: ScreenOverlayHeader, semantic: SemanticTrack, moment: NarrativeMomentRef, spec: ScreenOverlayItemSpec): ScreenOverlaySet {+ return realized(set, header, spec, projectMomentWindows({ itemId: spec.id, semantic, moment, expansion: spec.expansion, projection: spec.projection }));}export function sealScreenOverlayProgram(value: ScreenOverlayProgram): ScreenOverlayProgram {const normalized = { id: value.id,diff --git a/packages/screen-overlay/src/surface.ts b/packages/screen-overlay/src/surface.tsindex d26463f9..7e635c4b 100644--- a/packages/screen-overlay/src/surface.ts+++ b/packages/screen-overlay/src/surface.ts@@ -1,6 +1,5 @@import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { spatialTypes } from "@hypit/spatial";import type { StructuredElement, StructuredSurfaceHandler, SurfaceRecordDraft, SurfaceResolvedReference, MarkupAttributeValue } from "@hypit/markup";import type { TemporalDuration, TemporalPointExpression } from "@hypit/temporal";@@ -9,7 +8,7 @@ import { screenOverlayTypes } from "./manifest.js";import { sealScreenOverlayHeader, sealScreenOverlayItemSpec } from "./program.js";import type { ScreenOverlayComponent, ScreenOverlayItemSpec } from "./types.js";-const TIMING = ["during", "at", "for", "start", "end", "selection", "moment", "map", "occurrences"] as const;+const TIMING = ["during", "at", "for", "start", "end", "selection", "moment", "occurrences"] as const;function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;}@@ -62,7 +61,7 @@ function colors(element: StructuredElement, name: string): string[] {const values = text(element, name).split(",").map((value) => value.trim()).filter(Boolean);if (values.length === 0) throw new Error(`${element.name}.${name} requires colors.`); return values;}-type Binding = { readonly kind: "program" | "selection" | "moment"; readonly projection: ScreenOverlayItemSpec["projection"]; readonly source?: SurfaceResolvedReference; readonly map?: SurfaceResolvedReference };+type Binding = { readonly kind: "program" | "selection" | "moment"; readonly projection: ScreenOverlayItemSpec["projection"]; readonly source?: SurfaceResolvedReference };function binding(element: StructuredElement, resolve: (path: string) => SurfaceResolvedReference | undefined): Binding {const during = element.attributes.during; const at = element.attributes.at;const start = optionalText(element, "start"); const end = optionalText(element, "end");@@ -74,19 +73,16 @@ function binding(element: StructuredElement, resolve: (path: string) => SurfaceRreturn { kind: "program", projection: { start: { ref: "program.start" }, end: { ref: "program.end" } } };}return { kind: "selection", source: ref(during, `${element.name}.during`, narrativeTypes.selection, resolve),- map: ref(element.attributes.map, `${element.name}.map`, semanticMapTypes.complete, resolve),projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } } };}if (at !== undefined) return { kind: "moment", source: ref(at, `${element.name}.at`, narrativeTypes.moment, resolve),- map: ref(element.attributes.map, `${element.name}.map`, semanticMapTypes.complete, resolve),projection: { start: { ref: "moment.cue" }, end: { ref: "moment.cue", offset: duration(text(element, "for"), `${element.name}.for`) } } };if (start === undefined || end === undefined) throw new Error(`${element.name} explicit timing requires start and end.`);const selection = element.attributes.selection; const moment = element.attributes.moment;if (selection !== undefined && moment !== undefined) throw new Error(`${element.name} cannot bind Selection and Moment together.`);const projection = { start: point(start, `${element.name}.start`), end: point(end, `${element.name}.end`) };- if (selection !== undefined) return { kind: "selection", source: ref(selection, `${element.name}.selection`, narrativeTypes.selection, resolve), map: ref(element.attributes.map, `${element.name}.map`, semanticMapTypes.complete, resolve), projection };- if (moment !== undefined) return { kind: "moment", source: ref(moment, `${element.name}.moment`, narrativeTypes.moment, resolve), map: ref(element.attributes.map, `${element.name}.map`, semanticMapTypes.complete, resolve), projection };- if (element.attributes.map !== undefined) throw new Error(`${element.name}.map requires a temporal source.`);+ if (selection !== undefined) return { kind: "selection", source: ref(selection, `${element.name}.selection`, narrativeTypes.selection, resolve), projection };+ if (moment !== undefined) return { kind: "moment", source: ref(moment, `${element.name}.moment`, narrativeTypes.moment, resolve), projection };return { kind: "program", projection };}@@ -109,13 +105,13 @@ function content(element: StructuredElement): { readonly value: ScreenOverlayCom}export const decodeScreenOverlaySurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- allowed(element, ["id", "canvas", "space"]); const id = text(element, "id");+ allowed(element, ["id", "canvas", "semantic"]); const id = text(element, "id");const canvas = ref(element.attributes.canvas, `${element.name}.canvas`, spatialTypes.canvas, resolveReference);- const space = ref(element.attributes.space, `${element.name}.space`, programSpaceTypes.programSpace, resolveReference);+ const semantic = ref(element.attributes.semantic, `${element.name}.semantic`, semanticTrackTypes.track, resolveReference);const headerId = `${id}.header`; const records: SurfaceRecordDraft[] = [{ id: headerId, type: screenOverlayTypes.header,value: { kind: "inline", value: sealScreenOverlayHeader({ id }) }, range: element.range }];const fragmentItems: Parameters<typeof createScreenOverlayFragment>[0][number][] = [];- const inputs: Record<string, typeof canvas.ref> = { canvas: canvas.ref, header: { kind: "record", id: headerId }, space: space.ref };+ const inputs: Record<string, typeof canvas.ref> = { canvas: canvas.ref, header: { kind: "record", id: headerId }, semantic: semantic.ref };let index = 0;for (const child of element.children) {if (child.kind === "text") { if (child.value.trim()) throw new Error(`${element.name} accepts only component children.`); continue; }@@ -131,9 +127,9 @@ export const decodeScreenOverlaySurface: StructuredSurfaceHandler = ({ element,inputs[specName] = { kind: "record", id: specId };if (temporal.kind === "program") fragmentItems.push({ kind: "program", specName });else {- const mapName = `item-${suffix}-map`; const sourceName = `item-${suffix}-${temporal.kind}`;- inputs[mapName] = temporal.map!.ref; inputs[sourceName] = temporal.source!.ref;- fragmentItems.push({ kind: temporal.kind, specName, mapName, sourceName });+ const sourceName = `item-${suffix}-${temporal.kind}`;+ inputs[sourceName] = temporal.source!.ref;+ fragmentItems.push({ kind: temporal.kind, specName, sourceName });}}if (fragmentItems.length === 0) throw new Error(`${element.name} requires at least one component.`);diff --git a/packages/screen-overlay/test/screen-overlay.test.ts b/packages/screen-overlay/test/screen-overlay.test.tsindex 9512c112..6c14b55b 100644--- a/packages/screen-overlay/test/screen-overlay.test.ts+++ b/packages/screen-overlay/test/screen-overlay.test.ts@@ -1,6 +1,7 @@import assert from "node:assert/strict";import test from "node:test";import { fixtureDigest } from "../../../test/fixture-digest.js";+import { semanticTrackFixture } from "../../../test/semantic-track-fixture.js";import { artifactManifest } from "@hypit/artifact";import { registerTypeValidatorFacets } from "@hypit/component-kit";@@ -10,7 +11,7 @@ import { AuthorFrontendRegistry, compileSourceClosure, resolveCompiledSourceExpoimport { compileHyperframesDocument } from "@hypit/hyperframes";import { mediaManifest } from "@hypit/media";import { narrativeManifest } from "@hypit/narrative";-import { programSpaceDependency, programSpaceManifest, programSpaceTypes, sealProgramSpace } from "@hypit/program-space";+import { programSpaceManifest, sealProgramSpace } from "@hypit/program-space";import type { ModuleManifest } from "@hypit/protocol";import {appendProgramScreenOverlay,@@ -27,7 +28,7 @@ import {sealScreenOverlayItemSpec,} from "@hypit/screen-overlay";import type { ScreenOverlayComponent } from "@hypit/screen-overlay";-import { semanticMapManifest } from "@hypit/semantic-map";+import { semanticTrackDependency, semanticTrackManifest, semanticTrackTypes } from "@hypit/semantic-track";import { speechEvidenceManifest } from "@hypit/speech-evidence";import { speechManifest } from "@hypit/speech";import { sealCanvasSpace, spatialComponent, spatialDependency, spatialManifest, spatialTypes } from "@hypit/spatial";@@ -44,6 +45,7 @@ const space = sealProgramSpace({durationSec: 2, frameRate: { numerator: 30, denominator: 1 },});const header = sealScreenOverlayHeader({ id: "screen" });+const semantic = semanticTrackFixture(space);const components: readonly ScreenOverlayComponent[] = [{ kind: "flash", color: "#ffffff", intensity: 0.9, attackFrames: 2, holdFrames: 3, decayFrames: 5 },{ kind: "color-wash", color: "#2244ff", opacity: 0.2 },@@ -65,7 +67,7 @@ function trackFor(content: ScreenOverlayComponent, stackingOrder = 50) {expansion: { kind: "one" }, stackingOrder,});return renderScreenOverlay(canvas, space, finalizeScreenOverlay(- appendProgramScreenOverlay(createScreenOverlaySet(), header, space, spec), header,+ appendProgramScreenOverlay(createScreenOverlaySet(), header, semantic, spec), header,));}@@ -117,7 +119,7 @@ test("overlay Tracks interleave with peer Tracks only through absolute stacking"test("the package has no lower-composite, sibling Track, backdrop-filter or hidden audio port", () => {const fragment = createScreenOverlayFragment([{ kind: "program", specName: "spec" }]);- assert.deepEqual(fragment.inputs.map((input) => input.name), ["canvas", "header", "space", "spec"]);+ assert.deepEqual(fragment.inputs.map((input) => input.name), ["canvas", "header", "semantic", "spec"]);assert.equal(fragment.exports.some((output) => output.type.name === "AudioTrack"), false);assert.throws(() => sealScreenOverlayItemSpec({id: "blur",@@ -132,13 +134,13 @@ test("the self-described Screen Surface parses into a finite peer-Track graph",const fixtureSurfaceDigest = fixtureDigest("example.screen-inputs/surface@1");const fixtureSurface = {name: "inputs", tag: "Inputs", mode: "structured",- outputs: [spatialTypes.canvas, programSpaceTypes.programSpace],+ outputs: [spatialTypes.canvas, semanticTrackTypes.track],} as const;const fixtureManifest: ModuleManifest = {format: "hypit.module@1",name: fixtureModule.name,version: fixtureModule.version,- dependencies: [spatialDependency, programSpaceDependency],+ dependencies: [spatialDependency, semanticTrackDependency],types: [],capabilities: [],producers: [],@@ -150,7 +152,7 @@ test("the self-described Screen Surface parses into a finite peer-Track graph",programSpaceManifest,speechManifest,speechEvidenceManifest,- semanticMapManifest,+ semanticTrackManifest,spatialManifest,temporalManifest,visualIrManifest,@@ -162,7 +164,7 @@ test("the self-described Screen Surface parses into a finite peer-Track graph",registry.registerStructured({ module: fixtureModule, declaration: fixtureSurface, handler: ({ element }) => ({records: [{ id: "canvas", type: spatialTypes.canvas, value: { kind: "inline", value: canvas }, range: element.range },- { id: "space", type: programSpaceTypes.programSpace, value: { kind: "inline", value: space }, range: element.range },+ { id: "semantic", type: semanticTrackTypes.track, value: { kind: "inline", value: semantic }, range: element.range },],components: [],fragments: [],@@ -188,7 +190,7 @@ test("the self-described Screen Surface parses into a finite peer-Track graph",<import as="fixture" from="example.screen-inputs@1"/><import as="screen" from="@hypit/screen-overlay@1"/><fixture:Inputs/>- <screen:Track id="screen-fx" canvas={canvas} space={space}>+ <screen:Track id="screen-fx" canvas={canvas} semantic={semantic}><screen:Flash during="program" z="70" color="#ffffff" intensity="0.9" attack="2" hold="3" decay="5"/></screen:Track></svml>`,diff --git a/packages/script/README.md b/packages/script/README.mdindex e3ab0068..b5c850b9 100644--- a/packages/script/README.md+++ b/packages/script/README.md@@ -46,7 +46,7 @@ The Surface exports one full Narrative plus narrow, immutable views:- `script.caption.selection.<id>` is the display-word subset wholly owned by one Selection;- `script.selection.<id>` is a reusable explicit Selection.-Seedance consumes dialogue `Text`, Estimate and TTS consume speech `Text`, Speech Spine consumes the+Seedance consumes dialogue `Text`, Estimate and TTS consume speech `Text`, Speech Track consumes theSegment excerpt, and Caption consumes the explicit display and correspondence edges. None importsScript's parser AST. Another authoring package may produce the same ordinary Text and structuredNarrative contracts.diff --git a/packages/semantic-map/README.md b/packages/semantic-map/README.mddeleted file mode 100644index a567270a..00000000--- a/packages/semantic-map/README.md+++ /dev/null@@ -1,3 +0,0 @@-# `@hypit/semantic-map`--The located projection of authored Narrative identity onto ProgramSpace.diff --git a/packages/semantic-map/src/activation.ts b/packages/semantic-map/src/activation.tsdeleted file mode 100644index 0ec4e017..00000000--- a/packages/semantic-map/src/activation.ts+++ /dev/null@@ -1,3 +0,0 @@-import { semanticMapManifest } from "./index.js";-export const hypitPackage = { format: "hypit.node-package@1" as const, modules: [{ manifest: semanticMapManifest }] };-export default hypitPackage;diff --git a/packages/semantic-map/src/index.ts b/packages/semantic-map/src/index.tsdeleted file mode 100644index f20ee15d..00000000--- a/packages/semantic-map/src/index.ts+++ /dev/null@@ -1,11 +0,0 @@-export {- assertCompleteSemanticMapIdentity,- assertNarrativeMomentIdentity,- assertNarrativeSelectionIdentity,- momentFrames,- segmentFrameSpan,- selectionFrameSpans,- tokenFrameSpan,-} from "./location.js";-export { completeSemanticMapSchema, semanticMapDependency, semanticMapManifest, semanticMapModuleRef, semanticMapTypes } from "./manifest.js";-export type * from "./types.js";diff --git a/packages/semantic-map/src/location.ts b/packages/semantic-map/src/location.tsdeleted file mode 100644index 394c106f..00000000--- a/packages/semantic-map/src/location.ts+++ /dev/null@@ -1,122 +0,0 @@-import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrative";-import type { CompleteSemanticMap } from "./types.js";--export type LocatedFrameSpan = { readonly startFrame: number; readonly endFrameExclusive: number };--export function assertCompleteSemanticMapIdentity(map: CompleteSemanticMap): void {- if (!Array.isArray(map.tokens) || !Array.isArray(map.anchors)) throw new Error("CompleteSemanticMap is invalid");-}--export function assertNarrativeSelectionIdentity(selection: NarrativeSelectionRef): void {- if (- selection.id.length === 0- || selection.occurrences.length === 0- ) {- throw new Error("NarrativeSelection is invalid");- }-}--export function assertNarrativeMomentIdentity(moment: NarrativeMomentRef): void {- if (- moment.id.length === 0- || moment.occurrences.length === 0- ) {- throw new Error("NarrativeMoment is invalid");- }-}--/**- * Every marker resolves its affinity to one of the map's `2m + 2n` anchors while the- * Script was parsed, where the surrounding structure was known. Locating is a- * lookup: Token cuts and Segment cuts are equal citizens here.- */-function anchorFrames(map: CompleteSemanticMap): ReadonlyMap<string, number> {- return new Map(map.anchors.map((anchor) => [anchor.identity, anchor.frame]));-}--function frameFor(frames: ReadonlyMap<string, number>, anchorId: string, owner: string): number {- const frame = frames.get(anchorId);- if (frame === undefined) {- throw new Error(`${owner} names anchor ${anchorId}, which this SemanticMap does not contain`);- }- return frame;-}--/**- * Project every occurrence of one Script Selection onto one measured SemanticMap.- *- * One entry per occurrence, in Script order — the order the markers appear in the- * source. Segments may overlap in time, so Script order is not necessarily time- * order, and occurrences are located independently: they are never sorted,- * merged, clipped against one another or otherwise reconciled. A caller that- * needs time order sorts them itself.- *- * An occurrence whose two anchors meet or cross yields an empty or backwards- * span, returned as located. What a Selection covering no frames should show is- * a question about material, answered where that material is projected.- */-export function selectionFrameSpans(- map: CompleteSemanticMap,- selection: NarrativeSelectionRef,-): readonly LocatedFrameSpan[] {- assertCompleteSemanticMapIdentity(map);- assertNarrativeSelectionIdentity(selection);- const frames = anchorFrames(map);- return selection.occurrences.map((occurrence) => {- const startFrame = frameFor(frames, occurrence.startAnchorId, `NarrativeSelection ${selection.id}`);- const endFrameExclusive = frameFor(frames, occurrence.endAnchorId, `NarrativeSelection ${selection.id}`);- return { startFrame, endFrameExclusive };- });-}--/** Locate one authored Segment by its two structural anchors. */-export function segmentFrameSpan(- map: CompleteSemanticMap,- segment: NarrativeExcerpt,-): LocatedFrameSpan {- assertCompleteSemanticMapIdentity(map);- if (segment.kind !== "segment" || segment.id.length === 0) {- throw new Error("Narrative Segment excerpt is invalid");- }- const frames = anchorFrames(map);- return {- startFrame: frameFor(frames, `segment:${segment.id}:start`, `Narrative Segment ${segment.id}`),- endFrameExclusive: frameFor(frames, `segment:${segment.id}:end`, `Narrative Segment ${segment.id}`),- };-}--/**- * The measured window a run of Script tokens occupies: the first token's start- * and the last token's end, exactly as located. Undefined when any named token- * is absent, leaving the caller to decide what that means.- *- * This is the second way the same points are addressed. Markers address them by- * anchor; a component that renders the Script text itself addresses them by- * token, because its runs are computed rather than authored.- */-export function tokenFrameSpan(- map: CompleteSemanticMap,- tokenIds: readonly string[],-): LocatedFrameSpan | undefined {- assertCompleteSemanticMapIdentity(map);- if (tokenIds.length === 0) return undefined;- const timing = new Map(map.tokens.map((token) => [token.tokenId, token]));- const located = tokenIds.map((id) => timing.get(id));- if (located.some((token) => token === undefined)) return undefined;- return {- startFrame: located[0]!.startFrame,- endFrameExclusive: located.at(-1)!.endFrameExclusive,- };-}--/** Project every occurrence of one Script Moment onto one measured SemanticMap. */-export function momentFrames(- map: CompleteSemanticMap,- moment: NarrativeMomentRef,-): readonly number[] {- assertCompleteSemanticMapIdentity(map);- assertNarrativeMomentIdentity(moment);- const frames = anchorFrames(map);- return moment.occurrences.map((occurrence) =>- frameFor(frames, occurrence.anchorId, `NarrativeMoment ${moment.id}`));-}diff --git a/packages/semantic-map/src/manifest.ts b/packages/semantic-map/src/manifest.tsdeleted file mode 100644index eab6306e..00000000--- a/packages/semantic-map/src/manifest.ts+++ /dev/null@@ -1,15 +0,0 @@-import { narrativeDependency } from "@hypit/narrative";-import { speechEvidenceDependency } from "@hypit/speech-evidence";-import type { ModuleManifest, TypeRef, ValueSchema } from "@hypit/protocol";-const string = { kind: "string", minLength: 1 } as const;-const integer = { kind: "number", integer: true, minimum: 0 } as const;-const object = (fields: Readonly<Record<string, { readonly schema: ValueSchema; readonly optional?: boolean }>>): ValueSchema => ({ kind: "object", fields });-const timedToken = object({ tokenId: { schema: string }, segmentId: { schema: string }, startFrame: { schema: integer }, endFrameExclusive: { schema: integer } });-const point = object({ identity: { schema: string }, frame: { schema: integer } });-export const completeSemanticMapSchema: ValueSchema = object({- tokens: { schema: { kind: "array", items: timedToken } }, anchors: { schema: { kind: "array", items: point } } });-export const semanticMapModuleRef = { name: "@hypit/semantic-map", version: "1" } as const;-export const semanticMapTypes = { complete: { module: semanticMapModuleRef, name: "CompleteSemanticMap" } } satisfies Record<string, TypeRef>;-export const semanticMapManifest: ModuleManifest = { format: "hypit.module@1", name: semanticMapModuleRef.name, version: semanticMapModuleRef.version,- dependencies: [narrativeDependency, speechEvidenceDependency], types: [{ name: semanticMapTypes.complete.name }], capabilities: [], producers: [] };-export const semanticMapDependency = { module: semanticMapModuleRef } as const;diff --git a/packages/semantic-map/src/types.ts b/packages/semantic-map/src/types.tsdeleted file mode 100644index 56602740..00000000--- a/packages/semantic-map/src/types.ts+++ /dev/null@@ -1,17 +0,0 @@-export type TimedSpeechToken = {- readonly tokenId: string;- readonly segmentId: string;- readonly startFrame: number;- readonly endFrameExclusive: number;-};-export type SemanticTimePoint = { readonly identity: string; readonly frame: number };-/**- * The point map every marker addresses, plus the Segment and Token ranges- * consumers read windows from. For `m` Segments and `n` Tokens this contains- * `2m + 2n` structural boundaries; one Segment with `k` Tokens therefore has- * `2 + 2k` anchors. Anchors are the addressing truth.- */-export type CompleteSemanticMap = {- readonly tokens: readonly TimedSpeechToken[];- readonly anchors: readonly SemanticTimePoint[];-};diff --git a/packages/semantic-map/test/location.test.ts b/packages/semantic-map/test/location.test.tsdeleted file mode 100644index 97eaac86..00000000--- a/packages/semantic-map/test/location.test.ts+++ /dev/null@@ -1,152 +0,0 @@-import assert from "node:assert/strict";-import test from "node:test";--import { momentFrames, segmentFrameSpan, selectionFrameSpans } from "@hypit/semantic-map";-import type { CompleteSemanticMap } from "@hypit/semantic-map";-import type { NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrative";--/**- * Two Segments at 10 fps. Words are one frame wide with a one-frame gap, so a- * boundary that snaps to the wrong side of a gap is visible as a whole frame,- * and the two Segments are separated by a further frame of silence.- *- * opening: hello[0..1) end[2..3) segment 0..4- * second: start[5..6) here[7..8) segment 5..9- */-const anchor = (identity: string, frame: number) => ({ identity, frame });--const map = {- tokens: [],- anchors: [- anchor("segment:opening:start", 0),- anchor("segment:opening:token:1:start", 0),- anchor("segment:opening:token:1:end", 1),- anchor("segment:opening:token:2:start", 2),- anchor("segment:opening:token:2:end", 3),- anchor("segment:opening:end", 4),- anchor("segment:second:start", 5),- anchor("segment:second:token:1:start", 5),- anchor("segment:second:token:1:end", 6),- anchor("segment:second:token:2:start", 7),- anchor("segment:second:token:2:end", 8),- anchor("segment:second:end", 9),- ],-} as unknown as CompleteSemanticMap;--function selection(openAnchor: string, closeAnchor: string): NarrativeSelectionRef {- return {-- id: "x",- occurrences: [{- occurrence: 0,- startAnchorId: openAnchor,- endAnchorId: closeAnchor,- }],- } as NarrativeSelectionRef;-}--test("the default inward markers select exactly the marked word", () => {- // `@x end @/x` resolved at parse time to token 2's own cuts.- assert.deepEqual(- selectionFrameSpans(map, selection("segment:opening:token:2:start", "segment:opening:token:2:end")),- [{ startFrame: 2, endFrameExclusive: 3 }],- );-});--test("outward markers absorb the surrounding silence, not the neighbouring word", () => {- // `hello ~@x end @/x~ …` resolves to hello's END and the next word's START.- assert.deepEqual(- selectionFrameSpans(map, selection("segment:opening:token:1:end", "segment:opening:token:2:start")),- [{ startFrame: 1, endFrameExclusive: 2 }],- );-});--test("a Segment cut is an ordinary anchor, so a Selection may start at one", () => {- // `</opening><second> ~@x start …` — the left boundary is `second`'s own start- // cut, five frames after `opening` ended, never `opening`'s last word.- assert.deepEqual(- selectionFrameSpans(map, selection("segment:second:start", "segment:second:token:1:end")),- [{ startFrame: 5, endFrameExclusive: 6 }],- );- assert.deepEqual(- selectionFrameSpans(map, selection("segment:second:token:2:start", "segment:second:end")),- [{ startFrame: 7, endFrameExclusive: 9 }],- );-});--test("a whole Segment is exactly its two structural anchors", () => {- assert.deepEqual(segmentFrameSpan(map, {- kind: "segment", id: "second", tokenStart: 2, tokenEndExclusive: 4,- }), { startFrame: 5, endFrameExclusive: 9 });-});--test("a Moment locates one instant per occurrence", () => {- const moment = {-- id: "reveal",- occurrences: [- { occurrence: 0, anchorId: "segment:second:start" },- { occurrence: 1, anchorId: "segment:second:token:2:start" },- ],- } as NarrativeMomentRef;- assert.deepEqual(momentFrames(map, moment), [5, 7]);-});--test("an anchor the map does not contain names the Selection that asked for it", () => {- assert.throws(- () => selectionFrameSpans(map, selection("segment:missing:start", "segment:second:end")),- /NarrativeSelection x names anchor segment:missing:start/u,- );-});--test("overlapping occurrences pass through untouched, in Script order", () => {- // `@beat one @/beat … @beat three @/beat` where the two Segments overlap in- // time. Locating never sorts, merges or clips occurrences against each other:- // reconciling them is the caller's decision about its own material.- const overlapping = {-- id: "beat",- occurrences: [- {- occurrence: 0,- startAnchorId: "segment:opening:token:1:start",- endAnchorId: "segment:opening:end",- },- {- occurrence: 1,- startAnchorId: "segment:second:token:1:end",- endAnchorId: "segment:second:end",- },- ],- } as NarrativeSelectionRef;- // opening:token1:start = 0, opening:end = 4, second:token1:end = 6, second:end = 9- assert.deepEqual(selectionFrameSpans(map, overlapping), [- { startFrame: 0, endFrameExclusive: 4 },- { startFrame: 6, endFrameExclusive: 9 },- ]);-});--test("Script order is preserved even when it runs backwards in time", () => {- // Overlapping Segments make the second occurrence start before the first ends.- // The array still follows the Script; a caller that needs time order sorts.- const crossing = {-- id: "beat",- occurrences: [- {- occurrence: 0,- startAnchorId: "segment:second:token:1:start",- endAnchorId: "segment:second:end",- },- {- occurrence: 1,- startAnchorId: "segment:opening:token:1:start",- endAnchorId: "segment:opening:end",- },- ],- } as NarrativeSelectionRef;- assert.deepEqual(selectionFrameSpans(map, crossing), [- { startFrame: 5, endFrameExclusive: 9 },- { startFrame: 0, endFrameExclusive: 4 },- ]);-});diff --git a/packages/semantic-track/README.md b/packages/semantic-track/README.mdnew file mode 100644index 00000000..1dedad05--- /dev/null+++ b/packages/semantic-track/README.md@@ -0,0 +1,11 @@+# @hypit/semantic-track++`SemanticTrack` is the single global semantic-timeline fact. It is the continuous program skeleton:+every item is a real segment-local `SemanticTake`, and global frame positions are the prefix sum of+their normalized local frame counts. There are no gap items or empty items. A Take may carry only+audio or only visual media; only its word array may be empty. Visual placement is deliberately not+part of this semantic truth.++The package exposes pure projections to `ProgramSpace`, `VisualTrack` and `AudioTrack`, plus the+selection/segment/moment lookup functions used by semantic components. It performs no media I/O,+mixing, transcription or rendering.diff --git a/packages/semantic-map/package.json b/packages/semantic-track/package.jsonsimilarity index 57%rename from packages/semantic-map/package.jsonrename to packages/semantic-track/package.jsonindex f36367dd..b59dcf44 100644--- a/packages/semantic-map/package.json+++ b/packages/semantic-track/package.json@@ -1,5 +1,5 @@{- "name": "@hypit/semantic-map",+ "name": "@hypit/semantic-track","version": "0.0.0-dev","license": "SEE LICENSE IN LICENSE","private": true,@@ -11,8 +11,12 @@"activation": "./src/activation.ts"},"dependencies": {+ "@hypit/component-kit": "workspace:*",+ "@hypit/composition": "workspace:*",+ "@hypit/media": "workspace:*","@hypit/narrative": "workspace:*",+ "@hypit/program-space": "workspace:*","@hypit/protocol": "workspace:*",- "@hypit/speech-evidence": "workspace:*"+ "@hypit/speech": "workspace:*"}}diff --git a/packages/semantic-track/src/activation.ts b/packages/semantic-track/src/activation.tsnew file mode 100644index 00000000..b2ae4dc4--- /dev/null+++ b/packages/semantic-track/src/activation.ts@@ -0,0 +1,8 @@+import { semanticTrackComponent, semanticTrackManifest } from "./index.js";++export const hypitPackage = {+ format: "hypit.node-package@1" as const,+ modules: [{ manifest: semanticTrackManifest }],+ components: [semanticTrackComponent],+};+export default hypitPackage;diff --git a/packages/semantic-track/src/component.ts b/packages/semantic-track/src/component.tsnew file mode 100644index 00000000..8428b413--- /dev/null+++ b/packages/semantic-track/src/component.ts@@ -0,0 +1,19 @@+import type { ComponentPackage } from "@hypit/component-kit";+import type { StoredValue } from "@hypit/protocol";+import { canonicalize } from "@hypit/protocol";++import { semanticTrackProducers } from "./manifest.js";+import { projectSemanticAudioTrack, projectSemanticProgramSpace } from "./projection.js";+import type { SemanticTrack } from "./types.js";++function track(value: StoredValue | undefined): SemanticTrack {+ if (value?.kind !== "inline") throw new Error("SemanticTrack must be inline.");+ return value.value as unknown as SemanticTrack;+}++export const semanticTrackComponent = {+ producers: [+ { producer: semanticTrackProducers.projectProgramSpace, handler: ({ inputs }) => ({ outputs: { space: { kind: "inline", value: canonicalize(projectSemanticProgramSpace(track(inputs.track?.value))) } }, needs: {} }) },+ { producer: semanticTrackProducers.projectAudio, handler: ({ inputs }) => ({ outputs: { audio: { kind: "inline", value: canonicalize(projectSemanticAudioTrack(track(inputs.track?.value))) } }, needs: {} }) },+ ],+} satisfies ComponentPackage;diff --git a/packages/semantic-track/src/identity.ts b/packages/semantic-track/src/identity.tsnew file mode 100644index 00000000..41c6562b--- /dev/null+++ b/packages/semantic-track/src/identity.ts@@ -0,0 +1,50 @@+import { assertSemanticTakeIdentity } from "@hypit/speech";++import type { SemanticTrack, SemanticTrackSpan } from "./types.js";++export function sealSemanticTrack(value: SemanticTrack): SemanticTrack {+ const track = structuredClone(value);+ assertSemanticTrackIdentity(track);+ return track;+}++export function assertSemanticTrackIdentity(track: SemanticTrack): void {+ if (!track.id.trim() || track.items.length === 0) throw new Error("SemanticTrack must contain at least one item.");+ const segmentIds = new Set<string>();+ const tokenIds = new Set<string>();+ const anchorIds = new Set<string>();+ let frameRate: SemanticTrack["items"][number]["take"]["media"]["timeline"]["frameRate"] | undefined;+ for (const item of track.items) {+ assertSemanticTakeIdentity(item.take);+ const rate = item.take.media.timeline.frameRate;+ if (frameRate === undefined) frameRate = rate;+ else if (frameRate.numerator !== rate.numerator || frameRate.denominator !== rate.denominator) {+ throw new Error("SemanticTrack items must use one frame rate.");+ }+ const segmentId = item.take.segment.segmentId;+ if (segmentIds.has(segmentId)) throw new Error(`SemanticTrack repeats Segment ${segmentId}.`);+ segmentIds.add(segmentId);+ for (const token of item.take.tokens) {+ if (tokenIds.has(token.tokenId)) throw new Error(`SemanticTrack repeats Token ${token.tokenId}.`);+ tokenIds.add(token.tokenId);+ }+ for (const anchor of item.take.anchors) {+ if (anchorIds.has(anchor.identity)) throw new Error(`SemanticTrack repeats Anchor ${anchor.identity}.`);+ anchorIds.add(anchor.identity);+ }+ }+}++export function semanticTrackSpans(track: SemanticTrack): readonly SemanticTrackSpan[] {+ assertSemanticTrackIdentity(track);+ let frame = 0;+ return track.items.map((item) => {+ const startFrame = frame;+ frame += item.take.media.timeline.frameCount;+ return { item, startFrame, endFrameExclusive: frame };+ });+}++export function semanticTrackFrameCount(track: SemanticTrack): number {+ return semanticTrackSpans(track).at(-1)!.endFrameExclusive;+}diff --git a/packages/semantic-track/src/index.ts b/packages/semantic-track/src/index.tsnew file mode 100644index 00000000..51f8ca60--- /dev/null+++ b/packages/semantic-track/src/index.ts@@ -0,0 +1,6 @@+export { semanticTrackComponent } from "./component.js";+export { assertSemanticTrackIdentity, sealSemanticTrack, semanticTrackFrameCount, semanticTrackSpans } from "./identity.js";+export { assertNarrativeMomentIdentity, assertNarrativeSelectionIdentity, momentFrames, segmentFrameSpan, selectionFrameSpans, tokenFrameSpan } from "./location.js";+export { semanticTrackDependency, semanticTrackManifest, semanticTrackModuleRef, semanticTrackProducers, semanticTrackSchema, semanticTrackTypes } from "./manifest.js";+export { projectSemanticAudioTrack, projectSemanticProgramSpace } from "./projection.js";+export type * from "./types.js";diff --git a/packages/semantic-track/src/location.ts b/packages/semantic-track/src/location.tsnew file mode 100644index 00000000..7aac9869--- /dev/null+++ b/packages/semantic-track/src/location.ts@@ -0,0 +1,68 @@+import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrative";++import { assertSemanticTrackIdentity, semanticTrackSpans } from "./identity.js";+import type { LocatedFrameSpan, SemanticTrack } from "./types.js";++export function assertNarrativeSelectionIdentity(selection: NarrativeSelectionRef): void {+ if (!selection.id || selection.occurrences.length === 0) throw new Error("NarrativeSelection is invalid");+}++export function assertNarrativeMomentIdentity(moment: NarrativeMomentRef): void {+ if (!moment.id || moment.occurrences.length === 0) throw new Error("NarrativeMoment is invalid");+}++function anchorFrames(track: SemanticTrack): ReadonlyMap<string, number> {+ return new Map(semanticTrackSpans(track).flatMap(({ item, startFrame }) =>+ item.take.anchors.map((anchor) => [anchor.identity, startFrame + anchor.frame] as const)));+}++function frameFor(frames: ReadonlyMap<string, number>, anchorId: string, owner: string): number {+ const frame = frames.get(anchorId);+ if (frame === undefined) throw new Error(`${owner} names Anchor ${anchorId}, which this SemanticTrack does not contain.`);+ return frame;+}++export function selectionFrameSpans(+ track: SemanticTrack,+ selection: NarrativeSelectionRef,+): readonly LocatedFrameSpan[] {+ assertSemanticTrackIdentity(track);+ assertNarrativeSelectionIdentity(selection);+ const frames = anchorFrames(track);+ return selection.occurrences.map((occurrence) => ({+ startFrame: frameFor(frames, occurrence.startAnchorId, `NarrativeSelection ${selection.id}`),+ endFrameExclusive: frameFor(frames, occurrence.endAnchorId, `NarrativeSelection ${selection.id}`),+ }));+}++export function segmentFrameSpan(track: SemanticTrack, segment: NarrativeExcerpt): LocatedFrameSpan {+ assertSemanticTrackIdentity(track);+ if (segment.kind !== "segment" || !segment.id) throw new Error("Narrative Segment excerpt is invalid");+ const found = semanticTrackSpans(track).find(({ item }) => item.take.segment.segmentId === segment.id);+ if (found === undefined) throw new Error(`SemanticTrack does not contain Segment ${segment.id}.`);+ return { startFrame: found.startFrame, endFrameExclusive: found.endFrameExclusive };+}++export function tokenFrameSpan(track: SemanticTrack, tokenIds: readonly string[]): LocatedFrameSpan | undefined {+ assertSemanticTrackIdentity(track);+ if (tokenIds.length === 0) return undefined;+ const timing = new Map(semanticTrackSpans(track).flatMap(({ item, startFrame }) =>+ item.take.tokens.map((token) => [token.tokenId, {+ startFrame: startFrame + token.startFrame,+ endFrameExclusive: startFrame + token.endFrameExclusive,+ }] as const)));+ const located = tokenIds.map((id) => timing.get(id));+ if (located.some((item) => item === undefined)) return undefined;+ return {+ startFrame: located[0]!.startFrame,+ endFrameExclusive: located.at(-1)!.endFrameExclusive,+ };+}++export function momentFrames(track: SemanticTrack, moment: NarrativeMomentRef): readonly number[] {+ assertSemanticTrackIdentity(track);+ assertNarrativeMomentIdentity(moment);+ const frames = anchorFrames(track);+ return moment.occurrences.map((occurrence) =>+ frameFor(frames, occurrence.anchorId, `NarrativeMoment ${moment.id}`));+}diff --git a/packages/semantic-track/src/manifest.ts b/packages/semantic-track/src/manifest.tsnew file mode 100644index 00000000..6dd81b6c--- /dev/null+++ b/packages/semantic-track/src/manifest.ts@@ -0,0 +1,37 @@+import { compositionDependency, compositionTypes } from "@hypit/composition";+import { narrativeDependency } from "@hypit/narrative";+import { programSpaceDependency, programSpaceTypes } from "@hypit/program-space";+import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@hypit/protocol";+import { semanticTakeSchema, speechDependency } from "@hypit/speech";++export const semanticTrackModuleRef = { name: "@hypit/semantic-track", version: "1" } as const;+export const semanticTrackTypes = {+ track: { module: semanticTrackModuleRef, name: "SemanticTrack" },+} satisfies Record<string, TypeRef>;+export const semanticTrackProducers = {+ projectProgramSpace: { module: semanticTrackModuleRef, name: "project-program-space" },+ projectAudio: { module: semanticTrackModuleRef, name: "project-audio-track" },+} satisfies Record<string, ProducerRef>;++const string = { kind: "string", minLength: 1 } as const;+const object = (fields: Readonly<Record<string, { readonly schema: ValueSchema; readonly optional?: boolean }>>): ValueSchema => ({ kind: "object", fields });+export const semanticTrackSchema: ValueSchema = object({+ id: { schema: string },+ items: { schema: { kind: "array", minItems: 1, items: object({+ take: { schema: semanticTakeSchema },+ }) } },+});++export const semanticTrackManifest: ModuleManifest = {+ format: "hypit.module@1",+ name: semanticTrackModuleRef.name,+ version: semanticTrackModuleRef.version,+ dependencies: [speechDependency, narrativeDependency, programSpaceDependency, compositionDependency],+ types: [{ name: semanticTrackTypes.track.name }],+ capabilities: [],+ producers: [+ { name: semanticTrackProducers.projectProgramSpace.name, inputs: [{ name: "track", type: semanticTrackTypes.track }], outputs: [{ name: "space", type: programSpaceTypes.programSpace }], needs: [] },+ { name: semanticTrackProducers.projectAudio.name, inputs: [{ name: "track", type: semanticTrackTypes.track }], outputs: [{ name: "audio", type: compositionTypes.audioTrack }], needs: [] },+ ],+};+export const semanticTrackDependency = { module: semanticTrackModuleRef } as const;diff --git a/packages/semantic-track/src/projection.ts b/packages/semantic-track/src/projection.tsnew file mode 100644index 00000000..a88f564a--- /dev/null+++ b/packages/semantic-track/src/projection.ts@@ -0,0 +1,49 @@+import { sealAudioTrack } from "@hypit/composition";+import type { AudioTrack } from "@hypit/composition";+import { synchronizedMediaSampleFrames } from "@hypit/media";+import {+ programFrameSampleBoundary,+ sealProgramSpace,+} from "@hypit/program-space";+import type { ProgramSpace } from "@hypit/program-space";++import { assertSemanticTrackIdentity, semanticTrackFrameCount, semanticTrackSpans } from "./identity.js";+import type { SemanticTrack } from "./types.js";++export function projectSemanticProgramSpace(track: SemanticTrack): ProgramSpace {+ assertSemanticTrackIdentity(track);+ const frameRate = track.items[0]!.take.media.timeline.frameRate;+ return sealProgramSpace({+ durationSec: semanticTrackFrameCount(track) * frameRate.denominator / frameRate.numerator,+ frameRate,+ });+}++export function projectSemanticAudioTrack(track: SemanticTrack): AudioTrack {+ const space = projectSemanticProgramSpace(track);+ const clips = semanticTrackSpans(track).flatMap(({ item, startFrame, endFrameExclusive }) => {+ const audio = item.take.media.audio;+ if (audio === undefined) return [];+ const sourceSampleFrames = synchronizedMediaSampleFrames(item.take.media);+ const targetStartSample = programFrameSampleBoundary(space, startFrame, 48_000);+ const targetEndSampleExclusive = programFrameSampleBoundary(space, endFrameExclusive, 48_000);+ return [{+ id: item.take.segment.segmentId,+ artifact: audio.artifact,+ target: { startSample: targetStartSample, endSampleExclusive: targetEndSampleExclusive },+ source: {+ sampleFrames: sourceSampleFrames,+ startSample: 0,+ endSampleExclusive: sourceSampleFrames,+ loop: false,+ phaseSample: 0,+ },+ playbackRate: 1,+ pitch: "preserve" as const,+ gain: 1,+ fadeInSamples: 0,+ fadeOutSamples: 0,+ }];+ });+ return sealAudioTrack({ id: `${track.id}:audio`, clips });+}diff --git a/packages/semantic-track/src/types.ts b/packages/semantic-track/src/types.tsnew file mode 100644index 00000000..8398a3e4--- /dev/null+++ b/packages/semantic-track/src/types.ts@@ -0,0 +1,25 @@+import type { SemanticTake } from "@hypit/speech";++export type SemanticTrackItem = {+ readonly take: SemanticTake;+};++/**+ * The continuous program skeleton. Every item is a real SemanticTake and the+ * global frame domain is exactly their prefix sum: no gap or empty item exists.+ */+export type SemanticTrack = {+ readonly id: string;+ readonly items: readonly SemanticTrackItem[];+};++export type SemanticTrackSpan = {+ readonly item: SemanticTrackItem;+ readonly startFrame: number;+ readonly endFrameExclusive: number;+};++export type LocatedFrameSpan = {+ readonly startFrame: number;+ readonly endFrameExclusive: number;+};diff --git a/packages/speech-alignment/package.json b/packages/speech-alignment/package.jsonindex fb987db0..41d4f76f 100644--- a/packages/speech-alignment/package.json+++ b/packages/speech-alignment/package.json@@ -16,7 +16,6 @@"@hypit/narrative": "workspace:*","@hypit/program-space": "workspace:*","@hypit/protocol": "workspace:*",- "@hypit/semantic-map": "workspace:*","@hypit/speech": "workspace:*","@hypit/speech-evidence": "workspace:*"},diff --git a/packages/speech-alignment/src/local.ts b/packages/speech-alignment/src/local.tsindex d8dd8bc0..f40a839f 100644--- a/packages/speech-alignment/src/local.ts+++ b/packages/speech-alignment/src/local.ts@@ -53,10 +53,10 @@ export function alignSemanticTake(audio: media.audio.artifact,segments: [{ segmentId: excerpt.id, startFrame: 0, endFrameExclusive: media.timeline.frameCount }],};- const map = locateAlignedSegmentTiming(localNarrative, basis, evidence);+ const timing = locateAlignedSegmentTiming(localNarrative, basis, evidence);return materializeSemanticTake(localNarrative, {...excerpt,tokenStart: 0,tokenEndExclusive: localNarrative.tokens.length,- }, media, map);+ }, media, timing);}diff --git a/packages/speech-alignment/src/locate.ts b/packages/speech-alignment/src/locate.tsindex 4311b35b..d9c3e5fc 100644--- a/packages/speech-alignment/src/locate.ts+++ b/packages/speech-alignment/src/locate.ts@@ -10,13 +10,27 @@ import type {SpeechTranscriptPassage,SpeechWordEvidence,} from "@hypit/speech-evidence";-import type { CompleteSemanticMap, SemanticTimePoint, TimedSpeechToken } from "@hypit/semantic-map";import { alignWordGroups } from "./align.js";import { SpeechAlignmentError } from "./error.js";import { alignCharacters, alignmentCharacters } from "./normalize.js";import type { AlignmentGroup, TimedSpeechSegment } from "./types.js";+export type LocalTimedSpeechToken = {+ readonly tokenId: string;+ readonly segmentId: string;+ readonly startFrame: number;+ readonly endFrameExclusive: number;+};++export type LocalSemanticTimePoint = { readonly identity: string; readonly frame: number };++/** Private alignment result used only to materialize one SemanticTake. */+export type LocalSemanticTiming = {+ readonly tokens: readonly LocalTimedSpeechToken[];+ readonly anchors: readonly LocalSemanticTimePoint[];+};+/** Package-private clock used while aligning exactly one normalized Segment Take. */export type AlignmentBasis = {readonly programSpace: ProgramSpace;@@ -51,14 +65,16 @@ function fail(code: string, message: string): never {throw new SpeechAlignmentError(code, message);}-function sampleWindowPresent(value: {+function positiveSampleWindow(value: {readonly startSample?: number;readonly endSampleExclusive?: number;}): value is { readonly startSample: number; readonly endSampleExclusive: number } {- return value.startSample !== undefined && value.endSampleExclusive !== undefined;+ return value.startSample !== undefined+ && value.endSampleExclusive !== undefined+ && value.endSampleExclusive > value.startSample;}-/** Provider measurements may overlap or run backwards; only unusable sample coordinates are rejected. */+/** Raw provider measurements may overlap, collapse or run backwards; only unusable coordinates are rejected. */function validateSampleWindow(value: { readonly startSample?: number; readonly endSampleExclusive?: number },limit: number,@@ -92,7 +108,7 @@ function validateBasis(narrative: Narrative, basis: AlignmentBasis): void {const frameCount = programSpaceFrameCount(basis.programSpace);if (basis.audio.kind !== "blob" || !isDigest(basis.audio.digest)|| basis.audio.mediaType !== "audio/wav" || !Number.isSafeInteger(basis.audio.size) || basis.audio.size < 0) {- fail("SPEECH_AUDIO_DIGEST", "SpeechBasis audio BlobRef is invalid.");+ fail("SPEECH_AUDIO_DIGEST", "Speech alignment audio BlobRef is invalid.");}if (narrative.segments.length !== 1 || basis.segments.length !== 1) {fail("SPEECH_BASIS_SEGMENTS", "Speech alignment accepts exactly one normalized Segment Take.");@@ -139,7 +155,7 @@ function sampleMidpoint(value: {readonly startSample?: number;readonly endSampleExclusive?: number;}): number | undefined {- return sampleWindowPresent(value)+ return positiveSampleWindow(value)? (value.startSample + value.endSampleExclusive) / 2: undefined;}@@ -194,7 +210,7 @@ function syntheticCharTimes(wordIndex: number,word: SpeechWordEvidence,): TimedEvidenceChar[] {- const hasWindow = sampleWindowPresent(word);+ const hasWindow = positiveSampleWindow(word);return characters.map((value, index) => ({value,wordIndex,@@ -224,7 +240,7 @@ function timedCharsForWord(.flatMap((char) => alignmentCharacters(char.char).map((value) => ({value,wordIndex,- ...(sampleWindowPresent(char)+ ...(positiveSampleWindow(char)? { startSample: char.startSample, endSampleExclusive: char.endSampleExclusive }: {}),})));@@ -234,7 +250,7 @@ function timedCharsForWord(const pairs = alignCharacters(expected, supplied.map((char) => char.value));for (const pair of pairs) {const suppliedChar = supplied[pair.evidenceIndex]!;- if (!sampleWindowPresent(suppliedChar)) continue;+ if (!positiveSampleWindow(suppliedChar)) continue;fallback[pair.sourceIndex] = {value: expected[pair.sourceIndex]!,wordIndex,@@ -271,16 +287,14 @@ function groupWindow(const wordRun = words.slice(group.evidenceWordStart, group.evidenceWordEndExclusive);const charRun = chars.filter((char) =>char.wordIndex >= group.evidenceWordStart && char.wordIndex < group.evidenceWordEndExclusive);- const starts = [- ...wordRun.map((word) => word.startSample),- ...charRun.map((char) => char.startSample),- ].filter((value): value is number => value !== undefined);- const ends = [- ...wordRun.map((word) => word.endSampleExclusive),- ...charRun.map((char) => char.endSampleExclusive),- ].filter((value): value is number => value !== undefined);- if (!starts.length || !ends.length) return undefined;- return { startSample: Math.min(...starts), endSampleExclusive: Math.max(...ends) };+ const windows = [...wordRun, ...charRun].flatMap((item) => positiveSampleWindow(item)+ ? [{ startSample: item.startSample, endSampleExclusive: item.endSampleExclusive }]+ : []);+ if (!windows.length) return undefined;+ return {+ startSample: Math.min(...windows.map((item) => item.startSample)),+ endSampleExclusive: Math.max(...windows.map((item) => item.endSampleExclusive)),+ };}function locatePairedGroup(@@ -307,7 +321,7 @@ function locatePairedGroup(const evidenceForToken = pairs.filter((pair) => sourceCharacters[pair.sourceIndex]!.tokenOffset === tokenOffset).map((pair) => evidenceCharacters[pair.evidenceIndex]!)- .filter(sampleWindowPresent);+ .filter(positiveSampleWindow);let startSample = evidenceForToken[0]?.startSample;let endSampleExclusive = evidenceForToken.at(-1)?.endSampleExclusive;if (sourceRun.length === 1 && window) {@@ -319,7 +333,8 @@ function locatePairedGroup(endSampleExclusive = window.endSampleExclusive;}}- if (startSample === undefined || endSampleExclusive === undefined) continue;+ if (startSample === undefined || endSampleExclusive === undefined+ || endSampleExclusive <= startSample) continue;const sourceIndex = sourceIndexById.get(sourceRun[tokenOffset]!.id)!;output[sourceIndex] = { startSample, endSampleExclusive };}@@ -330,9 +345,12 @@ function speechBounds(segment: SegmentEvidence,basisSegment: AlignmentBasis["segments"][number],): { readonly start: number; readonly end: number } {- const spans = segment.speechActivity;+ const spans = segment.speechActivity.filter(positiveSampleWindow);return spans.length- ? { start: spans[0]!.startSample, end: spans.at(-1)!.endSampleExclusive }+ ? {+ start: Math.min(...spans.map((span) => span.startSample)),+ end: Math.max(...spans.map((span) => span.endSampleExclusive)),+ }: {start: programFrameSampleBoundary(basis.programSpace, basisSegment.startFrame, 16_000),end: programFrameSampleBoundary(basis.programSpace, basisSegment.endFrameExclusive, 16_000),@@ -373,26 +391,51 @@ function fillMissingTiming(return values as MutableTiming[];}-function frameForEvidenceSample(basis: AlignmentBasis, sample: number): number {+function floorFrameForEvidenceSample(basis: AlignmentBasis, sample: number): number {const numerator = BigInt(sample) * BigInt(basis.programSpace.frameRate.numerator);const denominator = 16_000n * BigInt(basis.programSpace.frameRate.denominator);- const frame = (numerator * 2n + denominator) / (denominator * 2n);+ const frame = numerator / denominator;if (frame > BigInt(Number.MAX_SAFE_INTEGER)) fail("SPEECH_FRAME", "Speech evidence exceeds the frame domain.");return Number(frame);}+function ceilFrameForEvidenceSample(basis: AlignmentBasis, sample: number): number {+ const numerator = BigInt(sample) * BigInt(basis.programSpace.frameRate.numerator);+ const denominator = 16_000n * BigInt(basis.programSpace.frameRate.denominator);+ const frame = (numerator + denominator - 1n) / denominator;+ if (frame > BigInt(Number.MAX_SAFE_INTEGER)) fail("SPEECH_FRAME", "Speech evidence exceeds the frame domain.");+ return Number(frame);+}++/** Project one completed sample timing onto every local video frame it touches. */+function projectSampleTiming(+ basis: AlignmentBasis,+ timing: MutableTiming,+): { readonly startFrame: number; readonly endFrameExclusive: number } {+ const frameCount = programSpaceFrameCount(basis.programSpace);+ const startFrame = Math.min(frameCount, floorFrameForEvidenceSample(basis, timing.startSample));+ const endFrameExclusive = Math.min(+ frameCount,+ ceilFrameForEvidenceSample(basis, timing.endSampleExclusive),+ );+ if (endFrameExclusive > startFrame) return { startFrame, endFrameExclusive };++ const containingFrame = Math.min(frameCount - 1, startFrame);+ return { startFrame: containingFrame, endFrameExclusive: containingFrame + 1 };+}+/** Locate every Script token from one provider-neutral acoustic evidence pass. */export function locateAlignedSegmentTiming(narrative: Narrative,basis: AlignmentBasis,evidence: AlignedTranscriptEvidence,-): CompleteSemanticMap {+): LocalSemanticTiming {validateBasis(narrative, basis);validateEvidence(basis, evidence);const evidenceBySegment = new Map(partitionEvidence(basis, evidence).map((segment) => [segment.sourceSegmentId, segment]));const timedSegments: TimedSpeechSegment[] = [];- const timedTokens: TimedSpeechToken[] = [];+ const timedTokens: LocalTimedSpeechToken[] = [];for (const segment of narrative.segments) {const aligned = evidenceBySegment.get(segment.id)!;@@ -404,13 +447,13 @@ export function locateAlignedSegmentTiming(const basisSegment = basis.segments.find((item) => item.segmentId === segment.id)!;const bounds = speechBounds(basis, aligned, basisSegment);const complete = fillMissingTiming(located, source, bounds.start, bounds.end);- timedTokens.push(...source.map((token, index): TimedSpeechToken => {+ timedTokens.push(...source.map((token, index): LocalTimedSpeechToken => {const timing = complete[index]!;+ const projected = projectSampleTiming(basis, timing);return {tokenId: token.id,segmentId: segment.id,- startFrame: frameForEvidenceSample(basis, timing.startSample),- endFrameExclusive: frameForEvidenceSample(basis, timing.endSampleExclusive),+ ...projected,};}));timedSegments.push({@@ -422,7 +465,7 @@ export function locateAlignedSegmentTiming(const tokensById = new Map(timedTokens.map((token) => [token.tokenId, token]));const segmentsById = new Map(timedSegments.map((segment) => [segment.segmentId, segment]));- const anchors: SemanticTimePoint[] = narrative.semanticIndex.anchors.map((anchor) => {+ const anchors: LocalSemanticTimePoint[] = narrative.semanticIndex.anchors.map((anchor) => {if (anchor.kind === "segment-start" || anchor.kind === "segment-end") {const segment = segmentsById.get(anchor.segmentId)!;return anchor.kind === "segment-start"diff --git a/packages/speech-alignment/src/materialize.ts b/packages/speech-alignment/src/materialize.tsindex 1249a602..330cee83 100644--- a/packages/speech-alignment/src/materialize.ts+++ b/packages/speech-alignment/src/materialize.ts@@ -1,9 +1,10 @@import type { Narrative, NarrativeExcerpt } from "@hypit/narrative";import type { SynchronizedMedia } from "@hypit/media";-import type { CompleteSemanticMap } from "@hypit/semantic-map";import type { SemanticTake } from "@hypit/speech";import { assertSemanticTakeIdentity, sealSemanticTake } from "@hypit/speech";+import type { LocalSemanticTiming } from "./locate.js";+function authoredSegment(narrative: Narrative, excerpt: NarrativeExcerpt): Narrative["segments"][number] {if (excerpt.kind !== "segment") throw new Error("SemanticTake materialization requires a Segment excerpt.");const segment = narrative.segments.find((candidate) => candidate.id === excerpt.id);@@ -28,11 +29,11 @@ export function materializeSemanticTake(narrative: Narrative,excerpt: NarrativeExcerpt,media: SynchronizedMedia,- map: CompleteSemanticMap,+ timing: LocalSemanticTiming,): SemanticTake {const segment = authoredSegment(narrative, excerpt);const localFrameCount = media.timeline.frameCount;- const timedById = new Map(map.tokens.map((token) => [token.tokenId, token]));+ const timedById = new Map(timing.tokens.map((token) => [token.tokenId, token]));const tokens = narrative.tokens.slice(segment.tokenStart, segment.tokenEndExclusive).map((token) => {const timed = timedById.get(token.id);if (timed === undefined || timed.segmentId !== segment.id) {@@ -48,7 +49,7 @@ export function materializeSemanticTake(endFrameExclusive: localFrame(timed.endFrameExclusive, localFrameCount, `Token ${token.id}`),};});- const anchorsById = new Map(map.anchors.map((anchor) => [anchor.identity, anchor]));+ const anchorsById = new Map(timing.anchors.map((anchor) => [anchor.identity, anchor]));const anchors = narrative.semanticIndex.anchors.filter((anchor) => anchor.segmentId === segment.id).map((anchor) => {diff --git a/packages/speech-alignment/src/types.ts b/packages/speech-alignment/src/types.tsindex d81c11e0..92d4054a 100644--- a/packages/speech-alignment/src/types.ts+++ b/packages/speech-alignment/src/types.ts@@ -5,7 +5,7 @@ export type {SpeechTranscriptPassage,SpeechWordEvidence,} from "@hypit/speech-evidence";-/** Private vocabulary of the alignment implementation, not part of SemanticMap. */+/** Private vocabulary of the alignment implementation, not part of SemanticTake. */export type AlignmentRelation =| "exact"| "split"diff --git a/packages/speech-alignment/test/speech-align.test.ts b/packages/speech-alignment/test/speech-align.test.tsindex 001ace81..0b68a538 100644--- a/packages/speech-alignment/test/speech-align.test.ts+++ b/packages/speech-alignment/test/speech-align.test.ts@@ -294,12 +294,53 @@ test("locating is total: every Script token carries a window", () => {});assert.equal(blind.tokens.length, narrative.tokens.length);assert.equal(blind.tokens.every((token) => Number.isSafeInteger(token.startFrame)- && Number.isSafeInteger(token.endFrameExclusive)), true);+ && Number.isSafeInteger(token.endFrameExclusive)+ && token.endFrameExclusive > token.startFrame), true);// The speaker said something else entirely; the Script is still fully located.const diverged = locate(narrative, { words: [{ text: "zzz", startSec: 0.2, endSec: 0.8 }] });assert.equal(diverged.tokens.length, narrative.tokens.length);- assert.equal(diverged.tokens.every((token) => Number.isSafeInteger(token.startFrame)), true);+ assert.equal(diverged.tokens.every((token) => token.endFrameExclusive > token.startFrame), true);+});++test("a collapsed WhisperX word is assigned the available interval between its neighbors", () => {+ const narrative = parseScript("collapsed.svml", "<line>well a lot</line>");+ const map = locate(narrative, {+ durationSec: 1,+ words: [+ { text: "well", startSec: 0.1, endSec: 0.4 },+ { text: "a", startSec: 0.43, endSec: 0.43 },+ { text: "lot", startSec: 0.5, endSec: 0.8 },+ ],+ });+ assert.deepEqual(map.tokens.map((token) => [token.startFrame, token.endFrameExclusive]), [+ [100, 400],+ [400, 500],+ [500, 800],+ ]);+});++test("three Script words may share the two video frames covered by one evidence word", () => {+ const narrative = parseScript("pigeonhole.svml", "<line>alpha beta gamma</line>");+ const original = speechBasis(narrative, 1);+ const programSpace = sealProgramSpace({+ durationSec: 1,+ frameRate: { numerator: 32, denominator: 1 },+ });+ const basis: AlignmentBasis = {+ ...original,+ programSpace,+ segments: [{ segmentId: "line", startFrame: 0, endFrameExclusive: 32 }],+ };+ const map = locateAlignedSegmentTiming(narrative, basis, evidence({+ basis,+ words: [{ text: "alphabetagamma", startSec: 10 / 32, endSec: 12 / 32 }],+ }));+ assert.deepEqual(map.tokens.map((token) => [token.startFrame, token.endFrameExclusive]), [+ [10, 11],+ [10, 12],+ [11, 12],+ ]);});test("Evidence is interpreted only through the explicitly connected alignment clock", () => {@@ -347,12 +388,7 @@ test("the final map is quantized once into the selected ProgramSpace", () => {assert.equal(map.anchors.every((anchor) => Number.isSafeInteger(anchor.frame)), true);});-test("a backwards character measurement reaches the map backwards, uncorrected", () => {- // WhisperX character times need not be ordered. The locator reports what it- // measured; deciding what a backwards window means belongs to whoever renders- // it, and silently flattening it here would hide the evidence. Three Script- // words merge into one spoken word, so the middle one inherits neither edge of- // the group envelope and its own characters decide its window.+test("a backwards character measurement is treated as missing token timing", () => {const narrative = parseScript("backwards.svml", "<line>can not now</line>");const map = locate(narrative, {endSec: 1,@@ -365,9 +401,5 @@ test("a backwards character measurement reaches the map backwards, uncorrected",),});const middle = map.tokens[1]!;- assert.equal(- middle.startFrame > middle.endFrameExclusive,- true,- `the inverted measurement survived, got ${middle.startFrame}..${middle.endFrameExclusive}`,- );+ assert.equal(middle.endFrameExclusive > middle.startFrame, true);});diff --git a/packages/speech-basis/README.md b/packages/speech-basis/README.mddeleted file mode 100644index 3b870678..00000000--- a/packages/speech-basis/README.md+++ /dev/null@@ -1,23 +0,0 @@-# `@hypit/speech-basis`--Reference Product/Projection package for an atomic speech take.--`SpeechBasis` is the indivisible Product that binds audio, visual clips and ProgramSpace. Its-identity belongs to the enclosing Core Record; the domain value does not repeat a self digest or-copy an upstream identity. Three ordinary deterministic Producers project it into `ProgramSpace`,-`VisualTrack` and `AudioTrack`. Core has no Projection primitive and never treats-these outputs as ports of one mutable workflow node.--This gives the author several independent Logical Outputs while preserving one shared generation-Operation. Selecting an Existing `SpeechBasis` stops generation and leaves both Projections usable;-selecting only an Existing visual Candidate does not change the independent audio path.--`speechBasisComponent` enumerates all three Producer facets. Their identities are checked against the-static `speechBasisManifest` and can be activated by any compatible compute Host selected by Source-imports without a Driver or Runtime source change. The component has no Provider,-Artifact access, queue or credentials.--The package intentionally publishes no Type validators: it does not own any of its input or output-Types. `@hypit/speech`, `@hypit/program-space` and `@hypit/composition` remain the-semantic owners of `SpeechBasis`, `ProgramSpace` and the generic Track Types, and supply any owner-validators required by those contracts.diff --git a/packages/speech-basis/package.json b/packages/speech-basis/package.jsondeleted file mode 100644index 29c32c48..00000000--- a/packages/speech-basis/package.json+++ /dev/null@@ -1,29 +0,0 @@-{- "name": "@hypit/speech-basis",- "version": "0.0.0-dev",- "license": "SEE LICENSE IN LICENSE",- "private": true,- "type": "module",- "exports": {- ".": "./src/index.ts"- },- "hypit": {- "activation": "./src/activation.ts"- },- "dependencies": {- "@hypit/component-kit": "workspace:*",- "@hypit/composition": "workspace:*",- "@hypit/elaborator": "workspace:*",- "@hypit/media-track": "workspace:*",- "@hypit/program-space": "workspace:*",- "@hypit/protocol": "workspace:*",- "@hypit/speech": "workspace:*"- },- "devDependencies": {- "@hypit/core": "workspace:*",- "@hypit/driver-node": "workspace:*",- "@hypit/media-pipeline": "workspace:*",- "@hypit/svs": "workspace:*",- "@hypit/validation": "workspace:*"- }-}diff --git a/packages/speech-basis/src/activation.ts b/packages/speech-basis/src/activation.tsdeleted file mode 100644index 0e1bcd4d..00000000--- a/packages/speech-basis/src/activation.ts+++ /dev/null@@ -1,8 +0,0 @@-import { speechBasisComponent, speechBasisManifest } from "./index.js";--export const hypitPackage = {- format: "hypit.node-package@1" as const,- modules: [{ manifest: speechBasisManifest }],- components: [speechBasisComponent],-};-export default hypitPackage;diff --git a/packages/speech-basis/src/component.ts b/packages/speech-basis/src/component.tsdeleted file mode 100644index ae3e9257..00000000--- a/packages/speech-basis/src/component.ts+++ /dev/null@@ -1,56 +0,0 @@-import type { ComponentPackage } from "@hypit/component-kit";-import type { SpeechBasis } from "@hypit/speech";-import type { StoredValue } from "@hypit/protocol";-import { canonicalize } from "@hypit/protocol";--import {- speechBasisProducers,-} from "./manifest.js";-import { projectSpeechAudioTrack, projectSpeechProgramSpace, projectSpeechVisual } from "./projection.js";--function speechBasis(value: StoredValue | undefined): SpeechBasis {- if (value?.kind !== "inline") throw new Error("SpeechBasis must be inline");- return value.value as unknown as SpeechBasis;-}--/** Pure projections of one atomic SpeechBasis; no Artifact read or external capability is hidden here. */-export const speechBasisComponent = {- producers: [- {- producer: speechBasisProducers.projectProgramSpace,- handler: ({ inputs }) => ({- outputs: {- programSpace: {- kind: "inline",- value: canonicalize(projectSpeechProgramSpace(speechBasis(inputs.basis?.value))),- },- },- needs: {},- }),- },- {- producer: speechBasisProducers.projectVisual,- handler: ({ inputs }) => ({- outputs: {- visual: {- kind: "inline",- value: canonicalize(projectSpeechVisual(speechBasis(inputs.basis?.value))),- },- },- needs: {},- }),- },- {- producer: speechBasisProducers.projectAudioTrack,- handler: ({ inputs }) => ({- outputs: {- track: {- kind: "inline",- value: canonicalize(projectSpeechAudioTrack(speechBasis(inputs.basis?.value))),- },- },- needs: {},- }),- },- ],-} satisfies ComponentPackage;diff --git a/packages/speech-basis/src/fragment.ts b/packages/speech-basis/src/fragment.tsdeleted file mode 100644index e26ed170..00000000--- a/packages/speech-basis/src/fragment.ts+++ /dev/null@@ -1,52 +0,0 @@-import { programSpaceTypes } from "@hypit/program-space";-import { speechTypes } from "@hypit/speech";-import type { SpeechBasis } from "@hypit/speech";-import { compositionTypes } from "@hypit/composition";-import { sealGraphFragment } from "@hypit/elaborator";--import { speechBasisProducers } from "./manifest.js";--const input = (name: string) => ({ kind: "fragment-input" as const, name });-const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });--/** Project the independently demandable facts carried by one SpeechBasis. */-export const speechBasisProjectionFragment = sealGraphFragment({- inputs: [{ name: "basis", type: speechTypes.basis }],- operations: [- {- id: "project-program-space",- producer: speechBasisProducers.projectProgramSpace,- inputs: { basis: input("basis") },- result: { kind: "output", name: "programSpace" },- },- {- id: "project-visual",- producer: speechBasisProducers.projectVisual,- inputs: { basis: input("basis") },- result: { kind: "output", name: "visual" },- },- {- id: "project-audio-track",- producer: speechBasisProducers.projectAudioTrack,- inputs: { basis: input("basis") },- result: { kind: "output", name: "track" },- },- ],- exports: [- {- name: "programSpace",- type: programSpaceTypes.programSpace,- root: operation("project-program-space"),- },- {- name: "visual",- type: compositionTypes.visualTrack,- root: operation("project-visual"),- },- {- name: "audioTrack",- type: compositionTypes.audioTrack,- root: operation("project-audio-track"),- },- ],-});diff --git a/packages/speech-basis/src/index.ts b/packages/speech-basis/src/index.tsdeleted file mode 100644index 002ee9b4..00000000--- a/packages/speech-basis/src/index.ts+++ /dev/null@@ -1,8 +0,0 @@-export { speechBasisComponent } from "./component.js";-export { speechBasisProjectionFragment } from "./fragment.js";-export {- speechBasisManifest,- speechBasisModuleRef,- speechBasisProducers,-} from "./manifest.js";-export { projectSpeechAudioTrack, projectSpeechProgramSpace, projectSpeechVisual } from "./projection.js";diff --git a/packages/speech-basis/src/manifest.ts b/packages/speech-basis/src/manifest.tsdeleted file mode 100644index b1bb2f1d..00000000--- a/packages/speech-basis/src/manifest.ts+++ /dev/null@@ -1,45 +0,0 @@-import { programSpaceDependency, programSpaceTypes } from "@hypit/program-space";-import { speechDependency, speechTypes } from "@hypit/speech";-import { compositionDependency, compositionTypes } from "@hypit/composition";-import type { ModuleManifest, ProducerRef } from "@hypit/protocol";--export const speechBasisModuleRef = { name: "@hypit/speech-basis", version: "1" } as const;--export const speechBasisProducers = {- projectAudioTrack: { module: speechBasisModuleRef, name: "project-audio-track" },- projectVisual: { module: speechBasisModuleRef, name: "project-visual" },- projectProgramSpace: { module: speechBasisModuleRef, name: "project-program-space" },-} satisfies Record<string, ProducerRef>;--export const speechBasisManifest: ModuleManifest = {- format: "hypit.module@1",- name: speechBasisModuleRef.name,- version: speechBasisModuleRef.version,- dependencies: [- speechDependency,- programSpaceDependency,- compositionDependency,- ],- types: [],- capabilities: [],- producers: [- {- name: speechBasisProducers.projectProgramSpace.name,- inputs: [{ name: "basis", type: speechTypes.basis }],- outputs: [{ name: "programSpace", type: programSpaceTypes.programSpace }],- needs: [],- },- {- name: speechBasisProducers.projectVisual.name,- inputs: [{ name: "basis", type: speechTypes.basis }],- outputs: [{ name: "visual", type: compositionTypes.visualTrack }],- needs: [],- },- {- name: speechBasisProducers.projectAudioTrack.name,- inputs: [{ name: "basis", type: speechTypes.basis }],- outputs: [{ name: "track", type: compositionTypes.audioTrack }],- needs: [],- },- ],-};diff --git a/packages/speech-basis/src/projection.ts b/packages/speech-basis/src/projection.tsdeleted file mode 100644index 48047e46..00000000--- a/packages/speech-basis/src/projection.ts+++ /dev/null@@ -1,68 +0,0 @@-import { programSpaceSampleFrames } from "@hypit/program-space";-import type { ProgramSpace } from "@hypit/program-space";-import { assertSpeechBasisIdentity } from "@hypit/speech";-import type { SpeechBasis } from "@hypit/speech";-import { lowerRestrictedSpeechVisualPresents } from "@hypit/media-track";-import { sealAudioTrack, sealVisualTrack } from "@hypit/composition";-import type { AudioTrack, VisualTrack } from "@hypit/composition";--export function projectSpeechProgramSpace(basis: SpeechBasis): ProgramSpace {- assertSpeechBasisIdentity(basis);- return basis.programSpace;-}--export function projectSpeechVisual(basis: SpeechBasis): VisualTrack {- assertSpeechBasisIdentity(basis);- const trackId = `speech-visual:${basis.segments.map((segment) => segment.segmentId).join("+")}`;- const spans = new Map(basis.segments.map((segment) => [segment.segmentId, {- startFrame: segment.startFrame,- endFrameExclusive: segment.endFrameExclusive,- }]));- return sealVisualTrack({- visualIr: "hypit.visual-ir@1",- id: trackId,- presents: lowerRestrictedSpeechVisualPresents(- trackId,- basis.programSpace,- basis.visualTrack.clips.map((clip) => {- const span = spans.get(clip.segmentId)!;- return {- id: clip.segmentId,- span,- artifact: clip.artifact,- extent: clip.extent,- frameRate: basis.programSpace.frameRate,- frameCount: span.endFrameExclusive - span.startFrame,- frame: clip.frame,- fit: clip.fit,- stackingOrder: clip.stackingOrder,- };- }),- ),- });-}--export function projectSpeechAudioTrack(basis: SpeechBasis): AudioTrack {- assertSpeechBasisIdentity(basis);- const sampleFrames = programSpaceSampleFrames(basis.programSpace, 48_000);- return sealAudioTrack({- id: `speech-audio:${basis.segments.map((segment) => segment.segmentId).join("+")}`,- clips: [{- id: "speech",- artifact: basis.audio,- target: { startSample: 0, endSampleExclusive: sampleFrames },- source: {- sampleFrames,- startSample: 0,- endSampleExclusive: sampleFrames,- loop: false,- phaseSample: 0,- },- playbackRate: 1,- pitch: "preserve",- gain: 1,- fadeInSamples: 0,- fadeOutSamples: 0,- }],- });-}diff --git a/packages/speech-basis/test/product.test.ts b/packages/speech-basis/test/product.test.tsdeleted file mode 100644index eb3afbb0..00000000--- a/packages/speech-basis/test/product.test.ts+++ /dev/null@@ -1,317 +0,0 @@-import { compositionComponent, spatialComponent, videoContractManifests } from "../../../test/support/video-domain.js";-import {- registerProducerFacets,- registerTypeValidatorFacets,-} from "@hypit/component-kit";-import { sealProgramSpace } from "@hypit/program-space";-import { sealSpeechBasis, speechDependency, speechTypes } from "@hypit/speech";-import type { SpeechBasis } from "@hypit/speech";-import { compositionTypes } from "@hypit/composition";-import { mediaPipelineManifest } from "@hypit/media-pipeline";-import assert from "node:assert/strict";-import test from "node:test";-import { fixtureDigest } from "../../../test/fixture-digest.js";--import {- createResolvedClosure,- link,- sealBuildRequest,- sealCompiledGraph,- sealRecord,- start,-} from "@hypit/core";-import { ProducerRegistry, NodeDriver } from "@hypit/driver-node";-import type {- BuildRequest,- CompiledGraph,- LinkedProgram,- ModuleManifest,- ProducerRef,- TypeRef,-} from "@hypit/protocol";-import {- projectSpeechAudioTrack,- projectSpeechProgramSpace,- projectSpeechVisual,- speechBasisComponent,- speechBasisManifest,- speechBasisProducers,-} from "@hypit/speech-basis";-import { TypeValidatorRegistry } from "@hypit/validation";-import { mediaTrackManifest } from "@hypit/media-track";-import { svsManifest } from "@hypit/svs";--const testModule = { name: "example.speech-basis-product", version: "0.0.0" } as const;-const requestType = { module: testModule, name: "SpeechRequest" } satisfies TypeRef;-const generateProducer = { module: testModule, name: "generate-speech-basis" } satisfies ProducerRef;--const testManifest: ModuleManifest = {- format: "hypit.module@1",- name: testModule.name,- version: testModule.version,- dependencies: [speechDependency],- types: [{ name: requestType.name }],- capabilities: [],- producers: [{- name: generateProducer.name,- inputs: [{ name: "request", type: requestType }],- outputs: [{ name: "take", type: speechTypes.basis }],- needs: [],- }],-};--const closure = createResolvedClosure([- ...videoContractManifests,- mediaPipelineManifest,- mediaTrackManifest,- svsManifest,- speechBasisManifest,- testManifest,-]);--function validatorRegistry(): TypeValidatorRegistry {- const registry = new TypeValidatorRegistry();- registerTypeValidatorFacets(registry, compositionComponent.validators);- registerTypeValidatorFacets(registry, spatialComponent.validators);- return registry;-}--function sampleTake(label = "generated"): SpeechBasis {- const durationSec = 2;- const programSpace = sealProgramSpace({- durationSec,- frameRate: { numerator: 30, denominator: 1 },- });- const audio = {- kind: "blob" as const,- digest: fixtureDigest(`${label}:audio`),- size: 12,- mediaType: "audio/wav",- };- const visual = {- kind: "blob" as const,- digest: fixtureDigest(`${label}:visual`),- size: 24,- mediaType: "video/mp4",- };- return sealSpeechBasis({- programSpace,- audio,- visualTrack: {- clips: [{- segmentId: "opening",- artifact: visual,- extent: { widthPx: 720, heightPx: 1280 },- frame: { xPx: 0, yPx: 0, widthPx: 720, heightPx: 1280 },- fit: {- sizing: "cover",- framePoint: { x: 0.5, y: 0.5 }, contentPoint: { x: 0.5, y: 0.5 },- offsetPx: { x: 0, y: 0 }, constraint: "bounded",- },- stackingOrder: 0,- }],- },- segments: [{- segmentId: "opening",- startFrame: 0,- endFrameExclusive: durationSec * 30,- }],- });-}--function createProgram(): LinkedProgram {- const request = sealRecord({- id: "request:opening",- type: requestType,- value: { kind: "inline", value: "Generate the opening speech take." },- });- return link(closure, [request]);-}--function createGraph(program: LinkedProgram): CompiledGraph {- const record = (id: string) => ({ kind: "record" as const, id });- const output = (id: string) => ({ kind: "logical-output" as const, id });- const operation = (id: string) => ({ kind: "operation-result" as const, operation: id });- const existingTake = sampleTake("approved");- const existingVisual = projectSpeechVisual(existingTake);- const existingVisualValue = { kind: "inline" as const, value: existingVisual };- return sealCompiledGraph({- outputs: [- {- id: "opening.take",- type: speechTypes.basis,- primary: "opening.take.generate",- },- {- id: "opening.audio",- type: compositionTypes.audioTrack,- primary: "opening.audio.project",- },- {- id: "opening.visual",- type: compositionTypes.visualTrack,- primary: "opening.visual.project",- },- ],- candidates: [- {- id: "opening.take.generate",- type: speechTypes.basis,- root: { kind: "operation", result: operation("generate-opening") },- },- {- id: "opening.take.existing",- type: speechTypes.basis,- root: {- kind: "value",- value: {- id: "provided:opening-take",- value: { kind: "inline", value: existingTake },- },- },- },- {- id: "opening.audio.project",- type: compositionTypes.audioTrack,- root: { kind: "operation", result: operation("project-opening-audio") },- },- {- id: "opening.visual.project",- type: compositionTypes.visualTrack,- root: { kind: "operation", result: operation("project-opening-visual") },- },- {- id: "opening.visual.existing",- type: compositionTypes.visualTrack,- root: {- kind: "value",- value: {- id: "provided:opening-visual",- value: existingVisualValue,- },- },- },- ],- operations: [- {- id: "generate-opening",- producer: generateProducer,- inputs: { request: record("request:opening") },- result: { kind: "output", name: "take", record: "take:opening" },- },- {- id: "project-opening-audio",- producer: speechBasisProducers.projectAudioTrack,- inputs: { basis: output("opening.take") },- result: { kind: "output", name: "track", record: "audio:opening" },- },- {- id: "project-opening-visual",- producer: speechBasisProducers.projectVisual,- inputs: { basis: output("opening.take") },- result: { kind: "output", name: "visual", record: "visual:opening" },- },- ],- });-}--function build(- targets: readonly string[],- satisfactionMap: Readonly<Record<string, string>> = {},-) {- const program = createProgram();- const sourceGraph = createGraph(program);- const graph = sealCompiledGraph({- outputs: sourceGraph.outputs.map((item) => ({- ...item,- primary: satisfactionMap[item.id] ?? item.primary,- })),- candidates: sourceGraph.candidates,- operations: sourceGraph.operations,- });- const request: BuildRequest = sealBuildRequest({- targets: targets.map((outputId) => ({ output: outputId })),- });- return start(program, graph, request);-}--function stepIds(state: ReturnType<typeof build>): string[] {- return state.plan.steps.map((step) => step.id).sort();-}--test("SpeechBasis is one Product and audio/visual are ordinary shared projections", () => {- const state = build(["opening.audio", "opening.visual"]);- assert.deepEqual(stepIds(state), [- "generate-opening",- "project-opening-audio",- "project-opening-visual",- ]);- assert.equal(state.plan.steps.filter((step) => step.id === "generate-opening").length, 1);- assert.equal(- state.plan.steps.find((step) => step.id === "project-opening-audio")?.inputs.basis,- "take:opening",- );- assert.equal(- state.plan.steps.find((step) => step.id === "project-opening-visual")?.inputs.basis,- "take:opening",- );-});--test("SpeechBasis projects to peer generic visual and audio Tracks", () => {- const take = sampleTake();- const visual = projectSpeechVisual(take);- const audio = projectSpeechAudioTrack(take);- const programSpace = projectSpeechProgramSpace(take);- assert.equal(visual.kind, "visual");- assert.equal(audio.kind, "audio");- assert.deepEqual(programSpace, take.programSpace);- assert.equal(- audio.clips[0]?.target.endSampleExclusive,- Math.round(visual.presents[0]!.span.endFrameExclusive * 48_000- * take.programSpace.frameRate.denominator / take.programSpace.frameRate.numerator),- );-});--test("a visual Candidate does not affect an independent audio path", () => {- const state = build(- ["opening.audio", "opening.visual"],- { "opening.visual": "opening.visual.existing" },- );- assert.deepEqual(stepIds(state), ["generate-opening", "project-opening-audio"]);- assert.equal(- state.plan.selections.find((selection) => selection.output === "opening.audio")?.record,- "audio:opening",- );- assert.equal(- state.plan.selections.find((selection) => selection.output === "opening.visual")?.record,- "provided:opening-visual",- );-});--test("selecting an Existing SpeechBasis stops generation but keeps both projections", () => {- const state = build(- ["opening.audio", "opening.visual"],- { "opening.take": "opening.take.existing" },- );- assert.deepEqual(stepIds(state), ["project-opening-audio", "project-opening-visual"]);- assert.equal(state.records.some((record) => record.id === "provided:opening-take"), true);- assert.equal(- state.plan.steps.find((step) => step.id === "project-opening-audio")?.inputs.basis,- "provided:opening-take",- );-});--test("the Build Machine executes one shared generation for both projected outputs", async () => {- let generations = 0;- const registry = new ProducerRegistry();- registry.registerProducer(generateProducer, () => {- generations += 1;- return { outputs: { take: { kind: "inline", value: sampleTake() } }, needs: {} };- });- registerProducerFacets(registry, speechBasisComponent.producers);- const result = await new NodeDriver({ producers: registry, validators: validatorRegistry() }).run(- build(["opening.audio", "opening.visual"]),- );- assert.equal(result.status, "complete");- assert.equal(generations, 1);-});diff --git a/packages/speech-spine/README.md b/packages/speech-spine/README.mddeleted file mode 100644index 7f7b8f04..00000000--- a/packages/speech-spine/README.md+++ /dev/null@@ -1,27 +0,0 @@-# @hypit/speech-spine--Trusted, static Graph Fragments for the first official speech-video Program.--This package contains no Core behavior, Provider credentials, queues, or API-clients. It only composes ordinary Producers from the leaf packages into-auditable reusable graph contributions:--- atomic SpeechBasis audio/visual projections after assembly;-- already-semantic Take assembly plus the provider-neutral speech SemanticMap;-- timed caption projection.--The package also owns `<speech:Spine>`. The Spine consumes the shared program `Clock` (with a-legacy inline `frame-rate` accepted during migration) and its restricted visual base-(`visual-frame`, `visual-appearance`, `visual-z`); each-`<speech:Take>` connects one self-contained `SemanticTake`. Media normalization, acoustic evidence,-and local semantic projection happen before the Spine is entered.--Visual Takes may override the Spine base with `frame`, `appearance` and `z`. That is the complete-visual authority of Speech Spine: one same-source visual clip may be placed and stacked. Motion,-transitions, sequences and independent pictures remain ordinary Media Tracks.--The Surface then expands an immutable append fold, one timeline-audio render and ordinary-SpeechBasis projections. Whitespace and child layout do not create ports, and Core receives no-variadic node, media-normalization rule, provider call, or global alignment special case.--The Elaborator expands them hygienically before SVML Core sees the graph.diff --git a/packages/speech-spine/src/activation.ts b/packages/speech-spine/src/activation.tsdeleted file mode 100644index ca207e1d..00000000--- a/packages/speech-spine/src/activation.ts+++ /dev/null@@ -1,19 +0,0 @@-import { createMarkupSurfaceHostFacet } from "@hypit/markup";-import {- decodeSpeechSpineSurface, speechSpineComponent, speechSpineManifest,- speechSpineModuleRef,- speechSpineMarkupSurfaces,-} from "./index.js";--export const hypitPackage = {- format: "hypit.node-package@1" as const,- modules: [{- manifest: speechSpineManifest,- }],- components: [speechSpineComponent],- hostFacets: [createMarkupSurfaceHostFacet({- module: speechSpineModuleRef,- declaration: speechSpineMarkupSurfaces.find((item) => item.name === "spine")!, handler: decodeSpeechSpineSurface,- })],-};-export default hypitPackage;diff --git a/packages/speech-spine/src/component.ts b/packages/speech-spine/src/component.tsdeleted file mode 100644index f5332902..00000000--- a/packages/speech-spine/src/component.ts+++ /dev/null@@ -1,88 +0,0 @@-import type { ComponentPackage } from "@hypit/component-kit";-import type { TimelineAudio } from "@hypit/media";-import type { StoredValue } from "@hypit/protocol";-import { canonicalize } from "@hypit/protocol";-import type { SemanticTake } from "@hypit/speech";-import type { ContentFit, SpatialFrame } from "@hypit/spatial";--import { speechSpineProducers } from "./manifest.js";-import { appendSpeechSpineAudioTake, appendSpeechSpineVisualTake, assembleSpeechBasis, compileSpeechSpineAudio, createSpeechSpineSet } from "./program.js";-import { assembleSemanticTakeMap } from "./semantic.js";-import type { SpeechSpineProgram, SpeechSpineSet, SpeechSpineVisualSpec } from "./types.js";--function inline<T>(value: StoredValue | undefined, subject: string): T {- if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);- return value.value as unknown as T;-}--export const speechSpineComponent = {- producers: [- {- producer: speechSpineProducers.createSet,- handler: () => ({- outputs: { set: { kind: "inline", value: canonicalize(createSpeechSpineSet()) } },- needs: {},- }),- },- {- producer: speechSpineProducers.appendAudioTake,- handler: ({ inputs }) => ({- outputs: { set: { kind: "inline", value: canonicalize(appendSpeechSpineAudioTake(- inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet"),- inline<SpeechSpineProgram>(inputs.program?.value, "SpeechSpineProgram"),- inline<SemanticTake>(inputs.take?.value, "SemanticTake"),- )) } },- needs: {},- }),- },- {- producer: speechSpineProducers.appendVisualTake,- handler: ({ inputs }) => ({- outputs: { set: { kind: "inline", value: canonicalize(appendSpeechSpineVisualTake(- inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet"),- inline<SpeechSpineProgram>(inputs.program?.value, "SpeechSpineProgram"),- inline<SemanticTake>(inputs.take?.value, "SemanticTake"),- inline<SpatialFrame>(inputs.frame?.value, "SpatialFrame"),- inline<ContentFit>(inputs.fit?.value, "ContentFit"),- inline<SpeechSpineVisualSpec>(inputs.visualSpec?.value, "SpeechSpineVisualSpec"),- )) } },- needs: {},- }),- },- {- producer: speechSpineProducers.compileAudio,- handler: ({ inputs }) => ({- outputs: { plan: { kind: "inline", value: canonicalize(compileSpeechSpineAudio(- inline<SpeechSpineProgram>(inputs.program?.value, "SpeechSpineProgram"),- inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet"),- )) } },- needs: {},- }),- },- {- producer: speechSpineProducers.assembleBasis,- handler: ({ inputs }) => ({- outputs: { basis: { kind: "inline", value: canonicalize(assembleSpeechBasis(- inline<SpeechSpineProgram>(inputs.program?.value, "SpeechSpineProgram"),- inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet"),- inline<TimelineAudio>(inputs.audio?.value, "TimelineAudio"),- )) } },- needs: {},- }),- },- {- producer: speechSpineProducers.assembleSemanticMap,- handler: ({ inputs }) => ({- outputs: {- map: {- kind: "inline",- value: canonicalize(assembleSemanticTakeMap(- inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet").takes.map((take) => take.semantic),- )),- },- },- needs: {},- }),- },- ],-} satisfies ComponentPackage;diff --git a/packages/speech-spine/src/fragment.ts b/packages/speech-spine/src/fragment.tsdeleted file mode 100644index 32f4554b..00000000--- a/packages/speech-spine/src/fragment.ts+++ /dev/null@@ -1,126 +0,0 @@-import { programSpaceTypes } from "@hypit/program-space";-import { speechTypes } from "@hypit/speech";-import { semanticMapTypes } from "@hypit/semantic-map";-import { compositionTypes } from "@hypit/composition";-import { sealGraphFragment } from "@hypit/elaborator";-import type { FragmentOperation } from "@hypit/elaborator";-import { mediaPipelineProducers } from "@hypit/media-pipeline";-import { speechBasisProducers } from "@hypit/speech-basis";-import { spatialTypes } from "@hypit/spatial";-import type { TypeRef } from "@hypit/protocol";--import { speechSpineProducers, speechSpineTypes } from "./manifest.js";-import type { SpeechSpineFragmentOptions } from "./types.js";--const input = (name: string) => ({ kind: "fragment-input" as const, name });-const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });--export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {- if (options.takes.length === 0) throw new Error("Speech Spine requires at least one Take");- const declaredInputs = new Map<string, TypeRef>();- const addInput = (name: string, type: TypeRef): void => {- if (!name) throw new Error("Speech Spine Fragment input names must not be empty");- const previous = declaredInputs.get(name);- if (previous !== undefined && (previous.module.name !== type.module.name- || previous.module.version !== type.module.version || previous.name !== type.name)) {- throw new Error(`Speech Spine Fragment input ${name} is reused with another Type`);- }- declaredInputs.set(name, type);- };- addInput("program", speechSpineTypes.spineProgram);- for (const take of options.takes) {- addInput(take.takeName, speechTypes.semanticTake);- if (take.visual !== undefined) {- addInput(take.visual.frameName, spatialTypes.frame);- addInput(take.visual.fitName, spatialTypes.fit);- addInput(take.visual.visualSpecName, speechSpineTypes.visualSpec);- }- }- const operations: FragmentOperation[] = [{- id: "spine:set:empty",- producer: speechSpineProducers.createSet,- inputs: {},- result: { kind: "output", name: "set" },- }];- let current = "spine:set:empty";- options.takes.forEach((take, index) => {- const id = `spine:set:append:${String(index + 1).padStart(4, "0")}`;- operations.push({- id,- producer: take.visual === undefined ? speechSpineProducers.appendAudioTake : speechSpineProducers.appendVisualTake,- inputs: {- set: operation(current),- program: input("program"),- take: input(take.takeName),- ...(take.visual === undefined ? {} : {- frame: input(take.visual.frameName),- fit: input(take.visual.fitName),- visualSpec: input(take.visual.visualSpecName),- }),- },- result: { kind: "output", name: "set" },- });- current = id;- });- operations.push(- {- id: "spine:audio:plan",- producer: speechSpineProducers.compileAudio,- inputs: { program: input("program"), set: operation(current) },- result: { kind: "output", name: "plan" },- },- {- id: "spine:audio:render",- producer: mediaPipelineProducers.renderAudio,- inputs: { plan: operation("spine:audio:plan") },- result: { kind: "need", name: "audio" },- },- {- id: "spine:basis",- producer: speechSpineProducers.assembleBasis,- inputs: { program: input("program"), set: operation(current), audio: operation("spine:audio:render") },- result: { kind: "output", name: "basis" },- },- {- id: "spine:space",- producer: speechBasisProducers.projectProgramSpace,- inputs: { basis: operation("spine:basis") },- result: { kind: "output", name: "programSpace" },- },- {- id: "spine:visual-track",- producer: speechBasisProducers.projectVisual,- inputs: { basis: operation("spine:basis") },- result: { kind: "output", name: "visual" },- },- {- id: "spine:audio-track",- producer: speechBasisProducers.projectAudioTrack,- inputs: { basis: operation("spine:basis") },- result: { kind: "output", name: "track" },- },- );- operations.push({- id: "spine:semantic-map",- producer: speechSpineProducers.assembleSemanticMap,- inputs: { set: operation(current) },- result: { kind: "output", name: "map" },- });- return sealGraphFragment({- inputs: [...declaredInputs.entries()].map(([name, type]) => ({ name, type })),- operations,- exports: [- { name: "basis", type: speechTypes.basis, root: operation("spine:basis") },- {- name: "space", type: programSpaceTypes.programSpace, root: operation("spine:space"),- },- {- name: "visual", type: compositionTypes.visualTrack, root: operation("spine:visual-track"),- },- {- name: "audioTrack", type: compositionTypes.audioTrack, root: operation("spine:audio-track"),- },- { name: "semanticMap", type: semanticMapTypes.complete, root: operation("spine:semantic-map") },- ],- });-}diff --git a/packages/speech-spine/src/index.ts b/packages/speech-spine/src/index.tsdeleted file mode 100644index 2ab27e7b..00000000--- a/packages/speech-spine/src/index.ts+++ /dev/null@@ -1,8 +0,0 @@-/** Official Speech Spine authoring and deterministic basis assembly. */-export { speechSpineComponent } from "./component.js";-export { createSpeechSpineFragment } from "./fragment.js";-export { speechSpineManifest, speechSpineMarkupSurfaces, speechSpineModuleRef, speechSpineProducers, speechSpineTypes, speechSpineProgramSchema, speechSpineSetSchema, speechSpineVisualSpecSchema } from "./manifest.js";-export { appendSpeechSpineAudioTake, appendSpeechSpineVisualTake, assembleSpeechBasis, assertSpeechSpineProgram, assertSpeechSpineSet, assertSpeechSpineVisualSpec, compileSpeechSpineAudio, createSpeechSpineSet, sealSpeechSpineProgram, sealSpeechSpineVisualSpec } from "./program.js";-export { assembleSemanticTakeMap } from "./semantic.js";-export { decodeSpeechSpineSurface } from "./surface.js";-export type * from "./types.js";diff --git a/packages/speech-spine/src/program.ts b/packages/speech-spine/src/program.tsdeleted file mode 100644index 2748552c..00000000--- a/packages/speech-spine/src/program.ts+++ /dev/null@@ -1,256 +0,0 @@-import { synchronizedMediaSampleFrames, verifyTimelineAudio } from "@hypit/media";-import type { TimelineAudio } from "@hypit/media";-import { assertProgramSpaceIdentity, sealProgramSpace } from "@hypit/program-space";-import type { ProgramSpace } from "@hypit/program-space";-import { assertSemanticTakeIdentity, sealSpeechBasis } from "@hypit/speech";-import type { SemanticTake, SpeechBasis } from "@hypit/speech";-import {- sealAudioProgramPlan,- verifyAudioProgramPlan,-} from "@hypit/media-pipeline";-import type { AudioProgramPlan } from "@hypit/media-pipeline";-import { canonicalize } from "@hypit/protocol";-import { assertContentFit, assertSpatialFrame } from "@hypit/spatial";-import type { ContentFit, SpatialFrame } from "@hypit/spatial";--import type {- SpeechSpineProgram,- SpeechSpineSet,- SpeechSpineTake,- SpeechSpineVisualSpec,-} from "./types.js";--function assert(condition: unknown, message: string): asserts condition {- if (!condition) throw new Error(message);-}--function programContent(value: SpeechSpineProgram): SpeechSpineProgram {- return {-- id: value.id,- frameRate: { ...value.frameRate },- };-}--export function sealSpeechSpineProgram(value: SpeechSpineProgram): SpeechSpineProgram {- return programContent(value);-}--export function assertSpeechSpineProgram(value: SpeechSpineProgram): void {- assert(value.id.trim().length > 0, "SpeechSpineProgram id must not be empty");- assert(Number.isSafeInteger(value.frameRate.numerator) && value.frameRate.numerator > 0- && Number.isSafeInteger(value.frameRate.denominator) && value.frameRate.denominator > 0,- "SpeechSpineProgram frame rate is invalid");-}--function setContent(value: SpeechSpineSet): SpeechSpineSet {- return canonicalize({-- takes: value.takes,- }) as unknown as SpeechSpineSet;-}--function sealSpeechSpineSet(value: SpeechSpineSet): SpeechSpineSet {- return setContent(value);-}--function assertTake(take: SpeechSpineTake, program: SpeechSpineProgram): void {- assertSemanticTakeIdentity(take.semantic);- const { media, segment } = take.semantic;- assert(media.audio !== undefined, `Speech Segment ${segment.segmentId} has no speech audio`);- assert(media.timeline.frameRate.numerator === program.frameRate.numerator- && media.timeline.frameRate.denominator === program.frameRate.denominator,- `Speech Segment ${segment.segmentId} uses another frame rate`);- assert((media.visual === undefined) === (take.visual === undefined),- `Speech Segment ${segment.segmentId} visual policy does not match its media`);- if (take.visual !== undefined) {- assertSpatialFrame(take.visual.frame);- assertContentFit(take.visual.fit);- assert(Number.isSafeInteger(take.visual.stackingOrder),- `Speech Segment ${segment.segmentId} visual stacking order is invalid`);- }-}--export function sealSpeechSpineVisualSpec(value: SpeechSpineVisualSpec): SpeechSpineVisualSpec {- const result = structuredClone(value);- assertSpeechSpineVisualSpec(result);- return result;-}--export function assertSpeechSpineVisualSpec(value: SpeechSpineVisualSpec): void {- assert(Number.isSafeInteger(value.stackingOrder), "SpeechSpineVisualSpec stacking order is invalid");-}--export function assertSpeechSpineSet(value: SpeechSpineSet): void {- const segments = new Set<string>();- for (const take of value.takes) {- assertSemanticTakeIdentity(take.semantic);- const segmentId = take.semantic.segment.segmentId;- assert(!segments.has(segmentId), `Speech Spine repeats Segment ${segmentId}`);- segments.add(segmentId);- }-}--export function createSpeechSpineSet(): SpeechSpineSet {- return sealSpeechSpineSet({-- takes: [],- });-}--function appendTake(- set: SpeechSpineSet,- program: SpeechSpineProgram,- take: SpeechSpineTake,-): SpeechSpineSet {- assertSpeechSpineSet(set);- assertSpeechSpineProgram(program);- assertTake(take, program);- assert(!set.takes.some((item) => item.semantic.segment.segmentId === take.semantic.segment.segmentId),- `Speech Spine repeats Segment ${take.semantic.segment.segmentId}`);- return sealSpeechSpineSet({-- takes: [...set.takes, take],- });-}--export function appendSpeechSpineAudioTake(- set: SpeechSpineSet,- program: SpeechSpineProgram,- semantic: SemanticTake,-): SpeechSpineSet {- assertSemanticTakeIdentity(semantic);- assert(semantic.media.visual === undefined,- `Speech Segment ${semantic.segment.segmentId} audio Take unexpectedly contains a visual stream`);- return appendTake(set, program, { semantic });-}--export function appendSpeechSpineVisualTake(- set: SpeechSpineSet,- program: SpeechSpineProgram,- semantic: SemanticTake,- frame: SpatialFrame,- fit: ContentFit,- visualSpec: SpeechSpineVisualSpec,-): SpeechSpineSet {- assertSpatialFrame(frame);- assertContentFit(fit);- assertSpeechSpineVisualSpec(visualSpec);- assertSemanticTakeIdentity(semantic);- assert(semantic.media.visual !== undefined, `Speech Segment ${semantic.segment.segmentId} visual Take has no visual stream`);- const take = {- semantic,- visual: {- frame: structuredClone(frame),- fit: structuredClone(fit),- stackingOrder: visualSpec.stackingOrder,- },- } satisfies SpeechSpineTake;- return appendTake(set, program, take);-}--function frameSample(frame: number, frameRate: SpeechSpineProgram["frameRate"]): number {- const numerator = BigInt(frame) * 48_000n * BigInt(frameRate.denominator);- const denominator = BigInt(frameRate.numerator);- const rounded = (numerator * 2n + denominator) / (denominator * 2n);- assert(rounded <= BigInt(Number.MAX_SAFE_INTEGER), "Speech Spine sample boundary exceeds safe arithmetic");- return Number(rounded);-}--function programSpace(program: SpeechSpineProgram, set: SpeechSpineSet): ProgramSpace {- assertSpeechSpineProgram(program);- assertSpeechSpineSet(set);- assert(set.takes.length > 0, "Speech Spine must contain at least one Take");- for (const take of set.takes) assertTake(take, program);- const frameCount = set.takes.reduce((sum, take) => sum + take.semantic.media.timeline.frameCount, 0);- const space = sealProgramSpace({- durationSec: frameCount * program.frameRate.denominator / program.frameRate.numerator,- frameRate: { ...program.frameRate },- });- assertProgramSpaceIdentity(space);- return space;-}--export function compileSpeechSpineAudio(program: SpeechSpineProgram, set: SpeechSpineSet): AudioProgramPlan {- const space = programSpace(program, set);- let frame = 0;- const clips = set.takes.map((take, index) => {- const { media, segment } = take.semantic;- const audio = media.audio!;- const startFrame = frame;- frame += media.timeline.frameCount;- const targetStartSample = frameSample(startFrame, program.frameRate);- const targetEndSampleExclusive = frameSample(frame, program.frameRate);- const sourceSampleFrames = synchronizedMediaSampleFrames(media);- assert(targetEndSampleExclusive - targetStartSample === sourceSampleFrames,- `Speech Segment ${segment.segmentId} audio does not exactly cover its normalized frame span`);- return {- id: `${String(index + 1).padStart(4, "0")}:${segment.segmentId}`,- artifact: audio.artifact,- targetStartSample,- targetEndSampleExclusive,- sourceSampleFrames,- sourceStartSample: 0,- sourceEndSampleExclusive: sourceSampleFrames,- sourceLoop: false,- sourcePhaseSample: 0,- playbackRate: 1,- pitch: "preserve" as const,- gain: 1,- fadeInSamples: 0,- fadeOutSamples: 0,- };- });- const plan = sealAudioProgramPlan({- frameRate: { ...space.frameRate },- frameCount: frame,- sampleRate: 48_000,- sampleFrames: frameSample(frame, program.frameRate),- clips,- mix: { normalize: false, limiter: "none" },- });- verifyAudioProgramPlan(plan);- return plan;-}--export function assembleSpeechBasis(- program: SpeechSpineProgram,- set: SpeechSpineSet,- audio: TimelineAudio,-): SpeechBasis {- const space = programSpace(program, set);- verifyTimelineAudio(audio);- assert(audio.sampleFrames === frameSample(- set.takes.reduce((sum, take) => sum + take.semantic.media.timeline.frameCount, 0),- program.frameRate,- ), "TimelineAudio does not cover this Speech Spine");- let frame = 0;- const segments = set.takes.map((take) => {- const startFrame = frame;- frame += take.semantic.media.timeline.frameCount;- return { segmentId: take.semantic.segment.segmentId, startFrame, endFrameExclusive: frame };- });- frame = 0;- const visualClips = set.takes.flatMap((take) => {- const { media, segment } = take.semantic;- frame += media.timeline.frameCount;- if (media.visual === undefined || take.visual === undefined) return [];- return [{- segmentId: segment.segmentId,- artifact: structuredClone(media.visual.artifact),- extent: {- widthPx: media.visual.width,- heightPx: media.visual.height,- },- frame: structuredClone(take.visual.frame),- fit: structuredClone(take.visual.fit),- stackingOrder: take.visual.stackingOrder,- }];- });- return sealSpeechBasis({- programSpace: space,- audio: structuredClone(audio.artifact),- visualTrack: { clips: visualClips },- segments,- });-}diff --git a/packages/speech-spine/src/semantic.ts b/packages/speech-spine/src/semantic.tsdeleted file mode 100644index 041fe42d..00000000--- a/packages/speech-spine/src/semantic.ts+++ /dev/null@@ -1,47 +0,0 @@-import type { CompleteSemanticMap, SemanticTimePoint, TimedSpeechToken } from "@hypit/semantic-map";-import { assertSemanticTakeIdentity } from "@hypit/speech";-import type { SemanticTake } from "@hypit/speech";--/**- * Compose local SemanticTakes in authored Spine order. The input takes already- * own their local media frame domains; Spine only translates their coordinates- * into one program frame domain and never re-identifies their words.- */-export function assembleSemanticTakeMap(takes: readonly SemanticTake[]): CompleteSemanticMap {- const tokens: TimedSpeechToken[] = [];- const anchors: SemanticTimePoint[] = [];- const segmentIds = new Set<string>();- const tokenIds = new Set<string>();- const anchorIds = new Set<string>();- let frameRate: SemanticTake["media"]["timeline"]["frameRate"] | undefined;- let offset = 0;- for (const take of takes) {- assertSemanticTakeIdentity(take);- if (frameRate === undefined) frameRate = take.media.timeline.frameRate;- else if (frameRate.numerator !== take.media.timeline.frameRate.numerator- || frameRate.denominator !== take.media.timeline.frameRate.denominator) {- throw new Error("Semantic Spine Takes must use one frame rate.");- }- if (segmentIds.has(take.segment.segmentId)) {- throw new Error(`Semantic Spine repeats Segment ${take.segment.segmentId}.`);- }- segmentIds.add(take.segment.segmentId);- for (const token of take.tokens) {- if (tokenIds.has(token.tokenId)) throw new Error(`Semantic Spine repeats Token ${token.tokenId}.`);- tokenIds.add(token.tokenId);- tokens.push({- tokenId: token.tokenId,- segmentId: token.segmentId,- startFrame: offset + token.startFrame,- endFrameExclusive: offset + token.endFrameExclusive,- });- }- for (const anchor of take.anchors) {- if (anchorIds.has(anchor.identity)) throw new Error(`Semantic Spine repeats Anchor ${anchor.identity}.`);- anchorIds.add(anchor.identity);- anchors.push({ identity: anchor.identity, frame: offset + anchor.frame });- }- offset += take.media.timeline.frameCount;- }- return { tokens, anchors };-}diff --git a/packages/speech-spine/src/types.ts b/packages/speech-spine/src/types.tsdeleted file mode 100644index ca5c9a9b..00000000--- a/packages/speech-spine/src/types.ts+++ /dev/null@@ -1,45 +0,0 @@-import type { ContentFit, SpatialFrame } from "@hypit/spatial";--/** Author-owned policy for one ordered Speech Spine. */-export type SpeechSpineProgram = {- readonly id: string;- readonly frameRate: {- readonly numerator: number;- readonly denominator: number;- };-};--export type SpeechSpineTake = {- readonly semantic: import("@hypit/speech").SemanticTake;- readonly visual?: {- readonly frame: SpatialFrame;- readonly fit: ContentFit;- readonly stackingOrder: number;- };-};--export type SpeechSpineVisualSpec = {- readonly stackingOrder: number;-};--/**- * Package-private immutable fold value. Its order is author meaning; Core sees- * only fixed-port append Operations for the concrete Spine declaration.- */-export type SpeechSpineSet = {- readonly takes: readonly SpeechSpineTake[];-};--export type SpeechSpineInput = {- readonly takeName: string;- readonly visual?: {- readonly frameName: string;- readonly fitName: string;- readonly visualSpecName: string;- };-};--export type SpeechSpineFragmentOptions = {- readonly name?: string;- readonly takes: readonly SpeechSpineInput[];-};diff --git a/packages/speech-spine/test/program.test.ts b/packages/speech-spine/test/program.test.tsdeleted file mode 100644index 70046393..00000000--- a/packages/speech-spine/test/program.test.ts+++ /dev/null@@ -1,103 +0,0 @@-import assert from "node:assert/strict";-import test from "node:test";-import { fixtureDigest } from "../../../test/fixture-digest.js";--import type { SynchronizedMedia, TimelineAudio } from "@hypit/media";-import type { SemanticTake } from "@hypit/speech";-import {- appendSpeechSpineAudioTake,- appendSpeechSpineVisualTake,- assembleSpeechBasis,- createSpeechSpineSet,- sealSpeechSpineProgram,- sealSpeechSpineVisualSpec,-} from "@hypit/speech-spine";--const frame = { xPx: 40, yPx: 80, widthPx: 640, heightPx: 900 };-const fit = {- sizing: "cover" as const,- framePoint: { x: 0.5, y: 0.5 },- contentPoint: { x: 0.5, y: 0.5 },- offsetPx: { x: 0, y: 0 },- constraint: "bounded" as const,-};--function synchronized(id: string, visual: boolean): SynchronizedMedia {- return {- timeline: {- frameRate: { numerator: 30, denominator: 1 },- frameCount: 30,- },- ...(visual ? { visual: {- artifact: { kind: "blob" as const, digest: fixtureDigest(`${id}:video`), size: 1, mediaType: "video/mp4" },- width: 720,- height: 1280,- } } : {}),- audio: {- artifact: { kind: "blob", digest: fixtureDigest(`${id}:audio`), size: 1, mediaType: "audio/wav" },- },- };-}--function semantic(id: string, visual: boolean): SemanticTake {- const media = synchronized(id, visual);- return {- media,- segment: {- segmentId: id,- startAnchorId: `segment:${id}:start`,- endAnchorId: `segment:${id}:end`,- startFrame: 0,- endFrameExclusive: media.timeline.frameCount,- },- tokens: [{- tokenId: `segment:${id}:token:1`,- segmentId: id,- text: id,- startAnchorId: `segment:${id}:token:1:start`,- endAnchorId: `segment:${id}:token:1:end`,- startFrame: 0,- endFrameExclusive: media.timeline.frameCount,- }],- anchors: [- { identity: `segment:${id}:start`, frame: 0 },- { identity: `segment:${id}:end`, frame: media.timeline.frameCount },- { identity: `segment:${id}:token:1:start`, frame: 0 },- { identity: `segment:${id}:token:1:end`, frame: media.timeline.frameCount },- ],- };-}--test("audio Takes lengthen the speech program without inventing a visual clip", () => {- const program = sealSpeechSpineProgram({-- id: "speech",- frameRate: { numerator: 30, denominator: 1 },- });- let set = createSpeechSpineSet();- set = appendSpeechSpineAudioTake(set, program, semantic("voiceover", false));- set = appendSpeechSpineVisualTake(- set,- program,- semantic("answer", true),- frame,- fit,- sealSpeechSpineVisualSpec({ stackingOrder: 30 }),- );- const audio: TimelineAudio = {- artifact: { kind: "blob", digest: fixtureDigest("speech:mix"), size: 1, mediaType: "audio/wav" },- sampleFrames: 96_000,- };- const basis = assembleSpeechBasis(program, set, audio);- assert.deepEqual(basis.segments.map((item) => item.segmentId), ["voiceover", "answer"]);- assert.equal(basis.programSpace.durationSec, 2);- assert.equal(basis.visualTrack.clips.length, 1);- assert.deepEqual(basis.visualTrack.clips[0], {- segmentId: "answer",- artifact: synchronized("answer", true).visual!.artifact,- extent: { widthPx: 720, heightPx: 1280 },- frame,- fit,- stackingOrder: 30,- });-});diff --git a/packages/speech-track/README.md b/packages/speech-track/README.mdnew file mode 100644index 00000000..58d3db49--- /dev/null+++ b/packages/speech-track/README.md@@ -0,0 +1,28 @@+# @hypit/speech-track++Trusted, static Graph Fragments for the first official speech-video Program.++This package contains no Core behavior, Provider credentials, queues, or API+clients. It only composes ordinary Producers from the leaf packages into+auditable reusable graph contributions:++- ordered, already-semantic Take assembly into one `SemanticTrack`;+- aligned peer VisualTrack and AudioTrack projections.++The package also owns `<speech:Track>`. The Track consumes a restricted visual base+(`visual-frame`, `visual-appearance`, `visual-z`); each+`<speech:Take>` connects one self-contained `SemanticTake`. Media normalization, acoustic evidence,+and local semantic projection happen before the Track is entered.++The shared frame rate and total ProgramSpace are derived from the ordered SemanticTakes. The Track+does not accept a second Clock or frame-rate declaration.++Visual Takes may override the Track base with `frame`, `appearance` and `z`. That is the complete+visual authority of Speech Track: one same-source visual clip may be placed and stacked. Motion,+transitions, sequences and independent pictures remain ordinary Media Tracks.++The Surface then expands an immutable append fold and ordinary SemanticTrack, VisualTrack and+AudioTrack projections. Whitespace and child layout do not create ports, and Core receives no+variadic node, media-normalization rule, provider call, or global alignment special case.++The Elaborator expands them hygienically before SVML Core sees the graph.diff --git a/packages/speech-spine/package.json b/packages/speech-track/package.jsonsimilarity index 73%rename from packages/speech-spine/package.jsonrename to packages/speech-track/package.jsonindex a5f0c843..e079c970 100644--- a/packages/speech-spine/package.json+++ b/packages/speech-track/package.json@@ -1,5 +1,5 @@{- "name": "@hypit/speech-spine",+ "name": "@hypit/speech-track","version": "0.0.0-dev","license": "SEE LICENSE IN LICENSE","private": true,@@ -11,24 +11,19 @@"activation": "./src/activation.ts"},"dependencies": {- "@hypit/artifact": "workspace:*","@hypit/component-kit": "workspace:*","@hypit/composition": "workspace:*","@hypit/elaborator": "workspace:*",- "@hypit/media": "workspace:*",- "@hypit/media-pipeline": "workspace:*",- "@hypit/program-space": "workspace:*",+ "@hypit/media-track": "workspace:*","@hypit/protocol": "workspace:*","@hypit/speech": "workspace:*",- "@hypit/speech-basis": "workspace:*",- "@hypit/semantic-map": "workspace:*",+ "@hypit/semantic-track": "workspace:*","@hypit/spatial": "workspace:*","@hypit/svs": "workspace:*","@hypit/markup": "workspace:*"},"devDependencies": {"@hypit/core": "workspace:*",- "@hypit/media-track": "workspace:*","@hypit/script": "workspace:*","@hypit/text": "workspace:*","@hypit/validation": "workspace:*"diff --git a/packages/speech-track/src/activation.ts b/packages/speech-track/src/activation.tsnew file mode 100644index 00000000..7ca63b1d--- /dev/null+++ b/packages/speech-track/src/activation.ts@@ -0,0 +1,19 @@+import { createMarkupSurfaceHostFacet } from "@hypit/markup";+import {+ decodeSpeechTrackSurface, speechTrackComponent, speechTrackManifest,+ speechTrackModuleRef,+ speechTrackMarkupSurfaces,+} from "./index.js";++export const hypitPackage = {+ format: "hypit.node-package@1" as const,+ modules: [{+ manifest: speechTrackManifest,+ }],+ components: [speechTrackComponent],+ hostFacets: [createMarkupSurfaceHostFacet({+ module: speechTrackModuleRef,+ declaration: speechTrackMarkupSurfaces.find((item) => item.name === "track")!, handler: decodeSpeechTrackSurface,+ })],+};+export default hypitPackage;diff --git a/packages/speech-track/src/component.ts b/packages/speech-track/src/component.tsnew file mode 100644index 00000000..a8d8b6e8--- /dev/null+++ b/packages/speech-track/src/component.ts@@ -0,0 +1,61 @@+import type { ComponentPackage } from "@hypit/component-kit";+import type { StoredValue } from "@hypit/protocol";+import { canonicalize } from "@hypit/protocol";+import type { SemanticTake } from "@hypit/speech";+import type { ContentFit, SpatialFrame } from "@hypit/spatial";++import { speechTrackProducers } from "./manifest.js";+import { appendSpeechTrackTake, assembleSpeechTrack, createSpeechTrackSet } from "./program.js";+import { projectSpeechTrackVisual } from "./projection.js";+import type { SemanticTrack } from "@hypit/semantic-track";+import type { SpeechTrackHeader, SpeechTrackSet, SpeechTrackVisualSpec } from "./types.js";++function inline<T>(value: StoredValue | undefined, subject: string): T {+ if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);+ return value.value as unknown as T;+}++export const speechTrackComponent = {+ producers: [+ {+ producer: speechTrackProducers.createSet,+ handler: () => ({+ outputs: { set: { kind: "inline", value: canonicalize(createSpeechTrackSet()) } },+ needs: {},+ }),+ },+ {+ producer: speechTrackProducers.appendTake,+ handler: ({ inputs }) => ({+ outputs: { set: { kind: "inline", value: canonicalize(appendSpeechTrackTake(+ inline<SpeechTrackSet>(inputs.set?.value, "SpeechTrackSet"),+ inline<SemanticTake>(inputs.take?.value, "SemanticTake"),+ inline<SpatialFrame>(inputs.frame?.value, "SpatialFrame"),+ inline<ContentFit>(inputs.fit?.value, "ContentFit"),+ inline<SpeechTrackVisualSpec>(inputs.visualSpec?.value, "SpeechTrackVisualSpec"),+ )) } },+ needs: {},+ }),+ },+ {+ producer: speechTrackProducers.assembleTrack,+ handler: ({ inputs }) => ({+ outputs: { track: { kind: "inline", value: canonicalize(assembleSpeechTrack(+ inline<SpeechTrackHeader>(inputs.header?.value, "SpeechTrackHeader"),+ inline<SpeechTrackSet>(inputs.set?.value, "SpeechTrackSet"),+ )) } },+ needs: {},+ }),+ },+ {+ producer: speechTrackProducers.projectVisual,+ handler: ({ inputs }) => ({+ outputs: { visual: { kind: "inline", value: canonicalize(projectSpeechTrackVisual(+ inline<SemanticTrack>(inputs.track?.value, "SemanticTrack"),+ inline<SpeechTrackSet>(inputs.set?.value, "SpeechTrackSet"),+ )) } },+ needs: {},+ }),+ },+ ],+} satisfies ComponentPackage;diff --git a/packages/speech-track/src/fragment.ts b/packages/speech-track/src/fragment.tsnew file mode 100644index 00000000..77859697--- /dev/null+++ b/packages/speech-track/src/fragment.ts@@ -0,0 +1,90 @@+import { speechTypes } from "@hypit/speech";+import { compositionTypes } from "@hypit/composition";+import { sealGraphFragment } from "@hypit/elaborator";+import type { FragmentOperation } from "@hypit/elaborator";+import { semanticTrackProducers, semanticTrackTypes } from "@hypit/semantic-track";+import { spatialTypes } from "@hypit/spatial";+import type { TypeRef } from "@hypit/protocol";++import { speechTrackProducers, speechTrackTypes } from "./manifest.js";+import type { SpeechTrackFragmentOptions } from "./types.js";++const input = (name: string) => ({ kind: "fragment-input" as const, name });+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });++export function createSpeechTrackFragment(options: SpeechTrackFragmentOptions) {+ if (options.takes.length === 0) throw new Error("Speech Track requires at least one Take");+ const declaredInputs = new Map<string, TypeRef>();+ const addInput = (name: string, type: TypeRef): void => {+ if (!name) throw new Error("Speech Track Fragment input names must not be empty");+ const previous = declaredInputs.get(name);+ if (previous !== undefined && (previous.module.name !== type.module.name+ || previous.module.version !== type.module.version || previous.name !== type.name)) {+ throw new Error(`Speech Track Fragment input ${name} is reused with another Type`);+ }+ declaredInputs.set(name, type);+ };+ addInput("header", speechTrackTypes.header);+ for (const take of options.takes) {+ addInput(take.takeName, speechTypes.semanticTake);+ addInput(take.visual.frameName, spatialTypes.frame);+ addInput(take.visual.fitName, spatialTypes.fit);+ addInput(take.visual.visualSpecName, speechTrackTypes.visualSpec);+ }+ const operations: FragmentOperation[] = [{+ id: "track:set:empty",+ producer: speechTrackProducers.createSet,+ inputs: {},+ result: { kind: "output", name: "set" },+ }];+ let current = "track:set:empty";+ options.takes.forEach((take, index) => {+ const id = `track:set:append:${String(index + 1).padStart(4, "0")}`;+ operations.push({+ id,+ producer: speechTrackProducers.appendTake,+ inputs: {+ set: operation(current),+ take: input(take.takeName),+ frame: input(take.visual.frameName),+ fit: input(take.visual.fitName),+ visualSpec: input(take.visual.visualSpecName),+ },+ result: { kind: "output", name: "set" },+ });+ current = id;+ });+ operations.push(+ {+ id: "track:semantic",+ producer: speechTrackProducers.assembleTrack,+ inputs: { header: input("header"), set: operation(current) },+ result: { kind: "output", name: "track" },+ },+ {+ id: "track:visual-track",+ producer: speechTrackProducers.projectVisual,+ inputs: { track: operation("track:semantic"), set: operation(current) },+ result: { kind: "output", name: "visual" },+ },+ {+ id: "track:audio-track",+ producer: semanticTrackProducers.projectAudio,+ inputs: { track: operation("track:semantic") },+ result: { kind: "output", name: "audio" },+ },+ );+ return sealGraphFragment({+ inputs: [...declaredInputs.entries()].map(([name, type]) => ({ name, type })),+ operations,+ exports: [+ { name: "semantic", type: semanticTrackTypes.track, root: operation("track:semantic") },+ {+ name: "visual", type: compositionTypes.visualTrack, root: operation("track:visual-track"),+ },+ {+ name: "audio", type: compositionTypes.audioTrack, root: operation("track:audio-track"),+ },+ ],+ });+}diff --git a/packages/speech-track/src/index.ts b/packages/speech-track/src/index.tsnew file mode 100644index 00000000..1880ff06--- /dev/null+++ b/packages/speech-track/src/index.ts@@ -0,0 +1,8 @@+/** Official Speech Track authoring and deterministic SemanticTrack assembly. */+export { speechTrackComponent } from "./component.js";+export { createSpeechTrackFragment } from "./fragment.js";+export { speechTrackManifest, speechTrackMarkupSurfaces, speechTrackModuleRef, speechTrackProducers, speechTrackTypes, speechTrackHeaderSchema, speechTrackSetSchema, speechTrackVisualSpecSchema } from "./manifest.js";+export { appendSpeechTrackTake, assembleSpeechTrack, assertSpeechTrackHeader, assertSpeechTrackSet, assertSpeechTrackVisualSpec, createSpeechTrackSet, sealSpeechTrackHeader, sealSpeechTrackVisualSpec } from "./program.js";+export { projectSpeechTrackVisual } from "./projection.js";+export { decodeSpeechTrackSurface } from "./surface.js";+export type * from "./types.js";diff --git a/packages/speech-spine/src/manifest.ts b/packages/speech-track/src/manifest.tssimilarity index 52%rename from packages/speech-spine/src/manifest.tsrename to packages/speech-track/src/manifest.tsindex dd6b667c..d7c02758 100644--- a/packages/speech-spine/src/manifest.ts+++ b/packages/speech-track/src/manifest.ts@@ -1,25 +1,16 @@import { readFile } from "node:fs/promises";-import { artifactDependency, artifactTypes } from "@hypit/artifact";-import { mediaDependency, mediaTypes } from "@hypit/media";-import { programSpaceDependency, programSpaceTypes } from "@hypit/program-space";import { semanticTakeSchema, speechDependency, speechTypes } from "@hypit/speech";import { compositionDependency, compositionTypes } from "@hypit/composition";-import {- mediaPipelineManifest,- mediaPipelineModuleRef,- mediaPipelineTypes,-} from "@hypit/media-pipeline";import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@hypit/protocol";-import { semanticMapDependency, semanticMapTypes } from "@hypit/semantic-map";-import { speechBasisManifest, speechBasisModuleRef } from "@hypit/speech-basis";+import { semanticTrackDependency, semanticTrackTypes } from "@hypit/semantic-track";import {contentFitSchema,spatialDependency,spatialFrameSchema,spatialTypes,} from "@hypit/spatial";-import { svsManifest, svsModuleRef, svsRecipeType } from "@hypit/svs";+import { svsModuleRef, svsRecipeType } from "@hypit/svs";const previewImage = (file: string) => ({mediaType: "image/png",@@ -27,37 +18,30 @@ const previewImage = (file: string) => ({open: async () => Uint8Array.from(await readFile(new URL(`../preview/${file}`, import.meta.url))),});-export const speechSpineModuleRef = { name: "@hypit/speech-spine", version: "1" } as const;-export const speechSpineTypes = {- spineProgram: { module: speechSpineModuleRef, name: "SpeechSpineProgram" },- spineSet: { module: speechSpineModuleRef, name: "SpeechSpineSet" },- visualSpec: { module: speechSpineModuleRef, name: "SpeechSpineVisualSpec" },+export const speechTrackModuleRef = { name: "@hypit/speech-track", version: "1" } as const;+export const speechTrackTypes = {+ header: { module: speechTrackModuleRef, name: "SpeechTrackHeader" },+ trackSet: { module: speechTrackModuleRef, name: "SpeechTrackSet" },+ visualSpec: { module: speechTrackModuleRef, name: "SpeechTrackVisualSpec" },} satisfies Record<string, TypeRef>;-export const speechSpineProducers = {- createSet: { module: speechSpineModuleRef, name: "create-spine-set" },- appendAudioTake: { module: speechSpineModuleRef, name: "append-spine-audio-take" },- appendVisualTake: { module: speechSpineModuleRef, name: "append-spine-visual-take" },- compileAudio: { module: speechSpineModuleRef, name: "compile-spine-audio" },- assembleBasis: { module: speechSpineModuleRef, name: "assemble-speech-basis" },- assembleSemanticMap: { module: speechSpineModuleRef, name: "assemble-semantic-map" },+export const speechTrackProducers = {+ createSet: { module: speechTrackModuleRef, name: "create-track-set" },+ appendTake: { module: speechTrackModuleRef, name: "append-track-take" },+ assembleTrack: { module: speechTrackModuleRef, name: "assemble-semantic-track" },+ projectVisual: { module: speechTrackModuleRef, name: "project-semantic-visual-track" },} satisfies Record<string, ProducerRef>;const string = { kind: "string", minLength: 1 } as const;-const integer = { kind: "number", integer: true, minimum: 1 } as const;const object = (fields: Readonly<Record<string, { readonly schema: ValueSchema; readonly optional?: boolean }>>): ValueSchema => ({kind: "object",fields,});-const frameRate = object({ numerator: { schema: integer }, denominator: { schema: integer } });--export const speechSpineProgramSchema: ValueSchema = object({+export const speechTrackHeaderSchema: ValueSchema = object({id: { schema: string },- frameRate: { schema: frameRate },});-export const speechSpineSetSchema: ValueSchema = object({-+export const speechTrackSetSchema: ValueSchema = object({takes: { schema: { kind: "array", items: object({semantic: { schema: semanticTakeSchema },visual: { schema: object({@@ -68,8 +52,7 @@ export const speechSpineSetSchema: ValueSchema = object({}) } },});-export const speechSpineVisualSpecSchema: ValueSchema = object({-+export const speechTrackVisualSpecSchema: ValueSchema = object({stackingOrder: { schema: { kind: "number", integer: true } },});@@ -93,24 +76,20 @@ const visualRecipeProperties = [summary: "Decides whether the placed picture is pulled back until it covers as much of the Frame as its size allows, or left exactly where the anchors and offsets put it." },] as const;-export const speechSpineMarkupSurfaces = [{- name: "spine",- tag: "Spine",+export const speechTrackMarkupSurfaces = [{+ name: "track",+ tag: "Track",mode: "structured",- outputs: [speechSpineTypes.spineProgram, speechSpineTypes.visualSpec, spatialTypes.fit,- speechTypes.basis, programSpaceTypes.programSpace,- compositionTypes.visualTrack, compositionTypes.audioTrack, semanticMapTypes.complete],+ outputs: [speechTrackTypes.header, speechTrackTypes.visualSpec, spatialTypes.fit,+ semanticTrackTypes.track,+ compositionTypes.visualTrack, compositionTypes.audioTrack],vocabulary: {- summary: "Folds ordered speech Takes into one SpeechBasis, and publishes the ProgramSpace every other Track is timed against together with the peer VisualTrack and AudioTrack the speech renders to.",- appearance: "The speaking picture itself, and the layer every other Track is stacked over. Each Take's own footage fills the Frame the Spine names, fitted by its Recipe, and the Takes run one after another in the order they are written, so the picture cuts from one to the next at each Take boundary with nothing between them. A Take may name its own Frame, so the picture can move or resize at a boundary; otherwise the framing holds. Nothing is drawn on top: titles, captions and cutaways are separate Tracks lying above this one.",- preview: previewImage("Spine.png"),+ summary: "Folds ordered SemanticTakes into one SemanticTrack, then publishes its aligned VisualTrack and AudioTrack projections.",+ appearance: "The speaking picture itself, and the layer every other Track is stacked over. Each Take's own footage fills the Frame the Track names, fitted by its Recipe, and the Takes run one after another in the order they are written, so the picture cuts from one to the next at each Take boundary with nothing between them. A Take may name its own Frame, so the picture can move or resize at a boundary; otherwise the framing holds. Nothing is drawn on top: titles, captions and cutaways are separate Tracks lying above this one.",+ preview: previewImage("Track.png"),attributes: [{ name: "id", kind: "identifier", required: true,- summary: "Names this Spine and prefixes every projection it publishes." },- { name: "clock", kind: "reference", required: false, accepts: [programSpaceTypes.clock],- summary: "Selects the authored frame clock shared by the normalized Takes." },- { name: "frame-rate", kind: "literal", required: false,- summary: "Legacy inline frame rate; write exactly one of clock or frame-rate." },+ summary: "Names this Track and prefixes every projection it publishes." },{ name: "visual-frame", kind: "reference", required: true, accepts: [spatialTypes.frame],summary: "Chooses the Frame every visual Take occupies unless the Take names its own." },{ name: "visual-appearance", kind: "reference", required: true, accepts: [svsRecipeType],@@ -124,118 +103,91 @@ export const speechSpineMarkupSurfaces = [{summary: "One self-contained SemanticTake, in document order.",attributes: [{ name: "source", kind: "reference", required: true, accepts: [speechTypes.semanticTake],- summary: "Chooses the normalized and locally aligned SemanticTake this Spine assembles." },+ summary: "Chooses the normalized and locally aligned SemanticTake this Track assembles." },{ name: "frame", kind: "reference", required: false, accepts: [spatialTypes.frame],- summary: "Chooses this Take's own Frame in place of the Spine's `visual-frame`." },+ summary: "Chooses this Take's own Frame in place of the Track's `visual-frame`." },{ name: "appearance", kind: "reference", required: false, accepts: [svsRecipeType],- summary: "Chooses this Take's own fit Recipe in place of the Spine's `visual-appearance`.",+ summary: "Chooses this Take's own fit Recipe in place of the Track's `visual-appearance`.",recipe: visualRecipeProperties },{ name: "z", kind: "literal", required: false,- summary: "Sets this Take's own stacking order in place of the Spine's `visual-z`." },+ summary: "Sets this Take's own stacking order in place of the Track's `visual-z`." },] },],ports: [- { name: "basis", type: speechTypes.basis,- summary: "The assembled SpeechBasis: the ordered Takes, their placed audio and their visual clips." },- { name: "space", type: programSpaceTypes.programSpace,- summary: "The speech coordinate space, the frame domain every other Track resolves its windows in." },+ { name: "semantic", type: semanticTrackTypes.track,+ summary: "The ordered semantic timeline whose items retain each self-contained SemanticTake." },{ name: "visual", type: compositionTypes.visualTrack,- summary: "The rendered speech picture, an ordinary peer VisualTrack." },- { name: "audioTrack", type: compositionTypes.audioTrack,- summary: "The rendered speech sound, an ordinary peer AudioTrack." },- { name: "semanticMap", type: semanticMapTypes.complete,- summary: "The complete global map assembled by translating each Take's local semantic frames." },+ summary: "The speech picture projected from the SemanticTrack, an ordinary peer VisualTrack." },+ { name: "audio", type: compositionTypes.audioTrack,+ summary: "The speech sound clips projected from the SemanticTrack, an ordinary peer AudioTrack." },],- example: `<speech:Spine id="speech" clock={clock}+ example: `<speech:Track id="speech"visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0"><speech:Take source={opening.take}/><speech:Take source={closing.take}/>-</speech:Spine>`,+</speech:Track>`,notes: [- "A Spine requires at least one Take and accepts no text content.",+ "A Track requires at least one Take and accepts no text content.","`visual-appearance` and a Take's `appearance` must each be an authored SVS Recipe declaring only the spatial fit properties listed for them; any other property is refused.",- "Media normalization, acoustic evidence and Segment alignment happen before a Take enters the Spine.",- "Placement and stacking are the complete visual authority of a Spine; motion, transitions and independent pictures remain ordinary Media Tracks.",+ "Media normalization, acoustic evidence and Segment alignment happen before a Take enters the Track.",+ "Placement and stacking are the complete visual authority of a Track; motion, transitions and independent pictures remain ordinary Media Tracks.",],},}] as const;-export const speechSpineManifest: ModuleManifest = {+export const speechTrackManifest: ModuleManifest = {format: "hypit.module@1",- name: speechSpineModuleRef.name,- version: speechSpineModuleRef.version,+ name: speechTrackModuleRef.name,+ version: speechTrackModuleRef.version,dependencies: [- artifactDependency,- mediaDependency,- programSpaceDependency,speechDependency,compositionDependency,spatialDependency,{ module: svsModuleRef },- { module: mediaPipelineModuleRef },- { module: speechBasisModuleRef },- semanticMapDependency,+ semanticTrackDependency,],types: [- { name: speechSpineTypes.spineProgram.name },- { name: speechSpineTypes.spineSet.name },- { name: speechSpineTypes.visualSpec.name },+ { name: speechTrackTypes.header.name },+ { name: speechTrackTypes.trackSet.name },+ { name: speechTrackTypes.visualSpec.name },],capabilities: [],producers: [{- name: speechSpineProducers.createSet.name,+ name: speechTrackProducers.createSet.name,inputs: [],- outputs: [{ name: "set", type: speechSpineTypes.spineSet }],+ outputs: [{ name: "set", type: speechTrackTypes.trackSet }],needs: [],},{- name: speechSpineProducers.appendAudioTake.name,+ name: speechTrackProducers.appendTake.name,inputs: [- { name: "set", type: speechSpineTypes.spineSet },- { name: "program", type: speechSpineTypes.spineProgram },- { name: "take", type: speechTypes.semanticTake },- ],- outputs: [{ name: "set", type: speechSpineTypes.spineSet }],- needs: [],- },- {- name: speechSpineProducers.appendVisualTake.name,- inputs: [- { name: "set", type: speechSpineTypes.spineSet },- { name: "program", type: speechSpineTypes.spineProgram },+ { name: "set", type: speechTrackTypes.trackSet },{ name: "take", type: speechTypes.semanticTake },{ name: "frame", type: spatialTypes.frame },{ name: "fit", type: spatialTypes.fit },- { name: "visualSpec", type: speechSpineTypes.visualSpec },+ { name: "visualSpec", type: speechTrackTypes.visualSpec },],- outputs: [{ name: "set", type: speechSpineTypes.spineSet }],+ outputs: [{ name: "set", type: speechTrackTypes.trackSet }],needs: [],},{- name: speechSpineProducers.compileAudio.name,+ name: speechTrackProducers.assembleTrack.name,inputs: [- { name: "program", type: speechSpineTypes.spineProgram },- { name: "set", type: speechSpineTypes.spineSet },+ { name: "header", type: speechTrackTypes.header },+ { name: "set", type: speechTrackTypes.trackSet },],- outputs: [{ name: "plan", type: mediaPipelineTypes.audioProgramPlan }],+ outputs: [{ name: "track", type: semanticTrackTypes.track }],needs: [],},{- name: speechSpineProducers.assembleBasis.name,+ name: speechTrackProducers.projectVisual.name,inputs: [- { name: "program", type: speechSpineTypes.spineProgram },- { name: "set", type: speechSpineTypes.spineSet },- { name: "audio", type: mediaTypes.timelineAudio },+ { name: "track", type: semanticTrackTypes.track },+ { name: "set", type: speechTrackTypes.trackSet },],- outputs: [{ name: "basis", type: speechTypes.basis }],- needs: [],- },- {- name: speechSpineProducers.assembleSemanticMap.name,- inputs: [{ name: "set", type: speechSpineTypes.spineSet }],- outputs: [{ name: "map", type: semanticMapTypes.complete }],+ outputs: [{ name: "visual", type: compositionTypes.visualTrack }],needs: [],},],diff --git a/packages/speech-track/src/program.ts b/packages/speech-track/src/program.tsnew file mode 100644index 00000000..17d3d970--- /dev/null+++ b/packages/speech-track/src/program.ts@@ -0,0 +1,128 @@+import { assertSemanticTakeIdentity } from "@hypit/speech";+import type { SemanticTake } from "@hypit/speech";+import { sealSemanticTrack } from "@hypit/semantic-track";+import type { SemanticTrack } from "@hypit/semantic-track";+import { canonicalize } from "@hypit/protocol";+import { assertContentFit, assertSpatialFrame } from "@hypit/spatial";+import type { ContentFit, SpatialFrame } from "@hypit/spatial";++import type {+ SpeechTrackHeader,+ SpeechTrackSet,+ SpeechTrackTake,+ SpeechTrackVisualSpec,+} from "./types.js";++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++function headerContent(value: SpeechTrackHeader): SpeechTrackHeader {+ return { id: value.id };+}++export function sealSpeechTrackHeader(value: SpeechTrackHeader): SpeechTrackHeader {+ return headerContent(value);+}++export function assertSpeechTrackHeader(value: SpeechTrackHeader): void {+ assert(value.id.trim().length > 0, "SpeechTrackHeader id must not be empty");+}++function setContent(value: SpeechTrackSet): SpeechTrackSet {+ return canonicalize({+ takes: value.takes,+ }) as unknown as SpeechTrackSet;+}++function sealSpeechTrackSet(value: SpeechTrackSet): SpeechTrackSet {+ return setContent(value);+}++function assertTake(take: SpeechTrackTake): void {+ assertSemanticTakeIdentity(take.semantic);+ const { media, segment } = take.semantic;+ assert((media.visual === undefined) === (take.visual === undefined),+ `Speech Segment ${segment.segmentId} visual policy does not match its media`);+ if (take.visual !== undefined) {+ assertSpatialFrame(take.visual.frame);+ assertContentFit(take.visual.fit);+ assert(Number.isSafeInteger(take.visual.stackingOrder),+ `Speech Segment ${segment.segmentId} visual stacking order is invalid`);+ }+}++export function sealSpeechTrackVisualSpec(value: SpeechTrackVisualSpec): SpeechTrackVisualSpec {+ const result = structuredClone(value);+ assertSpeechTrackVisualSpec(result);+ return result;+}++export function assertSpeechTrackVisualSpec(value: SpeechTrackVisualSpec): void {+ assert(Number.isSafeInteger(value.stackingOrder), "SpeechTrackVisualSpec stacking order is invalid");+}++export function assertSpeechTrackSet(value: SpeechTrackSet): void {+ const segments = new Set<string>();+ for (const take of value.takes) {+ assertSemanticTakeIdentity(take.semantic);+ const segmentId = take.semantic.segment.segmentId;+ assert(!segments.has(segmentId), `Speech Track repeats Segment ${segmentId}`);+ segments.add(segmentId);+ }+}++export function createSpeechTrackSet(): SpeechTrackSet {+ return sealSpeechTrackSet({+ takes: [],+ });+}++function appendTake(+ set: SpeechTrackSet,+ take: SpeechTrackTake,+): SpeechTrackSet {+ assertSpeechTrackSet(set);+ assertTake(take);+ assert(!set.takes.some((item) => item.semantic.segment.segmentId === take.semantic.segment.segmentId),+ `Speech Track repeats Segment ${take.semantic.segment.segmentId}`);+ return sealSpeechTrackSet({+ takes: [...set.takes, take],+ });+}++export function appendSpeechTrackTake(+ set: SpeechTrackSet,+ semantic: SemanticTake,+ frame: SpatialFrame,+ fit: ContentFit,+ visualSpec: SpeechTrackVisualSpec,+): SpeechTrackSet {+ assertSpatialFrame(frame);+ assertContentFit(fit);+ assertSpeechTrackVisualSpec(visualSpec);+ assertSemanticTakeIdentity(semantic);+ const take = {+ semantic,+ ...(semantic.media.visual === undefined ? {} : { visual: {+ frame: structuredClone(frame),+ fit: structuredClone(fit),+ stackingOrder: visualSpec.stackingOrder,+ } }),+ } satisfies SpeechTrackTake;+ return appendTake(set, take);+}++export function assembleSpeechTrack(+ header: SpeechTrackHeader,+ set: SpeechTrackSet,+): SemanticTrack {+ assertSpeechTrackHeader(header);+ assertSpeechTrackSet(set);+ assert(set.takes.length > 0, "Speech Track must contain at least one Take");+ for (const take of set.takes) assertTake(take);+ return sealSemanticTrack({+ id: header.id,+ items: set.takes.map((item) => ({ take: item.semantic })),+ });+}diff --git a/packages/speech-track/src/projection.ts b/packages/speech-track/src/projection.tsnew file mode 100644index 00000000..bafeeb2d--- /dev/null+++ b/packages/speech-track/src/projection.ts@@ -0,0 +1,40 @@+import { sealVisualTrack } from "@hypit/composition";+import type { VisualTrack } from "@hypit/composition";+import { lowerRestrictedSpeechVisualPresents } from "@hypit/media-track";+import { projectSemanticProgramSpace, semanticTrackSpans } from "@hypit/semantic-track";+import type { SemanticTrack } from "@hypit/semantic-track";+import { assertSpeechTrackSet } from "./program.js";+import type { SpeechTrackSet } from "./types.js";++export function projectSpeechTrackVisual(track: SemanticTrack, set: SpeechTrackSet): VisualTrack {+ assertSpeechTrackSet(set);+ const space = projectSemanticProgramSpace(track);+ const trackId = `${track.id}:visual`;+ const placementBySegment = new Map(set.takes.flatMap((item) => item.visual === undefined+ ? []+ : [[item.semantic.segment.segmentId, item.visual] as const]));+ return sealVisualTrack({+ visualIr: "hypit.visual-ir@1",+ id: trackId,+ presents: lowerRestrictedSpeechVisualPresents(+ trackId,+ space,+ semanticTrackSpans(track).flatMap(({ item, startFrame, endFrameExclusive }) => {+ const visual = item.take.media.visual;+ const placement = placementBySegment.get(item.take.segment.segmentId);+ if (visual === undefined || placement === undefined) return [];+ return [{+ id: item.take.segment.segmentId,+ span: { startFrame, endFrameExclusive },+ artifact: visual.artifact,+ extent: { widthPx: visual.width, heightPx: visual.height },+ frameRate: space.frameRate,+ frameCount: endFrameExclusive - startFrame,+ frame: placement.frame,+ fit: placement.fit,+ stackingOrder: placement.stackingOrder,+ }];+ }),+ ),+ });+}diff --git a/packages/speech-spine/src/surface.ts b/packages/speech-track/src/surface.tssimilarity index 77%rename from packages/speech-spine/src/surface.tsrename to packages/speech-track/src/surface.tsindex 5f9f85f1..a08af8a5 100644--- a/packages/speech-spine/src/surface.ts+++ b/packages/speech-track/src/surface.ts@@ -8,13 +8,12 @@ import {} from "@hypit/markup";import { sameType } from "@hypit/protocol";import { speechTypes } from "@hypit/speech";-import { programSpaceTypes, type ProgramClock } from "@hypit/program-space";import type { SvsRecipe } from "@hypit/svs";import { svsRecipeType } from "@hypit/svs";-import { createSpeechSpineFragment } from "./fragment.js";-import { speechSpineTypes } from "./manifest.js";-import { sealSpeechSpineProgram, sealSpeechSpineVisualSpec } from "./program.js";+import { createSpeechTrackFragment } from "./fragment.js";+import { speechTrackTypes } from "./manifest.js";+import { sealSpeechTrackHeader, sealSpeechTrackVisualSpec } from "./program.js";import {contentFitPropertyNames,decodeContentFitProperties,@@ -30,18 +29,6 @@ function allowedAttributes(element: StructuredElement, required: readonly string}}-function frameRate(element: StructuredElement): { readonly numerator: number; readonly denominator: number } {- const value = stringAttribute(element, "frame-rate");- const match = /^(\d+)(?:\/(\d+))?$/u.exec(value);- if (match === null) throw new Error(`${element.name}.frame-rate must be a positive rational such as 30 or 30000/1001`);- const numerator = Number(match[1]);- const denominator = Number(match[2] ?? "1");- if (!Number.isSafeInteger(numerator) || numerator <= 0 || !Number.isSafeInteger(denominator) || denominator <= 0) {- throw new Error(`${element.name}.frame-rate is invalid`);- }- return { numerator, denominator };-}-function integerAttribute(element: StructuredElement, name: string): number {const raw = stringAttribute(element, name);const value = Number(raw);@@ -96,11 +83,8 @@ function takes(element: StructuredElement): StructuredElement[] {return result;}-export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- allowedAttributes(element, ["id", "visual-frame", "visual-appearance", "visual-z"], ["clock", "frame-rate"]);- const hasClock = element.attributes.clock !== undefined;- const hasFrameRate = element.attributes["frame-rate"] !== undefined;- if (hasClock === hasFrameRate) throw new Error(`${element.name} requires exactly one of clock or frame-rate.`);+export const decodeSpeechTrackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ allowedAttributes(element, ["id", "visual-frame", "visual-appearance", "visual-z"], []);const id = stringAttribute(element, "id");const baseFrame = resolve(element, "visual-frame", spatialTypes.frame, resolveReference);const baseAppearanceReference = resolve(element, "visual-appearance", svsRecipeType, resolveReference);@@ -109,13 +93,6 @@ export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, re`${element.name}.visual-appearance`,);const baseZ = integerAttribute(element, "visual-z");- const rate = hasClock- ? (() => {- const clock = resolve(element, "clock", programSpaceTypes.clock, resolveReference);- if (clock.record?.value.kind !== "inline") throw new Error(`${element.name}.clock must be an authored Clock record.`);- return (clock.record.value.value as unknown as ProgramClock).frameRate;- })()- : frameRate(element);const frameInputNames = new Map<string, string>();const frameInputName = (value: SurfaceResolvedReference, fallback: string): string => {const key = JSON.stringify(value.ref);@@ -168,21 +145,16 @@ export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, rerange: take.range,};});- const programId = `${id}.program`;- const program = sealSpeechSpineProgram({-- id,- frameRate: rate,- });- const assembly = createSpeechSpineFragment({- name: `@hypit/speech-spine/surface/${id}@1`,+ const headerId = `${id}.header`;+ const header = sealSpeechTrackHeader({ id });+ const assembly = createSpeechTrackFragment({takes: declaredTakes.map(({ takeName, visual }) => ({takeName,- ...(visual === undefined ? {} : { visual: {+ visual: {frameName: visual.frameName,fitName: visual.fitName,visualSpecName: visual.visualSpecName,- } }),+ },})),});const emittedFits = new Set<string>();@@ -206,8 +178,8 @@ export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, reemittedSpecs.add(take.visual.visualSpecName);records.push({id: `${id}.${take.visual.visualSpecName}`,- type: speechSpineTypes.visualSpec,- value: { kind: "inline" as const, value: sealSpeechSpineVisualSpec({+ type: speechTrackTypes.visualSpec,+ value: { kind: "inline" as const, value: sealSpeechTrackVisualSpec({stackingOrder: take.visual.stackingOrder,}) },@@ -218,7 +190,7 @@ export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, re});return {records: [- { id: programId, type: speechSpineTypes.spineProgram, value: { kind: "inline", value: program }, range: element.range },+ { id: headerId, type: speechTrackTypes.header, value: { kind: "inline", value: header }, range: element.range },...visualRecords,],components: [@@ -226,7 +198,7 @@ export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, reid,fragment: assembly.id,inputs: {- program: { kind: "record", id: programId },+ header: { kind: "record", id: headerId },...Object.fromEntries(declaredTakes.flatMap((take) => [[take.takeName, take.source.ref],...(take.visual === undefined ? [] : [@@ -237,19 +209,16 @@ export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, re])),},outputs: {- basis: `${id}.basis`,- space: `${id}.space`,+ semantic: `${id}.semantic`,visual: `${id}.visual`,- audioTrack: `${id}.audioTrack`,- semanticMap: `${id}.semanticMap`,+ audio: `${id}.audio`,},range: element.range,},],fragments: [assembly],exports: [- `${id}.basis`, `${id}.space`, `${id}.visual`, `${id}.audioTrack`,- `${id}.semanticMap`,+ `${id}.semantic`, `${id}.visual`, `${id}.audio`,],};};diff --git a/packages/speech-track/src/types.ts b/packages/speech-track/src/types.tsnew file mode 100644index 00000000..658b3ad1--- /dev/null+++ b/packages/speech-track/src/types.ts@@ -0,0 +1,40 @@+import type { ContentFit, SpatialFrame } from "@hypit/spatial";++/** Author-owned policy for one ordered Speech Track. */+export type SpeechTrackHeader = {+ readonly id: string;+};++export type SpeechTrackTake = {+ readonly semantic: import("@hypit/speech").SemanticTake;+ readonly visual?: {+ readonly frame: SpatialFrame;+ readonly fit: ContentFit;+ readonly stackingOrder: number;+ };+};++export type SpeechTrackVisualSpec = {+ readonly stackingOrder: number;+};++/**+ * Package-private immutable fold value. Its order is author meaning; Core sees+ * only fixed-port append Operations for the concrete Track declaration.+ */+export type SpeechTrackSet = {+ readonly takes: readonly SpeechTrackTake[];+};++export type SpeechTrackInput = {+ readonly takeName: string;+ readonly visual: {+ readonly frameName: string;+ readonly fitName: string;+ readonly visualSpecName: string;+ };+};++export type SpeechTrackFragmentOptions = {+ readonly takes: readonly SpeechTrackInput[];+};diff --git a/packages/speech-track/test/program.test.ts b/packages/speech-track/test/program.test.tsnew file mode 100644index 00000000..d3860f26--- /dev/null+++ b/packages/speech-track/test/program.test.ts@@ -0,0 +1,66 @@+import assert from "node:assert/strict";+import test from "node:test";+import { fixtureDigest } from "../../../test/fixture-digest.js";++import type { SynchronizedMedia } from "@hypit/media";+import type { SemanticTake } from "@hypit/speech";+import {+ appendSpeechTrackTake,+ assembleSpeechTrack,+ createSpeechTrackSet,+ projectSpeechTrackVisual,+ sealSpeechTrackHeader,+ sealSpeechTrackVisualSpec,+} from "@hypit/speech-track";++const frame = { xPx: 40, yPx: 80, widthPx: 640, heightPx: 900 };+const fit = {+ sizing: "cover" as const,+ framePoint: { x: 0.5, y: 0.5 },+ contentPoint: { x: 0.5, y: 0.5 },+ offsetPx: { x: 0, y: 0 },+ constraint: "bounded" as const,+};++function semantic(id: string, visual: boolean): SemanticTake {+ const media: SynchronizedMedia = {+ timeline: { frameRate: { numerator: 30, denominator: 1 }, frameCount: 30 },+ ...(visual ? { visual: {+ artifact: { kind: "blob", digest: fixtureDigest(`${id}:video`), size: 1, mediaType: "video/mp4" },+ width: 720,+ height: 1280,+ } } : {}),+ audio: { artifact: { kind: "blob", digest: fixtureDigest(`${id}:audio`), size: 1, mediaType: "audio/wav" } },+ };+ return {+ media,+ segment: {+ segmentId: id,+ startAnchorId: `segment:${id}:start`,+ endAnchorId: `segment:${id}:end`,+ startFrame: 0,+ endFrameExclusive: 30,+ },+ tokens: [],+ anchors: [+ { identity: `segment:${id}:start`, frame: 0 },+ { identity: `segment:${id}:end`, frame: 30 },+ ],+ };+}++test("one Track assembles a continuous SemanticTrack and only visual Takes become presents", () => {+ const header = sealSpeechTrackHeader({ id: "speech" });+ const visualSpec = sealSpeechTrackVisualSpec({ stackingOrder: 30 });+ let set = createSpeechTrackSet();+ set = appendSpeechTrackTake(set, semantic("voiceover", false), frame, fit, visualSpec);+ set = appendSpeechTrackTake(set, semantic("answer", true), frame, fit, visualSpec);++ const semanticTrack = assembleSpeechTrack(header, set);+ const visualTrack = projectSpeechTrackVisual(semanticTrack, set);++ assert.deepEqual(semanticTrack.items.map((item) => item.take.segment.segmentId), ["voiceover", "answer"]);+ assert.equal(visualTrack.presents.length, 1);+ assert.equal(visualTrack.presents[0]?.id, "speech:visual:answer");+ assert.deepEqual(visualTrack.presents[0]?.span, { startFrame: 30, endFrameExclusive: 60 });+});diff --git a/packages/speech-spine/test/surface.test.ts b/packages/speech-track/test/surface.test.tssimilarity index 82%rename from packages/speech-spine/test/surface.test.tsrename to packages/speech-track/test/surface.test.tsindex fa35f5c1..ebdc002d 100644--- a/packages/speech-spine/test/surface.test.ts+++ b/packages/speech-track/test/surface.test.ts@@ -7,18 +7,18 @@ import type { SemanticTake } from "@hypit/speech";import { speechTypes } from "@hypit/speech";import { spatialTypes } from "@hypit/spatial";import { svsRecipeType } from "@hypit/svs";-import { decodeSpeechSpineSurface } from "@hypit/speech-spine";+import { decodeSpeechTrackSurface } from "@hypit/speech-track";const take: SemanticTake = {media: {timeline: { frameRate: { numerator: 30, denominator: 1 }, frameCount: 30 },visual: {- artifact: { kind: "blob", digest: fixtureDigest("spine:take:video"), size: 1, mediaType: "video/mp4" },+ artifact: { kind: "blob", digest: fixtureDigest("track:take:video"), size: 1, mediaType: "video/mp4" },width: 720,height: 1280,},audio: {- artifact: { kind: "blob", digest: fixtureDigest("spine:take:audio"), size: 1, mediaType: "audio/wav" },+ artifact: { kind: "blob", digest: fixtureDigest("track:take:audio"), size: 1, mediaType: "audio/wav" },},},segment: {@@ -61,13 +61,12 @@ function resolved(path: string): SurfaceResolvedReference | undefined {return undefined;}-test("Speech Spine accepts only already-semantic Segment Takes", async () => {+test("Speech Track accepts only already-semantic Segment Takes", async () => {const element: StructuredElement = {kind: "element",- name: "speech:Spine",+ name: "speech:Track",attributes: {id: "speech",- "frame-rate": "30","visual-frame": { kind: "reference", path: "full" },"visual-appearance": { kind: "reference", path: "speech-style" },"visual-z": "0",@@ -81,18 +80,16 @@ test("Speech Spine accepts only already-semantic Segment Takes", async () => {}],range: { start: 0, end: 70 },};- const output = await decodeSpeechSpineSurface({+ const output = await decodeSpeechTrackSurface({sourceName: "main.svml",element,resolveReference: resolved,- resolveAsset() { throw new Error("Speech Spine does not resolve assets"); },+ resolveAsset() { throw new Error("Speech Track does not resolve assets"); },});assert.deepEqual(output.components[0]?.outputs, {- basis: "speech.basis",- space: "speech.space",+ semantic: "speech.semantic",visual: "speech.visual",- audioTrack: "speech.audioTrack",- semanticMap: "speech.semanticMap",+ audio: "speech.audio",});- assert.equal(output.exports?.includes("speech.audio") ?? false, false);+ assert.equal(output.exports?.includes("speech.audio") ?? false, true);});diff --git a/packages/speech/README.md b/packages/speech/README.mdindex 298cc124..e669f5e2 100644--- a/packages/speech/README.md+++ b/packages/speech/README.md@@ -1,7 +1,7 @@# `@hypit/speech`-Public contracts for speech duration, normalized semantic Takes, assembled speech bases and provider--neutral evidence audio. A `SemanticTake` contains one normalized media product, one authored Segment,-its words and local frame anchors. `SpeechBasis` is only the assembled post-Spine product used to-project final peer VisualTrack and AudioTrack values. `SpeechEvidenceAudio` carries canonical 16 kHz-WAV bytes and their exact sample count; it never carries Script or Segment identity.+Public contracts for speech duration, normalized semantic Takes and provider-neutral evidence audio.+A `SemanticTake` contains one normalized media product, one authored Segment, its words and local+frame anchors. Ordered Takes are assembled by `@hypit/speech-track` into a `SemanticTrack`, which is+the program's continuous semantic skeleton. `SpeechEvidenceAudio` carries canonical 16 kHz WAV bytes+and their exact sample count; it never carries Script or Segment identity.diff --git a/packages/speech/package.json b/packages/speech/package.jsonindex a730f087..92a8cc29 100644--- a/packages/speech/package.json+++ b/packages/speech/package.json@@ -12,9 +12,6 @@},"dependencies": {"@hypit/media": "workspace:*",- "@hypit/program-space": "workspace:*",- "@hypit/protocol": "workspace:*",- "@hypit/semantic-map": "workspace:*",- "@hypit/spatial": "workspace:*"+ "@hypit/protocol": "workspace:*"}}diff --git a/packages/speech/src/identity.ts b/packages/speech/src/identity.tsindex 740d8243..c581dc12 100644--- a/packages/speech/src/identity.ts+++ b/packages/speech/src/identity.ts@@ -1,12 +1,6 @@import { isDigest } from "@hypit/protocol";-import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@hypit/program-space";import { verifySynchronizedMedia } from "@hypit/media";-import { assertContentFit, assertIntrinsicExtent, assertSpatialFrame } from "@hypit/spatial";-import type { SemanticTake, SpeechBasis, SpeechDuration, SpeechEvidenceAudio } from "./types.js";-function assertAudioBlob(value: SpeechBasis["audio"], label: string): void {- if (value.kind !== "blob" || !isDigest(value.digest) || !Number.isSafeInteger(value.size)- || value.size < 0 || value.mediaType !== "audio/wav") throw new Error(`${label} must be a canonical WAV BlobRef.`);-}+import type { SemanticTake, SpeechDuration, SpeechEvidenceAudio } from "./types.js";export function sealSpeechDuration(value: SpeechDuration): SpeechDuration { return value; }export function assertSpeechDurationIdentity(value: SpeechDuration): void {if (!Number.isFinite(value) || value <= 0) throw new Error("SpeechDuration is invalid.");@@ -17,7 +11,6 @@ export function speechEvidenceSampleBoundary(masterSampleBoundary: number): numbif (value > BigInt(Number.MAX_SAFE_INTEGER)) throw new Error("Speech evidence sample boundary exceeds safe arithmetic.");return Number(value);}-export function sealSpeechBasis(value: SpeechBasis): SpeechBasis { return structuredClone(value); }export function sealSpeechEvidenceAudio(value: SpeechEvidenceAudio): SpeechEvidenceAudio { return structuredClone(value); }export function sealSemanticTake(value: SemanticTake): SemanticTake { return structuredClone(value); }@@ -63,38 +56,3 @@ export function assertSpeechEvidenceAudioIdentity(value: SpeechEvidenceAudio): vthrow new Error("SpeechEvidenceAudio media identity is invalid.");}}-export function assertSpeechBasisIdentity(basis: SpeechBasis): void {- assertProgramSpaceIdentity(basis.programSpace);- assertAudioBlob(basis.audio, "SpeechBasis audio");- if (basis.segments.length === 0) throw new Error("SpeechBasis must contain at least one Segment.");- let previousEnd = 0;- const segmentIds = new Set<string>();- for (const segment of basis.segments) {- if (!segment.segmentId || segment.startFrame !== previousEnd- || !Number.isSafeInteger(segment.endFrameExclusive)- || segment.endFrameExclusive <= segment.startFrame- || segment.endFrameExclusive > programSpaceFrameCount(basis.programSpace)) {- throw new Error("SpeechBasis Segment is invalid or non-contiguous.");- }- if (segmentIds.has(segment.segmentId)) throw new Error(`SpeechBasis repeats Segment ${segment.segmentId}.`);- segmentIds.add(segment.segmentId);- previousEnd = segment.endFrameExclusive;- }- if (previousEnd !== programSpaceFrameCount(basis.programSpace)) {- throw new Error("SpeechBasis Segments do not cover ProgramSpace.");- }- const seenVisuals = new Set<string>();- for (const clip of basis.visualTrack.clips) {- if (!segmentIds.has(clip.segmentId) || seenVisuals.has(clip.segmentId)- || clip.artifact.kind !== "blob" || !isDigest(clip.artifact.digest)- || !Number.isSafeInteger(clip.artifact.size) || clip.artifact.size < 0- || !clip.artifact.mediaType.startsWith("video/")- || !Number.isSafeInteger(clip.stackingOrder)) {- throw new Error(`SpeechBasis visual clip ${clip.segmentId} is invalid.`);- }- seenVisuals.add(clip.segmentId);- assertIntrinsicExtent(clip.extent);- assertSpatialFrame(clip.frame);- assertContentFit(clip.fit);- }-}diff --git a/packages/speech/src/index.ts b/packages/speech/src/index.tsindex b79dda67..c48d94fa 100644--- a/packages/speech/src/index.ts+++ b/packages/speech/src/index.ts@@ -1,4 +1,4 @@-export { assertSemanticTakeIdentity, assertSpeechBasisIdentity, assertSpeechDurationIdentity, assertSpeechEvidenceAudioIdentity, sealSemanticTake, sealSpeechBasis, sealSpeechDuration, sealSpeechEvidenceAudio, speechEvidenceSampleBoundary } from "./identity.js";+export { assertSemanticTakeIdentity, assertSpeechDurationIdentity, assertSpeechEvidenceAudioIdentity, sealSemanticTake, sealSpeechDuration, sealSpeechEvidenceAudio, speechEvidenceSampleBoundary } from "./identity.js";export { speechDependency, speechManifest, speechModuleRef, speechTypes } from "./manifest.js";-export { semanticTakeSchema, speechBasisSchema, speechDurationSchema, speechEvidenceAudioSchema } from "./schema.js";+export { semanticTakeSchema, speechDurationSchema, speechEvidenceAudioSchema } from "./schema.js";export type * from "./types.js";diff --git a/packages/speech/src/manifest.ts b/packages/speech/src/manifest.tsindex dba40827..07c338b3 100644--- a/packages/speech/src/manifest.ts+++ b/packages/speech/src/manifest.ts@@ -1,17 +1,15 @@import { mediaDependency } from "@hypit/media";-import { programSpaceDependency } from "@hypit/program-space";import type { ModuleManifest, TypeRef } from "@hypit/protocol";-import { spatialDependency } from "@hypit/spatial";-import { semanticTakeSchema, speechBasisSchema, speechDurationSchema, speechEvidenceAudioSchema } from "./schema.js";+import { semanticTakeSchema, speechDurationSchema, speechEvidenceAudioSchema } from "./schema.js";export const speechModuleRef = { name: "@hypit/speech", version: "1" } as const;export const speechTypes = {- duration: { module: speechModuleRef, name: "SpeechDuration" }, basis: { module: speechModuleRef, name: "SpeechBasis" },+ duration: { module: speechModuleRef, name: "SpeechDuration" },evidenceAudio: { module: speechModuleRef, name: "SpeechEvidenceAudio" },semanticTake: { module: speechModuleRef, name: "SemanticTake" },} satisfies Record<string, TypeRef>;export const speechManifest: ModuleManifest = { format: "hypit.module@1", name: speechModuleRef.name, version: speechModuleRef.version,- dependencies: [mediaDependency, programSpaceDependency, spatialDependency],- types: [{ name: speechTypes.duration.name }, { name: speechTypes.basis.name },+ dependencies: [mediaDependency],+ types: [{ name: speechTypes.duration.name },{ name: speechTypes.evidenceAudio.name },{ name: speechTypes.semanticTake.name }],capabilities: [], producers: [] };diff --git a/packages/speech/src/schema.ts b/packages/speech/src/schema.tsindex 998a1e57..ca48a0c0 100644--- a/packages/speech/src/schema.ts+++ b/packages/speech/src/schema.ts@@ -1,7 +1,5 @@-import { programSpaceSchema } from "@hypit/program-space";import { synchronizedMediaSchema } from "@hypit/media";import type { ValueSchema } from "@hypit/protocol";-import { contentFitSchema, intrinsicExtentSchema, spatialFrameSchema } from "@hypit/spatial";const string = { kind: "string", minLength: 1 } as const;const number = { kind: "number", minimum: 0 } as const;const integer = { kind: "number", integer: true, minimum: 0 } as const;@@ -13,11 +11,6 @@ const audioBlobRef = object({size: { schema: integer },mediaType: { schema: { kind: "literal", value: "audio/wav" } },});-const segment = object({- segmentId: { schema: string },- startFrame: { schema: integer },- endFrameExclusive: { schema: integer },-});const semanticTakeToken = object({tokenId: { schema: string },segmentId: { schema: string },@@ -36,16 +29,6 @@ const semanticTakeSegment = object({endFrameExclusive: { schema: integer },});export const speechDurationSchema: ValueSchema = number;-export const speechBasisSchema: ValueSchema = object({- programSpace: { schema: programSpaceSchema }, audio: { schema: audioBlobRef },- visualTrack: { schema: object({ clips: { schema: { kind: "array", items: object({ segmentId: { schema: string }, artifact: { schema: object({- kind: { schema: { kind: "literal", value: "blob" } }, digest: { schema: digest }, size: { schema: integer }, mediaType: { schema: string },- }) }, extent: { schema: intrinsicExtentSchema },- frame: { schema: spatialFrameSchema }, fit: { schema: contentFitSchema },- stackingOrder: { schema: { kind: "number", integer: true } },- }) } } }) },- segments: { schema: { kind: "array", minItems: 1, items: segment } },-});export const speechEvidenceAudioSchema: ValueSchema = object({artifact: { schema: audioBlobRef },sampleFrames: { schema: { kind: "number", integer: true, minimum: 1 } },diff --git a/packages/speech/src/types.ts b/packages/speech/src/types.tsindex ea1c2acf..0394fc91 100644--- a/packages/speech/src/types.ts+++ b/packages/speech/src/types.ts@@ -1,27 +1,7 @@-import type { ProgramSpace } from "@hypit/program-space";-import type { BlobRef } from "@hypit/protocol";import type { SynchronizedMedia } from "@hypit/media";-import type { ContentFit, IntrinsicExtent, SpatialFrame } from "@hypit/spatial";-import type { SemanticTimePoint } from "@hypit/semantic-map";+import type { BlobRef } from "@hypit/protocol";/** Positive seconds. The graph port's TypeRef already carries the SpeechDuration identity. */export type SpeechDuration = number;-export type SpeechBasisSegment = {- readonly segmentId: string;- readonly startFrame: number;- readonly endFrameExclusive: number;-};-export type SpeechBasis = {- readonly programSpace: ProgramSpace; readonly audio: BlobRef;- readonly visualTrack: { readonly clips: readonly {- readonly segmentId: string;- readonly artifact: BlobRef;- readonly extent: IntrinsicExtent;- readonly frame: SpatialFrame;- readonly fit: ContentFit;- readonly stackingOrder: number;- }[] };- readonly segments: readonly SpeechBasisSegment[];-};export type SpeechEvidenceAudio = {readonly artifact: BlobRef;/** Exact 16 kHz mono PCM sample count. Format constants belong to this Type, not every value. */@@ -54,5 +34,5 @@ export type SemanticTake = {readonly endFrameExclusive: number;};readonly tokens: readonly SemanticTakeToken[];- readonly anchors: readonly SemanticTimePoint[];+ readonly anchors: readonly { readonly identity: string; readonly frame: number }[];};diff --git a/packages/studio/package.json b/packages/studio/package.jsonindex 6cc87fe8..dbee0169 100644--- a/packages/studio/package.json+++ b/packages/studio/package.json@@ -20,6 +20,7 @@"@hypit/runtime": "workspace:*","@hypit/runtime-host-node": "workspace:*","@hypit/script": "workspace:*",+ "@hypit/semantic-track": "workspace:*","@hypit/validation": "workspace:*","@hypit/video-cli": "workspace:*","@hypit/workspace": "workspace:*",diff --git a/packages/studio/src/adapters/audio.ts b/packages/studio/src/adapters/audio.tsnew file mode 100644index 00000000..d95e7da7--- /dev/null+++ b/packages/studio/src/adapters/audio.ts@@ -0,0 +1,24 @@+import type { StudioAdapter, StudioAdapterContext, StudioEntityDraft } from "./types.js";+import { childEntities, readonlyInteraction, sameSurfaceValue } from "./types.js";++type AudioTrackProgramValue = {+ readonly items?: readonly { readonly id: string; readonly window: { readonly startFrame: number; readonly endFrameExclusive: number } }[];+};++function projectAudio(context: StudioAdapterContext): readonly StudioEntityDraft[] {+ const program = sameSurfaceValue(context, "program") as AudioTrackProgramValue | undefined;+ if (program?.items === undefined) return context.generic();+ return childEntities(context, program.items.map((item) => ({+ id: item.id, startFrame: item.window.startFrame, endFrameExclusive: item.window.endFrameExclusive,+ stackOrder: Number.MIN_SAFE_INTEGER,+ })), "audio-clip", "waveform");+}++export const audioAdapters: readonly StudioAdapter[] = [+ { id: "audio-program", role: "realization", output: { type: "AudioTrackProgram", modules: ["@hypit/audio-track"] } },+ {+ id: "audio", role: "track", output: { type: "AudioTrack", surface: "track", modules: ["@hypit/audio-track"] },+ family: "audio", icon: "graphic_eq", interaction: readonlyInteraction,+ realizationPorts: ["program"], project: projectAudio,+ },+];diff --git a/packages/studio/src/adapters/generic.ts b/packages/studio/src/adapters/generic.tsnew file mode 100644index 00000000..87b766c7--- /dev/null+++ b/packages/studio/src/adapters/generic.ts@@ -0,0 +1,22 @@+import type { StudioAdapter } from "./types.js";+import { readonlyInteraction } from "./types.js";++export const genericAdapters: readonly StudioAdapter[] = [+ { id: "caption-plan", role: "caption-plan", output: { type: "CaptionPlan" } },+ {+ id: "text", role: "text", output: { type: "VisualTrack", surface: "track", modules: ["@hypit/typography-track"] },+ family: "text", icon: "title", interaction: readonlyInteraction,+ },+ {+ id: "caption", role: "caption", output: { type: "VisualTrack", surface: "track", modules: ["@hypit/caption-fine"] },+ family: "caption", icon: "subtitles", interaction: readonlyInteraction,+ dependencies: [{ type: "CaptionPlan", role: "caption-plan" }],+ },+ {+ id: "component", role: "track",+ output: { type: "VisualTrack", modules: ["@hypit/ranking", "@hypit/comment-sticker", "@hypit/screen-overlay", "@hypit/deck-track"] },+ family: "component", icon: "widgets", interaction: readonlyInteraction,+ },+ { id: "audio-fallback", role: "track", output: { type: "AudioTrack" }, family: "audio", icon: "graphic_eq", interaction: readonlyInteraction },+ { id: "visual-fallback", role: "track", output: { type: "VisualTrack" }, family: "visual", icon: "layers", interaction: readonlyInteraction },+];diff --git a/packages/studio/src/adapters/media.ts b/packages/studio/src/adapters/media.tsnew file mode 100644index 00000000..75c89408--- /dev/null+++ b/packages/studio/src/adapters/media.ts@@ -0,0 +1,34 @@+import type { StudioAdapter, StudioAdapterContext, StudioEntityDraft } from "./types.js";+import { childEntities, readonlyInteraction, sameSurfaceValue } from "./types.js";++type MediaTrackProgramValue = {+ readonly items?: readonly { readonly id: string; readonly span: { readonly startFrame: number; readonly endFrameExclusive: number }; readonly stacking: { readonly order: number } }[];+ readonly sequences?: readonly { readonly id: string; readonly span: { readonly startFrame: number; readonly endFrameExclusive: number }; readonly stacking: { readonly order: number } }[];+};++function projectMedia(context: StudioAdapterContext): readonly StudioEntityDraft[] {+ const program = sameSurfaceValue(context, "program") as MediaTrackProgramValue | undefined;+ if (program === undefined) return context.generic();+ const items = [...(program.items ?? []), ...(program.sequences ?? [])].map((item) => ({+ id: item.id, startFrame: item.span.startFrame, endFrameExclusive: item.span.endFrameExclusive,+ stackOrder: item.stacking.order,+ }));+ return childEntities(context, items, "media-item",+ context.track.type === "AudioTrack" ? "waveform" : "picture");+}++export const mediaAdapters: readonly StudioAdapter[] = [+ { id: "media-program", role: "realization", output: { type: "MediaTrackProgram", modules: ["@hypit/media-track"] } },+ {+ id: "media-visual", role: "media",+ output: { type: "VisualTrack", surface: "track", modules: ["@hypit/media-track"] },+ family: "media", icon: "movie", interaction: readonlyInteraction,+ realizationPorts: ["program"], project: projectMedia,+ },+ {+ id: "media-audio", role: "media",+ output: { type: "AudioTrack", surface: "track", modules: ["@hypit/media-track"] },+ family: "media", icon: "graphic_eq", interaction: readonlyInteraction,+ realizationPorts: ["program"], project: projectMedia,+ },+];diff --git a/packages/studio/src/adapters/ranking.ts b/packages/studio/src/adapters/ranking.tsnew file mode 100644index 00000000..1d2e1a37--- /dev/null+++ b/packages/studio/src/adapters/ranking.ts@@ -0,0 +1,65 @@+import type { StudioAdapter, StudioAdapterContext, StudioEntityDraft } from "./types.js";+import { readonlyInteraction } from "./types.js";++type RankingSchedule = {+ readonly variant?: string;+ readonly outer?: { readonly startFrame: number; readonly endFrameExclusive: number };+ readonly entries?: readonly ({ readonly itemId: string; readonly mode: "preset" } | {+ readonly itemId: string;+ readonly mode: "reveal";+ readonly active: { readonly startFrame: number; readonly endFrameExclusive: number };+ })[];+};++/** Column entities come from the public resolved Schedule, never renderer id conventions. */+function projectRanking(context: StudioAdapterContext): readonly StudioEntityDraft[] {+ const placement = context.placement;+ const scheduleRef = context.track.trace.outputPorts.find((port) => port.name === "schedule")?.ref;+ const schedule = scheduleRef === undefined ? undefined : context.values.get(scheduleRef) as RankingSchedule | undefined;+ if (placement === undefined || schedule?.outer === undefined || schedule.variant !== "column") return context.generic();+ const boardId = placement.id ?? context.track.outputRef;+ const parent = `${context.track.outputRef}:entity:${boardId}`;+ const group: StudioEntityDraft = {+ id: parent,+ authoredId: boardId,+ label: boardId,+ startFrame: schedule.outer.startFrame,+ endFrameExclusive: schedule.outer.endFrameExclusive,+ stackOrder: Math.max(...context.spans.map((span) => span.stackOrder), 0),+ elementRange: placement.range,+ renderIds: context.spans.map((span) => span.id),+ presentation: { entity: "ranking", shape: "group", depth: 0 },+ interaction: readonlyInteraction,+ };+ const children = new Map(placement.children.flatMap((child) =>+ child.id === undefined ? [] : [[child.id, child] as const]));+ const reveals = (schedule.entries ?? []).flatMap((entry): StudioEntityDraft[] => {+ if (entry.mode !== "reveal") return [];+ const child = children.get(entry.itemId);+ const markerId = child?.referenceAttributes.during?.split(".").at(-1);+ return [{+ id: `${context.track.outputRef}:entity:${entry.itemId}:reveal`,+ authoredId: entry.itemId,+ ...(markerId === undefined ? {} : { markerId }),+ label: child?.attributes.label ?? entry.itemId,+ startFrame: entry.active.startFrame,+ endFrameExclusive: entry.active.endFrameExclusive,+ stackOrder: group.stackOrder + 1,+ ...(child === undefined ? {} : { elementRange: child.range }),+ presentation: { entity: "ranking-reveal", shape: "window", parentId: parent, depth: 1 },+ interaction: readonlyInteraction,+ }];+ });+ return [group, ...reveals];+}++export const rankingAdapters: readonly StudioAdapter[] = [+ { id: "ranking-schedule", role: "realization", output: { type: "RankingSchedule", modules: ["@hypit/ranking"] } },+ {+ id: "ranking-column", role: "track",+ output: { type: "VisualTrack", surface: "column", modules: ["@hypit/ranking"] },+ family: "component", icon: "leaderboard", realizationPorts: ["schedule"],+ lane: { layout: "nested", boundFacets: false },+ interaction: readonlyInteraction, project: projectRanking,+ },+];diff --git a/packages/studio/src/adapters/speech.ts b/packages/studio/src/adapters/speech.tsnew file mode 100644index 00000000..729227dd--- /dev/null+++ b/packages/studio/src/adapters/speech.ts@@ -0,0 +1,41 @@+import type { StudioAdapter, StudioAdapterContext, StudioEntityDraft } from "./types.js";+import { childEntities, readonlyInteraction, sameSurfaceValue } from "./types.js";++type SemanticTrackValue = {+ readonly items?: readonly {+ readonly take: { readonly segment: { readonly segmentId: string; readonly startFrame: number; readonly endFrameExclusive: number } };+ }[];+};++function projectSpeech(context: StudioAdapterContext): readonly StudioEntityDraft[] {+ const semantic = sameSurfaceValue(context, "semantic") as SemanticTrackValue | undefined;+ if (semantic?.items === undefined) return context.generic();+ let cursor = 0;+ const items = semantic.items.map(({ take }) => {+ const count = take.segment.endFrameExclusive - take.segment.startFrame;+ const item = { id: take.segment.segmentId, startFrame: cursor, endFrameExclusive: cursor + count, stackOrder: 0 };+ cursor += count;+ return item;+ });+ return childEntities(context, items, "semantic-take",+ context.track.type === "AudioTrack" ? "waveform" : "picture");+}++export const speechAdapters: readonly StudioAdapter[] = [+ { id: "semantic-take", role: "semantic-take", output: { type: "SemanticTake" } },+ { id: "semantic-track", role: "semantic-track", output: { type: "SemanticTrack" } },+ {+ id: "speech-visual", role: "speech-visual",+ output: { type: "VisualTrack", surface: "track", modules: ["@hypit/speech-track"], siblingType: "SemanticTrack" },+ family: "speech", icon: "record_voice_over", interaction: readonlyInteraction,+ realizationPorts: ["semantic"], project: projectSpeech,+ lane: { layout: "flat", boundFacets: true },+ },+ {+ id: "speech-audio", role: "speech-audio",+ output: { type: "AudioTrack", surface: "track", modules: ["@hypit/speech-track"], siblingType: "SemanticTrack" },+ family: "speech", icon: "graphic_eq", interaction: readonlyInteraction,+ realizationPorts: ["semantic"], project: projectSpeech,+ lane: { layout: "flat", boundFacets: true },+ },+];diff --git a/packages/studio/src/adapters/types.ts b/packages/studio/src/adapters/types.tsnew file mode 100644index 00000000..495ecd8d--- /dev/null+++ b/packages/studio/src/adapters/types.ts@@ -0,0 +1,124 @@+import type { Placement } from "../observe.js";+import type { BuiltTrack } from "../programme.js";+import type {+ Range,+ StudioInteraction,+ StudioInspectorDescription,+ StudioLaneDescription,+ StudioTimelinePresentation,+ StudioTrackFamily,+} from "../shared.js";++export type StudioProjectionRole =+ | "semantic-take"+ | "semantic-track"+ | "caption-plan"+ | "realization"+ | "speech-visual"+ | "speech-audio"+ | "media"+ | "text"+ | "caption"+ | "track";++export type StudioSpan = {+ readonly id: string;+ readonly startFrame: number;+ readonly endFrameExclusive: number;+ readonly stackOrder: number;+};++export type StudioEntityDraft = {+ readonly id: string;+ readonly authoredId: string;+ readonly label: string;+ readonly startFrame: number;+ readonly endFrameExclusive: number;+ readonly stackOrder: number;+ readonly elementRange?: Range;+ readonly markerId?: string;+ readonly presentId?: string;+ readonly renderIds?: readonly string[];+ readonly presentation?: StudioTimelinePresentation;+ readonly interaction?: StudioInteraction;+};++export type StudioAdapterContext = {+ readonly track: BuiltTrack;+ readonly placement?: Placement;+ readonly spans: readonly StudioSpan[];+ readonly values: ReadonlyMap<string, unknown>;+ readonly generic: () => readonly StudioEntityDraft[];+};++export type StudioAdapter = {+ readonly id: string;+ readonly role: StudioProjectionRole;+ readonly output: {+ readonly type: string;+ readonly surface?: string;+ readonly modules?: readonly string[];+ readonly siblingType?: string;+ };+ readonly family?: StudioTrackFamily;+ readonly icon?: string;+ readonly realizationPorts?: readonly string[];+ readonly dependencies?: readonly {+ readonly type: string;+ readonly role: StudioProjectionRole;+ }[];+ readonly interaction?: StudioInteraction;+ readonly lane?: StudioLaneDescription;+ readonly inspector?: StudioInspectorDescription;+ readonly project?: (context: StudioAdapterContext) => readonly StudioEntityDraft[];+};++export const readonlyInteraction: StudioInteraction = {+ select: true,+ seek: "pointer",+ move: false,+ trimStart: false,+ trimEnd: false,+ canvasTransform: false,+ writeback: "none",+};++export const flatLane: StudioLaneDescription = { layout: "flat", boundFacets: false };++export const standardInspector: StudioInspectorDescription = {+ sections: ["authoring", "resolved", "composition", "identity", "run"],+};++export function sameSurfaceValue(context: StudioAdapterContext, port: string): unknown {+ const ref = context.track.trace.outputPorts.find((candidate) => candidate.name === port)?.ref;+ return ref === undefined ? undefined : context.values.get(ref);+}++export function childEntities(+ context: StudioAdapterContext,+ items: readonly { readonly id: string; readonly startFrame: number; readonly endFrameExclusive: number; readonly stackOrder: number }[],+ entity: StudioTimelinePresentation["entity"],+ shape: StudioTimelinePresentation["shape"],+): readonly StudioEntityDraft[] {+ const children = new Map(context.placement?.children.flatMap((child) =>+ child.id === undefined ? [] : [[child.id, child] as const]) ?? []);+ return items.map((item) => {+ const child = children.get(item.id);+ // A dedicated adapter does not reverse-engineer renderer ids. Only an exact+ // public identity is safe; richer renderer correspondence needs its own+ // declared realization rather than another naming convention.+ const render = context.spans.find((span) => span.id === item.id);+ return {+ id: `${context.track.outputRef}:entity:${item.id}`,+ authoredId: item.id,+ label: item.id,+ startFrame: item.startFrame,+ endFrameExclusive: item.endFrameExclusive,+ stackOrder: item.stackOrder,+ ...(child === undefined ? {} : { elementRange: child.range }),+ ...(render === undefined ? {} : { presentId: render.id, renderIds: [render.id] }),+ presentation: { entity, shape, depth: 0 },+ interaction: readonlyInteraction,+ };+ });+}diff --git a/packages/studio/src/compile.ts b/packages/studio/src/compile.tsindex 099b1dd2..b0858a7e 100644--- a/packages/studio/src/compile.ts+++ b/packages/studio/src/compile.ts@@ -1,11 +1,7 @@-import { readFileSync } from "node:fs";-import type { NodeCompiledSourceClosure } from "@hypit/compiler-node";import type { ArtifactAttachment } from "@hypit/workspace";-import type { StudioDomain } from "./domain.js";import type { Observations } from "./observe.js";-import { createObserver } from "./observe.js";export type ServedFile = {readonly mediaType: string;@@ -20,52 +16,15 @@ export type CompiledSource = {readonly exports: readonly { readonly name: string; readonly type: string; readonly ref: string }[];};-export class CompileFailure extends Error {- readonly range: { readonly start: number; readonly end: number } | undefined;-- constructor(message: string, range?: { readonly start: number; readonly end: number }) {- super(message);- this.name = "CompileFailure";- this.range = range;- }-}--async function bytesOf(attachment: ArtifactAttachment): Promise<Uint8Array> {- const chunks: Uint8Array[] = [];- let size = 0;- for await (const chunk of await attachment.open()) {- const copy = Uint8Array.from(chunk);- chunks.push(copy);- size += copy.byteLength;- }- const bytes = new Uint8Array(size);- let offset = 0;- for (const chunk of chunks) {- bytes.set(chunk, offset);- offset += chunk.byteLength;- }- return bytes;-}-/**- * Compile once through the official Node compiler. The observer wraps the- * selected Markup Surfaces only to retain source-to-output provenance.+ * Attach Studio's observations to the exact Author compilation a Run was+ * resolved against. Keeping this conversion separate lets the Run compiler and+ * Studio share one Author graph instead of compiling two look-alike graphs.*/-export async function compileSource(entryPath: string, domain: StudioDomain): Promise<CompiledSource> {- const observer = createObserver(domain.surfaces, (request) => {- const found = domain.resolveModule(request.from);- if (found === undefined) throw new CompileFailure(`No selected Source package satisfies ${request.from}.`);- return found;- });- const compiler = domain.createCompiler(observer.surfaces);- let compiled: NodeCompiledSourceClosure;- try {- compiled = await compiler.compileFile(entryPath);- } catch (error) {- const held = error as { readonly message?: string; readonly range?: { start: number; end: number } };- const last = observer.observations().placements.at(-1);- throw new CompileFailure(held.message ?? String(error), held.range ?? last?.range);- }+export async function observedCompiledSource(+ compiled: NodeCompiledSourceClosure,+ observations: Observations,+): Promise<CompiledSource> {const served = new Map<string, ServedFile>();for (const attachment of compiled.attachments) {served.set(attachment.artifact.digest, {@@ -75,7 +34,7 @@ export async function compileSource(entryPath: string, domain: StudioDomain): Pr}return {compiled,- observations: observer.observations(),+ observations,served,exports: compiled.exports.flatMap((item) => {if (!("id" in item.ref)) return [];@@ -84,6 +43,19 @@ export async function compileSource(entryPath: string, domain: StudioDomain): Pr};}-export function sourceText(path: string): string {- return readFileSync(path, "utf8");+async function bytesOf(attachment: ArtifactAttachment): Promise<Uint8Array> {+ const chunks: Uint8Array[] = [];+ let size = 0;+ for await (const chunk of await attachment.open()) {+ const copy = Uint8Array.from(chunk);+ chunks.push(copy);+ size += copy.byteLength;+ }+ const bytes = new Uint8Array(size);+ let offset = 0;+ for (const chunk of chunks) {+ bytes.set(chunk, offset);+ offset += chunk.byteLength;+ }+ return bytes;}diff --git a/packages/studio/src/observe.ts b/packages/studio/src/observe.tsindex 4ec26358..d032ecdd 100644--- a/packages/studio/src/observe.ts+++ b/packages/studio/src/observe.ts@@ -18,22 +18,31 @@ import type { Range } from "./shared.js";/** One authored element, where it was written, and what it produced. */export type Placement = {readonly tag: string;+ /** Resolved package owner; authored aliases are presentation, not identity. */+ readonly module: ModuleRef;+ /** Canonical Surface declaration name, independent of the alias used in Source. */+ readonly surface: string;readonly id?: string;readonly range: Range;/** Records this element sealed, so a value can be traced back to its tag. */readonly records: readonly string[];/** Graph outputs this element declared, named as the author would write them. */readonly outputs: readonly string[];+ /** Exact Surface output port to graph-output mapping. */+ readonly outputPorts: readonly { readonly name: string; readonly ref: string }[];/** Children the author wrote inside it, so a Clip can point at its own tag. */readonly children: readonly {readonly tag: string;readonly id?: string;readonly range: Range;+ readonly attributes: Readonly<Record<string, string>>;/** What the child itself points at, which is how it is placed. */readonly references: readonly string[];+ readonly referenceAttributes: Readonly<Record<string, string>>;}[];/** What the author wrote on it: plain text as written, references by path. */readonly attributes: Readonly<Record<string, string>>;+ readonly referenceAttributes: Readonly<Record<string, string>>;/** Paths this element and its children reference, in the order written. */readonly references: readonly string[];};@@ -68,11 +77,19 @@ function written(element: StructuredElement): Record<string, string> {/** Every whole-value reference an element points at, in written order. */function referenced(element: StructuredElement): string[] {- return Object.values(element.attributes)- .filter((value): value is { kind: "reference"; path: string } =>- typeof value === "object" && value !== null- && (value as { kind?: string }).kind === "reference")- .map((value) => value.path);+ return Object.values(referenceAttributes(element));+}++/** Whole-value references retained by attribute name for reversible interpretation. */+function referenceAttributes(element: StructuredElement): Record<string, string> {+ const held: Record<string, string> = {};+ for (const [name, value] of Object.entries(element.attributes)) {+ if (typeof value === "object" && value !== null+ && (value as { kind?: string }).kind === "reference") {+ held[name] = (value as { path: string }).path;+ }+ }+ return held;}export function createObserver(@@ -85,7 +102,7 @@ export function createObserver(// A Frontend may reach a Surface by name or by walking a module's whole list,// so both ways in are wrapped: an unwatched Surface decodes silently and the// preview loses the tag that placed the picture.- const watch = (found: RegisteredSurface | undefined): RegisteredSurface | undefined => {+ const watch = (found: RegisteredSurface | undefined, module: ModuleRef): RegisteredSurface | undefined => {// A raw Surface parses its own body and reports its own positions, so// there is nothing here to recover.if (found === undefined) return found;@@ -108,11 +125,16 @@ export function createObserver(const id = input.element.attributes.id;placements.push({tag: input.element.name,+ module: { ...module },+ surface: found.surface,...(typeof id === "string" ? { id } : {}),range: { start: input.element.range.start, end: input.element.range.end },records: output.records.map((record) => record.id),outputs: output.components.flatMap((component) => Object.values(component.outputs)),+ outputPorts: output.components.flatMap((component) =>+ Object.entries(component.outputs).map(([name, ref]) => ({ name, ref }))),attributes: written(input.element),+ referenceAttributes: referenceAttributes(input.element),references: [...referenced(input.element),...input.element.children@@ -127,7 +149,9 @@ export function createObserver(tag: child.name,...(typeof childId === "string" ? { id: childId } : {}),range: { start: child.range.start, end: child.range.end },+ attributes: written(child),references: referenced(child),+ referenceAttributes: referenceAttributes(child),};}),});@@ -138,9 +162,9 @@ export function createObserver(const watchedSurfaces: MarkupSurfaceRegistryLike = {surfaces(module) {- return surfaces.surfaces(module).map((found) => watch(found)!) as readonly RegisteredSurface[];+ return surfaces.surfaces(module).map((found) => watch(found, module)!) as readonly RegisteredSurface[];},- resolve(module, surface) { return watch(surfaces.resolve(module, surface)); },+ resolve(module, surface) { return watch(surfaces.resolve(module, surface), module); },};return {diff --git a/packages/studio/src/preview/render.ts b/packages/studio/src/preview/render.tsindex ab7928ed..ae72599f 100644--- a/packages/studio/src/preview/render.ts+++ b/packages/studio/src/preview/render.ts@@ -1,15 +1,12 @@-import { sealComposition } from "@hypit/composition";-import type { Composition, Track } from "@hypit/composition";+import type { Composition } from "@hypit/composition";import { compileHyperframesDocument, materializeHyperframesHtml } from "@hypit/hyperframes";import type { ProgramSpace } from "@hypit/program-space";import { injectRuntimeShim } from "./runtime-shim.js";export type RenderInput = {- readonly id: string;- readonly canvas: Composition["canvas"];+ readonly composition: Composition;readonly space: ProgramSpace;- readonly tracks: readonly Track[];/** Digests Studio can serve for material selected by the Run. */readonly served?: ReadonlySet<string>;/** The Track whose material carries the programme's speech. */@@ -24,11 +21,7 @@ export type RenderInput = {* material all come from the projection selected by the Run.*/export function renderPreview(input: RenderInput): string {- const document = compileHyperframesDocument(sealComposition({- id: input.id,- canvas: input.canvas,- tracks: input.tracks,- }), input.space);+ const document = compileHyperframesDocument(input.composition, input.space);const html = materializeHyperframesHtml(document, (artifact) => {// The only Artifacts a preview can reference are files the author already// has. Anything else would be a Provider's output, which does not exist yet,diff --git a/packages/studio/src/preview/runtime-shim.ts b/packages/studio/src/preview/runtime-shim.tsindex 6584e413..68b4b62a 100644--- a/packages/studio/src/preview/runtime-shim.ts+++ b/packages/studio/src/preview/runtime-shim.ts@@ -1,17 +1,9 @@/**- * The bridge between the transport bar and the composition inside the iframe.+ * Studio's frame driver for a compiled HyperFrames document.*- * A HyperFrames document drives its own motion. On load it samples every- * animated element frame by frame, cancels the CSS animation and sets- * `animation-name: none`, then waits for an `hf-seek` event carrying a time in- * seconds. Reaching for the Web Animations API here would find nothing to move,- * which is exactly how motion ends up frozen on its first keyframe.- *- * What the document does not do is decide which Presents are on screen, or run- * real material, because those are the driver's job. So this shim owns two- * things and delegates the third: it hides Presents outside their span, keeps- * material on the same instant, and asks the document to place its own- * animations.+ * HyperFrames remains the owner of layout and animation. Studio only supplies+ * an absolute Program frame, places every timed Present/material at that+ * instant, and waits until a discrete media seek is actually decoded.*/function shim(audibleTrack: string | undefined): string {return String.raw`@@ -28,109 +20,121 @@ function shim(audibleTrack: string | undefined): string {var frameSeconds = 1 / fps;var currentSeconds = 0;var playing = false;- var playbackFrame = 0;- var playbackStartedAt = 0;- var playbackHandle = 0;- var programmeDuration = 0;- var visualClips = [];+ var seekRevision = 0;if (root) {root.style.width = (root.getAttribute('data-width') || '0') + 'px';root.style.height = (root.getAttribute('data-height') || '0') + 'px';}- var clipNodes = document.querySelectorAll('.hypit-visual-present');- for (var clipIndex = 0; clipIndex < clipNodes.length; clipIndex++) {- var clipNode = clipNodes[clipIndex];- var clipStart = parseFloat(clipNode.getAttribute('data-start') || '0') || 0;- var clipDuration = parseFloat(clipNode.getAttribute('data-duration') || '0') || 0;- programmeDuration = Math.max(programmeDuration, clipStart + clipDuration);++ var visualClips = [];+ for (var clip of document.querySelectorAll('.hypit-visual-present')) {visualClips.push({- element: clipNode,- start: clipStart,- duration: clipDuration,- videos: clipNode.querySelectorAll('video'),- track: clipNode.getAttribute('data-hypit-track-id')+ element: clip,+ start: parseFloat(clip.getAttribute('data-start') || '0') || 0,+ duration: parseFloat(clip.getAttribute('data-duration') || '0') || 0+ });+ }+ // Sampling runs are separate video elements with their own absolute spans.+ // Addressing only their containing Present makes every run paint at once.+ var media = [];+ for (var element of document.querySelectorAll('.hypit-visual-present video')) {+ var present = element.closest('.hypit-visual-present');+ var ownStart = element.getAttribute('data-start');+ var ownDuration = element.getAttribute('data-duration');+ media.push({+ element: element,+ start: parseFloat(ownStart || (present && present.getAttribute('data-start')) || '0') || 0,+ duration: parseFloat(ownDuration || (present && present.getAttribute('data-duration')) || '0') || 0,+ mediaStart: parseFloat(element.getAttribute('data-media-start') || '0') || 0,+ rate: parseFloat(element.getAttribute('data-playback-rate') || '1') || 1,+ track: present && present.getAttribute('data-hypit-track-id')});}- /**- * Place the composition at one instant.- *- * Scrubbing and playing are different requests of the same material. A seek is- * a discrete jump, so material is paused and moved. Playback is continuous, so- * material runs on its own clock and is only corrected when it has drifted —- * setting currentTime every frame would ask the decoder for a fresh seek- * dozens of times a second, and the picture would fall behind the playhead- * while the transport insisted it was keeping time.- */++ function seekDecoded(element, target) {+ if (Math.abs(element.currentTime - target) < 0.0005 && element.readyState >= 2) return Promise.resolve();+ return new Promise(function (resolve) {+ var done = function () {+ element.removeEventListener('seeked', done);+ element.removeEventListener('error', done);+ resolve();+ };+ element.addEventListener('seeked', done, { once: true });+ element.addEventListener('error', done, { once: true });+ try { element.currentTime = target; }+ catch (error) { done(); }+ });+ }++ function placeHyperframes(seconds) {+ var pending = [];+ var event = new CustomEvent('hf-seek', { detail: {+ time: seconds,+ waitUntil: function (promise) { pending.push(Promise.resolve(promise)); }+ }});+ window.dispatchEvent(event);+ var compositionId = root && root.getAttribute('data-composition-id');+ var timeline = compositionId && window.__timelines && window.__timelines[compositionId];+ if (timeline) {+ if (typeof timeline.pause === 'function') timeline.pause();+ if (typeof timeline.totalTime === 'function') timeline.totalTime(seconds);+ else if (typeof timeline.seek === 'function') timeline.seek(seconds);+ }+ return Promise.all(pending);+ }+function apply(seconds, scrubbing) {currentSeconds = Math.max(0, seconds);- for (var i = 0; i < visualClips.length; i++) {- var record = visualClips[i];- var clip = record.element;- var local = seconds - record.start;+ var revision = ++seekRevision;+ var waits = [];+ for (var record of visualClips) {+ var local = currentSeconds - record.start;+ record.element.style.visibility = local >= 0 && local < record.duration ? 'visible' : 'hidden';+ }+ for (var record of media) {+ var local = currentSeconds - record.start;var inside = local >= 0 && local < record.duration;- var lastFrame = inside && local >= record.duration - frameSeconds - 0.000001;- clip.style.visibility = inside ? 'visible' : 'hidden';- var media = record.videos;- for (var m = 0; m < media.length; m++) {- var element = media[m];- var mediaStart = parseFloat(element.getAttribute('data-media-start') || '0') || 0;- var rate = parseFloat(element.getAttribute('data-playback-rate') || '1') || 1;- // Seek the centre of the requested source frame. An exact frame- // boundary lets the decoder choose either neighbouring picture.- var target = mediaStart + (Math.max(0, local) + frameSeconds / 2) * rate;- if (Number.isFinite(element.duration) && element.duration > 0) {- target = Math.min(target, Math.max(0, element.duration - 0.001));+ var element = record.element;+ element.style.visibility = inside ? 'visible' : 'hidden';+ element.muted = muted || scrubbing || audible === null || record.track !== audible;+ if (!inside) { element.pause(); continue; }+ var target = record.mediaStart + (local + frameSeconds / 2) * record.rate;+ if (Number.isFinite(element.duration) && element.duration > 0) {+ target = Math.min(target, Math.max(0, element.duration - 0.001));+ }+ if (scrubbing || local >= record.duration - frameSeconds - 0.000001) {+ element.pause();+ waits.push(seekDecoded(element, target));+ } else {+ // During continuous playback the decoder owns its clock. A frame event+ // only corrects visible drift; making every refresh a media seek is what+ // previously froze the preview between frames.+ if (Math.abs(element.currentTime - target) > 0.08) element.currentTime = target;+ element.playbackRate = record.rate;+ if (element.paused) {+ var started = element.play();+ if (started) started.catch(function () {});}- // HyperFrames renders a silent picture, because programme audio is a- // separate Track the media pipeline muxes in. A preview is not a- // render, and placing B-roll against speech means hearing the speech,- // so the Spine's own material is allowed to sound.- element.muted = muted || scrubbing || audible === null || record.track !== audible;- try {- if (!inside) { element.pause(); continue; }- // Do not let a video run through its media end. Some decoders expose- // the first decoded picture again after ended; pinning the final- // Program frame keeps the Segment transition frame-pure.- if (scrubbing || lastFrame) { element.pause(); element.currentTime = target; continue; }- // Quarter of a second is below the threshold of noticing and well- // above the jitter of a decoder keeping its own time.- if (Math.abs(element.currentTime - target) > 0.25) element.currentTime = target;- element.playbackRate = rate;- if (element.paused) { var played = element.play(); if (played) played.catch(function () {}); }- } catch (error) { /* the media is not ready yet */ }}}- // The document's own runtimes place every animated element and every text- // flow from their sampled timelines.- window.dispatchEvent(new CustomEvent('hf-seek', { detail: { time: seconds } }));- }- function stopClock() {- playing = false;- if (playbackHandle) cancelAnimationFrame(playbackHandle);- playbackHandle = 0;- }- function tick(now) {- if (!playing) return;- var seconds = playbackFrame / fps + Math.max(0, now - playbackStartedAt) / 1000;- if (programmeDuration > 0 && seconds >= programmeDuration) {- apply(Math.max(0, programmeDuration - frameSeconds), true);- stopClock();- return;- }- apply(seconds, false);- playbackHandle = requestAnimationFrame(tick);+ waits.push(placeHyperframes(currentSeconds));+ return Promise.all(waits).then(function () { return revision === seekRevision; });}+window.__hypitSetMuted = function (value) {muted = !!value;- apply(currentSeconds, !playing);+ return apply(currentSeconds, !playing);};window.__hypitSeekFrame = function (frame) {- stopClock();- apply(frame / fps, true);+ playing = false;+ return apply(frame / fps, true);+ };+ window.__hypitPlayFrame = function (frame) {+ playing = true;+ return apply(frame / fps, false);};- window.__hypitPlayFrame = function (frame) { apply(frame / fps, false); };- apply(0, true);- window.addEventListener('load', function () { apply(0, true); });+ window.__hypitFrameReady = apply(0, true);+ window.addEventListener('load', function () { window.__hypitFrameReady = apply(currentSeconds, true); });})();</script>`;}diff --git a/packages/studio/src/programme.ts b/packages/studio/src/programme.tsindex b6275014..2ba390f6 100644--- a/packages/studio/src/programme.ts+++ b/packages/studio/src/programme.ts@@ -1,14 +1,22 @@import type { ArtifactAttachment } from "@hypit/workspace";-import type { Digest, StoredValue } from "@hypit/protocol";+import type { BlobRef, Digest, StoredValue } from "@hypit/protocol";+import type { Composition } from "@hypit/composition";+import {+ projectSemanticProgramSpace,+ semanticTrackSpans,+} from "@hypit/semantic-track";+import type { SemanticTrack } from "@hypit/semantic-track";import type { StudioArchive } from "./archive.js";import type { CompiledSource, ServedFile } from "./compile.js";import type { StudioDomain } from "./domain.js";import { executeDeterministic, MemoryArtifactStore } from "./execute.js";import type { RunPlan } from "./run.js";+import type { StudioProjection } from "./studio-preflight.js";+import type { StudioProjectionRole } from "./studio-registry.js";const PLAYABLE = new Set(["VisualTrack", "AudioTrack"]);-const TIMING = "CompleteSemanticMap";+const TIMING = "SemanticTrack";export type BuiltTrack = {readonly name: string;@@ -16,12 +24,17 @@ export type BuiltTrack = {readonly outputRef: string;readonly candidateId?: string;readonly candidateOrigin: "run" | "source" | "none";+ readonly role: StudioProjectionRole;+ readonly trace: StudioProjection["trace"];readonly track: unknown;};export type Preview = {readonly source: CompiledSource;readonly tracks: readonly BuiltTrack[];+ /** Resolved adapter realizations keyed by exact graph output ref. */+ readonly values: ReadonlyMap<string, unknown>;+ readonly composition: Composition;readonly timing: "measured";readonly timingOutput?: { readonly name: string; readonly ref: string };readonly timingCandidateId?: string;@@ -38,40 +51,6 @@ export type Preview = {}[];};-function inlineRecord(compiled: unknown, id: string): unknown {- const records = (compiled as {- readonly program: {- readonly records: readonly { readonly id: string; readonly value: StoredValue }[];- };- }).program.records;- const found = records.find((record) => record.id === id);- return found?.value.kind === "inline" ? found.value.value : undefined;-}--function canvasOf(compiled: unknown): { width: number; height: number; clearColor: string } {- const records = (compiled as {- readonly program: { readonly records: readonly { readonly value: StoredValue }[] };- }).program.records;- let clearColor = "#000000";- let size: { width: number; height: number } | undefined;- for (const record of records) {- if (record.value.kind !== "inline") continue;- const held = record.value.value as {- readonly contract?: string;- readonly widthPx?: number;- readonly heightPx?: number;- readonly clearColor?: string;- readonly canvas?: { readonly clearColor?: string };- };- clearColor = held.clearColor ?? held.canvas?.clearColor ?? clearColor;- if (held.contract === "hypit.canvas-space@1"- && typeof held.widthPx === "number" && typeof held.heightPx === "number") {- size = { width: held.widthPx, height: held.heightPx };- }- }- return { ...(size ?? { width: 1080, height: 1920 }), clearColor };-}-async function bytesOf(attachment: ArtifactAttachment): Promise<Uint8Array> {const chunks: Uint8Array[] = [];let size = 0;@@ -91,25 +70,36 @@ async function bytesOf(attachment: ArtifactAttachment): Promise<Uint8Array> {type ExecutionState = Awaited<ReturnType<typeof executeDeterministic>>["state"];-function selectedValue(state: ExecutionState, output: string): StoredValue | undefined {+function selectedValue(+ state: ExecutionState,+ output: string,+): StoredValue | undefined {const selection = state.plan.selections.find((item) => item.output === output);if (selection === undefined) return undefined;- return state.records.find((item) => item.id === selection.record)?.value- ?? state.program.records.find((item) => item.id === selection.record)?.value;+ const executed = state.records.find((item) => item.id === selection.record)?.value;+ if (executed !== undefined) return executed;+ // Inline Source and Run candidates both belong to this freshly recompiled+ // revision. There is no stale-plan distinction to reconstruct here.+ return state.program.records.find((item) => item.id === selection.record)?.value;}-function programSpace(values: readonly StoredValue[]): unknown {- for (const stored of values) {- if (stored.kind !== "inline") continue;- const value = stored.value as {- readonly durationSec?: unknown;- readonly frameRate?: { readonly numerator?: unknown; readonly denominator?: unknown };- };- if (typeof value.durationSec === "number"- && typeof value.frameRate?.numerator === "number"- && typeof value.frameRate.denominator === "number") return value;- }- throw new Error("Studio projection produced no ProgramSpace.");+function compositionArtifacts(composition: Composition): readonly BlobRef[] {+ const found = new Map<Digest, BlobRef>();+ const visit = (value: unknown): void => {+ if (value === null || typeof value !== "object") return;+ if ((value as { readonly kind?: unknown }).kind === "blob") {+ const artifact = value as BlobRef;+ found.set(artifact.digest, artifact);+ return;+ }+ if (Array.isArray(value)) {+ for (const item of value) visit(item);+ return;+ }+ for (const item of Object.values(value)) visit(item);+ };+ visit(composition);+ return [...found.values()];}/** Build only the Studio-approved deterministic projection of one explicit Run. */@@ -118,6 +108,8 @@ export async function preview(input: {readonly run: RunPlan;readonly domain: StudioDomain;readonly outputRefs: readonly string[];+ readonly compositionRef: string;+ readonly projections: readonly StudioProjection[];readonly archive?: StudioArchive;}): Promise<Preview> {const targets = input.source.exports.filter((item) => input.outputRefs.includes(item.ref));@@ -163,39 +155,30 @@ export async function preview(input: {const satisfactions = new Map(input.run.run.graph.satisfactions.map((item) => [item.output, item.candidate]),);- const values = targets- .map((target) => selectedValue(executed.state, target.ref))- .filter((value): value is StoredValue => value !== undefined);const timingOutput = targets.find((target) => target.type === TIMING);const timingValue = timingOutput === undefined? undefined: selectedValue(executed.state, timingOutput.ref);if (timingOutput === undefined || timingValue?.kind !== "inline") {- throw new Error("Studio requires a resolved CompleteSemanticMap projection.");+ throw new Error("Studio requires a resolved SemanticTrack projection.");}- const map = timingValue.value as {- readonly anchors?: readonly { readonly identity: string; readonly frame: number }[];- };- const anchors = new Map((map.anchors ?? []).map((anchor) => [anchor.identity, anchor.frame]));- const narrative = input.source.exports.find((item) => item.type === "Narrative");- const narrativeValue = narrative === undefined ? undefined : inlineRecord(- input.source.compiled,- narrative.ref,- ) as {- readonly tokens?: readonly {- readonly id: string;- readonly startAnchorId: string;- readonly endAnchorId: string;- }[];- } | undefined;- const space = programSpace([- ...values,- ...executed.state.records.map((item) => item.value),- ...executed.state.program.records.map((item) => item.value),- ]);- const rate = (space as {- readonly frameRate: { readonly numerator: number; readonly denominator: number };- }).frameRate;+ const semantic = timingValue.value as unknown as SemanticTrack;+ const spans = semanticTrackSpans(semantic);+ const anchors = new Map(spans.flatMap(({ item, startFrame }) =>+ item.take.anchors.map((anchor) => [anchor.identity, startFrame + anchor.frame] as const)));+ const space = projectSemanticProgramSpace(semantic);+ const rate = space.frameRate;+ const compositionValue = selectedValue(executed.state, input.compositionRef);+ if (compositionValue?.kind !== "inline") {+ throw new Error("Studio Film composition did not produce an inline Composition.");+ }+ const composition = compositionValue.value as unknown as Composition;+ for (const artifact of compositionArtifacts(composition)) {+ if (served.has(artifact.digest)) continue;+ const bytes = await input.archive?.read(artifact.digest);+ if (bytes !== undefined) served.set(artifact.digest, { mediaType: artifact.mediaType, bytes });+ }+ const projectionByRef = new Map(input.projections.map((projection) => [projection.ref, projection]));const tracks: BuiltTrack[] = targets.flatMap((target) => {if (!PLAYABLE.has(target.type)) return [];const stored = selectedValue(executed.state, target.ref);@@ -203,28 +186,45 @@ export async function preview(input: {if (stored?.kind !== "inline") {throw new Error(`Studio projection ${target.name} did not produce an inline Track.`);}+ const projection = projectionByRef.get(target.ref);+ if (projection === undefined) {+ throw new Error(`Studio projection ${target.name} has no Studio trace.`);+ }return [{name: target.name,type: target.type,outputRef: target.ref,...(candidateId === undefined ? {} : { candidateId }),candidateOrigin: candidateId === undefined ? "source" as const : "run" as const,+ role: projection.role,+ trace: projection.trace,track: stored.value,}];});+ const values = new Map<string, unknown>();+ for (const target of targets) {+ const stored = selectedValue(executed.state, target.ref);+ if (stored?.kind === "inline") values.set(target.ref, stored.value);+ }const timingCandidateId = satisfactions.get(timingOutput.ref);return {source: input.source,tracks,+ values,+ composition,timing: "measured",timingOutput: { name: timingOutput.name, ref: timingOutput.ref },...(timingCandidateId === undefined ? {} : { timingCandidateId }),timingCandidateOrigin: timingCandidateId === undefined ? "source" : "run",served,- canvas: canvasOf(input.source.compiled),+ canvas: composition.canvas,frameRate: rate,space,anchors,- tokens: narrativeValue?.tokens ?? [],+ tokens: spans.flatMap(({ item }) => item.take.tokens.map((token) => ({+ id: token.tokenId,+ startAnchorId: token.startAnchorId,+ endAnchorId: token.endAnchorId,+ }))),};}diff --git a/packages/studio/src/run.ts b/packages/studio/src/run.tsindex 818b5d02..7102dfe0 100644--- a/packages/studio/src/run.ts+++ b/packages/studio/src/run.ts@@ -10,7 +10,10 @@ import {} from "@hypit/run";import type { StudioArchive } from "./archive.js";+import { observedCompiledSource } from "./compile.js";+import type { CompiledSource } from "./compile.js";import type { StudioDomain } from "./domain.js";+import { createObserver } from "./observe.js";export type RunCompilation = {readonly graph: {@@ -23,6 +26,8 @@ export type RunCompilation = {export type RunPlan = {readonly authorSource: string;+ /** The observed Author graph used by this exact Run compilation. */+ readonly source: CompiledSource;readonly run: RunCompilation;readonly targets: readonly string[];readonly attachments: readonly ArtifactAttachment[];@@ -43,8 +48,13 @@ export async function loadStudioRun(input: {frontends.register(frontend);}}+ const observer = createObserver(input.domain.surfaces, (request) => {+ const found = input.domain.resolveModule(request.from);+ if (found === undefined) throw new Error(`No selected Source package satisfies ${request.from}.`);+ return found;+ });const compiler = new NodeRunCompiler({- authorCompiler: input.domain.compiler,+ authorCompiler: input.domain.createCompiler(observer.surfaces),fragments,frontends,...(input.archive === undefined ? {} : {@@ -52,9 +62,11 @@ export async function loadStudioRun(input: {}),});const compiled = await compiler.compileFile(input.run);+ const source = await observedCompiledSource(compiled.author, observer.observations());const targets = compiled.run.graph.targets.map((target) => target.output);return {authorSource: compiled.authorSource,+ source,run: compiled.run as RunCompilation,targets,attachments: compiled.attachments,diff --git a/packages/studio/src/server.ts b/packages/studio/src/server.tsindex b4eff8fa..1b2c00cf 100644--- a/packages/studio/src/server.ts+++ b/packages/studio/src/server.ts@@ -1,12 +1,13 @@+import { watch } from "node:fs";+import type { FSWatcher } from "node:fs";import { writeFile } from "node:fs/promises";-import { resolve } from "node:path";import type { Plugin, ViteDevServer } from "vite";import type { StudioArchive } from "./archive.js";import type { ServedFile } from "./compile.js";import type { StudioDomain } from "./domain.js";-import type { RunPlan } from "./run.js";+import { loadStudioRun } from "./run.js";import { readStudioSession } from "./session.js";import type { Range, StudioFailure, StudioSnapshot } from "./shared.js";@@ -14,7 +15,6 @@ export type StudioPluginOptions = {readonly source: string;readonly runPath: string;readonly domain: StudioDomain;- readonly run: RunPlan;readonly archive?: StudioArchive;};@@ -44,22 +44,48 @@ export function studioPlugin(options: StudioPluginOptions): Plugin {let revision = 0;let server: ViteDevServer | undefined;let timer: ReturnType<typeof setTimeout> | undefined;+ let requestedRevision = 0;+ let currentSource = options.source;+ const watched = new Map<string, FSWatcher>();- const publish = async (): Promise<void> => {- revision += 1;+ const watchSource = (path: string): void => {+ if (watched.has(path)) return;try {+ watched.set(path, watch(path, () => schedule()));+ } catch {+ // Some Hosts may report virtual Source ids. They are still recompiled+ // whenever a real Source revision is scheduled; they simply emit no file event.+ }+ };++ const publish = async (attempt: number): Promise<void> => {+ try {+ // SVML and SVRun form one Studio source of truth. Recompile both for+ // every revision so a new Author graph is never executed through an old+ // Run plan.+ const run = await loadStudioRun({+ run: options.runPath,+ domain: options.domain,+ ...(options.archive === undefined ? {} : { archive: options.archive }),+ });+ currentSource = run.authorSource;+ watchSource(options.runPath);+ for (const unit of run.source.compiled.closure.units) watchSource(unit.id);const result = await readStudioSession({- source: options.source,domain: options.domain,- run: options.run,+ run,...(options.archive === undefined ? {} : { archive: options.archive }),- revision,+ revision: attempt,});+ if (attempt !== requestedRevision) return;+ revision = attempt;snapshot = result.snapshot;material = result.material;failure = undefined;server?.ws.send({ type: "custom", event: "studio:snapshot", data: snapshot });} catch (error) {+ if (attempt !== requestedRevision) return;+ revision = attempt;const range = rangeOf(error);failure = {revision,@@ -71,19 +97,17 @@ export function studioPlugin(options: StudioPluginOptions): Plugin {};const schedule = (): void => {+ const attempt = ++requestedRevision;if (timer !== undefined) clearTimeout(timer);- timer = setTimeout(() => void publish(), 80);+ timer = setTimeout(() => void publish(attempt), 80);};return {name: "hypit-studio",configureServer(value) {server = value;- value.watcher.add([options.source, options.runPath]);- value.watcher.on("change", (path) => {- const changed = resolve(path);- if (changed === options.source || changed === options.runPath || path.endsWith(".svs")) schedule();- });+ watchSource(options.runPath);+ watchSource(currentSource);value.middlewares.use((request, response, next) => {const url = new URL(request.url ?? "/", "http://studio.hypit.local");if (request.method === "PUT" && url.pathname === "/__studio/source") {@@ -103,8 +127,8 @@ export function studioPlugin(options: StudioPluginOptions): Plugin {json(response, 409, { error: "The Source changed outside Studio." });return;}- await writeFile(options.source, body.text, "utf8");- await publish();+ await writeFile(currentSource, body.text, "utf8");+ schedule();response.statusCode = 202;response.end();} catch (error) {@@ -119,9 +143,17 @@ export function studioPlugin(options: StudioPluginOptions): Plugin {}if (url.pathname === "/__studio/session") {void (async () => {- if (snapshot === undefined && failure === undefined) await publish();- if (snapshot !== undefined) json(response, 200, snapshot);- else json(response, 500, failure);+ if (snapshot === undefined && failure === undefined) {+ const attempt = ++requestedRevision;+ await publish(attempt);+ }+ if (failure !== undefined && (snapshot === undefined || failure.revision > snapshot.revision)) {+ json(response, 500, failure);+ } else if (snapshot !== undefined) {+ json(response, 200, snapshot);+ } else {+ json(response, 500, failure);+ }})();return;}@@ -144,6 +176,10 @@ export function studioPlugin(options: StudioPluginOptions): Plugin {next();});},- async closeBundle() { await options.archive?.close(); },+ async closeBundle() {+ for (const watcher of watched.values()) watcher.close();+ watched.clear();+ await options.archive?.close();+ },};}diff --git a/packages/studio/src/session.ts b/packages/studio/src/session.tsindex 312a7799..5cb430d5 100644--- a/packages/studio/src/session.ts+++ b/packages/studio/src/session.ts@@ -1,7 +1,6 @@import { readFileSync } from "node:fs";import type { StudioArchive } from "./archive.js";-import { compileSource } from "./compile.js";import type { ServedFile } from "./compile.js";import type { StudioDomain } from "./domain.js";import type { Observations } from "./observe.js";@@ -21,38 +20,37 @@ export type StudioSession = {};export async function readStudioSession(input: {- readonly source: string;readonly domain: StudioDomain;readonly run: RunPlan;readonly archive?: StudioArchive;readonly revision: number;}): Promise<StudioSession> {- const source = await compileSource(input.source, input.domain);+ const source = input.run.source;const inspection = inspectStudioRun(source, input.run);- const outputRefs = inspection.projections.map((projection) => projection.ref);+ const outputRefs = [+ inspection.filmComposition,+ ...inspection.projections.map((projection) => projection.ref),+ ];const built = await preview({source,run: input.run,domain: input.domain,outputRefs,+ compositionRef: inspection.filmComposition,+ projections: inspection.projections,...(input.archive === undefined ? {} : { archive: input.archive }),});- const tracks = built.tracks- .map((track) => track.track)- .filter((track): track is object => track !== undefined);const rendered = renderPreview({- id: "studio-preview",- canvas: built.canvas,+ composition: built.composition,space: built.space as never,- tracks: tracks as never,served: new Set(built.served.keys()),...(audible(built) === undefined ? {} : { audibleTrack: audible(built)! }),});- const text = readFileSync(input.source, "utf8");+ const text = readFileSync(input.run.authorSource, "utf8");return {snapshot: snapshot(built, {revision: input.revision,- path: input.source,+ path: input.run.authorSource,text,canvas: built.canvas,frameRate: built.frameRate,diff --git a/packages/studio/src/shared.ts b/packages/studio/src/shared.tsindex 6fabf2a3..223b1a1c 100644--- a/packages/studio/src/shared.ts+++ b/packages/studio/src/shared.ts@@ -9,6 +9,66 @@ export type Range = { readonly start: number; readonly end: number };export type CandidateOrigin = "run" | "source" | "none";export type CandidateStatus = "resolved" | "unresolved";+export type StudioTrackFamily =+ | "speech"+ | "media"+ | "text"+ | "caption"+ | "component"+ | "audio"+ | "visual";++export type StudioTimelinePresentation = {+ /** What one selectable box means, never how the renderer happened to split it. */+ readonly entity: "present" | "audio-clip" | "semantic-take" | "media-item" | "caption-cue" | "ranking" | "ranking-reveal";+ readonly shape: "block" | "picture" | "waveform" | "words" | "group" | "window";+ readonly parentId?: string;+ readonly depth: number;+};++export type StudioInteraction = {+ readonly select: boolean;+ readonly seek: "start" | "pointer" | "none";+ readonly move: boolean;+ readonly trimStart: boolean;+ readonly trimEnd: boolean;+ readonly canvasTransform: boolean;+ readonly writeback: "source" | "none";+};++export type StudioLaneDescription = {+ readonly layout: "flat" | "nested";+ readonly boundFacets: boolean;+};++export type StudioInspectorSection =+ | "authoring"+ | "resolved"+ | "material"+ | "composition"+ | "identity"+ | "run";++export type StudioInspectorDescription = {+ readonly sections: readonly StudioInspectorSection[];+};++/** Studio-owned interpretation of a terminal projection. */+export type StudioTrackBinding = {+ readonly family: StudioTrackFamily;+ readonly facet: "visual" | "audio";+ /** Visual/audio facets from one authored element share this identity. */+ readonly groupId: string;+ readonly icon: string;+ /** Stable Studio-local adapter id; fallback adapters remain explicit too. */+ readonly adapter: string;+ readonly lane: StudioLaneDescription;+ readonly inspector: StudioInspectorDescription;+ readonly authoredTag?: string;+ readonly references: readonly { readonly name: string; readonly type: string }[];+ readonly interaction: StudioInteraction;+};+/*** The graph edge that made a Studio projection exist. This is deliberately* small: it identifies the Run output and candidate without pulling the whole@@ -30,8 +90,10 @@ export type CandidateProvenance = {* moves it and only the picture knows where it ended up.*/export type Clip = {- /** The Present's own id, identical to the preview's data-hypit-present-id. */+ /** Studio identity. Output-qualified so sibling Track clips cannot collide. */readonly id: string;+ /** Renderer identity, present only when this clip paints a Visual Present. */+ readonly presentId?: string;/** The authored id this Present is named after, when it names one. */readonly authoredId: string;/** The Script marker that placed it, when something said put it there. */@@ -42,14 +104,20 @@ export type Clip = {/** Where that authored tag was written. */readonly elementRange?: Range;readonly stackOrder: number;+ readonly presentation: StudioTimelinePresentation;+ readonly interaction: StudioInteraction;+ /** Rendering identities implementing this author entity; optional for non-visual entities. */+ readonly renderIds: readonly string[];};export type Track = {+ /** Exact LogicalOutput ref; labels are not identities. */readonly id: string;readonly label: string;/** Render order in the timeline; 0 is the top row. */readonly row: number;readonly clips: readonly Clip[];+ readonly binding: StudioTrackBinding;/** Which resolved Run candidate produced this Track, or why it did not. */readonly provenance: CandidateProvenance;};@@ -122,7 +190,7 @@ export type SemanticTimeline = {readonly anchors: readonly SemanticAnchor[];readonly segments: readonly SemanticSegment[];readonly tokens: readonly SemanticToken[];- /** The CompleteSemanticMap candidate that supplied these frame anchors. */+ /** The SemanticTrack candidate that supplied these frame anchors. */readonly provenance: CandidateProvenance;};diff --git a/packages/studio/src/snapshot.ts b/packages/studio/src/snapshot.tsindex d668e397..09def553 100644--- a/packages/studio/src/snapshot.ts+++ b/packages/studio/src/snapshot.ts@@ -17,6 +17,12 @@ import type {} from "./shared.js";import type { Placement } from "./observe.js";import type { Preview } from "./programme.js";+import {+ bindStudioTrack,+ projectStudioTrack,+ sealStudioClip,+} from "./studio-registry.js";+import type { StudioEntityDraft } from "./studio-registry.js";type Present = {readonly id: string;@@ -155,7 +161,7 @@ function inlineRecord(compiled: unknown, id: string): unknown {/*** Project the compiled Narrative into the frame domain that the preview is* already using. No frontend timing is invented here: if an anchor is absent- * from the built map, the corresponding item is simply not drawable yet.+ * from the built SemanticTrack, the corresponding item is simply not drawable yet.*/function semanticTimeline(built: Preview, script: ScriptMap | undefined): SemanticTimeline | undefined {const exported = built.source.exports.find((item) => item.type === "Narrative");@@ -204,7 +210,7 @@ function semanticTimeline(built: Preview, script: ScriptMap | undefined): Semant});if (segments.length === 0 && tokens.length === 0) return undefined;const provenance: CandidateProvenance = {- output: built.timingOutput?.name ?? "CompleteSemanticMap",+ output: built.timingOutput?.name ?? "SemanticTrack",...(built.timingOutput?.ref === undefined ? {} : { outputRef: built.timingOutput.ref }),...(built.timingCandidateId === undefined ? {} : { candidateId: built.timingCandidateId }),origin: built.timingCandidateOrigin,@@ -269,7 +275,13 @@ export function snapshot(built: Preview, input: {};const tracks: Track[] = [];for (const item of built.tracks) {- const clips: Clip[] = spans(item.track, input.frameRate).map((span) => {+ const projectedSpans = spans(item.track, input.frameRate);+ const binding = bindStudioTrack(item);+ const placement = built.source.observations.placements.find((candidate) =>+ candidate.id === item.trace.authoredId+ && candidate.module.name === item.trace.module+ && candidate.surface === item.trace.surface);+ const generic = (): readonly StudioEntityDraft[] => projectedSpans.map((span) => {const where = locate(span.id, located);const named = [...new Set(span.id.split(/[:#+]/u))].find((part) => markers.has(part));const marker = named ?? (where === undefined ? undefined : markerFor(where.id));@@ -278,7 +290,8 @@ export function snapshot(built: Preview, input: {// make them one.const identity = where?.id ?? marker ?? item.name;return {- id: span.id,+ id: `${item.outputRef}:${span.id}`,+ ...(item.type === "VisualTrack" ? { presentId: span.id } : {}),authoredId: identity,...(marker === undefined ? {} : { markerId: marker }),label: where?.id ?? span.id,@@ -288,6 +301,13 @@ export function snapshot(built: Preview, input: {stackOrder: span.stackOrder,};});+ const clips: Clip[] = projectStudioTrack({+ track: item,+ ...(placement === undefined ? {} : { placement }),+ spans: projectedSpans,+ values: built.values,+ generic,+ }).map((draft) => sealStudioClip(item.outputRef, draft, binding));const provenance: CandidateProvenance = {output: item.name,outputRef: item.outputRef,@@ -297,10 +317,11 @@ export function snapshot(built: Preview, input: {errors: [],};tracks.push({- id: item.name,+ id: item.outputRef,label: item.name,row: 0,clips,+ binding,provenance,});}diff --git a/packages/studio/src/studio-preflight.ts b/packages/studio/src/studio-preflight.tsindex 2e9a6c9e..238c1af9 100644--- a/packages/studio/src/studio-preflight.ts+++ b/packages/studio/src/studio-preflight.ts@@ -11,6 +11,7 @@ import {roleFor,traceFor,tracedRealizations,+ tracedStudioRealizations,unique,} from "./studio-trace.js";import type { StudioProjectionRole, StudioTrace } from "./studio-trace.js";@@ -116,35 +117,39 @@ export function inspectStudioRun(source: CompiledSource, base: RunPlan): StudioI? [output.ref]: [];});- const speechSpines = source.observations.placements.filter((placement) =>- placement.tag.split(":").at(-1) === "Spine"+ const speechTracks = source.observations.placements.filter((placement) =>+ placement.outputs.some((ref) => outputFor(source, ref)?.type === "SemanticTrack")&& placement.outputs.some((ref) => {const output = outputFor(source, ref);return output !== undefined && filmTrackRefs.includes(output.ref);}));- const semanticTakeRefs = unique(speechSpines.flatMap((placement) => [+ const semanticTakeRefs = unique(speechTracks.flatMap((placement) => [...placement.references,...placement.children.flatMap((child) => child.references),])).flatMap((ref) => outputFor(source, ref)?.type === "SemanticTake"? [outputFor(source, ref)!.ref] : []);- const speechProjectionRefs = unique(speechSpines.flatMap((placement) => placement.outputs))+ const speechProjectionRefs = unique(speechTracks.flatMap((placement) => placement.outputs)).flatMap((ref) => {const output = outputFor(source, ref);return output !== undefined- && (output.type === "CompleteSemanticMap" || output.type === "VisualTrack" || output.type === "AudioTrack")+ && (output.type === "SemanticTrack" || output.type === "VisualTrack" || output.type === "AudioTrack")? [output.ref] : [];});- if (semanticTakeRefs.length === 0) issues.push("Film has no traceable SemanticTake / Speech Spine chain");+ if (semanticTakeRefs.length === 0) issues.push("Film has no traceable SemanticTake / Speech Track chain");const captionPlanRefs = unique(filmTrackRefs.flatMap((ref) =>tracedRealizations(source, ref).map((dependency) => dependency.ref)));+ const adapterRealizationRefs = unique(filmTrackRefs.flatMap((ref) =>+ tracedStudioRealizations(source, ref)));+const projectionRefs = unique([...filmTrackRefs,...speechProjectionRefs,...semanticTakeRefs,...captionPlanRefs,+ ...adapterRealizationRefs,]);const projections: StudioProjection[] = [];const derived: Omit<StudioProjection, "candidatePolicy">[] = [];diff --git a/packages/studio/src/studio-registry.ts b/packages/studio/src/studio-registry.tsnew file mode 100644index 00000000..a7b55126--- /dev/null+++ b/packages/studio/src/studio-registry.ts@@ -0,0 +1,129 @@+import { audioAdapters } from "./adapters/audio.js";+import { genericAdapters } from "./adapters/generic.js";+import { mediaAdapters } from "./adapters/media.js";+import { rankingAdapters } from "./adapters/ranking.js";+import { speechAdapters } from "./adapters/speech.js";+import type { StudioAdapter, StudioEntityDraft, StudioProjectionRole, StudioSpan } from "./adapters/types.js";+import { flatLane, readonlyInteraction, standardInspector } from "./adapters/types.js";+import type { Placement } from "./observe.js";+import type { BuiltTrack } from "./programme.js";+import type { Clip, StudioTrackBinding } from "./shared.js";++export type { StudioEntityDraft, StudioProjectionRole, StudioSpan } from "./adapters/types.js";++/** One Studio-local registry. Author packages and Core know nothing about it. */+const REGISTRY: readonly StudioAdapter[] = [+ ...rankingAdapters,+ ...speechAdapters,+ ...mediaAdapters,+ ...audioAdapters,+ ...genericAdapters,+];++function matches(+ adapter: StudioAdapter,+ type: string,+ placement: Placement | undefined,+ siblingTypes: readonly string[],+): boolean {+ const rule = adapter.output;+ return rule.type === type+ && (rule.surface === undefined || rule.surface === placement?.surface)+ && (rule.modules === undefined || (placement !== undefined && rule.modules.includes(placement.module.name)))+ && (rule.siblingType === undefined || siblingTypes.includes(rule.siblingType));+}++function adapterFor(+ type: string,+ placement: Placement | undefined,+ siblingTypes: readonly string[],+): StudioAdapter | undefined {+ return REGISTRY.find((adapter) => matches(adapter, type, placement, siblingTypes));+}++export function classifyStudioOutput(+ type: string,+ placement: Placement | undefined,+ siblingTypes: readonly string[],+): StudioProjectionRole | undefined {+ return adapterFor(type, placement, siblingTypes)?.role;+}++export function studioDependencyRole(+ role: StudioProjectionRole,+ type: string,+): StudioProjectionRole | undefined {+ return REGISTRY+ .filter((adapter) => adapter.role === role)+ .flatMap((adapter) => adapter.dependencies ?? [])+ .find((dependency) => dependency.type === type)?.role;+}++export function studioRealizationPorts(+ type: string,+ placement: Placement | undefined,+ siblingTypes: readonly string[],+): readonly string[] {+ return adapterFor(type, placement, siblingTypes)?.realizationPorts ?? [];+}++function trackAdapter(track: BuiltTrack): StudioAdapter {+ return REGISTRY.find((adapter) =>+ adapter.family !== undefined+ && adapter.role === track.role+ && adapter.output.type === track.type+ && (adapter.output.surface === undefined || adapter.output.surface === track.trace.surface)+ && (adapter.output.modules === undefined+ || (track.trace.module !== undefined && adapter.output.modules.includes(track.trace.module))))!;+}++export function bindStudioTrack(track: BuiltTrack): StudioTrackBinding {+ const adapter = trackAdapter(track);+ return {+ family: adapter.family!,+ facet: track.type === "AudioTrack" ? "audio" : "visual",+ groupId: track.trace.authoredId ?? track.outputRef,+ icon: adapter.icon!,+ adapter: adapter.id,+ lane: adapter.lane ?? flatLane,+ inspector: adapter.inspector ?? standardInspector,+ ...(track.trace.placement === undefined ? {} : { authoredTag: track.trace.placement }),+ references: track.trace.references.map(({ name, type }) => ({ name, type })),+ interaction: adapter.interaction ?? readonlyInteraction,+ };+}++export function projectStudioTrack(input: {+ readonly track: BuiltTrack;+ readonly placement?: Placement;+ readonly spans: readonly StudioSpan[];+ readonly values: ReadonlyMap<string, unknown>;+ readonly generic: () => readonly StudioEntityDraft[];+}): readonly StudioEntityDraft[] {+ return trackAdapter(input.track).project?.(input) ?? input.generic();+}++export function sealStudioClip(+ outputRef: string,+ draft: StudioEntityDraft,+ fallback: StudioTrackBinding,+): Clip {+ return {+ id: draft.id.startsWith(`${outputRef}:`) ? draft.id : `${outputRef}:${draft.id}`,+ ...(draft.presentId === undefined ? {} : { presentId: draft.presentId }),+ authoredId: draft.authoredId,+ ...(draft.markerId === undefined ? {} : { markerId: draft.markerId }),+ label: draft.label,+ startFrame: draft.startFrame,+ endFrameExclusive: draft.endFrameExclusive,+ ...(draft.elementRange === undefined ? {} : { elementRange: draft.elementRange }),+ stackOrder: draft.stackOrder,+ presentation: draft.presentation ?? {+ entity: fallback.facet === "audio" ? "audio-clip" : "present",+ shape: fallback.facet === "audio" ? "waveform" : "block",+ depth: 0,+ },+ interaction: draft.interaction ?? fallback.interaction,+ renderIds: draft.renderIds ?? (draft.presentId === undefined ? [] : [draft.presentId]),+ };+}diff --git a/packages/studio/src/studio-trace.ts b/packages/studio/src/studio-trace.tsindex 2deab0fe..b77e3e98 100644--- a/packages/studio/src/studio-trace.ts+++ b/packages/studio/src/studio-trace.ts@@ -1,16 +1,13 @@import type { CompiledSource } from "./compile.js";import type { Placement } from "./observe.js";+import {+ classifyStudioOutput,+ studioRealizationPorts,+ studioDependencyRole,+} from "./studio-registry.js";+import type { StudioProjectionRole } from "./studio-registry.js";-export type StudioProjectionRole =- | "semantic-take"- | "semantic-map"- | "caption-plan"- | "speech-visual"- | "speech-audio"- | "media"- | "text"- | "caption"- | "track";+export type { StudioProjectionRole } from "./studio-registry.js";export type StudioTraceDependency = {readonly name: string;@@ -20,6 +17,10 @@ export type StudioTraceDependency = {export type StudioTrace = {readonly placement?: string;+ readonly surface?: string;+ readonly module?: string;+ readonly authoredId?: string;+ readonly outputPorts: readonly { readonly name: string; readonly ref: string }[];readonly references: readonly StudioTraceDependency[];};@@ -29,40 +30,6 @@ export type StudioOutput = {readonly ref: string;};-type StudioOutputRule = {- readonly type: string;- readonly role: StudioProjectionRole;- readonly tag?: string;- readonly namespace?: string;-};--type StudioDependencyRule = {- readonly fromRole: StudioProjectionRole;- readonly type: string;- readonly role: StudioProjectionRole;-};--/**- * This table belongs entirely to Studio. It understands public Output types- * and observed Surface tags without asking domain packages to register UI- * knowledge.- */-const OUTPUT_RULES: readonly StudioOutputRule[] = [- { type: "SemanticTake", role: "semantic-take" },- { type: "CompleteSemanticMap", role: "semantic-map" },- { type: "CaptionPlan", role: "caption-plan" },- { type: "VisualTrack", role: "speech-visual", tag: "spine" },- { type: "AudioTrack", role: "speech-audio", tag: "spine" },- { type: "VisualTrack", role: "media", tag: "track", namespace: "media-track" },- { type: "AudioTrack", role: "media", tag: "track", namespace: "media-track" },- { type: "VisualTrack", role: "text", tag: "track", namespace: "text" },- { type: "VisualTrack", role: "caption", tag: "track", namespace: "caption-fine" },-];--const DEPENDENCY_RULES: readonly StudioDependencyRule[] = [- { fromRole: "caption", type: "CaptionPlan", role: "caption-plan" },-];-function lastTag(placement: Placement | undefined): string {return placement?.tag.split(":").at(-1) ?? "";}@@ -90,18 +57,16 @@ export function roleFor(source: CompiledSource, ref: string): StudioProjectionRoconst output = outputFor(source, ref);if (output === undefined) return undefined;const placement = placementFor(source, ref);- const tag = lastTag(placement).toLowerCase();- const namespace = placement?.tag.split(":")[0]?.toLowerCase();- return OUTPUT_RULES.find((rule) =>- rule.type === output.type- && (rule.tag === undefined || rule.tag === tag)- && (rule.namespace === undefined || rule.namespace === namespace)- )?.role ?? ((output.type === "VisualTrack" || output.type === "AudioTrack") ? "track" : undefined);+ const placementTypes = placement?.outputs.flatMap((candidate) => {+ const found = outputFor(source, candidate);+ return found === undefined ? [] : [found.type];+ }) ?? [];+ return classifyStudioOutput(output.type, placement, placementTypes);}export function traceFor(source: CompiledSource, ref: string): StudioTrace {const placement = placementFor(source, ref);- if (placement === undefined) return { references: [] };+ if (placement === undefined) return { outputPorts: [], references: [] };const refs = unique([...placement.references,...placement.children.flatMap((child) => child.references),@@ -111,10 +76,35 @@ export function traceFor(source: CompiledSource, ref: string): StudioTrace {});return {placement: placement.tag,+ surface: placement.surface,+ module: placement.module.name,+ ...(placement.id === undefined ? {} : { authoredId: placement.id }),+ outputPorts: placement.outputPorts.map((port) => ({+ name: port.name,+ ref: outputFor(source, port.ref)?.ref ?? port.ref,+ })),references: refs,};}+/** Additional same-Surface realizations an adapter needs beyond the terminal Track. */+export function tracedStudioRealizations(source: CompiledSource, ref: string): readonly string[] {+ const output = outputFor(source, ref);+ const placement = placementFor(source, ref);+ if (output === undefined || placement === undefined) return [];+ const siblingTypes = placement.outputPorts.flatMap((port) => {+ const found = outputFor(source, port.ref);+ return found === undefined ? [] : [found.type];+ });+ const ports = new Set(studioRealizationPorts(output.type, placement, siblingTypes));+ return placement.outputPorts+ .filter((port) => ports.has(port.name))+ .flatMap((port) => {+ const found = outputFor(source, port.ref);+ return found === undefined ? [] : [found.ref];+ });+}+export function tracedRealizations(source: CompiledSource,ref: string,@@ -122,8 +112,7 @@ export function tracedRealizations(const role = roleFor(source, ref);if (role === undefined) return [];return traceFor(source, ref).references.flatMap((dependency) => {- const rule = DEPENDENCY_RULES.find((item) =>- item.fromRole === role && item.type === dependency.type);- return rule === undefined ? [] : [{ ref: dependency.ref, role: rule.role }];+ const dependencyRole = studioDependencyRole(role, dependency.type);+ return dependencyRole === undefined ? [] : [{ ref: dependency.ref, role: dependencyRole }];});}diff --git a/packages/studio/src/style.css b/packages/studio/src/style.cssindex c62d25b1..960dd1ff 100644--- a/packages/studio/src/style.css+++ b/packages/studio/src/style.css@@ -272,6 +272,13 @@ button:focus-visible, textarea:focus-visible { outline: 1px solid var(--accent);.semantic-anchor-segment-start, .semantic-anchor-segment-end { opacity: .86; background: color-mix(in srgb, var(--semantic-tone) 82%, #fff 8%); }.semantic-segment.semantic-coarse .semantic-words, .semantic-segment.semantic-coarse .semantic-anchor { display: none; }.clip { --tone: var(--track); position: absolute; height: 40px; min-width: 3px; margin-top: 6px; display: flex; align-items: center; gap: 6px; padding: 0 8px; overflow: hidden; border: 1px solid color-mix(in srgb, var(--tone) 36%, var(--line)); border-radius: 5px; background: linear-gradient(180deg, color-mix(in srgb, var(--tone) 18%, #171a20), color-mix(in srgb, var(--tone) 10%, #111419)); color: var(--text); text-align: left; box-shadow: inset 0 1px rgba(255,255,255,.035); transition: border-color 100ms ease, box-shadow 100ms ease, filter 100ms ease, transform 100ms ease; touch-action: none; }+.clip-shape-group { padding-top: 2px; align-items: flex-start; background: linear-gradient(180deg, color-mix(in srgb, var(--tone) 13%, #171a20), color-mix(in srgb, var(--tone) 7%, #111419)); }+.clip-shape-group .clip-copy { flex-direction: row; align-items: baseline; gap: 6px; }+.clip-shape-window { z-index: 2; height: 19px; margin-top: 24px; padding: 0 5px; border-color: color-mix(in srgb, var(--tone) 62%, #fff 8%); border-radius: 3px; background: color-mix(in srgb, var(--tone) 35%, #171a20); }+.clip-shape-window .clip-icon { display: none; }+.clip-shape-window .clip-copy { flex-direction: row; align-items: center; gap: 4px; }+.clip-shape-window .clip-name { font-size: 8px; }+.clip-shape-window .clip-meta { font-size: 7px; }.clip:hover { z-index: 4; border-color: color-mix(in srgb, var(--tone) 66%, #fff 10%); filter: brightness(1.08); }.clip:active { cursor: grabbing; transform: translateY(1px); }.clip-icon { flex: none; color: color-mix(in srgb, var(--tone) 80%, #fff 15%); font-size: 14px; }diff --git a/packages/studio/src/ui/main.ts b/packages/studio/src/ui/main.tsindex 71e3dc18..09768e5a 100644--- a/packages/studio/src/ui/main.ts+++ b/packages/studio/src/ui/main.ts@@ -148,9 +148,13 @@ function renderInspector(snapshot: StudioSnapshot, clipId: string | undefined):const hero = document.createElement("div");hero.className = "selection-hero";hero.innerHTML = `- <span class="selection-icon"><span class="material-symbols-rounded">widgets</span></span>+ <span class="selection-icon"><span class="material-symbols-rounded"></span></span><div class="selection-title"><strong></strong><small></small></div>- <span class="selection-kind">Visual</span>`;+ <span class="selection-kind"></span>`;+ hero.querySelector(".selection-icon .material-symbols-rounded")!.textContent = track?.binding.icon ?? "widgets";+ hero.querySelector(".selection-kind")!.textContent = track === undefined+ ? "Track"+ : `${track.binding.family} · ${track.binding.facet}`;hero.querySelector("strong")!.textContent = clip.label;hero.querySelector("small")!.textContent = track?.label ?? "Visual track";@@ -165,13 +169,26 @@ function renderInspector(snapshot: StudioSnapshot, clipId: string | undefined):...(clip.markerId === undefined? []: [property("Script marker", clip.markerId, "property-wide property-code")]),- property("Present id", clip.id, "property-wide property-code"),+ property("Entity", clip.presentation.entity, "property-wide property-code"),+ ...(clip.presentId === undefined+ ? []+ : [property("Present id", clip.presentId, "property-wide property-code")]),]);const composition = group("Composition", [property("Track", track?.label ?? "—", "property-wide"),+ property("Family", track?.binding.family ?? "—"),+ property("Facet", track?.binding.facet ?? "—"),property("Stack order", String(clip.stackOrder)),property("Source", track?.provenance.origin ?? "source"),]);+ const authoring = group("Authoring", [+ property("Group", track?.binding.groupId ?? "—", "property-wide property-code"),+ property("Surface", track?.binding.authoredTag ?? "—", "property-wide property-code"),+ property("Adapter", track?.binding.adapter ?? "—", "property-wide property-code"),+ property("Writeback", clip.interaction.writeback),+ property("Move / trim", clip.interaction.move || clip.interaction.trimStart || clip.interaction.trimEnd ? "enabled" : "read-only"),+ property("Inputs", String(track?.binding.references.length ?? 0)),+ ]);const run = group("Run provenance", [property("Output", track?.provenance.output ?? "—", "property-wide property-code"),property("Candidate", track?.provenance.candidateId ?? "—", "property-wide property-code"),@@ -181,7 +198,19 @@ function renderInspector(snapshot: StudioSnapshot, clipId: string | undefined):? []: [property("Error", track.provenance.errors.join(" "), "property-wide property-code")]),]);- const sections: HTMLElement[] = [hero, timing, composition, identity, run];+ const available = {+ authoring,+ resolved: timing,+ material: undefined,+ composition,+ identity,+ run,+ } as const;+ const sections: HTMLElement[] = [+ hero,+ ...(track?.binding.inspector.sections ?? ["resolved", "composition", "identity", "run"])+ .flatMap((name) => available[name] === undefined ? [] : [available[name]]),+ ];inspector.replaceChildren(...sections);}diff --git a/packages/studio/src/ui/overlay.ts b/packages/studio/src/ui/overlay.tsindex e85eaafd..fadaf8a7 100644--- a/packages/studio/src/ui/overlay.ts+++ b/packages/studio/src/ui/overlay.ts@@ -71,7 +71,19 @@ export function createOverlay(store: Store, measure: Measure): Overlay {* motion moves an element across its span, so only what was drawn knows where* it ended up. A Present that has not mounted has no box to draw.*/- const drawnBox = (clip: Clip): Box | undefined => measure(clip.id);+ const drawnBox = (clip: Clip): Box | undefined => {+ const ids = clip.presentId === undefined ? clip.renderIds : [clip.presentId];+ const boxes = ids.flatMap((id) => {+ const found = measure(id);+ return found === undefined ? [] : [found];+ });+ if (boxes.length === 0) return undefined;+ const left = Math.min(...boxes.map((box) => box.xPx));+ const top = Math.min(...boxes.map((box) => box.yPx));+ const right = Math.max(...boxes.map((box) => box.xPx + box.widthPx));+ const bottom = Math.max(...boxes.map((box) => box.yPx + box.heightPx));+ return { xPx: left, yPx: top, widthPx: right - left, heightPx: bottom - top };+ };const draw = (): void => {if (last === undefined) return;diff --git a/packages/studio/src/ui/stage.ts b/packages/studio/src/ui/stage.tsindex 42908a06..fe68bcaf 100644--- a/packages/studio/src/ui/stage.ts+++ b/packages/studio/src/ui/stage.ts@@ -97,9 +97,9 @@ export function createStage(store: Store): Stage {const canvas = element.querySelector<HTMLElement>("[data-canvas]")!;type SeekWindow = Window & {- __hypitSeekFrame?: (frame: number) => void;- __hypitPlayFrame?: (frame: number) => void;- __hypitSetMuted?: (muted: boolean) => void;+ __hypitSeekFrame?: (frame: number) => Promise<boolean>;+ __hypitPlayFrame?: (frame: number) => Promise<boolean>;+ __hypitSetMuted?: (muted: boolean) => Promise<boolean>;};let state: State | undefined;let mounted = -1;@@ -112,6 +112,14 @@ export function createStage(store: Store): Stage {// carries on from the frame the author asked for.let fromFrame = 0;let began = 0;+ let placed = 0;++ const seekPicture = (frame: number): void => {+ const request = ++placed;+ void (iframe.contentWindow as SeekWindow | null)?.__hypitSeekFrame?.(frame).then((current) => {+ if (current && request === placed) overlay.refresh();+ });+ };const fps = (snapshot: StudioSnapshot): number =>snapshot.space.frameRate.numerator / snapshot.space.frameRate.denominator;@@ -143,7 +151,7 @@ export function createStage(store: Store): Stage {raf = 0;// Settle the picture on the frame the transport stopped at.if (wasPlaying && state !== undefined && ready) {- (iframe.contentWindow as SeekWindow | null)?.__hypitSeekFrame?.(state.playhead.frame);+ seekPicture(state.playhead.frame);}};@@ -183,7 +191,7 @@ export function createStage(store: Store): Stage {iframe.addEventListener("load", () => {ready = true;- if (state !== undefined) (iframe.contentWindow as SeekWindow | null)?.__hypitSeekFrame?.(state.playhead.frame);+ if (state !== undefined) seekPicture(state.playhead.frame);applyMuted();// The box could not be measured until now.overlay.refresh();@@ -222,10 +230,10 @@ export function createStage(store: Store): Stage {if (!ready) return;const frame = (iframe.contentWindow as SeekWindow | null);if (playing) frame?.__hypitPlayFrame?.(value.playhead.frame);- else frame?.__hypitSeekFrame?.(value.playhead.frame);+ else seekPicture(value.playhead.frame);// The overlay subscribed first, so it measured the picture as it was before// this seek. Redraw now that the picture has moved.- overlay.refresh();+ if (playing) overlay.refresh();});return { element, toggle };diff --git a/packages/studio/src/ui/timeline.ts b/packages/studio/src/ui/timeline.tsindex c3b11ea7..12622fac 100644--- a/packages/studio/src/ui/timeline.ts+++ b/packages/studio/src/ui/timeline.ts@@ -18,29 +18,8 @@ function timecode(seconds: number): string {return `${String(Math.floor(whole / 60)).padStart(2, "0")}:${String(whole % 60).padStart(2, "0")}`;}-function trackKind(label: string, id: string): string {- const value = `${label} ${id}`.toLowerCase();- if (value.includes("caption") || value.includes("subtitle")) return "caption";- if (value.includes("speech") || value.includes("spine") || value.includes("a-roll")) return "speech";- if (value.includes("audio") || value.includes("music") || value.includes("sound")) return "audio";- if (value.includes("ranking") || value.includes("comment") || value.includes("component")) return "component";- if (value.includes("media") || value.includes("b-roll") || value.includes("image") || value.includes("video")) return "media";- return "visual";-}--function trackIcon(kind: string): string {- return {- caption: "subtitles",- speech: "record_voice_over",- audio: "graphic_eq",- component: "widgets",- media: "movie",- visual: "layers",- }[kind] ?? "layers";-}-/**- * A cutting-room timeline read straight from the Source: the Speech Spine on the+ * A cutting-room timeline read straight from the Source: the Speech Track on the* bottom track, every Media Item above it on one shared track, exactly as an* editor would lay them out.*/@@ -312,11 +291,11 @@ export function createTimeline(store: Store): Timeline {}for (const track of snapshot.tracks) {- const kind = trackKind(track.label, track.id);+ const kind = track.binding.family;const label = document.createElement("div");label.className = `track-label track-${kind}`;label.innerHTML = `<span class="material-symbols-rounded track-icon"></span><span class="track-copy"><strong></strong><small></small></span><span class="track-state"></span>`;- label.querySelector(".track-icon")!.textContent = trackIcon(kind);+ label.querySelector(".track-icon")!.textContent = track.binding.icon;label.querySelector("strong")!.textContent = track.label;label.querySelector("small")!.textContent =`${track.clips.length} clip${track.clips.length === 1 ? "" : "s"}`;@@ -330,7 +309,8 @@ export function createTimeline(store: Store): Timeline {// everything else, so a folded Track shows the picture the viewer sees.const freeFrom: number[] = [];const rowOf = new Map<string, number>();- const ordered = [...track.clips].sort((a, b) =>+ const roots = track.clips.filter((clip) => clip.presentation.parentId === undefined);+ const ordered = [...roots].sort((a, b) =>b.stackOrder - a.stackOrder || a.startFrame - b.startFrame);for (const clip of ordered) {let row = freeFrom.findIndex((free) => free <= clip.startFrame);@@ -338,6 +318,13 @@ export function createTimeline(store: Store): Timeline {freeFrom[row] = clip.endFrameExclusive;rowOf.set(clip.id, row);}+ // Child entities are drawn inside their registered parent entity. They+ // describe its internal semantic windows and must not manufacture extra+ // Track rows merely because their resolved spans overlap the parent.+ for (const clip of track.clips) {+ const parentId = clip.presentation.parentId;+ if (parentId !== undefined) rowOf.set(clip.id, rowOf.get(parentId) ?? 0);+ }const depth = Math.max(1, freeFrom.length);const folded = depth > 1 && !opened.has(track.id);const shownRows = folded ? 1 : depth;@@ -368,7 +355,7 @@ export function createTimeline(store: Store): Timeline {if (to <= 0 || from >= 1) continue;const node = document.createElement("button");node.type = "button";- node.className = `clip clip-${kind}`;+ node.className = `clip clip-${kind} clip-shape-${clip.presentation.shape}`;node.dataset.clip = clip.id;const visibleFrom = Math.max(0, from);const visibleTo = Math.min(1, to);@@ -379,11 +366,11 @@ export function createTimeline(store: Store): Timeline {node.style.width = `${Math.max(0, visibleTo - visibleFrom) * 100}%`;node.title = `${clip.label} (${clip.startFrame}-${clip.endFrameExclusive}f)`;node.innerHTML = `- <span class="clip-edge clip-edge-left"></span>+ ${clip.interaction.trimStart ? '<span class="clip-edge clip-edge-left"></span>' : ""}<span class="material-symbols-rounded clip-icon"></span><span class="clip-copy"><span class="clip-name"></span><span class="clip-meta"></span></span>- <span class="clip-edge clip-edge-right"></span>`;- node.querySelector(".clip-icon")!.textContent = trackIcon(kind);+ ${clip.interaction.trimEnd ? '<span class="clip-edge clip-edge-right"></span>' : ""}`;+ node.querySelector(".clip-icon")!.textContent = track.binding.icon;node.querySelector(".clip-name")!.textContent = clip.label;node.querySelector(".clip-meta")!.textContent =`${((clip.endFrameExclusive - clip.startFrame) / fps(snapshot)).toFixed(2)}s`;@@ -394,6 +381,7 @@ export function createTimeline(store: Store): Timeline {if (tone !== undefined) node.classList.add(`tone-${tone}`);node.addEventListener("pointerdown", (event) => {event.stopPropagation();+ if (!clip.interaction.select) return;store.focus(frameAt(event.clientX), clip.id, "timeline");});nextClipNodes.push({ node, start: clip.startFrame, end: clip.endFrameExclusive, id: clip.id });diff --git a/packages/studio/start.ts b/packages/studio/start.tsindex 4391c229..2f606738 100644--- a/packages/studio/start.ts+++ b/packages/studio/start.ts@@ -4,7 +4,6 @@ import { fileURLToPath } from "node:url";import { createServer } from "vite";import { openStudioArchive } from "./src/archive.js";-import { compileSource } from "./src/compile.js";import { loadStudioDomain } from "./src/domain.js";import { loadStudioRun } from "./src/run.js";import { studioPlugin } from "./src/server.js";@@ -16,7 +15,7 @@ function usage(message?: string): never {if (message !== undefined) process.stderr.write(`${message}\n\n`);process.stderr.write(`Usage:pnpm studio -- --run <build.svrun> [--runtime <hypit.runtime.json>]- [--port <number>] [--package-root <directory>]+ [--port <number>] [--workspace <directory>] [--package-root <directory>]Studio opens one explicit Run Source, requires a Film/Render target and aresolved deterministic semantic projection, and writes only its Author SVML.@@ -44,7 +43,7 @@ const runArgument = values.get("run");if (runArgument === undefined || runArgument.trim().length === 0) usage("Missing --run");const runPath = resolve(invokedFrom, runArgument);const packageRoot = resolve(values.get("package-root") ?? invokedFrom);-const workspaceRoot = packageRoot;+const workspaceRoot = resolve(values.get("workspace") ?? dirname(runPath));const runtimeArgument = values.get("runtime");const runtimePath = runtimeArgument === undefined ? undefined : resolve(invokedFrom, runtimeArgument);const port = Number(values.get("port") ?? "5179");@@ -65,7 +64,7 @@ try {}const source = run.authorSource;try {- inspectStudioRun(await compileSource(source, domain), run);+ inspectStudioRun(run.source, run);} catch (error) {await archive?.close();throw error;@@ -81,7 +80,6 @@ const server = await createServer({source,runPath,domain,- run,...(archive === undefined ? {} : { archive }),})],});diff --git a/packages/temporal/README.md b/packages/temporal/README.mdindex 3818dac7..76ba184c 100644--- a/packages/temporal/README.md+++ b/packages/temporal/README.md@@ -1,7 +1,7 @@# `@hypit/temporal`Focused video-domain temporal projection. It locates authored Selection/Moment occurrences through-`@hypit/semantic-map`, expands strict `one`/`each` cardinality, evaluates exact rational point+`@hypit/semantic-track`, expands strict `one`/`each` cardinality, evaluates exact rational pointexpressions, intersects them with ProgramSpace and returns validated half-open frame spans.The package also provides pure sibling-window validation and triggered-stage scheduling. It definesdiff --git a/packages/temporal/package.json b/packages/temporal/package.jsonindex 0a20a5b4..51c4e9a6 100644--- a/packages/temporal/package.json+++ b/packages/temporal/package.json@@ -14,6 +14,6 @@"@hypit/narrative": "workspace:*","@hypit/program-space": "workspace:*","@hypit/protocol": "workspace:*",- "@hypit/semantic-map": "workspace:*"+ "@hypit/semantic-track": "workspace:*"}}diff --git a/packages/temporal/src/location.ts b/packages/temporal/src/location.tsindex 01cf437b..5d5928cb 100644--- a/packages/temporal/src/location.ts+++ b/packages/temporal/src/location.ts@@ -1,8 +1,12 @@import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrative";-import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@hypit/program-space";-import type { ProgramSpace } from "@hypit/program-space";-import { momentFrames, segmentFrameSpan, selectionFrameSpans } from "@hypit/semantic-map";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import { programSpaceFrameCount } from "@hypit/program-space";+import {+ momentFrames,+ projectSemanticProgramSpace,+ segmentFrameSpan,+ selectionFrameSpans,+} from "@hypit/semantic-track";+import type { SemanticTrack } from "@hypit/semantic-track";import type {LocatedMomentOccurrence,@@ -27,13 +31,12 @@ function assertLocatedFrame(frame: number, totalFrames: number, label: string):}export function locateSelectionOccurrences(- map: CompleteSemanticMap,+ semantic: SemanticTrack,selection: NarrativeSelectionRef,- space: ProgramSpace,): readonly LocatedSelectionOccurrence[] {- assertProgramSpaceIdentity(space);+ const space = projectSemanticProgramSpace(semantic);const totalFrames = programSpaceFrameCount(space);- const spans = selectionFrameSpans(map, selection);+ const spans = selectionFrameSpans(semantic, selection);if (spans.length !== selection.occurrences.length) {throw new Error(`NarrativeSelection ${selection.id} location cardinality changed.`);}@@ -53,13 +56,12 @@ export function locateSelectionOccurrences(}export function locateMomentOccurrences(- map: CompleteSemanticMap,+ semantic: SemanticTrack,moment: NarrativeMomentRef,- space: ProgramSpace,): readonly LocatedMomentOccurrence[] {- assertProgramSpaceIdentity(space);+ const space = projectSemanticProgramSpace(semantic);const totalFrames = programSpaceFrameCount(space);- const frames = momentFrames(map, moment);+ const frames = momentFrames(semantic, moment);if (frames.length !== moment.occurrences.length) {throw new Error(`NarrativeMoment ${moment.id} location cardinality changed.`);}@@ -76,19 +78,18 @@ export function locateMomentOccurrences(});}-export function locateProgramOccurrence(space: ProgramSpace): LocatedProgramOccurrence {- assertProgramSpaceIdentity(space);+export function locateProgramOccurrence(semantic: SemanticTrack): LocatedProgramOccurrence {+ const space = projectSemanticProgramSpace(semantic);return { id: "program", start: { frame: 0 }, end: { frame: programSpaceFrameCount(space) } };}export function locateSegmentOccurrence(- map: CompleteSemanticMap,+ semantic: SemanticTrack,segment: NarrativeExcerpt,- space: ProgramSpace,): LocatedSegmentOccurrence {- assertProgramSpaceIdentity(space);+ const space = projectSemanticProgramSpace(semantic);const totalFrames = programSpaceFrameCount(space);- const span = segmentFrameSpan(map, segment);+ const span = segmentFrameSpan(semantic, segment);assertLocatedFrame(span.startFrame, totalFrames, `Narrative Segment ${segment.id} start`);assertLocatedFrame(span.endFrameExclusive, totalFrames, `Narrative Segment ${segment.id} end`);if (span.endFrameExclusive <= span.startFrame) throw new Error(`Narrative Segment ${segment.id} has no positive frame span.`);diff --git a/packages/temporal/src/projection.ts b/packages/temporal/src/projection.tsindex d9424705..de9f041e 100644--- a/packages/temporal/src/projection.ts+++ b/packages/temporal/src/projection.ts@@ -1,7 +1,8 @@import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrative";import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@hypit/program-space";import type { ProgramSpace } from "@hypit/program-space";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import { projectSemanticProgramSpace } from "@hypit/semantic-track";+import type { SemanticTrack } from "@hypit/semantic-track";import { locateMomentOccurrences, locateProgramOccurrence, locateSegmentOccurrence, locateSelectionOccurrences } from "./location.js";import {@@ -138,15 +139,15 @@ function projectedId(itemId: string, occurrenceId: string): string {export function projectSelectionWindows(input: {readonly itemId: string;- readonly map: CompleteSemanticMap;+ readonly semantic: SemanticTrack;readonly selection: NarrativeSelectionRef;- readonly space: ProgramSpace;readonly expansion: OccurrenceExpansion;readonly projection: TemporalWindowProjection;}): readonly ProjectedOccurrence[] {- const program = locateProgramOccurrence(input.space);+ const space = projectSemanticProgramSpace(input.semantic);+ const program = locateProgramOccurrence(input.semantic);const occurrences = selectOccurrences(- locateSelectionOccurrences(input.map, input.selection, input.space),+ locateSelectionOccurrences(input.semantic, input.selection),input.expansion,input.projection,"selection",@@ -154,21 +155,21 @@ export function projectSelectionWindows(input: {);return occurrences.map((occurrence) => ({id: projectedId(input.itemId, occurrence.id),- span: projectTemporalWindow(input.projection, { program, selection: occurrence }, input.space),+ span: projectTemporalWindow(input.projection, { program, selection: occurrence }, space),}));}export function projectMomentWindows(input: {readonly itemId: string;- readonly map: CompleteSemanticMap;+ readonly semantic: SemanticTrack;readonly moment: NarrativeMomentRef;- readonly space: ProgramSpace;readonly expansion: OccurrenceExpansion;readonly projection: TemporalWindowProjection;}): readonly ProjectedOccurrence[] {- const program = locateProgramOccurrence(input.space);+ const space = projectSemanticProgramSpace(input.semantic);+ const program = locateProgramOccurrence(input.semantic);const occurrences = selectOccurrences(- locateMomentOccurrences(input.map, input.moment, input.space),+ locateMomentOccurrences(input.semantic, input.moment),input.expansion,input.projection,"moment",@@ -176,34 +177,35 @@ export function projectMomentWindows(input: {);return occurrences.map((occurrence) => ({id: projectedId(input.itemId, occurrence.id),- span: projectTemporalWindow(input.projection, { program, moment: occurrence }, input.space),+ span: projectTemporalWindow(input.projection, { program, moment: occurrence }, space),}));}export function projectProgramWindow(input: {readonly itemId: string;- readonly space: ProgramSpace;+ readonly semantic: SemanticTrack;readonly projection: TemporalWindowProjection;}): ProjectedOccurrence {- const program = locateProgramOccurrence(input.space);+ const space = projectSemanticProgramSpace(input.semantic);+ const program = locateProgramOccurrence(input.semantic);return {id: projectedId(input.itemId, program.id),- span: projectTemporalWindow(input.projection, { program }, input.space),+ span: projectTemporalWindow(input.projection, { program }, space),};}export function projectSegmentWindow(input: {readonly itemId: string;- readonly map: CompleteSemanticMap;+ readonly semantic: SemanticTrack;readonly segment: NarrativeExcerpt;- readonly space: ProgramSpace;readonly projection: TemporalWindowProjection;}): ProjectedOccurrence {- const program = locateProgramOccurrence(input.space);- const segment = locateSegmentOccurrence(input.map, input.segment, input.space);+ const space = projectSemanticProgramSpace(input.semantic);+ const program = locateProgramOccurrence(input.semantic);+ const segment = locateSegmentOccurrence(input.semantic, input.segment);return {id: projectedId(input.itemId, segment.id),- span: projectTemporalWindow(input.projection, { program, segment }, input.space),+ span: projectTemporalWindow(input.projection, { program, segment }, space),};}diff --git a/packages/temporal/test/temporal.test.ts b/packages/temporal/test/temporal.test.tsindex 7e65b79f..a34473f6 100644--- a/packages/temporal/test/temporal.test.ts+++ b/packages/temporal/test/temporal.test.ts@@ -4,7 +4,7 @@ import test from "node:test";import type { NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrative";import { programFrameSampleBoundary, programSpaceSampleFrames } from "@hypit/program-space";import type { ProgramSpace } from "@hypit/program-space";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import { semanticTrackFixture } from "../../../test/semantic-track-fixture.js";import {assertWindowRelation,@@ -22,8 +22,12 @@ const space: ProgramSpace = {frameRate: { numerator: 30, denominator: 1 },};-const map: CompleteSemanticMap = {- tokens: [],+const semantic = semanticTrackFixture(space, {+ segments: [+ { id: "opening", frameCount: 60 },+ { id: "answer", frameCount: 60 },+ { id: "ending", frameCount: 180 },+ ],anchors: [{ identity: "a", frame: 30 },{ identity: "b", frame: 60 },@@ -33,7 +37,7 @@ const map: CompleteSemanticMap = {{ identity: "segment:answer:start", frame: 60 },{ identity: "segment:answer:end", frame: 120 },],-};+});const selection = (id: string, occurrences: NarrativeSelectionRef["occurrences"]): NarrativeSelectionRef => ({id, occurrences,@@ -47,9 +51,8 @@ const seconds = (numerator: number, denominator = 1) => ({ unit: "seconds" as cotest("one Selection projects exact local points and stable occurrence identity", () => {const result = projectSelectionWindows({itemId: "card",- map,+ semantic,selection: selection("proof", [{ occurrence: 7, startAnchorId: "a", endAnchorId: "b" }]),- space,expansion: { kind: "one" },projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } },});@@ -62,9 +65,8 @@ test("one Selection projects exact local points and stable occurrence identity",test("one Segment projects from its own structural start and end anchors", () => {const result = projectSegmentWindow({itemId: "answer-card",- map,+ semantic,segment: { kind: "segment", id: "answer", tokenStart: 0, tokenEndExclusive: 1 },- space,projection: { start: { ref: "segment.start" }, end: { ref: "segment.end" } },});assert.deepEqual(result, {@@ -76,12 +78,11 @@ test("one Segment projects from its own structural start and end anchors", () =>test("each preserves source order, even when physical time is reversed between occurrences", () => {const result = projectSelectionWindows({itemId: "repeat",- map,+ semantic,selection: selection("mentions", [{ occurrence: 4, startAnchorId: "c", endAnchorId: "d" },{ occurrence: 9, startAnchorId: "a", endAnchorId: "b" },]),- space,expansion: { kind: "each" },projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } },});@@ -95,11 +96,11 @@ test("strict cardinality and occurrence-invariant each fail instead of choosing{ occurrence: 1, startAnchorId: "c", endAnchorId: "d" },]);assert.throws(() => projectSelectionWindows({- itemId: "one", map, selection: repeated, space, expansion: { kind: "one" },+ itemId: "one", semantic, selection: repeated, expansion: { kind: "one" },projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } },}), /exactly one occurrence/u);assert.throws(() => projectSelectionWindows({- itemId: "each", map, selection: repeated, space, expansion: { kind: "each" },+ itemId: "each", semantic, selection: repeated, expansion: { kind: "each" },projection: { start: { ref: "program.start" }, end: { ref: "program.end" } },}), /occurrence-invariant/u);});@@ -107,9 +108,8 @@ test("strict cardinality and occurrence-invariant each fail instead of choosingtest("negative intermediate points clip before nearest half-later frame quantization", () => {const result = projectMomentWindows({itemId: "lead",- map,+ semantic,moment: moment("cue", [{ occurrence: 0, anchorId: "a" }]),- space,expansion: { kind: "one" },projection: {start: { ref: "moment.cue", offset: seconds(-2) },@@ -126,7 +126,7 @@ test("program and absolute projections use exact rational frame-rate arithmetic"};const result = projectProgramWindow({itemId: "absolute",- space: ntsc,+ semantic: semanticTrackFixture(ntsc),projection: {start: { ref: "absolute", at: { unit: "milliseconds", value: 500 } },end: { ref: "absolute", at: seconds(1) },@@ -155,30 +155,30 @@ test("frame and authored durations enter one exact sample-boundary rule", () =>test("crossed source anchors are allowed until a projection actually consumes the reversal", () => {const crossed = selection("crossed", [{ occurrence: 0, startAnchorId: "d", endAnchorId: "a" }]);- assert.deepEqual(locateSelectionOccurrences(map, crossed, space)[0], {+ assert.deepEqual(locateSelectionOccurrences(semantic, crossed)[0], {id: "crossed#0", occurrence: 0, start: { frame: 120 }, end: { frame: 30 },});assert.throws(() => projectSelectionWindows({- itemId: "identity", map, selection: crossed, space, expansion: { kind: "one" },+ itemId: "identity", semantic, selection: crossed, expansion: { kind: "one" },projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } },}), /reversed raw window/u);assert.deepEqual(projectSelectionWindows({- itemId: "persist", map, selection: crossed, space, expansion: { kind: "one" },+ itemId: "persist", semantic, selection: crossed, expansion: { kind: "one" },projection: { start: { ref: "selection.start" }, end: { ref: "program.end" } },})[0]?.span, { startFrame: 120, endFrameExclusive: 300 });});test("zero, fully outside and sub-frame windows fail atomically", () => {assert.throws(() => projectProgramWindow({- itemId: "zero", space,+ itemId: "zero", semantic,projection: { start: { ref: "program.start" }, end: { ref: "program.start" } },}), /zero raw window/u);assert.throws(() => projectProgramWindow({- itemId: "outside", space,+ itemId: "outside", semantic,projection: { start: { ref: "program.end", offset: frames(1) }, end: { ref: "program.end", offset: frames(2) } },}), /does not intersect/u);assert.throws(() => projectProgramWindow({- itemId: "tiny", space,+ itemId: "tiny", semantic,projection: {start: { ref: "absolute", at: seconds(1, 100) },end: { ref: "absolute", at: seconds(7, 500) },@@ -188,9 +188,9 @@ test("zero, fully outside and sub-frame windows fail atomically", () => {test("disjoint validation checks every physical overlap without reordering the result", () => {const occurrences = projectMomentWindows({- itemId: "popup", map,+ itemId: "popup", semantic,moment: moment("hits", [{ occurrence: 9, anchorId: "c" }, { occurrence: 4, anchorId: "a" }]),- space, expansion: { kind: "each" },+ expansion: { kind: "each" },projection: { start: { ref: "moment.cue" }, end: { ref: "moment.cue", offset: seconds(3) } },});assert.equal(assertWindowRelation(occurrences, "independent"), occurrences);diff --git a/packages/typography-track/README.md b/packages/typography-track/README.mdindex 411225f2..a778b91a 100644--- a/packages/typography-track/README.md+++ b/packages/typography-track/README.md@@ -27,7 +27,7 @@ Content has two explicit author forms:<copy:Value id="headline">Intent, not timelines.</copy:Value>-<typo:Track id="titles" space={speech.space}>+<typo:Track id="titles" semantic={speech.semantic}><typo:Area id="headline" content={headline}placement={layout.headline} style={title-style} during="program"/>diff --git a/packages/typography-track/package.json b/packages/typography-track/package.jsonindex 57b26769..1a25cf61 100644--- a/packages/typography-track/package.json+++ b/packages/typography-track/package.json@@ -16,9 +16,9 @@"@hypit/elaborator": "workspace:*","@hypit/media": "workspace:*","@hypit/narrative": "workspace:*",- "@hypit/program-space": "workspace:*","@hypit/protocol": "workspace:*",- "@hypit/semantic-map": "workspace:*",+ "@hypit/program-space": "workspace:*",+ "@hypit/semantic-track": "workspace:*","@hypit/spatial": "workspace:*","@hypit/svs": "workspace:*","@hypit/temporal": "workspace:*",diff --git a/packages/typography-track/src/component.ts b/packages/typography-track/src/component.tsindex 69a0075a..04546c26 100644--- a/packages/typography-track/src/component.ts+++ b/packages/typography-track/src/component.ts@@ -1,7 +1,7 @@import type { ComponentPackage } from "@hypit/component-kit";import type { NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrative";-import type { ProgramSpace } from "@hypit/program-space";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import type { SemanticTrack } from "@hypit/semantic-track";+import { projectSemanticProgramSpace } from "@hypit/semantic-track";import type { CompositableSurfaceRef } from "@hypit/media";import type { SpatialFrame, SpatialPath, SpatialPoint } from "@hypit/spatial";import type { StoredValue } from "@hypit/protocol";@@ -59,7 +59,7 @@ export const typographyTrackComponent = {handler: ({ inputs }) => ({ outputs: { set: output(appendProgramTextItem(inline<TypographyTrackSet>(inputs.set?.value, "TypographyTrackSet"),inline<TypographyTrackHeader>(inputs.header?.value, "TypographyTrackHeader"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"),inline<TextPlacement>(inputs.placement?.value, "TextPlacement"),inline<TextItemSpec>(inputs.spec?.value, "TextItemSpec"),inline<TextStyle>(inputs.style?.value, "TextStyle"),@@ -71,9 +71,8 @@ export const typographyTrackComponent = {handler: ({ inputs }) => ({ outputs: { set: output(appendSelectionTextItem(inline<TypographyTrackSet>(inputs.set?.value, "TypographyTrackSet"),inline<TypographyTrackHeader>(inputs.header?.value, "TypographyTrackHeader"),- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"),inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelection"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),inline<TextPlacement>(inputs.placement?.value, "TextPlacement"),inline<TextItemSpec>(inputs.spec?.value, "TextItemSpec"),inline<TextStyle>(inputs.style?.value, "TextStyle"),@@ -85,9 +84,8 @@ export const typographyTrackComponent = {handler: ({ inputs }) => ({ outputs: { set: output(appendMomentTextItem(inline<TypographyTrackSet>(inputs.set?.value, "TypographyTrackSet"),inline<TypographyTrackHeader>(inputs.header?.value, "TypographyTrackHeader"),- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack"),inline<NarrativeMomentRef>(inputs.moment?.value, "NarrativeMoment"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),inline<TextPlacement>(inputs.placement?.value, "TextPlacement"),inline<TextItemSpec>(inputs.spec?.value, "TextItemSpec"),inline<TextStyle>(inputs.style?.value, "TextStyle"),@@ -104,14 +102,14 @@ export const typographyTrackComponent = {{producer: typographyTrackProducers.render,handler: ({ inputs }) => ({ outputs: { track: output(renderTypographyTrack(- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")),inline<TypographyTrackProgram>(inputs.program?.value, "TypographyTrackProgram"),)) }, needs: {} }),},{producer: typographyTrackProducers.renderMask,handler: ({ inputs }) => ({ outputs: { track: output(renderTextMaskTrack(- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ projectSemanticProgramSpace(inline<SemanticTrack>(inputs.semantic?.value, "SemanticTrack")),inline<TypographyTrackProgram>(inputs.program?.value, "TypographyTrackProgram"),inline<CompositableSurfaceRef>(inputs.material?.value, "CompositableSurfaceRef"),inline<TextMaskSpec>(inputs.spec?.value, "TextMaskSpec"),diff --git a/packages/typography-track/src/fragment.ts b/packages/typography-track/src/fragment.tsindex 238568dc..feea28f9 100644--- a/packages/typography-track/src/fragment.ts+++ b/packages/typography-track/src/fragment.ts@@ -1,7 +1,7 @@-import { programSpaceTypes } from "@hypit/program-space";import { compositionTypes } from "@hypit/composition";import type { VisualTrack } from "@hypit/composition";import { sealGraphFragment } from "@hypit/elaborator";+import { semanticTrackTypes } from "@hypit/semantic-track";import { typographyTrackProducers, typographyTrackTypes } from "./manifest.js";@@ -11,13 +11,13 @@ const operation = (id: string) => ({ kind: "fragment-operation" as const, operat/** Provider-free official lowering from TypographyTrackProgram to one peer VisualTrack. */export const typographyTrackFragment = sealGraphFragment({inputs: [- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "program", type: typographyTrackTypes.program },],operations: [{id: "render",producer: typographyTrackProducers.render,- inputs: { space: input("space"), program: input("program") },+ inputs: { semantic: input("semantic"), program: input("program") },result: { kind: "output", name: "track" },}],exports: [{diff --git a/packages/typography-track/src/manifest.ts b/packages/typography-track/src/manifest.tsindex 6c7017b7..702fbe7d 100644--- a/packages/typography-track/src/manifest.ts+++ b/packages/typography-track/src/manifest.ts@@ -1,8 +1,7 @@import { readFile } from "node:fs/promises";import { narrativeDependency, narrativeTypes } from "@hypit/narrative";-import { programSpaceDependency, programSpaceTypes } from "@hypit/program-space";-import { semanticMapDependency, semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackDependency, semanticTrackTypes } from "@hypit/semantic-track";import {compositionDependency,compositionTypes,@@ -445,10 +444,8 @@ export const typographyTrackMarkupSurfaces = [attributes: [{ name: "id", kind: "identifier", required: true,summary: "Names this Track and prefixes the identity of every item spec it seals." },- { name: "space", kind: "reference", required: true, accepts: [programSpaceTypes.programSpace],- summary: "Fixes the frame domain every item window is projected into." },- { name: "map", kind: "reference", required: false, accepts: [semanticMapTypes.complete],- summary: "Chooses the SemanticMap that turns the Selection or Moment an item binds into exact time." },+ { name: "semantic", kind: "reference", required: true, accepts: [semanticTrackTypes.track],+ summary: "Chooses the SemanticTrack that owns the frame domain and resolves every item window." },],children: [{ tag: "Point", cardinality: "many",@@ -515,7 +512,7 @@ export const typographyTrackMarkupSurfaces = [{ name: "track", type: compositionTypes.visualTrack,summary: "The rendered text, an ordinary peer VisualTrack." },],- example: `<text:Track id="titles" space={speech.space}>+ example: `<text:Track id="titles" semantic={speech.semantic}><text:Area id="title" placement={title-frame} style={title-style} during="program">EDIT MEANING, NOT TIMELINES</text:Area>@@ -524,7 +521,7 @@ export const typographyTrackMarkupSurfaces = [</text:Area></text:Track>`,notes: [- "A Track requires at least one `<Point>`, `<Area>` or `<Path>`, accepts no text content of its own, and refuses `map` when no item binds a Selection or a Moment.",+ "A Track requires at least one `<Point>`, `<Area>` or `<Path>` and accepts no text content of its own.","An item states exactly one window form: `during`, `at` with `for`, or `start` with `end`; `selection` and `moment` bind a start/end window and cannot be written together.","A point expression is `program.start`, `program.end`, `selection.start`, `selection.end` or `moment.cue`, each optionally offset by `+` or `-` and a duration, or a bare duration read as an absolute position.","An item written without `content` owns its own document: direct text becomes one paragraph, and `<P>` children carry rich runs instead; a document mixes neither `<P>` children with direct text nor direct text with nested elements.",@@ -540,8 +537,8 @@ export const typographyTrackMarkupSurfaces = [attributes: [{ name: "id", kind: "identifier", required: true,summary: "Names this Mask, under which its VisualTrack is published." },- { name: "space", kind: "reference", required: true, accepts: [programSpaceTypes.programSpace],- summary: "Fixes the frame domain the masked Track is rendered into." },+ { name: "semantic", kind: "reference", required: true, accepts: [semanticTrackTypes.track],+ summary: "Chooses the SemanticTrack whose frame domain the masked Track is rendered into." },{ name: "text", kind: "reference", required: true, accepts: [typographyTrackTypes.program],summary: "Chooses the authored Text Program whose items give the mask its shape and timing." },{ name: "material", kind: "reference", required: true, accepts: [mediaTypes.compositableSurface],@@ -555,7 +552,7 @@ export const typographyTrackMarkupSurfaces = [{ name: "track", type: compositionTypes.visualTrack,summary: "The masked picture, an ordinary peer VisualTrack." },],- example: `<text:Mask id="masked-titles" space={speech.space} text={mask-shape.program} material={material}/>`,+ example: `<text:Mask id="masked-titles" semantic={speech.semantic} text={mask-shape.program} material={material}/>`,notes: ["A Mask is written empty and accepts no children.","The material must be a still Surface; a timed material is refused and materializes through an independent package.",@@ -570,7 +567,7 @@ export const typographyTrackManifest: ModuleManifest = {format: "hypit.module@1",name: typographyTrackModuleRef.name,version: typographyTrackModuleRef.version,- dependencies: [programSpaceDependency, narrativeDependency, semanticMapDependency, spatialDependency, mediaDependency, compositionDependency, textDependency],+ dependencies: [narrativeDependency, semanticTrackDependency, spatialDependency, mediaDependency, compositionDependency, textDependency],types: [{ name: typographyTrackTypes.style.name },{ name: typographyTrackTypes.motion.name },@@ -589,11 +586,11 @@ export const typographyTrackManifest: ModuleManifest = {{ name: typographyTrackProducers.bindArea.name, inputs: [{ name: "frame", type: spatialTypes.frame }], outputs: [{ name: "placement", type: typographyTrackTypes.placement }], needs: [] },{ name: typographyTrackProducers.bindPath.name, inputs: [{ name: "path", type: spatialTypes.path }], outputs: [{ name: "placement", type: typographyTrackTypes.placement }], needs: [] },{ name: typographyTrackProducers.createSet.name, inputs: [], outputs: [{ name: "set", type: typographyTrackTypes.set }], needs: [] },- { name: typographyTrackProducers.appendProgram.name, inputs: [{ name: "set", type: typographyTrackTypes.set }, { name: "header", type: typographyTrackTypes.header }, { name: "space", type: programSpaceTypes.programSpace }, { name: "placement", type: typographyTrackTypes.placement }, { name: "spec", type: typographyTrackTypes.itemSpec }, { name: "style", type: typographyTrackTypes.style }, { name: "motion", type: typographyTrackTypes.motion }], outputs: [{ name: "set", type: typographyTrackTypes.set }], needs: [] },- { name: typographyTrackProducers.appendSelection.name, inputs: [{ name: "set", type: typographyTrackTypes.set }, { name: "header", type: typographyTrackTypes.header }, { name: "map", type: semanticMapTypes.complete }, { name: "selection", type: narrativeTypes.selection }, { name: "space", type: programSpaceTypes.programSpace }, { name: "placement", type: typographyTrackTypes.placement }, { name: "spec", type: typographyTrackTypes.itemSpec }, { name: "style", type: typographyTrackTypes.style }, { name: "motion", type: typographyTrackTypes.motion }], outputs: [{ name: "set", type: typographyTrackTypes.set }], needs: [] },- { name: typographyTrackProducers.appendMoment.name, inputs: [{ name: "set", type: typographyTrackTypes.set }, { name: "header", type: typographyTrackTypes.header }, { name: "map", type: semanticMapTypes.complete }, { name: "moment", type: narrativeTypes.moment }, { name: "space", type: programSpaceTypes.programSpace }, { name: "placement", type: typographyTrackTypes.placement }, { name: "spec", type: typographyTrackTypes.itemSpec }, { name: "style", type: typographyTrackTypes.style }, { name: "motion", type: typographyTrackTypes.motion }], outputs: [{ name: "set", type: typographyTrackTypes.set }], needs: [] },+ { name: typographyTrackProducers.appendProgram.name, inputs: [{ name: "set", type: typographyTrackTypes.set }, { name: "header", type: typographyTrackTypes.header }, { name: "semantic", type: semanticTrackTypes.track }, { name: "placement", type: typographyTrackTypes.placement }, { name: "spec", type: typographyTrackTypes.itemSpec }, { name: "style", type: typographyTrackTypes.style }, { name: "motion", type: typographyTrackTypes.motion }], outputs: [{ name: "set", type: typographyTrackTypes.set }], needs: [] },+ { name: typographyTrackProducers.appendSelection.name, inputs: [{ name: "set", type: typographyTrackTypes.set }, { name: "header", type: typographyTrackTypes.header }, { name: "semantic", type: semanticTrackTypes.track }, { name: "selection", type: narrativeTypes.selection }, { name: "placement", type: typographyTrackTypes.placement }, { name: "spec", type: typographyTrackTypes.itemSpec }, { name: "style", type: typographyTrackTypes.style }, { name: "motion", type: typographyTrackTypes.motion }], outputs: [{ name: "set", type: typographyTrackTypes.set }], needs: [] },+ { name: typographyTrackProducers.appendMoment.name, inputs: [{ name: "set", type: typographyTrackTypes.set }, { name: "header", type: typographyTrackTypes.header }, { name: "semantic", type: semanticTrackTypes.track }, { name: "moment", type: narrativeTypes.moment }, { name: "placement", type: typographyTrackTypes.placement }, { name: "spec", type: typographyTrackTypes.itemSpec }, { name: "style", type: typographyTrackTypes.style }, { name: "motion", type: typographyTrackTypes.motion }], outputs: [{ name: "set", type: typographyTrackTypes.set }], needs: [] },{ name: typographyTrackProducers.finalize.name, inputs: [{ name: "header", type: typographyTrackTypes.header }, { name: "set", type: typographyTrackTypes.set }], outputs: [{ name: "program", type: typographyTrackTypes.program }], needs: [] },- { name: typographyTrackProducers.render.name, inputs: [{ name: "space", type: programSpaceTypes.programSpace }, { name: "program", type: typographyTrackTypes.program }], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [] },- { name: typographyTrackProducers.renderMask.name, inputs: [{ name: "space", type: programSpaceTypes.programSpace }, { name: "program", type: typographyTrackTypes.program }, { name: "material", type: mediaTypes.compositableSurface }, { name: "spec", type: typographyTrackTypes.maskSpec }], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [] },+ { name: typographyTrackProducers.render.name, inputs: [{ name: "semantic", type: semanticTrackTypes.track }, { name: "program", type: typographyTrackTypes.program }], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [] },+ { name: typographyTrackProducers.renderMask.name, inputs: [{ name: "semantic", type: semanticTrackTypes.track }, { name: "program", type: typographyTrackTypes.program }, { name: "material", type: mediaTypes.compositableSurface }, { name: "spec", type: typographyTrackTypes.maskSpec }], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [] },],};diff --git a/packages/typography-track/src/program.ts b/packages/typography-track/src/program.tsindex 5760a5f4..4c17577b 100644--- a/packages/typography-track/src/program.ts+++ b/packages/typography-track/src/program.ts@@ -2,11 +2,11 @@ import type { NarrativeMomentRef, NarrativeSelectionRef } from "@hypit/narrativeimport { assertProgramSpaceIdentity, programSpaceFrameCount } from "@hypit/program-space";import type { ProgramSpace } from "@hypit/program-space";import {- assertCompleteSemanticMapIdentity,+ assertSemanticTrackIdentity,assertNarrativeMomentIdentity,assertNarrativeSelectionIdentity,-} from "@hypit/semantic-map";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+} from "@hypit/semantic-track";+import type { SemanticTrack } from "@hypit/semantic-track";import {assertSpatialFrame,assertSpatialPath,@@ -318,55 +318,51 @@ function append(export function appendProgramTextItem(set: TypographyTrackSet,header: TypographyTrackHeader,- space: ProgramSpace,+ semantic: SemanticTrack,placement: TextPlacement,spec: TextItemSpec,style: TextStyle,motion: TextMotion,): TypographyTrackSet {- assertProgramSpaceIdentity(space);+ assertSemanticTrackIdentity(semantic);assertTextPlacement(placement);if (spec.expansion.kind !== "one") throw new Error("Program Text uses one occurrence.");- return append(set, header, spec, style, motion, placement.geometry, [projectProgramWindow({ itemId: spec.id, space, projection: spec.projection })]);+ return append(set, header, spec, style, motion, placement.geometry, [projectProgramWindow({ itemId: spec.id, semantic, projection: spec.projection })]);}export function appendSelectionTextItem(set: TypographyTrackSet,header: TypographyTrackHeader,- map: CompleteSemanticMap,+ semantic: SemanticTrack,selection: NarrativeSelectionRef,- space: ProgramSpace,placement: TextPlacement,spec: TextItemSpec,style: TextStyle,motion: TextMotion,): TypographyTrackSet {- assertCompleteSemanticMapIdentity(map);+ assertSemanticTrackIdentity(semantic);assertNarrativeSelectionIdentity(selection);- assertProgramSpaceIdentity(space);assertTextPlacement(placement);return append(set, header, spec, style, motion, placement.geometry, assertWindowRelation(projectSelectionWindows({- itemId: spec.id, map, selection, space, projection: spec.projection, expansion: spec.expansion,+ itemId: spec.id, semantic, selection, projection: spec.projection, expansion: spec.expansion,}), "disjoint"));}export function appendMomentTextItem(set: TypographyTrackSet,header: TypographyTrackHeader,- map: CompleteSemanticMap,+ semantic: SemanticTrack,moment: NarrativeMomentRef,- space: ProgramSpace,placement: TextPlacement,spec: TextItemSpec,style: TextStyle,motion: TextMotion,): TypographyTrackSet {- assertCompleteSemanticMapIdentity(map);+ assertSemanticTrackIdentity(semantic);assertNarrativeMomentIdentity(moment);- assertProgramSpaceIdentity(space);assertTextPlacement(placement);return append(set, header, spec, style, motion, placement.geometry, assertWindowRelation(projectMomentWindows({- itemId: spec.id, map, moment, space, projection: spec.projection, expansion: spec.expansion,+ itemId: spec.id, semantic, moment, projection: spec.projection, expansion: spec.expansion,}), "disjoint"));}diff --git a/packages/typography-track/src/surface.ts b/packages/typography-track/src/surface.tsindex 4144b175..6096baf6 100644--- a/packages/typography-track/src/surface.ts+++ b/packages/typography-track/src/surface.ts@@ -1,6 +1,5 @@import { narrativeTypes } from "@hypit/narrative";-import { programSpaceTypes } from "@hypit/program-space";-import { semanticMapTypes } from "@hypit/semantic-map";+import { semanticTrackTypes } from "@hypit/semantic-track";import { compositionTypes } from "@hypit/composition";import type {VisualColorPaint,@@ -702,7 +701,6 @@ type FragmentItem = {};function createTrackFragment(id: string, items: readonly FragmentItem[]): GraphFragment {- const usesMap = items.some((item) => item.binding !== "program");const operations: FragmentOperation[] = [{ id: "text:set:empty", producer: typographyTrackProducers.createSet, inputs: {}, result: { kind: "output", name: "set" } }];let current = "text:set:empty";for (const [index, item] of items.entries()) {@@ -724,27 +722,26 @@ function createTrackFragment(id: string, items: readonly FragmentItem[]): GraphF}const append = `text:set:append:${String(index + 1).padStart(4, "0")}`;const common = {- set: operation(current), header: input("header"), space: input("space"), placement: operation(bind),+ set: operation(current), header: input("header"), semantic: input("semantic"), placement: operation(bind),spec: item.contentName === undefined ? input(item.specName) : operation(materialized),style: input(item.styleName), motion: input(item.motionName),};operations.push(item.binding === "program" ? {id: append, producer: typographyTrackProducers.appendProgram, inputs: common, result: { kind: "output", name: "set" },} : item.binding === "selection" ? {- id: append, producer: typographyTrackProducers.appendSelection, inputs: { ...common, map: input("map"), selection: input(item.sourceName!) }, result: { kind: "output", name: "set" },+ id: append, producer: typographyTrackProducers.appendSelection, inputs: { ...common, selection: input(item.sourceName!) }, result: { kind: "output", name: "set" },} : {- id: append, producer: typographyTrackProducers.appendMoment, inputs: { ...common, map: input("map"), moment: input(item.sourceName!) }, result: { kind: "output", name: "set" },+ id: append, producer: typographyTrackProducers.appendMoment, inputs: { ...common, moment: input(item.sourceName!) }, result: { kind: "output", name: "set" },});current = append;}operations.push({ id: "text:finalize", producer: typographyTrackProducers.finalize, inputs: { header: input("header"), set: operation(current) }, result: { kind: "output", name: "program" } },- { id: "text:render", producer: typographyTrackProducers.render, inputs: { space: input("space"), program: operation("text:finalize") }, result: { kind: "output", name: "track" } },+ { id: "text:render", producer: typographyTrackProducers.render, inputs: { semantic: input("semantic"), program: operation("text:finalize") }, result: { kind: "output", name: "track" } },);const inputEntries = [- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "header", type: typographyTrackTypes.header },- ...(usesMap ? [{ name: "map", type: semanticMapTypes.complete }] : []),...items.flatMap((item) => [{ name: item.geometryName, type: item.placementKind === "point" ? spatialTypes.point : item.placementKind === "area" ? spatialTypes.frame : spatialTypes.path },{ name: item.specName, type: item.contentName === undefined ? typographyTrackTypes.itemSpec : typographyTrackTypes.plainItemSpec },@@ -770,9 +767,9 @@ function createTrackFragment(id: string, items: readonly FragmentItem[]): GraphF}export const decodeTypographyTrackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- allowed(element, ["id", "space", "map"], ["id", "space"]);+ allowed(element, ["id", "semantic"], ["id", "semantic"]);const id = text(element, "id");- const space = reference(element.attributes.space, `${element.name}.space`, programSpaceTypes.programSpace, resolveReference);+ const semantic = reference(element.attributes.semantic, `${element.name}.semantic`, semanticTrackTypes.track, resolveReference);const headerId = `${id}.__header`;const records: SurfaceRecordDraft[] = [{id: headerId, type: typographyTrackTypes.header,@@ -849,9 +846,6 @@ export const decodeTypographyTrackSurface: StructuredSurfaceHandler = ({ element});}if (items.length === 0) throw new Error(`${element.name} requires at least one Point, Area or Path.`);- const usesMap = items.some((item) => item.binding !== "program");- const map = usesMap ? reference(element.attributes.map, `${element.name}.map`, semanticMapTypes.complete, resolveReference) : undefined;- if (!usesMap && element.attributes.map !== undefined) throw new Error(`${element.name}.map is unused.`);const fragmentItems: FragmentItem[] = items.map(({ suffix, binding, sourceName, placementKind, geometryName, specName, styleName, motionName, contentName }) => ({suffix, binding, placementKind, geometryName, specName, styleName, motionName,...(contentName === undefined ? {} : { contentName }), ...(sourceName === undefined ? {} : { sourceName }),@@ -862,7 +856,7 @@ export const decodeTypographyTrackSurface: StructuredSurfaceHandler = ({ elementcomponents: [{id, fragment: fragment.id,inputs: {- space: space.ref, header: { kind: "record", id: headerId }, ...(map === undefined ? {} : { map: map.ref }),+ semantic: semantic.ref, header: { kind: "record", id: headerId },...Object.fromEntries(items.flatMap((item) => [[item.geometryName, item.geometry.ref], [item.specName, { kind: "record" as const, id: item.specId }],[item.styleName, item.style.ref], [item.motionName, item.motion],@@ -879,7 +873,7 @@ export const decodeTypographyTrackSurface: StructuredSurfaceHandler = ({ elementfunction createMaskFragment(id: string): GraphFragment {const inputs = [- { name: "space", type: programSpaceTypes.programSpace },+ { name: "semantic", type: semanticTrackTypes.track },{ name: "program", type: typographyTrackTypes.program },{ name: "material", type: mediaTypes.compositableSurface },{ name: "spec", type: typographyTrackTypes.maskSpec },@@ -888,7 +882,7 @@ function createMaskFragment(id: string): GraphFragment {inputs,operations: [{id: "text-mask:render", producer: typographyTrackProducers.renderMask,- inputs: { space: input("space"), program: input("program"), material: input("material"), spec: input("spec") },+ inputs: { semantic: input("semantic"), program: input("program"), material: input("material"), spec: input("spec") },result: { kind: "output", name: "track" },}],exports: [{@@ -898,10 +892,10 @@ function createMaskFragment(id: string): GraphFragment {}export const decodeTypographyMaskSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- allowed(element, ["id", "space", "text", "material", "mode", "fit"], ["id", "space", "text", "material"]);+ allowed(element, ["id", "semantic", "text", "material", "mode", "fit"], ["id", "semantic", "text", "material"]);empty(element);const id = text(element, "id");- const space = reference(element.attributes.space, `${element.name}.space`, programSpaceTypes.programSpace, resolveReference);+ const semantic = reference(element.attributes.semantic, `${element.name}.semantic`, semanticTrackTypes.track, resolveReference);const program = reference(element.attributes.text, `${element.name}.text`, typographyTrackTypes.program, resolveReference);const material = reference(element.attributes.material, `${element.name}.material`, mediaTypes.compositableSurface, resolveReference);const specId = `${id}.__spec`;@@ -915,7 +909,7 @@ export const decodeTypographyMaskSurface: StructuredSurfaceHandler = ({ element,records: [{ id: specId, type: typographyTrackTypes.maskSpec, value: { kind: "inline", value: spec }, range: element.range }],components: [{id, fragment: fragment.id,- inputs: { space: space.ref, program: program.ref, material: material.ref, spec: { kind: "record", id: specId } },+ inputs: { semantic: semantic.ref, program: program.ref, material: material.ref, spec: { kind: "record", id: specId } },outputs: { track: `${id}.track` }, range: element.range,}],fragments: [fragment],diff --git a/packages/typography-track/test/typography-track.test.ts b/packages/typography-track/test/typography-track.test.tsindex 8835d2fa..c05ae737 100644--- a/packages/typography-track/test/typography-track.test.ts+++ b/packages/typography-track/test/typography-track.test.ts@@ -1,6 +1,7 @@import assert from "node:assert/strict";import test from "node:test";import { fixtureDigest } from "../../../test/fixture-digest.js";+import { semanticTrackFixture } from "../../../test/semantic-track-fixture.js";import { spatialComponent, videoContractManifests } from "../../../test/support/video-domain.js";import { registerTypeValidatorFacets } from "@hypit/component-kit";@@ -14,7 +15,7 @@ import type { NarrativeSelectionRef } from "@hypit/narrative";import { sealProgramSpace } from "@hypit/program-space";import { programSpaceDependency, programSpaceTypes } from "@hypit/program-space";import type { ModuleManifest } from "@hypit/protocol";-import type { CompleteSemanticMap } from "@hypit/semantic-map";+import { semanticTrackDependency, semanticTrackTypes } from "@hypit/semantic-track";import {appendSelectionTextItem,bindAreaTextPlacement,@@ -51,6 +52,10 @@ const space = sealProgramSpace({durationSec: 5,frameRate: { numerator: 30, denominator: 1 },});+const semantic = semanticTrackFixture(space, { anchors: [+ { identity: "selection:start", frame: 30 },+ { identity: "selection:end", frame: 60 },+] });const exactTestFont: FontArtifactRef = {sources: [{ artifact: {@@ -270,14 +275,7 @@ test("TypographyTrackProgram rejects a frame span outside ProgramSpace", () => {assert.throws(() => renderTypographyTrack(space, program), /outside ProgramSpace/u);});-test("Selection Text consumes explicit Selection, SemanticMap, Style, Motion and Placement edges", () => {- const map: CompleteSemanticMap = {- tokens: [],- anchors: [- { identity: "selection:start", frame: 30 },- { identity: "selection:end", frame: 60 },- ],- };+test("Selection Text consumes explicit Selection, SemanticTrack, Style, Motion and Placement edges", () => {const selection: NarrativeSelectionRef = {id: "callout",@@ -294,9 +292,8 @@ test("Selection Text consumes explicit Selection, SemanticMap, Style, Motion andconst program = finalizeTypographyTrack(header, appendSelectionTextItem(createTypographyTrackSet(),header,- map,+ semantic,selection,- space,bindAreaTextPlacement({ xPx: 108, yPx: 192, widthPx: 864, heightPx: 192 }),spec,textStyle("meaning", 80),@@ -311,7 +308,7 @@ test("the self-described Markup Surfaces compile Style, Motion and all three spaname: "inputs", tag: "Inputs", mode: "structured",outputs: [svsRecipeType,- programSpaceTypes.programSpace,+ semanticTrackTypes.track,spatialTypes.point,spatialTypes.frame,spatialTypes.path,@@ -325,7 +322,7 @@ test("the self-described Markup Surfaces compile Style, Motion and all three spaname: fixtureModule.name,version: fixtureModule.version,dependencies: [- programSpaceDependency,+ semanticTrackDependency,spatialDependency,mediaDependency,{@@ -384,7 +381,7 @@ test("the self-described Markup Surfaces compile Style, Motion and all three spa} },range: element.range,},- { id: "space", type: programSpaceTypes.programSpace, value: { kind: "inline", value: space }, range: element.range },+ { id: "semantic", type: semanticTrackTypes.track, value: { kind: "inline", value: semantic }, range: element.range },{ id: "title-point", type: spatialTypes.point, value: { kind: "inline", value: { xPx: 540, yPx: 120 } }, range: element.range },{ id: "body-frame", type: spatialTypes.frame, value: { kind: "inline", value: { xPx: 80, yPx: 220, widthPx: 920, heightPx: 520 } }, range: element.range },{ id: "arc", type: spatialTypes.path, value: { kind: "inline", value: { commands: [@@ -435,17 +432,17 @@ test("the self-described Markup Surfaces compile Style, Motion and all three spa<text:Keyframe at="1" opacity="1"/></text:Sequence></text:Motion>- <text:Track id="titles" space={space}>+ <text:Track id="titles" semantic={semantic}><text:Point id="hook" content={copy} placement={title-point} style={poster} during="program"/><text:Area id="body" placement={body-frame} style={poster} motion={arrive} during="program"><text:P id="first">Rich <text:Span style={poster}>inline text</text:Span><text:Break/>wraps.</text:P></text:Area><text:Path id="arc-title" placement={arc} style={poster} during="program">Along the path</text:Path></text:Track>- <text:Track id="mask-shape" space={space}>+ <text:Track id="mask-shape" semantic={semantic}><text:Area id="mask-word" placement={body-frame} style={mask-shape-style} during="program">MASK</text:Area></text:Track>- <text:Mask id="masked-titles" space={space} text={mask-shape.program} material={material}/>+ <text:Mask id="masked-titles" semantic={semantic} text={mask-shape.program} material={material}/></svml>`,},closure,@@ -607,13 +604,13 @@ test("a paragraph's source indentation is not part of its words", async () => {const fixtureSurface = {name: "inputs", tag: "Inputs", mode: "structured",outputs: [- svsRecipeType, programSpaceTypes.programSpace, spatialTypes.frame,+ svsRecipeType, semanticTrackTypes.track, spatialTypes.frame,mediaTypes.fontArtifact, textTypes.text,],} as const;const fixtureManifest: ModuleManifest = {format: "hypit.module@1", name: fixtureModule.name, version: fixtureModule.version,- dependencies: [programSpaceDependency, spatialDependency, mediaDependency,+ dependencies: [semanticTrackDependency, spatialDependency, mediaDependency,{ module: { name: svsManifest.name, version: svsManifest.version } }, textDependency],types: [], capabilities: [], producers: [],};@@ -627,7 +624,7 @@ test("a paragraph's source indentation is not part of its words", async () => {path: "text.editorial", properties: { "stack-order": 70, size: 44, "line-height": 1.15,"inline-size": "fixed", "block-size": "fixed", wrap: "word", overflow: "shrink", "minimum-scale": 0.65 } } },range: element.range },- { id: "space", type: programSpaceTypes.programSpace, value: { kind: "inline", value: space }, range: element.range },+ { id: "semantic", type: semanticTrackTypes.track, value: { kind: "inline", value: semantic }, range: element.range },{ id: "body-frame", type: spatialTypes.frame, value: { kind: "inline", value: { xPx: 80, yPx: 220, widthPx: 920, heightPx: 520 } }, range: element.range },{ id: "exact-font", type: mediaTypes.fontArtifact, value: { kind: "inline", value: exactTestFont }, range: element.range },],@@ -654,7 +651,7 @@ test("a paragraph's source indentation is not part of its words", async () => {<text:Style id="poster" recipe={editorial} font={exact-font}><text:Fill color="#f8fafc"/></text:Style>- <text:Track id="titles" space={space}>+ <text:Track id="titles" semantic={semantic}><text:Area id="body" placement={body-frame} style={poster} during="program"><text:P id="first">Top 5 Most Populardiff --git a/packages/video-cli/test/cli.test.ts b/packages/video-cli/test/cli.test.tsindex a77fc0a2..de627a41 100644--- a/packages/video-cli/test/cli.test.ts+++ b/packages/video-cli/test/cli.test.ts@@ -54,7 +54,7 @@ test("provider-free example plans from installed Source packages", async () => {readonly steps: readonly { readonly producer: { readonly name: string } }[];} }).plan;const producers = new Set(plan.steps.map((step) => step.producer.name));- assert.equal(producers.has("assemble-speech-basis"), true);+ assert.equal(producers.has("assemble-semantic-track"), true);assert.equal(producers.has("compile-composition"), true);assert.equal(producers.has("project-muxed-media"), true);assert.equal(plan.goals.length, 1);diff --git a/packages/video-cli/test/packages.ts b/packages/video-cli/test/packages.tsindex e24d2ec7..26a19223 100644--- a/packages/video-cli/test/packages.ts+++ b/packages/video-cli/test/packages.ts@@ -19,11 +19,10 @@ import runMarkup from "../../run-markup/src/activation.js";import script from "../../script/src/activation.js";import seedance from "../../seedance/src/activation.js";import speechAlignment from "../../speech-alignment/src/activation.js";-import speechSpine from "../../speech-spine/src/activation.js";-import speechBasis from "../../speech-basis/src/activation.js";+import speechTrack from "../../speech-track/src/activation.js";import speech from "../../speech/src/activation.js";import speechEvidence from "../../speech-evidence/src/activation.js";-import semanticMap from "../../semantic-map/src/activation.js";+import semanticTrack from "../../semantic-track/src/activation.js";import spatial from "../../spatial/src/activation.js";import temporal from "../../temporal/src/activation.js";import visualIr from "../../visual-ir/src/activation.js";@@ -45,7 +44,7 @@ export const videoTestPackages = [bind("@hypit/program-space", programSpace),bind("@hypit/speech", speech),bind("@hypit/speech-evidence", speechEvidence),- bind("@hypit/semantic-map", semanticMap),+ bind("@hypit/semantic-track", semanticTrack),bind("@hypit/visual-ir", visualIr),bind("@hypit/composition", composition),bind("@hypit/svs", svs),@@ -59,8 +58,7 @@ export const videoTestPackages = [bind("@hypit/caption-fine", captionFine),bind("@hypit/caption-gemini", captionGemini),bind("@hypit/speech-alignment", speechAlignment),- bind("@hypit/speech-basis", speechBasis),- bind("@hypit/speech-spine", speechSpine),+ bind("@hypit/speech-track", speechTrack),bind("@hypit/whisperx", whisperX),bind("@hypit/spatial", spatial),bind("@hypit/temporal", temporal),diff --git a/packages/whisperx/README.md b/packages/whisperx/README.mdindex 4c220478..d7c15479 100644--- a/packages/whisperx/README.md+++ b/packages/whisperx/README.md@@ -13,7 +13,7 @@ Segment. `@hypit/media-pipeline` projects that Take's 48 kHz audio to canonical`SpeechEvidenceAudio`; WhisperX sees only those bytes. A deterministic local alignment then combinesthe returned evidence with the one Segment and emits one self-contained `SemanticTake`.-There is no whole-program WhisperX pass. Speech Spine only receives already-semantic Takes and later+There is no whole-program WhisperX pass. Speech Track only receives already-semantic Takes and latertranslates their local frames when assembling the final ProgramSpace and complete semantic map.`@hypit/provider-whisperx-local` is the first concrete adapter; it validates and stages those bytesdiff --git a/packages/whisperx/src/fragment.ts b/packages/whisperx/src/fragment.tsindex f6bc0969..df7373bd 100644--- a/packages/whisperx/src/fragment.ts+++ b/packages/whisperx/src/fragment.ts@@ -1,7 +1,6 @@import { narrativeTypes } from "@hypit/narrative";import { mediaTypes } from "@hypit/media";import { speechTypes } from "@hypit/speech";-import { speechEvidenceTypes } from "@hypit/speech-evidence";import { sealGraphFragment } from "@hypit/elaborator";import { mediaPipelineProducers } from "@hypit/media-pipeline";import { speechAlignmentProducers } from "@hypit/speech-alignment";@@ -44,11 +43,6 @@ export const whisperXSemanticTakeFragment = sealGraphFragment({},],exports: [- {- name: "evidence",- type: speechEvidenceTypes.alignedTranscript,- root: operation("request-whisperx"),- },{name: "take",type: speechTypes.semanticTake,diff --git a/packages/whisperx/src/manifest.ts b/packages/whisperx/src/manifest.tsindex e7e835ae..b7f4b1ff 100644--- a/packages/whisperx/src/manifest.ts+++ b/packages/whisperx/src/manifest.ts@@ -18,7 +18,7 @@ export const whisperXMarkupSurfaces = [{name: "semantic-take",tag: "SemanticTake",mode: "structured",- outputs: [speechEvidenceTypes.alignedTranscript, speechTypes.semanticTake],+ outputs: [speechTypes.semanticTake],vocabulary: {summary:"Measures one normalized Take with WhisperX and aligns one authored Segment into a self-contained SemanticTake.",@@ -36,8 +36,6 @@ export const whisperXMarkupSurfaces = [{summary: "Selects the already normalized SynchronizedMedia measured by WhisperX." },],ports: [- { name: "evidence", type: speechEvidenceTypes.alignedTranscript,- summary: "The provider-neutral aligned transcript WhisperX returned, addressed as `<id>.evidence`." },{ name: "take", type: speechTypes.semanticTake,summary: "The normalized media plus this Segment's authored words and local frame anchors." },],diff --git a/packages/whisperx/src/surface.ts b/packages/whisperx/src/surface.tsindex e6fdfd66..c571d80e 100644--- a/packages/whisperx/src/surface.ts+++ b/packages/whisperx/src/surface.ts@@ -41,10 +41,7 @@ export const decodeWhisperXSemanticTakeSurface: StructuredSurfaceHandler = ({ elid,fragment: whisperXSemanticTakeFragment.id,inputs: { narrative: narrative.ref, segment: segment.ref, media: media.ref },- outputs: {- evidence: `${id}.evidence`,- take: `${id}.take`,- },+ outputs: { take: `${id}.take` },range: element.range,}],fragments: [whisperXSemanticTakeFragment],diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex 18d10175..50695d46 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -167,9 +167,9 @@ importers:'@hypit/seedream':specifier: workspace:*version: link:packages/seedream- '@hypit/semantic-map':+ '@hypit/semantic-track':specifier: workspace:*- version: link:packages/semantic-map+ version: link:packages/semantic-track'@hypit/spatial':specifier: workspace:*version: link:packages/spatial@@ -179,15 +179,12 @@ importers:'@hypit/speech-alignment':specifier: workspace:*version: link:packages/speech-alignment- '@hypit/speech-basis':- specifier: workspace:*- version: link:packages/speech-basis'@hypit/speech-evidence':specifier: workspace:*version: link:packages/speech-evidence- '@hypit/speech-spine':+ '@hypit/speech-track':specifier: workspace:*- version: link:packages/speech-spine+ version: link:packages/speech-track'@hypit/store-sqlite':specifier: workspace:*version: link:packages/store-sqlite@@ -284,9 +281,9 @@ importers:'@hypit/protocol':specifier: workspace:*version: link:../protocol- '@hypit/semantic-map':+ '@hypit/semantic-track':specifier: workspace:*- version: link:../semantic-map+ version: link:../semantic-track'@hypit/temporal':specifier: workspace:*version: link:../temporal@@ -351,9 +348,9 @@ importers:'@hypit/protocol':specifier: workspace:*version: link:../protocol- '@hypit/semantic-map':+ '@hypit/semantic-track':specifier: workspace:*- version: link:../semantic-map+ version: link:../semantic-trackdevDependencies:'@hypit/program-space':specifier: workspace:*@@ -400,9 +397,9 @@ importers:'@hypit/protocol':specifier: workspace:*version: link:../protocol- '@hypit/semantic-map':+ '@hypit/semantic-track':specifier: workspace:*- version: link:../semantic-map+ version: link:../semantic-track'@hypit/svs':specifier: workspace:*version: link:../svs@@ -495,9 +492,9 @@ importers:'@hypit/protocol':specifier: workspace:*version: link:../protocol- '@hypit/semantic-map':+ '@hypit/semantic-track':specifier: workspace:*- version: link:../semantic-map+ version: link:../semantic-track'@hypit/spatial':specifier: workspace:*version: link:../spatial@@ -610,9 +607,6 @@ importers:'@hypit/narrative':specifier: workspace:*version: link:../narrative- '@hypit/semantic-map':- specifier: workspace:*- version: link:../semantic-map'@hypit/spatial':specifier: workspace:*version: link:../spatial@@ -682,9 +676,9 @@ importers:'@hypit/protocol':specifier: workspace:*version: link:../protocol- '@hypit/semantic-map':+ '@hypit/semantic-track':specifier: workspace:*- version: link:../semantic-map+ version: link:../semantic-track'@hypit/spatial':specifier: workspace:*version: link:../spatial@@ -797,6 +791,9 @@ importers:'@hypit/protocol':specifier: workspace:*version: link:../protocol+ '@hypit/semantic-track':+ specifier: workspace:*+ version: link:../semantic-track'@hypit/spatial':specifier: workspace:*version: link:../spatial@@ -1502,9 +1499,9 @@ importers:'@hypit/protocol':specifier: workspace:*version: link:../protocol- '@hypit/semantic-map':+ '@hypit/semantic-track':specifier: workspace:*- version: link:../semantic-map+ version: link:../semantic-track'@hypit/spatial':specifier: workspace:*version: link:../spatial@@ -2011,9 +2008,9 @@ importers:'@hypit/protocol':specifier: workspace:*version: link:../protocol- '@hypit/semantic-map':+ '@hypit/semantic-track':specifier: workspace:*- version: link:../semantic-map+ version: link:../semantic-track'@hypit/spatial':specifier: workspace:*version: link:../spatial@@ -2115,6 +2112,9 @@ importers:'@hypit/protocol':specifier: workspace:*version: link:../protocol+ '@hypit/semantic-track':+ specifier: workspace:*+ version: link:../semantic-trackdevDependencies:'@hypit/core':specifier: workspace:*@@ -2122,6 +2122,9 @@ importers:'@hypit/driver-node':specifier: workspace:*version: link:../driver-node+ '@hypit/svs':+ specifier: workspace:*+ version: link:../svs'@hypit/validation':specifier: workspace:*version: link:../validation@@ -2265,9 +2268,9 @@ importers:'@hypit/protocol':specifier: workspace:*version: link:../protocol- '@hypit/semantic-map':+ '@hypit/semantic-track':specifier: workspace:*- version: link:../semantic-map+ version: link:../semantic-track'@hypit/spatial':specifier: workspace:*version: link:../spatial@@ -2373,17 +2376,29 @@ importers:specifier: workspace:*version: link:../text- packages/semantic-map:+ packages/semantic-track:dependencies:+ '@hypit/component-kit':+ specifier: workspace:*+ version: link:../component-kit+ '@hypit/composition':+ specifier: workspace:*+ version: link:../composition+ '@hypit/media':+ specifier: workspace:*+ version: link:../media'@hypit/narrative':specifier: workspace:*version: link:../narrative+ '@hypit/program-space':+ specifier: workspace:*+ version: link:../program-space'@hypit/protocol':specifier: workspace:*version: link:../protocol- '@hypit/speech-evidence':+ '@hypit/speech':specifier: workspace:*- version: link:../speech-evidence+ version: link:../speechpackages/source:dependencies:@@ -2418,18 +2433,9 @@ importers:'@hypit/media':specifier: workspace:*version: link:../media- '@hypit/program-space':- specifier: workspace:*- version: link:../program-space'@hypit/protocol':specifier: workspace:*version: link:../protocol- '@hypit/semantic-map':- specifier: workspace:*- version: link:../semantic-map- '@hypit/spatial':- specifier: workspace:*- version: link:../spatialpackages/speech-alignment:dependencies:@@ -2448,9 +2454,6 @@ importers:'@hypit/protocol':specifier: workspace:*version: link:../protocol- '@hypit/semantic-map':- specifier: workspace:*- version: link:../semantic-map'@hypit/speech':specifier: workspace:*version: link:../speech@@ -2462,57 +2465,14 @@ importers:specifier: workspace:*version: link:../script- packages/speech-basis:- dependencies:- '@hypit/component-kit':- specifier: workspace:*- version: link:../component-kit- '@hypit/composition':- specifier: workspace:*- version: link:../composition- '@hypit/elaborator':- specifier: workspace:*- version: link:../elaborator- '@hypit/media-track':- specifier: workspace:*- version: link:../media-track- '@hypit/program-space':- specifier: workspace:*- version: link:../program-space- '@hypit/protocol':- specifier: workspace:*- version: link:../protocol- '@hypit/speech':- specifier: workspace:*- version: link:../speech- devDependencies:- '@hypit/core':- specifier: workspace:*- version: link:../core- '@hypit/driver-node':- specifier: workspace:*- version: link:../driver-node- '@hypit/media-pipeline':- specifier: workspace:*- version: link:../media-pipeline- '@hypit/svs':- specifier: workspace:*- version: link:../svs- '@hypit/validation':- specifier: workspace:*- version: link:../validation-packages/speech-evidence:dependencies:'@hypit/protocol':specifier: workspace:*version: link:../protocol- packages/speech-spine:+ packages/speech-track:dependencies:- '@hypit/artifact':- specifier: workspace:*- version: link:../artifact'@hypit/component-kit':specifier: workspace:*version: link:../component-kit@@ -2525,30 +2485,21 @@ importers:'@hypit/markup':specifier: workspace:*version: link:../markup- '@hypit/media':- specifier: workspace:*- version: link:../media- '@hypit/media-pipeline':- specifier: workspace:*- version: link:../media-pipeline- '@hypit/program-space':+ '@hypit/media-track':specifier: workspace:*- version: link:../program-space+ version: link:../media-track'@hypit/protocol':specifier: workspace:*version: link:../protocol- '@hypit/semantic-map':+ '@hypit/semantic-track':specifier: workspace:*- version: link:../semantic-map+ version: link:../semantic-track'@hypit/spatial':specifier: workspace:*version: link:../spatial'@hypit/speech':specifier: workspace:*version: link:../speech- '@hypit/speech-basis':- specifier: workspace:*- version: link:../speech-basis'@hypit/svs':specifier: workspace:*version: link:../svs@@ -2556,9 +2507,6 @@ importers:'@hypit/core':specifier: workspace:*version: link:../core- '@hypit/media-track':- specifier: workspace:*- version: link:../media-track'@hypit/script':specifier: workspace:*version: link:../script@@ -2631,6 +2579,9 @@ importers:'@hypit/script':specifier: workspace:*version: link:../script+ '@hypit/semantic-track':+ specifier: workspace:*+ version: link:../semantic-track'@hypit/validation':specifier: workspace:*version: link:../validation@@ -2674,9 +2625,9 @@ importers:'@hypit/protocol':specifier: workspace:*version: link:../protocol- '@hypit/semantic-map':+ '@hypit/semantic-track':specifier: workspace:*- version: link:../semantic-map+ version: link:../semantic-trackpackages/text:dependencies:@@ -2734,9 +2685,9 @@ importers:'@hypit/protocol':specifier: workspace:*version: link:../protocol- '@hypit/semantic-map':+ '@hypit/semantic-track':specifier: workspace:*- version: link:../semantic-map+ version: link:../semantic-track'@hypit/spatial':specifier: workspace:*version: link:../spatialdiff --git a/services/whisperx/README.md b/services/whisperx/README.mdindex 2d87f720..db6bd733 100644--- a/services/whisperx/README.md+++ b/services/whisperx/README.md@@ -13,7 +13,8 @@ canonical 16 kHz mono PCM s16 WAV```It does not run FFmpeg, modify the authored script, split caption cues, infer SVML Segments, cache-Build results or create a SemanticMap. Missing WhisperX word timing stays missing.+Build results or create a SemanticTake. Missing WhisperX word timing stays missing; the author-side+semantic projection combines this evidence with one explicit Script Segment later.## Installdiff --git a/test/semantic-track-fixture.ts b/test/semantic-track-fixture.tsnew file mode 100644index 00000000..56ff604e--- /dev/null+++ b/test/semantic-track-fixture.ts@@ -0,0 +1,75 @@+import type { ProgramSpace } from "@hypit/program-space";+import type { SemanticTrack } from "@hypit/semantic-track";++type FixtureSegment = {+ readonly id: string;+ readonly frameCount: number;+};++type FixtureAnchor = {+ readonly identity: string;+ readonly frame: number;+};++const audioArtifact = {+ kind: "blob" as const,+ digest: `sha256:${"0".repeat(64)}` as const,+ size: 1,+ mediaType: "audio/wav",+};++/** Minimal real SemanticTrack for tests whose subject is timing rather than speech recognition. */+export function semanticTrackFixture(+ space: ProgramSpace,+ options: {+ readonly id?: string;+ readonly segments?: readonly FixtureSegment[];+ readonly anchors?: readonly FixtureAnchor[];+ } = {},+): SemanticTrack {+ const totalFrames = Math.round(space.durationSec * space.frameRate.numerator / space.frameRate.denominator);+ const segments = options.segments ?? [{ id: "fixture", frameCount: totalFrames }];+ if (segments.reduce((sum, segment) => sum + segment.frameCount, 0) !== totalFrames) {+ throw new Error("SemanticTrack fixture segments must fill ProgramSpace exactly.");+ }+ const anchors = options.anchors ?? [];+ const structuralAnchorIds = new Set(segments.flatMap((segment) => [+ `segment:${segment.id}:start`,+ `segment:${segment.id}:end`,+ ]));+ let startFrame = 0;+ return {+ id: options.id ?? "semantic",+ items: segments.map((segment, index) => {+ const endFrameExclusive = startFrame + segment.frameCount;+ const startAnchorId = `segment:${segment.id}:start`;+ const endAnchorId = `segment:${segment.id}:end`;+ const localAnchors = anchors+ .filter((anchor) => !structuralAnchorIds.has(anchor.identity)+ && anchor.frame >= startFrame+ && (index === segments.length - 1 ? anchor.frame <= endFrameExclusive : anchor.frame < endFrameExclusive))+ .map((anchor) => ({ identity: anchor.identity, frame: anchor.frame - startFrame }));+ const take = {+ media: {+ timeline: { frameRate: space.frameRate, frameCount: segment.frameCount },+ audio: { artifact: audioArtifact },+ },+ segment: {+ segmentId: segment.id,+ startAnchorId,+ endAnchorId,+ startFrame: 0,+ endFrameExclusive: segment.frameCount,+ },+ tokens: [],+ anchors: [+ { identity: startAnchorId, frame: 0 },+ { identity: endAnchorId, frame: segment.frameCount },+ ...localAnchors.filter((anchor) => anchor.identity !== startAnchorId && anchor.identity !== endAnchorId),+ ],+ };+ startFrame = endFrameExclusive;+ return { take };+ }),+ };+}diff --git a/test/support/video-domain.ts b/test/support/video-domain.tsindex 5824c016..87d801a5 100644--- a/test/support/video-domain.ts+++ b/test/support/video-domain.ts@@ -3,7 +3,7 @@ import { compositionComponent, compositionManifest } from "@hypit/composition";import { mediaComponent, mediaManifest } from "@hypit/media";import { narrativeManifest } from "@hypit/narrative";import { programSpaceManifest } from "@hypit/program-space";-import { semanticMapManifest } from "@hypit/semantic-map";+import { semanticTrackManifest } from "@hypit/semantic-track";import { spatialComponent, spatialManifest } from "@hypit/spatial";import { speechManifest } from "@hypit/speech";import { speechEvidenceManifest } from "@hypit/speech-evidence";@@ -18,7 +18,7 @@ export const videoContractManifests = [programSpaceManifest,speechManifest,speechEvidenceManifest,- semanticMapManifest,+ semanticTrackManifest,spatialManifest,temporalManifest,visualIrManifest,