Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
fix(speech): separate real and preview timing evidence
kashorin6007abb3ccf7
diff --git a/packages/semantic-take-estimate/README.md b/packages/semantic-take-estimate/README.mdindex 354ddc41..f12a95a4 100644--- a/packages/semantic-take-estimate/README.md+++ b/packages/semantic-take-estimate/README.md@@ -1,20 +1,18 @@# `@hypit/semantic-take-estimate`-An explicit, provider-free alternative to measured speech alignment for preview Builds.+An explicit, provider-free semantic step for preview Builds.```text-Narrative + Segment + normalized SynchronizedMedia + SpeechEstimatePolicy+Narrative + spoken Segment + normalized SynchronizedMedia + SpeechEstimatePolicy-> syllable-weighted Token windows with real gaps-> SemanticTake```-The package consumes an existing normalized media value. It knows nothing about how the video was-generated and does not run FFmpeg, OpenCV, WhisperX or any other external process. It preserves the-Script's exact Segment, Token and Anchor identities and estimates only their local frame positions.-The normalized media's complete frame count is the allocation budget: after short edge and word-gaps, every remaining frame is distributed across Tokens by pronunciation-unit weight. A longer-normalized video therefore stretches the whole Script across that longer interval instead of-leaving a fixed-length estimate at the beginning.+The package consumes existing normalized media. It knows nothing about how that media was made and+does not run an acoustic model. For a spoken Segment it preserves the Script's exact Segment, Token+and Anchor identities and predicts only their local frame positions. The media's complete frame+count is the allocation budget: after short edge and word gaps, every remaining frame is+distributed across Tokens by pronunciation-unit weight.```svml<estimated:SemanticTake id="opening-estimated" narrative={story}@@ -22,9 +20,19 @@ leaving a fixed-length estimate at the beginning.language="en" pace="normal" rounding="round"/>```-The delivery policy is the element's own: write it inline as above, or name an SVS Recipe with-`policy={recipes.speech.normal}` carrying the same properties. Nothing else in the graph computes-it; durations themselves are author literals, measured beforehand with `hypit measure`.+For a spoken Segment, write the delivery policy inline as above or name an SVS Recipe with+`policy={recipes.speech.normal}`. `hypit measure` helps the author choose the media duration and+balance delivery across spoken Segments; the prediction then distributes that known frame domain+across the authored words.++An empty Segment already has all of the semantic timing it can carry: its start and end. Omit the+policy and the Surface maps those two authored Anchors to the prepared media boundaries without a+prediction step:++```svml+<estimated:SemanticTake id="pause-preview" narrative={story}+ segment={story.segment.pause} media={pause-media.media}/>+```The result has the ordinary `@hypit/speech@1#SemanticTake` type. A Run can therefore choose thisBuild output while real A-roll is unavailable, then later choose the corresponding WhisperX output@@ -53,9 +61,9 @@ The package also registers `@hypit/semantic-take-estimate@1#semantic-take` as a| `media` | `@hypit/media@1#SynchronizedMedia` || `policy` | `@hypit/estimate@1#SpeechEstimatePolicy` |-Its export is `take`, an ordinary SemanticTake. The policy is a typed value with `language`,-`pace` or `rate`, `rounding` and optional `paddingSec`.-The Author Surface publishes its authored policy as `<id>.policy`. For an Author entry containing+Its export is `take`, an ordinary SemanticTake. For a spoken Segment, the policy is a typed value+with `language`, `pace` or `rate`, `rounding` and optional `paddingSec`; the Author Surface publishes+it as `<id>.policy`. For an Author entry containingthe `opening-estimated` declaration above, a Run can use:```svrun@@ -69,7 +77,7 @@ the `opening-estimated` declaration above, a Run can use:<satisfy output="opening-semantic.take" candidate="timing.take"/>```-This selects estimated timing for the measured Take while retaining the required prepared-media-dependency and any Run choice for its source bytes. The estimate allocates the media's complete-frame count; it neither generates nor normalizes that media. For a wordless Segment it maps the-Segment's two boundary Anchors directly to the media's first and final frame.+This selects predicted timing while retaining the required prepared-media dependency and any Run+choice for its source bytes. The estimate allocates the media's complete frame count; it neither+generates nor normalizes that media. An empty Segment instead uses the Surface's boundary branch+shown above and has no policy Fragment to select.diff --git a/packages/semantic-take-estimate/src/fragment.ts b/packages/semantic-take-estimate/src/fragment.tsindex 5d13955c..dc1faea0 100644--- a/packages/semantic-take-estimate/src/fragment.ts+++ b/packages/semantic-take-estimate/src/fragment.ts@@ -2,7 +2,7 @@ import { estimateTypes } from "@hypit/estimate";import { sealGraphFragment } from "@hypit/elaborator";import { mediaTypes } from "@hypit/media";import { narrativeTypes } from "@hypit/narrative";-import { speechTypes } from "@hypit/speech";+import { speechProducers, speechTypes } from "@hypit/speech";import { semanticTakeEstimateProducers } from "./manifest.js";@@ -29,3 +29,23 @@ export const semanticTakeEstimateFragment = sealGraphFragment({}],exports: [{ name: "take", type: speechTypes.semanticTake, root: operation("materialize") }],});++/** The zero-Token branch of the same preview Surface; media boundaries need no prediction policy. */+export const semanticTakeEstimateBoundaryFragment = sealGraphFragment({+ inputs: [+ { name: "narrative", type: narrativeTypes.narrative },+ { name: "segment", type: narrativeTypes.excerpt },+ { name: "media", type: mediaTypes.synchronized },+ ],+ operations: [{+ id: "materialize-boundaries",+ producer: speechProducers.materializeSegmentBoundaries,+ inputs: {+ narrative: input("narrative"),+ segment: input("segment"),+ media: input("media"),+ },+ result: { kind: "output", name: "take" },+ }],+ exports: [{ name: "take", type: speechTypes.semanticTake, root: operation("materialize-boundaries") }],+});diff --git a/packages/semantic-take-estimate/src/index.ts b/packages/semantic-take-estimate/src/index.tsindex fc100ee5..b2dda6ce 100644--- a/packages/semantic-take-estimate/src/index.ts+++ b/packages/semantic-take-estimate/src/index.ts@@ -1,5 +1,5 @@export { semanticTakeEstimateComponent } from "./component.js";-export { semanticTakeEstimateFragment } from "./fragment.js";+export { semanticTakeEstimateBoundaryFragment, semanticTakeEstimateFragment } from "./fragment.js";export {semanticTakeEstimateManifest,semanticTakeEstimateMarkupSurfaces,diff --git a/packages/semantic-take-estimate/src/manifest.ts b/packages/semantic-take-estimate/src/manifest.tsindex cfce1421..54a19812 100644--- a/packages/semantic-take-estimate/src/manifest.ts+++ b/packages/semantic-take-estimate/src/manifest.ts@@ -18,18 +18,18 @@ export const semanticTakeEstimateMarkupSurfaces = [{outputs: [estimateTypes.speechPolicy, speechTypes.semanticTake],vocabulary: {summary:- "Projects estimated word windows or wordless Segment boundaries onto normalized media and publishes a SemanticTake.",+ "Publishes a preview SemanticTake: predicted word windows for speech, or exact prepared-media boundaries for an empty Segment.",attributes: [{ name: "id", kind: "identifier", required: true,summary: "Names this estimated alignment and the SemanticTake it publishes." },{ name: "narrative", kind: "reference", required: true, accepts: [narrativeTypes.narrative],summary: "Selects the authored Narrative that owns every Token and Anchor identity." },{ name: "segment", kind: "reference", required: true, accepts: [narrativeTypes.excerpt],- summary: "Selects the single authored Segment represented by the normalized video." },+ summary: "Selects the single authored Segment represented by the prepared media." },{ name: "media", kind: "reference", required: true, accepts: [mediaTypes.synchronized],summary: "Selects the already normalized SynchronizedMedia whose exact frame domain receives the estimate." },{ name: "policy", kind: "reference", required: false, accepts: [svsRecipeType],- summary: "Selects an SVS Recipe carrying the delivery policy (language, pace or rate, rounding), in place of the inline attributes.",+ summary: "For a Segment with Tokens, selects an SVS Recipe carrying the delivery policy in place of inline attributes.",recipe: [{ name: "language", required: true, summary: "Selects the counting rules the Segment is read with, or detects them.", values: ["auto", "en", "zh", "ja", "es"] },{ name: "pace", required: false, summary: "Selects a named delivery density.", values: ["slow", "normal", "fast"] },@@ -37,7 +37,7 @@ export const semanticTakeEstimateMarkupSurfaces = [{{ name: "rounding", required: true, summary: "Selects how an estimated duration is rounded.", values: ["none", "round", "ceil"] },] },{ name: "language", kind: "literal", required: false, values: ["auto", "en", "zh", "ja", "es"],- summary: "Selects the counting rules the Segment is read with, or detects them from its Text." },+ summary: "For a Segment with Tokens, selects the counting rules or detects them from its Text." },{ name: "pace", kind: "literal", required: false, values: ["slow", "normal", "fast"],summary: "Selects a named delivery density for the chosen language." },{ name: "rate", kind: "literal", required: false,@@ -49,13 +49,14 @@ export const semanticTakeEstimateMarkupSurfaces = [{{ name: "take", type: speechTypes.semanticTake,summary: "The estimated SemanticTake, addressed as `<id>.take`." },{ name: "policy", type: estimateTypes.speechPolicy,- summary: "The sealed delivery policy this element weights Tokens with, addressed as `<id>.policy`." },+ summary: "For a Segment with Tokens, the sealed delivery policy addressed as `<id>.policy`." },],example: `<estimated:SemanticTake id="opening-estimated" narrative={story}segment={story.segment.opening} media={opening-media.media}language="en" pace="normal" rounding="round"/>`,notes: [- "The policy is the element's own: write it inline or name an SVS Recipe with policy; never both.",+ "A Segment with Tokens states its policy inline or names one SVS Recipe with policy.",+ "An empty Segment uses only id, narrative, segment and media; its prepared-media boundaries already contain all required timing.","This package does not measure audio and never claims WhisperX or other acoustic evidence.","Every spoken Token uses its original Script identity. A wordless Segment keeps its two authored boundary Anchors.","Pronunciation-unit weights determine relative word lengths; adjacent words and both media edges retain non-zero frame gaps.",diff --git a/packages/semantic-take-estimate/src/program.ts b/packages/semantic-take-estimate/src/program.tsindex 36e3d0ad..7c5d0474 100644--- a/packages/semantic-take-estimate/src/program.ts+++ b/packages/semantic-take-estimate/src/program.ts@@ -7,7 +7,7 @@ import type { SpeechEstimatePolicy } from "@hypit/estimate";import { verifySynchronizedMedia } from "@hypit/media";import type { SynchronizedMedia } from "@hypit/media";import type { Narrative, NarrativeExcerpt } from "@hypit/narrative";-import { materializeSemanticTake } from "@hypit/speech";+import { materializeSegmentBoundaryTake, materializeSemanticTake } from "@hypit/speech";import type { SemanticTake, SemanticTakeTimedToken, SemanticTakeTiming } from "@hypit/speech";const EDGE_GAP_SECONDS = 0.12;@@ -48,21 +48,11 @@ export function estimateSemanticTakeTiming(verifySynchronizedMedia(media);assertSpeechEstimatePolicy(policy);const tokens = segmentTokens(narrative, excerpt);- const segment = narrative.segments.find((candidate) => candidate.id === excerpt.id)!;if (tokens.length === 0) {- const frameByAnchor = new Map<string, number>([- [segment.startAnchorId, 0],- [segment.endAnchorId, media.timeline.frameCount],- ]);- const anchors = narrative.semanticIndex.anchors- .filter((anchor) => anchor.segmentId === excerpt.id)- .map((anchor) => {- const frame = frameByAnchor.get(anchor.id);- if (frame === undefined) throw new Error(`Estimated timing cannot locate authored Anchor ${anchor.id}.`);- return { identity: anchor.id, frame };- });- return { tokens: [], anchors };+ const take = materializeSegmentBoundaryTake(narrative, excerpt, media);+ return { tokens: take.tokens, anchors: take.anchors };}+ const segment = narrative.segments.find((candidate) => candidate.id === excerpt.id)!;const rate = media.timeline.frameRate.numerator / media.timeline.frameRate.denominator;const edgeGap = Math.max(1, Math.round(EDGE_GAP_SECONDS * rate));const wordGap = Math.max(1, Math.round(WORD_GAP_SECONDS * rate));diff --git a/packages/semantic-take-estimate/src/surface.ts b/packages/semantic-take-estimate/src/surface.tsindex 7e764e28..c4bf0180 100644--- a/packages/semantic-take-estimate/src/surface.ts+++ b/packages/semantic-take-estimate/src/surface.ts@@ -9,12 +9,13 @@ import type {} from "@hypit/markup";import { mediaTypes } from "@hypit/media";import { narrativeTypes } from "@hypit/narrative";+import type { NarrativeExcerpt } from "@hypit/narrative";import type { CanonicalValue } from "@hypit/protocol";import { sameType } from "@hypit/protocol";import { svsRecipeType } from "@hypit/svs";import type { SvsRecipe } from "@hypit/svs";-import { semanticTakeEstimateFragment } from "./fragment.js";+import { semanticTakeEstimateBoundaryFragment, semanticTakeEstimateFragment } from "./fragment.js";function reference(element: StructuredElement,@@ -32,6 +33,12 @@ function reference(return resolved;}+function authoredExcerpt(value: SurfaceResolvedReference): NarrativeExcerpt | undefined {+ return value.record?.value.kind === "inline"+ ? value.record.value.value as unknown as NarrativeExcerpt+ : undefined;+}+/*** The delivery policy the estimate weights Tokens with: an SVS Recipe named by `policy`, or the same* properties written inline. The element owns its policy; nothing elsewhere in the graph computes it.@@ -54,17 +61,37 @@ function policyOf(export const decodeSemanticTakeEstimateSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {const required = ["id", "narrative", "segment", "media"];- const allowed = new Set([...required, "policy", ...speechEstimatePolicyProperties]);const missing = required.filter((name) => element.attributes[name] === undefined);+ if (missing.length > 0) {+ throw new Error(`${element.name} requires ${required.join(", ")}`);+ }+ const narrative = reference(element, "narrative", narrativeTypes.narrative, resolveReference);+ const segment = reference(element, "segment", narrativeTypes.excerpt, resolveReference);+ const media = reference(element, "media", mediaTypes.synchronized, resolveReference);+ const excerpt = authoredExcerpt(segment);+ const hasNoTokens = excerpt !== undefined && excerpt.tokenStart === excerpt.tokenEndExclusive;+ const allowed = new Set(hasNoTokens ? required : [...required, "policy", ...speechEstimatePolicyProperties]);const unknown = Object.keys(element.attributes).filter((name) => !allowed.has(name));- if (missing.length > 0 || unknown.length > 0) {- throw new Error(`${element.name} requires ${required.join(", ")} and a policy (policy={recipe} or ${speechEstimatePolicyProperties.join(", ")})${- unknown.length === 0 ? "" : `; unknown ${unknown.join(", ")}`}`);+ if (unknown.length > 0) {+ throw new Error(`${element.name} has unknown or inapplicable attributes ${unknown.join(", ")}`);}if (element.children.some((child) => child.kind === "element" || child.value.trim().length > 0)) {throw new Error(`${element.name} must be empty`);}const id = textAttribute(element, "id");+ if (hasNoTokens) {+ return {+ records: [],+ components: [{+ id,+ fragment: semanticTakeEstimateBoundaryFragment.id,+ inputs: { narrative: narrative.ref, segment: segment.ref, media: media.ref },+ outputs: { take: `${id}.take` },+ range: element.range,+ }],+ fragments: [semanticTakeEstimateBoundaryFragment],+ };+ }const policyId = `${id}.policy`;return {records: [{@@ -77,9 +104,9 @@ export const decodeSemanticTakeEstimateSurface: StructuredSurfaceHandler = ({ elid,fragment: semanticTakeEstimateFragment.id,inputs: {- narrative: reference(element, "narrative", narrativeTypes.narrative, resolveReference).ref,- segment: reference(element, "segment", narrativeTypes.excerpt, resolveReference).ref,- media: reference(element, "media", mediaTypes.synchronized, resolveReference).ref,+ narrative: narrative.ref,+ segment: segment.ref,+ media: media.ref,policy: { kind: "record", id: policyId },},outputs: { take: `${id}.take` },diff --git a/packages/semantic-take-estimate/test/semantic-take-estimate.test.ts b/packages/semantic-take-estimate/test/semantic-take-estimate.test.tsindex d72a9f70..8c722d95 100644--- a/packages/semantic-take-estimate/test/semantic-take-estimate.test.ts+++ b/packages/semantic-take-estimate/test/semantic-take-estimate.test.ts@@ -2,16 +2,21 @@ import assert from "node:assert/strict";import test from "node:test";import { sealSpeechEstimatePolicy } from "@hypit/estimate";-import { sealSynchronizedMedia } from "@hypit/media";+import { mediaTypes, sealSynchronizedMedia } from "@hypit/media";+import { parseStructuredElement } from "@hypit/markup";+import type { SurfaceResolvedReference } from "@hypit/markup";import {assertCaptionDocumentIdentity,assertNarrativeExcerptIdentity,assertNarrativeIdentity,+ narrativeTypes,} from "@hypit/narrative";import type { Narrative } from "@hypit/narrative";+import { materializeSegmentBoundaryTake, speechProducers } from "@hypit/speech";import { fixtureResource } from "../../../test/fixture-resource.js";import {+ decodeSemanticTakeEstimateSurface,estimateSemanticTakeTiming,materializeEstimatedSemanticTake,} from "@hypit/semantic-take-estimate";@@ -150,7 +155,7 @@ test("a wordless Segment receives the prepared media boundaries without an audiocueBreaks: [],});- const take = materializeEstimatedSemanticTake(wordless, wordlessExcerpt, media, policy);+ const take = materializeSegmentBoundaryTake(wordless, wordlessExcerpt, media);assert.deepEqual(take.tokens, []);assert.deepEqual(take.anchors, [{ identity: "pause:start", frame: 0 },@@ -158,3 +163,59 @@ test("a wordless Segment receives the prepared media boundaries without an audio]);assert.equal(take.media.audio, undefined);});++test("the preview Surface sends an empty Segment straight to boundary materialization", async () => {+ const wordless: Narrative = {+ id: "wordless-preview",+ segments: [{+ id: "pause",+ startAnchorId: "pause:start",+ endAnchorId: "pause:end",+ tokenStart: 0,+ tokenEndExclusive: 0,+ }],+ tokens: [], turns: [], selections: [], moments: [],+ semanticIndex: { anchors: [+ { id: "pause:start", kind: "segment-start", segmentId: "pause" },+ { id: "pause:end", kind: "segment-end", segmentId: "pause" },+ ] },+ };+ const wordlessExcerpt = {+ narrativeId: wordless.id,+ kind: "segment" as const,+ id: "pause",+ tokenStart: 0,+ tokenEndExclusive: 0,+ };+ const authored = (+ path: string,+ type: SurfaceResolvedReference["type"],+ value: unknown,+ ): SurfaceResolvedReference => ({+ path,+ ref: { kind: "record", id: path },+ type,+ record: { id: path, type, value: { kind: "inline", value: value as never } },+ });+ const refs = new Map<string, SurfaceResolvedReference>([+ ["story", authored("story", narrativeTypes.narrative, wordless)],+ ["story.segment.pause", authored("story.segment.pause", narrativeTypes.excerpt, wordlessExcerpt)],+ ["pause-media.media", authored("pause-media.media", mediaTypes.synchronized, media)],+ ]);+ const output = await decodeSemanticTakeEstimateSurface({+ sourceName: "preview.svml",+ element: parseStructuredElement({ name: "preview.svml", text:+ '<estimate:SemanticTake id="pause-preview" narrative={story} segment={story.segment.pause} media={pause-media.media}/>',+ }, 0).element,+ resolveReference: (path) => refs.get(path),+ resolveAsset: () => { throw new Error("No assets are resolved by this test."); },+ });++ assert.equal(output.records.length, 0);+ assert.equal(output.fragments[0]?.operations[0]?.producer.name, speechProducers.materializeSegmentBoundaries.name);+ assert.deepEqual(output.components[0]?.inputs, {+ narrative: { kind: "record", id: "story" },+ segment: { kind: "record", id: "story.segment.pause" },+ media: { kind: "record", id: "pause-media.media" },+ });+});diff --git a/packages/speech/README.md b/packages/speech/README.mdindex c5fe4d8b..7e8f6c48 100644--- a/packages/speech/README.md+++ b/packages/speech/README.md@@ -9,3 +9,8 @@ A `SemanticTake` contains one normalized media product, one authored Segment, itframe anchors. Ordered Takes are assembled by `@hypit/speech-track` into a `SemanticTrack`, which isthe program's continuous semantic skeleton. `SpeechEvidenceAudio` carries canonical 16 kHz WAV bytesand their exact sample count; it never carries Script or Segment identity.++The package also owns deterministic boundary materialization for a Segment with no Tokens. It maps+the Segment start and end to the prepared media's frame domain and publishes the same+`SemanticTake` type. The preview and real-media author surfaces both select this operation for an+empty Segment. There are no word windows to predict or acoustic evidence to request.diff --git a/packages/speech/src/component.ts b/packages/speech/src/component.tsindex ccfa13bf..370382c8 100644--- a/packages/speech/src/component.ts+++ b/packages/speech/src/component.ts@@ -1,4 +1,7 @@import type { ComponentPackage } from "@hypit/component-kit";+import type { SynchronizedMedia } from "@hypit/media";+import type { Narrative, NarrativeExcerpt } from "@hypit/narrative";+import { canonicalize } from "@hypit/protocol";import type { StoredValue } from "@hypit/protocol";import {@@ -6,7 +9,8 @@ import {assertSpeechDurationIdentity,assertSpeechEvidenceAudioIdentity,} from "./identity.js";-import { speechTypes } from "./manifest.js";+import { materializeSegmentBoundaryTake } from "./materialize.js";+import { speechProducers, speechTypes } from "./manifest.js";import type { SemanticTake, SpeechDuration, SpeechEvidenceAudio } from "./types.js";function inline<T>(value: StoredValue, subject: string): T {@@ -20,4 +24,20 @@ export const speechComponent = {{ type: speechTypes.evidenceAudio, handler: ({ value }) => assertSpeechEvidenceAudioIdentity(inline<SpeechEvidenceAudio>(value, "SpeechEvidenceAudio")) },{ type: speechTypes.semanticTake, handler: ({ value }) => assertSemanticTakeIdentity(inline<SemanticTake>(value, "SemanticTake")) },],+ producers: [{+ producer: speechProducers.materializeSegmentBoundaries,+ handler: ({ inputs }) => ({+ outputs: {+ take: {+ kind: "inline",+ value: canonicalize(materializeSegmentBoundaryTake(+ inline<Narrative>(inputs.narrative!.value, "Narrative"),+ inline<NarrativeExcerpt>(inputs.segment!.value, "NarrativeExcerpt"),+ inline<SynchronizedMedia>(inputs.media!.value, "SynchronizedMedia"),+ )),+ },+ },+ needs: {},+ }),+ }],} satisfies ComponentPackage;diff --git a/packages/speech/src/index.ts b/packages/speech/src/index.tsindex aead4563..ca413bfd 100644--- a/packages/speech/src/index.ts+++ b/packages/speech/src/index.ts@@ -1,6 +1,6 @@export { assertSemanticTakeIdentity, assertSpeechDurationIdentity, assertSpeechEvidenceAudioIdentity, sealSemanticTake, sealSpeechDuration, sealSpeechEvidenceAudio, speechEvidenceSampleBoundary } from "./identity.js";export { speechComponent } from "./component.js";-export { materializeSemanticTake } from "./materialize.js";-export { speechDependency, speechManifest, speechModuleRef, speechTypes } from "./manifest.js";+export { materializeSegmentBoundaryTake, materializeSemanticTake } from "./materialize.js";+export { speechDependency, speechManifest, speechModuleRef, speechProducers, speechTypes } from "./manifest.js";export { semanticTakeSchema, speechDurationSchema, speechEvidenceAudioSchema } from "./schema.js";export type * from "./types.js";diff --git a/packages/speech/src/manifest.ts b/packages/speech/src/manifest.tsindex 07c338b3..ae17b07b 100644--- a/packages/speech/src/manifest.ts+++ b/packages/speech/src/manifest.ts@@ -1,5 +1,6 @@-import { mediaDependency } from "@hypit/media";-import type { ModuleManifest, TypeRef } from "@hypit/protocol";+import { mediaDependency, mediaTypes } from "@hypit/media";+import { narrativeDependency, narrativeTypes } from "@hypit/narrative";+import type { ModuleManifest, ProducerRef, TypeRef } from "@hypit/protocol";import { semanticTakeSchema, speechDurationSchema, speechEvidenceAudioSchema } from "./schema.js";export const speechModuleRef = { name: "@hypit/speech", version: "1" } as const;export const speechTypes = {@@ -7,10 +8,23 @@ export const speechTypes = {evidenceAudio: { module: speechModuleRef, name: "SpeechEvidenceAudio" },semanticTake: { module: speechModuleRef, name: "SemanticTake" },} satisfies Record<string, TypeRef>;+export const speechProducers = {+ materializeSegmentBoundaries: { module: speechModuleRef, name: "materialize-segment-boundaries" },+} satisfies Record<string, ProducerRef>;export const speechManifest: ModuleManifest = { format: "hypit.module@1", name: speechModuleRef.name, version: speechModuleRef.version,- dependencies: [mediaDependency],+ dependencies: [mediaDependency, narrativeDependency],types: [{ name: speechTypes.duration.name },{ name: speechTypes.evidenceAudio.name },{ name: speechTypes.semanticTake.name }],- capabilities: [], producers: [] };+ capabilities: [],+ producers: [{+ name: speechProducers.materializeSegmentBoundaries.name,+ inputs: [+ { name: "narrative", type: narrativeTypes.narrative },+ { name: "segment", type: narrativeTypes.excerpt },+ { name: "media", type: mediaTypes.synchronized },+ ],+ outputs: [{ name: "take", type: speechTypes.semanticTake }],+ needs: [],+ }] };export const speechDependency = { module: speechModuleRef } as const;diff --git a/packages/speech/src/materialize.ts b/packages/speech/src/materialize.tsindex afb3b834..2ee726c9 100644--- a/packages/speech/src/materialize.ts+++ b/packages/speech/src/materialize.ts@@ -82,3 +82,22 @@ export function materializeSemanticTake(assertSemanticTakeIdentity(take);return take;}++/** Materialize the exact media boundaries of an authored Segment that contains no Tokens. */+export function materializeSegmentBoundaryTake(+ narrative: Narrative,+ excerpt: NarrativeExcerpt,+ media: SynchronizedMedia,+): SemanticTake {+ const segment = authoredSegment(narrative, excerpt);+ if (segment.tokenStart !== segment.tokenEndExclusive) {+ throw new Error(`Segment ${segment.id} contains spoken Tokens and needs token timing.`);+ }+ return materializeSemanticTake(narrative, excerpt, media, {+ tokens: [],+ anchors: [+ { identity: segment.startAnchorId, frame: 0 },+ { identity: segment.endAnchorId, frame: media.timeline.frameCount },+ ],+ });+}diff --git a/packages/whisperx/README.md b/packages/whisperx/README.mdindex 37621057..b1c2cbf5 100644--- a/packages/whisperx/README.md+++ b/packages/whisperx/README.md@@ -9,12 +9,22 @@ The package contains no credentials, Python environment or queue. Providers tranrequest directly into provider-neutral `AlignedTranscriptEvidence`. There is no vendor-shapedEvidence wrapper or pass-through normalization node in the graph.-`<whisperx:SemanticTake>` consumes one normalized `SynchronizedMedia`, exactly one Script Segment,-and a required `language="en"` or `language="zh"`. The language is passed directly to WhisperX;-Script text and audio are never inspected to choose it implicitly. `@hypit/media-pipeline` projects-that Take's 48 kHz audio to canonical 16 kHz mono-`SpeechEvidenceAudio`; WhisperX sees only those bytes. A deterministic local alignment then combines-the returned evidence with the one Segment and emits one self-contained `SemanticTake`.+`<whisperx:SemanticTake>` is the real-media semantic Surface. It consumes one normalized+`SynchronizedMedia` and exactly one Script Segment. When that Segment contains Tokens, it also+requires `language="en"`, `language="zh"` or `language="es"`. The language is passed directly to+WhisperX; Script text and audio are not used to choose it implicitly. `@hypit/media-pipeline`+projects the Take's audio to canonical 16 kHz mono `SpeechEvidenceAudio`; WhisperX sees only those+bytes. A deterministic local alignment then combines the returned evidence with the Segment and+emits one self-contained `SemanticTake`.++When the authored Segment has no Tokens, write the same Surface without `language`. Its start and+end Anchors map directly to the prepared media's first and final frame. There are no words to align,+so this branch requests no evidence audio and no WhisperX capability:++```svml+<whisperx:SemanticTake id="pause" narrative={story}+ segment={story.segment.pause} media={pause-media.media}/>+```There is no whole-program WhisperX pass. Speech Track only receives already-semantic Takes and latertranslates their local frames when assembling the final ProgramSpace and complete semantic map.diff --git a/packages/whisperx/src/fragment.ts b/packages/whisperx/src/fragment.tsindex 4ea41268..98bd424c 100644--- a/packages/whisperx/src/fragment.ts+++ b/packages/whisperx/src/fragment.ts@@ -1,6 +1,6 @@import { narrativeTypes } from "@hypit/narrative";import { mediaTypes } from "@hypit/media";-import { speechTypes } from "@hypit/speech";+import { speechProducers, speechTypes } from "@hypit/speech";import { sealGraphFragment } from "@hypit/elaborator";import { mediaPipelineProducers } from "@hypit/media-pipeline";import { speechAlignmentProducers } from "@hypit/speech-alignment";@@ -54,3 +54,23 @@ export const whisperXSemanticTakeFragment = sealGraphFragment({},],});++/** The zero-Token branch of the same real-media Surface; media boundaries need no acoustic Need. */+export const whisperXBoundarySemanticTakeFragment = sealGraphFragment({+ inputs: [+ { name: "narrative", type: narrativeTypes.narrative },+ { name: "segment", type: narrativeTypes.excerpt },+ { name: "media", type: mediaTypes.synchronized },+ ],+ operations: [{+ id: "materialize-boundaries",+ producer: speechProducers.materializeSegmentBoundaries,+ inputs: {+ narrative: input("narrative"),+ segment: input("segment"),+ media: input("media"),+ },+ result: { kind: "output", name: "take" },+ }],+ exports: [{ name: "take", type: speechTypes.semanticTake, root: operation("materialize-boundaries") }],+});diff --git a/packages/whisperx/src/index.ts b/packages/whisperx/src/index.tsindex a39f9cdf..e7478996 100644--- a/packages/whisperx/src/index.ts+++ b/packages/whisperx/src/index.ts@@ -1,7 +1,7 @@export {whisperXComponent,} from "./component.js";-export { whisperXSemanticTakeFragment } from "./fragment.js";+export { whisperXBoundarySemanticTakeFragment, whisperXSemanticTakeFragment } from "./fragment.js";export {assertWhisperXEvidenceWav,verifyWhisperXAlignmentRequest,diff --git a/packages/whisperx/src/manifest.ts b/packages/whisperx/src/manifest.tsindex 24defab0..445d988f 100644--- a/packages/whisperx/src/manifest.ts+++ b/packages/whisperx/src/manifest.ts@@ -1,6 +1,6 @@import { speechDependency, speechTypes } from "@hypit/speech";import { mediaDependency, mediaTypes } from "@hypit/media";-import { narrativeTypes } from "@hypit/narrative";+import { narrativeDependency, narrativeTypes } from "@hypit/narrative";import { speechEvidenceDependency, speechEvidenceTypes } from "@hypit/speech-evidence";import type { CapabilityRef, ModuleManifest, ProducerRef, TypeRef } from "@hypit/protocol";import { mediaPipelineManifest, mediaPipelineModuleRef } from "@hypit/media-pipeline";@@ -24,7 +24,7 @@ export const whisperXMarkupSurfaces = [{outputs: [whisperXTypes.language, speechTypes.semanticTake],vocabulary: {summary:- "Measures one normalized Take with WhisperX and aligns one authored Segment into a self-contained SemanticTake.",+ "Publishes a real-media SemanticTake: WhisperX-aligned words for speech, or exact prepared-media boundaries for an empty Segment.",attributes: [{ name: "id", kind: "identifier", required: true,summary: "Names this alignment and prefixes the bindings it publishes." },@@ -36,10 +36,10 @@ export const whisperXMarkupSurfaces = [{summary: "Selects the single authored Segment performed by this Take." },{ name: "media", kind: "reference", required: true,accepts: [mediaTypes.synchronized],- summary: "Selects the already normalized SynchronizedMedia measured by WhisperX." },- { name: "language", kind: "literal", required: true,+ summary: "Selects the already normalized SynchronizedMedia represented by this Segment." },+ { name: "language", kind: "literal", required: false,values: ["en", "zh", "es"],- summary: "Explicitly selects the English, Chinese or Spanish WhisperX transcription and alignment models." },+ summary: "For a Segment with Tokens, explicitly selects the English, Chinese or Spanish WhisperX models." },],ports: [{ name: "take", type: speechTypes.semanticTake,@@ -48,8 +48,9 @@ export const whisperXMarkupSurfaces = [{example: `<whisperx:SemanticTake id="opening" narrative={story}segment={story.segment.opening} media={opening-media.media} language="en"/>`,notes: [- "All five attributes are required; the element accepts no children and no text content.",- "Language is never detected from Script text or audio; each alignment call states en, zh or es explicitly.",+ "A Segment with Tokens states all five attributes and sends its prepared audio for alignment.",+ "An empty Segment omits language and maps its authored start/end Anchors directly to the prepared-media boundaries.",+ "For speech, language is never detected from Script text or audio; each alignment call states en, zh or es explicitly.","Importing this package is what selects the WhisperX model family; the Runtime separately binds the alignment Need to an Endpoint.",],},@@ -64,6 +65,7 @@ export const whisperXManifest: ModuleManifest = {speechDependency,speechEvidenceDependency,mediaDependency,+ narrativeDependency,{ module: mediaPipelineModuleRef },{ module: speechAlignmentModuleRef },],diff --git a/packages/whisperx/src/surface.ts b/packages/whisperx/src/surface.tsindex 957a35e8..c5b9be7a 100644--- a/packages/whisperx/src/surface.ts+++ b/packages/whisperx/src/surface.ts@@ -8,9 +8,10 @@ import {} from "@hypit/markup";import { sameType } from "@hypit/protocol";import { narrativeTypes } from "@hypit/narrative";+import type { NarrativeExcerpt } from "@hypit/narrative";import { mediaTypes } from "@hypit/media";-import { whisperXSemanticTakeFragment } from "./fragment.js";+import { whisperXBoundarySemanticTakeFragment, whisperXSemanticTakeFragment } from "./fragment.js";import { whisperXTypes } from "./manifest.js";import type { WhisperXLanguage } from "./types.js";@@ -28,15 +29,38 @@ function reference(return value;}+function authoredExcerpt(value: SurfaceResolvedReference): NarrativeExcerpt | undefined {+ return value.record?.value.kind === "inline"+ ? value.record.value.value as unknown as NarrativeExcerpt+ : undefined;+}+export const decodeWhisperXSemanticTakeSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- exactAttributes(element, ["id", "narrative", "segment", "media", "language"]);+ const narrative = reference(element, "narrative", narrativeTypes.narrative, resolveReference);+ const segment = reference(element, "segment", narrativeTypes.excerpt, resolveReference);+ const media = reference(element, "media", mediaTypes.synchronized, resolveReference);+ const excerpt = authoredExcerpt(segment);+ const hasNoTokens = excerpt !== undefined && excerpt.tokenStart === excerpt.tokenEndExclusive;+ exactAttributes(element, hasNoTokens+ ? ["id", "narrative", "segment", "media"]+ : ["id", "narrative", "segment", "media", "language"]);if (element.children.some((child) => child.kind === "element" || child.value.trim())) {throw new Error(`${element.name} does not accept children`);}const id = stringAttribute(element, "id");- const narrative = reference(element, "narrative", narrativeTypes.narrative, resolveReference);- const segment = reference(element, "segment", narrativeTypes.excerpt, resolveReference);- const media = reference(element, "media", mediaTypes.synchronized, resolveReference);+ if (hasNoTokens) {+ return {+ records: [],+ components: [{+ id,+ fragment: whisperXBoundarySemanticTakeFragment.id,+ inputs: { narrative: narrative.ref, segment: segment.ref, media: media.ref },+ outputs: { take: `${id}.take` },+ range: element.range,+ }],+ fragments: [whisperXBoundarySemanticTakeFragment],+ };+ }const language = stringAttribute(element, "language");if (language !== "en" && language !== "zh" && language !== "es") {throw new Error(`${element.name}.language must be en, zh, or es`);diff --git a/packages/whisperx/test/whisperx.test.ts b/packages/whisperx/test/whisperx.test.tsindex 7d18f737..077f2e6a 100644--- a/packages/whisperx/test/whisperx.test.ts+++ b/packages/whisperx/test/whisperx.test.ts@@ -1,10 +1,16 @@-import { sealSpeechEvidenceAudio } from "@hypit/speech";+import { mediaTypes, sealSynchronizedMedia } from "@hypit/media";+import { parseStructuredElement } from "@hypit/markup";+import type { SurfaceResolvedReference } from "@hypit/markup";+import { narrativeTypes } from "@hypit/narrative";+import type { Narrative } from "@hypit/narrative";+import { sealSpeechEvidenceAudio, speechProducers } from "@hypit/speech";import type { SpeechEvidenceAudio } from "@hypit/speech";import assert from "node:assert/strict";import test from "node:test";import { fixtureResource } from "../../../test/fixture-resource.js";import {+ decodeWhisperXSemanticTakeSurface,whisperXRequestForEvidenceAudio,} from "@hypit/whisperx";@@ -28,3 +34,63 @@ test("WhisperX receives normalized bytes without authored Segment truth", () =>assert.equal(request.sampleFrames, 16_000);assert.equal(request.language, "es");});++test("the real-media Surface materializes an empty Segment from its media domain", async () => {+ const narrative: Narrative = {+ id: "wordless-real",+ segments: [{+ id: "pause",+ startAnchorId: "pause:start",+ endAnchorId: "pause:end",+ tokenStart: 0,+ tokenEndExclusive: 0,+ }],+ tokens: [], turns: [], selections: [], moments: [],+ semanticIndex: { anchors: [+ { id: "pause:start", kind: "segment-start", segmentId: "pause" },+ { id: "pause:end", kind: "segment-end", segmentId: "pause" },+ ] },+ };+ const excerpt = {+ narrativeId: narrative.id,+ kind: "segment" as const,+ id: "pause",+ tokenStart: 0,+ tokenEndExclusive: 0,+ };+ const media = sealSynchronizedMedia({+ timeline: { frameRate: { numerator: 30, denominator: 1 }, frameCount: 90 },+ visual: {+ artifact: { kind: "blob", resource: fixtureResource("whisperx:wordless"), size: 1, mediaType: "video/mp4" },+ width: 1_080,+ height: 1_920,+ },+ });+ const authored = (+ path: string,+ type: SurfaceResolvedReference["type"],+ value: unknown,+ ): SurfaceResolvedReference => ({+ path,+ ref: { kind: "record", id: path },+ type,+ record: { id: path, type, value: { kind: "inline", value: value as never } },+ });+ const refs = new Map<string, SurfaceResolvedReference>([+ ["story", authored("story", narrativeTypes.narrative, narrative)],+ ["story.segment.pause", authored("story.segment.pause", narrativeTypes.excerpt, excerpt)],+ ["pause-media.media", authored("pause-media.media", mediaTypes.synchronized, media)],+ ]);+ const output = await decodeWhisperXSemanticTakeSurface({+ sourceName: "real.svml",+ element: parseStructuredElement({ name: "real.svml", text:+ '<whisperx:SemanticTake id="pause" narrative={story} segment={story.segment.pause} media={pause-media.media}/>',+ }, 0).element,+ resolveReference: (path) => refs.get(path),+ resolveAsset: () => { throw new Error("No assets are resolved by this test."); },+ });++ assert.equal(output.records.length, 0);+ assert.equal(output.fragments[0]?.operations[0]?.producer.name, speechProducers.materializeSegmentBoundaries.name);+ assert.deepEqual(Object.keys(output.components[0]?.inputs ?? {}).sort(), ["media", "narrative", "segment"]);+});diff --git a/skills/hypit/references/production/media.md b/skills/hypit/references/production/media.mdindex 02fc1d7d..9bac3392 100644--- a/skills/hypit/references/production/media.md+++ b/skills/hypit/references/production/media.md@@ -89,15 +89,30 @@ performed audio from its Script and the person's Voice Reference. Normalize thatThe resulting SemanticTake publishes semantic time and sound through Speech Track without inventinga visual performance; the work's Media, Typography or MG Tracks supply the picture.-## Wordless passages use media boundaries+## Empty Segments use their media boundaries-An ordinary Script Segment such as `<empty></empty>` can carry a wordless passage. Its media gives-it a duration, so the resulting SemanticTake has the Segment's start/end anchors and no timed words.-It enters the same Speech Track assembly as a spoken Take.+An ordinary Script Segment such as `<empty></empty>` can carry a passage without words. Its prepared+media gives it a duration, so the resulting SemanticTake has the Segment's start/end Anchors and no+timed Tokens. It enters the same Speech Track assembly as a spoken Take.-Use `estimate:SemanticTake` with that empty Segment and prepared media. With no spoken Tokens to-locate, it maps the Segment start to frame zero and the Segment end to the media's frame count.-Action, music or visual rhythm can determine the chosen passage length.+Use the Surface that matches the evidence you have. For real media, the real semantic Surface maps+the empty Segment directly to its media domain:++```svml+<whisperx:SemanticTake id="pause" narrative={story}+ segment={story.segment.pause} media={pause-media.media}/>+```++For preview media, the prediction Surface performs the same boundary materialization:++```svml+<estimate:SemanticTake id="pause-preview" narrative={story}+ segment={story.segment.pause} media={pause-preview-media.media}/>+```++Neither branch calls WhisperX or predicts word windows because the Segment has no Tokens. Action,+music or visual rhythm can determine the media duration. Both publish the ordinary SemanticTake+consumed by Speech Track; there is no separate wordless downstream type.## Give a still a duration when that is its role