Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

fix(speech): separate real and preview timing evidence

kashorin6007abb3ccf7
diff --git a/packages/semantic-take-estimate/README.md b/packages/semantic-take-estimate/README.md
index 354ddc41..f12a95a4 100644
--- a/packages/semantic-take-estimate/README.md
+++ b/packages/semantic-take-estimate/README.md
@@ -1,20 +1,18 @@
# `@hypit/semantic-take-estimate`
-An explicit, provider-free alternative to measured speech alignment for preview Builds.
+An explicit, provider-free semantic step for preview Builds.
```text
-Narrative + Segment + normalized SynchronizedMedia + SpeechEstimatePolicy
+Narrative + spoken Segment + normalized SynchronizedMedia + SpeechEstimatePolicy
-> syllable-weighted Token windows with real gaps
-> SemanticTake
```
-The package consumes an existing normalized media value. It knows nothing about how the video was
-generated and does not run FFmpeg, OpenCV, WhisperX or any other external process. It preserves the
-Script's exact Segment, Token and Anchor identities and estimates only their local frame positions.
-The normalized media's complete frame count is the allocation budget: after short edge and word
-gaps, every remaining frame is distributed across Tokens by pronunciation-unit weight. A longer
-normalized video therefore stretches the whole Script across that longer interval instead of
-leaving a fixed-length estimate at the beginning.
+The package consumes existing normalized media. It knows nothing about how that media was made and
+does not run an acoustic model. For a spoken Segment it preserves the Script's exact Segment, Token
+and Anchor identities and predicts only their local frame positions. The media's complete frame
+count is the allocation budget: after short edge and word gaps, every remaining frame is
+distributed across Tokens by pronunciation-unit weight.
```svml
<estimated:SemanticTake id="opening-estimated" narrative={story}
@@ -22,9 +20,19 @@ leaving a fixed-length estimate at the beginning.
language="en" pace="normal" rounding="round"/>
```
-The delivery policy is the element's own: write it inline as above, or name an SVS Recipe with
-`policy={recipes.speech.normal}` carrying the same properties. Nothing else in the graph computes
-it; durations themselves are author literals, measured beforehand with `hypit measure`.
+For a spoken Segment, write the delivery policy inline as above or name an SVS Recipe with
+`policy={recipes.speech.normal}`. `hypit measure` helps the author choose the media duration and
+balance delivery across spoken Segments; the prediction then distributes that known frame domain
+across the authored words.
+
+An empty Segment already has all of the semantic timing it can carry: its start and end. Omit the
+policy and the Surface maps those two authored Anchors to the prepared media boundaries without a
+prediction step:
+
+```svml
+<estimated:SemanticTake id="pause-preview" narrative={story}
+ segment={story.segment.pause} media={pause-media.media}/>
+```
The result has the ordinary `@hypit/speech@1#SemanticTake` type. A Run can therefore choose this
Build output while real A-roll is unavailable, then later choose the corresponding WhisperX output
@@ -53,9 +61,9 @@ The package also registers `@hypit/semantic-take-estimate@1#semantic-take` as a
| `media` | `@hypit/media@1#SynchronizedMedia` |
| `policy` | `@hypit/estimate@1#SpeechEstimatePolicy` |
-Its export is `take`, an ordinary SemanticTake. The policy is a typed value with `language`,
-`pace` or `rate`, `rounding` and optional `paddingSec`.
-The Author Surface publishes its authored policy as `<id>.policy`. For an Author entry containing
+Its export is `take`, an ordinary SemanticTake. For a spoken Segment, the policy is a typed value
+with `language`, `pace` or `rate`, `rounding` and optional `paddingSec`; the Author Surface publishes
+it as `<id>.policy`. For an Author entry containing
the `opening-estimated` declaration above, a Run can use:
```svrun
@@ -69,7 +77,7 @@ the `opening-estimated` declaration above, a Run can use:
<satisfy output="opening-semantic.take" candidate="timing.take"/>
```
-This selects estimated timing for the measured Take while retaining the required prepared-media
-dependency and any Run choice for its source bytes. The estimate allocates the media's complete
-frame count; it neither generates nor normalizes that media. For a wordless Segment it maps the
-Segment's two boundary Anchors directly to the media's first and final frame.
+This selects predicted timing while retaining the required prepared-media dependency and any Run
+choice for its source bytes. The estimate allocates the media's complete frame count; it neither
+generates nor normalizes that media. An empty Segment instead uses the Surface's boundary branch
+shown above and has no policy Fragment to select.
diff --git a/packages/semantic-take-estimate/src/fragment.ts b/packages/semantic-take-estimate/src/fragment.ts
index 5d13955c..dc1faea0 100644
--- a/packages/semantic-take-estimate/src/fragment.ts
+++ b/packages/semantic-take-estimate/src/fragment.ts
@@ -2,7 +2,7 @@ import { estimateTypes } from "@hypit/estimate";
import { sealGraphFragment } from "@hypit/elaborator";
import { mediaTypes } from "@hypit/media";
import { narrativeTypes } from "@hypit/narrative";
-import { speechTypes } from "@hypit/speech";
+import { speechProducers, speechTypes } from "@hypit/speech";
import { semanticTakeEstimateProducers } from "./manifest.js";
@@ -29,3 +29,23 @@ export const semanticTakeEstimateFragment = sealGraphFragment({
}],
exports: [{ name: "take", type: speechTypes.semanticTake, root: operation("materialize") }],
});
+
+/** The zero-Token branch of the same preview Surface; media boundaries need no prediction policy. */
+export const semanticTakeEstimateBoundaryFragment = sealGraphFragment({
+ inputs: [
+ { name: "narrative", type: narrativeTypes.narrative },
+ { name: "segment", type: narrativeTypes.excerpt },
+ { name: "media", type: mediaTypes.synchronized },
+ ],
+ operations: [{
+ id: "materialize-boundaries",
+ producer: speechProducers.materializeSegmentBoundaries,
+ inputs: {
+ narrative: input("narrative"),
+ segment: input("segment"),
+ media: input("media"),
+ },
+ result: { kind: "output", name: "take" },
+ }],
+ exports: [{ name: "take", type: speechTypes.semanticTake, root: operation("materialize-boundaries") }],
+});
diff --git a/packages/semantic-take-estimate/src/index.ts b/packages/semantic-take-estimate/src/index.ts
index fc100ee5..b2dda6ce 100644
--- a/packages/semantic-take-estimate/src/index.ts
+++ b/packages/semantic-take-estimate/src/index.ts
@@ -1,5 +1,5 @@
export { semanticTakeEstimateComponent } from "./component.js";
-export { semanticTakeEstimateFragment } from "./fragment.js";
+export { semanticTakeEstimateBoundaryFragment, semanticTakeEstimateFragment } from "./fragment.js";
export {
semanticTakeEstimateManifest,
semanticTakeEstimateMarkupSurfaces,
diff --git a/packages/semantic-take-estimate/src/manifest.ts b/packages/semantic-take-estimate/src/manifest.ts
index cfce1421..54a19812 100644
--- a/packages/semantic-take-estimate/src/manifest.ts
+++ b/packages/semantic-take-estimate/src/manifest.ts
@@ -18,18 +18,18 @@ export const semanticTakeEstimateMarkupSurfaces = [{
outputs: [estimateTypes.speechPolicy, speechTypes.semanticTake],
vocabulary: {
summary:
- "Projects estimated word windows or wordless Segment boundaries onto normalized media and publishes a SemanticTake.",
+ "Publishes a preview SemanticTake: predicted word windows for speech, or exact prepared-media boundaries for an empty Segment.",
attributes: [
{ name: "id", kind: "identifier", required: true,
summary: "Names this estimated alignment and the SemanticTake it publishes." },
{ name: "narrative", kind: "reference", required: true, accepts: [narrativeTypes.narrative],
summary: "Selects the authored Narrative that owns every Token and Anchor identity." },
{ name: "segment", kind: "reference", required: true, accepts: [narrativeTypes.excerpt],
- summary: "Selects the single authored Segment represented by the normalized video." },
+ summary: "Selects the single authored Segment represented by the prepared media." },
{ name: "media", kind: "reference", required: true, accepts: [mediaTypes.synchronized],
summary: "Selects the already normalized SynchronizedMedia whose exact frame domain receives the estimate." },
{ name: "policy", kind: "reference", required: false, accepts: [svsRecipeType],
- summary: "Selects an SVS Recipe carrying the delivery policy (language, pace or rate, rounding), in place of the inline attributes.",
+ summary: "For a Segment with Tokens, selects an SVS Recipe carrying the delivery policy in place of inline attributes.",
recipe: [
{ name: "language", required: true, summary: "Selects the counting rules the Segment is read with, or detects them.", values: ["auto", "en", "zh", "ja", "es"] },
{ name: "pace", required: false, summary: "Selects a named delivery density.", values: ["slow", "normal", "fast"] },
@@ -37,7 +37,7 @@ export const semanticTakeEstimateMarkupSurfaces = [{
{ name: "rounding", required: true, summary: "Selects how an estimated duration is rounded.", values: ["none", "round", "ceil"] },
] },
{ name: "language", kind: "literal", required: false, values: ["auto", "en", "zh", "ja", "es"],
- summary: "Selects the counting rules the Segment is read with, or detects them from its Text." },
+ summary: "For a Segment with Tokens, selects the counting rules or detects them from its Text." },
{ name: "pace", kind: "literal", required: false, values: ["slow", "normal", "fast"],
summary: "Selects a named delivery density for the chosen language." },
{ name: "rate", kind: "literal", required: false,
@@ -49,13 +49,14 @@ export const semanticTakeEstimateMarkupSurfaces = [{
{ name: "take", type: speechTypes.semanticTake,
summary: "The estimated SemanticTake, addressed as `<id>.take`." },
{ name: "policy", type: estimateTypes.speechPolicy,
- summary: "The sealed delivery policy this element weights Tokens with, addressed as `<id>.policy`." },
+ summary: "For a Segment with Tokens, the sealed delivery policy addressed as `<id>.policy`." },
],
example: `<estimated:SemanticTake id="opening-estimated" narrative={story}
segment={story.segment.opening} media={opening-media.media}
language="en" pace="normal" rounding="round"/>`,
notes: [
- "The policy is the element's own: write it inline or name an SVS Recipe with policy; never both.",
+ "A Segment with Tokens states its policy inline or names one SVS Recipe with policy.",
+ "An empty Segment uses only id, narrative, segment and media; its prepared-media boundaries already contain all required timing.",
"This package does not measure audio and never claims WhisperX or other acoustic evidence.",
"Every spoken Token uses its original Script identity. A wordless Segment keeps its two authored boundary Anchors.",
"Pronunciation-unit weights determine relative word lengths; adjacent words and both media edges retain non-zero frame gaps.",
diff --git a/packages/semantic-take-estimate/src/program.ts b/packages/semantic-take-estimate/src/program.ts
index 36e3d0ad..7c5d0474 100644
--- a/packages/semantic-take-estimate/src/program.ts
+++ b/packages/semantic-take-estimate/src/program.ts
@@ -7,7 +7,7 @@ import type { SpeechEstimatePolicy } from "@hypit/estimate";
import { verifySynchronizedMedia } from "@hypit/media";
import type { SynchronizedMedia } from "@hypit/media";
import type { Narrative, NarrativeExcerpt } from "@hypit/narrative";
-import { materializeSemanticTake } from "@hypit/speech";
+import { materializeSegmentBoundaryTake, materializeSemanticTake } from "@hypit/speech";
import type { SemanticTake, SemanticTakeTimedToken, SemanticTakeTiming } from "@hypit/speech";
const EDGE_GAP_SECONDS = 0.12;
@@ -48,21 +48,11 @@ export function estimateSemanticTakeTiming(
verifySynchronizedMedia(media);
assertSpeechEstimatePolicy(policy);
const tokens = segmentTokens(narrative, excerpt);
- const segment = narrative.segments.find((candidate) => candidate.id === excerpt.id)!;
if (tokens.length === 0) {
- const frameByAnchor = new Map<string, number>([
- [segment.startAnchorId, 0],
- [segment.endAnchorId, media.timeline.frameCount],
- ]);
- const anchors = narrative.semanticIndex.anchors
- .filter((anchor) => anchor.segmentId === excerpt.id)
- .map((anchor) => {
- const frame = frameByAnchor.get(anchor.id);
- if (frame === undefined) throw new Error(`Estimated timing cannot locate authored Anchor ${anchor.id}.`);
- return { identity: anchor.id, frame };
- });
- return { tokens: [], anchors };
+ const take = materializeSegmentBoundaryTake(narrative, excerpt, media);
+ return { tokens: take.tokens, anchors: take.anchors };
}
+ const segment = narrative.segments.find((candidate) => candidate.id === excerpt.id)!;
const rate = media.timeline.frameRate.numerator / media.timeline.frameRate.denominator;
const edgeGap = Math.max(1, Math.round(EDGE_GAP_SECONDS * rate));
const wordGap = Math.max(1, Math.round(WORD_GAP_SECONDS * rate));
diff --git a/packages/semantic-take-estimate/src/surface.ts b/packages/semantic-take-estimate/src/surface.ts
index 7e764e28..c4bf0180 100644
--- a/packages/semantic-take-estimate/src/surface.ts
+++ b/packages/semantic-take-estimate/src/surface.ts
@@ -9,12 +9,13 @@ import type {
} from "@hypit/markup";
import { mediaTypes } from "@hypit/media";
import { narrativeTypes } from "@hypit/narrative";
+import type { NarrativeExcerpt } from "@hypit/narrative";
import type { CanonicalValue } from "@hypit/protocol";
import { sameType } from "@hypit/protocol";
import { svsRecipeType } from "@hypit/svs";
import type { SvsRecipe } from "@hypit/svs";
-import { semanticTakeEstimateFragment } from "./fragment.js";
+import { semanticTakeEstimateBoundaryFragment, semanticTakeEstimateFragment } from "./fragment.js";
function reference(
element: StructuredElement,
@@ -32,6 +33,12 @@ function reference(
return resolved;
}
+function authoredExcerpt(value: SurfaceResolvedReference): NarrativeExcerpt | undefined {
+ return value.record?.value.kind === "inline"
+ ? value.record.value.value as unknown as NarrativeExcerpt
+ : undefined;
+}
+
/**
* The delivery policy the estimate weights Tokens with: an SVS Recipe named by `policy`, or the same
* properties written inline. The element owns its policy; nothing elsewhere in the graph computes it.
@@ -54,17 +61,37 @@ function policyOf(
export const decodeSemanticTakeEstimateSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
const required = ["id", "narrative", "segment", "media"];
- const allowed = new Set([...required, "policy", ...speechEstimatePolicyProperties]);
const missing = required.filter((name) => element.attributes[name] === undefined);
+ if (missing.length > 0) {
+ throw new Error(`${element.name} requires ${required.join(", ")}`);
+ }
+ const narrative = reference(element, "narrative", narrativeTypes.narrative, resolveReference);
+ const segment = reference(element, "segment", narrativeTypes.excerpt, resolveReference);
+ const media = reference(element, "media", mediaTypes.synchronized, resolveReference);
+ const excerpt = authoredExcerpt(segment);
+ const hasNoTokens = excerpt !== undefined && excerpt.tokenStart === excerpt.tokenEndExclusive;
+ const allowed = new Set(hasNoTokens ? required : [...required, "policy", ...speechEstimatePolicyProperties]);
const unknown = Object.keys(element.attributes).filter((name) => !allowed.has(name));
- if (missing.length > 0 || unknown.length > 0) {
- throw new Error(`${element.name} requires ${required.join(", ")} and a policy (policy={recipe} or ${speechEstimatePolicyProperties.join(", ")})${
- unknown.length === 0 ? "" : `; unknown ${unknown.join(", ")}`}`);
+ if (unknown.length > 0) {
+ throw new Error(`${element.name} has unknown or inapplicable attributes ${unknown.join(", ")}`);
}
if (element.children.some((child) => child.kind === "element" || child.value.trim().length > 0)) {
throw new Error(`${element.name} must be empty`);
}
const id = textAttribute(element, "id");
+ if (hasNoTokens) {
+ return {
+ records: [],
+ components: [{
+ id,
+ fragment: semanticTakeEstimateBoundaryFragment.id,
+ inputs: { narrative: narrative.ref, segment: segment.ref, media: media.ref },
+ outputs: { take: `${id}.take` },
+ range: element.range,
+ }],
+ fragments: [semanticTakeEstimateBoundaryFragment],
+ };
+ }
const policyId = `${id}.policy`;
return {
records: [{
@@ -77,9 +104,9 @@ export const decodeSemanticTakeEstimateSurface: StructuredSurfaceHandler = ({ el
id,
fragment: semanticTakeEstimateFragment.id,
inputs: {
- narrative: reference(element, "narrative", narrativeTypes.narrative, resolveReference).ref,
- segment: reference(element, "segment", narrativeTypes.excerpt, resolveReference).ref,
- media: reference(element, "media", mediaTypes.synchronized, resolveReference).ref,
+ narrative: narrative.ref,
+ segment: segment.ref,
+ media: media.ref,
policy: { kind: "record", id: policyId },
},
outputs: { take: `${id}.take` },
diff --git a/packages/semantic-take-estimate/test/semantic-take-estimate.test.ts b/packages/semantic-take-estimate/test/semantic-take-estimate.test.ts
index d72a9f70..8c722d95 100644
--- a/packages/semantic-take-estimate/test/semantic-take-estimate.test.ts
+++ b/packages/semantic-take-estimate/test/semantic-take-estimate.test.ts
@@ -2,16 +2,21 @@ import assert from "node:assert/strict";
import test from "node:test";
import { sealSpeechEstimatePolicy } from "@hypit/estimate";
-import { sealSynchronizedMedia } from "@hypit/media";
+import { mediaTypes, sealSynchronizedMedia } from "@hypit/media";
+import { parseStructuredElement } from "@hypit/markup";
+import type { SurfaceResolvedReference } from "@hypit/markup";
import {
assertCaptionDocumentIdentity,
assertNarrativeExcerptIdentity,
assertNarrativeIdentity,
+ narrativeTypes,
} from "@hypit/narrative";
import type { Narrative } from "@hypit/narrative";
+import { materializeSegmentBoundaryTake, speechProducers } from "@hypit/speech";
import { fixtureResource } from "../../../test/fixture-resource.js";
import {
+ decodeSemanticTakeEstimateSurface,
estimateSemanticTakeTiming,
materializeEstimatedSemanticTake,
} from "@hypit/semantic-take-estimate";
@@ -150,7 +155,7 @@ test("a wordless Segment receives the prepared media boundaries without an audio
cueBreaks: [],
});
- const take = materializeEstimatedSemanticTake(wordless, wordlessExcerpt, media, policy);
+ const take = materializeSegmentBoundaryTake(wordless, wordlessExcerpt, media);
assert.deepEqual(take.tokens, []);
assert.deepEqual(take.anchors, [
{ identity: "pause:start", frame: 0 },
@@ -158,3 +163,59 @@ test("a wordless Segment receives the prepared media boundaries without an audio
]);
assert.equal(take.media.audio, undefined);
});
+
+test("the preview Surface sends an empty Segment straight to boundary materialization", async () => {
+ const wordless: Narrative = {
+ id: "wordless-preview",
+ segments: [{
+ id: "pause",
+ startAnchorId: "pause:start",
+ endAnchorId: "pause:end",
+ tokenStart: 0,
+ tokenEndExclusive: 0,
+ }],
+ tokens: [], turns: [], selections: [], moments: [],
+ semanticIndex: { anchors: [
+ { id: "pause:start", kind: "segment-start", segmentId: "pause" },
+ { id: "pause:end", kind: "segment-end", segmentId: "pause" },
+ ] },
+ };
+ const wordlessExcerpt = {
+ narrativeId: wordless.id,
+ kind: "segment" as const,
+ id: "pause",
+ tokenStart: 0,
+ tokenEndExclusive: 0,
+ };
+ const authored = (
+ path: string,
+ type: SurfaceResolvedReference["type"],
+ value: unknown,
+ ): SurfaceResolvedReference => ({
+ path,
+ ref: { kind: "record", id: path },
+ type,
+ record: { id: path, type, value: { kind: "inline", value: value as never } },
+ });
+ const refs = new Map<string, SurfaceResolvedReference>([
+ ["story", authored("story", narrativeTypes.narrative, wordless)],
+ ["story.segment.pause", authored("story.segment.pause", narrativeTypes.excerpt, wordlessExcerpt)],
+ ["pause-media.media", authored("pause-media.media", mediaTypes.synchronized, media)],
+ ]);
+ const output = await decodeSemanticTakeEstimateSurface({
+ sourceName: "preview.svml",
+ element: parseStructuredElement({ name: "preview.svml", text:
+ '<estimate:SemanticTake id="pause-preview" narrative={story} segment={story.segment.pause} media={pause-media.media}/>',
+ }, 0).element,
+ resolveReference: (path) => refs.get(path),
+ resolveAsset: () => { throw new Error("No assets are resolved by this test."); },
+ });
+
+ assert.equal(output.records.length, 0);
+ assert.equal(output.fragments[0]?.operations[0]?.producer.name, speechProducers.materializeSegmentBoundaries.name);
+ assert.deepEqual(output.components[0]?.inputs, {
+ narrative: { kind: "record", id: "story" },
+ segment: { kind: "record", id: "story.segment.pause" },
+ media: { kind: "record", id: "pause-media.media" },
+ });
+});
diff --git a/packages/speech/README.md b/packages/speech/README.md
index c5fe4d8b..7e8f6c48 100644
--- a/packages/speech/README.md
+++ b/packages/speech/README.md
@@ -9,3 +9,8 @@ A `SemanticTake` contains one normalized media product, one authored Segment, it
frame anchors. Ordered Takes are assembled by `@hypit/speech-track` into a `SemanticTrack`, which is
the program's continuous semantic skeleton. `SpeechEvidenceAudio` carries canonical 16 kHz WAV bytes
and their exact sample count; it never carries Script or Segment identity.
+
+The package also owns deterministic boundary materialization for a Segment with no Tokens. It maps
+the Segment start and end to the prepared media's frame domain and publishes the same
+`SemanticTake` type. The preview and real-media author surfaces both select this operation for an
+empty Segment. There are no word windows to predict or acoustic evidence to request.
diff --git a/packages/speech/src/component.ts b/packages/speech/src/component.ts
index ccfa13bf..370382c8 100644
--- a/packages/speech/src/component.ts
+++ b/packages/speech/src/component.ts
@@ -1,4 +1,7 @@
import type { ComponentPackage } from "@hypit/component-kit";
+import type { SynchronizedMedia } from "@hypit/media";
+import type { Narrative, NarrativeExcerpt } from "@hypit/narrative";
+import { canonicalize } from "@hypit/protocol";
import type { StoredValue } from "@hypit/protocol";
import {
@@ -6,7 +9,8 @@ import {
assertSpeechDurationIdentity,
assertSpeechEvidenceAudioIdentity,
} from "./identity.js";
-import { speechTypes } from "./manifest.js";
+import { materializeSegmentBoundaryTake } from "./materialize.js";
+import { speechProducers, speechTypes } from "./manifest.js";
import type { SemanticTake, SpeechDuration, SpeechEvidenceAudio } from "./types.js";
function inline<T>(value: StoredValue, subject: string): T {
@@ -20,4 +24,20 @@ export const speechComponent = {
{ type: speechTypes.evidenceAudio, handler: ({ value }) => assertSpeechEvidenceAudioIdentity(inline<SpeechEvidenceAudio>(value, "SpeechEvidenceAudio")) },
{ type: speechTypes.semanticTake, handler: ({ value }) => assertSemanticTakeIdentity(inline<SemanticTake>(value, "SemanticTake")) },
],
+ producers: [{
+ producer: speechProducers.materializeSegmentBoundaries,
+ handler: ({ inputs }) => ({
+ outputs: {
+ take: {
+ kind: "inline",
+ value: canonicalize(materializeSegmentBoundaryTake(
+ inline<Narrative>(inputs.narrative!.value, "Narrative"),
+ inline<NarrativeExcerpt>(inputs.segment!.value, "NarrativeExcerpt"),
+ inline<SynchronizedMedia>(inputs.media!.value, "SynchronizedMedia"),
+ )),
+ },
+ },
+ needs: {},
+ }),
+ }],
} satisfies ComponentPackage;
diff --git a/packages/speech/src/index.ts b/packages/speech/src/index.ts
index aead4563..ca413bfd 100644
--- a/packages/speech/src/index.ts
+++ b/packages/speech/src/index.ts
@@ -1,6 +1,6 @@
export { assertSemanticTakeIdentity, assertSpeechDurationIdentity, assertSpeechEvidenceAudioIdentity, sealSemanticTake, sealSpeechDuration, sealSpeechEvidenceAudio, speechEvidenceSampleBoundary } from "./identity.js";
export { speechComponent } from "./component.js";
-export { materializeSemanticTake } from "./materialize.js";
-export { speechDependency, speechManifest, speechModuleRef, speechTypes } from "./manifest.js";
+export { materializeSegmentBoundaryTake, materializeSemanticTake } from "./materialize.js";
+export { speechDependency, speechManifest, speechModuleRef, speechProducers, speechTypes } from "./manifest.js";
export { semanticTakeSchema, speechDurationSchema, speechEvidenceAudioSchema } from "./schema.js";
export type * from "./types.js";
diff --git a/packages/speech/src/manifest.ts b/packages/speech/src/manifest.ts
index 07c338b3..ae17b07b 100644
--- a/packages/speech/src/manifest.ts
+++ b/packages/speech/src/manifest.ts
@@ -1,5 +1,6 @@
-import { mediaDependency } from "@hypit/media";
-import type { ModuleManifest, TypeRef } from "@hypit/protocol";
+import { mediaDependency, mediaTypes } from "@hypit/media";
+import { narrativeDependency, narrativeTypes } from "@hypit/narrative";
+import type { ModuleManifest, ProducerRef, TypeRef } from "@hypit/protocol";
import { semanticTakeSchema, speechDurationSchema, speechEvidenceAudioSchema } from "./schema.js";
export const speechModuleRef = { name: "@hypit/speech", version: "1" } as const;
export const speechTypes = {
@@ -7,10 +8,23 @@ export const speechTypes = {
evidenceAudio: { module: speechModuleRef, name: "SpeechEvidenceAudio" },
semanticTake: { module: speechModuleRef, name: "SemanticTake" },
} satisfies Record<string, TypeRef>;
+export const speechProducers = {
+ materializeSegmentBoundaries: { module: speechModuleRef, name: "materialize-segment-boundaries" },
+} satisfies Record<string, ProducerRef>;
export const speechManifest: ModuleManifest = { format: "hypit.module@1", name: speechModuleRef.name, version: speechModuleRef.version,
- dependencies: [mediaDependency],
+ dependencies: [mediaDependency, narrativeDependency],
types: [{ name: speechTypes.duration.name },
{ name: speechTypes.evidenceAudio.name },
{ name: speechTypes.semanticTake.name }],
- capabilities: [], producers: [] };
+ capabilities: [],
+ producers: [{
+ name: speechProducers.materializeSegmentBoundaries.name,
+ inputs: [
+ { name: "narrative", type: narrativeTypes.narrative },
+ { name: "segment", type: narrativeTypes.excerpt },
+ { name: "media", type: mediaTypes.synchronized },
+ ],
+ outputs: [{ name: "take", type: speechTypes.semanticTake }],
+ needs: [],
+ }] };
export const speechDependency = { module: speechModuleRef } as const;
diff --git a/packages/speech/src/materialize.ts b/packages/speech/src/materialize.ts
index afb3b834..2ee726c9 100644
--- a/packages/speech/src/materialize.ts
+++ b/packages/speech/src/materialize.ts
@@ -82,3 +82,22 @@ export function materializeSemanticTake(
assertSemanticTakeIdentity(take);
return take;
}
+
+/** Materialize the exact media boundaries of an authored Segment that contains no Tokens. */
+export function materializeSegmentBoundaryTake(
+ narrative: Narrative,
+ excerpt: NarrativeExcerpt,
+ media: SynchronizedMedia,
+): SemanticTake {
+ const segment = authoredSegment(narrative, excerpt);
+ if (segment.tokenStart !== segment.tokenEndExclusive) {
+ throw new Error(`Segment ${segment.id} contains spoken Tokens and needs token timing.`);
+ }
+ return materializeSemanticTake(narrative, excerpt, media, {
+ tokens: [],
+ anchors: [
+ { identity: segment.startAnchorId, frame: 0 },
+ { identity: segment.endAnchorId, frame: media.timeline.frameCount },
+ ],
+ });
+}
diff --git a/packages/whisperx/README.md b/packages/whisperx/README.md
index 37621057..b1c2cbf5 100644
--- a/packages/whisperx/README.md
+++ b/packages/whisperx/README.md
@@ -9,12 +9,22 @@ The package contains no credentials, Python environment or queue. Providers tran
request directly into provider-neutral `AlignedTranscriptEvidence`. There is no vendor-shaped
Evidence wrapper or pass-through normalization node in the graph.
-`<whisperx:SemanticTake>` consumes one normalized `SynchronizedMedia`, exactly one Script Segment,
-and a required `language="en"` or `language="zh"`. The language is passed directly to WhisperX;
-Script text and audio are never inspected to choose it implicitly. `@hypit/media-pipeline` projects
-that Take's 48 kHz audio to canonical 16 kHz mono
-`SpeechEvidenceAudio`; WhisperX sees only those bytes. A deterministic local alignment then combines
-the returned evidence with the one Segment and emits one self-contained `SemanticTake`.
+`<whisperx:SemanticTake>` is the real-media semantic Surface. It consumes one normalized
+`SynchronizedMedia` and exactly one Script Segment. When that Segment contains Tokens, it also
+requires `language="en"`, `language="zh"` or `language="es"`. The language is passed directly to
+WhisperX; Script text and audio are not used to choose it implicitly. `@hypit/media-pipeline`
+projects the Take's audio to canonical 16 kHz mono `SpeechEvidenceAudio`; WhisperX sees only those
+bytes. A deterministic local alignment then combines the returned evidence with the Segment and
+emits one self-contained `SemanticTake`.
+
+When the authored Segment has no Tokens, write the same Surface without `language`. Its start and
+end Anchors map directly to the prepared media's first and final frame. There are no words to align,
+so this branch requests no evidence audio and no WhisperX capability:
+
+```svml
+<whisperx:SemanticTake id="pause" narrative={story}
+ segment={story.segment.pause} media={pause-media.media}/>
+```
There is no whole-program WhisperX pass. Speech Track only receives already-semantic Takes and later
translates their local frames when assembling the final ProgramSpace and complete semantic map.
diff --git a/packages/whisperx/src/fragment.ts b/packages/whisperx/src/fragment.ts
index 4ea41268..98bd424c 100644
--- a/packages/whisperx/src/fragment.ts
+++ b/packages/whisperx/src/fragment.ts
@@ -1,6 +1,6 @@
import { narrativeTypes } from "@hypit/narrative";
import { mediaTypes } from "@hypit/media";
-import { speechTypes } from "@hypit/speech";
+import { speechProducers, speechTypes } from "@hypit/speech";
import { sealGraphFragment } from "@hypit/elaborator";
import { mediaPipelineProducers } from "@hypit/media-pipeline";
import { speechAlignmentProducers } from "@hypit/speech-alignment";
@@ -54,3 +54,23 @@ export const whisperXSemanticTakeFragment = sealGraphFragment({
},
],
});
+
+/** The zero-Token branch of the same real-media Surface; media boundaries need no acoustic Need. */
+export const whisperXBoundarySemanticTakeFragment = sealGraphFragment({
+ inputs: [
+ { name: "narrative", type: narrativeTypes.narrative },
+ { name: "segment", type: narrativeTypes.excerpt },
+ { name: "media", type: mediaTypes.synchronized },
+ ],
+ operations: [{
+ id: "materialize-boundaries",
+ producer: speechProducers.materializeSegmentBoundaries,
+ inputs: {
+ narrative: input("narrative"),
+ segment: input("segment"),
+ media: input("media"),
+ },
+ result: { kind: "output", name: "take" },
+ }],
+ exports: [{ name: "take", type: speechTypes.semanticTake, root: operation("materialize-boundaries") }],
+});
diff --git a/packages/whisperx/src/index.ts b/packages/whisperx/src/index.ts
index a39f9cdf..e7478996 100644
--- a/packages/whisperx/src/index.ts
+++ b/packages/whisperx/src/index.ts
@@ -1,7 +1,7 @@
export {
whisperXComponent,
} from "./component.js";
-export { whisperXSemanticTakeFragment } from "./fragment.js";
+export { whisperXBoundarySemanticTakeFragment, whisperXSemanticTakeFragment } from "./fragment.js";
export {
assertWhisperXEvidenceWav,
verifyWhisperXAlignmentRequest,
diff --git a/packages/whisperx/src/manifest.ts b/packages/whisperx/src/manifest.ts
index 24defab0..445d988f 100644
--- a/packages/whisperx/src/manifest.ts
+++ b/packages/whisperx/src/manifest.ts
@@ -1,6 +1,6 @@
import { speechDependency, speechTypes } from "@hypit/speech";
import { mediaDependency, mediaTypes } from "@hypit/media";
-import { narrativeTypes } from "@hypit/narrative";
+import { narrativeDependency, narrativeTypes } from "@hypit/narrative";
import { speechEvidenceDependency, speechEvidenceTypes } from "@hypit/speech-evidence";
import type { CapabilityRef, ModuleManifest, ProducerRef, TypeRef } from "@hypit/protocol";
import { mediaPipelineManifest, mediaPipelineModuleRef } from "@hypit/media-pipeline";
@@ -24,7 +24,7 @@ export const whisperXMarkupSurfaces = [{
outputs: [whisperXTypes.language, speechTypes.semanticTake],
vocabulary: {
summary:
- "Measures one normalized Take with WhisperX and aligns one authored Segment into a self-contained SemanticTake.",
+ "Publishes a real-media SemanticTake: WhisperX-aligned words for speech, or exact prepared-media boundaries for an empty Segment.",
attributes: [
{ name: "id", kind: "identifier", required: true,
summary: "Names this alignment and prefixes the bindings it publishes." },
@@ -36,10 +36,10 @@ export const whisperXMarkupSurfaces = [{
summary: "Selects the single authored Segment performed by this Take." },
{ name: "media", kind: "reference", required: true,
accepts: [mediaTypes.synchronized],
- summary: "Selects the already normalized SynchronizedMedia measured by WhisperX." },
- { name: "language", kind: "literal", required: true,
+ summary: "Selects the already normalized SynchronizedMedia represented by this Segment." },
+ { name: "language", kind: "literal", required: false,
values: ["en", "zh", "es"],
- summary: "Explicitly selects the English, Chinese or Spanish WhisperX transcription and alignment models." },
+ summary: "For a Segment with Tokens, explicitly selects the English, Chinese or Spanish WhisperX models." },
],
ports: [
{ name: "take", type: speechTypes.semanticTake,
@@ -48,8 +48,9 @@ export const whisperXMarkupSurfaces = [{
example: `<whisperx:SemanticTake id="opening" narrative={story}
segment={story.segment.opening} media={opening-media.media} language="en"/>`,
notes: [
- "All five attributes are required; the element accepts no children and no text content.",
- "Language is never detected from Script text or audio; each alignment call states en, zh or es explicitly.",
+ "A Segment with Tokens states all five attributes and sends its prepared audio for alignment.",
+ "An empty Segment omits language and maps its authored start/end Anchors directly to the prepared-media boundaries.",
+ "For speech, language is never detected from Script text or audio; each alignment call states en, zh or es explicitly.",
"Importing this package is what selects the WhisperX model family; the Runtime separately binds the alignment Need to an Endpoint.",
],
},
@@ -64,6 +65,7 @@ export const whisperXManifest: ModuleManifest = {
speechDependency,
speechEvidenceDependency,
mediaDependency,
+ narrativeDependency,
{ module: mediaPipelineModuleRef },
{ module: speechAlignmentModuleRef },
],
diff --git a/packages/whisperx/src/surface.ts b/packages/whisperx/src/surface.ts
index 957a35e8..c5b9be7a 100644
--- a/packages/whisperx/src/surface.ts
+++ b/packages/whisperx/src/surface.ts
@@ -8,9 +8,10 @@ import {
} from "@hypit/markup";
import { sameType } from "@hypit/protocol";
import { narrativeTypes } from "@hypit/narrative";
+import type { NarrativeExcerpt } from "@hypit/narrative";
import { mediaTypes } from "@hypit/media";
-import { whisperXSemanticTakeFragment } from "./fragment.js";
+import { whisperXBoundarySemanticTakeFragment, whisperXSemanticTakeFragment } from "./fragment.js";
import { whisperXTypes } from "./manifest.js";
import type { WhisperXLanguage } from "./types.js";
@@ -28,15 +29,38 @@ function reference(
return value;
}
+function authoredExcerpt(value: SurfaceResolvedReference): NarrativeExcerpt | undefined {
+ return value.record?.value.kind === "inline"
+ ? value.record.value.value as unknown as NarrativeExcerpt
+ : undefined;
+}
+
export const decodeWhisperXSemanticTakeSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
- exactAttributes(element, ["id", "narrative", "segment", "media", "language"]);
+ const narrative = reference(element, "narrative", narrativeTypes.narrative, resolveReference);
+ const segment = reference(element, "segment", narrativeTypes.excerpt, resolveReference);
+ const media = reference(element, "media", mediaTypes.synchronized, resolveReference);
+ const excerpt = authoredExcerpt(segment);
+ const hasNoTokens = excerpt !== undefined && excerpt.tokenStart === excerpt.tokenEndExclusive;
+ exactAttributes(element, hasNoTokens
+ ? ["id", "narrative", "segment", "media"]
+ : ["id", "narrative", "segment", "media", "language"]);
if (element.children.some((child) => child.kind === "element" || child.value.trim())) {
throw new Error(`${element.name} does not accept children`);
}
const id = stringAttribute(element, "id");
- const narrative = reference(element, "narrative", narrativeTypes.narrative, resolveReference);
- const segment = reference(element, "segment", narrativeTypes.excerpt, resolveReference);
- const media = reference(element, "media", mediaTypes.synchronized, resolveReference);
+ if (hasNoTokens) {
+ return {
+ records: [],
+ components: [{
+ id,
+ fragment: whisperXBoundarySemanticTakeFragment.id,
+ inputs: { narrative: narrative.ref, segment: segment.ref, media: media.ref },
+ outputs: { take: `${id}.take` },
+ range: element.range,
+ }],
+ fragments: [whisperXBoundarySemanticTakeFragment],
+ };
+ }
const language = stringAttribute(element, "language");
if (language !== "en" && language !== "zh" && language !== "es") {
throw new Error(`${element.name}.language must be en, zh, or es`);
diff --git a/packages/whisperx/test/whisperx.test.ts b/packages/whisperx/test/whisperx.test.ts
index 7d18f737..077f2e6a 100644
--- a/packages/whisperx/test/whisperx.test.ts
+++ b/packages/whisperx/test/whisperx.test.ts
@@ -1,10 +1,16 @@
-import { sealSpeechEvidenceAudio } from "@hypit/speech";
+import { mediaTypes, sealSynchronizedMedia } from "@hypit/media";
+import { parseStructuredElement } from "@hypit/markup";
+import type { SurfaceResolvedReference } from "@hypit/markup";
+import { narrativeTypes } from "@hypit/narrative";
+import type { Narrative } from "@hypit/narrative";
+import { sealSpeechEvidenceAudio, speechProducers } from "@hypit/speech";
import type { SpeechEvidenceAudio } from "@hypit/speech";
import assert from "node:assert/strict";
import test from "node:test";
import { fixtureResource } from "../../../test/fixture-resource.js";
import {
+ decodeWhisperXSemanticTakeSurface,
whisperXRequestForEvidenceAudio,
} from "@hypit/whisperx";
@@ -28,3 +34,63 @@ test("WhisperX receives normalized bytes without authored Segment truth", () =>
assert.equal(request.sampleFrames, 16_000);
assert.equal(request.language, "es");
});
+
+test("the real-media Surface materializes an empty Segment from its media domain", async () => {
+ const narrative: Narrative = {
+ id: "wordless-real",
+ segments: [{
+ id: "pause",
+ startAnchorId: "pause:start",
+ endAnchorId: "pause:end",
+ tokenStart: 0,
+ tokenEndExclusive: 0,
+ }],
+ tokens: [], turns: [], selections: [], moments: [],
+ semanticIndex: { anchors: [
+ { id: "pause:start", kind: "segment-start", segmentId: "pause" },
+ { id: "pause:end", kind: "segment-end", segmentId: "pause" },
+ ] },
+ };
+ const excerpt = {
+ narrativeId: narrative.id,
+ kind: "segment" as const,
+ id: "pause",
+ tokenStart: 0,
+ tokenEndExclusive: 0,
+ };
+ const media = sealSynchronizedMedia({
+ timeline: { frameRate: { numerator: 30, denominator: 1 }, frameCount: 90 },
+ visual: {
+ artifact: { kind: "blob", resource: fixtureResource("whisperx:wordless"), size: 1, mediaType: "video/mp4" },
+ width: 1_080,
+ height: 1_920,
+ },
+ });
+ const authored = (
+ path: string,
+ type: SurfaceResolvedReference["type"],
+ value: unknown,
+ ): SurfaceResolvedReference => ({
+ path,
+ ref: { kind: "record", id: path },
+ type,
+ record: { id: path, type, value: { kind: "inline", value: value as never } },
+ });
+ const refs = new Map<string, SurfaceResolvedReference>([
+ ["story", authored("story", narrativeTypes.narrative, narrative)],
+ ["story.segment.pause", authored("story.segment.pause", narrativeTypes.excerpt, excerpt)],
+ ["pause-media.media", authored("pause-media.media", mediaTypes.synchronized, media)],
+ ]);
+ const output = await decodeWhisperXSemanticTakeSurface({
+ sourceName: "real.svml",
+ element: parseStructuredElement({ name: "real.svml", text:
+ '<whisperx:SemanticTake id="pause" narrative={story} segment={story.segment.pause} media={pause-media.media}/>',
+ }, 0).element,
+ resolveReference: (path) => refs.get(path),
+ resolveAsset: () => { throw new Error("No assets are resolved by this test."); },
+ });
+
+ assert.equal(output.records.length, 0);
+ assert.equal(output.fragments[0]?.operations[0]?.producer.name, speechProducers.materializeSegmentBoundaries.name);
+ assert.deepEqual(Object.keys(output.components[0]?.inputs ?? {}).sort(), ["media", "narrative", "segment"]);
+});
diff --git a/skills/hypit/references/production/media.md b/skills/hypit/references/production/media.md
index 02fc1d7d..9bac3392 100644
--- a/skills/hypit/references/production/media.md
+++ b/skills/hypit/references/production/media.md
@@ -89,15 +89,30 @@ performed audio from its Script and the person's Voice Reference. Normalize that
The resulting SemanticTake publishes semantic time and sound through Speech Track without inventing
a visual performance; the work's Media, Typography or MG Tracks supply the picture.
-## Wordless passages use media boundaries
+## Empty Segments use their media boundaries
-An ordinary Script Segment such as `<empty></empty>` can carry a wordless passage. Its media gives
-it a duration, so the resulting SemanticTake has the Segment's start/end anchors and no timed words.
-It enters the same Speech Track assembly as a spoken Take.
+An ordinary Script Segment such as `<empty></empty>` can carry a passage without words. Its prepared
+media gives it a duration, so the resulting SemanticTake has the Segment's start/end Anchors and no
+timed Tokens. It enters the same Speech Track assembly as a spoken Take.
-Use `estimate:SemanticTake` with that empty Segment and prepared media. With no spoken Tokens to
-locate, it maps the Segment start to frame zero and the Segment end to the media's frame count.
-Action, music or visual rhythm can determine the chosen passage length.
+Use the Surface that matches the evidence you have. For real media, the real semantic Surface maps
+the empty Segment directly to its media domain:
+
+```svml
+<whisperx:SemanticTake id="pause" narrative={story}
+ segment={story.segment.pause} media={pause-media.media}/>
+```
+
+For preview media, the prediction Surface performs the same boundary materialization:
+
+```svml
+<estimate:SemanticTake id="pause-preview" narrative={story}
+ segment={story.segment.pause} media={pause-preview-media.media}/>
+```
+
+Neither branch calls WhisperX or predicts word windows because the Segment has no Tokens. Action,
+music or visual rhythm can determine the media duration. Both publish the ordinary SemanticTake
+consumed by Speech Track; there is no separate wordless downstream type.
## Give a still a duration when that is its role