Skip to content

Code-first video & image, rendered natively on WebGPU.

Add signal-driven media time mapping (#58)

Okan Aslankane56c7f56f6fa
diff --git a/.changeset/signal-clock-context.md b/.changeset/signal-clock-context.md
new file mode 100644
index 00000000..cbd18f3f
--- /dev/null
+++ b/.changeset/signal-clock-context.md
@@ -0,0 +1,6 @@
+---
+"@framefields/core": patch
+"framefields": patch
+---
+
+Preserve the supplied frame progress when evaluating programmatic signals, and prevent cached frame-signal values from being reused across different frame rates or frame contexts.
diff --git a/.changeset/signal-time-remap.md b/.changeset/signal-time-remap.md
new file mode 100644
index 00000000..beb922e8
--- /dev/null
+++ b/.changeset/signal-time-remap.md
@@ -0,0 +1,7 @@
+---
+"@framefields/core": minor
+"@framefields/compositions": minor
+"framefields": minor
+---
+
+Drive media `timeRemap` from a signal: `{ kind: "signal", mode: "speed" | "source-time", signal }` accepts a signal instance or a named composition signal. Without frame hooks the signal is sampled up front; with hooks it is recorded live after each frame's hooks, replayed for seeks, audio, vision and the preview, and its source bounds are checked as it records.
diff --git a/nodes/node-compositor/SKILL.md b/nodes/node-compositor/SKILL.md
index 5f9e188d..d41825a5 100644
--- a/nodes/node-compositor/SKILL.md
+++ b/nodes/node-compositor/SKILL.md
@@ -134,7 +134,7 @@ Per-kind fields:
- **Charts**: there is no chart node kind. `Layer.chart` in the `framefields` SDK expands a chart into `box`, `shape` (`"path"`/`"circle"`) and `text` nodes, so a program document describes a chart with those kinds.
- **`media`**: `inputHandleId` (string, required — must match a connected input handle).
- **Standard Media (`Video`, `Image`, `GIF`, `SVG`, `Lottie`)**: `fit` (`"cover"` | `"contain"` | `"fill"` | `"none"`, default `"contain"`), `volume` (0–1), `muted`, `borderRadius`, `borderColor`, `borderWidth`.
- - **Video and audio timing**: `timeRemap` maps layer-local frames to source seconds, with an explicit `durationFrames`. Use `{ kind: "rate", sourceStartSec, rate }` with a nonzero rate (negative plays in reverse), or `{ kind: "keyframes", keyframes: [{ frame, sourceSec, interpolation? }, ...] }`. Keyframes span frame 0 through `durationFrames` and increase by frame; source time may increase, decrease, or hold. `interpolation` on the starting key applies to the following segment: `linear`, `ease-in`, `ease-out`, or `smooth`; all are monotonic. Equal source times hold video and silence audio. Rate changes alter audio pitch. Do not combine with `trimStartSec`; effects retain the composition clock.
+ - **Video and audio timing**: `timeRemap` maps layer-local frames to source seconds, with an explicit `durationFrames`. Use `{ kind: "rate", sourceStartSec, rate }` with a nonzero rate (negative plays in reverse), or `{ kind: "keyframes", keyframes: [{ frame, sourceSec, interpolation? }, ...] }`. Keyframes span frame 0 through `durationFrames` and increase by frame; source time may increase, decrease, or hold. `interpolation` on the starting key applies to the following segment: `linear`, `ease-in`, `ease-out`, or `smooth`; all are monotonic. Or `{ kind: "signal", mode: "speed" | "source-time", signal, sourceStartSec? }`: a signal instance or a named program signal read at each layer frame boundary on the composition clock (`speed` is integrated from `sourceStartSec`); it is sampled at compile time unless the host records it live. Equal source times hold video and silence audio. Rate changes alter audio pitch. Do not combine with `trimStartSec`; effects retain the composition clock.
- **Captions & Subtitles (`Caption` DataType)**: When bound to a `Caption` input (SRT source), `kind: "media"` renders time-synchronized subtitle cues:
- `fontSize` (number, default `48`): Font size in pixels.
- `fontFamily` (string, default `"Inter"`): Font family.
diff --git a/packages/compositions/src/program/evaluator.ts b/packages/compositions/src/program/evaluator.ts
index c62ccecd..c2bde79a 100644
--- a/packages/compositions/src/program/evaluator.ts
+++ b/packages/compositions/src/program/evaluator.ts
@@ -138,7 +138,7 @@ export function evaluateSpringOvershoot(
// ── Signal Sampler & Filter ───────────────────────────────────────────
-function extractRawSignalSample(
+export function extractRawSignalSample(
signalData: SignalDataSource,
frame: number,
fps: number,
diff --git a/packages/compositions/src/program/index.ts b/packages/compositions/src/program/index.ts
index 46550fc0..aa0489ae 100644
--- a/packages/compositions/src/program/index.ts
+++ b/packages/compositions/src/program/index.ts
@@ -11,6 +11,7 @@ export * from "./animation.js";
export * from "./evaluator.js";
export * from "./lint.js";
export * from "./mattes.js";
+export * from "./media-signal.js";
export * from "./path-morph.js";
export * from "./schema.js";
export * from "./sequence.js";
diff --git a/packages/compositions/src/program/media-signal.ts b/packages/compositions/src/program/media-signal.ts
new file mode 100644
index 00000000..a0c1c87c
--- /dev/null
+++ b/packages/compositions/src/program/media-signal.ts
@@ -0,0 +1,79 @@
+import {
+ compositionFrameContext,
+ isSignal,
+ type MediaTimeSignalRemap,
+ type ResolvedMediaTimeRemap,
+ sampleMediaSignal,
+} from "@framefields/core";
+import { extractRawSignalSample } from "./evaluator.js";
+
+/** Where a signal-driven media layer sits on the composition clock. */
+export interface MediaSignalLayer {
+ id?: string;
+ startFrame: number;
+ durationFrames: number;
+}
+
+export interface MediaSignalClock {
+ fps: number;
+ compositionDurationFrames: number;
+ /** Named composition and engine signals. */
+ signals?: Record<string, unknown>;
+}
+
+/**
+ * Read a `timeRemap` signal at a composition frame: a signal instance, a named
+ * signal, or a named engine signal (sample arrays, analysis channels).
+ */
+export function mediaSignalSampler(
+ binding: MediaTimeSignalRemap,
+ clock: MediaSignalClock,
+): (frame: number) => number {
+ const { fps, compositionDurationFrames } = clock;
+ if (
+ !Number.isInteger(compositionDurationFrames) ||
+ compositionDurationFrames <= 0
+ )
+ throw new Error(
+ "A signal timeRemap needs the composition duration (durationFrames or durationMs)",
+ );
+ const source =
+ typeof binding.signal === "string"
+ ? clock.signals?.[binding.signal]
+ : binding.signal;
+ if (source === undefined)
+ throw new Error(
+ `timeRemap signal '${String(binding.signal)}' is not a composition signal`,
+ );
+ if (isSignal(source))
+ return (frame) =>
+ Number(
+ source.get(
+ compositionFrameContext(frame, fps, compositionDurationFrames),
+ ),
+ );
+ return (frame) => extractRawSignalSample(source, frame, fps);
+}
+
+/**
+ * Sample a signal binding up front. Valid when nothing changes the signal
+ * while the composition renders; live signals are recorded per frame instead.
+ */
+export function sampleMediaTimeSignal(
+ binding: MediaTimeSignalRemap,
+ layer: MediaSignalLayer,
+ clock: MediaSignalClock,
+): ResolvedMediaTimeRemap {
+ const sample = mediaSignalSampler(binding, clock);
+ return {
+ kind: "sampled",
+ history: sampleMediaSignal({
+ mode: binding.mode,
+ fps: clock.fps,
+ startFrame: layer.startFrame,
+ durationFrames: layer.durationFrames,
+ sourceStartSec: binding.sourceStartSec,
+ sample,
+ }),
+ };
+}
diff --git a/packages/compositions/src/program/schema.test.ts b/packages/compositions/src/program/schema.test.ts
index ddcd94be..fadfef74 100644
--- a/packages/compositions/src/program/schema.test.ts
+++ b/packages/compositions/src/program/schema.test.ts
@@ -1,3 +1,4 @@
+import { Signal } from "@framefields/core";
import { describe, expect, it } from "vitest";
import {
CompositorProgramSchema,
@@ -1124,4 +1125,33 @@ describe("media time remapping schema", () => {
}).success,
).toBe(true);
});
+ it("accepts signal bindings by instance or composition signal name", () => {
+ const node = {
+ id: "clip",
+ kind: "media",
+ inputHandleId: "a.mp4",
+ dataType: "Video",
+ durationFrames: 30,
+ timeRemap: { kind: "signal", mode: "speed", signal: "pace" },
+ };
+ expect(LayoutNodeSchema.safeParse(node).success).toBe(true);
+ expect(
+ LayoutNodeSchema.safeParse({
+ ...node,
+ timeRemap: {
+ kind: "signal",
+ mode: "source-time",
+ signal: Signal.state(1),
+ },
+ }).success,
+ ).toBe(true);
+ for (const timeRemap of [
+ { kind: "signal", mode: "speed", signal: 3 },
+ { kind: "signal", mode: "source-time", signal: "p", sourceStartSec: 1 },
+ { kind: "signal", mode: "reverse", signal: "p" },
+ ])
+ expect(LayoutNodeSchema.safeParse({ ...node, timeRemap }).success).toBe(
+ false,
+ );
+ });
});
diff --git a/packages/compositions/src/program/schema.ts b/packages/compositions/src/program/schema.ts
index 84bd6489..311afc42 100644
--- a/packages/compositions/src/program/schema.ts
+++ b/packages/compositions/src/program/schema.ts
@@ -17,8 +17,8 @@
import {
Effect,
type EffectDomain,
- type MediaTimeRemap,
- MediaTimeRemapSchema,
+ type MediaTimeRemapInput,
+ MediaTimeRemapInputSchema,
type MeshAudioDeformConfig,
type TextSpan,
type TextSpanMark,
@@ -1127,7 +1127,7 @@ export interface MediaNode extends CompositionNodeBase {
/** Source in-point: seconds into the clip that play at the layer's startFrame. */
trimStartSec?: number;
/** Layer-local frame to absolute source time mapping. */
- timeRemap?: MediaTimeRemap;
+ timeRemap?: MediaTimeRemapInput;
// Caption styling
fontFamily?: string;
fontSize?: number;
@@ -1645,7 +1645,7 @@ export const MediaNodeSchema: z.ZodType<MediaNode> = z
volume: z.number().min(0).max(1).default(1),
muted: z.boolean().default(false),
trimStartSec: z.number().min(0).optional(),
- timeRemap: MediaTimeRemapSchema.optional(),
+ timeRemap: MediaTimeRemapInputSchema.optional(),
// Caption styling
fontFamily: z.string().optional(),
fontSize: z.number().positive().optional(),
diff --git a/packages/compositions/src/program/to-virtual-media.test.ts b/packages/compositions/src/program/to-virtual-media.test.ts
index e2305065..cb80967b 100644
--- a/packages/compositions/src/program/to-virtual-media.test.ts
+++ b/packages/compositions/src/program/to-virtual-media.test.ts
@@ -1,4 +1,9 @@
-import type { VirtualMediaData } from "@framefields/core";
+import {
+ mediaSourceTime,
+ type ResolvedMediaTimeRemap,
+ Signal,
+ type VirtualMediaData,
+} from "@framefields/core";
import { describe, expect, it } from "vitest";
import type { CompositorProgramConfig, LayoutNode } from "./schema.js";
import { compositorToProgram } from "./to-virtual-media.js";
@@ -70,6 +75,90 @@ describe("compositorToProgram", () => {
expect(vm.children[0].operation.durationFrames).toBe(60);
expect(vm.children[0].metadata.durationMs).toBe(2000);
});
+ it("resolves signal time remaps on the composition clock", () => {
+ const video = {
+ metadata: { durationMs: 10000 },
+ operation: { op: "source", dataType: "Video" },
+ children: [],
+ } as unknown as VirtualMediaData;
+ const layer = (timeRemap: unknown) => ({
+ width: 64,
+ height: 64,
+ fps: 4,
+ mode: "Video",
+ layout: [
+ {
+ id: "wrap",
+ kind: "box",
+ startFrame: 2,
+ children: [
+ {
+ id: "clip",
+ kind: "media",
+ inputHandleId: "source",
+ dataType: "Video",
+ startFrame: 1,
+ durationFrames: 4,
+ timeRemap,
+ },
+ ],
+ },
+ ],
+ });
+ const remapOf = (vm: VirtualMediaData) =>
+ (
+ vm.children[0].children[0].operation as unknown as {
+ timeRemap: ResolvedMediaTimeRemap;
+ }
+ ).timeRemap;
+ const options = {
+ isVideoMode: true,
+ fps: 4,
+ durationMs: 2500,
+ resolveMedia: () => video,
+ };
+ const instance = compositorToProgram(
+ layer({
+ kind: "signal",
+ mode: "source-time",
+ signal: Signal.builder({ type: "custom", fn: (ctx) => ctx.time }),
+ }) as unknown as CompositorProgramConfig,
+ options,
+ );
+ // The layer starts at composition frame 3: its frame 1 reads frame 4.
+ expect(mediaSourceTime(remapOf(instance), 1, 4)).toBe(1);
+ const named = compositorToProgram(
+ layer({
+ kind: "signal",
+ mode: "speed",
+ signal: "pace",
+ sourceStartSec: 2,
+ }) as unknown as CompositorProgramConfig,
+ { ...options, signals: { pace: [0, 0, 0, 2, 2, 2, 2, 2] } },
+ );
+ expect(mediaSourceTime(remapOf(named), 4, 4)).toBe(4);
+ expect(() =>
+ compositorToProgram(
+ layer({
+ kind: "signal",
+ mode: "speed",
+ signal: "missing",
+ }) as unknown as CompositorProgramConfig,
+ options,
+ ),
+ ).toThrow(/not a composition signal/);
+ expect(() =>
+ compositorToProgram(
+ layer({
+ kind: "signal",
+ mode: "source-time",
+ signal: "pace",
+ sourceStartSec: 1,
+ }) as unknown as CompositorProgramConfig,
+ { ...options, signals: { pace: [0] } },
+ ),
+ ).toThrow(/only to speed signals/);
+ });
it("builds the render tree 1:1 from the document", () => {
const vm = compositorToProgram(doc, { isVideoMode: true, fps: 30 });
const op = vm.operation as any;
diff --git a/packages/compositions/src/program/to-virtual-media.ts b/packages/compositions/src/program/to-virtual-media.ts
index 83999b12..9d32fd80 100644
--- a/packages/compositions/src/program/to-virtual-media.ts
+++ b/packages/compositions/src/program/to-virtual-media.ts
@@ -17,12 +17,18 @@ import {
type DataType,
type FrameContext,
getActiveMediaMetadata,
- MediaTimeRemapSchema,
+ MediaTimeRemapInputSchema,
+ type MediaTimeSignalRemap,
+ type ResolvedMediaTimeRemap,
type VirtualMediaData,
validateMediaTimeRemap,
} from "@framefields/core";
import { collectInlineSignals } from "./inline-signals.js";
import { hoistMattes } from "./mattes.js";
+import {
+ type MediaSignalLayer,
+ sampleMediaTimeSignal,
+} from "./media-signal.js";
import type {
CompositorProgramConfig,
LayoutNode,
@@ -46,8 +52,18 @@ export interface CompositorToProgramOptions {
effects?: ProgramEffect[];
/** Composition-wide vision config or bundle. */
vision?: unknown;
+ /**
+ * Resolve a signal `timeRemap` to the history its consumers read. By
+ * default the signal is sampled up front on the composition clock.
+ */
+ resolveMediaSignal?: MediaSignalResolver;
}
+export type MediaSignalResolver = (
+ binding: MediaTimeSignalRemap,
+ layer: MediaSignalLayer,
+) => ResolvedMediaTimeRemap;
+
function nodeToVirtualMedia(
node: LayoutNode,
resolveMedia: CompositorToProgramOptions["resolveMedia"],
@@ -55,9 +71,10 @@ function nodeToVirtualMedia(
inheritedOffset = 0,
compDurationMs?: number,
inheritedDurationFrames?: number,
+ resolveSignal?: MediaSignalResolver,
): VirtualMediaData {
if (node.kind === "media" && node.timeRemap !== undefined) {
- const parsed = MediaTimeRemapSchema.safeParse(node.timeRemap);
+ const parsed = MediaTimeRemapInputSchema.safeParse(node.timeRemap);
if (!parsed.success)
throw new Error(
`Invalid timeRemap on '${node.id}': ${parsed.error.message}`,
@@ -222,6 +239,7 @@ function nodeToVirtualMedia(
childOffset,
compDurationMs,
c.durationFrames === undefined ? nodeDuration : undefined,
+ resolveSignal,
);
});
}
@@ -315,6 +333,12 @@ function nodeToVirtualMedia(
}
}
+ const signalRemap =
+ node.kind === "media" && node.timeRemap?.kind === "signal"
+ ? node.timeRemap
+ : undefined;
+ if (signalRemap && !resolveSignal)
+ throw new Error(`Signal timeRemap on '${node.id}' has no resolver`);
return {
metadata: {
durationMs,
@@ -325,6 +349,14 @@ function nodeToVirtualMedia(
startFrame,
durationFrames: dur,
dataType,
+ ...(signalRemap &&
+ resolveSignal && {
+ timeRemap: resolveSignal(signalRemap, {
+ id: node.id,
+ startFrame,
+ durationFrames: dur,
+ }),
+ }),
},
children,
} as unknown as VirtualMediaData;
@@ -335,6 +367,18 @@ export function compositorToProgram(
options: CompositorToProgramOptions,
): VirtualMediaData {
const { isVideoMode, fps, durationMs, resolveMedia, signals } = options;
+ const programSignals = {
+ ...collectInlineSignals(config.layout),
+ ...(signals ?? config.signals ?? {}),
+ };
+ const resolveMediaSignal: MediaSignalResolver =
+ options.resolveMediaSignal ??
+ ((binding, layer) =>
+ sampleMediaTimeSignal(binding, layer, {
+ fps,
+ compositionDurationFrames: Math.round(((durationMs ?? 0) / 1000) * fps),
+ signals: programSignals,
+ }));
let rootMedia: VirtualMediaData = {
metadata: {
@@ -364,10 +408,7 @@ export function compositorToProgram(
layout: config.layout,
// Signal objects passed straight to `LayerAnimation.signal` ride on
// their tracks; fold them in under their generated handle ids.
- signals: {
- ...collectInlineSignals(config.layout),
- ...(signals ?? config.signals ?? {}),
- },
+ signals: programSignals,
...(config.onRequestFrame || options.onRequestFrame
? {
onRequestFrame: config.onRequestFrame ?? options.onRequestFrame,
@@ -375,7 +416,15 @@ export function compositorToProgram(
: {}),
},
children: hoistMattes(config.layout).map((n) =>
- nodeToVirtualMedia(n, resolveMedia, fps, 0, durationMs),
+ nodeToVirtualMedia(
+ n,
+ resolveMedia,
+ fps,
+ 0,
+ durationMs,
+ undefined,
+ resolveMediaSignal,
+ ),
),
} as unknown as VirtualMediaData;
diff --git a/packages/compositions/src/utils/audio-extractor.ts b/packages/compositions/src/utils/audio-extractor.ts
index 627a3e65..5acb7f45 100644
--- a/packages/compositions/src/utils/audio-extractor.ts
+++ b/packages/compositions/src/utils/audio-extractor.ts
@@ -1,10 +1,10 @@
import {
getActiveMediaMetadata,
getMediaType,
- type MediaTimeRemap,
mediaSourceIsHold,
mediaSourceRange,
mediaSourceTime,
+ type ResolvedMediaTimeRemap,
type VirtualMediaData,
} from "@framefields/core";
import { type AudioProcessor, audioRegistry } from "@framefields/node-sdk";
@@ -450,7 +450,7 @@ async function resampleMappedPCM(
outputRate: number,
length: number,
inputStart: number,
- map: MediaTimeRemap,
+ map: ResolvedMediaTimeRemap,
startFrame: number,
fps: number,
): Promise<Float32Array> {
@@ -670,7 +670,7 @@ export async function mixAudioTracks(
async function mappedChannels(
url: string,
length: number,
- map: MediaTimeRemap,
+ map: ResolvedMediaTimeRemap,
startFrame: number,
): Promise<Float32Array[] | null> {
const left = new Float32Array(length),
@@ -763,7 +763,7 @@ export async function mixAudioTracks(
inheritedVolume: number,
inheritedSpeed: number,
inheritedDurationFrames: number | null,
- inheritedRemap?: { map: MediaTimeRemap; offsetFrames: number },
+ inheritedRemap?: { map: ResolvedMediaTimeRemap; offsetFrames: number },
): Promise<Float32Array[] | null> {
if (!node || !isAudible(node)) return null;
const op = node.operation;
@@ -817,7 +817,7 @@ export async function mixAudioTracks(
const speedMultiplier = inheritedSpeed;
const nodeStartFrame = op.startFrame ?? op.timeline?.startFrame ?? 0;
- const ownRemap = (op as { timeRemap?: MediaTimeRemap }).timeRemap;
+ const ownRemap = (op as { timeRemap?: ResolvedMediaTimeRemap }).timeRemap;
const remap = ownRemap
? { map: ownRemap, offsetFrames: 0 }
: inheritedRemap
diff --git a/packages/compositions/src/utils/media-time-preflight.ts b/packages/compositions/src/utils/media-time-preflight.ts
index 4a336390..1e10d87b 100644
--- a/packages/compositions/src/utils/media-time-preflight.ts
+++ b/packages/compositions/src/utils/media-time-preflight.ts
@@ -1,6 +1,7 @@
import {
- type MediaTimeRemap,
+ type MediaSourceRange,
mediaSourceRange,
+ type ResolvedMediaTimeRemap,
type VirtualMediaData,
} from "@framefields/core";
import { inputStore } from "@framefields/webgpu-renderers";
@@ -29,13 +30,13 @@ export async function preflightMediaTime(
muted?: boolean;
volume?: number;
durationFrames?: number;
- timeRemap?: MediaTimeRemap;
+ timeRemap?: ResolvedMediaTimeRemap;
};
if (op?.op === "CompositorLayer" && op.timeRemap && op.durationFrames) {
const source = sourceOf(node);
if (!source)
throw new Error(`Time remap on '${op.id}' has no media source`);
- const range = mediaSourceRange(op.timeRemap, 0, op.durationFrames, fps);
+ const bounds: Array<{ first: number; last: number }> = [];
const input = await inputStore.acquire(source);
try {
const video =
@@ -54,8 +55,16 @@ export async function preflightMediaTime(
);
for (const track of [video, audio]) {
if (!track) continue;
- const first = await input.getFirstTimestamp([track]);
- const last = await track.computeDuration();
+ bounds.push({
+ first: await input.getFirstTimestamp([track]),
+ last: await track.computeDuration(),
+ });
+ }
+ } finally {
+ inputStore.release(source);
+ }
+ const check = (range: MediaSourceRange) => {
+ for (const { first, last } of bounds) {
if (
range.min < first - 1e-8 ||
range.max > last + 1e-8 ||
@@ -66,9 +75,11 @@ export async function preflightMediaTime(
);
}
}
- } finally {
- inputStore.release(source);
- }
+ };
+ // A live history is checked interval by interval as it is recorded.
+ if (op.timeRemap.kind === "recorded")
+ op.timeRemap.recorder.checkRange(check);
+ else check(mediaSourceRange(op.timeRemap, 0, op.durationFrames, fps));
}
for (const child of node.children ?? []) await walk(child);
};
diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts
index cecbd497..1ee3a5ba 100644
--- a/packages/core/src/index.ts
+++ b/packages/core/src/index.ts
@@ -2,6 +2,7 @@ export * from "./effects/index.js";
export * from "./errors.js";
export * from "./id-gen.js";
export * from "./media-time.js";
+export * from "./media-time-signal.js";
export * from "./node-metadata.js";
export * from "./render-diagnostics.js";
export * from "./signals/index.js";
diff --git a/packages/core/src/media-time-signal.test.ts b/packages/core/src/media-time-signal.test.ts
new file mode 100644
index 00000000..68e89dbf
--- /dev/null
+++ b/packages/core/src/media-time-signal.test.ts
@@ -0,0 +1,352 @@
+import { describe, expect, it } from "vitest";
+import {
+ mediaSourceIsHold,
+ mediaSourceRange,
+ mediaSourceRate,
+ mediaSourceTime,
+} from "./media-time.js";
+import {
+ MediaSignalRecorder,
+ sampledMediaSourceRange,
+ sampledMediaSourceTime,
+ sampleFrameMediaSignal,
+ sampleMediaSignal,
+} from "./media-time-signal.js";
+import { Signal } from "./signals/index.js";
+import { ProgrammaticSignal } from "./signals/programmatic.js";
+
+describe("sampled media signals", () => {
+ it("samples programmatic and engine signals on the composition clock", () => {
+ const frames: number[] = [];
+ const programmatic = new ProgrammaticSignal((ctx) => {
+ frames.push(ctx.frame);
+ expect(ctx.time).toBe(ctx.frame / 24);
+ expect(ctx.duration).toBe(4);
+ expect(ctx.progress).toBe(ctx.frame / 96);
+ return ctx.frame / 24;
+ });
+ const options = {
+ mode: "source-time" as const,
+ fps: 24,
+ startFrame: 24,
+ durationFrames: 2,
+ compositionDurationFrames: 96,
+ };
+ const map = sampleFrameMediaSignal({ ...options, signal: programmatic });
+ expect(frames).toEqual([24, 25, 26]);
+ expect(sampledMediaSourceTime(map, 1)).toBeCloseTo(25 / 24);
+ const engine = Signal.builder({
+ type: "custom",
+ fn: (ctx) => ctx.frame / 24,
+ });
+ const engineMap = sampleFrameMediaSignal({ ...options, signal: engine });
+ expect(sampledMediaSourceTime(engineMap, 1)).toBeCloseTo(25 / 24);
+ });
+ it("does not reuse a programmatic value across different frame contexts", () => {
+ const signal = new ProgrammaticSignal((ctx) => ctx.time + ctx.progress);
+ const context = {
+ frame: 24,
+ fps: 24,
+ time: 1,
+ duration: 4,
+ durationMs: 4000,
+ progress: 0.25,
+ deltaTime: 1 / 24,
+ };
+ expect(signal.get(context)).toBe(1.25);
+ expect(
+ signal.get({ ...context, fps: 48, time: 0.5, progress: 0.125 }),
+ ).toBe(0.625);
+ });
+ it("resamples an array signal when the same frame is read at a different fps", () => {
+ const signal = Signal.fromArray(
+ Array.from({ length: 30 }, (_, i) => i),
+ 24,
+ );
+ const context = {
+ frame: 24,
+ fps: 24,
+ time: 1,
+ duration: 4,
+ durationMs: 4000,
+ progress: 0.25,
+ deltaTime: 1 / 24,
+ };
+ expect(signal.get(context)).toBe(24);
+ expect(signal.get({ ...context, fps: 48, time: 0.5 })).toBe(12);
+ });
+ it("integrates linearly varying speed at fractional frames and supports random access", () => {
+ const map = sampleMediaSignal({
+ mode: "speed",
+ fps: 2,
+ startFrame: 4,
+ durationFrames: 2,
+ sourceStartSec: 1,
+ sample: (frame) => frame - 4,
+ });
+ expect(sampledMediaSourceTime(map, 0)).toBe(1);
+ expect(sampledMediaSourceTime(map, 0.5)).toBe(1.0625);
+ expect(sampledMediaSourceTime(map, 1)).toBe(1.25);
+ expect(sampledMediaSourceTime(map, 2)).toBe(2);
+ expect(sampledMediaSourceTime(map, 0.5)).toBe(1.0625);
+ });
+ it("finds source-time extrema between frame boundaries", () => {
+ const map = sampleMediaSignal({
+ mode: "speed",
+ fps: 1,
+ startFrame: 0,
+ durationFrames: 1,
+ sourceStartSec: 1,
+ sample: (f) => 1 - 2 * f,
+ });
+ expect(sampledMediaSourceRange(map, 0, 1)).toEqual({
+ min: 1,
+ max: 1.25,
+ maxIncluded: true,
+ });
+ });
+ it("interpolates source times and rejects non-finite samples", () => {
+ const map = sampleMediaSignal({
+ mode: "source-time",
+ fps: 24,
+ startFrame: 10,
+ durationFrames: 2,
+ sample: (frame) => 12 - frame,
+ });
+ expect(sampledMediaSourceTime(map, 0.5)).toBe(1.5);
+ expect(sampledMediaSourceRange(map, 0, 2)).toEqual({
+ min: 0,
+ max: 2,
+ maxIncluded: true,
+ });
+ expect(() =>
+ sampleMediaSignal({
+ mode: "source-time",
+ fps: 24,
+ startFrame: 0,
+ durationFrames: 2,
+ sample: () => NaN,
+ }),
+ ).toThrow(/Non-finite/);
+ });
+ it("rejects an overflow while integrating finite signal values", () => {
+ expect(() =>
+ sampleMediaSignal({
+ mode: "speed",
+ fps: 0.1,
+ startFrame: 0,
+ durationFrames: 1,
+ sample: () => Number.MAX_VALUE,
+ }),
+ ).toThrow(/overflow/);
+ });
+ it("distinguishes an exclusive upper boundary from a sampled upper bound", () => {
+ const ascending = sampleMediaSignal({
+ mode: "source-time",
+ fps: 24,
+ startFrame: 0,
+ durationFrames: 1,
+ sample: (frame) => frame,
+ });
+ expect(sampledMediaSourceRange(ascending, 0, 1)).toEqual({
+ min: 0,
+ max: 1,
+ maxIncluded: false,
+ });
+ const held = sampleMediaSignal({
+ mode: "speed",
+ fps: 24,
+ startFrame: 0,
+ durationFrames: 1,
+ sourceStartSec: 1,
+ sample: () => 0,
+ });
+ expect(sampledMediaSourceRange(held, 0, 1)).toEqual({
+ min: 1,
+ max: 1,
+ maxIncluded: true,
+ });
+ });
+ it("evaluates large finite signals without overflowing intermediate differences", () => {
+ const map = sampleMediaSignal({
+ mode: "speed",
+ fps: 1,
+ startFrame: 0,
+ durationFrames: 1,
+ sample: (f) => (f === 0 ? 1e308 : -1e308),
+ });
+ expect(sampledMediaSourceTime(map, 1)).toBe(0);
+ expect(sampledMediaSourceTime(map, 0.5)).toBeCloseTo(2.5e307);
+ expect(sampledMediaSourceRange(map, 0, 1).max).toBeCloseTo(2.5e307);
+ });
+});
+
+describe("live media signal history", () => {
+ it("exposes recorded boundaries to the shared media evaluators without looking ahead", () => {
+ const recorder = new MediaSignalRecorder({
+ mode: "speed",
+ fps: 2,
+ startFrame: 10,
+ durationFrames: 2,
+ sourceStartSec: 1,
+ });
+ const map = { kind: "recorded" as const, recorder };
+ recorder.record(10, 1);
+ expect(mediaSourceTime(map, 0, 2)).toBe(1);
+ expect(() => mediaSourceTime(map, 0.5, 2)).toThrow(/not been recorded/);
+ recorder.record(11, -1);
+ expect(mediaSourceTime(map, 0.5, 2)).toBe(1.125);
+ expect(mediaSourceRange(map, 0, 1, 2)).toEqual({
+ min: 1,
+ max: 1.125,
+ maxIncluded: true,
+ });
+ expect(mediaSourceRate(map, 0.5, 2)).toBe(0);
+ expect(mediaSourceIsHold(map, 0)).toBe(false);
+ expect(() => mediaSourceRange(map, 0, 2, 2)).toThrow(/not been recorded/);
+ });
+ it("uses recorded history through the shared media timing evaluators", () => {
+ const history = sampleMediaSignal({
+ mode: "speed",
+ fps: 2,
+ startFrame: 0,
+ durationFrames: 2,
+ sourceStartSec: 1,
+ sample: (f) => 1 - f,
+ });
+ const map = { kind: "sampled" as const, history };
+ expect(mediaSourceTime(map, 1, 2)).toBe(1.25);
+ expect(mediaSourceRate(map, 0.5, 2)).toBe(0.5);
+ expect(mediaSourceIsHold(map, 0)).toBe(false);
+ expect(mediaSourceRange(map, 0, 2, 2)).toEqual({
+ min: 1,
+ max: 1.25,
+ maxIncluded: true,
+ });
+ });
+ it("reports a circular signal dependency and recovers after an error", () => {
+ const a = new MediaSignalRecorder({
+ mode: "source-time",
+ fps: 24,
+ startFrame: 0,
+ durationFrames: 1,
+ });
+ const b = new MediaSignalRecorder({
+ mode: "source-time",
+ fps: 24,
+ startFrame: 0,
+ durationFrames: 1,
+ });
+ expect(() =>
+ a.recordFrom(0, () => {
+ b.recordFrom(0, () => {
+ a.recordFrom(0, () => 0);
+ return 0;
+ });
+ return 0;
+ }),
+ ).toThrow(/Circular media signal dependency/);
+ a.recordFrom(0, () => 0.5);
+ b.recordFrom(0, () => 0.25);
+ expect(a.at(0)).toBe(0.5);
+ expect(b.at(0)).toBe(0.25);
+ });
+ it("records mutable speed after each frame and replays fractional positions", () => {
+ const speed = Signal.state(1);
+ const recorder = new MediaSignalRecorder({
+ mode: "speed",
+ fps: 2,
+ startFrame: 10,
+ durationFrames: 2,
+ sourceStartSec: 2,
+ });
+ recorder.record(10, speed.get());
+ speed.set(-1);
+ recorder.record(11, speed.get());
+ recorder.record(12, speed.get());
+ expect(recorder.at(10)).toBe(2);
+ expect(recorder.at(11)).toBe(2);
+ expect(recorder.at(12)).toBe(1.5);
+ const history = recorder.finish();
+ expect(sampledMediaSourceTime(history, 0.5)).toBe(2.125);
+ expect(sampledMediaSourceTime(history, 2)).toBe(1.5);
+ speed.set(20);
+ expect(sampledMediaSourceTime(history, 2)).toBe(1.5);
+ });
+ it("samples live boundaries on demand up to the clock, never ahead", () => {
+ const speed = Signal.state(2);
+ const reads: number[] = [];
+ const recorder = new MediaSignalRecorder({
+ mode: "speed",
+ fps: 2,
+ startFrame: 4,
+ durationFrames: 3,
+ sample: (frame) => {
+ reads.push(frame);
+ return speed.get();
+ },
+ });
+ const map = { kind: "recorded" as const, recorder };
+ expect(() => mediaSourceTime(map, 0, 2)).toThrow(/not been recorded/);
+ recorder.advance(5);
+ expect(mediaSourceTime(map, 1, 2)).toBe(1);
+ expect(() => mediaSourceTime(map, 1.5, 2)).toThrow(/not been recorded/);
+ speed.set(0);
+ recorder.advance(6);
+ recorder.ensure(6);
+ expect(mediaSourceTime(map, 2, 2)).toBe(1.5);
+ expect(reads).toEqual([4, 5, 6]);
+ recorder.close();
+ expect(recorder.complete).toBe(true);
+ expect(mediaSourceTime(map, 3, 2)).toBe(1.5);
+ expect(mediaSourceIsHold(map, 2.5)).toBe(true);
+ });
+ it("reports a signal that reads its own layer's current source time", () => {
+ const recorder: MediaSignalRecorder = new MediaSignalRecorder({
+ mode: "source-time",
+ fps: 24,
+ startFrame: 0,
+ durationFrames: 2,
+ sample: (frame) => (frame === 0 ? 0 : recorder.at(frame) + 1),
+ });
+ recorder.advance(1);
+ expect(recorder.at(0)).toBe(0);
+ expect(() => recorder.at(1)).toThrow(/Circular media signal dependency/);
+ });
+ it("validates source bounds interval by interval as history arrives", () => {
+ const recorder = new MediaSignalRecorder({
+ mode: "source-time",
+ fps: 1,
+ startFrame: 0,
+ durationFrames: 3,
+ });
+ recorder.record(0, 1);
+ recorder.record(1, 2);
+ const ranges: Array<{ min: number; max: number; maxIncluded: boolean }> =
+ [];
+ recorder.checkRange((range) => {
+ ranges.push(range);
+ if (range.max > 2.5) throw new Error("outside the source");
+ });
+ recorder.record(2, 2.5);
+ expect(ranges).toEqual([
+ { min: 1, max: 2, maxIncluded: false },
+ { min: 2, max: 2.5, maxIncluded: false },
+ ]);
+ expect(() => recorder.record(3, 3)).toThrow(/outside the source/);
+ });
+ it("rejects missing, reordered, and conflicting frame samples", () => {
+ const recorder = new MediaSignalRecorder({
+ mode: "source-time",
+ fps: 24,
+ startFrame: 5,
+ durationFrames: 2,
+ });
+ expect(() => recorder.record(6, 1)).toThrow(/out of order/);
+ recorder.record(5, 0.5);
+ recorder.record(5, 0.5);
+ expect(() => recorder.record(5, 0.6)).toThrow(/different value/);
+ expect(() => recorder.at(6)).toThrow(/not been recorded/);
+ expect(() => recorder.finish()).toThrow(/incomplete/);
+ });
+});
diff --git a/packages/core/src/media-time-signal.ts b/packages/core/src/media-time-signal.ts
new file mode 100644
index 00000000..a890aa39
--- /dev/null
+++ b/packages/core/src/media-time-signal.ts
@@ -0,0 +1,403 @@
+import type { FrameContext, Signal } from "./signals/index.js";
+
+/** A sampled source-time curve or the integral of a sampled playback rate. */
+export type SampledMediaSignal =
+ | {
+ readonly mode: "source-time";
+ readonly values: Float64Array;
+ readonly fps: number;
+ }
+ | {
+ readonly mode: "speed";
+ readonly values: Float64Array;
+ readonly positions: Float64Array;
+ readonly fps: number;
+ };
+
+/**
+ * Sample a signal at every composition-frame boundary in layer order.
+ * Values between boundaries are linearly interpolated; speeds are integrated
+ * analytically, so picture and audio can query the same fractional frame.
+ */
+export function sampleMediaSignal(options: {
+ mode: "source-time" | "speed";
+ fps: number;
+ startFrame: number;
+ durationFrames: number;
+ sourceStartSec?: number;
+ sample: (compositionFrame: number) => number;
+}): SampledMediaSignal {
+ const recorder = new MediaSignalRecorder(options);
+ for (let i = 0; i <= options.durationFrames; i++) {
+ const frame = options.startFrame + i;
+ recorder.record(frame, options.sample(frame));
+ }
+ return recorder.finish();
+}
+
+/** Sample an engine signal with its composition clock and full frame context. */
+export function sampleFrameMediaSignal(options: {
+ mode: "source-time" | "speed";
+ fps: number;
+ startFrame: number;
+ durationFrames: number;
+ compositionDurationFrames: number;
+ sourceStartSec?: number;
+ signal: Signal<number>;
+}): SampledMediaSignal {
+ const { fps, compositionDurationFrames, signal } = options;
+ if (
+ !Number.isInteger(compositionDurationFrames) ||
+ compositionDurationFrames <= 0
+ )
+ throw new Error("Invalid composition duration");
+ return sampleMediaSignal({
+ ...options,
+ sample: (frame) =>
+ signal.get(
+ compositionFrameContext(frame, fps, compositionDurationFrames),
+ ),
+ });
+}
+
+/** The frame context a composition-clock signal sees at a composition frame. */
+export function compositionFrameContext(
+ frame: number,
+ fps: number,
+ compositionDurationFrames: number,
+): FrameContext {
+ return {
+ frame,
+ fps,
+ time: frame / fps,
+ duration: compositionDurationFrames / fps,
+ durationMs: (compositionDurationFrames / fps) * 1000,
+ progress: Math.min(1, frame / compositionDurationFrames),
+ deltaTime: 1 / fps,
+ };
+}
+
+/** Evaluate at any fractional local frame, independent of query order. */
+export function sampledMediaSourceTime(
+ map: SampledMediaSignal,
+ frame: number,
+): number {
+ if (!Number.isFinite(frame)) throw new Error("Invalid media signal frame");
+ const max = map.values.length - 1;
+ const f = Math.max(0, Math.min(max, frame));
+ const i = Math.min(Math.floor(f), max - 1);
+ const t = f - i;
+ const a = map.values[i];
+ const b = map.values[i + 1];
+ if (map.mode === "source-time") return a * (1 - t) + b * t;
+ return (
+ map.positions[i] + (a * (t - (t * t) / 2) + b * ((t * t) / 2)) / map.fps
+ );
+}
+
+/** Source-time bounds over a layer interval; `max` is exclusive unless included. */
+export interface MediaSourceRange {
+ min: number;
+ max: number;
+ maxIncluded: boolean;
+}
+
+/** Range includes direction changes inside a speed-signal interval. */
+export function sampledMediaSourceRange(
+ map: SampledMediaSignal,
+ from: number,
+ to: number,
+): MediaSourceRange {
+ if (!Number.isFinite(from) || !Number.isFinite(to))
+ throw new Error("Invalid media signal range");
+ const endFrame = map.values.length - 1;
+ const start = Math.max(0, Math.min(endFrame, Math.min(from, to)));
+ const end = Math.max(0, Math.min(endFrame, Math.max(from, to)));
+ let min = sampledMediaSourceTime(map, start);
+ let max = min;
+ let maxIncluded = start < end;
+ const add = (frame: number) => {
+ const value = sampledMediaSourceTime(map, frame);
+ min = Math.min(min, value);
+ if (value > max) {
+ max = value;
+ maxIncluded = frame < end;
+ } else if (value === max && frame < end) {
+ maxIncluded = true;
+ }
+ };
+ add(end);
+ for (let i = Math.floor(start); i < end; i++) {
+ if (i > start) add(i);
+ if (map.mode === "speed") {
+ const a = map.values[i],
+ b = map.values[i + 1];
+ if ((a < 0 && b > 0) || (a > 0 && b < 0)) {
+ const halfA = Math.abs(a) / 2;
+ const turn = i + halfA / (halfA + Math.abs(b) / 2);
+ if (turn > start && turn < end) add(turn);
+ }
+ }
+ }
+ return { min, max, maxIncluded };
+}
+
+/**
+ * Records a live signal at each composition-frame boundary, so its history can
+ * be replayed for audio, seeks and vision without evaluating it a second time.
+ * With a sampler, boundaries are evaluated on demand, but never past the
+ * frame the composition has reached ({@link advance}): no look-ahead.
+ */
+export class MediaSignalRecorder {
+ private readonly values: Float64Array;
+ private readonly positions: Float64Array | undefined;
+ private recorded = 0;
+ private evaluating = false;
+ private horizon = -1;
+ private sampler: ((frame: number) => number) | undefined;
+ private check: ((range: MediaSourceRange) => void) | undefined;
+ readonly startFrame: number;
+ readonly durationFrames: number;
+ readonly fps: number;
+ readonly mode: "source-time" | "speed";
+
+ constructor(options: {
+ mode: "source-time" | "speed";
+ fps: number;
+ startFrame: number;
+ durationFrames: number;
+ sourceStartSec?: number;
+ sample?: (compositionFrame: number) => number;
+ }) {
+ if (
+ !Number.isFinite(options.fps) ||
+ options.fps <= 0 ||
+ !Number.isInteger(options.startFrame) ||
+ options.startFrame < 0 ||
+ !Number.isInteger(options.durationFrames) ||
+ options.durationFrames <= 0
+ )
+ throw new Error("Invalid media signal timing");
+ const start = options.sourceStartSec ?? 0;
+ if (!Number.isFinite(start) || start < 0)
+ throw new Error("Invalid media signal source start");
+ this.mode = options.mode;
+ this.fps = options.fps;
+ this.startFrame = options.startFrame;
+ this.durationFrames = options.durationFrames;
+ this.sampler = options.sample;
+ this.values = new Float64Array(options.durationFrames + 1);
+ if (options.mode === "speed") {
+ this.positions = new Float64Array(options.durationFrames + 1);
+ this.positions[0] = start;
+ }
+ }
+
+ /** The first composition frame whose boundary is not yet recorded. */
+ get nextFrame(): number {
+ return this.startFrame + this.recorded;
+ }
+
+ get complete(): boolean {
+ return this.recorded === this.values.length;
+ }
+
+ /** Allow on-demand sampling through a composition frame the clock has reached. */
+ advance(frame: number): void {
+ if (frame > this.horizon) this.horizon = frame;
+ }
+
+ /**
+ * Validate source bounds as history arrives: the recorded part now, each
+ * further interval as it is recorded.
+ */
+ checkRange(check: (range: MediaSourceRange) => void): void {
+ this.check = check;
+ if (this.recorded > 1) check(this.range(0, this.recorded - 1));
+ }
+
+ /** Each boundary is recorded once in composition-frame order. */
+ record(frame: number, value: number): void {
+ const index = frame - this.startFrame;
+ if (!Number.isInteger(frame) || index < 0 || index > this.durationFrames)
+ throw new Error(`Media signal frame ${frame} is outside the layer`);
+ if (!Number.isFinite(value))
+ throw new Error(`Non-finite media signal at composition frame ${frame}`);
+ if (index < this.recorded) {
+ if (this.values[index] !== value)
+ throw new Error(
+ `Media signal frame ${frame} was already recorded with a different value`,
+ );
+ return;
+ }
+ if (index !== this.recorded)
+ throw new Error(`Media signal frame ${frame} was recorded out of order`);
+ this.values[index] = value;
+ if (this.positions && index > 0) {
+ const position =
+ this.positions[index - 1] +
+ (this.values[index - 1] / 2 + value / 2) / this.fps;
+ if (!Number.isFinite(position))
+ throw new Error(
+ `Media signal integration overflow at composition frame ${frame}`,
+ );
+ this.positions[index] = position;
+ }
+ if (this.check && index > 0) this.check(this.interval(index - 1));
+ this.recorded++;
+ }
+
+ /**
+ * Evaluate a boundary once, reporting recursive time dependencies. A
+ * boundary already recorded is kept, and `sample` is not called again.
+ */
+ recordFrom(frame: number, sample: () => number): void {
+ if (this.evaluating)
+ throw new Error(
+ `Circular media signal dependency at composition frame ${frame}`,
+ );
+ if (frame - this.startFrame < this.recorded) return;
+ this.evaluating = true;
+ try {
+ this.record(frame, sample());
+ } finally {
+ this.evaluating = false;
+ }
+ }
+
+ /** Sample every boundary through `frame` that the clock has reached. */
+ ensure(frame: number): void {
+ const sampler = this.sampler;
+ if (!sampler) return;
+ const last = Math.min(
+ frame,
+ this.horizon,
+ this.startFrame + this.durationFrames,
+ );
+ if (this.evaluating && frame >= this.nextFrame)
+ throw new Error(
+ `Circular media signal dependency at composition frame ${this.nextFrame}`,
+ );
+ for (let f = this.nextFrame; f <= last; f++)
+ this.recordFrom(f, () => sampler(f));
+ }
+
+ /**
+ * Close a layer that outlives the clock: the boundaries past the last
+ * rendered frame hold the last recorded value.
+ */
+ close(): void {
+ if (this.recorded === 0)
+ throw new Error("Media signal history has no recorded frames");
+ const last = this.values[this.recorded - 1];
+ while (!this.complete) this.record(this.nextFrame, last);
+ }
+
+ /** The known source position at an already-recorded frame boundary. */
+ at(frame: number): number {
+ this.ensure(frame);
+ const index = frame - this.startFrame;
+ if (!Number.isInteger(frame) || index < 0 || index >= this.recorded)
+ throw new Error(`Media signal frame ${frame} has not been recorded`);
+ return this.positions ? this.positions[index] : this.values[index];
+ }
+
+ /** Read a recorded local frame, including fractional audio sample positions. */
+ sourceTime(localFrame: number): number {
+ if (Number.isFinite(localFrame))
+ this.ensure(this.startFrame + Math.ceil(localFrame));
+ if (
+ !Number.isFinite(localFrame) ||
+ localFrame < 0 ||
+ localFrame > this.recorded - 1
+ )
+ throw new Error(
+ `Media signal local frame ${localFrame} has not been recorded`,
+ );
+ if (Number.isInteger(localFrame))
+ return this.at(this.startFrame + localFrame);
+ const i = Math.floor(localFrame);
+ const t = localFrame - i;
+ const a = this.values[i];
+ const b = this.values[i + 1];
+ return this.positions
+ ? this.positions[i] +
+ (a * (t - (t * t) / 2) + b * ((t * t) / 2)) / this.fps
+ : a * (1 - t) + b * t;
+ }
+
+ /** Read the instantaneous speed for an interval whose endpoints are known. */
+ rate(localFrame: number): number {
+ const i = this.knownInterval(localFrame);
+ const a = this.values[i];
+ const b = this.values[i + 1];
+ const t = localFrame - i;
+ return this.positions ? a * (1 - t) + b * t : (b - a) * this.fps;
+ }
+
+ isHold(localFrame: number): boolean {
+ const i = this.knownInterval(localFrame);
+ return this.positions
+ ? this.values[i] === 0 && this.values[i + 1] === 0
+ : this.values[i] === this.values[i + 1];
+ }
+
+ range(from: number, to: number): MediaSourceRange {
+ if (Number.isFinite(from) && Number.isFinite(to))
+ this.ensure(this.startFrame + Math.ceil(Math.max(from, to)));
+ if (
+ !Number.isFinite(from) ||
+ !Number.isFinite(to) ||
+ from < 0 ||
+ to < 0 ||
+ Math.max(from, to) > this.recorded - 1
+ )
+ throw new Error("Media signal range has not been recorded");
+ if (this.recorded === 1) {
+ const value = this.sourceTime(0);
+ return { min: value, max: value, maxIncluded: false };
+ }
+ return sampledMediaSourceRange(this.history(0, this.recorded), from, to);
+ }
+
+ /** A self-contained snapshot, available only when every boundary was recorded. */
+ finish(): SampledMediaSignal {
+ if (!this.complete) throw new Error("Media signal history is incomplete");
+ const values = this.values.slice();
+ return this.positions
+ ? {
+ mode: "speed",
+ fps: this.fps,
+ values,
+ positions: this.positions.slice(),
+ }
+ : { mode: "source-time", fps: this.fps, values };
+ }
+
+ private knownInterval(localFrame: number): number {
+ const i = Math.floor(localFrame);
+ if (Number.isFinite(localFrame)) this.ensure(this.startFrame + i + 1);
+ if (!Number.isFinite(localFrame) || i < 0 || i + 1 >= this.recorded)
+ throw new Error(
+ `Media signal interval ${localFrame} has not been recorded`,
+ );
+ return i;
+ }
+
+ /** Bounds of the interval starting at boundary `i`, its end exclusive. */
+ private interval(i: number): MediaSourceRange {
+ return sampledMediaSourceRange(this.history(i, i + 2), 0, 1);
+ }
+
+ private history(from: number, to: number): SampledMediaSignal {
+ const values = this.values.subarray(from, to);
+ return this.positions
+ ? {
+ mode: "speed",
+ fps: this.fps,
+ values,
+ positions: this.positions.subarray(from, to),
+ }
+ : { mode: "source-time", fps: this.fps, values };
+ }
+}
diff --git a/packages/core/src/media-time.ts b/packages/core/src/media-time.ts
index 5b88a548..2e39b041 100644
--- a/packages/core/src/media-time.ts
+++ b/packages/core/src/media-time.ts
@@ -1,4 +1,12 @@
import { z } from "zod";
+import {
+ type MediaSignalRecorder,
+ type MediaSourceRange,
+ type SampledMediaSignal,
+ sampledMediaSourceRange,
+ sampledMediaSourceTime,
+} from "./media-time-signal.js";
+import { isSignal, type Signal } from "./signals/index.js";
/** Maps layer-local frames to absolute source seconds. */
export const MediaTimeRemapSchema = z.discriminatedUnion("kind", [
@@ -34,13 +42,67 @@ export const MediaTimeRemapSchema = z.discriminatedUnion("kind", [
export type MediaTimeRemap = z.infer<typeof MediaTimeRemapSchema>;
+/**
+ * Drives source time from a signal on the composition clock: `"speed"` is
+ * playback rate (source seconds per second, integrated from `sourceStartSec`),
+ * `"source-time"` is absolute source seconds. `signal` is a signal instance or
+ * the name of a composition signal. The signal is read at every frame
+ * boundary of the layer, after that frame's hooks have run.
+ */
+export interface MediaTimeSignalRemap {
+ kind: "signal";
+ mode: "speed" | "source-time";
+ signal: Signal<number> | string;
+ sourceStartSec?: number;
+}
+
+export const MediaTimeSignalRemapSchema = z
+ .object({
+ kind: z.literal("signal"),
+ mode: z.enum(["speed", "source-time"]),
+ signal: z.union([
+ z.string().min(1),
+ z.custom<Signal<number>>(
+ isSignal,
+ "signal must be a Signal or the name of a composition signal",
+ ),
+ ]),
+ sourceStartSec: z.number().finite().nonnegative().optional(),
+ })
+ .strict();
+
+/** Every `timeRemap` an author can write: a fixed map or a signal binding. */
+export type MediaTimeRemapInput = MediaTimeRemap | MediaTimeSignalRemap;
+
+export const MediaTimeRemapInputSchema = z.discriminatedUnion("kind", [
+ ...MediaTimeRemapSchema.options,
+ MediaTimeSignalRemapSchema,
+]);
+
+/** A per-render signal history resolved by the media timing scheduler. */
+export type ResolvedMediaTimeRemap =
+ | MediaTimeRemap
+ | { readonly kind: "sampled"; readonly history: SampledMediaSignal }
+ | { readonly kind: "recorded"; readonly recorder: MediaSignalRecorder };
+
+function assertSampledFps(history: SampledMediaSignal, fps: number): void {
+ if (history.fps !== fps)
+ throw new Error(
+ `Media signal history fps ${history.fps} does not match composition fps ${fps}`,
+ );
+}
+
/** Validate the map against its explicit, exclusive layer duration. */
export function validateMediaTimeRemap(
- map: MediaTimeRemap,
+ map: MediaTimeRemapInput,
durationFrames: number,
): string | undefined {
if (!Number.isInteger(durationFrames) || durationFrames <= 0)
return "durationFrames must be a positive integer";
+ if (map.kind === "signal")
+ return map.mode === "source-time" && map.sourceStartSec !== undefined
+ ? "sourceStartSec applies only to speed signals"
+ : undefined;
if (map.kind === "rate") return;
const keys = map.keyframes;
if (
@@ -90,12 +152,23 @@ function shapeTime(
/** Absolute source seconds at a fractional layer-local frame. */
export function mediaSourceTime(
- map: MediaTimeRemap,
+ map: ResolvedMediaTimeRemap,
localFrame: number,
fps: number,
): number {
if (!Number.isFinite(localFrame) || !Number.isFinite(fps) || fps <= 0)
throw new Error("Invalid media time or fps");
+ if (map.kind === "recorded") {
+ if (map.recorder.fps !== fps)
+ throw new Error(
+ "Media signal recorder fps does not match composition fps",
+ );
+ return map.recorder.sourceTime(localFrame);
+ }
+ if (map.kind === "sampled") {
+ assertSampledFps(map.history, fps);
+ return sampledMediaSourceTime(map.history, localFrame);
+ }
if (map.kind === "rate")
return map.sourceStartSec + (localFrame / fps) * map.rate;
const keys = map.keyframes;
@@ -115,19 +188,46 @@ export function mediaSourceTime(
}
/** True only for a constant source-time interval, not a ramp with zero instantaneous velocity. */
-export function mediaSourceIsHold(map: MediaTimeRemap, frame: number): boolean {
+export function mediaSourceIsHold(
+ map: ResolvedMediaTimeRemap,
+ frame: number,
+): boolean {
if (map.kind === "rate") return false;
+ if (map.kind === "recorded") return map.recorder.isHold(frame);
+ if (map.kind === "sampled") {
+ const values = map.history.values;
+ const i = Math.max(0, Math.min(values.length - 2, Math.floor(frame)));
+ return map.history.mode === "speed"
+ ? values[i] === 0 && values[i + 1] === 0
+ : values[i] === values[i + 1];
+ }
const i = segmentIndex(map, frame);
return map.keyframes[i].sourceSec === map.keyframes[i + 1].sourceSec;
}
/** Source seconds per output second at a layer-local frame. */
export function mediaSourceRate(
- map: MediaTimeRemap,
+ map: ResolvedMediaTimeRemap,
frame: number,
fps: number,
): number {
if (map.kind === "rate") return map.rate;
+ if (map.kind === "recorded") {
+ if (map.recorder.fps !== fps)
+ throw new Error(
+ "Media signal recorder fps does not match composition fps",
+ );
+ return map.recorder.rate(frame);
+ }
+ if (map.kind === "sampled") {
+ assertSampledFps(map.history, fps);
+ const values = map.history.values;
+ const i = Math.max(0, Math.min(values.length - 2, Math.floor(frame)));
+ const t = Math.max(0, Math.min(1, frame - i));
+ return map.history.mode === "speed"
+ ? values[i] * (1 - t) + values[i + 1] * t
+ : (values[i + 1] - values[i]) * map.history.fps;
+ }
const i = segmentIndex(map, frame);
const a = map.keyframes[i];
const b = map.keyframes[i + 1];
@@ -145,11 +245,22 @@ export function mediaSourceRate(
/** Bounds over a local-frame interval, including intermediate turns. */
export function mediaSourceRange(
- map: MediaTimeRemap,
+ map: ResolvedMediaTimeRemap,
fromFrame: number,
toFrame: number,
fps: number,
-): { min: number; max: number; maxIncluded: boolean } {
+): MediaSourceRange {
+ if (map.kind === "recorded") {
+ if (map.recorder.fps !== fps)
+ throw new Error(
+ "Media signal recorder fps does not match composition fps",
+ );
+ return map.recorder.range(fromFrame, toFrame);
+ }
+ if (map.kind === "sampled") {
+ assertSampledFps(map.history, fps);
+ return sampledMediaSourceRange(map.history, fromFrame, toFrame);
+ }
const start = Math.min(fromFrame, toFrame),
end = Math.max(fromFrame, toFrame);
const first = mediaSourceTime(map, start, fps);
diff --git a/packages/core/src/signals/index.ts b/packages/core/src/signals/index.ts
index 6494153a..33d9a758 100644
--- a/packages/core/src/signals/index.ts
+++ b/packages/core/src/signals/index.ts
@@ -217,6 +217,7 @@ class FrameArraySignalImpl implements FrameSignal<number> {
public readonly fps: number;
private _samples: ArrayLike<number>;
private _cachedFrame = -1;
+ private _cachedFps = 0;
private _cachedValue = 0;
private _seekedFrame = -1;
private _subscribers = new Set<ComputedImpl<unknown>>();
@@ -248,17 +249,18 @@ class FrameArraySignalImpl implements FrameSignal<number> {
const currentFrame =
ctx?.frame ??
(this._seekedFrame >= 0 ? this._seekedFrame : frameSignal.value);
- if (this._cachedFrame === currentFrame) {
+ const compFps = ctx?.fps ?? this.fps;
+ if (this._cachedFrame === currentFrame && this._cachedFps === compFps) {
return this._cachedValue;
}
this._cachedFrame = currentFrame;
+ this._cachedFps = compFps;
if (!this._samples || this._samples.length === 0) {
this._cachedValue = 0;
return 0;
}
- const compFps = ctx?.fps ?? this.fps;
const targetFrame =
compFps !== this.fps && compFps > 0
? (currentFrame / compFps) * this.fps
@@ -279,6 +281,7 @@ class FrameArraySignalImpl implements FrameSignal<number> {
seekFrame(frame: number): void {
this._seekedFrame = frame;
this._cachedFrame = frame;
+ this._cachedFps = this.fps;
if (!this._samples || this._samples.length === 0) {
this._cachedValue = 0;
} else {
diff --git a/packages/core/src/signals/programmatic.ts b/packages/core/src/signals/programmatic.ts
index 3eb6f4ad..bb3621da 100644
--- a/packages/core/src/signals/programmatic.ts
+++ b/packages/core/src/signals/programmatic.ts
@@ -33,6 +33,7 @@ export class ProgrammaticSignal implements FrameSignal<number> {
private _fps: number;
private _durationFrames?: number;
private _cachedFrame = -1;
+ private _cachedContext?: FrameContext;
private _cachedValue = 0;
private _seekedFrame = -1;
// Holds the computeds that read this signal (registerSignalDependency adds them).
@@ -96,15 +97,12 @@ export class ProgrammaticSignal implements FrameSignal<number> {
ctx?.frame ??
(this._seekedFrame >= 0 ? this._seekedFrame : frameSignal.value);
- if (this._cachedFrame === currentFrame) {
- return this._cachedValue;
- }
-
const fps = ctx?.fps ?? this._fps;
const time = ctx?.time ?? (fps > 0 ? currentFrame / fps : 0);
const duration = ctx?.duration ?? this._durationFrames ?? 100;
const durationMs = ctx?.durationMs ?? (duration / fps) * 1000;
- const progress = duration > 0 ? currentFrame / duration : 0;
+ const progress =
+ ctx?.progress ?? (duration > 0 ? currentFrame / duration : 0);
const deltaTime = ctx?.deltaTime ?? (fps > 0 ? 1 / fps : 0.0416);
const resolvedCtx: FrameContext = {
@@ -116,8 +114,21 @@ export class ProgrammaticSignal implements FrameSignal<number> {
progress,
deltaTime,
};
+ const cached = this._cachedContext;
+ if (
+ this._cachedFrame === currentFrame &&
+ cached &&
+ cached.fps === fps &&
+ cached.time === time &&
+ cached.duration === duration &&
+ cached.durationMs === durationMs &&
+ cached.progress === progress &&
+ cached.deltaTime === deltaTime
+ )
+ return this._cachedValue;
this._cachedFrame = currentFrame;
+ this._cachedContext = resolvedCtx;
this._cachedValue = this._evaluator(resolvedCtx);
return this._cachedValue;
}
diff --git a/packages/framefields/src/clip-vision.test.ts b/packages/framefields/src/clip-vision.test.ts
index 5265f1d8..db15c1a2 100644
--- a/packages/framefields/src/clip-vision.test.ts
+++ b/packages/framefields/src/clip-vision.test.ts
@@ -1,6 +1,12 @@
+import { collectNodeOps } from "@framefields/compositions/program";
+import {
+ type MediaSignalRecorder,
+ type ResolvedMediaTimeRemap,
+ sampleMediaSignal,
+} from "@framefields/core";
import { describe, expect, it } from "vitest";
import { clipVisionFromResults } from "./clip-vision.js";
-import { Composition, Layer } from "./index.js";
+import { Composition, Layer, Signal } from "./index.js";
/** A 160×80 clip: frame f has the left wrist at (20 + 20f, 10) and a 40×40 square silhouette moving right. */
function clip(frames = 4, options = {}) {
@@ -95,6 +101,17 @@ describe("ClipVision", () => {
});
describe("remapped clip vision", () => {
+ it("uses recorded signal history for the same source clock", () => {
+ const history = sampleMediaSignal({
+ mode: "source-time",
+ fps: 24,
+ startFrame: 0,
+ durationFrames: 2,
+ sample: (frame) => frame / 24,
+ });
+ const c = clip(4, { timeRemap: { kind: "sampled", history } });
+ expect(c.person.point("leftWrist", 1).x).toBe(40);
+ });
it("samples reverse motion and covers intermediate turns in a readiness window", () => {
const c = clip(4, {
timeRemap: {
@@ -143,6 +160,30 @@ describe("remapped clip vision", () => {
expect(vision.startFrame).toBe(17);
expect(vision.clipFrame(19)).toBe(25);
});
+ it("shares a layer's live signal time with its render", () => {
+ const comp = new Composition({
+ width: 64,
+ height: 64,
+ fps: 24,
+ durationFrames: 48,
+ });
+ const speed = Signal.state(2);
+ const layer = Layer.video("clip.mp4", {
+ startFrame: 4,
+ durationFrames: 24,
+ timeRemap: { kind: "signal", mode: "speed", signal: speed },
+ });
+ comp.add(layer);
+ const vision = comp.vision(layer);
+ const op = collectNodeOps(comp.toVirtualMedia().children).find(
+ (node) => node.operation.dataType === "Video",
+ )?.operation as unknown as { timeRemap: ResolvedMediaTimeRemap };
+ expect(vision.options.timeRemap).toBe(op.timeRemap);
+ // No look-ahead: the clip's time is known once the clock reaches it.
+ expect(() => vision.clipFrame(5)).toThrow(/not been recorded/);
+ (op.timeRemap as { recorder: MediaSignalRecorder }).recorder.advance(5);
+ expect(vision.clipFrame(5)).toBe(2);
+ });
it("samples the same source clock as the media layer", () => {
const c = clip(4, {
startFrame: 10,
diff --git a/packages/framefields/src/clip-vision.ts b/packages/framefields/src/clip-vision.ts
index 1b52fb97..6cdf28b9 100644
--- a/packages/framefields/src/clip-vision.ts
+++ b/packages/framefields/src/clip-vision.ts
@@ -1,8 +1,8 @@
import {
type FrameSignal,
- type MediaTimeRemap,
mediaSourceRange,
mediaSourceTime,
+ type ResolvedMediaTimeRemap,
Signal,
} from "@framefields/core";
import {
@@ -23,7 +23,7 @@ export interface ClipVisionOptions {
/** Seconds trimmed off the clip's start, as on its media layer (default 0). */
trimStartSec?: number;
/** The media layer's source-time mapping. */
- timeRemap?: MediaTimeRemap;
+ timeRemap?: ResolvedMediaTimeRemap;
/** Media layer duration in composition frames, when known. */
durationFrames?: number;
/** Analysis frame rate (default: the composition's). */
diff --git a/packages/framefields/src/index.ts b/packages/framefields/src/index.ts
index b6b5bf0e..8aa08d63 100644
--- a/packages/framefields/src/index.ts
+++ b/packages/framefields/src/index.ts
@@ -156,6 +156,7 @@ import {
type MatteMode,
type MediaAnimatableProp,
type MediaNode,
+ type MediaSignalResolver,
type Model3DAnimatableProp,
type Model3DNode,
mergeAnimationTracks,
@@ -174,15 +175,19 @@ import {
type MediaMetadata,
type MediaOperation,
type MediaSourceInput,
- type MediaTimeRemap,
type MeshAudioDeformConfig,
ProgrammaticSignal,
type RenderDiagnostic,
+ type ResolvedMediaTimeRemap,
type Signal,
type VirtualMediaData,
} from "@framefields/core";
-export type { MediaTimeRemap } from "@framefields/core";
+export type {
+ MediaTimeRemap,
+ MediaTimeRemapInput,
+ MediaTimeSignalRemap,
+} from "@framefields/core";
import {
analyzeSequence as analyzeVisionFrames,
@@ -272,6 +277,11 @@ import {
Vision,
type VisionProps,
} from "./effects/index.js";
+import {
+ contextAtFrame,
+ MediaSignalTimeline,
+ remapKey,
+} from "./media-signals.js";
import { Particles, type ParticlesOptions } from "./particles.js";
import {
type FrameGridOptions,
@@ -1200,6 +1210,7 @@ export class Composition {
private _prepareHooks: Array<(ctx: FrameContext) => Promise<void> | void> =
[];
private readonly _clipVision = new Map<string, ClipVision>();
+ private _mediaSignals?: MediaSignalTimeline;
private _visionConfig?: VisionProps;
private _visionBundle?: VisionBundle;
private _effects: AnyEffect[] = [];
@@ -1253,7 +1264,11 @@ export class Composition {
? sourceOrLayer
: (sourceOrLayer.src ?? sourceOrLayer.inputHandleId);
if (typeof sourceOrLayer !== "string") {
- const layer = collectNodeOps(this.toVirtualMedia().children).find(
+ // A clip is analysed frame by frame, so its signal time is live.
+ const vm = this.toVirtualMedia({
+ resolveMediaSignal: this.mediaSignalResolver(true),
+ });
+ const layer = collectNodeOps(vm.children).find(
(node) =>
(node.operation as unknown as { id?: string }).id ===
sourceOrLayer.id,
@@ -1265,12 +1280,14 @@ export class Composition {
options = {
startFrame: layer.operation.startFrame ?? sourceOrLayer.startFrame,
trimStartSec: sourceOrLayer.trimStartSec,
- timeRemap: sourceOrLayer.timeRemap,
+ timeRemap: (
+ layer.operation as unknown as { timeRemap?: ResolvedMediaTimeRemap }
+ ).timeRemap,
durationFrames: sourceOrLayer.durationFrames,
...options,
};
}
- const key = `${source}\u0000${JSON.stringify(options)}`;
+ const key = `${source}\u0000${JSON.stringify({ ...options, timeRemap: remapKey(options.timeRemap) })}`;
let clip = this._clipVision.get(key);
if (!clip) {
clip = new ClipVision(source, this.fps, options);
@@ -1299,7 +1316,12 @@ export class Composition {
*/
private linkClipMattes(): void {
if (this._clipVision.size === 0) return;
- let starts: Map<string | undefined, number | undefined> | undefined;
+ let layers:
+ | Map<
+ string | undefined,
+ { startFrame?: number; timeRemap?: ResolvedMediaTimeRemap }
+ >
+ | undefined;
const walk = (node: unknown): void => {
if (!node || typeof node !== "object") return;
const n = node as {
@@ -1310,7 +1332,6 @@ export class Composition {
src?: string;
startFrame?: number;
trimStartSec?: number;
- timeRemap?: MediaTimeRemap;
vision?: unknown;
children?: unknown;
};
@@ -1321,15 +1342,21 @@ export class Composition {
n.vision instanceof Vision &&
!(CLIP_SUBJECT in n.vision)
) {
- starts ??= new Map(
- collectNodeOps(this.toVirtualMedia().children).map((node) => [
- (node.operation as unknown as { id?: string }).id,
- node.operation.startFrame,
- ]),
+ layers ??= new Map(
+ collectNodeOps(this.toVirtualMedia().children).map((node) => {
+ const op = node.operation as unknown as {
+ id?: string;
+ startFrame?: number;
+ timeRemap?: ResolvedMediaTimeRemap;
+ };
+ return [op.id, op];
+ }),
);
+ const layer = layers.get(n.id);
const clip = this.clipForMatte(n.src ?? n.inputHandleId, n.vision, {
- ...n,
- startFrame: starts.get(n.id) ?? n.startFrame,
+ trimStartSec: n.trimStartSec,
+ startFrame: layer?.startFrame ?? n.startFrame,
+ timeRemap: layer?.timeRemap,
});
if (clip) {
clip.keepSubjects();
@@ -1351,7 +1378,7 @@ export class Composition {
layer: {
startFrame?: number;
trimStartSec?: number;
- timeRemap?: MediaTimeRemap;
+ timeRemap?: ResolvedMediaTimeRemap;
},
): ClipVision | undefined {
const mode = v.mode ?? "passthrough";
@@ -1375,7 +1402,7 @@ export class Composition {
clip.source === source &&
clip.startFrame === (layer.startFrame ?? 0) &&
clip.trimStartSec === (layer.trimStartSec ?? 0) &&
- JSON.stringify(o.timeRemap) === JSON.stringify(layer.timeRemap) &&
+ remapKey(o.timeRemap) === remapKey(layer.timeRemap) &&
(o.variant ?? "s") === (v.variant ?? "s") &&
(o.confidence ?? 0.3) === (v.confidence ?? 0.3)
) {
@@ -1488,18 +1515,77 @@ export class Composition {
}
public notifyFrame(ctx: FrameContext): Promise<void> | undefined {
+ const skipped = this._mediaSignals?.framesBefore(ctx.frame) ?? [];
+ if (skipped.length === 0) return this.runFrame(ctx);
+ // Live signal time has no gaps: a seek forward runs the frames it skips.
+ return (async () => {
+ for (const frame of skipped)
+ await this.runFrame(contextAtFrame(ctx, frame));
+ await this.runFrame(ctx);
+ })();
+ }
+
+ private runFrame(ctx: FrameContext): Promise<void> | undefined {
for (const hook of this._frameHooks) {
hook(ctx);
}
+ const signals = this._mediaSignals;
+ signals?.advance(ctx.frame);
if (this._clipVision.size === 0 && this._prepareHooks.length === 0) {
+ signals?.reach(ctx.frame);
return undefined;
}
return (async () => {
await this.prepareVision(ctx.frame);
for (const hook of this._prepareHooks) await hook(ctx);
+ signals?.reach(ctx.frame);
})();
}
+ /**
+ * @internal Runs the frame clock through the last frame while a live
+ * signal `timeRemap` history is incomplete, so audio can be mixed before
+ * any frame renders (the preview does). Compile the composition first.
+ */
+ public async prepareMediaTime(): Promise<void> {
+ const durationMs = this.durationMs ?? (await this.computeDuration());
+ const last = Math.round((durationMs / 1000) * this.fps) - 1;
+ if (!this._mediaSignals?.pending || last < 0) return;
+ await this.notifyFrame(
+ contextAtFrame(
+ {
+ frame: 0,
+ fps: this.fps,
+ time: 0,
+ duration: durationMs / 1000,
+ durationMs,
+ progress: 0,
+ deltaTime: 1 / this.fps,
+ },
+ last,
+ ),
+ );
+ }
+
+ /**
+ * Resolves signal `timeRemap`s to one shared history per layer: sampled
+ * when nothing can change a signal mid-render, recorded live otherwise.
+ */
+ private mediaSignalResolver(
+ live = this.hasFrameHooks(),
+ durationMs = this.durationMs,
+ signals = this.signals,
+ ): MediaSignalResolver {
+ this._mediaSignals ??= new MediaSignalTimeline(this.fps);
+ return this._mediaSignals.resolver({
+ live,
+ compositionDurationFrames: Math.round(
+ ((durationMs ?? 0) / 1000) * this.fps,
+ ),
+ signals,
+ });
+ }
+
private hasFrameHooks(): boolean {
return (
this._frameHooks.length > 0 ||
@@ -2050,11 +2136,18 @@ export class Composition {
public toVirtualMedia(
options: Partial<CompositorToProgramOptions> = {},
): VirtualMediaData {
+ const durationMs = options.durationMs ?? this.durationMs;
+ const signals = options.signals ?? this.signals;
return compositorToProgram(this.toSpec(), {
isVideoMode: true,
fps: this.fps,
- durationMs: options.durationMs ?? this.durationMs,
- signals: options.signals ?? this.signals,
+ durationMs,
+ signals,
+ resolveMediaSignal: this.mediaSignalResolver(
+ undefined,
+ durationMs,
+ signals,
+ ),
...(this._effects.length > 0 && { effects: this._effects }),
...(this._visionConfig && { vision: this._visionConfig }),
...options,
@@ -2070,11 +2163,17 @@ export class Composition {
const durationMs =
options.durationMs ?? this.durationMs ?? (await this.computeDuration());
this.durationMs = durationMs;
+ const signals = options.signals ?? this.signals;
const vm = compositorToProgram(this.toSpec(), {
isVideoMode: true,
fps: this.fps,
durationMs,
- signals: options.signals ?? this.signals,
+ signals,
+ resolveMediaSignal: this.mediaSignalResolver(
+ undefined,
+ durationMs,
+ signals,
+ ),
...(this._effects.length > 0 && { effects: this._effects }),
...(this._visionConfig && { vision: this._visionConfig }),
...options,
@@ -2094,6 +2193,8 @@ export class Composition {
renderer?: HeadlessMediaRenderer;
} = {},
): Promise<Buffer> {
+ // Compile first: the frame's hooks then record its live signal time.
+ const vm = this.toVirtualMedia();
if (this.hasFrameHooks()) {
const targetFrame =
options.frame ??
@@ -2120,7 +2221,7 @@ export class Composition {
});
}
const renderer = options.renderer ?? new HeadlessMediaRenderer();
- return renderer.renderFrame(this.toVirtualMedia(), {
+ return renderer.renderFrame(vm, {
fps: this.fps,
...options,
});
diff --git a/packages/framefields/src/media-signals.ts b/packages/framefields/src/media-signals.ts
new file mode 100644
index 00000000..bb5300f2
--- /dev/null
+++ b/packages/framefields/src/media-signals.ts
@@ -0,0 +1,171 @@
+import {
+ type MediaSignalResolver,
+ mediaSignalSampler,
+ sampleMediaTimeSignal,
+} from "@framefields/compositions/program";
+import {
+ type FrameContext,
+ MediaSignalRecorder,
+ type MediaTimeSignalRemap,
+ type ResolvedMediaTimeRemap,
+} from "@framefields/core";
+
+interface Entry {
+ binding: MediaTimeSignalRemap;
+ live: boolean;
+ compositionFrames: number;
+ resolved: ResolvedMediaTimeRemap;
+}
+
+/**
+ * The timing histories behind a composition's signal `timeRemap`s. Each layer
+ * resolves to one history shared by every compile, so video, audio, vision and
+ * preview read the same source clock. Without frame hooks nothing can change
+ * a signal mid-render, so it is sampled up front. With hooks the signal is
+ * live: each boundary is recorded once, after its frame's hooks, in frame
+ * order, and replayed for seeks and audio.
+ */
+export class MediaSignalTimeline {
+ private readonly entries = new Map<string, Entry>();
+ private reached = -1;
+
+ constructor(private readonly fps: number) {}
+
+ resolver(options: {
+ live: boolean;
+ compositionDurationFrames: number;
+ signals: Record<string, unknown>;
+ }): MediaSignalResolver {
+ const { live, compositionDurationFrames } = options;
+ const clock = {
+ fps: this.fps,
+ compositionDurationFrames,
+ signals: options.signals,
+ };
+ return (binding, layer) => {
+ const key = `${layer.id}\u0000${layer.startFrame}\u0000${layer.durationFrames}`;
+ const cached = this.entries.get(key);
+ if (
+ cached &&
+ cached.live === live &&
+ cached.compositionFrames === compositionDurationFrames &&
+ sameBinding(cached.binding, binding)
+ )
+ return cached.resolved;
+ const resolved: ResolvedMediaTimeRemap = live
+ ? {
+ kind: "recorded",
+ recorder: new MediaSignalRecorder({
+ mode: binding.mode,
+ fps: this.fps,
+ startFrame: layer.startFrame,
+ durationFrames: layer.durationFrames,
+ sourceStartSec: binding.sourceStartSec,
+ sample: mediaSignalSampler(binding, clock),
+ }),
+ }
+ : sampleMediaTimeSignal(binding, layer, clock);
+ this.entries.set(key, {
+ binding,
+ live,
+ compositionFrames: compositionDurationFrames,
+ resolved,
+ });
+ return resolved;
+ };
+ }
+
+ /**
+ * Frames to run, in order, before `frame` so live histories have no gaps:
+ * a seek forward replays the frames it skipped. A seek back replays
+ * nothing; recorded history is reused.
+ */
+ framesBefore(frame: number): number[] {
+ const needs = this.recorders().some(
+ (r) => !r.complete && r.nextFrame < frame,
+ );
+ if (!needs) return [];
+ const frames: number[] = [];
+ for (let f = this.reached + 1; f < frame; f++) frames.push(f);
+ return frames;
+ }
+
+ /** Let live signals be read through `frame` (no look-ahead past the clock). */
+ advance(frame: number): void {
+ for (const r of this.recorders()) r.advance(frame);
+ }
+
+ /**
+ * Record each live boundary at `frame` once its hooks have run. At the
+ * composition's last frame, layers that outlive it hold their last value.
+ */
+ reach(frame: number): void {
+ if (frame > this.reached) this.reached = frame;
+ for (const [, entry] of this.entries) {
+ if (entry.resolved.kind !== "recorded") continue;
+ const recorder = entry.resolved.recorder;
+ recorder.ensure(frame);
+ if (
+ frame >= entry.compositionFrames - 1 &&
+ !recorder.complete &&
+ recorder.nextFrame > recorder.startFrame
+ )
+ recorder.close();
+ }
+ }
+
+ /** True while a live history still has boundaries to record. */
+ get pending(): boolean {
+ return this.recorders().some((r) => !r.complete);
+ }
+
+ private recorders(): MediaSignalRecorder[] {
+ const out: MediaSignalRecorder[] = [];
+ for (const { resolved } of this.entries.values())
+ if (resolved.kind === "recorded") out.push(resolved.recorder);
+ return out;
+ }
+}
+
+function sameBinding(
+ a: MediaTimeSignalRemap,
+ b: MediaTimeSignalRemap,
+): boolean {
+ return (
+ a.signal === b.signal &&
+ a.mode === b.mode &&
+ a.sourceStartSec === b.sourceStartSec
+ );
+}
+
+/** The frame context of another frame on the same clock. */
+export function contextAtFrame(ctx: FrameContext, frame: number): FrameContext {
+ const time = ctx.fps > 0 ? frame / ctx.fps : 0;
+ return {
+ ...ctx,
+ frame,
+ time,
+ progress:
+ ctx.duration > 0 ? Math.max(0, Math.min(1, time / ctx.duration)) : 0,
+ };
+}
+
+const remapIds = new WeakMap<object, number>();
+let nextRemapId = 0;
+
+/**
+ * A key equal for the same source clock: by value for a fixed map, by
+ * identity for a resolved signal history.
+ */
+export function remapKey(map: ResolvedMediaTimeRemap | undefined): string {
+ if (!map) return "";
+ if (map.kind === "rate" || map.kind === "keyframes")
+ return JSON.stringify(map);
+ const target = map.kind === "sampled" ? map.history : map.recorder;
+ let id = remapIds.get(target);
+ if (id === undefined) {
+ id = ++nextRemapId;
+ remapIds.set(target, id);
+ }
+ return `${map.kind}:${id}`;
+}
diff --git a/packages/framefields/src/media-time-render.test.ts b/packages/framefields/src/media-time-render.test.ts
index 8d8b491a..a39ad929 100644
--- a/packages/framefields/src/media-time-render.test.ts
+++ b/packages/framefields/src/media-time-render.test.ts
@@ -3,6 +3,11 @@ import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { decodeAudioSource } from "@framefields/compositions";
+import {
+ MediaSignalRecorder,
+ type ResolvedMediaTimeRemap,
+ sampleMediaSignal,
+} from "@framefields/core";
import { beforeAll, describe, expect, it } from "vitest";
import {
Blur,
@@ -137,6 +142,265 @@ beforeAll(async () => {
});
describe.skipIf(!hasFfmpeg)("remapped video", () => {
+ it("drives video and audio from a public signal timeRemap", async () => {
+ const comp = new Composition({
+ width: 16,
+ height: 16,
+ fps: 4,
+ durationFrames: 4,
+ });
+ comp.add(
+ Layer.video(sourceAv, {
+ width: 16,
+ height: 16,
+ durationFrames: 4,
+ timeRemap: {
+ kind: "signal",
+ mode: "speed",
+ signal: Signal.builder({ type: "custom", fn: () => 0.5 }),
+ },
+ }),
+ );
+ const third = getPixel(
+ await decodePngPixels(await comp.renderFrame({ frame: 2 })),
+ 8,
+ 8,
+ );
+ expect(third.g).toBeGreaterThan(200);
+ const { channels } = await new HeadlessMediaRenderer().renderAudio(
+ comp.toVirtualMedia(),
+ { fps: 4, sampleRate: 48000 },
+ );
+ const original = await decodeAudioSource(sourceAv);
+ if (!original) throw new Error("embedded audio did not decode");
+ expect(channels[0][24000]).toBeCloseTo(
+ Math.tanh(original.channels[0][12000]),
+ 4,
+ );
+ });
+ it("records a live signal after each frame hook, through seeks and audio", async () => {
+ const comp = new Composition({
+ width: 16,
+ height: 16,
+ fps: 4,
+ durationFrames: 4,
+ });
+ const speed = Signal.state(1);
+ const hooked: number[] = [];
+ comp.onRequestFrame((ctx) => {
+ hooked.push(ctx.frame);
+ speed.set(ctx.frame < 2 ? 1 : 0);
+ });
+ comp.addSignal("speed", speed);
+ comp.add(
+ Layer.video(sourceAv, {
+ width: 16,
+ height: 16,
+ durationFrames: 4,
+ timeRemap: { kind: "signal", mode: "speed", signal: "speed" },
+ }),
+ );
+ const at = async (frame: number) =>
+ getPixel(await decodePngPixels(await comp.renderFrame({ frame })), 8, 8);
+ // A seek forward runs the skipped frames' hooks in order: the source
+ // stops at 0.375s, inside the green frame.
+ const last = await at(3);
+ expect(hooked.slice(0, 4)).toEqual([0, 1, 2, 3]);
+ expect(last.g).toBeGreaterThan(200);
+ // Seeking back replays the recorded history.
+ speed.set(5);
+ const first = await at(0);
+ expect(first.r).toBeGreaterThan(200);
+ const { channels } = await new HeadlessMediaRenderer().renderAudio(
+ comp.toVirtualMedia(),
+ { fps: 4, sampleRate: 48000 },
+ );
+ const original = await decodeAudioSource(sourceAv);
+ if (!original) throw new Error("embedded audio did not decode");
+ expect(channels[0][6000]).toBeCloseTo(
+ Math.tanh(original.channels[0][6000]),
+ 4,
+ );
+ // The held tail is silent.
+ expect(channels[0][42000]).toBe(0);
+ });
+ it("finishes a live history before mixing audio ahead of any frame", async () => {
+ const comp = new Composition({
+ width: 16,
+ height: 16,
+ fps: 4,
+ durationFrames: 4,
+ });
+ const speed = Signal.state(0);
+ comp.onRequestFrame((ctx) => speed.set(ctx.frame < 2 ? 1 : 0));
+ comp.add(
+ Layer.video(sourceAv, {
+ width: 16,
+ height: 16,
+ durationFrames: 4,
+ timeRemap: { kind: "signal", mode: "speed", signal: speed },
+ }),
+ );
+ const vm = await comp.toVirtualMediaAsync();
+ const renderer = new HeadlessMediaRenderer();
+ await expect(renderer.renderAudio(vm, { fps: 4 })).rejects.toThrow(
+ /not been recorded/,
+ );
+ await comp.prepareMediaTime();
+ const { channels } = await renderer.renderAudio(vm, {
+ fps: 4,
+ sampleRate: 48000,
+ });
+ const original = await decodeAudioSource(sourceAv);
+ if (!original) throw new Error("embedded audio did not decode");
+ expect(channels[0][6000]).toBeCloseTo(
+ Math.tanh(original.channels[0][6000]),
+ 4,
+ );
+ expect(channels[0][42000]).toBe(0);
+ });
+ it("encodes a live signal timeRemap and validates its bounds as it records", async () => {
+ const make = (rate: number) => {
+ const comp = new Composition({
+ width: 16,
+ height: 16,
+ fps: 4,
+ durationFrames: 8,
+ });
+ const speed = Signal.state(0);
+ comp.onRequestFrame((ctx) => speed.set(ctx.frame < 4 ? rate : 0));
+ comp.add(
+ Layer.video(sourceAv, {
+ width: 16,
+ height: 16,
+ durationFrames: 8,
+ timeRemap: { kind: "signal", mode: "speed", signal: speed },
+ }),
+ );
+ return comp;
+ };
+ await expect(make(2).renderVideo()).rejects.toThrow(/outside/);
+ const result = await make(1).renderVideo();
+ try {
+ const bytes = execFileSync("ffmpeg", [
+ "-v",
+ "error",
+ "-i",
+ result.filePath,
+ "-f",
+ "rawvideo",
+ "-pix_fmt",
+ "rgb24",
+ "pipe:1",
+ ]);
+ const pixel = (frame: number, channel: number) =>
+ bytes[(frame * 16 * 16 + 8 * 16 + 8) * 3 + channel];
+ expect(bytes.length).toBe(8 * 16 * 16 * 3);
+ expect(pixel(0, 0)).toBeGreaterThan(200);
+ expect(pixel(2, 2)).toBeGreaterThan(200);
+ // Speed 1 then 0 from frame 4: the source holds at 0.875s (white).
+ for (const frame of [5, 7])
+ for (const channel of [0, 1, 2])
+ expect(pixel(frame, channel)).toBeGreaterThan(200);
+ } finally {
+ await fs.rm(result.filePath, { force: true });
+ }
+ });
+ it("replays a completed live signal history through video and audio", async () => {
+ const comp = new Composition({
+ width: 16,
+ height: 16,
+ fps: 4,
+ durationFrames: 4,
+ });
+ comp.add(
+ Layer.video(sourceAv, { width: 16, height: 16, durationFrames: 4 }),
+ );
+ const recorder = new MediaSignalRecorder({
+ mode: "speed",
+ fps: 4,
+ startFrame: 0,
+ durationFrames: 4,
+ });
+ for (let frame = 0; frame <= 4; frame++) recorder.record(frame, 0.5);
+ const vm = comp.toVirtualMedia();
+ (
+ vm.children[0].operation as unknown as {
+ timeRemap: ResolvedMediaTimeRemap;
+ }
+ ).timeRemap = { kind: "recorded", recorder };
+ const renderer = new HeadlessMediaRenderer();
+ const frame = getPixel(
+ await decodePngPixels(
+ await renderer.renderFrame(vm, { frame: 2, fps: 4 }),
+ ),
+ 8,
+ 8,
+ );
+ expect(frame.g).toBeGreaterThan(200);
+ const { channels } = await renderer.renderAudio(vm, {
+ fps: 4,
+ sampleRate: 48000,
+ });
+ const original = await decodeAudioSource(sourceAv);
+ if (!original) throw new Error("embedded audio did not decode");
+ expect(channels[0][24000]).toBeCloseTo(
+ Math.tanh(original.channels[0][12000]),
+ 4,
+ );
+ });
+ it("uses recorded signal history for both rendered video and audio", async () => {
+ const comp = new Composition({
+ width: 16,
+ height: 16,
+ fps: 4,
+ durationFrames: 4,
+ });
+ comp.add(
+ Layer.video(sourceAv, { width: 16, height: 16, durationFrames: 4 }),
+ );
+ const vm = comp.toVirtualMedia();
+ const history = sampleMediaSignal({
+ mode: "speed",
+ fps: 4,
+ startFrame: 0,
+ durationFrames: 4,
+ sample: () => 0.5,
+ });
+ (
+ vm.children[0].operation as unknown as {
+ timeRemap: ResolvedMediaTimeRemap;
+ }
+ ).timeRemap = { kind: "sampled", history };
+ const renderer = new HeadlessMediaRenderer();
+ const first = getPixel(
+ await decodePngPixels(
+ await renderer.renderFrame(vm, { frame: 0, fps: 4 }),
+ ),
+ 8,
+ 8,
+ );
+ const third = getPixel(
+ await decodePngPixels(
+ await renderer.renderFrame(vm, { frame: 2, fps: 4 }),
+ ),
+ 8,
+ 8,
+ );
+ expect(first.r).toBeGreaterThan(200);
+ expect(third.g).toBeGreaterThan(200);
+ const { channels } = await renderer.renderAudio(vm, {
+ fps: 4,
+ sampleRate: 48000,
+ });
+ const original = await decodeAudioSource(sourceAv);
+ if (!original) throw new Error("embedded audio did not decode");
+ for (const i of [10000, 24000, 36000])
+ expect(channels[0][i]).toBeCloseTo(
+ Math.tanh(original.channels[0][i / 2]),
+ 4,
+ );
+ });
it("selects reverse variable-frame-rate samples by their presentation timestamps", async () => {
const comp = new Composition({
width: 16,
diff --git a/packages/framefields/src/preview/index.ts b/packages/framefields/src/preview/index.ts
index 26b3c324..f976220d 100644
--- a/packages/framefields/src/preview/index.ts
+++ b/packages/framefields/src/preview/index.ts
@@ -34,6 +34,7 @@ interface PreviewTarget {
toVirtualMedia?(): VirtualMediaData;
toVirtualMediaAsync?(): Promise<VirtualMediaData>;
computeDuration?(): Promise<number>;
+ prepareMediaTime?(): Promise<void>;
fps?: number;
durationMs?: number;
width?: number;
@@ -245,6 +246,8 @@ export async function startPreview(
const vm = target.toVirtualMediaAsync
? await target.toVirtualMediaAsync()
: (target.toVirtualMedia as () => VirtualMediaData)();
+ // Live signal time is recorded frame by frame; finish it before mixing.
+ await target.prepareMediaTime?.();
return renderer.renderAudio(vm, { fps: meta.fps });
})().then(
({ channels, sampleRate }) => {
diff --git a/plugins/framefields/skills/framefields-compose/SKILL.md b/plugins/framefields/skills/framefields-compose/SKILL.md
index 625fba71..8fa2e436 100644
--- a/plugins/framefields/skills/framefields-compose/SKILL.md
+++ b/plugins/framefields/skills/framefields-compose/SKILL.md
@@ -124,6 +124,19 @@ const vision = comp.vision(clip);
// Or: timeRemap: { kind: "rate", sourceStartSec: 3, rate: -0.5 }
```
+A signal can drive the clock instead: `{ kind: "signal", mode: "speed", signal, sourceStartSec }` integrates playback rate (source seconds per second; negative reverses, 0 holds), and `mode: "source-time"` reads absolute source seconds. `signal` is a `Signal` or the name of a composition signal (`comp.addSignal`), read on the composition clock at every layer frame boundary and interpolated between them, so picture, audio and `comp.vision` share one clock.
+
+```ts
+const speed = Signal.state(1);
+comp.onRequestFrame((ctx) => speed.set(ctx.frame < 48 ? 1 : 0.25));
+comp.add(Layer.video("assets/clip.mp4", {
+ durationFrames: 96,
+ timeRemap: { kind: "signal", mode: "speed", signal: speed },
+}));
+```
+
+Without frame hooks (`onRequestFrame`, `onPrepareFrame`, `comp.vision`, physics) the signal is sampled up front. With them it is live: each boundary is recorded once, after that frame's `onRequestFrame` hooks, and never ahead of the frame being rendered. A seek forward runs the skipped frames' hooks in order; a seek back reuses the recorded history. Source bounds are checked as each frame records, and a layer that outlives the composition holds its last value. A signal that reads its own layer's current source time is reported as a circular dependency.
+
Keyframes must start at frame 0, end at `durationFrames`, and increase in frame. `interpolation` belongs to the starting key of a segment: `"linear"` (default), `"ease-in"`, `"ease-out"`, or `"smooth"`. These curves are monotonic and do not overshoot; `smooth` starts and ends at zero speed. Adjacent segments may have different speeds at their join. `comp.vision(layer)` requires the layer to have already been added; clip analysis runs in source order and is sampled on the mapped clock. Effects stay on the composition clock. Source ranges are checked before rendering. Audio uses resampling rather than pitch-preserving time stretch. Reverse video can render more slowly due to source seeking; headless decoding retains at most 48 MiB of cached frames per decoder.
## Animation System (`LayerAnimation`)