Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: make timeline preview play the source spans the export renders (#10866)

Adam Eivy13e5510dd732
diff --git a/client/src/components/media/VideoTimelineLanes.jsx b/client/src/components/media/VideoTimelineLanes.jsx
index 4e839db79e..8aec02874f 100644
--- a/client/src/components/media/VideoTimelineLanes.jsx
+++ b/client/src/components/media/VideoTimelineLanes.jsx
@@ -303,7 +303,7 @@ export const AudioRow = memo(function AudioRow({ track, onAdd }) {
* unambiguous, so this is where a removed bed gets paused; a detached but still
* playing `<audio>` keeps producing sound in some browsers.
*/
-export function BedAudio({ trackKey, src, registry }) {
+export function BedAudio({ trackKey, assetId, src, registry, onDuration }) {
const elRef = useRef(null);
useEffect(() => {
const el = elRef.current;
@@ -313,7 +313,31 @@ export function BedAudio({ trackKey, src, registry }) {
el.pause();
registry.delete(trackKey);
};
- }, [trackKey, registry]);
+ }, [trackKey, registry, src]);
+
+ // Report the file's real length so the preview can resolve this placement the
+ // way the export's probe does (see `resolvePlaybackLanes`). Only a finite,
+ // positive duration counts — a stream reports Infinity and a failed load
+ // never fires `loadedmetadata`, and both must leave the authored span alone.
+ // The listener belongs to this effect run: a changed asset or an unmount
+ // disarms it, so a late load can't attribute its length to another file.
+ useEffect(() => {
+ const el = elRef.current;
+ if (!el || !onDuration) return undefined;
+ let current = true;
+ const report = () => {
+ if (current && Number.isFinite(el.duration) && el.duration > 0) onDuration(assetId, el.duration);
+ };
+ if (el.readyState >= 1) report();
+ el.addEventListener('loadedmetadata', report);
+ return () => {
+ current = false;
+ el.removeEventListener('loadedmetadata', report);
+ };
+ }, [assetId, src, onDuration]);
+
if (!src) return null;
- return <audio ref={elRef} src={src} preload="auto" className="hidden" />;
+ // Keyed by `src` so a different file always mounts a fresh element: its
+ // readyState/duration can never be read back as the previous file's.
+ return <audio key={src} ref={elRef} src={src} preload="auto" className="hidden" />;
}
diff --git a/client/src/lib/README.md b/client/src/lib/README.md
index ecdcb27b5a..f7f79b96af 100644
--- a/client/src/lib/README.md
+++ b/client/src/lib/README.md
@@ -294,7 +294,7 @@ grep -i "what you want to do" client/src/lib/README.md
| `universeGraphLayout.js` | 2D positioning for the Universe Builder graph canvas: `seedPositions` (deterministic phyllotaxis), `stepLayout`/`settleLayout` (force sim with per-kind collision radius), `applyAnchors` (radial rings by kind, timeline lanes by first appearance), `computeFit`, `toScreen`/`toWorld`, `hitTest`, `radiusFor`, `GRAPH_LAYOUTS`. Distinct from `graphSimulation.js`, which settles the Brain/CoS graphs in 3D with no anchors, collision or screen fit. |
| `universeGraphPoster.js` | Print-ready infographic posters drawn from an `indexGraph` result: `POSTER_LAYOUTS` (cast roster, character dossier, place atlas, timeline strip, relationship web), `POSTER_SIZES`, `POSTER_THEMES`, `posterDimensions`, and `renderPoster(canvas, opts)` — the same call drives the on-screen preview (scale 1) and the 2× PNG download, so they can't diverge. |
| `universeBuilderCounts.js` | Pure prompt-count helpers for the Universe Builder render buttons (#2374), mirroring the server's compile/skip rules so inline counts match what `/render` will enqueue: `totalVariationCount`, `canonEntryHasContent`, `countCanonWithContent`, `renderPromptCount`, `scopedPromptCount`. The server `compilePrompts`/`synthesizeCanonPrompt` copy is authoritative — keep in sync. |
-| `videoTimelineModel.js` | Pure model helpers for the layered video timeline, mirroring `server/services/videoTimeline/segments.js` so the browser preview and the ffmpeg export make the same cut/geometry/mix decisions: `segmentDuration`/`timelineDuration`/`findSegmentAt` (project-time ↔ segment mapping), `fadeMultiplier` (linear, matching ffmpeg's default `tri` curve), `overlayOpacityAt`, `audioTrackStateAt`, `segmentVolumeAt` (clipVolume × segment volume × fade — the product the export writes into each segment audio chain, so the preview auditions the mix that renders), `assetUrl(assetKind, assetFile)`, the edit clamps `clampTrim` (keeps ≥ 1 frame, matching the server's CLIP_TOO_SHORT floor) and `fitFadePatch` (shrinks an over-long fade pair proportionally instead of letting the PATCH 400), the index summary `projectSummary(project, thumbnailFor)` (block count / duration / card thumbnail — reads the `segments` lane, NEVER the derived `clips` mirror, which omits stills), and the save helpers `stripKey`/`withKeys`/`timelinePatch`. Change a rule here and in the server module in the same commit. Fade fitting re-exports the shared `server/lib/videoTimelineFades.js` leaf. |
+| `videoTimelineModel.js` | Pure model helpers for the layered video timeline, mirroring `server/services/videoTimeline/segments.js` so the browser preview and the ffmpeg export make the same cut/geometry/mix decisions: `segmentDuration`/`timelineDuration`/`findSegmentAt` (project-time ↔ segment mapping), `fadeMultiplier` (linear, matching ffmpeg's default `tri` curve), `overlayOpacityAt`, `audioTrackStateAt`, `segmentVolumeAt` (clipVolume × segment volume × fade — the product the export writes into each segment audio chain, so the preview auditions the mix that renders), `assetUrl(assetKind, assetFile)`, the edit clamps `clampTrim` (keeps ≥ 1 frame, matching the server's CLIP_TOO_SHORT floor) and `fitFadePatch` (shrinks an over-long fade pair proportionally instead of letting the PATCH 400), the index summary `projectSummary(project, thumbnailFor)` (block count / duration / card thumbnail — reads the `segments` lane, NEVER the derived `clips` mirror, which omits stills), and the save helpers `stripKey`/`withKeys`/`timelinePatch`. Change a rule here and in the server module in the same commit. Fade fitting re-exports the shared `server/lib/videoTimelineFades.js` leaf, and the source-span rules come from `server/lib/videoTimelineSourceSpans.js`: `resolvePlaybackLanes(lanes, { sourceDurationFor, bedDurationFor })` turns the authored lanes into what the export renders (clip trims clamped to the history clip length, beds to their measured file length, fades refit; an unknown length keeps the authored span), `projectSummary` takes an optional `sourceDurationFor` so the index total agrees. |
| `wrSceneCursor.js` | Resolve which script scene the editor caret sits in (`sceneAtCursor`, `sceneAnchorIndex`) — inverse of WorkEditor's jump-to-scene text search; drives the live render preview's "scene at cursor" target. |
| `writingGuide.js` | Canonical Writers Room reference data + craft principles rendered by the Guide page (`/writers-room/guide`): `WRITING_LENGTH_TARGETS` (microfiction→novel word/char bands), `BOOK_LENGTH_ESTIMATES` (page-based), `WRITING_PRINCIPLES`, `PLANNED_ANALYSES` (e.g. the emotional-roadmap evaluator), and `classifyByWordCount(n)` for labelling a draft's length. Future word-count gauges / length checks read from here so targets don't drift from the docs. |
| `universeMarkdownFilename.js` | `slugifyUniverseName` / `universeMarkdownFilename` — client-side safe filename helpers for Universe Markdown world-bible downloads, kept in step with the server attachment name. |
diff --git a/client/src/lib/videoTimelineModel.js b/client/src/lib/videoTimelineModel.js
index eddcb4bc7b..93069ba181 100644
--- a/client/src/lib/videoTimelineModel.js
+++ b/client/src/lib/videoTimelineModel.js
@@ -6,7 +6,11 @@
* point of the lane model is that what the editor shows is what renders. When
* a rule changes on one side (fade curve, overlay placement, asset kinds),
* change it on the other in the same commit. Fade fitting is shared through
- * server/lib/videoTimelineFades.js rather than mirrored.
+ * server/lib/videoTimelineFades.js rather than mirrored, and so are the
+ * source-span rules (server/lib/videoTimelineSourceSpans.js): the export clamps
+ * a clip trim to the history clip's real length and a bed to its file's real
+ * length, so the preview plays `resolvePlaybackLanes()` — authored lanes run
+ * through those same rules — never the authored values directly.
*
* No React, no I/O — safe to import from the page, its child components, and
* tests alike.
@@ -15,7 +19,15 @@
import { clamp } from '../utils/formatters';
import { fitFades } from '../../../server/lib/videoTimelineFades.js';
+import {
+ historySourceDuration,
+ resolveClipSpan,
+ resolveStillFades,
+ resolveAudioSpan,
+} from '../../../server/lib/videoTimelineSourceSpans.js';
+
export { fitFades } from '../../../server/lib/videoTimelineFades.js';
+export { historySourceDuration };
export const IMAGE_ASSET_KINDS = ['images', 'video-thumbnails'];
export const AUDIO_ASSET_KINDS = ['audio', 'music'];
@@ -44,6 +56,55 @@ export const segmentDuration = (segment) => {
export const timelineDuration = (segments) => (segments || []).reduce((sum, s) => sum + segmentDuration(s), 0);
+const sameFades = (entry, fades) => (entry.fadeInSec || 0) === fades.fadeInSec && (entry.fadeOutSec || 0) === fades.fadeOutSec;
+
+/**
+ * The segment as the export will play it: a clip's trim clamped to the history
+ * clip's real length (`sourceDurationSec`, null when unknown) and its fades
+ * refit; a still's fades refit against its hold. Returns the SAME object when
+ * nothing had to move, so memoized blocks keep their identity.
+ */
+export const resolvePlaybackSegment = (segment, sourceDurationSec) => {
+ if (!segment) return segment;
+ if (segment.type === 'still') {
+ const fades = resolveStillFades(segment);
+ return sameFades(segment, fades) ? segment : { ...segment, ...fades };
+ }
+ const { inSec, outSec, fadeInSec, fadeOutSec } = resolveClipSpan(segment, sourceDurationSec);
+ if (outSec === segment.outSec && inSec === segment.inSec && sameFades(segment, { fadeInSec, fadeOutSec })) return segment;
+ return { ...segment, inSec, outSec, fadeInSec, fadeOutSec };
+};
+
+/** An audio bed as the export will play it, given its file's real length (null when unknown). */
+export const resolvePlaybackTrack = (track, fileDurationSec) => {
+ const span = resolveAudioSpan(track, fileDurationSec);
+ const same = span.offsetSec === (track.offsetSec || 0)
+ && span.durationSec === (track.durationSec || 0)
+ && sameFades(track, span);
+ return same ? track : { ...track, ...span };
+};
+
+/** Identity of a bed's audio file — what a loaded-duration report is keyed by. */
+export const bedAssetId = (track) => `${track.assetKind}/${track.assetFile}`;
+
+/**
+ * Authored lanes → the lanes the preview plays. `sourceDurationFor(clipId)` and
+ * `bedDurationFor(track)` return a known length in seconds or null; an unknown
+ * length leaves the authored span alone. Entries keep their `_key`s and lane
+ * positions, so selection and the dnd identity carry straight over. The authored
+ * lanes remain what the editor edits and saves — this is a view of them.
+ */
+export const resolvePlaybackLanes = ({ segments, audio }, { sourceDurationFor, bedDurationFor }) => ({
+ segments: (segments || []).map((segment) => resolvePlaybackSegment(
+ segment,
+ segment.type === 'still' ? null : sourceDurationFor(segment.clipId),
+ )),
+ audio: {
+ ...audio,
+ tracks: (audio?.tracks || []).map((track) => resolvePlaybackTrack(track, bedDurationFor(track))),
+ },
+});
+
/**
* Map project-time `t` to the (segmentIndex, withinSegmentSec) pair the
* preview needs. The strict `t < acc + dur` comparison falls through on exact
@@ -231,14 +292,21 @@ export const segmentVolumeAt = (segment, clipVolume, within) => {
* carries only clip segments, so a project built from stills would report zero
* blocks and zero seconds while playing back perfectly in the editor. A v1
* project that predates the lane falls back to `clips`, which for it IS the
- * lane. `thumbnailFor(clipId)` resolves a video clip's thumbnail filename.
+ * lane. `thumbnailFor(clipId)` resolves a video clip's thumbnail filename;
+ * the optional `sourceDurationFor(clipId)` (seconds or null) makes the total
+ * source-aware, matching what the editor previews and the export renders.
*/
-export const projectSummary = (project, thumbnailFor) => {
+export const projectSummary = (project, thumbnailFor, sourceDurationFor = null) => {
const segments = Array.isArray(project?.segments) ? project.segments : (project?.clips || []);
let totalSec = 0;
let firstThumb = null;
for (const segment of segments) {
- totalSec += segmentDuration(segment);
+ // With history durations available, count what the export will render
+ // rather than the trim as it was authored.
+ const playable = sourceDurationFor && segment.type !== 'still'
+ ? resolvePlaybackSegment(segment, sourceDurationFor(segment.clipId))
+ : segment;
+ totalSec += segmentDuration(playable);
if (firstThumb) continue;
firstThumb = segment.type === 'still'
? assetUrl(segment.assetKind, segment.assetFile)
diff --git a/client/src/lib/videoTimelineModel.test.js b/client/src/lib/videoTimelineModel.test.js
index 09ba82a2c1..4b80a0e342 100644
--- a/client/src/lib/videoTimelineModel.test.js
+++ b/client/src/lib/videoTimelineModel.test.js
@@ -19,6 +19,7 @@ import {
segmentVolumeAt,
projectSummary,
canvasAspectRatio,
+ resolvePlaybackLanes,
} from './videoTimelineModel';
const clip = (inSec, outSec) => ({ type: 'clip', clipId: 'c', inSec, outSec });
@@ -247,6 +248,19 @@ describe('projectSummary — the index must count the lane, not the mirror', ()
expect(summary).toMatchObject({ blockCount: 2, totalSec: 5 });
});
+ it('counts the source-clamped trim when history durations are supplied, like the export', () => {
+ const project = {
+ segments: [
+ { type: 'clip', clipId: 'c1', inSec: 0, outSec: 8 },
+ { type: 'still', assetKind: 'images', assetFile: 'plate.png', durationSec: 3 },
+ ],
+ };
+ expect(projectSummary(project, thumbFor, () => 1)).toMatchObject({ totalSec: 4 });
+ // An unknown source length keeps the authored trim.
+ expect(projectSummary(project, thumbFor, () => null)).toMatchObject({ totalSec: 11 });
+ expect(projectSummary(project, thumbFor)).toMatchObject({ totalSec: 11 });
+ });
+
it('does not report a stills-only project as empty', () => {
const summary = projectSummary({
segments: [{ type: 'still', assetKind: 'images', assetFile: 'plate.png', durationSec: 4 }],
@@ -344,3 +358,44 @@ describe('shared timeline fade contract', () => {
expect(previewFitFades).toBe(sharedFitFades);
});
});
+
+describe('resolvePlaybackLanes — the preview plays what the export renders', () => {
+ const lanes = {
+ segments: [
+ { _key: 's0', type: 'clip', clipId: 'c1', inSec: 0, outSec: 8, fadeInSec: 3, fadeOutSec: 3, volume: 1 },
+ { _key: 's1', type: 'clip', clipId: 'c2', inSec: 0, outSec: 4, fadeInSec: 0, fadeOutSec: 0, volume: 1 },
+ ],
+ audio: {
+ clipVolume: 1,
+ tracks: [{ _key: 'b0', assetKind: 'music', assetFile: 'bed.mp3', startSec: 0, offsetSec: 0, durationSec: 10, volume: 1, fadeInSec: 4, fadeOutSec: 4 }],
+ },
+ };
+ const resolve = (sources = {}, beds = {}) => resolvePlaybackLanes(lanes, {
+ sourceDurationFor: (id) => sources[id] ?? null,
+ bedDurationFor: (track) => beds[track.assetFile] ?? null,
+ });
+
+ it('matches the export for a clip shorter than its trim (resolveTimeline: 1s, fades 0.5s/0.5s)', () => {
+ const { segments } = resolve({ c1: 1 });
+ expect(timelineDuration(segments)).toBe(5);
+ // The second cut starts at 1s, not 8s.
+ expect(findSegmentAt(segments, 1.2)).toMatchObject({ index: 1, startAtProj: 1 });
+ expect(segments[0]).toMatchObject({ outSec: 1, fadeInSec: 0.5, fadeOutSec: 0.5, _key: 's0' });
+ expect(segmentVolumeAt(segments[0], 1, 0.5)).toBe(1);
+ });
+
+ it('matches the export for a short bed file (resolveTimeline: 2s, fades 1s/1s)', () => {
+ const [track] = resolve({}, { 'bed.mp3': 2 }).audio.tracks;
+ expect(track).toMatchObject({ offsetSec: 0, durationSec: 2, fadeInSec: 1, fadeOutSec: 1, _key: 'b0' });
+ expect(audioTrackStateAt(track, 0.5).volume).toBeCloseTo(0.5);
+ expect(audioTrackStateAt(track, 2).active).toBe(false);
+ });
+
+ it('keeps the authored spans, by identity, when no source length is known', () => {
+ const resolved = resolve();
+ expect(resolved.segments[1]).toBe(lanes.segments[1]);
+ // Fades already fit the authored span, so nothing moves.
+ expect(resolved.segments[0]).toMatchObject({ outSec: 8, fadeInSec: 3, fadeOutSec: 3 });
+ expect(resolved.audio.tracks[0]).toBe(lanes.audio.tracks[0]);
+ });
+});
diff --git a/client/src/pages/VideoTimeline.jsx b/client/src/pages/VideoTimeline.jsx
index 6f31a9534e..e65480315d 100644
--- a/client/src/pages/VideoTimeline.jsx
+++ b/client/src/pages/VideoTimeline.jsx
@@ -5,7 +5,7 @@ import PageSkeleton from '../components/ui/PageSkeleton';
import toast from '../components/ui/Toast';
import * as api from '../services/api';
import { formatDurationSec } from '../utils/formatters';
-import { projectSummary } from '../lib/videoTimelineModel';
+import { projectSummary, historySourceDuration } from '../lib/videoTimelineModel';
export default function VideoTimeline() {
const navigate = useNavigate();
@@ -54,7 +54,8 @@ export default function VideoTimeline() {
const projectStats = useMemo(() => {
const historyMap = new Map(history.map((h) => [h.id, h]));
const thumbnailFor = (clipId) => historyMap.get(clipId)?.thumbnail;
- return new Map(projects.map((p) => [p.id, projectSummary(p, thumbnailFor)]));
+ const sourceDurationFor = (clipId) => historySourceDuration(historyMap.get(clipId));
+ return new Map(projects.map((p) => [p.id, projectSummary(p, thumbnailFor, sourceDurationFor)]));
}, [projects, history]);
return (
diff --git a/client/src/pages/VideoTimelineEditor.jsx b/client/src/pages/VideoTimelineEditor.jsx
index bfb9fdeaa6..8f34d7ae98 100644
--- a/client/src/pages/VideoTimelineEditor.jsx
+++ b/client/src/pages/VideoTimelineEditor.jsx
@@ -39,6 +39,9 @@ import {
overlayOpacityAt,
audioTrackStateAt,
segmentVolumeAt,
+ resolvePlaybackLanes,
+ historySourceDuration,
+ bedAssetId,
clampTrim,
fitFadePatch,
timelinePatch,
@@ -118,6 +121,11 @@ function TimelineProjectEditor({ projectId }) {
const [musicTracks, setMusicTracks] = useState([]);
// Which catalogues have actually been fetched — see knownAbsent below.
const [loaded, setLoaded] = useState({ clips: false, images: false, music: false });
+ // Real lengths the browser measured for the beds' audio files, keyed by asset
+ // identity (`music/<file>`). The preview resolves each placement against its
+ // file's length exactly as the export's probe does; an asset with no entry
+ // (metadata not loaded yet, or the load failed) keeps its authored span.
+ const [bedDurations, setBedDurations] = useState({});
const [loading, setLoading] = useState(true);
const [error, setError] = useState(null);
// The three lanes live in ONE state object so a save always ships a
@@ -282,7 +290,19 @@ function TimelineProjectEditor({ projectId }) {
const isOverlayMissing = useCallback((ov) => knownAbsent(ov.assetKind, ov.assetFile), [knownAbsent]);
const isBedMissing = useCallback((tr) => knownAbsent(tr.assetKind, tr.assetFile), [knownAbsent]);
- const total = useMemo(() => timelineDuration(segments), [segments]);
+ // What the export will actually render: the authored lanes resolved against
+ // the history clips' real lengths and the beds' measured file lengths, by the
+ // same rules `resolveTimeline` applies server-side. Playback, block lengths,
+ // fades and overlay timing all read THESE; the inspector and the save path
+ // keep reading the authored `lanes`, so nothing probe-derived is persisted.
+ const playback = useMemo(() => resolvePlaybackLanes(lanes, {
+ sourceDurationFor: (clipId) => historySourceDuration(historyMap.get(clipId)),
+ bedDurationFor: (track) => bedDurations[bedAssetId(track)] ?? null,
+ }), [lanes, historyMap, bedDurations]);
+ const playSegments = playback.segments;
+ const playTracks = playback.audio.tracks;
+
+ const total = useMemo(() => timelineDuration(playSegments), [playSegments]);
const canvasAspect = useMemo(
() => canvasAspectRatio(segments, (clipId) => historyMap.get(clipId)),
[segments, historyMap],
@@ -518,10 +538,10 @@ function TimelineProjectEditor({ projectId }) {
}, [playing, total]);
const { index: activeIndex, within: activeWithin } = useMemo(
- () => (segments.length > 0 ? findSegmentAt(segments, t) : { index: -1, within: 0 }),
- [segments, t],
+ () => (playSegments.length > 0 ? findSegmentAt(playSegments, t) : { index: -1, within: 0 }),
+ [playSegments, t],
);
- const activeSegment = activeIndex >= 0 ? segments[activeIndex] : null;
+ const activeSegment = activeIndex >= 0 ? playSegments[activeIndex] : null;
// `within` advances every frame; the media-sync effects read it from a ref so
// they re-run on a segment CHANGE rather than once per animation frame.
const withinRef = useRef(0);
@@ -594,7 +614,7 @@ function TimelineProjectEditor({ projectId }) {
// Drive the bed <audio> elements from the same playhead the export mixes
// against, so what the user hears while scrubbing is what amix will produce.
useEffect(() => {
- for (const track of audio.tracks) {
+ for (const track of playTracks) {
const el = bedRefs.current.get(track._key);
if (!el) continue;
const state = audioTrackStateAt(track, t);
@@ -612,7 +632,7 @@ function TimelineProjectEditor({ projectId }) {
if (playing && el.paused) el.play().catch(() => {});
if (!playing && !el.paused) el.pause();
}
- }, [audio.tracks, t, playing, muted]);
+ }, [playTracks, t, playing, muted]);
// Stop every bed when the editor unmounts — a detached <audio> that was
// playing keeps producing sound in some browsers.
@@ -740,6 +760,24 @@ function TimelineProjectEditor({ projectId }) {
overlay: ov,
})), [overlays]);
+ // BedAudio reports a file's measured length here. Identical reports bail out
+ // so a re-fired `loadedmetadata` doesn't re-derive the playback lanes.
+ const reportBedDuration = useCallback((assetId, seconds) => {
+ setBedDurations((prev) => (prev[assetId] === seconds ? prev : { ...prev, [assetId]: seconds }));
+ }, []);
+ // Forget a file's length once no bed uses it: a removed bed's measurement must
+ // not resolve a later placement before that placement has loaded its own.
+ useEffect(() => {
+ const used = new Set(audio.tracks.map(bedAssetId));
+ setBedDurations((prev) => {
+ const stale = Object.keys(prev).filter((id) => !used.has(id));
+ if (stale.length === 0) return prev;
+ const next = { ...prev };
+ for (const id of stale) delete next[id];
+ return next;
+ });
+ }, [audio.tracks]);
+
const selectSegment = useCallback((key) => setSelection({ lane: 'segment', key }), []);
const removeSegment = useCallback((key) => removeFromLane('segment', key), [removeFromLane]);
const selectOverlay = useCallback((key) => setSelection({ lane: 'overlay', key }), []);
@@ -984,8 +1022,10 @@ function TimelineProjectEditor({ projectId }) {
<BedAudio
key={track._key}
trackKey={track._key}
+ assetId={bedAssetId(track)}
src={assetUrl(track.assetKind, track.assetFile)}
registry={bedRefs.current}
+ onDuration={reportBedDuration}
/>
))}
@@ -1058,7 +1098,7 @@ function TimelineProjectEditor({ projectId }) {
>
<SortableContext items={segmentKeys} strategy={horizontalListSortingStrategy}>
<div className="flex gap-1 items-stretch min-w-min py-1">
- {segments.map((segment) => (
+ {playSegments.map((segment) => (
<TimelineBlock
key={segment._key}
clip={segment}
@@ -1092,7 +1132,7 @@ function TimelineProjectEditor({ projectId }) {
<FloatingLane
title="Audio"
- entries={audio.tracks}
+ entries={playTracks}
emptyHint="Add a bed from the Audio tab"
tone="bg-port-success/15 border-port-success/40"
labelOf={bedLabel}
diff --git a/client/src/pages/VideoTimelineEditor.test.jsx b/client/src/pages/VideoTimelineEditor.test.jsx
index a90a21b1ca..d8fa56a807 100644
--- a/client/src/pages/VideoTimelineEditor.test.jsx
+++ b/client/src/pages/VideoTimelineEditor.test.jsx
@@ -472,3 +472,111 @@ describe('lane caps', () => {
await waitFor(() => expect(screen.getByRole('button', { name: 'Remove plate.png from timeline' })).toBeInTheDocument());
});
});
+
+// The saved lanes hold what the author ASKED for; the export resolves them
+// against the real sources (a shortened history clip, a short music file). The
+// preview must play that resolved timeline, or its cuts, fades and overlays run
+// on a different clock than the render (#10866).
+describe('source-aware playback', () => {
+ const seekTo = (seconds) => fireEvent.change(screen.getByRole('slider', { name: 'Playhead position' }), { target: { value: String(seconds) } });
+
+ it('advances past a clip whose source is shorter than its authored trim, matching the export', async () => {
+ // Authored 0-8s with 3s+3s fades, but the history entry is 1s long: the
+ // export renders 1s with the fades refit to 0.5s+0.5s.
+ api.history = [{ ...api.history[0], numFrames: 24, fps: 24 }];
+ api.project = project({
+ segments: [{ ...clipSegment, outSec: 8, fadeInSec: 3, fadeOutSec: 3 }, stillSegment],
+ overlays: [{ ...overlayEntry, startSec: 0, durationSec: 10, fadeOutSec: 2 }],
+ });
+ await renderEditor();
+
+ // Total is 1s clip + 3s still, not 8 + 3.
+ expect(screen.getByRole('slider', { name: 'Playhead position' })).toHaveAttribute('max', '4');
+
+ // Half a second in, the refit fade is already at full level: no scrim.
+ seekTo(0.5);
+ expect(screen.queryByTestId('fade-scrim')).not.toBeInTheDocument();
+
+ // The next cut lands at 1s, so the still is up at 1.5s.
+ const previewStill = () => document.querySelector('img.object-contain[src="/data/images/plate.png"]');
+ expect(previewStill()).toBeNull();
+ seekTo(1.5);
+ expect(previewStill()).not.toBeNull();
+
+ // The overlay's 10s window is clamped to the 4s timeline, so its fade-out
+ // starts at 2s and is half spent at 3s.
+ seekTo(3);
+ expect(Number(screen.getByTestId('overlay-preview').style.opacity)).toBeCloseTo(0.5);
+ });
+
+ it('keeps the authored spans when the history clip has no frame metadata', async () => {
+ api.history = [{ ...api.history[0], numFrames: undefined }];
+ api.project = project({ segments: [{ ...clipSegment, outSec: 8, fadeInSec: 3 }] });
+ await renderEditor();
+ expect(screen.getByRole('slider', { name: 'Playhead position' })).toHaveAttribute('max', '8');
+ });
+
+ describe('bed audio', () => {
+ const shortBed = { ...bedEntry, durationSec: 10, fadeInSec: 4, fadeOutSec: 4 };
+ // The lane block is the one carrying an inline width; a selected bed's
+ // inspector repeats the filename.
+ const bedBlock = () => screen.getAllByText('bed.mp3').map((node) => node.parentElement).find((node) => node.style.width);
+ const reportBedLength = (el, seconds) => {
+ Object.defineProperty(el, 'duration', { configurable: true, value: seconds });
+ act(() => { el.dispatchEvent(new Event('loadedmetadata')); });
+ };
+
+ it('plays a short file on the fade envelope the export builds once its length loads', async () => {
+ api.project = project({ segments: [clipSegment], audio: { clipVolume: 1, tracks: [shortBed] } });
+ const { container } = renderEditorRoute();
+ await awaitPageLoaded('Loading timeline project');
+ const el = container.querySelector('audio');
+
+ // Until the length is known the authored placement stands.
+ expect(bedBlock().style.width).toBe('600px');
+
+ reportBedLength(el, 2);
+ expect(bedBlock().style.width).toBe('120px');
+
+ // 2s file, 4s+4s authored fades refit to 1s+1s: gain is 0.5 at 0.5s.
+ seekTo(0.5);
+ await waitFor(() => expect(el.volume).toBeCloseTo(0.5));
+
+ // Past the end of the file the bed is inactive.
+ seekTo(3);
+ await waitFor(() => expect(el.volume).toBe(0));
+ });
+
+ it('ignores a length that is not finite and never persists a measured one', async () => {
+ api.project = project({ segments: [clipSegment], audio: { clipVolume: 1, tracks: [shortBed] } });
+ const { container, unmount } = renderEditorRoute();
+ await awaitPageLoaded('Loading timeline project');
+ const el = container.querySelector('audio');
+
+ reportBedLength(el, Infinity);
+ expect(bedBlock().style.width).toBe('600px');
+
+ reportBedLength(el, 2);
+ unmount();
+ // Nothing was edited, so nothing is saved; the measured span is a view.
+ expect(api.updateTimelineProject).not.toHaveBeenCalled();
+ });
+
+ it('forgets a removed bed so a later placement of the file starts from its own span', async () => {
+ api.project = project({ segments: [clipSegment], audio: { clipVolume: 1, tracks: [shortBed] } });
+ const { container } = renderEditorRoute();
+ await awaitPageLoaded('Loading timeline project');
+ reportBedLength(container.querySelector('audio'), 2);
+ expect(bedBlock().style.width).toBe('120px');
+
+ fireEvent.click(screen.getByRole('button', { name: 'Remove bed.mp3 from timeline' }));
+ expect(container.querySelector('audio')).toBeNull();
+
+ fireEvent.click(screen.getByRole('tab', { name: /Audio/ }));
+ fireEvent.click(screen.getByRole('button', { name: 'Add' }));
+ // The fresh placement is the 4s default (the timeline is 4s) until ITS
+ // element reports a length — the earlier measurement must not leak in.
+ expect(bedBlock().style.width).toBe('240px');
+ });
+ });
+});
diff --git a/server/lib/README.md b/server/lib/README.md
index f8d09ea38b..50b5744dc2 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -787,6 +787,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `providerTypes.js` | Pure shared `isCliProvider`, `isTuiProvider`, `isApiProvider`, `isProcessProvider`, and type-gated `isClaudeHarnessProvider` predicates. |
| `videoTimelineFades.js` | Browser-safe `fitFades()` proportionally fits a fade pair to its visible duration. Shared by timeline normalization/export and the editor preview/trim controls; legacy import paths re-export it. |
+| `videoTimelineSourceSpans.js` | Browser-safe source-span resolution for the video timeline: `resolveClipSpan` (trim clamped to the history clip length, fades refit), `resolveStillFades`, `resolveAudioSpan` (bed offset/length/fades fit to the real file length; an unknown length keeps the authored slice), `historySourceDuration`, `MIN_MEDIA_SEC`. One implementation for the ffmpeg export (`resolveTimeline`) and the editor preview, so both play the same timeline. |
| `voiceEcho.js` | Pure voice-echo detection helpers — tokenization, trigram building, thresholds shared by server and client echo gates. |
| `voiceEngines.js` | Shared TTS engine IDs, display metadata, supported-engine set, persisted configuration keys, and retired TTS config migration. |
| `voiceModelAssets.js` | Completion contract for downloaded voice model assets (Whisper `.bin`, Piper `.onnx` + `.onnx.json`). `inspectVoiceAsset(kind, path)` is the read-only readiness check (`verified` / `unverified` / `incomplete` / `missing` — stat plus one small JSON read, no hashing or network); `verifyVoiceAssetHashes` and `writeVoiceAssetReceipt` are the script-side halves that re-hash and record the `.portos-complete.json` receipt after promotion. `isManagedVoiceAsset` separates setup-script-owned files from user-supplied ones. Shared by `voice/bootstrap.js`, `voice/health.js` and `scripts/voice-asset.js`. |
diff --git a/server/lib/index.js b/server/lib/index.js
index d366bc35de..540f9e243e 100644
--- a/server/lib/index.js
+++ b/server/lib/index.js
@@ -755,6 +755,7 @@ export * from './creativeDirectorVideoCompiler.js';
export * from './providerTypes.js';
export * from './notificationTypes.js';
export * from './videoTimelineFades.js';
+export * from './videoTimelineSourceSpans.js';
// `tokenize` collides with `bm25.js`'s own tokenizer — namespaced like `runners`.
export * as voiceEcho from './voiceEcho.js';
export * from './voiceEngines.js';
diff --git a/server/lib/videoTimelineSourceSpans.js b/server/lib/videoTimelineSourceSpans.js
new file mode 100644
index 0000000000..fe94e65cc5
--- /dev/null
+++ b/server/lib/videoTimelineSourceSpans.js
@@ -0,0 +1,78 @@
+import { fitFades } from './videoTimelineFades.js';
+
+/**
+ * Browser-safe source-span resolution for the layered video timeline.
+ *
+ * A saved lane stores the span the author ASKED for; the sources can be shorter
+ * (a history entry whose clip was re-cut, a music file shorter than the
+ * guessed placement). The ffmpeg export resolves those spans against the real
+ * sources, and the editor preview must resolve them with the SAME rules or it
+ * plays a different timeline than the one that renders — a later cut lands
+ * seconds off and a fade envelope ramps on the wrong clock. Both sides call
+ * these functions (`resolveTimeline` in services/videoTimeline/local.js and the
+ * client model in client/src/lib/videoTimelineModel.js); no I/O, no React.
+ *
+ * "Unknown" is never "zero": a source whose length could not be established
+ * (no frame metadata, no ffprobe, a metadata load that failed) keeps the
+ * authored span rather than collapsing the lane entry to nothing.
+ */
+
+// Floor for a probe-clamped audio slice — atrim with start === end produces an
+// empty stream that amix rejects.
+export const MIN_MEDIA_SEC = 0.05;
+
+// Seconds, or null when the span cannot be established. Rejects anything that
+// is not a positive finite number so Infinity (a streamed media element) and 0
+// (a probe that found no duration) read as "unknown", not "available: forever"
+// or "available: nothing".
+const knownSeconds = (value) => (Number.isFinite(value) && value > 0 ? value : null);
+
+/** Length of a video-history entry's clip in seconds, or null when unknown. */
+export function historySourceDuration(entry) {
+ return entry?.numFrames && entry?.fps ? knownSeconds(entry.numFrames / entry.fps) : null;
+}
+
+/**
+ * The trim a clip segment actually plays: `outSec` clamped to the source length
+ * (when known) and the fade pair refit against what is left, so a fade authored
+ * for the stored trim cannot outlast a shortened clip (ffmpeg renders a fade
+ * with a negative start time as an all-black segment). `duration` is never
+ * negative — an `inSec` past the end of the source resolves to 0 and the caller
+ * decides whether that is an error (the export rejects it; the preview skips it).
+ */
+export function resolveClipSpan(segment, sourceDurationSec) {
+ const source = knownSeconds(sourceDurationSec);
+ const inSec = Math.max(0, segment.inSec || 0);
+ const outSec = source != null ? Math.min(segment.outSec || 0, source) : (segment.outSec || 0);
+ const duration = Math.max(0, outSec - inSec);
+ return { inSec, outSec, duration, ...fitFades(segment.fadeInSec, segment.fadeOutSec, duration) };
+}
+
+/** A still's fade pair refit against its own hold. */
+export function resolveStillFades(segment) {
+ return fitFades(segment.fadeInSec, segment.fadeOutSec, Math.max(0, segment.durationSec || 0));
+}
+
+/**
+ * The slice of an audio file a bed actually plays, given the file's real length.
+ * A probe reporting LESS than the requested slice is authoritative: the offset
+ * is pulled back inside the file, the length shrinks to what remains, and the
+ * fades are refit to the shortened length. An unknown length (`null`, a failed
+ * probe, a media element that never reported) returns the authored slice
+ * untouched — fades included, since nothing shortened them.
+ */
+export function resolveAudioSpan(track, fileDurationSec) {
+ const offsetSec = track.offsetSec || 0;
+ const durationSec = track.durationSec || 0;
+ const available = knownSeconds(fileDurationSec);
+ if (available == null) {
+ return { offsetSec, durationSec, fadeInSec: track.fadeInSec || 0, fadeOutSec: track.fadeOutSec || 0 };
+ }
+ const fittedOffset = Math.min(offsetSec, Math.max(0, available - MIN_MEDIA_SEC));
+ const fittedDuration = Math.min(durationSec, Math.max(MIN_MEDIA_SEC, available - fittedOffset));
+ return {
+ offsetSec: fittedOffset,
+ durationSec: fittedDuration,
+ ...fitFades(track.fadeInSec, track.fadeOutSec, fittedDuration),
+ };
+}
diff --git a/server/services/videoTimeline/local.js b/server/services/videoTimeline/local.js
index a5cd1b771b..01c6ea4a22 100644
--- a/server/services/videoTimeline/local.js
+++ b/server/services/videoTimeline/local.js
@@ -47,6 +47,7 @@ import {
fitFades,
} from './segments.js';
import { AUDIO_NORM, buildAudioBedMix, fmtSec } from './audioBedMix.js';
+import { historySourceDuration, resolveClipSpan, resolveStillFades, resolveAudioSpan } from '../../lib/videoTimelineSourceSpans.js';
const PROJECTS_FILE = join(PATHS.data, 'video-projects.json');
@@ -55,9 +56,6 @@ const PROJECTS_FILE = join(PATHS.data, 'video-projects.json');
const PROBE_CONCURRENCY = 8;
// A stills-only project has no clip to take geometry from.
const DEFAULT_CANVAS = { width: 1280, height: 720, fps: 24 };
-// Floor for a probe-clamped audio slice — atrim with start === end produces
-// an empty stream that amix rejects.
-const MIN_MEDIA_SEC = 0.05;
// Per-project render mutex map. Keyed by projectId so two different projects
// can render in parallel; same project re-render returns 409 with the
@@ -300,7 +298,7 @@ export async function resolveTimeline(rawProject) {
if (!assetPath) { missingAssets.push(`${seg.assetKind}/${seg.assetFile}`); continue; }
prepared.push({
kind: 'still', i, seg, assetPath, duration: seg.durationSec,
- fades: fitFades(seg.fadeInSec, seg.fadeOutSec, seg.durationSec),
+ fades: resolveStillFades(seg),
});
continue;
}
@@ -308,20 +306,18 @@ export async function resolveTimeline(rawProject) {
if (!entry) { missing.push(seg.clipId); continue; }
const videoPath = safeUnder(PATHS.videos, entry.filename);
if (!videoPath || !existsSync(videoPath)) { missing.push(seg.clipId); continue; }
- const sourceDuration = entry.numFrames && entry.fps ? entry.numFrames / entry.fps : null;
- const inSec = Math.max(0, seg.inSec);
- const outSec = sourceDuration != null ? Math.min(seg.outSec, sourceDuration) : seg.outSec;
- if (outSec - inSec < 1 / Math.max(1, entry.fps || 24)) {
+ // Shared with the editor preview (lib/videoTimelineSourceSpans.js): the
+ // trim is clamped to the history entry's real length and the fades refit
+ // against what is left, so a fade authored for the stored trim cannot
+ // outlast a clip that has since been shortened (a negative fade start
+ // renders as an all-black segment).
+ const { inSec, outSec, duration, fadeInSec, fadeOutSec } = resolveClipSpan(seg, historySourceDuration(entry));
+ if (duration < 1 / Math.max(1, entry.fps || 24)) {
throw new ServerError(`Clip ${i} trim too short — must be ≥ 1 frame`, {
status: 400, code: 'CLIP_TOO_SHORT', context: { index: i, clipId: seg.clipId },
});
}
- // A fade authored against the stored trim can outlast a clip that the
- // history entry has since shortened; rescale rather than emitting a
- // negative fade start (which renders as an all-black segment).
- const duration = outSec - inSec;
- const fades = fitFades(seg.fadeInSec, seg.fadeOutSec, duration);
- prepared.push({ kind: 'clip', i, seg, entry, videoPath, inSec, outSec, duration, fades });
+ prepared.push({ kind: 'clip', i, seg, entry, videoPath, inSec, outSec, duration, fades: { fadeInSec, fadeOutSec } });
}
const overlays = [];
@@ -358,15 +354,9 @@ export async function resolveTimeline(rawProject) {
for (const tr of audioTracks) {
const probed = durationByPath.get(tr.assetPath);
- // A probe reporting LESS than the requested slice is authoritative.
- // `null` (no ffprobe on PATH, unreadable container) must NOT collapse into
- // "0 seconds available" — leave the stored slice alone in that case.
- if (probed == null) continue;
- tr.offsetSec = Math.min(tr.offsetSec, Math.max(0, probed - MIN_MEDIA_SEC));
- tr.durationSec = Math.min(tr.durationSec, Math.max(MIN_MEDIA_SEC, probed - tr.offsetSec));
- const fitted = fitFades(tr.fadeInSec, tr.fadeOutSec, tr.durationSec);
- tr.fadeInSec = fitted.fadeInSec;
- tr.fadeOutSec = fitted.fadeOutSec;
+ // An unknown probe (`null`: no ffprobe on PATH, unreadable container) keeps
+ // the stored slice — it must not collapse into "0 seconds available".
+ Object.assign(tr, resolveAudioSpan(tr, probed));
}
const resolvedSegments = prepared.map((p) => (p.kind === 'still'
diff --git a/server/services/videoTimeline/local.resolve.test.js b/server/services/videoTimeline/local.resolve.test.js
index 8983f67f6a..4851d568af 100644
--- a/server/services/videoTimeline/local.resolve.test.js
+++ b/server/services/videoTimeline/local.resolve.test.js
@@ -160,6 +160,9 @@ describe('resolveTimeline — fades that a later change made too long', () => {
expect(segments[0].duration).toBe(1);
expect(segments[0].fadeInSec + segments[0].fadeOutSec).toBeCloseTo(1);
+ // The editor preview resolves through the same helper and pins these exact
+ // numbers in client/src/lib/videoTimelineModel.test.js.
+ expect([segments[0].fadeInSec, segments[0].fadeOutSec]).toEqual([0.5, 0.5]);
});
it('refits a still fade against its own hold', async () => {
@@ -205,6 +208,12 @@ describe('resolveTimeline — bed probing', () => {
const { audioTracks } = await resolveTimeline(bedProject({ fadeInSec: 5, fadeOutSec: 5 }));
expect(audioTracks[0].fadeInSec + audioTracks[0].fadeOutSec).toBeCloseTo(2);
});
+
+ it('resolves a short file to the span the editor preview pins (2s, fades 1s/1s)', async () => {
+ h.probedDuration = 2;
+ const { audioTracks } = await resolveTimeline(bedProject({ durationSec: 10, fadeInSec: 4, fadeOutSec: 4 }));
+ expect(audioTracks[0]).toMatchObject({ offsetSec: 0, durationSec: 2, fadeInSec: 1, fadeOutSec: 1 });
+ });
});
describe('resolveTimeline — audio presence', () => {