Skip to content

Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.

29.7 KB
Raw Blame

Timeline coordinate model — clip-anchored modifiers

A modifier (zoom, speed, annotation, full-camera) in the project document is anchored to a clip in that clip's own source time, not to an absolute timeline position. The single module that knows how the three coordinate frames relate — src/lib/ai-edition/timeline/timelineMap.ts — is the only place that translates between them. Every consumer — the timeline UI, the native preview, the export, the agent LLM, the captions layer — reads the modifier through that module and never mixes the frames by hand. The v4→v5 migration that introduced the clip anchor is documented in document-model.md; this file is about the model in force and the contract of timelineMap.

Two time reference frames

The document carries three coordinate systems. A raw clip is identity between source-time and raw-virtual-time (no speed is baked into its geometry), so within a clip raw↔source is a plain shift by the clip's own start offset. The interesting boundary is between the RAW ruler and the COMPRESSED playback sequence:

Frame Definition Who works in it
RAW virtual (currentTimeSec, the ruler, the playhead, document.timeline.clips[].timelineStartSec/timelineEndSec) The ruler the user manipulates. Trims still occupy their space. Region authoring, clip drag/resize/reorder, and the timeline ruler all happen here. NewEditorShell (transport, seek), V4Timeline (drag/resize of clips and pills), useTimeline (every region mutation), document.timeline ops
SOURCE (a clip's own media time: assetId + sourceStartSec/sourceEndSec) What the decoders and the native compositor actually advance. Region anchors (clipId + sourceStartSec + sourceEndSec) live here. NativeCompositorOverlay, useNativePlaybackSync, sceneDescription (→ native preview), documentExporter (multi-clip export)
COMPRESSED (the trim-narrowed resolvePlaybackSegments laid out back-to-back from 0) What the native free-run stream and the export frame counter walk through. Indexed by clipIndex into SceneDescription.clips. buildSceneDescription (sceneDescription.ts), the native compositor's clip_index field, exportMultiNative

The model splits the two so each consumer can read what it cares about without having to translate. Mixing them desyncs preview from render on zoom and trim: a region authored against the RAW ruler but projected against the COMPRESSED segment layout slips by exactly the trimmed duration, fires on the wrong clip, and pairs with the wrong camera. timelineMap is the single line that keeps that line of code from existing.

Clip-anchored modifiers

Every region (zoom / speed / annotation / layout / shader effect) is stored as one or more clip-anchored fragments — {clipId, sourceStartSec, sourceEndSec, …payload} — keyed to the clip it lives on, in that clip's source media time. A region the user drew across a clip boundary is stored as one fragment per covered clip; the fragments carry no marker tying them together, but the ruler still renders them as one pill because they share the same properties (see rule 1 below). Trims narrow a clip's kept source ranges, so an anchored fragment is hidden/clipped by the same interval math with no reprojection; reorder carries the fragment with its clipId, again with no reprojection. The schema field startMs/endMs is a derived cache of the fragment's current RAW ruler span — it is present so that un-migrated consumers and the agent LLM (which reasons in virtual seconds) keep reading the shape they always did, but the anchor is the source of truth. A structural op (move/duplicate/trim) re-derives startMs/endMs from the anchor; if the cache ever disagrees with the anchor, the anchor wins.

This is the data-level expression of the authoring-levels diagram in overview.md: modifiers are presented above the timeline in the UX but stored down on the clip in the data.

The timelineMap contract

Every public export of src/lib/ai-edition/timeline/timelineMap.ts. "Direction" is what flows in vs what flows out.

Function What it converts Direction
anchorRawRegionsToClips (:65) v4 RAW-virtual-ms region → one anchored fragment per covered clip (drops zero-length / off-timeline regions, and the sub-ms sliver a whole-ms edge leaves past a clip boundary) RAW-virtual → clip-anchored
anchorRegionsWithDerivedMs (:456) Same as above but never drops user data: emits {…fragment, startMs, endMs} for anchored regions and passes un-anchorable regions through with their original ms RAW-virtual → v5 stored shape
anchoredToRawSpanSec (:110) One anchored fragment → its current RAW-virtual span on the ruler clip-anchored → RAW-virtual
regionIdentityKey (:181) A region → canonical identity key (properties minus position/provenance); equal keys = "same kind, same look" region → identity string
coalesceByIdentity (:211) Set of identified spans → merged runs that touch and share an identity spans → pills
clampSpanAgainstNeighbours (:251) A desired span clamped against every neighbour, of any identity, backing off toward the span before the edit (a move keeps its length; no cascade) desired + span before the edit → clamped span
coalesceRegionsForRuler (:290) Region array → ruler pills (one entry per merged run, payload carried by member) regions → pills
resolvePillIds (:314) Region id → every region id under its pill (recomputed, not stored) id → ids
resolveRegionIds (:337) Region id → the ids of the one region it belongs to: its fragments across clip junctions, not a look-alike that touches it on the same clip (see below) id → ids
dropPillById / dropPillsByIds (:368 / :377) Delete every region under a pill (resolved from the merge rule) regions → regions
replacePillSpan Move/resize a pill: clamp non-annotation modifiers against neighbours, then re-anchor, keeping the selected id. Annotations may overlap and can carry a layer change in the same write. pill + clip layout → re-anchored fragments
segmentRawSpanSec (:481) One kept playback segment → its RAW-virtual extent segment → RAW span
projectRegionsToSource (:641) Region array → source-ms entries with clipIndex for native (anchored path uses anchor; unanchored path falls back to RAW mapping through each segment's own raw extent — never drops an un-anchorable region onto an unrelated clip). A region wholly under a trim is emitted once, marked underTrim, addressed by the segment the cut interrupts. A region a cut splits marks each piece's cut edge continuesBefore / continuesAfter, so Full Camera ramps only at its own edges and stays full across the cut RAW/anchored → source + clipIndex
resolveNativePosition (:759) RAW-virtual playhead → {clip, clipIndex, sourceTimeSec} for the active native decoder + paired camera (over a trimmed-out stretch it presents the removed frames themselves, borrowing the same segment index the modifiers under that cut borrow) RAW-virtual → source + clipIndex

The shared region rules are expressed once in this file rather than re-derived per kind. Annotations have the layer exception described below:

  1. Merge (coalesceByIdentity) — two regions of the same kind with the same identity that touch are one pill. How they became adjacent (authored side by side, split by a reorder then rejoined, …) is irrelevant; identity is what a region is, not where it came from.
  2. Repel (clampSpanAgainstNeighbours) — a moved or resized region may not overlap another region of its kind, whatever its identity. An edit clamps to the neighbour's edge; the neighbour never moves (no cascade). Which edge is decided by where the region was before the edit, so a moved region keeps its length and stops flush against the neighbour on the side it came from. A same-identity neighbour is a wall too: let through, it stayed stored under the moved region and an edit of either half reached both (#1133). Stopped flush, the two merge on display (rule 1). Annotations may overlap: their zIndex controls compositing, and the timeline displays separate tracks.

Annotation tracks

Overlapping annotations on separate tracks

regionTracks.ts packs only annotations into visible rows. Each annotation takes the highest free row below every higher annotation it overlaps. Nonoverlapping annotations share a row regardless of their saved zIndex; zoom and compact resize handles do not change the row count. Equal layers follow the document's compositing order. The timeline reports its exact rendered row count (including a drag preview) to the shell for panel sizing. Identical overlapping annotations remain separate pills, so selecting, editing, copying or deleting one cannot reach the other. Unambiguous touching fragments still coalesce across clip boundaries. If legacy objects share both an identity and a boundary, their fragments remain separately selectable instead of guessing their pairing. These annotation rules are explicit options to the pill operations, never inferred from the presence of zIndex.

Adding an annotation, from either the editor or MCP, uses the highest saved layer plus one. Dragging vertically places it above or below the annotations it overlaps at the drop time, using the target row and drag direction, even when the saved layers are equal. The live preview and persisted change use the same ordering plan. Horizontal moves and resizes keep the existing compositing order. Span and order changes persist in one history entry through save/load, preview and export. Speed, trim, zoom, camera and effect regions keep their single lanes and existing repulsion rules. The panel grows for annotation rows up to its height limit; additional rows remain vertically scrollable.

A kind with no properties (trim, full-camera) collapses to a constant identity, so its regions always merge — the long-standing trim behaviour, now derived from the general rule. clipId is in NON_IDENTITY_FIELDS and must stay there: it says where a region is, never what it is, and a trim ventilated across a clip boundary is necessarily 2+ rows that have to keep rendering and deleting as ONE pill.

A pill is not always one region, and a property edit reaches only the region the user pointed at (resolveRegionIds, #1017). A click on a pill selects the region under the pointer; editing it changes every fragment of that region across clip junctions, so a region drawn over a cut stays one pill, but leaves alone a region that merely matches it and touches it on the same clip. A region keeps at most one fragment per clip, so a same-clip neighbour is necessarily another region: the identities part and the pill splits. Overlapping non-annotation regions of equal identity still edit together, since changing one alone would break rule 2. Delete, move and resize keep acting on the whole pill; a move rebuilds it as one region that keeps the id it was grabbed by, so the selection survives. Without provenance, two regions meeting exactly at a clip junction cannot be told apart from one region's two fragments, so they edit as one.

Trims carry a clip anchor too (v7)

A trim is {assetId, clipId?, startSec, endSec} — startSec/endSec are its source window, so clipId was the only field it needed to become a clip-anchored region like any other. Before v7 it had none, and source time is per asset: two clips over the same media (a duplicated clip) share a coordinate space, so "which clip is this cut on?" had no answer and each reader invented one. That produced three symptoms from one cause — the transcript pane showed the cut on both clips, the ruler drew it on the first, and playback/export removed the span from both.

trimAppliesToClip (trim-mapping.ts) is the single definition of that question; buildClipSection, trimToTimelineSpan and resolvePlaybackSegments all route through it. A trim with no clipId — pre-v7, or the inter-clip cuts replaceTimeline mints, which lie outside every clip by construction — keeps the historical asset-wide meaning, so old documents render unchanged.

One asymmetry with the other kinds, on purpose: trimToTimelineSpan places an anchored trim on any overlap with its clip, but an un-anchored one only when the clip contains its start. Containment is doing the disambiguation in the un-anchored case; for an anchored trim it would be a second, stricter test that hides the pill of a cut resolvePlaybackSegments still applies — content removed with nothing on the ruler to click.

The same ambiguity at playback time

Storing the anchor settles which clip a cut is on; the preview still has to answer, sixty times a second, which clip is playing. It reads the <video>'s own source clock, and a source time is per asset, so the question is the same one — one layer up, against a moving target. VirtualPreview tracks the answer in activeClipIdRef and passes it into locateSourcePosition / locateKeptSegment / findNextKeptSegment (virtual-preview.ts); the rule those three now share is that a named clip is believed over anything inferred from (assetId, sourceTime). Three readers had each weakened it in their own way:

  • locateSourcePosition gave a clip an exclusive closing edge unless it was the last element of the clips array, and the named clip shared that edge. In the last ~50 ms of a clip — exactly when the rAF is deciding what to play next — the clip disowned its own final frames and the ambiguous scan handed them to a twin over the same recording, reporting the playhead near the END of that twin. Playback then saw "clip end reached" on a clip nothing follows and stopped, parking the playhead at the end of the timeline. Because the exception keyed off array position, whether it happened depended on clip ORDER: with a foreign clip laid down last there was no taker, the scan returned null and the rAF's timeline-order fallback quietly did the right thing. The closing edge is now inclusive for a named clip, and the scan resolves by strict containment first, falling back to the closing edge only if nobody claims the time — the same answer for a plain single-asset timeline, reached without consulting the order.
  • "Am I inside a cut?" scanned every kept segment by asset, so a twin that KEEPS the stretch answered for the clip that cuts it and the cut was never skipped during playback. locateKeptSegment restricts the question to the segments of the clip being played: a source time none of them covers is inside that clip's trim, however many other clips keep it.
  • findNextKeptSegment's source-clock fallback (for when the raw ruler lags behind the video) compared source positions across a whole asset. "Later in source time" only means anything within one clip: with a slice from late in a recording laid down BEFORE a trimmed slice from early in it, playing into that cut answered the first clip — raw start 0 — so playback jumped to the top of the timeline, replayed into the same cut, and looped. It is now scoped to the clip being played, and simply does not apply when no clip is named (the RAW-ruler test above is already the general answer).

Cursor telemetry is source time, and belongs to every clip that replays it

Auto-zoom reads the clicks in the recorded cursor telemetry, which is captured against the ORIGINAL media file: timeMs is the asset's SOURCE time, the same axis cursor-track.ts maps through locateSourcePosition. Zoom regions are authored in RAW TIMELINE ms — that is what anchorRegionsWithDerivedMs ventilates across clips. The two axes coincide for exactly one layout: a single clip, starting at 0, covering the whole recording. Feeding the planner's source-time output straight to the store therefore dropped every suggestion wherever [0, assetDuration] happens to fall on the ruler, i.e. on the first clip — "automatic zooms only decorate the first clip". buildAutoZoomSuggestionsForClips (zoom-suggestions.ts) does the projection: per clip, a plain shift (a raw clip is identity between source and raw-virtual time), so a click replayed by two clips over one recording yields one zoom on each. Each clip plans its own zooms from the samples inside its own source window, so no zoom spans a cut. The plans are then settled over the whole timeline, every recording at once: two zooms must leave a wide shot between them, even across a cut, and the edit's first and last seconds stay wide.

The same ambiguity in the transcript pane

Two clips over one media do not only share a source coordinate space — they share the transcript, which also belongs to the asset. The pane renders one block per clip, so the same AxcutWord is projected twice and word.id names a moment in the media, not a thing on screen. ClipWord.id = clipWordId(clip.id, word.id) (aggregated-transcript.ts) is the on-screen identity; the React key, data-word-id, the cue highlight and the caret anchor all use it, and findCueWordId selects its section by cue.clipId before returning one. The synthetic [silence] tokens make the scoping unconditional rather than a shared-media special case: withSilenceGaps numbers them from 1 inside each clip, so silence_1 exists in every block regardless of asset.

The caret/selection helpers at the bottom of RightPanes.tsx never parse that id — they read it off data-word-id and scope every query to one block's editor element, so they stay correct whatever the id looks like.

Invariants

A change that breaks one of these is wrong even if every test passes; treat them as the contract a reviewer can grade against. Each is asserted in timelineMap.test.ts.

  • Anchor wins over the cache. A region with a complete {clipId, sourceStartSec, sourceEndSec} lands on its source span regardless of what startMs/endMs say. (projectRegionsToSource, anchored path — see the "places an anchored region from its anchor, ignoring a stale startMs/endMs" test.)
  • RAW ↔ source within one clip is identity. A region's anchor [a, b] on a clip whose sourceStartSec = 0 lands on source [a, b]. Speed is not baked into the geometry — the clip is a flat re-mapping, never a stretch.
  • A trim is invisible to a region's source moment. Identity clip src[0,10] with a trim at [2,4]: a region authored at RAW [6,8] lands on source [6,8], not [8,10]. (resolveNativePosition / projectRegionsToSource "keeps a region on its source moment despite a trim before it".)
  • A region fully under a trim stays on its own clip, never leaks. Two clips of different assets whose source windows overlap numerically: a zoom fully trimmed away on its own clip must not re-appear on the later clip. It is not dropped either — a trim is marked by its pill and skipped during playback, but a user who moves the playhead onto it themselves sees what is underneath, modifiers included (issue #216). So it is emitted once with underTrim, borrowing the clipIndex of the segment the cut interrupts; the borrowed index is what pins it to one clip. Dropped only when its clip has no kept segment left to name it. (projectRegionsToSource "keeps a fully-trimmed region on its own clip instead of leaking it onto a later one".)
  • The render still cuts. underTrim tells native (scene.rs, regions.rs) to gate the region hard on its own span: full strength inside, nothing outside, no ease-in/out window and no chaining with a neighbouring zoom. An export never composes a frame at those source times, so the entry is inert there — without the gate a zoom's ease-in (up to 1.5 s before its start) would reach the kept frames beside the cut and the preview would show what the export does not.
  • A named clip beats anything inferred from (assetId, sourceTime), and clip ORDER is never an input. Asserted in virtual-preview.test.ts rather than timelineMap.test.ts: two clips over one recording, at the closing edge of the one being played, must resolve to that clip — and must do so identically however the clips are laid out, including with a foreign clip between or around them. The same rule governs "am I inside a cut?" (locateKeptSegment) and "what plays next?" (findNextKeptSegment), whose source-clock fallback may only speak about the clip it was given.
  • Same-identity, same-clip stays one pill; different-identity does not merge. Two regions of equal properties that touch → one pill; touch with one property changed → two pills, with no memory of ever having been one. (coalesceByIdentity and replacePillSpan "clamps a resize at a neighbouring pill of different properties (magnet)".)
  • A property edit reaches one region, not the whole pill. Editing one of two look-alikes that touch on one clip splits their pill; editing a fragment of a region drawn across a junction changes all its fragments. (resolveRegionIds "narrows a pill to one region, with every fragment of it (#1017)"; end to end through the store in useTimeline.test.ts, "changes the region picked and its fragments, not its look-alike, and the pill splits".)
  • Repel never cascades. A neighbour acts as a wall — the edited span stops at its edge and the neighbour never moves. (clampSpanAgainstNeighbours "stops at a neighbour on the right" / "left".)
  • Repel ignores identity for non-annotation modifiers. An identical region dragged into another stops flush instead of overlapping it; the pair then shows as one pill and an edit of either half splits it. (replacePillSpan "stops a speed / zoom / full-camera region flush against an identical one instead of overlapping it (#1133)".)
  • Repel never shortens a move. A region dragged into a neighbour keeps its length and stops against it on the side it came from, clip boundary or not; it is never cut short or thrown past it (#1008). (replacePillSpan "stops a zoom against a neighbour that ends on a clip boundary, keeping its length".)
  • No fragment is a rounding sliver. Region edges are whole ms and clip boundaries are not, so an edge placed on a junction misses it by up to half a ms. A piece whose stored whole-ms span would be empty is never stored as a fragment of its own; one that rounds to a millisecond stays, so the region keeps its length. (anchorRegionsWithDerivedMs "stores no zero-length fragment for an edge rounded onto a clip boundary", "keeps a half-millisecond piece that rounds to a whole millisecond".)
  • Identity ignores provenance. id, clipId, sourceStartSec, reason, origin, source, annotationSource, and the legacy groupId are not part of the identity key. Two regions that differ only in any of those still merge when adjacent. (regionIdentityKey "ignores position and provenance entirely" and the regression test "merges two independently authored regions that carry DIFFERENT legacy groupIds".)
  • Re-anchoring drops groupId. A v4 import may carry a groupId; after a write or migration it is gone. (anchorRegionsWithDerivedMs "drops groupId when re-anchoring, so it stops propagating".)
  • visibleSegments ordering matches SceneDescription.clips. The clipIndex emitted by projectRegionsToSource / resolveNativePosition lines up with the native stream because both consume the same array in the same order. Passing a different array (e.g. already-sorted segments) to the projection while sending the original to the native bridge silently desyncs the mapping.
  • A region that covers no clip is dropped by ventilation (anchorRawRegionsToClips / ventilateSpanAcrossClips), and anchorRegionsWithDerivedMs passes it through unanchored so the data is not lost. The two layers disagree on purpose — the low-level primitive is for explicit projection, the migration wrapper is for preserving user data.
  • A region authored on a clip that was later deleted disappears (anchor resolves to null, the fragment is not shown). Its id is preserved in the document so undo restores it with its anchor.

SSOT façades

When the timeline shape changes — a new field on a region, a new region kind, a change to the anchor contract — every one of these readers/writers must be updated in the same commit, or the project round-trips into an inconsistent state:

Consumer File What it reads / writes
Document layer (ops) src/lib/ai-edition/document/timeline.ts, document/layout.ts Shared clip/source anchoring and region CRUD; layouts maintain a continuous partition. UI, agent and MCP call these operations.
Schema and load upgrades src/lib/ai-edition/schema/index.ts Upgrades through v9 at load; parsing is pure. Full Camera becomes source-anchored layout passages with compatibility envelopes.
Timeline UI src/components/ai-edition/v4/V4Timeline.tsx Reads lanes through coalesceRegionsForRuler (:737, :745, :753, :763) and coalescedTrimGroups (:780); a click selects the region under the pointer, and a move/resize/delete acts on the whole pill via the store.
Inspector selection pane src/components/ai-edition/v4/FloatingInspector.tsx (SelectionPane, :735) Edits the selected region by id through useTimeline, which widens a property edit to that region's fragments (resolveRegionIds), never to the whole pill.
Store / authoring (UI) src/lib/ai-edition/store/useTimeline.ts Every add*, update*Span, removeRegion goes through anchorRegionsWithDerivedMs (:344, :482, :517, :556) and replacePillSpan (:753, :930, :1012, :1042) / dropPillsByIds (:1106-1118); every property edit (patchRegionById, :92) through resolveRegionIds.
Native preview scene src/native/sceneDescription.ts (:489, :508, :517, :531) Calls projectRegionsToSource for every region kind before serialising to the native compositor.
Native playback sync src/native/useNativePlaybackSync.ts (:22, :39) Resolves the RAW playhead through resolveNativePosition to feed setActiveClip / presentTime.
Native compositor overlay src/components/ai-edition/NativeCompositorOverlay.tsx (:3, :70) Same — resolveNativePosition(currentTimeSec, nativeClips, document.timeline.clips) — to keep preview in sync with the timeline ruler.
Multi-clip export src/components/ai-edition/ExportDialog.tsx MP4 and GIF consume the same scene builder and native geometry as preview.
Captions layer src/lib/ai-edition/captions/cues.ts (:14-15 and captionCuesToTextRegions) Caption cues are a derived view of the transcript; the projection to source at export goes through projectRegionsToSourceTime from region-ventilation, matching the multi-clip export path so preview and file agree.
Agent and MCP tools electron/ai-edition/agent-tools.ts, deep-agent/service.ts Share document operations, schemas, descriptions and mutation consent; expose logical layout segments and use the same partition as the UI.

Continuous layouts

The optional Dispositions band uses layoutRuns over layoutRegions; it replaces independent Full Camera pills. Moving a boundary edits both neighbours in one undo step, with no gap or overlap. Clip insertion, duplication, source-window edits and clear-timeline use the same anchored document operations.

The timeline's change-here button is available only within a clip's half-open [start, end) window; entering a gap or reaching the final end closes its menu. For a partitioned document, the Layout pane edits a resolved passage only: gaps disable its preset picker, while the final end resolves to the last passage. It never falls back to a whole-video preset write when no passage is active.

buildSceneDescription resolves both endpoints of a change per source clip. Native SceneLayoutRegion::sample drives rectangle geometry, the screen's clipping weight, reactive camera size and camera shadow with one eased progress value. The clipping box interpolates between the two layout slots. When a zoom is already running inside a contained layout, it keeps that viewport until the uninterrupted zoom settles; otherwise releasing the clip during a top/bottom → PiP change would inflate the footage and then shrink it again. Later independent PiP zooms retain their overflow. The derived scene carries the last contained-layout exit per original source clip, including through trims and subsequent free passages. Native checks continuity against the zoom sampler's screen clock, envelopes and connected-pan graph, so seeking does not require playback history or another authored document field.

Corner interpolation resolves each endpoint's real fallback radius. A camera appearing or disappearing retains its visible endpoint's relative corner radius as its rectangle scales; interpolating that fraction toward zero would square its corners before it vanishes. Changing a boolean crop policy at a boundary would expose enlarged footage abruptly; it must never substitute for interpolating the visible crop. Preview, MP4 and GIF share this planner.

The deterministic workbench includes layout anchors and playability, counts a uniform global layout change even without stored fragments, and checks tool-returned segments against the document. The camera scenarios use the advertised layout tool names.