Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.
Timeline coordinate model — clip-anchored modifiers
A modifier (zoom, speed, annotation, full-camera) in the project document is anchored
to a clip in that clip's own source time, not to an absolute timeline position. The
single module that knows how the three coordinate frames relate —
src/lib/ai-edition/timeline/timelineMap.ts —
is the only place that translates between them. Every consumer — the timeline UI, the
native preview, the export, the agent LLM, the captions layer — reads the modifier
through that module and never mixes the frames by hand. The v4→v5 migration that
introduced the clip anchor is documented in
document-model.md; this file is about the model in force and the
contract of timelineMap.
Two time reference frames
The document carries three coordinate systems. A raw clip is identity between source-time and raw-virtual-time (no speed is baked into its geometry), so within a clip raw↔source is a plain shift by the clip's own start offset. The interesting boundary is between the RAW ruler and the COMPRESSED playback sequence:
| Frame | Definition | Who works in it |
|---|---|---|
RAW virtual (currentTimeSec, the ruler, the playhead, document.timeline.clips[].timelineStartSec/timelineEndSec) |
The ruler the user manipulates. Trims still occupy their space. Region authoring, clip drag/resize/reorder, and the timeline ruler all happen here. | NewEditorShell (transport, seek), V4Timeline (drag/resize of clips and pills), useTimeline (every region mutation), document.timeline ops |
SOURCE (a clip's own media time: assetId + sourceStartSec/sourceEndSec) |
What the decoders and the native compositor actually advance. Region anchors (clipId + sourceStartSec + sourceEndSec) live here. |
NativeCompositorOverlay, useNativePlaybackSync, sceneDescription (→ native preview), documentExporter (multi-clip export) |
COMPRESSED (the trim-narrowed resolvePlaybackSegments laid out back-to-back from 0) |
What the native free-run stream and the export frame counter walk through. Indexed by clipIndex into SceneDescription.clips. |
buildSceneDescription (sceneDescription.ts), the native compositor's clip_index field, exportMultiNative |
The model splits the two so each consumer can read what it cares about without having
to translate. Mixing them desyncs preview from render on zoom and trim: a region
authored against the RAW ruler but projected against the COMPRESSED segment layout
slips by exactly the trimmed duration, fires on the wrong clip, and pairs with the
wrong camera. timelineMap is the single line that keeps that line of code from
existing.
Clip-anchored modifiers
Every region (zoom / speed / annotation / layout / shader effect) is stored as one or more
clip-anchored fragments — {clipId, sourceStartSec, sourceEndSec, …payload} — keyed
to the clip it lives on, in that clip's source media time. A region the user drew
across a clip boundary is stored as one fragment per covered clip; the fragments carry
no marker tying them together, but the ruler still renders them as one pill because
they share the same properties (see rule 1 below). Trims narrow a clip's kept source
ranges, so an anchored fragment is hidden/clipped by the same interval math with no
reprojection; reorder carries the fragment with its clipId, again with no
reprojection. The schema field startMs/endMs is a derived cache of the
fragment's current RAW ruler span — it is present so that un-migrated consumers and
the agent LLM (which reasons in virtual seconds) keep reading the shape they always
did, but the anchor is the source of truth. A structural op (move/duplicate/trim)
re-derives startMs/endMs from the anchor; if the cache ever disagrees with the
anchor, the anchor wins.
This is the data-level expression of the authoring-levels diagram in overview.md: modifiers are presented above the timeline in the UX but stored down on the clip in the data.
The timelineMap contract
Every public export of
src/lib/ai-edition/timeline/timelineMap.ts.
"Direction" is what flows in vs what flows out.
| Function | What it converts | Direction |
|---|---|---|
anchorRawRegionsToClips (:65) |
v4 RAW-virtual-ms region → one anchored fragment per covered clip (drops zero-length / off-timeline regions, and the sub-ms sliver a whole-ms edge leaves past a clip boundary) | RAW-virtual → clip-anchored |
anchorRegionsWithDerivedMs (:456) |
Same as above but never drops user data: emits {…fragment, startMs, endMs} for anchored regions and passes un-anchorable regions through with their original ms |
RAW-virtual → v5 stored shape |
anchoredToRawSpanSec (:110) |
One anchored fragment → its current RAW-virtual span on the ruler | clip-anchored → RAW-virtual |
regionIdentityKey (:181) |
A region → canonical identity key (properties minus position/provenance); equal keys = "same kind, same look" | region → identity string |
coalesceByIdentity (:211) |
Set of identified spans → merged runs that touch and share an identity | spans → pills |
clampSpanAgainstNeighbours (:251) |
A desired span clamped against every neighbour, of any identity, backing off toward the span before the edit (a move keeps its length; no cascade) | desired + span before the edit → clamped span |
coalesceRegionsForRuler (:290) |
Region array → ruler pills (one entry per merged run, payload carried by member) |
regions → pills |
resolvePillIds (:314) |
Region id → every region id under its pill (recomputed, not stored) | id → ids |
resolveRegionIds (:337) |
Region id → the ids of the one region it belongs to: its fragments across clip junctions, not a look-alike that touches it on the same clip (see below) | id → ids |
dropPillById / dropPillsByIds (:368 / :377) |
Delete every region under a pill (resolved from the merge rule) | regions → regions |
replacePillSpan |
Move/resize a pill: clamp non-annotation modifiers against neighbours, then re-anchor, keeping the selected id. Annotations may overlap and can carry a layer change in the same write. | pill + clip layout → re-anchored fragments |
segmentRawSpanSec (:481) |
One kept playback segment → its RAW-virtual extent | segment → RAW span |
projectRegionsToSource (:641) |
Region array → source-ms entries with clipIndex for native (anchored path uses anchor; unanchored path falls back to RAW mapping through each segment's own raw extent — never drops an un-anchorable region onto an unrelated clip). A region wholly under a trim is emitted once, marked underTrim, addressed by the segment the cut interrupts. A region a cut splits marks each piece's cut edge continuesBefore / continuesAfter, so Full Camera ramps only at its own edges and stays full across the cut |
RAW/anchored → source + clipIndex |
resolveNativePosition (:759) |
RAW-virtual playhead → {clip, clipIndex, sourceTimeSec} for the active native decoder + paired camera (over a trimmed-out stretch it presents the removed frames themselves, borrowing the same segment index the modifiers under that cut borrow) |
RAW-virtual → source + clipIndex |
The shared region rules are expressed once in this file rather than re-derived per kind. Annotations have the layer exception described below:
- Merge (
coalesceByIdentity) — two regions of the same kind with the same identity that touch are one pill. How they became adjacent (authored side by side, split by a reorder then rejoined, …) is irrelevant; identity is what a region is, not where it came from. - Repel (
clampSpanAgainstNeighbours) — a moved or resized region may not overlap another region of its kind, whatever its identity. An edit clamps to the neighbour's edge; the neighbour never moves (no cascade). Which edge is decided by where the region was before the edit, so a moved region keeps its length and stops flush against the neighbour on the side it came from. A same-identity neighbour is a wall too: let through, it stayed stored under the moved region and an edit of either half reached both (#1133). Stopped flush, the two merge on display (rule 1). Annotations may overlap: theirzIndexcontrols compositing, and the timeline displays separate tracks.
Annotation tracks

regionTracks.ts packs only annotations into visible rows. Each annotation takes the
highest free row below every higher annotation it overlaps. Nonoverlapping annotations
share a row regardless of their saved zIndex; zoom and compact resize handles do not
change the row count. Equal layers follow the document's compositing order. The timeline
reports its exact rendered row count (including a drag preview) to the shell for panel sizing.
Identical overlapping annotations remain separate pills, so selecting, editing, copying or
deleting one cannot reach the other. Unambiguous touching fragments still coalesce across
clip boundaries. If legacy objects share both an identity and a boundary, their fragments
remain separately selectable instead of guessing their pairing. These annotation rules
are explicit options to the pill operations, never inferred from the presence of zIndex.
Adding an annotation, from either the editor or MCP, uses the highest saved layer plus one. Dragging vertically places it above or below the annotations it overlaps at the drop time, using the target row and drag direction, even when the saved layers are equal. The live preview and persisted change use the same ordering plan. Horizontal moves and resizes keep the existing compositing order. Span and order changes persist in one history entry through save/load, preview and export. Speed, trim, zoom, camera and effect regions keep their single lanes and existing repulsion rules. The panel grows for annotation rows up to its height limit; additional rows remain vertically scrollable.
A kind with no properties (trim, full-camera) collapses to a constant identity, so its
regions always merge — the long-standing trim behaviour, now derived from the general
rule. clipId is in NON_IDENTITY_FIELDS and must stay there: it says where a region
is, never what it is, and a trim ventilated across a clip boundary is necessarily 2+ rows
that have to keep rendering and deleting as ONE pill.
A pill is not always one region, and a property edit reaches only the region the user
pointed at (resolveRegionIds, #1017). A click on a pill selects the region under the
pointer; editing it changes every fragment of that region across clip junctions, so a
region drawn over a cut stays one pill, but leaves alone a region that merely matches it
and touches it on the same clip. A region keeps at most one fragment per clip, so a
same-clip neighbour is necessarily another region: the identities part and the pill
splits. Overlapping non-annotation regions of equal identity still edit together, since
changing one alone would break rule 2. Delete, move and resize keep acting on the whole pill; a move rebuilds it as
one region that keeps the id it was grabbed by, so the selection survives. Without provenance,
two regions meeting exactly at a clip junction cannot be told apart from one region's two
fragments, so they edit as one.
Trims carry a clip anchor too (v7)
A trim is {assetId, clipId?, startSec, endSec} — startSec/endSec are its source
window, so clipId was the only field it needed to become a clip-anchored region like
any other. Before v7 it had none, and source time is per asset: two clips over the
same media (a duplicated clip) share a coordinate space, so "which clip is this cut on?"
had no answer and each reader invented one. That produced three symptoms from one cause —
the transcript pane showed the cut on both clips, the ruler drew it on the first, and
playback/export removed the span from both.
trimAppliesToClip
(trim-mapping.ts) is the single
definition of that question; buildClipSection, trimToTimelineSpan and
resolvePlaybackSegments all route through it. A trim with no clipId — pre-v7, or the
inter-clip cuts replaceTimeline mints, which lie outside every clip by construction —
keeps the historical asset-wide meaning, so old documents render unchanged.
One asymmetry with the other kinds, on purpose: trimToTimelineSpan places an anchored
trim on any overlap with its clip, but an un-anchored one only when the clip
contains its start. Containment is doing the disambiguation in the un-anchored case;
for an anchored trim it would be a second, stricter test that hides the pill of a cut
resolvePlaybackSegments still applies — content removed with nothing on the ruler to
click.
The same ambiguity at playback time
Storing the anchor settles which clip a cut is on; the preview still has to answer,
sixty times a second, which clip is playing. It reads the <video>'s own source clock,
and a source time is per asset, so the question is the same one — one layer up, against a
moving target. VirtualPreview tracks the answer in activeClipIdRef and passes it into
locateSourcePosition / locateKeptSegment / findNextKeptSegment
(virtual-preview.ts); the rule
those three now share is that a named clip is believed over anything inferred from
(assetId, sourceTime). Three readers had each weakened it in their own way:
locateSourcePositiongave a clip an exclusive closing edge unless it was the last element of the clips array, and the named clip shared that edge. In the last ~50 ms of a clip — exactly when the rAF is deciding what to play next — the clip disowned its own final frames and the ambiguous scan handed them to a twin over the same recording, reporting the playhead near the END of that twin. Playback then saw "clip end reached" on a clip nothing follows and stopped, parking the playhead at the end of the timeline. Because the exception keyed off array position, whether it happened depended on clip ORDER: with a foreign clip laid down last there was no taker, the scan returned null and the rAF's timeline-order fallback quietly did the right thing. The closing edge is now inclusive for a named clip, and the scan resolves by strict containment first, falling back to the closing edge only if nobody claims the time — the same answer for a plain single-asset timeline, reached without consulting the order.- "Am I inside a cut?" scanned every kept segment by asset, so a twin that KEEPS the
stretch answered for the clip that cuts it and the cut was never skipped during
playback.
locateKeptSegmentrestricts the question to the segments of the clip being played: a source time none of them covers is inside that clip's trim, however many other clips keep it. findNextKeptSegment's source-clock fallback (for when the raw ruler lags behind the video) compared source positions across a whole asset. "Later in source time" only means anything within one clip: with a slice from late in a recording laid down BEFORE a trimmed slice from early in it, playing into that cut answered the first clip — raw start 0 — so playback jumped to the top of the timeline, replayed into the same cut, and looped. It is now scoped to the clip being played, and simply does not apply when no clip is named (the RAW-ruler test above is already the general answer).
Cursor telemetry is source time, and belongs to every clip that replays it
Auto-zoom reads the clicks in the recorded cursor telemetry, which is captured against the
ORIGINAL media file: timeMs is the asset's SOURCE time, the same axis cursor-track.ts
maps through locateSourcePosition. Zoom regions are authored in RAW TIMELINE ms — that is
what anchorRegionsWithDerivedMs ventilates across clips. The two axes coincide for exactly
one layout: a single clip, starting at 0, covering the whole recording. Feeding the
planner's source-time output straight to the store therefore dropped every suggestion
wherever [0, assetDuration] happens to fall on the ruler, i.e. on the first clip —
"automatic zooms only decorate the first clip". buildAutoZoomSuggestionsForClips
(zoom-suggestions.ts) does the
projection: per clip, a plain shift (a raw clip is identity between source and raw-virtual
time), so a click replayed by two clips over one recording yields one zoom on each. Each
clip plans its own zooms from the samples inside its own source window, so no zoom spans a
cut. The plans are then settled over the whole timeline, every recording at once: two
zooms must leave a wide shot between them, even across a cut, and the edit's first and
last seconds stay wide.
The same ambiguity in the transcript pane
Two clips over one media do not only share a source coordinate space — they share the
transcript, which also belongs to the asset. The pane renders one block per clip, so
the same AxcutWord is projected twice and word.id names a moment in the media, not a
thing on screen. ClipWord.id = clipWordId(clip.id, word.id)
(aggregated-transcript.ts)
is the on-screen identity; the React key, data-word-id, the cue highlight and the caret
anchor all use it, and findCueWordId selects its section by cue.clipId before
returning one. The synthetic [silence] tokens make the scoping unconditional rather
than a shared-media special case: withSilenceGaps numbers them from 1 inside each clip,
so silence_1 exists in every block regardless of asset.
The caret/selection helpers at the bottom of RightPanes.tsx never parse that id — they
read it off data-word-id and scope every query to one block's editor element, so they
stay correct whatever the id looks like.
Invariants
A change that breaks one of these is wrong even if every test passes; treat them as
the contract a reviewer can grade against. Each is asserted in
timelineMap.test.ts.
- Anchor wins over the cache. A region with a complete
{clipId, sourceStartSec, sourceEndSec}lands on its source span regardless of whatstartMs/endMssay. (projectRegionsToSource, anchored path — see the "places an anchored region from its anchor, ignoring a stale startMs/endMs" test.) - RAW ↔ source within one clip is identity. A region's anchor
[a, b]on a clip whosesourceStartSec = 0lands on source[a, b]. Speed is not baked into the geometry — the clip is a flat re-mapping, never a stretch. - A trim is invisible to a region's source moment. Identity clip
src[0,10]with a trim at[2,4]: a region authored at RAW[6,8]lands on source[6,8], not[8,10]. (resolveNativePosition/projectRegionsToSource"keeps a region on its source moment despite a trim before it".) - A region fully under a trim stays on its own clip, never leaks. Two clips of
different assets whose source windows overlap numerically: a zoom fully trimmed away
on its own clip must not re-appear on the later clip. It is not dropped either — a trim
is marked by its pill and skipped during playback, but a user who moves the playhead
onto it themselves sees what is underneath, modifiers included (issue #216). So it is
emitted once with
underTrim, borrowing theclipIndexof the segment the cut interrupts; the borrowed index is what pins it to one clip. Dropped only when its clip has no kept segment left to name it. (projectRegionsToSource"keeps a fully-trimmed region on its own clip instead of leaking it onto a later one".) - The render still cuts.
underTrimtells native (scene.rs,regions.rs) to gate the region hard on its own span: full strength inside, nothing outside, no ease-in/out window and no chaining with a neighbouring zoom. An export never composes a frame at those source times, so the entry is inert there — without the gate a zoom's ease-in (up to 1.5 s before its start) would reach the kept frames beside the cut and the preview would show what the export does not. - A named clip beats anything inferred from (assetId, sourceTime), and clip ORDER is
never an input. Asserted in
virtual-preview.test.tsrather thantimelineMap.test.ts: two clips over one recording, at the closing edge of the one being played, must resolve to that clip — and must do so identically however the clips are laid out, including with a foreign clip between or around them. The same rule governs "am I inside a cut?" (locateKeptSegment) and "what plays next?" (findNextKeptSegment), whose source-clock fallback may only speak about the clip it was given. - Same-identity, same-clip stays one pill; different-identity does not merge. Two
regions of equal properties that touch → one pill; touch with one property
changed → two pills, with no memory of ever having been one. (
coalesceByIdentityandreplacePillSpan"clamps a resize at a neighbouring pill of different properties (magnet)".) - A property edit reaches one region, not the whole pill. Editing one of two
look-alikes that touch on one clip splits their pill; editing a fragment of a region
drawn across a junction changes all its fragments. (
resolveRegionIds"narrows a pill to one region, with every fragment of it (#1017)"; end to end through the store inuseTimeline.test.ts, "changes the region picked and its fragments, not its look-alike, and the pill splits".) - Repel never cascades. A neighbour acts as a wall — the edited span stops at its
edge and the neighbour never moves. (
clampSpanAgainstNeighbours"stops at a neighbour on the right" / "left".) - Repel ignores identity for non-annotation modifiers. An identical region dragged into another stops flush
instead of overlapping it; the pair then shows as one pill and an edit of either half
splits it. (
replacePillSpan"stops a speed / zoom / full-camera region flush against an identical one instead of overlapping it (#1133)".) - Repel never shortens a move. A region dragged into a neighbour
keeps its length and stops against it on the side it came from, clip boundary or not;
it is never cut short or thrown past it (#1008). (
replacePillSpan"stops a zoom against a neighbour that ends on a clip boundary, keeping its length".) - No fragment is a rounding sliver. Region edges are whole ms and clip boundaries are
not, so an edge placed on a junction misses it by up to half a ms. A piece whose stored
whole-ms span would be empty is never stored as a fragment of its own; one that rounds to
a millisecond stays, so the region keeps its length. (
anchorRegionsWithDerivedMs"stores no zero-length fragment for an edge rounded onto a clip boundary", "keeps a half-millisecond piece that rounds to a whole millisecond".) - Identity ignores provenance.
id,clipId,sourceStartSec,reason,origin,source,annotationSource, and the legacygroupIdare not part of the identity key. Two regions that differ only in any of those still merge when adjacent. (regionIdentityKey"ignores position and provenance entirely" and the regression test "merges two independently authored regions that carry DIFFERENT legacy groupIds".) - Re-anchoring drops
groupId. A v4 import may carry agroupId; after a write or migration it is gone. (anchorRegionsWithDerivedMs"drops groupId when re-anchoring, so it stops propagating".) visibleSegmentsordering matchesSceneDescription.clips. TheclipIndexemitted byprojectRegionsToSource/resolveNativePositionlines up with the native stream because both consume the same array in the same order. Passing a different array (e.g. already-sorted segments) to the projection while sending the original to the native bridge silently desyncs the mapping.- A region that covers no clip is dropped by ventilation (
anchorRawRegionsToClips/ventilateSpanAcrossClips), andanchorRegionsWithDerivedMspasses it through unanchored so the data is not lost. The two layers disagree on purpose — the low-level primitive is for explicit projection, the migration wrapper is for preserving user data. - A region authored on a clip that was later deleted disappears (anchor resolves to null, the fragment is not shown). Its id is preserved in the document so undo restores it with its anchor.
SSOT façades
When the timeline shape changes — a new field on a region, a new region kind, a change to the anchor contract — every one of these readers/writers must be updated in the same commit, or the project round-trips into an inconsistent state:
| Consumer | File | What it reads / writes |
|---|---|---|
| Document layer (ops) | src/lib/ai-edition/document/timeline.ts, document/layout.ts |
Shared clip/source anchoring and region CRUD; layouts maintain a continuous partition. UI, agent and MCP call these operations. |
| Schema and load upgrades | src/lib/ai-edition/schema/index.ts |
Upgrades through v9 at load; parsing is pure. Full Camera becomes source-anchored layout passages with compatibility envelopes. |
| Timeline UI | src/components/ai-edition/v4/V4Timeline.tsx |
Reads lanes through coalesceRegionsForRuler (:737, :745, :753, :763) and coalescedTrimGroups (:780); a click selects the region under the pointer, and a move/resize/delete acts on the whole pill via the store. |
| Inspector selection pane | src/components/ai-edition/v4/FloatingInspector.tsx (SelectionPane, :735) |
Edits the selected region by id through useTimeline, which widens a property edit to that region's fragments (resolveRegionIds), never to the whole pill. |
| Store / authoring (UI) | src/lib/ai-edition/store/useTimeline.ts |
Every add*, update*Span, removeRegion goes through anchorRegionsWithDerivedMs (:344, :482, :517, :556) and replacePillSpan (:753, :930, :1012, :1042) / dropPillsByIds (:1106-1118); every property edit (patchRegionById, :92) through resolveRegionIds. |
| Native preview scene | src/native/sceneDescription.ts (:489, :508, :517, :531) |
Calls projectRegionsToSource for every region kind before serialising to the native compositor. |
| Native playback sync | src/native/useNativePlaybackSync.ts (:22, :39) |
Resolves the RAW playhead through resolveNativePosition to feed setActiveClip / presentTime. |
| Native compositor overlay | src/components/ai-edition/NativeCompositorOverlay.tsx (:3, :70) |
Same — resolveNativePosition(currentTimeSec, nativeClips, document.timeline.clips) — to keep preview in sync with the timeline ruler. |
| Multi-clip export | src/components/ai-edition/ExportDialog.tsx |
MP4 and GIF consume the same scene builder and native geometry as preview. |
| Captions layer | src/lib/ai-edition/captions/cues.ts (:14-15 and captionCuesToTextRegions) |
Caption cues are a derived view of the transcript; the projection to source at export goes through projectRegionsToSourceTime from region-ventilation, matching the multi-clip export path so preview and file agree. |
| Agent and MCP tools | electron/ai-edition/agent-tools.ts, deep-agent/service.ts |
Share document operations, schemas, descriptions and mutation consent; expose logical layout segments and use the same partition as the UI. |
Continuous layouts
The optional Dispositions band uses layoutRuns over layoutRegions; it replaces
independent Full Camera pills. Moving a boundary edits both neighbours in one undo
step, with no gap or overlap. Clip insertion, duplication, source-window edits and
clear-timeline use the same anchored document operations.
The timeline's change-here button is available only within a clip's half-open
[start, end) window; entering a gap or reaching the final end closes its menu.
For a partitioned document, the Layout pane edits a resolved passage only: gaps
disable its preset picker, while the final end resolves to the last passage.
It never falls back to a whole-video preset write when no passage is active.
buildSceneDescription resolves both endpoints of a change per source clip. Native
SceneLayoutRegion::sample drives rectangle geometry, the screen's clipping weight,
reactive camera size and camera shadow with one eased progress value. The clipping
box interpolates between the two layout slots. When a zoom is already running inside
a contained layout, it keeps that viewport until the uninterrupted zoom settles;
otherwise releasing the clip during a top/bottom → PiP change would inflate the
footage and then shrink it again. Later independent PiP zooms retain their overflow.
The derived scene carries the last contained-layout exit per original source clip,
including through trims and subsequent free passages. Native checks continuity against
the zoom sampler's screen clock, envelopes and connected-pan graph, so seeking does
not require playback history or another authored document field.
Corner interpolation resolves each endpoint's real fallback radius. A camera appearing or disappearing retains its visible endpoint's relative corner radius as its rectangle scales; interpolating that fraction toward zero would square its corners before it vanishes. Changing a boolean crop policy at a boundary would expose enlarged footage abruptly; it must never substitute for interpolating the visible crop. Preview, MP4 and GIF share this planner.
The deterministic workbench includes layout anchors and playability, counts a uniform global layout change even without stored fragments, and checks tool-returned segments against the document. The camera scenarios use the advertised layout tool names.