Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.
AI agent
The optional AI editing layer lives in electron/ai-edition/ and src/components/ai-edition/. It connects the editor's chat UI to configured language models and exposes a fixed set of validated operations over the shared project document.
What it is and what gates it
The AI surface is always mounted. Without an API key configured the chat panel is a "no provider connected" welcome view, so the practical effect matches the old AI_FEATURES_ENABLED = false path: nothing agentic runs. (The chat rail entry itself stays — opening it shows the welcome view.) Configuring a provider (or signing in via OAuth) re-enables everything.
The boundary is intentionally narrow: only the LLM and agent UI are gated. The editing model, project panel, timeline, transcript and export surfaces ship to every user. Local Whisper transcription is privacy-preserving and is not gated.
The tool loop
The deep-agent service builds one createDeepAgent instance per user turn. It streams model text and tool lifecycle events to the renderer while a mutable document holder ensures that each call in the turn sees the preceding call's result.
sequenceDiagram
actor User
participant Chat as Chat service
participant Provider as Configured provider
participant Tools as Validated tool executor
participant Doc as Project document
User->>Chat: Send message
Chat->>Chat: Validate document and save pre-turn checkpoint
Chat->>Provider: History, document context, and fixed tool schemas
loop Agent iterations (createDeepAgent default cap: 10)
Provider-->>Chat: Text delta or tool call(s)
alt Tool calls requested
Chat->>Tools: Validate JSON arguments with shared Zod schema
Tools->>Doc: Apply validated operation to current snapshot
Doc-->>Tools: Updated document
Tools-->>Provider: JSON result or validation error
else Final answer
Provider-->>Chat: Final text
end
end
Chat-->>User: Streamed answer, tool summaries, and updated document
The agent is built with LangChain's createAgent, not deepagents' createDeepAgent, and pins recursionLimit: 1000 explicitly. createAgent alone would fall back to LangGraph's default of 25 steps, which an auto-enhance pass placing one trim per silence exceeds without difficulty.
Tool schema
The model never free-writes the project document. deep-agent/service.ts#buildTools builds the tools from TOOL_ARG_SCHEMAS, the registry also used by both MCP transports. TOOL_DESCRIPTIONS supplies their descriptions, and the Zod argument schemas come from agent-tools.ts. The executor parses JSON, validates arguments, and returns either a new schema-valid snapshot or an error. These surfaces and the public OPENSCREEN_TOOL_NAMES list are pinned to one another by deep-agent/service.test.ts. The tools operate on the same document model and layout mutations as manual editing.
| Tool | What it does | What it mutates |
|---|---|---|
getCurrentDocument |
Reads a compact project, asset, clip, trim, and modifier snapshot with explicit time bases. Each asset reports hasCameraTrack / cameraVisible / hasCursorTelemetry beside hasTranscript (hasCursorTelemetry is three-valued: true, false when the asset was checked and has none, null when it was not checked — never false for something we failed to look at), the document reports hasAnyCamera and autoFocusAll, and each zoom reports the renderedScale the viewer will see plus customScale / depthIsOverridden when a custom scale makes its depth inert, and hideCursor when set. Each annotation reports its type and box in the terms the writes take (box frame or footage, centre x/y, width/height), plus a text's look, an arrow's figureData or a blur's blurData; each audio track its gain and fades. Each zoom also reports where its camera moves sit on the ruler, outside the pill: zoomInFromSec, zoomOutUntilSec (speed-aware, from zoomTransitions in src/lib/ai-edition/timeline/zoom-transitions.ts), transitionSec, cutByTrim (in / out / both) when a trim cuts a move, and cutByJunction when a clip junction does (the compositor renders each clip on its own, so a move running past its clip's edge is lost too), so the export jumps there (#1028). A side cut by both is blamed on the cut nearest the pill, where the jump is. Chained zooms (at most 1.5 s apart on screen, no trim or clip junction between them) report the pan instead: panToZoomId / panUntilSec on the first, panFromZoomId on the second, in place of its zoom-out and zoom-in (chainedPans). |
Nothing. |
getTranscript |
Reads the transcript segments for an asset, or the primary asset, in full. On the production path a segment is one word, so a half-hour recording is a few thousand of them — there is no cap, and no per-model context budget to derive one from. | Nothing. |
getCursorTrack |
Reads the recorded pointer telemetry for an asset as REAL SAMPLES, reduced per axis against time (Douglas–Peucker) rather than summarised: every point returned is one that was recorded, carrying its source atSec, cx/cy, the virtualSec that addZoom takes (left off the points, with virtualEqualsSource on the envelope, when the two coincide everywhere), a shape index per distinct pointer bitmap, and kind / trimmed where they apply. visible:false marks a hidden pointer; absent means visible, including older sidecars. Visibility changes survive downsampling even when the pointer stays still. Answers available:false with reason:"no-sidecar" (checked, this asset has none) or reason:"unavailable" (could not be read from here), and the two are never conflated. |
Nothing. |
addTrim |
Adds one source-time cut inside a clip. Like the zoom writes and setTrim, it reports cutTransitions (zoom, side, seconds lost, by trim or junction) when the edit newly cuts a zoom's move in or out; a warning, never a refusal. |
timeline.trimRanges. |
addTrims |
Adds many cuts in one call, replaying addTrim per entry so the rules cannot drift apart. Each range stands alone: one that cannot be placed is refused by itself and named with its index and reason while the rest are applied, and the result leads with requested / appliedCount / refusedCount. Only a batch where nothing landed is an error. |
timeline.trimRanges. |
removeWords |
Cuts spoken words chosen by their getTranscriptWords IDs (fillers, a flubbed phrase, a duplicate take), qualified by assetId since IDs restart per asset. Every ID is resolved to one recorded asset, valid timestamps and exactly one covering clip before anything is written; a missing, invalid, duplicated, already-cut or ambiguous ID refuses the whole request. Words consecutive in the transcript go out as one addTrim span, so a phrase leaves no gaps playing. The result reports each word with the trim actually stored. The transcript is untouched. |
timeline.trimRanges. |
setTrim |
Moves or resizes an existing source-time trim. | The matching timeline.trimRanges entry. |
setClipRange |
Changes a clip's source in/out points and relays clips back-to-back. | The clip range and any anchored regions clamped or removed by the shared timeline mutator. |
moveClip |
Reorders a placed clip by naming the clip it should play before (null = last). Preserves every clip id, source range, trim and anchored modifier. |
Timeline clip order; anchored modifiers' derived ms follow their clip. |
replaceTimeline |
Rebuilds the primary-asset timeline from kept source-time intervals. Preserves the id, origin and label of every clip an interval reproduces exactly, carries existing trims through, and never touches another asset's trims. Refused when it would merge away, shorten or drop a clip, or when the intervals are not ascending (a reorder it cannot perform — the refusal points at moveClip). |
Timeline clips and trim ranges. |
addZoom |
Adds a clip-anchored zoom over virtual timeline time. depth is an ordinal selecting from ZOOM_DEPTH_SCALES (1.25×–5.0×, non-linear); the result reports the resulting renderedScale. It also takes the inspector's focusMode, hideCursor and customScale, under the same rules as setZoom. When the pointer track can be read for the footage under the span it also reports cursorAnchor: the focus the call used (default included) beside the median recorded pointer position over the span the zoom LANDED on, with spread, offset and the sample count. The source windows come from the stored fragments' own sourceStartSec/sourceEndSec, so a clamped or split zoom is described by the windows it really occupies; trimmed instants are left out; and the field is OMITTED — never guessed, never turned into an absence of data — when no track covers that footage. |
zoomRanges. |
addZooms |
Adds many zooms in one call, replaying addZoom per entry, with the same per-entry refusal and reporting contract as addTrims. Each applied entry carries its own cursorAnchor. |
zoomRanges. |
setZoom |
Moves, resizes, or restyles a zoom pill: level, focus, focusMode, hideCursor, 3D camera. Changing depth clears any customScale on that pill — otherwise the write is a no-op at render — and says so in the result. A customScale is written as the inspector writes a typed level (zoomLevelFor: to the hundredth, a table level as its depth) and refused past the clip's zoomScaleLimit; depth with customScale is refused. Reports cursorAnchor against the zoom's EFFECTIVE focus, read back off the document, so a span-only edit still learns what its unchanged focus now looks at. |
The clip-anchored zoomRanges fragments represented by that pill. |
addSpeed |
Adds a clip-anchored speed region over virtual timeline time. | legacyEditor.speedRegions. |
setSpeed |
Moves, resizes, or changes an existing speed pill. | The corresponding legacyEditor.speedRegions fragments. |
addAnnotation |
Adds a text, arrow (figure) or privacy blur annotation over virtual timeline time. An arrow or a blur is the editor's text switched through convertAnnotationKind, then given its box and its figureData / blurData (validated by figureDataSchema / blurDataSchema). Images are left out: they need a file. A freehand blur is refused, as the inspector no longer offers it. |
annotations. |
setAnnotation |
Moves, resizes, retypes or restyles an annotation. A type change goes through convertAnnotationKind with the footage under each fragment (annotationFootageRect), as the inspector's type switch does; figureData / blurData merge only the keys sent. Arguments of another kind (width on a text, blurData on an arrow) are refused. |
The corresponding clip-anchored annotations fragments. |
addLayoutChange |
Changes the layout from a RAW document-timeline time until the next change. Camera presets require linked camera footage. | The shared layoutRegions partition, or the global preset when uniform. |
setLayoutPreset |
Changes one passage by an ID from getCurrentDocument.layoutSegments; equal neighbours merge. An unchanged preset is refused. |
The same layout partition. |
moveLayoutBoundary |
Moves a passage's start in RAW seconds, updating both neighbours atomically. The first boundary is protected. | The same layout partition. |
addEffect |
Writes a shader effect (WGSL, docs/shader-effects.md) and poses it over virtual timeline time, or poses one of the project's effects again by effectId. The compositor checks the code first (checkShaderEffect); code that does not compile is refused with the error at the agent's line. The result gives the cost tier and, when the tier runs long, a recommendation, never a refusal. One effect runs at a time: two cannot overlap. |
shaderEffects (a new definition) and shaderEffectRanges. |
setEffect |
Moves, re-tunes (values) or retargets an effect pill. |
The pill's shaderEffectRanges fragments. |
setEffectDefinition |
Changes an effect's name, description, code or parameters, and so every region using it; new code is checked first. A removed parameter's values drop. | The shaderEffects entry and its regions' values. |
getEffect |
Reads one effect in full, code included; getCurrentDocument lists effects without their code. |
Read only. |
removeTrim |
Deletes a trim so its source span plays and exports again. | timeline.trimRanges. |
removeModifier |
Resolves and deletes a zoom, speed, annotation, layout change, audio, or effect by ID; removedIds lists the identifiers of the removed items. Audio IDs identify whole tracks (track groups), even when a track contains multiple stored fragments. Removing a layout change extends the preceding passage; the first passage is protected. |
The matching modifier collection. |
removeClip |
Deletes a placed clip, closes the gap, and drops effects anchored only to it. The result names the modifiers and trims it took with it. | Timeline clips and affected anchored modifiers. |
Clips and trims use source time. Zoom, speed, annotation and effect tools use virtual edited-timeline time; the executor converts these spans to the clip-anchored millisecond representation used by the document.
Layout point tools use RAW document-timeline seconds. getCurrentDocument.layoutSegments
exposes their logical IDs, bounds and presets. The document stores clip/source anchors;
buildSceneDescription projects them into source time for native preview and export.
Legacy Full Camera executors remain compatibility aliases over this partition and are
not advertised to the model or MCP clients. Migration-only camera envelopes are derived
from layout metadata rather than maintained as a second mutable collection.
Checkpoints and undo
Before a user message starts an agent turn, chat-service.ts clones the current
document and associates it with that user message. Explicit message rewind restores
that snapshot and truncates the later conversation. It is separate from Ctrl+Z:
completed tool edits accumulate in the agent's working document. runAgentTurn
captures the live document and its revision before the turn starts. On normal
completion or Stop, runChat returns the accumulated document, which the panel
applies once through applyAgentDocumentIfCurrent and the existing project store
save/history boundary. Each turn that applies edits has one project save and one
undo step. A revision conflict preserves the live document and offers Apply anyway.
Stopping a turn
electron/ipc/chatRuns.ts owns a main-process AbortController per active project.
The chat.cancel bridge action must match the sending WebContents, project, session,
and renderer-generated run ID; a stale stop cannot cancel the next turn. Concurrent
turns for the same project are refused. Other projects are independent.
The signal passes through the bridge service and chat runtime into both
createAgent and streamEvents. The runtime also stops awaiting an uncooperative
stream or read-only setup operation on abort. Tool calls remain serialized and
check the signal before execution and after asynchronous telemetry reads or shader
checks, before updating the document holder. Those read-only operations may finish
in the background; their results cannot commit edits or emit tool success after stop.
The turn lifetime seals queued tools on every exit, including normal completion.
Every streamed event carries the project, session and run ID. The panel ignores
foreign or stale events, cancels on project/session change or unmount, and discards
an abandoned turn's result. Window destruction aborts its run in main as well.
Stop stays in a stopping state until the turn replies and the accumulated document's
save settles. It preserves completed tool edits in one undo step, the partial answer,
and completed tool summaries. A stopped assistant message has stopped: true in the
existing chat sidecar, so reopening the conversation still distinguishes a user stop
from a provider failure. A later normal completion can win the race with a stop;
repeated stops are harmless.
Context management
chat-compaction.ts uses a four-characters-per-token estimate, adds tool-summary text, and compares history with an 80,000-token budget. Once a session has at least four messages and reaches 70% of that budget, it asks the active provider to summarize the older half at a user-message boundary and inserts an Earlier context assistant message while retaining recent turns. Compaction failure leaves history unchanged. The chat path also sends only the latest 20 stored messages to the agent.
chatBudget.ts duplicates the same lightweight estimate in the renderer so Electron-only code is not bundled into the UI. It drives the context-percentage badge and the manual compact action. The heuristic deliberately leaves room for the system prompt and tool payloads; it is not a provider tokenizer.
Sessions
Sessions are scoped first by project ID and then by session ID. Each session stores a generated ID, project ID, title, creation timestamp, messages, and per-user-message checkpoint references. The renderer can create, select, rename, delete, compact, and rewind sessions through the native bridge.
Sessions persist per project in a versioned sidecar, <projectId>.chat.json, beside the .openscreen document in userData/projects (chat-persistence.ts). It holds each session's title, timestamps, messages and compaction summary; never provider configuration or credentials. chat-service.ts caches a project's sessions on first use and queues every change to disk asynchronously, one queue per project, through a temp file, fsync and rename; a project deletion is queued behind its pending writes, and quitting the app waits for the queue. A malformed, foreign or older file is skipped, never an error, so it cannot stop the project from opening. Provider configuration and encrypted credentials are stored separately.
Document checkpoints stay in process memory. After a restart the conversation comes back, but its rewind controls do not: a message exposes a checkpoint id only while its snapshot is still held.
Known gaps
- Message checkpoints have no durable persistence, so rewind only reaches turns sent since the app started.
allowAgentEditshas no per-turn approval channel. When it is off the agent reads freely, is told by its system prompt to state the edit and ask, and has every write refused byexecuteAgentToolwith aconsent_requiredpayload; the returned document is withheld as well. But there is no way for the user to answer "yes, go ahead" for one turn — they have to re-enable the setting in Settings → AI, which is what the refusal tells the model to say.- Cursor telemetry is read through an injected
CursorTelemetryReader(electron/ipc/handlers.tsbuilds the only production one, behindresolveApprovedVideoPath). A runtime with no reader wired answersreason: "unavailable"on every call and reportshasCursorTelemetry: null— honest, and useless. There is no cache: a turn probes each asset's sidecar once, then reads it in full for every call inTOOLS_READING_CURSOR—getCursorTrackand the three zoom writes, which need it to reportcursorAnchor. N separateaddZoomcalls therefore re-read the same file N times;addZoomsis what keeps a whole zoom pass to one read. readCursorSidecar's normalizer flattensdouble-click/right-click/middle-clicktomovebefore the track is built, so those clicks reach the model as plain moves rather than as the events they were. The narrowing is upstream, inCursorRecordingSample.- The deep-agent instance is rebuilt for every turn without a LangGraph checkpointer, so stateful agent threads do not persist independently of the explicit chat history.