Skip to content

Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.

19.6 KB
Raw Blame

AI agent

The optional AI editing layer lives in electron/ai-edition/ and src/components/ai-edition/. It connects the editor's chat UI to configured language models and exposes a fixed set of validated operations over the shared project document.

What it is and what gates it

The AI surface is always mounted. Without an API key configured the chat panel is a "no provider connected" welcome view, so the practical effect matches the old AI_FEATURES_ENABLED = false path: nothing agentic runs. (The chat rail entry itself stays — opening it shows the welcome view.) Configuring a provider (or signing in via OAuth) re-enables everything.

The boundary is intentionally narrow: only the LLM and agent UI are gated. The editing model, project panel, timeline, transcript and export surfaces ship to every user. Local Whisper transcription is privacy-preserving and is not gated.

The tool loop

The deep-agent service builds one createDeepAgent instance per user turn. It streams model text and tool lifecycle events to the renderer while a mutable document holder ensures that each call in the turn sees the preceding call's result.

sequenceDiagram
    actor User
    participant Chat as Chat service
    participant Provider as Configured provider
    participant Tools as Validated tool executor
    participant Doc as Project document

    User->>Chat: Send message
    Chat->>Chat: Validate document and save pre-turn checkpoint
    Chat->>Provider: History, document context, and fixed tool schemas
    loop Agent iterations (createDeepAgent default cap: 10)
        Provider-->>Chat: Text delta or tool call(s)
        alt Tool calls requested
            Chat->>Tools: Validate JSON arguments with shared Zod schema
            Tools->>Doc: Apply validated operation to current snapshot
            Doc-->>Tools: Updated document
            Tools-->>Provider: JSON result or validation error
        else Final answer
            Provider-->>Chat: Final text
        end
    end
    Chat-->>User: Streamed answer, tool summaries, and updated document

The agent is built with LangChain's createAgent, not deepagents' createDeepAgent, and pins recursionLimit: 1000 explicitly. createAgent alone would fall back to LangGraph's default of 25 steps, which an auto-enhance pass placing one trim per silence exceeds without difficulty.

Tool schema

The model never free-writes the project document. deep-agent/service.ts#buildTools builds the tools from TOOL_ARG_SCHEMAS, the registry also used by both MCP transports. TOOL_DESCRIPTIONS supplies their descriptions, and the Zod argument schemas come from agent-tools.ts. The executor parses JSON, validates arguments, and returns either a new schema-valid snapshot or an error. These surfaces and the public OPENSCREEN_TOOL_NAMES list are pinned to one another by deep-agent/service.test.ts. The tools operate on the same document model and layout mutations as manual editing.

Tool What it does What it mutates
getCurrentDocument Reads a compact project, asset, clip, trim, and modifier snapshot with explicit time bases. Each asset reports hasCameraTrack / cameraVisible / hasCursorTelemetry beside hasTranscript (hasCursorTelemetry is three-valued: true, false when the asset was checked and has none, null when it was not checked — never false for something we failed to look at), the document reports hasAnyCamera and autoFocusAll, and each zoom reports the renderedScale the viewer will see plus customScale / depthIsOverridden when a custom scale makes its depth inert, and hideCursor when set. Each annotation reports its type and box in the terms the writes take (box frame or footage, centre x/y, width/height), plus a text's look, an arrow's figureData or a blur's blurData; each audio track its gain and fades. Each zoom also reports where its camera moves sit on the ruler, outside the pill: zoomInFromSec, zoomOutUntilSec (speed-aware, from zoomTransitions in src/lib/ai-edition/timeline/zoom-transitions.ts), transitionSec, cutByTrim (in / out / both) when a trim cuts a move, and cutByJunction when a clip junction does (the compositor renders each clip on its own, so a move running past its clip's edge is lost too), so the export jumps there (#1028). A side cut by both is blamed on the cut nearest the pill, where the jump is. Chained zooms (at most 1.5 s apart on screen, no trim or clip junction between them) report the pan instead: panToZoomId / panUntilSec on the first, panFromZoomId on the second, in place of its zoom-out and zoom-in (chainedPans). Nothing.
getTranscript Reads the transcript segments for an asset, or the primary asset, in full. On the production path a segment is one word, so a half-hour recording is a few thousand of them — there is no cap, and no per-model context budget to derive one from. Nothing.
getCursorTrack Reads the recorded pointer telemetry for an asset as REAL SAMPLES, reduced per axis against time (Douglas–Peucker) rather than summarised: every point returned is one that was recorded, carrying its source atSec, cx/cy, the virtualSec that addZoom takes (left off the points, with virtualEqualsSource on the envelope, when the two coincide everywhere), a shape index per distinct pointer bitmap, and kind / trimmed where they apply. visible:false marks a hidden pointer; absent means visible, including older sidecars. Visibility changes survive downsampling even when the pointer stays still. Answers available:false with reason:"no-sidecar" (checked, this asset has none) or reason:"unavailable" (could not be read from here), and the two are never conflated. Nothing.
addTrim Adds one source-time cut inside a clip. Like the zoom writes and setTrim, it reports cutTransitions (zoom, side, seconds lost, by trim or junction) when the edit newly cuts a zoom's move in or out; a warning, never a refusal. timeline.trimRanges.
addTrims Adds many cuts in one call, replaying addTrim per entry so the rules cannot drift apart. Each range stands alone: one that cannot be placed is refused by itself and named with its index and reason while the rest are applied, and the result leads with requested / appliedCount / refusedCount. Only a batch where nothing landed is an error. timeline.trimRanges.
removeWords Cuts spoken words chosen by their getTranscriptWords IDs (fillers, a flubbed phrase, a duplicate take), qualified by assetId since IDs restart per asset. Every ID is resolved to one recorded asset, valid timestamps and exactly one covering clip before anything is written; a missing, invalid, duplicated, already-cut or ambiguous ID refuses the whole request. Words consecutive in the transcript go out as one addTrim span, so a phrase leaves no gaps playing. The result reports each word with the trim actually stored. The transcript is untouched. timeline.trimRanges.
setTrim Moves or resizes an existing source-time trim. The matching timeline.trimRanges entry.
setClipRange Changes a clip's source in/out points and relays clips back-to-back. The clip range and any anchored regions clamped or removed by the shared timeline mutator.
moveClip Reorders a placed clip by naming the clip it should play before (null = last). Preserves every clip id, source range, trim and anchored modifier. Timeline clip order; anchored modifiers' derived ms follow their clip.
replaceTimeline Rebuilds the primary-asset timeline from kept source-time intervals. Preserves the id, origin and label of every clip an interval reproduces exactly, carries existing trims through, and never touches another asset's trims. Refused when it would merge away, shorten or drop a clip, or when the intervals are not ascending (a reorder it cannot perform — the refusal points at moveClip). Timeline clips and trim ranges.
addZoom Adds a clip-anchored zoom over virtual timeline time. depth is an ordinal selecting from ZOOM_DEPTH_SCALES (1.25×–5.0×, non-linear); the result reports the resulting renderedScale. It also takes the inspector's focusMode, hideCursor and customScale, under the same rules as setZoom. When the pointer track can be read for the footage under the span it also reports cursorAnchor: the focus the call used (default included) beside the median recorded pointer position over the span the zoom LANDED on, with spread, offset and the sample count. The source windows come from the stored fragments' own sourceStartSec/sourceEndSec, so a clamped or split zoom is described by the windows it really occupies; trimmed instants are left out; and the field is OMITTED — never guessed, never turned into an absence of data — when no track covers that footage. zoomRanges.
addZooms Adds many zooms in one call, replaying addZoom per entry, with the same per-entry refusal and reporting contract as addTrims. Each applied entry carries its own cursorAnchor. zoomRanges.
setZoom Moves, resizes, or restyles a zoom pill: level, focus, focusMode, hideCursor, 3D camera. Changing depth clears any customScale on that pill — otherwise the write is a no-op at render — and says so in the result. A customScale is written as the inspector writes a typed level (zoomLevelFor: to the hundredth, a table level as its depth) and refused past the clip's zoomScaleLimit; depth with customScale is refused. Reports cursorAnchor against the zoom's EFFECTIVE focus, read back off the document, so a span-only edit still learns what its unchanged focus now looks at. The clip-anchored zoomRanges fragments represented by that pill.
addSpeed Adds a clip-anchored speed region over virtual timeline time. legacyEditor.speedRegions.
setSpeed Moves, resizes, or changes an existing speed pill. The corresponding legacyEditor.speedRegions fragments.
addAnnotation Adds a text, arrow (figure) or privacy blur annotation over virtual timeline time. An arrow or a blur is the editor's text switched through convertAnnotationKind, then given its box and its figureData / blurData (validated by figureDataSchema / blurDataSchema). Images are left out: they need a file. A freehand blur is refused, as the inspector no longer offers it. annotations.
setAnnotation Moves, resizes, retypes or restyles an annotation. A type change goes through convertAnnotationKind with the footage under each fragment (annotationFootageRect), as the inspector's type switch does; figureData / blurData merge only the keys sent. Arguments of another kind (width on a text, blurData on an arrow) are refused. The corresponding clip-anchored annotations fragments.
addLayoutChange Changes the layout from a RAW document-timeline time until the next change. Camera presets require linked camera footage. The shared layoutRegions partition, or the global preset when uniform.
setLayoutPreset Changes one passage by an ID from getCurrentDocument.layoutSegments; equal neighbours merge. An unchanged preset is refused. The same layout partition.
moveLayoutBoundary Moves a passage's start in RAW seconds, updating both neighbours atomically. The first boundary is protected. The same layout partition.
addEffect Writes a shader effect (WGSL, docs/shader-effects.md) and poses it over virtual timeline time, or poses one of the project's effects again by effectId. The compositor checks the code first (checkShaderEffect); code that does not compile is refused with the error at the agent's line. The result gives the cost tier and, when the tier runs long, a recommendation, never a refusal. One effect runs at a time: two cannot overlap. shaderEffects (a new definition) and shaderEffectRanges.
setEffect Moves, re-tunes (values) or retargets an effect pill. The pill's shaderEffectRanges fragments.
setEffectDefinition Changes an effect's name, description, code or parameters, and so every region using it; new code is checked first. A removed parameter's values drop. The shaderEffects entry and its regions' values.
getEffect Reads one effect in full, code included; getCurrentDocument lists effects without their code. Read only.
removeTrim Deletes a trim so its source span plays and exports again. timeline.trimRanges.
removeModifier Resolves and deletes a zoom, speed, annotation, layout change, audio, or effect by ID; removedIds lists the identifiers of the removed items. Audio IDs identify whole tracks (track groups), even when a track contains multiple stored fragments. Removing a layout change extends the preceding passage; the first passage is protected. The matching modifier collection.
removeClip Deletes a placed clip, closes the gap, and drops effects anchored only to it. The result names the modifiers and trims it took with it. Timeline clips and affected anchored modifiers.

Clips and trims use source time. Zoom, speed, annotation and effect tools use virtual edited-timeline time; the executor converts these spans to the clip-anchored millisecond representation used by the document.

Layout point tools use RAW document-timeline seconds. getCurrentDocument.layoutSegments exposes their logical IDs, bounds and presets. The document stores clip/source anchors; buildSceneDescription projects them into source time for native preview and export. Legacy Full Camera executors remain compatibility aliases over this partition and are not advertised to the model or MCP clients. Migration-only camera envelopes are derived from layout metadata rather than maintained as a second mutable collection.

Checkpoints and undo

Before a user message starts an agent turn, chat-service.ts clones the current document and associates it with that user message. Explicit message rewind restores that snapshot and truncates the later conversation. It is separate from Ctrl+Z: completed tool edits accumulate in the agent's working document. runAgentTurn captures the live document and its revision before the turn starts. On normal completion or Stop, runChat returns the accumulated document, which the panel applies once through applyAgentDocumentIfCurrent and the existing project store save/history boundary. Each turn that applies edits has one project save and one undo step. A revision conflict preserves the live document and offers Apply anyway.

Stopping a turn

electron/ipc/chatRuns.ts owns a main-process AbortController per active project. The chat.cancel bridge action must match the sending WebContents, project, session, and renderer-generated run ID; a stale stop cannot cancel the next turn. Concurrent turns for the same project are refused. Other projects are independent.

The signal passes through the bridge service and chat runtime into both createAgent and streamEvents. The runtime also stops awaiting an uncooperative stream or read-only setup operation on abort. Tool calls remain serialized and check the signal before execution and after asynchronous telemetry reads or shader checks, before updating the document holder. Those read-only operations may finish in the background; their results cannot commit edits or emit tool success after stop. The turn lifetime seals queued tools on every exit, including normal completion.

Every streamed event carries the project, session and run ID. The panel ignores foreign or stale events, cancels on project/session change or unmount, and discards an abandoned turn's result. Window destruction aborts its run in main as well. Stop stays in a stopping state until the turn replies and the accumulated document's save settles. It preserves completed tool edits in one undo step, the partial answer, and completed tool summaries. A stopped assistant message has stopped: true in the existing chat sidecar, so reopening the conversation still distinguishes a user stop from a provider failure. A later normal completion can win the race with a stop; repeated stops are harmless.

Context management

chat-compaction.ts uses a four-characters-per-token estimate, adds tool-summary text, and compares history with an 80,000-token budget. Once a session has at least four messages and reaches 70% of that budget, it asks the active provider to summarize the older half at a user-message boundary and inserts an Earlier context assistant message while retaining recent turns. Compaction failure leaves history unchanged. The chat path also sends only the latest 20 stored messages to the agent.

chatBudget.ts duplicates the same lightweight estimate in the renderer so Electron-only code is not bundled into the UI. It drives the context-percentage badge and the manual compact action. The heuristic deliberately leaves room for the system prompt and tool payloads; it is not a provider tokenizer.

Sessions

Sessions are scoped first by project ID and then by session ID. Each session stores a generated ID, project ID, title, creation timestamp, messages, and per-user-message checkpoint references. The renderer can create, select, rename, delete, compact, and rewind sessions through the native bridge.

Sessions persist per project in a versioned sidecar, <projectId>.chat.json, beside the .openscreen document in userData/projects (chat-persistence.ts). It holds each session's title, timestamps, messages and compaction summary; never provider configuration or credentials. chat-service.ts caches a project's sessions on first use and queues every change to disk asynchronously, one queue per project, through a temp file, fsync and rename; a project deletion is queued behind its pending writes, and quitting the app waits for the queue. A malformed, foreign or older file is skipped, never an error, so it cannot stop the project from opening. Provider configuration and encrypted credentials are stored separately.

Document checkpoints stay in process memory. After a restart the conversation comes back, but its rewind controls do not: a message exposes a checkpoint id only while its snapshot is still held.

Known gaps

  • Message checkpoints have no durable persistence, so rewind only reaches turns sent since the app started.
  • allowAgentEdits has no per-turn approval channel. When it is off the agent reads freely, is told by its system prompt to state the edit and ask, and has every write refused by executeAgentTool with a consent_required payload; the returned document is withheld as well. But there is no way for the user to answer "yes, go ahead" for one turn — they have to re-enable the setting in Settings → AI, which is what the refusal tells the model to say.
  • Cursor telemetry is read through an injected CursorTelemetryReader (electron/ipc/handlers.ts builds the only production one, behind resolveApprovedVideoPath). A runtime with no reader wired answers reason: "unavailable" on every call and reports hasCursorTelemetry: null — honest, and useless. There is no cache: a turn probes each asset's sidecar once, then reads it in full for every call in TOOLS_READING_CURSOR — getCursorTrack and the three zoom writes, which need it to report cursorAnchor. N separate addZoom calls therefore re-read the same file N times; addZooms is what keeps a whole zoom pass to one read.
  • readCursorSidecar's normalizer flattens double-click / right-click / middle-click to move before the track is built, so those clicks reach the model as plain moves rather than as the events they were. The narrowing is upstream, in CursorRecordingSample.
  • The deep-agent instance is rebuilt for every turn without a LangGraph checkpointer, so stateful agent threads do not persist independently of the explicit chat history.