Skip to content

Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.

fix(effects): getFrames names a frame it could not render, and the review's doc fixes

EtienneLescotab6fe789f1d9
diff --git a/docs/shader-effects.md b/docs/shader-effects.md
index 598ff0179..ab4a96b5f 100644
--- a/docs/shader-effects.md
+++ b/docs/shader-effects.md
@@ -383,8 +383,10 @@ Une région choisit sa cible (`target`). Sans demande explicite, c'est `image`.
- **`image`** : l'image sans les textes. La passe s'insère avant les annotations et les
sous-titres. Un effet déforme l'écran, la webcam, le curseur et le fond, jamais le texte que
l'utilisateur a posé pour être lu.
-- **`text`** : les textes seuls (annotations et sous-titres). Les textes sont dessinés dans un
- calque transparent à part, l'effet s'y applique, puis le calque est composé par-dessus l'image
+- **`text`** : le calque posé sur l'image, c'est-à-dire les annotations de tout type (textes,
+ images, flèches) et les sous-titres. Un effet `text` touche donc aussi une flèche ou un logo :
+ les séparer casserait leur ordre de superposition (`zIndex`). Ce calque est dessiné à
+ part, transparent, l'effet s'y applique, puis le calque est composé par-dessus l'image
(« over » prémultiplié). L'image dessous reste intacte. Le calque n'existe que si une région
`text` couvre la frame.
- **`both`** : l'image finale, textes compris, par exemple pour un fondu au noir qui doit couvrir
@@ -586,8 +588,9 @@ lente.
frame, sur la grille à 60 i/s de `exportVideo`, et demande ces frames à l'addon
(`renderFrames`).
- **L'addon rend par la marche de l'export,** `timeline_walk::render_frames` : mêmes clips, mêmes
- vitesses, même frame source à chaque instant. Les frames non demandées ne sont ni décodées ni
- composées.
+ vitesses, même frame source à chaque instant. Les frames non demandées ne sont pas composées.
+ Le décodeur ne lit que ce qui mène à chaque frame demandée : depuis la précédente quand elle
+ est à moins de 2 s, sinon depuis une image clé.
- **Un saut de plus de 2 s repart d'une image clé** (`catch_up`). `seek_to` rend la première
frame à ou après le point visé ; on recule donc ce point jusqu'à tomber avant l'instant voulu,
puis on avance comme l'export : la frame montrée est la dernière dont le pts le précède.
diff --git a/electron/mcp/media-tools.test.ts b/electron/mcp/media-tools.test.ts
index cf3561763..aaaf3cb3a 100644
--- a/electron/mcp/media-tools.test.ts
+++ b/electron/mcp/media-tools.test.ts
@@ -836,6 +836,29 @@ describe("getFrames without an export that shows the project", () => {
renderFrames = async () => null;
expect(textOf(await call("getFrames", { projectId, atSec: [1] }))).toContain("no export yet");
});
+
+ it("names a frame the render did not return, and refuses a clip without media", async () => {
+ const { projectId } = await recordTake();
+ renderFrames = async () => [];
+ const short = await call("getFrames", { projectId, atSec: [1] });
+ expect(short.content).toContainEqual({
+ type: "text",
+ text: "No frame here: the rendered video ends before it.",
+ });
+
+ const document = await documents.getProject(projectId);
+ // The schema requires a path on every asset: a clip has no media when its asset is gone.
+ await documents.saveProject({
+ ...document,
+ timeline: {
+ ...document.timeline,
+ clips: document.timeline.clips.map((clip) => ({ ...clip, assetId: "gone" })),
+ },
+ });
+ const missing = await call("getFrames", { projectId, atSec: [1] });
+ expect(missing.isError).toBe(true);
+ expect(textOf(missing)).toContain("no media file");
+ });
});
describe("getFrames", () => {
diff --git a/electron/mcp/media-tools.ts b/electron/mcp/media-tools.ts
index 720ebcd5b..c112d2bda 100644
--- a/electron/mcp/media-tools.ts
+++ b/electron/mcp/media-tools.ts
@@ -523,6 +523,12 @@ export function createMediaTools(deps: MediaToolDeps): MediaTools {
why: string,
): Promise<CallToolResult | null> {
if (!deps.renderFrames) return null;
+ // The frame grid below counts every clip, the render only those with media: past a clip
+ // without one, the two would disagree on which frame a moment is.
+ const paths = new Map(document.assets.map((asset) => [asset.id, asset.originalPath]));
+ if (document.timeline.clips.some((clip) => !paths.get(clip.assetId))) {
+ return text("A clip on this timeline has no media file: its frames cannot be placed.", true);
+ }
const clips = buildNativeClipList(document);
if (clips.length === 0) return text("This project's timeline has no clip to show.", true);
const rulerEndSec = Math.max(0, ...document.timeline.clips.map((clip) => clip.timelineEndSec));
@@ -550,13 +556,11 @@ export function createMediaTools(deps: MediaToolDeps): MediaTools {
times.forEach((sec, i) => {
content.push({ type: "text", text: `${sec.toFixed(2)}s on the ruler` });
const frame = byIndex.get(indices[i]);
- if (frame) {
- content.push({
- type: "image",
- data: frame.jpeg.toString("base64"),
- mimeType: "image/jpeg",
- });
- }
+ content.push(
+ frame
+ ? { type: "image", data: frame.jpeg.toString("base64"), mimeType: "image/jpeg" }
+ : { type: "text", text: "No frame here: the rendered video ends before it." },
+ );
});
return { content };
}
diff --git a/technical-documentation/testing/manual-e2e-checklist.md b/technical-documentation/testing/manual-e2e-checklist.md
index 4f8a13754..b6a81f341 100644
--- a/technical-documentation/testing/manual-e2e-checklist.md
+++ b/technical-documentation/testing/manual-e2e-checklist.md
@@ -968,4 +968,4 @@ These checks are the human slice: an agent cannot click Apple's picker (see abov
| 2026-10-14 | `main` @ `2d98601c4` (includes c541fffaa, the recording-state fix), CI artifact `openscreen-mac-arm64` from build run 38598557320, DMG signed and notarized (`spctl`: accepted, Notarized Developer ID, M4LK7C6S84), installed over the build of the row above; About `2.0.0 (dmg)` | macOS 26.5 (25F71), Mac mini M1 (arm64), one 3840×2160 display at 2×, no camera (MDM) | **Pass, scoped to defect 1 of the row above** | Real OS input through computer-use, launched with `open -n --env OPENSCREEN_MAC_SOURCE_PICKER=legacy` (`-n` because another session's `Openscreen mcp` was running, #1214); MCP calls from an SDK StreamableHTTP client to the in-app server. **In-app `startRecording` (c541fffaa):** on a Finder window, with the microphone and the editable cursor requested while the bar had both off, the call returns in 4.9 s (was a 120 s timeout and a lost take); the editor closes, the bar records with the requested settings and one capture helper runs. `stopRecording` returns in 2.7 s with the `projectId` of the project the editor opens on, in Edit, one clip, no zoom (Record mode's Auto-zoom after recording is Off), take 1840×872, 26.9 s. The bar's microphone, cursor mode and last source are back to their values from before the take. `addZoom` on it shows live and Cmd+Z / Cmd+Shift+Z take it back and forth; `exportVideo` at 30 fps gives 806 frames for 26.87 s as predicted, and `getFrames` shows the zoom at 8 s. **Skipped:** the refusal while the editor has unsaved changes (the drag meant to hold the editor dirty never moved the pill, so the accepted `startRecording` that followed proves nothing either way); zooms after an in-app take with Auto-zoom On; the defects of the row above other than defect 1 (#1214, #1215, #1216 still open). |
| 2026-10-14 | `main` @ `61fa43812` (includes #1227, #1228, #1229, the fixes for #1215, #1214, #1216), CI artifact `openscreen-mac-arm64` from build run 38615516865, DMG signed and notarized (`spctl`: accepted, Notarized Developer ID, M4LK7C6S84); `app.asar` and the capture helper carry the new code; About `2.0.0 (dmg)` | macOS 26.5 (25F71), Mac mini M1 (arm64), one 3840×2160 display at 2× | **Pass on the three fixes, 1 new defect, 1 leftover** | Real OS input through computer-use for the Finder launch; MCP calls from SDK clients (stdio to `Openscreen mcp`, StreamableHTTP to the in-app server); numbers are ffprobe/ffmpeg on the files. Every OpenScreen process was quit first, another session's `Openscreen mcp` of an older build included, so no old binary could take the launch. **#1214:** with an `Openscreen mcp` client session open and no GUI, `open /Applications/Openscreen.app` brings the recording bar up in about 1 s (was: exit 0 and nothing); two quick `open`s with the GUI running leave one GUI; after a quit, a double-click on OpenScreen in Finder's Applications opens the bar. **#1215:** a GIF export (`gifSize: "small"`, 778×480) read by `getFrames` through the stdio server by `projectId` (1, 6, 11 s on the ruler) and by `path`, and through the in-app server by `projectId`, gives frames of distinct moments (was "media error 4"). **#1216:** a window take of a Finder window with `systemAudio: true` while `afplay` plays speech reads −10.0 dB mean through the stdio server and −10.0 dB through the in-app server under `legacy` (was −91 dB), video 1840×872, the window only. **New defect (P1 for the in-app server on macOS 15.2+):** on a default launch (no `OPENSCREEN_MAC_SOURCE_PICKER`), in-app `startRecording` with `window` ignores the window and opens Apple's picker ("Share a Window or Entire Screen"), which waits for a click, so the call fails after 120 s with "capture had not started"; the picker then stays on screen after the timeout, and even after the app quits, until someone presses Cancel. The stdio server and a `legacy` launch are not affected (4.3 s to start). **Leftover from #1214:** the `Openscreen mcp` process turns from `UIElement` to `Foreground` within 0.3 s of the launch LaunchServices hands it, and keeps a second OpenScreen icon in the Dock; a click on it is an `activate`, so it opens the GUI rather than nothing. A bare hidden Electron 41 with the same handler, with or without a hidden window, stays `UIElement`, so the cause is in the app's CLI process and not found yet. **Not covered:** the two failure paths of #1229's audio stream (ScreenCaptureKit was not made to refuse it). |
| 2026-10-14 | `main` @ `4f4bb997c` (includes #1237 and #1239, the fixes for #1232 and #1233), CI artifact `openscreen-mac-arm64` from build run 38627769728, DMG signed and notarized (`spctl`: accepted, Notarized Developer ID, M4LK7C6S84); About `2.0.0 (dmg)` | macOS 26.5 (25F71), Mac mini M1 (arm64), one 3840×2160 display at 2× | **Pass, scoped to #1232 and #1233** | Every OpenScreen process quit before the install, other sessions' `Openscreen mcp` included; MCP calls from SDK clients; numbers are ffprobe/ffmpeg on the files. **#1233:** with an `Openscreen mcp` client session open and no GUI, `open /Applications/Openscreen.app` starts the GUI and the `mcp` process stays `UIElement` at 0.5, 2.5 and 8.5 s (was `Foreground` within 0.3 s, with its own Dock icon). The two OpenScreen icons left in the Dock carry no running dot once the GUI quits: they are the Dock's recent-apps entries, not processes. **#1232:** on a default launch (no `OPENSCREEN_MAC_SOURCE_PICKER`, so Apple's picker owns the bar's sources), in-app `startRecording` with `window` and `systemAudio: true` returns in 5.0 s with no picker window on screen (was a 120 s timeout and a picker left behind); the take is the window only, 1840×872, system audio at −10.3 dB, and `stopRecording` returns in 2.7 s. **Not covered:** the refusal message when Screen Recording is denied (the grant was not revoked on this Mac). |
-| 2026-10-14 | dev, local branch `test/shader-fx-macos-e2e`: `feat/shader-effects-macos` (the Metal pass) with #1202 merged in locally, never pushed, plus a local-only line that writes each export's scene to a file. Compositor addon rebuilt from that tree (string probe `no Metal device` HIT, control `wallpaper image` HIT), `npm run build-vite` | macOS 26.5 (25F71), Mac mini M1, on mains power | Partial: agent-written shader effects only, pass | Headless, so no computer-use: `electron . --user-data-dir=<test profile> mcp` over stdio, and the addon loaded in Node for the live view (`createView`, `setScene`, `setParam` as `pushAllNativeParams` does, `presentTime`, `readFrame`). A 6 s 1920×1080 test pattern with a hand-written `.cursor.json` of three clicks. **Agent loop:** `addEffect` with a typo refused in 6 ms at the agent's line 10; the C.6 ripple accepted in 152 ms, real Metal compile included, `validated: true`, tier light; `exportVideo` then `getFrames` show the ripple after each click and nothing at 0.5 s. **Preview against export** (BT.709): 3.1/255 over the whole frame with the effect, 3.0 without; on the pixels the effect changes, 13 to 16/255 against the export with the effect and 14 to 16 against the one without on the same pixels with no effect, which is the codec on sharp edges (two exports of the same project decode to identical pixels). The live view held 29.7 fps against the 30 fps source with and without. **Export** of 180 frames at 1752×1080: 2469 ms without, 2428 with the ripple, 2375 with a heavier effect. **Driver note:** without the `setParam` calls the editor makes, the headless view draws the cursor at its default size, which reads as a preview/export mismatch. Not run: real capture, the editor UI (effects have no track until the UI PR), the in-app agent, a `text`-target effect over a real caption (covered by `an_annotation_stays_out_of_image_effects_and_inside_text_effects` on the Metal device). |
+| 2026-10-14 | dev, local branch `test/shader-fx-macos-e2e`: `feat/shader-effects-macos` (the Metal pass) with #1202 merged in locally, never pushed, plus a local-only line that writes each export's scene to a file. Compositor addon rebuilt from that tree (string probe `no Metal device` HIT, control `wallpaper image` HIT), `npm run build-vite` | macOS 26.5 (25F71), Mac mini M1, on mains power | Partial: agent-written shader effects only, pass | Headless, so no computer-use: `electron . --user-data-dir=<test profile> mcp` over stdio, and the addon loaded in Node for the live view (`createView`, `setScene`, `setParam` as `pushAllNativeParams` does, `presentTime`, `readFrame`). A 6 s 1920×1080 test pattern with a hand-written `.cursor.json` of three clicks. **Agent loop:** `addEffect` with a typo refused in 6 ms at the agent's line 10; the C.6 ripple accepted in 152 ms, real Metal compile included, `validated: true`, tier light; `exportVideo` then `getFrames` show the ripple after each click and nothing at 0.5 s. **Preview against export** (BT.709): 3.1/255 over the whole frame with the effect, 3.0 without; on the pixels the effect changes, 13 to 16/255 against the export with the effect and 14 to 16 against the one without on the same pixels with no effect, which is the codec on sharp edges (two exports of the same project decode to identical pixels). The live view held 29.7 fps against the 30 fps source with and without. **Export** of 180 frames at 1752×1080: 2469 ms without, 2428 with the ripple, 2375 with a heavier effect. **Driver note:** without the `setParam` calls the editor makes, the headless view draws the cursor at its default size, which reads as a preview/export mismatch. Not run: real capture, the editor UI (effects have no track until the UI PR), the in-app agent, a `text`-target effect over a real caption (`an_annotation_stays_out_of_image_effects_and_inside_text_effects` covers the routing of a figure annotation on the Metal device, not a caption). |