Skip to content

Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.

feat(mcp): getFrames reads a project's export at ruler seconds

EtienneLescot14147c88eb56
diff --git a/electron/mcp/media-tools.test.ts b/electron/mcp/media-tools.test.ts
index d0745748d..0470c15a6 100644
--- a/electron/mcp/media-tools.test.ts
+++ b/electron/mcp/media-tools.test.ts
@@ -399,6 +399,36 @@ describe("exportVideo", () => {
});
});
+describe("getFrames on a project", () => {
+ it("reads the last export at the ruler seconds the document tools use", async () => {
+ const { projectId } = await recordTake();
+ expect(textOf(await call("getFrames", { projectId, atSec: [1] }))).toContain("no export yet");
+
+ // A cut of 2 s early in the take: the ruler keeps a hole there, the export does not.
+ expect((await call("addTrim", { projectId, startSec: 2, endSec: 4 })).isError).toBeFalsy();
+ runners.results.export = { success: true, outputPath: path.join(dir, "demo.mp4") };
+ await call("exportVideo", { projectId, path: path.join(dir, "demo.mp4") });
+
+ const frames = await call("getFrames", { projectId, atSec: [1, 6] });
+ const labels = frames.content.flatMap((part) => (part.type === "text" ? [part.text] : []));
+ expect(labels.slice(1)).toEqual([
+ "1.00s on the ruler, 1.00s into the video",
+ "6.00s on the ruler, 4.00s into the video",
+ ]);
+ expect(labels[0]).not.toContain("changed after this export");
+
+ // An edit after the export is flagged, so a stale video is never taken for the edit.
+ await call("addAnnotation", { projectId, startSec: 5, endSec: 7, text: "New" });
+ const stale = await call("getFrames", { projectId, atSec: [6] });
+ expect(textOf(stale)).toContain("changed after this export");
+ });
+
+ it("takes a project or a file, not both", async () => {
+ expect((await call("getFrames", {})).isError).toBe(true);
+ expect((await call("getFrames", { projectId: "p", path: "x.mp4" })).isError).toBe(true);
+ });
+});
+
describe("getFrames", () => {
it("returns each frame as an image, labelled with its time", async () => {
const result = await call("getFrames", { path: "take.mp4", atSec: [1, 5] });
diff --git a/electron/mcp/media-tools.ts b/electron/mcp/media-tools.ts
index 463cd8658..8472b81d6 100644
--- a/electron/mcp/media-tools.ts
+++ b/electron/mcp/media-tools.ts
@@ -12,6 +12,10 @@ import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import type { CallToolResult } from "@modelcontextprotocol/sdk/types.js";
import { z } from "zod";
import type { CursorTelemetryPoint } from "../../src/components/video-editor/types";
+import {
+ projectRawTimelineSecToPlayback,
+ readSpeedRegions,
+} from "../../src/lib/ai-edition/document/timeline";
import type { AxcutDocument } from "../../src/lib/ai-edition/schema";
import {
appendAutoZoomSuggestions,
@@ -41,6 +45,7 @@ export interface MediaToolDeps {
| "addAsset"
| "saveProject"
| "filePathFor"
+ | "getProject"
| "getProjectForUpdate"
| "saveProjectIfUnchanged"
>;
@@ -139,7 +144,7 @@ const READ_ONLY = { readOnlyHint: true, destructiveHint: false, openWorldHint: f
export const MEDIA_PREAMBLE = [
"OpenScreen records the screen, turns each take into a project you edit with the document tools, and exports it as a video.",
- "A demo goes: listSources, then startRecording, then drive the app, then stopRecording. It returns the projectId and a checkpointId. Edit with the document tools, always passing that projectId: every edit is saved to the project straight away, and restoreCheckpoint with the checkpointId takes the project back to the raw take in one step. Then exportVideo, and getFrames on the export to check it before you hand it over.",
+ "A demo goes: listSources, then startRecording, then drive the app, then stopRecording. It returns the projectId and a checkpointId. Edit with the document tools, always passing that projectId: every edit is saved to the project straight away, and restoreCheckpoint with the checkpointId takes the project back to the raw take in one step. Then exportVideo, and getFrames with the projectId to check the export before you hand it over: it takes the same ruler seconds as the document tools, so a trim or a speed region never shifts what you look at.",
"Drive the app with the real mouse when you can: OpenScreen records the pointer, draws a clean cursor and zooms on each click. If you drive it from a script instead (Playwright, CDP), the screen pointer never moves: log each action as the script runs it, then hand the log to setPointerTrack after stopRecording.",
"These projects are in the OpenScreen app's project list, so the user can open one in the editor. Do not edit a project while the user has it open there: the editor would keep its own copy.",
"",
@@ -159,6 +164,8 @@ export function registerMediaTools(server: McpServer, deps: MediaToolDeps): void
let exporting = false;
/** When each take of this session started, by its video: setPointerTrack's clock. */
const startedAtByVideo = new Map<string, number>();
+ /** Each project's last export of this session, and when it was made: what getFrames reads. */
+ const lastExports = new Map<string, { path: string; at: string }>();
/** A runner's result, its hidden window closed: this server outlives every runner. */
async function finish(run: ReturnType<MediaToolDeps["runners"]["start"]>) {
@@ -450,7 +457,7 @@ export function registerMediaTools(server: McpServer, deps: MediaToolDeps): void
"exportVideo",
{
description:
- "Render a project to an MP4 (H.264, 60 fps) or a GIF with OpenScreen's compositor: the background, cursor, zooms, cuts, speed changes and annotations as edited. Returns the file's path; call getFrames on it to check the result.",
+ "Render a project to an MP4 (H.264, 60 fps) or a GIF with OpenScreen's compositor: the background, cursor, zooms, cuts, speed changes and annotations as edited. Returns the file's path; call getFrames with the projectId to check it at the ruler seconds you edited with. The video's shape follows the project's format (setAspectRatio).",
inputSchema: z.object({
projectId: z.string(),
path: z
@@ -461,7 +468,9 @@ export function registerMediaTools(server: McpServer, deps: MediaToolDeps): void
quality: z
.enum(["medium", "good", "source"])
.optional()
- .describe("medium is 720p, good is 1080p (default), source keeps the recording's size."),
+ .describe(
+ "The height: medium is 720 pixels, good is 1080 (default), source keeps the recording's. The width follows the format.",
+ ),
}),
annotations: { readOnlyHint: false, destructiveHint: false, openWorldHint: false },
},
@@ -507,6 +516,10 @@ export function registerMediaTools(server: McpServer, deps: MediaToolDeps): void
if (!result.success) {
return text(`The export failed: ${result.error ?? "unknown error"}`, true);
}
+ lastExports.set(args.projectId, {
+ path: result.outputPath ?? outPath,
+ at: new Date().toISOString(),
+ });
return text(
JSON.stringify({
path: result.outputPath,
@@ -524,37 +537,79 @@ export function registerMediaTools(server: McpServer, deps: MediaToolDeps): void
server.registerTool(
"getFrames",
{
- description: `Look at a video: still frames of a take (videoPath from stopRecording) or of an export, as images. Ask for the moments you need to check, such as each zoom and each callout. At most ${MAX_FRAMES} frames per call.`,
+ description: [
+ `Look at a video as still images, at most ${MAX_FRAMES} per call. Ask for the moments you need to check, such as each zoom and each callout.`,
+ "To check an edit, pass projectId: you get the project's last export, and atSec are the ruler seconds the document tools use, mapped through the trims and speed regions to the moment the export plays them. With path instead, you get any video file (a take's videoPath, say) and atSec are that file's own seconds.",
+ ].join("\n"),
inputSchema: z.object({
- path: z.string().describe("The video file."),
+ projectId: z.string().optional().describe("Look at this project's last export."),
+ path: z.string().optional().describe("Or look at this video file."),
atSec: z
.array(z.number().nonnegative())
.max(MAX_FRAMES)
.optional()
- .describe("Times in seconds of that video. A time past the end gives the last frame."),
+ .describe("Times in seconds: ruler seconds with projectId, the file's own with path."),
count: z
.number()
.int()
.min(1)
.max(MAX_FRAMES)
.optional()
- .describe("Frames spread evenly across the video, when atSec is not given. Default 6."),
+ .describe("Frames spread evenly, when atSec is not given. Default 6."),
}),
annotations: READ_ONLY,
},
async (args) => {
- const video = await deps.openVideo(path.resolve(args.path));
+ if ((args.projectId === undefined) === (args.path === undefined)) {
+ return text("Pass either projectId or path.", true);
+ }
+ let videoPath: string;
+ /** Ruler seconds → seconds of the exported file, when looking at a project. */
+ let toVideoSec = (sec: number) => sec;
+ let rulerEndSec: number | null = null;
+ let note = "";
+ if (args.projectId !== undefined) {
+ const exported = lastExports.get(args.projectId);
+ if (!exported) {
+ return text("This project has no export yet: call exportVideo first.", true);
+ }
+ const document = await deps.documents.getProject(args.projectId);
+ videoPath = exported.path;
+ toVideoSec = (sec) => programmeSecOf(document, sec);
+ rulerEndSec = Math.max(0, ...document.timeline.clips.map((clip) => clip.timelineEndSec));
+ if (document.project.updatedAt > exported.at) {
+ note = " The project changed after this export: export it again to see the changes.";
+ }
+ } else {
+ videoPath = path.resolve(args.path ?? "");
+ }
+
+ const video = await deps.openVideo(videoPath);
try {
- const times = frameTimes(video.durationSec, args.atSec ?? null, args.count ?? 6);
+ const asked =
+ args.atSec ??
+ (rulerEndSec === null ? null : frameTimes(rulerEndSec, null, args.count ?? 6));
+ const times = asked
+ ? asked.map((sec) => ({
+ asked: sec,
+ at: frameTimes(video.durationSec, [toVideoSec(sec)], 1)[0],
+ }))
+ : frameTimes(video.durationSec, null, args.count ?? 6).map((at) => ({ asked: at, at }));
const content: CallToolResult["content"] = [
{
type: "text",
- text: `${times.length} frame(s) of a ${video.width}x${video.height} video lasting ${video.durationSec.toFixed(2)}s.`,
+ text: `${times.length} frame(s) of a ${video.width}x${video.height} video lasting ${video.durationSec.toFixed(2)}s.${note}`,
},
];
- for (const timeSec of times) {
- const frame = await video.capture(timeSec, FRAME_WIDTH);
- content.push({ type: "text", text: `${timeSec.toFixed(2)}s` });
+ for (const { asked: askedSec, at } of times) {
+ const frame = await video.capture(at, FRAME_WIDTH);
+ content.push({
+ type: "text",
+ text:
+ rulerEndSec === null
+ ? `${at.toFixed(2)}s`
+ : `${askedSec.toFixed(2)}s on the ruler, ${at.toFixed(2)}s into the video`,
+ });
content.push({ type: "image", data: frame.data, mimeType: frame.mimeType });
}
return { content };
@@ -564,3 +619,16 @@ export function registerMediaTools(server: McpServer, deps: MediaToolDeps): void
},
);
}
+
+/**
+ * Where a ruler second plays in the export: the exporter's own projection, through the trims
+ * (holes in the ruler that the export skips) and the speed regions (spans it plays faster).
+ */
+function programmeSecOf(document: AxcutDocument, rulerSec: number): number {
+ return projectRawTimelineSecToPlayback(
+ document.timeline.clips,
+ document.timeline.trimRanges,
+ rulerSec,
+ readSpeedRegions(document),
+ );
+}
diff --git a/skills/openscreen-demo/SKILL.md b/skills/openscreen-demo/SKILL.md
index 48e142a61..40e2491ec 100644
--- a/skills/openscreen-demo/SKILL.md
+++ b/skills/openscreen-demo/SKILL.md
@@ -40,7 +40,7 @@ Write a storyboard and show it to the user in a few lines, then carry on unless
- Start the app and open it on the starting state of the first beat.
- Give it one clean window of about 1600×1000. For a web app, open it in an app window without browser chrome: `chrome --app=http://localhost:3000 --window-size=1600,1000 --user-data-dir=<temp folder>`. Its title is the page's `<title>`.
- Call `listSources` and pick a unique part of the window's title.
-- Close or hide anything personal. Recording one window keeps the rest of the desktop out.
+- **Only the app on screen.** No browser toolbar, address bar, local file path, user name, bookmark bar, translation prompt or notification in the recorded window. Recording one window keeps the rest of the desktop out.
## 4. Record
@@ -49,7 +49,7 @@ Write a storyboard and show it to the user in a few lines, then carry on unless
3. `stopRecording` with a `title`. Keep what it returns: `projectId`, `checkpointId`, `videoPath`, `recordingStartedAt`.
**How you drive the app decides the cursor.**
-- **Real mouse** (computer-use, or any tool that moves the actual pointer): OpenScreen records the pointer, draws a clean cursor and already zooms on each click.
+- **Real mouse** (computer-use, or any tool that moves the actual pointer): OpenScreen records the pointer, draws a clean cursor and already zooms on each click. If your mouse tool cannot click in the app's window (some only read browsers, for one), drive it from a script instead.
- **A script** (Playwright, CDP): the screen pointer never moves, so the take has no cursor and no zoom. In the script, log every action as it runs: `at` is `Date.now()`, `x` and `y` are the action's position as fractions of the window, `click` is true for a click. Hand the log to `setPointerTrack` after `stopRecording`; its description gives the formula from a page's coordinates. A script also gives the cleanest take: same timing on every run, no hesitation.
## 5. Check the take
@@ -58,11 +58,14 @@ Write a storyboard and show it to the user in a few lines, then carry on unless
## 6. Edit
-Use the document tools, always with the `projectId`. Read the project first with `getCurrentDocument`: its notes say which times are in the recording's seconds and which in the edited video's.
+Use the document tools, always with the `projectId`. Read the project first with `getCurrentDocument`: its notes say which times are in the recording's seconds and which on the editor's ruler.
+**One clock.** Place every effect in ruler seconds, and look at the result with `getFrames` and the `projectId`, which takes the same seconds. Never read times off the exported file: a cut or a speed-up shifts its clock away from the ruler.
+
+- **Format first:** `setAspectRatio` for where the video will be shown, `16:9` for X. Do it before placing text.
- **Cut dead time:** the setup at the start, the stop at the end, any wait. Use `addTrims`, or `addSpeed` to fast-forward a wait instead of cutting it.
-- **Zoom on the action:** the clicks already have zooms. Check them, then add or move zooms with `addZooms` and `setZoom` where a result appears without a click. Hold a zoom at least 1.5 s.
-- **Callouts:** one `addAnnotation` per beat, short text placed clear of the action.
+- **Zoom on the action:** clicks close together share one zoom, and the first 2.5 s stay wide. Check them, then add or move zooms with `addZooms` and `setZoom` where a result deserves its own. Hold a zoom at least 1.5 s.
+- **Callouts:** one `addAnnotation` per beat, short, placed clear of the action. Text sits on the output frame and stays put when the picture zooms. Use its look for hierarchy: a large title with `animation: "typewriter"`, step callouts with `animation: "pop"`. `setAnnotation` moves or restyles one.
- **Undo:** `restoreCheckpoint` with the `checkpointId` takes the project back to the raw take in one step.
Keep the rest as it is: the background, padding, shadow and cursor style are chosen to look good.
@@ -70,7 +73,7 @@ Keep the rest as it is: the background, padding, shadow and cursor style are cho
## 7. Export and review
1. `exportVideo` with the `projectId` and a path such as `demo/demo.mp4`.
-2. `getFrames` on the export, at the moments of your beats.
+2. `getFrames` with the `projectId`, at the ruler seconds of your beats.
3. Check every frame:
- Is each callout readable and clear of the action?
- Is each zoom on the thing that changes?