Skip to content

Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.

feat(mcp): rename a project, import and remove its media, create one

EtienneLescot37b373b3b94f
diff --git a/electron/ai-edition/agent-tools.ts b/electron/ai-edition/agent-tools.ts
index 34da2f2cd..e6bf21585 100644
--- a/electron/ai-edition/agent-tools.ts
+++ b/electron/ai-edition/agent-tools.ts
@@ -1242,7 +1242,7 @@ export function documentSnapshotForModel(
timeBaseNote:
"clips and trims are in source-time seconds; zooms, speedRegions, annotations, cameraFullscreenRegions and audioTracks are in virtual (edited-timeline) seconds: the editor's ruler, where a trim leaves a hole instead of shortening it and a speed region stretches nothing. The exported video's clock skips those holes and plays sped-up spans faster, so it differs from the ruler once there are any.",
audioNote:
- "audioTracks are imported voiceover / music files laid over the recording. They are clip-anchored like every other region, so they travel with their clip through reorder and trim, and they play at 1x whatever a speed region does to the picture under them. addAudio places an EXISTING asset of kind 'audio'; nothing here can import a file from disk or record one, so if the project has no audio asset, say so rather than inventing an id.",
+ "audioTracks are imported voiceover / music files laid over the recording. They are clip-anchored like every other region, so they travel with their clip through reorder and trim, and they play at 1x whatever a speed region does to the picture under them. addAudio places an EXISTING asset of kind 'audio': if the project has none, a file has to be imported first (importMedia, where it is offered), so never invent an id.",
zoomNote:
`renderedScale is what the viewer sees (depth is an ordinal, not a factor: ${ZOOM_DEPTH_LEGEND}). ` +
"When a zoom carries customScale it wins over depth and depthIsOverridden is true — " +
@@ -2975,7 +2975,7 @@ export function executeAgentTool(
`Unknown asset: ${assetId}.` +
(available.length
? ` Imported audio in this project: ${available.map((a) => `${a.id} (${a.label})`).join(", ")}.`
- : " This project has no imported audio; a file can only be imported or recorded from the editor, not from here."),
+ : " This project has no imported audio: a file has to be imported into it first (importMedia, where it is offered)."),
);
}
if (asset.kind !== "audio") {
diff --git a/electron/ai-edition/deep-agent/service.ts b/electron/ai-edition/deep-agent/service.ts
index 53d122e10..fa610515f 100644
--- a/electron/ai-edition/deep-agent/service.ts
+++ b/electron/ai-edition/deep-agent/service.ts
@@ -223,7 +223,7 @@ export const TOOL_DESCRIPTIONS: Record<string, string> = {
"Add a camera-fullscreen region over a span of the edited timeline (virtual seconds): the webcam fills the frame for that span. This only does something when the footage under that span comes from an asset with a linked webcam — check assets[].hasCameraTrack (or hasAnyCamera) in getCurrentDocument first. On footage with no camera the call is refused rather than storing a region that would render nothing; say so instead of retrying.",
setCameraFullscreen:
"Move or resize an existing camera-fullscreen region by id (virtual-timeline seconds). Only the fields you pass are changed. Refused if the new span lands on footage with no linked webcam.",
- addAudio: `Lay an ALREADY-IMPORTED audio file over the recording across a span of the edited timeline (virtual seconds): a voiceover, or a music bed. assetId must name an asset whose kind is 'audio' — getCurrentDocument lists them; nothing here can import a file from disk or record one, so if there is none, say so instead of guessing an id. Omit endSec to play the whole file from offsetSec. kind picks the lane ('voiceover' or 'music'). offsetSec is where in the FILE playback starts, gainDb its level (0 unchanged, negative ducks it), fadeInMs / fadeOutMs its fades (0 to ${AUDIO_FADE_MAX_MS} ms, default none; a track too short for both shares its span between them). A voiceover-lane track is also what gets transcribed, so the lane is not only cosmetic.`,
+ addAudio: `Lay an ALREADY-IMPORTED audio file over the recording across a span of the edited timeline (virtual seconds): a voiceover, or a music bed. assetId must name an asset whose kind is 'audio' — getCurrentDocument lists them. If there is none, a file has to be imported first (importMedia, where it is offered): never guess an id. Omit endSec to play the whole file from offsetSec. kind picks the lane ('voiceover' or 'music'). offsetSec is where in the FILE playback starts, gainDb its level (0 unchanged, negative ducks it), fadeInMs / fadeOutMs its fades (0 to ${AUDIO_FADE_MAX_MS} ms, default none; a track too short for both shares its span between them). A voiceover-lane track is also what gets transcribed, so the lane is not only cosmetic.`,
setAudio: `Move, resize, re-level, re-lane, mute, loop, fade or re-point an existing audio track by id (virtual-timeline seconds). Only the fields you pass are changed. Use it to duck a bed under narration (gainDb), to fade it in or out (fadeInMs / fadeOutMs, 0 to ${AUDIO_FADE_MAX_MS} ms), to shift what part of the file plays (offsetSec), or to move it between the voiceover and music lanes (kind). The whole track is edited, not one fragment of it, so a track split across a cut stays one thing.`,
removeTrim:
"Delete a trim range by id — the cut is undone and that span plays/exports again. This is how you 'remove a trim'; never re-add a trim to undo one.",
diff --git a/electron/ai-edition/document-service.ts b/electron/ai-edition/document-service.ts
index 8fea6d027..0d44274af 100644
--- a/electron/ai-edition/document-service.ts
+++ b/electron/ai-edition/document-service.ts
@@ -149,6 +149,75 @@ function isSupportedAudioPath(filePath: string): boolean {
return SUPPORTED_AUDIO_EXTENSIONS.has(ext);
}
+/** Why `input` cannot be filed as its kind of asset, or `null` when it can. */
+export function unsupportedAssetPath(input: Pick<AddAssetInput, "path" | "kind">): string | null {
+ if (!input.path) return "Asset path is required.";
+ if ((input.kind ?? "video") === "audio") {
+ return isSupportedAudioPath(input.path)
+ ? null
+ : `Unsupported audio extension: ${path.extname(input.path)} (supported: ${[...SUPPORTED_AUDIO_EXTENSIONS].join(", ")})`;
+ }
+ return isSupportedVideoPath(input.path)
+ ? null
+ : `Unsupported video extension: ${path.extname(input.path)} (supported: ${[...SUPPORTED_VIDEO_EXTENSIONS].join(", ")})`;
+}
+
+/** The kind a media file is imported as, from its extension: audio only when it cannot be a video. */
+export function assetKindForPath(filePath: string): "video" | "audio" {
+ return !isSupportedVideoPath(filePath) && isSupportedAudioPath(filePath) ? "audio" : "video";
+}
+
+/** `doc` with `asset` added to its media, as an import adds it. */
+export function documentWithAsset(doc: AxcutDocument, asset: AxcutAsset): AxcutDocument {
+ // An audio import is an overlay, never the thing the timeline is built
+ // around, so it must not claim the empty primaryAssetId slot — otherwise the
+ // first file dropped into a fresh project (a BGM track) would become its
+ // primary asset and the editor would try to lay out clips from a file with
+ // no video.
+ const claimsPrimary = asset.kind !== "audio" && !doc.project.primaryAssetId;
+ return {
+ ...doc,
+ assets: [...doc.assets, asset],
+ project: {
+ ...doc.project,
+ ...(claimsPrimary ? { primaryAssetId: asset.id } : {}),
+ updatedAt: new Date().toISOString(),
+ },
+ };
+}
+
+/** `doc` without the asset, its clips, trims and audio tracks. */
+export function documentWithoutAsset(doc: AxcutDocument, assetId: string): AxcutDocument {
+ const assets = doc.assets.filter((a) => a.id !== assetId);
+ // Primary is the thing the timeline is built around, so it must fall to the
+ // next VIDEO asset — never an audio overlay (issue #350), which can't be
+ // primary (see addAsset). Falling back to `assets[0]` would hand primary to
+ // an audio asset when the removed one was the last video.
+ const primaryAssetId =
+ doc.project.primaryAssetId === assetId
+ ? (assets.find((a) => a.kind !== "audio")?.id ?? undefined)
+ : doc.project.primaryAssetId;
+ const withoutAssetClips = doc.timeline.clips
+ .filter((clip) => clip.assetId === assetId)
+ .reduce((current, clip) => removeClip(current, clip.id), doc);
+ return {
+ ...withoutAssetClips,
+ assets,
+ // Drop imported audio tracks that referenced the removed asset — they
+ // would otherwise dangle, pointing at an asset the document no longer has.
+ audioTracks: withoutAssetClips.audioTracks.filter((t) => t.assetId !== assetId),
+ timeline: {
+ ...withoutAssetClips.timeline,
+ trimRanges: withoutAssetClips.timeline.trimRanges.filter((r) => r.assetId !== assetId),
+ },
+ project: {
+ ...withoutAssetClips.project,
+ primaryAssetId,
+ updatedAt: new Date().toISOString(),
+ },
+ };
+}
+
function safeProjectId(raw: string): string {
// ponytail: project ids are uuid-prefixed strings (e.g. "proj_<uuid>"). Reject
// anything that smells like path traversal before we ever touch the disk.
@@ -532,23 +601,8 @@ export class DocumentService {
async addAsset(projectId: string, input: AddAssetInput): Promise<AxcutDocument> {
const doc = await this.getProject(projectId);
- if (!input.path) {
- throw new ProjectFileError("Asset path is required.", projectId);
- }
- const kind = input.kind ?? "video";
- if (kind === "audio") {
- if (!isSupportedAudioPath(input.path)) {
- throw new ProjectFileError(
- `Unsupported audio extension: ${path.extname(input.path)} (supported: ${[...SUPPORTED_AUDIO_EXTENSIONS].join(", ")})`,
- projectId,
- );
- }
- } else if (!isSupportedVideoPath(input.path)) {
- throw new ProjectFileError(
- `Unsupported video extension: ${path.extname(input.path)} (supported: ${[...SUPPORTED_VIDEO_EXTENSIONS].join(", ")})`,
- projectId,
- );
- }
+ const unsupported = unsupportedAssetPath(input);
+ if (unsupported) throw new ProjectFileError(unsupported, projectId);
const absolutePath = path.isAbsolute(input.path) ? input.path : path.resolve(input.path);
// P3.1 — capture the file size at import. Non-fatal: a stat failure
// (network drive, permissions) just leaves sizeBytes undefined.
@@ -560,28 +614,13 @@ export class DocumentService {
}
const asset: AxcutAsset = {
id: createId("asset"),
- kind,
+ kind: input.kind ?? "video",
label: input.label?.trim() || path.basename(absolutePath),
originalPath: absolutePath,
sizeBytes,
cameraTrack: null,
};
- // An audio import is an overlay, never the thing the timeline is built
- // around, so it must not claim the empty primaryAssetId slot — otherwise the
- // first file dropped into a fresh project (a BGM track) would become its
- // primary asset and the editor would try to lay out clips from a file with
- // no video.
- const claimsPrimary = kind !== "audio" && !doc.project.primaryAssetId;
- const next: AxcutDocument = {
- ...doc,
- assets: [...doc.assets, asset],
- project: {
- ...doc.project,
- ...(claimsPrimary ? { primaryAssetId: asset.id } : {}),
- updatedAt: new Date().toISOString(),
- },
- };
- return this.saveProject(next);
+ return this.saveProject(documentWithAsset(doc, asset));
}
async removeAsset(projectId: string, assetId: string): Promise<AxcutDocument> {
@@ -589,35 +628,7 @@ export class DocumentService {
if (!doc.assets.some((a) => a.id === assetId)) {
throw new ProjectFileError(`Asset ${assetId} not found in project ${projectId}.`, projectId);
}
- const assets = doc.assets.filter((a) => a.id !== assetId);
- // Primary is the thing the timeline is built around, so it must fall to the
- // next VIDEO asset — never an audio overlay (issue #350), which can't be
- // primary (see addAsset). Falling back to `assets[0]` would hand primary to
- // an audio asset when the removed one was the last video.
- const primaryAssetId =
- doc.project.primaryAssetId === assetId
- ? (assets.find((a) => a.kind !== "audio")?.id ?? undefined)
- : doc.project.primaryAssetId;
- const withoutAssetClips = doc.timeline.clips
- .filter((clip) => clip.assetId === assetId)
- .reduce((current, clip) => removeClip(current, clip.id), doc);
- const next: AxcutDocument = {
- ...withoutAssetClips,
- assets,
- // Drop imported audio tracks that referenced the removed asset — they
- // would otherwise dangle, pointing at an asset the document no longer has.
- audioTracks: withoutAssetClips.audioTracks.filter((t) => t.assetId !== assetId),
- timeline: {
- ...withoutAssetClips.timeline,
- trimRanges: withoutAssetClips.timeline.trimRanges.filter((r) => r.assetId !== assetId),
- },
- project: {
- ...withoutAssetClips.project,
- primaryAssetId,
- updatedAt: new Date().toISOString(),
- },
- };
- return this.saveProject(next);
+ return this.saveProject(documentWithoutAsset(doc, assetId));
}
/**
diff --git a/electron/ipc/handlers.ts b/electron/ipc/handlers.ts
index 8cd32244d..426532884 100644
--- a/electron/ipc/handlers.ts
+++ b/electron/ipc/handlers.ts
@@ -5060,6 +5060,13 @@ export function registerIpcHandlers(
cursor: agentCursorTelemetryReader,
cameraSegmentation: () => loudnessService.probeSegmentation(),
version: app.getVersion(),
+ // importMedia: the editor's own probe, and the grant a picked file gets, so the editor
+ // can play a file a client imported into the project it has open.
+ openVideo: openVideoInHiddenWindow,
+ approveMedia: (filePath) => {
+ const media = normalizeVideoSourcePath(filePath);
+ if (media && hasAllowedImportMediaExtension(media)) approveFilePath(media);
+ },
};
// One runner, so one queue and one checkpoint store, for the in-app server and every
// `openscreen mcp` the app answers through its channel (app-channel.ts, started by main.ts).
diff --git a/electron/mcp/app-channel.ts b/electron/mcp/app-channel.ts
index 5a2a56f72..09d7f6be4 100644
--- a/electron/mcp/app-channel.ts
+++ b/electron/mcp/app-channel.ts
@@ -45,6 +45,7 @@ import {
type McpToolDeps,
type McpToolRunner,
} from "./openscreen-mcp-server";
+import { PROJECT_EDIT_TOOLS } from "./project-tools";
const DISCOVERY_FILE = "mcp-app.json";
/** Connect, token and MCP handshake: the app answers in milliseconds when it is alive. */
@@ -260,7 +261,7 @@ export async function connectToApp(
function isReadOnly(name: string): boolean {
if (name === LIST_PROJECTS_TOOL) return true;
if (name === "createCheckpoint" || name === "restoreCheckpoint") return false;
- return !isMutatingTool(name);
+ return !isMutatingTool(name) && !PROJECT_EDIT_TOOLS.has(name);
}
/**
diff --git a/electron/mcp/media-tools.test.ts b/electron/mcp/media-tools.test.ts
index 3b9c01d76..9ad599c27 100644
--- a/electron/mcp/media-tools.test.ts
+++ b/electron/mcp/media-tools.test.ts
@@ -309,6 +309,48 @@ describe("recording", () => {
});
});
+describe("createProject", () => {
+ it("creates an empty project the document tools can edit at once", async () => {
+ const created = await call("createProject", { title: " Blank " });
+ expect(created.isError).toBeFalsy();
+ const { projectId, checkpointId } = JSON.parse(textOf(created));
+ expect(checkpointId).toEqual(expect.any(String));
+ const document = await documents.getProject(projectId);
+ expect(document.project.title).toBe("Blank");
+ expect(document.assets).toEqual([]);
+ // Its checkpoint lets an edit land on the file straight away.
+ const renamed = await call("setProjectInfo", { projectId, title: "Named" });
+ expect(renamed.isError).toBeFalsy();
+ expect((await documents.getProject(projectId)).project.title).toBe("Named");
+ });
+
+ it("lays a video out as one clip, titled after the file, with the zooms of its clicks", async () => {
+ await fs.writeFile(videoPath, "video");
+ const created = await call("createProject", { videoPath });
+ expect(created.isError).toBeFalsy();
+ const result = JSON.parse(textOf(created));
+ expect(result).toMatchObject({ durationSec: 12 });
+ expect(result.autoZooms).toBeGreaterThan(0);
+ const document = await documents.getProject(result.projectId);
+ expect(document.project.title).toBe("take");
+ expect(document.timeline.clips).toHaveLength(1);
+ expect(document.assets[0]).toMatchObject({ originalPath: videoPath, durationSec: 12 });
+ });
+
+ it("creates nothing from a path it cannot use", async () => {
+ const missing = await call("createProject", { videoPath: path.join(dir, "gone.mp4") });
+ expect(missing.isError).toBe(true);
+ expect(textOf(missing)).toContain("No file");
+ const relative = await call("createProject", { videoPath: "take.mp4" });
+ expect(textOf(relative)).toContain("must be absolute");
+ await fs.writeFile(videoPath, "video");
+ openVideoFails = true;
+ const unreadable = await call("createProject", { videoPath });
+ expect(textOf(unreadable)).toContain("media error 4");
+ expect(await documents.listProjects()).toEqual([]);
+ });
+});
+
describe("pointerSamples", () => {
const startedAt = 10_000;
const samples = pointerSamples(
@@ -716,7 +758,7 @@ describe("the media tools in the app", () => {
allowed = false;
const app = await connectApp();
expect(app.getInstructions()).toContain(
- "startRecording and exportVideo are refused too while 'MCP edits'",
+ "startRecording, exportVideo and createProject are refused too while 'MCP edits'",
);
const started = await callOn(app, "startRecording", {});
expect(started.isError).toBe(true);
@@ -728,6 +770,9 @@ describe("the media tools in the app", () => {
});
expect(textOf(exported)).toContain("'MCP edits'");
expect(runners.started).toEqual([]);
+ const created = await callOn(app, "createProject", {});
+ expect(textOf(created)).toContain("'MCP edits'");
+ expect(await documents.listProjects()).toEqual([]);
// Reading stays open: listing what could be recorded changes nothing.
runners.results.sources = {
success: true,
diff --git a/electron/mcp/media-tools.ts b/electron/mcp/media-tools.ts
index 675abff3b..745526707 100644
--- a/electron/mcp/media-tools.ts
+++ b/electron/mcp/media-tools.ts
@@ -40,6 +40,7 @@ import {
type RunnerEdit,
textResult as text,
} from "./openscreen-mcp-server";
+import { mediaPathProblem } from "./project-tools";
/** What startRecording asks a recorder for. */
export type RecordRequest = Pick<
@@ -192,6 +193,7 @@ export function mediaPreamble(transport: McpTransport): string {
return [
"OpenScreen records the screen, turns each take into a project you edit with the document tools, and exports it as a video.",
"A demo goes: listSources, then startRecording, then drive the app, then stopRecording. It returns the projectId and a checkpointId. Edit with the document tools, always passing that projectId: every edit is saved to the project straight away, and restoreCheckpoint with the checkpointId takes the project back to the raw take in one step. Then exportVideo, and getFrames with the projectId to check the export before you hand it over: it takes the same ruler seconds as the document tools, so a trim or a speed region never shifts what you look at.",
+ "To edit a video you already have instead, createProject with its videoPath; importMedia adds a video or an audio file to a project.",
"Drive the app with the real mouse when you can: OpenScreen records the pointer, draws a clean cursor and zooms on the clicks as the editor does: clicks close in time and on screen share one zoom, and the first seconds and the end stay on the whole screen. If you drive it from a script instead (Playwright, CDP), the screen pointer never moves: log each action as the script runs it, then hand the log to setPointerTrack after stopRecording.",
TAKE_NOTES[transport],
].join("\n");
@@ -216,6 +218,11 @@ const STOP_RECORDING_DESCRIPTIONS: Record<McpTransport, string> = {
"Stop the recording and turn the take into a new project, laid out as the editor lays out a fresh recording: one clip, plus the zooms planned from the recorded clicks. Not one per click: clicks close in time and on screen share one zoom, a click in the first seconds or the one that stopped the take gets none, and autoZooms says how many there are. Returns the projectId to edit, a checkpointId that restores the raw take, and the take's videoPath.",
};
+const CREATE_PROJECT_NOTES: Record<McpTransport, string> = {
+ http: "The editor keeps the project it has open: the new one is in OpenScreen's project list, for the user to open.",
+ stdio: "It is in the OpenScreen app's project list, so the user can open it in its editor.",
+};
+
const EXPORT_PATH_DESCRIPTIONS: Record<McpTransport, string> = {
http: "Where to write the video: an absolute path ending in .mp4 or .gif. This server runs inside the app, so it has no working directory of yours to resolve a relative one against.",
stdio:
@@ -434,6 +441,15 @@ export function createMediaTools(deps: MediaToolDeps): MediaTools {
/** Each project's last export of this session, and when it was made: what getFrames reads. */
const lastExports = new Map<string, { path: string; at: Date; rendered: AxcutDocument }>();
+ /** A fresh project's checkpoint, so a client can edit it at once and take it back in one step. */
+ async function checkpointOf(projectId: string): Promise<string | null> {
+ const checkpoint = await deps.runTool("createCheckpoint", { projectId });
+ const checkpointText = checkpoint.content[0]?.type === "text" ? checkpoint.content[0].text : "";
+ return checkpoint.isError
+ ? null
+ : ((JSON.parse(checkpointText) as { checkpointId?: string }).checkpointId ?? null);
+ }
+
/** Stops the take, then hands back its project with a checkpoint of it. */
async function stopTake(
running: RunningTake,
@@ -447,15 +463,10 @@ export function createMediaTools(deps: MediaToolDeps): MediaTools {
}
startedAtByVideo.set(recorded.videoPath, running.startedAt);
const { projectId } = recorded;
- const checkpoint = await deps.runTool("createCheckpoint", { projectId });
- const checkpointText = checkpoint.content[0]?.type === "text" ? checkpoint.content[0].text : "";
- const checkpointId = checkpoint.isError
- ? null
- : (JSON.parse(checkpointText) as { checkpointId?: string }).checkpointId;
return text(
JSON.stringify({
projectId,
- checkpointId,
+ checkpointId: await checkpointOf(projectId),
videoPath: recorded.videoPath,
durationSec: recorded.durationSec,
recordingStartedAt: running.startedAt,
@@ -794,6 +805,70 @@ export function createMediaTools(deps: MediaToolDeps): MediaTools {
}
},
);
+
+ server.registerTool(
+ "createProject",
+ {
+ description: [
+ "Create a new project, in the look new projects start with in OpenScreen (the style preset marked for them, else the last edited project's), as the editor's New does. Empty, or from a video file laid out as one clip, as the editor lays out an opened video, with zooms planned from the clicks when the video is an OpenScreen recording with its pointer track; autoZooms says how many.",
+ `Returns the projectId to edit and a checkpointId that restores the project as it was created. Fill an empty one with importMedia. ${CREATE_PROJECT_NOTES[transport]}`,
+ ].join("\n"),
+ inputSchema: z.object({
+ title: z
+ .string()
+ .optional()
+ .describe(
+ "The project's title in OpenScreen. Default: the video's file name, or Untitled Project.",
+ ),
+ videoPath: z
+ .string()
+ .optional()
+ .describe(
+ "Absolute path of a video (.mp4, .mov, .m4v, .webm, .mkv, .avi, .wmv) to start from. Omit it for an empty project.",
+ ),
+ }),
+ annotations: { readOnlyHint: false, destructiveHint: false, openWorldHint: false },
+ },
+ async (args) => {
+ if (!deps.editsAllowed()) {
+ return text(
+ `No project was created: OpenScreen lets MCP clients create projects only while the user allows it. Ask the user to turn on ${MCP_EDITS_SETTING}.`,
+ true,
+ );
+ }
+ const title = args.title?.trim();
+ if (args.videoPath === undefined) {
+ const created = await deps.documents.createProject(title ?? "");
+ const projectId = created.project.id;
+ return text(JSON.stringify({ projectId, checkpointId: await checkpointOf(projectId) }));
+ }
+ const videoPath = args.videoPath;
+ const problem = await mediaPathProblem(videoPath, "video");
+ if (problem) return text(problem, true);
+ let project: Awaited<ReturnType<typeof createProjectFromVideo>>;
+ try {
+ project = await createProjectFromVideo(
+ deps,
+ videoPath,
+ title || path.parse(videoPath).name,
+ );
+ } catch (error) {
+ return text(
+ `No project was created: ${error instanceof Error ? error.message : String(error)}`,
+ true,
+ );
+ }
+ const projectId = project.document.project.id;
+ return text(
+ JSON.stringify({
+ projectId,
+ checkpointId: await checkpointOf(projectId),
+ durationSec: project.durationSec,
+ autoZooms: project.autoZooms,
+ }),
+ );
+ },
+ );
}
return { preamble: mediaPreamble(transport), register };
diff --git a/electron/mcp/openscreen-mcp-server.test.ts b/electron/mcp/openscreen-mcp-server.test.ts
index edb226ff5..85e0fbf2d 100644
--- a/electron/mcp/openscreen-mcp-server.test.ts
+++ b/electron/mcp/openscreen-mcp-server.test.ts
@@ -24,6 +24,7 @@ import {
MCP_ENDPOINT_PATH,
type McpApplyResult,
type McpDocumentHost,
+ type McpToolDeps,
mcpInstructions,
type RunningMcpServer,
startMcpHttpServer,
@@ -141,7 +142,11 @@ afterEach(async () => {
async function connect(
editor: FakeEditor,
- options: { editsAllowed?: boolean; headers?: Record<string, string> } = {},
+ options: {
+ editsAllowed?: boolean;
+ headers?: Record<string, string>;
+ media?: Pick<McpToolDeps, "openVideo" | "approveMedia">;
+ } = {},
) {
const editsAllowed = () => options.editsAllowed ?? true;
const deps = {
@@ -150,6 +155,7 @@ async function connect(
projects,
editsAllowed,
version: "0.0.0",
+ ...options.media,
};
const unused = () => {
throw new Error("not used by these tests");
@@ -192,6 +198,9 @@ describe("the MCP tool surface", () => {
expect(tools.map((t) => t.name)).toEqual([
LIST_PROJECTS_TOOL,
...OPENSCREEN_TOOL_NAMES,
+ "setProjectInfo",
+ "importMedia",
+ "removeMedia",
"createCheckpoint",
"restoreCheckpoint",
// The same as `openscreen mcp` serves (#1174).
@@ -201,6 +210,7 @@ describe("the MCP tool surface", () => {
"setPointerTrack",
"exportVideo",
"getFrames",
+ "createProject",
]);
for (const tool of tools.slice(1, 1 + OPENSCREEN_TOOL_NAMES.length)) {
expect(tool.description).toBe(TOOL_DESCRIPTIONS[tool.name]);
@@ -231,6 +241,8 @@ describe("the MCP tool surface", () => {
expect(byName.get(LIST_PROJECTS_TOOL)?.readOnlyHint).toBe(true);
expect(byName.get("addTrim")?.readOnlyHint).toBe(false);
expect(byName.get("removeClip")?.destructiveHint).toBe(true);
+ expect(byName.get("removeMedia")?.destructiveHint).toBe(true);
+ expect(byName.get("importMedia")?.readOnlyHint).toBe(false);
expect(byName.get("createCheckpoint")?.readOnlyHint).toBe(true);
expect(byName.get("restoreCheckpoint")?.destructiveHint).toBe(true);
});
@@ -247,7 +259,7 @@ describe("the MCP tool surface", () => {
"Without a projectId they act on the project open in the editor",
);
expect(instructions).toContain("startRecording hands over to OpenScreen's recording bar");
- expect(instructions).toContain("startRecording and exportVideo are refused too");
+ expect(instructions).toContain("startRecording, exportVideo and createProject are refused too");
expect(instructions).toContain("all of them are lost when OpenScreen quits.");
expect(instructions).toContain("turn on 'MCP edits' under MCP server in AI settings");
expect(instructions).not.toContain("Agent edits");
@@ -462,6 +474,147 @@ describe("checkpoints", () => {
});
});
+describe("the project's name and media", () => {
+ function probe(durationSec = 12) {
+ return vi.fn(async () => ({
+ durationSec,
+ width: 1920,
+ height: 1080,
+ close: () => undefined,
+ capture: async () => ({ mimeType: "image/jpeg" as const, data: "" }),
+ }));
+ }
+
+ function mediaFile(name: string): string {
+ const file = path.join(dir, name);
+ writeFileSync(file, "media");
+ return file;
+ }
+
+ it("renames the open project through the editor, trimmed as the editor commits it", async () => {
+ const editor = new FakeEditor();
+ const mcp = await connect(editor);
+ const result = await mcp.callTool({
+ name: "setProjectInfo",
+ arguments: { title: " Launch demo " },
+ });
+ expect(result.isError).toBeFalsy();
+ expect(editor.applied).toHaveLength(1);
+ expect(editor.document?.project.title).toBe("Launch demo");
+
+ const empty = await mcp.callTool({ name: "setProjectInfo", arguments: { title: " " } });
+ expect(empty.isError).toBe(true);
+ expect(editor.applied).toHaveLength(1);
+ });
+
+ it("imports a video into the open project, measured and readable by the editor", async () => {
+ const editor = new FakeEditor();
+ const openVideo = probe();
+ const approveMedia = vi.fn();
+ const mcp = await connect(editor, { media: { openVideo, approveMedia } });
+ const file = mediaFile("broll.mp4");
+ const result = await mcp.callTool({ name: "importMedia", arguments: { path: file } });
+ expect(result.isError).toBeFalsy();
+ const { assetId, kind, durationSec } = JSON.parse(resultText(result));
+ expect([kind, durationSec]).toEqual(["video", 12]);
+ expect(editor.applied).toHaveLength(1);
+ const asset = editor.document?.assets.find((a) => a.id === assetId);
+ expect(asset).toMatchObject({ originalPath: file, label: "broll.mp4", durationSec: 12 });
+ expect(asset?.video).toMatchObject({ width: 1920, height: 1080 });
+ // The project already has its footage: an import never takes its place.
+ expect(editor.document?.project.primaryAssetId).toBe("asset_1");
+ expect(approveMedia).toHaveBeenCalledWith(file);
+ });
+
+ it("files an audio-only extension as audio, and still imports a file it cannot measure", async () => {
+ const editor = new FakeEditor();
+ const openVideo = vi.fn(async () => {
+ throw new Error("media error 4");
+ });
+ const mcp = await connect(editor, { media: { openVideo } });
+ const result = await mcp.callTool({
+ name: "importMedia",
+ arguments: { path: mediaFile("voice.mp3"), label: "Voiceover" },
+ });
+ expect(result.isError).toBeFalsy();
+ expect(JSON.parse(resultText(result))).toMatchObject({
+ kind: "audio",
+ label: "Voiceover",
+ durationSec: null,
+ });
+ expect(editor.document?.assets.at(-1)?.kind).toBe("audio");
+ });
+
+ it("refuses a relative, missing or unsupported path before probing anything", async () => {
+ const editor = new FakeEditor();
+ const openVideo = probe();
+ const mcp = await connect(editor, { media: { openVideo } });
+ for (const [filePath, says] of [
+ ["broll.mp4", "must be absolute"],
+ [path.join(dir, "missing.mp4"), "No file"],
+ [mediaFile("notes.txt"), "Unsupported video extension"],
+ ]) {
+ const result = await mcp.callTool({ name: "importMedia", arguments: { path: filePath } });
+ expect(result.isError).toBe(true);
+ expect(resultText(result)).toContain(says);
+ }
+ expect(openVideo).not.toHaveBeenCalled();
+ expect(editor.applied).toHaveLength(0);
+ });
+
+ it("imports nothing while MCP edits is off", async () => {
+ const editor = new FakeEditor();
+ const openVideo = probe();
+ const mcp = await connect(editor, { editsAllowed: false, media: { openVideo } });
+ const refused = await mcp.callTool({
+ name: "importMedia",
+ arguments: { path: mediaFile("ok.mp4") },
+ });
+ expect(refused.isError).toBe(true);
+ expect(resultText(refused)).toContain("'MCP edits'");
+ expect(openVideo).not.toHaveBeenCalled();
+ expect(editor.applied).toHaveLength(0);
+ });
+
+ it("removes media only once the timeline no longer uses it", async () => {
+ const editor = new FakeEditor();
+ const mcp = await connect(editor, { media: { openVideo: probe() } });
+ const inUse = await mcp.callTool({ name: "removeMedia", arguments: { assetId: "asset_1" } });
+ expect(inUse.isError).toBe(true);
+ expect(resultText(inUse)).toContain("clip_1 (removeClip)");
+ expect(editor.applied).toHaveLength(0);
+
+ const imported = await mcp.callTool({
+ name: "importMedia",
+ arguments: { path: mediaFile("unused.mov") },
+ });
+ const { assetId } = JSON.parse(resultText(imported));
+ const removed = await mcp.callTool({ name: "removeMedia", arguments: { assetId } });
+ expect(removed.isError).toBeFalsy();
+ expect(editor.document?.assets.map((a) => a.id)).toEqual(["asset_1"]);
+ });
+
+ it("edits a project that is not open on its file, once it has a checkpoint", async () => {
+ await projects.saveProject(fixtureDocument("proj_2"));
+ const editor = new FakeEditor();
+ const mcp = await connect(editor, { media: { openVideo: probe() } });
+ const args = { projectId: "proj_2", path: mediaFile("music.wav") };
+ const refused = await mcp.callTool({ name: "importMedia", arguments: args });
+ expect(resultText(refused)).toContain("createCheckpoint");
+ await mcp.callTool({ name: "createCheckpoint", arguments: { projectId: "proj_2" } });
+ const result = await mcp.callTool({ name: "importMedia", arguments: args });
+ expect(result.isError).toBeFalsy();
+ await mcp.callTool({
+ name: "setProjectInfo",
+ arguments: { projectId: "proj_2", title: "Renamed" },
+ });
+ const saved = await projects.getProject("proj_2");
+ expect(saved.assets.map((a) => a.kind)).toEqual(["video", "audio"]);
+ expect(saved.project.title).toBe("Renamed");
+ expect(editor.applied).toHaveLength(0);
+ });
+});
+
describe("projects other than the open one", () => {
const addTrimArgs = { assetId: "asset_1", startSec: 5, endSec: 6 };
diff --git a/electron/mcp/openscreen-mcp-server.ts b/electron/mcp/openscreen-mcp-server.ts
index 4e4b3d02b..dd4688a1f 100644
--- a/electron/mcp/openscreen-mcp-server.ts
+++ b/electron/mcp/openscreen-mcp-server.ts
@@ -24,6 +24,8 @@
// back to where the turn started. For a project that is not open they are the
// only way back, so an edit to one is refused until it has a checkpoint. The
// in-app agent has no need for them: its whole turn is already a single apply.
+// Nor for the project's own edits (project-tools.ts: its name, its media), which
+// reach a file on disk the in-app agent has no way to name.
//
// Two transports serve these tools: the in-app server over HTTP (below) and
// `openscreen mcp` over stdio (`stdio-server.ts`), both with the media tools around
@@ -69,6 +71,17 @@ import {
type ProjectVersion,
} from "../ai-edition/document-service";
import type { MediaTools } from "./media-tools";
+import {
+ IMPORT_MEDIA_TOOL,
+ type ImportMediaDeps,
+ PROJECT_EDIT_TOOLS,
+ PROJECT_TOOLS,
+ prepareImportMedia,
+ REMOVE_MEDIA_TOOL,
+ removeMedia,
+ SET_PROJECT_INFO_TOOL,
+ setProjectInfo,
+} from "./project-tools";
export const MCP_SERVER_NAME = "openscreen";
export const MCP_ENDPOINT_PATH = "/mcp";
@@ -125,7 +138,7 @@ type ProjectRead = Awaited<ReturnType<McpProjectStore["getProjectForUpdate"]>>;
*/
export type McpTransport = "http" | "stdio";
-export interface McpToolDeps {
+export interface McpToolDeps extends ImportMediaDeps {
transport: McpTransport;
host: McpDocumentHost;
projects: McpProjectStore;
@@ -228,7 +241,9 @@ export function mcpInstructions(options: {
...CORE_PREAMBLES[transport],
CHECKPOINT_LIFETIMES[transport],
...(media && !editsAllowed
- ? [`startRecording and exportVideo are refused too while ${MCP_EDITS_SETTING} is off.`]
+ ? [
+ `startRecording, exportVideo and createProject are refused too while ${MCP_EDITS_SETTING} is off.`,
+ ]
: []),
"",
].join("\n");
@@ -497,10 +512,12 @@ export function createToolRunner(deps: McpToolDeps): McpToolRunner {
if (name === "createCheckpoint") return createCheckpoint(document);
if (name === "restoreCheckpoint") return restoreCheckpoint(document, args, editsAllowed);
// This server's own switch, so the refusal names it rather than the in-app agent's.
- if ((edit || isMutatingTool(name)) && !editsAllowed) {
+ if ((edit || isMutatingTool(name) || PROJECT_EDIT_TOOLS.has(name)) && !editsAllowed) {
return consentRequired(name, args, MCP_EDITS_SETTING);
}
if (edit) return edit(document);
+ if (name === SET_PROJECT_INFO_TOOL) return setProjectInfo(document, args);
+ if (name === REMOVE_MEDIA_TOOL) return removeMedia(document, args);
const availableByAssetId = await probeCursorTelemetry(document, deps.cursor);
return runDocumentTool(document, name, args, editsAllowed, {
cursor: deps.cursor,
@@ -541,11 +558,17 @@ export function createToolRunner(deps: McpToolDeps): McpToolRunner {
return (name, args, options = {}) => {
// The setting is read when the call's turn comes: it may change while it waits.
- const runNow = () =>
- run(name, args, {
- edit: options.edit,
- editsAllowed: options.editsAllowed ?? deps.editsAllowed(),
- });
+ const runNow = async () => {
+ const editsAllowed = options.editsAllowed ?? deps.editsAllowed();
+ let edit = options.edit;
+ // The file is checked and probed before the project is read: see prepareImportMedia.
+ if (name === IMPORT_MEDIA_TOOL && !edit && editsAllowed) {
+ const prepared = await prepareImportMedia(splitProjectId(args).args, deps);
+ if (typeof prepared === "string") return textResult(prepared, true);
+ edit = prepared;
+ }
+ return run(name, args, { edit, editsAllowed });
+ };
const next = queue.then(runNow, runNow);
queue = next.catch(() => undefined);
return next.catch((error) => textResult(`Tool failed: ${errorMessage(error)}`, true));
@@ -594,6 +617,21 @@ export function createOpenScreenMcpServer(
(args: unknown) => runTool(name, args),
);
}
+ for (const tool of PROJECT_TOOLS) {
+ server.registerTool(
+ tool.name,
+ {
+ description: tool.description,
+ inputSchema: tool.inputSchema.extend({ projectId: projectIdArg }),
+ annotations: {
+ readOnlyHint: false,
+ destructiveHint: tool.destructive,
+ openWorldHint: false,
+ },
+ },
+ (args: unknown) => runTool(tool.name, args),
+ );
+ }
for (const tool of checkpointTools(media !== undefined)) {
server.registerTool(
tool.name,
diff --git a/electron/mcp/project-tools.ts b/electron/mcp/project-tools.ts
new file mode 100644
index 000000000..bb51dc398
--- /dev/null
+++ b/electron/mcp/project-tools.ts
@@ -0,0 +1,210 @@
+// The project's own fields and its media, as MCP tools: rename the project, import a
+// file into its media, remove one. Run by the tool runner like any document edit
+// (openscreen-mcp-server.ts), so the project open in the editor changes there, live
+// and one Ctrl/Cmd+Z step, and any other project meets the same checkpoint rule. The
+// app's channel serves them too, so `openscreen mcp` hands them to the running app.
+//
+// Deleting a project is deliberately not here: it cannot be undone, and nothing an
+// agent makes needs it.
+
+import fs from "node:fs/promises";
+import path from "node:path";
+import { z } from "zod";
+import { createId } from "../../src/lib/ai-edition/document/ids";
+import type { AxcutAsset, AxcutDocument } from "../../src/lib/ai-edition/schema";
+import type { AgentToolExecution } from "../ai-edition/agent-tools";
+import {
+ assetKindForPath,
+ documentWithAsset,
+ documentWithoutAsset,
+ unsupportedAssetPath,
+} from "../ai-edition/document-service";
+import type { OpenedVideo } from "./hidden-video";
+
+export const SET_PROJECT_INFO_TOOL = "setProjectInfo";
+export const IMPORT_MEDIA_TOOL = "importMedia";
+export const REMOVE_MEDIA_TOOL = "removeMedia";
+
+/** The runner's edits to a project that are not the document executor's. */
+export const PROJECT_EDIT_TOOLS: ReadonlySet<string> = new Set([
+ SET_PROJECT_INFO_TOOL,
+ IMPORT_MEDIA_TOOL,
+ REMOVE_MEDIA_TOOL,
+]);
+
+const setProjectInfoArgs = z.object({
+ title: z.string().describe("The project's new name."),
+});
+
+const importMediaArgs = z.object({
+ path: z
+ .string()
+ .describe(
+ "Absolute path of the file. Video: .mp4, .mov, .m4v, .webm, .mkv, .avi, .wmv. Audio: .mp3, .wav, .m4a, .aac, .flac, .ogg, .oga, .opus, .webm.",
+ ),
+ kind: z
+ .enum(["video", "audio"])
+ .optional()
+ .describe(
+ "What the file is. Default: audio for an audio-only extension, video otherwise, so pass audio for a .webm that holds only sound.",
+ ),
+ label: z.string().optional().describe("Its name in the project's media. Default: the file name."),
+});
+
+const removeMediaArgs = z.object({
+ assetId: z.string().describe("The asset to remove, from getCurrentDocument's assets."),
+});
+
+export const PROJECT_TOOLS = [
+ {
+ name: SET_PROJECT_INFO_TOOL,
+ description:
+ "Rename the project: its title in OpenScreen's project list and in the editor's top bar, as renaming it there does. Changes nothing else.",
+ inputSchema: setProjectInfoArgs,
+ destructive: false,
+ },
+ {
+ name: IMPORT_MEDIA_TOOL,
+ description:
+ "Add a video or audio file from disk to the project's media, as the editor's Import does. It does not place it on the timeline. An audio file is then played by a track you lay with addAudio. A video becomes the project's footage when it has none yet (a project createProject made without a video): replaceTimeline then lays it out. Returns the new assetId and the file's durationSec, null when OpenScreen could not read it (the editor reads it again when it opens the project).",
+ inputSchema: importMediaArgs,
+ destructive: false,
+ },
+ {
+ name: REMOVE_MEDIA_TOOL,
+ description:
+ "Remove a file from the project's media. Refused while the timeline still uses it: remove its clips (removeClip) and its audio tracks (removeModifier) first. The file on disk is left alone.",
+ inputSchema: removeMediaArgs,
+ destructive: true,
+ },
+] as const;
+
+function failure(message: string): AgentToolExecution {
+ return { ok: false, resultJson: JSON.stringify({ error: message }) };
+}
+
+/** Why `filePath` cannot be imported as `kind`, or `null` when it can. */
+export async function mediaPathProblem(
+ filePath: string,
+ kind: "video" | "audio",
+): Promise<string | null> {
+ if (!path.isAbsolute(filePath)) return `The path must be absolute, got "${filePath}".`;
+ const unsupported = unsupportedAssetPath({ path: filePath, kind });
+ if (unsupported) return unsupported;
+ const stat = await fs.stat(filePath).catch(() => null);
+ if (!stat?.isFile()) return `No file at "${filePath}".`;
+ return null;
+}
+
+export function setProjectInfo(document: AxcutDocument, rawArgs: unknown): AgentToolExecution {
+ const parsed = setProjectInfoArgs.safeParse(rawArgs);
+ if (!parsed.success) return failure(parsed.error.message);
+ // Trimmed and never empty, as the editor's name field commits it.
+ const title = parsed.data.title.trim();
+ if (!title) return failure("title must not be empty.");
+ return {
+ ok: true,
+ resultJson: JSON.stringify({ ok: true, title }),
+ document: { ...document, project: { ...document.project, title } },
+ };
+}
+
+export function removeMedia(document: AxcutDocument, rawArgs: unknown): AgentToolExecution {
+ const parsed = removeMediaArgs.safeParse(rawArgs);
+ if (!parsed.success) return failure(parsed.error.message);
+ const { assetId } = parsed.data;
+ if (!document.assets.some((asset) => asset.id === assetId)) {
+ return failure(`Unknown asset: ${assetId}. getCurrentDocument lists the project's assets.`);
+ }
+ const clips = document.timeline.clips.filter((clip) => clip.assetId === assetId);
+ const tracks = document.audioTracks.filter((track) => track.assetId === assetId);
+ if (clips.length > 0 || tracks.length > 0) {
+ const uses = [
+ ...(clips.length ? [`clips ${clips.map((clip) => clip.id).join(", ")} (removeClip)`] : []),
+ ...(tracks.length
+ ? [`audio tracks ${tracks.map((track) => track.id).join(", ")} (removeModifier)`]
+ : []),
+ ];
+ return failure(
+ `Asset ${assetId} is still on the timeline, so nothing was removed. Remove its ${uses.join(" and its ")} first.`,
+ );
+ }
+ const next = documentWithoutAsset(document, assetId);
+ return {
+ ok: true,
+ resultJson: JSON.stringify({
+ ok: true,
+ removed: assetId,
+ primaryAssetId: next.project.primaryAssetId ?? null,
+ }),
+ document: next,
+ };
+}
+
+export interface ImportMediaDeps {
+ /** The duration probe the editor's footage goes through (hidden-video.ts). */
+ openVideo?(filePath: string): Promise<OpenedVideo>;
+ /** Lets the editor read the file, as picking it in the editor would. */
+ approveMedia?(filePath: string): void;
+}
+
+/**
+ * Checks and measures the file, then returns the edit that files it: computed before the
+ * runner reads the project, so the slow probe never holds a snapshot open. A string is
+ * why it cannot be imported.
+ */
+export async function prepareImportMedia(
+ rawArgs: unknown,
+ deps: ImportMediaDeps,
+): Promise<((document: AxcutDocument) => Promise<AgentToolExecution>) | string> {
+ const parsed = importMediaArgs.safeParse(rawArgs);
+ if (!parsed.success) return JSON.stringify({ error: parsed.error.message });
+ const { path: filePath, label } = parsed.data;
+ const kind = parsed.data.kind ?? assetKindForPath(filePath);
+ const problem = await mediaPathProblem(filePath, kind);
+ if (problem) return JSON.stringify({ error: problem });
+
+ const sizeBytes = (await fs.stat(filePath)).size;
+ // Not fatal, as in the editor: a file Chromium cannot decode may still be one the
+ // compositor plays, and the editor probes it again once it opens the project.
+ const probed = await deps.openVideo?.(filePath).then(
+ (video) => {
+ video.close();
+ return video;
+ },
+ () => null,
+ );
+ const durationSec =
+ probed && Number.isFinite(probed.durationSec) && probed.durationSec > 0
+ ? probed.durationSec
+ : undefined;
+ deps.approveMedia?.(filePath);
+
+ return async (document) => {
+ const asset: AxcutAsset = {
+ id: createId("asset"),
+ kind,
+ label: label?.trim() || path.basename(filePath),
+ originalPath: filePath,
+ sizeBytes,
+ cameraTrack: null,
+ ...(durationSec ? { durationSec } : {}),
+ ...(kind === "video" && probed && probed.width > 0
+ ? { video: { codec: "unknown", width: probed.width, height: probed.height, fps: 0 } }
+ : {}),
+ };
+ const next = documentWithAsset(document, asset);
+ return {
+ ok: true,
+ resultJson: JSON.stringify({
+ ok: true,
+ assetId: asset.id,
+ kind,
+ label: asset.label,
+ durationSec: durationSec ?? null,
+ primaryAssetId: next.project.primaryAssetId ?? null,
+ }),
+ document: next,
+ };
+ };
+}