Skip to content

Free open-source image editor for the browser

Let browser agents open an image from a URL through WebMCP (#58)

Rodrigo Belfioreba8b80f9ae07
diff --git a/API.md b/API.md
index 1f0bcee..280d649 100644
--- a/API.md
+++ b/API.md
@@ -31,7 +31,7 @@ These methods accept browser `File` objects and return `Promise<void>`. Await th
| `loadScene(file)` | Opens a scene file as a new document, replacing the current one and its history. |
| `recoverDraft()` | Opens the stored [draft](#drafts) as a new document, like `loadScene`. Rejects when no draft is stored or storage fails. |
| `discardDraft()` | Deletes the stored draft and its source files. |
-| `loadUrl(url)` | Fetches an image from a same-origin URL and loads it. |
+| `loadUrl(url)` | Fetches an image and loads it. A URL on another origin must allow CORS; the response's content type identifies an image whose URL has no extension. |
| `importXmp(file)` | Applies supported Camera Raw adjustments as one undoable edit. |
Loading calls are queued. XMP import is skipped if no document is ready; an invalid XMP import can reject without blocking later loads. Image and scene failures do not reject the loading promise. Without a document open, a failure leaves the workspace in its error state; with one, that document stays open and keeps its history. Either way `getState().failure` names the file and the error, and settings loaded in the same batch are skipped. Check `getState().documentId` or `failure` to confirm success. Loading completion does not guarantee the preview has rendered.
@@ -179,6 +179,7 @@ In browsers with [WebMCP](https://webmachinelearning.github.io/webmcp/), OpenLig
| Tool | Behavior |
| --- | --- |
| `get_state` | Returns the file, source size, frame, layers, comparison, and history; brush strokes and healing patches are counted rather than listed. |
+| `open_image` | Opens the image at an http(s) `url` like `loadUrl`, in place of the open photo, and returns the new state like `get_state`, or why the image could not open. |
| `set_adjustments`, `set_tone_curve` | Call `setAdjustments` and `setToneCurve`; `layerId` addresses a mask instead of the image. |
| `set_white_balance`, `set_color_mixer`, `set_details`, `set_vignette` | Call the matching command on the image or the first root effect. |
| `add_mask` | Adds a mask layer with the given mask on top of the stack as one edit and returns its `layerId`. |
@@ -187,4 +188,4 @@ In browsers with [WebMCP](https://webmachinelearning.github.io/webmcp/), OpenLig
| `undo`, `redo` | Call the matching command and return the history. |
| `set_preview` | Sets the preview `comparison`. |
-WebMCP passes only JSON, so the tools neither open nor export files: the agent edits the photo the user opened.
+WebMCP passes only JSON, so no tool takes or returns a file. An agent opens a local file, such as an image attached to its chat, by serving it over HTTP with CORS and passing its URL to `open_image`. From a public origin such as openlight.app, Chrome's [Local Network Access](https://developer.chrome.com/blog/local-network-access) asks the user before the page reaches a local address; an automated browser can grant the `local-network-access` permission instead.
diff --git a/src/app/loaders/image.ts b/src/app/loaders/image.ts
index 800cfdf..b579b69 100644
--- a/src/app/loaders/image.ts
+++ b/src/app/loaders/image.ts
@@ -29,14 +29,18 @@ export function createImageLoader(gpu: Gpu, workspace: Workspace) {
return workspace.open(file.name, () => loadDocument(gpu, file));
},
/** Enters the loading state before the fetch starts, so the empty state never shows. */
- loadUrl(url: string) {
- const name = url.slice(url.lastIndexOf("/") + 1);
+ async loadUrl(url: string) {
+ const { pathname } = new URL(url, location.href);
+ const name = decodeURIComponent(
+ pathname.slice(pathname.lastIndexOf("/") + 1),
+ );
return workspace.open(name, async () => {
const response = await fetch(url);
if (!response.ok) {
throw new Error(`${response.status} ${response.statusText}`);
}
- return loadDocument(gpu, new File([await response.blob()], name));
+ const blob = await response.blob();
+ return loadDocument(gpu, new File([blob], name, { type: blob.type }));
});
},
};
diff --git a/src/app/webmcp.ts b/src/app/webmcp.ts
index 3b96db4..d99787c 100644
--- a/src/app/webmcp.ts
+++ b/src/app/webmcp.ts
@@ -53,7 +53,7 @@ function defineTool<S extends z.ZodMiniType>(
annotations,
async execute(value) {
try {
- return run(parse(input, value, "Invalid input")) ?? "Done.";
+ return (await run(parse(input, value, "Invalid input"))) ?? "Done.";
} catch (error) {
// Returned, since browsers pass a thrown error to the agent without its message.
return String(error);
@@ -105,11 +105,24 @@ export function registerTools(
const tools = [
defineTool(
"get_state",
- "Describes the open photo: its file, source size in pixels, crop frame, layers from bottom to top with their IDs and settings, and undo history. Color mixer arrays list red, orange, yellow, green, aqua, blue, purple, and magenta. Call it first; when no photo is open, ask the user to open one.",
+ "Describes the open photo: its file, source size in pixels, crop frame, layers from bottom to top with their IDs and settings, and undo history. Color mixer arrays list red, orange, yellow, green, aqua, blue, purple, and magenta. Call it first; when no photo is open, open one with open_image or ask the user to open one.",
none,
() => describe(workspace, controls),
{ readOnlyHint: true },
),
+ defineTool(
+ "open_image",
+ "Opens the image at url in place of the open photo, with a new undo history, and describes it like get_state. The page fetches url, so another origin must allow CORS. To open a local file, such as an image attached to the chat, serve it over HTTP with CORS and pass its URL; the browser may first ask the user to let the page reach local addresses.",
+ z.strictObject({ url: z.url({ protocol: /^https?$/ }) }),
+ async ({ url }) => {
+ if (await controls.loadUrl(url)) {
+ return describe(workspace, controls);
+ }
+ const reason =
+ controls.getState().failure?.error ?? "another file opened instead";
+ throw Error(`Couldn't open ${url}: ${reason}`);
+ },
+ ),
defineTool(
"set_adjustments",
"Sets basic adjustments, keeping the ones omitted. Exposure is in stops from -5 to 5; the others go from -100 to 100, where 0 is neutral. Without layerId they tone the whole photo; with a mask's layerId they apply inside that mask.",
diff --git a/tests/webmcp.e2e.ts b/tests/webmcp.e2e.ts
index f5a6738..6d1fd8b 100644
--- a/tests/webmcp.e2e.ts
+++ b/tests/webmcp.e2e.ts
@@ -17,26 +17,33 @@ function callTool(page: Page, name: string, input: object = {}) {
);
}
-test("a browser agent edits, masks, crops, and undoes through WebMCP tools", async ({
+test("a browser agent opens, edits, masks, crops, and undoes through WebMCP tools", async ({
page,
}) => {
- await page.goto("/");
- await page.waitForFunction(
- async () => (await document.modelContext?.getTools())?.length,
+ // Another origin, with CORS and no extension, as an agent serves a local file.
+ const url = "http://localhost:8123/gray%20card";
+ await page.route(url, (route) =>
+ route.fulfill({
+ contentType: "image/svg+xml",
+ headers: { "access-control-allow-origin": "*" },
+ body: '<svg xmlns="http://www.w3.org/2000/svg" width="128" height="64"><rect width="128" height="64" fill="#808080"/></svg>',
+ }),
);
+ await page.goto("/");
+ // Polled from here, since waitForFunction takes a pending promise as truthy.
+ await expect
+ .poll(() =>
+ page.evaluate(
+ async () => (await document.modelContext?.getTools())?.length,
+ ),
+ )
+ .toBeGreaterThan(0);
expect(await callTool(page, "set_adjustments", { exposure: 1 })).toBe(
"Error: Load an image before editing.",
);
- await page.locator('input[type="file"]').setInputFiles({
- name: "gray.svg",
- mimeType: "image/svg+xml",
- buffer: Buffer.from(
- '<svg xmlns="http://www.w3.org/2000/svg" width="128" height="64"><rect width="128" height="64" fill="#808080"/></svg>',
- ),
- });
- await expect
- .poll(async () => JSON.parse(await callTool(page, "get_state")).sourceSize)
- .toEqual([128, 64]);
+ expect(JSON.parse(await callTool(page, "open_image", { url }))).toMatchObject(
+ { file: "gray card", sourceSize: [128, 64] },
+ );
const gray = await readImage(page);
expect(await callTool(page, "set_adjustments", { exposure: 9 })).toMatch(