Skip to content

Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.

Refuse the shell and a workspace write while a person holds the wheel (#247)

beardthelion0403be14817f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 350328b9..9acfaa3f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -36,6 +36,20 @@ uses. It still cannot address another pod, a node, or a cloud metadata endpoint.
destination and so permitted everything. That rule now covers the API server alone, and
`networkPolicy.kubernetesApiCidr` narrows it to your cluster's service range; left empty it stays as
it was, because a chart cannot know that range.
+### Taking the wheel stops the Bot's shell, not just its clicks
+
+While a person held the wheel the Bot was refused on the page, and not in the shell. `/exec` and a
+workspace write went through, so a Bot could keep running commands and rewriting its `/workspace`
+underneath somebody who had taken the browser at a login wall. The guard existed and covered
+navigation and the four page actions; the shell arrived later and was never wired to it.
+
+Every acting path now asks the same question in one place, so the property the documentation states
+is the property the computer has. Reading is deliberately not acting: `/files/read` and
+`/files/list` still answer while a person drives, because a Bot that has just been stopped still has
+to be able to say what it was doing.
+
+Nothing to configure. A Bot that acts during a takeover gets the refusal it already got for a click,
+and the trail records the attempt and the failure the same way.
### A finished turn shows the page it opened, not the one open now
diff --git a/agent-computer/src/authorisation.ts b/agent-computer/src/authorisation.ts
index 667f2d44..cbeabb10 100644
--- a/agent-computer/src/authorisation.ts
+++ b/agent-computer/src/authorisation.ts
@@ -47,3 +47,30 @@ export function offeredToken(headers: Headers, url: URL): string {
export function isOpenPath(pathname: string): boolean {
return pathname === "/health";
}
+
+/**
+ * Which paths act on the computer, and so are refused while a person holds the wheel.
+ *
+ * One list, asked once per request, rather than a check inside each handler. The shell is the reason:
+ * `/exec` arrived after the wheel existed and was never given the guard the page paths had, so a Bot
+ * could keep running commands and writing files underneath somebody who had taken the browser at a
+ * login wall. A per-handler check is exactly the thing the next endpoint forgets, which is how that
+ * happened; a list the dispatcher consults is one an endpoint has to be added to.
+ *
+ * Reading is not acting. `/files/read` and `/files/list` stay open so a Bot that has been stopped can
+ * still read its own notes and explain what it was doing, which is the answer the person handing the
+ * wheel back usually wants.
+ */
+const ACTING_PATHS = new Set([
+ "/navigate",
+ "/click",
+ "/type",
+ "/key",
+ "/scroll",
+ "/exec",
+ "/files/write",
+]);
+
+export function actsOnTheComputer(pathname: string): boolean {
+ return ACTING_PATHS.has(pathname);
+}
diff --git a/agent-computer/src/index.ts b/agent-computer/src/index.ts
index b4762635..5d7177a6 100644
--- a/agent-computer/src/index.ts
+++ b/agent-computer/src/index.ts
@@ -1,7 +1,12 @@
import { serve } from "bun";
import type { Page } from "playwright";
import { parseAriaSnapshot, type SnapshotElement } from "./aria-snapshot";
-import { isOpenPath, matchesToken, offeredToken } from "./authorisation";
+import {
+ actsOnTheComputer,
+ isOpenPath,
+ matchesToken,
+ offeredToken,
+} from "./authorisation";
import { isPlainBotId } from "./bot-id";
import {
type Control,
@@ -485,6 +490,26 @@ serve<StreamData>({
}
const session = sessionFor(botId);
+ /*
+ * The wheel, asked once for everything that acts.
+ *
+ * Refused here rather than inside each handler because the handler that forgets is the whole
+ * defect: the shell shipped without this check and ran commands underneath a person who had taken
+ * the browser at a login wall. `actsOnTheComputer` is the list, and a new acting endpoint is
+ * refused by being added to it rather than by remembering to repeat this.
+ */
+ if (actsOnTheComputer(url.pathname)) {
+ try {
+ session.control.assertBotMayAct();
+ } catch (error) {
+ // A person holding the wheel is not a failure of the action; the Bot should wait and say so.
+ if (error instanceof ControlError) {
+ return json({ error: error.message, humanHasControl: true }, 409);
+ }
+ throw error;
+ }
+ }
+
if (url.pathname === "/stream") {
/*
* The socket carries the Bot in the query because it cannot do it in a header. Every other call here names
@@ -696,7 +721,6 @@ serve<StreamData>({
const startedAt = Date.now();
try {
- session.control.assertBotMayAct();
const target = await currentPage(botId);
await target.goto(body.url, {
waitUntil: "domcontentloaded",
@@ -715,10 +739,6 @@ serve<StreamData>({
elapsedMs: Date.now() - startedAt,
});
} catch (error) {
- // A person holding the wheel is not a failed navigation; the Bot should wait.
- if (error instanceof ControlError) {
- return json({ error: error.message, humanHasControl: true }, 409);
- }
// The page is the Bot's working surface, so a failed navigation is reported rather than
// thrown: the transcript needs to say what happened, and the browser stays usable.
return json(
@@ -893,7 +913,6 @@ serve<StreamData>({
const startedAt = Date.now();
try {
- session.control.assertBotMayAct();
const target = await currentPage(botId);
const detail = await performAction(
session,
@@ -932,10 +951,6 @@ serve<StreamData>({
if (error instanceof StaleSnapshotError) {
return json({ error: error.message, stale: true }, 409);
}
- // 409 as well, and for the same reason: nothing is broken, the caller simply has to wait.
- if (error instanceof ControlError) {
- return json({ error: error.message, humanHasControl: true }, 409);
- }
return json({ error: describe(error, "The action failed.") }, 502);
}
}
diff --git a/agent-computer/tests/authorisation.test.ts b/agent-computer/tests/authorisation.test.ts
index 9ec6521b..34fcfdc4 100644
--- a/agent-computer/tests/authorisation.test.ts
+++ b/agent-computer/tests/authorisation.test.ts
@@ -1,5 +1,10 @@
import { describe, expect, test } from "bun:test";
-import { isOpenPath, matchesToken, offeredToken } from "../src/authorisation";
+import {
+ actsOnTheComputer,
+ isOpenPath,
+ matchesToken,
+ offeredToken,
+} from "../src/authorisation";
/**
* The check that stands in front of a Bot's browser.
@@ -88,3 +93,50 @@ describe("what an unauthenticated caller may reach", () => {
}
});
});
+
+/**
+ * Which paths the wheel stops.
+ *
+ * A person takes the wheel at a login wall precisely because they no longer want the Bot acting, and
+ * `control.ts` states the property outright: "While a person holds control every acting call from the
+ * Bot is refused". That was true of the page from the start and untrue of the shell, which arrived
+ * later (#62) and was never wired to the wheel, so a Bot could run a command and rewrite the
+ * workspace underneath somebody mid-sign-in.
+ *
+ * The list lives here, beside the other path decision, rather than in `index.ts`, for the reason the
+ * header of this file gives: a decision next to `chromium.launch()` cannot be tested without Chrome.
+ *
+ * Reading is not acting. `/files/read` and `/files/list` stay open so a Bot waiting to be handed the
+ * wheel back can still say what it was doing.
+ */
+describe("what the wheel stops while a person is driving", () => {
+ test("every path that acts on the computer, the shell and a workspace write included", () => {
+ for (const path of [
+ "/navigate",
+ "/click",
+ "/type",
+ "/key",
+ "/scroll",
+ "/exec",
+ "/files/write",
+ ]) {
+ expect(actsOnTheComputer(path)).toBeTrue();
+ }
+ });
+
+ test("reading, looking and the handover itself are not acting", () => {
+ for (const path of [
+ "/files/read",
+ "/files/list",
+ "/snapshot",
+ "/screenshot",
+ "/health",
+ "/control",
+ "/control/take",
+ "/control/release",
+ "/stream",
+ ]) {
+ expect(actsOnTheComputer(path)).toBeFalse();
+ }
+ });
+});