Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.
Make Web Bot Auth off unless WEB_BOT_AUTH=on (#775)
David McKayb4a3137d7b62
diff --git a/.env.example b/.env.exampleindex 733d0b49..063b112e 100644--- a/.env.example+++ b/.env.example@@ -339,11 +339,10 @@ COMPUTER_TOKEN=# Web Bot Auth: every request a Bot's browser makes is signed as coming from this deployment# (Ed25519, HTTP Message Signatures), and the public key is published at-# <OPENBOT_PUBLIC_URL>/.well-known/http-message-signatures-directory. Unset means on whenever-# OPENBOT_PUBLIC_URL (or BETTER_AUTH_URL) is https on a public address, because verifiers fetch the-# key from there; off otherwise, which includes every laptop. `on` also signs behind a private https-# address, and is refused for plain http or loopback. It only earns anything once the-# deployment is registered with Cloudflare, a manual step: see docs/configuration.md.+# <OPENBOT_PUBLIC_URL>/.well-known/http-message-signatures-directory. Off unless set to on, because+# it announces the deployment's URL to every site a Bot visits. `on` needs OPENBOT_PUBLIC_URL (or+# BETTER_AUTH_URL) to be https and not this machine, and is refused otherwise. It only earns anything+# once the deployment is registered with Cloudflare, a manual step: see docs/configuration.md.# WEB_BOT_AUTH=on# The signing key, an Ed25519 private key as a JSON Web Key. Unset derives one from# KEY_ENCRYPTION_KEY, the same on every replica, so nothing new is needed to start. Never logged.diff --git a/CHANGELOG.md b/CHANGELOG.mdindex cf3129c9..52663493 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,12 +8,10 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged.## Unreleased-**Before upgrading.** Four things change for an existing deployment:+**Before upgrading.** Three things change for an existing deployment:- A hosted computer's browser is now headed by default. It uses a little more memory than headlessdid; set `COMPUTER_BROWSER_MODE=headless` (Helm: `computers.browserMode: headless`) to keep the oldbehaviour.-- A deployment whose public URL is https on a public address starts signing its Bots' requests with- Web Bot Auth; `WEB_BOT_AUTH=off` (Helm: `config.webBotAuth: "off"`) prevents it.- Migration `0054_memory_on_intelligence` adds two columns to `personal_memories`. On a deploymententitled to Intelligence memory, the memory sweep then copies existing reviewed, switched-on memoriesinto Intelligence, a few people at a time each minute; it is idempotent and safe to interrupt.@@ -33,8 +31,9 @@ computer and leave the person a blank page to solve. Now:- A hosted computer runs headed Chromium on its own virtual display, behind the same live screen.Headless Chromium is what bot checks catch first. A computer still run headless no longer callsitself `HeadlessChrome`.-- With a public https URL, every request a Bot's browser sends is signed with Web Bot Auth and the- key is published at `/.well-known/http-message-signatures-directory`. Register the deployment once+- Web Bot Auth, off unless switched on: with `WEB_BOT_AUTH=on` (Helm: `config.webBotAuth: "on"`)+ and an https public URL, every request a Bot's browser sends is signed and the key is published at+ `/.well-known/http-message-signatures-directory`. Register the deployment oncewith Cloudflare (Bot Submission Form, Request Signature) for sites that allow signed agents to letBots through. See docs/configuration.md. Every site a Bot visits sees the deployment's URL.- A residential or ISP proxy is a first-class setting: `EGRESS_PROXY_DEFAULT`, or on Helmdiff --git a/charts/openbot/README.md b/charts/openbot/README.mdindex b69fbefb..0cb7f04b 100644--- a/charts/openbot/README.md+++ b/charts/openbot/README.md@@ -539,10 +539,11 @@ and hands back) is the designed fallback when they are not enough. The full explown Xvfb, in `shared` and `sandbox` mode alike, behind the same live screen. `headless` is stillthere and uses less memory; it is the browser bot checks are best at spotting.-**Web Bot Auth, on whenever `config.publicUrl` is https on a public address.** Every request a Bot's browser sends is-signed as coming from this deployment, and the key is published at-`<publicUrl>/.well-known/http-message-signatures-directory`. `config.webBotAuth: "on"` or `"off"`-decides it explicitly. The key is derived from `secrets.keyEncryptionKey`; `secrets.webBotAuthKey`+**Web Bot Auth, off unless `config.webBotAuth: "on"`.** Every request a Bot's browser sends is+then signed as coming from this deployment, and the key is published at+`<publicUrl>/.well-known/http-message-signatures-directory`. It announces the deployment's URL to+every site a Bot visits, and it needs an https `config.publicUrl` that Cloudflare can reach, so a+private deployment leaves it off. The key is derived from `secrets.keyEncryptionKey`; `secrets.webBotAuthKey`(an Ed25519 private JWK, key `web-bot-auth-key` in a Secret of your own) replaces it. It earnsnothing until the deployment is registered with Cloudflare, once, by hand:diff --git a/charts/openbot/values.yaml b/charts/openbot/values.yamlindex 8530147c..c803fb5a 100644--- a/charts/openbot/values.yaml+++ b/charts/openbot/values.yaml@@ -122,9 +122,9 @@ config:# Web Bot Auth: every request a Bot's browser makes is signed as coming from this deployment, and# the public key is published at `<publicUrl>/.well-known/http-message-signatures-directory`, so a# site behind Cloudflare that allows signed agents lets the Bot through instead of challenging it.- # Empty means on whenever `publicUrl` is https on a public address (verifiers must be able to fetch- # the key), off otherwise; `on` or `off` decides it, and `on` also signs behind a private address. Signing earns nothing until the deployment is registered- # with Cloudflare, a manual step described in the chart README. The key is derived from+ # Off unless "on": signing announces the deployment's URL to every site a Bot visits. "on" needs an+ # https `publicUrl` (verifiers fetch the key from it) and earns nothing until the deployment is+ # registered with Cloudflare, a manual step described in the chart README. The key is derived from# `secrets.keyEncryptionKey` unless `secrets.webBotAuthKey` gives one.webBotAuth: ""# Administrators by email address, comma-separated. Required unless singleUser is on.diff --git a/docs/architecture.md b/docs/architecture.mdindex 7b79231d..80c96f70 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -136,7 +136,7 @@ Handovers are audited as control events:While a person controls the browser, Bot actions are refused rather than queued. A run with no browser attached pauses instead, waiting on the handover.-The handover is also the designed fallback for a bot check a Bot cannot pass (a "Verify you are human" page, a CAPTCHA). OpenBot reduces how often that happens (headed Chromium, Web Bot Auth signatures, an optional residential proxy; see [Getting past bot checks](configuration.md#getting-past-bot-checks)) and never solves one on anyone's behalf: the Bot asks, the person solves it in the live screen, and hands back. A help request records the page it was made on, so a computer that restarted in between reopens that page rather than handing over a blank browser.+The handover is also the designed fallback for a bot check a Bot cannot pass (a "Verify you are human" page, a CAPTCHA). OpenBot reduces how often that happens (headed Chromium, optional Web Bot Auth signatures, an optional residential proxy; see [Getting past bot checks](configuration.md#getting-past-bot-checks)) and never solves one on anyone's behalf: the Bot asks, the person solves it in the live screen, and hands back. A help request records the page it was made on, so a computer that restarted in between reopens that page rather than handing over a blank browser.Secret entry is separate from chat content. The audit trail records that a secret was requested or supplied and the character count, not the secret value.diff --git a/docs/configuration.md b/docs/configuration.mdindex 7361d5d7..46b45eb5 100644--- a/docs/configuration.md+++ b/docs/configuration.md@@ -622,7 +622,7 @@ is still the record.| `COMPUTER_BROWSER_IDLE_MS` | How long an untouched browser is kept. 30 minutes by default; `0` keeps them resident. || `COMPUTER_BROWSER_BACKEND` | `managed` by default (full bundled Chromium); `local-chrome` opts into installed Chrome with dedicated profiles and a loopback API. || `COMPUTER_BROWSER_MODE` | Managed defaults to `headed` on Linux, which is every hosted computer (a container, a pod, the desktop app's engine): full Chromium on the computer's own Xvfb display. A computer run directly on macOS or Windows defaults to `headless`, where headed would open windows on the desktop. `headless` is full Chromium's new headless mode. Local Chrome requires `headed`. See [Getting past bot checks](#getting-past-bot-checks). |-| `WEB_BOT_AUTH` | `on` or `off`. Unset means on when the public URL is https on a public address, off otherwise. See [Web Bot Auth](#web-bot-auth). |+| `WEB_BOT_AUTH` | `on` or `off`. Unset means off. `on` needs an https public URL. See [Web Bot Auth](#web-bot-auth). || `WEB_BOT_AUTH_KEY` | The Web Bot Auth signing key, an Ed25519 private JWK. Unset derives one from `KEY_ENCRYPTION_KEY`. Never logged. || `OPENBOT_LOCAL_COMPUTER_DIR` | Local startup helper's absolute data root. Defaults to the platform's OpenBot user-data directory; contains `profiles/` and `workspace/`. || `COMPUTER_SUPERVISOR_URL` | Supervisor URL for per-Bot computers. If absent, Bots share `AGENT_COMPUTER_URL`. |@@ -829,8 +829,9 @@ for when none of them is enough.1. **A real browser.** Headed Chromium, the default for hosted computers (see above). HeadlessChromium is what bot checks are best at spotting, and is challenged even from a residentialaddress.-2. **Web Bot Auth.** The browser signs every request as coming from this deployment, the way- OpenAI's agent does. A site that allows signed agents lets it through. See below.+2. **Web Bot Auth, off unless you switch it on.** The browser signs every request as coming from+ this deployment, the way OpenAI's agent does. A site that allows signed agents lets it through.+ See below.3. **Where the traffic leaves from.** A datacenter address is the other thing bot checks key on. Aresidential or ISP proxy changes it. See [Residential or ISP proxy](#residential-or-isp-proxy).4. **The handoff, which is the designed fallback.** When a site still challenges (a "Verify you are@@ -852,13 +853,14 @@ Cloudflare's signed-agent scheme, built on HTTP Message Signatures (RFC 9421). Wthe browser as each request leaves it, so the egress filter still judges every connection, TLS isstill Chromium's own, cookies are unchanged, and no CORS preflight is added.-**When it is on.** `WEB_BOT_AUTH` unset means on when `OPENBOT_PUBLIC_URL` (or `BETTER_AUTH_URL`)-is https on a public address, and off otherwise. A verifier has to fetch the key from the internet-over HTTPS, so a laptop on `http://localhost:3001` would only be signing with a directory nobody can-read. A private address (`10.x`, `192.168.x`, `*.internal`, a single-label name) stays off unless-`WEB_BOT_AUTH=on`, because only the operator knows whether their verifier reaches it. `on` with plain-http or a loopback address is refused at start-up; `off` turns it off anywhere. On Helm,-`config.webBotAuth`.+**When it is on.** Only when `WEB_BOT_AUTH=on` (on Helm, `config.webBotAuth: "on"`). Unset or `off`+means off, whatever the address. An OpenBot deployment is private by design, and signing announces+it: every site a Bot visits sees the deployment's URL, and Cloudflare will only register a key+directory it can fetch from the internet. So it is a deliberate choice, made by an operator whose+public URL is reachable and who wants sites to know their Bots. `on` needs `OPENBOT_PUBLIC_URL` (or+`BETTER_AUTH_URL`) to be https and not this machine, and is refused at start-up otherwise. A private+https address (`10.x`, `192.168.x`, `*.internal`, a single-label name) is accepted, because only the+operator knows whether their verifier reaches it.**The key.** Derived from `KEY_ENCRYPTION_KEY` (one way, with HKDF), so every replica and restartsigns with the same key and nothing new is needed to start. `WEB_BOT_AUTH_KEY` names one instead, an@@ -883,7 +885,7 @@ Bot Auth read the same directory.**What it tells sites.** Every site a Bot visits, Cloudflare-fronted or not, sees this deployment'spublic URL in `Signature-Agent`. That is the point (an honest agent says who it is), and it is why-the switch exists.+it is off unless switched on.**What is signed.** Every request the browser sends, whatever sent it: pages, redirects,subresources, cross-site frames running in processes of their own, dedicated workers, and servicediff --git a/server/src/web-bot-auth.ts b/server/src/web-bot-auth.tsindex b72b9398..c2d48cf0 100644--- a/server/src/web-bot-auth.ts+++ b/server/src/web-bot-auth.ts@@ -99,13 +99,10 @@ function thumbprint(x: string): string {}/**- * Where a verifier could fetch the directory from: `public` for an https address on the internet,- * `private` for https on a private or single-label name (an intranet verifier, perhaps), and- * undefined for anything that is not https or is this machine.+ * The origin a verifier fetches the directory from, or undefined for anything that is not https or+ * is this machine, where no verifier could ever read it.*/-function httpsReach(- url: string | undefined,-): { origin: string; reach: "public" | "private" } | undefined {+function httpsOrigin(url: string | undefined): string | undefined {if (!url) return undefined;let parsed: URL;try {@@ -123,33 +120,17 @@ function httpsReach() {return undefined;}- const octets = host.split(".").map(Number);- const [a = -1, b = -1] = octets;- const privateAddress =- (octets.length === 4 &&- octets.every((octet) => Number.isInteger(octet)) &&- (a === 10 ||- (a === 172 && b >= 16 && b <= 31) ||- (a === 192 && b === 168) ||- (a === 169 && b === 254) ||- (a === 100 && b >= 64 && b <= 127))) ||- /^f[cd]|^fe[89ab]/.test(host) ||- !host.includes(".") ||- /\.(local|internal|lan|home\.arpa|test)$/.test(host);- return {- origin: parsed.origin,- reach: privateAddress ? "private" : "public",- };+ return parsed.origin;}/*** The deployment's Web Bot Auth identity, or undefined when it is off.*- * `WEB_BOT_AUTH=on|off`; unset means on whenever the deployment has a public HTTPS address- * (`OPENBOT_PUBLIC_URL`, else `BETTER_AUTH_URL`), because the directory has to be fetchable over- * HTTPS from the internet for the signatures to mean anything, and off otherwise: plain http, this- * machine, or a private address. `on` also signs with a private HTTPS address, and is refused for- * plain http or this machine rather than signing with a directory nobody can read.+ * Off unless `WEB_BOT_AUTH=on`. A signed request names the deployment's URL to every site a Bot+ * visits, and an OpenBot deployment is private by design, so signing is the operator's deliberate+ * choice, never inferred from the address. `on` needs an https `OPENBOT_PUBLIC_URL` (else+ * `BETTER_AUTH_URL`) that is not this machine, because that is where verifiers fetch the key; it is+ * refused for plain http or loopback rather than signing with a directory nobody can read.** The key is `WEB_BOT_AUTH_KEY` (an Ed25519 private JWK) when given, otherwise derived from* `KEY_ENCRYPTION_KEY` with HKDF, so every replica and every restart signs with the same key and a@@ -165,20 +146,13 @@ export function webBotAuthFromEnv(`WEB_BOT_AUTH must be on or off, not ${JSON.stringify(environment.WEB_BOT_AUTH)}.`,);}- if (switchValue === "off") return undefined;- const reach = httpsReach(options.publicUrl);- if (!reach) {- if (switchValue === "on") {- throw new Error(- "WEB_BOT_AUTH=on needs an HTTPS address in OPENBOT_PUBLIC_URL (or BETTER_AUTH_URL) other than this machine, where verifiers fetch the signing key.",- );- }- return undefined;+ if (switchValue !== "on") return undefined;+ const agent = httpsOrigin(options.publicUrl);+ if (!agent) {+ throw new Error(+ "WEB_BOT_AUTH=on needs an HTTPS address in OPENBOT_PUBLIC_URL (or BETTER_AUTH_URL) other than this machine, where verifiers fetch the signing key.",+ );}- // On by itself only where the internet can read the key. A private address is the operator's- // call, since only they know whether their verifier can reach it.- if (reach.reach === "private" && switchValue !== "on") return undefined;- const agent = reach.origin;const explicit = environment.WEB_BOT_AUTH_KEY?.trim();const privateJwk = explicit? parseJwk(explicit)diff --git a/server/tests/web-bot-auth.test.ts b/server/tests/web-bot-auth.test.tsindex d614e3d7..a1d964cc 100644--- a/server/tests/web-bot-auth.test.ts+++ b/server/tests/web-bot-auth.test.ts@@ -21,34 +21,25 @@ function generatedJwk(): JsonWebKey {return generateKeyPairSync("ed25519").privateKey.export({ format: "jwk" });}-describe("whether Web Bot Auth is on", () => {- test("on by default when the deployment has a public HTTPS address", () => {- const identity = webBotAuthFromEnv(- {},- { publicUrl: PUBLIC, keyEncryptionKey: KEY },- );- expect(identity?.agent).toBe(PUBLIC);- });+const ON = { WEB_BOT_AUTH: "on" };- // A verifier on the internet has to fetch the key over HTTPS, so anything else would sign with a- // directory nobody can read.+describe("whether Web Bot Auth is on", () => {+ // A signed request names the deployment to every site a Bot visits, and OpenBot is private by+ // design, so no address, however public, switches it on by itself.test.each([+ PUBLIC,undefined,"http://openbot.example.com","https://localhost:3001",- "https://127.0.0.1",- "https://app.localhost","https://10.0.0.5",- "https://192.168.1.20:3001","https://openbot.internal",- "https://intranet",- ])("off by default for %s", (publicUrl) => {+ ])("off by default, even for %s", (publicUrl) => {expect(webBotAuthFromEnv({}, { publicUrl, keyEncryptionKey: KEY }),).toBeUndefined();});- test("off when switched off, whatever the address", () => {+ test("off when switched off", () => {expect(webBotAuthFromEnv({ WEB_BOT_AUTH: "off" },@@ -57,30 +48,40 @@ describe("whether Web Bot Auth is on", () => {).toBeUndefined();});+ test("on when switched on, signing as the public URL", () => {+ expect(+ webBotAuthFromEnv(ON, { publicUrl: PUBLIC, keyEncryptionKey: KEY })+ ?.agent,+ ).toBe(PUBLIC);+ });++ // A verifier has to fetch the key over HTTPS, so anything else would sign with a directory nobody+ // can read.test.each(["http://openbot.example.com","https://localhost:3001",+ "https://127.0.0.1",+ "https://app.localhost",+ "https://[::1]:3001",undefined,])("on with %s, which nobody else can fetch over HTTPS, is refused at start-up",(publicUrl) => {expect(() =>- webBotAuthFromEnv(- { WEB_BOT_AUTH: "on" },- { publicUrl, keyEncryptionKey: KEY },- ),+ webBotAuthFromEnv(ON, { publicUrl, keyEncryptionKey: KEY }),).toThrow("WEB_BOT_AUTH=on needs an HTTPS address");},);// Only the operator knows whether their verifier can reach a private address.- test("on signs with a private HTTPS address", () => {+ test.each([+ "https://10.0.0.5",+ "https://openbot.internal",+ "https://intranet",+ ])("on signs with the private HTTPS address %s", (publicUrl) => {expect(- webBotAuthFromEnv(- { WEB_BOT_AUTH: "on" },- { publicUrl: "https://10.0.0.5", keyEncryptionKey: KEY },- )?.agent,- ).toBe("https://10.0.0.5");+ webBotAuthFromEnv(ON, { publicUrl, keyEncryptionKey: KEY })?.agent,+ ).toBe(publicUrl);});test("an unknown switch value is refused", () => {@@ -92,29 +93,32 @@ describe("whether Web Bot Auth is on", () => {).toThrow("WEB_BOT_AUTH must be on or off");});- test("reaches the configuration, and stays off in an ordinary local one", () => {+ test("reaches the configuration, and stays off without the switch", () => {expect(loadConfig(testEnvironment()).webBotAuth).toBeUndefined();expect(- loadConfig(testEnvironment({ OPENBOT_PUBLIC_URL: PUBLIC })).webBotAuth- ?.agent,+ loadConfig(testEnvironment({ OPENBOT_PUBLIC_URL: PUBLIC })).webBotAuth,+ ).toBeUndefined();+ expect(+ loadConfig(testEnvironment({ OPENBOT_PUBLIC_URL: PUBLIC, ...ON }))+ .webBotAuth?.agent,).toBe(PUBLIC);});});describe("the signing key", () => {test("is derived from the deployment key, the same on every replica and restart", () => {- const first = webBotAuthFromEnv(- {},- { publicUrl: PUBLIC, keyEncryptionKey: KEY },- );- const second = webBotAuthFromEnv(- {},- { publicUrl: PUBLIC, keyEncryptionKey: KEY },- );- const other = webBotAuthFromEnv(- {},- { publicUrl: PUBLIC, keyEncryptionKey: OTHER_KEY },- );+ const first = webBotAuthFromEnv(ON, {+ publicUrl: PUBLIC,+ keyEncryptionKey: KEY,+ });+ const second = webBotAuthFromEnv(ON, {+ publicUrl: PUBLIC,+ keyEncryptionKey: KEY,+ });+ const other = webBotAuthFromEnv(ON, {+ publicUrl: PUBLIC,+ keyEncryptionKey: OTHER_KEY,+ });expect(first?.privateJwk).toEqual(second?.privateJwk as JsonWebKey);expect(first?.keyid).not.toBe(other?.keyid);// One way: the deployment key is not the signing key or anything in the published directory.@@ -124,7 +128,7 @@ describe("the signing key", () => {test("is WEB_BOT_AUTH_KEY when given", () => {const jwk = generatedJwk();const identity = webBotAuthFromEnv(- { WEB_BOT_AUTH_KEY: JSON.stringify(jwk) },+ { ...ON, WEB_BOT_AUTH_KEY: JSON.stringify(jwk) },{ publicUrl: PUBLIC, keyEncryptionKey: KEY },);expect(identity?.publicJwk).toEqual({@@ -138,7 +142,7 @@ describe("the signing key", () => {const jwk = { ...generatedJwk(), x: generatedJwk().x };expect(() =>webBotAuthFromEnv(- { WEB_BOT_AUTH_KEY: JSON.stringify(jwk) },+ { ...ON, WEB_BOT_AUTH_KEY: JSON.stringify(jwk) },{ publicUrl: PUBLIC, keyEncryptionKey: KEY },),).toThrow("does not match");@@ -152,7 +156,7 @@ describe("the signing key", () => {let message = "";try {webBotAuthFromEnv(- { WEB_BOT_AUTH_KEY: value },+ { ...ON, WEB_BOT_AUTH_KEY: value },{ publicUrl: PUBLIC, keyEncryptionKey: KEY },);} catch (error) {@@ -164,20 +168,20 @@ describe("the signing key", () => {});test("is named by its RFC 7638 thumbprint, as the reference signer names it", async () => {- const identity = webBotAuthFromEnv(- {},- { publicUrl: PUBLIC, keyEncryptionKey: KEY },- );+ const identity = webBotAuthFromEnv(ON, {+ publicUrl: PUBLIC,+ keyEncryptionKey: KEY,+ });const signer = await signerFromJWK(identity?.privateJwk as JsonWebKey);expect(identity?.keyid).toBe(signer.keyid);});});describe("the key directory", () => {- const identity = webBotAuthFromEnv(- {},- { publicUrl: PUBLIC, keyEncryptionKey: KEY },- )!;+ const identity = webBotAuthFromEnv(ON, {+ publicUrl: PUBLIC,+ keyEncryptionKey: KEY,+ })!;test("publishes the public key and nothing private", async () => {const response = await directoryResponse(@@ -271,7 +275,7 @@ describe("the key directory", () => {test("is served at the well-known path when on, and not served when off", async () => {const on = createApp(- loadConfig(testEnvironment({ OPENBOT_PUBLIC_URL: PUBLIC })),+ loadConfig(testEnvironment({ OPENBOT_PUBLIC_URL: PUBLIC, ...ON })),);const served = await on.request(`${PUBLIC}${DIRECTORY_PATH}`);expect(served.status).toBe(200);@@ -287,10 +291,10 @@ describe("the key directory", () => {});describe("handing the key to a computer", () => {- const identity = webBotAuthFromEnv(- {},- { publicUrl: PUBLIC, keyEncryptionKey: KEY },- )!;+ const identity = webBotAuthFromEnv(ON, {+ publicUrl: PUBLIC,+ keyEncryptionKey: KEY,+ })!;async function pushedBody(webBotAuth?: typeof identity) {let body: unknown;