Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.
Close a computer's browsers before its memory runs out, and a Bot's idle tabs (#879)
David McKayf4083dd2b1cd
diff --git a/.env.example b/.env.exampleindex 0ff9ee6d..92595b42 100644--- a/.env.example+++ b/.env.example@@ -359,6 +359,20 @@ COMPUTER_TOKEN=# COMPUTER_BROWSER_BACKEND=managed# COMPUTER_BROWSER_MODE=headed+# When a computer closes a browser besides the cap (COMPUTER_MAX_BROWSERS) and the idle timeout+# (COMPUTER_BROWSER_IDLE_MS). At or above this share of its memory limit, read as the kubelet reads it+# (the working set, so the page cache does not count), it closes browsers nobody has used for a+# minute, least recently used first, until it is ten points under. Memory is read before each launch+# and once a minute, so a page that grows between two checks can still run it out of memory. A+# container with no memory limit is measured against the machine's memory; a computer outside a+# container closes nothing for memory. In the all-in-one image the share counts the database and the+# API server too, on purpose: the out-of-memory kill counts them as well. 0 switches it off.+# COMPUTER_MEMORY_HIGH_PERCENT=85+# How long a Bot's background tab may sit untouched before it is closed, in ms: a tab the Bot left+# behind, such as the page a sign-in popup opened from. The page the Bot is on, a person's own+# browser, and any tab of a Bot whose wheel a person holds are never closed. 0 keeps them.+# COMPUTER_TAB_IDLE_MS=600000+# Web Bot Auth: every request a Bot's browser makes is signed as coming from this deployment# (Ed25519, HTTP Message Signatures), and the public key is published at# <OPENBOT_PUBLIC_URL>/.well-known/http-message-signatures-directory. Off unless set to on, becausediff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex fe0c0082..228422a2 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -141,7 +141,7 @@ jobs:bun test tests/sign-in.test.ts tests/session-cookies.test.ts tests/follows-popup.test.tstests/demonstration-live.test.ts tests/webrtc-proxy.test.ts tests/web-bot-auth-browser.test.tstests/handoff-page-restore.test.ts tests/enter-text.test.ts tests/person-browser.test.ts tests/person-browser-computer.test.ts- tests/screencast-keys-browser.test.ts+ tests/screencast-keys-browser.test.ts tests/browser-memory.test.tsworking-directory: agent-computerenv:OPENBOT_COMPUTER_BROWSER: '1'diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f36937ac..ccb39f75 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,7 +8,7 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged.## Unreleased-**Before upgrading.** Fourteen things change for an existing deployment:+**Before upgrading.** Fifteen things change for an existing deployment:- A hosted computer's browser is now headed by default. It uses a little more memory than headlessdid; set `COMPUTER_BROWSER_MODE=headless` (Helm: `computers.browserMode: headless`) to keep the oldbehaviour.@@ -61,6 +61,55 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged.to keep the old behaviour.- Interfaces a Bot drew before the upgrade (A2UI 0.9 surfaces) no longer draw when a conversation isreopened; the messages around them stay. New ones use A2UI 1.0 (below).+- A computer now closes browsers nobody is using, least recently used first, when its memory+ reaches 85% of its limit, and closes a Bot's background tabs after ten minutes untouched (below).+ In the all-in-one image that share counts the database and the API server too.+ `COMPUTER_MEMORY_HIGH_PERCENT=0` and `COMPUTER_TAB_IDLE_MS=0` (Helm: `computers.memoryHighPercent`+ and `computers.tabIdleMs`) keep the old behaviour.++### A computer closes browsers nobody is using when its memory runs high++A computer runs one browser for each Bot that is browsing, and the only limits were a count and half+an hour untouched. Neither looked at memory, so a few heavy pages could take a container past its+memory limit well under the count, and the kernel then killed processes without regard to which Bot+was busy. On Kubernetes it kills everything in the container at once, so every Bot's browser closed+together and nothing could reach the computer until it came back. A computer now reads its own+memory the way the kubelet does (the working set, which leaves out the file cache the kernel can+drop), before it starts a browser and once a minute. At or above 85% of its limit it closes browsers+one at a time, least recently used first, until it is under 75%, so the next launch does not cross+the line again straight away. Launches that arrive together share one pass rather than closing a+browser each. A Bot whose browser was closed starts again where it left off on its next step, still+signed in, because its profile is on disk.++Only a browser nobody is using is closed. Never closed for memory: the browser being started, the+most recently used one, any browser used in the last minute (a Bot between two steps asks for+nothing while its model works out the next), a person's own browser, cloud or native, whose Recorder+session would end, and the browser of a Bot whose wheel a person holds or has been asked to take.+When every browser is in use, none is closed.++This is a check, not a guarantee. Memory is read before each launch and once a minute, so a page+that grows between two checks can still run the container out of memory. A container with no memory+limit is measured against the machine's memory, and a computer outside a container, such as OpenBot+Desktop's, closes nothing for memory. In the all-in-one image the database and the API server run in+the same container as the browsers, and the share counts them too. That is deliberate: the kernel's+out-of-memory kill counts everything in the container and can take the database down, and closing a+browser nobody is using first is how the database is protected. Each close is logged as+`computer-browser-closed-for-memory` with the working set and the limit in bytes.+`COMPUTER_MEMORY_HIGH_PERCENT` (Helm: `computers.memoryHighPercent`) sets the share where closing+starts, and `0` switches it off.++The tabs a Bot leaves behind are closed too. A Bot always works in its newest page, so an older tab,+such as the page a sign-in popup was opened from or a link a site opened in a new tab, is one it+cannot go back to, and it held memory for as long as the browser stayed up. A background tab that+nothing has used, navigated or brought to the front for ten minutes is now closed. The page the Bot+is on is never closed, and neither is any tab of a person's own browser or of a Bot whose wheel a+person holds. `COMPUTER_TAB_IDLE_MS` (Helm: `computers.tabIdleMs`) sets the time, and `0` keeps them.++Through Helm, `computers.browserIdleMs` and `computers.maxBrowsers` now reach the computer as+written. An hour (`browserIdleMs: 3600000`) used to render as `3.6e+06`, which the computer ignored+for its default of thirty minutes, and `browserIdleMs: 0` (keep browsers open) was dropped. All four+of these settings now also reach the computers `computers.mode: sandbox` makes, which had none of+them.### Generated interfaces use A2UI 1.0diff --git a/agent-computer/src/browser-eviction.ts b/agent-computer/src/browser-eviction.tsindex be87a708..bdcbb301 100644--- a/agent-computer/src/browser-eviction.ts+++ b/agent-computer/src/browser-eviction.ts@@ -1,5 +1,5 @@/**- * Which Bots' browsers to close, and when.+ * Which Bots' browsers to close, and when, and which of a Bot's tabs.** Its own file, with no Playwright import, for the reason `authorisation.ts` and `bot-id.ts` are:* `profiles.ts` imports Playwright at module scope, so anything living there needs a browser merely@@ -62,21 +62,96 @@ export function chooseEvictions(const entries = [...running];if (entries.length <= max) return [];// A cap of zero keeps nothing, the newest included; that is the mistake reading the setting guards.- const newest =- max > 0- ? entries.reduce((latest, entry) =>- entry[1].usedAt >= latest[1].usedAt ? entry : latest,- )- : undefined;+ const newest = max > 0 ? newestOf(entries) : undefined;+ return inClosingOrder(entries, newest, goesLast)+ .slice(0, entries.length - max)+ .map(([botId]) => botId);+}++/** The most recently used, which is the one a launch has just added. Ties go to the later entry. */+function newestOf(+ entries: [string, Evictable][],+): [string, Evictable] | undefined {+ if (entries.length === 0) return undefined;+ return entries.reduce((latest, entry) =>+ entry[1].usedAt >= latest[1].usedAt ? entry : latest,+ );+}++/**+ * The order browsers are closed in when some have to go: every Bot's before a person's native window+ * (`keptOpen`), and least recently used first within each. `spare` is left out, which is the newest.+ */+function inClosingOrder(+ entries: [string, Evictable][],+ spare: [string, Evictable] | undefined,+ goesLast: (botId: string) => boolean,+): [string, Evictable][] {return entries- .filter((entry) => entry !== newest)+ .filter((entry) => entry !== spare).sort(([aId, a], [bId, b]) =>Number(goesLast(aId)) - Number(goesLast(bId)) || a.usedAt - b.usedAt,+ );+}++/**+ * How long after its last use a browser still counts as in use, and so is never closed for memory.+ *+ * A browser's use is recorded when something asks for its page: a Bot's step, a person's input, the+ * live screen. Between two steps a Bot asks for nothing while its model works out the next one, which+ * can take tens of seconds, and nothing here can tell that pause from a Bot that has finished.+ * Closing the browser then closes it in the middle of a run: the page the Bot was on is gone, and its+ * next step starts a cold browser, which is the very jump in memory the close was meant to head off.+ * A minute covers a model's turn between steps. It also covers a browser that has just launched, so+ * the next launch never closes the one before it, which is what would otherwise turn a full computer+ * into a loop of each Bot's launch closing the last Bot's browser.+ */+export const RECENTLY_USED_MS = 60_000;++/**+ * Which to close next because the computer's memory runs high (`container-memory.ts`).+ *+ * One at a time, because the caller reads memory again after each close and stops as soon as it is+ * enough: a count says in advance how many have to go, and bytes do not. Least recently used first,+ * the cap's order, so the two never disagree about who goes first.+ *+ * Never a browser in use, which here means four things:+ * - The one being launched. The caller runs this before a launch and passes only the browsers+ * already running, so the Bot that asked is not among them.+ * - The most recently used of those, however long ago that was. So at least one browser is always+ * left, because one browser that is on its own too big for the computer is the limit's problem,+ * and closing it would only start it again on its next step.+ * - Any used within `recentlyUsedMs` ({@link RECENTLY_USED_MS}), which is a Bot in the middle of a+ * run or a browser that has just started.+ * - Any `spared` names, which is a Bot whose wheel a person holds or has been asked to take: closing+ * it closes it under somebody's hands, or under a page the Bot is waiting for somebody to work.+ *+ * And never a person's own browser, cloud or native. Their Recorder runs in it, and closing it ends+ * what they were recording without a word to them. It still counts toward the cap, which closes it+ * last; memory pressure is not a reason to end somebody's session.+ */+export function chooseForMemory(+ running: Iterable<[string, Evictable]>,+ now: number,+ {+ recentlyUsedMs = RECENTLY_USED_MS,+ spared = () => false,+ }: {+ recentlyUsedMs?: number;+ spared?: (botId: string) => boolean;+ } = {},+): string | undefined {+ const entries = [...running];+ const cutoff = now - recentlyUsedMs;+ return inClosingOrder(entries, newestOf(entries), () => false)+ .filter(+ ([botId, entry]) =>+ entry.usedAt <= cutoff && !isPersonBrowserId(botId) && !spared(botId),)- .slice(0, entries.length - max)- .map(([botId]) => botId);+ .map(([botId]) => botId)+ .at(0);}/**@@ -104,3 +179,40 @@ export function chooseIdle(.filter(([botId, entry]) => entry.usedAt <= cutoff && !stays(botId)).map(([botId]) => botId);}++/** Enough of a tab for the choice. Keeps this file independent of what else is on a page. */+export type IdleTab = {+ /** When the Bot or a person last acted on it, it navigated, or it was the page in front. */+ usedAt: number;+ /** The page the Bot is on, which is the one the live screen shows and its next action lands in. */+ front: boolean;+ /** A page that is never the Bot's to lose: an extension's own (the Recorder's) or DevTools. */+ kept?: boolean;+};++/**+ * Which of a Bot's background tabs to close because nothing has touched them.+ *+ * A Bot is always on its newest page (`chooseLivePage`): a window a site opens is where its work+ * went, and it has no way back to an older tab except that newer ones close. So a tab left behind+ * (the page a sign-in popup was opened from, a link that opened in a new tab and was never closed)+ * holds a renderer and its memory for nothing, for as long as the browser stays up. Closed past a+ * timeout rather than at once, because an opener whose popup is still mid sign-in is waiting for+ * that popup to come back to it.+ *+ * Never the page in front, which is the one the Bot is on, so closing tabs never moves it: it stays+ * the newest open page, which is what the Bot follows. And never anything when which page is in+ * front is not exactly one, because then any tab could be the one the Bot is on. That also means the+ * last tab is never closed, since a single tab is the one in front. A timeout of zero or less+ * switches this off.+ */+export function chooseIdleTabs<T extends IdleTab>(+ tabs: readonly T[],+ idleTimeoutMs: number,+ now: number,+): T[] {+ if (idleTimeoutMs <= 0) return [];+ if (tabs.filter((tab) => tab.front).length !== 1) return [];+ const cutoff = now - idleTimeoutMs;+ return tabs.filter((tab) => !tab.front && !tab.kept && tab.usedAt <= cutoff);+}diff --git a/agent-computer/src/container-memory.ts b/agent-computer/src/container-memory.tsnew file mode 100644index 00000000..613f3d41--- /dev/null+++ b/agent-computer/src/container-memory.ts@@ -0,0 +1,208 @@+/**+ * How much of its memory this computer is using, read the way Kubernetes reads it.+ *+ * WHY THIS EXISTS. A computer runs one Chromium per Bot that is browsing, and the only limits on them+ * were a count and an idle timeout. Neither looks at memory, so a few heavy pages could push the+ * container past its limit, and the kernel's OOM killer does not choose by use. On Kubernetes with+ * cgroup v2 the kubelet sets `memory.oom.group` (since 1.28), so one OOM kills every process in the+ * container and every Bot's browser on it goes at once; elsewhere the kernel kills the largest+ * process, which is somebody's browser picked by size. Knowing how close the container is lets the+ * computer close the least recently used browser first, which costs that Bot a relaunch and nothing+ * else, because its profile is on disk.+ *+ * WORKING SET, NOT USAGE. The cgroup's usage counts the page cache, and most of a page cache is files+ * the kernel will drop the moment anything needs the room: a container that wrote a large download+ * reads as nearly full while it is nowhere near being killed. Kubernetes judges a container by its+ * working set, which is usage less the inactive file cache, and that is what this reads, so the+ * computer measures itself the way the kubelet measures it. cAdvisor computes it in+ * `lib/container/libcontainer/handler.go`: `memory.current` less `inactive_file` on cgroup v2, and+ * `memory.usage_in_bytes` less `total_inactive_file` on cgroup v1, clamped at zero. Microsoft's AKS+ * guide to high memory consumption and the widely cited "Memory_working_set vs Memory_rss in+ * Kubernetes" describe the same rule.+ *+ * THE LIMIT IS WHICHEVER RUNS OUT FIRST. A container started with no limit (`docker run` with no+ * `--memory`, Docker Desktop's default) reports `max` on cgroup v2 and a number near 2^63 on cgroup+ * v1. Inside a container, the machine's own memory is the VM's, and that is what actually runs out,+ * so the limit is the smaller of the cgroup's and the machine's.+ *+ * NO CGROUP, NO READING. A computer that is not in a container (the one OpenBot Desktop runs on a+ * person's Mac for their own browser, or one started by hand on a Linux desktop, whose view of+ * `/sys/fs/cgroup` is the root, where these files do not exist) has nothing here to read, and closes+ * nothing for memory. The count cap is still there.+ *+ * Its own file, with no Playwright import, for the reason `browser-eviction.ts` is: the parsing is+ * the part with a wrong answer available, and the tests feed it the files real containers wrote.+ */++import { readFile } from "node:fs/promises";+import { totalmem } from "node:os";+import { join } from "node:path";++/** How much memory the computer is using, and how much it may. Both in bytes. */+export type MemoryReading = { workingSet: number; limit: number };++/** Read one cgroup file, or nothing when it is not there. Injected so a test can point it elsewhere. */+export type ReadCgroupFile = (path: string) => Promise<string | undefined>;++/** Where a container sees its own cgroup, under cgroup v2 and the v1 memory controller. */+export const CGROUP_ROOT = "/sys/fs/cgroup";++const readCgroupFile: ReadCgroupFile = (path) =>+ readFile(path, "utf8").catch(() => undefined);++/** A whole number of bytes, as a cgroup file writes one. Anything else is not a reading. */+function bytes(text: string | undefined): number | undefined {+ const trimmed = text?.trim();+ if (!trimmed || !/^\d+$/.test(trimmed)) return undefined;+ return Number(trimmed);+}++/** One counter from a `memory.stat` file, whose lines are `<name> <value>`. */+export function statValue(stat: string, name: string): number | undefined {+ for (const line of stat.split("\n")) {+ const [key, value] = line.trim().split(/\s+/);+ if (key === name) return bytes(value);+ }+ return undefined;+}++/**+ * Usage less the inactive file cache, clamped at zero, as cAdvisor does it. A `memory.stat` without+ * the counter leaves usage as it is, also as cAdvisor does.+ */+function workingSetOf(+ usage: number,+ stat: string | undefined,+ inactiveFile: string,+): number {+ const inactive =+ stat === undefined ? undefined : statValue(stat, inactiveFile);+ if (inactive === undefined) return usage;+ return usage < inactive ? 0 : usage - inactive;+}++/**+ * The limit that runs out first: the cgroup's, or the machine's when the cgroup has none or claims+ * more than the machine has. Cgroup v2 writes `max` for none; v1 writes a number near 2^63, which+ * the machine is always smaller than.+ */+function limitOf(+ cgroupLimit: string | undefined,+ machineBytes: number,+): number | undefined {+ if (cgroupLimit?.trim() === "max") return machineBytes;+ const limit = bytes(cgroupLimit);+ if (limit === undefined || limit === 0) return undefined;+ return Math.min(limit, machineBytes);+}++/**+ * The computer's memory from cgroup v2's `memory.current`, `memory.max` and `memory.stat`.+ *+ * Undefined when the usage or the limit cannot be read, which is a computer with no cgroup of its+ * own and so nothing to act on.+ */+export function cgroupV2Memory(+ files: { current?: string; max?: string; stat?: string },+ machineBytes: number,+): MemoryReading | undefined {+ const usage = bytes(files.current);+ const limit = limitOf(files.max, machineBytes);+ if (usage === undefined || limit === undefined) return undefined;+ return {+ workingSet: workingSetOf(usage, files.stat, "inactive_file"),+ limit,+ };+}++/**+ * The computer's memory from cgroup v1's `memory.usage_in_bytes`, `memory.limit_in_bytes` and+ * `memory.stat`, for a host that still mounts the v1 memory controller.+ */+export function cgroupV1Memory(+ files: { usage?: string; limit?: string; stat?: string },+ machineBytes: number,+): MemoryReading | undefined {+ const usage = bytes(files.usage);+ const limit = limitOf(files.limit, machineBytes);+ if (usage === undefined || limit === undefined) return undefined;+ return {+ workingSet: workingSetOf(usage, files.stat, "total_inactive_file"),+ limit,+ };+}++/**+ * Read the computer's memory now: cgroup v2 first, then v1, then nothing.+ *+ * Read again every time it is asked rather than cached, because the caller asks precisely when it+ * has just closed a browser and wants to know whether that was enough.+ */+export async function readContainerMemory({+ root = CGROUP_ROOT,+ read = readCgroupFile,+ machineBytes = totalmem(),+}: {+ root?: string;+ read?: ReadCgroupFile;+ machineBytes?: number;+} = {}): Promise<MemoryReading | undefined> {+ const [current, max, stat] = await Promise.all([+ read(join(root, "memory.current")),+ read(join(root, "memory.max")),+ read(join(root, "memory.stat")),+ ]);+ if (current !== undefined)+ return cgroupV2Memory({ current, max, stat }, machineBytes);+ const v1 = join(root, "memory");+ const [usage, limit, v1Stat] = await Promise.all([+ read(join(v1, "memory.usage_in_bytes")),+ read(join(v1, "memory.limit_in_bytes")),+ read(join(v1, "memory.stat")),+ ]);+ if (usage === undefined) return undefined;+ return cgroupV1Memory({ usage, limit, stat: v1Stat }, machineBytes);+}++/** Whether the working set is at or above a share of the limit. */+export function memoryAtOrAbove(+ reading: MemoryReading,+ percent: number,+): boolean {+ return reading.workingSet * 100 >= reading.limit * percent;+}++/**+ * Whether the computer is close enough to its limit to start closing browsers.+ *+ * At or above the share, not only above it, so a deployment that set 85 has acted by the time it+ * reads 85. A share of zero or less switches it off, as a timeout of zero switches off the idle+ * sweep.+ */+export function memoryRunsHigh(+ reading: MemoryReading,+ highPercent: number,+): boolean {+ if (highPercent <= 0) return false;+ return memoryAtOrAbove(reading, highPercent);+}++/**+ * How far under the share that starts the closing memory has to fall before the closing stops.+ *+ * Stopping at the same line it started at leaves the computer a browser's launch from crossing it+ * again, so the next Bot to start closes another browser, whose Bot starts again and closes another:+ * each launch paying for the last. Closing on to ten points under gives the launches after it room.+ * Ten points, not a setting of its own: it is a gap between two numbers, and the one an operator+ * chooses is where the closing starts.+ */+export const MEMORY_TARGET_POINTS_UNDER = 10;++/**+ * The share closing stops under: {@link MEMORY_TARGET_POINTS_UNDER} below where it starts, so 75 at+ * the default 85. At or under zero, which only a starting share of ten or less gives, closing goes on+ * while anything is left that may be closed.+ */+export function memoryTargetPercent(highPercent: number): number {+ return highPercent - MEMORY_TARGET_POINTS_UNDER;+}diff --git a/agent-computer/src/index.ts b/agent-computer/src/index.tsindex 62bad1fb..d79d3b9b 100644--- a/agent-computer/src/index.ts+++ b/agent-computer/src/index.ts@@ -310,7 +310,14 @@ const profiles = createProfiles(if (sessions.get(botId)) sessions.renewRun(botId);await sessions.get(botId)?.viewer.releaseAll(COMPUTER_STOPPED);},- { personDisplay: PERSON_DISPLAY },+ {+ personDisplay: PERSON_DISPLAY,+ // A Bot's browser a person is working in, or has been asked to, is never closed for memory, and+ // none of its tabs are closed for being idle. A person's own browser has no Bot's wheel to hold:+ // it is never closed for memory whoever is at it (`chooseForMemory`), and its tabs are never swept.+ heldByPerson: (botId) =>+ !isPersonBrowserId(botId) && sessions.personHasTheWheel(botId),+ },);/**diff --git a/agent-computer/src/profiles.ts b/agent-computer/src/profiles.tsindex 2cdf948d..d2201439 100644--- a/agent-computer/src/profiles.ts+++ b/agent-computer/src/profiles.ts@@ -38,8 +38,21 @@ import { join } from "node:path";import { type BrowserContext, chromium, type Page } from "playwright";import { isPersonBrowserId } from "../../shared/openbot-browser";import { profileDirectoryFor } from "./bot-id";-import { chooseEvictions, chooseIdle, keptOpen } from "./browser-eviction";+import {+ chooseEvictions,+ chooseForMemory,+ chooseIdle,+ chooseIdleTabs,+ keptOpen,+} from "./browser-eviction";import { type BrowserRuntime, browserRuntimeFromEnv } from "./browser-runtime";+import {+ type MemoryReading,+ memoryAtOrAbove,+ memoryRunsHigh,+ memoryTargetPercent,+ readContainerMemory,+} from "./container-memory";import { egressFor, egressLabel } from "./egress";import { numberFromEnv, settleWithin } from "./env";import { chooseLivePage } from "./live-page";@@ -304,9 +317,58 @@ const IDLE_TIMEOUT_MS = numberFromEnv("COMPUTER_BROWSER_IDLE_MS", 30 * 60_000, {zeroSwitchesItOff: true,});-/** How often the idle sweep looks. Cheap: it walks a map of at most `MAX_LIVE_BROWSERS`. */+/**+ * How often the idle sweep looks. Cheap: it walks a map of at most `MAX_LIVE_BROWSERS`, their tabs,+ * and three small files under `/sys/fs/cgroup`.+ */const IDLE_SWEEP_MS = 60_000;+/**+ * At what share of the computer's memory it starts closing browsers, least recently used first.+ *+ * The cap counts browsers, and a browser is not a fixed size: a few heavy pages can take a container+ * past its memory limit well under the cap, and the OOM killer does not choose by use. On Kubernetes+ * with cgroup v2 it kills every process in the container together, every Bot's browser at once, and+ * nothing can reach the computer until it comes back. So memory is read the way Kubernetes reads it+ * (`container-memory.ts`) before every launch, which is when memory jumps, and on every tick of the+ * idle sweep. At or above this share of the limit, browsers are closed one at a time, least recently+ * used first, until memory is ten points under it (`memoryTargetPercent`). Closing one costs its Bot+ * a relaunch and nothing else, the same as the cap.+ *+ * A check, not a guarantee. Between two checks a page can grow on its own, and memory can run out+ * before the next launch or the next minute comes round. What this removes is the computer filling+ * itself up one launch at a time with browsers nobody is using.+ *+ * Eighty-five leaves room for the launch that asked, and for a page that grows after the check,+ * before the limit is reached. Zero switches it off, as it does for the idle sweep. A computer with no+ * cgroup of its own (OpenBot Desktop's, on a person's Mac) has no limit to read and closes nothing+ * for memory either way. In the all-in-one image the share counts the database and the API server as+ * well, deliberately: the OOM kill counts them too, and a browser closed first is what keeps them up.+ */+const MEMORY_HIGH_PERCENT = numberFromEnv("COMPUTER_MEMORY_HIGH_PERCENT", 85, {+ zeroSwitchesItOff: true,+ max: 100,+});++/**+ * How long a Bot's background tab may sit untouched before the idle sweep closes it.+ *+ * A Bot is always on its newest page, so a tab it left behind (the page a sign-in popup opened from,+ * a link a site opened in a new tab) is one it cannot get back to, holding a renderer and its memory+ * for as long as the browser is up. Ten minutes is long enough for a popup's sign-in to come back to+ * the page that opened it, and a third of the thirty after which a whole idle browser is closed, so a+ * tab left behind in a browser that is still in use does not wait for the browser to go. See+ * `chooseIdleTabs` for what is never closed. Zero switches it off.+ *+ * No count of tabs alongside it. A count is the wrong measure for the same reason the browser cap is+ * not enough on its own: one heavy tab outweighs twenty light ones, and memory is now watched+ * directly. A count would also close tabs a site opened a moment ago, in the middle of the flow they+ * were opened for, where a timeout only ever closes what has been left alone.+ */+const TAB_IDLE_MS = numberFromEnv("COMPUTER_TAB_IDLE_MS", 10 * 60_000, {+ zeroSwitchesItOff: true,+});+/*** Told whenever a Bot's browser is closed and forgotten, before the close is waited on.*@@ -337,6 +399,26 @@ export type ProfilesOptions = {* as it is now.*/personDisplay?: PersonDisplay;+ /**+ * The computer's memory now, or nothing when it has no limit to read. The container's own cgroup+ * unless a test hands in the reading it wants the computer to see.+ */+ memory?: () => Promise<MemoryReading | undefined>;+ /** {@link MEMORY_HIGH_PERCENT}, for a test. */+ memoryHighPercent?: number;+ /** {@link TAB_IDLE_MS}, for a test that cannot wait ten minutes. */+ tabIdleMs?: number;+ /**+ * The clock every use, idle timeout and memory decision here is measured by. `Date.now` unless a+ * test needs a browser to have gone quiet a minute ago without waiting a minute for it.+ */+ now?: () => number;+ /**+ * Whether a person holds this Bot's wheel, or has been asked to take it. Such a browser is never+ * closed for memory, and none of its tabs are closed for being idle: a person may be working across+ * them. The computer's sessions answer it (`sessions.personHasTheWheel`); without it, nobody does.+ */+ heldByPerson?: (botId: string) => boolean;};/** Opening a person's browser, which is the one way it starts. See `person-browser.ts`. */@@ -366,6 +448,23 @@ export function createProfiles(DISPLAY: process.env.DISPLAY,WAYLAND_DISPLAY: process.env.WAYLAND_DISPLAY,};+ const readMemory = options.memory ?? (() => readContainerMemory());+ const memoryHighPercent = options.memoryHighPercent ?? MEMORY_HIGH_PERCENT;+ const tabIdleMs = options.tabIdleMs ?? TAB_IDLE_MS;+ const now = options.now ?? Date.now;+ const heldByPerson = options.heldByPerson ?? (() => false);+ /*+ * A wheel that cannot be read counts as held. The control state throws when its store has failed,+ * and the answer decides whether a browser or a tab is closed: keeping one is the mistake that+ * costs nothing, and a throw here must never fail the launch that asked.+ */+ const spared = (botId: string): boolean => {+ try {+ return heldByPerson(botId);+ } catch {+ return true;+ }+ };type LiveBrowser = {context: BrowserContext;@@ -400,6 +499,26 @@ export function createProfiles(const live = new Map<string, LiveBrowser>();/** A person's native window: the idle sweep leaves it and the cap takes it last. See `keptOpen`. */const stays = keptOpen(personWindow(runtime));+ /**+ * When each of a Bot's tabs was last used: acted on, navigated, or in front. Decides which+ * background tabs the idle sweep closes. Weak, so a closed page takes its entry with it.+ */+ const tabUsedAt = new WeakMap<Page, number>();+ const touchTab = (page: Page): void => {+ tabUsedAt.set(page, now());+ };+ /** A tab never seen before counts as used now, so one this process missed is not closed at once. */+ const tabLastUsed = (page: Page): number => {+ const at = tabUsedAt.get(page) ?? now();+ tabUsedAt.set(page, at);+ return at;+ };+ /** A tab that navigates is in use, a background one included: a site moving it on, a sign-in finishing. */+ const touchOnNavigation = (page: Page): void => {+ page.on("framenavigated", (frame) => {+ if (frame === page.mainFrame()) touchTab(page);+ });+ };/** Launches in flight, so a cold computer is started once however many callers ask at once. */const starting = new Map<string, Promise<Page>>();/** Closes and resets in flight, so a Bot's browser is never reopened from a profile being closed or deleted. */@@ -471,20 +590,125 @@ export function createProfiles(};/**- * Close browsers nothing has touched for a while.+ * Close browsers, least recently used first, once the computer's memory runs high, until it is ten+ * points under the line.+ *+ * One at a time, reading memory again after each close, because what one browser was holding is+ * not known until it has gone. Starts at {@link MEMORY_HIGH_PERCENT} and goes on to+ * `memoryTargetPercent` under it, so the launch after it does not cross the line again at once.+ * Stops when memory is there, when nothing is left that may be closed (`chooseForMemory`: never+ * the browser launching, the most recently used, one in use within the last minute, one a person+ * holds the wheel of, or a person's own), or when the same browser comes up twice.+ */+ const relieveMemoryOnce = async (): Promise<void> => {+ if (memoryHighPercent <= 0) return;+ // A reading that fails is a computer with nothing to act on, the same as one with no cgroup.+ const read = () => readMemory().catch(() => undefined);+ let reading = await read();+ if (!reading || !memoryRunsHigh(reading, memoryHighPercent)) return;+ const target = memoryTargetPercent(memoryHighPercent);+ const tried = new Set<string>();+ while (reading && memoryAtOrAbove(reading, target)) {+ const next = chooseForMemory(live.entries(), now(), { spared });+ if (!next || tried.has(next)) return;+ tried.add(next);+ console.info(+ JSON.stringify({+ type: "computer-browser-closed-for-memory",+ botId: next,+ workingSetBytes: reading.workingSet,+ limitBytes: reading.limit,+ }),+ );+ await evict(next, "the computer's memory ran high");+ reading = await read();+ }+ };++ /**+ * One pass at a time, which every caller waits on.+ *+ * A burst of launches each ran a pass of its own, and each read memory before any close had given+ * any back: a browser's memory is released when its Chromium exits, seconds after it was taken out+ * of the map. So K launches at once read the same high number and closed K browsers where one was+ * enough. A launch that arrives while a pass is running waits for that pass and launches after it.+ */+ let relieving: Promise<void> | undefined;+ const relieveMemory = (): Promise<void> => {+ relieving ??= relieveMemoryOnce().finally(() => {+ relieving = undefined;+ });+ return relieving;+ };++ /**+ * Close the background tabs of every Bot's browser that nothing has touched for a while.+ *+ * Never a person's browser, whose tabs are theirs and which has no Bot in it, and never a Bot's+ * while a person holds its wheel, because they may be working across its tabs. Within a Bot's+ * browser, `chooseIdleTabs` decides, and the page the Bot is on is never one of them, so it is+ * still the newest open page afterwards and the record does not move. The address is not logged:+ * a page left behind can be a sign-in link with a one-time token in it.+ */+ const closeIdleTabs = async (): Promise<number> => {+ if (tabIdleMs <= 0) return 0;+ const sweptAt = now();+ let closed = 0;+ for (const [botId, running] of [...live.entries()]) {+ if (running.person || isPersonBrowserId(botId) || spared(botId)) continue;+ // Closed while an earlier Bot's tabs were being closed: nothing of it is left to sweep.+ if (live.get(botId) !== running) continue;+ running.retarget();+ const tabs = running.context+ .pages()+ .filter((page) => !page.isClosed())+ .map((page) => ({+ page,+ usedAt: tabLastUsed(page),+ front: page === running.page,+ kept:+ page.url().startsWith("chrome-extension://") ||+ page.url().startsWith("devtools://"),+ }));+ for (const { page } of chooseIdleTabs(tabs, tabIdleMs, sweptAt)) {+ console.info(+ JSON.stringify({+ type: "computer-tab-closed",+ botId,+ reason: "it had been idle",+ }),+ );+ await page.close().catch(() => undefined);+ closed += 1;+ }+ }+ return closed;+ };++ /**+ * Close browsers nothing has touched for a while, then the tabs Bots left behind, then, if memory+ * still runs high, browsers by use.** The cap answers concurrent breadth; this answers a Bot used once last Tuesday whose browser is- * still resident and doing nothing for anybody.+ * still resident and doing nothing for anybody. In that order because each step is gentler than the+ * next: a browser nobody is using goes before a tab of one somebody is, and a tab nobody can reach+ * goes before a browser that is still in use.*/const sweepIdle = async (): Promise<void> => {for (const botId of chooseIdle(live.entries(),IDLE_TIMEOUT_MS,- Date.now(),+ now(),stays,)) {await evict(botId, "it had been idle");}+ const tabsClosed = await closeIdleTabs().catch(() => 0);+ // A closed tab's renderer exits a moment after the tab does, so memory read at once would still+ // count it, and a browser could be closed for memory the tabs had already given back.+ if (tabsClosed > 0)+ await new Promise((resolve) => setTimeout(resolve, CLOSE_SETTLE_MS));+ await relieveMemory();};const idleSweep = setInterval(() => {@@ -628,7 +852,7 @@ export function createProfiles(context,page,startedAt: new Date().toISOString(),- usedAt: Date.now(),+ usedAt: now(),retarget: () => {},...(sessionCookies ? { sessionCookies } : {}),person,@@ -723,8 +947,10 @@ export function createProfiles(existing?.context.browser()?.isConnected() &&!existing.page.isClosed()) {- // Touched on every use, which is what makes "least recently used" mean anything.- existing.usedAt = Date.now();+ // Touched on every use, which is what makes "least recently used" mean anything. The page in+ // front with it: a Bot's next action, and a person's input on the live screen, land there.+ existing.usedAt = now();+ touchTab(existing.page);return existing.page;}if (existing) {@@ -748,6 +974,9 @@ export function createProfiles(if (isPersonBrowserId(botId) && !open) throw new BrowserNotOpenError();const launch = (async () => {+ // Before the launch, which is when memory jumps: a cold Chromium and its first page are the+ // largest single step up a computer takes, and the OOM killer does not wait for the cap.+ await relieveMemory();const dir = directoryFor(botId);// A second native helper can target the same data root on another port. Do not remove an// active Chrome profile's lock; Chrome reports contention and handles its own stale locks.@@ -800,13 +1029,18 @@ export function createProfiles(context,page,startedAt: new Date().toISOString(),- usedAt: Date.now(),+ usedAt: now(),retarget: () => {},...(sessionCookies ? { sessionCookies } : {}),};+ touchTab(page);+ touchOnNavigation(page);record.retarget = () => {const next = chooseLivePage(context.pages());if (!next || next === record.page) return;+ // Both are in use at this moment: the one going behind was in front until now.+ touchTab(record.page);+ touchTab(next);record.page = next;console.info(JSON.stringify({@@ -828,6 +1062,8 @@ export function createProfiles(// in a new window is neither shown to the person taking the wheel nor reachable by input.context.on("page", (opened) => {if (!local) void attachPageNetwork(context, opened, network);+ touchTab(opened);+ touchOnNavigation(opened);record.retarget();// A popup closes itself when it succeeds, and the record must move back to the opener// rather than leave a closed page to be read as a dead browser.@@ -959,7 +1195,7 @@ export function createProfiles(const pages = () => personPages(running.context, person.recorderId);if (!windowPages(running.context).includes(running.page))return undefined;- running.usedAt = Date.now();+ running.usedAt = now();return {...person,context: running.context,diff --git a/agent-computer/src/sessions.ts b/agent-computer/src/sessions.tsindex 72177328..0b59f398 100644--- a/agent-computer/src/sessions.ts+++ b/agent-computer/src/sessions.ts@@ -102,15 +102,24 @@ export function createSessions(options: SessionsOptions) {for (const [botId, session] of [...sessions.entries()]) {if (session.viewer.occupied()) continue;if (options.isLive(botId)) continue;- if (- session.control.get().request?.status === "waiting" ||- session.control.get().holder === "human"- )- continue;+ if (wheelWithPerson(session)) continue;sessions.delete(botId);}}+ /**+ * Whether a person holds this Bot's wheel, or the Bot has asked one to take it and is waiting.+ *+ * Either way the browser is somebody's to work in or about to be, so it is kept: here, when the map+ * is trimmed, and by the computer when it closes browsers for memory or closes a Bot's idle tabs.+ * A request that has waited past its time has expired by the time this asks (`get` expires it), so+ * a Bot that asked and was never answered does not hold its browser for ever.+ */+ function wheelWithPerson(session: BotSession): boolean {+ const state = session.control.get();+ return state.request?.status === "waiting" || state.holder === "human";+ }+function sessionFor(botId: string): BotSession {const existing = sessions.get(botId);if (existing) return existing;@@ -164,6 +173,17 @@ export function createSessions(options: SessionsOptions) {return sessions.get(botId);},+ /**+ * Whether a person holds this Bot's wheel or has been asked to take it (`wheelWithPerson`).+ *+ * Asks without answering, like `get`: a Bot with no session has nobody at its wheel, and this is+ * asked on the paths that close browsers, where creating an entry would grow the map.+ */+ personHasTheWheel(botId: string): boolean {+ const session = sessions.get(botId);+ return session ? wheelWithPerson(session) : false;+ },+/*** Start a new run for this Bot, because its browser session is over.*diff --git a/agent-computer/tests/browser-eviction.test.ts b/agent-computer/tests/browser-eviction.test.tsindex 85f6f8ae..db569d12 100644--- a/agent-computer/tests/browser-eviction.test.ts+++ b/agent-computer/tests/browser-eviction.test.ts@@ -1,5 +1,12 @@import { describe, expect, test } from "bun:test";-import { chooseEvictions, chooseIdle, keptOpen } from "../src/browser-eviction";+import {+ chooseEvictions,+ chooseForMemory,+ chooseIdle,+ chooseIdleTabs,+ keptOpen,+ RECENTLY_USED_MS,+} from "../src/browser-eviction";import { numberFromEnv } from "../src/env";/**@@ -252,3 +259,172 @@ describe("a person's cloud browser among Bots' browsers", () => {]);});});++/**+ * Which browser to close next when the computer's memory runs high.+ *+ * The cap counts browsers and the idle sweep counts minutes, and neither looks at memory, so a few+ * heavy pages took a container past its limit and the OOM killer took every browser on it at once.+ * The computer now closes browsers one at a time while memory runs high, reading it again after each,+ * and never one that is in use: a Bot between two steps asks for nothing while its model thinks, and+ * closing its browser then closes it in the middle of a run.+ */+describe("closing a browser because memory runs high", () => {+ const NOW = 10_000_000;+ const MINUTE = 60_000;+ const PERSON = "person-0123456789abcdef0123456789abcdef";+ /** Bots used this long before NOW, in order. */+ const usedAgo = (...ago: number[]) => running(...ago.map((ms) => NOW - ms));++ test("closes the least recently used of the browsers nobody has used for a minute", () => {+ expect(+ chooseForMemory(usedAgo(5 * MINUTE, 9 * MINUTE, 2 * MINUTE, 1_000), NOW),+ ).toBe("bot-1");+ });++ test("never a Bot used seconds ago, which is a Bot whose model is working out its next step", () => {+ // bot-0 asked for its page ten seconds ago and has asked for nothing since. It is the least+ // recently used here and it is still in the middle of its run.+ expect(chooseForMemory(usedAgo(10_000, 5_000), NOW)).toBeUndefined();+ expect(chooseForMemory(usedAgo(10_000, 3 * MINUTE, 5_000), NOW)).toBe(+ "bot-1",+ );+ });++ test("a minute is the line: used exactly a minute ago may close, a moment inside it may not", () => {+ expect(chooseForMemory(usedAgo(MINUTE, 0), NOW)).toBe("bot-0");+ expect(chooseForMemory(usedAgo(MINUTE - 1, 0), NOW)).toBeUndefined();+ expect(RECENTLY_USED_MS).toBe(MINUTE);+ });++ test("never the most recently used browser running, however long ago that was", () => {+ // Before a launch the caller passes the browsers already running, and the one launching is not+ // among them. Of those, the newest stays even when every one is an hour quiet, so one is left.+ expect(chooseForMemory(usedAgo(60 * MINUTE), NOW)).toBeUndefined();+ expect(chooseForMemory(usedAgo(90 * MINUTE, 60 * MINUTE), NOW)).toBe(+ "bot-0",+ );+ expect(chooseForMemory(running(), NOW)).toBeUndefined();+ });++ test("a browser that has just launched is never what the next launch closes", () => {+ // The loop this rules out: each Bot's launch closing the browser the Bot before it just started.+ // Here the one just launched is not the newest (another Bot asked a second ago), so only the+ // minute keeps it.+ const justLaunched = new Map([+ ["just-launched", { usedAt: NOW - 2_000 }],+ ["asked-a-second-ago", { usedAt: NOW - 1_000 }],+ ["quiet", { usedAt: NOW - 20 * MINUTE }],+ ]);+ expect(chooseForMemory(justLaunched, NOW)).toBe("quiet");+ justLaunched.delete("quiet");+ expect(chooseForMemory(justLaunched, NOW)).toBeUndefined();+ });++ test("never a person's own browser, cloud or native, even the one used longest ago", () => {+ // Their Recorder runs in it, and closing it ends what they were recording without a word.+ const withPerson = new Map([+ [PERSON, { usedAt: NOW - 120 * MINUTE }],+ ...usedAgo(30 * MINUTE, 10 * MINUTE),+ ]);+ expect(chooseForMemory(withPerson, NOW)).toBe("bot-0");+ withPerson.delete("bot-0");+ expect(chooseForMemory(withPerson, NOW)).toBeUndefined();+ });++ test("never a browser whose wheel a person holds", () => {+ const held = (botId: string) => botId === "bot-0";+ expect(+ chooseForMemory(usedAgo(30 * MINUTE, 20 * MINUTE, 10 * MINUTE), NOW, {+ spared: held,+ }),+ ).toBe("bot-1");+ expect(+ chooseForMemory(usedAgo(30 * MINUTE, 10 * MINUTE), NOW, { spared: held }),+ ).toBeUndefined();+ });++ test("the window can be narrowed, which is how the browser tests run without waiting a minute", () => {+ expect(chooseForMemory(usedAgo(500, 0), NOW, { recentlyUsedMs: 100 })).toBe(+ "bot-0",+ );+ });+});++/**+ * Which of a Bot's background tabs to close because nobody has touched them.+ *+ * A Bot is always on its newest page, so a tab it left behind, the page a sign-in popup opened from+ * or a link a site opened in a new tab, is one it cannot go back to, and it held a renderer for as+ * long as the browser stayed up.+ */+describe("closing a Bot's idle background tabs", () => {+ const NOW = 1_000_000;+ const IDLE = 10 * 60_000;+ const tab = (name: string, usedAt: number, front = false, kept = false) => ({+ name,+ usedAt,+ front,+ kept,+ });+ const names = (tabs: { name: string }[]) => tabs.map((t) => t.name);++ test("closes a background tab past the timeout, and keeps the page in front", () => {+ const tabs = [+ tab("left behind", NOW - IDLE - 1),+ tab("in front", NOW - IDLE - 1, true),+ ];+ // The page in front is as old as the other and is still never closed: it is the one the Bot is on.+ expect(names(chooseIdleTabs(tabs, IDLE, NOW))).toEqual(["left behind"]);+ });++ test("keeps a background tab used inside the timeout, and closes one at it", () => {+ expect(+ names(+ chooseIdleTabs(+ [+ tab("recent", NOW - IDLE + 1),+ tab("at the line", NOW - IDLE),+ tab("in front", NOW, true),+ ],+ IDLE,+ NOW,+ ),+ ),+ ).toEqual(["at the line"]);+ });++ test("never the last tab", () => {+ expect(chooseIdleTabs([tab("only", 0, true)], IDLE, NOW)).toEqual([]);+ });++ test("closes nothing when which page is in front is not known", () => {+ // Any tab could then be the one the Bot is on, and closing it would move the Bot.+ expect(chooseIdleTabs([tab("a", 0), tab("b", 0)], IDLE, NOW)).toEqual([]);+ expect(+ chooseIdleTabs([tab("a", 0, true), tab("b", 0, true)], IDLE, NOW),+ ).toEqual([]);+ });++ test("never an extension's own page or DevTools", () => {+ expect(+ names(+ chooseIdleTabs(+ [+ tab("recorder", 0, false, true),+ tab("old", 0),+ tab("front", NOW, true),+ ],+ IDLE,+ NOW,+ ),+ ),+ ).toEqual(["old"]);+ });++ test("a timeout of zero switches it off", () => {+ expect(+ chooseIdleTabs([tab("old", 0), tab("front", NOW, true)], 0, NOW),+ ).toEqual([]);+ });+});diff --git a/agent-computer/tests/browser-memory.test.ts b/agent-computer/tests/browser-memory.test.tsnew file mode 100644index 00000000..d300790c--- /dev/null+++ b/agent-computer/tests/browser-memory.test.ts@@ -0,0 +1,379 @@+import { afterAll, afterEach, describe, expect, spyOn, test } from "bun:test";+import { mkdtemp, rm } from "node:fs/promises";+import { tmpdir } from "node:os";+import { join } from "node:path";+import type { BrowserContext, Page } from "playwright";+import type { MemoryReading } from "../src/container-memory";++/**+ * A computer that closes browsers before its memory runs out, and a Bot's tabs it left behind.+ *+ * The cap counted browsers and the idle sweep counted minutes, and neither looked at memory, so a few+ * heavy pages could take a container past its limit, where the OOM killer takes every browser on it+ * at once rather than one. These drive `createProfiles` with real Chromium, handing it the memory+ * reading a container would give, because the closes they cover are made from inside another Bot's+ * launch and from the idle sweep, and no request reaches either.+ *+ * Asked for by name, like the other tests that launch a real Chromium:+ *+ * cd agent-computer && bunx playwright install chromium+ * OPENBOT_COMPUTER_BROWSER=1 COMPUTER_BROWSER_MODE=headless bun test tests/browser-memory.test.ts+ */+const asked = process.env.OPENBOT_COMPUTER_BROWSER === "1";+const LAUNCH_TIMEOUT_MS = 120_000;++const html = (title: string) =>+ new Response(`<!doctype html><title>${title}</title><body>${title}</body>`, {+ headers: { "content-type": "text/html; charset=utf-8" },+ });+const site = Bun.serve({+ port: 0,+ fetch(request) {+ return html(new URL(request.url).pathname.slice(1) || "home");+ },+});+const origin = `http://127.0.0.1:${site.port}`;++const roots: string[] = [];+const said = spyOn(console, "info");++afterEach(() => {+ said.mockClear();+});++afterAll(async () => {+ site.stop(true);+ said.mockRestore();+ await Promise.all(+ roots.map((root) => rm(root, { recursive: true, force: true })),+ );+});++/** The structured lines the computer logged of one type. */+function logged(type: string): Record<string, unknown>[] {+ return said.mock.calls+ .map(([line]) => {+ try {+ return JSON.parse(String(line)) as Record<string, unknown>;+ } catch {+ return undefined;+ }+ })+ .filter((line): line is Record<string, unknown> => line?.type === type);+}++async function profilesFor(+ name: string,+ options: Parameters<typeof import("../src/profiles").createProfiles>[2],+) {+ const { createProfiles } = await import("../src/profiles");+ const root = await mkdtemp(join(tmpdir(), `openbot-${name}-`));+ roots.push(root);+ const told: string[] = [];+ const profiles = createProfiles(+ root,+ (botId) => {+ told.push(botId);+ },+ options,+ );+ return { profiles, told };+}++const MIB = 1024 ** 2;+const LIMIT = 1000 * MIB;+const MINUTE = 60_000;+const PERSON = "person-0123456789abcdef0123456789abcdef";++/**+ * A container whose memory is its browsers, read the way the cgroup reads it.+ *+ * Each browser holds the same share, counted while its Chromium is still connected, because that is+ * when the kernel gives its memory back: when the process exits, seconds after the computer took it+ * out of its map, not at the moment it decided to close it. The browsers are real; only their size+ * is set, so a test can say how many fit.+ */+function containerOf(percentEach: number) {+ const contexts = new Set<BrowserContext>();+ return {+ /** Count this page's browser from now on. */+ holds(page: Page): Page {+ contexts.add(page.context());+ return page;+ },+ memory: async (): Promise<MemoryReading> => ({+ workingSet:+ [...contexts].filter((context) => context.browser()?.isConnected())+ .length *+ percentEach *+ 10 *+ MIB,+ limit: LIMIT,+ }),+ };+}++/**+ * A clock that can be moved on, so every browser used so far reads as quiet for as long as a test+ * says, without the test waiting a minute for the window in which a browser counts as in use.+ */+function clock() {+ let ahead = 0;+ return {+ now: () => Date.now() + ahead,+ passes(ms: number) {+ ahead += ms;+ },+ };+}++const closedForMemory = () =>+ logged("computer-browser-closed-for-memory").map((line) => line.botId);++describe.skipIf(!asked)("closing browsers when memory runs high", () => {+ test(+ "a Bot used seconds ago is not closed when another launches; the quiet ones are",+ async () => {+ // Two browsers are 90%, over the line at 85.+ const container = containerOf(45);+ const time = clock();+ const { profiles, told } = await profilesFor("memory-recent", {+ memory: container.memory,+ now: time.now,+ });+ const open = async (botId: string) =>+ container.holds(await profiles.page(botId));+ try {+ await open("first");+ await open("second");+ // Memory is high, and both Bots asked for their pages seconds ago: either could be waiting on+ // its model between two steps. Neither is closed, and the third launches anyway.+ await open("third");+ expect(told).toEqual([]);+ expect(profiles.liveCount()).toBe(3);++ // Ten minutes on, the first Bot takes a step. The other two have been quiet all that time.+ time.passes(10 * MINUTE);+ await open("first");+ await open("fourth");++ // Both quiet ones close, down to under 75: three browsers were 135%, one is 45%.+ expect(told).toEqual(["second", "third"]);+ expect(profiles.isLive("first")).toBe(true);+ expect(profiles.isLive("fourth")).toBe(true);+ expect(logged("computer-browser-closed-for-memory")[0]).toEqual({+ type: "computer-browser-closed-for-memory",+ botId: "second",+ workingSetBytes: 3 * 450 * MIB,+ limitBytes: LIMIT,+ });+ } finally {+ await profiles.closeAll();+ }+ },+ LAUNCH_TIMEOUT_MS,+ );++ test(+ "a burst of launches closes only what memory needs, not one browser per launch",+ async () => {+ // Three browsers are 90%; one closed is 60%, under 75.+ const container = containerOf(30);+ const time = clock();+ const { profiles, told } = await profilesFor("memory-burst", {+ memory: container.memory,+ now: time.now,+ });+ const open = async (botId: string) =>+ container.holds(await profiles.page(botId));+ try {+ await open("a");+ await open("b");+ await open("c");+ time.passes(10 * MINUTE);++ // Three Bots at once. Each launch used to read memory itself, before the first close had+ // given anything back, and each closed a browser of its own.+ await Promise.all([open("d"), open("e"), open("f")]);++ expect(told).toEqual(["a"]);+ expect(closedForMemory()).toEqual(["a"]);+ expect(profiles.liveCount()).toBe(5);+ } finally {+ await profiles.closeAll();+ }+ },+ LAUNCH_TIMEOUT_MS,+ );++ test(+ "a Bot that comes back after its browser was closed does not start a cycle of closes",+ async () => {+ // Four browsers are 100%. One closed is 75%, which is under where closing starts (85) and not+ // under where it stops (75), so the pass goes on to 50%.+ const container = containerOf(25);+ const time = clock();+ const { profiles, told } = await profilesFor("memory-cycle", {+ memory: container.memory,+ now: time.now,+ });+ const open = async (botId: string) =>+ container.holds(await profiles.page(botId));+ try {+ for (const botId of ["a", "b", "c", "d"]) await open(botId);+ time.passes(10 * MINUTE);++ await open("e");+ // Two, not one: a pass that stopped at 75% left the next launch at 100% again.+ expect(told).toEqual(["a", "b"]);++ // Each Bot whose browser was closed comes straight back, as a Bot in the middle of its work+ // would. Each comeback used to close the least recently used browser, which by then was the+ // one the last comeback had just launched, and so on round the ring for ever.+ for (let round = 0; round < 8; round += 1) {+ const closed = told.find((botId) => !profiles.isLive(botId));+ if (!closed) break;+ await open(closed);+ }++ // Every quiet browser was closed once, and none that was in use: the Bots are all back.+ expect(told).toEqual(["a", "b", "c", "d"]);+ expect(new Set(told).size).toBe(told.length);+ for (const botId of ["a", "b", "c", "d", "e"])+ expect(profiles.isLive(botId)).toBe(true);+ } finally {+ await profiles.closeAll();+ }+ },+ LAUNCH_TIMEOUT_MS,+ );++ test(+ "a Bot a person is working in stays, and memory that will not come down leaves the newest",+ async () => {+ // Memory that reads full whatever is closed: one browser too big on its own. The pass has to+ // stop, not close its way to nothing.+ const time = clock();+ const { profiles, told } = await profilesFor("memory-held", {+ memory: async () => ({ workingSet: LIMIT, limit: LIMIT }),+ now: time.now,+ heldByPerson: (botId) => botId === "held",+ });+ try {+ await profiles.page("held");+ await profiles.page("other");+ await profiles.page("third");+ time.passes(10 * MINUTE);++ await profiles.page("asking");+ // "held" is a person's to work in and "third" was the most recently used: "other" goes.+ expect(told).toEqual(["other"]);++ // The sweep's tick reads full again. "third" is quiet and goes; "held" and the newest stay.+ await profiles.sweepIdleNow();+ expect(told).toEqual(["other", "third"]);+ expect(profiles.isLive("held")).toBe(true);+ expect(profiles.isLive("asking")).toBe(true);+ expect(profiles.liveCount()).toBe(2);+ } finally {+ await profiles.closeAll();+ }+ },+ LAUNCH_TIMEOUT_MS,+ );++ test(+ "a person's own browser is never closed for memory, even the one used longest ago",+ async () => {+ // Their Recorder runs in it, so closing it would end what they were recording without a word.+ const { findRecorder } = await import("../src/recorder");+ const found = await findRecorder(+ join(import.meta.dir, "fixtures", "recorder"),+ );+ if (!found.ok) throw new Error(found.error);+ const container = containerOf(30);+ const time = clock();+ const { profiles, told } = await profilesFor("memory-person", {+ memory: container.memory,+ now: time.now,+ });+ try {+ container.holds(+ await profiles.page(PERSON, { open: { recorder: found.recorder } }),+ );+ container.holds(await profiles.page("bot-a"));+ container.holds(await profiles.page("bot-b"));+ time.passes(10 * MINUTE);++ container.holds(await profiles.page("bot-c"));++ // The person's browser is the least recently used by ten minutes and stays; the quieter of+ // the two Bots goes in its place.+ expect(told).toEqual(["bot-a"]);+ expect(profiles.isLive(PERSON)).toBe(true);+ expect(profiles.person(PERSON)).toBeDefined();+ } finally {+ await profiles.closeAll();+ }+ },+ LAUNCH_TIMEOUT_MS,+ );+});++describe.skipIf(!asked)("closing a Bot's idle background tabs", () => {+ test(+ "the sweep closes a background tab left past the timeout, and the Bot stays on its page",+ async () => {+ let held = true;+ const made = await profilesFor("tabs", {+ memory: async () => undefined,+ // The shortest timeout the sweep acts on, so the test waits milliseconds rather than ten minutes.+ tabIdleMs: 1,+ heldByPerson: () => held,+ });+ const { profiles } = made;+ try {+ const behind = await profiles.page("tabs");+ await behind.goto(`${origin}/left-behind`, {+ waitUntil: "domcontentloaded",+ });+ const front = await behind.context().newPage();+ await front.goto(`${origin}/in-front`, {+ waitUntil: "domcontentloaded",+ });+ // The new tab is the Bot's page now, as a window a site opens would be.+ expect(await profiles.page("tabs")).toBe(front);+ await new Promise((resolve) => setTimeout(resolve, 25));++ // A person holds the wheel: they may be working across these tabs, so nothing is closed.+ await profiles.sweepIdleNow();+ expect(behind.isClosed()).toBe(false);++ held = false;+ await new Promise((resolve) => setTimeout(resolve, 25));+ said.mockClear();+ await profiles.sweepIdleNow();++ expect(behind.isClosed()).toBe(true);+ expect(front.isClosed()).toBe(false);+ expect(front.context().pages()).toEqual([front]);+ expect(logged("computer-tab-closed")).toEqual([+ {+ type: "computer-tab-closed",+ botId: "tabs",+ reason: "it had been idle",+ },+ ]);+ // Still the Bot's page: the record never moved, and the browser was not closed.+ expect(logged("computer-page-changed")).toEqual([]);+ const after = await profiles.page("tabs");+ expect(after).toBe(front);+ expect(after.url()).toBe(`${origin}/in-front`);+ expect(made.told).toEqual([]);+ } finally {+ await profiles.closeAll();+ }+ },+ LAUNCH_TIMEOUT_MS,+ );+});diff --git a/agent-computer/tests/container-memory.test.ts b/agent-computer/tests/container-memory.test.tsnew file mode 100644index 00000000..4cfa17f5--- /dev/null+++ b/agent-computer/tests/container-memory.test.ts@@ -0,0 +1,149 @@+import { describe, expect, test } from "bun:test";+import { mkdtemp, rm } from "node:fs/promises";+import { tmpdir } from "node:os";+import { join } from "node:path";+import {+ memoryAtOrAbove,+ memoryRunsHigh,+ memoryTargetPercent,+ readContainerMemory,+} from "../src/container-memory";++/**+ * How much of its memory a computer is using, from the files a real container wrote.+ *+ * Every fixture under `fixtures/cgroup` is laid out the way a container sees `/sys/fs/cgroup`, and+ * read here with the same reader the computer uses, only rooted elsewhere. The cgroup v2 ones were+ * captured from throwaway containers (ubuntu:24.04 on Docker 29.5.2, cgroup v2):+ *+ * v2-limit-512m docker run --rm --memory 512m, at rest+ * v2-no-limit docker run --rm with no --memory, so `memory.max` reads `max`+ * v2-page-cache --memory 256m after writing 96 MB to a file, which sits in the page cache+ * v2-shm-near-limit --memory 256m --shm-size 256m after writing 224 MB to /dev/shm, which is+ * where Chromium keeps the memory its processes share+ *+ * `v1-no-limit/memory/memory.stat` is a cgroup v1 file captured from a container with no limit, as+ * Apache Druid keeps it for its own tests (processing/src/test/resources/memory.stat); this machine's+ * kernel runs cgroup v2, which cannot mount the v1 memory controller. Its `memory.limit_in_bytes` is+ * the `hierarchical_memory_limit` that same file reports, and its `memory.usage_in_bytes` is the+ * file's `cache` plus `rss`, which is what v1 usage is made of.+ */++const FIXTURES = join(import.meta.dir, "fixtures", "cgroup");++/** The memory of the VM the v2 fixtures were captured in: `MemTotal` from its /proc/meminfo. */+const VM_BYTES = 16_353_252 * 1024;++const read = (fixture: string) =>+ readContainerMemory({+ root: join(FIXTURES, fixture),+ machineBytes: VM_BYTES,+ });++describe("reading a container's memory as Kubernetes reads it", () => {+ test("a container with a limit is judged against that limit", async () => {+ // `memory.current` less `inactive_file`, exactly as cAdvisor computes the working set.+ expect(await read("v2-limit-512m")).toEqual({+ workingSet: 1_228_800 - 16_384,+ limit: 512 * 1024 * 1024,+ });+ });++ test("a container with no limit is judged against the machine, which is what runs out", async () => {+ // `memory.max` reads `max`. Inside a container the machine is the VM, and its memory is the+ // limit the OOM killer actually enforces.+ expect(await read("v2-no-limit")).toEqual({+ workingSet: 1_568_768,+ limit: VM_BYTES,+ });+ });++ test("a file in the page cache is not memory the container is short of", async () => {+ /*+ * The case that makes it the working set rather than usage. After writing a 96 MB file, usage+ * reads 39% of a 256 MB limit, and nearly all of it is inactive file cache the kernel drops the+ * moment anything needs the room. Judged on usage, a Bot saving a large download would close+ * other Bots' browsers for memory nobody was short of.+ */+ const reading = await read("v2-page-cache");+ expect(reading).toEqual({+ workingSet: 105_312_256 - 100_982_784,+ limit: 256 * 1024 * 1024,+ });+ if (!reading) throw new Error("no reading");+ expect(105_312_256 / reading.limit).toBeGreaterThan(0.39);+ expect(memoryRunsHigh(reading, 30)).toBe(false);+ });++ test("memory held in /dev/shm counts, and near the limit it reads as high", async () => {+ // Shared memory cannot be dropped like a cache, and Chromium's processes share rendered frames+ // through it. 224 MB of it is 88% of a 256 MB limit.+ const reading = await read("v2-shm-near-limit");+ expect(reading).toEqual({+ workingSet: 236_740_608,+ limit: 256 * 1024 * 1024,+ });+ if (!reading) throw new Error("no reading");+ expect(memoryRunsHigh(reading, 85)).toBe(true);+ expect(memoryRunsHigh(reading, 90)).toBe(false);+ });++ test("cgroup v1, with no limit, is read from its own files and judged against the machine", async () => {+ // v1 writes no limit as a number near 2^63 rather than `max`, and its inactive cache is+ // `total_inactive_file`, which includes the cgroups beneath it.+ expect(await read("v1-no-limit")).toEqual({+ workingSet: 9_711_616 - 2_019_328,+ limit: VM_BYTES,+ });+ });++ test("a computer with no cgroup of its own reads nothing, and so closes nothing for memory", async () => {+ // OpenBot Desktop's computer on a Mac, or one started by hand on a Linux desktop, whose view of+ // /sys/fs/cgroup is the root, where none of these files exist.+ const empty = await mkdtemp(join(tmpdir(), "openbot-no-cgroup-"));+ try {+ expect(+ await readContainerMemory({ root: empty, machineBytes: VM_BYTES }),+ ).toBeUndefined();+ } finally {+ await rm(empty, { recursive: true, force: true });+ }+ });+});++describe("whether memory runs high", () => {+ const LIMIT = 1_000;++ test("at the share it acts, and a byte under it does not", () => {+ // At or above, so a deployment that said 85 has acted by the time it reads 85.+ expect(memoryRunsHigh({ workingSet: 850, limit: LIMIT }, 85)).toBe(true);+ expect(memoryRunsHigh({ workingSet: 849, limit: LIMIT }, 85)).toBe(false);+ });++ test("a share of zero switches it off", () => {+ expect(memoryRunsHigh({ workingSet: LIMIT, limit: LIMIT }, 0)).toBe(false);+ });+});++describe("where closing for memory stops", () => {+ const LIMIT = 1_000;++ test("ten points under where it starts, so 75 at the default 85", () => {+ expect(memoryTargetPercent(85)).toBe(75);+ expect(memoryTargetPercent(60)).toBe(50);+ });++ test("between the two lines nothing starts, and a pass that started goes on", () => {+ // 80% does not start a pass at 85, and a pass already closing at 85 does not stop at 80: it+ // closes on to under 75, so the next launch does not cross 85 again straight away.+ const at80 = { workingSet: 800, limit: LIMIT };+ expect(memoryRunsHigh(at80, 85)).toBe(false);+ expect(memoryAtOrAbove(at80, memoryTargetPercent(85))).toBe(true);+ expect(+ memoryAtOrAbove(+ { workingSet: 749, limit: LIMIT },+ memoryTargetPercent(85),+ ),+ ).toBe(false);+ });+});diff --git a/agent-computer/tests/fixtures/cgroup/v1-no-limit/memory/memory.limit_in_bytes b/agent-computer/tests/fixtures/cgroup/v1-no-limit/memory/memory.limit_in_bytesnew file mode 100644index 00000000..564113cf--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v1-no-limit/memory/memory.limit_in_bytes@@ -0,0 +1 @@+9223372036854771712diff --git a/agent-computer/tests/fixtures/cgroup/v1-no-limit/memory/memory.stat b/agent-computer/tests/fixtures/cgroup/v1-no-limit/memory/memory.statnew file mode 100644index 00000000..e68ddcee--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v1-no-limit/memory/memory.stat@@ -0,0 +1,36 @@+cache 7892992+rss 1818624+rss_huge 0+shmem 0+mapped_file 1327104+dirty 0+writeback 0+swap 0+pgpgin 8346+pgpgout 5975+pgfault 13137+pgmajfault 120+inactive_anon 0+active_anon 1757184+inactive_file 2019328+active_file 5873664+unevictable 0+hierarchical_memory_limit 9223372036854771712+hierarchical_memsw_limit 9223372036854771712+total_cache 7892992+total_rss 1818624+total_rss_huge 0+total_shmem 0+total_mapped_file 1327104+total_dirty 0+total_writeback 0+total_swap 0+total_pgpgin 8346+total_pgpgout 5975+total_pgfault 13137+total_pgmajfault 120+total_inactive_anon 0+total_active_anon 1757184+total_inactive_file 2019328+total_active_file 5873664+total_unevictable 0diff --git a/agent-computer/tests/fixtures/cgroup/v1-no-limit/memory/memory.usage_in_bytes b/agent-computer/tests/fixtures/cgroup/v1-no-limit/memory/memory.usage_in_bytesnew file mode 100644index 00000000..69f9852b--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v1-no-limit/memory/memory.usage_in_bytes@@ -0,0 +1 @@+9711616diff --git a/agent-computer/tests/fixtures/cgroup/v2-limit-512m/memory.current b/agent-computer/tests/fixtures/cgroup/v2-limit-512m/memory.currentnew file mode 100644index 00000000..9ff0b8de--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v2-limit-512m/memory.current@@ -0,0 +1 @@+1228800diff --git a/agent-computer/tests/fixtures/cgroup/v2-limit-512m/memory.max b/agent-computer/tests/fixtures/cgroup/v2-limit-512m/memory.maxnew file mode 100644index 00000000..1f585689--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v2-limit-512m/memory.max@@ -0,0 +1 @@+536870912diff --git a/agent-computer/tests/fixtures/cgroup/v2-limit-512m/memory.stat b/agent-computer/tests/fixtures/cgroup/v2-limit-512m/memory.statnew file mode 100644index 00000000..a6a5c98b--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v2-limit-512m/memory.stat@@ -0,0 +1,61 @@+anon 188416+file 77824+kernel 1069056+kernel_stack 32768+pagetables 86016+sec_pagetables 0+percpu 43920+sock 0+vmalloc 8192+shmem 0+zswap 0+zswapped 0+file_mapped 28672+file_dirty 0+file_writeback 0+swapcached 0+anon_thp 0+file_thp 0+shmem_thp 0+inactive_anon 0+active_anon 122880+inactive_file 16384+active_file 53248+unevictable 0+slab_reclaimable 95992+slab_unreclaimable 391144+slab 487136+workingset_refault_anon 0+workingset_refault_file 0+workingset_activate_anon 0+workingset_activate_file 0+workingset_restore_anon 0+workingset_restore_file 0+workingset_nodereclaim 0+pgdemote_kswapd 0+pgdemote_direct 0+pgdemote_khugepaged 0+pgscan 0+pgsteal 0+pgscan_kswapd 0+pgscan_direct 0+pgscan_khugepaged 0+pgsteal_kswapd 0+pgsteal_direct 0+pgsteal_khugepaged 0+pgfault 1473+pgmajfault 2+pgrefill 0+pgactivate 0+pgdeactivate 0+pglazyfree 0+pglazyfreed 0+swpin_zero 0+swpout_zero 0+zswpin 0+zswpout 0+zswpwb 0+thp_fault_alloc 0+thp_collapse_alloc 0+thp_swpout 0+thp_swpout_fallback 0diff --git a/agent-computer/tests/fixtures/cgroup/v2-no-limit/memory.current b/agent-computer/tests/fixtures/cgroup/v2-no-limit/memory.currentnew file mode 100644index 00000000..66fb7c4c--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v2-no-limit/memory.current@@ -0,0 +1 @@+1568768diff --git a/agent-computer/tests/fixtures/cgroup/v2-no-limit/memory.max b/agent-computer/tests/fixtures/cgroup/v2-no-limit/memory.maxnew file mode 100644index 00000000..355295a0--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v2-no-limit/memory.max@@ -0,0 +1 @@+maxdiff --git a/agent-computer/tests/fixtures/cgroup/v2-no-limit/memory.stat b/agent-computer/tests/fixtures/cgroup/v2-no-limit/memory.statnew file mode 100644index 00000000..c2f0ae55--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v2-no-limit/memory.stat@@ -0,0 +1,61 @@+anon 188416+file 0+kernel 1073152+kernel_stack 32768+pagetables 86016+sec_pagetables 0+percpu 44560+sock 0+vmalloc 8192+shmem 0+zswap 0+zswapped 0+file_mapped 0+file_dirty 0+file_writeback 0+swapcached 0+anon_thp 0+file_thp 0+shmem_thp 0+inactive_anon 0+active_anon 155648+inactive_file 0+active_file 0+unevictable 0+slab_reclaimable 90760+slab_unreclaimable 393544+slab 484304+workingset_refault_anon 0+workingset_refault_file 0+workingset_activate_anon 0+workingset_activate_file 0+workingset_restore_anon 0+workingset_restore_file 0+workingset_nodereclaim 0+pgdemote_kswapd 0+pgdemote_direct 0+pgdemote_khugepaged 0+pgscan 0+pgsteal 0+pgscan_kswapd 0+pgscan_direct 0+pgscan_khugepaged 0+pgsteal_kswapd 0+pgsteal_direct 0+pgsteal_khugepaged 0+pgfault 1469+pgmajfault 0+pgrefill 0+pgactivate 0+pgdeactivate 0+pglazyfree 0+pglazyfreed 0+swpin_zero 0+swpout_zero 0+zswpin 0+zswpout 0+zswpwb 0+thp_fault_alloc 0+thp_collapse_alloc 0+thp_swpout 0+thp_swpout_fallback 0diff --git a/agent-computer/tests/fixtures/cgroup/v2-page-cache/memory.current b/agent-computer/tests/fixtures/cgroup/v2-page-cache/memory.currentnew file mode 100644index 00000000..67b2df57--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v2-page-cache/memory.current@@ -0,0 +1 @@+105312256diff --git a/agent-computer/tests/fixtures/cgroup/v2-page-cache/memory.max b/agent-computer/tests/fixtures/cgroup/v2-page-cache/memory.maxnew file mode 100644index 00000000..853f47e7--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v2-page-cache/memory.max@@ -0,0 +1 @@+268435456diff --git a/agent-computer/tests/fixtures/cgroup/v2-page-cache/memory.stat b/agent-computer/tests/fixtures/cgroup/v2-page-cache/memory.statnew file mode 100644index 00000000..f0279ac4--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v2-page-cache/memory.stat@@ -0,0 +1,61 @@+anon 184320+file 101089280+kernel 3846144+kernel_stack 32768+pagetables 81920+sec_pagetables 0+percpu 43920+sock 0+vmalloc 8192+shmem 0+zswap 0+zswapped 0+file_mapped 0+file_dirty 0+file_writeback 0+swapcached 0+anon_thp 0+file_thp 0+shmem_thp 0+inactive_anon 0+active_anon 167936+inactive_file 100982784+active_file 106496+unevictable 0+slab_reclaimable 3079728+slab_unreclaimable 381048+slab 3460776+workingset_refault_anon 0+workingset_refault_file 30+workingset_activate_anon 0+workingset_activate_file 0+workingset_restore_anon 0+workingset_restore_file 0+workingset_nodereclaim 0+pgdemote_kswapd 0+pgdemote_direct 0+pgdemote_khugepaged 0+pgscan 0+pgsteal 0+pgscan_kswapd 0+pgscan_direct 0+pgscan_khugepaged 0+pgsteal_kswapd 0+pgsteal_direct 0+pgsteal_khugepaged 0+pgfault 1807+pgmajfault 2+pgrefill 0+pgactivate 0+pgdeactivate 0+pglazyfree 0+pglazyfreed 0+swpin_zero 0+swpout_zero 0+zswpin 0+zswpout 0+zswpwb 0+thp_fault_alloc 0+thp_collapse_alloc 0+thp_swpout 0+thp_swpout_fallback 0diff --git a/agent-computer/tests/fixtures/cgroup/v2-shm-near-limit/memory.current b/agent-computer/tests/fixtures/cgroup/v2-shm-near-limit/memory.currentnew file mode 100644index 00000000..e1ec59cb--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v2-shm-near-limit/memory.current@@ -0,0 +1 @@+236740608diff --git a/agent-computer/tests/fixtures/cgroup/v2-shm-near-limit/memory.max b/agent-computer/tests/fixtures/cgroup/v2-shm-near-limit/memory.maxnew file mode 100644index 00000000..853f47e7--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v2-shm-near-limit/memory.max@@ -0,0 +1 @@+268435456diff --git a/agent-computer/tests/fixtures/cgroup/v2-shm-near-limit/memory.stat b/agent-computer/tests/fixtures/cgroup/v2-shm-near-limit/memory.statnew file mode 100644index 00000000..5bd87d14--- /dev/null+++ b/agent-computer/tests/fixtures/cgroup/v2-shm-near-limit/memory.stat@@ -0,0 +1,61 @@+anon 188416+file 234881024+kernel 1392640+kernel_stack 32768+pagetables 81920+sec_pagetables 0+percpu 44560+sock 0+vmalloc 8192+shmem 234881024+zswap 0+zswapped 0+file_mapped 0+file_dirty 0+file_writeback 0+swapcached 0+anon_thp 0+file_thp 0+shmem_thp 0+inactive_anon 0+active_anon 235053056+inactive_file 0+active_file 0+unevictable 0+slab_reclaimable 613960+slab_unreclaimable 378376+slab 992336+workingset_refault_anon 0+workingset_refault_file 0+workingset_activate_anon 0+workingset_activate_file 0+workingset_restore_anon 0+workingset_restore_file 0+workingset_nodereclaim 0+pgdemote_kswapd 0+pgdemote_direct 0+pgdemote_khugepaged 0+pgscan 0+pgsteal 0+pgscan_kswapd 0+pgscan_direct 0+pgscan_khugepaged 0+pgsteal_kswapd 0+pgsteal_direct 0+pgsteal_khugepaged 0+pgfault 1780+pgmajfault 0+pgrefill 0+pgactivate 0+pgdeactivate 0+pglazyfree 0+pglazyfreed 0+swpin_zero 0+swpout_zero 0+zswpin 0+zswpout 0+zswpwb 0+thp_fault_alloc 0+thp_collapse_alloc 0+thp_swpout 0+thp_swpout_fallback 0diff --git a/agent-computer/tests/sessions.test.ts b/agent-computer/tests/sessions.test.tsindex 86fc152f..10b0e848 100644--- a/agent-computer/tests/sessions.test.ts+++ b/agent-computer/tests/sessions.test.ts@@ -167,3 +167,21 @@ test("a replacement context interrupts a live handoff; the same context keeps itexpect(session.control.get(request.id).request?.status).toBe("interrupted");expect(session.control.humanMayDrive()).toBe(false);});++test("a Bot whose wheel a person holds, or has been asked to take, says so without a session being made", async () => {+ // The computer asks this before closing a browser for memory or a Bot's idle tabs, and neither+ // may close one under a person working in it, or under a page the Bot is waiting for them to work.+ const sessions = createSessions({ isLive: () => true, mintRun: counting() });+ expect(sessions.personHasTheWheel("bot-1")).toBe(false);+ // Asking did not make one: the paths that ask are the ones that close things.+ expect(sessions.get("bot-1")).toBeUndefined();++ const session = sessions.for("bot-1");+ expect(sessions.personHasTheWheel("bot-1")).toBe(false);+ const request = session.control.requestHelp("Sign in").request!;+ expect(sessions.personHasTheWheel("bot-1")).toBe(true);+ await session.control.take(request.id);+ expect(sessions.personHasTheWheel("bot-1")).toBe(true);+ session.control.release(request.id);+ expect(sessions.personHasTheWheel("bot-1")).toBe(false);+});diff --git a/charts/openbot/templates/_helpers.tpl b/charts/openbot/templates/_helpers.tplindex 11ba5371..9bad6a0a 100644--- a/charts/openbot/templates/_helpers.tpl+++ b/charts/openbot/templates/_helpers.tpl@@ -679,7 +679,8 @@ podAntiAffinity:The computer settings this chart has values for, as a JSON list of env entries.One definition, read by the shared StatefulSet and by the per-Bot sandbox template, so a computer-cannot be headed in one mode and headless in the other, or proxied in one and not the other.+cannot be headed in one mode and headless in the other, proxied in one and not the other, or keep a+different number of browsers in one than the other.Each key is reached through a guard, because `helm upgrade --reuse-values` hands an existing releasenone of the keys this release added (see scripts/check-new-values-keys.ts). Absent `browserMode`@@ -691,6 +692,19 @@ leaves the computer to its own default, which is headed on Linux, the same thing{{- with $computers.browserMode -}}{{- $env = append $env (dict "name" "COMPUTER_BROWSER_MODE" "value" (toString .)) -}}{{- end -}}+{{- /*+ How many browsers a computer keeps, and when it closes them and their idle tabs. Asked whether+ anybody set each, not whether it is truthy: zero switches the idle closes and the memory check+ off, and `with` reads zero as empty and would drop it. Written out in digits+ (`openbot.wholeNumber`), because a whole number from a values file arrives as a float and+ rendered `browserIdleMs: 3600000` as `3.6e+06`, which the computer refused for its default.+*/ -}}+{{- range $name, $key := dict "COMPUTER_MAX_BROWSERS" "maxBrowsers" "COMPUTER_BROWSER_IDLE_MS" "browserIdleMs" "COMPUTER_MEMORY_HIGH_PERCENT" "memoryHighPercent" "COMPUTER_TAB_IDLE_MS" "tabIdleMs" -}}+{{- $value := index $computers $key -}}+{{- if and (not (kindIs "invalid" $value)) (ne (toString $value) "") -}}+{{- $env = append $env (dict "name" $name "value" (include "openbot.wholeNumber" $value)) -}}+{{- end -}}+{{- end -}}{{- $egressProxy := $computers.egressProxy | default dict -}}{{- if $egressProxy.enabled -}}{{- $env = append $env (dict "name" "EGRESS_PROXY_DEFAULT" "valueFrom" (dict "secretKeyRef" (dict@@ -827,3 +841,16 @@ than anything that names the cause.{{- end }}{{- end -}}+{{- /*+ A number from values written out in digits, which is how the computer reads its settings.++ A whole number in a values file arrives as a float, because Helm reads values through JSON, and a+ float renders in Go's shortest form: `tabIdleMs: 1200000` became `1.2e+06`. The computer takes+ digits only, so it fell back to its default without a word. A float with nothing after the point is+ written as the whole number it is; anything else (a string, a number from `--set`) as it came.+*/}}+{{- define "openbot.wholeNumber" -}}+{{- if kindIs "float64" . -}}+{{- if eq . (floor .) -}}{{ int64 . }}{{- else -}}{{ . }}{{- end -}}+{{- else -}}{{ . }}{{- end -}}+{{- end -}}diff --git a/charts/openbot/templates/computer/statefulset.yaml b/charts/openbot/templates/computer/statefulset.yamlindex fde0a21d..03d3db41 100644--- a/charts/openbot/templates/computer/statefulset.yaml+++ b/charts/openbot/templates/computer/statefulset.yaml@@ -155,14 +155,6 @@ spec:*/}}- name: COMPUTER_SEPARATE_WORKSPACESvalue: "on"- {{- with .Values.computers.maxBrowsers }}- - name: COMPUTER_MAX_BROWSERS- value: {{ . | quote }}- {{- end }}- {{- with .Values.computers.browserIdleMs }}- - name: COMPUTER_BROWSER_IDLE_MS- value: {{ . | quote }}- {{- end }}{{- with include "openbot.computerEnv" . | fromJsonArray }}{{ toYaml . | indent 12 }}{{- end }}diff --git a/charts/openbot/values.yaml b/charts/openbot/values.yamlindex 21b6d281..37064ca6 100644--- a/charts/openbot/values.yaml+++ b/charts/openbot/values.yaml@@ -404,6 +404,16 @@ computers:# the process default, which is a handful and thirty minutes.maxBrowsers: ""browserIdleMs: ""+ # At what share of the pod's memory limit the computer starts closing browsers nobody is using,+ # least recently used first, down to ten points under it. Measured as the kubelet measures it (the+ # working set), before each launch and once a minute: it stops idle browsers filling the pod one+ # launch at a time, and a page that grows between two checks can still run the pod out of memory.+ # Empty means 85; 0 switches it off.+ memoryHighPercent: ""+ # How long a Bot's background tab (one it left behind, such as the page a sign-in popup opened from)+ # is kept untouched before it is closed. The page the Bot is on is never closed. Empty means ten+ # minutes; 0 keeps them.+ tabIdleMs: ""# Which Chromium a Bot's computer runs. `headed` is full Chromium drawing to the pod's own X server# (Xvfb, started by the computer), behind the same streamed screen: it is what sites see from adiff --git a/docker-compose.yml b/docker-compose.ymlindex 306dde45..889f3b87 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -270,6 +270,12 @@ services:# minutes; closing a browser costs only a relaunch, because the profile is on the volume below.COMPUTER_MAX_BROWSERS: ${COMPUTER_MAX_BROWSERS:-}COMPUTER_BROWSER_IDLE_MS: ${COMPUTER_BROWSER_IDLE_MS:-}+ # At what share of the container's memory it starts closing browsers nobody is using, least+ # recently used first, down to ten points under (default 85), and how long a Bot's background+ # tab may sit untouched (default ten minutes). A container with no --memory is measured against+ # the machine's memory. 0 switches either off.+ COMPUTER_MEMORY_HIGH_PERCENT: ${COMPUTER_MEMORY_HIGH_PERCENT:-}+ COMPUTER_TAB_IDLE_MS: ${COMPUTER_TAB_IDLE_MS:-}# Every Bot without a computer of its own reaches this one, so each gets its own folder of# files and its shell starts there. The supervisor's per-Bot computers do not need it.COMPUTER_SEPARATE_WORKSPACES: "on"diff --git a/docker/s6/s6-rc.d/computer/run b/docker/s6/s6-rc.d/computer/runindex 972d55e7..5ea54a4a 100755--- a/docker/s6/s6-rc.d/computer/run+++ b/docker/s6/s6-rc.d/computer/run@@ -12,6 +12,12 @@## `with-contenv` is not decoration. Without it s6 starts a service with none of the container's# environment, and the failure is a config error naming a variable that is plainly set.+#+# COMPUTER_MEMORY_HIGH_PERCENT stays on here, and here it counts everything in this container: the+# database and the API server as well as the browsers, because the cgroup the computer reads is the+# container's. That is deliberate. The kernel's out-of-memory kill counts all of it too and can take+# the database down, so closing a browser nobody is using, before memory gets there, is how the+# database is protected. Set it to 0 to leave memory to the kernel.set -euif [ "${EMBEDDED_COMPUTER:-on}" != "on" ]; thenexec /bin/true@@ -26,7 +32,7 @@ for name in \BUN_INSTALL PLAYWRIGHT_BROWSERS_PATH SPIFFE_ENDPOINT_SOCKET \COMPUTER_TOKEN COMPUTER_BOT_ID COMPUTER_BROWSER_MODE COMPUTER_MAX_BROWSERS \COMPUTER_BROWSER_IDLE_MS COMPUTER_SANDBOX COMPUTER_SHELL_ENV COMPUTER_STOPPED \- COMPUTER_SEPARATE_WORKSPACES \+ COMPUTER_SEPARATE_WORKSPACES COMPUTER_MEMORY_HIGH_PERCENT COMPUTER_TAB_IDLE_MS \ACTION_TIMEOUT_MS NAVIGATION_TIMEOUT_MS AGENT_COMPUTER_ALLOW_PRIVATE_HOSTSdoeval "isset=\${$name+x}"diff --git a/docs/configuration.md b/docs/configuration.mdindex 3cda4bb2..350cedd2 100644--- a/docs/configuration.md+++ b/docs/configuration.md@@ -651,6 +651,8 @@ is still the record.| `COMPUTER_MAX_BROWSERS` | How many Bots may hold a running browser at once. `8` by default; the least recently used is closed past it. A person's browser in a native window (OpenBot Desktop) counts toward it and is closed only when no Bot's browser is left to close; a person's cloud browser is ranked by use like a Bot's. || `COMPUTER_SEPARATE_WORKSPACES` | `on` gives each Bot its own folder of files on a computer many Bots share, with its shell starting there. On for Helm's `shared` computer and the docker-compose `agent-computer`; off elsewhere. Files saved before it was on stay at the root of the volume. Commands can still read other folders: use `computers.mode: sandbox` to keep people apart. || `COMPUTER_BROWSER_IDLE_MS` | How long an untouched browser is kept. 30 minutes by default; `0` keeps them resident. A person's cloud browser takes the same rule: the Browser page touches it every second while it is open, so it closes once nobody is on that page. A person's native window (OpenBot Desktop) is never closed for being idle. |+| `COMPUTER_MEMORY_HIGH_PERCENT` | At what share of its memory limit a computer starts closing browsers nobody is using, least recently used first and one at a time, until it is ten points under that share (75% at the default). `85` by default; `0` switches it off. Memory is the container's working set, as the kubelet reads it (usage less the inactive page cache), checked before every launch and every minute, one pass at a time however many launches arrive together. It is a check, not a guarantee: a page that grows between two checks can still run the container out of memory. A container with no limit is measured against the machine's memory; a computer outside a container (OpenBot Desktop's) closes nothing for memory. Never closed for memory: the browser being launched, the most recently used one, any used in the last minute, a person's own browser (cloud or native), and a Bot's whose wheel a person holds or has been asked to take. In the all-in-one image the share counts the database and the API server as well as the browsers, on purpose: the kernel's out-of-memory kill counts them too and can take the database down, and closing a browser first is how the database is protected. Each close logs `computer-browser-closed-for-memory` with the working set and the limit in bytes. |+| `COMPUTER_TAB_IDLE_MS` | How long a Bot's background tab may sit untouched before it is closed. 10 minutes by default; `0` keeps them. A Bot is always on its newest page, so an older tab (the page a sign-in popup opened from, a link opened in a new tab) is one it cannot go back to. Navigating, being acted on, or being in front counts as use. Never closed: the page the Bot is on, a person's own browser, and any tab of a Bot whose wheel a person holds. || `OPENBOT_RECORDER_DIR` | The built OpenBot Recorder (`extension/dist` after `bun run build`), absolute. Every person's browser the computer opens has it loaded and connected; without it, or pointed at anything that is not a built Recorder, the computer still serves Bots and refuses to open a person's browser, saying why. The computer image sets it to the Recorder it carries. || `COMPUTER_PEOPLE_ONLY` | `on` for a computer that runs people's own browsers and no Bot (OpenBot Desktop sets it on its host-side computer); it refuses every Bot. A person's browser runs only on a computer of its own: this one, or one started for it alone (`COMPUTER_BOT_ID`, as the supervisor and a Sandbox start each computer). The shared computer refuses it, and so does the server, whose Browser page says why, because a Bot's shell there could read the person's logins off the disk. || `COMPUTER_BROWSER_BACKEND` | `managed` by default (full bundled Chromium); `local-chrome` opts into installed Chrome with dedicated profiles and a loopback API. |diff --git a/scripts/check-new-values-keys.ts b/scripts/check-new-values-keys.tsindex d80f7624..51f60d51 100644--- a/scripts/check-new-values-keys.ts+++ b/scripts/check-new-values-keys.ts@@ -15,6 +15,9 @@** bun scripts/check-new-values-keys.ts <ci-values.yaml> [--since v0.0.4]*/+import { mkdtemp, rm } from "node:fs/promises";+import { tmpdir } from "node:os";+import { join } from "node:path";import { parse, parseAllDocuments } from "yaml";const [valuesFile, ...rest] = process.argv.slice(2);@@ -461,4 +464,79 @@ for (const { path, variable } of offSwitches) {}console.log(`${path}=0 stays zero`);}++/*+ * The computer's own settings, which the chart passes through only when somebody set them, so a+ * blank leaves the computer's own default in force. Two things have gone wrong with that shape.+ *+ * ZERO. Each of these switches something off at zero, and `with` reads zero as empty, so a+ * deployment that wrote `0` got nothing rendered and the default left on.+ *+ * A WHOLE NUMBER FROM A VALUES FILE. Helm reads values files through JSON, so `tabIdleMs: 1200000`+ * arrives as a float and renders as `1.2e+06`. The computer reads digits only and falls back to its+ * default without a word. `--set` parses an integer, which is why this needs a file of its own.+ *+ * Only where the chart runs the shared computer, which is the workload that carries them.+ */+const computerSettings: Array<{+ path: string;+ variable: string;+ whole: number;+}> = [+ {+ path: "computers.memoryHighPercent",+ variable: "COMPUTER_MEMORY_HIGH_PERCENT",+ whole: 90,+ },+ {+ path: "computers.tabIdleMs",+ variable: "COMPUTER_TAB_IDLE_MS",+ whole: 1_200_000,+ },+ // Older than this check: an hour rendered as `3.6e+06`, and `0` ("keep browsers resident") as nothing.+ {+ path: "computers.browserIdleMs",+ variable: "COMPUTER_BROWSER_IDLE_MS",+ whole: 3_600_000,+ },+];+if (!baseline.out.includes("name: COMPUTER_SEPARATE_WORKSPACES")) {+ console.log(+ `${valuesFile} runs no shared computer, so its settings are not checked here`,+ );+} else {+ const scratch = await mkdtemp(join(tmpdir(), "openbot-values-"));+ try {+ for (const { path, variable, whole } of computerSettings) {+ const zero = render(["--set", `${path}=0`]);+ const zeroGot = zero.ok ? renderedValue(zero.out, variable) : undefined;+ if (zeroGot !== "0") {+ console.error(+ `::error::Setting ${path}=0 rendered ${variable}=${zeroGot ?? "nothing"} rather than 0. A zero is an off switch, not an absent value.`,+ );+ bad += 1;+ } else {+ console.log(`${path}=0 stays zero`);+ }++ const leaf = path.slice(path.lastIndexOf(".") + 1);+ const file = join(scratch, `${leaf}.yaml`);+ await Bun.write(file, `computers:\n ${leaf}: ${whole}\n`);+ const fromFile = render(["--values", file]);+ const fileGot = fromFile.ok+ ? renderedValue(fromFile.out, variable)+ : undefined;+ if (fileGot !== String(whole)) {+ console.error(+ `::error::${leaf}: ${whole} in a values file rendered ${variable}=${fileGot ?? "nothing"}, which the computer does not read as ${whole}.`,+ );+ bad += 1;+ } else {+ console.log(`${path} from a values file renders as ${fileGot}`);+ }+ }+ } finally {+ await rm(scratch, { recursive: true, force: true });+ }+}process.exit(bad === 0 ? 0 : 1);