Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.
Keep a finished suspension for the idle window, not for a day (#254)
beardthelionc6ed7020615e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5d41dd37..43d359ac 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -50,6 +50,22 @@ to be able to say what it was doing.Nothing to configure. A Bot that acts during a takeover gets the refusal it already got for a click,and the trail records the attempt and the failure the same way.+### A computer that was suspended once suspends again++Scale-to-zero worked once per Bot. A computer suspended, resumed, used and then left alone again was+offered for suspension on every sweep after that and never suspended, and stayed awake until the next+day. Nothing reported it, because a sweep that offers work and suspends nothing looks exactly like a+fleet that is busy.++The queue keys a suspension on the Bot id and keeps the finished row so that a late offer of the same+key collides with it rather than running the work twice. Both are right. What was wrong is that the+finished row was kept for a day, which is the window the other half of the sweep needs: a suspension+that keeps failing is held back that long before anything tries it again. One number could not be+both, so there are now two, and a finished suspension is kept for the idle window instead. That is+the same clock the offer runs on, so a Bot cannot come back round as idle until its row has gone.++Nothing to configure, and the sweep already runs on a schedule. A deployment where each Bot has its+own computer stops paying for browsers that were used once.### A finished turn shows the page it opened, not the one open nowdiff --git a/server/scripts/cull-idle-computers.ts b/server/scripts/cull-idle-computers.tsindex 626e55ae..43183ac3 100644--- a/server/scripts/cull-idle-computers.ts+++ b/server/scripts/cull-idle-computers.ts@@ -71,6 +71,20 @@ try {const purged = await queue.purge({kind: CULL_KIND,olderThanMs: 24 * 60 * 60 * 1000,+ /*+ * A FINISHED SUSPENSION IS KEPT FOR THE IDLE WINDOW, NOT FOR A DAY.+ *+ * The key is the Bot id, so the row left behind by a suspension is what the next one collides+ * with: a computer resumed, used, and left alone again was offered every sweep and swallowed+ * every time, and stayed awake until the row aged out a day later. Nobody saw it, because a+ * sweep that offers work and suspends nothing looks exactly like a fleet that is busy.+ *+ * The idle window is the right length because it is the same clock the offer runs on: a Bot+ * cannot qualify as idle again until this long after its last action, by which point its row has+ * gone. A day is still right for the other half, where the window is the backoff before a+ * suspension that keeps failing is tried again.+ */+ finishedOlderThanMs: config.computer.idleAfterMs,});/** And the screenshots, which had a reaper and nothing calling it.diff --git a/server/src/work/queue.ts b/server/src/work/queue.tsindex ce30b41b..30aa7ba5 100644--- a/server/src/work/queue.ts+++ b/server/src/work/queue.ts@@ -87,10 +87,18 @@ export type WorkQueue = {** Both kinds of done: finished, and given up on. An item at its attempt cap is not finished and was* reaped by nothing, so its key stayed occupied for ever and the work could never be offered again.+ *+ * TWO KINDS OF DONE, TWO WINDOWS. They are the same length only by coincidence. A finished row is+ * kept so a late offer of the same key collides with it, which needs to outlast a sweep; a row that+ * gave up is kept because the window is also the backoff before anything tries again, which wants+ * to be long. Kept as one number, a queue whose work repeats has to choose which of those to be+ * wrong about. `finishedOlderThanMs` defaults to `olderThanMs`, so a caller that has only one+ * answer keeps the behaviour it had.*/purge: (input: {kind: string;olderThanMs: number;+ finishedOlderThanMs?: number;maxAttempts?: number;}) => Promise<number>;};@@ -250,15 +258,21 @@ export function createWorkQueue(database: Database): WorkQueue {return Boolean(released);},- async purge({ kind, olderThanMs, maxAttempts = DEFAULT_MAX_ATTEMPTS }) {+ async purge({+ kind,+ olderThanMs,+ finishedOlderThanMs = olderThanMs,+ maxAttempts = DEFAULT_MAX_ATTEMPTS,+ }) {const cutoff = fromNow(-olderThanMs);+ const finishedCutoff = fromNow(-finishedOlderThanMs);const gone = await database.delete(workItems).where(and(eq(workItems.kind, kind),or(- lt(workItems.finishedAt, cutoff),+ lt(workItems.finishedAt, finishedCutoff),/** AND THE ONES THAT GAVE UP, which is the half this forgot.*diff --git a/server/tests/computer-culler.integration.test.ts b/server/tests/computer-culler.integration.test.tsindex 6f8555e0..22092d77 100644--- a/server/tests/computer-culler.integration.test.ts+++ b/server/tests/computer-culler.integration.test.ts@@ -219,4 +219,57 @@ describe("suspending computers nobody is using", () => {// Each exactly once, which is the point of claiming rather than sweeping.expect(new Set(stopped).size).toBe(stopped.length);});++ /*+ * The second idle window, which is the one that pays for scale-to-zero.+ *+ * A computer is suspended once and then resumed, used, and left alone again. Every sweep after the+ * first was offering work that the finished row silently swallowed, so the Bot stayed awake until+ * that row aged out a day later. The queue is right to keep the row: it is what stops the same key+ * running twice. It is the retention window for a finished suspension that has to be the idle+ * window rather than a day, which is what the sweep below passes.+ */+ test("a computer used again after it was suspended is suspended again", async () => {+ const botId = botOf("recycled");+ const { provider, stopped } = providerWith([+ { botId, status: "running", url: "http://c" },+ ]);+ const at = (iso: string) => () => new Date(iso);+ const sweep = async (whenIso: string) => {+ const options = {+ database,+ queue,+ provider,+ idleAfterMs,+ owner: "replica-1",+ now: at(whenIso),+ };+ await offerIdleComputers(options);+ const report = await suspendClaimedComputers(options);+ /*+ * What the CronJob does at the end of every sweep. Zero here rather than the idle window+ * because these rows are finished seconds ago in real time and `purge` reads the database's+ * clock, not this test's: the window being separate from the give-up one is the property+ * under test, not its length.+ */+ await queue.purge({+ kind: CULL_KIND,+ olderThanMs: 24 * 60 * 60_000,+ finishedOlderThanMs: 0,+ });+ return report;+ };++ await database.insert(auditEvents).values(ran(botId, minutesAgo(60)));+ expect((await sweep("2026-08-24T12:00:00Z")).suspended).toEqual([botId]);++ // Somebody comes back at 13:00, and it is quiet again by 14:00.+ await database+ .insert(auditEvents)+ .values(ran(botId, new Date("2026-08-24T13:00:00Z")));+ const second = await sweep("2026-08-24T14:00:00Z");++ expect(second.suspended).toEqual([botId]);+ expect(stopped).toEqual([botId, botId]);+ });});diff --git a/server/tests/cull-sweep-wiring.test.ts b/server/tests/cull-sweep-wiring.test.tsnew file mode 100644index 00000000..5c3d4164--- /dev/null+++ b/server/tests/cull-sweep-wiring.test.ts@@ -0,0 +1,40 @@+import { expect, test } from "bun:test";+import { readFileSync } from "node:fs";+import { join } from "node:path";++/**+ * What the sweep passes, asserted against the script's own source.+ *+ * The script cannot be imported to be tested: it opens a database and a computer provider at the top+ * level and runs a sweep as a side effect of loading. So the retention window it hands `purge` is the+ * one line in this fix that nothing executes, and every test around it passes the window itself,+ * which means they would all still be green with this line deleted.+ *+ * That is the failure this whole change is about, one level up: a value the caller never names+ * reaches nothing, and the sweep goes on looking like it worked. Read as text, the way+ * `tests/compose.test.ts` pins the variables Compose has to name, because the alternative is a fix+ * whose only load-bearing line is the untested one.+ *+ * WHAT THIS CANNOT SEE. It proves the argument is written and where its value comes from, not that+ * the sweep behaves. `computer-culler.integration.test.ts` owns the behaviour and runs the real+ * queue against a real database; this is only the wire between the two.+ */+const script = readFileSync(+ join(import.meta.dir, "..", "scripts", "cull-idle-computers.ts"),+ "utf8",+);++test("the cull sweep keeps a finished suspension for the configured idle window", () => {+ expect(script).toContain("finishedOlderThanMs: config.computer.idleAfterMs");+});++/**+ * And that the other half still gets its day.+ *+ * The two windows exist because they are different questions. A sweep that passed the idle window for+ * both would turn a suspension that keeps failing into one retried every few minutes, which is the+ * regression this fix could most easily cause and the one nothing else would report.+ */+test("and gives a suspension that gave up the longer window before it is tried again", () => {+ expect(script).toContain("olderThanMs: 24 * 60 * 60 * 1000");+});diff --git a/server/tests/work-queue.integration.test.ts b/server/tests/work-queue.integration.test.tsindex a7397e6a..14b7d847 100644--- a/server/tests/work-queue.integration.test.ts+++ b/server/tests/work-queue.integration.test.ts@@ -323,4 +323,53 @@ describe("claiming durable work", () => {expect(row?.attempts).toBe(3);expect(row?.why).toBe("the cluster said no");});++ /*+ * The two kinds of done, on their own clocks.+ *+ * They were one number, which a queue whose work repeats cannot afford: a finished row only has to+ * outlast a sweep, and a row that gave up is kept because that window is also the backoff before+ * anything tries the work again. Sharing it meant the culler either wedged its own key for a day+ * or retried a broken suspension every few minutes.+ */+ test("a finished row and one that gave up are swept on separate windows", async () => {+ await queue.offer({ kind, key: "done" });+ await queue.claim({ kind, owner: "replica-1", leaseMs: 30_000 });+ await queue.finish({ kind, key: "done", owner: "replica-1" });++ await queue.offer({ kind, key: "gave-up" });+ for (let attempt = 0; attempt < 3; attempt += 1) {+ await queue.claim({+ kind,+ owner: "replica-1",+ leaseMs: 30_000,+ maxAttempts: 3,+ });+ await queue.release({+ kind,+ key: "gave-up",+ owner: "replica-1",+ delayMs: 0,+ reason: "the cluster said no",+ });+ }++ // The finished one goes on its own short window; the one that gave up keeps its long one.+ expect(+ await queue.purge({+ kind,+ olderThanMs: 60_000,+ finishedOlderThanMs: 0,+ maxAttempts: 3,+ }),+ ).toBe(1);+ const [left] = await database+ .select({ key: workItems.key })+ .from(workItems)+ .where(eq(workItems.kind, kind));+ expect(left?.key).toBe("gave-up");++ // And omitting it leaves both halves on the one window, which is what every other caller does.+ expect(await queue.purge({ kind, olderThanMs: 0, maxAttempts: 3 })).toBe(1);+ });});