Skip to content

Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.

Keep a finished suspension for the idle window, not for a day (#254)

beardthelionc6ed7020615e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5d41dd37..43d359ac 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -50,6 +50,22 @@ to be able to say what it was doing.
Nothing to configure. A Bot that acts during a takeover gets the refusal it already got for a click,
and the trail records the attempt and the failure the same way.
+### A computer that was suspended once suspends again
+
+Scale-to-zero worked once per Bot. A computer suspended, resumed, used and then left alone again was
+offered for suspension on every sweep after that and never suspended, and stayed awake until the next
+day. Nothing reported it, because a sweep that offers work and suspends nothing looks exactly like a
+fleet that is busy.
+
+The queue keys a suspension on the Bot id and keeps the finished row so that a late offer of the same
+key collides with it rather than running the work twice. Both are right. What was wrong is that the
+finished row was kept for a day, which is the window the other half of the sweep needs: a suspension
+that keeps failing is held back that long before anything tries it again. One number could not be
+both, so there are now two, and a finished suspension is kept for the idle window instead. That is
+the same clock the offer runs on, so a Bot cannot come back round as idle until its row has gone.
+
+Nothing to configure, and the sweep already runs on a schedule. A deployment where each Bot has its
+own computer stops paying for browsers that were used once.
### A finished turn shows the page it opened, not the one open now
diff --git a/server/scripts/cull-idle-computers.ts b/server/scripts/cull-idle-computers.ts
index 626e55ae..43183ac3 100644
--- a/server/scripts/cull-idle-computers.ts
+++ b/server/scripts/cull-idle-computers.ts
@@ -71,6 +71,20 @@ try {
const purged = await queue.purge({
kind: CULL_KIND,
olderThanMs: 24 * 60 * 60 * 1000,
+ /*
+ * A FINISHED SUSPENSION IS KEPT FOR THE IDLE WINDOW, NOT FOR A DAY.
+ *
+ * The key is the Bot id, so the row left behind by a suspension is what the next one collides
+ * with: a computer resumed, used, and left alone again was offered every sweep and swallowed
+ * every time, and stayed awake until the row aged out a day later. Nobody saw it, because a
+ * sweep that offers work and suspends nothing looks exactly like a fleet that is busy.
+ *
+ * The idle window is the right length because it is the same clock the offer runs on: a Bot
+ * cannot qualify as idle again until this long after its last action, by which point its row has
+ * gone. A day is still right for the other half, where the window is the backoff before a
+ * suspension that keeps failing is tried again.
+ */
+ finishedOlderThanMs: config.computer.idleAfterMs,
});
/*
* And the screenshots, which had a reaper and nothing calling it.
diff --git a/server/src/work/queue.ts b/server/src/work/queue.ts
index ce30b41b..30aa7ba5 100644
--- a/server/src/work/queue.ts
+++ b/server/src/work/queue.ts
@@ -87,10 +87,18 @@ export type WorkQueue = {
*
* Both kinds of done: finished, and given up on. An item at its attempt cap is not finished and was
* reaped by nothing, so its key stayed occupied for ever and the work could never be offered again.
+ *
+ * TWO KINDS OF DONE, TWO WINDOWS. They are the same length only by coincidence. A finished row is
+ * kept so a late offer of the same key collides with it, which needs to outlast a sweep; a row that
+ * gave up is kept because the window is also the backoff before anything tries again, which wants
+ * to be long. Kept as one number, a queue whose work repeats has to choose which of those to be
+ * wrong about. `finishedOlderThanMs` defaults to `olderThanMs`, so a caller that has only one
+ * answer keeps the behaviour it had.
*/
purge: (input: {
kind: string;
olderThanMs: number;
+ finishedOlderThanMs?: number;
maxAttempts?: number;
}) => Promise<number>;
};
@@ -250,15 +258,21 @@ export function createWorkQueue(database: Database): WorkQueue {
return Boolean(released);
},
- async purge({ kind, olderThanMs, maxAttempts = DEFAULT_MAX_ATTEMPTS }) {
+ async purge({
+ kind,
+ olderThanMs,
+ finishedOlderThanMs = olderThanMs,
+ maxAttempts = DEFAULT_MAX_ATTEMPTS,
+ }) {
const cutoff = fromNow(-olderThanMs);
+ const finishedCutoff = fromNow(-finishedOlderThanMs);
const gone = await database
.delete(workItems)
.where(
and(
eq(workItems.kind, kind),
or(
- lt(workItems.finishedAt, cutoff),
+ lt(workItems.finishedAt, finishedCutoff),
/*
* AND THE ONES THAT GAVE UP, which is the half this forgot.
*
diff --git a/server/tests/computer-culler.integration.test.ts b/server/tests/computer-culler.integration.test.ts
index 6f8555e0..22092d77 100644
--- a/server/tests/computer-culler.integration.test.ts
+++ b/server/tests/computer-culler.integration.test.ts
@@ -219,4 +219,57 @@ describe("suspending computers nobody is using", () => {
// Each exactly once, which is the point of claiming rather than sweeping.
expect(new Set(stopped).size).toBe(stopped.length);
});
+
+ /*
+ * The second idle window, which is the one that pays for scale-to-zero.
+ *
+ * A computer is suspended once and then resumed, used, and left alone again. Every sweep after the
+ * first was offering work that the finished row silently swallowed, so the Bot stayed awake until
+ * that row aged out a day later. The queue is right to keep the row: it is what stops the same key
+ * running twice. It is the retention window for a finished suspension that has to be the idle
+ * window rather than a day, which is what the sweep below passes.
+ */
+ test("a computer used again after it was suspended is suspended again", async () => {
+ const botId = botOf("recycled");
+ const { provider, stopped } = providerWith([
+ { botId, status: "running", url: "http://c" },
+ ]);
+ const at = (iso: string) => () => new Date(iso);
+ const sweep = async (whenIso: string) => {
+ const options = {
+ database,
+ queue,
+ provider,
+ idleAfterMs,
+ owner: "replica-1",
+ now: at(whenIso),
+ };
+ await offerIdleComputers(options);
+ const report = await suspendClaimedComputers(options);
+ /*
+ * What the CronJob does at the end of every sweep. Zero here rather than the idle window
+ * because these rows are finished seconds ago in real time and `purge` reads the database's
+ * clock, not this test's: the window being separate from the give-up one is the property
+ * under test, not its length.
+ */
+ await queue.purge({
+ kind: CULL_KIND,
+ olderThanMs: 24 * 60 * 60_000,
+ finishedOlderThanMs: 0,
+ });
+ return report;
+ };
+
+ await database.insert(auditEvents).values(ran(botId, minutesAgo(60)));
+ expect((await sweep("2026-08-24T12:00:00Z")).suspended).toEqual([botId]);
+
+ // Somebody comes back at 13:00, and it is quiet again by 14:00.
+ await database
+ .insert(auditEvents)
+ .values(ran(botId, new Date("2026-08-24T13:00:00Z")));
+ const second = await sweep("2026-08-24T14:00:00Z");
+
+ expect(second.suspended).toEqual([botId]);
+ expect(stopped).toEqual([botId, botId]);
+ });
});
diff --git a/server/tests/cull-sweep-wiring.test.ts b/server/tests/cull-sweep-wiring.test.ts
new file mode 100644
index 00000000..5c3d4164
--- /dev/null
+++ b/server/tests/cull-sweep-wiring.test.ts
@@ -0,0 +1,40 @@
+import { expect, test } from "bun:test";
+import { readFileSync } from "node:fs";
+import { join } from "node:path";
+
+/**
+ * What the sweep passes, asserted against the script's own source.
+ *
+ * The script cannot be imported to be tested: it opens a database and a computer provider at the top
+ * level and runs a sweep as a side effect of loading. So the retention window it hands `purge` is the
+ * one line in this fix that nothing executes, and every test around it passes the window itself,
+ * which means they would all still be green with this line deleted.
+ *
+ * That is the failure this whole change is about, one level up: a value the caller never names
+ * reaches nothing, and the sweep goes on looking like it worked. Read as text, the way
+ * `tests/compose.test.ts` pins the variables Compose has to name, because the alternative is a fix
+ * whose only load-bearing line is the untested one.
+ *
+ * WHAT THIS CANNOT SEE. It proves the argument is written and where its value comes from, not that
+ * the sweep behaves. `computer-culler.integration.test.ts` owns the behaviour and runs the real
+ * queue against a real database; this is only the wire between the two.
+ */
+const script = readFileSync(
+ join(import.meta.dir, "..", "scripts", "cull-idle-computers.ts"),
+ "utf8",
+);
+
+test("the cull sweep keeps a finished suspension for the configured idle window", () => {
+ expect(script).toContain("finishedOlderThanMs: config.computer.idleAfterMs");
+});
+
+/**
+ * And that the other half still gets its day.
+ *
+ * The two windows exist because they are different questions. A sweep that passed the idle window for
+ * both would turn a suspension that keeps failing into one retried every few minutes, which is the
+ * regression this fix could most easily cause and the one nothing else would report.
+ */
+test("and gives a suspension that gave up the longer window before it is tried again", () => {
+ expect(script).toContain("olderThanMs: 24 * 60 * 60 * 1000");
+});
diff --git a/server/tests/work-queue.integration.test.ts b/server/tests/work-queue.integration.test.ts
index a7397e6a..14b7d847 100644
--- a/server/tests/work-queue.integration.test.ts
+++ b/server/tests/work-queue.integration.test.ts
@@ -323,4 +323,53 @@ describe("claiming durable work", () => {
expect(row?.attempts).toBe(3);
expect(row?.why).toBe("the cluster said no");
});
+
+ /*
+ * The two kinds of done, on their own clocks.
+ *
+ * They were one number, which a queue whose work repeats cannot afford: a finished row only has to
+ * outlast a sweep, and a row that gave up is kept because that window is also the backoff before
+ * anything tries the work again. Sharing it meant the culler either wedged its own key for a day
+ * or retried a broken suspension every few minutes.
+ */
+ test("a finished row and one that gave up are swept on separate windows", async () => {
+ await queue.offer({ kind, key: "done" });
+ await queue.claim({ kind, owner: "replica-1", leaseMs: 30_000 });
+ await queue.finish({ kind, key: "done", owner: "replica-1" });
+
+ await queue.offer({ kind, key: "gave-up" });
+ for (let attempt = 0; attempt < 3; attempt += 1) {
+ await queue.claim({
+ kind,
+ owner: "replica-1",
+ leaseMs: 30_000,
+ maxAttempts: 3,
+ });
+ await queue.release({
+ kind,
+ key: "gave-up",
+ owner: "replica-1",
+ delayMs: 0,
+ reason: "the cluster said no",
+ });
+ }
+
+ // The finished one goes on its own short window; the one that gave up keeps its long one.
+ expect(
+ await queue.purge({
+ kind,
+ olderThanMs: 60_000,
+ finishedOlderThanMs: 0,
+ maxAttempts: 3,
+ }),
+ ).toBe(1);
+ const [left] = await database
+ .select({ key: workItems.key })
+ .from(workItems)
+ .where(eq(workItems.kind, kind));
+ expect(left?.key).toBe("gave-up");
+
+ // And omitting it leaves both halves on the one window, which is what every other caller does.
+ expect(await queue.purge({ kind, olderThanMs: 0, maxAttempts: 3 })).toBe(1);
+ });
});