Skip to content

armgit / qm

Public Imported from GitHub
0

Multiplayer agent harness for work.

Make swarm message, notification, spawn and lifetime budgets opt-in (#2406)

Joshua Franceaac880b7ecf3
diff --git a/docs/swarms.md b/docs/swarms.md
index 649c98ca6..c28619c4e 100644
--- a/docs/swarms.md
+++ b/docs/swarms.md
@@ -66,9 +66,11 @@ unavailable model fails closed at dispatch.
Optional `settings` overrides the defaults below at initial spawn. The backend
can configure defaults with `SWARM_DEFAULTS`, a JSON object such as
`{"agents":16,"turnMs":900000}`. Values must be positive safe integers within the
-safety bounds; unknown settings are rejected. Resolved settings and the chosen
-`backend` are stored once. Identical initial requests can be retried; subsequent
-spawns inherit the settings and cannot reconfigure the swarm.
+safety bounds; unknown settings are rejected. Resolved settings and the chosen `backend` are stored once. Identical
+initial requests can be retried. Later spawns by the root session may pass
+`settings` again to change any setting except `turnMs` (for example, to raise
+`agents`); workers inherit the settings and cannot reconfigure the swarm. The
+backend cannot change after creation.
Omit `count` to spawn one; `contexts` also determines the count when provided. A
single `context` supplies the same initial JSON to every worker. Context may be any
@@ -108,7 +110,7 @@ never retires the existing forum.
### Identity, peers, and character
-`GET /v1/swarm` returns `id`, `self`, `peers`, `backend`, `settings`, and `expiresAt`. Peer
+`GET /v1/swarm` returns `id`, `self`, `peers`, `backend`, `settings`, and `expiresAt` when a work window is set. Peer
records keep trusted IDs, ancestry, session links, storage selection, and lifecycle
state separate from arbitrary `context` data. Only this swarm is listed, not every
session with the same owner, and never all private sessions in the organization.
@@ -154,8 +156,8 @@ delivery target, and the delivery layer also refuses swarm result delivery.
Set `notify:false` for a durable message without a wakeup. Sending to oneself does
not wake oneself. There are no automatic reply-to-reply notifications: agents
-explicitly decide whether a response warrants another turn. Every generated turn
-consumes the same finite notification budget.
+explicitly decide whether a response warrants another turn. When `notifications`
+is set, every generated turn consumes that budget.
### Ask, read, and tail
@@ -234,25 +236,30 @@ refreshes every few seconds only while workers are active and the tab is visible
| ------------------------------------------------ | -------------------- |
| `agents` (root and failed reservations included) | 32 |
| `depth` below root | 4 |
-| `spawnRequests` | 32 |
-| `messages` including initial work | 128 |
-| `notifications` including initial work | 256 |
| `contextBytes` / `textBytes` | 8,192 each |
-| `lifetimeMs` from first spawn | 3,600,000 |
+| `spawnRequests` | unset (no limit) |
+| `messages` including initial work | unset (no limit) |
+| `notifications` including initial work | unset (no limit) |
+| `lifetimeMs` from first spawn | unset (no limit) |
| `turnMs` per notification | 600,000 (10 minutes) |
| `waitMs` per read | 10,000 |
The root and at least one worker require `agents >= 2`. All other settings accept
any positive integer; `turnMs` is limited only by the 2,147,483,647 ms timer ceiling. Read requests can still use `waitMs:0`.
-Per-swarm budgets are reserved atomically with the messages and members consuming
-them. Both the orchestrator and independent worker cancellation honor the stored
-turn deadline; existing organization-wide limits can shorten it. Internal reconciliation limits, the 32-message read page, and
-bounded run retry counts are not swarm configuration.
-
-After expiration, history remains readable, but new work, notifications, and
-character changes are rejected. Human-initiated ordinary session turns remain
-ordinary QM actions. Workers and their durable results remain visible; computers
-can be retired through the existing sandbox inventory. There is no automatic
+`spawnRequests`, `messages`, `notifications`, and `lifetimeMs` are off unless a
+spawn or `SWARM_DEFAULTS` sets them. When set, they are enforced as budgets that are
+reserved atomically with the members and messages that consume them. Both the
+orchestrator and independent worker cancellation honor the stored turn deadline;
+existing organization-wide limits can shorten it. Internal reconciliation limits,
+the 32-message read page, and bounded run retry counts are not swarm
+configuration.
+
+When `lifetimeMs` is set and the window passes, history remains readable, but new
+work, notifications, and character changes are rejected until the root extends
+`lifetimeMs` on a later spawn. Swarms without `lifetimeMs` never expire. Human-initiated
+ordinary session turns remain ordinary QM actions.
+Workers and their durable results remain visible; computers can be retired
+through the existing sandbox inventory. There is no automatic
filesystem export on retirement: publish useful outputs through existing Files
or your version-control workflow first.
diff --git a/plugins/web-ui/src/swarm-strip.ts b/plugins/web-ui/src/swarm-strip.ts
index 31d7c8a68..152ab3fd7 100644
--- a/plugins/web-ui/src/swarm-strip.ts
+++ b/plugins/web-ui/src/swarm-strip.ts
@@ -18,7 +18,7 @@ interface SwarmView {
self: SwarmPeer;
peers: SwarmPeer[];
board?: { sandboxId: string };
- expiresAt: number;
+ expiresAt?: number;
}
type Control = "active" | "paused" | "stopped";
diff --git a/src/swarms/swarm-service.ts b/src/swarms/swarm-service.ts
index 91b108d47..ff32685e3 100644
--- a/src/swarms/swarm-service.ts
+++ b/src/swarms/swarm-service.ts
@@ -18,6 +18,7 @@ import { errMessage, swallow } from "../util/errors.ts";
import {
assertSwarmOpen,
SWARM_LIMITS,
+ swarmExpired,
type Swarm,
type SwarmMember,
type SwarmMessage,
@@ -71,7 +72,7 @@ export interface SwarmService {
board?: Swarm["board"];
backend: Swarm["backend"];
settings: SwarmSettings;
- expiresAt: number;
+ expiresAt?: number;
}>;
context(caller: SwarmCaller, context: unknown): Promise<SwarmMember>;
spawn(caller: SwarmCaller, input: SpawnInput): Promise<SwarmMember[]>;
@@ -331,7 +332,7 @@ export function createSwarmService(deps: {
settings,
backend: provider.name,
createdAt,
- expiresAt: createdAt + settings.lifetimeMs,
+ ...(settings.lifetimeMs === undefined ? {} : { expiresAt: createdAt + settings.lifetimeMs }),
...(forumSandboxId ? {} : { board: { sandboxId: randomUUID() } }),
members: [
{
@@ -372,7 +373,7 @@ export function createSwarmService(deps: {
if (!recipient || recipient.state === "reserved") continue;
const control = controlState(swarm, recipient.id);
if (control === "paused") continue;
- if (recipient.state === "failed" || control === "stopped" || Date.now() >= swarm.expiresAt) {
+ if (recipient.state === "failed" || control === "stopped" || swarmExpired(swarm)) {
await step(() =>
store.update(swarm.id, (current) => {
current.messages.find((item) => item.id === message.id)!.notifications[recipientId] = { state: "failed" };
@@ -483,12 +484,7 @@ export function createSwarmService(deps: {
const failed = current.members.find((peer) => peer.id === member.id)!;
if (failed.state !== "reserved") return;
failed.error = errMessage(error).slice(0, 500);
- if (
- provisioningTimedOut ||
- Date.now() >= deadline ||
- failed.attempts >= 3 ||
- Date.now() >= current.expiresAt
- ) {
+ if (provisioningTimedOut || Date.now() >= deadline || failed.attempts >= 3 || swarmExpired(current)) {
failed.state = "failed";
failed.cleanupPending = true;
}
@@ -579,7 +575,7 @@ export function createSwarmService(deps: {
...(swarm.board ? { board: swarm.board } : {}),
backend: swarm.backend,
settings: swarm.settings,
- expiresAt: swarm.expiresAt,
+ ...(swarm.expiresAt === undefined ? {} : { expiresAt: swarm.expiresAt }),
};
},
async context(caller, context) {
@@ -601,9 +597,10 @@ export function createSwarmService(deps: {
if (previous.signature !== fingerprint) throw new Error("requestId reused with different content");
return existing!.members.filter((member) => previous.memberIds.includes(member.id)).map(view);
}
- if (existing && (input.settings !== undefined || input.backend !== undefined))
- throw new Error("settings are only allowed on initial swarm creation");
- const settings = existing?.settings ?? resolveSwarmSettings(input.settings, defaults);
+ if (existing && input.backend !== undefined) throw new Error("backend is only allowed on initial swarm creation");
+ if (existing && input.settings !== undefined && auth.memberId !== existing.id)
+ throw new Error("only the root session can change swarm settings");
+ const settings = resolveSwarmSettings(input.settings, existing?.settings ?? defaults);
boundedText(input.text, settings.textBytes, "text");
const requestedRuntime = spawnRuntime(input);
const count = input.count ?? input.contexts?.length ?? 1;
@@ -613,7 +610,8 @@ export function createSwarmService(deps: {
const contexts = Array.from({ length: count }, (_value, index) =>
jsonContext(input.contexts ? input.contexts[index] : defaultContext, settings.contextBytes),
);
- if (!existing && count > settings.notifications) throw new Error("swarm notification budget exhausted");
+ if (!existing && count > (settings.notifications ?? Infinity))
+ throw new Error("swarm notification budget exhausted");
if (input.forumSandboxId) {
const forum = await deps.sandboxes.access(auth.actorId, input.forumSandboxId);
const session = await sessions.get(auth.sessionId);
@@ -625,20 +623,27 @@ export function createSwarmService(deps: {
if (previous.signature !== fingerprint) throw new Error("requestId reused with different content");
return;
}
- if (
- canonicalJson(settings) !== canonicalJson(swarm.settings) ||
- (input.backend !== undefined && input.backend !== swarm.backend)
- )
- throw new Error("conflicting initial swarm settings");
+ if (!existing) {
+ if (
+ canonicalJson(settings) !== canonicalJson(swarm.settings) ||
+ (input.backend !== undefined && input.backend !== swarm.backend)
+ )
+ throw new Error("conflicting initial swarm settings");
+ } else if (input.settings !== undefined) {
+ const next = resolveSwarmSettings(input.settings, swarm.settings);
+ if (next.turnMs !== swarm.settings.turnMs) throw new Error("turnMs cannot change after swarm creation");
+ swarm.settings = next;
+ if (next.lifetimeMs !== undefined) swarm.expiresAt = swarm.createdAt + next.lifetimeMs;
+ }
assertSwarmOpen(swarm);
const parent = swarm.members.find((member) => member.id === auth.memberId)!;
if (parent.depth >= swarm.settings.depth) throw new Error("swarm depth budget exhausted");
if (swarm.members.length + count > swarm.settings.agents) throw new Error("swarm agent budget exhausted");
- if (Object.keys(swarm.spawnRequests).length >= swarm.settings.spawnRequests)
+ if (Object.keys(swarm.spawnRequests).length >= (swarm.settings.spawnRequests ?? Infinity))
throw new Error("swarm spawn budget exhausted");
if (
- swarm.messages.length >= swarm.settings.messages ||
- swarm.notificationCount + count > swarm.settings.notifications
+ swarm.messages.length >= (swarm.settings.messages ?? Infinity) ||
+ swarm.notificationCount + count > (swarm.settings.notifications ?? Infinity)
)
throw new Error("swarm work budget exhausted");
const runtime = inheritRuntime(parent.runtime, requestedRuntime);
@@ -720,11 +725,12 @@ export function createSwarmService(deps: {
return;
}
assertSwarmOpen(swarm);
- if (swarm.messages.length >= swarm.settings.messages) throw new Error("swarm message budget exhausted");
+ if (swarm.messages.length >= (swarm.settings.messages ?? Infinity))
+ throw new Error("swarm message budget exhausted");
if (input.replyTo && !swarm.messages.some((message) => message.id === input.replyTo))
throw new Error("reply target is not in this swarm");
const recipients = input.notify === false ? [] : audience.filter((peer) => peer !== auth.memberId);
- if (swarm.notificationCount + recipients.length > swarm.settings.notifications)
+ if (swarm.notificationCount + recipients.length > (swarm.settings.notifications ?? Infinity))
throw new Error("swarm notification budget exhausted");
swarm.notificationCount += recipients.length;
swarm.messages.push({
diff --git a/src/swarms/swarm-settings.ts b/src/swarms/swarm-settings.ts
index 5d387924b..807bd2722 100644
--- a/src/swarms/swarm-settings.ts
+++ b/src/swarms/swarm-settings.ts
@@ -1,29 +1,29 @@
export const SWARM_DEFAULTS = {
agents: 32,
depth: 4,
- messages: 128,
- notifications: 256,
- spawnRequests: 32,
contextBytes: 8_192,
textBytes: 8_192,
waitMs: 10_000,
turnMs: 600_000,
- lifetimeMs: 3_600_000,
};
-export type SwarmSettings = typeof SWARM_DEFAULTS;
+const SWARM_CAPS = ["messages", "notifications", "spawnRequests", "lifetimeMs"] as const;
+
+export type SwarmSettings = typeof SWARM_DEFAULTS & Partial<Record<(typeof SWARM_CAPS)[number], number>>;
const MAX_TIMER_MS = 2_147_483_647;
export function resolveSwarmSettings(input: unknown = {}, defaults: SwarmSettings = SWARM_DEFAULTS): SwarmSettings {
if (!input || typeof input !== "object" || Array.isArray(input)) throw new Error("invalid swarm settings");
for (const key of Object.keys(input)) {
- if (!Object.hasOwn(SWARM_DEFAULTS, key)) throw new Error(`unknown swarm setting: ${key}`);
+ if (!Object.hasOwn(SWARM_DEFAULTS, key) && !SWARM_CAPS.includes(key as never))
+ throw new Error(`unknown swarm setting: ${key}`);
}
- const settings = { ...defaults, ...input };
- for (const key of Object.keys(SWARM_DEFAULTS) as Array<keyof SwarmSettings>) {
+ const settings: SwarmSettings = { ...defaults, ...input };
+ for (const key of [...Object.keys(SWARM_DEFAULTS), ...SWARM_CAPS] as Array<keyof SwarmSettings>) {
const value = settings[key];
- if (!Number.isSafeInteger(value) || value < 1 || (key === "turnMs" && value > MAX_TIMER_MS))
+ if (value === undefined && SWARM_CAPS.includes(key as never)) continue;
+ if (!Number.isSafeInteger(value) || value! < 1 || (key === "turnMs" && value! > MAX_TIMER_MS))
throw new Error(`invalid swarm setting: ${key}`);
}
if (settings.agents < 2) throw new Error("swarm settings must allow the root and a worker");
diff --git a/src/swarms/swarm-store.ts b/src/swarms/swarm-store.ts
index 3585e7e97..323bdfd94 100644
--- a/src/swarms/swarm-store.ts
+++ b/src/swarms/swarm-store.ts
@@ -54,7 +54,7 @@ export interface Swarm {
ownerId: string;
participants: string[];
createdAt: number;
- expiresAt: number;
+ expiresAt?: number;
template: SwarmTemplate;
settings: SwarmSettings;
backend: SandboxBackendName;
@@ -67,8 +67,12 @@ export interface Swarm {
pending: boolean;
}
+export function swarmExpired(swarm: Swarm): boolean {
+ return swarm.expiresAt !== undefined && Date.now() >= swarm.expiresAt;
+}
+
export function assertSwarmOpen(swarm: Swarm): void {
- if (Date.now() >= swarm.expiresAt) throw new NonRetryableTurnError("swarm work window expired");
+ if (swarmExpired(swarm)) throw new NonRetryableTurnError("swarm work window expired");
}
export interface SwarmStore {
diff --git a/test/swarm-settings.test.ts b/test/swarm-settings.test.ts
index 8fddbaf8a..478f8827b 100644
--- a/test/swarm-settings.test.ts
+++ b/test/swarm-settings.test.ts
@@ -25,6 +25,20 @@ test("backend defaults validate every setting and preserve unspecified defaults"
assert.deepEqual(resolveSwarmSettings(large), { ...SWARM_DEFAULTS, ...large });
});
+test("budget settings are optional, off by default, and honored from SWARM_DEFAULTS", () => {
+ const caps = { messages: 128, notifications: 256, spawnRequests: 32, lifetimeMs: 28_800_000 };
+ assert.deepEqual(resolveSwarmSettings({}), SWARM_DEFAULTS);
+ for (const key of Object.keys(caps)) {
+ assert.equal(key in SWARM_DEFAULTS, false);
+ for (const value of [0, -1, 1.5, "1", null])
+ assert.throws(() => resolveSwarmSettings({ [key]: value }), new RegExp(key));
+ }
+ assert.deepEqual(resolveSwarmSettings(caps), { ...SWARM_DEFAULTS, ...caps });
+ const swarmDefaults = loadConfig({ SWARM_DEFAULTS: JSON.stringify(caps) }).swarmDefaults;
+ assert.deepEqual(swarmDefaults, { ...SWARM_DEFAULTS, ...caps });
+ assert.deepEqual(resolveSwarmSettings({ messages: 500 }, swarmDefaults), { ...swarmDefaults, messages: 500 });
+});
+
test("resolved defaults, initial overrides, children, and restarted services share one immutable settings contract", async () => {
const f = await swarmFixture();
const defaults = resolveSwarmSettings({
@@ -45,8 +59,8 @@ test("resolved defaults, initial overrides, children, and restarted services sha
const restarted = createSwarmService({ ...f.serviceOptions, defaults: resolveSwarmSettings({ turnMs: 999 }) });
assert.deepEqual((await restarted.inspect(caller)).settings, { ...defaults, turnMs: 777 });
await assert.rejects(
- restarted.spawn(caller, { requestId: "change", text: "work", settings: { turnMs: 999 } }),
- /initial/,
+ restarted.spawn(caller, { requestId: "change", text: "work", settings: { depth: 5 } }),
+ /only the root/,
);
await assert.rejects(restarted.spawn(caller, { requestId: "child", text: "work" }), /depth/);
await assert.rejects(restarted.context(caller, { long: "x".repeat(20) }), /context/);
@@ -57,8 +71,24 @@ test("resolved defaults, initial overrides, children, and restarted services sha
const run = (await f.runs.list()).find((r) => r.request.swarm?.recipientId === sibling!.id)!;
assert.equal(run.request.turnWallClockMs, 777);
await assert.rejects(restarted.spawn(f.caller, { requestId: "excess", text: "work" }), /agent budget/);
+ await assert.rejects(
+ restarted.spawn(f.caller, { requestId: "turn", text: "work", settings: { agents: 4, turnMs: 999 } }),
+ /turnMs cannot change/,
+ );
+ const [extra] = await restarted.spawn(f.caller, {
+ requestId: "raise",
+ text: "work",
+ settings: { agents: 4, spawnRequests: 3 },
+ });
+ assert.ok(extra);
+ assert.deepEqual((await restarted.inspect(f.caller)).settings, {
+ ...defaults,
+ turnMs: 777,
+ agents: 4,
+ spawnRequests: 3,
+ });
const swarm = (await f.store.get(f.root.id))!;
- assert.equal(swarm.expiresAt - swarm.createdAt, 60_000);
+ assert.equal(swarm.expiresAt! - swarm.createdAt, 60_000);
});
test("canonical recipient sets deduplicate retries but reject changed sets", async () => {
@@ -160,3 +190,24 @@ test("operators can disable swarms without changing other background work", () =
assert.equal(disabled.backgroundWorkEnabled, true);
assert.throws(() => loadConfig({ SWARMS_ENABLED: "invalid" }), /SWARMS_ENABLED/);
});
+
+test("concurrent root settings changes both apply", async () => {
+ const f = await swarmFixture();
+ await f.service.spawn(f.caller, { requestId: "pool", text: "work" });
+ await f.service.sweep();
+ const update = f.store.update.bind(f.store);
+ const ready = Promise.withResolvers<void>();
+ let arrivals = 0;
+ f.store.update = async (...args) => {
+ if (++arrivals === 2) ready.resolve();
+ await ready.promise;
+ return update(...args);
+ };
+ await Promise.all([
+ f.service.spawn(f.caller, { requestId: "agents", text: "work", settings: { agents: 64 } }),
+ f.service.spawn(f.caller, { requestId: "depth", text: "work", settings: { depth: 8 } }),
+ ]);
+ const { settings } = (await f.store.get(f.root.id))!;
+ assert.equal(settings.agents, 64);
+ assert.equal(settings.depth, 8);
+});
diff --git a/test/swarm-simplification.test.ts b/test/swarm-simplification.test.ts
index e6221d03f..490dbf3bd 100644
--- a/test/swarm-simplification.test.ts
+++ b/test/swarm-simplification.test.ts
@@ -145,11 +145,27 @@ test("initial pool cannot commit after its work window expires", async (context)
assert.equal(await f.store.get(f.root.id), null);
});
+test("a swarm without lifetimeMs keeps accepting work long after it was created", async (context) => {
+ const f = await swarmFixture();
+ context.mock.timers.enable({ apis: ["Date"], now: Date.now() });
+ const [worker] = await f.service.spawn(f.caller, { requestId: "initial", text: "work" });
+ await f.service.sweep();
+ assert.equal((await f.store.get(f.root.id))!.expiresAt, undefined);
+ context.mock.timers.tick(30 * 86_400_000);
+ const human = { kind: "human" as const, actorId: "alice", sessionId: f.root.id };
+ const message = await f.service.send(human, { requestId: "later", audience: "all", text: "more work" });
+ await f.service.sweep();
+ assert.ok((await f.runs.list()).some((run) => run.request.swarm?.messageId === message.id));
+ assert.equal((await f.service.inspect(human)).peers.find((p) => p.id === worker!.id)?.state, "ready");
+ const [second] = await f.service.spawn(human, { requestId: "second", text: "more" });
+ assert.ok(second);
+});
+
test("initial work window uses one timestamp even when the clock advances between reads", async (context) => {
const f = await swarmFixture();
let now = Date.now();
context.mock.method(Date, "now", () => now++);
await f.service.spawn(f.caller, { requestId: "initial", text: "work", settings: { lifetimeMs: 60_000 } });
const swarm = (await f.store.get(f.root.id))!;
- assert.equal(swarm.expiresAt - swarm.createdAt, 60_000);
+ assert.equal(swarm.expiresAt! - swarm.createdAt, 60_000);
});
diff --git a/test/swarms.test.ts b/test/swarms.test.ts
index 2d12adcc3..217946775 100644
--- a/test/swarms.test.ts
+++ b/test/swarms.test.ts
@@ -606,7 +606,7 @@ test("swarm settings are fixed at creation and reject conflicting attempts", asy
assert.equal(swarm.settings.turnMs, 600_000);
await assert.rejects(
service.spawn(caller, { requestId: "second", text: "Work", settings: { turnMs: 300_000 } }),
- /settings are only allowed on initial swarm creation/,
+ /turnMs cannot change after swarm creation/,
);
});
@@ -642,17 +642,18 @@ test("recursive spawn depth is enforced independently of arbitrary character", a
test("notification and message budgets include initial work and prevent recursion loops", async () => {
const { service, caller, store, root } = await swarmFixture();
- await service.spawn(caller, { requestId: "one", text: "Work" });
+ const settings = { messages: 40, notifications: 20 };
+ await service.spawn(caller, { requestId: "one", text: "Work", settings });
await service.sweep();
await store.update(root.id, (swarm) => {
- swarm.notificationCount = SWARM_DEFAULTS.notifications;
+ swarm.notificationCount = settings.notifications;
});
await assert.rejects(
service.send(caller, { requestId: "notify", audience: "all", text: "Work" }),
/notification budget/,
);
await service.send(caller, { requestId: "quiet", audience: "all", text: "No wake", notify: false });
- for (let index = 2; index < SWARM_DEFAULTS.messages; index++) {
+ for (let index = 2; index < settings.messages; index++) {
await service.send(caller, { requestId: `quiet-${index}`, audience: "all", text: "Archive", notify: false });
}
await assert.rejects(
@@ -662,6 +663,25 @@ test("notification and message budgets include initial work and prevent recursio
assert.equal((await service.read(caller, {})).length, 32);
});
+test("spawn request budget applies only when set", async () => {
+ const { service, caller } = await swarmFixture();
+ await service.spawn(caller, { requestId: "one", text: "Work", settings: { spawnRequests: 2 } });
+ await service.spawn(caller, { requestId: "two", text: "Work" });
+ await assert.rejects(service.spawn(caller, { requestId: "three", text: "Work" }), /spawn budget/);
+});
+
+test("without caps, message, notification and spawn counts are unlimited", async () => {
+ const { service, caller, runs } = await swarmFixture();
+ await service.spawn(caller, { requestId: "one", count: 2, text: "Work", settings: { agents: 64 } });
+ await service.sweep();
+ for (let index = 0; index < 130; index++)
+ await service.send(caller, { requestId: `notify-${index}`, audience: "all", text: "Work" });
+ await service.sweep();
+ assert.equal((await runs.list({ limit: 1_000 })).filter((run) => run.request.swarm).length, 262);
+ for (let index = 0; index < 33; index++) await service.spawn(caller, { requestId: `spawn-${index}`, text: "Work" });
+ assert.equal((await service.inspect(caller)).peers.length, 36);
+});
+
test("restart between enqueue and outbox acknowledgement does not redeliver", async () => {
const { service, serviceOptions, caller, store, root, runs } = await swarmFixture();
await service.spawn(caller, { requestId: "pool", count: 2, text: "Work" });
@@ -847,6 +867,20 @@ test("expired swarms retain readable history but cannot reserve or wake more wor
assert.equal((await runs.list()).length, 1);
});
+test("the root can extend an expired work window", async () => {
+ const { service, caller, store, root } = await swarmFixture();
+ await service.spawn(caller, { requestId: "one", text: "Work", settings: { lifetimeMs: 60_000 } });
+ await store.update(root.id, (swarm) => {
+ swarm.createdAt -= 120_000;
+ swarm.expiresAt = swarm.createdAt + 60_000;
+ });
+ await assert.rejects(service.spawn(caller, { requestId: "two", text: "Work" }), /expired/);
+ await service.spawn(caller, { requestId: "three", text: "Work", settings: { lifetimeMs: 3_600_000 } });
+ const swarm = (await store.get(root.id))!;
+ assert.equal(swarm.expiresAt, swarm.createdAt + 3_600_000);
+ await service.send(caller, { requestId: "message", text: "Work", audience: "all" });
+});
+
test("memory store has transaction rollback and detached read parity", async () => {
const { service, caller, root, store } = await swarmFixture();
await service.spawn(caller, { requestId: "one", text: "Work" });