Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

Cheap machine reports: a fast path for reading-only heartbeats and one report call per machine (DB 2.5) (#2378)

0xken1558381b5d61b6
diff --git a/backend/integration/superTabMachineReports.test.ts b/backend/integration/superTabMachineReports.test.ts
new file mode 100644
index 000000000..6ac01cd1e
--- /dev/null
+++ b/backend/integration/superTabMachineReports.test.ts
@@ -0,0 +1,397 @@
+import { randomUUID } from 'node:crypto'
+import { execFile } from 'node:child_process'
+import { promisify } from 'node:util'
+import { join } from 'node:path'
+import { tmpdir } from 'node:os'
+import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from 'vitest'
+import { MongoMemoryReplSet } from 'mongodb-memory-server-core'
+import { PrismaClient } from '@prisma/client'
+import type { AuthUser } from '../src/lib/ssoAuth.js'
+import { SuperTabStore, type SwarmChanged } from '../src/superTabs/store.js'
+import { configureSwarmStamps } from '../src/superTabs/cache.js'
+import { MemoryStamps } from '../src/superTabs/cache.test-support.js'
+import { localReportMemory, machineReportLine, RECORD_TTL_MS, type ReportMemory } from '../src/superTabs/machineReports.js'
+import { SWEEP_EVERY_MS } from '../src/superTabs/liveness.js'
+import type { Command } from '../src/superTabs/commands.js'
+import type { MachineObservation } from '../src/superTabs/machines.js'
+import type { MachineData } from '../src/superTabs/parts/index.js'
+import { poolRows } from './hostRows.test-support.js'
+
+// Machine reports that change only the machine's reading skip the Swarm load (DB roadmap 2.5). A fresh loopback replica
+// set, created here; Redis is MemoryStamps and an in-process report memory. This suite never reads DATABASE_URL.
+let replica: MongoMemoryReplSet
+let db: PrismaClient
+let stamps: MemoryStamps
+const owner: AuthUser = { sub: '300000000000000000000001', email: '[email protected]', autonomousEnv: 'prod', role: 'user' }
+const host: AuthUser = { sub: '300000000000000000000003', email: '[email protected]', autonomousEnv: 'prod', role: 'user' }
+const config = { engine: 'codex', model: 'gpt-6.1-sol', effort: 'high', contextWindow: 200000, engineVersion: 'fixture' }
+const BEAT = 15_000
+const MINUTE = 60_000
+
+beforeAll(async () => {
+ replica = await MongoMemoryReplSet.create({ binary: { version: '7.0.24', downloadDir: join(tmpdir(), 'harness-super-tab-mongodb-binaries') },
+ replSet: { count: 1, ip: '127.0.0.1', storageEngine: 'wiredTiger' } })
+ const uri = replica.getUri(`swarm_machine_reports_test_${randomUUID().replaceAll('-', '_')}`)
+ await promisify(execFile)(join(process.cwd(), 'node_modules/.bin/prisma'), ['db', 'push', '--skip-generate'], {
+ env: { ...process.env, DATABASE_URL: uri }, timeout: 60000,
+ })
+ db = new PrismaClient({ datasources: { db: { url: uri } } })
+ for (const user of [owner, host]) await db.user.create({ data: { id: user.sub, externalId: `fixture-${user.sub}`, email: user.email, autonomousEnv: user.autonomousEnv } })
+ for (const machineId of ['lead-host', 'test-host', 'spare-host']) {
+ await db.machine.create({ data: { machineId, userId: host.sub, apiKey: `fixture-${machineId}`, managerId: 'fixture', autonomousEnv: 'prod', authMode: 'remote' } })
+ await db.machineShare.create({ data: { machineId, ownerId: host.sub, autonomousEnv: 'prod', emails: [owner.email] } })
+ }
+})
+afterAll(async () => {
+ await db?.$disconnect()
+ await replica?.stop()
+})
+beforeEach(() => { stamps = new MemoryStamps(); configureSwarmStamps(stamps) })
+afterEach(() => configureSwarmStamps(new MemoryStamps()))
+
+/** A memory whose sweeps are never due, so a test sees only the reason it is about. */
+const quiet = (memory: ReportMemory): ReportMemory => ({ get: (t, m) => memory.get(t, m), set: (t, m, r) => memory.set(t, m, r), sweepDue: async () => false })
+/** A server that remembers no full report: every reading runs in full. */
+const never: ReportMemory = { get: async () => undefined, set: async () => {}, sweepDue: async () => false }
+
+/** A Swarm whose Lead runs on lead-host and whose worker runs on test-host, working on a task the owner asked for. Swarms
+ * given one [shared] clock and memory are one server's. */
+async function swarm(memory?: (now: () => number) => ReportMemory, shared?: { clock: { now: number }; reports: ReportMemory }) {
+ const clock = shared?.clock ?? { now: 10_000_000 }
+ const pushes: SwarmChanged[] = []
+ const reports = shared?.reports ?? memory?.(() => clock.now) ?? quiet(localReportMemory(() => clock.now))
+ const store = (with_: ReportMemory = reports) => new SuperTabStore(db, () => clock.now, null, { reports: with_, swarmChanged: change => pushes.push(change) })
+ const timed = store()
+ const tab = (await timed.create(owner, randomUUID(), 'Reports', 'public')).id
+ const execute = (command: Command, user = owner) => timed.execute(user, tab, { operationId: randomUUID(), command })
+ const invitation = await execute({ type: 'invite.create', email: host.email, role: 'collaborator', conversationId: 'lead' })
+ await execute({ type: 'invite.accept', inviteId: invitation.result.inviteId as string }, host)
+ for (const [machineId, name] of [['lead-host', 'Lead Mac'], ['test-host', 'Worker Mac']]) {
+ await execute({ type: 'machine.contribute', machineId: machineId!, name: name!, cwd: '/repo', engines: ['codex'], maxAgents: 4 }, host)
+ }
+ const generations = new Map<string, string>()
+ for (const machineId of ['lead-host', 'test-host']) generations.set(machineId, (await poolRows(timed, host, machineId)).find(p => p.tabId === tab)!.machine.generation)
+ const harness = { lead: randomUUID(), worker: randomUUID() }
+ const add = async (conversationId: string, machineId: string, harnessId: string, name: string, lead = false) => String((await execute({ type: 'agent.add',
+ conversationId, machineId, harnessId, name, configuration: config, lead })).result.agentId)
+ const lead = await add('lead', 'lead-host', harness.lead, 'Lead', true)
+ const group = String((await execute({ type: 'conversation.create', title: 'Ship the parser', brief: 'Implement it',
+ checks: [{ description: 'Verified', independent: false }] })).result.conversationId)
+ const worker = await add(group, 'test-host', harness.worker, 'Worker')
+ await execute({ type: 'work.assign', conversationId: group, agentId: worker })
+ const reading = (machineId: string, running: string[], extra: Partial<MachineObservation> = {}): MachineObservation => ({
+ generation: generations.get(machineId)!, resources: { cpuPercent: 10, memoryTotalBytes: 32e9, memoryUsedBytes: 4e9, diskFreeBytes: 120e9,
+ diskTotalBytes: 500e9, projectDiskFreeBytes: 120e9, deviceType: 'Laptop', chip: 'Fixture', os: 'macOS', gpuPercent: null },
+ quotas: [], tools: {}, runningHarnessIds: running, stoppedHarnessIds: [], ...extra })
+ /** The worker's machine reports its session running, with what else [extra] says. */
+ const beat = (extra: Partial<MachineObservation> = {}, by = timed) => by.observeMachine(host, tab, 'test-host', reading('test-host', [harness.worker], extra))
+ const part = () => db.swarmMachine.findUniqueOrThrow({ where: { id: `${tab}:test-host` } })
+ const assignment = async () => (await timed.read(owner, tab)).state.conversations.find(c => c.id === group)!.assignment!
+ return { clock, pushes, reports, store, timed, tab, execute, lead, group, worker, harness, reading, beat, part, assignment }
+}
+type Scene = Awaited<ReturnType<typeof swarm>>
+
+/** Which path the one report [run] sends took. */
+async function pathOf(run: () => Promise<unknown>): Promise<'fast' | 'full'> {
+ machineReportLine(0)
+ let failed: unknown
+ try { await run() } catch (error) { failed = error }
+ const { fast, full } = machineReportLine(0)
+ if (fast + full !== 1) throw new Error(`expected one report, saw ${fast} fast and ${full} full`)
+ if (failed) throw failed
+ return fast ? 'fast' : 'full'
+}
+
+/** Heartbeats every 15 s until one is stored fast; the paths taken, at most three. */
+async function settle(f: Scene, extra: Partial<MachineObservation> = {}): Promise<Array<'fast' | 'full'>> {
+ const paths: Array<'fast' | 'full'> = []
+ while (paths.at(-1) !== 'fast' && paths.length < 3) {
+ paths.push(await pathOf(() => f.beat(extra)))
+ f.clock.now += BEAT
+ }
+ return paths
+}
+
+/** The part collections' operations MongoDB served while `run` ran, by collection (its profiler). */
+async function partOps(run: () => Promise<unknown>): Promise<Record<string, number>> {
+ await db.$runCommandRaw({ profile: 2 })
+ const since = new Date()
+ try {
+ await run()
+ const rows = await db.$runCommandRaw({ find: 'system.profile', filter: { ts: { $gte: { $date: since.toISOString() } },
+ ns: { $regex: '\\.Swarm(Head|Member|Agent|Group|Machine|Start|Progress)$' } }, projection: { _id: 0, ns: 1 }, batchSize: 10_000 }) as
+ { cursor: { firstBatch: Array<{ ns: string }> } }
+ const counts: Record<string, number> = {}
+ for (const { ns } of rows.cursor.firstBatch) counts[ns.slice(ns.lastIndexOf('.') + 1)] = (counts[ns.slice(ns.lastIndexOf('.') + 1)] ?? 0) + 1
+ return counts
+ } finally {
+ await db.$runCommandRaw({ profile: 0 })
+ }
+}
+
+describe('a reading-only heartbeat (DB 2.5)', () => {
+ it('runs in full once, then is stored without loading the Swarm', async () => {
+ const f = await swarm()
+ const paths = await settle(f)
+ expect(paths.at(-1)).toBe('fast')
+ expect(paths[0]).toBe('full')
+ })
+
+ it('stores exactly what the full path stores, Lent included, and pushes the same to the same people', async () => {
+ const f = await swarm()
+ await settle(f, { harnessProgress: { [f.harness.worker]: f.clock.now } })
+ const before = await f.part()
+ const heartbeat = { harnessProgress: { [f.harness.worker]: f.clock.now - 1_000 }, resources: { ...f.reading('test-host', []).resources, cpuPercent: 77 } }
+ f.pushes.length = 0
+ const fast = await (async () => { let version = 0; expect(await pathOf(async () => { version = (await f.beat(heartbeat)).version })).toBe('fast'); return version })()
+ const viaFast = { data: (await f.part()).data, pushes: [...f.pushes] }
+ // The same heartbeat again from the same moment, by a server that remembers nothing: the full path.
+ await db.swarmMachine.update({ where: { id: before.id }, data: { version: before.version, data: before.data as never, readAt: before.readAt } })
+ f.pushes.length = 0
+ const forgetful = f.store(never)
+ let full = 0
+ expect(await pathOf(async () => { full = (await f.beat(heartbeat, forgetful)).version })).toBe('full')
+ const viaFull = { data: (await f.part()).data, pushes: [...f.pushes] }
+ expect(fast).toBe(full)
+ expect(viaFast.data).toEqual(viaFull.data)
+ expect((viaFast.data as unknown as MachineData).contribution!.lent?.[0]?.busyMs).toBeGreaterThan(0)
+ // Each push carries its own write's stamp; who is told, and of what, is the same.
+ const pushed = (list: SwarmChanged[]) => list.map(({ lastChangedAt: _stamp, ...p }) => ({ ...p, userIds: [...p.userIds].sort() }))
+ expect(pushed(viaFast.pushes)).toEqual(pushed(viaFull.pushes))
+ expect(viaFast.pushes).toMatchObject([{ parts: ['machine'] }])
+ })
+
+ it('reads only the machine part among the Swarm\'s parts, writes it once, and moves only the machine stamp', async () => {
+ const f = await swarm()
+ await settle(f)
+ const bumps = stamps.bumps.length
+ let path: 'fast' | 'full' = 'full'
+ const ops = await partOps(async () => { path = await pathOf(() => f.beat()) })
+ expect(path).toBe('fast')
+ expect(Object.keys(ops)).toEqual(['SwarmMachine'])
+ // Its read, and its version-guarded write: Prisma's updateMany reads the matching ids, then updates them.
+ expect(ops.SwarmMachine).toBeLessThanOrEqual(3)
+ expect(stamps.bumps.slice(bumps)).toEqual([{ tabId: f.tab, scopes: ['machine'] }])
+ })
+})
+
+describe('every reason a report runs in full (DB 2.5 spec, Tests)', () => {
+ /** What changes before the heartbeat, the heartbeat itself, and which server it reaches. */
+ type Change = { extra?: Partial<MachineObservation>; by?: SuperTabStore }
+ const cases: Array<[string, (f: Scene) => Promise<Change | void>, { stopped?: true; progress?: true }?]> = [
+ ['its running sessions changed', async f => ({ extra: { runningHarnessIds: [f.harness.worker, 'another'] } })],
+ ['its stopped sessions changed', async () => ({ extra: { stoppedHarnessIds: ['gone'] } })],
+ ['it proves a session stopped', async () => ({ extra: { stoppedReservations: [{ harnessId: 'gone', generation: 'g1' }] } }), { stopped: true }],
+ ['a running session lost its progress entry', async () => ({ extra: { harnessProgress: {} } }), { progress: true }],
+ ['the Swarm changed since its last full report', async f => {
+ await f.execute({ type: 'message.send', conversationId: 'lead', text: 'hi', replyTo: null, recipientIds: [] })
+ }],
+ ['its record is missing', async f => ({ by: f.store(never) })],
+ ['the stamps cannot be read', async () => { stamps.down = true }],
+ ['its record is older than two minutes, whatever other machines wrote since', async f => {
+ await f.reports.set(f.tab, 'test-host', { ...(await f.reports.get(f.tab, 'test-host'))!, at: f.clock.now - RECORD_TTL_MS })
+ }],
+ ['the record says a Team account runs its agents', async f => {
+ await f.reports.set(f.tab, 'test-host', { ...(await f.reports.get(f.tab, 'test-host'))!, team: true })
+ }],
+ ['a bump of its Swarm is owed here', async f => {
+ const bump = stamps.bump.bind(stamps)
+ stamps.bump = async () => { throw new Error('Redis is unreachable') }
+ await f.execute({ type: 'message.send', conversationId: 'lead', text: 'owed', replyTo: null, recipientIds: [] })
+ stamps.bump = bump
+ }],
+ ['an engine suspension ends with this reading', async f => {
+ const row = await f.part()
+ const data = structuredClone(row.data) as unknown as MachineData
+ data.contribution!.engineSuspensions = { codex: { at: f.clock.now - 7 * 60 * MINUTE, fine: false } }
+ await db.swarmMachine.update({ where: { id: row.id }, data: { data: data as never } })
+ }],
+ ]
+ for (const [reason, change, baseline] of cases) {
+ it(reason, async () => {
+ const f = await swarm()
+ const settled: Partial<MachineObservation> = { ...baseline?.stopped ? { stoppedHarnessIds: ['gone'] } : {},
+ ...baseline?.progress ? { harnessProgress: { [f.harness.worker]: f.clock.now } } : {} }
+ expect((await settle(f, settled)).at(-1)).toBe('fast')
+ const { extra = {}, by = f.timed } = await change(f) ?? {}
+ expect(await pathOf(() => f.beat({ ...settled, ...extra }, by))).toBe('full')
+ })
+ }
+
+ it('a machine of the Swarm is being removed', async () => {
+ const f = await swarm()
+ expect((await settle(f)).at(-1)).toBe('fast')
+ // A spare machine leaves while online: it stays listed, cleaning, until its host confirms (machines.ts removal).
+ await f.execute({ type: 'machine.contribute', machineId: 'spare-host', name: 'Spare Mac', cwd: '/repo', engines: ['codex'], maxAgents: 1 }, host)
+ const generation = (await poolRows(f.timed, host, 'spare-host')).find(p => p.tabId === f.tab)!.machine.generation
+ await f.timed.observeMachine(host, f.tab, 'spare-host', { ...f.reading('test-host', []), generation })
+ await f.execute({ type: 'machine.remove', machineId: 'spare-host' }, host)
+ expect(await pathOf(() => f.beat())).toBe('full')
+ f.clock.now += BEAT
+ expect(await pathOf(() => f.beat())).toBe('full')
+ })
+
+ it('its sweep is due', async () => {
+ const f = await swarm(now => localReportMemory(now))
+ const start = f.clock.now
+ const paths: string[] = []
+ // The first report sweeps; within 30 s the reports may go fast; at 30 s the next one sweeps, in full.
+ for (const at of [0, 5_000, 10_000, 20_000, SWEEP_EVERY_MS]) {
+ f.clock.now = start + at
+ paths.push(await pathOf(() => f.beat()))
+ }
+ expect(paths[0]).toBe('full')
+ expect(paths[3]).toBe('fast')
+ expect(paths[4]).toBe('full')
+ })
+
+ it('a Team account\'s reading came in since: the owner\'s own sign-in is refused as the full report refuses it', async () => {
+ const f = await swarm()
+ expect((await settle(f)).at(-1)).toBe('fast')
+ // As a fast reading from the machine's Harness Team account stores it: same sessions, signed in as the team account.
+ const row = await f.part()
+ const data = structuredClone(row.data) as unknown as MachineData
+ data.contribution!.observation!.sessionKind = 'team'
+ await db.swarmMachine.update({ where: { id: row.id }, data: { data: data as never } })
+ try {
+ machineReportLine(0)
+ await expect(f.beat()).rejects.toMatchObject({ code: 'TEAM_HOST_ONLY' })
+ expect(machineReportLine(0)).toMatchObject({ fast: 0, full: 1 })
+ } finally {
+ // The machine's host form reads every Swarm it is in, and refuses this sign-in for one that records a Team account.
+ await db.swarmMachine.update({ where: { id: row.id }, data: { data: row.data as never } })
+ }
+ })
+
+ it('its contribution was regenerated: refused as revoked', async () => {
+ const f = await swarm()
+ expect((await settle(f)).at(-1)).toBe('fast')
+ await expect(f.beat({ generation: 'not-the-current-one' })).rejects.toMatchObject({ code: 'CONTRIBUTION_REVOKED' })
+ })
+})
+
+describe('time-based losses and Lent on the fast path', () => {
+ it('loses a worker\'s task by the first report after its lease ends, in full', async () => {
+ const f = await swarm()
+ // Its screen last changed long ago: the reports' renewal stops 25 minutes after it, its lease 5 minutes later.
+ const progress = { harnessProgress: { [f.harness.worker]: f.clock.now - 29 * MINUTE } }
+ expect((await settle(f, progress)).at(-1)).toBe('fast')
+ const deadline = (await f.reports.get(f.tab, 'test-host'))!.deadline
+ expect(deadline).toBeGreaterThan(f.clock.now)
+ // Heartbeats up to the deadline: fast while the record holds (its key expires 2 minutes after a full report).
+ const paths: string[] = []
+ while (f.clock.now + BEAT < deadline) {
+ f.clock.now += BEAT
+ paths.push(await pathOf(() => f.beat(progress)))
+ }
+ f.clock.now = deadline - 1
+ await f.beat(progress)
+ expect(paths).toContain('fast')
+ expect((await f.assignment()).status).toBe('working')
+ f.clock.now = deadline + 1
+ expect(await pathOf(() => f.beat(progress))).toBe('full')
+ expect((await f.assignment()).status).toBe('lost')
+ })
+
+ it('counts the same Lent after N fast heartbeats as after N full ones, also right after the running set changed', async () => {
+ const fast = await swarm(), full = await swarm()
+ const forgetful = full.store(never)
+ const run = async (f: Scene, by = f.timed) => {
+ const paths: string[] = []
+ // Not running yet, then running: the full report that changes the running set records the count for the next ones.
+ paths.push(await pathOf(() => by.observeMachine(host, f.tab, 'test-host', f.reading('test-host', []))))
+ for (let i = 0; i < 5; i++) {
+ f.clock.now += BEAT
+ paths.push(await pathOf(() => f.beat({}, by)))
+ }
+ return { paths, lent: ((await f.part()).data as unknown as MachineData).contribution!.lent }
+ }
+ const a = await run(fast), b = await run(full, forgetful)
+ expect(a.paths).toContain('fast')
+ expect(b.paths.every(p => p === 'full')).toBe(true)
+ expect(a.lent).toEqual(b.lent)
+ expect(a.lent?.[0]?.busyMs).toBe(4 * BEAT)
+ })
+
+ it('never lets a refused reading take the Swarm\'s sweep', async () => {
+ const f = await swarm(now => localReportMemory(now))
+ const lead = (extra: Partial<MachineObservation> = {}) => f.timed.observeMachine(host, f.tab, 'lead-host', { ...f.reading('lead-host', [f.harness.lead]), ...extra })
+ await lead()
+ await f.beat()
+ // The worker's machine goes silent past two minutes; a revoked reading arrives first, then the Lead machine's own.
+ f.clock.now += 125_000
+ await expect(lead({ generation: 'not-the-current-one' })).rejects.toMatchObject({ code: 'CONTRIBUTION_REVOKED' })
+ await lead()
+ expect((await f.assignment()).status).toBe('lost')
+ })
+
+ it('sweeps a Swarm once per 30 seconds between servers sharing one memory', async () => {
+ for (const shared of [true, false]) {
+ const f = await swarm(now => localReportMemory(now))
+ const other = f.store(shared ? f.reports : localReportMemory(() => f.clock.now))
+ const lead = (by: SuperTabStore) => by.observeMachine(host, f.tab, 'lead-host', f.reading('lead-host', [f.harness.lead]))
+ await lead(f.timed)
+ await f.beat()
+ // This server's sweep runs while the worker's machine is still fresh.
+ f.clock.now += 100_000
+ await lead(f.timed)
+ // 25 s on, the worker's machine has been silent past two minutes: only a server whose sweep is due loses its task.
+ f.clock.now += 25_000
+ await lead(other)
+ expect((await f.assignment()).status, shared ? 'shared clock' : 'own clock').toBe(shared ? 'working' : 'lost')
+ if (!shared) continue
+ f.clock.now += 5_000
+ await lead(other)
+ expect((await f.assignment()).status).toBe('lost')
+ }
+ })
+})
+
+describe('a machine\'s readings for all its Swarms in one call (DB 2.5)', () => {
+ /** [count] Swarms the worker's machine is in, on one server. */
+ async function server(count: number) {
+ const clock = { now: 10_000_000 }, reports = quiet(localReportMemory(() => clock.now))
+ const swarms: Scene[] = []
+ for (let i = 0; i < count; i++) swarms.push(await swarm(undefined, { clock, reports }))
+ return { clock, swarms, store: swarms[0]!.timed }
+ }
+ /** The `Machine` rows (collection `machines`) read while [run] ran. */
+ const machineReads = async (run: () => Promise<unknown>) => {
+ await db.$runCommandRaw({ profile: 2 })
+ const since = new Date()
+ try {
+ await run()
+ const rows = await db.$runCommandRaw({ find: 'system.profile', filter: { ts: { $gte: { $date: since.toISOString() } }, ns: { $regex: '\\.machines$' } },
+ projection: { _id: 0, ns: 1 } }) as { cursor: { firstBatch: unknown[] } }
+ return rows.cursor.firstBatch.length
+ } finally {
+ await db.$runCommandRaw({ profile: 0 })
+ }
+ }
+
+ it('answers each Swarm on its own: one stored fast, one in full, one revoked; the machine\'s owner checked once', async () => {
+ const { clock, swarms: [warm, cold, revoked], store } = await server(3)
+ expect((await settle(warm!)).at(-1)).toBe('fast')
+ clock.now += BEAT
+ let results: Array<{ tabId: string; error?: unknown }> = []
+ machineReportLine(0)
+ const reads = await machineReads(async () => {
+ results = await store.observeMachines(host, 'test-host', [{ tabId: warm!.tab, observation: warm!.reading('test-host', [warm!.harness.worker]) },
+ { tabId: cold!.tab, observation: cold!.reading('test-host', [cold!.harness.worker]) },
+ { tabId: revoked!.tab, observation: { ...revoked!.reading('test-host', [revoked!.harness.worker]), generation: 'not-the-current-one' } }])
+ })
+ expect(machineReportLine(0)).toMatchObject({ fast: 1, full: 2 })
+ expect(results.map(r => [r.tabId, (r.error as { code?: string } | undefined)?.code])).toEqual([[warm!.tab, undefined], [cold!.tab, undefined],
+ [revoked!.tab, 'CONTRIBUTION_REVOKED']])
+ expect(reads).toBe(1)
+ for (const f of [warm!, cold!]) expect(((await f.part()).data as unknown as MachineData).contribution!.observation!.receivedAt).toBe(clock.now)
+ })
+
+ it('stores a machine\'s readings for 21 Swarms in one call', async () => {
+ const { clock, swarms, store } = await server(21)
+ clock.now += BEAT
+ const results = await store.observeMachines(host, 'test-host', swarms.map(f => ({ tabId: f.tab, observation: f.reading('test-host', [f.harness.worker]) })))
+ expect(results.filter(r => r.error)).toEqual([])
+ for (const f of swarms) expect(((await f.part()).data as unknown as MachineData).contribution!.observation!.receivedAt).toBe(clock.now)
+ })
+})
diff --git a/backend/src/lib/teamSession.ts b/backend/src/lib/teamSession.ts
index d5c723462..f9d9581bf 100644
--- a/backend/src/lib/teamSession.ts
+++ b/backend/src/lib/teamSession.ts
@@ -119,6 +119,8 @@ export const TEAM_ROUTES: readonly TeamRoute[] = [
// has no git login of its own (routes/superTabs.ts).
{ method: 'POST', url: '/api/super-tabs/hosts/:machineId/starts/:tabId/:startId/github-token', scope: 'machine' },
{ method: 'POST', url: '/api/super-tabs/:tabId/machines/:machineId', scope: 'machine' },
+ // The same readings for all of its Swarms in one call (DB roadmap 2.5).
+ { method: 'POST', url: '/api/super-tabs/hosts/:machineId/readings', scope: 'machine' },
// Reads as one of its agents.
{ method: 'GET', url: '/api/super-tabs/:tabId', scope: 'agent' },
{ method: 'GET', url: '/api/super-tabs/:tabId/parts', scope: 'agent' },
diff --git a/backend/src/routes/superTabs.test.ts b/backend/src/routes/superTabs.test.ts
index 9e4e40890..b69ab13f7 100644
--- a/backend/src/routes/superTabs.test.ts
+++ b/backend/src/routes/superTabs.test.ts
@@ -50,7 +50,8 @@ import { registerAuthMiddleware } from '../middlewares/authMiddleware.js'
import { redeemHandoff, startHandoff } from '../lib/harnessSession.js'
import type { AuthUser } from '../lib/ssoAuth.js'
import { SuperTabError } from '../superTabs/model.js'
-import { HOT_BODY_LIMIT, NOT_AVAILABLE, superTabRoutes } from './superTabs.js'
+import { HOT_BODY_LIMIT, NOT_AVAILABLE, READINGS_MAX, superTabRoutes } from './superTabs.js'
+import { machineObservationSchema } from '../superTabs/machines.js'
import { logger } from '../utils/logger.js'
import { WORKSPACE_ACCESS_REVOKED } from '../superTabs/workspaceStore.js'
@@ -255,3 +256,53 @@ describe('the browser\'s own sign-in on Swarm routes', () => {
expect(seen).toEqual([])
})
})
+
+describe('a machine\'s readings for all its Swarms in one call (DB 2.5)', () => {
+ let app: FastifyInstance
+ let calls: Array<{ machineId: string; readings: Array<{ tabId: string }> }>
+ let answers: Array<{ tabId: string; error?: unknown }> | null
+ beforeEach(async () => {
+ calls = []
+ answers = null
+ app = Fastify()
+ app.setErrorHandler(errorHandler)
+ app.addHook('onRequest', async request => { request.user = { sub: 'u1', email: '[email protected]', autonomousEnv: 'prod', role: 'user' } as never })
+ const store = { observeMachines: async (_user: unknown, machineId: string, readings: Array<{ tabId: string }>) => {
+ calls.push({ machineId, readings })
+ return answers ?? readings.map(r => ({ tabId: r.tabId }))
+ } }
+ await app.register(superTabRoutes, { store: store as never, github: null, reads: {} as never, hosts: {} as never })
+ await app.ready()
+ })
+ afterEach(async () => { await app.close(); vi.restoreAllMocks() })
+ const post = (body: unknown) => app.inject({ method: 'POST', url: '/api/super-tabs/hosts/m1/readings', payload: body as object })
+ const readings = (n: number) => Array.from({ length: n }, (_, i) => ({ tabId: `tab-${i}`, observation: { any: 'shape' } }))
+
+ it('answers each Swarm on its own, as the per-Swarm route would answer it', async () => {
+ vi.spyOn(logger, 'error').mockImplementation(() => {})
+ vi.spyOn(logger, 'warn').mockImplementation(() => {})
+ answers = [{ tabId: 'tab-0' }, { tabId: 'tab-1', error: new SuperTabError('CONTRIBUTION_REVOKED', 'Read the current machine contribution.', 403) },
+ { tabId: 'tab-2', error: new SuperTabError('NOT_FOUND', 'This Super Tab is unavailable.', 404) },
+ { tabId: 'tab-3', error: machineObservationSchema.safeParse({ generation: 1 }).error }, { tabId: 'tab-4', error: new Error('boom') }]
+ const res = await post({ readings: readings(5) })
+ expect(res.statusCode).toBe(200)
+ expect(res.json().data.results).toEqual([{ tabId: 'tab-0', ok: true },
+ { tabId: 'tab-1', error: { code: 'CONTRIBUTION_REVOKED', message: 'Read the current machine contribution.', status: 403 } },
+ { tabId: 'tab-2', error: { ...NOT_AVAILABLE } },
+ { tabId: 'tab-3', error: { code: 'VALIDATION_ERROR', message: expect.any(String), status: 400 } },
+ { tabId: 'tab-4', error: { code: 'INTERNAL_ERROR', message: expect.any(String), status: 500 } }])
+ // Each observation reaches the store as sent: it is checked there, Swarm by Swarm.
+ expect(calls).toEqual([{ machineId: 'm1', readings: readings(5) }])
+ })
+
+ it(`takes up to ${READINGS_MAX} readings, each Swarm once, and refuses any other body before reaching the store`, async () => {
+ expect((await post({ readings: readings(READINGS_MAX) })).statusCode).toBe(200)
+ for (const body of [{ readings: readings(READINGS_MAX + 1) }, { readings: [] }, { readings: [...readings(1), ...readings(1)] },
+ { readings: [{ tabId: 'bad id!', observation: {} }] }, { readings: readings(1), extra: true }]) {
+ const res = await post(body)
+ expect(res.statusCode, JSON.stringify(body).slice(0, 80)).toBe(400)
+ expect(res.json().error.code).toBe('VALIDATION_ERROR')
+ }
+ expect(calls).toHaveLength(1)
+ })
+})
diff --git a/backend/src/routes/superTabs.ts b/backend/src/routes/superTabs.ts
index b287bf602..b76191500 100644
--- a/backend/src/routes/superTabs.ts
+++ b/backend/src/routes/superTabs.ts
@@ -1,5 +1,5 @@
import type { FastifyInstance, FastifyReply } from 'fastify'
-import { z } from 'zod'
+import { z, ZodError } from 'zod'
import { prisma } from '../lib/prisma.js'
import { env } from '../config/env.js'
import multipart from '@fastify/multipart'
@@ -10,6 +10,7 @@ import { operationSchema } from '../superTabs/commands.js'
import { fail, SuperTabError } from '../superTabs/model.js'
import { SuperTabStore } from '../superTabs/store.js'
import { redisPollMemory } from '../superTabs/pollMemory.js'
+import { redisReportMemory } from '../superTabs/machineReports.js'
import { routingPolicySaveSchema } from '../superTabs/routingPolicy.js'
import { nativeEventSchema, openNativeSchema } from '../superTabs/native.js'
import { machineObservationSchema } from '../superTabs/machines.js'
@@ -149,6 +150,28 @@ export const NOT_AVAILABLE = { code: 'NOT_FOUND', message: 'This is not availabl
const UNREACHABLE_CODES: ReadonlySet<string> = new Set(['NOT_FOUND', 'JOIN_UNAVAILABLE', WORKSPACE_ACCESS_REVOKED])
const unreachable = (error: SuperTabError) => error.unreachable || UNREACHABLE_CODES.has(error.code)
+/** At most this many Swarms' readings in one call; the most a machine was seen in is 21 (DB roadmap 2.5). */
+export const READINGS_MAX = 64
+/** A reading is a few KB, at most ~700 KB at its schema's limits (see HOT_BODY_LIMIT), so a batch gets a limit of its own. */
+export const READINGS_BODY_LIMIT = 8 * 1024 * 1024
+/** Each observation is checked on its own (machineObservationSchema, in the store), so one a backend refuses fails only its
+ * Swarm, and the host sends that one again through the per-Swarm route with its older fields (cli superTabs/machines.ts). */
+const readingsBody = z.object({ readings: z.array(z.object({ tabId: id, observation: z.unknown() }).strict()).min(1).max(READINGS_MAX)
+ .refine(list => new Set(list.map(r => r.tabId)).size === list.length, 'Name each Swarm once.') }).strict()
+
+/** One reading's refusal in a batch, answered as the per-Swarm route answers it: a Swarm the caller cannot reach gets the one
+ * NOT_AVAILABLE answer (that route is in NOT_AVAILABLE_ROUTES), anything unexpected a 500 for that Swarm alone, logged. */
+export function readingError(tabId: string, error: unknown): { code: string; message: string; status: number } {
+ if (error instanceof ZodError) return { code: 'VALIDATION_ERROR', message: error.issues[0]?.message ?? 'This reading is not valid.', status: 400 }
+ if (error instanceof SuperTabError) {
+ if (!unreachable(error)) return { code: error.code, message: error.message, status: error.status }
+ logger.warn('Swarm not available to the caller', { code: error.code, statusCode: error.status, reason: error.message, tabId, path: 'readings' })
+ return { ...NOT_AVAILABLE }
+ }
+ logger.error('[super-tab] a machine reading failed', error instanceof Error ? error : new Error(String(error)), { tabId })
+ return { code: 'INTERNAL_ERROR', message: 'This reading could not be stored. It is sent again with the next one.', status: 500 }
+}
+
async function respond(reply: FastifyReply, action: () => Promise<unknown>) {
try { sendSuccess(reply, await action()) } catch (error) {
if (error instanceof GitHubAppError) return sendError(reply, error.message, error.code, error.status, error.details)
@@ -199,6 +222,8 @@ async function productionStore(github: GitHubTokens | null): Promise<SuperTabSto
// Every server answers the quiet delivery poll from any server's full one, and tells the agent's machine of new work
// through whichever server holds its daemon socket (stability roadmap 2.4). Both only ever drop, never fail, on Redis.
polls: redisPollMemory(bus.pub),
+ // What each machine's full report saw, so its heartbeats skip the Swarm load, and one sweep clock per Swarm (DB 2.5).
+ reports: redisReportMemory(bus.pub),
deliveriesWaiting: announceSwarmDeliveries,
// And each machine whose host read a commit changed, so hosts read their Swarm changes only when told (WS Phase 2a).
hostsChanged: announceSwarmHostChanges,
@@ -324,6 +349,11 @@ export async function superTabRoutes(app: FastifyInstance, options: { store?: Su
// Worded as the desktop's GitHub access dialog is (design 2026-10-30): allowed, not approved.
return page(`${named.fullName} is allowed`, `Teammates' agents in ${where} can now push to it. You can go back to Harness.`)
})
+ // A machine's readings for all of its Swarms in one call, answered per Swarm (DB roadmap 2.5).
+ app.post<{ Params: z.infer<typeof hostParams>; Body: z.infer<typeof readingsBody> }>('/api/super-tabs/hosts/:machineId/readings',
+ { bodyLimit: READINGS_BODY_LIMIT, preHandler: [validateParams(hostParams), validateBody(readingsBody)] },
+ (req, reply) => respond(reply, async () => ({ results: (await store.observeMachines(req.user!, req.params.machineId, req.body.readings))
+ .map(({ tabId, error }) => error === undefined ? { tabId, ok: true as const } : { tabId, error: readingError(tabId, error) }) })))
app.get<{ Params: z.infer<typeof machineParams> }>('/api/super-tabs/:tabId/machines/:machineId/removal', { preHandler: validateParams(machineParams) },
(req, reply) => respond(reply, () => store.machineRemoval(req.user!, req.params.tabId, req.params.machineId)))
app.post<{ Params: z.infer<typeof machineParams>; Body: z.infer<typeof machineObservationSchema> }>('/api/super-tabs/:tabId/machines/:machineId',
diff --git a/backend/src/server.ts b/backend/src/server.ts
index 0ee7161b5..91286b6c1 100644
--- a/backend/src/server.ts
+++ b/backend/src/server.ts
@@ -35,6 +35,7 @@ import { registerWorkContext, startQueryStats } from './lib/queryStats.js'
import { registerRouteLatency, startRouteLatency } from './lib/routeLatency.js'
import { drainAllSockets, openSocketCount, RELEASE_UPGRADE_SLOT, type SlotSocket } from './lib/wsServer.js'
import { endpointOf, refusedUpgrade, startWsCounterLog, watchUpgrade } from './lib/wsAttribution.js'
+import { startMachineReportLog } from './superTabs/machineReports.js'
// Node ≥ 15 turns an unhandled rejection into a process exit. On a cluster worker that holds thousands of
// sockets, one DB/Redis hiccup inside a fire-and-forget promise would then drop every one of them and
@@ -159,6 +160,8 @@ async function start(): Promise<void> {
useSharedRateCounter(redisWindowCounter(pub))
// Each pod's socket counts once a minute: open, connects by client, short-lived, refused (WebSocket roadmap Phase 0).
startWsCounterLog()
+ // Each pod's machine reports a minute, stored without a Swarm load (fast) or in full (DB roadmap 2.5).
+ const stopMachineReportLog = startMachineReportLog()
configureSwarmStamps(redisSwarmStamps(), { reads: env.SWARM_READ_CACHE === 'on', maxBytes: env.SWARM_CACHE_MB * 2 ** 20 })
// Hot Swarm reads name their indexes once the database is seen to have them (superTabs/parts/indexed.ts).
watchGroupIndexes(prisma)
@@ -219,6 +222,7 @@ async function start(): Promise<void> {
// The last partial minute of both logs, once the drain and the usage flush have run their queries.
stopQueryStats()
stopRouteLatency()
+ stopMachineReportLog()
// The database stays connected through the drain, for the requests still being served (lib/prisma.ts).
await Promise.allSettled([prisma.$disconnect(), closeBus()])
process.exit(exitCode)
diff --git a/backend/src/superTabs/cache.test.ts b/backend/src/superTabs/cache.test.ts
index 58ec3f03f..2757126db 100644
--- a/backend/src/superTabs/cache.test.ts
+++ b/backend/src/superTabs/cache.test.ts
@@ -1,5 +1,5 @@
-import { describe, expect, it } from 'vitest'
-import { SwarmCache } from './cache.js'
+import { afterEach, describe, expect, it, vi } from 'vitest'
+import { configureSwarmStamps, noteChange, readStamps, stampOwed, swarmTransaction, SwarmCache } from './cache.js'
import { MemoryStamps } from './cache.test-support.js'
/** A loader that counts its calls and returns a fresh object each time, numbered. */
@@ -187,3 +187,33 @@ describe('SwarmCache', () => {
expect(load.calls).toBe(2)
})
})
+
+describe('the stamps a machine report reads (DB 2.5)', () => {
+ afterEach(() => configureSwarmStamps(null))
+ /** A database whose transaction runs its body on a client the cache records changes against. */
+ const db = { $transaction: async <T>(body: (tx: object) => Promise<T>) => body({}) }
+
+ it('reads them, or none without stamps, while they fail, or late', async () => {
+ expect(await readStamps(['t1'])).toBeNull()
+ const stamps = new MemoryStamps()
+ configureSwarmStamps(stamps)
+ await stamps.bump('t1', ['main'])
+ expect((await readStamps(['t1']))?.get('t1')).toEqual({ main: 1, machine: 0 })
+ stamps.down = true
+ expect(await readStamps(['t1'])).toBeNull()
+ stamps.down = false
+ stamps.slowMs = 300
+ expect(await readStamps(['t1'])).toBeNull()
+ })
+
+ it('knows a Swarm whose change committed here before Redis took its bump, until it is paid', async () => {
+ const stamps = new MemoryStamps()
+ configureSwarmStamps(stamps)
+ stamps.down = true
+ await swarmTransaction(db as never, async tx => noteChange(tx, 't1', 'main'))
+ expect([stampOwed('t1'), stampOwed('t2')]).toEqual([true, false])
+ stamps.down = false
+ await vi.waitFor(() => expect(stampOwed('t1')).toBe(false), { timeout: 5_000 })
+ expect(stamps.bumps).toEqual([{ tabId: 't1', scopes: ['main'] }])
+ })
+})
diff --git a/backend/src/superTabs/cache.ts b/backend/src/superTabs/cache.ts
index 0bf9f75da..95ac9d7b3 100644
--- a/backend/src/superTabs/cache.ts
+++ b/backend/src/superTabs/cache.ts
@@ -243,6 +243,24 @@ export function configureSwarmStamps(next: SwarmStamps | null, options: CacheOpt
}
export const swarmReadCache = (): SwarmCache | null => readCache
+/** These Swarms' stamps now, or null without stamps, when they fail, or past STAMP_TIMEOUT_MS: a reading-only machine
+ * report then runs in full (machineReports.ts). */
+export async function readStamps(tabIds: readonly string[]): Promise<Map<string, Stamp> | null> {
+ if (!stamps) return null
+ let timer: NodeJS.Timeout | undefined
+ const late = new Promise<null>(resolve => { timer = setTimeout(() => resolve(null), STAMP_TIMEOUT_MS) })
+ try {
+ return await Promise.race([stamps.read(tabIds), late])
+ } catch {
+ return null
+ } finally {
+ clearTimeout(timer)
+ }
+}
+
+/** Whether this process committed a change of the Swarm whose bump Redis has not taken yet: its stamps say nothing new. */
+export const stampOwed = (tabId: string): boolean => owed.has(tabId) || settling.has(tabId)
+
/** The Swarms a transaction changed, by stamp scope: bumped once it commits (`swarmTransaction`). */
type Changes = Map<string, Set<StampScope>>
const changesOf = new WeakMap<object, Changes>()
@@ -271,6 +289,8 @@ export async function swarmTransaction<T>(db: Pick<PrismaClient, '$transaction'>
}
const owed: Changes = new Map()
+/** Owed bumps being paid now, out of `owed` until Redis takes them or they are owed again. */
+const settling = new Set<string>()
let paying: NodeJS.Timeout | null = null
export const OWED_RETRY_MS = 1_000
@@ -294,6 +314,11 @@ async function payOwed(): Promise<void> {
paying = null
const due = [...owed]
owed.clear()
- await announce(new Map(due))
+ for (const [tabId] of due) settling.add(tabId)
+ try {
+ await announce(new Map(due))
+ } finally {
+ for (const [tabId] of due) settling.delete(tabId)
+ }
if (!owed.size && due.length) logger.info('[swarm-cache] owed stamp bumps paid', { swarms: due.length })
}
diff --git a/backend/src/superTabs/leaderboard/lent.test.ts b/backend/src/superTabs/leaderboard/lent.test.ts
index 4bb34bdfb..de2804700 100644
--- a/backend/src/superTabs/leaderboard/lent.test.ts
+++ b/backend/src/superTabs/leaderboard/lent.test.ts
@@ -1,8 +1,8 @@
import { describe, expect, it } from 'vitest'
import { MACHINE_FRESH_MS, type ContributedMachine } from '../machines.js'
import { createSuperTab, type Agent, type Conversation, type SuperTabState, type WorkStatus } from '../model.js'
-import { accrueLent } from './lent.js'
-import { weekStart } from './week.js'
+import { accrueLent, addLent, lentAgents } from './lent.js'
+import { WEEK_MS, weekStart } from './week.js'
// Monday 2026-10-26 00:00 UTC.
const MONDAY = Date.UTC(2026, 9, 26)
@@ -100,3 +100,30 @@ describe('lent agent-hours from machine readings', () => {
expect(pool.lent?.every(l => l.busyMs === 5_000)).toBe(true)
})
})
+
+describe('lent for a report that loads no Swarm (DB 2.5)', () => {
+ /** busy works on someone else's task, idle waits for review, and a week too old to keep is on the ledger. */
+ const scene = (running: string[]) => {
+ const pool = machine(running, MONDAY)
+ pool.lent = [{ week: MONDAY - 6 * WEEK_MS, busyMs: 1 }, { week: MONDAY, busyMs: 5 }]
+ return swarm(pool, [agent('busy'), agent('idle')], [task('busy'), task('idle', 'review')])
+ }
+
+ it('adds what accrueLent adds, from the count of agents it counts', () => {
+ for (const [running, now] of [[['h-busy', 'h-idle'], MONDAY + 30_000], [['h-busy'], MONDAY + 100_000], [[], MONDAY + 10_000],
+ [['h-idle'], MONDAY + 10_000]] as const) {
+ const full = scene([...running]), fast = scene([...running])
+ accrueLent(full, full.machines![0]!, now)
+ addLent(fast.machines![0]!, now, lentAgents(fast, fast.machines![0]!, fast.machines![0]!.observation!.runningHarnessIds))
+ expect(fast.machines![0]!.lent, JSON.stringify(running)).toEqual(full.machines![0]!.lent)
+ }
+ expect(lentAgents(scene([]), scene([]).machines![0]!, ['h-busy', 'h-idle'])).toBe(1)
+ })
+
+ it('writes nothing with no time since the last reading or no agent counted', () => {
+ const state = scene(['h-busy']), pool = state.machines![0]!
+ addLent(pool, MONDAY, 1)
+ addLent(pool, MONDAY + 30_000, 0)
+ expect(pool.lent).toEqual([{ week: MONDAY - 6 * WEEK_MS, busyMs: 1 }, { week: MONDAY, busyMs: 5 }])
+ })
+})
diff --git a/backend/src/superTabs/leaderboard/lent.ts b/backend/src/superTabs/leaderboard/lent.ts
index 77b2b3653..79dfa911b 100644
--- a/backend/src/superTabs/leaderboard/lent.ts
+++ b/backend/src/superTabs/leaderboard/lent.ts
@@ -18,19 +18,30 @@ export interface LentWeek { week: number; agentMs?: number; busyMs?: number }
* reading showed running and that is working on a task someone else asked for: one nobody named counts as someone else's.
* Call it before the new reading replaces the previous one. */
export function accrueLent(state: SuperTabState, pool: ContributedMachine, now: number): void {
- const previous = pool.observation
- if (!previous) return
- const elapsed = Math.min(Math.max(0, now - previous.receivedAt), MACHINE_FRESH_MS)
- const running = new Set(previous.runningHarnessIds)
+ addLent(pool, now, lentAgents(state, pool, pool.observation?.runningHarnessIds ?? []))
+}
+
+/** How many of the machine's agents among the [running] sessions Lent counts: each working on a task someone else asked
+ * for, never a Lead's or a private chat's. A reading-only report adds time for the count its last full report recorded. */
+export function lentAgents(state: SuperTabState, pool: ContributedMachine, running: readonly string[]): number {
+ const sessions = new Set(running)
const privateChats = state.conversations.filter(c => c.audience)
const busy = new Set(state.conversations.filter(c => !c.audience && c.assignment?.status === 'working' && c.requesterId !== pool.contributorId)
.map(c => c.assignment!.agentId))
- const counted = state.agents.filter(a => a.machineId === pool.machineId && running.has(a.harnessId) && busy.has(a.id)
- && !ledProject(state, a.id) && !privateChats.some(c => c.agentIds.includes(a.id)))
- if (!elapsed || !counted.length) return
+ return state.agents.filter(a => a.machineId === pool.machineId && sessions.has(a.harnessId) && busy.has(a.id)
+ && !ledProject(state, a.id) && !privateChats.some(c => c.agentIds.includes(a.id))).length
+}
+
+/** Credits [count] agents with the time since the machine's previous reading, at most one fresh reading's worth, in the
+ * current week. Call it before the new reading replaces the previous one. */
+export function addLent(pool: ContributedMachine, now: number, count: number): void {
+ const previous = pool.observation
+ if (!previous) return
+ const elapsed = Math.min(Math.max(0, now - previous.receivedAt), MACHINE_FRESH_MS)
+ if (!elapsed || !count) return
const week = weekStart(now)
const kept = (pool.lent ?? []).filter(l => l.week > week - LEDGER_WEEKS * WEEK_MS)
const current = kept.find(l => l.week === week) ?? kept[kept.push({ week }) - 1]!
- current.busyMs = (current.busyMs ?? 0) + elapsed * counted.length
+ current.busyMs = (current.busyMs ?? 0) + elapsed * count
pool.lent = kept
}
diff --git a/backend/src/superTabs/liveness.test.ts b/backend/src/superTabs/liveness.test.ts
index fff3ae978..425a69287 100644
--- a/backend/src/superTabs/liveness.test.ts
+++ b/backend/src/superTabs/liveness.test.ts
@@ -1,6 +1,6 @@
import { describe, expect, it } from 'vitest'
import type { AgentStart, Assignment, SuperTabState } from './model.js'
-import { LEASE_MS, QUIET_RUNNING_MS, sweepForLead, sweepLiveness } from './liveness.js'
+import { LEASE_MS, loseTasks, nextLossAt, QUIET_RUNNING_MS, sweepForLead, sweepLiveness } from './liveness.js'
import { MACHINE_FRESH_MS, startPreparing } from './machines.js'
import { split } from './parts/index.js'
@@ -427,3 +427,25 @@ describe('the state of each task, and the Lead woken when one stalls', () => {
expect(task(state).liveness).toBeUndefined()
})
})
+
+describe('the next time a machine\'s worker can lose its task (DB 2.5)', () => {
+ const progress = 1_000_000
+ /** Its machine reported at [at], its session's screen last changed at `progress`. */
+ const reported = (at: number, task = assignment({ leaseUntil: 1 })) => swarm(task, { receivedAt: at, harnessProgress: { h: progress } })
+ const ownLoss = (state: SuperTabState, now: number) => (loseTasks(state, now, worker => worker.machineId === 'm'), state.conversations[1]!.assignment!.status)
+
+ it('is when its lease ends, so a report up to then loses nothing and the first after it loses the task', () => {
+ const at = nextLossAt(reported(progress + 26 * MINUTE), 'm')!
+ expect(at).toBeGreaterThan(progress)
+ expect(ownLoss(reported(at), at)).toBe('working')
+ expect(ownLoss(reported(at + 1), at + 1)).toBe('lost')
+ })
+
+ it('is none for another machine, a task waiting for review, or no task', () => {
+ expect(nextLossAt(reported(progress), 'other')).toBeUndefined()
+ expect(nextLossAt(reported(progress, assignment({ status: 'review', result: 'Ready for review', leaseUntil: 1 })), 'm')).toBeUndefined()
+ const idle = reported(progress)
+ idle.conversations[1]!.assignment = null
+ expect(nextLossAt(idle, 'm')).toBeUndefined()
+ })
+})
diff --git a/backend/src/superTabs/liveness.ts b/backend/src/superTabs/liveness.ts
index 72bea5620..a79c0a85b 100644
--- a/backend/src/superTabs/liveness.ts
+++ b/backend/src/superTabs/liveness.ts
@@ -143,6 +143,15 @@ function limitReason(worker: Agent, now: number): string | null {
? `it hit a usage limit until ${worker.limitedUntil.slice(11, 16)} UTC` : null
}
+/** The earliest time a task held by a worker on [machineId] may be lost to its lease running out, as lostReason reads it now,
+ * or undefined when none can. Reports that keep its session running with progress no older can only move it later. */
+export function nextLossAt(state: SuperTabState, machineId: string): number | undefined {
+ const ends = workerTasks(state).filter(({ worker, assignment }) => worker.machineId === machineId && !waitingForReview(assignment))
+ .map(({ group, assignment, worker }) => leaseEnd(group, assignment, reportRenewedAt(state, worker)))
+ .filter((end): end is number => end !== undefined)
+ return ends.length ? Math.min(...ends) : undefined
+}
+
/** Why a worker has lost its task now, or null while it still holds it. Only a machine that has reported can be offline. */
function lostReason(state: SuperTabState, group: Conversation, worker: Agent, assignment: Assignment, now: number): string | null {
const pool = state.machines?.find(m => m.machineId === worker.machineId)
diff --git a/backend/src/superTabs/machineReports.test.ts b/backend/src/superTabs/machineReports.test.ts
new file mode 100644
index 000000000..cb60c9029
--- /dev/null
+++ b/backend/src/superTabs/machineReports.test.ts
@@ -0,0 +1,94 @@
+import { describe, expect, it, vi } from 'vitest'
+import { countReport, localReportMemory, machineReportLine, RECORD_TTL_MS, redisReportMemory, type ReportRecord } from './machineReports.js'
+import { SWEEP_EVERY_MS } from './liveness.js'
+
+/** The Redis commands the memory uses, with Redis' own expiry, on a clock the test moves; `down` fails every command. */
+function fakeRedis() {
+ let now = 0
+ const keys = new Map<string, { value: string | Map<string, string>; until: number }>()
+ const live = (key: string) => { const entry = keys.get(key); return entry && entry.until > now ? entry : undefined }
+ const redis = {
+ down: false,
+ tick: (ms: number) => { now += ms },
+ check() { if (redis.down) throw new Error('Redis is unreachable') },
+ async hget(key: string, field: string) { redis.check(); const entry = live(key); return entry ? (entry.value as Map<string, string>).get(field) ?? null : null },
+ async hset(key: string, field: string, value: string) {
+ redis.check()
+ const entry = live(key) ?? { value: new Map<string, string>(), until: Infinity }
+ ;(entry.value as Map<string, string>).set(field, value)
+ keys.set(key, entry)
+ return 1
+ },
+ async pexpire(key: string, ms: number) { redis.check(); const entry = live(key); if (entry) entry.until = now + ms; return entry ? 1 : 0 },
+ async set(key: string, value: string, px: 'PX', ms: number, nx: 'NX') {
+ redis.check()
+ if (px !== 'PX' || nx !== 'NX') throw new Error('expected PX and NX')
+ if (live(key)) return null
+ keys.set(key, { value, until: now + ms })
+ return 'OK' as const
+ },
+ }
+ return redis
+}
+
+const recordAt = (at: number): ReportRecord => ({ stamp: 7, lentCount: 2, removals: false, deadline: at + 60_000, team: false, members: ['p1'],
+ preparable: ['s1'], at })
+
+describe('what a full machine report leaves for the next one (DB 2.5)', () => {
+ it('reads back what was recorded, per machine, until two minutes after the Swarm\'s last record', async () => {
+ const redis = fakeRedis(), memory = redisReportMemory(redis)
+ await memory.set('t1', 'm1', recordAt(1))
+ expect(await memory.get('t1', 'm1')).toEqual(recordAt(1))
+ expect(await memory.get('t1', 'm2')).toBeUndefined()
+ redis.tick(RECORD_TTL_MS - 1)
+ await memory.set('t1', 'm2', recordAt(2))
+ redis.tick(RECORD_TTL_MS - 1)
+ expect(await memory.get('t1', 'm1')).toEqual(recordAt(1))
+ redis.tick(1)
+ expect(await memory.get('t1', 'm1')).toBeUndefined()
+ })
+
+ it('reads a record it cannot parse, or one Redis cannot give, as missing', async () => {
+ const redis = fakeRedis(), memory = redisReportMemory(redis)
+ await redis.hset('swarm:machine-full:t1', 'm1', '{"stamp":"x"}')
+ expect(await memory.get('t1', 'm1')).toBeUndefined()
+ await memory.set('t1', 'm1', recordAt(1))
+ redis.down = true
+ expect(await memory.get('t1', 'm1')).toBeUndefined()
+ await expect(memory.set('t1', 'm1', recordAt(2))).resolves.toBeUndefined()
+ })
+
+ it('sweeps a Swarm once per 30 seconds between every server sharing Redis', async () => {
+ const redis = fakeRedis(), a = redisReportMemory(redis), b = redisReportMemory(redis)
+ expect([await a.sweepDue('t1', 0), await b.sweepDue('t1', 0), await a.sweepDue('t2', 0)]).toEqual([true, false, true])
+ redis.tick(SWEEP_EVERY_MS - 1)
+ expect(await b.sweepDue('t1', SWEEP_EVERY_MS - 1)).toBe(false)
+ redis.tick(1)
+ expect([await b.sweepDue('t1', SWEEP_EVERY_MS), await a.sweepDue('t1', SWEEP_EVERY_MS)]).toEqual([true, false])
+ })
+
+ it('keeps the sweep clock on this server alone while Redis fails', async () => {
+ const redis = fakeRedis(), memory = redisReportMemory(redis)
+ redis.down = true
+ expect([await memory.sweepDue('t1', 0), await memory.sweepDue('t1', SWEEP_EVERY_MS - 1), await memory.sweepDue('t1', SWEEP_EVERY_MS)])
+ .toEqual([true, false, true])
+ })
+
+ it('remembers alone without Redis, records and sweeps both', async () => {
+ let now = 0
+ const memory = localReportMemory(() => now)
+ await memory.set('t1', 'm1', recordAt(0))
+ expect(await memory.get('t1', 'm1')).toEqual(recordAt(0))
+ now = RECORD_TTL_MS
+ expect(await memory.get('t1', 'm1')).toBeUndefined()
+ expect([await memory.sweepDue('t1', 0), await memory.sweepDue('t1', 1)]).toEqual([true, false])
+ })
+
+ it('counts fast and full reports for the minute line, and starts again after each', () => {
+ machineReportLine(60_000)
+ countReport('fast'); countReport('fast'); countReport('full')
+ expect(machineReportLine(60_000)).toMatchObject({ windowMs: 60_000, fast: 2, full: 1, pod: expect.any(String) })
+ expect(machineReportLine(60_000)).toMatchObject({ fast: 0, full: 0 })
+ vi.restoreAllMocks()
+ })
+})
diff --git a/backend/src/superTabs/machineReports.ts b/backend/src/superTabs/machineReports.ts
new file mode 100644
index 000000000..3c4502922
--- /dev/null
+++ b/backend/src/superTabs/machineReports.ts
@@ -0,0 +1,155 @@
+/**
+ * What a machine's full report leaves for the next one (DB roadmap 2.5, docs/superpowers/specs/
+ * 2026-10-30-machine-reports-fast-path-design.md): a heartbeat whose decisions cannot differ stores its reading without
+ * loading the Swarm, from what the last full report recorded here. Also the Swarm's sweep clock, one for every server, and
+ * this server's count of fast and full reports, logged each minute.
+ */
+import { z } from 'zod'
+import { INSTANCE_ID } from '../lib/instance.js'
+import { logger } from '../utils/logger.js'
+import { SWEEP_EVERY_MS } from './liveness.js'
+
+/** A record is read again only within this long of its Swarm's last full report. */
+export const RECORD_TTL_MS = 120_000
+/** The latest a fast report may follow its full one, whatever its workers' leases say. */
+export const DEADLINE_CAP_MS = 300_000
+
+/** What a full report saw, for the machine's next reports. */
+export interface ReportRecord {
+ /** The Swarm's `main` stamp read before the full report loaded it. */
+ stamp: number
+ /** This machine's agents Lent counts for the next report (leaderboard/lent.ts lentAgents). */
+ lentCount: number
+ /** Whether any machine of the Swarm was being removed (machines.ts expireRemovals). */
+ removals: boolean
+ /** The earliest a worker here may lose its task to time (liveness.ts nextLossAt), at most DEADLINE_CAP_MS on. */
+ deadline: number
+ /** Whether the Swarm's record says the machine is a Team account's (store.ts teamByRecord). */
+ team: boolean
+ /** Who is told of a reading (parts/commit.ts membersToTell): the Swarm's people. */
+ members: string[]
+ /** The starts this machine may say it is preparing (machines.ts preparableStart). */
+ preparable: string[]
+ /** When the full report ran. */
+ at: number
+}
+
+export interface ReportMemory {
+ get(tabId: string, machineId: string): Promise<ReportRecord | undefined>
+ set(tabId: string, machineId: string, record: ReportRecord): Promise<void>
+ /** Whether a report now runs the Swarm's time checks: yes at most once per SWEEP_EVERY_MS, whichever server asks. */
+ sweepDue(tabId: string, now: number): Promise<boolean>
+}
+
+/** The Redis commands the memory needs: the bus' fail-fast connection (lib/bus.ts `pub`) in production. */
+export interface ReportRedis {
+ hget(key: string, field: string): Promise<string | null>
+ hset(key: string, field: string, value: string): Promise<unknown>
+ pexpire(key: string, ms: number): Promise<unknown>
+ set(key: string, value: string, px: 'PX', ms: number, nx: 'NX'): Promise<string | null>
+}
+
+const record = z.object({ stamp: z.number().finite(), lentCount: z.number().int().nonnegative(), removals: z.boolean(), deadline: z.number().finite(),
+ team: z.boolean(), members: z.array(z.string()), preparable: z.array(z.string()), at: z.number().finite() })
+
+function parse(value: string | null): ReportRecord | undefined {
+ if (value === null) return undefined
+ try {
+ const read = record.safeParse(JSON.parse(value))
+ return read.success ? read.data : undefined
+ } catch {
+ return undefined
+ }
+}
+
+/** One server's memory: the default, and the sweep clock while Redis fails. */
+export function localReportMemory(now: () => number = Date.now): ReportMemory {
+ const records = new Map<string, { until: number; machines: Map<string, ReportRecord> }>()
+ const swept = new Map<string, number>()
+ return {
+ async get(tabId, machineId) {
+ const held = records.get(tabId)
+ if (!held || held.until <= now()) return undefined
+ return held.machines.get(machineId)
+ },
+ async set(tabId, machineId, value) {
+ const held = records.get(tabId)
+ const machines = held && held.until > now() ? held.machines : new Map<string, ReportRecord>()
+ records.set(tabId, { until: now() + RECORD_TTL_MS, machines: machines.set(machineId, value) })
+ for (const [id, entry] of records) if (entry.until <= now()) records.delete(id)
+ },
+ async sweepDue(tabId, at) {
+ const last = swept.get(tabId)
+ if (last !== undefined && at - last < SWEEP_EVERY_MS) return false
+ swept.set(tabId, at)
+ for (const [id, when] of swept) if (at - when >= SWEEP_EVERY_MS) swept.delete(id)
+ return true
+ },
+ }
+}
+
+/**
+ * The memory every server shares, in Redis: `swarm:machine-full:{tabId}` holds one field per machine and expires
+ * RECORD_TTL_MS after its last write; `swarm:swept:{tabId}` is the sweep clock. A record Redis cannot give reads as missing
+ * (that report runs in full); a sweep clock Redis cannot keep is kept by this server alone, as before.
+ */
+export function redisReportMemory(redis: ReportRedis, now: () => number = Date.now): ReportMemory {
+ const local = localReportMemory(now)
+ const key = (tabId: string) => `swarm:machine-full:${tabId}`
+ let failing = false
+ const failed = (what: string, error: unknown) => {
+ if (!failing) logger.warn('[machine-reports] Redis failed; reports run in full and sweeps are timed here until it answers', { what, error: String(error) })
+ failing = true
+ }
+ return {
+ async get(tabId, machineId) {
+ try {
+ const found = parse(await redis.hget(key(tabId), machineId))
+ failing = false
+ return found
+ } catch (error) {
+ failed('get', error)
+ return undefined
+ }
+ },
+ async set(tabId, machineId, value) {
+ try {
+ await redis.hset(key(tabId), machineId, JSON.stringify(value))
+ await redis.pexpire(key(tabId), RECORD_TTL_MS)
+ } catch (error) {
+ failed('set', error)
+ }
+ },
+ async sweepDue(tabId, at) {
+ try {
+ return await redis.set(`swarm:swept:${tabId}`, String(at), 'PX', SWEEP_EVERY_MS, 'NX') === 'OK'
+ } catch (error) {
+ failed('sweep', error)
+ return local.sweepDue(tabId, at)
+ }
+ },
+ }
+}
+
+const counts = { fast: 0, full: 0 }
+
+/** One machine reading stored, by which path. */
+export function countReport(path: 'fast' | 'full'): void {
+ counts[path]++
+}
+
+/** This server's counts since the last line, reset. */
+export function machineReportLine(windowMs: number): { pod: string; windowMs: number; fast: number; full: number } {
+ const line = { pod: INSTANCE_ID, windowMs, ...counts }
+ counts.fast = 0
+ counts.full = 0
+ return line
+}
+
+/** Logs `machine reports` every `intervalMs` while there were any; the stop logs the last partial window. */
+export function startMachineReportLog(intervalMs = 60_000): () => void {
+ const flush = () => { if (counts.fast || counts.full) logger.info('machine reports', machineReportLine(intervalMs)) }
+ const timer = setInterval(flush, intervalMs)
+ timer.unref()
+ return () => { clearInterval(timer); flush() }
+}
diff --git a/backend/src/superTabs/machines.test.ts b/backend/src/superTabs/machines.test.ts
index c83c78bfd..0fe589c6b 100644
--- a/backend/src/superTabs/machines.test.ts
+++ b/backend/src/superTabs/machines.test.ts
@@ -1,6 +1,7 @@
import { describe, expect, it } from 'vitest'
import { createSuperTab, SuperTabError, type AgentStart, type AgentStartRequest, type SuperTabState } from './model.js'
-import { admitMachineStart, machineSlots, machineObservationSchema, servedLocalModels, type ContributedMachine } from './machines.js'
+import { admitMachineStart, machineSlots, machineObservationSchema, preparableStart, readingOnly, servedLocalModels, storedObservation, type ContributedMachine,
+ type MachineObservation } from './machines.js'
const NOW = 1_000_000
const requirements = { memoryBytes: 1, diskBytes: 1, maxCpuPercent: 50, tools: ['git'], devices: [], quotaRequired: false }
@@ -200,3 +201,63 @@ it('keeps durable release receipts authoritative over stale observations and ret
f.state.harnessReservations = [{ machineId: 'host', harnessId: 'finished', generation: 'new-runtime', renewed: true }]
expect(machineSlots(f.state, pool)).toBe(1)
})
+
+describe('a machine\'s reading as a report stores it (DB 2.5)', () => {
+ const { receivedAt: _receivedAt, ...sent } = observation
+ const pool = (extra: Partial<ContributedMachine> = {}): ContributedMachine => ({ machineId: 'host', name: 'Mac', contributorId: 'owner',
+ membershipGeneration: 'tab', generation: 'g1', cwd: '/x', engines: ['claude'], maxAgents: 2, paused: false, ...extra })
+ const quota = (engine: string) => ({ engine, known: true, windows: [] })
+ const tokens = { opus: { input: 1, output: 2, cacheRead: 0, cacheWrite: 0 } }
+ const reading = (extra: Partial<MachineObservation> = {}): MachineObservation => ({ ...sent, runningHarnessIds: ['a'], stoppedHarnessIds: ['z'],
+ ...extra } as MachineObservation)
+ const stored = (extra: Partial<MachineObservation> = {}, at = NOW, preparable = (_id: string) => true) =>
+ storedObservation(pool(), reading(extra), { team: false, now: at, preparable })
+
+ it('keeps what a full report keeps: running sessions\' progress and tokens, its own preparing starts, contributed engines\' quotas', () => {
+ const kept = storedObservation(pool(), reading({ harnessProgress: { a: 5, b: 6 }, harnessTokens: { a: tokens, b: tokens },
+ preparingStartIds: ['s1', 's2'], quotas: [quota('claude'), quota('codex')], privacy: { mode: 'dedicated' } }),
+ { team: false, now: NOW, preparable: id => id === 's1' })
+ expect(kept).toMatchObject({ receivedAt: NOW, sessionKind: 'computer', harnessProgress: { a: 5 }, harnessTokens: { a: tokens },
+ preparingStartIds: ['s1'], runningHarnessIds: ['a'], stoppedHarnessIds: ['z'] })
+ expect(kept.quotas.map(q => q.engine)).toEqual(['claude'])
+ // Only a team session's privacy check is kept.
+ expect(kept.privacy).toBeUndefined()
+ const team = storedObservation(pool(), reading({ privacy: { mode: 'dedicated' } }), { team: true, now: NOW, preparable: () => true })
+ expect(team).toMatchObject({ sessionKind: 'team', privacy: { mode: 'dedicated', seenAt: NOW } })
+ })
+
+ it('names a start preparing only when this machine launched it and it neither runs nor failed', () => {
+ const start = (extra: Record<string, unknown>) => ({ id: 's', request: { machineId: 'host' }, dispatched: true, ...extra }) as unknown as AgentStart
+ expect(preparableStart(start({}), 'host')).toBe(true)
+ expect(preparableStart(start({}), 'other')).toBe(false)
+ expect(preparableStart(start({ dispatched: false }), 'host')).toBe(false)
+ expect(preparableStart(start({ harnessId: 'h' }), 'host')).toBe(false)
+ expect(preparableStart(start({ error: 'failed' }), 'host')).toBe(false)
+ })
+
+ it('reads as the reading alone only when every decision it feeds is the same', () => {
+ const before = stored({ harnessProgress: { a: 5 }, preparingStartIds: ['s1'] })
+ const same = (extra: Partial<MachineObservation>) => readingOnly(before, stored({ harnessProgress: { a: 5 }, preparingStartIds: ['s1'], ...extra }, NOW + 15_000))
+ // The reading alone: resources, quotas, tools, tokens, privacy, a later progress time, the same sets in another order.
+ expect(same({})).toBe(true)
+ expect(same({ resources: { ...sent.resources, cpuPercent: 90 }, tools: { git: true }, harnessTokens: { a: tokens } })).toBe(true)
+ expect(same({ harnessProgress: { a: 9 } })).toBe(true)
+ expect(readingOnly(stored({ runningHarnessIds: ['a', 'b'] }), stored({ runningHarnessIds: ['b', 'a'] }))).toBe(true)
+ // A decision may differ.
+ expect(same({ runningHarnessIds: ['a', 'b'] })).toBe(false)
+ expect(same({ stoppedHarnessIds: [] })).toBe(false)
+ expect(same({ stoppedReservations: [{ harnessId: 'z', generation: 'g1' }] })).toBe(false)
+ expect(same({ preparingStartIds: [] })).toBe(false)
+ // Progress renews a worker's lease (liveness.ts reportRenewedAt): a map added or gone, an entry gone or earlier.
+ expect(same({ harnessProgress: undefined })).toBe(false)
+ expect(same({ harnessProgress: {} })).toBe(false)
+ expect(same({ harnessProgress: { a: 4 } })).toBe(false)
+ expect(readingOnly(stored({}), stored({ harnessProgress: { a: 5 } }))).toBe(false)
+ expect(readingOnly(undefined, before)).toBe(false)
+ // Sessions are compared as sets: a session repeated is not one that stayed.
+ expect(readingOnly(stored({ runningHarnessIds: ['a', 'b'] }), stored({ runningHarnessIds: ['a', 'a'] }))).toBe(false)
+ // A reading from the other kind of session decides who may host the machine (store.ts teamByRecord).
+ const team = storedObservation(pool(), reading({ harnessProgress: { a: 5 }, preparingStartIds: ['s1'] }), { team: true, now: NOW + 15_000, preparable: () => true })
+ expect(readingOnly(before, team)).toBe(false)
+ })
+})
diff --git a/backend/src/superTabs/machines.ts b/backend/src/superTabs/machines.ts
index 27490b9cf..c4fc1e23a 100644
--- a/backend/src/superTabs/machines.ts
+++ b/backend/src/superTabs/machines.ts
@@ -159,6 +159,54 @@ export type MachineObservation = z.infer<typeof machineObservationSchema>
/** A machine reports every 15 seconds (cli/src/superTabs/machines.ts); three missed beats is offline. */
export const MACHINE_FRESH_MS = 45_000
+/** A machine's reading as its pool keeps it. */
+export type StoredReading = NonNullable<ContributedMachine['observation']>
+
+/** A start this machine may say it is preparing: one it launched that neither runs nor failed (liveness.ts). */
+export const preparableStart = (start: AgentStart, machineId: string): boolean =>
+ start.request.machineId === machineId && !!start.dispatched && !start.harnessId && !start.error
+
+/**
+ * What a report stores of [observation] on [pool]: progress and tokens of running sessions only (leases and attempts read
+ * them), the preparing starts [preparable] allows, the contributed engines' quotas, privacy only from a team session
+ * (observedPrivacy), and how the machine is signed in, from the session and never from the report (lib/teamSession.ts).
+ * A full report and a reading-only one (DB 2.5) store the same.
+ */
+export function storedObservation(pool: ContributedMachine, observation: MachineObservation,
+ opts: { team: boolean; now: number; preparable: (startId: string) => boolean }): StoredReading {
+ const { privacy: reported, harnessProgress, harnessTokens, preparingStartIds, ...reading } = observation
+ const running = (id: string) => observation.runningHarnessIds.includes(id)
+ const progress = harnessProgress && Object.fromEntries(Object.entries(harnessProgress).filter(([id]) => running(id)))
+ const tokens = harnessTokens && Object.fromEntries(Object.entries(harnessTokens).filter(([id]) => running(id)))
+ const preparing = preparingStartIds?.filter(opts.preparable)
+ const privacy = observedPrivacy(pool.observation?.privacy, reported, opts.team, opts.now)
+ return { ...reading, quotas: observation.quotas.filter(q => contributedEngines(pool, observation).includes(q.engine)), receivedAt: opts.now,
+ sessionKind: opts.team ? 'team' : 'computer', ...(privacy ? { privacy } : {}), ...(progress ? { harnessProgress: progress } : {}),
+ ...(tokens ? { harnessTokens: tokens } : {}), ...(preparing ? { preparingStartIds: preparing } : {}) }
+}
+
+const sameSet = (a: readonly string[] = [], b: readonly string[] = []) => {
+ const left = new Set(a), right = new Set(b)
+ return left.size === right.size && [...left].every(id => right.has(id))
+}
+
+/**
+ * Whether [next] changes nothing a report decides from, against [previous] (DB 2.5 spec, section 1 condition 2): the same
+ * running and stopped sessions, no stop proof, the same preparing starts. Progress renews leases (liveness.ts
+ * reportRenewedAt), so it also keeps its shape, and no session's progress goes back: a lease then ends no sooner than the
+ * last full report saw.
+ */
+export function readingOnly(previous: StoredReading | undefined, next: StoredReading): boolean {
+ // How the machine is signed in decides who may host it (store.ts teamByRecord reads the stored one): a reading from the
+ // other kind of session runs in full, so the full report's refusal is never skipped (review of #2378).
+ if (!previous || next.stoppedReservations?.length || previous.sessionKind !== next.sessionKind) return false
+ if (!sameSet(previous.runningHarnessIds, next.runningHarnessIds) || !sameSet(previous.stoppedHarnessIds, next.stoppedHarnessIds)
+ || !sameSet(previous.preparingStartIds, next.preparingStartIds)) return false
+ if (!previous.harnessProgress || !next.harnessProgress) return !previous.harnessProgress && !next.harnessProgress
+ const before = previous.harnessProgress, after = next.harnessProgress
+ return sameSet(Object.keys(before), Object.keys(after)) && Object.entries(before).every(([id, at]) => after[id]! >= at)
+}
+
/** Whether [start] is preparing now: launched, neither running nor failed, and named by a fresh report of its own machine.
* Read from that report, never written into the start, so a report a minute per start writes nothing more. */
export function startPreparing(state: SuperTabState, start: AgentStart, now: number): boolean {
diff --git a/backend/src/superTabs/parts/index.ts b/backend/src/superTabs/parts/index.ts
index 632e32e70..47cfde4a7 100644
--- a/backend/src/superTabs/parts/index.ts
+++ b/backend/src/superTabs/parts/index.ts
@@ -7,6 +7,6 @@ export * from './types.js'
export { clock, createClock } from './clock.js'
export { knownKind, reassemble, split, toLegacyState } from './shape.js'
-export { DORMANT_PAGE, insertParts, lastChangedAt, loadActive, loadAll, loadChanged, loadDormant, loadGroups, loadMachines, loadObserved, loadReadBase, loadStartsOfAgents } from './load.js'
+export { DORMANT_PAGE, insertParts, lastChangedAt, loadActive, loadAll, loadChanged, loadDormant, loadGroups, loadMachineParts, loadMachines, loadObserved, loadReadBase, loadStartsOfAgents } from './load.js'
export { loadLight, loadLightGroups } from './light.js'
export { guardedWrite, PartConflict, type PartWrite } from './write.js'
diff --git a/backend/src/superTabs/parts/load.ts b/backend/src/superTabs/parts/load.ts
index 76de6c052..5a5f17e84 100644
--- a/backend/src/superTabs/parts/load.ts
+++ b/backend/src/superTabs/parts/load.ts
@@ -93,6 +93,15 @@ export async function loadObserved(tx: Transaction, tabId: string, machineId: st
return { ...loaded, groups: rows.map(row => toPart<GroupData>('group', row)) }
}
+/** Machine parts by id (`${tabId}:${machineId}`), with their version, in one read by `_id`: what a reading-only report reads
+ * (DB 2.5, superTabs/machineReports.ts). Raw, so many ids are one plain `$in`. */
+export async function loadMachineParts(tx: Transaction, ids: string[]): Promise<Map<string, { version: number; deleted: boolean; data: MachineData }>> {
+ if (!ids.length) return new Map()
+ const rows = await tx.swarmMachine.aggregateRaw({ pipeline: [{ $match: { _id: { $in: ids } } }, { $project: { version: 1, deleted: 1, data: 1 } }] }) as
+ unknown as Array<{ _id: string; version: number; deleted: boolean; data: MachineData }>
+ return new Map(rows.map(row => [row._id, { version: Number(row.version), deleted: !!row.deleted, data: row.data }]))
+}
+
/** What the read cache keeps of a Swarm at its main stamp (cache.ts): `loadActive`'s parts with every live start, and no
* machines, which it keeps at their own stamp (`loadMachines`). Dormant groups and their like come by name (`loadGroups`). */
export function loadReadBase(tx: Transaction, tabId: string): Promise<SwarmParts> {
diff --git a/backend/src/superTabs/retry.ts b/backend/src/superTabs/retry.ts
index c210da1f7..7f6273173 100644
--- a/backend/src/superTabs/retry.ts
+++ b/backend/src/superTabs/retry.ts
@@ -8,7 +8,8 @@ const EXPIRED_TRANSACTION_ATTEMPTS = 2
const TRANSACTION_BACKOFF_MS = 50
const TRANSACTION_BACKOFF_CAP_MS = 1_500
const errorCode = (error: unknown): unknown => error && typeof error === 'object' && 'code' in error ? error.code : null
-function retryable(error: unknown): boolean {
+/** A transaction's failure that another attempt may not meet: a write conflict, a taken unique key, Mongo's transient ones. */
+export function retryable(error: unknown): boolean {
if (!error || typeof error !== 'object') return false
return ['P2034', 'P2002', 'P2028'].includes(errorCode(error) as string) || (error instanceof Error && error.message.includes('TransientTransactionError'))
}
diff --git a/backend/src/superTabs/store.ts b/backend/src/superTabs/store.ts
index a3ba7b053..f7d73aef3 100644
--- a/backend/src/superTabs/store.ts
+++ b/backend/src/superTabs/store.ts
@@ -1,4 +1,4 @@
-import { contributedEngines, expireRemovals, LASTING_TEAM_REFUSALS, machineContribution, machineObservationSchema, machineSlots, machineTeamWork, observedPrivacy, recordAgentPresence, removalPreview, startPreparing, swarmGridRow, teamAccountRequired, teamWorkRefusal, type MachineObservation } from './machines.js'
+import { expireRemovals, LASTING_TEAM_REFUSALS, machineContribution, machineObservationSchema, machineSlots, machineTeamWork, preparableStart, readingOnly, recordAgentPresence, removalPreview, startPreparing, storedObservation, swarmGridRow, teamAccountRequired, teamWorkRefusal, type MachineObservation } from './machines.js'
import { readingClearsSuspensions, suspendedEngines } from './engineHealth.js'
import { BUDDY_OWN_MACHINE_REFUSAL } from './buddyContract.js'
import { createHash } from 'node:crypto'
@@ -32,16 +32,18 @@ import { confidentialSessions, otherRegistrations, recordConfidential } from './
import { answersRequest, dirtyGroups, noteProjectReports, projectUpdateRequest, UNANSWERED_REQUEST_MS } from './projectUpdates.js'
import { cloneData } from './cloneData.js'
import { digestDue, digestMessageId, digestRequest, digestSince } from './digest.js'
-import { judgedMachines, loseTasks, sweepForLead, sweepLiveness, SWEEP_EVERY_MS, unpurgedNotice } from './liveness.js'
+import { judgedMachines, loseTasks, nextLossAt, sweepForLead, sweepLiveness, SWEEP_EVERY_MS, unpurgedNotice } from './liveness.js'
import { archivable, archiveLine, ASLEEP_FIELDS, asleepMatch, reviewClosable, reviewCloseLine, reviewPullRequests, stopIdleWorkers, stoppedMachines, type GroupActivity } from './tidy.js'
-import { clock, insertParts, loadActive, loadLightGroups, loadObserved, reassemble, split, type GroupData, type HeadData, type Part, type SwarmParts } from './parts/index.js'
+import { clock, insertParts, loadActive, loadLightGroups, loadMachineParts, loadObserved, reassemble, split, type GroupData, type HeadData, type MachineData, type Part, type SwarmParts } from './parts/index.js'
import { groupsBy, listingDrops, quietSince, readBy, withoutHints } from './parts/indexed.js'
import { memberSwarms } from './swarmRows.js'
import { commitParts, indexesOf, numberMessages, type Touch, type Written } from './parts/commit.js'
import { loadDeliveryParts } from './parts/light.js'
-import { guardedWrite } from './parts/write.js'
+import { guardedWrite, partId } from './parts/write.js'
+import { retryable } from './retry.js'
import { touchesFor } from './parts/touches.js'
-import { freeze, swarmReadCache, swarmTransaction, type CacheView, type SwarmCache } from './cache.js'
+import { freeze, readStamps, stampOwed, swarmReadCache, swarmTransaction, type CacheView, type SwarmCache } from './cache.js'
+import { countReport, DEADLINE_CAP_MS, localReportMemory, RECORD_TTL_MS, type ReportMemory, type ReportRecord } from './machineReports.js'
import { DIRECT, readLastChangedAt, readParts, type ReadNamed } from './cachedReads.js'
import { revokeNativeContexts, SuperTabNativeStore } from './native.js'
import { activeMember, addMember, hasWorkspaceAccess, listWorkspaces, memberId, WORKSPACE_ACCESS_REVOKED, workspaceForNewSwarm, requireWorkspaceAccess, requireWorkspaceRole, SwarmWorkspaces } from './workspaceStore.js'
@@ -54,7 +56,7 @@ import type { HoldReason } from './holdReason.js'
import { stuck } from './deliverySettle.js'
import { agentDisplayName } from './agentNames.js'
import { endedHolds, HOLDING, startedHolds, type WorkHold } from '../github/holds.js'
-import { accrueLent } from './leaderboard/lent.js'
+import { accrueLent, addLent, lentAgents } from './leaderboard/lent.js'
import { mayDrive, TERMINAL_ANSWER_MS, TERMINAL_EXPIRY_MS, TERMINAL_PICKUP_MS, TERMINAL_RECHECK_MS, type TerminalEffect } from './terminal.js'
import { costRows, failedStarts, summarizeCosts } from './costs/read.js'
import { APPROVE, APPROVED_DAYS, approvedReminder, LAPSED_DAYS, localDate, MAX_PROPOSALS, proposalQuestion, proposalTitle, REASK_DAYS, reflectionMessageId, routingReflection, type Approved, type Proposal, type ReflectionInput } from './costs/reflection.js'
@@ -290,6 +292,9 @@ export interface SuperTabStoreHooks {
/** When each caller's full delivery poll last succeeded, shared by every server (Redis in production, pollMemory.ts). Left
* out, this server remembers alone. */
polls?: PollMemory
+ /** What each machine's full report saw, for its next reports, and each Swarm's sweep clock, shared by every server (Redis
+ * in production, machineReports.ts). Left out, this server remembers alone. */
+ reports?: ReportMemory
/** A delivery now waits for this agent, committed: its machine's daemon socket tells its host to read it at once
* (lib/swarmDeliveryPushes.ts). Called after the write commits; it must not throw. */
deliveriesWaiting?: (waiting: DeliveriesWaiting) => void
@@ -301,6 +306,11 @@ export interface SuperTabStoreHooks {
swarmChanged?: (change: SwarmChanged) => void
}
+/** A report call's host checks that need no Swarm (SuperTabStore.machineHost): the caller's account row for the machine. */
+export interface MachineHost { owned: Machine | null }
+/** A machine part as a reading-only report reads it (parts/load.ts loadMachineParts). */
+export interface MachinePart { version: number; deleted: boolean; data: MachineData }
+
/** Which people's apps should read a Swarm again, the commit's time, and which of its stamps moved. */
export interface SwarmChanged { tabId: string; userIds: string[]; lastChangedAt: number; parts: Array<'main' | 'machine'> }
@@ -476,6 +486,7 @@ export class SuperTabStore {
constructor(readonly db: PrismaClient, readonly now: () => number = Date.now,
private readonly attachments: Attachments | null = null, private readonly hooks: SuperTabStoreHooks = {}) {
this.polls = hooks.polls ?? localPollMemory(now)
+ this.reports = hooks.reports ?? localReportMemory(now)
this.native = new SuperTabNativeStore({
transaction: body => transaction(db, body),
load: (tx, user, tabId, groupIds) => this.load(tx, user, tabId, { groupIds }),
@@ -2401,23 +2412,106 @@ export class SuperTabStore {
})
}
- /** When each Swarm's time checks last ran on a machine's report, on this server (liveness.ts SWEEP_EVERY_MS). */
- private readonly swept = new Map<string, number>()
+ private readonly reports: ReportMemory
- /** Whether a machine's report runs its Swarm's time checks now; decided once per report, not per transaction attempt. */
- private sweepDue(tabId: string, now: number): boolean {
- const last = this.swept.get(tabId)
- if (last !== undefined && now - last < SWEEP_EVERY_MS) return false
- this.swept.set(tabId, now)
- return true
+ /** The caller's own account row for the machine and the host checks that need no Swarm: once per report call, which a
+ * batch of readings shares (DB 2.5). */
+ async machineHost(user: AuthUser, machineId: string): Promise<MachineHost> {
+ const owned = await this.db.machine.findFirst({ where: { machineId, userId: user.sub, autonomousEnv: user.autonomousEnv, ...machineAlive } })
+ await this.hostFor(this.db, user, machineId, owned)
+ return { owned }
}
- async observeMachine(user: AuthUser, tabId: string, machineId: string, input: MachineObservation) {
+ /**
+ * A machine's reading for one Swarm. A reading whose decisions cannot differ from its last full report's is stored on its
+ * machine part alone (fastReading); any other runs in full. Whether it runs the Swarm's time checks is decided once, on a
+ * clock every server shares (machineReports.ts), and only for a reading that passed the checks: a refused one would
+ * hold up every server's sweep of that Swarm (review of #2378).
+ */
+ async observeMachine(user: AuthUser, tabId: string, machineId: string, input: MachineObservation, host?: MachineHost,
+ part?: MachinePart | null): Promise<{ version: number }> {
const observation = machineObservationSchema.parse(input)
- const sweep = this.sweepDue(tabId, this.now())
- return transaction(this.db, async tx => {
- const owned = await tx.machine.findFirst({ where: { machineId, userId: user.sub, autonomousEnv: user.autonomousEnv, ...machineAlive } })
- await this.hostFor(tx, user, machineId, owned)
+ const checked = host ?? await this.machineHost(user, machineId)
+ const fast = await this.fastReading(user, tabId, machineId, observation, checked, part)
+ countReport(fast && 'version' in fast ? 'fast' : 'full')
+ return fast && 'version' in fast ? fast : this.fullReading(user, tabId, machineId, observation, fast ? true : null, checked)
+ }
+
+ /**
+ * A machine's readings for several Swarms in one call (DB 2.5, POST /api/super-tabs/hosts/:machineId/readings): its host
+ * checks once, the machine parts of all of them in one read, then each reading as the per-Swarm route takes it, one after
+ * another. Each Swarm is answered on its own: a reading the schema refuses, or a Swarm the caller cannot reach, fails that
+ * Swarm alone, with the error the per-Swarm route would throw.
+ */
+ async observeMachines(user: AuthUser, machineId: string, readings: ReadonlyArray<{ tabId: string; observation: unknown }>):
+ Promise<Array<{ tabId: string; error?: unknown }>> {
+ const host = await this.machineHost(user, machineId)
+ const parts = await loadMachineParts(this.db, readings.map(r => partId(r.tabId, machineId)))
+ const results: Array<{ tabId: string; error?: unknown }> = []
+ for (const { tabId, observation } of readings) {
+ try {
+ await this.observeMachine(user, tabId, machineId, observation as MachineObservation, host, parts.get(partId(tabId, machineId)) ?? null)
+ results.push({ tabId })
+ } catch (error) {
+ results.push({ tabId, error })
+ }
+ }
+ return results
+ }
+
+ /**
+ * A heartbeat stored without loading its Swarm (DB 2.5 spec, sections 1–2), or null to run it in full. It needs the
+ * Swarm unchanged since this machine's last full report (its `main` stamp then, no bump owed here), no removal, none of
+ * its workers' leases due, and a reading that feeds no decision differently (machines.ts readingOnly). It stores what
+ * the full report would: the same reading, Lent for the count that report recorded, the same push to the same people.
+ */
+ private async fastReading(user: AuthUser, tabId: string, machineId: string, observation: MachineObservation, host: MachineHost,
+ loaded?: MachinePart | null): Promise<{ version: number } | { sweep: true } | null> {
+ if (stampOwed(tabId)) return null
+ const [stamps, record] = await Promise.all([readStamps([tabId]), this.reports.get(tabId, machineId)])
+ const now = this.now()
+ // A record holds for RECORD_TTL_MS after its own full report, whatever other machines of the Swarm write since.
+ if (!record || stamps?.get(tabId)?.main !== record.stamp || now >= record.deadline || now - record.at >= RECORD_TTL_MS || record.removals) return null
+ // The checks a full report makes with the Swarm, from what the last one saw of it; a refusal is the full report's to say.
+ if ((record.team && !isTeamSession(user)) || !host.owned || !machineBillingAllowsDataPlane(host.owned)) return null
+ const row = await this.db.superTab.findFirst({ where: { id: tabId, autonomousEnv: user.autonomousEnv }, select: { format: true } })
+ if (row?.format !== 2) return null
+ const part = loaded !== undefined ? loaded : (await loadMachineParts(this.db, [partId(tabId, machineId)])).get(partId(tabId, machineId))
+ const pool = part && !part.deleted ? part.data.contribution : null
+ if (!part || !pool || pool.contributorId !== user.sub || pool.generation !== observation.generation || pool.removal) return null
+ const data = structuredClone(part.data), mine = data.contribution!
+ const reading = storedObservation(mine, observation, { team: isTeamSession(user), now, preparable: id => record.preparable.includes(id) })
+ if (!readingOnly(mine.observation, reading)) return null
+ addLent(mine, now, record.lentCount)
+ mine.observation = reading
+ const suspended = JSON.stringify(mine.engineSuspensions)
+ readingClearsSuspensions(mine, now)
+ if (JSON.stringify(mine.engineSuspensions) !== suspended) return null
+ // Its sweep due: the full report runs it, with the slot this reading, which passed the checks, just took.
+ if (await this.reports.sweepDue(tabId, now)) return { sweep: true }
+ const stamp = clock.next()
+ let version: number
+ try {
+ version = await swarmTransaction(this.db, tx => guardedWrite(tx, tabId, { op: 'update', kind: 'machine', key: machineId, seen: part.version, data }, stamp))
+ } catch (error) {
+ // A conflict, or Mongo's transient refusal, is the full report's to retry.
+ if (retryable(error)) return null
+ throw error
+ }
+ try {
+ this.hooks.swarmChanged?.({ tabId, userIds: record.members, lastChangedAt: stamp, parts: ['machine'] })
+ } catch (error) {
+ logger.warn('[super-tab] reading push failed', { tabId, error: error instanceof Error ? error.message : String(error) })
+ }
+ return { version }
+ }
+
+ /** [sweep]: whether the Swarm's sweep slot was taken already, or null for this report to ask once its checks pass. */
+ private async fullReading(user: AuthUser, tabId: string, machineId: string, observation: MachineObservation, sweep: boolean | null,
+ { owned }: MachineHost): Promise<{ version: number }> {
+ // The Swarm's stamp before anything is loaded: the next reports may skip the load only while it holds (machineReports.ts).
+ const before = (await readStamps([tabId]))?.get(tabId)?.main
+ const { version, record } = await transaction(this.db, async (tx): Promise<{ version: number; record?: Omit<ReportRecord, 'stamp'> }> => {
// Only the conversations its checks read (parts/load.ts loadObserved): the whole Swarm was ~1.1 MB a report (#876).
const loaded = loadedFrom(await this.row(tx, user, tabId), await loadObserved(tx, tabId, machineId)), state = cloneData(loaded.state)
await this.hostFor(tx, user, machineId, owned, state)
@@ -2427,6 +2521,8 @@ export class SuperTabStore {
return found
})
if (!owned || !machineBillingAllowsDataPlane(owned)) fail('HOST_UNAVAILABLE', 'This execution host is unavailable.', 403)
+ // Asked once per report, never again on a retried attempt, which would find the slot taken by itself.
+ sweep ??= await this.reports.sweepDue(tabId, this.now())
const now = this.now()
// A machine that went silent while cleaning takes the fallback before this fresh reading can bring it back: its records
// go, its cleanup waits for this host, as the next sweep would have done (machines.ts expireRemovals).
@@ -2447,23 +2543,10 @@ export class SuperTabStore {
}
if (!state.machines?.includes(pool)) return saved()
recordAgentPresence(state, machineId, observation.runningHarnessIds, now)
- const team = isTeamSession(user)
- // Its privacy check counts only from the Harness Team account's own session; an ordinary sign-in's claim is dropped.
- // Progress is kept for running sessions only: their leases are read from it (liveness.ts).
- const { privacy: reported, harnessProgress, harnessTokens, preparingStartIds, ...reading } = observation
- const progress = harnessProgress && Object.fromEntries(Object.entries(harnessProgress).filter(([id]) => observation.runningHarnessIds.includes(id)))
- // Token counts too: an attempt's run reads its session's totals from them (costs/attempts.ts).
- const tokens = harnessTokens && Object.fromEntries(Object.entries(harnessTokens).filter(([id]) => observation.runningHarnessIds.includes(id)))
- // A machine says preparing only of the starts it launched that are neither running nor failed (liveness.ts).
- const preparing = preparingStartIds?.filter(id => state.starts?.some(s => s.id === id && s.request.machineId === machineId
- && s.dispatched && !s.harnessId && !s.error))
- const privacy = observedPrivacy(pool.observation?.privacy, reported, team, now)
+ const reading = storedObservation(pool, observation, { team: isTeamSession(user), now,
+ preparable: id => !!state.starts?.some(s => s.id === id && preparableStart(s, machineId)) })
accrueLent(state, pool, now)
- pool.observation = { ...reading, quotas: observation.quotas.filter(q => contributedEngines(pool, observation).includes(q.engine)), receivedAt: now,
- // How the machine is signed in, from the session that reported and never from the report (lib/teamSession.ts).
- sessionKind: team ? 'team' : 'computer', ...(privacy ? { privacy } : {}), ...(progress ? { harnessProgress: progress } : {}),
- ...(tokens ? { harnessTokens: tokens } : {}),
- ...(preparing ? { preparingStartIds: preparing } : {}) }
+ pool.observation = reading
// A reading that shows a suspended engine signed in with quota offered again (engineHealth.ts); posts nothing.
readingClearsSuspensions(pool, now)
// Current-generation physical stop proof also finishes a close whose membership was already withdrawn.
@@ -2481,8 +2564,16 @@ export class SuperTabStore {
// Its own agents lose their tasks at once (a stopped session); every 30 seconds it runs the Swarm's time checks too, so
// they keep running when the Lead's own machine is the one that died.
const checked = [...loseTasks(state, now, worker => worker.machineId === machineId), ...sweep ? sweepLiveness(state, now) : []]
- return saved(checked)
+ const written = await saved(checked)
+ // What the next reports of this machine need to skip the load, as this commit leaves the Swarm.
+ const mine = state.machines?.find(m => m.machineId === machineId)
+ return { ...written, ...mine ? { record: { lentCount: lentAgents(state, mine, mine.observation?.runningHarnessIds ?? []),
+ removals: !!state.machines?.some(m => m.removal), deadline: Math.min(nextLossAt(state, machineId) ?? Infinity, now + DEADLINE_CAP_MS),
+ team: teamByRecord(state, machineId), members: state.people.map(p => p.id),
+ preparable: (state.starts ?? []).filter(s => preparableStart(s, machineId)).map(s => s.id), at: now } } : {} }
})
+ if (record && before !== undefined && !stampOwed(tabId)) await this.reports.set(tabId, machineId, { ...record, stamp: before })
+ return { version }
}
async mayObserve(user: AuthUser, tabId: string, machineId: string, harnessId: string, machineOwnerId: string): Promise<boolean> {
diff --git a/backend/vitest.super-tabs.config.ts b/backend/vitest.super-tabs.config.ts
index 9547293a4..70a741197 100644
--- a/backend/vitest.super-tabs.config.ts
+++ b/backend/vitest.super-tabs.config.ts
@@ -3,7 +3,7 @@ import { defineConfig } from 'vitest/config'
export default defineConfig({ test: { include: ['integration/superTabs.test.ts', 'integration/superTabSharing.test.ts', 'integration/superTabGithub.test.ts', 'integration/superTabTeam.test.ts', 'integration/superTabWorkspaces.test.ts',
'integration/superTabRead.test.ts', 'integration/superTabReadMarks.test.ts', 'integration/superTabInbox.test.ts', 'integration/superTabLeaderboard.test.ts', 'integration/superTabMigration.test.ts', 'integration/superTabLiveness.test.ts',
'integration/superTabTidy.test.ts', 'integration/superTabProfile.test.ts', 'integration/superTabLegacyDrop.test.ts', 'integration/shareRaces.test.ts', 'integration/superTabIndexes.test.ts', 'integration/superTabRouteDecisions.test.ts',
- 'integration/superTabHotReads.test.ts', 'integration/swarmSweepStarts.test.ts', 'integration/superTabGroupPlans.test.ts', 'integration/superTabInvitationPlans.test.ts', 'integration/superTabCache.test.ts',
+ 'integration/superTabHotReads.test.ts', 'integration/swarmSweepStarts.test.ts', 'integration/superTabGroupPlans.test.ts', 'integration/superTabInvitationPlans.test.ts', 'integration/superTabCache.test.ts', 'integration/superTabMachineReports.test.ts',
'integration/superTabObserver.test.ts', 'integration/productUsageConflicts.test.ts', 'integration/productUsageReceipts.test.ts', 'integration/schemaRelease.test.ts', 'integration/superTabBuddies.test.ts', 'integration/superTabNotAvailable.test.ts',
'integration/superTabBuddyProbes.test.ts', 'integration/superTabBuddySessions.test.ts', 'integration/superTabCosts.test.ts', 'integration/harnessSharesList.test.ts', 'integration/atlasSnapshot.test.ts', 'integration/queryStats.test.ts',
'integration/superTabEvidence.test.ts', 'integration/superTabAttemptModels.test.ts', 'integration/superTabNamedReads.test.ts', 'integration/swarmSweepPlans.test.ts', 'integration/workerIndexPlans.test.ts', 'integration/superTabReflection.test.ts'], hookTimeout: 120000, testTimeout: 30000, setupFiles: ['integration/cacheOn.test-support.ts'],
diff --git a/cli/src/superTabs/cloneTrust.spec.ts b/cli/src/superTabs/cloneTrust.spec.ts
index 102028502..bfcd6a387 100644
--- a/cli/src/superTabs/cloneTrust.spec.ts
+++ b/cli/src/superTabs/cloneTrust.spec.ts
@@ -41,6 +41,12 @@ beforeEach(() => {
writeFileSync(join(process.env.CODEX_HOME!, 'config.toml'), 'model = "unchanged"\n')
})
afterEach(() => { vi.unstubAllEnvs(); rmSync(home, { recursive: true, force: true }) })
+/** A backend that serves the host form for [row] and takes every reading and command; the readings come in one call for all
+ * of the machine's Swarms (DB 2.5). */
+const takesAll = (row: unknown) => vi.fn(async (_core: unknown, _method: string, path: string, body?: unknown) => path.includes('/changes')
+ ? hostForm([], { pools: [row] as never })
+ : path.endsWith('/readings') ? { results: (body as { readings: Array<{ tabId: string }> }).readings.map(r => ({ tabId: r.tabId, ok: true })) } : {})
+
function git(cwd: string, ...args: string[]) {
return execFileSync('git', ['-c', 'user.name=Fixture', '-c', '[email protected]', '-C', cwd, ...args], {
env: { ...process.env, GIT_CONFIG_NOSYSTEM: '1', GIT_CONFIG_GLOBAL: '/dev/null', GIT_TERMINAL_PROMPT: '0' }, encoding: 'utf8',
@@ -98,7 +104,7 @@ describe('consented Swarm clones', () => {
const f = fixture(true), source = clone(join(f.oldRoot, 'project')), current = clone(join(f.root, 'project'))
const tree = join(home, 'harnesses', 'worktrees', 'project', 'worker'), unrelated = join(home, 'harnesses', 'worktrees', 'project', 'personal')
git(source, 'worktree', 'add', '-q', '-b', 'super-tab/worker', tree); git(source, 'worktree', 'add', '-q', '-b', 'personal', unrelated)
- const backend = vi.fn(async (_core: unknown, _method: string, path: string) => path.includes('/changes') ? hostForm([], { pools: [f.row] }) : {})
+ const backend = takesAll(f.row)
const start = vi.fn(), close = vi.fn()
const machine = new ContributedMachines(f.core, f.core.daemon.machineId(), { backend,
open: async () => ({ request: async () => ({}), close }), capabilities: f.deps.capacity, start, now: () => 1 } as never)
@@ -115,7 +121,7 @@ describe('consented Swarm clones', () => {
mkdirSync(profile); writeFileSync(join(profile, 'config.toml'), '')
Object.assign(f.row, { harnessIds: ['worker'] })
vi.mocked(f.core.agents.all).mockReturnValue([{ agentId: 'worker', engine: 'codex', cwd: tree, codexHome: profile }] as never)
- const backend = vi.fn(async (_core: unknown, _method: string, path: string) => path.includes('/changes') ? hostForm([], { pools: [f.row] }) : {})
+ const backend = takesAll(f.row)
const machine = new ContributedMachines(f.core, f.core.daemon.machineId(), { backend,
open: async () => ({ request: async () => ({}), close: vi.fn() }), capabilities: f.deps.capacity, start: vi.fn(), now: () => 1 } as never)
await machine.tick()
@@ -136,7 +142,7 @@ describe('consented Swarm clones', () => {
const f = fixture(true), source = clone(join(f.oldRoot, 'project'))
const config = join(process.env.CODEX_HOME!, 'config.toml')
rmSync(config); mkdirSync(config)
- const backend = vi.fn(async (_core: unknown, _method: string, path: string) => path.includes('/changes') ? hostForm([], { pools: [f.row] }) : {})
+ const backend = takesAll(f.row)
const machine = new ContributedMachines(f.core, f.core.daemon.machineId(), { backend,
open: async () => ({ request: async () => ({}), close: vi.fn() }), capabilities: f.deps.capacity, start: vi.fn(), now: () => 1 } as never)
await machine.tick()
@@ -166,7 +172,7 @@ describe('consented Swarm clones', () => {
it('publishes observations before repair and does no trust work on a second successful tick', async () => {
const f = fixture(true), source = clone(join(f.oldRoot, 'project'))
let now = 1
- const backend = vi.fn(async (_core: unknown, _method: string, path: string) => path.includes('/changes') ? hostForm([], { pools: [f.row] }) : {})
+ const backend = takesAll(f.row)
const machine = new ContributedMachines(f.core, f.core.daemon.machineId(), { backend,
open: async () => ({ request: async () => ({}), close: vi.fn() }), capabilities: f.deps.capacity, start: vi.fn(), now: () => now } as never)
await machine.tick()
@@ -191,7 +197,7 @@ describe('consented Swarm clones', () => {
const f = fixture(true), source = clone(join(f.oldRoot, 'project')), file = join(process.env.CODEX_HOME!, 'config.toml')
rmSync(file); mkdirSync(file)
let now = 1
- const backend = vi.fn(async (_core: unknown, _method: string, path: string) => path.includes('/changes') ? hostForm([], { pools: [f.row] }) : {})
+ const backend = takesAll(f.row)
const machine = new ContributedMachines(f.core, f.core.daemon.machineId(), { backend,
open: async () => ({ request: async () => ({}), close: vi.fn() }), capabilities: f.deps.capacity, start: vi.fn(), now: () => now } as never)
await machine.tick()
@@ -228,7 +234,7 @@ describe('consented Swarm clones', () => {
const actual = await vi.importActual<typeof import('../lib/gitProject.js')>('../lib/gitProject.js')
let unavailable = true, now = 1
vi.mocked(readGitProject).mockImplementation(async (path, options) => path === failed && unavailable ? { error: 'GIT_UNAVAILABLE' } : actual.readGitProject(path, options))
- const backend = vi.fn(async (_core: unknown, _method: string, path: string) => path.includes('/changes') ? hostForm([], { pools: [f.row] }) : {})
+ const backend = takesAll(f.row)
const machine = new ContributedMachines(f.core, f.core.daemon.machineId(), { backend,
open: async () => ({ request: async () => ({}), close: vi.fn() }), capabilities: f.deps.capacity, start: vi.fn(), now: () => now } as never)
try {
@@ -246,7 +252,7 @@ describe('consented Swarm clones', () => {
const claude = join(process.env.CLAUDE_CONFIG_DIR!, '.claude.json'), codex = join(process.env.CODEX_HOME!, 'config.toml')
writeFileSync(claude, 'invalid JSON'); rmSync(codex); mkdirSync(codex)
let now = 1
- const backend = vi.fn(async (_core: unknown, _method: string, path: string) => path.includes('/changes') ? hostForm([], { pools: [f.row] }) : {})
+ const backend = takesAll(f.row)
const machine = new ContributedMachines(f.core, f.core.daemon.machineId(), { backend,
open: async () => ({ request: async () => ({}), close: vi.fn() }), capabilities: f.deps.capacity, start: vi.fn(), now: () => now } as never)
await machine.tick()
@@ -262,7 +268,7 @@ describe('consented Swarm clones', () => {
const f = fixture(true), source = clone(join(f.oldRoot, 'project'))
Object.assign(f.row, { starts: [{ id: 'start', dispatched: true }] })
let now = 1
- const backend = vi.fn(async (_core: unknown, _method: string, path: string) => path.includes('/changes') ? hostForm([], { pools: [f.row] }) : {})
+ const backend = takesAll(f.row)
const start = vi.fn(async () => ({ ok: false }))
const machine = new ContributedMachines(f.core, f.core.daemon.machineId(), { backend,
open: async () => ({ request: async () => ({}), close: vi.fn() }), capabilities: f.deps.capacity, start, now: () => now } as never)
@@ -289,7 +295,7 @@ describe('consented Swarm clones', () => {
const f = fixture(true), source = clone(join(f.oldRoot, 'project')), codex = join(process.env.CODEX_HOME!, 'config.toml')
rmSync(codex)
let now = 1
- const backend = vi.fn(async (_core: unknown, _method: string, path: string) => path.includes('/changes') ? hostForm([], { pools: [f.row] }) : {})
+ const backend = takesAll(f.row)
const machine = new ContributedMachines(f.core, f.core.daemon.machineId(), { backend,
open: async () => ({ request: async () => ({}), close: vi.fn() }), capabilities: f.deps.capacity, start: vi.fn(), now: () => now } as never)
try {
@@ -308,7 +314,7 @@ describe('consented Swarm clones', () => {
it('repairs a newly registered Codex profile and worktree before the next observation is due', async () => {
const f = fixture(true), source = clone(join(f.oldRoot, 'project'))
let now = 1
- const backend = vi.fn(async (_core: unknown, _method: string, path: string) => path.includes('/changes') ? hostForm([], { pools: [f.row] }) : {})
+ const backend = takesAll(f.row)
const machine = new ContributedMachines(f.core, f.core.daemon.machineId(), { backend,
open: async () => ({ request: async () => ({}), close: vi.fn() }), capabilities: f.deps.capacity, start: vi.fn(), now: () => now } as never)
try {
@@ -330,7 +336,7 @@ describe('consented Swarm clones', () => {
])('discovers a later clone after one-shot $failure failure (existing=$existing)', async ({ existing, failure }) => {
const f = fixture(true), earlier = existing ? clone(join(f.oldRoot, 'earlier')) : undefined
let now = 1
- const backend = vi.fn(async (_core: unknown, _method: string, path: string) => path.includes('/changes') ? hostForm([], { pools: [f.row] }) : {})
+ const backend = takesAll(f.row)
const machine = new ContributedMachines(f.core, f.core.daemon.machineId(), { backend,
open: async () => ({ request: async () => ({}), close: vi.fn() }), capabilities: f.deps.capacity, start: vi.fn(), now: () => now } as never)
try {
@@ -354,7 +360,7 @@ describe('consented Swarm clones', () => {
const f = fixture(true), source = join(f.oldRoot, 'project')
mkdirSync(source, { recursive: true })
let now = 1
- const backend = vi.fn(async (_core: unknown, _method: string, path: string) => path.includes('/changes') ? hostForm([], { pools: [f.row] }) : {})
+ const backend = takesAll(f.row)
const machine = new ContributedMachines(f.core, f.core.daemon.machineId(), { backend,
open: async () => ({ request: async () => ({}), close: vi.fn() }), capabilities: f.deps.capacity, start: vi.fn(), now: () => now } as never)
try {
@@ -368,7 +374,7 @@ describe('consented Swarm clones', () => {
const f = fixture(true), source = clone(join(f.oldRoot, 'project'))
Object.assign(f.row, { starts: [{ id: 'start', dispatched: true }] })
let now = 1
- const backend = vi.fn(async (_core: unknown, _method: string, path: string) => path.includes('/changes') ? hostForm([], { pools: [f.row] }) : {})
+ const backend = takesAll(f.row)
const start = vi.fn(async () => ({ ok: false }))
const machine = new ContributedMachines(f.core, f.core.daemon.machineId(), { backend,
open: async () => ({ request: async () => ({}), close: vi.fn() }), capabilities: f.deps.capacity, start, now: () => now } as never)
@@ -388,7 +394,7 @@ describe('consented Swarm clones', () => {
const original = `[projects.${JSON.stringify(source)}]\ntrust_level = "untrusted"\n`
writeFileSync(codex, original)
let now = 1
- const backend = vi.fn(async (_core: unknown, _method: string, path: string) => path.includes('/changes') ? hostForm([], { pools: [f.row] }) : {})
+ const backend = takesAll(f.row)
const machine = new ContributedMachines(f.core, f.core.daemon.machineId(), { backend,
open: async () => ({ request: async () => ({}), close: vi.fn() }), capabilities: f.deps.capacity, start: vi.fn(), now: () => now } as never)
try {
diff --git a/cli/src/superTabs/host.spec.ts b/cli/src/superTabs/host.spec.ts
index 90c001bf1..74910ba2a 100644
--- a/cli/src/superTabs/host.spec.ts
+++ b/cli/src/superTabs/host.spec.ts
@@ -44,6 +44,8 @@ function setup(messageFields: Record<string, unknown> = {}, claimFields: Record<
let clock = 0
const call: CoreApi['account']['backend'] = vi.fn(async (method, path, body) => {
let data: unknown
+ // A backend from before one call for all of a machine's readings (DB 2.5): the host sends each Swarm's on its own.
+ if (path.endsWith('/readings')) return { status: 404, body: { message: `Route POST:${path} not found`, error: 'Not Found', statusCode: 404 } }
if (path.includes('/hosts/')) data = hostForm(removed ? [] : [row], { tabs: ['tab'] })
else if (path.includes('asAgentId=')) data = swarmReply(path, { conversations: [{ id: 'lead' }] })
else if (revoked) return { status: 403, body: { success: false, error: { code: 'FORBIDDEN', message: 'Grant removed' } } }
diff --git a/cli/src/superTabs/machines.spec.ts b/cli/src/superTabs/machines.spec.ts
index 8f8b002d6..e701e8951 100644
--- a/cli/src/superTabs/machines.spec.ts
+++ b/cli/src/superTabs/machines.spec.ts
@@ -6,7 +6,7 @@ import { join } from 'node:path'
import { afterEach, describe, expect, it, onTestFinished, vi } from 'vitest'
import type { RegisteredSession } from '../lib/registry.js'
import { fakeCore } from '../testing/fakeCore.js'
-import { apisOn, ContributedMachines, createQuietGit, liveOn, localModelsStamp, offered, refusedOn, servedOn, sharedWith, swarmUnconfirmedFile, swarmWorktreesFile } from './machines.js'
+import { apisOn, ContributedMachines, createQuietGit, liveOn, localModelsStamp, offered, READINGS_RETRY_MS, refusedOn, servedOn, sharedWith, swarmUnconfirmedFile, swarmWorktreesFile } from './machines.js'
import { SuperTabHostError } from './host.js'
import { startAgent } from './start.js'
import { hostForm, swarmReply } from './fixtures/read/index.js'
@@ -32,9 +32,27 @@ function fixture(env: NodeJS.ProcessEnv = {}, privacy: () => PrivacyObservation
const prepare = vi.fn(async () => {})
const log = vi.fn()
const modelsStamp = vi.fn(async () => stamp)
- const deps = { backend, open, capabilities, start, prepare, log, env, privacy, modelsStamp, relayed, now: () => now }
+ // The backend's one call for all of a machine's readings (DB 2.5), answered as `backend` answers each Swarm's own route,
+ // so every test reads the readings sent per Swarm; `readings` counts the calls themselves.
+ const readings = vi.fn(async (core: unknown, path: string, body: unknown) => {
+ const machineId = path.split('/')[4]
+ const results: unknown[] = []
+ for (const { tabId, observation } of (body as { readings: Array<{ tabId: string; observation: unknown }> }).readings) {
+ try {
+ await backend(core, 'POST', `/api/super-tabs/${tabId}/machines/${machineId}`, observation)
+ results.push({ tabId, ok: true })
+ } catch (error) {
+ const known = error instanceof SuperTabHostError ? error : new SuperTabHostError('BACKEND_UNAVAILABLE', error instanceof Error ? error.message : String(error))
+ results.push({ tabId, error: { code: known.code, message: known.message, status: known.status } })
+ }
+ }
+ return { results }
+ })
+ const wire = async (core: unknown, method: string, path: string, body?: unknown): Promise<unknown> =>
+ method === 'POST' && path.endsWith('/readings') ? readings(core, path, body) : backend(core, method, path, body)
+ const deps = { backend: wire, open, capabilities, start, prepare, log, env, privacy, modelsStamp, relayed, now: () => now }
const machine = new ContributedMachines(core, 'host', deps as never)
- return { deps, machine, pool, core, backend, open, request, close, capabilities, start, prepare, log, modelsStamp, advance: () => { now += 16000 }, step: (ms: number) => { now += ms },
+ return { deps, machine, pool, core, backend, readings, open, request, close, capabilities, start, prepare, log, modelsStamp, advance: () => { now += 16000 }, step: (ms: number) => { now += ms },
stamp: (next: string) => { stamp = next } }
}
@@ -124,7 +142,7 @@ describe('the pools of a contributed machine', () => {
let now = 1000
const feed = new HostFeed(f.core, 'host', { now: () => now, backend: f.backend })
await feed.refresh()
- const machine = new ContributedMachines(f.core, 'host', { backend: f.backend, open: f.open, capabilities: f.capabilities, start: f.start, prepare: f.prepare,
+ const machine = new ContributedMachines(f.core, 'host', { backend: f.deps.backend, open: f.open, capabilities: f.capabilities, start: f.start, prepare: f.prepare,
log: f.log, now: () => now, sources: { feed, reads: new SwarmReads(f.core, { backend: f.backend }) } } as never)
now += 1_999
await machine.tick()
@@ -197,7 +215,7 @@ describe('contributed machine executor', () => {
it('readies a start with the backend, as a team account when this daemon runs team work only', async () => {
for (const team of [false, true]) {
const f = fixture()
- const machine = new ContributedMachines(f.core, 'host', { backend: f.backend, open: f.open, capabilities: f.capabilities, start: f.start,
+ const machine = new ContributedMachines(f.core, 'host', { backend: f.deps.backend, open: f.open, capabilities: f.capabilities, start: f.start,
log: f.log, env: {}, privacy: () => null, team: () => team, now: () => 1000 } as never)
Object.assign(f.pool, { githubApp: true })
f.backend.mockImplementation(async (_core: unknown, _method: string, path: string): Promise<unknown> => path.includes('/hosts/host/changes') ? hostForm([], { pools: [f.pool] })
@@ -386,7 +404,7 @@ describe('contributed machine executor', () => {
function swarmFixture(git: ReturnType<typeof gitFor>['git']) {
const f = fixture()
Object.assign(f.pool.machine, { scope: 'swarm', cwd: swarmFolder() })
- const machine = new ContributedMachines(f.core, 'host', { backend: f.backend, open: f.open, capabilities: f.capabilities, start: f.start, prepare: f.prepare,
+ const machine = new ContributedMachines(f.core, 'host', { backend: f.deps.backend, open: f.open, capabilities: f.capabilities, start: f.start, prepare: f.prepare,
log: f.log, now: () => now, git } as never)
let now = 1000
return { ...f, machine, step: (ms: number) => { now += ms } }
@@ -2214,3 +2232,73 @@ describe('the allowance a machine reports', () => {
expect(posts().slice(-2)).toEqual([true, false])
})
})
+
+describe('one call for all of a machine\'s readings (DB 2.5)', () => {
+ /** The fixture in two Swarms, each with a waiting start; [refuse] answers a Swarm's own route for the readings it refuses. */
+ function twoSwarms(refuse: (tabId: string, body: Record<string, unknown>) => SuperTabHostError | null = () => null) {
+ const f = fixture()
+ const second = { ...f.pool, tabId: 'second-tab', starts: [{ id: 'second-start', dispatched: false }] }
+ const original = f.backend.getMockImplementation()!
+ f.backend.mockImplementation(async (core, method, path, body) => {
+ if (path.includes('/hosts/host/changes')) return hostForm([], { pools: [f.pool, second] })
+ const tabId = /^\/api\/super-tabs\/([^/]+)\/machines\//.exec(path)?.[1]
+ const refused = tabId ? refuse(tabId, body as Record<string, unknown>) : null
+ if (refused) throw refused
+ return original(core, method, path, body)
+ })
+ return f
+ }
+ const sentTo = (f: ReturnType<typeof fixture>) => f.backend.mock.calls.filter(c => c[1] === 'POST' && String(c[2]).includes('/machines/'))
+ .map(c => [/super-tabs\/([^/]+)\/machines/.exec(String(c[2]))![1], 'toolProbeErrors' in (c[3] as object)])
+ const missing = new SuperTabHostError('BACKEND_UNAVAILABLE', 'Route POST:/api/super-tabs/hosts/host/readings not found', 404)
+
+ it('sends every Swarm\'s reading in one call, then runs each Swarm\'s starts', async () => {
+ const f = twoSwarms()
+ await f.machine.tick()
+ expect(f.readings).toHaveBeenCalledOnce()
+ expect((f.readings.mock.calls[0]![2] as { readings: Array<{ tabId: string }> }).readings.map(r => r.tabId)).toEqual(['tab', 'second-tab'])
+ expect(f.start.mock.calls.map(call => ((call as unknown[])[0] as { startId: string }).startId)).toEqual(['start', 'second-start'])
+ })
+
+ it('sends each Swarm\'s reading on its own to a backend without that call, and asks again ten minutes later', async () => {
+ for (const refusal of [missing, new SuperTabHostError('TEAM_SESSION_LIMITED', 'This session cannot do that.', 403)]) {
+ const f = twoSwarms()
+ f.readings.mockRejectedValueOnce(refusal)
+ await f.machine.tick()
+ expect(sentTo(f).map(([tab]) => tab)).toEqual(['tab', 'second-tab'])
+ expect(f.start).toHaveBeenCalledTimes(2)
+ f.advance(); await f.machine.tick()
+ expect(f.readings, refusal.code).toHaveBeenCalledOnce()
+ f.step(READINGS_RETRY_MS); await f.machine.tick()
+ expect(f.readings).toHaveBeenCalledTimes(2)
+ }
+ })
+
+ it('sends one Swarm\'s reading again alone, in its older form, when the backend refuses a field for that Swarm', async () => {
+ const f = twoSwarms((tabId, body) => tabId === 'second-tab' && 'toolProbeErrors' in body
+ ? new SuperTabHostError('VALIDATION_ERROR', 'Unrecognized key: toolProbeErrors', 400) : null)
+ await f.machine.tick()
+ expect(f.readings).toHaveBeenCalledOnce()
+ // The batch carried both readings whole; only the refused Swarm's older one went alone, and was taken.
+ expect(sentTo(f)).toEqual([['tab', true], ['second-tab', true], ['second-tab', false]])
+ expect(f.log.mock.calls.flat().join('\n')).not.toMatch(/Could not send this machine's reading/)
+ })
+
+ it('runs no start in a Swarm whose answer says this machine lost access to it, and every other Swarm\'s starts', async () => {
+ const f = twoSwarms(tabId => tabId === 'second-tab' ? new SuperTabHostError('CONTRIBUTION_REVOKED', 'Read the current machine contribution.', 403) : null)
+ await f.machine.tick()
+ expect(f.start.mock.calls.map(call => ((call as unknown[])[0] as { startId: string }).startId)).toEqual(['start'])
+ })
+
+ it('sends again every Swarm\'s last reading in one call while a slow pass holds up the next', async () => {
+ const f = twoSwarms()
+ let launched!: () => void
+ f.start.mockImplementationOnce(() => new Promise(resolve => { launched = () => resolve({ ok: true }) }) as never)
+ const pass = f.machine.tick()
+ await vi.waitFor(() => expect(f.start).toHaveBeenCalledOnce())
+ f.step(15_000); f.machine.beat(); await new Promise(done => setTimeout(done, 0))
+ expect(f.readings).toHaveBeenCalledTimes(2)
+ expect((f.readings.mock.calls[1]![2] as { readings: Array<{ tabId: string }> }).readings.map(r => r.tabId)).toEqual(['tab', 'second-tab'])
+ launched(); await pass
+ })
+})
diff --git a/cli/src/superTabs/machines.ts b/cli/src/superTabs/machines.ts
index e4e1086c0..299625fdc 100644
--- a/cli/src/superTabs/machines.ts
+++ b/cli/src/superTabs/machines.ts
@@ -152,6 +152,20 @@ const CLEANING_WAIT = 'Waiting for this machine to finish cleaning up this Swarm
* hold up a delivery lease. Only a tab's explicit owner grant wakes this queue. */
/** A pool's starts this host launches: its agents', and its Buddies', which the backend lists apart so that a host from before
* Buddies never launches one as a Builder (backend read/hostRows.ts). */
+/** How long this machine sends one reading per Swarm after its backend had no route for all of them in one call (an older
+ * backend, DB roadmap 2.5): a backend deployed meanwhile is asked again within this long. */
+export const READINGS_RETRY_MS = 10 * 60_000
+/** The most Swarms' readings the backend takes in one call (backend routes/superTabs.ts READINGS_MAX). */
+const READINGS_MAX = 64
+/** The batch route is not there: an older backend answers 404, and its team gate 403 for a route it does not list. A body
+ * past its limit (413) is sent per Swarm too. */
+const readingsMissing = (error: unknown) => error instanceof SuperTabHostError
+ && (error.status === 404 || error.status === 413 || (error.status === 403 && error.code === 'TEAM_SESSION_LIMITED'))
+/** A reading the backend would not validate: the next, older form of it is tried (an older backend refusing a newer field). */
+const fieldRefused = (error: unknown) => error instanceof SuperTabHostError && error.status === 400 && error.code === 'VALIDATION_ERROR'
+/** One Swarm's reading of this pass, to send. */
+interface Reading { tab: string; observation: Record<string, unknown>; localModels: LocalModelReading[] | null; signature: string }
+
/** A host's starts: its workers', and its Buddies' and Product Managers', which the backend lists apart so that a host from
* before them never launches one as a worker. */
const startsOf = (row: Record<string, unknown>) => [...list(row.starts), ...list(row.buddyStarts), ...list(row.leadStarts)]
@@ -211,6 +225,8 @@ export class ContributedMachines {
private readonly cleanupRetries = new Map<string, { failures: number; at: number }>()
/** The last removal asked for each stopped agent's worktree, by tab and agent: the next one waits for it (removeStoppedWorktree). */
private readonly removals = new Map<string, Promise<void>>()
+ /** Until when this machine sends one reading per Swarm, its backend having no route for all of them in one call. */
+ private readingsRefusedUntil = 0
/** Until when the reading leaves out its privacy field, which the backend refused. */
private privacyRefusedUntil = 0
/** Until when the reading leaves out its progress, which the backend refused. */
@@ -367,12 +383,31 @@ export class ContributedMachines {
if (now - began >= BEAT_LIMIT_MS) return
const due = [...this.lastSent].filter(([tab, sent]) => !this.sending.has(tab) && now - sent.at >= MACHINE_HEARTBEAT_MS)
if (!due.length) return
+ const failed = (error: unknown) => this.log(`Could not send this machine's reading again: ${error instanceof Error ? error.message : String(error)}`)
this.beating = (async () => {
- for (const [, sent] of due) {
- // A failed one waits a whole beat too, rather than being asked again every second.
- sent.at = now
- try { await this.deps.backend(this.core, 'POST', sent.path, sent.body) }
- catch (error) { this.log(`Could not send this machine's reading again: ${error instanceof Error ? error.message : String(error)}`) }
+ // A failed one waits a whole beat too, rather than being asked again every second.
+ for (const [, sent] of due) sent.at = now
+ let alone = due
+ if (now >= this.readingsRefusedUntil) {
+ alone = []
+ for (let i = 0; i < due.length; i += READINGS_MAX) {
+ const chunk = due.slice(i, i + READINGS_MAX)
+ try {
+ const answer = object(await this.deps.backend(this.core, 'POST', `/api/super-tabs/hosts/${this.machineId}/readings`,
+ { readings: chunk.map(([tabId, sent]) => ({ tabId, observation: sent.body })) }))
+ for (const result of Array.isArray(answer.results) ? answer.results : []) {
+ const refusal = object(object(result).error)
+ if (object(result).ok !== true) failed(new Error(String(refusal.message ?? 'The backend did not take it.')))
+ }
+ } catch (error) {
+ if (!readingsMissing(error)) { failed(error); continue }
+ this.readingsRefusedUntil = now + READINGS_RETRY_MS
+ alone.push(...chunk)
+ }
+ }
+ }
+ for (const [, sent] of alone) {
+ try { await this.deps.backend(this.core, 'POST', sent.path, sent.body) } catch (error) { failed(error) }
}
})().finally(() => { this.beating = null })
}
@@ -398,6 +433,9 @@ export class ContributedMachines {
let local: Awaited<ReturnType<typeof openDaemon>> | null = null
let usage: { value: Record<string, unknown>; at: number } | null = null
try {
+ // Every Swarm's reading first, sent in one call (DB roadmap 2.5), then each Swarm's starts.
+ const looked: Array<{ row: Record<string, unknown>; pool: Record<string, unknown>; pending: ReturnType<typeof startsOf> }> = []
+ const due: Reading[] = []
for (const row of pools) {
try {
if (this.stopped || !identifier(row.tabId)) return
@@ -413,6 +451,7 @@ export class ContributedMachines {
}
const running = ids.filter(id => !!this.core.agents.byAgent(id) && !stopped.includes(id))
const pending = startsOf(row)
+ looked.push({ row, pool, pending })
const preparing = began >= this.preparingRefusedUntil ? pending.flatMap(start => this.preparing.has(String(start.id)) ? [String(start.id)] : []) : []
const tools = [...new Set([...COMMON_TOOLS, ...pending.flatMap(start => {
const wanted = object(object(start.request).requirements).tools
@@ -505,59 +544,22 @@ export class ContributedMachines {
quotas: allowances.map(q => ({ engine: q.engine, known: q.known, windows: q.windows })), runningHarnessIds: running, stoppedHarnessIds: stopped,
...(Array.isArray(row.reservations) ? { stoppedReservations: list(row.reservations).filter(r => stopped.includes(String(r.harnessId))
&& identifier(r.harnessId) && identifier(r.generation)).map(r => ({ harnessId: r.harnessId, generation: r.generation })) } : {}) }
- this.sending.add(String(row.tabId))
- try {
- await this.beating
- const path = `/api/super-tabs/${row.tabId}/machines/${this.machineId}`
- const refused = (error: unknown) => error instanceof SuperTabHostError && error.status === 400 && error.code === 'VALIDATION_ERROR'
- // Older backends reject the additive fields. The newest go alone first (the starts it is preparing, uncommitted work,
- // readiness, progress, then privacy): a backend that takes everything else keeps the local models it reads. Their
- // boolean-only view still safely refuses an unverified tool, and a reading without privacy is never one a Team
- // machine passes on.
- const { quotaReasons: _reasons, ...unreasoned } = observation
- const { harnessTokens: _tokens, ...untokened } = unreasoned
- const { preparingStartIds: _preparing, ...unprepared } = untokened
- const { harnessUncommitted: _uncommitted, ...known } = unprepared
- const { readiness: _readiness, ...ready } = known
- const { harnessProgress: _progress, ...withoutProgress } = ready
- const { privacy: _privacy, ...withoutPrivacy } = withoutProgress
- const { toolProbeErrors: _probeErrors, localModels: _served, refusedModels: _refused, ...legacy } = withoutPrivacy
- // A backend before `starting` refuses it: it gets the models the grid lists, as before, not none at all.
- const listedOnly = localModels?.some(m => m.starting)
- ? [{ ...withoutPrivacy, localModels: localModels.filter(m => !m.starting) }] : []
- const attempts = [...('quotaReasons' in observation ? [observation] : []), ...('harnessTokens' in unreasoned ? [unreasoned] : []), ...('preparingStartIds' in untokened ? [untokened] : []),
- ...('harnessUncommitted' in unprepared ? [unprepared] : []),
- ...('readiness' in known ? [known] : []), ready, ...('harnessProgress' in ready ? [withoutProgress] : []),
- ...('privacy' in withoutProgress ? [withoutPrivacy] : []), ...listedOnly,
- ...('toolProbeErrors' in observation || 'localModels' in observation || 'refusedModels' in observation ? [legacy] : [])]
- for (const [index, attempt] of attempts.entries()) {
- try {
- await this.deps.backend(this.core, 'POST', path, attempt)
- const { harnessUncommitted: _fresh, ...taken } = attempt as Record<string, unknown>
- this.lastSent.set(String(row.tabId), { path, body: taken, at: this.deps.now() })
- break
- } catch (error) {
- if (!refused(error) || index === attempts.length - 1) throw error
- if ('quotaReasons' in attempt) this.quotaReasonsRefusedUntil = began + FIELD_REFUSED_MS
- else if ('harnessTokens' in attempt) this.tokensRefusedUntil = began + FIELD_REFUSED_MS
- else if ('preparingStartIds' in attempt) this.preparingRefusedUntil = began + FIELD_REFUSED_MS
- else if ('harnessUncommitted' in attempt) this.uncommittedRefusedUntil = began + FIELD_REFUSED_MS
- else if ('readiness' in attempt) this.readinessRefusedUntil = began + FIELD_REFUSED_MS
- else if ('harnessProgress' in attempt) this.progressRefusedUntil = began + FIELD_REFUSED_MS
- else if ('privacy' in attempt) this.privacyRefusedUntil = began + FIELD_REFUSED_MS
- }
- }
- this.published.set(String(row.tabId), { at: began, signature, starting: !!localModels?.some(m => m.starting) })
- } finally { this.sending.delete(String(row.tabId)) }
- this.noteReading(String(row.tabId), null)
+ due.push({ tab: String(row.tabId), observation, localModels, signature })
} catch (error) {
+ // A reading this machine could not put together (its daemon unreachable, say): its starts still run.
this.failed = true
this.noteReading(String(row.tabId), error)
- // Only a refusal that this machine lost access to the Swarm stops its starts; a timeout, a 5xx or a reading the
- // backend would not validate says nothing about the starts, which keep their own receipts (Diego's Mac, 2026-10-28).
- if (lostAccess(error)) throw error
}
}
+ } catch { /* One unavailable project cannot block another contribution. */ }
+ }
+ const lost = await this.publish(due, began)
+ for (const { row, pool, pending } of looked) {
+ if (this.stopped) return
+ // Only a refusal that this machine lost access to the Swarm stops its starts; a timeout, a 5xx or a reading the
+ // backend would not validate says nothing about the starts, which keep their own receipts (Diego's Mac, 2026-10-28).
+ if (lost.has(String(row.tabId))) continue
+ try {
this.repairTrust ??= createSwarmCloneTrustRepair(this.core, this.deps.now)
try { await this.repairTrust(row) }
catch { /* Trust migration cannot prevent this pass from picking up pending starts. */ }
@@ -648,6 +650,121 @@ export class ContributedMachines {
}
} finally { local?.close() }
}
+ /**
+ * Sends this pass's readings: one call for every Swarm (DB roadmap 2.5), or one per Swarm to a backend without that route,
+ * asked again after READINGS_RETRY_MS. A Swarm's own refusal is handled as its own call's was: a field an older backend
+ * does not know sends that Swarm's older readings alone (sendAlone). Answers the Swarms this machine lost access to.
+ */
+ private async publish(due: Reading[], began: number): Promise<Set<string>> {
+ const lost = new Set<string>()
+ if (!due.length) return lost
+ for (const reading of due) this.sending.add(reading.tab)
+ try {
+ // A re-send landing after this pass's reading would put the older one back.
+ await this.beating
+ const answers = new Map<string, SuperTabHostError | null>()
+ let alone = due
+ if (began >= this.readingsRefusedUntil) {
+ alone = []
+ for (let i = 0; i < due.length; i += READINGS_MAX) {
+ const chunk = due.slice(i, i + READINGS_MAX)
+ try {
+ const answer = object(await this.deps.backend(this.core, 'POST', `/api/super-tabs/hosts/${this.machineId}/readings`,
+ { readings: chunk.map(reading => ({ tabId: reading.tab, observation: reading.observation })) }))
+ for (const result of Array.isArray(answer.results) ? answer.results.map(object) : []) {
+ const error = object(result.error)
+ answers.set(String(result.tabId), result.ok === true ? null : new SuperTabHostError(String(error.code ?? 'BACKEND_UNAVAILABLE'),
+ String(error.message ?? 'Could not reach this Super Tab.'), typeof error.status === 'number' ? error.status : 503))
+ }
+ } catch (error) {
+ if (readingsMissing(error)) {
+ this.readingsRefusedUntil = began + READINGS_RETRY_MS
+ alone.push(...chunk)
+ } else for (const reading of chunk) answers.set(reading.tab, error instanceof SuperTabHostError ? error : new SuperTabHostError('BACKEND_UNAVAILABLE', String(error)))
+ }
+ }
+ }
+ for (const reading of due) {
+ try {
+ if (alone.includes(reading)) await this.sendAlone(reading, began)
+ else {
+ const answer = answers.get(reading.tab)
+ if (answer === undefined) throw new SuperTabHostError('BACKEND_UNAVAILABLE', 'The backend did not answer for this Swarm.')
+ if (answer && !fieldRefused(answer)) throw answer
+ if (answer) await this.sendAlone(reading, began, answer)
+ else this.taken(reading.tab, reading.observation)
+ }
+ this.published.set(reading.tab, { at: began, signature: reading.signature, starting: !!reading.localModels?.some(m => m.starting) })
+ this.noteReading(reading.tab, null)
+ } catch (error) {
+ this.failed = true
+ this.noteReading(reading.tab, error)
+ if (lostAccess(error)) lost.add(reading.tab)
+ }
+ }
+ } finally {
+ for (const reading of due) this.sending.delete(reading.tab)
+ }
+ return lost
+ }
+
+ /** What [beat] sends again for a Swarm: the reading the backend took, without whether folders hold uncommitted work. */
+ private taken(tab: string, body: Record<string, unknown>) {
+ const { harnessUncommitted: _fresh, ...kept } = body
+ this.lastSent.set(tab, { path: `/api/super-tabs/${tab}/machines/${this.machineId}`, body: kept, at: this.deps.now() })
+ }
+
+ /**
+ * One Swarm's reading through its own route, newest form first. Older backends reject the additive fields. The newest go
+ * alone first (the starts it is preparing, uncommitted work, readiness, progress, then privacy): a backend that takes
+ * everything else keeps the local models it reads. Their boolean-only view still safely refuses an unverified tool, and a
+ * reading without privacy is never one a Team machine passes on. [refused]: the batch already refused the newest form.
+ */
+ private async sendAlone({ tab, observation, localModels }: Reading, began: number, refused?: SuperTabHostError): Promise<void> {
+ const path = `/api/super-tabs/${tab}/machines/${this.machineId}`
+ const { quotaReasons: _reasons, ...unreasoned } = observation
+ const { harnessTokens: _tokens, ...untokened } = unreasoned
+ const { preparingStartIds: _preparing, ...unprepared } = untokened
+ const { harnessUncommitted: _uncommitted, ...known } = unprepared
+ const { readiness: _readiness, ...ready } = known
+ const { harnessProgress: _progress, ...withoutProgress } = ready
+ const { privacy: _privacy, ...withoutPrivacy } = withoutProgress
+ const { toolProbeErrors: _probeErrors, localModels: _served, refusedModels: _refused, ...legacy } = withoutPrivacy
+ // A backend before `starting` refuses it: it gets the models the grid lists, as before, not none at all.
+ const listedOnly = localModels?.some(m => m.starting)
+ ? [{ ...withoutPrivacy, localModels: localModels.filter(m => !m.starting) }] : []
+ const attempts = [...('quotaReasons' in observation ? [observation] : []), ...('harnessTokens' in unreasoned ? [unreasoned] : []), ...('preparingStartIds' in untokened ? [untokened] : []),
+ ...('harnessUncommitted' in unprepared ? [unprepared] : []),
+ ...('readiness' in known ? [known] : []), ready, ...('harnessProgress' in ready ? [withoutProgress] : []),
+ ...('privacy' in withoutProgress ? [withoutPrivacy] : []), ...listedOnly,
+ ...('toolProbeErrors' in observation || 'localModels' in observation || 'refusedModels' in observation ? [legacy] : [])]
+ const refuse = (attempt: Record<string, unknown>) => {
+ if ('quotaReasons' in attempt) this.quotaReasonsRefusedUntil = began + FIELD_REFUSED_MS
+ else if ('harnessTokens' in attempt) this.tokensRefusedUntil = began + FIELD_REFUSED_MS
+ else if ('preparingStartIds' in attempt) this.preparingRefusedUntil = began + FIELD_REFUSED_MS
+ else if ('harnessUncommitted' in attempt) this.uncommittedRefusedUntil = began + FIELD_REFUSED_MS
+ else if ('readiness' in attempt) this.readinessRefusedUntil = began + FIELD_REFUSED_MS
+ else if ('harnessProgress' in attempt) this.progressRefusedUntil = began + FIELD_REFUSED_MS
+ else if ('privacy' in attempt) this.privacyRefusedUntil = began + FIELD_REFUSED_MS
+ }
+ // The batch sent the newest form, which is the first attempt: it is not sent again.
+ if (refused) {
+ if (attempts.length === 1) throw refused
+ refuse(attempts[0]!)
+ }
+ for (const [index, attempt] of attempts.entries()) {
+ if (refused && index === 0) continue
+ try {
+ await this.deps.backend(this.core, 'POST', path, attempt)
+ this.taken(tab, attempt)
+ return
+ } catch (error) {
+ if (!fieldRefused(error) || index === attempts.length - 1) throw error
+ refuse(attempt)
+ }
+ }
+ }
+
private unconfirmedLoaded = false
/** The launches an earlier run of this daemon kept ([swarmUnconfirmedFile]), once, before the first pass looks at them. */
private loadUnconfirmed(): void {
diff --git a/docs/superpowers/plans/2026-10-30-machine-reports-fast-path.md b/docs/superpowers/plans/2026-10-30-machine-reports-fast-path.md
new file mode 100644
index 000000000..bc629139e
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-30-machine-reports-fast-path.md
@@ -0,0 +1,158 @@
+# Cheap machine reports (DB roadmap 2.5) — Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** A heartbeat that changes only a machine's reading costs one read and one write of its machine part, and a machine sends one report call per pass for all its Swarms.
+
+**Architecture:** Every full report (today's `observeMachine`) records in Redis what the next report needs to skip the Swarm load: the `main` stamp read before it loaded, the lent count, the next loss deadline, the Team-by-agents answer, the members to push to and the preparable start ids. A later report whose decisions cannot differ (spec section 1) reads and writes only its machine part under the version guard; anything else runs in full. Sweeps take one Redis clock per Swarm across pods. A batch route takes all of a machine's readings; the CLI sends one call per pass and falls back per Swarm.
+
+**Tech Stack:** TypeScript, Fastify, Prisma on MongoDB, ioredis, vitest, `mongodb-memory-server`.
+
+**Spec:** `docs/superpowers/specs/2026-10-30-machine-reports-fast-path-design.md` (decisions final).
+
+## Global Constraints
+
+- Sweep clock: Redis `SET swarm:swept:{tabId} <now> NX PX 30000` (`SWEEP_EVERY_MS`); one sweep per Swarm per 30 s across pods.
+- Record: Redis hash `swarm:machine-full:{tabId}`, one field per machine (JSON), key expires 2 minutes (`120_000` ms) after its last write.
+- `nextDeadline` is capped at 5 minutes (`300_000` ms) after the full report.
+- Fast-path lent: `elapsed × lentCount`, elapsed capped at `MACHINE_FRESH_MS` (45 000), added to the current week as `accrueLent` does.
+- Redis or stamps unreadable, a record missing, a bump owed, any failed condition or a version conflict: the full path.
+- Replies of the per-Swarm route are unchanged (`{ version }`). The per-Swarm route gains the fast path too.
+- Batch route: `POST /api/super-tabs/hosts/:machineId/readings`, body `{ readings: [{ tabId, observation }] }`, answer `{ results: [{ tabId, ok: true } | { tabId, error: { code, message, status } }] }`; at most 64 readings, body limit 8 MiB.
+- CLI: route answers 404 (or 403 `TEAM_SESSION_LIMITED` from an older backend) → one call per Swarm, ask again after 10 minutes (`600_000` ms).
+- Log `machine reports` per minute with `fast` and `full` counts, per pod.
+- Do not commit or push until Ken asks; remove what becomes dead (`store.ts swept`, `sweepDue`).
+- Commands from `backend/`: unit `npx vitest run <file>`; integration `npx vitest run --config vitest.super-tabs.config.ts <file>`. CLI: `npm test` (`test:core` only if `cli/src/core` or `cli/src/services` changes).
+
+## Rulings on the spec (found while reading the code)
+
+1. **Progress feeds the loss decision.** `liveness.ts reportRenewedAt` reads `harnessProgress`: a running harness whose entry disappears renews nothing (its lease ends sooner), and a reading with no progress map renews at `receivedAt`. So spec condition 2 also requires, against the stored reading: `harnessProgress` present exactly when it was, the same running harnesses with an entry, and no entry earlier than stored. Then the lease can only end later than the full report computed, and `nextDeadline` stays a safe bound (anti-goal: no task left unlost).
+2. **The batch body is validated per reading.** One strict body schema would 400 every Swarm for one bad reading; each `observation` goes through `machineObservationSchema` alone and a failure is that Swarm's `VALIDATION_ERROR`.
+3. **Team sessions:** the new route joins `TEAM_ROUTES` with scope `machine`.
+4. **Unreachable refusals** (`NOT_FOUND`, `JOIN_UNAVAILABLE`, `WORKSPACE_ACCESS_REVOKED`, `unreachable`) answer that reading as `{ code: 'NOT_AVAILABLE', status: 404 }`, as the per-Swarm route does.
+
+## Review Focus
+
+1. A reading whose running harness drops its progress entry while running (ruling 1): full path, so a lapsed lease is judged — Task 1 (`readingOnly`) and Task 4 tests.
+2. A batch with one Swarm the caller lost (removed member, revoked pool): that Swarm's error only, the others stored — Task 5 test.
+3. Redis answering stamps but failing the record or sweep commands: reports still succeed in full, sweeps fall back to this process's clock — Task 2 test.
+4. A full report that itself wrote beyond its reading (presence, a lost task): the next report runs in full once, then fast — Task 4 test.
+5. An older backend (404) or a Team daemon on one (403 `TEAM_SESSION_LIMITED`): the CLI keeps reporting per Swarm and starts still run — Task 6 test.
+
+---
+
+### Task 1: Shared pure pieces: stored reading, lent count, next loss
+
+**Files:**
+- Modify: `backend/src/superTabs/machines.ts` (add `storedObservation`, `readingOnly`)
+- Modify: `backend/src/superTabs/leaderboard/lent.ts` (split `accrueLent`)
+- Modify: `backend/src/superTabs/liveness.ts` (add `nextLossAt`)
+- Modify: `backend/src/superTabs/store.ts` (`observeMachine` uses `storedObservation` and the split lent)
+- Test: `src/superTabs/machines.test.ts`, `src/superTabs/leaderboard/lent.test.ts`, `src/superTabs/liveness.test.ts`
+
+**Interfaces:**
+- Produces:
+ - `storedObservation(pool: ContributedMachine, observation: MachineObservation, opts: { team: boolean; now: number; preparable: (startId: string) => boolean }): NonNullable<ContributedMachine['observation']>` — exactly what `observeMachine` stores today (running-only progress/tokens, filtered `preparingStartIds`, `observedPrivacy`, quotas by `contributedEngines`, `sessionKind`, `receivedAt`).
+ - `readingOnly(stored: ContributedMachine['observation'] | undefined, observation: MachineObservation): boolean` — spec condition 2 plus ruling 1 (same running and stopped sets, no `stoppedReservations`, same `preparingStartIds` as a set, same progress shape, no entry earlier).
+ - `lentAgents(state: SuperTabState, pool: ContributedMachine, running: readonly string[]): number` and `addLent(pool: ContributedMachine, now: number, count: number): void`; `accrueLent(state, pool, now)` becomes `addLent(pool, now, lentAgents(state, pool, pool.observation?.runningHarnessIds ?? []))`.
+ - `nextLossAt(state: SuperTabState, machineId: string): number | undefined` — the earliest `leaseEnd` of this machine's workers' active tasks (as `lostReason` reads them), undefined when none can lapse.
+
+- [ ] **Step 1: Write the failing tests**
+ - `machines.test.ts`: `storedObservation` keeps progress/tokens of running harnesses only, keeps a preparing id only when `preparable` says so, drops quotas of engines not contributed, sets `sessionKind` `'team'`/`'computer'`; `readingOnly` is false for a changed running set, a changed stopped set, any `stoppedReservations`, changed preparing ids, a progress map added or removed, a running harness losing its entry, an entry earlier than stored; true when only resources, quotas, tools, tokens, privacy or a later progress time differ.
+ - `lent.test.ts`: `addLent` after `lentAgents` equals `accrueLent` on the same state (busyMs, pruned weeks, no write when count or elapsed is 0).
+ - `liveness.test.ts`: `nextLossAt` equals the lease end of the one active task; for `t = nextLossAt - 1`, `loseTasks` loses nothing, and for `t = nextLossAt + 1` it loses that task; undefined with no active work on the machine.
+- [ ] **Step 2: Run** `npx vitest run src/superTabs/machines.test.ts src/superTabs/leaderboard/lent.test.ts src/superTabs/liveness.test.ts` — FAIL: not exported.
+- [ ] **Step 3: Implement** the four functions; `observeMachine` calls `storedObservation` (with `preparable` from `state.starts`) and `accrueLent` unchanged in behavior.
+- [ ] **Step 4: Run** the three files plus `npx vitest run --config vitest.super-tabs.config.ts integration/superTabLiveness.test.ts` — PASS.
+
+### Task 2: Report memory: the record, the shared sweep clock, the counts
+
+**Files:**
+- Create: `backend/src/superTabs/machineReports.ts`
+- Modify: `backend/src/superTabs/cache.ts` (export `readStamps(tabIds)` and `stampOwed(tabId)`)
+- Modify: `backend/src/routes/superTabs.ts` (`productionStore` wires `reports: redisReportMemory(bus.pub)`), `backend/src/server.ts` (start/stop the log)
+- Test: `src/superTabs/machineReports.test.ts`, `src/superTabs/cache.test.ts`
+
+**Interfaces:**
+- Produces:
+ - `interface ReportRecord { stamp: number; lentCount: number; removals: boolean; deadline: number; team: boolean; members: string[]; preparable: string[]; at: number }`
+ - `interface ReportMemory { get(tabId: string, machineId: string): Promise<ReportRecord | undefined>; set(tabId: string, machineId: string, record: ReportRecord): Promise<void>; sweepDue(tabId: string, now: number): Promise<boolean> }`
+ - `interface ReportRedis { hget(key, field): Promise<string | null>; hset(key, field, value): Promise<unknown>; pexpire(key, ms): Promise<unknown>; set(key, value, px: 'PX', ms: number, nx: 'NX'): Promise<'OK' | null> }`
+ - `redisReportMemory(redis: ReportRedis): ReportMemory` (a failed `sweepDue` falls back to a local clock, a failed `get` reads as missing, a failed `set` is logged once); `localReportMemory(now: () => number): ReportMemory` (one process, the default).
+ - `RECORD_TTL_MS = 120_000`, `DEADLINE_CAP_MS = 300_000`.
+ - `countReport(path: 'fast' | 'full'): void`, `startMachineReportLog(intervalMs = 60_000): () => void` (logs `'machine reports', { pod, windowMs, fast, full }`, flushes on stop).
+ - `cache.ts`: `readStamps(tabIds: readonly string[]): Promise<Map<string, Stamp> | null>` (null without stamps or past `STAMP_TIMEOUT_MS`), `stampOwed(tabId: string): boolean`.
+
+- [ ] **Step 1: Write the failing tests** (`machineReports.test.ts`, a fake Redis with a movable clock as in `pollMemory.test.ts`): a record set then read back equal; gone 120 s after the key's last write; two memories over one fake Redis: `sweepDue` true once per 30 s between them; Redis failing `set NX`: `sweepDue` answers from the local clock; failing `hget`: `get` is undefined; the log line counts and resets. `cache.test.ts`: `readStamps` null with stamps down; `stampOwed` true after a failed bump until paid.
+- [ ] **Step 2: Run** `npx vitest run src/superTabs/machineReports.test.ts src/superTabs/cache.test.ts` — FAIL.
+- [ ] **Step 3: Implement**; wire production and the log beside `startWsCounterLog()`.
+- [ ] **Step 4: Run** the same files — PASS.
+
+### Task 3: The full path records, and sweeps by the shared clock
+
+**Files:**
+- Modify: `backend/src/superTabs/store.ts` (`SuperTabStoreHooks.reports?: ReportMemory`; delete `swept`/`sweepDue`; `observeMachine` full path)
+- Create: `backend/integration/superTabMachineReports.test.ts` (add to `vitest.super-tabs.config.ts` include)
+
+**Interfaces:**
+- Consumes: Task 1 `lentAgents`, `nextLossAt`; Task 2 `ReportMemory`, `readStamps`, `DEADLINE_CAP_MS`.
+- Produces: after a full report commits, `reports.set(tabId, machineId, record)` with `stamp` = the `main` stamp read before the transaction (no record when unreadable), `lentCount = lentAgents(state, pool, pool.observation.runningHarnessIds)` after the mutations, `removals = state.machines.some(m => m.removal)`, `deadline = min(nextLossAt ?? ∞, now + DEADLINE_CAP_MS)`, `team = teamByRecord(state, machineId)`, `members = state.people ids`, `preparable` = this machine's dispatched starts with no harness and no error.
+
+- [ ] **Step 1: Write the failing tests** (fixture as `superTabLiveness.test.ts swarm()`, with a shared `localReportMemory`/fake-Redis memory passed in hooks): a report writes the record with each field above; a report whose own write moved `main` records the stamp read before it; two stores sharing one memory sweep a Swarm once per 30 s (count `sweepLiveness` effects: a stalled start failed once).
+- [ ] **Step 2: Run** `npx vitest run --config vitest.super-tabs.config.ts integration/superTabMachineReports.test.ts` — FAIL.
+- [ ] **Step 3: Implement.**
+- [ ] **Step 4: Run** it and `integration/superTabLiveness.test.ts` (its per-store sweep tests: each store gets its own memory by default) — PASS.
+
+### Task 4: The fast path
+
+**Files:**
+- Modify: `backend/src/superTabs/store.ts` (`observeMachine` tries `fastReading` first; counts)
+- Modify: `backend/src/superTabs/parts/load.ts` (`loadMachineParts(db, tabIds, machineId)`: `_id $in` through `aggregateRaw`)
+- Test: `backend/integration/superTabMachineReports.test.ts`
+
+**Interfaces:**
+- Consumes: Tasks 1–3.
+- Produces: `private fastReading(user, tabId, machineId, observation, host: { owned: Machine | null }, part?: Part<MachineData>): Promise<{ version: number } | null>` — null means run in full. Order: stamps readable and equal to `record.stamp`; no bump owed; `now < record.deadline`; `!record.removals`; sweep not due (decided once per report, before either path); `hostFor` + `notTeamMachine` with `record.team`; billing; `store.row`; part live with `contribution`, `contributorId === user.sub`, `generation` equal, no `removal`; `readingOnly`; then `addLent(pool, now, record.lentCount)`, `storedObservation(..., preparable: id => record.preparable.includes(id))`, `readingClearsSuspensions` (changed → null); `guardedWrite` update in `swarmTransaction` (a conflict → null); after commit `hooks.swarmChanged({ tabId, userIds: record.members, lastChangedAt: stamp, parts: ['machine'] })`.
+
+- [ ] **Step 1: Write the failing tests:**
+ - **Same as full:** two Swarms written from one state (as `superTabRead.test.ts scene` does), one full report each at the same clock, then the same heartbeat at the same clock: Swarm A's store with the record (fast), Swarm B's with an empty memory (full). The machine parts' `data` are equal (including `lent`), and the `swarmChanged` calls are equal.
+ - **Each reason forces full** (assert through `countReport` or a spy hook, and the part read count): running set changed, stopped set changed, stop proof, preparing changed, generation changed, a command since the last full report, pending removal, sweep due, deadline passed, record missing, stamps down, bump owed, a suspension cleared, a running harness's progress entry dropped (ruling 1), a full report that wrote presence (next one full, the one after fast).
+ - **Deadline:** a worker whose lease ends between two heartbeats is lost by the first report after it.
+ - **Lent:** N fast heartbeats give the same `busyMs` as N full ones, including right after a full report that changed the running set.
+ - **Team machine:** the owner's ordinary sign-in reporting for a machine whose agent is on a Team account gets `TEAM_HOST_ONLY`.
+ - **Revoked:** a regenerated pool answers `CONTRIBUTION_REVOKED`.
+ - **Cost:** a fast heartbeat reads only `SwarmMachine` among part collections (plus `SuperTab`), writes once, bumps only `machine` (`partReads` and `MemoryStamps.bumps`).
+- [ ] **Step 2: Run** the file — FAIL.
+- [ ] **Step 3: Implement.**
+- [ ] **Step 4: Run** the file, `superTabLiveness`, `superTabTeam`, `superTabs`, `superTabCache`, `superTabRead` — PASS.
+
+### Task 5: The batch route
+
+**Files:**
+- Modify: `backend/src/superTabs/store.ts` (`observeMachines`), `backend/src/routes/superTabs.ts` (route, schema, body limit), `backend/src/lib/teamSession.ts` (`TEAM_ROUTES`)
+- Test: `src/routes/superTabs.test.ts`, `src/lib/teamSession.test.ts`, `backend/integration/superTabMachineReports.test.ts`
+
+**Interfaces:**
+- Consumes: Task 4 `fastReading`, the full path.
+- Produces: `observeMachines(user: AuthUser, machineId: string, readings: Array<{ tabId: string; observation: unknown }>): Promise<{ results: Array<{ tabId: string; ok: true } | { tabId: string; error: { code: string; message: string; status: number } }> }>` — identity (`teamHost`, owned row, `hostFor` without a Swarm) once; fast candidates' parts in one `loadMachineParts`; each reading validated alone, then fast else full, one at a time; errors per reading (ruling 4 for unreachable ones).
+
+- [ ] **Step 1: Write the failing tests:** route test (stub store): 64 readings accepted, 65 refused 400, a non-object reading answered per Swarm, team route listed; integration: one fast, one full, one revoked in one call → per-Swarm results; the `Machine` row read once for the call; a 21-Swarm call stores 21 readings.
+- [ ] **Step 2: Run** — FAIL. **Step 3: Implement.** **Step 4: Run** — PASS.
+
+### Task 6: The CLI sends one call per pass
+
+**Files:**
+- Modify: `cli/src/superTabs/machines.ts` (collect → send → starts; `sendAlone` ladder; `beat()` batches; `readingsRefusedUntil`)
+- Test: `cli/src/superTabs/machines.spec.ts`, `cli/src/superTabs/host.spec.ts` (`setup()` answers `/readings`)
+
+**Interfaces:**
+- Produces: `READINGS_RETRY_MS = 600_000`; `sendAlone(tab, observation, localModels, began, refused?)` — today's field-fallback ladder, starting after the attempt the batch refused.
+
+- [ ] **Step 1: Write the failing tests** (fixture's backend gets a batch emulator that answers per reading from the per-Swarm mock, so existing tests keep their meaning): one call per pass for two Swarms; `beat()` sends one call; a 404 (and a 403 `TEAM_SESSION_LIMITED`) → per-Swarm calls, no batch again until 10 minutes, then the batch; a `VALIDATION_ERROR` for one Swarm resends only it through the ladder; a lost-access error for one Swarm launches only the other Swarm's start.
+- [ ] **Step 2: Run** `npx vitest run src/superTabs/machines.spec.ts src/superTabs/host.spec.ts` — FAIL. **Step 3: Implement.** **Step 4: Run** `npm test` and `npm run typecheck` in `cli/` — PASS.
+
+### Task 7: Whole checks
+
+- [ ] `backend/`: `npm run typecheck`, `npm test`, `npm run test:super-tabs`; compare failures with a clean `origin/main`.
+- [ ] `cli/`: `npm test`, `npm run typecheck`.
+- [ ] After release (Ken): the spec's Measuring section, before and after.
diff --git a/docs/superpowers/specs/2026-10-30-machine-reports-fast-path-design.md b/docs/superpowers/specs/2026-10-30-machine-reports-fast-path-design.md
new file mode 100644
index 000000000..1122196d4
--- /dev/null
+++ b/docs/superpowers/specs/2026-10-30-machine-reports-fast-path-design.md
@@ -0,0 +1,198 @@
+# Design: cheap machine reports — a backend fast path and one report call per machine (DB roadmap 2.5)
+
+Date: 2026-10-30. Code read at `origin/main` (e08d53f96). Spec 4 in the CPU-ranked series. It replaces the roadmap's
+"project members and agents to the fields they use": those reads are 8% of what a report reads, so projecting them
+saves almost nothing.
+
+## Why
+
+`POST /api/super-tabs/:tabId/machines/:machineId` is the machine report. Every CLI daemon sends one per Swarm it
+contributes to (`cli/src/superTabs/machines.ts`): a pass every 5 s compares a signature (running and stopped
+harnesses, reservations, pending and preparing starts, pause, privacy, local models) and sends when it changed, when a
+model is starting, or when 15 s passed (`MACHINE_HEARTBEAT_MS`, so the Desktop can tell a silent machine from a quiet
+one). `beat()` resends the last reading while a slow pass runs.
+
+**Prod, 2026-10-30 (60 min of logs, 3 pods; Mongo read-only):**
+- 1.36 s of event-loop time per pod-minute, 12% of backend busy time; 136 reports per pod-minute, ~10 ms CPU and
+ ~10 queries each in one transaction; p95 85 ms; 7.2 MB read per pod-minute.
+- 24 machines report into 84 machine–Swarm pairs. Heartbeats alone are 84 × 4 = 336 a minute of the ~408 observed:
+ **about 82% of reports are heartbeats**.
+- Two machines contribute to 20 and 21 Swarms each and send ~40% of all reports, the same machine reading 20 times
+ every 15 s.
+- What a report reads per pod-minute: progress 2.4 MB (spec 2 removes it), every machine part of the Swarm 2.4 MB,
+ running and Lead conversations 1.3 MB, agents + members 0.6 MB, access checks (`Machine` ×2, `HarnessSession`,
+ `User`, `SuperTab`) ~0.3 MB. Reports do commit: 84 of the 92 live machine parts had a `receivedAt` under 2 minutes
+ old (Mongo, 2026-10-30).
+
+A heartbeat whose decisions are the same as last time pays the full price: load the Swarm, freeze, copy, run the
+checks, diff everything, and write only the machine's reading.
+
+## Goals
+
+- A heartbeat that changes nothing but the machine's reading costs one read and one write of that machine part, with
+ no Swarm load.
+- A machine sends one report call per pass for all its Swarms, not one per Swarm.
+- Everything a report decides (lost tasks, presence, removals, sweeps, lent hours, suspensions, cleanups) comes out
+ as it does today; liveness and the Desktop's offline rule are unchanged.
+
+## Non-goals
+
+- Changing the 15 s heartbeat, or deriving liveness from `adapter-ws` presence. Agents' Down state reads each Swarm's
+ `receivedAt`; changing that is a separate design.
+- The access checks' own cost beyond doing them once per call.
+- Reads other than the report's (`changes`, `messages`): their own specs.
+
+## Anti-goals
+
+- A fast path that skips a decision the full path would have made: a task not lost, a removal not ended, a sweep not
+ run, lent hours not counted.
+- A machine that loses access to a Swarm still getting its readings accepted.
+
+## Design decided
+
+### 1. When a report may take the fast path
+
+A report is a **reading-only report** when all of these hold:
+1. The Swarm's state is what this machine's last full report saw: the Swarm's `main` stamp now equals the `main`
+ stamp that full report read **before** it loaded (recorded in step 3). A report's own write that changes anything
+ beyond its reading moves `main`, so the next report runs in full once and records the new stamp.
+2. The decisions the observation feeds are unchanged against the stored machine part: the same `runningHarnessIds` and
+ `stoppedHarnessIds` (as sets), no `stoppedReservations`, the same `preparingStartIds`, the same pool `generation`
+ and contributor.
+3. The record says no machine of the Swarm had a pending `removal` (`expireRemovals` is time-based).
+4. No sweep is due for the Swarm. *(Author's call after the builder check.)* The sweep clock becomes one per Swarm
+ across pods (Redis `SET NX PX 30000`), not one per process (`store.ts swept`): today each of 3 pods sweeps each
+ Swarm every 30 s, which also forces ~2 full reports per Swarm per pod-minute. One clock means one sweep per Swarm
+ per 30 s, as the roadmap's Phase 4 item 5 already asks.
+5. `now` is before the record's `nextDeadline`: the earliest time-based judgment the full report saw coming for this
+ machine's workers (`liveness.ts lostReason`: lease end, quiet or report-renewal limits), capped at 5 minutes after
+ the full report. *(Author's call.)* This replaces a fixed 60 s window, which alone would force one full report in
+ four.
+6. Stamps are readable (Redis up). Without them every report runs in full, as today.
+
+Time-based losses of this machine's workers are still judged on time: by the deadline in condition 5, and by the
+Swarm's sweep every 30 s (`sweepLiveness`), as they are today when a report comes late. Losses a changed reading
+causes (a harness stopped) change the running or stopped set, so they run in full at once.
+
+In a Swarm that changes often (the busiest takes a command about every 9 s) condition 1 sends most reports in full;
+the fast path pays off in quiet Swarms and for quiet machines, and batching (section 4) still cuts calls and access
+checks everywhere. The per-minute fast/full counts say how the split comes out.
+
+Everything else in the observation (`resources`, `quotas`, `harnessProgress`, `harnessTokens`, `privacy`, tools,
+local models, `receivedAt`) is the machine's reading (`parts/commit.ts withoutReading`: `observation` and `lent`), and
+the full path writes it without touching anything else when the decisions are unchanged. `recordAttempts`, `endRuns`
+and `recordEvidence` act only on conversation changes, so a reading-only report leaves them idle.
+
+### 2. What the fast path does
+
+In one transaction on that one part:
+- read the machine part by id (`${tabId}:${machineId}`) and check conditions 2 and the pool's contributor/generation
+ (`CONTRIBUTION_REVOKED` as today);
+- build the new observation exactly as `observeMachine` does for the pool alone (`observedPrivacy`, the running-only
+ progress and tokens, quotas filtered by `contributedEngines`, `sessionKind`, `receivedAt`);
+- `readingClearsSuspensions(pool, now)` runs too; if it changes the pool beyond its reading, the report runs in full;
+- lent hours: `accrueLent` needs the Swarm's conversations, so the full report records `lentCount`: the number of this
+ machine's agents `accrueLent` would count **for the next report**, i.e. over the running set just stored and the
+ conversations as they stand after its commit (not the count it used for its own accrual, which read the previous
+ running set). The fast path adds `elapsed × lentCount` to the current week the way `accrueLent` does (elapsed capped
+ at `MACHINE_FRESH_MS`). With the state unchanged (condition 1) and the same running set (condition 2), that is what
+ `accrueLent` would add;
+- what the full path reads from the Swarm for a reading-only report, all fixed while `main` is unchanged, comes from
+ the record: whether the machine is a Team account by its agents (`teamByRecord`, the second `hostFor` check, security
+ re-check of #2089), the member ids the `swarm_changed` push with `parts: ['machine']` goes to (`membersToTell`), and
+ the start ids `preparingStartIds` may name (the filter on `state.starts`). The `SuperTab` row check (`store.row`:
+ the Swarm exists in the caller's environment) stays a read;
+- the same `swarm_changed` push to the same members as a reading-only full report sends;
+- a version-guarded update of the machine part (`guardedWrite`), which records a `machine`-scope stamp bump as a
+ reading-only write does today. A version conflict or any failed condition falls back to the full path.
+
+The access checks (`hostFor`: the machine is the caller's, alive, billing allows the data plane) run on the fast path
+as on the full one.
+
+### 3. The full-path record
+
+After every full report commits, the backend records for `(tabId, machineId)`: the `main` stamp read before the load,
+`lentCount`, whether any machine of the Swarm had a pending removal, `nextDeadline`, the Team-by-agents answer, the
+member ids to push to, the start ids preparing may name, and the time. A process whose stamp bump failed after a
+commit (cache.ts `owed`) runs every report of that Swarm in full until the bump is paid. It lives in Redis
+(`swarm:machine-full:{tabId}`, one field per machine, expiring 2 minutes after the last write), beside the stamps.
+Missing or unreadable means the next report runs in full.
+
+### 4. One report call per machine
+
+- A new route `POST /api/super-tabs/hosts/:machineId/readings` takes `{ readings: [{ tabId, observation }] }` (the same
+ observation schema as the per-Swarm route) and answers `{ results: [{ tabId, ok: true } | { tabId, error: { code,
+ message, status } }] }`. Each reading goes through the same code as the per-Swarm route (fast path, else full).
+ The caller's identity and the machine's ownership and billing are checked once per call; each Swarm's own access
+ is still checked per reading.
+- Fast-path readings of one call share one read of their machine parts (`_id $in`); full-path ones run one at a time,
+ as they would arrive today.
+- The CLI's pass collects the readings it would send and sends them in one call (`beat()` too), then runs each Swarm's
+ pending starts as today; a Swarm whose reading came back with a lost-access error runs no starts. A per-reading error is
+ handled as today's per-Swarm error for that Swarm (`lostAccess` stops that Swarm's starts; a `VALIDATION_ERROR` sends
+ that Swarm's reading again alone through the old route, so today's field fallbacks keep working).
+- A backend without the route answers 404: the CLI goes back to one call per Swarm and asks again after 10 minutes.
+ The per-Swarm route stays for older CLIs.
+
+## Left to the builder
+
+- Code layout: where the fast path lives (a function beside `observeMachine`, shared by both routes).
+- The exact Redis shape of the record, and the batch size limit of the new route (at least the 21 Swarms seen today).
+- How the CLI pass is reshaped to collect readings before sending, as long as starts are still picked up and the
+ signature/heartbeat rules are unchanged.
+
+## Tests
+
+- **Same decisions, fast or full:** for each reason a report must run in full (running set changed, stopped set
+ changed, stop proof, preparing changed, generation changed, Swarm changed since the last full report, pending removal,
+ sweep due, deadline passed, record missing, Redis down, bump owed, suspension cleared), the report takes the full
+ path. For a reading-only heartbeat before any deadline, the stored machine part after the fast path equals what the
+ full path would store (run both on copies of one Swarm and compare the machine part, including `lent`), and the
+ same push goes to the same members.
+- **Deadlines:** a worker whose lease ends between two heartbeats is lost by the first report after the deadline,
+ which runs in full.
+- **One sweep per Swarm:** two stores sharing one Redis sweep a Swarm once per 30 s between them.
+- **Lent hours:** a machine with counted agents gets the same `busyMs` after N fast heartbeats as after N full ones,
+ including right after a full report that changed the running set.
+- **Team machine:** a fast-path reading from the owner's ordinary sign-in for a machine whose agents run on a Team
+ account is refused as the full path refuses it.
+- **Revoked contribution:** a fast-path reading of a revoked or regenerated pool answers `CONTRIBUTION_REVOKED`.
+- **Cost:** a reading-only heartbeat reads only the machine part among the Swarm's part collections (plus the
+ `SuperTab` row), and writes once (`integration/superTabCache.test.ts` `partReads` style); it bumps the `machine` stamp
+ only.
+- **Batch route:** mixed results (one fast, one full, one revoked) in one call come back per Swarm; identity and
+ ownership are checked once; a 21-Swarm call works.
+- **CLI:** one call per pass for several Swarms; `beat()` batches; 404 falls back to per-Swarm calls and retries after
+ 10 minutes; a `VALIDATION_ERROR` for one Swarm resends it alone; a lost-access error for one Swarm stops only its
+ starts. Run `npm run test:core` if the change touches `src/core` or `src/services`.
+
+## Measuring
+
+Before and after, same weekday hours, prod logs:
+- calls per pod-minute to the per-Swarm route and to the new one (today 136 per pod-minute per-Swarm);
+- busy ms per pod-minute of both routes together (today 1.36 s);
+- queries and MB under them in `db by shape` (today ~10 queries a report, 7.2 MB per pod-minute);
+- with #2368's `callers`/`inclusive`, `observeMachine` and its loads in `cpu profile`.
+
+**Done when** the tests pass, the fast/full counts are logged per minute, and prod shows the report routes' busy time
+and queries falling, reported before and after (expected: about 2× once most daemons run the new CLI; ~24 calls
+instead of ~84 every 15 s; the fast share is highest in quiet Swarms and lowest in the busiest). The number is
+reported, not a merge gate.
+
+## Design (AGENTS.md rules)
+
+- **Growth:** the Redis record expires 2 minutes after its last write; nothing new in Mongo.
+- **Concurrency:** the fast path writes one machine part with its version guard, as the full path does; a conflict
+ falls back to the full path. Two pods may both run a full report for one Swarm, as today.
+- **Read cost:** a reading-only heartbeat reads one part; a batch reads all its fast-path parts in one query.
+- **Scale:** at 10× machines or Swarms per machine, calls grow with machines, not pairs, and most readings cost one
+ part. The first warning is the full-path share (log it per minute: fast vs full readings) and Redis calls.
+- **Compatibility:** old CLIs keep the per-Swarm route, which gains the fast path too. A new CLI on an old backend
+ falls back on 404. Replies of the per-Swarm route are unchanged.
+- **Migration + rollback:** nothing stored in Mongo changes shape (`lent` is written as today). Roll back the backend
+ and new CLIs fall back on 404; roll back the CLI and the per-Swarm route still has the fast path.
+- **Audience:** the machine's owner only, checked per call; each Swarm's contribution checked per reading.
+- **Identity:** the caller's own session, as today; never the machine owner's login on someone else's behalf.
+- **Failure:** any failed condition, conflict or missing record runs the full path; a batch reading's error is
+ answered for that Swarm, never dropped. The per-minute fast/full counts say when the fast path stops working.
+- **Platforms:** backend; CLI on macOS, Linux and Windows (no platform-specific code).