Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

perf(swarm): messages read what a page shows, receipts kept forever, thread summaries stored (spec 5 A, C, D) (#2391)

0xken15ed12a330cc72
diff --git a/backend/integration/superTabDeliveryCompaction.test.ts b/backend/integration/superTabDeliveryCompaction.test.ts
new file mode 100644
index 000000000..c9136a29f
--- /dev/null
+++ b/backend/integration/superTabDeliveryCompaction.test.ts
@@ -0,0 +1,251 @@
+import { randomUUID } from 'node:crypto'
+import { afterAll, beforeAll, describe, expect, it } from 'vitest'
+import type { MongoMemoryReplSet } from 'mongodb-memory-server-core'
+import type { Prisma, PrismaClient, SuperTabDelivery } from '@prisma/client'
+import type { AuthUser } from '../src/lib/ssoAuth.js'
+import { SuperTabStore } from '../src/superTabs/store.js'
+import type { Command } from '../src/superTabs/commands.js'
+import { COMPACT_AFTER_MS, compactDeliveries, startDeliveryCompaction } from '../src/superTabs/deliveryCompaction.js'
+import { listNamedIndexes, withoutHints } from '../src/superTabs/parts/indexed.js'
+import { planDatabase, plan, profiled, reads } from './plans.test-support.js'
+
+// Spec 5 phase C (docs/superpowers/specs/2026-10-30-swarm-messages-delta-and-retention-design.md): finished deliveries move
+// into their messages' receipts after 14 days; pages, hosts and notices read them as the rows they were.
+let replica: MongoMemoryReplSet
+let db: PrismaClient
+let store: SuperTabStore
+const owner: AuthUser = { sub: '600000000000000000000001', email: '[email protected]', autonomousEnv: 'prod', role: 'user' }
+const host: AuthUser = { sub: '600000000000000000000003', email: '[email protected]', autonomousEnv: 'prod', role: 'user' }
+const config = { engine: 'codex', model: 'fixture-model', effort: 'high', contextWindow: 200000, engineVersion: 'fixture' }
+const DAY = 86_400_000
+
+beforeAll(async () => {
+ ({ replica, db } = await planDatabase('swarm_compaction_test'))
+ store = new SuperTabStore(db)
+ await listNamedIndexes(db)
+ for (const user of [owner, host]) await db.user.create({ data: { id: user.sub, externalId: `fixture-${user.sub}`, email: user.email, autonomousEnv: 'prod' } })
+ await db.machine.create({ data: { machineId: 'test-host', userId: host.sub, apiKey: 'fixture-only', managerId: 'fixture', autonomousEnv: 'prod', authMode: 'remote' } })
+ await db.machineShare.create({ data: { machineId: 'test-host', ownerId: host.sub, autonomousEnv: 'prod', emails: [owner.email] } })
+})
+afterAll(async () => {
+ await db?.$disconnect()
+ await replica?.stop()
+})
+
+const run = (tab: string, command: Command, user = owner, asAgentId?: string) => store.execute(user, tab, { operationId: randomUUID(), command }, asAgentId)
+async function fixture() {
+ const tab = (await store.create(owner, randomUUID(), 'Receipts')).id
+ const lead = String((await run(tab, { type: 'agent.add', conversationId: 'lead', machineId: 'test-host', harnessId: randomUUID(), name: 'Lead',
+ configuration: config, lead: true })).result.agentId)
+ return { tab, lead }
+}
+/** A message to the Lead and its delivery, claimed by the Lead's host. */
+async function sent(tab: string, lead: string, text: string) {
+ const messageId = String((await run(tab, { type: 'message.send', conversationId: 'lead', text, replyTo: null, recipientIds: [] })).result.messageId)
+ const delivery = (await store.deliveries(host, tab, lead)).find(d => d.messageId === messageId)!
+ await store.delivery(host, tab, lead, delivery.id, { type: 'claim', claimId: `claim-${text}` })
+ return { messageId, deliveryId: delivery.id, claimId: `claim-${text}` }
+}
+/** A message to the Lead typed and answered. */
+async function answered(tab: string, lead: string, text: string) {
+ const one = await sent(tab, lead, text)
+ await store.delivery(host, tab, lead, one.deliveryId, { type: 'ack', claimId: one.claimId, receipt: `typed-${text}`, uncertain: false })
+ await run(tab, { type: 'message.send', conversationId: 'lead', text: `Re: ${text}`, replyTo: one.messageId, recipientIds: [] }, host, lead)
+ return one
+}
+/** Moves these deliveries' last change `days` back, as if nothing touched them since. */
+async function age(ids: string[], days: number) {
+ await db.$runCommandRaw({ update: 'SuperTabDelivery', updates: ids.map(id => ({ q: { _id: id },
+ u: { $set: { updatedAt: { $date: new Date(Date.now() - days * DAY).toISOString() } } } })) } as Prisma.InputJsonObject)
+}
+/** A delivery written straight into the database, for statuses no short flow reaches. */
+async function row(tab: string, lead: string, status: string, fields: Partial<SuperTabDelivery> = {}) {
+ const messageId = `${tab}-${randomUUID()}`
+ await db.superTabMessage.create({ data: { id: messageId, tabId: tab, conversationId: 'lead', sequence: 500, text: status, replyTo: null,
+ recipientIds: [fields.agentId ?? lead], mentions: [], thread: messageId, author: { kind: 'person', id: owner.sub, name: owner.email, membershipGeneration: 'g' } } })
+ return db.superTabDelivery.create({ data: { id: `${messageId}-d`, tabId: tab, conversationId: 'lead', messageId, agentId: lead, machineId: 'test-host',
+ machineOwnerId: host.sub, membershipGeneration: 'g', authorKind: 'person', authorId: owner.sub, authorGeneration: 'g', status, ...fields } })
+}
+const receiptOf = async (messageId: string) => (await db.$runCommandRaw({ find: 'SuperTabReceipt', filter: { _id: messageId } }) as unknown as
+ { cursor: { firstBatch: Array<{ receipts: Record<string, { deliveryId: string; status: string }>; deliveryIds: string[] }> } }).cursor.firstBatch[0]
+
+describe('compacting finished deliveries (spec 5 C)', () => {
+ it('moves only answered, cancelled and superseded deliveries whose last change is over 14 days old', async () => {
+ const { tab, lead } = await fixture()
+ const final = await Promise.all(['answered', 'cancelled', 'superseded'].map(status => row(tab, lead, status)))
+ const kept = await Promise.all(['delivered', 'confirmed', 'queued', 'claimed', 'uncertain'].map(status => row(tab, lead, status)))
+ const recent = await row(tab, lead, 'answered')
+ await age([...final, ...kept].map(d => d.id), 15)
+ await age([recent.id], 13)
+ // Delivered and confirmed ones stay rows, counted as unfinished.
+ expect(await compactDeliveries(db, Date.now())).toEqual({ compacted: 3, remaining: 0, unfinished: 2 })
+ const left = await db.superTabDelivery.findMany({ where: { tabId: tab }, select: { id: true } })
+ expect(left.map(d => d.id).sort()).toEqual([...kept, recent].map(d => d.id).sort())
+ for (const delivery of final) {
+ expect(await receiptOf(delivery.messageId)).toMatchObject({ deliveryIds: [delivery.id],
+ receipts: { [lead]: { deliveryId: delivery.id, status: delivery.status, machineOwnerId: host.sub } } })
+ }
+ expect(await compactDeliveries(db, Date.now())).toMatchObject({ compacted: 0, remaining: 0 })
+ })
+
+ it('keeps every agent of one message, compacted one after another or at once', async () => {
+ const { tab, lead } = await fixture()
+ const first = await row(tab, lead, 'answered')
+ const other = (agentId: string) => db.superTabDelivery.create({ data: { ...first, id: `${first.messageId}-${agentId}`, agentId, updatedAt: undefined } })
+ const second = await other('agent-b')
+ await age([first.id], 15)
+ await compactDeliveries(db, Date.now())
+ await age([second.id], 15)
+ await compactDeliveries(db, Date.now())
+ const third = await other('agent-c'), fourth = await other('agent-d')
+ await age([third.id, fourth.id], 15)
+ await Promise.allSettled([compactDeliveries(db, Date.now(), 1), compactDeliveries(db, Date.now(), 1)])
+ await compactDeliveries(db, Date.now())
+ const receipt = await receiptOf(first.messageId)
+ expect(Object.keys(receipt!.receipts).sort()).toEqual([lead, 'agent-b', 'agent-c', 'agent-d'].sort())
+ expect(receipt!.deliveryIds.sort()).toEqual([first, second, third, fourth].map(d => d.id).sort())
+ expect(await db.superTabDelivery.count({ where: { messageId: first.messageId } })).toBe(0)
+ })
+
+ it('leaves a delivery that changed after it was read, and moves it once it is old again', async () => {
+ const { tab, lead } = await fixture()
+ const delivery = await row(tab, lead, 'cancelled')
+ await age([delivery.id], 15)
+ let changed = false
+ // Another writer settles the delivery after the batch read it and before it writes (a host typing a held one).
+ const racing = new Proxy(db, { get(target, key, receiver) {
+ if (key !== '$transaction') return Reflect.get(target, key, receiver)
+ return (body: (tx: PrismaClient) => Promise<unknown>, options: unknown) => (target.$transaction as (b: unknown, o: unknown) => Promise<unknown>)(
+ (tx: PrismaClient) => body(new Proxy(tx, { get(inner, name, innerReceiver) {
+ const value = Reflect.get(inner, name, innerReceiver)
+ if (name !== '$runCommandRaw' || changed) return value
+ return async (...args: unknown[]) => {
+ changed = true
+ await db.superTabDelivery.update({ where: { id: delivery.id }, data: { receipt: 'typed late' } })
+ return (value as (...a: unknown[]) => unknown).apply(inner, args)
+ }
+ } })), options)
+ } }) as PrismaClient
+ expect(await compactDeliveries(racing, Date.now())).toMatchObject({ compacted: 0, remaining: 0 })
+ expect(changed).toBe(true)
+ expect(await db.superTabDelivery.findUnique({ where: { id: delivery.id } })).toMatchObject({ receipt: 'typed late' })
+ expect(await receiptOf(delivery.messageId)).toBeUndefined()
+ await age([delivery.id], 15)
+ expect(await compactDeliveries(db, Date.now())).toMatchObject({ compacted: 1, remaining: 0 })
+ })
+
+ it('reads a page the same before and after compaction, for a person and for the Lead', async () => {
+ const { tab, lead } = await fixture()
+ const done = [await answered(tab, lead, 'One'), await answered(tab, lead, 'Two')]
+ await sent(tab, lead, 'Waiting')
+ await age(done.map(d => d.deliveryId), 15)
+ const before = [await store.messages(owner, tab, 'lead', {}), await store.messages(host, tab, 'lead', {}, lead)]
+ expect((await compactDeliveries(db, Date.now())).compacted).toBe(2)
+ expect([await store.messages(owner, tab, 'lead', {}), await store.messages(host, tab, 'lead', {}, lead)]).toEqual(before)
+ })
+
+ it('stops after 20 batches and counts what is left as remaining', async () => {
+ const { tab, lead } = await fixture()
+ const rows = await Promise.all(Array.from({ length: 23 }, () => row(tab, lead, 'answered')))
+ await age(rows.map(d => d.id), 15)
+ expect(await compactDeliveries(db, Date.now(), 1)).toMatchObject({ compacted: 20, remaining: 3 })
+ expect(await compactDeliveries(db, Date.now(), 1)).toMatchObject({ compacted: 3, remaining: 0 })
+ })
+
+ it('leaves a delivery whose agent id cannot name a receipt as a row, without holding the others back', async () => {
+ const { tab, lead } = await fixture()
+ const odd = await row(tab, lead, 'answered', { agentId: 'agent.with.dots' })
+ const rows = [await row(tab, lead, 'answered'), await row(tab, lead, 'cancelled')]
+ await age([odd.id], 16)
+ await age(rows.map(d => d.id), 15)
+ try {
+ expect(await compactDeliveries(db, Date.now(), 1)).toMatchObject({ compacted: 2, remaining: 1 })
+ expect(await db.superTabDelivery.findUnique({ where: { id: odd.id } })).not.toBeNull()
+ expect(await receiptOf(odd.messageId)).toBeUndefined()
+ } finally {
+ await db.superTabDelivery.delete({ where: { id: odd.id } })
+ }
+ })
+
+ it('reads only what it moves, by its index', async () => {
+ const { tab, lead } = await fixture()
+ await Promise.all(Array.from({ length: 30 }, () => row(tab, lead, 'answered')))
+ const old = await Promise.all(Array.from({ length: 4 }, () => row(tab, lead, 'cancelled')))
+ await age(old.map(d => d.id), 15)
+ // As the worker reads, which lists no named index: the planner picks it.
+ const entries = await profiled(db, () => withoutHints(() => compactDeliveries(db, Date.now())))
+ const found = reads(entries, 'SuperTabDelivery', ['status', 'updatedAt'], ['_id'])
+ expect(found.length, JSON.stringify(plan(found))).toBeGreaterThan(0)
+ for (const entry of found) {
+ expect(entry.docsExamined, JSON.stringify(plan(found))).toBeLessThanOrEqual(4)
+ expect(entry.planSummary, JSON.stringify(plan(found))).toContain('status: 1, updatedAt: 1')
+ }
+ })
+
+ it('starts nothing when the kill switch is off', () => {
+ const untouchable = new Proxy({}, { get() { throw new Error('read the database') } }) as PrismaClient
+ const stop = startDeliveryCompaction(untouchable, 'off')
+ stop()
+ expect(COMPACT_AFTER_MS).toBe(14 * DAY)
+ })
+})
+
+describe('hosts read compacted deliveries (spec 5 C)', () => {
+ it('a host back after weeks offline gets a compacted delivery\'s final status; another person is refused', async () => {
+ const { tab, lead } = await fixture()
+ const done = await answered(tab, lead, 'Old')
+ await age([done.deliveryId], 15)
+ await compactDeliveries(db, Date.now())
+ expect(await db.superTabDelivery.findUnique({ where: { id: done.deliveryId } })).toBeNull()
+ expect(await store.deliveryReceipt(host, tab, lead, done.deliveryId)).toEqual({ id: done.deliveryId, status: 'answered' })
+ await expect(store.deliveryReceipt(owner, tab, lead, done.deliveryId)).rejects.toMatchObject({ code: 'FORBIDDEN', status: 403 })
+ await expect(store.deliveryReceipt(host, tab, 'another-agent', done.deliveryId)).rejects.toMatchObject({ status: 404 })
+ await expect(store.deliveryReceipt(host, tab, lead, 'not-a-delivery')).rejects.toMatchObject({ code: 'NOT_FOUND', status: 404 })
+ // A compacted delivery is finished: it is resolved, as its row was.
+ await expect(store.delivery(host, tab, lead, done.deliveryId, { type: 'ack', claimId: done.claimId, receipt: 'again', uncertain: false }))
+ .rejects.toMatchObject({ code: 'DELIVERY_RESOLVED' })
+ })
+
+ it('finds a compacted delivery by its index, examining only its receipt, and a page\'s receipts by message id', async () => {
+ const { tab, lead } = await fixture()
+ const done = await Promise.all(['A', 'B', 'C'].map(text => answered(tab, lead, text)))
+ await age(done.map(d => d.deliveryId), 15)
+ await compactDeliveries(db, Date.now())
+ const hostReads = await profiled(db, () => store.deliveryReceipt(host, tab, lead, done[1]!.deliveryId))
+ const byDelivery = reads(hostReads, 'SuperTabReceipt', ['deliveryIds'])
+ expect(byDelivery, JSON.stringify(plan(byDelivery))).toHaveLength(1)
+ expect(byDelivery[0]!.docsExamined, JSON.stringify(plan(byDelivery))).toBeLessThanOrEqual(1)
+ expect(byDelivery[0]!.command?.hint, `names its index: ${JSON.stringify(plan(byDelivery))}`).toBeTruthy()
+ const page = await profiled(db, () => store.messages(owner, tab, 'lead', {}))
+ const byMessage = reads(page, 'SuperTabReceipt', ['_id'])
+ expect(byMessage, JSON.stringify(plan(byMessage))).toHaveLength(1)
+ expect(byMessage[0]!.docsExamined, JSON.stringify(plan(byMessage))).toBeLessThanOrEqual(3)
+ expect(byMessage[0]!.command?.hint, `names its index: ${JSON.stringify(plan(byMessage))}`).toBeTruthy()
+ })
+
+ it('a host confirming it typed a held delivery the pass ended gets the same answer compacted or not', async () => {
+ const { tab, lead } = await fixture()
+ const ended = async (text: string) => {
+ const one = await sent(tab, lead, text)
+ await store.delivery(host, tab, lead, one.deliveryId, { type: 'hold', claimId: one.claimId, reason: 'team_waiting_user' })
+ // The delivery pass ends a hold it waited on too long, keeping its reason (deliverySettle.ts).
+ await db.superTabDelivery.update({ where: { id: one.deliveryId }, data: { status: 'cancelled' } })
+ return one
+ }
+ const live = await ended('Live'), compacted = await ended('Compacted')
+ await age([compacted.deliveryId], 15)
+ expect((await compactDeliveries(db, Date.now())).compacted).toBe(1)
+ const typed = async (one: typeof live) =>
+ await store.delivery(host, tab, lead, one.deliveryId, { type: 'ack', claimId: one.claimId, receipt: 'typed', uncertain: false }) as {
+ delivery: SuperTabDelivery; message: { id: string } }
+ const [a, b] = [await typed(live), await typed(compacted)]
+ const shape = (answer: typeof a, one: typeof live) => ({ ...answer, delivery: { ...answer.delivery, updatedAt: null, deliveredAt: null, createdAt: null,
+ id: answer.delivery.id === one.deliveryId, messageId: answer.delivery.messageId === one.messageId, claimId: answer.delivery.claimId === one.claimId },
+ message: { ...answer.message, id: answer.message.id === one.messageId, text: null, sequence: null, createdAt: null, thread: null } })
+ expect(shape(b, compacted)).toEqual(shape(a, live))
+ expect(b.delivery).toMatchObject({ status: 'delivered', receipt: 'typed', holdReason: null })
+ expect(await receiptOf(compacted.messageId)).toMatchObject({ receipts: {}, deliveryIds: [] })
+ const page = await store.messages(owner, tab, 'lead', {})
+ expect(page.messages.find(m => m.id === compacted.messageId)!.deliveries).toEqual([expect.objectContaining({ id: compacted.deliveryId, status: 'delivered' })])
+ })
+})
diff --git a/backend/integration/superTabLiveness.test.ts b/backend/integration/superTabLiveness.test.ts
index a0f783cb2..6c2972298 100644
--- a/backend/integration/superTabLiveness.test.ts
+++ b/backend/integration/superTabLiveness.test.ts
@@ -305,6 +305,19 @@ describe('work that loses its owner', () => {
expect(await db.superTabDelivery.count({ where: { tabId: f.tab, messageId: ids.lead } })).toBe(1)
})
+ it('gets no second delivery when its delivery was compacted into its receipts (spec 5 C)', async () => {
+ const f = await swarm()
+ f.clock.now += MINUTE
+ const ids = lostIds(f), compacted = stableId(ids.lead, f.lead)
+ await store(f, ids.lead, 'Stored for the Lead')
+ await db.$runCommandRaw({ update: 'SuperTabReceipt', updates: [{ q: { _id: ids.lead }, upsert: true, u: {
+ $set: { tabId: f.tab, conversationId: f.group, [`receipts.${f.lead}`]: { deliveryId: compacted, status: 'answered' } },
+ $addToSet: { deliveryIds: compacted } } }] })
+ await f.observe('test-host', [], { stoppedHarnessIds: [f.harness.worker] })
+ expect(await f.assignment()).toMatchObject({ status: 'lost' })
+ expect(await db.superTabDelivery.count({ where: { tabId: f.tab, messageId: ids.lead } })).toBe(0)
+ })
+
it('still refuses a stored message that is not this notice', async () => {
const f = await swarm()
f.clock.now += MINUTE
diff --git a/backend/integration/superTabMessagePages.test.ts b/backend/integration/superTabMessagePages.test.ts
new file mode 100644
index 000000000..1c348c2de
--- /dev/null
+++ b/backend/integration/superTabMessagePages.test.ts
@@ -0,0 +1,224 @@
+import { randomUUID } from 'node:crypto'
+import { afterAll, beforeAll, describe, expect, it } from 'vitest'
+import type { MongoMemoryReplSet } from 'mongodb-memory-server-core'
+import type { Prisma, PrismaClient, SuperTabMessage } from '@prisma/client'
+import type { AuthUser } from '../src/lib/ssoAuth.js'
+import { SuperTabStore } from '../src/superTabs/store.js'
+import type { Command } from '../src/superTabs/commands.js'
+import { agentOnly } from '../src/superTabs/model.js'
+import { receiptFields } from '../src/superTabs/pageReads.js'
+import { startMessageReadStats, type MessageReadKey } from '../src/superTabs/messageReadStats.js'
+import { planDatabase } from './plans.test-support.js'
+
+// Spec 5 phase A (docs/superpowers/specs/2026-10-30-swarm-messages-delta-and-retention-design.md): who reads messages how, a
+// page that decodes about the rows it shows, and receipts read the same whether their deliveries are live or compacted.
+let replica: MongoMemoryReplSet
+let db: PrismaClient
+let store: SuperTabStore
+const owner: AuthUser = { sub: '500000000000000000000001', email: '[email protected]', autonomousEnv: 'prod', role: 'user' }
+const host: AuthUser = { sub: '500000000000000000000003', email: '[email protected]', autonomousEnv: 'prod', role: 'user' }
+const config = { engine: 'codex', model: 'fixture-model', effort: 'high', contextWindow: 200000, engineVersion: 'fixture' }
+
+beforeAll(async () => {
+ ({ replica, db } = await planDatabase('swarm_pages_test'))
+ store = new SuperTabStore(db)
+ for (const user of [owner, host]) await db.user.create({ data: { id: user.sub, externalId: `fixture-${user.sub}`, email: user.email, autonomousEnv: 'prod' } })
+ await db.machine.create({ data: { machineId: 'test-host', userId: host.sub, apiKey: 'fixture-only', managerId: 'fixture', autonomousEnv: 'prod', authMode: 'remote' } })
+ await db.machineShare.create({ data: { machineId: 'test-host', ownerId: host.sub, autonomousEnv: 'prod', emails: [owner.email] } })
+})
+afterAll(async () => {
+ await db?.$disconnect()
+ await replica?.stop()
+})
+
+const run = (tab: string, command: Command, user = owner, asAgentId?: string) => store.execute(user, tab, { operationId: randomUUID(), command }, asAgentId)
+async function fixture() {
+ const tab = (await store.create(owner, randomUUID(), 'Pages')).id
+ const lead = String((await run(tab, { type: 'agent.add', conversationId: 'lead', machineId: 'test-host', harnessId: randomUUID(), name: 'Lead',
+ configuration: config, lead: true })).result.agentId)
+ return { tab, lead }
+}
+
+/** `count` messages in the Lead chat from sequence 1000 on; `hidden(i)` ones are agent-only notes to another agent. */
+async function seed(tab: string, count: number, hidden: (i: number) => boolean) {
+ const rows: Prisma.SuperTabMessageCreateManyInput[] = Array.from({ length: count }, (_, i) => {
+ const id = `${tab}-m${String(i).padStart(4, '0')}`
+ return { id, tabId: tab, conversationId: 'lead', sequence: 1000 + i, author: { kind: 'person', id: owner.sub, name: owner.email, membershipGeneration: 'g' },
+ text: `row ${i}`, replyTo: null, recipientIds: hidden(i) ? ['someone-else'] : [], mentions: [], thread: id,
+ ...(hidden(i) ? { source: { kind: 'agent-only' } } : {}) }
+ })
+ await db.superTabMessage.createMany({ data: rows })
+}
+
+/** The newest page and its cursor as every row of the conversation gives them, for a reader who sees what `sees` keeps. */
+async function expectedPage(tab: string, sees: (m: SuperTabMessage) => boolean) {
+ const all = await db.superTabMessage.findMany({ where: { tabId: tab, conversationId: 'lead' }, orderBy: [{ sequence: 'desc' }, { id: 'desc' }] })
+ const shown = all.filter(sees)
+ const page = shown.slice(0, 100)
+ return { ids: page.map(m => m.id).reverse(), next: shown.length > 100 ? { sequence: page.at(-1)!.sequence, id: page.at(-1)!.id } : null }
+}
+const personSees = (m: SuperTabMessage) => !agentOnly(m.source, m.recipientIds)
+const agentSees = (agentId: string) => (m: SuperTabMessage) => personSees(m) || m.recipientIds.includes(agentId)
+
+type Counted = MessageReadKey & { n: number; batches: number; decoded: number; shown: number }
+/** What `read` adds to the messages read log. */
+async function counted<T>(read: () => Promise<T>): Promise<{ value: T; rows: Counted[] }> {
+ const rows: Counted[] = []
+ const stop = startMessageReadStats({ everyMs: 3_600_000, log: (_message, context) => rows.push(...context.rows as Counted[]) })
+ try {
+ return { value: await read(), rows }
+ } finally {
+ stop()
+ }
+}
+
+describe('messages by reader (spec 5 A1)', () => {
+ it('counts an agent\'s older page and a person\'s newest page under their kinds, with batches and rows, naming nothing', async () => {
+ const { tab, lead } = await fixture()
+ await seed(tab, 150, () => false)
+ const edge = (await db.superTabMessage.findFirstOrThrow({ where: { tabId: tab, sequence: 1100 } }))
+ const { rows } = await counted(async () => {
+ await store.messages(host, tab, 'lead', { before: edge.sequence, beforeId: edge.id }, lead)
+ await store.messages(owner, tab, 'lead', {})
+ await store.messages(owner, tab, 'lead', { before: 1150 })
+ await store.catchUp(owner, tab, 'lead', 1140)
+ })
+ const row = (reader: string, cursor: string) => rows.find(r => r.reader === reader && r.cursor === cursor)
+ expect(row('agent', 'beforeId')).toMatchObject({ session: 'sso', n: 1, batches: 1, shown: 100 })
+ expect(row('person', 'latest')).toMatchObject({ session: 'sso', n: 1, batches: 1, decoded: 101, shown: 100 })
+ expect(row('person', 'before')).toMatchObject({ n: 1, shown: 100 })
+ expect(row('person', 'catchUp')).toMatchObject({ n: 1, batches: 1, decoded: 9, shown: 9 })
+ expect(JSON.stringify(rows)).not.toMatch(/[0-9a-f]{12,}|@|tab-/i)
+ })
+})
+
+describe('a page reads about the rows it shows (spec 5 A2)', () => {
+ it('a page with one hidden row in its first 101 decodes at most 101 + 10 rows and returns the same page and next as before', async () => {
+ const { tab } = await fixture()
+ await seed(tab, 300, i => i === 250)
+ const { value, rows } = await counted(() => store.messages(owner, tab, 'lead', {}))
+ const expected = await expectedPage(tab, personSees)
+ expect(value.messages.map(m => m.id)).toEqual(expected.ids)
+ expect(value.next).toEqual(expected.next)
+ expect(rows).toEqual([expect.objectContaining({ reader: 'person', cursor: 'latest', n: 1, shown: 100 })])
+ expect(rows[0]!.decoded).toBeLessThanOrEqual(111)
+ })
+
+ it('an agent reading a chat where most rows are notes to others still gets a full page, and the next page goes on from it', async () => {
+ const { tab, lead } = await fixture()
+ await seed(tab, 440, i => i % 4 !== 0)
+ const { value: first, rows } = await counted(() => store.messages(host, tab, 'lead', {}, lead))
+ // The old 101-row batches read 404 rows in 4 batches for this page; sized by the rate seen, two batches read about as many.
+ expect(rows[0]).toMatchObject({ reader: 'agent', cursor: 'latest', shown: 100 })
+ expect(rows[0]!.batches).toBeLessThanOrEqual(2)
+ expect(rows[0]!.decoded).toBeLessThanOrEqual(485)
+ const expected = await expectedPage(tab, agentSees(lead))
+ expect(first.messages).toHaveLength(100)
+ expect(first.messages.map(m => m.id)).toEqual(expected.ids)
+ expect(first.next).toEqual(expected.next)
+ const second = await store.messages(host, tab, 'lead', { before: first.next!.sequence, beforeId: first.next!.id }, lead)
+ expect(second.messages.length).toBeGreaterThan(0)
+ expect(second.messages.at(-1)!.sequence).toBeLessThan(first.messages[0]!.sequence)
+ })
+
+ it('a conversation that ends inside a short batch reads it once and stops', async () => {
+ const { tab } = await fixture()
+ await seed(tab, 105, i => i >= 100)
+ const { value, rows } = await counted(() => store.messages(owner, tab, 'lead', {}))
+ expect(value.messages.map(m => m.id)).toEqual((await expectedPage(tab, personSees)).ids)
+ expect(rows[0]!.batches).toBeLessThanOrEqual(2)
+ })
+})
+
+/** Moves deliveries into their messages' receipts and deletes them, as the compaction job does (deliveryCompaction.ts). */
+async function compactByHand(ids: string[]) {
+ for (const row of await db.superTabDelivery.findMany({ where: { id: { in: ids } } })) {
+ await db.$runCommandRaw({ update: 'SuperTabReceipt', updates: [{ q: { _id: row.messageId }, upsert: true,
+ u: { $set: { tabId: row.tabId, conversationId: row.conversationId, [`receipts.${row.agentId}`]: receiptFields(row) }, $addToSet: { deliveryIds: row.id } } }] })
+ await db.superTabDelivery.delete({ where: { id: row.id } })
+ }
+}
+
+describe('receipts read the same live or compacted (spec 5 A3)', () => {
+ /** Three messages to the Lead, each claimed, typed and answered, and a fourth still waiting. */
+ async function answered() {
+ const f = await fixture()
+ for (const text of ['One', 'Two', 'Three']) {
+ const sent = String((await run(f.tab, { type: 'message.send', conversationId: 'lead', text, replyTo: null, recipientIds: [] })).result.messageId)
+ const delivery = (await store.deliveries(host, f.tab, f.lead)).find(d => d.messageId === sent)!
+ await store.delivery(host, f.tab, f.lead, delivery.id, { type: 'claim', claimId: `claim-${text}` })
+ await store.delivery(host, f.tab, f.lead, delivery.id, { type: 'ack', claimId: `claim-${text}`, receipt: `typed-${text}`, uncertain: false })
+ await run(f.tab, { type: 'message.send', conversationId: 'lead', text: `Re: ${text}`, replyTo: sent, recipientIds: [] }, host, f.lead)
+ }
+ await run(f.tab, { type: 'message.send', conversationId: 'lead', text: 'Four', replyTo: null, recipientIds: [] })
+ const deliveries = await db.superTabDelivery.findMany({ where: { tabId: f.tab }, orderBy: { createdAt: 'asc' } })
+ return { ...f, deliveries }
+ }
+
+ it('a page reads the same whether its deliveries are live, compacted or half each', async () => {
+ const f = await answered()
+ expect(f.deliveries.map(d => d.status)).toEqual(['answered', 'answered', 'answered', 'queued'])
+ const live = await store.messages(owner, f.tab, 'lead', {}), liveToLead = await store.messages(host, f.tab, 'lead', {}, f.lead)
+ expect(live.messages.flatMap(m => m.deliveries)).toHaveLength(4)
+ await compactByHand(f.deliveries.slice(0, 2).map(d => d.id))
+ expect(await store.messages(owner, f.tab, 'lead', {})).toEqual(live)
+ await compactByHand([f.deliveries[2]!.id])
+ expect(await store.messages(owner, f.tab, 'lead', {})).toEqual(live)
+ expect(await store.messages(host, f.tab, 'lead', {}, f.lead)).toEqual(liveToLead)
+ })
+
+ it('keeps one order for a message\'s receipts whichever of them are compacted', async () => {
+ const { tab, lead } = await fixture()
+ const id = `${tab}-shared`, at = new Date()
+ await db.superTabMessage.create({ data: { id, tabId: tab, conversationId: 'lead', sequence: 900, text: 'To both', replyTo: null, recipientIds: [lead, 'agent-b'],
+ mentions: [], thread: id, author: { kind: 'person', id: owner.sub, name: owner.email, membershipGeneration: 'g' } } })
+ const delivery = (agentId: string) => ({ id: `${id}-${agentId}`, tabId: tab, conversationId: 'lead', messageId: id, agentId, machineId: 'test-host',
+ membershipGeneration: 'g', authorKind: 'person', authorId: owner.sub, authorGeneration: 'g', status: 'answered', createdAt: at })
+ await db.superTabDelivery.createMany({ data: [delivery(lead), delivery('agent-b')] })
+ const shown = async () => (await store.messages(owner, tab, 'lead', {})).messages.find(m => m.id === id)!.deliveries.map(d => d.agentId)
+ const before = await shown()
+ expect(before).toEqual([lead, 'agent-b'])
+ await compactByHand([`${id}-${lead}`])
+ expect(await shown()).toEqual(before)
+ })
+
+ it('a live row wins over a compacted one for the same agent', async () => {
+ const f = await answered()
+ const first = f.deliveries[0]!
+ const live = await store.messages(owner, f.tab, 'lead', {})
+ await compactByHand([first.id])
+ // A row for the same message and agent again (a host that settled it while it was being moved): it is what the page shows.
+ await db.superTabDelivery.create({ data: { ...first, status: 'confirmed', receipt: 'again' } })
+ const page = await store.messages(owner, f.tab, 'lead', {})
+ const row = page.messages.find(m => m.id === first.messageId)!
+ expect(row.deliveries).toHaveLength(1)
+ expect(row.deliveries[0]).toMatchObject({ id: first.id, status: 'confirmed', receipt: 'again' })
+ expect(page.messages.filter(m => m.id !== first.messageId)).toEqual(live.messages.filter(m => m.id !== first.messageId))
+ })
+
+ it('a compacted receipt for news a row does not name (a Buddy\'s) is shown only to that agent, as a live one is', async () => {
+ const { tab, lead } = await fixture()
+ // The Lead's delivery of a row that does not list it among its recipients: the shape of Buddy news (buddyNews.ts).
+ const id = `${tab}-news`
+ await db.superTabMessage.create({ data: { id, tabId: tab, conversationId: 'lead', sequence: 900, text: 'News', replyTo: null, recipientIds: [],
+ mentions: [], thread: id, author: { kind: 'person', id: owner.sub, name: owner.email, membershipGeneration: 'g' } } })
+ await db.superTabDelivery.create({ data: { id: `${id}-d`, tabId: tab, conversationId: 'lead', messageId: id, agentId: lead, machineId: 'test-host',
+ membershipGeneration: 'g', authorKind: 'person', authorId: owner.sub, authorGeneration: 'g', status: 'answered' } })
+ const receipts = async (user: AuthUser, asAgentId?: string) =>
+ (await store.messages(user, tab, 'lead', {}, asAgentId)).messages.find(m => m.id === id)!.deliveries.map(d => d.agentId)
+ expect(await receipts(owner)).toEqual([])
+ expect(await receipts(host, lead)).toEqual([lead])
+ await compactByHand([`${id}-d`])
+ expect(await receipts(owner)).toEqual([])
+ expect(await receipts(host, lead)).toEqual([lead])
+ })
+
+ it('no message read returns a receipts map or a compacted receipt\'s host fields', async () => {
+ const f = await answered()
+ await compactByHand(f.deliveries.slice(0, 3).map(d => d.id))
+ const replies = [await store.messages(owner, f.tab, 'lead', {}), await store.messages(host, f.tab, 'lead', {}, f.lead),
+ await store.messages(owner, f.tab, 'lead', { messageId: f.deliveries[0]!.messageId }), await store.catchUp(owner, f.tab, 'lead', 0),
+ await store.mentions(owner, f.tab, 'waiting')]
+ for (const reply of replies) expect(JSON.stringify(reply)).not.toMatch(/"receipts"|"deliveryIds"|"deliveryId"|"machineOwnerId"|"claimId"/)
+ })
+})
diff --git a/backend/integration/superTabThreads.test.ts b/backend/integration/superTabThreads.test.ts
new file mode 100644
index 000000000..ef6872795
--- /dev/null
+++ b/backend/integration/superTabThreads.test.ts
@@ -0,0 +1,191 @@
+import { randomUUID } from 'node:crypto'
+import { execFile } from 'node:child_process'
+import { promisify } from 'node:util'
+import { join } from 'node:path'
+import { tmpdir } from 'node:os'
+import { afterAll, beforeAll, describe, expect, it } from 'vitest'
+import { MongoMemoryReplSet } from 'mongodb-memory-server-core'
+import { PrismaClient, type Prisma } from '@prisma/client'
+import type { AuthUser } from '../src/lib/ssoAuth.js'
+import { SuperTabStore } from '../src/superTabs/store.js'
+import { migrateSwarm } from '../src/superTabs/migration/migrate.js'
+import { buddyChatId } from '../src/superTabs/buddies.js'
+import { refreshThreads, threadFacet } from '../src/superTabs/pageReads.js'
+import { backfillThreads } from '../src/superTabs/threadBackfill.js'
+import type { SuperTabState } from '../src/superTabs/model.js'
+import { writeLegacySwarm } from '../scripts/swarm-parts/legacy.js'
+import { NOW, outsider as outsiderActor, owner as ownerActor, peer as peerActor, swarm } from '../src/superTabs/read/fixture.test-support.js'
+import { profiledReads } from './profiledReads.test-support.js'
+import { filterOf, plan, profiled, reads } from './plans.test-support.js'
+
+// Spec 5 phase D (docs/superpowers/specs/2026-10-30-swarm-messages-delta-and-retention-design.md): a page reads its threads'
+// summaries from SuperTabThread once its Swarm is backfilled, and every reader's threads stay what the $facet gave them.
+// A fresh loopback replica set, created here. This suite never reads DATABASE_URL or an existing database.
+let replica: MongoMemoryReplSet
+let db: PrismaClient
+let store: SuperTabStore
+const user = (actor: { userId: string; email: string }): AuthUser => ({ sub: actor.userId, email: actor.email, autonomousEnv: 'prod', role: 'user' })
+const [owner, peer, outsider] = [ownerActor, peerActor, outsiderActor].map(user) as [AuthUser, AuthUser, AuthUser]
+const MIGRATED = NOW - 60_000
+
+beforeAll(async () => {
+ replica = await MongoMemoryReplSet.create({ binary: { version: '7.0.24', downloadDir: join(tmpdir(), 'harness-super-tab-mongodb-binaries') },
+ replSet: { count: 1, ip: '127.0.0.1', storageEngine: 'wiredTiger' } })
+ const uri = replica.getUri(`swarm_threads_test_${randomUUID().replaceAll('-', '_')}`)
+ await promisify(execFile)(join(process.cwd(), 'node_modules/.bin/prisma'), ['db', 'push', '--skip-generate'], {
+ env: { ...process.env, DATABASE_URL: uri }, timeout: 60000,
+ })
+ db = new PrismaClient({ datasources: { db: { url: uri } } })
+ for (const each of [owner, peer, outsider]) await db.user.create({ data: { id: each.sub, externalId: `fixture-${each.sub}`, email: each.email, autonomousEnv: 'prod' } })
+ // The owner's own machine runs the Swarm's agents, so reading as one of them needs no machine share.
+ await db.machine.create({ data: { machineId: 'host', userId: owner.sub, apiKey: 'fixture-only', managerId: 'fixture', autonomousEnv: 'prod', authMode: 'remote' } })
+ store = new SuperTabStore(db)
+})
+afterAll(async () => {
+ await db?.$disconnect()
+ await replica?.stop()
+})
+
+const person = (actor: AuthUser) => ({ kind: 'person', id: actor.sub, name: actor.email, membershipGeneration: 'gen' })
+const agent = (id: string) => ({ kind: 'agent', id, name: id, membershipGeneration: 'gen' })
+let sequence = 10, scenes = 0
+
+/** A migrated Swarm: a Lead, a worker in Web, and peer's Buddy, in the project chat and in its own Buddy chat. Web holds a lone
+ * root and threads with every kind of reply a reader may skip: a note to the Lead alone, a route row (no Buddy reads it), a
+ * tag. Its root `w-b` is a note to the worker alone. The Buddy chat holds a thread with replies of a Buddy session. */
+async function scene() {
+ const s = swarm(MIGRATED)
+ s.join(peerActor)
+ s.join(outsiderActor)
+ const lead = s.add('lead-harness', 'lead', true, ownerActor, 'host', owner.sub)
+ const web = s.group('Web')
+ const worker = s.add('worker-harness', web, false, ownerActor, 'host', owner.sub)
+ const buddy = s.add('buddy-harness', 'lead', false, ownerActor, 'host', owner.sub)
+ let buddyChat = ''
+ s.edit(state => {
+ const generation = state.people.find(p => p.id === peer.sub)!.generation
+ const projectId = state.projects![0]!.id
+ state.agents.find(a => a.id === buddy)!.buddyOf = { personId: peer.sub, generation }
+ buddyChat = buddyChatId(projectId, { id: peer.sub, generation })
+ state.conversations.push({ id: buddyChat, kind: 'buddy', projectId, title: 'Buddy', leadId: null, agentIds: [buddy], archived: false,
+ brief: '', checks: [], assignment: null, lastMessageSequence: 0, audience: { people: [{ id: peer.sub, generation }] } })
+ })
+ const tabId = `tab-${randomUUID()}`
+ await writeLegacySwarm(db, { state: { ...structuredClone(s.state), id: tabId } as SuperTabState, revision: 700 })
+ const report = await migrateSwarm(db, tabId, { now: () => MIGRATED })
+ if (!report.ok) throw new Error(report.error)
+ const note = (to: string) => ({ source: { kind: 'note', recipientsOnly: true }, recipientIds: [to] })
+ // Message ids are unique across Swarms: each scene's carry its own prefix.
+ const prefix = `s${++scenes}-`, id = (name: string) => prefix + name
+ const rows: Row[] = [
+ { id: 'w-lone', conversationId: web, thread: 'w-lone', author: person(owner) },
+ { id: 'w-a', conversationId: web, thread: 'w-a', author: person(owner) },
+ { id: 'w-a1', conversationId: web, thread: 'w-a', author: agent(worker) },
+ { id: 'w-a2', conversationId: web, thread: 'w-a', author: agent(worker), ...note(lead) },
+ { id: 'w-a3', conversationId: web, thread: 'w-a', author: agent(lead), source: { kind: 'route' } },
+ { id: 'w-a4', conversationId: web, thread: 'w-a', author: person(peer), mentions: [outsider.sub] },
+ { id: 'w-a5', conversationId: web, thread: 'w-a', author: person(outsider) },
+ { id: 'w-b', conversationId: web, thread: 'w-b', author: agent(lead), ...note(worker) },
+ { id: 'w-b1', conversationId: web, thread: 'w-b', author: agent(worker) },
+ { id: 'w-c', conversationId: web, thread: 'w-c', author: person(owner) },
+ { id: 'w-c1', conversationId: web, thread: 'w-c', author: agent(lead), source: { kind: 'route' } },
+ { id: 'b-root', conversationId: buddyChat, thread: 'b-root', author: person(peer) },
+ { id: 'b-r1', conversationId: buddyChat, thread: 'b-root', author: agent(buddy) },
+ { id: 'b-r2', conversationId: buddyChat, thread: 'b-root', author: person(peer), buddySession: true },
+ { id: 'b-r3', conversationId: buddyChat, thread: 'b-root', author: agent(buddy), buddySession: true },
+ ]
+ await insert(tabId, prefix, rows)
+ const readers: Array<[string, string, AuthUser, string?]> = [['owner', web, owner], ['peer', web, peer], ['outsider', web, outsider],
+ ['Buddy', web, owner, buddy], ['worker', web, owner, worker], ['Lead', web, owner, lead], ['peer in the Buddy chat', buddyChat, peer],
+ ['Buddy in its chat', buddyChat, owner, buddy]]
+ return { tabId, web, buddyChat, lead, worker, buddy, readers, prefix, id }
+}
+
+interface Row { id: string; conversationId: string; thread: string; author: object; mentions?: string[]; source?: object; recipientIds?: string[]; buddySession?: boolean }
+/** Rows written as an older release writes them, no summary touched; ids and threads named without the scene's prefix. */
+async function insert(tabId: string, prefix: string, rows: Row[]) {
+ await db.superTabMessage.createMany({ data: rows.map(row => ({ recipientIds: [], mentions: [], ...row, id: prefix + row.id, thread: prefix + row.thread,
+ tabId, text: row.id, sequence: ++sequence, createdAt: new Date() })) })
+}
+
+/** Each reader's newest page of its conversation. */
+const pages = async (s: Awaited<ReturnType<typeof scene>>) => Object.fromEntries(await Promise.all(s.readers.map(async ([name, conversationId, as, agentId]) =>
+ [name, await store.messages(as, s.tabId, conversationId, {}, agentId)] as const)))
+const replies = (page: { threads: Array<{ id: string; replies: number }> }) => Object.fromEntries(page.threads.map(t => [t.id.replace(/^s\d+-/, ''), t.replies]))
+const sorted = (rows: unknown[]) => rows.map(row => JSON.stringify(row)).sort()
+
+describe('thread summaries (spec 5 phase D)', () => {
+ it('gives every reader the same page from the summaries as from the $facet, lone roots and hidden replies included', async () => {
+ const s = await scene()
+ const before = await pages(s)
+ // Each reader skips its own kinds of reply: no summary counts a note to agents alone (a reader of `w-b`'s root still counts
+ // its reply), and a Buddy no route row, nor the session's replies in its own chat.
+ expect(Object.fromEntries(Object.entries(before).map(([name, page]) => [name, replies(page)]))).toEqual({
+ owner: { 'w-a': 4, 'w-c': 1 }, peer: { 'w-a': 4, 'w-c': 1 }, outsider: { 'w-a': 4, 'w-c': 1 }, Buddy: { 'w-a': 3 },
+ worker: { 'w-a': 4, 'w-b': 1, 'w-c': 1 }, Lead: { 'w-a': 4, 'w-c': 1 }, 'peer in the Buddy chat': { 'b-root': 3 }, 'Buddy in its chat': { 'b-root': 1 } })
+ expect(await db.superTabThread.count({ where: { tabId: s.tabId } })).toBe(0)
+
+ await backfillThreads(db, Date.now())
+ // One summary per thread with a reply: none for the lone root.
+ expect((await db.superTabThread.findMany({ where: { tabId: s.tabId }, select: { id: true } })).map(t => t.id).sort()).toEqual(['b-root', 'w-a', 'w-b', 'w-c'].map(s.id))
+ expect(await pages(s)).toEqual(before)
+ })
+
+ it('reads through the $facet until its Swarm is backfilled, and the summaries by id after', async () => {
+ const s = await scene()
+ const read = () => profiledReads(db, /\.SuperTab(Message|Thread)$/, () => store.messages(owner, s.tabId, s.web, {}))
+ const facet = await read()
+ expect(facet.byCollection.SuperTabThread).toBeUndefined()
+ await backfillThreads(db, Date.now())
+ const summaries = await read()
+ expect(summaries.value).toEqual(facet.value)
+ expect(summaries.byCollection).toEqual({ SuperTabMessage: facet.byCollection.SuperTabMessage! - 1, SuperTabThread: 1 })
+ })
+
+ it('rewrites a thread\'s summary in the transaction that sends a reply, a lone root\'s first one too', async () => {
+ const s = await scene()
+ await backfillThreads(db, Date.now())
+ for (const replyTo of ['w-a', 'w-lone'].map(s.id)) {
+ await store.execute(owner, s.tabId, { operationId: randomUUID(), command: { type: 'message.send', conversationId: s.web, text: 'More', replyTo,
+ recipientIds: [], mentions: [outsider.sub] } as never })
+ const stored = await db.superTabThread.findUniqueOrThrow({ where: { id: replyTo } })
+ const facet = await threadFacet(db, s.tabId, s.web, [replyTo])
+ expect(sorted(stored.parts as unknown[])).toEqual(sorted(facet.parts))
+ expect(sorted(stored.mentions as unknown[])).toEqual(sorted(facet.mentions))
+ }
+ expect(replies(await store.messages(owner, s.tabId, s.web, {}))).toEqual({ 'w-a': 5, 'w-c': 1, 'w-lone': 1 })
+ })
+
+ it('leaves no summary when the reply\'s transaction fails', async () => {
+ const s = await scene()
+ await expect(db.$transaction(async tx => {
+ await tx.superTabMessage.create({ data: { id: s.id('w-lone-r'), tabId: s.tabId, conversationId: s.web, thread: s.id('w-lone'), sequence: 999, text: 'x',
+ author: person(owner), recipientIds: [], mentions: [] } })
+ await refreshThreads(tx, s.tabId, s.web, [s.id('w-lone')])
+ expect(await tx.superTabThread.findUnique({ where: { id: s.id('w-lone') } })).not.toBeNull()
+ throw new Error('rolled back')
+ })).rejects.toThrow('rolled back')
+ expect(await db.superTabThread.findUnique({ where: { id: s.id('w-lone') } })).toBeNull()
+ })
+
+ it('counts at the next backfill a reply an older release wrote after the marker', async () => {
+ const s = await scene()
+ await backfillThreads(db, Date.now())
+ // An older release writes the reply as any command does (its conversation moves on), but leaves the summary as it was.
+ const summary = async () => (await db.$runCommandRaw({ find: 'SuperTabThread', filter: { _id: s.id('w-a') } }) as unknown as
+ { cursor: { firstBatch: Prisma.InputJsonObject[] } }).cursor.firstBatch[0]!
+ const before = await summary()
+ await store.execute(owner, s.tabId, { operationId: randomUUID(), command: { type: 'message.send', conversationId: s.web, text: 'Late', replyTo: s.id('w-a'),
+ recipientIds: [] } as never })
+ await db.$runCommandRaw({ update: 'SuperTabThread', updates: [{ q: { _id: s.id('w-a') }, u: before }] })
+ expect(replies(await store.messages(owner, s.tabId, s.web, {}))['w-a']).toBe(4)
+ const entries = await profiled(db, () => backfillThreads(db, Date.now()))
+ expect(replies(await store.messages(owner, s.tabId, s.web, {}))['w-a']).toBe(5)
+ // Only the changed conversation's newest messages are read again, never every message of each Swarm.
+ expect(plan(reads(entries, 'SuperTabMessage', ['tabId'], ['conversationId']))).toEqual([])
+ const walked = reads(entries, 'SuperTabMessage', ['conversationId'], ['thread']).filter(entry => JSON.stringify(filterOf(entry)).includes(s.tabId))
+ expect(walked.reduce((sum, entry) => sum + entry.docsExamined, 0), JSON.stringify(plan(walked))).toBeLessThanOrEqual(12)
+ const stored = await db.superTabThread.findUniqueOrThrow({ where: { id: s.id('w-a') } })
+ expect(sorted(stored.parts as unknown[])).toEqual(sorted((await threadFacet(db, s.tabId, s.web, [s.id('w-a')])).parts))
+ })
+})
diff --git a/backend/prisma/schema.prisma b/backend/prisma/schema.prisma
index 33d9f3840..2c1d2e879 100644
--- a/backend/prisma/schema.prisma
+++ b/backend/prisma/schema.prisma
@@ -1185,6 +1185,29 @@ model SuperTabThreadMark {
@@index([updatedAt])
}
+/// One thread with at least one reply, summed up as pageThreads' $facet gives it (pageReads.ts refreshThreads): rewritten in the
+/// transaction that inserts each reply, and read by root id instead of the $facet once its Swarm is backfilled
+/// (SuperTabThreadBackfill). No text; never returned as is. Kept with its messages.
+model SuperTabThread {
+ /// The root message's id.
+ id String @id @map("_id")
+ tabId String
+ conversationId String
+ /// The $facet's `parts` and `mentions` for this thread (pageReads.ts ThreadFacet).
+ parts Json
+ mentions Json
+ updatedAt DateTime
+
+ @@index([tabId, conversationId])
+}
+
+/// A Swarm whose thread summaries hold every reply created before `through` (threadBackfill.ts); its pages read them. Written
+/// by the singleton worker alone; id = the Swarm's id.
+model SuperTabThreadBackfill {
+ id String @id @map("_id")
+ through DateTime
+}
+
/// Where a Swarm sweep reads on from between passes and releases, one row per sweep (readMarks.ts sweepThreadMarks: id
/// "thread-marks", the last Swarm whose people check finished). Written by the singleton worker alone.
model SuperTabSweepCursor {
@@ -1321,6 +1344,23 @@ model SuperTabDelivery {
/// An ended task's open deliveries, cancelled with it (store.ts releaseRuns and stopped): by [tabId, status] alone the
/// planner sometimes walked every delivery of the Swarm (180 read for 1 in the benchmark).
@@index([tabId, assignmentId])
+ /// Finished deliveries by their last change: the hourly compaction reads only those past 14 days (deliveryCompaction.ts).
+ @@index([status, updatedAt])
+}
+
+// One message's finished deliveries, moved here 14 days after their last change (superTabs/deliveryCompaction.ts): message
+// history and its receipts are kept forever, while delivery rows hold only the recent and open ones. Never on the message
+// row, never returned whole: a page reads each receipt as a delivery and filters it per reader (pageReads.ts pageDeliveries).
+model SuperTabReceipt {
+ id String @id @map("_id") // the message id
+ tabId String
+ conversationId String
+ /// By agent id: the delivery row as it was, but for the fields above (pageReads.ts receiptFields).
+ receipts Json
+ /// The compacted deliveries' ids, so a host's receipt read finds one by its id (pageReads.ts compactedDelivery).
+ deliveryIds String[]
+
+ @@index([tabId, deliveryIds])
}
// One independent harness owns at most one active assignment across all Super Tabs.
diff --git a/backend/src/config/env.ts b/backend/src/config/env.ts
index b74888c2b..874bf30db 100644
--- a/backend/src/config/env.ts
+++ b/backend/src/config/env.ts
@@ -41,6 +41,10 @@ const envSchema = z.object({
SWARM_READ_CACHE: z.enum(['on', 'off']).default('on'),
// Its size per process, counted as the JSON size of what it holds.
SWARM_CACHE_MB: z.string().default('64').transform(Number),
+ // The worker's hourly move of finished deliveries older than 14 days into their messages' receipts
+ // (superTabs/deliveryCompaction.ts). 'off' is the kill switch: set it before rolling back to a release whose pages do not
+ // read compacted receipts.
+ SWARM_DELIVERY_COMPACTION: z.enum(['on', 'off']).default('on'),
// This backend instance's stable id (used to route manager-command replies via Redis in a
// multi-instance deploy). Defaults to the pm2 instance index or a per-process fallback.
diff --git a/backend/src/server.ts b/backend/src/server.ts
index 5b4736627..e0c6b2bc3 100644
--- a/backend/src/server.ts
+++ b/backend/src/server.ts
@@ -25,6 +25,7 @@ import { prisma } from './lib/prisma.js'
import { backendReadinessChecks, createReadiness, drainBeforeClosing, processDrain } from './lib/readiness.js'
import { configureSwarmStamps } from './superTabs/cache.js'
import { watchGroupIndexes } from './superTabs/parts/indexed.js'
+import { startMessageReadStats } from './superTabs/messageReadStats.js'
import { useSharedSsoProfileStore } from './lib/ssoAuth.js'
import { useSharedRateCounter } from './lib/rateLimit.js'
import { createRouteRateLimit } from './middlewares/routeRateLimit.js'
@@ -147,6 +148,8 @@ async function start(): Promise<void> {
// Every minute, how often Swarm transactions ran again and commands loaded twice (superTabs/retryStats.ts, DB 2.1).
const stopRetryStats = startRetryStats()
const stopRouteLatency = startRouteLatency(registerRouteLatency(app))
+ // Every minute: who reads messages how, with the rows each page decoded and showed (superTabs/messageReadStats.ts).
+ const stopMessageReadStats = startMessageReadStats()
app.setErrorHandler(errorHandler)
@@ -222,11 +225,12 @@ async function start(): Promise<void> {
await Promise.race([fastifyClosed, new Promise((resolve) => setTimeout(resolve, 5_000))])
// The sockets have closed, so no connect is left to count. Bounded: a slow database must not hold the exit.
await Promise.race([stopRelayUsage().catch(() => {}), new Promise((resolve) => setTimeout(resolve, 3_000))])
- // The last partial minute of both logs, once the drain and the usage flush have run their queries.
+ // The last partial minute of these logs, once the drain and the usage flush have run their queries.
stopQueryStats()
stopRetryStats()
stopRouteLatency()
stopMachineReportLog()
+ stopMessageReadStats()
// The database stays connected through the drain, for the requests still being served (lib/prisma.ts).
await Promise.allSettled([prisma.$disconnect(), closeBus()])
process.exit(exitCode)
diff --git a/backend/src/superTabs/deliveryCompaction.ts b/backend/src/superTabs/deliveryCompaction.ts
new file mode 100644
index 000000000..721d2c66c
--- /dev/null
+++ b/backend/src/superTabs/deliveryCompaction.ts
@@ -0,0 +1,93 @@
+import type { Prisma, PrismaClient, SuperTabDelivery } from '@prisma/client'
+import { logger } from '../utils/logger.js'
+import { readBy, retryStaleIndex } from './parts/indexed.js'
+import { receiptFields } from './pageReads.js'
+import { prismaRow } from './rawRows.js'
+import { retryTransient } from './retry.js'
+
+/**
+ * Message history is kept forever with its receipts, while delivery rows stop growing (spec 5 phase C): a finished delivery
+ * whose last change is 14 days old moves into its message's SuperTabReceipt, which page reads, hosts and notices read as the
+ * row (pageReads.ts). `delivered` and `confirmed` are not finished (a reply or a project report moves them to `answered`),
+ * nor are the open ones.
+ */
+export const FINAL_DELIVERY = ['answered', 'cancelled', 'superseded'] as const
+export const COMPACT_AFTER_MS = 14 * 86_400_000
+/** Batches per hourly run: up to 10 k deliveries, so the first runs also clear the backlog. Prod finishes ≤ 3.4 k a day. */
+const BATCHES_PER_RUN = 20
+/** Deliveries a reply or a project report may still answer: never compacted, counted so their growth shows in the log. */
+const UNFINISHED_DELIVERY = ['delivered', 'confirmed'] as const
+
+const old = (statuses: readonly string[], cutoff: Date) => ({ status: { $in: [...statuses] }, updatedAt: { $lt: { $date: cutoff.toISOString() } } })
+/** A receipt is stored under its agent's id as a field name: never one Mongo would read as a path or an operator. Ids are
+ * hex (model.ts stableId); another is left as a row, and never read again ahead of the rest. */
+const compactable = (cutoff: Date) => ({ ...old(FINAL_DELIVERY, cutoff), agentId: { $regex: '^[A-Za-z0-9_-]+$' } })
+
+/** One batch, in one transaction: the oldest finished deliveries past the cutoff, written into their messages' receipts by
+ * field path (two compactions of one message keep each other's agents) and deleted. A delivery changed since it was read
+ * is not deleted: the transaction fails and nothing of it is written. */
+async function compactBatch(db: PrismaClient, cutoff: Date, batch: number): Promise<number> {
+ return retryTransient(() => retryStaleIndex(() => db.$transaction(async tx => {
+ const moving = (await readBy(tx, 'SuperTabDelivery', 'compaction', [{ $match: compactable(cutoff) }, { $sort: { updatedAt: 1 } }, { $limit: batch }]))
+ .map(row => prismaRow<SuperTabDelivery>('SuperTabDelivery', row))
+ if (!moving.length) return 0
+ const byMessage = new Map<string, SuperTabDelivery[]>()
+ for (const row of moving) byMessage.set(row.messageId, [...byMessage.get(row.messageId) ?? [], row])
+ await tx.$runCommandRaw({ update: 'SuperTabReceipt', updates: [...byMessage.values()].map(deliveries => ({ q: { _id: deliveries[0]!.messageId }, upsert: true,
+ u: { $set: { tabId: deliveries[0]!.tabId, conversationId: deliveries[0]!.conversationId,
+ ...Object.fromEntries(deliveries.map(row => [`receipts.${row.agentId}`, receiptFields(row)])) },
+ $addToSet: { deliveryIds: { $each: deliveries.map(row => row.id) } } } })) } as Prisma.InputJsonObject)
+ const deleted = await tx.$runCommandRaw({ delete: 'SuperTabDelivery', deletes: moving.map(row => ({ limit: 1,
+ q: { _id: row.id, status: row.status, updatedAt: { $date: row.updatedAt.toISOString() } } })) } as Prisma.InputJsonObject) as unknown as { n: number }
+ if (deleted.n !== moving.length) throw new Error(`deliveries changed while compacting: ${moving.length - deleted.n}`)
+ return moving.length
+ }, TRANSACTION_OPTIONS)))
+}
+
+/** As the store's transactions (store.ts): a batch of 500 writes outlasts Prisma's 5 s default under load. */
+const TRANSACTION_OPTIONS = { maxWait: 5_000, timeout: 15_000 }
+
+/** A compacted delivery made a row again, and taken out of its message's receipts, in `tx`: a host that held it typed it
+ * after all (store.ts deliver). */
+export async function restoreDelivery(tx: Prisma.TransactionClient, row: SuperTabDelivery): Promise<void> {
+ await tx.superTabDelivery.create({ data: row })
+ await tx.$runCommandRaw({ update: 'SuperTabReceipt', updates: [{ q: { _id: row.messageId, tabId: row.tabId },
+ u: { $unset: { [`receipts.${row.agentId}`]: '' }, $pull: { deliveryIds: row.id } } }] })
+}
+
+/** Compacts finished deliveries older than 14 days at `now`, `batch` at a time, until none is left or 20 batches ran.
+ * `remaining`: those still old enough afterwards, the backlog the log warns on. `unfinished`: delivered or confirmed ones as
+ * old, which stay rows until a reply or a report answers them. */
+export async function compactDeliveries(db: PrismaClient, now: number, batch = 500): Promise<{ compacted: number; remaining: number; unfinished: number }> {
+ const cutoff = new Date(now - COMPACT_AFTER_MS)
+ let compacted = 0
+ for (let runs = 0; runs < BATCHES_PER_RUN; runs++) {
+ const moved = await compactBatch(db, cutoff, batch)
+ compacted += moved
+ if (moved < batch) break
+ }
+ const count = async (statuses: readonly string[]) => Number((await readBy(db as unknown as Prisma.TransactionClient, 'SuperTabDelivery', 'compaction',
+ [{ $match: old(statuses, cutoff) }, { $count: 'n' }]))[0]?.n ?? 0)
+ return { compacted, remaining: await count(FINAL_DELIVERY), unfinished: await count(UNFINISHED_DELIVERY) }
+}
+
+/** Runs the compaction now and then every `everyMs`, one run at a time, unless `setting` (env SWARM_DELIVERY_COMPACTION) is
+ * the kill switch. A failed run is logged and tried again next time. */
+export function startDeliveryCompaction(db: PrismaClient, setting: 'on' | 'off', everyMs = 3_600_000): () => void {
+ if (setting === 'off') {
+ logger.info('[swarm-deliveries] compaction is off (SWARM_DELIVERY_COMPACTION=off)')
+ return () => {}
+ }
+ let running = false
+ const run = () => {
+ if (running) return
+ running = true
+ void compactDeliveries(db, Date.now()).then(counts => logger.info('[swarm-deliveries] compacted', counts))
+ .catch(error => logger.warn('[swarm-deliveries] compaction failed', { error: error instanceof Error ? error.message : String(error) }))
+ .finally(() => { running = false })
+ }
+ run()
+ const timer = setInterval(run, everyMs)
+ timer.unref()
+ return () => clearInterval(timer)
+}
diff --git a/backend/src/superTabs/messageReadStats.test.ts b/backend/src/superTabs/messageReadStats.test.ts
new file mode 100644
index 000000000..cd2d7e447
--- /dev/null
+++ b/backend/src/superTabs/messageReadStats.test.ts
@@ -0,0 +1,50 @@
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+import { recordMessageRead, startMessageReadStats } from './messageReadStats.js'
+
+describe('messages by reader', () => {
+ let stop = () => {}
+ beforeEach(() => { vi.useFakeTimers() })
+ afterEach(() => {
+ stop()
+ vi.useRealTimers()
+ })
+
+ it('counts reads per reader and cursor, with batches and rows, and names nothing', () => {
+ const logged: Array<[string, Record<string, unknown>]> = []
+ stop = startMessageReadStats({ everyMs: 60_000, log: (message, context) => logged.push([message, context]) })
+ recordMessageRead({ reader: 'person', cursor: 'before', session: 'sso' }, 2, 202, 100)
+ recordMessageRead({ reader: 'person', cursor: 'before', session: 'sso' }, 2, 202, 100)
+ recordMessageRead({ reader: 'agent', cursor: 'latest', session: 'computer' }, 1, 101, 100)
+ vi.advanceTimersByTime(60_000)
+ expect(logged).toHaveLength(1)
+ const [message, context] = logged[0]!
+ expect(message).toBe('messages by reader')
+ expect(context.windowMs).toBe(60_000)
+ expect(context.rows).toEqual([
+ { reader: 'person', cursor: 'before', session: 'sso', n: 2, batches: 4, decoded: 404, shown: 200 },
+ { reader: 'agent', cursor: 'latest', session: 'computer', n: 1, batches: 1, decoded: 101, shown: 100 },
+ ])
+ expect(JSON.stringify(context)).not.toMatch(/[0-9a-f]{12,}|@/i)
+ })
+
+ it('logs nothing for a quiet minute, and starts each window empty', () => {
+ const logged: Array<Record<string, unknown>> = []
+ stop = startMessageReadStats({ everyMs: 60_000, log: (_message, context) => logged.push(context) })
+ recordMessageRead({ reader: 'buddy', cursor: 'after', session: 'sso' }, 1, 0, 0)
+ vi.advanceTimersByTime(180_000)
+ expect(logged).toEqual([{ windowMs: 60_000, rows: [{ reader: 'buddy', cursor: 'after', session: 'sso', n: 1, batches: 1, decoded: 0, shown: 0 }] }])
+ })
+
+ it('keeps nothing while stopped, and logs what is left when it stops', () => {
+ recordMessageRead({ reader: 'person', cursor: 'latest', session: 'sso' }, 1, 1, 1)
+ const logged: Array<Record<string, unknown>> = []
+ stop = startMessageReadStats({ everyMs: 60_000, log: (_message, context) => logged.push(context) })
+ stop()
+ expect(logged).toEqual([])
+ stop = startMessageReadStats({ everyMs: 60_000, log: (_message, context) => logged.push(context) })
+ recordMessageRead({ reader: 'person', cursor: 'catchUp', session: 'team' }, 1, 3, 2)
+ vi.advanceTimersByTime(10_000)
+ stop()
+ expect(logged).toEqual([{ windowMs: 10_000, rows: [{ reader: 'person', cursor: 'catchUp', session: 'team', n: 1, batches: 1, decoded: 3, shown: 2 }] }])
+ })
+})
diff --git a/backend/src/superTabs/messageReadStats.ts b/backend/src/superTabs/messageReadStats.ts
new file mode 100644
index 000000000..cba8a256f
--- /dev/null
+++ b/backend/src/superTabs/messageReadStats.ts
@@ -0,0 +1,51 @@
+import { logger } from '../utils/logger.js'
+
+/**
+ * Who reads messages and how (spec 5 A1): every minute, `messages by reader` logs each reader kind, cursor kind and sign-in
+ * session kind with its request count, the batches it read, the rows those batches decoded and the rows the pages showed.
+ * `db by shape` keeps only each minute's top shapes and says nothing of the caller. No id or text is kept.
+ */
+
+export interface MessageReadKey {
+ reader: 'person' | 'agent' | 'buddy'
+ /** `before` alone is Desktop's re-read of the page it shows; `beforeId` is scrolling back. */
+ cursor: 'latest' | 'after' | 'before' | 'beforeId' | 'thread' | 'messageId' | 'catchUp'
+ /** The Harness session the request signed in with (AuthUser.harnessSessionKind), `sso` for an Autonomous token. */
+ session: 'viewer' | 'computer' | 'team' | 'sso'
+}
+
+interface Counts { n: number; batches: number; decoded: number; shown: number }
+
+let window: Map<string, MessageReadKey & Counts> | undefined
+
+export function recordMessageRead(key: MessageReadKey, batches: number, decoded: number, shown: number): void {
+ if (!window) return
+ const id = `${key.reader}|${key.cursor}|${key.session}`
+ const row = window.get(id) ?? { reader: key.reader, cursor: key.cursor, session: key.session, n: 0, batches: 0, decoded: 0, shown: 0 }
+ row.n++
+ row.batches += batches
+ row.decoded += decoded
+ row.shown += shown
+ window.set(id, row)
+}
+
+/** Logs each window's reads, every `everyMs`, until stopped; stopping logs what is left. */
+export function startMessageReadStats({ everyMs = 60_000, log = (m, c) => logger.info(m, c) }: {
+ everyMs?: number; log?: (message: string, context: Record<string, unknown>) => void } = {}): () => void {
+ const counts = new Map<string, MessageReadKey & Counts>()
+ window = counts
+ let since = Date.now()
+ const flush = () => {
+ const now = Date.now()
+ if (counts.size) log('messages by reader', { windowMs: now - since, rows: [...counts.values()] })
+ counts.clear()
+ since = now
+ }
+ const timer = setInterval(flush, everyMs)
+ timer.unref()
+ return () => {
+ clearInterval(timer)
+ if (window === counts) window = undefined
+ flush()
+ }
+}
diff --git a/backend/src/superTabs/pageReads.ts b/backend/src/superTabs/pageReads.ts
index 54d458100..02a303c90 100644
--- a/backend/src/superTabs/pageReads.ts
+++ b/backend/src/superTabs/pageReads.ts
@@ -21,17 +21,67 @@ export async function pageReactions(tx: Transaction, tabId: string, messageIds:
export type PageDelivery = Pick<SuperTabDelivery, 'id' | 'messageId' | 'agentId' | 'status' | 'receipt' | 'holdReason' | 'holdSince'
| 'claimFailingSince' | 'claimFailures' | 'deliveredAt' | 'answeredAt' | 'createdAt' | 'updatedAt'>
-/** The deliveries of one page of messages: who each message went to and how far it got. As Prisma read them: dates as
- * Dates, and an optional field an older record lacks as null. */
+const date = (value: unknown) => value == null ? null : value instanceof Date ? value : new Date(ms(value))
+
+/** The deliveries of one page of messages: who each message went to and how far it got, from the live rows and the
+ * compacted receipts (deliveryCompaction.ts), a live row winning for the same agent. As Prisma read them: dates as Dates,
+ * and an optional field an older record lacks as null. A compacted receipt gives only these fields, never its host's. */
export async function pageDeliveries(tx: Transaction, tabId: string, messageIds: string[]): Promise<PageDelivery[]> {
const rows = await tx.superTabDelivery.aggregateRaw({ pipeline: [{ $match: { tabId, messageId: { $in: messageIds } } },
{ $project: { messageId: 1, agentId: 1, status: 1, receipt: 1, holdReason: 1, holdSince: 1, claimFailingSince: 1, claimFailures: 1, deliveredAt: 1, answeredAt: 1, createdAt: 1, updatedAt: 1 } }] }) as
unknown as Array<Record<string, unknown>>
- const date = (value: unknown) => value == null ? null : new Date(ms(value))
- return rows.map(row => ({ id: row._id as string, messageId: row.messageId as string, agentId: row.agentId as string, status: row.status as string,
- receipt: (row.receipt ?? null) as string | null, holdReason: (row.holdReason ?? null) as string | null, holdSince: date(row.holdSince),
- claimFailingSince: date(row.claimFailingSince), claimFailures: (row.claimFailures ?? null) as number | null,
- deliveredAt: date(row.deliveredAt), answeredAt: date(row.answeredAt), createdAt: date(row.createdAt)!, updatedAt: date(row.updatedAt)! }))
+ // After the rows, never beside them: a delivery compacted between the two reads is then in both, never in neither.
+ const compacted = await compactedReceipts(tx, tabId, messageIds)
+ const live = rows.map(row => pageDelivery({ ...row, id: row._id }))
+ const held = new Set(live.map(d => `${d.messageId}\0${d.agentId}`))
+ return [...live, ...compacted.filter(d => !held.has(`${d.messageId}\0${d.agentId}`)).map(pageDelivery)]
+}
+
+const pageDelivery = (row: Record<string, unknown>): PageDelivery => ({ id: row.id as string, messageId: row.messageId as string, agentId: row.agentId as string,
+ status: row.status as string, receipt: (row.receipt ?? null) as string | null, holdReason: (row.holdReason ?? null) as string | null, holdSince: date(row.holdSince),
+ claimFailingSince: date(row.claimFailingSince), claimFailures: (row.claimFailures ?? null) as number | null,
+ deliveredAt: date(row.deliveredAt), answeredAt: date(row.answeredAt), createdAt: date(row.createdAt)!, updatedAt: date(row.updatedAt)! })
+
+/** The fields a receipt document keeps of each of its deliveries, rather than per receipt. */
+const RECEIPT_DOCUMENT = new Set(['id', 'tabId', 'conversationId', 'messageId', 'agentId'])
+
+/** A delivery row as its compacted receipt stores it in its message's SuperTabReceipt (deliveryCompaction.ts), for a raw `$set`
+ * (dates as extended JSON): the row as it was, under its agent's id and with its own as `deliveryId`, without what the
+ * document holds once (its message, Swarm and conversation). The whole row is kept: a host that still holds the delivery is
+ * answered from it (store.ts deliveryReceipt, deliver), as from the row. */
+export function receiptFields(row: SuperTabDelivery): Prisma.InputJsonObject {
+ const stored: Record<string, Prisma.InputJsonValue | null> = { deliveryId: row.id }
+ for (const [key, value] of Object.entries(row)) {
+ if (!RECEIPT_DOCUMENT.has(key)) stored[key] = value instanceof Date ? { $date: value.toISOString() } : value as Prisma.InputJsonValue | null
+ }
+ return stored as Prisma.InputJsonObject
+}
+
+/** A receipt document as raw reads return it. */
+interface ReceiptDocument { _id: string; tabId: string; conversationId: string; receipts?: Record<string, Record<string, unknown>> }
+
+/** One compacted receipt as the delivery row it was: dates as Dates, a field the row did not have as null. */
+function compactedRow(document: ReceiptDocument, agentId: string, stored: Record<string, unknown>): SuperTabDelivery {
+ const row: Record<string, unknown> = {}
+ for (const [key, value] of Object.entries(stored)) {
+ row[key] = value && typeof value === 'object' && '$date' in value ? date(value) : value
+ }
+ const { deliveryId, ...fields } = row
+ return { ...fields, id: deliveryId as string, tabId: document.tabId, conversationId: document.conversationId, messageId: document._id, agentId } as SuperTabDelivery
+}
+
+/** The compacted receipts of these messages, one per agent. By the primary key; the Swarm checked in the match. */
+async function compactedReceipts(tx: Transaction, tabId: string, messageIds: string[]): Promise<SuperTabDelivery[]> {
+ const documents = await readBy(tx, 'SuperTabReceipt', 'id', [{ $match: { _id: { $in: messageIds }, tabId } }]) as unknown as ReceiptDocument[]
+ return documents.flatMap(document => Object.entries(document.receipts ?? {}).map(([agentId, stored]) => compactedRow(document, agentId, stored)))
+}
+
+/** A delivery of this agent that was compacted, as the row it was, found by its id (index [tabId, deliveryIds]); null when
+ * there is none. A host back after weeks offline still settles what its journal holds (store.ts deliveryReceipt, deliver). */
+export async function compactedDelivery(tx: Transaction, tabId: string, agentId: string, deliveryId: string): Promise<SuperTabDelivery | null> {
+ const [document] = await readBy(tx, 'SuperTabReceipt', 'delivery', [{ $match: { tabId, deliveryIds: deliveryId } }, { $limit: 1 }]) as unknown as ReceiptDocument[]
+ const stored = document?.receipts?.[agentId]
+ return document && stored?.deliveryId === deliveryId ? compactedRow(document, agentId, stored) : null
}
/** The deliveries a host is still told about (the rest are answered, cancelled or confirmed). */
@@ -143,22 +193,26 @@ export interface ThreadReply { thread: string; id: string; author: Author; menti
interface ThreadPart { thread: string; root: boolean; kind?: unknown; only?: unknown; to: boolean; session?: boolean | null }
/** One row of a part, by its place in the thread. */
interface ThreadPlace { s: number; i: string; at: unknown; author: Author; slot?: number }
+/** What the database sums up of some threads' rows (threadFacet), as it returns it: the same whoever reads. */
+export interface ThreadFacet {
+ parts: Array<{ _id: ThreadPart; count: number; oldest: ThreadPlace; newest: ThreadPlace; root?: { source?: unknown; recipientIds?: string[] } | null }>
+ mentions: Array<{ _id: ThreadPart & { person: string }; first: ThreadPlace }>
+}
/**
- * The threads rooted at `roots` in one conversation, roots included, oldest first, as a few rows that stand for all of them:
- * what the page's thread summaries are counted from (threads.ts). The database groups the rows of each part (ThreadPart) and
- * returns its oldest and newest row, how many it has (`count` on the newest), and where each person it tags is first tagged.
+ * The threads rooted at `roots` in one conversation, roots included, as the database sums them up: it groups the rows of each
+ * part (ThreadPart) and returns its oldest and newest row, how many it has, and where each person it tags is first tagged.
* That is every place a summary reads: first appearances, the newest reply of each author, the count and the last reply. So
* a poll that brings one reply of a 500-reply thread gets a few rows back, not 500, while the summaries stay what every
* row gave (integration/superTabIndexes.test.ts). Matched as the page reads are, so the thread index bounds it; the database
* still examines each of the threads' rows.
*/
-export async function pageThreads(tx: Transaction, tabId: string, conversationId: string, roots: string[]): Promise<ThreadReply[]> {
+export async function threadFacet(tx: Transaction, tabId: string, conversationId: string, roots: string[]): Promise<ThreadFacet> {
const root = { $eq: ['$_id', '$thread'] }
const part = { thread: '$thread', root, kind: '$source.kind', only: '$source.recipientsOnly',
to: { $gt: [{ $size: { $ifNull: ['$recipientIds', []] } }, 0] }, author: { kind: '$author.kind', id: '$author.id' },
session: { $eq: ['$buddySession', true] } }
- // (sequence, id) lead each place, so the database's min and max order rows as the sort below does (ids are ASCII).
+ // (sequence, id) lead each place, so the database's min and max order rows as threadReplies sorts them (ids are ASCII).
const place = { s: '$sequence', i: '$_id', at: '$createdAt', author: '$author' }
// No $sort: it makes the planner walk the conversation's sequence index instead.
const [found] = await readBy(tx, 'SuperTabMessage', 'thread', [{ $match: { tabId, conversationId, thread: { $in: roots } } },
@@ -168,11 +222,13 @@ export async function pageThreads(tx: Transaction, tabId: string, conversationId
root: { $first: { $cond: [root, { source: '$source', recipientIds: '$recipientIds' }, '$$REMOVE'] } } } }],
mentions: [{ $unwind: { path: '$mentions', includeArrayIndex: 'slot' } },
{ $group: { _id: { ...part, person: '$mentions' }, first: { $min: { s: '$sequence', i: '$_id', slot: { $toInt: '$slot' }, at: '$createdAt', author: '$author' } } } }],
- } }]) as unknown as Array<{
- parts: Array<{ _id: ThreadPart; count: number; oldest: ThreadPlace; newest: ThreadPlace; root?: { source?: unknown; recipientIds?: string[] } | null }>
- mentions: Array<{ _id: ThreadPart & { person: string }; first: ThreadPlace }>
- }>
- const parts = found?.parts ?? [], mentions = found?.mentions ?? []
+ } }]) as unknown as Array<Partial<ThreadFacet>>
+ return { parts: found?.parts ?? [], mentions: found?.mentions ?? [] }
+}
+
+/** The rows a page's thread summaries are counted from (threads.ts), oldest first, from what threadFacet gave: a few rows that
+ * stand for every row of each thread (`count` on the newest of each part), with what each reader skips marked. */
+export async function threadReplies(tx: Transaction, tabId: string, { parts, mentions }: ThreadFacet): Promise<ThreadReply[]> {
const copies = await deliveredCopies(tx, tabId, parts.flatMap(p => p._id.root && p.root
? [{ id: p._id.thread, source: p.root.source, recipientIds: p.root.recipientIds ?? [] }] : []))
// agentOnly reads only whether a row has recipients, not who they are.
@@ -193,3 +249,64 @@ export async function pageThreads(tx: Transaction, tabId: string, conversationId
return [...rows.values()].map(({ slots, ...reply }) => ({ ...reply, mentions: slots.sort((a, b) => a[0] - b[0]).map(([, person]) => person) }))
.sort((a, b) => a.sequence - b.sequence || (a.id < b.id ? -1 : 1))
}
+
+/**
+ * The rows the threads rooted at `roots` are summed up from (threadReplies). A Swarm whose summaries are backfilled
+ * (threadBackfill.ts) reads them by root id from SuperTabThread, one small document per thread with a reply; any other runs
+ * threadFacet over the threads' rows. A root with no summary has no reply, and threads.ts leaves such a thread out, so it is
+ * left out here too.
+ */
+export async function pageThreads(tx: Transaction, tabId: string, conversationId: string, roots: string[]): Promise<ThreadReply[]> {
+ if (!await threadsBackfilled(tx, tabId)) return threadReplies(tx, tabId, await threadFacet(tx, tabId, conversationId, roots))
+ const stored = await readBy(tx, 'SuperTabThread', 'id', [{ $match: { _id: { $in: roots }, tabId, conversationId } },
+ { $project: { parts: 1, mentions: 1 } }]) as unknown as ThreadFacet[]
+ return threadReplies(tx, tabId, { parts: stored.flatMap(t => t.parts), mentions: stored.flatMap(t => t.mentions) })
+}
+
+/**
+ * Rewrites the summaries of the threads rooted at `roots` in one conversation from their rows now: one document per thread
+ * with a reply, as threadFacet gives it. Run in the transaction that inserts a reply (refreshReplies), so a summary holds what
+ * its rows do: two replies at once write the same document, and one of their transactions runs again.
+ */
+export async function refreshThreads(tx: Transaction, tabId: string, conversationId: string, roots: string[]): Promise<void> {
+ if (!roots.length) return
+ const { parts, mentions } = await threadFacet(tx, tabId, conversationId, roots)
+ const updatedAt = { $date: new Date().toISOString() }
+ const updates = roots.flatMap(root => {
+ const own = parts.filter(p => p._id.thread === root)
+ if (!own.some(p => !p._id.root)) return []
+ return [{ q: { _id: root }, u: { $set: { tabId, conversationId, parts: own, mentions: mentions.filter(m => m._id.thread === root), updatedAt } },
+ upsert: true }]
+ })
+ // Raw, so the stored parts read back as the $facet returns them, dates as dates.
+ if (updates.length) await tx.$runCommandRaw({ update: 'SuperTabThread', updates: updates as unknown as Prisma.InputJsonArray })
+}
+
+/** Refreshes the summaries of the threads that `rows`, just inserted, reply in (refreshThreads). A row that is its own
+ * thread's root, or has none (written before threads), changes no summary. */
+export async function refreshReplies(tx: Transaction, tabId: string,
+ rows: ReadonlyArray<{ id: string; conversationId: string; thread?: string | null }>): Promise<void> {
+ const roots = new Map<string, Set<string>>()
+ for (const row of rows) {
+ if (row.thread && row.thread !== row.id) roots.set(row.conversationId, (roots.get(row.conversationId) ?? new Set()).add(row.thread))
+ }
+ for (const [conversationId, threads] of roots) await refreshThreads(tx, tabId, conversationId, [...threads])
+}
+
+/** How long this process trusts what it last read of a Swarm's backfill marker. */
+const BACKFILL_CHECK_MS = 60_000
+const backfilled = new Map<string, { done: boolean; at: number }>()
+
+/** Whether the Swarm's thread summaries are backfilled (threadBackfill.ts), as this process read it in the last minute. */
+async function threadsBackfilled(tx: Transaction, tabId: string): Promise<boolean> {
+ const known = backfilled.get(tabId), now = Date.now()
+ if (known && now - known.at < BACKFILL_CHECK_MS) return known.done
+ const done = !!await tx.superTabThreadBackfill.findUnique({ where: { id: tabId }, select: { id: true } })
+ backfilled.set(tabId, { done, at: now })
+ return done
+}
+
+/** Notes that this process backfilled the Swarm's summaries, so its own reads use them at once. */
+export function noteBackfilled(tabId: string): void {
+ backfilled.set(tabId, { done: true, at: Date.now() })
+}
diff --git a/backend/src/superTabs/parts/indexed.ts b/backend/src/superTabs/parts/indexed.ts
index 671df9c7b..5b510ca80 100644
--- a/backend/src/superTabs/parts/indexed.ts
+++ b/backend/src/superTabs/parts/indexed.ts
@@ -64,6 +64,13 @@ export const NAMED_INDEX = {
SuperTabDelivery: {
/** An agent's deliveries by status and age: its open ones, oldest first. */
agentOpen: { tabId: 1, agentId: 1, status: 1, createdAt: 1 },
+ /** Finished deliveries by their last change: those old enough to compact (deliveryCompaction.ts). */
+ compaction: { status: 1, updatedAt: 1 },
+ },
+ /** Compacted receipts (pageReads.ts): a page's by message id, and a host's by the compacted delivery's id. */
+ SuperTabReceipt: {
+ id: { _id: 1 },
+ delivery: { tabId: 1, deliveryIds: 1 },
},
/** A person's thread marks (readMarks.ts): `person`, every one or those moved since a cursor, the cap's count and oldest,
* and the sweep's people per Swarm; `age`, the sweep's marks untouched for 180 days. */
@@ -71,6 +78,8 @@ export const NAMED_INDEX = {
person: { tabId: 1, personId: 1, updatedAt: 1 },
age: { updatedAt: 1 },
},
+ /** A page's thread summaries, by root id (pageReads.ts pageThreads). */
+ SuperTabThread: { id: { _id: 1 } },
} as const
export type NamedCollection = keyof typeof NAMED_INDEX
export type NamedIndex<C extends NamedCollection> = keyof typeof NAMED_INDEX[C] & string
diff --git a/backend/src/superTabs/store.ts b/backend/src/superTabs/store.ts
index aa857f8d3..1d8d2cdd3 100644
--- a/backend/src/superTabs/store.ts
+++ b/backend/src/superTabs/store.ts
@@ -13,7 +13,8 @@ import { applyCommand, authorizeCommandAccess, operationSchema, PAIR_CHAIN_LIMIT
import { redactCredentials } from './credentials.js'
import type { AvatarUpload } from './profile.js'
import { reactionCounts, type ReactionTarget } from './reactions.js'
-import { agentOpenDeliveries, deliveredCopies, messagesById, messagesPast, OPEN_DELIVERY, type MessageEdge, pageDeliveries, pageReactions, pageThreads, type PageDelivery } from './pageReads.js'
+import { agentOpenDeliveries, compactedDelivery, deliveredCopies, messagesById, messagesPast, OPEN_DELIVERY, type MessageEdge, pageDeliveries, pageReactions, pageThreads, refreshReplies, type PageDelivery } from './pageReads.js'
+import { restoreDelivery } from './deliveryCompaction.js'
import { threadSummaries } from './threads.js'
import { attachmentMime, attachmentName, AttachmentTooLarge, AVATAR_PURPOSE, AVATAR_TYPES, checkedType, MAX_ATTACHMENT_BYTES, MAX_AVATAR_BYTES, objectKeyFor, peekType, publicUrl, trimPrefix,
type Attachment, type Attachments, type AttachmentStore, type AvatarStore } from './attachments.js'
@@ -66,6 +67,7 @@ import { closeAssignment, closeUnassignedStart, LIST_SIZE, newestConversations,
import { dropHeldTags, listMentions, markDone, mentionsWaiting, openTags, readMention, resolveByReply, resolveWhenWorked, writeMentions } from './mentions.js'
import { answerable, askDecisions, MAX_OPEN_DECISIONS, REFLECTION_DECISION, reflectionDecisions, expireReflections, fullFor, listDecisions, type DecisionList, readAskedDecision, recordAnswer, repeatOf, waitingFor, waitingIn, waitingInProject, writeDecisions } from './decisions.js'
import { CATCH_UP_SCAN, catchUpSummary, type CatchUp } from './catchUp.js'
+import { recordMessageRead, type MessageReadKey } from './messageReadStats.js'
import { DEFAULT_ROUTING_POLICY, effectivePolicy, routingPolicySaveSchema, type RoutingPolicy } from './routingPolicy.js'
import { invitedSwarms } from './invitedSwarms.js'
import { authenticatedEmail, authorizeSwarmJoin, configureSharing, joinSwarmSchema, requireSharingOwner,
@@ -407,6 +409,23 @@ const buddyNews = (message: Pick<SuperTabMessage, 'recipientIds'>, delivery: Pic
const readsRow = (actor: Actor, buddy: boolean) => (m: Pick<SuperTabMessage, 'source' | 'recipientIds'>): boolean =>
(!agentOnly(m.source, m.recipientIds) || (actor.kind === 'agent' && m.recipientIds.includes(actor.agentId))) && !(buddy && routingRow(m.source))
+/** A page shows 100 rows and reads a 101st to know there are more. After the first batch, each reads the rows still missing at
+ * the rate this reader has seen them so far, plus a tenth; twice what was read while none was readable; at least MIN_BATCH,
+ * at most MAX_BATCH (spec 5 A2). One hidden row costs 10 more rows, not a second batch of 101; a chat mostly of notes to
+ * others fills in about two batches, reading about what the old 101-row batches read. */
+const PAGE_BATCH = 101, MIN_BATCH = 10, MAX_BATCH = 400
+const nextBatch = (decoded: number, shown: number): number => Math.min(MAX_BATCH, Math.max(MIN_BATCH,
+ shown ? Math.ceil((PAGE_BATCH - shown) * decoded / shown * 1.1) : 2 * decoded))
+/** How a messages read counts in the read log (messageReadStats.ts). */
+const readerKind = (actor: Actor, buddy: boolean): MessageReadKey['reader'] => buddy ? 'buddy' : actor.kind
+const sessionKind = (user: AuthUser): MessageReadKey['session'] => user.harnessSessionKind ?? 'sso'
+/** A message's receipts in the order they were written, live or compacted alike: by creation, one write's in the order of
+ * its recipients (store.ts writeMessages), then by id. */
+const receiptOrder = (m: Pick<SuperTabMessage, 'recipientIds'>) => (a: PageDelivery, b: PageDelivery): number => {
+ const place = (d: PageDelivery) => { const at = m.recipientIds.indexOf(d.agentId); return at < 0 ? Infinity : at }
+ return a.createdAt.getTime() - b.createdAt.getTime() || place(a) - place(b) || (a.id < b.id ? -1 : a.id > b.id ? 1 : 0)
+}
+
/** A row as a Buddy reads it, without the machine it names (security reviews of #2115): a lost task's reason, in its text and
* its source (liveness.ts loseTask), a machine leaving the Swarm (machines.ts removalEnded, commands.ts removeMachine), and
* the machine a Product Manager started on (leadStart.ts leadJoined; Diego, 10-30) or a removed session is still held on
@@ -2050,6 +2069,8 @@ export class SuperTabStore {
}
}
if (rows.length) await tx.superTabMessage.createMany({ data: rows })
+ // A reply's thread summary, in this transaction (pageReads.ts refreshThreads).
+ await refreshReplies(tx, tabId, rows)
for (const message of messages) {
// Once committed, so a retried write logs once.
if (message.source?.kind === SESSION_ENDED_KIND && !kept.has(message.id)) {
@@ -2059,6 +2080,9 @@ export class SuperTabStore {
}
if (kept.size) {
const have = new Set((await tx.superTabDelivery.findMany({ where: { messageId: { in: [...kept] } }, select: { id: true } })).map(d => d.id))
+ // A delivery compacted into its message's receipts (deliveryCompaction.ts) was written too: never sent again.
+ const compacted = await readBy(tx, 'SuperTabReceipt', 'id', [{ $match: { _id: { $in: [...kept] }, tabId } }, { $project: { deliveryIds: 1 } }])
+ for (const receipt of compacted) for (const id of receipt.deliveryIds as string[] ?? []) have.add(id)
deliveries = deliveries.filter(d => !have.has(d.id))
}
if (deliveries.length) await tx.superTabDelivery.createMany({ data: deliveries })
@@ -2099,6 +2123,8 @@ export class SuperTabStore {
async messages(user: AuthUser, tabId: string, conversationId: string,
cursor: { after?: number; afterId?: string; before?: number; beforeId?: string; messageId?: string; thread?: string }, asAgentId?: string) {
const tx = this.db
+ const read: MessageReadKey['cursor'] = cursor.messageId ? 'messageId' : cursor.thread ? 'thread' : cursor.after !== undefined ? 'after'
+ : cursor.before === undefined ? 'latest' : cursor.beforeId === undefined ? 'before' : 'beforeId'
const { state } = await this.readLoad(user, tabId, { groupIds: [conversationId] })
const actor = await this.actor(tx, user, state, asAgentId)
const group = readConversation(state, actor, conversationId)
@@ -2133,17 +2159,23 @@ export class SuperTabStore {
: cursor.before !== undefined ? { sequence: cursor.before, id: cursor.beforeId } : undefined
// Hidden rows never shorten a page: Desktop reads on from the
// last row it shows, so the page reads past them until it has 101 rows this reader may see, or reaches the end.
- // Each batch reads on from the last row of the one before, never by an offset that walks every row before it again.
+ // Each batch reads on from the last row of the one before, never by an offset that walks every row before it again, and
+ // only about what is still missing (nextBatch): one hidden row once cost a whole second batch of 101.
const rows: SuperTabMessage[] = []
- for (let edge = start; rows.length <= 100;) {
- const batch = await messagesPast(tx, at, edge, forward, 101)
+ let batches = 0, decoded = 0
+ for (let edge = start, size = PAGE_BATCH; rows.length < PAGE_BATCH;) {
+ const batch = await messagesPast(tx, at, edge, forward, size)
+ batches++
+ decoded += batch.length
// A task chat's copy of a Lead's notice stored before it was marked is that notice: read by no one (pageReads.ts).
const copies = await deliveredCopies(tx, tabId, batch)
rows.push(...batch.filter(m => !copies.has(m.id) && readable(m)))
- if (batch.length < 101) break
+ if (batch.length < size) break
edge = { sequence: batch.at(-1)!.sequence, id: batch.at(-1)!.id }
+ size = nextBatch(decoded, rows.length)
}
const page = rows.slice(0, 100)
+ recordMessageRead({ reader: readerKind(actor, buddy), cursor: read, session: sessionKind(user) }, batches, decoded, page.length)
const next = rows.length > 100 ? { sequence: page.at(-1)!.sequence, id: page.at(-1)!.id } : null
if (!forward) page.reverse()
const reactions = page.length ? reactionCounts(state, await pageReactions(tx, tabId, page.map(m => m.id))) : new Map()
@@ -2173,7 +2205,7 @@ export class SuperTabStore {
// A session's own thread read carries the main-chat excerpt it started from (buddySessions.ts sessionContext).
const context = own && !forward && cursor.before === undefined && !cursor.messageId ? await this.sessionExcerpt(tx, tabId, conversationId, own.from, rowRead) : undefined
return { messages: shown.map(m => ({ ...m, mentions: m.mentions ?? [], reactions: reactions.get(m.id) ?? [],
- deliveries: deliveries.filter(d => d.messageId === m.id && shownTo(m, d)) })), next, threads, ...(context ? { context } : {}),
+ deliveries: deliveries.filter(d => d.messageId === m.id && shownTo(m, d)).sort(receiptOrder(m)) })), next, threads, ...(context ? { context } : {}),
// A poll for new messages learns here that a reaction changed on a message it already shows (reactions.ts).
reactionSeq: group.reactionSeq ?? 0 }
}
@@ -2206,6 +2238,7 @@ export class SuperTabStore {
// A task chat's copy of a Lead's notice stored before it was marked is read by no one, as on the page (pageReads.ts).
const copies = await deliveredCopies(this.db, tabId, scanned)
const shown = scanned.filter(m => !copies.has(m.id) && reads(m))
+ recordMessageRead({ reader: readerKind(actor, buddy), cursor: 'catchUp', session: sessionKind(user) }, 1, rows.length, shown.length)
return catchUpSummary(buddy ? shown.map(rowForBuddy) : shown, user.sub, { after, more: rows.length > CATCH_UP_SCAN, decisions })
}
@@ -2842,7 +2875,8 @@ export class SuperTabStore {
async deliveryReceipt(user: AuthUser, tabId: string, agentId: string, deliveryId: string) {
return transaction(this.db, async tx => {
const { state } = await this.load(tx, user, tabId)
- const row = await tx.superTabDelivery.findFirst({ where: { id: deliveryId, tabId, agentId } })
+ // A host back after weeks offline asks about a delivery compacted since (deliveryCompaction.ts): answered from its receipt.
+ const row = await tx.superTabDelivery.findFirst({ where: { id: deliveryId, tabId, agentId } }) ?? await compactedDelivery(tx, tabId, agentId, deliveryId)
if (!row) fail('NOT_FOUND', 'This delivery is unavailable.', 404)
const previous = state.agents.find(a => a.id === agentId && a.membershipGeneration === row.membershipGeneration)
const owner = row.machineOwnerId ?? previous?.machineOwnerId
@@ -2896,7 +2930,9 @@ export class SuperTabStore {
action: { type: 'claim'; claimId: string } | { type: 'ack'; claimId: string; receipt: string; uncertain: boolean }
| { type: 'hold'; claimId: string; reason: HoldReason }, reached: { authorized: boolean }) {
return transaction(this.db, async tx => {
- const delivery = await tx.superTabDelivery.findFirst({ where: { id: deliveryId, tabId, agentId } })
+ const live = await tx.superTabDelivery.findFirst({ where: { id: deliveryId, tabId, agentId } })
+ // A compacted delivery (deliveryCompaction.ts) is checked as its row was: finished, it is refused or resolved as before.
+ const delivery = live ?? await compactedDelivery(tx, tabId, agentId, deliveryId)
await this.row(tx, user, tabId)
// A native message may be delivered to its Lead in a different conversation from the terminal it came from.
const context = delivery?.nativeContextId ? await tx.superTabNativeContext.findFirst({
@@ -2932,6 +2968,8 @@ export class SuperTabStore {
const part = parts.members.find(p => p.key === touch.key)
if (part) await guardedWrite(tx, tabId, { op: 'touch', ...touch, seen: part.version }, stamp)
}
+ // Only a held delivery the pass ended can get here compacted: its host typed it after all, so it is a row again.
+ if (!live) await restoreDelivery(tx, delivery)
// A claim that gets in ends the failing record. Unset, never null: a missing field is a real absence, which the first failure's
// isSet: false branch matches and which leaves no keys in the document.
const updated = await tx.superTabDelivery.update({ where: { id: delivery.id }, data: { status, claimId: action.claimId,
diff --git a/backend/src/superTabs/threadBackfill.ts b/backend/src/superTabs/threadBackfill.ts
new file mode 100644
index 000000000..35f433e93
--- /dev/null
+++ b/backend/src/superTabs/threadBackfill.ts
@@ -0,0 +1,105 @@
+/**
+ * Thread summaries for replies no transaction summed up (spec 5 phase D): every thread with a reply when a Swarm has none yet,
+ * then, every 15 minutes, the threads with a reply created since an hour before the Swarm's marker. An older release
+ * writes replies without summaries, during a rollback or while a release rolls out; they are counted here. A Swarm's pages
+ * read its summaries only once its marker exists (pageReads.ts pageThreads).
+ */
+import type { PrismaClient } from '@prisma/client'
+import { logger } from '../utils/logger.js'
+import { messagesPast, noteBackfilled, refreshThreads, type MessageEdge } from './pageReads.js'
+import { groupsBy } from './parts/indexed.js'
+import { transaction } from './store.js'
+
+/** How far before its marker a Swarm's replies are read again: a reply's time is its writer's clock. */
+const SLACK_MS = 3_600_000
+/** Threads refreshed per transaction. */
+const BATCH = 100
+/** Messages read per step back through a changed conversation. */
+const TAIL = 200
+/** How often the backfill runs: replies an older release writes go uncounted at most this long. */
+const EVERY_MS = 15 * 60_000
+
+/** Refreshes the summaries of every Swarm's threads with a reply since its marker less an hour (all of them without one), then
+ * moves the marker to `now`. A Swarm that fails keeps its marker, is logged and is done at the next run. */
+export async function backfillThreads(db: PrismaClient, now: number): Promise<{ swarms: number; threads: number }> {
+ const marks = new Map((await db.superTabThreadBackfill.findMany()).map(mark => [mark.id, mark.through.getTime()]))
+ const swarms = await db.superTab.findMany({ select: { id: true } })
+ let done = 0, threads = 0
+ for (const { id: tabId } of swarms) {
+ try {
+ threads += await backfillSwarm(db, tabId, marks.get(tabId), now)
+ done++
+ } catch (error) {
+ logger.error('[swarm-threads] backfilling a Swarm failed; its pages keep their last summaries', error, { tabId })
+ }
+ }
+ return { swarms: done, threads }
+}
+
+async function backfillSwarm(db: PrismaClient, tabId: string, through: number | undefined, now: number): Promise<number> {
+ const replied = through === undefined ? await everyReply(db, tabId) : await repliesSince(db, tabId, through - SLACK_MS)
+ let threads = 0
+ for (const [conversationId, roots] of replied) {
+ threads += roots.size
+ const list = [...roots]
+ for (let i = 0; i < list.length; i += BATCH) {
+ await transaction(db, tx => refreshThreads(tx, tabId, conversationId, list.slice(i, i + BATCH)))
+ }
+ }
+ await db.superTabThreadBackfill.upsert({ where: { id: tabId }, create: { id: tabId, through: new Date(now) }, update: { through: new Date(now) } })
+ noteBackfilled(tabId)
+ return threads
+}
+
+type Replied = Map<string, Set<string>>
+const add = (replied: Replied, conversationId: string, thread: string) =>
+ replied.set(conversationId, (replied.get(conversationId) ?? new Set()).add(thread))
+
+/** Every thread of the Swarm with a reply, by conversation: its first backfill, which reads each of its messages once. */
+async function everyReply(db: PrismaClient, tabId: string): Promise<Replied> {
+ const rows = await db.superTabMessage.aggregateRaw({ pipeline: [
+ { $match: { tabId, thread: { $type: 'string' } } },
+ { $match: { $expr: { $ne: ['$thread', '$_id'] } } },
+ { $group: { _id: { conversationId: '$conversationId', thread: '$thread' } } }] }) as unknown as Array<{ _id: { conversationId: string; thread: string } }>
+ const replied: Replied = new Map()
+ for (const { _id } of rows) add(replied, _id.conversationId, _id.thread)
+ return replied
+}
+
+/** The threads with a reply created since `since`, by conversation. Only what changed is read: every message write moves its
+ * conversation's group (its last sequence), so the groups changed since then name the conversations (their changes index),
+ * and each is read newest first by its sequence index until its messages are older. History kept forever is never walked. */
+async function repliesSince(db: PrismaClient, tabId: string, since: number): Promise<Replied> {
+ const changed = await groupsBy(db, 'changes', [{ $match: { tabId, changedAt: { $gte: { $date: new Date(since).toISOString() } } } },
+ { $project: { _id: 0, key: 1 } }])
+ const replied: Replied = new Map()
+ for (const { key } of changed) {
+ const conversationId = key as string
+ for (let edge: MessageEdge | undefined; ;) {
+ const rows = await messagesPast(db, { tabId, conversationId }, edge, false, TAIL)
+ for (const row of rows) if (row.createdAt.getTime() >= since && row.thread && row.thread !== row.id) add(replied, conversationId, row.thread)
+ if (rows.length < TAIL || rows.at(-1)!.createdAt.getTime() < since) break
+ edge = { sequence: rows.at(-1)!.sequence, id: rows.at(-1)!.id }
+ }
+ }
+ return replied
+}
+
+/** Backfills now and then every `everyMs`, one run at a time, beside the worker's other jobs (pages read the $facet until a
+ * Swarm's marker exists, so nothing waits for it). Every run reads only what changed, so replies an older release wrote
+ * (a rollout, or a rollback of the API alone) are summed up within one interval. */
+export function startThreadBackfill(db: PrismaClient, everyMs = EVERY_MS): () => void {
+ let running = false
+ const run = () => {
+ if (running) return
+ running = true
+ const started = Date.now()
+ void backfillThreads(db, started).then(({ swarms, threads }) => logger.info('[swarm-threads] backfilled', { swarms, threads, ms: Date.now() - started }))
+ .catch(error => logger.error('[swarm-threads] backfill failed; it runs again at the next interval', error))
+ .finally(() => { running = false })
+ }
+ run()
+ const timer = setInterval(run, everyMs)
+ timer.unref()
+ return () => clearInterval(timer)
+}
diff --git a/backend/src/worker.ts b/backend/src/worker.ts
index a57b7a800..456f71f10 100644
--- a/backend/src/worker.ts
+++ b/backend/src/worker.ts
@@ -20,6 +20,8 @@ import { releaseSchema } from './lib/db/schemaRelease.js'
import { startSwarmSweeps } from './superTabs/sweeps.js'
import { startLegacyDrop } from './superTabs/migration/dropLegacyJob.js'
import { startDeliverySettle } from './superTabs/deliverySettle.js'
+import { startDeliveryCompaction } from './superTabs/deliveryCompaction.js'
+import { startThreadBackfill } from './superTabs/threadBackfill.js'
import { appPullRequests, markFinishedGroups, startSwarmArchive, unknownPullRequests } from './superTabs/archive.js'
import { SuperTabStore } from './superTabs/store.js'
import { announceSwarmDeliveries } from './lib/swarmDeliveryPushes.js'
@@ -66,6 +68,8 @@ async function start(): Promise<void> {
let stopLegacyDrop = (): void => {}
let stopArchive = (): void => {}
let stopDeliveries = (): void => {}
+ let stopCompaction = (): void => {}
+ let stopThreadBackfill = (): void => {}
let stopReceiptSweep = (): void => {}
let stopReflection = (): void => {}
const shutdown = (signal: string): void => {
@@ -78,8 +82,10 @@ async function start(): Promise<void> {
stopLegacyDrop()
stopArchive()
stopDeliveries()
+ stopCompaction()
stopReceiptSweep()
stopReflection()
+ stopThreadBackfill()
// The last partial minute, before the database closes.
stopQueryStats()
stopRetryStats()
@@ -107,6 +113,11 @@ async function start(): Promise<void> {
stopArchive = startSwarmArchive(prisma, swarms, () => githubApp ? appPullRequests(githubApp) : unknownPullRequests)
// A delivery that can no longer move ends as Not sent, instead of waiting for good (superTabs/deliverySettle.ts).
stopDeliveries = startDeliverySettle(prisma, swarms)
+ // Every hour, finished deliveries 14 days old move into their messages' receipts, kept forever (superTabs/deliveryCompaction.ts).
+ stopCompaction = startDeliveryCompaction(prisma, env.SWARM_DELIVERY_COMPACTION)
+ // Every 15 minutes, thread summaries for replies no write summed up: all of a Swarm's at first, then only what changed
+ // (superTabs/threadBackfill.ts). Pages read the summaries once a Swarm is backfilled.
+ stopThreadBackfill = startThreadBackfill(prisma)
// At 23:00 in each project's time zone, what its routed work cost and how it went, with routing changes asked of its owner
// (lib/routingReflection.ts).
stopReflection = startRoutingReflection(prisma, swarms)
diff --git a/backend/vitest.super-tabs.config.ts b/backend/vitest.super-tabs.config.ts
index 70a741197..6fd3c4d38 100644
--- a/backend/vitest.super-tabs.config.ts
+++ b/backend/vitest.super-tabs.config.ts
@@ -6,5 +6,6 @@ export default defineConfig({ test: { include: ['integration/superTabs.test.ts',
'integration/superTabHotReads.test.ts', 'integration/swarmSweepStarts.test.ts', 'integration/superTabGroupPlans.test.ts', 'integration/superTabInvitationPlans.test.ts', 'integration/superTabCache.test.ts', 'integration/superTabMachineReports.test.ts',
'integration/superTabObserver.test.ts', 'integration/productUsageConflicts.test.ts', 'integration/productUsageReceipts.test.ts', 'integration/schemaRelease.test.ts', 'integration/superTabBuddies.test.ts', 'integration/superTabNotAvailable.test.ts',
'integration/superTabBuddyProbes.test.ts', 'integration/superTabBuddySessions.test.ts', 'integration/superTabCosts.test.ts', 'integration/harnessSharesList.test.ts', 'integration/atlasSnapshot.test.ts', 'integration/queryStats.test.ts',
- 'integration/superTabEvidence.test.ts', 'integration/superTabAttemptModels.test.ts', 'integration/superTabNamedReads.test.ts', 'integration/swarmSweepPlans.test.ts', 'integration/workerIndexPlans.test.ts', 'integration/superTabReflection.test.ts'], hookTimeout: 120000, testTimeout: 30000, setupFiles: ['integration/cacheOn.test-support.ts'],
+ 'integration/superTabEvidence.test.ts', 'integration/superTabAttemptModels.test.ts', 'integration/superTabNamedReads.test.ts', 'integration/swarmSweepPlans.test.ts', 'integration/workerIndexPlans.test.ts', 'integration/superTabReflection.test.ts',
+ 'integration/superTabMessagePages.test.ts', 'integration/superTabDeliveryCompaction.test.ts', 'integration/superTabThreads.test.ts'], hookTimeout: 120000, testTimeout: 30000, setupFiles: ['integration/cacheOn.test-support.ts'],
fileParallelism: false, env: { NODE_ENV: 'test', HARNESS_CREDENTIAL_ENCRYPTION_KEY: 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=' } } })
diff --git a/docs/superpowers/plans/2026-10-30-swarm-messages-delta-and-retention.md b/docs/superpowers/plans/2026-10-30-swarm-messages-delta-and-retention.md
new file mode 100644
index 000000000..f27f86311
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-30-swarm-messages-delta-and-retention.md
@@ -0,0 +1,208 @@
+# Swarm messages: read what a page shows, receipt deltas, receipts kept forever — Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Cut the messages route's database reads and CPU, and keep message receipts forever while delivery rows stop growing.
+
+**Architecture:** Phase A (one release): a per-minute messages read log, continuation batches that read only what is missing, and a `SuperTabReceipt` collection that page reads merge. Phase B (only if A's log says so): receipt deltas on `?after` reads, used by Desktop and web. Phase C: a worker job (on by default) that moves final deliveries older than 14 days into `SuperTabReceipt`. Phase D: a `SuperTabThread` collection holding each replied thread's `$facet` result, refreshed when a reply is inserted, read by id instead of aggregating on every page.
+
+**Tech Stack:** TypeScript, Fastify, Prisma 5.22 on MongoDB, vitest + `mongodb-memory-server`; Flutter (Desktop and web share `desktop/lib/super_tabs/controller.dart`).
+
+**Spec:** `docs/superpowers/specs/2026-10-30-swarm-messages-delta-and-retention-design.md`
+
+## Global Constraints
+
+- Messages are kept forever; nothing here deletes a message.
+- Compact only `answered`, `cancelled`, `superseded`; never `delivered`, `confirmed`, `queued`, `claimed`, `uncertain`.
+- Compact after **14 days** since the delivery's `updatedAt`; delta slack **5 s**; a `since` older than **1 hour** answers without `receipts`.
+- Phase B goes only if Task 1's log shows Desktop/web re-reads are **at least 30%** of whole-page reads. Record the decision in the PR.
+- Compaction is **on by default** (Ken, 2026-10-30); env `SWARM_DELIVERY_COMPACTION=off` is the kill switch. Set it to `off` before rolling back to a release older than Phase A, whose page reads do not read `SuperTabReceipt`.
+- Receipts are never stored on a message document and never returned whole; every receipt shown goes through `shownTo`.
+- Do not commit or push until Ken asks; each task ends with a diff checkpoint. Remove dead code.
+- Backend commands from `backend/`: unit `npx vitest run <file>`; integration `npx vitest run --config vitest.super-tabs.config.ts <file>`. Desktop: `flutter test <file>` from `desktop/`.
+
+## Review Focus
+
+1. A host offline for more than 14 days asks `GET …/deliveries/:id` for a delivery that was compacted: it gets the final status, not 404 — Task 5 test.
+2. A notice re-written with `keepStored` after its deliveries were compacted is not delivered again — Task 5 test.
+3. A Buddy-news row's compacted receipt is hidden from everyone but that Buddy, as a live one is — Task 3 test.
+4. A conversation whose first 101 rows are mostly hidden (an agent reading a Lead chat full of notes to others) still fills its page — Task 2 test.
+5. A new Desktop talking to a backend without `since` (400 on unknown field) keeps polling — Task 7 test.
+
+---
+
+### Task 1: Messages read log (A1)
+
+**Files:**
+- Create: `backend/src/superTabs/messageReadStats.ts`
+- Modify: `backend/src/superTabs/store.ts` (`messages`, `catchUp`), `backend/src/server.ts` (start beside `startQueryStats`)
+- Test: `backend/src/superTabs/messageReadStats.test.ts`
+
+**Interfaces:**
+- Produces: `type MessageReadKey = { reader: 'person' | 'agent' | 'buddy'; cursor: 'latest' | 'after' | 'before' | 'beforeId' | 'thread' | 'messageId' | 'catchUp' }`; `recordMessageRead(key: MessageReadKey, batches: number, decoded: number, shown: number): void`; `startMessageReadStats(options?: { everyMs?: number; log?: (message: string, context: Record<string, unknown>) => void }): () => void` logging `'messages by reader'` each minute with `{ windowMs, rows: Array<MessageReadKey & { n, batches, decoded, shown }> }`.
+- `before` without `beforeId` is `'before'` (Desktop's refresh of the page it shows); with `beforeId` is `'beforeId'` (scrolling back).
+
+- [ ] **Step 1: Write the failing test** `'counts reads per reader and cursor, with batches and rows, and names nothing'`: record two `{person, before}` reads (2 batches, 202 decoded, 100 shown) and one `{agent, latest}` (1, 101, 100); run with `everyMs: 50`; expect one log row `{reader: 'person', cursor: 'before', n: 2, batches: 4, decoded: 404, shown: 200}` and one for the agent, and `JSON.stringify(context)` matching no id pattern.
+- [ ] **Step 2: Run** `npx vitest run src/superTabs/messageReadStats.test.ts` — FAIL, module missing.
+- [ ] **Step 3: Implement** the module (a `Map` keyed by `reader|cursor`, flushed per window) and call `recordMessageRead` once per `messages`/`catchUp` call with the batches, rows decoded (every row a batch returned) and rows shown; reader is `'buddy'` when `isBuddyReader(state, actor)`, else the actor's kind.
+- [ ] **Step 4: Run** the test and `npx vitest run src/superTabs` — PASS.
+- [ ] **Step 5: Checkpoint** `git diff --stat`.
+
+### Task 2: Continuation batches read what is missing (A2)
+
+**Files:**
+- Modify: `backend/src/superTabs/store.ts` (the `for (let edge = start; rows.length <= 100;)` loop in `messages`, and the same loop shape in `catchUp` if it pages the same way), `backend/src/superTabs/pageReads.ts` (`messagesPast` already takes `limit`)
+- Test: `backend/integration/superTabRead.test.ts` (new `describe('a page reads about the rows it shows (spec 5 A2)')`)
+
+**Interfaces:**
+- Consumes: Task 1's `recordMessageRead` counts, used by the test to read rows decoded.
+- Produces: no new export. Batch size after the first: `Math.max(MIN_BATCH, 101 - rows.length + 2 * hiddenSoFar)` with `MIN_BATCH = 10`.
+
+- [ ] **Step 1: Write the failing tests:**
+ - `'a page with one hidden row in its first 101 decodes at most 101 + 10 rows and returns the same page and next as before'`: a conversation of 300 messages, one agent-only note to another agent among the newest 101; read the latest page as a person; expect `decoded <= 111`, the 100 rows and `next` equal to the page the old loop returns (compute it in the test by reading 202 rows directly and filtering).
+ - `'an agent reading a chat where most rows are notes to others still gets a full page'`: 400 messages, 3 of every 4 agent-only notes to someone else; expect 100 rows shown and `next` set.
+- [ ] **Step 2: Run** `npx vitest run --config vitest.super-tabs.config.ts integration/superTabRead.test.ts -t "rows it shows"` — first test FAILS on `decoded` (202).
+- [ ] **Step 3: Implement** the batch size above in each loop; keep the stop rule (`rows.length > 100` or a short batch).
+- [ ] **Step 4: Run** `integration/superTabRead.test.ts`, `integration/superTabNamedReads.test.ts`, `integration/superTabHotReads.test.ts` — PASS.
+- [ ] **Step 5: Checkpoint** `git diff --stat`.
+
+### Task 3: `SuperTabReceipt` and page reads that merge it (A3)
+
+**Files:**
+- Modify: `backend/prisma/schema.prisma` (new model), `backend/src/superTabs/pageReads.ts` (`pageDeliveries`)
+- Test: `backend/integration/superTabRead.test.ts`
+
+**Interfaces:**
+- Produces: model
+ ```prisma
+ model SuperTabReceipt {
+ id String @id @map("_id") // the message id
+ tabId String
+ conversationId String
+ receipts Json // { [agentId]: CompactedReceipt }
+ deliveryIds String[] // the compacted deliveries' ids, so a host's receipt read finds one by id
+ @@index([tabId, conversationId])
+ @@index([tabId, deliveryIds])
+ }
+ ```
+ `type CompactedReceipt = Omit<PageDelivery, 'id' | 'messageId'> & { deliveryId: string; machineId: string; machineOwnerId: string | null; membershipGeneration: string }` exported from `pageReads.ts`; `pageDeliveries(tx, tabId, messageIds)` keeps its signature and returns live rows plus compacted ones (one `PageDelivery` per agent, `id` = the receipt's `deliveryId`; a live row wins). The host fields are never put into a `PageDelivery`.
+- [ ] **Step 1: Write the failing tests:**
+ - `'a page reads the same whether its deliveries are live, compacted or half each'`: answer three deliveries; read the page; move two into a `SuperTabReceipt` by hand and delete their rows; read again; deep-equal.
+ - `'a live row wins over a compacted one for the same agent'`.
+ - `'a compacted Buddy-news receipt is shown only to that Buddy'` (Review Focus 3).
+ - `'no message read returns a receipts map'`: `JSON.stringify` of `messages`, `catchUp`, the inbox and mentions replies has no `"receipts"` key.
+- [ ] **Step 2: Run** them — FAIL (model missing).
+- [ ] **Step 3: Implement** the model (schema push creates it) and the merge in `pageDeliveries` (one `findMany` by `_id $in` on `SuperTabReceipt` beside the deliveries read; `hotFilters.allowed.json` gains the entry if the lint asks).
+- [ ] **Step 4: Run** `integration/superTabRead.test.ts`, `src/superTabs/hotFilters.test.ts` — PASS.
+- [ ] **Step 5: Checkpoint** `git diff --stat`. **Phase A ends here: ship it, then read Task 1's log for a weekday before Tasks 6–7.**
+
+### Task 4: Compaction job (C)
+
+**Files:**
+- Create: `backend/src/superTabs/deliveryCompaction.ts`
+- Modify: `backend/src/worker.ts` (start beside `startDeliverySettle` unless `env.SWARM_DELIVERY_COMPACTION === 'off'`), `backend/src/config/env.ts` (default `'on'`)
+- Test: `backend/integration/superTabDeliveryCompaction.test.ts` (add to `vitest.super-tabs.config.ts` `include`)
+
+**Interfaces:**
+- Consumes: Task 3's model and `CompactedReceipt`.
+- Produces: `FINAL_DELIVERY = ['answered', 'cancelled', 'superseded'] as const`; `COMPACT_AFTER_MS = 14 * 86_400_000`; `compactDeliveries(db: PrismaClient, now: number, batch = 500): Promise<{ compacted: number; remaining: number }>`; `startDeliveryCompaction(db: PrismaClient, everyMs = 3_600_000): () => void` logging `'[swarm-deliveries] compacted'` with both numbers. Each hourly run calls `compactDeliveries` until it compacts nothing or 20 batches ran (up to 10 k per run; prod finishes ≤ 3.4 k deliveries a day), so the first runs also clear the backlog: no separate data migration.
+- Per batch, one transaction: read final deliveries with `updatedAt < now - COMPACT_AFTER_MS`, upsert each message's `SuperTabReceipt` with `$set: { 'receipts.<agentId>': … }` and `$addToSet: { deliveryIds: <id> }` (raw `updateOne` with `upsert`, field-path sets), delete those deliveries with `updatedAt` equal to what was read.
+- [ ] **Step 1: Write the failing tests:** only final deliveries over 14 days are compacted (a `delivered`, a `confirmed`, a `queued` and a 13-day `answered` stay); two concurrent `compactDeliveries` calls on one message keep both agents; a delivery updated between read and delete is skipped; a page is deep-equal before and after; `remaining` counts final deliveries still over 14 days.
+- [ ] **Step 2: Run** — FAIL (module missing).
+- [ ] **Step 3: Implement** the module and the worker wiring (on unless the switch says `off`; a test that `off` starts nothing).
+- [ ] **Step 4: Run** the new file and `integration/superTabRead.test.ts` — PASS.
+- [ ] **Step 5: Checkpoint** `git diff --stat`.
+
+### Task 5: Hosts and notices read compacted receipts (C)
+
+**Files:**
+- Modify: `backend/src/superTabs/store.ts` (`deliveryReceipt` ~2749, `delivery()` `typedAsEnded` ~2820, `writeMessages` `keepStored` ~2048)
+- Test: `backend/integration/superTabDeliveryCompaction.test.ts`
+
+**Interfaces:**
+- Consumes: Task 3's `SuperTabReceipt`, Task 4's `compactDeliveries`.
+- Produces: `compactedDelivery(tx, tabId, agentId, deliveryId): Promise<(CompactedReceipt & { messageId: string }) | null>` in `pageReads.ts`, found by `{ tabId, deliveryIds: deliveryId }` (index `[tabId, deliveryIds]`), then its `receipts[agentId]` with that `deliveryId`; `deliveryReceipt` and `typedAsEnded` fall back to it when no row exists, authorizing with its `machineOwnerId`/`machineId`/`membershipGeneration` exactly as with a row; `writeMessages(keepStored)` treats a message's compacted agent as an existing delivery. A plan test pins the lookup to `[tabId, deliveryIds]`.
+- [ ] **Step 1: Write the failing tests:** `deliveryReceipt` answers `{ id, status: 'answered' }` for a compacted delivery and refuses another machine owner (403) (Review Focus 1); a host confirming it typed a compacted cancelled-with-hold delivery gets the same answer as before compaction; `writeMessages(keepStored)` re-writing a notice whose deliveries were compacted creates no delivery (Review Focus 2).
+- [ ] **Step 2: Run** — FAIL (404 / re-delivered).
+- [ ] **Step 3: Implement** the three fallbacks through `compactedDelivery`.
+- [ ] **Step 4: Run** the file, the store's delivery suites (`integration/superTabs.test.ts -t deliver`) and the CLI host receipt tests (`cd ../cli && npx vitest run src/superTabs/host.spec.ts`) — PASS.
+- [ ] **Step 5: Checkpoint** `git diff --stat`.
+
+### Task 6: Receipt deltas on `?after` (B — only if the 30% gate passes)
+
+**Files:**
+- Modify: `backend/prisma/schema.prisma` (`SuperTabDelivery` gains `@@index([tabId, conversationId, updatedAt])`), `backend/src/routes/superTabs.ts:74` (`messageQuery` gains `since: z.coerce.number().int().nonnegative().safe().optional()`), `backend/src/superTabs/store.ts` (`messages`)
+- Test: `backend/integration/superTabRead.test.ts`
+
+**Interfaces:**
+- Produces: `messages(user, tabId, conversationId, cursor: { …; since?: number }, asAgentId?)` replies with `serverTime: number` (taken before any read) always, and `receipts?: Array<{ messageId: string; deliveries: PageDelivery[] }>` when `after` and `since` are given and `now - since <= 3_600_000`: messages not in `messages` whose deliveries have `updatedAt >= since - 5_000`, read by `messagesById`, kept only if `readable` and not a delivered copy, deliveries filtered by `shownTo`, compacted receipts merged as in Task 3.
+- [ ] **Step 1: Write the failing tests:** an answer after the last read appears in `receipts` on the next empty `?after`; one before does not; a hidden message's receipts never appear; `since` older than an hour gives no `receipts`; every reply has `serverTime`; an index plan test that the delta read uses `(tabId, conversationId, updatedAt)` (`integration/superTabHotReads.test.ts` style).
+- [ ] **Step 2: Run** — FAIL.
+- [ ] **Step 3: Implement.**
+- [ ] **Step 4: Run** `integration/superTabRead.test.ts`, `integration/superTabHotReads.test.ts` — PASS.
+- [ ] **Step 5: Checkpoint** `git diff --stat`.
+
+### Task 7: Desktop and web use the delta (B — only with Task 6)
+
+**Files:**
+- Modify: `desktop/lib/super_tabs/controller.dart` (`_query`, `_loadMessages`: the "No new text: refresh delivery receipts" branch)
+- Test: `desktop/test/super_tab_new_messages_test.dart` (or `super_tab_message_paging_test.dart`, whichever holds the receipt-refresh case)
+
+**Interfaces:**
+- Consumes: Task 6's `serverTime` and `receipts`.
+- Produces: per conversation, the last `serverTime` seen; `since` is added to `?after=` only once a reply from this backend carried `serverTime`, and dropped for the session after a 400.
+- [ ] **Step 1: Write the failing tests:** an empty `?after` read with `receipts` makes no `?before=<tail+1>` request and updates the held rows' deliveries; a reaction change still reads the page; a backend whose replies have no `serverTime` gets no `since` and keeps the old refresh; a 400 on `since` drops it and the next poll succeeds (Review Focus 5); the web reading policy behaves the same.
+- [ ] **Step 2: Run** `flutter test test/super_tab_new_messages_test.dart` — FAIL.
+- [ ] **Step 3: Implement.**
+- [ ] **Step 4: Run** the file and `flutter test test/super_tab_*` — PASS (compare failures with a clean `main`, which has known reds).
+- [ ] **Step 5: Checkpoint** `git diff --stat`.
+
+### Task 8: Thread summaries written with each reply (D)
+
+**Files:**
+- Modify: `backend/prisma/schema.prisma` (new model), `backend/src/superTabs/pageReads.ts` (split `pageThreads` into the `$facet` read and the post-processing), `backend/src/superTabs/store.ts` (`writeMessages`), `backend/src/superTabs/native.ts:235`
+- Test: `backend/integration/superTabThreads.test.ts` (new; add to `vitest.super-tabs.config.ts` `include`)
+
+**Interfaces:**
+- Produces: model
+ ```prisma
+ model SuperTabThread {
+ id String @id @map("_id") // the root message id
+ tabId String
+ conversationId String
+ parts Json // the $facet's `parts` for this one thread, as pageThreads reads them today
+ mentions Json // the $facet's `mentions` for this one thread
+ updatedAt DateTime
+ @@index([tabId, conversationId])
+ }
+ ```
+ In `pageReads.ts`: `threadFacet(tx, tabId, conversationId, roots: string[]): Promise<{ parts: FacetPart[]; mentions: FacetMention[] }>` (today's query, unchanged); `threadReplies(tx, tabId, facet): Promise<ThreadReply[]>` (today's post-processing, unchanged); `pageThreads` = `threadReplies(threadFacet(...))` until Task 9; `refreshThreads(tx, tabId, conversationId, roots: string[]): Promise<void>` upserts one `SuperTabThread` per root whose facet holds a reply (skips lone roots).
+- [ ] **Step 1: Write the failing tests:** a reply inserted through `store.execute` (`message.send` with `replyTo`) leaves a `SuperTabThread` whose `parts`/`mentions` deep-equal `threadFacet` for that root; a native reply (`native.ts`) does too; a lone root has no document; a reply that fails its transaction leaves no document.
+- [ ] **Step 2: Run** `npx vitest run --config vitest.super-tabs.config.ts integration/superTabThreads.test.ts` — FAIL.
+- [ ] **Step 3: Implement** the split (no behaviour change) and call `refreshThreads` for the distinct roots of inserted rows whose `thread` is set and not their own id, after the insert, in the same transaction, in both insert sites.
+- [ ] **Step 4: Run** the new file and `integration/superTabRead.test.ts` — PASS.
+- [ ] **Step 5: Checkpoint** `git diff --stat`.
+
+### Task 9: Backfill, per-Swarm switch, and reads from the collection (D)
+
+**Files:**
+- Create: `backend/src/superTabs/threadBackfill.ts`
+- Modify: `backend/src/superTabs/pageReads.ts` (`pageThreads`), `backend/src/worker.ts` (run after `releaseSchema`, before jobs start), `backend/prisma/schema.prisma` (marker model)
+- Test: `backend/integration/superTabThreads.test.ts`
+
+**Interfaces:**
+- Consumes: Task 8's `threadFacet`, `threadReplies`, `refreshThreads`.
+- Produces: model `SuperTabThreadBackfill { id String @id @map("_id") /* tabId */; through DateTime }`; `backfillThreads(db: PrismaClient, now: number): Promise<{ swarms: number; threads: number }>` — for each Swarm, refreshes every thread with a reply created after `marker.through - 3_600_000` (all of them when no marker), then sets `through = now`; `pageThreads(tx, tabId, conversationId, roots)` reads `SuperTabThread` by `_id $in roots` when the Swarm's marker exists (cached per process for 60 s), else `threadFacet` as today, and in both cases returns `threadReplies(...)`. A root without a document contributes what `threadFacet` gives for a lone root; prove it equal, or drop lone roots if `threadSummaries` ignores them.
+- [ ] **Step 1: Write the failing tests:** for a scene with lone roots and replied threads of every visibility class (agent-only notes to others, route rows, a Buddy session thread, mentions) and readers (owner, collaborator outside a private project, Buddy, agent), each reader's page `threads` deep-equals the `$facet` path; a Swarm without a marker reads through `threadFacet`; after backfill the page reads no `SuperTabMessage` aggregation for threads (count by the profiler); a reply written while the marker was stale (insert it with `refreshThreads` skipped, as an older release would) is counted after the next `backfillThreads`.
+- [ ] **Step 2: Run** them — FAIL.
+- [ ] **Step 3: Implement.**
+- [ ] **Step 4: Run** the file, `integration/superTabRead.test.ts`, `integration/superTabHotReads.test.ts` — PASS.
+- [ ] **Step 5: Checkpoint** `git diff --stat`.
+
+### Task 10: Checks and measurement
+
+- [ ] **Step 1:** `npm run typecheck` in `backend/`; `flutter analyze` in `desktop/` if Task 7 ran.
+- [ ] **Step 2:** `npm run test:super-tabs` and `npm test` in `backend/` — only failures already on clean `main`, listed in the PR.
+- [ ] **Step 3:** After Phase A's release, one weekday (07:00–16:00 VN) of `'messages by reader'` logs: whole-page reads by reader and cursor, decoded per shown (target < 1.2). Write the Phase B go/no-go into the PR with the numbers.
+- [ ] **Step 4:** After Phase C is switched on: the hourly `compacted`/`remaining` log, and weekly `SuperTabDelivery`/`SuperTabReceipt` counts and index sizes.
+- [ ] **Step 5:** After Phase D: the `$facet` shape under the messages route in `db by shape` (today ~100 a pod-minute, 3.4 MB) gone, replaced by a `SuperTabThread` read by id.
diff --git a/docs/superpowers/specs/2026-10-30-swarm-messages-delta-and-retention-design.md b/docs/superpowers/specs/2026-10-30-swarm-messages-delta-and-retention-design.md
new file mode 100644
index 000000000..f11ea48b2
--- /dev/null
+++ b/docs/superpowers/specs/2026-10-30-swarm-messages-delta-and-retention-design.md
@@ -0,0 +1,212 @@
+# Design: Swarm messages — read only what a page shows, receipts as deltas, and data that scales for years
+
+Date: 2026-10-30. Code read at `origin/main` (cdf21a617). Spec 5 in the CPU series; it covers the WebSocket roadmap's
+item 2.3 for messages and the database roadmap's Phase 5 for the message collections. Ken's decision (2026-10-30):
+**message history is kept forever.**
+
+## Why
+
+`GET /api/super-tabs/:tabId/conversations/:conversationId/messages` is the backend's largest database reader.
+
+**Prod, 2026-10-30 (60 min, 3 pods):** 298 requests per pod-minute. The `db by shape` log shows ~197 backward page reads
+per pod-minute (`SuperTabMessage` `sequence $lt`, 12.6 MB), plus the threads aggregation (`$facet`, 3.4 MB) and
+deliveries (`messageId $in`, 1.4 MB). That log keeps only each minute's top shapes (`lib/queryStats.ts`: 20 by bytes,
+10 by count, 10 by time), so small reads (an empty `?after` poll) are undercounted and **who sends the backward reads is
+not known**. Candidates, all in the code:
+- Desktop and the web build (`desktop/lib/super_tabs/controller.dart _loadMessages`, shared by `desktop/lib/web`):
+ after an empty `?after=<tail>` read, the whole last page again with `?before=<tail+1>`, to refresh receipts; scrolling
+ back; "holding" reads.
+- CLI agents (`cli/src/superTabs/service.ts:598`, the `messages` action): the latest page or a `before` page.
+- **Continuation batches inside one read** (`store.ts messages`): a page reads batches of 101 rows until it holds 101 it
+ may show. One hidden row in the first batch (an agent-only note to someone else, a route row, a delivered copy) makes
+ it read a whole second batch of 101. On a local copy of prod's busiest Swarm (`c51479bd…`), the owner's latest page
+ of the Lead chat decoded 202 rows to show 100.
+
+**Cost:** on that local copy a whole-page read costs **15 ms of process CPU** (decoding ~100–200 messages, the thread
+aggregation, deliveries); an `?after` read with nothing new **2.1 ms**.
+
+**Growth (prod, Swarm data since 2026-10-19):**
+
+| Collection | Now | Per day | In a year at this rate | Deleted today |
+|---|---|---|---|---|
+| `SuperTabMessage` | 43k docs, 41 MB | 5,471 | ~2 M docs, ~1.9 GB | never (and never will be: kept forever) |
+| `SuperTabDelivery` | 19k, 13 MB, 6 indexes | 2,376 | ~870 k, ~600 MB | never; 0 open now |
+
+At 10× users, 10× that. The M20 cluster has 4 GB of RAM per node with ~1 GB free; the first limit is indexes and
+working set against memory. Message reads stay index-bounded however long the history.
+
+## Goals
+
+- Know, per minute, who reads messages how: reader kind, cursor kind, batches and rows read against rows shown.
+- A page decodes about the rows it shows, never a whole extra batch for a few hidden rows.
+- If the numbers confirm it: a Desktop/web poll with nothing new never reads a whole page; changed receipts come as a
+ delta.
+- Message history is kept forever with its receipts, and delivery rows stop growing without bound.
+- A page reads its threads' summaries instead of aggregating every root's thread on each read.
+
+## Non-goals
+
+- Deleting or archiving messages.
+- Operation receipts' growth (`SuperTabOperation`, 7 k a day): native prompts and recoveries (`native.ts:178, 215`) and
+ `agent.remove` retries (`store.ts:1117`) read old receipts, so its expiry needs its own decision. Follow-up.
+- `SuperTabRouteDecision` growth: DB roadmap Phase 5.
+- Sharding. Message reads all name `tabId`; `SuperTabDelivery` also has machine-level indexes (`[machineId, status]`,
+ `[status, id]`) and reads by `_id`, so it is not shard-ready, and that is not changed here.
+
+## Anti-goals
+
+- A row, receipt or thread line shown to a reader who may not see it today (`readable`, `shownTo`, Buddy rules,
+ `deliveredCopies`).
+- A receipt that reads differently after its delivery is compacted, or a notice re-sent because its deliveries were
+ compacted.
+- A host that cannot settle a delivery it holds because the row is gone.
+
+## Design decided
+
+### Phase A — measure, stop over-reading, read compacted receipts (one release)
+
+**A1. Messages read log.** Every minute per pod, beside `db by shape`: request counts by reader kind (person, agent,
+Buddy) × cursor kind (latest, `after`, `before`, `thread`, `messageId`) × caller (the harness session's kind as the
+request's auth reads it: Desktop/web app vs CLI daemon), and for each: batches read, rows decoded, rows shown. No ids, no
+text.
+
+**A2. Continuation batches read what is missing.** After the first batch, a page reads only as many more rows as it
+still needs to reach 101 it may show, plus a margin (the builder picks it from the hidden-row rate, e.g. twice the
+hidden rows seen so far, at least 10), and loops while short. The page and its `next` cursor are unchanged. Applies to
+`messages`, threads and `catchUp` where they use the same loop.
+
+**A3. Compacted receipts are read.** A new collection `SuperTabReceipt` (one document per message: `_id` = message id,
+`tabId`, `conversationId`, `receipts` by agent id, and `deliveryIds`, indexed with `tabId`, so a host's receipt read
+finds a compacted delivery by its id) holds compacted deliveries (Phase C). A page read builds each
+message's `deliveries` from that message's live delivery rows plus its compacted receipts (a live row wins for the same
+agent), filtered by `shownTo` exactly as today. Receipts live in their own collection, never on the message: whole
+message rows are spread or returned in several places (`messagesPast` rows, `deliver()`, `mentions.ts:257`), and a
+receipt map there would bypass `shownTo` (security review 2 of #2178). Shipping this read before compaction is turned on
+makes a later rollback safe.
+
+### Phase B — receipt deltas (go only if A1 shows Desktop/web re-reads are at least 30% of whole-page reads)
+
+- `SuperTabDelivery` gains the index `(tabId, conversationId, updatedAt)`. Every delivery write goes through Prisma's
+ `update`, `updateMany`, `create` or `createMany`, which set `@updatedAt` (builder check, Prisma 5.22; the settle pass
+ already relies on it).
+- `messages` with `after` takes an optional `since` (the `serverTime` of the client's previous read). The reply adds
+ `receipts: [{ messageId, deliveries }]` for messages not in `messages` whose delivery rows changed at or after
+ `since − 5 s`: each changed message is read by id (`messagesById`), checked with `readable` and `deliveredCopies`, and
+ its deliveries filtered with `shownTo`, as on a page. The reply also carries `serverTime`, taken before any read.
+- A `since` older than 1 hour answers without `receipts`; the client falls back to its whole-page refresh once.
+- **Old backends refuse unknown query fields** (`agentQuery` is `.strict()`, `routes/superTabs.ts:37,74`): Desktop and
+ web send `since` only once a reply from this backend carried `serverTime`, and drop it again after a 400.
+- Desktop and web (one `controller.dart`): an empty `?after` read no longer reads `?before=<tail+1>`; the reply's
+ `receipts` are merged into the rows held. The whole-page read stays for a changed reaction (`reactionSeq`) and the first
+ read after a saved page. Web's own policy (`quietPoll` 15 s, no `messagesWhenChanged`) gets the same change.
+
+### Phase C — compact finished deliveries, keep receipts forever
+
+- Only **final** deliveries are compacted: `answered`, `cancelled`, `superseded`. `delivered` and `confirmed` are not
+ final (a reply or a `project.report` moves them to `answered`, `store.ts:1553, 1561`) and are never compacted; neither
+ are open ones (`queued`, `claimed`, `uncertain`).
+- A final delivery whose last update is over **14 days** old is moved into its message's `SuperTabReceipt` document
+ (`$set receipts.<agentId>`, so concurrent compactions of one message keep each other's agents) and deleted, in one
+ transaction per batch; a delivery that changed since it was read is skipped (filter on `updatedAt`). The compacted
+ receipt keeps the page fields (`PageDelivery`) **and** the host fields `deliveryReceipt` authorizes with (`machineId`,
+ `machineOwnerId`, `membershipGeneration`, `status`), never shown to readers.
+- **Hosts:** `deliveryReceipt` (`store.ts:2749`, the CLI's `checkReceipts`) answers a compacted delivery from its
+ receipt, so a host back after weeks offline still retires its journal entries. A host confirming it typed a cancelled,
+ held delivery (`typedAsEnded`) reads the compacted receipt the same way.
+- **Notices:** `writeMessages(keepStored)` (`store.ts:2048`) counts a compacted receipt as an existing delivery, so a
+ re-written notice is never sent again.
+- The job runs in the worker once an hour in bounded batches and logs how many it compacted and how many final
+ deliveries over 14 days old remain (the warning metric). It is **on by default** (Ken, 2026-10-30), with
+ `SWARM_DELIVERY_COMPACTION=off` as the kill switch. During the rolling deploy that ships it, pods still on the old
+ release show no receipts for compacted (14-day-old) messages and answer a host's receipt read for one with 404 until
+ they are replaced; the CLI retries.
+
+### Phase D — thread summaries in their own collection (Ken, 2026-10-30)
+
+**Why:** a thread is a field today (`SuperTabMessage.thread` = the root's id), and every page runs `pageThreads`
+(`pageReads.ts:156`), a `$facet` over all the page's roots, to count replies, their newest and oldest and their mentions,
+per visibility class. Prod: 10.4 k threads, **84% are the root alone**, 1,642 have replies (3 messages on average),
+~300 replies a day; yet the `$facet` runs on every page, ~100 a pod-minute, 3.4 MB a pod-minute (~20% of the route's
+bytes), and with history kept forever it only grows.
+
+**Design:**
+- A new collection `SuperTabThread`: one document per thread **with at least one reply**; `_id` = the root id, `tabId`,
+ `conversationId`, and the raw result today's `$facet` gives for that one thread (`parts` and `mentions`, keyed by the
+ same visibility classes: kind, recipients-only, has recipients, author, Buddy session, root). Index
+ `(tabId, conversationId)`. It holds no text and is never returned as is.
+- **Writes:** messages are insert-only (`store.ts writeMessages` and `native.ts`; nothing updates or deletes a message).
+ When a transaction inserts a reply (`thread` set and not its own id), it recomputes that thread's summary with today's
+ `$facet` for that one root and upserts the document, in the same transaction. So a summary always equals what the
+ `$facet` would compute; ~300 such writes a day.
+- **Reads:** `pageThreads` reads `SuperTabThread` by `_id $in` the page's roots and feeds the stored `parts` and
+ `mentions` to today's post-processing (`deliveredCopies` of roots, `agentOnly`, routing rows, Buddy sessions,
+ `threadSummaries`), which stays per reader as now. A root with no document has no reply; it yields what today's
+ `$facet` yields for a lone root (the builder proves the equality; if summaries drop lone roots, they are skipped).
+- **Backfill and switch:** the worker computes the summaries of every existing thread with replies (~1.6 k today, seconds)
+ and then records, per Swarm, that it is backfilled and through which time. A Swarm's reads use the collection only once
+ its marker exists; until then they run today's `$facet`. At every worker boot the backfill reruns for threads with a
+ reply newer than the marker's time minus 1 hour and moves the marker, so replies written by an older release during a
+ rollback are counted after roll-forward.
+
+## Left to the builder
+
+- A1's log name and exact keys; A2's margin; reply field names in Phase B (old clients ignore new fields).
+- Where the compaction job lives and its batch size.
+- How Desktop keeps the last `serverTime` per conversation.
+
+## Tests
+
+- **A1:** one `before` read by an agent and one latest read by a person are counted under their kinds, with batches and
+ rows; nothing names an id.
+- **A2:** a page whose first 101 rows hold 1 hidden row decodes at most ~101 + margin rows, and returns the same page and
+ `next` as today; a page with mostly hidden rows still fills; `catchUp` and threads unchanged.
+- **A3:** a page's reply is deep-equal whether a message's deliveries are live, compacted, or half each; a live row wins;
+ `shownTo` hides a compacted Buddy-news receipt as it hides a live one; no message read returns a `receipts` map.
+- **B:** a delivery answered after the last read comes back in `receipts` on the next empty `?after`; one before does not;
+ a hidden message's receipts never appear; Desktop sends `since` only after `serverTime` was seen, drops it after a 400,
+ and no longer reads `?before=<tail+1>` after an empty `?after`; a reaction change still reads the page; the web policy
+ behaves the same.
+- **C:** only `answered`/`cancelled`/`superseded` over 14 days are compacted; two concurrent compactions of one message
+ keep both agents; `deliveryReceipt` answers a compacted delivery with its final status and refuses another machine;
+ `writeMessages(keepStored)` writes no delivery for a compacted one; a page is unchanged across compaction.
+- **D:** for a conversation with lone roots, threads with replies of every visibility class (agent-only notes, route rows,
+ a Buddy session thread, mentions), each reader's `threads` from the collection deep-equals today's `$facet` result; a
+ reply inserted by `writeMessages` and by `native.ts` updates its thread in the same transaction; a Swarm without its
+ marker reads through the `$facet`; the boot rerun counts a reply written while the marker was stale.
+- Existing suites stay green: `integration/superTabRead.test.ts`, `superTabReadMarks`, `superTabHotReads`,
+ `superTabNamedReads`, the store's delivery tests, the CLI host's receipt tests, Desktop's `super_tabs` tests.
+
+## Measuring
+
+- A1's log, before Phase B: whole-page reads by caller and cursor; rows decoded per row shown.
+- The messages route's busy ms and bytes per pod-minute (`cpu by kind`, `db by shape`), before and after each phase.
+- Weekly: `SuperTabDelivery` and `SuperTabReceipt` counts and index sizes.
+
+**Done when** Phase A ships with A1's numbers reported and rows decoded per row shown under 1.2; Phase B (if it goes)
+cuts Desktop/web whole-page reads by at least 5×; Phase C keeps `SuperTabDelivery` near 14 days of final deliveries plus
+the non-final ones; Phase D removes the `$facet` from the messages route in `db by shape` (~100 a pod-minute, 3.4 MB
+today).
+
+## Design (AGENTS.md rules)
+
+- **Growth:** messages are kept forever (~5.5 k a day). Final deliveries move into `SuperTabReceipt` after 14 days (one
+ small document per message with receipts); delivery rows hold ~14 days plus non-final ones. `SuperTabThread` grows
+ with threads that have replies (~150 a day today), small documents, kept with their messages.
+- **Concurrency:** compaction writes receipts by field path in one transaction with the delete, skipping rows that
+ changed since read. Delta reads take `serverTime` before reading.
+- **Read cost:** a page decodes about the rows it shows; it reads live deliveries and compacted receipts for the page's
+ messages by id (two indexed reads), and thread summaries by id instead of the `$facet` (one read, ~16% of roots match). An empty poll with Phase B reads the new messages (none) and the conversation's
+ changed deliveries by the new index.
+- **Scale:** at 10× messages reads stay index-bounded; the first warning is index size against node RAM (`$collStats`
+ weekly) and the compaction backlog. Next steps are a larger tier, then a sharding design.
+- **Compatibility:** new reply fields only. Old Desktop/web keep their whole-page refresh. New clients send `since` only
+ to a backend that answered `serverTime`. Old CLIs read receipts through `deliveryReceipt`, which answers compacted ones.
+- **Migration + rollback:** Phase A adds a collection and reads it. Compaction is on by default; before rolling back to a
+ release older than Phase A, set `SWARM_DELIVERY_COMPACTION=off` (compacted receipts are then invisible to that old
+ release until it is rolled forward again; nothing is lost). The new index comes with the schema push.
+- **Audience:** every receipt is filtered per reader as on a page (`readable`, `deliveredCopies`, `shownTo`); compacted
+ receipts are never returned whole.
+- **Identity:** N/A because nothing here acts as or reads the machine owner's login.
+- **Failure:** a failed compaction batch logs and retries next hour; an unusable `since` answers without `receipts` and
+ the client falls back; a host asking about a compacted delivery gets its final status, not a 404.
+- **Platforms:** backend; Desktop (macOS, Windows, Linux) and the web build for Phase B.