The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
perf(swarm): messages read what a page shows, receipts kept forever, thread summaries stored (spec 5 A, C, D) (#2391)
0xken15ed12a330cc72
diff --git a/backend/integration/superTabDeliveryCompaction.test.ts b/backend/integration/superTabDeliveryCompaction.test.tsnew file mode 100644index 000000000..c9136a29f--- /dev/null+++ b/backend/integration/superTabDeliveryCompaction.test.ts@@ -0,0 +1,251 @@+import { randomUUID } from 'node:crypto'+import { afterAll, beforeAll, describe, expect, it } from 'vitest'+import type { MongoMemoryReplSet } from 'mongodb-memory-server-core'+import type { Prisma, PrismaClient, SuperTabDelivery } from '@prisma/client'+import type { AuthUser } from '../src/lib/ssoAuth.js'+import { SuperTabStore } from '../src/superTabs/store.js'+import type { Command } from '../src/superTabs/commands.js'+import { COMPACT_AFTER_MS, compactDeliveries, startDeliveryCompaction } from '../src/superTabs/deliveryCompaction.js'+import { listNamedIndexes, withoutHints } from '../src/superTabs/parts/indexed.js'+import { planDatabase, plan, profiled, reads } from './plans.test-support.js'++// Spec 5 phase C (docs/superpowers/specs/2026-10-30-swarm-messages-delta-and-retention-design.md): finished deliveries move+// into their messages' receipts after 14 days; pages, hosts and notices read them as the rows they were.+let replica: MongoMemoryReplSet+let db: PrismaClient+let store: SuperTabStore+const owner: AuthUser = { sub: '600000000000000000000001', email: '[email protected]', autonomousEnv: 'prod', role: 'user' }+const host: AuthUser = { sub: '600000000000000000000003', email: '[email protected]', autonomousEnv: 'prod', role: 'user' }+const config = { engine: 'codex', model: 'fixture-model', effort: 'high', contextWindow: 200000, engineVersion: 'fixture' }+const DAY = 86_400_000++beforeAll(async () => {+ ({ replica, db } = await planDatabase('swarm_compaction_test'))+ store = new SuperTabStore(db)+ await listNamedIndexes(db)+ for (const user of [owner, host]) await db.user.create({ data: { id: user.sub, externalId: `fixture-${user.sub}`, email: user.email, autonomousEnv: 'prod' } })+ await db.machine.create({ data: { machineId: 'test-host', userId: host.sub, apiKey: 'fixture-only', managerId: 'fixture', autonomousEnv: 'prod', authMode: 'remote' } })+ await db.machineShare.create({ data: { machineId: 'test-host', ownerId: host.sub, autonomousEnv: 'prod', emails: [owner.email] } })+})+afterAll(async () => {+ await db?.$disconnect()+ await replica?.stop()+})++const run = (tab: string, command: Command, user = owner, asAgentId?: string) => store.execute(user, tab, { operationId: randomUUID(), command }, asAgentId)+async function fixture() {+ const tab = (await store.create(owner, randomUUID(), 'Receipts')).id+ const lead = String((await run(tab, { type: 'agent.add', conversationId: 'lead', machineId: 'test-host', harnessId: randomUUID(), name: 'Lead',+ configuration: config, lead: true })).result.agentId)+ return { tab, lead }+}+/** A message to the Lead and its delivery, claimed by the Lead's host. */+async function sent(tab: string, lead: string, text: string) {+ const messageId = String((await run(tab, { type: 'message.send', conversationId: 'lead', text, replyTo: null, recipientIds: [] })).result.messageId)+ const delivery = (await store.deliveries(host, tab, lead)).find(d => d.messageId === messageId)!+ await store.delivery(host, tab, lead, delivery.id, { type: 'claim', claimId: `claim-${text}` })+ return { messageId, deliveryId: delivery.id, claimId: `claim-${text}` }+}+/** A message to the Lead typed and answered. */+async function answered(tab: string, lead: string, text: string) {+ const one = await sent(tab, lead, text)+ await store.delivery(host, tab, lead, one.deliveryId, { type: 'ack', claimId: one.claimId, receipt: `typed-${text}`, uncertain: false })+ await run(tab, { type: 'message.send', conversationId: 'lead', text: `Re: ${text}`, replyTo: one.messageId, recipientIds: [] }, host, lead)+ return one+}+/** Moves these deliveries' last change `days` back, as if nothing touched them since. */+async function age(ids: string[], days: number) {+ await db.$runCommandRaw({ update: 'SuperTabDelivery', updates: ids.map(id => ({ q: { _id: id },+ u: { $set: { updatedAt: { $date: new Date(Date.now() - days * DAY).toISOString() } } } })) } as Prisma.InputJsonObject)+}+/** A delivery written straight into the database, for statuses no short flow reaches. */+async function row(tab: string, lead: string, status: string, fields: Partial<SuperTabDelivery> = {}) {+ const messageId = `${tab}-${randomUUID()}`+ await db.superTabMessage.create({ data: { id: messageId, tabId: tab, conversationId: 'lead', sequence: 500, text: status, replyTo: null,+ recipientIds: [fields.agentId ?? lead], mentions: [], thread: messageId, author: { kind: 'person', id: owner.sub, name: owner.email, membershipGeneration: 'g' } } })+ return db.superTabDelivery.create({ data: { id: `${messageId}-d`, tabId: tab, conversationId: 'lead', messageId, agentId: lead, machineId: 'test-host',+ machineOwnerId: host.sub, membershipGeneration: 'g', authorKind: 'person', authorId: owner.sub, authorGeneration: 'g', status, ...fields } })+}+const receiptOf = async (messageId: string) => (await db.$runCommandRaw({ find: 'SuperTabReceipt', filter: { _id: messageId } }) as unknown as+ { cursor: { firstBatch: Array<{ receipts: Record<string, { deliveryId: string; status: string }>; deliveryIds: string[] }> } }).cursor.firstBatch[0]++describe('compacting finished deliveries (spec 5 C)', () => {+ it('moves only answered, cancelled and superseded deliveries whose last change is over 14 days old', async () => {+ const { tab, lead } = await fixture()+ const final = await Promise.all(['answered', 'cancelled', 'superseded'].map(status => row(tab, lead, status)))+ const kept = await Promise.all(['delivered', 'confirmed', 'queued', 'claimed', 'uncertain'].map(status => row(tab, lead, status)))+ const recent = await row(tab, lead, 'answered')+ await age([...final, ...kept].map(d => d.id), 15)+ await age([recent.id], 13)+ // Delivered and confirmed ones stay rows, counted as unfinished.+ expect(await compactDeliveries(db, Date.now())).toEqual({ compacted: 3, remaining: 0, unfinished: 2 })+ const left = await db.superTabDelivery.findMany({ where: { tabId: tab }, select: { id: true } })+ expect(left.map(d => d.id).sort()).toEqual([...kept, recent].map(d => d.id).sort())+ for (const delivery of final) {+ expect(await receiptOf(delivery.messageId)).toMatchObject({ deliveryIds: [delivery.id],+ receipts: { [lead]: { deliveryId: delivery.id, status: delivery.status, machineOwnerId: host.sub } } })+ }+ expect(await compactDeliveries(db, Date.now())).toMatchObject({ compacted: 0, remaining: 0 })+ })++ it('keeps every agent of one message, compacted one after another or at once', async () => {+ const { tab, lead } = await fixture()+ const first = await row(tab, lead, 'answered')+ const other = (agentId: string) => db.superTabDelivery.create({ data: { ...first, id: `${first.messageId}-${agentId}`, agentId, updatedAt: undefined } })+ const second = await other('agent-b')+ await age([first.id], 15)+ await compactDeliveries(db, Date.now())+ await age([second.id], 15)+ await compactDeliveries(db, Date.now())+ const third = await other('agent-c'), fourth = await other('agent-d')+ await age([third.id, fourth.id], 15)+ await Promise.allSettled([compactDeliveries(db, Date.now(), 1), compactDeliveries(db, Date.now(), 1)])+ await compactDeliveries(db, Date.now())+ const receipt = await receiptOf(first.messageId)+ expect(Object.keys(receipt!.receipts).sort()).toEqual([lead, 'agent-b', 'agent-c', 'agent-d'].sort())+ expect(receipt!.deliveryIds.sort()).toEqual([first, second, third, fourth].map(d => d.id).sort())+ expect(await db.superTabDelivery.count({ where: { messageId: first.messageId } })).toBe(0)+ })++ it('leaves a delivery that changed after it was read, and moves it once it is old again', async () => {+ const { tab, lead } = await fixture()+ const delivery = await row(tab, lead, 'cancelled')+ await age([delivery.id], 15)+ let changed = false+ // Another writer settles the delivery after the batch read it and before it writes (a host typing a held one).+ const racing = new Proxy(db, { get(target, key, receiver) {+ if (key !== '$transaction') return Reflect.get(target, key, receiver)+ return (body: (tx: PrismaClient) => Promise<unknown>, options: unknown) => (target.$transaction as (b: unknown, o: unknown) => Promise<unknown>)(+ (tx: PrismaClient) => body(new Proxy(tx, { get(inner, name, innerReceiver) {+ const value = Reflect.get(inner, name, innerReceiver)+ if (name !== '$runCommandRaw' || changed) return value+ return async (...args: unknown[]) => {+ changed = true+ await db.superTabDelivery.update({ where: { id: delivery.id }, data: { receipt: 'typed late' } })+ return (value as (...a: unknown[]) => unknown).apply(inner, args)+ }+ } })), options)+ } }) as PrismaClient+ expect(await compactDeliveries(racing, Date.now())).toMatchObject({ compacted: 0, remaining: 0 })+ expect(changed).toBe(true)+ expect(await db.superTabDelivery.findUnique({ where: { id: delivery.id } })).toMatchObject({ receipt: 'typed late' })+ expect(await receiptOf(delivery.messageId)).toBeUndefined()+ await age([delivery.id], 15)+ expect(await compactDeliveries(db, Date.now())).toMatchObject({ compacted: 1, remaining: 0 })+ })++ it('reads a page the same before and after compaction, for a person and for the Lead', async () => {+ const { tab, lead } = await fixture()+ const done = [await answered(tab, lead, 'One'), await answered(tab, lead, 'Two')]+ await sent(tab, lead, 'Waiting')+ await age(done.map(d => d.deliveryId), 15)+ const before = [await store.messages(owner, tab, 'lead', {}), await store.messages(host, tab, 'lead', {}, lead)]+ expect((await compactDeliveries(db, Date.now())).compacted).toBe(2)+ expect([await store.messages(owner, tab, 'lead', {}), await store.messages(host, tab, 'lead', {}, lead)]).toEqual(before)+ })++ it('stops after 20 batches and counts what is left as remaining', async () => {+ const { tab, lead } = await fixture()+ const rows = await Promise.all(Array.from({ length: 23 }, () => row(tab, lead, 'answered')))+ await age(rows.map(d => d.id), 15)+ expect(await compactDeliveries(db, Date.now(), 1)).toMatchObject({ compacted: 20, remaining: 3 })+ expect(await compactDeliveries(db, Date.now(), 1)).toMatchObject({ compacted: 3, remaining: 0 })+ })++ it('leaves a delivery whose agent id cannot name a receipt as a row, without holding the others back', async () => {+ const { tab, lead } = await fixture()+ const odd = await row(tab, lead, 'answered', { agentId: 'agent.with.dots' })+ const rows = [await row(tab, lead, 'answered'), await row(tab, lead, 'cancelled')]+ await age([odd.id], 16)+ await age(rows.map(d => d.id), 15)+ try {+ expect(await compactDeliveries(db, Date.now(), 1)).toMatchObject({ compacted: 2, remaining: 1 })+ expect(await db.superTabDelivery.findUnique({ where: { id: odd.id } })).not.toBeNull()+ expect(await receiptOf(odd.messageId)).toBeUndefined()+ } finally {+ await db.superTabDelivery.delete({ where: { id: odd.id } })+ }+ })++ it('reads only what it moves, by its index', async () => {+ const { tab, lead } = await fixture()+ await Promise.all(Array.from({ length: 30 }, () => row(tab, lead, 'answered')))+ const old = await Promise.all(Array.from({ length: 4 }, () => row(tab, lead, 'cancelled')))+ await age(old.map(d => d.id), 15)+ // As the worker reads, which lists no named index: the planner picks it.+ const entries = await profiled(db, () => withoutHints(() => compactDeliveries(db, Date.now())))+ const found = reads(entries, 'SuperTabDelivery', ['status', 'updatedAt'], ['_id'])+ expect(found.length, JSON.stringify(plan(found))).toBeGreaterThan(0)+ for (const entry of found) {+ expect(entry.docsExamined, JSON.stringify(plan(found))).toBeLessThanOrEqual(4)+ expect(entry.planSummary, JSON.stringify(plan(found))).toContain('status: 1, updatedAt: 1')+ }+ })++ it('starts nothing when the kill switch is off', () => {+ const untouchable = new Proxy({}, { get() { throw new Error('read the database') } }) as PrismaClient+ const stop = startDeliveryCompaction(untouchable, 'off')+ stop()+ expect(COMPACT_AFTER_MS).toBe(14 * DAY)+ })+})++describe('hosts read compacted deliveries (spec 5 C)', () => {+ it('a host back after weeks offline gets a compacted delivery\'s final status; another person is refused', async () => {+ const { tab, lead } = await fixture()+ const done = await answered(tab, lead, 'Old')+ await age([done.deliveryId], 15)+ await compactDeliveries(db, Date.now())+ expect(await db.superTabDelivery.findUnique({ where: { id: done.deliveryId } })).toBeNull()+ expect(await store.deliveryReceipt(host, tab, lead, done.deliveryId)).toEqual({ id: done.deliveryId, status: 'answered' })+ await expect(store.deliveryReceipt(owner, tab, lead, done.deliveryId)).rejects.toMatchObject({ code: 'FORBIDDEN', status: 403 })+ await expect(store.deliveryReceipt(host, tab, 'another-agent', done.deliveryId)).rejects.toMatchObject({ status: 404 })+ await expect(store.deliveryReceipt(host, tab, lead, 'not-a-delivery')).rejects.toMatchObject({ code: 'NOT_FOUND', status: 404 })+ // A compacted delivery is finished: it is resolved, as its row was.+ await expect(store.delivery(host, tab, lead, done.deliveryId, { type: 'ack', claimId: done.claimId, receipt: 'again', uncertain: false }))+ .rejects.toMatchObject({ code: 'DELIVERY_RESOLVED' })+ })++ it('finds a compacted delivery by its index, examining only its receipt, and a page\'s receipts by message id', async () => {+ const { tab, lead } = await fixture()+ const done = await Promise.all(['A', 'B', 'C'].map(text => answered(tab, lead, text)))+ await age(done.map(d => d.deliveryId), 15)+ await compactDeliveries(db, Date.now())+ const hostReads = await profiled(db, () => store.deliveryReceipt(host, tab, lead, done[1]!.deliveryId))+ const byDelivery = reads(hostReads, 'SuperTabReceipt', ['deliveryIds'])+ expect(byDelivery, JSON.stringify(plan(byDelivery))).toHaveLength(1)+ expect(byDelivery[0]!.docsExamined, JSON.stringify(plan(byDelivery))).toBeLessThanOrEqual(1)+ expect(byDelivery[0]!.command?.hint, `names its index: ${JSON.stringify(plan(byDelivery))}`).toBeTruthy()+ const page = await profiled(db, () => store.messages(owner, tab, 'lead', {}))+ const byMessage = reads(page, 'SuperTabReceipt', ['_id'])+ expect(byMessage, JSON.stringify(plan(byMessage))).toHaveLength(1)+ expect(byMessage[0]!.docsExamined, JSON.stringify(plan(byMessage))).toBeLessThanOrEqual(3)+ expect(byMessage[0]!.command?.hint, `names its index: ${JSON.stringify(plan(byMessage))}`).toBeTruthy()+ })++ it('a host confirming it typed a held delivery the pass ended gets the same answer compacted or not', async () => {+ const { tab, lead } = await fixture()+ const ended = async (text: string) => {+ const one = await sent(tab, lead, text)+ await store.delivery(host, tab, lead, one.deliveryId, { type: 'hold', claimId: one.claimId, reason: 'team_waiting_user' })+ // The delivery pass ends a hold it waited on too long, keeping its reason (deliverySettle.ts).+ await db.superTabDelivery.update({ where: { id: one.deliveryId }, data: { status: 'cancelled' } })+ return one+ }+ const live = await ended('Live'), compacted = await ended('Compacted')+ await age([compacted.deliveryId], 15)+ expect((await compactDeliveries(db, Date.now())).compacted).toBe(1)+ const typed = async (one: typeof live) =>+ await store.delivery(host, tab, lead, one.deliveryId, { type: 'ack', claimId: one.claimId, receipt: 'typed', uncertain: false }) as {+ delivery: SuperTabDelivery; message: { id: string } }+ const [a, b] = [await typed(live), await typed(compacted)]+ const shape = (answer: typeof a, one: typeof live) => ({ ...answer, delivery: { ...answer.delivery, updatedAt: null, deliveredAt: null, createdAt: null,+ id: answer.delivery.id === one.deliveryId, messageId: answer.delivery.messageId === one.messageId, claimId: answer.delivery.claimId === one.claimId },+ message: { ...answer.message, id: answer.message.id === one.messageId, text: null, sequence: null, createdAt: null, thread: null } })+ expect(shape(b, compacted)).toEqual(shape(a, live))+ expect(b.delivery).toMatchObject({ status: 'delivered', receipt: 'typed', holdReason: null })+ expect(await receiptOf(compacted.messageId)).toMatchObject({ receipts: {}, deliveryIds: [] })+ const page = await store.messages(owner, tab, 'lead', {})+ expect(page.messages.find(m => m.id === compacted.messageId)!.deliveries).toEqual([expect.objectContaining({ id: compacted.deliveryId, status: 'delivered' })])+ })+})diff --git a/backend/integration/superTabLiveness.test.ts b/backend/integration/superTabLiveness.test.tsindex a0f783cb2..6c2972298 100644--- a/backend/integration/superTabLiveness.test.ts+++ b/backend/integration/superTabLiveness.test.ts@@ -305,6 +305,19 @@ describe('work that loses its owner', () => {expect(await db.superTabDelivery.count({ where: { tabId: f.tab, messageId: ids.lead } })).toBe(1)})+ it('gets no second delivery when its delivery was compacted into its receipts (spec 5 C)', async () => {+ const f = await swarm()+ f.clock.now += MINUTE+ const ids = lostIds(f), compacted = stableId(ids.lead, f.lead)+ await store(f, ids.lead, 'Stored for the Lead')+ await db.$runCommandRaw({ update: 'SuperTabReceipt', updates: [{ q: { _id: ids.lead }, upsert: true, u: {+ $set: { tabId: f.tab, conversationId: f.group, [`receipts.${f.lead}`]: { deliveryId: compacted, status: 'answered' } },+ $addToSet: { deliveryIds: compacted } } }] })+ await f.observe('test-host', [], { stoppedHarnessIds: [f.harness.worker] })+ expect(await f.assignment()).toMatchObject({ status: 'lost' })+ expect(await db.superTabDelivery.count({ where: { tabId: f.tab, messageId: ids.lead } })).toBe(0)+ })+it('still refuses a stored message that is not this notice', async () => {const f = await swarm()f.clock.now += MINUTEdiff --git a/backend/integration/superTabMessagePages.test.ts b/backend/integration/superTabMessagePages.test.tsnew file mode 100644index 000000000..1c348c2de--- /dev/null+++ b/backend/integration/superTabMessagePages.test.ts@@ -0,0 +1,224 @@+import { randomUUID } from 'node:crypto'+import { afterAll, beforeAll, describe, expect, it } from 'vitest'+import type { MongoMemoryReplSet } from 'mongodb-memory-server-core'+import type { Prisma, PrismaClient, SuperTabMessage } from '@prisma/client'+import type { AuthUser } from '../src/lib/ssoAuth.js'+import { SuperTabStore } from '../src/superTabs/store.js'+import type { Command } from '../src/superTabs/commands.js'+import { agentOnly } from '../src/superTabs/model.js'+import { receiptFields } from '../src/superTabs/pageReads.js'+import { startMessageReadStats, type MessageReadKey } from '../src/superTabs/messageReadStats.js'+import { planDatabase } from './plans.test-support.js'++// Spec 5 phase A (docs/superpowers/specs/2026-10-30-swarm-messages-delta-and-retention-design.md): who reads messages how, a+// page that decodes about the rows it shows, and receipts read the same whether their deliveries are live or compacted.+let replica: MongoMemoryReplSet+let db: PrismaClient+let store: SuperTabStore+const owner: AuthUser = { sub: '500000000000000000000001', email: '[email protected]', autonomousEnv: 'prod', role: 'user' }+const host: AuthUser = { sub: '500000000000000000000003', email: '[email protected]', autonomousEnv: 'prod', role: 'user' }+const config = { engine: 'codex', model: 'fixture-model', effort: 'high', contextWindow: 200000, engineVersion: 'fixture' }++beforeAll(async () => {+ ({ replica, db } = await planDatabase('swarm_pages_test'))+ store = new SuperTabStore(db)+ for (const user of [owner, host]) await db.user.create({ data: { id: user.sub, externalId: `fixture-${user.sub}`, email: user.email, autonomousEnv: 'prod' } })+ await db.machine.create({ data: { machineId: 'test-host', userId: host.sub, apiKey: 'fixture-only', managerId: 'fixture', autonomousEnv: 'prod', authMode: 'remote' } })+ await db.machineShare.create({ data: { machineId: 'test-host', ownerId: host.sub, autonomousEnv: 'prod', emails: [owner.email] } })+})+afterAll(async () => {+ await db?.$disconnect()+ await replica?.stop()+})++const run = (tab: string, command: Command, user = owner, asAgentId?: string) => store.execute(user, tab, { operationId: randomUUID(), command }, asAgentId)+async function fixture() {+ const tab = (await store.create(owner, randomUUID(), 'Pages')).id+ const lead = String((await run(tab, { type: 'agent.add', conversationId: 'lead', machineId: 'test-host', harnessId: randomUUID(), name: 'Lead',+ configuration: config, lead: true })).result.agentId)+ return { tab, lead }+}++/** `count` messages in the Lead chat from sequence 1000 on; `hidden(i)` ones are agent-only notes to another agent. */+async function seed(tab: string, count: number, hidden: (i: number) => boolean) {+ const rows: Prisma.SuperTabMessageCreateManyInput[] = Array.from({ length: count }, (_, i) => {+ const id = `${tab}-m${String(i).padStart(4, '0')}`+ return { id, tabId: tab, conversationId: 'lead', sequence: 1000 + i, author: { kind: 'person', id: owner.sub, name: owner.email, membershipGeneration: 'g' },+ text: `row ${i}`, replyTo: null, recipientIds: hidden(i) ? ['someone-else'] : [], mentions: [], thread: id,+ ...(hidden(i) ? { source: { kind: 'agent-only' } } : {}) }+ })+ await db.superTabMessage.createMany({ data: rows })+}++/** The newest page and its cursor as every row of the conversation gives them, for a reader who sees what `sees` keeps. */+async function expectedPage(tab: string, sees: (m: SuperTabMessage) => boolean) {+ const all = await db.superTabMessage.findMany({ where: { tabId: tab, conversationId: 'lead' }, orderBy: [{ sequence: 'desc' }, { id: 'desc' }] })+ const shown = all.filter(sees)+ const page = shown.slice(0, 100)+ return { ids: page.map(m => m.id).reverse(), next: shown.length > 100 ? { sequence: page.at(-1)!.sequence, id: page.at(-1)!.id } : null }+}+const personSees = (m: SuperTabMessage) => !agentOnly(m.source, m.recipientIds)+const agentSees = (agentId: string) => (m: SuperTabMessage) => personSees(m) || m.recipientIds.includes(agentId)++type Counted = MessageReadKey & { n: number; batches: number; decoded: number; shown: number }+/** What `read` adds to the messages read log. */+async function counted<T>(read: () => Promise<T>): Promise<{ value: T; rows: Counted[] }> {+ const rows: Counted[] = []+ const stop = startMessageReadStats({ everyMs: 3_600_000, log: (_message, context) => rows.push(...context.rows as Counted[]) })+ try {+ return { value: await read(), rows }+ } finally {+ stop()+ }+}++describe('messages by reader (spec 5 A1)', () => {+ it('counts an agent\'s older page and a person\'s newest page under their kinds, with batches and rows, naming nothing', async () => {+ const { tab, lead } = await fixture()+ await seed(tab, 150, () => false)+ const edge = (await db.superTabMessage.findFirstOrThrow({ where: { tabId: tab, sequence: 1100 } }))+ const { rows } = await counted(async () => {+ await store.messages(host, tab, 'lead', { before: edge.sequence, beforeId: edge.id }, lead)+ await store.messages(owner, tab, 'lead', {})+ await store.messages(owner, tab, 'lead', { before: 1150 })+ await store.catchUp(owner, tab, 'lead', 1140)+ })+ const row = (reader: string, cursor: string) => rows.find(r => r.reader === reader && r.cursor === cursor)+ expect(row('agent', 'beforeId')).toMatchObject({ session: 'sso', n: 1, batches: 1, shown: 100 })+ expect(row('person', 'latest')).toMatchObject({ session: 'sso', n: 1, batches: 1, decoded: 101, shown: 100 })+ expect(row('person', 'before')).toMatchObject({ n: 1, shown: 100 })+ expect(row('person', 'catchUp')).toMatchObject({ n: 1, batches: 1, decoded: 9, shown: 9 })+ expect(JSON.stringify(rows)).not.toMatch(/[0-9a-f]{12,}|@|tab-/i)+ })+})++describe('a page reads about the rows it shows (spec 5 A2)', () => {+ it('a page with one hidden row in its first 101 decodes at most 101 + 10 rows and returns the same page and next as before', async () => {+ const { tab } = await fixture()+ await seed(tab, 300, i => i === 250)+ const { value, rows } = await counted(() => store.messages(owner, tab, 'lead', {}))+ const expected = await expectedPage(tab, personSees)+ expect(value.messages.map(m => m.id)).toEqual(expected.ids)+ expect(value.next).toEqual(expected.next)+ expect(rows).toEqual([expect.objectContaining({ reader: 'person', cursor: 'latest', n: 1, shown: 100 })])+ expect(rows[0]!.decoded).toBeLessThanOrEqual(111)+ })++ it('an agent reading a chat where most rows are notes to others still gets a full page, and the next page goes on from it', async () => {+ const { tab, lead } = await fixture()+ await seed(tab, 440, i => i % 4 !== 0)+ const { value: first, rows } = await counted(() => store.messages(host, tab, 'lead', {}, lead))+ // The old 101-row batches read 404 rows in 4 batches for this page; sized by the rate seen, two batches read about as many.+ expect(rows[0]).toMatchObject({ reader: 'agent', cursor: 'latest', shown: 100 })+ expect(rows[0]!.batches).toBeLessThanOrEqual(2)+ expect(rows[0]!.decoded).toBeLessThanOrEqual(485)+ const expected = await expectedPage(tab, agentSees(lead))+ expect(first.messages).toHaveLength(100)+ expect(first.messages.map(m => m.id)).toEqual(expected.ids)+ expect(first.next).toEqual(expected.next)+ const second = await store.messages(host, tab, 'lead', { before: first.next!.sequence, beforeId: first.next!.id }, lead)+ expect(second.messages.length).toBeGreaterThan(0)+ expect(second.messages.at(-1)!.sequence).toBeLessThan(first.messages[0]!.sequence)+ })++ it('a conversation that ends inside a short batch reads it once and stops', async () => {+ const { tab } = await fixture()+ await seed(tab, 105, i => i >= 100)+ const { value, rows } = await counted(() => store.messages(owner, tab, 'lead', {}))+ expect(value.messages.map(m => m.id)).toEqual((await expectedPage(tab, personSees)).ids)+ expect(rows[0]!.batches).toBeLessThanOrEqual(2)+ })+})++/** Moves deliveries into their messages' receipts and deletes them, as the compaction job does (deliveryCompaction.ts). */+async function compactByHand(ids: string[]) {+ for (const row of await db.superTabDelivery.findMany({ where: { id: { in: ids } } })) {+ await db.$runCommandRaw({ update: 'SuperTabReceipt', updates: [{ q: { _id: row.messageId }, upsert: true,+ u: { $set: { tabId: row.tabId, conversationId: row.conversationId, [`receipts.${row.agentId}`]: receiptFields(row) }, $addToSet: { deliveryIds: row.id } } }] })+ await db.superTabDelivery.delete({ where: { id: row.id } })+ }+}++describe('receipts read the same live or compacted (spec 5 A3)', () => {+ /** Three messages to the Lead, each claimed, typed and answered, and a fourth still waiting. */+ async function answered() {+ const f = await fixture()+ for (const text of ['One', 'Two', 'Three']) {+ const sent = String((await run(f.tab, { type: 'message.send', conversationId: 'lead', text, replyTo: null, recipientIds: [] })).result.messageId)+ const delivery = (await store.deliveries(host, f.tab, f.lead)).find(d => d.messageId === sent)!+ await store.delivery(host, f.tab, f.lead, delivery.id, { type: 'claim', claimId: `claim-${text}` })+ await store.delivery(host, f.tab, f.lead, delivery.id, { type: 'ack', claimId: `claim-${text}`, receipt: `typed-${text}`, uncertain: false })+ await run(f.tab, { type: 'message.send', conversationId: 'lead', text: `Re: ${text}`, replyTo: sent, recipientIds: [] }, host, f.lead)+ }+ await run(f.tab, { type: 'message.send', conversationId: 'lead', text: 'Four', replyTo: null, recipientIds: [] })+ const deliveries = await db.superTabDelivery.findMany({ where: { tabId: f.tab }, orderBy: { createdAt: 'asc' } })+ return { ...f, deliveries }+ }++ it('a page reads the same whether its deliveries are live, compacted or half each', async () => {+ const f = await answered()+ expect(f.deliveries.map(d => d.status)).toEqual(['answered', 'answered', 'answered', 'queued'])+ const live = await store.messages(owner, f.tab, 'lead', {}), liveToLead = await store.messages(host, f.tab, 'lead', {}, f.lead)+ expect(live.messages.flatMap(m => m.deliveries)).toHaveLength(4)+ await compactByHand(f.deliveries.slice(0, 2).map(d => d.id))+ expect(await store.messages(owner, f.tab, 'lead', {})).toEqual(live)+ await compactByHand([f.deliveries[2]!.id])+ expect(await store.messages(owner, f.tab, 'lead', {})).toEqual(live)+ expect(await store.messages(host, f.tab, 'lead', {}, f.lead)).toEqual(liveToLead)+ })++ it('keeps one order for a message\'s receipts whichever of them are compacted', async () => {+ const { tab, lead } = await fixture()+ const id = `${tab}-shared`, at = new Date()+ await db.superTabMessage.create({ data: { id, tabId: tab, conversationId: 'lead', sequence: 900, text: 'To both', replyTo: null, recipientIds: [lead, 'agent-b'],+ mentions: [], thread: id, author: { kind: 'person', id: owner.sub, name: owner.email, membershipGeneration: 'g' } } })+ const delivery = (agentId: string) => ({ id: `${id}-${agentId}`, tabId: tab, conversationId: 'lead', messageId: id, agentId, machineId: 'test-host',+ membershipGeneration: 'g', authorKind: 'person', authorId: owner.sub, authorGeneration: 'g', status: 'answered', createdAt: at })+ await db.superTabDelivery.createMany({ data: [delivery(lead), delivery('agent-b')] })+ const shown = async () => (await store.messages(owner, tab, 'lead', {})).messages.find(m => m.id === id)!.deliveries.map(d => d.agentId)+ const before = await shown()+ expect(before).toEqual([lead, 'agent-b'])+ await compactByHand([`${id}-${lead}`])+ expect(await shown()).toEqual(before)+ })++ it('a live row wins over a compacted one for the same agent', async () => {+ const f = await answered()+ const first = f.deliveries[0]!+ const live = await store.messages(owner, f.tab, 'lead', {})+ await compactByHand([first.id])+ // A row for the same message and agent again (a host that settled it while it was being moved): it is what the page shows.+ await db.superTabDelivery.create({ data: { ...first, status: 'confirmed', receipt: 'again' } })+ const page = await store.messages(owner, f.tab, 'lead', {})+ const row = page.messages.find(m => m.id === first.messageId)!+ expect(row.deliveries).toHaveLength(1)+ expect(row.deliveries[0]).toMatchObject({ id: first.id, status: 'confirmed', receipt: 'again' })+ expect(page.messages.filter(m => m.id !== first.messageId)).toEqual(live.messages.filter(m => m.id !== first.messageId))+ })++ it('a compacted receipt for news a row does not name (a Buddy\'s) is shown only to that agent, as a live one is', async () => {+ const { tab, lead } = await fixture()+ // The Lead's delivery of a row that does not list it among its recipients: the shape of Buddy news (buddyNews.ts).+ const id = `${tab}-news`+ await db.superTabMessage.create({ data: { id, tabId: tab, conversationId: 'lead', sequence: 900, text: 'News', replyTo: null, recipientIds: [],+ mentions: [], thread: id, author: { kind: 'person', id: owner.sub, name: owner.email, membershipGeneration: 'g' } } })+ await db.superTabDelivery.create({ data: { id: `${id}-d`, tabId: tab, conversationId: 'lead', messageId: id, agentId: lead, machineId: 'test-host',+ membershipGeneration: 'g', authorKind: 'person', authorId: owner.sub, authorGeneration: 'g', status: 'answered' } })+ const receipts = async (user: AuthUser, asAgentId?: string) =>+ (await store.messages(user, tab, 'lead', {}, asAgentId)).messages.find(m => m.id === id)!.deliveries.map(d => d.agentId)+ expect(await receipts(owner)).toEqual([])+ expect(await receipts(host, lead)).toEqual([lead])+ await compactByHand([`${id}-d`])+ expect(await receipts(owner)).toEqual([])+ expect(await receipts(host, lead)).toEqual([lead])+ })++ it('no message read returns a receipts map or a compacted receipt\'s host fields', async () => {+ const f = await answered()+ await compactByHand(f.deliveries.slice(0, 3).map(d => d.id))+ const replies = [await store.messages(owner, f.tab, 'lead', {}), await store.messages(host, f.tab, 'lead', {}, f.lead),+ await store.messages(owner, f.tab, 'lead', { messageId: f.deliveries[0]!.messageId }), await store.catchUp(owner, f.tab, 'lead', 0),+ await store.mentions(owner, f.tab, 'waiting')]+ for (const reply of replies) expect(JSON.stringify(reply)).not.toMatch(/"receipts"|"deliveryIds"|"deliveryId"|"machineOwnerId"|"claimId"/)+ })+})diff --git a/backend/integration/superTabThreads.test.ts b/backend/integration/superTabThreads.test.tsnew file mode 100644index 000000000..ef6872795--- /dev/null+++ b/backend/integration/superTabThreads.test.ts@@ -0,0 +1,191 @@+import { randomUUID } from 'node:crypto'+import { execFile } from 'node:child_process'+import { promisify } from 'node:util'+import { join } from 'node:path'+import { tmpdir } from 'node:os'+import { afterAll, beforeAll, describe, expect, it } from 'vitest'+import { MongoMemoryReplSet } from 'mongodb-memory-server-core'+import { PrismaClient, type Prisma } from '@prisma/client'+import type { AuthUser } from '../src/lib/ssoAuth.js'+import { SuperTabStore } from '../src/superTabs/store.js'+import { migrateSwarm } from '../src/superTabs/migration/migrate.js'+import { buddyChatId } from '../src/superTabs/buddies.js'+import { refreshThreads, threadFacet } from '../src/superTabs/pageReads.js'+import { backfillThreads } from '../src/superTabs/threadBackfill.js'+import type { SuperTabState } from '../src/superTabs/model.js'+import { writeLegacySwarm } from '../scripts/swarm-parts/legacy.js'+import { NOW, outsider as outsiderActor, owner as ownerActor, peer as peerActor, swarm } from '../src/superTabs/read/fixture.test-support.js'+import { profiledReads } from './profiledReads.test-support.js'+import { filterOf, plan, profiled, reads } from './plans.test-support.js'++// Spec 5 phase D (docs/superpowers/specs/2026-10-30-swarm-messages-delta-and-retention-design.md): a page reads its threads'+// summaries from SuperTabThread once its Swarm is backfilled, and every reader's threads stay what the $facet gave them.+// A fresh loopback replica set, created here. This suite never reads DATABASE_URL or an existing database.+let replica: MongoMemoryReplSet+let db: PrismaClient+let store: SuperTabStore+const user = (actor: { userId: string; email: string }): AuthUser => ({ sub: actor.userId, email: actor.email, autonomousEnv: 'prod', role: 'user' })+const [owner, peer, outsider] = [ownerActor, peerActor, outsiderActor].map(user) as [AuthUser, AuthUser, AuthUser]+const MIGRATED = NOW - 60_000++beforeAll(async () => {+ replica = await MongoMemoryReplSet.create({ binary: { version: '7.0.24', downloadDir: join(tmpdir(), 'harness-super-tab-mongodb-binaries') },+ replSet: { count: 1, ip: '127.0.0.1', storageEngine: 'wiredTiger' } })+ const uri = replica.getUri(`swarm_threads_test_${randomUUID().replaceAll('-', '_')}`)+ await promisify(execFile)(join(process.cwd(), 'node_modules/.bin/prisma'), ['db', 'push', '--skip-generate'], {+ env: { ...process.env, DATABASE_URL: uri }, timeout: 60000,+ })+ db = new PrismaClient({ datasources: { db: { url: uri } } })+ for (const each of [owner, peer, outsider]) await db.user.create({ data: { id: each.sub, externalId: `fixture-${each.sub}`, email: each.email, autonomousEnv: 'prod' } })+ // The owner's own machine runs the Swarm's agents, so reading as one of them needs no machine share.+ await db.machine.create({ data: { machineId: 'host', userId: owner.sub, apiKey: 'fixture-only', managerId: 'fixture', autonomousEnv: 'prod', authMode: 'remote' } })+ store = new SuperTabStore(db)+})+afterAll(async () => {+ await db?.$disconnect()+ await replica?.stop()+})++const person = (actor: AuthUser) => ({ kind: 'person', id: actor.sub, name: actor.email, membershipGeneration: 'gen' })+const agent = (id: string) => ({ kind: 'agent', id, name: id, membershipGeneration: 'gen' })+let sequence = 10, scenes = 0++/** A migrated Swarm: a Lead, a worker in Web, and peer's Buddy, in the project chat and in its own Buddy chat. Web holds a lone+ * root and threads with every kind of reply a reader may skip: a note to the Lead alone, a route row (no Buddy reads it), a+ * tag. Its root `w-b` is a note to the worker alone. The Buddy chat holds a thread with replies of a Buddy session. */+async function scene() {+ const s = swarm(MIGRATED)+ s.join(peerActor)+ s.join(outsiderActor)+ const lead = s.add('lead-harness', 'lead', true, ownerActor, 'host', owner.sub)+ const web = s.group('Web')+ const worker = s.add('worker-harness', web, false, ownerActor, 'host', owner.sub)+ const buddy = s.add('buddy-harness', 'lead', false, ownerActor, 'host', owner.sub)+ let buddyChat = ''+ s.edit(state => {+ const generation = state.people.find(p => p.id === peer.sub)!.generation+ const projectId = state.projects![0]!.id+ state.agents.find(a => a.id === buddy)!.buddyOf = { personId: peer.sub, generation }+ buddyChat = buddyChatId(projectId, { id: peer.sub, generation })+ state.conversations.push({ id: buddyChat, kind: 'buddy', projectId, title: 'Buddy', leadId: null, agentIds: [buddy], archived: false,+ brief: '', checks: [], assignment: null, lastMessageSequence: 0, audience: { people: [{ id: peer.sub, generation }] } })+ })+ const tabId = `tab-${randomUUID()}`+ await writeLegacySwarm(db, { state: { ...structuredClone(s.state), id: tabId } as SuperTabState, revision: 700 })+ const report = await migrateSwarm(db, tabId, { now: () => MIGRATED })+ if (!report.ok) throw new Error(report.error)+ const note = (to: string) => ({ source: { kind: 'note', recipientsOnly: true }, recipientIds: [to] })+ // Message ids are unique across Swarms: each scene's carry its own prefix.+ const prefix = `s${++scenes}-`, id = (name: string) => prefix + name+ const rows: Row[] = [+ { id: 'w-lone', conversationId: web, thread: 'w-lone', author: person(owner) },+ { id: 'w-a', conversationId: web, thread: 'w-a', author: person(owner) },+ { id: 'w-a1', conversationId: web, thread: 'w-a', author: agent(worker) },+ { id: 'w-a2', conversationId: web, thread: 'w-a', author: agent(worker), ...note(lead) },+ { id: 'w-a3', conversationId: web, thread: 'w-a', author: agent(lead), source: { kind: 'route' } },+ { id: 'w-a4', conversationId: web, thread: 'w-a', author: person(peer), mentions: [outsider.sub] },+ { id: 'w-a5', conversationId: web, thread: 'w-a', author: person(outsider) },+ { id: 'w-b', conversationId: web, thread: 'w-b', author: agent(lead), ...note(worker) },+ { id: 'w-b1', conversationId: web, thread: 'w-b', author: agent(worker) },+ { id: 'w-c', conversationId: web, thread: 'w-c', author: person(owner) },+ { id: 'w-c1', conversationId: web, thread: 'w-c', author: agent(lead), source: { kind: 'route' } },+ { id: 'b-root', conversationId: buddyChat, thread: 'b-root', author: person(peer) },+ { id: 'b-r1', conversationId: buddyChat, thread: 'b-root', author: agent(buddy) },+ { id: 'b-r2', conversationId: buddyChat, thread: 'b-root', author: person(peer), buddySession: true },+ { id: 'b-r3', conversationId: buddyChat, thread: 'b-root', author: agent(buddy), buddySession: true },+ ]+ await insert(tabId, prefix, rows)+ const readers: Array<[string, string, AuthUser, string?]> = [['owner', web, owner], ['peer', web, peer], ['outsider', web, outsider],+ ['Buddy', web, owner, buddy], ['worker', web, owner, worker], ['Lead', web, owner, lead], ['peer in the Buddy chat', buddyChat, peer],+ ['Buddy in its chat', buddyChat, owner, buddy]]+ return { tabId, web, buddyChat, lead, worker, buddy, readers, prefix, id }+}++interface Row { id: string; conversationId: string; thread: string; author: object; mentions?: string[]; source?: object; recipientIds?: string[]; buddySession?: boolean }+/** Rows written as an older release writes them, no summary touched; ids and threads named without the scene's prefix. */+async function insert(tabId: string, prefix: string, rows: Row[]) {+ await db.superTabMessage.createMany({ data: rows.map(row => ({ recipientIds: [], mentions: [], ...row, id: prefix + row.id, thread: prefix + row.thread,+ tabId, text: row.id, sequence: ++sequence, createdAt: new Date() })) })+}++/** Each reader's newest page of its conversation. */+const pages = async (s: Awaited<ReturnType<typeof scene>>) => Object.fromEntries(await Promise.all(s.readers.map(async ([name, conversationId, as, agentId]) =>+ [name, await store.messages(as, s.tabId, conversationId, {}, agentId)] as const)))+const replies = (page: { threads: Array<{ id: string; replies: number }> }) => Object.fromEntries(page.threads.map(t => [t.id.replace(/^s\d+-/, ''), t.replies]))+const sorted = (rows: unknown[]) => rows.map(row => JSON.stringify(row)).sort()++describe('thread summaries (spec 5 phase D)', () => {+ it('gives every reader the same page from the summaries as from the $facet, lone roots and hidden replies included', async () => {+ const s = await scene()+ const before = await pages(s)+ // Each reader skips its own kinds of reply: no summary counts a note to agents alone (a reader of `w-b`'s root still counts+ // its reply), and a Buddy no route row, nor the session's replies in its own chat.+ expect(Object.fromEntries(Object.entries(before).map(([name, page]) => [name, replies(page)]))).toEqual({+ owner: { 'w-a': 4, 'w-c': 1 }, peer: { 'w-a': 4, 'w-c': 1 }, outsider: { 'w-a': 4, 'w-c': 1 }, Buddy: { 'w-a': 3 },+ worker: { 'w-a': 4, 'w-b': 1, 'w-c': 1 }, Lead: { 'w-a': 4, 'w-c': 1 }, 'peer in the Buddy chat': { 'b-root': 3 }, 'Buddy in its chat': { 'b-root': 1 } })+ expect(await db.superTabThread.count({ where: { tabId: s.tabId } })).toBe(0)++ await backfillThreads(db, Date.now())+ // One summary per thread with a reply: none for the lone root.+ expect((await db.superTabThread.findMany({ where: { tabId: s.tabId }, select: { id: true } })).map(t => t.id).sort()).toEqual(['b-root', 'w-a', 'w-b', 'w-c'].map(s.id))+ expect(await pages(s)).toEqual(before)+ })++ it('reads through the $facet until its Swarm is backfilled, and the summaries by id after', async () => {+ const s = await scene()+ const read = () => profiledReads(db, /\.SuperTab(Message|Thread)$/, () => store.messages(owner, s.tabId, s.web, {}))+ const facet = await read()+ expect(facet.byCollection.SuperTabThread).toBeUndefined()+ await backfillThreads(db, Date.now())+ const summaries = await read()+ expect(summaries.value).toEqual(facet.value)+ expect(summaries.byCollection).toEqual({ SuperTabMessage: facet.byCollection.SuperTabMessage! - 1, SuperTabThread: 1 })+ })++ it('rewrites a thread\'s summary in the transaction that sends a reply, a lone root\'s first one too', async () => {+ const s = await scene()+ await backfillThreads(db, Date.now())+ for (const replyTo of ['w-a', 'w-lone'].map(s.id)) {+ await store.execute(owner, s.tabId, { operationId: randomUUID(), command: { type: 'message.send', conversationId: s.web, text: 'More', replyTo,+ recipientIds: [], mentions: [outsider.sub] } as never })+ const stored = await db.superTabThread.findUniqueOrThrow({ where: { id: replyTo } })+ const facet = await threadFacet(db, s.tabId, s.web, [replyTo])+ expect(sorted(stored.parts as unknown[])).toEqual(sorted(facet.parts))+ expect(sorted(stored.mentions as unknown[])).toEqual(sorted(facet.mentions))+ }+ expect(replies(await store.messages(owner, s.tabId, s.web, {}))).toEqual({ 'w-a': 5, 'w-c': 1, 'w-lone': 1 })+ })++ it('leaves no summary when the reply\'s transaction fails', async () => {+ const s = await scene()+ await expect(db.$transaction(async tx => {+ await tx.superTabMessage.create({ data: { id: s.id('w-lone-r'), tabId: s.tabId, conversationId: s.web, thread: s.id('w-lone'), sequence: 999, text: 'x',+ author: person(owner), recipientIds: [], mentions: [] } })+ await refreshThreads(tx, s.tabId, s.web, [s.id('w-lone')])+ expect(await tx.superTabThread.findUnique({ where: { id: s.id('w-lone') } })).not.toBeNull()+ throw new Error('rolled back')+ })).rejects.toThrow('rolled back')+ expect(await db.superTabThread.findUnique({ where: { id: s.id('w-lone') } })).toBeNull()+ })++ it('counts at the next backfill a reply an older release wrote after the marker', async () => {+ const s = await scene()+ await backfillThreads(db, Date.now())+ // An older release writes the reply as any command does (its conversation moves on), but leaves the summary as it was.+ const summary = async () => (await db.$runCommandRaw({ find: 'SuperTabThread', filter: { _id: s.id('w-a') } }) as unknown as+ { cursor: { firstBatch: Prisma.InputJsonObject[] } }).cursor.firstBatch[0]!+ const before = await summary()+ await store.execute(owner, s.tabId, { operationId: randomUUID(), command: { type: 'message.send', conversationId: s.web, text: 'Late', replyTo: s.id('w-a'),+ recipientIds: [] } as never })+ await db.$runCommandRaw({ update: 'SuperTabThread', updates: [{ q: { _id: s.id('w-a') }, u: before }] })+ expect(replies(await store.messages(owner, s.tabId, s.web, {}))['w-a']).toBe(4)+ const entries = await profiled(db, () => backfillThreads(db, Date.now()))+ expect(replies(await store.messages(owner, s.tabId, s.web, {}))['w-a']).toBe(5)+ // Only the changed conversation's newest messages are read again, never every message of each Swarm.+ expect(plan(reads(entries, 'SuperTabMessage', ['tabId'], ['conversationId']))).toEqual([])+ const walked = reads(entries, 'SuperTabMessage', ['conversationId'], ['thread']).filter(entry => JSON.stringify(filterOf(entry)).includes(s.tabId))+ expect(walked.reduce((sum, entry) => sum + entry.docsExamined, 0), JSON.stringify(plan(walked))).toBeLessThanOrEqual(12)+ const stored = await db.superTabThread.findUniqueOrThrow({ where: { id: s.id('w-a') } })+ expect(sorted(stored.parts as unknown[])).toEqual(sorted((await threadFacet(db, s.tabId, s.web, [s.id('w-a')])).parts))+ })+})diff --git a/backend/prisma/schema.prisma b/backend/prisma/schema.prismaindex 33d9f3840..2c1d2e879 100644--- a/backend/prisma/schema.prisma+++ b/backend/prisma/schema.prisma@@ -1185,6 +1185,29 @@ model SuperTabThreadMark {@@index([updatedAt])}+/// One thread with at least one reply, summed up as pageThreads' $facet gives it (pageReads.ts refreshThreads): rewritten in the+/// transaction that inserts each reply, and read by root id instead of the $facet once its Swarm is backfilled+/// (SuperTabThreadBackfill). No text; never returned as is. Kept with its messages.+model SuperTabThread {+ /// The root message's id.+ id String @id @map("_id")+ tabId String+ conversationId String+ /// The $facet's `parts` and `mentions` for this thread (pageReads.ts ThreadFacet).+ parts Json+ mentions Json+ updatedAt DateTime++ @@index([tabId, conversationId])+}++/// A Swarm whose thread summaries hold every reply created before `through` (threadBackfill.ts); its pages read them. Written+/// by the singleton worker alone; id = the Swarm's id.+model SuperTabThreadBackfill {+ id String @id @map("_id")+ through DateTime+}+/// Where a Swarm sweep reads on from between passes and releases, one row per sweep (readMarks.ts sweepThreadMarks: id/// "thread-marks", the last Swarm whose people check finished). Written by the singleton worker alone.model SuperTabSweepCursor {@@ -1321,6 +1344,23 @@ model SuperTabDelivery {/// An ended task's open deliveries, cancelled with it (store.ts releaseRuns and stopped): by [tabId, status] alone the/// planner sometimes walked every delivery of the Swarm (180 read for 1 in the benchmark).@@index([tabId, assignmentId])+ /// Finished deliveries by their last change: the hourly compaction reads only those past 14 days (deliveryCompaction.ts).+ @@index([status, updatedAt])+}++// One message's finished deliveries, moved here 14 days after their last change (superTabs/deliveryCompaction.ts): message+// history and its receipts are kept forever, while delivery rows hold only the recent and open ones. Never on the message+// row, never returned whole: a page reads each receipt as a delivery and filters it per reader (pageReads.ts pageDeliveries).+model SuperTabReceipt {+ id String @id @map("_id") // the message id+ tabId String+ conversationId String+ /// By agent id: the delivery row as it was, but for the fields above (pageReads.ts receiptFields).+ receipts Json+ /// The compacted deliveries' ids, so a host's receipt read finds one by its id (pageReads.ts compactedDelivery).+ deliveryIds String[]++ @@index([tabId, deliveryIds])}// One independent harness owns at most one active assignment across all Super Tabs.diff --git a/backend/src/config/env.ts b/backend/src/config/env.tsindex b74888c2b..874bf30db 100644--- a/backend/src/config/env.ts+++ b/backend/src/config/env.ts@@ -41,6 +41,10 @@ const envSchema = z.object({SWARM_READ_CACHE: z.enum(['on', 'off']).default('on'),// Its size per process, counted as the JSON size of what it holds.SWARM_CACHE_MB: z.string().default('64').transform(Number),+ // The worker's hourly move of finished deliveries older than 14 days into their messages' receipts+ // (superTabs/deliveryCompaction.ts). 'off' is the kill switch: set it before rolling back to a release whose pages do not+ // read compacted receipts.+ SWARM_DELIVERY_COMPACTION: z.enum(['on', 'off']).default('on'),// This backend instance's stable id (used to route manager-command replies via Redis in a// multi-instance deploy). Defaults to the pm2 instance index or a per-process fallback.diff --git a/backend/src/server.ts b/backend/src/server.tsindex 5b4736627..e0c6b2bc3 100644--- a/backend/src/server.ts+++ b/backend/src/server.ts@@ -25,6 +25,7 @@ import { prisma } from './lib/prisma.js'import { backendReadinessChecks, createReadiness, drainBeforeClosing, processDrain } from './lib/readiness.js'import { configureSwarmStamps } from './superTabs/cache.js'import { watchGroupIndexes } from './superTabs/parts/indexed.js'+import { startMessageReadStats } from './superTabs/messageReadStats.js'import { useSharedSsoProfileStore } from './lib/ssoAuth.js'import { useSharedRateCounter } from './lib/rateLimit.js'import { createRouteRateLimit } from './middlewares/routeRateLimit.js'@@ -147,6 +148,8 @@ async function start(): Promise<void> {// Every minute, how often Swarm transactions ran again and commands loaded twice (superTabs/retryStats.ts, DB 2.1).const stopRetryStats = startRetryStats()const stopRouteLatency = startRouteLatency(registerRouteLatency(app))+ // Every minute: who reads messages how, with the rows each page decoded and showed (superTabs/messageReadStats.ts).+ const stopMessageReadStats = startMessageReadStats()app.setErrorHandler(errorHandler)@@ -222,11 +225,12 @@ async function start(): Promise<void> {await Promise.race([fastifyClosed, new Promise((resolve) => setTimeout(resolve, 5_000))])// The sockets have closed, so no connect is left to count. Bounded: a slow database must not hold the exit.await Promise.race([stopRelayUsage().catch(() => {}), new Promise((resolve) => setTimeout(resolve, 3_000))])- // The last partial minute of both logs, once the drain and the usage flush have run their queries.+ // The last partial minute of these logs, once the drain and the usage flush have run their queries.stopQueryStats()stopRetryStats()stopRouteLatency()stopMachineReportLog()+ stopMessageReadStats()// The database stays connected through the drain, for the requests still being served (lib/prisma.ts).await Promise.allSettled([prisma.$disconnect(), closeBus()])process.exit(exitCode)diff --git a/backend/src/superTabs/deliveryCompaction.ts b/backend/src/superTabs/deliveryCompaction.tsnew file mode 100644index 000000000..721d2c66c--- /dev/null+++ b/backend/src/superTabs/deliveryCompaction.ts@@ -0,0 +1,93 @@+import type { Prisma, PrismaClient, SuperTabDelivery } from '@prisma/client'+import { logger } from '../utils/logger.js'+import { readBy, retryStaleIndex } from './parts/indexed.js'+import { receiptFields } from './pageReads.js'+import { prismaRow } from './rawRows.js'+import { retryTransient } from './retry.js'++/**+ * Message history is kept forever with its receipts, while delivery rows stop growing (spec 5 phase C): a finished delivery+ * whose last change is 14 days old moves into its message's SuperTabReceipt, which page reads, hosts and notices read as the+ * row (pageReads.ts). `delivered` and `confirmed` are not finished (a reply or a project report moves them to `answered`),+ * nor are the open ones.+ */+export const FINAL_DELIVERY = ['answered', 'cancelled', 'superseded'] as const+export const COMPACT_AFTER_MS = 14 * 86_400_000+/** Batches per hourly run: up to 10 k deliveries, so the first runs also clear the backlog. Prod finishes ≤ 3.4 k a day. */+const BATCHES_PER_RUN = 20+/** Deliveries a reply or a project report may still answer: never compacted, counted so their growth shows in the log. */+const UNFINISHED_DELIVERY = ['delivered', 'confirmed'] as const++const old = (statuses: readonly string[], cutoff: Date) => ({ status: { $in: [...statuses] }, updatedAt: { $lt: { $date: cutoff.toISOString() } } })+/** A receipt is stored under its agent's id as a field name: never one Mongo would read as a path or an operator. Ids are+ * hex (model.ts stableId); another is left as a row, and never read again ahead of the rest. */+const compactable = (cutoff: Date) => ({ ...old(FINAL_DELIVERY, cutoff), agentId: { $regex: '^[A-Za-z0-9_-]+$' } })++/** One batch, in one transaction: the oldest finished deliveries past the cutoff, written into their messages' receipts by+ * field path (two compactions of one message keep each other's agents) and deleted. A delivery changed since it was read+ * is not deleted: the transaction fails and nothing of it is written. */+async function compactBatch(db: PrismaClient, cutoff: Date, batch: number): Promise<number> {+ return retryTransient(() => retryStaleIndex(() => db.$transaction(async tx => {+ const moving = (await readBy(tx, 'SuperTabDelivery', 'compaction', [{ $match: compactable(cutoff) }, { $sort: { updatedAt: 1 } }, { $limit: batch }]))+ .map(row => prismaRow<SuperTabDelivery>('SuperTabDelivery', row))+ if (!moving.length) return 0+ const byMessage = new Map<string, SuperTabDelivery[]>()+ for (const row of moving) byMessage.set(row.messageId, [...byMessage.get(row.messageId) ?? [], row])+ await tx.$runCommandRaw({ update: 'SuperTabReceipt', updates: [...byMessage.values()].map(deliveries => ({ q: { _id: deliveries[0]!.messageId }, upsert: true,+ u: { $set: { tabId: deliveries[0]!.tabId, conversationId: deliveries[0]!.conversationId,+ ...Object.fromEntries(deliveries.map(row => [`receipts.${row.agentId}`, receiptFields(row)])) },+ $addToSet: { deliveryIds: { $each: deliveries.map(row => row.id) } } } })) } as Prisma.InputJsonObject)+ const deleted = await tx.$runCommandRaw({ delete: 'SuperTabDelivery', deletes: moving.map(row => ({ limit: 1,+ q: { _id: row.id, status: row.status, updatedAt: { $date: row.updatedAt.toISOString() } } })) } as Prisma.InputJsonObject) as unknown as { n: number }+ if (deleted.n !== moving.length) throw new Error(`deliveries changed while compacting: ${moving.length - deleted.n}`)+ return moving.length+ }, TRANSACTION_OPTIONS)))+}++/** As the store's transactions (store.ts): a batch of 500 writes outlasts Prisma's 5 s default under load. */+const TRANSACTION_OPTIONS = { maxWait: 5_000, timeout: 15_000 }++/** A compacted delivery made a row again, and taken out of its message's receipts, in `tx`: a host that held it typed it+ * after all (store.ts deliver). */+export async function restoreDelivery(tx: Prisma.TransactionClient, row: SuperTabDelivery): Promise<void> {+ await tx.superTabDelivery.create({ data: row })+ await tx.$runCommandRaw({ update: 'SuperTabReceipt', updates: [{ q: { _id: row.messageId, tabId: row.tabId },+ u: { $unset: { [`receipts.${row.agentId}`]: '' }, $pull: { deliveryIds: row.id } } }] })+}++/** Compacts finished deliveries older than 14 days at `now`, `batch` at a time, until none is left or 20 batches ran.+ * `remaining`: those still old enough afterwards, the backlog the log warns on. `unfinished`: delivered or confirmed ones as+ * old, which stay rows until a reply or a report answers them. */+export async function compactDeliveries(db: PrismaClient, now: number, batch = 500): Promise<{ compacted: number; remaining: number; unfinished: number }> {+ const cutoff = new Date(now - COMPACT_AFTER_MS)+ let compacted = 0+ for (let runs = 0; runs < BATCHES_PER_RUN; runs++) {+ const moved = await compactBatch(db, cutoff, batch)+ compacted += moved+ if (moved < batch) break+ }+ const count = async (statuses: readonly string[]) => Number((await readBy(db as unknown as Prisma.TransactionClient, 'SuperTabDelivery', 'compaction',+ [{ $match: old(statuses, cutoff) }, { $count: 'n' }]))[0]?.n ?? 0)+ return { compacted, remaining: await count(FINAL_DELIVERY), unfinished: await count(UNFINISHED_DELIVERY) }+}++/** Runs the compaction now and then every `everyMs`, one run at a time, unless `setting` (env SWARM_DELIVERY_COMPACTION) is+ * the kill switch. A failed run is logged and tried again next time. */+export function startDeliveryCompaction(db: PrismaClient, setting: 'on' | 'off', everyMs = 3_600_000): () => void {+ if (setting === 'off') {+ logger.info('[swarm-deliveries] compaction is off (SWARM_DELIVERY_COMPACTION=off)')+ return () => {}+ }+ let running = false+ const run = () => {+ if (running) return+ running = true+ void compactDeliveries(db, Date.now()).then(counts => logger.info('[swarm-deliveries] compacted', counts))+ .catch(error => logger.warn('[swarm-deliveries] compaction failed', { error: error instanceof Error ? error.message : String(error) }))+ .finally(() => { running = false })+ }+ run()+ const timer = setInterval(run, everyMs)+ timer.unref()+ return () => clearInterval(timer)+}diff --git a/backend/src/superTabs/messageReadStats.test.ts b/backend/src/superTabs/messageReadStats.test.tsnew file mode 100644index 000000000..cd2d7e447--- /dev/null+++ b/backend/src/superTabs/messageReadStats.test.ts@@ -0,0 +1,50 @@+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'+import { recordMessageRead, startMessageReadStats } from './messageReadStats.js'++describe('messages by reader', () => {+ let stop = () => {}+ beforeEach(() => { vi.useFakeTimers() })+ afterEach(() => {+ stop()+ vi.useRealTimers()+ })++ it('counts reads per reader and cursor, with batches and rows, and names nothing', () => {+ const logged: Array<[string, Record<string, unknown>]> = []+ stop = startMessageReadStats({ everyMs: 60_000, log: (message, context) => logged.push([message, context]) })+ recordMessageRead({ reader: 'person', cursor: 'before', session: 'sso' }, 2, 202, 100)+ recordMessageRead({ reader: 'person', cursor: 'before', session: 'sso' }, 2, 202, 100)+ recordMessageRead({ reader: 'agent', cursor: 'latest', session: 'computer' }, 1, 101, 100)+ vi.advanceTimersByTime(60_000)+ expect(logged).toHaveLength(1)+ const [message, context] = logged[0]!+ expect(message).toBe('messages by reader')+ expect(context.windowMs).toBe(60_000)+ expect(context.rows).toEqual([+ { reader: 'person', cursor: 'before', session: 'sso', n: 2, batches: 4, decoded: 404, shown: 200 },+ { reader: 'agent', cursor: 'latest', session: 'computer', n: 1, batches: 1, decoded: 101, shown: 100 },+ ])+ expect(JSON.stringify(context)).not.toMatch(/[0-9a-f]{12,}|@/i)+ })++ it('logs nothing for a quiet minute, and starts each window empty', () => {+ const logged: Array<Record<string, unknown>> = []+ stop = startMessageReadStats({ everyMs: 60_000, log: (_message, context) => logged.push(context) })+ recordMessageRead({ reader: 'buddy', cursor: 'after', session: 'sso' }, 1, 0, 0)+ vi.advanceTimersByTime(180_000)+ expect(logged).toEqual([{ windowMs: 60_000, rows: [{ reader: 'buddy', cursor: 'after', session: 'sso', n: 1, batches: 1, decoded: 0, shown: 0 }] }])+ })++ it('keeps nothing while stopped, and logs what is left when it stops', () => {+ recordMessageRead({ reader: 'person', cursor: 'latest', session: 'sso' }, 1, 1, 1)+ const logged: Array<Record<string, unknown>> = []+ stop = startMessageReadStats({ everyMs: 60_000, log: (_message, context) => logged.push(context) })+ stop()+ expect(logged).toEqual([])+ stop = startMessageReadStats({ everyMs: 60_000, log: (_message, context) => logged.push(context) })+ recordMessageRead({ reader: 'person', cursor: 'catchUp', session: 'team' }, 1, 3, 2)+ vi.advanceTimersByTime(10_000)+ stop()+ expect(logged).toEqual([{ windowMs: 10_000, rows: [{ reader: 'person', cursor: 'catchUp', session: 'team', n: 1, batches: 1, decoded: 3, shown: 2 }] }])+ })+})diff --git a/backend/src/superTabs/messageReadStats.ts b/backend/src/superTabs/messageReadStats.tsnew file mode 100644index 000000000..cba8a256f--- /dev/null+++ b/backend/src/superTabs/messageReadStats.ts@@ -0,0 +1,51 @@+import { logger } from '../utils/logger.js'++/**+ * Who reads messages and how (spec 5 A1): every minute, `messages by reader` logs each reader kind, cursor kind and sign-in+ * session kind with its request count, the batches it read, the rows those batches decoded and the rows the pages showed.+ * `db by shape` keeps only each minute's top shapes and says nothing of the caller. No id or text is kept.+ */++export interface MessageReadKey {+ reader: 'person' | 'agent' | 'buddy'+ /** `before` alone is Desktop's re-read of the page it shows; `beforeId` is scrolling back. */+ cursor: 'latest' | 'after' | 'before' | 'beforeId' | 'thread' | 'messageId' | 'catchUp'+ /** The Harness session the request signed in with (AuthUser.harnessSessionKind), `sso` for an Autonomous token. */+ session: 'viewer' | 'computer' | 'team' | 'sso'+}++interface Counts { n: number; batches: number; decoded: number; shown: number }++let window: Map<string, MessageReadKey & Counts> | undefined++export function recordMessageRead(key: MessageReadKey, batches: number, decoded: number, shown: number): void {+ if (!window) return+ const id = `${key.reader}|${key.cursor}|${key.session}`+ const row = window.get(id) ?? { reader: key.reader, cursor: key.cursor, session: key.session, n: 0, batches: 0, decoded: 0, shown: 0 }+ row.n+++ row.batches += batches+ row.decoded += decoded+ row.shown += shown+ window.set(id, row)+}++/** Logs each window's reads, every `everyMs`, until stopped; stopping logs what is left. */+export function startMessageReadStats({ everyMs = 60_000, log = (m, c) => logger.info(m, c) }: {+ everyMs?: number; log?: (message: string, context: Record<string, unknown>) => void } = {}): () => void {+ const counts = new Map<string, MessageReadKey & Counts>()+ window = counts+ let since = Date.now()+ const flush = () => {+ const now = Date.now()+ if (counts.size) log('messages by reader', { windowMs: now - since, rows: [...counts.values()] })+ counts.clear()+ since = now+ }+ const timer = setInterval(flush, everyMs)+ timer.unref()+ return () => {+ clearInterval(timer)+ if (window === counts) window = undefined+ flush()+ }+}diff --git a/backend/src/superTabs/pageReads.ts b/backend/src/superTabs/pageReads.tsindex 54d458100..02a303c90 100644--- a/backend/src/superTabs/pageReads.ts+++ b/backend/src/superTabs/pageReads.ts@@ -21,17 +21,67 @@ export async function pageReactions(tx: Transaction, tabId: string, messageIds:export type PageDelivery = Pick<SuperTabDelivery, 'id' | 'messageId' | 'agentId' | 'status' | 'receipt' | 'holdReason' | 'holdSince'| 'claimFailingSince' | 'claimFailures' | 'deliveredAt' | 'answeredAt' | 'createdAt' | 'updatedAt'>-/** The deliveries of one page of messages: who each message went to and how far it got. As Prisma read them: dates as- * Dates, and an optional field an older record lacks as null. */+const date = (value: unknown) => value == null ? null : value instanceof Date ? value : new Date(ms(value))++/** The deliveries of one page of messages: who each message went to and how far it got, from the live rows and the+ * compacted receipts (deliveryCompaction.ts), a live row winning for the same agent. As Prisma read them: dates as Dates,+ * and an optional field an older record lacks as null. A compacted receipt gives only these fields, never its host's. */export async function pageDeliveries(tx: Transaction, tabId: string, messageIds: string[]): Promise<PageDelivery[]> {const rows = await tx.superTabDelivery.aggregateRaw({ pipeline: [{ $match: { tabId, messageId: { $in: messageIds } } },{ $project: { messageId: 1, agentId: 1, status: 1, receipt: 1, holdReason: 1, holdSince: 1, claimFailingSince: 1, claimFailures: 1, deliveredAt: 1, answeredAt: 1, createdAt: 1, updatedAt: 1 } }] }) asunknown as Array<Record<string, unknown>>- const date = (value: unknown) => value == null ? null : new Date(ms(value))- return rows.map(row => ({ id: row._id as string, messageId: row.messageId as string, agentId: row.agentId as string, status: row.status as string,- receipt: (row.receipt ?? null) as string | null, holdReason: (row.holdReason ?? null) as string | null, holdSince: date(row.holdSince),- claimFailingSince: date(row.claimFailingSince), claimFailures: (row.claimFailures ?? null) as number | null,- deliveredAt: date(row.deliveredAt), answeredAt: date(row.answeredAt), createdAt: date(row.createdAt)!, updatedAt: date(row.updatedAt)! }))+ // After the rows, never beside them: a delivery compacted between the two reads is then in both, never in neither.+ const compacted = await compactedReceipts(tx, tabId, messageIds)+ const live = rows.map(row => pageDelivery({ ...row, id: row._id }))+ const held = new Set(live.map(d => `${d.messageId}\0${d.agentId}`))+ return [...live, ...compacted.filter(d => !held.has(`${d.messageId}\0${d.agentId}`)).map(pageDelivery)]+}++const pageDelivery = (row: Record<string, unknown>): PageDelivery => ({ id: row.id as string, messageId: row.messageId as string, agentId: row.agentId as string,+ status: row.status as string, receipt: (row.receipt ?? null) as string | null, holdReason: (row.holdReason ?? null) as string | null, holdSince: date(row.holdSince),+ claimFailingSince: date(row.claimFailingSince), claimFailures: (row.claimFailures ?? null) as number | null,+ deliveredAt: date(row.deliveredAt), answeredAt: date(row.answeredAt), createdAt: date(row.createdAt)!, updatedAt: date(row.updatedAt)! })++/** The fields a receipt document keeps of each of its deliveries, rather than per receipt. */+const RECEIPT_DOCUMENT = new Set(['id', 'tabId', 'conversationId', 'messageId', 'agentId'])++/** A delivery row as its compacted receipt stores it in its message's SuperTabReceipt (deliveryCompaction.ts), for a raw `$set`+ * (dates as extended JSON): the row as it was, under its agent's id and with its own as `deliveryId`, without what the+ * document holds once (its message, Swarm and conversation). The whole row is kept: a host that still holds the delivery is+ * answered from it (store.ts deliveryReceipt, deliver), as from the row. */+export function receiptFields(row: SuperTabDelivery): Prisma.InputJsonObject {+ const stored: Record<string, Prisma.InputJsonValue | null> = { deliveryId: row.id }+ for (const [key, value] of Object.entries(row)) {+ if (!RECEIPT_DOCUMENT.has(key)) stored[key] = value instanceof Date ? { $date: value.toISOString() } : value as Prisma.InputJsonValue | null+ }+ return stored as Prisma.InputJsonObject+}++/** A receipt document as raw reads return it. */+interface ReceiptDocument { _id: string; tabId: string; conversationId: string; receipts?: Record<string, Record<string, unknown>> }++/** One compacted receipt as the delivery row it was: dates as Dates, a field the row did not have as null. */+function compactedRow(document: ReceiptDocument, agentId: string, stored: Record<string, unknown>): SuperTabDelivery {+ const row: Record<string, unknown> = {}+ for (const [key, value] of Object.entries(stored)) {+ row[key] = value && typeof value === 'object' && '$date' in value ? date(value) : value+ }+ const { deliveryId, ...fields } = row+ return { ...fields, id: deliveryId as string, tabId: document.tabId, conversationId: document.conversationId, messageId: document._id, agentId } as SuperTabDelivery+}++/** The compacted receipts of these messages, one per agent. By the primary key; the Swarm checked in the match. */+async function compactedReceipts(tx: Transaction, tabId: string, messageIds: string[]): Promise<SuperTabDelivery[]> {+ const documents = await readBy(tx, 'SuperTabReceipt', 'id', [{ $match: { _id: { $in: messageIds }, tabId } }]) as unknown as ReceiptDocument[]+ return documents.flatMap(document => Object.entries(document.receipts ?? {}).map(([agentId, stored]) => compactedRow(document, agentId, stored)))+}++/** A delivery of this agent that was compacted, as the row it was, found by its id (index [tabId, deliveryIds]); null when+ * there is none. A host back after weeks offline still settles what its journal holds (store.ts deliveryReceipt, deliver). */+export async function compactedDelivery(tx: Transaction, tabId: string, agentId: string, deliveryId: string): Promise<SuperTabDelivery | null> {+ const [document] = await readBy(tx, 'SuperTabReceipt', 'delivery', [{ $match: { tabId, deliveryIds: deliveryId } }, { $limit: 1 }]) as unknown as ReceiptDocument[]+ const stored = document?.receipts?.[agentId]+ return document && stored?.deliveryId === deliveryId ? compactedRow(document, agentId, stored) : null}/** The deliveries a host is still told about (the rest are answered, cancelled or confirmed). */@@ -143,22 +193,26 @@ export interface ThreadReply { thread: string; id: string; author: Author; mentiinterface ThreadPart { thread: string; root: boolean; kind?: unknown; only?: unknown; to: boolean; session?: boolean | null }/** One row of a part, by its place in the thread. */interface ThreadPlace { s: number; i: string; at: unknown; author: Author; slot?: number }+/** What the database sums up of some threads' rows (threadFacet), as it returns it: the same whoever reads. */+export interface ThreadFacet {+ parts: Array<{ _id: ThreadPart; count: number; oldest: ThreadPlace; newest: ThreadPlace; root?: { source?: unknown; recipientIds?: string[] } | null }>+ mentions: Array<{ _id: ThreadPart & { person: string }; first: ThreadPlace }>+}/**- * The threads rooted at `roots` in one conversation, roots included, oldest first, as a few rows that stand for all of them:- * what the page's thread summaries are counted from (threads.ts). The database groups the rows of each part (ThreadPart) and- * returns its oldest and newest row, how many it has (`count` on the newest), and where each person it tags is first tagged.+ * The threads rooted at `roots` in one conversation, roots included, as the database sums them up: it groups the rows of each+ * part (ThreadPart) and returns its oldest and newest row, how many it has, and where each person it tags is first tagged.* That is every place a summary reads: first appearances, the newest reply of each author, the count and the last reply. So* a poll that brings one reply of a 500-reply thread gets a few rows back, not 500, while the summaries stay what every* row gave (integration/superTabIndexes.test.ts). Matched as the page reads are, so the thread index bounds it; the database* still examines each of the threads' rows.*/-export async function pageThreads(tx: Transaction, tabId: string, conversationId: string, roots: string[]): Promise<ThreadReply[]> {+export async function threadFacet(tx: Transaction, tabId: string, conversationId: string, roots: string[]): Promise<ThreadFacet> {const root = { $eq: ['$_id', '$thread'] }const part = { thread: '$thread', root, kind: '$source.kind', only: '$source.recipientsOnly',to: { $gt: [{ $size: { $ifNull: ['$recipientIds', []] } }, 0] }, author: { kind: '$author.kind', id: '$author.id' },session: { $eq: ['$buddySession', true] } }- // (sequence, id) lead each place, so the database's min and max order rows as the sort below does (ids are ASCII).+ // (sequence, id) lead each place, so the database's min and max order rows as threadReplies sorts them (ids are ASCII).const place = { s: '$sequence', i: '$_id', at: '$createdAt', author: '$author' }// No $sort: it makes the planner walk the conversation's sequence index instead.const [found] = await readBy(tx, 'SuperTabMessage', 'thread', [{ $match: { tabId, conversationId, thread: { $in: roots } } },@@ -168,11 +222,13 @@ export async function pageThreads(tx: Transaction, tabId: string, conversationIdroot: { $first: { $cond: [root, { source: '$source', recipientIds: '$recipientIds' }, '$$REMOVE'] } } } }],mentions: [{ $unwind: { path: '$mentions', includeArrayIndex: 'slot' } },{ $group: { _id: { ...part, person: '$mentions' }, first: { $min: { s: '$sequence', i: '$_id', slot: { $toInt: '$slot' }, at: '$createdAt', author: '$author' } } } }],- } }]) as unknown as Array<{- parts: Array<{ _id: ThreadPart; count: number; oldest: ThreadPlace; newest: ThreadPlace; root?: { source?: unknown; recipientIds?: string[] } | null }>- mentions: Array<{ _id: ThreadPart & { person: string }; first: ThreadPlace }>- }>- const parts = found?.parts ?? [], mentions = found?.mentions ?? []+ } }]) as unknown as Array<Partial<ThreadFacet>>+ return { parts: found?.parts ?? [], mentions: found?.mentions ?? [] }+}++/** The rows a page's thread summaries are counted from (threads.ts), oldest first, from what threadFacet gave: a few rows that+ * stand for every row of each thread (`count` on the newest of each part), with what each reader skips marked. */+export async function threadReplies(tx: Transaction, tabId: string, { parts, mentions }: ThreadFacet): Promise<ThreadReply[]> {const copies = await deliveredCopies(tx, tabId, parts.flatMap(p => p._id.root && p.root? [{ id: p._id.thread, source: p.root.source, recipientIds: p.root.recipientIds ?? [] }] : []))// agentOnly reads only whether a row has recipients, not who they are.@@ -193,3 +249,64 @@ export async function pageThreads(tx: Transaction, tabId: string, conversationIdreturn [...rows.values()].map(({ slots, ...reply }) => ({ ...reply, mentions: slots.sort((a, b) => a[0] - b[0]).map(([, person]) => person) })).sort((a, b) => a.sequence - b.sequence || (a.id < b.id ? -1 : 1))}++/**+ * The rows the threads rooted at `roots` are summed up from (threadReplies). A Swarm whose summaries are backfilled+ * (threadBackfill.ts) reads them by root id from SuperTabThread, one small document per thread with a reply; any other runs+ * threadFacet over the threads' rows. A root with no summary has no reply, and threads.ts leaves such a thread out, so it is+ * left out here too.+ */+export async function pageThreads(tx: Transaction, tabId: string, conversationId: string, roots: string[]): Promise<ThreadReply[]> {+ if (!await threadsBackfilled(tx, tabId)) return threadReplies(tx, tabId, await threadFacet(tx, tabId, conversationId, roots))+ const stored = await readBy(tx, 'SuperTabThread', 'id', [{ $match: { _id: { $in: roots }, tabId, conversationId } },+ { $project: { parts: 1, mentions: 1 } }]) as unknown as ThreadFacet[]+ return threadReplies(tx, tabId, { parts: stored.flatMap(t => t.parts), mentions: stored.flatMap(t => t.mentions) })+}++/**+ * Rewrites the summaries of the threads rooted at `roots` in one conversation from their rows now: one document per thread+ * with a reply, as threadFacet gives it. Run in the transaction that inserts a reply (refreshReplies), so a summary holds what+ * its rows do: two replies at once write the same document, and one of their transactions runs again.+ */+export async function refreshThreads(tx: Transaction, tabId: string, conversationId: string, roots: string[]): Promise<void> {+ if (!roots.length) return+ const { parts, mentions } = await threadFacet(tx, tabId, conversationId, roots)+ const updatedAt = { $date: new Date().toISOString() }+ const updates = roots.flatMap(root => {+ const own = parts.filter(p => p._id.thread === root)+ if (!own.some(p => !p._id.root)) return []+ return [{ q: { _id: root }, u: { $set: { tabId, conversationId, parts: own, mentions: mentions.filter(m => m._id.thread === root), updatedAt } },+ upsert: true }]+ })+ // Raw, so the stored parts read back as the $facet returns them, dates as dates.+ if (updates.length) await tx.$runCommandRaw({ update: 'SuperTabThread', updates: updates as unknown as Prisma.InputJsonArray })+}++/** Refreshes the summaries of the threads that `rows`, just inserted, reply in (refreshThreads). A row that is its own+ * thread's root, or has none (written before threads), changes no summary. */+export async function refreshReplies(tx: Transaction, tabId: string,+ rows: ReadonlyArray<{ id: string; conversationId: string; thread?: string | null }>): Promise<void> {+ const roots = new Map<string, Set<string>>()+ for (const row of rows) {+ if (row.thread && row.thread !== row.id) roots.set(row.conversationId, (roots.get(row.conversationId) ?? new Set()).add(row.thread))+ }+ for (const [conversationId, threads] of roots) await refreshThreads(tx, tabId, conversationId, [...threads])+}++/** How long this process trusts what it last read of a Swarm's backfill marker. */+const BACKFILL_CHECK_MS = 60_000+const backfilled = new Map<string, { done: boolean; at: number }>()++/** Whether the Swarm's thread summaries are backfilled (threadBackfill.ts), as this process read it in the last minute. */+async function threadsBackfilled(tx: Transaction, tabId: string): Promise<boolean> {+ const known = backfilled.get(tabId), now = Date.now()+ if (known && now - known.at < BACKFILL_CHECK_MS) return known.done+ const done = !!await tx.superTabThreadBackfill.findUnique({ where: { id: tabId }, select: { id: true } })+ backfilled.set(tabId, { done, at: now })+ return done+}++/** Notes that this process backfilled the Swarm's summaries, so its own reads use them at once. */+export function noteBackfilled(tabId: string): void {+ backfilled.set(tabId, { done: true, at: Date.now() })+}diff --git a/backend/src/superTabs/parts/indexed.ts b/backend/src/superTabs/parts/indexed.tsindex 671df9c7b..5b510ca80 100644--- a/backend/src/superTabs/parts/indexed.ts+++ b/backend/src/superTabs/parts/indexed.ts@@ -64,6 +64,13 @@ export const NAMED_INDEX = {SuperTabDelivery: {/** An agent's deliveries by status and age: its open ones, oldest first. */agentOpen: { tabId: 1, agentId: 1, status: 1, createdAt: 1 },+ /** Finished deliveries by their last change: those old enough to compact (deliveryCompaction.ts). */+ compaction: { status: 1, updatedAt: 1 },+ },+ /** Compacted receipts (pageReads.ts): a page's by message id, and a host's by the compacted delivery's id. */+ SuperTabReceipt: {+ id: { _id: 1 },+ delivery: { tabId: 1, deliveryIds: 1 },},/** A person's thread marks (readMarks.ts): `person`, every one or those moved since a cursor, the cap's count and oldest,* and the sweep's people per Swarm; `age`, the sweep's marks untouched for 180 days. */@@ -71,6 +78,8 @@ export const NAMED_INDEX = {person: { tabId: 1, personId: 1, updatedAt: 1 },age: { updatedAt: 1 },},+ /** A page's thread summaries, by root id (pageReads.ts pageThreads). */+ SuperTabThread: { id: { _id: 1 } },} as constexport type NamedCollection = keyof typeof NAMED_INDEXexport type NamedIndex<C extends NamedCollection> = keyof typeof NAMED_INDEX[C] & stringdiff --git a/backend/src/superTabs/store.ts b/backend/src/superTabs/store.tsindex aa857f8d3..1d8d2cdd3 100644--- a/backend/src/superTabs/store.ts+++ b/backend/src/superTabs/store.ts@@ -13,7 +13,8 @@ import { applyCommand, authorizeCommandAccess, operationSchema, PAIR_CHAIN_LIMITimport { redactCredentials } from './credentials.js'import type { AvatarUpload } from './profile.js'import { reactionCounts, type ReactionTarget } from './reactions.js'-import { agentOpenDeliveries, deliveredCopies, messagesById, messagesPast, OPEN_DELIVERY, type MessageEdge, pageDeliveries, pageReactions, pageThreads, type PageDelivery } from './pageReads.js'+import { agentOpenDeliveries, compactedDelivery, deliveredCopies, messagesById, messagesPast, OPEN_DELIVERY, type MessageEdge, pageDeliveries, pageReactions, pageThreads, refreshReplies, type PageDelivery } from './pageReads.js'+import { restoreDelivery } from './deliveryCompaction.js'import { threadSummaries } from './threads.js'import { attachmentMime, attachmentName, AttachmentTooLarge, AVATAR_PURPOSE, AVATAR_TYPES, checkedType, MAX_ATTACHMENT_BYTES, MAX_AVATAR_BYTES, objectKeyFor, peekType, publicUrl, trimPrefix,type Attachment, type Attachments, type AttachmentStore, type AvatarStore } from './attachments.js'@@ -66,6 +67,7 @@ import { closeAssignment, closeUnassignedStart, LIST_SIZE, newestConversations,import { dropHeldTags, listMentions, markDone, mentionsWaiting, openTags, readMention, resolveByReply, resolveWhenWorked, writeMentions } from './mentions.js'import { answerable, askDecisions, MAX_OPEN_DECISIONS, REFLECTION_DECISION, reflectionDecisions, expireReflections, fullFor, listDecisions, type DecisionList, readAskedDecision, recordAnswer, repeatOf, waitingFor, waitingIn, waitingInProject, writeDecisions } from './decisions.js'import { CATCH_UP_SCAN, catchUpSummary, type CatchUp } from './catchUp.js'+import { recordMessageRead, type MessageReadKey } from './messageReadStats.js'import { DEFAULT_ROUTING_POLICY, effectivePolicy, routingPolicySaveSchema, type RoutingPolicy } from './routingPolicy.js'import { invitedSwarms } from './invitedSwarms.js'import { authenticatedEmail, authorizeSwarmJoin, configureSharing, joinSwarmSchema, requireSharingOwner,@@ -407,6 +409,23 @@ const buddyNews = (message: Pick<SuperTabMessage, 'recipientIds'>, delivery: Picconst readsRow = (actor: Actor, buddy: boolean) => (m: Pick<SuperTabMessage, 'source' | 'recipientIds'>): boolean =>(!agentOnly(m.source, m.recipientIds) || (actor.kind === 'agent' && m.recipientIds.includes(actor.agentId))) && !(buddy && routingRow(m.source))+/** A page shows 100 rows and reads a 101st to know there are more. After the first batch, each reads the rows still missing at+ * the rate this reader has seen them so far, plus a tenth; twice what was read while none was readable; at least MIN_BATCH,+ * at most MAX_BATCH (spec 5 A2). One hidden row costs 10 more rows, not a second batch of 101; a chat mostly of notes to+ * others fills in about two batches, reading about what the old 101-row batches read. */+const PAGE_BATCH = 101, MIN_BATCH = 10, MAX_BATCH = 400+const nextBatch = (decoded: number, shown: number): number => Math.min(MAX_BATCH, Math.max(MIN_BATCH,+ shown ? Math.ceil((PAGE_BATCH - shown) * decoded / shown * 1.1) : 2 * decoded))+/** How a messages read counts in the read log (messageReadStats.ts). */+const readerKind = (actor: Actor, buddy: boolean): MessageReadKey['reader'] => buddy ? 'buddy' : actor.kind+const sessionKind = (user: AuthUser): MessageReadKey['session'] => user.harnessSessionKind ?? 'sso'+/** A message's receipts in the order they were written, live or compacted alike: by creation, one write's in the order of+ * its recipients (store.ts writeMessages), then by id. */+const receiptOrder = (m: Pick<SuperTabMessage, 'recipientIds'>) => (a: PageDelivery, b: PageDelivery): number => {+ const place = (d: PageDelivery) => { const at = m.recipientIds.indexOf(d.agentId); return at < 0 ? Infinity : at }+ return a.createdAt.getTime() - b.createdAt.getTime() || place(a) - place(b) || (a.id < b.id ? -1 : a.id > b.id ? 1 : 0)+}+/** A row as a Buddy reads it, without the machine it names (security reviews of #2115): a lost task's reason, in its text and* its source (liveness.ts loseTask), a machine leaving the Swarm (machines.ts removalEnded, commands.ts removeMachine), and* the machine a Product Manager started on (leadStart.ts leadJoined; Diego, 10-30) or a removed session is still held on@@ -2050,6 +2069,8 @@ export class SuperTabStore {}}if (rows.length) await tx.superTabMessage.createMany({ data: rows })+ // A reply's thread summary, in this transaction (pageReads.ts refreshThreads).+ await refreshReplies(tx, tabId, rows)for (const message of messages) {// Once committed, so a retried write logs once.if (message.source?.kind === SESSION_ENDED_KIND && !kept.has(message.id)) {@@ -2059,6 +2080,9 @@ export class SuperTabStore {}if (kept.size) {const have = new Set((await tx.superTabDelivery.findMany({ where: { messageId: { in: [...kept] } }, select: { id: true } })).map(d => d.id))+ // A delivery compacted into its message's receipts (deliveryCompaction.ts) was written too: never sent again.+ const compacted = await readBy(tx, 'SuperTabReceipt', 'id', [{ $match: { _id: { $in: [...kept] }, tabId } }, { $project: { deliveryIds: 1 } }])+ for (const receipt of compacted) for (const id of receipt.deliveryIds as string[] ?? []) have.add(id)deliveries = deliveries.filter(d => !have.has(d.id))}if (deliveries.length) await tx.superTabDelivery.createMany({ data: deliveries })@@ -2099,6 +2123,8 @@ export class SuperTabStore {async messages(user: AuthUser, tabId: string, conversationId: string,cursor: { after?: number; afterId?: string; before?: number; beforeId?: string; messageId?: string; thread?: string }, asAgentId?: string) {const tx = this.db+ const read: MessageReadKey['cursor'] = cursor.messageId ? 'messageId' : cursor.thread ? 'thread' : cursor.after !== undefined ? 'after'+ : cursor.before === undefined ? 'latest' : cursor.beforeId === undefined ? 'before' : 'beforeId'const { state } = await this.readLoad(user, tabId, { groupIds: [conversationId] })const actor = await this.actor(tx, user, state, asAgentId)const group = readConversation(state, actor, conversationId)@@ -2133,17 +2159,23 @@ export class SuperTabStore {: cursor.before !== undefined ? { sequence: cursor.before, id: cursor.beforeId } : undefined// Hidden rows never shorten a page: Desktop reads on from the// last row it shows, so the page reads past them until it has 101 rows this reader may see, or reaches the end.- // Each batch reads on from the last row of the one before, never by an offset that walks every row before it again.+ // Each batch reads on from the last row of the one before, never by an offset that walks every row before it again, and+ // only about what is still missing (nextBatch): one hidden row once cost a whole second batch of 101.const rows: SuperTabMessage[] = []- for (let edge = start; rows.length <= 100;) {- const batch = await messagesPast(tx, at, edge, forward, 101)+ let batches = 0, decoded = 0+ for (let edge = start, size = PAGE_BATCH; rows.length < PAGE_BATCH;) {+ const batch = await messagesPast(tx, at, edge, forward, size)+ batches+++ decoded += batch.length// A task chat's copy of a Lead's notice stored before it was marked is that notice: read by no one (pageReads.ts).const copies = await deliveredCopies(tx, tabId, batch)rows.push(...batch.filter(m => !copies.has(m.id) && readable(m)))- if (batch.length < 101) break+ if (batch.length < size) breakedge = { sequence: batch.at(-1)!.sequence, id: batch.at(-1)!.id }+ size = nextBatch(decoded, rows.length)}const page = rows.slice(0, 100)+ recordMessageRead({ reader: readerKind(actor, buddy), cursor: read, session: sessionKind(user) }, batches, decoded, page.length)const next = rows.length > 100 ? { sequence: page.at(-1)!.sequence, id: page.at(-1)!.id } : nullif (!forward) page.reverse()const reactions = page.length ? reactionCounts(state, await pageReactions(tx, tabId, page.map(m => m.id))) : new Map()@@ -2173,7 +2205,7 @@ export class SuperTabStore {// A session's own thread read carries the main-chat excerpt it started from (buddySessions.ts sessionContext).const context = own && !forward && cursor.before === undefined && !cursor.messageId ? await this.sessionExcerpt(tx, tabId, conversationId, own.from, rowRead) : undefinedreturn { messages: shown.map(m => ({ ...m, mentions: m.mentions ?? [], reactions: reactions.get(m.id) ?? [],- deliveries: deliveries.filter(d => d.messageId === m.id && shownTo(m, d)) })), next, threads, ...(context ? { context } : {}),+ deliveries: deliveries.filter(d => d.messageId === m.id && shownTo(m, d)).sort(receiptOrder(m)) })), next, threads, ...(context ? { context } : {}),// A poll for new messages learns here that a reaction changed on a message it already shows (reactions.ts).reactionSeq: group.reactionSeq ?? 0 }}@@ -2206,6 +2238,7 @@ export class SuperTabStore {// A task chat's copy of a Lead's notice stored before it was marked is read by no one, as on the page (pageReads.ts).const copies = await deliveredCopies(this.db, tabId, scanned)const shown = scanned.filter(m => !copies.has(m.id) && reads(m))+ recordMessageRead({ reader: readerKind(actor, buddy), cursor: 'catchUp', session: sessionKind(user) }, 1, rows.length, shown.length)return catchUpSummary(buddy ? shown.map(rowForBuddy) : shown, user.sub, { after, more: rows.length > CATCH_UP_SCAN, decisions })}@@ -2842,7 +2875,8 @@ export class SuperTabStore {async deliveryReceipt(user: AuthUser, tabId: string, agentId: string, deliveryId: string) {return transaction(this.db, async tx => {const { state } = await this.load(tx, user, tabId)- const row = await tx.superTabDelivery.findFirst({ where: { id: deliveryId, tabId, agentId } })+ // A host back after weeks offline asks about a delivery compacted since (deliveryCompaction.ts): answered from its receipt.+ const row = await tx.superTabDelivery.findFirst({ where: { id: deliveryId, tabId, agentId } }) ?? await compactedDelivery(tx, tabId, agentId, deliveryId)if (!row) fail('NOT_FOUND', 'This delivery is unavailable.', 404)const previous = state.agents.find(a => a.id === agentId && a.membershipGeneration === row.membershipGeneration)const owner = row.machineOwnerId ?? previous?.machineOwnerId@@ -2896,7 +2930,9 @@ export class SuperTabStore {action: { type: 'claim'; claimId: string } | { type: 'ack'; claimId: string; receipt: string; uncertain: boolean }| { type: 'hold'; claimId: string; reason: HoldReason }, reached: { authorized: boolean }) {return transaction(this.db, async tx => {- const delivery = await tx.superTabDelivery.findFirst({ where: { id: deliveryId, tabId, agentId } })+ const live = await tx.superTabDelivery.findFirst({ where: { id: deliveryId, tabId, agentId } })+ // A compacted delivery (deliveryCompaction.ts) is checked as its row was: finished, it is refused or resolved as before.+ const delivery = live ?? await compactedDelivery(tx, tabId, agentId, deliveryId)await this.row(tx, user, tabId)// A native message may be delivered to its Lead in a different conversation from the terminal it came from.const context = delivery?.nativeContextId ? await tx.superTabNativeContext.findFirst({@@ -2932,6 +2968,8 @@ export class SuperTabStore {const part = parts.members.find(p => p.key === touch.key)if (part) await guardedWrite(tx, tabId, { op: 'touch', ...touch, seen: part.version }, stamp)}+ // Only a held delivery the pass ended can get here compacted: its host typed it after all, so it is a row again.+ if (!live) await restoreDelivery(tx, delivery)// A claim that gets in ends the failing record. Unset, never null: a missing field is a real absence, which the first failure's// isSet: false branch matches and which leaves no keys in the document.const updated = await tx.superTabDelivery.update({ where: { id: delivery.id }, data: { status, claimId: action.claimId,diff --git a/backend/src/superTabs/threadBackfill.ts b/backend/src/superTabs/threadBackfill.tsnew file mode 100644index 000000000..35f433e93--- /dev/null+++ b/backend/src/superTabs/threadBackfill.ts@@ -0,0 +1,105 @@+/**+ * Thread summaries for replies no transaction summed up (spec 5 phase D): every thread with a reply when a Swarm has none yet,+ * then, every 15 minutes, the threads with a reply created since an hour before the Swarm's marker. An older release+ * writes replies without summaries, during a rollback or while a release rolls out; they are counted here. A Swarm's pages+ * read its summaries only once its marker exists (pageReads.ts pageThreads).+ */+import type { PrismaClient } from '@prisma/client'+import { logger } from '../utils/logger.js'+import { messagesPast, noteBackfilled, refreshThreads, type MessageEdge } from './pageReads.js'+import { groupsBy } from './parts/indexed.js'+import { transaction } from './store.js'++/** How far before its marker a Swarm's replies are read again: a reply's time is its writer's clock. */+const SLACK_MS = 3_600_000+/** Threads refreshed per transaction. */+const BATCH = 100+/** Messages read per step back through a changed conversation. */+const TAIL = 200+/** How often the backfill runs: replies an older release writes go uncounted at most this long. */+const EVERY_MS = 15 * 60_000++/** Refreshes the summaries of every Swarm's threads with a reply since its marker less an hour (all of them without one), then+ * moves the marker to `now`. A Swarm that fails keeps its marker, is logged and is done at the next run. */+export async function backfillThreads(db: PrismaClient, now: number): Promise<{ swarms: number; threads: number }> {+ const marks = new Map((await db.superTabThreadBackfill.findMany()).map(mark => [mark.id, mark.through.getTime()]))+ const swarms = await db.superTab.findMany({ select: { id: true } })+ let done = 0, threads = 0+ for (const { id: tabId } of swarms) {+ try {+ threads += await backfillSwarm(db, tabId, marks.get(tabId), now)+ done+++ } catch (error) {+ logger.error('[swarm-threads] backfilling a Swarm failed; its pages keep their last summaries', error, { tabId })+ }+ }+ return { swarms: done, threads }+}++async function backfillSwarm(db: PrismaClient, tabId: string, through: number | undefined, now: number): Promise<number> {+ const replied = through === undefined ? await everyReply(db, tabId) : await repliesSince(db, tabId, through - SLACK_MS)+ let threads = 0+ for (const [conversationId, roots] of replied) {+ threads += roots.size+ const list = [...roots]+ for (let i = 0; i < list.length; i += BATCH) {+ await transaction(db, tx => refreshThreads(tx, tabId, conversationId, list.slice(i, i + BATCH)))+ }+ }+ await db.superTabThreadBackfill.upsert({ where: { id: tabId }, create: { id: tabId, through: new Date(now) }, update: { through: new Date(now) } })+ noteBackfilled(tabId)+ return threads+}++type Replied = Map<string, Set<string>>+const add = (replied: Replied, conversationId: string, thread: string) =>+ replied.set(conversationId, (replied.get(conversationId) ?? new Set()).add(thread))++/** Every thread of the Swarm with a reply, by conversation: its first backfill, which reads each of its messages once. */+async function everyReply(db: PrismaClient, tabId: string): Promise<Replied> {+ const rows = await db.superTabMessage.aggregateRaw({ pipeline: [+ { $match: { tabId, thread: { $type: 'string' } } },+ { $match: { $expr: { $ne: ['$thread', '$_id'] } } },+ { $group: { _id: { conversationId: '$conversationId', thread: '$thread' } } }] }) as unknown as Array<{ _id: { conversationId: string; thread: string } }>+ const replied: Replied = new Map()+ for (const { _id } of rows) add(replied, _id.conversationId, _id.thread)+ return replied+}++/** The threads with a reply created since `since`, by conversation. Only what changed is read: every message write moves its+ * conversation's group (its last sequence), so the groups changed since then name the conversations (their changes index),+ * and each is read newest first by its sequence index until its messages are older. History kept forever is never walked. */+async function repliesSince(db: PrismaClient, tabId: string, since: number): Promise<Replied> {+ const changed = await groupsBy(db, 'changes', [{ $match: { tabId, changedAt: { $gte: { $date: new Date(since).toISOString() } } } },+ { $project: { _id: 0, key: 1 } }])+ const replied: Replied = new Map()+ for (const { key } of changed) {+ const conversationId = key as string+ for (let edge: MessageEdge | undefined; ;) {+ const rows = await messagesPast(db, { tabId, conversationId }, edge, false, TAIL)+ for (const row of rows) if (row.createdAt.getTime() >= since && row.thread && row.thread !== row.id) add(replied, conversationId, row.thread)+ if (rows.length < TAIL || rows.at(-1)!.createdAt.getTime() < since) break+ edge = { sequence: rows.at(-1)!.sequence, id: rows.at(-1)!.id }+ }+ }+ return replied+}++/** Backfills now and then every `everyMs`, one run at a time, beside the worker's other jobs (pages read the $facet until a+ * Swarm's marker exists, so nothing waits for it). Every run reads only what changed, so replies an older release wrote+ * (a rollout, or a rollback of the API alone) are summed up within one interval. */+export function startThreadBackfill(db: PrismaClient, everyMs = EVERY_MS): () => void {+ let running = false+ const run = () => {+ if (running) return+ running = true+ const started = Date.now()+ void backfillThreads(db, started).then(({ swarms, threads }) => logger.info('[swarm-threads] backfilled', { swarms, threads, ms: Date.now() - started }))+ .catch(error => logger.error('[swarm-threads] backfill failed; it runs again at the next interval', error))+ .finally(() => { running = false })+ }+ run()+ const timer = setInterval(run, everyMs)+ timer.unref()+ return () => clearInterval(timer)+}diff --git a/backend/src/worker.ts b/backend/src/worker.tsindex a57b7a800..456f71f10 100644--- a/backend/src/worker.ts+++ b/backend/src/worker.ts@@ -20,6 +20,8 @@ import { releaseSchema } from './lib/db/schemaRelease.js'import { startSwarmSweeps } from './superTabs/sweeps.js'import { startLegacyDrop } from './superTabs/migration/dropLegacyJob.js'import { startDeliverySettle } from './superTabs/deliverySettle.js'+import { startDeliveryCompaction } from './superTabs/deliveryCompaction.js'+import { startThreadBackfill } from './superTabs/threadBackfill.js'import { appPullRequests, markFinishedGroups, startSwarmArchive, unknownPullRequests } from './superTabs/archive.js'import { SuperTabStore } from './superTabs/store.js'import { announceSwarmDeliveries } from './lib/swarmDeliveryPushes.js'@@ -66,6 +68,8 @@ async function start(): Promise<void> {let stopLegacyDrop = (): void => {}let stopArchive = (): void => {}let stopDeliveries = (): void => {}+ let stopCompaction = (): void => {}+ let stopThreadBackfill = (): void => {}let stopReceiptSweep = (): void => {}let stopReflection = (): void => {}const shutdown = (signal: string): void => {@@ -78,8 +82,10 @@ async function start(): Promise<void> {stopLegacyDrop()stopArchive()stopDeliveries()+ stopCompaction()stopReceiptSweep()stopReflection()+ stopThreadBackfill()// The last partial minute, before the database closes.stopQueryStats()stopRetryStats()@@ -107,6 +113,11 @@ async function start(): Promise<void> {stopArchive = startSwarmArchive(prisma, swarms, () => githubApp ? appPullRequests(githubApp) : unknownPullRequests)// A delivery that can no longer move ends as Not sent, instead of waiting for good (superTabs/deliverySettle.ts).stopDeliveries = startDeliverySettle(prisma, swarms)+ // Every hour, finished deliveries 14 days old move into their messages' receipts, kept forever (superTabs/deliveryCompaction.ts).+ stopCompaction = startDeliveryCompaction(prisma, env.SWARM_DELIVERY_COMPACTION)+ // Every 15 minutes, thread summaries for replies no write summed up: all of a Swarm's at first, then only what changed+ // (superTabs/threadBackfill.ts). Pages read the summaries once a Swarm is backfilled.+ stopThreadBackfill = startThreadBackfill(prisma)// At 23:00 in each project's time zone, what its routed work cost and how it went, with routing changes asked of its owner// (lib/routingReflection.ts).stopReflection = startRoutingReflection(prisma, swarms)diff --git a/backend/vitest.super-tabs.config.ts b/backend/vitest.super-tabs.config.tsindex 70a741197..6fd3c4d38 100644--- a/backend/vitest.super-tabs.config.ts+++ b/backend/vitest.super-tabs.config.ts@@ -6,5 +6,6 @@ export default defineConfig({ test: { include: ['integration/superTabs.test.ts','integration/superTabHotReads.test.ts', 'integration/swarmSweepStarts.test.ts', 'integration/superTabGroupPlans.test.ts', 'integration/superTabInvitationPlans.test.ts', 'integration/superTabCache.test.ts', 'integration/superTabMachineReports.test.ts','integration/superTabObserver.test.ts', 'integration/productUsageConflicts.test.ts', 'integration/productUsageReceipts.test.ts', 'integration/schemaRelease.test.ts', 'integration/superTabBuddies.test.ts', 'integration/superTabNotAvailable.test.ts','integration/superTabBuddyProbes.test.ts', 'integration/superTabBuddySessions.test.ts', 'integration/superTabCosts.test.ts', 'integration/harnessSharesList.test.ts', 'integration/atlasSnapshot.test.ts', 'integration/queryStats.test.ts',- 'integration/superTabEvidence.test.ts', 'integration/superTabAttemptModels.test.ts', 'integration/superTabNamedReads.test.ts', 'integration/swarmSweepPlans.test.ts', 'integration/workerIndexPlans.test.ts', 'integration/superTabReflection.test.ts'], hookTimeout: 120000, testTimeout: 30000, setupFiles: ['integration/cacheOn.test-support.ts'],+ 'integration/superTabEvidence.test.ts', 'integration/superTabAttemptModels.test.ts', 'integration/superTabNamedReads.test.ts', 'integration/swarmSweepPlans.test.ts', 'integration/workerIndexPlans.test.ts', 'integration/superTabReflection.test.ts',+ 'integration/superTabMessagePages.test.ts', 'integration/superTabDeliveryCompaction.test.ts', 'integration/superTabThreads.test.ts'], hookTimeout: 120000, testTimeout: 30000, setupFiles: ['integration/cacheOn.test-support.ts'],fileParallelism: false, env: { NODE_ENV: 'test', HARNESS_CREDENTIAL_ENCRYPTION_KEY: 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=' } } })diff --git a/docs/superpowers/plans/2026-10-30-swarm-messages-delta-and-retention.md b/docs/superpowers/plans/2026-10-30-swarm-messages-delta-and-retention.mdnew file mode 100644index 000000000..f27f86311--- /dev/null+++ b/docs/superpowers/plans/2026-10-30-swarm-messages-delta-and-retention.md@@ -0,0 +1,208 @@+# Swarm messages: read what a page shows, receipt deltas, receipts kept forever — Implementation Plan++> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.++**Goal:** Cut the messages route's database reads and CPU, and keep message receipts forever while delivery rows stop growing.++**Architecture:** Phase A (one release): a per-minute messages read log, continuation batches that read only what is missing, and a `SuperTabReceipt` collection that page reads merge. Phase B (only if A's log says so): receipt deltas on `?after` reads, used by Desktop and web. Phase C: a worker job (on by default) that moves final deliveries older than 14 days into `SuperTabReceipt`. Phase D: a `SuperTabThread` collection holding each replied thread's `$facet` result, refreshed when a reply is inserted, read by id instead of aggregating on every page.++**Tech Stack:** TypeScript, Fastify, Prisma 5.22 on MongoDB, vitest + `mongodb-memory-server`; Flutter (Desktop and web share `desktop/lib/super_tabs/controller.dart`).++**Spec:** `docs/superpowers/specs/2026-10-30-swarm-messages-delta-and-retention-design.md`++## Global Constraints++- Messages are kept forever; nothing here deletes a message.+- Compact only `answered`, `cancelled`, `superseded`; never `delivered`, `confirmed`, `queued`, `claimed`, `uncertain`.+- Compact after **14 days** since the delivery's `updatedAt`; delta slack **5 s**; a `since` older than **1 hour** answers without `receipts`.+- Phase B goes only if Task 1's log shows Desktop/web re-reads are **at least 30%** of whole-page reads. Record the decision in the PR.+- Compaction is **on by default** (Ken, 2026-10-30); env `SWARM_DELIVERY_COMPACTION=off` is the kill switch. Set it to `off` before rolling back to a release older than Phase A, whose page reads do not read `SuperTabReceipt`.+- Receipts are never stored on a message document and never returned whole; every receipt shown goes through `shownTo`.+- Do not commit or push until Ken asks; each task ends with a diff checkpoint. Remove dead code.+- Backend commands from `backend/`: unit `npx vitest run <file>`; integration `npx vitest run --config vitest.super-tabs.config.ts <file>`. Desktop: `flutter test <file>` from `desktop/`.++## Review Focus++1. A host offline for more than 14 days asks `GET …/deliveries/:id` for a delivery that was compacted: it gets the final status, not 404 — Task 5 test.+2. A notice re-written with `keepStored` after its deliveries were compacted is not delivered again — Task 5 test.+3. A Buddy-news row's compacted receipt is hidden from everyone but that Buddy, as a live one is — Task 3 test.+4. A conversation whose first 101 rows are mostly hidden (an agent reading a Lead chat full of notes to others) still fills its page — Task 2 test.+5. A new Desktop talking to a backend without `since` (400 on unknown field) keeps polling — Task 7 test.++---++### Task 1: Messages read log (A1)++**Files:**+- Create: `backend/src/superTabs/messageReadStats.ts`+- Modify: `backend/src/superTabs/store.ts` (`messages`, `catchUp`), `backend/src/server.ts` (start beside `startQueryStats`)+- Test: `backend/src/superTabs/messageReadStats.test.ts`++**Interfaces:**+- Produces: `type MessageReadKey = { reader: 'person' | 'agent' | 'buddy'; cursor: 'latest' | 'after' | 'before' | 'beforeId' | 'thread' | 'messageId' | 'catchUp' }`; `recordMessageRead(key: MessageReadKey, batches: number, decoded: number, shown: number): void`; `startMessageReadStats(options?: { everyMs?: number; log?: (message: string, context: Record<string, unknown>) => void }): () => void` logging `'messages by reader'` each minute with `{ windowMs, rows: Array<MessageReadKey & { n, batches, decoded, shown }> }`.+- `before` without `beforeId` is `'before'` (Desktop's refresh of the page it shows); with `beforeId` is `'beforeId'` (scrolling back).++- [ ] **Step 1: Write the failing test** `'counts reads per reader and cursor, with batches and rows, and names nothing'`: record two `{person, before}` reads (2 batches, 202 decoded, 100 shown) and one `{agent, latest}` (1, 101, 100); run with `everyMs: 50`; expect one log row `{reader: 'person', cursor: 'before', n: 2, batches: 4, decoded: 404, shown: 200}` and one for the agent, and `JSON.stringify(context)` matching no id pattern.+- [ ] **Step 2: Run** `npx vitest run src/superTabs/messageReadStats.test.ts` — FAIL, module missing.+- [ ] **Step 3: Implement** the module (a `Map` keyed by `reader|cursor`, flushed per window) and call `recordMessageRead` once per `messages`/`catchUp` call with the batches, rows decoded (every row a batch returned) and rows shown; reader is `'buddy'` when `isBuddyReader(state, actor)`, else the actor's kind.+- [ ] **Step 4: Run** the test and `npx vitest run src/superTabs` — PASS.+- [ ] **Step 5: Checkpoint** `git diff --stat`.++### Task 2: Continuation batches read what is missing (A2)++**Files:**+- Modify: `backend/src/superTabs/store.ts` (the `for (let edge = start; rows.length <= 100;)` loop in `messages`, and the same loop shape in `catchUp` if it pages the same way), `backend/src/superTabs/pageReads.ts` (`messagesPast` already takes `limit`)+- Test: `backend/integration/superTabRead.test.ts` (new `describe('a page reads about the rows it shows (spec 5 A2)')`)++**Interfaces:**+- Consumes: Task 1's `recordMessageRead` counts, used by the test to read rows decoded.+- Produces: no new export. Batch size after the first: `Math.max(MIN_BATCH, 101 - rows.length + 2 * hiddenSoFar)` with `MIN_BATCH = 10`.++- [ ] **Step 1: Write the failing tests:**+ - `'a page with one hidden row in its first 101 decodes at most 101 + 10 rows and returns the same page and next as before'`: a conversation of 300 messages, one agent-only note to another agent among the newest 101; read the latest page as a person; expect `decoded <= 111`, the 100 rows and `next` equal to the page the old loop returns (compute it in the test by reading 202 rows directly and filtering).+ - `'an agent reading a chat where most rows are notes to others still gets a full page'`: 400 messages, 3 of every 4 agent-only notes to someone else; expect 100 rows shown and `next` set.+- [ ] **Step 2: Run** `npx vitest run --config vitest.super-tabs.config.ts integration/superTabRead.test.ts -t "rows it shows"` — first test FAILS on `decoded` (202).+- [ ] **Step 3: Implement** the batch size above in each loop; keep the stop rule (`rows.length > 100` or a short batch).+- [ ] **Step 4: Run** `integration/superTabRead.test.ts`, `integration/superTabNamedReads.test.ts`, `integration/superTabHotReads.test.ts` — PASS.+- [ ] **Step 5: Checkpoint** `git diff --stat`.++### Task 3: `SuperTabReceipt` and page reads that merge it (A3)++**Files:**+- Modify: `backend/prisma/schema.prisma` (new model), `backend/src/superTabs/pageReads.ts` (`pageDeliveries`)+- Test: `backend/integration/superTabRead.test.ts`++**Interfaces:**+- Produces: model+ ```prisma+ model SuperTabReceipt {+ id String @id @map("_id") // the message id+ tabId String+ conversationId String+ receipts Json // { [agentId]: CompactedReceipt }+ deliveryIds String[] // the compacted deliveries' ids, so a host's receipt read finds one by id+ @@index([tabId, conversationId])+ @@index([tabId, deliveryIds])+ }+ ```+ `type CompactedReceipt = Omit<PageDelivery, 'id' | 'messageId'> & { deliveryId: string; machineId: string; machineOwnerId: string | null; membershipGeneration: string }` exported from `pageReads.ts`; `pageDeliveries(tx, tabId, messageIds)` keeps its signature and returns live rows plus compacted ones (one `PageDelivery` per agent, `id` = the receipt's `deliveryId`; a live row wins). The host fields are never put into a `PageDelivery`.+- [ ] **Step 1: Write the failing tests:**+ - `'a page reads the same whether its deliveries are live, compacted or half each'`: answer three deliveries; read the page; move two into a `SuperTabReceipt` by hand and delete their rows; read again; deep-equal.+ - `'a live row wins over a compacted one for the same agent'`.+ - `'a compacted Buddy-news receipt is shown only to that Buddy'` (Review Focus 3).+ - `'no message read returns a receipts map'`: `JSON.stringify` of `messages`, `catchUp`, the inbox and mentions replies has no `"receipts"` key.+- [ ] **Step 2: Run** them — FAIL (model missing).+- [ ] **Step 3: Implement** the model (schema push creates it) and the merge in `pageDeliveries` (one `findMany` by `_id $in` on `SuperTabReceipt` beside the deliveries read; `hotFilters.allowed.json` gains the entry if the lint asks).+- [ ] **Step 4: Run** `integration/superTabRead.test.ts`, `src/superTabs/hotFilters.test.ts` — PASS.+- [ ] **Step 5: Checkpoint** `git diff --stat`. **Phase A ends here: ship it, then read Task 1's log for a weekday before Tasks 6–7.**++### Task 4: Compaction job (C)++**Files:**+- Create: `backend/src/superTabs/deliveryCompaction.ts`+- Modify: `backend/src/worker.ts` (start beside `startDeliverySettle` unless `env.SWARM_DELIVERY_COMPACTION === 'off'`), `backend/src/config/env.ts` (default `'on'`)+- Test: `backend/integration/superTabDeliveryCompaction.test.ts` (add to `vitest.super-tabs.config.ts` `include`)++**Interfaces:**+- Consumes: Task 3's model and `CompactedReceipt`.+- Produces: `FINAL_DELIVERY = ['answered', 'cancelled', 'superseded'] as const`; `COMPACT_AFTER_MS = 14 * 86_400_000`; `compactDeliveries(db: PrismaClient, now: number, batch = 500): Promise<{ compacted: number; remaining: number }>`; `startDeliveryCompaction(db: PrismaClient, everyMs = 3_600_000): () => void` logging `'[swarm-deliveries] compacted'` with both numbers. Each hourly run calls `compactDeliveries` until it compacts nothing or 20 batches ran (up to 10 k per run; prod finishes ≤ 3.4 k deliveries a day), so the first runs also clear the backlog: no separate data migration.+- Per batch, one transaction: read final deliveries with `updatedAt < now - COMPACT_AFTER_MS`, upsert each message's `SuperTabReceipt` with `$set: { 'receipts.<agentId>': … }` and `$addToSet: { deliveryIds: <id> }` (raw `updateOne` with `upsert`, field-path sets), delete those deliveries with `updatedAt` equal to what was read.+- [ ] **Step 1: Write the failing tests:** only final deliveries over 14 days are compacted (a `delivered`, a `confirmed`, a `queued` and a 13-day `answered` stay); two concurrent `compactDeliveries` calls on one message keep both agents; a delivery updated between read and delete is skipped; a page is deep-equal before and after; `remaining` counts final deliveries still over 14 days.+- [ ] **Step 2: Run** — FAIL (module missing).+- [ ] **Step 3: Implement** the module and the worker wiring (on unless the switch says `off`; a test that `off` starts nothing).+- [ ] **Step 4: Run** the new file and `integration/superTabRead.test.ts` — PASS.+- [ ] **Step 5: Checkpoint** `git diff --stat`.++### Task 5: Hosts and notices read compacted receipts (C)++**Files:**+- Modify: `backend/src/superTabs/store.ts` (`deliveryReceipt` ~2749, `delivery()` `typedAsEnded` ~2820, `writeMessages` `keepStored` ~2048)+- Test: `backend/integration/superTabDeliveryCompaction.test.ts`++**Interfaces:**+- Consumes: Task 3's `SuperTabReceipt`, Task 4's `compactDeliveries`.+- Produces: `compactedDelivery(tx, tabId, agentId, deliveryId): Promise<(CompactedReceipt & { messageId: string }) | null>` in `pageReads.ts`, found by `{ tabId, deliveryIds: deliveryId }` (index `[tabId, deliveryIds]`), then its `receipts[agentId]` with that `deliveryId`; `deliveryReceipt` and `typedAsEnded` fall back to it when no row exists, authorizing with its `machineOwnerId`/`machineId`/`membershipGeneration` exactly as with a row; `writeMessages(keepStored)` treats a message's compacted agent as an existing delivery. A plan test pins the lookup to `[tabId, deliveryIds]`.+- [ ] **Step 1: Write the failing tests:** `deliveryReceipt` answers `{ id, status: 'answered' }` for a compacted delivery and refuses another machine owner (403) (Review Focus 1); a host confirming it typed a compacted cancelled-with-hold delivery gets the same answer as before compaction; `writeMessages(keepStored)` re-writing a notice whose deliveries were compacted creates no delivery (Review Focus 2).+- [ ] **Step 2: Run** — FAIL (404 / re-delivered).+- [ ] **Step 3: Implement** the three fallbacks through `compactedDelivery`.+- [ ] **Step 4: Run** the file, the store's delivery suites (`integration/superTabs.test.ts -t deliver`) and the CLI host receipt tests (`cd ../cli && npx vitest run src/superTabs/host.spec.ts`) — PASS.+- [ ] **Step 5: Checkpoint** `git diff --stat`.++### Task 6: Receipt deltas on `?after` (B — only if the 30% gate passes)++**Files:**+- Modify: `backend/prisma/schema.prisma` (`SuperTabDelivery` gains `@@index([tabId, conversationId, updatedAt])`), `backend/src/routes/superTabs.ts:74` (`messageQuery` gains `since: z.coerce.number().int().nonnegative().safe().optional()`), `backend/src/superTabs/store.ts` (`messages`)+- Test: `backend/integration/superTabRead.test.ts`++**Interfaces:**+- Produces: `messages(user, tabId, conversationId, cursor: { …; since?: number }, asAgentId?)` replies with `serverTime: number` (taken before any read) always, and `receipts?: Array<{ messageId: string; deliveries: PageDelivery[] }>` when `after` and `since` are given and `now - since <= 3_600_000`: messages not in `messages` whose deliveries have `updatedAt >= since - 5_000`, read by `messagesById`, kept only if `readable` and not a delivered copy, deliveries filtered by `shownTo`, compacted receipts merged as in Task 3.+- [ ] **Step 1: Write the failing tests:** an answer after the last read appears in `receipts` on the next empty `?after`; one before does not; a hidden message's receipts never appear; `since` older than an hour gives no `receipts`; every reply has `serverTime`; an index plan test that the delta read uses `(tabId, conversationId, updatedAt)` (`integration/superTabHotReads.test.ts` style).+- [ ] **Step 2: Run** — FAIL.+- [ ] **Step 3: Implement.**+- [ ] **Step 4: Run** `integration/superTabRead.test.ts`, `integration/superTabHotReads.test.ts` — PASS.+- [ ] **Step 5: Checkpoint** `git diff --stat`.++### Task 7: Desktop and web use the delta (B — only with Task 6)++**Files:**+- Modify: `desktop/lib/super_tabs/controller.dart` (`_query`, `_loadMessages`: the "No new text: refresh delivery receipts" branch)+- Test: `desktop/test/super_tab_new_messages_test.dart` (or `super_tab_message_paging_test.dart`, whichever holds the receipt-refresh case)++**Interfaces:**+- Consumes: Task 6's `serverTime` and `receipts`.+- Produces: per conversation, the last `serverTime` seen; `since` is added to `?after=` only once a reply from this backend carried `serverTime`, and dropped for the session after a 400.+- [ ] **Step 1: Write the failing tests:** an empty `?after` read with `receipts` makes no `?before=<tail+1>` request and updates the held rows' deliveries; a reaction change still reads the page; a backend whose replies have no `serverTime` gets no `since` and keeps the old refresh; a 400 on `since` drops it and the next poll succeeds (Review Focus 5); the web reading policy behaves the same.+- [ ] **Step 2: Run** `flutter test test/super_tab_new_messages_test.dart` — FAIL.+- [ ] **Step 3: Implement.**+- [ ] **Step 4: Run** the file and `flutter test test/super_tab_*` — PASS (compare failures with a clean `main`, which has known reds).+- [ ] **Step 5: Checkpoint** `git diff --stat`.++### Task 8: Thread summaries written with each reply (D)++**Files:**+- Modify: `backend/prisma/schema.prisma` (new model), `backend/src/superTabs/pageReads.ts` (split `pageThreads` into the `$facet` read and the post-processing), `backend/src/superTabs/store.ts` (`writeMessages`), `backend/src/superTabs/native.ts:235`+- Test: `backend/integration/superTabThreads.test.ts` (new; add to `vitest.super-tabs.config.ts` `include`)++**Interfaces:**+- Produces: model+ ```prisma+ model SuperTabThread {+ id String @id @map("_id") // the root message id+ tabId String+ conversationId String+ parts Json // the $facet's `parts` for this one thread, as pageThreads reads them today+ mentions Json // the $facet's `mentions` for this one thread+ updatedAt DateTime+ @@index([tabId, conversationId])+ }+ ```+ In `pageReads.ts`: `threadFacet(tx, tabId, conversationId, roots: string[]): Promise<{ parts: FacetPart[]; mentions: FacetMention[] }>` (today's query, unchanged); `threadReplies(tx, tabId, facet): Promise<ThreadReply[]>` (today's post-processing, unchanged); `pageThreads` = `threadReplies(threadFacet(...))` until Task 9; `refreshThreads(tx, tabId, conversationId, roots: string[]): Promise<void>` upserts one `SuperTabThread` per root whose facet holds a reply (skips lone roots).+- [ ] **Step 1: Write the failing tests:** a reply inserted through `store.execute` (`message.send` with `replyTo`) leaves a `SuperTabThread` whose `parts`/`mentions` deep-equal `threadFacet` for that root; a native reply (`native.ts`) does too; a lone root has no document; a reply that fails its transaction leaves no document.+- [ ] **Step 2: Run** `npx vitest run --config vitest.super-tabs.config.ts integration/superTabThreads.test.ts` — FAIL.+- [ ] **Step 3: Implement** the split (no behaviour change) and call `refreshThreads` for the distinct roots of inserted rows whose `thread` is set and not their own id, after the insert, in the same transaction, in both insert sites.+- [ ] **Step 4: Run** the new file and `integration/superTabRead.test.ts` — PASS.+- [ ] **Step 5: Checkpoint** `git diff --stat`.++### Task 9: Backfill, per-Swarm switch, and reads from the collection (D)++**Files:**+- Create: `backend/src/superTabs/threadBackfill.ts`+- Modify: `backend/src/superTabs/pageReads.ts` (`pageThreads`), `backend/src/worker.ts` (run after `releaseSchema`, before jobs start), `backend/prisma/schema.prisma` (marker model)+- Test: `backend/integration/superTabThreads.test.ts`++**Interfaces:**+- Consumes: Task 8's `threadFacet`, `threadReplies`, `refreshThreads`.+- Produces: model `SuperTabThreadBackfill { id String @id @map("_id") /* tabId */; through DateTime }`; `backfillThreads(db: PrismaClient, now: number): Promise<{ swarms: number; threads: number }>` — for each Swarm, refreshes every thread with a reply created after `marker.through - 3_600_000` (all of them when no marker), then sets `through = now`; `pageThreads(tx, tabId, conversationId, roots)` reads `SuperTabThread` by `_id $in roots` when the Swarm's marker exists (cached per process for 60 s), else `threadFacet` as today, and in both cases returns `threadReplies(...)`. A root without a document contributes what `threadFacet` gives for a lone root; prove it equal, or drop lone roots if `threadSummaries` ignores them.+- [ ] **Step 1: Write the failing tests:** for a scene with lone roots and replied threads of every visibility class (agent-only notes to others, route rows, a Buddy session thread, mentions) and readers (owner, collaborator outside a private project, Buddy, agent), each reader's page `threads` deep-equals the `$facet` path; a Swarm without a marker reads through `threadFacet`; after backfill the page reads no `SuperTabMessage` aggregation for threads (count by the profiler); a reply written while the marker was stale (insert it with `refreshThreads` skipped, as an older release would) is counted after the next `backfillThreads`.+- [ ] **Step 2: Run** them — FAIL.+- [ ] **Step 3: Implement.**+- [ ] **Step 4: Run** the file, `integration/superTabRead.test.ts`, `integration/superTabHotReads.test.ts` — PASS.+- [ ] **Step 5: Checkpoint** `git diff --stat`.++### Task 10: Checks and measurement++- [ ] **Step 1:** `npm run typecheck` in `backend/`; `flutter analyze` in `desktop/` if Task 7 ran.+- [ ] **Step 2:** `npm run test:super-tabs` and `npm test` in `backend/` — only failures already on clean `main`, listed in the PR.+- [ ] **Step 3:** After Phase A's release, one weekday (07:00–16:00 VN) of `'messages by reader'` logs: whole-page reads by reader and cursor, decoded per shown (target < 1.2). Write the Phase B go/no-go into the PR with the numbers.+- [ ] **Step 4:** After Phase C is switched on: the hourly `compacted`/`remaining` log, and weekly `SuperTabDelivery`/`SuperTabReceipt` counts and index sizes.+- [ ] **Step 5:** After Phase D: the `$facet` shape under the messages route in `db by shape` (today ~100 a pod-minute, 3.4 MB) gone, replaced by a `SuperTabThread` read by id.diff --git a/docs/superpowers/specs/2026-10-30-swarm-messages-delta-and-retention-design.md b/docs/superpowers/specs/2026-10-30-swarm-messages-delta-and-retention-design.mdnew file mode 100644index 000000000..f11ea48b2--- /dev/null+++ b/docs/superpowers/specs/2026-10-30-swarm-messages-delta-and-retention-design.md@@ -0,0 +1,212 @@+# Design: Swarm messages — read only what a page shows, receipts as deltas, and data that scales for years++Date: 2026-10-30. Code read at `origin/main` (cdf21a617). Spec 5 in the CPU series; it covers the WebSocket roadmap's+item 2.3 for messages and the database roadmap's Phase 5 for the message collections. Ken's decision (2026-10-30):+**message history is kept forever.**++## Why++`GET /api/super-tabs/:tabId/conversations/:conversationId/messages` is the backend's largest database reader.++**Prod, 2026-10-30 (60 min, 3 pods):** 298 requests per pod-minute. The `db by shape` log shows ~197 backward page reads+per pod-minute (`SuperTabMessage` `sequence $lt`, 12.6 MB), plus the threads aggregation (`$facet`, 3.4 MB) and+deliveries (`messageId $in`, 1.4 MB). That log keeps only each minute's top shapes (`lib/queryStats.ts`: 20 by bytes,+10 by count, 10 by time), so small reads (an empty `?after` poll) are undercounted and **who sends the backward reads is+not known**. Candidates, all in the code:+- Desktop and the web build (`desktop/lib/super_tabs/controller.dart _loadMessages`, shared by `desktop/lib/web`):+ after an empty `?after=<tail>` read, the whole last page again with `?before=<tail+1>`, to refresh receipts; scrolling+ back; "holding" reads.+- CLI agents (`cli/src/superTabs/service.ts:598`, the `messages` action): the latest page or a `before` page.+- **Continuation batches inside one read** (`store.ts messages`): a page reads batches of 101 rows until it holds 101 it+ may show. One hidden row in the first batch (an agent-only note to someone else, a route row, a delivered copy) makes+ it read a whole second batch of 101. On a local copy of prod's busiest Swarm (`c51479bd…`), the owner's latest page+ of the Lead chat decoded 202 rows to show 100.++**Cost:** on that local copy a whole-page read costs **15 ms of process CPU** (decoding ~100–200 messages, the thread+aggregation, deliveries); an `?after` read with nothing new **2.1 ms**.++**Growth (prod, Swarm data since 2026-10-19):**++| Collection | Now | Per day | In a year at this rate | Deleted today |+|---|---|---|---|---|+| `SuperTabMessage` | 43k docs, 41 MB | 5,471 | ~2 M docs, ~1.9 GB | never (and never will be: kept forever) |+| `SuperTabDelivery` | 19k, 13 MB, 6 indexes | 2,376 | ~870 k, ~600 MB | never; 0 open now |++At 10× users, 10× that. The M20 cluster has 4 GB of RAM per node with ~1 GB free; the first limit is indexes and+working set against memory. Message reads stay index-bounded however long the history.++## Goals++- Know, per minute, who reads messages how: reader kind, cursor kind, batches and rows read against rows shown.+- A page decodes about the rows it shows, never a whole extra batch for a few hidden rows.+- If the numbers confirm it: a Desktop/web poll with nothing new never reads a whole page; changed receipts come as a+ delta.+- Message history is kept forever with its receipts, and delivery rows stop growing without bound.+- A page reads its threads' summaries instead of aggregating every root's thread on each read.++## Non-goals++- Deleting or archiving messages.+- Operation receipts' growth (`SuperTabOperation`, 7 k a day): native prompts and recoveries (`native.ts:178, 215`) and+ `agent.remove` retries (`store.ts:1117`) read old receipts, so its expiry needs its own decision. Follow-up.+- `SuperTabRouteDecision` growth: DB roadmap Phase 5.+- Sharding. Message reads all name `tabId`; `SuperTabDelivery` also has machine-level indexes (`[machineId, status]`,+ `[status, id]`) and reads by `_id`, so it is not shard-ready, and that is not changed here.++## Anti-goals++- A row, receipt or thread line shown to a reader who may not see it today (`readable`, `shownTo`, Buddy rules,+ `deliveredCopies`).+- A receipt that reads differently after its delivery is compacted, or a notice re-sent because its deliveries were+ compacted.+- A host that cannot settle a delivery it holds because the row is gone.++## Design decided++### Phase A — measure, stop over-reading, read compacted receipts (one release)++**A1. Messages read log.** Every minute per pod, beside `db by shape`: request counts by reader kind (person, agent,+Buddy) × cursor kind (latest, `after`, `before`, `thread`, `messageId`) × caller (the harness session's kind as the+request's auth reads it: Desktop/web app vs CLI daemon), and for each: batches read, rows decoded, rows shown. No ids, no+text.++**A2. Continuation batches read what is missing.** After the first batch, a page reads only as many more rows as it+still needs to reach 101 it may show, plus a margin (the builder picks it from the hidden-row rate, e.g. twice the+hidden rows seen so far, at least 10), and loops while short. The page and its `next` cursor are unchanged. Applies to+`messages`, threads and `catchUp` where they use the same loop.++**A3. Compacted receipts are read.** A new collection `SuperTabReceipt` (one document per message: `_id` = message id,+`tabId`, `conversationId`, `receipts` by agent id, and `deliveryIds`, indexed with `tabId`, so a host's receipt read+finds a compacted delivery by its id) holds compacted deliveries (Phase C). A page read builds each+message's `deliveries` from that message's live delivery rows plus its compacted receipts (a live row wins for the same+agent), filtered by `shownTo` exactly as today. Receipts live in their own collection, never on the message: whole+message rows are spread or returned in several places (`messagesPast` rows, `deliver()`, `mentions.ts:257`), and a+receipt map there would bypass `shownTo` (security review 2 of #2178). Shipping this read before compaction is turned on+makes a later rollback safe.++### Phase B — receipt deltas (go only if A1 shows Desktop/web re-reads are at least 30% of whole-page reads)++- `SuperTabDelivery` gains the index `(tabId, conversationId, updatedAt)`. Every delivery write goes through Prisma's+ `update`, `updateMany`, `create` or `createMany`, which set `@updatedAt` (builder check, Prisma 5.22; the settle pass+ already relies on it).+- `messages` with `after` takes an optional `since` (the `serverTime` of the client's previous read). The reply adds+ `receipts: [{ messageId, deliveries }]` for messages not in `messages` whose delivery rows changed at or after+ `since − 5 s`: each changed message is read by id (`messagesById`), checked with `readable` and `deliveredCopies`, and+ its deliveries filtered with `shownTo`, as on a page. The reply also carries `serverTime`, taken before any read.+- A `since` older than 1 hour answers without `receipts`; the client falls back to its whole-page refresh once.+- **Old backends refuse unknown query fields** (`agentQuery` is `.strict()`, `routes/superTabs.ts:37,74`): Desktop and+ web send `since` only once a reply from this backend carried `serverTime`, and drop it again after a 400.+- Desktop and web (one `controller.dart`): an empty `?after` read no longer reads `?before=<tail+1>`; the reply's+ `receipts` are merged into the rows held. The whole-page read stays for a changed reaction (`reactionSeq`) and the first+ read after a saved page. Web's own policy (`quietPoll` 15 s, no `messagesWhenChanged`) gets the same change.++### Phase C — compact finished deliveries, keep receipts forever++- Only **final** deliveries are compacted: `answered`, `cancelled`, `superseded`. `delivered` and `confirmed` are not+ final (a reply or a `project.report` moves them to `answered`, `store.ts:1553, 1561`) and are never compacted; neither+ are open ones (`queued`, `claimed`, `uncertain`).+- A final delivery whose last update is over **14 days** old is moved into its message's `SuperTabReceipt` document+ (`$set receipts.<agentId>`, so concurrent compactions of one message keep each other's agents) and deleted, in one+ transaction per batch; a delivery that changed since it was read is skipped (filter on `updatedAt`). The compacted+ receipt keeps the page fields (`PageDelivery`) **and** the host fields `deliveryReceipt` authorizes with (`machineId`,+ `machineOwnerId`, `membershipGeneration`, `status`), never shown to readers.+- **Hosts:** `deliveryReceipt` (`store.ts:2749`, the CLI's `checkReceipts`) answers a compacted delivery from its+ receipt, so a host back after weeks offline still retires its journal entries. A host confirming it typed a cancelled,+ held delivery (`typedAsEnded`) reads the compacted receipt the same way.+- **Notices:** `writeMessages(keepStored)` (`store.ts:2048`) counts a compacted receipt as an existing delivery, so a+ re-written notice is never sent again.+- The job runs in the worker once an hour in bounded batches and logs how many it compacted and how many final+ deliveries over 14 days old remain (the warning metric). It is **on by default** (Ken, 2026-10-30), with+ `SWARM_DELIVERY_COMPACTION=off` as the kill switch. During the rolling deploy that ships it, pods still on the old+ release show no receipts for compacted (14-day-old) messages and answer a host's receipt read for one with 404 until+ they are replaced; the CLI retries.++### Phase D — thread summaries in their own collection (Ken, 2026-10-30)++**Why:** a thread is a field today (`SuperTabMessage.thread` = the root's id), and every page runs `pageThreads`+(`pageReads.ts:156`), a `$facet` over all the page's roots, to count replies, their newest and oldest and their mentions,+per visibility class. Prod: 10.4 k threads, **84% are the root alone**, 1,642 have replies (3 messages on average),+~300 replies a day; yet the `$facet` runs on every page, ~100 a pod-minute, 3.4 MB a pod-minute (~20% of the route's+bytes), and with history kept forever it only grows.++**Design:**+- A new collection `SuperTabThread`: one document per thread **with at least one reply**; `_id` = the root id, `tabId`,+ `conversationId`, and the raw result today's `$facet` gives for that one thread (`parts` and `mentions`, keyed by the+ same visibility classes: kind, recipients-only, has recipients, author, Buddy session, root). Index+ `(tabId, conversationId)`. It holds no text and is never returned as is.+- **Writes:** messages are insert-only (`store.ts writeMessages` and `native.ts`; nothing updates or deletes a message).+ When a transaction inserts a reply (`thread` set and not its own id), it recomputes that thread's summary with today's+ `$facet` for that one root and upserts the document, in the same transaction. So a summary always equals what the+ `$facet` would compute; ~300 such writes a day.+- **Reads:** `pageThreads` reads `SuperTabThread` by `_id $in` the page's roots and feeds the stored `parts` and+ `mentions` to today's post-processing (`deliveredCopies` of roots, `agentOnly`, routing rows, Buddy sessions,+ `threadSummaries`), which stays per reader as now. A root with no document has no reply; it yields what today's+ `$facet` yields for a lone root (the builder proves the equality; if summaries drop lone roots, they are skipped).+- **Backfill and switch:** the worker computes the summaries of every existing thread with replies (~1.6 k today, seconds)+ and then records, per Swarm, that it is backfilled and through which time. A Swarm's reads use the collection only once+ its marker exists; until then they run today's `$facet`. At every worker boot the backfill reruns for threads with a+ reply newer than the marker's time minus 1 hour and moves the marker, so replies written by an older release during a+ rollback are counted after roll-forward.++## Left to the builder++- A1's log name and exact keys; A2's margin; reply field names in Phase B (old clients ignore new fields).+- Where the compaction job lives and its batch size.+- How Desktop keeps the last `serverTime` per conversation.++## Tests++- **A1:** one `before` read by an agent and one latest read by a person are counted under their kinds, with batches and+ rows; nothing names an id.+- **A2:** a page whose first 101 rows hold 1 hidden row decodes at most ~101 + margin rows, and returns the same page and+ `next` as today; a page with mostly hidden rows still fills; `catchUp` and threads unchanged.+- **A3:** a page's reply is deep-equal whether a message's deliveries are live, compacted, or half each; a live row wins;+ `shownTo` hides a compacted Buddy-news receipt as it hides a live one; no message read returns a `receipts` map.+- **B:** a delivery answered after the last read comes back in `receipts` on the next empty `?after`; one before does not;+ a hidden message's receipts never appear; Desktop sends `since` only after `serverTime` was seen, drops it after a 400,+ and no longer reads `?before=<tail+1>` after an empty `?after`; a reaction change still reads the page; the web policy+ behaves the same.+- **C:** only `answered`/`cancelled`/`superseded` over 14 days are compacted; two concurrent compactions of one message+ keep both agents; `deliveryReceipt` answers a compacted delivery with its final status and refuses another machine;+ `writeMessages(keepStored)` writes no delivery for a compacted one; a page is unchanged across compaction.+- **D:** for a conversation with lone roots, threads with replies of every visibility class (agent-only notes, route rows,+ a Buddy session thread, mentions), each reader's `threads` from the collection deep-equals today's `$facet` result; a+ reply inserted by `writeMessages` and by `native.ts` updates its thread in the same transaction; a Swarm without its+ marker reads through the `$facet`; the boot rerun counts a reply written while the marker was stale.+- Existing suites stay green: `integration/superTabRead.test.ts`, `superTabReadMarks`, `superTabHotReads`,+ `superTabNamedReads`, the store's delivery tests, the CLI host's receipt tests, Desktop's `super_tabs` tests.++## Measuring++- A1's log, before Phase B: whole-page reads by caller and cursor; rows decoded per row shown.+- The messages route's busy ms and bytes per pod-minute (`cpu by kind`, `db by shape`), before and after each phase.+- Weekly: `SuperTabDelivery` and `SuperTabReceipt` counts and index sizes.++**Done when** Phase A ships with A1's numbers reported and rows decoded per row shown under 1.2; Phase B (if it goes)+cuts Desktop/web whole-page reads by at least 5×; Phase C keeps `SuperTabDelivery` near 14 days of final deliveries plus+the non-final ones; Phase D removes the `$facet` from the messages route in `db by shape` (~100 a pod-minute, 3.4 MB+today).++## Design (AGENTS.md rules)++- **Growth:** messages are kept forever (~5.5 k a day). Final deliveries move into `SuperTabReceipt` after 14 days (one+ small document per message with receipts); delivery rows hold ~14 days plus non-final ones. `SuperTabThread` grows+ with threads that have replies (~150 a day today), small documents, kept with their messages.+- **Concurrency:** compaction writes receipts by field path in one transaction with the delete, skipping rows that+ changed since read. Delta reads take `serverTime` before reading.+- **Read cost:** a page decodes about the rows it shows; it reads live deliveries and compacted receipts for the page's+ messages by id (two indexed reads), and thread summaries by id instead of the `$facet` (one read, ~16% of roots match). An empty poll with Phase B reads the new messages (none) and the conversation's+ changed deliveries by the new index.+- **Scale:** at 10× messages reads stay index-bounded; the first warning is index size against node RAM (`$collStats`+ weekly) and the compaction backlog. Next steps are a larger tier, then a sharding design.+- **Compatibility:** new reply fields only. Old Desktop/web keep their whole-page refresh. New clients send `since` only+ to a backend that answered `serverTime`. Old CLIs read receipts through `deliveryReceipt`, which answers compacted ones.+- **Migration + rollback:** Phase A adds a collection and reads it. Compaction is on by default; before rolling back to a+ release older than Phase A, set `SWARM_DELIVERY_COMPACTION=off` (compacted receipts are then invisible to that old+ release until it is rolled forward again; nothing is lost). The new index comes with the schema push.+- **Audience:** every receipt is filtered per reader as on a page (`readable`, `deliveredCopies`, `shownTo`); compacted+ receipts are never returned whole.+- **Identity:** N/A because nothing here acts as or reads the machine owner's login.+- **Failure:** a failed compaction batch logs and retries next hour; an unusable `since` answers without `receipts` and+ the client falls back; a host asking about a compacted delivery gets its final status, not a 404.+- **Platforms:** backend; Desktop (macOS, Windows, Linux) and the web build for Phase B.