The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
A Swarm message that cannot be delivered says so: Not delivered yet, claims failing alarm, named host log (#2336)
0xdiego2516f2304d68bb
diff --git a/backend/integration/superTabIndexes.test.ts b/backend/integration/superTabIndexes.test.tsindex 957fa2280..6ea081309 100644--- a/backend/integration/superTabIndexes.test.ts+++ b/backend/integration/superTabIndexes.test.ts@@ -178,19 +178,20 @@ describe('message page reads', () => {const rows = await pageDeliveries(db, 'A', ['A-m150'])expect(rows.find(row => row.id === 'A-150-3')).toEqual({ id: 'A-150-3', messageId: 'A-m150', agentId: 'agent-3', status: 'delivered',receipt: 'typed', holdReason: 'busy', holdSince: new Date(T0 + 7000), deliveredAt: new Date(T0 + 8000), answeredAt: new Date(T0 + 9000),- createdAt: new Date(T0 + 150_000), updatedAt: expect.any(Date) })+ createdAt: new Date(T0 + 150_000), updatedAt: expect.any(Date), claimFailingSince: null, claimFailures: null })expect(rows.find(row => row.id === 'A-150-old')).toEqual({ id: 'A-150-old', messageId: 'A-m150', agentId: 'agent-old', status: 'answered',- receipt: null, holdReason: null, holdSince: null, deliveredAt: null, answeredAt: null, createdAt: new Date(T0), updatedAt: new Date(T0 + 1) })+ receipt: null, holdReason: null, holdSince: null, deliveredAt: null, answeredAt: null, createdAt: new Date(T0), updatedAt: new Date(T0 + 1), claimFailingSince: null, claimFailures: null })expect(rows.find(row => row.id === 'A-150-4')).toEqual({ id: 'A-150-4', messageId: 'A-m150', agentId: 'agent-4', status: 'answered',receipt: null, holdReason: null, holdSince: null, deliveredAt: null, answeredAt: null,- createdAt: new Date(T0 + 150_000), updatedAt: new Date(T0 + 150_500) })+ createdAt: new Date(T0 + 150_000), updatedAt: new Date(T0 + 150_500), claimFailingSince: null, claimFailures: null })})// The delivery read never had an order (no orderBy: it followed whichever index the planner took), so it is compared by id.// Reactions keep the tabId+messageId index order, which their counts' `by` lists show.it('returns the same rows as the Prisma reads it replaces', async () => {const select = { id: true, messageId: true, agentId: true, status: true, receipt: true,- holdReason: true, holdSince: true, deliveredAt: true, answeredAt: true, createdAt: true, updatedAt: true }+ holdReason: true, holdSince: true, deliveredAt: true, answeredAt: true, createdAt: true, updatedAt: true,+ claimFailingSince: true, claimFailures: true }const byId = (rows: Array<{ id: string }>) => [...rows].sort((a, b) => a.id.localeCompare(b.id))expect(byId(await pageDeliveries(db, 'A', PAGE))).toEqual(byId(await db.superTabDelivery.findMany({ where: { tabId: 'A', messageId: { in: PAGE } }, select })))expect(await pageReactions(db, 'A', PAGE)).toEqual(await db.superTabReaction.findMany({ where: { tabId: 'A', messageId: { in: PAGE } },diff --git a/backend/integration/superTabTidy.test.ts b/backend/integration/superTabTidy.test.tsindex 13813c05f..b7f00021e 100644--- a/backend/integration/superTabTidy.test.ts+++ b/backend/integration/superTabTidy.test.ts@@ -3,7 +3,7 @@ import { execFile } from 'node:child_process'import { promisify } from 'node:util'import { join } from 'node:path'import { tmpdir } from 'node:os'-import { afterAll, beforeAll, describe, expect, it } from 'vitest'+import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'import { MongoMemoryReplSet } from 'mongodb-memory-server-core'import { PrismaClient } from '@prisma/client'import type { AuthUser } from '../src/lib/ssoAuth.js'@@ -13,6 +13,8 @@ import type { MachineObservation } from '../src/superTabs/machines.js'import { poolRows, rosterRows } from './hostRows.test-support.js'import { archiveFinishedGroups, markFinishedGroups, type PullRequestState } from '../src/superTabs/archive.js'import { settleStuckDeliveries } from '../src/superTabs/deliverySettle.js'+import { SuperTabError } from '../src/superTabs/model.js'+import { logger } from '../src/utils/logger.js'// Nobody tidies a Swarm by hand: a worker idle for 15 minutes is stopped on its Lead's poll and its host is asked to close// it. A fresh loopback replica set, created here.@@ -550,4 +552,126 @@ describe('a delivery that cannot move', () => {expect(await f.timed.settleDeliveries(f.tab, [read], f.clock.now + HOUR + MINUTE)).toBe(1)expect(await db.superTabDelivery.findUniqueOrThrow({ where: { id } })).toMatchObject({ status: 'cancelled', holdReason: 'team_waiting_idle' })})++ // Prod 10-30, v1.3.71 (#2178): every Buddy claim answered SWARM_BUSY for two hours and nothing said so. A claim that fails+ // that way is now written on its delivery, outside the failed transaction (store.ts recordFailedClaim).+ describe('a claim that keeps failing', () => {+ /** Whether the stores from failingStore answer busy, and whether the record write itself is down. Reset before each test. */+ const busy = { on: false, recordDown: false, early: false, recordHangs: false }+ beforeEach(() => { busy.on = false; busy.recordDown = false; busy.early = false; busy.recordHangs = false })+ /** A store over the same database whose transactions answer busy while busy.on is set, as retryTransient ends 8 write+ * conflicts. By default the real body runs first (its authorization checks included) and the answer comes at commit, rolling+ * the body back; with busy.early the answer comes before any body runs. With busy.recordDown its single-document delivery+ * writes throw too; with busy.recordHangs they never answer. */+ const failingStore = (f: Fixture) => {+ const proxy = new Proxy(db, { get: (target, key) => {+ const tooBusy = () => new SuperTabError('SWARM_BUSY', 'This Swarm is busy. Try again in a moment.', 503)+ if (key === '$transaction') return (body: (tx: unknown) => Promise<unknown>, ...rest: unknown[]) => {+ if (!busy.on) return (target.$transaction as (...a: unknown[]) => unknown)(body, ...rest)+ if (busy.early) return Promise.reject(tooBusy())+ return (target.$transaction as (...a: unknown[]) => unknown)(async (tx: unknown) => { await body(tx); throw tooBusy() }, ...rest)+ }+ if (key === 'superTabDelivery' && (busy.recordDown || busy.recordHangs)) return new Proxy(target.superTabDelivery, { get: (model, name) =>+ name === 'updateMany' ? () => busy.recordHangs ? new Promise(() => {}) : Promise.reject(new Error('down')) : Reflect.get(model, name) })+ return Reflect.get(target, key)+ } }) as PrismaClient+ busy.on = true+ return new SuperTabStore(proxy, () => f.clock.now)+ }+ const claim = { type: 'claim', claimId: 'c' } as const+ const claimFields = ['claimFailingSince', 'claimFailedAt', 'claimFailures', 'claimFailure'] as const+ const row = (id: string) => db.superTabDelivery.findUniqueOrThrow({ where: { id } })++ it('records a busy claim on its delivery, keeps when it began and counts each one', async () => {+ const f = await swarm(); const id = await toLead(f, 'Hi')+ const flaky = failingStore(f)+ await expect(flaky.delivery(host, f.tab, f.lead, id, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY' })+ const first = f.clock.now+ f.clock.now += 30_000+ await expect(flaky.delivery(host, f.tab, f.lead, id, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY' })+ expect(await row(id)).toMatchObject({ status: 'queued', claimFailingSince: new Date(first),+ claimFailedAt: new Date(first + 30_000), claimFailures: 2, claimFailure: 'SWARM_BUSY' })+ })+ it('clears the record once a claim gets in', async () => {+ const f = await swarm(); const id = await toLead(f, 'Hi')+ await expect(failingStore(f).delivery(host, f.tab, f.lead, id, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY' })+ expect((await row(id)).claimFailures).toBe(1)+ await f.timed.delivery(host, f.tab, f.lead, id, claim)+ const claimed = await row(id)+ expect(claimed.status).toBe('claimed')+ // Unset, not null: the claim removes the keys, so "missing" stays a real absence that the first failure's isSet: false branch+ // matches (and starts the count at 0, since Prisma's increment of a missing count leaves it null). Prisma reads both as null.+ const [raw] = await db.superTabDelivery.findRaw({ filter: { _id: id } }) as unknown as Array<Record<string, unknown>>+ for (const field of claimFields) { expect(claimed[field], field).toBeNull(); expect(raw, field).not.toHaveProperty(field) }+ })+ it('records nothing for a delivery that is no longer queued, or for an ack', async () => {+ const f = await swarm(); const gone = await toLead(f, 'Gone'); const acked = await toLead(f, 'Acked')+ await db.superTabDelivery.update({ where: { id: gone }, data: { status: 'claimed', claimId: 'c' } })+ const flaky = failingStore(f)+ await expect(flaky.delivery(host, f.tab, f.lead, gone, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY' })+ for (const field of claimFields) expect((await row(gone))[field], field).toBeNull()+ await f.timed.delivery(host, f.tab, f.lead, acked, claim)+ await expect(flaky.delivery(host, f.tab, f.lead, acked, { type: 'ack', claimId: 'c', receipt: 'r', uncertain: false })).rejects.toMatchObject({ code: 'SWARM_BUSY' })+ for (const field of claimFields) expect((await row(acked))[field], field).toBeNull()+ })+ it('records nothing when busy is answered before the request was authorized', async () => {+ const f = await swarm(); const id = await toLead(f, 'Hi')+ const flaky = failingStore(f); busy.early = true+ await expect(flaky.delivery(host, f.tab, f.lead, id, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY' })+ for (const field of claimFields) expect((await row(id))[field], field).toBeNull()+ })+ it('answers busy within about 2 seconds even when the record write never returns', async () => {+ const f = await swarm(); const id = await toLead(f, 'Hi')+ const flaky = failingStore(f); busy.recordHangs = true+ const began = Date.now()+ await expect(flaky.delivery(host, f.tab, f.lead, id, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY', status: 503 })+ expect(Date.now() - began).toBeLessThan(4_000)+ }, 15_000)+ it('still throws the busy answer when the record itself cannot be written, and says so', async () => {+ const f = await swarm(); const id = await toLead(f, 'Hi')+ const flaky = failingStore(f); busy.recordDown = true+ const warn = vi.spyOn(logger, 'warn').mockImplementation(() => {})+ try {+ await expect(flaky.delivery(host, f.tab, f.lead, id, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY', status: 503 })+ expect(warn).toHaveBeenCalledWith('[swarm-deliveries] could not record a failed claim', expect.objectContaining({ tabId: f.tab, deliveryId: id }))+ } finally { warn.mockRestore() }+ })+ it('hands claimFailingSince and claimFailures to page readers', async () => {+ const f = await swarm(); const id = await toLead(f, 'Hi')+ await expect(failingStore(f).delivery(host, f.tab, f.lead, id, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY' })+ const page = await f.timed.messages(owner, f.tab, 'lead', {})+ const shown = page.messages.flatMap(m => m.deliveries).find(d => d.id === id)+ expect(shown).toMatchObject({ claimFailingSince: new Date(f.clock.now), claimFailures: 1 })+ expect(JSON.parse(JSON.stringify(shown))).toMatchObject({ claimFailingSince: new Date(f.clock.now).toISOString(), claimFailures: 1 })+ })+ it('raises claims failing for a delivery whose claims have failed 5 minutes and still fail, and cancels nothing', async () => {+ const f = await swarm(); const id = await toLead(f, 'Hi'); const flaky = failingStore(f)+ const began = f.clock.now+ await expect(flaky.delivery(host, f.tab, f.lead, id, claim)).rejects.toThrow()+ f.clock.now = began + 5 * MINUTE+ await expect(flaky.delivery(host, f.tab, f.lead, id, claim)).rejects.toThrow()+ const warn = vi.spyOn(logger, 'warn').mockImplementation(() => {})+ try {+ await settleStuckDeliveries(db, f.timed, began + 5 * MINUTE + 1_000)+ // Earlier tests leave rows in the shared database; this Swarm's line is the one that counts.+ expect(warn).toHaveBeenCalledWith('[swarm-deliveries] claims failing',+ { tabId: f.tab, deliveries: 1, oldestFailingSince: new Date(began).toISOString(), codes: ['SWARM_BUSY'] })+ expect(await status(id)).toBe('queued')+ } finally { warn.mockRestore() }+ })+ it('says nothing for a claim failing under 5 minutes, or one whose last failure is over 5 minutes old', async () => {+ const f = await swarm(); const young = await toLead(f, 'Young'); const old = await toLead(f, 'Old')+ const now = f.clock.now + 10 * MINUTE+ await db.superTabDelivery.update({ where: { id: young }, data: { claimFailingSince: new Date(now - 4 * MINUTE),+ claimFailedAt: new Date(now - 1_000), claimFailures: 3, claimFailure: 'SWARM_BUSY' } })+ await db.superTabDelivery.update({ where: { id: old }, data: { claimFailingSince: new Date(now - 20 * MINUTE),+ claimFailedAt: new Date(now - 6 * MINUTE), claimFailures: 9, claimFailure: 'SWARM_BUSY' } })+ const warn = vi.spyOn(logger, 'warn').mockImplementation(() => {})+ try {+ await settleStuckDeliveries(db, f.timed, now)+ expect(warn.mock.calls.filter(([line, fields]) => line === '[swarm-deliveries] claims failing' && (fields as { tabId: string }).tabId === f.tab)).toEqual([])+ expect(await status(young)).toBe('queued'); expect(await status(old)).toBe('queued')+ } finally { warn.mockRestore() }+ })+ })})diff --git a/backend/prisma/schema.prisma b/backend/prisma/schema.prismaindex 928dcff75..63c3ffd2f 100644--- a/backend/prisma/schema.prisma+++ b/backend/prisma/schema.prisma@@ -1269,6 +1269,12 @@ model SuperTabDelivery {// Optional on older records: backend first, then Desktop's inline receipt.holdReason String?holdSince DateTime?+ // A queued delivery whose claims keep answering SWARM_BUSY (prod 10-30, v1.3.71): written outside the failed claim, unset by+ // the claim that gets in. The settle pass alarms on them and page reads show the sender (store.ts recordFailedClaim).+ claimFailingSince DateTime?+ claimFailedAt DateTime?+ claimFailures Int?+ claimFailure String?deliveredAt DateTime?answeredAt DateTime?createdAt DateTime @default(now())diff --git a/backend/src/superTabs/deliverySettle.test.ts b/backend/src/superTabs/deliverySettle.test.tsnew file mode 100644index 000000000..72607d248--- /dev/null+++ b/backend/src/superTabs/deliverySettle.test.ts@@ -0,0 +1,26 @@+import { describe, expect, it } from 'vitest'+import { CLAIM_FAILING_ALARM_MS, claimsFailing } from './deliverySettle.js'++const NOW = 10_000_000+const row = (since: number | null, last: number | null, status = 'queued') => ({ status,+ claimFailingSince: since == null ? null : new Date(NOW - since), claimFailedAt: last == null ? null : new Date(NOW - last) }) as Parameters<typeof claimsFailing>[0]++describe('claimsFailing', () => {+ it('is true at exactly 5 minutes since the first failure and a last failure exactly 5 minutes old', () => {+ expect(claimsFailing(row(CLAIM_FAILING_ALARM_MS, CLAIM_FAILING_ALARM_MS), NOW)).toBe(true)+ })+ it('is false at 4:59 since the first failure', () => {+ expect(claimsFailing(row(CLAIM_FAILING_ALARM_MS - 1_000, 1_000), NOW)).toBe(false)+ })+ it('is false when the last failure is 5:01 old', () => {+ expect(claimsFailing(row(20 * 60_000, CLAIM_FAILING_ALARM_MS + 1_000), NOW)).toBe(false)+ })+ it('is false for a claimed delivery', () => {+ expect(claimsFailing(row(CLAIM_FAILING_ALARM_MS, 1_000, 'claimed'), NOW)).toBe(false)+ })+ it('is false with null fields', () => {+ expect(claimsFailing(row(null, null), NOW)).toBe(false)+ expect(claimsFailing(row(CLAIM_FAILING_ALARM_MS, null), NOW)).toBe(false)+ expect(claimsFailing(row(null, 1_000), NOW)).toBe(false)+ })+})diff --git a/backend/src/superTabs/deliverySettle.ts b/backend/src/superTabs/deliverySettle.tsindex 2fa522273..f1b8f2b98 100644--- a/backend/src/superTabs/deliverySettle.ts+++ b/backend/src/superTabs/deliverySettle.ts@@ -17,6 +17,9 @@ export const HOLD_LIMIT_MS = 60 * 60_000export const WAITING_ON_PERSON_LIMIT_MS = 24 * 60 * 60_000/** How long a person has to check a delivery its host could not confirm before it ends, as their Open terminal ends it. */export const UNCERTAIN_LIMIT_MS = 60 * 60_000+/** A queued delivery whose claims have failed this long, and failed again within this long, is stuck behind a Swarm that keeps+ * answering busy (prod 10-30, v1.3.71: every person's messages to their own Buddy sat queued for 2 hours, and nothing said so). */+export const CLAIM_FAILING_ALARM_MS = 5 * 60_000const SETTLE_EVERY_MS = 5 * 60_000const OPEN = ['queued', 'claimed', 'uncertain']const BATCH = 500@@ -31,13 +34,19 @@ export function stuck(delivery: Pick<SuperTabDelivery, 'status' | 'holdReason' |return since(delivery.holdSince) > (delivery.holdReason === 'team_waiting_user' ? WAITING_ON_PERSON_LIMIT_MS : HOLD_LIMIT_MS)}+/** Whether a delivery's claims keep failing: the alarm only logs it, because cancelling would drop a message a healthy host can still take. */+export function claimsFailing(delivery: Pick<SuperTabDelivery, 'status' | 'claimFailingSince' | 'claimFailedAt'>, now: number): boolean {+ if (delivery.status !== 'queued' || !delivery.claimFailingSince || !delivery.claimFailedAt) return false+ return now - delivery.claimFailingSince.getTime() >= CLAIM_FAILING_ALARM_MS && now - delivery.claimFailedAt.getTime() <= CLAIM_FAILING_ALARM_MS+}+/** A page of open deliveries, by id. Read by their own index (status, id) with a plain match: Prisma sends `status in` as an* $expr $or that no index can bound (pageReads.ts), and a pass must not read every delivery ever sent. */async function openDeliveries(db: PrismaClient, after: string | null): Promise<SuperTabDelivery[]> {const rows = await db.superTabDelivery.aggregateRaw({ pipeline: [{ $match: { status: { $in: OPEN }, ...after ? { _id: { $gt: after } } : {} } },{ $sort: { _id: 1 } }, { $limit: BATCH }] }) as unknown as Array<Record<string, unknown>>const date = (value: unknown) => value == null ? null : new Date(ms(value))- return rows.map(({ _id, ...row }) => ({ ...row, id: _id as string, holdSince: date(row.holdSince), deliveredAt: date(row.deliveredAt),+ return rows.map(({ _id, ...row }) => ({ ...row, id: _id as string, holdSince: date(row.holdSince), claimFailingSince: date(row.claimFailingSince), claimFailedAt: date(row.claimFailedAt), deliveredAt: date(row.deliveredAt),answeredAt: date(row.answeredAt), createdAt: date(row.createdAt)!, updatedAt: date(row.updatedAt)! }) as unknown as SuperTabDelivery)}@@ -53,6 +62,16 @@ export async function settleStuckDeliveries(db: PrismaClient, store: Pick<SuperTtry { settled += await store.settleDeliveries(tabId, tabRows, now) } catch (error) {logger.warn('[swarm-deliveries] could not settle', { tabId, error: error instanceof Error ? error.message : String(error) })}+ // Counted per page, so a Swarm whose rows span two 500-row pages logs one line per page. Rare; prod watch counts lines per tab.+ // Its own try/catch: a Swarm whose count throws must not stop the pass for the Swarms after it.+ try {+ const failing = tabRows.filter(row => claimsFailing(row, now))+ if (failing.length) logger.warn('[swarm-deliveries] claims failing', { tabId, deliveries: failing.length,+ oldestFailingSince: new Date(Math.min(...failing.map(row => row.claimFailingSince!.getTime()))).toISOString(),+ codes: [...new Set(failing.map(row => row.claimFailure).filter((code): code is string => !!code))].sort() })+ } catch (error) {+ logger.warn('[swarm-deliveries] could not count failing claims', { tabId, error: error instanceof Error ? error.message : String(error) })+ }}if (rows.length < BATCH) return settledafter = rows.at(-1)!.iddiff --git a/backend/src/superTabs/pageReads.ts b/backend/src/superTabs/pageReads.tsindex 1fb38fa01..7e0ac1f06 100644--- a/backend/src/superTabs/pageReads.ts+++ b/backend/src/superTabs/pageReads.ts@@ -19,17 +19,18 @@ export async function pageReactions(tx: Transaction, tabId: string, messageIds:}export type PageDelivery = Pick<SuperTabDelivery, 'id' | 'messageId' | 'agentId' | 'status' | 'receipt' | 'holdReason' | 'holdSince'- | 'deliveredAt' | 'answeredAt' | 'createdAt' | 'updatedAt'>+ | 'claimFailingSince' | 'claimFailures' | 'deliveredAt' | 'answeredAt' | 'createdAt' | 'updatedAt'>/** The deliveries of one page of messages: who each message went to and how far it got. As Prisma read them: dates as* Dates, and an optional field an older record lacks as null. */export async function pageDeliveries(tx: Transaction, tabId: string, messageIds: string[]): Promise<PageDelivery[]> {const rows = await tx.superTabDelivery.aggregateRaw({ pipeline: [{ $match: { tabId, messageId: { $in: messageIds } } },- { $project: { messageId: 1, agentId: 1, status: 1, receipt: 1, holdReason: 1, holdSince: 1, deliveredAt: 1, answeredAt: 1, createdAt: 1, updatedAt: 1 } }] }) as+ { $project: { messageId: 1, agentId: 1, status: 1, receipt: 1, holdReason: 1, holdSince: 1, claimFailingSince: 1, claimFailures: 1, deliveredAt: 1, answeredAt: 1, createdAt: 1, updatedAt: 1 } }] }) asunknown as Array<Record<string, unknown>>const date = (value: unknown) => value == null ? null : new Date(ms(value))return rows.map(row => ({ id: row._id as string, messageId: row.messageId as string, agentId: row.agentId as string, status: row.status as string,receipt: (row.receipt ?? null) as string | null, holdReason: (row.holdReason ?? null) as string | null, holdSince: date(row.holdSince),+ claimFailingSince: date(row.claimFailingSince), claimFailures: (row.claimFailures ?? null) as number | null,deliveredAt: date(row.deliveredAt), answeredAt: date(row.answeredAt), createdAt: date(row.createdAt)!, updatedAt: date(row.updatedAt)! }))}diff --git a/backend/src/superTabs/store.ts b/backend/src/superTabs/store.tsindex e93192143..332c1cf30 100644--- a/backend/src/superTabs/store.ts+++ b/backend/src/superTabs/store.ts@@ -200,6 +200,8 @@ export async function transaction<T>(db: Pick<PrismaClient, '$transaction'>, bod}}, sleep, error => error === staleIndex)}+/** The longest a failed claim's record may hold up the busy answer. */+const RECORD_FAILED_CLAIM_MS = 2_000const TRANSACTION_OPTIONS = { maxWait: 5_000, timeout: 15_000 }/** Each attempt's after-commit work, by that attempt's client: a retried attempt's list is dropped with its client. */const pendingCommits = new WeakMap<Transaction, Array<() => void>>()@@ -2646,6 +2648,42 @@ export class SuperTabStore {async delivery(user: AuthUser, tabId: string, agentId: string, deliveryId: string,action: { type: 'claim'; claimId: string } | { type: 'ack'; claimId: string; receipt: string; uncertain: boolean }| { type: 'hold'; claimId: string; reason: HoldReason }) {+ const reached = { authorized: false }+ try {+ return await this.deliver(user, tabId, agentId, deliveryId, action, reached)+ } catch (error) {+ // Prod 10-30, v1.3.71: 2 hours of SWARM_BUSY claims left no trace on the delivery. Retries end that way (retry.ts), so it is+ // the only code to record; the client still gets the original error. Only once an attempt got past the authorization+ // checks: a busy answer from before them says nothing about whether this caller may touch the delivery. The write is capped+ // so a struggling database cannot delay the 503 further.+ if (action.type === 'claim' && reached.authorized && error instanceof SuperTabError && error.code === 'SWARM_BUSY') {+ let timer: NodeJS.Timeout | undefined+ const cap = new Promise<void>(resolve => { timer = setTimeout(resolve, RECORD_FAILED_CLAIM_MS); timer.unref() })+ try { await Promise.race([this.recordFailedClaim(tabId, agentId, deliveryId, error.code), cap]) } finally { clearTimeout(timer) }+ }+ throw error+ }+ }++ /** A failed claim, written on its delivery by two single-document writes outside the failed transaction: they touch no Swarm+ * part, so they cannot meet the conflict that failed it. Only a delivery still queued; the first failure's time never moves+ * later. Best effort: a failed write is logged and never replaces the busy answer. */+ private async recordFailedClaim(tabId: string, agentId: string, deliveryId: string, code: string): Promise<void> {+ try {+ const at = new Date(this.now()), mine = { id: deliveryId, tabId, agentId, status: 'queued' }+ // A row an older backend wrote lacks the field, which Prisma on Mongo matches as isSet false, not null. The count starts at 0+ // here: Prisma's increment of a missing count leaves it null.+ await this.db.superTabDelivery.updateMany({ where: { ...mine, OR: [{ claimFailingSince: null }, { claimFailingSince: { isSet: false } }] },+ data: { claimFailingSince: at, claimFailures: 0 } })+ await this.db.superTabDelivery.updateMany({ where: mine, data: { claimFailedAt: at, claimFailure: code, claimFailures: { increment: 1 } } })+ } catch (error) {+ logger.warn('[swarm-deliveries] could not record a failed claim', { tabId, deliveryId, error: error instanceof Error ? error.message : String(error) })+ }+ }++ private async deliver(user: AuthUser, tabId: string, agentId: string, deliveryId: string,+ action: { type: 'claim'; claimId: string } | { type: 'ack'; claimId: string; receipt: string; uncertain: boolean }+ | { type: 'hold'; claimId: string; reason: HoldReason }, reached: { authorized: boolean }) {return transaction(this.db, async tx => {const delivery = await tx.superTabDelivery.findFirst({ where: { id: deliveryId, tabId, agentId } })await this.row(tx, user, tabId)@@ -2671,6 +2709,7 @@ export class SuperTabStore {if (delivery.claimId && delivery.claimId !== action.claimId) fail('CLAIM_CONFLICT', 'Reconcile the existing delivery before claiming it again.')if (action.type !== 'claim' && !delivery.claimId) fail('NOT_CLAIMED', 'Claim this instruction before acknowledging it.')if (action.type === 'hold') return this.hold(tx, delivery, action.reason)+ reached.authorized = trueconst status = action.type === 'claim' ? delivery.status === 'queued' ? 'claimed' : delivery.status : action.uncertain ? 'uncertain' : 'delivered'if (delivery.status === 'delivered' && action.type === 'ack' && (action.uncertain || delivery.receipt !== action.receipt)) {fail('RECEIPT_CONFLICT', 'A delivered instruction keeps its original host receipt.')@@ -2682,7 +2721,11 @@ export class SuperTabStore {const part = parts.members.find(p => p.key === touch.key)if (part) await guardedWrite(tx, tabId, { op: 'touch', ...touch, seen: part.version }, stamp)}+ // A claim that gets in ends the failing record. Unset, never null: a missing field is a real absence, which the first failure's+ // isSet: false branch matches and which leaves no keys in the document.const updated = await tx.superTabDelivery.update({ where: { id: delivery.id }, data: { status, claimId: action.claimId,+ ...(action.type === 'claim' ? { claimFailingSince: { unset: true }, claimFailedAt: { unset: true }, claimFailures: { unset: true },+ claimFailure: { unset: true } } : {}),...(action.type === 'ack' ? { receipt: action.receipt, holdReason: null, holdSince: null,...(!action.uncertain ? { deliveredAt: delivery.deliveredAt ?? new Date(this.now()) } : {}) } : {}) } })const row = await tx.superTabMessage.findUnique({ where: { id: delivery.messageId } })diff --git a/cli/src/superTabs/host.spec.ts b/cli/src/superTabs/host.spec.tsindex a5fa0238c..26904146b 100644--- a/cli/src/superTabs/host.spec.ts+++ b/cli/src/superTabs/host.spec.ts@@ -736,13 +736,14 @@ describe('Super Tab native delivery handoff', () => {describe('a delivery poll or claim that keeps failing', () => {/** The calls the host makes for one failing path, counted as the backend refuses them with `status`. */- function failing(t: ReturnType<typeof setup>, refuse: (method: string, path: string) => boolean, status = 503) {+ function failing(t: ReturnType<typeof setup>, refuse: (method: string, path: string) => boolean, status = 503,+ error: () => { code: string; message: string } = () => ({ code: 'SWARM_BUSY', message: 'This Swarm is busy. Try again in a moment.' })) {const original = vi.mocked(t.core.account.backend).getMockImplementation()!const calls = { n: 0 }vi.mocked(t.core.account.backend).mockImplementation(async (method, path, body) => {if (!refuse(method, path)) return original(method, path, body)calls.n++- return { status, body: { success: false, error: { code: 'SWARM_BUSY', message: 'This Swarm is busy. Try again in a moment.' } } }+ return { status, body: { success: false, error: error() } }})return calls}@@ -771,6 +772,51 @@ describe('a delivery poll or claim that keeps failing', () => {}})+ it('says which delivery keeps failing to go through, once, then at most every 5 minutes', async () => {+ const t = setup(), host = t.open()+ failing(t, (method, path) => method === 'POST' && path.endsWith('/deliveries/delivery'))+ await host.tick()+ const said = () => t.log.mock.calls.map(c => c[0] as string).filter(l => l.includes('delivery delivery') || l.includes('Delivery delivery') || l.includes('This Swarm is busy'))+ expect(said()).toEqual(['Could not hand delivery delivery to agent agent (session harness) in Swarm tab: This Swarm is busy. Try again in a moment. (SWARM_BUSY). '+ + 'Failing since 1970-01-01T00:00:00.000Z, 1 try; next try in 1 s.'])+ for (let i = 0; i < 9; i++) { t.wait(30_000); await host.tick() } // 4.5 min of failing claims+ expect(said()).toHaveLength(1)+ t.wait(30_000); await host.tick() // 5 min since the last line+ expect(said()).toHaveLength(2)+ expect(said()[1]).toMatch(/^Could not hand delivery delivery .*Failing since 1970-01-01T00:00:00.000Z, \d+ tries; next try in 30 s\.$/)+ })++ it('says a different failure at once, and the same one again only after 5 minutes', async () => {+ // Prod 10-30: accept() is more than the claim (ack, downloads, checks), so a new error inside the window must not hide.+ const t = setup(), host = t.open()+ let err = { code: 'SWARM_BUSY', message: 'This Swarm is busy. Try again in a moment.' }+ failing(t, (method, path) => method === 'POST' && path.endsWith('/deliveries/delivery'), 503, () => err)+ await host.tick()+ const said = () => t.log.mock.calls.map(c => c[0] as string).filter(l => l.startsWith('Could not hand delivery delivery'))+ expect(said()).toHaveLength(1)+ err = { code: 'INTERNAL', message: 'The delivery has no message.' }+ t.wait(1_000); await host.tick()+ expect(said()).toHaveLength(2)+ expect(said()[1]).toContain('The delivery has no message. (INTERNAL)')+ t.wait(2_000); await host.tick()+ t.wait(4_000); await host.tick()+ expect(said()).toHaveLength(2)+ })++ it('says once when a delivery that had failed goes through', async () => {+ const t = setup(), host = t.open()+ let refuse = true+ failing(t, (method, path) => refuse && method === 'POST' && path.endsWith('/deliveries/delivery'))+ await host.tick()+ t.wait(1_000); await host.tick()+ refuse = false+ t.wait(2_000); await host.tick()+ const claimed = () => t.log.mock.calls.map(c => c[0] as string).filter(l => l.startsWith('Delivery delivery went through'))+ expect(claimed()).toEqual(['Delivery delivery went through after 3 tries, 0 min.'])+ t.wait(1_000); await host.tick()+ expect(claimed()).toHaveLength(1)+ })+it('claims at once, on the next second, after the backend answers again', async () => {const t = setup(), host = t.open()let refuse = truediff --git a/cli/src/superTabs/host.ts b/cli/src/superTabs/host.tsindex b3b96ac6a..fad3b58c8 100644--- a/cli/src/superTabs/host.ts+++ b/cli/src/superTabs/host.ts@@ -180,6 +180,9 @@ const JOIN_FINAL = new Set(['MACHINES_FULL', 'OWNER_REQUIRED'])* added to an overloaded backend (production at 5 pods, 2026-10-27). */const DELIVERY_RETRY_MS = 1_000const DELIVERY_RETRY_MAX_MS = 30_000+/** Prod 10-30: a claim refused every 30 s said the backend's bare words 223 times in 2 hours. The same failure of a delivery is+ * named at most this often. */+const CLAIM_FAILURE_LOG_MS = 5 * 60_000/** Stability roadmap 2.4 (2026-10-28): every host read each running agent's delivery list each second, almost always to find* nothing. A server that says it pushes (a `super_tab_deliveries` notice over the daemon socket, backend lib/adapterWs.ts)* names the agent when work waits; an idle agent's list is then read only this often, as the safety net for a lost push.@@ -279,7 +282,7 @@ export class SuperTabHost {private nextReplacementsAt = 0private timer: ReturnType<typeof setTimeout> | null = null/** Per agent's delivery list and per delivery's claim, the failures in a row and when it may be asked again. */- private readonly deliveryRetries = new Map<string, { failures: number; retryAt: number; since: number; list?: string }>()+ private readonly deliveryRetries = new Map<string, { failures: number; retryAt: number; since: number; saidAt?: number; said?: string; list?: string }>()private native: NativeRecorder | null = nullprivate machines: ContributedMachines | null = nullprivate readonly machineDeps: Partial<MachineDeps>@@ -757,7 +760,7 @@ export class SuperTabHost {for (const id of last.ids) seen.add(id)continue}- let listRead = false+ let listRead = false, claimError: unknowntry {const deliveries = await backend(this.core, 'GET', path)if (this.stopped) return@@ -778,10 +781,30 @@ export class SuperTabHost {// claimed one with no journal is never typed, only reported as uncertain; a delivered one only acknowledged.const entry = this.journal.entries[delivery.id]if ((entry ? entry.stage === 'prepared' : !delivery.claimId) && !this.mayRead(target, delivery)) continue- try { await this.accept(target, delivery, path); this.deliveryRetries.delete(claimKey) } catch (error) {+ try {+ await this.accept(target, delivery, path)+ const failed = this.deliveryRetries.get(claimKey)+ this.deliveryRetries.delete(claimKey)+ if (failed && failed.failures > 0) this.log(`Delivery ${delivery.id} went through after ${failed.failures + 1} tries, ${Math.round((this.now() - failed.since) / 60_000)} min.`)+ } catch (error) {// Answered (or resolved by a person) between the roster read and this call: nothing is owed, and the// agent's other deliveries go on.- if (!(error instanceof SuperTabHostError) || error.code !== 'DELIVERY_RESOLVED') { this.retryLater(claimKey, listKey); throw error }+ if (!(error instanceof SuperTabHostError) || error.code !== 'DELIVERY_RESOLVED') {+ // Prod 10-30: the bare message below named no delivery or agent. Named here, and only now and then: this retries for hours.+ // accept() is more than the claim (ack POST, downloads, message and journal checks; ack failures are a known open risk),+ // so the line names no step, and a different error than the last one said is told at once, not hidden by the window.+ const retry = this.retryLater(claimKey, listKey), now = this.now(), code = error instanceof SuperTabHostError ? ` (${error.code})` : ''+ const what = `${error instanceof Error ? error.message : String(error)}${code}`+ if (retry.saidAt === undefined || retry.said !== what || now - retry.saidAt >= CLAIM_FAILURE_LOG_MS) {+ retry.saidAt = now+ retry.said = what+ this.log(`Could not hand delivery ${delivery.id} to agent ${target.agent.id} (session ${target.agent.harnessId.slice(0, 8)}) in Swarm ${target.tabId}: `+ + `${what}. Failing since ${new Date(retry.since).toISOString()}, ${retry.failures} ${retry.failures === 1 ? 'try' : 'tries'}; `+ + `next try in ${Math.round((retry.retryAt - now) / 1_000)} s.`)+ }+ claimError = error+ throw error+ }this.deliveryRetries.delete(claimKey)this.retire(delivery.id as string)}@@ -790,7 +813,7 @@ export class SuperTabHost {} catch (error) {// Authority checks and network failures both withhold input. A later fresh read may grant it again.const message = error instanceof Error ? error.message : String(error)- if (listRead) this.log(message)+ if (listRead) { if (error !== claimError) this.log(message) }else {// 2026-10-28: a Lead's list failed for 45 minutes while this log said only the backend's words, with no agent.const retry = this.retryLater(listKey), code = error instanceof SuperTabHostError ? ` (${error.code})` : ''diff --git a/desktop/lib/super_tabs/message_receipt.dart b/desktop/lib/super_tabs/message_receipt.dartindex 34d1e6838..927c1d6e1 100644--- a/desktop/lib/super_tabs/message_receipt.dart+++ b/desktop/lib/super_tabs/message_receipt.dart@@ -1,5 +1,6 @@import 'dart:async';+import 'package:clock/clock.dart';import 'package:flutter/material.dart';import '../core/models.dart';@@ -11,6 +12,12 @@ import 'controller.dart';import 'machine_status.dart';import 'model.dart';+/// How long a queued delivery's claims fail before its sender is told, and how+/// long a queued message waits with no claim fields before the same (prod 10-30,+/// v1.3.71: claims failed for two hours and nothing showed).+const superTabClaimFailingNotice = Duration(seconds: 60);+const superTabQueuedNotice = Duration(minutes: 2);+/// The line under the chat's newest own message, shown only while it sends,/// when it was not sent, or when its delivery needs a hand; a message that/// arrived carries none, as in Slack.@@ -23,7 +30,7 @@ class SuperTabMessageReceipt extends StatefulWidget {required this.onAddLead,required this.machineName,this.app,- this.now = DateTime.now,+ this.now,});final SuperTabMessage message;final SuperTabController controller;@@ -31,7 +38,9 @@ class SuperTabMessageReceipt extends StatefulWidget {final void Function(SuperTabDelivery delivery, String resolution) onResolve;final VoidCallback onAddLead;final String Function(String machineId) machineName;- final DateTime Function() now;++ /// Defaults to the package clock, which tests move (a const default cannot).+ final DateTime Function()? now;@overrideState<SuperTabMessageReceipt> createState() => _SuperTabMessageReceiptState();@@ -74,7 +83,7 @@ class _SuperTabMessageReceiptState extends State<SuperTabMessageReceipt>if (mounted) setState(() {});}- void _clock(bool needed) {+ void _clock(bool needed, [Duration? until]) {_timer?.cancel();final lifecycle = WidgetsBinding.instance.lifecycleState;if (!needed ||@@ -82,7 +91,9 @@ class _SuperTabMessageReceiptState extends State<SuperTabMessageReceipt>(lifecycle != null && lifecycle != AppLifecycleState.resumed)) {return;}- _timer = Timer(const Duration(milliseconds: 500), _changed);+ // A notice threshold is one wake-up at its time, not a tick every 500 ms.+ final wait = until ?? const Duration(milliseconds: 500);+ _timer = Timer(wait.isNegative ? Duration.zero : wait, _changed);}@override@@ -101,7 +112,7 @@ class _SuperTabMessageReceiptState extends State<SuperTabMessageReceipt>final name =c.snapshot?.agentName(delivery?.agentId ?? '', fallback: 'Lead') ??'Lead';- final now = widget.now();+ final now = widget.now?.call() ?? clock.now();final machine = c.snapshot?.machines.where((m) => m['machineId'] == agent?.machineId).firstOrNull;@@ -129,6 +140,32 @@ class _SuperTabMessageReceiptState extends State<SuperTabMessageReceipt>delivery?.pending == true &&message.createdAt != null &&now.difference(message.createdAt!) < const Duration(seconds: 3);+ // A queued delivery to a running agent on an online, readable Swarm that is+ // not getting in: its claims fail, or no claim fields came (an older backend)+ // and it just sits. Either reason shows the line, so it never drops out+ // when the other one starts.+ final queued =+ delivery?.status == 'queued' &&+ agent != null &&+ agent.downSince == null &&+ !offline &&+ !c.backendUnreachable;+ final failingSince = delivery?.claimFailingSince;+ final failingAt = failingSince?.add(superTabClaimFailingNotice);+ final queuedAt = message.createdAt?.add(superTabQueuedNotice);+ final stuck =+ queued &&+ ((failingAt != null && !failingAt.isAfter(now)) ||+ (queuedAt != null && !queuedAt.isAfter(now)));+ // Until a threshold is crossed the receipt must tick by itself: no snapshot+ // arrives to rebuild it. The earlier of the two is the next to cross.+ final upcoming = [+ failingAt,+ queuedAt,+ ].whereType<DateTime>().where((t) => t.isAfter(now));+ final nextAt = queued && !stuck && upcoming.isNotEmpty+ ? upcoming.reduce((a, b) => a.isBefore(b) ? a : b)+ : null;final error = Theme.of(context).colorScheme.error;var color = error;var icon = AppIcons.triangleAlert;@@ -175,12 +212,24 @@ class _SuperTabMessageReceiptState extends State<SuperTabMessageReceipt>note = readingAt == null? "$host is offline. Your message goes in when it's back online.": "$host was last seen ${superTabAgo(now.difference(readingAt))}. Your message goes in when it's back online.";+ } else if (stuck) {+ // Queued to an online computer with no line read as delivered (prod 10-30).+ // Nothing to do for the sender: Send again would queue behind the same failure.+ color = grid.AppPalette.warn;+ icon = AppIcons.clock;+ title = 'Not delivered yet';+ final host =+ machine?['name'] as String? ?? widget.machineName(agent.machineId);+ note = failingSince != null+ ? "$host is online but couldn't take this message yet. Harness keeps trying, and it goes in as soon as it can."+ : "$host is online but hasn't picked this message up yet. Harness keeps trying, and it goes in as soon as it can.";} else if (fresh) {color = DesktopChrome.muted;title = 'Sending…';sending = true;}- _clock(fresh || (delivery?.pending == true && offline));+ final ticking = fresh || (delivery?.pending == true && offline);+ _clock(ticking || nextAt != null, ticking ? null : nextAt?.difference(now));if (title == null) return const SizedBox.shrink();final style = superTabReceiptStyle(color);final parts = <Widget>[diff --git a/desktop/lib/super_tabs/model.dart b/desktop/lib/super_tabs/model.dartindex 5dacaaf1c..1b8890c26 100644--- a/desktop/lib/super_tabs/model.dart+++ b/desktop/lib/super_tabs/model.dart@@ -858,11 +858,18 @@ class SuperTabDelivery {status = row['status'] as String? ?? '',holdReason = row['holdReason'] as String?,holdSince = _date(row['holdSince']),+ claimFailingSince = _date(row['claimFailingSince']),+ claimFailures = (row['claimFailures'] as num?)?.toInt(),deliveredAt = _date(row['deliveredAt']),answeredAt = _date(row['answeredAt']),updatedAt = _date(row['updatedAt']);final String? id, holdReason;final String agentId, status;++ /// Since when the backend's claims of this delivery fail, and how many have+ /// (prod 10-30, v1.3.71). Null from an older backend, and once a claim works.+ final DateTime? claimFailingSince;+ final int? claimFailures;final DateTime? holdSince, deliveredAt, answeredAt, updatedAt;static DateTime? _date(Object? value) =>value is String ? DateTime.tryParse(value) : null;diff --git a/desktop/test/super_tab_receipt_test.dart b/desktop/test/super_tab_receipt_test.dartindex 2bf8c6a42..8b477ae3c 100644--- a/desktop/test/super_tab_receipt_test.dart+++ b/desktop/test/super_tab_receipt_test.dart@@ -1,5 +1,6 @@import 'dart:io';+import 'package:clock/clock.dart';import 'package:flutter/material.dart';import 'package:flutter_test/flutter_test.dart';import 'package:harness/shared/theme/app_theme.dart' as grid;@@ -16,7 +17,8 @@ class ReceiptFixture {final WidgetTester tester;final String? status;final server = ConversationServer();- final now = DateTime.now();+ // The receipt reads the package clock, which a test moves; real time would drift from it.+ final now = clock.now();late final SuperTabController controller;String? opened;bool addedLead = false;@@ -377,4 +379,169 @@ void main() {await f.close();},);++ // Not delivered yet: prod 10-30 (v1.3.71), claims failed for two hours while+ // every queued message looked delivered.+ const couldntTake =+ "Team Mac is online but couldn't take this message yet. Harness keeps trying, and it goes in as soon as it can.";+ const notPickedUp =+ "Team Mac is online but hasn't picked this message up yet. Harness keeps trying, and it goes in as soon as it can.";++ testWidgets(+ 'receipt: a queued message whose claims fail for a minute reads Not delivered yet',+ (tester) async {+ final f = ReceiptFixture(tester, status: 'queued');+ await f.show(+ prepare: () => f.delivery['claimFailingSince'] = f.now+ .subtract(const Duration(seconds: 61))+ .toIso8601String(),+ );+ expect(find.text('Not delivered yet'), findsOneWidget);+ expect(find.text(couldntTake), findsOneWidget);+ expect(find.text('Send again'), findsNothing);+ await f.close();+ },+ );++ testWidgets(+ 'receipt: a queued message no computer picked up for 2 minutes reads Not delivered yet',+ (tester) async {+ final f = ReceiptFixture(tester, status: 'queued');+ await f.show(+ prepare: () => f.message['createdAt'] = f.now+ .subtract(const Duration(minutes: 2, seconds: 1))+ .toIso8601String(),+ );+ expect(find.text('Not delivered yet'), findsOneWidget);+ expect(find.text(notPickedUp), findsOneWidget);+ await f.close();+ },+ );++ testWidgets('receipt: offline still wins over Not delivered yet', (+ tester,+ ) async {+ final f = ReceiptFixture(tester, status: 'queued');+ await f.show(+ prepare: () {+ f.delivery['claimFailingSince'] = f.now+ .subtract(const Duration(minutes: 5))+ .toIso8601String();+ (f.server.state['machines'] as List)+ .first['observation']['receivedAt'] = f.now+ .subtract(const Duration(minutes: 3))+ .millisecondsSinceEpoch;+ },+ );+ expect(find.text("Lead's computer is offline"), findsOneWidget);+ expect(find.text('Not delivered yet'), findsNothing);+ await f.close();+ });++ testWidgets(+ 'receipt: a stopped recipient that has been queued for minutes shows neither case',+ (tester) async {+ final f = ReceiptFixture(tester, status: 'queued');+ await f.show(+ prepare: () {+ (f.server.state['agents'] as List).first['downSince'] = f.now+ .subtract(const Duration(minutes: 10))+ .toIso8601String();+ f.message['createdAt'] = f.now+ .subtract(const Duration(minutes: 3))+ .toIso8601String();+ },+ );+ expect(find.text('Not delivered yet'), findsNothing);+ await f.close();+ },+ );++ testWidgets(+ 'receipt: a stopped recipient with failing claims shows no Not delivered yet',+ (tester) async {+ final f = ReceiptFixture(tester, status: 'queued');+ await f.show(+ prepare: () {+ (f.server.state['agents'] as List).first['downSince'] = f.now+ .subtract(const Duration(minutes: 10))+ .toIso8601String();+ f.delivery['claimFailingSince'] = f.now+ .subtract(const Duration(minutes: 5))+ .toIso8601String();+ },+ );+ expect(find.text('Not delivered yet'), findsNothing);+ await f.close();+ },+ );++ testWidgets(+ 'receipt: a delivery to an agent not in the Swarm shows nothing and does not throw',+ (tester) async {+ final f = ReceiptFixture(tester, status: 'queued');+ await f.show(+ prepare: () {+ f.delivery['agentId'] = 'ghost-agent';+ f.message['createdAt'] = f.now+ .subtract(const Duration(minutes: 3))+ .toIso8601String();+ },+ );+ expect(find.text('Not delivered yet'), findsNothing);+ await f.close();+ },+ );++ testWidgets(+ 'receipt: a message queued for minutes keeps its line when the first claim failure arrives',+ (tester) async {+ final f = ReceiptFixture(tester, status: 'queued');+ await f.show(+ prepare: () {+ f.message['createdAt'] = f.now+ .subtract(const Duration(minutes: 5))+ .toIso8601String();+ f.delivery['claimFailingSince'] = f.now+ .subtract(const Duration(seconds: 10))+ .toIso8601String();+ },+ );+ expect(find.text('Not delivered yet'), findsOneWidget);+ expect(find.text(couldntTake), findsOneWidget);+ await f.close();+ },+ );++ testWidgets(+ 'receipt: a claimed message with leftover claim fields stays quiet',+ (tester) async {+ final f = ReceiptFixture(tester);+ await f.show(+ prepare: () => f.delivery['claimFailingSince'] = f.now+ .subtract(const Duration(minutes: 5))+ .toIso8601String(),+ );+ f.expectQuiet();+ expect(find.text('Not delivered yet'), findsNothing);+ await f.close();+ },+ );++ testWidgets(+ 'receipt: below both thresholds nothing shows, and the line comes by the receipt\'s own clock',+ (tester) async {+ final f = ReceiptFixture(tester, status: 'queued');+ await f.show(+ prepare: () => f.message['createdAt'] = clock+ .now()+ .subtract(const Duration(minutes: 1, seconds: 58))+ .toIso8601String(),+ );+ expect(find.text('Not delivered yet'), findsNothing);+ await tester.pump(const Duration(seconds: 3));+ expect(find.text('Not delivered yet'), findsOneWidget);+ await f.close();+ },+ );}diff --git a/docs/superpowers/plans/2026-10-30-swarm-stuck-delivery-visible.md b/docs/superpowers/plans/2026-10-30-swarm-stuck-delivery-visible.mdnew file mode 100644index 000000000..ca83a4675--- /dev/null+++ b/docs/superpowers/plans/2026-10-30-swarm-stuck-delivery-visible.md@@ -0,0 +1,459 @@+# A Swarm message that cannot be delivered says so: Implementation Plan++> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.++**Goal:** When a Swarm delivery keeps failing to be claimed, three things change:++- the sender sees **Not delivered yet**;+- prod watch gets a `claims failing` alarm line;+- the host's log names the claim.++**Architecture:** The backend records each `SWARM_BUSY` claim failure on its `SuperTabDelivery` row, with standalone+writes outside the failed transaction. A successful claim clears the record. The delivery settle pass logs an alarm+for Swarms whose queued deliveries keep failing, and cancels none of them. Page reads carry `claimFailingSince` and+`claimFailures`. The Flutter receipt (Desktop and web) turns those fields, or a queued message's age, into an amber line.+The CLI host replaces its bare busy line with a named, rate-limited one.++**Tech Stack:**+- Backend: TypeScript, Fastify, Prisma on MongoDB, vitest with mongodb-memory-server.+- Desktop: Flutter/Dart, flutter_test, package:clock.+- CLI: TypeScript, vitest.++**Spec:** `docs/superpowers/specs/2026-10-30-swarm-stuck-delivery-visible-design.md`++## Global Constraints++- Work only in the worktree `.claude/worktrees/stuckdelivery` (branch `feat/stuck-delivery-visible`). Never touch the+ main checkout.+- Commit only when Diego asks. The commit steps below are prepared commands: run them only once asked. Stage only this+ plan's files.+- The fresh worktree has no `node_modules`. Run `npm ci` once in `backend/` and in `cli/` before the first test.+ Run `npx prisma generate` in `backend/` after the schema change.+- Backend integration test command prefix (`B`):+ `cd backend && NODE_ENV=test HARNESS_CREDENTIAL_ENCRYPTION_KEY=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= npx vitest run --config vitest.super-tabs.config.ts integration/superTabTidy.test.ts`+- No message text, agent names or person ids in any new log line.+- The client's answer to a failed claim does not change: still 503 `SWARM_BUSY`.+- Thresholds, copied from the spec:+ - Desktop: claim failing for **60 s** (`superTabClaimFailingNotice`); queued for **2 min** (`superTabQueuedNotice`).+ - Backend alarm: `CLAIM_FAILING_ALARM_MS` = **5 min**.+ - CLI: logs at most once every **5 min** per delivery (`CLAIM_FAILURE_LOG_MS`).+- Copy, exact text:+ - title `Not delivered yet`;+ - note `$host is online but couldn't take this message yet. Harness keeps trying, and it goes in as soon as it can.`;+ - note `$host is online but hasn't picked this message up yet. Harness keeps trying, and it goes in as soon as it can.`+- Comments say why, naming the incident ("prod 10-30, v1.3.71"), in the surrounding files' density.++## Review Focus++1. A claim failing on a delivery that is no longer `queued` (revoked or cancelled while in flight) must write nothing.+ The record's filter includes `status: 'queued'`. Test in Task 1.+2. A failing `ack` or `hold` records nothing. Only claims do. Test in Task 1.+3. A delivery that left `queued` but still carries leftover fields shows no line in the receipt. Test in Task 3.+4. The alarm pass must keep settling the other Swarms when one Swarm's alarm count throws, and must cancel no failing+ row. Test in Task 2.+5. The receipt's own clock makes the line appear without a new snapshot, using `clock.now` so tests can drive it.+ Test in Task 3.++---++### Task 1: Backend records a failed claim on its delivery, and page reads carry it++**Files:**+- Modify: `backend/prisma/schema.prisma` (model `SuperTabDelivery`, ~line 1249)+- Modify: `backend/src/superTabs/store.ts` (`delivery()`, ~line 2606)+- Modify: `backend/src/superTabs/pageReads.ts` (`PageDelivery`, `pageDeliveries`, ~lines 21-34)+- Test: `backend/integration/superTabTidy.test.ts`. Add a `describe('a claim that keeps failing', …)` nested inside+ `describe('a delivery that cannot move', …)`, so it reuses `toLead`, `status` and `HOUR`.++**Interfaces:**+- Produces these fields on `SuperTabDelivery`, all optional: `claimFailingSince DateTime?`, `claimFailedAt DateTime?`,+ `claimFailures Int?`, `claimFailure String?`.+- Produces `PageDelivery` with `claimFailingSince: Date | null` and `claimFailures: number | null`.+- Produces `private async recordFailedClaim(tabId: string, agentId: string, deliveryId: string, code: string): Promise<void>`+ on `SuperTabStore`.++- [ ] **Step 1: Write the failing tests**++All of them use one helper inside the new describe. `failingStore(f)` returns+`new SuperTabStore(proxy, () => f.clock.now)`, where `proxy` is a `Proxy` over `db`:++- its `$transaction` throws `new SuperTabError('SWARM_BUSY', 'This Swarm is busy. Try again in a moment.', 503)` while+ `busy.on` is true;+- every other property passes through to `db`.++Import `SuperTabError` from `../src/superTabs/model.js`.++```ts+it('records a busy claim on its delivery, keeps when it began and counts each one', async () => {+ const f = await swarm(); const id = await toLead(f, 'Hi')+ const flaky = failingStore(f)+ await expect(flaky.delivery(host, f.tab, f.lead, id, { type: 'claim', claimId: 'c' })).rejects.toMatchObject({ code: 'SWARM_BUSY' })+ const first = f.clock.now+ f.clock.now += 30_000+ await expect(flaky.delivery(host, f.tab, f.lead, id, { type: 'claim', claimId: 'c' })).rejects.toMatchObject({ code: 'SWARM_BUSY' })+ expect(await db.superTabDelivery.findUniqueOrThrow({ where: { id } })).toMatchObject({ status: 'queued',+ claimFailingSince: new Date(first), claimFailedAt: new Date(first + 30_000), claimFailures: 2, claimFailure: 'SWARM_BUSY' })+})+it('clears the record once a claim gets in', async () => { /* fail once with flaky, then f.timed.delivery(... claim 'c') succeeds;+ the row has status 'claimed' and claimFailingSince, claimFailedAt, claimFailures, claimFailure all undefined (unset, not null) */ })+it('records nothing for a delivery that is no longer queued, or for an ack', async () => { /* (a) cancel the row with+ db.superTabDelivery.update({ status: 'cancelled' }), flaky claim rejects, no claim* fields; (b) a real claim, then a flaky ack+ rejects SWARM_BUSY, no claim* fields */ })+it('still throws the busy answer when the record itself cannot be written, and says so', async () => { /* proxy also makes+ superTabDelivery.updateMany throw new Error('down'); delivery() rejects with code SWARM_BUSY;+ vi.spyOn(logger, 'warn') was called with '[swarm-deliveries] could not record a failed claim', expect.objectContaining({ tabId: f.tab, deliveryId: id }) */ })+it('hands claimFailingSince and claimFailures to page readers', async () => { /* after one flaky failure, f.timed.read(owner, f.tab) page+ of 'lead' has the message's delivery with claimFailingSince equal to that time and claimFailures 1 (as the page serialises dates) */ })+```++For the last test, find how this file's other tests read a page's deliveries. If none does, use `pageDeliveries(tx, …)`+inside `db.$transaction`, imported from `../src/superTabs/pageReads.js`.++- [ ] **Step 2: Run the tests to verify they fail**++Run: `B -t "a claim that keeps failing"`+Expected: FAIL. TypeScript or Prisma reject the unknown `claimFailingSince` fields, or the assertions find them+undefined.++- [ ] **Step 3: Add the four optional fields to `SuperTabDelivery` in `schema.prisma`, then run `npx prisma generate`**++Put a one-line comment above them naming prod 10-30 and the settle pass alarm. Add no index.++- [ ] **Step 4: Implement recording and clearing in `store.ts`**++In `delivery()`:++1. Wrap the `return transaction(...)` in a `try`/`catch`. When `action.type === 'claim'` and the error is a+ `SuperTabError` with code `SWARM_BUSY`, `await this.recordFailedClaim(...)`. Then rethrow the original error.+2. In the successful claim's existing `tx.superTabDelivery.update`, add `{ unset: true }` for all four fields, only when+ `action.type === 'claim'`.++`recordFailedClaim` makes two writes. Both match `{ id: deliveryId, tabId, agentId, status: 'queued' }`:++1. `this.db.superTabDelivery.updateMany`, with `OR: [{ claimFailingSince: null }, { claimFailingSince: { isSet: false } }]`+ added to the filter, setting `claimFailingSince: new Date(this.now())`.+2. `this.db.superTabDelivery.updateMany` setting `claimFailedAt: new Date(this.now())` and `claimFailure: code`, with+ `claimFailures: { increment: 1 }`.++Catch any error from either write and log+`logger.warn('[swarm-deliveries] could not record a failed claim', { tabId, deliveryId, error })`. Never let it+replace the thrown error.++- [ ] **Step 5: Project the two fields in `pageReads.ts`**++Add `claimFailingSince` and `claimFailures` to the `PageDelivery` pick and to the `$project`. Map them as+`date(row.claimFailingSince)` and `(row.claimFailures ?? null) as number | null`.++Then find every place a page delivery is turned into JSON for clients, by grepping `pageDeliveries(` and `holdSince` in+`backend/src/superTabs`. Make sure the two fields reach the response the same way `holdSince` does. If a serialiser+lists fields explicitly, add them there.++- [ ] **Step 6: Run the tests and the typecheck**++Run: `B -t "a claim that keeps failing"`, then `cd backend && npm run typecheck`+Expected: 5 passed; tsc exits 0.++Then run the whole file, `B`. Expected: every test passes. The existing settle tests must be unchanged.++- [ ] **Step 7: Commit (only once Diego asks)**++```bash+git add backend/prisma/schema.prisma backend/src/superTabs/store.ts backend/src/superTabs/pageReads.ts backend/integration/superTabTidy.test.ts+git commit -m "fix(swarm): a claim that keeps failing is recorded on its delivery"+```++---++### Task 2: The settle pass logs `claims failing`++**Files:**+- Modify: `backend/src/superTabs/deliverySettle.ts`+- Test: `backend/integration/superTabTidy.test.ts`, in the same `describe('a claim that keeps failing', …)`++**Interfaces:**+- Consumes the Task 1 fields on the rows `openDeliveries` returns.+- Produces `export const CLAIM_FAILING_ALARM_MS = 5 * 60_000`.+- Produces `export function claimsFailing(delivery: Pick<SuperTabDelivery, 'status' | 'claimFailingSince' | 'claimFailedAt'>, now: number): boolean`.+ It is true when all of these hold: `status === 'queued'`, `now - claimFailingSince >= CLAIM_FAILING_ALARM_MS`, and+ `now - claimFailedAt <= CLAIM_FAILING_ALARM_MS`.++- [ ] **Step 1: Write the failing tests**++```ts+it('raises claims failing for a delivery whose claims have failed 5 minutes and still fail, and cancels nothing', async () => {+ const f = await swarm(); const id = await toLead(f, 'Hi'); const flaky = failingStore(f)+ const began = f.clock.now+ await expect(flaky.delivery(host, f.tab, f.lead, id, { type: 'claim', claimId: 'c' })).rejects.toThrow()+ f.clock.now = began + 5 * MINUTE+ await expect(flaky.delivery(host, f.tab, f.lead, id, { type: 'claim', claimId: 'c' })).rejects.toThrow()+ const warn = vi.spyOn(logger, 'warn')+ await settleStuckDeliveries(db, f.timed, began + 5 * MINUTE + 1_000)+ expect(warn).toHaveBeenCalledWith('[swarm-deliveries] claims failing',+ { tabId: f.tab, deliveries: 1, oldestFailingSince: new Date(began).toISOString(), codes: ['SWARM_BUSY'] })+ expect(await status(id)).toBe('queued')+ warn.mockRestore()+})+it('says nothing for a claim failing under 5 minutes, or one whose last failure is over 5 minutes old', async () => { /* two+ deliveries: one failing 4 min; one whose claimFailedAt is 6 min before the pass (set the fields directly with db.update).+ No 'claims failing' warn; both stay queued */ })+```++The pure `claimsFailing` boundaries also get a unit test, in a new `backend/src/superTabs/deliverySettle.test.ts`:++- true at exactly 5 min since and 5 min since the last failure;+- false at 4:59 since;+- false at a last failure 5:01 old;+- false for `claimed`;+- false with null fields.++- [ ] **Step 2: Run the tests to verify they fail**++Run: `B -t "claims failing|says nothing for a claim"`, then+`cd backend && NODE_ENV=test HARNESS_CREDENTIAL_ENCRYPTION_KEY=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= npx vitest run src/superTabs/deliverySettle.test.ts`+Expected: FAIL. `claimsFailing` is not exported, and no warn is logged.++- [ ] **Step 3: Implement it in `deliverySettle.ts`**++1. Map `claimFailingSince` and `claimFailedAt` to Dates in `openDeliveries`, as `holdSince` is.+2. In `settleStuckDeliveries`, per Swarm, **after** its `store.settleDeliveries` try/catch, count+ `tabRows.filter(row => claimsFailing(row, now))`.+3. When the count is non-zero, log `logger.warn('[swarm-deliveries] claims failing', { tabId, deliveries,+ oldestFailingSince, codes })`. Here `oldestFailingSince` is the earliest `claimFailingSince` as ISO text, and `codes`+ is the sorted distinct `claimFailure` values.+4. The count and the log sit inside their own try/catch, which logs `[swarm-deliveries] could not count failing claims`,+ so one Swarm never stops the pass.++`stuck()` is unchanged.++A Swarm whose rows span two 500-row pages logs one line per page. Accept that, and say so in a comment. It is rare, and+prod watch counts lines per tab.++- [ ] **Step 4: Run the tests to verify they pass**++Run: the two commands from Step 2, then `B`+Expected: all pass.++- [ ] **Step 5: Commit (only once Diego asks)**++```bash+git add backend/src/superTabs/deliverySettle.ts backend/src/superTabs/deliverySettle.test.ts backend/integration/superTabTidy.test.ts+git commit -m "feat(swarm): the delivery pass raises claims failing for prod watch"+```++---++### Task 3: Desktop and web show Not delivered yet++**Files:**+- Modify: `desktop/lib/super_tabs/model.dart` (`SuperTabDelivery`, ~line 851)+- Modify: `desktop/lib/super_tabs/message_receipt.dart` (`build`, `_clock`, the `now` default)+- Test: `desktop/test/super_tab_receipt_test.dart`++**Interfaces:**+- Consumes `claimFailingSince` (ISO string) and `claimFailures` (int) on a page delivery, from Task 1.+- Produces `SuperTabDelivery.claimFailingSince` (`DateTime?`) and `SuperTabDelivery.claimFailures` (`int?`).+- Produces `const superTabClaimFailingNotice = Duration(seconds: 60)` and `const superTabQueuedNotice = Duration(minutes: 2)`+ in `message_receipt.dart`.++- [ ] **Step 1: Write the failing tests**++Use `ReceiptFixture` as it is.++- Its default `createdAt` is 2 min old and its default status is `claimed`.+- For `queued`, pass `ReceiptFixture(tester, status: 'queued')`.+- To make the recipient stopped, set `downSince` on the fixture's agent in `prepare`.++Each new test ends with `await f.close()`.++```dart+testWidgets('receipt: a queued message whose claims fail for a minute reads Not delivered yet', (tester) async {+ final f = ReceiptFixture(tester, status: 'queued');+ await f.show(prepare: () => f.delivery['claimFailingSince'] = f.now.subtract(const Duration(seconds: 61)).toIso8601String());+ expect(find.text('Not delivered yet'), findsOneWidget);+ expect(find.text("Team Mac is online but couldn't take this message yet. Harness keeps trying, and it goes in as soon as it can."), findsOneWidget);+ expect(find.text('Send again'), findsNothing);+ await f.close();+});+```++Then these tests, each with its own assertions:++- **`receipt: a queued message no computer picked up for 2 minutes reads Not delivered yet`**+ - No claim fields. `createdAt` is 2 min 1 s old.+ - Expect the title and the "hasn't picked this message up yet" note.+- **`receipt: offline still wins over Not delivered yet`**+ - Claim failing, and the machine's `receivedAt` is 3 min old.+ - Expect `"Lead's computer is offline"`, and no `Not delivered yet`.+- **`receipt: a stopped recipient shows neither case`**+ - The agent has `downSince` set, the delivery has no claim fields, and the message is 3 min old.+ - Expect `find.text('Not delivered yet')` to find nothing.+- **`receipt: a claimed message with leftover claim fields stays quiet`**+ - Status `claimed`, with `claimFailingSince` 5 min old.+ - Expect `f.expectQuiet()`.+- **`receipt: below both thresholds nothing shows, and the line comes by the receipt's own clock`**+ - Status `queued`, `createdAt` = `clock.now()` minus 1 min 58 s, no claim fields.+ - Expect nothing.+ - Then `await tester.pump(const Duration(seconds: 3))`, with no refresh.+ - Expect `Not delivered yet`.++- [ ] **Step 2: Run the tests to verify they fail**++Run: `cd desktop && flutter test test/super_tab_receipt_test.dart`+Expected: the six new tests FAIL, because no `Not delivered yet` is found. Every existing test passes.++- [ ] **Step 3: Read the fields in `model.dart`**++`claimFailingSince = _date(row['claimFailingSince'])` and `claimFailures = (row['claimFailures'] as num?)?.toInt()`.+Keep the class comment true: these are optional, and an older backend has neither.++- [ ] **Step 4: Add the branch, the clock and the constants in `message_receipt.dart`**++- **Default clock.** Change the widget's default `now` from `DateTime.now` to `clock.now` (`package:clock/clock.dart`, as+ `controller.dart` uses). flutter_test fakes that clock, so a pump moves it.+- **Branch placement.** Place the new branch after the `pending == true && offline` branch and before `fresh`.+- **Condition.** It holds when all of these are true:+ - `delivery?.status == 'queued'`+ - `!offline`+ - `!c.backendUnreachable`+ - either `claimFailingSince` is at least `superTabClaimFailingNotice` before `now` (the "couldn't take" note), or+ `claimFailingSince == null`, `message.createdAt` is at least `superTabQueuedNotice` before `now`, and+ `agent?.downSince == null` (the "hasn't picked" note)+- **Look.** `color = grid.AppPalette.warn`, `icon = AppIcons.clock`, `title = 'Not delivered yet'`, `note` = the copy,+ with `$host` worked out as in the offline branch. No actions.+- **Timer.** `_clock(needed)` must also be true while the delivery is `queued`, the recipient is online, and either+ threshold can still be crossed (not yet crossed, and its condition could hold).+ - The timer fires at `min(500 ms, time to the nearest threshold)`.+ - Once neither threshold can still be crossed, no timer runs.++- [ ] **Step 5: Run the tests and analyze**++Run: `cd desktop && flutter test test/super_tab_receipt_test.dart && flutter analyze lib/super_tabs test/super_tab_receipt_test.dart`+Expected: all tests pass; analyze reports "No issues found!".++Moving `now` to `clock.now` can make an existing test fail. If one does, find out why first. A fixture time read from+`DateTime.now()` against the fake clock is the likely cause. Ledger the ruling, and fix the fixture rather than the+widget.++- [ ] **Step 6: Commit (only once Diego asks)**++```bash+git add desktop/lib/super_tabs/model.dart desktop/lib/super_tabs/message_receipt.dart desktop/test/super_tab_receipt_test.dart+git commit -m "feat(desktop): a queued message that is not getting in reads Not delivered yet"+```++---++### Task 4: The CLI host names a failing claim++**Files:**+- Modify: `cli/src/superTabs/host.ts`: the `deliveryRetries` type (~line 271), the claim catch and the `listRead` log+ (~lines 760-790), and the constants near `DELIVERY_RETRY_MS` (~line 170).+- Test: `cli/src/superTabs/host.spec.ts`, in `describe('a delivery poll or claim that keeps failing', …)`++**Interfaces:**+- Consumes the existing `retryLater(key, list?)`, which returns `{ failures, retryAt, since, list? }`.+- Produces `const CLAIM_FAILURE_LOG_MS = 5 * 60_000`.+- Produces an optional `saidAt?: number` on the `deliveryRetries` entry type.++- [ ] **Step 1: Write the failing tests**++They use that describe's `failing(t, refuse)` helper and `t.wait(ms)`. The setup's ids are: delivery `delivery`, agent+`agent`, session `harness`, Swarm `tab`. Its clock starts at 0.++```ts+it('says whose claim keeps failing, once, then at most every 5 minutes', async () => {+ const t = setup(), host = t.open()+ failing(t, (method, path) => method === 'POST' && path.endsWith('/deliveries/delivery'))+ await host.tick()+ const said = () => t.log.mock.calls.map(c => c[0] as string).filter(l => l.includes('delivery delivery') || l.includes('This Swarm is busy'))+ expect(said()).toEqual(['Could not claim delivery delivery for agent agent (session harness) in Swarm tab: This Swarm is busy. Try again in a moment. (SWARM_BUSY). '+ + 'Failing since 1970-01-01T00:00:00.000Z, 1 try; next try in 1 s.'])+ for (let i = 0; i < 9; i++) { t.wait(30_000); await host.tick() } // 4.5 min of failing claims+ expect(said()).toHaveLength(1)+ t.wait(30_000); await host.tick() // 5 min since the last line+ expect(said()).toHaveLength(2)+ expect(said()[1]).toMatch(/^Could not claim delivery delivery .*Failing since 1970-01-01T00:00:00.000Z, \d+ tries; next try in 30 s\.$/)+})+it('says once when a claim that had failed gets in', async () => { /* refuse = true for 2 claims, then false; after the+ successful tick the log has 'Delivery delivery claimed after 3 tries, 0 min.' exactly once, and no further line on later ticks */ })+```++- [ ] **Step 2: Run the tests to verify they fail**++Run: `cd cli && npx vitest run src/superTabs/host.spec.ts -t "claim keeps failing|claim that had failed"`+Expected: FAIL. The log holds the bare `This Swarm is busy. Try again in a moment.` line.++- [ ] **Step 3: Implement it in `host.ts`**++**The failure line.** In the claim's catch, where `retryLater(claimKey, listKey)` is already called, keep its returned+entry. Log the named line when either holds:++- it is the entry's first failure;+- `now - saidAt >= CLAIM_FAILURE_LOG_MS`.++Then set `saidAt = now`. The line has the same shape as the existing "Could not read the deliveries…" line:++```+Could not claim delivery <id> for agent <agentId> (session <harnessId.slice(0, 8)>) in Swarm <tabId>: <message><code>.+Failing since <ISO>, <n> try|tries; next try in <s> s.+```++`<code>` is ` (CODE)` for a `SuperTabHostError` and empty otherwise.++**No bare line.** The outer `if (listRead) this.log(message)` must not log again for a claim the inner catch already+spoke for, whether or not the inner catch logged on this pass. Mark such errors, for example in a `WeakSet<object>` of+claim errors, and skip them there. Other errors after a read list keep the bare line, as today.++**The success line.** Where a claim succeeds (`this.deliveryRetries.delete(claimKey)` after `accept`), look first for an+entry with `failures > 0`. If there is one, log+`Delivery <id> claimed after <failures + 1> tries, <Math.round((now - since) / 60_000)> min.`++Add a comment naming prod 10-30: 223 bare lines in 2 hours.++- [ ] **Step 4: Run the tests and the typecheck**++Run: `cd cli && npx vitest run src/superTabs/host.spec.ts && npm run typecheck`+Expected: the whole file passes, including the existing backoff tests; tsc exits 0.++- [ ] **Step 5: Commit (only once Diego asks)**++```bash+git add cli/src/superTabs/host.ts cli/src/superTabs/host.spec.ts+git commit -m "fix(cli): a Swarm claim that keeps failing is named in the log"+```++---++### Task 5: Docs, and the final checks++**Files:**+- Modify: `docs/swarms.md`. Add a short paragraph after **Lead updates**, or wherever delivery receipts are described.+ Grep "Not sent" first. It says, plainly:+ - a message that its recipient's online computer cannot take within a minute, or has not picked up in two, reads+ **Not delivered yet**;+ - Harness keeps trying;+ - prod watch alarms on `claims failing`.++ Name the incident date.++- [ ] **Step 1: Write the paragraph.**++- [ ] **Step 2: Run every affected check together, and record each result**++Run:++- `B` (the whole integration file)+- the `deliverySettle.test.ts` unit run+- `cd backend && npm run typecheck`+- `cd desktop && flutter test test/super_tab_receipt_test.dart && flutter analyze lib/super_tabs`+- `cd cli && npx vitest run src/superTabs/host.spec.ts && npm run typecheck`++Expected: all pass. `host.ts` is not under `src/core`, `src/services` or `src/harnessd`, so no 100% coverage gate+applies.++- [ ] **Step 3: Commit (only once Diego asks)**++```bash+git add docs/swarms.md docs/superpowers/specs/2026-10-30-swarm-stuck-delivery-visible-design.md docs/superpowers/plans/2026-10-30-swarm-stuck-delivery-visible.md+git commit -m "docs(swarm): Not delivered yet and the claims failing alarm"+```diff --git a/docs/superpowers/specs/2026-10-30-swarm-stuck-delivery-visible-design.md b/docs/superpowers/specs/2026-10-30-swarm-stuck-delivery-visible-design.mdnew file mode 100644index 000000000..49eae5f8d--- /dev/null+++ b/docs/superpowers/specs/2026-10-30-swarm-stuck-delivery-visible-design.md@@ -0,0 +1,241 @@+# A Swarm message that cannot be delivered says so++Status: design · 2026-10-30 · owner Diego · approved in chat 2026-10-30++## Why++On 2026-10-30 no message a person sent their own Buddy reached it from 09:06 to about 11:01 (VN). Backend 1.3.71+(#2178) made a Buddy delivery's claim touch the same part twice with one version. The second write always conflicted,+`retryTransient` gave up after 8 attempts, and the claim answered `SWARM_BUSY`. Hotfix #2295 fixed the cause.++The delivery itself was never lost. `SuperTabDelivery` is a durable queue (`queued → claimed → delivered`): the row+stayed `queued` and went in once claims worked again. What failed was that **nobody could tell**:++- **The sender** saw nothing. `desktop/lib/super_tabs/message_receipt.dart` shows a line only while a message sends+ (under 3 s), when it was not sent or could not be confirmed, or when the recipient's computer is offline. A `queued`+ message to an online computer has no line, which reads as delivered.+- **The backend** answered every claim with the same busy it gives during a deploy. `stuck()` in+ `backend/src/superTabs/deliverySettle.ts` returns false for every reachable `queued` delivery, so the settle pass+ never looked at it. Nothing raised an alarm.+- **The host** logged 223 bare `[super-tabs] This Swarm is busy. Try again in a moment.` lines (`host.ts` logs a failed+ claim's message alone when its list was read). None of them names the delivery, the agent or how long it has failed.++The trigger was Diego asking whether his Buddy had died. This design makes the same failure visible to the sender+within about a minute, and to prod watch within five.++The herdr-core idea of a seq/missed event stream was considered and dropped: deliveries are already durable rows, so+nothing is missed. The problem was a stuck delivery that nobody noticed.++## Scope++In:++1. Backend records each failed claim on its delivery row and clears the record when a claim succeeds.+2. The delivery settle pass logs an alarm line for a Swarm whose queued deliveries keep failing to be claimed.+3. Desktop and the web app (one Flutter receipt) show **Not delivered yet** for a queued message that is not getting in.+4. The CLI host's log names the failing delivery and is rate limited.++Out:++- The phone app (`mobile/` is its own package). It gets the same rule in a follow-up, from the same fields.+- The round dial and the Harness Pro.+- A Buddy that is down. Its chat already offers **Start**, and a delivery to a stopped agent is not this failure.+- Failed acknowledgements (`ack`). See Open risks.+- Changing what `retryTransient` or the error handler answer. A claim that keeps conflicting still answers+ `SWARM_BUSY`, and the host still retries it.++## Design++### 1. Backend: a failed claim is written on its delivery++**Fields.** `SuperTabDelivery` (`backend/prisma/schema.prisma`) gains four optional fields. Mongo needs no migration,+and older rows read them as null.++| Field | Type | Meaning |+| --- | --- | --- |+| `claimFailingSince` | `DateTime?` | The first failed claim since the last successful one |+| `claimFailedAt` | `DateTime?` | The latest failed claim |+| `claimFailures` | `Int?` | Failed claims since `claimFailingSince` |+| `claimFailure` | `String?` | The latest failure's code (today always `SWARM_BUSY`) |++**Recording.** `SuperTabStore.delivery` (`backend/src/superTabs/store.ts`) runs its `transaction(...)` inside a catch,+for `action.type === 'claim'` only, and only for a request that got past the claim's authorization and status checks on+some attempt. `deliver()` sets a flag right after those checks, so a busy answer that came before them (the first read,+under load) writes nothing on a delivery the caller may not touch. The catch records when the thrown error is:++- a `SuperTabError` with code `SWARM_BUSY`. `retryTransient` (`retry.ts`) ends both kinds of retryable failure that way:+ a write conflict after 8 attempts, and an expired transaction (`P2028`) after 2. So the claim never sees a raw+ `P2028`, and nothing else needs matching.++It records with two standalone single-document writes, **outside** the failed transaction. Both match+`{ id: deliveryId, tabId, agentId, status: 'queued' }`.++1. An `updateMany` that also matches a missing or null `claimFailingSince` and sets it to `now`. With Prisma on Mongo,+ a field an older row never had needs `isSet: false` beside `null`. This keeps the first failure's time from ever+ moving later.+2. An `updateMany` that sets `claimFailedAt = now` and `claimFailure = code`, and increments `claimFailures`. A missing+ count counts from 0.++Those writes touch no Swarm part and no other row, so they cannot meet the conflict that failed the claim. They are+capped at 2 s (`RECORD_FAILED_CLAIM_MS`), so a struggling database never holds the busy answer back. They are best+effort: if it fails, that is logged as `warn [swarm-deliveries] could not record a failed claim` `{tabId, deliveryId,+error}` and the original error is still thrown unchanged. The client's answer (503 `SWARM_BUSY` with Retry-After) does+not change.++Other refusals (`DELIVERY_REVOKED`, `DELIVERY_RESOLVED`, `CLAIM_CONFLICT` and so on) are answers, not failures, and+record nothing.++**Clearing.** Only a successful claim clears the record: its `tx.superTabDelivery.update` in `delivery()` unsets all four fields+(`{ unset: true }`). It must not write `null`, because Mongo's `$inc` refuses a null count, so the next failure's+increment would throw.++Every other write that moves a delivery out of `queued` leaves the fields as they are. That covers an ack, a cancel, the+settle pass and archiving. Every reader below ignores the fields unless the status is `queued`, so the leftovers are+harmless.++**Reading.** `pageDeliveries` (`backend/src/superTabs/pageReads.ts`) projects `claimFailingSince` and+`claimFailures` beside the fields it reads today, and `PageDelivery` gains them. Every reader of a page already reads+its deliveries this way, so the Desktop and web receipts get the fields with no new request. `claimFailure` and+`claimFailedAt` stay server-side: they are for the alarm and for diagnosis, and the person has nothing to do with a+code.++**Cost.** One extra single-document write per failed claim. A host retries a failing claim at most every 30 s+(`DELIVERY_RETRY_MAX_MS`). In the 10-30 incident that would have been about 240 writes across 2 hours for one+person's Buddy, against a delivery index that already exists. No new index is needed: the alarm reads rows the settle+pass already reads.++### 2. Backend: the settle pass raises the alarm++`settleStuckDeliveries` already reads every open delivery every 5 minutes, page by page, grouped by Swarm. After it+settles a Swarm's rows, it also counts that Swarm's **failing** rows from the same page. A row is failing when:++- `status === 'queued'`,+- `claimFailingSince` is at least `CLAIM_FAILING_ALARM_MS` (5 minutes) before `now`, and+- `claimFailedAt` is within `CLAIM_FAILING_ALARM_MS` of `now`, so claims are still failing rather than abandoned.++For each Swarm with at least one failing row, the pass logs one line:++```+warn [swarm-deliveries] claims failing { tabId, deliveries, oldestFailingSince, codes }+```++- `deliveries` is the count.+- `oldestFailingSince` is the earliest `claimFailingSince` as ISO text.+- `codes` is the distinct `claimFailure` values.++The line carries no message text, no agent names and no person ids. The tab id is already in every Swarm log.++This is the line prod watch alarms on. On 10-30 it would first have been logged at the first pass after 09:11 (5 minutes+of failing). The pass **does not cancel** these deliveries:++- A cancelled delivery reads **Not sent** and its text is gone from the recipient.+- Once the cause is fixed, a still-queued one goes in, late but whole.+- `stuck()` is unchanged.++`openDeliveries` must also return the two new date fields as Dates, as it already does for `holdSince`.++### 3. Desktop and web: Not delivered yet++`SuperTabDelivery` (`desktop/lib/super_tabs/model.dart`) reads `claimFailingSince` (a date) and `claimFailures` (an+int). Both are null from an older backend.++`SuperTabMessageReceipt.build` gains one branch. It goes after **offline** and before **Sending…**, so an offline+computer still says it is offline. It applies when the receipt delivery's `status == 'queued'`, the recipient's+computer is not offline, the Swarm is readable (`!c.backendUnreachable`), the recipient is in the Swarm and running+(`downSince == null`), and either of these holds:++| Case | Condition | Note text |+| --- | --- | --- |+| Claims failing | `claimFailingSince` is at least 60 s ago (`superTabClaimFailingNotice`) | "$host is online but couldn't take this message yet. Harness keeps trying, and it goes in as soon as it can." |+| Not picked up | the message's `createdAt` is at least 2 min ago (`superTabQueuedNotice`); the note follows whether `claimFailingSince` is set, so the line never drops out when a first claim failure arrives | "$host is online but hasn't picked this message up yet. Harness keeps trying, and it goes in as soon as it can." |++- **Title:** "Not delivered yet". It uses the warn colour (`grid.AppPalette.warn`) and the clock icon the app already+ uses for waiting. If there is none, the plan picks the nearest existing `AppIcons` mark. No new icon.+- **No actions.** The sender has nothing to do. **Send again** would make a second delivery behind the same failure.+- **`$host`** is the machine name, as the offline note already names it+ (`machine?['name'] ?? widget.machineName(agent.machineId)`).+- **The line disappears by itself** once the delivery leaves `queued`, at the next page read. That is one poll, or a+ push.+- **Clock.** The widget's `_clock(needed)` 500 ms timer runs today only while fresh or offline. It must also run while a+ queued delivery is below either threshold, so the line appears without another rebuild. When neither threshold can+ still be crossed (not queued, or already shown), no timer runs.++The web app gets the same line, because it renders the same widget.++The Leads conversation and a Buddy chat follow the same rule. Their receipts read the same delivery.++### 4. CLI host: the log names the delivery++In `cli/src/superTabs/host.ts`, the branch that logs `message` alone when the list was read but a claim failed+(`if (listRead) this.log(message)`) logs this instead:++```+Could not hand delivery <deliveryId> to agent <agentId> (session <harnessId 8>) in Swarm <tabId>: <message> (<code>).+Failing since <ISO>, <n> tries; next try in <s> s.+```++It uses the `deliveryRetries` entry `retryLater(claimKey, listKey)` already keeps (`since`, `failures`, `retryAt`).+That is the same shape as the existing "Could not read the deliveries…" line.++**Rate.**++- The line names the delivery, not the claim: `accept()` also acknowledges and downloads files, and an ack that keeps+ failing is the open risk below.+- It logs on a delivery's first failure, at once when the error's text or code differs from the last line, and+ otherwise at most once every 5 minutes per delivery while it keeps failing.+- When a delivery that had failed goes through, the host logs `Delivery <id> went through after <n> tries, <m> min.`+ once.+- A failure that is not a `SuperTabHostError` (a network error) is logged the same way, with no code.++At the 10-30 rate this is about 25 named lines instead of 223 bare ones.++The message text is never logged, as today.++## Compatibility++- **Additive fields.** An older Desktop, web build or CLI ignores them. An older backend sends none, and the receipt+ then uses only the 2-minute rule.+- **No change to any request or answer shape** beyond the two added fields on page deliveries.+- **Rollback** leaves the fields unread on existing rows. They are harmless and clear on the next successful claim.++## Testing++**Backend** (`backend/src/superTabs/*.test.ts`, vitest):++- A claim whose transaction throws `SWARM_BUSY` records the four fields with `claimFailures = 1`. A second failure keeps+ `claimFailingSince` and increments the count.+- A `DELIVERY_REVOKED` records nothing.+- A successful claim after failures clears all four fields.+- When the recording write fails, the original error is still thrown and a warn is logged.+- The settle pass logs `claims failing` for a queued row failing 5 min or more and still failing. It logs nothing for one+ failing under 5 min, or one whose last failure is older than 5 min. It cancels none of them.+- `pageDeliveries` returns `claimFailingSince` and `claimFailures`.++**Backend integration** (`backend/integration/`): a claim forced to answer `SWARM_BUSY`, as+`harnessSharesList.test.ts` forces `readParts`. The page read then carries `claimFailingSince`, and a later real claim+clears it.++**Desktop** (`desktop/test/super_tab_receipt_test.dart`):++- Each case shows "Not delivered yet" with its own note at and after its threshold, and nothing before it.+- Offline wins over it.+- A delivered message shows nothing.+- An older backend's delivery (no fields) uses the 2-minute rule.+- A stopped agent (`downSince` set) shows neither case.+- The line appears by the widget's own clock, without a new snapshot.++**CLI** (`cli/src/superTabs/host.spec.ts`):++- A claim failing with `SWARM_BUSY` logs the named line once.+- Further failures inside 5 minutes log nothing; the next one after 5 minutes logs again.+- A different error inside the 5 minutes logs at once.+- Success after failures logs the "went through after" line.++## Open risks++- **Failed acks.** An ack that keeps failing leaves a delivery `claimed`, although the agent has the text. After+ `CLAIM_QUIET_MS` (10 min) the settle pass cancels it, and the sender reads **Not sent** for a message that arrived. This+ design does not change that. It is a separate fix, worth raising after this ships.+- **The 2-minute rule's false positives.** These are agents that are running but legitimately not reading: a host that+ updates for longer than 2 minutes, or `UPDATE_REQUIRED` backoff. In each of these the message truly is not delivered+ yet, so the line is honest. The note says Harness keeps trying, which is true.diff --git a/docs/swarms.md b/docs/swarms.mdindex fab92146f..822e577b5 100644--- a/docs/swarms.md+++ b/docs/swarms.md@@ -640,6 +640,30 @@ summary, or `text: null` when nothing merits another message. Human questions,serious blockers and completion are reported immediately. This does not starta second summarizing model or forward every terminal turn.+## A message that is not getting in++A message waits as a delivery until its recipient's computer takes it. Your newest message shows a line under it only+while it sends, or when it needs attention.++**Not delivered yet** is that line, in amber, with no actions. It shows while the recipient's computer is online and+either:++- has failed to take the message for a minute, or+- has not picked it up for two minutes while the agent runs.++Harness keeps trying, and the line goes once the message is in. An offline computer still reads as offline instead.++Behind it, the backend writes each claim that ends busy on the delivery (`claimFailingSince`, `claimFailures`) and+clears that record when a claim gets in. Every 5 minutes the delivery pass logs `[swarm-deliveries] claims failing`+for a Swarm whose queued messages have failed to be claimed for 5 minutes and still fail. Prod watch alarms on that+line. The pass cancels none of them: once the cause is fixed, they go in late but whole.++The host's log names each delivery that keeps failing (delivery, agent, since when, how many tries). It logs at once,+again when the error changes, and otherwise at most every 5 minutes, and it says once when the delivery gets in.++This comes from 2026-10-30. A backend release made every claim of a person's message to their own Buddy end busy for+two hours. Nothing was lost, but nobody could tell.+## Switching Swarms off on a computerSwarms are on by default. To run a computer's daemon without them, set