Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

A Swarm message that cannot be delivered says so: Not delivered yet, claims failing alarm, named host log (#2336)

0xdiego2516f2304d68bb
diff --git a/backend/integration/superTabIndexes.test.ts b/backend/integration/superTabIndexes.test.ts
index 957fa2280..6ea081309 100644
--- a/backend/integration/superTabIndexes.test.ts
+++ b/backend/integration/superTabIndexes.test.ts
@@ -178,19 +178,20 @@ describe('message page reads', () => {
const rows = await pageDeliveries(db, 'A', ['A-m150'])
expect(rows.find(row => row.id === 'A-150-3')).toEqual({ id: 'A-150-3', messageId: 'A-m150', agentId: 'agent-3', status: 'delivered',
receipt: 'typed', holdReason: 'busy', holdSince: new Date(T0 + 7000), deliveredAt: new Date(T0 + 8000), answeredAt: new Date(T0 + 9000),
- createdAt: new Date(T0 + 150_000), updatedAt: expect.any(Date) })
+ createdAt: new Date(T0 + 150_000), updatedAt: expect.any(Date), claimFailingSince: null, claimFailures: null })
expect(rows.find(row => row.id === 'A-150-old')).toEqual({ id: 'A-150-old', messageId: 'A-m150', agentId: 'agent-old', status: 'answered',
- receipt: null, holdReason: null, holdSince: null, deliveredAt: null, answeredAt: null, createdAt: new Date(T0), updatedAt: new Date(T0 + 1) })
+ receipt: null, holdReason: null, holdSince: null, deliveredAt: null, answeredAt: null, createdAt: new Date(T0), updatedAt: new Date(T0 + 1), claimFailingSince: null, claimFailures: null })
expect(rows.find(row => row.id === 'A-150-4')).toEqual({ id: 'A-150-4', messageId: 'A-m150', agentId: 'agent-4', status: 'answered',
receipt: null, holdReason: null, holdSince: null, deliveredAt: null, answeredAt: null,
- createdAt: new Date(T0 + 150_000), updatedAt: new Date(T0 + 150_500) })
+ createdAt: new Date(T0 + 150_000), updatedAt: new Date(T0 + 150_500), claimFailingSince: null, claimFailures: null })
})
// The delivery read never had an order (no orderBy: it followed whichever index the planner took), so it is compared by id.
// Reactions keep the tabId+messageId index order, which their counts' `by` lists show.
it('returns the same rows as the Prisma reads it replaces', async () => {
const select = { id: true, messageId: true, agentId: true, status: true, receipt: true,
- holdReason: true, holdSince: true, deliveredAt: true, answeredAt: true, createdAt: true, updatedAt: true }
+ holdReason: true, holdSince: true, deliveredAt: true, answeredAt: true, createdAt: true, updatedAt: true,
+ claimFailingSince: true, claimFailures: true }
const byId = (rows: Array<{ id: string }>) => [...rows].sort((a, b) => a.id.localeCompare(b.id))
expect(byId(await pageDeliveries(db, 'A', PAGE))).toEqual(byId(await db.superTabDelivery.findMany({ where: { tabId: 'A', messageId: { in: PAGE } }, select })))
expect(await pageReactions(db, 'A', PAGE)).toEqual(await db.superTabReaction.findMany({ where: { tabId: 'A', messageId: { in: PAGE } },
diff --git a/backend/integration/superTabTidy.test.ts b/backend/integration/superTabTidy.test.ts
index 13813c05f..b7f00021e 100644
--- a/backend/integration/superTabTidy.test.ts
+++ b/backend/integration/superTabTidy.test.ts
@@ -3,7 +3,7 @@ import { execFile } from 'node:child_process'
import { promisify } from 'node:util'
import { join } from 'node:path'
import { tmpdir } from 'node:os'
-import { afterAll, beforeAll, describe, expect, it } from 'vitest'
+import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'
import { MongoMemoryReplSet } from 'mongodb-memory-server-core'
import { PrismaClient } from '@prisma/client'
import type { AuthUser } from '../src/lib/ssoAuth.js'
@@ -13,6 +13,8 @@ import type { MachineObservation } from '../src/superTabs/machines.js'
import { poolRows, rosterRows } from './hostRows.test-support.js'
import { archiveFinishedGroups, markFinishedGroups, type PullRequestState } from '../src/superTabs/archive.js'
import { settleStuckDeliveries } from '../src/superTabs/deliverySettle.js'
+import { SuperTabError } from '../src/superTabs/model.js'
+import { logger } from '../src/utils/logger.js'
// Nobody tidies a Swarm by hand: a worker idle for 15 minutes is stopped on its Lead's poll and its host is asked to close
// it. A fresh loopback replica set, created here.
@@ -550,4 +552,126 @@ describe('a delivery that cannot move', () => {
expect(await f.timed.settleDeliveries(f.tab, [read], f.clock.now + HOUR + MINUTE)).toBe(1)
expect(await db.superTabDelivery.findUniqueOrThrow({ where: { id } })).toMatchObject({ status: 'cancelled', holdReason: 'team_waiting_idle' })
})
+
+ // Prod 10-30, v1.3.71 (#2178): every Buddy claim answered SWARM_BUSY for two hours and nothing said so. A claim that fails
+ // that way is now written on its delivery, outside the failed transaction (store.ts recordFailedClaim).
+ describe('a claim that keeps failing', () => {
+ /** Whether the stores from failingStore answer busy, and whether the record write itself is down. Reset before each test. */
+ const busy = { on: false, recordDown: false, early: false, recordHangs: false }
+ beforeEach(() => { busy.on = false; busy.recordDown = false; busy.early = false; busy.recordHangs = false })
+ /** A store over the same database whose transactions answer busy while busy.on is set, as retryTransient ends 8 write
+ * conflicts. By default the real body runs first (its authorization checks included) and the answer comes at commit, rolling
+ * the body back; with busy.early the answer comes before any body runs. With busy.recordDown its single-document delivery
+ * writes throw too; with busy.recordHangs they never answer. */
+ const failingStore = (f: Fixture) => {
+ const proxy = new Proxy(db, { get: (target, key) => {
+ const tooBusy = () => new SuperTabError('SWARM_BUSY', 'This Swarm is busy. Try again in a moment.', 503)
+ if (key === '$transaction') return (body: (tx: unknown) => Promise<unknown>, ...rest: unknown[]) => {
+ if (!busy.on) return (target.$transaction as (...a: unknown[]) => unknown)(body, ...rest)
+ if (busy.early) return Promise.reject(tooBusy())
+ return (target.$transaction as (...a: unknown[]) => unknown)(async (tx: unknown) => { await body(tx); throw tooBusy() }, ...rest)
+ }
+ if (key === 'superTabDelivery' && (busy.recordDown || busy.recordHangs)) return new Proxy(target.superTabDelivery, { get: (model, name) =>
+ name === 'updateMany' ? () => busy.recordHangs ? new Promise(() => {}) : Promise.reject(new Error('down')) : Reflect.get(model, name) })
+ return Reflect.get(target, key)
+ } }) as PrismaClient
+ busy.on = true
+ return new SuperTabStore(proxy, () => f.clock.now)
+ }
+ const claim = { type: 'claim', claimId: 'c' } as const
+ const claimFields = ['claimFailingSince', 'claimFailedAt', 'claimFailures', 'claimFailure'] as const
+ const row = (id: string) => db.superTabDelivery.findUniqueOrThrow({ where: { id } })
+
+ it('records a busy claim on its delivery, keeps when it began and counts each one', async () => {
+ const f = await swarm(); const id = await toLead(f, 'Hi')
+ const flaky = failingStore(f)
+ await expect(flaky.delivery(host, f.tab, f.lead, id, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY' })
+ const first = f.clock.now
+ f.clock.now += 30_000
+ await expect(flaky.delivery(host, f.tab, f.lead, id, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY' })
+ expect(await row(id)).toMatchObject({ status: 'queued', claimFailingSince: new Date(first),
+ claimFailedAt: new Date(first + 30_000), claimFailures: 2, claimFailure: 'SWARM_BUSY' })
+ })
+ it('clears the record once a claim gets in', async () => {
+ const f = await swarm(); const id = await toLead(f, 'Hi')
+ await expect(failingStore(f).delivery(host, f.tab, f.lead, id, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY' })
+ expect((await row(id)).claimFailures).toBe(1)
+ await f.timed.delivery(host, f.tab, f.lead, id, claim)
+ const claimed = await row(id)
+ expect(claimed.status).toBe('claimed')
+ // Unset, not null: the claim removes the keys, so "missing" stays a real absence that the first failure's isSet: false branch
+ // matches (and starts the count at 0, since Prisma's increment of a missing count leaves it null). Prisma reads both as null.
+ const [raw] = await db.superTabDelivery.findRaw({ filter: { _id: id } }) as unknown as Array<Record<string, unknown>>
+ for (const field of claimFields) { expect(claimed[field], field).toBeNull(); expect(raw, field).not.toHaveProperty(field) }
+ })
+ it('records nothing for a delivery that is no longer queued, or for an ack', async () => {
+ const f = await swarm(); const gone = await toLead(f, 'Gone'); const acked = await toLead(f, 'Acked')
+ await db.superTabDelivery.update({ where: { id: gone }, data: { status: 'claimed', claimId: 'c' } })
+ const flaky = failingStore(f)
+ await expect(flaky.delivery(host, f.tab, f.lead, gone, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY' })
+ for (const field of claimFields) expect((await row(gone))[field], field).toBeNull()
+ await f.timed.delivery(host, f.tab, f.lead, acked, claim)
+ await expect(flaky.delivery(host, f.tab, f.lead, acked, { type: 'ack', claimId: 'c', receipt: 'r', uncertain: false })).rejects.toMatchObject({ code: 'SWARM_BUSY' })
+ for (const field of claimFields) expect((await row(acked))[field], field).toBeNull()
+ })
+ it('records nothing when busy is answered before the request was authorized', async () => {
+ const f = await swarm(); const id = await toLead(f, 'Hi')
+ const flaky = failingStore(f); busy.early = true
+ await expect(flaky.delivery(host, f.tab, f.lead, id, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY' })
+ for (const field of claimFields) expect((await row(id))[field], field).toBeNull()
+ })
+ it('answers busy within about 2 seconds even when the record write never returns', async () => {
+ const f = await swarm(); const id = await toLead(f, 'Hi')
+ const flaky = failingStore(f); busy.recordHangs = true
+ const began = Date.now()
+ await expect(flaky.delivery(host, f.tab, f.lead, id, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY', status: 503 })
+ expect(Date.now() - began).toBeLessThan(4_000)
+ }, 15_000)
+ it('still throws the busy answer when the record itself cannot be written, and says so', async () => {
+ const f = await swarm(); const id = await toLead(f, 'Hi')
+ const flaky = failingStore(f); busy.recordDown = true
+ const warn = vi.spyOn(logger, 'warn').mockImplementation(() => {})
+ try {
+ await expect(flaky.delivery(host, f.tab, f.lead, id, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY', status: 503 })
+ expect(warn).toHaveBeenCalledWith('[swarm-deliveries] could not record a failed claim', expect.objectContaining({ tabId: f.tab, deliveryId: id }))
+ } finally { warn.mockRestore() }
+ })
+ it('hands claimFailingSince and claimFailures to page readers', async () => {
+ const f = await swarm(); const id = await toLead(f, 'Hi')
+ await expect(failingStore(f).delivery(host, f.tab, f.lead, id, claim)).rejects.toMatchObject({ code: 'SWARM_BUSY' })
+ const page = await f.timed.messages(owner, f.tab, 'lead', {})
+ const shown = page.messages.flatMap(m => m.deliveries).find(d => d.id === id)
+ expect(shown).toMatchObject({ claimFailingSince: new Date(f.clock.now), claimFailures: 1 })
+ expect(JSON.parse(JSON.stringify(shown))).toMatchObject({ claimFailingSince: new Date(f.clock.now).toISOString(), claimFailures: 1 })
+ })
+ it('raises claims failing for a delivery whose claims have failed 5 minutes and still fail, and cancels nothing', async () => {
+ const f = await swarm(); const id = await toLead(f, 'Hi'); const flaky = failingStore(f)
+ const began = f.clock.now
+ await expect(flaky.delivery(host, f.tab, f.lead, id, claim)).rejects.toThrow()
+ f.clock.now = began + 5 * MINUTE
+ await expect(flaky.delivery(host, f.tab, f.lead, id, claim)).rejects.toThrow()
+ const warn = vi.spyOn(logger, 'warn').mockImplementation(() => {})
+ try {
+ await settleStuckDeliveries(db, f.timed, began + 5 * MINUTE + 1_000)
+ // Earlier tests leave rows in the shared database; this Swarm's line is the one that counts.
+ expect(warn).toHaveBeenCalledWith('[swarm-deliveries] claims failing',
+ { tabId: f.tab, deliveries: 1, oldestFailingSince: new Date(began).toISOString(), codes: ['SWARM_BUSY'] })
+ expect(await status(id)).toBe('queued')
+ } finally { warn.mockRestore() }
+ })
+ it('says nothing for a claim failing under 5 minutes, or one whose last failure is over 5 minutes old', async () => {
+ const f = await swarm(); const young = await toLead(f, 'Young'); const old = await toLead(f, 'Old')
+ const now = f.clock.now + 10 * MINUTE
+ await db.superTabDelivery.update({ where: { id: young }, data: { claimFailingSince: new Date(now - 4 * MINUTE),
+ claimFailedAt: new Date(now - 1_000), claimFailures: 3, claimFailure: 'SWARM_BUSY' } })
+ await db.superTabDelivery.update({ where: { id: old }, data: { claimFailingSince: new Date(now - 20 * MINUTE),
+ claimFailedAt: new Date(now - 6 * MINUTE), claimFailures: 9, claimFailure: 'SWARM_BUSY' } })
+ const warn = vi.spyOn(logger, 'warn').mockImplementation(() => {})
+ try {
+ await settleStuckDeliveries(db, f.timed, now)
+ expect(warn.mock.calls.filter(([line, fields]) => line === '[swarm-deliveries] claims failing' && (fields as { tabId: string }).tabId === f.tab)).toEqual([])
+ expect(await status(young)).toBe('queued'); expect(await status(old)).toBe('queued')
+ } finally { warn.mockRestore() }
+ })
+ })
})
diff --git a/backend/prisma/schema.prisma b/backend/prisma/schema.prisma
index 928dcff75..63c3ffd2f 100644
--- a/backend/prisma/schema.prisma
+++ b/backend/prisma/schema.prisma
@@ -1269,6 +1269,12 @@ model SuperTabDelivery {
// Optional on older records: backend first, then Desktop's inline receipt.
holdReason String?
holdSince DateTime?
+ // A queued delivery whose claims keep answering SWARM_BUSY (prod 10-30, v1.3.71): written outside the failed claim, unset by
+ // the claim that gets in. The settle pass alarms on them and page reads show the sender (store.ts recordFailedClaim).
+ claimFailingSince DateTime?
+ claimFailedAt DateTime?
+ claimFailures Int?
+ claimFailure String?
deliveredAt DateTime?
answeredAt DateTime?
createdAt DateTime @default(now())
diff --git a/backend/src/superTabs/deliverySettle.test.ts b/backend/src/superTabs/deliverySettle.test.ts
new file mode 100644
index 000000000..72607d248
--- /dev/null
+++ b/backend/src/superTabs/deliverySettle.test.ts
@@ -0,0 +1,26 @@
+import { describe, expect, it } from 'vitest'
+import { CLAIM_FAILING_ALARM_MS, claimsFailing } from './deliverySettle.js'
+
+const NOW = 10_000_000
+const row = (since: number | null, last: number | null, status = 'queued') => ({ status,
+ claimFailingSince: since == null ? null : new Date(NOW - since), claimFailedAt: last == null ? null : new Date(NOW - last) }) as Parameters<typeof claimsFailing>[0]
+
+describe('claimsFailing', () => {
+ it('is true at exactly 5 minutes since the first failure and a last failure exactly 5 minutes old', () => {
+ expect(claimsFailing(row(CLAIM_FAILING_ALARM_MS, CLAIM_FAILING_ALARM_MS), NOW)).toBe(true)
+ })
+ it('is false at 4:59 since the first failure', () => {
+ expect(claimsFailing(row(CLAIM_FAILING_ALARM_MS - 1_000, 1_000), NOW)).toBe(false)
+ })
+ it('is false when the last failure is 5:01 old', () => {
+ expect(claimsFailing(row(20 * 60_000, CLAIM_FAILING_ALARM_MS + 1_000), NOW)).toBe(false)
+ })
+ it('is false for a claimed delivery', () => {
+ expect(claimsFailing(row(CLAIM_FAILING_ALARM_MS, 1_000, 'claimed'), NOW)).toBe(false)
+ })
+ it('is false with null fields', () => {
+ expect(claimsFailing(row(null, null), NOW)).toBe(false)
+ expect(claimsFailing(row(CLAIM_FAILING_ALARM_MS, null), NOW)).toBe(false)
+ expect(claimsFailing(row(null, 1_000), NOW)).toBe(false)
+ })
+})
diff --git a/backend/src/superTabs/deliverySettle.ts b/backend/src/superTabs/deliverySettle.ts
index 2fa522273..f1b8f2b98 100644
--- a/backend/src/superTabs/deliverySettle.ts
+++ b/backend/src/superTabs/deliverySettle.ts
@@ -17,6 +17,9 @@ export const HOLD_LIMIT_MS = 60 * 60_000
export const WAITING_ON_PERSON_LIMIT_MS = 24 * 60 * 60_000
/** How long a person has to check a delivery its host could not confirm before it ends, as their Open terminal ends it. */
export const UNCERTAIN_LIMIT_MS = 60 * 60_000
+/** A queued delivery whose claims have failed this long, and failed again within this long, is stuck behind a Swarm that keeps
+ * answering busy (prod 10-30, v1.3.71: every person's messages to their own Buddy sat queued for 2 hours, and nothing said so). */
+export const CLAIM_FAILING_ALARM_MS = 5 * 60_000
const SETTLE_EVERY_MS = 5 * 60_000
const OPEN = ['queued', 'claimed', 'uncertain']
const BATCH = 500
@@ -31,13 +34,19 @@ export function stuck(delivery: Pick<SuperTabDelivery, 'status' | 'holdReason' |
return since(delivery.holdSince) > (delivery.holdReason === 'team_waiting_user' ? WAITING_ON_PERSON_LIMIT_MS : HOLD_LIMIT_MS)
}
+/** Whether a delivery's claims keep failing: the alarm only logs it, because cancelling would drop a message a healthy host can still take. */
+export function claimsFailing(delivery: Pick<SuperTabDelivery, 'status' | 'claimFailingSince' | 'claimFailedAt'>, now: number): boolean {
+ if (delivery.status !== 'queued' || !delivery.claimFailingSince || !delivery.claimFailedAt) return false
+ return now - delivery.claimFailingSince.getTime() >= CLAIM_FAILING_ALARM_MS && now - delivery.claimFailedAt.getTime() <= CLAIM_FAILING_ALARM_MS
+}
+
/** A page of open deliveries, by id. Read by their own index (status, id) with a plain match: Prisma sends `status in` as an
* $expr $or that no index can bound (pageReads.ts), and a pass must not read every delivery ever sent. */
async function openDeliveries(db: PrismaClient, after: string | null): Promise<SuperTabDelivery[]> {
const rows = await db.superTabDelivery.aggregateRaw({ pipeline: [{ $match: { status: { $in: OPEN }, ...after ? { _id: { $gt: after } } : {} } },
{ $sort: { _id: 1 } }, { $limit: BATCH }] }) as unknown as Array<Record<string, unknown>>
const date = (value: unknown) => value == null ? null : new Date(ms(value))
- return rows.map(({ _id, ...row }) => ({ ...row, id: _id as string, holdSince: date(row.holdSince), deliveredAt: date(row.deliveredAt),
+ return rows.map(({ _id, ...row }) => ({ ...row, id: _id as string, holdSince: date(row.holdSince), claimFailingSince: date(row.claimFailingSince), claimFailedAt: date(row.claimFailedAt), deliveredAt: date(row.deliveredAt),
answeredAt: date(row.answeredAt), createdAt: date(row.createdAt)!, updatedAt: date(row.updatedAt)! }) as unknown as SuperTabDelivery)
}
@@ -53,6 +62,16 @@ export async function settleStuckDeliveries(db: PrismaClient, store: Pick<SuperT
try { settled += await store.settleDeliveries(tabId, tabRows, now) } catch (error) {
logger.warn('[swarm-deliveries] could not settle', { tabId, error: error instanceof Error ? error.message : String(error) })
}
+ // Counted per page, so a Swarm whose rows span two 500-row pages logs one line per page. Rare; prod watch counts lines per tab.
+ // Its own try/catch: a Swarm whose count throws must not stop the pass for the Swarms after it.
+ try {
+ const failing = tabRows.filter(row => claimsFailing(row, now))
+ if (failing.length) logger.warn('[swarm-deliveries] claims failing', { tabId, deliveries: failing.length,
+ oldestFailingSince: new Date(Math.min(...failing.map(row => row.claimFailingSince!.getTime()))).toISOString(),
+ codes: [...new Set(failing.map(row => row.claimFailure).filter((code): code is string => !!code))].sort() })
+ } catch (error) {
+ logger.warn('[swarm-deliveries] could not count failing claims', { tabId, error: error instanceof Error ? error.message : String(error) })
+ }
}
if (rows.length < BATCH) return settled
after = rows.at(-1)!.id
diff --git a/backend/src/superTabs/pageReads.ts b/backend/src/superTabs/pageReads.ts
index 1fb38fa01..7e0ac1f06 100644
--- a/backend/src/superTabs/pageReads.ts
+++ b/backend/src/superTabs/pageReads.ts
@@ -19,17 +19,18 @@ export async function pageReactions(tx: Transaction, tabId: string, messageIds:
}
export type PageDelivery = Pick<SuperTabDelivery, 'id' | 'messageId' | 'agentId' | 'status' | 'receipt' | 'holdReason' | 'holdSince'
- | 'deliveredAt' | 'answeredAt' | 'createdAt' | 'updatedAt'>
+ | 'claimFailingSince' | 'claimFailures' | 'deliveredAt' | 'answeredAt' | 'createdAt' | 'updatedAt'>
/** The deliveries of one page of messages: who each message went to and how far it got. As Prisma read them: dates as
* Dates, and an optional field an older record lacks as null. */
export async function pageDeliveries(tx: Transaction, tabId: string, messageIds: string[]): Promise<PageDelivery[]> {
const rows = await tx.superTabDelivery.aggregateRaw({ pipeline: [{ $match: { tabId, messageId: { $in: messageIds } } },
- { $project: { messageId: 1, agentId: 1, status: 1, receipt: 1, holdReason: 1, holdSince: 1, deliveredAt: 1, answeredAt: 1, createdAt: 1, updatedAt: 1 } }] }) as
+ { $project: { messageId: 1, agentId: 1, status: 1, receipt: 1, holdReason: 1, holdSince: 1, claimFailingSince: 1, claimFailures: 1, deliveredAt: 1, answeredAt: 1, createdAt: 1, updatedAt: 1 } }] }) as
unknown as Array<Record<string, unknown>>
const date = (value: unknown) => value == null ? null : new Date(ms(value))
return rows.map(row => ({ id: row._id as string, messageId: row.messageId as string, agentId: row.agentId as string, status: row.status as string,
receipt: (row.receipt ?? null) as string | null, holdReason: (row.holdReason ?? null) as string | null, holdSince: date(row.holdSince),
+ claimFailingSince: date(row.claimFailingSince), claimFailures: (row.claimFailures ?? null) as number | null,
deliveredAt: date(row.deliveredAt), answeredAt: date(row.answeredAt), createdAt: date(row.createdAt)!, updatedAt: date(row.updatedAt)! }))
}
diff --git a/backend/src/superTabs/store.ts b/backend/src/superTabs/store.ts
index e93192143..332c1cf30 100644
--- a/backend/src/superTabs/store.ts
+++ b/backend/src/superTabs/store.ts
@@ -200,6 +200,8 @@ export async function transaction<T>(db: Pick<PrismaClient, '$transaction'>, bod
}
}, sleep, error => error === staleIndex)
}
+/** The longest a failed claim's record may hold up the busy answer. */
+const RECORD_FAILED_CLAIM_MS = 2_000
const TRANSACTION_OPTIONS = { maxWait: 5_000, timeout: 15_000 }
/** Each attempt's after-commit work, by that attempt's client: a retried attempt's list is dropped with its client. */
const pendingCommits = new WeakMap<Transaction, Array<() => void>>()
@@ -2646,6 +2648,42 @@ export class SuperTabStore {
async delivery(user: AuthUser, tabId: string, agentId: string, deliveryId: string,
action: { type: 'claim'; claimId: string } | { type: 'ack'; claimId: string; receipt: string; uncertain: boolean }
| { type: 'hold'; claimId: string; reason: HoldReason }) {
+ const reached = { authorized: false }
+ try {
+ return await this.deliver(user, tabId, agentId, deliveryId, action, reached)
+ } catch (error) {
+ // Prod 10-30, v1.3.71: 2 hours of SWARM_BUSY claims left no trace on the delivery. Retries end that way (retry.ts), so it is
+ // the only code to record; the client still gets the original error. Only once an attempt got past the authorization
+ // checks: a busy answer from before them says nothing about whether this caller may touch the delivery. The write is capped
+ // so a struggling database cannot delay the 503 further.
+ if (action.type === 'claim' && reached.authorized && error instanceof SuperTabError && error.code === 'SWARM_BUSY') {
+ let timer: NodeJS.Timeout | undefined
+ const cap = new Promise<void>(resolve => { timer = setTimeout(resolve, RECORD_FAILED_CLAIM_MS); timer.unref() })
+ try { await Promise.race([this.recordFailedClaim(tabId, agentId, deliveryId, error.code), cap]) } finally { clearTimeout(timer) }
+ }
+ throw error
+ }
+ }
+
+ /** A failed claim, written on its delivery by two single-document writes outside the failed transaction: they touch no Swarm
+ * part, so they cannot meet the conflict that failed it. Only a delivery still queued; the first failure's time never moves
+ * later. Best effort: a failed write is logged and never replaces the busy answer. */
+ private async recordFailedClaim(tabId: string, agentId: string, deliveryId: string, code: string): Promise<void> {
+ try {
+ const at = new Date(this.now()), mine = { id: deliveryId, tabId, agentId, status: 'queued' }
+ // A row an older backend wrote lacks the field, which Prisma on Mongo matches as isSet false, not null. The count starts at 0
+ // here: Prisma's increment of a missing count leaves it null.
+ await this.db.superTabDelivery.updateMany({ where: { ...mine, OR: [{ claimFailingSince: null }, { claimFailingSince: { isSet: false } }] },
+ data: { claimFailingSince: at, claimFailures: 0 } })
+ await this.db.superTabDelivery.updateMany({ where: mine, data: { claimFailedAt: at, claimFailure: code, claimFailures: { increment: 1 } } })
+ } catch (error) {
+ logger.warn('[swarm-deliveries] could not record a failed claim', { tabId, deliveryId, error: error instanceof Error ? error.message : String(error) })
+ }
+ }
+
+ private async deliver(user: AuthUser, tabId: string, agentId: string, deliveryId: string,
+ action: { type: 'claim'; claimId: string } | { type: 'ack'; claimId: string; receipt: string; uncertain: boolean }
+ | { type: 'hold'; claimId: string; reason: HoldReason }, reached: { authorized: boolean }) {
return transaction(this.db, async tx => {
const delivery = await tx.superTabDelivery.findFirst({ where: { id: deliveryId, tabId, agentId } })
await this.row(tx, user, tabId)
@@ -2671,6 +2709,7 @@ export class SuperTabStore {
if (delivery.claimId && delivery.claimId !== action.claimId) fail('CLAIM_CONFLICT', 'Reconcile the existing delivery before claiming it again.')
if (action.type !== 'claim' && !delivery.claimId) fail('NOT_CLAIMED', 'Claim this instruction before acknowledging it.')
if (action.type === 'hold') return this.hold(tx, delivery, action.reason)
+ reached.authorized = true
const status = action.type === 'claim' ? delivery.status === 'queued' ? 'claimed' : delivery.status : action.uncertain ? 'uncertain' : 'delivered'
if (delivery.status === 'delivered' && action.type === 'ack' && (action.uncertain || delivery.receipt !== action.receipt)) {
fail('RECEIPT_CONFLICT', 'A delivered instruction keeps its original host receipt.')
@@ -2682,7 +2721,11 @@ export class SuperTabStore {
const part = parts.members.find(p => p.key === touch.key)
if (part) await guardedWrite(tx, tabId, { op: 'touch', ...touch, seen: part.version }, stamp)
}
+ // A claim that gets in ends the failing record. Unset, never null: a missing field is a real absence, which the first failure's
+ // isSet: false branch matches and which leaves no keys in the document.
const updated = await tx.superTabDelivery.update({ where: { id: delivery.id }, data: { status, claimId: action.claimId,
+ ...(action.type === 'claim' ? { claimFailingSince: { unset: true }, claimFailedAt: { unset: true }, claimFailures: { unset: true },
+ claimFailure: { unset: true } } : {}),
...(action.type === 'ack' ? { receipt: action.receipt, holdReason: null, holdSince: null,
...(!action.uncertain ? { deliveredAt: delivery.deliveredAt ?? new Date(this.now()) } : {}) } : {}) } })
const row = await tx.superTabMessage.findUnique({ where: { id: delivery.messageId } })
diff --git a/cli/src/superTabs/host.spec.ts b/cli/src/superTabs/host.spec.ts
index a5fa0238c..26904146b 100644
--- a/cli/src/superTabs/host.spec.ts
+++ b/cli/src/superTabs/host.spec.ts
@@ -736,13 +736,14 @@ describe('Super Tab native delivery handoff', () => {
describe('a delivery poll or claim that keeps failing', () => {
/** The calls the host makes for one failing path, counted as the backend refuses them with `status`. */
- function failing(t: ReturnType<typeof setup>, refuse: (method: string, path: string) => boolean, status = 503) {
+ function failing(t: ReturnType<typeof setup>, refuse: (method: string, path: string) => boolean, status = 503,
+ error: () => { code: string; message: string } = () => ({ code: 'SWARM_BUSY', message: 'This Swarm is busy. Try again in a moment.' })) {
const original = vi.mocked(t.core.account.backend).getMockImplementation()!
const calls = { n: 0 }
vi.mocked(t.core.account.backend).mockImplementation(async (method, path, body) => {
if (!refuse(method, path)) return original(method, path, body)
calls.n++
- return { status, body: { success: false, error: { code: 'SWARM_BUSY', message: 'This Swarm is busy. Try again in a moment.' } } }
+ return { status, body: { success: false, error: error() } }
})
return calls
}
@@ -771,6 +772,51 @@ describe('a delivery poll or claim that keeps failing', () => {
}
})
+ it('says which delivery keeps failing to go through, once, then at most every 5 minutes', async () => {
+ const t = setup(), host = t.open()
+ failing(t, (method, path) => method === 'POST' && path.endsWith('/deliveries/delivery'))
+ await host.tick()
+ const said = () => t.log.mock.calls.map(c => c[0] as string).filter(l => l.includes('delivery delivery') || l.includes('Delivery delivery') || l.includes('This Swarm is busy'))
+ expect(said()).toEqual(['Could not hand delivery delivery to agent agent (session harness) in Swarm tab: This Swarm is busy. Try again in a moment. (SWARM_BUSY). '
+ + 'Failing since 1970-01-01T00:00:00.000Z, 1 try; next try in 1 s.'])
+ for (let i = 0; i < 9; i++) { t.wait(30_000); await host.tick() } // 4.5 min of failing claims
+ expect(said()).toHaveLength(1)
+ t.wait(30_000); await host.tick() // 5 min since the last line
+ expect(said()).toHaveLength(2)
+ expect(said()[1]).toMatch(/^Could not hand delivery delivery .*Failing since 1970-01-01T00:00:00.000Z, \d+ tries; next try in 30 s\.$/)
+ })
+
+ it('says a different failure at once, and the same one again only after 5 minutes', async () => {
+ // Prod 10-30: accept() is more than the claim (ack, downloads, checks), so a new error inside the window must not hide.
+ const t = setup(), host = t.open()
+ let err = { code: 'SWARM_BUSY', message: 'This Swarm is busy. Try again in a moment.' }
+ failing(t, (method, path) => method === 'POST' && path.endsWith('/deliveries/delivery'), 503, () => err)
+ await host.tick()
+ const said = () => t.log.mock.calls.map(c => c[0] as string).filter(l => l.startsWith('Could not hand delivery delivery'))
+ expect(said()).toHaveLength(1)
+ err = { code: 'INTERNAL', message: 'The delivery has no message.' }
+ t.wait(1_000); await host.tick()
+ expect(said()).toHaveLength(2)
+ expect(said()[1]).toContain('The delivery has no message. (INTERNAL)')
+ t.wait(2_000); await host.tick()
+ t.wait(4_000); await host.tick()
+ expect(said()).toHaveLength(2)
+ })
+
+ it('says once when a delivery that had failed goes through', async () => {
+ const t = setup(), host = t.open()
+ let refuse = true
+ failing(t, (method, path) => refuse && method === 'POST' && path.endsWith('/deliveries/delivery'))
+ await host.tick()
+ t.wait(1_000); await host.tick()
+ refuse = false
+ t.wait(2_000); await host.tick()
+ const claimed = () => t.log.mock.calls.map(c => c[0] as string).filter(l => l.startsWith('Delivery delivery went through'))
+ expect(claimed()).toEqual(['Delivery delivery went through after 3 tries, 0 min.'])
+ t.wait(1_000); await host.tick()
+ expect(claimed()).toHaveLength(1)
+ })
+
it('claims at once, on the next second, after the backend answers again', async () => {
const t = setup(), host = t.open()
let refuse = true
diff --git a/cli/src/superTabs/host.ts b/cli/src/superTabs/host.ts
index b3b96ac6a..fad3b58c8 100644
--- a/cli/src/superTabs/host.ts
+++ b/cli/src/superTabs/host.ts
@@ -180,6 +180,9 @@ const JOIN_FINAL = new Set(['MACHINES_FULL', 'OWNER_REQUIRED'])
* added to an overloaded backend (production at 5 pods, 2026-10-27). */
const DELIVERY_RETRY_MS = 1_000
const DELIVERY_RETRY_MAX_MS = 30_000
+/** Prod 10-30: a claim refused every 30 s said the backend's bare words 223 times in 2 hours. The same failure of a delivery is
+ * named at most this often. */
+const CLAIM_FAILURE_LOG_MS = 5 * 60_000
/** Stability roadmap 2.4 (2026-10-28): every host read each running agent's delivery list each second, almost always to find
* nothing. A server that says it pushes (a `super_tab_deliveries` notice over the daemon socket, backend lib/adapterWs.ts)
* names the agent when work waits; an idle agent's list is then read only this often, as the safety net for a lost push.
@@ -279,7 +282,7 @@ export class SuperTabHost {
private nextReplacementsAt = 0
private timer: ReturnType<typeof setTimeout> | null = null
/** Per agent's delivery list and per delivery's claim, the failures in a row and when it may be asked again. */
- private readonly deliveryRetries = new Map<string, { failures: number; retryAt: number; since: number; list?: string }>()
+ private readonly deliveryRetries = new Map<string, { failures: number; retryAt: number; since: number; saidAt?: number; said?: string; list?: string }>()
private native: NativeRecorder | null = null
private machines: ContributedMachines | null = null
private readonly machineDeps: Partial<MachineDeps>
@@ -757,7 +760,7 @@ export class SuperTabHost {
for (const id of last.ids) seen.add(id)
continue
}
- let listRead = false
+ let listRead = false, claimError: unknown
try {
const deliveries = await backend(this.core, 'GET', path)
if (this.stopped) return
@@ -778,10 +781,30 @@ export class SuperTabHost {
// claimed one with no journal is never typed, only reported as uncertain; a delivered one only acknowledged.
const entry = this.journal.entries[delivery.id]
if ((entry ? entry.stage === 'prepared' : !delivery.claimId) && !this.mayRead(target, delivery)) continue
- try { await this.accept(target, delivery, path); this.deliveryRetries.delete(claimKey) } catch (error) {
+ try {
+ await this.accept(target, delivery, path)
+ const failed = this.deliveryRetries.get(claimKey)
+ this.deliveryRetries.delete(claimKey)
+ if (failed && failed.failures > 0) this.log(`Delivery ${delivery.id} went through after ${failed.failures + 1} tries, ${Math.round((this.now() - failed.since) / 60_000)} min.`)
+ } catch (error) {
// Answered (or resolved by a person) between the roster read and this call: nothing is owed, and the
// agent's other deliveries go on.
- if (!(error instanceof SuperTabHostError) || error.code !== 'DELIVERY_RESOLVED') { this.retryLater(claimKey, listKey); throw error }
+ if (!(error instanceof SuperTabHostError) || error.code !== 'DELIVERY_RESOLVED') {
+ // Prod 10-30: the bare message below named no delivery or agent. Named here, and only now and then: this retries for hours.
+ // accept() is more than the claim (ack POST, downloads, message and journal checks; ack failures are a known open risk),
+ // so the line names no step, and a different error than the last one said is told at once, not hidden by the window.
+ const retry = this.retryLater(claimKey, listKey), now = this.now(), code = error instanceof SuperTabHostError ? ` (${error.code})` : ''
+ const what = `${error instanceof Error ? error.message : String(error)}${code}`
+ if (retry.saidAt === undefined || retry.said !== what || now - retry.saidAt >= CLAIM_FAILURE_LOG_MS) {
+ retry.saidAt = now
+ retry.said = what
+ this.log(`Could not hand delivery ${delivery.id} to agent ${target.agent.id} (session ${target.agent.harnessId.slice(0, 8)}) in Swarm ${target.tabId}: `
+ + `${what}. Failing since ${new Date(retry.since).toISOString()}, ${retry.failures} ${retry.failures === 1 ? 'try' : 'tries'}; `
+ + `next try in ${Math.round((retry.retryAt - now) / 1_000)} s.`)
+ }
+ claimError = error
+ throw error
+ }
this.deliveryRetries.delete(claimKey)
this.retire(delivery.id as string)
}
@@ -790,7 +813,7 @@ export class SuperTabHost {
} catch (error) {
// Authority checks and network failures both withhold input. A later fresh read may grant it again.
const message = error instanceof Error ? error.message : String(error)
- if (listRead) this.log(message)
+ if (listRead) { if (error !== claimError) this.log(message) }
else {
// 2026-10-28: a Lead's list failed for 45 minutes while this log said only the backend's words, with no agent.
const retry = this.retryLater(listKey), code = error instanceof SuperTabHostError ? ` (${error.code})` : ''
diff --git a/desktop/lib/super_tabs/message_receipt.dart b/desktop/lib/super_tabs/message_receipt.dart
index 34d1e6838..927c1d6e1 100644
--- a/desktop/lib/super_tabs/message_receipt.dart
+++ b/desktop/lib/super_tabs/message_receipt.dart
@@ -1,5 +1,6 @@
import 'dart:async';
+import 'package:clock/clock.dart';
import 'package:flutter/material.dart';
import '../core/models.dart';
@@ -11,6 +12,12 @@ import 'controller.dart';
import 'machine_status.dart';
import 'model.dart';
+/// How long a queued delivery's claims fail before its sender is told, and how
+/// long a queued message waits with no claim fields before the same (prod 10-30,
+/// v1.3.71: claims failed for two hours and nothing showed).
+const superTabClaimFailingNotice = Duration(seconds: 60);
+const superTabQueuedNotice = Duration(minutes: 2);
+
/// The line under the chat's newest own message, shown only while it sends,
/// when it was not sent, or when its delivery needs a hand; a message that
/// arrived carries none, as in Slack.
@@ -23,7 +30,7 @@ class SuperTabMessageReceipt extends StatefulWidget {
required this.onAddLead,
required this.machineName,
this.app,
- this.now = DateTime.now,
+ this.now,
});
final SuperTabMessage message;
final SuperTabController controller;
@@ -31,7 +38,9 @@ class SuperTabMessageReceipt extends StatefulWidget {
final void Function(SuperTabDelivery delivery, String resolution) onResolve;
final VoidCallback onAddLead;
final String Function(String machineId) machineName;
- final DateTime Function() now;
+
+ /// Defaults to the package clock, which tests move (a const default cannot).
+ final DateTime Function()? now;
@override
State<SuperTabMessageReceipt> createState() => _SuperTabMessageReceiptState();
@@ -74,7 +83,7 @@ class _SuperTabMessageReceiptState extends State<SuperTabMessageReceipt>
if (mounted) setState(() {});
}
- void _clock(bool needed) {
+ void _clock(bool needed, [Duration? until]) {
_timer?.cancel();
final lifecycle = WidgetsBinding.instance.lifecycleState;
if (!needed ||
@@ -82,7 +91,9 @@ class _SuperTabMessageReceiptState extends State<SuperTabMessageReceipt>
(lifecycle != null && lifecycle != AppLifecycleState.resumed)) {
return;
}
- _timer = Timer(const Duration(milliseconds: 500), _changed);
+ // A notice threshold is one wake-up at its time, not a tick every 500 ms.
+ final wait = until ?? const Duration(milliseconds: 500);
+ _timer = Timer(wait.isNegative ? Duration.zero : wait, _changed);
}
@override
@@ -101,7 +112,7 @@ class _SuperTabMessageReceiptState extends State<SuperTabMessageReceipt>
final name =
c.snapshot?.agentName(delivery?.agentId ?? '', fallback: 'Lead') ??
'Lead';
- final now = widget.now();
+ final now = widget.now?.call() ?? clock.now();
final machine = c.snapshot?.machines
.where((m) => m['machineId'] == agent?.machineId)
.firstOrNull;
@@ -129,6 +140,32 @@ class _SuperTabMessageReceiptState extends State<SuperTabMessageReceipt>
delivery?.pending == true &&
message.createdAt != null &&
now.difference(message.createdAt!) < const Duration(seconds: 3);
+ // A queued delivery to a running agent on an online, readable Swarm that is
+ // not getting in: its claims fail, or no claim fields came (an older backend)
+ // and it just sits. Either reason shows the line, so it never drops out
+ // when the other one starts.
+ final queued =
+ delivery?.status == 'queued' &&
+ agent != null &&
+ agent.downSince == null &&
+ !offline &&
+ !c.backendUnreachable;
+ final failingSince = delivery?.claimFailingSince;
+ final failingAt = failingSince?.add(superTabClaimFailingNotice);
+ final queuedAt = message.createdAt?.add(superTabQueuedNotice);
+ final stuck =
+ queued &&
+ ((failingAt != null && !failingAt.isAfter(now)) ||
+ (queuedAt != null && !queuedAt.isAfter(now)));
+ // Until a threshold is crossed the receipt must tick by itself: no snapshot
+ // arrives to rebuild it. The earlier of the two is the next to cross.
+ final upcoming = [
+ failingAt,
+ queuedAt,
+ ].whereType<DateTime>().where((t) => t.isAfter(now));
+ final nextAt = queued && !stuck && upcoming.isNotEmpty
+ ? upcoming.reduce((a, b) => a.isBefore(b) ? a : b)
+ : null;
final error = Theme.of(context).colorScheme.error;
var color = error;
var icon = AppIcons.triangleAlert;
@@ -175,12 +212,24 @@ class _SuperTabMessageReceiptState extends State<SuperTabMessageReceipt>
note = readingAt == null
? "$host is offline. Your message goes in when it's back online."
: "$host was last seen ${superTabAgo(now.difference(readingAt))}. Your message goes in when it's back online.";
+ } else if (stuck) {
+ // Queued to an online computer with no line read as delivered (prod 10-30).
+ // Nothing to do for the sender: Send again would queue behind the same failure.
+ color = grid.AppPalette.warn;
+ icon = AppIcons.clock;
+ title = 'Not delivered yet';
+ final host =
+ machine?['name'] as String? ?? widget.machineName(agent.machineId);
+ note = failingSince != null
+ ? "$host is online but couldn't take this message yet. Harness keeps trying, and it goes in as soon as it can."
+ : "$host is online but hasn't picked this message up yet. Harness keeps trying, and it goes in as soon as it can.";
} else if (fresh) {
color = DesktopChrome.muted;
title = 'Sending…';
sending = true;
}
- _clock(fresh || (delivery?.pending == true && offline));
+ final ticking = fresh || (delivery?.pending == true && offline);
+ _clock(ticking || nextAt != null, ticking ? null : nextAt?.difference(now));
if (title == null) return const SizedBox.shrink();
final style = superTabReceiptStyle(color);
final parts = <Widget>[
diff --git a/desktop/lib/super_tabs/model.dart b/desktop/lib/super_tabs/model.dart
index 5dacaaf1c..1b8890c26 100644
--- a/desktop/lib/super_tabs/model.dart
+++ b/desktop/lib/super_tabs/model.dart
@@ -858,11 +858,18 @@ class SuperTabDelivery {
status = row['status'] as String? ?? '',
holdReason = row['holdReason'] as String?,
holdSince = _date(row['holdSince']),
+ claimFailingSince = _date(row['claimFailingSince']),
+ claimFailures = (row['claimFailures'] as num?)?.toInt(),
deliveredAt = _date(row['deliveredAt']),
answeredAt = _date(row['answeredAt']),
updatedAt = _date(row['updatedAt']);
final String? id, holdReason;
final String agentId, status;
+
+ /// Since when the backend's claims of this delivery fail, and how many have
+ /// (prod 10-30, v1.3.71). Null from an older backend, and once a claim works.
+ final DateTime? claimFailingSince;
+ final int? claimFailures;
final DateTime? holdSince, deliveredAt, answeredAt, updatedAt;
static DateTime? _date(Object? value) =>
value is String ? DateTime.tryParse(value) : null;
diff --git a/desktop/test/super_tab_receipt_test.dart b/desktop/test/super_tab_receipt_test.dart
index 2bf8c6a42..8b477ae3c 100644
--- a/desktop/test/super_tab_receipt_test.dart
+++ b/desktop/test/super_tab_receipt_test.dart
@@ -1,5 +1,6 @@
import 'dart:io';
+import 'package:clock/clock.dart';
import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:harness/shared/theme/app_theme.dart' as grid;
@@ -16,7 +17,8 @@ class ReceiptFixture {
final WidgetTester tester;
final String? status;
final server = ConversationServer();
- final now = DateTime.now();
+ // The receipt reads the package clock, which a test moves; real time would drift from it.
+ final now = clock.now();
late final SuperTabController controller;
String? opened;
bool addedLead = false;
@@ -377,4 +379,169 @@ void main() {
await f.close();
},
);
+
+ // Not delivered yet: prod 10-30 (v1.3.71), claims failed for two hours while
+ // every queued message looked delivered.
+ const couldntTake =
+ "Team Mac is online but couldn't take this message yet. Harness keeps trying, and it goes in as soon as it can.";
+ const notPickedUp =
+ "Team Mac is online but hasn't picked this message up yet. Harness keeps trying, and it goes in as soon as it can.";
+
+ testWidgets(
+ 'receipt: a queued message whose claims fail for a minute reads Not delivered yet',
+ (tester) async {
+ final f = ReceiptFixture(tester, status: 'queued');
+ await f.show(
+ prepare: () => f.delivery['claimFailingSince'] = f.now
+ .subtract(const Duration(seconds: 61))
+ .toIso8601String(),
+ );
+ expect(find.text('Not delivered yet'), findsOneWidget);
+ expect(find.text(couldntTake), findsOneWidget);
+ expect(find.text('Send again'), findsNothing);
+ await f.close();
+ },
+ );
+
+ testWidgets(
+ 'receipt: a queued message no computer picked up for 2 minutes reads Not delivered yet',
+ (tester) async {
+ final f = ReceiptFixture(tester, status: 'queued');
+ await f.show(
+ prepare: () => f.message['createdAt'] = f.now
+ .subtract(const Duration(minutes: 2, seconds: 1))
+ .toIso8601String(),
+ );
+ expect(find.text('Not delivered yet'), findsOneWidget);
+ expect(find.text(notPickedUp), findsOneWidget);
+ await f.close();
+ },
+ );
+
+ testWidgets('receipt: offline still wins over Not delivered yet', (
+ tester,
+ ) async {
+ final f = ReceiptFixture(tester, status: 'queued');
+ await f.show(
+ prepare: () {
+ f.delivery['claimFailingSince'] = f.now
+ .subtract(const Duration(minutes: 5))
+ .toIso8601String();
+ (f.server.state['machines'] as List)
+ .first['observation']['receivedAt'] = f.now
+ .subtract(const Duration(minutes: 3))
+ .millisecondsSinceEpoch;
+ },
+ );
+ expect(find.text("Lead's computer is offline"), findsOneWidget);
+ expect(find.text('Not delivered yet'), findsNothing);
+ await f.close();
+ });
+
+ testWidgets(
+ 'receipt: a stopped recipient that has been queued for minutes shows neither case',
+ (tester) async {
+ final f = ReceiptFixture(tester, status: 'queued');
+ await f.show(
+ prepare: () {
+ (f.server.state['agents'] as List).first['downSince'] = f.now
+ .subtract(const Duration(minutes: 10))
+ .toIso8601String();
+ f.message['createdAt'] = f.now
+ .subtract(const Duration(minutes: 3))
+ .toIso8601String();
+ },
+ );
+ expect(find.text('Not delivered yet'), findsNothing);
+ await f.close();
+ },
+ );
+
+ testWidgets(
+ 'receipt: a stopped recipient with failing claims shows no Not delivered yet',
+ (tester) async {
+ final f = ReceiptFixture(tester, status: 'queued');
+ await f.show(
+ prepare: () {
+ (f.server.state['agents'] as List).first['downSince'] = f.now
+ .subtract(const Duration(minutes: 10))
+ .toIso8601String();
+ f.delivery['claimFailingSince'] = f.now
+ .subtract(const Duration(minutes: 5))
+ .toIso8601String();
+ },
+ );
+ expect(find.text('Not delivered yet'), findsNothing);
+ await f.close();
+ },
+ );
+
+ testWidgets(
+ 'receipt: a delivery to an agent not in the Swarm shows nothing and does not throw',
+ (tester) async {
+ final f = ReceiptFixture(tester, status: 'queued');
+ await f.show(
+ prepare: () {
+ f.delivery['agentId'] = 'ghost-agent';
+ f.message['createdAt'] = f.now
+ .subtract(const Duration(minutes: 3))
+ .toIso8601String();
+ },
+ );
+ expect(find.text('Not delivered yet'), findsNothing);
+ await f.close();
+ },
+ );
+
+ testWidgets(
+ 'receipt: a message queued for minutes keeps its line when the first claim failure arrives',
+ (tester) async {
+ final f = ReceiptFixture(tester, status: 'queued');
+ await f.show(
+ prepare: () {
+ f.message['createdAt'] = f.now
+ .subtract(const Duration(minutes: 5))
+ .toIso8601String();
+ f.delivery['claimFailingSince'] = f.now
+ .subtract(const Duration(seconds: 10))
+ .toIso8601String();
+ },
+ );
+ expect(find.text('Not delivered yet'), findsOneWidget);
+ expect(find.text(couldntTake), findsOneWidget);
+ await f.close();
+ },
+ );
+
+ testWidgets(
+ 'receipt: a claimed message with leftover claim fields stays quiet',
+ (tester) async {
+ final f = ReceiptFixture(tester);
+ await f.show(
+ prepare: () => f.delivery['claimFailingSince'] = f.now
+ .subtract(const Duration(minutes: 5))
+ .toIso8601String(),
+ );
+ f.expectQuiet();
+ expect(find.text('Not delivered yet'), findsNothing);
+ await f.close();
+ },
+ );
+
+ testWidgets(
+ 'receipt: below both thresholds nothing shows, and the line comes by the receipt\'s own clock',
+ (tester) async {
+ final f = ReceiptFixture(tester, status: 'queued');
+ await f.show(
+ prepare: () => f.message['createdAt'] = clock
+ .now()
+ .subtract(const Duration(minutes: 1, seconds: 58))
+ .toIso8601String(),
+ );
+ expect(find.text('Not delivered yet'), findsNothing);
+ await tester.pump(const Duration(seconds: 3));
+ expect(find.text('Not delivered yet'), findsOneWidget);
+ await f.close();
+ },
+ );
}
diff --git a/docs/superpowers/plans/2026-10-30-swarm-stuck-delivery-visible.md b/docs/superpowers/plans/2026-10-30-swarm-stuck-delivery-visible.md
new file mode 100644
index 000000000..ca83a4675
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-30-swarm-stuck-delivery-visible.md
@@ -0,0 +1,459 @@
+# A Swarm message that cannot be delivered says so: Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** When a Swarm delivery keeps failing to be claimed, three things change:
+
+- the sender sees **Not delivered yet**;
+- prod watch gets a `claims failing` alarm line;
+- the host's log names the claim.
+
+**Architecture:** The backend records each `SWARM_BUSY` claim failure on its `SuperTabDelivery` row, with standalone
+writes outside the failed transaction. A successful claim clears the record. The delivery settle pass logs an alarm
+for Swarms whose queued deliveries keep failing, and cancels none of them. Page reads carry `claimFailingSince` and
+`claimFailures`. The Flutter receipt (Desktop and web) turns those fields, or a queued message's age, into an amber line.
+The CLI host replaces its bare busy line with a named, rate-limited one.
+
+**Tech Stack:**
+- Backend: TypeScript, Fastify, Prisma on MongoDB, vitest with mongodb-memory-server.
+- Desktop: Flutter/Dart, flutter_test, package:clock.
+- CLI: TypeScript, vitest.
+
+**Spec:** `docs/superpowers/specs/2026-10-30-swarm-stuck-delivery-visible-design.md`
+
+## Global Constraints
+
+- Work only in the worktree `.claude/worktrees/stuckdelivery` (branch `feat/stuck-delivery-visible`). Never touch the
+ main checkout.
+- Commit only when Diego asks. The commit steps below are prepared commands: run them only once asked. Stage only this
+ plan's files.
+- The fresh worktree has no `node_modules`. Run `npm ci` once in `backend/` and in `cli/` before the first test.
+ Run `npx prisma generate` in `backend/` after the schema change.
+- Backend integration test command prefix (`B`):
+ `cd backend && NODE_ENV=test HARNESS_CREDENTIAL_ENCRYPTION_KEY=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= npx vitest run --config vitest.super-tabs.config.ts integration/superTabTidy.test.ts`
+- No message text, agent names or person ids in any new log line.
+- The client's answer to a failed claim does not change: still 503 `SWARM_BUSY`.
+- Thresholds, copied from the spec:
+ - Desktop: claim failing for **60 s** (`superTabClaimFailingNotice`); queued for **2 min** (`superTabQueuedNotice`).
+ - Backend alarm: `CLAIM_FAILING_ALARM_MS` = **5 min**.
+ - CLI: logs at most once every **5 min** per delivery (`CLAIM_FAILURE_LOG_MS`).
+- Copy, exact text:
+ - title `Not delivered yet`;
+ - note `$host is online but couldn't take this message yet. Harness keeps trying, and it goes in as soon as it can.`;
+ - note `$host is online but hasn't picked this message up yet. Harness keeps trying, and it goes in as soon as it can.`
+- Comments say why, naming the incident ("prod 10-30, v1.3.71"), in the surrounding files' density.
+
+## Review Focus
+
+1. A claim failing on a delivery that is no longer `queued` (revoked or cancelled while in flight) must write nothing.
+ The record's filter includes `status: 'queued'`. Test in Task 1.
+2. A failing `ack` or `hold` records nothing. Only claims do. Test in Task 1.
+3. A delivery that left `queued` but still carries leftover fields shows no line in the receipt. Test in Task 3.
+4. The alarm pass must keep settling the other Swarms when one Swarm's alarm count throws, and must cancel no failing
+ row. Test in Task 2.
+5. The receipt's own clock makes the line appear without a new snapshot, using `clock.now` so tests can drive it.
+ Test in Task 3.
+
+---
+
+### Task 1: Backend records a failed claim on its delivery, and page reads carry it
+
+**Files:**
+- Modify: `backend/prisma/schema.prisma` (model `SuperTabDelivery`, ~line 1249)
+- Modify: `backend/src/superTabs/store.ts` (`delivery()`, ~line 2606)
+- Modify: `backend/src/superTabs/pageReads.ts` (`PageDelivery`, `pageDeliveries`, ~lines 21-34)
+- Test: `backend/integration/superTabTidy.test.ts`. Add a `describe('a claim that keeps failing', …)` nested inside
+ `describe('a delivery that cannot move', …)`, so it reuses `toLead`, `status` and `HOUR`.
+
+**Interfaces:**
+- Produces these fields on `SuperTabDelivery`, all optional: `claimFailingSince DateTime?`, `claimFailedAt DateTime?`,
+ `claimFailures Int?`, `claimFailure String?`.
+- Produces `PageDelivery` with `claimFailingSince: Date | null` and `claimFailures: number | null`.
+- Produces `private async recordFailedClaim(tabId: string, agentId: string, deliveryId: string, code: string): Promise<void>`
+ on `SuperTabStore`.
+
+- [ ] **Step 1: Write the failing tests**
+
+All of them use one helper inside the new describe. `failingStore(f)` returns
+`new SuperTabStore(proxy, () => f.clock.now)`, where `proxy` is a `Proxy` over `db`:
+
+- its `$transaction` throws `new SuperTabError('SWARM_BUSY', 'This Swarm is busy. Try again in a moment.', 503)` while
+ `busy.on` is true;
+- every other property passes through to `db`.
+
+Import `SuperTabError` from `../src/superTabs/model.js`.
+
+```ts
+it('records a busy claim on its delivery, keeps when it began and counts each one', async () => {
+ const f = await swarm(); const id = await toLead(f, 'Hi')
+ const flaky = failingStore(f)
+ await expect(flaky.delivery(host, f.tab, f.lead, id, { type: 'claim', claimId: 'c' })).rejects.toMatchObject({ code: 'SWARM_BUSY' })
+ const first = f.clock.now
+ f.clock.now += 30_000
+ await expect(flaky.delivery(host, f.tab, f.lead, id, { type: 'claim', claimId: 'c' })).rejects.toMatchObject({ code: 'SWARM_BUSY' })
+ expect(await db.superTabDelivery.findUniqueOrThrow({ where: { id } })).toMatchObject({ status: 'queued',
+ claimFailingSince: new Date(first), claimFailedAt: new Date(first + 30_000), claimFailures: 2, claimFailure: 'SWARM_BUSY' })
+})
+it('clears the record once a claim gets in', async () => { /* fail once with flaky, then f.timed.delivery(... claim 'c') succeeds;
+ the row has status 'claimed' and claimFailingSince, claimFailedAt, claimFailures, claimFailure all undefined (unset, not null) */ })
+it('records nothing for a delivery that is no longer queued, or for an ack', async () => { /* (a) cancel the row with
+ db.superTabDelivery.update({ status: 'cancelled' }), flaky claim rejects, no claim* fields; (b) a real claim, then a flaky ack
+ rejects SWARM_BUSY, no claim* fields */ })
+it('still throws the busy answer when the record itself cannot be written, and says so', async () => { /* proxy also makes
+ superTabDelivery.updateMany throw new Error('down'); delivery() rejects with code SWARM_BUSY;
+ vi.spyOn(logger, 'warn') was called with '[swarm-deliveries] could not record a failed claim', expect.objectContaining({ tabId: f.tab, deliveryId: id }) */ })
+it('hands claimFailingSince and claimFailures to page readers', async () => { /* after one flaky failure, f.timed.read(owner, f.tab) page
+ of 'lead' has the message's delivery with claimFailingSince equal to that time and claimFailures 1 (as the page serialises dates) */ })
+```
+
+For the last test, find how this file's other tests read a page's deliveries. If none does, use `pageDeliveries(tx, …)`
+inside `db.$transaction`, imported from `../src/superTabs/pageReads.js`.
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `B -t "a claim that keeps failing"`
+Expected: FAIL. TypeScript or Prisma reject the unknown `claimFailingSince` fields, or the assertions find them
+undefined.
+
+- [ ] **Step 3: Add the four optional fields to `SuperTabDelivery` in `schema.prisma`, then run `npx prisma generate`**
+
+Put a one-line comment above them naming prod 10-30 and the settle pass alarm. Add no index.
+
+- [ ] **Step 4: Implement recording and clearing in `store.ts`**
+
+In `delivery()`:
+
+1. Wrap the `return transaction(...)` in a `try`/`catch`. When `action.type === 'claim'` and the error is a
+ `SuperTabError` with code `SWARM_BUSY`, `await this.recordFailedClaim(...)`. Then rethrow the original error.
+2. In the successful claim's existing `tx.superTabDelivery.update`, add `{ unset: true }` for all four fields, only when
+ `action.type === 'claim'`.
+
+`recordFailedClaim` makes two writes. Both match `{ id: deliveryId, tabId, agentId, status: 'queued' }`:
+
+1. `this.db.superTabDelivery.updateMany`, with `OR: [{ claimFailingSince: null }, { claimFailingSince: { isSet: false } }]`
+ added to the filter, setting `claimFailingSince: new Date(this.now())`.
+2. `this.db.superTabDelivery.updateMany` setting `claimFailedAt: new Date(this.now())` and `claimFailure: code`, with
+ `claimFailures: { increment: 1 }`.
+
+Catch any error from either write and log
+`logger.warn('[swarm-deliveries] could not record a failed claim', { tabId, deliveryId, error })`. Never let it
+replace the thrown error.
+
+- [ ] **Step 5: Project the two fields in `pageReads.ts`**
+
+Add `claimFailingSince` and `claimFailures` to the `PageDelivery` pick and to the `$project`. Map them as
+`date(row.claimFailingSince)` and `(row.claimFailures ?? null) as number | null`.
+
+Then find every place a page delivery is turned into JSON for clients, by grepping `pageDeliveries(` and `holdSince` in
+`backend/src/superTabs`. Make sure the two fields reach the response the same way `holdSince` does. If a serialiser
+lists fields explicitly, add them there.
+
+- [ ] **Step 6: Run the tests and the typecheck**
+
+Run: `B -t "a claim that keeps failing"`, then `cd backend && npm run typecheck`
+Expected: 5 passed; tsc exits 0.
+
+Then run the whole file, `B`. Expected: every test passes. The existing settle tests must be unchanged.
+
+- [ ] **Step 7: Commit (only once Diego asks)**
+
+```bash
+git add backend/prisma/schema.prisma backend/src/superTabs/store.ts backend/src/superTabs/pageReads.ts backend/integration/superTabTidy.test.ts
+git commit -m "fix(swarm): a claim that keeps failing is recorded on its delivery"
+```
+
+---
+
+### Task 2: The settle pass logs `claims failing`
+
+**Files:**
+- Modify: `backend/src/superTabs/deliverySettle.ts`
+- Test: `backend/integration/superTabTidy.test.ts`, in the same `describe('a claim that keeps failing', …)`
+
+**Interfaces:**
+- Consumes the Task 1 fields on the rows `openDeliveries` returns.
+- Produces `export const CLAIM_FAILING_ALARM_MS = 5 * 60_000`.
+- Produces `export function claimsFailing(delivery: Pick<SuperTabDelivery, 'status' | 'claimFailingSince' | 'claimFailedAt'>, now: number): boolean`.
+ It is true when all of these hold: `status === 'queued'`, `now - claimFailingSince >= CLAIM_FAILING_ALARM_MS`, and
+ `now - claimFailedAt <= CLAIM_FAILING_ALARM_MS`.
+
+- [ ] **Step 1: Write the failing tests**
+
+```ts
+it('raises claims failing for a delivery whose claims have failed 5 minutes and still fail, and cancels nothing', async () => {
+ const f = await swarm(); const id = await toLead(f, 'Hi'); const flaky = failingStore(f)
+ const began = f.clock.now
+ await expect(flaky.delivery(host, f.tab, f.lead, id, { type: 'claim', claimId: 'c' })).rejects.toThrow()
+ f.clock.now = began + 5 * MINUTE
+ await expect(flaky.delivery(host, f.tab, f.lead, id, { type: 'claim', claimId: 'c' })).rejects.toThrow()
+ const warn = vi.spyOn(logger, 'warn')
+ await settleStuckDeliveries(db, f.timed, began + 5 * MINUTE + 1_000)
+ expect(warn).toHaveBeenCalledWith('[swarm-deliveries] claims failing',
+ { tabId: f.tab, deliveries: 1, oldestFailingSince: new Date(began).toISOString(), codes: ['SWARM_BUSY'] })
+ expect(await status(id)).toBe('queued')
+ warn.mockRestore()
+})
+it('says nothing for a claim failing under 5 minutes, or one whose last failure is over 5 minutes old', async () => { /* two
+ deliveries: one failing 4 min; one whose claimFailedAt is 6 min before the pass (set the fields directly with db.update).
+ No 'claims failing' warn; both stay queued */ })
+```
+
+The pure `claimsFailing` boundaries also get a unit test, in a new `backend/src/superTabs/deliverySettle.test.ts`:
+
+- true at exactly 5 min since and 5 min since the last failure;
+- false at 4:59 since;
+- false at a last failure 5:01 old;
+- false for `claimed`;
+- false with null fields.
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `B -t "claims failing|says nothing for a claim"`, then
+`cd backend && NODE_ENV=test HARNESS_CREDENTIAL_ENCRYPTION_KEY=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= npx vitest run src/superTabs/deliverySettle.test.ts`
+Expected: FAIL. `claimsFailing` is not exported, and no warn is logged.
+
+- [ ] **Step 3: Implement it in `deliverySettle.ts`**
+
+1. Map `claimFailingSince` and `claimFailedAt` to Dates in `openDeliveries`, as `holdSince` is.
+2. In `settleStuckDeliveries`, per Swarm, **after** its `store.settleDeliveries` try/catch, count
+ `tabRows.filter(row => claimsFailing(row, now))`.
+3. When the count is non-zero, log `logger.warn('[swarm-deliveries] claims failing', { tabId, deliveries,
+ oldestFailingSince, codes })`. Here `oldestFailingSince` is the earliest `claimFailingSince` as ISO text, and `codes`
+ is the sorted distinct `claimFailure` values.
+4. The count and the log sit inside their own try/catch, which logs `[swarm-deliveries] could not count failing claims`,
+ so one Swarm never stops the pass.
+
+`stuck()` is unchanged.
+
+A Swarm whose rows span two 500-row pages logs one line per page. Accept that, and say so in a comment. It is rare, and
+prod watch counts lines per tab.
+
+- [ ] **Step 4: Run the tests to verify they pass**
+
+Run: the two commands from Step 2, then `B`
+Expected: all pass.
+
+- [ ] **Step 5: Commit (only once Diego asks)**
+
+```bash
+git add backend/src/superTabs/deliverySettle.ts backend/src/superTabs/deliverySettle.test.ts backend/integration/superTabTidy.test.ts
+git commit -m "feat(swarm): the delivery pass raises claims failing for prod watch"
+```
+
+---
+
+### Task 3: Desktop and web show Not delivered yet
+
+**Files:**
+- Modify: `desktop/lib/super_tabs/model.dart` (`SuperTabDelivery`, ~line 851)
+- Modify: `desktop/lib/super_tabs/message_receipt.dart` (`build`, `_clock`, the `now` default)
+- Test: `desktop/test/super_tab_receipt_test.dart`
+
+**Interfaces:**
+- Consumes `claimFailingSince` (ISO string) and `claimFailures` (int) on a page delivery, from Task 1.
+- Produces `SuperTabDelivery.claimFailingSince` (`DateTime?`) and `SuperTabDelivery.claimFailures` (`int?`).
+- Produces `const superTabClaimFailingNotice = Duration(seconds: 60)` and `const superTabQueuedNotice = Duration(minutes: 2)`
+ in `message_receipt.dart`.
+
+- [ ] **Step 1: Write the failing tests**
+
+Use `ReceiptFixture` as it is.
+
+- Its default `createdAt` is 2 min old and its default status is `claimed`.
+- For `queued`, pass `ReceiptFixture(tester, status: 'queued')`.
+- To make the recipient stopped, set `downSince` on the fixture's agent in `prepare`.
+
+Each new test ends with `await f.close()`.
+
+```dart
+testWidgets('receipt: a queued message whose claims fail for a minute reads Not delivered yet', (tester) async {
+ final f = ReceiptFixture(tester, status: 'queued');
+ await f.show(prepare: () => f.delivery['claimFailingSince'] = f.now.subtract(const Duration(seconds: 61)).toIso8601String());
+ expect(find.text('Not delivered yet'), findsOneWidget);
+ expect(find.text("Team Mac is online but couldn't take this message yet. Harness keeps trying, and it goes in as soon as it can."), findsOneWidget);
+ expect(find.text('Send again'), findsNothing);
+ await f.close();
+});
+```
+
+Then these tests, each with its own assertions:
+
+- **`receipt: a queued message no computer picked up for 2 minutes reads Not delivered yet`**
+ - No claim fields. `createdAt` is 2 min 1 s old.
+ - Expect the title and the "hasn't picked this message up yet" note.
+- **`receipt: offline still wins over Not delivered yet`**
+ - Claim failing, and the machine's `receivedAt` is 3 min old.
+ - Expect `"Lead's computer is offline"`, and no `Not delivered yet`.
+- **`receipt: a stopped recipient shows neither case`**
+ - The agent has `downSince` set, the delivery has no claim fields, and the message is 3 min old.
+ - Expect `find.text('Not delivered yet')` to find nothing.
+- **`receipt: a claimed message with leftover claim fields stays quiet`**
+ - Status `claimed`, with `claimFailingSince` 5 min old.
+ - Expect `f.expectQuiet()`.
+- **`receipt: below both thresholds nothing shows, and the line comes by the receipt's own clock`**
+ - Status `queued`, `createdAt` = `clock.now()` minus 1 min 58 s, no claim fields.
+ - Expect nothing.
+ - Then `await tester.pump(const Duration(seconds: 3))`, with no refresh.
+ - Expect `Not delivered yet`.
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `cd desktop && flutter test test/super_tab_receipt_test.dart`
+Expected: the six new tests FAIL, because no `Not delivered yet` is found. Every existing test passes.
+
+- [ ] **Step 3: Read the fields in `model.dart`**
+
+`claimFailingSince = _date(row['claimFailingSince'])` and `claimFailures = (row['claimFailures'] as num?)?.toInt()`.
+Keep the class comment true: these are optional, and an older backend has neither.
+
+- [ ] **Step 4: Add the branch, the clock and the constants in `message_receipt.dart`**
+
+- **Default clock.** Change the widget's default `now` from `DateTime.now` to `clock.now` (`package:clock/clock.dart`, as
+ `controller.dart` uses). flutter_test fakes that clock, so a pump moves it.
+- **Branch placement.** Place the new branch after the `pending == true && offline` branch and before `fresh`.
+- **Condition.** It holds when all of these are true:
+ - `delivery?.status == 'queued'`
+ - `!offline`
+ - `!c.backendUnreachable`
+ - either `claimFailingSince` is at least `superTabClaimFailingNotice` before `now` (the "couldn't take" note), or
+ `claimFailingSince == null`, `message.createdAt` is at least `superTabQueuedNotice` before `now`, and
+ `agent?.downSince == null` (the "hasn't picked" note)
+- **Look.** `color = grid.AppPalette.warn`, `icon = AppIcons.clock`, `title = 'Not delivered yet'`, `note` = the copy,
+ with `$host` worked out as in the offline branch. No actions.
+- **Timer.** `_clock(needed)` must also be true while the delivery is `queued`, the recipient is online, and either
+ threshold can still be crossed (not yet crossed, and its condition could hold).
+ - The timer fires at `min(500 ms, time to the nearest threshold)`.
+ - Once neither threshold can still be crossed, no timer runs.
+
+- [ ] **Step 5: Run the tests and analyze**
+
+Run: `cd desktop && flutter test test/super_tab_receipt_test.dart && flutter analyze lib/super_tabs test/super_tab_receipt_test.dart`
+Expected: all tests pass; analyze reports "No issues found!".
+
+Moving `now` to `clock.now` can make an existing test fail. If one does, find out why first. A fixture time read from
+`DateTime.now()` against the fake clock is the likely cause. Ledger the ruling, and fix the fixture rather than the
+widget.
+
+- [ ] **Step 6: Commit (only once Diego asks)**
+
+```bash
+git add desktop/lib/super_tabs/model.dart desktop/lib/super_tabs/message_receipt.dart desktop/test/super_tab_receipt_test.dart
+git commit -m "feat(desktop): a queued message that is not getting in reads Not delivered yet"
+```
+
+---
+
+### Task 4: The CLI host names a failing claim
+
+**Files:**
+- Modify: `cli/src/superTabs/host.ts`: the `deliveryRetries` type (~line 271), the claim catch and the `listRead` log
+ (~lines 760-790), and the constants near `DELIVERY_RETRY_MS` (~line 170).
+- Test: `cli/src/superTabs/host.spec.ts`, in `describe('a delivery poll or claim that keeps failing', …)`
+
+**Interfaces:**
+- Consumes the existing `retryLater(key, list?)`, which returns `{ failures, retryAt, since, list? }`.
+- Produces `const CLAIM_FAILURE_LOG_MS = 5 * 60_000`.
+- Produces an optional `saidAt?: number` on the `deliveryRetries` entry type.
+
+- [ ] **Step 1: Write the failing tests**
+
+They use that describe's `failing(t, refuse)` helper and `t.wait(ms)`. The setup's ids are: delivery `delivery`, agent
+`agent`, session `harness`, Swarm `tab`. Its clock starts at 0.
+
+```ts
+it('says whose claim keeps failing, once, then at most every 5 minutes', async () => {
+ const t = setup(), host = t.open()
+ failing(t, (method, path) => method === 'POST' && path.endsWith('/deliveries/delivery'))
+ await host.tick()
+ const said = () => t.log.mock.calls.map(c => c[0] as string).filter(l => l.includes('delivery delivery') || l.includes('This Swarm is busy'))
+ expect(said()).toEqual(['Could not claim delivery delivery for agent agent (session harness) in Swarm tab: This Swarm is busy. Try again in a moment. (SWARM_BUSY). '
+ + 'Failing since 1970-01-01T00:00:00.000Z, 1 try; next try in 1 s.'])
+ for (let i = 0; i < 9; i++) { t.wait(30_000); await host.tick() } // 4.5 min of failing claims
+ expect(said()).toHaveLength(1)
+ t.wait(30_000); await host.tick() // 5 min since the last line
+ expect(said()).toHaveLength(2)
+ expect(said()[1]).toMatch(/^Could not claim delivery delivery .*Failing since 1970-01-01T00:00:00.000Z, \d+ tries; next try in 30 s\.$/)
+})
+it('says once when a claim that had failed gets in', async () => { /* refuse = true for 2 claims, then false; after the
+ successful tick the log has 'Delivery delivery claimed after 3 tries, 0 min.' exactly once, and no further line on later ticks */ })
+```
+
+- [ ] **Step 2: Run the tests to verify they fail**
+
+Run: `cd cli && npx vitest run src/superTabs/host.spec.ts -t "claim keeps failing|claim that had failed"`
+Expected: FAIL. The log holds the bare `This Swarm is busy. Try again in a moment.` line.
+
+- [ ] **Step 3: Implement it in `host.ts`**
+
+**The failure line.** In the claim's catch, where `retryLater(claimKey, listKey)` is already called, keep its returned
+entry. Log the named line when either holds:
+
+- it is the entry's first failure;
+- `now - saidAt >= CLAIM_FAILURE_LOG_MS`.
+
+Then set `saidAt = now`. The line has the same shape as the existing "Could not read the deliveries…" line:
+
+```
+Could not claim delivery <id> for agent <agentId> (session <harnessId.slice(0, 8)>) in Swarm <tabId>: <message><code>.
+Failing since <ISO>, <n> try|tries; next try in <s> s.
+```
+
+`<code>` is ` (CODE)` for a `SuperTabHostError` and empty otherwise.
+
+**No bare line.** The outer `if (listRead) this.log(message)` must not log again for a claim the inner catch already
+spoke for, whether or not the inner catch logged on this pass. Mark such errors, for example in a `WeakSet<object>` of
+claim errors, and skip them there. Other errors after a read list keep the bare line, as today.
+
+**The success line.** Where a claim succeeds (`this.deliveryRetries.delete(claimKey)` after `accept`), look first for an
+entry with `failures > 0`. If there is one, log
+`Delivery <id> claimed after <failures + 1> tries, <Math.round((now - since) / 60_000)> min.`
+
+Add a comment naming prod 10-30: 223 bare lines in 2 hours.
+
+- [ ] **Step 4: Run the tests and the typecheck**
+
+Run: `cd cli && npx vitest run src/superTabs/host.spec.ts && npm run typecheck`
+Expected: the whole file passes, including the existing backoff tests; tsc exits 0.
+
+- [ ] **Step 5: Commit (only once Diego asks)**
+
+```bash
+git add cli/src/superTabs/host.ts cli/src/superTabs/host.spec.ts
+git commit -m "fix(cli): a Swarm claim that keeps failing is named in the log"
+```
+
+---
+
+### Task 5: Docs, and the final checks
+
+**Files:**
+- Modify: `docs/swarms.md`. Add a short paragraph after **Lead updates**, or wherever delivery receipts are described.
+ Grep "Not sent" first. It says, plainly:
+ - a message that its recipient's online computer cannot take within a minute, or has not picked up in two, reads
+ **Not delivered yet**;
+ - Harness keeps trying;
+ - prod watch alarms on `claims failing`.
+
+ Name the incident date.
+
+- [ ] **Step 1: Write the paragraph.**
+
+- [ ] **Step 2: Run every affected check together, and record each result**
+
+Run:
+
+- `B` (the whole integration file)
+- the `deliverySettle.test.ts` unit run
+- `cd backend && npm run typecheck`
+- `cd desktop && flutter test test/super_tab_receipt_test.dart && flutter analyze lib/super_tabs`
+- `cd cli && npx vitest run src/superTabs/host.spec.ts && npm run typecheck`
+
+Expected: all pass. `host.ts` is not under `src/core`, `src/services` or `src/harnessd`, so no 100% coverage gate
+applies.
+
+- [ ] **Step 3: Commit (only once Diego asks)**
+
+```bash
+git add docs/swarms.md docs/superpowers/specs/2026-10-30-swarm-stuck-delivery-visible-design.md docs/superpowers/plans/2026-10-30-swarm-stuck-delivery-visible.md
+git commit -m "docs(swarm): Not delivered yet and the claims failing alarm"
+```
diff --git a/docs/superpowers/specs/2026-10-30-swarm-stuck-delivery-visible-design.md b/docs/superpowers/specs/2026-10-30-swarm-stuck-delivery-visible-design.md
new file mode 100644
index 000000000..49eae5f8d
--- /dev/null
+++ b/docs/superpowers/specs/2026-10-30-swarm-stuck-delivery-visible-design.md
@@ -0,0 +1,241 @@
+# A Swarm message that cannot be delivered says so
+
+Status: design · 2026-10-30 · owner Diego · approved in chat 2026-10-30
+
+## Why
+
+On 2026-10-30 no message a person sent their own Buddy reached it from 09:06 to about 11:01 (VN). Backend 1.3.71
+(#2178) made a Buddy delivery's claim touch the same part twice with one version. The second write always conflicted,
+`retryTransient` gave up after 8 attempts, and the claim answered `SWARM_BUSY`. Hotfix #2295 fixed the cause.
+
+The delivery itself was never lost. `SuperTabDelivery` is a durable queue (`queued → claimed → delivered`): the row
+stayed `queued` and went in once claims worked again. What failed was that **nobody could tell**:
+
+- **The sender** saw nothing. `desktop/lib/super_tabs/message_receipt.dart` shows a line only while a message sends
+ (under 3 s), when it was not sent or could not be confirmed, or when the recipient's computer is offline. A `queued`
+ message to an online computer has no line, which reads as delivered.
+- **The backend** answered every claim with the same busy it gives during a deploy. `stuck()` in
+ `backend/src/superTabs/deliverySettle.ts` returns false for every reachable `queued` delivery, so the settle pass
+ never looked at it. Nothing raised an alarm.
+- **The host** logged 223 bare `[super-tabs] This Swarm is busy. Try again in a moment.` lines (`host.ts` logs a failed
+ claim's message alone when its list was read). None of them names the delivery, the agent or how long it has failed.
+
+The trigger was Diego asking whether his Buddy had died. This design makes the same failure visible to the sender
+within about a minute, and to prod watch within five.
+
+The herdr-core idea of a seq/missed event stream was considered and dropped: deliveries are already durable rows, so
+nothing is missed. The problem was a stuck delivery that nobody noticed.
+
+## Scope
+
+In:
+
+1. Backend records each failed claim on its delivery row and clears the record when a claim succeeds.
+2. The delivery settle pass logs an alarm line for a Swarm whose queued deliveries keep failing to be claimed.
+3. Desktop and the web app (one Flutter receipt) show **Not delivered yet** for a queued message that is not getting in.
+4. The CLI host's log names the failing delivery and is rate limited.
+
+Out:
+
+- The phone app (`mobile/` is its own package). It gets the same rule in a follow-up, from the same fields.
+- The round dial and the Harness Pro.
+- A Buddy that is down. Its chat already offers **Start**, and a delivery to a stopped agent is not this failure.
+- Failed acknowledgements (`ack`). See Open risks.
+- Changing what `retryTransient` or the error handler answer. A claim that keeps conflicting still answers
+ `SWARM_BUSY`, and the host still retries it.
+
+## Design
+
+### 1. Backend: a failed claim is written on its delivery
+
+**Fields.** `SuperTabDelivery` (`backend/prisma/schema.prisma`) gains four optional fields. Mongo needs no migration,
+and older rows read them as null.
+
+| Field | Type | Meaning |
+| --- | --- | --- |
+| `claimFailingSince` | `DateTime?` | The first failed claim since the last successful one |
+| `claimFailedAt` | `DateTime?` | The latest failed claim |
+| `claimFailures` | `Int?` | Failed claims since `claimFailingSince` |
+| `claimFailure` | `String?` | The latest failure's code (today always `SWARM_BUSY`) |
+
+**Recording.** `SuperTabStore.delivery` (`backend/src/superTabs/store.ts`) runs its `transaction(...)` inside a catch,
+for `action.type === 'claim'` only, and only for a request that got past the claim's authorization and status checks on
+some attempt. `deliver()` sets a flag right after those checks, so a busy answer that came before them (the first read,
+under load) writes nothing on a delivery the caller may not touch. The catch records when the thrown error is:
+
+- a `SuperTabError` with code `SWARM_BUSY`. `retryTransient` (`retry.ts`) ends both kinds of retryable failure that way:
+ a write conflict after 8 attempts, and an expired transaction (`P2028`) after 2. So the claim never sees a raw
+ `P2028`, and nothing else needs matching.
+
+It records with two standalone single-document writes, **outside** the failed transaction. Both match
+`{ id: deliveryId, tabId, agentId, status: 'queued' }`.
+
+1. An `updateMany` that also matches a missing or null `claimFailingSince` and sets it to `now`. With Prisma on Mongo,
+ a field an older row never had needs `isSet: false` beside `null`. This keeps the first failure's time from ever
+ moving later.
+2. An `updateMany` that sets `claimFailedAt = now` and `claimFailure = code`, and increments `claimFailures`. A missing
+ count counts from 0.
+
+Those writes touch no Swarm part and no other row, so they cannot meet the conflict that failed the claim. They are
+capped at 2 s (`RECORD_FAILED_CLAIM_MS`), so a struggling database never holds the busy answer back. They are best
+effort: if it fails, that is logged as `warn [swarm-deliveries] could not record a failed claim` `{tabId, deliveryId,
+error}` and the original error is still thrown unchanged. The client's answer (503 `SWARM_BUSY` with Retry-After) does
+not change.
+
+Other refusals (`DELIVERY_REVOKED`, `DELIVERY_RESOLVED`, `CLAIM_CONFLICT` and so on) are answers, not failures, and
+record nothing.
+
+**Clearing.** Only a successful claim clears the record: its `tx.superTabDelivery.update` in `delivery()` unsets all four fields
+(`{ unset: true }`). It must not write `null`, because Mongo's `$inc` refuses a null count, so the next failure's
+increment would throw.
+
+Every other write that moves a delivery out of `queued` leaves the fields as they are. That covers an ack, a cancel, the
+settle pass and archiving. Every reader below ignores the fields unless the status is `queued`, so the leftovers are
+harmless.
+
+**Reading.** `pageDeliveries` (`backend/src/superTabs/pageReads.ts`) projects `claimFailingSince` and
+`claimFailures` beside the fields it reads today, and `PageDelivery` gains them. Every reader of a page already reads
+its deliveries this way, so the Desktop and web receipts get the fields with no new request. `claimFailure` and
+`claimFailedAt` stay server-side: they are for the alarm and for diagnosis, and the person has nothing to do with a
+code.
+
+**Cost.** One extra single-document write per failed claim. A host retries a failing claim at most every 30 s
+(`DELIVERY_RETRY_MAX_MS`). In the 10-30 incident that would have been about 240 writes across 2 hours for one
+person's Buddy, against a delivery index that already exists. No new index is needed: the alarm reads rows the settle
+pass already reads.
+
+### 2. Backend: the settle pass raises the alarm
+
+`settleStuckDeliveries` already reads every open delivery every 5 minutes, page by page, grouped by Swarm. After it
+settles a Swarm's rows, it also counts that Swarm's **failing** rows from the same page. A row is failing when:
+
+- `status === 'queued'`,
+- `claimFailingSince` is at least `CLAIM_FAILING_ALARM_MS` (5 minutes) before `now`, and
+- `claimFailedAt` is within `CLAIM_FAILING_ALARM_MS` of `now`, so claims are still failing rather than abandoned.
+
+For each Swarm with at least one failing row, the pass logs one line:
+
+```
+warn [swarm-deliveries] claims failing { tabId, deliveries, oldestFailingSince, codes }
+```
+
+- `deliveries` is the count.
+- `oldestFailingSince` is the earliest `claimFailingSince` as ISO text.
+- `codes` is the distinct `claimFailure` values.
+
+The line carries no message text, no agent names and no person ids. The tab id is already in every Swarm log.
+
+This is the line prod watch alarms on. On 10-30 it would first have been logged at the first pass after 09:11 (5 minutes
+of failing). The pass **does not cancel** these deliveries:
+
+- A cancelled delivery reads **Not sent** and its text is gone from the recipient.
+- Once the cause is fixed, a still-queued one goes in, late but whole.
+- `stuck()` is unchanged.
+
+`openDeliveries` must also return the two new date fields as Dates, as it already does for `holdSince`.
+
+### 3. Desktop and web: Not delivered yet
+
+`SuperTabDelivery` (`desktop/lib/super_tabs/model.dart`) reads `claimFailingSince` (a date) and `claimFailures` (an
+int). Both are null from an older backend.
+
+`SuperTabMessageReceipt.build` gains one branch. It goes after **offline** and before **Sending…**, so an offline
+computer still says it is offline. It applies when the receipt delivery's `status == 'queued'`, the recipient's
+computer is not offline, the Swarm is readable (`!c.backendUnreachable`), the recipient is in the Swarm and running
+(`downSince == null`), and either of these holds:
+
+| Case | Condition | Note text |
+| --- | --- | --- |
+| Claims failing | `claimFailingSince` is at least 60 s ago (`superTabClaimFailingNotice`) | "$host is online but couldn't take this message yet. Harness keeps trying, and it goes in as soon as it can." |
+| Not picked up | the message's `createdAt` is at least 2 min ago (`superTabQueuedNotice`); the note follows whether `claimFailingSince` is set, so the line never drops out when a first claim failure arrives | "$host is online but hasn't picked this message up yet. Harness keeps trying, and it goes in as soon as it can." |
+
+- **Title:** "Not delivered yet". It uses the warn colour (`grid.AppPalette.warn`) and the clock icon the app already
+ uses for waiting. If there is none, the plan picks the nearest existing `AppIcons` mark. No new icon.
+- **No actions.** The sender has nothing to do. **Send again** would make a second delivery behind the same failure.
+- **`$host`** is the machine name, as the offline note already names it
+ (`machine?['name'] ?? widget.machineName(agent.machineId)`).
+- **The line disappears by itself** once the delivery leaves `queued`, at the next page read. That is one poll, or a
+ push.
+- **Clock.** The widget's `_clock(needed)` 500 ms timer runs today only while fresh or offline. It must also run while a
+ queued delivery is below either threshold, so the line appears without another rebuild. When neither threshold can
+ still be crossed (not queued, or already shown), no timer runs.
+
+The web app gets the same line, because it renders the same widget.
+
+The Leads conversation and a Buddy chat follow the same rule. Their receipts read the same delivery.
+
+### 4. CLI host: the log names the delivery
+
+In `cli/src/superTabs/host.ts`, the branch that logs `message` alone when the list was read but a claim failed
+(`if (listRead) this.log(message)`) logs this instead:
+
+```
+Could not hand delivery <deliveryId> to agent <agentId> (session <harnessId 8>) in Swarm <tabId>: <message> (<code>).
+Failing since <ISO>, <n> tries; next try in <s> s.
+```
+
+It uses the `deliveryRetries` entry `retryLater(claimKey, listKey)` already keeps (`since`, `failures`, `retryAt`).
+That is the same shape as the existing "Could not read the deliveries…" line.
+
+**Rate.**
+
+- The line names the delivery, not the claim: `accept()` also acknowledges and downloads files, and an ack that keeps
+ failing is the open risk below.
+- It logs on a delivery's first failure, at once when the error's text or code differs from the last line, and
+ otherwise at most once every 5 minutes per delivery while it keeps failing.
+- When a delivery that had failed goes through, the host logs `Delivery <id> went through after <n> tries, <m> min.`
+ once.
+- A failure that is not a `SuperTabHostError` (a network error) is logged the same way, with no code.
+
+At the 10-30 rate this is about 25 named lines instead of 223 bare ones.
+
+The message text is never logged, as today.
+
+## Compatibility
+
+- **Additive fields.** An older Desktop, web build or CLI ignores them. An older backend sends none, and the receipt
+ then uses only the 2-minute rule.
+- **No change to any request or answer shape** beyond the two added fields on page deliveries.
+- **Rollback** leaves the fields unread on existing rows. They are harmless and clear on the next successful claim.
+
+## Testing
+
+**Backend** (`backend/src/superTabs/*.test.ts`, vitest):
+
+- A claim whose transaction throws `SWARM_BUSY` records the four fields with `claimFailures = 1`. A second failure keeps
+ `claimFailingSince` and increments the count.
+- A `DELIVERY_REVOKED` records nothing.
+- A successful claim after failures clears all four fields.
+- When the recording write fails, the original error is still thrown and a warn is logged.
+- The settle pass logs `claims failing` for a queued row failing 5 min or more and still failing. It logs nothing for one
+ failing under 5 min, or one whose last failure is older than 5 min. It cancels none of them.
+- `pageDeliveries` returns `claimFailingSince` and `claimFailures`.
+
+**Backend integration** (`backend/integration/`): a claim forced to answer `SWARM_BUSY`, as
+`harnessSharesList.test.ts` forces `readParts`. The page read then carries `claimFailingSince`, and a later real claim
+clears it.
+
+**Desktop** (`desktop/test/super_tab_receipt_test.dart`):
+
+- Each case shows "Not delivered yet" with its own note at and after its threshold, and nothing before it.
+- Offline wins over it.
+- A delivered message shows nothing.
+- An older backend's delivery (no fields) uses the 2-minute rule.
+- A stopped agent (`downSince` set) shows neither case.
+- The line appears by the widget's own clock, without a new snapshot.
+
+**CLI** (`cli/src/superTabs/host.spec.ts`):
+
+- A claim failing with `SWARM_BUSY` logs the named line once.
+- Further failures inside 5 minutes log nothing; the next one after 5 minutes logs again.
+- A different error inside the 5 minutes logs at once.
+- Success after failures logs the "went through after" line.
+
+## Open risks
+
+- **Failed acks.** An ack that keeps failing leaves a delivery `claimed`, although the agent has the text. After
+ `CLAIM_QUIET_MS` (10 min) the settle pass cancels it, and the sender reads **Not sent** for a message that arrived. This
+ design does not change that. It is a separate fix, worth raising after this ships.
+- **The 2-minute rule's false positives.** These are agents that are running but legitimately not reading: a host that
+ updates for longer than 2 minutes, or `UPDATE_REQUIRED` backoff. In each of these the message truly is not delivered
+ yet, so the line is honest. The note says Harness keeps trying, which is true.
diff --git a/docs/swarms.md b/docs/swarms.md
index fab92146f..822e577b5 100644
--- a/docs/swarms.md
+++ b/docs/swarms.md
@@ -640,6 +640,30 @@ summary, or `text: null` when nothing merits another message. Human questions,
serious blockers and completion are reported immediately. This does not start
a second summarizing model or forward every terminal turn.
+## A message that is not getting in
+
+A message waits as a delivery until its recipient's computer takes it. Your newest message shows a line under it only
+while it sends, or when it needs attention.
+
+**Not delivered yet** is that line, in amber, with no actions. It shows while the recipient's computer is online and
+either:
+
+- has failed to take the message for a minute, or
+- has not picked it up for two minutes while the agent runs.
+
+Harness keeps trying, and the line goes once the message is in. An offline computer still reads as offline instead.
+
+Behind it, the backend writes each claim that ends busy on the delivery (`claimFailingSince`, `claimFailures`) and
+clears that record when a claim gets in. Every 5 minutes the delivery pass logs `[swarm-deliveries] claims failing`
+for a Swarm whose queued messages have failed to be claimed for 5 minutes and still fail. Prod watch alarms on that
+line. The pass cancels none of them: once the cause is fixed, they go in late but whole.
+
+The host's log names each delivery that keeps failing (delivery, agent, since when, how many tries). It logs at once,
+again when the error changes, and otherwise at most every 5 minutes, and it says once when the delivery gets in.
+
+This comes from 2026-10-30. A backend release made every claim of a person's message to their own Buddy end busy for
+two hours. Nothing was lost, but nobody could tell.
+
## Switching Swarms off on a computer
Swarms are on by default. To run a computer's daemon without them, set