Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

Backend: usage receipts live 4 days, not 36; old ones swept behind a receipt rule (#2145)

harness-for-autonomous[bot]91b32d3bb2fd
diff --git a/backend/README.md b/backend/README.md
index 1308e76b4..2d2b9f4da 100644
--- a/backend/README.md
+++ b/backend/README.md
@@ -227,7 +227,7 @@ The product usage collections have distinct purposes:
| --- | --- | --- |
| `product_usage_preferences` | Account ID, enabled flag, revision | Saved account preference; no usage history |
| `product_usage_totals` | UTC day, allowlisted metric, sum (plus DB primary key) | 396 days, approximately 13 months |
-| `product_usage_receipts` | Random report UUID and expiration date | Scheduled expiry 36 days from the report day |
+| `product_usage_receipts` | Random report UUID, expiration date and its life in days (required on every new write once the worker's receipt sweep installs the collection rule) | Scheduled expiry 4 days from the report day (36 until 2026-10-29) |
| `product_usage_activity` | HMAC subject (`PRODUCT_USAGE_SUBJECT_KEY`), UTC day, surfaces, worked flag, sign-up day, staff flag, expiry | 10 days (TTL); deleted on opt-out and `/api/analytics/delete`; written only while sharing is on and the key is set |
No individual report body or arrival timestamp is stored. Receipt insertion and increments share a
diff --git a/backend/integration/oldUsageServer.test-support.ts b/backend/integration/oldUsageServer.test-support.ts
new file mode 100644
index 000000000..437e8b03f
--- /dev/null
+++ b/backend/integration/oldUsageServer.test-support.ts
@@ -0,0 +1,23 @@
+import type { PrismaClient } from '@prisma/client'
+import { incrementUsageTotals } from '../src/lib/productUsageWrite.js'
+
+const DAY = 86_400_000
+
+/**
+ * A server released before receipts lived 4 days (main at 3571a6cef, src/lib/productUsage.ts ingestProductUsage and
+ * commitReports): it accepts a report up to 35 days old and, in one transaction, looks its receipt up, writes it as
+ * `{ _id, expiresAt: day + 36 }` with no life, and adds its totals. [afterLookup] holds it between the lookup and the write.
+ */
+export async function oldServerIngest(db: PrismaClient, report: { id: string; day: string; counters: Record<string, number> },
+ now: Date, afterLookup?: () => Promise<void>): Promise<{ accepted: boolean; reason?: string }> {
+ const stamp = Date.parse(report.day), today = Math.floor(now.getTime() / DAY) * DAY
+ if (!(stamp <= today && stamp > today - 35 * DAY)) return { accepted: false, reason: 'EXPIRED' }
+ await db.$transaction(async tx => {
+ if (await tx.productUsageReceipt.findFirst({ where: { id: report.id }, select: { id: true } })) return
+ await afterLookup?.()
+ // The released call, word for word: this Prisma client writes no `life` when given none, as the released one could not.
+ await tx.productUsageReceipt.createMany({ data: [{ id: report.id, expiresAt: new Date(stamp + 36 * DAY) }] })
+ await incrementUsageTotals(tx, Object.entries(report.counters).map(([metric, value]) => ({ dayUtc: new Date(report.day), metric, value })))
+ })
+ return { accepted: true }
+}
diff --git a/backend/integration/productUsageReceipts.test.ts b/backend/integration/productUsageReceipts.test.ts
new file mode 100644
index 000000000..be0be30e0
--- /dev/null
+++ b/backend/integration/productUsageReceipts.test.ts
@@ -0,0 +1,234 @@
+import { randomUUID } from 'node:crypto'
+import { execFile } from 'node:child_process'
+import { promisify } from 'node:util'
+import { join } from 'node:path'
+import { tmpdir } from 'node:os'
+import { afterAll, beforeAll, beforeEach, expect, it, vi } from 'vitest'
+import { MongoMemoryReplSet } from 'mongodb-memory-server-core'
+import { PrismaClient } from '@prisma/client'
+import { oldServerIngest } from './oldUsageServer.test-support.js'
+
+// A fresh loopback replica set, created here; nothing reads an existing database. Receipts lived 36 days until 2026-10-29
+// (about 4M of them in production) and live 4 now. Here the shorter life was released on 30 October and the clock reads
+// 2 November, 03:00 UTC: every receipt below is seeded by this file.
+let replica: MongoMemoryReplSet
+let db: PrismaClient
+let usage: typeof import('../src/lib/productUsage.js')
+let sweep: typeof import('../src/lib/productUsageReceiptSweep.js')
+
+beforeAll(async () => {
+ replica = await MongoMemoryReplSet.create({ binary: { version: '7.0.24', downloadDir: join(tmpdir(), 'harness-super-tab-mongodb-binaries') },
+ replSet: { count: 1, ip: '127.0.0.1', storageEngine: 'wiredTiger' } })
+ const uri = replica.getUri(`product_usage_receipts_${randomUUID().replaceAll('-', '_')}`)
+ await promisify(execFile)(join(process.cwd(), 'node_modules/.bin/prisma'), ['db', 'push', '--skip-generate'], {
+ env: { ...process.env, DATABASE_URL: uri }, timeout: 60000,
+ })
+ process.env.DATABASE_URL = uri
+ vi.resetModules()
+ db = (await import('../src/lib/prisma.js')).prisma
+ usage = await import('../src/lib/productUsage.js')
+ sweep = await import('../src/lib/productUsageReceiptSweep.js')
+}, 120000)
+afterAll(async () => {
+ await db?.$disconnect()
+ await replica?.stop()
+})
+
+const DAY = 86_400_000
+const now = new Date('2026-11-02T03:00:00Z')
+const at = (day: string, days: number) => new Date(Date.parse(day) + days * DAY)
+/** Every UTC day from [first] to [last], both included. */
+const days = (first: string, last: string) => Array.from({ length: (Date.parse(last) - Date.parse(first)) / DAY + 1 },
+ (_, i) => at(first, i).toISOString().slice(0, 10))
+const ids = (style: string, day: string) => [0, 1, 2].map(n => `${style}-${day}-${n}`)
+
+// Three receipts a report day. Old servers wrote day + 36 up to the release (30 October included), new ones day + 4.
+const OLD_DAYS = days('2026-09-28', '2026-10-30')
+const NEW_DAYS = days('2026-10-30', '2026-11-02')
+// The sweep removes the old receipts whose 4-day life is over (report days 3 to 29 October), apart from those Mongo's expiry
+// removes within 5 days anyway (day + 36 up to 7 November). The 30th's new life ends on 3 November, after this clock.
+const KEPT = [...days('2026-09-28', '2026-10-02'), '2026-10-30'].flatMap(day => ids('old', day))
+ .concat(NEW_DAYS.flatMap(day => ids('new', day)), ['expired-old-2026-09-25'])
+const TOTALS = [{ dayUtc: new Date('2026-10-20'), metric: 'turns_started', value: 41 }, { dayUtc: new Date('2026-11-01'), metric: 'turns_started', value: 7 }]
+
+beforeEach(async () => {
+ // The database as it is before the release: no receipt rule yet.
+ await db.$runCommandRaw({ collMod: 'product_usage_receipts', validator: {}, validationLevel: 'off' })
+ await db.productUsageReceipt.deleteMany()
+ await db.productUsageTotal.deleteMany()
+ await db.productUsageReceipt.createMany({ data: [
+ ...OLD_DAYS.flatMap(day => ids('old', day).map(id => ({ id, expiresAt: at(day, 36) }))),
+ ...NEW_DAYS.flatMap(day => ids('new', day).map(id => ({ id, expiresAt: at(day, 4), life: 4 }))),
+ // Expired already (1 November); Mongo's expiry removes it, not the cleanup.
+ { id: 'expired-old-2026-09-25', expiresAt: at('2026-09-25', 36) },
+ ] })
+ await db.productUsageTotal.createMany({ data: TOTALS })
+})
+
+/** Resolves once [ready] holds, checking every 10 ms. */
+async function until(ready: () => Promise<boolean>): Promise<void> {
+ while (!await ready()) await new Promise(resolve => setTimeout(resolve, 10))
+}
+/** How many installs of a rule on the receipts wait for a lock now. */
+async function installsWaiting(): Promise<number> {
+ const admin = new PrismaClient({ datasources: { db: { url: replica.getUri('admin') } } })
+ try {
+ const ops = await admin.$runCommandRaw({ aggregate: 1, pipeline: [{ $currentOp: {} },
+ { $match: { 'command.collMod': 'product_usage_receipts', waitingForLock: true } }], cursor: {} }) as { cursor: { firstBatch: unknown[] } }
+ return ops.cursor.firstBatch.length
+ } finally { await admin.$disconnect() }
+}
+const deferred = () => { let resolve!: () => void; const promise = new Promise<void>(r => { resolve = r }); return { promise, resolve } }
+const pass = (at = now) => sweep.sweepOldReceipts(db, at, { chunk: 10, chunks: 4, pauseMs: 30 })
+const left = async () => (await db.productUsageReceipt.findMany({ select: { id: true } })).map(r => r.id).sort()
+const totals = async () => db.productUsageTotal.findMany({ select: { dayUtc: true, metric: true, value: true }, orderBy: { dayUtc: 'asc' } })
+
+it('removes only old receipts outside the new life, a few chunks a pass with a pause, then nothing', async () => {
+ const started = performance.now()
+ expect(await pass()).toEqual({ removed: 40, done: false })
+ expect(performance.now() - started).toBeGreaterThanOrEqual(3 * 30)
+ expect(await pass()).toEqual({ removed: 40, done: false })
+ expect(await pass()).toEqual({ removed: 1, done: false })
+ expect(await left()).toEqual(KEPT.sort())
+ // The 30 October's old receipts are still inside their new life (to 3 November): kept, and the sweep goes on.
+ expect(await pass()).toEqual({ removed: 0, done: false })
+ const later = new Date('2026-11-03T03:00:00Z')
+ expect(await pass(later)).toEqual({ removed: 3, done: true })
+ expect(await pass(later)).toEqual({ removed: 0, done: true })
+ expect(await left()).toEqual(KEPT.filter(id => !id.startsWith('old-2026-10-30')).sort())
+ expect(await totals()).toEqual(TOTALS)
+})
+
+it('sends commands to the receipts collection only', async () => {
+ await db.$runCommandRaw({ profile: 2 })
+ const since = new Date()
+ expect(await pass()).toMatchObject({ removed: 40 })
+ await db.$runCommandRaw({ profile: 0 })
+ const answer = await db.$runCommandRaw({ find: 'system.profile', filter: { ts: { $gt: { $date: since.toISOString() } } },
+ projection: { ns: 1 }, batchSize: 100_000, singleBatch: true }) as { cursor: { firstBatch: Array<{ ns: string }> } }
+ const collections = new Set(answer.cursor.firstBatch.map(entry => entry.ns.split('.').slice(1).join('.')))
+ expect([...collections].filter(name => name !== 'system.profile' && name !== '$cmd')).toEqual(['product_usage_receipts'])
+})
+
+const late = (day = '2026-10-29') => ({ id: randomUUID(), day, counters: { turns_ended: 2 } })
+const turnsEnded = async (day: string) =>
+ (await db.productUsageTotal.findUnique({ where: { dayUtc_metric: { dayUtc: new Date(day), metric: 'turns_ended' } } }))?.value
+const drain = async (at = now) => { while ((await pass(at)).removed) { /* the next chunks */ } }
+const account = '6a0000000000000000000c01'
+const signedIn = (report: ReturnType<typeof late>, at = now) => usage.ingestProductUsage(account, { schemaVersion: 1, revision: 0,
+ ...report, context: { surface: 'cli', release: '0.4.92', platform: 'macos', channel: 'production' } }, at)
+
+it('an old server still running cannot count a late report twice across the sweep (review of c20be500d)', async () => {
+ // 2 November 00:01: no report for today yet, and an old server takes a 29 October report.
+ const midnight = new Date('2026-11-02T00:01:00Z'), report = late()
+ expect(await oldServerIngest(db, report, midnight)).toEqual({ accepted: true })
+ expect(await turnsEnded(report.day)).toBe(2)
+ await drain(midnight)
+ expect(await db.productUsageReceipt.findUnique({ where: { id: report.id } })).toBeNull()
+ // The client retries through the old server: it may no longer write a receipt, so nothing is added.
+ await expect(oldServerIngest(db, report, midnight)).rejects.toThrow(/validation/i)
+ expect(await signedIn(report, midnight)).toEqual({ accepted: false, reason: 'EXPIRED' })
+ expect(await turnsEnded(report.day)).toBe(2)
+})
+
+it('removes nothing until every receipt write must carry the new life', async () => {
+ const refusing = new Proxy(db, { get: (target, key) => key === '$runCommandRaw'
+ ? (command: Record<string, unknown>) => 'collMod' in command ? Promise.reject(new Error('not authorized')) : target.$runCommandRaw(command as never)
+ : Reflect.get(target, key) })
+ await expect(sweep.sweepOldReceipts(refusing, now, { chunk: 10, chunks: 4, pauseMs: 0 })).rejects.toThrow('not authorized')
+ expect((await left()).length).toBe(112)
+ await pass()
+ const [info] = (await db.$runCommandRaw({ listCollections: 1, filter: { name: 'product_usage_receipts' } }) as
+ { cursor: { firstBatch: Array<{ options: { validator?: unknown; validationLevel?: string } }> } }).cursor.firstBatch
+ expect(info.options).toMatchObject({ validator: { $jsonSchema: { required: ['life'] } }, validationLevel: 'strict', validationAction: 'error' })
+})
+
+it('mixed versions count a report inside the window once, whichever server takes each copy', async () => {
+ // Before the sweep: the old server takes it first; the new server sees its receipt.
+ const first = late('2026-11-01')
+ expect(await oldServerIngest(db, first, now)).toEqual({ accepted: true })
+ expect(await signedIn(first)).toEqual({ accepted: true })
+ await drain()
+ expect(await db.productUsageReceipt.findUnique({ where: { id: first.id } })).not.toBeNull()
+ expect(await signedIn(first)).toEqual({ accepted: true })
+ // After it: the old server is refused, the new server counts it, and the old server then sees the new receipt.
+ const second = { ...late('2026-11-01'), counters: { turns_ended: 5 } }
+ await expect(oldServerIngest(db, second, now)).rejects.toThrow(/validation/i)
+ expect(await signedIn(second)).toEqual({ accepted: true })
+ expect(await oldServerIngest(db, second, now)).toEqual({ accepted: true })
+ expect(await turnsEnded('2026-11-01')).toBe(7)
+})
+
+it('an old server between its receipt lookup and its write while the rule goes up counts at most once', async () => {
+ const report = late(), lookedUp = deferred(), proceed = deferred()
+ const old = oldServerIngest(db, report, now, async () => { lookedUp.resolve(); await proceed.promise })
+ await lookedUp.promise
+ // The rule's install queues behind the old server's open transaction, which then writes: it commits under the old rule,
+ // or Mongo ends it for the lock it now waits on. Either way, nothing after the install is written without a life.
+ let settled = false
+ const sweeping = pass().finally(() => { settled = true })
+ await until(async () => settled || (await installsWaiting()) > 0)
+ expect(settled).toBe(false)
+ proceed.resolve()
+ const [first] = await Promise.allSettled([old, sweeping])
+ await drain()
+ await Promise.allSettled([oldServerIngest(db, report, now), oldServerIngest(db, report, now)])
+ expect(await turnsEnded(report.day)).toBe(first.status === 'fulfilled' ? 2 : undefined)
+ await expect(oldServerIngest(db, late(), now)).rejects.toThrow(/validation/i)
+})
+
+it('two workers at once (a rolling deploy) still remove each receipt once and nothing else', async () => {
+ const passes = await Promise.all([pass(), pass(), pass()])
+ expect(passes.map(p => p.done)).toEqual([false, false, false])
+ while ((await pass()).removed) { /* the rest */ }
+ expect(await left()).toEqual(KEPT.sort())
+ expect(await totals()).toEqual(TOTALS)
+})
+
+it('the worker\'s sweep runs a pass a minute and stops by itself once nothing is left', async () => {
+ const lines: string[] = []
+ const log = { info: (message: string) => lines.push(message), warn: (message: string) => lines.push(message) }
+ await db.productUsageReceipt.deleteMany({ where: { id: { startsWith: 'old-2026-10-30' } } })
+ const stop = sweep.startReceiptSweep(db, { everyMs: 20, chunk: 50, chunks: 1, pauseMs: 0, now: () => now, log })
+ try {
+ await vi.waitFor(() => expect(lines.at(-1)).toMatch(/done/), { timeout: 5000 })
+ const settled = lines.length
+ await new Promise(resolve => setTimeout(resolve, 100))
+ expect(lines.length).toBe(settled)
+ } finally { stop() }
+ expect(lines.filter(line => line.includes('pass'))).toHaveLength(2)
+ expect(await left()).toEqual(KEPT.filter(id => !id.startsWith('old-2026-10-30')).sort())
+})
+
+it('a failed pass is logged and the next minute\'s pass carries on', async () => {
+ const lines: string[] = []
+ const log = { info: (message: string) => lines.push(message), warn: (message: string) => lines.push(message) }
+ await db.productUsageReceipt.deleteMany({ where: { id: { startsWith: 'old-2026-10-30' } } })
+ let calls = 0
+ // The database refuses the first command only, as a lost connection or a primary stepping down would.
+ const flaky = new Proxy(db, { get: (target, key) => key === '$runCommandRaw'
+ ? (command: never) => calls++ ? target.$runCommandRaw(command) : Promise.reject(new Error('connection closed'))
+ : Reflect.get(target, key) })
+ const stop = sweep.startReceiptSweep(flaky, { everyMs: 20, chunk: 100, chunks: 1, pauseMs: 0, now: () => now, log })
+ try {
+ await vi.waitFor(() => expect(lines.at(-1)).toMatch(/done/), { timeout: 5000 })
+ } finally { stop() }
+ expect(lines[0]).toMatch(/pass failed; retrying next minute/)
+ expect(await left()).toEqual(KEPT.filter(id => !id.startsWith('old-2026-10-30')).sort())
+})
+
+it('counting stays exact: a report inside the window counts once across the sweep, an older one adds nothing', async () => {
+ const context = { surface: 'cli', release: '0.4.92', platform: 'macos', channel: 'production' } as const
+ const report = { schemaVersion: 1 as const, revision: 0, id: randomUUID(), day: '2026-11-01', context, counters: { turns_ended: 2 } }
+ const turnsEnded = async () => (await db.productUsageTotal.findUnique({ where: { dayUtc_metric: { dayUtc: new Date('2026-11-01'), metric: 'turns_ended' } } }))?.value
+ expect(await usage.ingestProductUsage('6a0000000000000000000c01', report, now)).toEqual({ accepted: true })
+ expect(await db.productUsageReceipt.findUnique({ where: { id: report.id } })).toEqual({ id: report.id, expiresAt: new Date('2026-11-05'), life: 4 })
+ while ((await pass()).removed) { /* every old receipt outside the new life */ }
+ expect(await usage.ingestProductUsage('6a0000000000000000000c01', report, now)).toEqual({ accepted: true })
+ expect(await turnsEnded()).toBe(2)
+
+ const late = { ...report, id: randomUUID(), day: '2026-10-30' }
+ expect(await usage.ingestProductUsage('6a0000000000000000000c01', late, now)).toEqual({ accepted: false, reason: 'EXPIRED' })
+ expect(await db.productUsageReceipt.findUnique({ where: { id: late.id } })).toBeNull()
+ expect(await db.productUsageTotal.findMany({ where: { dayUtc: new Date('2026-10-30') } })).toEqual([])
+}, 60000)
diff --git a/backend/integration/schemaRelease.test.ts b/backend/integration/schemaRelease.test.ts
index beac5164e..538b5b0f9 100644
--- a/backend/integration/schemaRelease.test.ts
+++ b/backend/integration/schemaRelease.test.ts
@@ -7,6 +7,7 @@ import { MongoMemoryReplSet } from 'mongodb-memory-server-core'
import { PrismaClient } from '@prisma/client'
import { ensureIndex } from '../src/lib/db/indexes.js'
import { releaseSchema } from '../src/lib/db/schemaRelease.js'
+import { ensureReceiptRule } from '../src/lib/productUsageReceiptSweep.js'
import { DIRECT, readLight, readParts } from '../src/superTabs/cachedReads.js'
import * as parts from '../src/superTabs/parts/index.js'
import { swarm } from '../src/superTabs/read/fixture.test-support.js'
@@ -142,6 +143,35 @@ describe('a release\'s schema step', () => {
}, 240_000)
})
+describe('the usage receipt rule (lib/productUsageReceiptSweep.ts)', () => {
+ const rule = async (db: PrismaClient) => ((await db.$runCommandRaw({ listCollections: 1, filter: { name: 'product_usage_receipts' } }) as
+ { cursor: { firstBatch: Array<{ options: Record<string, unknown> }> } }).cursor.firstBatch[0]?.options)
+
+ it('survives a release, with or without a schema change, and the release does not touch it', async () => {
+ const { db, target } = await database()
+ await releaseSchema(db, target)
+ await ensureReceiptRule(db)
+ const installed = await rule(db)
+ expect(installed).toMatchObject({ validator: { $jsonSchema: { required: ['life'] } }, validationLevel: 'strict', validationAction: 'error' })
+ const since = new Date()
+ await releaseSchema(db, target)
+ await releaseSchema(db, { ...target, schema: await schemaWith(groupIndex('@@index([tabId, purgeAt])')) })
+ expect(await rule(db)).toEqual(installed)
+ expect((await indexChanges(db, since)).filter(change => change.includes('product_usage_receipts'))).toEqual([])
+ await db.$disconnect()
+ }, 240_000)
+
+ it('is installed once: a second install sends no change', async () => {
+ const { db, target } = await database()
+ await releaseSchema(db, target)
+ await ensureReceiptRule(db)
+ const since = new Date()
+ await ensureReceiptRule(db)
+ expect(await indexChanges(db, since)).toEqual([])
+ await db.$disconnect()
+ }, 240_000)
+})
+
describe('ensureIndex', () => {
it('creates the collection and its index when neither exists', async () => {
const { db } = await database()
diff --git a/backend/prisma/schema.prisma b/backend/prisma/schema.prisma
index 18b12bfe2..d667eb3b8 100644
--- a/backend/prisma/schema.prisma
+++ b/backend/prisma/schema.prisma
@@ -560,6 +560,10 @@ model ProductUsageTotal {
model ProductUsageReceipt {
id String @id @map("_id")
expiresAt DateTime
+ /// Days the receipt lives past its report day (types/productUsageReceipt.ts). Required by the collection's rule once the
+ /// receipt sweep installs it (lib/productUsageReceiptSweep.ts): a server that writes receipts without it cannot count.
+ /// Absent on receipts written before 2026-10-29.
+ life Int?
@@map("product_usage_receipts")
/// Expires these rows (lib/productUsage.ts ensureProductUsageExpiry). Declared so db push keeps it; the worker sets its expiry.
diff --git a/backend/src/lib/productUsage.test.ts b/backend/src/lib/productUsage.test.ts
index 526655ea3..5eea1ed12 100644
--- a/backend/src/lib/productUsage.test.ts
+++ b/backend/src/lib/productUsage.test.ts
@@ -204,7 +204,7 @@ describe('aggregate ingestion', () => {
],
})
expect(db.productUsageReceipt.createMany).toHaveBeenCalledExactlyOnceWith({ data: [
- { id: report.id, expiresAt: new Date('2026-11-18T00:00:00.000Z') }] })
+ { id: report.id, expiresAt: new Date('2026-10-17T00:00:00.000Z'), life: 4 }] })
const stored = JSON.stringify([db.$runCommandRaw.mock.calls, db.productUsageReceipt.createMany.mock.calls])
for (const forbidden of ['private-account', 'userId', 'machineId', 'agentId', 'revision', 'counters', 'receivedAt']) expect(stored).not.toContain(forbidden)
})
@@ -219,10 +219,15 @@ describe('aggregate ingestion', () => {
expect(db.productUsageReceipt.createMany).not.toHaveBeenCalled()
expect(db.$runCommandRaw).not.toHaveBeenCalled()
})
- it.each(['2026-09-08', '2026-10-14', '2026-02-30', 'bad'])('refuses expired/future/invalid day %s', async day => {
+ it.each(['2026-10-10', '2026-09-08', '2026-10-14', '2026-02-30', 'bad'])('refuses expired/future/invalid day %s', async day => {
expect(await ingestProductUsage('a', { ...report, day }, now)).toEqual({ accepted: false, reason: 'EXPIRED' })
expect(db.$transaction).not.toHaveBeenCalled()
})
+ it('accepts the last day of the 3-day window, its receipt kept a day past the window', async () => {
+ expect(await ingestProductUsage('a', { ...report, day: '2026-10-11' }, now)).toEqual({ accepted: true })
+ expect(db.productUsageReceipt.createMany).toHaveBeenCalledExactlyOnceWith({ data: [
+ { id: report.id, expiresAt: new Date('2026-10-15T00:00:00.000Z'), life: 4 }] })
+ })
it('accepts a previously committed receipt without incrementing again', async () => {
db.productUsageReceipt.findMany.mockResolvedValue([{ id: report.id }])
expect(await ingestProductUsage('a', report, now)).toEqual({ accepted: true })
diff --git a/backend/src/lib/productUsage.ts b/backend/src/lib/productUsage.ts
index cd1a8f145..098d2a585 100644
--- a/backend/src/lib/productUsage.ts
+++ b/backend/src/lib/productUsage.ts
@@ -8,6 +8,7 @@ import { prisma } from './prisma.js'
import { ensureIndex } from './db/indexes.js'
import { PRODUCT_USAGE_DAYS, type ProductUsageChoice, type ProductUsageReport } from '../types/productUsage.js'
import { ACTIVITY_DAYS, type ActivitySurface } from '../types/productUsageActivity.js'
+import { PRODUCT_USAGE_RECEIPT_DAYS, PRODUCT_USAGE_RECEIPT_EXPIRY_DAYS } from '../types/productUsageReceipt.js'
import { incrementUsageTotals, type UsageIncrement } from './productUsageWrite.js'
import { createUsageBatch } from './productUsageBatch.js'
import { createRelayTotals } from './productUsageRelay.js'
@@ -39,7 +40,7 @@ function validDay(day: string, now: Date): boolean {
const stamp = Date.parse(day + 'T00:00:00.000Z')
const today = Math.floor(now.getTime() / DAY) * DAY
return Number.isFinite(stamp) && new Date(stamp).toISOString().slice(0, 10) === day
- && stamp <= today && stamp > today - PRODUCT_USAGE_DAYS * DAY
+ && stamp <= today && stamp > today - PRODUCT_USAGE_RECEIPT_DAYS * DAY
}
/** [productUsageChoice] for several accounts at once, in two reads: a batch's permission checks. */
@@ -141,7 +142,8 @@ async function commitReports(reports: QueuedReport[]): Promise<IngestResult[]> {
for (const r of permitted) if (!received.has(r.id)) { received.add(r.id); fresh.push(r) }
if (!fresh.length) return results
await tx.productUsageReceipt.createMany({ data: fresh.map(r => ({
- id: r.id, expiresAt: new Date(Date.parse(r.day) + (PRODUCT_USAGE_DAYS + 1) * DAY) })) })
+ // [life]: once the receipt sweep's rule is installed, a server that does not write it cannot count (productUsageReceiptSweep.ts).
+ id: r.id, expiresAt: new Date(Date.parse(r.day) + PRODUCT_USAGE_RECEIPT_EXPIRY_DAYS * DAY), life: PRODUCT_USAGE_RECEIPT_EXPIRY_DAYS })) })
const apart = await apartPrefixes(tx, fresh)
await incrementUsageTotals(tx, summed(fresh.flatMap((r, i) =>
apart[i] ? r.increments.map(inc => ({ ...inc, metric: apart[i] + inc.metric })) : r.increments)))
diff --git a/backend/src/lib/productUsageReceiptSweep.ts b/backend/src/lib/productUsageReceiptSweep.ts
new file mode 100644
index 000000000..5da09dbae
--- /dev/null
+++ b/backend/src/lib/productUsageReceiptSweep.ts
@@ -0,0 +1,112 @@
+import { setTimeout as delay } from 'node:timers/promises'
+import type { PrismaClient } from '@prisma/client'
+import { PRODUCT_USAGE_RECEIPT_EXPIRY_DAYS } from '../types/productUsageReceipt.js'
+import { logger } from '../utils/logger.js'
+
+/**
+ * Removes, a little at a time, the usage receipts written while a receipt lived 36 days (until 2026-10-29: about 4M of
+ * them, most of the database's index size), once their new life is over. Mongo's expiry would take 36 days to.
+ *
+ * A receipt is `{ _id: report id, expiresAt: report day + N days }`, nothing else, so which life it was given shows only in
+ * its expiresAt. A new receipt expires by today + 4 days; an old one at day + 36, and its new life ended at day + 4. So an
+ * old receipt is removed when its expiresAt is past today + 5 (a day more than any new receipt, for a server clock ahead of
+ * this one and a pass across midnight) and at most now + 32 days. Old receipts below that range expire within 5 days.
+ *
+ * Before any removal, the collection's rule ([RECEIPT_RULE]) refuses every receipt written without its `life`: a server
+ * released before 2026-10-29 (a rollout in progress, a rollback) still accepts reports 35 days old, and would count one
+ * again once its receipt is gone. Its receipt write and its totals share one transaction, so with the rule it counts nothing
+ * and its client retries later. No server's presence is inferred: the rule holds whatever runs. Mongo applies a rule to
+ * writes from then on only (receipts are never updated), so installing it reads no existing receipt.
+ */
+const COLLECTION = 'product_usage_receipts'
+const DAY = 86_400_000
+const OLD_EXPIRY_DAYS = 36
+/** About 2,000 a minute, in four deletes with a pause between them: the backlog clears in about a day. */
+export const RECEIPT_SWEEP = { chunk: 500, chunks: 4, pauseMs: 250, everyMs: 60_000 } as const
+
+/** Every receipt write must carry its life; old receipts, never written again, are not checked. */
+export const RECEIPT_RULE = { validator: { $jsonSchema: { required: ['life'] } }, validationLevel: 'strict', validationAction: 'error' } as const
+
+export interface SweepOptions { chunk: number; chunks: number; pauseMs: number }
+export interface SweepPass {
+ removed: number
+ /** No old receipt is left that this sweep would ever remove. */
+ done: boolean
+}
+
+const date = (ms: number) => ({ $date: new Date(ms).toISOString() })
+const startOfDay = (now: Date) => Math.floor(now.getTime() / DAY) * DAY
+
+/** Installs [RECEIPT_RULE] unless it is there already. collMod locks the collection only for the change itself; a receipt
+ * write in flight finishes first, under the old rule, and every later write is checked. */
+export async function ensureReceiptRule(db: PrismaClient): Promise<void> {
+ const answer = await db.$runCommandRaw({ listCollections: 1, filter: { name: COLLECTION } } as never) as
+ { cursor: { firstBatch: Array<{ options?: Record<string, unknown> }> } }
+ const options = answer.cursor.firstBatch[0]?.options ?? {}
+ if ((Object.keys(RECEIPT_RULE) as Array<keyof typeof RECEIPT_RULE>)
+ .every(key => JSON.stringify(options[key]) === JSON.stringify(RECEIPT_RULE[key]))) return
+ await db.$runCommandRaw({ collMod: COLLECTION, ...RECEIPT_RULE } as never)
+}
+
+async function anyReceipt(db: PrismaClient, filter: Record<string, unknown>): Promise<boolean> {
+ const answer = await db.$runCommandRaw({ find: COLLECTION, filter, projection: { _id: 1 }, limit: 1, singleBatch: true } as never) as
+ { cursor: { firstBatch: unknown[] } }
+ return answer.cursor.firstBatch.length > 0
+}
+
+/** One chunk: up to [chunk] old receipts in expiresAt order, then exactly those, their range checked again. */
+async function removeChunk(db: PrismaClient, now: Date, chunk: number): Promise<number> {
+ const range = { expiresAt: { $gt: date(startOfDay(now) + (PRODUCT_USAGE_RECEIPT_EXPIRY_DAYS + 1) * DAY),
+ $lte: date(now.getTime() + (OLD_EXPIRY_DAYS - PRODUCT_USAGE_RECEIPT_EXPIRY_DAYS) * DAY) } }
+ const found = await db.$runCommandRaw({ find: COLLECTION, filter: range, projection: { _id: 1 }, sort: { expiresAt: 1 },
+ hint: 'product_usage_expiry', limit: chunk, singleBatch: true } as never) as { cursor: { firstBatch: Array<{ _id: string }> } }
+ const ids = found.cursor.firstBatch.map(receipt => receipt._id)
+ if (!ids.length) return 0
+ const answer = await db.$runCommandRaw({ delete: COLLECTION, deletes: [{ q: { _id: { $in: ids }, ...range }, limit: 0 }] } as never) as
+ { n?: number; writeErrors?: unknown[] }
+ if (answer.writeErrors?.length) throw new Error('The database refused some receipt removals')
+ return answer.n ?? 0
+}
+
+/** One pass: the rule, then at most [chunks] × [chunk] receipts. Safe to repeat, and from two processes at once. */
+export async function sweepOldReceipts(db: PrismaClient, now = new Date(), options: SweepOptions = RECEIPT_SWEEP): Promise<SweepPass> {
+ await ensureReceiptRule(db)
+ let removed = 0
+ for (let i = 0; i < options.chunks; i++) {
+ if (i) await delay(options.pauseMs)
+ const n = await removeChunk(db, now, options.chunk)
+ removed += n
+ if (n < options.chunk) break
+ }
+ const done = !await anyReceipt(db, { expiresAt: { $gt: date(startOfDay(now) + (PRODUCT_USAGE_RECEIPT_EXPIRY_DAYS + 1) * DAY) } })
+ return { removed, done }
+}
+
+/** Runs a pass a minute in the singleton worker until nothing is left to remove, then stops; a restart checks once more. */
+export function startReceiptSweep(db: PrismaClient, settings: Partial<SweepOptions & { everyMs: number; now: () => Date;
+ log: Pick<typeof logger, 'info' | 'warn'> }> = {}): () => void {
+ const { everyMs, now, log, ...options } = { ...RECEIPT_SWEEP, now: () => new Date(), log: logger, ...settings }
+ let running = false, total = 0
+ const run = async () => {
+ if (running) return
+ running = true
+ try {
+ const pass = await sweepOldReceipts(db, now(), options)
+ total += pass.removed
+ if (pass.removed) log.info('[usage-receipts] pass', { removed: pass.removed, removedSinceStart: total })
+ if (pass.done) {
+ stop()
+ log.info('[usage-receipts] done: no old receipts left', { removedSinceStart: total })
+ }
+ } catch (error) {
+ log.warn('[usage-receipts] pass failed; retrying next minute', { error: error instanceof Error ? error.message : String(error) })
+ } finally {
+ running = false
+ }
+ }
+ const timer = setInterval(() => void run(), everyMs)
+ timer.unref()
+ const stop = () => clearInterval(timer)
+ void run()
+ return stop
+}
diff --git a/backend/src/types/productUsageReceipt.ts b/backend/src/types/productUsageReceipt.ts
new file mode 100644
index 000000000..1fd6b3bbf
--- /dev/null
+++ b/backend/src/types/productUsageReceipt.ts
@@ -0,0 +1,8 @@
+/** Shared by the write path (lib/productUsage.ts) and the one-time receipt cleanup (scripts/product-usage/): no imports,
+ * so the script loads without the server's environment. */
+
+/** How many UTC days, today included, a signed-in report is accepted for. Its receipt, which only stops a retry counting
+ * twice, is kept one day longer ([PRODUCT_USAGE_RECEIPT_EXPIRY_DAYS]): a report accepted in the window's last moment
+ * commits a moment later, and Mongo's expiry runs about once a minute. 36 days held about 4M receipts (2026-10-29). */
+export const PRODUCT_USAGE_RECEIPT_DAYS = 3
+export const PRODUCT_USAGE_RECEIPT_EXPIRY_DAYS = PRODUCT_USAGE_RECEIPT_DAYS + 1
diff --git a/backend/src/worker.ts b/backend/src/worker.ts
index 70d711877..963850fca 100644
--- a/backend/src/worker.ts
+++ b/backend/src/worker.ts
@@ -26,6 +26,7 @@ import { GitHubApp, githubAppConfig } from './github/app.js'
import { env } from './config/env.js'
import { machineBillingWorkerService } from './services/MachineBillingWorkerService.js'
import { productUsagePeopleRollupService } from './services/ProductUsagePeopleRollupService.js'
+import { startReceiptSweep } from './lib/productUsageReceiptSweep.js'
import { logger } from './utils/logger.js'
// Singleton background process. HTTP/WS server.ts runs as a PM2 cluster and must never own schema
@@ -55,6 +56,7 @@ async function start(): Promise<void> {
let stopSweeps = (): void => {}
let stopArchive = (): void => {}
let stopDeliveries = (): void => {}
+ let stopReceiptSweep = (): void => {}
const shutdown = (signal: string): void => {
logger.info(`Received ${signal}, stopping backend worker...`)
const hardExit = setTimeout(() => process.exit(0), 5000)
@@ -64,6 +66,7 @@ async function start(): Promise<void> {
stopSweeps()
stopArchive()
stopDeliveries()
+ stopReceiptSweep()
void Promise.allSettled([prisma.$disconnect(), closeBus()]).finally(() => process.exit(0))
}
process.on('SIGINT', () => shutdown('SIGINT'))
@@ -71,6 +74,8 @@ async function start(): Promise<void> {
machineBillingWorkerService.start()
productUsagePeopleRollupService.start()
+ // Usage receipts from when they lived 36 days, a few at a time until none are left (lib/productUsageReceiptSweep.ts).
+ stopReceiptSweep = startReceiptSweep(prisma)
// The worker's sweeps and archive write Swarms: they bump the stamps the API processes read by. It reads too little to keep a cache.
configureSwarmStamps(redisSwarmStamps(), { reads: false })
// Archive notes and Workspace removals write messages too: their recipients' hosts hear of them at once, and of the
diff --git a/backend/vitest.super-tabs.config.ts b/backend/vitest.super-tabs.config.ts
index ed30d29ee..83a89921f 100644
--- a/backend/vitest.super-tabs.config.ts
+++ b/backend/vitest.super-tabs.config.ts
@@ -3,6 +3,6 @@ import { defineConfig } from 'vitest/config'
export default defineConfig({ test: { include: ['integration/superTabs.test.ts', 'integration/superTabSharing.test.ts', 'integration/superTabGithub.test.ts', 'integration/superTabTeam.test.ts', 'integration/superTabWorkspaces.test.ts',
'integration/superTabRead.test.ts', 'integration/superTabReadMarks.test.ts', 'integration/superTabLeaderboard.test.ts', 'integration/superTabMigration.test.ts', 'integration/superTabLiveness.test.ts', 'integration/superTabTidy.test.ts',
'integration/superTabProfile.test.ts', 'integration/superTabLegacyDrop.test.ts', 'integration/shareRaces.test.ts',
- 'integration/superTabIndexes.test.ts', 'integration/superTabHotReads.test.ts', 'integration/superTabGroupPlans.test.ts', 'integration/superTabInvitationPlans.test.ts', 'integration/superTabCache.test.ts', 'integration/superTabObserver.test.ts', 'integration/productUsageConflicts.test.ts', 'integration/schemaRelease.test.ts', 'integration/superTabBuddies.test.ts', 'integration/superTabBuddyProbes.test.ts', 'integration/superTabCosts.test.ts',
+ 'integration/superTabIndexes.test.ts', 'integration/superTabHotReads.test.ts', 'integration/superTabGroupPlans.test.ts', 'integration/superTabInvitationPlans.test.ts', 'integration/superTabCache.test.ts', 'integration/superTabObserver.test.ts', 'integration/productUsageConflicts.test.ts', 'integration/productUsageReceipts.test.ts', 'integration/schemaRelease.test.ts', 'integration/superTabBuddies.test.ts', 'integration/superTabBuddyProbes.test.ts', 'integration/superTabCosts.test.ts',
'integration/harnessSharesList.test.ts'], hookTimeout: 120000, testTimeout: 30000, setupFiles: ['integration/cacheOn.test-support.ts'],
fileParallelism: false, env: { NODE_ENV: 'test', HARNESS_CREDENTIAL_ENCRYPTION_KEY: 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=' } } })
diff --git a/docs/product-measurement.md b/docs/product-measurement.md
index d636b064d..4e4cf1fe5 100644
--- a/docs/product-measurement.md
+++ b/docs/product-measurement.md
@@ -360,7 +360,9 @@ interrupted Codex leaves behind, no longer refuses a resume: it is kept as it is
From CLI 0.4.3, an unconfirmed runtime counts as unknown and a concurrently changed harness as
cancelled (legacy aggregate: unknown), rather than failed. Compare releases separately across this
definition change; a lower failure rate alone is not proof the product improved.
-Receipts expire within 36 days and totals within 396 days through the existing Mongo TTL indexes.
+The backend accepts a signed-in report for today and the two UTC days before it; an older one answers `EXPIRED`, adds
+nothing, and the CLI and Desktop count it as `health_<surface>_report_expired`. Receipts expire 4 days after the report
+day (36 until 2026-10-29) and totals within 396 days through the existing Mongo TTL indexes.
No account deduplication markers or stable hashes enter these totals. People-based daily counts and
D1/D2/D3/D7 returns come from one mark per sharing person per day under an HMAC pseudonym (no time,
machine, agent, device, country or content), kept 10 days and then only as anonymous `people_*` totals;