Where agent teams gather. Cross-platform team chat where AI agents are first-class teammates — with cloud or bring-your-own (Claude Code / Codex) brains.
TypeScript
fix(db): lock contention must not fail the deploy on its first attempt (#288)
OKNOTOKd8d63fb02ec5
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex e86e6c4..1208a61 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -48,6 +48,7 @@ npm run test:integration # integration suite, see the env var belownpm run guard:big-brain # architecture guard, see belownpm run guard:llm-tracked # architecture guard, see belownpm run guard:engine-registry # architecture guard, see below+npm run guard:migration-locks # schema guard, see below````npm run test:integration` needs a **dedicated** database and does nothing@@ -93,6 +94,33 @@ guard script will fail your build if you break them:unknown to `byoa-claude`, so its runs quietly bill to the wrong engine.`npm run guard:engine-registry` checks this.+## Writing a schema migration++Migrations run from a pre-deploy Job, against production, **while the old Pods+are still serving traffic**, and `ensureSchema` pins that session at+`lock_timeout = '5s'`. DDL that needs an `ACCESS EXCLUSIVE` lock on a hot table+has five seconds to get it before the statement aborts with `55P03`; even when+it does get the lock, every query behind it waits for the whole statement.++Once a migration has been applied its SQL is checksum-pinned and can never be+rewritten — so a lock-taking migration cannot be fixed afterwards, only worked+around. `npm run guard:migration-locks` rejects the two patterns that have cost+this project production time:++- **`ADD COLUMN … DEFAULT <volatile>`.** PostgreSQL's metadata-only fast path+ applies only to non-volatile defaults; `gen_random_uuid()`, `random()`,+ `nextval()` and friends force a full table and index rewrite. Write it as+ nullable column → batched backfill → `SET DEFAULT` →+ `CHECK (… IS NOT NULL) NOT VALID` → `VALIDATE CONSTRAINT`. `now()` is+ `STABLE`, not volatile, and is fine.+- **`CREATE INDEX` without `CONCURRENTLY` on an existing table.** It blocks+ writes for the whole build and deadlocks against live writers. Use+ `CREATE INDEX CONCURRENTLY` with `transactional: false`, as migrations 0005+ and 0006 do. Indexing a table the same migration creates is fine.++Transactional migrations that still lose a lock race are retried with backoff+before the Job fails, so a brief contention spike does not need a redeploy.+The multi-agent coordination model (how N agents share a room withoutcolliding, and why the prompt is kept deliberately minimal) is documented in[`docs/COORDINATION.md`](docs/COORDINATION.md) — read it before touching thediff --git a/package.json b/package.jsonindex f319b62..fb6c8c4 100644--- a/package.json+++ b/package.json@@ -31,6 +31,7 @@"guard:big-brain": "node scripts/guard-big-brain.mjs","guard:llm-tracked": "node scripts/guard-llm-tracked.mjs","guard:engine-registry": "node scripts/guard-engine-registry.mjs",+ "guard:migration-locks": "node scripts/guard-migration-locks.mjs","lint": "biome lint .","lint:fix": "biome lint --write .","test": "node --import tsx --test \"server/src/__tests__/**/*.test.ts\" \"tests/**/*.test.ts\" \"workers/**/*.test.ts\"",diff --git a/scripts/guard-migration-locks.d.mts b/scripts/guard-migration-locks.d.mtsnew file mode 100644index 0000000..e6a8e28--- /dev/null+++ b/scripts/guard-migration-locks.d.mts@@ -0,0 +1,17 @@+// Type declarations for the migration lock guard (scripts/guard-migration-locks.mjs).+export interface MigrationLockProblem {+ /** Repo-relative path of the offending migration. */+ where: string+ /** What it locks, and how to write it instead. */+ why: string+}+/** Strip block and line comments so prose about a pattern is not read as the pattern. */+export function stripComments(source: string): string+/** Lower-cased names of the tables a migration creates itself. */+export function tablesCreatedIn(sql: string): Set<string>+/** Every lock hazard in one migration's source text. */+export function scanMigration(file: string, source: string): MigrationLockProblem[]+/** Scan every migration except the grandfathered ones. */+export function scanRepo(): MigrationLockProblem[]+/** Applied migrations whose checksum-pinned SQL cannot be rewritten, and why. */+export function grandfathered(): Map<string, string>diff --git a/scripts/guard-migration-locks.mjs b/scripts/guard-migration-locks.mjsnew file mode 100644index 0000000..1a3b952--- /dev/null+++ b/scripts/guard-migration-locks.mjs@@ -0,0 +1,173 @@+#!/usr/bin/env node+/**+ * guard-migration-locks — a migration may not take a table-wide lock it does+ * not need.+ *+ * Migrations run from a pre-deploy Job, against production, WHILE the old Pods+ * are still serving traffic. `ensureSchema` pins that session at+ * `lock_timeout = '5s'`, so DDL that needs an AccessExclusiveLock on a hot+ * table has five seconds to get it before the statement aborts with `55P03`+ * and the deploy fails. Even when it does get the lock, everything else queues+ * behind it for as long as the statement runs.+ *+ * Migration SQL is checksum-pinned once applied, so a migration written this+ * way cannot be fixed afterwards — only worked around. That is why this is a+ * pre-merge guard and not a runtime check.+ *+ * Two patterns are rejected:+ *+ * 1. ADD COLUMN with a VOLATILE default.+ *+ * PostgreSQL's metadata-only fast path for `ADD COLUMN … DEFAULT` applies+ * only to non-volatile defaults. A volatile one — gen_random_uuid(),+ * random(), clock_timestamp(), nextval() — forces a full table and index+ * rewrite under AccessExclusiveLock. Migration 0004 did exactly this to+ * `participants`, a table on the hot path of nearly every query in the+ * product; it is allowlisted below because its checksum is pinned and its+ * SQL can no longer be changed.+ *+ * Write it as: nullable column (metadata only) → batched backfill →+ * SET DEFAULT → `CHECK (… IS NOT NULL) NOT VALID` → VALIDATE CONSTRAINT.+ * Each step is short, and none of them holds a lock across the backfill.+ *+ * 2. CREATE INDEX without CONCURRENTLY on a pre-existing table.+ *+ * A plain CREATE INDEX blocks writes to the table for the whole build. On+ * a hot table under live traffic that wait deadlocks (`40P01`) against+ * concurrent writers, and because the build then never commits, every Pod+ * retries it on boot and none can start — which is how the+ * conversations.members GIN build wedged production. Use CREATE INDEX+ * CONCURRENTLY with `transactional: false`, as 0005 and 0006 do.+ *+ * Indexing a table the SAME migration creates is fine and is not flagged:+ * nothing can be reading or writing a table that did not exist a statement+ * ago.+ *+ * A guard that quietly stops checking is worse than no guard, so `scanRepo()`+ * reports a problem if the migrations directory ever reads as empty.+ *+ * Run: node scripts/guard-migration-locks.mjs+ */+import { readdirSync, readFileSync } from 'node:fs'+import { join } from 'node:path'+import { fileURLToPath } from 'node:url'++const ROOT = join(fileURLToPath(new URL('.', import.meta.url)), '..')+const MIGRATIONS_DIR = 'server/src/db/migrations'++/**+ * Migrations whose SQL is already applied in production and checksum-pinned,+ * so it cannot be rewritten. Each entry records what it does and why it stays.+ */+const GRANDFATHERED = new Map([+ [+ '0004-agent-runtime-assignment.ts',+ 'ADD COLUMN … DEFAULT gen_random_uuid()::text rewrites participants under ' ++ 'AccessExclusiveLock. Applied and checksum-pinned; changing the SQL would fail ' ++ 'the checksum check on every existing database.',+ ],+])++/** VOLATILE in PostgreSQL's function catalog, so a default built from one+ * defeats the metadata-only ADD COLUMN fast path. `now()` and+ * `current_timestamp` are STABLE, not volatile, and are deliberately absent. */+const VOLATILE_DEFAULTS = [+ 'gen_random_uuid',+ 'uuid_generate_v1',+ 'uuid_generate_v4',+ 'random',+ 'clock_timestamp',+ 'timeofday',+ 'nextval',+]++/** Strip comments so prose describing a pattern is never mistaken for it —+ * 0005's header explains why "a plain CREATE INDEX" is wrong, in a comment. */+export function stripComments(source) {+ return source+ .replace(/\/\*[\s\S]*?\*\//g, ' ')+ .replace(/(^|[^:])\/\/[^\n]*/g, '$1 ')+}++/** Tables this file creates itself — indexing them is free of live traffic. */+export function tablesCreatedIn(sql) {+ const names = new Set()+ for (const m of sql.matchAll(/CREATE\s+TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?([a-z_][a-z0-9_]*)/gi)) {+ names.add(m[1].toLowerCase())+ }+ return names+}++/** Every lock hazard in one migration's source text. */+export function scanMigration(file, source) {+ const sql = stripComments(source).replace(/\s+/g, ' ')+ const created = tablesCreatedIn(sql)+ const problems = []++ // ADD COLUMN … DEFAULT <volatile>. Bounded to the clause: the default runs to+ // the next comma or statement end, so a volatile call in a LATER column's+ // definition is attributed to that column, not this one.+ for (const m of sql.matchAll(/ADD\s+COLUMN\s+(?:IF\s+NOT\s+EXISTS\s+)?([a-z_][a-z0-9_]*)[^,;]*?DEFAULT\s+([^,;]+)/gi)) {+ const [, column, expression] = m+ const volatileFn = VOLATILE_DEFAULTS.find((fn) => new RegExp(`\\b${fn}\\s*\\(`, 'i').test(expression))+ if (!volatileFn) continue+ problems.push({+ where: `${MIGRATIONS_DIR}/${file}`,+ why: `ADD COLUMN ${column} DEFAULT … ${volatileFn}() is VOLATILE, so PostgreSQL rewrites the whole `+ + 'table under AccessExclusiveLock instead of taking the metadata-only fast path. Add the column '+ + 'nullable, backfill in batches, then SET DEFAULT and validate a NOT VALID check constraint.',+ })+ }++ // CREATE [UNIQUE] INDEX … ON <table>, without CONCURRENTLY, on a table that+ // already exists.+ for (const m of sql.matchAll(/CREATE\s+(?:UNIQUE\s+)?INDEX\s+(CONCURRENTLY\s+)?(?:IF\s+NOT\s+EXISTS\s+)?[a-z_][a-z0-9_$]*\s+ON\s+(?:ONLY\s+)?([a-z_][a-z0-9_]*)/gi)) {+ const [, concurrently, table] = m+ if (concurrently) continue+ if (created.has(table.toLowerCase())) continue+ problems.push({+ where: `${MIGRATIONS_DIR}/${file}`,+ why: `CREATE INDEX on the pre-existing table "${table}" blocks writes for the whole build and `+ + 'deadlocks against live writers. Use CREATE INDEX CONCURRENTLY with transactional: false.',+ })+ }++ return problems+}++export function scanRepo() {+ const files = readdirSync(join(ROOT, MIGRATIONS_DIR))+ .filter((name) => /^\d{4}-.+\.ts$/.test(name))+ .sort()++ if (files.length === 0) {+ return [{+ where: MIGRATIONS_DIR,+ why: 'no migration files found — the guard is looking in the wrong place and would pass vacuously',+ }]+ }++ const problems = []+ for (const file of files) {+ if (GRANDFATHERED.has(file)) continue+ problems.push(...scanMigration(file, readFileSync(join(ROOT, MIGRATIONS_DIR, file), 'utf8')))+ }+ return problems+}++/** Exported so the test can prove the allowlist still describes a real hazard+ * rather than silently covering a migration that no longer has one. */+export function grandfathered() {+ return GRANDFATHERED+}++if (import.meta.url === `file://${process.argv[1]}`) {+ const problems = scanRepo()+ if (problems.length > 0) {+ console.error('🚨 migration would lock a live table:\n')+ for (const problem of problems) console.error(` ${problem.where}\n ${problem.why}\n`)+ console.error('Migration SQL is checksum-pinned once applied — it cannot be fixed after the fact.')+ process.exit(1)+ }+ console.log('✅ migration lock guard: no migration takes a table-wide lock it does not need.')+}diff --git a/server/src/__tests__/guard-migration-locks.test.ts b/server/src/__tests__/guard-migration-locks.test.tsnew file mode 100644index 0000000..8693e59--- /dev/null+++ b/server/src/__tests__/guard-migration-locks.test.ts@@ -0,0 +1,119 @@+/**+ * CI enforcement for the migration lock guard.+ *+ * Migrations run from a pre-deploy Job while the old Pods still serve traffic,+ * and `ensureSchema` pins that session at `lock_timeout = '5s'`. DDL that needs+ * an AccessExclusiveLock on a hot table therefore has five seconds to get it+ * before the statement aborts with `55P03` — and once a migration is applied,+ * its SQL is checksum-pinned and can never be rewritten. Catching the pattern+ * before merge is the only place it can be caught at all.+ *+ * See scripts/guard-migration-locks.mjs.+ */+import { test } from 'node:test'+import assert from 'node:assert/strict'+// The guard is plain ESM tooling; tsx lets this .ts test import the .mjs.+import {+ grandfathered,+ scanMigration,+ scanRepo,+ stripComments,+ tablesCreatedIn,+} from '../../../scripts/guard-migration-locks.mjs'+import { readFileSync, readdirSync } from 'node:fs'+import { join } from 'node:path'++const ROOT = join(import.meta.dirname, '..', '..', '..')+const MIGRATIONS = join(ROOT, 'server/src/db/migrations')+const migration = (file: string): string => readFileSync(join(MIGRATIONS, file), 'utf8')++test('no migration takes a table-wide lock it does not need', () => {+ const problems = scanRepo()+ assert.deepEqual(+ problems, [],+ '\n🚨 migration would lock a live table:\n' ++ problems.map((p: { where: string; why: string }) => ` ${p.where}\n ${p.why}`).join('\n'),+ )+})++test('the guard is looking at the real migration files', () => {+ // If the directory or the filename pattern ever moved, scanRepo() would have+ // nothing to scan and would pass vacuously. scanRepo() reports that as a+ // problem; pin the happy path here too.+ const files = readdirSync(MIGRATIONS).filter((name) => /^\d{4}-.+\.ts$/.test(name))+ assert.ok(files.length >= 8, `expected the real migration set, found ${files.length}`)+})++// --- the guard actually catches both hazards (so it is not a no-op) ---++test('a volatile ADD COLUMN default is caught', () => {+ // Exactly migration 0004's shape, which rewrites `participants` — a table on+ // the hot path of nearly every query — under AccessExclusiveLock.+ const problems = scanMigration(+ 'x.ts',+ 'ALTER TABLE participants ADD COLUMN runtime_assignment_id TEXT NOT NULL DEFAULT gen_random_uuid()::text;',+ )+ assert.equal(problems.length, 1)+ assert.match(problems[0].why, /VOLATILE/)+ assert.match(problems[0].why, /nullable/)++ // And the real file is caught when it is not treated as grandfathered, which+ // is what keeps the allowlist entry honest.+ const live = scanMigration('0004.ts', migration('0004-agent-runtime-assignment.ts'))+ assert.equal(live.length, 1, 'the allowlisted migration must still be a real hazard')+})++test('a non-volatile default is not caught', () => {+ // now() is STABLE, and a literal is IMMUTABLE — both take PostgreSQL's+ // metadata-only fast path, so flagging them would be a false positive that+ // teaches people to ignore the guard.+ assert.deepEqual(scanMigration('x.ts', "ALTER TABLE computers ADD COLUMN engine_defaults JSONB NOT NULL DEFAULT '{}'::jsonb;"), [])+ assert.deepEqual(scanMigration('x.ts', 'ALTER TABLE messages ADD COLUMN seen_at TIMESTAMPTZ NOT NULL DEFAULT now();'), [])+ assert.deepEqual(scanMigration('x.ts', 'ALTER TABLE participants ADD COLUMN provider_profile TEXT;'), [])+})++test('a plain CREATE INDEX on a pre-existing table is caught', () => {+ const problems = scanMigration('x.ts', 'CREATE INDEX idx_messages_convo ON messages (conversation_id);')+ assert.equal(problems.length, 1)+ assert.match(problems[0].why, /CONCURRENTLY/)++ assert.deepEqual(+ scanMigration('x.ts', 'CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_messages_convo ON messages (conversation_id);'),+ [],+ )+})++test('indexing a table the same migration creates is not caught', () => {+ // Nothing can be reading or writing a table that did not exist a statement+ // ago, so CONCURRENTLY buys nothing there — and would force the migration+ // out of its transaction for no reason.+ assert.deepEqual(+ scanMigration('x.ts', `+ CREATE TABLE conversation_members (id TEXT PRIMARY KEY, conversation_id TEXT NOT NULL);+ CREATE INDEX idx_conversation_members_participant ON conversation_members (conversation_id);+ `),+ [],+ )+ assert.ok(tablesCreatedIn('CREATE TABLE IF NOT EXISTS agent_routing_claims (id TEXT)').has('agent_routing_claims'))+})++test('prose about a bad pattern is not mistaken for the pattern', () => {+ // 0005's header comment explains why "a plain CREATE INDEX" is wrong. A guard+ // that flagged its own documentation would be unusable.+ assert.deepEqual(scanMigration('0005.ts', migration('0005-search-trigram-index.ts')), [])+ assert.match(stripComments('/* CREATE INDEX idx ON messages (a) */ SELECT 1'), /^\s+SELECT 1$/)+ assert.match(stripComments('// ADD COLUMN x DEFAULT gen_random_uuid()\nSELECT 1'), /^\s*\nSELECT 1$/)+})++test('every grandfathered migration says what it does and why it stays', () => {+ const entries = grandfathered()+ assert.ok(entries.size > 0, 'the allowlist should still cover migration 0004')+ for (const [file, reason] of entries) {+ assert.ok(+ readdirSync(MIGRATIONS).includes(file),+ `allowlist names ${file}, which no longer exists — drop the entry`,+ )+ assert.ok(reason.length > 40, `${file} needs a real reason, not "${reason}"`)+ assert.match(reason, /checksum/i, `${file}'s reason must say why the SQL cannot be changed`)+ }+})diff --git a/server/src/__tests__/migration-transaction-atomicity.test.ts b/server/src/__tests__/migration-transaction-atomicity.test.tsindex 5230bd0..6dbba8b 100644--- a/server/src/__tests__/migration-transaction-atomicity.test.ts+++ b/server/src/__tests__/migration-transaction-atomicity.test.ts@@ -153,3 +153,127 @@ test('rollback failure still preserves original migration error', async () => {/primary migration error/,)})++/** A PostgreSQL error as node-postgres surfaces it: the SQLSTATE is on `code`. */+function pgError(code: string, message: string): Error & { code: string } {+ return Object.assign(new Error(message), { code })+}++test('a migration that loses a lock race is retried after a clean rollback', async () => {+ const { client, calls } = fakeClient()+ let attempts = 0+ const slept: number[] = []++ const migration: VersionedMigration = {+ version: 4,+ name: '0004-lock-contention',+ checksum: 'f'.repeat(64),+ sourceChecksum: 'f'.repeat(64),+ transactional: true,+ up: async () => {+ attempts += 1+ calls.push({ sql: `ALTER TABLE participants ADD COLUMN x TEXT -- attempt ${attempts}` })+ // `ensureSchema` pins lock_timeout at 5s, so DDL on a hot table under+ // live traffic aborts with 55P03 long before it does any work.+ if (attempts < 3) throw pgError('55P03', 'canceling statement due to lock timeout')+ },+ }++ await applyPendingMigration(client, migration, { sleep: async (ms) => { slept.push(ms) } })++ assert.equal(attempts, 3)+ const statements = calls.map((c) => c.sql)+ // Each failed attempt rolls back completely, so attempt N+1 starts from+ // exactly the state attempt N saw. No partial schema, no partial ledger.+ assert.equal(statements.filter((s) => s === 'BEGIN').length, 3)+ assert.equal(statements.filter((s) => s === 'ROLLBACK').length, 2)+ assert.equal(statements.filter((s) => s === 'COMMIT').length, 1)+ assert.equal(statements.filter((s) => s.startsWith('INSERT INTO schema_migrations')).length, 1)+ assert.equal(slept.length, 2, 'each retry must back off')+ assert.ok(slept[1] > slept[0], `backoff must grow, got ${slept.join(', ')}`)+})++test('deadlock and serialization failures retry too; a real error never does', async () => {+ for (const code of ['40P01', '55P03', '40001']) {+ const { client } = fakeClient()+ let attempts = 0+ const migration: VersionedMigration = {+ version: 4,+ name: `0004-${code}`,+ checksum: 'f'.repeat(64),+ sourceChecksum: 'f'.repeat(64),+ transactional: true,+ up: async () => {+ attempts += 1+ if (attempts < 2) throw pgError(code, `transient ${code}`)+ },+ }+ await applyPendingMigration(client, migration, { sleep: async () => {} })+ assert.equal(attempts, 2, `${code} should have been retried once`)+ }++ // A missing relation, a failed precheck, a constraint violation: the deploy+ // has to stop on the first attempt and say so.+ const { client } = fakeClient()+ let realAttempts = 0+ await assert.rejects(+ () => applyPendingMigration(client, {+ version: 4,+ name: '0004-real-error',+ checksum: 'f'.repeat(64),+ sourceChecksum: 'f'.repeat(64),+ transactional: true,+ up: async () => {+ realAttempts += 1+ throw pgError('42P01', 'relation "nope" does not exist')+ },+ }, { sleep: async () => {} }),+ /relation "nope" does not exist/,+ )+ assert.equal(realAttempts, 1, 'a real error must not be retried')+})++test('sustained lock contention gives up and fails the deploy', async () => {+ const { client } = fakeClient()+ let attempts = 0++ await assert.rejects(+ () => applyPendingMigration(client, {+ version: 4,+ name: '0004-always-locked',+ checksum: 'f'.repeat(64),+ sourceChecksum: 'f'.repeat(64),+ transactional: true,+ up: async () => {+ attempts += 1+ throw pgError('55P03', 'canceling statement due to lock timeout')+ },+ }, { sleep: async () => {}, maxAttempts: 4 }),+ /lock timeout/,+ )+ assert.equal(attempts, 4, 'the budget is finite — a wedged table must not hang the Job forever')+})++test('a non-transactional migration is never retried wholesale', async () => {+ // 0001 retries per statement internally and 0005/0006 build indexes+ // CONCURRENTLY; re-running either from here could re-execute work that+ // already committed, because there is no transaction to roll back.+ const { client } = fakeClient()+ let attempts = 0++ await assert.rejects(+ () => applyPendingMigration(client, {+ version: 5,+ name: '0005-concurrent',+ checksum: 'f'.repeat(64),+ sourceChecksum: 'f'.repeat(64),+ transactional: false,+ up: async () => {+ attempts += 1+ throw pgError('55P03', 'canceling statement due to lock timeout')+ },+ }, { sleep: async () => {} }),+ /lock timeout/,+ )+ assert.equal(attempts, 1)+})diff --git a/server/src/db/migrate.ts b/server/src/db/migrate.tsindex 6134ab4..cc101b7 100644--- a/server/src/db/migrate.ts+++ b/server/src/db/migrate.ts@@ -2618,25 +2618,77 @@ const VERSIONED_MIGRATIONS: readonly VersionedMigration[] = [},]+/** How many times a transactional migration may lose a lock race before the+ * deploy gives up. Matches the frozen baseline's per-statement budget. */+const MIGRATION_LOCK_ATTEMPTS = 6++/** `true` for the transient lock-contention SQLSTATEs — the ones where the+ * migration never started, so a clean retry is the same as a first attempt. */+function isTransientLockError(err: unknown): boolean {+ const code = (err as { code?: unknown } | null)?.code+ return typeof code === 'string' && BASELINE_STATEMENT_LOCK_CODES.has(code)+}++/**+ * Apply one migration, retrying a transactional one that loses a lock race.+ *+ * `ensureSchema` pins the migration session at `lock_timeout = '5s'` — the+ * right call, since a migration that queues indefinitely for an+ * AccessExclusiveLock parks every later query behind it — but it means DDL that+ * cannot get its lock within five seconds aborts with `55P03` and, without+ * this loop, takes the whole pre-deploy Job down with it. The Job runs while+ * the old Pods are still serving, so on a busy table that is not an unlikely+ * outcome; it is the normal one, and it repeats on every rerun.+ *+ * Migration 0004 is the standing example. `ALTER TABLE participants ADD COLUMN+ * runtime_assignment_id TEXT NOT NULL DEFAULT gen_random_uuid()::text` takes+ * AccessExclusiveLock on `participants`, which sits on the hot path of nearly+ * every query in the product. Its SQL is checksum-pinned and cannot be edited+ * — see scripts/guard-migration-locks.mjs for the authoring rule that keeps the+ * next one from being written the same way.+ *+ * Only `transactional` migrations retry. A failed transaction has rolled back+ * completely, so attempt two starts from exactly the state attempt one saw. The+ * `transactional: false` ones (the frozen baseline and the CONCURRENTLY index+ * builds) manage their own idempotence and their own retries, and re-running+ * them wholesale from here could re-execute work that already committed.+ *+ * Only the transient lock SQLSTATEs retry: a real error — a missing relation, a+ * constraint violation, a failed precheck — still fails the deploy on its first+ * attempt, loudly, which is the point of the pre-deploy Job.+ */export async function applyPendingMigration(client: import('pg').PoolClient,migration: VersionedMigration,+ opts: { sleep?: (ms: number) => Promise<void>; maxAttempts?: number } = {},): Promise<void> {+ const sleep = opts.sleep ?? ((ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms)))+ const maxAttempts = opts.maxAttempts ?? MIGRATION_LOCK_ATTEMPTSconst started = Date.now()console.log(`[db] applying migration ${migration.version} ${migration.name}`)if (migration.transactional !== false) {- await client.query('BEGIN')- try {- await migration.up(client)- await client.query(- `INSERT INTO schema_migrations (version, name, checksum, execution_ms)- VALUES ($1, $2, $3, $4)`,- [migration.version, migration.name, migration.checksum, Date.now() - started],- )- await client.query('COMMIT')- } catch (err) {- await client.query('ROLLBACK').catch(() => { /* swallow rollback err */ })- throw err+ for (let attempt = 1; ; attempt++) {+ const attemptStarted = Date.now()+ await client.query('BEGIN')+ try {+ await migration.up(client)+ await client.query(+ `INSERT INTO schema_migrations (version, name, checksum, execution_ms)+ VALUES ($1, $2, $3, $4)`,+ [migration.version, migration.name, migration.checksum, Date.now() - attemptStarted],+ )+ await client.query('COMMIT')+ break+ } catch (err) {+ await client.query('ROLLBACK').catch(() => { /* swallow rollback err */ })+ if (!isTransientLockError(err) || attempt >= maxAttempts) throw err+ const backoffMs = Math.min(8_000, 500 * 2 ** (attempt - 1)) + Math.floor(Math.random() * 250)+ console.warn(+ `[db] migration ${migration.version} ${migration.name} hit ${(err as { code: string }).code} ` ++ `(attempt ${attempt}/${maxAttempts}) — retrying in ${backoffMs}ms`,+ )+ await sleep(backoffMs)+ }}} else {await migration.up(client)