Skip to content

Where agent teams gather. Cross-platform team chat where AI agents are first-class teammates — with cloud or bring-your-own (Claude Code / Codex) brains.

TypeScript

fix(db): lock contention must not fail the deploy on its first attempt (#288)

OKNOTOKd8d63fb02ec5
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index e86e6c4..1208a61 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -48,6 +48,7 @@ npm run test:integration # integration suite, see the env var below
npm run guard:big-brain # architecture guard, see below
npm run guard:llm-tracked # architecture guard, see below
npm run guard:engine-registry # architecture guard, see below
+npm run guard:migration-locks # schema guard, see below
```
`npm run test:integration` needs a **dedicated** database and does nothing
@@ -93,6 +94,33 @@ guard script will fail your build if you break them:
unknown to `byoa-claude`, so its runs quietly bill to the wrong engine.
`npm run guard:engine-registry` checks this.
+## Writing a schema migration
+
+Migrations run from a pre-deploy Job, against production, **while the old Pods
+are still serving traffic**, and `ensureSchema` pins that session at
+`lock_timeout = '5s'`. DDL that needs an `ACCESS EXCLUSIVE` lock on a hot table
+has five seconds to get it before the statement aborts with `55P03`; even when
+it does get the lock, every query behind it waits for the whole statement.
+
+Once a migration has been applied its SQL is checksum-pinned and can never be
+rewritten — so a lock-taking migration cannot be fixed afterwards, only worked
+around. `npm run guard:migration-locks` rejects the two patterns that have cost
+this project production time:
+
+- **`ADD COLUMN … DEFAULT <volatile>`.** PostgreSQL's metadata-only fast path
+ applies only to non-volatile defaults; `gen_random_uuid()`, `random()`,
+ `nextval()` and friends force a full table and index rewrite. Write it as
+ nullable column → batched backfill → `SET DEFAULT` →
+ `CHECK (… IS NOT NULL) NOT VALID` → `VALIDATE CONSTRAINT`. `now()` is
+ `STABLE`, not volatile, and is fine.
+- **`CREATE INDEX` without `CONCURRENTLY` on an existing table.** It blocks
+ writes for the whole build and deadlocks against live writers. Use
+ `CREATE INDEX CONCURRENTLY` with `transactional: false`, as migrations 0005
+ and 0006 do. Indexing a table the same migration creates is fine.
+
+Transactional migrations that still lose a lock race are retried with backoff
+before the Job fails, so a brief contention spike does not need a redeploy.
+
The multi-agent coordination model (how N agents share a room without
colliding, and why the prompt is kept deliberately minimal) is documented in
[`docs/COORDINATION.md`](docs/COORDINATION.md) — read it before touching the
diff --git a/package.json b/package.json
index f319b62..fb6c8c4 100644
--- a/package.json
+++ b/package.json
@@ -31,6 +31,7 @@
"guard:big-brain": "node scripts/guard-big-brain.mjs",
"guard:llm-tracked": "node scripts/guard-llm-tracked.mjs",
"guard:engine-registry": "node scripts/guard-engine-registry.mjs",
+ "guard:migration-locks": "node scripts/guard-migration-locks.mjs",
"lint": "biome lint .",
"lint:fix": "biome lint --write .",
"test": "node --import tsx --test \"server/src/__tests__/**/*.test.ts\" \"tests/**/*.test.ts\" \"workers/**/*.test.ts\"",
diff --git a/scripts/guard-migration-locks.d.mts b/scripts/guard-migration-locks.d.mts
new file mode 100644
index 0000000..e6a8e28
--- /dev/null
+++ b/scripts/guard-migration-locks.d.mts
@@ -0,0 +1,17 @@
+// Type declarations for the migration lock guard (scripts/guard-migration-locks.mjs).
+export interface MigrationLockProblem {
+ /** Repo-relative path of the offending migration. */
+ where: string
+ /** What it locks, and how to write it instead. */
+ why: string
+}
+/** Strip block and line comments so prose about a pattern is not read as the pattern. */
+export function stripComments(source: string): string
+/** Lower-cased names of the tables a migration creates itself. */
+export function tablesCreatedIn(sql: string): Set<string>
+/** Every lock hazard in one migration's source text. */
+export function scanMigration(file: string, source: string): MigrationLockProblem[]
+/** Scan every migration except the grandfathered ones. */
+export function scanRepo(): MigrationLockProblem[]
+/** Applied migrations whose checksum-pinned SQL cannot be rewritten, and why. */
+export function grandfathered(): Map<string, string>
diff --git a/scripts/guard-migration-locks.mjs b/scripts/guard-migration-locks.mjs
new file mode 100644
index 0000000..1a3b952
--- /dev/null
+++ b/scripts/guard-migration-locks.mjs
@@ -0,0 +1,173 @@
+#!/usr/bin/env node
+/**
+ * guard-migration-locks — a migration may not take a table-wide lock it does
+ * not need.
+ *
+ * Migrations run from a pre-deploy Job, against production, WHILE the old Pods
+ * are still serving traffic. `ensureSchema` pins that session at
+ * `lock_timeout = '5s'`, so DDL that needs an AccessExclusiveLock on a hot
+ * table has five seconds to get it before the statement aborts with `55P03`
+ * and the deploy fails. Even when it does get the lock, everything else queues
+ * behind it for as long as the statement runs.
+ *
+ * Migration SQL is checksum-pinned once applied, so a migration written this
+ * way cannot be fixed afterwards — only worked around. That is why this is a
+ * pre-merge guard and not a runtime check.
+ *
+ * Two patterns are rejected:
+ *
+ * 1. ADD COLUMN with a VOLATILE default.
+ *
+ * PostgreSQL's metadata-only fast path for `ADD COLUMN … DEFAULT` applies
+ * only to non-volatile defaults. A volatile one — gen_random_uuid(),
+ * random(), clock_timestamp(), nextval() — forces a full table and index
+ * rewrite under AccessExclusiveLock. Migration 0004 did exactly this to
+ * `participants`, a table on the hot path of nearly every query in the
+ * product; it is allowlisted below because its checksum is pinned and its
+ * SQL can no longer be changed.
+ *
+ * Write it as: nullable column (metadata only) → batched backfill →
+ * SET DEFAULT → `CHECK (… IS NOT NULL) NOT VALID` → VALIDATE CONSTRAINT.
+ * Each step is short, and none of them holds a lock across the backfill.
+ *
+ * 2. CREATE INDEX without CONCURRENTLY on a pre-existing table.
+ *
+ * A plain CREATE INDEX blocks writes to the table for the whole build. On
+ * a hot table under live traffic that wait deadlocks (`40P01`) against
+ * concurrent writers, and because the build then never commits, every Pod
+ * retries it on boot and none can start — which is how the
+ * conversations.members GIN build wedged production. Use CREATE INDEX
+ * CONCURRENTLY with `transactional: false`, as 0005 and 0006 do.
+ *
+ * Indexing a table the SAME migration creates is fine and is not flagged:
+ * nothing can be reading or writing a table that did not exist a statement
+ * ago.
+ *
+ * A guard that quietly stops checking is worse than no guard, so `scanRepo()`
+ * reports a problem if the migrations directory ever reads as empty.
+ *
+ * Run: node scripts/guard-migration-locks.mjs
+ */
+import { readdirSync, readFileSync } from 'node:fs'
+import { join } from 'node:path'
+import { fileURLToPath } from 'node:url'
+
+const ROOT = join(fileURLToPath(new URL('.', import.meta.url)), '..')
+const MIGRATIONS_DIR = 'server/src/db/migrations'
+
+/**
+ * Migrations whose SQL is already applied in production and checksum-pinned,
+ * so it cannot be rewritten. Each entry records what it does and why it stays.
+ */
+const GRANDFATHERED = new Map([
+ [
+ '0004-agent-runtime-assignment.ts',
+ 'ADD COLUMN … DEFAULT gen_random_uuid()::text rewrites participants under ' +
+ 'AccessExclusiveLock. Applied and checksum-pinned; changing the SQL would fail ' +
+ 'the checksum check on every existing database.',
+ ],
+])
+
+/** VOLATILE in PostgreSQL's function catalog, so a default built from one
+ * defeats the metadata-only ADD COLUMN fast path. `now()` and
+ * `current_timestamp` are STABLE, not volatile, and are deliberately absent. */
+const VOLATILE_DEFAULTS = [
+ 'gen_random_uuid',
+ 'uuid_generate_v1',
+ 'uuid_generate_v4',
+ 'random',
+ 'clock_timestamp',
+ 'timeofday',
+ 'nextval',
+]
+
+/** Strip comments so prose describing a pattern is never mistaken for it —
+ * 0005's header explains why "a plain CREATE INDEX" is wrong, in a comment. */
+export function stripComments(source) {
+ return source
+ .replace(/\/\*[\s\S]*?\*\//g, ' ')
+ .replace(/(^|[^:])\/\/[^\n]*/g, '$1 ')
+}
+
+/** Tables this file creates itself — indexing them is free of live traffic. */
+export function tablesCreatedIn(sql) {
+ const names = new Set()
+ for (const m of sql.matchAll(/CREATE\s+TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?([a-z_][a-z0-9_]*)/gi)) {
+ names.add(m[1].toLowerCase())
+ }
+ return names
+}
+
+/** Every lock hazard in one migration's source text. */
+export function scanMigration(file, source) {
+ const sql = stripComments(source).replace(/\s+/g, ' ')
+ const created = tablesCreatedIn(sql)
+ const problems = []
+
+ // ADD COLUMN … DEFAULT <volatile>. Bounded to the clause: the default runs to
+ // the next comma or statement end, so a volatile call in a LATER column's
+ // definition is attributed to that column, not this one.
+ for (const m of sql.matchAll(/ADD\s+COLUMN\s+(?:IF\s+NOT\s+EXISTS\s+)?([a-z_][a-z0-9_]*)[^,;]*?DEFAULT\s+([^,;]+)/gi)) {
+ const [, column, expression] = m
+ const volatileFn = VOLATILE_DEFAULTS.find((fn) => new RegExp(`\\b${fn}\\s*\\(`, 'i').test(expression))
+ if (!volatileFn) continue
+ problems.push({
+ where: `${MIGRATIONS_DIR}/${file}`,
+ why: `ADD COLUMN ${column} DEFAULT … ${volatileFn}() is VOLATILE, so PostgreSQL rewrites the whole `
+ + 'table under AccessExclusiveLock instead of taking the metadata-only fast path. Add the column '
+ + 'nullable, backfill in batches, then SET DEFAULT and validate a NOT VALID check constraint.',
+ })
+ }
+
+ // CREATE [UNIQUE] INDEX … ON <table>, without CONCURRENTLY, on a table that
+ // already exists.
+ for (const m of sql.matchAll(/CREATE\s+(?:UNIQUE\s+)?INDEX\s+(CONCURRENTLY\s+)?(?:IF\s+NOT\s+EXISTS\s+)?[a-z_][a-z0-9_$]*\s+ON\s+(?:ONLY\s+)?([a-z_][a-z0-9_]*)/gi)) {
+ const [, concurrently, table] = m
+ if (concurrently) continue
+ if (created.has(table.toLowerCase())) continue
+ problems.push({
+ where: `${MIGRATIONS_DIR}/${file}`,
+ why: `CREATE INDEX on the pre-existing table "${table}" blocks writes for the whole build and `
+ + 'deadlocks against live writers. Use CREATE INDEX CONCURRENTLY with transactional: false.',
+ })
+ }
+
+ return problems
+}
+
+export function scanRepo() {
+ const files = readdirSync(join(ROOT, MIGRATIONS_DIR))
+ .filter((name) => /^\d{4}-.+\.ts$/.test(name))
+ .sort()
+
+ if (files.length === 0) {
+ return [{
+ where: MIGRATIONS_DIR,
+ why: 'no migration files found — the guard is looking in the wrong place and would pass vacuously',
+ }]
+ }
+
+ const problems = []
+ for (const file of files) {
+ if (GRANDFATHERED.has(file)) continue
+ problems.push(...scanMigration(file, readFileSync(join(ROOT, MIGRATIONS_DIR, file), 'utf8')))
+ }
+ return problems
+}
+
+/** Exported so the test can prove the allowlist still describes a real hazard
+ * rather than silently covering a migration that no longer has one. */
+export function grandfathered() {
+ return GRANDFATHERED
+}
+
+if (import.meta.url === `file://${process.argv[1]}`) {
+ const problems = scanRepo()
+ if (problems.length > 0) {
+ console.error('🚨 migration would lock a live table:\n')
+ for (const problem of problems) console.error(` ${problem.where}\n ${problem.why}\n`)
+ console.error('Migration SQL is checksum-pinned once applied — it cannot be fixed after the fact.')
+ process.exit(1)
+ }
+ console.log('✅ migration lock guard: no migration takes a table-wide lock it does not need.')
+}
diff --git a/server/src/__tests__/guard-migration-locks.test.ts b/server/src/__tests__/guard-migration-locks.test.ts
new file mode 100644
index 0000000..8693e59
--- /dev/null
+++ b/server/src/__tests__/guard-migration-locks.test.ts
@@ -0,0 +1,119 @@
+/**
+ * CI enforcement for the migration lock guard.
+ *
+ * Migrations run from a pre-deploy Job while the old Pods still serve traffic,
+ * and `ensureSchema` pins that session at `lock_timeout = '5s'`. DDL that needs
+ * an AccessExclusiveLock on a hot table therefore has five seconds to get it
+ * before the statement aborts with `55P03` — and once a migration is applied,
+ * its SQL is checksum-pinned and can never be rewritten. Catching the pattern
+ * before merge is the only place it can be caught at all.
+ *
+ * See scripts/guard-migration-locks.mjs.
+ */
+import { test } from 'node:test'
+import assert from 'node:assert/strict'
+// The guard is plain ESM tooling; tsx lets this .ts test import the .mjs.
+import {
+ grandfathered,
+ scanMigration,
+ scanRepo,
+ stripComments,
+ tablesCreatedIn,
+} from '../../../scripts/guard-migration-locks.mjs'
+import { readFileSync, readdirSync } from 'node:fs'
+import { join } from 'node:path'
+
+const ROOT = join(import.meta.dirname, '..', '..', '..')
+const MIGRATIONS = join(ROOT, 'server/src/db/migrations')
+const migration = (file: string): string => readFileSync(join(MIGRATIONS, file), 'utf8')
+
+test('no migration takes a table-wide lock it does not need', () => {
+ const problems = scanRepo()
+ assert.deepEqual(
+ problems, [],
+ '\n🚨 migration would lock a live table:\n' +
+ problems.map((p: { where: string; why: string }) => ` ${p.where}\n ${p.why}`).join('\n'),
+ )
+})
+
+test('the guard is looking at the real migration files', () => {
+ // If the directory or the filename pattern ever moved, scanRepo() would have
+ // nothing to scan and would pass vacuously. scanRepo() reports that as a
+ // problem; pin the happy path here too.
+ const files = readdirSync(MIGRATIONS).filter((name) => /^\d{4}-.+\.ts$/.test(name))
+ assert.ok(files.length >= 8, `expected the real migration set, found ${files.length}`)
+})
+
+// --- the guard actually catches both hazards (so it is not a no-op) ---
+
+test('a volatile ADD COLUMN default is caught', () => {
+ // Exactly migration 0004's shape, which rewrites `participants` — a table on
+ // the hot path of nearly every query — under AccessExclusiveLock.
+ const problems = scanMigration(
+ 'x.ts',
+ 'ALTER TABLE participants ADD COLUMN runtime_assignment_id TEXT NOT NULL DEFAULT gen_random_uuid()::text;',
+ )
+ assert.equal(problems.length, 1)
+ assert.match(problems[0].why, /VOLATILE/)
+ assert.match(problems[0].why, /nullable/)
+
+ // And the real file is caught when it is not treated as grandfathered, which
+ // is what keeps the allowlist entry honest.
+ const live = scanMigration('0004.ts', migration('0004-agent-runtime-assignment.ts'))
+ assert.equal(live.length, 1, 'the allowlisted migration must still be a real hazard')
+})
+
+test('a non-volatile default is not caught', () => {
+ // now() is STABLE, and a literal is IMMUTABLE — both take PostgreSQL's
+ // metadata-only fast path, so flagging them would be a false positive that
+ // teaches people to ignore the guard.
+ assert.deepEqual(scanMigration('x.ts', "ALTER TABLE computers ADD COLUMN engine_defaults JSONB NOT NULL DEFAULT '{}'::jsonb;"), [])
+ assert.deepEqual(scanMigration('x.ts', 'ALTER TABLE messages ADD COLUMN seen_at TIMESTAMPTZ NOT NULL DEFAULT now();'), [])
+ assert.deepEqual(scanMigration('x.ts', 'ALTER TABLE participants ADD COLUMN provider_profile TEXT;'), [])
+})
+
+test('a plain CREATE INDEX on a pre-existing table is caught', () => {
+ const problems = scanMigration('x.ts', 'CREATE INDEX idx_messages_convo ON messages (conversation_id);')
+ assert.equal(problems.length, 1)
+ assert.match(problems[0].why, /CONCURRENTLY/)
+
+ assert.deepEqual(
+ scanMigration('x.ts', 'CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_messages_convo ON messages (conversation_id);'),
+ [],
+ )
+})
+
+test('indexing a table the same migration creates is not caught', () => {
+ // Nothing can be reading or writing a table that did not exist a statement
+ // ago, so CONCURRENTLY buys nothing there — and would force the migration
+ // out of its transaction for no reason.
+ assert.deepEqual(
+ scanMigration('x.ts', `
+ CREATE TABLE conversation_members (id TEXT PRIMARY KEY, conversation_id TEXT NOT NULL);
+ CREATE INDEX idx_conversation_members_participant ON conversation_members (conversation_id);
+ `),
+ [],
+ )
+ assert.ok(tablesCreatedIn('CREATE TABLE IF NOT EXISTS agent_routing_claims (id TEXT)').has('agent_routing_claims'))
+})
+
+test('prose about a bad pattern is not mistaken for the pattern', () => {
+ // 0005's header comment explains why "a plain CREATE INDEX" is wrong. A guard
+ // that flagged its own documentation would be unusable.
+ assert.deepEqual(scanMigration('0005.ts', migration('0005-search-trigram-index.ts')), [])
+ assert.match(stripComments('/* CREATE INDEX idx ON messages (a) */ SELECT 1'), /^\s+SELECT 1$/)
+ assert.match(stripComments('// ADD COLUMN x DEFAULT gen_random_uuid()\nSELECT 1'), /^\s*\nSELECT 1$/)
+})
+
+test('every grandfathered migration says what it does and why it stays', () => {
+ const entries = grandfathered()
+ assert.ok(entries.size > 0, 'the allowlist should still cover migration 0004')
+ for (const [file, reason] of entries) {
+ assert.ok(
+ readdirSync(MIGRATIONS).includes(file),
+ `allowlist names ${file}, which no longer exists — drop the entry`,
+ )
+ assert.ok(reason.length > 40, `${file} needs a real reason, not "${reason}"`)
+ assert.match(reason, /checksum/i, `${file}'s reason must say why the SQL cannot be changed`)
+ }
+})
diff --git a/server/src/__tests__/migration-transaction-atomicity.test.ts b/server/src/__tests__/migration-transaction-atomicity.test.ts
index 5230bd0..6dbba8b 100644
--- a/server/src/__tests__/migration-transaction-atomicity.test.ts
+++ b/server/src/__tests__/migration-transaction-atomicity.test.ts
@@ -153,3 +153,127 @@ test('rollback failure still preserves original migration error', async () => {
/primary migration error/,
)
})
+
+/** A PostgreSQL error as node-postgres surfaces it: the SQLSTATE is on `code`. */
+function pgError(code: string, message: string): Error & { code: string } {
+ return Object.assign(new Error(message), { code })
+}
+
+test('a migration that loses a lock race is retried after a clean rollback', async () => {
+ const { client, calls } = fakeClient()
+ let attempts = 0
+ const slept: number[] = []
+
+ const migration: VersionedMigration = {
+ version: 4,
+ name: '0004-lock-contention',
+ checksum: 'f'.repeat(64),
+ sourceChecksum: 'f'.repeat(64),
+ transactional: true,
+ up: async () => {
+ attempts += 1
+ calls.push({ sql: `ALTER TABLE participants ADD COLUMN x TEXT -- attempt ${attempts}` })
+ // `ensureSchema` pins lock_timeout at 5s, so DDL on a hot table under
+ // live traffic aborts with 55P03 long before it does any work.
+ if (attempts < 3) throw pgError('55P03', 'canceling statement due to lock timeout')
+ },
+ }
+
+ await applyPendingMigration(client, migration, { sleep: async (ms) => { slept.push(ms) } })
+
+ assert.equal(attempts, 3)
+ const statements = calls.map((c) => c.sql)
+ // Each failed attempt rolls back completely, so attempt N+1 starts from
+ // exactly the state attempt N saw. No partial schema, no partial ledger.
+ assert.equal(statements.filter((s) => s === 'BEGIN').length, 3)
+ assert.equal(statements.filter((s) => s === 'ROLLBACK').length, 2)
+ assert.equal(statements.filter((s) => s === 'COMMIT').length, 1)
+ assert.equal(statements.filter((s) => s.startsWith('INSERT INTO schema_migrations')).length, 1)
+ assert.equal(slept.length, 2, 'each retry must back off')
+ assert.ok(slept[1] > slept[0], `backoff must grow, got ${slept.join(', ')}`)
+})
+
+test('deadlock and serialization failures retry too; a real error never does', async () => {
+ for (const code of ['40P01', '55P03', '40001']) {
+ const { client } = fakeClient()
+ let attempts = 0
+ const migration: VersionedMigration = {
+ version: 4,
+ name: `0004-${code}`,
+ checksum: 'f'.repeat(64),
+ sourceChecksum: 'f'.repeat(64),
+ transactional: true,
+ up: async () => {
+ attempts += 1
+ if (attempts < 2) throw pgError(code, `transient ${code}`)
+ },
+ }
+ await applyPendingMigration(client, migration, { sleep: async () => {} })
+ assert.equal(attempts, 2, `${code} should have been retried once`)
+ }
+
+ // A missing relation, a failed precheck, a constraint violation: the deploy
+ // has to stop on the first attempt and say so.
+ const { client } = fakeClient()
+ let realAttempts = 0
+ await assert.rejects(
+ () => applyPendingMigration(client, {
+ version: 4,
+ name: '0004-real-error',
+ checksum: 'f'.repeat(64),
+ sourceChecksum: 'f'.repeat(64),
+ transactional: true,
+ up: async () => {
+ realAttempts += 1
+ throw pgError('42P01', 'relation "nope" does not exist')
+ },
+ }, { sleep: async () => {} }),
+ /relation "nope" does not exist/,
+ )
+ assert.equal(realAttempts, 1, 'a real error must not be retried')
+})
+
+test('sustained lock contention gives up and fails the deploy', async () => {
+ const { client } = fakeClient()
+ let attempts = 0
+
+ await assert.rejects(
+ () => applyPendingMigration(client, {
+ version: 4,
+ name: '0004-always-locked',
+ checksum: 'f'.repeat(64),
+ sourceChecksum: 'f'.repeat(64),
+ transactional: true,
+ up: async () => {
+ attempts += 1
+ throw pgError('55P03', 'canceling statement due to lock timeout')
+ },
+ }, { sleep: async () => {}, maxAttempts: 4 }),
+ /lock timeout/,
+ )
+ assert.equal(attempts, 4, 'the budget is finite — a wedged table must not hang the Job forever')
+})
+
+test('a non-transactional migration is never retried wholesale', async () => {
+ // 0001 retries per statement internally and 0005/0006 build indexes
+ // CONCURRENTLY; re-running either from here could re-execute work that
+ // already committed, because there is no transaction to roll back.
+ const { client } = fakeClient()
+ let attempts = 0
+
+ await assert.rejects(
+ () => applyPendingMigration(client, {
+ version: 5,
+ name: '0005-concurrent',
+ checksum: 'f'.repeat(64),
+ sourceChecksum: 'f'.repeat(64),
+ transactional: false,
+ up: async () => {
+ attempts += 1
+ throw pgError('55P03', 'canceling statement due to lock timeout')
+ },
+ }, { sleep: async () => {} }),
+ /lock timeout/,
+ )
+ assert.equal(attempts, 1)
+})
diff --git a/server/src/db/migrate.ts b/server/src/db/migrate.ts
index 6134ab4..cc101b7 100644
--- a/server/src/db/migrate.ts
+++ b/server/src/db/migrate.ts
@@ -2618,25 +2618,77 @@ const VERSIONED_MIGRATIONS: readonly VersionedMigration[] = [
},
]
+/** How many times a transactional migration may lose a lock race before the
+ * deploy gives up. Matches the frozen baseline's per-statement budget. */
+const MIGRATION_LOCK_ATTEMPTS = 6
+
+/** `true` for the transient lock-contention SQLSTATEs — the ones where the
+ * migration never started, so a clean retry is the same as a first attempt. */
+function isTransientLockError(err: unknown): boolean {
+ const code = (err as { code?: unknown } | null)?.code
+ return typeof code === 'string' && BASELINE_STATEMENT_LOCK_CODES.has(code)
+}
+
+/**
+ * Apply one migration, retrying a transactional one that loses a lock race.
+ *
+ * `ensureSchema` pins the migration session at `lock_timeout = '5s'` — the
+ * right call, since a migration that queues indefinitely for an
+ * AccessExclusiveLock parks every later query behind it — but it means DDL that
+ * cannot get its lock within five seconds aborts with `55P03` and, without
+ * this loop, takes the whole pre-deploy Job down with it. The Job runs while
+ * the old Pods are still serving, so on a busy table that is not an unlikely
+ * outcome; it is the normal one, and it repeats on every rerun.
+ *
+ * Migration 0004 is the standing example. `ALTER TABLE participants ADD COLUMN
+ * runtime_assignment_id TEXT NOT NULL DEFAULT gen_random_uuid()::text` takes
+ * AccessExclusiveLock on `participants`, which sits on the hot path of nearly
+ * every query in the product. Its SQL is checksum-pinned and cannot be edited
+ * — see scripts/guard-migration-locks.mjs for the authoring rule that keeps the
+ * next one from being written the same way.
+ *
+ * Only `transactional` migrations retry. A failed transaction has rolled back
+ * completely, so attempt two starts from exactly the state attempt one saw. The
+ * `transactional: false` ones (the frozen baseline and the CONCURRENTLY index
+ * builds) manage their own idempotence and their own retries, and re-running
+ * them wholesale from here could re-execute work that already committed.
+ *
+ * Only the transient lock SQLSTATEs retry: a real error — a missing relation, a
+ * constraint violation, a failed precheck — still fails the deploy on its first
+ * attempt, loudly, which is the point of the pre-deploy Job.
+ */
export async function applyPendingMigration(
client: import('pg').PoolClient,
migration: VersionedMigration,
+ opts: { sleep?: (ms: number) => Promise<void>; maxAttempts?: number } = {},
): Promise<void> {
+ const sleep = opts.sleep ?? ((ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms)))
+ const maxAttempts = opts.maxAttempts ?? MIGRATION_LOCK_ATTEMPTS
const started = Date.now()
console.log(`[db] applying migration ${migration.version} ${migration.name}`)
if (migration.transactional !== false) {
- await client.query('BEGIN')
- try {
- await migration.up(client)
- await client.query(
- `INSERT INTO schema_migrations (version, name, checksum, execution_ms)
- VALUES ($1, $2, $3, $4)`,
- [migration.version, migration.name, migration.checksum, Date.now() - started],
- )
- await client.query('COMMIT')
- } catch (err) {
- await client.query('ROLLBACK').catch(() => { /* swallow rollback err */ })
- throw err
+ for (let attempt = 1; ; attempt++) {
+ const attemptStarted = Date.now()
+ await client.query('BEGIN')
+ try {
+ await migration.up(client)
+ await client.query(
+ `INSERT INTO schema_migrations (version, name, checksum, execution_ms)
+ VALUES ($1, $2, $3, $4)`,
+ [migration.version, migration.name, migration.checksum, Date.now() - attemptStarted],
+ )
+ await client.query('COMMIT')
+ break
+ } catch (err) {
+ await client.query('ROLLBACK').catch(() => { /* swallow rollback err */ })
+ if (!isTransientLockError(err) || attempt >= maxAttempts) throw err
+ const backoffMs = Math.min(8_000, 500 * 2 ** (attempt - 1)) + Math.floor(Math.random() * 250)
+ console.warn(
+ `[db] migration ${migration.version} ${migration.name} hit ${(err as { code: string }).code} ` +
+ `(attempt ${attempt}/${maxAttempts}) — retrying in ${backoffMs}ms`,
+ )
+ await sleep(backoffMs)
+ }
}
} else {
await migration.up(client)