OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.
fix(api): code migrations always run all pending ones, in file order (#592)
Carl-Gerhard Lindesvärd03e63cd88851
diff --git a/.env.example b/.env.exampleindex a7cb1bb49..dd3d48ddd 100644--- a/.env.example+++ b/.env.example@@ -10,10 +10,10 @@ CLICKHOUSE_URL="http://localhost:23123/openpanel"ENCRYPTION_KEY=""# CODE MIGRATIONS-# Lets the api/worker apply pending ClickHouse code migrations once it is-# listening (blocking ones first, then online data moves); a Redis lock makes-# one process do it. Self-hosted installs set it; without it, run them with-# `bun run --filter @openpanel/db migrate:deploy` (blocking only) or by name.+# Lets the api/worker apply every pending ClickHouse code migration, in order,+# once it is listening; a Redis lock makes one process do it. Self-hosted+# installs set it; without it, run them all with+# `bun run --filter @openpanel/db migrate:deploy`.# AUTOMATIC_MIGRATIONS="true"# OBJECT-STORE EXPORT (S3/GCS) tuning — optional, sensible defaults shown.diff --git a/AGENTS.md b/AGENTS.mdindex 1cd4ed807..5285f831d 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -159,35 +159,28 @@ on the same data.## Migrations-The api starts serving the moment its container starts. Only Prisma runs before it;-with `AUTOMATIC_MIGRATIONS=true` (self-hosted installs set it, cloud decides per-deploy) the api applies pending ClickHouse code migrations itself once it is listening,-one process at a time (`startCodeMigrations` in-`packages/core/src/modules/code-migration/`), every blocking one first, then the online-ones. So write every change in two parts:--- **Blocking** (the default): the fast breaking change the new code needs — a new- table or column, a metadata-only `ALTER`. Seconds, whatever the data size: until it- lands, the new code runs against the old schema, so keep that window short. The- runner warns when one takes over 30 s. A failed migration is retried, so make- each statement safe to repeat (`IF NOT EXISTS`, `IF EXISTS`) for when an earlier- statement of the same migration already ran.-- **Online** (`export const online = true`): anything that grows with the data —- rebuilding or copying a table, a backfill. It runs after the blocking ones, in the- background, and it must:- - stay correct while ingestion and the dashboard read and write the table. If the- write path reads the table back (the profile buffer does), copy before you swap;- otherwise swap first and copy after (`28-sessions-sort-key-id.ts`);- - be safe to rerun from any point, since a process can die mid-way and another one- resumes it;- - never be something a blocking migration depends on.--A change that needs both ships as two migrations: a blocking one that lets the new code-work with the old data, then an online one that moves the data. Time a blocking-migration against a large seed (`bun run seed --size xl`) before calling it fast.-`migrate:deploy` runs only the blocking ones, so a fresh database has its schema before-anything seeds it; set `AUTOMATIC_MIGRATIONS=true` for your api to apply the online ones,-or run one by name (`bun ./scripts/migrate-code.ts <file>` in `packages/db`).+The api starts serving the moment its container starts; only Prisma runs before it.+ClickHouse code migrations run while it serves, always all pending ones, always in+file order:++- With `AUTOMATIC_MIGRATIONS=true` (self-hosted installs) the api applies them itself+ once it is listening, one process at a time (`startCodeMigrations` in+ `packages/core/src/modules/code-migration/`).+- Without it (cloud), the operator runs `migrate:deploy` (or+ `bun ./scripts/migrate-code.ts` in `packages/db`) against the running deployment.+ A file name as argument runs just that one.++So until every migration has landed, the new code runs against the old schema, and a+long data move holds back every migration numbered after it. Write migrations so that:++- each one is safe to rerun from any point: a process can die mid-way and another one+ resumes it, and a failed one is retried (`IF NOT EXISTS`, `IF EXISTS`, check before+ you swap);+- a data move stays correct while ingestion and the dashboard read and write the+ table. If the write path reads the table back (the profile buffer does), copy before+ you swap; otherwise swap first and copy after (`28-sessions-sort-key-id.ts`);+- a schema change the new code needs comes before a long data move in the same+ release, so it lands in seconds rather than after the copy.## Code stylediff --git a/apps/api/src/main.ts b/apps/api/src/main.tsindex af897847a..a9ba1f797 100644--- a/apps/api/src/main.ts+++ b/apps/api/src/main.ts@@ -491,8 +491,8 @@ async function main() {);});- // Right after listening, so the api serves from the start and the fast- // blocking migrations land within seconds; see AGENTS.md → Migrations.+ // Right after listening, so the api serves while pending migrations run;+ // see AGENTS.md → Migrations.const codeMigrations = config.AUTOMATIC_MIGRATIONS? startCodeMigrations({redis: deps.redis,diff --git a/docs/DOCKER_IMAGES.md b/docs/DOCKER_IMAGES.mdindex 6090c8a46..9ccf7ca3f 100644--- a/docs/DOCKER_IMAGES.md+++ b/docs/DOCKER_IMAGES.md@@ -122,10 +122,10 @@ cd /app/packages/db && prisma migrate deployand, with `AUTOMATIC_MIGRATIONS=true` (set by the shipped compose `.env` andthe Coolify template), the api applies pending ClickHouse code migrations-itself once it is listening (Carl, 2026-10-20: the api starts at once, fast-breaking changes first, long data moves after; see AGENTS.md → Migrations). Installs whose-compose file still runs `cd /app/packages/core && bun scripts/migrate-code.ts`-first keep working; that step simply finds less to do.+itself, in order, once it is listening (see AGENTS.md → Migrations). Installs+whose compose file still runs `cd /app/packages/core && bun scripts/migrate-code.ts`+first keep working, but wait for every pending migration before the api starts+until `openpanel doctor` rewrites the command.- **The image ships the Prisma CLI** (Carl, 2026-10-20), installed on its own in`/opt/prisma` and on `PATH`, engines included, so a container start reachesdiff --git a/packages/core/src/modules/code-migration/code-migration.service.ts b/packages/core/src/modules/code-migration/code-migration.service.tsindex 86861271f..a6995cf57 100644--- a/packages/core/src/modules/code-migration/code-migration.service.ts+++ b/packages/core/src/modules/code-migration/code-migration.service.ts@@ -29,7 +29,7 @@ return 0`);export interface CodeMigrationRunner {- /** Unfinished migrations in run order: blocking first, then online. */+ /** Unfinished migrations in file order. */getPending: () => Promise<string[]>;/** Runs one migration and records it as finished; throws on failure. */run: (file: string) => Promise<void>;@@ -73,7 +73,7 @@ type AttemptResult = 'finished' | 'retry';* Every process calls this once it listens; the lock picks one to do the* work, so replicas never run a migration twice at once. The others check* back every `retryIntervalMs`, so a holder that dies mid-way is replaced and- * the migration resumes, which online migrations must support.+ * the migration resumes, which every migration must support.*/export function startCodeMigrations(deps: CodeMigrationDeps,@@ -106,7 +106,7 @@ export function startCodeMigrations(return;}// Another process may take over now; two copies running at once is the- // one thing an online migration cannot survive.+ // one thing a migration cannot survive.isLockLost = true;logger.error({ err: error }, message);migrations.abort();diff --git a/packages/db/src/code-migrations/28-sessions-sort-key-id.ts b/packages/db/src/code-migrations/28-sessions-sort-key-id.tsindex 58accd0b3..98abb72fa 100644--- a/packages/db/src/code-migrations/28-sessions-sort-key-id.ts+++ b/packages/db/src/code-migrations/28-sessions-sort-key-id.ts@@ -217,9 +217,6 @@ async function getDaysToCopy(source: string) {return rows.map((row) => row.day);}-/** Runs in the background once the api is up; see the rerun notes above. */-export const online = true;-export async function up(env: CodeMigrationEnv) {const isClustered = getIsCluster(env);const isDry = process.argv.includes('--dry');diff --git a/packages/db/src/code-migrations/migrate.ts b/packages/db/src/code-migrations/migrate.tsindex 1d3d41864..2749448f7 100644--- a/packages/db/src/code-migrations/migrate.ts+++ b/packages/db/src/code-migrations/migrate.ts@@ -14,20 +14,9 @@ import {} from './helpers';const MIGRATIONS_DIR = dirname(fileURLToPath(import.meta.url));-// The api is already serving while blocking migrations run, against the old-// schema until they finish, so they must stay fast.-const SLOW_BLOCKING_MIGRATION_MS = 30_000;-/**- * A code migration module. The api applies pending ones after it starts- * listening (`startCodeMigrations` in @openpanel/core): every blocking one- * first, then the `online` ones. `online` marks work that grows with the data:- * it must stay correct under live traffic and resume after an interruption,- * and no blocking migration may depend on it.- */interface CodeMigration {up: (env: CodeMigrationEnv) => Promise<void>;- online?: boolean;}function migrationVersion(file: string) {@@ -91,23 +80,11 @@ async function assertClusterModeMatchesDatabase(env: CodeMigrationEnv) {}}-/**- * Runs one migration and records it as finished. Throws on failure. Warns- * when a blocking migration is slow enough to be a long window of the api- * running against the old schema.- */+/** Runs one migration and records it as finished. Throws on failure. */export async function runMigration(file: string, env: CodeMigrationEnv) {await assertClusterModeMatchesDatabase(env);const migration = await loadMigration(file);- const startedAt = performance.now();await migration.up(env);- const durationMs = performance.now() - startedAt;- if (!migration.online && durationMs > SLOW_BLOCKING_MIGRATION_MS) {- printBoxMessage('🐢 Slow blocking migration', [- `${file} took ${Math.round(durationMs / 1000)}s; the api ran against the old schema meanwhile.`,- 'Work that grows with the data belongs in an online migration (AGENTS.md → Migrations).',- ]);- }if (!(getIsDry() || getShouldIgnoreRecord())) {await db.codeMigration.upsert({where: { name: file },@@ -118,24 +95,12 @@ export async function runMigration(file: string, env: CodeMigrationEnv) {}/**- * Unfinished migrations in the order the api runs them: blocking ones first,- * so a fast breaking change never waits behind a long online data move.+ * Unfinished migrations in file order. Never reordered: a migration may+ * depend on everything numbered before it, data moves included.*/export async function getPendingMigrations() {const finished = await getFinishedMigrationNames();- const blocking: string[] = [];- const online: string[] = [];- for (const file of listMigrationFiles()) {- if (finished.has(file)) {- continue;- }- if ((await loadMigration(file)).online) {- online.push(file);- } else {- blocking.push(file);- }- }- return [...blocking, ...online];+ return listMigrationFiles().filter((file) => !finished.has(file));}async function runMigrationOrExit(file: string, env: CodeMigrationEnv) {@@ -152,35 +117,23 @@ async function runMigrationOrExit(file: string, env: CodeMigrationEnv) {}/**- * The command-line runner (`migrate:deploy`, for development, CI and- * operators): every unfinished blocking migration, in order, so a fresh- * database has its schema before anything seeds it. Online ones are left to- * the api. A migration named on the command line runs either way.+ * The command-line runner (`migrate:deploy`, and operators who do not set+ * AUTOMATIC_MIGRATIONS): every pending migration, in order. The api may keep+ * serving meanwhile. A migration named on the command line runs on its own.*/export async function runCodeMigrations(env: CodeMigrationEnv) {const args = process.argv.slice(2);const named = args.filter((arg) => !arg.startsWith('--'))[0];const finished = await getFinishedMigrationNames();- const unfinished = listMigrationFiles().filter((file) => !finished.has(file));- const online = new Set<string>();- for (const file of unfinished) {- if ((await loadMigration(file)).online) {- online.add(file);- }- }+ const pending = await getPendingMigrations();printBoxMessage('📋 Plan', ['\t✅ Finished:',...[...finished].map((name) => `\t- ${name}`),'','\t🔄 Will run now:',- ...unfinished- .filter((file) => !online.has(file))- .map((file) => `\t- ${file}`),- '',- '\t🌙 Online, applied by the api once it is up:',- ...[...online].map((file) => `\t- ${file}`),+ ...(named ? [named] : pending).map((file) => `\t- ${file}`),]);printBoxMessage('🤝 Config', [@@ -205,10 +158,8 @@ export async function runCodeMigrations(env: CodeMigrationEnv) {if (named) {await runMigrationOrExit(named, env);} else {- for (const file of unfinished) {- if (!online.has(file)) {- await runMigrationOrExit(file, env);- }+ for (const file of pending) {+ await runMigrationOrExit(file, env);}}