Skip to content

OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.

fix(api): code migrations always run all pending ones, in file order (#592)

Carl-Gerhard Lindesvärd03e63cd88851
diff --git a/.env.example b/.env.example
index a7cb1bb49..dd3d48ddd 100644
--- a/.env.example
+++ b/.env.example
@@ -10,10 +10,10 @@ CLICKHOUSE_URL="http://localhost:23123/openpanel"
ENCRYPTION_KEY=""
# CODE MIGRATIONS
-# Lets the api/worker apply pending ClickHouse code migrations once it is
-# listening (blocking ones first, then online data moves); a Redis lock makes
-# one process do it. Self-hosted installs set it; without it, run them with
-# `bun run --filter @openpanel/db migrate:deploy` (blocking only) or by name.
+# Lets the api/worker apply every pending ClickHouse code migration, in order,
+# once it is listening; a Redis lock makes one process do it. Self-hosted
+# installs set it; without it, run them all with
+# `bun run --filter @openpanel/db migrate:deploy`.
# AUTOMATIC_MIGRATIONS="true"
# OBJECT-STORE EXPORT (S3/GCS) tuning — optional, sensible defaults shown.
diff --git a/AGENTS.md b/AGENTS.md
index 1cd4ed807..5285f831d 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -159,35 +159,28 @@ on the same data.
## Migrations
-The api starts serving the moment its container starts. Only Prisma runs before it;
-with `AUTOMATIC_MIGRATIONS=true` (self-hosted installs set it, cloud decides per
-deploy) the api applies pending ClickHouse code migrations itself once it is listening,
-one process at a time (`startCodeMigrations` in
-`packages/core/src/modules/code-migration/`), every blocking one first, then the online
-ones. So write every change in two parts:
-
-- **Blocking** (the default): the fast breaking change the new code needs — a new
- table or column, a metadata-only `ALTER`. Seconds, whatever the data size: until it
- lands, the new code runs against the old schema, so keep that window short. The
- runner warns when one takes over 30 s. A failed migration is retried, so make
- each statement safe to repeat (`IF NOT EXISTS`, `IF EXISTS`) for when an earlier
- statement of the same migration already ran.
-- **Online** (`export const online = true`): anything that grows with the data —
- rebuilding or copying a table, a backfill. It runs after the blocking ones, in the
- background, and it must:
- - stay correct while ingestion and the dashboard read and write the table. If the
- write path reads the table back (the profile buffer does), copy before you swap;
- otherwise swap first and copy after (`28-sessions-sort-key-id.ts`);
- - be safe to rerun from any point, since a process can die mid-way and another one
- resumes it;
- - never be something a blocking migration depends on.
-
-A change that needs both ships as two migrations: a blocking one that lets the new code
-work with the old data, then an online one that moves the data. Time a blocking
-migration against a large seed (`bun run seed --size xl`) before calling it fast.
-`migrate:deploy` runs only the blocking ones, so a fresh database has its schema before
-anything seeds it; set `AUTOMATIC_MIGRATIONS=true` for your api to apply the online ones,
-or run one by name (`bun ./scripts/migrate-code.ts <file>` in `packages/db`).
+The api starts serving the moment its container starts; only Prisma runs before it.
+ClickHouse code migrations run while it serves, always all pending ones, always in
+file order:
+
+- With `AUTOMATIC_MIGRATIONS=true` (self-hosted installs) the api applies them itself
+ once it is listening, one process at a time (`startCodeMigrations` in
+ `packages/core/src/modules/code-migration/`).
+- Without it (cloud), the operator runs `migrate:deploy` (or
+ `bun ./scripts/migrate-code.ts` in `packages/db`) against the running deployment.
+ A file name as argument runs just that one.
+
+So until every migration has landed, the new code runs against the old schema, and a
+long data move holds back every migration numbered after it. Write migrations so that:
+
+- each one is safe to rerun from any point: a process can die mid-way and another one
+ resumes it, and a failed one is retried (`IF NOT EXISTS`, `IF EXISTS`, check before
+ you swap);
+- a data move stays correct while ingestion and the dashboard read and write the
+ table. If the write path reads the table back (the profile buffer does), copy before
+ you swap; otherwise swap first and copy after (`28-sessions-sort-key-id.ts`);
+- a schema change the new code needs comes before a long data move in the same
+ release, so it lands in seconds rather than after the copy.
## Code style
diff --git a/apps/api/src/main.ts b/apps/api/src/main.ts
index af897847a..a9ba1f797 100644
--- a/apps/api/src/main.ts
+++ b/apps/api/src/main.ts
@@ -491,8 +491,8 @@ async function main() {
);
});
- // Right after listening, so the api serves from the start and the fast
- // blocking migrations land within seconds; see AGENTS.md → Migrations.
+ // Right after listening, so the api serves while pending migrations run;
+ // see AGENTS.md → Migrations.
const codeMigrations = config.AUTOMATIC_MIGRATIONS
? startCodeMigrations({
redis: deps.redis,
diff --git a/docs/DOCKER_IMAGES.md b/docs/DOCKER_IMAGES.md
index 6090c8a46..9ccf7ca3f 100644
--- a/docs/DOCKER_IMAGES.md
+++ b/docs/DOCKER_IMAGES.md
@@ -122,10 +122,10 @@ cd /app/packages/db && prisma migrate deploy
and, with `AUTOMATIC_MIGRATIONS=true` (set by the shipped compose `.env` and
the Coolify template), the api applies pending ClickHouse code migrations
-itself once it is listening (Carl, 2026-10-20: the api starts at once, fast
-breaking changes first, long data moves after; see AGENTS.md → Migrations). Installs whose
-compose file still runs `cd /app/packages/core && bun scripts/migrate-code.ts`
-first keep working; that step simply finds less to do.
+itself, in order, once it is listening (see AGENTS.md → Migrations). Installs
+whose compose file still runs `cd /app/packages/core && bun scripts/migrate-code.ts`
+first keep working, but wait for every pending migration before the api starts
+until `openpanel doctor` rewrites the command.
- **The image ships the Prisma CLI** (Carl, 2026-10-20), installed on its own in
`/opt/prisma` and on `PATH`, engines included, so a container start reaches
diff --git a/packages/core/src/modules/code-migration/code-migration.service.ts b/packages/core/src/modules/code-migration/code-migration.service.ts
index 86861271f..a6995cf57 100644
--- a/packages/core/src/modules/code-migration/code-migration.service.ts
+++ b/packages/core/src/modules/code-migration/code-migration.service.ts
@@ -29,7 +29,7 @@ return 0
`);
export interface CodeMigrationRunner {
- /** Unfinished migrations in run order: blocking first, then online. */
+ /** Unfinished migrations in file order. */
getPending: () => Promise<string[]>;
/** Runs one migration and records it as finished; throws on failure. */
run: (file: string) => Promise<void>;
@@ -73,7 +73,7 @@ type AttemptResult = 'finished' | 'retry';
* Every process calls this once it listens; the lock picks one to do the
* work, so replicas never run a migration twice at once. The others check
* back every `retryIntervalMs`, so a holder that dies mid-way is replaced and
- * the migration resumes, which online migrations must support.
+ * the migration resumes, which every migration must support.
*/
export function startCodeMigrations(
deps: CodeMigrationDeps,
@@ -106,7 +106,7 @@ export function startCodeMigrations(
return;
}
// Another process may take over now; two copies running at once is the
- // one thing an online migration cannot survive.
+ // one thing a migration cannot survive.
isLockLost = true;
logger.error({ err: error }, message);
migrations.abort();
diff --git a/packages/db/src/code-migrations/28-sessions-sort-key-id.ts b/packages/db/src/code-migrations/28-sessions-sort-key-id.ts
index 58accd0b3..98abb72fa 100644
--- a/packages/db/src/code-migrations/28-sessions-sort-key-id.ts
+++ b/packages/db/src/code-migrations/28-sessions-sort-key-id.ts
@@ -217,9 +217,6 @@ async function getDaysToCopy(source: string) {
return rows.map((row) => row.day);
}
-/** Runs in the background once the api is up; see the rerun notes above. */
-export const online = true;
-
export async function up(env: CodeMigrationEnv) {
const isClustered = getIsCluster(env);
const isDry = process.argv.includes('--dry');
diff --git a/packages/db/src/code-migrations/migrate.ts b/packages/db/src/code-migrations/migrate.ts
index 1d3d41864..2749448f7 100644
--- a/packages/db/src/code-migrations/migrate.ts
+++ b/packages/db/src/code-migrations/migrate.ts
@@ -14,20 +14,9 @@ import {
} from './helpers';
const MIGRATIONS_DIR = dirname(fileURLToPath(import.meta.url));
-// The api is already serving while blocking migrations run, against the old
-// schema until they finish, so they must stay fast.
-const SLOW_BLOCKING_MIGRATION_MS = 30_000;
-/**
- * A code migration module. The api applies pending ones after it starts
- * listening (`startCodeMigrations` in @openpanel/core): every blocking one
- * first, then the `online` ones. `online` marks work that grows with the data:
- * it must stay correct under live traffic and resume after an interruption,
- * and no blocking migration may depend on it.
- */
interface CodeMigration {
up: (env: CodeMigrationEnv) => Promise<void>;
- online?: boolean;
}
function migrationVersion(file: string) {
@@ -91,23 +80,11 @@ async function assertClusterModeMatchesDatabase(env: CodeMigrationEnv) {
}
}
-/**
- * Runs one migration and records it as finished. Throws on failure. Warns
- * when a blocking migration is slow enough to be a long window of the api
- * running against the old schema.
- */
+/** Runs one migration and records it as finished. Throws on failure. */
export async function runMigration(file: string, env: CodeMigrationEnv) {
await assertClusterModeMatchesDatabase(env);
const migration = await loadMigration(file);
- const startedAt = performance.now();
await migration.up(env);
- const durationMs = performance.now() - startedAt;
- if (!migration.online && durationMs > SLOW_BLOCKING_MIGRATION_MS) {
- printBoxMessage('🐢 Slow blocking migration', [
- `${file} took ${Math.round(durationMs / 1000)}s; the api ran against the old schema meanwhile.`,
- 'Work that grows with the data belongs in an online migration (AGENTS.md → Migrations).',
- ]);
- }
if (!(getIsDry() || getShouldIgnoreRecord())) {
await db.codeMigration.upsert({
where: { name: file },
@@ -118,24 +95,12 @@ export async function runMigration(file: string, env: CodeMigrationEnv) {
}
/**
- * Unfinished migrations in the order the api runs them: blocking ones first,
- * so a fast breaking change never waits behind a long online data move.
+ * Unfinished migrations in file order. Never reordered: a migration may
+ * depend on everything numbered before it, data moves included.
*/
export async function getPendingMigrations() {
const finished = await getFinishedMigrationNames();
- const blocking: string[] = [];
- const online: string[] = [];
- for (const file of listMigrationFiles()) {
- if (finished.has(file)) {
- continue;
- }
- if ((await loadMigration(file)).online) {
- online.push(file);
- } else {
- blocking.push(file);
- }
- }
- return [...blocking, ...online];
+ return listMigrationFiles().filter((file) => !finished.has(file));
}
async function runMigrationOrExit(file: string, env: CodeMigrationEnv) {
@@ -152,35 +117,23 @@ async function runMigrationOrExit(file: string, env: CodeMigrationEnv) {
}
/**
- * The command-line runner (`migrate:deploy`, for development, CI and
- * operators): every unfinished blocking migration, in order, so a fresh
- * database has its schema before anything seeds it. Online ones are left to
- * the api. A migration named on the command line runs either way.
+ * The command-line runner (`migrate:deploy`, and operators who do not set
+ * AUTOMATIC_MIGRATIONS): every pending migration, in order. The api may keep
+ * serving meanwhile. A migration named on the command line runs on its own.
*/
export async function runCodeMigrations(env: CodeMigrationEnv) {
const args = process.argv.slice(2);
const named = args.filter((arg) => !arg.startsWith('--'))[0];
const finished = await getFinishedMigrationNames();
- const unfinished = listMigrationFiles().filter((file) => !finished.has(file));
- const online = new Set<string>();
- for (const file of unfinished) {
- if ((await loadMigration(file)).online) {
- online.add(file);
- }
- }
+ const pending = await getPendingMigrations();
printBoxMessage('📋 Plan', [
'\t✅ Finished:',
...[...finished].map((name) => `\t- ${name}`),
'',
'\t🔄 Will run now:',
- ...unfinished
- .filter((file) => !online.has(file))
- .map((file) => `\t- ${file}`),
- '',
- '\t🌙 Online, applied by the api once it is up:',
- ...[...online].map((file) => `\t- ${file}`),
+ ...(named ? [named] : pending).map((file) => `\t- ${file}`),
]);
printBoxMessage('🤝 Config', [
@@ -205,10 +158,8 @@ export async function runCodeMigrations(env: CodeMigrationEnv) {
if (named) {
await runMigrationOrExit(named, env);
} else {
- for (const file of unfinished) {
- if (!online.has(file)) {
- await runMigrationOrExit(file, env);
- }
+ for (const file of pending) {
+ await runMigrationOrExit(file, env);
}
}