Skip to content

OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.

ralph(M16-004): Prove a FRESH install migrates cleanly: 154 Prisma migrations and 27 ClickHouse code-migrations against brand-new databases, and settle the unconfirmed 3-init-ch failure

ralpha5d6f4fabab3
diff --git a/docs/MIGRATIONS_FRESH_INSTALL.md b/docs/MIGRATIONS_FRESH_INSTALL.md
new file mode 100644
index 000000000..49bb5e92c
--- /dev/null
+++ b/docs/MIGRATIONS_FRESH_INSTALL.md
@@ -0,0 +1,326 @@
+# A fresh install's migrations, measured
+
+**Task M16-004. All numbers below come from runs executed on this box on
+2026-09-12 between 19:50 and 20:00 UTC**, against the local Postgres (5432) and
+ClickHouse (8123), on scratch databases that were dropped afterwards.
+
+Carl's scope (2026-09-12): *"Only thing I want here is our existing migrations
+will work in a similar fashion"*, narrowed to *"I don't feel worried about 2.
+Just try number 1 on new databases is good enough here."* So this is
+**fresh databases only** — upgrade-in-place is deliberately not exercised.
+
+## Verdict, first
+
+| Question | Answer |
+|---|---|
+| Does a brand-new install migrate cleanly, end to end? | **Yes.** Every Prisma migration and every code-migration applied, first try, no errors. |
+| Is the sequence idempotent on a second boot? | **Yes** — and also when the ledger is empty but the tables already exist (extra test below). |
+| Is the `3-init-ch` failure report real? | **The failure is real and reproduces byte for byte — but it is an artifact of the harness M15-203 used, not of a fresh install.** Neither the original explanation nor the operator's hypothesis is the cause. See [`3-init-ch`](#deliverable-2--the-3-init-ch-verdict). |
+| Anything genuinely broken found? | One real hazard, **not** on the first boot: if the *first* boot is interrupted part-way through `3-init-ch`, the *next* boot takes the self-hosting upgrade path on what is really a fresh install. Reported, not fixed — see [Why nothing was fixed](#why-nothing-was-fixed). |
+
+## The shipped command, verified not assumed
+
+`self-hosting/docker-compose.template.yml:152`, `self-hosting/coolify.yml:132`
+and `.github/smoke/docker-compose.yml:85` all run the same two lines in the api
+container's start command, before `bun run start`:
+
+```sh
+cd /app/packages/db && bunx [email protected] migrate deploy
+cd /app/packages/core && bun scripts/migrate-code.ts
+```
+
+`packages/core/scripts/migrate-code.ts` is a one-line forwarder to
+`@openpanel/db/scripts/migrate-code`, which reads the environment and calls
+`runCodeMigrations` (`packages/db/src/code-migrations/migrate.ts`). Both were
+run exactly as written.
+
+## The surface is 153 + 24, not 154 + 27
+
+The task text says 154 Prisma migrations and 27 code-migrations. Measured:
+
+```console
+$ find packages/db/prisma/migrations -mindepth 1 -maxdepth 1 -type d | wc -l
+153
+$ ls packages/db/src/code-migrations/*.ts | grep -E "/[0-9]+-" | wc -l
+24
+```
+
+`ls packages/db/prisma/migrations | wc -l` returns 154 because it counts
+`migration_lock.toml`, which is not a migration. Prisma agrees: *"153 migrations
+found in prisma/migrations"*. The 24 code-migrations are files `1`…`22` with two
+`20-` and two `21-` files. The runner's own plan box lists exactly those 24.
+
+## Reproducing it safely
+
+**The local ClickHouse holds the prod-copy dataset every golden test replays
+against. Read this section before running anything.**
+
+Pointing `CLICKHOUSE_URL` at a scratch database is **not** isolation, for two
+reasons that are easy to miss:
+
+1. `getExistingTables()` (`packages/db/src/clickhouse/migration.ts:174-187`)
+ asks `SELECT name FROM system.tables WHERE database = 'openpanel'` — the
+ database name is a **literal**, not the one in `CLICKHOUSE_URL`. A run
+ pointed at a scratch database still reads the real `openpanel`.
+2. `4-add-sessions.ts:111-114` writes `INSERT INTO openpanel.sessions ... FROM
+ openpanel.events` — fully qualified. A run pointed at a scratch database
+ would still **write** to the real `openpanel`.
+
+So the drill puts a proxy between the migrations and ClickHouse
+(`docs/migrations-fresh-install/ch-scratch-proxy.ts`) which rewrites every
+`openpanel` token — in the URL and in the SQL — to the scratch database name,
+and refuses outright any request that still names `openpanel` or
+`openpanel_test`. That is also what makes the run *faithful*: the migration's
+own hardcoded database name resolves to the empty scratch database, which is
+what it would find on a real fresh install.
+
+```bash
+# run 1 + run 2, on scratch databases, with the proxy
+bash docs/migrations-fresh-install/fresh-install-drill.sh rewrite
+
+# M15-203's setup: only CLICKHOUSE_URL points at a scratch database
+bash docs/migrations-fresh-install/fresh-install-drill.sh guard
+
+# drops ONLY the scratch databases (refuses openpanel / openpanel_test / postgres)
+bash docs/migrations-fresh-install/fresh-install-teardown.sh
+```
+
+The drill creates `openpanel_m16_004` in both Postgres and ClickHouse
+(`SCRATCH_DB` overrides the name), prints the protected databases' table counts
+before and after, and dies if something is already listening on the proxy port —
+a stale proxy would silently serve the run against the wrong database.
+
+The migrations write a `<migration>.sql` dump next to each migration file
+(`writeSqlDump`, `helpers.ts:79-92`). Those files are **untracked and not
+gitignored**, so `fresh-install-teardown.sh` removes them
+(`rm -f packages/db/src/code-migrations/*.sql`) along with the databases. They are also how M15-203's report came to cite a `3-init-ch.sql` that
+does not exist in the repo: it is generated by the failing run itself.
+
+## Deliverable 1 — what actually happened
+
+`bash docs/migrations-fresh-install/fresh-install-drill.sh rewrite`, 2026-09-12
+19:54:44 UTC, whole drill **7.8 s wall clock** (`time`):
+
+| Step | Result | Time |
+|---|---|---|
+| RUN 1 `bunx [email protected] migrate deploy` | `153 migrations found` → **"All migrations have been successfully applied."** | 3 s |
+| RUN 1 `bun scripts/migrate-code.ts` | all **24** ran, **"Migrations finished"**, exit 0 | 2 s |
+| RUN 2 `bunx [email protected] migrate deploy` | **"No pending migrations to apply."** | 2 s |
+| RUN 2 `bun scripts/migrate-code.ts` | 24 × **"✅ Already Migrated ✅"**, **"Migrations finished"** | 0 s |
+
+**There was no first failure. All 177 applied cleanly** (153 Prisma + 24
+code-migrations).
+
+State after each run, read from the scratch databases themselves:
+
+```
+_prisma_migrations finished: 153
+__code_migrations rows: 24
+clickhouse tables: 33 (25 tables/MVs + 8 .inner_id MV storage tables)
+```
+
+identical after run 2.
+
+The config the run reported: `isClustered: false`, `isSelfHosting: true`
+(`SELF_HOSTED=true`, which is what `self-hosting/.env.template:2`,
+`self-hosting/coolify.yml` and `.github/smoke/docker-compose.yml` all set — so
+it is the path every self-hoster takes).
+
+### The fresh schema against the live one
+
+```console
+$ diff <live openpanel tables> <fresh scratch tables>
+17a18,19
+> profile_event_property_summary_mv
+> profile_event_summary_mv
+```
+
+A fresh install produces **exactly the live prod-copy table set plus two MVs**.
+That is expected, not drift: migration 20's header says the pre-20 summary MVs
+"are left in place and keep receiving inserts. Once the new ones are verified,
+dropping them is a one-line follow-up migration" — so a fresh install carries
+both generations, and the prod copy simply does not have the older pair.
+
+The dated tables (`events_20251123`, `sessions_20251123`, `profiles_20260504`)
+are created by a *fresh* run too — they are the rename-swap halves of
+`8-order-keys` and `16-restructure-profiles`, not leftovers.
+
+## Deliverable 2 — the `3-init-ch` verdict
+
+**The failure reproduces exactly, and it is a property of the harness, not of a
+fresh install.**
+
+Reproduced with `fresh-install-drill.sh guard` (2026-09-12 19:51 UTC), which is
+M15-203's setup — `CLICKHOUSE_URL` pointed at a scratch database and nothing
+else:
+
+```
+Failed on query RENAME TABLE cohort_events_mv TO cohort_events_mv_tmp
+error: Table `openpanel_m16_004`.`cohort_events_mv` doesn't exist.
+ type: "UNKNOWN_TABLE",
+ code: "60"
+```
+
+Character for character M15-203's error, with only the scratch database's name
+different (theirs said `openpanel_m15_203`). The proxy log shows why, in one
+pair of lines — the request is routed at the scratch database while the SQL asks
+about `openpanel`:
+
+```
+[POST] /?query_id=...&database=openpanel_m16_004
+SELECT name FROM system.tables WHERE database = 'openpanel'
+FORMAT JSONEachRow
+[POST] /?query_id=...&database=openpanel_m16_004
+RENAME TABLE cohort_events_mv TO cohort_events_mv_tmp
+```
+
+`getExistingTables()` answered with the **prod copy's 23 tables** (the local
+`openpanel`, alphabetically first `cohort_events_mv`), so
+`isSelfHostingOld = existingTables.length !== 0 && isSelfHosting` was true, and
+the rename block emitted a rename for every one of them — against a database
+where none of them exist.
+
+Three explanations, judged against that evidence:
+
+1. **M15-203's stated mechanism — "the migration opens with `RENAME TABLE
+ cohort_events_mv TO cohort_events_mv_tmp`, so it cannot run against an empty
+ ClickHouse": WRONG.** In the fresh run the same migration's first statement
+ is `CREATE DATABASE IF NOT EXISTS openpanel` and it emits **zero** renames
+ (`grep -c RENAME` over the run's ClickHouse traffic: the only 6 renames in
+ the whole 24-migration sequence are the deliberate table swaps in
+ `8-order-keys` and `16-restructure-profiles`). The generated
+ `3-init-ch.sql`'s first line after a fresh run is `CREATE DATABASE IF NOT
+ EXISTS openpanel;`.
+2. **The operator's hypothesis — "migrations 1 and 2 run first and CREATE
+ tables, so by migration 3 `existingTables.length !== 0` is true": WRONG, and
+ disprovable by reading the two files.** `1-settings.ts` is three lines and a
+ no-op (`// Deprecate migration`); `2-accounts.ts` only touches Postgres, and
+ on a fresh install it returns immediately (`No users found, skipping
+ migration`). Neither creates a ClickHouse table. In the fresh run, migration
+ 3 saw an empty `existingTables` and took no rename path.
+3. **What actually happened: the hardcoded database name in
+ `getExistingTables()`.** On a real fresh install the target database *is*
+ `openpanel`, so the literal matches and the answer is empty. On a box where
+ `openpanel` already holds data and the run is aimed elsewhere — which is
+ exactly how you would try to test a fresh install without endangering the
+ prod copy — the answer comes from the wrong database, and the upgrade path
+ fires. The report was an artifact of the only safe-looking way to run the
+ test.
+
+**So: the `3-init-ch` box is not a fresh-install bug, and M15-203's conclusion
+("a fresh self-host install is left partially migrated") does not follow from
+it.** A fresh install with `SELF_HOSTED=true` completes all 24 migrations.
+
+### But there is a real hazard next door: the retry
+
+`3-init-ch` is not safely retryable on a self-hosted install. The ledger row in
+`__code_migrations` is written **after** the migration finishes
+(`migrate.ts:96-108`), so if the first boot dies part-way through it — disk
+full, ClickHouse restart, container killed — the tables it already created stay
+and the ledger row does not. On the next boot `existingTables.length !== 0` is
+true, `isSelfHosting` is true, and the upgrade path runs on what is really a
+fresh install. The shipped start command makes this easy to reach: it is one
+`sh -c` without `set -e`, so a failed migration still lets `bun run start`
+execute, and the operator sees a booting api.
+
+Demonstrated directly (2026-09-12 19:57 UTC), on fresh scratch databases:
+
+```console
+# boot 1, interrupted: migration 3 runs, its ledger row is never written
+$ bun scripts/migrate-code.ts 3-init-ch.ts --no-record
+Migrations finished # 13 ClickHouse tables created, 0 ledger rows
+
+# boot 2: the retry
+$ bun scripts/migrate-code.ts
+Migrations finished # exit 0 — and:
+9 _tmp tables now exist (every table migration 3 created, renamed aside)
+0 INSERT statements ran (migration 3's two data-move branches need a V1
+ marker table; the renamed tables were empty anyway)
+```
+
+and the operator is told to delete them:
+
+```
+│ ⚠️ Please run the following command to clean up unused tables:
+│ docker compose exec -it op-ch clickhouse-client --query "DROP TABLE IF EXISTS openpanel.cohort_events_mv_tmp"
+│ ...
+```
+
+On a fresh install those tables are empty, so nothing is lost — the cost is 9
+orphaned tables and a frightening message. The same path on a populated
+database is what makes it worth recording: with the ledger cleared and the
+scratch database fully migrated, a re-run renamed **all 25** tables aside,
+created 25 empty ones, moved nothing out of them (migration 3's two
+data-move branches need `events_replicated` or `events_v2`, and neither exists
+on a V2 install), and still exited 0.
+
+## Deliverable 3 — idempotency
+
+The second run of the whole sequence changed nothing and errored nowhere:
+
+```
+========== RUN 2 (idempotency) — prisma migrate deploy ==========
+153 migrations found in prisma/migrations
+No pending migrations to apply.
+========== RUN 2 (idempotency) — code migrations ==========
+│ ✅ Already Migrated ✅ (× 24)
+Migrations finished
+========== state after run 2 (ok) ==========
+[{"applied":153}] [{"code_migrations":24}] 33 clickhouse tables
+```
+
+Counts identical to run 1; 0 s of ClickHouse work.
+
+Note **why** it is idempotent: the code-migration half is skipped wholesale by
+the `__code_migrations` ledger, so the second boot executes no ClickHouse SQL at
+all. That is a property of the ledger, not of the SQL. The stronger property was
+tested separately — ledger deleted, all 24 replayed against the already-migrated
+scratch database — and **all 24 succeeded** (exit 0, "Migrations finished"), at
+the cost of the rename shuffle described above. Both properties hold; only the
+first one is what a normal second boot exercises.
+
+## Why nothing was fixed
+
+The fresh-install path works, so there is nothing to fix there. The retry hazard
+is real, but the obvious fix is not safe, and the task's own constraint is that
+a fix which breaks upgrades is worse than the bug:
+
+- The narrow fix would be to fire the rename only when a **V1 marker** table
+ exists — `events_v2`, `events_replicated` or one of the `*_distributed` names
+ the migration already probes — rather than on any table at all. That works for
+ a non-clustered install.
+- It does **not** work clustered. With `isClustered`, `createTable` creates
+ `events_replicated` *itself* (`migration.ts:103-115`), so a fresh clustered
+ install that retries would match the marker anyway — and would additionally
+ match `isSelfHostingPostCluster`, which tries to move data out of
+ `events_replicated_tmp`.
+- A very old self-host with plain `events` (no `_v2`, no `_replicated`) matches
+ no marker, so the marker fix would silently change its behaviour from
+ "rename aside, create new" to "keep the old table, schema and all".
+- Making the retry safe properly needs a persistent "V2 init already started
+ here" marker, which is a new mechanism and a new place to put it — a decision,
+ not a patch.
+
+`getExistingTables()` lives in `packages/db/src/clickhouse/migration.ts`, which
+is outside this task's scope (`packages/db/src/code-migrations/**`, `docs/**`)
+in any case.
+
+## Still unknown
+
+- **Non-default ClickHouse database names are not supported and nothing says
+ so.** `getExistingTables()` and `4-add-sessions.ts` hardcode `openpanel`, and
+ `3-init-ch.ts` hardcodes `CREATE DATABASE IF NOT EXISTS openpanel`. A
+ self-hoster who points `CLICKHOUSE_URL` at a differently-named database gets a
+ half-populated `openpanel` alongside it. Pre-existing V1 behaviour; not
+ measured beyond noticing it.
+- **Clustered fresh install.** Everything here ran `isClustered: false`. The
+ clustered path creates different tables and has the sharper version of the
+ retry hazard; it needs a real cluster to test.
+- **This was not run inside the api image.** The commands are the shipped ones
+ and were run from the same repo layout the image ships, but `bunx
+ [email protected]` fetching the CLI at container start (and the registry reach
+ that needs) is covered by `.github/smoke`, not by this drill.
+- **The `.sql` dumps are untracked and not gitignored.** Adding
+ `packages/db/src/code-migrations/.gitignore` would stop a future run dirtying
+ the tree and stop the next reader citing a generated file as source, as
+ M15-203 did. Not done here: it is not this task's change to make.
diff --git a/docs/TECH_DEBT.md b/docs/TECH_DEBT.md
index 7eb9606d3..31fcf17fc 100644
--- a/docs/TECH_DEBT.md
+++ b/docs/TECH_DEBT.md
@@ -1766,24 +1766,46 @@ Two corrections to the paragraph above, both measured 2026-09-11:
The compose files run `cd /app/packages/core && bun scripts/migrate-code.ts`
instead, which is the shell that reads the environment for it.
-### `3-init-ch.ts` cannot run against an empty ClickHouse database
-
-Found while proving M15-203's start command; **outside that task's scope, left
-unfixed**. `packages/core/src/code-migrations/3-init-ch.ts` (and its paired
-`3-init-ch.sql:1`) opens with
-
-```
-RENAME TABLE cohort_events_mv TO cohort_events_mv_tmp
-```
-
-so on a database where that materialized view does not exist yet the migration
-dies with `Table <db>.cohort_events_mv doesn't exist (UNKNOWN_TABLE, code 60)`.
-Measured 2026-09-11 twice, against two freshly created ClickHouse databases —
-once inside the built api image, once on the host with
-`bun packages/core/scripts/migrate-code.ts` — with identical output, so it is
-neither an image nor a container property. Migrations 1 and 2 apply first, so a
-fresh self-host install is left partially migrated rather than untouched. The
-api still boots: the shipped start command is one `sh -c` without `set -e`.
+### ~~`3-init-ch.ts` cannot run against an empty ClickHouse database~~ — WRONG, corrected by M16-004 (2026-09-12)
+
+M15-203 reported, while proving its start command, that
+`3-init-ch.ts` "opens with `RENAME TABLE cohort_events_mv TO
+cohort_events_mv_tmp`" and therefore "cannot run against an empty ClickHouse
+database", dying with `UNKNOWN_TABLE (code 60)`, leaving a fresh self-host
+install partially migrated. **The failure it saw is real and reproduces exactly.
+The explanation is not.** M16-004 ran the whole sequence on fresh scratch
+databases — full evidence in `docs/MIGRATIONS_FRESH_INSTALL.md`:
+
+- **A brand-new install migrates cleanly.** 153 Prisma migrations then all 24
+ code-migrations, `SELF_HOSTED=true`, first try, no errors (2026-09-12). On a
+ fresh database `3-init-ch.ts` emits **no rename at all** — its first statement
+ is `CREATE DATABASE IF NOT EXISTS openpanel` — because the rename block sits
+ behind `isSelfHostingOld = existingTables.length !== 0 && isSelfHosting`.
+- **The `3-init-ch.sql:1` the report cites is generated by the failing run
+ itself** (`writeSqlDump`), not a repo file. It is untracked and not
+ gitignored.
+- **Migrations 1 and 2 are not the cause either** (the operator's 2026-09-12
+ hypothesis): `1-settings.ts` is a no-op and `2-accounts.ts` only touches
+ Postgres, so neither makes `existingTables` non-empty.
+- **The real cause of the observed failure is the test harness.**
+ `getExistingTables()` asks `... WHERE database = 'openpanel'` with the name as
+ a literal, so a run aimed at a scratch database still reads the real
+ `openpanel`. On the M15-203 box that answered with the prod copy's tables,
+ `isSelfHostingOld` went true, and the renames were emitted against a database
+ where those tables do not exist. Reproduced byte for byte in `guard` mode.
+- **What IS real, and is not a first-boot bug:** `3-init-ch.ts` is not safely
+ retryable when `SELF_HOSTED=true`. Its ledger row is written only after it
+ finishes, so an interrupted first boot leaves its tables behind without the
+ row, and the next boot takes the self-hosting upgrade path — renaming every
+ table aside and creating empty ones. Demonstrated 2026-09-12; empty tables on
+ a fresh install, so no data loss there, but 9 orphaned `_tmp` tables and a
+ "please drop these" message. **Left unfixed deliberately**: the marker-based
+ fix breaks the clustered case (`createTable` creates `events_replicated`
+ itself) and changes behaviour for a pre-`events_v2` self-host, and Carl scoped
+ upgrade-in-place out of testing — a fix that makes fresh installs work by
+ breaking upgrades is worse than the bug.
+- Still true from the original note: the shipped start command is one `sh -c`
+ without `set -e`, so a failed migration still lets the api boot.
### Also recorded
diff --git a/docs/migrations-fresh-install/ch-scratch-proxy.ts b/docs/migrations-fresh-install/ch-scratch-proxy.ts
new file mode 100644
index 000000000..a39c9632d
--- /dev/null
+++ b/docs/migrations-fresh-install/ch-scratch-proxy.ts
@@ -0,0 +1,146 @@
+/**
+ * A ClickHouse HTTP proxy that lets a fresh-install migration drill run on a
+ * box whose `openpanel` database already holds the prod-copy dataset.
+ *
+ * Why it has to exist: `getExistingTables()`
+ * (`packages/db/src/clickhouse/migration.ts`) asks
+ * `SELECT name FROM system.tables WHERE database = 'openpanel'` — the database
+ * name is a literal, not the one in `CLICKHOUSE_URL` — and `4-add-sessions.ts`
+ * writes `INSERT INTO openpanel.sessions ... FROM openpanel.events`. Pointing
+ * `CLICKHOUSE_URL` at a scratch database is therefore NOT isolation: the run
+ * still reads, and would still write, the real `openpanel`.
+ *
+ * Modes:
+ * rewrite every `openpanel` token (URL and body) becomes SCRATCH_DATABASE,
+ * so the migration's own hardcoded name resolves to the scratch
+ * database. This is what a genuinely fresh install looks like.
+ * guard pass through untouched — used to reproduce a run that points only
+ * CLICKHOUSE_URL at a scratch database — but refuse any statement
+ * that is not a read and still names `openpanel` / `openpanel_test`.
+ *
+ * Both modes refuse anything naming `openpanel_test`, and every statement is
+ * appended to LOG_FILE.
+ */
+
+const UPSTREAM_URL = process.env.UPSTREAM_CLICKHOUSE_URL ?? 'http://127.0.0.1:8123';
+const LISTEN_PORT = Number(process.env.PROXY_PORT ?? 8199);
+const SCRATCH_DATABASE = process.env.SCRATCH_DATABASE ?? '';
+const MODE = process.env.PROXY_MODE ?? 'rewrite';
+const LOG_FILE = process.env.PROXY_LOG ?? '/tmp/ch-scratch-proxy.log';
+
+// Two spellings of the same token: `.test()` on a /g regex carries lastIndex
+// between calls, so the guard gets its own non-global copy.
+const PROTECTED_DATABASE_TOKEN_ALL = /\bopenpanel\b/g;
+const PROTECTED_DATABASE_TOKEN = /\bopenpanel\b/;
+const TEST_DATABASE_TOKEN = /\bopenpanel_test\b/;
+const READ_STATEMENT = /^\s*(SELECT|WITH|SHOW|DESC|DESCRIBE|EXISTS|EXPLAIN)\b/i;
+const REFUSED_STATUS = 400;
+
+if (MODE === 'rewrite' && !SCRATCH_DATABASE) {
+ throw new Error('SCRATCH_DATABASE is required in rewrite mode');
+}
+if (SCRATCH_DATABASE === 'openpanel' || SCRATCH_DATABASE === 'openpanel_test') {
+ throw new Error(`refusing to use ${SCRATCH_DATABASE} as a scratch database`);
+}
+
+await Bun.write(LOG_FILE, '');
+const logHandle = Bun.file(LOG_FILE).writer();
+const append = (line: string) => {
+ logHandle.write(`${line}\n`);
+ logHandle.flush();
+};
+
+/** ClickHouse clients may gzip the request body; rewriting needs the plain text. */
+async function readBody(request: Request): Promise<string> {
+ const raw = new Uint8Array(await request.arrayBuffer());
+ if (request.headers.get('content-encoding') === 'gzip') {
+ return new TextDecoder().decode(Bun.gunzipSync(raw));
+ }
+ return new TextDecoder().decode(raw);
+}
+
+const rewrite = (text: string) =>
+ MODE === 'rewrite'
+ ? text.replace(PROTECTED_DATABASE_TOKEN_ALL, SCRATCH_DATABASE)
+ : text;
+
+/**
+ * `statement` is the SQL alone — the read/write classification has to look at
+ * the query text, not at the URL it arrived on.
+ */
+const isRefused = (target: string, statement: string) => {
+ if (TEST_DATABASE_TOKEN.test(target) || TEST_DATABASE_TOKEN.test(statement)) {
+ return 'names openpanel_test';
+ }
+ if (MODE === 'rewrite') {
+ // Nothing may reach ClickHouse still naming the real database.
+ return PROTECTED_DATABASE_TOKEN.test(target) ||
+ PROTECTED_DATABASE_TOKEN.test(statement)
+ ? 'still names openpanel'
+ : null;
+ }
+ if (PROTECTED_DATABASE_TOKEN.test(statement) && !READ_STATEMENT.test(statement)) {
+ return 'non-read statement naming openpanel';
+ }
+ if (PROTECTED_DATABASE_TOKEN.test(target)) {
+ return 'request routed at the openpanel database';
+ }
+ return null;
+};
+
+const server = Bun.serve({
+ port: LISTEN_PORT,
+ hostname: '127.0.0.1',
+ idleTimeout: 255,
+ async fetch(request) {
+ const incoming = new URL(request.url);
+ const body = request.method === 'GET' ? '' : await readBody(request);
+
+ const outgoing = new URL(
+ rewrite(incoming.pathname + incoming.search),
+ UPSTREAM_URL
+ );
+ const outgoingBody = rewrite(body);
+ const target = `${outgoing.pathname}?${outgoing.searchParams.get('database') ?? ''}`;
+ const statement = outgoingBody || (outgoing.searchParams.get('query') ?? '');
+ const inspected = `${outgoing.pathname}${outgoing.search}\n${outgoingBody}`;
+
+ const refusal = isRefused(target, statement);
+ if (refusal) {
+ append(`[REFUSED: ${refusal}] ${inspected}`);
+ return new Response(`proxy refused request: ${refusal}\n`, {
+ status: REFUSED_STATUS,
+ });
+ }
+
+ append(`[${request.method}] ${outgoing.pathname}${outgoing.search}\n${outgoingBody}`);
+
+ const headers = new Headers(request.headers);
+ // The body is rewritten, so any length/encoding the client set is stale.
+ headers.delete('content-length');
+ headers.delete('content-encoding');
+
+ const response = await fetch(outgoing, {
+ method: request.method,
+ headers,
+ body: request.method === 'GET' ? undefined : outgoingBody,
+ });
+
+ // `fetch` already decompressed the upstream body; forwarding ClickHouse's
+ // `Content-Encoding: gzip` would make the client decompress it twice.
+ const responseHeaders = new Headers(response.headers);
+ responseHeaders.delete('content-encoding');
+ responseHeaders.delete('content-length');
+
+ return new Response(response.body, {
+ status: response.status,
+ statusText: response.statusText,
+ headers: responseHeaders,
+ });
+ },
+});
+
+append(`--- proxy up mode=${MODE} scratch=${SCRATCH_DATABASE || '(none)'} ---`);
+console.log(
+ `ch-scratch-proxy listening on http://127.0.0.1:${server.port} mode=${MODE} scratch=${SCRATCH_DATABASE || '(none)'}`
+);
diff --git a/docs/migrations-fresh-install/fresh-install-drill.sh b/docs/migrations-fresh-install/fresh-install-drill.sh
new file mode 100755
index 000000000..65b8e5111
--- /dev/null
+++ b/docs/migrations-fresh-install/fresh-install-drill.sh
@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# Fresh-install migration drill (M16-004).
+#
+# Runs the shipped container start command's migration half — 153 Prisma
+# migrations then the 24 code-migrations — against scratch databases, twice,
+# and prints what each step did.
+#
+# bash docs/migrations-fresh-install/fresh-install-drill.sh rewrite
+# bash docs/migrations-fresh-install/fresh-install-drill.sh guard
+#
+# `rewrite` is the real drill: ch-scratch-proxy.ts maps the `openpanel`
+# database name the migrations hardcode onto the scratch database, so the run
+# sees a genuinely empty install. `guard` reproduces M15-203's setup, where
+# only CLICKHOUSE_URL pointed at a scratch database.
+#
+# Teardown is `fresh-install-teardown.sh`. Nothing here touches the `openpanel`
+# or `openpanel_test` databases; the proxy refuses any request that would.
+set -euo pipefail
+
+MODE="${1:-rewrite}"
+SCRATCH_DB="${SCRATCH_DB:-openpanel_m16_004}"
+PROXY_PORT="${PROXY_PORT:-8199}"
+PROXY_LOG="${PROXY_LOG:-/tmp/ch-scratch-proxy-${MODE}.log}"
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+PG_BASE_URL="${PG_BASE_URL:-postgresql://postgres:postgres@localhost:5432}"
+PG_ADMIN_URL="$PG_BASE_URL/postgres"
+CH_ADMIN_URL="${CH_ADMIN_URL:-http://127.0.0.1:8123}"
+
+case "$SCRATCH_DB" in
+ openpanel|openpanel_test|postgres)
+ echo "refusing to use '$SCRATCH_DB' as a scratch database" >&2
+ exit 1
+ ;;
+esac
+
+pg_query() { PG_QUERY_URL="$1" bun -e "
+const sql = new Bun.SQL(process.env.PG_QUERY_URL);
+const rows = await sql.unsafe(process.argv[1]);
+if (Array.isArray(rows) && rows.length) console.log(JSON.stringify(rows));
+await sql.end();
+" "$2"; }
+
+# The maintenance connection (creates/drops databases) and the scratch database
+# itself are different connections — reporting through the wrong one reports the
+# dev database's state.
+pg_admin() { pg_query "$PG_ADMIN_URL" "$1"; }
+pg_scratch() { pg_query "$PG_BASE_URL/$SCRATCH_DB" "$1"; }
+
+ch_admin() { curl -sS --fail-with-body "$CH_ADMIN_URL/" --data-binary "$1"; }
+
+banner() { printf '\n========== %s ==========\n' "$1"; }
+
+banner "0. scratch databases"
+pg_admin "DROP DATABASE IF EXISTS \"$SCRATCH_DB\""
+pg_admin "CREATE DATABASE \"$SCRATCH_DB\""
+ch_admin "DROP DATABASE IF EXISTS $SCRATCH_DB"
+ch_admin "CREATE DATABASE $SCRATCH_DB"
+echo "postgres: $SCRATCH_DB created"
+echo "clickhouse: $SCRATCH_DB created, tables: $(ch_admin "SELECT count() FROM system.tables WHERE database='$SCRATCH_DB'")"
+
+banner "0b. protected databases before the run"
+ch_admin "SELECT database, count() FROM system.tables WHERE database IN ('openpanel','openpanel_test') GROUP BY database ORDER BY database FORMAT TSV"
+
+if curl -sS "http://127.0.0.1:$PROXY_PORT/" --data-binary "SELECT 1" >/dev/null 2>&1; then
+ echo "port $PROXY_PORT already answers — a stale proxy would silently serve this run" >&2
+ exit 1
+fi
+
+if [ "$MODE" = "rewrite" ]; then
+ PROXY_MODE=rewrite SCRATCH_DATABASE="$SCRATCH_DB" PROXY_PORT="$PROXY_PORT" PROXY_LOG="$PROXY_LOG" \
+ bun "$REPO_ROOT/docs/migrations-fresh-install/ch-scratch-proxy.ts" &
+else
+ PROXY_MODE=guard PROXY_PORT="$PROXY_PORT" PROXY_LOG="$PROXY_LOG" \
+ bun "$REPO_ROOT/docs/migrations-fresh-install/ch-scratch-proxy.ts" &
+fi
+PROXY_PID=$!
+trap 'kill $PROXY_PID 2>/dev/null || true' EXIT
+until curl -sS "http://127.0.0.1:$PROXY_PORT/?query=SELECT%201" >/dev/null 2>&1; do sleep 0.2; done
+
+export DATABASE_URL="$PG_BASE_URL/$SCRATCH_DB?schema=public"
+export DATABASE_URL_DIRECT="$DATABASE_URL"
+export CLICKHOUSE_URL="http://127.0.0.1:$PROXY_PORT/$SCRATCH_DB"
+export SELF_HOSTED=true
+
+run_once() {
+ banner "$1 — prisma migrate deploy"
+ local started=$SECONDS
+ (cd "$REPO_ROOT/packages/db" && bunx [email protected] migrate deploy) || return 1
+ echo "[timing] $1 prisma migrate deploy: $((SECONDS - started))s"
+
+ banner "$1 — code migrations"
+ started=$SECONDS
+ (cd "$REPO_ROOT/packages/core" && bun scripts/migrate-code.ts) || return 1
+ echo "[timing] $1 code migrations: $((SECONDS - started))s"
+}
+
+FIRST_RESULT=ok
+run_once "RUN 1" || FIRST_RESULT=failed
+
+banner "state after run 1 ($FIRST_RESULT)"
+pg_scratch "SELECT count(*)::int AS applied FROM \"_prisma_migrations\" WHERE finished_at IS NOT NULL" || true
+pg_scratch "SELECT count(*)::int AS code_migrations FROM \"__code_migrations\"" || true
+ch_admin "SELECT count() FROM system.tables WHERE database='$SCRATCH_DB'"
+
+SECOND_RESULT=ok
+run_once "RUN 2 (idempotency)" || SECOND_RESULT=failed
+
+banner "state after run 2 ($SECOND_RESULT)"
+pg_scratch "SELECT count(*)::int AS applied FROM \"_prisma_migrations\" WHERE finished_at IS NOT NULL" || true
+pg_scratch "SELECT count(*)::int AS code_migrations FROM \"__code_migrations\"" || true
+ch_admin "SELECT count() FROM system.tables WHERE database='$SCRATCH_DB'"
+
+banner "protected databases after the run"
+ch_admin "SELECT database, count() FROM system.tables WHERE database IN ('openpanel','openpanel_test') GROUP BY database ORDER BY database FORMAT TSV"
+
+banner "verdict"
+echo "mode=$MODE run1=$FIRST_RESULT run2=$SECOND_RESULT proxy log: $PROXY_LOG"
diff --git a/docs/migrations-fresh-install/fresh-install-teardown.sh b/docs/migrations-fresh-install/fresh-install-teardown.sh
new file mode 100755
index 000000000..8554588c9
--- /dev/null
+++ b/docs/migrations-fresh-install/fresh-install-teardown.sh
@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+# Drops ONLY the scratch databases the drill created, and removes the `.sql`
+# dumps the migrations write next to themselves (untracked build artifacts).
+set -euo pipefail
+
+SCRATCH_DB="${SCRATCH_DB:-openpanel_m16_004}"
+PG_ADMIN_URL="${PG_ADMIN_URL:-postgresql://postgres:postgres@localhost:5432/postgres}"
+CH_ADMIN_URL="${CH_ADMIN_URL:-http://127.0.0.1:8123}"
+
+case "$SCRATCH_DB" in
+ openpanel|openpanel_test|postgres)
+ echo "refusing to drop '$SCRATCH_DB'" >&2
+ exit 1
+ ;;
+esac
+
+PG_ADMIN_URL="$PG_ADMIN_URL" bun -e "
+const sql = new Bun.SQL(process.env.PG_ADMIN_URL);
+await sql.unsafe(\`DROP DATABASE IF EXISTS \"\${process.argv[1]}\"\`);
+await sql.end();
+" "$SCRATCH_DB"
+curl -sS --fail-with-body "$CH_ADMIN_URL/" --data-binary "DROP DATABASE IF EXISTS $SCRATCH_DB"
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+rm -f "$REPO_ROOT"/packages/db/src/code-migrations/*.sql
+
+echo "dropped scratch databases named $SCRATCH_DB and removed the .sql dumps"