OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.
ralph(M16-004): Prove a FRESH install migrates cleanly: 154 Prisma migrations and 27 ClickHouse code-migrations against brand-new databases, and settle the unconfirmed 3-init-ch failure
ralpha5d6f4fabab3
diff --git a/docs/MIGRATIONS_FRESH_INSTALL.md b/docs/MIGRATIONS_FRESH_INSTALL.mdnew file mode 100644index 000000000..49bb5e92c--- /dev/null+++ b/docs/MIGRATIONS_FRESH_INSTALL.md@@ -0,0 +1,326 @@+# A fresh install's migrations, measured++**Task M16-004. All numbers below come from runs executed on this box on+2026-09-12 between 19:50 and 20:00 UTC**, against the local Postgres (5432) and+ClickHouse (8123), on scratch databases that were dropped afterwards.++Carl's scope (2026-09-12): *"Only thing I want here is our existing migrations+will work in a similar fashion"*, narrowed to *"I don't feel worried about 2.+Just try number 1 on new databases is good enough here."* So this is+**fresh databases only** — upgrade-in-place is deliberately not exercised.++## Verdict, first++| Question | Answer |+|---|---|+| Does a brand-new install migrate cleanly, end to end? | **Yes.** Every Prisma migration and every code-migration applied, first try, no errors. |+| Is the sequence idempotent on a second boot? | **Yes** — and also when the ledger is empty but the tables already exist (extra test below). |+| Is the `3-init-ch` failure report real? | **The failure is real and reproduces byte for byte — but it is an artifact of the harness M15-203 used, not of a fresh install.** Neither the original explanation nor the operator's hypothesis is the cause. See [`3-init-ch`](#deliverable-2--the-3-init-ch-verdict). |+| Anything genuinely broken found? | One real hazard, **not** on the first boot: if the *first* boot is interrupted part-way through `3-init-ch`, the *next* boot takes the self-hosting upgrade path on what is really a fresh install. Reported, not fixed — see [Why nothing was fixed](#why-nothing-was-fixed). |++## The shipped command, verified not assumed++`self-hosting/docker-compose.template.yml:152`, `self-hosting/coolify.yml:132`+and `.github/smoke/docker-compose.yml:85` all run the same two lines in the api+container's start command, before `bun run start`:++```sh+cd /app/packages/db && bunx [email protected] migrate deploy+cd /app/packages/core && bun scripts/migrate-code.ts+```++`packages/core/scripts/migrate-code.ts` is a one-line forwarder to+`@openpanel/db/scripts/migrate-code`, which reads the environment and calls+`runCodeMigrations` (`packages/db/src/code-migrations/migrate.ts`). Both were+run exactly as written.++## The surface is 153 + 24, not 154 + 27++The task text says 154 Prisma migrations and 27 code-migrations. Measured:++```console+$ find packages/db/prisma/migrations -mindepth 1 -maxdepth 1 -type d | wc -l+153+$ ls packages/db/src/code-migrations/*.ts | grep -E "/[0-9]+-" | wc -l+24+```++`ls packages/db/prisma/migrations | wc -l` returns 154 because it counts+`migration_lock.toml`, which is not a migration. Prisma agrees: *"153 migrations+found in prisma/migrations"*. The 24 code-migrations are files `1`…`22` with two+`20-` and two `21-` files. The runner's own plan box lists exactly those 24.++## Reproducing it safely++**The local ClickHouse holds the prod-copy dataset every golden test replays+against. Read this section before running anything.**++Pointing `CLICKHOUSE_URL` at a scratch database is **not** isolation, for two+reasons that are easy to miss:++1. `getExistingTables()` (`packages/db/src/clickhouse/migration.ts:174-187`)+ asks `SELECT name FROM system.tables WHERE database = 'openpanel'` — the+ database name is a **literal**, not the one in `CLICKHOUSE_URL`. A run+ pointed at a scratch database still reads the real `openpanel`.+2. `4-add-sessions.ts:111-114` writes `INSERT INTO openpanel.sessions ... FROM+ openpanel.events` — fully qualified. A run pointed at a scratch database+ would still **write** to the real `openpanel`.++So the drill puts a proxy between the migrations and ClickHouse+(`docs/migrations-fresh-install/ch-scratch-proxy.ts`) which rewrites every+`openpanel` token — in the URL and in the SQL — to the scratch database name,+and refuses outright any request that still names `openpanel` or+`openpanel_test`. That is also what makes the run *faithful*: the migration's+own hardcoded database name resolves to the empty scratch database, which is+what it would find on a real fresh install.++```bash+# run 1 + run 2, on scratch databases, with the proxy+bash docs/migrations-fresh-install/fresh-install-drill.sh rewrite++# M15-203's setup: only CLICKHOUSE_URL points at a scratch database+bash docs/migrations-fresh-install/fresh-install-drill.sh guard++# drops ONLY the scratch databases (refuses openpanel / openpanel_test / postgres)+bash docs/migrations-fresh-install/fresh-install-teardown.sh+```++The drill creates `openpanel_m16_004` in both Postgres and ClickHouse+(`SCRATCH_DB` overrides the name), prints the protected databases' table counts+before and after, and dies if something is already listening on the proxy port —+a stale proxy would silently serve the run against the wrong database.++The migrations write a `<migration>.sql` dump next to each migration file+(`writeSqlDump`, `helpers.ts:79-92`). Those files are **untracked and not+gitignored**, so `fresh-install-teardown.sh` removes them+(`rm -f packages/db/src/code-migrations/*.sql`) along with the databases. They are also how M15-203's report came to cite a `3-init-ch.sql` that+does not exist in the repo: it is generated by the failing run itself.++## Deliverable 1 — what actually happened++`bash docs/migrations-fresh-install/fresh-install-drill.sh rewrite`, 2026-09-12+19:54:44 UTC, whole drill **7.8 s wall clock** (`time`):++| Step | Result | Time |+|---|---|---|+| RUN 1 `bunx [email protected] migrate deploy` | `153 migrations found` → **"All migrations have been successfully applied."** | 3 s |+| RUN 1 `bun scripts/migrate-code.ts` | all **24** ran, **"Migrations finished"**, exit 0 | 2 s |+| RUN 2 `bunx [email protected] migrate deploy` | **"No pending migrations to apply."** | 2 s |+| RUN 2 `bun scripts/migrate-code.ts` | 24 × **"✅ Already Migrated ✅"**, **"Migrations finished"** | 0 s |++**There was no first failure. All 177 applied cleanly** (153 Prisma + 24+code-migrations).++State after each run, read from the scratch databases themselves:++```+_prisma_migrations finished: 153+__code_migrations rows: 24+clickhouse tables: 33 (25 tables/MVs + 8 .inner_id MV storage tables)+```++identical after run 2.++The config the run reported: `isClustered: false`, `isSelfHosting: true`+(`SELF_HOSTED=true`, which is what `self-hosting/.env.template:2`,+`self-hosting/coolify.yml` and `.github/smoke/docker-compose.yml` all set — so+it is the path every self-hoster takes).++### The fresh schema against the live one++```console+$ diff <live openpanel tables> <fresh scratch tables>+17a18,19+> profile_event_property_summary_mv+> profile_event_summary_mv+```++A fresh install produces **exactly the live prod-copy table set plus two MVs**.+That is expected, not drift: migration 20's header says the pre-20 summary MVs+"are left in place and keep receiving inserts. Once the new ones are verified,+dropping them is a one-line follow-up migration" — so a fresh install carries+both generations, and the prod copy simply does not have the older pair.++The dated tables (`events_20251123`, `sessions_20251123`, `profiles_20260504`)+are created by a *fresh* run too — they are the rename-swap halves of+`8-order-keys` and `16-restructure-profiles`, not leftovers.++## Deliverable 2 — the `3-init-ch` verdict++**The failure reproduces exactly, and it is a property of the harness, not of a+fresh install.**++Reproduced with `fresh-install-drill.sh guard` (2026-09-12 19:51 UTC), which is+M15-203's setup — `CLICKHOUSE_URL` pointed at a scratch database and nothing+else:++```+Failed on query RENAME TABLE cohort_events_mv TO cohort_events_mv_tmp+error: Table `openpanel_m16_004`.`cohort_events_mv` doesn't exist.+ type: "UNKNOWN_TABLE",+ code: "60"+```++Character for character M15-203's error, with only the scratch database's name+different (theirs said `openpanel_m15_203`). The proxy log shows why, in one+pair of lines — the request is routed at the scratch database while the SQL asks+about `openpanel`:++```+[POST] /?query_id=...&database=openpanel_m16_004+SELECT name FROM system.tables WHERE database = 'openpanel'+FORMAT JSONEachRow+[POST] /?query_id=...&database=openpanel_m16_004+RENAME TABLE cohort_events_mv TO cohort_events_mv_tmp+```++`getExistingTables()` answered with the **prod copy's 23 tables** (the local+`openpanel`, alphabetically first `cohort_events_mv`), so+`isSelfHostingOld = existingTables.length !== 0 && isSelfHosting` was true, and+the rename block emitted a rename for every one of them — against a database+where none of them exist.++Three explanations, judged against that evidence:++1. **M15-203's stated mechanism — "the migration opens with `RENAME TABLE+ cohort_events_mv TO cohort_events_mv_tmp`, so it cannot run against an empty+ ClickHouse": WRONG.** In the fresh run the same migration's first statement+ is `CREATE DATABASE IF NOT EXISTS openpanel` and it emits **zero** renames+ (`grep -c RENAME` over the run's ClickHouse traffic: the only 6 renames in+ the whole 24-migration sequence are the deliberate table swaps in+ `8-order-keys` and `16-restructure-profiles`). The generated+ `3-init-ch.sql`'s first line after a fresh run is `CREATE DATABASE IF NOT+ EXISTS openpanel;`.+2. **The operator's hypothesis — "migrations 1 and 2 run first and CREATE+ tables, so by migration 3 `existingTables.length !== 0` is true": WRONG, and+ disprovable by reading the two files.** `1-settings.ts` is three lines and a+ no-op (`// Deprecate migration`); `2-accounts.ts` only touches Postgres, and+ on a fresh install it returns immediately (`No users found, skipping+ migration`). Neither creates a ClickHouse table. In the fresh run, migration+ 3 saw an empty `existingTables` and took no rename path.+3. **What actually happened: the hardcoded database name in+ `getExistingTables()`.** On a real fresh install the target database *is*+ `openpanel`, so the literal matches and the answer is empty. On a box where+ `openpanel` already holds data and the run is aimed elsewhere — which is+ exactly how you would try to test a fresh install without endangering the+ prod copy — the answer comes from the wrong database, and the upgrade path+ fires. The report was an artifact of the only safe-looking way to run the+ test.++**So: the `3-init-ch` box is not a fresh-install bug, and M15-203's conclusion+("a fresh self-host install is left partially migrated") does not follow from+it.** A fresh install with `SELF_HOSTED=true` completes all 24 migrations.++### But there is a real hazard next door: the retry++`3-init-ch` is not safely retryable on a self-hosted install. The ledger row in+`__code_migrations` is written **after** the migration finishes+(`migrate.ts:96-108`), so if the first boot dies part-way through it — disk+full, ClickHouse restart, container killed — the tables it already created stay+and the ledger row does not. On the next boot `existingTables.length !== 0` is+true, `isSelfHosting` is true, and the upgrade path runs on what is really a+fresh install. The shipped start command makes this easy to reach: it is one+`sh -c` without `set -e`, so a failed migration still lets `bun run start`+execute, and the operator sees a booting api.++Demonstrated directly (2026-09-12 19:57 UTC), on fresh scratch databases:++```console+# boot 1, interrupted: migration 3 runs, its ledger row is never written+$ bun scripts/migrate-code.ts 3-init-ch.ts --no-record+Migrations finished # 13 ClickHouse tables created, 0 ledger rows++# boot 2: the retry+$ bun scripts/migrate-code.ts+Migrations finished # exit 0 — and:+9 _tmp tables now exist (every table migration 3 created, renamed aside)+0 INSERT statements ran (migration 3's two data-move branches need a V1+ marker table; the renamed tables were empty anyway)+```++and the operator is told to delete them:++```+│ ⚠️ Please run the following command to clean up unused tables:+│ docker compose exec -it op-ch clickhouse-client --query "DROP TABLE IF EXISTS openpanel.cohort_events_mv_tmp"+│ ...+```++On a fresh install those tables are empty, so nothing is lost — the cost is 9+orphaned tables and a frightening message. The same path on a populated+database is what makes it worth recording: with the ledger cleared and the+scratch database fully migrated, a re-run renamed **all 25** tables aside,+created 25 empty ones, moved nothing out of them (migration 3's two+data-move branches need `events_replicated` or `events_v2`, and neither exists+on a V2 install), and still exited 0.++## Deliverable 3 — idempotency++The second run of the whole sequence changed nothing and errored nowhere:++```+========== RUN 2 (idempotency) — prisma migrate deploy ==========+153 migrations found in prisma/migrations+No pending migrations to apply.+========== RUN 2 (idempotency) — code migrations ==========+│ ✅ Already Migrated ✅ (× 24)+Migrations finished+========== state after run 2 (ok) ==========+[{"applied":153}] [{"code_migrations":24}] 33 clickhouse tables+```++Counts identical to run 1; 0 s of ClickHouse work.++Note **why** it is idempotent: the code-migration half is skipped wholesale by+the `__code_migrations` ledger, so the second boot executes no ClickHouse SQL at+all. That is a property of the ledger, not of the SQL. The stronger property was+tested separately — ledger deleted, all 24 replayed against the already-migrated+scratch database — and **all 24 succeeded** (exit 0, "Migrations finished"), at+the cost of the rename shuffle described above. Both properties hold; only the+first one is what a normal second boot exercises.++## Why nothing was fixed++The fresh-install path works, so there is nothing to fix there. The retry hazard+is real, but the obvious fix is not safe, and the task's own constraint is that+a fix which breaks upgrades is worse than the bug:++- The narrow fix would be to fire the rename only when a **V1 marker** table+ exists — `events_v2`, `events_replicated` or one of the `*_distributed` names+ the migration already probes — rather than on any table at all. That works for+ a non-clustered install.+- It does **not** work clustered. With `isClustered`, `createTable` creates+ `events_replicated` *itself* (`migration.ts:103-115`), so a fresh clustered+ install that retries would match the marker anyway — and would additionally+ match `isSelfHostingPostCluster`, which tries to move data out of+ `events_replicated_tmp`.+- A very old self-host with plain `events` (no `_v2`, no `_replicated`) matches+ no marker, so the marker fix would silently change its behaviour from+ "rename aside, create new" to "keep the old table, schema and all".+- Making the retry safe properly needs a persistent "V2 init already started+ here" marker, which is a new mechanism and a new place to put it — a decision,+ not a patch.++`getExistingTables()` lives in `packages/db/src/clickhouse/migration.ts`, which+is outside this task's scope (`packages/db/src/code-migrations/**`, `docs/**`)+in any case.++## Still unknown++- **Non-default ClickHouse database names are not supported and nothing says+ so.** `getExistingTables()` and `4-add-sessions.ts` hardcode `openpanel`, and+ `3-init-ch.ts` hardcodes `CREATE DATABASE IF NOT EXISTS openpanel`. A+ self-hoster who points `CLICKHOUSE_URL` at a differently-named database gets a+ half-populated `openpanel` alongside it. Pre-existing V1 behaviour; not+ measured beyond noticing it.+- **Clustered fresh install.** Everything here ran `isClustered: false`. The+ clustered path creates different tables and has the sharper version of the+ retry hazard; it needs a real cluster to test.+- **This was not run inside the api image.** The commands are the shipped ones+ and were run from the same repo layout the image ships, but `bunx+ [email protected]` fetching the CLI at container start (and the registry reach+ that needs) is covered by `.github/smoke`, not by this drill.+- **The `.sql` dumps are untracked and not gitignored.** Adding+ `packages/db/src/code-migrations/.gitignore` would stop a future run dirtying+ the tree and stop the next reader citing a generated file as source, as+ M15-203 did. Not done here: it is not this task's change to make.diff --git a/docs/TECH_DEBT.md b/docs/TECH_DEBT.mdindex 7eb9606d3..31fcf17fc 100644--- a/docs/TECH_DEBT.md+++ b/docs/TECH_DEBT.md@@ -1766,24 +1766,46 @@ Two corrections to the paragraph above, both measured 2026-09-11:The compose files run `cd /app/packages/core && bun scripts/migrate-code.ts`instead, which is the shell that reads the environment for it.-### `3-init-ch.ts` cannot run against an empty ClickHouse database--Found while proving M15-203's start command; **outside that task's scope, left-unfixed**. `packages/core/src/code-migrations/3-init-ch.ts` (and its paired-`3-init-ch.sql:1`) opens with--```-RENAME TABLE cohort_events_mv TO cohort_events_mv_tmp-```--so on a database where that materialized view does not exist yet the migration-dies with `Table <db>.cohort_events_mv doesn't exist (UNKNOWN_TABLE, code 60)`.-Measured 2026-09-11 twice, against two freshly created ClickHouse databases —-once inside the built api image, once on the host with-`bun packages/core/scripts/migrate-code.ts` — with identical output, so it is-neither an image nor a container property. Migrations 1 and 2 apply first, so a-fresh self-host install is left partially migrated rather than untouched. The-api still boots: the shipped start command is one `sh -c` without `set -e`.+### ~~`3-init-ch.ts` cannot run against an empty ClickHouse database~~ — WRONG, corrected by M16-004 (2026-09-12)++M15-203 reported, while proving its start command, that+`3-init-ch.ts` "opens with `RENAME TABLE cohort_events_mv TO+cohort_events_mv_tmp`" and therefore "cannot run against an empty ClickHouse+database", dying with `UNKNOWN_TABLE (code 60)`, leaving a fresh self-host+install partially migrated. **The failure it saw is real and reproduces exactly.+The explanation is not.** M16-004 ran the whole sequence on fresh scratch+databases — full evidence in `docs/MIGRATIONS_FRESH_INSTALL.md`:++- **A brand-new install migrates cleanly.** 153 Prisma migrations then all 24+ code-migrations, `SELF_HOSTED=true`, first try, no errors (2026-09-12). On a+ fresh database `3-init-ch.ts` emits **no rename at all** — its first statement+ is `CREATE DATABASE IF NOT EXISTS openpanel` — because the rename block sits+ behind `isSelfHostingOld = existingTables.length !== 0 && isSelfHosting`.+- **The `3-init-ch.sql:1` the report cites is generated by the failing run+ itself** (`writeSqlDump`), not a repo file. It is untracked and not+ gitignored.+- **Migrations 1 and 2 are not the cause either** (the operator's 2026-09-12+ hypothesis): `1-settings.ts` is a no-op and `2-accounts.ts` only touches+ Postgres, so neither makes `existingTables` non-empty.+- **The real cause of the observed failure is the test harness.**+ `getExistingTables()` asks `... WHERE database = 'openpanel'` with the name as+ a literal, so a run aimed at a scratch database still reads the real+ `openpanel`. On the M15-203 box that answered with the prod copy's tables,+ `isSelfHostingOld` went true, and the renames were emitted against a database+ where those tables do not exist. Reproduced byte for byte in `guard` mode.+- **What IS real, and is not a first-boot bug:** `3-init-ch.ts` is not safely+ retryable when `SELF_HOSTED=true`. Its ledger row is written only after it+ finishes, so an interrupted first boot leaves its tables behind without the+ row, and the next boot takes the self-hosting upgrade path — renaming every+ table aside and creating empty ones. Demonstrated 2026-09-12; empty tables on+ a fresh install, so no data loss there, but 9 orphaned `_tmp` tables and a+ "please drop these" message. **Left unfixed deliberately**: the marker-based+ fix breaks the clustered case (`createTable` creates `events_replicated`+ itself) and changes behaviour for a pre-`events_v2` self-host, and Carl scoped+ upgrade-in-place out of testing — a fix that makes fresh installs work by+ breaking upgrades is worse than the bug.+- Still true from the original note: the shipped start command is one `sh -c`+ without `set -e`, so a failed migration still lets the api boot.### Also recordeddiff --git a/docs/migrations-fresh-install/ch-scratch-proxy.ts b/docs/migrations-fresh-install/ch-scratch-proxy.tsnew file mode 100644index 000000000..a39c9632d--- /dev/null+++ b/docs/migrations-fresh-install/ch-scratch-proxy.ts@@ -0,0 +1,146 @@+/**+ * A ClickHouse HTTP proxy that lets a fresh-install migration drill run on a+ * box whose `openpanel` database already holds the prod-copy dataset.+ *+ * Why it has to exist: `getExistingTables()`+ * (`packages/db/src/clickhouse/migration.ts`) asks+ * `SELECT name FROM system.tables WHERE database = 'openpanel'` — the database+ * name is a literal, not the one in `CLICKHOUSE_URL` — and `4-add-sessions.ts`+ * writes `INSERT INTO openpanel.sessions ... FROM openpanel.events`. Pointing+ * `CLICKHOUSE_URL` at a scratch database is therefore NOT isolation: the run+ * still reads, and would still write, the real `openpanel`.+ *+ * Modes:+ * rewrite every `openpanel` token (URL and body) becomes SCRATCH_DATABASE,+ * so the migration's own hardcoded name resolves to the scratch+ * database. This is what a genuinely fresh install looks like.+ * guard pass through untouched — used to reproduce a run that points only+ * CLICKHOUSE_URL at a scratch database — but refuse any statement+ * that is not a read and still names `openpanel` / `openpanel_test`.+ *+ * Both modes refuse anything naming `openpanel_test`, and every statement is+ * appended to LOG_FILE.+ */++const UPSTREAM_URL = process.env.UPSTREAM_CLICKHOUSE_URL ?? 'http://127.0.0.1:8123';+const LISTEN_PORT = Number(process.env.PROXY_PORT ?? 8199);+const SCRATCH_DATABASE = process.env.SCRATCH_DATABASE ?? '';+const MODE = process.env.PROXY_MODE ?? 'rewrite';+const LOG_FILE = process.env.PROXY_LOG ?? '/tmp/ch-scratch-proxy.log';++// Two spellings of the same token: `.test()` on a /g regex carries lastIndex+// between calls, so the guard gets its own non-global copy.+const PROTECTED_DATABASE_TOKEN_ALL = /\bopenpanel\b/g;+const PROTECTED_DATABASE_TOKEN = /\bopenpanel\b/;+const TEST_DATABASE_TOKEN = /\bopenpanel_test\b/;+const READ_STATEMENT = /^\s*(SELECT|WITH|SHOW|DESC|DESCRIBE|EXISTS|EXPLAIN)\b/i;+const REFUSED_STATUS = 400;++if (MODE === 'rewrite' && !SCRATCH_DATABASE) {+ throw new Error('SCRATCH_DATABASE is required in rewrite mode');+}+if (SCRATCH_DATABASE === 'openpanel' || SCRATCH_DATABASE === 'openpanel_test') {+ throw new Error(`refusing to use ${SCRATCH_DATABASE} as a scratch database`);+}++await Bun.write(LOG_FILE, '');+const logHandle = Bun.file(LOG_FILE).writer();+const append = (line: string) => {+ logHandle.write(`${line}\n`);+ logHandle.flush();+};++/** ClickHouse clients may gzip the request body; rewriting needs the plain text. */+async function readBody(request: Request): Promise<string> {+ const raw = new Uint8Array(await request.arrayBuffer());+ if (request.headers.get('content-encoding') === 'gzip') {+ return new TextDecoder().decode(Bun.gunzipSync(raw));+ }+ return new TextDecoder().decode(raw);+}++const rewrite = (text: string) =>+ MODE === 'rewrite'+ ? text.replace(PROTECTED_DATABASE_TOKEN_ALL, SCRATCH_DATABASE)+ : text;++/**+ * `statement` is the SQL alone — the read/write classification has to look at+ * the query text, not at the URL it arrived on.+ */+const isRefused = (target: string, statement: string) => {+ if (TEST_DATABASE_TOKEN.test(target) || TEST_DATABASE_TOKEN.test(statement)) {+ return 'names openpanel_test';+ }+ if (MODE === 'rewrite') {+ // Nothing may reach ClickHouse still naming the real database.+ return PROTECTED_DATABASE_TOKEN.test(target) ||+ PROTECTED_DATABASE_TOKEN.test(statement)+ ? 'still names openpanel'+ : null;+ }+ if (PROTECTED_DATABASE_TOKEN.test(statement) && !READ_STATEMENT.test(statement)) {+ return 'non-read statement naming openpanel';+ }+ if (PROTECTED_DATABASE_TOKEN.test(target)) {+ return 'request routed at the openpanel database';+ }+ return null;+};++const server = Bun.serve({+ port: LISTEN_PORT,+ hostname: '127.0.0.1',+ idleTimeout: 255,+ async fetch(request) {+ const incoming = new URL(request.url);+ const body = request.method === 'GET' ? '' : await readBody(request);++ const outgoing = new URL(+ rewrite(incoming.pathname + incoming.search),+ UPSTREAM_URL+ );+ const outgoingBody = rewrite(body);+ const target = `${outgoing.pathname}?${outgoing.searchParams.get('database') ?? ''}`;+ const statement = outgoingBody || (outgoing.searchParams.get('query') ?? '');+ const inspected = `${outgoing.pathname}${outgoing.search}\n${outgoingBody}`;++ const refusal = isRefused(target, statement);+ if (refusal) {+ append(`[REFUSED: ${refusal}] ${inspected}`);+ return new Response(`proxy refused request: ${refusal}\n`, {+ status: REFUSED_STATUS,+ });+ }++ append(`[${request.method}] ${outgoing.pathname}${outgoing.search}\n${outgoingBody}`);++ const headers = new Headers(request.headers);+ // The body is rewritten, so any length/encoding the client set is stale.+ headers.delete('content-length');+ headers.delete('content-encoding');++ const response = await fetch(outgoing, {+ method: request.method,+ headers,+ body: request.method === 'GET' ? undefined : outgoingBody,+ });++ // `fetch` already decompressed the upstream body; forwarding ClickHouse's+ // `Content-Encoding: gzip` would make the client decompress it twice.+ const responseHeaders = new Headers(response.headers);+ responseHeaders.delete('content-encoding');+ responseHeaders.delete('content-length');++ return new Response(response.body, {+ status: response.status,+ statusText: response.statusText,+ headers: responseHeaders,+ });+ },+});++append(`--- proxy up mode=${MODE} scratch=${SCRATCH_DATABASE || '(none)'} ---`);+console.log(+ `ch-scratch-proxy listening on http://127.0.0.1:${server.port} mode=${MODE} scratch=${SCRATCH_DATABASE || '(none)'}`+);diff --git a/docs/migrations-fresh-install/fresh-install-drill.sh b/docs/migrations-fresh-install/fresh-install-drill.shnew file mode 100755index 000000000..65b8e5111--- /dev/null+++ b/docs/migrations-fresh-install/fresh-install-drill.sh@@ -0,0 +1,117 @@+#!/usr/bin/env bash+# Fresh-install migration drill (M16-004).+#+# Runs the shipped container start command's migration half — 153 Prisma+# migrations then the 24 code-migrations — against scratch databases, twice,+# and prints what each step did.+#+# bash docs/migrations-fresh-install/fresh-install-drill.sh rewrite+# bash docs/migrations-fresh-install/fresh-install-drill.sh guard+#+# `rewrite` is the real drill: ch-scratch-proxy.ts maps the `openpanel`+# database name the migrations hardcode onto the scratch database, so the run+# sees a genuinely empty install. `guard` reproduces M15-203's setup, where+# only CLICKHOUSE_URL pointed at a scratch database.+#+# Teardown is `fresh-install-teardown.sh`. Nothing here touches the `openpanel`+# or `openpanel_test` databases; the proxy refuses any request that would.+set -euo pipefail++MODE="${1:-rewrite}"+SCRATCH_DB="${SCRATCH_DB:-openpanel_m16_004}"+PROXY_PORT="${PROXY_PORT:-8199}"+PROXY_LOG="${PROXY_LOG:-/tmp/ch-scratch-proxy-${MODE}.log}"+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"+PG_BASE_URL="${PG_BASE_URL:-postgresql://postgres:postgres@localhost:5432}"+PG_ADMIN_URL="$PG_BASE_URL/postgres"+CH_ADMIN_URL="${CH_ADMIN_URL:-http://127.0.0.1:8123}"++case "$SCRATCH_DB" in+ openpanel|openpanel_test|postgres)+ echo "refusing to use '$SCRATCH_DB' as a scratch database" >&2+ exit 1+ ;;+esac++pg_query() { PG_QUERY_URL="$1" bun -e "+const sql = new Bun.SQL(process.env.PG_QUERY_URL);+const rows = await sql.unsafe(process.argv[1]);+if (Array.isArray(rows) && rows.length) console.log(JSON.stringify(rows));+await sql.end();+" "$2"; }++# The maintenance connection (creates/drops databases) and the scratch database+# itself are different connections — reporting through the wrong one reports the+# dev database's state.+pg_admin() { pg_query "$PG_ADMIN_URL" "$1"; }+pg_scratch() { pg_query "$PG_BASE_URL/$SCRATCH_DB" "$1"; }++ch_admin() { curl -sS --fail-with-body "$CH_ADMIN_URL/" --data-binary "$1"; }++banner() { printf '\n========== %s ==========\n' "$1"; }++banner "0. scratch databases"+pg_admin "DROP DATABASE IF EXISTS \"$SCRATCH_DB\""+pg_admin "CREATE DATABASE \"$SCRATCH_DB\""+ch_admin "DROP DATABASE IF EXISTS $SCRATCH_DB"+ch_admin "CREATE DATABASE $SCRATCH_DB"+echo "postgres: $SCRATCH_DB created"+echo "clickhouse: $SCRATCH_DB created, tables: $(ch_admin "SELECT count() FROM system.tables WHERE database='$SCRATCH_DB'")"++banner "0b. protected databases before the run"+ch_admin "SELECT database, count() FROM system.tables WHERE database IN ('openpanel','openpanel_test') GROUP BY database ORDER BY database FORMAT TSV"++if curl -sS "http://127.0.0.1:$PROXY_PORT/" --data-binary "SELECT 1" >/dev/null 2>&1; then+ echo "port $PROXY_PORT already answers — a stale proxy would silently serve this run" >&2+ exit 1+fi++if [ "$MODE" = "rewrite" ]; then+ PROXY_MODE=rewrite SCRATCH_DATABASE="$SCRATCH_DB" PROXY_PORT="$PROXY_PORT" PROXY_LOG="$PROXY_LOG" \+ bun "$REPO_ROOT/docs/migrations-fresh-install/ch-scratch-proxy.ts" &+else+ PROXY_MODE=guard PROXY_PORT="$PROXY_PORT" PROXY_LOG="$PROXY_LOG" \+ bun "$REPO_ROOT/docs/migrations-fresh-install/ch-scratch-proxy.ts" &+fi+PROXY_PID=$!+trap 'kill $PROXY_PID 2>/dev/null || true' EXIT+until curl -sS "http://127.0.0.1:$PROXY_PORT/?query=SELECT%201" >/dev/null 2>&1; do sleep 0.2; done++export DATABASE_URL="$PG_BASE_URL/$SCRATCH_DB?schema=public"+export DATABASE_URL_DIRECT="$DATABASE_URL"+export CLICKHOUSE_URL="http://127.0.0.1:$PROXY_PORT/$SCRATCH_DB"+export SELF_HOSTED=true++run_once() {+ banner "$1 — prisma migrate deploy"+ local started=$SECONDS+ (cd "$REPO_ROOT/packages/db" && bunx [email protected] migrate deploy) || return 1+ echo "[timing] $1 prisma migrate deploy: $((SECONDS - started))s"++ banner "$1 — code migrations"+ started=$SECONDS+ (cd "$REPO_ROOT/packages/core" && bun scripts/migrate-code.ts) || return 1+ echo "[timing] $1 code migrations: $((SECONDS - started))s"+}++FIRST_RESULT=ok+run_once "RUN 1" || FIRST_RESULT=failed++banner "state after run 1 ($FIRST_RESULT)"+pg_scratch "SELECT count(*)::int AS applied FROM \"_prisma_migrations\" WHERE finished_at IS NOT NULL" || true+pg_scratch "SELECT count(*)::int AS code_migrations FROM \"__code_migrations\"" || true+ch_admin "SELECT count() FROM system.tables WHERE database='$SCRATCH_DB'"++SECOND_RESULT=ok+run_once "RUN 2 (idempotency)" || SECOND_RESULT=failed++banner "state after run 2 ($SECOND_RESULT)"+pg_scratch "SELECT count(*)::int AS applied FROM \"_prisma_migrations\" WHERE finished_at IS NOT NULL" || true+pg_scratch "SELECT count(*)::int AS code_migrations FROM \"__code_migrations\"" || true+ch_admin "SELECT count() FROM system.tables WHERE database='$SCRATCH_DB'"++banner "protected databases after the run"+ch_admin "SELECT database, count() FROM system.tables WHERE database IN ('openpanel','openpanel_test') GROUP BY database ORDER BY database FORMAT TSV"++banner "verdict"+echo "mode=$MODE run1=$FIRST_RESULT run2=$SECOND_RESULT proxy log: $PROXY_LOG"diff --git a/docs/migrations-fresh-install/fresh-install-teardown.sh b/docs/migrations-fresh-install/fresh-install-teardown.shnew file mode 100755index 000000000..8554588c9--- /dev/null+++ b/docs/migrations-fresh-install/fresh-install-teardown.sh@@ -0,0 +1,26 @@+#!/usr/bin/env bash+# Drops ONLY the scratch databases the drill created, and removes the `.sql`+# dumps the migrations write next to themselves (untracked build artifacts).+set -euo pipefail++SCRATCH_DB="${SCRATCH_DB:-openpanel_m16_004}"+PG_ADMIN_URL="${PG_ADMIN_URL:-postgresql://postgres:postgres@localhost:5432/postgres}"+CH_ADMIN_URL="${CH_ADMIN_URL:-http://127.0.0.1:8123}"++case "$SCRATCH_DB" in+ openpanel|openpanel_test|postgres)+ echo "refusing to drop '$SCRATCH_DB'" >&2+ exit 1+ ;;+esac++PG_ADMIN_URL="$PG_ADMIN_URL" bun -e "+const sql = new Bun.SQL(process.env.PG_ADMIN_URL);+await sql.unsafe(\`DROP DATABASE IF EXISTS \"\${process.argv[1]}\"\`);+await sql.end();+" "$SCRATCH_DB"+curl -sS --fail-with-body "$CH_ADMIN_URL/" --data-binary "DROP DATABASE IF EXISTS $SCRATCH_DB"+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"+rm -f "$REPO_ROOT"/packages/db/src/code-migrations/*.sql++echo "dropped scratch databases named $SCRATCH_DB and removed the .sql dumps"