Skip to content

OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.

ralph(M12-009): DELETE the builders: query-builder.ts, its test and sql-builder.ts gone, sqlstring removed from every package.json, sql.ts joins the db barrel, p12-grep-gates --assert exits 0, goldens 137/137

ralph4925d2f7a082
diff --git a/apps/start/package.json b/apps/start/package.json
index f4a9b7765..2e79d55e6 100644
--- a/apps/start/package.json
+++ b/apps/start/package.json
@@ -150,7 +150,6 @@
"short-unique-id": "^5.0.3",
"slugify": "^1.6.6",
"sonner": "^1.4.0",
- "sqlstring": "^2.3.3",
"superjson": "^2.2.2",
"tailwind-merge": "^3.3.1",
"tailwindcss": "^4.0.6",
diff --git a/docs/TECH_DEBT.md b/docs/TECH_DEBT.md
index b9a37e8d0..b7ef60b5b 100644
--- a/docs/TECH_DEBT.md
+++ b/docs/TECH_DEBT.md
@@ -1154,3 +1154,188 @@ package imports it — the wire-constant types were inlined into `src/index.ts`
in a pre-rewrite commit (`fix(sdk): inline validation types in dts...`) and
the manifest entry never followed. Safe to delete whenever that
`package.json` is next touched.
+
+### 2026-09-07 — M12-009, ADR-013 decision 21 closed: the builders are deleted and `sqlstring` is out of the repo
+
+**RESOLVED: the "clix coexistence" exception.** ADR-013's P8 deletion had been
+deferred since M7 and was last restated by `M9-CLEANUP-001` in
+`packages/db/index.ts`:
+
+> `clix` (query-builder.ts) and `createSqlBuilder` (sql-builder.ts) are dead
+> per ADR-013, but 7 live call sites across @openpanel/core still import them
+> directly and have not been converted onto the `sql` tag yet — see
+> M9-CLEANUP-001's report. Deleting these two ahead of that conversion is
+> BLOCKED, not done.
+
+M12-002..008 emptied that import graph; M12-009 deleted the three files
+(`git rm`), took the barrel comment with them, and removed `sqlstring` /
+`@types/sqlstring` from every workspace that declared it (`packages/db`,
+`packages/core`, `apps/start` — `apps/api` had already lost its copy). Both
+sections of this register that cite `query-builder.ts` / `sql-builder.ts` as
+exempt targets of `core-uses-ctx-not-db-internals` are now describing paths
+that do not exist; `packages/db/src/clickhouse/sql.ts` is the only one left,
+and it is also the one the rule keeps exempt.
+
+**Final `bash tooling/gates/p12-grep-gates.sh --report`, ralph 2026-09-07:**
+
+```
+ sqlstring clix sql-builder file
+----------------------------------------------------------------------------
+----------------------------------------------------------------------------
+ 0 0 0 TOTAL
+```
+
+`--assert` exits `0`: `OK: sqlstring = 0, clix = 0, sql-builder = 0; all three
+definer files are gone.`
+
+**The leftovers M12-008 disclosed, converted here.** The gate read
+`sqlstring 16 / clix 1 / sql-builder 1` on clean `30d236fa` (measured by
+stashing this task's tree and re-running `--report`), spread over eight files:
+`apps/start/package.json` 1, `packages/core/package.json` 2, `group-buffer.ts`
+3, `profile-backfill-buffer.ts` 3, `profile-buffer.ts` 2,
+`packages/db/package.json` 2, `packages/db/src/clickhouse/client.ts` 3, and
+`packages/db/index.ts` 1 clix + 1 sql-builder. The gate does not scan
+`packages/db/scripts/`, but the two scripts there imported `clix` and
+`sqlstring` and had to convert too — removing the dependency removes the module
+they import.
+
+Every changed statement was executed against local ClickHouse in **both** forms
+through the same `@clickhouse/client` — V1 as `query`, V2 as `query` +
+`query_params` — on the same data. **8 statements, all IDENTICAL.** Full SQL,
+params, row counts and timings: `packages/core/src/buffers/buffers.sql.proof.md`
+and `packages/db/scripts/scripts.sql.proof.md`.
+
+| Site | Statement | Rows | old → new |
+|---|---|---:|---|
+| `core/src/buffers/group-buffer.ts:84` | `SELECT … FROM groups FINAL` | 1 | 33ms → 8ms |
+| `core/src/buffers/profile-buffer.ts:222` | batch fetch, no date filter | 2 | 20ms → 20ms |
+| `core/src/buffers/profile-buffer.ts:222` | batch fetch, 2-day filter | 0 | 9ms → 8ms |
+| `core/src/buffers/profile-backfill-buffer.ts:96` | lightweight `UPDATE … CASE` | 3 read back | 7ms → 6ms |
+| `db/scripts/ch-update-sessions-with-revenue.ts:45` | `SELECT id FROM sessions` | 2 | 5ms → 5ms |
+| `db/scripts/ch-update-sessions-with-revenue.ts:79` | `ALTER TABLE … UPDATE multiIf` | 4 read back | 11ms → 14ms |
+| `db/scripts/ch-copy-from-remote.ts:95` | `SELECT * FROM remote(…)` | 2 | 7ms → 9ms |
+| `db/scripts/ch-copy-from-remote.ts:97` | `INSERT … SELECT * FROM remote(…)` | 3 → 5 | 9ms → 9ms |
+
+Reads ran against the local prod-copy `openpanel`; the two mutations and the
+`INSERT` ran against a throwaway `m12009_scratch` database, dropped afterwards
+— never against `openpanel` or `openpanel_test`. `remote()` accepts bound
+params (`{p:String}` for the host and credentials, `{p:Identifier}` for the
+database and table), verified before the conversion was written, which is what
+takes the remote password out of the query text the script logs. No `IN` was
+converted to `GLOBAL IN` or back: every `IN` touched takes a literal set, not
+a subquery, so `docs/ENVIRONMENT.md`'s distributed-`IN` trap does not apply.
+
+Timings are single-node and directional only (ClickHouse 26.1.3.52, 4 vCPU);
+production is 2 shards × 2 replicas.
+
+`packages/db/src/clickhouse/client.ts`'s `toDate(str, interval?)` was deleted
+rather than converted: it existed only to `sqlstring.escape` a date-shaped
+string into query text, it had zero importers in the repo, and keeping it
+would have kept a manual-escaping helper alive that ADR-013 forbids.
+
+**`packages/db/src/clickhouse/sql.ts` joined the barrel**, as its own header
+promised. `packages/core` still reaches it by its deep path in all 55 files:
+the barrel also re-exports `clickhouse/client.ts` and `prisma-client.ts`,
+which construct a ClickHouse client array and a `PrismaClient` at module load,
+so a barrel import from core acquires a second, request-scope-less client —
+which is exactly what `.dependency-cruiser.cjs`'s
+`core-uses-ctx-not-db-internals` forbids (its comment: "prisma-client.ts,
+clickhouse/client.ts, logger.ts **and the barrel** are NOT exempt"). Nothing
+outside `packages/core` imported the deep path.
+
+**Debt noticed, not fixed (outside this task's scope — the file is at the repo
+root, not under any of this task's scope globs):** `.dependency-cruiser.cjs`'s
+`core-uses-ctx-not-db-internals` still exempts
+`^packages/db/src/clickhouse/query-builder\.ts$` and
+`^packages/db/src/sql-builder\.ts$` in its `to.pathNot`, and its comment still
+explains both. They are dead regex branches now — harmless (nothing can match
+them), but the next task that touches that file should delete both lines and
+the two sentences describing them.
+
+#### `packages/core/src/modules/chart/src/compiled.ts` — kept, with its caller census
+
+M12-003 kept `compiled.ts` and its reviewer checked the reason: neither filter
+compiler (`filter-where.ts`, `table-filter-where.ts`) nor the field resolver
+(`field-resolution.ts`'s `getSelectPropertyKey`) returns anything that crosses
+`compiledText` any more — they all return `SqlFragment`s — so what is left are
+the chart builders splicing a name they generated themselves.
+`compiledTextWithProfileRefs` was deleted outright at that time. That still
+holds at M12-009: `compiled.ts` has **29 `compiledText(` splice sites on 27
+lines across 8 files**, so it
+is NOT deleted, and this is the census the M12-009 criterion asks for.
+
+`compiledText` stays a named export in one file rather than a `sql.raw()`
+scattered across the module precisely so this census is a grep:
+
+```
+$ grep -rn "compiledText(" packages/core/src --include=*.ts | grep -v '/compiled.ts:'
+```
+
+| file:line | what it splices | where the text comes from |
+|---|---|---|
+| `chart/src/chart.sql.ts:175` | a CTE name in `WITH <name> AS (…)` | keys of the `ctes` record the builder itself assembles — the profile CTE name and `` `cohort-<id>` `` (below) |
+| `chart/src/chart.sql.ts:274` | the `"profile.<field>"` CTE alias | `field`, validated on the same line by `sql.id(field, PROFILE_CTE_FIELDS)` against a closed column set (`id`,
+`properties`, plus `PROFILE_CTE_SCALAR_FIELDS`) |
+| `chart/src/chart.sql.ts:340` | `` `cohort-<id>` `` as a JOIN target | `cohortId`, through `assertCohortId` (below) |
+| `chart/src/chart.sql.ts:434` | `label_<n>` breakdown alias | `\`label_${index + 1}\`` — a loop index |
+| `chart/src/chart.sql.ts:435` | the same `label_<n>` as a GROUP BY key | same |
+| `chart/src/chart.sql.ts:469` | the aggregate keyword (`sum`/`avg`/`max`/`min`) | `MATH_FUNCTION_BY_SEGMENT[event.segment]` — a 4-entry literal `Record` at `chart.sql.ts:58`; an unknown segment yields `undefined`, not the input |
+| `chart/src/chart.sql.ts:580` | `label_<n>` in a window `PARTITION BY` | a loop index |
+| `chart/src/funnel.sql.ts:78` | `, 'strict_increase'` — the `windowFunnel` mode | `STRICT_INCREASE_MODE`, a file constant; the only branch is an env flag |
+| `chart/src/funnel.sql.ts:188` | `b_<index>` breakdown alias | a loop index |
+| `chart/src/funnel.sql.ts:307` | `b_<index>` as a GROUP BY key | a loop index |
+| `chart/src/funnel.sql.ts:366` | `b_<index>` in SELECT and GROUP BY | a loop index |
+| `chart/src/funnel.sql.ts:391` | `trim(ifNull(toString(b_<n>), ''))` | a loop index inside a fixed template; the compared VALUE beside it binds as `sql.string` |
+| `chart/src/conversion.sql.ts:117` | `b_<index>` outer-select alias | a loop index |
+| `chart/conversion.service.ts:137` | `b_<index>` alias | a loop index |
+| `chart/src/retention.sql.ts:170` | `toDate` / `toStartOfWeek` / `toStartOfMonth` | `SQL_START_OF[interval]`, a 5-entry `Record` keyed by the `IRetentionInterval` union |
+| `chart/src/retention.sql.ts:171` | the `dateDiff` / `INTERVAL` unit | `SQL_INTERVAL[interval]`, same shape |
+| `chart/src/retention.sql.ts:172` | `>=` vs `=` | `COUNT_CRITERIA[criteria]`, keyed by the `IRetentionCriteria` union |
+| `chart/src/retention.sql.ts:200` | `interval_<n>_users` alias | `String(index)` over `range(0..diffInterval)` |
+| `chart/src/retention.sql.ts:206` | `interval_<n>_users` / `_user_count` | same |
+| `chart/src/sankey.sql.ts:28` | the `arrayFilter(…) as events_deduped` expression | a file-level constant, no interpolation |
+| `chart/src/sankey.sql.ts:38` | the `if(arrayFirstIndex(…))` truncation | a file-level constant, no interpolation |
+| `chart/src/sankey.sql.ts:48` | the `arrayJoin(arrayMap(…))` transition subselect | a file-level constant, no interpolation |
+| `chart/src/sankey.sql.ts:187` | `arraySlice(events, start_index, …)` | a file-level constant, no interpolation |
+| `chart/src/sankey.sql.ts:201` (×2) | the base CTE name | `transitionsFrom`'s two callers pass the string literals `'session_paths_base'` and `'between_sessions'` |
+| `overview/src/overview.sql.ts:115` | the `WITH FILL … STEP toInterval<Unit>(1)` step | `toIntervalStep(interval)`, a `switch` over the `IInterval` union that throws on anything else |
+| `chart/src/field-resolution.ts:203` | `` `cohort-<id>` `` CTE name | `cohortId` — see below |
+| `chart/src/field-resolution.ts:505` | `` `profile.properties.<key>` `` CTE alias | `key` — see below |
+
+**Nothing above is a value, and nothing above is a compiler's output.** Every
+row is an alias, a CTE name, a SQL keyword or a static expression. The two
+compilers' and the resolver's outputs reach the statement as `SqlFragment`s and
+never pass through `compiledText`.
+
+**Two rows are name-shaped text derived from a user-chosen NAME, and are
+flagged rather than buried** — they are the same two M12-003's reviewer read,
+and both are guarded so that `compiledText` receives only a validated
+identifier, never the raw input:
+
+- **`` `cohort-<id>` ``** (`field-resolution.ts:203`, `chart.sql.ts:340`,
+ and the `ctes` key at `chart.sql.ts:175`). `cohortId` comes from a saved
+ report's breakdown config. `chart.sql.ts:119`'s `assertCohortId` tests it
+ against `/^[A-Za-z0-9_-]+$/` and **throws `ChartCohortIdError`** otherwise —
+ it never falls back to inlining, which is what ADR-013 R3 requires of an
+ identifier path. The name needs the text seam because a dash makes `sql.id`
+ reject it and `{x:Identifier}` is not accepted in a `WITH … AS` position.
+ `field-resolution.ts:202`'s `getCohortCteName` has no in-repo caller today
+ beyond the two barrel re-exports (`chart.service.ts:133`, `index.ts:338`);
+ `chart.sql.ts` builds the same name behind `assertCohortId` itself.
+- **`` `profile.properties.<key>` ``** (`field-resolution.ts:505`). `key` is
+ the tail of a `profile.properties.<key>` field reference in the chart config.
+ `collectProfilePropertyKeys` (`field-resolution.ts:465`, guard at `:484`)
+ **rejects any key
+ containing a backtick or a backslash** — the only two characters that are
+ special inside a backtick-quoted ClickHouse identifier — and such keys are
+ simply not narrowed: the full `properties` Map stays selected and their refs
+ keep the Map access. The alias has three dot-separated parts, so `sql.id`
+ cannot express it. The key's *value* twin on the same line binds normally
+ (`properties[${sql.string(k)}]`).
+
+Both are the identifier tier of ADR-013, implemented as a guard local to the
+call site rather than through `sql.id`, because `sql.id`'s grammar (bare or
+once-qualified, no dashes, no backticks) cannot spell either name. **If a
+future task widens `sql.id` to accept a backtick-quoted, multi-part name,
+these two rows and `compiled.ts` with them can go.** That is the standing exit
+condition; recorded here, not done in M12-009 (out of its scope).
diff --git a/packages/core/package.json b/packages/core/package.json
index 0eea1dd4d..5406436ce 100644
--- a/packages/core/package.json
+++ b/packages/core/package.json
@@ -68,7 +68,6 @@
"sharp": "^0.33.5",
"short-unique-id": "^5.0.3",
"slugify": "^1.6.6",
- "sqlstring": "^2.3.3",
"superjson": "^1.13.3",
"ua-parser-js": "^2.0.6",
"undici": "^8.10.0",
@@ -84,7 +83,6 @@
"@types/luxon": "^3.7.1",
"@types/qrcode": "^1.5.6",
"@types/ramda": "^0.31.1",
- "@types/sqlstring": "^2.3.2",
"@types/unzipper": "^0.10.10",
"@types/uuid": "^10.0.0",
"bun-types": "1.4.0",
diff --git a/packages/core/src/buffers/base-buffer.ts b/packages/core/src/buffers/base-buffer.ts
index c607cf0bc..2dfe051c8 100644
--- a/packages/core/src/buffers/base-buffer.ts
+++ b/packages/core/src/buffers/base-buffer.ts
@@ -3,7 +3,7 @@ import type { ClickHouseSettings } from '@clickhouse/client';
import { getRedisCache } from '@openpanel/redis';
import type { Logger } from '../logger';
import type { ServiceDeps } from '../services';
-import { type ChScope, chQuery } from '../shared/ch-query';
+import { type ChQueryInput, type ChScope, chQuery } from '../shared/ch-query';
import { generateSecureId } from '../shared/id';
// GENUINE CYCLE, kept lazy: v1-compat.ts -> services.ts -> (every service) ->
@@ -131,7 +131,7 @@ export class BaseBuffer {
}
/** `chQuery` bound to this buffer's client and its own logger. */
- protected async chQuery<T extends object>(query: string): Promise<T[]> {
+ protected async chQuery<T extends object>(query: ChQueryInput): Promise<T[]> {
const scope: ChScope = { ch: await this.resolveCh(), logger: this.logger };
return chQuery<T>(scope, query);
}
diff --git a/packages/core/src/buffers/buffers.sql.proof.md b/packages/core/src/buffers/buffers.sql.proof.md
new file mode 100644
index 000000000..9ad2cdc89
--- /dev/null
+++ b/packages/core/src/buffers/buffers.sql.proof.md
@@ -0,0 +1,141 @@
+# buffers off `sqlstring` — V1 → V2 result-set proof (M12-009)
+
+M12-009 deletes `sqlstring` from every workspace manifest, so the three buffer
+files that still imported it had to move onto the `sql` tag first. V1 is the
+`sqlstring.escape(...)` text each file built at `30d236fa`; V2 is the tagged
+statement in this commit. Both forms were rendered in one Bun process and
+executed through the same `@clickhouse/client` (`format: 'JSONCompact'`) — V1
+as `query`, V2 as `query` + `query_params`. `meta` and `data` were compared
+field for field.
+
+- **Date**: 2026-09-07, run by ralph on this box.
+- **Reads**: local prod-copy `openpanel` (319M events), read-only.
+- **Mutation**: the `profile-backfill-buffer` lightweight `UPDATE` is a write,
+ so it ran against a throwaway `m12009_scratch` database seeded with three
+ rows, never against `openpanel` or `openpanel_test`. The database was dropped
+ after the run.
+- **Machine**: single-node ClickHouse 26.1.3.52 on a 4-vCPU box. Timings are
+ directional only; production is 2 shards × 2 replicas
+ (`docs/ENVIRONMENT.md`).
+- **Verdict**: **4 statements, all IDENTICAL.**
+
+| # | Site | Statement | Rows | old → new |
+|---|---|---|---:|---|
+| 1 | `group-buffer.ts:84` | `SELECT … FROM groups FINAL WHERE project_id/id` | 1 | 33ms → 8ms |
+| 2 | `profile-buffer.ts:222` | batch fetch, `withDateFilter=false` | 2 | 20ms → 20ms |
+| 3 | `profile-buffer.ts:222` | batch fetch, `withDateFilter=true` | 0 | 9ms → 8ms |
+| 4 | `profile-backfill-buffer.ts:96` | lightweight `UPDATE … CASE session_id` | 3 read back | 7ms → 6ms |
+
+## IN / GLOBAL IN
+
+Every `IN` touched here takes a **literal set** — `(p.id, p.project_id) IN
+((…), (…))` and `(project_id, session_id) IN ((…), (…))` — not a subquery. A
+literal set is evaluated identically on every shard, so `docs/ENVIRONMENT.md`'s
+`IN` vs `GLOBAL IN` trap does not apply and **no `IN` was converted in either
+direction**. Binding the tuple members as `{pN:String}` does not change that:
+the set is still spelled out in the query text, one placeholder per member.
+
+## Run output
+
+```
+==============================================================================
+CASE: group-buffer: SELECT ... FROM groups FINAL [db=openpanel]
+--- OLD (clix/sqlstring text) ---
+SELECT project_id, id, type, name, properties, created_at
+ FROM groups FINAL
+ WHERE project_id = 'acade'
+ AND id = 'acade'
+ AND deleted = 0
+--- NEW (sql tag) ---
+SELECT project_id, id, type, name, properties, created_at
+ FROM groups FINAL
+ WHERE project_id = {p1:String}
+ AND id = {p2:String}
+ AND deleted = 0
+--- params ---
+{"p1":"acade","p2":"acade"}
+old: ok=true rows=1 33ms
+new: ok=true rows=1 8ms
+RESULT SETS: IDENTICAL
+==============================================================================
+CASE: profile-buffer: batch fetch (withDateFilter=false) [db=openpanel]
+--- OLD (clix/sqlstring text) ---
+SELECT id, project_id, argMax(first_name, p.last_seen_at) AS first_name, argMax(last_name, p.last_seen_at) AS last_name, argMax(email, p.last_seen_at) AS email, argMax(avatar, p.last_seen_at) AS avatar, argMax(properties, p.last_seen_at) AS properties, argMax(is_external, p.last_seen_at) AS is_external, argMax(groups, p.last_seen_at) AS groups, min(created_at) AS created_at, max(p.last_seen_at) AS last_seen_at
+ FROM profiles AS p
+ WHERE (p.id, p.project_id) IN (('03b7d7d74db08200f7fb3dc912abb51e', '0byte'), ('07deba9fc77b280a8dbc7f823d5c2319', '0byte'))
+
+ GROUP BY p.id, p.project_id
+--- NEW (sql tag) ---
+SELECT id, project_id, argMax(first_name, p.last_seen_at) AS first_name, argMax(last_name, p.last_seen_at) AS last_name, argMax(email, p.last_seen_at) AS email, argMax(avatar, p.last_seen_at) AS avatar, argMax(properties, p.last_seen_at) AS properties, argMax(is_external, p.last_seen_at) AS is_external, argMax(groups, p.last_seen_at) AS groups, min(created_at) AS created_at, max(p.last_seen_at) AS last_seen_at
+ FROM profiles AS p
+ WHERE (p.id, p.project_id) IN (({p1:String}, {p2:String}), ({p3:String}, {p4:String}))
+
+ GROUP BY p.id, p.project_id
+--- params ---
+{"p1":"03b7d7d74db08200f7fb3dc912abb51e","p2":"0byte","p3":"07deba9fc77b280a8dbc7f823d5c2319","p4":"0byte"}
+old: ok=true rows=2 20ms
+new: ok=true rows=2 20ms
+RESULT SETS: IDENTICAL
+==============================================================================
+CASE: profile-buffer: batch fetch (withDateFilter=true) [db=openpanel]
+--- OLD (clix/sqlstring text) ---
+SELECT id, project_id, argMax(first_name, p.last_seen_at) AS first_name, argMax(last_name, p.last_seen_at) AS last_name, argMax(email, p.last_seen_at) AS email, argMax(avatar, p.last_seen_at) AS avatar, argMax(properties, p.last_seen_at) AS properties, argMax(is_external, p.last_seen_at) AS is_external, argMax(groups, p.last_seen_at) AS groups, min(created_at) AS created_at, max(p.last_seen_at) AS last_seen_at
+ FROM profiles AS p
+ WHERE (p.id, p.project_id) IN (('03b7d7d74db08200f7fb3dc912abb51e', '0byte'), ('07deba9fc77b280a8dbc7f823d5c2319', '0byte'))
+ AND p.last_seen_at > now() - INTERVAL 2 DAY
+ GROUP BY p.id, p.project_id
+--- NEW (sql tag) ---
+SELECT id, project_id, argMax(first_name, p.last_seen_at) AS first_name, argMax(last_name, p.last_seen_at) AS last_name, argMax(email, p.last_seen_at) AS email, argMax(avatar, p.last_seen_at) AS avatar, argMax(properties, p.last_seen_at) AS properties, argMax(is_external, p.last_seen_at) AS is_external, argMax(groups, p.last_seen_at) AS groups, min(created_at) AS created_at, max(p.last_seen_at) AS last_seen_at
+ FROM profiles AS p
+ WHERE (p.id, p.project_id) IN (({p1:String}, {p2:String}), ({p3:String}, {p4:String}))
+ AND p.last_seen_at > now() - INTERVAL 2 DAY
+ GROUP BY p.id, p.project_id
+--- params ---
+{"p1":"03b7d7d74db08200f7fb3dc912abb51e","p2":"0byte","p3":"07deba9fc77b280a8dbc7f823d5c2319","p4":"0byte"}
+old: ok=true rows=0 9ms
+new: ok=true rows=0 8ms
+RESULT SETS: IDENTICAL
+```
+
+The `profile-backfill-buffer` case, run against `m12009_scratch`:
+
+```
+==============================================================================
+CASE: profile-backfill-buffer: lightweight UPDATE ... CASE session_id [db=m12009_scratch]
+--- OLD ---
+UPDATE events_replicated
+ SET profile_id = CASE session_id
+ WHEN 's1' THEN 'prof-1'
+WHEN 's2' THEN 'prof-2'
+ END
+ WHERE (project_id, session_id) IN (('p1', 's1'),('p1', 's2'))
+ AND created_at > now() - INTERVAL 6 HOURS
+--- NEW ---
+UPDATE events_replicated
+ SET profile_id = CASE session_id
+ WHEN {p1:String} THEN {p2:String}
+WHEN {p3:String} THEN {p4:String}
+ END
+ WHERE (project_id, session_id) IN (({p5:String}, {p6:String}), ({p7:String}, {p8:String}))
+ AND created_at > now() - INTERVAL 6 HOURS
+--- params ---
+{"p1":"s1","p2":"prof-1","p3":"s2","p4":"prof-2","p5":"p1","p6":"s1","p7":"p1","p8":"s2"}
+old: ok=true 7ms
+new: ok=true 6ms
+old rows after: {"meta":[{"name":"project_id","type":"String"},{"name":"session_id","type":"String"},{"name":"profile_id","type":"String"}],"data":[["p1","s1","prof-1"],["p1","s2","prof-2"],["p2","s3","keep"]]}
+new rows after: {"meta":[{"name":"project_id","type":"String"},{"name":"session_id","type":"String"},{"name":"profile_id","type":"String"}],"data":[["p1","s1","prof-1"],["p1","s2","prof-2"],["p2","s3","keep"]]}
+RESULT SETS: IDENTICAL
+```
+
+`m12009_scratch.events_replicated` needed `enable_block_number_column = 1` for
+`allow_experimental_lightweight_update` to apply — a property of the throwaway
+fixture table, not of the statement; both forms failed identically before it was
+set and both succeeded identically after.
+
+## Whitespace-only differences, stated
+
+`profile-backfill-buffer.ts`'s tuple list joins on `', '` where V1 joined on
+`','`, and `profile-buffer.ts`'s aggregate column list is now a
+`sql.join([...], ', ')` of one fragment per column instead of a
+`String[].join(', ')`. Both render the same tokens; the run above is over the
+rendered text, so the comparison covers it.
diff --git a/packages/core/src/buffers/group-buffer.ts b/packages/core/src/buffers/group-buffer.ts
index 0ad19b9ee..241c65256 100644
--- a/packages/core/src/buffers/group-buffer.ts
+++ b/packages/core/src/buffers/group-buffer.ts
@@ -1,6 +1,6 @@
+import { sql } from '@openpanel/db/src/clickhouse/sql';
import { getRedisCache, type Redis } from '@openpanel/redis';
import shallowEqual from 'fast-deep-equal';
-import sqlstring from 'sqlstring';
import { formatClickhouseDate } from '../shared/ch-dates';
import { TABLE_NAMES } from '../shared/ch-tables';
import { getSafeJson } from '../shared/json';
@@ -80,11 +80,11 @@ export class GroupBuffer extends BaseBuffer {
projectId: string,
id: string
): Promise<IGroupCacheEntry | null> {
- const rows = await this.chQuery<IGroupCacheEntry>(`
+ const rows = await this.chQuery<IGroupCacheEntry>(sql`
SELECT project_id, id, type, name, properties, created_at
- FROM ${TABLE_NAMES.groups} FINAL
- WHERE project_id = ${sqlstring.escape(projectId)}
- AND id = ${sqlstring.escape(id)}
+ FROM ${sql.id(TABLE_NAMES.groups)} FINAL
+ WHERE project_id = ${sql.string(projectId)}
+ AND id = ${sql.string(id)}
AND deleted = 0
`);
return rows[0] ?? null;
diff --git a/packages/core/src/buffers/profile-backfill-buffer.ts b/packages/core/src/buffers/profile-backfill-buffer.ts
index b2ade70ef..24a0ec438 100644
--- a/packages/core/src/buffers/profile-backfill-buffer.ts
+++ b/packages/core/src/buffers/profile-backfill-buffer.ts
@@ -1,5 +1,5 @@
+import { sql } from '@openpanel/db/src/clickhouse/sql';
import { getRedisCache, type Redis } from '@openpanel/redis';
-import sqlstring from 'sqlstring';
import { getReplicatedTableName, TABLE_NAMES } from '../shared/ch-tables';
import { getSafeJson } from '../shared/json';
import { BaseBuffer, type BufferDeps } from './base-buffer';
@@ -77,29 +77,32 @@ export class ProfileBackfillBuffer extends BaseBuffer {
const chStart = performance.now();
for (const chunk of chunks) {
- const caseClause = chunk
- .map(
+ const caseClause = sql.join(
+ chunk.map(
({ sessionId, profileId }) =>
- `WHEN ${sqlstring.escape(sessionId)} THEN ${sqlstring.escape(profileId)}`
- )
- .join('\n');
- const tupleList = chunk
- .map(
+ sql`WHEN ${sql.string(sessionId)} THEN ${sql.string(profileId)}`
+ ),
+ '\n'
+ );
+ const tupleList = sql.join(
+ chunk.map(
({ projectId, sessionId }) =>
- `(${sqlstring.escape(projectId)}, ${sqlstring.escape(sessionId)})`
- )
- .join(',');
+ sql`(${sql.string(projectId)}, ${sql.string(sessionId)})`
+ ),
+ ', '
+ );
- const query = `
- UPDATE ${table}
+ const statement = sql`
+ UPDATE ${sql.id(table)}
SET profile_id = CASE session_id
${caseClause}
END
WHERE (project_id, session_id) IN (${tupleList})
- AND created_at > now() - INTERVAL 6 HOURS`;
+ AND created_at > now() - INTERVAL 6 HOURS`.toStatement();
await ch.command({
- query,
+ query: statement.query,
+ query_params: statement.query_params,
clickhouse_settings: {
mutations_sync: '0',
allow_experimental_lightweight_update: '1',
diff --git a/packages/core/src/buffers/profile-buffer.ts b/packages/core/src/buffers/profile-buffer.ts
index 6cf07250e..be9f8bc0a 100644
--- a/packages/core/src/buffers/profile-buffer.ts
+++ b/packages/core/src/buffers/profile-buffer.ts
@@ -1,6 +1,6 @@
+import { sql } from '@openpanel/db/src/clickhouse/sql';
import { getRedisCache, type Redis } from '@openpanel/redis';
import { omit, uniq } from 'ramda';
-import sqlstring from 'sqlstring';
import type { IClickhouseProfile } from '../modules/profile/profile.service';
import { TABLE_NAMES } from '../shared/ch-tables';
import { getSafeJson } from '../shared/json';
@@ -25,22 +25,25 @@ const PROFILE_COLUMNS =
// bare column refs against the alias list first — causing ILLEGAL_AGGREGATION
// inside the argMax/max calls. Qualifying with `p.` bypasses the alias
// lookup and binds to the raw column.
-const PROFILE_LATEST_AGGREGATE_COLUMNS = [
- 'id',
- 'project_id',
- 'argMax(first_name, p.last_seen_at) AS first_name',
- 'argMax(last_name, p.last_seen_at) AS last_name',
- 'argMax(email, p.last_seen_at) AS email',
- 'argMax(avatar, p.last_seen_at) AS avatar',
- 'argMax(properties, p.last_seen_at) AS properties',
- 'argMax(is_external, p.last_seen_at) AS is_external',
- 'argMax(groups, p.last_seen_at) AS groups',
- // created_at doesn't change between rows for the same profile, but
- // min() is safe and matches "first seen" semantics if there ever is a
- // discrepancy.
- 'min(created_at) AS created_at',
- 'max(p.last_seen_at) AS last_seen_at',
-].join(', ');
+const PROFILE_LATEST_AGGREGATE_COLUMNS = sql.join(
+ [
+ sql`id`,
+ sql`project_id`,
+ sql`argMax(first_name, p.last_seen_at) AS first_name`,
+ sql`argMax(last_name, p.last_seen_at) AS last_name`,
+ sql`argMax(email, p.last_seen_at) AS email`,
+ sql`argMax(avatar, p.last_seen_at) AS avatar`,
+ sql`argMax(properties, p.last_seen_at) AS properties`,
+ sql`argMax(is_external, p.last_seen_at) AS is_external`,
+ sql`argMax(groups, p.last_seen_at) AS groups`,
+ // created_at doesn't change between rows for the same profile, but
+ // min() is safe and matches "first seen" semantics if there ever is a
+ // discrepancy.
+ sql`min(created_at) AS created_at`,
+ sql`max(p.last_seen_at) AS last_seen_at`,
+ ],
+ ', '
+);
export class ProfileBuffer extends BaseBuffer {
private readonly batchSize = process.env.PROFILE_BUFFER_BATCH_SIZE
@@ -202,12 +205,13 @@ export class ProfileBuffer extends BaseBuffer {
withDateFilter: boolean
) => {
for (const chunk of this.chunks(group, this.fetchChunkSize)) {
- const tuples = chunk
- .map(
+ const tuples = sql.join(
+ chunk.map(
(p) =>
- `(${sqlstring.escape(String(p.id))}, ${sqlstring.escape(p.project_id)})`
- )
- .join(', ');
+ sql`(${sql.string(String(p.id))}, ${sql.string(p.project_id)})`
+ ),
+ ', '
+ );
try {
// Table alias `p` is required: without it, WHERE's `last_seen_at`
// resolves to the SELECT-list aggregate alias `max(last_seen_at) AS
@@ -215,10 +219,10 @@ export class ProfileBuffer extends BaseBuffer {
// (CH ILLEGAL_AGGREGATION). Qualifying with `p.` bypasses the alias
// lookup and binds to the raw column.
const rows = await this.chQuery<IClickhouseProfile>(
- `SELECT ${PROFILE_LATEST_AGGREGATE_COLUMNS}
- FROM ${TABLE_NAMES.profiles} AS p
+ sql`SELECT ${PROFILE_LATEST_AGGREGATE_COLUMNS}
+ FROM ${sql.id(TABLE_NAMES.profiles)} AS p
WHERE (p.id, p.project_id) IN (${tuples})
- ${withDateFilter ? 'AND p.last_seen_at > now() - INTERVAL 2 DAY' : ''}
+ ${withDateFilter ? sql`AND p.last_seen_at > now() - INTERVAL 2 DAY` : sql.empty}
GROUP BY p.id, p.project_id`
);
for (const row of rows) {
diff --git a/packages/db/index.ts b/packages/db/index.ts
index 22cb91d5e..310cb433a 100644
--- a/packages/db/index.ts
+++ b/packages/db/index.ts
@@ -1,10 +1,4 @@
export * from './src/clickhouse/client';
-// `clix` (query-builder.ts) and `createSqlBuilder` (sql-builder.ts) are dead
-// per ADR-013, but 7 live call sites across @openpanel/core still import
-// them directly and have not been converted onto the `sql` tag yet — see
-// M9-CLEANUP-001's report. Deleting these two ahead of that conversion is
-// BLOCKED, not done.
-export * from './src/clickhouse/query-builder';
+export * from './src/clickhouse/sql';
export * from './src/prisma-client';
-export * from './src/sql-builder';
export * from './src/types';
diff --git a/packages/db/package.json b/packages/db/package.json
index 32e591824..431c09b31 100644
--- a/packages/db/package.json
+++ b/packages/db/package.json
@@ -25,13 +25,11 @@
"pino": "catalog:",
"pino-pretty": "catalog:",
"prisma-json-types-generator": "^3.1.1",
- "sqlstring": "^2.3.3",
"zod": "catalog:"
},
"devDependencies": {
"@openpanel/tsconfig": "workspace:*",
"@types/node": "catalog:",
- "@types/sqlstring": "^2.3.2",
"prisma": "^6.14.0",
"typescript": "catalog:"
}
diff --git a/packages/db/scripts/ch-copy-from-remote.ts b/packages/db/scripts/ch-copy-from-remote.ts
index 18e05445e..3b8ad98c2 100644
--- a/packages/db/scripts/ch-copy-from-remote.ts
+++ b/packages/db/scripts/ch-copy-from-remote.ts
@@ -1,9 +1,8 @@
import { stdin as input, stdout as output } from 'node:process';
import { createInterface } from 'node:readline/promises';
import { parseArgs } from 'node:util';
-import sqlstring from 'sqlstring';
import { ch } from '../src/clickhouse/client';
-import { clix } from '../src/clickhouse/query-builder';
+import { sql } from '../src/clickhouse/sql';
async function main() {
const rl = createInterface({ input, output });
@@ -31,14 +30,20 @@ async function main() {
const host =
getArg(values.host) || (await rl.question('Remote Host (IP/Domain): '));
- if (!host) throw new Error('Host is required');
+ if (!host) {
+ throw new Error('Host is required');
+ }
const user = getArg(values.user) || (await rl.question('Remote User: '));
- if (!user) throw new Error('User is required');
+ if (!user) {
+ throw new Error('User is required');
+ }
const password =
getArg(values.password) || (await rl.question('Remote Password: '));
- if (!password) throw new Error('Password is required');
+ if (!password) {
+ throw new Error('Password is required');
+ }
const dbName =
getArg(values.db) ||
@@ -48,17 +53,21 @@ async function main() {
const startDate =
getArg(values.start) ||
(await rl.question('Start Date (YYYY-MM-DD HH:mm:ss): '));
- if (!startDate) throw new Error('Start date is required');
+ if (!startDate) {
+ throw new Error('Start date is required');
+ }
const endDate =
getArg(values.end) ||
(await rl.question('End Date (YYYY-MM-DD HH:mm:ss): '));
- if (!endDate) throw new Error('End date is required');
+ if (!endDate) {
+ throw new Error('End date is required');
+ }
const projectIdsInput =
getArg(values.projects) ||
(await rl.question(
- 'Project IDs (comma separated, leave empty for all): ',
+ 'Project IDs (comma separated, leave empty for all): '
));
const projectIds = projectIdsInput
? projectIdsInput.split(',').map((s: string) => s.trim())
@@ -71,27 +80,29 @@ async function main() {
for (const table of tables) {
console.log(`Processing table: ${table}`);
- // Build the SELECT part using the query builder
- // We use sqlstring to escape the remote function arguments
- const remoteTable = `remote(${sqlstring.escape(host)}, ${sqlstring.escape(dbName)}, ${sqlstring.escape(table)}, ${sqlstring.escape(user)}, ${sqlstring.escape(password)})`;
+ // Every remote() argument is a bound param, so the credentials never
+ // enter the query text — which is what makes the log line below safe to
+ // print. `dbName` and `table` are identifiers: bound as {x:Identifier}
+ // inside remote(), validated by sql.id() in the INSERT target, where a
+ // param is not accepted.
+ const remoteTable = sql`remote(${sql.string(host)}, ${sql.identifier(dbName)}, ${sql.identifier(table)}, ${sql.string(user)}, ${sql.string(password)})`;
- const queryBuilder = clix(ch)
- .from(remoteTable)
- .select(['*'])
- .where('created_at', 'BETWEEN', [startDate, endDate]);
+ const projectFilter =
+ projectIds.length > 0
+ ? sql` AND project_id IN ${sql.array('String', projectIds)}`
+ : sql.empty;
- if (projectIds.length > 0) {
- queryBuilder.where('project_id', 'IN', projectIds);
- }
+ const selectQuery = sql`SELECT * FROM ${remoteTable} WHERE created_at BETWEEN ${sql.string(startDate)} AND ${sql.string(endDate)}${projectFilter}`;
- const selectQuery = queryBuilder.toSQL();
- const insertQuery = `INSERT INTO ${dbName}.${table} ${selectQuery}`;
+ const insertQuery =
+ sql`INSERT INTO ${sql.id(dbName)}.${sql.id(table)} ${selectQuery}`.toStatement();
- console.log(`Executing: ${insertQuery}`);
+ console.log(`Executing: ${insertQuery.query}`);
// try {
// await ch.command({
- // query: insertQuery,
+ // query: insertQuery.query,
+ // query_params: insertQuery.query_params,
// });
// console.log(`✅ Copied ${table} successfully`);
// } catch (error) {
diff --git a/packages/db/scripts/ch-update-sessions-with-revenue.ts b/packages/db/scripts/ch-update-sessions-with-revenue.ts
index 93ecb22eb..76ca68613 100644
--- a/packages/db/scripts/ch-update-sessions-with-revenue.ts
+++ b/packages/db/scripts/ch-update-sessions-with-revenue.ts
@@ -1,13 +1,22 @@
-import { TABLE_NAMES, ch } from '../src/clickhouse/client';
-import { clix } from '../src/clickhouse/query-builder';
+// Local fixture script: give a couple of sessions per hour a random revenue.
+//
+// M12-009 took it off `clix` (deleted with ADR-013 decision 21) and onto the
+// `sql` tag. Both statements bind every value as a `{pN:Type}` param instead
+// of splicing it into the text, which is also what removes the hand-rolled
+// `'${id}'` quoting the ALTER used to do.
+import { ch, chQuery, TABLE_NAMES } from '../src/clickhouse/client';
+import { sql } from '../src/clickhouse/sql';
const START_DATE = new Date('2025-11-10T00:00:00Z');
const END_DATE = new Date('2025-11-20T23:00:00Z');
const SESSIONS_PER_HOUR = 2;
+const PROJECT_ID = 'public-web';
+const MS_PER_HOUR = 60 * 60 * 1000;
+const PAUSE_BETWEEN_MUTATIONS_MS = 500;
// Revenue between $10 (1000 cents) and $200 (20000 cents)
const MIN_REVENUE = 1000;
-const MAX_REVENUE = 20000;
+const MAX_REVENUE = 20_000;
function getRandomRevenue() {
return (
@@ -15,26 +24,30 @@ function getRandomRevenue() {
);
}
+/** `YYYY-MM-DD HH:mm:ss` — the literal form clix escaped a `Date` to. */
+function datetime(date: Date): string {
+ return date.toISOString().slice(0, 19).replace('T', ' ');
+}
+
async function main() {
console.log(
- `Starting revenue update for sessions between ${START_DATE.toISOString()} and ${END_DATE.toISOString()}`,
+ `Starting revenue update for sessions between ${START_DATE.toISOString()} and ${END_DATE.toISOString()}`
);
let currentDate = new Date(START_DATE);
while (currentDate < END_DATE) {
- const nextHour = new Date(currentDate.getTime() + 60 * 60 * 1000);
+ const nextHour = new Date(currentDate.getTime() + MS_PER_HOUR);
console.log(`Processing hour: ${currentDate.toISOString()}`);
// 1. Pick random sessions for this hour
- const sessions = await clix(ch)
- .from(TABLE_NAMES.sessions)
- .select(['id'])
- .where('created_at', '>=', currentDate)
- .andWhere('created_at', '<', nextHour)
- .where('project_id', '=', 'public-web')
- .limit(SESSIONS_PER_HOUR)
- .execute();
+ const sessions = await chQuery<{ id: string }>(sql`
+ SELECT id
+ FROM ${sql.id(TABLE_NAMES.sessions)}
+ WHERE created_at >= ${sql.string(datetime(currentDate))}
+ AND created_at < ${sql.string(datetime(nextHour))}
+ AND project_id = ${sql.string(PROJECT_ID)}
+ LIMIT ${sql.uint64(SESSIONS_PER_HOUR)}`);
if (sessions.length === 0) {
console.log(`No sessions found for ${currentDate.toISOString()}`);
@@ -42,44 +55,42 @@ async function main() {
continue;
}
- const sessionIds = sessions.map((s: any) => s.id);
+ const sessionIds = sessions.map((s) => s.id);
console.log(
- `Found ${sessionIds.length} sessions to update: ${sessionIds.join(', ')}`,
+ `Found ${sessionIds.length} sessions to update: ${sessionIds.join(', ')}`
);
- // 2. Construct update query
- // We want to assign a DIFFERENT random revenue to each session
- // Query: ALTER TABLE sessions UPDATE revenue = if(id='id1', rev1, if(id='id2', rev2, ...)) WHERE id IN ('id1', 'id2', ...)
-
- const updates: { id: string; revenue: number }[] = [];
-
- for (const id of sessionIds) {
- const revenue = getRandomRevenue();
- updates.push({ id, revenue });
- }
-
- // Build nested if() for the update expression
- // ClickHouse doesn't have CASE WHEN in UPDATE expression in the same way, but if() works.
- // Actually multiIf is cleaner: multiIf(id='id1', rev1, id='id2', rev2, revenue)
+ // 2. Assign a DIFFERENT random revenue to each session. ClickHouse has no
+ // CASE WHEN in an UPDATE expression, so multiIf carries the mapping and
+ // falls back to the existing `revenue`.
+ const updates = sessionIds.map((id) => ({
+ id,
+ revenue: getRandomRevenue(),
+ }));
- const conditions = updates
- .map((u) => `id = '${u.id}', ${u.revenue}`)
- .join(', ');
- const updateExpr = `multiIf(${conditions}, revenue)`;
+ const updateExpr = sql`multiIf(${sql.join(
+ updates.map(
+ (u) => sql`id = ${sql.string(u.id)}, ${sql.float64(u.revenue)}`
+ ),
+ ', '
+ )}, revenue)`;
- const idsStr = sessionIds.map((id: string) => `'${id}'`).join(', ');
- const query = `ALTER TABLE ${TABLE_NAMES.sessions} UPDATE revenue = ${updateExpr} WHERE id IN (${idsStr})`;
+ const statement =
+ sql`ALTER TABLE ${sql.id(TABLE_NAMES.sessions)} UPDATE revenue = ${updateExpr} WHERE id IN ${sql.array('String', sessionIds)}`.toStatement();
- console.log(`Executing update: ${query}`);
+ console.log(`Executing update: ${statement.query}`);
try {
await ch.command({
- query,
+ query: statement.query,
+ query_params: statement.query_params,
});
console.log('Update command sent.');
// Wait a bit to not overload mutations if running on a large range
- await new Promise((resolve) => setTimeout(resolve, 500));
+ await new Promise((resolve) =>
+ setTimeout(resolve, PAUSE_BETWEEN_MUTATIONS_MS)
+ );
} catch (error) {
console.error('Failed to update sessions:', error);
}
diff --git a/packages/db/scripts/scripts.sql.proof.md b/packages/db/scripts/scripts.sql.proof.md
new file mode 100644
index 000000000..24c3bfbfe
--- /dev/null
+++ b/packages/db/scripts/scripts.sql.proof.md
@@ -0,0 +1,119 @@
+# `packages/db/scripts` off `clix` / `sqlstring` — result-set proof (M12-009)
+
+M12-009 deletes `query-builder.ts`, `sql-builder.ts` and the `sqlstring`
+dependency. Two operator scripts in this directory were the last things in
+`packages/db` importing either, so they move onto the `sql` tag in the same
+commit. (The gate `tooling/gates/p12-grep-gates.sh` does not scan `scripts/`;
+these two had to convert anyway because removing `sqlstring` from
+`packages/db/package.json` removes the module they imported.)
+
+V1 is each script's text at `30d236fa`; V2 is the tagged statement in this
+commit. Both were rendered in one Bun process and executed through the same
+`@clickhouse/client` — V1 as `query`, V2 as `query` + `query_params` — and the
+result sets (or, for the writes, the table read back afterwards) compared field
+for field.
+
+- **Date**: 2026-09-07, run by ralph on this box.
+- **Database**: a throwaway `m12009_scratch`, seeded per case and dropped after
+ the run. Both scripts are writers (`ALTER TABLE … UPDATE`, `INSERT … SELECT`),
+ so neither could be pointed at the prod-copy `openpanel` or at
+ `openpanel_test`.
+- **Machine**: single-node ClickHouse 26.1.3.52, 4 vCPU. Timings directional
+ only; production is 2 shards × 2 replicas (`docs/ENVIRONMENT.md`).
+- **Verdict**: **4 statements, all IDENTICAL.**
+
+| # | Script | Statement | Rows | old → new |
+|---|---|---|---:|---|
+| 1 | `ch-update-sessions-with-revenue.ts:45` | `SELECT id FROM sessions … LIMIT {n}` | 2 | 5ms → 5ms |
+| 2 | `ch-update-sessions-with-revenue.ts:79` | `ALTER TABLE … UPDATE revenue = multiIf(…)` | 4 read back | 11ms → 14ms |
+| 3 | `ch-copy-from-remote.ts:95` | `SELECT * FROM remote(…)` | 2 | 7ms → 9ms |
+| 4 | `ch-copy-from-remote.ts:97` | `INSERT INTO db.table SELECT * FROM remote(…)` | 3 → 5 | 9ms → 9ms |
+
+## What binding buys here, concretely
+
+`remote()` accepts bound parameters in every argument position — `{p:String}`
+for the host, user and password, `{p:Identifier}` for the database and table.
+That was verified before the conversion was written (case 3 above) and it is
+the whole point: `ch-copy-from-remote.ts` **logs the query it is about to run**,
+and V1's text carried the remote password in it. It no longer does — the
+`--- params ---` line in the run below is the only place the credential
+appears, and the script never prints it.
+
+The `INSERT INTO <db>.<table>` target is the one position a `{x:Identifier}`
+param is not accepted, so it goes through `sql.id()`, which throws on anything
+that is not a bare identifier rather than falling back to interpolation
+(ADR-013 R3).
+
+## Type choices
+
+- `revenue` is `Float64` (`system.columns`), so the `multiIf` branches bind as
+ `sql.float64` — the type ClickHouse infers for V1's bare integer literal in
+ that expression, not a wider guess.
+- `LIMIT` takes `sql.uint64`.
+- The two date bounds bind as `String`, exactly the form V1 quoted, so
+ `date_time_input_format: 'best_effort'` parses them identically.
+
+## IN / GLOBAL IN
+
+`WHERE id IN ('a','b')` became `WHERE id IN {p:Array(String)}`. Both are a
+**literal set**, not a subquery, so they evaluate identically on every shard and
+`docs/ENVIRONMENT.md`'s `IN` vs `GLOBAL IN` trap does not apply. No `IN` was
+converted in either direction. `remote()` is a table function evaluated on the
+initiator, unchanged by the rewrite.
+
+## Run output
+
+```
+==============================================================================
+CASE: ch-update-sessions-with-revenue: SELECT id FROM sessions [db=m12009_scratch]
+--- OLD ---
+SELECT id FROM sessions WHERE created_at >= '2025-11-10 00:00:00' AND created_at < '2025-11-10 01:00:00' AND project_id = 'public-web' LIMIT 2
+--- NEW ---
+SELECT id
+ FROM sessions
+ WHERE created_at >= {p1:String}
+ AND created_at < {p2:String}
+ AND project_id = {p3:String}
+ LIMIT {p4:UInt64}
+--- params ---
+{"p1":"2025-11-10 00:00:00","p2":"2025-11-10 01:00:00","p3":"public-web","p4":2}
+old: ok=true 5ms
+new: ok=true 5ms
+old: {"meta":[{"name":"id","type":"String"}],"data":[["sess-a"],["sess-b"]]}
+new: {"meta":[{"name":"id","type":"String"}],"data":[["sess-a"],["sess-b"]]}
+RESULT SETS: IDENTICAL
+==============================================================================
+CASE: ch-update-sessions-with-revenue: ALTER TABLE ... UPDATE multiIf [db=m12009_scratch]
+--- OLD ---
+ALTER TABLE sessions UPDATE revenue = multiIf(id = 'sess-a', 1500, id = 'sess-b', 20000, revenue) WHERE id IN ('sess-a', 'sess-b')
+--- NEW ---
+ALTER TABLE sessions UPDATE revenue = multiIf(id = {p1:String}, {p2:Float64}, id = {p3:String}, {p4:Float64}, revenue) WHERE id IN {p5:Array(String)}
+--- params ---
+{"p1":"sess-a","p2":1500,"p3":"sess-b","p4":20000,"p5":["sess-a","sess-b"]}
+old: ok=true 11ms
+new: ok=true 14ms
+old rows after: {"meta":[{"name":"id","type":"String"},{"name":"revenue","type":"Float64"}],"data":[["sess-a",1500],["sess-b",20000],["sess-c",0],["sess-d",0]]}
+new rows after: {"meta":[{"name":"id","type":"String"},{"name":"revenue","type":"Float64"}],"data":[["sess-a",1500],["sess-b",20000],["sess-c",0],["sess-d",0]]}
+RESULT SETS: IDENTICAL
+==============================================================================
+CASE: ch-copy-from-remote: INSERT INTO <db>.<table> SELECT * FROM remote(...) [db=m12009_scratch]
+--- OLD SELECT ---
+SELECT * FROM remote('127.0.0.1:9000', 'm12009_scratch', 'copy_target', 'default', '') WHERE created_at BETWEEN '2025-11-10 00:00:00' AND '2025-11-11 00:00:00' AND project_id IN ('p1', 'p2')
+--- NEW SELECT ---
+SELECT * FROM remote({p1:String}, {p2:Identifier}, {p3:Identifier}, {p4:String}, {p5:String}) WHERE created_at BETWEEN {p6:String} AND {p7:String} AND project_id IN {p8:Array(String)}
+--- params (note: credentials are NOT in the text) ---
+{"p1":"127.0.0.1:9000","p2":"m12009_scratch","p3":"copy_target","p4":"default","p5":"","p6":"2025-11-10 00:00:00","p7":"2025-11-11 00:00:00","p8":["p1","p2"]}
+old: ok=true 7ms
+new: ok=true 9ms
+old: {"meta":[{"name":"project_id","type":"String"},{"name":"created_at","type":"DateTime64(3)"},{"name":"name","type":"String"}],"data":[["p1","2025-11-10 00:05:00.000","a"],["p2","2025-11-10 00:06:00.000","b"]]}
+new: {"meta":[{"name":"project_id","type":"String"},{"name":"created_at","type":"DateTime64(3)"},{"name":"name","type":"String"}],"data":[["p1","2025-11-10 00:05:00.000","a"],["p2","2025-11-10 00:06:00.000","b"]]}
+SELECT RESULT SETS: IDENTICAL
+--- OLD INSERT ---
+INSERT INTO m12009_scratch.copy_target SELECT * FROM remote('127.0.0.1:9000', 'm12009_scratch', 'copy_target', 'default', '') WHERE created_at BETWEEN '2025-11-10 00:00:00' AND '2025-11-11 00:00:00' AND project_id IN ('p1', 'p2')
+--- NEW INSERT ---
+INSERT INTO m12009_scratch.copy_target SELECT * FROM remote({p1:String}, {p2:Identifier}, {p3:Identifier}, {p4:String}, {p5:String}) WHERE created_at BETWEEN {p6:String} AND {p7:String} AND project_id IN {p8:Array(String)}
+old INSERT: ok=true 9ms
+new INSERT: ok=true 9ms
+count before: {"meta":[{"name":"count()","type":"UInt64"}],"data":[[3]]} after old: {"meta":[{"name":"count()","type":"UInt64"}],"data":[[5]]} after new: {"meta":[{"name":"count()","type":"UInt64"}],"data":[[5]]}
+INSERT EFFECT: IDENTICAL
+```
diff --git a/packages/db/src/clickhouse/client.ts b/packages/db/src/clickhouse/client.ts
index 12c4ff23b..70b4675c3 100644
--- a/packages/db/src/clickhouse/client.ts
+++ b/packages/db/src/clickhouse/client.ts
@@ -5,8 +5,6 @@ import type {
} from '@clickhouse/client';
import { ClickHouseLogLevel, createClient } from '@clickhouse/client';
import type { NodeClickHouseClientConfigOptions } from '@clickhouse/client/dist/config';
-import type { IInterval } from '@openpanel/core/modules/report/report.constants';
-import sqlstring from 'sqlstring';
import { createLogger } from '../logger';
import { RoundRobinPicker, withRoundRobinRetry } from './round-robin';
import { type SqlFragment, toStatement } from './sql';
@@ -407,23 +405,6 @@ export function formatClickhouseDate(
.replace(/(\.\d{3})?Z+$/, '');
}
-export function toDate(str: string, interval?: IInterval) {
- // If it does not match the regex it's a column name eg 'created_at'
- if (!interval || interval === 'minute' || interval === 'hour') {
- if (str.match(/\d{4}-\d{2}-\d{2}/)) {
- return sqlstring.escape(str);
- }
-
- return str;
- }
-
- if (str.match(/\d{4}-\d{2}-\d{2}/)) {
- return `toDate(${sqlstring.escape(str.split(' ')[0])})`;
- }
-
- return `toDate(${str})`;
-}
-
export function convertClickhouseDateToJs(date: string) {
return new Date(`${date.replace(' ', 'T')}Z`);
}
diff --git a/packages/db/src/clickhouse/query-builder.test.ts b/packages/db/src/clickhouse/query-builder.test.ts
deleted file mode 100644
index 17e47393a..000000000
--- a/packages/db/src/clickhouse/query-builder.test.ts
+++ /dev/null
@@ -1,88 +0,0 @@
-import type { ClickHouseClient } from '@clickhouse/client';
-import { describe, expect, it } from 'vitest';
-import { clix } from './query-builder';
-
-const fakeClient = {} as ClickHouseClient;
-
-const buildWhereSQL = (column: string, op: any, value: any) =>
- clix(fakeClient)
- .select(['*'])
- .from('events')
- .where(column, op, value)
- .toSQL();
-
-describe('query-builder escapeValue', () => {
- it('fully escapes user paths that contain a date substring', () => {
- const sql = buildWhereSQL(
- 'e.path',
- '=',
- '/blog/2024-03-17-supabase-activity-scheduler',
- );
-
- expect(sql).toContain(
- "e.path = '/blog/2024-03-17-supabase-activity-scheduler'",
- );
- // Make sure the date is NOT quoted independently (the original bug).
- expect(sql).not.toContain("/blog/'2024-03-17'");
- });
-
- it('does not allow SQL injection via a date-substring payload', () => {
- const sql = buildWhereSQL(
- 'e.path',
- '=',
- "/blog/2024-01-01' OR '1'='1",
- );
-
- // Single quotes in the value must be escaped, not break out of the string.
- expect(sql).toContain(
- "e.path = '/blog/2024-01-01\\' OR \\'1\\'=\\'1'",
- );
- expect(sql).not.toMatch(/OR '1'='1[^']/);
- });
-
- it('escapes a plain ISO date string', () => {
- const sql = buildWhereSQL('e.created_at', '=', '2026-04-29 00:00:00');
- expect(sql).toContain("e.created_at = '2026-04-29 00:00:00'");
- });
-
- it('keeps clix.datetime(date, "toDateTime") as a SQL fragment with the date quoted', () => {
- const sql = buildWhereSQL(
- 'e.created_at',
- '=',
- clix.datetime('2026-04-29T00:00:00.000Z', 'toDateTime'),
- );
- expect(sql).toContain("e.created_at = toDateTime('2026-04-29 00:00:00')");
- });
-
- it('handles BETWEEN with clix.datetime wrapped values', () => {
- const sql = clix(fakeClient)
- .select(['*'])
- .from('events')
- .where('e.created_at', 'BETWEEN', [
- clix.datetime('2026-04-29T00:00:00.000Z', 'toDateTime'),
- clix.datetime('2026-05-07T00:00:00.000Z', 'toDateTime'),
- ])
- .toSQL();
-
- expect(sql).toContain(
- "e.created_at BETWEEN toDateTime('2026-04-29 00:00:00') AND toDateTime('2026-05-07 00:00:00')",
- );
- });
-
- it('escapes Date objects as datetime literals', () => {
- const sql = buildWhereSQL(
- 'e.created_at',
- '=',
- new Date('2026-04-29T00:00:00.000Z'),
- );
- expect(sql).toContain("e.created_at = '2026-04-29 00:00:00'");
- });
-
- it('does not treat an arbitrary string with embedded date as raw SQL', () => {
- // Function-call-like prefix but not a real wrapper — ensures we don't
- // accidentally promote arbitrary text to raw SQL.
- const sql = buildWhereSQL('e.path', '=', 'event-2026-04-15-launch');
- expect(sql).toContain("e.path = 'event-2026-04-15-launch'");
- expect(sql).not.toContain("'2026-04-15'");
- });
-});
diff --git a/packages/db/src/clickhouse/query-builder.ts b/packages/db/src/clickhouse/query-builder.ts
deleted file mode 100644
index d42885852..000000000
--- a/packages/db/src/clickhouse/query-builder.ts
+++ /dev/null
@@ -1,791 +0,0 @@
-import type { ClickHouseClient, ResponseJSON } from '@clickhouse/client';
-import type { IInterval } from '@openpanel/core/modules/report/report.constants';
-import sqlstring from 'sqlstring';
-
-type SqlValue = string | number | boolean | Date | null | Expression;
-type SqlParam = SqlValue | SqlValue[];
-type Operator =
- | '='
- | '>'
- | '<'
- | '>='
- | '<='
- | '!='
- | 'IN'
- | 'NOT IN'
- | 'LIKE'
- | 'NOT LIKE'
- | 'IS NULL'
- | 'IS NOT NULL'
- | 'BETWEEN';
-
-type CTE = {
- name: string;
- query: Query | string;
-};
-
-type JoinType = 'INNER' | 'LEFT' | 'RIGHT' | 'FULL' | 'CROSS' | 'LEFT ANY';
-
-type WhereCondition = {
- condition: string;
- operator: 'AND' | 'OR';
- isGroup?: boolean;
-};
-
-type ConditionalCallback = (query: Query) => void;
-
-class Expression {
- constructor(private expression: string) {}
-
- toString() {
- return this.expression;
- }
-}
-
-export class Query<T = any> {
- private _select: (string | Expression)[] = [];
- private _except: string[] = [];
- private _from?: string | Expression;
- private _where: WhereCondition[] = [];
- private _groupBy: string[] = [];
- private _rollup = false;
- private _having: { condition: string; operator: 'AND' | 'OR' }[] = [];
- private _orderBy: {
- column: string;
- direction: 'ASC' | 'DESC';
- }[] = [];
- private _limit?: number;
- private _offset?: number;
- private _final = false;
- private _settings: Record<string, string> = {};
- private _ctes: CTE[] = [];
- private _joins: {
- type: JoinType;
- table: string | Expression | Query;
- condition: string;
- alias?: string;
- }[] = [];
- private _skipNext = false;
- private _rawJoins: string[] = [];
- private _fill?: {
- from: string | Date;
- to: string | Date;
- step: string;
- };
- private _transform?: Record<string, (item: T) => any>;
- private _union?: Query;
- private _dateRegex = /\d{4}-\d{2}-\d{2}([\s:\d.]+)?/g;
- private _dateValueRegex =
- /^(?:[a-zA-Z]\w*\()?\d{4}-\d{2}-\d{2}(?:[\s:\d.]+)?\)?$/;
- constructor(
- private client: ClickHouseClient,
- private timezone: string
- ) {}
-
- // Select methods
- select<U>(
- columns: (string | Expression | null | undefined | false)[],
- type: 'merge' | 'replace' = 'replace'
- ): Query<U> {
- if (this._skipNext) {
- return this as unknown as Query<U>;
- }
- if (type === 'merge') {
- this._select = [
- ...this._select,
- ...columns.filter((col): col is string | Expression => Boolean(col)),
- ];
- } else {
- this._select = columns.filter((col): col is string | Expression =>
- Boolean(col)
- );
- }
- return this as unknown as Query<U>;
- }
-
- except(columns: string[]): this {
- this._except = [...this._except, ...columns];
- return this;
- }
-
- rollup(): this {
- this._rollup = true;
- return this;
- }
-
- // From methods
- from(table: string | Expression, final = false): this {
- this._from = table;
- this._final = final;
- return this;
- }
-
- union(query: Query): this {
- this._union = query;
- return this;
- }
-
- // Where methods
- private escapeValue(value: SqlParam): string {
- if (value === null) {
- return 'NULL';
- }
- if (value instanceof Expression) {
- return `(${value.toString()})`;
- }
- if (Array.isArray(value)) {
- return `(${value.map((v) => this.escapeValue(v)).join(', ')})`;
- }
-
- if (
- (typeof value === 'string' && this._dateValueRegex.test(value)) ||
- value instanceof Date
- ) {
- return this.escapeDate(value);
- }
-
- return sqlstring.escape(value);
- }
-
- where(column: string, operator: Operator, value?: SqlParam): this {
- if (this._skipNext) {
- return this;
- }
- const condition = this.buildCondition(column, operator, value);
- this._where.push({ condition, operator: 'AND' });
- return this;
- }
-
- public buildCondition(
- column: string,
- operator: Operator,
- value?: SqlParam
- ): string {
- switch (operator) {
- case 'IS NULL':
- return `${column} IS NULL`;
- case 'IS NOT NULL':
- return `${column} IS NOT NULL`;
- case 'BETWEEN':
- if (Array.isArray(value) && value.length === 2) {
- return `${column} BETWEEN ${this.escapeValue(value[0]!)} AND ${this.escapeValue(value[1]!)}`;
- }
- throw new Error('BETWEEN operator requires an array of two values');
- case 'IN':
- case 'NOT IN':
- if (!(Array.isArray(value) || value instanceof Expression)) {
- throw new Error(`${operator} operator requires an array value`);
- }
- return `${column} ${operator} ${this.escapeValue(value)}`;
- default:
- return `${column} ${operator} ${this.escapeValue(value!)}`;
- }
- }
-
- andWhere(column: string, operator: Operator, value?: SqlParam): this {
- const condition = this.buildCondition(column, operator, value);
- this._where.push({ condition, operator: 'AND' });
- return this;
- }
-
- rawWhere(condition: string): this {
- if (condition) {
- this._where.push({ condition, operator: 'AND' });
- }
- return this;
- }
-
- orWhere(column: string, operator: Operator, value?: SqlParam): this {
- const condition = this.buildCondition(column, operator, value);
- this._where.push({ condition, operator: 'OR' });
- return this;
- }
-
- // Group by methods
- groupBy(columns: (string | null | undefined | false)[]): this {
- this._groupBy = columns.filter((col): col is string => Boolean(col));
- return this;
- }
-
- // Having methods
- having(column: string, operator: Operator, value: SqlParam): this {
- const condition = this.buildCondition(column, operator, value);
- this._having.push({ condition, operator: 'AND' });
- return this;
- }
-
- rawHaving(condition: string): this {
- if (condition) {
- this._having.push({ condition, operator: 'AND' });
- }
- return this;
- }
-
- andHaving(column: string, operator: Operator, value: SqlParam): this {
- const condition = this.buildCondition(column, operator, value);
- this._having.push({ condition, operator: 'AND' });
- return this;
- }
-
- orHaving(column: string, operator: Operator, value: SqlParam): this {
- const condition = this.buildCondition(column, operator, value);
- this._having.push({ condition, operator: 'OR' });
- return this;
- }
-
- // Order by methods
- orderBy(column: string, direction: 'ASC' | 'DESC' = 'ASC'): this {
- if (this._skipNext) {
- return this;
- }
- this._orderBy.push({ column, direction });
- return this;
- }
-
- // Limit and offset
- limit(limit?: number): this {
- if (limit !== undefined) {
- this._limit = limit;
- }
- return this;
- }
-
- offset(offset?: number): this {
- if (offset !== undefined) {
- this._offset = offset;
- }
- return this;
- }
-
- // Settings
- settings(settings: Record<string, string>): this {
- Object.assign(this._settings, settings);
- return this;
- }
-
- with(name: string, query: Query | string): this {
- this._ctes.push({ name, query });
- return this;
- }
-
- // Fill
- fill(
- from: string | Date | Expression,
- to: string | Date | Expression,
- step: string | Expression
- ): this {
- this._fill = {
- from:
- from instanceof Expression ? from.toString() : this.escapeDate(from),
- to: to instanceof Expression ? to.toString() : this.escapeDate(to),
- step: step instanceof Expression ? step.toString() : step,
- };
- return this;
- }
-
- private escapeDate(value: string | Date): string {
- if (value instanceof Date) {
- return sqlstring.escape(clix.datetime(value));
- }
-
- return value.replaceAll(this._dateRegex, (match) => {
- return sqlstring.escape(match);
- });
- }
-
- // Add join methods
- join(table: string | Expression, condition: string, alias?: string): this {
- return this.joinWithType('INNER', table, condition, alias);
- }
-
- innerJoin(
- table: string | Expression,
- condition: string,
- alias?: string
- ): this {
- return this.joinWithType('INNER', table, condition, alias);
- }
-
- leftJoin(
- table: string | Expression | Query,
- condition: string,
- alias?: string
- ): this {
- return this.joinWithType('LEFT', table, condition, alias);
- }
-
- leftAnyJoin(
- table: string | Expression | Query,
- condition: string,
- alias?: string
- ): this {
- return this.joinWithType('LEFT ANY', table, condition, alias);
- }
-
- rightJoin(
- table: string | Expression,
- condition: string,
- alias?: string
- ): this {
- return this.joinWithType('RIGHT', table, condition, alias);
- }
-
- fullJoin(
- table: string | Expression,
- condition: string,
- alias?: string
- ): this {
- return this.joinWithType('FULL', table, condition, alias);
- }
-
- crossJoin(table: string | Expression, alias?: string): this {
- return this.joinWithType('CROSS', table, '', alias);
- }
-
- rawJoin(sql: string): this {
- if (this._skipNext) {
- return this;
- }
- this._rawJoins.push(sql);
- return this;
- }
-
- private joinWithType(
- type: JoinType,
- table: string | Expression | Query,
- condition: string,
- alias?: string
- ): this {
- if (this._skipNext) {
- return this;
- }
- this._joins.push({
- type,
- table,
- condition: this.escapeDate(condition),
- alias,
- });
- return this;
- }
-
- // Add methods for grouping conditions
- whereGroup(): WhereGroupBuilder {
- return new WhereGroupBuilder(this, 'AND');
- }
-
- orWhereGroup(): WhereGroupBuilder {
- return new WhereGroupBuilder(this, 'OR');
- }
-
- // Update buildQuery method's WHERE section
- private buildWhereConditions(conditions: WhereCondition[]): string {
- return conditions
- .map((w, i) => {
- const condition = w.isGroup ? `(${w.condition})` : w.condition;
- return i === 0 ? condition : `${w.operator} ${condition}`;
- })
- .join(' ');
- }
-
- private buildQuery(): string {
- const parts: string[] = [];
-
- // Add WITH clause if CTEs exist
- if (this._ctes.length > 0) {
- const cteStatements = this._ctes.map((cte) => {
- const queryStr =
- typeof cte.query === 'string' ? cte.query : cte.query.toSQL();
- return `${cte.name} AS (${queryStr})`;
- });
- parts.push(`WITH ${cteStatements.join(', ')}`);
- }
-
- // SELECT
- if (this._select.length > 0) {
- parts.push(
- 'SELECT',
- this._select
- // Important: Expressions are treated as raw SQL; do not run escapeDate()
- // on them, otherwise any embedded date strings get double-escaped
- // (e.g. ''2025-12-16 23:59:59'') which ClickHouse rejects.
- .map((col) =>
- col instanceof Expression ? col.toString() : this.escapeDate(col)
- )
- .join(', ')
- );
- } else {
- parts.push('SELECT *');
- }
-
- if (this._except.length > 0) {
- parts.push('EXCEPT', `(${this._except.map(this.escapeDate).join(', ')})`);
- }
-
- // FROM
- if (this._from) {
- if (this._from instanceof Expression) {
- parts.push(`FROM (${this._from.toString()})`);
- } else {
- parts.push(`FROM ${this._from}${this._final ? ' FINAL' : ''}`);
- }
-
- // Add joins
- this._joins.forEach((join) => {
- const aliasClause = join.alias ? ` ${join.alias} ` : ' ';
- const conditionStr = join.condition ? `ON ${join.condition}` : '';
- parts.push(
- `${join.type} JOIN ${join.table instanceof Query ? `(${join.table.toSQL()})` : join.table instanceof Expression ? `(${join.table.toString()})` : join.table}${aliasClause}${conditionStr}`
- );
- });
- // Add raw joins (e.g. ARRAY JOIN)
- this._rawJoins.forEach((join) => {
- parts.push(join);
- });
- }
-
- // WHERE
- if (this._where.length > 0) {
- parts.push('WHERE', this.buildWhereConditions(this._where));
- }
-
- // GROUP BY
- if (this._groupBy.length > 0) {
- parts.push('GROUP BY', this._groupBy.join(', '));
- }
-
- if (this._rollup) {
- parts.push('WITH ROLLUP');
- }
-
- // HAVING
- if (this._having.length > 0) {
- const conditions = this._having.map((h, i) => {
- return i === 0 ? h.condition : `${h.operator} ${h.condition}`;
- });
- parts.push('HAVING', conditions.join(' '));
- }
-
- // ORDER BY
- if (this._orderBy.length > 0) {
- const orderBy = this._orderBy.map((o) => {
- const col = o.column;
- return `${col} ${o.direction}`;
- });
- parts.push('ORDER BY', orderBy.join(', '));
- }
-
- // Add FILL clause after ORDER BY
- if (this._fill) {
- const fromDate =
- this._fill.from instanceof Date
- ? clix.datetime(this._fill.from)
- : this._fill.from;
- const toDate =
- this._fill.to instanceof Date
- ? clix.datetime(this._fill.to)
- : this._fill.to;
-
- parts.push('WITH FILL');
- parts.push(`FROM ${fromDate}`);
- parts.push(`TO ${toDate}`);
- parts.push(`STEP ${this._fill.step}`);
- }
-
- // LIMIT & OFFSET
- if (this._limit !== undefined) {
- parts.push(`LIMIT ${this._limit}`);
- if (this._offset !== undefined) {
- parts.push(`OFFSET ${this._offset}`);
- }
- }
-
- // SETTINGS
- if (Object.keys(this._settings).length > 0) {
- const settings = Object.entries(this._settings)
- .map(([key, value]) => `${key} = ${value}`)
- .join(', ');
- parts.push(`SETTINGS ${settings}`);
- }
-
- if (this._union) {
- parts.push('UNION ALL', this._union.buildQuery());
- }
-
- return parts.join(' ');
- }
-
- transformJson<E extends ResponseJSON<any>>(json: E): E {
- const keys = Object.keys(json.data[0] || {});
- const response = {
- ...json,
- data: json.data.map((item) => {
- return keys.reduce((acc, key) => {
- const meta = json.meta?.find((m) => m.name === key);
- const transformer = this._transform?.[key];
-
- if (transformer) {
- return {
- ...acc,
- [key]: transformer(item),
- };
- }
-
- return {
- ...acc,
- [key]:
- item[key] && meta?.type.includes('Int')
- ? Number.parseFloat(item[key] as string)
- : item[key],
- };
- }, {} as T);
- }),
- };
- return response;
- }
-
- transform(transformations: Record<string, (item: T) => any>): this {
- this._transform = transformations;
- return this;
- }
-
- // Execution methods
- async execute(): Promise<T[]> {
- const query = this.buildQuery();
- // console.log(
- // 'query',
- // `${query.replaceAll('\n', ' ').replaceAll('\t', ' ').replaceAll('\r', ' ')} SETTINGS session_timezone = '${this.timezone}'`,
- // );
-
- const result = await this.client.query({
- query,
- clickhouse_settings: {
- session_timezone: this.timezone,
- },
- });
- const json = await result.json<T>();
- return this.transformJson(json).data;
- }
-
- // Debug methods
- toSQL(): string {
- return this.buildQuery();
- }
-
- // Add method to add where conditions (for internal use)
- _addWhereCondition(condition: WhereCondition): this {
- this._where.push(condition);
- return this;
- }
-
- if(condition: any): this {
- this._skipNext = !condition;
- return this;
- }
-
- endIf(): this {
- this._skipNext = false;
- return this;
- }
-
- // Add method for callback-style conditionals
- when(condition: boolean, callback?: ConditionalCallback): this {
- if (condition && callback) {
- callback(this);
- }
- return this;
- }
-
- clone(): Query<T> {
- return new Query(this.client, this.timezone).merge(this);
- }
-
- // Add merge method
- merge(query: Query): this {
- if (this._skipNext) {
- return this;
- }
-
- this._from = query._from;
-
- this._select = [...this._select, ...query._select];
-
- this._except = [...this._except, ...query._except];
-
- // Merge WHERE conditions
- this._where = [...this._where, ...query._where];
-
- // Merge CTEs
- this._ctes = [...this._ctes, ...query._ctes];
-
- // Merge JOINS
- this._joins = [...this._joins, ...query._joins];
- this._rawJoins = [...this._rawJoins, ...query._rawJoins];
-
- // Merge settings
- this._settings = { ...this._settings, ...query._settings };
-
- // Take the most restrictive LIMIT
- if (query._limit !== undefined) {
- this._limit =
- this._limit === undefined
- ? query._limit
- : Math.min(this._limit, query._limit);
- }
-
- // Merge ORDER BY
- this._orderBy = [...this._orderBy, ...query._orderBy];
-
- // Merge GROUP BY
- this._groupBy = [...this._groupBy, ...query._groupBy];
-
- // Merge HAVING conditions
- this._having = [...this._having, ...query._having];
-
- return this;
- }
-}
-
-// Add this new class for building where groups
-export class WhereGroupBuilder {
- private conditions: WhereCondition[] = [];
-
- constructor(
- private query: Query,
- private groupOperator: 'AND' | 'OR'
- ) {}
-
- where(column: string, operator: Operator, value?: SqlParam): this {
- const condition = this.query.buildCondition(column, operator, value);
- this.conditions.push({ condition, operator: 'AND' });
- return this;
- }
-
- andWhere(column: string, operator: Operator, value?: SqlParam): this {
- const condition = this.query.buildCondition(column, operator, value);
- this.conditions.push({ condition, operator: 'AND' });
- return this;
- }
-
- rawWhere(condition: string): this {
- this.conditions.push({ condition, operator: 'AND' });
- return this;
- }
-
- orWhere(column: string, operator: Operator, value?: SqlParam): this {
- const condition = this.query.buildCondition(column, operator, value);
- this.conditions.push({ condition, operator: 'OR' });
- return this;
- }
-
- end(): Query {
- const groupCondition = this.conditions
- .map((c, i) => (i === 0 ? c.condition : `${c.operator} ${c.condition}`))
- .join(' ');
-
- this.query._addWhereCondition({
- condition: groupCondition,
- operator: this.groupOperator,
- isGroup: true,
- });
-
- return this.query;
- }
-}
-
-// Helper function to create a new query
-export function clix(client: ClickHouseClient, timezone?: string): Query {
- return new Query(client, timezone ?? 'UTC');
-}
-
-clix.exp = (expr: string | Query<any>) =>
- new Expression(expr instanceof Query ? expr.toSQL() : expr);
-clix.date = (date: string | Date, wrapper?: string) => {
- const dateStr = new Date(date).toISOString().slice(0, 10);
- return wrapper ? `${wrapper}(${dateStr})` : dateStr;
-};
-clix.datetime = (date: string | Date, wrapper?: string) => {
- const datetime = new Date(date).toISOString().slice(0, 19).replace('T', ' ');
- return wrapper ? `${wrapper}(${datetime})` : datetime;
-};
-clix.dynamicDatetime = (date: string | Date, interval: IInterval) => {
- if (interval === 'month' || interval === 'week') {
- return clix.date(date);
- }
- return clix.datetime(date);
-};
-
-clix.toStartOf = (node: string, interval: IInterval, timezone?: string) => {
- switch (interval) {
- case 'minute': {
- return `toStartOfMinute(${node})`;
- }
- case 'hour': {
- return `toStartOfHour(${node})`;
- }
- case 'day': {
- return `toStartOfDay(${node})`;
- }
- case 'week': {
- return `toStartOfWeek(toDateTime(${node}))`;
- }
- case 'month': {
- return `toStartOfMonth(toDateTime(${node}))`;
- }
- }
-};
-clix.toStartOfInterval = (
- node: string,
- interval: IInterval,
- origin: string | Date
-) => {
- switch (interval) {
- case 'minute': {
- return `toStartOfInterval(toDateTime(${node}), INTERVAL 1 minute, toDateTime(${clix.datetime(origin)}))`;
- }
- case 'hour': {
- return `toStartOfInterval(toDateTime(${node}), INTERVAL 1 hour, toDateTime(${clix.datetime(origin)}))`;
- }
- case 'day': {
- return `toStartOfInterval(toDateTime(${node}), INTERVAL 1 day, toDateTime(${clix.datetime(origin)}))`;
- }
- case 'week': {
- return `toStartOfInterval(toDateTime(${node}), INTERVAL 1 week, toDateTime(${clix.datetime(origin)}))`;
- }
- case 'month': {
- return `toStartOfInterval(toDateTime(${node}), INTERVAL 1 month, toDateTime(${clix.datetime(origin)}))`;
- }
- }
-};
-clix.toInterval = (node: string, interval: IInterval) => {
- switch (interval) {
- case 'minute': {
- return `toIntervalMinute(${node})`;
- }
- case 'hour': {
- return `toIntervalHour(${node})`;
- }
- case 'day': {
- return `toIntervalDay(${node})`;
- }
- case 'week': {
- return `toIntervalWeek(${node})`;
- }
- case 'month': {
- return `toIntervalMonth(${node})`;
- }
- }
-};
-clix.toDate = (node: string, interval: IInterval) => {
- switch (interval) {
- case 'day':
- case 'week':
- case 'month': {
- return `toDate(${node})`;
- }
- default: {
- return `toDateTime(${node})`;
- }
- }
-};
-// Export types
-export type { SqlValue, SqlParam, Operator };
diff --git a/packages/db/src/clickhouse/sql.test.ts b/packages/db/src/clickhouse/sql.test.ts
index 91a65b5b5..08e315c05 100644
--- a/packages/db/src/clickhouse/sql.test.ts
+++ b/packages/db/src/clickhouse/sql.test.ts
@@ -247,6 +247,62 @@ describe('injection (ported from query-builder.test.ts)', () => {
expect(query).toBe('SELECT * FROM events WHERE e.path = {p1:String}');
expect(query_params.p1).toBe('{p1:String}');
});
+
+ // M12-009 completed the port: the four cases below were still only asserted
+ // against clix when query-builder.test.ts was deleted. Each keeps that
+ // test's payload; what changes is the mechanism the payload proves —
+ // clix asserted on the escaped literal it produced, the tag asserts the
+ // value never reaches the text at all.
+
+ it('binds a plain ISO date string instead of quoting it', () => {
+ const { query, query_params } =
+ sql`SELECT * FROM events WHERE e.created_at = ${sql.string('2026-04-29 00:00:00')}`.toStatement();
+
+ expect(query).toBe('SELECT * FROM events WHERE e.created_at = {p1:String}');
+ expect(query_params.p1).toBe('2026-04-29 00:00:00');
+ });
+
+ it('binds a Date as a datetime value, not a literal in the text', () => {
+ const { query, query_params } =
+ sql`SELECT * FROM events WHERE e.created_at = ${sql.dateTime64('2026-04-29 00:00:00.000')}`.toStatement();
+
+ expect(query).toBe(
+ 'SELECT * FROM events WHERE e.created_at = {p1:DateTime64(3)}'
+ );
+ expect(query_params.p1).toBe('2026-04-29 00:00:00.000');
+ });
+
+ it('keeps a toDateTime() wrapper as SQL text with the value bound', () => {
+ // clix.datetime(date, 'toDateTime') produced the call AND the quoted
+ // literal in one string. The wrapper is structure and stays raw; the
+ // value is a value and binds.
+ const { query, query_params } =
+ sql`SELECT * FROM events WHERE e.created_at = toDateTime(${sql.string('2026-04-29 00:00:00')})`.toStatement();
+
+ expect(query).toBe(
+ 'SELECT * FROM events WHERE e.created_at = toDateTime({p1:String})'
+ );
+ expect(query_params.p1).toBe('2026-04-29 00:00:00');
+ });
+
+ it('handles BETWEEN with two wrapped datetime values', () => {
+ const { query, query_params } =
+ sql`SELECT * FROM events WHERE e.created_at BETWEEN toDateTime(${sql.string('2026-04-29 00:00:00')}) AND toDateTime(${sql.string('2026-05-07 00:00:00')})`.toStatement();
+
+ expect(query).toBe(
+ 'SELECT * FROM events WHERE e.created_at BETWEEN toDateTime({p1:String}) AND toDateTime({p2:String})'
+ );
+ expect(query_params.p1).toBe('2026-04-29 00:00:00');
+ expect(query_params.p2).toBe('2026-05-07 00:00:00');
+ });
+
+ it('does not promote an arbitrary string with an embedded date to SQL', () => {
+ const { query, query_params } = bind('event-2026-04-15-launch');
+
+ expect(query).toBe('SELECT * FROM events WHERE e.path = {p1:String}');
+ expect(query).not.toContain('2026-04-15');
+ expect(query_params.p1).toBe('event-2026-04-15-launch');
+ });
});
describe('param constructors', () => {
diff --git a/packages/db/src/clickhouse/sql.ts b/packages/db/src/clickhouse/sql.ts
index b78b31648..32a70e673 100644
--- a/packages/db/src/clickhouse/sql.ts
+++ b/packages/db/src/clickhouse/sql.ts
@@ -18,9 +18,17 @@
* Placeholder names are assigned at render time from a single counter, so
* fragments nest and repeat without colliding (R2).
*
- * Not re-exported from the package barrel during the clix coexistence window:
- * `query-builder.ts` exports its own `SqlParam` type, and `export *` from both
- * is ambiguous. It joins the barrel when clix is deleted.
+ * Exported from the package barrel (M12-009). It could not be during the clix
+ * coexistence window — `query-builder.ts` exported its own `SqlParam` type and
+ * `export *` from both was ambiguous — and that file is now deleted.
+ *
+ * `packages/core` still imports this module by its deep path rather than
+ * through the barrel: the barrel also re-exports `clickhouse/client.ts` and
+ * `prisma-client.ts`, which construct a ClickHouse client array and a
+ * PrismaClient at module load, so a barrel import from core would acquire a
+ * second, request-scope-less client. `.dependency-cruiser.cjs`'s
+ * `core-uses-ctx-not-db-internals` encodes exactly that, exempting this file
+ * by path and not the barrel.
*/
/** Auto-generated placeholders are `{p1:Type}`, `{p2:Type}`, … */
diff --git a/packages/db/src/sql-builder.ts b/packages/db/src/sql-builder.ts
deleted file mode 100644
index d20d85dac..000000000
--- a/packages/db/src/sql-builder.ts
+++ /dev/null
@@ -1,94 +0,0 @@
-import { TABLE_NAMES } from './clickhouse/client';
-
-export interface SqlBuilderObject {
- where: Record<string, string>;
- having: Record<string, string>;
- select: Record<string, string>;
- groupBy: Record<string, string>;
- orderBy: Record<string, string>;
- from: string;
- joins: Record<string, string>;
- ctes: Record<string, string>;
- limit: number | undefined;
- offset: number | undefined;
- fill: string | undefined;
-}
-
-export function createSqlBuilder() {
- const join = (obj: Record<string, string> | string[], joiner: string) =>
- Object.values(obj).filter(Boolean).join(joiner);
-
- const sb: SqlBuilderObject = {
- where: {},
- from: `${TABLE_NAMES.events} e`,
- select: {},
- groupBy: {},
- orderBy: {},
- having: {},
- joins: {},
- ctes: {},
- limit: undefined,
- offset: undefined,
- fill: undefined,
- };
-
- const getWhere = () =>
- Object.keys(sb.where).length ? `WHERE ${join(sb.where, ' AND ')}` : '';
- const getHaving = () =>
- Object.keys(sb.having).length ? `HAVING ${join(sb.having, ' AND ')}` : '';
- const getFrom = () => `FROM ${sb.from}`;
- const getSelect = () =>
- `SELECT ${Object.keys(sb.select).length ? join(sb.select, ', ') : '*'}`;
- const getGroupBy = () =>
- Object.keys(sb.groupBy).length ? `GROUP BY ${join(sb.groupBy, ', ')}` : '';
- const getOrderBy = () =>
- Object.keys(sb.orderBy).length ? `ORDER BY ${join(sb.orderBy, ', ')}` : '';
- const getLimit = () => (sb.limit ? `LIMIT ${sb.limit}` : '');
- const getOffset = () => (sb.offset ? `OFFSET ${sb.offset}` : '');
- const getJoins = () =>
- Object.keys(sb.joins).length ? join(sb.joins, ' ') : '';
- const getFill = () => (sb.fill ? `WITH FILL ${sb.fill}` : '');
- const getWith = () => {
- const cteEntries = Object.entries(sb.ctes);
- if (cteEntries.length === 0) return '';
- const cteClauses = cteEntries.map(
- ([name, query]) => `${name} AS (${query})`,
- );
- return `WITH ${cteClauses.join(', ')} `;
- };
-
- return {
- sb,
- join,
- getWhere,
- getFrom,
- getSelect,
- getGroupBy,
- getOrderBy,
- getHaving,
- getJoins,
- getFill,
- getWith,
- with: (name: string, query: string) => {
- sb.ctes[name] = query;
- },
- getSql: () => {
- const sql = [
- getWith(),
- getSelect(),
- getFrom(),
- getJoins(),
- getWhere(),
- getGroupBy(),
- getHaving(),
- getOrderBy(),
- getLimit(),
- getOffset(),
- getFill(),
- ]
- .filter(Boolean)
- .join(' ');
- return sql;
- },
- };
-}
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index dea0301f8..d6829d42c 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -658,9 +658,6 @@ importers:
sonner:
specifier: ^1.4.0
- sqlstring:
- specifier: ^2.3.3
- version: 2.3.3
superjson:
specifier: ^2.2.2
version: 2.2.2
@@ -939,9 +936,6 @@ importers:
slugify:
specifier: ^1.6.6
version: 1.6.9
- sqlstring:
- specifier: ^2.3.3
- version: 2.3.3
superjson:
specifier: ^1.13.3
version: 1.13.3
@@ -982,9 +976,6 @@ importers:
'@types/ramda':
specifier: ^0.31.1
version: 0.31.1
- '@types/sqlstring':
- specifier: ^2.3.2
- version: 2.3.2
'@types/unzipper':
specifier: ^0.10.10
version: 0.10.11
@@ -1033,9 +1024,6 @@ importers:
prisma-json-types-generator:
specifier: ^3.1.1
- sqlstring:
- specifier: ^2.3.3
- version: 2.3.3
zod:
specifier: 'catalog:'
version: 4.3.6
@@ -1046,9 +1034,6 @@ importers:
'@types/node':
specifier: 'catalog:'
version: 24.10.1
- '@types/sqlstring':
- specifier: ^2.3.2
- version: 2.3.2
prisma:
specifier: ^6.14.0
@@ -9276,9 +9261,6 @@ packages:
resolution: {integrity: sha512-5JcVt1u5HDmlXkwOD2nslZVllBBc7HDuOICfiZah2Z0is8M8g+ddAEawbmd3VjedfDHBzxCaXLs07QEmb7y54g==}
- '@types/[email protected]':
- resolution: {integrity: sha512-lVRe4Iz9UNgiHelKVo8QlC8fb5nfY8+p+jNQNE+UVsuuVlQnWhyWmQ/wF5pE8Ys6TdjfVpqTG9O9i2vi6E0+Sg==}
-
resolution: {integrity: sha512-9aEbYZ3TbYMznPdcdr3SmIrLXwC/AKZXQeCf9Pgao5CKb8CyHuEX5jzWPTkvregvhRJHcpRO6BFoGW9ycaOkYw==}
@@ -16595,10 +16577,6 @@ packages:
resolution: {integrity: sha512-D9cPgkvLlV3t3IzL0D0YLvGA9Ahk4PcvVwUbN0dSGr1aP0Nrt4AEnTUbuGvquEC0mA64Gqt1fzirlRs5ibXx8g==}
- resolution: {integrity: sha512-qC9iz2FlN7DQl3+wjwn3802RTyjCx7sDvfQEXchwa6CWOx07/WVfh91gBmQ9fahw8snwGEWU3xGzOt4tFyHLxg==}
- engines: {node: '>= 0.6'}
-
resolution: {integrity: sha512-poPs1GuctQLpiJ/1Pb8e+5b5lju9hQU7wxJ6NkYVUw7ZZExeRoYwyiaOekal+rDZc99MO/J2y9+SGFpHBKRSpQ==}
engines: {node: '>=20.16.0'}
@@ -28827,8 +28805,6 @@ snapshots:
'@types/[email protected]': {}
- '@types/[email protected]': {}
-
'@types/[email protected]': {}
'@types/[email protected]': {}
@@ -38310,8 +38286,6 @@ snapshots:
-
dependencies:
cookie-es: 2.0.1