OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.
ralph(M12-009): DELETE the builders: query-builder.ts, its test and sql-builder.ts gone, sqlstring removed from every package.json, sql.ts joins the db barrel, p12-grep-gates --assert exits 0, goldens 137/137
ralph4925d2f7a082
diff --git a/apps/start/package.json b/apps/start/package.jsonindex f4a9b7765..2e79d55e6 100644--- a/apps/start/package.json+++ b/apps/start/package.json@@ -150,7 +150,6 @@"short-unique-id": "^5.0.3","slugify": "^1.6.6","sonner": "^1.4.0",- "sqlstring": "^2.3.3","superjson": "^2.2.2","tailwind-merge": "^3.3.1","tailwindcss": "^4.0.6",diff --git a/docs/TECH_DEBT.md b/docs/TECH_DEBT.mdindex b9a37e8d0..b7ef60b5b 100644--- a/docs/TECH_DEBT.md+++ b/docs/TECH_DEBT.md@@ -1154,3 +1154,188 @@ package imports it — the wire-constant types were inlined into `src/index.ts`in a pre-rewrite commit (`fix(sdk): inline validation types in dts...`) andthe manifest entry never followed. Safe to delete whenever that`package.json` is next touched.++### 2026-09-07 — M12-009, ADR-013 decision 21 closed: the builders are deleted and `sqlstring` is out of the repo++**RESOLVED: the "clix coexistence" exception.** ADR-013's P8 deletion had been+deferred since M7 and was last restated by `M9-CLEANUP-001` in+`packages/db/index.ts`:++> `clix` (query-builder.ts) and `createSqlBuilder` (sql-builder.ts) are dead+> per ADR-013, but 7 live call sites across @openpanel/core still import them+> directly and have not been converted onto the `sql` tag yet — see+> M9-CLEANUP-001's report. Deleting these two ahead of that conversion is+> BLOCKED, not done.++M12-002..008 emptied that import graph; M12-009 deleted the three files+(`git rm`), took the barrel comment with them, and removed `sqlstring` /+`@types/sqlstring` from every workspace that declared it (`packages/db`,+`packages/core`, `apps/start` — `apps/api` had already lost its copy). Both+sections of this register that cite `query-builder.ts` / `sql-builder.ts` as+exempt targets of `core-uses-ctx-not-db-internals` are now describing paths+that do not exist; `packages/db/src/clickhouse/sql.ts` is the only one left,+and it is also the one the rule keeps exempt.++**Final `bash tooling/gates/p12-grep-gates.sh --report`, ralph 2026-09-07:**++```+ sqlstring clix sql-builder file+----------------------------------------------------------------------------+----------------------------------------------------------------------------+ 0 0 0 TOTAL+```++`--assert` exits `0`: `OK: sqlstring = 0, clix = 0, sql-builder = 0; all three+definer files are gone.`++**The leftovers M12-008 disclosed, converted here.** The gate read+`sqlstring 16 / clix 1 / sql-builder 1` on clean `30d236fa` (measured by+stashing this task's tree and re-running `--report`), spread over eight files:+`apps/start/package.json` 1, `packages/core/package.json` 2, `group-buffer.ts`+3, `profile-backfill-buffer.ts` 3, `profile-buffer.ts` 2,+`packages/db/package.json` 2, `packages/db/src/clickhouse/client.ts` 3, and+`packages/db/index.ts` 1 clix + 1 sql-builder. The gate does not scan+`packages/db/scripts/`, but the two scripts there imported `clix` and+`sqlstring` and had to convert too — removing the dependency removes the module+they import.++Every changed statement was executed against local ClickHouse in **both** forms+through the same `@clickhouse/client` — V1 as `query`, V2 as `query` ++`query_params` — on the same data. **8 statements, all IDENTICAL.** Full SQL,+params, row counts and timings: `packages/core/src/buffers/buffers.sql.proof.md`+and `packages/db/scripts/scripts.sql.proof.md`.++| Site | Statement | Rows | old → new |+|---|---|---:|---|+| `core/src/buffers/group-buffer.ts:84` | `SELECT … FROM groups FINAL` | 1 | 33ms → 8ms |+| `core/src/buffers/profile-buffer.ts:222` | batch fetch, no date filter | 2 | 20ms → 20ms |+| `core/src/buffers/profile-buffer.ts:222` | batch fetch, 2-day filter | 0 | 9ms → 8ms |+| `core/src/buffers/profile-backfill-buffer.ts:96` | lightweight `UPDATE … CASE` | 3 read back | 7ms → 6ms |+| `db/scripts/ch-update-sessions-with-revenue.ts:45` | `SELECT id FROM sessions` | 2 | 5ms → 5ms |+| `db/scripts/ch-update-sessions-with-revenue.ts:79` | `ALTER TABLE … UPDATE multiIf` | 4 read back | 11ms → 14ms |+| `db/scripts/ch-copy-from-remote.ts:95` | `SELECT * FROM remote(…)` | 2 | 7ms → 9ms |+| `db/scripts/ch-copy-from-remote.ts:97` | `INSERT … SELECT * FROM remote(…)` | 3 → 5 | 9ms → 9ms |++Reads ran against the local prod-copy `openpanel`; the two mutations and the+`INSERT` ran against a throwaway `m12009_scratch` database, dropped afterwards+— never against `openpanel` or `openpanel_test`. `remote()` accepts bound+params (`{p:String}` for the host and credentials, `{p:Identifier}` for the+database and table), verified before the conversion was written, which is what+takes the remote password out of the query text the script logs. No `IN` was+converted to `GLOBAL IN` or back: every `IN` touched takes a literal set, not+a subquery, so `docs/ENVIRONMENT.md`'s distributed-`IN` trap does not apply.++Timings are single-node and directional only (ClickHouse 26.1.3.52, 4 vCPU);+production is 2 shards × 2 replicas.++`packages/db/src/clickhouse/client.ts`'s `toDate(str, interval?)` was deleted+rather than converted: it existed only to `sqlstring.escape` a date-shaped+string into query text, it had zero importers in the repo, and keeping it+would have kept a manual-escaping helper alive that ADR-013 forbids.++**`packages/db/src/clickhouse/sql.ts` joined the barrel**, as its own header+promised. `packages/core` still reaches it by its deep path in all 55 files:+the barrel also re-exports `clickhouse/client.ts` and `prisma-client.ts`,+which construct a ClickHouse client array and a `PrismaClient` at module load,+so a barrel import from core acquires a second, request-scope-less client —+which is exactly what `.dependency-cruiser.cjs`'s+`core-uses-ctx-not-db-internals` forbids (its comment: "prisma-client.ts,+clickhouse/client.ts, logger.ts **and the barrel** are NOT exempt"). Nothing+outside `packages/core` imported the deep path.++**Debt noticed, not fixed (outside this task's scope — the file is at the repo+root, not under any of this task's scope globs):** `.dependency-cruiser.cjs`'s+`core-uses-ctx-not-db-internals` still exempts+`^packages/db/src/clickhouse/query-builder\.ts$` and+`^packages/db/src/sql-builder\.ts$` in its `to.pathNot`, and its comment still+explains both. They are dead regex branches now — harmless (nothing can match+them), but the next task that touches that file should delete both lines and+the two sentences describing them.++#### `packages/core/src/modules/chart/src/compiled.ts` — kept, with its caller census++M12-003 kept `compiled.ts` and its reviewer checked the reason: neither filter+compiler (`filter-where.ts`, `table-filter-where.ts`) nor the field resolver+(`field-resolution.ts`'s `getSelectPropertyKey`) returns anything that crosses+`compiledText` any more — they all return `SqlFragment`s — so what is left are+the chart builders splicing a name they generated themselves.+`compiledTextWithProfileRefs` was deleted outright at that time. That still+holds at M12-009: `compiled.ts` has **29 `compiledText(` splice sites on 27+lines across 8 files**, so it+is NOT deleted, and this is the census the M12-009 criterion asks for.++`compiledText` stays a named export in one file rather than a `sql.raw()`+scattered across the module precisely so this census is a grep:++```+$ grep -rn "compiledText(" packages/core/src --include=*.ts | grep -v '/compiled.ts:'+```++| file:line | what it splices | where the text comes from |+|---|---|---|+| `chart/src/chart.sql.ts:175` | a CTE name in `WITH <name> AS (…)` | keys of the `ctes` record the builder itself assembles — the profile CTE name and `` `cohort-<id>` `` (below) |+| `chart/src/chart.sql.ts:274` | the `"profile.<field>"` CTE alias | `field`, validated on the same line by `sql.id(field, PROFILE_CTE_FIELDS)` against a closed column set (`id`,+`properties`, plus `PROFILE_CTE_SCALAR_FIELDS`) |+| `chart/src/chart.sql.ts:340` | `` `cohort-<id>` `` as a JOIN target | `cohortId`, through `assertCohortId` (below) |+| `chart/src/chart.sql.ts:434` | `label_<n>` breakdown alias | `\`label_${index + 1}\`` — a loop index |+| `chart/src/chart.sql.ts:435` | the same `label_<n>` as a GROUP BY key | same |+| `chart/src/chart.sql.ts:469` | the aggregate keyword (`sum`/`avg`/`max`/`min`) | `MATH_FUNCTION_BY_SEGMENT[event.segment]` — a 4-entry literal `Record` at `chart.sql.ts:58`; an unknown segment yields `undefined`, not the input |+| `chart/src/chart.sql.ts:580` | `label_<n>` in a window `PARTITION BY` | a loop index |+| `chart/src/funnel.sql.ts:78` | `, 'strict_increase'` — the `windowFunnel` mode | `STRICT_INCREASE_MODE`, a file constant; the only branch is an env flag |+| `chart/src/funnel.sql.ts:188` | `b_<index>` breakdown alias | a loop index |+| `chart/src/funnel.sql.ts:307` | `b_<index>` as a GROUP BY key | a loop index |+| `chart/src/funnel.sql.ts:366` | `b_<index>` in SELECT and GROUP BY | a loop index |+| `chart/src/funnel.sql.ts:391` | `trim(ifNull(toString(b_<n>), ''))` | a loop index inside a fixed template; the compared VALUE beside it binds as `sql.string` |+| `chart/src/conversion.sql.ts:117` | `b_<index>` outer-select alias | a loop index |+| `chart/conversion.service.ts:137` | `b_<index>` alias | a loop index |+| `chart/src/retention.sql.ts:170` | `toDate` / `toStartOfWeek` / `toStartOfMonth` | `SQL_START_OF[interval]`, a 5-entry `Record` keyed by the `IRetentionInterval` union |+| `chart/src/retention.sql.ts:171` | the `dateDiff` / `INTERVAL` unit | `SQL_INTERVAL[interval]`, same shape |+| `chart/src/retention.sql.ts:172` | `>=` vs `=` | `COUNT_CRITERIA[criteria]`, keyed by the `IRetentionCriteria` union |+| `chart/src/retention.sql.ts:200` | `interval_<n>_users` alias | `String(index)` over `range(0..diffInterval)` |+| `chart/src/retention.sql.ts:206` | `interval_<n>_users` / `_user_count` | same |+| `chart/src/sankey.sql.ts:28` | the `arrayFilter(…) as events_deduped` expression | a file-level constant, no interpolation |+| `chart/src/sankey.sql.ts:38` | the `if(arrayFirstIndex(…))` truncation | a file-level constant, no interpolation |+| `chart/src/sankey.sql.ts:48` | the `arrayJoin(arrayMap(…))` transition subselect | a file-level constant, no interpolation |+| `chart/src/sankey.sql.ts:187` | `arraySlice(events, start_index, …)` | a file-level constant, no interpolation |+| `chart/src/sankey.sql.ts:201` (×2) | the base CTE name | `transitionsFrom`'s two callers pass the string literals `'session_paths_base'` and `'between_sessions'` |+| `overview/src/overview.sql.ts:115` | the `WITH FILL … STEP toInterval<Unit>(1)` step | `toIntervalStep(interval)`, a `switch` over the `IInterval` union that throws on anything else |+| `chart/src/field-resolution.ts:203` | `` `cohort-<id>` `` CTE name | `cohortId` — see below |+| `chart/src/field-resolution.ts:505` | `` `profile.properties.<key>` `` CTE alias | `key` — see below |++**Nothing above is a value, and nothing above is a compiler's output.** Every+row is an alias, a CTE name, a SQL keyword or a static expression. The two+compilers' and the resolver's outputs reach the statement as `SqlFragment`s and+never pass through `compiledText`.++**Two rows are name-shaped text derived from a user-chosen NAME, and are+flagged rather than buried** — they are the same two M12-003's reviewer read,+and both are guarded so that `compiledText` receives only a validated+identifier, never the raw input:++- **`` `cohort-<id>` ``** (`field-resolution.ts:203`, `chart.sql.ts:340`,+ and the `ctes` key at `chart.sql.ts:175`). `cohortId` comes from a saved+ report's breakdown config. `chart.sql.ts:119`'s `assertCohortId` tests it+ against `/^[A-Za-z0-9_-]+$/` and **throws `ChartCohortIdError`** otherwise —+ it never falls back to inlining, which is what ADR-013 R3 requires of an+ identifier path. The name needs the text seam because a dash makes `sql.id`+ reject it and `{x:Identifier}` is not accepted in a `WITH … AS` position.+ `field-resolution.ts:202`'s `getCohortCteName` has no in-repo caller today+ beyond the two barrel re-exports (`chart.service.ts:133`, `index.ts:338`);+ `chart.sql.ts` builds the same name behind `assertCohortId` itself.+- **`` `profile.properties.<key>` ``** (`field-resolution.ts:505`). `key` is+ the tail of a `profile.properties.<key>` field reference in the chart config.+ `collectProfilePropertyKeys` (`field-resolution.ts:465`, guard at `:484`)+ **rejects any key+ containing a backtick or a backslash** — the only two characters that are+ special inside a backtick-quoted ClickHouse identifier — and such keys are+ simply not narrowed: the full `properties` Map stays selected and their refs+ keep the Map access. The alias has three dot-separated parts, so `sql.id`+ cannot express it. The key's *value* twin on the same line binds normally+ (`properties[${sql.string(k)}]`).++Both are the identifier tier of ADR-013, implemented as a guard local to the+call site rather than through `sql.id`, because `sql.id`'s grammar (bare or+once-qualified, no dashes, no backticks) cannot spell either name. **If a+future task widens `sql.id` to accept a backtick-quoted, multi-part name,+these two rows and `compiled.ts` with them can go.** That is the standing exit+condition; recorded here, not done in M12-009 (out of its scope).diff --git a/packages/core/package.json b/packages/core/package.jsonindex 0eea1dd4d..5406436ce 100644--- a/packages/core/package.json+++ b/packages/core/package.json@@ -68,7 +68,6 @@"sharp": "^0.33.5","short-unique-id": "^5.0.3","slugify": "^1.6.6",- "sqlstring": "^2.3.3","superjson": "^1.13.3","ua-parser-js": "^2.0.6","undici": "^8.10.0",@@ -84,7 +83,6 @@"@types/luxon": "^3.7.1","@types/qrcode": "^1.5.6","@types/ramda": "^0.31.1",- "@types/sqlstring": "^2.3.2","@types/unzipper": "^0.10.10","@types/uuid": "^10.0.0","bun-types": "1.4.0",diff --git a/packages/core/src/buffers/base-buffer.ts b/packages/core/src/buffers/base-buffer.tsindex c607cf0bc..2dfe051c8 100644--- a/packages/core/src/buffers/base-buffer.ts+++ b/packages/core/src/buffers/base-buffer.ts@@ -3,7 +3,7 @@ import type { ClickHouseSettings } from '@clickhouse/client';import { getRedisCache } from '@openpanel/redis';import type { Logger } from '../logger';import type { ServiceDeps } from '../services';-import { type ChScope, chQuery } from '../shared/ch-query';+import { type ChQueryInput, type ChScope, chQuery } from '../shared/ch-query';import { generateSecureId } from '../shared/id';// GENUINE CYCLE, kept lazy: v1-compat.ts -> services.ts -> (every service) ->@@ -131,7 +131,7 @@ export class BaseBuffer {}/** `chQuery` bound to this buffer's client and its own logger. */- protected async chQuery<T extends object>(query: string): Promise<T[]> {+ protected async chQuery<T extends object>(query: ChQueryInput): Promise<T[]> {const scope: ChScope = { ch: await this.resolveCh(), logger: this.logger };return chQuery<T>(scope, query);}diff --git a/packages/core/src/buffers/buffers.sql.proof.md b/packages/core/src/buffers/buffers.sql.proof.mdnew file mode 100644index 000000000..9ad2cdc89--- /dev/null+++ b/packages/core/src/buffers/buffers.sql.proof.md@@ -0,0 +1,141 @@+# buffers off `sqlstring` — V1 → V2 result-set proof (M12-009)++M12-009 deletes `sqlstring` from every workspace manifest, so the three buffer+files that still imported it had to move onto the `sql` tag first. V1 is the+`sqlstring.escape(...)` text each file built at `30d236fa`; V2 is the tagged+statement in this commit. Both forms were rendered in one Bun process and+executed through the same `@clickhouse/client` (`format: 'JSONCompact'`) — V1+as `query`, V2 as `query` + `query_params`. `meta` and `data` were compared+field for field.++- **Date**: 2026-09-07, run by ralph on this box.+- **Reads**: local prod-copy `openpanel` (319M events), read-only.+- **Mutation**: the `profile-backfill-buffer` lightweight `UPDATE` is a write,+ so it ran against a throwaway `m12009_scratch` database seeded with three+ rows, never against `openpanel` or `openpanel_test`. The database was dropped+ after the run.+- **Machine**: single-node ClickHouse 26.1.3.52 on a 4-vCPU box. Timings are+ directional only; production is 2 shards × 2 replicas+ (`docs/ENVIRONMENT.md`).+- **Verdict**: **4 statements, all IDENTICAL.**++| # | Site | Statement | Rows | old → new |+|---|---|---|---:|---|+| 1 | `group-buffer.ts:84` | `SELECT … FROM groups FINAL WHERE project_id/id` | 1 | 33ms → 8ms |+| 2 | `profile-buffer.ts:222` | batch fetch, `withDateFilter=false` | 2 | 20ms → 20ms |+| 3 | `profile-buffer.ts:222` | batch fetch, `withDateFilter=true` | 0 | 9ms → 8ms |+| 4 | `profile-backfill-buffer.ts:96` | lightweight `UPDATE … CASE session_id` | 3 read back | 7ms → 6ms |++## IN / GLOBAL IN++Every `IN` touched here takes a **literal set** — `(p.id, p.project_id) IN+((…), (…))` and `(project_id, session_id) IN ((…), (…))` — not a subquery. A+literal set is evaluated identically on every shard, so `docs/ENVIRONMENT.md`'s+`IN` vs `GLOBAL IN` trap does not apply and **no `IN` was converted in either+direction**. Binding the tuple members as `{pN:String}` does not change that:+the set is still spelled out in the query text, one placeholder per member.++## Run output++```+==============================================================================+CASE: group-buffer: SELECT ... FROM groups FINAL [db=openpanel]+--- OLD (clix/sqlstring text) ---+SELECT project_id, id, type, name, properties, created_at+ FROM groups FINAL+ WHERE project_id = 'acade'+ AND id = 'acade'+ AND deleted = 0+--- NEW (sql tag) ---+SELECT project_id, id, type, name, properties, created_at+ FROM groups FINAL+ WHERE project_id = {p1:String}+ AND id = {p2:String}+ AND deleted = 0+--- params ---+{"p1":"acade","p2":"acade"}+old: ok=true rows=1 33ms+new: ok=true rows=1 8ms+RESULT SETS: IDENTICAL+==============================================================================+CASE: profile-buffer: batch fetch (withDateFilter=false) [db=openpanel]+--- OLD (clix/sqlstring text) ---+SELECT id, project_id, argMax(first_name, p.last_seen_at) AS first_name, argMax(last_name, p.last_seen_at) AS last_name, argMax(email, p.last_seen_at) AS email, argMax(avatar, p.last_seen_at) AS avatar, argMax(properties, p.last_seen_at) AS properties, argMax(is_external, p.last_seen_at) AS is_external, argMax(groups, p.last_seen_at) AS groups, min(created_at) AS created_at, max(p.last_seen_at) AS last_seen_at+ FROM profiles AS p+ WHERE (p.id, p.project_id) IN (('03b7d7d74db08200f7fb3dc912abb51e', '0byte'), ('07deba9fc77b280a8dbc7f823d5c2319', '0byte'))++ GROUP BY p.id, p.project_id+--- NEW (sql tag) ---+SELECT id, project_id, argMax(first_name, p.last_seen_at) AS first_name, argMax(last_name, p.last_seen_at) AS last_name, argMax(email, p.last_seen_at) AS email, argMax(avatar, p.last_seen_at) AS avatar, argMax(properties, p.last_seen_at) AS properties, argMax(is_external, p.last_seen_at) AS is_external, argMax(groups, p.last_seen_at) AS groups, min(created_at) AS created_at, max(p.last_seen_at) AS last_seen_at+ FROM profiles AS p+ WHERE (p.id, p.project_id) IN (({p1:String}, {p2:String}), ({p3:String}, {p4:String}))++ GROUP BY p.id, p.project_id+--- params ---+{"p1":"03b7d7d74db08200f7fb3dc912abb51e","p2":"0byte","p3":"07deba9fc77b280a8dbc7f823d5c2319","p4":"0byte"}+old: ok=true rows=2 20ms+new: ok=true rows=2 20ms+RESULT SETS: IDENTICAL+==============================================================================+CASE: profile-buffer: batch fetch (withDateFilter=true) [db=openpanel]+--- OLD (clix/sqlstring text) ---+SELECT id, project_id, argMax(first_name, p.last_seen_at) AS first_name, argMax(last_name, p.last_seen_at) AS last_name, argMax(email, p.last_seen_at) AS email, argMax(avatar, p.last_seen_at) AS avatar, argMax(properties, p.last_seen_at) AS properties, argMax(is_external, p.last_seen_at) AS is_external, argMax(groups, p.last_seen_at) AS groups, min(created_at) AS created_at, max(p.last_seen_at) AS last_seen_at+ FROM profiles AS p+ WHERE (p.id, p.project_id) IN (('03b7d7d74db08200f7fb3dc912abb51e', '0byte'), ('07deba9fc77b280a8dbc7f823d5c2319', '0byte'))+ AND p.last_seen_at > now() - INTERVAL 2 DAY+ GROUP BY p.id, p.project_id+--- NEW (sql tag) ---+SELECT id, project_id, argMax(first_name, p.last_seen_at) AS first_name, argMax(last_name, p.last_seen_at) AS last_name, argMax(email, p.last_seen_at) AS email, argMax(avatar, p.last_seen_at) AS avatar, argMax(properties, p.last_seen_at) AS properties, argMax(is_external, p.last_seen_at) AS is_external, argMax(groups, p.last_seen_at) AS groups, min(created_at) AS created_at, max(p.last_seen_at) AS last_seen_at+ FROM profiles AS p+ WHERE (p.id, p.project_id) IN (({p1:String}, {p2:String}), ({p3:String}, {p4:String}))+ AND p.last_seen_at > now() - INTERVAL 2 DAY+ GROUP BY p.id, p.project_id+--- params ---+{"p1":"03b7d7d74db08200f7fb3dc912abb51e","p2":"0byte","p3":"07deba9fc77b280a8dbc7f823d5c2319","p4":"0byte"}+old: ok=true rows=0 9ms+new: ok=true rows=0 8ms+RESULT SETS: IDENTICAL+```++The `profile-backfill-buffer` case, run against `m12009_scratch`:++```+==============================================================================+CASE: profile-backfill-buffer: lightweight UPDATE ... CASE session_id [db=m12009_scratch]+--- OLD ---+UPDATE events_replicated+ SET profile_id = CASE session_id+ WHEN 's1' THEN 'prof-1'+WHEN 's2' THEN 'prof-2'+ END+ WHERE (project_id, session_id) IN (('p1', 's1'),('p1', 's2'))+ AND created_at > now() - INTERVAL 6 HOURS+--- NEW ---+UPDATE events_replicated+ SET profile_id = CASE session_id+ WHEN {p1:String} THEN {p2:String}+WHEN {p3:String} THEN {p4:String}+ END+ WHERE (project_id, session_id) IN (({p5:String}, {p6:String}), ({p7:String}, {p8:String}))+ AND created_at > now() - INTERVAL 6 HOURS+--- params ---+{"p1":"s1","p2":"prof-1","p3":"s2","p4":"prof-2","p5":"p1","p6":"s1","p7":"p1","p8":"s2"}+old: ok=true 7ms+new: ok=true 6ms+old rows after: {"meta":[{"name":"project_id","type":"String"},{"name":"session_id","type":"String"},{"name":"profile_id","type":"String"}],"data":[["p1","s1","prof-1"],["p1","s2","prof-2"],["p2","s3","keep"]]}+new rows after: {"meta":[{"name":"project_id","type":"String"},{"name":"session_id","type":"String"},{"name":"profile_id","type":"String"}],"data":[["p1","s1","prof-1"],["p1","s2","prof-2"],["p2","s3","keep"]]}+RESULT SETS: IDENTICAL+```++`m12009_scratch.events_replicated` needed `enable_block_number_column = 1` for+`allow_experimental_lightweight_update` to apply — a property of the throwaway+fixture table, not of the statement; both forms failed identically before it was+set and both succeeded identically after.++## Whitespace-only differences, stated++`profile-backfill-buffer.ts`'s tuple list joins on `', '` where V1 joined on+`','`, and `profile-buffer.ts`'s aggregate column list is now a+`sql.join([...], ', ')` of one fragment per column instead of a+`String[].join(', ')`. Both render the same tokens; the run above is over the+rendered text, so the comparison covers it.diff --git a/packages/core/src/buffers/group-buffer.ts b/packages/core/src/buffers/group-buffer.tsindex 0ad19b9ee..241c65256 100644--- a/packages/core/src/buffers/group-buffer.ts+++ b/packages/core/src/buffers/group-buffer.ts@@ -1,6 +1,6 @@+import { sql } from '@openpanel/db/src/clickhouse/sql';import { getRedisCache, type Redis } from '@openpanel/redis';import shallowEqual from 'fast-deep-equal';-import sqlstring from 'sqlstring';import { formatClickhouseDate } from '../shared/ch-dates';import { TABLE_NAMES } from '../shared/ch-tables';import { getSafeJson } from '../shared/json';@@ -80,11 +80,11 @@ export class GroupBuffer extends BaseBuffer {projectId: string,id: string): Promise<IGroupCacheEntry | null> {- const rows = await this.chQuery<IGroupCacheEntry>(`+ const rows = await this.chQuery<IGroupCacheEntry>(sql`SELECT project_id, id, type, name, properties, created_at- FROM ${TABLE_NAMES.groups} FINAL- WHERE project_id = ${sqlstring.escape(projectId)}- AND id = ${sqlstring.escape(id)}+ FROM ${sql.id(TABLE_NAMES.groups)} FINAL+ WHERE project_id = ${sql.string(projectId)}+ AND id = ${sql.string(id)}AND deleted = 0`);return rows[0] ?? null;diff --git a/packages/core/src/buffers/profile-backfill-buffer.ts b/packages/core/src/buffers/profile-backfill-buffer.tsindex b2ade70ef..24a0ec438 100644--- a/packages/core/src/buffers/profile-backfill-buffer.ts+++ b/packages/core/src/buffers/profile-backfill-buffer.ts@@ -1,5 +1,5 @@+import { sql } from '@openpanel/db/src/clickhouse/sql';import { getRedisCache, type Redis } from '@openpanel/redis';-import sqlstring from 'sqlstring';import { getReplicatedTableName, TABLE_NAMES } from '../shared/ch-tables';import { getSafeJson } from '../shared/json';import { BaseBuffer, type BufferDeps } from './base-buffer';@@ -77,29 +77,32 @@ export class ProfileBackfillBuffer extends BaseBuffer {const chStart = performance.now();for (const chunk of chunks) {- const caseClause = chunk- .map(+ const caseClause = sql.join(+ chunk.map(({ sessionId, profileId }) =>- `WHEN ${sqlstring.escape(sessionId)} THEN ${sqlstring.escape(profileId)}`- )- .join('\n');- const tupleList = chunk- .map(+ sql`WHEN ${sql.string(sessionId)} THEN ${sql.string(profileId)}`+ ),+ '\n'+ );+ const tupleList = sql.join(+ chunk.map(({ projectId, sessionId }) =>- `(${sqlstring.escape(projectId)}, ${sqlstring.escape(sessionId)})`- )- .join(',');+ sql`(${sql.string(projectId)}, ${sql.string(sessionId)})`+ ),+ ', '+ );- const query = `- UPDATE ${table}+ const statement = sql`+ UPDATE ${sql.id(table)}SET profile_id = CASE session_id${caseClause}ENDWHERE (project_id, session_id) IN (${tupleList})- AND created_at > now() - INTERVAL 6 HOURS`;+ AND created_at > now() - INTERVAL 6 HOURS`.toStatement();await ch.command({- query,+ query: statement.query,+ query_params: statement.query_params,clickhouse_settings: {mutations_sync: '0',allow_experimental_lightweight_update: '1',diff --git a/packages/core/src/buffers/profile-buffer.ts b/packages/core/src/buffers/profile-buffer.tsindex 6cf07250e..be9f8bc0a 100644--- a/packages/core/src/buffers/profile-buffer.ts+++ b/packages/core/src/buffers/profile-buffer.ts@@ -1,6 +1,6 @@+import { sql } from '@openpanel/db/src/clickhouse/sql';import { getRedisCache, type Redis } from '@openpanel/redis';import { omit, uniq } from 'ramda';-import sqlstring from 'sqlstring';import type { IClickhouseProfile } from '../modules/profile/profile.service';import { TABLE_NAMES } from '../shared/ch-tables';import { getSafeJson } from '../shared/json';@@ -25,22 +25,25 @@ const PROFILE_COLUMNS =// bare column refs against the alias list first — causing ILLEGAL_AGGREGATION// inside the argMax/max calls. Qualifying with `p.` bypasses the alias// lookup and binds to the raw column.-const PROFILE_LATEST_AGGREGATE_COLUMNS = [- 'id',- 'project_id',- 'argMax(first_name, p.last_seen_at) AS first_name',- 'argMax(last_name, p.last_seen_at) AS last_name',- 'argMax(email, p.last_seen_at) AS email',- 'argMax(avatar, p.last_seen_at) AS avatar',- 'argMax(properties, p.last_seen_at) AS properties',- 'argMax(is_external, p.last_seen_at) AS is_external',- 'argMax(groups, p.last_seen_at) AS groups',- // created_at doesn't change between rows for the same profile, but- // min() is safe and matches "first seen" semantics if there ever is a- // discrepancy.- 'min(created_at) AS created_at',- 'max(p.last_seen_at) AS last_seen_at',-].join(', ');+const PROFILE_LATEST_AGGREGATE_COLUMNS = sql.join(+ [+ sql`id`,+ sql`project_id`,+ sql`argMax(first_name, p.last_seen_at) AS first_name`,+ sql`argMax(last_name, p.last_seen_at) AS last_name`,+ sql`argMax(email, p.last_seen_at) AS email`,+ sql`argMax(avatar, p.last_seen_at) AS avatar`,+ sql`argMax(properties, p.last_seen_at) AS properties`,+ sql`argMax(is_external, p.last_seen_at) AS is_external`,+ sql`argMax(groups, p.last_seen_at) AS groups`,+ // created_at doesn't change between rows for the same profile, but+ // min() is safe and matches "first seen" semantics if there ever is a+ // discrepancy.+ sql`min(created_at) AS created_at`,+ sql`max(p.last_seen_at) AS last_seen_at`,+ ],+ ', '+);export class ProfileBuffer extends BaseBuffer {private readonly batchSize = process.env.PROFILE_BUFFER_BATCH_SIZE@@ -202,12 +205,13 @@ export class ProfileBuffer extends BaseBuffer {withDateFilter: boolean) => {for (const chunk of this.chunks(group, this.fetchChunkSize)) {- const tuples = chunk- .map(+ const tuples = sql.join(+ chunk.map((p) =>- `(${sqlstring.escape(String(p.id))}, ${sqlstring.escape(p.project_id)})`- )- .join(', ');+ sql`(${sql.string(String(p.id))}, ${sql.string(p.project_id)})`+ ),+ ', '+ );try {// Table alias `p` is required: without it, WHERE's `last_seen_at`// resolves to the SELECT-list aggregate alias `max(last_seen_at) AS@@ -215,10 +219,10 @@ export class ProfileBuffer extends BaseBuffer {// (CH ILLEGAL_AGGREGATION). Qualifying with `p.` bypasses the alias// lookup and binds to the raw column.const rows = await this.chQuery<IClickhouseProfile>(- `SELECT ${PROFILE_LATEST_AGGREGATE_COLUMNS}- FROM ${TABLE_NAMES.profiles} AS p+ sql`SELECT ${PROFILE_LATEST_AGGREGATE_COLUMNS}+ FROM ${sql.id(TABLE_NAMES.profiles)} AS pWHERE (p.id, p.project_id) IN (${tuples})- ${withDateFilter ? 'AND p.last_seen_at > now() - INTERVAL 2 DAY' : ''}+ ${withDateFilter ? sql`AND p.last_seen_at > now() - INTERVAL 2 DAY` : sql.empty}GROUP BY p.id, p.project_id`);for (const row of rows) {diff --git a/packages/db/index.ts b/packages/db/index.tsindex 22cb91d5e..310cb433a 100644--- a/packages/db/index.ts+++ b/packages/db/index.ts@@ -1,10 +1,4 @@export * from './src/clickhouse/client';-// `clix` (query-builder.ts) and `createSqlBuilder` (sql-builder.ts) are dead-// per ADR-013, but 7 live call sites across @openpanel/core still import-// them directly and have not been converted onto the `sql` tag yet — see-// M9-CLEANUP-001's report. Deleting these two ahead of that conversion is-// BLOCKED, not done.-export * from './src/clickhouse/query-builder';+export * from './src/clickhouse/sql';export * from './src/prisma-client';-export * from './src/sql-builder';export * from './src/types';diff --git a/packages/db/package.json b/packages/db/package.jsonindex 32e591824..431c09b31 100644--- a/packages/db/package.json+++ b/packages/db/package.json@@ -25,13 +25,11 @@"pino": "catalog:","pino-pretty": "catalog:","prisma-json-types-generator": "^3.1.1",- "sqlstring": "^2.3.3","zod": "catalog:"},"devDependencies": {"@openpanel/tsconfig": "workspace:*","@types/node": "catalog:",- "@types/sqlstring": "^2.3.2","prisma": "^6.14.0","typescript": "catalog:"}diff --git a/packages/db/scripts/ch-copy-from-remote.ts b/packages/db/scripts/ch-copy-from-remote.tsindex 18e05445e..3b8ad98c2 100644--- a/packages/db/scripts/ch-copy-from-remote.ts+++ b/packages/db/scripts/ch-copy-from-remote.ts@@ -1,9 +1,8 @@import { stdin as input, stdout as output } from 'node:process';import { createInterface } from 'node:readline/promises';import { parseArgs } from 'node:util';-import sqlstring from 'sqlstring';import { ch } from '../src/clickhouse/client';-import { clix } from '../src/clickhouse/query-builder';+import { sql } from '../src/clickhouse/sql';async function main() {const rl = createInterface({ input, output });@@ -31,14 +30,20 @@ async function main() {const host =getArg(values.host) || (await rl.question('Remote Host (IP/Domain): '));- if (!host) throw new Error('Host is required');+ if (!host) {+ throw new Error('Host is required');+ }const user = getArg(values.user) || (await rl.question('Remote User: '));- if (!user) throw new Error('User is required');+ if (!user) {+ throw new Error('User is required');+ }const password =getArg(values.password) || (await rl.question('Remote Password: '));- if (!password) throw new Error('Password is required');+ if (!password) {+ throw new Error('Password is required');+ }const dbName =getArg(values.db) ||@@ -48,17 +53,21 @@ async function main() {const startDate =getArg(values.start) ||(await rl.question('Start Date (YYYY-MM-DD HH:mm:ss): '));- if (!startDate) throw new Error('Start date is required');+ if (!startDate) {+ throw new Error('Start date is required');+ }const endDate =getArg(values.end) ||(await rl.question('End Date (YYYY-MM-DD HH:mm:ss): '));- if (!endDate) throw new Error('End date is required');+ if (!endDate) {+ throw new Error('End date is required');+ }const projectIdsInput =getArg(values.projects) ||(await rl.question(- 'Project IDs (comma separated, leave empty for all): ',+ 'Project IDs (comma separated, leave empty for all): '));const projectIds = projectIdsInput? projectIdsInput.split(',').map((s: string) => s.trim())@@ -71,27 +80,29 @@ async function main() {for (const table of tables) {console.log(`Processing table: ${table}`);- // Build the SELECT part using the query builder- // We use sqlstring to escape the remote function arguments- const remoteTable = `remote(${sqlstring.escape(host)}, ${sqlstring.escape(dbName)}, ${sqlstring.escape(table)}, ${sqlstring.escape(user)}, ${sqlstring.escape(password)})`;+ // Every remote() argument is a bound param, so the credentials never+ // enter the query text — which is what makes the log line below safe to+ // print. `dbName` and `table` are identifiers: bound as {x:Identifier}+ // inside remote(), validated by sql.id() in the INSERT target, where a+ // param is not accepted.+ const remoteTable = sql`remote(${sql.string(host)}, ${sql.identifier(dbName)}, ${sql.identifier(table)}, ${sql.string(user)}, ${sql.string(password)})`;- const queryBuilder = clix(ch)- .from(remoteTable)- .select(['*'])- .where('created_at', 'BETWEEN', [startDate, endDate]);+ const projectFilter =+ projectIds.length > 0+ ? sql` AND project_id IN ${sql.array('String', projectIds)}`+ : sql.empty;- if (projectIds.length > 0) {- queryBuilder.where('project_id', 'IN', projectIds);- }+ const selectQuery = sql`SELECT * FROM ${remoteTable} WHERE created_at BETWEEN ${sql.string(startDate)} AND ${sql.string(endDate)}${projectFilter}`;- const selectQuery = queryBuilder.toSQL();- const insertQuery = `INSERT INTO ${dbName}.${table} ${selectQuery}`;+ const insertQuery =+ sql`INSERT INTO ${sql.id(dbName)}.${sql.id(table)} ${selectQuery}`.toStatement();- console.log(`Executing: ${insertQuery}`);+ console.log(`Executing: ${insertQuery.query}`);// try {// await ch.command({- // query: insertQuery,+ // query: insertQuery.query,+ // query_params: insertQuery.query_params,// });// console.log(`✅ Copied ${table} successfully`);// } catch (error) {diff --git a/packages/db/scripts/ch-update-sessions-with-revenue.ts b/packages/db/scripts/ch-update-sessions-with-revenue.tsindex 93ecb22eb..76ca68613 100644--- a/packages/db/scripts/ch-update-sessions-with-revenue.ts+++ b/packages/db/scripts/ch-update-sessions-with-revenue.ts@@ -1,13 +1,22 @@-import { TABLE_NAMES, ch } from '../src/clickhouse/client';-import { clix } from '../src/clickhouse/query-builder';+// Local fixture script: give a couple of sessions per hour a random revenue.+//+// M12-009 took it off `clix` (deleted with ADR-013 decision 21) and onto the+// `sql` tag. Both statements bind every value as a `{pN:Type}` param instead+// of splicing it into the text, which is also what removes the hand-rolled+// `'${id}'` quoting the ALTER used to do.+import { ch, chQuery, TABLE_NAMES } from '../src/clickhouse/client';+import { sql } from '../src/clickhouse/sql';const START_DATE = new Date('2025-11-10T00:00:00Z');const END_DATE = new Date('2025-11-20T23:00:00Z');const SESSIONS_PER_HOUR = 2;+const PROJECT_ID = 'public-web';+const MS_PER_HOUR = 60 * 60 * 1000;+const PAUSE_BETWEEN_MUTATIONS_MS = 500;// Revenue between $10 (1000 cents) and $200 (20000 cents)const MIN_REVENUE = 1000;-const MAX_REVENUE = 20000;+const MAX_REVENUE = 20_000;function getRandomRevenue() {return (@@ -15,26 +24,30 @@ function getRandomRevenue() {);}+/** `YYYY-MM-DD HH:mm:ss` — the literal form clix escaped a `Date` to. */+function datetime(date: Date): string {+ return date.toISOString().slice(0, 19).replace('T', ' ');+}+async function main() {console.log(- `Starting revenue update for sessions between ${START_DATE.toISOString()} and ${END_DATE.toISOString()}`,+ `Starting revenue update for sessions between ${START_DATE.toISOString()} and ${END_DATE.toISOString()}`);let currentDate = new Date(START_DATE);while (currentDate < END_DATE) {- const nextHour = new Date(currentDate.getTime() + 60 * 60 * 1000);+ const nextHour = new Date(currentDate.getTime() + MS_PER_HOUR);console.log(`Processing hour: ${currentDate.toISOString()}`);// 1. Pick random sessions for this hour- const sessions = await clix(ch)- .from(TABLE_NAMES.sessions)- .select(['id'])- .where('created_at', '>=', currentDate)- .andWhere('created_at', '<', nextHour)- .where('project_id', '=', 'public-web')- .limit(SESSIONS_PER_HOUR)- .execute();+ const sessions = await chQuery<{ id: string }>(sql`+ SELECT id+ FROM ${sql.id(TABLE_NAMES.sessions)}+ WHERE created_at >= ${sql.string(datetime(currentDate))}+ AND created_at < ${sql.string(datetime(nextHour))}+ AND project_id = ${sql.string(PROJECT_ID)}+ LIMIT ${sql.uint64(SESSIONS_PER_HOUR)}`);if (sessions.length === 0) {console.log(`No sessions found for ${currentDate.toISOString()}`);@@ -42,44 +55,42 @@ async function main() {continue;}- const sessionIds = sessions.map((s: any) => s.id);+ const sessionIds = sessions.map((s) => s.id);console.log(- `Found ${sessionIds.length} sessions to update: ${sessionIds.join(', ')}`,+ `Found ${sessionIds.length} sessions to update: ${sessionIds.join(', ')}`);- // 2. Construct update query- // We want to assign a DIFFERENT random revenue to each session- // Query: ALTER TABLE sessions UPDATE revenue = if(id='id1', rev1, if(id='id2', rev2, ...)) WHERE id IN ('id1', 'id2', ...)-- const updates: { id: string; revenue: number }[] = [];-- for (const id of sessionIds) {- const revenue = getRandomRevenue();- updates.push({ id, revenue });- }-- // Build nested if() for the update expression- // ClickHouse doesn't have CASE WHEN in UPDATE expression in the same way, but if() works.- // Actually multiIf is cleaner: multiIf(id='id1', rev1, id='id2', rev2, revenue)+ // 2. Assign a DIFFERENT random revenue to each session. ClickHouse has no+ // CASE WHEN in an UPDATE expression, so multiIf carries the mapping and+ // falls back to the existing `revenue`.+ const updates = sessionIds.map((id) => ({+ id,+ revenue: getRandomRevenue(),+ }));- const conditions = updates- .map((u) => `id = '${u.id}', ${u.revenue}`)- .join(', ');- const updateExpr = `multiIf(${conditions}, revenue)`;+ const updateExpr = sql`multiIf(${sql.join(+ updates.map(+ (u) => sql`id = ${sql.string(u.id)}, ${sql.float64(u.revenue)}`+ ),+ ', '+ )}, revenue)`;- const idsStr = sessionIds.map((id: string) => `'${id}'`).join(', ');- const query = `ALTER TABLE ${TABLE_NAMES.sessions} UPDATE revenue = ${updateExpr} WHERE id IN (${idsStr})`;+ const statement =+ sql`ALTER TABLE ${sql.id(TABLE_NAMES.sessions)} UPDATE revenue = ${updateExpr} WHERE id IN ${sql.array('String', sessionIds)}`.toStatement();- console.log(`Executing update: ${query}`);+ console.log(`Executing update: ${statement.query}`);try {await ch.command({- query,+ query: statement.query,+ query_params: statement.query_params,});console.log('Update command sent.');// Wait a bit to not overload mutations if running on a large range- await new Promise((resolve) => setTimeout(resolve, 500));+ await new Promise((resolve) =>+ setTimeout(resolve, PAUSE_BETWEEN_MUTATIONS_MS)+ );} catch (error) {console.error('Failed to update sessions:', error);}diff --git a/packages/db/scripts/scripts.sql.proof.md b/packages/db/scripts/scripts.sql.proof.mdnew file mode 100644index 000000000..24c3bfbfe--- /dev/null+++ b/packages/db/scripts/scripts.sql.proof.md@@ -0,0 +1,119 @@+# `packages/db/scripts` off `clix` / `sqlstring` — result-set proof (M12-009)++M12-009 deletes `query-builder.ts`, `sql-builder.ts` and the `sqlstring`+dependency. Two operator scripts in this directory were the last things in+`packages/db` importing either, so they move onto the `sql` tag in the same+commit. (The gate `tooling/gates/p12-grep-gates.sh` does not scan `scripts/`;+these two had to convert anyway because removing `sqlstring` from+`packages/db/package.json` removes the module they imported.)++V1 is each script's text at `30d236fa`; V2 is the tagged statement in this+commit. Both were rendered in one Bun process and executed through the same+`@clickhouse/client` — V1 as `query`, V2 as `query` + `query_params` — and the+result sets (or, for the writes, the table read back afterwards) compared field+for field.++- **Date**: 2026-09-07, run by ralph on this box.+- **Database**: a throwaway `m12009_scratch`, seeded per case and dropped after+ the run. Both scripts are writers (`ALTER TABLE … UPDATE`, `INSERT … SELECT`),+ so neither could be pointed at the prod-copy `openpanel` or at+ `openpanel_test`.+- **Machine**: single-node ClickHouse 26.1.3.52, 4 vCPU. Timings directional+ only; production is 2 shards × 2 replicas (`docs/ENVIRONMENT.md`).+- **Verdict**: **4 statements, all IDENTICAL.**++| # | Script | Statement | Rows | old → new |+|---|---|---|---:|---|+| 1 | `ch-update-sessions-with-revenue.ts:45` | `SELECT id FROM sessions … LIMIT {n}` | 2 | 5ms → 5ms |+| 2 | `ch-update-sessions-with-revenue.ts:79` | `ALTER TABLE … UPDATE revenue = multiIf(…)` | 4 read back | 11ms → 14ms |+| 3 | `ch-copy-from-remote.ts:95` | `SELECT * FROM remote(…)` | 2 | 7ms → 9ms |+| 4 | `ch-copy-from-remote.ts:97` | `INSERT INTO db.table SELECT * FROM remote(…)` | 3 → 5 | 9ms → 9ms |++## What binding buys here, concretely++`remote()` accepts bound parameters in every argument position — `{p:String}`+for the host, user and password, `{p:Identifier}` for the database and table.+That was verified before the conversion was written (case 3 above) and it is+the whole point: `ch-copy-from-remote.ts` **logs the query it is about to run**,+and V1's text carried the remote password in it. It no longer does — the+`--- params ---` line in the run below is the only place the credential+appears, and the script never prints it.++The `INSERT INTO <db>.<table>` target is the one position a `{x:Identifier}`+param is not accepted, so it goes through `sql.id()`, which throws on anything+that is not a bare identifier rather than falling back to interpolation+(ADR-013 R3).++## Type choices++- `revenue` is `Float64` (`system.columns`), so the `multiIf` branches bind as+ `sql.float64` — the type ClickHouse infers for V1's bare integer literal in+ that expression, not a wider guess.+- `LIMIT` takes `sql.uint64`.+- The two date bounds bind as `String`, exactly the form V1 quoted, so+ `date_time_input_format: 'best_effort'` parses them identically.++## IN / GLOBAL IN++`WHERE id IN ('a','b')` became `WHERE id IN {p:Array(String)}`. Both are a+**literal set**, not a subquery, so they evaluate identically on every shard and+`docs/ENVIRONMENT.md`'s `IN` vs `GLOBAL IN` trap does not apply. No `IN` was+converted in either direction. `remote()` is a table function evaluated on the+initiator, unchanged by the rewrite.++## Run output++```+==============================================================================+CASE: ch-update-sessions-with-revenue: SELECT id FROM sessions [db=m12009_scratch]+--- OLD ---+SELECT id FROM sessions WHERE created_at >= '2025-11-10 00:00:00' AND created_at < '2025-11-10 01:00:00' AND project_id = 'public-web' LIMIT 2+--- NEW ---+SELECT id+ FROM sessions+ WHERE created_at >= {p1:String}+ AND created_at < {p2:String}+ AND project_id = {p3:String}+ LIMIT {p4:UInt64}+--- params ---+{"p1":"2025-11-10 00:00:00","p2":"2025-11-10 01:00:00","p3":"public-web","p4":2}+old: ok=true 5ms+new: ok=true 5ms+old: {"meta":[{"name":"id","type":"String"}],"data":[["sess-a"],["sess-b"]]}+new: {"meta":[{"name":"id","type":"String"}],"data":[["sess-a"],["sess-b"]]}+RESULT SETS: IDENTICAL+==============================================================================+CASE: ch-update-sessions-with-revenue: ALTER TABLE ... UPDATE multiIf [db=m12009_scratch]+--- OLD ---+ALTER TABLE sessions UPDATE revenue = multiIf(id = 'sess-a', 1500, id = 'sess-b', 20000, revenue) WHERE id IN ('sess-a', 'sess-b')+--- NEW ---+ALTER TABLE sessions UPDATE revenue = multiIf(id = {p1:String}, {p2:Float64}, id = {p3:String}, {p4:Float64}, revenue) WHERE id IN {p5:Array(String)}+--- params ---+{"p1":"sess-a","p2":1500,"p3":"sess-b","p4":20000,"p5":["sess-a","sess-b"]}+old: ok=true 11ms+new: ok=true 14ms+old rows after: {"meta":[{"name":"id","type":"String"},{"name":"revenue","type":"Float64"}],"data":[["sess-a",1500],["sess-b",20000],["sess-c",0],["sess-d",0]]}+new rows after: {"meta":[{"name":"id","type":"String"},{"name":"revenue","type":"Float64"}],"data":[["sess-a",1500],["sess-b",20000],["sess-c",0],["sess-d",0]]}+RESULT SETS: IDENTICAL+==============================================================================+CASE: ch-copy-from-remote: INSERT INTO <db>.<table> SELECT * FROM remote(...) [db=m12009_scratch]+--- OLD SELECT ---+SELECT * FROM remote('127.0.0.1:9000', 'm12009_scratch', 'copy_target', 'default', '') WHERE created_at BETWEEN '2025-11-10 00:00:00' AND '2025-11-11 00:00:00' AND project_id IN ('p1', 'p2')+--- NEW SELECT ---+SELECT * FROM remote({p1:String}, {p2:Identifier}, {p3:Identifier}, {p4:String}, {p5:String}) WHERE created_at BETWEEN {p6:String} AND {p7:String} AND project_id IN {p8:Array(String)}+--- params (note: credentials are NOT in the text) ---+{"p1":"127.0.0.1:9000","p2":"m12009_scratch","p3":"copy_target","p4":"default","p5":"","p6":"2025-11-10 00:00:00","p7":"2025-11-11 00:00:00","p8":["p1","p2"]}+old: ok=true 7ms+new: ok=true 9ms+old: {"meta":[{"name":"project_id","type":"String"},{"name":"created_at","type":"DateTime64(3)"},{"name":"name","type":"String"}],"data":[["p1","2025-11-10 00:05:00.000","a"],["p2","2025-11-10 00:06:00.000","b"]]}+new: {"meta":[{"name":"project_id","type":"String"},{"name":"created_at","type":"DateTime64(3)"},{"name":"name","type":"String"}],"data":[["p1","2025-11-10 00:05:00.000","a"],["p2","2025-11-10 00:06:00.000","b"]]}+SELECT RESULT SETS: IDENTICAL+--- OLD INSERT ---+INSERT INTO m12009_scratch.copy_target SELECT * FROM remote('127.0.0.1:9000', 'm12009_scratch', 'copy_target', 'default', '') WHERE created_at BETWEEN '2025-11-10 00:00:00' AND '2025-11-11 00:00:00' AND project_id IN ('p1', 'p2')+--- NEW INSERT ---+INSERT INTO m12009_scratch.copy_target SELECT * FROM remote({p1:String}, {p2:Identifier}, {p3:Identifier}, {p4:String}, {p5:String}) WHERE created_at BETWEEN {p6:String} AND {p7:String} AND project_id IN {p8:Array(String)}+old INSERT: ok=true 9ms+new INSERT: ok=true 9ms+count before: {"meta":[{"name":"count()","type":"UInt64"}],"data":[[3]]} after old: {"meta":[{"name":"count()","type":"UInt64"}],"data":[[5]]} after new: {"meta":[{"name":"count()","type":"UInt64"}],"data":[[5]]}+INSERT EFFECT: IDENTICAL+```diff --git a/packages/db/src/clickhouse/client.ts b/packages/db/src/clickhouse/client.tsindex 12c4ff23b..70b4675c3 100644--- a/packages/db/src/clickhouse/client.ts+++ b/packages/db/src/clickhouse/client.ts@@ -5,8 +5,6 @@ import type {} from '@clickhouse/client';import { ClickHouseLogLevel, createClient } from '@clickhouse/client';import type { NodeClickHouseClientConfigOptions } from '@clickhouse/client/dist/config';-import type { IInterval } from '@openpanel/core/modules/report/report.constants';-import sqlstring from 'sqlstring';import { createLogger } from '../logger';import { RoundRobinPicker, withRoundRobinRetry } from './round-robin';import { type SqlFragment, toStatement } from './sql';@@ -407,23 +405,6 @@ export function formatClickhouseDate(.replace(/(\.\d{3})?Z+$/, '');}-export function toDate(str: string, interval?: IInterval) {- // If it does not match the regex it's a column name eg 'created_at'- if (!interval || interval === 'minute' || interval === 'hour') {- if (str.match(/\d{4}-\d{2}-\d{2}/)) {- return sqlstring.escape(str);- }-- return str;- }-- if (str.match(/\d{4}-\d{2}-\d{2}/)) {- return `toDate(${sqlstring.escape(str.split(' ')[0])})`;- }-- return `toDate(${str})`;-}-export function convertClickhouseDateToJs(date: string) {return new Date(`${date.replace(' ', 'T')}Z`);}diff --git a/packages/db/src/clickhouse/query-builder.test.ts b/packages/db/src/clickhouse/query-builder.test.tsdeleted file mode 100644index 17e47393a..000000000--- a/packages/db/src/clickhouse/query-builder.test.ts+++ /dev/null@@ -1,88 +0,0 @@-import type { ClickHouseClient } from '@clickhouse/client';-import { describe, expect, it } from 'vitest';-import { clix } from './query-builder';--const fakeClient = {} as ClickHouseClient;--const buildWhereSQL = (column: string, op: any, value: any) =>- clix(fakeClient)- .select(['*'])- .from('events')- .where(column, op, value)- .toSQL();--describe('query-builder escapeValue', () => {- it('fully escapes user paths that contain a date substring', () => {- const sql = buildWhereSQL(- 'e.path',- '=',- '/blog/2024-03-17-supabase-activity-scheduler',- );-- expect(sql).toContain(- "e.path = '/blog/2024-03-17-supabase-activity-scheduler'",- );- // Make sure the date is NOT quoted independently (the original bug).- expect(sql).not.toContain("/blog/'2024-03-17'");- });-- it('does not allow SQL injection via a date-substring payload', () => {- const sql = buildWhereSQL(- 'e.path',- '=',- "/blog/2024-01-01' OR '1'='1",- );-- // Single quotes in the value must be escaped, not break out of the string.- expect(sql).toContain(- "e.path = '/blog/2024-01-01\\' OR \\'1\\'=\\'1'",- );- expect(sql).not.toMatch(/OR '1'='1[^']/);- });-- it('escapes a plain ISO date string', () => {- const sql = buildWhereSQL('e.created_at', '=', '2026-04-29 00:00:00');- expect(sql).toContain("e.created_at = '2026-04-29 00:00:00'");- });-- it('keeps clix.datetime(date, "toDateTime") as a SQL fragment with the date quoted', () => {- const sql = buildWhereSQL(- 'e.created_at',- '=',- clix.datetime('2026-04-29T00:00:00.000Z', 'toDateTime'),- );- expect(sql).toContain("e.created_at = toDateTime('2026-04-29 00:00:00')");- });-- it('handles BETWEEN with clix.datetime wrapped values', () => {- const sql = clix(fakeClient)- .select(['*'])- .from('events')- .where('e.created_at', 'BETWEEN', [- clix.datetime('2026-04-29T00:00:00.000Z', 'toDateTime'),- clix.datetime('2026-05-07T00:00:00.000Z', 'toDateTime'),- ])- .toSQL();-- expect(sql).toContain(- "e.created_at BETWEEN toDateTime('2026-04-29 00:00:00') AND toDateTime('2026-05-07 00:00:00')",- );- });-- it('escapes Date objects as datetime literals', () => {- const sql = buildWhereSQL(- 'e.created_at',- '=',- new Date('2026-04-29T00:00:00.000Z'),- );- expect(sql).toContain("e.created_at = '2026-04-29 00:00:00'");- });-- it('does not treat an arbitrary string with embedded date as raw SQL', () => {- // Function-call-like prefix but not a real wrapper — ensures we don't- // accidentally promote arbitrary text to raw SQL.- const sql = buildWhereSQL('e.path', '=', 'event-2026-04-15-launch');- expect(sql).toContain("e.path = 'event-2026-04-15-launch'");- expect(sql).not.toContain("'2026-04-15'");- });-});diff --git a/packages/db/src/clickhouse/query-builder.ts b/packages/db/src/clickhouse/query-builder.tsdeleted file mode 100644index d42885852..000000000--- a/packages/db/src/clickhouse/query-builder.ts+++ /dev/null@@ -1,791 +0,0 @@-import type { ClickHouseClient, ResponseJSON } from '@clickhouse/client';-import type { IInterval } from '@openpanel/core/modules/report/report.constants';-import sqlstring from 'sqlstring';--type SqlValue = string | number | boolean | Date | null | Expression;-type SqlParam = SqlValue | SqlValue[];-type Operator =- | '='- | '>'- | '<'- | '>='- | '<='- | '!='- | 'IN'- | 'NOT IN'- | 'LIKE'- | 'NOT LIKE'- | 'IS NULL'- | 'IS NOT NULL'- | 'BETWEEN';--type CTE = {- name: string;- query: Query | string;-};--type JoinType = 'INNER' | 'LEFT' | 'RIGHT' | 'FULL' | 'CROSS' | 'LEFT ANY';--type WhereCondition = {- condition: string;- operator: 'AND' | 'OR';- isGroup?: boolean;-};--type ConditionalCallback = (query: Query) => void;--class Expression {- constructor(private expression: string) {}-- toString() {- return this.expression;- }-}--export class Query<T = any> {- private _select: (string | Expression)[] = [];- private _except: string[] = [];- private _from?: string | Expression;- private _where: WhereCondition[] = [];- private _groupBy: string[] = [];- private _rollup = false;- private _having: { condition: string; operator: 'AND' | 'OR' }[] = [];- private _orderBy: {- column: string;- direction: 'ASC' | 'DESC';- }[] = [];- private _limit?: number;- private _offset?: number;- private _final = false;- private _settings: Record<string, string> = {};- private _ctes: CTE[] = [];- private _joins: {- type: JoinType;- table: string | Expression | Query;- condition: string;- alias?: string;- }[] = [];- private _skipNext = false;- private _rawJoins: string[] = [];- private _fill?: {- from: string | Date;- to: string | Date;- step: string;- };- private _transform?: Record<string, (item: T) => any>;- private _union?: Query;- private _dateRegex = /\d{4}-\d{2}-\d{2}([\s:\d.]+)?/g;- private _dateValueRegex =- /^(?:[a-zA-Z]\w*\()?\d{4}-\d{2}-\d{2}(?:[\s:\d.]+)?\)?$/;- constructor(- private client: ClickHouseClient,- private timezone: string- ) {}-- // Select methods- select<U>(- columns: (string | Expression | null | undefined | false)[],- type: 'merge' | 'replace' = 'replace'- ): Query<U> {- if (this._skipNext) {- return this as unknown as Query<U>;- }- if (type === 'merge') {- this._select = [- ...this._select,- ...columns.filter((col): col is string | Expression => Boolean(col)),- ];- } else {- this._select = columns.filter((col): col is string | Expression =>- Boolean(col)- );- }- return this as unknown as Query<U>;- }-- except(columns: string[]): this {- this._except = [...this._except, ...columns];- return this;- }-- rollup(): this {- this._rollup = true;- return this;- }-- // From methods- from(table: string | Expression, final = false): this {- this._from = table;- this._final = final;- return this;- }-- union(query: Query): this {- this._union = query;- return this;- }-- // Where methods- private escapeValue(value: SqlParam): string {- if (value === null) {- return 'NULL';- }- if (value instanceof Expression) {- return `(${value.toString()})`;- }- if (Array.isArray(value)) {- return `(${value.map((v) => this.escapeValue(v)).join(', ')})`;- }-- if (- (typeof value === 'string' && this._dateValueRegex.test(value)) ||- value instanceof Date- ) {- return this.escapeDate(value);- }-- return sqlstring.escape(value);- }-- where(column: string, operator: Operator, value?: SqlParam): this {- if (this._skipNext) {- return this;- }- const condition = this.buildCondition(column, operator, value);- this._where.push({ condition, operator: 'AND' });- return this;- }-- public buildCondition(- column: string,- operator: Operator,- value?: SqlParam- ): string {- switch (operator) {- case 'IS NULL':- return `${column} IS NULL`;- case 'IS NOT NULL':- return `${column} IS NOT NULL`;- case 'BETWEEN':- if (Array.isArray(value) && value.length === 2) {- return `${column} BETWEEN ${this.escapeValue(value[0]!)} AND ${this.escapeValue(value[1]!)}`;- }- throw new Error('BETWEEN operator requires an array of two values');- case 'IN':- case 'NOT IN':- if (!(Array.isArray(value) || value instanceof Expression)) {- throw new Error(`${operator} operator requires an array value`);- }- return `${column} ${operator} ${this.escapeValue(value)}`;- default:- return `${column} ${operator} ${this.escapeValue(value!)}`;- }- }-- andWhere(column: string, operator: Operator, value?: SqlParam): this {- const condition = this.buildCondition(column, operator, value);- this._where.push({ condition, operator: 'AND' });- return this;- }-- rawWhere(condition: string): this {- if (condition) {- this._where.push({ condition, operator: 'AND' });- }- return this;- }-- orWhere(column: string, operator: Operator, value?: SqlParam): this {- const condition = this.buildCondition(column, operator, value);- this._where.push({ condition, operator: 'OR' });- return this;- }-- // Group by methods- groupBy(columns: (string | null | undefined | false)[]): this {- this._groupBy = columns.filter((col): col is string => Boolean(col));- return this;- }-- // Having methods- having(column: string, operator: Operator, value: SqlParam): this {- const condition = this.buildCondition(column, operator, value);- this._having.push({ condition, operator: 'AND' });- return this;- }-- rawHaving(condition: string): this {- if (condition) {- this._having.push({ condition, operator: 'AND' });- }- return this;- }-- andHaving(column: string, operator: Operator, value: SqlParam): this {- const condition = this.buildCondition(column, operator, value);- this._having.push({ condition, operator: 'AND' });- return this;- }-- orHaving(column: string, operator: Operator, value: SqlParam): this {- const condition = this.buildCondition(column, operator, value);- this._having.push({ condition, operator: 'OR' });- return this;- }-- // Order by methods- orderBy(column: string, direction: 'ASC' | 'DESC' = 'ASC'): this {- if (this._skipNext) {- return this;- }- this._orderBy.push({ column, direction });- return this;- }-- // Limit and offset- limit(limit?: number): this {- if (limit !== undefined) {- this._limit = limit;- }- return this;- }-- offset(offset?: number): this {- if (offset !== undefined) {- this._offset = offset;- }- return this;- }-- // Settings- settings(settings: Record<string, string>): this {- Object.assign(this._settings, settings);- return this;- }-- with(name: string, query: Query | string): this {- this._ctes.push({ name, query });- return this;- }-- // Fill- fill(- from: string | Date | Expression,- to: string | Date | Expression,- step: string | Expression- ): this {- this._fill = {- from:- from instanceof Expression ? from.toString() : this.escapeDate(from),- to: to instanceof Expression ? to.toString() : this.escapeDate(to),- step: step instanceof Expression ? step.toString() : step,- };- return this;- }-- private escapeDate(value: string | Date): string {- if (value instanceof Date) {- return sqlstring.escape(clix.datetime(value));- }-- return value.replaceAll(this._dateRegex, (match) => {- return sqlstring.escape(match);- });- }-- // Add join methods- join(table: string | Expression, condition: string, alias?: string): this {- return this.joinWithType('INNER', table, condition, alias);- }-- innerJoin(- table: string | Expression,- condition: string,- alias?: string- ): this {- return this.joinWithType('INNER', table, condition, alias);- }-- leftJoin(- table: string | Expression | Query,- condition: string,- alias?: string- ): this {- return this.joinWithType('LEFT', table, condition, alias);- }-- leftAnyJoin(- table: string | Expression | Query,- condition: string,- alias?: string- ): this {- return this.joinWithType('LEFT ANY', table, condition, alias);- }-- rightJoin(- table: string | Expression,- condition: string,- alias?: string- ): this {- return this.joinWithType('RIGHT', table, condition, alias);- }-- fullJoin(- table: string | Expression,- condition: string,- alias?: string- ): this {- return this.joinWithType('FULL', table, condition, alias);- }-- crossJoin(table: string | Expression, alias?: string): this {- return this.joinWithType('CROSS', table, '', alias);- }-- rawJoin(sql: string): this {- if (this._skipNext) {- return this;- }- this._rawJoins.push(sql);- return this;- }-- private joinWithType(- type: JoinType,- table: string | Expression | Query,- condition: string,- alias?: string- ): this {- if (this._skipNext) {- return this;- }- this._joins.push({- type,- table,- condition: this.escapeDate(condition),- alias,- });- return this;- }-- // Add methods for grouping conditions- whereGroup(): WhereGroupBuilder {- return new WhereGroupBuilder(this, 'AND');- }-- orWhereGroup(): WhereGroupBuilder {- return new WhereGroupBuilder(this, 'OR');- }-- // Update buildQuery method's WHERE section- private buildWhereConditions(conditions: WhereCondition[]): string {- return conditions- .map((w, i) => {- const condition = w.isGroup ? `(${w.condition})` : w.condition;- return i === 0 ? condition : `${w.operator} ${condition}`;- })- .join(' ');- }-- private buildQuery(): string {- const parts: string[] = [];-- // Add WITH clause if CTEs exist- if (this._ctes.length > 0) {- const cteStatements = this._ctes.map((cte) => {- const queryStr =- typeof cte.query === 'string' ? cte.query : cte.query.toSQL();- return `${cte.name} AS (${queryStr})`;- });- parts.push(`WITH ${cteStatements.join(', ')}`);- }-- // SELECT- if (this._select.length > 0) {- parts.push(- 'SELECT',- this._select- // Important: Expressions are treated as raw SQL; do not run escapeDate()- // on them, otherwise any embedded date strings get double-escaped- // (e.g. ''2025-12-16 23:59:59'') which ClickHouse rejects.- .map((col) =>- col instanceof Expression ? col.toString() : this.escapeDate(col)- )- .join(', ')- );- } else {- parts.push('SELECT *');- }-- if (this._except.length > 0) {- parts.push('EXCEPT', `(${this._except.map(this.escapeDate).join(', ')})`);- }-- // FROM- if (this._from) {- if (this._from instanceof Expression) {- parts.push(`FROM (${this._from.toString()})`);- } else {- parts.push(`FROM ${this._from}${this._final ? ' FINAL' : ''}`);- }-- // Add joins- this._joins.forEach((join) => {- const aliasClause = join.alias ? ` ${join.alias} ` : ' ';- const conditionStr = join.condition ? `ON ${join.condition}` : '';- parts.push(- `${join.type} JOIN ${join.table instanceof Query ? `(${join.table.toSQL()})` : join.table instanceof Expression ? `(${join.table.toString()})` : join.table}${aliasClause}${conditionStr}`- );- });- // Add raw joins (e.g. ARRAY JOIN)- this._rawJoins.forEach((join) => {- parts.push(join);- });- }-- // WHERE- if (this._where.length > 0) {- parts.push('WHERE', this.buildWhereConditions(this._where));- }-- // GROUP BY- if (this._groupBy.length > 0) {- parts.push('GROUP BY', this._groupBy.join(', '));- }-- if (this._rollup) {- parts.push('WITH ROLLUP');- }-- // HAVING- if (this._having.length > 0) {- const conditions = this._having.map((h, i) => {- return i === 0 ? h.condition : `${h.operator} ${h.condition}`;- });- parts.push('HAVING', conditions.join(' '));- }-- // ORDER BY- if (this._orderBy.length > 0) {- const orderBy = this._orderBy.map((o) => {- const col = o.column;- return `${col} ${o.direction}`;- });- parts.push('ORDER BY', orderBy.join(', '));- }-- // Add FILL clause after ORDER BY- if (this._fill) {- const fromDate =- this._fill.from instanceof Date- ? clix.datetime(this._fill.from)- : this._fill.from;- const toDate =- this._fill.to instanceof Date- ? clix.datetime(this._fill.to)- : this._fill.to;-- parts.push('WITH FILL');- parts.push(`FROM ${fromDate}`);- parts.push(`TO ${toDate}`);- parts.push(`STEP ${this._fill.step}`);- }-- // LIMIT & OFFSET- if (this._limit !== undefined) {- parts.push(`LIMIT ${this._limit}`);- if (this._offset !== undefined) {- parts.push(`OFFSET ${this._offset}`);- }- }-- // SETTINGS- if (Object.keys(this._settings).length > 0) {- const settings = Object.entries(this._settings)- .map(([key, value]) => `${key} = ${value}`)- .join(', ');- parts.push(`SETTINGS ${settings}`);- }-- if (this._union) {- parts.push('UNION ALL', this._union.buildQuery());- }-- return parts.join(' ');- }-- transformJson<E extends ResponseJSON<any>>(json: E): E {- const keys = Object.keys(json.data[0] || {});- const response = {- ...json,- data: json.data.map((item) => {- return keys.reduce((acc, key) => {- const meta = json.meta?.find((m) => m.name === key);- const transformer = this._transform?.[key];-- if (transformer) {- return {- ...acc,- [key]: transformer(item),- };- }-- return {- ...acc,- [key]:- item[key] && meta?.type.includes('Int')- ? Number.parseFloat(item[key] as string)- : item[key],- };- }, {} as T);- }),- };- return response;- }-- transform(transformations: Record<string, (item: T) => any>): this {- this._transform = transformations;- return this;- }-- // Execution methods- async execute(): Promise<T[]> {- const query = this.buildQuery();- // console.log(- // 'query',- // `${query.replaceAll('\n', ' ').replaceAll('\t', ' ').replaceAll('\r', ' ')} SETTINGS session_timezone = '${this.timezone}'`,- // );-- const result = await this.client.query({- query,- clickhouse_settings: {- session_timezone: this.timezone,- },- });- const json = await result.json<T>();- return this.transformJson(json).data;- }-- // Debug methods- toSQL(): string {- return this.buildQuery();- }-- // Add method to add where conditions (for internal use)- _addWhereCondition(condition: WhereCondition): this {- this._where.push(condition);- return this;- }-- if(condition: any): this {- this._skipNext = !condition;- return this;- }-- endIf(): this {- this._skipNext = false;- return this;- }-- // Add method for callback-style conditionals- when(condition: boolean, callback?: ConditionalCallback): this {- if (condition && callback) {- callback(this);- }- return this;- }-- clone(): Query<T> {- return new Query(this.client, this.timezone).merge(this);- }-- // Add merge method- merge(query: Query): this {- if (this._skipNext) {- return this;- }-- this._from = query._from;-- this._select = [...this._select, ...query._select];-- this._except = [...this._except, ...query._except];-- // Merge WHERE conditions- this._where = [...this._where, ...query._where];-- // Merge CTEs- this._ctes = [...this._ctes, ...query._ctes];-- // Merge JOINS- this._joins = [...this._joins, ...query._joins];- this._rawJoins = [...this._rawJoins, ...query._rawJoins];-- // Merge settings- this._settings = { ...this._settings, ...query._settings };-- // Take the most restrictive LIMIT- if (query._limit !== undefined) {- this._limit =- this._limit === undefined- ? query._limit- : Math.min(this._limit, query._limit);- }-- // Merge ORDER BY- this._orderBy = [...this._orderBy, ...query._orderBy];-- // Merge GROUP BY- this._groupBy = [...this._groupBy, ...query._groupBy];-- // Merge HAVING conditions- this._having = [...this._having, ...query._having];-- return this;- }-}--// Add this new class for building where groups-export class WhereGroupBuilder {- private conditions: WhereCondition[] = [];-- constructor(- private query: Query,- private groupOperator: 'AND' | 'OR'- ) {}-- where(column: string, operator: Operator, value?: SqlParam): this {- const condition = this.query.buildCondition(column, operator, value);- this.conditions.push({ condition, operator: 'AND' });- return this;- }-- andWhere(column: string, operator: Operator, value?: SqlParam): this {- const condition = this.query.buildCondition(column, operator, value);- this.conditions.push({ condition, operator: 'AND' });- return this;- }-- rawWhere(condition: string): this {- this.conditions.push({ condition, operator: 'AND' });- return this;- }-- orWhere(column: string, operator: Operator, value?: SqlParam): this {- const condition = this.query.buildCondition(column, operator, value);- this.conditions.push({ condition, operator: 'OR' });- return this;- }-- end(): Query {- const groupCondition = this.conditions- .map((c, i) => (i === 0 ? c.condition : `${c.operator} ${c.condition}`))- .join(' ');-- this.query._addWhereCondition({- condition: groupCondition,- operator: this.groupOperator,- isGroup: true,- });-- return this.query;- }-}--// Helper function to create a new query-export function clix(client: ClickHouseClient, timezone?: string): Query {- return new Query(client, timezone ?? 'UTC');-}--clix.exp = (expr: string | Query<any>) =>- new Expression(expr instanceof Query ? expr.toSQL() : expr);-clix.date = (date: string | Date, wrapper?: string) => {- const dateStr = new Date(date).toISOString().slice(0, 10);- return wrapper ? `${wrapper}(${dateStr})` : dateStr;-};-clix.datetime = (date: string | Date, wrapper?: string) => {- const datetime = new Date(date).toISOString().slice(0, 19).replace('T', ' ');- return wrapper ? `${wrapper}(${datetime})` : datetime;-};-clix.dynamicDatetime = (date: string | Date, interval: IInterval) => {- if (interval === 'month' || interval === 'week') {- return clix.date(date);- }- return clix.datetime(date);-};--clix.toStartOf = (node: string, interval: IInterval, timezone?: string) => {- switch (interval) {- case 'minute': {- return `toStartOfMinute(${node})`;- }- case 'hour': {- return `toStartOfHour(${node})`;- }- case 'day': {- return `toStartOfDay(${node})`;- }- case 'week': {- return `toStartOfWeek(toDateTime(${node}))`;- }- case 'month': {- return `toStartOfMonth(toDateTime(${node}))`;- }- }-};-clix.toStartOfInterval = (- node: string,- interval: IInterval,- origin: string | Date-) => {- switch (interval) {- case 'minute': {- return `toStartOfInterval(toDateTime(${node}), INTERVAL 1 minute, toDateTime(${clix.datetime(origin)}))`;- }- case 'hour': {- return `toStartOfInterval(toDateTime(${node}), INTERVAL 1 hour, toDateTime(${clix.datetime(origin)}))`;- }- case 'day': {- return `toStartOfInterval(toDateTime(${node}), INTERVAL 1 day, toDateTime(${clix.datetime(origin)}))`;- }- case 'week': {- return `toStartOfInterval(toDateTime(${node}), INTERVAL 1 week, toDateTime(${clix.datetime(origin)}))`;- }- case 'month': {- return `toStartOfInterval(toDateTime(${node}), INTERVAL 1 month, toDateTime(${clix.datetime(origin)}))`;- }- }-};-clix.toInterval = (node: string, interval: IInterval) => {- switch (interval) {- case 'minute': {- return `toIntervalMinute(${node})`;- }- case 'hour': {- return `toIntervalHour(${node})`;- }- case 'day': {- return `toIntervalDay(${node})`;- }- case 'week': {- return `toIntervalWeek(${node})`;- }- case 'month': {- return `toIntervalMonth(${node})`;- }- }-};-clix.toDate = (node: string, interval: IInterval) => {- switch (interval) {- case 'day':- case 'week':- case 'month': {- return `toDate(${node})`;- }- default: {- return `toDateTime(${node})`;- }- }-};-// Export types-export type { SqlValue, SqlParam, Operator };diff --git a/packages/db/src/clickhouse/sql.test.ts b/packages/db/src/clickhouse/sql.test.tsindex 91a65b5b5..08e315c05 100644--- a/packages/db/src/clickhouse/sql.test.ts+++ b/packages/db/src/clickhouse/sql.test.ts@@ -247,6 +247,62 @@ describe('injection (ported from query-builder.test.ts)', () => {expect(query).toBe('SELECT * FROM events WHERE e.path = {p1:String}');expect(query_params.p1).toBe('{p1:String}');});++ // M12-009 completed the port: the four cases below were still only asserted+ // against clix when query-builder.test.ts was deleted. Each keeps that+ // test's payload; what changes is the mechanism the payload proves —+ // clix asserted on the escaped literal it produced, the tag asserts the+ // value never reaches the text at all.++ it('binds a plain ISO date string instead of quoting it', () => {+ const { query, query_params } =+ sql`SELECT * FROM events WHERE e.created_at = ${sql.string('2026-04-29 00:00:00')}`.toStatement();++ expect(query).toBe('SELECT * FROM events WHERE e.created_at = {p1:String}');+ expect(query_params.p1).toBe('2026-04-29 00:00:00');+ });++ it('binds a Date as a datetime value, not a literal in the text', () => {+ const { query, query_params } =+ sql`SELECT * FROM events WHERE e.created_at = ${sql.dateTime64('2026-04-29 00:00:00.000')}`.toStatement();++ expect(query).toBe(+ 'SELECT * FROM events WHERE e.created_at = {p1:DateTime64(3)}'+ );+ expect(query_params.p1).toBe('2026-04-29 00:00:00.000');+ });++ it('keeps a toDateTime() wrapper as SQL text with the value bound', () => {+ // clix.datetime(date, 'toDateTime') produced the call AND the quoted+ // literal in one string. The wrapper is structure and stays raw; the+ // value is a value and binds.+ const { query, query_params } =+ sql`SELECT * FROM events WHERE e.created_at = toDateTime(${sql.string('2026-04-29 00:00:00')})`.toStatement();++ expect(query).toBe(+ 'SELECT * FROM events WHERE e.created_at = toDateTime({p1:String})'+ );+ expect(query_params.p1).toBe('2026-04-29 00:00:00');+ });++ it('handles BETWEEN with two wrapped datetime values', () => {+ const { query, query_params } =+ sql`SELECT * FROM events WHERE e.created_at BETWEEN toDateTime(${sql.string('2026-04-29 00:00:00')}) AND toDateTime(${sql.string('2026-05-07 00:00:00')})`.toStatement();++ expect(query).toBe(+ 'SELECT * FROM events WHERE e.created_at BETWEEN toDateTime({p1:String}) AND toDateTime({p2:String})'+ );+ expect(query_params.p1).toBe('2026-04-29 00:00:00');+ expect(query_params.p2).toBe('2026-05-07 00:00:00');+ });++ it('does not promote an arbitrary string with an embedded date to SQL', () => {+ const { query, query_params } = bind('event-2026-04-15-launch');++ expect(query).toBe('SELECT * FROM events WHERE e.path = {p1:String}');+ expect(query).not.toContain('2026-04-15');+ expect(query_params.p1).toBe('event-2026-04-15-launch');+ });});describe('param constructors', () => {diff --git a/packages/db/src/clickhouse/sql.ts b/packages/db/src/clickhouse/sql.tsindex b78b31648..32a70e673 100644--- a/packages/db/src/clickhouse/sql.ts+++ b/packages/db/src/clickhouse/sql.ts@@ -18,9 +18,17 @@* Placeholder names are assigned at render time from a single counter, so* fragments nest and repeat without colliding (R2).*- * Not re-exported from the package barrel during the clix coexistence window:- * `query-builder.ts` exports its own `SqlParam` type, and `export *` from both- * is ambiguous. It joins the barrel when clix is deleted.+ * Exported from the package barrel (M12-009). It could not be during the clix+ * coexistence window — `query-builder.ts` exported its own `SqlParam` type and+ * `export *` from both was ambiguous — and that file is now deleted.+ *+ * `packages/core` still imports this module by its deep path rather than+ * through the barrel: the barrel also re-exports `clickhouse/client.ts` and+ * `prisma-client.ts`, which construct a ClickHouse client array and a+ * PrismaClient at module load, so a barrel import from core would acquire a+ * second, request-scope-less client. `.dependency-cruiser.cjs`'s+ * `core-uses-ctx-not-db-internals` encodes exactly that, exempting this file+ * by path and not the barrel.*//** Auto-generated placeholders are `{p1:Type}`, `{p2:Type}`, … */diff --git a/packages/db/src/sql-builder.ts b/packages/db/src/sql-builder.tsdeleted file mode 100644index d20d85dac..000000000--- a/packages/db/src/sql-builder.ts+++ /dev/null@@ -1,94 +0,0 @@-import { TABLE_NAMES } from './clickhouse/client';--export interface SqlBuilderObject {- where: Record<string, string>;- having: Record<string, string>;- select: Record<string, string>;- groupBy: Record<string, string>;- orderBy: Record<string, string>;- from: string;- joins: Record<string, string>;- ctes: Record<string, string>;- limit: number | undefined;- offset: number | undefined;- fill: string | undefined;-}--export function createSqlBuilder() {- const join = (obj: Record<string, string> | string[], joiner: string) =>- Object.values(obj).filter(Boolean).join(joiner);-- const sb: SqlBuilderObject = {- where: {},- from: `${TABLE_NAMES.events} e`,- select: {},- groupBy: {},- orderBy: {},- having: {},- joins: {},- ctes: {},- limit: undefined,- offset: undefined,- fill: undefined,- };-- const getWhere = () =>- Object.keys(sb.where).length ? `WHERE ${join(sb.where, ' AND ')}` : '';- const getHaving = () =>- Object.keys(sb.having).length ? `HAVING ${join(sb.having, ' AND ')}` : '';- const getFrom = () => `FROM ${sb.from}`;- const getSelect = () =>- `SELECT ${Object.keys(sb.select).length ? join(sb.select, ', ') : '*'}`;- const getGroupBy = () =>- Object.keys(sb.groupBy).length ? `GROUP BY ${join(sb.groupBy, ', ')}` : '';- const getOrderBy = () =>- Object.keys(sb.orderBy).length ? `ORDER BY ${join(sb.orderBy, ', ')}` : '';- const getLimit = () => (sb.limit ? `LIMIT ${sb.limit}` : '');- const getOffset = () => (sb.offset ? `OFFSET ${sb.offset}` : '');- const getJoins = () =>- Object.keys(sb.joins).length ? join(sb.joins, ' ') : '';- const getFill = () => (sb.fill ? `WITH FILL ${sb.fill}` : '');- const getWith = () => {- const cteEntries = Object.entries(sb.ctes);- if (cteEntries.length === 0) return '';- const cteClauses = cteEntries.map(- ([name, query]) => `${name} AS (${query})`,- );- return `WITH ${cteClauses.join(', ')} `;- };-- return {- sb,- join,- getWhere,- getFrom,- getSelect,- getGroupBy,- getOrderBy,- getHaving,- getJoins,- getFill,- getWith,- with: (name: string, query: string) => {- sb.ctes[name] = query;- },- getSql: () => {- const sql = [- getWith(),- getSelect(),- getFrom(),- getJoins(),- getWhere(),- getGroupBy(),- getHaving(),- getOrderBy(),- getLimit(),- getOffset(),- getFill(),- ]- .filter(Boolean)- .join(' ');- return sql;- },- };-}diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex dea0301f8..d6829d42c 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -658,9 +658,6 @@ importers:sonner:specifier: ^1.4.0- sqlstring:- specifier: ^2.3.3- version: 2.3.3superjson:specifier: ^2.2.2version: 2.2.2@@ -939,9 +936,6 @@ importers:slugify:specifier: ^1.6.6version: 1.6.9- sqlstring:- specifier: ^2.3.3- version: 2.3.3superjson:specifier: ^1.13.3version: 1.13.3@@ -982,9 +976,6 @@ importers:'@types/ramda':specifier: ^0.31.1version: 0.31.1- '@types/sqlstring':- specifier: ^2.3.2- version: 2.3.2'@types/unzipper':specifier: ^0.10.10version: 0.10.11@@ -1033,9 +1024,6 @@ importers:prisma-json-types-generator:specifier: ^3.1.1- sqlstring:- specifier: ^2.3.3- version: 2.3.3zod:specifier: 'catalog:'version: 4.3.6@@ -1046,9 +1034,6 @@ importers:'@types/node':specifier: 'catalog:'version: 24.10.1- '@types/sqlstring':- specifier: ^2.3.2- version: 2.3.2prisma:specifier: ^6.14.0version: 6.14.0([email protected])([email protected])@@ -9276,9 +9261,6 @@ packages:'@types/[email protected]':resolution: {integrity: sha512-5JcVt1u5HDmlXkwOD2nslZVllBBc7HDuOICfiZah2Z0is8M8g+ddAEawbmd3VjedfDHBzxCaXLs07QEmb7y54g==}- '@types/[email protected]':- resolution: {integrity: sha512-lVRe4Iz9UNgiHelKVo8QlC8fb5nfY8+p+jNQNE+UVsuuVlQnWhyWmQ/wF5pE8Ys6TdjfVpqTG9O9i2vi6E0+Sg==}-'@types/[email protected]':resolution: {integrity: sha512-9aEbYZ3TbYMznPdcdr3SmIrLXwC/AKZXQeCf9Pgao5CKb8CyHuEX5jzWPTkvregvhRJHcpRO6BFoGW9ycaOkYw==}@@ -16595,10 +16577,6 @@ packages:resolution: {integrity: sha512-D9cPgkvLlV3t3IzL0D0YLvGA9Ahk4PcvVwUbN0dSGr1aP0Nrt4AEnTUbuGvquEC0mA64Gqt1fzirlRs5ibXx8g==}- resolution: {integrity: sha512-qC9iz2FlN7DQl3+wjwn3802RTyjCx7sDvfQEXchwa6CWOx07/WVfh91gBmQ9fahw8snwGEWU3xGzOt4tFyHLxg==}- engines: {node: '>= 0.6'}-resolution: {integrity: sha512-poPs1GuctQLpiJ/1Pb8e+5b5lju9hQU7wxJ6NkYVUw7ZZExeRoYwyiaOekal+rDZc99MO/J2y9+SGFpHBKRSpQ==}engines: {node: '>=20.16.0'}@@ -28827,8 +28805,6 @@ snapshots:'@types/[email protected]': {}- '@types/[email protected]': {}-'@types/[email protected]': {}'@types/[email protected]': {}@@ -38310,8 +38286,6 @@ snapshots:- [email protected]: {}-dependencies:cookie-es: 2.0.1