OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.
ralph(M38-001): Explicit PREWHERE on profilesByIdsQuery
ralpha5de15cc5f38
diff --git a/packages/core/src/modules/profile/src/profile.sql.proof.md b/packages/core/src/modules/profile/src/profile.sql.proof.mdindex 6e1290b0e..3f54c85ad 100644--- a/packages/core/src/modules/profile/src/profile.sql.proof.md+++ b/packages/core/src/modules/profile/src/profile.sql.proof.md@@ -426,3 +426,94 @@ reports a difference that is not a change.spelling. Both routes now accept `range` / `startDate` / `endDate`, and the default is `3m`, thesame as here. The default is applied on the routes only: the MCP tools that call the sameservice without dates still read all time. The proof is in `chart/src/retention.sql.proof.md`, section "M34-002".++## M38-001 — explicit PREWHERE on `profilesByIdsQuery` (2026-09-16)++`FINAL` turns off automatic PREWHERE (`optimize_move_to_prewhere_if_final = 0` on this server), so+the V1 shape read all 11 columns for every row in the project's key range. The statement now reads:++```sql+SELECT id, first_name, last_name, email, avatar, properties, project_id, is_external, created_at, last_seen_at, groups FROM profiles FINAL PREWHERE project_id = {p1:String} AND id IN {p2:Array(String)}+```++The setting was **not** changed server-wide or in `CLICKHOUSE_SETTINGS`. The PREWHERE is on this one+statement only. Both conditions use sort-key columns only, and every version of a+`ReplacingMergeTree` row shares them, so filtering before the `FINAL` merge drops no version that+the merge would keep. The `IN` takes a bound literal array, not a subquery, so `GLOBAL IN` does not+apply. On a `Distributed` table, the shards receive the PREWHERE with the rest of the statement, and+`FINAL` stays per shard, as before.++**Method.** 2026-09-16, 17:40–17:55 UTC, single node, ClickHouse 26.1.3.52, prod copy `openpanel`+(SELECT only), HTTP with `use_query_cache=0` and the client's settings. The ids were bound as+`{p2:Array(String)}`, the same as production. **The page cache was warm and not controlled.**+Every pair was compared three ways: the row count, sha256 of the sorted `FORMAT TSVRaw` output,+and `cityHash64(arraySort(groupArray(cityHash64(tuple(*)))))` computed on the server. The hashes+are truncated to 16 hex characters. No rows left the box or were printed.++**Result sets: identical in all 18 prod-copy pairs and all 9 duplicate-version pairs.** Only the+unordered row order differs. The statement has no `ORDER BY`, and all six `getProfiles` /+`getProfilesCached` callers index the result by `id` (see "Callers" below).++| project | id set | rows | sorted sha (old = new) | server hash (old = new) |+|---|---|--:|---|--:|+| verdict | 200 sampled + 2 missing | 200 | 990ff37720bbd9ee | 2207503090764657581 |+| verdict | 1,000 sampled | 1000 | 52f97c198baf11aa | 10633293728532801169 |+| verdict | 2026-08-20 bucket, first 200 | 113 | 89e5dfe568bc840f | 3782160794143776797 |+| verdict | 2026-08-20 bucket, ≤1,000 | 557 | f923bf2f66e0f529 | 9998950015958726181 |+| bayse | 200 sampled + 2 missing | 200 | a3c5fa45f03e566f | 5808497201729836461 |+| bayse | 1,000 sampled | 1000 | 6bccde35694a0d5d | 15599780444145327832 |+| bayse | 2026-08-20 bucket, first 200 | 67 | 787b5b61b92fab5e | 15475558395705370151 |+| bayse | 2026-08-20 bucket, ≤1,000 | 476 | ab1ba0dac490c5f6 | 17356756279215469892 |+| earlysalary-production | 200 sampled + 2 missing | 200 | c672481141af7754 | 10996104250063144912 |+| earlysalary-production | 1,000 sampled | 1000 | e385c4fe9b40e97b | 4868776334728909166 |+| earlysalary-production | 2026-08-20 bucket, first 200 | 195 | 7fd9304dd609e859 | 1770102371015330504 |+| earlysalary-production | 2026-08-20 bucket, ≤1,000 | 974 | f1ad3f126d5466aa | 16534687301997138277 |+| website-8103 | 200 sampled + 2 missing | 200 | fc35ccc87636163b | 14505134278598945611 |+| website-8103 | 1,000 sampled | 1000 | 5a906205e04747c6 | 6368711441762070328 |+| chatpaper | 200 sampled + 2 missing | 200 | caf0a176212258cb | 9471910246293331129 |+| chatpaper | 1,000 sampled | 1000 | 5c278c8382b2413b | 13601208562850535735 |+| chatpaper | `screen_view` 2026-08-20 bucket, first 200 | 200 | aa019808e0770ee3 | 17930342762468771172 |+| chatpaper | `screen_view` 2026-08-20 bucket, 1,000 | 1000 | 01cb23d19b21b6e9 | 12578565101048520051 |++Key to the id sets. "Sampled" means `ORDER BY cityHash64(id) LIMIT n` over the project's profiles.+"Missing" means one unknown id and one quote-injection string. "Bucket" means+`DISTINCT profile_id` from `events` on that day. `website-8103` has no non-empty bucket ids.++**Duplicate versions.** The prod copy has no un-merged duplicates (M37-002 §4), so it cannot+exercise the `FINAL` merge. A scratch database, `openpanel_m38`, held a copy of the `profiles` DDL+with **synthetic** rows only: 5 inserts with merges stopped, 20 parts, and 75,000 rows over 15,000+`(project_id, id)` keys. Each key had 5 versions, with `created_at` spread across 4 partitions and+tied `last_seen_at` values. Old and new returned the same result for 200, 1,000 and 5,000 ids in each+of 3 projects:++| project | ids | rows | sorted sha (old = new) |+|---|--:|--:|---|+| proj-a | 201 / 1000 / 5000 | 200 / 1000 / 5000 | e0eb58de4d3d5c46 / 715b257a3034cabb / f0394f87bd48b50a |+| proj-b | 201 / 1000 / 5000 | 200 / 1000 / 5000 | 0027b01ee499da6f / d1da354936467673 / 881ec0da794c33ca |+| proj-c | 201 / 1000 / 5000 | 200 / 1000 / 5000 | c32dab9e4f45d2a5 / 4e594a31dd87bf2f / 2f73bcde4061e82e |++**Cost** (`X-ClickHouse-Summary`, `FORMAT Null`, warm cache). `chatpaper` runs 3 times each and the+other projects once:++| project / id set | read_rows (old = new) | bytes old → new | ClickHouse ms old → new |+|---|--:|--:|--:|+| chatpaper, bucket 200 | 1,809,792 | 686.2 → **191.4 MiB** | 278–280 → **105–108** |+| chatpaper, bucket 1,000 | 1,850,752 | 701.6 → **196.3 MiB** | 285–287 → **111–117** |+| chatpaper, 200 sampled | 1,793,408 | 680.0 → 372.2 MiB | 275–281 → 159–167 |+| chatpaper, 1,000 sampled | 1,850,752 | 701.6 → 663.8 MiB | 290–298 → 217–220 |+| verdict, bucket ≤1,000 | 916,864 | 379.8 → 136.1 MiB | 149 → 67 |+| bayse, bucket ≤1,000 | 400,928 | 197.8 → 149.5 MiB | 86 → 63 |+| earlysalary-production, bucket ≤1,000 | 959,588 | 464.6 → 125.5 MiB | 169 → 62 |+| website-8103, 1,000 sampled | 57,184 | 27.2 → 13.5 MiB | 19 → 15 |++These reproduce M37-002's figures of 683 → 194 MiB and 702 → 196 MiB. As predicted, `read_rows` does+not fall. The bytes saved depend on the id set. Ids drawn uniformly by hash across the whole table+save less: 1,000 sampled `chatpaper` ids read 664 MiB. The likely cause is that those ids touch more+parts and partitions, so more wide-column ranges are read. This was not verified. No case read more.++**Callers.** `getProfiles` / `getProfilesCached` are called from `event.service.ts:433`+(`attachProfiles`, Map by id), `session.service.ts:296` (Map by id), `cohort.service.ts:919` and+`group.service.ts:355` (Map by id, re-ordered by the page's ids), `realtime.service.ts:365` (Map by+id) and `profile.service.ts:581` (`find` by id, in `powerUsers` row order). None of these reads the+statement's row order. The chart drill-down (`chart.service.ts:213`, `getProfilesInBatches`)+flattens batches in ClickHouse order, and that order was never defined (M37-002 §5).diff --git a/packages/core/src/modules/profile/src/sql.test.ts b/packages/core/src/modules/profile/src/sql.test.tsindex 9b5afe940..446380ceb 100644--- a/packages/core/src/modules/profile/src/sql.test.ts+++ b/packages/core/src/modules/profile/src/sql.test.ts@@ -35,6 +35,7 @@ const WINDOW = {endDate: '2026-09-16 23:59:59',};const COLUMN_LIST = PROFILE_COLUMNS.join(', ');+const PLAIN_WHERE = /\sWHERE\s/;function collapse(text: string): string {return text.replace(/\s+/g, ' ').trim();@@ -99,15 +100,16 @@ describe('single-profile lookups', () => {expect(row.query_params).toEqual({ p1: PROJECT_ID, p2: HOSTILE });});- test('profilesByIdsQuery binds the id list as one Array(String) param', () => {+ test('profilesByIdsQuery filters by sort key in PREWHERE, binding ids as one Array(String)', () => {const { query, query_params } = profilesByIdsQuery({projectId: PROJECT_ID,ids: ['a', HOSTILE],}).toStatement();expect(collapse(query)).toContain(- 'WHERE project_id = {p1:String} AND id IN {p2:Array(String)}'+ 'FROM profiles FINAL PREWHERE project_id = {p1:String} AND id IN {p2:Array(String)}');+ expect(query).not.toMatch(PLAIN_WHERE);expect(query).not.toContain(HOSTILE);expect(query_params).toEqual({ p1: PROJECT_ID, p2: ['a', HOSTILE] });});diff --git a/packages/core/src/modules/profile/src/sql.ts b/packages/core/src/modules/profile/src/sql.tsindex db445a893..f7081edcd 100644--- a/packages/core/src/modules/profile/src/sql.ts+++ b/packages/core/src/modules/profile/src/sql.ts@@ -161,10 +161,13 @@ export function profilesByIdsQuery(query: {projectId: string;ids: readonly string[];}): SqlFragment {+ // FINAL turns off automatic PREWHERE, so without it every column (mostly+ // `properties`) is read for the whole project range. Safe before the FINAL+ // merge: both conditions are sort-key columns, shared by every row version.return sql`SELECT ${PROFILE_COLUMNS_FRAGMENT}FROM ${sql.id(TABLE.profiles)} FINAL- WHERE+ PREWHEREproject_id = ${sql.string(query.projectId)} ANDid IN ${sql.array('String', query.ids)}`;