Skip to content

OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.

ralph(M39-001): sessionByIdQuery: explicit PREWHERE, keep FINAL

ralph200463f969b4
diff --git a/packages/core/src/modules/session/src/sql.proof.md b/packages/core/src/modules/session/src/sql.proof.md
new file mode 100644
index 000000000..cf002947e
--- /dev/null
+++ b/packages/core/src/modules/session/src/sql.proof.md
@@ -0,0 +1,73 @@
+# session/src/sql.ts — `sessionByIdQuery` PREWHERE proof (M39-001)
+
+`sessionByIdQuery` keeps `FINAL`. It now reads
+`FROM sessions FINAL PREWHERE id = … AND project_id = … WHERE sign = 1`; before, all three
+conditions were in `WHERE`. This is rank 1 of `docs/FINAL_USAGE.md` §7 (M38-002).
+
+**Why it helps.** On this server `optimize_move_to_prewhere_if_final = 0`, and `FINAL` widens the
+`idx_id` bloom-filter hit (`use_skip_indexes_if_final_exact_mode = 1`). With `FINAL`, every column
+was read for every widened granule. `EXPLAIN indexes=1` is the same for both spellings: bayse
+`idx_id` Parts 2/9, Granules 4/101, then `FINAL` widens this to Parts 9/2, Granules 173/4. The
+statements differ only in which columns are read for those granules. `EXPLAIN actions=1` on the
+new statement shows `Prewhere filter column: and(equals(id, …), equals(project_id, …))`, and the
+outer `Filter column: equals(__table1.sign, 1_UInt8)`.
+
+**Why it is safe.** `project_id` is a sort-key column. `id` is not, but a −1 row is a full copy
+of its +1 (`buffers/session-buffer.ts`), so both rows of a pair pass the PREWHERE and still
+collapse. **`sign` stays in `WHERE`.** A PREWHERE on `sign` drops the −1 before the collapse, so
+the stale +1 survives (control below).
+
+## Setup
+
+- **When and where:** 2026-09-16, 21:30–21:34 UTC, on the local prod-copy ClickHouse
+ (`127.0.0.1:8123`, database `openpanel`), with load average 2.9–3.9.
+- **Access:** SELECT and EXPLAIN only. `openpanel.events` read 321,192,117 rows.
+- **Request settings:** every request used `use_query_cache=0`, `use_query_condition_cache=0` and
+ `wait_end_of_query=1`. Ids were bound as `{p1:String}`/`{p2:String}`, exactly as the builder
+ renders them.
+- **Where the numbers come from:** `read_rows`, `read_bytes`, `elapsed_ns` and `memory_usage` are
+ read off `X-ClickHouse-Summary`, from `FORMAT Null` runs. `memory_usage` is the summary figure,
+ not a sampled peak.
+- **Caching:** page cache was not controlled, so **every number is warm**.
+
+## Result-set equality
+
+Each check runs `SELECT count(), sum(h), cityHash64(groupArray(h)) FROM (SELECT
+sipHash64(formatRow('TSV', *)) h FROM (<statement>))`. `sum(h)` is the order-independent
+result-set hash.
+
+| project | ids checked | old = new |
+|---|---|--:|
+| verdict | top 3 by row count + 5 `cityHash64(id,'m39')` samples | 8/8 |
+| bayse | same | 8/8 |
+| earlysalary-production | same | 8/8 |
+| chatpaper | same | 8/8 |
+| e2e-sessions | same (top id: 4 rows) | 8/8 |
+| bayse | the id that reads the most (1,613,380 rows; M38-002's case) | 1/1 (2 rows) |
+| website-8103 | no `sessions` rows on this box; unknown id | 0 rows = 0 rows |
+| **e2e-sessions, unmerged** | **every id with a −1 row (21 ids, 21 −1 rows)** | **21/21** |
+| e2e-sessions | 50 sampled ids with >1 `version` | 50/50 |
+
+Controls on the 21 unmerged ids, which show that this probe can detect a wrong answer:
+- **No `FINAL`** differs on **21/21**: 2 rows instead of 1, because the stale +1 is returned.
+- **`FINAL PREWHERE id, project_id, sign = 1`** also differs on **21/21**, with the same hashes as
+ no `FINAL`.
+
+## Cost
+
+Each variant was run 4 times on each project's top id (ordered by row count, then by id). The
+bayse row uses the widest id instead. Each cell shows `read_rows`, then MiB read, the four run
+times in ms, and `memory_usage` in MiB.
+
+| project | old (`FINAL WHERE`) | new (`FINAL PREWHERE`) |
+|---|---|---|
+| **bayse (widest id)** | 1,613,380 · 364 MiB · **858/808/820/819 ms** · 432–475 MiB | 1,613,380 · 39 MiB · **35/32/42/28 ms** · 26 MiB |
+| bayse (top id) | 90,109 · 21 · 87/106/81/88 · 66–70 | 90,109 · 4 · 17/16/15/16 · 17 |
+| chatpaper | 5,913,071 · 1,308 · 3124/2912/2900/3072 · 438–450 | 5,913,071 · 148 · 75/55/55/56 · 34–38 |
+| verdict | 146,882 · 28 · 99/113/114/104 · 82–94 | 146,882 · 4 · 17/20/26/20 · 17 |
+| earlysalary-production | 545,014 · 149 · 314/308/339/328 · 265–299 | 545,014 · 15 · 28/25/28/26 · 25 |
+| e2e-sessions | 141,898 · 35 · 108/96/96/99 · 96–120 | 141,898 · 5 · 16/17/18/16 · 17 |
+
+M38-002 measured bayse at 846–994 ms → 32–36 ms and 363 → 38 MiB. This run reproduces those
+figures: 808–858 ms → 28–42 ms and 364 → 39 MiB. `read_rows` does not change, because `FINAL`
+still widens the ranges. Only the `id`/key columns are read for rows that the PREWHERE rejects.
diff --git a/packages/core/src/modules/session/src/sql.test.ts b/packages/core/src/modules/session/src/sql.test.ts
index 5dac08717..9725859c2 100644
--- a/packages/core/src/modules/session/src/sql.test.ts
+++ b/packages/core/src/modules/session/src/sql.test.ts
@@ -149,13 +149,13 @@ describe('replay and byId', () => {
});
});
- test('sessionByIdQuery and sessionHasReplayQuery keep V1 text', () => {
+ test('sessionByIdQuery prewheres the id and keeps sign under FINAL; sessionHasReplayQuery keeps V1 text', () => {
const byId = sessionByIdQuery({
sessionId: 'sess-1',
projectId: PROJECT_ID,
}).toStatement();
expect(collapse(byId.query)).toBe(
- 'SELECT * FROM sessions FINAL WHERE id = {p1:String} AND project_id = {p2:String} AND sign = 1'
+ 'SELECT * FROM sessions FINAL PREWHERE id = {p1:String} AND project_id = {p2:String} WHERE sign = 1'
);
const hasReplay = sessionHasReplayQuery({
diff --git a/packages/core/src/modules/session/src/sql.ts b/packages/core/src/modules/session/src/sql.ts
index 75d4d915e..059c5d288 100644
--- a/packages/core/src/modules/session/src/sql.ts
+++ b/packages/core/src/modules/session/src/sql.ts
@@ -237,12 +237,16 @@ export function sessionByIdQuery(query: {
sessionId: string;
projectId: string;
}): SqlFragment {
+ // FINAL turns off automatic PREWHERE, so without it every column is read for
+ // every granule FINAL widens the id's skip-index hit to. Safe before the
+ // merge: a -1 row is a copy of its +1, so both pass. `sign` must stay in
+ // WHERE — filtering it first drops the -1 and resurrects the stale +1.
return sql`
SELECT *
FROM ${sql.id(TABLE.sessions)} FINAL
- WHERE id = ${sql.string(query.sessionId)}
+ PREWHERE id = ${sql.string(query.sessionId)}
AND project_id = ${sql.string(query.projectId)}
- AND sign = 1
+ WHERE sign = 1
`;
}