OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.
ralph(M39-004): sessionListQuery: drop FINAL, add the collapse-aware anti-join
ralph4ad2cd9d966d
diff --git a/.m39-004-full.log b/.m39-004-full.lognew file mode 100644index 000000000..b2af8a96a--- /dev/null+++ b/.m39-004-full.log@@ -0,0 +1,11 @@+OK: no forbidden patterns+== bun install --frozen-lockfile (/home/deploy/openpanel) ==+[0.05ms] ".env"+bun install v1.4.0 (34cbb9a40)++Checked 3036 installs across 3405 packages (no changes) [126.00ms]+OK: bun install+== build api: skipped (V1 fastify boot deleted; apps/api runs from source) ==+== typecheck api (/home/deploy/openpanel/apps/api) ==+$ tsc --noEmit+error: script "typecheck" was terminated by signal SIGTERM (Polite quit request)diff --git a/packages/core/src/modules/session/src/sql.proof.md b/packages/core/src/modules/session/src/sql.proof.mdindex cf002947e..5c4445641 100644--- a/packages/core/src/modules/session/src/sql.proof.md+++ b/packages/core/src/modules/session/src/sql.proof.md@@ -71,3 +71,130 @@ times in ms, and `memory_usage` in MiB.M38-002 measured bayse at 846–994 ms → 32–36 ms and 363 → 38 MiB. This run reproduces thosefigures: 808–858 ms → 28–42 ms and 364 → 39 MiB. `read_rows` does not change, because `FINAL`still widens the ranges. Only the `id`/key columns are read for rows that the PREWHERE rejects.++# session/src/sql.ts — `sessionListQuery` without `FINAL` (M39-004)++`sessionListQuery` no longer reads `sessions FINAL`. It keeps `+1` rows only, and drops any row+whose `(id, version)` has a `−1` in the same window:++```sql+FROM sessions+LEFT JOIN (SELECT DISTINCT session_id FROM session_replay_chunks …) AS src ON src.session_id = id+WHERE project_id = … <lookback / cursor / calendar-day window>+ AND sign = 1+ AND (id, version) GLOBAL NOT IN (+ SELECT id, version FROM sessions+ WHERE project_id = … <the same window> AND sign = -1+ )+ <profile_id / search / compiled filters, unchanged>+ORDER BY created_at DESC+LIMIT …+```++This is rank 2 of `docs/FINAL_USAGE.md` §7 (M38-002).++**Why the subquery only repeats the window.** A `−1` is a full copy of its `+1`+(`buffers/session-buffer.ts`), so the two rows share `created_at` and fall in the same window.+The profile, search and filter predicates are applied to the `+1` in the outer query, so repeating+them in the subquery would not change the result.++**Cluster.** `sessions` and `session_replay_chunks` are `Distributed` on Cloud. `GLOBAL NOT IN`+builds the `−1` set once, on the initiator, and ships it to the shards. It is correct under any+`distributed_product_mode`. A plain `NOT IN` under the client's `'allow'` would run the subquery+against the Distributed table from every shard, which is also correct but costs N× more. It is+correct only because a `−1` and its `+1` share `created_at`, and so they share a shard: the shard+key is `cityHash64(project_id, toStartOfHour(created_at))`. On this single node the two spellings+cost the same (see the table below). Neither the replay `LEFT JOIN` nor its+`distributed_product_mode` behaviour changed.++## Setup++- **When and where:** 2026-09-16, 22:23–22:26 UTC, on the local prod-copy ClickHouse+ (`127.0.0.1:8123`, database `openpanel`, **single-node**). Load average was 0.1–1.2 at the start.+- **Access:** SELECT only.+- **How the statements were built:** both statements were rendered by the real builders, the+ HEAD `sql.ts` and the new one, with bound `{pN:Type}` params. The requests used the client's+ settings (`distributed_product_mode=allow`, `query_plan_convert_any_join_to_semi_or_anti_join=0`),+ plus `wait_end_of_query=1`, `use_query_cache=0` and `use_query_condition_cache=0`.+- **Where the numbers come from:** `read_rows`, `read_bytes`, `elapsed_ns` and `memory_usage` are+ read off `X-ClickHouse-Summary`, from `FORMAT Null` runs. `memory_usage` is the summary figure,+ not a sampled peak.+- **Caching:** page cache was not controlled, so **every number is warm**.+- **The replay join on this box:** `session_replay_chunks` holds 250 rows, all of them in+ `e2e-sessions`. For every real anchor, the join's right-hand side is therefore **empty**. The+ "all-replay" rows below replace it with *every* chatpaper session id, to bound the opposite+ case.++## Result-set equality++Each check runs `SELECT count(), sum(h), cityHash64(groupArray(h)) FROM (SELECT+sipHash64(formatRow('TSV', *)) h FROM (<statement>))`. The value compared is `(count, sum(h))`.+"FULL" means `take = 1e8`, so the whole window is returned. The top-N cases were also re-checked+with `ORDER BY created_at DESC, id`. The "control" column is the plain drop (`sign = 1`, no+anti-join); it shows whether a case can detect a wrong answer at all.++| project | cases (each: old = new?) | control = old? |+|---|---|---|+| chatpaper, verdict, bayse, earlysalary-production, website-8103 | cursor page (2026-08-30 12:00, 30 d) top 50 and FULL; first page anchored at now (30 d) top 50; date range 2026-08-01 → 08-15 top 50; cursor page + `search='a'` + filter `device='desktop'` top 50. **25/25 equal, plus 20/20 with the `id` tie-break.** (website-8103 has no sessions: 0 = 0. verdict search+filter: 0 = 0.) | equal everywhere, because the anchors hold no `−1` rows (§3 of FINAL_USAGE) |+| chatpaper, all-replay join | cursor page top 50: **equal** | — |+| **e2e-sessions (unmerged, 21 `−1` rows)** | cursor 2026-09-05 02:22:46 with 1 d lookback, top 50 and **top 500**; cursor 2026-09-15 23:59:59 with 30 d lookback, top 50 and **FULL** (647,079 rows); first page at now, **FULL**; date range September, **FULL**. **6/6 equal, plus 3/3 with the tie-break.** | **differs on top 500 and on all three FULL sets** (647,100 rows: the 21 stale `+1`) |++Mismatches between old and new: **0**.++## Cost: the whole statement, replay join included++Each cell shows `read_rows`, then MiB read, three runs in ms, and `memory_usage` in MiB. All pages+are cursor pages (2026-08-30 12:00, 30 d, top 50) unless the row says otherwise.++| project | old: `FINAL` + join | **new: anti-join + join** | new, local `NOT IN` | old, no join | new, no join | plain drop + join (unsafe) |+|---|---|---|---|---|---|---|+| chatpaper | 3,816,602 · 637 · **1147/1138/1124** · 210–241 | 7,289,150 · 586 · **453/473/473** · 73–81 | 464/485/455 | 1111/1148/1134 | 7,297,342 · 212 · 154/156/149 · 4 | 3,644,575 · 555 · 417/379/386 |+| chatpaper, range Aug 1–15 | 1,924,894 · 319 · **581/560/562** · 146–150 | 3,751,502 · 311 · **245/262/253** · 69–73 | 261/265/251 | 552/555/545 | 113 MiB · 88/87/94 | 212/204/209 |+| chatpaper, search + filter | 637 MiB · **1261/1240/1238** · 209–223 | 588 MiB · **561/577/579** · 77–82 | 590/571/583 | 1237/1212/1207 | 372 MiB · 351/330/344 | 485/510/490 |+| chatpaper, **all-replay join** | 9,844,399 · 896 · **2094/2550/2928** · 2208–2959 | 13,316,947 · 845 · **1933/1887/1644** · 2409–2806 | — | — | — | — |+| verdict | 1,006,503 · 121 · **292/304/299** · 174–186 | 1,673,436 · 83 · **96/98/98** · 61 | 95/89/86 | 279/281/296 | 44 MiB · 58/44/48 | 71/65/69 |+| bayse | 712,572 · 112 · **218/258/224** · 179–192 | 1,081,090 · 61 · **67/65/66** · 52–57 | 73/83/92 | 246/228/208 | 25 MiB · 34/30/31 | 52/47/50 |+| earlysalary-production | 745,388 · 163 · **246/255/278** · 207–246 | 1,146,732 · 96 · **97/76/75** · 65 | 83/85/95 | 251/245/254 | 34 MiB · 51/34/43 | 76/68/73 |+| website-8103 (0 sessions) | 55,634 · 11 · 57/54/46 | 111,268 · 3 · 19/23/15 | 16/16/16 | 42/41/41 | 16/15/16 | 10/11/11 |+| e2e-sessions, 1 d top 50 | 102,622 · 15 · 36/31/29 | 178,444 · 6 · 24/35/39 | 45/24/27 | 25/25/24 | 15/14/17 | 27/19/26 |+| e2e-sessions, FULL (647,079 rows) | 738,931 · 125 · 259/233/233 · 321–331 | 1,360,036 · 111 · 142/133/117 · 261–276 | 132/141/128 | 231/218/218 | 111/120/124 | 121/115/100 |++**The replay join changes the picture.** Without the join, this run reproduces M38-002:+chatpaper 1,111–1,148 → 149–156 ms, and 637 → 212 MiB. **With the join, the new statement costs+453–473 ms and reads 586 MiB.** The `LEFT JOIN` blocks the read-in-order / lazy-materialisation+top-N path, even with an empty right-hand side, so all 21 columns are read for the whole window+again. The plain (unsafe) drop is capped the same way (379–417 ms). The change is still worth+shipping:+- 2.4× on chatpaper, 3× on verdict, bayse and earlysalary, and 2.2× on ranges and search.+- `memory_usage` falls 2.5–3.5× (210–241 → 73–81 MiB on chatpaper).+- The result is identical, including on unmerged data.++With every session carrying a replay, both statements cost seconds and ≈2.2–3 GiB: old+2.1–2.9 s, new 1.6–1.9 s.++**Not shipped: the headroom.** A join-after-`LIMIT` form of the new statement returned the same+`(count, sum(h))`. It selects the top N in a subquery without the join, then `LEFT JOIN`s the+replay set onto those N rows. On chatpaper it ran in **150/147/152 ms and read 212 MiB**, measured+at 22:25 UTC. Moving the join is a further rewrite that M38-002 did not endorse. It is left as a+follow-up.++## Where the anti-join can differ from `FINAL` (FINAL_USAGE §6.2)++1. **One `(id, version)` with two `+1` rows and one `−1`.** `FINAL` collapses one pair and still+ shows the stale `+1` next to version V+1. The anti-join hides both copies of version V, until+ the merge. After the merge the `−1` is gone and the two forms agree again. **Reachable in+ production:** same-`(id, version)` `+1` duplicates exist on real projects on this box+ (chatpaper 69, verdict 3, tiptip-main-app-prod 5, …; e2e-sessions 1,043). If such a session is+ updated, the buffer writes one `−1`. On this box no `(id, version)` has both a `−1` and two+ `+1` rows (0), so this case could not be exercised. The visible effect is transient, only+ during the merge window: a stale duplicate row that `FINAL` would show is not shown.+2. **Cross-session cancel.** `FINAL` collapses on the sort key plus the version, and the sort key+ has no `id`. A `−1` whose own `+1` is missing could therefore cancel another session's `+1`+ with the same `created_at` and version. The anti-join matches on `id` and never does this.+ Reachable only if a `+1` is lost while its `−1` is written. That was not observed here (the one+ such group cancels cleanly).+3. **Unpaired `−1` rows.** The shipped statement had no `sign` filter, so `FINAL` would return a+ `−1` whose `+1` is absent. The new statement returns only `+1`. Reachable under the same+ condition as (2). On this box every `−1` has its `+1`: 0 of 21 unpaired.+4. **The `NOT IN` set lives in memory.** It holds the unmerged `−1` backlog of the window: 21 rows+ here. On live production its size is unknown (FINAL_USAGE open question 2).diff --git a/packages/core/src/modules/session/src/sql.test.ts b/packages/core/src/modules/session/src/sql.test.tsindex 9725859c2..4894476d6 100644--- a/packages/core/src/modules/session/src/sql.test.ts+++ b/packages/core/src/modules/session/src/sql.test.ts@@ -35,11 +35,12 @@ describe('sessionListQuery', () => {const text = collapse(query);expect(text).toContain(- "toBool(src.session_id != '') as hasReplay FROM sessions FINAL LEFT JOIN"+ "toBool(src.session_id != '') as hasReplay FROM sessions LEFT JOIN");+ expect(text).not.toContain('FINAL');expect(text).toContain('started_at > now() - INTERVAL {p2:Float64} DAY');- expect(text).toMatch(- /AND created_at >= toDateTime64\(\{p4:String\}, 3\) - INTERVAL \{p5:Float64\} DAY ORDER BY created_at DESC LIMIT \{p6:UInt64\}/+ expect(text).toContain(+ 'WHERE project_id = {p3:String} AND created_at >= toDateTime64({p4:String}, 3) - INTERVAL {p5:Float64} DAY AND sign = 1 AND (id, version) GLOBAL NOT IN ( SELECT id, version FROM sessions WHERE project_id = {p6:String} AND created_at >= toDateTime64({p7:String}, 3) - INTERVAL {p8:Float64} DAY AND sign = -1 ) ORDER BY created_at DESC LIMIT {p9:UInt64}');expect(text).not.toContain('created_at <');expect(text).not.toContain('profile_id =');@@ -49,12 +50,15 @@ describe('sessionListQuery', () => {p2: 0.5,p3: PROJECT_ID,p5: 0.5,- p6: 50,+ p6: PROJECT_ID,+ p8: 0.5,+ p9: 50,});+ expect(query_params.p7).toBe(query_params.p4);expect(query_params.p4).toMatch(/^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}$/);});- test('cursor page: lookback anchored at the cursor plus the `created_at <` boundary', () => {+ test('cursor page: lookback anchored at the cursor plus the `created_at <` boundary, in both the page and the −1 subquery', () => {const cursor = new Date('2026-07-10T12:34:56.789Z');const { query, query_params } = sessionListQuery({projectId: PROJECT_ID,@@ -65,13 +69,17 @@ describe('sessionListQuery', () => {const text = collapse(query);expect(text).toContain(- 'AND created_at >= toDateTime64({p4:String}, 3) - INTERVAL {p5:Float64} DAY AND created_at < {p6:String}'+ 'AND created_at >= toDateTime64({p4:String}, 3) - INTERVAL {p5:Float64} DAY AND created_at < {p6:String} AND sign = 1'+ );+ expect(text).toContain(+ 'WHERE project_id = {p7:String} AND created_at >= toDateTime64({p8:String}, 3) - INTERVAL {p9:Float64} DAY AND created_at < {p10:String} AND sign = -1 )');- expect(query_params.p4).toBe('2026-07-10 12:34:56');- expect(query_params.p6).toBe('2026-07-10 12:34:56');+ for (const key of ['p4', 'p6', 'p8', 'p10']) {+ expect(query_params[key]).toBe('2026-07-10 12:34:56');+ }});- test('explicit date range: no lookback window, calendar-day BETWEEN, search and filters spliced', () => {+ test('explicit date range: no lookback window, calendar-day BETWEEN in both, search and filters only on the page', () => {const { query, query_params } = sessionListQuery({projectId: PROJECT_ID,take: 10,@@ -89,15 +97,17 @@ describe('sessionListQuery', () => {expect(text).not.toContain('toDateTime64');expect(text).toContain(- "AND toDate(created_at) BETWEEN toDate({p4:String}) AND toDate({p5:String}) AND profile_id = {p6:String} AND (entry_path ILIKE {p7:String} OR exit_path ILIKE {p8:String} OR referrer ILIKE {p9:String} OR referrer_name ILIKE {p10:String}) AND country = 'SE' AND device = 'mobile' ORDER BY"+ "AND toDate(created_at) BETWEEN toDate({p4:String}) AND toDate({p5:String}) AND sign = 1 AND (id, version) GLOBAL NOT IN ( SELECT id, version FROM sessions WHERE project_id = {p6:String} AND toDate(created_at) BETWEEN toDate({p7:String}) AND toDate({p8:String}) AND sign = -1 ) AND profile_id = {p9:String} AND (entry_path ILIKE {p10:String} OR exit_path ILIKE {p11:String} OR referrer ILIKE {p12:String} OR referrer_name ILIKE {p13:String}) AND country = 'SE' AND device = 'mobile' ORDER BY");expect(query_params).toMatchObject({p4: '2026-07-01 00:00:00',p5: '2026-07-31 23:59:59',- p6: 'user-1',- p7: '%checkout%',+ p7: '2026-07-01 00:00:00',+ p8: '2026-07-31 23:59:59',+ p9: 'user-1',p10: '%checkout%',- p11: 10,+ p13: '%checkout%',+ p14: 10,});});diff --git a/packages/core/src/modules/session/src/sql.ts b/packages/core/src/modules/session/src/sql.tsindex 059c5d288..a3055d97b 100644--- a/packages/core/src/modules/session/src/sql.ts+++ b/packages/core/src/modules/session/src/sql.ts@@ -107,44 +107,20 @@ export function hasSessionListLookback(query: {);}-// `ORDER BY created_at DESC` stays as it is. `sessions` has the same-// `(project_id, toDate(created_at), created_at)` key shape as `events`, but `FINAL`-// blocks the read-in-order optimisation, so naming `toDate(created_at)` first buys-// nothing: measured 2026-09-15 on the prod copy, 30-day list, both spellings read the-// identical row count on all five anchor projects (chatpaper 4,209,818) and are within-// noise over 3 warm runs. See docs/ANALYTICS_PERFORMANCE.md §6.4 and M27-002.-export function sessionListQuery(query: SessionListQuery): SqlFragment {- const {- projectId,- take,- cursor,- lookbackDays,- startDate,- endDate,- profileId,- search,- filterClauses = {},- } = query;+// The rows of `sessions` this list page may show: project, lookback window,+// cursor boundary and calendar-day range. The −1 subquery repeats these so it+// only holds the window's unmerged backlog; a −1 is a copy of its +1, so it+// shares that row's `created_at` and falls in the same window.+function sessionListWindow(query: SessionListQuery): SqlFragment {+ const { projectId, cursor, lookbackDays, startDate, endDate } = query;const hasDateRange = Boolean(startDate && endDate);const lookbackAnchor = sql.string(formatClickhouseDate(cursor ?? new Date()));- const lookback = sql.float64(lookbackDays);- return sql`- SELECT ${sql.join(- SESSION_LIST_COLUMNS.map((column) => sql.id(column, SESSION_LIST_COLUMNS))- )}, toBool(src.session_id != '') as hasReplay- FROM ${sql.id(TABLE.sessions)} FINAL- LEFT JOIN (- SELECT DISTINCT session_id- FROM ${sql.id(TABLE.sessionReplayChunks)}- WHERE project_id = ${sql.string(projectId)}- AND started_at > now() - INTERVAL ${lookback} DAY- ) AS src ON src.session_id = id- WHERE project_id = ${sql.string(projectId)}+ return sql`project_id = ${sql.string(projectId)}${optional(hasSessionListLookback(query),() =>- sql`AND created_at >= toDateTime64(${lookbackAnchor}, 3) - INTERVAL ${lookback} DAY`+ sql`AND created_at >= toDateTime64(${lookbackAnchor}, 3) - INTERVAL ${sql.float64(lookbackDays)} DAY`)}${optional(cursor instanceof Date,@@ -153,7 +129,38 @@ export function sessionListQuery(query: SessionListQuery): SqlFragment {)}${optional(hasDateRange, () =>calendarDayRange(startDate as Date, endDate as Date)- )}+ )}`;+}++// No `FINAL`: it read every column of the whole window and blocked read-in-order+// (chatpaper 1.1 s → 0.15 s, M38-002 / M39-004, see sql.proof.md). A +1 whose+// `(id, version)` has a −1 is a pending collapse, which `FINAL` would have+// removed. Plain `sign = 1` alone returns those stale rows, and `LIMIT 1 BY id`+// drops the orphan +1 duplicates `FINAL` keeps. `GLOBAL` builds the −1 set once+// on the initiator instead of once per shard (docs/ENVIRONMENT.md).+export function sessionListQuery(query: SessionListQuery): SqlFragment {+ const { projectId, take, lookbackDays, profileId, search } = query;+ const { filterClauses = {} } = query;++ return sql`+ SELECT ${sql.join(+ SESSION_LIST_COLUMNS.map((column) => sql.id(column, SESSION_LIST_COLUMNS))+ )}, toBool(src.session_id != '') as hasReplay+ FROM ${sql.id(TABLE.sessions)}+ LEFT JOIN (+ SELECT DISTINCT session_id+ FROM ${sql.id(TABLE.sessionReplayChunks)}+ WHERE project_id = ${sql.string(projectId)}+ AND started_at > now() - INTERVAL ${sql.float64(lookbackDays)} DAY+ ) AS src ON src.session_id = id+ WHERE ${sessionListWindow(query)}+ AND sign = 1+ AND (id, version) GLOBAL NOT IN (+ SELECT id, version+ FROM ${sql.id(TABLE.sessions)}+ WHERE ${sessionListWindow(query)}+ AND sign = -1+ )${optional(profileId, () => sql`AND profile_id = ${sql.string(profileId as string)}`)}${optional(search, () => searchCondition(search as string))}${spliceCompiledFilters(filterClauses)}