Skip to content

OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.

ralph(M39-004): sessionListQuery: drop FINAL, add the collapse-aware anti-join

ralph4ad2cd9d966d
diff --git a/.m39-004-full.log b/.m39-004-full.log
new file mode 100644
index 000000000..b2af8a96a
--- /dev/null
+++ b/.m39-004-full.log
@@ -0,0 +1,11 @@
+OK: no forbidden patterns
+== bun install --frozen-lockfile (/home/deploy/openpanel) ==
+[0.05ms] ".env"
+bun install v1.4.0 (34cbb9a40)
+
+Checked 3036 installs across 3405 packages (no changes) [126.00ms]
+OK: bun install
+== build api: skipped (V1 fastify boot deleted; apps/api runs from source) ==
+== typecheck api (/home/deploy/openpanel/apps/api) ==
+$ tsc --noEmit
+error: script "typecheck" was terminated by signal SIGTERM (Polite quit request)
diff --git a/packages/core/src/modules/session/src/sql.proof.md b/packages/core/src/modules/session/src/sql.proof.md
index cf002947e..5c4445641 100644
--- a/packages/core/src/modules/session/src/sql.proof.md
+++ b/packages/core/src/modules/session/src/sql.proof.md
@@ -71,3 +71,130 @@ times in ms, and `memory_usage` in MiB.
M38-002 measured bayse at 846–994 ms → 32–36 ms and 363 → 38 MiB. This run reproduces those
figures: 808–858 ms → 28–42 ms and 364 → 39 MiB. `read_rows` does not change, because `FINAL`
still widens the ranges. Only the `id`/key columns are read for rows that the PREWHERE rejects.
+
+# session/src/sql.ts — `sessionListQuery` without `FINAL` (M39-004)
+
+`sessionListQuery` no longer reads `sessions FINAL`. It keeps `+1` rows only, and drops any row
+whose `(id, version)` has a `−1` in the same window:
+
+```sql
+FROM sessions
+LEFT JOIN (SELECT DISTINCT session_id FROM session_replay_chunks …) AS src ON src.session_id = id
+WHERE project_id = … <lookback / cursor / calendar-day window>
+ AND sign = 1
+ AND (id, version) GLOBAL NOT IN (
+ SELECT id, version FROM sessions
+ WHERE project_id = … <the same window> AND sign = -1
+ )
+ <profile_id / search / compiled filters, unchanged>
+ORDER BY created_at DESC
+LIMIT …
+```
+
+This is rank 2 of `docs/FINAL_USAGE.md` §7 (M38-002).
+
+**Why the subquery only repeats the window.** A `−1` is a full copy of its `+1`
+(`buffers/session-buffer.ts`), so the two rows share `created_at` and fall in the same window.
+The profile, search and filter predicates are applied to the `+1` in the outer query, so repeating
+them in the subquery would not change the result.
+
+**Cluster.** `sessions` and `session_replay_chunks` are `Distributed` on Cloud. `GLOBAL NOT IN`
+builds the `−1` set once, on the initiator, and ships it to the shards. It is correct under any
+`distributed_product_mode`. A plain `NOT IN` under the client's `'allow'` would run the subquery
+against the Distributed table from every shard, which is also correct but costs N× more. It is
+correct only because a `−1` and its `+1` share `created_at`, and so they share a shard: the shard
+key is `cityHash64(project_id, toStartOfHour(created_at))`. On this single node the two spellings
+cost the same (see the table below). Neither the replay `LEFT JOIN` nor its
+`distributed_product_mode` behaviour changed.
+
+## Setup
+
+- **When and where:** 2026-09-16, 22:23–22:26 UTC, on the local prod-copy ClickHouse
+ (`127.0.0.1:8123`, database `openpanel`, **single-node**). Load average was 0.1–1.2 at the start.
+- **Access:** SELECT only.
+- **How the statements were built:** both statements were rendered by the real builders, the
+ HEAD `sql.ts` and the new one, with bound `{pN:Type}` params. The requests used the client's
+ settings (`distributed_product_mode=allow`, `query_plan_convert_any_join_to_semi_or_anti_join=0`),
+ plus `wait_end_of_query=1`, `use_query_cache=0` and `use_query_condition_cache=0`.
+- **Where the numbers come from:** `read_rows`, `read_bytes`, `elapsed_ns` and `memory_usage` are
+ read off `X-ClickHouse-Summary`, from `FORMAT Null` runs. `memory_usage` is the summary figure,
+ not a sampled peak.
+- **Caching:** page cache was not controlled, so **every number is warm**.
+- **The replay join on this box:** `session_replay_chunks` holds 250 rows, all of them in
+ `e2e-sessions`. For every real anchor, the join's right-hand side is therefore **empty**. The
+ "all-replay" rows below replace it with *every* chatpaper session id, to bound the opposite
+ case.
+
+## Result-set equality
+
+Each check runs `SELECT count(), sum(h), cityHash64(groupArray(h)) FROM (SELECT
+sipHash64(formatRow('TSV', *)) h FROM (<statement>))`. The value compared is `(count, sum(h))`.
+"FULL" means `take = 1e8`, so the whole window is returned. The top-N cases were also re-checked
+with `ORDER BY created_at DESC, id`. The "control" column is the plain drop (`sign = 1`, no
+anti-join); it shows whether a case can detect a wrong answer at all.
+
+| project | cases (each: old = new?) | control = old? |
+|---|---|---|
+| chatpaper, verdict, bayse, earlysalary-production, website-8103 | cursor page (2026-08-30 12:00, 30 d) top 50 and FULL; first page anchored at now (30 d) top 50; date range 2026-08-01 → 08-15 top 50; cursor page + `search='a'` + filter `device='desktop'` top 50. **25/25 equal, plus 20/20 with the `id` tie-break.** (website-8103 has no sessions: 0 = 0. verdict search+filter: 0 = 0.) | equal everywhere, because the anchors hold no `−1` rows (§3 of FINAL_USAGE) |
+| chatpaper, all-replay join | cursor page top 50: **equal** | — |
+| **e2e-sessions (unmerged, 21 `−1` rows)** | cursor 2026-09-05 02:22:46 with 1 d lookback, top 50 and **top 500**; cursor 2026-09-15 23:59:59 with 30 d lookback, top 50 and **FULL** (647,079 rows); first page at now, **FULL**; date range September, **FULL**. **6/6 equal, plus 3/3 with the tie-break.** | **differs on top 500 and on all three FULL sets** (647,100 rows: the 21 stale `+1`) |
+
+Mismatches between old and new: **0**.
+
+## Cost: the whole statement, replay join included
+
+Each cell shows `read_rows`, then MiB read, three runs in ms, and `memory_usage` in MiB. All pages
+are cursor pages (2026-08-30 12:00, 30 d, top 50) unless the row says otherwise.
+
+| project | old: `FINAL` + join | **new: anti-join + join** | new, local `NOT IN` | old, no join | new, no join | plain drop + join (unsafe) |
+|---|---|---|---|---|---|---|
+| chatpaper | 3,816,602 · 637 · **1147/1138/1124** · 210–241 | 7,289,150 · 586 · **453/473/473** · 73–81 | 464/485/455 | 1111/1148/1134 | 7,297,342 · 212 · 154/156/149 · 4 | 3,644,575 · 555 · 417/379/386 |
+| chatpaper, range Aug 1–15 | 1,924,894 · 319 · **581/560/562** · 146–150 | 3,751,502 · 311 · **245/262/253** · 69–73 | 261/265/251 | 552/555/545 | 113 MiB · 88/87/94 | 212/204/209 |
+| chatpaper, search + filter | 637 MiB · **1261/1240/1238** · 209–223 | 588 MiB · **561/577/579** · 77–82 | 590/571/583 | 1237/1212/1207 | 372 MiB · 351/330/344 | 485/510/490 |
+| chatpaper, **all-replay join** | 9,844,399 · 896 · **2094/2550/2928** · 2208–2959 | 13,316,947 · 845 · **1933/1887/1644** · 2409–2806 | — | — | — | — |
+| verdict | 1,006,503 · 121 · **292/304/299** · 174–186 | 1,673,436 · 83 · **96/98/98** · 61 | 95/89/86 | 279/281/296 | 44 MiB · 58/44/48 | 71/65/69 |
+| bayse | 712,572 · 112 · **218/258/224** · 179–192 | 1,081,090 · 61 · **67/65/66** · 52–57 | 73/83/92 | 246/228/208 | 25 MiB · 34/30/31 | 52/47/50 |
+| earlysalary-production | 745,388 · 163 · **246/255/278** · 207–246 | 1,146,732 · 96 · **97/76/75** · 65 | 83/85/95 | 251/245/254 | 34 MiB · 51/34/43 | 76/68/73 |
+| website-8103 (0 sessions) | 55,634 · 11 · 57/54/46 | 111,268 · 3 · 19/23/15 | 16/16/16 | 42/41/41 | 16/15/16 | 10/11/11 |
+| e2e-sessions, 1 d top 50 | 102,622 · 15 · 36/31/29 | 178,444 · 6 · 24/35/39 | 45/24/27 | 25/25/24 | 15/14/17 | 27/19/26 |
+| e2e-sessions, FULL (647,079 rows) | 738,931 · 125 · 259/233/233 · 321–331 | 1,360,036 · 111 · 142/133/117 · 261–276 | 132/141/128 | 231/218/218 | 111/120/124 | 121/115/100 |
+
+**The replay join changes the picture.** Without the join, this run reproduces M38-002:
+chatpaper 1,111–1,148 → 149–156 ms, and 637 → 212 MiB. **With the join, the new statement costs
+453–473 ms and reads 586 MiB.** The `LEFT JOIN` blocks the read-in-order / lazy-materialisation
+top-N path, even with an empty right-hand side, so all 21 columns are read for the whole window
+again. The plain (unsafe) drop is capped the same way (379–417 ms). The change is still worth
+shipping:
+- 2.4× on chatpaper, 3× on verdict, bayse and earlysalary, and 2.2× on ranges and search.
+- `memory_usage` falls 2.5–3.5× (210–241 → 73–81 MiB on chatpaper).
+- The result is identical, including on unmerged data.
+
+With every session carrying a replay, both statements cost seconds and ≈2.2–3 GiB: old
+2.1–2.9 s, new 1.6–1.9 s.
+
+**Not shipped: the headroom.** A join-after-`LIMIT` form of the new statement returned the same
+`(count, sum(h))`. It selects the top N in a subquery without the join, then `LEFT JOIN`s the
+replay set onto those N rows. On chatpaper it ran in **150/147/152 ms and read 212 MiB**, measured
+at 22:25 UTC. Moving the join is a further rewrite that M38-002 did not endorse. It is left as a
+follow-up.
+
+## Where the anti-join can differ from `FINAL` (FINAL_USAGE §6.2)
+
+1. **One `(id, version)` with two `+1` rows and one `−1`.** `FINAL` collapses one pair and still
+ shows the stale `+1` next to version V+1. The anti-join hides both copies of version V, until
+ the merge. After the merge the `−1` is gone and the two forms agree again. **Reachable in
+ production:** same-`(id, version)` `+1` duplicates exist on real projects on this box
+ (chatpaper 69, verdict 3, tiptip-main-app-prod 5, …; e2e-sessions 1,043). If such a session is
+ updated, the buffer writes one `−1`. On this box no `(id, version)` has both a `−1` and two
+ `+1` rows (0), so this case could not be exercised. The visible effect is transient, only
+ during the merge window: a stale duplicate row that `FINAL` would show is not shown.
+2. **Cross-session cancel.** `FINAL` collapses on the sort key plus the version, and the sort key
+ has no `id`. A `−1` whose own `+1` is missing could therefore cancel another session's `+1`
+ with the same `created_at` and version. The anti-join matches on `id` and never does this.
+ Reachable only if a `+1` is lost while its `−1` is written. That was not observed here (the one
+ such group cancels cleanly).
+3. **Unpaired `−1` rows.** The shipped statement had no `sign` filter, so `FINAL` would return a
+ `−1` whose `+1` is absent. The new statement returns only `+1`. Reachable under the same
+ condition as (2). On this box every `−1` has its `+1`: 0 of 21 unpaired.
+4. **The `NOT IN` set lives in memory.** It holds the unmerged `−1` backlog of the window: 21 rows
+ here. On live production its size is unknown (FINAL_USAGE open question 2).
diff --git a/packages/core/src/modules/session/src/sql.test.ts b/packages/core/src/modules/session/src/sql.test.ts
index 9725859c2..4894476d6 100644
--- a/packages/core/src/modules/session/src/sql.test.ts
+++ b/packages/core/src/modules/session/src/sql.test.ts
@@ -35,11 +35,12 @@ describe('sessionListQuery', () => {
const text = collapse(query);
expect(text).toContain(
- "toBool(src.session_id != '') as hasReplay FROM sessions FINAL LEFT JOIN"
+ "toBool(src.session_id != '') as hasReplay FROM sessions LEFT JOIN"
);
+ expect(text).not.toContain('FINAL');
expect(text).toContain('started_at > now() - INTERVAL {p2:Float64} DAY');
- expect(text).toMatch(
- /AND created_at >= toDateTime64\(\{p4:String\}, 3\) - INTERVAL \{p5:Float64\} DAY ORDER BY created_at DESC LIMIT \{p6:UInt64\}/
+ expect(text).toContain(
+ 'WHERE project_id = {p3:String} AND created_at >= toDateTime64({p4:String}, 3) - INTERVAL {p5:Float64} DAY AND sign = 1 AND (id, version) GLOBAL NOT IN ( SELECT id, version FROM sessions WHERE project_id = {p6:String} AND created_at >= toDateTime64({p7:String}, 3) - INTERVAL {p8:Float64} DAY AND sign = -1 ) ORDER BY created_at DESC LIMIT {p9:UInt64}'
);
expect(text).not.toContain('created_at <');
expect(text).not.toContain('profile_id =');
@@ -49,12 +50,15 @@ describe('sessionListQuery', () => {
p2: 0.5,
p3: PROJECT_ID,
p5: 0.5,
- p6: 50,
+ p6: PROJECT_ID,
+ p8: 0.5,
+ p9: 50,
});
+ expect(query_params.p7).toBe(query_params.p4);
expect(query_params.p4).toMatch(/^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}$/);
});
- test('cursor page: lookback anchored at the cursor plus the `created_at <` boundary', () => {
+ test('cursor page: lookback anchored at the cursor plus the `created_at <` boundary, in both the page and the −1 subquery', () => {
const cursor = new Date('2026-07-10T12:34:56.789Z');
const { query, query_params } = sessionListQuery({
projectId: PROJECT_ID,
@@ -65,13 +69,17 @@ describe('sessionListQuery', () => {
const text = collapse(query);
expect(text).toContain(
- 'AND created_at >= toDateTime64({p4:String}, 3) - INTERVAL {p5:Float64} DAY AND created_at < {p6:String}'
+ 'AND created_at >= toDateTime64({p4:String}, 3) - INTERVAL {p5:Float64} DAY AND created_at < {p6:String} AND sign = 1'
+ );
+ expect(text).toContain(
+ 'WHERE project_id = {p7:String} AND created_at >= toDateTime64({p8:String}, 3) - INTERVAL {p9:Float64} DAY AND created_at < {p10:String} AND sign = -1 )'
);
- expect(query_params.p4).toBe('2026-07-10 12:34:56');
- expect(query_params.p6).toBe('2026-07-10 12:34:56');
+ for (const key of ['p4', 'p6', 'p8', 'p10']) {
+ expect(query_params[key]).toBe('2026-07-10 12:34:56');
+ }
});
- test('explicit date range: no lookback window, calendar-day BETWEEN, search and filters spliced', () => {
+ test('explicit date range: no lookback window, calendar-day BETWEEN in both, search and filters only on the page', () => {
const { query, query_params } = sessionListQuery({
projectId: PROJECT_ID,
take: 10,
@@ -89,15 +97,17 @@ describe('sessionListQuery', () => {
expect(text).not.toContain('toDateTime64');
expect(text).toContain(
- "AND toDate(created_at) BETWEEN toDate({p4:String}) AND toDate({p5:String}) AND profile_id = {p6:String} AND (entry_path ILIKE {p7:String} OR exit_path ILIKE {p8:String} OR referrer ILIKE {p9:String} OR referrer_name ILIKE {p10:String}) AND country = 'SE' AND device = 'mobile' ORDER BY"
+ "AND toDate(created_at) BETWEEN toDate({p4:String}) AND toDate({p5:String}) AND sign = 1 AND (id, version) GLOBAL NOT IN ( SELECT id, version FROM sessions WHERE project_id = {p6:String} AND toDate(created_at) BETWEEN toDate({p7:String}) AND toDate({p8:String}) AND sign = -1 ) AND profile_id = {p9:String} AND (entry_path ILIKE {p10:String} OR exit_path ILIKE {p11:String} OR referrer ILIKE {p12:String} OR referrer_name ILIKE {p13:String}) AND country = 'SE' AND device = 'mobile' ORDER BY"
);
expect(query_params).toMatchObject({
p4: '2026-07-01 00:00:00',
p5: '2026-07-31 23:59:59',
- p6: 'user-1',
- p7: '%checkout%',
+ p7: '2026-07-01 00:00:00',
+ p8: '2026-07-31 23:59:59',
+ p9: 'user-1',
p10: '%checkout%',
- p11: 10,
+ p13: '%checkout%',
+ p14: 10,
});
});
diff --git a/packages/core/src/modules/session/src/sql.ts b/packages/core/src/modules/session/src/sql.ts
index 059c5d288..a3055d97b 100644
--- a/packages/core/src/modules/session/src/sql.ts
+++ b/packages/core/src/modules/session/src/sql.ts
@@ -107,44 +107,20 @@ export function hasSessionListLookback(query: {
);
}
-// `ORDER BY created_at DESC` stays as it is. `sessions` has the same
-// `(project_id, toDate(created_at), created_at)` key shape as `events`, but `FINAL`
-// blocks the read-in-order optimisation, so naming `toDate(created_at)` first buys
-// nothing: measured 2026-09-15 on the prod copy, 30-day list, both spellings read the
-// identical row count on all five anchor projects (chatpaper 4,209,818) and are within
-// noise over 3 warm runs. See docs/ANALYTICS_PERFORMANCE.md §6.4 and M27-002.
-export function sessionListQuery(query: SessionListQuery): SqlFragment {
- const {
- projectId,
- take,
- cursor,
- lookbackDays,
- startDate,
- endDate,
- profileId,
- search,
- filterClauses = {},
- } = query;
+// The rows of `sessions` this list page may show: project, lookback window,
+// cursor boundary and calendar-day range. The −1 subquery repeats these so it
+// only holds the window's unmerged backlog; a −1 is a copy of its +1, so it
+// shares that row's `created_at` and falls in the same window.
+function sessionListWindow(query: SessionListQuery): SqlFragment {
+ const { projectId, cursor, lookbackDays, startDate, endDate } = query;
const hasDateRange = Boolean(startDate && endDate);
const lookbackAnchor = sql.string(formatClickhouseDate(cursor ?? new Date()));
- const lookback = sql.float64(lookbackDays);
- return sql`
- SELECT ${sql.join(
- SESSION_LIST_COLUMNS.map((column) => sql.id(column, SESSION_LIST_COLUMNS))
- )}, toBool(src.session_id != '') as hasReplay
- FROM ${sql.id(TABLE.sessions)} FINAL
- LEFT JOIN (
- SELECT DISTINCT session_id
- FROM ${sql.id(TABLE.sessionReplayChunks)}
- WHERE project_id = ${sql.string(projectId)}
- AND started_at > now() - INTERVAL ${lookback} DAY
- ) AS src ON src.session_id = id
- WHERE project_id = ${sql.string(projectId)}
+ return sql`project_id = ${sql.string(projectId)}
${optional(
hasSessionListLookback(query),
() =>
- sql`AND created_at >= toDateTime64(${lookbackAnchor}, 3) - INTERVAL ${lookback} DAY`
+ sql`AND created_at >= toDateTime64(${lookbackAnchor}, 3) - INTERVAL ${sql.float64(lookbackDays)} DAY`
)}
${optional(
cursor instanceof Date,
@@ -153,7 +129,38 @@ export function sessionListQuery(query: SessionListQuery): SqlFragment {
)}
${optional(hasDateRange, () =>
calendarDayRange(startDate as Date, endDate as Date)
- )}
+ )}`;
+}
+
+// No `FINAL`: it read every column of the whole window and blocked read-in-order
+// (chatpaper 1.1 s → 0.15 s, M38-002 / M39-004, see sql.proof.md). A +1 whose
+// `(id, version)` has a −1 is a pending collapse, which `FINAL` would have
+// removed. Plain `sign = 1` alone returns those stale rows, and `LIMIT 1 BY id`
+// drops the orphan +1 duplicates `FINAL` keeps. `GLOBAL` builds the −1 set once
+// on the initiator instead of once per shard (docs/ENVIRONMENT.md).
+export function sessionListQuery(query: SessionListQuery): SqlFragment {
+ const { projectId, take, lookbackDays, profileId, search } = query;
+ const { filterClauses = {} } = query;
+
+ return sql`
+ SELECT ${sql.join(
+ SESSION_LIST_COLUMNS.map((column) => sql.id(column, SESSION_LIST_COLUMNS))
+ )}, toBool(src.session_id != '') as hasReplay
+ FROM ${sql.id(TABLE.sessions)}
+ LEFT JOIN (
+ SELECT DISTINCT session_id
+ FROM ${sql.id(TABLE.sessionReplayChunks)}
+ WHERE project_id = ${sql.string(projectId)}
+ AND started_at > now() - INTERVAL ${sql.float64(lookbackDays)} DAY
+ ) AS src ON src.session_id = id
+ WHERE ${sessionListWindow(query)}
+ AND sign = 1
+ AND (id, version) GLOBAL NOT IN (
+ SELECT id, version
+ FROM ${sql.id(TABLE.sessions)}
+ WHERE ${sessionListWindow(query)}
+ AND sign = -1
+ )
${optional(profileId, () => sql`AND profile_id = ${sql.string(profileId as string)}`)}
${optional(search, () => searchCondition(search as string))}
${spliceCompiledFilters(filterClauses)}