πΈ Rybbit - open-source and privacy-friendly alternative to Google Analytics that is 10x more intuitive.
perf(clickhouse): cut mutation-driven parts-lock contention on the hot ingest path (#1131)
Goldflagc65b39312b5e
diff --git a/clickhouse/REPLAY_METADATA_V2.md b/clickhouse/REPLAY_METADATA_V2.mdnew file mode 100644index 000000000..da2a35cd0--- /dev/null+++ b/clickhouse/REPLAY_METADATA_V2.md@@ -0,0 +1,199 @@+# Cutover: `session_replay_metadata` β `session_replay_metadata_v2`++The old table is a `ReplacingMergeTree` holding one cumulative row per session.+Keeping that row current meant every replay batch re-read the whole session+(`SELECT MIN/MAX/COUNT/SUM β¦ FROM session_replay_events WHERE session_id = β¦`)+and rewrote it. In one six-day window that read scanned **818 billion rows β+76% of everything the cluster read** β and the rewrites left **2.5 million+single-row parts**.++`session_replay_metadata_v2` is an `AggregatingMergeTree`. Each batch inserts+only what it observed and the engine combines rows at merge time, so ingest+never reads back. Column names and their post-merge meanings are unchanged;+readers already used `FINAL`, which is all that is needed.++## Known limitation: snapshot columns merge independently++`page_url`, the geo fields, `browser`, `language` and the rest use+`SimpleAggregateFunction(anyLast, β¦)`, so each one merges on its own. The old+`ReplacingMergeTree(created_at)` picked one whole winning row, so a session+whose batches disagreed always showed a single coherent snapshot; the new table+can assemble one from several batches.++Measured over 30 days of production, of the 423 sessions carrying more than one+metadata version:++| field | sessions that differed |+| --- | --- |+| `page_url` | 62 |+| `screen_width` / `screen_height` | 28 / 32 |+| `region`, `city`, `lat` | 18 |+| `language` | 13 |+| `country` | 11 |+| `browser`, `operating_system`, `device_type`, `channel`, `hostname`, `referrer`, `user_id` | 0 |++`screen_width`/`screen_height` are unaffected β they use `max`, which is what+the old code computed anyway.++The geo group is the real exposure: `country`, `region`, `city` and `lat`/`lon`+are only meaningful together, and roughly 4% of multi-version sessions could+show a city and a country taken from different batches.++**This was weighed and accepted in August 2026.** The exposure is narrow, the+affected fields are all still true of the session, and the alternatives cost+more than the defect. The rest of this section records what the options were, so+the decision can be revisited if mixed geo ever turns out to mislead someone.++Making the snapshot coherent means versioning it as a unit β either one+`SimpleAggregateFunction(max, Tuple(version, β¦))`, or `argMaxState` columns read+with `argMaxMerge` and `GROUP BY`. Both change every read site, and the tuple+form carries a sharp edge: `SELECT snapshot.page_url FROM β¦ FINAL` silently+returns the *unmerged* value, because ClickHouse resolves the named element as a+subcolumn read and skips the merge. Positional access (`snapshot.2`) merges+correctly, as does selecting the whole tuple in a subquery first.++## Columns that are gone++Two columns are gone:++- `duration_ms` β derived at read time as+ `dateDiff('millisecond', start_time, end_time)`, because a single batch only+ knows its own slice of the session. `start_time`/`end_time` are+ `DateTime64(3)` for exactly this reason: at second resolution a 900 ms replay+ would derive a duration of 0.+- `created_at` β there is no version column to order by any more.++The application creates the new table on boot. The steps below move the+existing data across and are **not** run by the app.++## 1. Deploy++Deploying repoints both writes and reads at `session_replay_metadata_v2`. From+this moment the old table is frozen β nothing writes to it.++Sessions already in flight keep working: the backfill below copies their+totals so far, and post-deploy batches add their own increments on top. The+aggregate columns sum, so the two halves combine to the correct total.++## 2. Backfill++Run once, after the deploy:++```sql+INSERT INTO session_replay_metadata_v2+SELECT+ site_id, session_id, user_id, identified_user_id,+ start_time,+ -- The old bounds are second-resolution but the old `duration_ms` was exact,+ -- so rebuild `end_time` from it rather than copying the rounded column β+ -- otherwise every historical replay's duration is re-derived to the second.+ if(duration_ms IS NULL,+ CAST(toDateTime64(end_time, 3) AS Nullable(DateTime64(3))),+ CAST(toDateTime64(start_time, 3) + toIntervalMillisecond(assumeNotNull(duration_ms)) AS Nullable(DateTime64(3)))) AS end_time,+ event_count, compressed_size_bytes,+ page_url, country, region, city, lat, lon,+ browser, browser_version, operating_system, operating_system_version,+ language, screen_width, screen_height, device_type,+ channel, hostname, referrer, has_replay_data+FROM session_replay_metadata+FINAL+WHERE start_time >= now() - INTERVAL 30 DAY;+```++`start_time` still lands on a whole second for backfilled rows, because that is+all the old table recorded. Rows written after the deploy carry true+millisecond bounds on both ends.++`FINAL` matters: without it the old table's superseded versions are copied too,+and because the new engine **sums** `event_count` rather than replacing it,+every session would be inflated by its own history.++For the same reason this statement is **not idempotent**. Running it twice+doubles `event_count` and `compressed_size_bytes` for every session. If it is+interrupted, clear the table before retrying:++```sql+TRUNCATE TABLE session_replay_metadata_v2;+```++That is safe only while the deploy is fresh β it also discards post-deploy+increments, which the backfill cannot recover.++The 30-day bound matches the table's TTL; older rows would be deleted on the+next TTL pass anyway.++## 3. Verify++Totals should agree, modulo sessions that received events between the two+reads:++```sql+SELECT count() AS sessions, sum(event_count) AS events+FROM session_replay_metadata FINAL+WHERE start_time >= now() - INTERVAL 30 DAY;++SELECT count() AS sessions, sum(event_count) AS events+FROM session_replay_metadata_v2 FINAL+WHERE start_time >= now() - INTERVAL 30 DAY;+```++Also confirm ingest is no longer reading the session back:++```sql+SELECT count()+FROM system.query_log+WHERE event_time > now() - INTERVAL 10 MINUTE+ AND query LIKE '%FROM session_replay_events%'+ AND query LIKE '%compressed_size_bytes%';+```++Expect zero.++## 4. Drop the old table++Once the dashboard has been checked against real sessions and you are happy to+lose the ability to roll back:++```sql+DROP TABLE session_replay_metadata;+```++Nothing in the application references it after the deploy; it is kept only as+the rollback path.++---++# Related: clearing impossible partitions++Replay event timestamps come from the browser, so devices with broken clocks+wrote rows dated 2032, 2035, 2064, 2076 and 2090. Those partitions grow the+partition list permanently and, because the TTL is `toDateTime(start_time) + 30 DAY`, they+never expire.++Ingest now corrects this (`server/src/services/replay/replayClockSkew.ts`): a+batch whose median timestamp is more than a day ahead, or more than 30 days+behind, is shifted onto server time as a whole, preserving the gaps between+events so playback still reconstructs. **No new impossible partitions are+created after the deploy.**++The existing ones still need removing. List them first:++```sql+SELECT table, partition, sum(rows) AS rows, count() AS parts+FROM system.parts+WHERE active AND database = currentDatabase()+ AND table IN ('session_replay_events', 'session_replay_metadata_v2')+ AND partition > formatDateTime(now() + INTERVAL 2 MONTH, '%Y%m')+GROUP BY table, partition+ORDER BY table, partition;+```++Then drop each one by name β a few hundred rows of unplayable recordings from+devices whose clocks were decades out:++```sql+ALTER TABLE session_replay_events DROP PARTITION '209007';+```++Check the listing rather than copying partition ids from here; the set will+have changed.diff --git a/client/src/api/admin/hooks/useSites.ts b/client/src/api/admin/hooks/useSites.tsindex 02cc8c67b..257373843 100644--- a/client/src/api/admin/hooks/useSites.ts+++ b/client/src/api/admin/hooks/useSites.ts@@ -31,8 +31,17 @@ export function useSiteHasData(siteId: string) {}return fetchSiteHasData(siteId).then(data => data.hasData);},- refetchInterval: 5000,+ // Poll only until the site reports data. Once it flips true it can never+ // flip back, so continuing to poll re-asks a settled question every 5 s for+ // as long as the tab stays open β which is where the bulk of this+ // endpoint's call volume came from.+ //+ // A site that genuinely has no data yet never stops polling, so that case+ // sets the floor: 30 s is still well inside the "paste the script, watch it+ // light up" loop this powers, at a sixth of the requests.+ refetchInterval: query => (query.state.data === true ? false : 30_000),staleTime: Infinity,+ enabled: !!siteId,});}diff --git a/client/src/lib/defaultTimeRange.test.ts b/client/src/lib/defaultTimeRange.test.tsnew file mode 100644index 000000000..03ea16681--- /dev/null+++ b/client/src/lib/defaultTimeRange.test.ts@@ -0,0 +1,19 @@+import { describe, expect, it } from "vitest";+import { capDashboardDefaultRange } from "./defaultTimeRange";++describe("capDashboardDefaultRange", () => {+ it("caps an all-time default on the users page", () => {+ expect(capDashboardDefaultRange("all-time", "/1/users")).toBe("last-30-days");+ });++ it("leaves the all-time default alone everywhere else", () => {+ expect(capDashboardDefaultRange("all-time", "/1")).toBe("all-time");+ expect(capDashboardDefaultRange("all-time", "/1/sessions")).toBe("all-time");+ expect(capDashboardDefaultRange("all-time", "/1/user/abc")).toBe("all-time");+ });++ it("never widens or rewrites a bounded default", () => {+ expect(capDashboardDefaultRange("today", "/1/users")).toBe("today");+ expect(capDashboardDefaultRange("last-7-days", "/1/users")).toBe("last-7-days");+ });+});diff --git a/client/src/lib/defaultTimeRange.ts b/client/src/lib/defaultTimeRange.tsindex a4ed4fac3..b7368b1a9 100644--- a/client/src/lib/defaultTimeRange.ts+++ b/client/src/lib/defaultTimeRange.ts@@ -182,6 +182,30 @@ export function getDashboardTimeForRange(value: string, timezone: string): Time}}+// Some pages cannot afford an all-time default. The users list aggregates every+// event a site has ever recorded β the count query behind it scanned 5.6 B rows+// over six days β which is an expensive way to answer a question the visitor has+// not asked yet.+//+// This caps the *default* only, and it belongs here rather than on the page:+// every caller below already skips the stored default when the URL carries a+// time, and an explicit selection is written to the URL. So picking All time+// from the selector still works and still survives leaving and returning; what+// no longer happens is landing on it unasked.+const DEFAULT_RANGE_CAPS: { appliesTo: (pathname: string) => boolean; cap: DashboardDefaultTimeRange }[] = [+ { appliesTo: pathname => pathname.endsWith("/users"), cap: "last-30-days" },+];++export function capDashboardDefaultRange(+ range: DashboardDefaultTimeRange,+ pathname: string+): DashboardDefaultTimeRange {+ if (range !== "all-time") return range;+ return DEFAULT_RANGE_CAPS.find(rule => rule.appliesTo(pathname))?.cap ?? range;+}+export function getStoredDashboardDefaultTime(timezone: string): Time {- return getDashboardTimeForRange(getStoredDashboardDefaultTimeRange(), timezone);+ const stored = getStoredDashboardDefaultTimeRange();+ const range = typeof window === "undefined" ? stored : capDashboardDefaultRange(stored, window.location.pathname);+ return getDashboardTimeForRange(range, timezone);}diff --git a/server/src/api/admin/getClickhouseStats.ts b/server/src/api/admin/getClickhouseStats.tsindex d7b0ccd28..fe2bdc37c 100644--- a/server/src/api/admin/getClickhouseStats.ts+++ b/server/src/api/admin/getClickhouseStats.ts@@ -50,7 +50,7 @@ export async function getClickhouseStats(request: FastifyRequest<{ Querystring:count() as parts_countFROM system.partsWHERE database = currentDatabase() AND active = 1- AND table IN ('events', 'session_replay_events', 'session_replay_metadata')+ AND table IN ('events', 'session_replay_events', 'session_replay_metadata_v2')GROUP BY tableORDER BY total_rows DESC`,@@ -79,7 +79,7 @@ export async function getClickhouseStats(request: FastifyRequest<{ Querystring:GROUP BY dateUNION ALLSELECT toDate(start_time) as date, 'session_replay_metadata' as table, count() as rows_inserted- FROM session_replay_metadata+ FROM session_replay_metadata_v2${dateFilterStartTime}GROUP BY date)diff --git a/server/src/api/analytics/sessions/getSessions.ts b/server/src/api/analytics/sessions/getSessions.tsindex 50d712ca3..2771a8d65 100644--- a/server/src/api/analytics/sessions/getSessions.ts+++ b/server/src/api/analytics/sessions/getSessions.ts@@ -162,7 +162,7 @@ export const buildSessionsQuery = (query: GetSessionsRequest["Querystring"], sit),ReplaySessions AS (SELECT DISTINCT session_id- FROM session_replay_metadata+ FROM session_replay_metadata_v2FINALWHERE site_id = {siteId:Int32}AND event_count >= 2diff --git a/server/src/api/analytics/users/deleteUser.ts b/server/src/api/analytics/users/deleteUser.tsindex abca880b6..ca435b03c 100644--- a/server/src/api/analytics/users/deleteUser.ts+++ b/server/src/api/analytics/users/deleteUser.ts@@ -63,7 +63,7 @@ export async function deleteUser(req: FastifyRequest<DeleteUserRequest>, res: Fa}await Promise.all(- ["events", "session_replay_events", "session_replay_metadata"].map(table =>+ ["events", "session_replay_events", "session_replay_metadata_v2"].map(table =>clickhouse.command({query: `DELETE FROM ${table} WHERE ${userCondition}`,query_params: queryParams,diff --git a/server/src/api/analytics/users/identifyUser.ts b/server/src/api/analytics/users/identifyUser.tsindex 69496d144..a33cf6deb 100644--- a/server/src/api/analytics/users/identifyUser.ts+++ b/server/src/api/analytics/users/identifyUser.ts@@ -76,9 +76,12 @@ export async function identifyUser(req: FastifyRequest<IdentifyUserRequest>, resset: { userId: user_id },});- // Fire-and-forget: the ClickHouse mutation can take a while on large sites,- // so don't hold the response on it. No backfill window β the operator is- // explicitly asserting this device's history belongs to this user.+ // Queued, not immediate: the backfill is batched with every other identity+ // assigned in the flush interval, so it lands within a few minutes rather+ // than on this request. It was already fire-and-forget, so nothing in the+ // response depended on it. No backfill window β the operator is explicitly+ // asserting this device's history belongs to this user, and an unbounded+ // mutation is exactly the kind worth batching.backfillIdentifiedUserId(siteId, anonymous_id, user_id, null);return res.send({ success: true });diff --git a/server/src/api/sites/getSiteHasData.ts b/server/src/api/sites/getSiteHasData.tsindex 075d42d6c..61c46e546 100644--- a/server/src/api/sites/getSiteHasData.ts+++ b/server/src/api/sites/getSiteHasData.ts@@ -5,20 +5,26 @@ export async function getSiteHasData(request: FastifyRequest<{ Params: { siteId:const { siteId } = request.params;try {- // Check if site has data using original method- const pageviewsData: { count: number }[] = await clickhouse+ // `count(*)` reads every row for the site just to answer "any?" β 134.8 B+ // rows across the poll's 289 k calls in one six-day window. LIMIT 1 alone+ // is close to a wash, because parallel readers each fetch a granule before+ // the limit stops them; pinning to a single thread is what turns this into+ // a two-granule read.+ const rows: { has_data: number }[] = await clickhouse.query({- query: `SELECT count(*) as count FROM events WHERE site_id = {siteId:Int32}`,+ query: `SELECT 1 AS has_data FROM events WHERE site_id = {siteId:Int32} LIMIT 1`,format: "JSONEachRow",query_params: {siteId: Number(siteId),},+ clickhouse_settings: {+ max_threads: 1,+ },}).then(res => res.json());- const hasData = pageviewsData[0].count > 0;return {- hasData,+ hasData: rows.length > 0,};} catch (error) {request.log.error({ err: error }, "Error checking if site has data");diff --git a/server/src/db/clickhouse/schema/core.ts b/server/src/db/clickhouse/schema/core.tsindex 3d4cac0be..344ef1bce 100644--- a/server/src/db/clickhouse/schema/core.ts+++ b/server/src/db/clickhouse/schema/core.ts@@ -319,4 +319,79 @@ export async function initializeCoreTables() {ADD COLUMN IF NOT EXISTS identified_user_id String DEFAULT ''`);++ // Successor to session_replay_metadata. The old table is a+ // ReplacingMergeTree holding one cumulative row per session, so every replay+ // batch had to re-derive that row: SELECT MIN/MAX/COUNT/SUM over the whole+ // session, then rewrite it. That read scanned 818 B rows in six days β 76% of+ // everything the cluster read β and the rewrites left 2.5 M single-row parts.+ //+ // Aggregating the columns instead lets each batch insert only what it+ // observed, and the engine does the combining. Nothing reads back during+ // ingest. Every column keeps its name and its post-merge meaning, so readers+ // only need FINAL, which they already used.+ //+ // duration_ms and created_at are deliberately absent: duration is derived+ // from the merged bounds at read time, and there is no version column to+ // order by any more.+ await execClickhouseInitStep(+ "create session replay metadata v2 table",+ `+ CREATE TABLE IF NOT EXISTS session_replay_metadata_v2 (+ site_id UInt16,+ session_id String,+ user_id SimpleAggregateFunction(anyLast, String),+ -- Rows arrive with '' until the visitor identifies, and max() over a+ -- String makes any real id win over the empty one.+ identified_user_id SimpleAggregateFunction(max, String),+ -- Millisecond resolution, matching session_replay_events.timestamp:+ -- duration is now derived from these bounds instead of being stored,+ -- and second-resolution columns would floor a 900ms replay to 0.+ start_time SimpleAggregateFunction(min, DateTime64(3)),+ end_time SimpleAggregateFunction(max, Nullable(DateTime64(3))),+ event_count SimpleAggregateFunction(sum, UInt64),+ compressed_size_bytes SimpleAggregateFunction(sum, UInt64),+ -- KNOWN LIMITATION. These merge independently, so a session whose+ -- batches disagreed can assemble a row from more than one of them β+ -- the old ReplacingMergeTree always returned one whole batch's+ -- snapshot. Measured over 30 days of production, of the 423 sessions+ -- that had more than one metadata version: page_url differed in 62,+ -- region/city/lat in 18, country in 11, language in 13; browser,+ -- operating_system, device_type, channel, hostname, referrer and+ -- user_id never differed at all.+ --+ -- The geo group is the one that matters, because those fields are only+ -- meaningful together: ~4% of multi-version sessions could show a city+ -- and a country drawn from different batches. Making them coherent+ -- needs one versioned snapshot (a max() over a leading-version tuple,+ -- or argMaxState + GROUP BY), which changes every read site.+ --+ -- Weighed and accepted, 2026-08: the exposure is narrow and the fix+ -- costs more than the defect. Don't re-open it without new evidence+ -- that mixed geo is actually misleading someone.+ page_url SimpleAggregateFunction(anyLast, String),+ country SimpleAggregateFunction(anyLast, LowCardinality(FixedString(2))),+ region SimpleAggregateFunction(anyLast, LowCardinality(String)),+ city SimpleAggregateFunction(anyLast, String),+ lat SimpleAggregateFunction(anyLast, Float64),+ lon SimpleAggregateFunction(anyLast, Float64),+ browser SimpleAggregateFunction(anyLast, LowCardinality(String)),+ browser_version SimpleAggregateFunction(anyLast, LowCardinality(String)),+ operating_system SimpleAggregateFunction(anyLast, LowCardinality(String)),+ operating_system_version SimpleAggregateFunction(anyLast, LowCardinality(String)),+ language SimpleAggregateFunction(anyLast, LowCardinality(String)),+ screen_width SimpleAggregateFunction(max, UInt16),+ screen_height SimpleAggregateFunction(max, UInt16),+ device_type SimpleAggregateFunction(anyLast, LowCardinality(String)),+ channel SimpleAggregateFunction(anyLast, String),+ hostname SimpleAggregateFunction(anyLast, String),+ referrer SimpleAggregateFunction(anyLast, String),+ has_replay_data SimpleAggregateFunction(max, UInt8)+ )+ ENGINE = AggregatingMergeTree()+ PARTITION BY toYYYYMM(start_time)+ ORDER BY (site_id, session_id)+ TTL toDateTime(start_time) + INTERVAL 30 DAY+ `+ );}diff --git a/server/src/index.ts b/server/src/index.tsindex 00402dd07..0753bc1a4 100644--- a/server/src/index.ts+++ b/server/src/index.ts@@ -179,6 +179,7 @@ import { createCorsOptionsDelegate, createRejectUntrustedOriginHook } from "./liimport { IS_CLOUD } from "./lib/const.js";import { logger } from "./lib/logger/logger.js";import { registerRequestLogging } from "./lib/logger/requestLogging.js";+import { identityBackfillQueue } from "./services/tracker/identityBackfillQueue.js";import { reengagementService } from "./services/reengagement/reengagementService.js";import { telemetryService } from "./services/telemetryService.js";import { handleIdentify } from "./services/tracker/identifyService.js";@@ -622,6 +623,10 @@ const shutdown = async (signal: string) => {await server.close();server.log.info("Server closed");+ // Identity backfills are buffered for several minutes to keep mutation+ // submissions rare; without this, a deploy drops whatever is still pending.+ await identityBackfillQueue.drainCompletely();+// Clear the timeout since we're doneclearTimeout(forceExitTimeout);diff --git a/server/src/services/replay/replayClockSkew.test.ts b/server/src/services/replay/replayClockSkew.test.tsnew file mode 100644index 000000000..f12f8b9a1--- /dev/null+++ b/server/src/services/replay/replayClockSkew.test.ts@@ -0,0 +1,89 @@+import { describe, expect, it } from "vitest";+import { correctReplayClockSkew, MAX_FUTURE_SKEW_MS, MAX_PAST_SKEW_MS } from "./replayClockSkew.js";++const NOW = Date.UTC(2026, 7, 19, 12, 0, 0);+const batch = (...offsets: number[]) => offsets.map(timestamp => ({ timestamp, type: "2" }));++describe("correctReplayClockSkew", () => {+ it("leaves a healthy batch untouched", () => {+ const events = batch(NOW - 3000, NOW - 2000, NOW - 1000);+ const result = correctReplayClockSkew(events, NOW);+ expect(result.skewMs).toBe(0);+ expect(result.events).toBe(events);+ });++ it("shifts a batch from a device whose clock is decades ahead", () => {+ const year2090 = Date.UTC(2090, 6, 2, 21, 31, 45);+ const events = batch(year2090, year2090 + 1000, year2090 + 5000);+ const result = correctReplayClockSkew(events, NOW);++ expect(result.skewMs).toBeGreaterThan(0);+ for (const event of result.events) {+ expect(event.timestamp).toBeLessThanOrEqual(NOW + MAX_FUTURE_SKEW_MS);+ expect(event.timestamp).toBeGreaterThanOrEqual(NOW - MAX_PAST_SKEW_MS);+ }+ });++ it("preserves the gaps between events, so playback timing survives", () => {+ const year2076 = Date.UTC(2076, 1, 3, 12, 56, 50);+ const gaps = [0, 250, 1750, 187_000];+ const { events } = correctReplayClockSkew(+ batch(...gaps.map(g => year2076 + g)),+ NOW+ );++ const shifted = events.map(e => e.timestamp - events[0].timestamp);+ expect(shifted).toEqual(gaps);+ });++ it("is not dragged off by a single corrupt timestamp in a healthy batch", () => {+ const events = batch(NOW - 3000, NOW - 2000, NOW - 1000, Date.UTC(2064, 0, 5));+ const { events: corrected, skewMs } = correctReplayClockSkew(events, NOW);++ expect(skewMs).toBe(0);+ expect(corrected.slice(0, 3).map(e => e.timestamp)).toEqual([NOW - 3000, NOW - 2000, NOW - 1000]);+ // the outlier alone is clamped back into range+ expect(corrected[3].timestamp).toBe(NOW + MAX_FUTURE_SKEW_MS);+ });++ it("pulls forward a batch older than the replay TTL", () => {+ const ancient = NOW - MAX_PAST_SKEW_MS - 90 * 24 * 60 * 60 * 1000;+ const { events, skewMs } = correctReplayClockSkew(batch(ancient, ancient + 1000), NOW);++ expect(skewMs).toBeLessThan(0);+ expect(events[0].timestamp).toBeGreaterThanOrEqual(NOW - MAX_PAST_SKEW_MS);+ });++ it("handles an empty batch", () => {+ expect(correctReplayClockSkew([], NOW)).toEqual({ events: [], skewMs: 0 });+ });++ // Non-finite timestamps land on the bound they were heading for, never NaN.+ it("clamps a lone Infinity timestamp instead of returning NaN", () => {+ const { events, skewMs } = correctReplayClockSkew([{ timestamp: Infinity }], NOW);++ expect(events[0].timestamp).toBe(NOW + MAX_FUTURE_SKEW_MS);+ expect(Number.isFinite(skewMs)).toBe(true);+ });++ it("clamps a lone -Infinity timestamp", () => {+ const { events } = correctReplayClockSkew([{ timestamp: -Infinity }], NOW);++ expect(events[0].timestamp).toBe(NOW - MAX_PAST_SKEW_MS);+ });++ it("does not let one non-finite timestamp shift its healthy neighbours", () => {+ const { events } = correctReplayClockSkew(+ [{ timestamp: NOW }, { timestamp: NOW + 100 }, { timestamp: Infinity }, { timestamp: NOW + 300 }],+ NOW+ );++ expect(events.map(event => event.timestamp)).toEqual([NOW, NOW + 100, NOW + MAX_FUTURE_SKEW_MS, NOW + 300]);+ });++ it("falls back to now for a NaN timestamp", () => {+ const { events } = correctReplayClockSkew([{ timestamp: NaN }], NOW);++ expect(events[0].timestamp).toBe(NOW);+ });+});diff --git a/server/src/services/replay/replayClockSkew.ts b/server/src/services/replay/replayClockSkew.tsnew file mode 100644index 000000000..bdddd13f6--- /dev/null+++ b/server/src/services/replay/replayClockSkew.ts@@ -0,0 +1,85 @@+// Replay event timestamps come from the browser, not the server, so they carry+// whatever the device's clock says. Most are fine; a handful of devices are+// years out. Those events landed in partitions dated 2032, 2064, 2076 and 2090,+// which grows the partition list without bound and β because the table's TTL is+// `start_time + 30 DAY` β leaves rows that never expire.+//+// Individual timestamps cannot simply be pinned to `now`: rrweb reconstructs+// playback from the gaps between them, so flattening a batch would destroy the+// recording. The skew is a property of the device, not of any one event, so+// correct it as one offset applied to the whole batch and keep every delta.++// A day of tolerance absorbs ordinary clock drift and timezone-confused+// devices without reshaping their batches.+export const MAX_FUTURE_SKEW_MS = 24 * 60 * 60 * 1000;++// Matches the replay TTL: anything older than this is already unreadable, so+// there is nothing to preserve by keeping its original date.+export const MAX_PAST_SKEW_MS = 30 * 24 * 60 * 60 * 1000;++// A browser can put a non-finite value on the wire: JSON `1e400` parses as+// Infinity and `z.number()` admits it. Left alone it poisons the median and+// then every corrected timestamp with NaN β which ClickHouse cannot encode, so+// one bad event would fail the whole batch and the recorder would retry it+// forever. Treat it as the most extreme possible skew and let the clamp below+// pull it back to a bound, the same as any other unusable clock.+function sanitize(timestamp: number, nowMs: number, lowerBound: number, upperBound: number): number {+ if (Number.isFinite(timestamp)) return timestamp;+ if (timestamp === Infinity) return upperBound;+ if (timestamp === -Infinity) return lowerBound;+ // NaN carries no ordering at all, so there is no bound to prefer.+ return nowMs;+}++function median(values: number[]): number {+ const sorted = [...values].sort((a, b) => a - b);+ const mid = Math.floor(sorted.length / 2);+ return sorted.length % 2 === 0 ? (sorted[mid - 1] + sorted[mid]) / 2 : sorted[mid];+}++export type ClockSkewCorrection<T> = {+ events: T[];+ /** How far the device's clock was off, in ms. 0 when nothing was corrected. */+ skewMs: number;+};++/**+ * Shifts a batch of client-timestamped events onto the server's clock when the+ * device is implausibly far off, preserving the interval between every pair of+ * events.+ *+ * The offset is taken from the median rather than the max so that one corrupt+ * timestamp cannot drag an otherwise healthy batch backwards; a per-event clamp+ * afterwards catches those stragglers individually.+ */+export function correctReplayClockSkew<T extends { timestamp: number }>(+ events: T[],+ nowMs: number+): ClockSkewCorrection<T> {+ if (events.length === 0) return { events, skewMs: 0 };++ const upperBound = nowMs + MAX_FUTURE_SKEW_MS;+ const lowerBound = nowMs - MAX_PAST_SKEW_MS;++ const timestamps = events.map(event => sanitize(event.timestamp, nowMs, lowerBound, upperBound));+ const anchor = median(timestamps);++ let skewMs = 0;+ if (anchor > upperBound) skewMs = anchor - nowMs;+ else if (anchor < lowerBound) skewMs = anchor - nowMs;++ // Returning the original array is only safe when nothing needed sanitizing;+ // otherwise the caller would get the non-finite values straight back.+ const allFinite = events.every(event => Number.isFinite(event.timestamp));+ if (skewMs === 0 && allFinite && timestamps.every(t => t <= upperBound && t >= lowerBound)) {+ return { events, skewMs: 0 };+ }++ return {+ events: events.map((event, index) => ({+ ...event,+ timestamp: Math.min(Math.max(timestamps[index] - skewMs, lowerBound), upperBound),+ })),+ skewMs,+ };+}diff --git a/server/src/services/replay/sessionReplayIngestService.ts b/server/src/services/replay/sessionReplayIngestService.tsindex 9ca52c4b6..167dbbebe 100644--- a/server/src/services/replay/sessionReplayIngestService.ts+++ b/server/src/services/replay/sessionReplayIngestService.ts@@ -1,13 +1,26 @@import { DateTime } from "luxon";import { clickhouse } from "../../db/clickhouse/clickhouse.js";import { RecordSessionReplayRequest } from "../../types/sessionReplay.js";-import { processResults } from "../../api/analytics/utils/utils.js";+import { createServiceLogger } from "../../lib/logger/logger.js";+import { correctReplayClockSkew } from "./replayClockSkew.js";import { parseTrackingData } from "./trackingUtils.js";import { sessionsService } from "../sessions/sessionsService.js";import { userIdService } from "../userId/userIdService.js";import { r2Storage } from "../storage/r2StorageService.js";import { siteConfig } from "../../lib/siteConfig.js";+const logger = createServiceLogger("session-replay-ingest");++/** What a single batch observed about its session β no history involved. */+type BatchStats = {+ startTime: Date;+ endTime: Date;+ eventCount: number;+ compressedSizeBytes: number;+ screenWidth: number;+ screenHeight: number;+};+export interface RequestMetadata {userAgent: string;ipAddress: string;@@ -25,7 +38,14 @@ export class SessionReplayIngestService {request: RecordSessionReplayRequest,requestMeta?: RequestMetadata): Promise<void> {- const { userId: clientUserId, events, metadata } = request;+ const { userId: clientUserId, events: rawEvents, metadata } = request;++ // Device clocks, not ours. Correct the whole batch onto server time before+ // anything downstream partitions or TTLs on these values.+ const { events, skewMs } = correctReplayClockSkew(rawEvents, Date.now());+ if (skewMs !== 0) {+ logger.warn({ siteId, skewMs, eventCount: events.length }, "Corrected replay clock skew");+ }// Always generate device fingerprint (anonymous user ID) server-sideconst deviceFingerprint = await userIdService.generateUserId(@@ -121,7 +141,28 @@ export class SessionReplayIngestService {// Update or insert metadataif (metadata) {- await this.updateSessionMetadata(siteId, sessionId, userId, identifiedUserId, metadata, requestMeta);+ // An events-free batch still carries metadata worth recording, but it has+ // no timestamps to bound it β anchor those on server time rather than+ // letting Math.min of nothing produce an epoch-dated row.+ const timestamps = events.length > 0 ? events.map(event => event.timestamp) : [Date.now()];+ const batchStats: BatchStats = {+ startTime: new Date(Math.min(...timestamps)),+ endTime: new Date(Math.max(...timestamps)),+ eventCount: eventsToInsert.length,+ compressedSizeBytes: eventsToInsert.reduce((total, event) => total + event.event_size_bytes, 0),+ screenWidth: metadata.viewportWidth || 0,+ screenHeight: metadata.viewportHeight || 0,+ };++ await this.updateSessionMetadata(+ siteId,+ sessionId,+ userId,+ identifiedUserId,+ metadata,+ batchStats,+ requestMeta+ );}}@@ -131,42 +172,13 @@ export class SessionReplayIngestService {userId: string,identifiedUserId: string,metadata: any,+ batchStats: BatchStats,requestMeta?: RequestMetadata): Promise<void> {- // Get existing session info from events table- const sessionInfo = await clickhouse.query({- query: `- SELECT- MIN(timestamp) as start_time,- MAX(timestamp) as end_time,- COUNT() as event_count,- SUM(event_size_bytes) as compressed_size_bytes,- MAX(viewport_width) as screen_width,- MAX(viewport_height) as screen_height- FROM session_replay_events- WHERE site_id = {siteId:UInt16}- AND session_id = {sessionId:String}- `,- query_params: { siteId, sessionId },- format: "JSONEachRow",- });-- type SessionInfoResult = {- start_time: string;- end_time: string | null;- event_count: number;- compressed_size_bytes: number;- screen_width: number | null;- screen_height: number | null;- };-- const sessionResults = await processResults<SessionInfoResult>(sessionInfo);-- if (!sessionResults || sessionResults.length === 0) return;-- const sessionReplayData = sessionResults[0];-- // Parse tracking data from request metadata+ // Append-only. This used to read the session back first β MIN/MAX/COUNT/SUM+ // across every row the session had ever written, on every batch β purely to+ // recompute a cumulative row. The aggregating table does that combining at+ // merge time, so a batch now writes only what it saw.let trackingData: any = {};if (requestMeta?.userAgent) {try {@@ -181,33 +193,26 @@ export class SessionReplayIngestService {urlObj.search || "", // querystring from URLhostname,metadata.language || "", // language from client- sessionReplayData.screen_width || metadata.viewportWidth || 0,- sessionReplayData.screen_height || metadata.viewportHeight || 0+ batchStats.screenWidth || metadata.viewportWidth || 0,+ batchStats.screenHeight || metadata.viewportHeight || 0);} catch (error) {- console.error("Error parsing tracking data for session replay:", error);+ logger.error({ siteId, sessionId, err: error }, "Error parsing tracking data for session replay");}}- // Calculate duration- const startTime = new Date(sessionReplayData.start_time);- const endTime = sessionReplayData.end_time ? new Date(sessionReplayData.end_time) : null;- const durationMs = endTime ? endTime.getTime() - startTime.getTime() : null;-- // Insert or update metadataawait clickhouse.insert({- table: "session_replay_metadata",+ table: "session_replay_metadata_v2",values: [{site_id: siteId,session_id: sessionId,user_id: userId,identified_user_id: identifiedUserId,- start_time: DateTime.fromJSDate(startTime).toFormat("yyyy-MM-dd HH:mm:ss"),- end_time: endTime ? DateTime.fromJSDate(endTime).toFormat("yyyy-MM-dd HH:mm:ss") : null,- duration_ms: durationMs,- event_count: sessionReplayData.event_count || 0,- compressed_size_bytes: sessionReplayData.compressed_size_bytes || 0,+ start_time: DateTime.fromJSDate(batchStats.startTime).toFormat("yyyy-MM-dd HH:mm:ss.SSS"),+ end_time: DateTime.fromJSDate(batchStats.endTime).toFormat("yyyy-MM-dd HH:mm:ss.SSS"),+ event_count: batchStats.eventCount,+ compressed_size_bytes: batchStats.compressedSizeBytes,page_url: metadata.pageUrl || "",country: trackingData.country || "",region: trackingData.region || "",@@ -219,8 +224,8 @@ export class SessionReplayIngestService {operating_system: trackingData.operatingSystem || "",operating_system_version: trackingData.operatingSystemVersion || "",language: trackingData.language || "",- screen_width: sessionReplayData.screen_width || metadata?.viewportWidth || 0,- screen_height: sessionReplayData.screen_height || metadata?.viewportHeight || 0,+ screen_width: batchStats.screenWidth || metadata?.viewportWidth || 0,+ screen_height: batchStats.screenHeight || metadata?.viewportHeight || 0,device_type: trackingData.deviceType || "",channel: trackingData.channel || "",hostname: trackingData.hostname || "",diff --git a/server/src/services/replay/sessionReplayQueryService.ts b/server/src/services/replay/sessionReplayQueryService.tsindex 419bb0fb8..ab5f564ff 100644--- a/server/src/services/replay/sessionReplayQueryService.ts+++ b/server/src/services/replay/sessionReplayQueryService.ts@@ -15,6 +15,40 @@ import { matchesUser } from "../../api/analytics/utils/effectiveUserId.js";* Service responsible for querying/retrieving session replay data* Handles listing sessions and getting replay events*/+const DURATION_MS = `dateDiff('millisecond', start_time, end_time)`;++// SELECT * is not usable against an aggregating table: it would return the raw+// per-batch columns and omit the derived duration. List what readers consume.+const METADATA_COLUMNS = `+ site_id,+ session_id,+ user_id,+ identified_user_id,+ start_time,+ end_time,+ ${DURATION_MS} AS duration_ms,+ event_count,+ compressed_size_bytes,+ page_url,+ country,+ region,+ city,+ lat,+ lon,+ browser,+ browser_version,+ operating_system,+ operating_system_version,+ language,+ screen_width,+ screen_height,+ device_type,+ channel,+ hostname,+ referrer,+ has_replay_data+`;+export class SessionReplayQueryService {async getSessionReplayList(siteId: number,@@ -40,7 +74,10 @@ export class SessionReplayQueryService {}if (minDuration !== undefined) {- whereConditions.push(`duration_ms >= {minDuration:UInt32}`);+ // Derived from the merged bounds rather than stored: each batch only+ // knows its own slice of the session, so duration is only meaningful+ // after FINAL has combined them.+ whereConditions.push(`${DURATION_MS} >= {minDuration:UInt32}`);queryParams.minDuration = minDuration * 1000; // Convert seconds to milliseconds}@@ -55,7 +92,7 @@ export class SessionReplayQueryService {if (filterStatement) {sessionIdsSubquery = `SELECT DISTINCT srm.session_id- FROM session_replay_metadata srm+ FROM session_replay_metadata_v2 srmFINALWHERE srm.site_id = {siteId:UInt16}AND srm.session_id IN (@@ -79,7 +116,7 @@ export class SessionReplayQueryService {identified_user_id,start_time,end_time,- duration_ms,+ ${DURATION_MS} AS duration_ms,page_url,event_count,country,@@ -92,7 +129,7 @@ export class SessionReplayQueryService {device_type,screen_width,screen_height- FROM session_replay_metadata+ FROM session_replay_metadata_v2FINALWHERE ${whereConditions.join(" AND ")}AND event_count >= 2@@ -120,10 +157,10 @@ export class SessionReplayQueryService {// Get metadataconst metadataResult = await clickhouse.query({query: `- SELECT *- FROM session_replay_metadata+ SELECT ${METADATA_COLUMNS}+ FROM session_replay_metadata_v2FINAL- WHERE site_id = {siteId:UInt16}+ WHERE site_id = {siteId:UInt16}AND session_id = {sessionId:String}LIMIT 1`,@@ -250,8 +287,8 @@ export class SessionReplayQueryService {async getSessionReplayMetadata(siteId: number, sessionId: string): Promise<SessionReplayMetadata | null> {const result = await clickhouse.query({query: `- SELECT *- FROM session_replay_metadata+ SELECT ${METADATA_COLUMNS}+ FROM session_replay_metadata_v2FINALWHERE site_id = {siteId:UInt16}AND session_id = {sessionId:String}@@ -310,7 +347,7 @@ export class SessionReplayQueryService {await clickhouse.command({query: `- DELETE FROM session_replay_metadata+ DELETE FROM session_replay_metadata_v2WHERE site_id = {siteId:UInt16}AND session_id = {sessionId:String}`,diff --git a/server/src/services/sites/siteConfigurationLifecycle.ts b/server/src/services/sites/siteConfigurationLifecycle.tsindex 365f043ce..3b82918c2 100644--- a/server/src/services/sites/siteConfigurationLifecycle.ts+++ b/server/src/services/sites/siteConfigurationLifecycle.ts@@ -375,7 +375,7 @@ class SiteConfigurationLifecycle {query_params: { id: siteId },}),clickhouse.command({- query: "DELETE FROM session_replay_metadata WHERE site_id = {id:UInt32}",+ query: "DELETE FROM session_replay_metadata_v2 WHERE site_id = {id:UInt32}",query_params: { id: siteId },}),]);diff --git a/server/src/services/telemetryService.ts b/server/src/services/telemetryService.tsindex 34ad65550..b5a5129ab 100644--- a/server/src/services/telemetryService.ts+++ b/server/src/services/telemetryService.ts@@ -55,7 +55,7 @@ class TelemetryService {// Get table row counts from ClickHouseprivate async getTableCounts() {- const tables = ["events", "session_replay_events", "session_replay_metadata"];+ const tables = ["events", "session_replay_events", "session_replay_metadata_v2"];const counts: Record<string, number> = {};diff --git a/server/src/services/tracker/identifyService.ts b/server/src/services/tracker/identifyService.tsindex 143a27a8b..492085010 100644--- a/server/src/services/tracker/identifyService.ts+++ b/server/src/services/tracker/identifyService.ts@@ -2,9 +2,9 @@ import { FastifyReply, FastifyRequest } from "fastify";import { eq, and, sql } from "drizzle-orm";import { z } from "zod";import { db } from "../../db/postgres/postgres.js";-import { clickhouse } from "../../db/clickhouse/clickhouse.js";import { userProfiles, userAliases } from "../../db/postgres/schema.js";import { siteConfig } from "../../lib/siteConfig.js";+import { identityBackfillQueue } from "./identityBackfillQueue.js";import { userIdService } from "../userId/userIdService.js";import { resolveClientIp } from "./resolveClientIp.js";import { createServiceLogger } from "../../lib/logger/logger.js";@@ -42,33 +42,18 @@ const BACKFILL_DAYS = 30;// days: null backfills the device's full history β only for explicit admin// actions (dashboard identify), where the operator asserts the whole history// belongs to this user and the unbounded partition scan is a one-off.-export async function backfillIdentifiedUserId(+// Queues the assignment rather than mutating immediately. Each mutation+// submission takes the MergeTree parts lock, and at production identify rates+// that lock was being taken every ~12 seconds per table, stalling concurrent+// inserts and selects. The queue collapses an interval's worth of identities+// into one mutation per table; see identityBackfillQueue.ts.+export function backfillIdentifiedUserId(siteId: number,anonymousId: string,userId: string,days: number | null = BACKFILL_DAYS) {- try {- // session_replay_metadata has no `timestamp` column; its time column is- // `start_time`. Using `timestamp` there throws ClickHouse error 47- // (UNKNOWN_IDENTIFIER), so map each table to its actual time column.- const tables: Array<{ name: string; timeColumn: string }> = [- { name: "events", timeColumn: "timestamp" },- { name: "session_replay_events", timeColumn: "timestamp" },- { name: "session_replay_metadata", timeColumn: "start_time" },- ];- for (const { name, timeColumn } of tables) {- await clickhouse.command({- query: `ALTER TABLE ${name} UPDATE identified_user_id = {userId: String} WHERE site_id = {siteId: UInt16} AND user_id = {anonymousId: String} AND identified_user_id = ''${- days !== null ? ` AND ${timeColumn} >= now() - INTERVAL {days: UInt16} DAY` : ""- }`,- query_params: { userId, siteId, anonymousId, ...(days !== null ? { days } : {}) },- });- }- logger.info({ siteId, anonymousId, userId }, "Backfilled identified_user_id in ClickHouse");- } catch (error) {- logger.error({ siteId, anonymousId, userId, err: error }, "Error backfilling identified_user_id");- }+ identityBackfillQueue.enqueue({ siteId, anonymousId, userId }, days);}export async function handleIdentify(request: FastifyRequest, reply: FastifyReply) {diff --git a/server/src/services/tracker/identityBackfillQueue.test.ts b/server/src/services/tracker/identityBackfillQueue.test.tsnew file mode 100644index 000000000..eac6db09a--- /dev/null+++ b/server/src/services/tracker/identityBackfillQueue.test.ts@@ -0,0 +1,198 @@+import { beforeEach, describe, expect, it, vi } from "vitest";++const mocks = vi.hoisted(() => ({+ command: vi.fn(),+}));++vi.mock("../../db/clickhouse/clickhouse.js", () => ({+ clickhouse: { command: mocks.command },+}));++vi.mock("../../lib/logger/logger.js", () => ({+ createServiceLogger: () => ({ info: vi.fn(), warn: vi.fn(), error: vi.fn() }),+}));++vi.useFakeTimers();+const { identityBackfillQueue } = await import("./identityBackfillQueue.js");++const TABLE_COUNT = 3;++/** Every key interpolated into the mutations of the most recent flush. */+function flushedKeys() {+ return mocks.command.mock.calls.map(([{ query_params }]) => query_params.keys);+}++function deferred() {+ let resolve!: () => void;+ let reject!: (error: Error) => void;+ const promise = new Promise<void>((res, rej) => {+ resolve = res;+ reject = rej;+ });+ return { promise, resolve, reject };+}++beforeEach(async () => {+ // Drain anything a previous test left buffered so each starts empty.+ mocks.command.mockReset();+ mocks.command.mockResolvedValue(undefined);+ await identityBackfillQueue.flush();+ mocks.command.mockReset();+ mocks.command.mockResolvedValue(undefined);+});++describe("identityBackfillQueue", () => {+ it("collapses many identifies into one mutation per table", async () => {+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "user-a" }, 30);+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-b", userId: "user-b" }, 30);++ await identityBackfillQueue.flush();++ expect(mocks.command).toHaveBeenCalledTimes(TABLE_COUNT);+ for (const keys of flushedKeys()) {+ expect(keys).toEqual(["7:anon-a", "7:anon-b"]);+ }+ });++ it("keeps the first assignment when a device identifies twice in a window", async () => {+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "first" }, 30);+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "second" }, 30);++ await identityBackfillQueue.flush();++ expect(mocks.command.mock.calls[0][0].query_params.userIds).toEqual(["first"]);+ });++ it("keeps windows apart so an all-history backfill does not widen the routine ones", async () => {+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "user-a" }, 30);+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-b", userId: "user-b" }, null);++ await identityBackfillQueue.flush();++ const windowed = mocks.command.mock.calls.filter(([{ query_params }]) => query_params.days === 30);+ const unbounded = mocks.command.mock.calls.filter(([{ query_params }]) => query_params.days === undefined);++ expect(windowed).toHaveLength(TABLE_COUNT);+ expect(unbounded).toHaveLength(TABLE_COUNT);+ expect(windowed[0][0].query).toContain("INTERVAL {days: UInt16} DAY");+ expect(unbounded[0][0].query).not.toContain("INTERVAL");+ });++ it("retries assignments whose mutation failed instead of dropping them", async () => {+ mocks.command.mockRejectedValueOnce(new Error("ClickHouse timeout"));++ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "user-a" }, 30);+ await identityBackfillQueue.flush();+ expect(mocks.command).toHaveBeenCalledTimes(TABLE_COUNT);++ mocks.command.mockReset();+ mocks.command.mockResolvedValue(undefined);++ await identityBackfillQueue.flush();++ expect(mocks.command).toHaveBeenCalledTimes(TABLE_COUNT);+ expect(mocks.command.mock.calls[0][0].query_params.keys).toEqual(["7:anon-a"]);+ });++ it("gives up after repeated failures rather than retrying forever", async () => {+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "user-a" }, 30);++ for (let attempt = 0; attempt < 3; attempt++) {+ mocks.command.mockReset();+ mocks.command.mockRejectedValue(new Error("ClickHouse down"));+ await identityBackfillQueue.flush();+ expect(mocks.command).toHaveBeenCalledTimes(TABLE_COUNT);+ }++ mocks.command.mockReset();+ mocks.command.mockResolvedValue(undefined);+ await identityBackfillQueue.flush();++ expect(mocks.command).not.toHaveBeenCalled();+ });++ it("drains identities that arrived while a flush was in progress", async () => {+ const inFlight = deferred();+ mocks.command.mockReturnValueOnce(inFlight.promise);++ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "user-a" }, 30);+ const first = identityBackfillQueue.flush();++ // Arrives after the first flush took its snapshot.+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-b", userId: "user-b" }, 30);++ inFlight.resolve();+ await first;+ await identityBackfillQueue.flush();++ const allKeys = flushedKeys().flat();+ expect(allKeys).toContain("7:anon-a");+ expect(allKeys).toContain("7:anon-b");+ });++ it("waits for an in-flight flush so shutdown does not abandon it", async () => {+ const inFlight = deferred();+ mocks.command.mockReturnValueOnce(inFlight.promise);++ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "user-a" }, 30);+ const first = identityBackfillQueue.flush();++ let shutdownDone = false;+ // What server shutdown does: flush() must not resolve while work is in flight.+ const shutdown = identityBackfillQueue.flush().then(() => {+ shutdownDone = true;+ });++ await Promise.resolve();+ expect(shutdownDone).toBe(false);++ inFlight.resolve();+ await first;+ await shutdown;++ expect(shutdownDone).toBe(true);+ });++ it("splits an oversized backlog across several mutations", async () => {+ const inFlight = deferred();+ mocks.command.mockReturnValueOnce(inFlight.promise);++ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "held", userId: "u" }, 30);+ const first = identityBackfillQueue.flush();++ // Everything below arrives while that flush is stuck, so it all lands in+ // one backlog rather than being spread over separate windows.+ for (let i = 0; i < 5001; i++) {+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: `anon-${i}`, userId: `user-${i}` }, 30);+ }++ inFlight.resolve();+ await first;+ // Joins the end of the chain, including the drains the size trigger queued.+ await identityBackfillQueue.flush();++ const mutations = mocks.command.mock.calls.map(([{ query_params }]) => query_params.keys);+ expect(mutations.every(keys => keys.length <= 5000)).toBe(true);++ const covered = new Set(mutations.flat());+ for (let i = 0; i < 5001; i++) {+ expect(covered.has(`7:anon-${i}`)).toBe(true);+ }+ expect(covered.has("7:held")).toBe(true);+ });++ it("retries a shutdown failure instead of exiting on top of it", async () => {+ mocks.command.mockRejectedValueOnce(new Error("ClickHouse blip"));++ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "user-a" }, 30);+ await identityBackfillQueue.drainCompletely();++ // One failed table, then a clean retry round: nothing is left behind.+ expect(mocks.command).toHaveBeenCalledTimes(TABLE_COUNT * 2);++ mocks.command.mockReset();+ mocks.command.mockResolvedValue(undefined);+ await identityBackfillQueue.flush();+ expect(mocks.command).not.toHaveBeenCalled();+ });+});diff --git a/server/src/services/tracker/identityBackfillQueue.ts b/server/src/services/tracker/identityBackfillQueue.tsnew file mode 100644index 000000000..6522bfdd7--- /dev/null+++ b/server/src/services/tracker/identityBackfillQueue.ts@@ -0,0 +1,200 @@+import { clickhouse } from "../../db/clickhouse/clickhouse.js";+import { createServiceLogger } from "../../lib/logger/logger.js";++// Identify calls arrive continuously, and each one used to submit three+// `ALTER TABLE β¦ UPDATE` mutations immediately. Mutation submission takes the+// MergeTree parts lock, so a steady identify rate turns into a steady stream of+// lock acquisitions: measured at a 12-second median gap on `events`, which is+// short enough that inserts and selects are near-permanently contending. The+// table is not slower on average β it periodically freezes, and p90 insert+// latency runs ~100x baseline inside a submission window.+//+// Buffering fixes the frequency rather than the cost. One flush covers every+// identity assigned in the interval, so N identifies cost one mutation per+// table instead of N, and the gaps between them are long enough for the lock to+// stay uncontended in between.+const FLUSH_INTERVAL_MS = 5 * 60 * 1000;++// A mutation interpolates one array element per identity, so this bounds the+// statement rather than the buffer: crossing it triggers an early flush, and a+// flush that finds more than this β identifies that arrived while the previous+// one was in flight β splits into several mutations rather than one huge one.+const MAX_IDENTITIES_PER_MUTATION = 5000;++// A mutation that fails is retried on later flushes. Re-running a partially+// applied backfill is harmless β the `identified_user_id = ''` guard makes the+// tables that already succeeded a no-op β but a permanently failing assignment+// must not circulate forever.+const MAX_ATTEMPTS = 3;++type BackfillTable = { name: string; timeColumn: string };++// session_replay_metadata has no `timestamp` column; its time column is+// `start_time`. Using `timestamp` there throws ClickHouse error 47+// (UNKNOWN_IDENTIFIER), so map each table to its actual time column.+const TABLES: BackfillTable[] = [+ { name: "events", timeColumn: "timestamp" },+ { name: "session_replay_events", timeColumn: "timestamp" },+ { name: "session_replay_metadata_v2", timeColumn: "start_time" },+];++export type IdentityAssignment = {+ siteId: number;+ anonymousId: string;+ userId: string;+};++type PendingIdentity = IdentityAssignment & { attempts: number };++class IdentityBackfillQueue {+ // Grouped by backfill window, because the window is part of the mutation's+ // WHERE clause: folding a `days: null` admin backfill in with the routine+ // 30-day ones would widen every assignment to a full-history partition scan.+ private pending = new Map<number | null, Map<string, PendingIdentity>>();++ // Flushes are serialised by chaining rather than by a boolean guard. A guard+ // would make a concurrent flush() a silent no-op, which loses two things: a+ // size-triggered flush during a slow one would never run, and shutdown's+ // `await flush()` would return while work was still in flight.+ private chain: Promise<void> = Promise.resolve();++ private logger = createServiceLogger("identity-backfill-queue");++ constructor() {+ setInterval(() => {+ void this.flush();+ }, FLUSH_INTERVAL_MS);+ }++ enqueue(assignment: IdentityAssignment, days: number | null) {+ this.add({ ...assignment, attempts: 0 }, days);++ const group = this.pending.get(days);+ if (group && group.size >= MAX_IDENTITIES_PER_MUTATION) {+ void this.flush();+ }+ }++ private add(entry: PendingIdentity, days: number | null) {+ let group = this.pending.get(days);+ if (!group) {+ group = new Map();+ this.pending.set(days, group);+ }++ // First assignment wins, matching the un-batched behaviour: the mutation+ // only touches rows where identified_user_id is still empty, so a second+ // identify for the same device in the same window used to find nothing left+ // to update.+ const key = `${entry.siteId}:${entry.anonymousId}`;+ if (!group.has(key)) {+ group.set(key, entry);+ }+ }++ /** Resolves once everything queued at call time has been attempted. */+ flush(): Promise<void> {+ this.chain = this.chain.then(() => this.drain());+ return this.chain;+ }++ /**+ * Drains repeatedly until nothing is left to retry. Only for shutdown: during+ * normal operation a failed assignment should wait for the next interval+ * rather than hot-looping against a ClickHouse that is already struggling,+ * but at shutdown there is no next interval, so a requeued retry would be+ * thrown away by process exit.+ */+ async drainCompletely(): Promise<void> {+ for (let round = 0; round < MAX_ATTEMPTS; round++) {+ await this.flush();+ if (this.pending.size === 0) return;+ }+ }++ private async drain() {+ const groups = [...this.pending.entries()].filter(([, assignments]) => assignments.size > 0);+ if (groups.length === 0) return;++ this.pending = new Map();++ for (const [days, assignments] of groups) {+ const queued = [...assignments.values()];+ for (let start = 0; start < queued.length; start += MAX_IDENTITIES_PER_MUTATION) {+ await this.runBackfill(days, queued.slice(start, start + MAX_IDENTITIES_PER_MUTATION));+ }+ }+ }++ private async runBackfill(days: number | null, assignments: PendingIdentity[]) {+ // Anonymous ids are salted per site, so a bare `user_id IN (β¦)` would be+ // correct in practice β but it would also let one site's list match another+ // site's rows. Pairing the two into a single key keeps the mutation exact,+ // while the separate `site_id IN (β¦)` keeps the primary-key prefix usable+ // for pruning.+ const keys = assignments.map(a => `${a.siteId}:${a.anonymousId}`);+ const userIds = assignments.map(a => a.userId);+ const siteIds = [...new Set(assignments.map(a => a.siteId))];++ const failedTables: string[] = [];++ for (const { name, timeColumn } of TABLES) {+ try {+ await clickhouse.command({+ query: `+ ALTER TABLE ${name}+ UPDATE identified_user_id = transform(+ concat(toString(site_id), ':', user_id),+ {keys: Array(String)},+ {userIds: Array(String)},+ identified_user_id+ )+ WHERE site_id IN {siteIds: Array(UInt16)}+ AND concat(toString(site_id), ':', user_id) IN {keys: Array(String)}+ AND identified_user_id = ''${+ days !== null ? `\n AND ${timeColumn} >= now() - INTERVAL {days: UInt16} DAY` : ""+ }+ `,+ query_params: { keys, userIds, siteIds, ...(days !== null ? { days } : {}) },+ });+ } catch (error) {+ failedTables.push(name);+ this.logger.error(+ { table: name, identities: assignments.length, days, err: error },+ "Error backfilling identified_user_id"+ );+ }+ }++ if (failedTables.length === 0) {+ this.logger.info({ identities: assignments.length, days }, "Flushed identity backfill");+ return;+ }++ // Requeue rather than drop: the queue is the only thing that knows about+ // these assignments. The alias row in Postgres already exists, so a later+ // identify for the same device takes the "alias unchanged" path and never+ // schedules another backfill β a dropped failure leaves those rows+ // anonymous permanently.+ const retryable = assignments.filter(a => a.attempts + 1 < MAX_ATTEMPTS);+ const exhausted = assignments.length - retryable.length;++ for (const assignment of retryable) {+ this.add({ ...assignment, attempts: assignment.attempts + 1 }, days);+ }++ this.logger.warn(+ { failedTables, requeued: retryable.length, dropped: exhausted, days },+ "Identity backfill partially failed"+ );++ if (exhausted > 0) {+ this.logger.error(+ { dropped: exhausted, days, maxAttempts: MAX_ATTEMPTS },+ "Giving up on identity backfill; those rows stay anonymous"+ );+ }+ }+}++export const identityBackfillQueue = new IdentityBackfillQueue();diff --git a/server/src/services/usageService.ts b/server/src/services/usageService.tsindex 9326749cf..6ec11cc52 100644--- a/server/src/services/usageService.ts+++ b/server/src/services/usageService.ts@@ -238,7 +238,7 @@ class UsageService {SELECTsite_id,uniq(session_id) as count- FROM session_replay_metadata+ FROM session_replay_metadata_v2WHERE start_time >= toDate({periodStart:String})GROUP BY site_id`,