Skip to content

🐸 Rybbit - open-source and privacy-friendly alternative to Google Analytics that is 10x more intuitive.

perf(clickhouse): cut mutation-driven parts-lock contention on the hot ingest path (#1131)

Goldflagc65b39312b5e
diff --git a/clickhouse/REPLAY_METADATA_V2.md b/clickhouse/REPLAY_METADATA_V2.md
new file mode 100644
index 000000000..da2a35cd0
--- /dev/null
+++ b/clickhouse/REPLAY_METADATA_V2.md
@@ -0,0 +1,199 @@
+# Cutover: `session_replay_metadata` β†’ `session_replay_metadata_v2`
+
+The old table is a `ReplacingMergeTree` holding one cumulative row per session.
+Keeping that row current meant every replay batch re-read the whole session
+(`SELECT MIN/MAX/COUNT/SUM … FROM session_replay_events WHERE session_id = …`)
+and rewrote it. In one six-day window that read scanned **818 billion rows β€”
+76% of everything the cluster read** β€” and the rewrites left **2.5 million
+single-row parts**.
+
+`session_replay_metadata_v2` is an `AggregatingMergeTree`. Each batch inserts
+only what it observed and the engine combines rows at merge time, so ingest
+never reads back. Column names and their post-merge meanings are unchanged;
+readers already used `FINAL`, which is all that is needed.
+
+## Known limitation: snapshot columns merge independently
+
+`page_url`, the geo fields, `browser`, `language` and the rest use
+`SimpleAggregateFunction(anyLast, …)`, so each one merges on its own. The old
+`ReplacingMergeTree(created_at)` picked one whole winning row, so a session
+whose batches disagreed always showed a single coherent snapshot; the new table
+can assemble one from several batches.
+
+Measured over 30 days of production, of the 423 sessions carrying more than one
+metadata version:
+
+| field | sessions that differed |
+| --- | --- |
+| `page_url` | 62 |
+| `screen_width` / `screen_height` | 28 / 32 |
+| `region`, `city`, `lat` | 18 |
+| `language` | 13 |
+| `country` | 11 |
+| `browser`, `operating_system`, `device_type`, `channel`, `hostname`, `referrer`, `user_id` | 0 |
+
+`screen_width`/`screen_height` are unaffected β€” they use `max`, which is what
+the old code computed anyway.
+
+The geo group is the real exposure: `country`, `region`, `city` and `lat`/`lon`
+are only meaningful together, and roughly 4% of multi-version sessions could
+show a city and a country taken from different batches.
+
+**This was weighed and accepted in August 2026.** The exposure is narrow, the
+affected fields are all still true of the session, and the alternatives cost
+more than the defect. The rest of this section records what the options were, so
+the decision can be revisited if mixed geo ever turns out to mislead someone.
+
+Making the snapshot coherent means versioning it as a unit β€” either one
+`SimpleAggregateFunction(max, Tuple(version, …))`, or `argMaxState` columns read
+with `argMaxMerge` and `GROUP BY`. Both change every read site, and the tuple
+form carries a sharp edge: `SELECT snapshot.page_url FROM … FINAL` silently
+returns the *unmerged* value, because ClickHouse resolves the named element as a
+subcolumn read and skips the merge. Positional access (`snapshot.2`) merges
+correctly, as does selecting the whole tuple in a subquery first.
+
+## Columns that are gone
+
+Two columns are gone:
+
+- `duration_ms` β€” derived at read time as
+ `dateDiff('millisecond', start_time, end_time)`, because a single batch only
+ knows its own slice of the session. `start_time`/`end_time` are
+ `DateTime64(3)` for exactly this reason: at second resolution a 900 ms replay
+ would derive a duration of 0.
+- `created_at` β€” there is no version column to order by any more.
+
+The application creates the new table on boot. The steps below move the
+existing data across and are **not** run by the app.
+
+## 1. Deploy
+
+Deploying repoints both writes and reads at `session_replay_metadata_v2`. From
+this moment the old table is frozen β€” nothing writes to it.
+
+Sessions already in flight keep working: the backfill below copies their
+totals so far, and post-deploy batches add their own increments on top. The
+aggregate columns sum, so the two halves combine to the correct total.
+
+## 2. Backfill
+
+Run once, after the deploy:
+
+```sql
+INSERT INTO session_replay_metadata_v2
+SELECT
+ site_id, session_id, user_id, identified_user_id,
+ start_time,
+ -- The old bounds are second-resolution but the old `duration_ms` was exact,
+ -- so rebuild `end_time` from it rather than copying the rounded column β€”
+ -- otherwise every historical replay's duration is re-derived to the second.
+ if(duration_ms IS NULL,
+ CAST(toDateTime64(end_time, 3) AS Nullable(DateTime64(3))),
+ CAST(toDateTime64(start_time, 3) + toIntervalMillisecond(assumeNotNull(duration_ms)) AS Nullable(DateTime64(3)))) AS end_time,
+ event_count, compressed_size_bytes,
+ page_url, country, region, city, lat, lon,
+ browser, browser_version, operating_system, operating_system_version,
+ language, screen_width, screen_height, device_type,
+ channel, hostname, referrer, has_replay_data
+FROM session_replay_metadata
+FINAL
+WHERE start_time >= now() - INTERVAL 30 DAY;
+```
+
+`start_time` still lands on a whole second for backfilled rows, because that is
+all the old table recorded. Rows written after the deploy carry true
+millisecond bounds on both ends.
+
+`FINAL` matters: without it the old table's superseded versions are copied too,
+and because the new engine **sums** `event_count` rather than replacing it,
+every session would be inflated by its own history.
+
+For the same reason this statement is **not idempotent**. Running it twice
+doubles `event_count` and `compressed_size_bytes` for every session. If it is
+interrupted, clear the table before retrying:
+
+```sql
+TRUNCATE TABLE session_replay_metadata_v2;
+```
+
+That is safe only while the deploy is fresh β€” it also discards post-deploy
+increments, which the backfill cannot recover.
+
+The 30-day bound matches the table's TTL; older rows would be deleted on the
+next TTL pass anyway.
+
+## 3. Verify
+
+Totals should agree, modulo sessions that received events between the two
+reads:
+
+```sql
+SELECT count() AS sessions, sum(event_count) AS events
+FROM session_replay_metadata FINAL
+WHERE start_time >= now() - INTERVAL 30 DAY;
+
+SELECT count() AS sessions, sum(event_count) AS events
+FROM session_replay_metadata_v2 FINAL
+WHERE start_time >= now() - INTERVAL 30 DAY;
+```
+
+Also confirm ingest is no longer reading the session back:
+
+```sql
+SELECT count()
+FROM system.query_log
+WHERE event_time > now() - INTERVAL 10 MINUTE
+ AND query LIKE '%FROM session_replay_events%'
+ AND query LIKE '%compressed_size_bytes%';
+```
+
+Expect zero.
+
+## 4. Drop the old table
+
+Once the dashboard has been checked against real sessions and you are happy to
+lose the ability to roll back:
+
+```sql
+DROP TABLE session_replay_metadata;
+```
+
+Nothing in the application references it after the deploy; it is kept only as
+the rollback path.
+
+---
+
+# Related: clearing impossible partitions
+
+Replay event timestamps come from the browser, so devices with broken clocks
+wrote rows dated 2032, 2035, 2064, 2076 and 2090. Those partitions grow the
+partition list permanently and, because the TTL is `toDateTime(start_time) + 30 DAY`, they
+never expire.
+
+Ingest now corrects this (`server/src/services/replay/replayClockSkew.ts`): a
+batch whose median timestamp is more than a day ahead, or more than 30 days
+behind, is shifted onto server time as a whole, preserving the gaps between
+events so playback still reconstructs. **No new impossible partitions are
+created after the deploy.**
+
+The existing ones still need removing. List them first:
+
+```sql
+SELECT table, partition, sum(rows) AS rows, count() AS parts
+FROM system.parts
+WHERE active AND database = currentDatabase()
+ AND table IN ('session_replay_events', 'session_replay_metadata_v2')
+ AND partition > formatDateTime(now() + INTERVAL 2 MONTH, '%Y%m')
+GROUP BY table, partition
+ORDER BY table, partition;
+```
+
+Then drop each one by name β€” a few hundred rows of unplayable recordings from
+devices whose clocks were decades out:
+
+```sql
+ALTER TABLE session_replay_events DROP PARTITION '209007';
+```
+
+Check the listing rather than copying partition ids from here; the set will
+have changed.
diff --git a/client/src/api/admin/hooks/useSites.ts b/client/src/api/admin/hooks/useSites.ts
index 02cc8c67b..257373843 100644
--- a/client/src/api/admin/hooks/useSites.ts
+++ b/client/src/api/admin/hooks/useSites.ts
@@ -31,8 +31,17 @@ export function useSiteHasData(siteId: string) {
}
return fetchSiteHasData(siteId).then(data => data.hasData);
},
- refetchInterval: 5000,
+ // Poll only until the site reports data. Once it flips true it can never
+ // flip back, so continuing to poll re-asks a settled question every 5 s for
+ // as long as the tab stays open β€” which is where the bulk of this
+ // endpoint's call volume came from.
+ //
+ // A site that genuinely has no data yet never stops polling, so that case
+ // sets the floor: 30 s is still well inside the "paste the script, watch it
+ // light up" loop this powers, at a sixth of the requests.
+ refetchInterval: query => (query.state.data === true ? false : 30_000),
staleTime: Infinity,
+ enabled: !!siteId,
});
}
diff --git a/client/src/lib/defaultTimeRange.test.ts b/client/src/lib/defaultTimeRange.test.ts
new file mode 100644
index 000000000..03ea16681
--- /dev/null
+++ b/client/src/lib/defaultTimeRange.test.ts
@@ -0,0 +1,19 @@
+import { describe, expect, it } from "vitest";
+import { capDashboardDefaultRange } from "./defaultTimeRange";
+
+describe("capDashboardDefaultRange", () => {
+ it("caps an all-time default on the users page", () => {
+ expect(capDashboardDefaultRange("all-time", "/1/users")).toBe("last-30-days");
+ });
+
+ it("leaves the all-time default alone everywhere else", () => {
+ expect(capDashboardDefaultRange("all-time", "/1")).toBe("all-time");
+ expect(capDashboardDefaultRange("all-time", "/1/sessions")).toBe("all-time");
+ expect(capDashboardDefaultRange("all-time", "/1/user/abc")).toBe("all-time");
+ });
+
+ it("never widens or rewrites a bounded default", () => {
+ expect(capDashboardDefaultRange("today", "/1/users")).toBe("today");
+ expect(capDashboardDefaultRange("last-7-days", "/1/users")).toBe("last-7-days");
+ });
+});
diff --git a/client/src/lib/defaultTimeRange.ts b/client/src/lib/defaultTimeRange.ts
index a4ed4fac3..b7368b1a9 100644
--- a/client/src/lib/defaultTimeRange.ts
+++ b/client/src/lib/defaultTimeRange.ts
@@ -182,6 +182,30 @@ export function getDashboardTimeForRange(value: string, timezone: string): Time
}
}
+// Some pages cannot afford an all-time default. The users list aggregates every
+// event a site has ever recorded β€” the count query behind it scanned 5.6 B rows
+// over six days β€” which is an expensive way to answer a question the visitor has
+// not asked yet.
+//
+// This caps the *default* only, and it belongs here rather than on the page:
+// every caller below already skips the stored default when the URL carries a
+// time, and an explicit selection is written to the URL. So picking All time
+// from the selector still works and still survives leaving and returning; what
+// no longer happens is landing on it unasked.
+const DEFAULT_RANGE_CAPS: { appliesTo: (pathname: string) => boolean; cap: DashboardDefaultTimeRange }[] = [
+ { appliesTo: pathname => pathname.endsWith("/users"), cap: "last-30-days" },
+];
+
+export function capDashboardDefaultRange(
+ range: DashboardDefaultTimeRange,
+ pathname: string
+): DashboardDefaultTimeRange {
+ if (range !== "all-time") return range;
+ return DEFAULT_RANGE_CAPS.find(rule => rule.appliesTo(pathname))?.cap ?? range;
+}
+
export function getStoredDashboardDefaultTime(timezone: string): Time {
- return getDashboardTimeForRange(getStoredDashboardDefaultTimeRange(), timezone);
+ const stored = getStoredDashboardDefaultTimeRange();
+ const range = typeof window === "undefined" ? stored : capDashboardDefaultRange(stored, window.location.pathname);
+ return getDashboardTimeForRange(range, timezone);
}
diff --git a/server/src/api/admin/getClickhouseStats.ts b/server/src/api/admin/getClickhouseStats.ts
index d7b0ccd28..fe2bdc37c 100644
--- a/server/src/api/admin/getClickhouseStats.ts
+++ b/server/src/api/admin/getClickhouseStats.ts
@@ -50,7 +50,7 @@ export async function getClickhouseStats(request: FastifyRequest<{ Querystring:
count() as parts_count
FROM system.parts
WHERE database = currentDatabase() AND active = 1
- AND table IN ('events', 'session_replay_events', 'session_replay_metadata')
+ AND table IN ('events', 'session_replay_events', 'session_replay_metadata_v2')
GROUP BY table
ORDER BY total_rows DESC
`,
@@ -79,7 +79,7 @@ export async function getClickhouseStats(request: FastifyRequest<{ Querystring:
GROUP BY date
UNION ALL
SELECT toDate(start_time) as date, 'session_replay_metadata' as table, count() as rows_inserted
- FROM session_replay_metadata
+ FROM session_replay_metadata_v2
${dateFilterStartTime}
GROUP BY date
)
diff --git a/server/src/api/analytics/sessions/getSessions.ts b/server/src/api/analytics/sessions/getSessions.ts
index 50d712ca3..2771a8d65 100644
--- a/server/src/api/analytics/sessions/getSessions.ts
+++ b/server/src/api/analytics/sessions/getSessions.ts
@@ -162,7 +162,7 @@ export const buildSessionsQuery = (query: GetSessionsRequest["Querystring"], sit
),
ReplaySessions AS (
SELECT DISTINCT session_id
- FROM session_replay_metadata
+ FROM session_replay_metadata_v2
FINAL
WHERE site_id = {siteId:Int32}
AND event_count >= 2
diff --git a/server/src/api/analytics/users/deleteUser.ts b/server/src/api/analytics/users/deleteUser.ts
index abca880b6..ca435b03c 100644
--- a/server/src/api/analytics/users/deleteUser.ts
+++ b/server/src/api/analytics/users/deleteUser.ts
@@ -63,7 +63,7 @@ export async function deleteUser(req: FastifyRequest<DeleteUserRequest>, res: Fa
}
await Promise.all(
- ["events", "session_replay_events", "session_replay_metadata"].map(table =>
+ ["events", "session_replay_events", "session_replay_metadata_v2"].map(table =>
clickhouse.command({
query: `DELETE FROM ${table} WHERE ${userCondition}`,
query_params: queryParams,
diff --git a/server/src/api/analytics/users/identifyUser.ts b/server/src/api/analytics/users/identifyUser.ts
index 69496d144..a33cf6deb 100644
--- a/server/src/api/analytics/users/identifyUser.ts
+++ b/server/src/api/analytics/users/identifyUser.ts
@@ -76,9 +76,12 @@ export async function identifyUser(req: FastifyRequest<IdentifyUserRequest>, res
set: { userId: user_id },
});
- // Fire-and-forget: the ClickHouse mutation can take a while on large sites,
- // so don't hold the response on it. No backfill window β€” the operator is
- // explicitly asserting this device's history belongs to this user.
+ // Queued, not immediate: the backfill is batched with every other identity
+ // assigned in the flush interval, so it lands within a few minutes rather
+ // than on this request. It was already fire-and-forget, so nothing in the
+ // response depended on it. No backfill window β€” the operator is explicitly
+ // asserting this device's history belongs to this user, and an unbounded
+ // mutation is exactly the kind worth batching.
backfillIdentifiedUserId(siteId, anonymous_id, user_id, null);
return res.send({ success: true });
diff --git a/server/src/api/sites/getSiteHasData.ts b/server/src/api/sites/getSiteHasData.ts
index 075d42d6c..61c46e546 100644
--- a/server/src/api/sites/getSiteHasData.ts
+++ b/server/src/api/sites/getSiteHasData.ts
@@ -5,20 +5,26 @@ export async function getSiteHasData(request: FastifyRequest<{ Params: { siteId:
const { siteId } = request.params;
try {
- // Check if site has data using original method
- const pageviewsData: { count: number }[] = await clickhouse
+ // `count(*)` reads every row for the site just to answer "any?" β€” 134.8 B
+ // rows across the poll's 289 k calls in one six-day window. LIMIT 1 alone
+ // is close to a wash, because parallel readers each fetch a granule before
+ // the limit stops them; pinning to a single thread is what turns this into
+ // a two-granule read.
+ const rows: { has_data: number }[] = await clickhouse
.query({
- query: `SELECT count(*) as count FROM events WHERE site_id = {siteId:Int32}`,
+ query: `SELECT 1 AS has_data FROM events WHERE site_id = {siteId:Int32} LIMIT 1`,
format: "JSONEachRow",
query_params: {
siteId: Number(siteId),
},
+ clickhouse_settings: {
+ max_threads: 1,
+ },
})
.then(res => res.json());
- const hasData = pageviewsData[0].count > 0;
return {
- hasData,
+ hasData: rows.length > 0,
};
} catch (error) {
request.log.error({ err: error }, "Error checking if site has data");
diff --git a/server/src/db/clickhouse/schema/core.ts b/server/src/db/clickhouse/schema/core.ts
index 3d4cac0be..344ef1bce 100644
--- a/server/src/db/clickhouse/schema/core.ts
+++ b/server/src/db/clickhouse/schema/core.ts
@@ -319,4 +319,79 @@ export async function initializeCoreTables() {
ADD COLUMN IF NOT EXISTS identified_user_id String DEFAULT ''
`
);
+
+ // Successor to session_replay_metadata. The old table is a
+ // ReplacingMergeTree holding one cumulative row per session, so every replay
+ // batch had to re-derive that row: SELECT MIN/MAX/COUNT/SUM over the whole
+ // session, then rewrite it. That read scanned 818 B rows in six days β€” 76% of
+ // everything the cluster read β€” and the rewrites left 2.5 M single-row parts.
+ //
+ // Aggregating the columns instead lets each batch insert only what it
+ // observed, and the engine does the combining. Nothing reads back during
+ // ingest. Every column keeps its name and its post-merge meaning, so readers
+ // only need FINAL, which they already used.
+ //
+ // duration_ms and created_at are deliberately absent: duration is derived
+ // from the merged bounds at read time, and there is no version column to
+ // order by any more.
+ await execClickhouseInitStep(
+ "create session replay metadata v2 table",
+ `
+ CREATE TABLE IF NOT EXISTS session_replay_metadata_v2 (
+ site_id UInt16,
+ session_id String,
+ user_id SimpleAggregateFunction(anyLast, String),
+ -- Rows arrive with '' until the visitor identifies, and max() over a
+ -- String makes any real id win over the empty one.
+ identified_user_id SimpleAggregateFunction(max, String),
+ -- Millisecond resolution, matching session_replay_events.timestamp:
+ -- duration is now derived from these bounds instead of being stored,
+ -- and second-resolution columns would floor a 900ms replay to 0.
+ start_time SimpleAggregateFunction(min, DateTime64(3)),
+ end_time SimpleAggregateFunction(max, Nullable(DateTime64(3))),
+ event_count SimpleAggregateFunction(sum, UInt64),
+ compressed_size_bytes SimpleAggregateFunction(sum, UInt64),
+ -- KNOWN LIMITATION. These merge independently, so a session whose
+ -- batches disagreed can assemble a row from more than one of them β€”
+ -- the old ReplacingMergeTree always returned one whole batch's
+ -- snapshot. Measured over 30 days of production, of the 423 sessions
+ -- that had more than one metadata version: page_url differed in 62,
+ -- region/city/lat in 18, country in 11, language in 13; browser,
+ -- operating_system, device_type, channel, hostname, referrer and
+ -- user_id never differed at all.
+ --
+ -- The geo group is the one that matters, because those fields are only
+ -- meaningful together: ~4% of multi-version sessions could show a city
+ -- and a country drawn from different batches. Making them coherent
+ -- needs one versioned snapshot (a max() over a leading-version tuple,
+ -- or argMaxState + GROUP BY), which changes every read site.
+ --
+ -- Weighed and accepted, 2026-08: the exposure is narrow and the fix
+ -- costs more than the defect. Don't re-open it without new evidence
+ -- that mixed geo is actually misleading someone.
+ page_url SimpleAggregateFunction(anyLast, String),
+ country SimpleAggregateFunction(anyLast, LowCardinality(FixedString(2))),
+ region SimpleAggregateFunction(anyLast, LowCardinality(String)),
+ city SimpleAggregateFunction(anyLast, String),
+ lat SimpleAggregateFunction(anyLast, Float64),
+ lon SimpleAggregateFunction(anyLast, Float64),
+ browser SimpleAggregateFunction(anyLast, LowCardinality(String)),
+ browser_version SimpleAggregateFunction(anyLast, LowCardinality(String)),
+ operating_system SimpleAggregateFunction(anyLast, LowCardinality(String)),
+ operating_system_version SimpleAggregateFunction(anyLast, LowCardinality(String)),
+ language SimpleAggregateFunction(anyLast, LowCardinality(String)),
+ screen_width SimpleAggregateFunction(max, UInt16),
+ screen_height SimpleAggregateFunction(max, UInt16),
+ device_type SimpleAggregateFunction(anyLast, LowCardinality(String)),
+ channel SimpleAggregateFunction(anyLast, String),
+ hostname SimpleAggregateFunction(anyLast, String),
+ referrer SimpleAggregateFunction(anyLast, String),
+ has_replay_data SimpleAggregateFunction(max, UInt8)
+ )
+ ENGINE = AggregatingMergeTree()
+ PARTITION BY toYYYYMM(start_time)
+ ORDER BY (site_id, session_id)
+ TTL toDateTime(start_time) + INTERVAL 30 DAY
+ `
+ );
}
diff --git a/server/src/index.ts b/server/src/index.ts
index 00402dd07..0753bc1a4 100644
--- a/server/src/index.ts
+++ b/server/src/index.ts
@@ -179,6 +179,7 @@ import { createCorsOptionsDelegate, createRejectUntrustedOriginHook } from "./li
import { IS_CLOUD } from "./lib/const.js";
import { logger } from "./lib/logger/logger.js";
import { registerRequestLogging } from "./lib/logger/requestLogging.js";
+import { identityBackfillQueue } from "./services/tracker/identityBackfillQueue.js";
import { reengagementService } from "./services/reengagement/reengagementService.js";
import { telemetryService } from "./services/telemetryService.js";
import { handleIdentify } from "./services/tracker/identifyService.js";
@@ -622,6 +623,10 @@ const shutdown = async (signal: string) => {
await server.close();
server.log.info("Server closed");
+ // Identity backfills are buffered for several minutes to keep mutation
+ // submissions rare; without this, a deploy drops whatever is still pending.
+ await identityBackfillQueue.drainCompletely();
+
// Clear the timeout since we're done
clearTimeout(forceExitTimeout);
diff --git a/server/src/services/replay/replayClockSkew.test.ts b/server/src/services/replay/replayClockSkew.test.ts
new file mode 100644
index 000000000..f12f8b9a1
--- /dev/null
+++ b/server/src/services/replay/replayClockSkew.test.ts
@@ -0,0 +1,89 @@
+import { describe, expect, it } from "vitest";
+import { correctReplayClockSkew, MAX_FUTURE_SKEW_MS, MAX_PAST_SKEW_MS } from "./replayClockSkew.js";
+
+const NOW = Date.UTC(2026, 7, 19, 12, 0, 0);
+const batch = (...offsets: number[]) => offsets.map(timestamp => ({ timestamp, type: "2" }));
+
+describe("correctReplayClockSkew", () => {
+ it("leaves a healthy batch untouched", () => {
+ const events = batch(NOW - 3000, NOW - 2000, NOW - 1000);
+ const result = correctReplayClockSkew(events, NOW);
+ expect(result.skewMs).toBe(0);
+ expect(result.events).toBe(events);
+ });
+
+ it("shifts a batch from a device whose clock is decades ahead", () => {
+ const year2090 = Date.UTC(2090, 6, 2, 21, 31, 45);
+ const events = batch(year2090, year2090 + 1000, year2090 + 5000);
+ const result = correctReplayClockSkew(events, NOW);
+
+ expect(result.skewMs).toBeGreaterThan(0);
+ for (const event of result.events) {
+ expect(event.timestamp).toBeLessThanOrEqual(NOW + MAX_FUTURE_SKEW_MS);
+ expect(event.timestamp).toBeGreaterThanOrEqual(NOW - MAX_PAST_SKEW_MS);
+ }
+ });
+
+ it("preserves the gaps between events, so playback timing survives", () => {
+ const year2076 = Date.UTC(2076, 1, 3, 12, 56, 50);
+ const gaps = [0, 250, 1750, 187_000];
+ const { events } = correctReplayClockSkew(
+ batch(...gaps.map(g => year2076 + g)),
+ NOW
+ );
+
+ const shifted = events.map(e => e.timestamp - events[0].timestamp);
+ expect(shifted).toEqual(gaps);
+ });
+
+ it("is not dragged off by a single corrupt timestamp in a healthy batch", () => {
+ const events = batch(NOW - 3000, NOW - 2000, NOW - 1000, Date.UTC(2064, 0, 5));
+ const { events: corrected, skewMs } = correctReplayClockSkew(events, NOW);
+
+ expect(skewMs).toBe(0);
+ expect(corrected.slice(0, 3).map(e => e.timestamp)).toEqual([NOW - 3000, NOW - 2000, NOW - 1000]);
+ // the outlier alone is clamped back into range
+ expect(corrected[3].timestamp).toBe(NOW + MAX_FUTURE_SKEW_MS);
+ });
+
+ it("pulls forward a batch older than the replay TTL", () => {
+ const ancient = NOW - MAX_PAST_SKEW_MS - 90 * 24 * 60 * 60 * 1000;
+ const { events, skewMs } = correctReplayClockSkew(batch(ancient, ancient + 1000), NOW);
+
+ expect(skewMs).toBeLessThan(0);
+ expect(events[0].timestamp).toBeGreaterThanOrEqual(NOW - MAX_PAST_SKEW_MS);
+ });
+
+ it("handles an empty batch", () => {
+ expect(correctReplayClockSkew([], NOW)).toEqual({ events: [], skewMs: 0 });
+ });
+
+ // Non-finite timestamps land on the bound they were heading for, never NaN.
+ it("clamps a lone Infinity timestamp instead of returning NaN", () => {
+ const { events, skewMs } = correctReplayClockSkew([{ timestamp: Infinity }], NOW);
+
+ expect(events[0].timestamp).toBe(NOW + MAX_FUTURE_SKEW_MS);
+ expect(Number.isFinite(skewMs)).toBe(true);
+ });
+
+ it("clamps a lone -Infinity timestamp", () => {
+ const { events } = correctReplayClockSkew([{ timestamp: -Infinity }], NOW);
+
+ expect(events[0].timestamp).toBe(NOW - MAX_PAST_SKEW_MS);
+ });
+
+ it("does not let one non-finite timestamp shift its healthy neighbours", () => {
+ const { events } = correctReplayClockSkew(
+ [{ timestamp: NOW }, { timestamp: NOW + 100 }, { timestamp: Infinity }, { timestamp: NOW + 300 }],
+ NOW
+ );
+
+ expect(events.map(event => event.timestamp)).toEqual([NOW, NOW + 100, NOW + MAX_FUTURE_SKEW_MS, NOW + 300]);
+ });
+
+ it("falls back to now for a NaN timestamp", () => {
+ const { events } = correctReplayClockSkew([{ timestamp: NaN }], NOW);
+
+ expect(events[0].timestamp).toBe(NOW);
+ });
+});
diff --git a/server/src/services/replay/replayClockSkew.ts b/server/src/services/replay/replayClockSkew.ts
new file mode 100644
index 000000000..bdddd13f6
--- /dev/null
+++ b/server/src/services/replay/replayClockSkew.ts
@@ -0,0 +1,85 @@
+// Replay event timestamps come from the browser, not the server, so they carry
+// whatever the device's clock says. Most are fine; a handful of devices are
+// years out. Those events landed in partitions dated 2032, 2064, 2076 and 2090,
+// which grows the partition list without bound and β€” because the table's TTL is
+// `start_time + 30 DAY` β€” leaves rows that never expire.
+//
+// Individual timestamps cannot simply be pinned to `now`: rrweb reconstructs
+// playback from the gaps between them, so flattening a batch would destroy the
+// recording. The skew is a property of the device, not of any one event, so
+// correct it as one offset applied to the whole batch and keep every delta.
+
+// A day of tolerance absorbs ordinary clock drift and timezone-confused
+// devices without reshaping their batches.
+export const MAX_FUTURE_SKEW_MS = 24 * 60 * 60 * 1000;
+
+// Matches the replay TTL: anything older than this is already unreadable, so
+// there is nothing to preserve by keeping its original date.
+export const MAX_PAST_SKEW_MS = 30 * 24 * 60 * 60 * 1000;
+
+// A browser can put a non-finite value on the wire: JSON `1e400` parses as
+// Infinity and `z.number()` admits it. Left alone it poisons the median and
+// then every corrected timestamp with NaN β€” which ClickHouse cannot encode, so
+// one bad event would fail the whole batch and the recorder would retry it
+// forever. Treat it as the most extreme possible skew and let the clamp below
+// pull it back to a bound, the same as any other unusable clock.
+function sanitize(timestamp: number, nowMs: number, lowerBound: number, upperBound: number): number {
+ if (Number.isFinite(timestamp)) return timestamp;
+ if (timestamp === Infinity) return upperBound;
+ if (timestamp === -Infinity) return lowerBound;
+ // NaN carries no ordering at all, so there is no bound to prefer.
+ return nowMs;
+}
+
+function median(values: number[]): number {
+ const sorted = [...values].sort((a, b) => a - b);
+ const mid = Math.floor(sorted.length / 2);
+ return sorted.length % 2 === 0 ? (sorted[mid - 1] + sorted[mid]) / 2 : sorted[mid];
+}
+
+export type ClockSkewCorrection<T> = {
+ events: T[];
+ /** How far the device's clock was off, in ms. 0 when nothing was corrected. */
+ skewMs: number;
+};
+
+/**
+ * Shifts a batch of client-timestamped events onto the server's clock when the
+ * device is implausibly far off, preserving the interval between every pair of
+ * events.
+ *
+ * The offset is taken from the median rather than the max so that one corrupt
+ * timestamp cannot drag an otherwise healthy batch backwards; a per-event clamp
+ * afterwards catches those stragglers individually.
+ */
+export function correctReplayClockSkew<T extends { timestamp: number }>(
+ events: T[],
+ nowMs: number
+): ClockSkewCorrection<T> {
+ if (events.length === 0) return { events, skewMs: 0 };
+
+ const upperBound = nowMs + MAX_FUTURE_SKEW_MS;
+ const lowerBound = nowMs - MAX_PAST_SKEW_MS;
+
+ const timestamps = events.map(event => sanitize(event.timestamp, nowMs, lowerBound, upperBound));
+ const anchor = median(timestamps);
+
+ let skewMs = 0;
+ if (anchor > upperBound) skewMs = anchor - nowMs;
+ else if (anchor < lowerBound) skewMs = anchor - nowMs;
+
+ // Returning the original array is only safe when nothing needed sanitizing;
+ // otherwise the caller would get the non-finite values straight back.
+ const allFinite = events.every(event => Number.isFinite(event.timestamp));
+ if (skewMs === 0 && allFinite && timestamps.every(t => t <= upperBound && t >= lowerBound)) {
+ return { events, skewMs: 0 };
+ }
+
+ return {
+ events: events.map((event, index) => ({
+ ...event,
+ timestamp: Math.min(Math.max(timestamps[index] - skewMs, lowerBound), upperBound),
+ })),
+ skewMs,
+ };
+}
diff --git a/server/src/services/replay/sessionReplayIngestService.ts b/server/src/services/replay/sessionReplayIngestService.ts
index 9ca52c4b6..167dbbebe 100644
--- a/server/src/services/replay/sessionReplayIngestService.ts
+++ b/server/src/services/replay/sessionReplayIngestService.ts
@@ -1,13 +1,26 @@
import { DateTime } from "luxon";
import { clickhouse } from "../../db/clickhouse/clickhouse.js";
import { RecordSessionReplayRequest } from "../../types/sessionReplay.js";
-import { processResults } from "../../api/analytics/utils/utils.js";
+import { createServiceLogger } from "../../lib/logger/logger.js";
+import { correctReplayClockSkew } from "./replayClockSkew.js";
import { parseTrackingData } from "./trackingUtils.js";
import { sessionsService } from "../sessions/sessionsService.js";
import { userIdService } from "../userId/userIdService.js";
import { r2Storage } from "../storage/r2StorageService.js";
import { siteConfig } from "../../lib/siteConfig.js";
+const logger = createServiceLogger("session-replay-ingest");
+
+/** What a single batch observed about its session β€” no history involved. */
+type BatchStats = {
+ startTime: Date;
+ endTime: Date;
+ eventCount: number;
+ compressedSizeBytes: number;
+ screenWidth: number;
+ screenHeight: number;
+};
+
export interface RequestMetadata {
userAgent: string;
ipAddress: string;
@@ -25,7 +38,14 @@ export class SessionReplayIngestService {
request: RecordSessionReplayRequest,
requestMeta?: RequestMetadata
): Promise<void> {
- const { userId: clientUserId, events, metadata } = request;
+ const { userId: clientUserId, events: rawEvents, metadata } = request;
+
+ // Device clocks, not ours. Correct the whole batch onto server time before
+ // anything downstream partitions or TTLs on these values.
+ const { events, skewMs } = correctReplayClockSkew(rawEvents, Date.now());
+ if (skewMs !== 0) {
+ logger.warn({ siteId, skewMs, eventCount: events.length }, "Corrected replay clock skew");
+ }
// Always generate device fingerprint (anonymous user ID) server-side
const deviceFingerprint = await userIdService.generateUserId(
@@ -121,7 +141,28 @@ export class SessionReplayIngestService {
// Update or insert metadata
if (metadata) {
- await this.updateSessionMetadata(siteId, sessionId, userId, identifiedUserId, metadata, requestMeta);
+ // An events-free batch still carries metadata worth recording, but it has
+ // no timestamps to bound it β€” anchor those on server time rather than
+ // letting Math.min of nothing produce an epoch-dated row.
+ const timestamps = events.length > 0 ? events.map(event => event.timestamp) : [Date.now()];
+ const batchStats: BatchStats = {
+ startTime: new Date(Math.min(...timestamps)),
+ endTime: new Date(Math.max(...timestamps)),
+ eventCount: eventsToInsert.length,
+ compressedSizeBytes: eventsToInsert.reduce((total, event) => total + event.event_size_bytes, 0),
+ screenWidth: metadata.viewportWidth || 0,
+ screenHeight: metadata.viewportHeight || 0,
+ };
+
+ await this.updateSessionMetadata(
+ siteId,
+ sessionId,
+ userId,
+ identifiedUserId,
+ metadata,
+ batchStats,
+ requestMeta
+ );
}
}
@@ -131,42 +172,13 @@ export class SessionReplayIngestService {
userId: string,
identifiedUserId: string,
metadata: any,
+ batchStats: BatchStats,
requestMeta?: RequestMetadata
): Promise<void> {
- // Get existing session info from events table
- const sessionInfo = await clickhouse.query({
- query: `
- SELECT
- MIN(timestamp) as start_time,
- MAX(timestamp) as end_time,
- COUNT() as event_count,
- SUM(event_size_bytes) as compressed_size_bytes,
- MAX(viewport_width) as screen_width,
- MAX(viewport_height) as screen_height
- FROM session_replay_events
- WHERE site_id = {siteId:UInt16}
- AND session_id = {sessionId:String}
- `,
- query_params: { siteId, sessionId },
- format: "JSONEachRow",
- });
-
- type SessionInfoResult = {
- start_time: string;
- end_time: string | null;
- event_count: number;
- compressed_size_bytes: number;
- screen_width: number | null;
- screen_height: number | null;
- };
-
- const sessionResults = await processResults<SessionInfoResult>(sessionInfo);
-
- if (!sessionResults || sessionResults.length === 0) return;
-
- const sessionReplayData = sessionResults[0];
-
- // Parse tracking data from request metadata
+ // Append-only. This used to read the session back first β€” MIN/MAX/COUNT/SUM
+ // across every row the session had ever written, on every batch β€” purely to
+ // recompute a cumulative row. The aggregating table does that combining at
+ // merge time, so a batch now writes only what it saw.
let trackingData: any = {};
if (requestMeta?.userAgent) {
try {
@@ -181,33 +193,26 @@ export class SessionReplayIngestService {
urlObj.search || "", // querystring from URL
hostname,
metadata.language || "", // language from client
- sessionReplayData.screen_width || metadata.viewportWidth || 0,
- sessionReplayData.screen_height || metadata.viewportHeight || 0
+ batchStats.screenWidth || metadata.viewportWidth || 0,
+ batchStats.screenHeight || metadata.viewportHeight || 0
);
} catch (error) {
- console.error("Error parsing tracking data for session replay:", error);
+ logger.error({ siteId, sessionId, err: error }, "Error parsing tracking data for session replay");
}
}
- // Calculate duration
- const startTime = new Date(sessionReplayData.start_time);
- const endTime = sessionReplayData.end_time ? new Date(sessionReplayData.end_time) : null;
- const durationMs = endTime ? endTime.getTime() - startTime.getTime() : null;
-
- // Insert or update metadata
await clickhouse.insert({
- table: "session_replay_metadata",
+ table: "session_replay_metadata_v2",
values: [
{
site_id: siteId,
session_id: sessionId,
user_id: userId,
identified_user_id: identifiedUserId,
- start_time: DateTime.fromJSDate(startTime).toFormat("yyyy-MM-dd HH:mm:ss"),
- end_time: endTime ? DateTime.fromJSDate(endTime).toFormat("yyyy-MM-dd HH:mm:ss") : null,
- duration_ms: durationMs,
- event_count: sessionReplayData.event_count || 0,
- compressed_size_bytes: sessionReplayData.compressed_size_bytes || 0,
+ start_time: DateTime.fromJSDate(batchStats.startTime).toFormat("yyyy-MM-dd HH:mm:ss.SSS"),
+ end_time: DateTime.fromJSDate(batchStats.endTime).toFormat("yyyy-MM-dd HH:mm:ss.SSS"),
+ event_count: batchStats.eventCount,
+ compressed_size_bytes: batchStats.compressedSizeBytes,
page_url: metadata.pageUrl || "",
country: trackingData.country || "",
region: trackingData.region || "",
@@ -219,8 +224,8 @@ export class SessionReplayIngestService {
operating_system: trackingData.operatingSystem || "",
operating_system_version: trackingData.operatingSystemVersion || "",
language: trackingData.language || "",
- screen_width: sessionReplayData.screen_width || metadata?.viewportWidth || 0,
- screen_height: sessionReplayData.screen_height || metadata?.viewportHeight || 0,
+ screen_width: batchStats.screenWidth || metadata?.viewportWidth || 0,
+ screen_height: batchStats.screenHeight || metadata?.viewportHeight || 0,
device_type: trackingData.deviceType || "",
channel: trackingData.channel || "",
hostname: trackingData.hostname || "",
diff --git a/server/src/services/replay/sessionReplayQueryService.ts b/server/src/services/replay/sessionReplayQueryService.ts
index 419bb0fb8..ab5f564ff 100644
--- a/server/src/services/replay/sessionReplayQueryService.ts
+++ b/server/src/services/replay/sessionReplayQueryService.ts
@@ -15,6 +15,40 @@ import { matchesUser } from "../../api/analytics/utils/effectiveUserId.js";
* Service responsible for querying/retrieving session replay data
* Handles listing sessions and getting replay events
*/
+const DURATION_MS = `dateDiff('millisecond', start_time, end_time)`;
+
+// SELECT * is not usable against an aggregating table: it would return the raw
+// per-batch columns and omit the derived duration. List what readers consume.
+const METADATA_COLUMNS = `
+ site_id,
+ session_id,
+ user_id,
+ identified_user_id,
+ start_time,
+ end_time,
+ ${DURATION_MS} AS duration_ms,
+ event_count,
+ compressed_size_bytes,
+ page_url,
+ country,
+ region,
+ city,
+ lat,
+ lon,
+ browser,
+ browser_version,
+ operating_system,
+ operating_system_version,
+ language,
+ screen_width,
+ screen_height,
+ device_type,
+ channel,
+ hostname,
+ referrer,
+ has_replay_data
+`;
+
export class SessionReplayQueryService {
async getSessionReplayList(
siteId: number,
@@ -40,7 +74,10 @@ export class SessionReplayQueryService {
}
if (minDuration !== undefined) {
- whereConditions.push(`duration_ms >= {minDuration:UInt32}`);
+ // Derived from the merged bounds rather than stored: each batch only
+ // knows its own slice of the session, so duration is only meaningful
+ // after FINAL has combined them.
+ whereConditions.push(`${DURATION_MS} >= {minDuration:UInt32}`);
queryParams.minDuration = minDuration * 1000; // Convert seconds to milliseconds
}
@@ -55,7 +92,7 @@ export class SessionReplayQueryService {
if (filterStatement) {
sessionIdsSubquery = `
SELECT DISTINCT srm.session_id
- FROM session_replay_metadata srm
+ FROM session_replay_metadata_v2 srm
FINAL
WHERE srm.site_id = {siteId:UInt16}
AND srm.session_id IN (
@@ -79,7 +116,7 @@ export class SessionReplayQueryService {
identified_user_id,
start_time,
end_time,
- duration_ms,
+ ${DURATION_MS} AS duration_ms,
page_url,
event_count,
country,
@@ -92,7 +129,7 @@ export class SessionReplayQueryService {
device_type,
screen_width,
screen_height
- FROM session_replay_metadata
+ FROM session_replay_metadata_v2
FINAL
WHERE ${whereConditions.join(" AND ")}
AND event_count >= 2
@@ -120,10 +157,10 @@ export class SessionReplayQueryService {
// Get metadata
const metadataResult = await clickhouse.query({
query: `
- SELECT *
- FROM session_replay_metadata
+ SELECT ${METADATA_COLUMNS}
+ FROM session_replay_metadata_v2
FINAL
- WHERE site_id = {siteId:UInt16}
+ WHERE site_id = {siteId:UInt16}
AND session_id = {sessionId:String}
LIMIT 1
`,
@@ -250,8 +287,8 @@ export class SessionReplayQueryService {
async getSessionReplayMetadata(siteId: number, sessionId: string): Promise<SessionReplayMetadata | null> {
const result = await clickhouse.query({
query: `
- SELECT *
- FROM session_replay_metadata
+ SELECT ${METADATA_COLUMNS}
+ FROM session_replay_metadata_v2
FINAL
WHERE site_id = {siteId:UInt16}
AND session_id = {sessionId:String}
@@ -310,7 +347,7 @@ export class SessionReplayQueryService {
await clickhouse.command({
query: `
- DELETE FROM session_replay_metadata
+ DELETE FROM session_replay_metadata_v2
WHERE site_id = {siteId:UInt16}
AND session_id = {sessionId:String}
`,
diff --git a/server/src/services/sites/siteConfigurationLifecycle.ts b/server/src/services/sites/siteConfigurationLifecycle.ts
index 365f043ce..3b82918c2 100644
--- a/server/src/services/sites/siteConfigurationLifecycle.ts
+++ b/server/src/services/sites/siteConfigurationLifecycle.ts
@@ -375,7 +375,7 @@ class SiteConfigurationLifecycle {
query_params: { id: siteId },
}),
clickhouse.command({
- query: "DELETE FROM session_replay_metadata WHERE site_id = {id:UInt32}",
+ query: "DELETE FROM session_replay_metadata_v2 WHERE site_id = {id:UInt32}",
query_params: { id: siteId },
}),
]);
diff --git a/server/src/services/telemetryService.ts b/server/src/services/telemetryService.ts
index 34ad65550..b5a5129ab 100644
--- a/server/src/services/telemetryService.ts
+++ b/server/src/services/telemetryService.ts
@@ -55,7 +55,7 @@ class TelemetryService {
// Get table row counts from ClickHouse
private async getTableCounts() {
- const tables = ["events", "session_replay_events", "session_replay_metadata"];
+ const tables = ["events", "session_replay_events", "session_replay_metadata_v2"];
const counts: Record<string, number> = {};
diff --git a/server/src/services/tracker/identifyService.ts b/server/src/services/tracker/identifyService.ts
index 143a27a8b..492085010 100644
--- a/server/src/services/tracker/identifyService.ts
+++ b/server/src/services/tracker/identifyService.ts
@@ -2,9 +2,9 @@ import { FastifyReply, FastifyRequest } from "fastify";
import { eq, and, sql } from "drizzle-orm";
import { z } from "zod";
import { db } from "../../db/postgres/postgres.js";
-import { clickhouse } from "../../db/clickhouse/clickhouse.js";
import { userProfiles, userAliases } from "../../db/postgres/schema.js";
import { siteConfig } from "../../lib/siteConfig.js";
+import { identityBackfillQueue } from "./identityBackfillQueue.js";
import { userIdService } from "../userId/userIdService.js";
import { resolveClientIp } from "./resolveClientIp.js";
import { createServiceLogger } from "../../lib/logger/logger.js";
@@ -42,33 +42,18 @@ const BACKFILL_DAYS = 30;
// days: null backfills the device's full history β€” only for explicit admin
// actions (dashboard identify), where the operator asserts the whole history
// belongs to this user and the unbounded partition scan is a one-off.
-export async function backfillIdentifiedUserId(
+// Queues the assignment rather than mutating immediately. Each mutation
+// submission takes the MergeTree parts lock, and at production identify rates
+// that lock was being taken every ~12 seconds per table, stalling concurrent
+// inserts and selects. The queue collapses an interval's worth of identities
+// into one mutation per table; see identityBackfillQueue.ts.
+export function backfillIdentifiedUserId(
siteId: number,
anonymousId: string,
userId: string,
days: number | null = BACKFILL_DAYS
) {
- try {
- // session_replay_metadata has no `timestamp` column; its time column is
- // `start_time`. Using `timestamp` there throws ClickHouse error 47
- // (UNKNOWN_IDENTIFIER), so map each table to its actual time column.
- const tables: Array<{ name: string; timeColumn: string }> = [
- { name: "events", timeColumn: "timestamp" },
- { name: "session_replay_events", timeColumn: "timestamp" },
- { name: "session_replay_metadata", timeColumn: "start_time" },
- ];
- for (const { name, timeColumn } of tables) {
- await clickhouse.command({
- query: `ALTER TABLE ${name} UPDATE identified_user_id = {userId: String} WHERE site_id = {siteId: UInt16} AND user_id = {anonymousId: String} AND identified_user_id = ''${
- days !== null ? ` AND ${timeColumn} >= now() - INTERVAL {days: UInt16} DAY` : ""
- }`,
- query_params: { userId, siteId, anonymousId, ...(days !== null ? { days } : {}) },
- });
- }
- logger.info({ siteId, anonymousId, userId }, "Backfilled identified_user_id in ClickHouse");
- } catch (error) {
- logger.error({ siteId, anonymousId, userId, err: error }, "Error backfilling identified_user_id");
- }
+ identityBackfillQueue.enqueue({ siteId, anonymousId, userId }, days);
}
export async function handleIdentify(request: FastifyRequest, reply: FastifyReply) {
diff --git a/server/src/services/tracker/identityBackfillQueue.test.ts b/server/src/services/tracker/identityBackfillQueue.test.ts
new file mode 100644
index 000000000..eac6db09a
--- /dev/null
+++ b/server/src/services/tracker/identityBackfillQueue.test.ts
@@ -0,0 +1,198 @@
+import { beforeEach, describe, expect, it, vi } from "vitest";
+
+const mocks = vi.hoisted(() => ({
+ command: vi.fn(),
+}));
+
+vi.mock("../../db/clickhouse/clickhouse.js", () => ({
+ clickhouse: { command: mocks.command },
+}));
+
+vi.mock("../../lib/logger/logger.js", () => ({
+ createServiceLogger: () => ({ info: vi.fn(), warn: vi.fn(), error: vi.fn() }),
+}));
+
+vi.useFakeTimers();
+const { identityBackfillQueue } = await import("./identityBackfillQueue.js");
+
+const TABLE_COUNT = 3;
+
+/** Every key interpolated into the mutations of the most recent flush. */
+function flushedKeys() {
+ return mocks.command.mock.calls.map(([{ query_params }]) => query_params.keys);
+}
+
+function deferred() {
+ let resolve!: () => void;
+ let reject!: (error: Error) => void;
+ const promise = new Promise<void>((res, rej) => {
+ resolve = res;
+ reject = rej;
+ });
+ return { promise, resolve, reject };
+}
+
+beforeEach(async () => {
+ // Drain anything a previous test left buffered so each starts empty.
+ mocks.command.mockReset();
+ mocks.command.mockResolvedValue(undefined);
+ await identityBackfillQueue.flush();
+ mocks.command.mockReset();
+ mocks.command.mockResolvedValue(undefined);
+});
+
+describe("identityBackfillQueue", () => {
+ it("collapses many identifies into one mutation per table", async () => {
+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "user-a" }, 30);
+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-b", userId: "user-b" }, 30);
+
+ await identityBackfillQueue.flush();
+
+ expect(mocks.command).toHaveBeenCalledTimes(TABLE_COUNT);
+ for (const keys of flushedKeys()) {
+ expect(keys).toEqual(["7:anon-a", "7:anon-b"]);
+ }
+ });
+
+ it("keeps the first assignment when a device identifies twice in a window", async () => {
+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "first" }, 30);
+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "second" }, 30);
+
+ await identityBackfillQueue.flush();
+
+ expect(mocks.command.mock.calls[0][0].query_params.userIds).toEqual(["first"]);
+ });
+
+ it("keeps windows apart so an all-history backfill does not widen the routine ones", async () => {
+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "user-a" }, 30);
+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-b", userId: "user-b" }, null);
+
+ await identityBackfillQueue.flush();
+
+ const windowed = mocks.command.mock.calls.filter(([{ query_params }]) => query_params.days === 30);
+ const unbounded = mocks.command.mock.calls.filter(([{ query_params }]) => query_params.days === undefined);
+
+ expect(windowed).toHaveLength(TABLE_COUNT);
+ expect(unbounded).toHaveLength(TABLE_COUNT);
+ expect(windowed[0][0].query).toContain("INTERVAL {days: UInt16} DAY");
+ expect(unbounded[0][0].query).not.toContain("INTERVAL");
+ });
+
+ it("retries assignments whose mutation failed instead of dropping them", async () => {
+ mocks.command.mockRejectedValueOnce(new Error("ClickHouse timeout"));
+
+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "user-a" }, 30);
+ await identityBackfillQueue.flush();
+ expect(mocks.command).toHaveBeenCalledTimes(TABLE_COUNT);
+
+ mocks.command.mockReset();
+ mocks.command.mockResolvedValue(undefined);
+
+ await identityBackfillQueue.flush();
+
+ expect(mocks.command).toHaveBeenCalledTimes(TABLE_COUNT);
+ expect(mocks.command.mock.calls[0][0].query_params.keys).toEqual(["7:anon-a"]);
+ });
+
+ it("gives up after repeated failures rather than retrying forever", async () => {
+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "user-a" }, 30);
+
+ for (let attempt = 0; attempt < 3; attempt++) {
+ mocks.command.mockReset();
+ mocks.command.mockRejectedValue(new Error("ClickHouse down"));
+ await identityBackfillQueue.flush();
+ expect(mocks.command).toHaveBeenCalledTimes(TABLE_COUNT);
+ }
+
+ mocks.command.mockReset();
+ mocks.command.mockResolvedValue(undefined);
+ await identityBackfillQueue.flush();
+
+ expect(mocks.command).not.toHaveBeenCalled();
+ });
+
+ it("drains identities that arrived while a flush was in progress", async () => {
+ const inFlight = deferred();
+ mocks.command.mockReturnValueOnce(inFlight.promise);
+
+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "user-a" }, 30);
+ const first = identityBackfillQueue.flush();
+
+ // Arrives after the first flush took its snapshot.
+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-b", userId: "user-b" }, 30);
+
+ inFlight.resolve();
+ await first;
+ await identityBackfillQueue.flush();
+
+ const allKeys = flushedKeys().flat();
+ expect(allKeys).toContain("7:anon-a");
+ expect(allKeys).toContain("7:anon-b");
+ });
+
+ it("waits for an in-flight flush so shutdown does not abandon it", async () => {
+ const inFlight = deferred();
+ mocks.command.mockReturnValueOnce(inFlight.promise);
+
+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "user-a" }, 30);
+ const first = identityBackfillQueue.flush();
+
+ let shutdownDone = false;
+ // What server shutdown does: flush() must not resolve while work is in flight.
+ const shutdown = identityBackfillQueue.flush().then(() => {
+ shutdownDone = true;
+ });
+
+ await Promise.resolve();
+ expect(shutdownDone).toBe(false);
+
+ inFlight.resolve();
+ await first;
+ await shutdown;
+
+ expect(shutdownDone).toBe(true);
+ });
+
+ it("splits an oversized backlog across several mutations", async () => {
+ const inFlight = deferred();
+ mocks.command.mockReturnValueOnce(inFlight.promise);
+
+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "held", userId: "u" }, 30);
+ const first = identityBackfillQueue.flush();
+
+ // Everything below arrives while that flush is stuck, so it all lands in
+ // one backlog rather than being spread over separate windows.
+ for (let i = 0; i < 5001; i++) {
+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: `anon-${i}`, userId: `user-${i}` }, 30);
+ }
+
+ inFlight.resolve();
+ await first;
+ // Joins the end of the chain, including the drains the size trigger queued.
+ await identityBackfillQueue.flush();
+
+ const mutations = mocks.command.mock.calls.map(([{ query_params }]) => query_params.keys);
+ expect(mutations.every(keys => keys.length <= 5000)).toBe(true);
+
+ const covered = new Set(mutations.flat());
+ for (let i = 0; i < 5001; i++) {
+ expect(covered.has(`7:anon-${i}`)).toBe(true);
+ }
+ expect(covered.has("7:held")).toBe(true);
+ });
+
+ it("retries a shutdown failure instead of exiting on top of it", async () => {
+ mocks.command.mockRejectedValueOnce(new Error("ClickHouse blip"));
+
+ identityBackfillQueue.enqueue({ siteId: 7, anonymousId: "anon-a", userId: "user-a" }, 30);
+ await identityBackfillQueue.drainCompletely();
+
+ // One failed table, then a clean retry round: nothing is left behind.
+ expect(mocks.command).toHaveBeenCalledTimes(TABLE_COUNT * 2);
+
+ mocks.command.mockReset();
+ mocks.command.mockResolvedValue(undefined);
+ await identityBackfillQueue.flush();
+ expect(mocks.command).not.toHaveBeenCalled();
+ });
+});
diff --git a/server/src/services/tracker/identityBackfillQueue.ts b/server/src/services/tracker/identityBackfillQueue.ts
new file mode 100644
index 000000000..6522bfdd7
--- /dev/null
+++ b/server/src/services/tracker/identityBackfillQueue.ts
@@ -0,0 +1,200 @@
+import { clickhouse } from "../../db/clickhouse/clickhouse.js";
+import { createServiceLogger } from "../../lib/logger/logger.js";
+
+// Identify calls arrive continuously, and each one used to submit three
+// `ALTER TABLE … UPDATE` mutations immediately. Mutation submission takes the
+// MergeTree parts lock, so a steady identify rate turns into a steady stream of
+// lock acquisitions: measured at a 12-second median gap on `events`, which is
+// short enough that inserts and selects are near-permanently contending. The
+// table is not slower on average β€” it periodically freezes, and p90 insert
+// latency runs ~100x baseline inside a submission window.
+//
+// Buffering fixes the frequency rather than the cost. One flush covers every
+// identity assigned in the interval, so N identifies cost one mutation per
+// table instead of N, and the gaps between them are long enough for the lock to
+// stay uncontended in between.
+const FLUSH_INTERVAL_MS = 5 * 60 * 1000;
+
+// A mutation interpolates one array element per identity, so this bounds the
+// statement rather than the buffer: crossing it triggers an early flush, and a
+// flush that finds more than this β€” identifies that arrived while the previous
+// one was in flight β€” splits into several mutations rather than one huge one.
+const MAX_IDENTITIES_PER_MUTATION = 5000;
+
+// A mutation that fails is retried on later flushes. Re-running a partially
+// applied backfill is harmless β€” the `identified_user_id = ''` guard makes the
+// tables that already succeeded a no-op β€” but a permanently failing assignment
+// must not circulate forever.
+const MAX_ATTEMPTS = 3;
+
+type BackfillTable = { name: string; timeColumn: string };
+
+// session_replay_metadata has no `timestamp` column; its time column is
+// `start_time`. Using `timestamp` there throws ClickHouse error 47
+// (UNKNOWN_IDENTIFIER), so map each table to its actual time column.
+const TABLES: BackfillTable[] = [
+ { name: "events", timeColumn: "timestamp" },
+ { name: "session_replay_events", timeColumn: "timestamp" },
+ { name: "session_replay_metadata_v2", timeColumn: "start_time" },
+];
+
+export type IdentityAssignment = {
+ siteId: number;
+ anonymousId: string;
+ userId: string;
+};
+
+type PendingIdentity = IdentityAssignment & { attempts: number };
+
+class IdentityBackfillQueue {
+ // Grouped by backfill window, because the window is part of the mutation's
+ // WHERE clause: folding a `days: null` admin backfill in with the routine
+ // 30-day ones would widen every assignment to a full-history partition scan.
+ private pending = new Map<number | null, Map<string, PendingIdentity>>();
+
+ // Flushes are serialised by chaining rather than by a boolean guard. A guard
+ // would make a concurrent flush() a silent no-op, which loses two things: a
+ // size-triggered flush during a slow one would never run, and shutdown's
+ // `await flush()` would return while work was still in flight.
+ private chain: Promise<void> = Promise.resolve();
+
+ private logger = createServiceLogger("identity-backfill-queue");
+
+ constructor() {
+ setInterval(() => {
+ void this.flush();
+ }, FLUSH_INTERVAL_MS);
+ }
+
+ enqueue(assignment: IdentityAssignment, days: number | null) {
+ this.add({ ...assignment, attempts: 0 }, days);
+
+ const group = this.pending.get(days);
+ if (group && group.size >= MAX_IDENTITIES_PER_MUTATION) {
+ void this.flush();
+ }
+ }
+
+ private add(entry: PendingIdentity, days: number | null) {
+ let group = this.pending.get(days);
+ if (!group) {
+ group = new Map();
+ this.pending.set(days, group);
+ }
+
+ // First assignment wins, matching the un-batched behaviour: the mutation
+ // only touches rows where identified_user_id is still empty, so a second
+ // identify for the same device in the same window used to find nothing left
+ // to update.
+ const key = `${entry.siteId}:${entry.anonymousId}`;
+ if (!group.has(key)) {
+ group.set(key, entry);
+ }
+ }
+
+ /** Resolves once everything queued at call time has been attempted. */
+ flush(): Promise<void> {
+ this.chain = this.chain.then(() => this.drain());
+ return this.chain;
+ }
+
+ /**
+ * Drains repeatedly until nothing is left to retry. Only for shutdown: during
+ * normal operation a failed assignment should wait for the next interval
+ * rather than hot-looping against a ClickHouse that is already struggling,
+ * but at shutdown there is no next interval, so a requeued retry would be
+ * thrown away by process exit.
+ */
+ async drainCompletely(): Promise<void> {
+ for (let round = 0; round < MAX_ATTEMPTS; round++) {
+ await this.flush();
+ if (this.pending.size === 0) return;
+ }
+ }
+
+ private async drain() {
+ const groups = [...this.pending.entries()].filter(([, assignments]) => assignments.size > 0);
+ if (groups.length === 0) return;
+
+ this.pending = new Map();
+
+ for (const [days, assignments] of groups) {
+ const queued = [...assignments.values()];
+ for (let start = 0; start < queued.length; start += MAX_IDENTITIES_PER_MUTATION) {
+ await this.runBackfill(days, queued.slice(start, start + MAX_IDENTITIES_PER_MUTATION));
+ }
+ }
+ }
+
+ private async runBackfill(days: number | null, assignments: PendingIdentity[]) {
+ // Anonymous ids are salted per site, so a bare `user_id IN (…)` would be
+ // correct in practice β€” but it would also let one site's list match another
+ // site's rows. Pairing the two into a single key keeps the mutation exact,
+ // while the separate `site_id IN (…)` keeps the primary-key prefix usable
+ // for pruning.
+ const keys = assignments.map(a => `${a.siteId}:${a.anonymousId}`);
+ const userIds = assignments.map(a => a.userId);
+ const siteIds = [...new Set(assignments.map(a => a.siteId))];
+
+ const failedTables: string[] = [];
+
+ for (const { name, timeColumn } of TABLES) {
+ try {
+ await clickhouse.command({
+ query: `
+ ALTER TABLE ${name}
+ UPDATE identified_user_id = transform(
+ concat(toString(site_id), ':', user_id),
+ {keys: Array(String)},
+ {userIds: Array(String)},
+ identified_user_id
+ )
+ WHERE site_id IN {siteIds: Array(UInt16)}
+ AND concat(toString(site_id), ':', user_id) IN {keys: Array(String)}
+ AND identified_user_id = ''${
+ days !== null ? `\n AND ${timeColumn} >= now() - INTERVAL {days: UInt16} DAY` : ""
+ }
+ `,
+ query_params: { keys, userIds, siteIds, ...(days !== null ? { days } : {}) },
+ });
+ } catch (error) {
+ failedTables.push(name);
+ this.logger.error(
+ { table: name, identities: assignments.length, days, err: error },
+ "Error backfilling identified_user_id"
+ );
+ }
+ }
+
+ if (failedTables.length === 0) {
+ this.logger.info({ identities: assignments.length, days }, "Flushed identity backfill");
+ return;
+ }
+
+ // Requeue rather than drop: the queue is the only thing that knows about
+ // these assignments. The alias row in Postgres already exists, so a later
+ // identify for the same device takes the "alias unchanged" path and never
+ // schedules another backfill β€” a dropped failure leaves those rows
+ // anonymous permanently.
+ const retryable = assignments.filter(a => a.attempts + 1 < MAX_ATTEMPTS);
+ const exhausted = assignments.length - retryable.length;
+
+ for (const assignment of retryable) {
+ this.add({ ...assignment, attempts: assignment.attempts + 1 }, days);
+ }
+
+ this.logger.warn(
+ { failedTables, requeued: retryable.length, dropped: exhausted, days },
+ "Identity backfill partially failed"
+ );
+
+ if (exhausted > 0) {
+ this.logger.error(
+ { dropped: exhausted, days, maxAttempts: MAX_ATTEMPTS },
+ "Giving up on identity backfill; those rows stay anonymous"
+ );
+ }
+ }
+}
+
+export const identityBackfillQueue = new IdentityBackfillQueue();
diff --git a/server/src/services/usageService.ts b/server/src/services/usageService.ts
index 9326749cf..6ec11cc52 100644
--- a/server/src/services/usageService.ts
+++ b/server/src/services/usageService.ts
@@ -238,7 +238,7 @@ class UsageService {
SELECT
site_id,
uniq(session_id) as count
- FROM session_replay_metadata
+ FROM session_replay_metadata_v2
WHERE start_time >= toDate({periodStart:String})
GROUP BY site_id
`,