Skip to content

🐸 Rybbit - open-source and privacy-friendly alternative to Google Analytics that is 10x more intuitive.

fix(clickhouse): pin every time column to UTC and send explicit-UTC instants (#1207)

Goldflagc9b8f5db62fa
diff --git a/docs/content/docs/(docs)/self-hosting-guides/self-hosting-advanced.mdx b/docs/content/docs/(docs)/self-hosting-guides/self-hosting-advanced.mdx
index 90568e398..7f2e21629 100644
--- a/docs/content/docs/(docs)/self-hosting-guides/self-hosting-advanced.mdx
+++ b/docs/content/docs/(docs)/self-hosting-guides/self-hosting-advanced.mdx
@@ -87,3 +87,42 @@ IMAGE_TAG=latest
HOST_BACKEND_PORT="3001:3001"
HOST_CLIENT_PORT="3002:3002"
```
+
+## ClickHouse Timezone
+
+Rybbit stores every timestamp as a UTC instant and converts to your dashboard timezone when it queries. Every time column is declared `DateTime('UTC')`, and the backend writes timestamps with an explicit UTC offset, so the ClickHouse server's own `<timezone>` setting does not change what gets stored. Leave it at the default (UTC) unless you have a reason not to; the backend logs a warning at startup if it is anything else.
+
+### Upgrading from 2.9.1 or earlier on a non-UTC ClickHouse
+
+Before this change the backend wrote timezone-less strings, which ClickHouse read in the server's timezone. If your ClickHouse was configured for anything but UTC, two things need attention after upgrading:
+
+1. **Month partitions.** Existing rows keep the month partition computed in the old timezone. The backend converts the columns to UTC on startup without rewriting data, and afterwards rows that fall within the old offset of a month boundary (for Europe/Berlin, the last one or two hours of each month) sit in a partition ClickHouse no longer expects. A query whose whole time window falls inside those hours will miss them. To fix it, stop the backend (`docker compose stop backend`) and rebuild each partitioned table with a copy and swap. For `events`:
+
+ ```sql
+ -- How many rows are affected (0 means there is nothing to do):
+ SELECT count() FROM events WHERE _partition_id != toString(toYYYYMM(timestamp, 'UTC'));
+
+ CREATE TABLE events_rebuilt AS events;
+ INSERT INTO events_rebuilt SELECT * FROM events;
+ -- Check both tables have the same count(), then:
+ EXCHANGE TABLES events AND events_rebuilt;
+ DROP TABLE events_rebuilt; -- once you are satisfied
+ ```
+
+ Repeat for `bot_events` and `bot_observations` (partitioned on `timestamp`), and `session_replay_events` (`timestamp`), `session_replay_metadata` and `session_replay_metadata_v2` (`start_time`). With `LITE_DASHBOARD=true`, do the same for `sessions_mv_target` (`start_time`) and the `*_hourly_mv_target` tables (`event_hour`); `session_hourly_mv_target` is rebuilt by its view every hour and can be skipped. On `ReplacingMergeTree` and `AggregatingMergeTree` tables compare `count()` with `FINAL`, since background merges can change the raw count. Materialized views stay attached to the swapped-in table. The copy rewrites every row, so run it off-peak. Step 2's copy also re-partitions every row it writes, so a table you fix there can skip this step. Start the backend when you are done, unless you continue with step 2.
+
+2. **Shifted timestamps.** Keep the backend stopped for this step too: any event that arrives between the copy and the swap ends up only in the table you drop. Rows written by rybbit-backend 2.9.0 or 2.9.1 while ClickHouse was not in UTC were stored shifted by the server's offset (the dashboard looked frozen at the upgrade time). Rows written by 2.8.0 or earlier were shifted only if the backend container's `TZ` differed from the ClickHouse timezone. Rybbit cannot correct these automatically: a shifted row and a correct row can carry the same timestamp, so only you know which rows the old backend wrote. If you want them fixed, use the same copy-and-swap on the `events` table with your own bounds, then rebuild the rollups from it:
+
+ ```sql
+ CREATE TABLE events_fixed AS events;
+ INSERT INTO events_fixed
+ SELECT * REPLACE (
+ -- Re-read the stored wall-clock, rendered in the old server timezone, as UTC.
+ if(<written by the old backend>, toDateTime(toString(toTimeZone(timestamp, 'Europe/Berlin')), 'UTC'), timestamp) AS timestamp,
+ if(<written by the old backend>, toDateTime64(toString(toTimeZone(timestamp_ms, 'Europe/Berlin')), 3, 'UTC'), timestamp_ms) AS timestamp_ms
+ ) FROM events;
+ -- Check counts and a few known rows, then:
+ EXCHANGE TABLES events AND events_fixed;
+ ```
+
+ Replace `<written by the old backend>` with a predicate that is true only for rows the old backend wrote, for example a `timestamp` range that ends before the upgraded backend started, minus the server offset. Run it once: applying it twice shifts rows twice. Do it before importing historical data from another tool. Afterwards, if you run with `LITE_DASHBOARD=true`, rebuild its rollups with `docker compose run --rm --no-deps backend node dist/scripts/backfillLiteDashboardMVs.js --truncate` (the rollup tables only exist with that setting). Drop `events_fixed`, and only then start the backend. The same applies to `bot_events`, `bot_observations` and the two `session_replay_metadata` tables; replay event timestamps were always written as epoch numbers and are correct as they are.
diff --git a/server/src/api/admin/getAdminOrganizations.ts b/server/src/api/admin/getAdminOrganizations.ts
index 75a92be14..181742027 100644
--- a/server/src/api/admin/getAdminOrganizations.ts
+++ b/server/src/api/admin/getAdminOrganizations.ts
@@ -104,7 +104,9 @@ export async function getAdminOrganizations(request: FastifyRequest, reply: Fast
.where(allOrgIds.length > 0 ? inArray(sites.organizationId, allOrgIds) : undefined);
// Get event counts for the past 24 hours and 30 days from ClickHouse
- const now = DateTime.now();
+ // Bounds are rendered and parsed in UTC so neither Node's nor ClickHouse's
+ // timezone shifts the window.
+ const now = DateTime.utc();
const yesterday = now.minus({ hours: 24 });
const thirtyDaysAgo = now.minus({ days: 30 });
@@ -121,8 +123,8 @@ export async function getAdminOrganizations(request: FastifyRequest, reply: Fast
FROM
hourly_events_by_site_mv_target
WHERE
- event_hour >= toDateTime('${yesterday.toFormat("yyyy-MM-dd HH:mm:ss")}') AND
- event_hour <= toDateTime('${now.toFormat("yyyy-MM-dd HH:mm:ss")}')
+ event_hour >= toDateTime('${yesterday.toFormat("yyyy-MM-dd HH:mm:ss")}', 'UTC') AND
+ event_hour <= toDateTime('${now.toFormat("yyyy-MM-dd HH:mm:ss")}', 'UTC')
GROUP BY
site_id
`,
@@ -145,8 +147,8 @@ export async function getAdminOrganizations(request: FastifyRequest, reply: Fast
FROM
hourly_events_by_site_mv_target
WHERE
- event_hour >= toDateTime('${thirtyDaysAgo.toFormat("yyyy-MM-dd HH:mm:ss")}') AND
- event_hour <= toDateTime('${now.toFormat("yyyy-MM-dd HH:mm:ss")}')
+ event_hour >= toDateTime('${thirtyDaysAgo.toFormat("yyyy-MM-dd HH:mm:ss")}', 'UTC') AND
+ event_hour <= toDateTime('${now.toFormat("yyyy-MM-dd HH:mm:ss")}', 'UTC')
GROUP BY
site_id
`,
diff --git a/server/src/api/admin/getAdminSites.ts b/server/src/api/admin/getAdminSites.ts
index d3ac36e56..6c9e750fa 100644
--- a/server/src/api/admin/getAdminSites.ts
+++ b/server/src/api/admin/getAdminSites.ts
@@ -39,7 +39,9 @@ export async function getAdminSites(request: FastifyRequest, reply: FastifyReply
const orgSubscriptionMap = await getOrganizationSubscriptions(organizationsData, false);
// Get event counts for the past 24 hours and 30 days from Clickhouse
- const now = DateTime.now();
+ // Bounds are rendered and parsed in UTC so neither Node's nor ClickHouse's
+ // timezone shifts the window.
+ const now = DateTime.utc();
const yesterday = now.minus({ hours: 24 });
const thirtyDaysAgo = now.minus({ days: 30 });
@@ -52,8 +54,8 @@ export async function getAdminSites(request: FastifyRequest, reply: FastifyReply
FROM
hourly_events_by_site_mv_target
WHERE
- event_hour >= toDateTime('${yesterday.toFormat("yyyy-MM-dd HH:mm:ss")}') AND
- event_hour <= toDateTime('${now.toFormat("yyyy-MM-dd HH:mm:ss")}')
+ event_hour >= toDateTime('${yesterday.toFormat("yyyy-MM-dd HH:mm:ss")}', 'UTC') AND
+ event_hour <= toDateTime('${now.toFormat("yyyy-MM-dd HH:mm:ss")}', 'UTC')
GROUP BY
site_id
`,
@@ -69,8 +71,8 @@ export async function getAdminSites(request: FastifyRequest, reply: FastifyReply
FROM
hourly_events_by_site_mv_target
WHERE
- event_hour >= toDateTime('${thirtyDaysAgo.toFormat("yyyy-MM-dd HH:mm:ss")}') AND
- event_hour <= toDateTime('${now.toFormat("yyyy-MM-dd HH:mm:ss")}')
+ event_hour >= toDateTime('${thirtyDaysAgo.toFormat("yyyy-MM-dd HH:mm:ss")}', 'UTC') AND
+ event_hour <= toDateTime('${now.toFormat("yyyy-MM-dd HH:mm:ss")}', 'UTC')
GROUP BY
site_id
`,
diff --git a/server/src/api/analytics/events/getEvents.ts b/server/src/api/analytics/events/getEvents.ts
index f314f95d9..11e9e7acc 100644
--- a/server/src/api/analytics/events/getEvents.ts
+++ b/server/src/api/analytics/events/getEvents.ts
@@ -92,7 +92,7 @@ export const buildEventsQuery = (query: GetEventsRequest["Querystring"], siteId:
WHERE
site_id = {siteId:Int32}
${EVENT_TYPE_FILTER}
- AND timestamp > toDateTime64({sinceTimestamp:String}, 3)
+ AND timestamp > toDateTime64({sinceTimestamp:String}, 3, 'UTC')
${filterStatement}
ORDER BY timestamp DESC
LIMIT 500
@@ -114,7 +114,9 @@ export const buildEventsQuery = (query: GetEventsRequest["Querystring"], siteId:
};
if (before_timestamp) {
- cursorCondition = `AND timestamp < toDateTime64({beforeTimestamp:String}, 3)`;
+ // The API renders timestamps in UTC and the client echoes them back;
+ // parse in UTC too, not in the server timezone.
+ cursorCondition = `AND timestamp < toDateTime64({beforeTimestamp:String}, 3, 'UTC')`;
queryParams.beforeTimestamp = before_timestamp;
}
diff --git a/server/src/db/clickhouse/clickhouse.test.ts b/server/src/db/clickhouse/clickhouse.test.ts
index b8b4b248c..2b9b8af78 100644
--- a/server/src/db/clickhouse/clickhouse.test.ts
+++ b/server/src/db/clickhouse/clickhouse.test.ts
@@ -2,27 +2,33 @@ import { beforeEach, describe, expect, it, vi } from "vitest";
const state = vi.hoisted(() => ({
existingSessionHourlyView: undefined as string | undefined,
+ existingColumns: {} as Record<string, { name: string; type: string }[]>,
+ timezone: "UTC",
}));
const mocks = vi.hoisted(() => ({
exec: vi.fn(),
+ insert: vi.fn(),
query: vi.fn(),
logger: {
debug: vi.fn(),
error: vi.fn(),
info: vi.fn(),
+ warn: vi.fn(),
},
}));
vi.mock("@clickhouse/client", () => ({
createClient: () => ({
exec: mocks.exec,
+ insert: mocks.insert,
query: mocks.query,
}),
}));
+// Both flags on so every CREATE TABLE the server can issue is under test.
vi.mock("../../lib/const.js", () => ({
- IS_CLOUD: false,
+ IS_CLOUD: true,
LITE_DASHBOARD: true,
}));
@@ -44,21 +50,28 @@ describe("session hourly materialized view initialization", () => {
beforeEach(() => {
vi.clearAllMocks();
state.existingSessionHourlyView = undefined;
+ state.existingColumns = {};
+ state.timezone = "UTC";
mocks.exec.mockResolvedValue(undefined);
- mocks.query.mockImplementation(async ({ query }: { query: string }) => {
- if (query.includes("FROM system.columns")) {
- return { json: async () => [] };
+ mocks.query.mockImplementation(
+ async ({ query, query_params }: { query: string; query_params?: { table?: string } }) => {
+ if (query.includes("FROM system.columns")) {
+ return { json: async () => state.existingColumns[query_params?.table ?? ""] ?? [] };
+ }
+ if (query.includes("timezone()")) {
+ return { json: async () => [{ timezone: state.timezone }] };
+ }
+
+ if (query.includes("FROM system.tables")) {
+ return {
+ json: async () =>
+ state.existingSessionHourlyView ? [{ create_table_query: state.existingSessionHourlyView }] : [],
+ };
+ }
+
+ throw new Error(`Unexpected ClickHouse query: ${query}`);
}
-
- if (query.includes("FROM system.tables")) {
- return {
- json: async () =>
- state.existingSessionHourlyView ? [{ create_table_query: state.existingSessionHourlyView }] : [],
- };
- }
-
- throw new Error(`Unexpected ClickHouse query: ${query}`);
- });
+ );
});
it("creates the hourly rollup from finalized sessions instead of raw events", async () => {
@@ -79,8 +92,8 @@ describe("session hourly materialized view initialization", () => {
const createEvents = queries.find(query => /CREATE TABLE IF NOT EXISTS events\b/.test(query));
const alterEvents = queries.find(query => /ALTER TABLE events\b/.test(query));
- expect(createEvents).toContain("timestamp_ms DateTime64(3)");
- expect(alterEvents).toContain("timestamp_ms DateTime64(3) DEFAULT toDateTime64(timestamp, 3)");
+ expect(createEvents).toContain("timestamp_ms DateTime64(3, 'UTC')");
+ expect(alterEvents).toContain("timestamp_ms DateTime64(3, 'UTC') DEFAULT toDateTime64(timestamp, 3)");
});
it("replaces the old five-minute raw-events definition", async () => {
@@ -115,3 +128,117 @@ describe("session hourly materialized view initialization", () => {
expect(executedQueries()).not.toContainEqual(expect.stringMatching(/DROP VIEW/));
});
});
+
+// A bare `DateTime` column takes the server's timezone, so on a self-hosted
+// ClickHouse configured for anything but UTC every stored instant would read
+// back shifted. Every time column has to carry an explicit 'UTC', and tables
+// created before that was declared have to be migrated on startup.
+describe("UTC time columns", () => {
+ beforeEach(() => {
+ vi.clearAllMocks();
+ state.existingSessionHourlyView = undefined;
+ state.existingColumns = {};
+ state.timezone = "UTC";
+ mocks.exec.mockResolvedValue(undefined);
+ mocks.query.mockImplementation(
+ async ({ query, query_params }: { query: string; query_params?: { table?: string } }) => {
+ if (query.includes("FROM system.columns")) {
+ return { json: async () => state.existingColumns[query_params?.table ?? ""] ?? [] };
+ }
+ if (query.includes("timezone()")) {
+ return { json: async () => [{ timezone: state.timezone }] };
+ }
+ if (query.includes("FROM system.tables")) {
+ return { json: async () => [] };
+ }
+ throw new Error(`Unexpected ClickHouse query: ${query}`);
+ }
+ );
+ });
+
+ it("declares every DateTime column in the DDL with an explicit UTC timezone", async () => {
+ await initializeClickhouse();
+
+ const ddl = executedQueries().filter(query => /CREATE TABLE|ALTER TABLE/.test(query));
+ expect(ddl.length).toBeGreaterThan(0);
+
+ // Matches type tokens only: `toDateTime(...)` / `toDateTime64(...)` calls
+ // have no word boundary before the `D`, so they are left alone.
+ const timeTypes = ddl.flatMap(query => query.match(/\bDateTime(?:64)?(?:\([^()]*\))?/g) ?? []);
+ expect(timeTypes.length).toBeGreaterThan(0);
+ const bare = timeTypes.filter(type => !type.includes("'UTC'"));
+ expect(bare).toEqual([]);
+ });
+
+ it("pins existing bare time columns to UTC with a metadata-only ALTER", async () => {
+ state.existingColumns = {
+ events: [
+ { name: "timestamp", type: "DateTime" },
+ { name: "timestamp_ms", type: "DateTime64(3)" },
+ ],
+ session_replay_metadata_v2: [
+ { name: "start_time", type: "SimpleAggregateFunction(min, DateTime64(3))" },
+ { name: "end_time", type: "SimpleAggregateFunction(max, Nullable(DateTime64(3)))" },
+ ],
+ sessions_mv_target: [{ name: "start_time", type: "SimpleAggregateFunction(min, DateTime)" }],
+ };
+
+ await initializeClickhouse();
+
+ const modifies = executedQueries().filter(query => /MODIFY COLUMN/.test(query));
+ expect(modifies.find(query => /ALTER TABLE events\b/.test(query))).toContain(
+ "MODIFY COLUMN timestamp DateTime('UTC'),\n MODIFY COLUMN timestamp_ms DateTime64(3, 'UTC')"
+ );
+ expect(modifies.find(query => /ALTER TABLE session_replay_metadata_v2\b/.test(query))).toContain(
+ "MODIFY COLUMN end_time SimpleAggregateFunction(max, Nullable(DateTime64(3, 'UTC')))"
+ );
+ expect(modifies.find(query => /ALTER TABLE sessions_mv_target\b/.test(query))).toContain(
+ "MODIFY COLUMN start_time SimpleAggregateFunction(min, DateTime('UTC'))"
+ );
+ expect(modifies.some(query => /ALTER TABLE bot_events\b/.test(query))).toBe(false);
+ });
+
+ it("leaves columns that already carry UTC untouched", async () => {
+ state.existingColumns = {
+ events: [
+ { name: "timestamp", type: "DateTime('UTC')" },
+ { name: "timestamp_ms", type: "DateTime64(3, 'UTC')" },
+ ],
+ overview_hourly_mv_target: [{ name: "event_hour", type: "DateTime('UTC')" }],
+ };
+
+ await initializeClickhouse();
+
+ expect(executedQueries().some(query => /MODIFY COLUMN/.test(query))).toBe(false);
+ });
+
+ it("stays quiet about the server timezone when it is UTC", async () => {
+ await initializeClickhouse();
+
+ expect(mocks.logger.warn).not.toHaveBeenCalled();
+ });
+
+ it("warns about a non-UTC server", async () => {
+ state.timezone = "Europe/Berlin";
+
+ await initializeClickhouse();
+
+ expect(mocks.logger.warn).toHaveBeenCalledWith(
+ expect.objectContaining({ timezone: "Europe/Berlin" }),
+ expect.stringContaining("not UTC")
+ );
+ });
+
+ it("keeps starting when the timezone migration fails", async () => {
+ state.existingColumns = { events: [{ name: "timestamp", type: "DateTime" }] };
+ mocks.exec.mockImplementation(async ({ query }: { query: string }) => {
+ if (/MODIFY COLUMN/.test(query)) throw new Error("ALTER refused");
+ });
+
+ await expect(initializeClickhouse()).resolves.toBeUndefined();
+ expect(mocks.logger.error).toHaveBeenCalledWith(
+ expect.objectContaining({ step: "pin events time columns to UTC" }),
+ expect.any(String)
+ );
+ });
+});
diff --git a/server/src/db/clickhouse/clickhouse.ts b/server/src/db/clickhouse/clickhouse.ts
index 1f10e9e83..f2f6b7c2e 100644
--- a/server/src/db/clickhouse/clickhouse.ts
+++ b/server/src/db/clickhouse/clickhouse.ts
@@ -1,4 +1,6 @@
import { IS_CLOUD, LITE_DASHBOARD } from "../../lib/const.js";
+import { clickhouse } from "./client.js";
+import { clickhouseInitLogger } from "./initUtils.js";
import { initializeCloudTables } from "./schema/cloud.js";
import { initializeCoreTables } from "./schema/core.js";
import { initializeLiteDashboardMVs } from "./schema/liteDashboard.js";
@@ -7,6 +9,20 @@ import { provisionQueryUser } from "./queryUser.js";
export { clickhouse, clickhouseQuery } from "./client.js";
export const initializeClickhouse = async () => {
+ // Every column is declared in UTC and every writer sends explicit-UTC
+ // instants, so the server timezone no longer changes what is stored. It
+ // still shows up in ad-hoc SQL, and installs upgraded from a non-UTC server
+ // have month partitions to rebuild, so say so once at boot.
+ const result = await clickhouse.query({ query: "SELECT timezone() AS timezone", format: "JSONEachRow" });
+ const timezone = (await result.json<{ timezone: string }>())[0]?.timezone ?? "UTC";
+ if (timezone !== "UTC") {
+ clickhouseInitLogger.warn(
+ { timezone },
+ "ClickHouse server timezone is not UTC; Rybbit stores and queries UTC explicitly, but set <timezone>UTC</timezone> " +
+ "to keep ad-hoc SQL unambiguous. If this install ran before its columns were pinned to UTC, see docs: self-hosting-advanced β†’ ClickHouse timezone"
+ );
+ }
+
await initializeCoreTables();
if (IS_CLOUD) {
diff --git a/server/src/db/clickhouse/client.test.ts b/server/src/db/clickhouse/client.test.ts
new file mode 100644
index 000000000..fbcbe29bf
--- /dev/null
+++ b/server/src/db/clickhouse/client.test.ts
@@ -0,0 +1,19 @@
+import { describe, expect, it, vi } from "vitest";
+
+const mocks = vi.hoisted(() => ({ createClient: vi.fn((_options: unknown) => ({})) }));
+
+vi.mock("@clickhouse/client", () => ({ createClient: mocks.createClient }));
+
+// Writers serialise instants as ISO-8601 with an explicit `Z` (dateTime.ts).
+// ClickHouse's default `basic` parser rejects the offset, so the write client
+// has to ask for best_effort or every insert fails with CANNOT_PARSE_INPUT.
+describe("clickhouse write client", () => {
+ it("parses inserted timestamps with best_effort", async () => {
+ await import("./client.js");
+
+ const writeClientOptions = mocks.createClient.mock.calls[0][0] as {
+ clickhouse_settings?: Record<string, unknown>;
+ };
+ expect(writeClientOptions.clickhouse_settings).toMatchObject({ date_time_input_format: "best_effort" });
+ });
+});
diff --git a/server/src/db/clickhouse/client.ts b/server/src/db/clickhouse/client.ts
index 7307c7c8a..00ad1faf1 100644
--- a/server/src/db/clickhouse/client.ts
+++ b/server/src/db/clickhouse/client.ts
@@ -8,6 +8,13 @@ export const clickhouse = createClient({
database: process.env.CLICKHOUSE_DB,
password: process.env.CLICKHOUSE_PASSWORD,
request_timeout: CLICKHOUSE_REQUEST_TIMEOUT_MS,
+ clickhouse_settings: {
+ // Writers serialise every instant as ISO-8601 with an explicit `Z` (see
+ // dateTime.ts). The default `basic` parser rejects the offset; best_effort
+ // honours it, so the stored epoch no longer depends on the server timezone.
+ // Only affects parsing of inserted text, not date functions in queries.
+ date_time_input_format: "best_effort",
+ },
});
// Least-privilege connection for user-authored SQL (custom query page and
diff --git a/server/src/db/clickhouse/dateTime.test.ts b/server/src/db/clickhouse/dateTime.test.ts
new file mode 100644
index 000000000..c2969e6cd
--- /dev/null
+++ b/server/src/db/clickhouse/dateTime.test.ts
@@ -0,0 +1,52 @@
+import { DateTime, Settings } from "luxon";
+import { afterEach, describe, expect, it } from "vitest";
+import { isClickHouseDateTime, toClickHouseDateTime } from "./dateTime.js";
+
+const originalZone = Settings.defaultZone;
+
+describe("toClickHouseDateTime", () => {
+ afterEach(() => {
+ Settings.defaultZone = originalZone;
+ });
+
+ it("renders every input kind as an explicit-UTC ISO string", () => {
+ expect(toClickHouseDateTime("2026-09-15T12:57:22.789Z")).toBe("2026-09-15T12:57:22.789Z");
+ expect(toClickHouseDateTime("2026-09-15T14:57:22.789+02:00")).toBe("2026-09-15T12:57:22.789Z");
+ expect(toClickHouseDateTime("2026-09-15T12:57:22")).toBe("2026-09-15T12:57:22.000Z");
+ expect(toClickHouseDateTime("2026-09-15 12:57:22")).toBe("2026-09-15T12:57:22.000Z");
+ expect(toClickHouseDateTime(1_700_000_000_123)).toBe("2023-11-14T22:13:20.123Z");
+ expect(toClickHouseDateTime(new Date(Date.UTC(2026, 0, 15, 7, 57, 22)))).toBe("2026-01-15T07:57:22.000Z");
+ expect(toClickHouseDateTime(DateTime.fromISO("2026-01-15T07:57:22-05:00", { setZone: true }))).toBe(
+ "2026-01-15T12:57:22.000Z"
+ );
+ });
+
+ it("ignores the process timezone", () => {
+ for (const zone of ["Europe/Berlin", "Asia/Shanghai", "America/New_York"]) {
+ Settings.defaultZone = zone;
+ expect(toClickHouseDateTime("2026-09-15T12:57:22.789Z")).toBe("2026-09-15T12:57:22.789Z");
+ expect(toClickHouseDateTime("2026-09-15 12:57:22")).toBe("2026-09-15T12:57:22.000Z");
+ expect(toClickHouseDateTime(new Date("2026-09-15T12:57:22.789Z"))).toBe("2026-09-15T12:57:22.789Z");
+ }
+ });
+
+ it("keeps the repeated daylight-saving hour distinct", () => {
+ Settings.defaultZone = "Europe/Berlin";
+ expect(toClickHouseDateTime("2026-10-25T02:30:00.999+02:00")).toBe("2026-10-25T00:30:00.999Z");
+ expect(toClickHouseDateTime("2026-10-25T02:30:00.999+01:00")).toBe("2026-10-25T01:30:00.999Z");
+ });
+
+ it("tells a real instant from a well-shaped impossible one", () => {
+ expect(isClickHouseDateTime("2024-02-29 12:00:00")).toBe(true);
+ expect(isClickHouseDateTime("2024-02-31 12:00:00")).toBe(false);
+ expect(isClickHouseDateTime("2024-02-31T12:00:00Z")).toBe(false);
+ expect(isClickHouseDateTime("2024-06-15T25:00:00Z")).toBe(false);
+ });
+
+ it("rejects values ClickHouse could not parse", () => {
+ expect(() => toClickHouseDateTime("not-a-date")).toThrow(RangeError);
+ expect(() => toClickHouseDateTime("")).toThrow(RangeError);
+ expect(() => toClickHouseDateTime(NaN)).toThrow(RangeError);
+ expect(() => toClickHouseDateTime(new Date("nope"))).toThrow(RangeError);
+ });
+});
diff --git a/server/src/db/clickhouse/dateTime.ts b/server/src/db/clickhouse/dateTime.ts
new file mode 100644
index 000000000..ab4ce740a
--- /dev/null
+++ b/server/src/db/clickhouse/dateTime.ts
@@ -0,0 +1,46 @@
+import { DateTime } from "luxon";
+
+export type ClickHouseDateTimeInput = DateTime | Date | number | string;
+
+/**
+ * Serialises an instant for a ClickHouse `DateTime` / `DateTime64` column.
+ *
+ * ClickHouse reads a timezone-less `YYYY-MM-DD hh:mm:ss` string in the
+ * column's timezone, and a bare `DateTime` column takes whatever timezone the
+ * server is configured with. Rybbit's query layer assumes every stored value
+ * is a UTC instant, so on a self-hosted server set to Europe/Berlin a naive
+ * string landed two hours off. Every table Rybbit creates now declares its
+ * time columns as `'UTC'`, but writers still never send a naive string: an
+ * explicit `Z` pins the instant whatever the server or column timezone is.
+ *
+ * ClickHouse only accepts the ISO offset under
+ * `date_time_input_format=best_effort`, which the write client sets globally.
+ * A `DateTime` column truncates the milliseconds; `DateTime64(3)` keeps them.
+ *
+ * Accepts a Luxon DateTime, a JS Date, epoch milliseconds, an ISO-8601 string
+ * (with or without an offset β€” none means UTC), or a ClickHouse-style
+ * `YYYY-MM-DD hh:mm:ss` string, which is read as UTC. Throws on anything else
+ * rather than let ClickHouse reject the whole batch with a parse error.
+ */
+export function toClickHouseDateTime(input: ClickHouseDateTimeInput): string {
+ const instant = toInstant(input);
+ if (!instant.isValid) {
+ throw new RangeError(`Cannot serialise ${JSON.stringify(input)} as a ClickHouse DateTime`);
+ }
+ return instant.toUTC().toISO() as string;
+}
+
+/** Whether toClickHouseDateTime would accept the value: a real instant, not just the right shape. */
+export function isClickHouseDateTime(input: ClickHouseDateTimeInput): boolean {
+ return toInstant(input).isValid;
+}
+
+function toInstant(input: ClickHouseDateTimeInput): DateTime {
+ if (DateTime.isDateTime(input)) return input;
+ if (input instanceof Date) return DateTime.fromJSDate(input);
+ if (typeof input === "number") return DateTime.fromMillis(input);
+
+ const iso = DateTime.fromISO(input, { zone: "utc", setZone: true });
+ if (iso.isValid) return iso;
+ return DateTime.fromSQL(input, { zone: "utc" });
+}
diff --git a/server/src/db/clickhouse/initUtils.ts b/server/src/db/clickhouse/initUtils.ts
index 06e3c6015..f3b317d5a 100644
--- a/server/src/db/clickhouse/initUtils.ts
+++ b/server/src/db/clickhouse/initUtils.ts
@@ -1,5 +1,6 @@
import { createServiceLogger } from "../../lib/logger/logger.js";
import { CLICKHOUSE_REQUEST_TIMEOUT_MS, clickhouse } from "./client.js";
+import type { TimeColumnDefinition } from "./timeColumns.js";
export const clickhouseInitLogger = createServiceLogger("clickhouse");
@@ -26,10 +27,11 @@ export async function execClickhouseInitStep(
}
}
+/** Existing columns of a table, keyed by name, valued by ClickHouse type text. */
export async function getTableColumns(table: string) {
const result = await clickhouse.query({
query: `
- SELECT name
+ SELECT name, type
FROM system.columns
WHERE database = currentDatabase()
AND table = {table:String}
@@ -38,8 +40,56 @@ export async function getTableColumns(table: string) {
format: "JSONEachRow",
});
- const rows = await result.json<{ name: string }>();
- return new Set(rows.map(row => row.name));
+ const rows = await result.json<{ name: string; type: string }>();
+ return new Map(rows.map(row => [row.name, row.type]));
+}
+
+/**
+ * Pins the timezone of a table's existing time columns to UTC.
+ *
+ * A bare `DateTime` column takes the server's configured timezone, so a
+ * self-hosted ClickHouse set to Europe/Berlin read every stored epoch two hours
+ * off from what the UTC-explicit query layer assumes. The stored value is an
+ * epoch whichever timezone the type carries, so `MODIFY COLUMN` between
+ * timezones is metadata-only: no part is rewritten, and ClickHouse allows it
+ * on partition-key, sorting-key and SimpleAggregateFunction columns alike
+ * (verified on 26.3). Columns that already carry `'UTC'` are left alone, and a
+ * column the table does not have yet is skipped β€” the CREATE / ADD COLUMN
+ * statements declare it in UTC.
+ *
+ * On a server whose timezone is not UTC, existing month partitions keep the
+ * names they got in that timezone; the self-hosting docs cover rebuilding them.
+ *
+ * Optional because writers send explicit-UTC instants regardless: a failed
+ * ALTER leaves the pre-fix behaviour for reads on a non-UTC server, which is
+ * not worth refusing to start over.
+ */
+export async function ensureUtcTimeColumns(table: string, columns: TimeColumnDefinition[]) {
+ const existingColumns = await getTableColumns(table);
+ const outdatedColumns = columns.filter(column => {
+ const currentType = existingColumns.get(column.name);
+ return currentType !== undefined && !currentType.includes("'UTC'");
+ });
+
+ if (outdatedColumns.length === 0) {
+ clickhouseInitLogger.debug({ table }, "Time columns are already pinned to UTC");
+ return;
+ }
+
+ clickhouseInitLogger.info(
+ { table, columns: outdatedColumns.map(column => column.name) },
+ "Pinning time columns to UTC"
+ );
+
+ await execClickhouseInitStep(
+ `pin ${table} time columns to UTC`,
+ `
+ ALTER TABLE ${table}
+ ${outdatedColumns.map(column => `MODIFY COLUMN ${column.name} ${column.type}`).join(",\n ")}
+ `,
+ { optional: true, lockAcquireTimeoutSeconds: 15 }
+ );
+
}
export async function getTableCreateQuery(table: string) {
diff --git a/server/src/db/clickhouse/schema/cloud.ts b/server/src/db/clickhouse/schema/cloud.ts
index 3208c9d80..0444718bf 100644
--- a/server/src/db/clickhouse/schema/cloud.ts
+++ b/server/src/db/clickhouse/schema/cloud.ts
@@ -1,4 +1,5 @@
-import { execClickhouseInitStep } from "../initUtils.js";
+import { ensureUtcTimeColumns, execClickhouseInitStep } from "../initUtils.js";
+import { UTC_TIME_COLUMNS } from "../timeColumns.js";
// Hourly per-site event counts, used by cloud usage tracking / billing.
export async function initializeCloudTables() {
@@ -6,7 +7,7 @@ export async function initializeCloudTables() {
"create hourly events by site target table",
`
CREATE TABLE IF NOT EXISTS hourly_events_by_site_mv_target (
- event_hour DateTime, -- The specific hour
+ event_hour DateTime('UTC'), -- The specific hour
site_id UInt16,
event_count UInt64 -- The count of events for that site in that hour
)
@@ -16,6 +17,7 @@ export async function initializeCloudTables() {
TTL event_hour + INTERVAL 60 DAY
`
);
+ await ensureUtcTimeColumns("hourly_events_by_site_mv_target", UTC_TIME_COLUMNS.hourly_events_by_site_mv_target);
await execClickhouseInitStep(
"create hourly events by site materialized view",
diff --git a/server/src/db/clickhouse/schema/core.ts b/server/src/db/clickhouse/schema/core.ts
index 35c7401c4..dc4cd6c47 100644
--- a/server/src/db/clickhouse/schema/core.ts
+++ b/server/src/db/clickhouse/schema/core.ts
@@ -1,4 +1,10 @@
-import { clickhouseInitLogger as logger, execClickhouseInitStep, getTableColumns } from "../initUtils.js";
+import {
+ clickhouseInitLogger as logger,
+ ensureUtcTimeColumns,
+ execClickhouseInitStep,
+ getTableColumns,
+} from "../initUtils.js";
+import { UTC_TIME_COLUMNS } from "../timeColumns.js";
type ColumnDefinition = {
name: string;
@@ -9,7 +15,10 @@ const EVENTS_COLUMNS_TO_ENSURE: ColumnDefinition[] = [
// `timestamp` is part of the MergeTree key and ClickHouse forbids widening a
// key column in place. Keep it for partition pruning; use this additive
// column wherever event order matters. Old rows read as whole-second values.
- { name: "timestamp_ms", definition: "timestamp_ms DateTime64(3) DEFAULT toDateTime64(timestamp, 3) AFTER timestamp" },
+ {
+ name: "timestamp_ms",
+ definition: "timestamp_ms DateTime64(3, 'UTC') DEFAULT toDateTime64(timestamp, 3) AFTER timestamp",
+ },
{ name: "lcp", definition: "lcp Nullable(Float64)" },
{ name: "cls", definition: "cls Nullable(Float64)" },
{ name: "inp", definition: "inp Nullable(Float64)" },
@@ -101,8 +110,8 @@ export async function initializeCoreTables() {
`
CREATE TABLE IF NOT EXISTS events (
site_id UInt16,
- timestamp DateTime,
- timestamp_ms DateTime64(3) DEFAULT toDateTime64(timestamp, 3),
+ timestamp DateTime('UTC'),
+ timestamp_ms DateTime64(3, 'UTC') DEFAULT toDateTime64(timestamp, 3),
session_id String,
user_id String,
hostname String,
@@ -153,13 +162,14 @@ export async function initializeCoreTables() {
// TABLE above was missing these columns). Only columns that are actually
// missing are altered in β€” even a no-op ALTER needs the events table ALTER lock.
await ensureEventsColumns();
+ await ensureUtcTimeColumns("events", UTC_TIME_COLUMNS.events);
await execClickhouseInitStep(
"create bot events table",
`
CREATE TABLE IF NOT EXISTS bot_events (
site_id UInt16,
- timestamp DateTime,
+ timestamp DateTime('UTC'),
session_id String,
user_id String,
hostname String,
@@ -205,6 +215,7 @@ export async function initializeCoreTables() {
);
await ensureBotEventsColumns("bot_events");
+ await ensureUtcTimeColumns("bot_events", UTC_TIME_COLUMNS.bot_events);
// Forensic mirror of bot_events for sites with bot blocking turned OFF: the
// event is still tracked normally, and this row is the only trace that
@@ -215,7 +226,7 @@ export async function initializeCoreTables() {
`
CREATE TABLE IF NOT EXISTS bot_observations (
site_id UInt16,
- timestamp DateTime,
+ timestamp DateTime('UTC'),
session_id String,
user_id String,
hostname String,
@@ -261,6 +272,7 @@ export async function initializeCoreTables() {
);
await ensureBotEventsColumns("bot_observations");
+ await ensureUtcTimeColumns("bot_observations", UTC_TIME_COLUMNS.bot_observations);
await execClickhouseInitStep(
"create session replay events table",
@@ -269,7 +281,7 @@ export async function initializeCoreTables() {
site_id UInt16,
session_id String,
user_id String,
- timestamp DateTime64(3),
+ timestamp DateTime64(3, 'UTC'),
event_type LowCardinality(String),
event_data String,
event_data_key Nullable(String), -- R2 storage key for cloud deployments
@@ -296,6 +308,7 @@ export async function initializeCoreTables() {
ADD COLUMN IF NOT EXISTS identified_user_id String DEFAULT ''
`
);
+ await ensureUtcTimeColumns("session_replay_events", UTC_TIME_COLUMNS.session_replay_events);
await execClickhouseInitStep(
"create session replay metadata table",
@@ -304,8 +317,8 @@ export async function initializeCoreTables() {
site_id UInt16,
session_id String,
user_id String,
- start_time DateTime,
- end_time Nullable(DateTime),
+ start_time DateTime('UTC'),
+ end_time Nullable(DateTime('UTC')),
duration_ms Nullable(UInt32),
event_count UInt32,
compressed_size_bytes UInt32,
@@ -327,7 +340,7 @@ export async function initializeCoreTables() {
hostname String,
referrer String,
has_replay_data UInt8 DEFAULT 1,
- created_at DateTime DEFAULT now()
+ created_at DateTime('UTC') DEFAULT now()
)
ENGINE = ReplacingMergeTree(created_at)
PARTITION BY toYYYYMM(start_time)
@@ -343,6 +356,7 @@ export async function initializeCoreTables() {
ADD COLUMN IF NOT EXISTS identified_user_id String DEFAULT ''
`
);
+ await ensureUtcTimeColumns("session_replay_metadata", UTC_TIME_COLUMNS.session_replay_metadata);
// Successor to session_replay_metadata. The old table is a
// ReplacingMergeTree holding one cumulative row per session, so every replay
@@ -371,8 +385,8 @@ export async function initializeCoreTables() {
-- Millisecond resolution, matching session_replay_events.timestamp:
-- duration is now derived from these bounds instead of being stored,
-- and second-resolution columns would floor a 900ms replay to 0.
- start_time SimpleAggregateFunction(min, DateTime64(3)),
- end_time SimpleAggregateFunction(max, Nullable(DateTime64(3))),
+ start_time SimpleAggregateFunction(min, DateTime64(3, 'UTC')),
+ end_time SimpleAggregateFunction(max, Nullable(DateTime64(3, 'UTC'))),
event_count SimpleAggregateFunction(sum, UInt64),
compressed_size_bytes SimpleAggregateFunction(sum, UInt64),
-- KNOWN LIMITATION. These merge independently, so a session whose
@@ -418,4 +432,5 @@ export async function initializeCoreTables() {
TTL toDateTime(start_time) + INTERVAL 30 DAY
`
);
+ await ensureUtcTimeColumns("session_replay_metadata_v2", UTC_TIME_COLUMNS.session_replay_metadata_v2);
}
diff --git a/server/src/db/clickhouse/schema/liteDashboard.ts b/server/src/db/clickhouse/schema/liteDashboard.ts
index ef86f74bb..b2985fe38 100644
--- a/server/src/db/clickhouse/schema/liteDashboard.ts
+++ b/server/src/db/clickhouse/schema/liteDashboard.ts
@@ -1,4 +1,10 @@
-import { clickhouseInitLogger as logger, execClickhouseInitStep, getTableCreateQuery } from "../initUtils.js";
+import {
+ clickhouseInitLogger as logger,
+ ensureUtcTimeColumns,
+ execClickhouseInitStep,
+ getTableCreateQuery,
+} from "../initUtils.js";
+import { UTC_TIME_COLUMNS } from "../timeColumns.js";
const SESSION_HOURLY_MV_NAME = "session_hourly_mv";
const SESSION_HOURLY_MV_REFRESH_INTERVAL = "REFRESH EVERY 1 HOUR";
@@ -80,8 +86,8 @@ export async function initializeLiteDashboardMVs() {
site_id UInt16,
session_id String,
user_id String,
- start_time SimpleAggregateFunction(min, DateTime),
- end_time SimpleAggregateFunction(max, DateTime),
+ start_time SimpleAggregateFunction(min, DateTime('UTC')),
+ end_time SimpleAggregateFunction(max, DateTime('UTC')),
pageviews SimpleAggregateFunction(sum, UInt64),
events SimpleAggregateFunction(sum, UInt64),
country LowCardinality(FixedString(2)),
@@ -90,13 +96,14 @@ export async function initializeLiteDashboardMVs() {
browser LowCardinality(String),
operating_system LowCardinality(String),
hostname String,
- last_seen SimpleAggregateFunction(max, DateTime)
+ last_seen SimpleAggregateFunction(max, DateTime('UTC'))
)
ENGINE = AggregatingMergeTree()
PARTITION BY toYYYYMM(start_time)
ORDER BY (site_id, session_id)
`
);
+ await ensureUtcTimeColumns("sessions_mv_target", UTC_TIME_COLUMNS.sessions_mv_target);
await execClickhouseInitStep(
"create sessions rollup materialized view",
@@ -129,7 +136,7 @@ export async function initializeLiteDashboardMVs() {
`
CREATE TABLE IF NOT EXISTS overview_hourly_mv_target (
site_id UInt16,
- event_hour DateTime,
+ event_hour DateTime('UTC'),
pageviews SimpleAggregateFunction(sum, UInt64),
events SimpleAggregateFunction(sum, UInt64),
users AggregateFunction(uniq, String),
@@ -140,6 +147,7 @@ export async function initializeLiteDashboardMVs() {
ORDER BY (site_id, event_hour)
`
);
+ await ensureUtcTimeColumns("overview_hourly_mv_target", UTC_TIME_COLUMNS.overview_hourly_mv_target);
await execClickhouseInitStep(
"create hourly overview rollup materialized view",
@@ -165,7 +173,7 @@ export async function initializeLiteDashboardMVs() {
`
CREATE TABLE IF NOT EXISTS pathname_hourly_mv_target (
site_id UInt16,
- event_hour DateTime,
+ event_hour DateTime('UTC'),
pathname String,
hostname String,
pageviews SimpleAggregateFunction(sum, UInt64),
@@ -177,6 +185,7 @@ export async function initializeLiteDashboardMVs() {
ORDER BY (site_id, event_hour, pathname)
`
);
+ await ensureUtcTimeColumns("pathname_hourly_mv_target", UTC_TIME_COLUMNS.pathname_hourly_mv_target);
await execClickhouseInitStep(
"create hourly pathname rollup materialized view",
@@ -203,7 +212,7 @@ export async function initializeLiteDashboardMVs() {
`
CREATE TABLE IF NOT EXISTS country_hourly_mv_target (
site_id UInt16,
- event_hour DateTime,
+ event_hour DateTime('UTC'),
country LowCardinality(FixedString(2)),
region LowCardinality(String),
pageviews SimpleAggregateFunction(sum, UInt64),
@@ -215,6 +224,7 @@ export async function initializeLiteDashboardMVs() {
ORDER BY (site_id, event_hour, country, region)
`
);
+ await ensureUtcTimeColumns("country_hourly_mv_target", UTC_TIME_COLUMNS.country_hourly_mv_target);
await execClickhouseInitStep(
"create hourly country rollup materialized view",
@@ -240,7 +250,7 @@ export async function initializeLiteDashboardMVs() {
`
CREATE TABLE IF NOT EXISTS device_type_hourly_mv_target (
site_id UInt16,
- event_hour DateTime,
+ event_hour DateTime('UTC'),
device_type LowCardinality(String),
pageviews SimpleAggregateFunction(sum, UInt64),
users AggregateFunction(uniq, String),
@@ -251,6 +261,7 @@ export async function initializeLiteDashboardMVs() {
ORDER BY (site_id, event_hour, device_type)
`
);
+ await ensureUtcTimeColumns("device_type_hourly_mv_target", UTC_TIME_COLUMNS.device_type_hourly_mv_target);
await execClickhouseInitStep(
"create hourly device type rollup materialized view",
@@ -279,7 +290,7 @@ export async function initializeLiteDashboardMVs() {
`
CREATE TABLE IF NOT EXISTS session_hourly_mv_target (
site_id UInt16,
- session_hour DateTime,
+ session_hour DateTime('UTC'),
sessions UInt64,
pageviews UInt64,
users AggregateFunction(uniq, String),
@@ -291,6 +302,7 @@ export async function initializeLiteDashboardMVs() {
ORDER BY (site_id, session_hour)
`
);
+ await ensureUtcTimeColumns("session_hourly_mv_target", UTC_TIME_COLUMNS.session_hourly_mv_target);
// CREATE IF NOT EXISTS cannot update an already-installed view. Inspect the
// stored definition so deployments replace the old five-minute raw-events
diff --git a/server/src/db/clickhouse/timeColumns.ts b/server/src/db/clickhouse/timeColumns.ts
new file mode 100644
index 000000000..5c2105d2f
--- /dev/null
+++ b/server/src/db/clickhouse/timeColumns.ts
@@ -0,0 +1,45 @@
+/**
+ * Every DateTime / DateTime64 column Rybbit stores in ClickHouse, with the
+ * UTC-pinned type the CREATE statements declare. The startup migration
+ * (ensureUtcTimeColumns) uses this to heal tables created before the timezone
+ * was explicit. Types must match the CREATE statements in schema/*.ts exactly.
+ */
+export type TimeColumnDefinition = {
+ name: string;
+ /** The full target type, e.g. `DateTime('UTC')` or `SimpleAggregateFunction(min, DateTime64(3, 'UTC'))`. */
+ type: string;
+};
+
+const EVENT_HOUR: TimeColumnDefinition[] = [{ name: "event_hour", type: "DateTime('UTC')" }];
+
+export const UTC_TIME_COLUMNS = {
+ events: [
+ { name: "timestamp", type: "DateTime('UTC')" },
+ { name: "timestamp_ms", type: "DateTime64(3, 'UTC')" },
+ ],
+ bot_events: [{ name: "timestamp", type: "DateTime('UTC')" }],
+ bot_observations: [{ name: "timestamp", type: "DateTime('UTC')" }],
+ session_replay_events: [{ name: "timestamp", type: "DateTime64(3, 'UTC')" }],
+ session_replay_metadata: [
+ { name: "start_time", type: "DateTime('UTC')" },
+ { name: "end_time", type: "Nullable(DateTime('UTC'))" },
+ { name: "created_at", type: "DateTime('UTC')" },
+ ],
+ session_replay_metadata_v2: [
+ { name: "start_time", type: "SimpleAggregateFunction(min, DateTime64(3, 'UTC'))" },
+ { name: "end_time", type: "SimpleAggregateFunction(max, Nullable(DateTime64(3, 'UTC')))" },
+ ],
+ // Cloud only.
+ hourly_events_by_site_mv_target: EVENT_HOUR,
+ // LITE_DASHBOARD only.
+ sessions_mv_target: [
+ { name: "start_time", type: "SimpleAggregateFunction(min, DateTime('UTC'))" },
+ { name: "end_time", type: "SimpleAggregateFunction(max, DateTime('UTC'))" },
+ { name: "last_seen", type: "SimpleAggregateFunction(max, DateTime('UTC'))" },
+ ],
+ overview_hourly_mv_target: EVENT_HOUR,
+ pathname_hourly_mv_target: EVENT_HOUR,
+ country_hourly_mv_target: EVENT_HOUR,
+ device_type_hourly_mv_target: EVENT_HOUR,
+ session_hourly_mv_target: [{ name: "session_hour", type: "DateTime('UTC')" }],
+} satisfies Record<string, TimeColumnDefinition[]>;
diff --git a/server/src/scripts/backfillLiteDashboardMVs.ts b/server/src/scripts/backfillLiteDashboardMVs.ts
index 930c9301b..be9bc5d78 100644
--- a/server/src/scripts/backfillLiteDashboardMVs.ts
+++ b/server/src/scripts/backfillLiteDashboardMVs.ts
@@ -214,7 +214,7 @@ async function main() {
console.log(`\n=== ${table} β†’ ${target} ===`);
for await (const month of monthRanges(from, to)) {
- const where = `WHERE timestamp >= toDateTime('${month.start}') AND timestamp < toDateTime('${month.end}')`;
+ const where = `WHERE timestamp >= toDateTime('${month.start}', 'UTC') AND timestamp < toDateTime('${month.end}', 'UTC')`;
const select = buildSelectQuery(table, where);
const insertSql = `INSERT INTO ${target}\n${select}`;
diff --git a/server/src/services/import/importQuotaTracker.test.ts b/server/src/services/import/importQuotaTracker.test.ts
index bc766cac9..22eb21c52 100644
--- a/server/src/services/import/importQuotaTracker.test.ts
+++ b/server/src/services/import/importQuotaTracker.test.ts
@@ -62,7 +62,11 @@ describe("ImportQuotaTracker", () => {
describe("canImportBatch", () => {
it("should allow all timestamps when under the monthly limit", () => {
const tracker = makeTracker({}, 10, "202312");
- const result = tracker.canImportBatch(["2024-06-01 00:00:00", "2024-06-10 08:15:30", "2024-06-14 23:59:59"]);
+ const result = tracker.canImportBatch([
+ "2024-06-01T00:00:00.000Z",
+ "2024-06-10T08:15:30.000Z",
+ "2024-06-14T23:59:59.000Z",
+ ]);
expect(result).toEqual([0, 1, 2]);
});
@@ -74,10 +78,10 @@ describe("ImportQuotaTracker", () => {
it("should account for existing usage in a month", () => {
const tracker = makeTracker({ "202405": 3 }, 5, "202312");
const result = tracker.canImportBatch([
- "2024-05-01 10:00:00",
- "2024-05-02 10:00:00",
- "2024-05-03 10:00:00",
- "2024-05-04 10:00:00",
+ "2024-05-01T10:00:00.000Z",
+ "2024-05-02T10:00:00.000Z",
+ "2024-05-03T10:00:00.000Z",
+ "2024-05-04T10:00:00.000Z",
]);
// Only 2 slots remain in May (3 used out of 5).
expect(result).toEqual([0, 1]);
@@ -86,52 +90,52 @@ describe("ImportQuotaTracker", () => {
it("should track quota per month independently in a batch spanning months", () => {
const tracker = makeTracker({}, 2, "202312");
const result = tracker.canImportBatch([
- "2024-05-01 10:00:00",
- "2024-06-01 10:00:00",
- "2024-05-02 10:00:00",
- "2024-06-02 10:00:00",
- "2024-05-03 10:00:00", // 3rd May event exceeds May's limit
+ "2024-05-01T10:00:00.000Z",
+ "2024-06-01T10:00:00.000Z",
+ "2024-05-02T10:00:00.000Z",
+ "2024-06-02T10:00:00.000Z",
+ "2024-05-03T10:00:00.000Z", // 3rd May event exceeds May's limit
]);
expect(result).toEqual([0, 1, 2, 3]);
});
it("should reject events for a month exactly at the limit", () => {
const tracker = makeTracker({ "202406": 5 }, 5, "202312");
- expect(tracker.canImportBatch(["2024-06-01 10:00:00"])).toEqual([]);
+ expect(tracker.canImportBatch(["2024-06-01T10:00:00.000Z"])).toEqual([]);
});
it("should allow exactly one more event when one below the limit", () => {
const tracker = makeTracker({ "202406": 4 }, 5, "202312");
- expect(tracker.canImportBatch(["2024-06-01 10:00:00", "2024-06-02 10:00:00"])).toEqual([0]);
+ expect(tracker.canImportBatch(["2024-06-01T10:00:00.000Z", "2024-06-02T10:00:00.000Z"])).toEqual([0]);
});
it("should accumulate increments across successive calls", () => {
const tracker = makeTracker({}, 3, "202312");
- expect(tracker.canImportBatch(["2024-06-01 10:00:00", "2024-06-02 10:00:00"])).toEqual([0, 1]);
- expect(tracker.canImportBatch(["2024-06-03 10:00:00", "2024-06-04 10:00:00"])).toEqual([0]);
- expect(tracker.canImportBatch(["2024-06-05 10:00:00"])).toEqual([]);
+ expect(tracker.canImportBatch(["2024-06-01T10:00:00.000Z", "2024-06-02T10:00:00.000Z"])).toEqual([0, 1]);
+ expect(tracker.canImportBatch(["2024-06-03T10:00:00.000Z", "2024-06-04T10:00:00.000Z"])).toEqual([0]);
+ expect(tracker.canImportBatch(["2024-06-05T10:00:00.000Z"])).toEqual([]);
});
it("should reject future timestamps", () => {
const tracker = makeTracker({}, 10, "202312");
- expect(tracker.canImportBatch(["2024-06-15 12:00:01", "2025-01-01 00:00:00"])).toEqual([]);
+ expect(tracker.canImportBatch(["2024-06-15T12:00:01.000Z", "2025-01-01T00:00:00.000Z"])).toEqual([]);
});
it("should allow a timestamp exactly at the current time", () => {
const tracker = makeTracker({}, 10, "202312");
// dt > now is false when equal, so it is not treated as future.
- expect(tracker.canImportBatch(["2024-06-15 12:00:00"])).toEqual([0]);
+ expect(tracker.canImportBatch(["2024-06-15T12:00:00.000Z"])).toEqual([0]);
});
it("should reject timestamps older than the allowed window", () => {
const tracker = makeTracker({}, 10, "202312");
- expect(tracker.canImportBatch(["2023-11-30 23:59:59", "2020-01-01 00:00:00"])).toEqual([]);
+ expect(tracker.canImportBatch(["2023-11-30T23:59:59.000Z", "2020-01-01T00:00:00.000Z"])).toEqual([]);
});
it("should allow timestamps exactly in the oldest allowed month", () => {
const tracker = makeTracker({}, 10, "202312");
// "202312" < "202312" is false, so the boundary month is allowed.
- expect(tracker.canImportBatch(["2023-12-01 00:00:00"])).toEqual([0]);
+ expect(tracker.canImportBatch(["2023-12-01T00:00:00.000Z"])).toEqual([0]);
});
it("should reject invalid timestamp strings", () => {
@@ -139,9 +143,9 @@ describe("ImportQuotaTracker", () => {
expect(
tracker.canImportBatch([
"not-a-date",
- "2024-06-15T10:00:00", // ISO "T" separator does not match the expected format
+ "2024-06-15 10:00:00", // legacy naive form; mappers emit explicit-UTC ISO now
"",
- "2024-13-01 10:00:00", // invalid month
+ "2024-13-01T10:00:00.000Z", // invalid month
])
).toEqual([]);
});
@@ -149,16 +153,16 @@ describe("ImportQuotaTracker", () => {
it("should not consume quota for rejected timestamps", () => {
const tracker = makeTracker({}, 1, "202312");
const result = tracker.canImportBatch([
- "2025-01-01 00:00:00", // future, rejected
+ "2025-01-01T00:00:00.000Z", // future, rejected
"bad-timestamp", // invalid, rejected
- "2024-06-01 10:00:00", // valid, takes the single slot
+ "2024-06-01T10:00:00.000Z", // valid, takes the single slot
]);
expect(result).toEqual([2]);
});
it("should keep valid events interleaved with rejected ones", () => {
const tracker = makeTracker({}, 10, "202312");
- const result = tracker.canImportBatch(["2024-06-01 10:00:00", "garbage", "2024-06-02 10:00:00"]);
+ const result = tracker.canImportBatch(["2024-06-01T10:00:00.000Z", "garbage", "2024-06-02T10:00:00.000Z"]);
expect(result).toEqual([0, 2]);
});
@@ -166,14 +170,16 @@ describe("ImportQuotaTracker", () => {
const tracker = makeTracker({}, Infinity, "190001");
// Self-hosted: quota accounting is disabled, but timestamp validation
// applies exactly like the cloud path.
- expect(tracker.canImportBatch(["not-a-date", "2099-01-01 00:00:00", "2024-06-01 10:00:00"])).toEqual([2]);
+ expect(tracker.canImportBatch(["not-a-date", "2099-01-01T00:00:00.000Z", "2024-06-01T10:00:00.000Z"])).toEqual([
+ 2,
+ ]);
});
it("should ignore the historical window when the limit is Infinity", () => {
// The window is tier/quota-derived, so it is disabled for self-hosted
// even if oldestAllowedMonth would otherwise reject the timestamp.
const tracker = makeTracker({}, Infinity, "202401");
- expect(tracker.canImportBatch(["2020-01-01 00:00:00"])).toEqual([0]);
+ expect(tracker.canImportBatch(["2020-01-01T00:00:00.000Z"])).toEqual([0]);
});
});
@@ -225,9 +231,9 @@ describe("ImportQuotaTracker", () => {
source: "free",
eventLimit: 2,
});
- expect(tracker.canImportBatch(["2024-06-01 10:00:00", "2024-06-02 10:00:00", "2024-06-03 10:00:00"])).toEqual([
- 0, 1,
- ]);
+ expect(
+ tracker.canImportBatch(["2024-06-01T10:00:00.000Z", "2024-06-02T10:00:00.000Z", "2024-06-03T10:00:00.000Z"])
+ ).toEqual([0, 1]);
});
it("should throw when the organization is not found", async () => {
diff --git a/server/src/services/import/importQuotaTracker.ts b/server/src/services/import/importQuotaTracker.ts
index a508b43da..026df4854 100644
--- a/server/src/services/import/importQuotaTracker.ts
+++ b/server/src/services/import/importQuotaTracker.ts
@@ -167,7 +167,8 @@ export class ImportQuotaTracker {
for (let i = 0; i < timestamps.length; i++) {
const timestamp = timestamps[i];
- const dt = DateTime.fromFormat(timestamp, "yyyy-MM-dd HH:mm:ss", { zone: "utc" });
+ // Mappers emit the explicit-UTC ISO form that ClickHouse is given.
+ const dt = DateTime.fromISO(timestamp, { zone: "utc", setZone: true });
if (!dt.isValid || dt > now) {
continue;
diff --git a/server/src/services/import/mappers/plausible.test.ts b/server/src/services/import/mappers/plausible.test.ts
index 67fa5ea48..dafba3c49 100644
--- a/server/src/services/import/mappers/plausible.test.ts
+++ b/server/src/services/import/mappers/plausible.test.ts
@@ -37,7 +37,7 @@ describe("PlausibleImportMapper", () => {
expect(result).toHaveLength(1);
expect(result[0]).toMatchObject({
site_id: 1,
- timestamp: "2024-06-15 14:30:00",
+ timestamp: "2024-06-15T14:30:00.000Z",
session_id: TEST_SESSION_ID,
user_id: TEST_USER_ID,
hostname: "example.com",
@@ -81,6 +81,16 @@ describe("PlausibleImportMapper", () => {
expect(result[0].props).toEqual({ plan: "pro" });
});
+ it("drops a well-shaped impossible date without losing the rest of the batch", () => {
+ const events = [
+ makeEvent({ timestamp: "2024-06-15 14:30:00" }),
+ makeEvent({ timestamp: "2024-02-31 12:00:00" }),
+ makeEvent({ timestamp: "2024-06-16 09:00:00" }),
+ ];
+ const result = PlausibleImportMapper.transform(events, 1, "import-1");
+ expect(result.map(row => row.timestamp)).toEqual(["2024-06-15T14:30:00.000Z", "2024-06-16T09:00:00.000Z"]);
+ });
+
it("should skip events with invalid timestamps", () => {
const events = [makeEvent({ timestamp: "not-a-date" })];
const result = PlausibleImportMapper.transform(events, 1, "import-1");
@@ -115,11 +125,7 @@ describe("PlausibleImportMapper", () => {
];
const result = PlausibleImportMapper.transform(events, 1, "import-1");
expect(result).toHaveLength(3);
- expect(result.map((e) => e.pathname)).toEqual([
- "/page-1",
- "/page-2",
- "/page-3",
- ]);
+ expect(result.map(e => e.pathname)).toEqual(["/page-1", "/page-2", "/page-3"]);
});
it("should skip events with hostname exceeding max length", () => {
@@ -147,9 +153,7 @@ describe("PlausibleImportMapper", () => {
});
it("should compute channel from referrer", () => {
- const events = [
- makeEvent({ referrer: "https://google.com", querystring: "" }),
- ];
+ const events = [makeEvent({ referrer: "https://google.com", querystring: "" })];
const result = PlausibleImportMapper.transform(events, 1, "import-1");
expect(result).toHaveLength(1);
// getChannel should classify google.com as organic search
diff --git a/server/src/services/import/mappers/plausible.ts b/server/src/services/import/mappers/plausible.ts
index ba9d73430..ba9cdc597 100644
--- a/server/src/services/import/mappers/plausible.ts
+++ b/server/src/services/import/mappers/plausible.ts
@@ -1,3 +1,4 @@
+import { isClickHouseDateTime, toClickHouseDateTime } from "../../../db/clickhouse/dateTime.js";
import { clearSelfReferrer, getAllUrlParams } from "../../tracker/utils.js";
import { getChannel } from "../../tracker/getChannel.js";
import { RybbitEvent } from "./rybbit.js";
@@ -14,7 +15,10 @@ export class PlausibleImportMapper {
.string()
.regex(
/^\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01]) ([01]\d|2[0-3]):([0-5]\d):([0-5]\d)$/
- ),
+ )
+ // The regex accepts 2024-02-31; a row that is not a real instant is
+ // dropped here instead of failing the whole batch at serialisation.
+ .refine(isClickHouseDateTime),
session_id: z.string().uuid(),
user_id: z.string().uuid(),
hostname: z.string().max(253),
@@ -66,7 +70,7 @@ export class PlausibleImportMapper {
acc.push({
site_id: site,
- timestamp: data.timestamp,
+ timestamp: toClickHouseDateTime(data.timestamp),
session_id: data.session_id,
user_id: data.user_id,
hostname: data.hostname,
diff --git a/server/src/services/import/mappers/simpleAnalytics.test.ts b/server/src/services/import/mappers/simpleAnalytics.test.ts
index 80f0383bd..e92ccfb1d 100644
--- a/server/src/services/import/mappers/simpleAnalytics.test.ts
+++ b/server/src/services/import/mappers/simpleAnalytics.test.ts
@@ -54,7 +54,7 @@ describe("SimpleAnalyticsImportMapper", () => {
expect(result).toHaveLength(1);
expect(result[0]).toMatchObject({
site_id: 1,
- timestamp: "2024-06-15 14:30:00",
+ timestamp: "2024-06-15T14:30:00.000Z",
session_id: TEST_SESSION_ID,
user_id: TEST_UUID,
hostname: "example.com",
@@ -94,23 +94,36 @@ describe("SimpleAnalyticsImportMapper", () => {
});
describe("timestamp reformatting", () => {
- it("should reformat the ISO timestamp to ClickHouse format", () => {
+ it("should normalise the ISO timestamp to the explicit-UTC form ClickHouse is given", () => {
const result = SimpleAnalyticsImportMapper.transform(
[makeEvent({ added_iso: "2023-01-05T09:07:03Z" })],
1,
"i"
);
- expect(result[0].timestamp).toBe("2023-01-05 09:07:03");
+ expect(result[0].timestamp).toBe("2023-01-05T09:07:03.000Z");
});
- it("should drop fractional seconds", () => {
+ it("should keep fractional seconds", () => {
const result = SimpleAnalyticsImportMapper.transform(
[makeEvent({ added_iso: "2024-06-15T14:30:00.123Z" })],
1,
"i"
);
expect(result).toHaveLength(1);
- expect(result[0].timestamp).toBe("2024-06-15 14:30:00");
+ expect(result[0].timestamp).toBe("2024-06-15T14:30:00.123Z");
+ });
+
+ it("drops a well-shaped impossible date without losing the rest of the batch", () => {
+ const result = SimpleAnalyticsImportMapper.transform(
+ [
+ makeEvent({ added_iso: "2024-06-15T14:30:00Z" }),
+ makeEvent({ added_iso: "2024-02-31T12:00:00Z" }),
+ makeEvent({ added_iso: "2024-06-16T09:00:00Z" }),
+ ],
+ 1,
+ "i"
+ );
+ expect(result.map(row => row.timestamp)).toEqual(["2024-06-15T14:30:00.000Z", "2024-06-16T09:00:00.000Z"]);
});
it("should drop rows with non-ISO timestamps", () => {
diff --git a/server/src/services/import/mappers/simpleAnalytics.ts b/server/src/services/import/mappers/simpleAnalytics.ts
index 01e9c91d0..9355e54cb 100644
--- a/server/src/services/import/mappers/simpleAnalytics.ts
+++ b/server/src/services/import/mappers/simpleAnalytics.ts
@@ -3,7 +3,7 @@ import { getChannel } from "../../tracker/getChannel.js";
import { RybbitEvent } from "./rybbit.js";
import { z } from "zod";
import { UAParser } from "ua-parser-js";
-import { DateTime } from "luxon";
+import { isClickHouseDateTime, toClickHouseDateTime } from "../../../db/clickhouse/dateTime.js";
import { getDeviceType } from "../../../utils.js";
import { deriveKeyOnlySchema } from "./utils.js";
@@ -11,7 +11,9 @@ export type SimpleAnalyticsEvent = z.input<typeof SimpleAnalyticsImportMapper.si
export class SimpleAnalyticsImportMapper {
private static readonly simpleAnalyticsEventSchema = z.object({
- added_iso: z.string().datetime(),
+ // datetime() checks the shape, not the calendar; 2024-02-31 would
+ // otherwise throw at serialisation and fail the whole batch.
+ added_iso: z.string().datetime().refine(isClickHouseDateTime),
country_code: z
.string()
.regex(/^[A-Z]{2}$/)
@@ -52,7 +54,7 @@ export class SimpleAnalyticsImportMapper {
acc.push({
site_id: site,
- timestamp: DateTime.fromISO(data.added_iso).toFormat("yyyy-MM-dd HH:mm:ss"),
+ timestamp: toClickHouseDateTime(data.added_iso),
session_id: data.session_id,
user_id: data.uuid,
hostname: data.hostname,
diff --git a/server/src/services/import/mappers/umami.test.ts b/server/src/services/import/mappers/umami.test.ts
index 696c1b520..1bf0ca4bc 100644
--- a/server/src/services/import/mappers/umami.test.ts
+++ b/server/src/services/import/mappers/umami.test.ts
@@ -39,7 +39,7 @@ describe("UmamiImportMapper", () => {
expect(result).toHaveLength(1);
expect(result[0]).toMatchObject({
site_id: 1,
- timestamp: "2024-06-15 14:30:00",
+ timestamp: "2024-06-15T14:30:00.000Z",
session_id: TEST_SESSION_ID,
user_id: "visitor-1",
hostname: "example.com",
@@ -250,6 +250,19 @@ describe("UmamiImportMapper", () => {
expect(result[0].country).toBe("");
});
+ it("drops a well-shaped impossible date without losing the rest of the batch", () => {
+ const result = UmamiImportMapper.transform(
+ [
+ makeEvent({ created_at: "2024-06-15 14:30:00" }),
+ makeEvent({ created_at: "2024-02-31 12:00:00" }),
+ makeEvent({ created_at: "2024-06-16 09:00:00" }),
+ ],
+ 1,
+ "i"
+ );
+ expect(result.map(row => row.timestamp)).toEqual(["2024-06-15T14:30:00.000Z", "2024-06-16T09:00:00.000Z"]);
+ });
+
it("should drop rows with an invalid timestamp format", () => {
for (const created_at of [
"2024-06-15T14:30:00", // ISO "T" separator
diff --git a/server/src/services/import/mappers/umami.ts b/server/src/services/import/mappers/umami.ts
index 10114fd35..3c5d01176 100644
--- a/server/src/services/import/mappers/umami.ts
+++ b/server/src/services/import/mappers/umami.ts
@@ -1,3 +1,4 @@
+import { isClickHouseDateTime, toClickHouseDateTime } from "../../../db/clickhouse/dateTime.js";
import { clearSelfReferrer, getAllUrlParams } from "../../tracker/utils.js";
import { getChannel } from "../../tracker/getChannel.js";
import { RybbitEvent } from "./rybbit.js";
@@ -104,7 +105,12 @@ export class UmamiImportMapper {
event_type: z.enum(["1", "2"]),
event_name: z.string().max(256),
distinct_id: z.string().max(64),
- created_at: z.string().regex(/^\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01]) ([01]\d|2[0-3]):([0-5]\d):([0-5]\d)$/),
+ created_at: z
+ .string()
+ .regex(/^\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01]) ([01]\d|2[0-3]):([0-5]\d):([0-5]\d)$/)
+ // The regex accepts 2024-02-31; a row that is not a real instant is
+ // dropped here instead of failing the whole batch at serialisation.
+ .refine(isClickHouseDateTime),
});
static readonly umamiEventKeyOnlySchema = deriveKeyOnlySchema(UmamiImportMapper.umamiEventSchema);
@@ -131,7 +137,7 @@ export class UmamiImportMapper {
acc.push({
site_id: site,
- timestamp: data.created_at,
+ timestamp: toClickHouseDateTime(data.created_at),
session_id: data.session_id,
user_id: data.distinct_id,
hostname: data.hostname,
diff --git a/server/src/services/replay/sessionReplayIngestService.ts b/server/src/services/replay/sessionReplayIngestService.ts
index 167dbbebe..c28df69fb 100644
--- a/server/src/services/replay/sessionReplayIngestService.ts
+++ b/server/src/services/replay/sessionReplayIngestService.ts
@@ -1,5 +1,5 @@
-import { DateTime } from "luxon";
import { clickhouse } from "../../db/clickhouse/clickhouse.js";
+import { toClickHouseDateTime } from "../../db/clickhouse/dateTime.js";
import { RecordSessionReplayRequest } from "../../types/sessionReplay.js";
import { createServiceLogger } from "../../lib/logger/logger.js";
import { correctReplayClockSkew } from "./replayClockSkew.js";
@@ -98,7 +98,7 @@ export class SessionReplayIngestService {
session_id: sessionId,
user_id: userId,
identified_user_id: identifiedUserId,
- timestamp: event.timestamp,
+ timestamp: toClickHouseDateTime(event.timestamp),
event_type: event.type,
event_data: "", // Empty string when using R2
event_data_key: r2BatchKey,
@@ -116,7 +116,7 @@ export class SessionReplayIngestService {
session_id: sessionId,
user_id: userId,
identified_user_id: identifiedUserId,
- timestamp: event.timestamp,
+ timestamp: toClickHouseDateTime(event.timestamp),
event_type: event.type,
event_data: serializedData,
event_data_key: null,
@@ -209,8 +209,8 @@ export class SessionReplayIngestService {
session_id: sessionId,
user_id: userId,
identified_user_id: identifiedUserId,
- start_time: DateTime.fromJSDate(batchStats.startTime).toFormat("yyyy-MM-dd HH:mm:ss.SSS"),
- end_time: DateTime.fromJSDate(batchStats.endTime).toFormat("yyyy-MM-dd HH:mm:ss.SSS"),
+ start_time: toClickHouseDateTime(batchStats.startTime),
+ end_time: toClickHouseDateTime(batchStats.endTime),
event_count: batchStats.eventCount,
compressed_size_bytes: batchStats.compressedSizeBytes,
page_url: metadata.pageUrl || "",
diff --git a/server/src/services/tracker/botBlocking/botEventQueue.ts b/server/src/services/tracker/botBlocking/botEventQueue.ts
index aa27eb9a8..2cbfd57f4 100644
--- a/server/src/services/tracker/botBlocking/botEventQueue.ts
+++ b/server/src/services/tracker/botBlocking/botEventQueue.ts
@@ -1,5 +1,5 @@
-import { DateTime } from "luxon";
import { clickhouse } from "../../../db/clickhouse/clickhouse.js";
+import { toClickHouseDateTime } from "../../../db/clickhouse/dateTime.js";
import { getLocation } from "../../../db/geolocation/geolocation.js";
import { createServiceLogger } from "../../../lib/logger/logger.js";
import { getDeviceType } from "../../../utils.js";
@@ -50,7 +50,7 @@ class BotEventQueue {
return {
site_id: event.site_id,
- timestamp: DateTime.fromISO(event.timestamp).toFormat("yyyy-MM-dd HH:mm:ss"),
+ timestamp: toClickHouseDateTime(event.timestamp),
session_id: event.sessionId,
user_id: event.userId,
hostname: event.hostname || "",
diff --git a/server/src/services/tracker/pageviewQueue.test.ts b/server/src/services/tracker/pageviewQueue.test.ts
index 39046d33e..a6990cf5c 100644
--- a/server/src/services/tracker/pageviewQueue.test.ts
+++ b/server/src/services/tracker/pageviewQueue.test.ts
@@ -1,4 +1,5 @@
-import { beforeEach, describe, expect, it, vi } from "vitest";
+import { Settings } from "luxon";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type UAParser from "ua-parser-js";
import type { TotalTrackingPayload } from "./utils.js";
@@ -27,6 +28,7 @@ vi.mock("./botBlocking/datacenterAsns.js", () => ({
vi.useFakeTimers();
const { pageviewQueue } = await import("./pageviewQueue.js");
+const originalZone = Settings.defaultZone;
function makePayload(overrides: Partial<TotalTrackingPayload & { sessionId: string }> = {}) {
return {
@@ -56,13 +58,17 @@ async function flush() {
await vi.advanceTimersByTimeAsync(1000);
}
-describe("pageviewQueue ASN enrichment", () => {
+describe("pageviewQueue", () => {
beforeEach(() => {
vi.clearAllMocks();
mocks.getLocation.mockResolvedValue({});
mocks.insert.mockResolvedValue(undefined);
});
+ afterEach(() => {
+ Settings.defaultZone = originalZone;
+ });
+
it("stores the ASN, org, and datacenter classification for a recognized IP", async () => {
mocks.lookupAsn.mockReturnValue({ asn: 16509, organization: "AMAZON-02" });
mocks.isDatacenterAsn.mockReturnValue(true);
@@ -101,7 +107,35 @@ describe("pageviewQueue ASN enrichment", () => {
await vi.advanceTimersByTimeAsync(1000);
const row = mocks.insert.mock.calls[0][0].values[0];
- expect(row.timestamp).toBe("2026-08-28 12:34:56");
- expect(row.timestamp_ms).toBe("2026-08-28 12:34:56.789");
+ expect(row.timestamp).toBe("2026-08-28T12:34:56.789Z");
+ expect(row.timestamp_ms).toBe("2026-08-28T12:34:56.789Z");
+ });
+
+ it.each([
+ ["UTC", "2026-09-15T12:57:22.789Z", "2026-09-15T12:57:22.789Z"],
+ ["Europe/Berlin", "2026-09-15T14:57:22.789+02:00", "2026-09-15T12:57:22.789Z"],
+ ["Asia/Shanghai", "2026-01-15T20:57:22.001+08:00", "2026-01-15T12:57:22.001Z"],
+ ["America/New_York", "2026-01-15T07:57:22.000-05:00", "2026-01-15T12:57:22.000Z"],
+ ])("sends explicit-UTC timestamps when the backend runs in %s", async (zone, timestamp, expected) => {
+ Settings.defaultZone = zone;
+ await pageviewQueue.add(makePayload({ timestamp }));
+ await vi.advanceTimersByTimeAsync(1000);
+
+ const row = mocks.insert.mock.calls[0][0].values[0];
+ expect(row.timestamp).toBe(expected);
+ expect(row.timestamp_ms).toBe(expected);
+ });
+
+ it("keeps the repeated daylight-saving hour distinct", async () => {
+ Settings.defaultZone = "Europe/Berlin";
+ await pageviewQueue.add(makePayload({ timestamp: "2026-10-25T02:30:00.999+02:00" }));
+ await pageviewQueue.add(makePayload({ timestamp: "2026-10-25T02:30:00.999+01:00" }));
+ await vi.advanceTimersByTimeAsync(1000);
+
+ const rows = mocks.insert.mock.calls[0][0].values;
+ expect(rows.map((row: { timestamp_ms: string }) => row.timestamp_ms)).toEqual([
+ "2026-10-25T00:30:00.999Z",
+ "2026-10-25T01:30:00.999Z",
+ ]);
});
});
diff --git a/server/src/services/tracker/pageviewQueue.ts b/server/src/services/tracker/pageviewQueue.ts
index 7b3be3a9c..b2ed8b9a6 100644
--- a/server/src/services/tracker/pageviewQueue.ts
+++ b/server/src/services/tracker/pageviewQueue.ts
@@ -1,5 +1,5 @@
-import { DateTime } from "luxon";
import { clickhouse } from "../../db/clickhouse/clickhouse.js";
+import { toClickHouseDateTime } from "../../db/clickhouse/dateTime.js";
import { lookupAsn } from "../../db/geolocation/asn.js";
import { getLocation } from "../../db/geolocation/geolocation.js";
import { createServiceLogger } from "../../lib/logger/logger.js";
@@ -79,15 +79,17 @@ class PageviewQueue {
// Get all URL parameters for the url_parameters map
const allUrlParams = getAllUrlParams(pv.querystring || "");
- const receivedAt = DateTime.fromISO(pv.timestamp).toUTC();
+ const receivedAt = toClickHouseDateTime(pv.timestamp);
return {
site_id: pv.site_id,
// Keep the original DateTime column for the MergeTree key and
// whole-second time filters; timestamp_ms preserves arrival order
// for funnels and entry/exit attribution.
- timestamp: receivedAt.toFormat("yyyy-MM-dd HH:mm:ss"),
- timestamp_ms: receivedAt.toFormat("yyyy-MM-dd HH:mm:ss.SSS"),
+ // Both get the same explicit-UTC instant; the DateTime column
+ // truncates to the second on its own.
+ timestamp: receivedAt,
+ timestamp_ms: receivedAt,
session_id: pv.sessionId,
user_id: pv.userId, // Always the device fingerprint
identified_user_id: pv.identifiedUserId || "", // Custom user ID when identified