Skip to content

🐸 Rybbit - open-source and privacy-friendly alternative to Google Analytics that is 10x more intuitive.

Enhance user identification in analytics queries

Bill Yang3e1e1ec2a9eb
diff --git a/docs/content/docs/(docs)/identify-users.mdx b/docs/content/docs/(docs)/identify-users.mdx
index d6a455d62..bf2d91f40 100644
--- a/docs/content/docs/(docs)/identify-users.mdx
+++ b/docs/content/docs/(docs)/identify-users.mdx
@@ -110,6 +110,7 @@ if (currentUserId) {
## Important Notes
+- **Unique user counts**: Every user count in the dashboard β€” Unique Users, the users chart, the Users table, traits, and retention β€” keys on the custom user ID wherever one is present, falling back to the anonymous fingerprint only for visitors who have never identified. Two people sharing an IP address and browser count as two users, and one person on a laptop and a phone counts as one. Sites that never call `identify()` are unaffected. Because the count follows the IDs you pass, sending a fresh ID per visit (rather than a stable per-person ID) will inflate it.
- **Identified sessions**: When a custom user ID is present, session assignment includes both the anonymous device fingerprint and the custom user ID. Different identified users therefore remain in separate sessions even when they share the same IP address and user agent.
- **Shared devices**: If multiple users log in from the same device (same IP address and user agent), calling `identify()` with a new user ID updates the alias to point to the new user. Future events will be attributed to the most recently identified user. Past events that were already backfilled retain their original attribution.
- **Cross-device tracking**: Identifying the same user on different devices creates separate aliases. Events from each device are linked to the user independently via the `identified_user_id` field.
diff --git a/server/src/api/analytics/getOverview.test.ts b/server/src/api/analytics/getOverview.test.ts
new file mode 100644
index 000000000..6da6a5a7c
--- /dev/null
+++ b/server/src/api/analytics/getOverview.test.ts
@@ -0,0 +1,59 @@
+import { describe, expect, it, vi } from "vitest";
+
+vi.mock("../../db/clickhouse/clickhouse.js", () => ({
+ clickhouse: { query: vi.fn() },
+}));
+vi.mock("../../db/postgres/postgres.js", () => ({
+ db: {},
+}));
+
+import { FilterParams } from "@rybbit/shared";
+import { buildOverviewQuery } from "./getOverview.js";
+import { buildOverviewBucketedQuery } from "./getOverviewBucketed.js";
+
+const SITE_ID = 1;
+
+const baseParams = (overrides: Partial<Record<string, unknown>> = {}) =>
+ ({
+ start_date: "2026-07-17",
+ end_date: "2026-07-23",
+ time_zone: "UTC",
+ filters: "",
+ bucket: "day",
+ ...overrides,
+ }) as FilterParams & { bucket: "day" };
+
+describe("unique user counting", () => {
+ describe("buildOverviewQuery", () => {
+ it("counts users by identity, falling back to the device fingerprint", () => {
+ const sql = buildOverviewQuery(baseParams(), SITE_ID);
+
+ expect(sql).toContain("COUNT(DISTINCT f.effective_user_id) AS users");
+ expect(sql).toContain("anyIf(identified_user_id, identified_user_id != '')");
+ expect(sql).toContain("anyLast(user_id)");
+ });
+
+ it("resolves the identity once per session rather than per event", () => {
+ const sql = buildOverviewQuery(baseParams(), SITE_ID);
+
+ // The identity expression must sit inside the session-grouped CTE, so a
+ // visitor whose first pageview predates identify() still counts once.
+ const sessionCteBody = sql.split("FilteredSessionsWithStats AS (")[1].split("GROUP BY session_id")[0];
+ expect(sessionCteBody).toContain("anyIf(identified_user_id");
+ });
+
+ it("never aliases the identity back onto user_id", () => {
+ // Shadowing a column with an alias used inside its own expression is a
+ // cyclic-alias error in ClickHouse.
+ expect(buildOverviewQuery(baseParams(), SITE_ID)).not.toMatch(/anyLast\(user_id\)\) AS user_id/);
+ });
+ });
+
+ describe("buildOverviewBucketedQuery", () => {
+ it("counts users by identity, falling back to the device fingerprint", () => {
+ const sql = buildOverviewBucketedQuery(baseParams(), SITE_ID);
+
+ expect(sql).toContain("COUNT(DISTINCT COALESCE(NULLIF(identified_user_id, ''), user_id)) AS users");
+ });
+ });
+});
diff --git a/server/src/api/analytics/getOverview.ts b/server/src/api/analytics/getOverview.ts
index fa83b3cfc..3bd1fdc81 100644
--- a/server/src/api/analytics/getOverview.ts
+++ b/server/src/api/analytics/getOverview.ts
@@ -3,6 +3,7 @@ import { FastifyReply, FastifyRequest } from "fastify";
import { getFilterStatement } from "./utils/getFilterStatement.js";
import { getTimeStatement } from "./utils/utils.js";
import { analyticsRoute, runAnalyticsQuery } from "./utils/analyticsQuery.js";
+import { EFFECTIVE_SESSION_USER_ID } from "./utils/effectiveUserId.js";
type GetOverviewResponse = {
sessions: number;
@@ -32,7 +33,9 @@ export const buildOverviewQuery = (params: FilterParams, siteId: number) => {
FilteredSessionsWithStats AS (
SELECT
session_id,
- anyLast(user_id) AS user_id,
+ -- Aliased away from \`user_id\` on purpose: an alias that shadows a column
+ -- referenced inside its own expression is a cyclic-alias error in ClickHouse.
+ ${EFFECTIVE_SESSION_USER_ID} AS effective_user_id,
MIN(timestamp) AS start_time,
MAX(timestamp) AS end_time,
countIf(type = 'pageview') AS filtered_pageviews
@@ -49,7 +52,7 @@ export const buildOverviewQuery = (params: FilterParams, siteId: number) => {
sumIf(1, asp.total_pageviews_in_session = 1) / COUNT() * 100 AS bounce_rate,
AVG(f.end_time - f.start_time) AS session_duration,
SUM(f.filtered_pageviews) AS pageviews,
- COUNT(DISTINCT f.user_id) AS users
+ COUNT(DISTINCT f.effective_user_id) AS users
FROM FilteredSessionsWithStats f
LEFT JOIN AllSessionPageviews asp ON f.session_id = asp.session_id`;
};
diff --git a/server/src/api/analytics/getOverviewBucketed.ts b/server/src/api/analytics/getOverviewBucketed.ts
index 730fb1a05..f7baeaf57 100644
--- a/server/src/api/analytics/getOverviewBucketed.ts
+++ b/server/src/api/analytics/getOverviewBucketed.ts
@@ -11,6 +11,7 @@ import {
import { getFilterStatement } from "./utils/getFilterStatement.js";
import { TimeBucket } from "./types.js";
import { analyticsRoute, runAnalyticsQuery } from "./utils/analyticsQuery.js";
+import { effectiveUserId } from "./utils/effectiveUserId.js";
function getTimeStatementFill(params: FilterParams, bucket: TimeBucket) {
const { params: validatedParams, bucket: validatedBucket } = validateTimeStatementFillParams(params, bucket);
@@ -168,7 +169,7 @@ FULL JOIN
SELECT
toDateTime(${TimeBucketToFn[bucket]}(toTimeZone(timestamp, ${tzEscaped}))) AS time,
countIf(type = 'pageview') AS pageviews,
- COUNT(DISTINCT user_id) AS users
+ COUNT(DISTINCT ${effectiveUserId()}) AS users
FROM events
WHERE
site_id = {siteId:Int32}
diff --git a/server/src/api/analytics/getRetention.ts b/server/src/api/analytics/getRetention.ts
index 9b56d39aa..31cccbc65 100644
--- a/server/src/api/analytics/getRetention.ts
+++ b/server/src/api/analytics/getRetention.ts
@@ -1,5 +1,6 @@
import { FastifyReply, FastifyRequest } from "fastify";
import { analyticsRoute, runAnalyticsQuery } from "./utils/analyticsQuery.js";
+import { effectiveUserId } from "./utils/effectiveUserId.js";
// Define the expected shape of a single data row from the query
interface RetentionDataRow {
@@ -32,7 +33,7 @@ export const buildRetentionQuery = (retentionMode: "day" | "week") => {
WITH UserFirstPeriod AS (
SELECT
-- Use effective user ID: identified_user_id for identified users, user_id for anonymous
- COALESCE(NULLIF(identified_user_id, ''), user_id) AS effective_user_id,
+ ${effectiveUserId()} AS effective_user_id,
${periodFunction}(min(timestamp)${retentionMode === "week" ? ", 1" : ""}) AS cohort_period
FROM events
WHERE site_id = {siteId:UInt16}
@@ -42,7 +43,7 @@ WITH UserFirstPeriod AS (
),
PeriodActivity AS (
SELECT DISTINCT
- COALESCE(NULLIF(identified_user_id, ''), user_id) AS effective_user_id,
+ ${effectiveUserId()} AS effective_user_id,
${periodFunction}(timestamp${retentionMode === "week" ? ", 1" : ""}) AS activity_period
FROM events
WHERE site_id = {siteId:UInt16}
diff --git a/server/src/api/analytics/users/getUserTraits.ts b/server/src/api/analytics/users/getUserTraits.ts
index 4ee875274..a4fe8a027 100644
--- a/server/src/api/analytics/users/getUserTraits.ts
+++ b/server/src/api/analytics/users/getUserTraits.ts
@@ -2,6 +2,7 @@ import { FastifyReply, FastifyRequest } from "fastify";
import { sql } from "drizzle-orm";
import { db } from "../../../db/postgres/postgres.js";
import { analyticsRoute, runAnalyticsQuery } from "../utils/analyticsQuery.js";
+import { effectiveUserId } from "../utils/effectiveUserId.js";
export interface GetUserTraitKeysRequest {
Params: { siteId: string };
@@ -86,7 +87,7 @@ export const getUserTraitValues = analyticsRoute<GetUserTraitValuesRequest>(
// Use events.identified_user_id to avoid conflict with the argMax alias
export const buildTraitValueUsersQuery = () => `
SELECT
- COALESCE(NULLIF(events.identified_user_id, ''), events.user_id) AS effective_user_id,
+ ${effectiveUserId("events")} AS effective_user_id,
argMax(events.user_id, timestamp) AS user_id,
argMax(events.identified_user_id, timestamp) AS identified_user_id,
argMax(country, timestamp) AS country,
diff --git a/server/src/api/analytics/users/getUsers.ts b/server/src/api/analytics/users/getUsers.ts
index 25b9841f5..6e3238dfd 100644
--- a/server/src/api/analytics/users/getUsers.ts
+++ b/server/src/api/analytics/users/getUsers.ts
@@ -6,6 +6,7 @@ import { FilterParams } from "@rybbit/shared";
import { getFilterStatement } from "../utils/getFilterStatement.js";
import { SESSION_CHANNEL_AGG, SESSION_REFERRER_AGG } from "../utils/sessionAttribution.js";
import { analyticsRoute, runAnalyticsQuery } from "../utils/analyticsQuery.js";
+import { effectiveUserId } from "../utils/effectiveUserId.js";
export type GetUsersResponse = {
user_id: string; // Device fingerprint
@@ -88,7 +89,7 @@ FROM (
`
: `
SELECT
- count(DISTINCT COALESCE(NULLIF(events.identified_user_id, ''), events.user_id)) AS total_count
+ count(DISTINCT ${effectiveUserId("events")}) AS total_count
FROM events
WHERE
site_id = {siteId:Int32}
@@ -106,7 +107,7 @@ WHERE
WITH AggregatedUsers AS (
SELECT
-- Group by effective user: identified_user_id for identified users, user_id (device) for anonymous
- COALESCE(NULLIF(events.identified_user_id, ''), events.user_id) AS effective_user_id,
+ ${effectiveUserId("events")} AS effective_user_id,
argMax(user_id, timestamp) AS user_id,
argMax(identified_user_id, timestamp) AS identified_user_id,
argMax(country, timestamp) AS country,
diff --git a/server/src/api/analytics/utils/effectiveUserId.ts b/server/src/api/analytics/utils/effectiveUserId.ts
new file mode 100644
index 000000000..03a86fc86
--- /dev/null
+++ b/server/src/api/analytics/utils/effectiveUserId.ts
@@ -0,0 +1,40 @@
+/**
+ * How a "user" is counted across analytics queries.
+ *
+ * `user_id` is an anonymous device fingerprint β€” sha256(bucketed IP + user agent).
+ * It collides whenever visitors share an egress IP and a browser build (corporate
+ * proxies, VPNs, Private Relay, CGNAT, managed fleets) and it splits whenever one
+ * person uses two devices. `identified_user_id` is the stable ID the site passes to
+ * `identify()`, so it is strictly better information wherever it is present.
+ *
+ * Every user-facing count therefore keys on the identity when there is one and falls
+ * back to the fingerprint otherwise. For sites that never call `identify()` the
+ * column is '' on every row and these expressions collapse to plain `user_id`,
+ * leaving their numbers byte-identical.
+ */
+
+/**
+ * Event-level key. Use when the query aggregates raw events and has no session
+ * grain to resolve against.
+ *
+ * Caveat: the first pageview of a visit fires before `identify()` resolves, so
+ * those rows carry '' and count under the fingerprint β€” one visitor can register
+ * as two users in the window where they cross over. Prefer
+ * {@link EFFECTIVE_SESSION_USER_ID} when the query already groups by session.
+ */
+export const effectiveUserId = (tableAlias = "") => {
+ const prefix = tableAlias ? `${tableAlias}.` : "";
+ return `COALESCE(NULLIF(${prefix}identified_user_id, ''), ${prefix}user_id)`;
+};
+
+/**
+ * Session-level key, for use inside a `GROUP BY session_id` aggregation.
+ *
+ * Sessions are keyed per identified user (see sessionsService.getSessionKey), so a
+ * session's events either all carry the same identity or none do β€” taking any
+ * non-empty `identified_user_id` in the session is well defined, and it avoids the
+ * pre-identify split that the event-level key has. Pre-#1045 rows, where several
+ * identities could share one anonymous session, resolve to whichever identity the
+ * aggregate happens to see; that is no worse than the fingerprint they resolve to today.
+ */
+export const EFFECTIVE_SESSION_USER_ID = `COALESCE(NULLIF(anyIf(identified_user_id, identified_user_id != ''), ''), anyLast(user_id))`;