OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.
perf(ch): projection for the property-key autocomplete + value cap (migration 19) (#446)
Ritik Jained7bdddbadbe
diff --git a/apps/public/content/docs/self-hosting/environment-variables.mdx b/apps/public/content/docs/self-hosting/environment-variables.mdxindex 9507dc49b..bb4442ebf 100644--- a/apps/public/content/docs/self-hosting/environment-variables.mdx+++ b/apps/public/content/docs/self-hosting/environment-variables.mdx@@ -1066,6 +1066,19 @@ Batch size for data import operations.IMPORT_BATCH_SIZE=10000```+### EVENT_PROPERTY_VALUE_AUTOCOMPLETE_LIMIT++**Type**: `number` (positive integer)+**Required**: No+**Default**: `500`++Cap on distinct values returned per event property to the filter autocomplete. High-cardinality keys (ids, urls, tokens) can hold millions of distinct values; the most recent ones win. Invalid values keep the default.++**Example**:+```bash+EVENT_PROPERTY_VALUE_AUTOCOMPLETE_LIMIT=1000+```+### IP_HEADER_ORDER**Type**: `string` (comma-separated)diff --git a/packages/db/code-migrations/19-event-property-values-projections.ts b/packages/db/code-migrations/19-event-property-values-projections.tsnew file mode 100644index 000000000..c99c9f5f6--- /dev/null+++ b/packages/db/code-migrations/19-event-property-values-projections.ts@@ -0,0 +1,217 @@+import { TABLE_NAMES } from '../src/clickhouse/client';+import {+ chMigrationClient,+ runClickhouseMigrationCommands,+} from '../src/clickhouse/migration';+import { getIsCluster } from './helpers';++/**+ * Aggregating projection for the property-key autocomplete dropdown.+ *+ * The key picker reads event_property_values_mv, which is ORDER BY+ * (project_id, name, property_key, property_value). Two shapes exist:+ *+ * no event selected -> WHERE project_id ... GROUP BY property_key+ * event selected -> WHERE project_id AND name ... GROUP BY property_key+ *+ * Neither is cheap on that sort key. The first can seek only to project_id+ * and must then aggregate the project's whole slice (on our deployment,+ * 1.57B rows to return ~6.4K distinct keys). The second prunes to the+ * event's slice, but that slice still holds every key AND every value for+ * the event — measured 120M rows read on average, p95 15.9s.+ *+ * One aggregating projection answers both:+ *+ * epv_keys -> (project_id, name, property_key), max(created_at)+ *+ * With an event selected, (project_id, name) is a seekable prefix, so the+ * read prunes to that event's keys. With no event, the query's GROUP BY is+ * a subset of the projection's, so the optimizer still substitutes it and+ * aggregates over the (tiny) projection instead of the table — verified+ * with EXPLAIN for both shapes. The optimizer rewrites the existing picker+ * queries transparently; no application change.+ *+ * Sizing: the projection is one row per (project, event, key) — 30,212 rows+ * against a 1.637B-row MV on our deployment. Including `name` costs ~3.4x+ * the rows of a (project_id, property_key)-only projection while covering+ * the event-selected shape that one cannot serve at all (a projection is+ * only substitutable when it contains EVERY column the query references,+ * so any filter on `name` disqualifies a projection that lacks it).+ *+ * NOT included: a (project_id, property_key, property_value) projection for+ * the value picker. Its grain is 95.4% of the MV's row count — a near-+ * complete second copy of the table — and it can only serve value lookups+ * with no event selected, since the event-selected shape already gets a+ * full three-column prefix seek on the base table. Measured on our+ * deployment: 2 such queries in 30 days.+ *+ * Notes:+ * - The projection lives on the MV's STORAGE table: the implicit+ * `.inner_id.<uuid>` table on a single node, `<mv>_replicated` when+ * clustered.+ * - AggregatingMergeTree refuses ADD PROJECTION while+ * deduplicate_merge_projection_mode is 'throw' (the default);+ * 'rebuild' keeps projections correct across dedup merges.+ * - The migration ADDs the projection and submits MATERIALIZE PROJECTION+ * for existing parts by default. The mutation is asynchronous (the+ * migration doesn't block on it) and idempotent; queries stay correct+ * over mixed parts while it runs — ClickHouse reads the projection from+ * parts that have it and the base table from those that don't (observed+ * in EXPLAIN as two read nodes). Fresh installs no-op. Progress:+ *+ * SELECT * FROM system.mutations WHERE command LIKE '%epv_%';+ *+ * - The mutation rewrites projection data for every part, so deployments+ * with a very large MV that want to control WHEN the backfill runs can+ * apply the setting + ADD PROJECTION + MATERIALIZE PROJECTION statements+ * manually before upgrading (off-peak), against the storage table. The+ * migration skips re-materializing only when the projection already+ * exists AND system.mutations records a completed MATERIALIZE for it —+ * presence alone could be a crashed earlier attempt's ADD, and if+ * mutation state can't be read the migration materializes anyway+ * (idempotent; redundant work beats a silent skip). Note that because+ * ADD uses IF NOT EXISTS, a manual pre-apply must use the definition+ * below verbatim — a same-named projection with a different grain would+ * be kept as-is.+ */++const MV = TABLE_NAMES.event_property_values_mv;++const PROJECTIONS = [+ {+ name: 'epv_keys',+ def: 'SELECT project_id, name, property_key, max(created_at) AS created_at GROUP BY project_id, name, property_key',+ },+];++// The projection lives on the MV's storage table: the implicit inner table+// (`.inner_id.<uuid>`) on a single node, or `<mv>_replicated` when clustered.+async function resolveStorageTable(isClustered: boolean): Promise<string> {+ if (isClustered) {+ return `${MV}_replicated`;+ }+ const res = await chMigrationClient.query({+ query: `SELECT concat('.inner_id.', toString(uuid)) AS t+ FROM system.tables WHERE database = currentDatabase() AND name = '${MV}'`,+ format: 'JSONEachRow',+ });+ const rows = await res.json<{ t: string }>();+ if (!rows[0]?.t) {+ throw new Error(`${MV}: inner storage table not found`);+ }+ return rows[0].t;+}++// Token-boundary match so a similarly named projection (epv_keys_v2) can+// never satisfy the checks — substring/LIKE matching would.+const nameBoundary = (name: string) =>+ new RegExp(`${name}([^A-Za-z0-9_]|$)`);++async function existingProjections(storage: string): Promise<Set<string>> {+ const res = await chMigrationClient.query({+ query: `SHOW CREATE TABLE \`${storage}\``,+ format: 'JSONEachRow',+ });+ const [row] = await res.json<{ statement: string }>();+ return new Set(+ PROJECTIONS.filter((p) =>+ nameBoundary(`PROJECTION ${p.name}`).test(row?.statement ?? ''),+ ).map((p) => p.name),+ );+}++async function materializedProjections(+ storage: string,+ isClustered: boolean,+): Promise<Set<string>> {+ try {+ // Clustered: a completed mutation on the connected host doesn't prove+ // the other hosts finished (or even received) theirs — require every+ // host in the cluster to report one, per projection.+ const source = isClustered+ ? `clusterAllReplicas('{cluster}', system.mutations)`+ : 'system.mutations';+ // (hostName(), tcpPort()) rather than hostName() alone: multiple+ // replicas on one machine share a hostname, and collapsing them could+ // count an incomplete replica as done.+ const res = await chMigrationClient.query({+ query: `SELECT concat(hostName(), ':', toString(tcpPort())) AS host, command FROM ${source}+ WHERE database = currentDatabase() AND table = '${storage}'+ AND command LIKE '%MATERIALIZE PROJECTION%'+ AND is_done = 1`,+ format: 'JSONEachRow',+ });+ const rows = await res.json<{ host: string; command: string }>();++ let totalHosts = 1;+ if (isClustered) {+ const hostsRes = await chMigrationClient.query({+ query: `SELECT countDistinct((hostName(), tcpPort())) AS c FROM clusterAllReplicas('{cluster}', system.one)`,+ format: 'JSONEachRow',+ });+ const [hostsRow] = await hostsRes.json<{ c: string | number }>();+ totalHosts = Number(hostsRow?.c ?? 0);+ if (totalHosts === 0) {+ return new Set();+ }+ }++ return new Set(+ PROJECTIONS.filter((p) => {+ const matcher = nameBoundary(`MATERIALIZE PROJECTION ${p.name}`);+ const hosts = new Set(+ rows.filter((r) => matcher.test(r.command)).map((r) => r.host),+ );+ return hosts.size >= totalHosts;+ }).map((p) => p.name),+ );+ } catch {+ // Can't verify (e.g. no grant on system.mutations / cluster functions) —+ // materialize; the mutations are idempotent and redundant work beats a+ // silent skip.+ return new Set();+ }+}++export async function up() {+ const isClustered = getIsCluster();+ const storage = await resolveStorageTable(isClustered);+ const tbl = `\`${storage}\``;+ const onCluster = isClustered ? " ON CLUSTER '{cluster}'" : '';++ // Skip a projection's backfill only when it exists AND a completed+ // MATERIALIZE mutation is on record (a manual pre-apply, see header) —+ // presence alone could be a crashed earlier attempt's ADD.+ const existing = await existingProjections(storage);+ const materialized = await materializedProjections(storage, isClustered);++ await runClickhouseMigrationCommands([+ `ALTER TABLE ${tbl}${onCluster} MODIFY SETTING deduplicate_merge_projection_mode = 'rebuild'`,+ ...PROJECTIONS.map(+ (p) =>+ `ALTER TABLE ${tbl}${onCluster} ADD PROJECTION IF NOT EXISTS ${p.name} (${p.def})`,+ ),+ ...PROJECTIONS.filter(+ (p) => !(existing.has(p.name) && materialized.has(p.name)),+ ).map(+ (p) => `ALTER TABLE ${tbl}${onCluster} MATERIALIZE PROJECTION ${p.name}`,+ ),+ ]);+}++export async function down() {+ const isClustered = getIsCluster();+ const storage = await resolveStorageTable(isClustered);+ const tbl = `\`${storage}\``;+ const onCluster = isClustered ? " ON CLUSTER '{cluster}'" : '';++ await runClickhouseMigrationCommands([+ ...PROJECTIONS.map(+ (p) => `ALTER TABLE ${tbl}${onCluster} DROP PROJECTION IF EXISTS ${p.name}`,+ ),+ // Restore the engine default ('throw'). Safe here because this down()+ // just dropped the only projections on the table; if you've added your+ // own projections to it, keep 'rebuild'.+ `ALTER TABLE ${tbl}${onCluster} MODIFY SETTING deduplicate_merge_projection_mode = 'throw'`,+ ]);+}diff --git a/packages/trpc/src/routers/chart.ts b/packages/trpc/src/routers/chart.tsindex 88cc29f14..aea2980cc 100644--- a/packages/trpc/src/routers/chart.ts+++ b/packages/trpc/src/routers/chart.ts@@ -59,6 +59,27 @@ const cacher = cacheMiddleware(60);*/const EVENT_PROPERTY_KEY_LIMIT = 50_000;+/**+ * Cap on distinct values returned per event property to the filter+ * autocomplete. High-cardinality keys (ids, urls, session tokens) can hold+ * millions of distinct values — returning them all is useless for a picker+ * and heavy for ClickHouse and the browser alike. Most recent values win.+ * Env-tunable via EVENT_PROPERTY_VALUE_AUTOCOMPLETE_LIMIT (positive+ * integer; invalid values keep the default).+ */+const EVENT_PROPERTY_VALUE_AUTOCOMPLETE_LIMIT_RAW =+ process.env.EVENT_PROPERTY_VALUE_AUTOCOMPLETE_LIMIT;+const EVENT_PROPERTY_VALUE_AUTOCOMPLETE_LIMIT_PARSED =+ EVENT_PROPERTY_VALUE_AUTOCOMPLETE_LIMIT_RAW &&+ /^\d+$/.test(EVENT_PROPERTY_VALUE_AUTOCOMPLETE_LIMIT_RAW)+ ? Number(EVENT_PROPERTY_VALUE_AUTOCOMPLETE_LIMIT_RAW)+ : Number.NaN;+const EVENT_PROPERTY_VALUE_AUTOCOMPLETE_LIMIT =+ Number.isSafeInteger(EVENT_PROPERTY_VALUE_AUTOCOMPLETE_LIMIT_PARSED) &&+ EVENT_PROPERTY_VALUE_AUTOCOMPLETE_LIMIT_PARSED > 0+ ? EVENT_PROPERTY_VALUE_AUTOCOMPLETE_LIMIT_PARSED+ : 500;+const chartProcedure = publicProcedure.use(async ({ ctx, next, getRawInput }) => {const rawInput = (await getRawInput()) as {@@ -344,7 +365,11 @@ export const chartRouter = createTRPCRouter({.where('project_id', '=', projectId).where('property_key', '=', property.replace(/^properties\./, '')).groupBy(['property_value'])- .orderBy('created_at', 'DESC');+ // Recency + key tie-break for a stable list under the cap — same+ // rationale as the key picker above.+ .orderBy('created_at', 'DESC')+ .orderBy('property_value', 'ASC')+ .limit(EVENT_PROPERTY_VALUE_AUTOCOMPLETE_LIMIT);if (event && event !== '*') {query.where('name', '=', event);