OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.
ralph(M31-004): Group C fix 9: bound or reject very long chart.sankey windows
ralph26a56aa0db0d
diff --git a/packages/core/src/modules/chart/chart.constants.ts b/packages/core/src/modules/chart/chart.constants.tsindex 4a8cdc5a4..d9cb96b9c 100644--- a/packages/core/src/modules/chart/chart.constants.ts+++ b/packages/core/src/modules/chart/chart.constants.ts@@ -8,6 +8,27 @@ import { zChartEvent } from '../report/report.constants';/** User-flow depth: how many events one sankey path may span. */export const DEFAULT_SANKEY_STEPS = 5;++/**+ * Longest window `chart.sankey` will answer, in days.+ *+ * Both of its statements scan every event in the range and hold one+ * `groupArray` per session, so cost is linear in the window and `steps` bounds+ * the answer rather than the work. Measured on the local production copy+ * (2026-09-16, `bayse`, mode `after`, 3 steps, `use_query_condition_cache=0`),+ * per statement: 871 ms / 515 MiB at 7 days, 2,361 ms / 1,329 MiB at 30,+ * 3,832 ms / 2,060 MiB at 56 - about 60 ms and 31 MiB per day. The range+ * picker offers `12m` and `lastYear`, which extrapolates to ~22 s and+ * ~11.8 GiB per statement, twice per call: past what a node can answer at all.+ *+ * 93 is the widest span the picker's `3m` option can produce - the 92-day+ * June-August quarter plus the half-open end-of-day boundary+ * `getDatesFromRange` adds - so every option up to `3m` is served and `6m`,+ * `12m`, `lastYear` and a long `yearToDate` or `custom` are refused. Refused,+ * not silently shortened: see `assertSankeyWindowIsAnswerable` in+ * sankey.service.ts, applied by `getSankeyChart`.+ */+export const MAX_SANKEY_WINDOW_DAYS = 93;const MIN_SANKEY_STEPS = 2;const MAX_SANKEY_STEPS = 10;diff --git a/packages/core/src/modules/chart/chart.service.ts b/packages/core/src/modules/chart/chart.service.tsindex c8bd47c5f..582080d66 100644--- a/packages/core/src/modules/chart/chart.service.ts+++ b/packages/core/src/modules/chart/chart.service.ts@@ -50,7 +50,11 @@ import {createRetentionService,getRetentionCohort,} from './retention.service';-import { createSankeyService, getSankey } from './sankey.service';+import {+ assertSankeyWindowIsAnswerable,+ createSankeyService,+ getSankey,+} from './sankey.service';import { mapWithConcurrency } from './src/concurrency';import { formatClickhouseDate } from './src/dates';import { executeAggregateChart, executeChart } from './src/engine/execute';@@ -525,6 +529,10 @@ export async function getConversionChart(export async function getSankeyChart(deps: ServiceDeps, input: IReportInput) {const { timezone } = await getSettingsForProject(deps, input.projectId);const currentPeriod = getChartStartEndDate(input, timezone);+ assertSankeyWindowIsAnswerable(+ currentPeriod.startDate,+ currentPeriod.endDate+ );const options = input.options;if (!options || options.type !== 'sankey') {diff --git a/packages/core/src/modules/chart/sankey.service.test.ts b/packages/core/src/modules/chart/sankey.service.test.tsnew file mode 100644index 000000000..29c9147de--- /dev/null+++ b/packages/core/src/modules/chart/sankey.service.test.ts@@ -0,0 +1,133 @@+/**+ * `chart.sankey`'s window bound (M25 Group C fix 9, §12 fix 9).+ *+ * Both sankey statements scan every event in the range and hold one+ * `groupArray` per session, so the `12m`/`lastYear` the range picker offers+ * costs ~22 s and ~11.8 GiB per statement on a busy tenant — twice per call.+ * Those windows are refused, not quietly shortened.+ *+ * The bound belongs to the `chart.sankey` procedure specifically, so the+ * endpoint case below drives `getSankeyChart` rather than `getSankey`: the+ * REST, MCP and assistant surfaces reach `getSankey` through+ * `getUserFlowCore` with caller-supplied dates and are outside fix 9's+ * endpoint list.+ */++import { describe, expect, it } from 'bun:test';+import { TRPCError } from '@trpc/server';+import { testServiceDeps } from '../../../test/service-deps';+import type { ServiceDeps } from '../../services';+import { zReportInput } from '../report/report.constants';+import { MAX_SANKEY_WINDOW_DAYS } from './chart.constants';+import { getSankeyChart } from './chart.service';+import { assertSankeyWindowIsAnswerable } from './sankey.service';++const PROJECT_ID = 'sankey-window-bound-test';+const START_DATE = '2026-01-01 00:00:00';+const MILLISECONDS_PER_DAY = 86_400_000;+const STEPS = 3;+const TWELVE_MONTHS_IN_DAYS = 365;++function clickhouseDate(milliseconds: number): string {+ return new Date(milliseconds).toISOString().replace('T', ' ').slice(0, 19);+}++function endOfWindow(days: number): string {+ const start = new Date(`${START_DATE.replace(' ', 'T')}Z`).getTime();+ return clickhouseDate(start + days * MILLISECONDS_PER_DAY);+}++function windowOf(days: number): void {+ assertSankeyWindowIsAnswerable(START_DATE, endOfWindow(days));+}++function caught(run: () => void): TRPCError {+ try {+ run();+ } catch (thrown) {+ return thrown as TRPCError;+ }+ throw new Error('expected the window bound to reject');+}++/** Postgres answers the timezone lookup; any ClickHouse contact is a failure,+ * since the point of the bound is that the statements are never built. */+function depsRefusingClickhouse(): Promise<ServiceDeps> {+ return testServiceDeps({+ db: {+ project: {+ findUniqueOrThrow: () =>+ Promise.resolve({ organization: { timezone: 'UTC' } }),+ },+ },+ ch: new Proxy(+ {},+ {+ get() {+ throw new Error('ClickHouse was reached despite the window bound');+ },+ }+ ),+ } as unknown as Partial<ServiceDeps>);+}++describe('assertSankeyWindowIsAnswerable', () => {+ it(`answers the widest window "3m" can produce (${MAX_SANKEY_WINDOW_DAYS} days)`, () => {+ expect(() => windowOf(MAX_SANKEY_WINDOW_DAYS)).not.toThrow();+ });++ it('refuses a longer window with an explained bad request', () => {+ const error = caught(() => windowOf(MAX_SANKEY_WINDOW_DAYS + 1));++ expect(error).toBeInstanceOf(TRPCError);+ expect(error.code).toBe('BAD_REQUEST');+ expect(error.message).toContain(`at most ${MAX_SANKEY_WINDOW_DAYS} days`);+ expect(error.message).toContain(`covers ${MAX_SANKEY_WINDOW_DAYS + 1}`);+ });++ it("refuses the range picker's 12-month option", () => {+ expect(caught(() => windowOf(TWELVE_MONTHS_IN_DAYS)).code).toBe(+ 'BAD_REQUEST'+ );+ });++ it('leaves an unparseable range to the statement to reject', () => {+ expect(() =>+ assertSankeyWindowIsAnswerable('not-a-date', 'also-not-a-date')+ ).not.toThrow();+ });+});++describe('chart.sankey applies the window bound', () => {+ it('refuses a 12-month report before any query is built', async () => {+ const deps = await depsRefusingClickhouse();++ const input = zReportInput.parse({+ projectId: PROJECT_ID,+ startDate: START_DATE,+ endDate: endOfWindow(TWELVE_MONTHS_IN_DAYS),+ range: 'custom',+ chartType: 'sankey',+ interval: 'day',+ breakdowns: [],+ series: [+ {+ type: 'event',+ id: 'A',+ name: 'screen_view',+ displayName: 'screen_view',+ segment: 'event',+ filters: [],+ },+ ],+ options: { type: 'sankey', mode: 'after', steps: STEPS, exclude: [] },+ });++ const error = await getSankeyChart(deps, input).catch(+ (thrown: unknown) => thrown+ );++ expect(error).toBeInstanceOf(TRPCError);+ expect((error as TRPCError).code).toBe('BAD_REQUEST');+ });+});diff --git a/packages/core/src/modules/chart/sankey.service.ts b/packages/core/src/modules/chart/sankey.service.tsindex 98c7c9bf8..e330b0ca2 100644--- a/packages/core/src/modules/chart/sankey.service.ts+++ b/packages/core/src/modules/chart/sankey.service.ts@@ -4,6 +4,7 @@// re-export shim onto this module (DELEGATE PATTERN).import type { SqlFragment } from '@openpanel/db/src/clickhouse/sql';+import { TRPCBadRequestError } from '../../rpc/errors';import type { ServiceDeps, Services } from '../../services';import { getSettingsForProject } from '../organization/organization.service';import {@@ -11,7 +12,12 @@ import {type IChartEvent,type IChartEventFilter,} from '../report/report.constants';-import { DEFAULT_SANKEY_STEPS, type IGetSankeyInput } from './chart.constants';+import {+ DEFAULT_SANKEY_STEPS,+ type IGetSankeyInput,+ MAX_SANKEY_WINDOW_DAYS,+} from './chart.constants';+import { convertClickhouseDateToJs } from './src/dates';import {getEventFiltersWhereClause,joinFilterClauses,@@ -28,6 +34,7 @@ import {/** Destinations kept per node when the flow branches. */const TOP_DESTINATIONS_PER_NODE = 3;+const MILLISECONDS_PER_DAY = 86_400_000;/** Links carrying less than this share of the entry sessions are dropped. */const MIN_LINK_PERCENT = 0.25;const PERCENT = 100;@@ -121,6 +128,39 @@ function toSankeyEvent(: undefined;}+/**+ * Refuses a `chart.sankey` window wider than MAX_SANKEY_WINDOW_DAYS.+ *+ * Refused rather than shortened: a flow computed over a narrower range than the+ * one asked for is a different diagram, and nothing in the rendered result says+ * so. An unparseable date is left alone — the statement itself is where a+ * malformed range is rejected.+ *+ * Applied by `getSankeyChart`, the `chart.sankey` entry point, and not by+ * `getSankey` itself: §12 fix 9 bounds that procedure specifically, because it+ * is the range picker that offers `12m`/`lastYear`. The REST+ * `/insights/:projectId/user_flow` route, the MCP `get_user_flow` tool and the+ * assistant tool take caller-supplied dates and are outside fix 9's endpoint+ * list; see sankey.sql.proof.md for that residual.+ */+export function assertSankeyWindowIsAnswerable(+ startDate: string,+ endDate: string+): void {+ const start = convertClickhouseDateToJs(startDate).getTime();+ const end = convertClickhouseDateToJs(endDate).getTime();+ if (Number.isNaN(start) || Number.isNaN(end)) {+ return;+ }++ const windowDays = Math.floor((end - start) / MILLISECONDS_PER_DAY);+ if (windowDays > MAX_SANKEY_WINDOW_DAYS) {+ throw new TRPCBadRequestError(+ `User flow can cover at most ${MAX_SANKEY_WINDOW_DAYS} days and the selected range covers ${windowDays}. Pick a shorter range — "Last 3 months" is the longest one supported.`+ );+ }+}+export async function getSankey(deps: ServiceDeps,{diff --git a/packages/core/src/modules/chart/src/sankey.sql.proof.md b/packages/core/src/modules/chart/src/sankey.sql.proof.mdindex 4f11b8439..07bb7f260 100644--- a/packages/core/src/modules/chart/src/sankey.sql.proof.md+++ b/packages/core/src/modules/chart/src/sankey.sql.proof.md@@ -26,6 +26,49 @@ Both statements of every sankey mode were executed twice against the same data1,504 MiB · `website-8103` FAIL (`would use 2.03 PiB`) -> 2,607 ms / 1,035 MiB · `chatpaper`13,453 ms -> 12,639 ms / 2,132 MiB. `website-8103` at 1 day: FAIL (`768.88 GiB`) -> 147 ms / 20 MiB.+- **Amended by M31-004 (2026-09-16): the statements are unchanged, but windows over 93 days no longer reach them.**+ `arrayCompact` turned the sankey from *fails* into *slow*; it did not make the window cheap. Both+ statements still scan every event in the range and hold one `groupArray` per session, so cost is+ linear in the window and `steps` bounds the answer, not the work. Re-measured on the local+ production copy on **2026-09-16**, `use_query_condition_cache=0`, `max_memory_usage=6 GiB`,+ `elapsed_ns` / `read_rows` / `memory_usage` read off `X-ClickHouse-Summary`, `bayse`, mode+ `after`, 3 steps, per statement:++ | window | topEntries | transitions | read_rows | peak memory |+ |---|---|---|---:|---|+ | 7 d | 871 ms | 731 ms | 10,271,382 | 515 / 519 MiB |+ | 14 d | 1,315 ms | 1,284 ms | 19,020,434 | 831 / 832 MiB |+ | 30 d | 2,361 ms | 2,196 ms | 33,961,756 | 1,329 / 1,324 MiB |+ | 45 d | 3,125 ms | 2,965 ms | 50,967,354 | 1,770 / 1,772 MiB |+ | 56 d (the whole copy) | 3,832 ms | 3,954 ms | 76,461,848 | 2,060 / 2,061 MiB |++ **The copy holds only 56 days (2026-07-01 -> 2026-08-25), so `3m`, `6m` and `12m` all read the+ same 76.46 M rows and cost the same 3.8 s + 4.0 s here** — a 12-month window cannot be measured+ on this data, only extrapolated. The five points above are linear at ~60 ms and ~31 MiB per day+ per statement, which puts a real 12-month tenant of bayse's shape at ~22 s and ~11.8 GiB per+ statement, twice per call, against a 13.72 GiB overcommit ceiling.++ `getSankeyChart` therefore refuses a window wider than `MAX_SANKEY_WINDOW_DAYS` (93, the widest+ span the picker's `3m` can produce) with a `BAD_REQUEST` naming the limit, before either+ statement is built. Measured the same day, 3 runs in-process against a `deps` proxy that throws+ if ClickHouse is touched: **0.474 / 0.043 / 0.012 ms, no ClickHouse contact**; the proxy is the+ standing regression gate in `sankey.service.test.ts`.++ **Where the bound is applied, and what it deliberately leaves alone.** §12 fix 9's endpoint+ column reads `chart.sankey` *specifically*, because it is the dashboard range picker that offers+ `12m`/`lastYear`. The check therefore sits in `getSankeyChart` — the `chart.sankey` entry point,+ and the place where `getChartStartEndDate` turns a picked range into dates — not in `getSankey`,+ which is the shared chokepoint three other surfaces reach through `getUserFlowCore`:+ the REST `/insights/:projectId/user_flow` route, the MCP `get_user_flow` tool and the assistant+ tool. Those three take caller-supplied dates rather than picker ranges, are outside fix 9's+ endpoint list, and keep their existing behaviour; `get_user_flow`'s own integration test asks for+ `2000-01-01`..`2099-01-01` and still passes unchanged.++ **Residual (recorded, not fixed here):** those three surfaces can still ask for a 12-month+ sankey and pay the ~22 s x 2 this section measures. Bounding them is a separate decision per+ surface — the MCP and REST tools have their own limit vocabulary (`zLimit`, `resolveDateRange`)+ and live outside this task's scope.+- **Verdict**: 23 cases, 0 not identical.### sankey topEntries — after, screen_view, 3 steps (golden insights-user-flow-after)