Skip to content

OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.

ralph(M31-004): Group C fix 9: bound or reject very long chart.sankey windows

ralph26a56aa0db0d
diff --git a/packages/core/src/modules/chart/chart.constants.ts b/packages/core/src/modules/chart/chart.constants.ts
index 4a8cdc5a4..d9cb96b9c 100644
--- a/packages/core/src/modules/chart/chart.constants.ts
+++ b/packages/core/src/modules/chart/chart.constants.ts
@@ -8,6 +8,27 @@ import { zChartEvent } from '../report/report.constants';
/** User-flow depth: how many events one sankey path may span. */
export const DEFAULT_SANKEY_STEPS = 5;
+
+/**
+ * Longest window `chart.sankey` will answer, in days.
+ *
+ * Both of its statements scan every event in the range and hold one
+ * `groupArray` per session, so cost is linear in the window and `steps` bounds
+ * the answer rather than the work. Measured on the local production copy
+ * (2026-09-16, `bayse`, mode `after`, 3 steps, `use_query_condition_cache=0`),
+ * per statement: 871 ms / 515 MiB at 7 days, 2,361 ms / 1,329 MiB at 30,
+ * 3,832 ms / 2,060 MiB at 56 - about 60 ms and 31 MiB per day. The range
+ * picker offers `12m` and `lastYear`, which extrapolates to ~22 s and
+ * ~11.8 GiB per statement, twice per call: past what a node can answer at all.
+ *
+ * 93 is the widest span the picker's `3m` option can produce - the 92-day
+ * June-August quarter plus the half-open end-of-day boundary
+ * `getDatesFromRange` adds - so every option up to `3m` is served and `6m`,
+ * `12m`, `lastYear` and a long `yearToDate` or `custom` are refused. Refused,
+ * not silently shortened: see `assertSankeyWindowIsAnswerable` in
+ * sankey.service.ts, applied by `getSankeyChart`.
+ */
+export const MAX_SANKEY_WINDOW_DAYS = 93;
const MIN_SANKEY_STEPS = 2;
const MAX_SANKEY_STEPS = 10;
diff --git a/packages/core/src/modules/chart/chart.service.ts b/packages/core/src/modules/chart/chart.service.ts
index c8bd47c5f..582080d66 100644
--- a/packages/core/src/modules/chart/chart.service.ts
+++ b/packages/core/src/modules/chart/chart.service.ts
@@ -50,7 +50,11 @@ import {
createRetentionService,
getRetentionCohort,
} from './retention.service';
-import { createSankeyService, getSankey } from './sankey.service';
+import {
+ assertSankeyWindowIsAnswerable,
+ createSankeyService,
+ getSankey,
+} from './sankey.service';
import { mapWithConcurrency } from './src/concurrency';
import { formatClickhouseDate } from './src/dates';
import { executeAggregateChart, executeChart } from './src/engine/execute';
@@ -525,6 +529,10 @@ export async function getConversionChart(
export async function getSankeyChart(deps: ServiceDeps, input: IReportInput) {
const { timezone } = await getSettingsForProject(deps, input.projectId);
const currentPeriod = getChartStartEndDate(input, timezone);
+ assertSankeyWindowIsAnswerable(
+ currentPeriod.startDate,
+ currentPeriod.endDate
+ );
const options = input.options;
if (!options || options.type !== 'sankey') {
diff --git a/packages/core/src/modules/chart/sankey.service.test.ts b/packages/core/src/modules/chart/sankey.service.test.ts
new file mode 100644
index 000000000..29c9147de
--- /dev/null
+++ b/packages/core/src/modules/chart/sankey.service.test.ts
@@ -0,0 +1,133 @@
+/**
+ * `chart.sankey`'s window bound (M25 Group C fix 9, §12 fix 9).
+ *
+ * Both sankey statements scan every event in the range and hold one
+ * `groupArray` per session, so the `12m`/`lastYear` the range picker offers
+ * costs ~22 s and ~11.8 GiB per statement on a busy tenant — twice per call.
+ * Those windows are refused, not quietly shortened.
+ *
+ * The bound belongs to the `chart.sankey` procedure specifically, so the
+ * endpoint case below drives `getSankeyChart` rather than `getSankey`: the
+ * REST, MCP and assistant surfaces reach `getSankey` through
+ * `getUserFlowCore` with caller-supplied dates and are outside fix 9's
+ * endpoint list.
+ */
+
+import { describe, expect, it } from 'bun:test';
+import { TRPCError } from '@trpc/server';
+import { testServiceDeps } from '../../../test/service-deps';
+import type { ServiceDeps } from '../../services';
+import { zReportInput } from '../report/report.constants';
+import { MAX_SANKEY_WINDOW_DAYS } from './chart.constants';
+import { getSankeyChart } from './chart.service';
+import { assertSankeyWindowIsAnswerable } from './sankey.service';
+
+const PROJECT_ID = 'sankey-window-bound-test';
+const START_DATE = '2026-01-01 00:00:00';
+const MILLISECONDS_PER_DAY = 86_400_000;
+const STEPS = 3;
+const TWELVE_MONTHS_IN_DAYS = 365;
+
+function clickhouseDate(milliseconds: number): string {
+ return new Date(milliseconds).toISOString().replace('T', ' ').slice(0, 19);
+}
+
+function endOfWindow(days: number): string {
+ const start = new Date(`${START_DATE.replace(' ', 'T')}Z`).getTime();
+ return clickhouseDate(start + days * MILLISECONDS_PER_DAY);
+}
+
+function windowOf(days: number): void {
+ assertSankeyWindowIsAnswerable(START_DATE, endOfWindow(days));
+}
+
+function caught(run: () => void): TRPCError {
+ try {
+ run();
+ } catch (thrown) {
+ return thrown as TRPCError;
+ }
+ throw new Error('expected the window bound to reject');
+}
+
+/** Postgres answers the timezone lookup; any ClickHouse contact is a failure,
+ * since the point of the bound is that the statements are never built. */
+function depsRefusingClickhouse(): Promise<ServiceDeps> {
+ return testServiceDeps({
+ db: {
+ project: {
+ findUniqueOrThrow: () =>
+ Promise.resolve({ organization: { timezone: 'UTC' } }),
+ },
+ },
+ ch: new Proxy(
+ {},
+ {
+ get() {
+ throw new Error('ClickHouse was reached despite the window bound');
+ },
+ }
+ ),
+ } as unknown as Partial<ServiceDeps>);
+}
+
+describe('assertSankeyWindowIsAnswerable', () => {
+ it(`answers the widest window "3m" can produce (${MAX_SANKEY_WINDOW_DAYS} days)`, () => {
+ expect(() => windowOf(MAX_SANKEY_WINDOW_DAYS)).not.toThrow();
+ });
+
+ it('refuses a longer window with an explained bad request', () => {
+ const error = caught(() => windowOf(MAX_SANKEY_WINDOW_DAYS + 1));
+
+ expect(error).toBeInstanceOf(TRPCError);
+ expect(error.code).toBe('BAD_REQUEST');
+ expect(error.message).toContain(`at most ${MAX_SANKEY_WINDOW_DAYS} days`);
+ expect(error.message).toContain(`covers ${MAX_SANKEY_WINDOW_DAYS + 1}`);
+ });
+
+ it("refuses the range picker's 12-month option", () => {
+ expect(caught(() => windowOf(TWELVE_MONTHS_IN_DAYS)).code).toBe(
+ 'BAD_REQUEST'
+ );
+ });
+
+ it('leaves an unparseable range to the statement to reject', () => {
+ expect(() =>
+ assertSankeyWindowIsAnswerable('not-a-date', 'also-not-a-date')
+ ).not.toThrow();
+ });
+});
+
+describe('chart.sankey applies the window bound', () => {
+ it('refuses a 12-month report before any query is built', async () => {
+ const deps = await depsRefusingClickhouse();
+
+ const input = zReportInput.parse({
+ projectId: PROJECT_ID,
+ startDate: START_DATE,
+ endDate: endOfWindow(TWELVE_MONTHS_IN_DAYS),
+ range: 'custom',
+ chartType: 'sankey',
+ interval: 'day',
+ breakdowns: [],
+ series: [
+ {
+ type: 'event',
+ id: 'A',
+ name: 'screen_view',
+ displayName: 'screen_view',
+ segment: 'event',
+ filters: [],
+ },
+ ],
+ options: { type: 'sankey', mode: 'after', steps: STEPS, exclude: [] },
+ });
+
+ const error = await getSankeyChart(deps, input).catch(
+ (thrown: unknown) => thrown
+ );
+
+ expect(error).toBeInstanceOf(TRPCError);
+ expect((error as TRPCError).code).toBe('BAD_REQUEST');
+ });
+});
diff --git a/packages/core/src/modules/chart/sankey.service.ts b/packages/core/src/modules/chart/sankey.service.ts
index 98c7c9bf8..e330b0ca2 100644
--- a/packages/core/src/modules/chart/sankey.service.ts
+++ b/packages/core/src/modules/chart/sankey.service.ts
@@ -4,6 +4,7 @@
// re-export shim onto this module (DELEGATE PATTERN).
import type { SqlFragment } from '@openpanel/db/src/clickhouse/sql';
+import { TRPCBadRequestError } from '../../rpc/errors';
import type { ServiceDeps, Services } from '../../services';
import { getSettingsForProject } from '../organization/organization.service';
import {
@@ -11,7 +12,12 @@ import {
type IChartEvent,
type IChartEventFilter,
} from '../report/report.constants';
-import { DEFAULT_SANKEY_STEPS, type IGetSankeyInput } from './chart.constants';
+import {
+ DEFAULT_SANKEY_STEPS,
+ type IGetSankeyInput,
+ MAX_SANKEY_WINDOW_DAYS,
+} from './chart.constants';
+import { convertClickhouseDateToJs } from './src/dates';
import {
getEventFiltersWhereClause,
joinFilterClauses,
@@ -28,6 +34,7 @@ import {
/** Destinations kept per node when the flow branches. */
const TOP_DESTINATIONS_PER_NODE = 3;
+const MILLISECONDS_PER_DAY = 86_400_000;
/** Links carrying less than this share of the entry sessions are dropped. */
const MIN_LINK_PERCENT = 0.25;
const PERCENT = 100;
@@ -121,6 +128,39 @@ function toSankeyEvent(
: undefined;
}
+/**
+ * Refuses a `chart.sankey` window wider than MAX_SANKEY_WINDOW_DAYS.
+ *
+ * Refused rather than shortened: a flow computed over a narrower range than the
+ * one asked for is a different diagram, and nothing in the rendered result says
+ * so. An unparseable date is left alone — the statement itself is where a
+ * malformed range is rejected.
+ *
+ * Applied by `getSankeyChart`, the `chart.sankey` entry point, and not by
+ * `getSankey` itself: §12 fix 9 bounds that procedure specifically, because it
+ * is the range picker that offers `12m`/`lastYear`. The REST
+ * `/insights/:projectId/user_flow` route, the MCP `get_user_flow` tool and the
+ * assistant tool take caller-supplied dates and are outside fix 9's endpoint
+ * list; see sankey.sql.proof.md for that residual.
+ */
+export function assertSankeyWindowIsAnswerable(
+ startDate: string,
+ endDate: string
+): void {
+ const start = convertClickhouseDateToJs(startDate).getTime();
+ const end = convertClickhouseDateToJs(endDate).getTime();
+ if (Number.isNaN(start) || Number.isNaN(end)) {
+ return;
+ }
+
+ const windowDays = Math.floor((end - start) / MILLISECONDS_PER_DAY);
+ if (windowDays > MAX_SANKEY_WINDOW_DAYS) {
+ throw new TRPCBadRequestError(
+ `User flow can cover at most ${MAX_SANKEY_WINDOW_DAYS} days and the selected range covers ${windowDays}. Pick a shorter range — "Last 3 months" is the longest one supported.`
+ );
+ }
+}
+
export async function getSankey(
deps: ServiceDeps,
{
diff --git a/packages/core/src/modules/chart/src/sankey.sql.proof.md b/packages/core/src/modules/chart/src/sankey.sql.proof.md
index 4f11b8439..07bb7f260 100644
--- a/packages/core/src/modules/chart/src/sankey.sql.proof.md
+++ b/packages/core/src/modules/chart/src/sankey.sql.proof.md
@@ -26,6 +26,49 @@ Both statements of every sankey mode were executed twice against the same data
1,504 MiB · `website-8103` FAIL (`would use 2.03 PiB`) -> 2,607 ms / 1,035 MiB · `chatpaper`
13,453 ms -> 12,639 ms / 2,132 MiB. `website-8103` at 1 day: FAIL (`768.88 GiB`) -> 147 ms / 20 MiB.
+- **Amended by M31-004 (2026-09-16): the statements are unchanged, but windows over 93 days no longer reach them.**
+ `arrayCompact` turned the sankey from *fails* into *slow*; it did not make the window cheap. Both
+ statements still scan every event in the range and hold one `groupArray` per session, so cost is
+ linear in the window and `steps` bounds the answer, not the work. Re-measured on the local
+ production copy on **2026-09-16**, `use_query_condition_cache=0`, `max_memory_usage=6 GiB`,
+ `elapsed_ns` / `read_rows` / `memory_usage` read off `X-ClickHouse-Summary`, `bayse`, mode
+ `after`, 3 steps, per statement:
+
+ | window | topEntries | transitions | read_rows | peak memory |
+ |---|---|---|---:|---|
+ | 7 d | 871 ms | 731 ms | 10,271,382 | 515 / 519 MiB |
+ | 14 d | 1,315 ms | 1,284 ms | 19,020,434 | 831 / 832 MiB |
+ | 30 d | 2,361 ms | 2,196 ms | 33,961,756 | 1,329 / 1,324 MiB |
+ | 45 d | 3,125 ms | 2,965 ms | 50,967,354 | 1,770 / 1,772 MiB |
+ | 56 d (the whole copy) | 3,832 ms | 3,954 ms | 76,461,848 | 2,060 / 2,061 MiB |
+
+ **The copy holds only 56 days (2026-07-01 -> 2026-08-25), so `3m`, `6m` and `12m` all read the
+ same 76.46 M rows and cost the same 3.8 s + 4.0 s here** — a 12-month window cannot be measured
+ on this data, only extrapolated. The five points above are linear at ~60 ms and ~31 MiB per day
+ per statement, which puts a real 12-month tenant of bayse's shape at ~22 s and ~11.8 GiB per
+ statement, twice per call, against a 13.72 GiB overcommit ceiling.
+
+ `getSankeyChart` therefore refuses a window wider than `MAX_SANKEY_WINDOW_DAYS` (93, the widest
+ span the picker's `3m` can produce) with a `BAD_REQUEST` naming the limit, before either
+ statement is built. Measured the same day, 3 runs in-process against a `deps` proxy that throws
+ if ClickHouse is touched: **0.474 / 0.043 / 0.012 ms, no ClickHouse contact**; the proxy is the
+ standing regression gate in `sankey.service.test.ts`.
+
+ **Where the bound is applied, and what it deliberately leaves alone.** §12 fix 9's endpoint
+ column reads `chart.sankey` *specifically*, because it is the dashboard range picker that offers
+ `12m`/`lastYear`. The check therefore sits in `getSankeyChart` — the `chart.sankey` entry point,
+ and the place where `getChartStartEndDate` turns a picked range into dates — not in `getSankey`,
+ which is the shared chokepoint three other surfaces reach through `getUserFlowCore`:
+ the REST `/insights/:projectId/user_flow` route, the MCP `get_user_flow` tool and the assistant
+ tool. Those three take caller-supplied dates rather than picker ranges, are outside fix 9's
+ endpoint list, and keep their existing behaviour; `get_user_flow`'s own integration test asks for
+ `2000-01-01`..`2099-01-01` and still passes unchanged.
+
+ **Residual (recorded, not fixed here):** those three surfaces can still ask for a 12-month
+ sankey and pay the ~22 s x 2 this section measures. Bounding them is a separate decision per
+ surface — the MCP and REST tools have their own limit vocabulary (`zLimit`, `resolveDateRange`)
+ and live outside this task's scope.
+
- **Verdict**: 23 cases, 0 not identical.
### sankey topEntries — after, screen_view, 3 steps (golden insights-user-flow-after)