Skip to content

OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.

ralph(P1-007): P1 exit proof: e2e + stress + goldens green on the Kafka-only path

ralph541c2e3d749e
diff --git a/apps/api/e2e/README.md b/apps/api/e2e/README.md
index 8487466e1..1658154d5 100644
--- a/apps/api/e2e/README.md
+++ b/apps/api/e2e/README.md
@@ -52,3 +52,48 @@ endpoint, so it never waits for the 5-minute reaper cron.
- Overridable: `E2E_API_URL` (default `:3333`), `E2E_WORKER_URL` (default `:9999`).
- The harness and the stack **must share the same `SESSION_TIMEOUT_MS`** — the
harness derives its idle waits from it.
+
+## P1 baseline — measured throughput (2026-09-02)
+
+`BASE-002` never produced a V1 stress number (both databases were unusable on
+this box at the time — `verification/benchmarks/02-api-throughput.md`), so this
+run **is** the baseline later phases compare against. Recorded here because the
+number has to outlive the task that measured it.
+
+| | |
+|---|---|
+| transport | Kafka/Redpanda only (`KAFKA_BROKERS=localhost:19092`); GroupMQ deleted in P1-001 |
+| rev | `dc7516df` + this commit, `rewrite/v2` |
+| machine | 4 vCPU AMD EPYC-Milan, 15 GiB RAM, Linux 6.8.0-137, Node v24.20.0 — api, worker, Redpanda, ClickHouse, Postgres and Redis all on the same 4 cores |
+| harness | `e2e:sessions:stress`, `SESSION_TIMEOUT_MS=4000` |
+
+**Baseline (documented defaults — 500 sessions × 3 events, concurrency 25):**
+`emit` 2.0s median over 10 consecutive green runs (1.9–2.1s) =
+**≈750 events/s**, one `POST /track` per event. Full run (emit → settle →
+reaper drain → reconcile) 15–21s wall, 13/13 checks, 10/10 runs green.
+
+**Concurrency sweep (2000 sessions × 3 events):** 25 → 759/s, 50 → 779/s,
+100 → 789/s. Flat: the ceiling is the single API process on a shared 4-core
+box, not client concurrency — consistent with DISC-021's per-process
+`producer.send()` serialisation (`idempotent: true`, `maxInFlightRequests: 1`).
+The ADR-004 producer redesign is the named lever, and is a separate decision.
+
+Not measured here: per-request P50/P95/P99. `track()` in `lib.ts` has no timing
+instrumentation and adding a load generator is out of this task's scope.
+
+### One V1 property this harness has to work around
+
+`/track` answers with a session id it derives from the Redis session blob — but
+that blob is written by the **worker**, asynchronously. Until it exists the API
+falls back to an id that is deterministic per `SESSION_TIMEOUT_MS`-wide time
+bucket (`apps/api/src/utils/ids.ts`). At the 30-minute production default a
+bucket boundary is effectively never hit mid-visit; at the harness's compressed
+4s window it is hit inside the emit ramp, and every session in flight at that
+instant is told a *different* id for its next event.
+
+The rows still land — they just carry a session id the harness was never given.
+That is why `reconcile` counts by **device within the run window** and not by
+the echoed session ids: scoped by session id the same run reads 2490/2500 and
+looks like event loss when nothing was lost. Verified against the same data:
+device-scoped 2500, session-scoped 2490, `session_start`/`session_end` 500 in
+both. Pre-existing V1 behaviour, unrelated to the transport.
diff --git a/apps/api/e2e/session-stress.ts b/apps/api/e2e/session-stress.ts
index 907a90d20..4bf5331de 100644
--- a/apps/api/e2e/session-stress.ts
+++ b/apps/api/e2e/session-stress.ts
@@ -11,7 +11,8 @@
* SESSION_TIMEOUT_MS=4000 pnpm --filter @openpanel/api e2e:sessions:stress
*
* Tunables (env): E2E_SESSIONS (500), E2E_CONCURRENCY (25),
- * E2E_EVENTS_PER_SESSION (3), E2E_DRAIN_TIMEOUT_MS (120000).
+ * E2E_EVENTS_PER_SESSION (3), E2E_DRAIN_TIMEOUT_MS (120000),
+ * E2E_RECONCILE_TIMEOUT_MS (30000).
*/
import {
@@ -53,6 +54,21 @@ const DRAIN_TIMEOUT_MS = Number.parseInt(
10
);
+// drain() exits on session_end + a clean Redis, which says nothing about the
+// run's trailing regular events — those can still sit in the event buffer.
+const RECONCILE_TIMEOUT_MS = Number.parseInt(
+ process.env.E2E_RECONCILE_TIMEOUT_MS || '30000',
+ 10
+);
+const RECONCILE_INTERVAL_MS = 1500;
+// Consecutive samples with no new rows before we call the buffer drained and
+// let the assertion report whatever is actually there.
+const RECONCILE_STABLE_SAMPLES = 3;
+// The reconcile window opens this far before the run starts: a session_start
+// is backdated 100ms from the event that opened it, and the first requests
+// leave within milliseconds of the captured start.
+const RECONCILE_WINDOW_LEAD_MS = 2000;
+
type Session = { sessionId: string; deviceId: string };
// Unique IP per session → unique device → unique session. Namespaced by runId
@@ -189,16 +205,56 @@ async function drain(sessions: Session[]) {
check('redis: no session blobs leaked (sampled)', leaked === 0, `${leaked}/${sample.length} leaked`);
}
-async function reconcile(sessions: Session[]) {
- scenario('reconcile: ClickHouse event counts');
- const ids = sessions.map((s) => s.sessionId);
- const n = ids.length;
- const inList = ids.map((id) => `'${id}'`).join(',');
+/**
+ * Count this run's rows by event name, scoped to the run's DEVICES rather than
+ * to the session ids `/track` echoed back.
+ *
+ * Why the device: until the worker has persisted the session blob, the API
+ * answers from a session id that is deterministic per SESSION_TIMEOUT_MS-wide
+ * time bucket (apps/api/src/utils/ids.ts). At the harness's compressed 4s
+ * window a bucket boundary falls inside the emit ramp, so an in-flight
+ * session's next event is told a different id — the row still lands, under a
+ * session id the harness was never given. The device is the stable identity
+ * for "what this run sent"; `since` keeps earlier runs' rows out, because a
+ * device id repeats across runs whenever the ip/ua/salt triple does.
+ */
+async function countEventsByName(deviceIds: string[], since: Date) {
+ const inList = deviceIds.map((id) => `'${id}'`).join(',');
+ const from = new Date(since.getTime() - RECONCILE_WINDOW_LEAD_MS)
+ .toISOString()
+ .replace('T', ' ')
+ .replace('Z', '');
const rows = await chQuery<{ name: string; c: string }>(
- `SELECT name, count() AS c FROM events WHERE project_id = '${PROJECT_ID}' AND session_id IN (${inList}) GROUP BY name ORDER BY c DESC`
+ `SELECT name, count() AS c FROM events WHERE project_id = '${PROJECT_ID}' AND device_id IN (${inList}) AND created_at >= toDateTime64('${from}', 3) GROUP BY name ORDER BY c DESC`
);
const byName = new Map(rows.map((r) => [r.name, Number(r.c)]));
const total = [...byName.values()].reduce((a, b) => a + b, 0);
+ return { rows, byName, total };
+}
+
+async function reconcile(sessions: Session[], since: Date) {
+ scenario('reconcile: ClickHouse event counts');
+ const devices = sessions.map((s) => s.deviceId);
+ const n = sessions.length;
+ const expectedTotal = n * (EVENTS_PER_SESSION + 2);
+
+ let counts = await countEventsByName(devices, since);
+ const deadline = Date.now() + RECONCILE_TIMEOUT_MS;
+ let stableSamples = 0;
+ while (
+ counts.total < expectedTotal &&
+ stableSamples < RECONCILE_STABLE_SAMPLES &&
+ Date.now() < deadline
+ ) {
+ console.log(` …events ${counts.total}/${expectedTotal}, flushing`);
+ await triggerCron('flushEvents');
+ await sleep(RECONCILE_INTERVAL_MS);
+ const next = await countEventsByName(devices, since);
+ stableSamples = next.total > counts.total ? 0 : stableSamples + 1;
+ counts = next;
+ }
+
+ const { rows, byName, total } = counts;
console.log(` event breakdown: ${rows.map((r) => `${r.name}=${r.c}`).join(' ')}`);
const starts = byName.get('session_start') ?? 0;
@@ -208,8 +264,8 @@ async function reconcile(sessions: Session[]) {
check('session_end == sessions', ends === n, `${ends} vs ${n}`);
check(
'total events == sessions × (events + start + end)',
- total === n * (EVENTS_PER_SESSION + 2),
- `${total} vs ${n * (EVENTS_PER_SESSION + 2)}`
+ total === expectedTotal,
+ `${total} vs ${expectedTotal}`
);
}
@@ -221,10 +277,13 @@ async function main() {
await preflight();
await ensureFixtures();
+ // Captured before the first event so the reconcile window covers every row
+ // of this run, including session_start (backdated 100ms).
+ const runStartedAt = new Date();
const sessions = await emit();
await settle(sessions);
await drain(sessions);
- await reconcile(sessions);
+ await reconcile(sessions, runStartedAt);
await shutdown(summarize());
}