OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.
ralph(P1-007): P1 exit proof: e2e + stress + goldens green on the Kafka-only path
ralph541c2e3d749e
diff --git a/apps/api/e2e/README.md b/apps/api/e2e/README.mdindex 8487466e1..1658154d5 100644--- a/apps/api/e2e/README.md+++ b/apps/api/e2e/README.md@@ -52,3 +52,48 @@ endpoint, so it never waits for the 5-minute reaper cron.- Overridable: `E2E_API_URL` (default `:3333`), `E2E_WORKER_URL` (default `:9999`).- The harness and the stack **must share the same `SESSION_TIMEOUT_MS`** — theharness derives its idle waits from it.++## P1 baseline — measured throughput (2026-09-02)++`BASE-002` never produced a V1 stress number (both databases were unusable on+this box at the time — `verification/benchmarks/02-api-throughput.md`), so this+run **is** the baseline later phases compare against. Recorded here because the+number has to outlive the task that measured it.++| | |+|---|---|+| transport | Kafka/Redpanda only (`KAFKA_BROKERS=localhost:19092`); GroupMQ deleted in P1-001 |+| rev | `dc7516df` + this commit, `rewrite/v2` |+| machine | 4 vCPU AMD EPYC-Milan, 15 GiB RAM, Linux 6.8.0-137, Node v24.20.0 — api, worker, Redpanda, ClickHouse, Postgres and Redis all on the same 4 cores |+| harness | `e2e:sessions:stress`, `SESSION_TIMEOUT_MS=4000` |++**Baseline (documented defaults — 500 sessions × 3 events, concurrency 25):**+`emit` 2.0s median over 10 consecutive green runs (1.9–2.1s) =+**≈750 events/s**, one `POST /track` per event. Full run (emit → settle →+reaper drain → reconcile) 15–21s wall, 13/13 checks, 10/10 runs green.++**Concurrency sweep (2000 sessions × 3 events):** 25 → 759/s, 50 → 779/s,+100 → 789/s. Flat: the ceiling is the single API process on a shared 4-core+box, not client concurrency — consistent with DISC-021's per-process+`producer.send()` serialisation (`idempotent: true`, `maxInFlightRequests: 1`).+The ADR-004 producer redesign is the named lever, and is a separate decision.++Not measured here: per-request P50/P95/P99. `track()` in `lib.ts` has no timing+instrumentation and adding a load generator is out of this task's scope.++### One V1 property this harness has to work around++`/track` answers with a session id it derives from the Redis session blob — but+that blob is written by the **worker**, asynchronously. Until it exists the API+falls back to an id that is deterministic per `SESSION_TIMEOUT_MS`-wide time+bucket (`apps/api/src/utils/ids.ts`). At the 30-minute production default a+bucket boundary is effectively never hit mid-visit; at the harness's compressed+4s window it is hit inside the emit ramp, and every session in flight at that+instant is told a *different* id for its next event.++The rows still land — they just carry a session id the harness was never given.+That is why `reconcile` counts by **device within the run window** and not by+the echoed session ids: scoped by session id the same run reads 2490/2500 and+looks like event loss when nothing was lost. Verified against the same data:+device-scoped 2500, session-scoped 2490, `session_start`/`session_end` 500 in+both. Pre-existing V1 behaviour, unrelated to the transport.diff --git a/apps/api/e2e/session-stress.ts b/apps/api/e2e/session-stress.tsindex 907a90d20..4bf5331de 100644--- a/apps/api/e2e/session-stress.ts+++ b/apps/api/e2e/session-stress.ts@@ -11,7 +11,8 @@* SESSION_TIMEOUT_MS=4000 pnpm --filter @openpanel/api e2e:sessions:stress** Tunables (env): E2E_SESSIONS (500), E2E_CONCURRENCY (25),- * E2E_EVENTS_PER_SESSION (3), E2E_DRAIN_TIMEOUT_MS (120000).+ * E2E_EVENTS_PER_SESSION (3), E2E_DRAIN_TIMEOUT_MS (120000),+ * E2E_RECONCILE_TIMEOUT_MS (30000).*/import {@@ -53,6 +54,21 @@ const DRAIN_TIMEOUT_MS = Number.parseInt(10);+// drain() exits on session_end + a clean Redis, which says nothing about the+// run's trailing regular events — those can still sit in the event buffer.+const RECONCILE_TIMEOUT_MS = Number.parseInt(+ process.env.E2E_RECONCILE_TIMEOUT_MS || '30000',+ 10+);+const RECONCILE_INTERVAL_MS = 1500;+// Consecutive samples with no new rows before we call the buffer drained and+// let the assertion report whatever is actually there.+const RECONCILE_STABLE_SAMPLES = 3;+// The reconcile window opens this far before the run starts: a session_start+// is backdated 100ms from the event that opened it, and the first requests+// leave within milliseconds of the captured start.+const RECONCILE_WINDOW_LEAD_MS = 2000;+type Session = { sessionId: string; deviceId: string };// Unique IP per session → unique device → unique session. Namespaced by runId@@ -189,16 +205,56 @@ async function drain(sessions: Session[]) {check('redis: no session blobs leaked (sampled)', leaked === 0, `${leaked}/${sample.length} leaked`);}-async function reconcile(sessions: Session[]) {- scenario('reconcile: ClickHouse event counts');- const ids = sessions.map((s) => s.sessionId);- const n = ids.length;- const inList = ids.map((id) => `'${id}'`).join(',');+/**+ * Count this run's rows by event name, scoped to the run's DEVICES rather than+ * to the session ids `/track` echoed back.+ *+ * Why the device: until the worker has persisted the session blob, the API+ * answers from a session id that is deterministic per SESSION_TIMEOUT_MS-wide+ * time bucket (apps/api/src/utils/ids.ts). At the harness's compressed 4s+ * window a bucket boundary falls inside the emit ramp, so an in-flight+ * session's next event is told a different id — the row still lands, under a+ * session id the harness was never given. The device is the stable identity+ * for "what this run sent"; `since` keeps earlier runs' rows out, because a+ * device id repeats across runs whenever the ip/ua/salt triple does.+ */+async function countEventsByName(deviceIds: string[], since: Date) {+ const inList = deviceIds.map((id) => `'${id}'`).join(',');+ const from = new Date(since.getTime() - RECONCILE_WINDOW_LEAD_MS)+ .toISOString()+ .replace('T', ' ')+ .replace('Z', '');const rows = await chQuery<{ name: string; c: string }>(- `SELECT name, count() AS c FROM events WHERE project_id = '${PROJECT_ID}' AND session_id IN (${inList}) GROUP BY name ORDER BY c DESC`+ `SELECT name, count() AS c FROM events WHERE project_id = '${PROJECT_ID}' AND device_id IN (${inList}) AND created_at >= toDateTime64('${from}', 3) GROUP BY name ORDER BY c DESC`);const byName = new Map(rows.map((r) => [r.name, Number(r.c)]));const total = [...byName.values()].reduce((a, b) => a + b, 0);+ return { rows, byName, total };+}++async function reconcile(sessions: Session[], since: Date) {+ scenario('reconcile: ClickHouse event counts');+ const devices = sessions.map((s) => s.deviceId);+ const n = sessions.length;+ const expectedTotal = n * (EVENTS_PER_SESSION + 2);++ let counts = await countEventsByName(devices, since);+ const deadline = Date.now() + RECONCILE_TIMEOUT_MS;+ let stableSamples = 0;+ while (+ counts.total < expectedTotal &&+ stableSamples < RECONCILE_STABLE_SAMPLES &&+ Date.now() < deadline+ ) {+ console.log(` …events ${counts.total}/${expectedTotal}, flushing`);+ await triggerCron('flushEvents');+ await sleep(RECONCILE_INTERVAL_MS);+ const next = await countEventsByName(devices, since);+ stableSamples = next.total > counts.total ? 0 : stableSamples + 1;+ counts = next;+ }++ const { rows, byName, total } = counts;console.log(` event breakdown: ${rows.map((r) => `${r.name}=${r.c}`).join(' ')}`);const starts = byName.get('session_start') ?? 0;@@ -208,8 +264,8 @@ async function reconcile(sessions: Session[]) {check('session_end == sessions', ends === n, `${ends} vs ${n}`);check('total events == sessions × (events + start + end)',- total === n * (EVENTS_PER_SESSION + 2),- `${total} vs ${n * (EVENTS_PER_SESSION + 2)}`+ total === expectedTotal,+ `${total} vs ${expectedTotal}`);}@@ -221,10 +277,13 @@ async function main() {await preflight();await ensureFixtures();+ // Captured before the first event so the reconcile window covers every row+ // of this run, including session_start (backdated 100ms).+ const runStartedAt = new Date();const sessions = await emit();await settle(sessions);await drain(sessions);- await reconcile(sessions);+ await reconcile(sessions, runStartedAt);await shutdown(summarize());}