OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.
hub testing
Carl-Gerhard Lindesvärd890207babb8f
diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.mddeleted file mode 100644index cf5295c5b..000000000--- a/.claude/CLAUDE.md+++ /dev/null@@ -1,168 +0,0 @@-# CLAUDE.md--NEVER CALL FORMAT! WE'LL FORMAT IN THE FUTURE WHEN WE HAVE MERGED ALL BIG PRS!--This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.--## Project Overview--Openpanel is an open-source web/product analytics platform (Mixpanel alternative). It's a **bun monorepo** with apps, packages, tooling, and SDKs.--## Common Commands--```bash-# Development-bun run dev # Run apps/api (ROLE=all, port 3333) + apps/start dashboard (port 3000) in parallel-bun run dev:public # Run public/docs site only-bun run dock:up / dock:down # Start/stop Docker (PostgreSQL, Redis, ClickHouse)--# Code quality-bun run check # Lint (Biome via Ultracite) + dependency-cruiser (check:deps)-bun run fix # Auto-fix lint/format issues-bun run typecheck # Typecheck all packages--# Testing-bun run test # @openpanel/core, @openpanel/db, @openpanel/api and start's own suites, then root vitest for the rest-cd packages/core && bun test # core's own suite (ADR-010; bun:test cannot run under vitest)-bunx vitest run <path> # Run a single vitest test file--# Database-bun run codegen # Generate the Prisma client + core's geo/bot/ASN data-bun run migrate # Run Prisma migrations (dev)-bun run migrate:deploy # Deploy migrations (production - never run this)--# Docker utilities-bun run dock:ch # ClickHouse CLI-bun run dock:redis # Redis CLI-```--## Architecture--### Apps--| App | Stack | Port | Purpose |-|-----|-------|------|---------|-| `apps/api` | Elysia on Bun, `ROLE=api\|worker\|all` | 3000 (`API_PORT`) | REST/tRPC API, BullMQ workers, Kafka ingest consumer — one process, role-gated |-| `apps/start` | TanStack Start (Vite + React 19) | 3000 (dev default; run alongside `apps/api` needs a distinct `API_PORT`) | Dashboard SPA |-| `apps/public` | Next.js 16 + Fumadocs | - | Marketing/docs site |--The standalone worker app is gone — M9 merged it into `apps/api`, selected at boot by `ROLE`.--### Key Packages--| Package | Purpose |-|---------|---------|-| `packages/core` | The backend: every module (`<name>.rpc.ts` / `.routes.ts` / `.jobs.ts` / `.service.ts`), the tRPC router, Elysia route surfaces, the BullMQ job registry, auth macros, the ClickHouse `sql` tag consumer code, shared utilities. `apps/api` is a thin shell that mounts it |-| `packages/db` | Prisma ORM (PostgreSQL) + ClickHouse client, the `sql` tagged-template query builder, migrations |-| `packages/redis` | Redis client (Bun `RedisClient`) + caching, locks, pub/sub |-| `packages/email` | React Email templates via Resend |-| `packages/payments` | Billing/subscription integration |-| `packages/sdks/*` | Client SDKs (web, react, next, express, react-native, etc.) |--### Data Flow--1. **Event ingestion**: Client SDKs → `apps/api` `/track` → Kafka/Redpanda topic `events`-2. **Processing**: `apps/api`'s Kafka consumer (any non-`api` role) batches events into ClickHouse-3. **Dashboard queries**: `apps/start` → tRPC (fetch adapter) → `apps/api` → ClickHouse (analytics) / PostgreSQL (config)-4. **Real-time**: WebSocket via Elysia, pub/sub via Redis--### Three-Database Strategy--- **PostgreSQL**: Relational data (users, orgs, projects, dashboards). Managed by Prisma.-- **ClickHouse**: Analytics event storage (OLAP). High-volume reads/writes.-- **Redis**: Caching, BullMQ job queues, rate limiting, pub/sub.--### Dashboard (apps/start)--Uses TanStack Router with file-based routing (`src/routes/`). State management via Redux Toolkit. UI built on Radix primitives + Tailwind v4. Charts via Recharts. Modals in `src/modals/`.--### API (apps/api)--Elysia on Bun, mounting `@openpanel/core`'s routers. No bundler — `bun run src/main.ts` runs the TypeScript directly (ADR-010). `main.ts` boots deps, `app.ts` wires the four Elysia scopes (root, dashboard, public API, ops); `ROLE` selects whether workers, schedulers and the Kafka consumer also start in this process.------## Core Principles--Write code that is **accessible, performant, type-safe, and maintainable**. Focus on clarity and explicit intent over brevity.--### Type Safety & Explicitness--- Use explicit types for function parameters and return values when they enhance clarity-- Prefer `unknown` over `any` when the type is genuinely unknown-- Use const assertions (`as const`) for immutable values and literal types-- Leverage TypeScript's type narrowing instead of type assertions-- Use meaningful variable names instead of magic numbers - extract constants with descriptive names--### Modern JavaScript/TypeScript--- Use arrow functions for callbacks and short functions-- Prefer `for...of` loops over `.forEach()` and indexed `for` loops-- Use optional chaining (`?.`) and nullish coalescing (`??`) for safer property access-- Prefer template literals over string concatenation-- Use destructuring for object and array assignments-- Use `const` by default, `let` only when reassignment is needed, never `var`--### Async & Promises--- Always `await` promises in async functions - don't forget to use the return value-- Use `async/await` syntax instead of promise chains for better readability-- Handle errors appropriately in async code with try-catch blocks-- Don't use async functions as Promise executors--### React & JSX--- Use function components over class components-- Call hooks at the top level only, never conditionally-- Specify all dependencies in hook dependency arrays correctly-- Use the `key` prop for elements in iterables (prefer unique IDs over array indices)-- Nest children between opening and closing tags instead of passing as props-- Don't define components inside other components-- Use semantic HTML and ARIA attributes for accessibility:- - Provide meaningful alt text for images- - Use proper heading hierarchy- - Add labels for form inputs- - Include keyboard event handlers alongside mouse events- - Use semantic elements (`<button>`, `<nav>`, etc.) instead of divs with roles--### Error Handling & Debugging--- Remove `console.log`, `debugger`, and `alert` statements from production code-- Throw `Error` objects with descriptive messages, not strings or other values-- Use `try-catch` blocks meaningfully - don't catch errors just to rethrow them-- Prefer early returns over nested conditionals for error cases--### Code Organization--- Keep functions focused and under reasonable cognitive complexity limits-- Extract complex conditions into well-named boolean variables-- Use early returns to reduce nesting-- Prefer simple conditionals over nested ternary operators-- Group related code together and separate concerns--### Security--- Add `rel="noopener"` when using `target="_blank"` on links-- Avoid `dangerouslySetInnerHTML` unless absolutely necessary-- Don't use `eval()` or assign directly to `document.cookie`-- Validate and sanitize user input--### Performance--- Avoid spread syntax in accumulators within loops-- Use top-level regex literals instead of creating them in loops-- Prefer specific imports over namespace imports-- Avoid barrel files (index files that re-export everything)-- Use proper image components (e.g., Next.js `<Image>`) over `<img>` tags--### Framework-Specific Guidance--**Next.js:**-- Use Next.js `<Image>` component for images-- Use `next/head` or App Router metadata API for head elements-- Use Server Components for async data fetching instead of async Client Components--**React 19+:**-- Use ref as a prop instead of `React.forwardRef`--**Solid/Svelte/Vue/Qwik:**-- Use `class` and `for` attributes (not `className` or `htmlFor`)diff --git a/.gitignore b/.gitignoreindex 482baa8ac..02e89ab2f 100644--- a/.gitignore+++ b/.gitignore@@ -195,3 +195,14 @@ dist.worktree-port.worktree-dbWORKTREE.md++# seed output and playwright auth state+.seed.json+apps/start/e2e/.auth/++# playwright output+apps/start/test-results/+apps/start/playwright-report/++# SQL dumps the ClickHouse code-migration runner writes next to each migration+packages/db/src/code-migrations/*.sqldiff --git a/AGENTS.md b/AGENTS.mdnew file mode 100644index 000000000..660a8ead2--- /dev/null+++ b/AGENTS.md@@ -0,0 +1,191 @@+# OpenPanel — agent guide++**If `WORKTREE.md` exists in the repo root, read it before anything else.** The tooling+that created this worktree generates it: URLs, ports, database names, the seeded login+and client secrets, where the process logs are and how to restart a process. Never guess+a URL or port — take them from `WORKTREE.md` or from your shell env (`$DASHBOARD_URL`,+`$API_URL`, `$DATABASE_URL`, `$CLICKHOUSE_URL`, `$REDIS_URL`).++Backend conventions (module layout, procedure builders, queues, tests) are the+contract in `packages/core/AGENTS.md`. Read it before touching `packages/core`.++## What this is++Open-source product analytics (a Mixpanel alternative). Bun monorepo.++| Path | What | Stack |+|---|---|---|+| `packages/core` | The backend: every module, the tRPC router, Elysia routes, BullMQ jobs, ingest | TypeScript |+| `apps/api` | Thin shell that mounts core; `ROLE=api\|worker\|all` picks what one process runs | Elysia on Bun |+| `apps/start` | The dashboard | TanStack Start, React 19, Redux Toolkit, Tailwind v4, Recharts |+| `packages/db` | Prisma (Postgres) + ClickHouse client and the `sql` tag (`src/clickhouse/sql.ts`) | |+| `packages/seed` | Deterministic seed data for Postgres + ClickHouse (`bun run seed`) | |+| `packages/redis`, `packages/email`, `packages/payments`, `packages/sdks/*` | Redis client/locks/pubsub, React Email, billing, client SDKs | |+| `apps/public` | Marketing + docs | Next.js, Fumadocs |++Data flow: SDK → `POST /track` (`packages/core/src/modules/ingest`) → Kafka → the+worker role → ClickHouse (`events`, `sessions`, `profiles`, materialized views).+Dashboard → tRPC (`$API_URL/trpc`) → ClickHouse for analytics, Postgres for config.++## Your environment++You are usually inside a **worktree** under `.worktrees/<name>` with its own port block,+its own Postgres/ClickHouse database (`openpanel_<name>`), Redis db index and Kafka+topics. `WORKTREE.md` says which, and whether the processes (`web`, `api`, `studio`) are+supervised for you.++- **Supervised processes**: `WORKTREE.md` gives the log files and the exact restart+ command. Prefer restarting over starting your own copy — two processes on one port is+ the usual cause of "my change does nothing". If the supervisor is gone, your shell has+ the same env it used, so `bun run --filter @openpanel/api dev` (or `start`) starts a+ process on the right port. `tail -f` the log while you test.+- Env is injected into your shell and wins over `.env`; `.env` in a worktree is a copy of+ the main checkout's and mostly irrelevant.+- Never point at another worktree's database or edit files under `.worktrees/`.+- Migrations: `bun run --filter @openpanel/db migrate:deploy` runs both Prisma and the+ ClickHouse code migrations against *your* databases (already run when the worktree was+ created and seeded).++Without `WORKTREE.md` you are in a plain checkout: `.env` applies, `bun run dock:up`+starts Postgres/Redis/ClickHouse/Redpanda, and `bun run dev` starts api + dashboard.+The dashboard's server side runs under the Cloudflare Vite plugin, which reads+`apps/start/.env.local` (and wrangler `vars`) rather than your shell — set `API_URL` and+`DASHBOARD_URL` there, or export `CLOUDFLARE_INCLUDE_PROCESS_ENV=true` to make it use+the shell env.++## Data++- `bun run seed` — seeds a user, organization, projects and clients into Postgres and+ realistic events/sessions/profiles into ClickHouse. `--size small` (default, ~14+ days), `medium`, `large`, `xl` (tens of millions of events, for query-performance+ work), or `--days N --sessions-per-day N`. `--reset` wipes the seeded projects first.+ Same `--seed` ⇒ identical data. `--dry-run` prints the row estimate.+- `.seed.json` (gitignored, repo root) holds what was seeded: login, organization and+ project ids, client ids and secrets, and per project which funnels and breakdowns the+ data is built to show. `WORKTREE.md` repeats the important ones. The four projects+ (web, SaaS, shop, app) differ in identity: only identified projects have retention;+ see `packages/seed/README.md`.+- Direct database access: `bun run --filter @openpanel/db studio` (Prisma Studio on+ `$STUDIO_PORT`), or `psql "$DATABASE_URL"` / `clickhouse-client` / `redis-cli` with the+ URLs from your env.++## How to verify your work++Verify like a user would, then like CI would. Pick the rows that match what you changed;+a one-line fix does not need the full suite, a query or ingest change always does.++| You changed | Verify with | Also run |+|---|---|---|+| Dashboard UI (`apps/start`) | Open `$DASHBOARD_URL` with the Playwright MCP if you have one (`browser_*` tools; the local HTTPS certificate is self-signed, ignore that), log in with `.seed.json`, use the feature, **take a screenshot** (`browser_take_screenshot`; you get the image back — look at it) | `bunx vitest run` in `apps/start`, then add/adjust a spec in `apps/start/e2e/` and `bun run e2e` |+| tRPC procedure / REST route (`packages/core`) | Call it (recipes below) with seeded ids and check the JSON; for auth changes, try the wrong role too | `bun test --isolate` in `packages/core` for that module |+| ClickHouse query / chart / overview | Run old and new against the seeded data and diff results; check the numbers on the dashboard match a manual query | `packages/core` tests for the module; `bun run --filter @openpanel/db test` if you touched `packages/db` |+| Ingest / worker / buffers / sessions | `bun run send journey --project acme-shop` (live events through Kafka → worker → ClickHouse), then query `events`/`sessions`/`profiles` for that session id | `packages/core` ingest + buffer tests, `apps/api/e2e/` for session lifecycle |+| MCP tools (`packages/core/src/modules/mcp`) | Call the product's MCP against *your* API and data: as the `openpanel` MCP server if one is configured for you, else over HTTP (`.seed.json` → `mcp.url`, recipe below); compare its answer with a direct query | `packages/core` mcp tests |+| Schema / migration | `bun run --filter @openpanel/db migrate:deploy` on your isolated database, then `bun run seed --reset` and the seed tests (`bun run --filter @openpanel/seed test`) | `bun run typecheck` |+| Anything shared (`packages/shared`, `packages/db`, config, deps) | — | `bun run typecheck` and `bun run test` in full |++Always before you finish: `bun run typecheck`, and `npx ultracite fix <files you changed>`+(never bare or on `.` — the repo is not bulk-formatted). Report what you ran and what you saw;+a screenshot or a query result is evidence, "should work" is not.++Sending real events (`bun run send`, uses the seeded clients and `$API_URL`):++```bash+bun run send journey --project acme-shop --sessions 2 # replay a generated journey live+bun run send track purchase --project acme-shop --path /order/1 --prop value=99 --prop payment=card --revenue 99+bun run send identify usr_123 --project acme-saas --email [email protected] --first-name Ada --prop plan=pro+```++Request recipes (ids and secrets from `.seed.json`):++```bash+# Track through the real pipeline+curl -sS -X POST "$API_URL/track" \+ -H 'content-type: application/json' \+ -H "openpanel-client-id: $CLIENT_ID" -H "openpanel-client-secret: $CLIENT_SECRET" \+ -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/128.0 Safari/537.36' \+ -H 'x-client-ip: 81.2.69.142' \+ -d '{"type":"track","payload":{"name":"screen_view","properties":{"__path":"/pricing"}}}'++# Management API (root client)+curl -sS "$API_URL/manage/projects" -H "openpanel-client-id: $ROOT_ID" -H "openpanel-client-secret: $ROOT_SECRET"++# tRPC as the seeded user: sign in, keep the `session` cookie, call any procedure+curl -sS -c cookies.txt -X POST "$API_URL/trpc/auth.signInEmail" -H 'content-type: application/json' \+ -d '{"json":{"email":"[email protected]","password":"openpanel"}}'+curl -sS -b cookies.txt "$API_URL/trpc/organization.list"++# The product's MCP (streamable HTTP; url with token is `.seed.json` → mcp.url)+curl -sS -X POST "$MCP_URL" -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \+ -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'+```++Payload shapes for `/track` are in `packages/core/src/modules/ingest/ingest.constants.ts`.+`apps/api/e2e/` is an HTTP harness for session-lifecycle stress tests.++### Splitting verification across sub-agents++Default: verify your own change yourself; it is small and you have the context. Spawn+sub-agents only when the change touches several surfaces at once (API + dashboard + MCP,+or a query used in many reports) — then one sub-agent per surface, in parallel, each with a+concrete checklist ("open X, do Y, expect Z; report the numbers/screenshot"). They share+your worktree and databases, so they must not migrate, reseed or restart processes; only+you do that, before they start. Merge their reports into yours; do not forward them raw.++## How to find things++- A backend module is `packages/core/src/modules/<name>/` with `<name>.rpc.ts`+ (tRPC), `.routes.ts` (REST), `.jobs.ts` (BullMQ), `.service.ts`, `.constants.ts`+ (zod schemas + constants, safe to import from the frontend). `Cmd+P report.service`.+- Dashboard routes: `apps/start/src/routes/` (TanStack file routing); modals in+ `src/modals/`; tRPC client in `src/integrations/tanstack-query/`.+- ClickHouse queries use the `sql` tag from `@openpanel/db` (`packages/db/src/clickhouse/sql.ts`).+ Table names: `packages/db/src/clickhouse/client.ts` (`TABLE_NAMES`). Schema: the+ numbered files in `packages/db/src/code-migrations/`; a flat copy is `test/clickhouse-schema.sql`.+- Env schema: `apps/api/src/config/env.ts`. Auth: `packages/core/src/modules/auth/`.+- The write path (what ends up in ClickHouse): `packages/core/src/modules/event/event.service.ts`+ (`createEvent`), `packages/core/src/buffers/session-buffer.ts`, `profile-buffer.ts`.++## ClickHouse++Local is a single node; production is sharded (2 shards × 2 replicas, `Distributed`+tables). A plain `IN (subquery)` on a distributed table silently returns per-shard+results — use `GLOBAL IN` or a join. Never commit ClickHouse SQL you have not run+against your worktree's database; when replacing a query, diff old and new results+on the same data.++## Code style++Formatting and linting is ultracite (Biome), **only on the files you changed**. Do not+hand-format against it and do not disable rules to make a check pass.++- Constants over magic numbers: named, at the top of the file or in the module's+ `<name>.constants.ts`.+- Names reveal purpose; no abbreviations unless universal.+- Comments say *why*, briefly. Never narrate what the code does. If a function needs+ a comment to explain what it does, split it.+- Single responsibility, early returns, no nested ternaries, extract conditions into+ named booleans.+- DRY at the narrowest level that fits: the module first, `packages/core/src/shared/`+ when several backend modules need it, a dedicated package when the frontend or SDKs+ need it too. Do not invent a cross-module abstraction for two call sites.+- `unknown` over `any`; `as const`; narrowing over assertions; `for…of` over `forEach`.+- React: function components, hooks at top level, `key` from ids, ref as a prop+ (React 19), semantic elements with labels and keyboard handlers.+- No `console.log` / `debugger` left behind. Throw `Error` objects.++Leave the code you touch cleaner than you found it — inside the task's scope only.+Do not refactor, upgrade dependencies or fix unrelated things you notice; mention+them instead.++## Git++One branch per worktree, created from `rewrite/v2`. Commit when a step is done and+verified, with a message that says what and why. Never push, never add+remotes, never rewrite history that is not yours. `WORKTREE.md`, `.seed.json` and+`.worktree-*` are gitignored — keep them that way.++## Ralph++If `/home/deploy/rewrite-openpanel` exists you are the autonomous rewrite loop:+`CLAUDE.ralph.md` overrides this file.diff --git a/CLAUDE.md b/CLAUDE.mdindex d04d5c843..43c994c2d 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -1,111 +1 @@-# OpenPanel V2 — autonomous rewrite--This repository is being rewritten by an autonomous loop. Read this before doing-anything.--## Safety--Your world is exactly two directories: this repository and-`/home/deploy/rewrite-openpanel` (the controller). Do not read, list, or modify-anything else on this machine.--Never use production credentials — production does not exist on this machine,-and nothing here should reach it. Local Postgres / ClickHouse / Redis only.--## Environment--- **No docker.** `bun run dock:up` will fail. Postgres (5432), Redis (6379) and- ClickHouse (8123) are already running on localhost, started by another user.- Do not try to start, stop or install them.-- **No sudo.** Anything needing root is a human task. Say so; do not attempt it.-- Writes are confined to this repo and `/home/deploy/rewrite-openpanel`. Every- other path on the box is read-only to you.--## Commands--```bash-bun install-bun run codegen # prisma client; needs .env-bun run typecheck-bun run check # ultracite + check:deps-bun run test-```--`typecheck` and `test` fail until `.env` exists and `bun run codegen` has run.-If a task depends on them and they are not ready, report `BLOCKED` rather than-inventing a workaround.--## ClickHouse--Never commit ClickHouse SQL you have not executed. Run it against the local-server (`http://127.0.0.1:8123`, database `openpanel`) and report timing and row-count. When replacing a query, diff the old and new result sets on the same data.--Local is a single node; production is 2 shards x 2 replicas. Before editing any-query against a `Distributed` table read-`/home/deploy/rewrite-openpanel/docs/ENVIRONMENT.md` - a plain `IN (subquery)` on a-distributed table silently returns per-shard results.--## Architecture--Architecture decisions live in `/home/deploy/rewrite-openpanel/decisions`.-**Accepted ADRs are binding.** If you need a decision no accepted ADR covers,-stop and say `BLOCKED: needs ADR — <question>`. Do not decide it yourself.--The target architecture is `/home/deploy/rewrite-openpanel/docs/TARGET_ARCHITECTURE.md`.-The documented behaviour of the system you are replacing is in-`/home/deploy/rewrite-openpanel/docs/current/`. Read the relevant document before-changing that area — it exists so you do not have to rediscover it.--## Ralph--Do not modify:--```-/home/deploy/rewrite-openpanel/plan/tasks.json-/home/deploy/rewrite-openpanel/state-/home/deploy/rewrite-openpanel/verification-```--The orchestrator owns task state and owns the definition of "passing".--## Git--Do not commit. The orchestrator manages commits and the `rewrite/v2` branch.-There are no push credentials on this machine, by design - all commits are local-and a human pushes them later. Never try to add a remote or authenticate to one.--## Code style--- **Formatting and linting: ultracite** (the Biome preset) — but **only ever on- the files you changed**, named explicitly (`npx ultracite fix <files...>` or- `biome check --write <files...>`). NEVER run it bare or on `.`: the repo has- never been bulk-formatted, so an unscoped run rewrites thousands of files,- buries your real diff, and fails the task. (This is why format-on-save hooks- are disabled here.) Do not hand-format against it, and do not disable its- rules to make a task pass.--- **Constants over magic numbers** — named, descriptive, at the top of the file- or in the module's `<name>.constants.ts`.-- **Meaningful names** — reveal purpose; no abbreviations unless universal.-- **Smart comments** — never narrate what code does; comment only the why, and- keep it very short. APIs, non-obvious side effects and invariants excepted.-- **Single responsibility** — small, focused functions; if it needs a comment- to explain what it does, split it.-- **DRY** — single sources of truth, shared at the narrowest level that fits:- inside the module first; `core/shared/` when several backend modules need it;- a **dedicated package** when anything beyond the backend (frontend, SDKs,- other apps) needs it — core is not a grab-bag for cross-app code. Exception,- by accepted decision: module `*.constants.ts` files are consumed by the- frontend via deep paths into core, not via a package. Never invent a- cross-module abstraction just to deduplicate two call sites.-- **Clean structure & encapsulation** — related code together, implementation- hidden, nested conditionals extracted into well-named functions.-- **Leave the code you touch cleaner than you found it — inside the task's- scope only.** Refactoring outside scope is a task of its own; propose it,- don't do it. Same for technical debt you notice: record it, move on.--## Scope--Do exactly the assigned task. Do not opportunistically refactor, upgrade-dependencies, or fix unrelated things you notice along the way.diff --git a/CLAUDE.ralph.md b/CLAUDE.ralph.mdnew file mode 100644index 000000000..d04d5c843--- /dev/null+++ b/CLAUDE.ralph.md@@ -0,0 +1,111 @@+# OpenPanel V2 — autonomous rewrite++This repository is being rewritten by an autonomous loop. Read this before doing+anything.++## Safety++Your world is exactly two directories: this repository and+`/home/deploy/rewrite-openpanel` (the controller). Do not read, list, or modify+anything else on this machine.++Never use production credentials — production does not exist on this machine,+and nothing here should reach it. Local Postgres / ClickHouse / Redis only.++## Environment++- **No docker.** `bun run dock:up` will fail. Postgres (5432), Redis (6379) and+ ClickHouse (8123) are already running on localhost, started by another user.+ Do not try to start, stop or install them.+- **No sudo.** Anything needing root is a human task. Say so; do not attempt it.+- Writes are confined to this repo and `/home/deploy/rewrite-openpanel`. Every+ other path on the box is read-only to you.++## Commands++```bash+bun install+bun run codegen # prisma client; needs .env+bun run typecheck+bun run check # ultracite + check:deps+bun run test+```++`typecheck` and `test` fail until `.env` exists and `bun run codegen` has run.+If a task depends on them and they are not ready, report `BLOCKED` rather than+inventing a workaround.++## ClickHouse++Never commit ClickHouse SQL you have not executed. Run it against the local+server (`http://127.0.0.1:8123`, database `openpanel`) and report timing and row+count. When replacing a query, diff the old and new result sets on the same data.++Local is a single node; production is 2 shards x 2 replicas. Before editing any+query against a `Distributed` table read+`/home/deploy/rewrite-openpanel/docs/ENVIRONMENT.md` - a plain `IN (subquery)` on a+distributed table silently returns per-shard results.++## Architecture++Architecture decisions live in `/home/deploy/rewrite-openpanel/decisions`.+**Accepted ADRs are binding.** If you need a decision no accepted ADR covers,+stop and say `BLOCKED: needs ADR — <question>`. Do not decide it yourself.++The target architecture is `/home/deploy/rewrite-openpanel/docs/TARGET_ARCHITECTURE.md`.+The documented behaviour of the system you are replacing is in+`/home/deploy/rewrite-openpanel/docs/current/`. Read the relevant document before+changing that area — it exists so you do not have to rediscover it.++## Ralph++Do not modify:++```+/home/deploy/rewrite-openpanel/plan/tasks.json+/home/deploy/rewrite-openpanel/state+/home/deploy/rewrite-openpanel/verification+```++The orchestrator owns task state and owns the definition of "passing".++## Git++Do not commit. The orchestrator manages commits and the `rewrite/v2` branch.+There are no push credentials on this machine, by design - all commits are local+and a human pushes them later. Never try to add a remote or authenticate to one.++## Code style++- **Formatting and linting: ultracite** (the Biome preset) — but **only ever on+ the files you changed**, named explicitly (`npx ultracite fix <files...>` or+ `biome check --write <files...>`). NEVER run it bare or on `.`: the repo has+ never been bulk-formatted, so an unscoped run rewrites thousands of files,+ buries your real diff, and fails the task. (This is why format-on-save hooks+ are disabled here.) Do not hand-format against it, and do not disable its+ rules to make a task pass.++- **Constants over magic numbers** — named, descriptive, at the top of the file+ or in the module's `<name>.constants.ts`.+- **Meaningful names** — reveal purpose; no abbreviations unless universal.+- **Smart comments** — never narrate what code does; comment only the why, and+ keep it very short. APIs, non-obvious side effects and invariants excepted.+- **Single responsibility** — small, focused functions; if it needs a comment+ to explain what it does, split it.+- **DRY** — single sources of truth, shared at the narrowest level that fits:+ inside the module first; `core/shared/` when several backend modules need it;+ a **dedicated package** when anything beyond the backend (frontend, SDKs,+ other apps) needs it — core is not a grab-bag for cross-app code. Exception,+ by accepted decision: module `*.constants.ts` files are consumed by the+ frontend via deep paths into core, not via a package. Never invent a+ cross-module abstraction just to deduplicate two call sites.+- **Clean structure & encapsulation** — related code together, implementation+ hidden, nested conditionals extracted into well-named functions.+- **Leave the code you touch cleaner than you found it — inside the task's+ scope only.** Refactoring outside scope is a task of its own; propose it,+ don't do it. Same for technical debt you notice: record it, move on.++## Scope++Do exactly the assigned task. Do not opportunistically refactor, upgrade+dependencies, or fix unrelated things you notice along the way.diff --git a/apps/start/e2e/auth.setup.ts b/apps/start/e2e/auth.setup.tsnew file mode 100644index 000000000..35576a927--- /dev/null+++ b/apps/start/e2e/auth.setup.ts@@ -0,0 +1,21 @@+// Logs in once through the real form and saves the session cookie; every+// other spec starts authenticated (Playwright's storageState pattern).++import { expect, test as setup } from '@playwright/test';+import { readSeedManifest } from './seed-manifest';++const AUTH_STATE_FILE = 'e2e/.auth/user.json';++setup('sign in as the seeded user', async ({ page }) => {+ const manifest = readSeedManifest();+ await page.goto('/login');+ // The form is server-rendered; a click before hydration submits it natively as a GET.+ await page.waitForLoadState('networkidle');+ await page.getByLabel('Email').fill(manifest.user.email);+ await page.getByLabel('Password').fill(manifest.user.password);+ await page.getByRole('button', { name: 'Sign in', exact: true }).click();+ await expect(page).toHaveURL(new RegExp(`/${manifest.organizationId}`), {+ timeout: 15_000,+ });+ await page.context().storageState({ path: AUTH_STATE_FILE });+});diff --git a/apps/start/e2e/fixtures.ts b/apps/start/e2e/fixtures.tsnew file mode 100644index 000000000..6a522b65f--- /dev/null+++ b/apps/start/e2e/fixtures.ts@@ -0,0 +1,12 @@+import { test as base } from '@playwright/test';+import { readSeedManifest, type SeedManifest } from './seed-manifest';++export const test = base.extend<{ seed: SeedManifest }>({+ // Playwright requires the destructured first argument, even when unused.+ // biome-ignore lint/correctness/noEmptyPattern: Playwright fixture signature+ seed: async ({}, use) => {+ await use(readSeedManifest());+ },+});++export { expect } from '@playwright/test';diff --git a/apps/start/e2e/overview.spec.ts b/apps/start/e2e/overview.spec.tsnew file mode 100644index 000000000..a903975d1--- /dev/null+++ b/apps/start/e2e/overview.spec.ts@@ -0,0 +1,45 @@+import { expect, test } from './fixtures';+import { busiestProject } from './seed-manifest';++test('the organization lists the seeded projects', async ({ page, seed }) => {+ await page.goto(`/${seed.organizationId}`);+ for (const project of seed.projects) {+ await expect(page.getByText(project.name, { exact: true })).toBeVisible();+ }+});++test('the overview renders its metric cards for a seeded project', async ({+ page,+ seed,+}) => {+ const project = busiestProject(seed);+ await page.goto(`/${seed.organizationId}/${project.id}`);+ for (const metric of [+ 'Unique Visitors',+ 'Sessions',+ 'Pageviews',+ 'Bounce Rate',+ ]) {+ await expect(+ page.getByRole('button', { name: new RegExp(`^${metric}`) })+ ).toBeVisible();+ }+});++test('sessions and profiles pages list seeded rows', async ({ page, seed }) => {+ const project = busiestProject(seed);+ const base = `/${seed.organizationId}/${project.id}`;++ await page.goto(`${base}/sessions`);+ await expect(+ page.getByRole('heading', { name: 'Sessions', level: 1 })+ ).toBeVisible();+ await expect(+ page.locator(`a[href^="${base}/sessions/"]`).first()+ ).toBeVisible();++ await page.goto(`${base}/profiles`);+ await expect(+ page.locator(`a[href^="${base}/profiles/"]`).first()+ ).toBeVisible();+});diff --git a/apps/start/e2e/seed-manifest.ts b/apps/start/e2e/seed-manifest.tsnew file mode 100644index 000000000..411c5ae64--- /dev/null+++ b/apps/start/e2e/seed-manifest.ts@@ -0,0 +1,52 @@+// The seed's `.seed.json` at the repo root: who to log in as and which+// projects carry data. `bun run seed` writes it.++import { existsSync, readFileSync } from 'node:fs';+import { resolve } from 'node:path';++export interface SeedManifest {+ user: { email: string; password: string };+ organizationId: string;+ rootClient: { id: string; secret: string };+ projects: {+ id: string;+ name: string;+ archetype: string;+ clientId: string;+ clientSecret: string;+ sessions: number;+ events: number;+ }[];+ range: { from: string; to: string };+}++const MANIFEST_PATH = resolve(+ import.meta.dirname,+ '..',+ '..',+ '..',+ '.seed.json'+);++export function readSeedManifest(): SeedManifest {+ if (!existsSync(MANIFEST_PATH)) {+ throw new Error(+ `No ${MANIFEST_PATH}. Run \`bun run seed\` first; the e2e suite logs in with what it seeded.`+ );+ }+ return JSON.parse(readFileSync(MANIFEST_PATH, 'utf8')) as SeedManifest;+}++/** The seeded project with the most sessions; the overview specs look at it. */+export function busiestProject(+ manifest: SeedManifest+): SeedManifest['projects'][number] {+ const [first, ...rest] = manifest.projects;+ if (!first) {+ throw new Error('.seed.json lists no projects');+ }+ return rest.reduce(+ (best, project) => (project.sessions > best.sessions ? project : best),+ first+ );+}diff --git a/apps/start/package.json b/apps/start/package.jsonindex 03f31c1b7..5a65133bd 100644--- a/apps/start/package.json+++ b/apps/start/package.json@@ -10,6 +10,7 @@"build": "bun run with-env vite build","serve": "vite preview","test": "vitest run",+ "e2e": "playwright test","format": "biome format","lint": "biome lint","check": "biome check",@@ -161,6 +162,7 @@"@biomejs/biome": "1.9.4","@cloudflare/vite-plugin": "1.33.2","@openpanel/db": "workspace:*",+ "@playwright/test": "1.63.0","@tanstack/devtools-event-client": "0.3.3","@testing-library/dom": "10.4.1","@testing-library/react": "16.3.0",diff --git a/apps/start/playwright.config.ts b/apps/start/playwright.config.tsnew file mode 100644index 000000000..bd6ff1cb3--- /dev/null+++ b/apps/start/playwright.config.ts@@ -0,0 +1,30 @@+// Runs against an already-running dashboard (a worktree's processes are supervised+// for it), so there is no `webServer` block. `DASHBOARD_URL` names it; a plain+// checkout sets it or falls back to the dev port.++import { defineConfig, devices } from '@playwright/test';++const DEV_DASHBOARD_URL = 'http://localhost:3000';+const AUTH_STATE_FILE = 'e2e/.auth/user.json';++export default defineConfig({+ testDir: 'e2e',+ fullyParallel: false,+ workers: 1,+ retries: process.env.CI ? 1 : 0,+ reporter: process.env.CI ? 'github' : 'list',+ use: {+ baseURL: process.env.DASHBOARD_URL ?? DEV_DASHBOARD_URL,+ // A worktree's HTTPS certificate is from a local CA Playwright's Chromium does not trust.+ ignoreHTTPSErrors: true,+ trace: 'retain-on-failure',+ },+ projects: [+ { name: 'setup', testMatch: /auth\.setup\.ts/ },+ {+ name: 'chromium',+ use: { ...devices['Desktop Chrome'], storageState: AUTH_STATE_FILE },+ dependencies: ['setup'],+ },+ ],+});diff --git a/bun.lock b/bun.lockindex c833506e6..952ba2146 100644--- a/bun.lock+++ b/bun.lock@@ -238,6 +238,7 @@"@biomejs/biome": "1.9.4","@cloudflare/vite-plugin": "1.33.2","@openpanel/db": "workspace:*",+ "@playwright/test": "1.63.0","@tanstack/devtools-event-client": "0.3.3","@testing-library/dom": "10.4.1","@testing-library/react": "16.3.0",@@ -553,6 +554,21 @@"typescript": "catalog:",},},+ "packages/seed": {+ "name": "@openpanel/seed",+ "version": "0.0.1",+ "dependencies": {+ "@node-rs/argon2": "2.0.2",+ "@openpanel/db": "workspace:*",+ "@openpanel/shared": "workspace:*",+ },+ "devDependencies": {+ "@openpanel/tsconfig": "workspace:*",+ "@types/node": "catalog:",+ "bun-types": "1.4.0",+ "typescript": "catalog:",+ },+ },"packages/shared": {"name": "@openpanel/shared","version": "0.0.1",@@ -1579,6 +1595,8 @@"@openpanel/sdk-info": ["@openpanel/sdk-info@workspace:packages/sdks/_info"],+ "@openpanel/seed": ["@openpanel/seed@workspace:packages/seed"],+"@openpanel/shared": ["@openpanel/shared@workspace:packages/shared"],"@openpanel/testbed": ["@openpanel/testbed@workspace:apps/testbed"],@@ -1903,6 +1921,8 @@"@pkgjs/parseargs": ["@pkgjs/[email protected]", "", {}, "sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg=="],+ "@playwright/test": ["@playwright/[email protected]", "", { "dependencies": { "playwright": "1.63.0" }, "bin": { "playwright": "cli.js" } }, "sha512-oxMK4vllB9RK5NQ2l1pq1IfOf2AvnEuj/vYGDj0H2nMtmtZpKtCwt/l00GEO6xjGfpBNAvjovvYdCm50dRQkpQ=="],+"@polar-sh/sdk": ["@polar-sh/[email protected]", "", { "dependencies": { "standardwebhooks": "^1.0.0", "zod": "^3.25.65 || ^4.0.0" } }, "sha512-9UYb70iKjJCtWYlu0OF5HLYBLmkxHwqr2RlXwuxXQgRGqq56IQWlVG+NO7e1YJ7I5GW0CBHhGIjRbQ9hYM6ycQ=="],"@polka/url": ["@polka/[email protected]", "", {}, "sha512-wwQAWhWSuHaag8c4q/KN/vCoeOJYshAIvMQwD4GpSb3OiZklFfvAgmj0VCBBImRpuF/aFgIRzllXlVX93Jevww=="],@@ -4851,6 +4871,10 @@"pkg-types": ["[email protected]", "", { "dependencies": { "confbox": "^0.3.1", "exsolve": "^1.1.1", "pathe": "^2.0.3" } }, "sha512-j/lCFdcppV0JxWpCEITdbDltBxPP6cHT+yNJ6Go2OgoSA9518X847X9z0p6LtA4Nc16+eQzCZjRrWanTGvHJ5w=="],+ "playwright": ["[email protected]", "", { "dependencies": { "playwright-core": "1.63.0" }, "bin": { "playwright": "cli.js" } }, "sha512-+7ziBLidS4NaNCdt57SUDT+wYmmd5fmiQejUic/kb+YsYSCPyOOE9sebzMjNmQrsnNpDJqd4WHvV/8lfKfUDUg=="],++ "playwright-core": ["[email protected]", "", { "bin": { "playwright-core": "cli.js" } }, "sha512-rYCsBF/M5HjUch52bbtVONEFjv6Xu8sm8h72dNlR5bzIE1fvC/bxgspzkjSfU+MweEMmPM8KJebG6nnyxo5mCg=="],+"please-upgrade-node": ["[email protected]", "", { "dependencies": { "semver-compare": "^1.0.0" } }, "sha512-gQR3WpIgNIKwBMVLkpMUeR3e1/E1y42bqDQZfql+kDeXd8COYfM8PQA4X6y7a8u9Ua9FHmsrrmirW2vHs45hWg=="],"plist": ["[email protected]", "", { "dependencies": { "@xmldom/xmldom": "^0.9.10", "base64-js": "^1.5.1", "xmlbuilder": "^15.1.1" } }, "sha512-ZIfcLJC+7E7FBFnDxm9MPmt7D+DidyQ26lewieO75AdhA2ayMtsJSES0iWzqJQbcVRSrTufQoy0DR94xHue0oA=="],diff --git a/hub.config.ts b/hub.config.tsindex 70c17639d..a4f030a6f 100644--- a/hub.config.ts+++ b/hub.config.ts@@ -1,3 +1,17 @@+import { readFileSync } from 'node:fs';+import { join } from 'node:path';++/** The product's MCP on this worktree's API, authenticated with the seed's root client. */+function seededMcp(w: { dir: string; url(process: string): string }): Record<string, { url: string }> {+ try {+ const seed = JSON.parse(readFileSync(join(w.dir, '.seed.json'), 'utf8'));+ if (typeof seed?.mcp?.token !== 'string') return {};+ return { openpanel: { url: `${w.url('api')}/mcp?token=${seed.mcp.token}` } };+ } catch {+ return {};+ }+}+// The one file the hub reads from this repo. Its shape is `hub.config.d.ts` in the hub repo,// which type-checks this file in its own tests; a field the hub does not know fails there.export default {@@ -6,7 +20,7 @@ export default {worktreesDir: '.worktrees',github: 'Openpanel-dev/openpanel',domains: { local: 'local.openpanel.cc', remote: 'openpanel.cc' },- install: 'bun install && bun run --filter @openpanel/db codegen',+ install: 'bun install && bun run --filter @openpanel/db codegen && bunx playwright install chromium',// port = 21000 + block×100 + offset. Main checkout is block 0.processes: {@@ -18,6 +32,10 @@ export default {// Injected as process env. Wins over .env because dotenv-cli runs without -o.env: (w) => ({+ // The dashboard's SSR runs under the Cloudflare Vite plugin, whose worker only sees wrangler+ // vars and .env.local unless told to take the process env; without this it would SSR+ // against production's API_URL.+ CLOUDFLARE_INCLUDE_PROCESS_ENV: 'true',API_PORT: w.port('api'),API_HOST: '127.0.0.1', // Bun.serve defaults to 0.0.0.0WEB_PORT: w.port('web'),@@ -48,6 +66,12 @@ export default {redis: { index: (block) => block }, // always per worktreekafka: { topics: (n) => [`events-${n}`, `events-${n}-dlq`] },migrate: 'bun run --filter @openpanel/db migrate:deploy',+ // Presets of packages/seed; each writes .seed.json and prints the logins it created.+ seed: {+ small: 'bun run --filter @openpanel/seed seed -- --size small --reset',+ medium: 'bun run --filter @openpanel/seed seed -- --size medium --reset',+ large: 'bun run --filter @openpanel/seed seed -- --size large --reset',+ },},checks: {@@ -65,6 +89,16 @@ export default {kafka: (w) => `docker compose -f ${w.root}/docker-compose.yml exec op-rp sh -c 'rpk topic list; echo; echo "this worktree: ${w.db.kafkaTopics.join(' ')}"; exec sh'`,},+ // Every agent the hub launches can drive the dashboard in a real browser (--ignore-https-errors+ // only skips certificate checks for the hub's local CA; cookies and login are unaffected), and+ // talks to this worktree's own OpenPanel MCP once the seed has written its root-client token.+ agents: {+ mcp: (w) => ({+ playwright: { command: 'npx', args: ['-y', '@playwright/mcp@latest', '--isolated', '--ignore-https-errors'] },+ ...seededMcp(w),+ }),+ },+stack: { up: 'docker compose up -d op-db op-kv op-ch op-rp', cwd: '.' },auth: process.platform === 'darwin' ? 'off' : 'password',issues: { github: true, userjot: true },diff --git a/package.json b/package.jsonindex f503e2964..5ea4f119c 100644--- a/package.json+++ b/package.json@@ -52,7 +52,7 @@"vue": "3.5.25"},"scripts": {- "test": "bun run --filter @openpanel/core test && bun run --filter @openpanel/db test && bun run --filter @openpanel/api test && bun run --filter start test && vitest run",+ "test": "bun run --filter @openpanel/core test && bun run --filter @openpanel/db test && bun run --filter @openpanel/api test && bun run --filter @openpanel/seed test && bun run --filter start test && vitest run","gen:bots": "bun run --filter @openpanel/core gen:bots","gen:referrers": "bun run --filter @openpanel/core gen:referrers","dock:up": "docker compose up -d",@@ -63,6 +63,9 @@"codegen": "bun run --filter @openpanel/db codegen && bun run --filter @openpanel/core codegen","migrate": "bun run --filter @openpanel/db migrate","migrate:deploy": "bun run --filter @openpanel/db migrate:deploy",+ "seed": "bun run --filter @openpanel/seed seed --",+ "send": "bun run --filter @openpanel/seed send --",+ "e2e": "bun run --filter start e2e --","postinstall:workspaces": "bun run --filter public postinstall && bun run --filter @openpanel/testbed postinstall","dev": "bun run --filter @openpanel/api testing & bun run --filter start dev & wait","dev:public": "bun run --filter public dev",diff --git a/packages/seed/README.md b/packages/seed/README.mdnew file mode 100644index 000000000..53b6b5dde--- /dev/null+++ b/packages/seed/README.md@@ -0,0 +1,60 @@+# @openpanel/seed++Deterministic seed data for a worktree: one login, one organization, four projects+with realistic traffic in ClickHouse. `bun run seed` from the repo root.++```+bun run seed # small: 14 days, ~5k sessions+bun run seed --size large --reset # 90 days, ~1M sessions; --reset clears earlier seeded rows+bun run seed --days 3 --sessions-per-day 500 --projects saas,app --seed 7+bun run seed --dry-run # the plan only+```++Same flags + seed ⇒ identical rows.++`bun run send` sends *live* requests to `$API_URL` with the seeded clients — through the real+ingest pipeline, unlike the seed's direct inserts:++```+bun run send journey --project acme-shop --sessions 2 # generate a journey with the archetype, replay it+bun run send track purchase --project acme-shop --path /order/1 --prop value=99 --revenue 99+bun run send identify usr_123 --project acme-saas --email [email protected] --prop plan=pro+``` Output: `.seed.json` at the repo root (login,+ids, client secrets, the product's MCP endpoint with a root-client token, and the funnels+below), for agents and the e2e suite.++## What the data is built to show++| Project | Identity | Funnels (breakdowns) | Cohort-worthy behaviours |+|---|---|---|---|+| **Acme Web** `acme-web` (website) | Anonymous; a device is new every UTC day (ingest's salted id). ~18 % sign up and are identified from then on. | Signup: `screen_view → plan_selected → signup_started → signup_completed` (referrer, device, country, plan). Demo: `cta_clicked → demo_requested` | signed up, requested demo, subscribed to newsletter |+| **Acme SaaS** `acme-saas` (logged-in product) | Everyone signs up/logs in on the first session ⇒ real retention. | Activation: `signup_completed → project_created → sdk_installed → first_event_received` (signup_source, company_size, role, sdk). Upgrade: `pricing_viewed → checkout_started → subscription_started` (plan, interval) | invited a teammate, exported, connected an integration, paid plan, churned — each changes retention |+| **Acme Shop** `acme-shop` (ecommerce) | Anonymous shoppers; ~40 % of buyers create an account at first purchase. | Checkout: `product_viewed → add_to_cart → checkout_started → shipping_info_added → payment_info_added → purchase` (category, brand, payment, shipping, coupon, device). Search: `search → product_viewed → add_to_cart` | has purchased, has account, used coupon, wishlisted; `purchase.first_purchase` |+| **Acme App** `acme-app` (React Native) | ~85 % sign up during onboarding. | Onboarding: `app_opened → onboarding_step_completed → signup_completed → notification_permission → post_created` (platform, app_version). Premium: `paywall_viewed → trial_started → subscription_started` (trigger, interval) | notifications granted, premium, has posted, platform |++Conversions (`EventMeta.conversion`) are flagged per project. Revenue events carry+whole-unit amounts (`events.revenue` is `UInt64`).++## How it is generated++- **Traffic**: per archetype an hourly curve, weekday curve, slow trend, an AR(1)+ drift, rare spikes/dips; Poisson per hour (`traffic.ts`).+- **People**: a per-project pool bucketed by first-seen day. A returning session picks+ a cohort through the archetype's retention curve (`sameDay`, `day1`, `day7`,+ `day30`) and a visitor through its stickiness (paid plan, notifications, has+ purchased…). Anonymous device ids rotate at UTC midnight (`world.ts`).+- **Journeys**: one state machine per archetype (`archetypes/*.ts`) with log-normal+ dwell times, state that persists across visits (cart, plan, onboarding step) and+ `identify()` at signup/login/purchase.+- **Rows**: shaped like the worker's (`rows.ts`) — `session_start`/`session_end`+ synthesized, one collapsed `sessions` row, one `profiles` row per device window or+ person. Devices/browsers come from real user agents through the real parser.+- **Isolation**: depends on `@openpanel/db` and `@openpanel/shared` only; row shapes are+ copied from core and checked against `test/clickhouse-schema.sql` by+ `clickhouse-rows.test.ts`.++## Not modelled (yet)++Bots, server-side events, late/duplicate events, groups/aliases, imports, geo-local+time of day, and very high property cardinality. For volume beyond `xl`, the plan is a+ClickHouse-side `--amplify` that copies a seeded slice with shifted dates.diff --git a/packages/seed/package.json b/packages/seed/package.jsonnew file mode 100644index 000000000..418ffcb5c--- /dev/null+++ b/packages/seed/package.json@@ -0,0 +1,25 @@+{+ "name": "@openpanel/seed",+ "version": "0.0.1",+ "private": true,+ "type": "module",+ "sideEffects": false,+ "scripts": {+ "seed": "bun run with-env bun src/cli.ts",+ "test": "bun test --isolate",+ "typecheck": "tsc --noEmit",+ "with-env": "dotenv -e ../../.env -c --",+ "send": "bun run with-env bun src/send.ts"+ },+ "dependencies": {+ "@node-rs/argon2": "2.0.2",+ "@openpanel/db": "workspace:*",+ "@openpanel/shared": "workspace:*"+ },+ "devDependencies": {+ "@openpanel/tsconfig": "workspace:*",+ "@types/node": "catalog:",+ "bun-types": "1.4.0",+ "typescript": "catalog:"+ }+}diff --git a/packages/seed/src/archetypes/app.ts b/packages/seed/src/archetypes/app.tsnew file mode 100644index 000000000..18aab2aa8--- /dev/null+++ b/packages/seed/src/archetypes/app.ts@@ -0,0 +1,319 @@+// Acme App: a native social app (React Native SDK). Most users sign up on+// the first open and are identified from then on. Funnels: onboarding+// (app_opened → onboarding → signup → notification permission → first post)+// and monetization (paywall_viewed → trial_started → subscription_started).+// Cohort-worthy: platform, notifications granted, premium, has posted.++import { newPerson } from '../data/people';+import type { Visitor } from '../model';+import type { Rng } from '../rng';+import type { Archetype, Journey } from './archetype';++const SCREENS = {+ home: ['Home', 'Home'],+ feed: ['Feed', 'Feed'],+ post: ['Post', 'Post'],+ create: ['CreatePost', 'New post'],+ profile: ['Profile', 'Profile'],+ settings: ['Settings', 'Settings'],+ onboarding: ['Onboarding', 'Onboarding'],+ paywall: ['Paywall', 'Go premium'],+ search: ['Search', 'Search'],+} as const;++const POST_CATEGORIES = [+ 'travel',+ 'food',+ 'fitness',+ 'tech',+ 'music',+ 'art',+ 'pets',+] as const;+const MEDIA = ['photo', 'video', 'text'] as const;+const SHARE_CHANNELS = [+ 'message',+ 'copy_link',+ 'instagram',+ 'external',+] as const;+const PUSH_CAMPAIGNS = [+ 'daily-digest',+ 'new-follower',+ 'trending',+ 'comment-reply',+] as const;+const OPEN_SOURCES = ['icon', 'push', 'deeplink', 'widget'] as const;+const APP_VERSIONS = ['3.4.0', '3.4.1', '3.5.0', '3.5.2', '3.6.0'] as const;+const PAYWALL_TRIGGERS = [+ 'post_limit',+ 'hd_upload',+ 'analytics',+ 'settings',+] as const;+/** Post ids are drawn from a large pool so `post_viewed.post_id` breaks down like a real feed. */+const POST_POOL = 50_000;++const IDENTIFIED_SHARE = 0.85;+const SIGNUP_COMPLETED_CHANCE = 0.8;+const NOTIFICATION_GRANTED_CHANCE = 0.55;+const PUSH_OPEN_CHANCE = 0.18;+const POST_CHANCE = 0.55;+const LIKE_CHANCE = 0.5;+const COMMENT_CHANCE = 0.12;+const SHARE_CHANCE = 0.08;+const CREATE_POST_CHANCE = 0.12;+const SEARCH_CHANCE = 0.15;+const PAYWALL_CHANCE = 0.06;+const TRIAL_CHANCE = 0.35;+const SUBSCRIBE_CHANCE = 0.5;+const PREMIUM_MONTHLY_PRICE = 5;+const PREMIUM_YEARLY_PRICE = 40;++interface AppState {+ onboarded?: boolean;+ notifications?: boolean;+ posts?: number;+ trial?: boolean;+ premium?: boolean;+ appVersion?: string;+}++function stateOf(visitor: Visitor): AppState {+ return visitor.state as AppState;+}++function platformOf(visitor: Visitor): string {+ return visitor.device.os === 'iOS' ? 'ios' : 'android';+}++function onboarding(+ rng: Rng,+ journey: Journey,+ visitor: Visitor,+ state: AppState+): boolean {+ journey.view(...SCREENS.onboarding, { step: 1 });+ journey.event('onboarding_step_completed', { step: 1, name: 'welcome' });+ journey.view(...SCREENS.onboarding, { step: 2 });+ journey.event('onboarding_step_completed', {+ step: 2,+ name: 'interests',+ interests: rng.int(1, 5),+ });+ if (visitor.person && rng.chance(SIGNUP_COMPLETED_CHANCE)) {+ const method = rng.one(['apple', 'google', 'email']);+ journey.event('signup_started', { method });+ journey.identify(visitor.person);+ journey.event('signup_completed', { method, ...visitor.person.properties });+ } else if (visitor.person) {+ return false;+ } else {+ journey.event('signup_skipped');+ }+ const granted = rng.chance(NOTIFICATION_GRANTED_CHANCE);+ journey.event('notification_permission', { granted });+ state.notifications = granted;+ state.onboarded = true;+ return true;+}++function feed(rng: Rng, journey: Journey, state: AppState): void {+ journey.view(...SCREENS.feed);+ const scrolls = rng.int(1, 6);+ for (let i = 0; i < scrolls; i++) {+ if (!rng.chance(POST_CHANCE)) {+ continue;+ }+ const postId = `p_${rng.int(1, POST_POOL)}`;+ const category = rng.one(POST_CATEGORIES);+ journey.view(...SCREENS.post, { post_id: postId });+ journey.event('post_viewed', {+ post_id: postId,+ category,+ media: rng.one(MEDIA),+ author_id: `usr_${rng.hex(6)}`,+ });+ if (rng.chance(LIKE_CHANCE)) {+ journey.event('post_liked', { post_id: postId, category });+ }+ if (rng.chance(COMMENT_CHANCE)) {+ journey.event('comment_posted', {+ post_id: postId,+ length: rng.int(5, 200),+ });+ }+ if (rng.chance(SHARE_CHANCE)) {+ journey.event('post_shared', {+ post_id: postId,+ channel: rng.one(SHARE_CHANNELS),+ });+ }+ }+ if (rng.chance(SEARCH_CHANCE)) {+ journey.view(...SCREENS.search);+ journey.event('search', {+ query: rng.one(POST_CATEGORIES),+ results: rng.int(0, 50),+ });+ }+ if (rng.chance(CREATE_POST_CHANCE)) {+ journey.view(...SCREENS.create);+ const media = rng.one(MEDIA);+ journey.event('post_created', {+ media,+ category: rng.one(POST_CATEGORIES),+ first_post: (state.posts ?? 0) === 0,+ });+ state.posts = (state.posts ?? 0) + 1;+ }+}++function monetization(rng: Rng, journey: Journey, state: AppState): void {+ if (state.premium || !rng.chance(PAYWALL_CHANCE)) {+ return;+ }+ const trigger = rng.one(PAYWALL_TRIGGERS);+ journey.view(...SCREENS.paywall, { trigger });+ journey.event('paywall_viewed', { trigger });+ if (!state.trial) {+ if (!rng.chance(TRIAL_CHANCE)) {+ return;+ }+ journey.event('trial_started', { plan: 'premium', days: 7, trigger });+ state.trial = true;+ return;+ }+ if (!rng.chance(SUBSCRIBE_CHANCE)) {+ return;+ }+ const interval = rng.one(['monthly', 'yearly']);+ const price =+ interval === 'yearly' ? PREMIUM_YEARLY_PRICE : PREMIUM_MONTHLY_PRICE;+ journey.event('subscription_started', {+ plan: 'premium',+ interval,+ price,+ trigger,+ });+ journey.revenue(price, { plan: 'premium', interval });+ state.premium = true;+}++export const appArchetype: Archetype = {+ id: 'app',+ projectName: 'Acme App',+ origin: '',+ domain: null,+ types: ['app'],+ sdk: { name: 'react-native', version: '1.0.3' },+ shape: {+ hourly: [+ 2, 1.2, 0.7, 0.4, 0.3, 0.5, 1.5, 3.5, 4, 3.5, 3, 3.5, 4.5, 4, 3.5, 3.5, 4,+ 5, 6, 7, 7.5, 7, 5.5, 3.5,+ ],+ dayOfWeek: [1.2, 0.95, 0.9, 0.9, 0.95, 1.05, 1.25],+ trendPerWeek: 0.03,+ spikeChance: 0.03,+ dipChance: 0.02,+ spikeRange: [1.5, 3],+ dipRange: [0.5, 0.8],+ },+ trafficShare: 0.1,+ returningShare: 0.65,+ retention: { sameDay: 0.45, day1: 0.45, day7: 0.25, day30: 0.15 },+ stickiness(visitor) {+ const state = stateOf(visitor);+ let value = visitor.identified ? 1.5 : 0.7;+ if (state.notifications) {+ value += 1;+ }+ if (state.premium) {+ value += 1.5;+ }+ if ((state.posts ?? 0) > 0) {+ value += 1;+ }+ return value;+ },+ mobileShare: 1,+ deviceClass: 'native',+ referrers: [{ weight: 1, value: null }],+ campaignChance: 0,+ dwellMedianSeconds: 12,+ utcOffsetHours: -5,+ newPerson(rng) {+ if (!rng.chance(IDENTIFIED_SHARE)) {+ return null;+ }+ const person = newPerson(rng, { business: false });+ person.properties = { plan: 'free', interests: String(rng.int(1, 5)) };+ return person;+ },+ identity:+ '~85% sign up during onboarding and are identified from then on; the rest stay anonymous devices.',+ conversions: ['signup_completed', 'subscription_started', 'post_created'],+ funnels: [+ {+ name: 'Onboarding',+ steps: [+ 'app_opened',+ 'onboarding_step_completed',+ 'signup_completed',+ 'notification_permission',+ 'post_created',+ ],+ breakdowns: ['platform', 'app_version', 'method', 'granted'],+ },+ {+ name: 'Premium',+ steps: ['paywall_viewed', 'trial_started', 'subscription_started'],+ breakdowns: ['trigger', 'platform', 'interval'],+ },+ ],+ journey(rng, journey, visitor) {+ const state = stateOf(visitor);+ state.appVersion ??= rng.one(APP_VERSIONS);+ const platform = platformOf(visitor);+ const common = { platform, app_version: state.appVersion };++ if (state.notifications && rng.chance(PUSH_OPEN_CHANCE)) {+ journey.event('push_opened', {+ campaign: rng.one(PUSH_CAMPAIGNS),+ ...common,+ });+ journey.event('app_opened', { source: 'push', ...common });+ } else {+ journey.event('app_opened', {+ source: rng.one(OPEN_SOURCES.filter((source) => source !== 'push')),+ ...common,+ });+ }+ journey.view(...SCREENS.home);++ if (!(state.onboarded || onboarding(rng, journey, visitor, state))) {+ return;+ }+ feed(rng, journey, state);+ if (rng.chance(0.2)) {+ journey.view(...SCREENS.profile);+ }+ if (rng.chance(0.08)) {+ journey.view(...SCREENS.settings);+ }+ if (visitor.identified) {+ monetization(rng, journey, state);+ if (visitor.person) {+ visitor.person.properties.plan = state.premium+ ? 'premium'+ : state.trial+ ? 'trial'+ : 'free';+ visitor.person.properties.platform = platform;+ visitor.person.properties.notifications = String(+ state.notifications ?? false+ );+ }+ }+ },+};diff --git a/packages/seed/src/archetypes/archetype.ts b/packages/seed/src/archetypes/archetype.tsnew file mode 100644index 000000000..7fd626e77--- /dev/null+++ b/packages/seed/src/archetypes/archetype.ts@@ -0,0 +1,169 @@+import type { ReferrerMix } from '../data/referrers';+import type { Person, Visitor } from '../model';+import type { Rng } from '../rng';+import type { TrafficShape } from '../traffic';++export type ArchetypeId = 'website' | 'saas' | 'ecommerce' | 'app';++export type ProjectType = 'website' | 'app' | 'backend';++/**+ * Share of a cohort that comes back after n days. Piecewise exponential+ * between the points; `sameDay` is a second visit within the first day.+ */+export interface RetentionCurve {+ sameDay: number;+ day1: number;+ day7: number;+ day30: number;+}++export interface SeedFunnel {+ name: string;+ steps: readonly string[];+ /** Properties worth breaking the funnel down by. */+ breakdowns: readonly string[];+}++export interface Archetype {+ id: ArchetypeId;+ projectName: string;+ /** The site the events come from; `''` for a native app (screens are named routes). */+ origin: string;+ domain: string | null;+ types: ProjectType[];+ sdk: { name: string; version: string };+ shape: TrafficShape;+ /** This project's share of the run's sessions per day. */+ trafficShare: number;+ /** Share of sessions that come from a visitor seen before. */+ returningShare: number;+ retention: RetentionCurve;+ /** Relative likelihood that this visitor is the one who comes back; 1 is the baseline. */+ stickiness(visitor: Visitor): number;+ mobileShare: number;+ deviceClass: 'web' | 'native';+ referrers: ReferrerMix;+ campaignChance: number;+ dwellMedianSeconds: number;+ /** Shifts the hourly curve so a European site peaks in European daytime. */+ utcOffsetHours: number;+ /** The person behind a visitor who will identify at some point; null for a never-identified one. */+ newPerson(rng: Rng): Person | null;+ /** One line on who gets identified and when, for `.seed.json`. */+ identity: string;+ /** Events the dashboard should treat as conversions (`EventMeta.conversion`). */+ conversions: readonly string[];+ /** The funnels this project's data is built to show, for `.seed.json` and whoever builds reports. */+ funnels: readonly SeedFunnel[];+ journey(rng: Rng, journey: Journey, visitor: Visitor): void;+}++const CUSTOM_EVENT_GAP_MEDIAN_SECONDS = 3;+const CUSTOM_EVENT_GAP_SIGMA = 0.7;+const REVENUE_GAP_MEDIAN_SECONDS = 2;+const REVENUE_GAP_SIGMA = 0.5;+const DWELL_SIGMA = 0.8;+const MS_PER_SECOND = 1000;++export interface JourneyEvent {+ name: string;+ path: string;+ title: string;+ at: Date;+ properties: Record<string, unknown>;+ revenue?: number;+ /** Who the event belongs to once identified; null while anonymous. */+ person: Person | null;+}++/** Appends events while advancing a clock, so timestamps read like a person browsing. */+export class Journey {+ readonly events: JourneyEvent[] = [];+ private cursor: Date;+ private person: Person | null;++ constructor(+ private readonly rng: Rng,+ start: Date,+ private readonly dwellMedianSeconds: number,+ person: Person | null+ ) {+ this.cursor = start;+ this.person = person;+ }++ get identified(): Person | null {+ return this.person;+ }++ /** From here on events carry this person; what `op.identify()` does in the SDK. */+ identify(person: Person): void {+ this.person = person;+ }++ view(+ path: string,+ title: string,+ properties: Record<string, unknown> = {}+ ): void {+ this.events.push({+ name: 'screen_view',+ path,+ title,+ at: this.cursor,+ properties,+ person: this.person,+ });+ this.advance(+ this.rng.logNormalSeconds(this.dwellMedianSeconds, DWELL_SIGMA)+ );+ }++ event(name: string, properties: Record<string, unknown> = {}): void {+ this.advance(+ this.rng.logNormalSeconds(+ CUSTOM_EVENT_GAP_MEDIAN_SECONDS,+ CUSTOM_EVENT_GAP_SIGMA+ )+ );+ this.events.push({+ name,+ ...this.currentScreen(),+ at: this.cursor,+ properties,+ person: this.person,+ });+ }++ /** Amount in whole currency units, integer: `events.revenue` is UInt64. */+ revenue(amount: number, properties: Record<string, unknown> = {}): void {+ this.advance(+ this.rng.logNormalSeconds(REVENUE_GAP_MEDIAN_SECONDS, REVENUE_GAP_SIGMA)+ );+ this.events.push({+ name: 'revenue',+ ...this.currentScreen(),+ at: this.cursor,+ properties,+ revenue: amount,+ person: this.person,+ });+ }++ private advance(seconds: number): void {+ this.cursor = new Date(+ this.cursor.getTime() + Math.round(seconds * MS_PER_SECOND)+ );+ }++ private currentScreen(): { path: string; title: string } {+ for (let i = this.events.length - 1; i >= 0; i--) {+ const event = this.events[i];+ if (event?.name === 'screen_view') {+ return { path: event.path, title: event.title };+ }+ }+ return { path: '', title: '' };+ }+}diff --git a/packages/seed/src/archetypes/ecommerce.ts b/packages/seed/src/archetypes/ecommerce.tsnew file mode 100644index 000000000..614ab5161--- /dev/null+++ b/packages/seed/src/archetypes/ecommerce.ts@@ -0,0 +1,391 @@+// Acme Shop: mostly anonymous shoppers; an account (identify) is created at+// the first purchase for a share of them. Funnel: product_viewed →+// add_to_cart → checkout_started → shipping_info_added → payment_info_added+// → purchase. Cohort-worthy: has purchased, has an account, used a coupon,+// wishlisted.++import { newPerson } from '../data/people';+import { REFERRERS } from '../data/referrers';+import type { Visitor } from '../model';+import type { Rng } from '../rng';+import type { Archetype, Journey } from './archetype';++const CATEGORIES = [+ 'sneakers',+ 'boots',+ 'sandals',+ 'running',+ 'kids',+ 'accessories',+] as const;+const BRANDS = ['Acme', 'Northwind', 'Vela', 'Kestrel', 'Orbit'] as const;+const SIZES = [+ '36',+ '37',+ '38',+ '39',+ '40',+ '41',+ '42',+ '43',+ '44',+ '45',+ '46',+] as const;+const COLORS = ['black', 'white', 'navy', 'sand', 'olive', 'red'] as const;+const PAYMENTS = [+ 'card',+ 'paypal',+ 'klarna',+ 'apple_pay',+ 'google_pay',+] as const;+const SHIPPING = ['standard', 'express', 'pickup'] as const;+const COUPONS = ['WELCOME10', 'SUMMER20', 'FREESHIP', 'VIP15'] as const;+const LOYALTY_TIERS = ['bronze', 'silver', 'gold'] as const;+/** Ten of each category, so `product_viewed.sku` has real cardinality. */+const PRODUCTS_PER_CATEGORY = 10;++interface Product {+ sku: string;+ name: string;+ category: (typeof CATEGORIES)[number];+ brand: (typeof BRANDS)[number];+ price: number;+}++function catalogue(): Product[] {+ const products: Product[] = [];+ for (const [categoryIndex, category] of CATEGORIES.entries()) {+ for (let n = 1; n <= PRODUCTS_PER_CATEGORY; n++) {+ products.push({+ sku: `${category.slice(0, 3).toUpperCase()}-${String(n).padStart(3, '0')}`,+ name: `${category[0]?.toUpperCase()}${category.slice(1)} ${n}`,+ category,+ brand: BRANDS[(categoryIndex + n) % BRANDS.length] ?? 'Acme',+ price: 39 + ((categoryIndex * 37 + n * 23) % 180),+ });+ }+ }+ return products;+}++const PRODUCTS = catalogue();++const ACCOUNT_SHARE = 0.4;+const BOUNCE_CHANCE = 0.38;+const SEARCH_CHANCE = 0.2;+const FILTER_CHANCE = 0.25;+const WISHLIST_CHANCE = 0.06;+const ADD_TO_CART_CHANCE = 0.22;+const REMOVE_FROM_CART_CHANCE = 0.15;+const CHECKOUT_CHANCE = 0.45;+const SHIPPING_CHANCE = 0.85;+const PAYMENT_CHANCE = 0.8;+const PURCHASE_CHANCE = 0.85;+const COUPON_CHANCE = 0.2;+const REVIEW_CHANCE = 0.08;+const SHIPPING_PRICE = 9;++interface ShopState {+ cart?: Product[];+ purchases?: number;+ couponUsed?: boolean;+ wishlisted?: boolean;+}++function stateOf(visitor: Visitor): ShopState {+ return visitor.state as ShopState;+}++function viewProduct(rng: Rng, journey: Journey, product: Product): void {+ journey.view(+ `/product/${product.sku.toLowerCase()}`,+ `${product.name} — Acme Shop`,+ { sku: product.sku, category: product.category }+ );+ journey.event('product_viewed', {+ sku: product.sku,+ name: product.name,+ category: product.category,+ brand: product.brand,+ price: product.price,+ in_stock: rng.chance(0.92),+ rating: rng.int(3, 5),+ });+ if (rng.chance(0.3)) {+ journey.event('product_image_zoomed', { sku: product.sku });+ }+}++function cartValue(cart: readonly Product[]): number {+ let total = 0;+ for (const product of cart) {+ total += product.price;+ }+ return total;+}++function checkout(+ rng: Rng,+ journey: Journey,+ visitor: Visitor,+ state: ShopState,+ cart: Product[]+): void {+ journey.view('/checkout', 'Checkout');+ const value = cartValue(cart);+ journey.event('checkout_started', {+ items: cart.length,+ value,+ currency: 'USD',+ });+ if (!rng.chance(SHIPPING_CHANCE)) {+ return;+ }+ const shipping = rng.one(SHIPPING);+ journey.event('shipping_info_added', {+ method: shipping,+ country: visitor.geo.country,+ });+ if (!rng.chance(PAYMENT_CHANCE)) {+ return;+ }+ const payment = rng.one(PAYMENTS);+ journey.event('payment_info_added', { method: payment });+ let coupon: string | null = null;+ if (rng.chance(COUPON_CHANCE)) {+ coupon = rng.one(COUPONS);+ journey.event('coupon_applied', {+ code: coupon,+ discount_percent: coupon === 'FREESHIP' ? 0 : Number(coupon.slice(-2)),+ });+ state.couponUsed = true;+ }+ if (!rng.chance(PURCHASE_CHANCE)) {+ return;+ }+ // The first order creates the account for those who make one.+ if (visitor.person && !visitor.identified) {+ journey.identify(visitor.person);+ journey.event('account_created', {+ method: rng.one(['email', 'google', 'apple']),+ at: 'checkout',+ });+ }+ const orderId = `ord_${rng.hex(8)}`;+ const total =+ value ++ (shipping === 'pickup' || coupon === 'FREESHIP' ? 0 : SHIPPING_PRICE);+ journey.view(`/order/${orderId}`, 'Thank you');+ journey.event('purchase', {+ order_id: orderId,+ items: cart.length,+ value: total,+ currency: 'USD',+ payment,+ shipping,+ coupon,+ first_purchase: (state.purchases ?? 0) === 0,+ categories: [...new Set(cart.map((product) => product.category))].join(','),+ });+ journey.revenue(total, { order_id: orderId, currency: 'USD' });+ state.purchases = (state.purchases ?? 0) + 1;+ state.cart = [];+ if (rng.chance(REVIEW_CHANCE)) {+ const reviewed = rng.one(cart);+ journey.event('review_submitted', {+ sku: reviewed.sku,+ rating: rng.int(2, 5),+ });+ }+}++export const ecommerceArchetype: Archetype = {+ id: 'ecommerce',+ projectName: 'Acme Shop',+ origin: 'https://shop.acme.test',+ domain: 'https://shop.acme.test',+ types: ['website'],+ sdk: { name: 'web', version: '1.0.5' },+ shape: {+ hourly: [+ 1.5, 0.8, 0.5, 0.3, 0.3, 0.4, 0.8, 1.5, 2.5, 3.5, 4, 4.5, 4.5, 4, 4, 4,+ 4.5, 5, 5.5, 6.5, 7, 6, 4.5, 3,+ ],+ dayOfWeek: [1.15, 0.9, 0.9, 0.95, 1.0, 1.1, 1.25],+ trendPerWeek: 0.01,+ spikeChance: 0.05,+ dipChance: 0.02,+ spikeRange: [2, 4],+ dipRange: [0.4, 0.7],+ },+ trafficShare: 0.25,+ returningShare: 0.4,+ retention: { sameDay: 0.18, day1: 0.2, day7: 0.1, day30: 0.06 },+ stickiness(visitor) {+ const state = stateOf(visitor);+ let value = 1;+ if ((state.purchases ?? 0) > 0) {+ value += 1.5;+ }+ if (visitor.identified) {+ value += 1;+ }+ if (state.wishlisted) {+ value += 0.5;+ }+ return value;+ },+ mobileShare: 0.65,+ deviceClass: 'web',+ referrers: [+ { weight: 30, value: null },+ { weight: 28, value: REFERRERS.google },+ { weight: 12, value: REFERRERS.instagram },+ { weight: 10, value: REFERRERS.facebook },+ { weight: 5, value: REFERRERS.youTube },+ { weight: 4, value: REFERRERS.bing },+ { weight: 3, value: REFERRERS.reddit },+ { weight: 3, value: REFERRERS.gmail },+ { weight: 2, value: REFERRERS.twitter },+ ],+ campaignChance: 0.3,+ dwellMedianSeconds: 20,+ utcOffsetHours: 0,+ newPerson(rng) {+ if (!rng.chance(ACCOUNT_SHARE)) {+ return null;+ }+ const person = newPerson(rng, { business: false });+ person.properties = {+ loyalty_tier: rng.one(LOYALTY_TIERS),+ newsletter: String(rng.chance(0.5)),+ };+ return person;+ },+ identity:+ 'Anonymous shoppers; an account is created at the first purchase for ~40% of buyers, who are identified from then on.',+ conversions: ['purchase', 'account_created'],+ funnels: [+ {+ name: 'Checkout',+ steps: [+ 'product_viewed',+ 'add_to_cart',+ 'checkout_started',+ 'shipping_info_added',+ 'payment_info_added',+ 'purchase',+ ],+ breakdowns: [+ 'category',+ 'brand',+ 'payment',+ 'shipping',+ 'device',+ 'country',+ 'coupon',+ ],+ },+ {+ name: 'Search to cart',+ steps: ['search', 'product_viewed', 'add_to_cart'],+ breakdowns: ['query', 'category'],+ },+ ],+ journey(rng, journey, visitor) {+ const state = stateOf(visitor);+ state.cart ??= [];++ const entry = rng.pick([+ { weight: 45, value: 'home' },+ { weight: 30, value: 'product' },+ { weight: 25, value: 'category' },+ ] as const);+ if (entry === 'home') {+ journey.view('/', 'Acme Shop');+ } else if (entry === 'category') {+ const category = rng.one(CATEGORIES);+ journey.view(`/category/${category}`, `${category} — Acme Shop`, {+ category,+ });+ } else {+ viewProduct(rng, journey, rng.one(PRODUCTS));+ }+ if (rng.chance(BOUNCE_CHANCE)) {+ return;+ }++ if (rng.chance(SEARCH_CHANCE)) {+ const query = rng.one([+ 'white sneakers',+ 'boots',+ 'running shoes',+ 'sale',+ 'kids',+ 'sandals',+ ]);+ journey.event('search', { query, results: rng.int(0, 40) });+ }+ if (rng.chance(FILTER_CHANCE)) {+ journey.event('filter_applied', {+ facet: rng.one(['size', 'color', 'brand', 'price']),+ value: rng.one([...SIZES, ...COLORS, ...BRANDS]),+ });+ }++ const browses = rng.int(1, 4);+ for (let i = 0; i < browses; i++) {+ const product = rng.one(PRODUCTS);+ if (rng.chance(0.4)) {+ journey.view(+ `/category/${product.category}`,+ `${product.category} — Acme Shop`,+ { category: product.category }+ );+ }+ viewProduct(rng, journey, product);+ if (rng.chance(WISHLIST_CHANCE)) {+ journey.event('wishlist_added', {+ sku: product.sku,+ category: product.category,+ });+ state.wishlisted = true;+ }+ if (rng.chance(ADD_TO_CART_CHANCE)) {+ journey.event('add_to_cart', {+ sku: product.sku,+ name: product.name,+ category: product.category,+ brand: product.brand,+ price: product.price,+ quantity: 1,+ size: rng.one(SIZES),+ color: rng.one(COLORS),+ });+ state.cart.push(product);+ }+ }+ if (state.cart.length === 0) {+ return;+ }++ journey.view('/cart', 'Your cart', {+ items: state.cart.length,+ value: cartValue(state.cart),+ });+ if (rng.chance(REMOVE_FROM_CART_CHANCE)) {+ const removed = state.cart.pop();+ if (removed) {+ journey.event('remove_from_cart', {+ sku: removed.sku,+ category: removed.category,+ });+ }+ }+ if (state.cart.length > 0 && rng.chance(CHECKOUT_CHANCE)) {+ checkout(rng, journey, visitor, state, state.cart);+ }+ },+};diff --git a/packages/seed/src/archetypes/index.ts b/packages/seed/src/archetypes/index.tsnew file mode 100644index 000000000..c45604056--- /dev/null+++ b/packages/seed/src/archetypes/index.ts@@ -0,0 +1,26 @@+import { appArchetype } from './app';+import type { Archetype } from './archetype';+import { ecommerceArchetype } from './ecommerce';+import { saasArchetype } from './saas';+import { websiteArchetype } from './website';++export const ARCHETYPES: readonly Archetype[] = [+ websiteArchetype,+ saasArchetype,+ ecommerceArchetype,+ appArchetype,+];++export const ARCHETYPE_IDS = ARCHETYPES.map((archetype) => archetype.id);++export function archetypeById(id: string): Archetype {+ const archetype = ARCHETYPES.find((candidate) => candidate.id === id);+ if (!archetype) {+ throw new Error(+ `Unknown archetype "${id}". Known: ${ARCHETYPE_IDS.join(', ')}`+ );+ }+ return archetype;+}++export type { Archetype, ArchetypeId } from './archetype';diff --git a/packages/seed/src/archetypes/saas.ts b/packages/seed/src/archetypes/saas.tsnew file mode 100644index 000000000..e4e5d698c--- /dev/null+++ b/packages/seed/src/archetypes/saas.ts@@ -0,0 +1,391 @@+// Acme SaaS: a logged-in product. Everyone identifies on the first session,+// so retention is real. Funnels: activation (signup → project → SDK → first+// event) spread over visits, and monetization (pricing → checkout →+// subscription). Cohort-worthy behaviours: invited a teammate, exported,+// connected an integration, paid plan.++import { newPerson } from '../data/people';+import { REFERRERS } from '../data/referrers';+import type { Person, Visitor } from '../model';+import type { Rng, Weighted } from '../rng';+import type { Archetype, Journey } from './archetype';++interface Plan {+ name: string;+ price: number;+}++const PLANS: readonly Weighted<Plan>[] = [+ { weight: 70, value: { name: 'free', price: 0 } },+ { weight: 18, value: { name: 'pro', price: 29 } },+ { weight: 9, value: { name: 'team', price: 99 } },+ { weight: 3, value: { name: 'enterprise', price: 299 } },+];+const PAID_PLANS = PLANS.filter((plan) => plan.value.price > 0);+const ROLES = ['founder', 'engineer', 'product', 'marketing', 'data'] as const;+const COMPANY_SIZES = [+ '1',+ '2-10',+ '11-50',+ '51-200',+ '201-1000',+ '1000+',+] as const;+const SIGNUP_SOURCES = [+ 'organic',+ 'referral',+ 'ads',+ 'producthunt',+ 'docs',+] as const;+const ONBOARDING_STEPS = [+ 'organization',+ 'project',+ 'install',+ 'verify',+] as const;+const PROJECT_TYPES = ['website', 'app', 'backend'] as const;+const SDKS = [+ 'web',+ 'react',+ 'next',+ 'react-native',+ 'node',+ 'python',+] as const;+const FEATURES = [+ 'funnels',+ 'retention',+ 'cohorts',+ 'pages',+ 'realtime',+ 'profiles',+ 'insights',+ 'ai-assistant',+] as const;+const INTEGRATIONS = ['slack', 'discord', 'webhook', 'gsc', 'stripe'] as const;+const REPORT_TYPES = [+ 'overview',+ 'funnel',+ 'retention',+ 'pages',+ 'events',+ 'cohort',+] as const;+const CANCEL_REASONS = [+ 'too_expensive',+ 'missing_feature',+ 'not_using',+ 'switched',+] as const;+const PROJECT_IDS = [+ 'web',+ 'ios',+ 'android',+ 'landing',+ 'internal',+ 'staging',+] as const;++const SIGNUP_COMPLETED_CHANCE = 0.85;+const ONBOARDING_STEP_CHANCE = 0.75;+const SDK_INSTALLED_CHANCE = 0.55;+const FIRST_EVENT_CHANCE = 0.8;+const PRICING_VIEWED_CHANCE = 0.06;+const CHECKOUT_CHANCE = 0.45;+const SUBSCRIBED_CHANCE = 0.6;+const CHURN_CHANCE = 0.004;+const INVITE_CHANCE = 0.05;+const EXPORT_CHANCE = 0.05;+const INTEGRATION_CHANCE = 0.03;+const REPORT_CREATED_CHANCE = 0.15;+const FEATURE_CHANCE = 0.5;+const SETTINGS_CHANCE = 0.1;++interface SaasState {+ plan?: Plan;+ onboardingStep?: number;+ projectCreated?: boolean;+ sdkInstalled?: boolean;+ firstEvent?: boolean;+ invited?: boolean;+ exported?: boolean;+ integrated?: boolean;+ churned?: boolean;+}++function stateOf(visitor: Visitor): SaasState {+ return visitor.state as SaasState;+}++function personProps(person: Person): Record<string, string> {+ return person.properties;+}++/** Activation is spread over visits: each session moves the user at most a step or two. */+function activation(rng: Rng, journey: Journey, state: SaasState): void {+ const step = state.onboardingStep ?? 0;+ if (step < ONBOARDING_STEPS.length) {+ if (!rng.chance(ONBOARDING_STEP_CHANCE)) {+ return;+ }+ journey.view('/onboarding', 'Onboarding', { step: step + 1 });+ journey.event('onboarding_step_completed', {+ step: step + 1,+ name: ONBOARDING_STEPS[step],+ });+ state.onboardingStep = step + 1;+ if (ONBOARDING_STEPS[step] === 'project') {+ journey.event('project_created', {+ type: rng.one(PROJECT_TYPES),+ timezone: rng.one([+ 'UTC',+ 'Europe/Stockholm',+ 'America/New_York',+ 'Asia/Tokyo',+ ]),+ });+ state.projectCreated = true;+ }+ return;+ }+ if (+ state.projectCreated &&+ !state.sdkInstalled &&+ rng.chance(SDK_INSTALLED_CHANCE)+ ) {+ journey.view('/projects/web/connect', 'Connect your app');+ journey.event('sdk_installed', {+ sdk: rng.one(SDKS),+ framework: rng.one(['next', 'vite', 'remix', 'expo', 'express', 'none']),+ });+ state.sdkInstalled = true;+ return;+ }+ if (+ state.sdkInstalled &&+ !state.firstEvent &&+ rng.chance(FIRST_EVENT_CHANCE)+ ) {+ journey.event('first_event_received', {+ latency_seconds: rng.int(5, 900),+ event_name: rng.one(['screen_view', 'signup', 'purchase']),+ });+ state.firstEvent = true;+ }+}++function monetization(+ rng: Rng,+ journey: Journey,+ state: SaasState,+ person: Person+): void {+ const plan = state.plan ?? { name: 'free', price: 0 };+ if (plan.price === 0 && rng.chance(PRICING_VIEWED_CHANCE)) {+ journey.view('/settings/billing', 'Billing');+ journey.event('pricing_viewed', { current_plan: plan.name });+ if (!rng.chance(CHECKOUT_CHANCE)) {+ return;+ }+ const chosen = rng.pick(PAID_PLANS);+ const interval = rng.one(['monthly', 'yearly']);+ journey.event('checkout_started', { plan: chosen.name, interval });+ if (!rng.chance(SUBSCRIBED_CHANCE)) {+ return;+ }+ const seats = chosen.name === 'pro' ? 1 : rng.int(2, 25);+ journey.event('subscription_started', {+ plan: chosen.name,+ interval,+ seats,+ mrr: chosen.price,+ previous_plan: plan.name,+ });+ journey.revenue(interval === 'yearly' ? chosen.price * 10 : chosen.price, {+ plan: chosen.name,+ interval,+ seats,+ });+ state.plan = chosen;+ person.properties.plan = chosen.name;+ return;+ }+ if (plan.price > 0 && rng.chance(CHURN_CHANCE)) {+ journey.view('/settings/billing', 'Billing');+ journey.event('subscription_cancelled', {+ plan: plan.name,+ reason: rng.one(CANCEL_REASONS),+ months_active: rng.int(1, 18),+ });+ state.plan = { name: 'free', price: 0 };+ state.churned = true;+ person.properties.plan = 'free';+ }+}++function usage(rng: Rng, journey: Journey, state: SaasState): void {+ const visits = rng.int(1, 3);+ for (let i = 0; i < visits; i++) {+ const projectId = rng.one(PROJECT_IDS);+ journey.view(`/projects/${projectId}`, `Project ${projectId}`, {+ project: projectId,+ });+ if (rng.chance(FEATURE_CHANCE)) {+ const feature = rng.one(FEATURES);+ journey.view(`/projects/${projectId}/${feature}`, feature, {+ project: projectId,+ });+ journey.event('feature_used', { feature, project: projectId });+ }+ if (rng.chance(0.6)) {+ journey.event('report_viewed', {+ report_type: rng.one(REPORT_TYPES),+ range: rng.one(['7d', '30d', '90d', 'custom']),+ project: projectId,+ });+ }+ if (rng.chance(REPORT_CREATED_CHANCE)) {+ journey.event('report_created', {+ chart: rng.one(['line', 'bar', 'funnel', 'retention', 'pie']),+ breakdowns: rng.int(0, 2),+ project: projectId,+ });+ }+ if (rng.chance(EXPORT_CHANCE)) {+ journey.event('export_completed', {+ format: rng.one(['csv', 'json']),+ rows: rng.int(100, 200_000),+ });+ state.exported = true;+ }+ }+ if (rng.chance(INVITE_CHANCE)) {+ journey.view('/settings/members', 'Members');+ journey.event('invite_sent', { role: rng.one(['admin', 'member']) });+ state.invited = true;+ } else if (rng.chance(INTEGRATION_CHANCE)) {+ journey.view('/settings/integrations', 'Integrations');+ journey.event('integration_connected', { provider: rng.one(INTEGRATIONS) });+ state.integrated = true;+ } else if (rng.chance(SETTINGS_CHANCE)) {+ journey.view('/settings', 'Settings');+ }+}++export const saasArchetype: Archetype = {+ id: 'saas',+ projectName: 'Acme SaaS',+ origin: 'https://app.acme.test',+ domain: 'https://app.acme.test',+ types: ['website', 'backend'],+ sdk: { name: 'web', version: '1.0.5' },+ shape: {+ hourly: [+ 0.5, 0.3, 0.2, 0.2, 0.3, 0.6, 1.5, 3.5, 6, 7, 7, 6, 5.5, 6.5, 7, 6.5, 6,+ 5, 3.5, 2.5, 2, 1.5, 1, 0.8,+ ],+ dayOfWeek: [0.2, 1.2, 1.25, 1.25, 1.2, 1.0, 0.2],+ trendPerWeek: 0.015,+ spikeChance: 0.01,+ dipChance: 0.03,+ spikeRange: [1.5, 2.5],+ dipRange: [0.3, 0.6],+ },+ trafficShare: 0.2,+ returningShare: 0.75,+ retention: { sameDay: 0.35, day1: 0.55, day7: 0.4, day30: 0.3 },+ stickiness(visitor) {+ const state = stateOf(visitor);+ if (state.churned) {+ return 0.3;+ }+ let value = (state.plan?.price ?? 0) > 0 ? 3 : 1;+ if (state.invited) {+ value += 1;+ }+ if (state.firstEvent) {+ value += 0.5;+ }+ return value;+ },+ mobileShare: 0.12,+ deviceClass: 'web',+ referrers: [+ { weight: 70, value: null },+ { weight: 15, value: REFERRERS.google },+ { weight: 8, value: REFERRERS.gmail },+ { weight: 4, value: REFERRERS.gitHub },+ { weight: 3, value: REFERRERS.linkedIn },+ ],+ campaignChance: 0.03,+ dwellMedianSeconds: 45,+ utcOffsetHours: -2,+ newPerson(rng) {+ const person = newPerson(rng, { business: true });+ person.properties = {+ plan: 'free',+ role: rng.one(ROLES),+ company_size: rng.one(COMPANY_SIZES),+ signup_source: rng.one(SIGNUP_SOURCES),+ };+ return person;+ },+ identity:+ 'Everyone signs up or logs in on their first session and is identified from then on; retention is real.',+ conversions: ['signup_completed', 'sdk_installed', 'subscription_started'],+ funnels: [+ {+ name: 'Activation',+ steps: [+ 'signup_completed',+ 'project_created',+ 'sdk_installed',+ 'first_event_received',+ ],+ breakdowns: ['signup_source', 'company_size', 'role', 'sdk'],+ },+ {+ name: 'Upgrade',+ steps: ['pricing_viewed', 'checkout_started', 'subscription_started'],+ breakdowns: ['plan', 'interval', 'company_size'],+ },+ ],+ journey(rng, journey, visitor) {+ const state = stateOf(visitor);+ const person = visitor.person;+ if (!person) {+ return;+ }++ if (!visitor.identified) {+ journey.view('/signup', 'Create your account');+ const method = rng.one(['email', 'github', 'google']);+ journey.event('signup_started', { method });+ if (!rng.chance(SIGNUP_COMPLETED_CHANCE)) {+ return;+ }+ journey.identify(person);+ journey.event('signup_completed', { method, ...personProps(person) });+ state.plan = rng.pick(PLANS.filter((plan) => plan.value.price === 0));+ } else if (visitor.sessions === 0 || rng.chance(0.15)) {+ // A known user on a new device, or a session that expired.+ journey.view('/login', 'Log in');+ journey.event('login', {+ method: rng.one(['email', 'github', 'google']),+ });+ }++ journey.view('/dashboard', 'Dashboard', {+ plan: state.plan?.name ?? 'free',+ });+ activation(rng, journey, state);+ if (+ state.onboardingStep !== undefined &&+ state.onboardingStep >= ONBOARDING_STEPS.length+ ) {+ usage(rng, journey, state);+ monetization(rng, journey, state, person);+ }+ },+};diff --git a/packages/seed/src/archetypes/website.ts b/packages/seed/src/archetypes/website.tsnew file mode 100644index 000000000..16114103c--- /dev/null+++ b/packages/seed/src/archetypes/website.ts@@ -0,0 +1,244 @@+// Acme Web: a marketing + docs site. Anonymous traffic; a share of visitors+// eventually signs up, which identifies them. Funnels: landing → pricing →+// signup, and docs → demo request.++import { newPerson } from '../data/people';+import { REFERRERS } from '../data/referrers';+import type { Archetype } from './archetype';++const BLOG_TOPICS = [+ 'product-analytics',+ 'privacy',+ 'self-hosting',+ 'retention',+ 'funnels',+ 'attribution',+ 'sdk',+ 'clickhouse',+ 'pricing',+ 'changelog',+];+const BLOG_POSTS_PER_TOPIC = 6;+const DOCS_PAGES = [+ ['get-started', 'Get started'],+ ['sdks/web', 'Web SDK'],+ ['sdks/react', 'React SDK'],+ ['sdks/react-native', 'React Native SDK'],+ ['sdks/node', 'Node SDK'],+ ['api/track', 'Track API'],+ ['api/export', 'Export API'],+ ['guides/funnels', 'Funnels'],+ ['guides/retention', 'Retention'],+ ['guides/cohorts', 'Cohorts'],+ ['self-hosting', 'Self-hosting'],+ ['self-hosting/clickhouse', 'ClickHouse tuning'],+] as const;+const COMPANY_SIZES = [+ '1',+ '2-10',+ '11-50',+ '51-200',+ '201-1000',+ '1000+',+] as const;+const ROLES = ['founder', 'engineer', 'product', 'marketing', 'data'] as const;+const PLANS = ['free', 'pro', 'team'] as const;++const WILL_SIGN_UP_SHARE = 0.18;+const BOUNCE_CHANCE = 0.42;+const CTA_CHANCE = 0.25;+const PRICING_CHANCE = 0.35;+const DOCS_CHANCE = 0.3;+const DOCS_SEARCH_CHANCE = 0.3;+const NEWSLETTER_CHANCE = 0.04;+const DEMO_CHANCE = 0.02;+const CONTACT_CHANCE = 0.02;+const SIGNUP_STARTED_CHANCE = 0.4;+const SIGNUP_COMPLETED_CHANCE = 0.7;+const IDENTIFIED_STICKINESS = 3;++export const websiteArchetype: Archetype = {+ id: 'website',+ projectName: 'Acme Web',+ origin: 'https://www.acme.test',+ domain: 'https://www.acme.test',+ types: ['website'],+ sdk: { name: 'web', version: '1.0.5' },+ shape: {+ hourly: [+ 1, 0.6, 0.4, 0.3, 0.3, 0.5, 1, 2.5, 4, 5, 5.5, 5.5, 5, 5.5, 5.5, 5, 4.5,+ 4, 3.5, 3.5, 3, 2.5, 2, 1.5,+ ],+ dayOfWeek: [0.55, 1.1, 1.15, 1.15, 1.1, 0.95, 0.55],+ trendPerWeek: 0.02,+ spikeChance: 0.04,+ dipChance: 0.02,+ spikeRange: [2, 6],+ dipRange: [0.2, 0.5],+ },+ trafficShare: 0.45,+ returningShare: 0.3,+ // Anonymous visitors are new devices tomorrow; this is how often the same person comes back.+ retention: { sameDay: 0.1, day1: 0.14, day7: 0.06, day30: 0.03 },+ stickiness: (visitor) => (visitor.identified ? IDENTIFIED_STICKINESS : 1),+ mobileShare: 0.45,+ deviceClass: 'web',+ referrers: [+ { weight: 35, value: null },+ { weight: 25, value: REFERRERS.google },+ { weight: 4, value: REFERRERS.bing },+ { weight: 3, value: REFERRERS.duckDuckGo },+ { weight: 6, value: REFERRERS.twitter },+ { weight: 5, value: REFERRERS.hackerNews },+ { weight: 4, value: REFERRERS.reddit },+ { weight: 4, value: REFERRERS.linkedIn },+ { weight: 4, value: REFERRERS.gitHub },+ { weight: 3, value: REFERRERS.chatGpt },+ { weight: 2, value: REFERRERS.claude },+ { weight: 2, value: REFERRERS.medium },+ { weight: 2, value: REFERRERS.productHunt },+ { weight: 1, value: REFERRERS.youTube },+ ],+ campaignChance: 0.12,+ dwellMedianSeconds: 25,+ utcOffsetHours: 1,+ newPerson(rng) {+ if (!rng.chance(WILL_SIGN_UP_SHARE)) {+ return null;+ }+ const person = newPerson(rng, { business: rng.chance(0.6) });+ person.properties = {+ company_size: rng.one(COMPANY_SIZES),+ role: rng.one(ROLES),+ plan: rng.one(PLANS),+ };+ return person;+ },+ identity:+ 'Anonymous; a device is new every UTC day. Visitors who sign up are identified from then on (~18%).',+ conversions: ['signup_completed', 'demo_requested'],+ funnels: [+ {+ name: 'Signup',+ steps: [+ 'screen_view',+ 'plan_selected',+ 'signup_started',+ 'signup_completed',+ ],+ breakdowns: ['referrer_name', 'device', 'country', 'plan'],+ },+ {+ name: 'Demo request',+ steps: ['screen_view', 'cta_clicked', 'demo_requested'],+ breakdowns: ['cta', 'company_size'],+ },+ ],+ journey(rng, journey, visitor) {+ const entry = rng.pick([+ { weight: 55, value: 'home' },+ { weight: 25, value: 'blog' },+ { weight: 20, value: 'docs' },+ ] as const);++ if (entry === 'home') {+ journey.view('/', 'Acme — product analytics');+ } else if (entry === 'blog') {+ const topic = rng.one(BLOG_TOPICS);+ const n = rng.int(1, BLOG_POSTS_PER_TOPIC);+ journey.view(`/blog/${topic}-${n}`, `${topic} #${n} — Acme blog`, {+ topic,+ });+ } else {+ const [slug, title] = rng.one(DOCS_PAGES);+ journey.view(`/docs/${slug}`, title);+ }++ if (rng.chance(BOUNCE_CHANCE)) {+ return;+ }++ if (rng.chance(CTA_CHANCE)) {+ journey.event('cta_clicked', {+ cta: rng.one(['start-free', 'book-demo', 'view-docs']),+ location: rng.one(['hero', 'nav', 'footer']),+ });+ }+ if (rng.chance(DOCS_CHANCE)) {+ const [slug, title] = rng.one(DOCS_PAGES);+ journey.view(`/docs/${slug}`, title);+ if (rng.chance(DOCS_SEARCH_CHANCE)) {+ journey.event('docs_searched', {+ query: rng.one([+ 'identify',+ 'self host',+ 'react native',+ 'export',+ 'funnel',+ ]),+ results: rng.int(0, 12),+ });+ }+ if (rng.chance(0.15)) {+ journey.event('docs_feedback', {+ helpful: rng.chance(0.8),+ page: slug,+ });+ }+ }++ const wantsPricing =+ rng.chance(PRICING_CHANCE) ||+ (visitor.person !== null && !visitor.identified && rng.chance(0.5));+ if (wantsPricing) {+ journey.view('/pricing', 'Pricing');+ if (rng.chance(0.3)) {+ journey.event('pricing_toggled', {+ interval: rng.one(['monthly', 'yearly']),+ });+ }+ if (+ visitor.person &&+ !visitor.identified &&+ rng.chance(SIGNUP_STARTED_CHANCE)+ ) {+ const plan = visitor.person.properties.plan ?? 'free';+ journey.event('plan_selected', {+ plan,+ interval: rng.one(['monthly', 'yearly']),+ });+ journey.view('/signup', 'Create your account', { plan });+ const method = rng.one(['email', 'github', 'google']);+ journey.event('signup_started', { method, plan });+ if (rng.chance(SIGNUP_COMPLETED_CHANCE)) {+ // The app signs the new user in; from here on the device is identified.+ journey.identify(visitor.person);+ journey.event('signup_completed', {+ method,+ plan,+ ...visitor.person.properties,+ });+ journey.view('/welcome', 'Welcome');+ }+ }+ }++ if (rng.chance(NEWSLETTER_CHANCE)) {+ journey.event('newsletter_subscribed', {+ placement: entry === 'blog' ? 'blog-footer' : 'homepage',+ });+ }+ if (rng.chance(DEMO_CHANCE)) {+ journey.view('/demo', 'Book a demo');+ journey.event('demo_requested', {+ company_size: rng.one(COMPANY_SIZES),+ use_case: rng.one(['web', 'mobile', 'saas', 'ecommerce']),+ });+ } else if (rng.chance(CONTACT_CHANCE)) {+ journey.view('/contact', 'Contact');+ journey.event('contact_form_submitted', {+ topic: rng.one(['sales', 'support', 'partnership']),+ });+ }+ },+};diff --git a/packages/seed/src/cli.ts b/packages/seed/src/cli.tsnew file mode 100644index 000000000..6d20c9177--- /dev/null+++ b/packages/seed/src/cli.ts@@ -0,0 +1,228 @@+// `bun run seed [--size small|medium|large|xl] [--days N --sessions-per-day N]+// [--seed N] [--projects website,saas,...] [--reset] [--dry-run] [--out path]`+//+// Targets whatever DATABASE_URL and CLICKHOUSE_URL name (a worktree's own+// databases). Same flags and seed ⇒ identical data.++import { resolve } from 'node:path';+import { parseArgs } from 'node:util';+import { db } from '@openpanel/db';+import { ARCHETYPE_IDS, ARCHETYPES, archetypeById } from './archetypes';+import { type GenerateConfig, generate, planDays } from './generator';+import { projectIdFor } from './ids';+import { buildManifest, describeManifest, writeManifest } from './manifest';+import { recordProjectStats, seedPostgres } from './postgres';+import { resetProjects } from './reset';+import { Rng } from './rng';+import {+ DEFAULT_SEED,+ DEFAULT_SIZE,+ MANIFEST_FILE,+ SIZE_PRESETS,+ type SizePreset,+} from './seed.constants';+import { ClickhouseSink } from './sink';++/** Sessions carry ~7 rows once session_start/session_end are added; only for the estimate. */+const EVENTS_PER_SESSION_ESTIMATE = 7;+const DEFAULT_VARIANCE = 1;+const MS_PER_DAY = 86_400_000;++const USAGE = `Usage: bun run seed [options]++ --size <${Object.keys(SIZE_PRESETS).join('|')}> preset (default: ${DEFAULT_SIZE})+ --days <n> override the preset's day count+ --sessions-per-day <n> override the preset's sessions per day (all projects)+ --seed <n> deterministic seed (default: ${DEFAULT_SEED})+ --projects <a,b> archetypes to seed (default: ${ARCHETYPE_IDS.join(',')})+ --variance <x> 0 regular … 1 lively … 2 chaotic (default: ${DEFAULT_VARIANCE})+ --reset delete earlier seeded rows for these projects first+ --dry-run print the plan and exit+ --out <path> where to write the manifest (default: <repo>/${MANIFEST_FILE})+ --help+`;++function isPreset(value: string): value is SizePreset {+ return value in SIZE_PRESETS;+}++function integer(+ value: string | undefined,+ name: string,+ fallback: number+): number {+ if (value === undefined) {+ return fallback;+ }+ const parsed = Number(value);+ if (!Number.isInteger(parsed) || parsed < 0) {+ throw new Error(`--${name} must be a non-negative integer`);+ }+ return parsed;+}++function maskedUrl(url: string | undefined): string {+ if (!url) {+ return '<unset>';+ }+ try {+ const parsed = new URL(url);+ if (parsed.password) {+ parsed.password = '***';+ }+ return parsed.toString();+ } catch {+ return url;+ }+}++function main(): Promise<void> {+ const { values } = parseArgs({+ args: process.argv.slice(2),+ options: {+ size: { type: 'string' },+ days: { type: 'string' },+ 'sessions-per-day': { type: 'string' },+ seed: { type: 'string' },+ projects: { type: 'string' },+ variance: { type: 'string' },+ reset: { type: 'boolean', default: false },+ 'dry-run': { type: 'boolean', default: false },+ out: { type: 'string' },+ help: { type: 'boolean', default: false },+ },+ });++ if (values.help) {+ process.stdout.write(USAGE);+ return Promise.resolve();+ }++ const size = values.size ?? DEFAULT_SIZE;+ if (!isPreset(size)) {+ throw new Error(`Unknown size "${size}". ${USAGE}`);+ }+ const preset = SIZE_PRESETS[size];+ const days = integer(values.days, 'days', preset.days);+ const sessionsPerDay = integer(+ values['sessions-per-day'],+ 'sessions-per-day',+ preset.sessionsPerDay+ );+ const seed = integer(values.seed, 'seed', DEFAULT_SEED);+ const variance =+ values.variance === undefined ? DEFAULT_VARIANCE : Number(values.variance);+ const archetypes = values.projects+ ? values.projects.split(',').map((id) => archetypeById(id.trim()))+ : ARCHETYPES;+ const outPath =+ values.out ?? resolve(import.meta.dir, '..', '..', '..', MANIFEST_FILE);++ return run({+ size,+ days,+ sessionsPerDay,+ seed,+ variance,+ archetypes,+ reset: values.reset,+ dryRun: values['dry-run'],+ outPath,+ });+}++async function run(options: {+ size: SizePreset;+ days: number;+ sessionsPerDay: number;+ seed: number;+ variance: number;+ archetypes: typeof ARCHETYPES;+ reset: boolean;+ dryRun: boolean;+ outPath: string;+}): Promise<void> {+ const now = new Date();+ const config: GenerateConfig = {+ seed: options.seed,+ sessionsPerDay: options.sessionsPerDay,+ days: options.days,+ now,+ variance: options.variance,+ projects: options.archetypes.map((archetype) => ({+ archetype,+ projectId: projectIdFor(archetype),+ })),+ };++ const plannedSessions = planDays(config).reduce(+ (sum, day) => sum + day.sessions,+ 0+ );+ console.log(`Postgres: ${maskedUrl(process.env.DATABASE_URL)}`);+ console.log(`ClickHouse: ${maskedUrl(process.env.CLICKHOUSE_URL)}`);+ console.log(+ `Plan: ${options.days} days × ${options.sessionsPerDay} sessions/day (size ${options.size}, seed ${options.seed}) ≈ ${plannedSessions.toLocaleString()} sessions, ≈ ${(plannedSessions * EVENTS_PER_SESSION_ESTIMATE).toLocaleString()} events`+ );+ if (options.dryRun) {+ return;+ }++ const rng = new Rng(`${options.seed}/postgres`);+ const postgres = await seedPostgres(rng, options.archetypes);+ console.log(+ `Postgres ready: organization ${postgres.organizationId}, ${postgres.projects.length} projects`+ );++ if (options.reset) {+ console.log('Resetting earlier seeded rows…');+ await resetProjects(postgres.projects.map((project) => project.id));+ }++ const sink = new ClickhouseSink();+ const started = Date.now();+ const stats = await generate(config, sink, (progress) => {+ process.stdout.write(+ `\r${progress.day} ${progress.sessions.toLocaleString()} sessions, ${progress.events.toLocaleString()} events`.padEnd(+ 80+ )+ );+ });+ process.stdout.write('\n');+ console.log(`Inserted in ${((Date.now() - started) / 1000).toFixed(1)}s`);++ const perProject = new Map<string, { sessions: number; events: number }>();+ for (const project of stats) {+ perProject.set(project.projectId, {+ sessions: project.sessions,+ events: project.events,+ });+ await recordProjectStats(project.projectId, {+ eventsCount: project.events,+ firstEventAt: project.firstEventAt,+ });+ }++ const manifest = buildManifest({+ seed: options.seed,+ size: options.size,+ days: options.days,+ sessionsPerDay: options.sessionsPerDay,+ from: new Date(now.getTime() - options.days * MS_PER_DAY),+ to: now,+ postgres,+ apiUrl: process.env.API_URL ?? null,+ perProject,+ totals: sink.counts(),+ });+ writeManifest(options.outPath, manifest);+ console.log(describeManifest(manifest));+ console.log(`Manifest: ${options.outPath}`);+}++try {+ await main();+} finally {+ await db.$disconnect();+}+process.exit(0);diff --git a/packages/seed/src/clickhouse-rows.test.ts b/packages/seed/src/clickhouse-rows.test.tsnew file mode 100644index 000000000..7b09062a3--- /dev/null+++ b/packages/seed/src/clickhouse-rows.test.ts@@ -0,0 +1,109 @@+// The row shapes are copied from core, not imported. This is what keeps them+// honest: every key the seed writes must be a column, and every column must+// be written, for the three tables it inserts into.++import { describe, expect, test } from 'bun:test';+import { readFileSync } from 'node:fs';+import { resolve } from 'node:path';+import { ARCHETYPES } from './archetypes';+import type {+ ClickhouseEventRow,+ ClickhouseProfileRow,+ ClickhouseSessionRow,+} from './clickhouse-rows';+import { generate } from './generator';+import { projectIdFor } from './ids';+import type { RowSink, SinkCounts } from './sink';++const SCHEMA_FILE = resolve(+ import.meta.dir,+ '..',+ '..',+ '..',+ 'test',+ 'clickhouse-schema.sql'+);++/** Column names of one `CREATE TABLE` in the flat schema; indexes and projections are skipped. */+function schemaColumns(table: string): Set<string> {+ const sql = readFileSync(SCHEMA_FILE, 'utf8');+ const start = sql.indexOf(`CREATE TABLE IF NOT EXISTS ${table}\n`);+ if (start < 0) {+ throw new Error(`${table} not in ${SCHEMA_FILE}`);+ }+ const body = sql.slice(start, sql.indexOf('\nENGINE', start));+ const columns = new Set<string>();+ for (const match of body.matchAll(/^\s+`([a-z_]+)`/gm)) {+ columns.add(match[1] ?? '');+ }+ return columns;+}++class CollectingSink implements RowSink {+ events: ClickhouseEventRow[] = [];+ sessions: ClickhouseSessionRow[] = [];+ profiles: ClickhouseProfileRow[] = [];+ addEvents(rows: readonly ClickhouseEventRow[]): void {+ this.events.push(...rows);+ }+ addSession(row: ClickhouseSessionRow): void {+ this.sessions.push(row);+ }+ addProfiles(rows: readonly ClickhouseProfileRow[]): void {+ this.profiles.push(...rows);+ }+ flushFull(): Promise<void> {+ return Promise.resolve();+ }+ flush(): Promise<void> {+ return Promise.resolve();+ }+ counts(): SinkCounts {+ return {+ events: this.events.length,+ sessions: this.sessions.length,+ profiles: this.profiles.length,+ };+ }+}++async function sampleRows(): Promise<CollectingSink> {+ const sink = new CollectingSink();+ await generate(+ {+ seed: 1,+ sessionsPerDay: 60,+ days: 1,+ now: new Date('2026-09-20T12:00:00.000Z'),+ variance: 1,+ projects: ARCHETYPES.map((archetype) => ({+ archetype,+ projectId: projectIdFor(archetype),+ })),+ },+ sink,+ () => undefined+ );+ return sink;+}++function keysOf(rows: readonly object[]): Set<string> {+ const keys = new Set<string>();+ for (const row of rows) {+ for (const key of Object.keys(row)) {+ keys.add(key);+ }+ }+ return keys;+}++describe('seed rows match the ClickHouse schema', () => {+ for (const table of ['events', 'sessions', 'profiles'] as const) {+ test(table, async () => {+ const sink = await sampleRows();+ const written = keysOf(sink[table]);+ const columns = schemaColumns(table);+ expect([...written].sort()).toEqual([...columns].sort());+ });+ }+});diff --git a/packages/seed/src/clickhouse-rows.ts b/packages/seed/src/clickhouse-rows.tsnew file mode 100644index 000000000..d14798d7e--- /dev/null+++ b/packages/seed/src/clickhouse-rows.ts@@ -0,0 +1,100 @@+// The rows the seed inserts. Copied from core's `IClickhouseEvent`,+// `IClickhouseSession` and `IClickhouseProfile` rather than imported: like+// code-migrations, the seed is a script with direct database access and+// stays out of the backend's dependency graph. `clickhouse-rows.test.ts`+// checks these shapes against the schema.++export interface ClickhouseEventRow {+ id: string;+ name: string;+ device_id: string;+ profile_id: string;+ project_id: string;+ session_id: string;+ path: string;+ origin: string;+ referrer: string;+ referrer_name: string;+ referrer_type: string;+ duration: number;+ properties: Record<string, string>;+ created_at: string;+ country: string;+ city: string;+ region: string;+ longitude: number | null;+ latitude: number | null;+ os: string;+ os_version: string;+ browser: string;+ browser_version: string;+ device: string;+ brand: string;+ model: string;+ imported_at: string | null;+ inserted_at: string;+ sdk_name: string;+ sdk_version: string;+ revenue?: number;+ groups: string[];+}++export interface ClickhouseSessionRow {+ id: string;+ profile_id: string;+ event_count: number;+ screen_view_count: number;+ entry_path: string;+ entry_origin: string;+ exit_path: string;+ exit_origin: string;+ created_at: string;+ ended_at: string;+ referrer: string;+ referrer_name: string;+ referrer_type: string;+ os: string;+ os_version: string;+ browser: string;+ browser_version: string;+ device: string;+ brand: string;+ model: string;+ country: string;+ region: string;+ city: string;+ longitude: number | null;+ latitude: number | null;+ is_bounce: boolean;+ project_id: string;+ device_id: string;+ duration: number;+ utm_medium: string;+ utm_source: string;+ utm_campaign: string;+ utm_content: string;+ utm_term: string;+ revenue: number;+ sign: 1 | -1;+ version: number;+ groups: string[];+}++export interface ClickhouseProfileRow {+ id: string;+ first_name: string;+ last_name: string;+ email: string;+ avatar: string;+ properties: Record<string, string>;+ project_id: string;+ is_external: boolean;+ created_at: string;+ last_seen_at: string;+ groups: string[];+}++/** `DateTime64(3)` in UTC, the format the worker writes: `yyyy-MM-dd HH:mm:ss.SSS`. */+export function toClickhouseDateTime(date: Date): string {+ return date.toISOString().replace('T', ' ').replace('Z', '');+}diff --git a/packages/seed/src/data/devices.ts b/packages/seed/src/data/devices.tsnew file mode 100644index 000000000..9b2d289f9--- /dev/null+++ b/packages/seed/src/data/devices.ts@@ -0,0 +1,114 @@+// User agents only; os/browser/device fields come from the real parser at+// session time, so the seed can never disagree with the ingest worker.++import type { Weighted } from '../rng';++export const DESKTOP_USER_AGENTS: readonly Weighted<string>[] = [+ {+ weight: 40,+ value:+ 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36',+ },+ {+ weight: 25,+ value:+ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36',+ },+ {+ weight: 12,+ value:+ 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Safari/605.1.15',+ },+ {+ weight: 8,+ value:+ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0',+ },+ {+ weight: 8,+ value:+ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Edg/128.0.0.0',+ },+ {+ weight: 5,+ value:+ 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36',+ },+ {+ weight: 2,+ value:+ 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0',+ },+];++export const MOBILE_USER_AGENTS: readonly Weighted<string>[] = [+ {+ weight: 30,+ value:+ 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Mobile/15E148 Safari/604.1',+ },+ {+ weight: 15,+ value:+ 'Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0 Mobile/15E148 Safari/604.1',+ },+ {+ weight: 8,+ value:+ 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/128.0.6613.98 Mobile/15E148 Safari/604.1',+ },+ {+ weight: 18,+ value:+ 'Mozilla/5.0 (Linux; Android 14; SM-S928B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Mobile Safari/537.36',+ },+ {+ weight: 12,+ value:+ 'Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Mobile Safari/537.36',+ },+ {+ weight: 7,+ value:+ 'Mozilla/5.0 (Linux; Android 13; SM-A546B) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/25.0 Chrome/121.0.0.0 Mobile Safari/537.36',+ },+ {+ weight: 5,+ value:+ 'Mozilla/5.0 (Linux; Android 14; 2312DRA50G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36',+ },+ {+ weight: 5,+ value:+ 'Mozilla/5.0 (iPad; CPU OS 17_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Mobile/15E148 Safari/604.1',+ },+];++/** What the React Native SDK reports: the WebView user agent. */+export const NATIVE_APP_USER_AGENTS: readonly Weighted<string>[] = [+ {+ weight: 30,+ value:+ 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148',+ },+ {+ weight: 25,+ value:+ 'Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148',+ },+ {+ weight: 20,+ value:+ 'Mozilla/5.0 (Linux; Android 14; SM-S928B Build/UP1A.231005.007; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/128.0.0.0 Mobile Safari/537.36',+ },+ {+ weight: 15,+ value:+ 'Mozilla/5.0 (Linux; Android 14; Pixel 8 Pro Build/AP2A.240805.005; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/128.0.0.0 Mobile Safari/537.36',+ },+ {+ weight: 10,+ value:+ 'Mozilla/5.0 (Linux; Android 13; SM-A546B Build/TP1A.220624.014; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/126.0.0.0 Mobile Safari/537.36',+ },+];diff --git a/packages/seed/src/data/geo.ts b/packages/seed/src/data/geo.tsnew file mode 100644index 000000000..d842a395b--- /dev/null+++ b/packages/seed/src/data/geo.ts@@ -0,0 +1,343 @@+import type { Geo } from '../model';+import type { Rng, Weighted } from '../rng';++interface City {+ country: string;+ region: string;+ city: string;+ latitude: number;+ longitude: number;+ /** A /16 GeoIP attributes to roughly this region; only decorates `x-client-ip`-style data. */+ ipPrefix: string;+}++const CITIES: readonly Weighted<City>[] = [+ {+ weight: 14,+ value: {+ country: 'SE',+ region: 'Stockholm',+ city: 'Stockholm',+ latitude: 59.33,+ longitude: 18.07,+ ipPrefix: '83.233',+ },+ },+ {+ weight: 4,+ value: {+ country: 'SE',+ region: 'Västra Götaland',+ city: 'Gothenburg',+ latitude: 57.71,+ longitude: 11.97,+ ipPrefix: '83.250',+ },+ },+ {+ weight: 3,+ value: {+ country: 'SE',+ region: 'Skåne',+ city: 'Malmö',+ latitude: 55.6,+ longitude: 13.0,+ ipPrefix: '90.230',+ },+ },+ {+ weight: 16,+ value: {+ country: 'US',+ region: 'California',+ city: 'San Francisco',+ latitude: 37.77,+ longitude: -122.42,+ ipPrefix: '104.28',+ },+ },+ {+ weight: 10,+ value: {+ country: 'US',+ region: 'New York',+ city: 'New York',+ latitude: 40.71,+ longitude: -74.01,+ ipPrefix: '74.72',+ },+ },+ {+ weight: 5,+ value: {+ country: 'US',+ region: 'Texas',+ city: 'Austin',+ latitude: 30.27,+ longitude: -97.74,+ ipPrefix: '70.112',+ },+ },+ {+ weight: 4,+ value: {+ country: 'US',+ region: 'Washington',+ city: 'Seattle',+ latitude: 47.61,+ longitude: -122.33,+ ipPrefix: '73.140',+ },+ },+ {+ weight: 9,+ value: {+ country: 'GB',+ region: 'England',+ city: 'London',+ latitude: 51.51,+ longitude: -0.13,+ ipPrefix: '81.2',+ },+ },+ {+ weight: 3,+ value: {+ country: 'GB',+ region: 'England',+ city: 'Manchester',+ latitude: 53.48,+ longitude: -2.24,+ ipPrefix: '86.20',+ },+ },+ {+ weight: 7,+ value: {+ country: 'DE',+ region: 'Berlin',+ city: 'Berlin',+ latitude: 52.52,+ longitude: 13.41,+ ipPrefix: '77.12',+ },+ },+ {+ weight: 3,+ value: {+ country: 'DE',+ region: 'Bavaria',+ city: 'Munich',+ latitude: 48.14,+ longitude: 11.58,+ ipPrefix: '84.153',+ },+ },+ {+ weight: 4,+ value: {+ country: 'NL',+ region: 'North Holland',+ city: 'Amsterdam',+ latitude: 52.37,+ longitude: 4.9,+ ipPrefix: '77.160',+ },+ },+ {+ weight: 4,+ value: {+ country: 'FR',+ region: 'Île-de-France',+ city: 'Paris',+ latitude: 48.86,+ longitude: 2.35,+ ipPrefix: '82.64',+ },+ },+ {+ weight: 3,+ value: {+ country: 'ES',+ region: 'Madrid',+ city: 'Madrid',+ latitude: 40.42,+ longitude: -3.7,+ ipPrefix: '81.32',+ },+ },+ {+ weight: 3,+ value: {+ country: 'DK',+ region: 'Capital Region',+ city: 'Copenhagen',+ latitude: 55.68,+ longitude: 12.57,+ ipPrefix: '80.62',+ },+ },+ {+ weight: 3,+ value: {+ country: 'NO',+ region: 'Oslo',+ city: 'Oslo',+ latitude: 59.91,+ longitude: 10.75,+ ipPrefix: '84.208',+ },+ },+ {+ weight: 3,+ value: {+ country: 'FI',+ region: 'Uusimaa',+ city: 'Helsinki',+ latitude: 60.17,+ longitude: 24.94,+ ipPrefix: '85.76',+ },+ },+ {+ weight: 3,+ value: {+ country: 'PL',+ region: 'Masovian',+ city: 'Warsaw',+ latitude: 52.23,+ longitude: 21.01,+ ipPrefix: '83.4',+ },+ },+ {+ weight: 4,+ value: {+ country: 'CA',+ region: 'Ontario',+ city: 'Toronto',+ latitude: 43.65,+ longitude: -79.38,+ ipPrefix: '99.224',+ },+ },+ {+ weight: 4,+ value: {+ country: 'AU',+ region: 'New South Wales',+ city: 'Sydney',+ latitude: -33.87,+ longitude: 151.21,+ ipPrefix: '1.128',+ },+ },+ {+ weight: 4,+ value: {+ country: 'IN',+ region: 'Karnataka',+ city: 'Bengaluru',+ latitude: 12.97,+ longitude: 77.59,+ ipPrefix: '49.204',+ },+ },+ {+ weight: 3,+ value: {+ country: 'BR',+ region: 'São Paulo',+ city: 'São Paulo',+ latitude: -23.55,+ longitude: -46.63,+ ipPrefix: '177.220',+ },+ },+ {+ weight: 3,+ value: {+ country: 'JP',+ region: 'Tokyo',+ city: 'Tokyo',+ latitude: 35.68,+ longitude: 139.69,+ ipPrefix: '126.0',+ },+ },+ {+ weight: 2,+ value: {+ country: 'SG',+ region: '',+ city: 'Singapore',+ latitude: 1.35,+ longitude: 103.82,+ ipPrefix: '116.14',+ },+ },+ {+ weight: 2,+ value: {+ country: 'ZA',+ region: 'Western Cape',+ city: 'Cape Town',+ latitude: -33.92,+ longitude: 18.42,+ ipPrefix: '41.13',+ },+ },+ {+ weight: 2,+ value: {+ country: 'MX',+ region: 'Mexico City',+ city: 'Mexico City',+ latitude: 19.43,+ longitude: -99.13,+ ipPrefix: '187.188',+ },+ },+ {+ weight: 2,+ value: {+ country: 'IE',+ region: 'Leinster',+ city: 'Dublin',+ latitude: 53.35,+ longitude: -6.26,+ ipPrefix: '86.43',+ },+ },+ {+ weight: 2,+ value: {+ country: 'CH',+ region: 'Zurich',+ city: 'Zurich',+ latitude: 47.38,+ longitude: 8.54,+ ipPrefix: '85.5',+ },+ },+];++/** Lat/lon jitter so map views do not stack every visitor of a city on one pixel. */+const COORDINATE_JITTER = 0.05;+const COORDINATE_DECIMALS = 4;++function round(value: number): number {+ return Number(value.toFixed(COORDINATE_DECIMALS));+}++export function newGeo(rng: Rng): Geo {+ const city = rng.pick(CITIES);+ return {+ ip: `${city.ipPrefix}.${rng.int(1, 254)}.${rng.int(1, 254)}`,+ country: city.country,+ region: city.region,+ city: city.city,+ latitude: round(city.latitude + rng.normal() * COORDINATE_JITTER),+ longitude: round(city.longitude + rng.normal() * COORDINATE_JITTER),+ };+}diff --git a/packages/seed/src/data/people.ts b/packages/seed/src/data/people.tsnew file mode 100644index 000000000..fce3d2211--- /dev/null+++ b/packages/seed/src/data/people.ts@@ -0,0 +1,136 @@+import type { Person } from '../model';+import type { Rng } from '../rng';++const FIRST_NAMES = [+ 'Anna',+ 'Erik',+ 'Maja',+ 'Oskar',+ 'Elsa',+ 'Liam',+ 'Olivia',+ 'Noah',+ 'Emma',+ 'Lucas',+ 'Sofia',+ 'Hugo',+ 'Alice',+ 'William',+ 'Ella',+ 'Leo',+ 'Mia',+ 'Elias',+ 'Julia',+ 'Axel',+ 'Nora',+ 'Felix',+ 'Ida',+ 'Vincent',+ 'Clara',+ 'Adam',+ 'Freja',+ 'Isak',+ 'Astrid',+ 'Theo',+ 'James',+ 'Charlotte',+ 'Benjamin',+ 'Amelia',+ 'Henry',+ 'Harper',+ 'Jack',+ 'Evelyn',+ 'Owen',+ 'Luna',+ 'Mateo',+ 'Camila',+ 'Yuki',+ 'Haruto',+ 'Aarav',+ 'Priya',+ 'Chen',+ 'Mei',+ 'Lena',+ 'Jonas',+];++const LAST_NAMES = [+ 'Andersson',+ 'Johansson',+ 'Karlsson',+ 'Nilsson',+ 'Eriksson',+ 'Larsson',+ 'Olsson',+ 'Persson',+ 'Smith',+ 'Johnson',+ 'Williams',+ 'Brown',+ 'Jones',+ 'Garcia',+ 'Miller',+ 'Davis',+ 'Martinez',+ 'Müller',+ 'Schmidt',+ 'Schneider',+ 'Fischer',+ 'Weber',+ 'Meyer',+ 'Dubois',+ 'Martin',+ 'Bernard',+ 'Rossi',+ 'Russo',+ 'Ferrari',+ 'Silva',+ 'Santos',+ 'Tanaka',+ 'Suzuki',+ 'Sharma',+ 'Patel',+ 'Wang',+];++const EMAIL_DOMAINS: readonly string[] = [+ 'gmail.com',+ 'outlook.com',+ 'icloud.com',+ 'proton.me',+ 'hey.com',+ 'yahoo.com',+];+const COMPANY_DOMAINS: readonly string[] = [+ 'acme.io',+ 'northwind.co',+ 'globex.com',+ 'initech.dev',+ 'umbrella.app',+ 'hooli.xyz',+];++const AVATAR_BASE = 'https://api.dicebear.com/9.x/notionists/svg?seed=';++function slugify(value: string): string {+ return value+ .normalize('NFD')+ .replace(/[̀-ͯ]/g, '')+ .toLowerCase()+ .replace(/[^a-z0-9]+/g, '');+}++export function newPerson(rng: Rng, options: { business: boolean }): Person {+ const firstName = rng.one(FIRST_NAMES);+ const lastName = rng.one(LAST_NAMES);+ const domain = rng.one(options.business ? COMPANY_DOMAINS : EMAIL_DOMAINS);+ const handle = `${slugify(firstName)}.${slugify(lastName)}${rng.int(1, 999)}`;+ return {+ id: `usr_${rng.hex(12)}`,+ firstName,+ lastName,+ email: `${handle}@${domain}`,+ avatar: `${AVATAR_BASE}${handle}`,+ properties: {},+ };+}diff --git a/packages/seed/src/data/referrers.ts b/packages/seed/src/data/referrers.tsnew file mode 100644index 000000000..da142ab9a--- /dev/null+++ b/packages/seed/src/data/referrers.ts@@ -0,0 +1,120 @@+// The referrers the seed sends traffic from. Name and type are what core's+// `parseReferrer` (its generated referrer catalogue) answers for each URL,+// copied here so the seed stays out of core's dependency graph; `null` is+// direct traffic.++import type { Referrer } from '../model';+import type { Weighted } from '../rng';++export const REFERRERS = {+ google: { url: 'https://www.google.com', name: 'Google', type: 'search' },+ bing: { url: 'https://www.bing.com', name: 'Bing', type: 'search' },+ duckDuckGo: {+ url: 'https://duckduckgo.com',+ name: 'DuckDuckGo',+ type: 'search',+ },+ twitter: { url: 'https://t.co', name: 'Twitter', type: 'social' },+ facebook: { url: 'https://m.facebook.com', name: 'Facebook', type: 'social' },+ reddit: { url: 'https://www.reddit.com', name: 'Reddit', type: 'social' },+ hackerNews: {+ url: 'https://news.ycombinator.com',+ name: 'Hacker News',+ type: 'social',+ },+ linkedIn: {+ url: 'https://www.linkedin.com',+ name: 'LinkedIn',+ type: 'social',+ },+ instagram: {+ url: 'https://l.instagram.com',+ name: 'Instagram',+ type: 'social',+ },+ youTube: { url: 'https://www.youtube.com', name: 'Youtube', type: 'social' },+ gitHub: { url: 'https://github.com', name: 'GitHub', type: 'tech' },+ chatGpt: { url: 'https://chatgpt.com', name: 'ChatGPT', type: 'ai' },+ claude: { url: 'https://claude.ai', name: 'Claude', type: 'ai' },+ medium: { url: 'https://medium.com', name: 'Medium', type: 'content' },+ substack: { url: 'https://substack.com', name: 'Substack', type: 'content' },+ gmail: { url: 'https://mail.google.com', name: 'Gmail', type: 'email' },+ productHunt: {+ url: 'https://www.producthunt.com',+ name: 'Product Hunt',+ type: 'social',+ },+} as const satisfies Record<string, Referrer>;++export type ReferrerMix = readonly Weighted<Referrer | null>[];++export interface Campaign {+ source: string;+ medium: string;+ name: string;+}++export const CAMPAIGNS: readonly Weighted<Campaign>[] = [+ { weight: 5, value: { source: 'google', medium: 'cpc', name: 'brand' } },+ { weight: 4, value: { source: 'google', medium: 'cpc', name: 'generic' } },+ {+ weight: 3,+ value: { source: 'facebook', medium: 'paid_social', name: 'autumn-sale' },+ },+ {+ weight: 3,+ value: { source: 'instagram', medium: 'paid_social', name: 'new-arrivals' },+ },+ {+ weight: 2,+ value: { source: 'newsletter', medium: 'email', name: 'weekly-digest' },+ },+ {+ weight: 2,+ value: { source: 'linkedin', medium: 'paid_social', name: 'b2b-launch' },+ },+ {+ weight: 1,+ value: { source: 'producthunt', medium: 'referral', name: 'launch' },+ },+];++/** By name and by the hostname's own label, the two ways a `utm_source` names a referrer. */+const KNOWN_SOURCES: Record<string, Referrer> = Object.fromEntries(+ Object.values(REFERRERS).flatMap((referrer) => {+ const label =+ new URL(referrer.url).hostname+ .replace(/^(www|m|l)\./, '')+ .split('.')[0] ?? '';+ return [+ [referrer.name.toLowerCase(), referrer],+ [label, referrer],+ ];+ })+);++/**+ * What the ingest handler stores for a landing event: the referrer's URL, but a+ * `utm_source` that names a known referrer overrides its name and type, and+ * an unknown one becomes the name with no type (`getReferrerWithQuery`).+ */+export function sessionReferrer(+ referrer: Referrer | null,+ query: Record<string, string> | undefined+): Referrer {+ const source = (+ query?.utm_source ??+ query?.ref ??+ query?.utm_referrer ??+ ''+ ).toLowerCase();+ const fromQuery =+ source === ''+ ? null+ : (KNOWN_SOURCES[source] ?? { url: '', name: source, type: '' });+ return {+ url: referrer?.url ?? '',+ name: fromQuery?.name || referrer?.name || referrer?.url || '',+ type: fromQuery?.type || referrer?.type || '',+ };+}diff --git a/packages/seed/src/generator.test.ts b/packages/seed/src/generator.test.tsnew file mode 100644index 000000000..7492bef48--- /dev/null+++ b/packages/seed/src/generator.test.ts@@ -0,0 +1,369 @@+import { describe, expect, test } from 'bun:test';+import { createHash } from 'node:crypto';+import { ARCHETYPES } from './archetypes';+import type {+ ClickhouseEventRow,+ ClickhouseProfileRow,+ ClickhouseSessionRow,+} from './clickhouse-rows';+import { type GenerateConfig, generate, planDays } from './generator';+import { projectIdFor } from './ids';+import type { RowSink, SinkCounts } from './sink';++class MemorySink implements RowSink {+ events: ClickhouseEventRow[] = [];+ sessions: ClickhouseSessionRow[] = [];+ profiles: ClickhouseProfileRow[] = [];++ addEvents(rows: readonly ClickhouseEventRow[]): void {+ this.events.push(...rows);+ }+ addSession(row: ClickhouseSessionRow): void {+ this.sessions.push(row);+ }+ addProfiles(rows: readonly ClickhouseProfileRow[]): void {+ this.profiles.push(...rows);+ }+ flushFull(): Promise<void> {+ return Promise.resolve();+ }+ flush(): Promise<void> {+ return Promise.resolve();+ }+ counts(): SinkCounts {+ return {+ events: this.events.length,+ sessions: this.sessions.length,+ profiles: this.profiles.length,+ };+ }+}++const NOW = new Date('2026-09-20T12:00:00.000Z');+const ignoreProgress = () => undefined;++function config(overrides: Partial<GenerateConfig> = {}): GenerateConfig {+ return {+ seed: 7,+ sessionsPerDay: 200,+ days: 3,+ now: NOW,+ variance: 1,+ projects: ARCHETYPES.map((archetype) => ({+ archetype,+ projectId: projectIdFor(archetype),+ })),+ ...overrides,+ };+}++async function run(+ overrides: Partial<GenerateConfig> = {}+): Promise<MemorySink> {+ const sink = new MemorySink();+ await generate(config(overrides), sink, ignoreProgress);+ return sink;+}++/** `inserted_at` is stamped at insert time by design; everything else must be reproducible. */+function fingerprint(sink: MemorySink): string {+ const hash = createHash('sha256');+ for (const event of sink.events) {+ hash.update(JSON.stringify({ ...event, inserted_at: undefined }));+ }+ for (const session of sink.sessions) {+ hash.update(JSON.stringify(session));+ }+ for (const profile of sink.profiles) {+ hash.update(JSON.stringify(profile));+ }+ return hash.digest('hex');+}++describe('generate', () => {+ test('is deterministic for a seed and differs across seeds', async () => {+ const [first, second, other] = await Promise.all([+ run(),+ run(),+ run({ seed: 8 }),+ ]);+ expect(first.events.length).toBeGreaterThan(0);+ expect(fingerprint(first)).toBe(fingerprint(second));+ expect(fingerprint(other)).not.toBe(fingerprint(first));+ });++ test('plans roughly the requested volume', () => {+ const planned = planDays(config()).reduce(+ (sum, day) => sum + day.sessions,+ 0+ );+ expect(planned).toBeGreaterThan(200 * 3 * 0.4);+ expect(planned).toBeLessThan(200 * 3 * 2.5);+ });++ test('every session has one session_start, one session_end and a matching sessions row', async () => {+ const sink = await run();+ const bySession = new Map<string, ClickhouseEventRow[]>();+ for (const event of sink.events) {+ const list = bySession.get(event.session_id) ?? [];+ list.push(event);+ bySession.set(event.session_id, list);+ }+ expect(bySession.size).toBe(sink.sessions.length);++ for (const session of sink.sessions) {+ const events = bySession.get(session.id);+ expect(events).toBeDefined();+ if (!events) {+ continue;+ }+ const names = events.map((event) => event.name);+ expect(names.filter((name) => name === 'session_start')).toHaveLength(1);+ expect(names.filter((name) => name === 'session_end')).toHaveLength(1);+ expect(session.sign).toBe(1);+ expect(session.screen_view_count + session.event_count).toBe(+ events.length - 2+ );+ expect(session.is_bounce).toBe(session.screen_view_count <= 1);+ expect(session.duration).toBeGreaterThanOrEqual(0);+ for (const event of events) {+ expect(event.project_id).toBe(session.project_id);+ expect(event.device_id).toBe(session.device_id);+ expect(event.profile_id).not.toBe('');+ }+ const end = events.find((event) => event.name === 'session_end');+ expect(end?.properties.__bounce).toBe(String(session.is_bounce));+ expect(end?.duration).toBe(session.duration);+ }+ });++ test('events are ordered within a session and never in the future', async () => {+ const sink = await run();+ for (const event of sink.events) {+ expect(new Date(`${event.created_at}Z`).getTime()).toBeLessThanOrEqual(+ NOW.getTime() + 1000+ );+ }+ });++ test('profiles cover every profile_id the events reference, once each', async () => {+ const sink = await run();+ const referenced = new Set(+ sink.events.map((event) => `${event.project_id}/${event.profile_id}`)+ );+ const written = sink.profiles.map(+ (profile) => `${profile.project_id}/${profile.id}`+ );+ expect(new Set(written).size).toBe(written.length);+ for (const key of referenced) {+ expect(written).toContain(key);+ }+ const identified = sink.profiles.filter((profile) => profile.is_external);+ expect(identified.length).toBeGreaterThan(0);+ for (const profile of identified) {+ expect(profile.email).toContain('@');+ }+ });++ test('device and referrer fields come from the real parsers', async () => {+ const sink = await run();+ const devices = new Set(sink.events.map((event) => event.device));+ expect(+ [...devices].every((device) =>+ ['desktop', 'mobile', 'tablet'].includes(device)+ )+ ).toBe(true);+ const referrerTypes = new Set(+ sink.sessions.map((session) => session.referrer_type)+ );+ expect(referrerTypes.has('search')).toBe(true);+ expect(referrerTypes.has('social')).toBe(true);+ const withUtm = sink.sessions.filter(+ (session) => session.utm_source !== ''+ );+ expect(withUtm.length).toBeGreaterThan(0);+ });+});++const MS_PER_DAY = 86_400_000;++function dayOf(event: ClickhouseEventRow): string {+ return event.created_at.slice(0, 10);+}++function eventsOf(+ sink: MemorySink,+ projectId: string,+ name?: string+): ClickhouseEventRow[] {+ return sink.events.filter(+ (event) =>+ event.project_id === projectId &&+ (name === undefined || event.name === name)+ );+}++describe('identity and retention', () => {+ test('anonymous devices are recognised within a UTC day and never across days', async () => {+ const sink = await run({ days: 10 });+ const daysPerDevice = new Map<string, Set<string>>();+ const sessionsPerDevice = new Map<string, Set<string>>();+ for (const event of eventsOf(sink, 'acme-web')) {+ if (event.profile_id !== event.device_id) {+ continue;+ }+ daysPerDevice.set(+ event.device_id,+ (daysPerDevice.get(event.device_id) ?? new Set()).add(dayOf(event))+ );+ sessionsPerDevice.set(+ event.device_id,+ (sessionsPerDevice.get(event.device_id) ?? new Set()).add(+ event.session_id+ )+ );+ }+ expect(daysPerDevice.size).toBeGreaterThan(100);+ for (const days of daysPerDevice.values()) {+ expect(days.size).toBe(1);+ }+ const sameDayReturns = [...sessionsPerDevice.values()].filter(+ (sessions) => sessions.size >= 2+ ).length;+ expect(sameDayReturns).toBeGreaterThan(10);+ });++ test('identified users keep their profile across days, so retention exists', async () => {+ const sink = await run({ days: 14, sessionsPerDay: 300 });+ for (const projectId of ['acme-saas', 'acme-app']) {+ const firstDay = new Map<string, number>();+ const activeDays = new Map<string, Set<number>>();+ for (const event of eventsOf(sink, projectId)) {+ if (event.profile_id === event.device_id) {+ continue;+ }+ const day = Math.floor(+ new Date(`${event.created_at}Z`).getTime() / MS_PER_DAY+ );+ firstDay.set(+ event.profile_id,+ Math.min(firstDay.get(event.profile_id) ?? day, day)+ );+ activeDays.set(+ event.profile_id,+ (activeDays.get(event.profile_id) ?? new Set()).add(day)+ );+ }+ // Cohorts old enough to have a day-7: those who started in the first 5 days.+ const cutoff = Math.min(...firstDay.values()) + 5;+ const cohort = [...firstDay].filter(([, day]) => day <= cutoff);+ const retainedDay7 = cohort.filter(([id, day]) =>+ activeDays.get(id)?.has(day + 7)+ ).length;+ expect(cohort.length).toBeGreaterThan(50);+ expect(retainedDay7 / cohort.length).toBeGreaterThan(0.1);+ }+ });++ test('a device that identifies mid-session leaves both an anonymous and a person profile', async () => {+ const sink = await run({ days: 5 });+ const signups = eventsOf(sink, 'acme-web', 'signup_completed');+ expect(signups.length).toBeGreaterThan(0);+ const profileIds = new Set(+ sink.profiles+ .filter((profile) => profile.project_id === 'acme-web')+ .map((profile) => profile.id)+ );+ for (const signup of signups) {+ expect(signup.profile_id).not.toBe(signup.device_id);+ expect(profileIds.has(signup.profile_id)).toBe(true);+ expect(profileIds.has(signup.device_id)).toBe(true);+ }+ });+});++describe('funnels', () => {+ /** Within-session funnels link by session; the SaaS activation funnel spans visits, so by person. */+ const FUNNELS: Record<+ string,+ { by: 'session_id' | 'profile_id'; steps: string[] }+ > = {+ 'acme-shop': {+ by: 'session_id',+ steps: [+ 'product_viewed',+ 'add_to_cart',+ 'checkout_started',+ 'shipping_info_added',+ 'payment_info_added',+ 'purchase',+ ],+ },+ 'acme-saas': {+ by: 'profile_id',+ steps: [+ 'signup_completed',+ 'project_created',+ 'sdk_installed',+ 'first_event_received',+ ],+ },+ 'acme-app': {+ by: 'profile_id',+ steps: [+ 'app_opened',+ 'signup_completed',+ 'notification_permission',+ 'post_created',+ ],+ },+ 'acme-web': {+ by: 'session_id',+ steps: [+ 'screen_view',+ 'plan_selected',+ 'signup_started',+ 'signup_completed',+ ],+ },+ };++ test('each step is reached by fewer users than the one before, and the last by some', async () => {+ const sink = await run({ days: 10, sessionsPerDay: 300 });+ for (const [projectId, { by, steps }] of Object.entries(FUNNELS)) {+ // Funnel semantics: a user counts for a step only having done every step before it.+ let reached: Set<string> | null = null;+ const counts: number[] = [];+ for (const step of steps) {+ const users = new Set(+ eventsOf(sink, projectId, step).map((event) => event[by])+ );+ const previous: Set<string> = reached ?? users;+ reached = new Set([...previous].filter((user) => users.has(user)));+ counts.push(reached.size);+ }+ expect(counts.at(-1)).toBeGreaterThan(0);+ expect(counts.at(-1)).toBeLessThan(counts[0] ?? 0);+ }+ });++ test('purchases carry breakdown-worthy properties and revenue', async () => {+ const sink = await run({ days: 10, sessionsPerDay: 300 });+ const purchases = eventsOf(sink, 'acme-shop', 'purchase');+ expect(purchases.length).toBeGreaterThan(5);+ const payments = new Set(+ purchases.map((event) => event.properties.payment)+ );+ expect(payments.size).toBeGreaterThan(1);+ expect(+ purchases.every((event) => event.properties.order_id?.startsWith('ord_'))+ ).toBe(true);+ const revenue = eventsOf(sink, 'acme-shop', 'revenue');+ expect(revenue.length).toBe(purchases.length);+ expect(+ revenue.every(+ (event) => Number.isInteger(event.revenue) && (event.revenue ?? 0) > 0+ )+ ).toBe(true);+ });+});diff --git a/packages/seed/src/generator.ts b/packages/seed/src/generator.tsnew file mode 100644index 000000000..222de45c0--- /dev/null+++ b/packages/seed/src/generator.ts@@ -0,0 +1,229 @@+// Walks every project day by day, hour by hour, and turns each planned+// session into rows. Sessions are produced in time order per project so the+// returning-visitor pool never hands out a visitor from the future.++import type { Archetype } from './archetypes/archetype';+import { Journey } from './archetypes/archetype';+import { CAMPAIGNS, type Campaign } from './data/referrers';+import { Rng } from './rng';+import { buildSessionRows } from './rows';+import type { RowSink } from './sink';+import { TrafficModel } from './traffic';+import { World } from './world';++export interface ProjectTarget {+ archetype: Archetype;+ projectId: string;+}++export interface GenerateConfig {+ seed: number;+ /** Sessions per day across all projects; each archetype takes its `trafficShare`. */+ sessionsPerDay: number;+ days: number;+ now: Date;+ variance: number;+ projects: readonly ProjectTarget[];+}++export interface DayPlan {+ archetype: Archetype;+ date: Date;+ sessions: number;+}++export interface ProjectStats {+ projectId: string;+ sessions: number;+ events: number;+ firstEventAt: Date | null;+}++export interface Progress {+ day: string;+ sessions: number;+ events: number;+}++const MS_PER_DAY = 86_400_000;+const MS_PER_HOUR = 3_600_000;+const HOURS_PER_DAY = 24;+const SECONDS_PER_HOUR = 3600;+const PROGRESS_EVERY_SESSIONS = 5000;++function utcMidnight(date: Date): Date {+ return new Date(+ Date.UTC(date.getUTCFullYear(), date.getUTCMonth(), date.getUTCDate())+ );+}++function trafficModel(+ config: GenerateConfig,+ archetype: Archetype+): TrafficModel {+ return new TrafficModel(archetype.id, archetype.shape, {+ seed: String(config.seed),+ variance: config.variance,+ now: config.now,+ });+}++/** The days the run covers, oldest first, each ending before `now`. */+function daysOf(config: GenerateConfig): Date[] {+ const today = utcMidnight(config.now);+ const days: Date[] = [];+ for (let offset = config.days - 1; offset >= 0; offset--) {+ days.push(new Date(today.getTime() - offset * MS_PER_DAY));+ }+ return days;+}++export function planDays(config: GenerateConfig): DayPlan[] {+ const plan: DayPlan[] = [];+ for (const { archetype } of config.projects) {+ const model = trafficModel(config, archetype);+ const base = config.sessionsPerDay * archetype.trafficShare;+ for (const date of daysOf(config)) {+ plan.push({+ archetype,+ date,+ sessions: model.dayFor(date, base).visitors,+ });+ }+ }+ return plan;+}++function utmFor(campaign: Campaign): Record<string, string> {+ return {+ utm_source: campaign.source,+ utm_medium: campaign.medium,+ utm_campaign: campaign.name,+ };+}++/** Session start times within one UTC day, following the archetype's local-hour curve. */+function sessionStarts(+ rng: Rng,+ model: TrafficModel,+ archetype: Archetype,+ date: Date,+ total: number,+ now: Date+): Date[] {+ const weights = model.hourWeights(date);+ const starts: Date[] = [];+ for (let utcHour = 0; utcHour < HOURS_PER_DAY; utcHour++) {+ const localHour =+ (((utcHour + archetype.utcOffsetHours) % HOURS_PER_DAY) + HOURS_PER_DAY) %+ HOURS_PER_DAY;+ const count = rng.poisson(total * (weights[localHour] ?? 0));+ for (let i = 0; i < count; i++) {+ const start = new Date(+ date.getTime() ++ utcHour * MS_PER_HOUR ++ rng.int(0, SECONDS_PER_HOUR - 1) * 1000+ );+ if (start.getTime() < now.getTime()) {+ starts.push(start);+ }+ }+ }+ return starts.sort((a, b) => a.getTime() - b.getTime());+}++export async function generate(+ config: GenerateConfig,+ sink: RowSink,+ onProgress: (progress: Progress) => void+): Promise<ProjectStats[]> {+ const stats: ProjectStats[] = [];+ const days = daysOf(config);+ let sessionsSinceProgress = 0;++ for (const { archetype, projectId } of config.projects) {+ const model = trafficModel(config, archetype);+ const base = config.sessionsPerDay * archetype.trafficShare;+ const world = new World(+ archetype,+ projectId,+ days[0] ?? config.now,+ (rows) => sink.addProfiles(rows)+ );+ const projectStats: ProjectStats = {+ projectId,+ sessions: 0,+ events: 0,+ firstEventAt: null,+ };++ for (const date of days) {+ const dayKey = date.toISOString().slice(0, 10);+ const rng = new Rng(`${config.seed}/${archetype.id}/${dayKey}`);+ const planned = model.dayFor(date, base).visitors;++ for (const [index, start] of sessionStarts(+ rng,+ model,+ archetype,+ date,+ planned,+ config.now+ ).entries()) {+ const visitor = world.pick(rng, start);+ const referrer = rng.pick(archetype.referrers);+ const utm = rng.chance(archetype.campaignChance)+ ? utmFor(rng.pick(CAMPAIGNS))+ : {};+ // A returning identified user is recognised from the first event; a first visit starts anonymous.+ const journey = new Journey(+ rng,+ start,+ archetype.dwellMedianSeconds,+ visitor.identified ? visitor.person : null+ );+ archetype.journey(rng, journey, visitor);+ if (journey.events.length === 0) {+ continue;+ }++ const rows = buildSessionRows(rng, {+ id: rng.sessionId(),+ projectId,+ archetype,+ visitor,+ referrer,+ utm,+ events: journey.events,+ });+ world.touch(visitor, rows.endedAt, {+ identified: rows.identified,+ anonymousEvents: rows.anonymousEvents,+ });+ sink.addEvents(rows.events);+ sink.addSession(rows.session);++ projectStats.sessions += 1;+ projectStats.events += rows.events.length;+ projectStats.firstEventAt ??= start;+ sessionsSinceProgress += 1;+ if (sessionsSinceProgress >= PROGRESS_EVERY_SESSIONS || index === 0) {+ sessionsSinceProgress = 0;+ const counts = sink.counts();+ onProgress({+ day: `${archetype.id} ${dayKey}`,+ sessions: counts.sessions,+ events: counts.events,+ });+ }+ await sink.flushFull();+ }+ }++ world.drain();+ stats.push(projectStats);+ }++ await sink.flush();+ return stats;+}diff --git a/packages/seed/src/ids.ts b/packages/seed/src/ids.tsnew file mode 100644index 000000000..5ca134c2d--- /dev/null+++ b/packages/seed/src/ids.ts@@ -0,0 +1,23 @@+import { createHash } from 'node:crypto';+import { slug } from '@openpanel/shared';+import type { Archetype } from './archetypes/archetype';+import { formatUuid } from './rng';+import { ID_NAMESPACE } from './seed.constants';++function namespaceBytes(): Buffer {+ return Buffer.from(ID_NAMESPACE.replace(/-/g, ''), 'hex');+}++/** RFC 4122 v5: the same name always yields the same uuid, so re-runs upsert. */+export function deterministicUuid(name: string): string {+ const digest = createHash('sha1')+ .update(namespaceBytes())+ .update(name)+ .digest();+ return formatUuid(digest.subarray(0, 16).toString('hex'), '5');+}++/** The same slug id the onboarding flow would mint for the project name. */+export function projectIdFor(archetype: Archetype): string {+ return slug(archetype.projectName);+}diff --git a/packages/seed/src/manifest.ts b/packages/seed/src/manifest.tsnew file mode 100644index 000000000..5733710f8--- /dev/null+++ b/packages/seed/src/manifest.ts@@ -0,0 +1,114 @@+// `.seed.json`: what was seeded, for agents, the Playwright suite and whatever tooling+// describes the worktree to them.++import { writeFileSync } from 'node:fs';+import type { PostgresSeed } from './postgres';+import { SEED_USER } from './seed.constants';++export interface SeedManifest {+ seed: number;+ size: string;+ days: number;+ sessionsPerDay: number;+ range: { from: string; to: string };+ seededAt: string;+ user: { email: string; password: string };+ organizationId: string;+ rootClient: { id: string; secret: string };+ /** The API this seed was made for (from `API_URL`), and the product's own MCP endpoint on it. */+ apiUrl: string | null;+ mcp: { url: string; token: string } | null;+ projects: {+ id: string;+ name: string;+ archetype: string;+ clientId: string;+ clientSecret: string;+ sessions: number;+ events: number;+ /** How users are identified in this project: what retention can show. */+ identity: string;+ conversions: string[];+ funnels: { name: string; steps: string[]; breakdowns: string[] }[];+ }[];+ totals: { sessions: number; events: number; profiles: number };+}++/** What the MCP settings page hands out: `<api>/mcp?token=base64(clientId:clientSecret)` for a read or root client. */+function mcpEndpoint(+ apiUrl: string | null,+ client: { id: string; secret: string }+): { url: string; token: string } | null {+ if (!apiUrl) {+ return null;+ }+ const token = Buffer.from(`${client.id}:${client.secret}`).toString('base64');+ return { url: `${apiUrl.replace(/\/$/, '')}/mcp?token=${token}`, token };+}++export function buildManifest(input: {+ seed: number;+ size: string;+ days: number;+ sessionsPerDay: number;+ from: Date;+ to: Date;+ postgres: PostgresSeed;+ apiUrl: string | null;+ perProject: Map<string, { sessions: number; events: number }>;+ totals: { sessions: number; events: number; profiles: number };+}): SeedManifest {+ return {+ seed: input.seed,+ size: input.size,+ days: input.days,+ sessionsPerDay: input.sessionsPerDay,+ range: { from: input.from.toISOString(), to: input.to.toISOString() },+ seededAt: new Date().toISOString(),+ user: { email: SEED_USER.email, password: SEED_USER.password },+ organizationId: input.postgres.organizationId,+ rootClient: input.postgres.rootClient,+ apiUrl: input.apiUrl,+ mcp: mcpEndpoint(input.apiUrl, input.postgres.rootClient),+ projects: input.postgres.projects.map((project) => ({+ id: project.id,+ name: project.name,+ archetype: project.archetype,+ clientId: project.client.id,+ clientSecret: project.client.secret,+ sessions: input.perProject.get(project.id)?.sessions ?? 0,+ events: input.perProject.get(project.id)?.events ?? 0,+ identity: project.identity,+ conversions: [...project.conversions],+ funnels: project.funnels.map((funnel) => ({+ name: funnel.name,+ steps: [...funnel.steps],+ breakdowns: [...funnel.breakdowns],+ })),+ })),+ totals: input.totals,+ };+}++export function writeManifest(path: string, manifest: SeedManifest): void {+ writeFileSync(path, `${JSON.stringify(manifest, null, 2)}\n`);+}++export function describeManifest(manifest: SeedManifest): string {+ const lines = [+ `Seeded ${manifest.totals.sessions.toLocaleString()} sessions / ${manifest.totals.events.toLocaleString()} events / ${manifest.totals.profiles.toLocaleString()} profiles`,+ `Range: ${manifest.range.from.slice(0, 10)} → ${manifest.range.to.slice(0, 10)} (${manifest.days} days, size ${manifest.size}, seed ${manifest.seed})`,+ `Login: ${manifest.user.email} / ${manifest.user.password}`,+ `Organization: ${manifest.organizationId}`,+ 'Projects:',+ ...manifest.projects.map(+ (project) =>+ ` ${project.name} (${project.archetype}) id ${project.id} client ${project.clientId} secret ${project.clientSecret}`+ ),+ `Root client: ${manifest.rootClient.id} secret ${manifest.rootClient.secret}`,+ manifest.mcp+ ? `MCP (streamable HTTP, root client): ${manifest.mcp.url}`+ : 'MCP: set API_URL to get the endpoint',+ ];+ return lines.join('\n');+}diff --git a/packages/seed/src/model.ts b/packages/seed/src/model.tsnew file mode 100644index 000000000..54feed5c8--- /dev/null+++ b/packages/seed/src/model.ts@@ -0,0 +1,65 @@+// The in-memory shapes the generator produces before they become ClickHouse rows.++import type { parseUserAgent } from '@openpanel/shared/server';++export type ParsedDevice = ReturnType<typeof parseUserAgent>;++export interface Geo {+ ip: string;+ country: string;+ region: string;+ city: string;+ latitude: number;+ longitude: number;+}++export interface Person {+ id: string;+ firstName: string;+ lastName: string;+ email: string;+ avatar: string;+ properties: Record<string, string>;+}++export interface Referrer {+ url: string;+ name: string;+ type: string;+}++/**+ * The device id ingest mints from user agent + IP + a salt that rotates at+ * midnight UTC: the same browser is one device within a UTC day and a new+ * one the next. Anonymous analytics only ever sees these.+ */+export interface DeviceWindow {+ id: string;+ /** The UTC day this id is valid for. */+ day: string;+ firstSeen: Date;+ lastSeen: Date;+ /** Whether any event was sent anonymously from it, which is what leaves a profile behind. */+ anonymousEvents: boolean;+}++/** A person (or an unidentified browser) that can come back. */+export interface Visitor {+ userAgent: string;+ device: ParsedDevice;+ geo: Geo;+ person: Person | null;+ /** `identify()` has been sent once, so later sessions start identified. */+ identified: boolean;+ window: DeviceWindow;+ /** Days since the run's first day, for retention cohorts. */+ cohortDay: number;+ sessions: number;+ firstSeen: Date;+ lastSeen: Date;+ /** What the worker snapshots onto the profile at each session boundary. */+ lastPath: string;+ lastReferrer: Referrer | null;+ /** Archetype-specific memory (plan, cart, ...). */+ state: Record<string, unknown>;+}diff --git a/packages/seed/src/postgres.ts b/packages/seed/src/postgres.tsnew file mode 100644index 000000000..aacc1f2fb--- /dev/null+++ b/packages/seed/src/postgres.ts@@ -0,0 +1,210 @@+// The Postgres side: one login, one organization, one project per archetype+// and a write client each, plus a root client for /manage. Mirrors what the+// onboarding flow creates (onboarding.service.ts), with deterministic ids so a+// re-run updates in place.++import { hash as argon2Hash } from '@node-rs/argon2';+import { db } from '@openpanel/db';+import { hashPassword } from '@openpanel/shared/server';+import type { Archetype } from './archetypes/archetype';+import { deterministicUuid, projectIdFor } from './ids';+import type { Rng } from './rng';+import {+ ORGANIZATION_ADMIN_ROLE,+ SEED_ORGANIZATION,+ SEED_USER,+ TRIAL_DAYS,+ TRIAL_EVENTS_LIMIT,+} from './seed.constants';++const MS_PER_DAY = 86_400_000;+const SECRET_HEX_CHARS = 20;+/** Sign-in accepts `$argon2` hashes made with these parameters; copied from core's auth `hashPassword`. */+const ARGON2_OPTIONS = {+ memoryCost: 19_456,+ timeCost: 2,+ outputLen: 32,+ parallelism: 1,+} as const;++export interface SeededClient {+ id: string;+ secret: string;+}++export interface SeededProject {+ id: string;+ name: string;+ archetype: Archetype['id'];+ client: SeededClient;+ identity: string;+ conversions: readonly string[];+ funnels: Archetype['funnels'];+}++export interface PostgresSeed {+ userId: string;+ organizationId: string;+ rootClient: SeededClient;+ projects: SeededProject[];+}++function newSecret(rng: Rng): string {+ return `sec_${rng.hex(SECRET_HEX_CHARS)}`;+}++async function upsertUser(): Promise<string> {+ const user = await db.user.upsert({+ where: { email: SEED_USER.email },+ create: {+ id: SEED_USER.id,+ email: SEED_USER.email,+ firstName: SEED_USER.firstName,+ lastName: SEED_USER.lastName,+ },+ update: {+ firstName: SEED_USER.firstName,+ lastName: SEED_USER.lastName,+ deletedAt: null,+ },+ });+ const password = await argon2Hash(SEED_USER.password, ARGON2_OPTIONS);+ const account = await db.account.findFirst({+ where: { userId: user.id, provider: 'email' },+ });+ if (account) {+ await db.account.update({ where: { id: account.id }, data: { password } });+ } else {+ await db.account.create({+ data: { userId: user.id, provider: 'email', email: user.email, password },+ });+ }+ return user.id;+}++async function upsertOrganization(userId: string): Promise<string> {+ const organization = await db.organization.upsert({+ where: { id: SEED_ORGANIZATION.id },+ create: {+ id: SEED_ORGANIZATION.id,+ name: SEED_ORGANIZATION.name,+ createdByUserId: userId,+ subscriptionStatus: 'trialing',+ subscriptionEndsAt: new Date(Date.now() + TRIAL_DAYS * MS_PER_DAY),+ subscriptionPeriodEventsLimit: TRIAL_EVENTS_LIMIT,+ timezone: 'UTC',+ onboarding: '',+ },+ update: { name: SEED_ORGANIZATION.name, deleteAt: null },+ });+ await db.member.upsert({+ where: {+ organizationId_userId: { organizationId: organization.id, userId },+ },+ create: {+ organizationId: organization.id,+ userId,+ email: SEED_USER.email,+ role: ORGANIZATION_ADMIN_ROLE,+ },+ update: { role: ORGANIZATION_ADMIN_ROLE },+ });+ return organization.id;+}++/** Ingest verifies client secrets with scrypt (`verifyPassword`), so that is what the seed stores. */+async function upsertClient(+ rng: Rng,+ input: {+ name: string;+ organizationId: string;+ projectId: string | null;+ type: 'write' | 'root';+ }+): Promise<SeededClient> {+ const id = deterministicUuid(`client:${input.projectId ?? 'root'}`);+ const secret = newSecret(rng);+ await db.client.upsert({+ where: { id },+ create: {+ id,+ name: input.name,+ organizationId: input.organizationId,+ projectId: input.projectId,+ type: input.type,+ secret: await hashPassword(secret),+ },+ update: { name: input.name, secret: await hashPassword(secret) },+ });+ return { id, secret };+}++async function upsertProject(+ rng: Rng,+ organizationId: string,+ archetype: Archetype+): Promise<SeededProject> {+ const id = projectIdFor(archetype);+ await db.project.upsert({+ where: { id },+ create: {+ id,+ name: archetype.projectName,+ organizationId,+ types: archetype.types,+ domain: archetype.domain,+ cors: archetype.origin ? ['*'] : [],+ },+ update: {+ name: archetype.projectName,+ types: archetype.types,+ domain: archetype.domain,+ deleteAt: null,+ },+ });+ const client = await upsertClient(rng, {+ name: `${archetype.projectName} Client`,+ organizationId,+ projectId: id,+ type: 'write',+ });+ return {+ id,+ name: archetype.projectName,+ archetype: archetype.id,+ client,+ identity: archetype.identity,+ conversions: archetype.conversions,+ funnels: archetype.funnels,+ };+}++export async function seedPostgres(+ rng: Rng,+ archetypes: readonly Archetype[]+): Promise<PostgresSeed> {+ const userId = await upsertUser();+ const organizationId = await upsertOrganization(userId);+ const rootClient = await upsertClient(rng, {+ name: 'Seed root client',+ organizationId,+ projectId: null,+ type: 'root',+ });+ const projects: SeededProject[] = [];+ for (const archetype of archetypes) {+ projects.push(await upsertProject(rng, organizationId, archetype));+ }+ return { userId, organizationId, rootClient, projects };+}++/** What the ingest path maintains on the project row once events arrive. */+export async function recordProjectStats(+ projectId: string,+ stats: { eventsCount: number; firstEventAt: Date | null }+): Promise<void> {+ await db.project.update({+ where: { id: projectId },+ data: { eventsCount: stats.eventsCount, firstEventAt: stats.firstEventAt },+ });+}diff --git a/packages/seed/src/reset.ts b/packages/seed/src/reset.tsnew file mode 100644index 000000000..c05e012e2--- /dev/null+++ b/packages/seed/src/reset.ts@@ -0,0 +1,37 @@+// Removes what an earlier run seeded for the same projects. Postgres rows are+// upserted in place, so only ClickHouse needs clearing; the materialized views+// are not rebuilt from `events` on delete and must be cleared explicitly.++import { ch } from '@openpanel/db';++const TABLES_WITH_PROJECT_ROWS = ['events', 'sessions', 'profiles'] as const;++const MATERIALIZED_VIEWS_WITH_PROJECT_ROWS = [+ 'dau_mv',+ 'distinct_event_names_mv',+ 'event_property_values_mv',+ 'cohort_events_mv',+ 'event_profile_summary_mv',+ 'event_property_profile_summary_mv',+] as const;++function quoted(value: string): string {+ return `'${value.replace(/'/g, "''")}'`;+}++export async function resetProjects(+ projectIds: readonly string[]+): Promise<void> {+ const list = projectIds.map(quoted).join(', ');+ for (const table of TABLES_WITH_PROJECT_ROWS) {+ await ch.command({+ query: `DELETE FROM ${table} WHERE project_id IN (${list})`,+ });+ }+ for (const view of MATERIALIZED_VIEWS_WITH_PROJECT_ROWS) {+ await ch.command({+ query: `ALTER TABLE ${view} DELETE WHERE project_id IN (${list})`,+ clickhouse_settings: { mutations_sync: '2' },+ });+ }+}diff --git a/packages/seed/src/rng.ts b/packages/seed/src/rng.tsnew file mode 100644index 000000000..ea8a5f4be--- /dev/null+++ b/packages/seed/src/rng.ts@@ -0,0 +1,185 @@+// A small deterministic PRNG. Every random choice in the seed goes through an+// `Rng` derived from the run's seed, so the same flags produce the same rows.++/* biome-ignore-all lint/suspicious/noBitwiseOperators: sfc32 is defined in 32-bit integer arithmetic */++import { createHash } from 'node:crypto';++const UUID_VARIANT_NIBBLES = ['8', '9', 'a', 'b'] as const;++/** 32 hex chars into RFC 4122 shape, stamping the version and variant nibbles. */+export function formatUuid(hex: string, version: '4' | '5'): string {+ const variant =+ UUID_VARIANT_NIBBLES[+ Number.parseInt(hex[16] ?? '0', 16) % UUID_VARIANT_NIBBLES.length+ ] ?? '8';+ return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${version}${hex.slice(13, 16)}-${variant}${hex.slice(17, 20)}-${hex.slice(20, 32)}`;+}++export interface Weighted<T> {+ value: T;+ weight: number;+}++const UINT32_RANGE = 4_294_967_296;+const POISSON_NORMAL_APPROXIMATION_THRESHOLD = 60;++function seedWords(seed: string): [number, number, number, number] {+ const digest = createHash('sha256').update(seed).digest();+ return [+ digest.readUInt32LE(0),+ digest.readUInt32LE(4),+ digest.readUInt32LE(8),+ digest.readUInt32LE(12),+ ];+}++/** sfc32: fast, 128-bit state, good enough statistics for synthetic traffic. */+export class Rng {+ private a: number;+ private b: number;+ private c: number;+ private d: number;+ private spareNormal: number | null = null;++ constructor(private readonly seed: string) {+ [this.a, this.b, this.c, this.d] = seedWords(seed);+ for (let i = 0; i < 12; i++) {+ this.next();+ }+ }++ /** A child generator whose stream is independent of the parent's position. */+ fork(...salt: (string | number)[]): Rng {+ return new Rng(`${this.seed}/${salt.join('/')}`);+ }++ private next(): number {+ this.a >>>= 0;+ this.b >>>= 0;+ this.c >>>= 0;+ this.d >>>= 0;+ let t = (this.a + this.b) | 0;+ this.a = this.b ^ (this.b >>> 9);+ this.b = (this.c + (this.c << 3)) | 0;+ this.c = (this.c << 21) | (this.c >>> 11);+ this.d = (this.d + 1) | 0;+ t = (t + this.d) | 0;+ this.c = (this.c + t) | 0;+ return t >>> 0;+ }++ /** Uniform in [0, 1). */+ float(): number {+ return this.next() / UINT32_RANGE;+ }++ /** Uniform integer in [low, high]. */+ int(low: number, high: number): number {+ if (high <= low) {+ return low;+ }+ return low + Math.floor(this.float() * (high - low + 1));+ }++ chance(probability: number): boolean {+ return this.float() < probability;+ }++ one<T>(items: readonly T[]): T {+ const item = items[Math.floor(this.float() * items.length)];+ if (item === undefined) {+ throw new Error('Rng.one called with an empty list');+ }+ return item;+ }++ pick<T>(items: readonly Weighted<T>[]): T {+ let total = 0;+ for (const item of items) {+ total += item.weight;+ }+ let r = this.float() * total;+ for (const item of items) {+ r -= item.weight;+ if (r <= 0) {+ return item.value;+ }+ }+ const last = items.at(-1);+ if (last === undefined) {+ throw new Error('Rng.pick called with an empty list');+ }+ return last.value;+ }++ /** Standard normal via Box–Muller. */+ normal(): number {+ if (this.spareNormal !== null) {+ const spare = this.spareNormal;+ this.spareNormal = null;+ return spare;+ }+ let u = 0;+ while (u === 0) {+ u = this.float();+ }+ const v = this.float();+ const radius = Math.sqrt(-2 * Math.log(u));+ this.spareNormal = radius * Math.sin(2 * Math.PI * v);+ return radius * Math.cos(2 * Math.PI * v);+ }++ /** Right-skewed seconds around a median: most dwell times are short, a few are very long. */+ logNormalSeconds(median: number, sigma: number): number {+ return median * Math.exp(this.normal() * sigma);+ }++ poisson(lambda: number): number {+ if (lambda <= 0) {+ return 0;+ }+ if (lambda > POISSON_NORMAL_APPROXIMATION_THRESHOLD) {+ return Math.max(+ 0,+ Math.round(lambda + Math.sqrt(lambda) * this.normal())+ );+ }+ const limit = Math.exp(-lambda);+ let count = 0;+ let product = 1;+ for (;;) {+ product *= this.float();+ if (product <= limit) {+ return count;+ }+ count++;+ }+ }++ bytes(length: number): Buffer {+ const out = Buffer.alloc(length);+ for (let offset = 0; offset < length; offset += 4) {+ const word = this.next();+ for (let b = 0; b < 4 && offset + b < length; b++) {+ out[offset + b] = (word >>> (b * 8)) & 0xff;+ }+ }+ return out;+ }++ hex(chars: number): string {+ return this.bytes(Math.ceil(chars / 2))+ .toString('hex')+ .slice(0, chars);+ }++ uuid(): string {+ return formatUuid(this.bytes(16).toString('hex'), '4');+ }++ /** The shape the ingest mints for session ids: 16 bytes, base64url, 22 chars. */+ sessionId(): string {+ return this.bytes(16).toString('base64url');+ }+}diff --git a/packages/seed/src/rows.ts b/packages/seed/src/rows.tsnew file mode 100644index 000000000..ee5cdbdcb--- /dev/null+++ b/packages/seed/src/rows.ts@@ -0,0 +1,363 @@+// From a journey to ClickHouse rows, shaped the way the worker shapes them:+// `createEvent` for events, the session buffer's `newSession` / `extendSession`+// for the one sessions row, and the ingest handler's synthesized+// `session_start` / `session_end`. Kept in step with core by hand, on purpose:+// the seed stays out of the backend's dependency graph.++import { parsePath, toDots } from '@openpanel/shared';+import type { Archetype, JourneyEvent } from './archetypes/archetype';+import {+ type ClickhouseEventRow,+ type ClickhouseProfileRow,+ type ClickhouseSessionRow,+ toClickhouseDateTime,+} from './clickhouse-rows';+import { sessionReferrer } from './data/referrers';+import type { Referrer, Visitor } from './model';+import type { Rng } from './rng';+import {+ SESSION_END_OFFSET_MS,+ SESSION_START_OFFSET_MS,+} from './seed.constants';++export interface SessionInput {+ id: string;+ projectId: string;+ archetype: Archetype;+ visitor: Visitor;+ referrer: Referrer | null;+ utm: Record<string, string>;+ events: readonly JourneyEvent[];+}++export interface SessionRows {+ events: ClickhouseEventRow[];+ session: ClickhouseSessionRow;+ endedAt: Date;+ /** Whether the session ended identified, and whether any event went out anonymously. */+ identified: boolean;+ anonymousEvents: boolean;+}++/** What the ingest handler derives from one SDK request before the rows are shaped. */+interface EventPayload {+ name: string;+ profileId: string;+ createdAt: Date;+ path: string;+ origin: string;+ properties: Record<string, unknown>;+ duration: number;+ revenue?: number;+}++const UTM_KEYS = [+ 'utm_medium',+ 'utm_source',+ 'utm_campaign',+ 'utm_content',+ 'utm_term',+] as const;++/** The SDK sends `__path` as a full URL on the web and a bare screen name in a native app. */+function eventUrl(+ archetype: Archetype,+ event: JourneyEvent,+ utm: Record<string, string>,+ isLanding: boolean+): string {+ if (archetype.deviceClass === 'native') {+ return event.path;+ }+ const query =+ isLanding && Object.keys(utm).length > 0+ ? `?${new URLSearchParams(utm).toString()}`+ : '';+ return `${archetype.origin}${event.path}${query}`;+}++function utmOf(+ properties: Record<string, unknown>,+ key: (typeof UTM_KEYS)[number]+): string {+ const query = properties.__query as Record<string, unknown> | undefined;+ const value = query?.[key];+ return value ? String(value) : '';+}++/** `createEvent`: every column, with the anonymous fallback `profile_id = device_id`. */+function eventRow(+ input: SessionInput,+ referrer: Referrer,+ payload: EventPayload,+ id: string+): ClickhouseEventRow {+ const { visitor, archetype } = input;+ return {+ id,+ name: payload.name,+ device_id: visitor.window.id,+ profile_id: payload.profileId || visitor.window.id,+ project_id: input.projectId,+ session_id: input.id,+ properties: toDots(payload.properties),+ path: payload.path,+ origin: payload.origin,+ created_at: toClickhouseDateTime(payload.createdAt),+ country: visitor.geo.country,+ city: visitor.geo.city,+ region: visitor.geo.region,+ longitude: visitor.geo.longitude,+ latitude: visitor.geo.latitude,+ os: visitor.device.os ?? '',+ os_version: visitor.device.osVersion ?? '',+ browser: visitor.device.browser ?? '',+ browser_version: visitor.device.browserVersion ?? '',+ device: visitor.device.device ?? '',+ brand: visitor.device.brand ?? '',+ model: visitor.device.model ?? '',+ duration: payload.duration,+ referrer: referrer.url,+ referrer_name: referrer.name,+ referrer_type: referrer.type,+ imported_at: null,+ inserted_at: toClickhouseDateTime(new Date()),+ sdk_name: archetype.sdk.name,+ sdk_version: archetype.sdk.version,+ revenue: payload.revenue,+ groups: [],+ };+}++/** The session buffer's `newSession`. */+function openSession(+ input: SessionInput,+ referrer: Referrer,+ first: EventPayload+): ClickhouseSessionRow {+ const { visitor } = input;+ const createdAt = toClickhouseDateTime(first.createdAt);+ const isScreenView = first.name === 'screen_view';+ return {+ id: input.id,+ project_id: input.projectId,+ device_id: visitor.window.id,+ profile_id: first.profileId || visitor.window.id,+ is_bounce: true,+ created_at: createdAt,+ ended_at: createdAt,+ event_count: isScreenView ? 0 : 1,+ screen_view_count: isScreenView ? 1 : 0,+ entry_path: first.path,+ entry_origin: first.origin,+ exit_path: first.path,+ exit_origin: first.origin,+ revenue: first.name === 'revenue' ? (first.revenue ?? 0) : 0,+ referrer: referrer.url,+ referrer_name: referrer.name,+ referrer_type: referrer.type,+ os: visitor.device.os ?? '',+ os_version: visitor.device.osVersion ?? '',+ browser: visitor.device.browser ?? '',+ browser_version: visitor.device.browserVersion ?? '',+ device: visitor.device.device ?? '',+ brand: visitor.device.brand ?? '',+ model: visitor.device.model ?? '',+ country: visitor.geo.country,+ region: visitor.geo.region,+ city: visitor.geo.city,+ longitude: visitor.geo.longitude,+ latitude: visitor.geo.latitude,+ duration: first.duration,+ utm_medium: utmOf(first.properties, 'utm_medium'),+ utm_source: utmOf(first.properties, 'utm_source'),+ utm_campaign: utmOf(first.properties, 'utm_campaign'),+ utm_content: utmOf(first.properties, 'utm_content'),+ utm_term: utmOf(first.properties, 'utm_term'),+ groups: [],+ sign: 1,+ version: 1,+ };+}++/** The session buffer's `extendSession`, for events that arrive in order. */+function extendSession(+ session: ClickhouseSessionRow,+ payload: EventPayload,+ deviceId: string+): void {+ session.version += 1;+ session.ended_at = toClickhouseDateTime(payload.createdAt);+ if (payload.path) {+ session.exit_path = payload.path;+ }+ if (payload.origin) {+ session.exit_origin = payload.origin;+ }+ if (!session.entry_path && payload.path) {+ session.entry_path = payload.path;+ }+ if (!session.entry_origin && payload.origin) {+ session.entry_origin = payload.origin;+ }+ session.duration =+ payload.createdAt.getTime() - new Date(`${session.created_at}Z`).getTime();+ if (payload.name === 'revenue') {+ session.revenue += payload.revenue ?? 0;+ }+ if (payload.name === 'screen_view' && payload.path) {+ session.screen_view_count += 1;+ } else {+ session.event_count += 1;+ }+ if (session.screen_view_count > 1) {+ session.is_bounce = false;+ }+ if (payload.profileId && payload.profileId !== deviceId) {+ session.profile_id = payload.profileId;+ }+}++export function buildSessionRows(rng: Rng, input: SessionInput): SessionRows {+ const { archetype, visitor, utm } = input;+ const payloads: EventPayload[] = [];+ let referrer: Referrer = { url: '', name: '', type: '' };++ for (const [index, event] of input.events.entries()) {+ const isLanding = index === 0;+ const { path, origin, query, hash } = parsePath(+ eventUrl(archetype, event, utm, isLanding)+ );+ if (isLanding) {+ referrer = sessionReferrer(input.referrer, query);+ }+ payloads.push({+ name: event.name,+ profileId: event.person?.id ?? '',+ createdAt: event.at,+ path,+ origin,+ // `__path` / `__referrer` never reach the row; `__hash` and `__query` are added by the handler.+ properties: {+ ...event.properties,+ __title: event.title || undefined,+ __hash: hash,+ __query: query,+ },+ duration: 0,+ revenue: event.name === 'revenue' ? event.revenue : undefined,+ });+ }++ const first = payloads[0];+ const last = payloads.at(-1);+ if (!(first && last)) {+ throw new Error('A session needs at least one event');+ }++ const session = openSession(input, referrer, first);+ for (const payload of payloads.slice(1)) {+ extendSession(session, payload, visitor.window.id);+ }++ const endedAt = new Date(last.createdAt.getTime());+ const sessionStart: EventPayload = {+ ...first,+ name: 'session_start',+ createdAt: new Date(first.createdAt.getTime() - SESSION_START_OFFSET_MS),+ };+ const sessionEnd: EventPayload = {+ ...last,+ name: 'session_end',+ properties: { ...last.properties, __bounce: session.is_bounce },+ duration: session.duration,+ path: session.exit_path,+ createdAt: new Date(endedAt.getTime() + SESSION_END_OFFSET_MS),+ profileId: session.profile_id,+ };++ visitor.lastPath = first.path;+ visitor.lastReferrer = referrer;++ return {+ events: [sessionStart, ...payloads, sessionEnd].map((payload) =>+ eventRow(input, referrer, payload, rng.uuid())+ ),+ session,+ endedAt,+ identified: last.profileId !== '',+ anonymousEvents: payloads.some((payload) => payload.profileId === ''),+ };+}++/** The properties `createEvent` snapshots onto a profile at a session boundary. */+function profileSnapshot(visitor: Visitor): Record<string, string> {+ const snapshot: Record<string, string | number | null | undefined> = {+ path: visitor.lastPath,+ country: visitor.geo.country,+ city: visitor.geo.city,+ region: visitor.geo.region,+ longitude: visitor.geo.longitude,+ latitude: visitor.geo.latitude,+ os: visitor.device.os,+ os_version: visitor.device.osVersion,+ browser: visitor.device.browser,+ browser_version: visitor.device.browserVersion,+ device: visitor.device.device,+ brand: visitor.device.brand,+ model: visitor.device.model,+ referrer: visitor.lastReferrer?.url,+ referrer_name: visitor.lastReferrer?.name,+ referrer_type: visitor.lastReferrer?.type,+ };+ const properties: Record<string, string> = {};+ for (const [key, value] of Object.entries(snapshot)) {+ if (value !== undefined && value !== null && value !== '') {+ properties[key] = String(value);+ }+ }+ return properties;+}++/** The anonymous profile a device window leaves behind (`profile_id = device_id`). */+export function deviceProfileRow(+ visitor: Visitor,+ projectId: string+): ClickhouseProfileRow {+ return {+ id: visitor.window.id,+ first_name: '',+ last_name: '',+ email: '',+ avatar: '',+ is_external: false,+ properties: profileSnapshot(visitor),+ project_id: projectId,+ created_at: toClickhouseDateTime(visitor.window.firstSeen),+ last_seen_at: toClickhouseDateTime(visitor.window.lastSeen),+ groups: [],+ };+}++/** The identified person's profile: name, email, avatar and the archetype's person properties. */+export function personProfileRow(+ visitor: Visitor,+ projectId: string+): ClickhouseProfileRow {+ if (!visitor.person) {+ throw new Error('personProfileRow needs an identified visitor');+ }+ return {+ id: visitor.person.id,+ first_name: visitor.person.firstName,+ last_name: visitor.person.lastName,+ email: visitor.person.email,+ avatar: visitor.person.avatar,+ is_external: true,+ properties: { ...profileSnapshot(visitor), ...visitor.person.properties },+ project_id: projectId,+ created_at: toClickhouseDateTime(visitor.firstSeen),+ last_seen_at: toClickhouseDateTime(visitor.lastSeen),+ groups: [],+ };+}diff --git a/packages/seed/src/seed.constants.ts b/packages/seed/src/seed.constants.tsnew file mode 100644index 000000000..51b211d8b--- /dev/null+++ b/packages/seed/src/seed.constants.ts@@ -0,0 +1,44 @@+export const DEFAULT_SEED = 42;++/** The login every seeded database has; also written to `.seed.json`. */+export const SEED_USER = {+ id: 'user_seed_admin',+ email: '[email protected]',+ password: 'openpanel',+ firstName: 'Seed',+ lastName: 'Admin',+} as const;++export const SEED_ORGANIZATION = { id: 'acme', name: 'Acme' } as const;++export const ORGANIZATION_ADMIN_ROLE = 'org:admin';++/** Mirrors the trial the onboarding flow grants, so billing gates behave the same. */+export const TRIAL_DAYS = 365;+export const TRIAL_EVENTS_LIMIT = 100_000_000;++export const SIZE_PRESETS = {+ small: { days: 14, sessionsPerDay: 300 },+ medium: { days: 30, sessionsPerDay: 2000 },+ large: { days: 90, sessionsPerDay: 12_000 },+ xl: { days: 365, sessionsPerDay: 30_000 },+} as const;++export type SizePreset = keyof typeof SIZE_PRESETS;++export const DEFAULT_SIZE: SizePreset = 'small';++/** Rows per ClickHouse insert; JSONEachRow at this size stays well under a second. */+export const INSERT_BATCH_ROWS = 50_000;++/** uuid v5 namespace for every deterministic id the seed mints. */+export const ID_NAMESPACE = '6f5f2b0e-3f7c-4a3e-9c1e-2d0b4a6e8c11';++/** Where `session_start` / `session_end` sit relative to the real events (mirrors the worker). */+export const SESSION_START_OFFSET_MS = 100;+export const SESSION_END_OFFSET_MS = 1000;++/** Returning-visitor pool per project; evicted visitors never come back. */+export const VISITOR_POOL_SIZE = 20_000;++export const MANIFEST_FILE = '.seed.json';diff --git a/packages/seed/src/seed.integration.test.ts b/packages/seed/src/seed.integration.test.tsnew file mode 100644index 000000000..8b648a749--- /dev/null+++ b/packages/seed/src/seed.integration.test.ts@@ -0,0 +1,157 @@+// Seeds two days into the suite's isolated databases and checks what the+// dashboard's queries rely on. Pins the env before any database import, the+// same way test/bun-preload.ts does for @openpanel/db.++import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import {+ TEST_CLICKHOUSE_URL,+ TEST_DATABASE_URL,+ TEST_REDIS_URL,+} from '../../../test/databases';++process.env.DATABASE_URL = TEST_DATABASE_URL;+process.env.CLICKHOUSE_URL = TEST_CLICKHOUSE_URL;+process.env.REDIS_URL = TEST_REDIS_URL;+process.env.SELF_HOSTED = 'true';++const { bootstrapTestDatabases } = await import(+ '../../../test/bootstrap-databases'+);+const { ch, db } = await import('@openpanel/db');+const { ARCHETYPES } = await import('./archetypes');+const { generate } = await import('./generator');+const { projectIdFor } = await import('./ids');+const { seedPostgres } = await import('./postgres');+const { resetProjects } = await import('./reset');+const { Rng } = await import('./rng');+const { SEED_ORGANIZATION, SEED_USER } = await import('./seed.constants');+const { ClickhouseSink } = await import('./sink');++const ignoreProgress = () => undefined;+const SEED = 3;+const DAYS = 2;+const SESSIONS_PER_DAY = 120;+const PROJECT_IDS = ARCHETYPES.map(projectIdFor);++async function count(query: string): Promise<number> {+ const result = await ch.query({ query, format: 'JSONEachRow' });+ const rows = (await result.json()) as { value: string | number }[];+ return Number(rows[0]?.value ?? 0);+}++function inList(): string {+ return PROJECT_IDS.map((id) => `'${id}'`).join(', ');+}++describe('seed against real databases', () => {+ beforeAll(async () => {+ await bootstrapTestDatabases();+ await seedPostgres(new Rng(`${SEED}/postgres`), ARCHETYPES);+ await resetProjects(PROJECT_IDS);+ const sink = new ClickhouseSink();+ await generate(+ {+ seed: SEED,+ sessionsPerDay: SESSIONS_PER_DAY,+ days: DAYS,+ now: new Date(),+ variance: 1,+ projects: ARCHETYPES.map((archetype) => ({+ archetype,+ projectId: projectIdFor(archetype),+ })),+ },+ sink,+ ignoreProgress+ );+ });++ afterAll(async () => {+ await resetProjects(PROJECT_IDS);+ await db.organization+ .delete({ where: { id: SEED_ORGANIZATION.id } })+ .catch(() => null);+ await db.user.delete({ where: { id: SEED_USER.id } }).catch(() => null);+ await db.$disconnect();+ });++ test('postgres has the login, organization, projects and clients', async () => {+ const user = await db.user.findUnique({+ where: { email: SEED_USER.email },+ include: { accounts: true, membership: true },+ });+ expect(+ user?.accounts.some(+ (account) =>+ account.provider === 'email' &&+ account.password?.startsWith('$argon2')+ )+ ).toBe(true);+ expect(+ user?.membership.some(+ (member) => member.organizationId === SEED_ORGANIZATION.id+ )+ ).toBe(true);+ const projects = await db.project.findMany({+ where: { organizationId: SEED_ORGANIZATION.id },+ include: { clients: true },+ });+ expect(projects.map((project) => project.id).sort()).toEqual(+ [...PROJECT_IDS].sort()+ );+ for (const project of projects) {+ expect(project.clients).toHaveLength(1);+ // scrypt `salt.hex`, what ingest's verifyPassword expects — never argon2.+ expect(project.clients[0]?.secret).toMatch(/^[0-9a-f]+\.[0-9a-f]+$/);+ }+ });++ test('sessions collapse to one row per session and match the events', async () => {+ const sessions = await count(+ `SELECT count() AS value FROM sessions FINAL WHERE project_id IN (${inList()})`+ );+ const distinct = await count(+ `SELECT uniqExact(session_id) AS value FROM events WHERE project_id IN (${inList()})`+ );+ const starts = await count(+ `SELECT count() AS value FROM events WHERE name = 'session_start' AND project_id IN (${inList()})`+ );+ expect(sessions).toBeGreaterThan(DAYS * SESSIONS_PER_DAY * 0.3);+ expect(distinct).toBe(sessions);+ expect(starts).toBe(sessions);+ });++ test('bounce rate and durations are in a realistic range', async () => {+ const bounced = await count(+ `SELECT countIf(is_bounce) AS value FROM sessions FINAL WHERE project_id IN (${inList()})`+ );+ const total = await count(+ `SELECT count() AS value FROM sessions FINAL WHERE project_id IN (${inList()})`+ );+ const rate = bounced / total;+ expect(rate).toBeGreaterThan(0.1);+ expect(rate).toBeLessThan(0.8);+ const averageSeconds = await count(+ `SELECT avg(duration) / 1000 AS value FROM sessions FINAL WHERE project_id IN (${inList()}) AND NOT is_bounce`+ );+ expect(averageSeconds).toBeGreaterThan(10);+ });++ test('the materialized views the overview reads are populated', async () => {+ expect(+ await count(+ `SELECT uniqMerge(profile_id) AS value FROM dau_mv WHERE project_id IN (${inList()})`+ )+ ).toBeGreaterThan(0);+ expect(+ await count(+ `SELECT count() AS value FROM distinct_event_names_mv WHERE project_id IN (${inList()})`+ )+ ).toBeGreaterThan(0);+ expect(+ await count(+ `SELECT count() AS value FROM profiles FINAL WHERE project_id IN (${inList()}) AND is_external`+ )+ ).toBeGreaterThan(0);+ });+});diff --git a/packages/seed/src/send.ts b/packages/seed/src/send.tsnew file mode 100644index 000000000..2cde726fa--- /dev/null+++ b/packages/seed/src/send.ts@@ -0,0 +1,347 @@+// `bun run send` — send real requests to a worktree's API with the seeded clients:+// one event, an identify, or a whole generated journey replayed through `/track`+// (ingest → Kafka → worker → ClickHouse), unlike `bun run seed`, which inserts+// directly. For testing ingest, live views and anything that reacts to new events.+//+// bun run send track <event> [--project acme-shop] [--path /pricing] [--profile usr_x] [--prop k=v]... [--revenue 99]+// bun run send identify <profileId> [--project] [--email] [--first-name] [--last-name] [--prop k=v]...+// bun run send journey [--project acme-shop] [--sessions 1] [--realtime] [--seed 7]+// options: --url <api> (default $API_URL), --client <id> --secret <secret> (default: the project's seeded client)++import { existsSync, readFileSync } from 'node:fs';+import { resolve } from 'node:path';+import { setTimeout as sleep } from 'node:timers/promises';+import { parseArgs } from 'node:util';+import { ARCHETYPES, archetypeById } from './archetypes';+import {+ type Archetype,+ Journey,+ type JourneyEvent,+} from './archetypes/archetype';+import { CAMPAIGNS } from './data/referrers';+import { projectIdFor } from './ids';+import type { Person, Visitor } from './model';+import { Rng } from './rng';+import { MANIFEST_FILE } from './seed.constants';+import { World } from './world';++const USAGE = `Usage: bun run send <track|identify|journey> [options]++ track <event> send one event --path /x --profile usr_x --prop k=v --revenue 99+ identify <profileId> send an identify --email a@b --first-name A --last-name B --prop k=v+ journey generate a session with the project's archetype and replay it live+ --sessions N (default 1) --realtime (real dwell times; default caps gaps at 1s) --seed N++ --project <id> which seeded project (default: acme-web)+ --url <api> default $API_URL+ --client <id> --secret <secret> override the project's seeded write client+`;++/** Gaps between replayed events are capped so a journey takes seconds, not minutes. */+const MAX_GAP_MS = 1000;+const MANIFEST_PATH = resolve(import.meta.dir, '..', '..', '..', MANIFEST_FILE);+const DEFAULT_PROJECT = 'acme-web';++interface Manifest {+ projects: {+ id: string;+ archetype: string;+ clientId: string;+ clientSecret: string;+ }[];+}++interface Target {+ url: string;+ clientId: string;+ clientSecret: string;+ projectId: string;+ archetype: Archetype;+}++function readManifest(): Manifest | null {+ if (!existsSync(MANIFEST_PATH)) {+ return null;+ }+ return JSON.parse(readFileSync(MANIFEST_PATH, 'utf8')) as Manifest;+}++function parseProps(values: string[] | undefined): Record<string, unknown> {+ const properties: Record<string, unknown> = {};+ for (const entry of values ?? []) {+ const index = entry.indexOf('=');+ if (index <= 0) {+ throw new Error(`--prop expects key=value, got "${entry}"`);+ }+ const raw = entry.slice(index + 1);+ const asNumber = Number(raw);+ properties[entry.slice(0, index)] =+ raw === 'true'+ ? true+ : raw === 'false'+ ? false+ : raw !== '' && !Number.isNaN(asNumber)+ ? asNumber+ : raw;+ }+ return properties;+}++function resolveTarget(values: {+ url?: string;+ project?: string;+ client?: string;+ secret?: string;+}): Target {+ const url = (values.url ?? process.env.API_URL ?? '').replace(/\/$/, '');+ if (!url) {+ throw new Error('No API: pass --url or set API_URL.');+ }+ const projectId = values.project ?? DEFAULT_PROJECT;+ const archetype =+ ARCHETYPES.find((candidate) => projectIdFor(candidate) === projectId) ??+ archetypeById(projectId);+ const manifest = readManifest();+ const seeded = manifest?.projects.find((project) => project.id === projectId);+ const clientId = values.client ?? seeded?.clientId;+ const clientSecret = values.secret ?? seeded?.clientSecret;+ if (!(clientId && clientSecret)) {+ throw new Error(+ `No client for ${projectId}: run \`bun run seed\` first or pass --client and --secret.`+ );+ }+ return { url, clientId, clientSecret, projectId, archetype };+}++interface Sender {+ userAgent: string;+ ip: string;+}++async function post(+ target: Target,+ sender: Sender,+ body: unknown+): Promise<string> {+ const response = await fetch(`${target.url}/track`, {+ method: 'POST',+ headers: {+ 'content-type': 'application/json',+ 'openpanel-client-id': target.clientId,+ 'openpanel-client-secret': target.clientSecret,+ 'openpanel-sdk-name': target.archetype.sdk.name,+ 'openpanel-sdk-version': target.archetype.sdk.version,+ 'user-agent': sender.userAgent,+ 'x-client-ip': sender.ip,+ },+ body: JSON.stringify(body),+ });+ const text = await response.text();+ return `${response.status} ${text}`;+}++function trackBody(+ name: string,+ properties: Record<string, unknown>,+ profileId?: string+) {+ return {+ type: 'track',+ payload: { name, properties, ...(profileId ? { profileId } : {}) },+ };+}++function identifyBody(person: Person) {+ // Empty strings fail validation (`avatar` must be a URL); send only what is set.+ const optional = Object.fromEntries(+ Object.entries({+ firstName: person.firstName,+ lastName: person.lastName,+ email: person.email,+ avatar: person.avatar,+ }).filter(([, value]) => value !== '')+ );+ return {+ type: 'identify',+ payload: {+ profileId: person.id,+ ...optional,+ properties: person.properties,+ },+ };+}++/** The SDK's view of a journey event: `__path` as the full URL (or screen name), `__title`, custom properties. */+function sdkProperties(+ target: Target,+ event: JourneyEvent,+ landingQuery: string+): Record<string, unknown> {+ const path =+ target.archetype.deviceClass === 'native'+ ? event.path+ : `${target.archetype.origin}${event.path}${landingQuery}`;+ return {+ ...event.properties,+ __path: path,+ __title: event.title || undefined,+ ...(event.revenue !== undefined ? { __revenue: event.revenue } : {}),+ };+}++async function replay(+ target: Target,+ visitor: Visitor,+ events: readonly JourneyEvent[],+ utm: Record<string, string>,+ realtime: boolean+): Promise<void> {+ const sender = { userAgent: visitor.userAgent, ip: visitor.geo.ip };+ let identified: Person | null = visitor.identified ? visitor.person : null;+ let previous: Date | null = null;+ for (const [index, event] of events.entries()) {+ if (previous) {+ const gap = event.at.getTime() - previous.getTime();+ await sleep(realtime ? gap : Math.min(gap, MAX_GAP_MS));+ }+ previous = event.at;+ if (event.person && event.person !== identified) {+ identified = event.person;+ console.log(+ `identify ${event.person.id} → ${await post(target, sender, identifyBody(event.person))}`+ );+ }+ const landingQuery =+ index === 0 && Object.keys(utm).length+ ? `?${new URLSearchParams(utm)}`+ : '';+ const body = trackBody(+ event.name,+ sdkProperties(target, event, landingQuery),+ identified?.id+ );+ console.log(+ `${event.name.padEnd(28)} ${event.path.padEnd(28)} → ${await post(target, sender, body)}`+ );+ }+}++async function main(): Promise<void> {+ const { values, positionals } = parseArgs({+ args: process.argv.slice(2),+ allowPositionals: true,+ options: {+ project: { type: 'string' },+ url: { type: 'string' },+ client: { type: 'string' },+ secret: { type: 'string' },+ path: { type: 'string' },+ profile: { type: 'string' },+ prop: { type: 'string', multiple: true },+ revenue: { type: 'string' },+ email: { type: 'string' },+ 'first-name': { type: 'string' },+ 'last-name': { type: 'string' },+ sessions: { type: 'string' },+ realtime: { type: 'boolean', default: false },+ seed: { type: 'string' },+ help: { type: 'boolean', default: false },+ },+ });+ const [command, argument] = positionals;+ if (values.help || !command) {+ process.stdout.write(USAGE);+ return;+ }+ const target = resolveTarget(values);+ const rng = new Rng(`send/${values.seed ?? Date.now()}`);+ const world = new World(+ target.archetype,+ target.projectId,+ new Date(),+ () => undefined+ );+ const visitor = world.pick(rng, new Date());+ const sender = { userAgent: visitor.userAgent, ip: visitor.geo.ip };+ console.log(+ `→ ${target.url} project ${target.projectId} client ${target.clientId}`+ );++ if (command === 'track') {+ if (!argument) {+ throw new Error('track needs an event name');+ }+ const path = values.path ?? '/';+ const properties = {+ ...parseProps(values.prop),+ __path:+ target.archetype.deviceClass === 'native'+ ? path+ : `${target.archetype.origin}${path}`,+ ...(values.revenue ? { __revenue: Number(values.revenue) } : {}),+ };+ console.log(+ await post(+ target,+ sender,+ trackBody(argument, properties, values.profile)+ )+ );+ return;+ }+ if (command === 'identify') {+ if (!argument) {+ throw new Error('identify needs a profile id');+ }+ const person: Person = {+ id: argument,+ firstName: values['first-name'] ?? '',+ lastName: values['last-name'] ?? '',+ email: values.email ?? '',+ avatar: '',+ properties: Object.fromEntries(+ Object.entries(parseProps(values.prop)).map(([key, value]) => [+ key,+ String(value),+ ])+ ),+ };+ console.log(await post(target, sender, identifyBody(person)));+ return;+ }+ if (command === 'journey') {+ const sessions = Number(values.sessions ?? 1);+ for (let i = 0; i < sessions; i++) {+ const current = i === 0 ? visitor : world.pick(rng, new Date());+ const campaign = rng.chance(target.archetype.campaignChance)+ ? rng.pick(CAMPAIGNS)+ : null;+ const utm: Record<string, string> = campaign+ ? {+ utm_source: campaign.source,+ utm_medium: campaign.medium,+ utm_campaign: campaign.name,+ }+ : {};+ const journey = new Journey(+ rng,+ new Date(),+ target.archetype.dwellMedianSeconds,+ current.identified ? current.person : null+ );+ target.archetype.journey(rng, journey, current);+ console.log(+ `\nsession ${i + 1}: ${journey.events.length} events, ${current.person ? `person ${current.person.id}` : 'anonymous'}`+ );+ await replay(target, current, journey.events, utm, values.realtime);+ world.touch(current, new Date(), {+ identified: journey.identified !== null,+ anonymousEvents: journey.events.some((event) => event.person === null),+ });+ }+ return;+ }+ throw new Error(`Unknown command "${command}".\n${USAGE}`);+}++await main();diff --git a/packages/seed/src/sink.ts b/packages/seed/src/sink.tsnew file mode 100644index 000000000..135beae19--- /dev/null+++ b/packages/seed/src/sink.ts@@ -0,0 +1,109 @@+// Batches rows per table and inserts them as JSONEachRow through the shared+// ClickHouse client, so the seed lands in whatever database CLICKHOUSE_URL names.+// Adds are synchronous (the visitor pool evicts from inside a sync path); the+// generator awaits `flushFull()` between sessions.++import { ch, TABLE_NAMES } from '@openpanel/db';+import type {+ ClickhouseEventRow,+ ClickhouseProfileRow,+ ClickhouseSessionRow,+} from './clickhouse-rows';+import { INSERT_BATCH_ROWS } from './seed.constants';++export interface SinkCounts {+ events: number;+ sessions: number;+ profiles: number;+}++class TableBuffer<Row> {+ private rows: Row[] = [];+ count = 0;++ constructor(private readonly table: string) {}++ add(rows: readonly Row[]): void {+ for (const row of rows) {+ this.rows.push(row);+ }+ this.count += rows.length;+ }++ get full(): boolean {+ return this.rows.length >= INSERT_BATCH_ROWS;+ }++ async flush(): Promise<void> {+ if (this.rows.length === 0) {+ return;+ }+ const batch = this.rows;+ this.rows = [];+ await ch.insert({+ table: this.table,+ values: batch,+ format: 'JSONEachRow',+ });+ }+}++/** What the generator needs from a sink; tests substitute an in-memory one. */+export interface RowSink {+ addEvents(rows: readonly ClickhouseEventRow[]): void;+ addSession(row: ClickhouseSessionRow): void;+ addProfiles(rows: readonly ClickhouseProfileRow[]): void;+ flushFull(): Promise<void>;+ flush(): Promise<void>;+ counts(): SinkCounts;+}++export class ClickhouseSink implements RowSink {+ private readonly events = new TableBuffer<ClickhouseEventRow>(+ TABLE_NAMES.events+ );+ private readonly sessions = new TableBuffer<ClickhouseSessionRow>(+ TABLE_NAMES.sessions+ );+ private readonly profiles = new TableBuffer<ClickhouseProfileRow>(+ TABLE_NAMES.profiles+ );++ private get buffers(): TableBuffer<unknown>[] {+ return [this.events, this.sessions, this.profiles];+ }++ addEvents(rows: readonly ClickhouseEventRow[]): void {+ this.events.add(rows);+ }++ addSession(row: ClickhouseSessionRow): void {+ this.sessions.add([row]);+ }++ addProfiles(rows: readonly ClickhouseProfileRow[]): void {+ this.profiles.add(rows);+ }++ async flushFull(): Promise<void> {+ for (const buffer of this.buffers) {+ if (buffer.full) {+ await buffer.flush();+ }+ }+ }++ async flush(): Promise<void> {+ for (const buffer of this.buffers) {+ await buffer.flush();+ }+ }++ counts(): SinkCounts {+ return {+ events: this.events.count,+ sessions: this.sessions.count,+ profiles: this.profiles.count,+ };+ }+}diff --git a/packages/seed/src/traffic.ts b/packages/seed/src/traffic.tsnew file mode 100644index 000000000..9d2ed1ebe--- /dev/null+++ b/packages/seed/src/traffic.ts@@ -0,0 +1,168 @@+// Turns an archetype's shape (hourly curve, weekday curve) into visitor counts+// per day and per hour. Ported from openpanel-mock's opmock/internal/traffic.+//+// The daily multiplier is a product of:+// trend slow growth/decline, anchored so "today" == 1.0+// walk a mean-reverting random walk across days (weeks drift)+// weekday the archetype's day-of-week curve+// noise small independent per-day jitter+// anomaly rare spike or dip days+// Everything is a pure function of (seed, key, day), so seeding twice gives+// the same curve.++import { Rng } from './rng';++export interface TrafficShape {+ /** Relative weight of each local hour, 0..23; scale is irrelevant. */+ hourly: readonly number[];+ /** Multiplies the daily total; index 0 = Sunday. */+ dayOfWeek: readonly number[];+ /** Fractional growth per week, e.g. 0.02 = +2 %/week, relative to today. */+ trendPerWeek: number;+ spikeChance: number;+ dipChance: number;+ spikeRange: readonly [number, number];+ dipRange: readonly [number, number];+}++export interface TrafficParams {+ seed: string;+ /** 0 = perfectly regular, 1 = lively, > 1 = chaotic. */+ variance: number;+ now: Date;+}++export interface Anomaly {+ kind: 'none' | 'spike' | 'dip';+ multiplier: number;+}++export interface TrafficDay {+ date: Date;+ visitors: number;+ weekday: number;+ trend: number;+ walk: number;+ noise: number;+ anomaly: Anomaly;+ multiplier: number;+}++const MS_PER_DAY = 86_400_000;+const HOURS_PER_DAY = 24;+/** An arbitrary fixed anchor so any day's walk value is computable on its own. */+const WALK_EPOCH_DAY = Date.UTC(2020, 0, 1);+const WALK_MEAN_REVERSION = 0.9;+const WALK_SIGMA = 0.1;+const WALK_MIN = 0.35;+const WALK_MAX = 3;+const NOISE_SIGMA = 0.06;+const NOISE_MIN = 0.5;+const HOUR_JITTER_SIGMA = 0.12;+const HOUR_JITTER_MIN = 0.2;+const MAX_ANOMALY_VARIANCE = 2;++function dayIndex(date: Date): number {+ const midnight = Date.UTC(+ date.getUTCFullYear(),+ date.getUTCMonth(),+ date.getUTCDate()+ );+ return Math.floor((midnight - WALK_EPOCH_DAY) / MS_PER_DAY);+}++function isoDay(date: Date): string {+ return date.toISOString().slice(0, 10);+}++export class TrafficModel {+ private walk: number[] = [];++ constructor(+ private readonly key: string,+ private readonly shape: TrafficShape,+ private readonly params: TrafficParams+ ) {}++ private rng(salt: string, day: Date): Rng {+ return new Rng(`${this.params.seed}/${this.key}/${salt}/${isoDay(day)}`);+ }++ /** AR(1): w[i] = 0.9·w[i-1] + N(0, σ), replayed from the epoch and cached. */+ private walkAt(index: number): number {+ if (index < 0) {+ return 0;+ }+ if (index < this.walk.length) {+ return this.walk[index] ?? 0;+ }+ const rng = new Rng(`${this.params.seed}/${this.key}/walk`);+ const sigma = WALK_SIGMA * this.params.variance;+ this.walk = [];+ let value = 0;+ for (let i = 0; i <= index; i++) {+ value = WALK_MEAN_REVERSION * value + rng.normal() * sigma;+ this.walk.push(value);+ }+ return this.walk[index] ?? 0;+ }++ private anomaly(day: Date): Anomaly {+ const rng = this.rng('anomaly', day);+ const roll = rng.float();+ const variance = Math.min(this.params.variance, MAX_ANOMALY_VARIANCE);+ if (roll < this.shape.spikeChance * variance) {+ const [low, high] = this.shape.spikeRange;+ return { kind: 'spike', multiplier: low + rng.float() * (high - low) };+ }+ if (roll < (this.shape.spikeChance + this.shape.dipChance) * variance) {+ const [low, high] = this.shape.dipRange;+ return { kind: 'dip', multiplier: low + rng.float() * (high - low) };+ }+ return { kind: 'none', multiplier: 1 };+ }++ dayFor(day: Date, baseVisitors: number): TrafficDay {+ const index = dayIndex(day);+ const weeksFromNow = (index - dayIndex(this.params.now)) / 7;+ const trend = (1 + this.shape.trendPerWeek) ** weeksFromNow;+ const walk = Math.min(+ WALK_MAX,+ Math.max(WALK_MIN, Math.exp(this.walkAt(index)))+ );+ const noise = Math.max(+ NOISE_MIN,+ 1 + this.rng('noise', day).normal() * NOISE_SIGMA * this.params.variance+ );+ const weekday = this.shape.dayOfWeek[day.getUTCDay()] ?? 1;+ const anomaly = this.anomaly(day);+ const multiplier = trend * walk * noise * weekday * anomaly.multiplier;+ return {+ date: day,+ visitors: Math.round(baseVisitors * multiplier),+ weekday,+ trend,+ walk,+ noise,+ anomaly,+ multiplier,+ };+ }++ /** Fraction of a day's visitors per hour, with a little per-day jitter on the curve. */+ hourWeights(day: Date): number[] {+ const rng = this.rng('hours', day);+ const weights: number[] = [];+ let sum = 0;+ for (let hour = 0; hour < HOURS_PER_DAY; hour++) {+ const jitter = Math.max(+ HOUR_JITTER_MIN,+ 1 + rng.normal() * HOUR_JITTER_SIGMA * this.params.variance+ );+ const weight = (this.shape.hourly[hour] ?? 0) * jitter;+ weights.push(weight);+ sum += weight;+ }+ return weights.map((weight) => weight / sum);+ }+}diff --git a/packages/seed/src/world.ts b/packages/seed/src/world.tsnew file mode 100644index 000000000..87daac8ab--- /dev/null+++ b/packages/seed/src/world.ts@@ -0,0 +1,232 @@+// Per-project population: who comes back, and as whom. Visitors live in+// cohorts by first-seen day; a returning session picks a cohort through the+// archetype's retention curve, then a visitor through its stickiness. The+// device id rotates at UTC midnight like ingest's salted id, so anonymous+// traffic is only recognised within a day while identified people persist.+// Bounded: the oldest cohort is dropped when the pool is full, which is what+// lets profiles be written exactly once.++import { parseUserAgent } from '@openpanel/shared/server';+import type { Archetype, RetentionCurve } from './archetypes/archetype';+import type { ClickhouseProfileRow } from './clickhouse-rows';+import {+ DESKTOP_USER_AGENTS,+ MOBILE_USER_AGENTS,+ NATIVE_APP_USER_AGENTS,+} from './data/devices';+import { newGeo } from './data/geo';+import type { DeviceWindow, ParsedDevice, Visitor } from './model';+import type { Rng, Weighted } from './rng';+import { deviceProfileRow, personProfileRow } from './rows';+import { VISITOR_POOL_SIZE } from './seed.constants';++const DEVICE_ID_HEX_CHARS = 32;+const MS_PER_DAY = 86_400_000;+/** How many candidates a returning pick tries before settling for the last one. */+const STICKINESS_ATTEMPTS = 6;+const MAX_STICKINESS = 4;++const parsedDevices = new Map<string, ParsedDevice>();++function parseDevice(userAgent: string): ParsedDevice {+ let parsed = parsedDevices.get(userAgent);+ if (!parsed) {+ parsed = parseUserAgent(userAgent);+ parsedDevices.set(userAgent, parsed);+ }+ return parsed;+}++function utcDay(date: Date): string {+ return date.toISOString().slice(0, 10);+}++/** Share of a cohort active `days` after its first day, interpolated on a log scale. */+export function retentionAt(curve: RetentionCurve, days: number): number {+ if (days <= 0) {+ return curve.sameDay;+ }+ const points: [number, number][] = [+ [1, curve.day1],+ [7, curve.day7],+ [30, curve.day30],+ ];+ for (let i = 0; i < points.length - 1; i++) {+ const [fromDay, fromValue] = points[i] as [number, number];+ const [toDay, toValue] = points[i + 1] as [number, number];+ if (days <= toDay) {+ const t = (days - fromDay) / (toDay - fromDay);+ return fromValue * (toValue / fromValue) ** t;+ }+ }+ // Past day 30 the 7→30 decay continues.+ const weeklyDecay = (curve.day30 / curve.day7) ** (7 / 23);+ return curve.day30 * weeklyDecay ** ((days - 30) / 7);+}++export class World {+ private readonly cohorts = new Map<number, Visitor[]>();+ private size = 0;++ constructor(+ private readonly archetype: Archetype,+ private readonly projectId: string,+ private readonly runStart: Date,+ private readonly onProfiles: (rows: ClickhouseProfileRow[]) => void+ ) {}++ private dayIndex(at: Date): number {+ return Math.floor((at.getTime() - this.runStart.getTime()) / MS_PER_DAY);+ }++ /** A returning visitor with probability `returningShare`, else a new one added to today's cohort. */+ pick(rng: Rng, at: Date): Visitor {+ const visitor =+ (this.size > 0 && rng.chance(this.archetype.returningShare)+ ? this.returning(rng, at)+ : null) ?? this.newVisitor(rng, at);+ this.rotateDevice(rng, visitor, at);+ return visitor;+ }++ private returning(rng: Rng, at: Date): Visitor | null {+ const today = this.dayIndex(at);+ const weighted: Weighted<Visitor[]>[] = [];+ for (const [day, visitors] of this.cohorts) {+ const weight =+ visitors.length * retentionAt(this.archetype.retention, today - day);+ if (weight > 0) {+ weighted.push({ value: visitors, weight });+ }+ }+ if (weighted.length === 0) {+ return null;+ }+ const cohort = rng.pick(weighted);+ let candidate: Visitor | null = null;+ for (let attempt = 0; attempt < STICKINESS_ATTEMPTS; attempt++) {+ candidate = cohort[rng.int(0, cohort.length - 1)] ?? null;+ // A visitor "in the future" relative to this session cannot be its visitor.+ if (!candidate || candidate.lastSeen.getTime() > at.getTime()) {+ candidate = null;+ continue;+ }+ if (+ rng.chance(+ Math.min(MAX_STICKINESS, this.archetype.stickiness(candidate)) /+ MAX_STICKINESS+ )+ ) {+ return candidate;+ }+ }+ return candidate;+ }++ private newVisitor(rng: Rng, at: Date): Visitor {+ const userAgent = this.pickUserAgent(rng);+ const day = this.dayIndex(at);+ const visitor: Visitor = {+ userAgent,+ device: parseDevice(userAgent),+ geo: newGeo(rng),+ person: this.archetype.newPerson(rng),+ identified: false,+ window: this.newWindow(rng, at),+ cohortDay: day,+ sessions: 0,+ firstSeen: at,+ lastSeen: at,+ lastPath: '',+ lastReferrer: null,+ state: {},+ };+ if (this.size >= VISITOR_POOL_SIZE) {+ this.dropOldestCohort();+ }+ const cohort = this.cohorts.get(day) ?? [];+ cohort.push(visitor);+ this.cohorts.set(day, cohort);+ this.size += 1;+ return visitor;+ }++ private newWindow(rng: Rng, at: Date): DeviceWindow {+ return {+ id: rng.hex(DEVICE_ID_HEX_CHARS),+ day: utcDay(at),+ firstSeen: at,+ lastSeen: at,+ anonymousEvents: false,+ };+ }++ /** A new UTC day means a new salted id; the old one's anonymous profile is final. */+ private rotateDevice(rng: Rng, visitor: Visitor, at: Date): void {+ if (visitor.window.day === utcDay(at)) {+ return;+ }+ this.flushDevice(visitor);+ visitor.window = this.newWindow(rng, at);+ }++ private flushDevice(visitor: Visitor): void {+ if (visitor.window.anonymousEvents) {+ this.onProfiles([deviceProfileRow(visitor, this.projectId)]);+ }+ }++ /** Records a finished session: who it was sent as, and when it ended. */+ touch(+ visitor: Visitor,+ end: Date,+ options: { identified: boolean; anonymousEvents: boolean }+ ): void {+ visitor.sessions += 1;+ visitor.identified ||= options.identified;+ visitor.window.anonymousEvents ||= options.anonymousEvents;+ if (end.getTime() > visitor.lastSeen.getTime()) {+ visitor.lastSeen = end;+ }+ if (end.getTime() > visitor.window.lastSeen.getTime()) {+ visitor.window.lastSeen = end;+ }+ }++ private dropOldestCohort(): void {+ const oldest = Math.min(...this.cohorts.keys());+ const visitors = this.cohorts.get(oldest) ?? [];+ this.cohorts.delete(oldest);+ this.size -= visitors.length;+ this.flushVisitors(visitors);+ }++ private flushVisitors(visitors: readonly Visitor[]): void {+ for (const visitor of visitors) {+ this.flushDevice(visitor);+ if (visitor.identified && visitor.person) {+ this.onProfiles([personProfileRow(visitor, this.projectId)]);+ }+ }+ }++ /** Writes every remaining profile; call once generation is done. */+ drain(): void {+ for (const visitors of this.cohorts.values()) {+ this.flushVisitors(visitors);+ }+ this.cohorts.clear();+ this.size = 0;+ }++ private pickUserAgent(rng: Rng): string {+ if (this.archetype.deviceClass === 'native') {+ return rng.pick(NATIVE_APP_USER_AGENTS);+ }+ return rng.pick(+ rng.chance(this.archetype.mobileShare)+ ? MOBILE_USER_AGENTS+ : DESKTOP_USER_AGENTS+ );+ }+}diff --git a/packages/seed/tsconfig.json b/packages/seed/tsconfig.jsonnew file mode 100644index 000000000..4ca92684f--- /dev/null+++ b/packages/seed/tsconfig.json@@ -0,0 +1,10 @@+{+ "extends": "@openpanel/tsconfig/base.json",+ "compilerOptions": {+ "tsBuildInfoFile": "node_modules/.cache/tsbuildinfo.json",+ "strictNullChecks": true,+ "types": ["bun-types", "node"]+ },+ "include": ["src"],+ "exclude": ["node_modules"]+}