Skip to content

OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.

hub testing

Carl-Gerhard Lindesvärd890207babb8f
diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md
deleted file mode 100644
index cf5295c5b..000000000
--- a/.claude/CLAUDE.md
+++ /dev/null
@@ -1,168 +0,0 @@
-# CLAUDE.md
-
-NEVER CALL FORMAT! WE'LL FORMAT IN THE FUTURE WHEN WE HAVE MERGED ALL BIG PRS!
-
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
-
-## Project Overview
-
-Openpanel is an open-source web/product analytics platform (Mixpanel alternative). It's a **bun monorepo** with apps, packages, tooling, and SDKs.
-
-## Common Commands
-
-```bash
-# Development
-bun run dev # Run apps/api (ROLE=all, port 3333) + apps/start dashboard (port 3000) in parallel
-bun run dev:public # Run public/docs site only
-bun run dock:up / dock:down # Start/stop Docker (PostgreSQL, Redis, ClickHouse)
-
-# Code quality
-bun run check # Lint (Biome via Ultracite) + dependency-cruiser (check:deps)
-bun run fix # Auto-fix lint/format issues
-bun run typecheck # Typecheck all packages
-
-# Testing
-bun run test # @openpanel/core, @openpanel/db, @openpanel/api and start's own suites, then root vitest for the rest
-cd packages/core && bun test # core's own suite (ADR-010; bun:test cannot run under vitest)
-bunx vitest run <path> # Run a single vitest test file
-
-# Database
-bun run codegen # Generate the Prisma client + core's geo/bot/ASN data
-bun run migrate # Run Prisma migrations (dev)
-bun run migrate:deploy # Deploy migrations (production - never run this)
-
-# Docker utilities
-bun run dock:ch # ClickHouse CLI
-bun run dock:redis # Redis CLI
-```
-
-## Architecture
-
-### Apps
-
-| App | Stack | Port | Purpose |
-|-----|-------|------|---------|
-| `apps/api` | Elysia on Bun, `ROLE=api\|worker\|all` | 3000 (`API_PORT`) | REST/tRPC API, BullMQ workers, Kafka ingest consumer — one process, role-gated |
-| `apps/start` | TanStack Start (Vite + React 19) | 3000 (dev default; run alongside `apps/api` needs a distinct `API_PORT`) | Dashboard SPA |
-| `apps/public` | Next.js 16 + Fumadocs | - | Marketing/docs site |
-
-The standalone worker app is gone — M9 merged it into `apps/api`, selected at boot by `ROLE`.
-
-### Key Packages
-
-| Package | Purpose |
-|---------|---------|
-| `packages/core` | The backend: every module (`<name>.rpc.ts` / `.routes.ts` / `.jobs.ts` / `.service.ts`), the tRPC router, Elysia route surfaces, the BullMQ job registry, auth macros, the ClickHouse `sql` tag consumer code, shared utilities. `apps/api` is a thin shell that mounts it |
-| `packages/db` | Prisma ORM (PostgreSQL) + ClickHouse client, the `sql` tagged-template query builder, migrations |
-| `packages/redis` | Redis client (Bun `RedisClient`) + caching, locks, pub/sub |
-| `packages/email` | React Email templates via Resend |
-| `packages/payments` | Billing/subscription integration |
-| `packages/sdks/*` | Client SDKs (web, react, next, express, react-native, etc.) |
-
-### Data Flow
-
-1. **Event ingestion**: Client SDKs → `apps/api` `/track` → Kafka/Redpanda topic `events`
-2. **Processing**: `apps/api`'s Kafka consumer (any non-`api` role) batches events into ClickHouse
-3. **Dashboard queries**: `apps/start` → tRPC (fetch adapter) → `apps/api` → ClickHouse (analytics) / PostgreSQL (config)
-4. **Real-time**: WebSocket via Elysia, pub/sub via Redis
-
-### Three-Database Strategy
-
-- **PostgreSQL**: Relational data (users, orgs, projects, dashboards). Managed by Prisma.
-- **ClickHouse**: Analytics event storage (OLAP). High-volume reads/writes.
-- **Redis**: Caching, BullMQ job queues, rate limiting, pub/sub.
-
-### Dashboard (apps/start)
-
-Uses TanStack Router with file-based routing (`src/routes/`). State management via Redux Toolkit. UI built on Radix primitives + Tailwind v4. Charts via Recharts. Modals in `src/modals/`.
-
-### API (apps/api)
-
-Elysia on Bun, mounting `@openpanel/core`'s routers. No bundler — `bun run src/main.ts` runs the TypeScript directly (ADR-010). `main.ts` boots deps, `app.ts` wires the four Elysia scopes (root, dashboard, public API, ops); `ROLE` selects whether workers, schedulers and the Kafka consumer also start in this process.
----
-
-## Core Principles
-
-Write code that is **accessible, performant, type-safe, and maintainable**. Focus on clarity and explicit intent over brevity.
-
-### Type Safety & Explicitness
-
-- Use explicit types for function parameters and return values when they enhance clarity
-- Prefer `unknown` over `any` when the type is genuinely unknown
-- Use const assertions (`as const`) for immutable values and literal types
-- Leverage TypeScript's type narrowing instead of type assertions
-- Use meaningful variable names instead of magic numbers - extract constants with descriptive names
-
-### Modern JavaScript/TypeScript
-
-- Use arrow functions for callbacks and short functions
-- Prefer `for...of` loops over `.forEach()` and indexed `for` loops
-- Use optional chaining (`?.`) and nullish coalescing (`??`) for safer property access
-- Prefer template literals over string concatenation
-- Use destructuring for object and array assignments
-- Use `const` by default, `let` only when reassignment is needed, never `var`
-
-### Async & Promises
-
-- Always `await` promises in async functions - don't forget to use the return value
-- Use `async/await` syntax instead of promise chains for better readability
-- Handle errors appropriately in async code with try-catch blocks
-- Don't use async functions as Promise executors
-
-### React & JSX
-
-- Use function components over class components
-- Call hooks at the top level only, never conditionally
-- Specify all dependencies in hook dependency arrays correctly
-- Use the `key` prop for elements in iterables (prefer unique IDs over array indices)
-- Nest children between opening and closing tags instead of passing as props
-- Don't define components inside other components
-- Use semantic HTML and ARIA attributes for accessibility:
- - Provide meaningful alt text for images
- - Use proper heading hierarchy
- - Add labels for form inputs
- - Include keyboard event handlers alongside mouse events
- - Use semantic elements (`<button>`, `<nav>`, etc.) instead of divs with roles
-
-### Error Handling & Debugging
-
-- Remove `console.log`, `debugger`, and `alert` statements from production code
-- Throw `Error` objects with descriptive messages, not strings or other values
-- Use `try-catch` blocks meaningfully - don't catch errors just to rethrow them
-- Prefer early returns over nested conditionals for error cases
-
-### Code Organization
-
-- Keep functions focused and under reasonable cognitive complexity limits
-- Extract complex conditions into well-named boolean variables
-- Use early returns to reduce nesting
-- Prefer simple conditionals over nested ternary operators
-- Group related code together and separate concerns
-
-### Security
-
-- Add `rel="noopener"` when using `target="_blank"` on links
-- Avoid `dangerouslySetInnerHTML` unless absolutely necessary
-- Don't use `eval()` or assign directly to `document.cookie`
-- Validate and sanitize user input
-
-### Performance
-
-- Avoid spread syntax in accumulators within loops
-- Use top-level regex literals instead of creating them in loops
-- Prefer specific imports over namespace imports
-- Avoid barrel files (index files that re-export everything)
-- Use proper image components (e.g., Next.js `<Image>`) over `<img>` tags
-
-### Framework-Specific Guidance
-
-**Next.js:**
-- Use Next.js `<Image>` component for images
-- Use `next/head` or App Router metadata API for head elements
-- Use Server Components for async data fetching instead of async Client Components
-
-**React 19+:**
-- Use ref as a prop instead of `React.forwardRef`
-
-**Solid/Svelte/Vue/Qwik:**
-- Use `class` and `for` attributes (not `className` or `htmlFor`)
diff --git a/.gitignore b/.gitignore
index 482baa8ac..02e89ab2f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -195,3 +195,14 @@ dist
.worktree-port
.worktree-db
WORKTREE.md
+
+# seed output and playwright auth state
+.seed.json
+apps/start/e2e/.auth/
+
+# playwright output
+apps/start/test-results/
+apps/start/playwright-report/
+
+# SQL dumps the ClickHouse code-migration runner writes next to each migration
+packages/db/src/code-migrations/*.sql
diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 000000000..660a8ead2
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1,191 @@
+# OpenPanel — agent guide
+
+**If `WORKTREE.md` exists in the repo root, read it before anything else.** The tooling
+that created this worktree generates it: URLs, ports, database names, the seeded login
+and client secrets, where the process logs are and how to restart a process. Never guess
+a URL or port — take them from `WORKTREE.md` or from your shell env (`$DASHBOARD_URL`,
+`$API_URL`, `$DATABASE_URL`, `$CLICKHOUSE_URL`, `$REDIS_URL`).
+
+Backend conventions (module layout, procedure builders, queues, tests) are the
+contract in `packages/core/AGENTS.md`. Read it before touching `packages/core`.
+
+## What this is
+
+Open-source product analytics (a Mixpanel alternative). Bun monorepo.
+
+| Path | What | Stack |
+|---|---|---|
+| `packages/core` | The backend: every module, the tRPC router, Elysia routes, BullMQ jobs, ingest | TypeScript |
+| `apps/api` | Thin shell that mounts core; `ROLE=api\|worker\|all` picks what one process runs | Elysia on Bun |
+| `apps/start` | The dashboard | TanStack Start, React 19, Redux Toolkit, Tailwind v4, Recharts |
+| `packages/db` | Prisma (Postgres) + ClickHouse client and the `sql` tag (`src/clickhouse/sql.ts`) | |
+| `packages/seed` | Deterministic seed data for Postgres + ClickHouse (`bun run seed`) | |
+| `packages/redis`, `packages/email`, `packages/payments`, `packages/sdks/*` | Redis client/locks/pubsub, React Email, billing, client SDKs | |
+| `apps/public` | Marketing + docs | Next.js, Fumadocs |
+
+Data flow: SDK → `POST /track` (`packages/core/src/modules/ingest`) → Kafka → the
+worker role → ClickHouse (`events`, `sessions`, `profiles`, materialized views).
+Dashboard → tRPC (`$API_URL/trpc`) → ClickHouse for analytics, Postgres for config.
+
+## Your environment
+
+You are usually inside a **worktree** under `.worktrees/<name>` with its own port block,
+its own Postgres/ClickHouse database (`openpanel_<name>`), Redis db index and Kafka
+topics. `WORKTREE.md` says which, and whether the processes (`web`, `api`, `studio`) are
+supervised for you.
+
+- **Supervised processes**: `WORKTREE.md` gives the log files and the exact restart
+ command. Prefer restarting over starting your own copy — two processes on one port is
+ the usual cause of "my change does nothing". If the supervisor is gone, your shell has
+ the same env it used, so `bun run --filter @openpanel/api dev` (or `start`) starts a
+ process on the right port. `tail -f` the log while you test.
+- Env is injected into your shell and wins over `.env`; `.env` in a worktree is a copy of
+ the main checkout's and mostly irrelevant.
+- Never point at another worktree's database or edit files under `.worktrees/`.
+- Migrations: `bun run --filter @openpanel/db migrate:deploy` runs both Prisma and the
+ ClickHouse code migrations against *your* databases (already run when the worktree was
+ created and seeded).
+
+Without `WORKTREE.md` you are in a plain checkout: `.env` applies, `bun run dock:up`
+starts Postgres/Redis/ClickHouse/Redpanda, and `bun run dev` starts api + dashboard.
+The dashboard's server side runs under the Cloudflare Vite plugin, which reads
+`apps/start/.env.local` (and wrangler `vars`) rather than your shell — set `API_URL` and
+`DASHBOARD_URL` there, or export `CLOUDFLARE_INCLUDE_PROCESS_ENV=true` to make it use
+the shell env.
+
+## Data
+
+- `bun run seed` — seeds a user, organization, projects and clients into Postgres and
+ realistic events/sessions/profiles into ClickHouse. `--size small` (default, ~14
+ days), `medium`, `large`, `xl` (tens of millions of events, for query-performance
+ work), or `--days N --sessions-per-day N`. `--reset` wipes the seeded projects first.
+ Same `--seed` ⇒ identical data. `--dry-run` prints the row estimate.
+- `.seed.json` (gitignored, repo root) holds what was seeded: login, organization and
+ project ids, client ids and secrets, and per project which funnels and breakdowns the
+ data is built to show. `WORKTREE.md` repeats the important ones. The four projects
+ (web, SaaS, shop, app) differ in identity: only identified projects have retention;
+ see `packages/seed/README.md`.
+- Direct database access: `bun run --filter @openpanel/db studio` (Prisma Studio on
+ `$STUDIO_PORT`), or `psql "$DATABASE_URL"` / `clickhouse-client` / `redis-cli` with the
+ URLs from your env.
+
+## How to verify your work
+
+Verify like a user would, then like CI would. Pick the rows that match what you changed;
+a one-line fix does not need the full suite, a query or ingest change always does.
+
+| You changed | Verify with | Also run |
+|---|---|---|
+| Dashboard UI (`apps/start`) | Open `$DASHBOARD_URL` with the Playwright MCP if you have one (`browser_*` tools; the local HTTPS certificate is self-signed, ignore that), log in with `.seed.json`, use the feature, **take a screenshot** (`browser_take_screenshot`; you get the image back — look at it) | `bunx vitest run` in `apps/start`, then add/adjust a spec in `apps/start/e2e/` and `bun run e2e` |
+| tRPC procedure / REST route (`packages/core`) | Call it (recipes below) with seeded ids and check the JSON; for auth changes, try the wrong role too | `bun test --isolate` in `packages/core` for that module |
+| ClickHouse query / chart / overview | Run old and new against the seeded data and diff results; check the numbers on the dashboard match a manual query | `packages/core` tests for the module; `bun run --filter @openpanel/db test` if you touched `packages/db` |
+| Ingest / worker / buffers / sessions | `bun run send journey --project acme-shop` (live events through Kafka → worker → ClickHouse), then query `events`/`sessions`/`profiles` for that session id | `packages/core` ingest + buffer tests, `apps/api/e2e/` for session lifecycle |
+| MCP tools (`packages/core/src/modules/mcp`) | Call the product's MCP against *your* API and data: as the `openpanel` MCP server if one is configured for you, else over HTTP (`.seed.json` → `mcp.url`, recipe below); compare its answer with a direct query | `packages/core` mcp tests |
+| Schema / migration | `bun run --filter @openpanel/db migrate:deploy` on your isolated database, then `bun run seed --reset` and the seed tests (`bun run --filter @openpanel/seed test`) | `bun run typecheck` |
+| Anything shared (`packages/shared`, `packages/db`, config, deps) | — | `bun run typecheck` and `bun run test` in full |
+
+Always before you finish: `bun run typecheck`, and `npx ultracite fix <files you changed>`
+(never bare or on `.` — the repo is not bulk-formatted). Report what you ran and what you saw;
+a screenshot or a query result is evidence, "should work" is not.
+
+Sending real events (`bun run send`, uses the seeded clients and `$API_URL`):
+
+```bash
+bun run send journey --project acme-shop --sessions 2 # replay a generated journey live
+bun run send track purchase --project acme-shop --path /order/1 --prop value=99 --prop payment=card --revenue 99
+bun run send identify usr_123 --project acme-saas --email [email protected] --first-name Ada --prop plan=pro
+```
+
+Request recipes (ids and secrets from `.seed.json`):
+
+```bash
+# Track through the real pipeline
+curl -sS -X POST "$API_URL/track" \
+ -H 'content-type: application/json' \
+ -H "openpanel-client-id: $CLIENT_ID" -H "openpanel-client-secret: $CLIENT_SECRET" \
+ -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/128.0 Safari/537.36' \
+ -H 'x-client-ip: 81.2.69.142' \
+ -d '{"type":"track","payload":{"name":"screen_view","properties":{"__path":"/pricing"}}}'
+
+# Management API (root client)
+curl -sS "$API_URL/manage/projects" -H "openpanel-client-id: $ROOT_ID" -H "openpanel-client-secret: $ROOT_SECRET"
+
+# tRPC as the seeded user: sign in, keep the `session` cookie, call any procedure
+curl -sS -c cookies.txt -X POST "$API_URL/trpc/auth.signInEmail" -H 'content-type: application/json' \
+ -d '{"json":{"email":"[email protected]","password":"openpanel"}}'
+curl -sS -b cookies.txt "$API_URL/trpc/organization.list"
+
+# The product's MCP (streamable HTTP; url with token is `.seed.json` → mcp.url)
+curl -sS -X POST "$MCP_URL" -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
+ -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
+```
+
+Payload shapes for `/track` are in `packages/core/src/modules/ingest/ingest.constants.ts`.
+`apps/api/e2e/` is an HTTP harness for session-lifecycle stress tests.
+
+### Splitting verification across sub-agents
+
+Default: verify your own change yourself; it is small and you have the context. Spawn
+sub-agents only when the change touches several surfaces at once (API + dashboard + MCP,
+or a query used in many reports) — then one sub-agent per surface, in parallel, each with a
+concrete checklist ("open X, do Y, expect Z; report the numbers/screenshot"). They share
+your worktree and databases, so they must not migrate, reseed or restart processes; only
+you do that, before they start. Merge their reports into yours; do not forward them raw.
+
+## How to find things
+
+- A backend module is `packages/core/src/modules/<name>/` with `<name>.rpc.ts`
+ (tRPC), `.routes.ts` (REST), `.jobs.ts` (BullMQ), `.service.ts`, `.constants.ts`
+ (zod schemas + constants, safe to import from the frontend). `Cmd+P report.service`.
+- Dashboard routes: `apps/start/src/routes/` (TanStack file routing); modals in
+ `src/modals/`; tRPC client in `src/integrations/tanstack-query/`.
+- ClickHouse queries use the `sql` tag from `@openpanel/db` (`packages/db/src/clickhouse/sql.ts`).
+ Table names: `packages/db/src/clickhouse/client.ts` (`TABLE_NAMES`). Schema: the
+ numbered files in `packages/db/src/code-migrations/`; a flat copy is `test/clickhouse-schema.sql`.
+- Env schema: `apps/api/src/config/env.ts`. Auth: `packages/core/src/modules/auth/`.
+- The write path (what ends up in ClickHouse): `packages/core/src/modules/event/event.service.ts`
+ (`createEvent`), `packages/core/src/buffers/session-buffer.ts`, `profile-buffer.ts`.
+
+## ClickHouse
+
+Local is a single node; production is sharded (2 shards × 2 replicas, `Distributed`
+tables). A plain `IN (subquery)` on a distributed table silently returns per-shard
+results — use `GLOBAL IN` or a join. Never commit ClickHouse SQL you have not run
+against your worktree's database; when replacing a query, diff old and new results
+on the same data.
+
+## Code style
+
+Formatting and linting is ultracite (Biome), **only on the files you changed**. Do not
+hand-format against it and do not disable rules to make a check pass.
+
+- Constants over magic numbers: named, at the top of the file or in the module's
+ `<name>.constants.ts`.
+- Names reveal purpose; no abbreviations unless universal.
+- Comments say *why*, briefly. Never narrate what the code does. If a function needs
+ a comment to explain what it does, split it.
+- Single responsibility, early returns, no nested ternaries, extract conditions into
+ named booleans.
+- DRY at the narrowest level that fits: the module first, `packages/core/src/shared/`
+ when several backend modules need it, a dedicated package when the frontend or SDKs
+ need it too. Do not invent a cross-module abstraction for two call sites.
+- `unknown` over `any`; `as const`; narrowing over assertions; `for…of` over `forEach`.
+- React: function components, hooks at top level, `key` from ids, ref as a prop
+ (React 19), semantic elements with labels and keyboard handlers.
+- No `console.log` / `debugger` left behind. Throw `Error` objects.
+
+Leave the code you touch cleaner than you found it — inside the task's scope only.
+Do not refactor, upgrade dependencies or fix unrelated things you notice; mention
+them instead.
+
+## Git
+
+One branch per worktree, created from `rewrite/v2`. Commit when a step is done and
+verified, with a message that says what and why. Never push, never add
+remotes, never rewrite history that is not yours. `WORKTREE.md`, `.seed.json` and
+`.worktree-*` are gitignored — keep them that way.
+
+## Ralph
+
+If `/home/deploy/rewrite-openpanel` exists you are the autonomous rewrite loop:
+`CLAUDE.ralph.md` overrides this file.
diff --git a/CLAUDE.md b/CLAUDE.md
index d04d5c843..43c994c2d 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -1,111 +1 @@
-# OpenPanel V2 — autonomous rewrite
-
-This repository is being rewritten by an autonomous loop. Read this before doing
-anything.
-
-## Safety
-
-Your world is exactly two directories: this repository and
-`/home/deploy/rewrite-openpanel` (the controller). Do not read, list, or modify
-anything else on this machine.
-
-Never use production credentials — production does not exist on this machine,
-and nothing here should reach it. Local Postgres / ClickHouse / Redis only.
-
-## Environment
-
-- **No docker.** `bun run dock:up` will fail. Postgres (5432), Redis (6379) and
- ClickHouse (8123) are already running on localhost, started by another user.
- Do not try to start, stop or install them.
-- **No sudo.** Anything needing root is a human task. Say so; do not attempt it.
-- Writes are confined to this repo and `/home/deploy/rewrite-openpanel`. Every
- other path on the box is read-only to you.
-
-## Commands
-
-```bash
-bun install
-bun run codegen # prisma client; needs .env
-bun run typecheck
-bun run check # ultracite + check:deps
-bun run test
-```
-
-`typecheck` and `test` fail until `.env` exists and `bun run codegen` has run.
-If a task depends on them and they are not ready, report `BLOCKED` rather than
-inventing a workaround.
-
-## ClickHouse
-
-Never commit ClickHouse SQL you have not executed. Run it against the local
-server (`http://127.0.0.1:8123`, database `openpanel`) and report timing and row
-count. When replacing a query, diff the old and new result sets on the same data.
-
-Local is a single node; production is 2 shards x 2 replicas. Before editing any
-query against a `Distributed` table read
-`/home/deploy/rewrite-openpanel/docs/ENVIRONMENT.md` - a plain `IN (subquery)` on a
-distributed table silently returns per-shard results.
-
-## Architecture
-
-Architecture decisions live in `/home/deploy/rewrite-openpanel/decisions`.
-**Accepted ADRs are binding.** If you need a decision no accepted ADR covers,
-stop and say `BLOCKED: needs ADR — <question>`. Do not decide it yourself.
-
-The target architecture is `/home/deploy/rewrite-openpanel/docs/TARGET_ARCHITECTURE.md`.
-The documented behaviour of the system you are replacing is in
-`/home/deploy/rewrite-openpanel/docs/current/`. Read the relevant document before
-changing that area — it exists so you do not have to rediscover it.
-
-## Ralph
-
-Do not modify:
-
-```
-/home/deploy/rewrite-openpanel/plan/tasks.json
-/home/deploy/rewrite-openpanel/state
-/home/deploy/rewrite-openpanel/verification
-```
-
-The orchestrator owns task state and owns the definition of "passing".
-
-## Git
-
-Do not commit. The orchestrator manages commits and the `rewrite/v2` branch.
-There are no push credentials on this machine, by design - all commits are local
-and a human pushes them later. Never try to add a remote or authenticate to one.
-
-## Code style
-
-- **Formatting and linting: ultracite** (the Biome preset) — but **only ever on
- the files you changed**, named explicitly (`npx ultracite fix <files...>` or
- `biome check --write <files...>`). NEVER run it bare or on `.`: the repo has
- never been bulk-formatted, so an unscoped run rewrites thousands of files,
- buries your real diff, and fails the task. (This is why format-on-save hooks
- are disabled here.) Do not hand-format against it, and do not disable its
- rules to make a task pass.
-
-- **Constants over magic numbers** — named, descriptive, at the top of the file
- or in the module's `<name>.constants.ts`.
-- **Meaningful names** — reveal purpose; no abbreviations unless universal.
-- **Smart comments** — never narrate what code does; comment only the why, and
- keep it very short. APIs, non-obvious side effects and invariants excepted.
-- **Single responsibility** — small, focused functions; if it needs a comment
- to explain what it does, split it.
-- **DRY** — single sources of truth, shared at the narrowest level that fits:
- inside the module first; `core/shared/` when several backend modules need it;
- a **dedicated package** when anything beyond the backend (frontend, SDKs,
- other apps) needs it — core is not a grab-bag for cross-app code. Exception,
- by accepted decision: module `*.constants.ts` files are consumed by the
- frontend via deep paths into core, not via a package. Never invent a
- cross-module abstraction just to deduplicate two call sites.
-- **Clean structure & encapsulation** — related code together, implementation
- hidden, nested conditionals extracted into well-named functions.
-- **Leave the code you touch cleaner than you found it — inside the task's
- scope only.** Refactoring outside scope is a task of its own; propose it,
- don't do it. Same for technical debt you notice: record it, move on.
-
-## Scope
-
-Do exactly the assigned task. Do not opportunistically refactor, upgrade
-dependencies, or fix unrelated things you notice along the way.
diff --git a/CLAUDE.ralph.md b/CLAUDE.ralph.md
new file mode 100644
index 000000000..d04d5c843
--- /dev/null
+++ b/CLAUDE.ralph.md
@@ -0,0 +1,111 @@
+# OpenPanel V2 — autonomous rewrite
+
+This repository is being rewritten by an autonomous loop. Read this before doing
+anything.
+
+## Safety
+
+Your world is exactly two directories: this repository and
+`/home/deploy/rewrite-openpanel` (the controller). Do not read, list, or modify
+anything else on this machine.
+
+Never use production credentials — production does not exist on this machine,
+and nothing here should reach it. Local Postgres / ClickHouse / Redis only.
+
+## Environment
+
+- **No docker.** `bun run dock:up` will fail. Postgres (5432), Redis (6379) and
+ ClickHouse (8123) are already running on localhost, started by another user.
+ Do not try to start, stop or install them.
+- **No sudo.** Anything needing root is a human task. Say so; do not attempt it.
+- Writes are confined to this repo and `/home/deploy/rewrite-openpanel`. Every
+ other path on the box is read-only to you.
+
+## Commands
+
+```bash
+bun install
+bun run codegen # prisma client; needs .env
+bun run typecheck
+bun run check # ultracite + check:deps
+bun run test
+```
+
+`typecheck` and `test` fail until `.env` exists and `bun run codegen` has run.
+If a task depends on them and they are not ready, report `BLOCKED` rather than
+inventing a workaround.
+
+## ClickHouse
+
+Never commit ClickHouse SQL you have not executed. Run it against the local
+server (`http://127.0.0.1:8123`, database `openpanel`) and report timing and row
+count. When replacing a query, diff the old and new result sets on the same data.
+
+Local is a single node; production is 2 shards x 2 replicas. Before editing any
+query against a `Distributed` table read
+`/home/deploy/rewrite-openpanel/docs/ENVIRONMENT.md` - a plain `IN (subquery)` on a
+distributed table silently returns per-shard results.
+
+## Architecture
+
+Architecture decisions live in `/home/deploy/rewrite-openpanel/decisions`.
+**Accepted ADRs are binding.** If you need a decision no accepted ADR covers,
+stop and say `BLOCKED: needs ADR — <question>`. Do not decide it yourself.
+
+The target architecture is `/home/deploy/rewrite-openpanel/docs/TARGET_ARCHITECTURE.md`.
+The documented behaviour of the system you are replacing is in
+`/home/deploy/rewrite-openpanel/docs/current/`. Read the relevant document before
+changing that area — it exists so you do not have to rediscover it.
+
+## Ralph
+
+Do not modify:
+
+```
+/home/deploy/rewrite-openpanel/plan/tasks.json
+/home/deploy/rewrite-openpanel/state
+/home/deploy/rewrite-openpanel/verification
+```
+
+The orchestrator owns task state and owns the definition of "passing".
+
+## Git
+
+Do not commit. The orchestrator manages commits and the `rewrite/v2` branch.
+There are no push credentials on this machine, by design - all commits are local
+and a human pushes them later. Never try to add a remote or authenticate to one.
+
+## Code style
+
+- **Formatting and linting: ultracite** (the Biome preset) — but **only ever on
+ the files you changed**, named explicitly (`npx ultracite fix <files...>` or
+ `biome check --write <files...>`). NEVER run it bare or on `.`: the repo has
+ never been bulk-formatted, so an unscoped run rewrites thousands of files,
+ buries your real diff, and fails the task. (This is why format-on-save hooks
+ are disabled here.) Do not hand-format against it, and do not disable its
+ rules to make a task pass.
+
+- **Constants over magic numbers** — named, descriptive, at the top of the file
+ or in the module's `<name>.constants.ts`.
+- **Meaningful names** — reveal purpose; no abbreviations unless universal.
+- **Smart comments** — never narrate what code does; comment only the why, and
+ keep it very short. APIs, non-obvious side effects and invariants excepted.
+- **Single responsibility** — small, focused functions; if it needs a comment
+ to explain what it does, split it.
+- **DRY** — single sources of truth, shared at the narrowest level that fits:
+ inside the module first; `core/shared/` when several backend modules need it;
+ a **dedicated package** when anything beyond the backend (frontend, SDKs,
+ other apps) needs it — core is not a grab-bag for cross-app code. Exception,
+ by accepted decision: module `*.constants.ts` files are consumed by the
+ frontend via deep paths into core, not via a package. Never invent a
+ cross-module abstraction just to deduplicate two call sites.
+- **Clean structure & encapsulation** — related code together, implementation
+ hidden, nested conditionals extracted into well-named functions.
+- **Leave the code you touch cleaner than you found it — inside the task's
+ scope only.** Refactoring outside scope is a task of its own; propose it,
+ don't do it. Same for technical debt you notice: record it, move on.
+
+## Scope
+
+Do exactly the assigned task. Do not opportunistically refactor, upgrade
+dependencies, or fix unrelated things you notice along the way.
diff --git a/apps/start/e2e/auth.setup.ts b/apps/start/e2e/auth.setup.ts
new file mode 100644
index 000000000..35576a927
--- /dev/null
+++ b/apps/start/e2e/auth.setup.ts
@@ -0,0 +1,21 @@
+// Logs in once through the real form and saves the session cookie; every
+// other spec starts authenticated (Playwright's storageState pattern).
+
+import { expect, test as setup } from '@playwright/test';
+import { readSeedManifest } from './seed-manifest';
+
+const AUTH_STATE_FILE = 'e2e/.auth/user.json';
+
+setup('sign in as the seeded user', async ({ page }) => {
+ const manifest = readSeedManifest();
+ await page.goto('/login');
+ // The form is server-rendered; a click before hydration submits it natively as a GET.
+ await page.waitForLoadState('networkidle');
+ await page.getByLabel('Email').fill(manifest.user.email);
+ await page.getByLabel('Password').fill(manifest.user.password);
+ await page.getByRole('button', { name: 'Sign in', exact: true }).click();
+ await expect(page).toHaveURL(new RegExp(`/${manifest.organizationId}`), {
+ timeout: 15_000,
+ });
+ await page.context().storageState({ path: AUTH_STATE_FILE });
+});
diff --git a/apps/start/e2e/fixtures.ts b/apps/start/e2e/fixtures.ts
new file mode 100644
index 000000000..6a522b65f
--- /dev/null
+++ b/apps/start/e2e/fixtures.ts
@@ -0,0 +1,12 @@
+import { test as base } from '@playwright/test';
+import { readSeedManifest, type SeedManifest } from './seed-manifest';
+
+export const test = base.extend<{ seed: SeedManifest }>({
+ // Playwright requires the destructured first argument, even when unused.
+ // biome-ignore lint/correctness/noEmptyPattern: Playwright fixture signature
+ seed: async ({}, use) => {
+ await use(readSeedManifest());
+ },
+});
+
+export { expect } from '@playwright/test';
diff --git a/apps/start/e2e/overview.spec.ts b/apps/start/e2e/overview.spec.ts
new file mode 100644
index 000000000..a903975d1
--- /dev/null
+++ b/apps/start/e2e/overview.spec.ts
@@ -0,0 +1,45 @@
+import { expect, test } from './fixtures';
+import { busiestProject } from './seed-manifest';
+
+test('the organization lists the seeded projects', async ({ page, seed }) => {
+ await page.goto(`/${seed.organizationId}`);
+ for (const project of seed.projects) {
+ await expect(page.getByText(project.name, { exact: true })).toBeVisible();
+ }
+});
+
+test('the overview renders its metric cards for a seeded project', async ({
+ page,
+ seed,
+}) => {
+ const project = busiestProject(seed);
+ await page.goto(`/${seed.organizationId}/${project.id}`);
+ for (const metric of [
+ 'Unique Visitors',
+ 'Sessions',
+ 'Pageviews',
+ 'Bounce Rate',
+ ]) {
+ await expect(
+ page.getByRole('button', { name: new RegExp(`^${metric}`) })
+ ).toBeVisible();
+ }
+});
+
+test('sessions and profiles pages list seeded rows', async ({ page, seed }) => {
+ const project = busiestProject(seed);
+ const base = `/${seed.organizationId}/${project.id}`;
+
+ await page.goto(`${base}/sessions`);
+ await expect(
+ page.getByRole('heading', { name: 'Sessions', level: 1 })
+ ).toBeVisible();
+ await expect(
+ page.locator(`a[href^="${base}/sessions/"]`).first()
+ ).toBeVisible();
+
+ await page.goto(`${base}/profiles`);
+ await expect(
+ page.locator(`a[href^="${base}/profiles/"]`).first()
+ ).toBeVisible();
+});
diff --git a/apps/start/e2e/seed-manifest.ts b/apps/start/e2e/seed-manifest.ts
new file mode 100644
index 000000000..411c5ae64
--- /dev/null
+++ b/apps/start/e2e/seed-manifest.ts
@@ -0,0 +1,52 @@
+// The seed's `.seed.json` at the repo root: who to log in as and which
+// projects carry data. `bun run seed` writes it.
+
+import { existsSync, readFileSync } from 'node:fs';
+import { resolve } from 'node:path';
+
+export interface SeedManifest {
+ user: { email: string; password: string };
+ organizationId: string;
+ rootClient: { id: string; secret: string };
+ projects: {
+ id: string;
+ name: string;
+ archetype: string;
+ clientId: string;
+ clientSecret: string;
+ sessions: number;
+ events: number;
+ }[];
+ range: { from: string; to: string };
+}
+
+const MANIFEST_PATH = resolve(
+ import.meta.dirname,
+ '..',
+ '..',
+ '..',
+ '.seed.json'
+);
+
+export function readSeedManifest(): SeedManifest {
+ if (!existsSync(MANIFEST_PATH)) {
+ throw new Error(
+ `No ${MANIFEST_PATH}. Run \`bun run seed\` first; the e2e suite logs in with what it seeded.`
+ );
+ }
+ return JSON.parse(readFileSync(MANIFEST_PATH, 'utf8')) as SeedManifest;
+}
+
+/** The seeded project with the most sessions; the overview specs look at it. */
+export function busiestProject(
+ manifest: SeedManifest
+): SeedManifest['projects'][number] {
+ const [first, ...rest] = manifest.projects;
+ if (!first) {
+ throw new Error('.seed.json lists no projects');
+ }
+ return rest.reduce(
+ (best, project) => (project.sessions > best.sessions ? project : best),
+ first
+ );
+}
diff --git a/apps/start/package.json b/apps/start/package.json
index 03f31c1b7..5a65133bd 100644
--- a/apps/start/package.json
+++ b/apps/start/package.json
@@ -10,6 +10,7 @@
"build": "bun run with-env vite build",
"serve": "vite preview",
"test": "vitest run",
+ "e2e": "playwright test",
"format": "biome format",
"lint": "biome lint",
"check": "biome check",
@@ -161,6 +162,7 @@
"@biomejs/biome": "1.9.4",
"@cloudflare/vite-plugin": "1.33.2",
"@openpanel/db": "workspace:*",
+ "@playwright/test": "1.63.0",
"@tanstack/devtools-event-client": "0.3.3",
"@testing-library/dom": "10.4.1",
"@testing-library/react": "16.3.0",
diff --git a/apps/start/playwright.config.ts b/apps/start/playwright.config.ts
new file mode 100644
index 000000000..bd6ff1cb3
--- /dev/null
+++ b/apps/start/playwright.config.ts
@@ -0,0 +1,30 @@
+// Runs against an already-running dashboard (a worktree's processes are supervised
+// for it), so there is no `webServer` block. `DASHBOARD_URL` names it; a plain
+// checkout sets it or falls back to the dev port.
+
+import { defineConfig, devices } from '@playwright/test';
+
+const DEV_DASHBOARD_URL = 'http://localhost:3000';
+const AUTH_STATE_FILE = 'e2e/.auth/user.json';
+
+export default defineConfig({
+ testDir: 'e2e',
+ fullyParallel: false,
+ workers: 1,
+ retries: process.env.CI ? 1 : 0,
+ reporter: process.env.CI ? 'github' : 'list',
+ use: {
+ baseURL: process.env.DASHBOARD_URL ?? DEV_DASHBOARD_URL,
+ // A worktree's HTTPS certificate is from a local CA Playwright's Chromium does not trust.
+ ignoreHTTPSErrors: true,
+ trace: 'retain-on-failure',
+ },
+ projects: [
+ { name: 'setup', testMatch: /auth\.setup\.ts/ },
+ {
+ name: 'chromium',
+ use: { ...devices['Desktop Chrome'], storageState: AUTH_STATE_FILE },
+ dependencies: ['setup'],
+ },
+ ],
+});
diff --git a/bun.lock b/bun.lock
index c833506e6..952ba2146 100644
--- a/bun.lock
+++ b/bun.lock
@@ -238,6 +238,7 @@
"@biomejs/biome": "1.9.4",
"@cloudflare/vite-plugin": "1.33.2",
"@openpanel/db": "workspace:*",
+ "@playwright/test": "1.63.0",
"@tanstack/devtools-event-client": "0.3.3",
"@testing-library/dom": "10.4.1",
"@testing-library/react": "16.3.0",
@@ -553,6 +554,21 @@
"typescript": "catalog:",
},
},
+ "packages/seed": {
+ "name": "@openpanel/seed",
+ "version": "0.0.1",
+ "dependencies": {
+ "@node-rs/argon2": "2.0.2",
+ "@openpanel/db": "workspace:*",
+ "@openpanel/shared": "workspace:*",
+ },
+ "devDependencies": {
+ "@openpanel/tsconfig": "workspace:*",
+ "@types/node": "catalog:",
+ "bun-types": "1.4.0",
+ "typescript": "catalog:",
+ },
+ },
"packages/shared": {
"name": "@openpanel/shared",
"version": "0.0.1",
@@ -1579,6 +1595,8 @@
"@openpanel/sdk-info": ["@openpanel/sdk-info@workspace:packages/sdks/_info"],
+ "@openpanel/seed": ["@openpanel/seed@workspace:packages/seed"],
+
"@openpanel/shared": ["@openpanel/shared@workspace:packages/shared"],
"@openpanel/testbed": ["@openpanel/testbed@workspace:apps/testbed"],
@@ -1903,6 +1921,8 @@
"@pkgjs/parseargs": ["@pkgjs/[email protected]", "", {}, "sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg=="],
+ "@playwright/test": ["@playwright/[email protected]", "", { "dependencies": { "playwright": "1.63.0" }, "bin": { "playwright": "cli.js" } }, "sha512-oxMK4vllB9RK5NQ2l1pq1IfOf2AvnEuj/vYGDj0H2nMtmtZpKtCwt/l00GEO6xjGfpBNAvjovvYdCm50dRQkpQ=="],
+
"@polar-sh/sdk": ["@polar-sh/[email protected]", "", { "dependencies": { "standardwebhooks": "^1.0.0", "zod": "^3.25.65 || ^4.0.0" } }, "sha512-9UYb70iKjJCtWYlu0OF5HLYBLmkxHwqr2RlXwuxXQgRGqq56IQWlVG+NO7e1YJ7I5GW0CBHhGIjRbQ9hYM6ycQ=="],
"@polka/url": ["@polka/[email protected]", "", {}, "sha512-wwQAWhWSuHaag8c4q/KN/vCoeOJYshAIvMQwD4GpSb3OiZklFfvAgmj0VCBBImRpuF/aFgIRzllXlVX93Jevww=="],
@@ -4851,6 +4871,10 @@
"pkg-types": ["[email protected]", "", { "dependencies": { "confbox": "^0.3.1", "exsolve": "^1.1.1", "pathe": "^2.0.3" } }, "sha512-j/lCFdcppV0JxWpCEITdbDltBxPP6cHT+yNJ6Go2OgoSA9518X847X9z0p6LtA4Nc16+eQzCZjRrWanTGvHJ5w=="],
+ "playwright": ["[email protected]", "", { "dependencies": { "playwright-core": "1.63.0" }, "bin": { "playwright": "cli.js" } }, "sha512-+7ziBLidS4NaNCdt57SUDT+wYmmd5fmiQejUic/kb+YsYSCPyOOE9sebzMjNmQrsnNpDJqd4WHvV/8lfKfUDUg=="],
+
+ "playwright-core": ["[email protected]", "", { "bin": { "playwright-core": "cli.js" } }, "sha512-rYCsBF/M5HjUch52bbtVONEFjv6Xu8sm8h72dNlR5bzIE1fvC/bxgspzkjSfU+MweEMmPM8KJebG6nnyxo5mCg=="],
+
"please-upgrade-node": ["[email protected]", "", { "dependencies": { "semver-compare": "^1.0.0" } }, "sha512-gQR3WpIgNIKwBMVLkpMUeR3e1/E1y42bqDQZfql+kDeXd8COYfM8PQA4X6y7a8u9Ua9FHmsrrmirW2vHs45hWg=="],
"plist": ["[email protected]", "", { "dependencies": { "@xmldom/xmldom": "^0.9.10", "base64-js": "^1.5.1", "xmlbuilder": "^15.1.1" } }, "sha512-ZIfcLJC+7E7FBFnDxm9MPmt7D+DidyQ26lewieO75AdhA2ayMtsJSES0iWzqJQbcVRSrTufQoy0DR94xHue0oA=="],
diff --git a/hub.config.ts b/hub.config.ts
index 70c17639d..a4f030a6f 100644
--- a/hub.config.ts
+++ b/hub.config.ts
@@ -1,3 +1,17 @@
+import { readFileSync } from 'node:fs';
+import { join } from 'node:path';
+
+/** The product's MCP on this worktree's API, authenticated with the seed's root client. */
+function seededMcp(w: { dir: string; url(process: string): string }): Record<string, { url: string }> {
+ try {
+ const seed = JSON.parse(readFileSync(join(w.dir, '.seed.json'), 'utf8'));
+ if (typeof seed?.mcp?.token !== 'string') return {};
+ return { openpanel: { url: `${w.url('api')}/mcp?token=${seed.mcp.token}` } };
+ } catch {
+ return {};
+ }
+}
+
// The one file the hub reads from this repo. Its shape is `hub.config.d.ts` in the hub repo,
// which type-checks this file in its own tests; a field the hub does not know fails there.
export default {
@@ -6,7 +20,7 @@ export default {
worktreesDir: '.worktrees',
github: 'Openpanel-dev/openpanel',
domains: { local: 'local.openpanel.cc', remote: 'openpanel.cc' },
- install: 'bun install && bun run --filter @openpanel/db codegen',
+ install: 'bun install && bun run --filter @openpanel/db codegen && bunx playwright install chromium',
// port = 21000 + block×100 + offset. Main checkout is block 0.
processes: {
@@ -18,6 +32,10 @@ export default {
// Injected as process env. Wins over .env because dotenv-cli runs without -o.
env: (w) => ({
+ // The dashboard's SSR runs under the Cloudflare Vite plugin, whose worker only sees wrangler
+ // vars and .env.local unless told to take the process env; without this it would SSR
+ // against production's API_URL.
+ CLOUDFLARE_INCLUDE_PROCESS_ENV: 'true',
API_PORT: w.port('api'),
API_HOST: '127.0.0.1', // Bun.serve defaults to 0.0.0.0
WEB_PORT: w.port('web'),
@@ -48,6 +66,12 @@ export default {
redis: { index: (block) => block }, // always per worktree
kafka: { topics: (n) => [`events-${n}`, `events-${n}-dlq`] },
migrate: 'bun run --filter @openpanel/db migrate:deploy',
+ // Presets of packages/seed; each writes .seed.json and prints the logins it created.
+ seed: {
+ small: 'bun run --filter @openpanel/seed seed -- --size small --reset',
+ medium: 'bun run --filter @openpanel/seed seed -- --size medium --reset',
+ large: 'bun run --filter @openpanel/seed seed -- --size large --reset',
+ },
},
checks: {
@@ -65,6 +89,16 @@ export default {
kafka: (w) => `docker compose -f ${w.root}/docker-compose.yml exec op-rp sh -c 'rpk topic list; echo; echo "this worktree: ${w.db.kafkaTopics.join(' ')}"; exec sh'`,
},
+ // Every agent the hub launches can drive the dashboard in a real browser (--ignore-https-errors
+ // only skips certificate checks for the hub's local CA; cookies and login are unaffected), and
+ // talks to this worktree's own OpenPanel MCP once the seed has written its root-client token.
+ agents: {
+ mcp: (w) => ({
+ playwright: { command: 'npx', args: ['-y', '@playwright/mcp@latest', '--isolated', '--ignore-https-errors'] },
+ ...seededMcp(w),
+ }),
+ },
+
stack: { up: 'docker compose up -d op-db op-kv op-ch op-rp', cwd: '.' },
auth: process.platform === 'darwin' ? 'off' : 'password',
issues: { github: true, userjot: true },
diff --git a/package.json b/package.json
index f503e2964..5ea4f119c 100644
--- a/package.json
+++ b/package.json
@@ -52,7 +52,7 @@
"vue": "3.5.25"
},
"scripts": {
- "test": "bun run --filter @openpanel/core test && bun run --filter @openpanel/db test && bun run --filter @openpanel/api test && bun run --filter start test && vitest run",
+ "test": "bun run --filter @openpanel/core test && bun run --filter @openpanel/db test && bun run --filter @openpanel/api test && bun run --filter @openpanel/seed test && bun run --filter start test && vitest run",
"gen:bots": "bun run --filter @openpanel/core gen:bots",
"gen:referrers": "bun run --filter @openpanel/core gen:referrers",
"dock:up": "docker compose up -d",
@@ -63,6 +63,9 @@
"codegen": "bun run --filter @openpanel/db codegen && bun run --filter @openpanel/core codegen",
"migrate": "bun run --filter @openpanel/db migrate",
"migrate:deploy": "bun run --filter @openpanel/db migrate:deploy",
+ "seed": "bun run --filter @openpanel/seed seed --",
+ "send": "bun run --filter @openpanel/seed send --",
+ "e2e": "bun run --filter start e2e --",
"postinstall:workspaces": "bun run --filter public postinstall && bun run --filter @openpanel/testbed postinstall",
"dev": "bun run --filter @openpanel/api testing & bun run --filter start dev & wait",
"dev:public": "bun run --filter public dev",
diff --git a/packages/seed/README.md b/packages/seed/README.md
new file mode 100644
index 000000000..53b6b5dde
--- /dev/null
+++ b/packages/seed/README.md
@@ -0,0 +1,60 @@
+# @openpanel/seed
+
+Deterministic seed data for a worktree: one login, one organization, four projects
+with realistic traffic in ClickHouse. `bun run seed` from the repo root.
+
+```
+bun run seed # small: 14 days, ~5k sessions
+bun run seed --size large --reset # 90 days, ~1M sessions; --reset clears earlier seeded rows
+bun run seed --days 3 --sessions-per-day 500 --projects saas,app --seed 7
+bun run seed --dry-run # the plan only
+```
+
+Same flags + seed ⇒ identical rows.
+
+`bun run send` sends *live* requests to `$API_URL` with the seeded clients — through the real
+ingest pipeline, unlike the seed's direct inserts:
+
+```
+bun run send journey --project acme-shop --sessions 2 # generate a journey with the archetype, replay it
+bun run send track purchase --project acme-shop --path /order/1 --prop value=99 --revenue 99
+bun run send identify usr_123 --project acme-saas --email [email protected] --prop plan=pro
+``` Output: `.seed.json` at the repo root (login,
+ids, client secrets, the product's MCP endpoint with a root-client token, and the funnels
+below), for agents and the e2e suite.
+
+## What the data is built to show
+
+| Project | Identity | Funnels (breakdowns) | Cohort-worthy behaviours |
+|---|---|---|---|
+| **Acme Web** `acme-web` (website) | Anonymous; a device is new every UTC day (ingest's salted id). ~18 % sign up and are identified from then on. | Signup: `screen_view → plan_selected → signup_started → signup_completed` (referrer, device, country, plan). Demo: `cta_clicked → demo_requested` | signed up, requested demo, subscribed to newsletter |
+| **Acme SaaS** `acme-saas` (logged-in product) | Everyone signs up/logs in on the first session ⇒ real retention. | Activation: `signup_completed → project_created → sdk_installed → first_event_received` (signup_source, company_size, role, sdk). Upgrade: `pricing_viewed → checkout_started → subscription_started` (plan, interval) | invited a teammate, exported, connected an integration, paid plan, churned — each changes retention |
+| **Acme Shop** `acme-shop` (ecommerce) | Anonymous shoppers; ~40 % of buyers create an account at first purchase. | Checkout: `product_viewed → add_to_cart → checkout_started → shipping_info_added → payment_info_added → purchase` (category, brand, payment, shipping, coupon, device). Search: `search → product_viewed → add_to_cart` | has purchased, has account, used coupon, wishlisted; `purchase.first_purchase` |
+| **Acme App** `acme-app` (React Native) | ~85 % sign up during onboarding. | Onboarding: `app_opened → onboarding_step_completed → signup_completed → notification_permission → post_created` (platform, app_version). Premium: `paywall_viewed → trial_started → subscription_started` (trigger, interval) | notifications granted, premium, has posted, platform |
+
+Conversions (`EventMeta.conversion`) are flagged per project. Revenue events carry
+whole-unit amounts (`events.revenue` is `UInt64`).
+
+## How it is generated
+
+- **Traffic**: per archetype an hourly curve, weekday curve, slow trend, an AR(1)
+ drift, rare spikes/dips; Poisson per hour (`traffic.ts`).
+- **People**: a per-project pool bucketed by first-seen day. A returning session picks
+ a cohort through the archetype's retention curve (`sameDay`, `day1`, `day7`,
+ `day30`) and a visitor through its stickiness (paid plan, notifications, has
+ purchased…). Anonymous device ids rotate at UTC midnight (`world.ts`).
+- **Journeys**: one state machine per archetype (`archetypes/*.ts`) with log-normal
+ dwell times, state that persists across visits (cart, plan, onboarding step) and
+ `identify()` at signup/login/purchase.
+- **Rows**: shaped like the worker's (`rows.ts`) — `session_start`/`session_end`
+ synthesized, one collapsed `sessions` row, one `profiles` row per device window or
+ person. Devices/browsers come from real user agents through the real parser.
+- **Isolation**: depends on `@openpanel/db` and `@openpanel/shared` only; row shapes are
+ copied from core and checked against `test/clickhouse-schema.sql` by
+ `clickhouse-rows.test.ts`.
+
+## Not modelled (yet)
+
+Bots, server-side events, late/duplicate events, groups/aliases, imports, geo-local
+time of day, and very high property cardinality. For volume beyond `xl`, the plan is a
+ClickHouse-side `--amplify` that copies a seeded slice with shifted dates.
diff --git a/packages/seed/package.json b/packages/seed/package.json
new file mode 100644
index 000000000..418ffcb5c
--- /dev/null
+++ b/packages/seed/package.json
@@ -0,0 +1,25 @@
+{
+ "name": "@openpanel/seed",
+ "version": "0.0.1",
+ "private": true,
+ "type": "module",
+ "sideEffects": false,
+ "scripts": {
+ "seed": "bun run with-env bun src/cli.ts",
+ "test": "bun test --isolate",
+ "typecheck": "tsc --noEmit",
+ "with-env": "dotenv -e ../../.env -c --",
+ "send": "bun run with-env bun src/send.ts"
+ },
+ "dependencies": {
+ "@node-rs/argon2": "2.0.2",
+ "@openpanel/db": "workspace:*",
+ "@openpanel/shared": "workspace:*"
+ },
+ "devDependencies": {
+ "@openpanel/tsconfig": "workspace:*",
+ "@types/node": "catalog:",
+ "bun-types": "1.4.0",
+ "typescript": "catalog:"
+ }
+}
diff --git a/packages/seed/src/archetypes/app.ts b/packages/seed/src/archetypes/app.ts
new file mode 100644
index 000000000..18aab2aa8
--- /dev/null
+++ b/packages/seed/src/archetypes/app.ts
@@ -0,0 +1,319 @@
+// Acme App: a native social app (React Native SDK). Most users sign up on
+// the first open and are identified from then on. Funnels: onboarding
+// (app_opened → onboarding → signup → notification permission → first post)
+// and monetization (paywall_viewed → trial_started → subscription_started).
+// Cohort-worthy: platform, notifications granted, premium, has posted.
+
+import { newPerson } from '../data/people';
+import type { Visitor } from '../model';
+import type { Rng } from '../rng';
+import type { Archetype, Journey } from './archetype';
+
+const SCREENS = {
+ home: ['Home', 'Home'],
+ feed: ['Feed', 'Feed'],
+ post: ['Post', 'Post'],
+ create: ['CreatePost', 'New post'],
+ profile: ['Profile', 'Profile'],
+ settings: ['Settings', 'Settings'],
+ onboarding: ['Onboarding', 'Onboarding'],
+ paywall: ['Paywall', 'Go premium'],
+ search: ['Search', 'Search'],
+} as const;
+
+const POST_CATEGORIES = [
+ 'travel',
+ 'food',
+ 'fitness',
+ 'tech',
+ 'music',
+ 'art',
+ 'pets',
+] as const;
+const MEDIA = ['photo', 'video', 'text'] as const;
+const SHARE_CHANNELS = [
+ 'message',
+ 'copy_link',
+ 'instagram',
+ 'external',
+] as const;
+const PUSH_CAMPAIGNS = [
+ 'daily-digest',
+ 'new-follower',
+ 'trending',
+ 'comment-reply',
+] as const;
+const OPEN_SOURCES = ['icon', 'push', 'deeplink', 'widget'] as const;
+const APP_VERSIONS = ['3.4.0', '3.4.1', '3.5.0', '3.5.2', '3.6.0'] as const;
+const PAYWALL_TRIGGERS = [
+ 'post_limit',
+ 'hd_upload',
+ 'analytics',
+ 'settings',
+] as const;
+/** Post ids are drawn from a large pool so `post_viewed.post_id` breaks down like a real feed. */
+const POST_POOL = 50_000;
+
+const IDENTIFIED_SHARE = 0.85;
+const SIGNUP_COMPLETED_CHANCE = 0.8;
+const NOTIFICATION_GRANTED_CHANCE = 0.55;
+const PUSH_OPEN_CHANCE = 0.18;
+const POST_CHANCE = 0.55;
+const LIKE_CHANCE = 0.5;
+const COMMENT_CHANCE = 0.12;
+const SHARE_CHANCE = 0.08;
+const CREATE_POST_CHANCE = 0.12;
+const SEARCH_CHANCE = 0.15;
+const PAYWALL_CHANCE = 0.06;
+const TRIAL_CHANCE = 0.35;
+const SUBSCRIBE_CHANCE = 0.5;
+const PREMIUM_MONTHLY_PRICE = 5;
+const PREMIUM_YEARLY_PRICE = 40;
+
+interface AppState {
+ onboarded?: boolean;
+ notifications?: boolean;
+ posts?: number;
+ trial?: boolean;
+ premium?: boolean;
+ appVersion?: string;
+}
+
+function stateOf(visitor: Visitor): AppState {
+ return visitor.state as AppState;
+}
+
+function platformOf(visitor: Visitor): string {
+ return visitor.device.os === 'iOS' ? 'ios' : 'android';
+}
+
+function onboarding(
+ rng: Rng,
+ journey: Journey,
+ visitor: Visitor,
+ state: AppState
+): boolean {
+ journey.view(...SCREENS.onboarding, { step: 1 });
+ journey.event('onboarding_step_completed', { step: 1, name: 'welcome' });
+ journey.view(...SCREENS.onboarding, { step: 2 });
+ journey.event('onboarding_step_completed', {
+ step: 2,
+ name: 'interests',
+ interests: rng.int(1, 5),
+ });
+ if (visitor.person && rng.chance(SIGNUP_COMPLETED_CHANCE)) {
+ const method = rng.one(['apple', 'google', 'email']);
+ journey.event('signup_started', { method });
+ journey.identify(visitor.person);
+ journey.event('signup_completed', { method, ...visitor.person.properties });
+ } else if (visitor.person) {
+ return false;
+ } else {
+ journey.event('signup_skipped');
+ }
+ const granted = rng.chance(NOTIFICATION_GRANTED_CHANCE);
+ journey.event('notification_permission', { granted });
+ state.notifications = granted;
+ state.onboarded = true;
+ return true;
+}
+
+function feed(rng: Rng, journey: Journey, state: AppState): void {
+ journey.view(...SCREENS.feed);
+ const scrolls = rng.int(1, 6);
+ for (let i = 0; i < scrolls; i++) {
+ if (!rng.chance(POST_CHANCE)) {
+ continue;
+ }
+ const postId = `p_${rng.int(1, POST_POOL)}`;
+ const category = rng.one(POST_CATEGORIES);
+ journey.view(...SCREENS.post, { post_id: postId });
+ journey.event('post_viewed', {
+ post_id: postId,
+ category,
+ media: rng.one(MEDIA),
+ author_id: `usr_${rng.hex(6)}`,
+ });
+ if (rng.chance(LIKE_CHANCE)) {
+ journey.event('post_liked', { post_id: postId, category });
+ }
+ if (rng.chance(COMMENT_CHANCE)) {
+ journey.event('comment_posted', {
+ post_id: postId,
+ length: rng.int(5, 200),
+ });
+ }
+ if (rng.chance(SHARE_CHANCE)) {
+ journey.event('post_shared', {
+ post_id: postId,
+ channel: rng.one(SHARE_CHANNELS),
+ });
+ }
+ }
+ if (rng.chance(SEARCH_CHANCE)) {
+ journey.view(...SCREENS.search);
+ journey.event('search', {
+ query: rng.one(POST_CATEGORIES),
+ results: rng.int(0, 50),
+ });
+ }
+ if (rng.chance(CREATE_POST_CHANCE)) {
+ journey.view(...SCREENS.create);
+ const media = rng.one(MEDIA);
+ journey.event('post_created', {
+ media,
+ category: rng.one(POST_CATEGORIES),
+ first_post: (state.posts ?? 0) === 0,
+ });
+ state.posts = (state.posts ?? 0) + 1;
+ }
+}
+
+function monetization(rng: Rng, journey: Journey, state: AppState): void {
+ if (state.premium || !rng.chance(PAYWALL_CHANCE)) {
+ return;
+ }
+ const trigger = rng.one(PAYWALL_TRIGGERS);
+ journey.view(...SCREENS.paywall, { trigger });
+ journey.event('paywall_viewed', { trigger });
+ if (!state.trial) {
+ if (!rng.chance(TRIAL_CHANCE)) {
+ return;
+ }
+ journey.event('trial_started', { plan: 'premium', days: 7, trigger });
+ state.trial = true;
+ return;
+ }
+ if (!rng.chance(SUBSCRIBE_CHANCE)) {
+ return;
+ }
+ const interval = rng.one(['monthly', 'yearly']);
+ const price =
+ interval === 'yearly' ? PREMIUM_YEARLY_PRICE : PREMIUM_MONTHLY_PRICE;
+ journey.event('subscription_started', {
+ plan: 'premium',
+ interval,
+ price,
+ trigger,
+ });
+ journey.revenue(price, { plan: 'premium', interval });
+ state.premium = true;
+}
+
+export const appArchetype: Archetype = {
+ id: 'app',
+ projectName: 'Acme App',
+ origin: '',
+ domain: null,
+ types: ['app'],
+ sdk: { name: 'react-native', version: '1.0.3' },
+ shape: {
+ hourly: [
+ 2, 1.2, 0.7, 0.4, 0.3, 0.5, 1.5, 3.5, 4, 3.5, 3, 3.5, 4.5, 4, 3.5, 3.5, 4,
+ 5, 6, 7, 7.5, 7, 5.5, 3.5,
+ ],
+ dayOfWeek: [1.2, 0.95, 0.9, 0.9, 0.95, 1.05, 1.25],
+ trendPerWeek: 0.03,
+ spikeChance: 0.03,
+ dipChance: 0.02,
+ spikeRange: [1.5, 3],
+ dipRange: [0.5, 0.8],
+ },
+ trafficShare: 0.1,
+ returningShare: 0.65,
+ retention: { sameDay: 0.45, day1: 0.45, day7: 0.25, day30: 0.15 },
+ stickiness(visitor) {
+ const state = stateOf(visitor);
+ let value = visitor.identified ? 1.5 : 0.7;
+ if (state.notifications) {
+ value += 1;
+ }
+ if (state.premium) {
+ value += 1.5;
+ }
+ if ((state.posts ?? 0) > 0) {
+ value += 1;
+ }
+ return value;
+ },
+ mobileShare: 1,
+ deviceClass: 'native',
+ referrers: [{ weight: 1, value: null }],
+ campaignChance: 0,
+ dwellMedianSeconds: 12,
+ utcOffsetHours: -5,
+ newPerson(rng) {
+ if (!rng.chance(IDENTIFIED_SHARE)) {
+ return null;
+ }
+ const person = newPerson(rng, { business: false });
+ person.properties = { plan: 'free', interests: String(rng.int(1, 5)) };
+ return person;
+ },
+ identity:
+ '~85% sign up during onboarding and are identified from then on; the rest stay anonymous devices.',
+ conversions: ['signup_completed', 'subscription_started', 'post_created'],
+ funnels: [
+ {
+ name: 'Onboarding',
+ steps: [
+ 'app_opened',
+ 'onboarding_step_completed',
+ 'signup_completed',
+ 'notification_permission',
+ 'post_created',
+ ],
+ breakdowns: ['platform', 'app_version', 'method', 'granted'],
+ },
+ {
+ name: 'Premium',
+ steps: ['paywall_viewed', 'trial_started', 'subscription_started'],
+ breakdowns: ['trigger', 'platform', 'interval'],
+ },
+ ],
+ journey(rng, journey, visitor) {
+ const state = stateOf(visitor);
+ state.appVersion ??= rng.one(APP_VERSIONS);
+ const platform = platformOf(visitor);
+ const common = { platform, app_version: state.appVersion };
+
+ if (state.notifications && rng.chance(PUSH_OPEN_CHANCE)) {
+ journey.event('push_opened', {
+ campaign: rng.one(PUSH_CAMPAIGNS),
+ ...common,
+ });
+ journey.event('app_opened', { source: 'push', ...common });
+ } else {
+ journey.event('app_opened', {
+ source: rng.one(OPEN_SOURCES.filter((source) => source !== 'push')),
+ ...common,
+ });
+ }
+ journey.view(...SCREENS.home);
+
+ if (!(state.onboarded || onboarding(rng, journey, visitor, state))) {
+ return;
+ }
+ feed(rng, journey, state);
+ if (rng.chance(0.2)) {
+ journey.view(...SCREENS.profile);
+ }
+ if (rng.chance(0.08)) {
+ journey.view(...SCREENS.settings);
+ }
+ if (visitor.identified) {
+ monetization(rng, journey, state);
+ if (visitor.person) {
+ visitor.person.properties.plan = state.premium
+ ? 'premium'
+ : state.trial
+ ? 'trial'
+ : 'free';
+ visitor.person.properties.platform = platform;
+ visitor.person.properties.notifications = String(
+ state.notifications ?? false
+ );
+ }
+ }
+ },
+};
diff --git a/packages/seed/src/archetypes/archetype.ts b/packages/seed/src/archetypes/archetype.ts
new file mode 100644
index 000000000..7fd626e77
--- /dev/null
+++ b/packages/seed/src/archetypes/archetype.ts
@@ -0,0 +1,169 @@
+import type { ReferrerMix } from '../data/referrers';
+import type { Person, Visitor } from '../model';
+import type { Rng } from '../rng';
+import type { TrafficShape } from '../traffic';
+
+export type ArchetypeId = 'website' | 'saas' | 'ecommerce' | 'app';
+
+export type ProjectType = 'website' | 'app' | 'backend';
+
+/**
+ * Share of a cohort that comes back after n days. Piecewise exponential
+ * between the points; `sameDay` is a second visit within the first day.
+ */
+export interface RetentionCurve {
+ sameDay: number;
+ day1: number;
+ day7: number;
+ day30: number;
+}
+
+export interface SeedFunnel {
+ name: string;
+ steps: readonly string[];
+ /** Properties worth breaking the funnel down by. */
+ breakdowns: readonly string[];
+}
+
+export interface Archetype {
+ id: ArchetypeId;
+ projectName: string;
+ /** The site the events come from; `''` for a native app (screens are named routes). */
+ origin: string;
+ domain: string | null;
+ types: ProjectType[];
+ sdk: { name: string; version: string };
+ shape: TrafficShape;
+ /** This project's share of the run's sessions per day. */
+ trafficShare: number;
+ /** Share of sessions that come from a visitor seen before. */
+ returningShare: number;
+ retention: RetentionCurve;
+ /** Relative likelihood that this visitor is the one who comes back; 1 is the baseline. */
+ stickiness(visitor: Visitor): number;
+ mobileShare: number;
+ deviceClass: 'web' | 'native';
+ referrers: ReferrerMix;
+ campaignChance: number;
+ dwellMedianSeconds: number;
+ /** Shifts the hourly curve so a European site peaks in European daytime. */
+ utcOffsetHours: number;
+ /** The person behind a visitor who will identify at some point; null for a never-identified one. */
+ newPerson(rng: Rng): Person | null;
+ /** One line on who gets identified and when, for `.seed.json`. */
+ identity: string;
+ /** Events the dashboard should treat as conversions (`EventMeta.conversion`). */
+ conversions: readonly string[];
+ /** The funnels this project's data is built to show, for `.seed.json` and whoever builds reports. */
+ funnels: readonly SeedFunnel[];
+ journey(rng: Rng, journey: Journey, visitor: Visitor): void;
+}
+
+const CUSTOM_EVENT_GAP_MEDIAN_SECONDS = 3;
+const CUSTOM_EVENT_GAP_SIGMA = 0.7;
+const REVENUE_GAP_MEDIAN_SECONDS = 2;
+const REVENUE_GAP_SIGMA = 0.5;
+const DWELL_SIGMA = 0.8;
+const MS_PER_SECOND = 1000;
+
+export interface JourneyEvent {
+ name: string;
+ path: string;
+ title: string;
+ at: Date;
+ properties: Record<string, unknown>;
+ revenue?: number;
+ /** Who the event belongs to once identified; null while anonymous. */
+ person: Person | null;
+}
+
+/** Appends events while advancing a clock, so timestamps read like a person browsing. */
+export class Journey {
+ readonly events: JourneyEvent[] = [];
+ private cursor: Date;
+ private person: Person | null;
+
+ constructor(
+ private readonly rng: Rng,
+ start: Date,
+ private readonly dwellMedianSeconds: number,
+ person: Person | null
+ ) {
+ this.cursor = start;
+ this.person = person;
+ }
+
+ get identified(): Person | null {
+ return this.person;
+ }
+
+ /** From here on events carry this person; what `op.identify()` does in the SDK. */
+ identify(person: Person): void {
+ this.person = person;
+ }
+
+ view(
+ path: string,
+ title: string,
+ properties: Record<string, unknown> = {}
+ ): void {
+ this.events.push({
+ name: 'screen_view',
+ path,
+ title,
+ at: this.cursor,
+ properties,
+ person: this.person,
+ });
+ this.advance(
+ this.rng.logNormalSeconds(this.dwellMedianSeconds, DWELL_SIGMA)
+ );
+ }
+
+ event(name: string, properties: Record<string, unknown> = {}): void {
+ this.advance(
+ this.rng.logNormalSeconds(
+ CUSTOM_EVENT_GAP_MEDIAN_SECONDS,
+ CUSTOM_EVENT_GAP_SIGMA
+ )
+ );
+ this.events.push({
+ name,
+ ...this.currentScreen(),
+ at: this.cursor,
+ properties,
+ person: this.person,
+ });
+ }
+
+ /** Amount in whole currency units, integer: `events.revenue` is UInt64. */
+ revenue(amount: number, properties: Record<string, unknown> = {}): void {
+ this.advance(
+ this.rng.logNormalSeconds(REVENUE_GAP_MEDIAN_SECONDS, REVENUE_GAP_SIGMA)
+ );
+ this.events.push({
+ name: 'revenue',
+ ...this.currentScreen(),
+ at: this.cursor,
+ properties,
+ revenue: amount,
+ person: this.person,
+ });
+ }
+
+ private advance(seconds: number): void {
+ this.cursor = new Date(
+ this.cursor.getTime() + Math.round(seconds * MS_PER_SECOND)
+ );
+ }
+
+ private currentScreen(): { path: string; title: string } {
+ for (let i = this.events.length - 1; i >= 0; i--) {
+ const event = this.events[i];
+ if (event?.name === 'screen_view') {
+ return { path: event.path, title: event.title };
+ }
+ }
+ return { path: '', title: '' };
+ }
+}
diff --git a/packages/seed/src/archetypes/ecommerce.ts b/packages/seed/src/archetypes/ecommerce.ts
new file mode 100644
index 000000000..614ab5161
--- /dev/null
+++ b/packages/seed/src/archetypes/ecommerce.ts
@@ -0,0 +1,391 @@
+// Acme Shop: mostly anonymous shoppers; an account (identify) is created at
+// the first purchase for a share of them. Funnel: product_viewed →
+// add_to_cart → checkout_started → shipping_info_added → payment_info_added
+// → purchase. Cohort-worthy: has purchased, has an account, used a coupon,
+// wishlisted.
+
+import { newPerson } from '../data/people';
+import { REFERRERS } from '../data/referrers';
+import type { Visitor } from '../model';
+import type { Rng } from '../rng';
+import type { Archetype, Journey } from './archetype';
+
+const CATEGORIES = [
+ 'sneakers',
+ 'boots',
+ 'sandals',
+ 'running',
+ 'kids',
+ 'accessories',
+] as const;
+const BRANDS = ['Acme', 'Northwind', 'Vela', 'Kestrel', 'Orbit'] as const;
+const SIZES = [
+ '36',
+ '37',
+ '38',
+ '39',
+ '40',
+ '41',
+ '42',
+ '43',
+ '44',
+ '45',
+ '46',
+] as const;
+const COLORS = ['black', 'white', 'navy', 'sand', 'olive', 'red'] as const;
+const PAYMENTS = [
+ 'card',
+ 'paypal',
+ 'klarna',
+ 'apple_pay',
+ 'google_pay',
+] as const;
+const SHIPPING = ['standard', 'express', 'pickup'] as const;
+const COUPONS = ['WELCOME10', 'SUMMER20', 'FREESHIP', 'VIP15'] as const;
+const LOYALTY_TIERS = ['bronze', 'silver', 'gold'] as const;
+/** Ten of each category, so `product_viewed.sku` has real cardinality. */
+const PRODUCTS_PER_CATEGORY = 10;
+
+interface Product {
+ sku: string;
+ name: string;
+ category: (typeof CATEGORIES)[number];
+ brand: (typeof BRANDS)[number];
+ price: number;
+}
+
+function catalogue(): Product[] {
+ const products: Product[] = [];
+ for (const [categoryIndex, category] of CATEGORIES.entries()) {
+ for (let n = 1; n <= PRODUCTS_PER_CATEGORY; n++) {
+ products.push({
+ sku: `${category.slice(0, 3).toUpperCase()}-${String(n).padStart(3, '0')}`,
+ name: `${category[0]?.toUpperCase()}${category.slice(1)} ${n}`,
+ category,
+ brand: BRANDS[(categoryIndex + n) % BRANDS.length] ?? 'Acme',
+ price: 39 + ((categoryIndex * 37 + n * 23) % 180),
+ });
+ }
+ }
+ return products;
+}
+
+const PRODUCTS = catalogue();
+
+const ACCOUNT_SHARE = 0.4;
+const BOUNCE_CHANCE = 0.38;
+const SEARCH_CHANCE = 0.2;
+const FILTER_CHANCE = 0.25;
+const WISHLIST_CHANCE = 0.06;
+const ADD_TO_CART_CHANCE = 0.22;
+const REMOVE_FROM_CART_CHANCE = 0.15;
+const CHECKOUT_CHANCE = 0.45;
+const SHIPPING_CHANCE = 0.85;
+const PAYMENT_CHANCE = 0.8;
+const PURCHASE_CHANCE = 0.85;
+const COUPON_CHANCE = 0.2;
+const REVIEW_CHANCE = 0.08;
+const SHIPPING_PRICE = 9;
+
+interface ShopState {
+ cart?: Product[];
+ purchases?: number;
+ couponUsed?: boolean;
+ wishlisted?: boolean;
+}
+
+function stateOf(visitor: Visitor): ShopState {
+ return visitor.state as ShopState;
+}
+
+function viewProduct(rng: Rng, journey: Journey, product: Product): void {
+ journey.view(
+ `/product/${product.sku.toLowerCase()}`,
+ `${product.name} — Acme Shop`,
+ { sku: product.sku, category: product.category }
+ );
+ journey.event('product_viewed', {
+ sku: product.sku,
+ name: product.name,
+ category: product.category,
+ brand: product.brand,
+ price: product.price,
+ in_stock: rng.chance(0.92),
+ rating: rng.int(3, 5),
+ });
+ if (rng.chance(0.3)) {
+ journey.event('product_image_zoomed', { sku: product.sku });
+ }
+}
+
+function cartValue(cart: readonly Product[]): number {
+ let total = 0;
+ for (const product of cart) {
+ total += product.price;
+ }
+ return total;
+}
+
+function checkout(
+ rng: Rng,
+ journey: Journey,
+ visitor: Visitor,
+ state: ShopState,
+ cart: Product[]
+): void {
+ journey.view('/checkout', 'Checkout');
+ const value = cartValue(cart);
+ journey.event('checkout_started', {
+ items: cart.length,
+ value,
+ currency: 'USD',
+ });
+ if (!rng.chance(SHIPPING_CHANCE)) {
+ return;
+ }
+ const shipping = rng.one(SHIPPING);
+ journey.event('shipping_info_added', {
+ method: shipping,
+ country: visitor.geo.country,
+ });
+ if (!rng.chance(PAYMENT_CHANCE)) {
+ return;
+ }
+ const payment = rng.one(PAYMENTS);
+ journey.event('payment_info_added', { method: payment });
+ let coupon: string | null = null;
+ if (rng.chance(COUPON_CHANCE)) {
+ coupon = rng.one(COUPONS);
+ journey.event('coupon_applied', {
+ code: coupon,
+ discount_percent: coupon === 'FREESHIP' ? 0 : Number(coupon.slice(-2)),
+ });
+ state.couponUsed = true;
+ }
+ if (!rng.chance(PURCHASE_CHANCE)) {
+ return;
+ }
+ // The first order creates the account for those who make one.
+ if (visitor.person && !visitor.identified) {
+ journey.identify(visitor.person);
+ journey.event('account_created', {
+ method: rng.one(['email', 'google', 'apple']),
+ at: 'checkout',
+ });
+ }
+ const orderId = `ord_${rng.hex(8)}`;
+ const total =
+ value +
+ (shipping === 'pickup' || coupon === 'FREESHIP' ? 0 : SHIPPING_PRICE);
+ journey.view(`/order/${orderId}`, 'Thank you');
+ journey.event('purchase', {
+ order_id: orderId,
+ items: cart.length,
+ value: total,
+ currency: 'USD',
+ payment,
+ shipping,
+ coupon,
+ first_purchase: (state.purchases ?? 0) === 0,
+ categories: [...new Set(cart.map((product) => product.category))].join(','),
+ });
+ journey.revenue(total, { order_id: orderId, currency: 'USD' });
+ state.purchases = (state.purchases ?? 0) + 1;
+ state.cart = [];
+ if (rng.chance(REVIEW_CHANCE)) {
+ const reviewed = rng.one(cart);
+ journey.event('review_submitted', {
+ sku: reviewed.sku,
+ rating: rng.int(2, 5),
+ });
+ }
+}
+
+export const ecommerceArchetype: Archetype = {
+ id: 'ecommerce',
+ projectName: 'Acme Shop',
+ origin: 'https://shop.acme.test',
+ domain: 'https://shop.acme.test',
+ types: ['website'],
+ sdk: { name: 'web', version: '1.0.5' },
+ shape: {
+ hourly: [
+ 1.5, 0.8, 0.5, 0.3, 0.3, 0.4, 0.8, 1.5, 2.5, 3.5, 4, 4.5, 4.5, 4, 4, 4,
+ 4.5, 5, 5.5, 6.5, 7, 6, 4.5, 3,
+ ],
+ dayOfWeek: [1.15, 0.9, 0.9, 0.95, 1.0, 1.1, 1.25],
+ trendPerWeek: 0.01,
+ spikeChance: 0.05,
+ dipChance: 0.02,
+ spikeRange: [2, 4],
+ dipRange: [0.4, 0.7],
+ },
+ trafficShare: 0.25,
+ returningShare: 0.4,
+ retention: { sameDay: 0.18, day1: 0.2, day7: 0.1, day30: 0.06 },
+ stickiness(visitor) {
+ const state = stateOf(visitor);
+ let value = 1;
+ if ((state.purchases ?? 0) > 0) {
+ value += 1.5;
+ }
+ if (visitor.identified) {
+ value += 1;
+ }
+ if (state.wishlisted) {
+ value += 0.5;
+ }
+ return value;
+ },
+ mobileShare: 0.65,
+ deviceClass: 'web',
+ referrers: [
+ { weight: 30, value: null },
+ { weight: 28, value: REFERRERS.google },
+ { weight: 12, value: REFERRERS.instagram },
+ { weight: 10, value: REFERRERS.facebook },
+ { weight: 5, value: REFERRERS.youTube },
+ { weight: 4, value: REFERRERS.bing },
+ { weight: 3, value: REFERRERS.reddit },
+ { weight: 3, value: REFERRERS.gmail },
+ { weight: 2, value: REFERRERS.twitter },
+ ],
+ campaignChance: 0.3,
+ dwellMedianSeconds: 20,
+ utcOffsetHours: 0,
+ newPerson(rng) {
+ if (!rng.chance(ACCOUNT_SHARE)) {
+ return null;
+ }
+ const person = newPerson(rng, { business: false });
+ person.properties = {
+ loyalty_tier: rng.one(LOYALTY_TIERS),
+ newsletter: String(rng.chance(0.5)),
+ };
+ return person;
+ },
+ identity:
+ 'Anonymous shoppers; an account is created at the first purchase for ~40% of buyers, who are identified from then on.',
+ conversions: ['purchase', 'account_created'],
+ funnels: [
+ {
+ name: 'Checkout',
+ steps: [
+ 'product_viewed',
+ 'add_to_cart',
+ 'checkout_started',
+ 'shipping_info_added',
+ 'payment_info_added',
+ 'purchase',
+ ],
+ breakdowns: [
+ 'category',
+ 'brand',
+ 'payment',
+ 'shipping',
+ 'device',
+ 'country',
+ 'coupon',
+ ],
+ },
+ {
+ name: 'Search to cart',
+ steps: ['search', 'product_viewed', 'add_to_cart'],
+ breakdowns: ['query', 'category'],
+ },
+ ],
+ journey(rng, journey, visitor) {
+ const state = stateOf(visitor);
+ state.cart ??= [];
+
+ const entry = rng.pick([
+ { weight: 45, value: 'home' },
+ { weight: 30, value: 'product' },
+ { weight: 25, value: 'category' },
+ ] as const);
+ if (entry === 'home') {
+ journey.view('/', 'Acme Shop');
+ } else if (entry === 'category') {
+ const category = rng.one(CATEGORIES);
+ journey.view(`/category/${category}`, `${category} — Acme Shop`, {
+ category,
+ });
+ } else {
+ viewProduct(rng, journey, rng.one(PRODUCTS));
+ }
+ if (rng.chance(BOUNCE_CHANCE)) {
+ return;
+ }
+
+ if (rng.chance(SEARCH_CHANCE)) {
+ const query = rng.one([
+ 'white sneakers',
+ 'boots',
+ 'running shoes',
+ 'sale',
+ 'kids',
+ 'sandals',
+ ]);
+ journey.event('search', { query, results: rng.int(0, 40) });
+ }
+ if (rng.chance(FILTER_CHANCE)) {
+ journey.event('filter_applied', {
+ facet: rng.one(['size', 'color', 'brand', 'price']),
+ value: rng.one([...SIZES, ...COLORS, ...BRANDS]),
+ });
+ }
+
+ const browses = rng.int(1, 4);
+ for (let i = 0; i < browses; i++) {
+ const product = rng.one(PRODUCTS);
+ if (rng.chance(0.4)) {
+ journey.view(
+ `/category/${product.category}`,
+ `${product.category} — Acme Shop`,
+ { category: product.category }
+ );
+ }
+ viewProduct(rng, journey, product);
+ if (rng.chance(WISHLIST_CHANCE)) {
+ journey.event('wishlist_added', {
+ sku: product.sku,
+ category: product.category,
+ });
+ state.wishlisted = true;
+ }
+ if (rng.chance(ADD_TO_CART_CHANCE)) {
+ journey.event('add_to_cart', {
+ sku: product.sku,
+ name: product.name,
+ category: product.category,
+ brand: product.brand,
+ price: product.price,
+ quantity: 1,
+ size: rng.one(SIZES),
+ color: rng.one(COLORS),
+ });
+ state.cart.push(product);
+ }
+ }
+ if (state.cart.length === 0) {
+ return;
+ }
+
+ journey.view('/cart', 'Your cart', {
+ items: state.cart.length,
+ value: cartValue(state.cart),
+ });
+ if (rng.chance(REMOVE_FROM_CART_CHANCE)) {
+ const removed = state.cart.pop();
+ if (removed) {
+ journey.event('remove_from_cart', {
+ sku: removed.sku,
+ category: removed.category,
+ });
+ }
+ }
+ if (state.cart.length > 0 && rng.chance(CHECKOUT_CHANCE)) {
+ checkout(rng, journey, visitor, state, state.cart);
+ }
+ },
+};
diff --git a/packages/seed/src/archetypes/index.ts b/packages/seed/src/archetypes/index.ts
new file mode 100644
index 000000000..c45604056
--- /dev/null
+++ b/packages/seed/src/archetypes/index.ts
@@ -0,0 +1,26 @@
+import { appArchetype } from './app';
+import type { Archetype } from './archetype';
+import { ecommerceArchetype } from './ecommerce';
+import { saasArchetype } from './saas';
+import { websiteArchetype } from './website';
+
+export const ARCHETYPES: readonly Archetype[] = [
+ websiteArchetype,
+ saasArchetype,
+ ecommerceArchetype,
+ appArchetype,
+];
+
+export const ARCHETYPE_IDS = ARCHETYPES.map((archetype) => archetype.id);
+
+export function archetypeById(id: string): Archetype {
+ const archetype = ARCHETYPES.find((candidate) => candidate.id === id);
+ if (!archetype) {
+ throw new Error(
+ `Unknown archetype "${id}". Known: ${ARCHETYPE_IDS.join(', ')}`
+ );
+ }
+ return archetype;
+}
+
+export type { Archetype, ArchetypeId } from './archetype';
diff --git a/packages/seed/src/archetypes/saas.ts b/packages/seed/src/archetypes/saas.ts
new file mode 100644
index 000000000..e4e5d698c
--- /dev/null
+++ b/packages/seed/src/archetypes/saas.ts
@@ -0,0 +1,391 @@
+// Acme SaaS: a logged-in product. Everyone identifies on the first session,
+// so retention is real. Funnels: activation (signup → project → SDK → first
+// event) spread over visits, and monetization (pricing → checkout →
+// subscription). Cohort-worthy behaviours: invited a teammate, exported,
+// connected an integration, paid plan.
+
+import { newPerson } from '../data/people';
+import { REFERRERS } from '../data/referrers';
+import type { Person, Visitor } from '../model';
+import type { Rng, Weighted } from '../rng';
+import type { Archetype, Journey } from './archetype';
+
+interface Plan {
+ name: string;
+ price: number;
+}
+
+const PLANS: readonly Weighted<Plan>[] = [
+ { weight: 70, value: { name: 'free', price: 0 } },
+ { weight: 18, value: { name: 'pro', price: 29 } },
+ { weight: 9, value: { name: 'team', price: 99 } },
+ { weight: 3, value: { name: 'enterprise', price: 299 } },
+];
+const PAID_PLANS = PLANS.filter((plan) => plan.value.price > 0);
+const ROLES = ['founder', 'engineer', 'product', 'marketing', 'data'] as const;
+const COMPANY_SIZES = [
+ '1',
+ '2-10',
+ '11-50',
+ '51-200',
+ '201-1000',
+ '1000+',
+] as const;
+const SIGNUP_SOURCES = [
+ 'organic',
+ 'referral',
+ 'ads',
+ 'producthunt',
+ 'docs',
+] as const;
+const ONBOARDING_STEPS = [
+ 'organization',
+ 'project',
+ 'install',
+ 'verify',
+] as const;
+const PROJECT_TYPES = ['website', 'app', 'backend'] as const;
+const SDKS = [
+ 'web',
+ 'react',
+ 'next',
+ 'react-native',
+ 'node',
+ 'python',
+] as const;
+const FEATURES = [
+ 'funnels',
+ 'retention',
+ 'cohorts',
+ 'pages',
+ 'realtime',
+ 'profiles',
+ 'insights',
+ 'ai-assistant',
+] as const;
+const INTEGRATIONS = ['slack', 'discord', 'webhook', 'gsc', 'stripe'] as const;
+const REPORT_TYPES = [
+ 'overview',
+ 'funnel',
+ 'retention',
+ 'pages',
+ 'events',
+ 'cohort',
+] as const;
+const CANCEL_REASONS = [
+ 'too_expensive',
+ 'missing_feature',
+ 'not_using',
+ 'switched',
+] as const;
+const PROJECT_IDS = [
+ 'web',
+ 'ios',
+ 'android',
+ 'landing',
+ 'internal',
+ 'staging',
+] as const;
+
+const SIGNUP_COMPLETED_CHANCE = 0.85;
+const ONBOARDING_STEP_CHANCE = 0.75;
+const SDK_INSTALLED_CHANCE = 0.55;
+const FIRST_EVENT_CHANCE = 0.8;
+const PRICING_VIEWED_CHANCE = 0.06;
+const CHECKOUT_CHANCE = 0.45;
+const SUBSCRIBED_CHANCE = 0.6;
+const CHURN_CHANCE = 0.004;
+const INVITE_CHANCE = 0.05;
+const EXPORT_CHANCE = 0.05;
+const INTEGRATION_CHANCE = 0.03;
+const REPORT_CREATED_CHANCE = 0.15;
+const FEATURE_CHANCE = 0.5;
+const SETTINGS_CHANCE = 0.1;
+
+interface SaasState {
+ plan?: Plan;
+ onboardingStep?: number;
+ projectCreated?: boolean;
+ sdkInstalled?: boolean;
+ firstEvent?: boolean;
+ invited?: boolean;
+ exported?: boolean;
+ integrated?: boolean;
+ churned?: boolean;
+}
+
+function stateOf(visitor: Visitor): SaasState {
+ return visitor.state as SaasState;
+}
+
+function personProps(person: Person): Record<string, string> {
+ return person.properties;
+}
+
+/** Activation is spread over visits: each session moves the user at most a step or two. */
+function activation(rng: Rng, journey: Journey, state: SaasState): void {
+ const step = state.onboardingStep ?? 0;
+ if (step < ONBOARDING_STEPS.length) {
+ if (!rng.chance(ONBOARDING_STEP_CHANCE)) {
+ return;
+ }
+ journey.view('/onboarding', 'Onboarding', { step: step + 1 });
+ journey.event('onboarding_step_completed', {
+ step: step + 1,
+ name: ONBOARDING_STEPS[step],
+ });
+ state.onboardingStep = step + 1;
+ if (ONBOARDING_STEPS[step] === 'project') {
+ journey.event('project_created', {
+ type: rng.one(PROJECT_TYPES),
+ timezone: rng.one([
+ 'UTC',
+ 'Europe/Stockholm',
+ 'America/New_York',
+ 'Asia/Tokyo',
+ ]),
+ });
+ state.projectCreated = true;
+ }
+ return;
+ }
+ if (
+ state.projectCreated &&
+ !state.sdkInstalled &&
+ rng.chance(SDK_INSTALLED_CHANCE)
+ ) {
+ journey.view('/projects/web/connect', 'Connect your app');
+ journey.event('sdk_installed', {
+ sdk: rng.one(SDKS),
+ framework: rng.one(['next', 'vite', 'remix', 'expo', 'express', 'none']),
+ });
+ state.sdkInstalled = true;
+ return;
+ }
+ if (
+ state.sdkInstalled &&
+ !state.firstEvent &&
+ rng.chance(FIRST_EVENT_CHANCE)
+ ) {
+ journey.event('first_event_received', {
+ latency_seconds: rng.int(5, 900),
+ event_name: rng.one(['screen_view', 'signup', 'purchase']),
+ });
+ state.firstEvent = true;
+ }
+}
+
+function monetization(
+ rng: Rng,
+ journey: Journey,
+ state: SaasState,
+ person: Person
+): void {
+ const plan = state.plan ?? { name: 'free', price: 0 };
+ if (plan.price === 0 && rng.chance(PRICING_VIEWED_CHANCE)) {
+ journey.view('/settings/billing', 'Billing');
+ journey.event('pricing_viewed', { current_plan: plan.name });
+ if (!rng.chance(CHECKOUT_CHANCE)) {
+ return;
+ }
+ const chosen = rng.pick(PAID_PLANS);
+ const interval = rng.one(['monthly', 'yearly']);
+ journey.event('checkout_started', { plan: chosen.name, interval });
+ if (!rng.chance(SUBSCRIBED_CHANCE)) {
+ return;
+ }
+ const seats = chosen.name === 'pro' ? 1 : rng.int(2, 25);
+ journey.event('subscription_started', {
+ plan: chosen.name,
+ interval,
+ seats,
+ mrr: chosen.price,
+ previous_plan: plan.name,
+ });
+ journey.revenue(interval === 'yearly' ? chosen.price * 10 : chosen.price, {
+ plan: chosen.name,
+ interval,
+ seats,
+ });
+ state.plan = chosen;
+ person.properties.plan = chosen.name;
+ return;
+ }
+ if (plan.price > 0 && rng.chance(CHURN_CHANCE)) {
+ journey.view('/settings/billing', 'Billing');
+ journey.event('subscription_cancelled', {
+ plan: plan.name,
+ reason: rng.one(CANCEL_REASONS),
+ months_active: rng.int(1, 18),
+ });
+ state.plan = { name: 'free', price: 0 };
+ state.churned = true;
+ person.properties.plan = 'free';
+ }
+}
+
+function usage(rng: Rng, journey: Journey, state: SaasState): void {
+ const visits = rng.int(1, 3);
+ for (let i = 0; i < visits; i++) {
+ const projectId = rng.one(PROJECT_IDS);
+ journey.view(`/projects/${projectId}`, `Project ${projectId}`, {
+ project: projectId,
+ });
+ if (rng.chance(FEATURE_CHANCE)) {
+ const feature = rng.one(FEATURES);
+ journey.view(`/projects/${projectId}/${feature}`, feature, {
+ project: projectId,
+ });
+ journey.event('feature_used', { feature, project: projectId });
+ }
+ if (rng.chance(0.6)) {
+ journey.event('report_viewed', {
+ report_type: rng.one(REPORT_TYPES),
+ range: rng.one(['7d', '30d', '90d', 'custom']),
+ project: projectId,
+ });
+ }
+ if (rng.chance(REPORT_CREATED_CHANCE)) {
+ journey.event('report_created', {
+ chart: rng.one(['line', 'bar', 'funnel', 'retention', 'pie']),
+ breakdowns: rng.int(0, 2),
+ project: projectId,
+ });
+ }
+ if (rng.chance(EXPORT_CHANCE)) {
+ journey.event('export_completed', {
+ format: rng.one(['csv', 'json']),
+ rows: rng.int(100, 200_000),
+ });
+ state.exported = true;
+ }
+ }
+ if (rng.chance(INVITE_CHANCE)) {
+ journey.view('/settings/members', 'Members');
+ journey.event('invite_sent', { role: rng.one(['admin', 'member']) });
+ state.invited = true;
+ } else if (rng.chance(INTEGRATION_CHANCE)) {
+ journey.view('/settings/integrations', 'Integrations');
+ journey.event('integration_connected', { provider: rng.one(INTEGRATIONS) });
+ state.integrated = true;
+ } else if (rng.chance(SETTINGS_CHANCE)) {
+ journey.view('/settings', 'Settings');
+ }
+}
+
+export const saasArchetype: Archetype = {
+ id: 'saas',
+ projectName: 'Acme SaaS',
+ origin: 'https://app.acme.test',
+ domain: 'https://app.acme.test',
+ types: ['website', 'backend'],
+ sdk: { name: 'web', version: '1.0.5' },
+ shape: {
+ hourly: [
+ 0.5, 0.3, 0.2, 0.2, 0.3, 0.6, 1.5, 3.5, 6, 7, 7, 6, 5.5, 6.5, 7, 6.5, 6,
+ 5, 3.5, 2.5, 2, 1.5, 1, 0.8,
+ ],
+ dayOfWeek: [0.2, 1.2, 1.25, 1.25, 1.2, 1.0, 0.2],
+ trendPerWeek: 0.015,
+ spikeChance: 0.01,
+ dipChance: 0.03,
+ spikeRange: [1.5, 2.5],
+ dipRange: [0.3, 0.6],
+ },
+ trafficShare: 0.2,
+ returningShare: 0.75,
+ retention: { sameDay: 0.35, day1: 0.55, day7: 0.4, day30: 0.3 },
+ stickiness(visitor) {
+ const state = stateOf(visitor);
+ if (state.churned) {
+ return 0.3;
+ }
+ let value = (state.plan?.price ?? 0) > 0 ? 3 : 1;
+ if (state.invited) {
+ value += 1;
+ }
+ if (state.firstEvent) {
+ value += 0.5;
+ }
+ return value;
+ },
+ mobileShare: 0.12,
+ deviceClass: 'web',
+ referrers: [
+ { weight: 70, value: null },
+ { weight: 15, value: REFERRERS.google },
+ { weight: 8, value: REFERRERS.gmail },
+ { weight: 4, value: REFERRERS.gitHub },
+ { weight: 3, value: REFERRERS.linkedIn },
+ ],
+ campaignChance: 0.03,
+ dwellMedianSeconds: 45,
+ utcOffsetHours: -2,
+ newPerson(rng) {
+ const person = newPerson(rng, { business: true });
+ person.properties = {
+ plan: 'free',
+ role: rng.one(ROLES),
+ company_size: rng.one(COMPANY_SIZES),
+ signup_source: rng.one(SIGNUP_SOURCES),
+ };
+ return person;
+ },
+ identity:
+ 'Everyone signs up or logs in on their first session and is identified from then on; retention is real.',
+ conversions: ['signup_completed', 'sdk_installed', 'subscription_started'],
+ funnels: [
+ {
+ name: 'Activation',
+ steps: [
+ 'signup_completed',
+ 'project_created',
+ 'sdk_installed',
+ 'first_event_received',
+ ],
+ breakdowns: ['signup_source', 'company_size', 'role', 'sdk'],
+ },
+ {
+ name: 'Upgrade',
+ steps: ['pricing_viewed', 'checkout_started', 'subscription_started'],
+ breakdowns: ['plan', 'interval', 'company_size'],
+ },
+ ],
+ journey(rng, journey, visitor) {
+ const state = stateOf(visitor);
+ const person = visitor.person;
+ if (!person) {
+ return;
+ }
+
+ if (!visitor.identified) {
+ journey.view('/signup', 'Create your account');
+ const method = rng.one(['email', 'github', 'google']);
+ journey.event('signup_started', { method });
+ if (!rng.chance(SIGNUP_COMPLETED_CHANCE)) {
+ return;
+ }
+ journey.identify(person);
+ journey.event('signup_completed', { method, ...personProps(person) });
+ state.plan = rng.pick(PLANS.filter((plan) => plan.value.price === 0));
+ } else if (visitor.sessions === 0 || rng.chance(0.15)) {
+ // A known user on a new device, or a session that expired.
+ journey.view('/login', 'Log in');
+ journey.event('login', {
+ method: rng.one(['email', 'github', 'google']),
+ });
+ }
+
+ journey.view('/dashboard', 'Dashboard', {
+ plan: state.plan?.name ?? 'free',
+ });
+ activation(rng, journey, state);
+ if (
+ state.onboardingStep !== undefined &&
+ state.onboardingStep >= ONBOARDING_STEPS.length
+ ) {
+ usage(rng, journey, state);
+ monetization(rng, journey, state, person);
+ }
+ },
+};
diff --git a/packages/seed/src/archetypes/website.ts b/packages/seed/src/archetypes/website.ts
new file mode 100644
index 000000000..16114103c
--- /dev/null
+++ b/packages/seed/src/archetypes/website.ts
@@ -0,0 +1,244 @@
+// Acme Web: a marketing + docs site. Anonymous traffic; a share of visitors
+// eventually signs up, which identifies them. Funnels: landing → pricing →
+// signup, and docs → demo request.
+
+import { newPerson } from '../data/people';
+import { REFERRERS } from '../data/referrers';
+import type { Archetype } from './archetype';
+
+const BLOG_TOPICS = [
+ 'product-analytics',
+ 'privacy',
+ 'self-hosting',
+ 'retention',
+ 'funnels',
+ 'attribution',
+ 'sdk',
+ 'clickhouse',
+ 'pricing',
+ 'changelog',
+];
+const BLOG_POSTS_PER_TOPIC = 6;
+const DOCS_PAGES = [
+ ['get-started', 'Get started'],
+ ['sdks/web', 'Web SDK'],
+ ['sdks/react', 'React SDK'],
+ ['sdks/react-native', 'React Native SDK'],
+ ['sdks/node', 'Node SDK'],
+ ['api/track', 'Track API'],
+ ['api/export', 'Export API'],
+ ['guides/funnels', 'Funnels'],
+ ['guides/retention', 'Retention'],
+ ['guides/cohorts', 'Cohorts'],
+ ['self-hosting', 'Self-hosting'],
+ ['self-hosting/clickhouse', 'ClickHouse tuning'],
+] as const;
+const COMPANY_SIZES = [
+ '1',
+ '2-10',
+ '11-50',
+ '51-200',
+ '201-1000',
+ '1000+',
+] as const;
+const ROLES = ['founder', 'engineer', 'product', 'marketing', 'data'] as const;
+const PLANS = ['free', 'pro', 'team'] as const;
+
+const WILL_SIGN_UP_SHARE = 0.18;
+const BOUNCE_CHANCE = 0.42;
+const CTA_CHANCE = 0.25;
+const PRICING_CHANCE = 0.35;
+const DOCS_CHANCE = 0.3;
+const DOCS_SEARCH_CHANCE = 0.3;
+const NEWSLETTER_CHANCE = 0.04;
+const DEMO_CHANCE = 0.02;
+const CONTACT_CHANCE = 0.02;
+const SIGNUP_STARTED_CHANCE = 0.4;
+const SIGNUP_COMPLETED_CHANCE = 0.7;
+const IDENTIFIED_STICKINESS = 3;
+
+export const websiteArchetype: Archetype = {
+ id: 'website',
+ projectName: 'Acme Web',
+ origin: 'https://www.acme.test',
+ domain: 'https://www.acme.test',
+ types: ['website'],
+ sdk: { name: 'web', version: '1.0.5' },
+ shape: {
+ hourly: [
+ 1, 0.6, 0.4, 0.3, 0.3, 0.5, 1, 2.5, 4, 5, 5.5, 5.5, 5, 5.5, 5.5, 5, 4.5,
+ 4, 3.5, 3.5, 3, 2.5, 2, 1.5,
+ ],
+ dayOfWeek: [0.55, 1.1, 1.15, 1.15, 1.1, 0.95, 0.55],
+ trendPerWeek: 0.02,
+ spikeChance: 0.04,
+ dipChance: 0.02,
+ spikeRange: [2, 6],
+ dipRange: [0.2, 0.5],
+ },
+ trafficShare: 0.45,
+ returningShare: 0.3,
+ // Anonymous visitors are new devices tomorrow; this is how often the same person comes back.
+ retention: { sameDay: 0.1, day1: 0.14, day7: 0.06, day30: 0.03 },
+ stickiness: (visitor) => (visitor.identified ? IDENTIFIED_STICKINESS : 1),
+ mobileShare: 0.45,
+ deviceClass: 'web',
+ referrers: [
+ { weight: 35, value: null },
+ { weight: 25, value: REFERRERS.google },
+ { weight: 4, value: REFERRERS.bing },
+ { weight: 3, value: REFERRERS.duckDuckGo },
+ { weight: 6, value: REFERRERS.twitter },
+ { weight: 5, value: REFERRERS.hackerNews },
+ { weight: 4, value: REFERRERS.reddit },
+ { weight: 4, value: REFERRERS.linkedIn },
+ { weight: 4, value: REFERRERS.gitHub },
+ { weight: 3, value: REFERRERS.chatGpt },
+ { weight: 2, value: REFERRERS.claude },
+ { weight: 2, value: REFERRERS.medium },
+ { weight: 2, value: REFERRERS.productHunt },
+ { weight: 1, value: REFERRERS.youTube },
+ ],
+ campaignChance: 0.12,
+ dwellMedianSeconds: 25,
+ utcOffsetHours: 1,
+ newPerson(rng) {
+ if (!rng.chance(WILL_SIGN_UP_SHARE)) {
+ return null;
+ }
+ const person = newPerson(rng, { business: rng.chance(0.6) });
+ person.properties = {
+ company_size: rng.one(COMPANY_SIZES),
+ role: rng.one(ROLES),
+ plan: rng.one(PLANS),
+ };
+ return person;
+ },
+ identity:
+ 'Anonymous; a device is new every UTC day. Visitors who sign up are identified from then on (~18%).',
+ conversions: ['signup_completed', 'demo_requested'],
+ funnels: [
+ {
+ name: 'Signup',
+ steps: [
+ 'screen_view',
+ 'plan_selected',
+ 'signup_started',
+ 'signup_completed',
+ ],
+ breakdowns: ['referrer_name', 'device', 'country', 'plan'],
+ },
+ {
+ name: 'Demo request',
+ steps: ['screen_view', 'cta_clicked', 'demo_requested'],
+ breakdowns: ['cta', 'company_size'],
+ },
+ ],
+ journey(rng, journey, visitor) {
+ const entry = rng.pick([
+ { weight: 55, value: 'home' },
+ { weight: 25, value: 'blog' },
+ { weight: 20, value: 'docs' },
+ ] as const);
+
+ if (entry === 'home') {
+ journey.view('/', 'Acme — product analytics');
+ } else if (entry === 'blog') {
+ const topic = rng.one(BLOG_TOPICS);
+ const n = rng.int(1, BLOG_POSTS_PER_TOPIC);
+ journey.view(`/blog/${topic}-${n}`, `${topic} #${n} — Acme blog`, {
+ topic,
+ });
+ } else {
+ const [slug, title] = rng.one(DOCS_PAGES);
+ journey.view(`/docs/${slug}`, title);
+ }
+
+ if (rng.chance(BOUNCE_CHANCE)) {
+ return;
+ }
+
+ if (rng.chance(CTA_CHANCE)) {
+ journey.event('cta_clicked', {
+ cta: rng.one(['start-free', 'book-demo', 'view-docs']),
+ location: rng.one(['hero', 'nav', 'footer']),
+ });
+ }
+ if (rng.chance(DOCS_CHANCE)) {
+ const [slug, title] = rng.one(DOCS_PAGES);
+ journey.view(`/docs/${slug}`, title);
+ if (rng.chance(DOCS_SEARCH_CHANCE)) {
+ journey.event('docs_searched', {
+ query: rng.one([
+ 'identify',
+ 'self host',
+ 'react native',
+ 'export',
+ 'funnel',
+ ]),
+ results: rng.int(0, 12),
+ });
+ }
+ if (rng.chance(0.15)) {
+ journey.event('docs_feedback', {
+ helpful: rng.chance(0.8),
+ page: slug,
+ });
+ }
+ }
+
+ const wantsPricing =
+ rng.chance(PRICING_CHANCE) ||
+ (visitor.person !== null && !visitor.identified && rng.chance(0.5));
+ if (wantsPricing) {
+ journey.view('/pricing', 'Pricing');
+ if (rng.chance(0.3)) {
+ journey.event('pricing_toggled', {
+ interval: rng.one(['monthly', 'yearly']),
+ });
+ }
+ if (
+ visitor.person &&
+ !visitor.identified &&
+ rng.chance(SIGNUP_STARTED_CHANCE)
+ ) {
+ const plan = visitor.person.properties.plan ?? 'free';
+ journey.event('plan_selected', {
+ plan,
+ interval: rng.one(['monthly', 'yearly']),
+ });
+ journey.view('/signup', 'Create your account', { plan });
+ const method = rng.one(['email', 'github', 'google']);
+ journey.event('signup_started', { method, plan });
+ if (rng.chance(SIGNUP_COMPLETED_CHANCE)) {
+ // The app signs the new user in; from here on the device is identified.
+ journey.identify(visitor.person);
+ journey.event('signup_completed', {
+ method,
+ plan,
+ ...visitor.person.properties,
+ });
+ journey.view('/welcome', 'Welcome');
+ }
+ }
+ }
+
+ if (rng.chance(NEWSLETTER_CHANCE)) {
+ journey.event('newsletter_subscribed', {
+ placement: entry === 'blog' ? 'blog-footer' : 'homepage',
+ });
+ }
+ if (rng.chance(DEMO_CHANCE)) {
+ journey.view('/demo', 'Book a demo');
+ journey.event('demo_requested', {
+ company_size: rng.one(COMPANY_SIZES),
+ use_case: rng.one(['web', 'mobile', 'saas', 'ecommerce']),
+ });
+ } else if (rng.chance(CONTACT_CHANCE)) {
+ journey.view('/contact', 'Contact');
+ journey.event('contact_form_submitted', {
+ topic: rng.one(['sales', 'support', 'partnership']),
+ });
+ }
+ },
+};
diff --git a/packages/seed/src/cli.ts b/packages/seed/src/cli.ts
new file mode 100644
index 000000000..6d20c9177
--- /dev/null
+++ b/packages/seed/src/cli.ts
@@ -0,0 +1,228 @@
+// `bun run seed [--size small|medium|large|xl] [--days N --sessions-per-day N]
+// [--seed N] [--projects website,saas,...] [--reset] [--dry-run] [--out path]`
+//
+// Targets whatever DATABASE_URL and CLICKHOUSE_URL name (a worktree's own
+// databases). Same flags and seed ⇒ identical data.
+
+import { resolve } from 'node:path';
+import { parseArgs } from 'node:util';
+import { db } from '@openpanel/db';
+import { ARCHETYPE_IDS, ARCHETYPES, archetypeById } from './archetypes';
+import { type GenerateConfig, generate, planDays } from './generator';
+import { projectIdFor } from './ids';
+import { buildManifest, describeManifest, writeManifest } from './manifest';
+import { recordProjectStats, seedPostgres } from './postgres';
+import { resetProjects } from './reset';
+import { Rng } from './rng';
+import {
+ DEFAULT_SEED,
+ DEFAULT_SIZE,
+ MANIFEST_FILE,
+ SIZE_PRESETS,
+ type SizePreset,
+} from './seed.constants';
+import { ClickhouseSink } from './sink';
+
+/** Sessions carry ~7 rows once session_start/session_end are added; only for the estimate. */
+const EVENTS_PER_SESSION_ESTIMATE = 7;
+const DEFAULT_VARIANCE = 1;
+const MS_PER_DAY = 86_400_000;
+
+const USAGE = `Usage: bun run seed [options]
+
+ --size <${Object.keys(SIZE_PRESETS).join('|')}> preset (default: ${DEFAULT_SIZE})
+ --days <n> override the preset's day count
+ --sessions-per-day <n> override the preset's sessions per day (all projects)
+ --seed <n> deterministic seed (default: ${DEFAULT_SEED})
+ --projects <a,b> archetypes to seed (default: ${ARCHETYPE_IDS.join(',')})
+ --variance <x> 0 regular … 1 lively … 2 chaotic (default: ${DEFAULT_VARIANCE})
+ --reset delete earlier seeded rows for these projects first
+ --dry-run print the plan and exit
+ --out <path> where to write the manifest (default: <repo>/${MANIFEST_FILE})
+ --help
+`;
+
+function isPreset(value: string): value is SizePreset {
+ return value in SIZE_PRESETS;
+}
+
+function integer(
+ value: string | undefined,
+ name: string,
+ fallback: number
+): number {
+ if (value === undefined) {
+ return fallback;
+ }
+ const parsed = Number(value);
+ if (!Number.isInteger(parsed) || parsed < 0) {
+ throw new Error(`--${name} must be a non-negative integer`);
+ }
+ return parsed;
+}
+
+function maskedUrl(url: string | undefined): string {
+ if (!url) {
+ return '<unset>';
+ }
+ try {
+ const parsed = new URL(url);
+ if (parsed.password) {
+ parsed.password = '***';
+ }
+ return parsed.toString();
+ } catch {
+ return url;
+ }
+}
+
+function main(): Promise<void> {
+ const { values } = parseArgs({
+ args: process.argv.slice(2),
+ options: {
+ size: { type: 'string' },
+ days: { type: 'string' },
+ 'sessions-per-day': { type: 'string' },
+ seed: { type: 'string' },
+ projects: { type: 'string' },
+ variance: { type: 'string' },
+ reset: { type: 'boolean', default: false },
+ 'dry-run': { type: 'boolean', default: false },
+ out: { type: 'string' },
+ help: { type: 'boolean', default: false },
+ },
+ });
+
+ if (values.help) {
+ process.stdout.write(USAGE);
+ return Promise.resolve();
+ }
+
+ const size = values.size ?? DEFAULT_SIZE;
+ if (!isPreset(size)) {
+ throw new Error(`Unknown size "${size}". ${USAGE}`);
+ }
+ const preset = SIZE_PRESETS[size];
+ const days = integer(values.days, 'days', preset.days);
+ const sessionsPerDay = integer(
+ values['sessions-per-day'],
+ 'sessions-per-day',
+ preset.sessionsPerDay
+ );
+ const seed = integer(values.seed, 'seed', DEFAULT_SEED);
+ const variance =
+ values.variance === undefined ? DEFAULT_VARIANCE : Number(values.variance);
+ const archetypes = values.projects
+ ? values.projects.split(',').map((id) => archetypeById(id.trim()))
+ : ARCHETYPES;
+ const outPath =
+ values.out ?? resolve(import.meta.dir, '..', '..', '..', MANIFEST_FILE);
+
+ return run({
+ size,
+ days,
+ sessionsPerDay,
+ seed,
+ variance,
+ archetypes,
+ reset: values.reset,
+ dryRun: values['dry-run'],
+ outPath,
+ });
+}
+
+async function run(options: {
+ size: SizePreset;
+ days: number;
+ sessionsPerDay: number;
+ seed: number;
+ variance: number;
+ archetypes: typeof ARCHETYPES;
+ reset: boolean;
+ dryRun: boolean;
+ outPath: string;
+}): Promise<void> {
+ const now = new Date();
+ const config: GenerateConfig = {
+ seed: options.seed,
+ sessionsPerDay: options.sessionsPerDay,
+ days: options.days,
+ now,
+ variance: options.variance,
+ projects: options.archetypes.map((archetype) => ({
+ archetype,
+ projectId: projectIdFor(archetype),
+ })),
+ };
+
+ const plannedSessions = planDays(config).reduce(
+ (sum, day) => sum + day.sessions,
+ 0
+ );
+ console.log(`Postgres: ${maskedUrl(process.env.DATABASE_URL)}`);
+ console.log(`ClickHouse: ${maskedUrl(process.env.CLICKHOUSE_URL)}`);
+ console.log(
+ `Plan: ${options.days} days × ${options.sessionsPerDay} sessions/day (size ${options.size}, seed ${options.seed}) ≈ ${plannedSessions.toLocaleString()} sessions, ≈ ${(plannedSessions * EVENTS_PER_SESSION_ESTIMATE).toLocaleString()} events`
+ );
+ if (options.dryRun) {
+ return;
+ }
+
+ const rng = new Rng(`${options.seed}/postgres`);
+ const postgres = await seedPostgres(rng, options.archetypes);
+ console.log(
+ `Postgres ready: organization ${postgres.organizationId}, ${postgres.projects.length} projects`
+ );
+
+ if (options.reset) {
+ console.log('Resetting earlier seeded rows…');
+ await resetProjects(postgres.projects.map((project) => project.id));
+ }
+
+ const sink = new ClickhouseSink();
+ const started = Date.now();
+ const stats = await generate(config, sink, (progress) => {
+ process.stdout.write(
+ `\r${progress.day} ${progress.sessions.toLocaleString()} sessions, ${progress.events.toLocaleString()} events`.padEnd(
+ 80
+ )
+ );
+ });
+ process.stdout.write('\n');
+ console.log(`Inserted in ${((Date.now() - started) / 1000).toFixed(1)}s`);
+
+ const perProject = new Map<string, { sessions: number; events: number }>();
+ for (const project of stats) {
+ perProject.set(project.projectId, {
+ sessions: project.sessions,
+ events: project.events,
+ });
+ await recordProjectStats(project.projectId, {
+ eventsCount: project.events,
+ firstEventAt: project.firstEventAt,
+ });
+ }
+
+ const manifest = buildManifest({
+ seed: options.seed,
+ size: options.size,
+ days: options.days,
+ sessionsPerDay: options.sessionsPerDay,
+ from: new Date(now.getTime() - options.days * MS_PER_DAY),
+ to: now,
+ postgres,
+ apiUrl: process.env.API_URL ?? null,
+ perProject,
+ totals: sink.counts(),
+ });
+ writeManifest(options.outPath, manifest);
+ console.log(describeManifest(manifest));
+ console.log(`Manifest: ${options.outPath}`);
+}
+
+try {
+ await main();
+} finally {
+ await db.$disconnect();
+}
+process.exit(0);
diff --git a/packages/seed/src/clickhouse-rows.test.ts b/packages/seed/src/clickhouse-rows.test.ts
new file mode 100644
index 000000000..7b09062a3
--- /dev/null
+++ b/packages/seed/src/clickhouse-rows.test.ts
@@ -0,0 +1,109 @@
+// The row shapes are copied from core, not imported. This is what keeps them
+// honest: every key the seed writes must be a column, and every column must
+// be written, for the three tables it inserts into.
+
+import { describe, expect, test } from 'bun:test';
+import { readFileSync } from 'node:fs';
+import { resolve } from 'node:path';
+import { ARCHETYPES } from './archetypes';
+import type {
+ ClickhouseEventRow,
+ ClickhouseProfileRow,
+ ClickhouseSessionRow,
+} from './clickhouse-rows';
+import { generate } from './generator';
+import { projectIdFor } from './ids';
+import type { RowSink, SinkCounts } from './sink';
+
+const SCHEMA_FILE = resolve(
+ import.meta.dir,
+ '..',
+ '..',
+ '..',
+ 'test',
+ 'clickhouse-schema.sql'
+);
+
+/** Column names of one `CREATE TABLE` in the flat schema; indexes and projections are skipped. */
+function schemaColumns(table: string): Set<string> {
+ const sql = readFileSync(SCHEMA_FILE, 'utf8');
+ const start = sql.indexOf(`CREATE TABLE IF NOT EXISTS ${table}\n`);
+ if (start < 0) {
+ throw new Error(`${table} not in ${SCHEMA_FILE}`);
+ }
+ const body = sql.slice(start, sql.indexOf('\nENGINE', start));
+ const columns = new Set<string>();
+ for (const match of body.matchAll(/^\s+`([a-z_]+)`/gm)) {
+ columns.add(match[1] ?? '');
+ }
+ return columns;
+}
+
+class CollectingSink implements RowSink {
+ events: ClickhouseEventRow[] = [];
+ sessions: ClickhouseSessionRow[] = [];
+ profiles: ClickhouseProfileRow[] = [];
+ addEvents(rows: readonly ClickhouseEventRow[]): void {
+ this.events.push(...rows);
+ }
+ addSession(row: ClickhouseSessionRow): void {
+ this.sessions.push(row);
+ }
+ addProfiles(rows: readonly ClickhouseProfileRow[]): void {
+ this.profiles.push(...rows);
+ }
+ flushFull(): Promise<void> {
+ return Promise.resolve();
+ }
+ flush(): Promise<void> {
+ return Promise.resolve();
+ }
+ counts(): SinkCounts {
+ return {
+ events: this.events.length,
+ sessions: this.sessions.length,
+ profiles: this.profiles.length,
+ };
+ }
+}
+
+async function sampleRows(): Promise<CollectingSink> {
+ const sink = new CollectingSink();
+ await generate(
+ {
+ seed: 1,
+ sessionsPerDay: 60,
+ days: 1,
+ now: new Date('2026-09-20T12:00:00.000Z'),
+ variance: 1,
+ projects: ARCHETYPES.map((archetype) => ({
+ archetype,
+ projectId: projectIdFor(archetype),
+ })),
+ },
+ sink,
+ () => undefined
+ );
+ return sink;
+}
+
+function keysOf(rows: readonly object[]): Set<string> {
+ const keys = new Set<string>();
+ for (const row of rows) {
+ for (const key of Object.keys(row)) {
+ keys.add(key);
+ }
+ }
+ return keys;
+}
+
+describe('seed rows match the ClickHouse schema', () => {
+ for (const table of ['events', 'sessions', 'profiles'] as const) {
+ test(table, async () => {
+ const sink = await sampleRows();
+ const written = keysOf(sink[table]);
+ const columns = schemaColumns(table);
+ expect([...written].sort()).toEqual([...columns].sort());
+ });
+ }
+});
diff --git a/packages/seed/src/clickhouse-rows.ts b/packages/seed/src/clickhouse-rows.ts
new file mode 100644
index 000000000..d14798d7e
--- /dev/null
+++ b/packages/seed/src/clickhouse-rows.ts
@@ -0,0 +1,100 @@
+// The rows the seed inserts. Copied from core's `IClickhouseEvent`,
+// `IClickhouseSession` and `IClickhouseProfile` rather than imported: like
+// code-migrations, the seed is a script with direct database access and
+// stays out of the backend's dependency graph. `clickhouse-rows.test.ts`
+// checks these shapes against the schema.
+
+export interface ClickhouseEventRow {
+ id: string;
+ name: string;
+ device_id: string;
+ profile_id: string;
+ project_id: string;
+ session_id: string;
+ path: string;
+ origin: string;
+ referrer: string;
+ referrer_name: string;
+ referrer_type: string;
+ duration: number;
+ properties: Record<string, string>;
+ created_at: string;
+ country: string;
+ city: string;
+ region: string;
+ longitude: number | null;
+ latitude: number | null;
+ os: string;
+ os_version: string;
+ browser: string;
+ browser_version: string;
+ device: string;
+ brand: string;
+ model: string;
+ imported_at: string | null;
+ inserted_at: string;
+ sdk_name: string;
+ sdk_version: string;
+ revenue?: number;
+ groups: string[];
+}
+
+export interface ClickhouseSessionRow {
+ id: string;
+ profile_id: string;
+ event_count: number;
+ screen_view_count: number;
+ entry_path: string;
+ entry_origin: string;
+ exit_path: string;
+ exit_origin: string;
+ created_at: string;
+ ended_at: string;
+ referrer: string;
+ referrer_name: string;
+ referrer_type: string;
+ os: string;
+ os_version: string;
+ browser: string;
+ browser_version: string;
+ device: string;
+ brand: string;
+ model: string;
+ country: string;
+ region: string;
+ city: string;
+ longitude: number | null;
+ latitude: number | null;
+ is_bounce: boolean;
+ project_id: string;
+ device_id: string;
+ duration: number;
+ utm_medium: string;
+ utm_source: string;
+ utm_campaign: string;
+ utm_content: string;
+ utm_term: string;
+ revenue: number;
+ sign: 1 | -1;
+ version: number;
+ groups: string[];
+}
+
+export interface ClickhouseProfileRow {
+ id: string;
+ first_name: string;
+ last_name: string;
+ email: string;
+ avatar: string;
+ properties: Record<string, string>;
+ project_id: string;
+ is_external: boolean;
+ created_at: string;
+ last_seen_at: string;
+ groups: string[];
+}
+
+/** `DateTime64(3)` in UTC, the format the worker writes: `yyyy-MM-dd HH:mm:ss.SSS`. */
+export function toClickhouseDateTime(date: Date): string {
+ return date.toISOString().replace('T', ' ').replace('Z', '');
+}
diff --git a/packages/seed/src/data/devices.ts b/packages/seed/src/data/devices.ts
new file mode 100644
index 000000000..9b2d289f9
--- /dev/null
+++ b/packages/seed/src/data/devices.ts
@@ -0,0 +1,114 @@
+// User agents only; os/browser/device fields come from the real parser at
+// session time, so the seed can never disagree with the ingest worker.
+
+import type { Weighted } from '../rng';
+
+export const DESKTOP_USER_AGENTS: readonly Weighted<string>[] = [
+ {
+ weight: 40,
+ value:
+ 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36',
+ },
+ {
+ weight: 25,
+ value:
+ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36',
+ },
+ {
+ weight: 12,
+ value:
+ 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Safari/605.1.15',
+ },
+ {
+ weight: 8,
+ value:
+ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0',
+ },
+ {
+ weight: 8,
+ value:
+ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Edg/128.0.0.0',
+ },
+ {
+ weight: 5,
+ value:
+ 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36',
+ },
+ {
+ weight: 2,
+ value:
+ 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0',
+ },
+];
+
+export const MOBILE_USER_AGENTS: readonly Weighted<string>[] = [
+ {
+ weight: 30,
+ value:
+ 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Mobile/15E148 Safari/604.1',
+ },
+ {
+ weight: 15,
+ value:
+ 'Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0 Mobile/15E148 Safari/604.1',
+ },
+ {
+ weight: 8,
+ value:
+ 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/128.0.6613.98 Mobile/15E148 Safari/604.1',
+ },
+ {
+ weight: 18,
+ value:
+ 'Mozilla/5.0 (Linux; Android 14; SM-S928B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Mobile Safari/537.36',
+ },
+ {
+ weight: 12,
+ value:
+ 'Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Mobile Safari/537.36',
+ },
+ {
+ weight: 7,
+ value:
+ 'Mozilla/5.0 (Linux; Android 13; SM-A546B) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/25.0 Chrome/121.0.0.0 Mobile Safari/537.36',
+ },
+ {
+ weight: 5,
+ value:
+ 'Mozilla/5.0 (Linux; Android 14; 2312DRA50G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36',
+ },
+ {
+ weight: 5,
+ value:
+ 'Mozilla/5.0 (iPad; CPU OS 17_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Mobile/15E148 Safari/604.1',
+ },
+];
+
+/** What the React Native SDK reports: the WebView user agent. */
+export const NATIVE_APP_USER_AGENTS: readonly Weighted<string>[] = [
+ {
+ weight: 30,
+ value:
+ 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148',
+ },
+ {
+ weight: 25,
+ value:
+ 'Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148',
+ },
+ {
+ weight: 20,
+ value:
+ 'Mozilla/5.0 (Linux; Android 14; SM-S928B Build/UP1A.231005.007; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/128.0.0.0 Mobile Safari/537.36',
+ },
+ {
+ weight: 15,
+ value:
+ 'Mozilla/5.0 (Linux; Android 14; Pixel 8 Pro Build/AP2A.240805.005; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/128.0.0.0 Mobile Safari/537.36',
+ },
+ {
+ weight: 10,
+ value:
+ 'Mozilla/5.0 (Linux; Android 13; SM-A546B Build/TP1A.220624.014; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/126.0.0.0 Mobile Safari/537.36',
+ },
+];
diff --git a/packages/seed/src/data/geo.ts b/packages/seed/src/data/geo.ts
new file mode 100644
index 000000000..d842a395b
--- /dev/null
+++ b/packages/seed/src/data/geo.ts
@@ -0,0 +1,343 @@
+import type { Geo } from '../model';
+import type { Rng, Weighted } from '../rng';
+
+interface City {
+ country: string;
+ region: string;
+ city: string;
+ latitude: number;
+ longitude: number;
+ /** A /16 GeoIP attributes to roughly this region; only decorates `x-client-ip`-style data. */
+ ipPrefix: string;
+}
+
+const CITIES: readonly Weighted<City>[] = [
+ {
+ weight: 14,
+ value: {
+ country: 'SE',
+ region: 'Stockholm',
+ city: 'Stockholm',
+ latitude: 59.33,
+ longitude: 18.07,
+ ipPrefix: '83.233',
+ },
+ },
+ {
+ weight: 4,
+ value: {
+ country: 'SE',
+ region: 'Västra Götaland',
+ city: 'Gothenburg',
+ latitude: 57.71,
+ longitude: 11.97,
+ ipPrefix: '83.250',
+ },
+ },
+ {
+ weight: 3,
+ value: {
+ country: 'SE',
+ region: 'Skåne',
+ city: 'Malmö',
+ latitude: 55.6,
+ longitude: 13.0,
+ ipPrefix: '90.230',
+ },
+ },
+ {
+ weight: 16,
+ value: {
+ country: 'US',
+ region: 'California',
+ city: 'San Francisco',
+ latitude: 37.77,
+ longitude: -122.42,
+ ipPrefix: '104.28',
+ },
+ },
+ {
+ weight: 10,
+ value: {
+ country: 'US',
+ region: 'New York',
+ city: 'New York',
+ latitude: 40.71,
+ longitude: -74.01,
+ ipPrefix: '74.72',
+ },
+ },
+ {
+ weight: 5,
+ value: {
+ country: 'US',
+ region: 'Texas',
+ city: 'Austin',
+ latitude: 30.27,
+ longitude: -97.74,
+ ipPrefix: '70.112',
+ },
+ },
+ {
+ weight: 4,
+ value: {
+ country: 'US',
+ region: 'Washington',
+ city: 'Seattle',
+ latitude: 47.61,
+ longitude: -122.33,
+ ipPrefix: '73.140',
+ },
+ },
+ {
+ weight: 9,
+ value: {
+ country: 'GB',
+ region: 'England',
+ city: 'London',
+ latitude: 51.51,
+ longitude: -0.13,
+ ipPrefix: '81.2',
+ },
+ },
+ {
+ weight: 3,
+ value: {
+ country: 'GB',
+ region: 'England',
+ city: 'Manchester',
+ latitude: 53.48,
+ longitude: -2.24,
+ ipPrefix: '86.20',
+ },
+ },
+ {
+ weight: 7,
+ value: {
+ country: 'DE',
+ region: 'Berlin',
+ city: 'Berlin',
+ latitude: 52.52,
+ longitude: 13.41,
+ ipPrefix: '77.12',
+ },
+ },
+ {
+ weight: 3,
+ value: {
+ country: 'DE',
+ region: 'Bavaria',
+ city: 'Munich',
+ latitude: 48.14,
+ longitude: 11.58,
+ ipPrefix: '84.153',
+ },
+ },
+ {
+ weight: 4,
+ value: {
+ country: 'NL',
+ region: 'North Holland',
+ city: 'Amsterdam',
+ latitude: 52.37,
+ longitude: 4.9,
+ ipPrefix: '77.160',
+ },
+ },
+ {
+ weight: 4,
+ value: {
+ country: 'FR',
+ region: 'Île-de-France',
+ city: 'Paris',
+ latitude: 48.86,
+ longitude: 2.35,
+ ipPrefix: '82.64',
+ },
+ },
+ {
+ weight: 3,
+ value: {
+ country: 'ES',
+ region: 'Madrid',
+ city: 'Madrid',
+ latitude: 40.42,
+ longitude: -3.7,
+ ipPrefix: '81.32',
+ },
+ },
+ {
+ weight: 3,
+ value: {
+ country: 'DK',
+ region: 'Capital Region',
+ city: 'Copenhagen',
+ latitude: 55.68,
+ longitude: 12.57,
+ ipPrefix: '80.62',
+ },
+ },
+ {
+ weight: 3,
+ value: {
+ country: 'NO',
+ region: 'Oslo',
+ city: 'Oslo',
+ latitude: 59.91,
+ longitude: 10.75,
+ ipPrefix: '84.208',
+ },
+ },
+ {
+ weight: 3,
+ value: {
+ country: 'FI',
+ region: 'Uusimaa',
+ city: 'Helsinki',
+ latitude: 60.17,
+ longitude: 24.94,
+ ipPrefix: '85.76',
+ },
+ },
+ {
+ weight: 3,
+ value: {
+ country: 'PL',
+ region: 'Masovian',
+ city: 'Warsaw',
+ latitude: 52.23,
+ longitude: 21.01,
+ ipPrefix: '83.4',
+ },
+ },
+ {
+ weight: 4,
+ value: {
+ country: 'CA',
+ region: 'Ontario',
+ city: 'Toronto',
+ latitude: 43.65,
+ longitude: -79.38,
+ ipPrefix: '99.224',
+ },
+ },
+ {
+ weight: 4,
+ value: {
+ country: 'AU',
+ region: 'New South Wales',
+ city: 'Sydney',
+ latitude: -33.87,
+ longitude: 151.21,
+ ipPrefix: '1.128',
+ },
+ },
+ {
+ weight: 4,
+ value: {
+ country: 'IN',
+ region: 'Karnataka',
+ city: 'Bengaluru',
+ latitude: 12.97,
+ longitude: 77.59,
+ ipPrefix: '49.204',
+ },
+ },
+ {
+ weight: 3,
+ value: {
+ country: 'BR',
+ region: 'São Paulo',
+ city: 'São Paulo',
+ latitude: -23.55,
+ longitude: -46.63,
+ ipPrefix: '177.220',
+ },
+ },
+ {
+ weight: 3,
+ value: {
+ country: 'JP',
+ region: 'Tokyo',
+ city: 'Tokyo',
+ latitude: 35.68,
+ longitude: 139.69,
+ ipPrefix: '126.0',
+ },
+ },
+ {
+ weight: 2,
+ value: {
+ country: 'SG',
+ region: '',
+ city: 'Singapore',
+ latitude: 1.35,
+ longitude: 103.82,
+ ipPrefix: '116.14',
+ },
+ },
+ {
+ weight: 2,
+ value: {
+ country: 'ZA',
+ region: 'Western Cape',
+ city: 'Cape Town',
+ latitude: -33.92,
+ longitude: 18.42,
+ ipPrefix: '41.13',
+ },
+ },
+ {
+ weight: 2,
+ value: {
+ country: 'MX',
+ region: 'Mexico City',
+ city: 'Mexico City',
+ latitude: 19.43,
+ longitude: -99.13,
+ ipPrefix: '187.188',
+ },
+ },
+ {
+ weight: 2,
+ value: {
+ country: 'IE',
+ region: 'Leinster',
+ city: 'Dublin',
+ latitude: 53.35,
+ longitude: -6.26,
+ ipPrefix: '86.43',
+ },
+ },
+ {
+ weight: 2,
+ value: {
+ country: 'CH',
+ region: 'Zurich',
+ city: 'Zurich',
+ latitude: 47.38,
+ longitude: 8.54,
+ ipPrefix: '85.5',
+ },
+ },
+];
+
+/** Lat/lon jitter so map views do not stack every visitor of a city on one pixel. */
+const COORDINATE_JITTER = 0.05;
+const COORDINATE_DECIMALS = 4;
+
+function round(value: number): number {
+ return Number(value.toFixed(COORDINATE_DECIMALS));
+}
+
+export function newGeo(rng: Rng): Geo {
+ const city = rng.pick(CITIES);
+ return {
+ ip: `${city.ipPrefix}.${rng.int(1, 254)}.${rng.int(1, 254)}`,
+ country: city.country,
+ region: city.region,
+ city: city.city,
+ latitude: round(city.latitude + rng.normal() * COORDINATE_JITTER),
+ longitude: round(city.longitude + rng.normal() * COORDINATE_JITTER),
+ };
+}
diff --git a/packages/seed/src/data/people.ts b/packages/seed/src/data/people.ts
new file mode 100644
index 000000000..fce3d2211
--- /dev/null
+++ b/packages/seed/src/data/people.ts
@@ -0,0 +1,136 @@
+import type { Person } from '../model';
+import type { Rng } from '../rng';
+
+const FIRST_NAMES = [
+ 'Anna',
+ 'Erik',
+ 'Maja',
+ 'Oskar',
+ 'Elsa',
+ 'Liam',
+ 'Olivia',
+ 'Noah',
+ 'Emma',
+ 'Lucas',
+ 'Sofia',
+ 'Hugo',
+ 'Alice',
+ 'William',
+ 'Ella',
+ 'Leo',
+ 'Mia',
+ 'Elias',
+ 'Julia',
+ 'Axel',
+ 'Nora',
+ 'Felix',
+ 'Ida',
+ 'Vincent',
+ 'Clara',
+ 'Adam',
+ 'Freja',
+ 'Isak',
+ 'Astrid',
+ 'Theo',
+ 'James',
+ 'Charlotte',
+ 'Benjamin',
+ 'Amelia',
+ 'Henry',
+ 'Harper',
+ 'Jack',
+ 'Evelyn',
+ 'Owen',
+ 'Luna',
+ 'Mateo',
+ 'Camila',
+ 'Yuki',
+ 'Haruto',
+ 'Aarav',
+ 'Priya',
+ 'Chen',
+ 'Mei',
+ 'Lena',
+ 'Jonas',
+];
+
+const LAST_NAMES = [
+ 'Andersson',
+ 'Johansson',
+ 'Karlsson',
+ 'Nilsson',
+ 'Eriksson',
+ 'Larsson',
+ 'Olsson',
+ 'Persson',
+ 'Smith',
+ 'Johnson',
+ 'Williams',
+ 'Brown',
+ 'Jones',
+ 'Garcia',
+ 'Miller',
+ 'Davis',
+ 'Martinez',
+ 'Müller',
+ 'Schmidt',
+ 'Schneider',
+ 'Fischer',
+ 'Weber',
+ 'Meyer',
+ 'Dubois',
+ 'Martin',
+ 'Bernard',
+ 'Rossi',
+ 'Russo',
+ 'Ferrari',
+ 'Silva',
+ 'Santos',
+ 'Tanaka',
+ 'Suzuki',
+ 'Sharma',
+ 'Patel',
+ 'Wang',
+];
+
+const EMAIL_DOMAINS: readonly string[] = [
+ 'gmail.com',
+ 'outlook.com',
+ 'icloud.com',
+ 'proton.me',
+ 'hey.com',
+ 'yahoo.com',
+];
+const COMPANY_DOMAINS: readonly string[] = [
+ 'acme.io',
+ 'northwind.co',
+ 'globex.com',
+ 'initech.dev',
+ 'umbrella.app',
+ 'hooli.xyz',
+];
+
+const AVATAR_BASE = 'https://api.dicebear.com/9.x/notionists/svg?seed=';
+
+function slugify(value: string): string {
+ return value
+ .normalize('NFD')
+ .replace(/[̀-ͯ]/g, '')
+ .toLowerCase()
+ .replace(/[^a-z0-9]+/g, '');
+}
+
+export function newPerson(rng: Rng, options: { business: boolean }): Person {
+ const firstName = rng.one(FIRST_NAMES);
+ const lastName = rng.one(LAST_NAMES);
+ const domain = rng.one(options.business ? COMPANY_DOMAINS : EMAIL_DOMAINS);
+ const handle = `${slugify(firstName)}.${slugify(lastName)}${rng.int(1, 999)}`;
+ return {
+ id: `usr_${rng.hex(12)}`,
+ firstName,
+ lastName,
+ email: `${handle}@${domain}`,
+ avatar: `${AVATAR_BASE}${handle}`,
+ properties: {},
+ };
+}
diff --git a/packages/seed/src/data/referrers.ts b/packages/seed/src/data/referrers.ts
new file mode 100644
index 000000000..da142ab9a
--- /dev/null
+++ b/packages/seed/src/data/referrers.ts
@@ -0,0 +1,120 @@
+// The referrers the seed sends traffic from. Name and type are what core's
+// `parseReferrer` (its generated referrer catalogue) answers for each URL,
+// copied here so the seed stays out of core's dependency graph; `null` is
+// direct traffic.
+
+import type { Referrer } from '../model';
+import type { Weighted } from '../rng';
+
+export const REFERRERS = {
+ google: { url: 'https://www.google.com', name: 'Google', type: 'search' },
+ bing: { url: 'https://www.bing.com', name: 'Bing', type: 'search' },
+ duckDuckGo: {
+ url: 'https://duckduckgo.com',
+ name: 'DuckDuckGo',
+ type: 'search',
+ },
+ twitter: { url: 'https://t.co', name: 'Twitter', type: 'social' },
+ facebook: { url: 'https://m.facebook.com', name: 'Facebook', type: 'social' },
+ reddit: { url: 'https://www.reddit.com', name: 'Reddit', type: 'social' },
+ hackerNews: {
+ url: 'https://news.ycombinator.com',
+ name: 'Hacker News',
+ type: 'social',
+ },
+ linkedIn: {
+ url: 'https://www.linkedin.com',
+ name: 'LinkedIn',
+ type: 'social',
+ },
+ instagram: {
+ url: 'https://l.instagram.com',
+ name: 'Instagram',
+ type: 'social',
+ },
+ youTube: { url: 'https://www.youtube.com', name: 'Youtube', type: 'social' },
+ gitHub: { url: 'https://github.com', name: 'GitHub', type: 'tech' },
+ chatGpt: { url: 'https://chatgpt.com', name: 'ChatGPT', type: 'ai' },
+ claude: { url: 'https://claude.ai', name: 'Claude', type: 'ai' },
+ medium: { url: 'https://medium.com', name: 'Medium', type: 'content' },
+ substack: { url: 'https://substack.com', name: 'Substack', type: 'content' },
+ gmail: { url: 'https://mail.google.com', name: 'Gmail', type: 'email' },
+ productHunt: {
+ url: 'https://www.producthunt.com',
+ name: 'Product Hunt',
+ type: 'social',
+ },
+} as const satisfies Record<string, Referrer>;
+
+export type ReferrerMix = readonly Weighted<Referrer | null>[];
+
+export interface Campaign {
+ source: string;
+ medium: string;
+ name: string;
+}
+
+export const CAMPAIGNS: readonly Weighted<Campaign>[] = [
+ { weight: 5, value: { source: 'google', medium: 'cpc', name: 'brand' } },
+ { weight: 4, value: { source: 'google', medium: 'cpc', name: 'generic' } },
+ {
+ weight: 3,
+ value: { source: 'facebook', medium: 'paid_social', name: 'autumn-sale' },
+ },
+ {
+ weight: 3,
+ value: { source: 'instagram', medium: 'paid_social', name: 'new-arrivals' },
+ },
+ {
+ weight: 2,
+ value: { source: 'newsletter', medium: 'email', name: 'weekly-digest' },
+ },
+ {
+ weight: 2,
+ value: { source: 'linkedin', medium: 'paid_social', name: 'b2b-launch' },
+ },
+ {
+ weight: 1,
+ value: { source: 'producthunt', medium: 'referral', name: 'launch' },
+ },
+];
+
+/** By name and by the hostname's own label, the two ways a `utm_source` names a referrer. */
+const KNOWN_SOURCES: Record<string, Referrer> = Object.fromEntries(
+ Object.values(REFERRERS).flatMap((referrer) => {
+ const label =
+ new URL(referrer.url).hostname
+ .replace(/^(www|m|l)\./, '')
+ .split('.')[0] ?? '';
+ return [
+ [referrer.name.toLowerCase(), referrer],
+ [label, referrer],
+ ];
+ })
+);
+
+/**
+ * What the ingest handler stores for a landing event: the referrer's URL, but a
+ * `utm_source` that names a known referrer overrides its name and type, and
+ * an unknown one becomes the name with no type (`getReferrerWithQuery`).
+ */
+export function sessionReferrer(
+ referrer: Referrer | null,
+ query: Record<string, string> | undefined
+): Referrer {
+ const source = (
+ query?.utm_source ??
+ query?.ref ??
+ query?.utm_referrer ??
+ ''
+ ).toLowerCase();
+ const fromQuery =
+ source === ''
+ ? null
+ : (KNOWN_SOURCES[source] ?? { url: '', name: source, type: '' });
+ return {
+ url: referrer?.url ?? '',
+ name: fromQuery?.name || referrer?.name || referrer?.url || '',
+ type: fromQuery?.type || referrer?.type || '',
+ };
+}
diff --git a/packages/seed/src/generator.test.ts b/packages/seed/src/generator.test.ts
new file mode 100644
index 000000000..7492bef48
--- /dev/null
+++ b/packages/seed/src/generator.test.ts
@@ -0,0 +1,369 @@
+import { describe, expect, test } from 'bun:test';
+import { createHash } from 'node:crypto';
+import { ARCHETYPES } from './archetypes';
+import type {
+ ClickhouseEventRow,
+ ClickhouseProfileRow,
+ ClickhouseSessionRow,
+} from './clickhouse-rows';
+import { type GenerateConfig, generate, planDays } from './generator';
+import { projectIdFor } from './ids';
+import type { RowSink, SinkCounts } from './sink';
+
+class MemorySink implements RowSink {
+ events: ClickhouseEventRow[] = [];
+ sessions: ClickhouseSessionRow[] = [];
+ profiles: ClickhouseProfileRow[] = [];
+
+ addEvents(rows: readonly ClickhouseEventRow[]): void {
+ this.events.push(...rows);
+ }
+ addSession(row: ClickhouseSessionRow): void {
+ this.sessions.push(row);
+ }
+ addProfiles(rows: readonly ClickhouseProfileRow[]): void {
+ this.profiles.push(...rows);
+ }
+ flushFull(): Promise<void> {
+ return Promise.resolve();
+ }
+ flush(): Promise<void> {
+ return Promise.resolve();
+ }
+ counts(): SinkCounts {
+ return {
+ events: this.events.length,
+ sessions: this.sessions.length,
+ profiles: this.profiles.length,
+ };
+ }
+}
+
+const NOW = new Date('2026-09-20T12:00:00.000Z');
+const ignoreProgress = () => undefined;
+
+function config(overrides: Partial<GenerateConfig> = {}): GenerateConfig {
+ return {
+ seed: 7,
+ sessionsPerDay: 200,
+ days: 3,
+ now: NOW,
+ variance: 1,
+ projects: ARCHETYPES.map((archetype) => ({
+ archetype,
+ projectId: projectIdFor(archetype),
+ })),
+ ...overrides,
+ };
+}
+
+async function run(
+ overrides: Partial<GenerateConfig> = {}
+): Promise<MemorySink> {
+ const sink = new MemorySink();
+ await generate(config(overrides), sink, ignoreProgress);
+ return sink;
+}
+
+/** `inserted_at` is stamped at insert time by design; everything else must be reproducible. */
+function fingerprint(sink: MemorySink): string {
+ const hash = createHash('sha256');
+ for (const event of sink.events) {
+ hash.update(JSON.stringify({ ...event, inserted_at: undefined }));
+ }
+ for (const session of sink.sessions) {
+ hash.update(JSON.stringify(session));
+ }
+ for (const profile of sink.profiles) {
+ hash.update(JSON.stringify(profile));
+ }
+ return hash.digest('hex');
+}
+
+describe('generate', () => {
+ test('is deterministic for a seed and differs across seeds', async () => {
+ const [first, second, other] = await Promise.all([
+ run(),
+ run(),
+ run({ seed: 8 }),
+ ]);
+ expect(first.events.length).toBeGreaterThan(0);
+ expect(fingerprint(first)).toBe(fingerprint(second));
+ expect(fingerprint(other)).not.toBe(fingerprint(first));
+ });
+
+ test('plans roughly the requested volume', () => {
+ const planned = planDays(config()).reduce(
+ (sum, day) => sum + day.sessions,
+ 0
+ );
+ expect(planned).toBeGreaterThan(200 * 3 * 0.4);
+ expect(planned).toBeLessThan(200 * 3 * 2.5);
+ });
+
+ test('every session has one session_start, one session_end and a matching sessions row', async () => {
+ const sink = await run();
+ const bySession = new Map<string, ClickhouseEventRow[]>();
+ for (const event of sink.events) {
+ const list = bySession.get(event.session_id) ?? [];
+ list.push(event);
+ bySession.set(event.session_id, list);
+ }
+ expect(bySession.size).toBe(sink.sessions.length);
+
+ for (const session of sink.sessions) {
+ const events = bySession.get(session.id);
+ expect(events).toBeDefined();
+ if (!events) {
+ continue;
+ }
+ const names = events.map((event) => event.name);
+ expect(names.filter((name) => name === 'session_start')).toHaveLength(1);
+ expect(names.filter((name) => name === 'session_end')).toHaveLength(1);
+ expect(session.sign).toBe(1);
+ expect(session.screen_view_count + session.event_count).toBe(
+ events.length - 2
+ );
+ expect(session.is_bounce).toBe(session.screen_view_count <= 1);
+ expect(session.duration).toBeGreaterThanOrEqual(0);
+ for (const event of events) {
+ expect(event.project_id).toBe(session.project_id);
+ expect(event.device_id).toBe(session.device_id);
+ expect(event.profile_id).not.toBe('');
+ }
+ const end = events.find((event) => event.name === 'session_end');
+ expect(end?.properties.__bounce).toBe(String(session.is_bounce));
+ expect(end?.duration).toBe(session.duration);
+ }
+ });
+
+ test('events are ordered within a session and never in the future', async () => {
+ const sink = await run();
+ for (const event of sink.events) {
+ expect(new Date(`${event.created_at}Z`).getTime()).toBeLessThanOrEqual(
+ NOW.getTime() + 1000
+ );
+ }
+ });
+
+ test('profiles cover every profile_id the events reference, once each', async () => {
+ const sink = await run();
+ const referenced = new Set(
+ sink.events.map((event) => `${event.project_id}/${event.profile_id}`)
+ );
+ const written = sink.profiles.map(
+ (profile) => `${profile.project_id}/${profile.id}`
+ );
+ expect(new Set(written).size).toBe(written.length);
+ for (const key of referenced) {
+ expect(written).toContain(key);
+ }
+ const identified = sink.profiles.filter((profile) => profile.is_external);
+ expect(identified.length).toBeGreaterThan(0);
+ for (const profile of identified) {
+ expect(profile.email).toContain('@');
+ }
+ });
+
+ test('device and referrer fields come from the real parsers', async () => {
+ const sink = await run();
+ const devices = new Set(sink.events.map((event) => event.device));
+ expect(
+ [...devices].every((device) =>
+ ['desktop', 'mobile', 'tablet'].includes(device)
+ )
+ ).toBe(true);
+ const referrerTypes = new Set(
+ sink.sessions.map((session) => session.referrer_type)
+ );
+ expect(referrerTypes.has('search')).toBe(true);
+ expect(referrerTypes.has('social')).toBe(true);
+ const withUtm = sink.sessions.filter(
+ (session) => session.utm_source !== ''
+ );
+ expect(withUtm.length).toBeGreaterThan(0);
+ });
+});
+
+const MS_PER_DAY = 86_400_000;
+
+function dayOf(event: ClickhouseEventRow): string {
+ return event.created_at.slice(0, 10);
+}
+
+function eventsOf(
+ sink: MemorySink,
+ projectId: string,
+ name?: string
+): ClickhouseEventRow[] {
+ return sink.events.filter(
+ (event) =>
+ event.project_id === projectId &&
+ (name === undefined || event.name === name)
+ );
+}
+
+describe('identity and retention', () => {
+ test('anonymous devices are recognised within a UTC day and never across days', async () => {
+ const sink = await run({ days: 10 });
+ const daysPerDevice = new Map<string, Set<string>>();
+ const sessionsPerDevice = new Map<string, Set<string>>();
+ for (const event of eventsOf(sink, 'acme-web')) {
+ if (event.profile_id !== event.device_id) {
+ continue;
+ }
+ daysPerDevice.set(
+ event.device_id,
+ (daysPerDevice.get(event.device_id) ?? new Set()).add(dayOf(event))
+ );
+ sessionsPerDevice.set(
+ event.device_id,
+ (sessionsPerDevice.get(event.device_id) ?? new Set()).add(
+ event.session_id
+ )
+ );
+ }
+ expect(daysPerDevice.size).toBeGreaterThan(100);
+ for (const days of daysPerDevice.values()) {
+ expect(days.size).toBe(1);
+ }
+ const sameDayReturns = [...sessionsPerDevice.values()].filter(
+ (sessions) => sessions.size >= 2
+ ).length;
+ expect(sameDayReturns).toBeGreaterThan(10);
+ });
+
+ test('identified users keep their profile across days, so retention exists', async () => {
+ const sink = await run({ days: 14, sessionsPerDay: 300 });
+ for (const projectId of ['acme-saas', 'acme-app']) {
+ const firstDay = new Map<string, number>();
+ const activeDays = new Map<string, Set<number>>();
+ for (const event of eventsOf(sink, projectId)) {
+ if (event.profile_id === event.device_id) {
+ continue;
+ }
+ const day = Math.floor(
+ new Date(`${event.created_at}Z`).getTime() / MS_PER_DAY
+ );
+ firstDay.set(
+ event.profile_id,
+ Math.min(firstDay.get(event.profile_id) ?? day, day)
+ );
+ activeDays.set(
+ event.profile_id,
+ (activeDays.get(event.profile_id) ?? new Set()).add(day)
+ );
+ }
+ // Cohorts old enough to have a day-7: those who started in the first 5 days.
+ const cutoff = Math.min(...firstDay.values()) + 5;
+ const cohort = [...firstDay].filter(([, day]) => day <= cutoff);
+ const retainedDay7 = cohort.filter(([id, day]) =>
+ activeDays.get(id)?.has(day + 7)
+ ).length;
+ expect(cohort.length).toBeGreaterThan(50);
+ expect(retainedDay7 / cohort.length).toBeGreaterThan(0.1);
+ }
+ });
+
+ test('a device that identifies mid-session leaves both an anonymous and a person profile', async () => {
+ const sink = await run({ days: 5 });
+ const signups = eventsOf(sink, 'acme-web', 'signup_completed');
+ expect(signups.length).toBeGreaterThan(0);
+ const profileIds = new Set(
+ sink.profiles
+ .filter((profile) => profile.project_id === 'acme-web')
+ .map((profile) => profile.id)
+ );
+ for (const signup of signups) {
+ expect(signup.profile_id).not.toBe(signup.device_id);
+ expect(profileIds.has(signup.profile_id)).toBe(true);
+ expect(profileIds.has(signup.device_id)).toBe(true);
+ }
+ });
+});
+
+describe('funnels', () => {
+ /** Within-session funnels link by session; the SaaS activation funnel spans visits, so by person. */
+ const FUNNELS: Record<
+ string,
+ { by: 'session_id' | 'profile_id'; steps: string[] }
+ > = {
+ 'acme-shop': {
+ by: 'session_id',
+ steps: [
+ 'product_viewed',
+ 'add_to_cart',
+ 'checkout_started',
+ 'shipping_info_added',
+ 'payment_info_added',
+ 'purchase',
+ ],
+ },
+ 'acme-saas': {
+ by: 'profile_id',
+ steps: [
+ 'signup_completed',
+ 'project_created',
+ 'sdk_installed',
+ 'first_event_received',
+ ],
+ },
+ 'acme-app': {
+ by: 'profile_id',
+ steps: [
+ 'app_opened',
+ 'signup_completed',
+ 'notification_permission',
+ 'post_created',
+ ],
+ },
+ 'acme-web': {
+ by: 'session_id',
+ steps: [
+ 'screen_view',
+ 'plan_selected',
+ 'signup_started',
+ 'signup_completed',
+ ],
+ },
+ };
+
+ test('each step is reached by fewer users than the one before, and the last by some', async () => {
+ const sink = await run({ days: 10, sessionsPerDay: 300 });
+ for (const [projectId, { by, steps }] of Object.entries(FUNNELS)) {
+ // Funnel semantics: a user counts for a step only having done every step before it.
+ let reached: Set<string> | null = null;
+ const counts: number[] = [];
+ for (const step of steps) {
+ const users = new Set(
+ eventsOf(sink, projectId, step).map((event) => event[by])
+ );
+ const previous: Set<string> = reached ?? users;
+ reached = new Set([...previous].filter((user) => users.has(user)));
+ counts.push(reached.size);
+ }
+ expect(counts.at(-1)).toBeGreaterThan(0);
+ expect(counts.at(-1)).toBeLessThan(counts[0] ?? 0);
+ }
+ });
+
+ test('purchases carry breakdown-worthy properties and revenue', async () => {
+ const sink = await run({ days: 10, sessionsPerDay: 300 });
+ const purchases = eventsOf(sink, 'acme-shop', 'purchase');
+ expect(purchases.length).toBeGreaterThan(5);
+ const payments = new Set(
+ purchases.map((event) => event.properties.payment)
+ );
+ expect(payments.size).toBeGreaterThan(1);
+ expect(
+ purchases.every((event) => event.properties.order_id?.startsWith('ord_'))
+ ).toBe(true);
+ const revenue = eventsOf(sink, 'acme-shop', 'revenue');
+ expect(revenue.length).toBe(purchases.length);
+ expect(
+ revenue.every(
+ (event) => Number.isInteger(event.revenue) && (event.revenue ?? 0) > 0
+ )
+ ).toBe(true);
+ });
+});
diff --git a/packages/seed/src/generator.ts b/packages/seed/src/generator.ts
new file mode 100644
index 000000000..222de45c0
--- /dev/null
+++ b/packages/seed/src/generator.ts
@@ -0,0 +1,229 @@
+// Walks every project day by day, hour by hour, and turns each planned
+// session into rows. Sessions are produced in time order per project so the
+// returning-visitor pool never hands out a visitor from the future.
+
+import type { Archetype } from './archetypes/archetype';
+import { Journey } from './archetypes/archetype';
+import { CAMPAIGNS, type Campaign } from './data/referrers';
+import { Rng } from './rng';
+import { buildSessionRows } from './rows';
+import type { RowSink } from './sink';
+import { TrafficModel } from './traffic';
+import { World } from './world';
+
+export interface ProjectTarget {
+ archetype: Archetype;
+ projectId: string;
+}
+
+export interface GenerateConfig {
+ seed: number;
+ /** Sessions per day across all projects; each archetype takes its `trafficShare`. */
+ sessionsPerDay: number;
+ days: number;
+ now: Date;
+ variance: number;
+ projects: readonly ProjectTarget[];
+}
+
+export interface DayPlan {
+ archetype: Archetype;
+ date: Date;
+ sessions: number;
+}
+
+export interface ProjectStats {
+ projectId: string;
+ sessions: number;
+ events: number;
+ firstEventAt: Date | null;
+}
+
+export interface Progress {
+ day: string;
+ sessions: number;
+ events: number;
+}
+
+const MS_PER_DAY = 86_400_000;
+const MS_PER_HOUR = 3_600_000;
+const HOURS_PER_DAY = 24;
+const SECONDS_PER_HOUR = 3600;
+const PROGRESS_EVERY_SESSIONS = 5000;
+
+function utcMidnight(date: Date): Date {
+ return new Date(
+ Date.UTC(date.getUTCFullYear(), date.getUTCMonth(), date.getUTCDate())
+ );
+}
+
+function trafficModel(
+ config: GenerateConfig,
+ archetype: Archetype
+): TrafficModel {
+ return new TrafficModel(archetype.id, archetype.shape, {
+ seed: String(config.seed),
+ variance: config.variance,
+ now: config.now,
+ });
+}
+
+/** The days the run covers, oldest first, each ending before `now`. */
+function daysOf(config: GenerateConfig): Date[] {
+ const today = utcMidnight(config.now);
+ const days: Date[] = [];
+ for (let offset = config.days - 1; offset >= 0; offset--) {
+ days.push(new Date(today.getTime() - offset * MS_PER_DAY));
+ }
+ return days;
+}
+
+export function planDays(config: GenerateConfig): DayPlan[] {
+ const plan: DayPlan[] = [];
+ for (const { archetype } of config.projects) {
+ const model = trafficModel(config, archetype);
+ const base = config.sessionsPerDay * archetype.trafficShare;
+ for (const date of daysOf(config)) {
+ plan.push({
+ archetype,
+ date,
+ sessions: model.dayFor(date, base).visitors,
+ });
+ }
+ }
+ return plan;
+}
+
+function utmFor(campaign: Campaign): Record<string, string> {
+ return {
+ utm_source: campaign.source,
+ utm_medium: campaign.medium,
+ utm_campaign: campaign.name,
+ };
+}
+
+/** Session start times within one UTC day, following the archetype's local-hour curve. */
+function sessionStarts(
+ rng: Rng,
+ model: TrafficModel,
+ archetype: Archetype,
+ date: Date,
+ total: number,
+ now: Date
+): Date[] {
+ const weights = model.hourWeights(date);
+ const starts: Date[] = [];
+ for (let utcHour = 0; utcHour < HOURS_PER_DAY; utcHour++) {
+ const localHour =
+ (((utcHour + archetype.utcOffsetHours) % HOURS_PER_DAY) + HOURS_PER_DAY) %
+ HOURS_PER_DAY;
+ const count = rng.poisson(total * (weights[localHour] ?? 0));
+ for (let i = 0; i < count; i++) {
+ const start = new Date(
+ date.getTime() +
+ utcHour * MS_PER_HOUR +
+ rng.int(0, SECONDS_PER_HOUR - 1) * 1000
+ );
+ if (start.getTime() < now.getTime()) {
+ starts.push(start);
+ }
+ }
+ }
+ return starts.sort((a, b) => a.getTime() - b.getTime());
+}
+
+export async function generate(
+ config: GenerateConfig,
+ sink: RowSink,
+ onProgress: (progress: Progress) => void
+): Promise<ProjectStats[]> {
+ const stats: ProjectStats[] = [];
+ const days = daysOf(config);
+ let sessionsSinceProgress = 0;
+
+ for (const { archetype, projectId } of config.projects) {
+ const model = trafficModel(config, archetype);
+ const base = config.sessionsPerDay * archetype.trafficShare;
+ const world = new World(
+ archetype,
+ projectId,
+ days[0] ?? config.now,
+ (rows) => sink.addProfiles(rows)
+ );
+ const projectStats: ProjectStats = {
+ projectId,
+ sessions: 0,
+ events: 0,
+ firstEventAt: null,
+ };
+
+ for (const date of days) {
+ const dayKey = date.toISOString().slice(0, 10);
+ const rng = new Rng(`${config.seed}/${archetype.id}/${dayKey}`);
+ const planned = model.dayFor(date, base).visitors;
+
+ for (const [index, start] of sessionStarts(
+ rng,
+ model,
+ archetype,
+ date,
+ planned,
+ config.now
+ ).entries()) {
+ const visitor = world.pick(rng, start);
+ const referrer = rng.pick(archetype.referrers);
+ const utm = rng.chance(archetype.campaignChance)
+ ? utmFor(rng.pick(CAMPAIGNS))
+ : {};
+ // A returning identified user is recognised from the first event; a first visit starts anonymous.
+ const journey = new Journey(
+ rng,
+ start,
+ archetype.dwellMedianSeconds,
+ visitor.identified ? visitor.person : null
+ );
+ archetype.journey(rng, journey, visitor);
+ if (journey.events.length === 0) {
+ continue;
+ }
+
+ const rows = buildSessionRows(rng, {
+ id: rng.sessionId(),
+ projectId,
+ archetype,
+ visitor,
+ referrer,
+ utm,
+ events: journey.events,
+ });
+ world.touch(visitor, rows.endedAt, {
+ identified: rows.identified,
+ anonymousEvents: rows.anonymousEvents,
+ });
+ sink.addEvents(rows.events);
+ sink.addSession(rows.session);
+
+ projectStats.sessions += 1;
+ projectStats.events += rows.events.length;
+ projectStats.firstEventAt ??= start;
+ sessionsSinceProgress += 1;
+ if (sessionsSinceProgress >= PROGRESS_EVERY_SESSIONS || index === 0) {
+ sessionsSinceProgress = 0;
+ const counts = sink.counts();
+ onProgress({
+ day: `${archetype.id} ${dayKey}`,
+ sessions: counts.sessions,
+ events: counts.events,
+ });
+ }
+ await sink.flushFull();
+ }
+ }
+
+ world.drain();
+ stats.push(projectStats);
+ }
+
+ await sink.flush();
+ return stats;
+}
diff --git a/packages/seed/src/ids.ts b/packages/seed/src/ids.ts
new file mode 100644
index 000000000..5ca134c2d
--- /dev/null
+++ b/packages/seed/src/ids.ts
@@ -0,0 +1,23 @@
+import { createHash } from 'node:crypto';
+import { slug } from '@openpanel/shared';
+import type { Archetype } from './archetypes/archetype';
+import { formatUuid } from './rng';
+import { ID_NAMESPACE } from './seed.constants';
+
+function namespaceBytes(): Buffer {
+ return Buffer.from(ID_NAMESPACE.replace(/-/g, ''), 'hex');
+}
+
+/** RFC 4122 v5: the same name always yields the same uuid, so re-runs upsert. */
+export function deterministicUuid(name: string): string {
+ const digest = createHash('sha1')
+ .update(namespaceBytes())
+ .update(name)
+ .digest();
+ return formatUuid(digest.subarray(0, 16).toString('hex'), '5');
+}
+
+/** The same slug id the onboarding flow would mint for the project name. */
+export function projectIdFor(archetype: Archetype): string {
+ return slug(archetype.projectName);
+}
diff --git a/packages/seed/src/manifest.ts b/packages/seed/src/manifest.ts
new file mode 100644
index 000000000..5733710f8
--- /dev/null
+++ b/packages/seed/src/manifest.ts
@@ -0,0 +1,114 @@
+// `.seed.json`: what was seeded, for agents, the Playwright suite and whatever tooling
+// describes the worktree to them.
+
+import { writeFileSync } from 'node:fs';
+import type { PostgresSeed } from './postgres';
+import { SEED_USER } from './seed.constants';
+
+export interface SeedManifest {
+ seed: number;
+ size: string;
+ days: number;
+ sessionsPerDay: number;
+ range: { from: string; to: string };
+ seededAt: string;
+ user: { email: string; password: string };
+ organizationId: string;
+ rootClient: { id: string; secret: string };
+ /** The API this seed was made for (from `API_URL`), and the product's own MCP endpoint on it. */
+ apiUrl: string | null;
+ mcp: { url: string; token: string } | null;
+ projects: {
+ id: string;
+ name: string;
+ archetype: string;
+ clientId: string;
+ clientSecret: string;
+ sessions: number;
+ events: number;
+ /** How users are identified in this project: what retention can show. */
+ identity: string;
+ conversions: string[];
+ funnels: { name: string; steps: string[]; breakdowns: string[] }[];
+ }[];
+ totals: { sessions: number; events: number; profiles: number };
+}
+
+/** What the MCP settings page hands out: `<api>/mcp?token=base64(clientId:clientSecret)` for a read or root client. */
+function mcpEndpoint(
+ apiUrl: string | null,
+ client: { id: string; secret: string }
+): { url: string; token: string } | null {
+ if (!apiUrl) {
+ return null;
+ }
+ const token = Buffer.from(`${client.id}:${client.secret}`).toString('base64');
+ return { url: `${apiUrl.replace(/\/$/, '')}/mcp?token=${token}`, token };
+}
+
+export function buildManifest(input: {
+ seed: number;
+ size: string;
+ days: number;
+ sessionsPerDay: number;
+ from: Date;
+ to: Date;
+ postgres: PostgresSeed;
+ apiUrl: string | null;
+ perProject: Map<string, { sessions: number; events: number }>;
+ totals: { sessions: number; events: number; profiles: number };
+}): SeedManifest {
+ return {
+ seed: input.seed,
+ size: input.size,
+ days: input.days,
+ sessionsPerDay: input.sessionsPerDay,
+ range: { from: input.from.toISOString(), to: input.to.toISOString() },
+ seededAt: new Date().toISOString(),
+ user: { email: SEED_USER.email, password: SEED_USER.password },
+ organizationId: input.postgres.organizationId,
+ rootClient: input.postgres.rootClient,
+ apiUrl: input.apiUrl,
+ mcp: mcpEndpoint(input.apiUrl, input.postgres.rootClient),
+ projects: input.postgres.projects.map((project) => ({
+ id: project.id,
+ name: project.name,
+ archetype: project.archetype,
+ clientId: project.client.id,
+ clientSecret: project.client.secret,
+ sessions: input.perProject.get(project.id)?.sessions ?? 0,
+ events: input.perProject.get(project.id)?.events ?? 0,
+ identity: project.identity,
+ conversions: [...project.conversions],
+ funnels: project.funnels.map((funnel) => ({
+ name: funnel.name,
+ steps: [...funnel.steps],
+ breakdowns: [...funnel.breakdowns],
+ })),
+ })),
+ totals: input.totals,
+ };
+}
+
+export function writeManifest(path: string, manifest: SeedManifest): void {
+ writeFileSync(path, `${JSON.stringify(manifest, null, 2)}\n`);
+}
+
+export function describeManifest(manifest: SeedManifest): string {
+ const lines = [
+ `Seeded ${manifest.totals.sessions.toLocaleString()} sessions / ${manifest.totals.events.toLocaleString()} events / ${manifest.totals.profiles.toLocaleString()} profiles`,
+ `Range: ${manifest.range.from.slice(0, 10)} → ${manifest.range.to.slice(0, 10)} (${manifest.days} days, size ${manifest.size}, seed ${manifest.seed})`,
+ `Login: ${manifest.user.email} / ${manifest.user.password}`,
+ `Organization: ${manifest.organizationId}`,
+ 'Projects:',
+ ...manifest.projects.map(
+ (project) =>
+ ` ${project.name} (${project.archetype}) id ${project.id} client ${project.clientId} secret ${project.clientSecret}`
+ ),
+ `Root client: ${manifest.rootClient.id} secret ${manifest.rootClient.secret}`,
+ manifest.mcp
+ ? `MCP (streamable HTTP, root client): ${manifest.mcp.url}`
+ : 'MCP: set API_URL to get the endpoint',
+ ];
+ return lines.join('\n');
+}
diff --git a/packages/seed/src/model.ts b/packages/seed/src/model.ts
new file mode 100644
index 000000000..54feed5c8
--- /dev/null
+++ b/packages/seed/src/model.ts
@@ -0,0 +1,65 @@
+// The in-memory shapes the generator produces before they become ClickHouse rows.
+
+import type { parseUserAgent } from '@openpanel/shared/server';
+
+export type ParsedDevice = ReturnType<typeof parseUserAgent>;
+
+export interface Geo {
+ ip: string;
+ country: string;
+ region: string;
+ city: string;
+ latitude: number;
+ longitude: number;
+}
+
+export interface Person {
+ id: string;
+ firstName: string;
+ lastName: string;
+ email: string;
+ avatar: string;
+ properties: Record<string, string>;
+}
+
+export interface Referrer {
+ url: string;
+ name: string;
+ type: string;
+}
+
+/**
+ * The device id ingest mints from user agent + IP + a salt that rotates at
+ * midnight UTC: the same browser is one device within a UTC day and a new
+ * one the next. Anonymous analytics only ever sees these.
+ */
+export interface DeviceWindow {
+ id: string;
+ /** The UTC day this id is valid for. */
+ day: string;
+ firstSeen: Date;
+ lastSeen: Date;
+ /** Whether any event was sent anonymously from it, which is what leaves a profile behind. */
+ anonymousEvents: boolean;
+}
+
+/** A person (or an unidentified browser) that can come back. */
+export interface Visitor {
+ userAgent: string;
+ device: ParsedDevice;
+ geo: Geo;
+ person: Person | null;
+ /** `identify()` has been sent once, so later sessions start identified. */
+ identified: boolean;
+ window: DeviceWindow;
+ /** Days since the run's first day, for retention cohorts. */
+ cohortDay: number;
+ sessions: number;
+ firstSeen: Date;
+ lastSeen: Date;
+ /** What the worker snapshots onto the profile at each session boundary. */
+ lastPath: string;
+ lastReferrer: Referrer | null;
+ /** Archetype-specific memory (plan, cart, ...). */
+ state: Record<string, unknown>;
+}
diff --git a/packages/seed/src/postgres.ts b/packages/seed/src/postgres.ts
new file mode 100644
index 000000000..aacc1f2fb
--- /dev/null
+++ b/packages/seed/src/postgres.ts
@@ -0,0 +1,210 @@
+// The Postgres side: one login, one organization, one project per archetype
+// and a write client each, plus a root client for /manage. Mirrors what the
+// onboarding flow creates (onboarding.service.ts), with deterministic ids so a
+// re-run updates in place.
+
+import { hash as argon2Hash } from '@node-rs/argon2';
+import { db } from '@openpanel/db';
+import { hashPassword } from '@openpanel/shared/server';
+import type { Archetype } from './archetypes/archetype';
+import { deterministicUuid, projectIdFor } from './ids';
+import type { Rng } from './rng';
+import {
+ ORGANIZATION_ADMIN_ROLE,
+ SEED_ORGANIZATION,
+ SEED_USER,
+ TRIAL_DAYS,
+ TRIAL_EVENTS_LIMIT,
+} from './seed.constants';
+
+const MS_PER_DAY = 86_400_000;
+const SECRET_HEX_CHARS = 20;
+/** Sign-in accepts `$argon2` hashes made with these parameters; copied from core's auth `hashPassword`. */
+const ARGON2_OPTIONS = {
+ memoryCost: 19_456,
+ timeCost: 2,
+ outputLen: 32,
+ parallelism: 1,
+} as const;
+
+export interface SeededClient {
+ id: string;
+ secret: string;
+}
+
+export interface SeededProject {
+ id: string;
+ name: string;
+ archetype: Archetype['id'];
+ client: SeededClient;
+ identity: string;
+ conversions: readonly string[];
+ funnels: Archetype['funnels'];
+}
+
+export interface PostgresSeed {
+ userId: string;
+ organizationId: string;
+ rootClient: SeededClient;
+ projects: SeededProject[];
+}
+
+function newSecret(rng: Rng): string {
+ return `sec_${rng.hex(SECRET_HEX_CHARS)}`;
+}
+
+async function upsertUser(): Promise<string> {
+ const user = await db.user.upsert({
+ where: { email: SEED_USER.email },
+ create: {
+ id: SEED_USER.id,
+ email: SEED_USER.email,
+ firstName: SEED_USER.firstName,
+ lastName: SEED_USER.lastName,
+ },
+ update: {
+ firstName: SEED_USER.firstName,
+ lastName: SEED_USER.lastName,
+ deletedAt: null,
+ },
+ });
+ const password = await argon2Hash(SEED_USER.password, ARGON2_OPTIONS);
+ const account = await db.account.findFirst({
+ where: { userId: user.id, provider: 'email' },
+ });
+ if (account) {
+ await db.account.update({ where: { id: account.id }, data: { password } });
+ } else {
+ await db.account.create({
+ data: { userId: user.id, provider: 'email', email: user.email, password },
+ });
+ }
+ return user.id;
+}
+
+async function upsertOrganization(userId: string): Promise<string> {
+ const organization = await db.organization.upsert({
+ where: { id: SEED_ORGANIZATION.id },
+ create: {
+ id: SEED_ORGANIZATION.id,
+ name: SEED_ORGANIZATION.name,
+ createdByUserId: userId,
+ subscriptionStatus: 'trialing',
+ subscriptionEndsAt: new Date(Date.now() + TRIAL_DAYS * MS_PER_DAY),
+ subscriptionPeriodEventsLimit: TRIAL_EVENTS_LIMIT,
+ timezone: 'UTC',
+ onboarding: '',
+ },
+ update: { name: SEED_ORGANIZATION.name, deleteAt: null },
+ });
+ await db.member.upsert({
+ where: {
+ organizationId_userId: { organizationId: organization.id, userId },
+ },
+ create: {
+ organizationId: organization.id,
+ userId,
+ email: SEED_USER.email,
+ role: ORGANIZATION_ADMIN_ROLE,
+ },
+ update: { role: ORGANIZATION_ADMIN_ROLE },
+ });
+ return organization.id;
+}
+
+/** Ingest verifies client secrets with scrypt (`verifyPassword`), so that is what the seed stores. */
+async function upsertClient(
+ rng: Rng,
+ input: {
+ name: string;
+ organizationId: string;
+ projectId: string | null;
+ type: 'write' | 'root';
+ }
+): Promise<SeededClient> {
+ const id = deterministicUuid(`client:${input.projectId ?? 'root'}`);
+ const secret = newSecret(rng);
+ await db.client.upsert({
+ where: { id },
+ create: {
+ id,
+ name: input.name,
+ organizationId: input.organizationId,
+ projectId: input.projectId,
+ type: input.type,
+ secret: await hashPassword(secret),
+ },
+ update: { name: input.name, secret: await hashPassword(secret) },
+ });
+ return { id, secret };
+}
+
+async function upsertProject(
+ rng: Rng,
+ organizationId: string,
+ archetype: Archetype
+): Promise<SeededProject> {
+ const id = projectIdFor(archetype);
+ await db.project.upsert({
+ where: { id },
+ create: {
+ id,
+ name: archetype.projectName,
+ organizationId,
+ types: archetype.types,
+ domain: archetype.domain,
+ cors: archetype.origin ? ['*'] : [],
+ },
+ update: {
+ name: archetype.projectName,
+ types: archetype.types,
+ domain: archetype.domain,
+ deleteAt: null,
+ },
+ });
+ const client = await upsertClient(rng, {
+ name: `${archetype.projectName} Client`,
+ organizationId,
+ projectId: id,
+ type: 'write',
+ });
+ return {
+ id,
+ name: archetype.projectName,
+ archetype: archetype.id,
+ client,
+ identity: archetype.identity,
+ conversions: archetype.conversions,
+ funnels: archetype.funnels,
+ };
+}
+
+export async function seedPostgres(
+ rng: Rng,
+ archetypes: readonly Archetype[]
+): Promise<PostgresSeed> {
+ const userId = await upsertUser();
+ const organizationId = await upsertOrganization(userId);
+ const rootClient = await upsertClient(rng, {
+ name: 'Seed root client',
+ organizationId,
+ projectId: null,
+ type: 'root',
+ });
+ const projects: SeededProject[] = [];
+ for (const archetype of archetypes) {
+ projects.push(await upsertProject(rng, organizationId, archetype));
+ }
+ return { userId, organizationId, rootClient, projects };
+}
+
+/** What the ingest path maintains on the project row once events arrive. */
+export async function recordProjectStats(
+ projectId: string,
+ stats: { eventsCount: number; firstEventAt: Date | null }
+): Promise<void> {
+ await db.project.update({
+ where: { id: projectId },
+ data: { eventsCount: stats.eventsCount, firstEventAt: stats.firstEventAt },
+ });
+}
diff --git a/packages/seed/src/reset.ts b/packages/seed/src/reset.ts
new file mode 100644
index 000000000..c05e012e2
--- /dev/null
+++ b/packages/seed/src/reset.ts
@@ -0,0 +1,37 @@
+// Removes what an earlier run seeded for the same projects. Postgres rows are
+// upserted in place, so only ClickHouse needs clearing; the materialized views
+// are not rebuilt from `events` on delete and must be cleared explicitly.
+
+import { ch } from '@openpanel/db';
+
+const TABLES_WITH_PROJECT_ROWS = ['events', 'sessions', 'profiles'] as const;
+
+const MATERIALIZED_VIEWS_WITH_PROJECT_ROWS = [
+ 'dau_mv',
+ 'distinct_event_names_mv',
+ 'event_property_values_mv',
+ 'cohort_events_mv',
+ 'event_profile_summary_mv',
+ 'event_property_profile_summary_mv',
+] as const;
+
+function quoted(value: string): string {
+ return `'${value.replace(/'/g, "''")}'`;
+}
+
+export async function resetProjects(
+ projectIds: readonly string[]
+): Promise<void> {
+ const list = projectIds.map(quoted).join(', ');
+ for (const table of TABLES_WITH_PROJECT_ROWS) {
+ await ch.command({
+ query: `DELETE FROM ${table} WHERE project_id IN (${list})`,
+ });
+ }
+ for (const view of MATERIALIZED_VIEWS_WITH_PROJECT_ROWS) {
+ await ch.command({
+ query: `ALTER TABLE ${view} DELETE WHERE project_id IN (${list})`,
+ clickhouse_settings: { mutations_sync: '2' },
+ });
+ }
+}
diff --git a/packages/seed/src/rng.ts b/packages/seed/src/rng.ts
new file mode 100644
index 000000000..ea8a5f4be
--- /dev/null
+++ b/packages/seed/src/rng.ts
@@ -0,0 +1,185 @@
+// A small deterministic PRNG. Every random choice in the seed goes through an
+// `Rng` derived from the run's seed, so the same flags produce the same rows.
+
+/* biome-ignore-all lint/suspicious/noBitwiseOperators: sfc32 is defined in 32-bit integer arithmetic */
+
+import { createHash } from 'node:crypto';
+
+const UUID_VARIANT_NIBBLES = ['8', '9', 'a', 'b'] as const;
+
+/** 32 hex chars into RFC 4122 shape, stamping the version and variant nibbles. */
+export function formatUuid(hex: string, version: '4' | '5'): string {
+ const variant =
+ UUID_VARIANT_NIBBLES[
+ Number.parseInt(hex[16] ?? '0', 16) % UUID_VARIANT_NIBBLES.length
+ ] ?? '8';
+ return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${version}${hex.slice(13, 16)}-${variant}${hex.slice(17, 20)}-${hex.slice(20, 32)}`;
+}
+
+export interface Weighted<T> {
+ value: T;
+ weight: number;
+}
+
+const UINT32_RANGE = 4_294_967_296;
+const POISSON_NORMAL_APPROXIMATION_THRESHOLD = 60;
+
+function seedWords(seed: string): [number, number, number, number] {
+ const digest = createHash('sha256').update(seed).digest();
+ return [
+ digest.readUInt32LE(0),
+ digest.readUInt32LE(4),
+ digest.readUInt32LE(8),
+ digest.readUInt32LE(12),
+ ];
+}
+
+/** sfc32: fast, 128-bit state, good enough statistics for synthetic traffic. */
+export class Rng {
+ private a: number;
+ private b: number;
+ private c: number;
+ private d: number;
+ private spareNormal: number | null = null;
+
+ constructor(private readonly seed: string) {
+ [this.a, this.b, this.c, this.d] = seedWords(seed);
+ for (let i = 0; i < 12; i++) {
+ this.next();
+ }
+ }
+
+ /** A child generator whose stream is independent of the parent's position. */
+ fork(...salt: (string | number)[]): Rng {
+ return new Rng(`${this.seed}/${salt.join('/')}`);
+ }
+
+ private next(): number {
+ this.a >>>= 0;
+ this.b >>>= 0;
+ this.c >>>= 0;
+ this.d >>>= 0;
+ let t = (this.a + this.b) | 0;
+ this.a = this.b ^ (this.b >>> 9);
+ this.b = (this.c + (this.c << 3)) | 0;
+ this.c = (this.c << 21) | (this.c >>> 11);
+ this.d = (this.d + 1) | 0;
+ t = (t + this.d) | 0;
+ this.c = (this.c + t) | 0;
+ return t >>> 0;
+ }
+
+ /** Uniform in [0, 1). */
+ float(): number {
+ return this.next() / UINT32_RANGE;
+ }
+
+ /** Uniform integer in [low, high]. */
+ int(low: number, high: number): number {
+ if (high <= low) {
+ return low;
+ }
+ return low + Math.floor(this.float() * (high - low + 1));
+ }
+
+ chance(probability: number): boolean {
+ return this.float() < probability;
+ }
+
+ one<T>(items: readonly T[]): T {
+ const item = items[Math.floor(this.float() * items.length)];
+ if (item === undefined) {
+ throw new Error('Rng.one called with an empty list');
+ }
+ return item;
+ }
+
+ pick<T>(items: readonly Weighted<T>[]): T {
+ let total = 0;
+ for (const item of items) {
+ total += item.weight;
+ }
+ let r = this.float() * total;
+ for (const item of items) {
+ r -= item.weight;
+ if (r <= 0) {
+ return item.value;
+ }
+ }
+ const last = items.at(-1);
+ if (last === undefined) {
+ throw new Error('Rng.pick called with an empty list');
+ }
+ return last.value;
+ }
+
+ /** Standard normal via Box–Muller. */
+ normal(): number {
+ if (this.spareNormal !== null) {
+ const spare = this.spareNormal;
+ this.spareNormal = null;
+ return spare;
+ }
+ let u = 0;
+ while (u === 0) {
+ u = this.float();
+ }
+ const v = this.float();
+ const radius = Math.sqrt(-2 * Math.log(u));
+ this.spareNormal = radius * Math.sin(2 * Math.PI * v);
+ return radius * Math.cos(2 * Math.PI * v);
+ }
+
+ /** Right-skewed seconds around a median: most dwell times are short, a few are very long. */
+ logNormalSeconds(median: number, sigma: number): number {
+ return median * Math.exp(this.normal() * sigma);
+ }
+
+ poisson(lambda: number): number {
+ if (lambda <= 0) {
+ return 0;
+ }
+ if (lambda > POISSON_NORMAL_APPROXIMATION_THRESHOLD) {
+ return Math.max(
+ 0,
+ Math.round(lambda + Math.sqrt(lambda) * this.normal())
+ );
+ }
+ const limit = Math.exp(-lambda);
+ let count = 0;
+ let product = 1;
+ for (;;) {
+ product *= this.float();
+ if (product <= limit) {
+ return count;
+ }
+ count++;
+ }
+ }
+
+ bytes(length: number): Buffer {
+ const out = Buffer.alloc(length);
+ for (let offset = 0; offset < length; offset += 4) {
+ const word = this.next();
+ for (let b = 0; b < 4 && offset + b < length; b++) {
+ out[offset + b] = (word >>> (b * 8)) & 0xff;
+ }
+ }
+ return out;
+ }
+
+ hex(chars: number): string {
+ return this.bytes(Math.ceil(chars / 2))
+ .toString('hex')
+ .slice(0, chars);
+ }
+
+ uuid(): string {
+ return formatUuid(this.bytes(16).toString('hex'), '4');
+ }
+
+ /** The shape the ingest mints for session ids: 16 bytes, base64url, 22 chars. */
+ sessionId(): string {
+ return this.bytes(16).toString('base64url');
+ }
+}
diff --git a/packages/seed/src/rows.ts b/packages/seed/src/rows.ts
new file mode 100644
index 000000000..ee5cdbdcb
--- /dev/null
+++ b/packages/seed/src/rows.ts
@@ -0,0 +1,363 @@
+// From a journey to ClickHouse rows, shaped the way the worker shapes them:
+// `createEvent` for events, the session buffer's `newSession` / `extendSession`
+// for the one sessions row, and the ingest handler's synthesized
+// `session_start` / `session_end`. Kept in step with core by hand, on purpose:
+// the seed stays out of the backend's dependency graph.
+
+import { parsePath, toDots } from '@openpanel/shared';
+import type { Archetype, JourneyEvent } from './archetypes/archetype';
+import {
+ type ClickhouseEventRow,
+ type ClickhouseProfileRow,
+ type ClickhouseSessionRow,
+ toClickhouseDateTime,
+} from './clickhouse-rows';
+import { sessionReferrer } from './data/referrers';
+import type { Referrer, Visitor } from './model';
+import type { Rng } from './rng';
+import {
+ SESSION_END_OFFSET_MS,
+ SESSION_START_OFFSET_MS,
+} from './seed.constants';
+
+export interface SessionInput {
+ id: string;
+ projectId: string;
+ archetype: Archetype;
+ visitor: Visitor;
+ referrer: Referrer | null;
+ utm: Record<string, string>;
+ events: readonly JourneyEvent[];
+}
+
+export interface SessionRows {
+ events: ClickhouseEventRow[];
+ session: ClickhouseSessionRow;
+ endedAt: Date;
+ /** Whether the session ended identified, and whether any event went out anonymously. */
+ identified: boolean;
+ anonymousEvents: boolean;
+}
+
+/** What the ingest handler derives from one SDK request before the rows are shaped. */
+interface EventPayload {
+ name: string;
+ profileId: string;
+ createdAt: Date;
+ path: string;
+ origin: string;
+ properties: Record<string, unknown>;
+ duration: number;
+ revenue?: number;
+}
+
+const UTM_KEYS = [
+ 'utm_medium',
+ 'utm_source',
+ 'utm_campaign',
+ 'utm_content',
+ 'utm_term',
+] as const;
+
+/** The SDK sends `__path` as a full URL on the web and a bare screen name in a native app. */
+function eventUrl(
+ archetype: Archetype,
+ event: JourneyEvent,
+ utm: Record<string, string>,
+ isLanding: boolean
+): string {
+ if (archetype.deviceClass === 'native') {
+ return event.path;
+ }
+ const query =
+ isLanding && Object.keys(utm).length > 0
+ ? `?${new URLSearchParams(utm).toString()}`
+ : '';
+ return `${archetype.origin}${event.path}${query}`;
+}
+
+function utmOf(
+ properties: Record<string, unknown>,
+ key: (typeof UTM_KEYS)[number]
+): string {
+ const query = properties.__query as Record<string, unknown> | undefined;
+ const value = query?.[key];
+ return value ? String(value) : '';
+}
+
+/** `createEvent`: every column, with the anonymous fallback `profile_id = device_id`. */
+function eventRow(
+ input: SessionInput,
+ referrer: Referrer,
+ payload: EventPayload,
+ id: string
+): ClickhouseEventRow {
+ const { visitor, archetype } = input;
+ return {
+ id,
+ name: payload.name,
+ device_id: visitor.window.id,
+ profile_id: payload.profileId || visitor.window.id,
+ project_id: input.projectId,
+ session_id: input.id,
+ properties: toDots(payload.properties),
+ path: payload.path,
+ origin: payload.origin,
+ created_at: toClickhouseDateTime(payload.createdAt),
+ country: visitor.geo.country,
+ city: visitor.geo.city,
+ region: visitor.geo.region,
+ longitude: visitor.geo.longitude,
+ latitude: visitor.geo.latitude,
+ os: visitor.device.os ?? '',
+ os_version: visitor.device.osVersion ?? '',
+ browser: visitor.device.browser ?? '',
+ browser_version: visitor.device.browserVersion ?? '',
+ device: visitor.device.device ?? '',
+ brand: visitor.device.brand ?? '',
+ model: visitor.device.model ?? '',
+ duration: payload.duration,
+ referrer: referrer.url,
+ referrer_name: referrer.name,
+ referrer_type: referrer.type,
+ imported_at: null,
+ inserted_at: toClickhouseDateTime(new Date()),
+ sdk_name: archetype.sdk.name,
+ sdk_version: archetype.sdk.version,
+ revenue: payload.revenue,
+ groups: [],
+ };
+}
+
+/** The session buffer's `newSession`. */
+function openSession(
+ input: SessionInput,
+ referrer: Referrer,
+ first: EventPayload
+): ClickhouseSessionRow {
+ const { visitor } = input;
+ const createdAt = toClickhouseDateTime(first.createdAt);
+ const isScreenView = first.name === 'screen_view';
+ return {
+ id: input.id,
+ project_id: input.projectId,
+ device_id: visitor.window.id,
+ profile_id: first.profileId || visitor.window.id,
+ is_bounce: true,
+ created_at: createdAt,
+ ended_at: createdAt,
+ event_count: isScreenView ? 0 : 1,
+ screen_view_count: isScreenView ? 1 : 0,
+ entry_path: first.path,
+ entry_origin: first.origin,
+ exit_path: first.path,
+ exit_origin: first.origin,
+ revenue: first.name === 'revenue' ? (first.revenue ?? 0) : 0,
+ referrer: referrer.url,
+ referrer_name: referrer.name,
+ referrer_type: referrer.type,
+ os: visitor.device.os ?? '',
+ os_version: visitor.device.osVersion ?? '',
+ browser: visitor.device.browser ?? '',
+ browser_version: visitor.device.browserVersion ?? '',
+ device: visitor.device.device ?? '',
+ brand: visitor.device.brand ?? '',
+ model: visitor.device.model ?? '',
+ country: visitor.geo.country,
+ region: visitor.geo.region,
+ city: visitor.geo.city,
+ longitude: visitor.geo.longitude,
+ latitude: visitor.geo.latitude,
+ duration: first.duration,
+ utm_medium: utmOf(first.properties, 'utm_medium'),
+ utm_source: utmOf(first.properties, 'utm_source'),
+ utm_campaign: utmOf(first.properties, 'utm_campaign'),
+ utm_content: utmOf(first.properties, 'utm_content'),
+ utm_term: utmOf(first.properties, 'utm_term'),
+ groups: [],
+ sign: 1,
+ version: 1,
+ };
+}
+
+/** The session buffer's `extendSession`, for events that arrive in order. */
+function extendSession(
+ session: ClickhouseSessionRow,
+ payload: EventPayload,
+ deviceId: string
+): void {
+ session.version += 1;
+ session.ended_at = toClickhouseDateTime(payload.createdAt);
+ if (payload.path) {
+ session.exit_path = payload.path;
+ }
+ if (payload.origin) {
+ session.exit_origin = payload.origin;
+ }
+ if (!session.entry_path && payload.path) {
+ session.entry_path = payload.path;
+ }
+ if (!session.entry_origin && payload.origin) {
+ session.entry_origin = payload.origin;
+ }
+ session.duration =
+ payload.createdAt.getTime() - new Date(`${session.created_at}Z`).getTime();
+ if (payload.name === 'revenue') {
+ session.revenue += payload.revenue ?? 0;
+ }
+ if (payload.name === 'screen_view' && payload.path) {
+ session.screen_view_count += 1;
+ } else {
+ session.event_count += 1;
+ }
+ if (session.screen_view_count > 1) {
+ session.is_bounce = false;
+ }
+ if (payload.profileId && payload.profileId !== deviceId) {
+ session.profile_id = payload.profileId;
+ }
+}
+
+export function buildSessionRows(rng: Rng, input: SessionInput): SessionRows {
+ const { archetype, visitor, utm } = input;
+ const payloads: EventPayload[] = [];
+ let referrer: Referrer = { url: '', name: '', type: '' };
+
+ for (const [index, event] of input.events.entries()) {
+ const isLanding = index === 0;
+ const { path, origin, query, hash } = parsePath(
+ eventUrl(archetype, event, utm, isLanding)
+ );
+ if (isLanding) {
+ referrer = sessionReferrer(input.referrer, query);
+ }
+ payloads.push({
+ name: event.name,
+ profileId: event.person?.id ?? '',
+ createdAt: event.at,
+ path,
+ origin,
+ // `__path` / `__referrer` never reach the row; `__hash` and `__query` are added by the handler.
+ properties: {
+ ...event.properties,
+ __title: event.title || undefined,
+ __hash: hash,
+ __query: query,
+ },
+ duration: 0,
+ revenue: event.name === 'revenue' ? event.revenue : undefined,
+ });
+ }
+
+ const first = payloads[0];
+ const last = payloads.at(-1);
+ if (!(first && last)) {
+ throw new Error('A session needs at least one event');
+ }
+
+ const session = openSession(input, referrer, first);
+ for (const payload of payloads.slice(1)) {
+ extendSession(session, payload, visitor.window.id);
+ }
+
+ const endedAt = new Date(last.createdAt.getTime());
+ const sessionStart: EventPayload = {
+ ...first,
+ name: 'session_start',
+ createdAt: new Date(first.createdAt.getTime() - SESSION_START_OFFSET_MS),
+ };
+ const sessionEnd: EventPayload = {
+ ...last,
+ name: 'session_end',
+ properties: { ...last.properties, __bounce: session.is_bounce },
+ duration: session.duration,
+ path: session.exit_path,
+ createdAt: new Date(endedAt.getTime() + SESSION_END_OFFSET_MS),
+ profileId: session.profile_id,
+ };
+
+ visitor.lastPath = first.path;
+ visitor.lastReferrer = referrer;
+
+ return {
+ events: [sessionStart, ...payloads, sessionEnd].map((payload) =>
+ eventRow(input, referrer, payload, rng.uuid())
+ ),
+ session,
+ endedAt,
+ identified: last.profileId !== '',
+ anonymousEvents: payloads.some((payload) => payload.profileId === ''),
+ };
+}
+
+/** The properties `createEvent` snapshots onto a profile at a session boundary. */
+function profileSnapshot(visitor: Visitor): Record<string, string> {
+ const snapshot: Record<string, string | number | null | undefined> = {
+ path: visitor.lastPath,
+ country: visitor.geo.country,
+ city: visitor.geo.city,
+ region: visitor.geo.region,
+ longitude: visitor.geo.longitude,
+ latitude: visitor.geo.latitude,
+ os: visitor.device.os,
+ os_version: visitor.device.osVersion,
+ browser: visitor.device.browser,
+ browser_version: visitor.device.browserVersion,
+ device: visitor.device.device,
+ brand: visitor.device.brand,
+ model: visitor.device.model,
+ referrer: visitor.lastReferrer?.url,
+ referrer_name: visitor.lastReferrer?.name,
+ referrer_type: visitor.lastReferrer?.type,
+ };
+ const properties: Record<string, string> = {};
+ for (const [key, value] of Object.entries(snapshot)) {
+ if (value !== undefined && value !== null && value !== '') {
+ properties[key] = String(value);
+ }
+ }
+ return properties;
+}
+
+/** The anonymous profile a device window leaves behind (`profile_id = device_id`). */
+export function deviceProfileRow(
+ visitor: Visitor,
+ projectId: string
+): ClickhouseProfileRow {
+ return {
+ id: visitor.window.id,
+ first_name: '',
+ last_name: '',
+ email: '',
+ avatar: '',
+ is_external: false,
+ properties: profileSnapshot(visitor),
+ project_id: projectId,
+ created_at: toClickhouseDateTime(visitor.window.firstSeen),
+ last_seen_at: toClickhouseDateTime(visitor.window.lastSeen),
+ groups: [],
+ };
+}
+
+/** The identified person's profile: name, email, avatar and the archetype's person properties. */
+export function personProfileRow(
+ visitor: Visitor,
+ projectId: string
+): ClickhouseProfileRow {
+ if (!visitor.person) {
+ throw new Error('personProfileRow needs an identified visitor');
+ }
+ return {
+ id: visitor.person.id,
+ first_name: visitor.person.firstName,
+ last_name: visitor.person.lastName,
+ email: visitor.person.email,
+ avatar: visitor.person.avatar,
+ is_external: true,
+ properties: { ...profileSnapshot(visitor), ...visitor.person.properties },
+ project_id: projectId,
+ created_at: toClickhouseDateTime(visitor.firstSeen),
+ last_seen_at: toClickhouseDateTime(visitor.lastSeen),
+ groups: [],
+ };
+}
diff --git a/packages/seed/src/seed.constants.ts b/packages/seed/src/seed.constants.ts
new file mode 100644
index 000000000..51b211d8b
--- /dev/null
+++ b/packages/seed/src/seed.constants.ts
@@ -0,0 +1,44 @@
+export const DEFAULT_SEED = 42;
+
+/** The login every seeded database has; also written to `.seed.json`. */
+export const SEED_USER = {
+ id: 'user_seed_admin',
+ email: '[email protected]',
+ password: 'openpanel',
+ firstName: 'Seed',
+ lastName: 'Admin',
+} as const;
+
+export const SEED_ORGANIZATION = { id: 'acme', name: 'Acme' } as const;
+
+export const ORGANIZATION_ADMIN_ROLE = 'org:admin';
+
+/** Mirrors the trial the onboarding flow grants, so billing gates behave the same. */
+export const TRIAL_DAYS = 365;
+export const TRIAL_EVENTS_LIMIT = 100_000_000;
+
+export const SIZE_PRESETS = {
+ small: { days: 14, sessionsPerDay: 300 },
+ medium: { days: 30, sessionsPerDay: 2000 },
+ large: { days: 90, sessionsPerDay: 12_000 },
+ xl: { days: 365, sessionsPerDay: 30_000 },
+} as const;
+
+export type SizePreset = keyof typeof SIZE_PRESETS;
+
+export const DEFAULT_SIZE: SizePreset = 'small';
+
+/** Rows per ClickHouse insert; JSONEachRow at this size stays well under a second. */
+export const INSERT_BATCH_ROWS = 50_000;
+
+/** uuid v5 namespace for every deterministic id the seed mints. */
+export const ID_NAMESPACE = '6f5f2b0e-3f7c-4a3e-9c1e-2d0b4a6e8c11';
+
+/** Where `session_start` / `session_end` sit relative to the real events (mirrors the worker). */
+export const SESSION_START_OFFSET_MS = 100;
+export const SESSION_END_OFFSET_MS = 1000;
+
+/** Returning-visitor pool per project; evicted visitors never come back. */
+export const VISITOR_POOL_SIZE = 20_000;
+
+export const MANIFEST_FILE = '.seed.json';
diff --git a/packages/seed/src/seed.integration.test.ts b/packages/seed/src/seed.integration.test.ts
new file mode 100644
index 000000000..8b648a749
--- /dev/null
+++ b/packages/seed/src/seed.integration.test.ts
@@ -0,0 +1,157 @@
+// Seeds two days into the suite's isolated databases and checks what the
+// dashboard's queries rely on. Pins the env before any database import, the
+// same way test/bun-preload.ts does for @openpanel/db.
+
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import {
+ TEST_CLICKHOUSE_URL,
+ TEST_DATABASE_URL,
+ TEST_REDIS_URL,
+} from '../../../test/databases';
+
+process.env.DATABASE_URL = TEST_DATABASE_URL;
+process.env.CLICKHOUSE_URL = TEST_CLICKHOUSE_URL;
+process.env.REDIS_URL = TEST_REDIS_URL;
+process.env.SELF_HOSTED = 'true';
+
+const { bootstrapTestDatabases } = await import(
+ '../../../test/bootstrap-databases'
+);
+const { ch, db } = await import('@openpanel/db');
+const { ARCHETYPES } = await import('./archetypes');
+const { generate } = await import('./generator');
+const { projectIdFor } = await import('./ids');
+const { seedPostgres } = await import('./postgres');
+const { resetProjects } = await import('./reset');
+const { Rng } = await import('./rng');
+const { SEED_ORGANIZATION, SEED_USER } = await import('./seed.constants');
+const { ClickhouseSink } = await import('./sink');
+
+const ignoreProgress = () => undefined;
+const SEED = 3;
+const DAYS = 2;
+const SESSIONS_PER_DAY = 120;
+const PROJECT_IDS = ARCHETYPES.map(projectIdFor);
+
+async function count(query: string): Promise<number> {
+ const result = await ch.query({ query, format: 'JSONEachRow' });
+ const rows = (await result.json()) as { value: string | number }[];
+ return Number(rows[0]?.value ?? 0);
+}
+
+function inList(): string {
+ return PROJECT_IDS.map((id) => `'${id}'`).join(', ');
+}
+
+describe('seed against real databases', () => {
+ beforeAll(async () => {
+ await bootstrapTestDatabases();
+ await seedPostgres(new Rng(`${SEED}/postgres`), ARCHETYPES);
+ await resetProjects(PROJECT_IDS);
+ const sink = new ClickhouseSink();
+ await generate(
+ {
+ seed: SEED,
+ sessionsPerDay: SESSIONS_PER_DAY,
+ days: DAYS,
+ now: new Date(),
+ variance: 1,
+ projects: ARCHETYPES.map((archetype) => ({
+ archetype,
+ projectId: projectIdFor(archetype),
+ })),
+ },
+ sink,
+ ignoreProgress
+ );
+ });
+
+ afterAll(async () => {
+ await resetProjects(PROJECT_IDS);
+ await db.organization
+ .delete({ where: { id: SEED_ORGANIZATION.id } })
+ .catch(() => null);
+ await db.user.delete({ where: { id: SEED_USER.id } }).catch(() => null);
+ await db.$disconnect();
+ });
+
+ test('postgres has the login, organization, projects and clients', async () => {
+ const user = await db.user.findUnique({
+ where: { email: SEED_USER.email },
+ include: { accounts: true, membership: true },
+ });
+ expect(
+ user?.accounts.some(
+ (account) =>
+ account.provider === 'email' &&
+ account.password?.startsWith('$argon2')
+ )
+ ).toBe(true);
+ expect(
+ user?.membership.some(
+ (member) => member.organizationId === SEED_ORGANIZATION.id
+ )
+ ).toBe(true);
+ const projects = await db.project.findMany({
+ where: { organizationId: SEED_ORGANIZATION.id },
+ include: { clients: true },
+ });
+ expect(projects.map((project) => project.id).sort()).toEqual(
+ [...PROJECT_IDS].sort()
+ );
+ for (const project of projects) {
+ expect(project.clients).toHaveLength(1);
+ // scrypt `salt.hex`, what ingest's verifyPassword expects — never argon2.
+ expect(project.clients[0]?.secret).toMatch(/^[0-9a-f]+\.[0-9a-f]+$/);
+ }
+ });
+
+ test('sessions collapse to one row per session and match the events', async () => {
+ const sessions = await count(
+ `SELECT count() AS value FROM sessions FINAL WHERE project_id IN (${inList()})`
+ );
+ const distinct = await count(
+ `SELECT uniqExact(session_id) AS value FROM events WHERE project_id IN (${inList()})`
+ );
+ const starts = await count(
+ `SELECT count() AS value FROM events WHERE name = 'session_start' AND project_id IN (${inList()})`
+ );
+ expect(sessions).toBeGreaterThan(DAYS * SESSIONS_PER_DAY * 0.3);
+ expect(distinct).toBe(sessions);
+ expect(starts).toBe(sessions);
+ });
+
+ test('bounce rate and durations are in a realistic range', async () => {
+ const bounced = await count(
+ `SELECT countIf(is_bounce) AS value FROM sessions FINAL WHERE project_id IN (${inList()})`
+ );
+ const total = await count(
+ `SELECT count() AS value FROM sessions FINAL WHERE project_id IN (${inList()})`
+ );
+ const rate = bounced / total;
+ expect(rate).toBeGreaterThan(0.1);
+ expect(rate).toBeLessThan(0.8);
+ const averageSeconds = await count(
+ `SELECT avg(duration) / 1000 AS value FROM sessions FINAL WHERE project_id IN (${inList()}) AND NOT is_bounce`
+ );
+ expect(averageSeconds).toBeGreaterThan(10);
+ });
+
+ test('the materialized views the overview reads are populated', async () => {
+ expect(
+ await count(
+ `SELECT uniqMerge(profile_id) AS value FROM dau_mv WHERE project_id IN (${inList()})`
+ )
+ ).toBeGreaterThan(0);
+ expect(
+ await count(
+ `SELECT count() AS value FROM distinct_event_names_mv WHERE project_id IN (${inList()})`
+ )
+ ).toBeGreaterThan(0);
+ expect(
+ await count(
+ `SELECT count() AS value FROM profiles FINAL WHERE project_id IN (${inList()}) AND is_external`
+ )
+ ).toBeGreaterThan(0);
+ });
+});
diff --git a/packages/seed/src/send.ts b/packages/seed/src/send.ts
new file mode 100644
index 000000000..2cde726fa
--- /dev/null
+++ b/packages/seed/src/send.ts
@@ -0,0 +1,347 @@
+// `bun run send` — send real requests to a worktree's API with the seeded clients:
+// one event, an identify, or a whole generated journey replayed through `/track`
+// (ingest → Kafka → worker → ClickHouse), unlike `bun run seed`, which inserts
+// directly. For testing ingest, live views and anything that reacts to new events.
+//
+// bun run send track <event> [--project acme-shop] [--path /pricing] [--profile usr_x] [--prop k=v]... [--revenue 99]
+// bun run send identify <profileId> [--project] [--email] [--first-name] [--last-name] [--prop k=v]...
+// bun run send journey [--project acme-shop] [--sessions 1] [--realtime] [--seed 7]
+// options: --url <api> (default $API_URL), --client <id> --secret <secret> (default: the project's seeded client)
+
+import { existsSync, readFileSync } from 'node:fs';
+import { resolve } from 'node:path';
+import { setTimeout as sleep } from 'node:timers/promises';
+import { parseArgs } from 'node:util';
+import { ARCHETYPES, archetypeById } from './archetypes';
+import {
+ type Archetype,
+ Journey,
+ type JourneyEvent,
+} from './archetypes/archetype';
+import { CAMPAIGNS } from './data/referrers';
+import { projectIdFor } from './ids';
+import type { Person, Visitor } from './model';
+import { Rng } from './rng';
+import { MANIFEST_FILE } from './seed.constants';
+import { World } from './world';
+
+const USAGE = `Usage: bun run send <track|identify|journey> [options]
+
+ track <event> send one event --path /x --profile usr_x --prop k=v --revenue 99
+ identify <profileId> send an identify --email a@b --first-name A --last-name B --prop k=v
+ journey generate a session with the project's archetype and replay it live
+ --sessions N (default 1) --realtime (real dwell times; default caps gaps at 1s) --seed N
+
+ --project <id> which seeded project (default: acme-web)
+ --url <api> default $API_URL
+ --client <id> --secret <secret> override the project's seeded write client
+`;
+
+/** Gaps between replayed events are capped so a journey takes seconds, not minutes. */
+const MAX_GAP_MS = 1000;
+const MANIFEST_PATH = resolve(import.meta.dir, '..', '..', '..', MANIFEST_FILE);
+const DEFAULT_PROJECT = 'acme-web';
+
+interface Manifest {
+ projects: {
+ id: string;
+ archetype: string;
+ clientId: string;
+ clientSecret: string;
+ }[];
+}
+
+interface Target {
+ url: string;
+ clientId: string;
+ clientSecret: string;
+ projectId: string;
+ archetype: Archetype;
+}
+
+function readManifest(): Manifest | null {
+ if (!existsSync(MANIFEST_PATH)) {
+ return null;
+ }
+ return JSON.parse(readFileSync(MANIFEST_PATH, 'utf8')) as Manifest;
+}
+
+function parseProps(values: string[] | undefined): Record<string, unknown> {
+ const properties: Record<string, unknown> = {};
+ for (const entry of values ?? []) {
+ const index = entry.indexOf('=');
+ if (index <= 0) {
+ throw new Error(`--prop expects key=value, got "${entry}"`);
+ }
+ const raw = entry.slice(index + 1);
+ const asNumber = Number(raw);
+ properties[entry.slice(0, index)] =
+ raw === 'true'
+ ? true
+ : raw === 'false'
+ ? false
+ : raw !== '' && !Number.isNaN(asNumber)
+ ? asNumber
+ : raw;
+ }
+ return properties;
+}
+
+function resolveTarget(values: {
+ url?: string;
+ project?: string;
+ client?: string;
+ secret?: string;
+}): Target {
+ const url = (values.url ?? process.env.API_URL ?? '').replace(/\/$/, '');
+ if (!url) {
+ throw new Error('No API: pass --url or set API_URL.');
+ }
+ const projectId = values.project ?? DEFAULT_PROJECT;
+ const archetype =
+ ARCHETYPES.find((candidate) => projectIdFor(candidate) === projectId) ??
+ archetypeById(projectId);
+ const manifest = readManifest();
+ const seeded = manifest?.projects.find((project) => project.id === projectId);
+ const clientId = values.client ?? seeded?.clientId;
+ const clientSecret = values.secret ?? seeded?.clientSecret;
+ if (!(clientId && clientSecret)) {
+ throw new Error(
+ `No client for ${projectId}: run \`bun run seed\` first or pass --client and --secret.`
+ );
+ }
+ return { url, clientId, clientSecret, projectId, archetype };
+}
+
+interface Sender {
+ userAgent: string;
+ ip: string;
+}
+
+async function post(
+ target: Target,
+ sender: Sender,
+ body: unknown
+): Promise<string> {
+ const response = await fetch(`${target.url}/track`, {
+ method: 'POST',
+ headers: {
+ 'content-type': 'application/json',
+ 'openpanel-client-id': target.clientId,
+ 'openpanel-client-secret': target.clientSecret,
+ 'openpanel-sdk-name': target.archetype.sdk.name,
+ 'openpanel-sdk-version': target.archetype.sdk.version,
+ 'user-agent': sender.userAgent,
+ 'x-client-ip': sender.ip,
+ },
+ body: JSON.stringify(body),
+ });
+ const text = await response.text();
+ return `${response.status} ${text}`;
+}
+
+function trackBody(
+ name: string,
+ properties: Record<string, unknown>,
+ profileId?: string
+) {
+ return {
+ type: 'track',
+ payload: { name, properties, ...(profileId ? { profileId } : {}) },
+ };
+}
+
+function identifyBody(person: Person) {
+ // Empty strings fail validation (`avatar` must be a URL); send only what is set.
+ const optional = Object.fromEntries(
+ Object.entries({
+ firstName: person.firstName,
+ lastName: person.lastName,
+ email: person.email,
+ avatar: person.avatar,
+ }).filter(([, value]) => value !== '')
+ );
+ return {
+ type: 'identify',
+ payload: {
+ profileId: person.id,
+ ...optional,
+ properties: person.properties,
+ },
+ };
+}
+
+/** The SDK's view of a journey event: `__path` as the full URL (or screen name), `__title`, custom properties. */
+function sdkProperties(
+ target: Target,
+ event: JourneyEvent,
+ landingQuery: string
+): Record<string, unknown> {
+ const path =
+ target.archetype.deviceClass === 'native'
+ ? event.path
+ : `${target.archetype.origin}${event.path}${landingQuery}`;
+ return {
+ ...event.properties,
+ __path: path,
+ __title: event.title || undefined,
+ ...(event.revenue !== undefined ? { __revenue: event.revenue } : {}),
+ };
+}
+
+async function replay(
+ target: Target,
+ visitor: Visitor,
+ events: readonly JourneyEvent[],
+ utm: Record<string, string>,
+ realtime: boolean
+): Promise<void> {
+ const sender = { userAgent: visitor.userAgent, ip: visitor.geo.ip };
+ let identified: Person | null = visitor.identified ? visitor.person : null;
+ let previous: Date | null = null;
+ for (const [index, event] of events.entries()) {
+ if (previous) {
+ const gap = event.at.getTime() - previous.getTime();
+ await sleep(realtime ? gap : Math.min(gap, MAX_GAP_MS));
+ }
+ previous = event.at;
+ if (event.person && event.person !== identified) {
+ identified = event.person;
+ console.log(
+ `identify ${event.person.id} → ${await post(target, sender, identifyBody(event.person))}`
+ );
+ }
+ const landingQuery =
+ index === 0 && Object.keys(utm).length
+ ? `?${new URLSearchParams(utm)}`
+ : '';
+ const body = trackBody(
+ event.name,
+ sdkProperties(target, event, landingQuery),
+ identified?.id
+ );
+ console.log(
+ `${event.name.padEnd(28)} ${event.path.padEnd(28)} → ${await post(target, sender, body)}`
+ );
+ }
+}
+
+async function main(): Promise<void> {
+ const { values, positionals } = parseArgs({
+ args: process.argv.slice(2),
+ allowPositionals: true,
+ options: {
+ project: { type: 'string' },
+ url: { type: 'string' },
+ client: { type: 'string' },
+ secret: { type: 'string' },
+ path: { type: 'string' },
+ profile: { type: 'string' },
+ prop: { type: 'string', multiple: true },
+ revenue: { type: 'string' },
+ email: { type: 'string' },
+ 'first-name': { type: 'string' },
+ 'last-name': { type: 'string' },
+ sessions: { type: 'string' },
+ realtime: { type: 'boolean', default: false },
+ seed: { type: 'string' },
+ help: { type: 'boolean', default: false },
+ },
+ });
+ const [command, argument] = positionals;
+ if (values.help || !command) {
+ process.stdout.write(USAGE);
+ return;
+ }
+ const target = resolveTarget(values);
+ const rng = new Rng(`send/${values.seed ?? Date.now()}`);
+ const world = new World(
+ target.archetype,
+ target.projectId,
+ new Date(),
+ () => undefined
+ );
+ const visitor = world.pick(rng, new Date());
+ const sender = { userAgent: visitor.userAgent, ip: visitor.geo.ip };
+ console.log(
+ `→ ${target.url} project ${target.projectId} client ${target.clientId}`
+ );
+
+ if (command === 'track') {
+ if (!argument) {
+ throw new Error('track needs an event name');
+ }
+ const path = values.path ?? '/';
+ const properties = {
+ ...parseProps(values.prop),
+ __path:
+ target.archetype.deviceClass === 'native'
+ ? path
+ : `${target.archetype.origin}${path}`,
+ ...(values.revenue ? { __revenue: Number(values.revenue) } : {}),
+ };
+ console.log(
+ await post(
+ target,
+ sender,
+ trackBody(argument, properties, values.profile)
+ )
+ );
+ return;
+ }
+ if (command === 'identify') {
+ if (!argument) {
+ throw new Error('identify needs a profile id');
+ }
+ const person: Person = {
+ id: argument,
+ firstName: values['first-name'] ?? '',
+ lastName: values['last-name'] ?? '',
+ email: values.email ?? '',
+ avatar: '',
+ properties: Object.fromEntries(
+ Object.entries(parseProps(values.prop)).map(([key, value]) => [
+ key,
+ String(value),
+ ])
+ ),
+ };
+ console.log(await post(target, sender, identifyBody(person)));
+ return;
+ }
+ if (command === 'journey') {
+ const sessions = Number(values.sessions ?? 1);
+ for (let i = 0; i < sessions; i++) {
+ const current = i === 0 ? visitor : world.pick(rng, new Date());
+ const campaign = rng.chance(target.archetype.campaignChance)
+ ? rng.pick(CAMPAIGNS)
+ : null;
+ const utm: Record<string, string> = campaign
+ ? {
+ utm_source: campaign.source,
+ utm_medium: campaign.medium,
+ utm_campaign: campaign.name,
+ }
+ : {};
+ const journey = new Journey(
+ rng,
+ new Date(),
+ target.archetype.dwellMedianSeconds,
+ current.identified ? current.person : null
+ );
+ target.archetype.journey(rng, journey, current);
+ console.log(
+ `\nsession ${i + 1}: ${journey.events.length} events, ${current.person ? `person ${current.person.id}` : 'anonymous'}`
+ );
+ await replay(target, current, journey.events, utm, values.realtime);
+ world.touch(current, new Date(), {
+ identified: journey.identified !== null,
+ anonymousEvents: journey.events.some((event) => event.person === null),
+ });
+ }
+ return;
+ }
+ throw new Error(`Unknown command "${command}".\n${USAGE}`);
+}
+
+await main();
diff --git a/packages/seed/src/sink.ts b/packages/seed/src/sink.ts
new file mode 100644
index 000000000..135beae19
--- /dev/null
+++ b/packages/seed/src/sink.ts
@@ -0,0 +1,109 @@
+// Batches rows per table and inserts them as JSONEachRow through the shared
+// ClickHouse client, so the seed lands in whatever database CLICKHOUSE_URL names.
+// Adds are synchronous (the visitor pool evicts from inside a sync path); the
+// generator awaits `flushFull()` between sessions.
+
+import { ch, TABLE_NAMES } from '@openpanel/db';
+import type {
+ ClickhouseEventRow,
+ ClickhouseProfileRow,
+ ClickhouseSessionRow,
+} from './clickhouse-rows';
+import { INSERT_BATCH_ROWS } from './seed.constants';
+
+export interface SinkCounts {
+ events: number;
+ sessions: number;
+ profiles: number;
+}
+
+class TableBuffer<Row> {
+ private rows: Row[] = [];
+ count = 0;
+
+ constructor(private readonly table: string) {}
+
+ add(rows: readonly Row[]): void {
+ for (const row of rows) {
+ this.rows.push(row);
+ }
+ this.count += rows.length;
+ }
+
+ get full(): boolean {
+ return this.rows.length >= INSERT_BATCH_ROWS;
+ }
+
+ async flush(): Promise<void> {
+ if (this.rows.length === 0) {
+ return;
+ }
+ const batch = this.rows;
+ this.rows = [];
+ await ch.insert({
+ table: this.table,
+ values: batch,
+ format: 'JSONEachRow',
+ });
+ }
+}
+
+/** What the generator needs from a sink; tests substitute an in-memory one. */
+export interface RowSink {
+ addEvents(rows: readonly ClickhouseEventRow[]): void;
+ addSession(row: ClickhouseSessionRow): void;
+ addProfiles(rows: readonly ClickhouseProfileRow[]): void;
+ flushFull(): Promise<void>;
+ flush(): Promise<void>;
+ counts(): SinkCounts;
+}
+
+export class ClickhouseSink implements RowSink {
+ private readonly events = new TableBuffer<ClickhouseEventRow>(
+ TABLE_NAMES.events
+ );
+ private readonly sessions = new TableBuffer<ClickhouseSessionRow>(
+ TABLE_NAMES.sessions
+ );
+ private readonly profiles = new TableBuffer<ClickhouseProfileRow>(
+ TABLE_NAMES.profiles
+ );
+
+ private get buffers(): TableBuffer<unknown>[] {
+ return [this.events, this.sessions, this.profiles];
+ }
+
+ addEvents(rows: readonly ClickhouseEventRow[]): void {
+ this.events.add(rows);
+ }
+
+ addSession(row: ClickhouseSessionRow): void {
+ this.sessions.add([row]);
+ }
+
+ addProfiles(rows: readonly ClickhouseProfileRow[]): void {
+ this.profiles.add(rows);
+ }
+
+ async flushFull(): Promise<void> {
+ for (const buffer of this.buffers) {
+ if (buffer.full) {
+ await buffer.flush();
+ }
+ }
+ }
+
+ async flush(): Promise<void> {
+ for (const buffer of this.buffers) {
+ await buffer.flush();
+ }
+ }
+
+ counts(): SinkCounts {
+ return {
+ events: this.events.count,
+ sessions: this.sessions.count,
+ profiles: this.profiles.count,
+ };
+ }
+}
diff --git a/packages/seed/src/traffic.ts b/packages/seed/src/traffic.ts
new file mode 100644
index 000000000..9d2ed1ebe
--- /dev/null
+++ b/packages/seed/src/traffic.ts
@@ -0,0 +1,168 @@
+// Turns an archetype's shape (hourly curve, weekday curve) into visitor counts
+// per day and per hour. Ported from openpanel-mock's opmock/internal/traffic.
+//
+// The daily multiplier is a product of:
+// trend slow growth/decline, anchored so "today" == 1.0
+// walk a mean-reverting random walk across days (weeks drift)
+// weekday the archetype's day-of-week curve
+// noise small independent per-day jitter
+// anomaly rare spike or dip days
+// Everything is a pure function of (seed, key, day), so seeding twice gives
+// the same curve.
+
+import { Rng } from './rng';
+
+export interface TrafficShape {
+ /** Relative weight of each local hour, 0..23; scale is irrelevant. */
+ hourly: readonly number[];
+ /** Multiplies the daily total; index 0 = Sunday. */
+ dayOfWeek: readonly number[];
+ /** Fractional growth per week, e.g. 0.02 = +2 %/week, relative to today. */
+ trendPerWeek: number;
+ spikeChance: number;
+ dipChance: number;
+ spikeRange: readonly [number, number];
+ dipRange: readonly [number, number];
+}
+
+export interface TrafficParams {
+ seed: string;
+ /** 0 = perfectly regular, 1 = lively, > 1 = chaotic. */
+ variance: number;
+ now: Date;
+}
+
+export interface Anomaly {
+ kind: 'none' | 'spike' | 'dip';
+ multiplier: number;
+}
+
+export interface TrafficDay {
+ date: Date;
+ visitors: number;
+ weekday: number;
+ trend: number;
+ walk: number;
+ noise: number;
+ anomaly: Anomaly;
+ multiplier: number;
+}
+
+const MS_PER_DAY = 86_400_000;
+const HOURS_PER_DAY = 24;
+/** An arbitrary fixed anchor so any day's walk value is computable on its own. */
+const WALK_EPOCH_DAY = Date.UTC(2020, 0, 1);
+const WALK_MEAN_REVERSION = 0.9;
+const WALK_SIGMA = 0.1;
+const WALK_MIN = 0.35;
+const WALK_MAX = 3;
+const NOISE_SIGMA = 0.06;
+const NOISE_MIN = 0.5;
+const HOUR_JITTER_SIGMA = 0.12;
+const HOUR_JITTER_MIN = 0.2;
+const MAX_ANOMALY_VARIANCE = 2;
+
+function dayIndex(date: Date): number {
+ const midnight = Date.UTC(
+ date.getUTCFullYear(),
+ date.getUTCMonth(),
+ date.getUTCDate()
+ );
+ return Math.floor((midnight - WALK_EPOCH_DAY) / MS_PER_DAY);
+}
+
+function isoDay(date: Date): string {
+ return date.toISOString().slice(0, 10);
+}
+
+export class TrafficModel {
+ private walk: number[] = [];
+
+ constructor(
+ private readonly key: string,
+ private readonly shape: TrafficShape,
+ private readonly params: TrafficParams
+ ) {}
+
+ private rng(salt: string, day: Date): Rng {
+ return new Rng(`${this.params.seed}/${this.key}/${salt}/${isoDay(day)}`);
+ }
+
+ /** AR(1): w[i] = 0.9·w[i-1] + N(0, σ), replayed from the epoch and cached. */
+ private walkAt(index: number): number {
+ if (index < 0) {
+ return 0;
+ }
+ if (index < this.walk.length) {
+ return this.walk[index] ?? 0;
+ }
+ const rng = new Rng(`${this.params.seed}/${this.key}/walk`);
+ const sigma = WALK_SIGMA * this.params.variance;
+ this.walk = [];
+ let value = 0;
+ for (let i = 0; i <= index; i++) {
+ value = WALK_MEAN_REVERSION * value + rng.normal() * sigma;
+ this.walk.push(value);
+ }
+ return this.walk[index] ?? 0;
+ }
+
+ private anomaly(day: Date): Anomaly {
+ const rng = this.rng('anomaly', day);
+ const roll = rng.float();
+ const variance = Math.min(this.params.variance, MAX_ANOMALY_VARIANCE);
+ if (roll < this.shape.spikeChance * variance) {
+ const [low, high] = this.shape.spikeRange;
+ return { kind: 'spike', multiplier: low + rng.float() * (high - low) };
+ }
+ if (roll < (this.shape.spikeChance + this.shape.dipChance) * variance) {
+ const [low, high] = this.shape.dipRange;
+ return { kind: 'dip', multiplier: low + rng.float() * (high - low) };
+ }
+ return { kind: 'none', multiplier: 1 };
+ }
+
+ dayFor(day: Date, baseVisitors: number): TrafficDay {
+ const index = dayIndex(day);
+ const weeksFromNow = (index - dayIndex(this.params.now)) / 7;
+ const trend = (1 + this.shape.trendPerWeek) ** weeksFromNow;
+ const walk = Math.min(
+ WALK_MAX,
+ Math.max(WALK_MIN, Math.exp(this.walkAt(index)))
+ );
+ const noise = Math.max(
+ NOISE_MIN,
+ 1 + this.rng('noise', day).normal() * NOISE_SIGMA * this.params.variance
+ );
+ const weekday = this.shape.dayOfWeek[day.getUTCDay()] ?? 1;
+ const anomaly = this.anomaly(day);
+ const multiplier = trend * walk * noise * weekday * anomaly.multiplier;
+ return {
+ date: day,
+ visitors: Math.round(baseVisitors * multiplier),
+ weekday,
+ trend,
+ walk,
+ noise,
+ anomaly,
+ multiplier,
+ };
+ }
+
+ /** Fraction of a day's visitors per hour, with a little per-day jitter on the curve. */
+ hourWeights(day: Date): number[] {
+ const rng = this.rng('hours', day);
+ const weights: number[] = [];
+ let sum = 0;
+ for (let hour = 0; hour < HOURS_PER_DAY; hour++) {
+ const jitter = Math.max(
+ HOUR_JITTER_MIN,
+ 1 + rng.normal() * HOUR_JITTER_SIGMA * this.params.variance
+ );
+ const weight = (this.shape.hourly[hour] ?? 0) * jitter;
+ weights.push(weight);
+ sum += weight;
+ }
+ return weights.map((weight) => weight / sum);
+ }
+}
diff --git a/packages/seed/src/world.ts b/packages/seed/src/world.ts
new file mode 100644
index 000000000..87daac8ab
--- /dev/null
+++ b/packages/seed/src/world.ts
@@ -0,0 +1,232 @@
+// Per-project population: who comes back, and as whom. Visitors live in
+// cohorts by first-seen day; a returning session picks a cohort through the
+// archetype's retention curve, then a visitor through its stickiness. The
+// device id rotates at UTC midnight like ingest's salted id, so anonymous
+// traffic is only recognised within a day while identified people persist.
+// Bounded: the oldest cohort is dropped when the pool is full, which is what
+// lets profiles be written exactly once.
+
+import { parseUserAgent } from '@openpanel/shared/server';
+import type { Archetype, RetentionCurve } from './archetypes/archetype';
+import type { ClickhouseProfileRow } from './clickhouse-rows';
+import {
+ DESKTOP_USER_AGENTS,
+ MOBILE_USER_AGENTS,
+ NATIVE_APP_USER_AGENTS,
+} from './data/devices';
+import { newGeo } from './data/geo';
+import type { DeviceWindow, ParsedDevice, Visitor } from './model';
+import type { Rng, Weighted } from './rng';
+import { deviceProfileRow, personProfileRow } from './rows';
+import { VISITOR_POOL_SIZE } from './seed.constants';
+
+const DEVICE_ID_HEX_CHARS = 32;
+const MS_PER_DAY = 86_400_000;
+/** How many candidates a returning pick tries before settling for the last one. */
+const STICKINESS_ATTEMPTS = 6;
+const MAX_STICKINESS = 4;
+
+const parsedDevices = new Map<string, ParsedDevice>();
+
+function parseDevice(userAgent: string): ParsedDevice {
+ let parsed = parsedDevices.get(userAgent);
+ if (!parsed) {
+ parsed = parseUserAgent(userAgent);
+ parsedDevices.set(userAgent, parsed);
+ }
+ return parsed;
+}
+
+function utcDay(date: Date): string {
+ return date.toISOString().slice(0, 10);
+}
+
+/** Share of a cohort active `days` after its first day, interpolated on a log scale. */
+export function retentionAt(curve: RetentionCurve, days: number): number {
+ if (days <= 0) {
+ return curve.sameDay;
+ }
+ const points: [number, number][] = [
+ [1, curve.day1],
+ [7, curve.day7],
+ [30, curve.day30],
+ ];
+ for (let i = 0; i < points.length - 1; i++) {
+ const [fromDay, fromValue] = points[i] as [number, number];
+ const [toDay, toValue] = points[i + 1] as [number, number];
+ if (days <= toDay) {
+ const t = (days - fromDay) / (toDay - fromDay);
+ return fromValue * (toValue / fromValue) ** t;
+ }
+ }
+ // Past day 30 the 7→30 decay continues.
+ const weeklyDecay = (curve.day30 / curve.day7) ** (7 / 23);
+ return curve.day30 * weeklyDecay ** ((days - 30) / 7);
+}
+
+export class World {
+ private readonly cohorts = new Map<number, Visitor[]>();
+ private size = 0;
+
+ constructor(
+ private readonly archetype: Archetype,
+ private readonly projectId: string,
+ private readonly runStart: Date,
+ private readonly onProfiles: (rows: ClickhouseProfileRow[]) => void
+ ) {}
+
+ private dayIndex(at: Date): number {
+ return Math.floor((at.getTime() - this.runStart.getTime()) / MS_PER_DAY);
+ }
+
+ /** A returning visitor with probability `returningShare`, else a new one added to today's cohort. */
+ pick(rng: Rng, at: Date): Visitor {
+ const visitor =
+ (this.size > 0 && rng.chance(this.archetype.returningShare)
+ ? this.returning(rng, at)
+ : null) ?? this.newVisitor(rng, at);
+ this.rotateDevice(rng, visitor, at);
+ return visitor;
+ }
+
+ private returning(rng: Rng, at: Date): Visitor | null {
+ const today = this.dayIndex(at);
+ const weighted: Weighted<Visitor[]>[] = [];
+ for (const [day, visitors] of this.cohorts) {
+ const weight =
+ visitors.length * retentionAt(this.archetype.retention, today - day);
+ if (weight > 0) {
+ weighted.push({ value: visitors, weight });
+ }
+ }
+ if (weighted.length === 0) {
+ return null;
+ }
+ const cohort = rng.pick(weighted);
+ let candidate: Visitor | null = null;
+ for (let attempt = 0; attempt < STICKINESS_ATTEMPTS; attempt++) {
+ candidate = cohort[rng.int(0, cohort.length - 1)] ?? null;
+ // A visitor "in the future" relative to this session cannot be its visitor.
+ if (!candidate || candidate.lastSeen.getTime() > at.getTime()) {
+ candidate = null;
+ continue;
+ }
+ if (
+ rng.chance(
+ Math.min(MAX_STICKINESS, this.archetype.stickiness(candidate)) /
+ MAX_STICKINESS
+ )
+ ) {
+ return candidate;
+ }
+ }
+ return candidate;
+ }
+
+ private newVisitor(rng: Rng, at: Date): Visitor {
+ const userAgent = this.pickUserAgent(rng);
+ const day = this.dayIndex(at);
+ const visitor: Visitor = {
+ userAgent,
+ device: parseDevice(userAgent),
+ geo: newGeo(rng),
+ person: this.archetype.newPerson(rng),
+ identified: false,
+ window: this.newWindow(rng, at),
+ cohortDay: day,
+ sessions: 0,
+ firstSeen: at,
+ lastSeen: at,
+ lastPath: '',
+ lastReferrer: null,
+ state: {},
+ };
+ if (this.size >= VISITOR_POOL_SIZE) {
+ this.dropOldestCohort();
+ }
+ const cohort = this.cohorts.get(day) ?? [];
+ cohort.push(visitor);
+ this.cohorts.set(day, cohort);
+ this.size += 1;
+ return visitor;
+ }
+
+ private newWindow(rng: Rng, at: Date): DeviceWindow {
+ return {
+ id: rng.hex(DEVICE_ID_HEX_CHARS),
+ day: utcDay(at),
+ firstSeen: at,
+ lastSeen: at,
+ anonymousEvents: false,
+ };
+ }
+
+ /** A new UTC day means a new salted id; the old one's anonymous profile is final. */
+ private rotateDevice(rng: Rng, visitor: Visitor, at: Date): void {
+ if (visitor.window.day === utcDay(at)) {
+ return;
+ }
+ this.flushDevice(visitor);
+ visitor.window = this.newWindow(rng, at);
+ }
+
+ private flushDevice(visitor: Visitor): void {
+ if (visitor.window.anonymousEvents) {
+ this.onProfiles([deviceProfileRow(visitor, this.projectId)]);
+ }
+ }
+
+ /** Records a finished session: who it was sent as, and when it ended. */
+ touch(
+ visitor: Visitor,
+ end: Date,
+ options: { identified: boolean; anonymousEvents: boolean }
+ ): void {
+ visitor.sessions += 1;
+ visitor.identified ||= options.identified;
+ visitor.window.anonymousEvents ||= options.anonymousEvents;
+ if (end.getTime() > visitor.lastSeen.getTime()) {
+ visitor.lastSeen = end;
+ }
+ if (end.getTime() > visitor.window.lastSeen.getTime()) {
+ visitor.window.lastSeen = end;
+ }
+ }
+
+ private dropOldestCohort(): void {
+ const oldest = Math.min(...this.cohorts.keys());
+ const visitors = this.cohorts.get(oldest) ?? [];
+ this.cohorts.delete(oldest);
+ this.size -= visitors.length;
+ this.flushVisitors(visitors);
+ }
+
+ private flushVisitors(visitors: readonly Visitor[]): void {
+ for (const visitor of visitors) {
+ this.flushDevice(visitor);
+ if (visitor.identified && visitor.person) {
+ this.onProfiles([personProfileRow(visitor, this.projectId)]);
+ }
+ }
+ }
+
+ /** Writes every remaining profile; call once generation is done. */
+ drain(): void {
+ for (const visitors of this.cohorts.values()) {
+ this.flushVisitors(visitors);
+ }
+ this.cohorts.clear();
+ this.size = 0;
+ }
+
+ private pickUserAgent(rng: Rng): string {
+ if (this.archetype.deviceClass === 'native') {
+ return rng.pick(NATIVE_APP_USER_AGENTS);
+ }
+ return rng.pick(
+ rng.chance(this.archetype.mobileShare)
+ ? MOBILE_USER_AGENTS
+ : DESKTOP_USER_AGENTS
+ );
+ }
+}
diff --git a/packages/seed/tsconfig.json b/packages/seed/tsconfig.json
new file mode 100644
index 000000000..4ca92684f
--- /dev/null
+++ b/packages/seed/tsconfig.json
@@ -0,0 +1,10 @@
+{
+ "extends": "@openpanel/tsconfig/base.json",
+ "compilerOptions": {
+ "tsBuildInfoFile": "node_modules/.cache/tsbuildinfo.json",
+ "strictNullChecks": true,
+ "types": ["bun-types", "node"]
+ },
+ "include": ["src"],
+ "exclude": ["node_modules"]
+}