Skip to content

OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.

ralph(M12-010): Polish: fika swept from jobs/naming.ts, root package.json scripts point only at workspaces and scripts that exist, CLAUDE.md / .claude/CLAUDE.md / AGENTS.md rewritten for the current tree and paths

ralphbcf07cbc7c45
diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md
index 6ada80df8..a9cf654c1 100644
--- a/.claude/CLAUDE.md
+++ b/.claude/CLAUDE.md
@@ -12,22 +12,22 @@ Openpanel is an open-source web/product analytics platform (Mixpanel alternative
```bash
# Development
-pnpm dev # Run all services (api, worker, dashboard) in parallel
+pnpm dev # Run apps/api (ROLE=all, port 3333) + apps/start dashboard (port 3000) in parallel
pnpm dev:public # Run public/docs site only
pnpm dock:up / dock:down # Start/stop Docker (PostgreSQL, Redis, ClickHouse)
# Code quality
-pnpm check # Lint check (Biome via Ultracite)
+pnpm check # Lint (Biome via Ultracite) + dependency-cruiser (check:deps)
pnpm fix # Auto-fix lint/format issues
pnpm typecheck # Typecheck all packages
# Testing
-pnpm test # Run all tests (vitest)
-pnpm vitest run <path> # Run a single test file
-# Workspace: packages/* and apps/* (excluding apps/start)
+pnpm test # vitest run — packages/* and apps/* (excluding apps/start, packages/core)
+cd packages/core && bun test # core's own suite (ADR-010; bun:test cannot run under vitest)
+pnpm vitest run <path> # Run a single vitest test file
# Database
-pnpm codegen # Generate Prisma types + geo data
+pnpm codegen # Generate the Prisma client + core's geo/bot/ASN data
pnpm migrate # Run Prisma migrations (dev)
pnpm migrate:deploy # Deploy migrations (production - never run this)
@@ -42,37 +42,35 @@ pnpm dock:redis # Redis CLI
| App | Stack | Port | Purpose |
|-----|-------|------|---------|
-| `apps/api` | Fastify + tRPC | 3333 | REST/RPC API server |
-| `apps/start` | TanStack Start (Vite + React 19) | 3000 | Dashboard SPA |
+| `apps/api` | Elysia on Bun, `ROLE=api\|worker\|all` | 3000 (`API_PORT`) | REST/tRPC API, BullMQ workers, Kafka ingest consumer — one process, role-gated |
+| `apps/start` | TanStack Start (Vite + React 19) | 3000 (dev default; run alongside `apps/api` needs a distinct `API_PORT`) | Dashboard SPA |
| `apps/public` | Next.js 16 + Fumadocs | - | Marketing/docs site |
-| `apps/worker` | Express + BullMQ | 9999 | Background job processor |
+
+The standalone worker app is gone — M9 merged it into `apps/api`, selected at boot by `ROLE`.
### Key Packages
| Package | Purpose |
|---------|---------|
-| `packages/db` | Prisma ORM (PostgreSQL) + ClickHouse client |
-| `packages/trpc` | tRPC router definitions, context, middleware |
-| `packages/auth` | Authentication (Arctic OAuth, Oslo sessions, argon2) |
-| `packages/queue` | BullMQ + GroupMQ job queue definitions |
-| `packages/redis` | Redis client + LRU caching |
-| `packages/validation` | Zod schemas shared across apps |
-| `packages/common` | Shared utilities (date-fns, ua-parser, nanoid) |
+| `packages/core` | The backend: every module (`<name>.rpc.ts` / `.routes.ts` / `.jobs.ts` / `.service.ts`), the tRPC router, Elysia route surfaces, the BullMQ job registry, auth macros, the ClickHouse `sql` tag consumer code, shared utilities. `apps/api` is a thin shell that mounts it |
+| `packages/db` | Prisma ORM (PostgreSQL) + ClickHouse client, the `sql` tagged-template query builder, migrations |
+| `packages/redis` | Redis client (Bun `RedisClient`) + caching, locks, pub/sub |
| `packages/email` | React Email templates via Resend |
+| `packages/payments` | Billing/subscription integration |
| `packages/sdks/*` | Client SDKs (web, react, next, express, react-native, etc.) |
### Data Flow
-1. **Event ingestion**: Client SDKs → `apps/api` (track routes) → Redis queue
-2. **Processing**: `apps/worker` picks up jobs from BullMQ, batches events into ClickHouse
-3. **Dashboard queries**: `apps/start` → tRPC → `apps/api` → ClickHouse (analytics) / PostgreSQL (config)
-4. **Real-time**: WebSocket via Fastify, pub/sub via Redis
+1. **Event ingestion**: Client SDKs → `apps/api` `/track` → Kafka/Redpanda topic `events`
+2. **Processing**: `apps/api`'s Kafka consumer (any non-`api` role) batches events into ClickHouse
+3. **Dashboard queries**: `apps/start` → tRPC (fetch adapter) → `apps/api` → ClickHouse (analytics) / PostgreSQL (config)
+4. **Real-time**: WebSocket via Elysia, pub/sub via Redis
### Three-Database Strategy
- **PostgreSQL**: Relational data (users, orgs, projects, dashboards). Managed by Prisma.
- **ClickHouse**: Analytics event storage (OLAP). High-volume reads/writes.
-- **Redis**: Caching, job queues (BullMQ), rate limiting, pub/sub.
+- **Redis**: Caching, BullMQ job queues, rate limiting, pub/sub.
### Dashboard (apps/start)
@@ -80,7 +78,7 @@ Uses TanStack Router with file-based routing (`src/routes/`). State management v
### API (apps/api)
-Fastify server with tRPC integration. Route files in `src/routes/`. Hooks for IP extraction, request logging, timestamps. Built with `tsdown`.
+Elysia on Bun, mounting `@openpanel/core`'s routers. No bundler — `bun run src/main.ts` runs the TypeScript directly (ADR-010). `main.ts` boots deps, `app.ts` wires the four Elysia scopes (root, dashboard, public API, ops); `ROLE` selects whether workers, schedulers and the Kafka consumer also start in this process.
---
## Core Principles
diff --git a/.claude/commands/pr-review.md b/.claude/commands/pr-review.md
index 5bcf18242..80b8a8348 100644
--- a/.claude/commands/pr-review.md
+++ b/.claude/commands/pr-review.md
@@ -12,21 +12,21 @@ Review the current PR by running `git diff main...HEAD` and examining all change
## Authorization & Data Access
-- Every tRPC procedure that accesses project/org data uses the appropriate access check (`getProjectAccess`, `getOrganizationAccess`, `getClientAccess` from `@openpanel/db`)
+- Every `protectedProcedure` gets the `enforceAccess` middleware (`rpc/base.ts`) for free — it reads the raw, top-level `projectId`/`organizationId` and calls `ctx.services.auth`, which is `shared/access.ts`'s single `createAccessChecks` ladder, bound once centrally in `auth.service.ts` (M10-002)
- All ClickHouse queries filter by `project_id` — no cross-project data leaks
- All Prisma queries scope to the authenticated user's org/project — no missing `organizationId`/`projectId` where clauses
- No client-provided IDs trusted for authorization without server-side validation
-- Note: all `protectedProcedure` which either have `organizationId` or `projectId` will be ensured correct access in the middleware
+- Note: `enforceAccess` only checks `projectId`/`organizationId` in *input* — secondary object ids (reportId, dashboardId, …) need an in-handler check (ADR-011)
## Architecture: Service Layer
-- Data fetching and mutation logic lives in `packages/db/src/services/` — not inline in tRPC routers or API route handlers
+- Data fetching and mutation logic lives in `packages/core/src/modules/<name>/<name>.service.ts` — not inline in tRPC routers or route handlers
- tRPC routers should call service functions, not query ClickHouse/Prisma directly
-- New queue job types defined in `packages/queue/src/queues.ts`, not inline in app code
+- New job types are declared in the owning module's `<name>.jobs.ts` and spread into `packages/core/src/jobs.registry.ts` — not inline in app code. A cron schedule is the job's own `cron` field (ADR-021), not a second registry
## Architecture: Validation
-- Zod schemas used in more than one package/app belong in `packages/validation/src/` — not defined locally in a router or component
+- Zod schemas used by more than one module, or by `apps/start`/an SDK, belong in the owning module's `<name>.constants.ts` (imports only zod, other constants, or types — ADR-008) — not defined locally in a router or component
- All tRPC procedures use `.input(zodSchema)` for input validation
- Schemas used only within a single file can stay local
@@ -48,8 +48,8 @@ Review the current PR by running `git diff main...HEAD` and examining all change
## Clickhouse
- Queries should be optimized for very large datasets
-- `chQuery` can be used for simple queries
-- `clix` should be used when we have a lot of dynamic sql queries
+- Static queries are raw SQL with ClickHouse-native `{name:Type}` bound params
+- Dynamic composition (chart engine, filters, breakdowns) uses `@openpanel/db`'s `sql` tagged template — never `${}` of a raw value, never hand-rolled string concatenation (ADR-013)
---
diff --git a/CLAUDE.md b/CLAUDE.md
index 5797fb5f0..d3fb55bdc 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -6,7 +6,7 @@ anything.
## Safety
Your world is exactly two directories: this repository and
-`/data/apps/rewrite-openpanel` (the controller). Do not read, list, or modify
+`/home/deploy/rewrite-openpanel` (the controller). Do not read, list, or modify
anything else on this machine.
Never use production credentials — production does not exist on this machine,
@@ -18,7 +18,7 @@ and nothing here should reach it. Local Postgres / ClickHouse / Redis only.
ClickHouse (8123) are already running on localhost, started by another user.
Do not try to start, stop or install them.
- **No sudo.** Anything needing root is a human task. Say so; do not attempt it.
-- Writes are confined to this repo and `/data/apps/rewrite-openpanel`. Every
+- Writes are confined to this repo and `/home/deploy/rewrite-openpanel`. Every
other path on the box is read-only to you.
## Commands
@@ -27,7 +27,7 @@ and nothing here should reach it. Local Postgres / ClickHouse / Redis only.
pnpm install
pnpm codegen # prisma client; needs .env
pnpm run typecheck
-pnpm run lint
+pnpm check # ultracite + check:deps
pnpm test
```
@@ -43,18 +43,18 @@ count. When replacing a query, diff the old and new result sets on the same data
Local is a single node; production is 2 shards x 2 replicas. Before editing any
query against a `Distributed` table read
-`/data/apps/rewrite-openpanel/docs/ENVIRONMENT.md` - a plain `IN (subquery)` on a
+`/home/deploy/rewrite-openpanel/docs/ENVIRONMENT.md` - a plain `IN (subquery)` on a
distributed table silently returns per-shard results.
## Architecture
-Architecture decisions live in `/data/apps/rewrite-openpanel/decisions`.
+Architecture decisions live in `/home/deploy/rewrite-openpanel/decisions`.
**Accepted ADRs are binding.** If you need a decision no accepted ADR covers,
stop and say `BLOCKED: needs ADR — <question>`. Do not decide it yourself.
-The target architecture is `/data/apps/rewrite-openpanel/docs/TARGET_ARCHITECTURE.md`.
+The target architecture is `/home/deploy/rewrite-openpanel/docs/TARGET_ARCHITECTURE.md`.
The documented behaviour of the system you are replacing is in
-`/data/apps/rewrite-openpanel/docs/current/`. Read the relevant document before
+`/home/deploy/rewrite-openpanel/docs/current/`. Read the relevant document before
changing that area — it exists so you do not have to rediscover it.
## Ralph
@@ -62,9 +62,9 @@ changing that area — it exists so you do not have to rediscover it.
Do not modify:
```
-/data/apps/rewrite-openpanel/plan/tasks.json
-/data/apps/rewrite-openpanel/state
-/data/apps/rewrite-openpanel/verification
+/home/deploy/rewrite-openpanel/plan/tasks.json
+/home/deploy/rewrite-openpanel/state
+/home/deploy/rewrite-openpanel/verification
```
The orchestrator owns task state and owns the definition of "passing".
diff --git a/docs/TECH_DEBT.md b/docs/TECH_DEBT.md
index b7ef60b5b..65ea89aaa 100644
--- a/docs/TECH_DEBT.md
+++ b/docs/TECH_DEBT.md
@@ -1339,3 +1339,32 @@ once-qualified, no dashes, no backticks) cannot spell either name. **If a
future task widens `sql.id` to accept a backtick-quoted, multi-part name,
these two rows and `compiled.ts` with them can go.** That is the standing exit
condition; recorded here, not done in M12-009 (out of its scope).
+
+## Root scripts (M12-010)
+
+Every `--filter <name>` in the root `package.json` verified against the
+workspace it selects and the script that workspace actually exports.
+`gen:bots` pointed at `api`, which has no such script — the generator has
+always lived in `core`; fixed to match `gen:referrers`, which was already
+correct. `dev` assumed a `testing` script existed on every dev-facing
+workspace (a V1-era convention); only `apps/api` still has one, so
+`pnpm -r --parallel testing` silently ran nothing but the API. Fixed to start
+`apps/api` and `apps/start`'s dashboard explicitly, on non-colliding ports
+(both default to 3000) — verified with a real 10s boot (Kafka consumer joined,
+7 BullMQ workers started, cron upserted, Vite ready) and a clean `SIGTERM`
+shutdown, 2026-09-07.
+
+| Script | Target workspace(s) | Target script | Exists |
+|---|---|---|---|
+| `gen:bots` | `core` | `gen:bots` | yes |
+| `gen:referrers` | `core` | `gen:referrers` | yes |
+| `dev` | `api` (testing), `start` (dev) | `testing`, `dev` | yes |
+| `dev:public` | `public` | `dev` | yes |
+| `db:codegen` | `db` | `codegen` | yes |
+| `codegen` | `db`, `core` | `codegen` (both) | yes |
+| `migrate` | `db` | `migrate` | yes |
+| `migrate:deploy` | `db` | `migrate:deploy` | yes |
+
+`typecheck`, `check`, `check:workspace`, `check:deps` and `fix` run across the
+whole workspace (`pnpm -r` / no filter) and name no specific package, so they
+are not in this table.
diff --git a/package.json b/package.json
index c7352f4c3..c2271226c 100644
--- a/package.json
+++ b/package.json
@@ -8,7 +8,7 @@
"packageManager": "[email protected]",
"scripts": {
"test": "vitest run",
- "gen:bots": "pnpm -r --filter api gen:bots",
+ "gen:bots": "pnpm -r --filter core gen:bots",
"gen:referrers": "pnpm -r --filter core gen:referrers",
"dock:up": "docker compose up -d",
"dock:down": "docker compose down",
@@ -18,7 +18,7 @@
"codegen": "pnpm -r --filter db --filter core run codegen",
"migrate": "pnpm -r --filter db run migrate",
"migrate:deploy": "pnpm -r --filter db run migrate:deploy",
- "dev": "pnpm -r --parallel testing",
+ "dev": "pnpm --filter api run testing & pnpm --filter start run dev & wait",
"dev:public": "pnpm -r --filter public dev",
"typecheck": "pnpm -r --no-bail typecheck",
"update-simple-git-hooks": "npx simple-git-hooks",
diff --git a/packages/core/src/jobs/naming.test.ts b/packages/core/src/jobs/naming.test.ts
index 2d71efc38..59eae683b 100644
--- a/packages/core/src/jobs/naming.test.ts
+++ b/packages/core/src/jobs/naming.test.ts
@@ -48,7 +48,7 @@ test('the golden table covers exactly the queues the registry declares', () => {
).toEqual(golden);
});
-test('braces appear only under QUEUE_CLUSTER — the divergence from fika', () => {
+test('braces appear only under QUEUE_CLUSTER', () => {
expect(queueKey('cron')).toBe('cron');
expect(queueKey('cron', {})).toBe('cron');
expect(queueKey('cron', { cluster: false })).toBe('cron');
diff --git a/packages/core/src/jobs/naming.ts b/packages/core/src/jobs/naming.ts
index 87443e3e9..9896f2e45 100644
--- a/packages/core/src/jobs/naming.ts
+++ b/packages/core/src/jobs/naming.ts
@@ -13,11 +13,9 @@ export interface QueueKeyOptions {
/**
* The Redis key a queue lives under.
*
- * Deliberate divergence from the fika template this is ported from: fika
- * braces unconditionally, we brace only under `QUEUE_CLUSTER`, because V1's
- * `getQueueName` did and every existing queue in every deployment is named
- * that way (ADR-005). `-` joins the namespace because BullMQ rejects a `:`
- * in a queue name.
+ * Braces only under `QUEUE_CLUSTER`, because V1's `getQueueName` did and every
+ * existing queue in every deployment is named that way (ADR-005). `-` joins
+ * the namespace because BullMQ rejects a `:` in a queue name.
*/
export function queueKey(name: string, options?: QueueKeyOptions): string {
const scoped = options?.namespace ? `${name}-${options.namespace}` : name;