OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.
OPS: Grafana / alert migration for the V1 → V2 metrics cutover
Audience: whoever owns the OpenPanel dashboards and alerts. This is a written spec, not a change. No dashboard, alert or datasource was edited by the work that produced it — deliberately (M9 cutover discipline).
Written for M9-002. Sources: packages/core/src/metrics.ts (the one registry),
packages/core/src/http/http.metrics.ts, packages/core/src/jobs/jobs.metrics.ts,
apps/api/src/main.ts (the ROLE table), and — for the V1 side —
[email protected]/dist/fastify-metrics.js and
apps/worker/src/metrics.ts.
Where to put this file: it lives here because the task's scope globs cover
packages/**and not a repo-rootdocs/. Move it todocs/OPS_GRAFANA_MIGRATION.mdwith onegit mvif that is where you want it.
0. TL;DR — what a dashboard owner must do
- No metric name changes. Every panel and alert keeps working by name.
routelabel VALUES change for a handful of routes (§2). Any panel or alert that pinsroute="…"for one of those needs its literal updated.- Ten new
<queue>_*_countseries appear (§3.1). Nothing to fix; you may want new panels. buffer_add_*for the replay, group and profile buffers steps up from near-zero (§3.3). It looks like a regression on a graph. It is a fix.- Bull-board moved from
http://<worker>:3040/tohttp://<host>:<port>/bullboard, and is now behind the dashboard session cookie (§4). /metricsis unchanged: same path, same container port 3000, served in every ROLE.
1. Metric names and labels: unchanged, on purpose
fastify-metrics is deleted with Fastify (ADR-002). Its two route series are
re-created by one Elysia hook, reproducing the plugin's own definitions:
| Series | Type | Labels | V1 source | V2 source |
|---|---|---|---|---|
http_request_duration_seconds |
histogram | method, route, status_code |
fastify-metrics default |
core/src/http/http.metrics.ts |
http_request_summary_seconds |
summary | method, route, status_code |
fastify-metrics default |
same file |
Both still observe seconds, not milliseconds — a millisecond observation on
the same series would multiply every existing percentile by 1000, so it is
pinned by a test (http.metrics.test.ts).
Also unchanged in name and labels: the default process/GC metrics,
job_duration_ms{name,status}, kafka_events_*_total, the twelve buffer_*
series, the twelve session-lifecycle series, and the five
<queue>_{active,delayed,failed,completed,waiting}_count gauges.
job_duration_ms keeps its failure-only semantics: V1 observed it
exclusively in the workers' failed listener, so every percentile ever computed
from it is a percentile of failures. Success timings, if you ever want them,
must get a new series name.
2. The route label: same name, different values
This is the one thing that needs action.
V1's rule: the label was Fastify's routeOptions.url — the declared route
pattern including the plugin prefix, with Fastify's :param syntax.
V2's rule: the label is Elysia's route — the matched route pattern, with
Elysia's :param syntax.
The two syntaxes agree, so most values are byte-identical: every
/insights/:projectId/..., /manage/..., /export/..., /import/events,
/track/device-id, /healthz/live and so on carries the same literal it does
today. The differences are structural, and they are these:
2.1 Routes whose label changes
| What | V1 route label |
V2 route label |
Why |
|---|---|---|---|
| Track | /track/ |
/track |
V1's label was prefix + the route '/', giving a trailing slash. V2 declares the full path. |
| Profile | /profile/ |
/profile |
same |
| Legacy event | /event/ |
/event |
same (the route itself is KEPT — ADR-004/ADR-015 acceptance notes) |
| tRPC | /trpc/:path |
/trpc/* |
the fetch adapter derives the path by slicing the prefix, and a batched request puts commas in that segment, which :path cannot match (ADR-009) |
| OpenAPI UI | /documentation |
/openapi |
@fastify/swagger-ui → @elysiajs/openapi |
| OpenAPI document | /documentation/json |
/openapi.json |
same |
| Bull-board | (separate worker port, /) |
/bullboard, and its sub-routes |
§4 |
| Debug cron | (separate worker port, /debug/cron/:type) |
/debug/cron/:type on the main port |
§4 |
2.2 Routes whose label disappears
V1 route label |
Why it goes |
|---|---|
/misc/og/clear, /misc/favicon/clear |
deleted (removal register entry 6) |
/mcp GET and DELETE |
deleted (removal register entry 2); POST /mcp stays |
the whole worker-side / bull-board tree on :3040/:3050 |
merged onto the main port |
2.3 Behaviour at the edges, reproduced deliberately
| Case | V1 (fastify-metrics) |
V2 |
|---|---|---|
HEAD / OPTIONS |
not measured (methodBlacklist default) |
not measured |
| a request matching no route (404) | not measured (registeredRoutesOnly default true) |
not measured |
| a handler that threw and was mapped to 5xx | measured — onResponse fires |
measured — the hook is onAfterResponse, which likewise fires |
| status code grouping | ungrouped (groupStatusCodes default false) |
ungrouped |
2.4 One thing this table cannot promise yet
At the time of writing, main.ts mounts only the ops surface; the dashboard
and public-API surfaces mount at M9-004, when the V1 Fastify boot is deleted.
The rules above are final, and §2.1 lists every difference in the route set that
exists today — but re-read §2.1 against the M9-004 commit before you edit a
dashboard, because that is the commit in which the values actually start being
produced.
Two route-set observations that belong to that commit, not to this one, and are recorded here so they are not discovered from a graph:
packages/core/src/modules/tools/tools.routes.tsdeclares/site-checkerand/ip-lookupat the root, where V1 served them under the router prefix/tools. Whoever mounts the public-API surface has to restore the prefix; until then both the URL and the metric label would differ./manage/references*,/manage/projects/:projectId/dashboards*,POST /event,/healthcheckand/trpc/*are still V1-only routes.
3. Series that appear, disappear, or change level
3.1 New: the queue gauges cover all seven queues
V1 registered <queue>_{active,delayed,failed,completed,waiting}_count for
two queues — sessions and cron. V2 registers them for all seven
(ADR-018's continuity register). Ten new series, none of the existing ten
changed:
notification_{active,delayed,failed,completed,waiting}_count
import_{...}_count
insights_{...}_count
gsc_{...}_count
cohortCompute_{...}_count
The cohortCompute spelling is intentional and permanent — the registry key
was NOT renamed (ADR-005 acceptance note), so COHORTCOMPUTE_CONCURRENCY keeps
working and the series name matches the queue.
3.2 Gone: the GroupMQ shard series
Removed with GroupMQ itself (ADR-004). Any panel or alert still selecting them will read "No data" rather than error. Delete those panels.
3.3 Level change: buffer_add_* steps up
V1 installed the buffer add observers only inside the worker process, so every
API-side add — the replay, group and profile buffers on the /track path — was
invisible. V2 has one registry per process and every role builds the buffers, so
those adds now count.
On a graph this looks like a regression at the cutover. It is a fix. Expect
buffer_add_* for replay, group and profile to step up from near-zero;
event and session are already counted in V1 and should not move.
If you have an alert with an absolute threshold on any buffer_add_* series,
re-baseline it after the cutover.
3.4 Where each collector registers now
/metrics is served in every role, but not every role registers every
collector — a scrape-time gauge costs a Redis round trip per project, and ten
API replicas exposing the session gauges would multiply that Redis load for no
new information.
| Collector | ROLE=api |
ROLE=worker |
ROLE=all |
|---|---|---|---|
| default process metrics | yes | yes | yes |
http_request_* |
yes | yes | yes |
<queue>_*_count |
no | yes | yes |
buffer_* |
no | yes | yes |
sessions_active_total, sessions_projects_active, sessions_hwm_lag_ms |
no | yes | yes |
job_duration_ms, kafka_events_* |
declared, never observed | observed | observed |
Consequence for a dashboard: a panel that today sums a queue or buffer gauge
across every scrape target keeps working, because only the consuming replicas
ever exposed a non-trivial value. A panel that asserts presence on an API
target (e.g. absent(buffer_event_count)) will fire. Scope such alerts to the
consuming job.
4. Two URLs moved (deploy note, not a dashboard note)
The worker's second HTTP server is gone; one process serves one port in every role (ADR-007).
| What | V1 | V2 |
|---|---|---|
| Bull-board | http://<worker>:3040/ (and :3050), unauthenticated on that port |
http://<host>:<port>/bullboard, behind the dashboard session cookie |
| Debug cron trigger | http://<worker>:3040/debug/cron/:type |
/debug/cron/:type on the main port, still NODE_ENV != production only |
/metrics |
:3030 (api), :3040/:3050 (workers) |
unchanged — same path, container port still 3000 |
Bull-board is mounted only when the role consumes and DISABLE_BULLBOARD is
unset ('1' and 'true' are both still accepted as "disable"). It now returns
401 rather than the UI to an unauthenticated caller.
Unchanged and deliberate: pausing the cron queue from bull-board halts all
buffer flushing (docs/ANSWERS.md §3 — "known!").
5. Checklist
- Search dashboards and alert rules for
route="/track/",route="/profile/",route="/event/",route="/trpc/:path",route="/documentation"and update the literals per §2.1. - Delete panels selecting the GroupMQ shard series (§3.2).
- Re-baseline absolute-threshold alerts on
buffer_add_*for the replay, group and profile buffers (§3.3). - Scope any
absent()/up-style alert on a queue, buffer or session gauge to the consuming job only (§3.4). - Optionally add panels for the five newly-covered queues (§3.1).
- Update the bull-board bookmark and tell operators it now needs a dashboard login (§4).
- Re-read §2.1 against the M9-004 commit before editing anything (§2.4).