Your favorite shift management application!
feat(telemetry): store pings in Cloudflare D1 and publish a public statistics page (#224)
panteLa21bb6988717
diff --git a/components/telemetry-consent-dialog.tsx b/components/telemetry-consent-dialog.tsxindex 1d0b32e..d3115b3 100644--- a/components/telemetry-consent-dialog.tsx+++ b/components/telemetry-consent-dialog.tsx@@ -70,6 +70,10 @@ export function TelemetryConsentDialog({ open, onDecide }: TelemetryConsentDialo<span aria-hidden>•</span>{t("telemetry.consent.pointRevocable")}</li>+ <li className="flex gap-2">+ <span aria-hidden>•</span>+ {t("telemetry.consent.pointPublic")}+ </li></ul><details className="rounded-lg border border-line">diff --git a/docs/TELEMETRY.md b/docs/TELEMETRY.mdindex 070f62f..0d1ce7b 100644--- a/docs/TELEMETRY.md+++ b/docs/TELEMETRY.md@@ -122,9 +122,14 @@ The default endpoint is a small Cloudflare Worker; its source lives in this repo- Accepts only `POST` requests up to 16 KB.- Validates the payload against the current schema version (`telemetry-hub/src/schemas/v1.ts`). A body that isn't valid JSON is answered with `400` and one larger than 16 KB with `413`; a payload that parses but fails validation, or carries an unknown/retired `schemaVersion`, is answered with `200` and discarded. Either way the fire-and-forget sender ignores the status and never retries.-- Forwards a valid payload to [PostHog](https://posthog.com/) (`telemetry-hub/src/targets/posthog.ts`), explicitly enumerating each field it passes on rather than forwarding the request body as-is, so an unexpected extra field can never leak through.-- Sends `$ip: null` and `$geoip_disable: true` with every event, so PostHog neither stores the request IP nor derives a country or city from it — without this, the Cloudflare PoP nearest the sending instance would approximate its location.-- **Stores nothing itself.** There is no database, no request log, and no persistence layer in the Worker — a payload it can't forward to PostHog is simply lost, not queued or written anywhere.+- Writes a valid payload as one row into a [Cloudflare D1](https://developers.cloudflare.com/d1/) database (`telemetry-hub/src/targets/d1.ts`), with each field going into its own named column rather than the request body being stored as-is, so an unexpected extra field can never leak through. **No third party is involved** — the data goes to the operator's own Cloudflare account and nowhere else.+- **Never reads the connecting IP**, and the table has no column that could hold one — neither an IP nor a country or city derived from it is stored. The `received_at` timestamp on each row is the Worker's own clock, not the instance's. Cloudflare still sees the connecting IP in its own edge logs, as it does for any hosted Worker, but the Worker code never touches it.+- **Keeps rows indefinitely.** There is no expiry job: a ping written today is still there next year, and repeated pings from the same instance accumulate as separate rows linked by `instanceId`. Nothing beyond the fields listed above is stored, and there is no request log or queue — a payload the Worker cannot write is simply lost, not retried.+- **Publishes aggregated statistics on a public page**, at `GET /` on the hub's URL (`https://telemetry-bettershift.pantelx.com` by default), with the same figures as JSON at `GET /data.json`. Both are computed hourly by a scheduled job (`src/aggregate.ts`) and served from that precomputed result, so the page never queries the raw ping table directly.+- **Only aggregates appear there** — counts and shares grouped across instances. The published data contains no instance id, no raw row, and no per-instance timestamp; the closest thing to a timestamp is `computedAt`, the time the hourly job last ran, which is not tied to any individual instance.+- **There is no small-group threshold.** A value reported by only one instance — a Node version nobody else is on, a timezone unique to one deployment — is published as such, with an instance count of `1`. An earlier design collapsed small groups into an "other" bucket; the maintainer removed it, since the previous telemetry backend had no such floor either. If you are the only instance reporting a given value, that fact is visible on the public page. The one size limit that does exist is unrelated to this: each breakdown lists at most its 25 largest values and sums the rest into one `remainder`-marked entry, purely so a page with an attacker-controlled, unbounded field (custom field types) cannot grow without limit — it is not a privacy measure, and nothing is hidden for being rare.+- **Distributions count instances that reported within the last 30 days** — one vote per instance, from its newest ping in that window. The "reports over time" history on the page covers a longer, 90-day window instead.+- **`schemaVersion` stays at `1` for this change.** The page publishes aggregates computed from data instances were already sending; the payload itself is unchanged, so no instance is asked for fresh consent. Deciding whether that is acceptable is exactly what this section is for — an admin who already agreed to `schemaVersion: 1` should judge that consent against what is written here, not against a version bump that didn't happen.---diff --git a/messages/cs.json b/messages/cs.jsonindex da3bd78..bf0a1e3 100644--- a/messages/cs.json+++ b/messages/cs.json@@ -1339,6 +1339,7 @@"pointAnonymous": "Žádná jména, adresy, e-maily ani obsah kalendářů","pointBuckets": "Velikosti se uvádějí pouze jako rozsah, například „6-20 uživatelů“","pointRevocable": "Kdykoli odvolatelné v panelu správce",+ "pointPublic": "Souhrnná čísla se zobrazují na veřejné stránce se statistikami","previewToggle": "Zobrazit náhled dat k odeslání","docsLink": "Úplná dokumentace","accept": "Odeslat",diff --git a/messages/de.json b/messages/de.jsonindex d7a48b9..e82c369 100644--- a/messages/de.json+++ b/messages/de.json@@ -1339,6 +1339,7 @@"pointAnonymous": "Keine Namen, Adressen, E-Mails oder Kalenderinhalte","pointBuckets": "Größenangaben nur als Bereich, etwa „6-20 Nutzer“","pointRevocable": "Jederzeit im Admin-Bereich widerrufbar",+ "pointPublic": "Aggregierte Kennzahlen erscheinen auf einer öffentlichen Statistikseite","previewToggle": "Vorschau der zu sendenden Daten anzeigen","docsLink": "Vollständige Dokumentation","accept": "Senden",diff --git a/messages/en.json b/messages/en.jsonindex 374cc83..8c31689 100644--- a/messages/en.json+++ b/messages/en.json@@ -1339,6 +1339,7 @@"pointAnonymous": "No names, addresses, emails, or calendar content","pointBuckets": "Size figures only as a range, e.g. “6-20 users”","pointRevocable": "Revocable at any time in the admin area",+ "pointPublic": "Aggregated figures appear on a public statistics page","previewToggle": "Show preview of data to be sent","docsLink": "Full documentation","accept": "Send",diff --git a/messages/es.json b/messages/es.jsonindex 7c36f65..6948658 100644--- a/messages/es.json+++ b/messages/es.json@@ -1339,6 +1339,7 @@"pointAnonymous": "Sin nombres, direcciones, correos electrónicos ni contenido de calendarios","pointBuckets": "Los tamaños solo se indican como rango, por ejemplo «6-20 usuarios»","pointRevocable": "Revocable en cualquier momento desde el panel de administración",+ "pointPublic": "Las cifras agregadas aparecen en una página de estadísticas pública","previewToggle": "Mostrar vista previa de los datos a enviar","docsLink": "Documentación completa","accept": "Enviar",diff --git a/messages/fr.json b/messages/fr.jsonindex ed538af..964baca 100755--- a/messages/fr.json+++ b/messages/fr.json@@ -1339,6 +1339,7 @@"pointAnonymous": "Aucun nom, adresse, e-mail ni contenu de calendrier","pointBuckets": "Les tailles ne sont indiquées que sous forme de plage, par exemple « 6-20 utilisateurs »","pointRevocable": "Révocable à tout moment depuis le panneau d'administration",+ "pointPublic": "Les chiffres agrégés apparaissent sur une page de statistiques publique","previewToggle": "Afficher l'aperçu des données à envoyer","docsLink": "Documentation complète","accept": "Envoyer",diff --git a/messages/it.json b/messages/it.jsonindex afe7f0c..d25517f 100644--- a/messages/it.json+++ b/messages/it.json@@ -1339,6 +1339,7 @@"pointAnonymous": "Nessun nome, indirizzo, email o contenuto dei calendari","pointBuckets": "Le dimensioni sono indicate solo come intervallo, ad esempio «6-20 utenti»","pointRevocable": "Revocabile in qualsiasi momento dal Pannello Admin",+ "pointPublic": "Le cifre aggregate compaiono in una pagina di statistiche pubblica","previewToggle": "Mostra l'anteprima dei dati da inviare","docsLink": "Documentazione completa","accept": "Invia",diff --git a/telemetry-hub/.env.example b/telemetry-hub/.env.exampledeleted file mode 100644index f41e038..0000000--- a/telemetry-hub/.env.example+++ /dev/null@@ -1,16 +0,0 @@-# Settings for scripts/create-posthog-dashboard.mjs — copy to .env (gitignored), then `npm run dashboard`.-# Not the Worker's config: that lives in wrangler.toml and .dev.vars.--# Personal key (phx_...) with the dashboard:write, insight:write and project:read scopes.-POSTHOG_PERSONAL_API_KEY=--# Optional. Only needed if the key can see more than one project.-# POSTHOG_PROJECT_ID=--# Optional. App host, not the eu.i.posthog.com ingest host.-# POSTHOG_HOST=https://eu.posthog.com--# Optional. Instance ids to leave out of the "ohne Test-Instanzen" dashboard (comma-separated).-# EXCLUDE_INSTANCE_IDS=--# Optional. DASHBOARD_NAME, and DASHBOARD_VARIANTS=all,no-dev,dev-only,no-idsdiff --git a/telemetry-hub/README.md b/telemetry-hub/README.mdindex 42f724e..a4ada4a 100644--- a/telemetry-hub/README.md+++ b/telemetry-hub/README.md@@ -1,6 +1,6 @@# BetterShift Telemetry Hub-A small Cloudflare Worker that receives the opt-in, anonymous instance pings of BetterShift and forwards them to [PostHog](https://posthog.com/). It stores nothing itself.+A small Cloudflare Worker that receives the opt-in, anonymous instance pings of BetterShift and stores them in a [Cloudflare D1](https://developers.cloudflare.com/d1/) database. Reports are plain SQL, run from the command line.You only need this if you want to **run your own receiver**. By default, instances send to the maintainer's hub, and an admin can opt out completely. What is sent is documented field by field in [`docs/TELEMETRY.md`](../docs/TELEMETRY.md).@@ -13,19 +13,21 @@ The hub is deployed on its own and is not part of the Docker image.3. [Point Your Instances at It](#point-your-instances-at-it)4. [Test It Locally](#test-it-locally)5. [What the Worker Does](#what-the-worker-does)-6. [The PostHog Dashboards](#the-posthog-dashboards)-7. [Operating It](#operating-it)-8. [Project Layout](#project-layout)+6. [The Public Statistics Page](#the-public-statistics-page)+7. [Reading the Data](#reading-the-data)+8. [Operating It](#operating-it)+9. [Project Layout](#project-layout)---## What You Need-- A **Cloudflare account** (the free plan is enough).-- A **PostHog project**, cloud EU or US. Use the _project API key_ (`phc_...`), not a personal key.+- A **Cloudflare account** (the free plan is enough — see [Cost](#cost)).- **Node.js 20.12 or newer** on the machine you deploy from.- Optional: a domain on Cloudflare, if you want a URL like `telemetry.example.com`. Without one, the Worker is served from a free `*.workers.dev` address.+There is no third-party service and no API key anywhere in this project.+## Set It UpAll commands run in this directory (`telemetry-hub/`).@@ -37,35 +39,43 @@ npm installnpx wrangler login```-### 2. Adjust `wrangler.toml`+### 2. Create the database++```bash+npx wrangler d1 create bettershift-telemetry+```++Wrangler prints a `database_id`. Paste it into the `[[d1_databases]]` block in `wrangler.toml`, replacing `REPLACE_WITH_YOUR_DATABASE_ID`. The id is not a secret and belongs in the file.++### 3. Adjust `wrangler.toml`The committed file describes the maintainer's deployment. Change it before your first deploy:-| Setting | What to do |-| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |-| `name` | Any unique Worker name. |-| `routes` | Replace `pattern` and `zone_name` with your own hostname and the Cloudflare zone it belongs to. Cloudflare creates the DNS record and certificate for you. |-| `POSTHOG_HOST` | The **ingest** host of your PostHog region: `https://eu.i.posthog.com` or `https://us.i.posthog.com`. It must match the region of the project key (see below). |+| Setting | What to do |+| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `name` | Any unique Worker name. |+| `routes` | Replace `pattern` and `zone_name` with your own hostname and the Cloudflare zone it belongs to. Cloudflare creates the DNS record and certificate for you. |+| `database_id` | From step 2. |+| `database_name` | Only if you named the database something else. It also has to match the name in the `migrate` scripts in `package.json`. |+| `binding` | Leave it alone unless you also change the matching key in `Env` in `src/index.ts`. A mismatch is silent: the Worker still answers `200` and drops the ping. |No domain? Delete the whole `routes` block and add `workers_dev = true` instead. After the deploy, Wrangler prints the `https://<name>.<your-subdomain>.workers.dev` URL.-### 3. Store the PostHog key as a secret+### 4. Create the table```bash-npx wrangler secret put POSTHOG_API_KEY+npm run migrate```-Paste the project API key (PostHog → Project settings → Project API key). It is stored encrypted in Cloudflare and is never written to `wrangler.toml` or the repository.--A key only works against its own region. An EU project used with `us.i.posthog.com` (or the other way round) is rejected, and because the Worker always answers `200`, you would see no error on the sender's side. See [Operating It](#operating-it) for where to look.+This applies everything in `migrations/` to the deployed database. Run it again after adding a migration; already-applied ones are skipped — that includes `migrations/0002_create_aggregates.sql`, added for the public statistics page below, so `npm run migrate` needs a second run on a database that was already set up before it existed.-### 4. Deploy+### 5. Deploy```bashnpm run deploy```-### 5. Check that it arrived+### 6. Check that it arrivedSend one valid ping with an id you can recognise later:@@ -75,9 +85,13 @@ curl -i -X POST https://YOUR-HUB-URL \-d '{"schemaVersion":1,"instanceId":"smoke-test","sentAt":"2026-01-01T00:00:00.000Z","app":{"version":"0.0.0","isDev":true,"migrations":0},"runtime":{"node":"24.0.0","arch":"x64","platform":"linux","sqlite":"3.45.1","timezone":"UTC"},"config":{"authEnabled":true,"guestAccess":false,"registrationOpen":false,"defaultLocale":"en","updateCheckEnabled":true,"rateLimitOverrides":0},"scale":{"users":"1-5","calendars":"1-5","shifts":"1-5","presets":"0","notes":"0","bundles":"0","shares":"0","accessTokens":"0","signups":"0"},"features":{"externalSyncs":"0","calendarViewOverrides":"0","customFields":{"count":"0","types":[]},"archivedPresets":false,"splitShifts":false},"health":{"uptimeHours":1,"syncRuns24h":0,"syncFailures24h":0}}'```-You should get `HTTP/2 200`, and an `instance_ping` event for `smoke-test` should show up in PostHog → Activity within a minute.+You should get `HTTP/2 200`, and the ping should show up in:++```bash+npm run stats -- recent+```-**A `200` alone proves nothing.** The Worker also answers `200` to payloads it discards and to failed forwards, on purpose (see [What the Worker Does](#what-the-worker-does)). The event in PostHog is the real confirmation.+**A `200` alone proves nothing.** The Worker also answers `200` to payloads it discards and to failed writes, on purpose (see [What the Worker Does](#what-the-worker-does)). The row is the real confirmation.## Point Your Instances at It@@ -91,68 +105,129 @@ Use the bare URL without a path. Telemetry itself stays opt-in: the instance adm## Test It Locally-Create `.dev.vars` in this directory:--```env-POSTHOG_API_KEY=phc_your_key-POSTHOG_HOST=https://eu.i.posthog.com+```bash+npm run migrate:local+npm run dev```-Then start the dev server:+The Worker then listens on `http://localhost:8787` against a local SQLite file under `.wrangler/` — nothing touches the deployed database. Use the `curl` command from step 6 against that URL, then:```bash-npm run dev+npm run stats -- --local```-The Worker then listens on `http://localhost:8787`. Use the `curl` command from step 5 against that URL. `.dev.vars` is gitignored and only read by `wrangler dev`.+To check the public page and `/data.json` without sending pings by hand, seed deterministic fixture data instead:-Use a throwaway PostHog project for this. Local test pings otherwise mix into your real data.+```bash+npm run seed:local -- --reset+```++This deletes all local pings and inserts a fixed set of instances, never against `--remote`. `npx wrangler dev --test-scheduled` lets you then trigger the hourly cron by hand (`curl "http://localhost:8787/__scheduled?cron=17+*+*+*+*"`) to compute the aggregate from the seeded rows, and reload `/` or `/data.json` to see it.`npm run typecheck` checks the types.## What the Worker Does-| Request | Answer |-| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------- |-| Any method except `POST` | `405` |-| Body larger than 16 KB | `413` |-| Body is not valid JSON | `400` |-| Valid JSON that fails validation, or unknown version | `200`, discarded |-| Valid v1 payload | `200`, forwarded to PostHog. If PostHog rejects it, still `200`; the error message is logged. |+Routing resolves the path first, then the method:++| Request | Answer |+| --------------------------------------- | ------------------------------------------------------------------------------------ |+| `GET /` | The public statistics page (HTML). |+| `GET /data.json` | The same statistics as JSON. |+| `POST /` | Ingest — validated and written to D1, see below. |+| Any other path | `404` |+| A known path with the wrong method | `405` (e.g. `PUT /`, or `POST /data.json`) |-The senders are fire-and-forget and never retry, so a payload the Worker cannot forward is lost. That is by design: the Worker has no database, queue or request log.+For `POST /` specifically:-Every event is sent to PostHog with `$ip: null` and `$geoip_disable: true`, so PostHog stores neither the sender's IP nor a location derived from it. Person profiles are switched off as well, since an instance is not a person. Each field is passed on explicitly; nothing is spread from the request body.+| Request | Answer |+| ---------------------------------------------------- | ------------------------------------------------------------------------------------------ |+| Body larger than 16 KB | `413` |+| Body is not valid JSON | `400` |+| Valid JSON that fails validation, or unknown version | `200`, discarded |+| Valid v1 payload | `200`, written to D1. If the write fails, still `200`; the error message is logged. |-Cloudflare itself still sees the connecting IP in its own edge logs, like for any hosted Worker. The Worker code never reads or stores it.+The senders are fire-and-forget and never retry, so a payload the Worker cannot write is lost. That is by design: there is no queue and no request log.-## The PostHog Dashboards+Each field is written to its own column explicitly; nothing is spread from the request body, so an unexpected extra field cannot reach the database. The Worker never reads the connecting IP, and no column holds one — the stored `received_at` is the Worker's own clock, because the instance's `sent_at` can be skewed.-`scripts/create-posthog-dashboard.mjs` builds ready-made dashboards (growth, versions, environment, sizes, configuration, features, health) from the `instance_ping` events. It uses a **personal** API key, not the project key of the Worker.+Cloudflare itself still sees the connecting IP in its own edge logs, like for any hosted Worker.++Rows are kept indefinitely. There is no expiry job: at one ping per instance per day the table stays small for years (see [Cost](#cost)).++## The Public Statistics Page++`GET /` renders an HTML page — and `GET /data.json` the same data as JSON — from a single precomputed row in the `aggregates` table, so the read path never touches `instance_pings` (`src/aggregate.ts`, `src/page.ts`). A `[triggers]` cron in `wrangler.toml` (`17 * * * *`, off the hour on purpose) recomputes that row once an hour by running `buildAggregate()` and replacing it wholesale via `storeAggregate()` — a run in progress never leaves a partially updated page. Until the first cron run after deploy, the page shows a neutral empty state.++The aggregate only ever holds counts and shares grouped across instances: no instance id, no raw row, no per-instance timestamp appears in it, and `/data.json` is served with `Access-Control-Allow-Origin: *` since it carries nothing that needs to stay private to a caller. What exactly is published, including the fact that there is no small-group threshold, is documented in [`docs/TELEMETRY.md`](../docs/TELEMETRY.md#where-it-goes).++## Reading the Data++Every report in `queries/` is a plain SQL file. Run them all, or by name:++```bash+npm run stats # every report+npm run stats -- versions sizes # just these two+npm run stats -- --local # against the local database+```++| Report | What it answers | Window |+| -------------------- | ------------------------------------------------------------------ | ------------------------- |+| `instances-daily` | How many instances reported per day | All history |+| `instances-new` | New installs per day (first time an instance id was seen) | All history |+| `versions` | Which app versions are in use | Last 30 days |+| `environment` | Node, platform, arch, SQLite and timezone distribution | Last 30 days |+| `configuration` | Auth, guest access, registration, locale, update check | Last 30 days |+| `sizes` | Instance size buckets across users, calendars, shifts, … | Last 30 days |+| `features` | Feature adoption | Last 30 days |+| `custom-field-types` | Which custom field types are actually used | Last 30 days |+| `health` | Uptime and external-sync failures | Last 30 days |+| `recent` | The last 25 raw pings | All history |++Distribution reports (`versions`, `environment`, `configuration`, `sizes`, `features`, `custom-field-types`, `health`) read the `latest_pings` view — one row per instance, its newest ping — bounded to instances whose newest ping is within the last 30 days, the same window `src/aggregate.ts` uses for the public page and `/data.json`. That keeps the reports and the page describing the same population instead of quietly diverging (an instance that stopped reporting 90 days ago drops out of both at once). `instances-daily`, `instances-new` and `recent` are genuinely historical or raw and stay unbounded, reading `instance_pings` directly.++For anything ad hoc:```bash-cp .env.example .env # then fill in POSTHOG_PERSONAL_API_KEY-npm run dashboard -- --dry-run-npm run dashboard+npx wrangler d1 execute bettershift-telemetry --remote \+ --command "SELECT app_version, COUNT(*) FROM latest_pings GROUP BY app_version"```-The script also creates variants without dev builds, with dev builds only, and without your own test instances (`EXCLUDE_INSTANCE_IDS`, for example `smoke-test`). Run it again at any time: existing dashboards are matched by name and updated in place. All options are described in `.env.example` and in the header of the script.+The `queries/` reports and this ad-hoc SQL run against the raw `instance_pings` table and need `wrangler` access to the database. The public page at `GET /` and `GET /data.json` (see [The Public Statistics Page](#the-public-statistics-page)) is the equivalent view for anyone without that access — it needs no credentials because it never touches `instance_pings` directly, only the hourly precomputed aggregate.++## Cost++Everything here fits in the Cloudflare free plan, which covers 100,000 D1 row writes and 5 million row reads per day and 5 GB of storage. One instance writes one row per day, and a row is roughly 300 bytes — a hundred instances reporting daily for a year is about 36,000 rows, some 11 MB, well inside the write limit.-If your PostHog project is in the US region, set `POSTHOG_HOST=https://us.posthog.com` in `.env` (the app host, not the `i.` ingest host).+Reads are a separate budget, and the hourly cron is what spends it, not the writes above. `buildAggregate()` (`src/aggregate.ts`) resolves `latest_pings` — each instance's newest ping — once per run, then computes every breakdown (versions, environment, configuration, sizes, features, custom field types, health) from that one filtered, materialized set instead of re-deriving it per breakdown.++A run scans the ping history three times in total: once to materialize that set, once for `newLast30Days`, which needs each instance's *first* ping rather than its newest, and once for the 90-day history series. At the 36,000-row example above that is on the order of 80,000 rows per run, so 24 runs a day come to roughly **1.9M rows/day** — inside the 5M/day limit, but not by a margin worth spending carelessly.++Querying `latest_pings` independently for each of those seven breakdowns instead — as an earlier version of this query did — reads the full ping history again every time: roughly 9× per run, about 7.8M rows/day at the same example, over the free-tier read limit with no warning beyond the cron silently failing. `EXPLAIN QUERY PLAN` on `buildAggregate()`'s query shows a single `MATERIALIZE active` step feeding every breakdown, which is what keeps this bounded as the ping history grows. That `MATERIALIZED` hint is load-bearing: without it SQLite inlines the CTE per reference and the nine-scan behaviour returns silently, with identical results.++Since 1 September 2026 Cloudflare returns errors on free accounts that exceed the daily row limits instead of billing for them, so the plan cannot turn into a surprise invoice.## Operating It-- **Live logs:** `npx wrangler tail`. A failed forward appears as one line with the PostHog status, never the payload.-- **Nothing shows up in PostHog:** check, in this order, that the ping reached the Worker (`wrangler tail` while sending), that the key belongs to the region set in `POSTHOG_HOST`, and that the sender is a real v1 payload (an invalid one is discarded silently).-- **Rotating the key:** run `npx wrangler secret put POSTHOG_API_KEY` again. No redeploy is needed.+- **Live logs:** `npx wrangler tail`. A failed write appears as one line with the error message, never the payload.+- **Nothing shows up:** check, in this order, that the ping reached the Worker (`wrangler tail` while sending), that `npm run migrate` has been run against the deployed database, and that the sender is a real v1 payload (an invalid one is discarded silently).+- **Deleting an instance's data:** `npx wrangler d1 execute bettershift-telemetry --remote --command "DELETE FROM instance_pings WHERE instance_id = '<id>'"`.+## Project Layout```text-src/index.ts request handling: method, size, JSON, validation, forward+src/index.ts routing: path then method, size, JSON, validation, storesrc/schemas/v1.ts payload type and validator for schemaVersion 1-src/targets/posthog.ts maps a payload to a PostHog event-scripts/ create-posthog-dashboard.mjs-wrangler.toml Worker name, route and the PostHog host+src/targets/d1.ts maps a payload to a row and inserts it+src/aggregate.ts builds and (re)stores the public statistics, run hourly by the cron+src/page.ts renders the public statistics page from a stored aggregate+migrations/ D1 schema, applied with `npm run migrate`+ 0001_create_instance_pings.sql the raw ping table+ 0002_create_aggregates.sql the one-row precomputed aggregate table+queries/ one SQL report per file, run with `npm run stats`+scripts/stats.mjs runs the reports through wrangler and prints them+scripts/seed-local.mjs fills the local D1 with deterministic fixture pings+wrangler.toml Worker name, route, the D1 binding and the cron trigger```-This project is self-contained and does not import from the main app. `src/schemas/v1.ts` mirrors `TelemetryPayload` in [`lib/telemetry/schema.ts`](../lib/telemetry/schema.ts) by value, so a change to the payload has to be made in both places. Until a new `schemaVersion` exists, anything that is not a valid v1 payload is discarded.+This project is self-contained and does not import from the main app. `src/schemas/v1.ts` mirrors `TelemetryPayload` in [`lib/telemetry/schema.ts`](../lib/telemetry/schema.ts) by value, and `migrations/` mirrors it again as columns, so a change to the payload has to be made in all three places. Until a new `schemaVersion` exists, anything that is not a valid v1 payload is discarded.diff --git a/telemetry-hub/migrations/0001_create_instance_pings.sql b/telemetry-hub/migrations/0001_create_instance_pings.sqlnew file mode 100644index 0000000..1c2fbf1--- /dev/null+++ b/telemetry-hub/migrations/0001_create_instance_pings.sql@@ -0,0 +1,65 @@+-- One row per received ping, never updated. Mirrors PayloadV1 in src/schemas/v1.ts+-- column by column, so a payload change needs a new migration here too.+CREATE TABLE IF NOT EXISTS instance_pings (+ id INTEGER PRIMARY KEY AUTOINCREMENT,++ -- The Worker's own clock. sent_at is the instance's and can be skewed or wrong.+ received_at TEXT NOT NULL,+ ping_day TEXT NOT NULL,+ sent_at TEXT NOT NULL,+ instance_id TEXT,+ schema_version INTEGER NOT NULL,++ app_version TEXT NOT NULL,+ app_is_dev INTEGER NOT NULL,+ app_migrations INTEGER NOT NULL,++ runtime_node TEXT NOT NULL,+ runtime_arch TEXT NOT NULL,+ runtime_platform TEXT NOT NULL,+ runtime_sqlite TEXT NOT NULL,+ runtime_timezone TEXT NOT NULL,++ config_auth_enabled INTEGER NOT NULL,+ config_guest_access INTEGER NOT NULL,+ config_registration_open INTEGER NOT NULL,+ config_default_locale TEXT NOT NULL,+ config_update_check_enabled INTEGER NOT NULL,+ config_rate_limit_overrides INTEGER NOT NULL,++ scale_users TEXT NOT NULL,+ scale_calendars TEXT NOT NULL,+ scale_shifts TEXT NOT NULL,+ scale_presets TEXT NOT NULL,+ scale_notes TEXT NOT NULL,+ scale_bundles TEXT NOT NULL,+ scale_shares TEXT NOT NULL,+ scale_access_tokens TEXT NOT NULL,+ scale_signups TEXT NOT NULL,++ features_external_syncs TEXT NOT NULL,+ features_calendar_view_overrides TEXT NOT NULL,+ features_custom_fields_count TEXT NOT NULL,+ -- JSON array; query it with json_each(), see queries/custom-field-types.sql.+ features_custom_field_types TEXT NOT NULL,+ features_archived_presets INTEGER NOT NULL,+ features_split_shifts INTEGER NOT NULL,++ health_uptime_hours REAL NOT NULL,+ health_sync_runs_24h INTEGER NOT NULL,+ health_sync_failures_24h INTEGER NOT NULL+);++CREATE INDEX IF NOT EXISTS idx_instance_pings_ping_day ON instance_pings (ping_day);+CREATE INDEX IF NOT EXISTS idx_instance_pings_instance ON instance_pings (instance_id, received_at);+CREATE INDEX IF NOT EXISTS idx_instance_pings_version ON instance_pings (app_version);++-- The newest ping per instance -- what every distribution query reads, so a chatty+-- instance cannot outvote a quiet one. A ping without an instance_id cannot be+-- attributed to one, so each such ping counts as its own instance.+CREATE VIEW IF NOT EXISTS latest_pings AS+SELECT *+FROM instance_pings+WHERE id IN (+ SELECT MAX(id) FROM instance_pings GROUP BY COALESCE(instance_id, 'anon:' || id)+);diff --git a/telemetry-hub/migrations/0002_create_aggregates.sql b/telemetry-hub/migrations/0002_create_aggregates.sqlnew file mode 100644index 0000000..ec5ac28--- /dev/null+++ b/telemetry-hub/migrations/0002_create_aggregates.sql@@ -0,0 +1,7 @@+-- Precomputed public statistics, one row with key = 'public'. The cron handler+-- replaces it wholesale, so there is never a partially updated aggregate.+CREATE TABLE IF NOT EXISTS aggregates (+ key TEXT PRIMARY KEY,+ json TEXT NOT NULL,+ computed_at TEXT NOT NULL+);diff --git a/telemetry-hub/package.json b/telemetry-hub/package.jsonindex 20dedde..061a9dc 100644--- a/telemetry-hub/package.json+++ b/telemetry-hub/package.json@@ -5,7 +5,11 @@"scripts": {"dev": "wrangler dev","deploy": "wrangler deploy",- "dashboard": "node scripts/create-posthog-dashboard.mjs",+ "migrate": "wrangler d1 migrations apply bettershift-telemetry --remote",+ "migrate:local": "wrangler d1 migrations apply bettershift-telemetry --local",+ "seed:local": "node scripts/seed-local.mjs",+ "stats": "node scripts/stats.mjs",+ "tail": "wrangler tail","typecheck": "tsc --noEmit"},"devDependencies": {diff --git a/telemetry-hub/queries/configuration.sql b/telemetry-hub/queries/configuration.sqlnew file mode 100644index 0000000..c6aae2b--- /dev/null+++ b/telemetry-hub/queries/configuration.sql@@ -0,0 +1,17 @@+-- How instances are configured, for the current population -- instances whose+-- newest ping is within the last 30 days, matching the public page and /data.json.+-- Boolean settings show as 1 / 0.+-- json_object + json_each unpivots the columns; D1 caps a compound SELECT at 5 terms.+SELECT t.key AS setting, t.value AS value, COUNT(*) AS instances+FROM latest_pings AS p,+ json_each(json_object(+ 'auth_enabled', p.config_auth_enabled,+ 'guest_access', p.config_guest_access,+ 'registration_open', p.config_registration_open,+ 'update_check_enabled', p.config_update_check_enabled,+ 'default_locale', p.config_default_locale,+ 'rate_limit_overrides', p.config_rate_limit_overrides+ )) AS t+WHERE datetime(p.received_at) >= datetime('now', '-30 days')+GROUP BY t.key, t.value+ORDER BY setting, instances DESC;diff --git a/telemetry-hub/queries/custom-field-types.sql b/telemetry-hub/queries/custom-field-types.sqlnew file mode 100644index 0000000..7049a38--- /dev/null+++ b/telemetry-hub/queries/custom-field-types.sql@@ -0,0 +1,9 @@+-- Which custom field types are actually used, for the current population --+-- instances whose newest ping is within the last 30 days, matching the public page+-- and /data.json. json_each expands the stored JSON array, so an instance using+-- three types contributes to three rows.+SELECT t.value AS field_type, COUNT(DISTINCT p.id) AS instances+FROM latest_pings AS p, json_each(p.features_custom_field_types) AS t+WHERE datetime(p.received_at) >= datetime('now', '-30 days')+GROUP BY t.value+ORDER BY instances DESC, field_type;diff --git a/telemetry-hub/queries/environment.sql b/telemetry-hub/queries/environment.sqlnew file mode 100644index 0000000..94dde2d--- /dev/null+++ b/telemetry-hub/queries/environment.sql@@ -0,0 +1,16 @@+-- Runtime distribution across every environment dimension at once, for the current+-- population -- instances whose newest ping is within the last 30 days, matching+-- the public page and /data.json.+-- json_object + json_each unpivots the columns; D1 caps a compound SELECT at 5 terms.+SELECT t.key AS dimension, t.value AS value, COUNT(*) AS instances+FROM latest_pings AS p,+ json_each(json_object(+ 'node', p.runtime_node,+ 'platform', p.runtime_platform,+ 'arch', p.runtime_arch,+ 'sqlite', p.runtime_sqlite,+ 'timezone', p.runtime_timezone+ )) AS t+WHERE datetime(p.received_at) >= datetime('now', '-30 days')+GROUP BY t.key, t.value+ORDER BY dimension, instances DESC;diff --git a/telemetry-hub/queries/features.sql b/telemetry-hub/queries/features.sqlnew file mode 100644index 0000000..934d6c6--- /dev/null+++ b/telemetry-hub/queries/features.sql@@ -0,0 +1,16 @@+-- Feature adoption, for the current population -- instances whose newest ping is+-- within the last 30 days, matching the public page and /data.json. Boolean+-- features show as 1 / 0, the rest as buckets.+-- json_object + json_each unpivots the columns; D1 caps a compound SELECT at 5 terms.+SELECT t.key AS dimension, t.value AS value, COUNT(*) AS instances+FROM latest_pings AS p,+ json_each(json_object(+ 'external_syncs', p.features_external_syncs,+ 'calendar_view_overrides', p.features_calendar_view_overrides,+ 'custom_fields_count', p.features_custom_fields_count,+ 'archived_presets', p.features_archived_presets,+ 'split_shifts', p.features_split_shifts+ )) AS t+WHERE datetime(p.received_at) >= datetime('now', '-30 days')+GROUP BY t.key, t.value+ORDER BY dimension, instances DESC;diff --git a/telemetry-hub/queries/health.sql b/telemetry-hub/queries/health.sqlnew file mode 100644index 0000000..e2c0d6f--- /dev/null+++ b/telemetry-hub/queries/health.sql@@ -0,0 +1,10 @@+-- Uptime and external-sync health, for the current population -- instances whose+-- newest ping is within the last 30 days, matching the public page and /data.json.+SELECT COUNT(*) AS instances,+ ROUND(AVG(health_uptime_hours), 1) AS avg_uptime_hours,+ MAX(health_uptime_hours) AS max_uptime_hours,+ SUM(health_sync_runs_24h) AS sync_runs_24h,+ SUM(health_sync_failures_24h) AS sync_failures_24h,+ SUM(CASE WHEN health_sync_failures_24h > 0 THEN 1 ELSE 0 END) AS instances_with_failures+FROM latest_pings+WHERE datetime(received_at) >= datetime('now', '-30 days');diff --git a/telemetry-hub/queries/instances-daily.sql b/telemetry-hub/queries/instances-daily.sqlnew file mode 100644index 0000000..fe20847--- /dev/null+++ b/telemetry-hub/queries/instances-daily.sql@@ -0,0 +1,9 @@+-- Instances seen per day. A ping without an instance id counts as its own instance.+SELECT ping_day AS day,+ COUNT(DISTINCT COALESCE(instance_id, 'anon:' || id)) AS instances,+ COUNT(*) AS pings,+ SUM(CASE WHEN app_is_dev = 1 THEN 1 ELSE 0 END) AS dev_pings+FROM instance_pings+GROUP BY ping_day+ORDER BY day DESC+LIMIT 60;diff --git a/telemetry-hub/queries/instances-new.sql b/telemetry-hub/queries/instances-new.sqlnew file mode 100644index 0000000..7c1f697--- /dev/null+++ b/telemetry-hub/queries/instances-new.sql@@ -0,0 +1,11 @@+-- New installs per day: the first day each instance id was ever seen.+SELECT first_day AS day, COUNT(*) AS new_instances+FROM (+ SELECT instance_id, MIN(ping_day) AS first_day+ FROM instance_pings+ WHERE instance_id IS NOT NULL+ GROUP BY instance_id+)+GROUP BY first_day+ORDER BY day DESC+LIMIT 60;diff --git a/telemetry-hub/queries/recent.sql b/telemetry-hub/queries/recent.sqlnew file mode 100644index 0000000..7ff4195--- /dev/null+++ b/telemetry-hub/queries/recent.sql@@ -0,0 +1,6 @@+-- The most recent pings, unaggregated -- the quickest check that ingest still works.+SELECT received_at, instance_id, app_version, app_is_dev,+ runtime_platform, runtime_node, scale_users, scale_shifts, health_uptime_hours+FROM instance_pings+ORDER BY id DESC+LIMIT 25;diff --git a/telemetry-hub/queries/sizes.sql b/telemetry-hub/queries/sizes.sqlnew file mode 100644index 0000000..81fb9c6--- /dev/null+++ b/telemetry-hub/queries/sizes.sql@@ -0,0 +1,20 @@+-- Instance sizes, for the current population -- instances whose newest ping is+-- within the last 30 days, matching the public page and /data.json. Values are the+-- buckets documented in docs/TELEMETRY.md, never exact counts.+-- json_object + json_each unpivots the columns; D1 caps a compound SELECT at 5 terms.+SELECT t.key AS dimension, t.value AS bucket, COUNT(*) AS instances+FROM latest_pings AS p,+ json_each(json_object(+ 'users', p.scale_users,+ 'calendars', p.scale_calendars,+ 'shifts', p.scale_shifts,+ 'presets', p.scale_presets,+ 'notes', p.scale_notes,+ 'bundles', p.scale_bundles,+ 'shares', p.scale_shares,+ 'access_tokens', p.scale_access_tokens,+ 'signups', p.scale_signups+ )) AS t+WHERE datetime(p.received_at) >= datetime('now', '-30 days')+GROUP BY t.key, t.value+ORDER BY dimension, instances DESC;diff --git a/telemetry-hub/queries/versions.sql b/telemetry-hub/queries/versions.sqlnew file mode 100644index 0000000..ea1d409--- /dev/null+++ b/telemetry-hub/queries/versions.sql@@ -0,0 +1,11 @@+-- App versions in use, one vote per instance (its newest ping). Bounded to the+-- current population -- an instance whose newest ping is more than 30 days old+-- does not count, matching the public page and /data.json.+SELECT app_version,+ COUNT(*) AS instances,+ SUM(CASE WHEN app_is_dev = 1 THEN 1 ELSE 0 END) AS dev_builds,+ MAX(received_at) AS last_seen+FROM latest_pings+WHERE datetime(received_at) >= datetime('now', '-30 days')+GROUP BY app_version+ORDER BY instances DESC, app_version DESC;diff --git a/telemetry-hub/scripts/create-posthog-dashboard.mjs b/telemetry-hub/scripts/create-posthog-dashboard.mjsdeleted file mode 100644index c0c91b7..0000000--- a/telemetry-hub/scripts/create-posthog-dashboard.mjs+++ /dev/null@@ -1,456 +0,0 @@-#!/usr/bin/env node-// Creates the "BetterShift Instanzen" dashboards in PostHog from the instance_ping events.-//-// npm run dashboard [-- --dry-run]-//-// Settings come from telemetry-hub/.env (copy .env.example) or the environment; the environment wins.-// Only POSTHOG_PERSONAL_API_KEY is required, everything else has a default:-// POSTHOG_PERSONAL_API_KEY personal key (phx_..., not the phc_ project key) with the-// dashboard:write and insight:write scopes, plus project:read for-// finding the project id-// POSTHOG_PROJECT_ID only needed if the key can see more than one project-// POSTHOG_HOST app host, default https://eu.posthog.com (not the eu.i. ingest host)-// DASHBOARD_NAME default "BetterShift Instanzen"-// DASHBOARD_VARIANTS all,no-dev,dev-only,no-ids (default: all four)-// EXCLUDE_INSTANCE_IDS id1,id2 for the no-ids dashboard-//-// One run builds up to four dashboards with the same insights and a different filter each:-// all "<name>" no filter-// no-dev "<name> · ohne Dev-Builds" instances running a dev build left out-// dev-only "<name> · nur Dev-Builds" only instances running a dev build-// no-ids "<name> · ohne Test-Instanzen" EXCLUDE_INSTANCE_IDS left out-// A dashboard that already exists (matched by name) is updated in place, insight by insight,-// so new instance ids can be added later by running the script again.--import { existsSync } from "node:fs";-import { fileURLToPath } from "node:url";--// loadEnvFile never overrides variables that are already set.-const envFile = fileURLToPath(new URL("../.env", import.meta.url));-if (existsSync(envFile)) process.loadEnvFile(envFile);--const args = new Set(process.argv.slice(2));-const dryRun = args.has("--dry-run");--const HOST = (process.env.POSTHOG_HOST || "https://eu.posthog.com").replace(/\/$/, "");-const KEY = process.env.POSTHOG_PERSONAL_API_KEY;-let PROJECT = process.env.POSTHOG_PROJECT_ID;-const DASHBOARD_NAME = process.env.DASHBOARD_NAME || "BetterShift Instanzen";-const EVENT = "instance_ping";-const TAG = "bettershift-telemetry";--function fail(message) {- console.error(`Fehler: ${message}`);- process.exit(1);-}--const excluded = (process.env.EXCLUDE_INSTANCE_IDS || "")- .split(",")- .map((id) => id.trim())- .filter(Boolean);-// The ids end up inside SQL, so only the characters of a UUID are accepted.-for (const id of excluded) {- if (!/^[A-Za-z0-9-]+$/.test(id)) fail(`Ungültige Instanz-ID in EXCLUDE_INSTANCE_IDS: ${id}`);-}--// Dev builds send like any other; app_is_dev only exists to tell them apart here.-const IS_DEV = `toString(coalesce(properties.app_is_dev, 'false')) IN ('true', '1')`;-const VARIANTS = {- all: { suffix: "", condition: "" },- "no-dev": { suffix: " · ohne Dev-Builds", condition: `NOT (${IS_DEV})` },- "dev-only": { suffix: " · nur Dev-Builds", condition: IS_DEV },- "no-ids": {- suffix: " · ohne Test-Instanzen",- condition: excluded.length ? `distinct_id NOT IN (${excluded.map((id) => `'${id}'`).join(", ")})` : "",- },-};--const requested = (process.env.DASHBOARD_VARIANTS || "").split(",").map((v) => v.trim()).filter(Boolean);-for (const key of requested) {- if (!VARIANTS[key]) fail(`Unbekannte Variante in DASHBOARD_VARIANTS: ${key} (erlaubt: ${Object.keys(VARIANTS).join(", ")})`);-}-if (requested.includes("no-ids") && !excluded.length) fail("Variante no-ids braucht EXCLUDE_INSTANCE_IDS.");-const variantKeys = (requested.length ? requested : Object.keys(VARIANTS)).filter(- (key) => key !== "no-ids" || excluded.length-);-if (!requested.length && !excluded.length) console.log("Ohne EXCLUDE_INSTANCE_IDS entfällt das Dashboard \"ohne Test-Instanzen\".");---/** All insights of one dashboard; `condition` is the HogQL filter of its variant (may be empty). */-function buildInsights(condition) {- const andNotExcluded = condition ? ` AND ${condition}` : "";- const trendProperties = condition ? [{ type: "hogql", key: condition }] : [];-- // ---------------------------------------------------------------------------- // Query builders- // ----------------------------------------------------------------------------- /** Trends over unique instances (distinct_id), never over raw pings. */- function trends({ display, interval = "week", from = "-30d", math = "dau", mathProperty, breakdown }) {- return {- kind: "InsightVizNode",- source: {- kind: "TrendsQuery",- series: [- {- kind: "EventsNode",- event: EVENT,- name: EVENT,- math,- ...(mathProperty && { math_property: mathProperty, math_property_type: "event_properties" }),- },- ],- interval,- dateRange: { date_from: from },- properties: trendProperties,- filterTestAccounts: false,- ...(breakdown && { breakdownFilter: { breakdowns: [{ property: breakdown, type: "event" }] } }),- trendsFilter: { display },- version: 2,- },- };- }-- function sql(query) {- return { kind: "DataVisualizationNode", source: { kind: "HogQLQuery", query: query.trim() } };- }-- /** Latest ping per instance in the last 30 days, so every instance counts once. */- function latestPerInstance(columns) {- return `- SELECT distinct_id, ${columns}- FROM events- WHERE event = '${EVENT}' AND timestamp > now() - INTERVAL 30 DAY${andNotExcluded}- GROUP BY distinct_id`;- }-- const bars = (name, description, breakdown) => ({- name,- description,- query: trends({ display: "ActionsBarValue", breakdown }),- });- const pie = (name, description, breakdown) => ({- name,- description,- query: trends({ display: "ActionsPie", breakdown }),- });-- // A boolean breakdown would only show a bare instance count, so yes/no flags get an explicit table.- const yesNo = (name, property) => ({- name,- description: "Jede Instanz zählt einmal, mit dem Wert ihres letzten Pings (letzte 30 Tage).",- query: sql(`- SELECT multiIf(flag IN ('true', '1'), 'Ja', flag IN ('false', '0'), 'Nein', 'Unbekannt') AS Wert, count() AS Instanzen- FROM (${latestPerInstance(`toString(argMax(properties.${property}, timestamp)) AS flag`)})- GROUP BY Wert- ORDER BY Instanzen DESC`),- });-- const BY_INSTANCE = "Aktive Instanzen der letzten 30 Tage, aufgeteilt nach dem gemeldeten Wert.";-- // ---------------------------------------------------------------------------- // Insights, in dashboard order- // ----------------------------------------------------------------------------- const insights = [- // Growth- {- name: "Wachstum · Aktive Instanzen (7 Tage)",- description: "Instanzen, die in den letzten 7 Tagen mindestens einmal gemeldet haben.",- query: trends({ display: "BoldNumber", from: "-7d", interval: "day" }),- },- {- name: "Wachstum · Bekannte Instanzen (gesamt)",- description: "Alle Instanzen, die je einen Ping gesendet haben.",- query: sql(`- SELECT count(DISTINCT distinct_id) AS instances- FROM events- WHERE event = '${EVENT}'${andNotExcluded}`),- },- {- name: "Wachstum · Neue Instanzen (30 Tage)",- description: "Instanzen, deren erster Ping in den letzten 30 Tagen liegt.",- query: sql(`- SELECT count() AS new_instances- FROM (- SELECT distinct_id, min(timestamp) AS first_seen- FROM events- WHERE event = '${EVENT}'${andNotExcluded}- GROUP BY distinct_id- HAVING first_seen > now() - INTERVAL 30 DAY- )`),- },- {- name: "Wachstum · Aktive Instanzen pro Woche",- description: "Eindeutige Instanzen je Woche, unabhängig von der Zahl der Pings.",- query: trends({ display: "ActionsLineGraph", from: "-180d" }),- },- {- name: "Wachstum · Neu, wiederkehrend, inaktiv",- description: "Lebenszyklus der Instanzen je Woche.",- query: {- kind: "InsightVizNode",- source: {- kind: "LifecycleQuery",- series: [{ kind: "EventsNode", event: EVENT, name: EVENT, math: "total" }],- interval: "week",- dateRange: { date_from: "-90d" },- properties: trendProperties,- filterTestAccounts: false,- lifecycleFilter: { showLegend: true },- },- },- },- {- name: "Wachstum · Verbleib (Retention)",- description: "Wie viele Instanzen nach ihrem ersten Ping in den Folgewochen weiter melden.",- query: {- kind: "InsightVizNode",- source: {- kind: "RetentionQuery",- dateRange: { date_from: "-12w" },- properties: trendProperties,- filterTestAccounts: false,- retentionFilter: {- period: "Week",- totalIntervals: 12,- retentionType: "retention_first_time",- targetEntity: { id: EVENT, name: EVENT, type: "events" },- returningEntity: { id: EVENT, name: EVENT, type: "events" },- },- },- },- },-- // Versions- bars("Version · Aktive Instanzen je Version", BY_INSTANCE, "app_version"),- {- name: "Version · Aktuelle Version je Instanz",- description: "Jede Instanz zählt einmal, mit ihrer zuletzt gemeldeten Version.",- query: sql(`- SELECT version, count() AS instances- FROM (${latestPerInstance("argMax(properties.app_version, timestamp) AS version")})- GROUP BY version- ORDER BY instances DESC`),- },- yesNo("Version · Dev-Builds", "app_is_dev"),- bars("Version · Datenbank-Migrationen", `${BY_INSTANCE} Zeigt, wie aktuell das Schema ist.`, "app_migrations"),-- // Environment- bars("Umgebung · Node.js", BY_INSTANCE, "runtime_node"),- pie("Umgebung · Plattform", BY_INSTANCE, "runtime_platform"),- pie("Umgebung · Architektur", BY_INSTANCE, "runtime_arch"),- bars("Umgebung · SQLite", BY_INSTANCE, "runtime_sqlite"),- bars("Umgebung · Zeitzone", BY_INSTANCE, "runtime_timezone"),- pie("Umgebung · Standardsprache", BY_INSTANCE, "config_default_locale"),-- // Scale (buckets)- bars("Größe · Nutzer", `${BY_INSTANCE} Größenklassen, keine genauen Zahlen.`, "scale_users"),- bars("Größe · Kalender", `${BY_INSTANCE} Größenklassen, keine genauen Zahlen.`, "scale_calendars"),- bars("Größe · Schichten", `${BY_INSTANCE} Größenklassen, keine genauen Zahlen.`, "scale_shifts"),- bars("Größe · Schichtvorlagen", `${BY_INSTANCE} Größenklassen, keine genauen Zahlen.`, "scale_presets"),- bars("Größe · Notizen", `${BY_INSTANCE} Größenklassen, keine genauen Zahlen.`, "scale_notes"),- bars("Größe · Berechtigungs-Bundles", `${BY_INSTANCE} Größenklassen, keine genauen Zahlen.`, "scale_bundles"),- bars("Größe · Freigaben", `${BY_INSTANCE} Größenklassen, keine genauen Zahlen.`, "scale_shares"),- bars("Größe · Freigabe-Links", `${BY_INSTANCE} Größenklassen, keine genauen Zahlen.`, "scale_access_tokens"),- bars("Größe · Schicht-Anmeldungen", `${BY_INSTANCE} Größenklassen, keine genauen Zahlen.`, "scale_signups"),-- // Configuration- yesNo("Konfiguration · Authentifizierung aktiv", "config_auth_enabled"),- yesNo("Konfiguration · Gastzugang erlaubt", "config_guest_access"),- yesNo("Konfiguration · Registrierung offen", "config_registration_open"),- yesNo("Konfiguration · Update-Prüfung aktiv", "config_update_check_enabled"),- bars("Konfiguration · Rate-Limit-Überschreibungen", `${BY_INSTANCE} Anzahl gesetzter RATE_LIMIT_*-Variablen.`, "config_rate_limit_overrides"),-- // Features- bars("Funktionen · Externe Synchronisierungen", `${BY_INSTANCE} Größenklassen.`, "features_external_syncs"),- bars("Funktionen · Kalender mit eigener Ansicht", `${BY_INSTANCE} Größenklassen.`, "features_calendar_view_overrides"),- bars("Funktionen · Eigene Felder (Anzahl)", `${BY_INSTANCE} Größenklassen.`, "features_custom_fields_count"),- {- name: "Funktionen · Typen eigener Felder",- description: "Wie viele Instanzen welchen Feldtyp nutzen (letzte 30 Tage).",- query: sql(`- SELECT field_type, count(DISTINCT distinct_id) AS instances- FROM (- SELECT- distinct_id,- arrayJoin(JSONExtractArrayRaw(coalesce(properties.features_custom_field_types, '[]'))) AS raw_type,- trim(BOTH '"' FROM raw_type) AS field_type- FROM events- WHERE event = '${EVENT}' AND timestamp > now() - INTERVAL 30 DAY${andNotExcluded}- )- GROUP BY field_type- ORDER BY instances DESC`),- },- yesNo("Funktionen · Archivierte Vorlagen", "features_archived_presets"),- yesNo("Funktionen · Geteilte Schichten", "features_split_shifts"),-- // Operations- {- name: "Betrieb · Ø Laufzeit (Stunden)",- description: "Durchschnittliche Zeit seit dem letzten Neustart, je Woche.",- query: trends({ display: "ActionsLineGraph", math: "avg", mathProperty: "health_uptime_hours", from: "-90d" }),- },- {- name: "Betrieb · Ø Sync-Läufe je Instanz (24 h)",- description: "Durchschnitt der gemeldeten Sync-Läufe der letzten 24 Stunden, je Woche.",- query: trends({ display: "ActionsLineGraph", math: "avg", mathProperty: "health_sync_runs_24h", from: "-90d" }),- },- {- name: "Betrieb · Ø Sync-Fehler je Instanz (24 h)",- description: "Durchschnitt der gemeldeten Sync-Fehler der letzten 24 Stunden, je Woche.",- query: trends({ display: "ActionsLineGraph", math: "avg", mathProperty: "health_sync_failures_24h", from: "-90d" }),- },- {- name: "Betrieb · Instanzen mit Sync-Fehlern",- description: "Instanzen, deren letzter Ping Sync-Fehler in den letzten 24 Stunden meldet.",- query: sql(`- SELECT distinct_id, version, sync_runs, sync_failures, last_seen- FROM (${latestPerInstance(`- argMax(properties.app_version, timestamp) AS version,- argMax(properties.health_sync_runs_24h, timestamp) AS sync_runs,- argMax(properties.health_sync_failures_24h, timestamp) AS sync_failures,- max(timestamp) AS last_seen`)})- WHERE sync_failures > 0- ORDER BY sync_failures DESC`),- },-- // Overview table- {- name: "Instanzen · Letzter Stand je Instanz",- description: "Eine Zeile je Instanz mit den Werten ihres letzten Pings.",- query: sql(`- SELECT- distinct_id,- max(timestamp) AS last_seen,- argMax(properties.app_version, timestamp) AS version,- argMax(properties.runtime_platform, timestamp) AS platform,- argMax(properties.runtime_arch, timestamp) AS arch,- argMax(properties.runtime_node, timestamp) AS node,- argMax(properties.scale_users, timestamp) AS users,- argMax(properties.scale_calendars, timestamp) AS calendars,- argMax(properties.scale_shifts, timestamp) AS shifts,- argMax(properties.health_uptime_hours, timestamp) AS uptime_hours- FROM events- WHERE event = '${EVENT}' AND timestamp > now() - INTERVAL 30 DAY${andNotExcluded}- GROUP BY distinct_id- ORDER BY last_seen DESC`),- },- ];-- return insights;-}--// ----------------------------------------------------------------------------// API-// -----------------------------------------------------------------------------async function api(method, path, body) {- return request(method, `/api/projects/${PROJECT}${path}`, body);-}--async function request(method, path, body) {- const response = await fetch(`${HOST}${path}`, {- method,- headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" },- body: body ? JSON.stringify(body) : undefined,- });- const text = await response.text();- if (!response.ok) throw new Error(`${method} ${path} -> ${response.status} ${text.slice(0, 300)}`);- return text ? JSON.parse(text) : {};-}--/** Uses the only project the key can see; with several (or no project:read scope) the id must be set. */-async function findProject() {- let projects;- try {- projects = (await request("GET", "/api/projects/")).results ?? [];- } catch (error) {- fail(`POSTHOG_PROJECT_ID fehlt und die Projektliste ließ sich nicht laden (${error.message}).`);- }- if (projects.length !== 1) {- const list = projects.map((p) => ` ${p.id} ${p.name}`).join("\n");- fail(`POSTHOG_PROJECT_ID fehlt und der Key sieht ${projects.length} Projekte:\n${list}`);- }- console.log(`Projekt "${projects[0].name}" (ID ${projects[0].id}) automatisch gewählt.`);- return projects[0].id;-}--/** Creates or updates the insights of one dashboard; returns the names that failed. */-async function syncInsights(dashboardId, existingTiles, insights) {- const byName = new Map(existingTiles.filter((t) => t.insight).map((t) => [t.insight.name, t.insight.id]));- const failures = [];- for (const insight of insights) {- const id = byName.get(insight.name);- try {- if (id) {- await api("PATCH", `/insights/${id}/`, { description: insight.description, query: insight.query });- } else {- await api("POST", "/insights/", {- name: insight.name,- description: insight.description,- query: insight.query,- dashboards: [dashboardId],- tags: [TAG],- });- }- console.log(` ${id ? "aktual." : "neu "} ${insight.name}`);- } catch (error) {- failures.push(insight.name);- console.error(` FEHLER ${insight.name}\n ${error.message}`);- }- }- return failures;-}--async function main() {- const plan = variantKeys.map((key) => ({- key,- name: DASHBOARD_NAME + VARIANTS[key].suffix,- insights: buildInsights(VARIANTS[key].condition),- }));-- if (dryRun) {- for (const { key, name, insights } of plan) {- console.log(`Trockenlauf: ${insights.length} Insights für "${name}" (${key})`);- }- for (const insight of plan[0].insights) console.log(` - ${insight.name}`);- if (excluded.length) console.log(`Ausgeschlossene Instanzen: ${excluded.length}`);- return;- }-- if (!KEY) {- fail("POSTHOG_PERSONAL_API_KEY fehlt. In telemetry-hub/.env eintragen (Vorlage: .env.example) oder als Variable setzen.");- }- if (!PROJECT) PROJECT = await findProject();-- const existing = (await api("GET", "/dashboards/?limit=200")).results ?? [];- let failed = 0;-- for (const { name, insights } of plan) {- const found = existing.find((d) => d.name === name && !d.deleted);- let dashboard = found;- let tiles = [];- if (found) {- // The list endpoint carries no tiles, so fetch the dashboard itself.- tiles = (await api("GET", `/dashboards/${found.id}/`)).tiles ?? [];- console.log(`\nDashboard "${name}" existiert (ID ${found.id}), wird aktualisiert.`);- } else {- dashboard = await api("POST", "/dashboards/", {- name,- description: "Anonyme Instanz-Pings von BetterShift. Gezählt werden eindeutige Instanzen, nicht Pings.",- tags: [TAG],- });- console.log(`\nDashboard "${name}" angelegt (ID ${dashboard.id}).`);- }-- const failures = await syncInsights(dashboard.id, tiles, insights);- failed += failures.length;- console.log(`${insights.length - failures.length} von ${insights.length} Insights synchronisiert.`);- console.log(`${HOST}/project/${PROJECT}/dashboard/${dashboard.id}`);- }-- if (failed) process.exit(1);-}--main().catch((error) => fail(error.message));diff --git a/telemetry-hub/scripts/seed-local.mjs b/telemetry-hub/scripts/seed-local.mjsnew file mode 100755index 0000000..607c317--- /dev/null+++ b/telemetry-hub/scripts/seed-local.mjs@@ -0,0 +1,72 @@+#!/usr/bin/env node+// Fills the LOCAL D1 with deterministic pings so the statistics page can be+// verified against known numbers. Never run against --remote.+//+// npm run seed:local add the fixture rows+// npm run seed:local -- --reset delete all pings first++import { spawnSync } from "node:child_process";+import { readFileSync } from "node:fs";+import { dirname, join } from "node:path";+import { fileURLToPath } from "node:url";++const root = join(dirname(fileURLToPath(import.meta.url)), "..");+const config = readFileSync(join(root, "wrangler.toml"), "utf8");+const databaseName = config.match(/^\s*database_name\s*=\s*"([^"]+)"/m)?.[1];+if (!databaseName) {+ console.error("No database_name found in wrangler.toml.");+ process.exit(1);+}++const day = (offset) => {+ const d = new Date();+ d.setUTCDate(d.getUTCDate() - offset);+ return d.toISOString();+};++// 6 instances: four current, one 90 days stale, one with a unique timezone (n=1).+const fixtures = [+ { id: "inst-a", ago: 0, version: "1.4.2", dev: 0, node: "24.0.0", tz: "Europe/Berlin", users: "6-20" },+ { id: "inst-b", ago: 0, version: "1.4.2", dev: 0, node: "24.0.0", tz: "Europe/Berlin", users: "1-5" },+ { id: "inst-c", ago: 2, version: "1.4.2", dev: 0, node: "24.0.0", tz: "Europe/Berlin", users: "1-5" },+ { id: "inst-d", ago: 10, version: "1.3.0", dev: 1, node: "22.11.0", tz: "Pacific/Chatham", users: "1-5" },+ { id: "inst-e", ago: 12, version: "1.4.2", dev: 0, node: "24.0.0", tz: "Europe/Berlin", users: "1-5" },+ { id: "inst-stale", ago: 90, version: "0.9.0", dev: 0, node: "20.0.0", tz: "UTC", users: "1-5" },+];++const values = fixtures+ .map((f) => {+ const at = day(f.ago);+ return `('${at}','${at.slice(0, 10)}','${at}','${f.id}',1,'${f.version}',${f.dev},30,'${f.node}','x64','linux','3.46.0','${f.tz}',1,0,0,'de',1,0,'${f.users}','1-5','1-5','0','0','0','0','0','0','0','0','0','[]',0,0,100.0,1,0)`;+ })+ .join(",\n");++const columns = [+ "received_at", "ping_day", "sent_at", "instance_id", "schema_version",+ "app_version", "app_is_dev", "app_migrations",+ "runtime_node", "runtime_arch", "runtime_platform", "runtime_sqlite", "runtime_timezone",+ "config_auth_enabled", "config_guest_access", "config_registration_open",+ "config_default_locale", "config_update_check_enabled", "config_rate_limit_overrides",+ "scale_users", "scale_calendars", "scale_shifts", "scale_presets", "scale_notes",+ "scale_bundles", "scale_shares", "scale_access_tokens", "scale_signups",+ "features_external_syncs", "features_calendar_view_overrides",+ "features_custom_fields_count", "features_custom_field_types",+ "features_archived_presets", "features_split_shifts",+ "health_uptime_hours", "health_sync_runs_24h", "health_sync_failures_24h",+].join(", ");++const statements = [];+if (process.argv.includes("--reset")) statements.push("DELETE FROM instance_pings;");+statements.push(`INSERT INTO instance_pings (${columns}) VALUES\n${values};`);++const result = spawnSync(+ "npx",+ ["wrangler", "d1", "execute", databaseName, "--local", "--command", statements.join("\n"), "--json"],+ { cwd: root, encoding: "utf8" },+);++if (result.status !== 0) {+ console.error(result.stdout?.trim() || result.stderr?.trim());+ process.exit(1);+}+console.log(`Seeded ${fixtures.length} pings into the local database.`);diff --git a/telemetry-hub/scripts/stats.mjs b/telemetry-hub/scripts/stats.mjsnew file mode 100644index 0000000..6be3dc0--- /dev/null+++ b/telemetry-hub/scripts/stats.mjs@@ -0,0 +1,107 @@+#!/usr/bin/env node+// Runs the canned reports in queries/ against the D1 database and prints them.+//+// npm run stats every report, against the deployed database+// npm run stats -- versions sizes only those two+// npm run stats -- --local against the local `wrangler dev` database+//+// Requires `npx wrangler login` for the remote database.++import { spawnSync } from "node:child_process";+import { readFileSync, readdirSync } from "node:fs";+import { dirname, join } from "node:path";+import { fileURLToPath } from "node:url";++const root = join(dirname(fileURLToPath(import.meta.url)), "..");+const queriesDir = join(root, "queries");++const args = process.argv.slice(2);+const local = args.includes("--local");+const wanted = args.filter((arg) => !arg.startsWith("--"));++const config = readFileSync(join(root, "wrangler.toml"), "utf8");+const databaseName = config.match(/^\s*database_name\s*=\s*"([^"]+)"/m)?.[1];+if (!databaseName) {+ console.error("No database_name found in wrangler.toml.");+ process.exit(1);+}++const available = readdirSync(queriesDir)+ .filter((file) => file.endsWith(".sql"))+ .map((file) => file.replace(/\.sql$/, ""))+ .sort();++const unknown = wanted.filter((name) => !available.includes(name));+if (unknown.length > 0) {+ console.error(`Unknown report(s): ${unknown.join(", ")}`);+ console.error(`Available: ${available.join(", ")}`);+ process.exit(1);+}++const reports = wanted.length > 0 ? wanted : available;+let failed = false;++for (const name of reports) {+ // --command, not --file: against a remote database wrangler treats a file as an+ // import and reports only statistics, never the selected rows. Comment lines are+ // stripped because a value starting with "--" is parsed as further CLI flags.+ const sql = readFileSync(join(queriesDir, `${name}.sql`), "utf8")+ .split("\n")+ .filter((line) => !line.trim().startsWith("--"))+ .join("\n")+ .trim();++ const result = spawnSync(+ "npx",+ [+ "wrangler",+ "d1",+ "execute",+ databaseName,+ local ? "--local" : "--remote",+ "--command",+ sql,+ "--json",+ ],+ { cwd: root, encoding: "utf8" },+ );++ console.log(`\n── ${name} ${"─".repeat(Math.max(0, 60 - name.length))}`);++ // A rejected query is reported as a `{ error }` object on stdout, so stdout is+ // worth parsing even when the exit code is non-zero -- stderr only carries noise.+ const start = result.stdout?.search(/[[{]/) ?? -1;+ let parsed;+ if (start !== -1) {+ try {+ parsed = JSON.parse(result.stdout.slice(start));+ } catch {+ parsed = undefined;+ }+ }++ if (parsed && !Array.isArray(parsed)) {+ failed = true;+ console.error(parsed.error?.text ?? JSON.stringify(parsed));+ continue;+ }++ if (!parsed) {+ failed = true;+ console.error(+ result.stderr?.trim() ||+ result.stdout?.trim() ||+ `wrangler exited with ${result.status}`,+ );+ continue;+ }++ const rows = parsed.flatMap((entry) => entry.results ?? []);+ if (rows.length === 0) {+ console.log("(no rows)");+ } else {+ console.table(rows);+ }+}++process.exit(failed ? 1 : 0);diff --git a/telemetry-hub/src/aggregate.ts b/telemetry-hub/src/aggregate.tsnew file mode 100644index 0000000..8d4cea8--- /dev/null+++ b/telemetry-hub/src/aggregate.ts@@ -0,0 +1,333 @@+export interface DistributionEntry {+ value: string;+ instances: number;+ share: number;+ /** Set only on the size-guard remainder entry; see MAX_DISTRIBUTION_ENTRIES. */+ remainder?: true;+}++export interface VersionEntry extends DistributionEntry {+ devInstances: number;+}++export interface HistoryPoint {+ day: string;+ instances: number;+}++export interface PublicAggregate {+ computedAt: string;+ instances: { total: number; activeLast7Days: number; newLast30Days: number };+ history: HistoryPoint[];+ versions: VersionEntry[];+ environment: Record<string, DistributionEntry[]>;+ configuration: Record<string, DistributionEntry[]>;+ sizes: Record<string, DistributionEntry[]>;+ features: Record<string, DistributionEntry[]>;+ customFieldTypes: DistributionEntry[];+ health: { avgUptimeHours: number; instancesWithSyncFailures: number };+}++const ACTIVE_WINDOW_DAYS = 30;+const HISTORY_DAYS = 90;+// Payload string fields are unbounded and the hub is unauthenticated.+const MAX_VALUE_LENGTH = 64;+// SIZE GUARD, NOT A PRIVACY RULE: a single ping can carry arbitrarily many+// distinct values (features.customFields.types is an unbounded string array), and+// every distinct value becomes a row in /data.json and a bar on the page. The long+// tail past this many entries is summed into one remainder entry so both stay+// bounded. Nothing is hidden for being rare — small counts are published as-is.+const MAX_DISTRIBUTION_ENTRIES = 25;++const remainderLabel = (values: number): string => `other (${values} more values)`;++const iso = (now: Date, daysAgo: number): string =>+ new Date(now.getTime() - daysAgo * 86_400_000).toISOString();++const truncate = (value: string): string =>+ value.length > MAX_VALUE_LENGTH ? value.slice(0, MAX_VALUE_LENGTH) : value;++interface RawGroup {+ key: string;+ value: string | number | null;+ instances: number;+}++// The GROUP BY queries have no ORDER BY, so entries tied on count could otherwise+// straddle the cap differently between runs — by name one hour, in the remainder+// the next. The value breaks the tie deterministically.+const byCountThenValue = (+ a: { value: string; instances: number },+ b: { value: string; instances: number },+): number => b.instances - a.instances || (a.value < b.value ? -1 : a.value > b.value ? 1 : 0);++const share = (instances: number, base: number): number =>+ base > 0 ? Math.round((instances / base) * 10_000) / 10_000 : 0;++/** Sorts by size, caps the long tail (see MAX_DISTRIBUTION_ENTRIES), adds shares. */+function collapse(rows: RawGroup[], base: number): DistributionEntry[] {+ const sorted = rows+ .map((row) => ({ value: truncate(String(row.value)), instances: row.instances }))+ .sort(byCountThenValue);++ const kept = sorted.slice(0, MAX_DISTRIBUTION_ENTRIES);+ const tail = sorted.slice(MAX_DISTRIBUTION_ENTRIES);++ const entries: DistributionEntry[] = kept.map((row) => ({+ value: row.value,+ instances: row.instances,+ share: share(row.instances, base),+ }));++ if (tail.length > 0) {+ const instances = tail.reduce((sum, row) => sum + row.instances, 0);+ entries.push({+ value: remainderLabel(tail.length),+ instances,+ share: share(instances, base),+ remainder: true,+ });+ }+ return entries;+}++/** Same tail cap as collapse(), plus a dev-instance count carried per entry. */+function buildVersions(+ rows: Array<{ value: string; instances: number; dev_instances: number }>,+ base: number,+): VersionEntry[] {+ const sorted = rows+ .map((row) => ({+ value: truncate(row.value),+ instances: row.instances,+ devInstances: Number(row.dev_instances ?? 0),+ }))+ .sort(byCountThenValue);++ const kept = sorted.slice(0, MAX_DISTRIBUTION_ENTRIES);+ const tail = sorted.slice(MAX_DISTRIBUTION_ENTRIES);++ const entries: VersionEntry[] = kept.map((row) => ({+ value: row.value,+ instances: row.instances,+ share: share(row.instances, base),+ devInstances: row.devInstances,+ }));++ if (tail.length > 0) {+ const instances = tail.reduce((sum, row) => sum + row.instances, 0);+ entries.push({+ value: remainderLabel(tail.length),+ instances,+ share: share(instances, base),+ devInstances: tail.reduce((sum, row) => sum + row.devInstances, 0),+ remainder: true,+ });+ }+ return entries;+}++function groupByKey(rows: RawGroup[], base: number): Record<string, DistributionEntry[]> {+ const buckets = new Map<string, RawGroup[]>();+ for (const row of rows) {+ const list = buckets.get(row.key) ?? [];+ list.push(row);+ buckets.set(row.key, list);+ }+ const out: Record<string, DistributionEntry[]> = {};+ for (const [key, list] of buckets) out[key] = collapse(list, base);+ return out;+}++// Unpivots columns into (key, value) pairs, reading the already-filtered `active`+// CTE (see buildAggregate) instead of the view directly. D1 caps a compound SELECT+// at five terms, so UNION ALL is not an option for the columns themselves.+const unpivot = (pairs: string) => `+ SELECT t.key AS key, t.value AS value, COUNT(*) AS instances+ FROM active AS p, json_each(json_object(${pairs})) AS t+ GROUP BY t.key, t.value`;++// Wraps a (key?, value, instances[, extra]) query as a JSON array so it can sit+// in its own scalar-subquery column of the single mega-query below. That keeps+// every one of these reports reading the same MATERIALIZED `active` CTE instead+// of each re-deriving "the latest ping per instance" from the full ping history.+const asJsonArray = (columns: string, innerSql: string) =>+ `(SELECT COALESCE(json_group_array(json_object(${columns})), '[]') FROM (${innerSql}))`;++const VERSIONS_SQL = `+ SELECT app_version AS value, COUNT(*) AS instances,+ SUM(CASE WHEN app_is_dev = 1 THEN 1 ELSE 0 END) AS dev_instances+ FROM active GROUP BY app_version`;++const FIELD_TYPES_SQL = `+ SELECT t.value AS value, COUNT(DISTINCT p.id) AS instances+ FROM active AS p, json_each(p.features_custom_field_types) AS t+ GROUP BY t.value`;++/**+ * Every report below used to read the `latest_pings` view independently (nine+ * references in total), and the view itself has no time bound: each reference+ * re-scanned the entire ping history to find each instance's newest row before+ * filtering it down to the active window. At the README's own worked example+ * (~36,000 rows/year) that is roughly 9x the necessary reads per cron run.+ *+ * Instead, `active` is materialized exactly once (one scan of `instance_pings`,+ * enforced with the MATERIALIZED hint so SQLite doesn't just inline the CTE per+ * reference) and every report reads that bounded, already-filtered result as a+ * scalar subquery of one row -- one statement, one full scan, regardless of how+ * many breakdowns are computed from it.+ */+const MEGA_SQL = `+ WITH active AS MATERIALIZED (SELECT * FROM latest_pings WHERE received_at >= ?1)+ SELECT+ (SELECT COUNT(*) FROM active) AS total,+ (SELECT COUNT(*) FROM active WHERE received_at >= ?2) AS active_7d,+ (SELECT COUNT(*) FROM (+ SELECT instance_id FROM instance_pings+ WHERE instance_id IS NOT NULL+ GROUP BY instance_id HAVING MIN(received_at) >= ?1+ )) AS new_30d,+ ${asJsonArray("'value', value, 'instances', instances, 'dev_instances', dev_instances", VERSIONS_SQL)} AS versions_json,+ ${asJsonArray(+ "'key', key, 'value', value, 'instances', instances",+ unpivot(+ `'node', p.runtime_node, 'platform', p.runtime_platform, 'arch', p.runtime_arch,+ 'sqlite', p.runtime_sqlite, 'timezone', p.runtime_timezone`,+ ),+ )} AS environment_json,+ ${asJsonArray(+ "'key', key, 'value', value, 'instances', instances",+ unpivot(+ `'auth_enabled', p.config_auth_enabled, 'guest_access', p.config_guest_access,+ 'registration_open', p.config_registration_open,+ 'update_check_enabled', p.config_update_check_enabled,+ 'default_locale', p.config_default_locale,+ 'rate_limit_overrides', p.config_rate_limit_overrides`,+ ),+ )} AS configuration_json,+ ${asJsonArray(+ "'key', key, 'value', value, 'instances', instances",+ unpivot(+ `'users', p.scale_users, 'calendars', p.scale_calendars, 'shifts', p.scale_shifts,+ 'presets', p.scale_presets, 'notes', p.scale_notes, 'bundles', p.scale_bundles,+ 'shares', p.scale_shares, 'access_tokens', p.scale_access_tokens,+ 'signups', p.scale_signups`,+ ),+ )} AS sizes_json,+ ${asJsonArray(+ "'key', key, 'value', value, 'instances', instances",+ unpivot(+ `'external_syncs', p.features_external_syncs,+ 'calendar_view_overrides', p.features_calendar_view_overrides,+ 'custom_fields_count', p.features_custom_fields_count,+ 'archived_presets', p.features_archived_presets,+ 'split_shifts', p.features_split_shifts`,+ ),+ )} AS features_json,+ ${asJsonArray("'value', value, 'instances', instances", FIELD_TYPES_SQL)} AS field_types_json,+ (SELECT ROUND(AVG(health_uptime_hours), 1) FROM active) AS avg_uptime,+ (SELECT SUM(CASE WHEN health_sync_failures_24h > 0 THEN 1 ELSE 0 END) FROM active) AS with_failures`;++interface VersionJsonRow {+ value: string;+ instances: number;+ dev_instances: number;+}++// Falling back to [] would otherwise publish an all-zeros aggregate with nothing+// in the logs, so every unexpected shape is reported before it is swallowed.+function parseJsonArray<T>(value: unknown): T[] {+ if (typeof value !== "string") {+ console.error(`Aggregate column was ${typeof value}, expected a JSON string`);+ return [];+ }+ try {+ const parsed: unknown = JSON.parse(value);+ if (Array.isArray(parsed)) return parsed as T[];+ console.error("Aggregate column parsed to a non-array");+ return [];+ } catch {+ console.error("Aggregate column is not valid JSON");+ return [];+ }+}++export async function buildAggregate(db: D1Database, now: Date = new Date()): Promise<PublicAggregate> {+ const activeCutoff = iso(now, ACTIVE_WINDOW_DAYS);+ const weekCutoff = iso(now, 7);+ const historyCutoff = iso(now, HISTORY_DAYS).slice(0, 10);++ const [mega, history] = await db.batch([+ db.prepare(MEGA_SQL).bind(activeCutoff, weekCutoff),+ db+ .prepare(+ `SELECT ping_day AS day, COUNT(DISTINCT COALESCE(instance_id, 'anon:' || id)) AS instances+ FROM instance_pings WHERE ping_day >= ?1 GROUP BY ping_day ORDER BY day`,+ )+ .bind(historyCutoff),+ ]);++ const megaRow = (mega.results[0] ?? {}) as Record<string, unknown>;+ const base = Number(megaRow.total ?? 0);++ const instances = {+ total: base,+ activeLast7Days: Number(megaRow.active_7d ?? 0),+ newLast30Days: Number(megaRow.new_30d ?? 0),+ };++ const versionRows = parseJsonArray<VersionJsonRow>(megaRow.versions_json);++ return {+ computedAt: now.toISOString(),+ instances,+ history: history.results as unknown as HistoryPoint[],+ versions: buildVersions(+ versionRows.map((r) => ({+ value: r.value,+ instances: Number(r.instances),+ dev_instances: Number(r.dev_instances),+ })),+ base,+ ),+ environment: groupByKey(parseJsonArray<RawGroup>(megaRow.environment_json), base),+ configuration: groupByKey(parseJsonArray<RawGroup>(megaRow.configuration_json), base),+ sizes: groupByKey(parseJsonArray<RawGroup>(megaRow.sizes_json), base),+ features: groupByKey(parseJsonArray<RawGroup>(megaRow.features_json), base),+ customFieldTypes: collapse(+ parseJsonArray<RawGroup>(megaRow.field_types_json).map((r) => ({ ...r, key: "types" })),+ base,+ ),+ health: {+ avgUptimeHours: Number(megaRow.avg_uptime ?? 0),+ instancesWithSyncFailures: Number(megaRow.with_failures ?? 0),+ },+ };+}++export async function storeAggregate(db: D1Database, aggregate: PublicAggregate): Promise<void> {+ const result = await db+ .prepare("INSERT OR REPLACE INTO aggregates (key, json, computed_at) VALUES ('public', ?1, ?2)")+ .bind(JSON.stringify(aggregate), aggregate.computedAt)+ .run();+ if (!result.success) throw new Error("Aggregate write did not report success");+}++export async function readAggregate(db: D1Database): Promise<PublicAggregate | null> {+ let row: { json: string } | null;+ try {+ row = await db.prepare("SELECT json FROM aggregates WHERE key = 'public'").first<{ json: string }>();+ } catch (error) {+ // A D1 read error (e.g. the table is missing on an un-migrated database)+ // degrades to the empty state instead of a 500; see src/page.ts / src/index.ts.+ console.error(error instanceof Error ? error.message : "Aggregate read failed");+ return null;+ }+ if (!row) return null;+ try {+ return JSON.parse(row.json) as PublicAggregate;+ } catch {+ // A corrupt row is treated exactly like a missing one.+ return null;+ }+}diff --git a/telemetry-hub/src/index.ts b/telemetry-hub/src/index.tsindex 09302bb..6e56946 100644--- a/telemetry-hub/src/index.ts+++ b/telemetry-hub/src/index.ts@@ -1,20 +1,45 @@+import { buildAggregate, readAggregate, storeAggregate } from "./aggregate";+import { renderPage } from "./page";import { parseV1 } from "./schemas/v1";-import { sendToPostHog } from "./targets/posthog";+import { storeInD1 } from "./targets/d1";+// The key must match the `binding` of the [[d1_databases]] block in wrangler.toml.interface Env {- POSTHOG_API_KEY: string;- // Optional ingest host, e.g. https://us.i.posthog.com; defaults to EU.- POSTHOG_HOST?: string;+ bettershift_telemetry: D1Database;}const MAX_BODY_BYTES = 16 * 1024;export default {async fetch(request: Request, env: Env): Promise<Response> {- if (request.method !== "POST") {- return new Response(null, { status: 405 });+ const { pathname } = new URL(request.url);++ if (pathname === "/data.json") {+ if (request.method !== "GET") return new Response(null, { status: 405 });+ const aggregate = await readAggregate(env.bettershift_telemetry);+ return new Response(JSON.stringify(aggregate ?? { computedAt: null }), {+ headers: {+ "Content-Type": "application/json; charset=utf-8",+ "Cache-Control": "public, max-age=300, s-maxage=900",+ "Access-Control-Allow-Origin": "*",+ },+ });}+ if (pathname !== "/") return new Response(null, { status: 404 });++ if (request.method === "GET") {+ const aggregate = await readAggregate(env.bettershift_telemetry);+ return new Response(renderPage(aggregate), {+ headers: {+ "Content-Type": "text/html; charset=utf-8",+ "Cache-Control": "public, max-age=300, s-maxage=900",+ },+ });+ }++ if (request.method !== "POST") return new Response(null, { status: 405 });+// Content-Length only: a chunked body skips this check, which is acceptable// because Workers cap the request size first and parseV1 gates every field.const length = Number(request.headers.get("content-length") ?? "0");@@ -37,12 +62,23 @@ export default {}try {- await sendToPostHog(payload, env.POSTHOG_API_KEY, env.POSTHOG_HOST || undefined);+ await storeInD1(payload, env.bettershift_telemetry);} catch (error) {- // The sender still gets a 200. Only the message is logged (a status, never the payload).- console.error(error instanceof Error ? error.message : "PostHog forward failed");+ // The sender still gets a 200. Only the message is logged (never the payload).+ console.error(error instanceof Error ? error.message : "D1 write failed");}return new Response(null, { status: 200 });},++ async scheduled(_event: ScheduledController, env: Env, _ctx: ExecutionContext): Promise<void> {+ try {+ await storeAggregate(env.bettershift_telemetry, await buildAggregate(env.bettershift_telemetry));+ } catch (error) {+ // db.batch + INSERT OR REPLACE already keep the previous aggregate intact;+ // rethrow so Cloudflare's cron success metric reflects the failure.+ console.error(error instanceof Error ? error.message : "Aggregate run failed");+ throw error;+ }+ },};diff --git a/telemetry-hub/src/page.ts b/telemetry-hub/src/page.tsnew file mode 100644index 0000000..e0481f3--- /dev/null+++ b/telemetry-hub/src/page.ts@@ -0,0 +1,588 @@+import type { DistributionEntry, HistoryPoint, PublicAggregate, VersionEntry } from "./aggregate";++const HISTORY_DAYS = 90;+// Mirrors MAX_DISTRIBUTION_ENTRIES in aggregate.ts, for the footer wording only.+const MAX_DISTRIBUTION_ENTRIES = 25;+const REPO = "https://github.com/panteLx/bettershift";+const DOCS_URL = `${REPO}/blob/main/docs/TELEMETRY.md`;++const ESCAPES: Record<string, string> = {+ "&": "&",+ "<": "<",+ ">": ">",+ '"': """,+ "'": "'",+};++function escapeHtml(value: unknown): string {+ return String(value).replace(/[&<>"']/g, (char) => ESCAPES[char] ?? char);+}++const esc = escapeHtml;++function count(n: number): string {+ return Number.isFinite(n) ? Math.round(n).toLocaleString("en-US") : "0";+}++/** Keeps the one decimal aggregate.ts computes for avgUptimeHours. */+function decimal(n: number): string {+ if (!Number.isFinite(n)) return "0";+ return (Math.round(n * 10) / 10).toLocaleString("en-US", { maximumFractionDigits: 1 });+}++function percent(share: number): string {+ if (!Number.isFinite(share)) return "0%";+ const pct = share * 100;+ if (pct > 0 && pct < 1) return "<1%";+ return `${pct >= 10 || Number.isInteger(pct) ? Math.round(pct) : Math.round(pct * 10) / 10}%`;+}++function plural(n: number, one: string, many: string): string {+ return n === 1 ? one : many;+}++/** Renders the ISO timestamp as a stable UTC string; a bad value is passed through. */+function formatTimestamp(iso: string): string {+ const date = new Date(iso);+ if (Number.isNaN(date.getTime())) return iso;+ const pad = (n: number) => String(n).padStart(2, "0");+ return (+ `${date.getUTCFullYear()}-${pad(date.getUTCMonth() + 1)}-${pad(date.getUTCDate())} ` ++ `${pad(date.getUTCHours())}:${pad(date.getUTCMinutes())} UTC`+ );+}++const DIMENSION_LABELS: Record<string, string> = {+ versions: "Release",+ types: "Type in use",+ node: "Node.js version",+ platform: "Operating system",+ arch: "CPU architecture",+ sqlite: "SQLite version",+ timezone: "Time zone",+ auth_enabled: "Authentication enabled",+ guest_access: "Guest access allowed",+ registration_open: "Open registration",+ update_check_enabled: "Update check enabled",+ default_locale: "Default language",+ rate_limit_overrides: "Rate-limit overrides set",+ users: "User accounts",+ calendars: "Calendars",+ shifts: "Shifts",+ presets: "Shift presets",+ notes: "Notes",+ bundles: "Permission bundles",+ shares: "Calendar shares",+ access_tokens: "Share links",+ signups: "Shift signups",+ external_syncs: "External calendar syncs",+ calendar_view_overrides: "Calendars with a pinned view",+ custom_fields_count: "Custom field definitions",+ archived_presets: "Has archived presets",+ split_shifts: "Split shifts enabled",+};++const BOOLEAN_DIMENSIONS = new Set([+ "auth_enabled",+ "guest_access",+ "registration_open",+ "update_check_enabled",+ "archived_presets",+ "split_shifts",+]);++const ORDER: Record<string, string[]> = {+ environment: ["node", "platform", "arch", "sqlite", "timezone"],+ configuration: [+ "auth_enabled",+ "guest_access",+ "registration_open",+ "update_check_enabled",+ "default_locale",+ "rate_limit_overrides",+ ],+ sizes: [+ "users",+ "calendars",+ "shifts",+ "presets",+ "notes",+ "bundles",+ "shares",+ "access_tokens",+ "signups",+ ],+ features: [+ "external_syncs",+ "calendar_view_overrides",+ "custom_fields_count",+ "archived_presets",+ "split_shifts",+ ],+};++function dimensionLabel(key: string): string {+ return DIMENSION_LABELS[key] ?? key.replace(/_/g, " ");+}++function valueLabel(dimension: string, value: string): string {+ if (BOOLEAN_DIMENSIONS.has(dimension)) {+ if (value === "1" || value === "true") return "Yes";+ if (value === "0" || value === "false") return "No";+ }+ if (value === "unknown") return "not set";+ return value;+}++/** Orders known dimensions first, then anything the aggregate added later. */+function orderedKeys(group: Record<string, DistributionEntry[]>, section: string): string[] {+ const preferred = ORDER[section] ?? [];+ const present = Object.keys(group).filter((key) => group[key] && group[key].length > 0);+ const known = preferred.filter((key) => present.includes(key));+ const rest = present.filter((key) => !preferred.includes(key)).sort();+ return [...known, ...rest];+}++interface BarOptions {+ dimension: string;+ note?: (entry: DistributionEntry, index: number) => string | null;+}++// Structural, not by label: reported values are attacker-supplied and could+// otherwise impersonate the remainder entry's styling.+const isRemainder = (entry: DistributionEntry): boolean => entry.remainder === true;++/**+ * Entries are indexed by position, never by `value`: values are truncated to 64+ * characters, so two entries can legitimately carry the same label.+ */+function renderBars(entries: DistributionEntry[], options: BarOptions): string {+ const rows = entries+ .map((entry, index) => {+ const label = valueLabel(options.dimension, entry.value);+ const remainder = isRemainder(entry);+ // A multi-select dimension counts an instance once per value it holds, so a+ // summed remainder can exceed the population. A share above 1 is not a share+ // there — while in a single-select dimension it would signal a counting bug,+ // which must stay visible, so the suppression is scoped to the remainder.+ const unshareable = remainder && (entry.share || 0) > 1;+ const width = Math.max(0.8, Math.min(100, (entry.share || 0) * 100));+ const note = options.note?.(entry, index);+ return `<li class="bar${remainder ? " bar-other" : ""}">+ <div class="bar-head">+ <span class="bar-label">${esc(label)}${remainder ? '<span class="bar-hint"> — long tail, summed to keep this page small</span>' : ""}</span>+ <span class="bar-value">${unshareable ? "" : `<b>${esc(percent(entry.share))}</b>`}<span class="bar-count">${esc(count(entry.instances))}</span></span>+ </div>+ <div class="track">${+ // No claimable share, so no fill: an empty track beats drawing the widest bar+ // on the card for a number the page just refused to print.+ unshareable ? "" : `<div class="fill" style="width:${esc(width.toFixed(1))}%"></div>`+ }</div>+ ${note ? `<p class="bar-note">${esc(note)}</p>` : ""}+</li>`;+ })+ .join("\n");++ return `<ul class="bars">\n${rows}\n</ul>`;+}++function renderGroup(key: string, entries: DistributionEntry[], note?: BarOptions["note"]): string {+ return `<div class="dim">+ <h3>${esc(dimensionLabel(key))}</h3>+ ${renderBars(entries, { dimension: key, note })}+</div>`;+}++function renderDistributionSection(+ id: string,+ title: string,+ caption: string,+ group: Record<string, DistributionEntry[]>,+): string {+ const keys = orderedKeys(group ?? {}, id);+ if (keys.length === 0) return "";+ const body = keys.map((key) => renderGroup(key, group[key])).join("\n");+ return `<section id="${esc(id)}">+ <h2>${esc(title)}</h2>+ <p class="caption">${esc(caption)}</p>+ <div class="dims">+${body}+ </div>+</section>`;+}++function renderVersions(versions: VersionEntry[]): string {+ if (versions.length === 0) return "";+ const note = (_entry: DistributionEntry, index: number) => {+ const dev = versions[index]?.devInstances ?? 0;+ return dev > 0 ? `includes ${count(dev)} development ${plural(dev, "build", "builds")}` : null;+ };+ return `<section id="versions">+ <h2>Versions in use</h2>+ <p class="caption">The release each reporting instance was running when it last checked in.</p>+ <div class="dims">+${renderGroup("versions", versions, note)}+ </div>+</section>`;+}++const DAY_MS = 86_400_000;++function dayNumber(day: string): number | null {+ const match = /^(\d{4})-(\d{2})-(\d{2})$/.exec(day);+ if (!match) return null;+ const time = Date.UTC(Number(match[1]), Number(match[2]) - 1, Number(match[3]));+ return Number.isNaN(time) ? null : Math.round(time / DAY_MS);+}++function niceMax(value: number): number {+ if (value <= 5) return Math.max(1, value);+ const magnitude = Math.pow(10, Math.floor(Math.log10(value)));+ for (const step of [1, 1.5, 2, 2.5, 3, 4, 5, 6, 8, 10]) {+ const candidate = step * magnitude;+ if (candidate >= value) return candidate;+ }+ return value;+}++interface Point {+ day: string;+ n: number;+ instances: number;+}++/**+ * A sparse series: a day on which nothing reported is absent from the data, not+ * zero. Gaps stay gaps — zero-filling would draw a dip that never happened. Only+ * runs of consecutive days are joined by a line.+ */+function renderHistory(points: HistoryPoint[]): string {+ const caption = `Distinct instances that reported on each of the last ${HISTORY_DAYS} days.`;++ const parsed: Point[] = points+ .map((point) => ({+ day: String(point.day),+ n: dayNumber(String(point.day)),+ instances: Number(point.instances),+ }))+ .filter(+ (point): point is Point => point.n !== null && Number.isFinite(point.instances) && point.instances > 0,+ )+ .sort((a, b) => a.n - b.n);++ if (parsed.length === 0) {+ return `<section id="history">+ <h2>Reports over time</h2>+ <p class="caption">${esc(caption)}</p>+ <div class="empty-chart">+ <div class="empty-baseline"></div>+ <p class="empty-title">Nothing to plot yet</p>+ <p class="empty-body">No instance has reported on any day in this window, so there is no series to draw. The chart appears with the first day that carries a report.</p>+ </div>+</section>`;+ }++ const width = 720;+ const height = 220;+ const padL = 40;+ const padR = 16;+ const padT = 28;+ const padB = 30;+ const plotW = width - padL - padR;+ const plotH = height - padT - padB;++ const first = parsed[0];+ const last = parsed[parsed.length - 1];+ const span = Math.max(1, last.n - first.n);+ const yMax = niceMax(Math.max(...parsed.map((point) => point.instances)));++ const x = (n: number) => padL + (parsed.length === 1 ? plotW / 2 : ((n - first.n) / span) * plotW);+ const y = (v: number) => padT + plotH - (v / yMax) * plotH;++ // Split into runs of calendar-consecutive days; a break in the data breaks the line.+ const runs: Point[][] = [];+ for (const point of parsed) {+ const current = runs[runs.length - 1];+ if (current && point.n - current[current.length - 1].n === 1) current.push(point);+ else runs.push([point]);+ }++ const lines = runs+ .filter((run) => run.length > 1)+ .map(+ (run) =>+ `<polyline class="line" points="${esc(+ run.map((point) => `${x(point.n).toFixed(1)},${y(point.instances).toFixed(1)}`).join(" "),+ )}" />`,+ )+ .join("");++ // Dots stay affordable while the series is short; otherwise only isolated days+ // and the endpoint are marked, so a gap still reads as a gap.+ const dense = parsed.length > 20;+ const dots = parsed+ .filter((point, index) => {+ if (!dense) return true;+ const prev = parsed[index - 1];+ const next = parsed[index + 1];+ const isolated = (!prev || point.n - prev.n > 1) && (!next || next.n - point.n > 1);+ return isolated || point === last;+ })+ .map(+ (point) =>+ `<circle class="dot" cx="${esc(x(point.n).toFixed(1))}" cy="${esc(y(point.instances).toFixed(1))}" r="4"><title>${esc(point.day)}: ${esc(count(point.instances))} ${esc(plural(point.instances, "instance", "instances"))}</title></circle>`,+ )+ .join("");++ // Instance counts are integers: a mid tick at 2.5 labelled "3" would put a+ // 3-instance day below its own gridline, so it is only drawn when it is whole.+ const tickValues = yMax % 2 === 0 ? [0, yMax / 2, yMax] : [0, yMax];+ const ticks = tickValues.map(+ (value) =>+ `<line class="grid" x1="${esc(padL)}" x2="${esc(width - padR)}" y1="${esc(y(value).toFixed(1))}" y2="${esc(y(value).toFixed(1))}" />` ++ `<text class="tick" x="${esc(padL - 8)}" y="${esc((y(value) + 4).toFixed(1))}" text-anchor="end">${esc(count(value))}</text>`,+ );++ // padT leaves room for the end label to sit above the last point without clipping.+ const endLabel = `<text class="end-label" x="${esc((x(last.n) - 8).toFixed(1))}" y="${esc((y(last.instances) - 12).toFixed(1))}" text-anchor="end">${esc(count(last.instances))}</text>`;++ const gapNote =+ parsed.length < span + 1+ ? "Where the line breaks, no instance reported that day at all. A break is not a drop to zero."+ : "Every day in this range carries at least one report, so the line is unbroken.";++ return `<section id="history">+ <h2>Reports over time</h2>+ <p class="caption">${esc(caption)}</p>+ <div class="chart">+ <svg viewBox="0 0 ${esc(width)} ${esc(height)}" role="img" aria-label="${esc(`Instances reporting per day, ${first.day} to ${last.day}`)}" preserveAspectRatio="xMidYMid meet">+ ${ticks.join("")}+ ${lines}+ ${dots}+ ${endLabel}+ <text class="tick" x="${esc(padL)}" y="${esc(height - 8)}">${esc(first.day)}</text>+ <text class="tick" x="${esc(width - padR)}" y="${esc(height - 8)}" text-anchor="end">${esc(last.day)}</text>+ </svg>+ </div>+ <p class="note">${esc(gapNote)}</p>+</section>`;+}++function renderHealth(health: PublicAggregate["health"]): string {+ // Zero is a true and useful answer here — "no instance reported a failed sync"+ // is a statement worth publishing, so this section never disappears.+ const uptime = Number(health?.avgUptimeHours ?? 0);+ const failures = Number(health?.instancesWithSyncFailures ?? 0);+ return `<section id="health">+ <h2>Health</h2>+ <p class="caption">Two plain figures, across the instances that reported.</p>+ <div class="figures">+ <div class="figure"><span class="figure-value">${esc(decimal(uptime))} h</span><span class="figure-label">Average uptime since the last restart</span></div>+ <div class="figure"><span class="figure-value">${esc(count(failures))}</span><span class="figure-label">Instances with a failed calendar sync in the last 24 hours</span></div>+ </div>+</section>`;+}++/** Nothing is being withheld here — no instance has reported in the window. */+function renderNoReportsNotice(hasHistory: boolean): string {+ const tail = hasHistory+ ? " The chart below still reaches back 90 days, so the last reports that did come in are plotted there."+ : "";+ return `<section id="no-reports">+ <div class="notice">+ <p class="notice-title">No instance has reported in the last 30 days</p>+ <p>There is nothing to break down, because nothing has come in. That is the normal state for a freshly deployed hub, and it is also what this page shows once every instance that once reported has aged out of the 30-day window. Nothing is being withheld.${tail}</p>+ </div>+</section>`;+}++const STYLES = `+:root{color-scheme:light;--plane:#f9f9f7;--surface:#fcfcfb;--ink:#0b0b0b;--ink-2:#52514e;--muted:#898781;--grid:#e1e0d9;--border:rgba(11,11,11,.10);--series:#2a78d6;--track:#cde2fb;--accent-soft:#eef4fd}+@media (prefers-color-scheme:dark){:root{color-scheme:dark;--plane:#0d0d0d;--surface:#1a1a19;--ink:#fff;--ink-2:#c3c2b7;--muted:#898781;--grid:#2c2c2a;--border:rgba(255,255,255,.10);--series:#3987e5;--track:#0d366b;--accent-soft:#14202e}}+*{box-sizing:border-box}+html{-webkit-text-size-adjust:100%}+body{margin:0;background:var(--plane);color:var(--ink);font:400 16px/1.55 system-ui,-apple-system,"Segoe UI",sans-serif;overflow-wrap:break-word}+.wrap{max-width:860px;margin:0 auto;padding:40px 16px 56px}+a{color:var(--series)}+h1{font-size:28px;line-height:1.2;letter-spacing:-.02em;margin:0 0 8px}+h2{font-size:13px;letter-spacing:.08em;text-transform:uppercase;color:var(--ink-2);margin:0 0 4px;font-weight:600}+h3{font-size:14px;font-weight:600;margin:0 0 10px;color:var(--ink)}+p{margin:0 0 12px}+.lede{font-size:17px;color:var(--ink-2);margin:0 0 20px;max-width:56ch}+.promise{border:1px solid var(--border);border-left:3px solid var(--series);background:var(--accent-soft);border-radius:10px;padding:14px 16px;margin:0 0 36px}+.promise p{margin:0 0 6px;font-size:14px;color:var(--ink-2)}+.promise p:last-child{margin:0}+.promise b{color:var(--ink)}+section{margin:0 0 40px}+.caption{color:var(--muted);font-size:13px;margin:0 0 16px;max-width:64ch}+.note{color:var(--muted);font-size:13px;margin:10px 0 0;max-width:64ch}+.tiles{display:grid;grid-template-columns:repeat(auto-fit,minmax(150px,1fr));gap:12px;margin:0 0 40px}+.tile{background:var(--surface);border:1px solid var(--border);border-radius:12px;padding:18px}+.tile-value{display:block;font-size:30px;font-weight:600;letter-spacing:-.02em;line-height:1.1}+.tile.hero .tile-value{font-size:52px}+.tile-label{display:block;color:var(--ink-2);font-size:13px;margin-top:6px}+.chart{background:var(--surface);border:1px solid var(--border);border-radius:12px;padding:12px 8px}+.chart svg{display:block;width:100%;height:auto}+.grid{stroke:var(--grid);stroke-width:1}+.line{fill:none;stroke:var(--series);stroke-width:2;stroke-linejoin:round;stroke-linecap:round}+.dot{fill:var(--series);stroke:var(--surface);stroke-width:2}+.tick{fill:var(--muted);font-size:11px;font-family:system-ui,-apple-system,"Segoe UI",sans-serif}+.end-label{fill:var(--ink-2);font-size:12px;font-weight:600;font-family:system-ui,-apple-system,"Segoe UI",sans-serif}+.empty-chart{background:var(--surface);border:1px solid var(--border);border-radius:12px;padding:22px 18px}+.empty-baseline{height:1px;background:repeating-linear-gradient(90deg,var(--grid) 0 6px,transparent 6px 12px);margin-bottom:16px}+.empty-title{font-weight:600;margin:0 0 6px}+.empty-body{color:var(--ink-2);font-size:14px;margin:0;max-width:64ch}+.dims{display:grid;grid-template-columns:repeat(auto-fit,minmax(250px,1fr));gap:14px}+.dim{background:var(--surface);border:1px solid var(--border);border-radius:12px;padding:16px}+.bars{list-style:none;margin:0;padding:0}+.bar+.bar{margin-top:12px}+.bar-head{display:flex;align-items:baseline;justify-content:space-between;gap:10px;margin-bottom:5px}+.bar-label{font-size:14px;min-width:0}+.bar-other .bar-label{color:var(--ink-2);font-style:italic}+.bar-value{font-size:13px;color:var(--ink-2);white-space:nowrap;font-variant-numeric:tabular-nums}+.bar-value b{color:var(--ink);font-weight:600}+.bar-count{color:var(--muted);margin-left:8px}+.track{height:10px;background:var(--track);border-radius:2px;overflow:hidden}+.fill{height:100%;background:var(--series);border-radius:0 4px 4px 0}+.bar-note{color:var(--muted);font-size:12px;margin:5px 0 0}+.bar-hint{color:var(--muted);font-size:12px;font-style:normal}+.figures{display:grid;grid-template-columns:repeat(auto-fit,minmax(220px,1fr));gap:12px}+.figure{background:var(--surface);border:1px solid var(--border);border-radius:12px;padding:18px}+.figure-value{display:block;font-size:30px;font-weight:600;letter-spacing:-.02em}+.figure-label{display:block;color:var(--ink-2);font-size:13px;margin-top:6px}+.notice{background:var(--surface);border:1px solid var(--border);border-left:3px solid var(--series);border-radius:10px;padding:18px}+.notice p{color:var(--ink-2);font-size:14px;margin:0}+.notice p.notice-title{font-size:16px;font-weight:600;color:var(--ink);margin:0 0 10px}+footer{border-top:1px solid var(--border);padding-top:20px;color:var(--muted);font-size:13px}+footer p{margin:0 0 8px;max-width:72ch}+footer .stamp{color:var(--ink-2);font-variant-numeric:tabular-nums}+@media (max-width:420px){.wrap{padding:28px 16px 40px}h1{font-size:24px}.tile.hero .tile-value{font-size:44px}}+`;++function renderFooter(computedAt: string | null): string {+ const stamp = computedAt+ ? `Figures computed ${esc(formatTimestamp(computedAt))}, recomputed hourly.`+ : "No figures have been computed yet.";+ return `<footer>+ <p class="stamp">${stamp}</p>+ <p>BetterShift telemetry is <b>opt-in and off by default</b>. A self-hosted instance sends nothing until an administrator explicitly agrees; declining is permanent, and nothing here was counted without that consent.</p>+ <p>Percentages are shares of the instances that reported within the last 30 days — not of all BetterShift installations. A single instance can hold more than one value in some breakdowns (the custom field types it uses, for one), so the shares in a card need not add up to 100%.</p>+ <p>Each breakdown lists its ${esc(String(MAX_DISTRIBUTION_ENTRIES))} largest values and sums everything past that into one <i>other (N more values)</i> entry. That is a size guard — the reported strings are arbitrary and a page has to stay small — not a privacy rule: nothing is withheld for being rare, and a value held by a single instance is published with its count of one.</p>+ <p>Every field an instance can send is documented in <a href="${esc(DOCS_URL)}" rel="noopener">docs/TELEMETRY.md</a>.</p>+</footer>`;+}++function page(main: string, computedAt: string | null): string {+ return `<!doctype html>+<html lang="en">+<head>+<meta charset="utf-8">+<meta name="viewport" content="width=device-width,initial-scale=1">+<meta name="description" content="Anonymous, opt-in usage statistics from self-hosted BetterShift instances.">+<title>BetterShift — public statistics</title>+<style>${STYLES}</style>+</head>+<body>+<div class="wrap">+<header>+<h1>BetterShift in the wild</h1>+<p class="lede">What self-hosted BetterShift instances look like, as reported by the ones whose administrators chose to share.</p>+<div class="promise">+<p><b>Nobody is counted without agreeing first.</b> Telemetry is opt-in and off by default; an instance sends nothing until an admin says yes.</p>+<p><b>No names, no URLs, no IP addresses.</b> Sizes travel as buckets, never as exact counts.</p>+<p><b>One snapshot a day, nothing in between.</b> No page tracking, no analytics script, no cookie — and every field an instance can send is documented before an admin agrees to it.</p>+</div>+</header>+${main}+${renderFooter(computedAt)}+</div>+</body>+</html>+`;+}++export function renderPage(aggregate: PublicAggregate | null): string {+ if (!aggregate || !aggregate.instances) {+ return page(+ `<section id="empty">+ <div class="notice">+ <p class="notice-title">No data yet</p>+ <p>No statistics have been computed so far. This page fills in once instances have opted in and reported — until then there is genuinely nothing to show.</p>+ </div>+</section>`,+ null,+ );+ }++ const instances = aggregate.instances;+ const total = Number(instances.total ?? 0);++ const tiles = `<section id="headline">+ <h2>Reporting instances</h2>+ <p class="caption">Instances that sent a report within the last 30 days.</p>+ <div class="tiles">+ <div class="tile hero"><span class="tile-value">${esc(count(total))}</span><span class="tile-label">${esc(plural(total, "instance reporting", "instances reporting"))}</span></div>+ <div class="tile"><span class="tile-value">${esc(count(Number(instances.activeLast7Days ?? 0)))}</span><span class="tile-label">Active in the last 7 days</span></div>+ <div class="tile"><span class="tile-value">${esc(count(Number(instances.newLast30Days ?? 0)))}</span><span class="tile-label">First seen in the last 30 days</span></div>+ </div>+</section>`;++ const history = Array.isArray(aggregate.history) ? aggregate.history : [];++ // The history window is 90 days and the headline window is 30, so days can+ // outlive the instances that reported them. Keep the chart in that case.+ if (total <= 0) {+ const hasHistory = history.length > 0;+ const main = tiles + renderNoReportsNotice(hasHistory) + (hasHistory ? renderHistory(history) : "");+ return page(main, aggregate.computedAt ?? null);+ }++ const versions = Array.isArray(aggregate.versions) ? aggregate.versions : [];+ const environment = aggregate.environment ?? {};+ const configuration = aggregate.configuration ?? {};+ const sizes = aggregate.sizes ?? {};+ const features = aggregate.features ?? {};+ const customFieldTypes = Array.isArray(aggregate.customFieldTypes) ? aggregate.customFieldTypes : [];++ const customFields =+ customFieldTypes.length > 0+ ? `<section id="custom-fields">+ <h2>Custom field types</h2>+ <p class="caption">Which custom field types are in use. Field keys and labels are never collected — only the type. An instance appears once per type it uses, so these counts describe types, not a split of the instance population.</p>+ <div class="dims">+${renderGroup("types", customFieldTypes)}+ </div>+</section>`+ : "";++ const main = [+ tiles,+ renderHistory(history),+ renderVersions(versions),+ renderDistributionSection("environment", "Runtime environment", "What the app runs on.", environment),+ renderDistributionSection(+ "configuration",+ "Configuration",+ "How instances are set up. Each setting is counted on its own.",+ configuration,+ ),+ renderDistributionSection(+ "sizes",+ "Size of an instance",+ "Reported as buckets, never as exact counts.",+ sizes,+ ),+ renderDistributionSection(+ "features",+ "Feature use",+ "Which optional parts of the app are in use.",+ features,+ ),+ customFields,+ renderHealth(aggregate.health),+ ]+ .filter(Boolean)+ .join("\n");++ return page(main, aggregate.computedAt ?? null);+}diff --git a/telemetry-hub/src/targets/d1.ts b/telemetry-hub/src/targets/d1.tsnew file mode 100644index 0000000..0ae1d57--- /dev/null+++ b/telemetry-hub/src/targets/d1.ts@@ -0,0 +1,96 @@+import type { PayloadV1 } from "../schemas/v1";++const COLUMNS = [+ "received_at",+ "ping_day",+ "sent_at",+ "instance_id",+ "schema_version",+ "app_version",+ "app_is_dev",+ "app_migrations",+ "runtime_node",+ "runtime_arch",+ "runtime_platform",+ "runtime_sqlite",+ "runtime_timezone",+ "config_auth_enabled",+ "config_guest_access",+ "config_registration_open",+ "config_default_locale",+ "config_update_check_enabled",+ "config_rate_limit_overrides",+ "scale_users",+ "scale_calendars",+ "scale_shifts",+ "scale_presets",+ "scale_notes",+ "scale_bundles",+ "scale_shares",+ "scale_access_tokens",+ "scale_signups",+ "features_external_syncs",+ "features_calendar_view_overrides",+ "features_custom_fields_count",+ "features_custom_field_types",+ "features_archived_presets",+ "features_split_shifts",+ "health_uptime_hours",+ "health_sync_runs_24h",+ "health_sync_failures_24h",+] as const;++const INSERT = `INSERT INTO instance_pings (${COLUMNS.join(", ")}) VALUES (${COLUMNS.map(() => "?").join(", ")})`;++const bool = (value: boolean): number => (value ? 1 : 0);++// Enumerates target columns rather than spreading, so an unknown input field cannot pass through.+export async function storeInD1(payload: PayloadV1, db: D1Database): Promise<void> {+ const receivedAt = new Date().toISOString();++ const result = await db+ .prepare(INSERT)+ .bind(+ receivedAt,+ receivedAt.slice(0, 10),+ payload.sentAt,+ payload.instanceId,+ payload.schemaVersion,+ payload.app.version,+ bool(payload.app.isDev),+ payload.app.migrations,+ payload.runtime.node,+ payload.runtime.arch,+ payload.runtime.platform,+ payload.runtime.sqlite,+ payload.runtime.timezone,+ bool(payload.config.authEnabled),+ bool(payload.config.guestAccess),+ bool(payload.config.registrationOpen),+ payload.config.defaultLocale,+ bool(payload.config.updateCheckEnabled),+ payload.config.rateLimitOverrides,+ payload.scale.users,+ payload.scale.calendars,+ payload.scale.shifts,+ payload.scale.presets,+ payload.scale.notes,+ payload.scale.bundles,+ payload.scale.shares,+ payload.scale.accessTokens,+ payload.scale.signups,+ payload.features.externalSyncs,+ payload.features.calendarViewOverrides,+ payload.features.customFields.count,+ JSON.stringify(payload.features.customFields.types),+ bool(payload.features.archivedPresets),+ bool(payload.features.splitShifts),+ payload.health.uptimeHours,+ payload.health.syncRuns24h,+ payload.health.syncFailures24h,+ )+ .run();++ // D1 reports a rejected write in the result rather than by throwing.+ if (!result.success) throw new Error("D1 insert did not report success");+}diff --git a/telemetry-hub/src/targets/posthog.ts b/telemetry-hub/src/targets/posthog.tsdeleted file mode 100644index 52f6ea4..0000000--- a/telemetry-hub/src/targets/posthog.ts+++ /dev/null@@ -1,67 +0,0 @@-import type { PayloadV1 } from "../schemas/v1";--// The ingest host is region-specific, and so is the project API key (US: https://us.i.posthog.com).-const DEFAULT_POSTHOG_HOST = "https://eu.i.posthog.com";--// Enumerates target fields rather than spreading, so an unknown input field cannot pass through.-export async function sendToPostHog(- payload: PayloadV1,- apiKey: string,- host: string = DEFAULT_POSTHOG_HOST,-): Promise<void> {- const body = {- api_key: apiKey,- event: "instance_ping",- distinct_id: payload.instanceId ?? "anonymous",- timestamp: payload.sentAt,- properties: {- // The Worker runs in the PoP nearest the sending instance, so PostHog would- // geolocate that IP and approximate every instance's location.- $ip: null,- $geoip_disable: true,- // Keeps PostHog from creating a person profile per instance -- there is no person here.- $process_person_profile: false,- schema_version: payload.schemaVersion,- app_version: payload.app.version,- app_is_dev: payload.app.isDev,- app_migrations: payload.app.migrations,- runtime_node: payload.runtime.node,- runtime_arch: payload.runtime.arch,- runtime_platform: payload.runtime.platform,- runtime_sqlite: payload.runtime.sqlite,- runtime_timezone: payload.runtime.timezone,- config_auth_enabled: payload.config.authEnabled,- config_guest_access: payload.config.guestAccess,- config_registration_open: payload.config.registrationOpen,- config_default_locale: payload.config.defaultLocale,- config_update_check_enabled: payload.config.updateCheckEnabled,- config_rate_limit_overrides: payload.config.rateLimitOverrides,- scale_users: payload.scale.users,- scale_calendars: payload.scale.calendars,- scale_shifts: payload.scale.shifts,- scale_presets: payload.scale.presets,- scale_notes: payload.scale.notes,- scale_bundles: payload.scale.bundles,- scale_shares: payload.scale.shares,- scale_access_tokens: payload.scale.accessTokens,- scale_signups: payload.scale.signups,- features_external_syncs: payload.features.externalSyncs,- features_calendar_view_overrides: payload.features.calendarViewOverrides,- features_custom_fields_count: payload.features.customFields.count,- features_custom_field_types: payload.features.customFields.types,- features_archived_presets: payload.features.archivedPresets,- features_split_shifts: payload.features.splitShifts,- health_uptime_hours: payload.health.uptimeHours,- health_sync_runs_24h: payload.health.syncRuns24h,- health_sync_failures_24h: payload.health.syncFailures24h,- },- };-- const response = await fetch(`${host.replace(/\/$/, "")}/i/v0/e/`, {- method: "POST",- headers: { "Content-Type": "application/json" },- body: JSON.stringify(body),- });- // fetch resolves on 4xx/5xx, so a rejected key or wrong region would otherwise vanish.- if (!response.ok) throw new Error(`PostHog responded with ${response.status}`);-}diff --git a/telemetry-hub/wrangler.toml b/telemetry-hub/wrangler.tomlindex c68336c..3464282 100644--- a/telemetry-hub/wrangler.toml+++ b/telemetry-hub/wrangler.toml@@ -9,5 +9,12 @@ routes = [[dev]ip = "0.0.0.0"-[vars]-POSTHOG_HOST = "https://eu.i.posthog.com"\ No newline at end of file+[[d1_databases]]+binding = "bettershift_telemetry"+database_name = "bettershift-telemetry"+database_id = "79d67a9c-d7bc-4514-af1e-f24dd1a94e24"+migrations_dir = "migrations"++# Off the hour: the top of the hour is the busiest slot on Cloudflare's scheduler.+[triggers]+crons = ["17 * * * *"]