Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

docs: alerts handbook page and enterprise OOM tracker (SRE-237) (#16712)

Amr Elmohamady4967ac05e76b
diff --git a/brain/knowledge/data-storage-observability/enterprise-oom-tracker.md b/brain/knowledge/data-storage-observability/enterprise-oom-tracker.md
new file mode 100644
index 00000000000..01c55b8efd5
--- /dev/null
+++ b/brain/knowledge/data-storage-observability/enterprise-oom-tracker.md
@@ -0,0 +1,26 @@
+---
+icon: 🧠
+---
+
+# Enterprise OOM Tracker
+
+Daily job that finds enterprise Cloud platforms whose runs end in `MEMORY_LIMIT_EXCEEDED`, so CS/SRE can reach out before the customer escalates (SRE-237). Those runs end as a flow-run status, not a failed BullMQ job, so nothing else alerts on them.
+
+**Script** — `/root/platform-oom-tracker/platform-oom-tracker.js` on `ap-devops`, not in this repo. Runs daily at 12:00 Cairo time and processes yesterday (UTC day). The box's Debian cron has no `CRON_TZ`, so the crontab fires at 09:00 and 10:00 UTC behind a `TZ=Africa/Cairo date +%H = 12` guard, which keeps it on 12:00 through Egypt's DST switch. Log: `/var/log/platform-oom-tracker.log`.
+**Source** — the read-only Postgres replica (`OOM_PG_URL` + `OOM_PG_SSL_CA` in the script's own `.env`), never the prod `/root/queue/.env`.
+**Table** — ClickHouse `ops.platform_oom_daily` on the `Activepieces` service: one row per `(day, platform_id)` with OOM > 0, carrying `platform_name`, `plan`, `oom_runs`, `distinct_flows_oom`, `total_runs`. `ReplacingMergeTree`, so read with `FINAL`; insert user `oom_tracker`.
+**Alert** — every enterprise platform with at least one OOM run yesterday, no dedup. One Discord message per day, posted straight to a channel webhook (`DISCORD_OOM_WEBHOOK_URL`), not through Betterstack, so no incident and no auto-created Linear ticket carrying customer names. One line per platform, sorted by OOM runs, trimmed to Discord's 2,000-character limit with "…and N more"; nothing is sent on a day with no OOMs.
+**Dashboard** — HyperDX [Enterprise OOM Tracker](https://hyperdx.clickhouse.cloud/dashboards/6ad7451181c0f6e143965a5e): top platforms, OOM runs and platforms hit per day, and the per-day list each alert sends.
+**Heartbeat** — Betterstack heartbeat 512932 "platform-oom-tracker (SRE-237)", 1d period + 3h grace, pinged only after insert and alert succeed; `/fail` on error.
+**Enterprise** — `plan` not in `free, appsumo, free_legacy, plus*, team*, standard, old_unlimited_trial*`. See the plan-name gotcha in *EE Platform Plans & Billing*.
+
+## Running it
+
+- `node platform-oom-tracker.js --day YYYY-MM-DD` reprocesses one day; safe to repeat.
+- `--backfill N --no-alert` loads the last N days without alerting. Run it under `nohup`: a day takes 25–90 s.
+
+## Gotchas
+
+- **The replica has `statement_timeout = 1min`.** The script sets its own session timeout (20 min); a query copied out of it into another client will be cancelled.
+- **An `sslmode=` param in `OOM_PG_URL` silently discards `OOM_PG_SSL_CA`** in `pg` 8.x (the parsed URL overrides the `ssl` object), and DO console URLs ship `?sslmode=require`. The script refuses a URL with `ssl*` params.
+- **ClickHouse resolves a `SELECT` alias everywhere in the query**, so `sumIf(oom_runs, …) AS oom_runs` followed by another `sumIf(oom_runs, …)` fails with `ILLEGAL_AGGREGATION`. Alias results with a different name.
diff --git a/brain/knowledge/data-storage-observability/index.md b/brain/knowledge/data-storage-observability/index.md
index 7d7dc7b8482..01b8435b58b 100644
--- a/brain/knowledge/data-storage-observability/index.md
+++ b/brain/knowledge/data-storage-observability/index.md
@@ -104,3 +104,4 @@ Streams platform/project events to webhook URLs in real time — internal AP flo
- **Analytics** — usage reporting
- **Audit Logs** — the persisted security-action record
- **Logging & Metrics (evlog)** — wide events, drains, log-based metrics, `system.snapshot`, OTLP queue gauge
+- **Enterprise OOM Tracker** — daily `MEMORY_LIMIT_EXCEEDED` count per enterprise platform in ClickHouse, with a daily Discord report of every affected platform
diff --git a/brain/knowledge/execution-runtime/benchmark-cli.md b/brain/knowledge/execution-runtime/benchmark-cli.md
index 512eb4a8303..1f4fa43ccd5 100644
--- a/brain/knowledge/execution-runtime/benchmark-cli.md
+++ b/brain/knowledge/execution-runtime/benchmark-cli.md
@@ -6,6 +6,8 @@ icon: ⏱️
`npx @activepieces/cli benchmark` load-tests a deployment's sync-webhook path and attributes *where* latency goes, so a self-hosted setup can be compared apples-to-apples against Activepieces' published reference numbers. Available in CE, EE, Cloud.
+Nightly CI results (`.github/workflows/benchmark.yml`, scheduled runs only) are stored in ClickHouse and charted on the HyperDX [Nightly Benchmark](https://hyperdx.clickhouse.cloud/dashboards/6ab2f0392d54d4a63406098f) dashboard.
+
### How it works
- Builds a `webhook → data-mapper → return-response` flow. Instead of a raw `--concurrency`, it **auto-discovers the deployment shape** (`GET /v1/worker-machines`) and drives load = the effective **execution slot** count, so a healthy deploy queues ~zero by construction. Any queue-wait it reports is a real finding (usually driven concurrency > slots).
- Attributes latency from each `FlowRun.timeline` into QUEUE / PROVISION / BOOT / RUN, cross-checked against live BullMQ queue depth.
diff --git a/brain/knowledge/platform-editions-ee/ee-platform-plans-billing.md b/brain/knowledge/platform-editions-ee/ee-platform-plans-billing.md
index b628badd8bc..8a89370f89e 100644
--- a/brain/knowledge/platform-editions-ee/ee-platform-plans-billing.md
+++ b/brain/knowledge/platform-editions-ee/ee-platform-plans-billing.md
@@ -49,6 +49,7 @@ Billing and entitlements are powered by [Autumn](https://useautumn.com). Each pl
- **`Balance.nextResetAt` is the earliest reset across *every* grant, not the plan's** — so the credits card dates the wrong pool. Autumn's `auto_enable` free plan sits underneath every paid customer and keeps contributing its daily 100-credit bucket, so `nextResetAt` is always ~24h away while the plan's own allowance renews months out: a large `one_off` customer whose plan grant is a year away reads a next-day reset because the free tier's 100 credits reset then. The per-grant dates are in `balance.breakdown[].reset.resetsAt`, keyed by `plan_id`; `toBalanceCache` now takes the reset belonging to the **largest** `includedGrant`, which picks the plan grant without needing the current plan id threaded through `writeBalance`/`writeCustomerStateCaches` — safe because the free tier's 100/day is smaller than any paid allowance, and only wrong if a paid plan ever grants under 100 credits. It falls back to the aggregate when there is a single grant, so free-only platforms are untouched. The same aggregation inflates `granted` by the free tier's 100 on top of the plan grant, which is left alone — the customer really can spend all of it. Note this is a *different* problem from the reset-cadence bullet above: that one is about the interval (day vs month vs year) being absent from the balance, this one is about the date being the wrong grant's.
- **`platform.usage.creditsRemaining` collapses three states into two values, and none of them tells you whether to gate the user.** `getUsage` maps the provider's `CreditsUsage` as `isNil(credits) ? 0 : credits.remaining`, while `toCreditsUsage` (`autumn-billing.ts`) nulls `remaining` for an unlimited balance — so **`null` means unlimited** and **`0` means either genuinely empty or unknown** (Autumn unreachable, cold cache); the whole `usage` object is `undefined` only on CE. Percent-based UI survives the unknown case by accident, not by design: `creditsUsed` is 0 too, so `total` is 0 and `billingUtils.percentUsed` returns 0, landing on the quiet branch. What `creditsRemaining` never carries is whether credits are *enforced* — that is the separate `platform.billingEnforced` flag (decision 000020), and a platform can meter a finite balance with the flag off. Any credit-nag surface has to read both; a surface driven by the percentage alone nags customers whose runs nothing is actually blocking, and the louder the surface (a non-dismissible dashboard banner rather than a sidebar figure) the worse that reads. The web side normalises it once as `useCreditsUsage().isBillingEnforced` (`features/billing/hooks/use-credits-usage.ts`) and gates on it in three places — `billingUtils.shouldShowCreditsAlert`, `CreditsActionButton`, and `useCreditsState`'s `showLowCreditsWarning`; note the chat hook originally gated only its *exhausted* branch, so the warning branch leaked for a while. Because `false` also means "cold enforcement cache", every one of these fails silent rather than open, matching decision 000020.
- **`platform_plan.plan` still carries pre-Autumn plan names on any platform that has not been read since the migration deployed.** There was no `'free'` plan before Autumn: the Cloud free tier was `plan = 'standard'` (`STANDARD_CLOUD_PLAN`), and the old `PlanName` enum held only `STANDARD`, `ENTERPRISE`, and `APPSUMO_ACTIVEPIECES_TIER1..6`. The column is rewritten to an Autumn plan id only by `refreshEntitlements` via `mapAutumnFeaturesToPlatformPlan`, which fires lazily on a plan read, so a dormant platform keeps `'standard'` indefinitely. Any cohort query written as `plan = 'free'` therefore selects only the platforms that have been active since the deploy and silently skips the dormant ones, which are usually the exact population a grandfathering or migration pass is meant to catch. Match `'standard'` as well. The console cannot supply a substitute date either: `autumn_customers.created_at` is when AP enrolled the platform, not when it signed up, and enrolment is lazy, so it says nothing about what plan a platform held on a given date. The whole table also only begins at the 2026-07-23 Autumn catalog go-live.
+- **No helper says whether a Cloud platform is enterprise.** `PlanName.ENTERPRISE` is checked nowhere server-side, `isCloudPlanButNotEnterprise` only detects free plans, and sales mints a new Autumn plan id per deal, often named after the customer. The working definition is the `default:` arm of `concurrencyLimitForCloudPlan` (`rate-limiter-interceptor.ts`), minus the legacy non-enterprise ids that arm misses: `standard` (old free tier) and `old_unlimited_trial*`. `internal` *is* enterprise. Those two misses also get the enterprise concurrency limit today.
- **Per-project credit usage is reconstructed from Autumn events, and any per-source split rides `properties.source`.** `getCreditUsage` (`autumn-utils.ts`) calls `aggregateEvents` on `apCredits` grouped by `properties.projectId` — there is no per-project balance, only the summed event stream. Every credit event carries `properties.source` (one of `CreditUsageSource` = `flow_run` | `ai` | `chat`; stamped in `sendTrackEvent` as `{ source, ...properties }`), so to break out a slice like AI usage (`ai` + `chat`) you run *additional* `aggregateEvents` passes with `filterBy: { source }` and merge by project — `filterBy` is AND with one value per key, so there is no OR: it's one call per source. The `projects-usage` table's `aiCreditsUsed` column is built exactly this way; no new tracking was added. CE's default provider returns `{ total: 0, byProject: [] }`, so any such column is `0` on Community.
- `CONSUMABLE_AUTUMN_FEATURE_IDS` (`apCredits`, `appSumoAiCredits`) is the source of truth splitting the two billing mechanics: consumables are prepaid balances the customer tops up (units added to a depleting pool); every other billable feature (e.g. seats) is a recurring per-unit quantity edited and charged each period — never "topped up".
- **A top-up only works if the plan's prepaid item is `interval: one_off`** — Autumn fires auto top-ups (and one-click credit purchases) exclusively against a one-off prepaid purchase path; a prepaid item priced `interval: month` is a *selectable monthly bucket* (a recurring subscription quantity), so there is nothing for the top-up to buy and it silently no-ops. Verified in sandbox 2026-08-01: `team`'s `apCredits` prepaid item is `one_off` and tops up; `free_legacy` and `appsumo` carry the `appSumoAiCredits` prepaid item at `interval: month` / `reset: month`, and a customer with the control enabled (threshold 170, card on file) crossed the threshold twice via `balances.track` with no purchase and `prepaid_grant` stuck at 0. `reset: month` on a top-up item is wrong for a second reason — it would wipe purchased credits each cycle. Nothing warns you: `toBillableFeatures` (`autumn-billing.ts`) surfaces any item with `billingMethod === 'prepaid'` regardless of interval, so the UI advertises a price the catalog cannot sell.
diff --git a/docs/docs.json b/docs/docs.json
index 401c2ffd0d6..47bdf36a2fe 100644
--- a/docs/docs.json
+++ b/docs/docs.json
@@ -209,6 +209,7 @@
"handbook/engineering/playbooks/releases",
"handbook/engineering/playbooks/canary-deployment",
"handbook/engineering/playbooks/queue-metrics",
+ "handbook/engineering/playbooks/alerts",
"handbook/engineering/playbooks/infrastructure",
"handbook/engineering/playbooks/database-migration",
"handbook/engineering/playbooks/structured-logging",
diff --git a/docs/handbook/engineering/playbooks/alerts.mdx b/docs/handbook/engineering/playbooks/alerts.mdx
new file mode 100644
index 00000000000..78cc54cded7
--- /dev/null
+++ b/docs/handbook/engineering/playbooks/alerts.mdx
@@ -0,0 +1,28 @@
+---
+title: 'Alerts'
+icon: 'bell'
+---
+
+Automated reports and alerts that tell us something is wrong before a customer does.
+
+## Enterprise OOM Report
+
+Runs that run out of memory end with status `MEMORY_LIMIT_EXCEEDED`. They are not failed queue jobs, so queue alerts never see them. This report surfaces them so we can reach out with a fix (bigger dedicated workers, flow optimization) before the customer escalates.
+
+- **Runs:** as a daily cron job on the DevOps machine, reading the read-only Postgres replica and storing one row per platform per day in ClickHouse.
+- **Who:** enterprise Cloud platforms only, every platform with at least one OOM run the previous day.
+- **Where:** one message in the internal Discord channel, one line per platform with OOM runs, flows affected and OOM rate. Nothing is sent on a quiet day.
+- **History:** the **Enterprise OOM Tracker** dashboard in HyperDX shows trends and every past day's report.
+- **If it stops:** a Betterstack heartbeat alerts when the daily job has not run.
+
+## Nightly Benchmark
+
+The `Benchmark` GitHub Actions workflow (`.github/workflows/benchmark.yml`) runs every night at 03:00 UTC, once per worker shape (`shared` and `dedicated`).
+
+- **Results:** each scheduled run stores one row per shape in ClickHouse. The **Nightly Benchmark** dashboard in HyperDX charts throughput, latency, queue wait and worker resources over time.
+- **Regression alert:** `benchmark/detect-regression.mjs` compares the run with the median of the last 14 runs and alerts Betterstack when a metric is more than 5% worse on 2 runs in a row. It stays silent until a shape has 3 runs of history.
+- **Pipeline alert:** on scheduled runs only, the job's last step opens a `bench-pipeline-<shape>` alert in Betterstack when anything in the job failed, and resolves it on the next green scheduled run.
+
+<Warning>
+Manual runs (`workflow_dispatch`) skip the ClickHouse writes and both Betterstack alert steps. A green manual run does not prove these steps work or resolve an existing pipeline alert. Check the next scheduled run after changing them or their secrets.
+</Warning>