AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
docs: alerts handbook page and enterprise OOM tracker (SRE-237) (#16712)
Amr Elmohamady4967ac05e76b
diff --git a/brain/knowledge/data-storage-observability/enterprise-oom-tracker.md b/brain/knowledge/data-storage-observability/enterprise-oom-tracker.mdnew file mode 100644index 00000000000..01c55b8efd5--- /dev/null+++ b/brain/knowledge/data-storage-observability/enterprise-oom-tracker.md@@ -0,0 +1,26 @@+---+icon: 🧠+---++# Enterprise OOM Tracker++Daily job that finds enterprise Cloud platforms whose runs end in `MEMORY_LIMIT_EXCEEDED`, so CS/SRE can reach out before the customer escalates (SRE-237). Those runs end as a flow-run status, not a failed BullMQ job, so nothing else alerts on them.++**Script** — `/root/platform-oom-tracker/platform-oom-tracker.js` on `ap-devops`, not in this repo. Runs daily at 12:00 Cairo time and processes yesterday (UTC day). The box's Debian cron has no `CRON_TZ`, so the crontab fires at 09:00 and 10:00 UTC behind a `TZ=Africa/Cairo date +%H = 12` guard, which keeps it on 12:00 through Egypt's DST switch. Log: `/var/log/platform-oom-tracker.log`.+**Source** — the read-only Postgres replica (`OOM_PG_URL` + `OOM_PG_SSL_CA` in the script's own `.env`), never the prod `/root/queue/.env`.+**Table** — ClickHouse `ops.platform_oom_daily` on the `Activepieces` service: one row per `(day, platform_id)` with OOM > 0, carrying `platform_name`, `plan`, `oom_runs`, `distinct_flows_oom`, `total_runs`. `ReplacingMergeTree`, so read with `FINAL`; insert user `oom_tracker`.+**Alert** — every enterprise platform with at least one OOM run yesterday, no dedup. One Discord message per day, posted straight to a channel webhook (`DISCORD_OOM_WEBHOOK_URL`), not through Betterstack, so no incident and no auto-created Linear ticket carrying customer names. One line per platform, sorted by OOM runs, trimmed to Discord's 2,000-character limit with "…and N more"; nothing is sent on a day with no OOMs.+**Dashboard** — HyperDX [Enterprise OOM Tracker](https://hyperdx.clickhouse.cloud/dashboards/6ad7451181c0f6e143965a5e): top platforms, OOM runs and platforms hit per day, and the per-day list each alert sends.+**Heartbeat** — Betterstack heartbeat 512932 "platform-oom-tracker (SRE-237)", 1d period + 3h grace, pinged only after insert and alert succeed; `/fail` on error.+**Enterprise** — `plan` not in `free, appsumo, free_legacy, plus*, team*, standard, old_unlimited_trial*`. See the plan-name gotcha in *EE Platform Plans & Billing*.++## Running it++- `node platform-oom-tracker.js --day YYYY-MM-DD` reprocesses one day; safe to repeat.+- `--backfill N --no-alert` loads the last N days without alerting. Run it under `nohup`: a day takes 25–90 s.++## Gotchas++- **The replica has `statement_timeout = 1min`.** The script sets its own session timeout (20 min); a query copied out of it into another client will be cancelled.+- **An `sslmode=` param in `OOM_PG_URL` silently discards `OOM_PG_SSL_CA`** in `pg` 8.x (the parsed URL overrides the `ssl` object), and DO console URLs ship `?sslmode=require`. The script refuses a URL with `ssl*` params.+- **ClickHouse resolves a `SELECT` alias everywhere in the query**, so `sumIf(oom_runs, …) AS oom_runs` followed by another `sumIf(oom_runs, …)` fails with `ILLEGAL_AGGREGATION`. Alias results with a different name.diff --git a/brain/knowledge/data-storage-observability/index.md b/brain/knowledge/data-storage-observability/index.mdindex 7d7dc7b8482..01b8435b58b 100644--- a/brain/knowledge/data-storage-observability/index.md+++ b/brain/knowledge/data-storage-observability/index.md@@ -104,3 +104,4 @@ Streams platform/project events to webhook URLs in real time — internal AP flo- **Analytics** — usage reporting- **Audit Logs** — the persisted security-action record- **Logging & Metrics (evlog)** — wide events, drains, log-based metrics, `system.snapshot`, OTLP queue gauge+- **Enterprise OOM Tracker** — daily `MEMORY_LIMIT_EXCEEDED` count per enterprise platform in ClickHouse, with a daily Discord report of every affected platformdiff --git a/brain/knowledge/execution-runtime/benchmark-cli.md b/brain/knowledge/execution-runtime/benchmark-cli.mdindex 512eb4a8303..1f4fa43ccd5 100644--- a/brain/knowledge/execution-runtime/benchmark-cli.md+++ b/brain/knowledge/execution-runtime/benchmark-cli.md@@ -6,6 +6,8 @@ icon: ⏱️`npx @activepieces/cli benchmark` load-tests a deployment's sync-webhook path and attributes *where* latency goes, so a self-hosted setup can be compared apples-to-apples against Activepieces' published reference numbers. Available in CE, EE, Cloud.+Nightly CI results (`.github/workflows/benchmark.yml`, scheduled runs only) are stored in ClickHouse and charted on the HyperDX [Nightly Benchmark](https://hyperdx.clickhouse.cloud/dashboards/6ab2f0392d54d4a63406098f) dashboard.+### How it works- Builds a `webhook → data-mapper → return-response` flow. Instead of a raw `--concurrency`, it **auto-discovers the deployment shape** (`GET /v1/worker-machines`) and drives load = the effective **execution slot** count, so a healthy deploy queues ~zero by construction. Any queue-wait it reports is a real finding (usually driven concurrency > slots).- Attributes latency from each `FlowRun.timeline` into QUEUE / PROVISION / BOOT / RUN, cross-checked against live BullMQ queue depth.diff --git a/brain/knowledge/platform-editions-ee/ee-platform-plans-billing.md b/brain/knowledge/platform-editions-ee/ee-platform-plans-billing.mdindex b628badd8bc..8a89370f89e 100644--- a/brain/knowledge/platform-editions-ee/ee-platform-plans-billing.md+++ b/brain/knowledge/platform-editions-ee/ee-platform-plans-billing.md@@ -49,6 +49,7 @@ Billing and entitlements are powered by [Autumn](https://useautumn.com). Each pl- **`Balance.nextResetAt` is the earliest reset across *every* grant, not the plan's** — so the credits card dates the wrong pool. Autumn's `auto_enable` free plan sits underneath every paid customer and keeps contributing its daily 100-credit bucket, so `nextResetAt` is always ~24h away while the plan's own allowance renews months out: a large `one_off` customer whose plan grant is a year away reads a next-day reset because the free tier's 100 credits reset then. The per-grant dates are in `balance.breakdown[].reset.resetsAt`, keyed by `plan_id`; `toBalanceCache` now takes the reset belonging to the **largest** `includedGrant`, which picks the plan grant without needing the current plan id threaded through `writeBalance`/`writeCustomerStateCaches` — safe because the free tier's 100/day is smaller than any paid allowance, and only wrong if a paid plan ever grants under 100 credits. It falls back to the aggregate when there is a single grant, so free-only platforms are untouched. The same aggregation inflates `granted` by the free tier's 100 on top of the plan grant, which is left alone — the customer really can spend all of it. Note this is a *different* problem from the reset-cadence bullet above: that one is about the interval (day vs month vs year) being absent from the balance, this one is about the date being the wrong grant's.- **`platform.usage.creditsRemaining` collapses three states into two values, and none of them tells you whether to gate the user.** `getUsage` maps the provider's `CreditsUsage` as `isNil(credits) ? 0 : credits.remaining`, while `toCreditsUsage` (`autumn-billing.ts`) nulls `remaining` for an unlimited balance — so **`null` means unlimited** and **`0` means either genuinely empty or unknown** (Autumn unreachable, cold cache); the whole `usage` object is `undefined` only on CE. Percent-based UI survives the unknown case by accident, not by design: `creditsUsed` is 0 too, so `total` is 0 and `billingUtils.percentUsed` returns 0, landing on the quiet branch. What `creditsRemaining` never carries is whether credits are *enforced* — that is the separate `platform.billingEnforced` flag (decision 000020), and a platform can meter a finite balance with the flag off. Any credit-nag surface has to read both; a surface driven by the percentage alone nags customers whose runs nothing is actually blocking, and the louder the surface (a non-dismissible dashboard banner rather than a sidebar figure) the worse that reads. The web side normalises it once as `useCreditsUsage().isBillingEnforced` (`features/billing/hooks/use-credits-usage.ts`) and gates on it in three places — `billingUtils.shouldShowCreditsAlert`, `CreditsActionButton`, and `useCreditsState`'s `showLowCreditsWarning`; note the chat hook originally gated only its *exhausted* branch, so the warning branch leaked for a while. Because `false` also means "cold enforcement cache", every one of these fails silent rather than open, matching decision 000020.- **`platform_plan.plan` still carries pre-Autumn plan names on any platform that has not been read since the migration deployed.** There was no `'free'` plan before Autumn: the Cloud free tier was `plan = 'standard'` (`STANDARD_CLOUD_PLAN`), and the old `PlanName` enum held only `STANDARD`, `ENTERPRISE`, and `APPSUMO_ACTIVEPIECES_TIER1..6`. The column is rewritten to an Autumn plan id only by `refreshEntitlements` via `mapAutumnFeaturesToPlatformPlan`, which fires lazily on a plan read, so a dormant platform keeps `'standard'` indefinitely. Any cohort query written as `plan = 'free'` therefore selects only the platforms that have been active since the deploy and silently skips the dormant ones, which are usually the exact population a grandfathering or migration pass is meant to catch. Match `'standard'` as well. The console cannot supply a substitute date either: `autumn_customers.created_at` is when AP enrolled the platform, not when it signed up, and enrolment is lazy, so it says nothing about what plan a platform held on a given date. The whole table also only begins at the 2026-07-23 Autumn catalog go-live.+- **No helper says whether a Cloud platform is enterprise.** `PlanName.ENTERPRISE` is checked nowhere server-side, `isCloudPlanButNotEnterprise` only detects free plans, and sales mints a new Autumn plan id per deal, often named after the customer. The working definition is the `default:` arm of `concurrencyLimitForCloudPlan` (`rate-limiter-interceptor.ts`), minus the legacy non-enterprise ids that arm misses: `standard` (old free tier) and `old_unlimited_trial*`. `internal` *is* enterprise. Those two misses also get the enterprise concurrency limit today.- **Per-project credit usage is reconstructed from Autumn events, and any per-source split rides `properties.source`.** `getCreditUsage` (`autumn-utils.ts`) calls `aggregateEvents` on `apCredits` grouped by `properties.projectId` — there is no per-project balance, only the summed event stream. Every credit event carries `properties.source` (one of `CreditUsageSource` = `flow_run` | `ai` | `chat`; stamped in `sendTrackEvent` as `{ source, ...properties }`), so to break out a slice like AI usage (`ai` + `chat`) you run *additional* `aggregateEvents` passes with `filterBy: { source }` and merge by project — `filterBy` is AND with one value per key, so there is no OR: it's one call per source. The `projects-usage` table's `aiCreditsUsed` column is built exactly this way; no new tracking was added. CE's default provider returns `{ total: 0, byProject: [] }`, so any such column is `0` on Community.- `CONSUMABLE_AUTUMN_FEATURE_IDS` (`apCredits`, `appSumoAiCredits`) is the source of truth splitting the two billing mechanics: consumables are prepaid balances the customer tops up (units added to a depleting pool); every other billable feature (e.g. seats) is a recurring per-unit quantity edited and charged each period — never "topped up".- **A top-up only works if the plan's prepaid item is `interval: one_off`** — Autumn fires auto top-ups (and one-click credit purchases) exclusively against a one-off prepaid purchase path; a prepaid item priced `interval: month` is a *selectable monthly bucket* (a recurring subscription quantity), so there is nothing for the top-up to buy and it silently no-ops. Verified in sandbox 2026-08-01: `team`'s `apCredits` prepaid item is `one_off` and tops up; `free_legacy` and `appsumo` carry the `appSumoAiCredits` prepaid item at `interval: month` / `reset: month`, and a customer with the control enabled (threshold 170, card on file) crossed the threshold twice via `balances.track` with no purchase and `prepaid_grant` stuck at 0. `reset: month` on a top-up item is wrong for a second reason — it would wipe purchased credits each cycle. Nothing warns you: `toBillableFeatures` (`autumn-billing.ts`) surfaces any item with `billingMethod === 'prepaid'` regardless of interval, so the UI advertises a price the catalog cannot sell.diff --git a/docs/docs.json b/docs/docs.jsonindex 401c2ffd0d6..47bdf36a2fe 100644--- a/docs/docs.json+++ b/docs/docs.json@@ -209,6 +209,7 @@"handbook/engineering/playbooks/releases","handbook/engineering/playbooks/canary-deployment","handbook/engineering/playbooks/queue-metrics",+ "handbook/engineering/playbooks/alerts","handbook/engineering/playbooks/infrastructure","handbook/engineering/playbooks/database-migration","handbook/engineering/playbooks/structured-logging",diff --git a/docs/handbook/engineering/playbooks/alerts.mdx b/docs/handbook/engineering/playbooks/alerts.mdxnew file mode 100644index 00000000000..78cc54cded7--- /dev/null+++ b/docs/handbook/engineering/playbooks/alerts.mdx@@ -0,0 +1,28 @@+---+title: 'Alerts'+icon: 'bell'+---++Automated reports and alerts that tell us something is wrong before a customer does.++## Enterprise OOM Report++Runs that run out of memory end with status `MEMORY_LIMIT_EXCEEDED`. They are not failed queue jobs, so queue alerts never see them. This report surfaces them so we can reach out with a fix (bigger dedicated workers, flow optimization) before the customer escalates.++- **Runs:** as a daily cron job on the DevOps machine, reading the read-only Postgres replica and storing one row per platform per day in ClickHouse.+- **Who:** enterprise Cloud platforms only, every platform with at least one OOM run the previous day.+- **Where:** one message in the internal Discord channel, one line per platform with OOM runs, flows affected and OOM rate. Nothing is sent on a quiet day.+- **History:** the **Enterprise OOM Tracker** dashboard in HyperDX shows trends and every past day's report.+- **If it stops:** a Betterstack heartbeat alerts when the daily job has not run.++## Nightly Benchmark++The `Benchmark` GitHub Actions workflow (`.github/workflows/benchmark.yml`) runs every night at 03:00 UTC, once per worker shape (`shared` and `dedicated`).++- **Results:** each scheduled run stores one row per shape in ClickHouse. The **Nightly Benchmark** dashboard in HyperDX charts throughput, latency, queue wait and worker resources over time.+- **Regression alert:** `benchmark/detect-regression.mjs` compares the run with the median of the last 14 runs and alerts Betterstack when a metric is more than 5% worse on 2 runs in a row. It stays silent until a shape has 3 runs of history.+- **Pipeline alert:** on scheduled runs only, the job's last step opens a `bench-pipeline-<shape>` alert in Betterstack when anything in the job failed, and resolves it on the next green scheduled run.++<Warning>+Manual runs (`workflow_dispatch`) skip the ClickHouse writes and both Betterstack alert steps. A green manual run does not prove these steps work or resolve an existing pipeline alert. Check the next scheduled run after changing them or their secrets.+</Warning>