AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
docs: alerts handbook covers queue and host alerts, OOM report moves to Box (#16866)
Amr Elmohamady935d47addd99
diff --git a/brain/knowledge/data-storage-observability/enterprise-oom-tracker.md b/brain/knowledge/data-storage-observability/enterprise-oom-tracker.mdindex 01c55b8efd5..e1c83b7dfaf 100644--- a/brain/knowledge/data-storage-observability/enterprise-oom-tracker.md+++ b/brain/knowledge/data-storage-observability/enterprise-oom-tracker.md@@ -9,7 +9,7 @@ Daily job that finds enterprise Cloud platforms whose runs end in `MEMORY_LIMIT_**Script** — `/root/platform-oom-tracker/platform-oom-tracker.js` on `ap-devops`, not in this repo. Runs daily at 12:00 Cairo time and processes yesterday (UTC day). The box's Debian cron has no `CRON_TZ`, so the crontab fires at 09:00 and 10:00 UTC behind a `TZ=Africa/Cairo date +%H = 12` guard, which keeps it on 12:00 through Egypt's DST switch. Log: `/var/log/platform-oom-tracker.log`.**Source** — the read-only Postgres replica (`OOM_PG_URL` + `OOM_PG_SSL_CA` in the script's own `.env`), never the prod `/root/queue/.env`.**Table** — ClickHouse `ops.platform_oom_daily` on the `Activepieces` service: one row per `(day, platform_id)` with OOM > 0, carrying `platform_name`, `plan`, `oom_runs`, `distinct_flows_oom`, `total_runs`. `ReplacingMergeTree`, so read with `FINAL`; insert user `oom_tracker`.-**Alert** — every enterprise platform with at least one OOM run yesterday, no dedup. One Discord message per day, posted straight to a channel webhook (`DISCORD_OOM_WEBHOOK_URL`), not through Betterstack, so no incident and no auto-created Linear ticket carrying customer names. One line per platform, sorted by OOM runs, trimmed to Discord's 2,000-character limit with "…and N more"; nothing is sent on a day with no OOMs.+**Alert** — one Box signal per day (`POST $SIGNALS_URL`, `fingerprint: enterprise-oom-<day>`) when any enterprise platform had an OOM run yesterday; nothing on a quiet day. The `message` is the investigation prompt and the `stack` lists platforms by id only (plan, OOM runs, flows, total runs, rate). The Box agent reads the replica and ClickHouse itself, keeps every customer name in a private org artifact, and posts the Discord summary with that link. Moved off the direct Discord webhook on 2026-10-26; `DISCORD_OOM_WEBHOOK_URL` is still in `.env` but unused.**Dashboard** — HyperDX [Enterprise OOM Tracker](https://hyperdx.clickhouse.cloud/dashboards/6ad7451181c0f6e143965a5e): top platforms, OOM runs and platforms hit per day, and the per-day list each alert sends.**Heartbeat** — Betterstack heartbeat 512932 "platform-oom-tracker (SRE-237)", 1d period + 3h grace, pinged only after insert and alert succeed; `/fail` on error.**Enterprise** — `plan` not in `free, appsumo, free_legacy, plus*, team*, standard, old_unlimited_trial*`. See the plan-name gotcha in *EE Platform Plans & Billing*.@@ -23,4 +23,6 @@ Daily job that finds enterprise Cloud platforms whose runs end in `MEMORY_LIMIT_- **The replica has `statement_timeout = 1min`.** The script sets its own session timeout (20 min); a query copied out of it into another client will be cancelled.- **An `sslmode=` param in `OOM_PG_URL` silently discards `OOM_PG_SSL_CA`** in `pg` 8.x (the parsed URL overrides the `ssl` object), and DO console URLs ship `?sslmode=require`. The script refuses a URL with `ssl*` params.+- **`internal` counts as enterprise.** "Enterprise" is any plan not in `NON_ENTERPRISE_PLANS`, so the `internal` plan (our own platforms) shows up in the report.+- **A manual `--day` run sends a real signal** and starts an agent. Rerunning a day Box already has only adds to that day's issue, because the fingerprint is the day.- **ClickHouse resolves a `SELECT` alias everywhere in the query**, so `sumIf(oom_runs, …) AS oom_runs` followed by another `sumIf(oom_runs, …)` fails with `ILLEGAL_AGGREGATION`. Alias results with a different name.diff --git a/brain/knowledge/data-storage-observability/index.md b/brain/knowledge/data-storage-observability/index.mdindex 01b8435b58b..bc8c3923328 100644--- a/brain/knowledge/data-storage-observability/index.md+++ b/brain/knowledge/data-storage-observability/index.md@@ -104,4 +104,4 @@ Streams platform/project events to webhook URLs in real time — internal AP flo- **Analytics** — usage reporting- **Audit Logs** — the persisted security-action record- **Logging & Metrics (evlog)** — wide events, drains, log-based metrics, `system.snapshot`, OTLP queue gauge-- **Enterprise OOM Tracker** — daily `MEMORY_LIMIT_EXCEEDED` count per enterprise platform in ClickHouse, with a daily Discord report of every affected platform+- **Enterprise OOM Tracker** — daily `MEMORY_LIMIT_EXCEEDED` count per enterprise platform in ClickHouse, with a daily Box signal whose agent finds each platform's causediff --git a/docs/handbook/engineering/playbooks/alerts.mdx b/docs/handbook/engineering/playbooks/alerts.mdxindex 32fd3908133..bd8ba877524 100644--- a/docs/handbook/engineering/playbooks/alerts.mdx+++ b/docs/handbook/engineering/playbooks/alerts.mdx@@ -5,16 +5,38 @@ icon: 'bell'Automated reports and alerts that tell us something is wrong before a customer does.+Every alert below except the AI chat alerts ends up in **Box**. Box groups repeats of the same alert into one issue by fingerprint and starts a Claude agent only for a new issue, or for a resolved one that comes back. The agent investigates, files the Linear ticket for the team that owns the cause, and opens a pull request when the fix is small. Resolve the Box issue once its fix has shipped, and never ignore one: an ignored issue never starts an agent again.+## Enterprise OOM ReportRuns that run out of memory end with status `MEMORY_LIMIT_EXCEEDED`. They are not failed queue jobs, so queue alerts never see them. This report surfaces them so we can reach out with a fix (bigger dedicated workers, flow optimization) before the customer escalates.- **Runs:** as a daily cron job on the DevOps machine, reading the read-only Postgres replica and storing one row per platform per day in ClickHouse.- **Who:** enterprise Cloud platforms only, every platform with at least one OOM run the previous day.-- **Where:** one message in the internal Discord channel, one line per platform with OOM runs, flows affected and OOM rate. Nothing is sent on a quiet day.+- **Where:** one Box signal per day (`enterprise-oom-<day>`), listing each platform by id with OOM runs, flows affected and OOM rate. The agent finds why each platform runs out of memory from the replica and the OOM history, decides whether it is the customer's flow design or a platform or piece bug, and posts a short summary to Discord. Nothing is sent on a quiet day.+- **Customer data:** platform, project and flow names go only into a private artifact for the Activepieces org. The Linear ticket, pull requests and the Discord summary carry platform ids and the artifact link.- **History:** the **Enterprise OOM Tracker** dashboard in HyperDX shows trends and every past day's report.- **If it stops:** a Betterstack heartbeat alerts when the daily job has not run.+## Queue and Host Alerts++ClickStack (HyperDX) alerts on the production logs, all sending to the **Box Inbox** webhook. Each one fires separately per group, so each failing error signature or struggling host sends its own alert.++| Alert | Fires when | Checked every | Grouped by |+| --- | --- | --- | --- |+| Queue job failure | a `workerJobs` job logs `job.failed` | 15 min | error signature |+| Worker RSS high | a worker's memory passes 950 MB (the container is killed at about 1 GB) | 15 min | worker host |+| App RSS high | an app server's memory passes 3,500 MB | 15 min | app host |+| Worker event-loop wedged | a worker's event-loop delay (p99) passes 90 s | 5 min | worker host |+| App event-loop wedged | an app server's event-loop delay (p99) passes 5 s | 5 min | app host |++- **Where:** the [HyperDX alerts page](https://hyperdx.clickhouse.cloud/alerts). Each alert's message links to the matching log search.+- **Memory and event-loop data** comes from the `system.snapshot` event every process logs once a minute.++## AI Chat Alerts++Two ClickStack alerts owned by the AI team (AI-20) post to the `#ai-team` Discord channel instead of Box: **Failed chat turns per hour** (fires at 8 or more) and **Avg input tokens per chat turn**, checked daily (fires above 750k, about 1.25 times normal).+## Nightly BenchmarkThe `Benchmark` GitHub Actions workflow (`.github/workflows/benchmark.yml`) runs every night at 03:00 UTC, once per worker shape (`shared` and `dedicated`).