Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

fix(monitoring): retune graphql soup volume and websocket surge monitors (#9547)

teoaf59731781e3
diff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.md
index a95a3d96e0..aa9c24e694 100644
--- a/infra/stacks/monitoring/README.md
+++ b/infra/stacks/monitoring/README.md
@@ -112,8 +112,9 @@ The gate is that preview:
(critical 2 GB → 250 MB, warning 3 GB → 500 MB) so 2 GiB instances stop
alerting at their normal level.
- **Already-deployed monitors update their `message`** (GoAlert handles instead
- of the channel), and the GraphQL soup volume, gateway connections and task
- count monitors their `query` and thresholds.
+ of the channel), and the GraphQL soup volume, websocket upgrade, gateway
+ connections and task count monitors their `query` and thresholds (volume
+ and websocket upgrades also gain recovery thresholds).
- **90 resources to create**: 70 monitors (31 prod threshold and anomaly, 19 GraphQL soup, 9
notification retention, 10 WhatsApp session, 1 native call push), 2 webhooks,
7 log-based metrics, 3 span-based metrics and 1 APM retention filter (6, 3
diff --git a/infra/stacks/monitoring/monitors/graphql-soup.ts b/infra/stacks/monitoring/monitors/graphql-soup.ts
index 819ee1a090..5f56ac3d85 100644
--- a/infra/stacks/monitoring/monitors/graphql-soup.ts
+++ b/infra/stacks/monitoring/monitors/graphql-soup.ts
@@ -4,9 +4,10 @@
// remove them freely once the rollout settles.
//
// Thresholds sit above the 7 days before the 2026-10-11 ramp: none of these
-// would have fired in that week except where noted. Volume, gateway
-// connections and task count were raised on 2026-10-29 above the 7 days
-// before it, after the rollout outgrew the pre-ramp numbers. PostHog covers
+// would have fired in that week except where noted. Gateway connections and
+// task count were raised on 2026-10-29 above the 7 days before it, after the
+// rollout outgrew the pre-ramp numbers; request volume and websocket upgrades
+// were raised again on 2026-11-04 from post-rollout traffic. PostHog covers
// the client side (exceptions, browser cache and draft-queue logs, usage) on
// https://us.posthog.com/project/299018/dashboard/2198651.
//
@@ -45,6 +46,8 @@ function rolloutMonitor(
description: string;
critical: string;
warning: string;
+ criticalRecovery?: string;
+ warningRecovery?: string;
service: string;
severe?: boolean;
}
@@ -62,7 +65,12 @@ ${ROLLOUT_CONTEXT}
${notify({ pages: args.severe ?? false })}`,
tags: ['env:prod', `service:${args.service}`, 'rollout:graphql-soup'],
priority: args.severe ? '2' : '3',
- monitorThresholds: { critical: args.critical, warning: args.warning },
+ monitorThresholds: {
+ critical: args.critical,
+ warning: args.warning,
+ criticalRecovery: args.criticalRecovery,
+ warningRecovery: args.warningRecovery,
+ },
renotifyInterval: RENOTIFY_MINUTES,
// APM and log counts are sparse at night; a full window would skip
// evaluations, and an empty count must read as zero rather than alert.
@@ -178,27 +186,37 @@ rolloutMonitor('graphql-soup-unauthenticated-mutations', {
});
// Volume. There is no rate limit on the endpoint, so a client retry loop or
-// reconnect herd shows up here before it shows up as errors. Over the 7 days
-// to 2026-10-29 the average was 16k requests per 15 minutes, peak 42k.
+// reconnect herd shows up here before it shows up as errors. Thresholds sit
+// at about 1.5x and 2x the 15-minute peaks of the 7 days to 2026-11-04, after
+// the flag reached 100% and traffic levelled off; none would have fired in
+// the 14 days to then. Recovery thresholds keep a value hovering at a
+// threshold from flapping. Retune if the client polls or reconnects more.
rolloutMonitor('graphql-soup-request-volume', {
name: '[PROD] GraphQL soup: request volume surge',
type: 'query alert',
- query: `sum(last_15m):sum:trace.http.server.request.hits{${GRAPHQL}}.as_count() > 50000`,
+ query: `sum(last_15m):sum:trace.http.server.request.hits{${GRAPHQL}}.as_count() > 100000`,
description:
- 'POST /dss/items/soup/graphql received {{value}} requests in 15 minutes (normal is about 16k, weekly peak 42k). Check for a client retry loop before scaling.',
- critical: '50000',
- warning: '45000',
+ 'POST /dss/items/soup/graphql received {{value}} requests in 15 minutes (weekday median about 27k, US daytime 39k-46k, 7-day peak 51k). Roughly double the daytime rate points to a client retry loop; check it before scaling.',
+ critical: '100000',
+ criticalRecovery: '80000',
+ warning: '75000',
+ warningRecovery: '60000',
service: 'cloud-storage-service',
});
+// Each deploy drains tasks and their clients reconnect at once: 130-340
+// upgrades in a minute, up to 733 in 15 minutes. Clients also reconnect at
+// about minute 27 of every hour (65-120 in that minute).
rolloutMonitor('graphql-soup-websocket-surge', {
name: '[PROD] GraphQL soup: websocket upgrade surge',
type: 'query alert',
- query: `sum(last_15m):sum:trace.http.server.request.hits{${GRAPHQL_WS}}.as_count() > 300`,
+ query: `sum(last_15m):sum:trace.http.server.request.hits{${GRAPHQL_WS}}.as_count() > 1800`,
description:
- 'GraphQL websocket upgrades reached {{value}} in 15 minutes (pre-rollout max was 168). A spike after a deploy or network blip is a reconnect herd.',
- critical: '300',
- warning: '150',
+ 'GraphQL websocket upgrades reached {{value}} in 15 minutes (normal 100-460, 7-day peak 733 when a deploy drained tasks). Above that, clients are reconnecting in a loop rather than once.',
+ critical: '1800',
+ criticalRecovery: '1400',
+ warning: '1200',
+ warningRecovery: '950',
service: 'cloud-storage-service',
});