Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
fix(notifications): enable hourly retention in dev and prod (#9057)
Sean Ayeff66f7ac95ee
diff --git a/docs/runbooks/notification-retention.md b/docs/runbooks/notification-retention.mdindex 0609fdd0b2..5eda39b4db 100644--- a/docs/runbooks/notification-retention.md+++ b/docs/runbooks/notification-retention.md@@ -1,9 +1,15 @@# Notification retentionThe internal `notification_retention_handler` Lambda removes notifications older-than 30 complete UTC days. Its EventBridge schedule is **disabled** in both dev-and prod, and its event payload defaults to **dry-run**. Deploying the code or-migrations alone does not purge history.+than 30 complete UTC days. Both dev and prod stack configs **enable** its+hourly EventBridge schedule in **deletion mode**. Deploying either+notification-service stack activates cleanup, including the historical backlog;+committing the config or applying migrations alone does not. These are desired+settings, not proof of the deployed schedule's state. Merging to `main` triggers+normal dev deployment; a production release applies the prod settings. Complete+the rollout prerequisites before merging or releasing this activation. Missing+stack settings still default to disabled/dry-run, and manual invocations with+`{}` remain read-only.## Semantics@@ -18,7 +24,7 @@ migrations alone does not purge history.device registrations, and email-send bookkeeping are not removed.- Old unread notifications disappear. Items whose only feed-placement reason wasan expired notification may disappear from that feed, not from source storage.-- This is eventual cleanup, normally within a five-minute schedule interval plus+- This is eventual cleanup, normally within a one-hour schedule interval plusexecution time after expiry once caught up. Backlogs, failures, or locked rowscan extend that delay. There is no new read-time expiration filter.- **Replay protection is intentionally out of scope.** Parent-row existence still@@ -137,10 +143,13 @@ at Kafka: it does **not** measure time queued before PostgreSQL NOTIFY is forwaror acknowledge client delivery. Existing best-effort loss/reconnect semanticsremain unchanged; no durable outbox is added by this observability change.-Before the next deletion canary, after approved deployments:+Before first enabling deletion in an environment, verify these prerequisites;+code approval does not establish that the live checks have passed:-1. Deploy the notification-service stack (extension + worker) and DSS (consumer- telemetry). Keep both cleanup schedules disabled and dry-run enabled.+1. For a telemetry-only deployment of the notification-service stack (extension+ + worker) and DSS (consumer telemetry), explicitly override the checked-in+ retention settings to disabled/dry-run before deploying. The checked-in+ settings otherwise enable deletion on deployment.2. Invoke explicit dry runs and verify the complete invocation and batch/readspans in Datadog, with the correct service/env and no exporter/flush errors.3. Confirm health logs from each running DSS instance, including healthy zero@@ -157,8 +166,12 @@ Before the next deletion canary, after approved deployments:## Rollout-Production deletion needs explicit approval and a verified backup/PITR recovery-procedure. Do not run a historical `DELETE` inside a migration.+Deployment needs explicit approval; production deletion also needs a verified+backup/PITR recovery procedure. The checked-in dev and prod settings select the+final deletion stage below; they do not verify the rollout prerequisites.+For an environment that has not completed them, explicitly override the config+to disabled/dry-run before its first deployment and follow the staged checks.+Do not run a historical `DELETE` inside a migration.1. Apply the additive migrations through the normal approved database workflow.The retention index is built concurrently in its own nontransactional@@ -166,9 +179,10 @@ procedure. Do not run a historical `DELETE` inside a migration.changing its contract. Verify the index is valid. If a concurrent build fails,inspect its invalid index and repair/retry under the database deploymentprocedure rather than marking the migration successful.-2. Deploy the notification-service stack. Its deployment configuration also builds- the new Lambda. Keep `notification_retention_enabled: false` and- `notification_retention_dry_run: true` initially.+2. For an initial staged rollout, set `notification_retention_enabled: false`+ and `notification_retention_dry_run: true` before deploying the+ notification-service stack. These override the checked-in deletion settings.+ Its deployment configuration also builds the Lambda.3. Inspect age distribution, recipient/receipt fanout, and query plans againstrepresentative data. Use plain `EXPLAIN` for production deletion plans;`EXPLAIN ANALYZE DELETE` actually deletes data.@@ -184,8 +198,9 @@ procedure. Do not run a historical `DELETE` inside a migration.5. After approval and reviewing dry-run logs, change`notification_retention_dry_run` to `false` and deploy. Drain existing historywith these same bounded batches; do not substitute a bulk deletion. Initial- capacity is at most 10,000 parents per invocation (usually less). Verify- steady-state capacity exceeds incoming expiration volume before enabling prod.+ capacity is at most 10,000 parents per hourly invocation (usually less), or+ 240,000 per day. Verify steady-state capacity exceeds incoming expiration+ volume before enabling prod.6. Monitor application latency, database CPU/I/O, WAL and replica lag, dead tuples,vacuum progress, NOTIFY warnings/queue pressure, and realtime consumer lag.Pause the schedule on unacceptable impact. Do not automatically run@@ -199,8 +214,10 @@ retention period or batch caps.## Monitoring and stopping-Infrastructure alarms on Lambda errors, throttles, and no invocations for-15 minutes while the schedule is enabled. Each sweep logs the cutoff,+Infrastructure alarms on Lambda errors, throttles, and three consecutive hourly+periods without invocations while the schedule is enabled. The invocation alarm+fills missing hourly samples with zero and requires all three periods to breach,+so older invocations cannot mask a stopped schedule. Each sweep logs the cutoff,`sampled_expired`, `oldest_expired_at`, committed parent `deleted` count,attempted `batches`, and stopping reason. Each committed batch is also logged,so partial progress remains visible if a later batch fails.@@ -209,7 +226,7 @@ Track `cutoff - oldest_expired_at` as expiration backlog age. During backfill itshould trend downward; once caught up, sustained growth indicates insufficientcapacity, long-lived locks, or a failing high-fanout parent. A zero-row batch canmean rows were locked, not necessarily that every expired row has been removed.-Do not run a full `COUNT(*)` over the backlog every five minutes.+Do not run a full `COUNT(*)` over the backlog on each sweep.Disable `notification_retention_enabled` and deploy to stop future scheduledinvocations. An already-running invocation may still commit batches. Emergencydiff --git a/infra/bun.lock b/infra/bun.lockindex fc9099622b..ee962ca24a 100644--- a/infra/bun.lock+++ b/infra/bun.lock@@ -190,6 +190,9 @@"stacks/notification-service": {"name": "notification-service-stack","version": "0.0.0",+ "devDependencies": {+ "yaml": "^2.8.1",+ },},"stacks/observability": {"name": "observability-stack",@@ -927,6 +930,8 @@"yallist": ["[email protected]", "", {}, "sha512-YgvUTfwqyc7UXVMrB+SImsVYSmTS8X/tSrtdNZMImM+n7+QTriRXyXim0mBrTXNeqzVF0KWGgHPeiyViFFrNDw=="],+ "yaml": ["[email protected]", "", { "bin": { "yaml": "bin.mjs" } }, "sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw=="],+"yargs": ["[email protected]", "", { "dependencies": { "cliui": "^8.0.1", "escalade": "^3.1.1", "get-caller-file": "^2.0.5", "require-directory": "^2.1.1", "string-width": "^4.2.3", "y18n": "^5.0.5", "yargs-parser": "^21.1.1" } }, "sha512-GZtjxm/J/4TSxuL3FNYjCmLktBTnIw/rVmKSIyKeYAZpmJB2ig9VauCC5xsa82GNKVKDAqpOn3KVzNt0zmrU0g=="],"yargs-parser": ["[email protected]", "", {}, "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw=="],diff --git a/infra/stacks/notification-service/AGENTS.md b/infra/stacks/notification-service/AGENTS.mdindex 0eb2136d80..a1f2b3c2f0 100644--- a/infra/stacks/notification-service/AGENTS.md+++ b/infra/stacks/notification-service/AGENTS.md@@ -10,12 +10,15 @@ notification-retention Lambda. Runtime business policy lives in Rust.From `infra/`:- `bun test stacks/notification-service/retention.test.ts`- `bun run check`-- `bunx biome check stacks/notification-service`+- Run `biome ci --changed --no-errors-on-unmatched --error-on-warnings` inside+ the Nix shell to match CI; the Bun-installed formatter can differ.## Gotchas-- Keep retention disabled/dry-run until explicitly approved. Deployment does not- authorize deletion; `docs/runbooks/notification-retention.md` owns rollout.+- Dev and prod configs enable hourly 30-day retention in deletion mode.+ Deploying either stack activates cleanup; deployment still needs+ explicit approval and the checks in `docs/runbooks/notification-retention.md`.+- Keep the disabled/dry-run defaults for missing config and empty manual events.- The retention function needs its pinned x86_64 Datadog extension to receivegRPC OTLP; the ECS sidecar default alone does not work in Lambda.- Pass the existing Datadog key ARN, not its value, and scope the role to that key.diff --git a/infra/stacks/notification-service/Pulumi.dev.yaml b/infra/stacks/notification-service/Pulumi.dev.yamlindex b0537a4f73..564c6aca5c 100644--- a/infra/stacks/notification-service/Pulumi.dev.yaml+++ b/infra/stacks/notification-service/Pulumi.dev.yaml@@ -2,8 +2,8 @@ config:aws-native:region: us-east-1aws:region: us-east-1notification-service:macro_db_secret_key: macro-db-dev- notification-service:notification_retention_enabled: false- notification-service:notification_retention_dry_run: true+ notification-service:notification_retention_enabled: true+ notification-service:notification_retention_dry_run: falsenotification-service:jwt_secret_key: fusionauth-jwt-secret-devnotification-service:fusionauth_client_id: fusionauth-client-id-key-devnotification-service:fusionauth_issuer: fusionauth-dev.macro.comdiff --git a/infra/stacks/notification-service/Pulumi.prod.yaml b/infra/stacks/notification-service/Pulumi.prod.yamlindex 2c9076cd67..0938db23ea 100644--- a/infra/stacks/notification-service/Pulumi.prod.yaml+++ b/infra/stacks/notification-service/Pulumi.prod.yaml@@ -2,8 +2,8 @@ config:aws-native:region: us-east-1aws:region: us-east-1notification-service:macro_db_secret_key: macro-db-prod- notification-service:notification_retention_enabled: false- notification-service:notification_retention_dry_run: true+ notification-service:notification_retention_enabled: true+ notification-service:notification_retention_dry_run: falsenotification-service:jwt_secret_key: fusionauth-jwt-secret-prodnotification-service:fusionauth_client_id: fusionauth-client-id-key-prodnotification-service:fusionauth_issuer: auth.macro.comdiff --git a/infra/stacks/notification-service/package.json b/infra/stacks/notification-service/package.jsonindex 646ebd3bb0..16ba64f9fa 100644--- a/infra/stacks/notification-service/package.json+++ b/infra/stacks/notification-service/package.json@@ -3,5 +3,8 @@"version": "0.0.0","private": true,"license": "MIT",- "main": "index.ts"+ "main": "index.ts",+ "devDependencies": {+ "yaml": "^2.8.1"+ }}diff --git a/infra/stacks/notification-service/retention.test.ts b/infra/stacks/notification-service/retention.test.tsindex 8b66b7c2c2..b9a1875417 100644--- a/infra/stacks/notification-service/retention.test.ts+++ b/infra/stacks/notification-service/retention.test.ts@@ -1,5 +1,7 @@+import { file } from 'bun';import { beforeAll, expect, mock, test } from 'bun:test';import * as pulumi from '@pulumi/pulumi';+import { parse } from 'yaml';const resources: pulumi.runtime.MockResourceArgs[] = [];const secretArn =@@ -70,8 +72,11 @@ beforeAll(async () => {await pulumi.runtime.waitForRPCs();});-function resource(type: string): pulumi.runtime.MockResourceArgs {- const value = resources.find((r) => r.type === type);+function resource(+ type: string,+ registered = resources+): pulumi.runtime.MockResourceArgs {+ const value = registered.find((r) => r.type === type);if (!value) throw new Error(`Missing ${type}`);return value;}@@ -107,6 +112,85 @@ test('extension access is scoped to the existing Datadog key, not all secrets',]);});+test.each(['dev', 'prod'])('%s uses hourly retention', async (stack) => {+ const parsed: unknown = parse(+ await file(new URL(`./Pulumi.${stack}.yaml`, import.meta.url)).text()+ );+ if (+ !parsed ||+ typeof parsed !== 'object' ||+ !('config' in parsed) ||+ !parsed.config ||+ typeof parsed.config !== 'object'+ ) {+ throw new Error('Missing stack config');+ }+ pulumi.runtime.setAllConfig({+ 'aws:region': 'us-east-1',+ ...Object.fromEntries(+ Object.entries(parsed.config)+ .filter(([, value]) => typeof value === 'boolean')+ .map(([key, value]) => [key, String(value)])+ ),+ });+ const config = new pulumi.Config('notification-service');+ const start = resources.length;+ const { NotificationRetention } = await import('./retention');+ new NotificationRetention(`retention-${stack}`, {+ databaseUrl: pulumi.secret('test-database'),+ enabled: config.getBoolean('notification_retention_enabled') ?? false,+ dryRun: config.getBoolean('notification_retention_dry_run') ?? true,+ vpc: {+ vpcId: 'vpc-test',+ publicSubnetIds: [],+ privateSubnetIds: ['subnet-test'],+ },+ tags: { environment: stack },+ });+ await pulumi.runtime.waitForRPCs();+ const registered = resources.slice(start);+ expect(+ resource('aws:cloudwatch/eventRule:EventRule', registered).inputs+ ).toMatchObject({ state: 'ENABLED', scheduleExpression: 'rate(1 hour)' });+ expect(+ JSON.parse(+ resource('aws:cloudwatch/eventTarget:EventTarget', registered).inputs+ .input+ )+ ).toEqual({ dry_run: false });+ expect(+ registered.find(+ (r) =>+ r.type === 'aws:cloudwatch/metricAlarm:MetricAlarm' &&+ r.name === 'notification_retention_handler-missing-invocations'+ )?.inputs+ ).toMatchObject({+ metricQueries: [+ {+ id: 'invocations',+ metric: {+ namespace: 'AWS/Lambda',+ metricName: 'Invocations',+ period: 3600,+ stat: 'Sum',+ },+ returnData: false,+ },+ {+ id: 'hourly_invocations',+ expression: 'FILL(invocations, 0)',+ returnData: true,+ },+ ],+ evaluationPeriods: 3,+ datapointsToAlarm: 3,+ threshold: 1,+ comparisonOperator: 'LessThanThreshold',+ actionsEnabled: true,+ treatMissingData: 'breaching',+ });+});+test('telemetry does not enable cleanup or duplicate the existing log path', () => {expect(resource('aws:cloudwatch/eventRule:EventRule').inputs.state).toBe('DISABLED'diff --git a/infra/stacks/notification-service/retention.ts b/infra/stacks/notification-service/retention.tsindex 99f867597e..1705586f43 100644--- a/infra/stacks/notification-service/retention.ts+++ b/infra/stacks/notification-service/retention.ts@@ -34,7 +34,7 @@ type Args = {tags: Record<string, string>;};-/** Isolated maintenance worker: disabled initially, then preview before deleting. */+/** Isolated maintenance worker with explicit scheduling and dry-run controls. */export class NotificationRetention extends pulumi.ComponentResource {readonly lambda: aws.lambda.Function;@@ -116,7 +116,7 @@ export class NotificationRetention extends pulumi.ComponentResource {const rule = new aws.cloudwatch.EventRule(`${BASE_NAME}-schedule`,{- scheduleExpression: 'rate(5 minutes)',+ scheduleExpression: 'rate(1 hour)',state: enabled ? 'ENABLED' : 'DISABLED',tags,},@@ -175,12 +175,27 @@ export class NotificationRetention extends pulumi.ComponentResource {new aws.cloudwatch.MetricAlarm(`${BASE_NAME}-missing-invocations`,{- namespace: 'AWS/Lambda',- metricName: 'Invocations',- dimensions: { FunctionName: this.lambda.name },- statistic: 'Sum',- period: 900,- evaluationPeriods: 1,+ metricQueries: [+ {+ id: 'invocations',+ metric: {+ namespace: 'AWS/Lambda',+ metricName: 'Invocations',+ dimensions: { FunctionName: this.lambda.name },+ period: 3600,+ stat: 'Sum',+ },+ returnData: false,+ },+ {+ id: 'hourly_invocations',+ // Fill silent hours so older samples cannot mask a stopped schedule.+ expression: 'FILL(invocations, 0)',+ returnData: true,+ },+ ],+ evaluationPeriods: 3,+ datapointsToAlarm: 3,threshold: 1,comparisonOperator: 'LessThanThreshold',treatMissingData: enabled ? 'breaching' : 'notBreaching',diff --git a/services/notification_retention_handler/AGENTS.md b/services/notification_retention_handler/AGENTS.mdindex b26ce493f3..318d4da956 100644--- a/services/notification_retention_handler/AGENTS.md+++ b/services/notification_retention_handler/AGENTS.md@@ -10,7 +10,7 @@ Internal scheduled notification cleanup. The owning notification domain enforces- `src/main.rs`: one-connection pool, retained telemetry provider, post-invocationflush. Close handler spans before flushing; never shut down warm providers.- `src/handler.rs`: explicit dry-run default, invocation deadline and request ID.-- `infra/stacks/notification-service/retention.ts`: disabled schedule, pinned+- `infra/stacks/notification-service/retention.ts`: configurable schedule, pinnedDatadog extension, scoped key access and nonduplicated log forwarding.## Checks@@ -26,7 +26,8 @@ Internal scheduled notification cleanup. The owning notification domain enforcesof already-committed deletion. Flush time comes from reserved handler headroom.- Unit tests cannot prove the deployed extension exported traces. Verify anexplicitly approved dry run in Datadog before production cleanup.-- The schedule stays disabled until the rollout is explicitly approved.+- Dev and prod stack configs enable scheduled deletion every hour;+ deployment still needs explicit approval and the runbook's rollout checks.## Related