Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

fix(notifications): enable hourly retention in dev and prod (#9057)

Sean Ayeff66f7ac95ee
diff --git a/docs/runbooks/notification-retention.md b/docs/runbooks/notification-retention.md
index 0609fdd0b2..5eda39b4db 100644
--- a/docs/runbooks/notification-retention.md
+++ b/docs/runbooks/notification-retention.md
@@ -1,9 +1,15 @@
# Notification retention
The internal `notification_retention_handler` Lambda removes notifications older
-than 30 complete UTC days. Its EventBridge schedule is **disabled** in both dev
-and prod, and its event payload defaults to **dry-run**. Deploying the code or
-migrations alone does not purge history.
+than 30 complete UTC days. Both dev and prod stack configs **enable** its
+hourly EventBridge schedule in **deletion mode**. Deploying either
+notification-service stack activates cleanup, including the historical backlog;
+committing the config or applying migrations alone does not. These are desired
+settings, not proof of the deployed schedule's state. Merging to `main` triggers
+normal dev deployment; a production release applies the prod settings. Complete
+the rollout prerequisites before merging or releasing this activation. Missing
+stack settings still default to disabled/dry-run, and manual invocations with
+`{}` remain read-only.
## Semantics
@@ -18,7 +24,7 @@ migrations alone does not purge history.
device registrations, and email-send bookkeeping are not removed.
- Old unread notifications disappear. Items whose only feed-placement reason was
an expired notification may disappear from that feed, not from source storage.
-- This is eventual cleanup, normally within a five-minute schedule interval plus
+- This is eventual cleanup, normally within a one-hour schedule interval plus
execution time after expiry once caught up. Backlogs, failures, or locked rows
can extend that delay. There is no new read-time expiration filter.
- **Replay protection is intentionally out of scope.** Parent-row existence still
@@ -137,10 +143,13 @@ at Kafka: it does **not** measure time queued before PostgreSQL NOTIFY is forwar
or acknowledge client delivery. Existing best-effort loss/reconnect semantics
remain unchanged; no durable outbox is added by this observability change.
-Before the next deletion canary, after approved deployments:
+Before first enabling deletion in an environment, verify these prerequisites;
+code approval does not establish that the live checks have passed:
-1. Deploy the notification-service stack (extension + worker) and DSS (consumer
- telemetry). Keep both cleanup schedules disabled and dry-run enabled.
+1. For a telemetry-only deployment of the notification-service stack (extension
+ + worker) and DSS (consumer telemetry), explicitly override the checked-in
+ retention settings to disabled/dry-run before deploying. The checked-in
+ settings otherwise enable deletion on deployment.
2. Invoke explicit dry runs and verify the complete invocation and batch/read
spans in Datadog, with the correct service/env and no exporter/flush errors.
3. Confirm health logs from each running DSS instance, including healthy zero
@@ -157,8 +166,12 @@ Before the next deletion canary, after approved deployments:
## Rollout
-Production deletion needs explicit approval and a verified backup/PITR recovery
-procedure. Do not run a historical `DELETE` inside a migration.
+Deployment needs explicit approval; production deletion also needs a verified
+backup/PITR recovery procedure. The checked-in dev and prod settings select the
+final deletion stage below; they do not verify the rollout prerequisites.
+For an environment that has not completed them, explicitly override the config
+to disabled/dry-run before its first deployment and follow the staged checks.
+Do not run a historical `DELETE` inside a migration.
1. Apply the additive migrations through the normal approved database workflow.
The retention index is built concurrently in its own nontransactional
@@ -166,9 +179,10 @@ procedure. Do not run a historical `DELETE` inside a migration.
changing its contract. Verify the index is valid. If a concurrent build fails,
inspect its invalid index and repair/retry under the database deployment
procedure rather than marking the migration successful.
-2. Deploy the notification-service stack. Its deployment configuration also builds
- the new Lambda. Keep `notification_retention_enabled: false` and
- `notification_retention_dry_run: true` initially.
+2. For an initial staged rollout, set `notification_retention_enabled: false`
+ and `notification_retention_dry_run: true` before deploying the
+ notification-service stack. These override the checked-in deletion settings.
+ Its deployment configuration also builds the Lambda.
3. Inspect age distribution, recipient/receipt fanout, and query plans against
representative data. Use plain `EXPLAIN` for production deletion plans;
`EXPLAIN ANALYZE DELETE` actually deletes data.
@@ -184,8 +198,9 @@ procedure. Do not run a historical `DELETE` inside a migration.
5. After approval and reviewing dry-run logs, change
`notification_retention_dry_run` to `false` and deploy. Drain existing history
with these same bounded batches; do not substitute a bulk deletion. Initial
- capacity is at most 10,000 parents per invocation (usually less). Verify
- steady-state capacity exceeds incoming expiration volume before enabling prod.
+ capacity is at most 10,000 parents per hourly invocation (usually less), or
+ 240,000 per day. Verify steady-state capacity exceeds incoming expiration
+ volume before enabling prod.
6. Monitor application latency, database CPU/I/O, WAL and replica lag, dead tuples,
vacuum progress, NOTIFY warnings/queue pressure, and realtime consumer lag.
Pause the schedule on unacceptable impact. Do not automatically run
@@ -199,8 +214,10 @@ retention period or batch caps.
## Monitoring and stopping
-Infrastructure alarms on Lambda errors, throttles, and no invocations for
-15 minutes while the schedule is enabled. Each sweep logs the cutoff,
+Infrastructure alarms on Lambda errors, throttles, and three consecutive hourly
+periods without invocations while the schedule is enabled. The invocation alarm
+fills missing hourly samples with zero and requires all three periods to breach,
+so older invocations cannot mask a stopped schedule. Each sweep logs the cutoff,
`sampled_expired`, `oldest_expired_at`, committed parent `deleted` count,
attempted `batches`, and stopping reason. Each committed batch is also logged,
so partial progress remains visible if a later batch fails.
@@ -209,7 +226,7 @@ Track `cutoff - oldest_expired_at` as expiration backlog age. During backfill it
should trend downward; once caught up, sustained growth indicates insufficient
capacity, long-lived locks, or a failing high-fanout parent. A zero-row batch can
mean rows were locked, not necessarily that every expired row has been removed.
-Do not run a full `COUNT(*)` over the backlog every five minutes.
+Do not run a full `COUNT(*)` over the backlog on each sweep.
Disable `notification_retention_enabled` and deploy to stop future scheduled
invocations. An already-running invocation may still commit batches. Emergency
diff --git a/infra/bun.lock b/infra/bun.lock
index fc9099622b..ee962ca24a 100644
--- a/infra/bun.lock
+++ b/infra/bun.lock
@@ -190,6 +190,9 @@
"stacks/notification-service": {
"name": "notification-service-stack",
"version": "0.0.0",
+ "devDependencies": {
+ "yaml": "^2.8.1",
+ },
},
"stacks/observability": {
"name": "observability-stack",
@@ -927,6 +930,8 @@
"yallist": ["[email protected]", "", {}, "sha512-YgvUTfwqyc7UXVMrB+SImsVYSmTS8X/tSrtdNZMImM+n7+QTriRXyXim0mBrTXNeqzVF0KWGgHPeiyViFFrNDw=="],
+ "yaml": ["[email protected]", "", { "bin": { "yaml": "bin.mjs" } }, "sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw=="],
+
"yargs": ["[email protected]", "", { "dependencies": { "cliui": "^8.0.1", "escalade": "^3.1.1", "get-caller-file": "^2.0.5", "require-directory": "^2.1.1", "string-width": "^4.2.3", "y18n": "^5.0.5", "yargs-parser": "^21.1.1" } }, "sha512-GZtjxm/J/4TSxuL3FNYjCmLktBTnIw/rVmKSIyKeYAZpmJB2ig9VauCC5xsa82GNKVKDAqpOn3KVzNt0zmrU0g=="],
"yargs-parser": ["[email protected]", "", {}, "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw=="],
diff --git a/infra/stacks/notification-service/AGENTS.md b/infra/stacks/notification-service/AGENTS.md
index 0eb2136d80..a1f2b3c2f0 100644
--- a/infra/stacks/notification-service/AGENTS.md
+++ b/infra/stacks/notification-service/AGENTS.md
@@ -10,12 +10,15 @@ notification-retention Lambda. Runtime business policy lives in Rust.
From `infra/`:
- `bun test stacks/notification-service/retention.test.ts`
- `bun run check`
-- `bunx biome check stacks/notification-service`
+- Run `biome ci --changed --no-errors-on-unmatched --error-on-warnings` inside
+ the Nix shell to match CI; the Bun-installed formatter can differ.
## Gotchas
-- Keep retention disabled/dry-run until explicitly approved. Deployment does not
- authorize deletion; `docs/runbooks/notification-retention.md` owns rollout.
+- Dev and prod configs enable hourly 30-day retention in deletion mode.
+ Deploying either stack activates cleanup; deployment still needs
+ explicit approval and the checks in `docs/runbooks/notification-retention.md`.
+- Keep the disabled/dry-run defaults for missing config and empty manual events.
- The retention function needs its pinned x86_64 Datadog extension to receive
gRPC OTLP; the ECS sidecar default alone does not work in Lambda.
- Pass the existing Datadog key ARN, not its value, and scope the role to that key.
diff --git a/infra/stacks/notification-service/Pulumi.dev.yaml b/infra/stacks/notification-service/Pulumi.dev.yaml
index b0537a4f73..564c6aca5c 100644
--- a/infra/stacks/notification-service/Pulumi.dev.yaml
+++ b/infra/stacks/notification-service/Pulumi.dev.yaml
@@ -2,8 +2,8 @@ config:
aws-native:region: us-east-1
aws:region: us-east-1
notification-service:macro_db_secret_key: macro-db-dev
- notification-service:notification_retention_enabled: false
- notification-service:notification_retention_dry_run: true
+ notification-service:notification_retention_enabled: true
+ notification-service:notification_retention_dry_run: false
notification-service:jwt_secret_key: fusionauth-jwt-secret-dev
notification-service:fusionauth_client_id: fusionauth-client-id-key-dev
notification-service:fusionauth_issuer: fusionauth-dev.macro.com
diff --git a/infra/stacks/notification-service/Pulumi.prod.yaml b/infra/stacks/notification-service/Pulumi.prod.yaml
index 2c9076cd67..0938db23ea 100644
--- a/infra/stacks/notification-service/Pulumi.prod.yaml
+++ b/infra/stacks/notification-service/Pulumi.prod.yaml
@@ -2,8 +2,8 @@ config:
aws-native:region: us-east-1
aws:region: us-east-1
notification-service:macro_db_secret_key: macro-db-prod
- notification-service:notification_retention_enabled: false
- notification-service:notification_retention_dry_run: true
+ notification-service:notification_retention_enabled: true
+ notification-service:notification_retention_dry_run: false
notification-service:jwt_secret_key: fusionauth-jwt-secret-prod
notification-service:fusionauth_client_id: fusionauth-client-id-key-prod
notification-service:fusionauth_issuer: auth.macro.com
diff --git a/infra/stacks/notification-service/package.json b/infra/stacks/notification-service/package.json
index 646ebd3bb0..16ba64f9fa 100644
--- a/infra/stacks/notification-service/package.json
+++ b/infra/stacks/notification-service/package.json
@@ -3,5 +3,8 @@
"version": "0.0.0",
"private": true,
"license": "MIT",
- "main": "index.ts"
+ "main": "index.ts",
+ "devDependencies": {
+ "yaml": "^2.8.1"
+ }
}
diff --git a/infra/stacks/notification-service/retention.test.ts b/infra/stacks/notification-service/retention.test.ts
index 8b66b7c2c2..b9a1875417 100644
--- a/infra/stacks/notification-service/retention.test.ts
+++ b/infra/stacks/notification-service/retention.test.ts
@@ -1,5 +1,7 @@
+import { file } from 'bun';
import { beforeAll, expect, mock, test } from 'bun:test';
import * as pulumi from '@pulumi/pulumi';
+import { parse } from 'yaml';
const resources: pulumi.runtime.MockResourceArgs[] = [];
const secretArn =
@@ -70,8 +72,11 @@ beforeAll(async () => {
await pulumi.runtime.waitForRPCs();
});
-function resource(type: string): pulumi.runtime.MockResourceArgs {
- const value = resources.find((r) => r.type === type);
+function resource(
+ type: string,
+ registered = resources
+): pulumi.runtime.MockResourceArgs {
+ const value = registered.find((r) => r.type === type);
if (!value) throw new Error(`Missing ${type}`);
return value;
}
@@ -107,6 +112,85 @@ test('extension access is scoped to the existing Datadog key, not all secrets',
]);
});
+test.each(['dev', 'prod'])('%s uses hourly retention', async (stack) => {
+ const parsed: unknown = parse(
+ await file(new URL(`./Pulumi.${stack}.yaml`, import.meta.url)).text()
+ );
+ if (
+ !parsed ||
+ typeof parsed !== 'object' ||
+ !('config' in parsed) ||
+ !parsed.config ||
+ typeof parsed.config !== 'object'
+ ) {
+ throw new Error('Missing stack config');
+ }
+ pulumi.runtime.setAllConfig({
+ 'aws:region': 'us-east-1',
+ ...Object.fromEntries(
+ Object.entries(parsed.config)
+ .filter(([, value]) => typeof value === 'boolean')
+ .map(([key, value]) => [key, String(value)])
+ ),
+ });
+ const config = new pulumi.Config('notification-service');
+ const start = resources.length;
+ const { NotificationRetention } = await import('./retention');
+ new NotificationRetention(`retention-${stack}`, {
+ databaseUrl: pulumi.secret('test-database'),
+ enabled: config.getBoolean('notification_retention_enabled') ?? false,
+ dryRun: config.getBoolean('notification_retention_dry_run') ?? true,
+ vpc: {
+ vpcId: 'vpc-test',
+ publicSubnetIds: [],
+ privateSubnetIds: ['subnet-test'],
+ },
+ tags: { environment: stack },
+ });
+ await pulumi.runtime.waitForRPCs();
+ const registered = resources.slice(start);
+ expect(
+ resource('aws:cloudwatch/eventRule:EventRule', registered).inputs
+ ).toMatchObject({ state: 'ENABLED', scheduleExpression: 'rate(1 hour)' });
+ expect(
+ JSON.parse(
+ resource('aws:cloudwatch/eventTarget:EventTarget', registered).inputs
+ .input
+ )
+ ).toEqual({ dry_run: false });
+ expect(
+ registered.find(
+ (r) =>
+ r.type === 'aws:cloudwatch/metricAlarm:MetricAlarm' &&
+ r.name === 'notification_retention_handler-missing-invocations'
+ )?.inputs
+ ).toMatchObject({
+ metricQueries: [
+ {
+ id: 'invocations',
+ metric: {
+ namespace: 'AWS/Lambda',
+ metricName: 'Invocations',
+ period: 3600,
+ stat: 'Sum',
+ },
+ returnData: false,
+ },
+ {
+ id: 'hourly_invocations',
+ expression: 'FILL(invocations, 0)',
+ returnData: true,
+ },
+ ],
+ evaluationPeriods: 3,
+ datapointsToAlarm: 3,
+ threshold: 1,
+ comparisonOperator: 'LessThanThreshold',
+ actionsEnabled: true,
+ treatMissingData: 'breaching',
+ });
+});
+
test('telemetry does not enable cleanup or duplicate the existing log path', () => {
expect(resource('aws:cloudwatch/eventRule:EventRule').inputs.state).toBe(
'DISABLED'
diff --git a/infra/stacks/notification-service/retention.ts b/infra/stacks/notification-service/retention.ts
index 99f867597e..1705586f43 100644
--- a/infra/stacks/notification-service/retention.ts
+++ b/infra/stacks/notification-service/retention.ts
@@ -34,7 +34,7 @@ type Args = {
tags: Record<string, string>;
};
-/** Isolated maintenance worker: disabled initially, then preview before deleting. */
+/** Isolated maintenance worker with explicit scheduling and dry-run controls. */
export class NotificationRetention extends pulumi.ComponentResource {
readonly lambda: aws.lambda.Function;
@@ -116,7 +116,7 @@ export class NotificationRetention extends pulumi.ComponentResource {
const rule = new aws.cloudwatch.EventRule(
`${BASE_NAME}-schedule`,
{
- scheduleExpression: 'rate(5 minutes)',
+ scheduleExpression: 'rate(1 hour)',
state: enabled ? 'ENABLED' : 'DISABLED',
tags,
},
@@ -175,12 +175,27 @@ export class NotificationRetention extends pulumi.ComponentResource {
new aws.cloudwatch.MetricAlarm(
`${BASE_NAME}-missing-invocations`,
{
- namespace: 'AWS/Lambda',
- metricName: 'Invocations',
- dimensions: { FunctionName: this.lambda.name },
- statistic: 'Sum',
- period: 900,
- evaluationPeriods: 1,
+ metricQueries: [
+ {
+ id: 'invocations',
+ metric: {
+ namespace: 'AWS/Lambda',
+ metricName: 'Invocations',
+ dimensions: { FunctionName: this.lambda.name },
+ period: 3600,
+ stat: 'Sum',
+ },
+ returnData: false,
+ },
+ {
+ id: 'hourly_invocations',
+ // Fill silent hours so older samples cannot mask a stopped schedule.
+ expression: 'FILL(invocations, 0)',
+ returnData: true,
+ },
+ ],
+ evaluationPeriods: 3,
+ datapointsToAlarm: 3,
threshold: 1,
comparisonOperator: 'LessThanThreshold',
treatMissingData: enabled ? 'breaching' : 'notBreaching',
diff --git a/services/notification_retention_handler/AGENTS.md b/services/notification_retention_handler/AGENTS.md
index b26ce493f3..318d4da956 100644
--- a/services/notification_retention_handler/AGENTS.md
+++ b/services/notification_retention_handler/AGENTS.md
@@ -10,7 +10,7 @@ Internal scheduled notification cleanup. The owning notification domain enforces
- `src/main.rs`: one-connection pool, retained telemetry provider, post-invocation
flush. Close handler spans before flushing; never shut down warm providers.
- `src/handler.rs`: explicit dry-run default, invocation deadline and request ID.
-- `infra/stacks/notification-service/retention.ts`: disabled schedule, pinned
+- `infra/stacks/notification-service/retention.ts`: configurable schedule, pinned
Datadog extension, scoped key access and nonduplicated log forwarding.
## Checks
@@ -26,7 +26,8 @@ Internal scheduled notification cleanup. The owning notification domain enforces
of already-committed deletion. Flush time comes from reserved handler headroom.
- Unit tests cannot prove the deployed extension exported traces. Verify an
explicitly approved dry run in Datadog before production cleanup.
-- The schedule stays disabled until the rollout is explicitly approved.
+- Dev and prod stack configs enable scheduled deletion every hour;
+ deployment still needs explicit approval and the runbook's rollout checks.
## Related