Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
Notification retention
The internal notification_retention_handler Lambda removes notifications older
than 30 complete UTC days. Both dev and prod stack configs enable its
hourly EventBridge schedule in deletion mode. Deploying either
notification-service stack activates cleanup, including the historical backlog;
committing the config or applying migrations alone does not. These are desired
settings, not proof of the deployed schedule's state. Merging to main triggers
normal dev deployment; a production release applies the prod settings. Complete
the rollout prerequisites before merging or releasing this activation. Missing
stack settings still default to disabled/dry-run, and manual invocations with
{} remain read-only.
Semantics
- The cutoff is the database clock at invocation start minus 30 days, converted
to UTC for the schema's timezone-less
notification.created_atcolumn. - Delete only parents with
created_at < cutoff. Equality and newer timestamps survive. Reading, completing, reopening, or sending does not extend retention. - All recipient states (
unseen,seen,done) and soft-deleted recipients expire with the parent. The parent age wins even if recipient timestamps differ. - Foreign keys cascade to
user_notificationandnotification_message_receipt. Messages, documents, other source entities, preferences, unsubscribe records, device registrations, and email-send bookkeeping are not removed. - Old unread notifications disappear. Items whose only feed-placement reason was an expired notification may disappear from that feed, not from source storage.
- This is eventual cleanup, normally within a one-hour schedule interval plus execution time after expiry once caught up. Backlogs, failures, or locked rows can extend that delay. There is no new read-time expiration filter.
- Replay protection is intentionally out of scope. Parent-row existence still provides notification-ID deduplication. After retention deletes a parent, an old event replay can recreate it with a fresh timestamp. Do not assume forever idempotency when replaying old ingress events.
Execution and safety limits
The notification domain owns policy and batching in
crates/notification/src/domain/retention.rs. Its outbound adapter owns checked
SQL and transactions in crates/notification/src/outbound/retention.rs. The
Lambda only parses an internal event, computes remaining invocation time, and
calls the domain service. There is no HTTP route or user-callable cleanup API.
IAM controls invocation; do not grant end users permission to invoke this Lambda.
Each invocation:
- Computes one cutoff using the database clock and previews at most 100 expired
parents.
sampled_expiredis not a total backlog count. - In deletion mode, selects oldest-first using
(created_at, id), at most 100 parents per transaction. A matching B-tree index avoids a history-wide sort. - Acquires a nonblocking transaction-scoped advisory lock, then uses
FOR UPDATE SKIP LOCKED. Another cleanup transaction causes this invocation to stop. Locked parents are retried on future invocations; there is no watermark that could permanently skip them. - Deletes parents, allowing ordinary FK cascades and delete triggers, and commits each batch. No recipient payloads are returned to the worker.
- Pauses 250 ms between batches. Caps work at 100 attempted batches and a 100-second invocation budget, leaving 15 seconds of headroom before starting another batch. The Lambda timeout is 120 seconds and remaining Lambda time is also respected.
The dedicated pool has one connection and a five-second acquisition timeout. Statements time out after five seconds and lock waits after one second. Idle transactions time out after ten seconds, so a stalled/frozen worker cannot hold locks indefinitely. The handler also bounds the entire call by its remaining invocation budget, including pool/network waits. A delete statement/lock timeout rolls back that batch; the service halves the next batch size. A timeout at one parent fails the invocation and raises the error alarm. Other errors also fail, without undoing previously committed batches. A subsequent invocation resumes naturally. Lambda concurrency is one; the advisory lock also protects against other instances of this worker.
Parent limits do not bound recipient/receipt fanout. A single huge notification can still exceed the timeout and require operator investigation. Never raise limits or disable constraints/triggers merely to force progress.
Realtime effects
The existing parent-delete trigger publishes user_notification_deletes events
before recipients cascade away. The migration splits recipients into JSON chunks
with a 7000-byte recipient-array budget, leaving room below PostgreSQL's 8000-byte
NOTIFY limit. Existing consumers accept the same event shape for each chunk.
An individual malformed/oversized recipient is warned about and skipped; a
publication failure remains best-effort, as in the previous trigger.
LISTEN/NOTIFY is not durable. Disconnected clients/listeners must reconcile on
refetch/reconnect; this change does not add durable tombstones or an outbox.
Watch notification-service/Kafka/connection-gateway load during backfill. Do not
turn off delete events to speed up the purge.
Tracing and consumer health
The retention Lambda includes the pinned x86_64 Datadog Extension v101, with an
OTLP gRPC receiver on 127.0.0.1:4317. The extension retrieves the existing
Datadog key by secret ARN; its role can read only that key, not unrelated secrets.
No API key value is copied into the Lambda environment. If that secret uses a
customer-managed KMS key, verify narrowly scoped decrypt permission before deploy.
CloudWatch → Firehose remains the log path; extension log shipping is disabled to
avoid duplicates. Runtime identity is service:notification-retention, with
DD_ENV set to the stack.
The handler span is notification.retention.invoke, searchable by
faas.invocation_id. After that span closes, the worker flushes completed spans
with a five-second caller deadline before Lambda can freeze the process. It does
not shut down the provider between warm invocations. Flush failure logs an error
but preserves the cleanup result: committed deletion must not be retried just
because telemetry failed. The existing ten-second invocation headroom covers
this flush. The extension must be checked live after an approved deployment;
unit tests or a green Lambda result do not prove Datadog received its traces.
DSS notification consumers intentionally assign all partitions from latest,
without committing offsets or joining durable consumer groups. Consequently MSK
MaxOffsetLag is absent for macro.notifications; don't change delivery semantics
just to manufacture that metric.
Each consumer now reports telemetry_event:notification_consumer_health every
ten seconds through normal service logs, with an opaque consumer_instance:
max_offset_backlog/total_offset_backlog: broker high watermark minus the next application-delivery position. Before first delivery, use next fetch position plus prefetched records. Offset gaps mean this is an approximation, not an exact count of outstanding business events.prefetched_messages: records buffered locally, not just records still on the broker. This prevents a full fetch queue from looking caught up.known_partitions,unknown_partitions,disconnected_partitions, andconsumer_health: distinguish valid measurements from missing offsets, disconnected brokers, and startup.- A separate reporter detects callbacks older than 30 seconds (including old snapshots delivered late), even when the consuming task is stalled. Stale or partial backlog measurements are omitted, never reported as zero. Dropping the consumer stops its reporter.
The manual infra/stacks/monitoring stack adds nine log monitors and a rollout
dashboard. Alerts isolate consumer instances; a healthy process cannot mask a
lagging one. Retired-instance alerts resolve on missing data, and a separate
per-environment heartbeat alarm catches loss of all consumer telemetry. Starting
thresholds are warning 100 / critical 1000 for the five-minute average of maximum
partition backlog; tune against real traffic before enabling production cleanup.
The notification.delete.publish span covers forwarding a database deletion,
with notification ID, recipient count, and publication outcome, but no recipient
IDs or payload. A log monitor catches failed forwarding. Consumer backlog begins
at Kafka: it does not measure time queued before PostgreSQL NOTIFY is forwarded,
or acknowledge client delivery. Existing best-effort loss/reconnect semantics
remain unchanged; no durable outbox is added by this observability change.
Before first enabling deletion in an environment, verify these prerequisites; code approval does not establish that the live checks have passed:
- For a telemetry-only deployment of the notification-service stack (extension
- worker) and DSS (consumer telemetry), explicitly override the checked-in retention settings to disabled/dry-run before deploying. The checked-in settings otherwise enable deletion on deployment.
- Invoke explicit dry runs and verify the complete invocation and batch/read spans in Datadog, with the correct service/env and no exporter/flush errors.
- Confirm health logs from each running DSS instance, including healthy zero backlog when idle; investigate missing/unknown/stale samples rather than accepting an empty chart as healthy.
- Validate and deploy the monitoring stack separately. Verify the incoming
attributes are parsed as numeric measures (
max_offset_backlog,deleted) and grouping attributes (consumer_instance,consumer_health) in Datadog; configure missing measures/facets before relying on aggregation charts. The authenticated preview must show nine new monitors and one dashboard, without modifying adopted resources. - Only then approve a bounded dev deletion canary and inspect DB/API health, publisher failures, and consumer backlog draining before considering prod.
Rollout
Deployment needs explicit approval; production deletion also needs a verified
backup/PITR recovery procedure. The checked-in dev and prod settings select the
final deletion stage below; they do not verify the rollout prerequisites.
For an environment that has not completed them, explicitly override the config
to disabled/dry-run before its first deployment and follow the staged checks.
Do not run a historical DELETE inside a migration.
Apply the additive migrations through the normal approved database workflow. The retention index is built concurrently in its own nontransactional migration; the other migration replaces the delete-event function without changing its contract. Verify the index is valid. If a concurrent build fails, inspect its invalid index and repair/retry under the database deployment procedure rather than marking the migration successful.
For an initial staged rollout, set
notification_retention_enabled: falseandnotification_retention_dry_run: truebefore deploying the notification-service stack. These override the checked-in deletion settings. Its deployment configuration also builds the Lambda.Inspect age distribution, recipient/receipt fanout, and query plans against representative data. Use plain
EXPLAINfor production deletion plans;EXPLAIN ANALYZE DELETEactually deletes data.Enable the schedule in dry-run mode first:
\cd infra/stacks/notification-service pulumi config set notification_retention_enabled true --stack <dev-or-prod> pulumi config set notification_retention_dry_run true --stack <dev-or-prod> # Review/apply through the normal deployment workflow.After approval and reviewing dry-run logs, change
notification_retention_dry_runtofalseand deploy. Drain existing history with these same bounded batches; do not substitute a bulk deletion. Initial capacity is at most 10,000 parents per hourly invocation (usually less), or 240,000 per day. Verify steady-state capacity exceeds incoming expiration volume before enabling prod.Monitor application latency, database CPU/I/O, WAL and replica lag, dead tuples, vacuum progress, NOTIFY warnings/queue pressure, and realtime consumer lag. Pause the schedule on unacceptable impact. Do not automatically run
VACUUM FULL; ordinary vacuum reclaims space for reuse, not necessarily for returning to the filesystem.
A manual invocation with {} or {"dry_run": true} is read-only.
{"dry_run": false} explicitly opts into deletion. Unknown fields, alternate
spellings, and nonboolean settings are rejected. Invocation cannot override the
retention period or batch caps.
Monitoring and stopping
Infrastructure alarms on Lambda errors, throttles, and three consecutive hourly
periods without invocations while the schedule is enabled. The invocation alarm
fills missing hourly samples with zero and requires all three periods to breach,
so older invocations cannot mask a stopped schedule. Each sweep logs the cutoff,
sampled_expired, oldest_expired_at, committed parent deleted count,
attempted batches, and stopping reason. Each committed batch is also logged,
so partial progress remains visible if a later batch fails.
Track cutoff - oldest_expired_at as expiration backlog age. During backfill it
should trend downward; once caught up, sustained growth indicates insufficient
capacity, long-lived locks, or a failing high-fanout parent. A zero-row batch can
mean rows were locked, not necessarily that every expired row has been removed.
Do not run a full COUNT(*) over the backlog on each sweep.
Disable notification_retention_enabled and deploy to stop future scheduled
invocations. An already-running invocation may still commit batches. Emergency
operational controls may also stop future invokes; they do not undo commits.
Disabling the worker or rolling back migrations cannot restore deleted data.
Recovery requires the approved backup/PITR process.
Retention bounds age, not cardinality. Busy users can still have many notifications within 30 days; pagination of currently unbounded consumers remains separate work.
Local verification
Run from the repository root with local PostgreSQL and SQLX_OFFLINE unset:
cargo test -p kafka_util -p macro_entrypoint -p notification -p notification_retention_handler -- --test-threads=8
# Only if notification SQL or schema changed:
nix develop --command just prepare_db
just check
just clippy
\cd infra
bun test stacks/notification-service/retention.test.ts
bun test stacks/monitoring/monitors/notification-retention.test.ts
bun run check
Tests cover cutoff boundaries, parent-vs-recipient age, all states, cascades, read-only previews, deterministic bounded ordering, retry behavior, skipped locks, transaction advisory locking, timeout rollback, non-UTC sessions, byte-bounded multi-recipient delete events, invocation caps, and fail-closed event parsing. Observability tests also cover warm-invocation trace flushing and exporter failures, a real librdkafka mock broker without offset commits, stale/partial consumer statistics, broker disconnects, reporter cancellation, and the Pulumi collector/secret/monitor configuration. Hosted Datadog query validation and trace delivery still require an authenticated preview and approved deployment.