Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

chore(infra): fix WAF adoption and structured Datadog trace parsing (#7152)

Wolf Mermelstein86d0f93a0d37
diff --git a/infra/stacks/cloud-storage-service/WAF_RUNBOOK.md b/infra/stacks/cloud-storage-service/WAF_RUNBOOK.md
index 0b153aba14..62c2737dea 100644
--- a/infra/stacks/cloud-storage-service/WAF_RUNBOOK.md
+++ b/infra/stacks/cloud-storage-service/WAF_RUNBOOK.md
@@ -1,6 +1,6 @@
# Cloud Storage WAF Adoption
-The production WAF component adopts the existing Web ACL, CloudWatch log group, and WAF logging configuration through Pulumi resource `import` options. These resources are also protected. The existing Datadog Forwarder and `ip_safety` rule group are referenced by ARN rather than duplicated. ALB associations are outside this component.
+The production WAF component manages the existing Web ACL, CloudWatch log group, and WAF logging configuration. They were adopted, with their parent component, in [production update #336](https://app.pulumi.com/macro-inc/cloud-storage-service/prod/updates/336) on 2026-09-28 and are protected from deletion. The existing Datadog Forwarder and `ip_safety` rule group are referenced by ARN rather than duplicated. ALB associations are outside this component.
## Current Coverage
@@ -16,21 +16,21 @@ Browser telemetry must be enabled for the affected user (`enable-browser-otel` o
## Resource Configuration
-`Pulumi.prod.yaml` supplies `waf_account_id`, `waf_web_acl_name`, `waf_web_acl_id`, `waf_ip_safety_rule_group_arn`, `waf_log_group_name`, and `waf_datadog_forwarder_arn` in the `cloud-storage-service` namespace. The region comes from the existing `aws:region` setting. The component derives the Web ACL ARN from that configuration and uses the configured ACL name in the Datadog pipeline filter and its parser sample.
+`Pulumi.prod.yaml` supplies `waf_account_id`, `waf_web_acl_name`, `waf_web_acl_id`, `waf_ip_safety_rule_group_arn`, `waf_log_group_name`, and `waf_datadog_forwarder_arn` in the `cloud-storage-service` namespace. The region comes from the existing `aws:region` setting. The component derives the Web ACL ARN from that configuration and uses the derived ARN in the Datadog pipeline filter.
WAF configuration is read only when the production component is instantiated; other stacks do not need these keys.
-## First Deployment
+## Deployment After Adoption
1. Run `pulumi preview --stack prod` from `infra/stacks/cloud-storage-service` with credentials for the configured `waf_account_id` and Datadog US5.
-2. Confirm the Web ACL, log group, and logging configuration are imports followed only by in-place updates. Do not proceed if Pulumi proposes a replacement or deletion for any of them.
+2. Confirm the Web ACL, log group, and logging configuration have only in-place updates or no changes. Do not proceed if Pulumi proposes a replacement or deletion for any of them.
3. Confirm the Web ACL is still attached to the intended ALBs using `aws wafv2 list-resources-for-web-acl`. This stack does not create or change associations.
-4. Confirm the existing Lambda policy has no statement named `AllowCloudWatchLogsAwsWafProd`. If it does, import that permission before deployment or rename the statement only after determining ownership.
-5. Confirm the log group has capacity for another subscription filter and no existing filter already sends these events to the same Forwarder. CloudWatch subscriptions process only events written after the filter is created; this configuration does not replay retained logs.
-6. Review the intended in-place policy changes explicitly: request bodies, sensitive headers, and query strings receive data protection; URI paths, query strings, and sensitive headers are redacted in the logging configuration; sampled requests are disabled; both `CategoryHttpLibrary` and `SignalNonBrowserUserAgent` become non-terminating counts; and `SQLi_BODY` becomes a label that is blocked everywhere except the exact channel-message POST route. Non-body SQLi rules remain active on that route.
+4. Confirm the Forwarder permission `AllowCloudWatchLogsAwsWafProd` and log subscription remain managed by this component. Do not create duplicate permissions or subscriptions.
+5. Confirm the log group has exactly one subscription sending these events to the configured Forwarder. CloudWatch subscriptions process only events written after the filter is created; this configuration does not replay retained logs.
+6. Review the intended in-place policy changes explicitly: request bodies, sensitive headers, and query strings receive data protection; URI paths, query strings, and sensitive headers are configured for logging redaction; sampled requests are disabled; both `CategoryHttpLibrary` and `SignalNonBrowserUserAgent` become non-terminating counts; and `SQLi_BODY` becomes a label that is blocked everywhere except the exact channel-message POST route. Non-body SQLi rules remain active on that route.
7. Confirm CloudWatch log retention is unchanged. The component reads and preserves the existing log group retention instead of imposing a new retention period.
-Do not run a separate `pulumi import`: the configuration-derived import IDs in `waf-observability.ts` perform adoption on the first update and remain harmless after the resources are in stack state.
+The one-time `import` options have been removed now that the resources are in production state. The Web ACL importer accepts `id/name/scope`, but stores only the UUID as the resource ID; retaining that composite import option after CLI adoption makes Pulumi propose a replacement. Existing resources must retain their current logical names and component parent.
## Datadog Pipeline Order
@@ -42,4 +42,10 @@ Before the first deployment, confirm no other Pulumi stack manages the singleton
2. Verify all pre-existing pipeline IDs remain in their original relative order.
3. Coordinate pipeline-order deployments because the Datadog API exposes one organization-global order; concurrent external edits can race an update.
-The pipeline filter includes the production Web ACL name so it does not remap unrelated WAF logs. It parses WAF JSON, extracts W3C `traceparent` IDs as lowercase 32-character trace IDs and 16-character span IDs, and uses the trace, span, and service remappers supported by the installed Datadog provider.
+The Forwarder delivers WAF events as structured JSON attributes, without a raw `message`. The pipeline matches the exact production `webaclId`, selects the `traceparent` value from `httpRequest.headers` using an array processor, and parses W3C `traceparent` IDs as lowercase 32-character trace IDs and 16-character span IDs, and uses the trace, span, and service remappers supported by the installed Datadog provider.
+
+## Production Verification
+
+Production updates [#337](https://app.pulumi.com/macro-inc/cloud-storage-service/prod/updates/337) and [#338](https://app.pulumi.com/macro-inc/cloud-storage-service/prod/updates/338) enabled forwarding and corrected the pipeline for structured Forwarder events. A controlled SQLi query against the WAF-protected static-file ALB returned 403, reached Datadog, and was searchable by its W3C `trace_id` and `span_id`. Authorization and query data were redacted. The subsequent scoped preview reported 141 unchanged resources.
+
+[AWS logging redaction depends on the rule's match component](https://docs.aws.amazon.com/waf/latest/developerguide/logging-management.html): configuring `UriPath` does not guarantee that every event hides its path. The verification event retained `/`. Body, sensitive-header, and query-string data protection are configured separately at Web ACL level. This test verifies trace metadata on the log; it does not verify linkage to an exported browser span.
diff --git a/infra/stacks/cloud-storage-service/waf-observability.ts b/infra/stacks/cloud-storage-service/waf-observability.ts
index d4d79d9011..accca3acc5 100644
--- a/infra/stacks/cloud-storage-service/waf-observability.ts
+++ b/infra/stacks/cloud-storage-service/waf-observability.ts
@@ -85,11 +85,6 @@ export class WafObservability extends pulumi.ComponentResource {
const datadogForwarderArn = config.require('waf_datadog_forwarder_arn');
const childOptions = { parent: this, protect: true };
- const adoptedOptions = {
- parent: this,
- protect: true,
- };
-
const messagePostPaths = new aws.wafv2.RegexPatternSet(
`${name}-message-post-paths`,
{
@@ -144,6 +139,7 @@ export class WafObservability extends pulumi.ComponentResource {
{
awsManagedRulesBotControlRuleSet: {
inspectionLevel: 'COMMON',
+ enableMachineLearning: false,
},
},
],
@@ -209,10 +205,7 @@ export class WafObservability extends pulumi.ComponentResource {
],
visibilityConfig: visibilityConfig(webAclName),
},
- {
- ...adoptedOptions,
- import: `${webAclId}/${webAclName}/REGIONAL`,
- }
+ childOptions
);
const existingLogGroup = aws.cloudwatch.getLogGroupOutput(
@@ -225,10 +218,7 @@ export class WafObservability extends pulumi.ComponentResource {
name: logGroupName,
retentionInDays: existingLogGroup.retentionInDays,
},
- {
- ...adoptedOptions,
- import: logGroupName,
- }
+ childOptions
);
const loggingConfiguration = new aws.wafv2.WebAclLoggingConfiguration(
@@ -255,10 +245,7 @@ export class WafObservability extends pulumi.ComponentResource {
{ uriPath: {} },
],
},
- {
- ...adoptedOptions,
- import: webAclArn,
- }
+ childOptions
);
const forwarderPermission = new aws.lambda.Permission(
@@ -297,31 +284,23 @@ export class WafObservability extends pulumi.ComponentResource {
isEnabled: true,
filters: [
{
- query: `"${webAclName}"`,
+ query: `@webaclId:"${webAclArn}"`,
},
],
tags: ['env:prod', 'service:cloud-storage-service', 'source:waf'],
processors: [
{
- grokParser: {
- name: 'Parse AWS WAF JSON',
+ arrayProcessor: {
+ name: 'Extract W3C traceparent header',
isEnabled: true,
- source: 'message',
- samples: [
- JSON.stringify({
- timestamp: 0,
- webaclId: webAclArn,
- action: 'BLOCK',
- httpRequest: {
- clientIp: '192.0.2.1',
- uri: '/',
- httpMethod: 'GET',
- },
- }),
- ],
- grok: {
- matchRules: 'waf_json %{data::json}',
- supportRules: '',
+ operation: {
+ select: {
+ source: 'httpRequest.headers',
+ filter:
+ 'name:traceparent OR name:Traceparent OR name:TraceParent',
+ valueToExtract: 'value',
+ target: 'waf.traceparent',
+ },
},
},
},
@@ -329,15 +308,14 @@ export class WafObservability extends pulumi.ComponentResource {
grokParser: {
name: 'Parse W3C traceparent',
isEnabled: true,
- source: 'message',
+ source: 'waf.traceparent',
samples: [
- '{"httpRequest":{"headers":[{"name":"traceparent","value":"00-4BF92F3577B34DA6A3CE929D0E0E4736-00F067AA0BA902B7-01"}]}}',
+ '00-4BF92F3577B34DA6A3CE929D0E0E4736-00F067AA0BA902B7-01',
],
grok: {
matchRules:
- 'traceparent %{traceparentPrefix}%{regex("[0-9A-Fa-f]{32}"):waf.trace_id:lowercase}-%{regex("[0-9A-Fa-f]{16}"):waf.span_id:lowercase}-%{regex("[0-9A-Fa-f]{2}")}%{data}',
- supportRules:
- 'traceparentPrefix .*"name"\\s*:\\s*"[Tt][Rr][Aa][Cc][Ee][Pp][Aa][Rr][Ee][Nn][Tt]"\\s*,\\s*"value"\\s*:\\s*"00-',
+ 'traceparent 00-%{regex("[0-9A-Fa-f]{32}"):waf.trace_id:lowercase}-%{regex("[0-9A-Fa-f]{16}"):waf.span_id:lowercase}-%{regex("[0-9A-Fa-f]{2}")}',
+ supportRules: '',
},
},
},