Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

feat(observability): correlate WAF logs with traces (#6109)

Wolf Mermelsteine1f8d0fd01b8
diff --git a/infra/stacks/cloud-storage-service/Pulumi.prod.yaml b/infra/stacks/cloud-storage-service/Pulumi.prod.yaml
index 73e2f6c0f2..e4960521a5 100644
--- a/infra/stacks/cloud-storage-service/Pulumi.prod.yaml
+++ b/infra/stacks/cloud-storage-service/Pulumi.prod.yaml
@@ -1,6 +1,13 @@
config:
aws-native:region: us-east-1
aws:region: us-east-1
+ datadog:apiUrl: https://api.us5.datadoghq.com/
+ cloud-storage-service:waf_account_id: "569036502058"
+ cloud-storage-service:waf_web_acl_name: macro-cloud-storage-prod
+ cloud-storage-service:waf_web_acl_id: 0f1059a5-1ee4-4b57-a429-88df830d5091
+ cloud-storage-service:waf_ip_safety_rule_group_arn: arn:aws:wafv2:us-east-1:569036502058:regional/rulegroup/ip_safety/0a1bfeec-4d6c-4afe-bf35-b93e06c65f9b
+ cloud-storage-service:waf_log_group_name: aws-waf-logs-macro-cloud-storage-prod
+ cloud-storage-service:waf_datadog_forwarder_arn: arn:aws:lambda:us-east-1:569036502058:function:DatadogIntegration-ForwarderStack-BS3QDP-Forwarder-O0hyWQ9Yq4uQ
cloud-storage-service:macro_db_secret_key: macro-db-prod
cloud-storage-service:macro_db_proxy_secret_key: macrodb-rds-proxy-prod
cloud-storage-service:macro_db_readonly_secret_key: macro-db-readonly-prod
diff --git a/infra/stacks/cloud-storage-service/WAF_RUNBOOK.md b/infra/stacks/cloud-storage-service/WAF_RUNBOOK.md
new file mode 100644
index 0000000000..0b153aba14
--- /dev/null
+++ b/infra/stacks/cloud-storage-service/WAF_RUNBOOK.md
@@ -0,0 +1,45 @@
+# Cloud Storage WAF Adoption
+
+The production WAF component adopts the existing Web ACL, CloudWatch log group, and WAF logging configuration through Pulumi resource `import` options. These resources are also protected. The existing Datadog Forwarder and `ip_safety` rule group are referenced by ARN rather than duplicated. ALB associations are outside this component.
+
+## Current Coverage
+
+A read-only AWS check on 2026-09-28 found no WAF association on the production shared gateway ALB. The configured Web ACL remains associated with the document-processing, document-cognition, and static-file-service ALBs. The old cloud-storage-service ALB association no longer exists and must not be imported.
+
+This component observes requests evaluated by the existing Web ACL; it does not add WAF coverage to the shared gateway. Recheck attachments before deployment. Attaching a WAF to the shared gateway affects multiple services and requires a separate policy review and rollout.
+
+## Correlation
+
+The existing browser `safeFetch` instrumentation already sends W3C `traceparent` headers to Macro services. This component extracts those trace and span IDs from WAF events so a block or challenge can be linked to its browser request without a backend span. No fetch, retry, request-ID, or CORS behavior changes are needed.
+
+Browser telemetry must be enabled for the affected user (`enable-browser-otel` or its build override), and the browser trace must reach Datadog. Deploy this production-only component and verify a new WAF event links to the corresponding trace; merging to main deploys dev and does not enable the production WAF pipeline by itself.
+
+## Resource Configuration
+
+`Pulumi.prod.yaml` supplies `waf_account_id`, `waf_web_acl_name`, `waf_web_acl_id`, `waf_ip_safety_rule_group_arn`, `waf_log_group_name`, and `waf_datadog_forwarder_arn` in the `cloud-storage-service` namespace. The region comes from the existing `aws:region` setting. The component derives the Web ACL ARN from that configuration and uses the configured ACL name in the Datadog pipeline filter and its parser sample.
+
+WAF configuration is read only when the production component is instantiated; other stacks do not need these keys.
+
+## First Deployment
+
+1. Run `pulumi preview --stack prod` from `infra/stacks/cloud-storage-service` with credentials for the configured `waf_account_id` and Datadog US5.
+2. Confirm the Web ACL, log group, and logging configuration are imports followed only by in-place updates. Do not proceed if Pulumi proposes a replacement or deletion for any of them.
+3. Confirm the Web ACL is still attached to the intended ALBs using `aws wafv2 list-resources-for-web-acl`. This stack does not create or change associations.
+4. Confirm the existing Lambda policy has no statement named `AllowCloudWatchLogsAwsWafProd`. If it does, import that permission before deployment or rename the statement only after determining ownership.
+5. Confirm the log group has capacity for another subscription filter and no existing filter already sends these events to the same Forwarder. CloudWatch subscriptions process only events written after the filter is created; this configuration does not replay retained logs.
+6. Review the intended in-place policy changes explicitly: request bodies, sensitive headers, and query strings receive data protection; URI paths, query strings, and sensitive headers are redacted in the logging configuration; sampled requests are disabled; both `CategoryHttpLibrary` and `SignalNonBrowserUserAgent` become non-terminating counts; and `SQLi_BODY` becomes a label that is blocked everywhere except the exact channel-message POST route. Non-body SQLi rules remain active on that route.
+7. Confirm CloudWatch log retention is unchanged. The component reads and preserves the existing log group retention instead of imposing a new retention period.
+
+Do not run a separate `pulumi import`: the configuration-derived import IDs in `waf-observability.ts` perform adoption on the first update and remain harmless after the resources are in stack state.
+
+## Datadog Pipeline Order
+
+`LogsPipelineOrder` represents the complete organization-global pipeline order. The stack reads the current order during deployment, removes any existing occurrence of this pipeline, and prepends it. This preserves unrelated pipeline IDs while ensuring its trace, span, and service remappers run before other matching pipelines.
+
+Before the first deployment, confirm no other Pulumi stack manages the singleton Datadog pipeline-order resource. Then:
+
+1. Verify that Datadog placed `Macro Cloud Storage AWS WAF` first in the pipeline order and that matching sample logs are processed.
+2. Verify all pre-existing pipeline IDs remain in their original relative order.
+3. Coordinate pipeline-order deployments because the Datadog API exposes one organization-global order; concurrent external edits can race an update.
+
+The pipeline filter includes the production Web ACL name so it does not remap unrelated WAF logs. It parses WAF JSON, extracts W3C `traceparent` IDs as lowercase 32-character trace IDs and 16-character span IDs, and uses the trace, span, and service remappers supported by the installed Datadog provider.
diff --git a/infra/stacks/cloud-storage-service/index.ts b/infra/stacks/cloud-storage-service/index.ts
index 62d7a58986..a36213c7be 100644
--- a/infra/stacks/cloud-storage-service/index.ts
+++ b/infra/stacks/cloud-storage-service/index.ts
@@ -26,6 +26,7 @@ import {
} from './document-upload-finalizer-lambda';
import { CalendarReminderDispatchQueue } from './calendar-reminder-dispatch-queue';
import { ReminderDispatchQueue } from './reminder-dispatch-queue';
+import { WafObservability } from './waf-observability';
const tags = {
environment: stack,
@@ -328,6 +329,10 @@ const cloudStorageService = new CloudStorageService(
}
);
+if (stack === 'prod') {
+ new WafObservability('cloud-storage-service-prod', { protect: true });
+}
+
export const cloudStorageServiceRoleArn = cloudStorageService.role.arn;
export const cloudStorageServiceSgId = cloudStorageService.serviceSg.id;
export const cloudStorageServiceUrl = DOCUMENT_STORAGE_GATEWAY_URL;
diff --git a/infra/stacks/cloud-storage-service/waf-observability.ts b/infra/stacks/cloud-storage-service/waf-observability.ts
new file mode 100644
index 0000000000..d4d79d9011
--- /dev/null
+++ b/infra/stacks/cloud-storage-service/waf-observability.ts
@@ -0,0 +1,552 @@
+import * as aws from '@pulumi/aws';
+import * as datadog from '@pulumi/datadog';
+import * as pulumi from '@pulumi/pulumi';
+
+const SENSITIVE_HEADERS = [
+ 'authorization',
+ 'cookie',
+ 'x-api-key',
+ 'x-cal-signature-256',
+ 'x-document-storage-service-auth-key',
+ 'x-document-storage-service-session-id',
+ 'x-document-storage-service-user-id',
+ 'x-email-link-id',
+ 'x-hub-signature-256',
+ 'x-internal-auth-key',
+ 'x-internal-fusionauth-user-id',
+ 'x-internal-macro-user-id',
+ 'x-internal-macro-organization-id',
+ 'x-macro-bot-token',
+ 'x-macro-channel-bot-token',
+ 'x-macro-bot-for-fusionauth-user-id',
+ 'x-macro-bot-for-macro-user-id',
+ 'x-macro-bot-for-organization-id',
+ 'x-macro-internal-call',
+ 'x-macro-signature',
+ 'x-macro-user-api-key',
+ 'x-permissions-token',
+] as const;
+
+const visibilityConfig = (metricName: string) => ({
+ cloudwatchMetricsEnabled: true,
+ metricName,
+ sampledRequestsEnabled: false,
+});
+
+const managedRule = (
+ name: string,
+ priority: number,
+ managedRuleGroupStatement: aws.types.input.wafv2.WebAclRuleStatementManagedRuleGroupStatement
+): aws.types.input.wafv2.WebAclRule => ({
+ name,
+ priority,
+ overrideAction: { none: {} },
+ statement: { managedRuleGroupStatement },
+ visibilityConfig: visibilityConfig(name),
+});
+
+const messagePostStatement = (
+ regexPatternSetArn: pulumi.Input<string>
+): aws.types.input.wafv2.WebAclRuleStatement => ({
+ andStatement: {
+ statements: [
+ {
+ byteMatchStatement: {
+ fieldToMatch: { method: {} },
+ positionalConstraint: 'EXACTLY',
+ searchString: 'POST',
+ textTransformations: [{ priority: 0, type: 'NONE' }],
+ },
+ },
+ {
+ regexPatternSetReferenceStatement: {
+ arn: regexPatternSetArn,
+ fieldToMatch: { uriPath: {} },
+ textTransformations: [{ priority: 0, type: 'NONE' }],
+ },
+ },
+ ],
+ },
+});
+
+export class WafObservability extends pulumi.ComponentResource {
+ constructor(name: string, opts?: pulumi.ComponentResourceOptions) {
+ super('macro:cloud-storage:WafObservability', name, {}, opts);
+
+ // Read WAF config only when instantiated; non-production stacks omit it.
+ const config = new pulumi.Config();
+ const accountId = config.require('waf_account_id');
+ const region = new pulumi.Config('aws').require('region');
+ const webAclName = config.require('waf_web_acl_name');
+ const webAclId = config.require('waf_web_acl_id');
+ const webAclArn = `arn:aws:wafv2:${region}:${accountId}:regional/webacl/${webAclName}/${webAclId}`;
+ const ipSafetyRuleGroupArn = config.require('waf_ip_safety_rule_group_arn');
+ const logGroupName = config.require('waf_log_group_name');
+ const datadogForwarderArn = config.require('waf_datadog_forwarder_arn');
+
+ const childOptions = { parent: this, protect: true };
+ const adoptedOptions = {
+ parent: this,
+ protect: true,
+ };
+
+ const messagePostPaths = new aws.wafv2.RegexPatternSet(
+ `${name}-message-post-paths`,
+ {
+ name: 'cloud-storage-message-post-prod',
+ description: 'Exact channel message creation URI paths',
+ scope: 'REGIONAL',
+ regularExpressions: [
+ {
+ regexString:
+ '^/channels/[0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[89AaBb][0-9A-Fa-f]{3}-[0-9A-Fa-f]{12}/message$',
+ },
+ ],
+ },
+ childOptions
+ );
+
+ const webAcl = new aws.wafv2.WebAcl(
+ `${name}-web-acl`,
+ {
+ name: webAclName,
+ description: 'Cloud Storage WAF for production resources',
+ scope: 'REGIONAL',
+ defaultAction: { allow: {} },
+ dataProtectionConfig: {
+ dataProtections: [
+ {
+ action: 'SUBSTITUTION',
+ field: { fieldType: 'BODY' },
+ },
+ {
+ action: 'SUBSTITUTION',
+ field: {
+ fieldType: 'SINGLE_HEADER',
+ fieldKeys: [...SENSITIVE_HEADERS],
+ },
+ },
+ {
+ action: 'SUBSTITUTION',
+ field: { fieldType: 'QUERY_STRING' },
+ },
+ ],
+ },
+ rules: [
+ managedRule('AWS-AWSManagedRulesAmazonIpReputationList', 0, {
+ name: 'AWSManagedRulesAmazonIpReputationList',
+ vendorName: 'AWS',
+ }),
+ managedRule('AWS-AWSManagedRulesBotControlRuleSet', 1, {
+ name: 'AWSManagedRulesBotControlRuleSet',
+ vendorName: 'AWS',
+ managedRuleGroupConfigs: [
+ {
+ awsManagedRulesBotControlRuleSet: {
+ inspectionLevel: 'COMMON',
+ },
+ },
+ ],
+ ruleActionOverrides: [
+ {
+ name: 'SignalNonBrowserUserAgent',
+ actionToUse: { count: {} },
+ },
+ {
+ name: 'CategoryHttpLibrary',
+ actionToUse: { count: {} },
+ },
+ ],
+ }),
+ managedRule('AWS-AWSManagedRulesKnownBadInputsRuleSet', 2, {
+ name: 'AWSManagedRulesKnownBadInputsRuleSet',
+ vendorName: 'AWS',
+ }),
+ managedRule('AWS-AWSManagedRulesSQLiRuleSet', 3, {
+ name: 'AWSManagedRulesSQLiRuleSet',
+ vendorName: 'AWS',
+ ruleActionOverrides: [
+ {
+ name: 'SQLi_BODY',
+ actionToUse: { count: {} },
+ },
+ ],
+ }),
+ {
+ name: 'BlockSQLiBody',
+ priority: 4,
+ action: { block: {} },
+ statement: {
+ andStatement: {
+ statements: [
+ {
+ labelMatchStatement: {
+ key: 'awswaf:managed:aws:sql-database:SQLi_Body',
+ scope: 'LABEL',
+ },
+ },
+ {
+ notStatement: {
+ statement: messagePostStatement(messagePostPaths.arn),
+ },
+ },
+ ],
+ },
+ },
+ visibilityConfig: visibilityConfig('BlockSQLiBody'),
+ },
+ {
+ name: 'ip_safety',
+ priority: 5,
+ overrideAction: { none: {} },
+ statement: {
+ ruleGroupReferenceStatement: {
+ arn: ipSafetyRuleGroupArn,
+ },
+ },
+ visibilityConfig: visibilityConfig('ip_safety'),
+ },
+ ],
+ visibilityConfig: visibilityConfig(webAclName),
+ },
+ {
+ ...adoptedOptions,
+ import: `${webAclId}/${webAclName}/REGIONAL`,
+ }
+ );
+
+ const existingLogGroup = aws.cloudwatch.getLogGroupOutput(
+ { name: logGroupName },
+ { parent: this }
+ );
+ const logGroup = new aws.cloudwatch.LogGroup(
+ `${name}-log-group`,
+ {
+ name: logGroupName,
+ retentionInDays: existingLogGroup.retentionInDays,
+ },
+ {
+ ...adoptedOptions,
+ import: logGroupName,
+ }
+ );
+
+ const loggingConfiguration = new aws.wafv2.WebAclLoggingConfiguration(
+ `${name}-logging`,
+ {
+ resourceArn: webAcl.arn,
+ logDestinationConfigs: [logGroup.arn],
+ loggingFilter: {
+ defaultBehavior: 'DROP',
+ filters: [
+ {
+ behavior: 'KEEP',
+ requirement: 'MEETS_ANY',
+ conditions: [
+ { actionCondition: { action: 'BLOCK' } },
+ { actionCondition: { action: 'CHALLENGE' } },
+ ],
+ },
+ ],
+ },
+ redactedFields: [
+ ...SENSITIVE_HEADERS.map((name) => ({ singleHeader: { name } })),
+ { queryString: {} },
+ { uriPath: {} },
+ ],
+ },
+ {
+ ...adoptedOptions,
+ import: webAclArn,
+ }
+ );
+
+ const forwarderPermission = new aws.lambda.Permission(
+ `${name}-forwarder-permission`,
+ {
+ statementId: 'AllowCloudWatchLogsAwsWafProd',
+ action: 'lambda:InvokeFunction',
+ function: datadogForwarderArn,
+ principal: `logs.${region}.amazonaws.com`,
+ sourceAccount: accountId,
+ sourceArn: pulumi.interpolate`${logGroup.arn}:*`,
+ },
+ childOptions
+ );
+
+ new aws.cloudwatch.LogSubscriptionFilter(
+ `${name}-forwarder-subscription`,
+ {
+ name: 'datadog-forwarder',
+ destinationArn: datadogForwarderArn,
+ filterPattern: '{ ($.action = "BLOCK") || ($.action = "CHALLENGE") }',
+ logGroup: logGroup.name,
+ },
+ {
+ ...childOptions,
+ dependsOn: [forwarderPermission, loggingConfiguration],
+ }
+ );
+
+ const datadogPipeline = new datadog.LogsCustomPipeline(
+ `${name}-datadog-pipeline`,
+ {
+ name: 'Macro Cloud Storage AWS WAF',
+ description:
+ 'Parses and normalizes blocked or challenged AWS WAF requests for cloud-storage-service.',
+ isEnabled: true,
+ filters: [
+ {
+ query: `"${webAclName}"`,
+ },
+ ],
+ tags: ['env:prod', 'service:cloud-storage-service', 'source:waf'],
+ processors: [
+ {
+ grokParser: {
+ name: 'Parse AWS WAF JSON',
+ isEnabled: true,
+ source: 'message',
+ samples: [
+ JSON.stringify({
+ timestamp: 0,
+ webaclId: webAclArn,
+ action: 'BLOCK',
+ httpRequest: {
+ clientIp: '192.0.2.1',
+ uri: '/',
+ httpMethod: 'GET',
+ },
+ }),
+ ],
+ grok: {
+ matchRules: 'waf_json %{data::json}',
+ supportRules: '',
+ },
+ },
+ },
+ {
+ grokParser: {
+ name: 'Parse W3C traceparent',
+ isEnabled: true,
+ source: 'message',
+ samples: [
+ '{"httpRequest":{"headers":[{"name":"traceparent","value":"00-4BF92F3577B34DA6A3CE929D0E0E4736-00F067AA0BA902B7-01"}]}}',
+ ],
+ grok: {
+ matchRules:
+ 'traceparent %{traceparentPrefix}%{regex("[0-9A-Fa-f]{32}"):waf.trace_id:lowercase}-%{regex("[0-9A-Fa-f]{16}"):waf.span_id:lowercase}-%{regex("[0-9A-Fa-f]{2}")}%{data}',
+ supportRules:
+ 'traceparentPrefix .*"name"\\s*:\\s*"[Tt][Rr][Aa][Cc][Ee][Pp][Aa][Rr][Ee][Nn][Tt]"\\s*,\\s*"value"\\s*:\\s*"00-',
+ },
+ },
+ },
+ {
+ attributeRemapper: {
+ name: 'Normalize WAF web ACL ARN',
+ isEnabled: true,
+ sources: ['webaclId'],
+ sourceType: 'attribute',
+ target: 'waf.web_acl_arn',
+ targetType: 'attribute',
+ preserveSource: true,
+ overrideOnConflict: false,
+ },
+ },
+ {
+ attributeRemapper: {
+ name: 'Normalize WAF request ID',
+ isEnabled: true,
+ sources: ['httpRequest.requestId'],
+ sourceType: 'attribute',
+ target: 'waf.request_id',
+ targetType: 'attribute',
+ preserveSource: true,
+ overrideOnConflict: false,
+ },
+ },
+ {
+ attributeRemapper: {
+ name: 'Normalize WAF rule ID',
+ isEnabled: true,
+ sources: ['terminatingRuleId'],
+ sourceType: 'attribute',
+ target: 'waf.rule_id',
+ targetType: 'attribute',
+ preserveSource: true,
+ overrideOnConflict: false,
+ },
+ },
+ {
+ attributeRemapper: {
+ name: 'Normalize WAF action',
+ isEnabled: true,
+ sources: ['action'],
+ sourceType: 'attribute',
+ target: 'waf.action',
+ targetType: 'attribute',
+ preserveSource: true,
+ overrideOnConflict: false,
+ },
+ },
+ {
+ attributeRemapper: {
+ name: 'Normalize client IP',
+ isEnabled: true,
+ sources: ['httpRequest.clientIp'],
+ sourceType: 'attribute',
+ target: 'network.client.ip',
+ targetType: 'attribute',
+ preserveSource: true,
+ overrideOnConflict: false,
+ },
+ },
+ {
+ attributeRemapper: {
+ name: 'Normalize HTTP method',
+ isEnabled: true,
+ sources: ['httpRequest.httpMethod'],
+ sourceType: 'attribute',
+ target: 'http.method',
+ targetType: 'attribute',
+ preserveSource: true,
+ overrideOnConflict: false,
+ },
+ },
+ {
+ attributeRemapper: {
+ name: 'Normalize HTTP version',
+ isEnabled: true,
+ sources: ['httpRequest.httpVersion'],
+ sourceType: 'attribute',
+ target: 'http.version',
+ targetType: 'attribute',
+ preserveSource: true,
+ overrideOnConflict: false,
+ },
+ },
+ {
+ geoIpParser: {
+ name: 'Enrich client IP',
+ isEnabled: true,
+ sources: ['network.client.ip'],
+ target: 'network.client.geoip',
+ },
+ },
+ {
+ dateRemapper: {
+ name: 'Use WAF event time',
+ isEnabled: true,
+ sources: ['timestamp'],
+ },
+ },
+ {
+ stringBuilderProcessor: {
+ name: 'Set service',
+ isEnabled: true,
+ target: 'waf.service',
+ template: 'cloud-storage-service',
+ isReplaceMissing: true,
+ },
+ },
+ {
+ categoryProcessor: {
+ name: 'Set WAF response status',
+ isEnabled: true,
+ target: 'waf.status_category',
+ categories: [
+ {
+ name: '403',
+ filter: { query: '@action:BLOCK' },
+ },
+ {
+ name: '202',
+ filter: { query: '@action:CHALLENGE' },
+ },
+ ],
+ },
+ },
+ {
+ attributeRemapper: {
+ name: 'Normalize WAF response status',
+ isEnabled: true,
+ sources: ['waf.status_category'],
+ sourceType: 'attribute',
+ target: 'http.response.status_code',
+ targetType: 'attribute',
+ targetFormat: 'integer',
+ preserveSource: false,
+ overrideOnConflict: true,
+ },
+ },
+ {
+ stringBuilderProcessor: {
+ name: 'Set environment',
+ isEnabled: true,
+ target: 'env',
+ template: 'prod',
+ isReplaceMissing: true,
+ },
+ },
+ {
+ stringBuilderProcessor: {
+ name: 'Set source',
+ isEnabled: true,
+ target: 'source',
+ template: 'waf',
+ isReplaceMissing: true,
+ },
+ },
+ {
+ traceIdRemapper: {
+ name: 'Remap trace ID',
+ isEnabled: true,
+ sources: ['waf.trace_id'],
+ },
+ },
+ {
+ spanIdRemapper: {
+ name: 'Remap span ID',
+ isEnabled: true,
+ sources: ['waf.span_id'],
+ },
+ },
+ {
+ serviceRemapper: {
+ name: 'Remap service',
+ isEnabled: true,
+ sources: ['waf.service'],
+ },
+ },
+ ],
+ },
+ {
+ parent: this,
+ protect: true,
+ }
+ );
+
+ const currentPipelineOrder = datadog.getLogsPipelinesOrderOutput();
+ new datadog.LogsPipelineOrder(
+ `${name}-datadog-pipeline-order`,
+ {
+ name: 'macro-cloud-storage-waf-pipeline-order',
+ pipelines: pulumi
+ .all([currentPipelineOrder.pipelineIds, datadogPipeline.id])
+ .apply(([pipelineIds, pipelineId]) => [
+ pipelineId,
+ ...pipelineIds.filter((id) => id !== pipelineId),
+ ]),
+ },
+ {
+ parent: this,
+ protect: true,
+ }
+ );
+
+ this.registerOutputs({
+ webAclArn: webAcl.arn,
+ logGroupName: logGroup.name,
+ });
+ }
+}