The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
Split the Asymptote reader into its own Terraform modules
shukieshah68a7962c8abf
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 6a643b58..1c67777c 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -302,7 +302,7 @@ jobs:- name: Check Terraform modulesrun: |terraform fmt -check -recursive -diff terraform- for module in terraform/aws-s3 terraform/gcs; do+ for module in terraform/*/; doterraform -chdir="$module" init -backend=false -input=falseterraform -chdir="$module" validateterraform -chdir="$module" testdiff --git a/CLAUDE.md b/CLAUDE.mdindex 89ddc972..f74a72ec 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -18,7 +18,7 @@ Beacon Endpoint Agent is a local-only endpoint telemetry agent for AI runtimes.- `spec/lenses`: the Lenses format — single-file HTML views of one trace that the local dashboard runs in a sandboxed, network-less frame and feeds once through `window.beacon.getTrace()`. The contract types (`LensManifestV1`, `LensDataV1`) live in `pkg/asymptoteobserve/lens.go`, and `lens_test.go` pins the spec's examples and manifest schema to them. The dashboard builds the data in `BuildLensData` (`/api/lens-data`), lists lenses at `/api/lenses`, and serves each lens at `/lenses/frame/<id>` under the spec's CSP with `lenses/prelude.js` (which defines `window.beacon`) injected ahead of it; that route is the only one that may be framed. Built-in lenses are `cli/beacon/internal/endpoint/dashboard/lenses/*.lens.html`, and `activity` is byte-identical to the spec example (a test enforces it). The session page (`static/session.html`, `initLensTabs` in `app.js`) shows lenses as tabs fed by `/api/lens-data?session=`, which projects exactly the events that page lists; it is the host half of the protocol and closes a lens that errors, stalls, or navigates its own frame. Trust model (SPEC "Trust model"): the opaque origin, sandbox and CSP keep a lens from touching Beacon or the dashboard and block ordinary network APIs, but cannot fully prevent a lens from sending trace data out through browser features no CSP governs (WebRTC, resource hints, speculative loading), so an installed lens is code the person trusts with the trace data it is shown; never describe the sandbox as making exfiltration impossible. `prelude.js` removes the `RTC*` globals and guards the document (refuses creating, inserting or parsing in nested frames, `<link>` elements and `<script type=speculationrules>`, parsing markup in the real call's document and context and inserting exactly the checked nodes so `<noscript>`/foreign-content parser differentials cannot slip through, refuses `document.write` and `execCommand("insertHTML")`, and a mutation observer removes those from the file's own markup); that is best-effort hardening, not a boundary, and the hostile `probe` fixture in `testdata/lens-e2e` pins each covered path. Installed lenses live in the lens store (`cli/beacon/internal/endpoint/lensstore`, `<BaseDir>/lenses/<id>.lens.html`, 0600 under 0700), written only by `beacon lenses add|remove` (`cli/beacon/cmd/lenses.go`; `add` prints the trust warning and confirms on a TTY, and refuses without one or in CI unless `--yes`); the dashboard rescans it per request and never writes it, and a store lens can never shadow a built-in. `asymptoteobserve.CheckLensFile` is the one file check every installer, server and linter uses. User docs are `docs/concepts/lenses.mdx` and `docs/cli/lenses.mdx`; keep them in sync with the commands and built-in lenses. The authoring loop is `beacon lenses spec` (the spec carried in the binary as `dashboard/lenses/SPEC.md`, kept byte-identical to `spec/lenses/SPEC.md` by a test), `beacon lenses data` (the exact `LensDataV1` for a session), `beacon lenses lint` (`lensstore.Lint`; built-ins must lint clean) and `beacon lenses preview` (the dashboard with `Options.LensFiles`, re-read per reload).- `beacon-sandbox`: harness that runs a real Claude Code session in a disposable sandbox and checks what Beacon captured. See `beacon-sandbox/AGENTS.md`.- `packaging`: macOS, Linux, and Windows packaging and deployment assets.-- `terraform`: customer-run Terraform modules for the cloud side of object storage forwarding. Each module always creates a new, dedicated bucket and never manages an existing one. `aws-s3` creates the hardened bucket, the put-only `beacon-endpoint-writer` IAM user, and optionally `beacon-s3-reader` for Beacon Cloud's S3 data connector, which reads with an access key because ClickPipes on the GCP-hosted ClickHouse rejects IAM roles. The reader comes with an SQS queue per stream (runtime, inventory), each with a dead-letter queue, and the bucket notification that feeds them, so the ClickPipes run in any-order mode; without a queue a pipe reads in key order and skips objects that land behind its cursor, which many Macs uploading on their own timers cause. `gcs` creates the bucket, a `beacon-endpoint-writer` service account with `objectCreator` limited to the prefix by an IAM condition, an optional reader, and with `grant_beacon_read_access` an `objectViewer` grant for Beacon's `beacon-clickpipes-reader` plus a Pub/Sub topic, bucket notification and one subscription per stream matching the managed path's (60 s ack, 7-day retention, never expiring, `objectId` prefix filter). The queue URLs and subscription paths are outputs, ready for the provisioning script's `--queue-url`. Neither module creates keys unless asked (`create_access_keys`, `create_writer_key`), so no secret lands in state by default. Resource names and the `forwarder_settings` output match the packaged `install-forwarder.sh` helpers and the manual guides. docs.beacon.sh (`docs/log-forwarding/{aws-s3,gcs}-terraform.mdx`) documents the bucket and writer only. The Beacon reader, its queues and subscriptions, and the S3 data connector are the enterprise self-hosted path and are documented only in beacon-mdr's docs (docs.asymptotelabs.ai), so keep them out of `docs/`; `/manage/configure-s3-data-connector` redirects there. Both sites pin the module to a release tag; keep them in sync with the variables.+- `terraform`: customer-run Terraform modules for the cloud side of object storage forwarding. The base modules `aws-s3` and `gcs` always create a new, dedicated bucket (never an existing one) and the writer every Mac's forwarder uses: the put-only `beacon-endpoint-writer` IAM user, or a `beacon-endpoint-writer` service account with `objectCreator` limited to the prefix by an IAM condition. The enterprise self-hosted path, where Asymptote reads that bucket through ClickHouse, is two separate modules that take the base module's `bucket` output: `aws-s3-asymptote-reader` (an `asymptote-s3-reader` IAM user, because ClickPipes on the GCP-hosted ClickHouse rejects IAM roles, plus an SQS queue and dead-letter queue per stream and the bucket notification, which it owns) and `gcs-asymptote-reader` (`objectViewer` for `beacon-clickpipes-reader@asymptote-labs`, a Pub/Sub topic, the notification and a subscription per stream with the managed path's settings). The queues exist because a ClickPipe without one reads in key order and skips objects that land behind its cursor, which Macs uploading on their own timers cause; their URLs and subscription paths are outputs for the provisioning script's `--queue-url`. Keep everything Asymptote-specific out of the base modules. No module creates keys unless asked (`create_writer_access_key`, `create_writer_key`, `create_access_key`), so no secret lands in state by default. Resource names and `forwarder_settings` match the packaged `install-forwarder.sh` helpers and the manual guides. docs.beacon.sh (`docs/log-forwarding/{aws-s3,gcs}-terraform.mdx`) documents the base modules only. The reader modules and the self-hosted setup are documented only on docs.asymptotelabs.ai (beacon-mdr `docs/deployment/self-hosted-{s3,gcs}.mdx`), so keep them out of `docs/`; `/manage/configure-s3-data-connector` redirects there. Both sites pin the modules to a release tag; keep them in sync with the variables.- `docs`: the published documentation site (docs.asymptotelabs.ai).Do not recreate or depend on removed `asymptote` mirror trees. Keep new work focused on the Beacon paths above.@@ -136,8 +136,7 @@ go test ./...Run the Terraform module tests (the providers are mocked, so no cloud account is needed):```bash-cd terraform/aws-s3 && terraform init -backend=false && terraform test-cd ../gcs && terraform init -backend=false && terraform test+for module in terraform/*/; do (cd "$module" && terraform init -backend=false && terraform test); done```Run the managed runtime plugin/extension checks (each verifies the checked-in copy matches its embedded twin, then runs its tests):@@ -568,7 +567,7 @@ unpacked through Chrome's developer mode.- `browser-extension` (Node 22, matrix over `chromium` and `msedge`): `npm run check`, `npm run test:unit`, `npm run build`, a check that the build is a loadable MV3 extension, `npm run lint:firefox` (the Firefox build plus `addons-linter --warnings-as-errors`), and the Playwright replay e2e running the real extension headless in that leg's browser (`BROWSER_CHANNEL`) against a local HTTPS replay server; the HTML report and traces upload on failure. Brave, Opera, Vivaldi and Arc have no Playwright channel; `BROWSER_EXECUTABLE` runs the same e2e against their binaries for the manual checklist in `docs/runtimes/browser-extension-chromium.mdx`.- `lens-e2e` (ubuntu): runs the real dashboard handler with hostile fixture lenses and the full `rules/` corpus installed (`cli/beacon/internal/endpoint/dashboard/e2eserver`, `testdata/lens-e2e`) and drives the session page in headless Chromium. Every escape from the lens sandbox must be refused, and a lens that throws, stalls or navigates must give way to the full session.- `beacon-sandbox` (ubuntu): `go build`, `go vet`, and `go test` for the verification harness. Hermetic — no Modal account, no API key, no paid resources.-- `terraform` (ubuntu): `terraform fmt -check`, `validate`, and `terraform test` (mocked providers) for `terraform/aws-s3` and `terraform/gcs`, and `validate` for each example.+- `terraform` (ubuntu): `terraform fmt -check`, `validate`, and `terraform test` (mocked providers) for every module under `terraform/`, and `validate` for each example.- `cross-build` (ubuntu): builds all five release targets and asserts the Linux binaries are statically linked (no `PT_INTERP`).- `package-smoke` (macOS): `packaging/macos/test-endpoint-scripts.sh`, a `build-pkg.sh` payload check, and `packaging/macos/smoke-endpoint.sh`.- `linux-systemd-smoke` and `linux-package-smoke` (ubuntu): the real systemd install path, and installing the built `.deb` in a systemd container.diff --git a/docs/docs.json b/docs/docs.jsonindex 63aae3ca..96c31735 100644--- a/docs/docs.json+++ b/docs/docs.json@@ -678,7 +678,7 @@"redirects": [{"source": "/manage/configure-s3-data-connector",- "destination": "https://docs.asymptotelabs.ai/manage/configure-s3-data-connector"+ "destination": "https://docs.asymptotelabs.ai/deployment/self-hosted-s3"},{"source": "/log-forwarding/s3",diff --git a/docs/log-forwarding/aws-s3-terraform.mdx b/docs/log-forwarding/aws-s3-terraform.mdxindex 180c1bac..ce8909ca 100644--- a/docs/log-forwarding/aws-s3-terraform.mdx+++ b/docs/log-forwarding/aws-s3-terraform.mdx@@ -160,9 +160,9 @@ Set any of these in the `module "beacon_s3"` block and run `terraform apply` agaObjects below the prefix are deleted a year after upload. Versioning is on by default, so a deleted or replaced object stays recoverable for `noncurrent_version_expiration_days` (default 30). Set `versioning_enabled = false` to turn versioning off.</Accordion>- <Accordion title="Create the access keys with Terraform">+ <Accordion title="Create the access key with Terraform">```hcl- create_access_keys = true+ create_writer_access_key = true```Terraform creates the writer's key. Read the secret with `terraform output -raw writer_secret_access_key` after adding a matching `output` block. The secret is stored in Terraform state, so use this only when your state is encrypted and access is restricted.@@ -188,7 +188,7 @@ Set any of these in the `module "beacon_s3"` block and run `terraform apply` aga| --- | --- || `bucket_name`, `region`, `prefix`, `storage_class` | The forwarder settings, also together as `forwarder_settings` || `writer_user_name`, `writer_user_arn` | The IAM user every Mac uploads as |-| `writer_access_key_id`, `writer_secret_access_key` | Only with `create_access_keys = true` |+| `writer_access_key_id`, `writer_secret_access_key` | Only with `create_writer_access_key = true` |## Rotate the writer keydiff --git a/terraform/README.md b/terraform/README.mdindex d70e531e..fc3fc960 100644--- a/terraform/README.md+++ b/terraform/README.md@@ -7,8 +7,13 @@ your cloud credentials.| Module | Creates | Guide || --- | --- | --- |-| [`aws-s3`](aws-s3) | An S3 bucket, a write-only IAM user, and optionally a read-only user for Beacon Cloud | [AWS S3 with Terraform](https://docs.beacon.sh/log-forwarding/aws-s3-terraform) |-| [`gcs`](gcs) | A Cloud Storage bucket, a create-only service account, and optionally read access for Beacon Cloud | [Google Cloud Storage with Terraform](https://docs.beacon.sh/log-forwarding/gcs-terraform) |+| [`aws-s3`](aws-s3) | A new S3 bucket and a write-only IAM user | [AWS S3 with Terraform](https://docs.beacon.sh/log-forwarding/aws-s3-terraform) |+| [`gcs`](gcs) | A new Cloud Storage bucket and a create-only service account | [Google Cloud Storage with Terraform](https://docs.beacon.sh/log-forwarding/gcs-terraform) |+| [`aws-s3-asymptote-reader`](aws-s3-asymptote-reader) | Read-only access for Asymptote to an `aws-s3` bucket, with new-object SQS queues | [Asymptote with your own S3 bucket](https://docs.asymptotelabs.ai/deployment/self-hosted-s3) |+| [`gcs-asymptote-reader`](gcs-asymptote-reader) | Read access for Asymptote to a `gcs` bucket, with new-object Pub/Sub subscriptions | [Asymptote with your own Cloud Storage bucket](https://docs.asymptotelabs.ai/deployment/self-hosted-gcs) |++The reader modules take the base module's `bucket` output, so the bucket is always one the+base module created.Use a module from your own configuration and pin it to a Beacon release:@@ -21,19 +26,20 @@ module "beacon_s3" {}```-Neither module creates keys unless you ask it to (`create_access_keys`,-`create_writer_key`), so by default no secret is written to Terraform state. The-guides show how to create the key with the AWS CLI or gcloud instead.+No module creates keys unless you ask it to (`create_writer_access_key`,+`create_writer_key`, `create_access_key`), so by default no secret is written to+Terraform state. The guides show how to create the keys with the AWS CLI or gcloud+instead.## DevelopmentThe tests mock the cloud providers, so they need no account:```bash-cd terraform/aws-s3 # or terraform/gcs+cd terraform/aws-s3 # or any other module directoryterraform init -backend=falseterraform test```-CI runs `terraform fmt -check`, `validate` and `test` for both modules, and+CI runs `terraform fmt -check`, `validate` and `test` for every module, and`validate` for each example.diff --git a/terraform/aws-s3-asymptote-reader/README.md b/terraform/aws-s3-asymptote-reader/README.mdnew file mode 100644index 00000000..d17c7aa7--- /dev/null+++ b/terraform/aws-s3-asymptote-reader/README.md@@ -0,0 +1,33 @@+# Asymptote reader for a Beacon S3 bucket++Gives Asymptote read-only access to a Beacon bucket created by the [`aws-s3`](../aws-s3)+module, for customers who keep Beacon telemetry in their own bucket and use Asymptote's+dashboard:++- `asymptote-s3-reader`, an IAM user that can list the bucket, read objects below the+ Beacon prefix, and consume the two queues below;+- an SQS queue per stream (`runtime`, `inventory`), each with a dead-letter queue, that S3+ notifies of every new object, so Asymptote loads objects in any order;+- the bucket's notification configuration, which this module owns.++```hcl+module "beacon_s3" {+ source = "github.com/asymptote-labs/agent-beacon//terraform/aws-s3?ref=<release tag>"++ bucket_name = "acme-beacon-telemetry"+ prefix = "beacon-prod"+}++module "asymptote_reader" {+ source = "github.com/asymptote-labs/agent-beacon//terraform/aws-s3-asymptote-reader?ref=<release tag>"++ bucket = module.beacon_s3.bucket+}+```++Use a provider in the bucket's region: S3 can only notify queues in its own region. Create+the reader's access key with the AWS CLI, so its secret never enters Terraform state, and+send Asymptote the key with the `source_url` and `queue_urls` outputs.++The full walkthrough is on the Asymptote docs:+[Asymptote with your own S3 bucket](https://docs.asymptotelabs.ai/deployment/self-hosted-s3).diff --git a/terraform/aws-s3-asymptote-reader/examples/basic/main.tf b/terraform/aws-s3-asymptote-reader/examples/basic/main.tfnew file mode 100644index 00000000..3f4106a7--- /dev/null+++ b/terraform/aws-s3-asymptote-reader/examples/basic/main.tf@@ -0,0 +1,83 @@+# A Beacon bucket that Asymptote can read: the aws-s3 module creates the bucket and the writer,+# and this module adds Asymptote's read-only user and the new-object queues.+#+# terraform init+# terraform apply -var region=us-west-2 -var bucket_name=<globally-unique-name>+# aws iam create-access-key --user-name "$(terraform output -raw writer_user_name)"+# aws iam create-access-key --user-name "$(terraform output -raw asymptote_reader_user_name)"+#+# Outside this repository, point both sources at a release:+# source = "github.com/asymptote-labs/agent-beacon//terraform/aws-s3?ref=<release tag>"+# source = "github.com/asymptote-labs/agent-beacon//terraform/aws-s3-asymptote-reader?ref=<release tag>"++terraform {+ required_version = ">= 1.5.0"++ required_providers {+ aws = {+ source = "hashicorp/aws"+ version = ">= 5.0"+ }+ }+}++variable "region" {+ description = "AWS region for the bucket and the queues."+ type = string+}++variable "bucket_name" {+ description = "Globally unique name for the new bucket."+ type = string+}++variable "prefix" {+ description = "Root prefix for Beacon objects."+ type = string+ default = "beacon-prod"+}++provider "aws" {+ region = var.region+}++module "beacon_s3" {+ source = "../../../aws-s3"++ bucket_name = var.bucket_name+ prefix = var.prefix+}++module "asymptote_reader" {+ source = "../.."++ bucket = module.beacon_s3.bucket+}++output "bucket_name" {+ value = module.beacon_s3.bucket_name+}++output "region" {+ value = module.beacon_s3.region+}++output "prefix" {+ value = module.beacon_s3.prefix+}++output "writer_user_name" {+ value = module.beacon_s3.writer_user_name+}++output "asymptote_reader_user_name" {+ value = module.asymptote_reader.user_name+}++output "asymptote_source_url" {+ value = module.asymptote_reader.source_url+}++output "asymptote_queue_urls" {+ value = module.asymptote_reader.queue_urls+}diff --git a/terraform/aws-s3-asymptote-reader/main.tf b/terraform/aws-s3-asymptote-reader/main.tfnew file mode 100644index 00000000..a1422ec7--- /dev/null+++ b/terraform/aws-s3-asymptote-reader/main.tf@@ -0,0 +1,160 @@+# Read-only access for Asymptote to a Beacon bucket created by the aws-s3 module.+#+# Asymptote's ClickPipes read the bucket in "any order" mode: S3 announces every new object on an+# SQS queue, and the pipe loads what the queue names. Without the queues a pipe lists objects in+# key order and skips any that sort before ones it has already loaded, which happens whenever Macs+# upload out of step.++data "aws_partition" "current" {}++data "aws_caller_identity" "current" {}++locals {+ streams = toset(["runtime", "inventory"])++ objects_arn = "${var.bucket.arn}/${var.bucket.prefix}/*"++ # SQS names allow letters, digits, hyphens and underscores; IAM user names allow a few more.+ queue_base = replace(var.user_name, "/[^A-Za-z0-9_-]/", "-")++ # S3's certificate for virtual-hosted URLs covers one label, so a dotted bucket name needs path style.+ s3_host = "s3.${var.bucket.region}.${data.aws_partition.current.dns_suffix}"+ source_url = (+ strcontains(var.bucket.name, ".")+ ? "https://${local.s3_host}/${var.bucket.name}/${var.bucket.prefix}"+ : "https://${var.bucket.name}.${local.s3_host}/${var.bucket.prefix}"+ )++ statements = concat(+ [+ {+ Sid = "ListBeaconBucket"+ Effect = "Allow"+ Action = ["s3:ListBucket", "s3:GetBucketLocation"]+ Resource = var.bucket.arn+ },+ {+ Sid = "ReadBeaconObjects"+ Effect = "Allow"+ Action = "s3:GetObject"+ Resource = local.objects_arn+ },+ # The four actions ClickPipes needs to consume the new-object queues.+ {+ Sid = "ConsumeBeaconNotifications"+ Effect = "Allow"+ Action = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]+ Resource = [for queue in aws_sqs_queue.reader : queue.arn]+ },+ {+ Sid = "FindBeaconQueues"+ Effect = "Allow"+ Action = "sqs:ListQueues"+ Resource = "*"+ },+ ],+ var.bucket.kms_key_arn == null ? [] : [+ {+ Sid = "DecryptBeaconObjects"+ Effect = "Allow"+ Action = "kms:Decrypt"+ Resource = var.bucket.kms_key_arn+ },+ ],+ )+}++resource "aws_iam_user" "reader" {+ name = var.user_name+ path = var.iam_path+ permissions_boundary = var.permissions_boundary_arn+ tags = var.tags+}++resource "aws_iam_user_policy" "reader" {+ name = "BeaconTelemetryReadOnly"+ user = aws_iam_user.reader.name+ policy = jsonencode({+ Version = "2012-10-17"+ Statement = local.statements+ })+}++resource "aws_iam_access_key" "reader" {+ count = var.create_access_key ? 1 : 0++ user = aws_iam_user.reader.name+}++# A message that fails ten deliveries moves here instead of being dropped, so it can be redriven.+resource "aws_sqs_queue" "dead_letter" {+ for_each = local.streams++ name = "${local.queue_base}-${each.key}-dlq"+ message_retention_seconds = 1209600+ sqs_managed_sse_enabled = true+ tags = var.tags+}++resource "aws_sqs_queue" "reader" {+ for_each = local.streams++ name = "${local.queue_base}-${each.key}"+ message_retention_seconds = 1209600+ visibility_timeout_seconds = 300+ sqs_managed_sse_enabled = true+ tags = var.tags++ redrive_policy = jsonencode({+ deadLetterTargetArn = aws_sqs_queue.dead_letter[each.key].arn+ maxReceiveCount = 10+ })+}++resource "aws_sqs_queue_policy" "reader" {+ for_each = local.streams++ queue_url = aws_sqs_queue.reader[each.key].id+ policy = jsonencode({+ Version = "2012-10-17"+ Statement = [+ {+ Sid = "AllowBucketNotifications"+ Effect = "Allow"+ Principal = { Service = "s3.amazonaws.com" }+ Action = "sqs:SendMessage"+ Resource = aws_sqs_queue.reader[each.key].arn+ Condition = {+ ArnEquals = { "aws:SourceArn" = var.bucket.arn }+ StringEquals = { "aws:SourceAccount" = data.aws_caller_identity.current.account_id }+ }+ },+ ]+ })+}++# S3 keeps one notification configuration per bucket. The aws-s3 module never sets one, so this+# module owns it.+resource "aws_s3_bucket_notification" "this" {+ bucket = var.bucket.name++ dynamic "queue" {+ for_each = local.streams+ content {+ id = "beacon-${queue.key}"+ queue_arn = aws_sqs_queue.reader[queue.key].arn+ events = ["s3:ObjectCreated:*"]+ filter_prefix = "${var.bucket.prefix}/${queue.key}/"+ }+ }++ lifecycle {+ precondition {+ condition = split(":", aws_sqs_queue.reader["runtime"].arn)[3] == var.bucket.region+ error_message = "S3 can only notify queues in the bucket's own region. Use a provider in ${var.bucket.region} for this module."+ }+ }++ # S3 checks that it may send to each queue when the configuration is written.+ depends_on = [aws_sqs_queue_policy.reader]+}diff --git a/terraform/aws-s3-asymptote-reader/outputs.tf b/terraform/aws-s3-asymptote-reader/outputs.tfnew file mode 100644index 00000000..1899cf82--- /dev/null+++ b/terraform/aws-s3-asymptote-reader/outputs.tf@@ -0,0 +1,30 @@+output "user_name" {+ description = "Read-only IAM user Asymptote reads the bucket with. Create its access key with: aws iam create-access-key --user-name <this value>"+ value = aws_iam_user.reader.name+}++output "user_arn" {+ description = "ARN of the reader IAM user."+ value = aws_iam_user.reader.arn+}++output "access_key_id" {+ description = "Reader access key ID, when create_access_key = true."+ value = try(aws_iam_access_key.reader[0].id, null)+}++output "secret_access_key" {+ description = "Reader secret access key, when create_access_key = true."+ value = try(aws_iam_access_key.reader[0].secret, null)+ sensitive = true+}++output "source_url" {+ description = "The bucket location to send Asymptote."+ value = local.source_url+}++output "queue_urls" {+ description = "The new-object queues to send Asymptote, by stream."+ value = { for stream, queue in aws_sqs_queue.reader : stream => queue.url }+}diff --git a/terraform/aws-s3-asymptote-reader/tests/module.tftest.hcl b/terraform/aws-s3-asymptote-reader/tests/module.tftest.hclnew file mode 100644index 00000000..47199026--- /dev/null+++ b/terraform/aws-s3-asymptote-reader/tests/module.tftest.hcl@@ -0,0 +1,217 @@+# Hermetic: the AWS provider is mocked, so these runs need no account or credentials.++mock_provider "aws" {+ mock_data "aws_partition" {+ defaults = {+ partition = "aws"+ dns_suffix = "amazonaws.com"+ }+ }++ mock_data "aws_caller_identity" {+ defaults = {+ account_id = "111122223333"+ }+ }++ mock_resource "aws_sqs_queue" {+ defaults = {+ arn = "arn:aws:sqs:us-west-2:111122223333:mock"+ }+ }+}++variables {+ bucket = {+ name = "acme-beacon-telemetry"+ arn = "arn:aws:s3:::acme-beacon-telemetry"+ region = "us-west-2"+ prefix = "beacon-prod"+ }+}++run "reader_reads_only_the_prefix_and_its_queues" {+ command = apply++ assert {+ condition = jsondecode(aws_iam_user_policy.reader.policy).Statement == [+ {+ Sid = "ListBeaconBucket"+ Effect = "Allow"+ Action = ["s3:ListBucket", "s3:GetBucketLocation"]+ Resource = "arn:aws:s3:::acme-beacon-telemetry"+ },+ {+ Sid = "ReadBeaconObjects"+ Effect = "Allow"+ Action = "s3:GetObject"+ Resource = "arn:aws:s3:::acme-beacon-telemetry/beacon-prod/*"+ },+ {+ Sid = "ConsumeBeaconNotifications"+ Effect = "Allow"+ Action = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]+ Resource = [aws_sqs_queue.reader["inventory"].arn, aws_sqs_queue.reader["runtime"].arn]+ },+ {+ Sid = "FindBeaconQueues"+ Effect = "Allow"+ Action = "sqs:ListQueues"+ Resource = "*"+ },+ ]+ error_message = "The reader may list the bucket, read only the Beacon prefix, and consume its two queues."+ }++ assert {+ condition = aws_iam_user.reader.name == "asymptote-s3-reader" && aws_iam_user_policy.reader.name == "BeaconTelemetryReadOnly"+ error_message = "The reader has the documented names."+ }++ assert {+ condition = length(aws_iam_access_key.reader) == 0+ error_message = "The access key, whose secret would land in state, must be opt-in."+ }+}++run "each_stream_gets_a_queue" {+ command = apply++ assert {+ condition = (+ aws_sqs_queue.reader["runtime"].name == "asymptote-s3-reader-runtime"+ && aws_sqs_queue.reader["inventory"].name == "asymptote-s3-reader-inventory"+ && aws_sqs_queue.dead_letter["runtime"].name == "asymptote-s3-reader-runtime-dlq"+ && aws_sqs_queue.dead_letter["inventory"].name == "asymptote-s3-reader-inventory-dlq"+ )+ error_message = "Each stream gets a queue and a dead-letter queue named after the reader."+ }++ assert {+ condition = alltrue([+ for stream in ["runtime", "inventory"] :+ aws_sqs_queue.reader[stream].sqs_managed_sse_enabled+ && aws_sqs_queue.reader[stream].message_retention_seconds == 1209600+ && aws_sqs_queue.reader[stream].visibility_timeout_seconds == 300+ && jsondecode(aws_sqs_queue.reader[stream].redrive_policy) == {+ deadLetterTargetArn = aws_sqs_queue.dead_letter[stream].arn+ maxReceiveCount = 10+ }+ ])+ error_message = "Queues are encrypted, keep messages 14 days, and move a message to the dead-letter queue after ten failed deliveries."+ }++ assert {+ condition = alltrue([+ for stream in ["runtime", "inventory"] :+ jsondecode(aws_sqs_queue_policy.reader[stream].policy).Statement == [+ {+ Sid = "AllowBucketNotifications"+ Effect = "Allow"+ Principal = { Service = "s3.amazonaws.com" }+ Action = "sqs:SendMessage"+ Resource = aws_sqs_queue.reader[stream].arn+ Condition = {+ ArnEquals = { "aws:SourceArn" = "arn:aws:s3:::acme-beacon-telemetry" }+ StringEquals = { "aws:SourceAccount" = "111122223333" }+ }+ },+ ]+ ])+ error_message = "Only this bucket, in this account, may send to the queues."+ }++ assert {+ condition = aws_s3_bucket_notification.this.bucket == "acme-beacon-telemetry" && {+ for queue in aws_s3_bucket_notification.this.queue : queue.filter_prefix => {+ arn = queue.queue_arn+ events = queue.events+ }+ } == {+ "beacon-prod/runtime/" = { arn = aws_sqs_queue.reader["runtime"].arn, events = toset(["s3:ObjectCreated:*"]) }+ "beacon-prod/inventory/" = { arn = aws_sqs_queue.reader["inventory"].arn, events = toset(["s3:ObjectCreated:*"]) }+ }+ error_message = "Every new object below each stream's prefix is announced on that stream's queue."+ }++ assert {+ condition = output.queue_urls == {+ runtime = aws_sqs_queue.reader["runtime"].url+ inventory = aws_sqs_queue.reader["inventory"].url+ }+ error_message = "queue_urls gives the provisioning script's --queue-url values."+ }++ assert {+ condition = output.source_url == "https://acme-beacon-telemetry.s3.us-west-2.amazonaws.com/beacon-prod"+ error_message = "The source URL is the bucket's regional endpoint plus the prefix."+ }+}++run "dotted_bucket_names_get_a_path_style_url" {+ command = apply++ variables {+ bucket = {+ name = "acme.beacon.telemetry"+ arn = "arn:aws:s3:::acme.beacon.telemetry"+ region = "us-west-2"+ prefix = "beacon-prod"+ }+ }++ assert {+ condition = output.source_url == "https://s3.us-west-2.amazonaws.com/acme.beacon.telemetry/beacon-prod"+ error_message = "A dotted bucket name fails TLS on a virtual-hosted URL, so the URL must be path style."+ }+}++run "kms_bucket_needs_decrypt" {+ command = apply++ variables {+ bucket = {+ name = "acme-beacon-telemetry"+ arn = "arn:aws:s3:::acme-beacon-telemetry"+ region = "us-west-2"+ prefix = "beacon-prod"+ kms_key_arn = "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"+ }+ }++ assert {+ condition = contains(jsondecode(aws_iam_user_policy.reader.policy).Statement, {+ Sid = "DecryptBeaconObjects"+ Effect = "Allow"+ Action = "kms:Decrypt"+ Resource = "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"+ })+ error_message = "The reader needs kms:Decrypt to read an SSE-KMS bucket."+ }+}++run "queue_names_drop_characters_sqs_refuses" {+ command = plan++ variables {+ user_name = "asymptote.reader@acme"+ }++ assert {+ condition = aws_sqs_queue.reader["runtime"].name == "asymptote-reader-acme-runtime"+ error_message = "IAM user names allow . and @, which SQS names do not."+ }+}++run "access_key_when_asked" {+ command = plan++ variables {+ create_access_key = true+ }++ assert {+ condition = length(aws_iam_access_key.reader) == 1+ error_message = "create_access_key creates one key."+ }+}diff --git a/terraform/aws-s3-asymptote-reader/tests/partition.tftest.hcl b/terraform/aws-s3-asymptote-reader/tests/partition.tftest.hclnew file mode 100644index 00000000..59e5c97b--- /dev/null+++ b/terraform/aws-s3-asymptote-reader/tests/partition.tftest.hcl@@ -0,0 +1,34 @@+# The module builds URLs from the partition, so it also works in AWS China and GovCloud.++mock_provider "aws" {+ mock_data "aws_partition" {+ defaults = {+ partition = "aws-cn"+ dns_suffix = "amazonaws.com.cn"+ }+ }++ mock_resource "aws_sqs_queue" {+ defaults = {+ arn = "arn:aws-cn:sqs:cn-north-1:111122223333:mock"+ }+ }+}++variables {+ bucket = {+ name = "acme-beacon-telemetry"+ arn = "arn:aws-cn:s3:::acme-beacon-telemetry"+ region = "cn-north-1"+ prefix = "beacon-prod"+ }+}++run "source_url_follows_the_partition" {+ command = apply++ assert {+ condition = output.source_url == "https://acme-beacon-telemetry.s3.cn-north-1.amazonaws.com.cn/beacon-prod"+ error_message = "The source URL uses the partition's DNS suffix."+ }+}diff --git a/terraform/aws-s3-asymptote-reader/tests/region.tftest.hcl b/terraform/aws-s3-asymptote-reader/tests/region.tftest.hclnew file mode 100644index 00000000..7962e7be--- /dev/null+++ b/terraform/aws-s3-asymptote-reader/tests/region.tftest.hcl@@ -0,0 +1,24 @@+# S3 can only notify queues in the bucket's own region.++mock_provider "aws" {+ mock_resource "aws_sqs_queue" {+ defaults = {+ arn = "arn:aws:sqs:eu-west-1:111122223333:mock"+ }+ }+}++variables {+ bucket = {+ name = "acme-beacon-telemetry"+ arn = "arn:aws:s3:::acme-beacon-telemetry"+ region = "us-west-2"+ prefix = "beacon-prod"+ }+}++run "refuses_queues_in_another_region" {+ command = apply++ expect_failures = [aws_s3_bucket_notification.this]+}diff --git a/terraform/aws-s3-asymptote-reader/variables.tf b/terraform/aws-s3-asymptote-reader/variables.tfnew file mode 100644index 00000000..abc27484--- /dev/null+++ b/terraform/aws-s3-asymptote-reader/variables.tf@@ -0,0 +1,45 @@+variable "bucket" {+ description = "The Beacon bucket to read, as the aws-s3 module's bucket output: bucket = module.beacon_s3.bucket."+ type = object({+ name = string+ arn = string+ region = string+ prefix = string+ kms_key_arn = optional(string)+ })+}++variable "user_name" {+ description = "Name of the read-only IAM user Asymptote reads the bucket with. Its queues are named after it: <name>-runtime and <name>-inventory, each with a -dlq."+ type = string+ default = "asymptote-s3-reader"+}++variable "create_access_key" {+ description = "Also create the reader's access key. The secret key is then stored in Terraform state, so leave this false unless your state is encrypted and access-controlled, and create the key with the AWS CLI instead."+ type = bool+ default = false+}++variable "iam_path" {+ description = "Path for the IAM user."+ type = string+ default = "/"++ validation {+ condition = can(regex("^/([A-Za-z0-9+=,.@_-]+/)*$", var.iam_path))+ error_message = "iam_path must begin and end with a slash, such as / or /beacon/."+ }+}++variable "permissions_boundary_arn" {+ description = "ARN of a permissions boundary policy to attach to the IAM user, if your organization requires one."+ type = string+ default = null+}++variable "tags" {+ description = "Tags to apply to every resource the module creates."+ type = map(string)+ default = {}+}diff --git a/terraform/aws-s3-asymptote-reader/versions.tf b/terraform/aws-s3-asymptote-reader/versions.tfnew file mode 100644index 00000000..b088b0f1--- /dev/null+++ b/terraform/aws-s3-asymptote-reader/versions.tf@@ -0,0 +1,10 @@+terraform {+ required_version = ">= 1.5.0"++ required_providers {+ aws = {+ source = "hashicorp/aws"+ version = ">= 5.0"+ }+ }+}diff --git a/terraform/aws-s3/README.md b/terraform/aws-s3/README.mdindex 6dab9139..89e3abe0 100644--- a/terraform/aws-s3/README.md+++ b/terraform/aws-s3/README.md@@ -7,9 +7,8 @@ Creates the AWS side of Beacon's S3 forwarding:lifecycle rule scoped to the Beacon prefix;- `beacon-endpoint-writer`, an IAM user that can only `s3:PutObject` below the prefix(plus `s3:ListBucket` for the forwarder's startup health check);-- optionally `beacon-s3-reader`, a read-only IAM user for Beacon Cloud's S3 data- connector (`create_beacon_reader = true`), with an SQS queue per stream that S3- notifies of every new object, so Beacon loads objects in any order.+- optionally the writer's access key (`create_writer_access_key = true`), though the+ docs create it with the AWS CLI so its secret stays out of Terraform state.```hclprovider "aws" {diff --git a/terraform/aws-s3/iam.tf b/terraform/aws-s3/iam.tfindex 6652bf6b..4725fc53 100644--- a/terraform/aws-s3/iam.tf+++ b/terraform/aws-s3/iam.tf@@ -25,44 +25,6 @@ locals {},],)-- reader_statements = concat(- [- {- Sid = "ListBeaconBucket"- Effect = "Allow"- Action = ["s3:ListBucket", "s3:GetBucketLocation"]- Resource = local.bucket_arn- },- {- Sid = "ReadBeaconObjects"- Effect = "Allow"- Action = "s3:GetObject"- Resource = local.beacon_objects_arn- },- # The four actions ClickPipes needs to consume the new-object queues.- {- Sid = "ConsumeBeaconNotifications"- Effect = "Allow"- Action = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]- Resource = [for queue in aws_sqs_queue.beacon_reader : queue.arn]- },- {- Sid = "FindBeaconQueues"- Effect = "Allow"- Action = "sqs:ListQueues"- Resource = "*"- },- ],- var.kms_key_arn == null ? [] : [- {- Sid = "DecryptBeaconObjects"- Effect = "Allow"- Action = "kms:Decrypt"- Resource = var.kms_key_arn- },- ],- )}# The identity every Mac's forwarder uses. It can create objects below the Beacon prefix, and it@@ -84,34 +46,7 @@ resource "aws_iam_user_policy" "writer" {}resource "aws_iam_access_key" "writer" {- count = var.create_access_keys ? 1 : 0+ count = var.create_writer_access_key ? 1 : 0user = aws_iam_user.writer.name}--# Beacon Cloud's S3 data connector reads with an access key; it cannot assume a role in your account.-resource "aws_iam_user" "beacon_reader" {- count = var.create_beacon_reader ? 1 : 0-- name = var.beacon_reader_user_name- path = var.iam_path- permissions_boundary = var.permissions_boundary_arn- tags = var.tags-}--resource "aws_iam_user_policy" "beacon_reader" {- count = var.create_beacon_reader ? 1 : 0-- name = "BeaconTelemetryReadOnly"- user = aws_iam_user.beacon_reader[0].name- policy = jsonencode({- Version = "2012-10-17"- Statement = local.reader_statements- })-}--resource "aws_iam_access_key" "beacon_reader" {- count = var.create_beacon_reader && var.create_access_keys ? 1 : 0-- user = aws_iam_user.beacon_reader[0].name-}diff --git a/terraform/aws-s3/main.tf b/terraform/aws-s3/main.tfindex aff4c1f9..244b0e56 100644--- a/terraform/aws-s3/main.tf+++ b/terraform/aws-s3/main.tf@@ -1,20 +1,10 @@data "aws_partition" "current" {}-data "aws_caller_identity" "current" {}-locals {bucket_arn = "arn:${data.aws_partition.current.partition}:s3:::${var.bucket_name}"# Beacon writes <prefix>/runtime/date=YYYY-MM-DD/... and <prefix>/inventory/date=YYYY-MM-DD/...beacon_objects_arn = "${local.bucket_arn}/${var.prefix}/*"-- # S3's certificate for virtual-hosted URLs covers one label, so a dotted bucket name needs path style.- s3_host = "s3.${aws_s3_bucket.this.region}.${data.aws_partition.current.dns_suffix}"- beacon_reader_source_url = (- strcontains(var.bucket_name, ".")- ? "https://${local.s3_host}/${var.bucket_name}/${var.prefix}"- : "https://${var.bucket_name}.${local.s3_host}/${var.prefix}"- )}# A dedicated bucket for Beacon telemetry, always created by the module.diff --git a/terraform/aws-s3/notifications.tf b/terraform/aws-s3/notifications.tfdeleted file mode 100644index d9c0b529..00000000--- a/terraform/aws-s3/notifications.tf+++ /dev/null@@ -1,78 +0,0 @@-# Beacon Cloud's ClickPipes read the bucket in "any order" mode: S3 announces every new object on-# an SQS queue, and the pipe loads what the queue names. Without the queues a pipe lists objects in-# key order and skips any that sort before ones it has already loaded, which happens whenever Macs-# upload out of step.--locals {- reader_streams = var.create_beacon_reader ? toset(["runtime", "inventory"]) : toset([])-- # SQS names allow letters, digits, hyphens and underscores; IAM user names allow a few more.- reader_queue_base = replace(var.beacon_reader_user_name, "/[^A-Za-z0-9_-]/", "-")-}--# A message that fails ten deliveries moves here instead of being dropped, so it can be redriven.-resource "aws_sqs_queue" "beacon_reader_dead_letter" {- for_each = local.reader_streams-- name = "${local.reader_queue_base}-${each.key}-dlq"- message_retention_seconds = 1209600- sqs_managed_sse_enabled = true- tags = var.tags-}--resource "aws_sqs_queue" "beacon_reader" {- for_each = local.reader_streams-- name = "${local.reader_queue_base}-${each.key}"- message_retention_seconds = 1209600- visibility_timeout_seconds = 300- sqs_managed_sse_enabled = true- tags = var.tags-- redrive_policy = jsonencode({- deadLetterTargetArn = aws_sqs_queue.beacon_reader_dead_letter[each.key].arn- maxReceiveCount = 10- })-}--resource "aws_sqs_queue_policy" "beacon_reader" {- for_each = local.reader_streams-- queue_url = aws_sqs_queue.beacon_reader[each.key].id- policy = jsonencode({- Version = "2012-10-17"- Statement = [- {- Sid = "AllowBucketNotifications"- Effect = "Allow"- Principal = { Service = "s3.amazonaws.com" }- Action = "sqs:SendMessage"- Resource = aws_sqs_queue.beacon_reader[each.key].arn- Condition = {- ArnEquals = { "aws:SourceArn" = local.bucket_arn }- StringEquals = { "aws:SourceAccount" = data.aws_caller_identity.current.account_id }- }- },- ]- })-}--# S3 keeps one notification configuration per bucket. The module owns this bucket, so it owns that too.-resource "aws_s3_bucket_notification" "this" {- count = var.create_beacon_reader ? 1 : 0-- bucket = aws_s3_bucket.this.id-- dynamic "queue" {- for_each = local.reader_streams- content {- id = "beacon-${queue.key}"- queue_arn = aws_sqs_queue.beacon_reader[queue.key].arn- events = ["s3:ObjectCreated:*"]- filter_prefix = "${var.prefix}/${queue.key}/"- }- }-- # S3 checks that it may send to each queue when the configuration is written.- depends_on = [aws_sqs_queue_policy.beacon_reader]-}diff --git a/terraform/aws-s3/outputs.tf b/terraform/aws-s3/outputs.tfindex 6c8b856d..ce6a80b0 100644--- a/terraform/aws-s3/outputs.tf+++ b/terraform/aws-s3/outputs.tf@@ -18,6 +18,17 @@ output "prefix" {value = var.prefix}+output "bucket" {+ description = "The bucket as one object, for the aws-s3-asymptote-reader module: bucket = module.beacon_s3.bucket."+ value = {+ name = aws_s3_bucket.this.bucket+ arn = local.bucket_arn+ region = aws_s3_bucket.this.region+ prefix = var.prefix+ kms_key_arn = var.kms_key_arn+ }+}+output "storage_class" {description = "Storage class the forwarder writes objects with (BEACON_S3_STORAGE_CLASS)."value = var.storage_class@@ -44,43 +55,12 @@ output "writer_user_arn" {}output "writer_access_key_id" {- description = "Writer access key ID, when create_access_keys = true."+ description = "Writer access key ID, when create_writer_access_key = true."value = try(aws_iam_access_key.writer[0].id, null)}output "writer_secret_access_key" {- description = "Writer secret access key, when create_access_keys = true. Read it with terraform output -raw writer_secret_access_key."+ description = "Writer secret access key, when create_writer_access_key = true. Read it with terraform output -raw writer_secret_access_key."value = try(aws_iam_access_key.writer[0].secret, null)sensitive = true}--output "beacon_reader_user_name" {- description = "Read-only IAM user for Beacon Cloud's S3 data connector, when create_beacon_reader = true."- value = try(aws_iam_user.beacon_reader[0].name, null)-}--output "beacon_reader_user_arn" {- description = "ARN of the Beacon reader IAM user, when create_beacon_reader = true."- value = try(aws_iam_user.beacon_reader[0].arn, null)-}--output "beacon_reader_access_key_id" {- description = "Beacon reader access key ID, when create_beacon_reader and create_access_keys are both true."- value = try(aws_iam_access_key.beacon_reader[0].id, null)-}--output "beacon_reader_secret_access_key" {- description = "Beacon reader secret access key, when create_beacon_reader and create_access_keys are both true."- value = try(aws_iam_access_key.beacon_reader[0].secret, null)- sensitive = true-}--output "beacon_reader_source_url" {- description = "The location to give Beacon for the S3 data connector."- value = local.beacon_reader_source_url-}--output "beacon_reader_queue_urls" {- description = "The new-object queues to give Beacon for the S3 data connector, by stream, when create_beacon_reader = true."- value = var.create_beacon_reader ? { for stream, queue in aws_sqs_queue.beacon_reader : stream => queue.url } : null-}diff --git a/terraform/aws-s3/tests/module.tftest.hcl b/terraform/aws-s3/tests/module.tftest.hclindex 6c4852e5..804ae734 100644--- a/terraform/aws-s3/tests/module.tftest.hcl+++ b/terraform/aws-s3/tests/module.tftest.hcl@@ -8,12 +8,6 @@ mock_provider "aws" {}}- mock_data "aws_caller_identity" {- defaults = {- account_id = "111122223333"- }- }-mock_resource "aws_s3_bucket" {defaults = {bucket = "acme-beacon-telemetry"@@ -58,13 +52,8 @@ run "no_access_keys_by_default" {command = planassert {- condition = length(aws_iam_access_key.writer) == 0 && length(aws_iam_access_key.beacon_reader) == 0- error_message = "Access keys, whose secrets would land in state, must be opt-in."- }-- assert {- condition = length(aws_iam_user.beacon_reader) == 0- error_message = "The Beacon reader must be opt-in."+ condition = length(aws_iam_access_key.writer) == 0+ error_message = "The access key, whose secret would land in state, must be opt-in."}}@@ -145,30 +134,22 @@ run "forwarder_settings_match_the_helper" {}assert {- condition = output.beacon_reader_source_url == "https://acme-beacon-telemetry.s3.us-west-2.amazonaws.com/beacon-prod"- error_message = "The data connector URL is the bucket's regional endpoint plus the prefix."- }-}--run "source_url_uses_path_style_for_dotted_bucket_names" {- command = apply-- variables {- bucket_name = "acme.beacon.telemetry"- }-- assert {- condition = output.beacon_reader_source_url == "https://s3.us-west-2.amazonaws.com/acme.beacon.telemetry/beacon-prod"- error_message = "A dotted bucket name fails TLS on a virtual-hosted URL, so the URL must be path style."+ condition = (+ output.bucket.name == "acme-beacon-telemetry"+ && output.bucket.arn == "arn:aws:s3:::acme-beacon-telemetry"+ && output.bucket.region == "us-west-2"+ && output.bucket.prefix == "beacon-prod"+ && output.bucket.kms_key_arn == null+ )+ error_message = "The bucket output carries what the reader module needs."}}run "kms_key_encrypts_and_is_granted" {- command = apply+ command = planvariables {- kms_key_arn = "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"- create_beacon_reader = true+ kms_key_arn = "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"}assert {@@ -189,16 +170,6 @@ run "kms_key_encrypts_and_is_granted" {})error_message = "The writer needs kms:GenerateDataKey to upload to an SSE-KMS bucket."}-- assert {- condition = contains(jsondecode(aws_iam_user_policy.beacon_reader[0].policy).Statement, {- Sid = "DecryptBeaconObjects"- Effect = "Allow"- Action = "kms:Decrypt"- Resource = "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"- })- error_message = "The Beacon reader needs kms:Decrypt to read an SSE-KMS bucket."- }}run "healthcheck_permission_can_be_dropped" {@@ -214,161 +185,16 @@ run "healthcheck_permission_can_be_dropped" {}}-run "beacon_reader_reads_only_the_prefix" {- command = apply-- variables {- create_beacon_reader = true- }-- assert {- condition = jsondecode(aws_iam_user_policy.beacon_reader[0].policy).Statement == [- {- Sid = "ListBeaconBucket"- Effect = "Allow"- Action = ["s3:ListBucket", "s3:GetBucketLocation"]- Resource = "arn:aws:s3:::acme-beacon-telemetry"- },- {- Sid = "ReadBeaconObjects"- Effect = "Allow"- Action = "s3:GetObject"- Resource = "arn:aws:s3:::acme-beacon-telemetry/beacon-prod/*"- },- {- Sid = "ConsumeBeaconNotifications"- Effect = "Allow"- Action = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]- Resource = [aws_sqs_queue.beacon_reader["inventory"].arn, aws_sqs_queue.beacon_reader["runtime"].arn]- },- {- Sid = "FindBeaconQueues"- Effect = "Allow"- Action = "sqs:ListQueues"- Resource = "*"- },- ]- error_message = "The Beacon reader may list the bucket, read only the Beacon prefix, and consume its two queues."- }-- assert {- condition = aws_iam_user.beacon_reader[0].name == "beacon-s3-reader" && length(aws_iam_access_key.beacon_reader) == 0- error_message = "The reader is created without a key unless create_access_keys is set."- }-}--run "beacon_reader_gets_new_object_queues" {- command = apply-- variables {- create_beacon_reader = true- }-- assert {- condition = (- aws_sqs_queue.beacon_reader["runtime"].name == "beacon-s3-reader-runtime"- && aws_sqs_queue.beacon_reader["inventory"].name == "beacon-s3-reader-inventory"- && aws_sqs_queue.beacon_reader_dead_letter["runtime"].name == "beacon-s3-reader-runtime-dlq"- && aws_sqs_queue.beacon_reader_dead_letter["inventory"].name == "beacon-s3-reader-inventory-dlq"- )- error_message = "Each stream gets a queue and a dead-letter queue named after the reader."- }-- assert {- condition = alltrue([- for stream in ["runtime", "inventory"] :- aws_sqs_queue.beacon_reader[stream].sqs_managed_sse_enabled- && aws_sqs_queue.beacon_reader[stream].message_retention_seconds == 1209600- && aws_sqs_queue.beacon_reader[stream].visibility_timeout_seconds == 300- && jsondecode(aws_sqs_queue.beacon_reader[stream].redrive_policy) == {- deadLetterTargetArn = aws_sqs_queue.beacon_reader_dead_letter[stream].arn- maxReceiveCount = 10- }- ])- error_message = "Queues are encrypted, keep messages 14 days, and move a message to the dead-letter queue after ten failed deliveries."- }-- assert {- condition = alltrue([- for stream in ["runtime", "inventory"] :- jsondecode(aws_sqs_queue_policy.beacon_reader[stream].policy).Statement == [- {- Sid = "AllowBucketNotifications"- Effect = "Allow"- Principal = { Service = "s3.amazonaws.com" }- Action = "sqs:SendMessage"- Resource = aws_sqs_queue.beacon_reader[stream].arn- Condition = {- ArnEquals = { "aws:SourceArn" = "arn:aws:s3:::acme-beacon-telemetry" }- StringEquals = { "aws:SourceAccount" = "111122223333" }- }- },- ]- ])- error_message = "Only this bucket, in this account, may send to the queues."- }-- assert {- condition = {- for queue in aws_s3_bucket_notification.this[0].queue : queue.filter_prefix => {- arn = queue.queue_arn- events = queue.events- }- } == {- "beacon-prod/runtime/" = { arn = aws_sqs_queue.beacon_reader["runtime"].arn, events = toset(["s3:ObjectCreated:*"]) }- "beacon-prod/inventory/" = { arn = aws_sqs_queue.beacon_reader["inventory"].arn, events = toset(["s3:ObjectCreated:*"]) }- }- error_message = "Every new object below each stream's prefix is announced on that stream's queue."- }-- assert {- condition = output.beacon_reader_queue_urls == {- runtime = aws_sqs_queue.beacon_reader["runtime"].url- inventory = aws_sqs_queue.beacon_reader["inventory"].url- }- error_message = "beacon_reader_queue_urls gives the provisioning script's --queue-url values."- }-}--run "queue_names_drop_characters_sqs_refuses" {- command = plan-- variables {- create_beacon_reader = true- beacon_reader_user_name = "beacon.reader@acme"- }-- assert {- condition = aws_sqs_queue.beacon_reader["runtime"].name == "beacon-reader-acme-runtime"- error_message = "IAM user names allow . and @, which SQS names do not."- }-}--run "no_queues_without_the_reader" {- command = apply-- assert {- condition = (- length(aws_sqs_queue.beacon_reader) == 0- && length(aws_sqs_queue.beacon_reader_dead_letter) == 0- && length(aws_s3_bucket_notification.this) == 0- && output.beacon_reader_queue_urls == null- )- error_message = "Queues and the bucket notification exist only with the Beacon reader."- }-}--run "access_keys_when_asked" {+run "writer_access_key_when_asked" {command = planvariables {- create_beacon_reader = true- create_access_keys = true+ create_writer_access_key = true}assert {- condition = length(aws_iam_access_key.writer) == 1 && length(aws_iam_access_key.beacon_reader) == 1- error_message = "create_access_keys creates one key per user."+ condition = length(aws_iam_access_key.writer) == 1+ error_message = "create_writer_access_key creates one key."}}diff --git a/terraform/aws-s3/tests/partition.tftest.hcl b/terraform/aws-s3/tests/partition.tftest.hclindex ae90c39b..61167862 100644--- a/terraform/aws-s3/tests/partition.tftest.hcl+++ b/terraform/aws-s3/tests/partition.tftest.hcl@@ -24,11 +24,6 @@ variables {run "urls_and_arns_follow_the_partition" {command = apply- assert {- condition = output.beacon_reader_source_url == "https://acme-beacon-telemetry.s3.cn-north-1.amazonaws.com.cn/beacon-prod"- error_message = "The data connector URL uses the partition's DNS suffix."- }-assert {condition = output.bucket_arn == "arn:aws-cn:s3:::acme-beacon-telemetry"error_message = "The bucket ARN uses the partition."diff --git a/terraform/aws-s3/variables.tf b/terraform/aws-s3/variables.tfindex 6691f57c..cf6a0674 100644--- a/terraform/aws-s3/variables.tf+++ b/terraform/aws-s3/variables.tf@@ -42,7 +42,7 @@ variable "storage_class" {}variable "kms_key_arn" {- description = "ARN of a customer managed KMS key to make the bucket's default encryption (SSE-KMS with an S3 Bucket Key). The writer gets kms:GenerateDataKey and the Beacon reader gets kms:Decrypt on it. Leave null for SSE-S3."+ description = "ARN of a customer managed KMS key to make the bucket's default encryption (SSE-KMS with an S3 Bucket Key). The writer gets kms:GenerateDataKey on it. Leave null for SSE-S3."type = stringdefault = null@@ -98,26 +98,14 @@ variable "allow_writer_healthcheck" {default = true}-variable "create_beacon_reader" {- description = "Create a read-only IAM user for Beacon Cloud's S3 data connector, which reads your bucket with an access key. It can read only the Beacon prefix. The module also creates an SQS queue per stream, with a dead-letter queue, that S3 notifies of every new object, so the connector loads objects in any order."- type = bool- default = false-}--variable "beacon_reader_user_name" {- description = "Name of the read-only IAM user created when create_beacon_reader = true. Its queues are named after it: <name>-runtime and <name>-inventory, each with a -dlq."- type = string- default = "beacon-s3-reader"-}--variable "create_access_keys" {- description = "Also create an access key for the writer (and for the Beacon reader, when created). The secret keys are then stored in Terraform state, so leave this false unless your state is encrypted and access-controlled, and create the keys with the AWS CLI instead."+variable "create_writer_access_key" {+ description = "Also create the writer's access key. The secret key is then stored in Terraform state, so leave this false unless your state is encrypted and access-controlled, and create the key with the AWS CLI instead."type = booldefault = false}variable "iam_path" {- description = "Path for the IAM users."+ description = "Path for the writer IAM user."type = stringdefault = "/"@@ -128,7 +116,7 @@ variable "iam_path" {}variable "permissions_boundary_arn" {- description = "ARN of a permissions boundary policy to attach to the IAM users, if your organization requires one."+ description = "ARN of a permissions boundary policy to attach to the writer IAM user, if your organization requires one."type = stringdefault = null}diff --git a/terraform/gcs-asymptote-reader/README.md b/terraform/gcs-asymptote-reader/README.mdnew file mode 100644index 00000000..bdce661d--- /dev/null+++ b/terraform/gcs-asymptote-reader/README.md@@ -0,0 +1,32 @@+# Asymptote reader for a Beacon Cloud Storage bucket++Gives Asymptote read access to a Beacon bucket created by the [`gcs`](../gcs) module, for+customers who keep Beacon telemetry in their own bucket and use Asymptote's dashboard:++- `roles/storage.objectViewer` on the bucket for Asymptote's reader,+ `[email protected]`. No key is involved;+- a Pub/Sub topic the bucket notifies of every new object below the Beacon prefix, with one+ subscription per stream (`runtime`, `inventory`) that Asymptote's reader may consume, so+ Asymptote loads objects in any order.++```hcl+module "beacon_gcs" {+ source = "github.com/asymptote-labs/agent-beacon//terraform/gcs?ref=<release tag>"++ project = "acme-security"+ location = "us-west1"+ bucket_name = "acme-beacon-telemetry"+ prefix = "beacon-prod"+}++module "asymptote_reader" {+ source = "github.com/asymptote-labs/agent-beacon//terraform/gcs-asymptote-reader?ref=<release tag>"++ bucket = module.beacon_gcs.bucket+}+```++Send Asymptote the bucket name, the prefix and the `subscriptions` output.++The full walkthrough is on the Asymptote docs:+[Asymptote with your own Cloud Storage bucket](https://docs.asymptotelabs.ai/deployment/self-hosted-gcs).diff --git a/terraform/gcs-asymptote-reader/examples/basic/main.tf b/terraform/gcs-asymptote-reader/examples/basic/main.tfnew file mode 100644index 00000000..a94f6c6b--- /dev/null+++ b/terraform/gcs-asymptote-reader/examples/basic/main.tf@@ -0,0 +1,77 @@+# A Beacon bucket that Asymptote can read: the gcs module creates the bucket and the writer, and+# this module adds Asymptote's read access and the new-object subscriptions.+#+# terraform init+# terraform apply -var project=<project-id> -var location=us-west1 -var bucket_name=<globally-unique-name>+# gcloud iam service-accounts keys create beacon-gcs-writer.json --iam-account "$(terraform output -raw writer_email)"+#+# Outside this repository, point both sources at a release:+# source = "github.com/asymptote-labs/agent-beacon//terraform/gcs?ref=<release tag>"+# source = "github.com/asymptote-labs/agent-beacon//terraform/gcs-asymptote-reader?ref=<release tag>"++terraform {+ required_version = ">= 1.5.0"++ required_providers {+ google = {+ source = "hashicorp/google"+ version = ">= 5.0"+ }+ }+}++variable "project" {+ description = "Google Cloud project for the bucket, service account and subscriptions."+ type = string+}++variable "location" {+ description = "Bucket location, such as us-west1."+ type = string+}++variable "bucket_name" {+ description = "Globally unique name for the new bucket."+ type = string+}++variable "prefix" {+ description = "Root prefix for Beacon objects."+ type = string+ default = "beacon-prod"+}++provider "google" {+ project = var.project+}++module "beacon_gcs" {+ source = "../../../gcs"++ project = var.project+ location = var.location+ bucket_name = var.bucket_name+ prefix = var.prefix+}++module "asymptote_reader" {+ source = "../.."++ bucket = module.beacon_gcs.bucket+}++output "bucket_name" {+ value = module.beacon_gcs.bucket_name+}++output "prefix" {+ value = module.beacon_gcs.prefix+}++output "writer_email" {+ value = module.beacon_gcs.writer_email+}++output "asymptote_subscriptions" {+ value = module.asymptote_reader.subscriptions+}diff --git a/terraform/gcs-asymptote-reader/main.tf b/terraform/gcs-asymptote-reader/main.tfnew file mode 100644index 00000000..ed32cc6c--- /dev/null+++ b/terraform/gcs-asymptote-reader/main.tf@@ -0,0 +1,86 @@+# Read access for Asymptote to a Beacon bucket created by the gcs module.+#+# Asymptote's ClickPipes read the bucket in "any order" mode: Cloud Storage announces every new+# object on Pub/Sub, and the pipe loads what its subscription names. Without them a pipe lists+# objects in name order and skips any that sort before ones it has already loaded, which happens+# whenever Macs upload out of step. The subscriptions match the ones Asymptote's managed ingest uses.++locals {+ streams = toset(["runtime", "inventory"])+}++resource "google_project_service" "pubsub" {+ count = var.enable_apis ? 1 : 0++ project = var.bucket.project+ service = "pubsub.googleapis.com"+ disable_on_destroy = false+}++# Asymptote reads with its own service account, so no key is involved.+resource "google_storage_bucket_iam_member" "reader" {+ bucket = var.bucket.name+ role = "roles/storage.objectViewer"+ member = var.reader_member+}++# Cloud Storage publishes notifications as its own service agent.+data "google_storage_project_service_account" "this" {+ project = var.bucket.project+}++resource "google_pubsub_topic" "objects" {+ project = var.bucket.project+ name = "asymptote-${var.bucket.name}-objects"+ labels = var.labels++ depends_on = [google_project_service.pubsub]+}++resource "google_pubsub_topic_iam_member" "storage_agent" {+ project = var.bucket.project+ topic = google_pubsub_topic.objects.name+ role = "roles/pubsub.publisher"+ member = "serviceAccount:${data.google_storage_project_service_account.this.email_address}"+}++resource "google_storage_notification" "objects" {+ bucket = var.bucket.name+ topic = google_pubsub_topic.objects.id+ payload_format = "JSON_API_V1"+ event_types = ["OBJECT_FINALIZE"]+ object_name_prefix = "${var.bucket.prefix}/"++ # Cloud Storage checks that it may publish to the topic when the notification is created.+ depends_on = [google_pubsub_topic_iam_member.storage_agent]+}++resource "google_pubsub_subscription" "reader" {+ for_each = local.streams++ project = var.bucket.project+ name = "asymptote-${var.bucket.name}-${each.key}"+ topic = google_pubsub_topic.objects.id+ filter = "attributes.eventType = \"OBJECT_FINALIZE\" AND hasPrefix(attributes.objectId, \"${var.bucket.prefix}/${each.key}/\")"+ ack_deadline_seconds = 60+ message_retention_duration = "604800s"+ labels = var.labels++ # An empty TTL never expires the subscription. The default expires it after 31 quiet days,+ # which would silently stop ingestion from a fleet that was offline for a month.+ expiration_policy {+ ttl = ""+ }+}++resource "google_pubsub_subscription_iam_member" "reader" {+ for_each = {+ for pair in setproduct(local.streams, ["roles/pubsub.subscriber", "roles/pubsub.viewer"]) :+ "${pair[0]}:${pair[1]}" => { stream = pair[0], role = pair[1] }+ }++ project = var.bucket.project+ subscription = google_pubsub_subscription.reader[each.value.stream].name+ role = each.value.role+ member = var.reader_member+}diff --git a/terraform/gcs-asymptote-reader/outputs.tf b/terraform/gcs-asymptote-reader/outputs.tfnew file mode 100644index 00000000..bbd635fb--- /dev/null+++ b/terraform/gcs-asymptote-reader/outputs.tf@@ -0,0 +1,9 @@+output "subscriptions" {+ description = "The new-object subscriptions to send Asymptote, by stream."+ value = { for stream, sub in google_pubsub_subscription.reader : stream => sub.id }+}++output "topic" {+ description = "The Pub/Sub topic the bucket publishes new objects to."+ value = google_pubsub_topic.objects.id+}diff --git a/terraform/gcs-asymptote-reader/tests/module.tftest.hcl b/terraform/gcs-asymptote-reader/tests/module.tftest.hclnew file mode 100644index 00000000..514c0443--- /dev/null+++ b/terraform/gcs-asymptote-reader/tests/module.tftest.hcl@@ -0,0 +1,109 @@+# Hermetic: the Google provider is mocked, so these runs need no project or credentials.++mock_provider "google" {+ mock_resource "google_pubsub_topic" {+ defaults = {+ id = "projects/acme-security/topics/asymptote-acme-beacon-telemetry-objects"+ }+ }++ mock_data "google_storage_project_service_account" {+ defaults = {+ email_address = "[email protected]"+ }+ }+}++variables {+ bucket = {+ name = "acme-beacon-telemetry"+ project = "acme-security"+ prefix = "beacon-prod"+ }+}++run "asymptote_can_read_the_bucket" {+ command = plan++ assert {+ condition = (+ google_storage_bucket_iam_member.reader.bucket == "acme-beacon-telemetry"+ && google_storage_bucket_iam_member.reader.role == "roles/storage.objectViewer"+ && google_storage_bucket_iam_member.reader.member == "serviceAccount:[email protected]"+ )+ error_message = "Asymptote's reader gets objectViewer on the bucket."+ }+}++run "new_objects_are_announced_per_stream" {+ command = apply++ assert {+ condition = (+ google_pubsub_topic.objects.name == "asymptote-acme-beacon-telemetry-objects"+ && google_pubsub_topic_iam_member.storage_agent.role == "roles/pubsub.publisher"+ && google_pubsub_topic_iam_member.storage_agent.member == "serviceAccount:[email protected]"+ )+ error_message = "Cloud Storage's service agent publishes new-object notifications to a topic for this bucket."+ }++ assert {+ condition = (+ google_storage_notification.objects.bucket == "acme-beacon-telemetry"+ && google_storage_notification.objects.event_types == toset(["OBJECT_FINALIZE"])+ && google_storage_notification.objects.object_name_prefix == "beacon-prod/"+ && google_storage_notification.objects.payload_format == "JSON_API_V1"+ )+ error_message = "Every new object below the prefix is announced."+ }++ assert {+ condition = alltrue([+ for stream in ["runtime", "inventory"] :+ google_pubsub_subscription.reader[stream].name == "asymptote-acme-beacon-telemetry-${stream}"+ && google_pubsub_subscription.reader[stream].filter == "attributes.eventType = \"OBJECT_FINALIZE\" AND hasPrefix(attributes.objectId, \"beacon-prod/${stream}/\")"+ && google_pubsub_subscription.reader[stream].ack_deadline_seconds == 60+ && google_pubsub_subscription.reader[stream].message_retention_duration == "604800s"+ && one(google_pubsub_subscription.reader[stream].expiration_policy).ttl == ""+ ])+ error_message = "Each stream has a never-expiring subscription filtered to its prefix, with the managed path's settings."+ }++ assert {+ condition = toset([+ for binding in google_pubsub_subscription_iam_member.reader : "${binding.subscription}|${binding.role}|${binding.member}"+ ]) == toset(flatten([+ for stream in ["runtime", "inventory"] : [+ for role in ["roles/pubsub.subscriber", "roles/pubsub.viewer"] :+ "${google_pubsub_subscription.reader[stream].name}|${role}|serviceAccount:[email protected]"+ ]+ ]))+ error_message = "Asymptote's reader may consume and view both subscriptions, and nothing else gets access."+ }++ assert {+ condition = output.subscriptions == {+ runtime = google_pubsub_subscription.reader["runtime"].id+ inventory = google_pubsub_subscription.reader["inventory"].id+ }+ error_message = "subscriptions gives the provisioning script's --queue-url values."+ }++ assert {+ condition = google_project_service.pubsub[0].service == "pubsub.googleapis.com" && google_project_service.pubsub[0].disable_on_destroy == false+ error_message = "The module enables Pub/Sub and never disables it."+ }+}++run "apis_left_alone_when_asked" {+ command = plan++ variables {+ enable_apis = false+ }++ assert {+ condition = length(google_project_service.pubsub) == 0+ error_message = "enable_apis = false leaves the project's APIs alone."+ }+}diff --git a/terraform/gcs-asymptote-reader/variables.tf b/terraform/gcs-asymptote-reader/variables.tfnew file mode 100644index 00000000..d4e05adf--- /dev/null+++ b/terraform/gcs-asymptote-reader/variables.tf@@ -0,0 +1,26 @@+variable "bucket" {+ description = "The Beacon bucket to read, as the gcs module's bucket output: bucket = module.beacon_gcs.bucket."+ type = object({+ name = string+ project = string+ prefix = string+ })+}++variable "reader_member" {+ description = "Asymptote's reader identity. Change it only if Asymptote gives you a different one."+ type = string+ default = "serviceAccount:[email protected]"+}++variable "enable_apis" {+ description = "Enable the Pub/Sub API on the project. It is left enabled when the module is destroyed."+ type = bool+ default = true+}++variable "labels" {+ description = "Labels to apply to the Pub/Sub topic and subscriptions."+ type = map(string)+ default = {}+}diff --git a/terraform/gcs-asymptote-reader/versions.tf b/terraform/gcs-asymptote-reader/versions.tfnew file mode 100644index 00000000..55b5fcf4--- /dev/null+++ b/terraform/gcs-asymptote-reader/versions.tf@@ -0,0 +1,10 @@+terraform {+ required_version = ">= 1.5.0"++ required_providers {+ google = {+ source = "hashicorp/google"+ version = ">= 5.0"+ }+ }+}diff --git a/terraform/gcs/README.md b/terraform/gcs/README.mdindex b7ad66af..8c2d2581 100644--- a/terraform/gcs/README.md+++ b/terraform/gcs/README.md@@ -7,10 +7,8 @@ Creates the Google Cloud side of Beacon's GCS forwarding:scoped to the Beacon prefix;- `beacon-endpoint-writer`, a service account with `roles/storage.objectCreator` onthe bucket, limited by an IAM condition to objects below the prefix;-- optionally a read-only `beacon-endpoint-reader` (`create_reader = true`) and read- access for Beacon Cloud (`grant_beacon_read_access = true`), which also publishes- every new object to Pub/Sub with a subscription per stream, so Beacon loads objects in- any order.+- optionally a read-only `beacon-endpoint-reader` (`create_reader = true`) for+ checking uploads.```hclprovider "google" {diff --git a/terraform/gcs/main.tf b/terraform/gcs/main.tfindex c2f5e845..edbe83c1 100644--- a/terraform/gcs/main.tf+++ b/terraform/gcs/main.tf@@ -1,15 +1,9 @@locals {- apis = var.enable_apis ? toset(concat(- ["storage.googleapis.com", "iam.googleapis.com"],- var.grant_beacon_read_access ? ["pubsub.googleapis.com"] : [],- )) : toset([])+ apis = var.enable_apis ? toset(["storage.googleapis.com", "iam.googleapis.com"]) : toset([])# Beacon writes <prefix>/runtime/date=YYYY-MM-DD/... and <prefix>/inventory/date=YYYY-MM-DD/...# The condition is built from the variable, not the bucket resource, so it is known at plan time.writer_condition = "resource.name.startsWith(\"projects/_/buckets/${var.bucket_name}/objects/${var.prefix}/\")"-- # Cloud Storage's service agent encrypts with a customer-managed key and publishes notifications.- needs_storage_agent = var.kms_key_name != null && var.grant_kms_to_storage_agent || var.grant_beacon_read_access}resource "google_project_service" "this" {@@ -21,15 +15,16 @@ resource "google_project_service" "this" {}data "google_storage_project_service_account" "this" {- count = local.needs_storage_agent ? 1 : 0+ count = var.kms_key_name != null && var.grant_kms_to_storage_agent ? 1 : 0project = var.projectdepends_on = [google_project_service.this]}+# Cloud Storage encrypts with a customer-managed key as its own service agent, not as the writer.resource "google_kms_crypto_key_iam_member" "storage_agent" {- count = var.kms_key_name != null && var.grant_kms_to_storage_agent ? 1 : 0+ count = length(data.google_storage_project_service_account.this)crypto_key_id = var.kms_key_namerole = "roles/cloudkms.cryptoKeyEncrypterDecrypter"@@ -134,11 +129,3 @@ resource "google_storage_bucket_iam_member" "reader" {role = "roles/storage.objectViewer"member = "serviceAccount:${google_service_account.reader[0].email}"}--resource "google_storage_bucket_iam_member" "beacon_reader" {- count = var.grant_beacon_read_access ? 1 : 0-- bucket = google_storage_bucket.this.name- role = "roles/storage.objectViewer"- member = var.beacon_reader_member-}diff --git a/terraform/gcs/notifications.tf b/terraform/gcs/notifications.tfdeleted file mode 100644index acad267d..00000000--- a/terraform/gcs/notifications.tf+++ /dev/null@@ -1,70 +0,0 @@-# Beacon Cloud's ClickPipes read the bucket in "any order" mode: Cloud Storage announces every new-# object on Pub/Sub, and the pipe loads what its subscription names. Without them a pipe lists-# objects in name order and skips any that sort before ones it has already loaded, which happens-# whenever Macs upload out of step. The subscriptions match the ones Beacon's managed ingest uses.--locals {- reader_streams = var.grant_beacon_read_access ? toset(["runtime", "inventory"]) : toset([])-}--resource "google_pubsub_topic" "beacon_reader" {- count = var.grant_beacon_read_access ? 1 : 0-- project = var.project- name = "beacon-${var.bucket_name}-objects"- labels = var.labels-- depends_on = [google_project_service.this]-}--resource "google_pubsub_topic_iam_member" "storage_agent" {- count = var.grant_beacon_read_access ? 1 : 0-- project = var.project- topic = google_pubsub_topic.beacon_reader[0].name- role = "roles/pubsub.publisher"- member = "serviceAccount:${data.google_storage_project_service_account.this[0].email_address}"-}--resource "google_storage_notification" "beacon_reader" {- count = var.grant_beacon_read_access ? 1 : 0-- bucket = google_storage_bucket.this.name- topic = google_pubsub_topic.beacon_reader[0].id- payload_format = "JSON_API_V1"- event_types = ["OBJECT_FINALIZE"]- object_name_prefix = "${var.prefix}/"-- # Cloud Storage checks that it may publish to the topic when the notification is created.- depends_on = [google_pubsub_topic_iam_member.storage_agent]-}--resource "google_pubsub_subscription" "beacon_reader" {- for_each = local.reader_streams-- project = var.project- name = "beacon-${var.bucket_name}-${each.key}"- topic = google_pubsub_topic.beacon_reader[0].id- filter = "attributes.eventType = \"OBJECT_FINALIZE\" AND hasPrefix(attributes.objectId, \"${var.prefix}/${each.key}/\")"- ack_deadline_seconds = 60- message_retention_duration = "604800s"- labels = var.labels-- # An empty TTL never expires the subscription. The default expires it after 31 quiet days,- # which would silently stop ingestion from a fleet that was offline for a month.- expiration_policy {- ttl = ""- }-}--resource "google_pubsub_subscription_iam_member" "beacon_reader" {- for_each = {- for pair in setproduct(local.reader_streams, ["roles/pubsub.subscriber", "roles/pubsub.viewer"]) :- "${pair[0]}:${pair[1]}" => { stream = pair[0], role = pair[1] }- }-- project = var.project- subscription = google_pubsub_subscription.beacon_reader[each.value.stream].name- role = each.value.role- member = var.beacon_reader_member-}diff --git a/terraform/gcs/outputs.tf b/terraform/gcs/outputs.tfindex 948f345c..ba6cf94e 100644--- a/terraform/gcs/outputs.tf+++ b/terraform/gcs/outputs.tf@@ -13,6 +13,15 @@ output "prefix" {value = var.prefix}+output "bucket" {+ description = "The bucket as one object, for the gcs-asymptote-reader module: bucket = module.beacon_gcs.bucket."+ value = {+ name = google_storage_bucket.this.name+ project = var.project+ prefix = var.prefix+ }+}+output "storage_class" {description = "Storage class the forwarder writes objects with (BEACON_GCS_STORAGE_CLASS)."value = var.storage_class@@ -42,8 +51,3 @@ output "reader_email" {description = "Read-only service account, when create_reader = true."value = try(google_service_account.reader[0].email, null)}--output "beacon_reader_subscriptions" {- description = "The new-object subscriptions to give Beacon, by stream, when grant_beacon_read_access = true."- value = var.grant_beacon_read_access ? { for stream, sub in google_pubsub_subscription.beacon_reader : stream => sub.id } : null-}diff --git a/terraform/gcs/tests/module.tftest.hcl b/terraform/gcs/tests/module.tftest.hclindex eaa448ce..634cafbc 100644--- a/terraform/gcs/tests/module.tftest.hcl+++ b/terraform/gcs/tests/module.tftest.hcl@@ -14,12 +14,6 @@ mock_provider "google" {}}- mock_resource "google_pubsub_topic" {- defaults = {- id = "projects/acme-security/topics/beacon-acme-beacon-telemetry-objects"- }- }-mock_data "google_storage_project_service_account" {defaults = {email_address = "[email protected]"@@ -53,7 +47,7 @@ run "writer_can_only_create_below_the_prefix" {}}-run "no_key_reader_or_beacon_access_by_default" {+run "no_key_or_reader_by_default" {command = planassert {@@ -61,9 +55,8 @@ run "no_key_reader_or_beacon_access_by_default" {length(google_service_account_key.writer) == 0&& length(google_service_account.reader) == 0&& length(google_storage_bucket_iam_member.reader) == 0- && length(google_storage_bucket_iam_member.beacon_reader) == 0)- error_message = "Keys, the reader and Beacon read access must all be opt-in."+ error_message = "The key and the reader must be opt-in."}}@@ -115,109 +108,13 @@ run "forwarder_settings_match_the_helper" {}assert {- condition = output.writer_email == "[email protected]"- error_message = "writer_email is the service account's email."- }-}--run "beacon_read_access_when_asked" {- command = plan-- variables {- grant_beacon_read_access = true- create_reader = true- }-- assert {- condition = (- google_storage_bucket_iam_member.beacon_reader[0].role == "roles/storage.objectViewer"- && google_storage_bucket_iam_member.beacon_reader[0].member == "serviceAccount:[email protected]"- && length(google_storage_bucket_iam_member.beacon_reader[0].condition) == 0- )- error_message = "Beacon's reader gets objectViewer on the bucket."+ condition = output.bucket.name == "acme-beacon-telemetry" && output.bucket.project == "acme-security" && output.bucket.prefix == "beacon-prod"+ error_message = "The bucket output carries what the reader module needs."}assert {- condition = google_storage_bucket_iam_member.reader[0].role == "roles/storage.objectViewer" && google_service_account.reader[0].account_id == "beacon-endpoint-reader"- error_message = "create_reader adds the read-only service account the manual guides use."- }-}--run "beacon_read_access_brings_new_object_subscriptions" {- command = apply-- variables {- grant_beacon_read_access = true- }-- assert {- condition = (- google_pubsub_topic.beacon_reader[0].name == "beacon-acme-beacon-telemetry-objects"- && google_pubsub_topic_iam_member.storage_agent[0].role == "roles/pubsub.publisher"- && google_pubsub_topic_iam_member.storage_agent[0].member == "serviceAccount:[email protected]"- )- error_message = "Cloud Storage's service agent publishes new-object notifications to a topic for this bucket."- }-- assert {- condition = (- google_storage_notification.beacon_reader[0].event_types == toset(["OBJECT_FINALIZE"])- && google_storage_notification.beacon_reader[0].object_name_prefix == "beacon-prod/"- && google_storage_notification.beacon_reader[0].payload_format == "JSON_API_V1"- )- error_message = "Every new object below the prefix is announced."- }-- assert {- condition = alltrue([- for stream in ["runtime", "inventory"] :- google_pubsub_subscription.beacon_reader[stream].name == "beacon-acme-beacon-telemetry-${stream}"- && google_pubsub_subscription.beacon_reader[stream].filter == "attributes.eventType = \"OBJECT_FINALIZE\" AND hasPrefix(attributes.objectId, \"beacon-prod/${stream}/\")"- && google_pubsub_subscription.beacon_reader[stream].ack_deadline_seconds == 60- && google_pubsub_subscription.beacon_reader[stream].message_retention_duration == "604800s"- && one(google_pubsub_subscription.beacon_reader[stream].expiration_policy).ttl == ""- ])- error_message = "Each stream has a never-expiring subscription filtered to its prefix, with the managed path's settings."- }-- assert {- condition = toset([- for binding in google_pubsub_subscription_iam_member.beacon_reader : "${binding.subscription}|${binding.role}|${binding.member}"- ]) == toset(flatten([- for stream in ["runtime", "inventory"] : [- for role in ["roles/pubsub.subscriber", "roles/pubsub.viewer"] :- "${google_pubsub_subscription.beacon_reader[stream].name}|${role}|serviceAccount:[email protected]"- ]- ]))- error_message = "Beacon's reader may consume and view both subscriptions, and nothing else gets access."- }-- assert {- condition = output.beacon_reader_subscriptions == {- runtime = google_pubsub_subscription.beacon_reader["runtime"].id- inventory = google_pubsub_subscription.beacon_reader["inventory"].id- }- error_message = "beacon_reader_subscriptions gives the provisioning script's --queue-url values."- }-- assert {- condition = contains(keys(google_project_service.this), "pubsub.googleapis.com")- error_message = "Pub/Sub is enabled when Beacon reads the bucket."- }-}--run "no_pubsub_without_beacon_read_access" {- command = apply-- assert {- condition = (- length(google_pubsub_topic.beacon_reader) == 0- && length(google_storage_notification.beacon_reader) == 0- && length(google_pubsub_subscription.beacon_reader) == 0- && output.beacon_reader_subscriptions == null- && !contains(keys(google_project_service.this), "pubsub.googleapis.com")- )- error_message = "The topic, notification and subscriptions exist only with Beacon read access."+ condition = output.writer_email == "[email protected]"+ error_message = "writer_email is the service account's email."}}@@ -293,3 +190,16 @@ run "rejects_short_kms_name" {expect_failures = [var.kms_key_name]}++run "reader_when_asked" {+ command = plan++ variables {+ create_reader = true+ }++ assert {+ condition = google_storage_bucket_iam_member.reader[0].role == "roles/storage.objectViewer" && google_service_account.reader[0].account_id == "beacon-endpoint-reader"+ error_message = "create_reader adds the read-only service account the manual guides use."+ }+}diff --git a/terraform/gcs/variables.tf b/terraform/gcs/variables.tfindex 119a226f..ac3718ab 100644--- a/terraform/gcs/variables.tf+++ b/terraform/gcs/variables.tf@@ -95,7 +95,7 @@ variable "force_destroy" {}variable "enable_apis" {- description = "Enable the Cloud Storage and IAM APIs on the project, and Pub/Sub when grant_beacon_read_access = true. They are left enabled when the module is destroyed."+ description = "Enable the Cloud Storage and IAM APIs on the project. They are left enabled when the module is destroyed."type = booldefault = true}@@ -124,20 +124,8 @@ variable "reader_account_id" {default = "beacon-endpoint-reader"}-variable "grant_beacon_read_access" {- description = "Grant Beacon Cloud read access to the bucket (roles/storage.objectViewer for beacon_reader_member), so Beacon can ingest the telemetry you store there. No key is involved. The module also publishes every new object to a Pub/Sub topic with one subscription per stream that Beacon's reader may consume, so Beacon loads objects in any order."- type = bool- default = false-}--variable "beacon_reader_member" {- description = "Beacon Cloud's reader identity. Change it only if Beacon gives you a different one."- type = string- default = "serviceAccount:[email protected]"-}-variable "labels" {- description = "Labels to apply to the bucket, and to the Pub/Sub topic and subscriptions when they exist."+ description = "Labels to apply to the bucket."type = map(string)default = {}}