Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

Split the Asymptote reader into its own Terraform modules

shukieshah68a7962c8abf
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 6a643b58..1c67777c 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -302,7 +302,7 @@ jobs:
- name: Check Terraform modules
run: |
terraform fmt -check -recursive -diff terraform
- for module in terraform/aws-s3 terraform/gcs; do
+ for module in terraform/*/; do
terraform -chdir="$module" init -backend=false -input=false
terraform -chdir="$module" validate
terraform -chdir="$module" test
diff --git a/CLAUDE.md b/CLAUDE.md
index 89ddc972..f74a72ec 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -18,7 +18,7 @@ Beacon Endpoint Agent is a local-only endpoint telemetry agent for AI runtimes.
- `spec/lenses`: the Lenses format — single-file HTML views of one trace that the local dashboard runs in a sandboxed, network-less frame and feeds once through `window.beacon.getTrace()`. The contract types (`LensManifestV1`, `LensDataV1`) live in `pkg/asymptoteobserve/lens.go`, and `lens_test.go` pins the spec's examples and manifest schema to them. The dashboard builds the data in `BuildLensData` (`/api/lens-data`), lists lenses at `/api/lenses`, and serves each lens at `/lenses/frame/<id>` under the spec's CSP with `lenses/prelude.js` (which defines `window.beacon`) injected ahead of it; that route is the only one that may be framed. Built-in lenses are `cli/beacon/internal/endpoint/dashboard/lenses/*.lens.html`, and `activity` is byte-identical to the spec example (a test enforces it). The session page (`static/session.html`, `initLensTabs` in `app.js`) shows lenses as tabs fed by `/api/lens-data?session=`, which projects exactly the events that page lists; it is the host half of the protocol and closes a lens that errors, stalls, or navigates its own frame. Trust model (SPEC "Trust model"): the opaque origin, sandbox and CSP keep a lens from touching Beacon or the dashboard and block ordinary network APIs, but cannot fully prevent a lens from sending trace data out through browser features no CSP governs (WebRTC, resource hints, speculative loading), so an installed lens is code the person trusts with the trace data it is shown; never describe the sandbox as making exfiltration impossible. `prelude.js` removes the `RTC*` globals and guards the document (refuses creating, inserting or parsing in nested frames, `<link>` elements and `<script type=speculationrules>`, parsing markup in the real call's document and context and inserting exactly the checked nodes so `<noscript>`/foreign-content parser differentials cannot slip through, refuses `document.write` and `execCommand("insertHTML")`, and a mutation observer removes those from the file's own markup); that is best-effort hardening, not a boundary, and the hostile `probe` fixture in `testdata/lens-e2e` pins each covered path. Installed lenses live in the lens store (`cli/beacon/internal/endpoint/lensstore`, `<BaseDir>/lenses/<id>.lens.html`, 0600 under 0700), written only by `beacon lenses add|remove` (`cli/beacon/cmd/lenses.go`; `add` prints the trust warning and confirms on a TTY, and refuses without one or in CI unless `--yes`); the dashboard rescans it per request and never writes it, and a store lens can never shadow a built-in. `asymptoteobserve.CheckLensFile` is the one file check every installer, server and linter uses. User docs are `docs/concepts/lenses.mdx` and `docs/cli/lenses.mdx`; keep them in sync with the commands and built-in lenses. The authoring loop is `beacon lenses spec` (the spec carried in the binary as `dashboard/lenses/SPEC.md`, kept byte-identical to `spec/lenses/SPEC.md` by a test), `beacon lenses data` (the exact `LensDataV1` for a session), `beacon lenses lint` (`lensstore.Lint`; built-ins must lint clean) and `beacon lenses preview` (the dashboard with `Options.LensFiles`, re-read per reload).
- `beacon-sandbox`: harness that runs a real Claude Code session in a disposable sandbox and checks what Beacon captured. See `beacon-sandbox/AGENTS.md`.
- `packaging`: macOS, Linux, and Windows packaging and deployment assets.
-- `terraform`: customer-run Terraform modules for the cloud side of object storage forwarding. Each module always creates a new, dedicated bucket and never manages an existing one. `aws-s3` creates the hardened bucket, the put-only `beacon-endpoint-writer` IAM user, and optionally `beacon-s3-reader` for Beacon Cloud's S3 data connector, which reads with an access key because ClickPipes on the GCP-hosted ClickHouse rejects IAM roles. The reader comes with an SQS queue per stream (runtime, inventory), each with a dead-letter queue, and the bucket notification that feeds them, so the ClickPipes run in any-order mode; without a queue a pipe reads in key order and skips objects that land behind its cursor, which many Macs uploading on their own timers cause. `gcs` creates the bucket, a `beacon-endpoint-writer` service account with `objectCreator` limited to the prefix by an IAM condition, an optional reader, and with `grant_beacon_read_access` an `objectViewer` grant for Beacon's `beacon-clickpipes-reader` plus a Pub/Sub topic, bucket notification and one subscription per stream matching the managed path's (60 s ack, 7-day retention, never expiring, `objectId` prefix filter). The queue URLs and subscription paths are outputs, ready for the provisioning script's `--queue-url`. Neither module creates keys unless asked (`create_access_keys`, `create_writer_key`), so no secret lands in state by default. Resource names and the `forwarder_settings` output match the packaged `install-forwarder.sh` helpers and the manual guides. docs.beacon.sh (`docs/log-forwarding/{aws-s3,gcs}-terraform.mdx`) documents the bucket and writer only. The Beacon reader, its queues and subscriptions, and the S3 data connector are the enterprise self-hosted path and are documented only in beacon-mdr's docs (docs.asymptotelabs.ai), so keep them out of `docs/`; `/manage/configure-s3-data-connector` redirects there. Both sites pin the module to a release tag; keep them in sync with the variables.
+- `terraform`: customer-run Terraform modules for the cloud side of object storage forwarding. The base modules `aws-s3` and `gcs` always create a new, dedicated bucket (never an existing one) and the writer every Mac's forwarder uses: the put-only `beacon-endpoint-writer` IAM user, or a `beacon-endpoint-writer` service account with `objectCreator` limited to the prefix by an IAM condition. The enterprise self-hosted path, where Asymptote reads that bucket through ClickHouse, is two separate modules that take the base module's `bucket` output: `aws-s3-asymptote-reader` (an `asymptote-s3-reader` IAM user, because ClickPipes on the GCP-hosted ClickHouse rejects IAM roles, plus an SQS queue and dead-letter queue per stream and the bucket notification, which it owns) and `gcs-asymptote-reader` (`objectViewer` for `beacon-clickpipes-reader@asymptote-labs`, a Pub/Sub topic, the notification and a subscription per stream with the managed path's settings). The queues exist because a ClickPipe without one reads in key order and skips objects that land behind its cursor, which Macs uploading on their own timers cause; their URLs and subscription paths are outputs for the provisioning script's `--queue-url`. Keep everything Asymptote-specific out of the base modules. No module creates keys unless asked (`create_writer_access_key`, `create_writer_key`, `create_access_key`), so no secret lands in state by default. Resource names and `forwarder_settings` match the packaged `install-forwarder.sh` helpers and the manual guides. docs.beacon.sh (`docs/log-forwarding/{aws-s3,gcs}-terraform.mdx`) documents the base modules only. The reader modules and the self-hosted setup are documented only on docs.asymptotelabs.ai (beacon-mdr `docs/deployment/self-hosted-{s3,gcs}.mdx`), so keep them out of `docs/`; `/manage/configure-s3-data-connector` redirects there. Both sites pin the modules to a release tag; keep them in sync with the variables.
- `docs`: the published documentation site (docs.asymptotelabs.ai).
Do not recreate or depend on removed `asymptote` mirror trees. Keep new work focused on the Beacon paths above.
@@ -136,8 +136,7 @@ go test ./...
Run the Terraform module tests (the providers are mocked, so no cloud account is needed):
```bash
-cd terraform/aws-s3 && terraform init -backend=false && terraform test
-cd ../gcs && terraform init -backend=false && terraform test
+for module in terraform/*/; do (cd "$module" && terraform init -backend=false && terraform test); done
```
Run the managed runtime plugin/extension checks (each verifies the checked-in copy matches its embedded twin, then runs its tests):
@@ -568,7 +567,7 @@ unpacked through Chrome's developer mode.
- `browser-extension` (Node 22, matrix over `chromium` and `msedge`): `npm run check`, `npm run test:unit`, `npm run build`, a check that the build is a loadable MV3 extension, `npm run lint:firefox` (the Firefox build plus `addons-linter --warnings-as-errors`), and the Playwright replay e2e running the real extension headless in that leg's browser (`BROWSER_CHANNEL`) against a local HTTPS replay server; the HTML report and traces upload on failure. Brave, Opera, Vivaldi and Arc have no Playwright channel; `BROWSER_EXECUTABLE` runs the same e2e against their binaries for the manual checklist in `docs/runtimes/browser-extension-chromium.mdx`.
- `lens-e2e` (ubuntu): runs the real dashboard handler with hostile fixture lenses and the full `rules/` corpus installed (`cli/beacon/internal/endpoint/dashboard/e2eserver`, `testdata/lens-e2e`) and drives the session page in headless Chromium. Every escape from the lens sandbox must be refused, and a lens that throws, stalls or navigates must give way to the full session.
- `beacon-sandbox` (ubuntu): `go build`, `go vet`, and `go test` for the verification harness. Hermetic — no Modal account, no API key, no paid resources.
-- `terraform` (ubuntu): `terraform fmt -check`, `validate`, and `terraform test` (mocked providers) for `terraform/aws-s3` and `terraform/gcs`, and `validate` for each example.
+- `terraform` (ubuntu): `terraform fmt -check`, `validate`, and `terraform test` (mocked providers) for every module under `terraform/`, and `validate` for each example.
- `cross-build` (ubuntu): builds all five release targets and asserts the Linux binaries are statically linked (no `PT_INTERP`).
- `package-smoke` (macOS): `packaging/macos/test-endpoint-scripts.sh`, a `build-pkg.sh` payload check, and `packaging/macos/smoke-endpoint.sh`.
- `linux-systemd-smoke` and `linux-package-smoke` (ubuntu): the real systemd install path, and installing the built `.deb` in a systemd container.
diff --git a/docs/docs.json b/docs/docs.json
index 63aae3ca..96c31735 100644
--- a/docs/docs.json
+++ b/docs/docs.json
@@ -678,7 +678,7 @@
"redirects": [
{
"source": "/manage/configure-s3-data-connector",
- "destination": "https://docs.asymptotelabs.ai/manage/configure-s3-data-connector"
+ "destination": "https://docs.asymptotelabs.ai/deployment/self-hosted-s3"
},
{
"source": "/log-forwarding/s3",
diff --git a/docs/log-forwarding/aws-s3-terraform.mdx b/docs/log-forwarding/aws-s3-terraform.mdx
index 180c1bac..ce8909ca 100644
--- a/docs/log-forwarding/aws-s3-terraform.mdx
+++ b/docs/log-forwarding/aws-s3-terraform.mdx
@@ -160,9 +160,9 @@ Set any of these in the `module "beacon_s3"` block and run `terraform apply` aga
Objects below the prefix are deleted a year after upload. Versioning is on by default, so a deleted or replaced object stays recoverable for `noncurrent_version_expiration_days` (default 30). Set `versioning_enabled = false` to turn versioning off.
</Accordion>
- <Accordion title="Create the access keys with Terraform">
+ <Accordion title="Create the access key with Terraform">
```hcl
- create_access_keys = true
+ create_writer_access_key = true
```
Terraform creates the writer's key. Read the secret with `terraform output -raw writer_secret_access_key` after adding a matching `output` block. The secret is stored in Terraform state, so use this only when your state is encrypted and access is restricted.
@@ -188,7 +188,7 @@ Set any of these in the `module "beacon_s3"` block and run `terraform apply` aga
| --- | --- |
| `bucket_name`, `region`, `prefix`, `storage_class` | The forwarder settings, also together as `forwarder_settings` |
| `writer_user_name`, `writer_user_arn` | The IAM user every Mac uploads as |
-| `writer_access_key_id`, `writer_secret_access_key` | Only with `create_access_keys = true` |
+| `writer_access_key_id`, `writer_secret_access_key` | Only with `create_writer_access_key = true` |
## Rotate the writer key
diff --git a/terraform/README.md b/terraform/README.md
index d70e531e..fc3fc960 100644
--- a/terraform/README.md
+++ b/terraform/README.md
@@ -7,8 +7,13 @@ your cloud credentials.
| Module | Creates | Guide |
| --- | --- | --- |
-| [`aws-s3`](aws-s3) | An S3 bucket, a write-only IAM user, and optionally a read-only user for Beacon Cloud | [AWS S3 with Terraform](https://docs.beacon.sh/log-forwarding/aws-s3-terraform) |
-| [`gcs`](gcs) | A Cloud Storage bucket, a create-only service account, and optionally read access for Beacon Cloud | [Google Cloud Storage with Terraform](https://docs.beacon.sh/log-forwarding/gcs-terraform) |
+| [`aws-s3`](aws-s3) | A new S3 bucket and a write-only IAM user | [AWS S3 with Terraform](https://docs.beacon.sh/log-forwarding/aws-s3-terraform) |
+| [`gcs`](gcs) | A new Cloud Storage bucket and a create-only service account | [Google Cloud Storage with Terraform](https://docs.beacon.sh/log-forwarding/gcs-terraform) |
+| [`aws-s3-asymptote-reader`](aws-s3-asymptote-reader) | Read-only access for Asymptote to an `aws-s3` bucket, with new-object SQS queues | [Asymptote with your own S3 bucket](https://docs.asymptotelabs.ai/deployment/self-hosted-s3) |
+| [`gcs-asymptote-reader`](gcs-asymptote-reader) | Read access for Asymptote to a `gcs` bucket, with new-object Pub/Sub subscriptions | [Asymptote with your own Cloud Storage bucket](https://docs.asymptotelabs.ai/deployment/self-hosted-gcs) |
+
+The reader modules take the base module's `bucket` output, so the bucket is always one the
+base module created.
Use a module from your own configuration and pin it to a Beacon release:
@@ -21,19 +26,20 @@ module "beacon_s3" {
}
```
-Neither module creates keys unless you ask it to (`create_access_keys`,
-`create_writer_key`), so by default no secret is written to Terraform state. The
-guides show how to create the key with the AWS CLI or gcloud instead.
+No module creates keys unless you ask it to (`create_writer_access_key`,
+`create_writer_key`, `create_access_key`), so by default no secret is written to
+Terraform state. The guides show how to create the keys with the AWS CLI or gcloud
+instead.
## Development
The tests mock the cloud providers, so they need no account:
```bash
-cd terraform/aws-s3 # or terraform/gcs
+cd terraform/aws-s3 # or any other module directory
terraform init -backend=false
terraform test
```
-CI runs `terraform fmt -check`, `validate` and `test` for both modules, and
+CI runs `terraform fmt -check`, `validate` and `test` for every module, and
`validate` for each example.
diff --git a/terraform/aws-s3-asymptote-reader/README.md b/terraform/aws-s3-asymptote-reader/README.md
new file mode 100644
index 00000000..d17c7aa7
--- /dev/null
+++ b/terraform/aws-s3-asymptote-reader/README.md
@@ -0,0 +1,33 @@
+# Asymptote reader for a Beacon S3 bucket
+
+Gives Asymptote read-only access to a Beacon bucket created by the [`aws-s3`](../aws-s3)
+module, for customers who keep Beacon telemetry in their own bucket and use Asymptote's
+dashboard:
+
+- `asymptote-s3-reader`, an IAM user that can list the bucket, read objects below the
+ Beacon prefix, and consume the two queues below;
+- an SQS queue per stream (`runtime`, `inventory`), each with a dead-letter queue, that S3
+ notifies of every new object, so Asymptote loads objects in any order;
+- the bucket's notification configuration, which this module owns.
+
+```hcl
+module "beacon_s3" {
+ source = "github.com/asymptote-labs/agent-beacon//terraform/aws-s3?ref=<release tag>"
+
+ bucket_name = "acme-beacon-telemetry"
+ prefix = "beacon-prod"
+}
+
+module "asymptote_reader" {
+ source = "github.com/asymptote-labs/agent-beacon//terraform/aws-s3-asymptote-reader?ref=<release tag>"
+
+ bucket = module.beacon_s3.bucket
+}
+```
+
+Use a provider in the bucket's region: S3 can only notify queues in its own region. Create
+the reader's access key with the AWS CLI, so its secret never enters Terraform state, and
+send Asymptote the key with the `source_url` and `queue_urls` outputs.
+
+The full walkthrough is on the Asymptote docs:
+[Asymptote with your own S3 bucket](https://docs.asymptotelabs.ai/deployment/self-hosted-s3).
diff --git a/terraform/aws-s3-asymptote-reader/examples/basic/main.tf b/terraform/aws-s3-asymptote-reader/examples/basic/main.tf
new file mode 100644
index 00000000..3f4106a7
--- /dev/null
+++ b/terraform/aws-s3-asymptote-reader/examples/basic/main.tf
@@ -0,0 +1,83 @@
+# A Beacon bucket that Asymptote can read: the aws-s3 module creates the bucket and the writer,
+# and this module adds Asymptote's read-only user and the new-object queues.
+#
+# terraform init
+# terraform apply -var region=us-west-2 -var bucket_name=<globally-unique-name>
+# aws iam create-access-key --user-name "$(terraform output -raw writer_user_name)"
+# aws iam create-access-key --user-name "$(terraform output -raw asymptote_reader_user_name)"
+#
+# Outside this repository, point both sources at a release:
+# source = "github.com/asymptote-labs/agent-beacon//terraform/aws-s3?ref=<release tag>"
+# source = "github.com/asymptote-labs/agent-beacon//terraform/aws-s3-asymptote-reader?ref=<release tag>"
+
+terraform {
+ required_version = ">= 1.5.0"
+
+ required_providers {
+ aws = {
+ source = "hashicorp/aws"
+ version = ">= 5.0"
+ }
+ }
+}
+
+variable "region" {
+ description = "AWS region for the bucket and the queues."
+ type = string
+}
+
+variable "bucket_name" {
+ description = "Globally unique name for the new bucket."
+ type = string
+}
+
+variable "prefix" {
+ description = "Root prefix for Beacon objects."
+ type = string
+ default = "beacon-prod"
+}
+
+provider "aws" {
+ region = var.region
+}
+
+module "beacon_s3" {
+ source = "../../../aws-s3"
+
+ bucket_name = var.bucket_name
+ prefix = var.prefix
+}
+
+module "asymptote_reader" {
+ source = "../.."
+
+ bucket = module.beacon_s3.bucket
+}
+
+output "bucket_name" {
+ value = module.beacon_s3.bucket_name
+}
+
+output "region" {
+ value = module.beacon_s3.region
+}
+
+output "prefix" {
+ value = module.beacon_s3.prefix
+}
+
+output "writer_user_name" {
+ value = module.beacon_s3.writer_user_name
+}
+
+output "asymptote_reader_user_name" {
+ value = module.asymptote_reader.user_name
+}
+
+output "asymptote_source_url" {
+ value = module.asymptote_reader.source_url
+}
+
+output "asymptote_queue_urls" {
+ value = module.asymptote_reader.queue_urls
+}
diff --git a/terraform/aws-s3-asymptote-reader/main.tf b/terraform/aws-s3-asymptote-reader/main.tf
new file mode 100644
index 00000000..a1422ec7
--- /dev/null
+++ b/terraform/aws-s3-asymptote-reader/main.tf
@@ -0,0 +1,160 @@
+# Read-only access for Asymptote to a Beacon bucket created by the aws-s3 module.
+#
+# Asymptote's ClickPipes read the bucket in "any order" mode: S3 announces every new object on an
+# SQS queue, and the pipe loads what the queue names. Without the queues a pipe lists objects in
+# key order and skips any that sort before ones it has already loaded, which happens whenever Macs
+# upload out of step.
+
+data "aws_partition" "current" {}
+
+data "aws_caller_identity" "current" {}
+
+locals {
+ streams = toset(["runtime", "inventory"])
+
+ objects_arn = "${var.bucket.arn}/${var.bucket.prefix}/*"
+
+ # SQS names allow letters, digits, hyphens and underscores; IAM user names allow a few more.
+ queue_base = replace(var.user_name, "/[^A-Za-z0-9_-]/", "-")
+
+ # S3's certificate for virtual-hosted URLs covers one label, so a dotted bucket name needs path style.
+ s3_host = "s3.${var.bucket.region}.${data.aws_partition.current.dns_suffix}"
+ source_url = (
+ strcontains(var.bucket.name, ".")
+ ? "https://${local.s3_host}/${var.bucket.name}/${var.bucket.prefix}"
+ : "https://${var.bucket.name}.${local.s3_host}/${var.bucket.prefix}"
+ )
+
+ statements = concat(
+ [
+ {
+ Sid = "ListBeaconBucket"
+ Effect = "Allow"
+ Action = ["s3:ListBucket", "s3:GetBucketLocation"]
+ Resource = var.bucket.arn
+ },
+ {
+ Sid = "ReadBeaconObjects"
+ Effect = "Allow"
+ Action = "s3:GetObject"
+ Resource = local.objects_arn
+ },
+ # The four actions ClickPipes needs to consume the new-object queues.
+ {
+ Sid = "ConsumeBeaconNotifications"
+ Effect = "Allow"
+ Action = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]
+ Resource = [for queue in aws_sqs_queue.reader : queue.arn]
+ },
+ {
+ Sid = "FindBeaconQueues"
+ Effect = "Allow"
+ Action = "sqs:ListQueues"
+ Resource = "*"
+ },
+ ],
+ var.bucket.kms_key_arn == null ? [] : [
+ {
+ Sid = "DecryptBeaconObjects"
+ Effect = "Allow"
+ Action = "kms:Decrypt"
+ Resource = var.bucket.kms_key_arn
+ },
+ ],
+ )
+}
+
+resource "aws_iam_user" "reader" {
+ name = var.user_name
+ path = var.iam_path
+ permissions_boundary = var.permissions_boundary_arn
+ tags = var.tags
+}
+
+resource "aws_iam_user_policy" "reader" {
+ name = "BeaconTelemetryReadOnly"
+ user = aws_iam_user.reader.name
+ policy = jsonencode({
+ Version = "2012-10-17"
+ Statement = local.statements
+ })
+}
+
+resource "aws_iam_access_key" "reader" {
+ count = var.create_access_key ? 1 : 0
+
+ user = aws_iam_user.reader.name
+}
+
+# A message that fails ten deliveries moves here instead of being dropped, so it can be redriven.
+resource "aws_sqs_queue" "dead_letter" {
+ for_each = local.streams
+
+ name = "${local.queue_base}-${each.key}-dlq"
+ message_retention_seconds = 1209600
+ sqs_managed_sse_enabled = true
+ tags = var.tags
+}
+
+resource "aws_sqs_queue" "reader" {
+ for_each = local.streams
+
+ name = "${local.queue_base}-${each.key}"
+ message_retention_seconds = 1209600
+ visibility_timeout_seconds = 300
+ sqs_managed_sse_enabled = true
+ tags = var.tags
+
+ redrive_policy = jsonencode({
+ deadLetterTargetArn = aws_sqs_queue.dead_letter[each.key].arn
+ maxReceiveCount = 10
+ })
+}
+
+resource "aws_sqs_queue_policy" "reader" {
+ for_each = local.streams
+
+ queue_url = aws_sqs_queue.reader[each.key].id
+ policy = jsonencode({
+ Version = "2012-10-17"
+ Statement = [
+ {
+ Sid = "AllowBucketNotifications"
+ Effect = "Allow"
+ Principal = { Service = "s3.amazonaws.com" }
+ Action = "sqs:SendMessage"
+ Resource = aws_sqs_queue.reader[each.key].arn
+ Condition = {
+ ArnEquals = { "aws:SourceArn" = var.bucket.arn }
+ StringEquals = { "aws:SourceAccount" = data.aws_caller_identity.current.account_id }
+ }
+ },
+ ]
+ })
+}
+
+# S3 keeps one notification configuration per bucket. The aws-s3 module never sets one, so this
+# module owns it.
+resource "aws_s3_bucket_notification" "this" {
+ bucket = var.bucket.name
+
+ dynamic "queue" {
+ for_each = local.streams
+ content {
+ id = "beacon-${queue.key}"
+ queue_arn = aws_sqs_queue.reader[queue.key].arn
+ events = ["s3:ObjectCreated:*"]
+ filter_prefix = "${var.bucket.prefix}/${queue.key}/"
+ }
+ }
+
+ lifecycle {
+ precondition {
+ condition = split(":", aws_sqs_queue.reader["runtime"].arn)[3] == var.bucket.region
+ error_message = "S3 can only notify queues in the bucket's own region. Use a provider in ${var.bucket.region} for this module."
+ }
+ }
+
+ # S3 checks that it may send to each queue when the configuration is written.
+ depends_on = [aws_sqs_queue_policy.reader]
+}
diff --git a/terraform/aws-s3-asymptote-reader/outputs.tf b/terraform/aws-s3-asymptote-reader/outputs.tf
new file mode 100644
index 00000000..1899cf82
--- /dev/null
+++ b/terraform/aws-s3-asymptote-reader/outputs.tf
@@ -0,0 +1,30 @@
+output "user_name" {
+ description = "Read-only IAM user Asymptote reads the bucket with. Create its access key with: aws iam create-access-key --user-name <this value>"
+ value = aws_iam_user.reader.name
+}
+
+output "user_arn" {
+ description = "ARN of the reader IAM user."
+ value = aws_iam_user.reader.arn
+}
+
+output "access_key_id" {
+ description = "Reader access key ID, when create_access_key = true."
+ value = try(aws_iam_access_key.reader[0].id, null)
+}
+
+output "secret_access_key" {
+ description = "Reader secret access key, when create_access_key = true."
+ value = try(aws_iam_access_key.reader[0].secret, null)
+ sensitive = true
+}
+
+output "source_url" {
+ description = "The bucket location to send Asymptote."
+ value = local.source_url
+}
+
+output "queue_urls" {
+ description = "The new-object queues to send Asymptote, by stream."
+ value = { for stream, queue in aws_sqs_queue.reader : stream => queue.url }
+}
diff --git a/terraform/aws-s3-asymptote-reader/tests/module.tftest.hcl b/terraform/aws-s3-asymptote-reader/tests/module.tftest.hcl
new file mode 100644
index 00000000..47199026
--- /dev/null
+++ b/terraform/aws-s3-asymptote-reader/tests/module.tftest.hcl
@@ -0,0 +1,217 @@
+# Hermetic: the AWS provider is mocked, so these runs need no account or credentials.
+
+mock_provider "aws" {
+ mock_data "aws_partition" {
+ defaults = {
+ partition = "aws"
+ dns_suffix = "amazonaws.com"
+ }
+ }
+
+ mock_data "aws_caller_identity" {
+ defaults = {
+ account_id = "111122223333"
+ }
+ }
+
+ mock_resource "aws_sqs_queue" {
+ defaults = {
+ arn = "arn:aws:sqs:us-west-2:111122223333:mock"
+ }
+ }
+}
+
+variables {
+ bucket = {
+ name = "acme-beacon-telemetry"
+ arn = "arn:aws:s3:::acme-beacon-telemetry"
+ region = "us-west-2"
+ prefix = "beacon-prod"
+ }
+}
+
+run "reader_reads_only_the_prefix_and_its_queues" {
+ command = apply
+
+ assert {
+ condition = jsondecode(aws_iam_user_policy.reader.policy).Statement == [
+ {
+ Sid = "ListBeaconBucket"
+ Effect = "Allow"
+ Action = ["s3:ListBucket", "s3:GetBucketLocation"]
+ Resource = "arn:aws:s3:::acme-beacon-telemetry"
+ },
+ {
+ Sid = "ReadBeaconObjects"
+ Effect = "Allow"
+ Action = "s3:GetObject"
+ Resource = "arn:aws:s3:::acme-beacon-telemetry/beacon-prod/*"
+ },
+ {
+ Sid = "ConsumeBeaconNotifications"
+ Effect = "Allow"
+ Action = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]
+ Resource = [aws_sqs_queue.reader["inventory"].arn, aws_sqs_queue.reader["runtime"].arn]
+ },
+ {
+ Sid = "FindBeaconQueues"
+ Effect = "Allow"
+ Action = "sqs:ListQueues"
+ Resource = "*"
+ },
+ ]
+ error_message = "The reader may list the bucket, read only the Beacon prefix, and consume its two queues."
+ }
+
+ assert {
+ condition = aws_iam_user.reader.name == "asymptote-s3-reader" && aws_iam_user_policy.reader.name == "BeaconTelemetryReadOnly"
+ error_message = "The reader has the documented names."
+ }
+
+ assert {
+ condition = length(aws_iam_access_key.reader) == 0
+ error_message = "The access key, whose secret would land in state, must be opt-in."
+ }
+}
+
+run "each_stream_gets_a_queue" {
+ command = apply
+
+ assert {
+ condition = (
+ aws_sqs_queue.reader["runtime"].name == "asymptote-s3-reader-runtime"
+ && aws_sqs_queue.reader["inventory"].name == "asymptote-s3-reader-inventory"
+ && aws_sqs_queue.dead_letter["runtime"].name == "asymptote-s3-reader-runtime-dlq"
+ && aws_sqs_queue.dead_letter["inventory"].name == "asymptote-s3-reader-inventory-dlq"
+ )
+ error_message = "Each stream gets a queue and a dead-letter queue named after the reader."
+ }
+
+ assert {
+ condition = alltrue([
+ for stream in ["runtime", "inventory"] :
+ aws_sqs_queue.reader[stream].sqs_managed_sse_enabled
+ && aws_sqs_queue.reader[stream].message_retention_seconds == 1209600
+ && aws_sqs_queue.reader[stream].visibility_timeout_seconds == 300
+ && jsondecode(aws_sqs_queue.reader[stream].redrive_policy) == {
+ deadLetterTargetArn = aws_sqs_queue.dead_letter[stream].arn
+ maxReceiveCount = 10
+ }
+ ])
+ error_message = "Queues are encrypted, keep messages 14 days, and move a message to the dead-letter queue after ten failed deliveries."
+ }
+
+ assert {
+ condition = alltrue([
+ for stream in ["runtime", "inventory"] :
+ jsondecode(aws_sqs_queue_policy.reader[stream].policy).Statement == [
+ {
+ Sid = "AllowBucketNotifications"
+ Effect = "Allow"
+ Principal = { Service = "s3.amazonaws.com" }
+ Action = "sqs:SendMessage"
+ Resource = aws_sqs_queue.reader[stream].arn
+ Condition = {
+ ArnEquals = { "aws:SourceArn" = "arn:aws:s3:::acme-beacon-telemetry" }
+ StringEquals = { "aws:SourceAccount" = "111122223333" }
+ }
+ },
+ ]
+ ])
+ error_message = "Only this bucket, in this account, may send to the queues."
+ }
+
+ assert {
+ condition = aws_s3_bucket_notification.this.bucket == "acme-beacon-telemetry" && {
+ for queue in aws_s3_bucket_notification.this.queue : queue.filter_prefix => {
+ arn = queue.queue_arn
+ events = queue.events
+ }
+ } == {
+ "beacon-prod/runtime/" = { arn = aws_sqs_queue.reader["runtime"].arn, events = toset(["s3:ObjectCreated:*"]) }
+ "beacon-prod/inventory/" = { arn = aws_sqs_queue.reader["inventory"].arn, events = toset(["s3:ObjectCreated:*"]) }
+ }
+ error_message = "Every new object below each stream's prefix is announced on that stream's queue."
+ }
+
+ assert {
+ condition = output.queue_urls == {
+ runtime = aws_sqs_queue.reader["runtime"].url
+ inventory = aws_sqs_queue.reader["inventory"].url
+ }
+ error_message = "queue_urls gives the provisioning script's --queue-url values."
+ }
+
+ assert {
+ condition = output.source_url == "https://acme-beacon-telemetry.s3.us-west-2.amazonaws.com/beacon-prod"
+ error_message = "The source URL is the bucket's regional endpoint plus the prefix."
+ }
+}
+
+run "dotted_bucket_names_get_a_path_style_url" {
+ command = apply
+
+ variables {
+ bucket = {
+ name = "acme.beacon.telemetry"
+ arn = "arn:aws:s3:::acme.beacon.telemetry"
+ region = "us-west-2"
+ prefix = "beacon-prod"
+ }
+ }
+
+ assert {
+ condition = output.source_url == "https://s3.us-west-2.amazonaws.com/acme.beacon.telemetry/beacon-prod"
+ error_message = "A dotted bucket name fails TLS on a virtual-hosted URL, so the URL must be path style."
+ }
+}
+
+run "kms_bucket_needs_decrypt" {
+ command = apply
+
+ variables {
+ bucket = {
+ name = "acme-beacon-telemetry"
+ arn = "arn:aws:s3:::acme-beacon-telemetry"
+ region = "us-west-2"
+ prefix = "beacon-prod"
+ kms_key_arn = "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
+ }
+ }
+
+ assert {
+ condition = contains(jsondecode(aws_iam_user_policy.reader.policy).Statement, {
+ Sid = "DecryptBeaconObjects"
+ Effect = "Allow"
+ Action = "kms:Decrypt"
+ Resource = "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
+ })
+ error_message = "The reader needs kms:Decrypt to read an SSE-KMS bucket."
+ }
+}
+
+run "queue_names_drop_characters_sqs_refuses" {
+ command = plan
+
+ variables {
+ user_name = "asymptote.reader@acme"
+ }
+
+ assert {
+ condition = aws_sqs_queue.reader["runtime"].name == "asymptote-reader-acme-runtime"
+ error_message = "IAM user names allow . and @, which SQS names do not."
+ }
+}
+
+run "access_key_when_asked" {
+ command = plan
+
+ variables {
+ create_access_key = true
+ }
+
+ assert {
+ condition = length(aws_iam_access_key.reader) == 1
+ error_message = "create_access_key creates one key."
+ }
+}
diff --git a/terraform/aws-s3-asymptote-reader/tests/partition.tftest.hcl b/terraform/aws-s3-asymptote-reader/tests/partition.tftest.hcl
new file mode 100644
index 00000000..59e5c97b
--- /dev/null
+++ b/terraform/aws-s3-asymptote-reader/tests/partition.tftest.hcl
@@ -0,0 +1,34 @@
+# The module builds URLs from the partition, so it also works in AWS China and GovCloud.
+
+mock_provider "aws" {
+ mock_data "aws_partition" {
+ defaults = {
+ partition = "aws-cn"
+ dns_suffix = "amazonaws.com.cn"
+ }
+ }
+
+ mock_resource "aws_sqs_queue" {
+ defaults = {
+ arn = "arn:aws-cn:sqs:cn-north-1:111122223333:mock"
+ }
+ }
+}
+
+variables {
+ bucket = {
+ name = "acme-beacon-telemetry"
+ arn = "arn:aws-cn:s3:::acme-beacon-telemetry"
+ region = "cn-north-1"
+ prefix = "beacon-prod"
+ }
+}
+
+run "source_url_follows_the_partition" {
+ command = apply
+
+ assert {
+ condition = output.source_url == "https://acme-beacon-telemetry.s3.cn-north-1.amazonaws.com.cn/beacon-prod"
+ error_message = "The source URL uses the partition's DNS suffix."
+ }
+}
diff --git a/terraform/aws-s3-asymptote-reader/tests/region.tftest.hcl b/terraform/aws-s3-asymptote-reader/tests/region.tftest.hcl
new file mode 100644
index 00000000..7962e7be
--- /dev/null
+++ b/terraform/aws-s3-asymptote-reader/tests/region.tftest.hcl
@@ -0,0 +1,24 @@
+# S3 can only notify queues in the bucket's own region.
+
+mock_provider "aws" {
+ mock_resource "aws_sqs_queue" {
+ defaults = {
+ arn = "arn:aws:sqs:eu-west-1:111122223333:mock"
+ }
+ }
+}
+
+variables {
+ bucket = {
+ name = "acme-beacon-telemetry"
+ arn = "arn:aws:s3:::acme-beacon-telemetry"
+ region = "us-west-2"
+ prefix = "beacon-prod"
+ }
+}
+
+run "refuses_queues_in_another_region" {
+ command = apply
+
+ expect_failures = [aws_s3_bucket_notification.this]
+}
diff --git a/terraform/aws-s3-asymptote-reader/variables.tf b/terraform/aws-s3-asymptote-reader/variables.tf
new file mode 100644
index 00000000..abc27484
--- /dev/null
+++ b/terraform/aws-s3-asymptote-reader/variables.tf
@@ -0,0 +1,45 @@
+variable "bucket" {
+ description = "The Beacon bucket to read, as the aws-s3 module's bucket output: bucket = module.beacon_s3.bucket."
+ type = object({
+ name = string
+ arn = string
+ region = string
+ prefix = string
+ kms_key_arn = optional(string)
+ })
+}
+
+variable "user_name" {
+ description = "Name of the read-only IAM user Asymptote reads the bucket with. Its queues are named after it: <name>-runtime and <name>-inventory, each with a -dlq."
+ type = string
+ default = "asymptote-s3-reader"
+}
+
+variable "create_access_key" {
+ description = "Also create the reader's access key. The secret key is then stored in Terraform state, so leave this false unless your state is encrypted and access-controlled, and create the key with the AWS CLI instead."
+ type = bool
+ default = false
+}
+
+variable "iam_path" {
+ description = "Path for the IAM user."
+ type = string
+ default = "/"
+
+ validation {
+ condition = can(regex("^/([A-Za-z0-9+=,.@_-]+/)*$", var.iam_path))
+ error_message = "iam_path must begin and end with a slash, such as / or /beacon/."
+ }
+}
+
+variable "permissions_boundary_arn" {
+ description = "ARN of a permissions boundary policy to attach to the IAM user, if your organization requires one."
+ type = string
+ default = null
+}
+
+variable "tags" {
+ description = "Tags to apply to every resource the module creates."
+ type = map(string)
+ default = {}
+}
diff --git a/terraform/aws-s3-asymptote-reader/versions.tf b/terraform/aws-s3-asymptote-reader/versions.tf
new file mode 100644
index 00000000..b088b0f1
--- /dev/null
+++ b/terraform/aws-s3-asymptote-reader/versions.tf
@@ -0,0 +1,10 @@
+terraform {
+ required_version = ">= 1.5.0"
+
+ required_providers {
+ aws = {
+ source = "hashicorp/aws"
+ version = ">= 5.0"
+ }
+ }
+}
diff --git a/terraform/aws-s3/README.md b/terraform/aws-s3/README.md
index 6dab9139..89e3abe0 100644
--- a/terraform/aws-s3/README.md
+++ b/terraform/aws-s3/README.md
@@ -7,9 +7,8 @@ Creates the AWS side of Beacon's S3 forwarding:
lifecycle rule scoped to the Beacon prefix;
- `beacon-endpoint-writer`, an IAM user that can only `s3:PutObject` below the prefix
(plus `s3:ListBucket` for the forwarder's startup health check);
-- optionally `beacon-s3-reader`, a read-only IAM user for Beacon Cloud's S3 data
- connector (`create_beacon_reader = true`), with an SQS queue per stream that S3
- notifies of every new object, so Beacon loads objects in any order.
+- optionally the writer's access key (`create_writer_access_key = true`), though the
+ docs create it with the AWS CLI so its secret stays out of Terraform state.
```hcl
provider "aws" {
diff --git a/terraform/aws-s3/iam.tf b/terraform/aws-s3/iam.tf
index 6652bf6b..4725fc53 100644
--- a/terraform/aws-s3/iam.tf
+++ b/terraform/aws-s3/iam.tf
@@ -25,44 +25,6 @@ locals {
},
],
)
-
- reader_statements = concat(
- [
- {
- Sid = "ListBeaconBucket"
- Effect = "Allow"
- Action = ["s3:ListBucket", "s3:GetBucketLocation"]
- Resource = local.bucket_arn
- },
- {
- Sid = "ReadBeaconObjects"
- Effect = "Allow"
- Action = "s3:GetObject"
- Resource = local.beacon_objects_arn
- },
- # The four actions ClickPipes needs to consume the new-object queues.
- {
- Sid = "ConsumeBeaconNotifications"
- Effect = "Allow"
- Action = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]
- Resource = [for queue in aws_sqs_queue.beacon_reader : queue.arn]
- },
- {
- Sid = "FindBeaconQueues"
- Effect = "Allow"
- Action = "sqs:ListQueues"
- Resource = "*"
- },
- ],
- var.kms_key_arn == null ? [] : [
- {
- Sid = "DecryptBeaconObjects"
- Effect = "Allow"
- Action = "kms:Decrypt"
- Resource = var.kms_key_arn
- },
- ],
- )
}
# The identity every Mac's forwarder uses. It can create objects below the Beacon prefix, and it
@@ -84,34 +46,7 @@ resource "aws_iam_user_policy" "writer" {
}
resource "aws_iam_access_key" "writer" {
- count = var.create_access_keys ? 1 : 0
+ count = var.create_writer_access_key ? 1 : 0
user = aws_iam_user.writer.name
}
-
-# Beacon Cloud's S3 data connector reads with an access key; it cannot assume a role in your account.
-resource "aws_iam_user" "beacon_reader" {
- count = var.create_beacon_reader ? 1 : 0
-
- name = var.beacon_reader_user_name
- path = var.iam_path
- permissions_boundary = var.permissions_boundary_arn
- tags = var.tags
-}
-
-resource "aws_iam_user_policy" "beacon_reader" {
- count = var.create_beacon_reader ? 1 : 0
-
- name = "BeaconTelemetryReadOnly"
- user = aws_iam_user.beacon_reader[0].name
- policy = jsonencode({
- Version = "2012-10-17"
- Statement = local.reader_statements
- })
-}
-
-resource "aws_iam_access_key" "beacon_reader" {
- count = var.create_beacon_reader && var.create_access_keys ? 1 : 0
-
- user = aws_iam_user.beacon_reader[0].name
-}
diff --git a/terraform/aws-s3/main.tf b/terraform/aws-s3/main.tf
index aff4c1f9..244b0e56 100644
--- a/terraform/aws-s3/main.tf
+++ b/terraform/aws-s3/main.tf
@@ -1,20 +1,10 @@
data "aws_partition" "current" {}
-data "aws_caller_identity" "current" {}
-
locals {
bucket_arn = "arn:${data.aws_partition.current.partition}:s3:::${var.bucket_name}"
# Beacon writes <prefix>/runtime/date=YYYY-MM-DD/... and <prefix>/inventory/date=YYYY-MM-DD/...
beacon_objects_arn = "${local.bucket_arn}/${var.prefix}/*"
-
- # S3's certificate for virtual-hosted URLs covers one label, so a dotted bucket name needs path style.
- s3_host = "s3.${aws_s3_bucket.this.region}.${data.aws_partition.current.dns_suffix}"
- beacon_reader_source_url = (
- strcontains(var.bucket_name, ".")
- ? "https://${local.s3_host}/${var.bucket_name}/${var.prefix}"
- : "https://${var.bucket_name}.${local.s3_host}/${var.prefix}"
- )
}
# A dedicated bucket for Beacon telemetry, always created by the module.
diff --git a/terraform/aws-s3/notifications.tf b/terraform/aws-s3/notifications.tf
deleted file mode 100644
index d9c0b529..00000000
--- a/terraform/aws-s3/notifications.tf
+++ /dev/null
@@ -1,78 +0,0 @@
-# Beacon Cloud's ClickPipes read the bucket in "any order" mode: S3 announces every new object on
-# an SQS queue, and the pipe loads what the queue names. Without the queues a pipe lists objects in
-# key order and skips any that sort before ones it has already loaded, which happens whenever Macs
-# upload out of step.
-
-locals {
- reader_streams = var.create_beacon_reader ? toset(["runtime", "inventory"]) : toset([])
-
- # SQS names allow letters, digits, hyphens and underscores; IAM user names allow a few more.
- reader_queue_base = replace(var.beacon_reader_user_name, "/[^A-Za-z0-9_-]/", "-")
-}
-
-# A message that fails ten deliveries moves here instead of being dropped, so it can be redriven.
-resource "aws_sqs_queue" "beacon_reader_dead_letter" {
- for_each = local.reader_streams
-
- name = "${local.reader_queue_base}-${each.key}-dlq"
- message_retention_seconds = 1209600
- sqs_managed_sse_enabled = true
- tags = var.tags
-}
-
-resource "aws_sqs_queue" "beacon_reader" {
- for_each = local.reader_streams
-
- name = "${local.reader_queue_base}-${each.key}"
- message_retention_seconds = 1209600
- visibility_timeout_seconds = 300
- sqs_managed_sse_enabled = true
- tags = var.tags
-
- redrive_policy = jsonencode({
- deadLetterTargetArn = aws_sqs_queue.beacon_reader_dead_letter[each.key].arn
- maxReceiveCount = 10
- })
-}
-
-resource "aws_sqs_queue_policy" "beacon_reader" {
- for_each = local.reader_streams
-
- queue_url = aws_sqs_queue.beacon_reader[each.key].id
- policy = jsonencode({
- Version = "2012-10-17"
- Statement = [
- {
- Sid = "AllowBucketNotifications"
- Effect = "Allow"
- Principal = { Service = "s3.amazonaws.com" }
- Action = "sqs:SendMessage"
- Resource = aws_sqs_queue.beacon_reader[each.key].arn
- Condition = {
- ArnEquals = { "aws:SourceArn" = local.bucket_arn }
- StringEquals = { "aws:SourceAccount" = data.aws_caller_identity.current.account_id }
- }
- },
- ]
- })
-}
-
-# S3 keeps one notification configuration per bucket. The module owns this bucket, so it owns that too.
-resource "aws_s3_bucket_notification" "this" {
- count = var.create_beacon_reader ? 1 : 0
-
- bucket = aws_s3_bucket.this.id
-
- dynamic "queue" {
- for_each = local.reader_streams
- content {
- id = "beacon-${queue.key}"
- queue_arn = aws_sqs_queue.beacon_reader[queue.key].arn
- events = ["s3:ObjectCreated:*"]
- filter_prefix = "${var.prefix}/${queue.key}/"
- }
- }
-
- # S3 checks that it may send to each queue when the configuration is written.
- depends_on = [aws_sqs_queue_policy.beacon_reader]
-}
diff --git a/terraform/aws-s3/outputs.tf b/terraform/aws-s3/outputs.tf
index 6c8b856d..ce6a80b0 100644
--- a/terraform/aws-s3/outputs.tf
+++ b/terraform/aws-s3/outputs.tf
@@ -18,6 +18,17 @@ output "prefix" {
value = var.prefix
}
+output "bucket" {
+ description = "The bucket as one object, for the aws-s3-asymptote-reader module: bucket = module.beacon_s3.bucket."
+ value = {
+ name = aws_s3_bucket.this.bucket
+ arn = local.bucket_arn
+ region = aws_s3_bucket.this.region
+ prefix = var.prefix
+ kms_key_arn = var.kms_key_arn
+ }
+}
+
output "storage_class" {
description = "Storage class the forwarder writes objects with (BEACON_S3_STORAGE_CLASS)."
value = var.storage_class
@@ -44,43 +55,12 @@ output "writer_user_arn" {
}
output "writer_access_key_id" {
- description = "Writer access key ID, when create_access_keys = true."
+ description = "Writer access key ID, when create_writer_access_key = true."
value = try(aws_iam_access_key.writer[0].id, null)
}
output "writer_secret_access_key" {
- description = "Writer secret access key, when create_access_keys = true. Read it with terraform output -raw writer_secret_access_key."
+ description = "Writer secret access key, when create_writer_access_key = true. Read it with terraform output -raw writer_secret_access_key."
value = try(aws_iam_access_key.writer[0].secret, null)
sensitive = true
}
-
-output "beacon_reader_user_name" {
- description = "Read-only IAM user for Beacon Cloud's S3 data connector, when create_beacon_reader = true."
- value = try(aws_iam_user.beacon_reader[0].name, null)
-}
-
-output "beacon_reader_user_arn" {
- description = "ARN of the Beacon reader IAM user, when create_beacon_reader = true."
- value = try(aws_iam_user.beacon_reader[0].arn, null)
-}
-
-output "beacon_reader_access_key_id" {
- description = "Beacon reader access key ID, when create_beacon_reader and create_access_keys are both true."
- value = try(aws_iam_access_key.beacon_reader[0].id, null)
-}
-
-output "beacon_reader_secret_access_key" {
- description = "Beacon reader secret access key, when create_beacon_reader and create_access_keys are both true."
- value = try(aws_iam_access_key.beacon_reader[0].secret, null)
- sensitive = true
-}
-
-output "beacon_reader_source_url" {
- description = "The location to give Beacon for the S3 data connector."
- value = local.beacon_reader_source_url
-}
-
-output "beacon_reader_queue_urls" {
- description = "The new-object queues to give Beacon for the S3 data connector, by stream, when create_beacon_reader = true."
- value = var.create_beacon_reader ? { for stream, queue in aws_sqs_queue.beacon_reader : stream => queue.url } : null
-}
diff --git a/terraform/aws-s3/tests/module.tftest.hcl b/terraform/aws-s3/tests/module.tftest.hcl
index 6c4852e5..804ae734 100644
--- a/terraform/aws-s3/tests/module.tftest.hcl
+++ b/terraform/aws-s3/tests/module.tftest.hcl
@@ -8,12 +8,6 @@ mock_provider "aws" {
}
}
- mock_data "aws_caller_identity" {
- defaults = {
- account_id = "111122223333"
- }
- }
-
mock_resource "aws_s3_bucket" {
defaults = {
bucket = "acme-beacon-telemetry"
@@ -58,13 +52,8 @@ run "no_access_keys_by_default" {
command = plan
assert {
- condition = length(aws_iam_access_key.writer) == 0 && length(aws_iam_access_key.beacon_reader) == 0
- error_message = "Access keys, whose secrets would land in state, must be opt-in."
- }
-
- assert {
- condition = length(aws_iam_user.beacon_reader) == 0
- error_message = "The Beacon reader must be opt-in."
+ condition = length(aws_iam_access_key.writer) == 0
+ error_message = "The access key, whose secret would land in state, must be opt-in."
}
}
@@ -145,30 +134,22 @@ run "forwarder_settings_match_the_helper" {
}
assert {
- condition = output.beacon_reader_source_url == "https://acme-beacon-telemetry.s3.us-west-2.amazonaws.com/beacon-prod"
- error_message = "The data connector URL is the bucket's regional endpoint plus the prefix."
- }
-}
-
-run "source_url_uses_path_style_for_dotted_bucket_names" {
- command = apply
-
- variables {
- bucket_name = "acme.beacon.telemetry"
- }
-
- assert {
- condition = output.beacon_reader_source_url == "https://s3.us-west-2.amazonaws.com/acme.beacon.telemetry/beacon-prod"
- error_message = "A dotted bucket name fails TLS on a virtual-hosted URL, so the URL must be path style."
+ condition = (
+ output.bucket.name == "acme-beacon-telemetry"
+ && output.bucket.arn == "arn:aws:s3:::acme-beacon-telemetry"
+ && output.bucket.region == "us-west-2"
+ && output.bucket.prefix == "beacon-prod"
+ && output.bucket.kms_key_arn == null
+ )
+ error_message = "The bucket output carries what the reader module needs."
}
}
run "kms_key_encrypts_and_is_granted" {
- command = apply
+ command = plan
variables {
- kms_key_arn = "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
- create_beacon_reader = true
+ kms_key_arn = "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
}
assert {
@@ -189,16 +170,6 @@ run "kms_key_encrypts_and_is_granted" {
})
error_message = "The writer needs kms:GenerateDataKey to upload to an SSE-KMS bucket."
}
-
- assert {
- condition = contains(jsondecode(aws_iam_user_policy.beacon_reader[0].policy).Statement, {
- Sid = "DecryptBeaconObjects"
- Effect = "Allow"
- Action = "kms:Decrypt"
- Resource = "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
- })
- error_message = "The Beacon reader needs kms:Decrypt to read an SSE-KMS bucket."
- }
}
run "healthcheck_permission_can_be_dropped" {
@@ -214,161 +185,16 @@ run "healthcheck_permission_can_be_dropped" {
}
}
-run "beacon_reader_reads_only_the_prefix" {
- command = apply
-
- variables {
- create_beacon_reader = true
- }
-
- assert {
- condition = jsondecode(aws_iam_user_policy.beacon_reader[0].policy).Statement == [
- {
- Sid = "ListBeaconBucket"
- Effect = "Allow"
- Action = ["s3:ListBucket", "s3:GetBucketLocation"]
- Resource = "arn:aws:s3:::acme-beacon-telemetry"
- },
- {
- Sid = "ReadBeaconObjects"
- Effect = "Allow"
- Action = "s3:GetObject"
- Resource = "arn:aws:s3:::acme-beacon-telemetry/beacon-prod/*"
- },
- {
- Sid = "ConsumeBeaconNotifications"
- Effect = "Allow"
- Action = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]
- Resource = [aws_sqs_queue.beacon_reader["inventory"].arn, aws_sqs_queue.beacon_reader["runtime"].arn]
- },
- {
- Sid = "FindBeaconQueues"
- Effect = "Allow"
- Action = "sqs:ListQueues"
- Resource = "*"
- },
- ]
- error_message = "The Beacon reader may list the bucket, read only the Beacon prefix, and consume its two queues."
- }
-
- assert {
- condition = aws_iam_user.beacon_reader[0].name == "beacon-s3-reader" && length(aws_iam_access_key.beacon_reader) == 0
- error_message = "The reader is created without a key unless create_access_keys is set."
- }
-}
-
-run "beacon_reader_gets_new_object_queues" {
- command = apply
-
- variables {
- create_beacon_reader = true
- }
-
- assert {
- condition = (
- aws_sqs_queue.beacon_reader["runtime"].name == "beacon-s3-reader-runtime"
- && aws_sqs_queue.beacon_reader["inventory"].name == "beacon-s3-reader-inventory"
- && aws_sqs_queue.beacon_reader_dead_letter["runtime"].name == "beacon-s3-reader-runtime-dlq"
- && aws_sqs_queue.beacon_reader_dead_letter["inventory"].name == "beacon-s3-reader-inventory-dlq"
- )
- error_message = "Each stream gets a queue and a dead-letter queue named after the reader."
- }
-
- assert {
- condition = alltrue([
- for stream in ["runtime", "inventory"] :
- aws_sqs_queue.beacon_reader[stream].sqs_managed_sse_enabled
- && aws_sqs_queue.beacon_reader[stream].message_retention_seconds == 1209600
- && aws_sqs_queue.beacon_reader[stream].visibility_timeout_seconds == 300
- && jsondecode(aws_sqs_queue.beacon_reader[stream].redrive_policy) == {
- deadLetterTargetArn = aws_sqs_queue.beacon_reader_dead_letter[stream].arn
- maxReceiveCount = 10
- }
- ])
- error_message = "Queues are encrypted, keep messages 14 days, and move a message to the dead-letter queue after ten failed deliveries."
- }
-
- assert {
- condition = alltrue([
- for stream in ["runtime", "inventory"] :
- jsondecode(aws_sqs_queue_policy.beacon_reader[stream].policy).Statement == [
- {
- Sid = "AllowBucketNotifications"
- Effect = "Allow"
- Principal = { Service = "s3.amazonaws.com" }
- Action = "sqs:SendMessage"
- Resource = aws_sqs_queue.beacon_reader[stream].arn
- Condition = {
- ArnEquals = { "aws:SourceArn" = "arn:aws:s3:::acme-beacon-telemetry" }
- StringEquals = { "aws:SourceAccount" = "111122223333" }
- }
- },
- ]
- ])
- error_message = "Only this bucket, in this account, may send to the queues."
- }
-
- assert {
- condition = {
- for queue in aws_s3_bucket_notification.this[0].queue : queue.filter_prefix => {
- arn = queue.queue_arn
- events = queue.events
- }
- } == {
- "beacon-prod/runtime/" = { arn = aws_sqs_queue.beacon_reader["runtime"].arn, events = toset(["s3:ObjectCreated:*"]) }
- "beacon-prod/inventory/" = { arn = aws_sqs_queue.beacon_reader["inventory"].arn, events = toset(["s3:ObjectCreated:*"]) }
- }
- error_message = "Every new object below each stream's prefix is announced on that stream's queue."
- }
-
- assert {
- condition = output.beacon_reader_queue_urls == {
- runtime = aws_sqs_queue.beacon_reader["runtime"].url
- inventory = aws_sqs_queue.beacon_reader["inventory"].url
- }
- error_message = "beacon_reader_queue_urls gives the provisioning script's --queue-url values."
- }
-}
-
-run "queue_names_drop_characters_sqs_refuses" {
- command = plan
-
- variables {
- create_beacon_reader = true
- beacon_reader_user_name = "beacon.reader@acme"
- }
-
- assert {
- condition = aws_sqs_queue.beacon_reader["runtime"].name == "beacon-reader-acme-runtime"
- error_message = "IAM user names allow . and @, which SQS names do not."
- }
-}
-
-run "no_queues_without_the_reader" {
- command = apply
-
- assert {
- condition = (
- length(aws_sqs_queue.beacon_reader) == 0
- && length(aws_sqs_queue.beacon_reader_dead_letter) == 0
- && length(aws_s3_bucket_notification.this) == 0
- && output.beacon_reader_queue_urls == null
- )
- error_message = "Queues and the bucket notification exist only with the Beacon reader."
- }
-}
-
-run "access_keys_when_asked" {
+run "writer_access_key_when_asked" {
command = plan
variables {
- create_beacon_reader = true
- create_access_keys = true
+ create_writer_access_key = true
}
assert {
- condition = length(aws_iam_access_key.writer) == 1 && length(aws_iam_access_key.beacon_reader) == 1
- error_message = "create_access_keys creates one key per user."
+ condition = length(aws_iam_access_key.writer) == 1
+ error_message = "create_writer_access_key creates one key."
}
}
diff --git a/terraform/aws-s3/tests/partition.tftest.hcl b/terraform/aws-s3/tests/partition.tftest.hcl
index ae90c39b..61167862 100644
--- a/terraform/aws-s3/tests/partition.tftest.hcl
+++ b/terraform/aws-s3/tests/partition.tftest.hcl
@@ -24,11 +24,6 @@ variables {
run "urls_and_arns_follow_the_partition" {
command = apply
- assert {
- condition = output.beacon_reader_source_url == "https://acme-beacon-telemetry.s3.cn-north-1.amazonaws.com.cn/beacon-prod"
- error_message = "The data connector URL uses the partition's DNS suffix."
- }
-
assert {
condition = output.bucket_arn == "arn:aws-cn:s3:::acme-beacon-telemetry"
error_message = "The bucket ARN uses the partition."
diff --git a/terraform/aws-s3/variables.tf b/terraform/aws-s3/variables.tf
index 6691f57c..cf6a0674 100644
--- a/terraform/aws-s3/variables.tf
+++ b/terraform/aws-s3/variables.tf
@@ -42,7 +42,7 @@ variable "storage_class" {
}
variable "kms_key_arn" {
- description = "ARN of a customer managed KMS key to make the bucket's default encryption (SSE-KMS with an S3 Bucket Key). The writer gets kms:GenerateDataKey and the Beacon reader gets kms:Decrypt on it. Leave null for SSE-S3."
+ description = "ARN of a customer managed KMS key to make the bucket's default encryption (SSE-KMS with an S3 Bucket Key). The writer gets kms:GenerateDataKey on it. Leave null for SSE-S3."
type = string
default = null
@@ -98,26 +98,14 @@ variable "allow_writer_healthcheck" {
default = true
}
-variable "create_beacon_reader" {
- description = "Create a read-only IAM user for Beacon Cloud's S3 data connector, which reads your bucket with an access key. It can read only the Beacon prefix. The module also creates an SQS queue per stream, with a dead-letter queue, that S3 notifies of every new object, so the connector loads objects in any order."
- type = bool
- default = false
-}
-
-variable "beacon_reader_user_name" {
- description = "Name of the read-only IAM user created when create_beacon_reader = true. Its queues are named after it: <name>-runtime and <name>-inventory, each with a -dlq."
- type = string
- default = "beacon-s3-reader"
-}
-
-variable "create_access_keys" {
- description = "Also create an access key for the writer (and for the Beacon reader, when created). The secret keys are then stored in Terraform state, so leave this false unless your state is encrypted and access-controlled, and create the keys with the AWS CLI instead."
+variable "create_writer_access_key" {
+ description = "Also create the writer's access key. The secret key is then stored in Terraform state, so leave this false unless your state is encrypted and access-controlled, and create the key with the AWS CLI instead."
type = bool
default = false
}
variable "iam_path" {
- description = "Path for the IAM users."
+ description = "Path for the writer IAM user."
type = string
default = "/"
@@ -128,7 +116,7 @@ variable "iam_path" {
}
variable "permissions_boundary_arn" {
- description = "ARN of a permissions boundary policy to attach to the IAM users, if your organization requires one."
+ description = "ARN of a permissions boundary policy to attach to the writer IAM user, if your organization requires one."
type = string
default = null
}
diff --git a/terraform/gcs-asymptote-reader/README.md b/terraform/gcs-asymptote-reader/README.md
new file mode 100644
index 00000000..bdce661d
--- /dev/null
+++ b/terraform/gcs-asymptote-reader/README.md
@@ -0,0 +1,32 @@
+# Asymptote reader for a Beacon Cloud Storage bucket
+
+Gives Asymptote read access to a Beacon bucket created by the [`gcs`](../gcs) module, for
+customers who keep Beacon telemetry in their own bucket and use Asymptote's dashboard:
+
+- `roles/storage.objectViewer` on the bucket for Asymptote's reader,
+ `[email protected]`. No key is involved;
+- a Pub/Sub topic the bucket notifies of every new object below the Beacon prefix, with one
+ subscription per stream (`runtime`, `inventory`) that Asymptote's reader may consume, so
+ Asymptote loads objects in any order.
+
+```hcl
+module "beacon_gcs" {
+ source = "github.com/asymptote-labs/agent-beacon//terraform/gcs?ref=<release tag>"
+
+ project = "acme-security"
+ location = "us-west1"
+ bucket_name = "acme-beacon-telemetry"
+ prefix = "beacon-prod"
+}
+
+module "asymptote_reader" {
+ source = "github.com/asymptote-labs/agent-beacon//terraform/gcs-asymptote-reader?ref=<release tag>"
+
+ bucket = module.beacon_gcs.bucket
+}
+```
+
+Send Asymptote the bucket name, the prefix and the `subscriptions` output.
+
+The full walkthrough is on the Asymptote docs:
+[Asymptote with your own Cloud Storage bucket](https://docs.asymptotelabs.ai/deployment/self-hosted-gcs).
diff --git a/terraform/gcs-asymptote-reader/examples/basic/main.tf b/terraform/gcs-asymptote-reader/examples/basic/main.tf
new file mode 100644
index 00000000..a94f6c6b
--- /dev/null
+++ b/terraform/gcs-asymptote-reader/examples/basic/main.tf
@@ -0,0 +1,77 @@
+# A Beacon bucket that Asymptote can read: the gcs module creates the bucket and the writer, and
+# this module adds Asymptote's read access and the new-object subscriptions.
+#
+# terraform init
+# terraform apply -var project=<project-id> -var location=us-west1 -var bucket_name=<globally-unique-name>
+# gcloud iam service-accounts keys create beacon-gcs-writer.json --iam-account "$(terraform output -raw writer_email)"
+#
+# Outside this repository, point both sources at a release:
+# source = "github.com/asymptote-labs/agent-beacon//terraform/gcs?ref=<release tag>"
+# source = "github.com/asymptote-labs/agent-beacon//terraform/gcs-asymptote-reader?ref=<release tag>"
+
+terraform {
+ required_version = ">= 1.5.0"
+
+ required_providers {
+ google = {
+ source = "hashicorp/google"
+ version = ">= 5.0"
+ }
+ }
+}
+
+variable "project" {
+ description = "Google Cloud project for the bucket, service account and subscriptions."
+ type = string
+}
+
+variable "location" {
+ description = "Bucket location, such as us-west1."
+ type = string
+}
+
+variable "bucket_name" {
+ description = "Globally unique name for the new bucket."
+ type = string
+}
+
+variable "prefix" {
+ description = "Root prefix for Beacon objects."
+ type = string
+ default = "beacon-prod"
+}
+
+provider "google" {
+ project = var.project
+}
+
+module "beacon_gcs" {
+ source = "../../../gcs"
+
+ project = var.project
+ location = var.location
+ bucket_name = var.bucket_name
+ prefix = var.prefix
+}
+
+module "asymptote_reader" {
+ source = "../.."
+
+ bucket = module.beacon_gcs.bucket
+}
+
+output "bucket_name" {
+ value = module.beacon_gcs.bucket_name
+}
+
+output "prefix" {
+ value = module.beacon_gcs.prefix
+}
+
+output "writer_email" {
+ value = module.beacon_gcs.writer_email
+}
+
+output "asymptote_subscriptions" {
+ value = module.asymptote_reader.subscriptions
+}
diff --git a/terraform/gcs-asymptote-reader/main.tf b/terraform/gcs-asymptote-reader/main.tf
new file mode 100644
index 00000000..ed32cc6c
--- /dev/null
+++ b/terraform/gcs-asymptote-reader/main.tf
@@ -0,0 +1,86 @@
+# Read access for Asymptote to a Beacon bucket created by the gcs module.
+#
+# Asymptote's ClickPipes read the bucket in "any order" mode: Cloud Storage announces every new
+# object on Pub/Sub, and the pipe loads what its subscription names. Without them a pipe lists
+# objects in name order and skips any that sort before ones it has already loaded, which happens
+# whenever Macs upload out of step. The subscriptions match the ones Asymptote's managed ingest uses.
+
+locals {
+ streams = toset(["runtime", "inventory"])
+}
+
+resource "google_project_service" "pubsub" {
+ count = var.enable_apis ? 1 : 0
+
+ project = var.bucket.project
+ service = "pubsub.googleapis.com"
+ disable_on_destroy = false
+}
+
+# Asymptote reads with its own service account, so no key is involved.
+resource "google_storage_bucket_iam_member" "reader" {
+ bucket = var.bucket.name
+ role = "roles/storage.objectViewer"
+ member = var.reader_member
+}
+
+# Cloud Storage publishes notifications as its own service agent.
+data "google_storage_project_service_account" "this" {
+ project = var.bucket.project
+}
+
+resource "google_pubsub_topic" "objects" {
+ project = var.bucket.project
+ name = "asymptote-${var.bucket.name}-objects"
+ labels = var.labels
+
+ depends_on = [google_project_service.pubsub]
+}
+
+resource "google_pubsub_topic_iam_member" "storage_agent" {
+ project = var.bucket.project
+ topic = google_pubsub_topic.objects.name
+ role = "roles/pubsub.publisher"
+ member = "serviceAccount:${data.google_storage_project_service_account.this.email_address}"
+}
+
+resource "google_storage_notification" "objects" {
+ bucket = var.bucket.name
+ topic = google_pubsub_topic.objects.id
+ payload_format = "JSON_API_V1"
+ event_types = ["OBJECT_FINALIZE"]
+ object_name_prefix = "${var.bucket.prefix}/"
+
+ # Cloud Storage checks that it may publish to the topic when the notification is created.
+ depends_on = [google_pubsub_topic_iam_member.storage_agent]
+}
+
+resource "google_pubsub_subscription" "reader" {
+ for_each = local.streams
+
+ project = var.bucket.project
+ name = "asymptote-${var.bucket.name}-${each.key}"
+ topic = google_pubsub_topic.objects.id
+ filter = "attributes.eventType = \"OBJECT_FINALIZE\" AND hasPrefix(attributes.objectId, \"${var.bucket.prefix}/${each.key}/\")"
+ ack_deadline_seconds = 60
+ message_retention_duration = "604800s"
+ labels = var.labels
+
+ # An empty TTL never expires the subscription. The default expires it after 31 quiet days,
+ # which would silently stop ingestion from a fleet that was offline for a month.
+ expiration_policy {
+ ttl = ""
+ }
+}
+
+resource "google_pubsub_subscription_iam_member" "reader" {
+ for_each = {
+ for pair in setproduct(local.streams, ["roles/pubsub.subscriber", "roles/pubsub.viewer"]) :
+ "${pair[0]}:${pair[1]}" => { stream = pair[0], role = pair[1] }
+ }
+
+ project = var.bucket.project
+ subscription = google_pubsub_subscription.reader[each.value.stream].name
+ role = each.value.role
+ member = var.reader_member
+}
diff --git a/terraform/gcs-asymptote-reader/outputs.tf b/terraform/gcs-asymptote-reader/outputs.tf
new file mode 100644
index 00000000..bbd635fb
--- /dev/null
+++ b/terraform/gcs-asymptote-reader/outputs.tf
@@ -0,0 +1,9 @@
+output "subscriptions" {
+ description = "The new-object subscriptions to send Asymptote, by stream."
+ value = { for stream, sub in google_pubsub_subscription.reader : stream => sub.id }
+}
+
+output "topic" {
+ description = "The Pub/Sub topic the bucket publishes new objects to."
+ value = google_pubsub_topic.objects.id
+}
diff --git a/terraform/gcs-asymptote-reader/tests/module.tftest.hcl b/terraform/gcs-asymptote-reader/tests/module.tftest.hcl
new file mode 100644
index 00000000..514c0443
--- /dev/null
+++ b/terraform/gcs-asymptote-reader/tests/module.tftest.hcl
@@ -0,0 +1,109 @@
+# Hermetic: the Google provider is mocked, so these runs need no project or credentials.
+
+mock_provider "google" {
+ mock_resource "google_pubsub_topic" {
+ defaults = {
+ id = "projects/acme-security/topics/asymptote-acme-beacon-telemetry-objects"
+ }
+ }
+
+ mock_data "google_storage_project_service_account" {
+ defaults = {
+ email_address = "[email protected]"
+ }
+ }
+}
+
+variables {
+ bucket = {
+ name = "acme-beacon-telemetry"
+ project = "acme-security"
+ prefix = "beacon-prod"
+ }
+}
+
+run "asymptote_can_read_the_bucket" {
+ command = plan
+
+ assert {
+ condition = (
+ google_storage_bucket_iam_member.reader.bucket == "acme-beacon-telemetry"
+ && google_storage_bucket_iam_member.reader.role == "roles/storage.objectViewer"
+ && google_storage_bucket_iam_member.reader.member == "serviceAccount:[email protected]"
+ )
+ error_message = "Asymptote's reader gets objectViewer on the bucket."
+ }
+}
+
+run "new_objects_are_announced_per_stream" {
+ command = apply
+
+ assert {
+ condition = (
+ google_pubsub_topic.objects.name == "asymptote-acme-beacon-telemetry-objects"
+ && google_pubsub_topic_iam_member.storage_agent.role == "roles/pubsub.publisher"
+ && google_pubsub_topic_iam_member.storage_agent.member == "serviceAccount:[email protected]"
+ )
+ error_message = "Cloud Storage's service agent publishes new-object notifications to a topic for this bucket."
+ }
+
+ assert {
+ condition = (
+ google_storage_notification.objects.bucket == "acme-beacon-telemetry"
+ && google_storage_notification.objects.event_types == toset(["OBJECT_FINALIZE"])
+ && google_storage_notification.objects.object_name_prefix == "beacon-prod/"
+ && google_storage_notification.objects.payload_format == "JSON_API_V1"
+ )
+ error_message = "Every new object below the prefix is announced."
+ }
+
+ assert {
+ condition = alltrue([
+ for stream in ["runtime", "inventory"] :
+ google_pubsub_subscription.reader[stream].name == "asymptote-acme-beacon-telemetry-${stream}"
+ && google_pubsub_subscription.reader[stream].filter == "attributes.eventType = \"OBJECT_FINALIZE\" AND hasPrefix(attributes.objectId, \"beacon-prod/${stream}/\")"
+ && google_pubsub_subscription.reader[stream].ack_deadline_seconds == 60
+ && google_pubsub_subscription.reader[stream].message_retention_duration == "604800s"
+ && one(google_pubsub_subscription.reader[stream].expiration_policy).ttl == ""
+ ])
+ error_message = "Each stream has a never-expiring subscription filtered to its prefix, with the managed path's settings."
+ }
+
+ assert {
+ condition = toset([
+ for binding in google_pubsub_subscription_iam_member.reader : "${binding.subscription}|${binding.role}|${binding.member}"
+ ]) == toset(flatten([
+ for stream in ["runtime", "inventory"] : [
+ for role in ["roles/pubsub.subscriber", "roles/pubsub.viewer"] :
+ "${google_pubsub_subscription.reader[stream].name}|${role}|serviceAccount:[email protected]"
+ ]
+ ]))
+ error_message = "Asymptote's reader may consume and view both subscriptions, and nothing else gets access."
+ }
+
+ assert {
+ condition = output.subscriptions == {
+ runtime = google_pubsub_subscription.reader["runtime"].id
+ inventory = google_pubsub_subscription.reader["inventory"].id
+ }
+ error_message = "subscriptions gives the provisioning script's --queue-url values."
+ }
+
+ assert {
+ condition = google_project_service.pubsub[0].service == "pubsub.googleapis.com" && google_project_service.pubsub[0].disable_on_destroy == false
+ error_message = "The module enables Pub/Sub and never disables it."
+ }
+}
+
+run "apis_left_alone_when_asked" {
+ command = plan
+
+ variables {
+ enable_apis = false
+ }
+
+ assert {
+ condition = length(google_project_service.pubsub) == 0
+ error_message = "enable_apis = false leaves the project's APIs alone."
+ }
+}
diff --git a/terraform/gcs-asymptote-reader/variables.tf b/terraform/gcs-asymptote-reader/variables.tf
new file mode 100644
index 00000000..d4e05adf
--- /dev/null
+++ b/terraform/gcs-asymptote-reader/variables.tf
@@ -0,0 +1,26 @@
+variable "bucket" {
+ description = "The Beacon bucket to read, as the gcs module's bucket output: bucket = module.beacon_gcs.bucket."
+ type = object({
+ name = string
+ project = string
+ prefix = string
+ })
+}
+
+variable "reader_member" {
+ description = "Asymptote's reader identity. Change it only if Asymptote gives you a different one."
+ type = string
+ default = "serviceAccount:[email protected]"
+}
+
+variable "enable_apis" {
+ description = "Enable the Pub/Sub API on the project. It is left enabled when the module is destroyed."
+ type = bool
+ default = true
+}
+
+variable "labels" {
+ description = "Labels to apply to the Pub/Sub topic and subscriptions."
+ type = map(string)
+ default = {}
+}
diff --git a/terraform/gcs-asymptote-reader/versions.tf b/terraform/gcs-asymptote-reader/versions.tf
new file mode 100644
index 00000000..55b5fcf4
--- /dev/null
+++ b/terraform/gcs-asymptote-reader/versions.tf
@@ -0,0 +1,10 @@
+terraform {
+ required_version = ">= 1.5.0"
+
+ required_providers {
+ google = {
+ source = "hashicorp/google"
+ version = ">= 5.0"
+ }
+ }
+}
diff --git a/terraform/gcs/README.md b/terraform/gcs/README.md
index b7ad66af..8c2d2581 100644
--- a/terraform/gcs/README.md
+++ b/terraform/gcs/README.md
@@ -7,10 +7,8 @@ Creates the Google Cloud side of Beacon's GCS forwarding:
scoped to the Beacon prefix;
- `beacon-endpoint-writer`, a service account with `roles/storage.objectCreator` on
the bucket, limited by an IAM condition to objects below the prefix;
-- optionally a read-only `beacon-endpoint-reader` (`create_reader = true`) and read
- access for Beacon Cloud (`grant_beacon_read_access = true`), which also publishes
- every new object to Pub/Sub with a subscription per stream, so Beacon loads objects in
- any order.
+- optionally a read-only `beacon-endpoint-reader` (`create_reader = true`) for
+ checking uploads.
```hcl
provider "google" {
diff --git a/terraform/gcs/main.tf b/terraform/gcs/main.tf
index c2f5e845..edbe83c1 100644
--- a/terraform/gcs/main.tf
+++ b/terraform/gcs/main.tf
@@ -1,15 +1,9 @@
locals {
- apis = var.enable_apis ? toset(concat(
- ["storage.googleapis.com", "iam.googleapis.com"],
- var.grant_beacon_read_access ? ["pubsub.googleapis.com"] : [],
- )) : toset([])
+ apis = var.enable_apis ? toset(["storage.googleapis.com", "iam.googleapis.com"]) : toset([])
# Beacon writes <prefix>/runtime/date=YYYY-MM-DD/... and <prefix>/inventory/date=YYYY-MM-DD/...
# The condition is built from the variable, not the bucket resource, so it is known at plan time.
writer_condition = "resource.name.startsWith(\"projects/_/buckets/${var.bucket_name}/objects/${var.prefix}/\")"
-
- # Cloud Storage's service agent encrypts with a customer-managed key and publishes notifications.
- needs_storage_agent = var.kms_key_name != null && var.grant_kms_to_storage_agent || var.grant_beacon_read_access
}
resource "google_project_service" "this" {
@@ -21,15 +15,16 @@ resource "google_project_service" "this" {
}
data "google_storage_project_service_account" "this" {
- count = local.needs_storage_agent ? 1 : 0
+ count = var.kms_key_name != null && var.grant_kms_to_storage_agent ? 1 : 0
project = var.project
depends_on = [google_project_service.this]
}
+# Cloud Storage encrypts with a customer-managed key as its own service agent, not as the writer.
resource "google_kms_crypto_key_iam_member" "storage_agent" {
- count = var.kms_key_name != null && var.grant_kms_to_storage_agent ? 1 : 0
+ count = length(data.google_storage_project_service_account.this)
crypto_key_id = var.kms_key_name
role = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
@@ -134,11 +129,3 @@ resource "google_storage_bucket_iam_member" "reader" {
role = "roles/storage.objectViewer"
member = "serviceAccount:${google_service_account.reader[0].email}"
}
-
-resource "google_storage_bucket_iam_member" "beacon_reader" {
- count = var.grant_beacon_read_access ? 1 : 0
-
- bucket = google_storage_bucket.this.name
- role = "roles/storage.objectViewer"
- member = var.beacon_reader_member
-}
diff --git a/terraform/gcs/notifications.tf b/terraform/gcs/notifications.tf
deleted file mode 100644
index acad267d..00000000
--- a/terraform/gcs/notifications.tf
+++ /dev/null
@@ -1,70 +0,0 @@
-# Beacon Cloud's ClickPipes read the bucket in "any order" mode: Cloud Storage announces every new
-# object on Pub/Sub, and the pipe loads what its subscription names. Without them a pipe lists
-# objects in name order and skips any that sort before ones it has already loaded, which happens
-# whenever Macs upload out of step. The subscriptions match the ones Beacon's managed ingest uses.
-
-locals {
- reader_streams = var.grant_beacon_read_access ? toset(["runtime", "inventory"]) : toset([])
-}
-
-resource "google_pubsub_topic" "beacon_reader" {
- count = var.grant_beacon_read_access ? 1 : 0
-
- project = var.project
- name = "beacon-${var.bucket_name}-objects"
- labels = var.labels
-
- depends_on = [google_project_service.this]
-}
-
-resource "google_pubsub_topic_iam_member" "storage_agent" {
- count = var.grant_beacon_read_access ? 1 : 0
-
- project = var.project
- topic = google_pubsub_topic.beacon_reader[0].name
- role = "roles/pubsub.publisher"
- member = "serviceAccount:${data.google_storage_project_service_account.this[0].email_address}"
-}
-
-resource "google_storage_notification" "beacon_reader" {
- count = var.grant_beacon_read_access ? 1 : 0
-
- bucket = google_storage_bucket.this.name
- topic = google_pubsub_topic.beacon_reader[0].id
- payload_format = "JSON_API_V1"
- event_types = ["OBJECT_FINALIZE"]
- object_name_prefix = "${var.prefix}/"
-
- # Cloud Storage checks that it may publish to the topic when the notification is created.
- depends_on = [google_pubsub_topic_iam_member.storage_agent]
-}
-
-resource "google_pubsub_subscription" "beacon_reader" {
- for_each = local.reader_streams
-
- project = var.project
- name = "beacon-${var.bucket_name}-${each.key}"
- topic = google_pubsub_topic.beacon_reader[0].id
- filter = "attributes.eventType = \"OBJECT_FINALIZE\" AND hasPrefix(attributes.objectId, \"${var.prefix}/${each.key}/\")"
- ack_deadline_seconds = 60
- message_retention_duration = "604800s"
- labels = var.labels
-
- # An empty TTL never expires the subscription. The default expires it after 31 quiet days,
- # which would silently stop ingestion from a fleet that was offline for a month.
- expiration_policy {
- ttl = ""
- }
-}
-
-resource "google_pubsub_subscription_iam_member" "beacon_reader" {
- for_each = {
- for pair in setproduct(local.reader_streams, ["roles/pubsub.subscriber", "roles/pubsub.viewer"]) :
- "${pair[0]}:${pair[1]}" => { stream = pair[0], role = pair[1] }
- }
-
- project = var.project
- subscription = google_pubsub_subscription.beacon_reader[each.value.stream].name
- role = each.value.role
- member = var.beacon_reader_member
-}
diff --git a/terraform/gcs/outputs.tf b/terraform/gcs/outputs.tf
index 948f345c..ba6cf94e 100644
--- a/terraform/gcs/outputs.tf
+++ b/terraform/gcs/outputs.tf
@@ -13,6 +13,15 @@ output "prefix" {
value = var.prefix
}
+output "bucket" {
+ description = "The bucket as one object, for the gcs-asymptote-reader module: bucket = module.beacon_gcs.bucket."
+ value = {
+ name = google_storage_bucket.this.name
+ project = var.project
+ prefix = var.prefix
+ }
+}
+
output "storage_class" {
description = "Storage class the forwarder writes objects with (BEACON_GCS_STORAGE_CLASS)."
value = var.storage_class
@@ -42,8 +51,3 @@ output "reader_email" {
description = "Read-only service account, when create_reader = true."
value = try(google_service_account.reader[0].email, null)
}
-
-output "beacon_reader_subscriptions" {
- description = "The new-object subscriptions to give Beacon, by stream, when grant_beacon_read_access = true."
- value = var.grant_beacon_read_access ? { for stream, sub in google_pubsub_subscription.beacon_reader : stream => sub.id } : null
-}
diff --git a/terraform/gcs/tests/module.tftest.hcl b/terraform/gcs/tests/module.tftest.hcl
index eaa448ce..634cafbc 100644
--- a/terraform/gcs/tests/module.tftest.hcl
+++ b/terraform/gcs/tests/module.tftest.hcl
@@ -14,12 +14,6 @@ mock_provider "google" {
}
}
- mock_resource "google_pubsub_topic" {
- defaults = {
- id = "projects/acme-security/topics/beacon-acme-beacon-telemetry-objects"
- }
- }
-
mock_data "google_storage_project_service_account" {
defaults = {
email_address = "[email protected]"
@@ -53,7 +47,7 @@ run "writer_can_only_create_below_the_prefix" {
}
}
-run "no_key_reader_or_beacon_access_by_default" {
+run "no_key_or_reader_by_default" {
command = plan
assert {
@@ -61,9 +55,8 @@ run "no_key_reader_or_beacon_access_by_default" {
length(google_service_account_key.writer) == 0
&& length(google_service_account.reader) == 0
&& length(google_storage_bucket_iam_member.reader) == 0
- && length(google_storage_bucket_iam_member.beacon_reader) == 0
)
- error_message = "Keys, the reader and Beacon read access must all be opt-in."
+ error_message = "The key and the reader must be opt-in."
}
}
@@ -115,109 +108,13 @@ run "forwarder_settings_match_the_helper" {
}
assert {
- condition = output.writer_email == "[email protected]"
- error_message = "writer_email is the service account's email."
- }
-}
-
-run "beacon_read_access_when_asked" {
- command = plan
-
- variables {
- grant_beacon_read_access = true
- create_reader = true
- }
-
- assert {
- condition = (
- google_storage_bucket_iam_member.beacon_reader[0].role == "roles/storage.objectViewer"
- && google_storage_bucket_iam_member.beacon_reader[0].member == "serviceAccount:[email protected]"
- && length(google_storage_bucket_iam_member.beacon_reader[0].condition) == 0
- )
- error_message = "Beacon's reader gets objectViewer on the bucket."
+ condition = output.bucket.name == "acme-beacon-telemetry" && output.bucket.project == "acme-security" && output.bucket.prefix == "beacon-prod"
+ error_message = "The bucket output carries what the reader module needs."
}
assert {
- condition = google_storage_bucket_iam_member.reader[0].role == "roles/storage.objectViewer" && google_service_account.reader[0].account_id == "beacon-endpoint-reader"
- error_message = "create_reader adds the read-only service account the manual guides use."
- }
-}
-
-run "beacon_read_access_brings_new_object_subscriptions" {
- command = apply
-
- variables {
- grant_beacon_read_access = true
- }
-
- assert {
- condition = (
- google_pubsub_topic.beacon_reader[0].name == "beacon-acme-beacon-telemetry-objects"
- && google_pubsub_topic_iam_member.storage_agent[0].role == "roles/pubsub.publisher"
- && google_pubsub_topic_iam_member.storage_agent[0].member == "serviceAccount:[email protected]"
- )
- error_message = "Cloud Storage's service agent publishes new-object notifications to a topic for this bucket."
- }
-
- assert {
- condition = (
- google_storage_notification.beacon_reader[0].event_types == toset(["OBJECT_FINALIZE"])
- && google_storage_notification.beacon_reader[0].object_name_prefix == "beacon-prod/"
- && google_storage_notification.beacon_reader[0].payload_format == "JSON_API_V1"
- )
- error_message = "Every new object below the prefix is announced."
- }
-
- assert {
- condition = alltrue([
- for stream in ["runtime", "inventory"] :
- google_pubsub_subscription.beacon_reader[stream].name == "beacon-acme-beacon-telemetry-${stream}"
- && google_pubsub_subscription.beacon_reader[stream].filter == "attributes.eventType = \"OBJECT_FINALIZE\" AND hasPrefix(attributes.objectId, \"beacon-prod/${stream}/\")"
- && google_pubsub_subscription.beacon_reader[stream].ack_deadline_seconds == 60
- && google_pubsub_subscription.beacon_reader[stream].message_retention_duration == "604800s"
- && one(google_pubsub_subscription.beacon_reader[stream].expiration_policy).ttl == ""
- ])
- error_message = "Each stream has a never-expiring subscription filtered to its prefix, with the managed path's settings."
- }
-
- assert {
- condition = toset([
- for binding in google_pubsub_subscription_iam_member.beacon_reader : "${binding.subscription}|${binding.role}|${binding.member}"
- ]) == toset(flatten([
- for stream in ["runtime", "inventory"] : [
- for role in ["roles/pubsub.subscriber", "roles/pubsub.viewer"] :
- "${google_pubsub_subscription.beacon_reader[stream].name}|${role}|serviceAccount:[email protected]"
- ]
- ]))
- error_message = "Beacon's reader may consume and view both subscriptions, and nothing else gets access."
- }
-
- assert {
- condition = output.beacon_reader_subscriptions == {
- runtime = google_pubsub_subscription.beacon_reader["runtime"].id
- inventory = google_pubsub_subscription.beacon_reader["inventory"].id
- }
- error_message = "beacon_reader_subscriptions gives the provisioning script's --queue-url values."
- }
-
- assert {
- condition = contains(keys(google_project_service.this), "pubsub.googleapis.com")
- error_message = "Pub/Sub is enabled when Beacon reads the bucket."
- }
-}
-
-run "no_pubsub_without_beacon_read_access" {
- command = apply
-
- assert {
- condition = (
- length(google_pubsub_topic.beacon_reader) == 0
- && length(google_storage_notification.beacon_reader) == 0
- && length(google_pubsub_subscription.beacon_reader) == 0
- && output.beacon_reader_subscriptions == null
- && !contains(keys(google_project_service.this), "pubsub.googleapis.com")
- )
- error_message = "The topic, notification and subscriptions exist only with Beacon read access."
+ condition = output.writer_email == "[email protected]"
+ error_message = "writer_email is the service account's email."
}
}
@@ -293,3 +190,16 @@ run "rejects_short_kms_name" {
expect_failures = [var.kms_key_name]
}
+
+run "reader_when_asked" {
+ command = plan
+
+ variables {
+ create_reader = true
+ }
+
+ assert {
+ condition = google_storage_bucket_iam_member.reader[0].role == "roles/storage.objectViewer" && google_service_account.reader[0].account_id == "beacon-endpoint-reader"
+ error_message = "create_reader adds the read-only service account the manual guides use."
+ }
+}
diff --git a/terraform/gcs/variables.tf b/terraform/gcs/variables.tf
index 119a226f..ac3718ab 100644
--- a/terraform/gcs/variables.tf
+++ b/terraform/gcs/variables.tf
@@ -95,7 +95,7 @@ variable "force_destroy" {
}
variable "enable_apis" {
- description = "Enable the Cloud Storage and IAM APIs on the project, and Pub/Sub when grant_beacon_read_access = true. They are left enabled when the module is destroyed."
+ description = "Enable the Cloud Storage and IAM APIs on the project. They are left enabled when the module is destroyed."
type = bool
default = true
}
@@ -124,20 +124,8 @@ variable "reader_account_id" {
default = "beacon-endpoint-reader"
}
-variable "grant_beacon_read_access" {
- description = "Grant Beacon Cloud read access to the bucket (roles/storage.objectViewer for beacon_reader_member), so Beacon can ingest the telemetry you store there. No key is involved. The module also publishes every new object to a Pub/Sub topic with one subscription per stream that Beacon's reader may consume, so Beacon loads objects in any order."
- type = bool
- default = false
-}
-
-variable "beacon_reader_member" {
- description = "Beacon Cloud's reader identity. Change it only if Beacon gives you a different one."
- type = string
- default = "serviceAccount:[email protected]"
-}
-
variable "labels" {
- description = "Labels to apply to the bucket, and to the Pub/Sub topic and subscriptions when they exist."
+ description = "Labels to apply to the bucket."
type = map(string)
default = {}
}