Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

Keep the enterprise self-hosted read path out of docs.beacon.sh

shukieshahe2d5c8c9cf65
diff --git a/CLAUDE.md b/CLAUDE.md
index 012ce4aa..89ddc972 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -18,7 +18,7 @@ Beacon Endpoint Agent is a local-only endpoint telemetry agent for AI runtimes.
- `spec/lenses`: the Lenses format — single-file HTML views of one trace that the local dashboard runs in a sandboxed, network-less frame and feeds once through `window.beacon.getTrace()`. The contract types (`LensManifestV1`, `LensDataV1`) live in `pkg/asymptoteobserve/lens.go`, and `lens_test.go` pins the spec's examples and manifest schema to them. The dashboard builds the data in `BuildLensData` (`/api/lens-data`), lists lenses at `/api/lenses`, and serves each lens at `/lenses/frame/<id>` under the spec's CSP with `lenses/prelude.js` (which defines `window.beacon`) injected ahead of it; that route is the only one that may be framed. Built-in lenses are `cli/beacon/internal/endpoint/dashboard/lenses/*.lens.html`, and `activity` is byte-identical to the spec example (a test enforces it). The session page (`static/session.html`, `initLensTabs` in `app.js`) shows lenses as tabs fed by `/api/lens-data?session=`, which projects exactly the events that page lists; it is the host half of the protocol and closes a lens that errors, stalls, or navigates its own frame. Trust model (SPEC "Trust model"): the opaque origin, sandbox and CSP keep a lens from touching Beacon or the dashboard and block ordinary network APIs, but cannot fully prevent a lens from sending trace data out through browser features no CSP governs (WebRTC, resource hints, speculative loading), so an installed lens is code the person trusts with the trace data it is shown; never describe the sandbox as making exfiltration impossible. `prelude.js` removes the `RTC*` globals and guards the document (refuses creating, inserting or parsing in nested frames, `<link>` elements and `<script type=speculationrules>`, parsing markup in the real call's document and context and inserting exactly the checked nodes so `<noscript>`/foreign-content parser differentials cannot slip through, refuses `document.write` and `execCommand("insertHTML")`, and a mutation observer removes those from the file's own markup); that is best-effort hardening, not a boundary, and the hostile `probe` fixture in `testdata/lens-e2e` pins each covered path. Installed lenses live in the lens store (`cli/beacon/internal/endpoint/lensstore`, `<BaseDir>/lenses/<id>.lens.html`, 0600 under 0700), written only by `beacon lenses add|remove` (`cli/beacon/cmd/lenses.go`; `add` prints the trust warning and confirms on a TTY, and refuses without one or in CI unless `--yes`); the dashboard rescans it per request and never writes it, and a store lens can never shadow a built-in. `asymptoteobserve.CheckLensFile` is the one file check every installer, server and linter uses. User docs are `docs/concepts/lenses.mdx` and `docs/cli/lenses.mdx`; keep them in sync with the commands and built-in lenses. The authoring loop is `beacon lenses spec` (the spec carried in the binary as `dashboard/lenses/SPEC.md`, kept byte-identical to `spec/lenses/SPEC.md` by a test), `beacon lenses data` (the exact `LensDataV1` for a session), `beacon lenses lint` (`lensstore.Lint`; built-ins must lint clean) and `beacon lenses preview` (the dashboard with `Options.LensFiles`, re-read per reload).
- `beacon-sandbox`: harness that runs a real Claude Code session in a disposable sandbox and checks what Beacon captured. See `beacon-sandbox/AGENTS.md`.
- `packaging`: macOS, Linux, and Windows packaging and deployment assets.
-- `terraform`: customer-run Terraform modules for the cloud side of object storage forwarding. Each module always creates a new, dedicated bucket and never manages an existing one. `aws-s3` creates the hardened bucket, the put-only `beacon-endpoint-writer` IAM user, and optionally `beacon-s3-reader` for Beacon Cloud's S3 data connector, which reads with an access key because ClickPipes on the GCP-hosted ClickHouse rejects IAM roles. The reader comes with an SQS queue per stream (runtime, inventory), each with a dead-letter queue, and the bucket notification that feeds them, so the ClickPipes run in any-order mode; without a queue a pipe reads in key order and skips objects that land behind its cursor, which many Macs uploading on their own timers cause. `gcs` creates the bucket, a `beacon-endpoint-writer` service account with `objectCreator` limited to the prefix by an IAM condition, an optional reader, and with `grant_beacon_read_access` an `objectViewer` grant for Beacon's `beacon-clickpipes-reader` plus a Pub/Sub topic, bucket notification and one subscription per stream matching the managed path's (60 s ack, 7-day retention, never expiring, `objectId` prefix filter). The queue URLs and subscription paths are outputs, ready for the provisioning script's `--queue-url`. Neither module creates keys unless asked (`create_access_keys`, `create_writer_key`), so no secret lands in state by default. Resource names and the `forwarder_settings` output match the packaged `install-forwarder.sh` helpers and the manual guides. Docs are `docs/log-forwarding/{aws-s3,gcs}-terraform.mdx` and the connector steps in `docs/manage/configure-s3-data-connector.mdx`, mirrored in beacon-mdr's docs, with the module pinned to a release tag; keep both in sync with the variables.
+- `terraform`: customer-run Terraform modules for the cloud side of object storage forwarding. Each module always creates a new, dedicated bucket and never manages an existing one. `aws-s3` creates the hardened bucket, the put-only `beacon-endpoint-writer` IAM user, and optionally `beacon-s3-reader` for Beacon Cloud's S3 data connector, which reads with an access key because ClickPipes on the GCP-hosted ClickHouse rejects IAM roles. The reader comes with an SQS queue per stream (runtime, inventory), each with a dead-letter queue, and the bucket notification that feeds them, so the ClickPipes run in any-order mode; without a queue a pipe reads in key order and skips objects that land behind its cursor, which many Macs uploading on their own timers cause. `gcs` creates the bucket, a `beacon-endpoint-writer` service account with `objectCreator` limited to the prefix by an IAM condition, an optional reader, and with `grant_beacon_read_access` an `objectViewer` grant for Beacon's `beacon-clickpipes-reader` plus a Pub/Sub topic, bucket notification and one subscription per stream matching the managed path's (60 s ack, 7-day retention, never expiring, `objectId` prefix filter). The queue URLs and subscription paths are outputs, ready for the provisioning script's `--queue-url`. Neither module creates keys unless asked (`create_access_keys`, `create_writer_key`), so no secret lands in state by default. Resource names and the `forwarder_settings` output match the packaged `install-forwarder.sh` helpers and the manual guides. docs.beacon.sh (`docs/log-forwarding/{aws-s3,gcs}-terraform.mdx`) documents the bucket and writer only. The Beacon reader, its queues and subscriptions, and the S3 data connector are the enterprise self-hosted path and are documented only in beacon-mdr's docs (docs.asymptotelabs.ai), so keep them out of `docs/`; `/manage/configure-s3-data-connector` redirects there. Both sites pin the module to a release tag; keep them in sync with the variables.
- `docs`: the published documentation site (docs.asymptotelabs.ai).
Do not recreate or depend on removed `asymptote` mirror trees. Keep new work focused on the Beacon paths above.
diff --git a/docs/docs.json b/docs/docs.json
index 79368b67..63aae3ca 100644
--- a/docs/docs.json
+++ b/docs/docs.json
@@ -676,6 +676,10 @@
]
},
"redirects": [
+ {
+ "source": "/manage/configure-s3-data-connector",
+ "destination": "https://docs.asymptotelabs.ai/manage/configure-s3-data-connector"
+ },
{
"source": "/log-forwarding/s3",
"destination": "/guides/macos-object-storage-system"
diff --git a/docs/log-forwarding/aws-s3-terraform.mdx b/docs/log-forwarding/aws-s3-terraform.mdx
index c9df5610..180c1bac 100644
--- a/docs/log-forwarding/aws-s3-terraform.mdx
+++ b/docs/log-forwarding/aws-s3-terraform.mdx
@@ -11,7 +11,6 @@ The AWS CLI steps in the rollout guides create the same resources. Follow either
- **A new S3 bucket just for Beacon**, with public access blocked, ACLs disabled, HTTPS required, default encryption (SSE-S3, or SSE-KMS with your key), and versioning. The module always creates its own bucket and does not manage existing ones.
- **`beacon-endpoint-writer`**, an IAM user that can upload objects below your Beacon prefix and can't read or delete them.
-- **Optionally, `beacon-s3-reader`**, a read-only IAM user for [Beacon Cloud's S3 data connector](/manage/configure-s3-data-connector), with two SQS queues that tell Beacon about each new object.
Every Mac's forwarder writes both of Beacon's logs to the bucket:
@@ -152,7 +151,7 @@ Set any of these in the `module "beacon_s3"` block and run `terraform apply` aga
kms_key_arn = "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
```
- The bucket uses SSE-KMS with an S3 Bucket Key. The writer gets `kms:GenerateDataKey` and the Beacon reader gets `kms:Decrypt` on the key. The key's own policy must allow IAM policies in your account to grant access, which the default key policy does.
+ The bucket uses SSE-KMS with an S3 Bucket Key. The writer gets `kms:GenerateDataKey` on the key. The key's own policy must allow IAM policies in your account to grant access, which the default key policy does.
</Accordion>
<Accordion title="Delete old objects">
```hcl
@@ -161,32 +160,17 @@ Set any of these in the `module "beacon_s3"` block and run `terraform apply` aga
Objects below the prefix are deleted a year after upload. Versioning is on by default, so a deleted or replaced object stays recoverable for `noncurrent_version_expiration_days` (default 30). Set `versioning_enabled = false` to turn versioning off.
</Accordion>
- <Accordion title="Let Beacon Cloud read the bucket">
- ```hcl
- create_beacon_reader = true
- ```
-
- This adds `beacon-s3-reader`, which can list the bucket and read objects below the prefix. Create its key the same way as the writer's:
-
- ```bash
- aws iam create-access-key --user-name beacon-s3-reader --output json
- ```
-
- It also adds two SQS queues, `beacon-s3-reader-runtime` and `beacon-s3-reader-inventory`, each with a dead-letter queue. S3 announces every new object below the prefix on its stream's queue, and Beacon loads what the queues name. Without them Beacon would read objects in name order and could skip ones that arrive late, which happens when many Macs upload at once.
-
- Send Beacon the key, the `beacon_reader_source_url` output, and the two URLs in `beacon_reader_queue_urls`. See [Configure S3 Data Connector](/manage/configure-s3-data-connector).
- </Accordion>
<Accordion title="Create the access keys with Terraform">
```hcl
create_access_keys = true
```
- Terraform creates the writer's key, and the reader's when you have one. Read the secret with `terraform output -raw writer_secret_access_key` after adding a matching `output` block. The secret is stored in Terraform state, so use this only when your state is encrypted and access is restricted.
+ Terraform creates the writer's key. Read the secret with `terraform output -raw writer_secret_access_key` after adding a matching `output` block. The secret is stored in Terraform state, so use this only when your state is encrypted and access is restricted.
</Accordion>
<Accordion title="Match your organization's IAM rules">
- - `writer_user_name` and `beacon_reader_user_name` rename the users.
- - `iam_path` puts them under a path such as `/beacon/`.
- - `permissions_boundary_arn` attaches a permissions boundary.
+ - `writer_user_name` renames the writer.
+ - `iam_path` puts it under a path such as `/beacon/`.
+ - `permissions_boundary_arn` attaches a permissions boundary to it.
- `tags` tags every resource the module creates.
</Accordion>
<Accordion title="Drop the health-check permission">
@@ -204,9 +188,6 @@ Set any of these in the `module "beacon_s3"` block and run `terraform apply` aga
| --- | --- |
| `bucket_name`, `region`, `prefix`, `storage_class` | The forwarder settings, also together as `forwarder_settings` |
| `writer_user_name`, `writer_user_arn` | The IAM user every Mac uploads as |
-| `beacon_reader_user_name`, `beacon_reader_user_arn` | The Beacon reader, when `create_beacon_reader = true` |
-| `beacon_reader_source_url` | The location to send Beacon for the S3 data connector |
-| `beacon_reader_queue_urls` | The new-object queues to send Beacon, by stream |
| `writer_access_key_id`, `writer_secret_access_key` | Only with `create_access_keys = true` |
## Rotate the writer key
@@ -255,7 +236,4 @@ Rotation happens outside Terraform. IAM allows two keys per user, so the old key
<Card title="Google Cloud Storage with Terraform" icon="cloud" href="/log-forwarding/gcs-terraform">
The same setup for a Cloud Storage bucket.
</Card>
- <Card title="Configure S3 Data Connector" icon="database" href="/manage/configure-s3-data-connector">
- Let Beacon Cloud read the telemetry in your bucket.
- </Card>
</Columns>
diff --git a/docs/log-forwarding/gcs-terraform.mdx b/docs/log-forwarding/gcs-terraform.mdx
index 1d9fc5cc..664692c4 100644
--- a/docs/log-forwarding/gcs-terraform.mdx
+++ b/docs/log-forwarding/gcs-terraform.mdx
@@ -12,7 +12,6 @@ The gcloud steps in the rollout guides create the same resources. Follow either
- **A new Cloud Storage bucket just for Beacon**, with uniform bucket-level access, public access prevention enforced, versioning, and optionally your Cloud KMS key. The module always creates its own bucket and does not manage existing ones.
- **`beacon-endpoint-writer`**, a service account that can create objects below your Beacon prefix and can't list, read, or delete them.
- **`beacon-endpoint-reader`**, a read-only service account for checking uploads.
-- **Optionally, read access for Beacon Cloud**, so Beacon can ingest the telemetry you store, with Pub/Sub subscriptions that tell Beacon about each new object.
Every Mac's forwarder writes both of Beacon's logs to the bucket:
@@ -173,17 +172,6 @@ Set any of these in the `module "beacon_gcs"` block and run `terraform apply` ag
Objects below the prefix are deleted a year after upload. Versioning is on by default, so a deleted or replaced object stays recoverable for `noncurrent_version_expiration_days` (default 30). Set `versioning_enabled = false` to turn versioning off.
</Accordion>
- <Accordion title="Let Beacon Cloud read the bucket">
- ```hcl
- grant_beacon_read_access = true
- ```
-
- This grants `roles/storage.objectViewer` on the bucket to Beacon's reader, `[email protected]`. No key is involved.
-
- It also enables Pub/Sub and creates a topic that the bucket notifies of every new object below the prefix, with one subscription per stream that Beacon's reader can consume. Beacon loads what the subscriptions name. Without them Beacon would read objects in name order and could skip ones that arrive late, which happens when many Macs upload at once.
-
- Send Beacon the bucket name, the prefix, and the two subscriptions in `beacon_reader_subscriptions`.
- </Accordion>
<Accordion title="Create the writer's key with Terraform">
```hcl
create_writer_key = true
@@ -206,7 +194,6 @@ Set any of these in the `module "beacon_gcs"` block and run `terraform apply` ag
| `bucket_url` | `gs://<bucket>/<prefix>/` |
| `writer_email` | The service account every Mac uploads as |
| `reader_email` | The read-only service account, when `create_reader = true` |
-| `beacon_reader_subscriptions` | The new-object subscriptions to send Beacon, by stream, when `grant_beacon_read_access = true` |
| `writer_private_key` | Only with `create_writer_key = true` |
## Rotate the writer key
diff --git a/docs/manage/configure-s3-data-connector.mdx b/docs/manage/configure-s3-data-connector.mdx
deleted file mode 100644
index 56e2637e..00000000
--- a/docs/manage/configure-s3-data-connector.mdx
+++ /dev/null
@@ -1,253 +0,0 @@
----
-title: "Configure S3 Data Connector"
-description: "Configure read-only S3 access for Beacon ingestion."
----
-
-## Overview
-
-Beacon can ingest customer-hosted event files from a private AWS S3 bucket or prefix. Beacon reads them with the access key of a read-only IAM user you create. That user can list the bucket, read objects below the prefix you choose, and consume two notification queues, and nothing else.
-
-Beacon can't assume an IAM role in your account for this connector, so use an IAM user's access key.
-
-Do not make the bucket public. Do not grant write or delete permissions.
-
-## Create the read-only user and queues
-
-Beacon reads the bucket with the access key of a read-only IAM user. It also needs two SQS queues, one per stream, that S3 notifies of every new object. Beacon loads the objects the queues name, in any order. Without them Beacon would read objects in name order and could skip ones that arrive late, which happens when many Macs upload at once.
-
-Use a bucket dedicated to Beacon telemetry: the notification setup below replaces any notifications already configured on the bucket.
-
-<Tabs>
- <Tab title="Terraform">
- If you created the bucket with [AWS S3 with Terraform](/log-forwarding/aws-s3-terraform), turn on the reader in the module block and apply:
-
- ```hcl
- module "beacon_s3" {
- # ...your existing settings...
- create_beacon_reader = true
- }
- ```
-
- The module creates `beacon-s3-reader` with the policy shown in the other tab, including `kms:Decrypt` when you set `kms_key_arn`, the two queues with their dead-letter queues, and the bucket notification. Create the reader's access key:
-
- ```bash
- aws iam create-access-key --user-name beacon-s3-reader --output json
- ```
-
- The `beacon_reader_queue_urls` output holds the two queue URLs.
- </Tab>
- <Tab title="AWS CLI">
- **Set your values.** Use the bucket's region and the prefix that holds the event files:
-
- ```bash
- export AWS_REGION="<bucket region>"
- export BEACON_S3_BUCKET="<bucket>"
- export BEACON_S3_PREFIX="beacon-prod"
- export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
- ```
-
- **Create the queues.** Each stream gets a queue that only this bucket may send to, and a dead-letter queue that keeps a message after ten failed deliveries:
-
- ```bash
- for stream in runtime inventory; do
- dlq_url="$(aws sqs create-queue --region "$AWS_REGION" \
- --queue-name "beacon-s3-reader-${stream}-dlq" \
- --attributes MessageRetentionPeriod=1209600,SqsManagedSseEnabled=true \
- --query QueueUrl --output text)"
- dlq_arn="$(aws sqs get-queue-attributes --region "$AWS_REGION" --queue-url "$dlq_url" \
- --attribute-names QueueArn --query Attributes.QueueArn --output text)"
- queue_arn="arn:aws:sqs:${AWS_REGION}:${AWS_ACCOUNT_ID}:beacon-s3-reader-${stream}"
- cat > "beacon-s3-reader-${stream}.json" <<EOF
- {
- "MessageRetentionPeriod": "1209600",
- "VisibilityTimeout": "300",
- "SqsManagedSseEnabled": "true",
- "RedrivePolicy": "{\"deadLetterTargetArn\":\"${dlq_arn}\",\"maxReceiveCount\":10}",
- "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"s3.amazonaws.com\"},\"Action\":\"sqs:SendMessage\",\"Resource\":\"${queue_arn}\",\"Condition\":{\"ArnEquals\":{\"aws:SourceArn\":\"arn:aws:s3:::${BEACON_S3_BUCKET}\"},\"StringEquals\":{\"aws:SourceAccount\":\"${AWS_ACCOUNT_ID}\"}}}]}"
- }
- EOF
- aws sqs create-queue --region "$AWS_REGION" \
- --queue-name "beacon-s3-reader-${stream}" \
- --attributes "file://beacon-s3-reader-${stream}.json"
- done
- ```
-
- The loop prints the `QueueUrl` of the `runtime` and `inventory` queues. Keep both to send to Beacon.
-
- **Send new-object notifications to the queues:**
-
- ```bash
- cat > beacon-notifications.json <<EOF
- {
- "QueueConfigurations": [
- {
- "Id": "beacon-runtime",
- "QueueArn": "arn:aws:sqs:${AWS_REGION}:${AWS_ACCOUNT_ID}:beacon-s3-reader-runtime",
- "Events": ["s3:ObjectCreated:*"],
- "Filter": {"Key": {"FilterRules": [{"Name": "prefix", "Value": "${BEACON_S3_PREFIX}/runtime/"}]}}
- },
- {
- "Id": "beacon-inventory",
- "QueueArn": "arn:aws:sqs:${AWS_REGION}:${AWS_ACCOUNT_ID}:beacon-s3-reader-inventory",
- "Events": ["s3:ObjectCreated:*"],
- "Filter": {"Key": {"FilterRules": [{"Name": "prefix", "Value": "${BEACON_S3_PREFIX}/inventory/"}]}}
- }
- ]
- }
- EOF
- aws s3api put-bucket-notification-configuration \
- --bucket "$BEACON_S3_BUCKET" \
- --notification-configuration file://beacon-notifications.json
- ```
-
- It prints nothing when it works.
-
- **Create the user** and give it read-only access to the prefix and the two queues:
-
- ```bash
- aws iam create-user --user-name beacon-s3-reader
-
- aws iam put-user-policy \
- --user-name beacon-s3-reader \
- --policy-name BeaconTelemetryReadOnly \
- --policy-document "$(cat <<EOF
- {
- "Version": "2012-10-17",
- "Statement": [
- {
- "Effect": "Allow",
- "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
- "Resource": "arn:aws:s3:::${BEACON_S3_BUCKET}"
- },
- {
- "Effect": "Allow",
- "Action": "s3:GetObject",
- "Resource": "arn:aws:s3:::${BEACON_S3_BUCKET}/${BEACON_S3_PREFIX}/*"
- },
- {
- "Effect": "Allow",
- "Action": ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"],
- "Resource": [
- "arn:aws:sqs:${AWS_REGION}:${AWS_ACCOUNT_ID}:beacon-s3-reader-runtime",
- "arn:aws:sqs:${AWS_REGION}:${AWS_ACCOUNT_ID}:beacon-s3-reader-inventory"
- ]
- },
- {
- "Effect": "Allow",
- "Action": "sqs:ListQueues",
- "Resource": "*"
- }
- ]
- }
- EOF
- )"
- ```
-
- If the bucket uses SSE-KMS, also allow `kms:Decrypt` on the key, and make sure the key policy lets IAM policies in your account grant it:
-
- ```json title="SSE-KMS decrypt statement"
- {
- "Effect": "Allow",
- "Action": "kms:Decrypt",
- "Resource": "YOUR_KMS_KEY_ARN"
- }
- ```
-
- **Create the access key:**
-
- ```bash
- aws iam create-access-key --user-name beacon-s3-reader --output json
- ```
- </Tab>
-</Tabs>
-
-AWS shows the secret only once. Keep it in your password manager until you send it to Beacon.
-
-## Send Beacon the details
-
-Send these through the secure channel your Beacon contact gives you. Don't send the secret access key by email or chat.
-
-- S3 bucket name and AWS region
-- Prefix for the event files, such as `s3://your-bucket/beacon-prod/`
-- The two queue URLs, for `runtime` and `inventory`
-- The access key ID and secret access key
-- KMS key ARN, only if SSE-KMS is enabled
-- For files that Beacon's forwarder didn't write: the file format (ideally JSONL), delivery cadence, approximate event volume, and a small sample file or example JSONL row
-
-To rotate the key later, create a second key for `beacon-s3-reader`, send it to Beacon, and delete the old key once Beacon confirms the switch.
-
-## Using Google Cloud Storage
-
-For a Cloud Storage bucket, no key is involved. Beacon reads with its own service account, `[email protected]`, and needs a Pub/Sub subscription per stream for the same reason S3 needs the queues.
-
-<Tabs>
- <Tab title="Terraform">
- Set `grant_beacon_read_access = true` in the [Terraform module](/log-forwarding/gcs-terraform) and apply. Send Beacon the bucket name, the prefix, and the two subscriptions in `beacon_reader_subscriptions`.
- </Tab>
- <Tab title="gcloud">
- ```bash
- export GCP_PROJECT="<project-id>"
- export BEACON_GCS_BUCKET="<bucket>"
- export BEACON_GCS_PREFIX="beacon-prod"
- export BEACON_READER="serviceAccount:[email protected]"
- export BEACON_TOPIC="beacon-${BEACON_GCS_BUCKET}-objects"
-
- # Let Beacon read the bucket.
- gcloud storage buckets add-iam-policy-binding "gs://${BEACON_GCS_BUCKET}" \
- --member "$BEACON_READER" --role roles/storage.objectViewer
-
- # Publish every new object below the prefix to a topic.
- gcloud pubsub topics create "$BEACON_TOPIC" --project "$GCP_PROJECT"
- gcloud pubsub topics add-iam-policy-binding "$BEACON_TOPIC" --project "$GCP_PROJECT" \
- --member "serviceAccount:$(gcloud storage service-agent --project "$GCP_PROJECT" | tr -d '[:space:]')" \
- --role roles/pubsub.publisher
- gcloud storage buckets notifications create "gs://${BEACON_GCS_BUCKET}" \
- --topic "projects/${GCP_PROJECT}/topics/${BEACON_TOPIC}" \
- --event-types OBJECT_FINALIZE \
- --object-prefix "${BEACON_GCS_PREFIX}/" \
- --payload-format json
-
- # One subscription per stream that Beacon may consume. They never expire.
- for stream in runtime inventory; do
- gcloud pubsub subscriptions create "beacon-${BEACON_GCS_BUCKET}-${stream}" \
- --project "$GCP_PROJECT" \
- --topic "$BEACON_TOPIC" \
- --message-filter "attributes.eventType = \"OBJECT_FINALIZE\" AND hasPrefix(attributes.objectId, \"${BEACON_GCS_PREFIX}/${stream}/\")" \
- --ack-deadline 60 \
- --message-retention-duration 7d \
- --expiration-period never
- for role in roles/pubsub.subscriber roles/pubsub.viewer; do
- gcloud pubsub subscriptions add-iam-policy-binding "beacon-${BEACON_GCS_BUCKET}-${stream}" \
- --project "$GCP_PROJECT" --member "$BEACON_READER" --role "$role"
- done
- done
- ```
-
- Send Beacon the bucket name, the prefix, and the two subscriptions: `projects/<project-id>/subscriptions/beacon-<bucket>-runtime` and `-inventory`.
- </Tab>
-</Tabs>
-
-## Event file requirements
-
-Use newline-delimited JSON, with one complete JSON object per line. Each event should include a UTC-compatible timestamp and stable fields for user, host, repository, branch, agent runtime, event action, and event category when available.
-
-Share a representative sample file before production ingestion. The sample should show realistic field names and value shapes, but should not include secrets.
-
-Beacon's own S3 forwarder already writes this format, as gzip-compressed JSONL below `<prefix>/runtime/` and `<prefix>/inventory/`.
-
-## Related
-
-<Columns cols={2}>
- <Card title="Beacon Cloud" icon="cloud" href="/deployment/managed">
- Review managed visibility, governance, and investigation workflows.
- </Card>
- <Card title="AWS S3 with Terraform" icon="bucket" href="/log-forwarding/aws-s3-terraform">
- Create the bucket, writer, Beacon reader, and queues with Terraform.
- </Card>
- <Card title="Object Storage Forwarding" icon="bucket" href="/guides/macos-object-storage-system">
- Configure customer-managed Beacon event forwarding into S3.
- </Card>
- <Card title="Endpoint Event Schema" icon="code" href="/telemetry-schema/event-schema">
- Review normalized event fields and example payloads.
- </Card>
-</Columns>
diff --git a/docs/manage/onboarding.mdx b/docs/manage/onboarding.mdx
index 55674433..aafe89f6 100644
--- a/docs/manage/onboarding.mdx
+++ b/docs/manage/onboarding.mdx
@@ -13,7 +13,4 @@ Use these guides to connect managed data sources during Beacon onboarding.
<Card title="Connect Slack" icon="slack" href="/manage/connect-slack">
Send Beacon alert notifications to a Slack channel.
</Card>
- <Card title="Configure S3 Data Connector" icon="bucket" href="/manage/configure-s3-data-connector">
- Grant read-only access for managed event ingestion from S3.
- </Card>
</Columns>