The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
Fix GCS object encoding for HTTP readers
shukieshah82ba703ec3a2
diff --git a/cli/beacon/internal/endpoint/gcs/gcs_test.go b/cli/beacon/internal/endpoint/gcs/gcs_test.goindex 65e57874..1266d595 100644--- a/cli/beacon/internal/endpoint/gcs/gcs_test.go+++ b/cli/beacon/internal/endpoint/gcs/gcs_test.go@@ -102,9 +102,7 @@ func TestVectorConfigUsesGCSCloudStorageSinkAndPreservesJSONShape(t *testing.T)`key_prefix = "${BEACON_GCS_PREFIX:-beacon}/inventory/date=%F/"`,`filename_time_format = "%s"`,`filename_append_uuid = true`,- `filename_extension = "jsonl.gz"`,- `compression = "gzip"`,- `content_encoding = "gzip"`,+ `filename_extension = "jsonl"`,`content_type = "application/x-ndjson"`,`storage_class = "${BEACON_GCS_STORAGE_CLASS:-STANDARD}"`,`codec = "json"`,@@ -120,6 +118,8 @@ func TestVectorConfigUsesGCSCloudStorageSinkAndPreservesJSONShape(t *testing.T)}}for _, forbidden := range []string{+ `compression = "gzip"`,+ `content_encoding`,`.input_tokens =`,`.output_tokens =`,`.cache_read_input_tokens =`,diff --git a/cli/beacon/internal/endpoint/gcs/pack/README.md b/cli/beacon/internal/endpoint/gcs/pack/README.mdindex 79a85944..2bf68176 100644--- a/cli/beacon/internal/endpoint/gcs/pack/README.md+++ b/cli/beacon/internal/endpoint/gcs/pack/README.md@@ -96,8 +96,8 @@ gcloud storage cat "gs://${BEACON_GCS_BUCKET}/${BEACON_GCS_PREFIX}/runtime/smokeFor production, use the generated Vector config as a customer-managed host-agentforwarding template. Beacon remains the local JSONL producer; Vector tails`runtime.jsonl` and `inventory_state.jsonl`, checkpoints file offsets in its-`data_dir`, batches Beacon events, and writes gzip-compressed newline-delimited-JSON objects into Google Cloud Storage.+`data_dir`, batches Beacon events, and writes newline-delimited JSON objects+into Google Cloud Storage.Install Vector using your normal endpoint management tooling, then copy thegenerated config into Vector's config directory. On a macOS system-mode Beacon@@ -140,8 +140,10 @@ parallel usage fields.The template uses date-partitioned `key_prefix`, `filename_time_format = "%s"`,and `filename_append_uuid = true` so production forwarding does not overwrite-previous GCS objects. It also sets `compression = "gzip"`,-`content_encoding = "gzip"`, and `content_type = "application/x-ndjson"`.+previous GCS objects. It writes uncompressed `.jsonl` with+`content_type = "application/x-ndjson"`. This avoids ambiguous GCS+`Content-Encoding` and filename-extension behavior that causes HTTP readers+such as ClickHouse to decompress an object twice.Runtime log forwarding starts at the end of the active log to avoid backfillinghistorical session activity when Vector is first installed. Inventory forwardingstarts at the beginning of `inventory_state.jsonl` so the first snapshot is not@@ -170,7 +172,7 @@ be confirmed with Google Cloud tooling:```bashgcloud storage ls "gs://${BEACON_GCS_BUCKET}/${BEACON_GCS_PREFIX}/runtime/**"-gcloud storage cat "gs://${BEACON_GCS_BUCKET}/${BEACON_GCS_PREFIX}/runtime/date=<date>/<object>.jsonl.gz" | gzip -dc | grep "Beacon endpoint GCS validation event"+gcloud storage cat "gs://${BEACON_GCS_BUCKET}/${BEACON_GCS_PREFIX}/runtime/date=<date>/<object>.jsonl" | grep "Beacon endpoint GCS validation event"```Expected validation fields:diff --git a/cli/beacon/internal/endpoint/gcs/pack/vector.toml.tmpl b/cli/beacon/internal/endpoint/gcs/pack/vector.toml.tmplindex 0e002d6c..34d1db00 100644--- a/cli/beacon/internal/endpoint/gcs/pack/vector.toml.tmpl+++ b/cli/beacon/internal/endpoint/gcs/pack/vector.toml.tmpl@@ -39,9 +39,7 @@ bucket = "${BEACON_GCS_BUCKET}"key_prefix = "${BEACON_GCS_PREFIX:-beacon}/runtime/date=%F/"filename_time_format = "%s"filename_append_uuid = true-filename_extension = "jsonl.gz"-compression = "gzip"-content_encoding = "gzip"+filename_extension = "jsonl"content_type = "application/x-ndjson"storage_class = "${BEACON_GCS_STORAGE_CLASS:-STANDARD}"@@ -70,9 +68,7 @@ bucket = "${BEACON_GCS_BUCKET}"key_prefix = "${BEACON_GCS_PREFIX:-beacon}/inventory/date=%F/"filename_time_format = "%s"filename_append_uuid = true-filename_extension = "jsonl.gz"-compression = "gzip"-content_encoding = "gzip"+filename_extension = "jsonl"content_type = "application/x-ndjson"storage_class = "${BEACON_GCS_STORAGE_CLASS:-STANDARD}"diff --git a/docs/changelog/cli.mdx b/docs/changelog/cli.mdxindex 59560d77..0bcfeabb 100644--- a/docs/changelog/cli.mdx+++ b/docs/changelog/cli.mdx@@ -347,7 +347,7 @@ Stay up to date with the latest changes to the Agent Beacon CLI.### v0.0.38 · June 3, 2026- **Google Cloud Storage forwarding support** · Added `beacon endpoint gcs` commands for generating customer-managed GCS forwarding content over Beacon's local `runtime.jsonl`.-- **GCS content pack** · Added generated setup guidance, a Vector `gcp_cloud_storage` forwarding template, one-shot `gcloud storage` or `gsutil` smoke-test script, and sample events for gzip-compressed NDJSON objects with date-partitioned keys.+- **GCS content pack** · Added generated setup guidance, a Vector `gcp_cloud_storage` forwarding template, one-shot `gcloud storage` or `gsutil` smoke-test script, and sample events for NDJSON objects with date-partitioned keys.- **Customer-managed Google Cloud controls** · Kept Google Cloud credentials, service accounts, workload identity, bucket IAM, lifecycle, retention, and encryption settings outside Beacon endpoint configuration.### v0.0.37 · June 3, 2026diff --git a/docs/cli/gcs.mdx b/docs/cli/gcs.mdxindex 675cceec..672baf57 100644--- a/docs/cli/gcs.mdx+++ b/docs/cli/gcs.mdx@@ -5,7 +5,7 @@ description: "Generate Google Cloud Storage forwarding content and validation ev## Forwarding Command-Use `beacon endpoint gcs` to generate Google Cloud Storage forwarding content for Beacon endpoint events. The generated pack keeps Beacon as a local JSONL producer and helps your customer-managed Vector agent upload runtime and inventory JSONL to a GCS bucket as gzip-compressed NDJSON.+Use `beacon endpoint gcs` to generate Google Cloud Storage forwarding content for Beacon endpoint events. The generated pack keeps Beacon as a local JSONL producer and helps your customer-managed Vector agent upload runtime and inventory JSONL to a GCS bucket as NDJSON.Beacon does not store Google Cloud credentials, service accounts, workload identity settings, bucket IAM, lifecycle rules, retention policies, or encryption settings. Keep those values in Google Cloud, Vector, endpoint-management policy, or deployment tooling.@@ -36,7 +36,7 @@ beacon endpoint gcs [command]## beacon endpoint gcs print-config-`beacon endpoint gcs print-config` prints a Vector configuration that tails the selected Beacon runtime JSONL log and writes gzip-compressed NDJSON objects to Google Cloud Storage.+`beacon endpoint gcs print-config` prints a Vector configuration that tails the selected Beacon runtime JSONL log and writes NDJSON objects to Google Cloud Storage.```bash title="Print the configuration"beacon endpoint gcs print-config@@ -150,7 +150,7 @@ The validation command writes the local event only. Confirm remote delivery with```textgcloud storage ls "gs://${BEACON_GCS_BUCKET}/${BEACON_GCS_PREFIX}/**"-gcloud storage cat "gs://${BEACON_GCS_BUCKET}/${BEACON_GCS_PREFIX}/runtime/date=<date>/<object>.jsonl.gz" | gzip -dc | grep "Beacon endpoint GCS validation event"+gcloud storage cat "gs://${BEACON_GCS_BUCKET}/${BEACON_GCS_PREFIX}/runtime/date=<date>/<object>.jsonl" | grep "Beacon endpoint GCS validation event"```Expected validation fields:diff --git a/docs/concepts/vector-forwarding.mdx b/docs/concepts/vector-forwarding.mdxindex 4f9d96ac..f04843e5 100644--- a/docs/concepts/vector-forwarding.mdx+++ b/docs/concepts/vector-forwarding.mdx@@ -262,7 +262,7 @@ If events do not appear, check the same ownership boundary in order: Beacon shouThe same Vector pattern powers several Beacon content packs:- [AWS S3](/guides/macos-object-storage-system#aws-s3) uses Vector's `aws_s3` sink to write gzip NDJSON objects.-- [Google Cloud Storage](/guides/macos-object-storage-system#google-cloud-storage) uses Vector's [`gcp_cloud_storage` sink](https://vector.dev/docs/reference/configuration/sinks/gcp_cloud_storage/) to write gzip NDJSON objects.+- [Google Cloud Storage](/guides/macos-object-storage-system#google-cloud-storage) uses Vector's [`gcp_cloud_storage` sink](https://vector.dev/docs/reference/configuration/sinks/gcp_cloud_storage/) to write NDJSON objects.- [AWS CloudWatch Logs](/log-forwarding/cloudwatch) uses Vector's `aws_cloudwatch_logs` sink to write parsed Beacon events into a log group.- [Sumo Logic](/log-forwarding/sumo), [Rapid7 InsightIDR](/log-forwarding/rapid7), and [Falcon LogScale](/log-forwarding/falcon) use HTTP-based Vector sinks with destination-specific headers, endpoints, and batching.diff --git a/docs/guides/jamf-gcs-mdm.mdx b/docs/guides/jamf-gcs-mdm.mdxindex 518a6666..45faebf6 100644--- a/docs/guides/jamf-gcs-mdm.mdx+++ b/docs/guides/jamf-gcs-mdm.mdx@@ -218,9 +218,9 @@ gcloud storage ls \--impersonate-service-account "$BEACON_GCS_READER_EMAIL"gcloud storage cat \- "gs://${BEACON_GCS_BUCKET}/${BEACON_GCS_PREFIX}/runtime/date=<YYYY-MM-DD>/<object>.jsonl.gz" \+ "gs://${BEACON_GCS_BUCKET}/${BEACON_GCS_PREFIX}/runtime/date=<YYYY-MM-DD>/<object>.jsonl" \--impersonate-service-account "$BEACON_GCS_READER_EMAIL" |- gzip -dc | grep "Beacon endpoint GCS validation event"+ grep "Beacon endpoint GCS validation event"```The administrator running impersonation also needsdiff --git a/docs/guides/macos-object-storage-system.mdx b/docs/guides/macos-object-storage-system.mdxindex 613055fb..0c05a24d 100644--- a/docs/guides/macos-object-storage-system.mdx+++ b/docs/guides/macos-object-storage-system.mdx@@ -378,8 +378,8 @@ aws s3 cp \gzip -dc | grep "Beacon endpoint S3 validation event"gcloud storage cat \- "gs://${BEACON_GCS_BUCKET}/${BEACON_GCS_PREFIX}/runtime/date=<YYYY-MM-DD>/<object>.jsonl.gz" |- gzip -dc | grep "Beacon endpoint GCS validation event"+ "gs://${BEACON_GCS_BUCKET}/${BEACON_GCS_PREFIX}/runtime/date=<YYYY-MM-DD>/<object>.jsonl" |+ grep "Beacon endpoint GCS validation event"```The endpoint writer identities are intentionally write-only. Run thesediff --git a/docs/log-forwarding/index.mdx b/docs/log-forwarding/index.mdxindex f8523288..01354749 100644--- a/docs/log-forwarding/index.mdx+++ b/docs/log-forwarding/index.mdx@@ -225,7 +225,7 @@ for bucket creation, IAM, credentials, Vector setup, and validation.### Google Cloud Storage-For Google Cloud Storage, generate Beacon's GCS content pack and configure a customer-managed Vector host agent to write gzip-compressed NDJSON objects into a bucket.+For Google Cloud Storage, generate Beacon's GCS content pack and configure a customer-managed Vector host agent to write NDJSON objects into a bucket.```bash title="Generate the integration pack"beacon endpoint gcs install-pack --system --output ./beacon-gcs-packdiff --git a/packaging/macos/jamf/claude/gcs/install-forwarder.sh b/packaging/macos/jamf/claude/gcs/install-forwarder.shindex 01ef5170..ddd49c50 100755--- a/packaging/macos/jamf/claude/gcs/install-forwarder.sh+++ b/packaging/macos/jamf/claude/gcs/install-forwarder.sh@@ -197,9 +197,7 @@ bucket = "\${BEACON_GCS_BUCKET}"key_prefix = "\${BEACON_GCS_PREFIX:-beacon}/runtime/date=%F/"filename_time_format = "%s"filename_append_uuid = true-filename_extension = "jsonl.gz"-compression = "gzip"-content_encoding = "gzip"+filename_extension = "jsonl"content_type = "application/x-ndjson"storage_class = "\${BEACON_GCS_STORAGE_CLASS:-STANDARD}"@@ -228,9 +226,7 @@ bucket = "\${BEACON_GCS_BUCKET}"key_prefix = "\${BEACON_GCS_PREFIX:-beacon}/inventory/date=%F/"filename_time_format = "%s"filename_append_uuid = true-filename_extension = "jsonl.gz"-compression = "gzip"-content_encoding = "gzip"+filename_extension = "jsonl"content_type = "application/x-ndjson"storage_class = "\${BEACON_GCS_STORAGE_CLASS:-STANDARD}"diff --git a/packaging/macos/test-endpoint-scripts.sh b/packaging/macos/test-endpoint-scripts.shindex 03b5d59f..c6cb54cd 100644--- a/packaging/macos/test-endpoint-scripts.sh+++ b/packaging/macos/test-endpoint-scripts.sh@@ -858,7 +858,7 @@ for expected in \'key_prefix = "${BEACON_GCS_PREFIX:-beacon}/runtime/date=%F/"' \'key_prefix = "${BEACON_GCS_PREFIX:-beacon}/inventory/date=%F/"' \'read_from = "beginning"' \- 'compression = "gzip"' \+ 'filename_extension = "jsonl"' \'filename_append_uuid = true' \'[sinks.beacon_runtime_gcs.healthcheck]' \'[sinks.beacon_inventory_gcs.healthcheck]' \@@ -870,8 +870,8 @@ dofidone-if grep -q 'beacon-gcs-test-bucket\|service_account\|private_key\|\.input_tokens =\|\.output_tokens =\|\.cost_usd =' "$GCS_FORWARDER_BASE/gcs-vector.toml"; then- echo "GCS Vector config should not contain destination values, credentials, or parallel usage fields" >&2+if grep -q 'beacon-gcs-test-bucket\|service_account\|private_key\|content_encoding\|compression = "gzip"\|\.input_tokens =\|\.output_tokens =\|\.cost_usd =' "$GCS_FORWARDER_BASE/gcs-vector.toml"; then+ echo "GCS Vector config should not contain destination values, credentials, gzip metadata, or parallel usage fields" >&2exit 1fifor expected in \