The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
Document S3 token usage forwarding
shukieshah676e065ca46e
diff --git a/cli/beacon/internal/endpoint/s3/pack/README.md b/cli/beacon/internal/endpoint/s3/pack/README.mdindex ade89bb3..346e1125 100644--- a/cli/beacon/internal/endpoint/s3/pack/README.md+++ b/cli/beacon/internal/endpoint/s3/pack/README.md@@ -138,6 +138,14 @@ Beacon event per line, without a Vector wrapper. Runtime activity is writtenunder `${BEACON_S3_PREFIX}/runtime/date=.../`; inventory telemetry is writtenunder `${BEACON_S3_PREFIX}/inventory/date=.../`.+When an event contains canonical token or cost attribution under+`gen_ai.usage`, the Vector template preserves the original nested payload and+also emits top-level analytics fields for downstream warehouses such as+ClickHouse: `input_tokens`, `output_tokens`, `cache_read_input_tokens`,+`cache_creation_input_tokens`, `reasoning_output_tokens`, and `cost_usd`.+`cost_usd` is runtime-reported only; Beacon does not derive cost from local+pricing tables.+The template uses date-partitioned `key_prefix`, `filename_time_format = "%s"`,and `filename_append_uuid = true` so production forwarding does not overwriteprevious S3 objects. Runtime log forwarding starts at the end of the active logdiff --git a/docs/cli/s3.mdx b/docs/cli/s3.mdxindex fef09b5b..e1ebf923 100644--- a/docs/cli/s3.mdx+++ b/docs/cli/s3.mdx@@ -82,6 +82,12 @@ beacon endpoint s3 install-pack --output ./beacon-s3-packThe pack includes setup instructions, a one-shot AWS CLI smoke-test script, a Vector `aws_s3` forwarding template, and sample Beacon endpoint events.+The generated Vector template preserves each Beacon event's canonical+`gen_ai.usage` payload and also flattens token and runtime-reported cost fields+to top-level JSON columns for downstream analytics: `input_tokens`,+`output_tokens`, `cache_read_input_tokens`, `cache_creation_input_tokens`,+`reasoning_output_tokens`, and `cost_usd`.+### ExamplesGenerate a content pack for the default per-user install:diff --git a/docs/log-forwarding/s3.mdx b/docs/log-forwarding/s3.mdxindex b4b3404f..19467ddc 100644--- a/docs/log-forwarding/s3.mdx+++ b/docs/log-forwarding/s3.mdx@@ -117,6 +117,8 @@ In managed deployments, provide `BEACON_S3_BUCKET`, optional `BEACON_S3_PREFIX`,The template expects a Vector version with the `file` source, `remap` transform, and `aws_s3` sink. It parses each Beacon JSONL line and re-encodes the original Beacon event as JSON with newline-delimited framing so S3 receives one Beacon event per line, without a Vector wrapper.+When a runtime event includes canonical token or cost attribution under `gen_ai.usage`, the generated Vector config preserves that nested payload and also emits top-level analytics fields for downstream warehouses such as ClickHouse: `input_tokens`, `output_tokens`, `cache_read_input_tokens`, `cache_creation_input_tokens`, `reasoning_output_tokens`, and `cost_usd`. `cost_usd` is runtime-reported only; Beacon does not derive cost from local pricing tables.+The template uses date-partitioned `key_prefix`, `filename_time_format = "%s"`, and `filename_append_uuid = true` so production forwarding does not overwrite previous S3 objects. It also sets `compression = "gzip"`, `content_encoding = "gzip"`, and `content_type = "application/x-ndjson"`.If you adapt the config or use another forwarder, it should: