Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

Document S3 token usage forwarding

shukieshah676e065ca46e
diff --git a/cli/beacon/internal/endpoint/s3/pack/README.md b/cli/beacon/internal/endpoint/s3/pack/README.md
index ade89bb3..346e1125 100644
--- a/cli/beacon/internal/endpoint/s3/pack/README.md
+++ b/cli/beacon/internal/endpoint/s3/pack/README.md
@@ -138,6 +138,14 @@ Beacon event per line, without a Vector wrapper. Runtime activity is written
under `${BEACON_S3_PREFIX}/runtime/date=.../`; inventory telemetry is written
under `${BEACON_S3_PREFIX}/inventory/date=.../`.
+When an event contains canonical token or cost attribution under
+`gen_ai.usage`, the Vector template preserves the original nested payload and
+also emits top-level analytics fields for downstream warehouses such as
+ClickHouse: `input_tokens`, `output_tokens`, `cache_read_input_tokens`,
+`cache_creation_input_tokens`, `reasoning_output_tokens`, and `cost_usd`.
+`cost_usd` is runtime-reported only; Beacon does not derive cost from local
+pricing tables.
+
The template uses date-partitioned `key_prefix`, `filename_time_format = "%s"`,
and `filename_append_uuid = true` so production forwarding does not overwrite
previous S3 objects. Runtime log forwarding starts at the end of the active log
diff --git a/docs/cli/s3.mdx b/docs/cli/s3.mdx
index fef09b5b..e1ebf923 100644
--- a/docs/cli/s3.mdx
+++ b/docs/cli/s3.mdx
@@ -82,6 +82,12 @@ beacon endpoint s3 install-pack --output ./beacon-s3-pack
The pack includes setup instructions, a one-shot AWS CLI smoke-test script, a Vector `aws_s3` forwarding template, and sample Beacon endpoint events.
+The generated Vector template preserves each Beacon event's canonical
+`gen_ai.usage` payload and also flattens token and runtime-reported cost fields
+to top-level JSON columns for downstream analytics: `input_tokens`,
+`output_tokens`, `cache_read_input_tokens`, `cache_creation_input_tokens`,
+`reasoning_output_tokens`, and `cost_usd`.
+
### Examples
Generate a content pack for the default per-user install:
diff --git a/docs/log-forwarding/s3.mdx b/docs/log-forwarding/s3.mdx
index b4b3404f..19467ddc 100644
--- a/docs/log-forwarding/s3.mdx
+++ b/docs/log-forwarding/s3.mdx
@@ -117,6 +117,8 @@ In managed deployments, provide `BEACON_S3_BUCKET`, optional `BEACON_S3_PREFIX`,
The template expects a Vector version with the `file` source, `remap` transform, and `aws_s3` sink. It parses each Beacon JSONL line and re-encodes the original Beacon event as JSON with newline-delimited framing so S3 receives one Beacon event per line, without a Vector wrapper.
+When a runtime event includes canonical token or cost attribution under `gen_ai.usage`, the generated Vector config preserves that nested payload and also emits top-level analytics fields for downstream warehouses such as ClickHouse: `input_tokens`, `output_tokens`, `cache_read_input_tokens`, `cache_creation_input_tokens`, `reasoning_output_tokens`, and `cost_usd`. `cost_usd` is runtime-reported only; Beacon does not derive cost from local pricing tables.
+
The template uses date-partitioned `key_prefix`, `filename_time_format = "%s"`, and `filename_append_uuid = true` so production forwarding does not overwrite previous S3 objects. It also sets `compression = "gzip"`, `content_encoding = "gzip"`, and `content_type = "application/x-ndjson"`.
If you adapt the config or use another forwarder, it should: