:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
PostHog Log Capture Service
A service that receives OpenTelemetry Protocol (OTLP) logs via HTTP and processes them based on team authentication.
Features
- Receives OTLP logs via HTTP on
/v1/logsand/i/v1/logsendpoints - Supports Protobuf and JSON formats
- Supports JSONL (JSON Lines) format for multiple log batches
- Authenticates clients using Bearer tokens or query parameters
- Associates logs with specific team IDs
- Health check endpoints
- Prometheus metrics
Configuration
The service is configured using environment variables:
| Variable | Default | Description |
|---|---|---|
| BIND_HOST | :: | Host to bind the HTTP server |
| BIND_PORT | 4318 | Port for the HTTP server |
| MANAGEMENT_BIND_HOST | :: | Host to bind the health check and metrics server |
| MANAGEMENT_BIND_PORT | 8080 | Port for the health check and metrics server |
| MAX_REQUEST_BODY_SIZE_BYTES | 2097152 | Rejects larger request bodies, before and after gzip decompression |
| FIREHOSE_MAX_REQUEST_BODY_SIZE_BYTES | 8388608 | Body cap for the Amazon Data Firehose route only |
| DROP_EVENTS_BY_TOKEN | (none) | Comma-separated tokens to drop |
| MAX_BACKFILL_DAYS | 0 | How far back a request may ask to keep its own timestamps. 0 refuses every backfill_days request |
Authentication
Clients must authenticate by sending a valid token either:
- In the Authorization header:
Authorization: Bearer your-project-api-key
- As a query parameter:
POST /v1/logs?token=your-project-token
The token is your PostHog project token.
Backdated logs
A log record whose timestamp is more than 24 hours behind the ingest time is normally replaced
with the ingest time, and the original is kept on the $originalTimestamp attribute. That guard
protects every query range on the team from one client with a broken clock.
A historical import needs the original timestamps. Widen the past bound for one request with the
backfill_days query parameter, on POST /v1/logs and POST /i/v1/logs only. The Datadog and
Prometheus routes do not accept it, because the Datadog agent cannot send a query string at all.
Traces and metrics do not accept it either.
POST /i/v1/logs?backfill_days=540
The value must be between 1 and MAX_BACKFILL_DAYS. A request outside that range is answered 400,
rather than being narrowed to the default, because an import whose timestamps are quietly replaced
looks successful and writes most of its records onto the ingest time.
MAX_BACKFILL_DAYS is per-deployment. Which teams may use it is decided by the logs consumer's
LOGS_BACKFILL_ENABLED_TEAMS list, which is empty in production by default. This service has no team
ID, so it marks each message from a backfill_days request with a backfill_days header, and the
logs consumer drops the whole message when the team is not on the list. That drop also happens
after a 200, and is counted per team in the records_dropped_backfill_not_enabled usage metric.
Row age cannot replace the header, because a backfill request can carry recent records.
The future bound stays at 24 hours whatever backfill_days says. A timestamp ahead of the
ingest time is a client clock error in every case, and accepting one would let a single client
write rows past the end of every other query range on the team.
A backfill keeps only the records that are still inside the team's retention. A record whose
timestamp plus its retention is already in the past is dropped by the logs consumer before it is
written, even though this service answered the request with a 200. Dropped records are counted
per team in the records_dropped_retention_expired usage metric. A record that is kept expires
retention_days after it is imported, not after its own timestamp.
Each message carries a min_timestamp header, the earliest record timestamp in it, so the
consumer only decodes and checks the messages that can hold an expired record.
Response codes
| Status | Meaning | Client behavior |
|---|---|---|
| 200 | Accepted | — |
| 400 | Body could not be decoded as OTLP protobuf or JSON, or backfill_days was rejected |
Permanent |
| 401 | No token, or a token that cannot be a project API key (for example a phx_ personal API key) |
Permanent, so the client stops and surfaces the misconfiguration |
| 413 | Body over MAX_REQUEST_BODY_SIZE_BYTES |
Permanent |
The 401 covers shape only: empty, over 64 characters, non-ASCII, containing a null byte, or
prefixed phx_. None of those can be a project API key, so answering 200 and dropping the
records downstream only hides the misconfiguration from whoever is sending the data. This is
the same check, and the same status, that event capture applies to the same input.
One misconfiguration is still answered 200: a well-formed token that belongs to no project. Resolving a token to a team needs Postgres, which this service does not have, so those records are still dropped by the ingestion consumer. Event capture does not validate this at the edge either.
A project over its billing quota is also still answered 200 and dropped by the consumer.
Rejections are counted on capture_logs_requests_rejected_total{reason, signal}, where reason
is one of missing_token, dropped_token or invalid_token.
Running the Service
From Source
cargo run --bin capture_logs
With Docker
docker build -t posthog/capture-logs .
docker run -p 8000:8000 posthog/capture-logs
Sending Logs
You can configure any OpenTelemetry-compatible client to send logs to this service. The service accepts:
Single JSON Format
Standard OTLP ExportLogsServiceRequest as JSON:
curl -X POST http://localhost:8000/v1/logs \
-H "Authorization: Bearer your-api-key" \
-H "Content-Type: application/json" \
-d '{"resourceLogs":[{"resource":{"attributes":[]},"scopeLogs":[{"logRecords":[{"body":{"stringValue":"Hello World"}}]}]}]}'
JSONL Format (JSON Lines)
Multiple ExportLogsServiceRequest objects, one per line:
curl -X POST http://localhost:8000/v1/logs \
-H "Authorization: Bearer your-api-key" \
-H "Content-Type: application/json" \
-d $'{"resourceLogs":[{"resource":{},"scopeLogs":[{"logRecords":[{"body":{"stringValue":"Log 1"}}]}]}]}\n{"resourceLogs":[{"resource":{},"scopeLogs":[{"logRecords":[{"body":{"stringValue":"Log 2"}}]}]}]}'
Protobuf Format
Standard OTLP protobuf encoding is also supported.
Requirements:
- Set the HTTP endpoint to
http://your-service-host:8000/v1/logs - Include your PostHog project token in the Authorization header or as a query parameter
- Use standard OTLP log format (JSON, JSONL, or Protobuf)
Amazon Data Firehose (CloudWatch Logs)
POST /i/v1/logs/aws/firehose implements the Firehose HTTP endpoint destination contract.
A customer creates a Firehose stream with this URL as its HTTP endpoint and the project API key as the stream's access key, then subscribes CloudWatch log groups to the stream.
- Auth: the project API key in
X-Amz-Firehose-Access-Key, or a BearerAuthorizationheader for manual testing. A pastedBearerprefix in the access key field is tolerated. The same shape check as every other route applies. - Each Firehose record is one CloudWatch subscription batch (base64, usually gzip).
Every
logEventsentry becomes a log row.CONTROL_MESSAGErecords are acknowledged and skipped. A record that is not a CloudWatch envelope (VPC flow logs, WAF) is split on newlines. A record that cannot be decoded at all is skipped and counted, so one bad record does not make Firehose retry the good ones. - Row mapping:
body= message;timestamp= the event timestamp;service.nameis inferred from the log group (/aws/lambda/<fn>→<fn>,/aws/rds/instance/<id>/...→<id>, else the group name) unless a Firehose common attributeservice.nameis set. Resource attributes arecloud.provider=aws,cloud.account.id,cloud.region(fromX-Amz-Firehose-Source-Arn),aws.log.group.name,aws.log.stream.name, plus any common attributes; the AWS facts win over a common attribute of the same name. Severity comes from a JSONlevel/severitykey or a leadingERROR/WARN/INFO/DEBUGtoken, elseinfo. - Rows are chunked into Kafka messages sized at half the producer's
message.max.bytes, because a decoded Firehose request can be far larger than one message. Batches are produced concurrently. Delivery is at-least-once: a Kafka failure returns 500, Firehose redelivers the whole request with the same request id, and the batches already produced are duplicated. - Size limits: the request body is capped by
FIREHOSE_MAX_REQUEST_BODY_SIZE_BYTES(default 8 MiB, above the 2 MiB used by the other routes because Firehose buffers whole MiB and base64 adds a third), and the decoded records of one request are capped at eight times that. Either limit answers 413, which Firehose treats as permanent, so the failed batch lands in the customer's S3 backup bucket. - Every response carries the contract body:
{"requestId", "timestamp"}on 200, plus"errorMessage"on 400 (unparseable body, no decodable record), 401 (missing or invalid access key), 413 and 500 (Kafka). Rejections raised by the body-size and decompression layers are re-shaped into the same body. - Recommended stream settings: buffer 1 MB / 60 s, GZIP content encoding, retry 300 s, S3 backup for failed data only.
Metrics: capture_logs_firehose_records_total{kind} (data, control, raw, invalid) and capture_logs_firehose_events_total.
Request outcomes are on the shared http_requests_total{path,status} and rejected tokens on capture_logs_requests_rejected_total.
Endpoints
Log Ingestion
POST /v1/logs- Accept OTLP logs (JSON, JSONL, or Protobuf)POST /i/v1/logs- Alternative endpoint for OTLP logsPOST /i/v1/logs/datadog[/<token>]- Datadog agent intakePOST /i/v1/logs/aws/firehose- Amazon Data Firehose HTTP endpoint destinationOPTIONS /v1/logs- CORS preflight supportOPTIONS /i/v1/logs- CORS preflight support
Management
/- Basic information page/_readiness- Readiness probe for Kubernetes/_liveness- Liveness probe for Kubernetes/metrics- Prometheus metrics
Development
Running Tests
cargo test
Building in Release Mode
cargo build --release