:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Session Recording Ingestion
This module handles ingestion of session recording data, processing it, and storing it for later playback.
Overview
The SessionRecordingIngester consumes session recording events from Kafka and:
- Validates and filters messages (team checks, rate limiting, restrictions)
- Batches events by session for efficient storage
- Writes session data to object storage (S3) as compressed blocks
- Publishes metadata to ClickHouse for querying and playback
- Extracts console logs for separate storage and search
- Handles failures via dead letter queue and overflow topics
Navigation URLs
Navigation URL extraction ignores $json_ld custom events, so their URLs do not add page visits or clear pending dead-click detection.
Other events retain the existing extraction behavior: read data.href, then fall back to data.payload.href.
This includes the $pageview and $url_changed custom events used for SPA navigation.
The ML mirror extracts URL metadata separately through the native anonymizer. A URL on a mirrored JSON-LD event identifies its page; it does not represent a navigation.
Local Development
Prerequisites
- Docker (for local infrastructure)
- Python environment (for Django migrations)
Setup
Start the required services (Kafka, MinIO, Postgres, Redis, ClickHouse):
hogli dev:setupOr manually, with the profile that carries the DynamoDB the keystore tests need:
docker compose -f docker-compose.dev.yml --profile dynamodb upSet up the test database (creates test_posthog DB and runs migrations):
pnpm setup:test
Testing
Running E2E Tests
pnpm jest src/session-recording/consumer.e2e.test.ts
Tests will fail if required infrastructure is not available, with a message indicating which services are missing.
To skip E2E tests when running the full suite:
pnpm jest --testPathIgnorePatterns=e2e
Test Coverage
The E2E tests validate the full pipeline by:
- Producing test messages to the input Kafka topic
- Verifying data is correctly written to S3
- Verifying metadata is correctly aggregated in ClickHouse
- Using snapshot testing to capture and verify behavior
Adding New Test Cases
Add a new entry to the
testCasesarray inconsumer.e2e.test.ts:name: Short identifier used in the snapshot namedescription: What the test verifiescreatePayloads: Function returningPayloadConfig[]with test dataexpectedOutcome:'written'(data should appear in S3) or'dropped'(rejected)
Run tests with the update flag to generate the new snapshot:
pnpm jest src/session-recording/consumer.e2e.test.ts -uReview the generated snapshot in
__snapshots__/to ensure it captures expected behavior.
Updating Snapshots
If refactoring changes the output format (but behavior is correct), update snapshots:
pnpm jest src/session-recording/consumer.e2e.test.ts -u
Always review snapshot diffs carefully - they should reflect intentional changes only.