:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Session replay structured data index
The ML mirror extracts a sparse index for full snapshots, $json_ld custom events, and URL changes.
Every version stores separate Parquet index files.
V2 encrypts the projected entries and also retains the original entries in encrypted block metadata.
V3 stores the projected entries as plain Parquet columns.
Use it to find candidate labels and block locations without downloading DOM payloads.
The native anonymizer extracts metadata after scrubbing and passes it beside the serialized recording bytes.
The ordinary replay ingestion path does not extract this index.
See ML replay data for identifier formats and dataset paths.
Storage and lookup
Encrypted v2 index and metadata
V2 writes block-metadata/v2/<YYYY-MM>/part-<writer>-<time>-<sequence>.parquet in the recording bucket.
The partition follows the session's UTC start month, including late arrivals.
The separate eval index uses these paths:
block-metadata-replay-index/v2/2026-09/kind=json_ld/part-<writer>-<time>-<sequence>.parquet
block-metadata-replay-index/v2/2026-09/kind=full_snapshot/part-<writer>-<time>-<sequence>.parquet
block-metadata-replay-index/v2/2026-09/kind=page/part-<writer>-<time>-<sequence>.parquet
The session's UTC start month selects the partition for every kind, including events and uploads in later months.
Each Parquet row exposes real team and session IDs and contains an encrypted array of projected entries from one block and kind.
Use ReplayV2Reader.replay_index(object_keys, kind, excluded_team_ids=training_team_ids) to decrypt and flatten these entries.
The sink applies the URL matching rules below before encryption.
The index uses the recording's session key, so the same consent and deletion checks apply.
The JSON-LD payload remains in the referenced replay block.
For evals, exclude the union of real team IDs from every partition used to train the model. A separate month does not imply a separate team. Apply the exclusion to labels, snapshots, and pages before pairing them. Keep these IDs in the preparation manifest and remove them from model inputs.
Use ReplayV2Reader.metadata() to decrypt selected metadata objects with live key and consent checks.
Each returned block contains raw team and session IDs, its storage key and byte range, replay_index_entries, and replay_index_truncated.
The entries retain their original fields, including windowId, eventTimestamp, eventIndex, and rootTypes.
These original metadata entries are separate from the projected, URL-enriched entries returned by replay_index().
When reading original entries through metadata() instead of the separate index, data preparation must validate and flatten them before applying the pairing rules below.
Attach the block's identifiers and storage location to each entry, and apply the URL matching rules in this guide.
Map windowId to window_id, eventTimestamp to event_ts_ms, eventIndex to event_index, and rootTypes to root_types.
Map the block's replay_index_truncated flag to block_index_truncated.
The encrypted reader returns metadata; it does not flatten the entries.
Use ReplayV2Reader.recording() to fetch and decrypt selected recording blocks before inspecting their events.
Plain v3 index and metadata
A session that starts at or after the v3 cutoff writes to the v3 bucket.
V3 writes block-metadata/v3/<YYYY-MM>/part-<writer>-<time>-<sequence>.parquet and block-metadata-replay-index/v3/<YYYY-MM>/kind=<kind>/part-<writer>-<time>-<sequence>.parquet, partitioned by the session's UTC start month.
Both files use plain Parquet columns with real team and session IDs, so a reader queries them without a key.
The index has the v1 columns, including the scrubbed url.
The metadata keeps the v1 columns, including the scrubbed first_url and urls, and has no replay_index_entries.
Deletion does not remove rows from these files, and the sink does not check session keys for v3 rows.
The recording blocks that the rows of a deleted session point to become unreadable when the session key is shredded and its cached copies expire.
A v3 prefix can also hold objects in the sealed format, mostly under 2026-09.
A sealed object has a payload column that holds an encrypted JSON envelope, and a plain object has no payload column.
A reader checks each object in every month, decrypts a sealed object as it decrypts v2 objects, and reads a plain object directly.
The two formats carry some metadata fields differently:
| Field | Sealed payload (JSON) | Plain object (Parquet) |
|---|---|---|
| Block start and end | first_ts_ms and last_ts_ms, in epoch milliseconds |
first_ts and last_ts, as UTC timestamps |
| Identifier scheme | format_version, which is 2 |
No format_version column: the /v3/ path shows that the team_id and session_id columns hold raw IDs |
| Session start | session_start_ts_ms |
No column: decode it from the UUIDv7 session ID |
| Index entries | replay_index_entries and replay_index_truncated |
No column: read the separate index |
Legacy v1 sparse index
The legacy metadata sink writes the index to the same bucket as block metadata.
Its prefix is <block-metadata-prefix>-replay-index/v1/.
For the default prefix, files have this form:
block-metadata-replay-index/v1/kind=json_ld/session_start_date=2026-09-01/part-<writer>-<time>-<sequence>.parquet
block-metadata-replay-index/v1/kind=full_snapshot/session_start_date=2026-09-01/part-<writer>-<time>-<sequence>.parquet
block-metadata-replay-index/v1/kind=page/session_start_date=2026-09-01/part-<writer>-<time>-<sequence>.parquet
Each legacy row identifies pseudonymized team and session IDs, the recording window, an event timestamp, and a zero-based event index within the decompressed block. The block key and inclusive byte range locate the independently compressed block. Timestamps use doubles so fractional milliseconds survive an exact join. Window IDs match the IDs in the scrubbed recording lines. The index contains no DOM nodes or JSON-LD payload text.
A json_ld row contains an optional url and up to 64 distinct root types.
The schema also accepts full_snapshot_ts_ms, but the URL-only SDK change does not send this optional reference.
Types come from root objects, root arrays, and @graph members.
Nested entity properties, such as a product's offers, do not contribute types.
The types help select candidates; read the payload before deciding which label to use.
The legacy index partitions each row by its session's UTC start date, decoded from UUIDv7.
Entries for one session stay under one date, including separate blocks and late arrivals across midnight.
This partition helps session lookups and cross-block joins. An arrival-date partition would make ingestion-time scans simpler, but would spread one session across dates.
The index omits non-v7 session IDs and starts outside the interval from seven days before the block's last event through that event.
Normal replay storage and block metadata continue for those sessions.
For an event-time search, include the preceding seven session-start dates and filter event_ts_ms.
Full snapshot URLs
The metadata consumer enriches the projected index rows of every version before writing Parquet.
Apply these same matching rules if preparing original entries from decrypted v2 block metadata instead.
For JSON-LD, it copies the URL from the page entry at the same event index, then omits that duplicate page row.
For a full snapshot, it uses a URL entry at the same event index or the immediately preceding event index.
The URL entry must belong to the same window and have a timestamp no later than the snapshot.
An explicit URL already on the index entry takes precedence.
This uses the block metadata that the shared recorder already produces.
It adds no state or behavior to the shared replay recorder and does not change stored rrweb events.
Payload boundaries within a block do not affect the lookup, but any intervening event prevents the preceding-event association.
A block boundary or reversed timestamps can leave the snapshot without a URL.
Readers can use durable page rows to find candidates across blocks, without an ingestion-side session cache.
The sparse index identifies URL-bearing events, not their original rrweb types. An adjacent URL is useful candidate metadata, but does not prove that the source was a Meta event. The shared recorder omits Meta events without URLs, so the index cannot reliably identify every unknown-URL boundary. Payload validation must account for this limit.
Pairing labels with snapshots
Pair labels with snapshots in Athena. Athena reads the plain v1 and v3 index files. It cannot read sealed entries, which are the v2 index and the sealed objects under the v3 prefix. Decrypt those entries with the encrypted reader, and apply the same rules to them.
An Athena table over the v3 index uses these columns and partitions:
CREATE EXTERNAL TABLE replay_index_v3 (
team_id string,
session_id string,
window_id string,
session_start_ts_ms double,
event_ts_ms double,
block_index_truncated boolean,
event_index int,
full_snapshot_ts_ms double,
root_types array<string>,
url string,
block_s3_key string,
block_byte_start double,
block_byte_end double,
payload binary
)
PARTITIONED BY (session_month string, kind string)
STORED AS PARQUET
LOCATION 's3://<v3-bucket>/block-metadata-replay-index/v3/'
TBLPROPERTIES (
'projection.enabled' = 'true',
'projection.session_month.type' = 'date',
'projection.session_month.format' = 'yyyy-MM',
'projection.session_month.range' = '2026-09,NOW',
'projection.session_month.interval' = '1',
'projection.session_month.interval.unit' = 'MONTHS',
'projection.kind.type' = 'enum',
'projection.kind.values' = 'json_ld,full_snapshot,page',
'storage.location.template' = 's3://<v3-bucket>/block-metadata-replay-index/v3/${session_month}/kind=${kind}/'
)
The v3 path does not write its month as key=value, so the table finds each partition through storage.location.template.
The kind partition supplies the kind value, because an Athena table cannot use one name for a column and a partition.
Only a sealed object has a payload value, so the query keeps the rows where payload is null.
A v1 table uses the same columns at block-metadata-replay-index/v1/, with a session_start_date partition in place of session_month.
Set the session months and the training team IDs in index_rows before you run the query.
For v1, filter on session_start_date in place of session_month.
The query finds candidates by URL and time, including separate blocks and out-of-order arrivals.
Its one-second Meta gap and two-second label gap are example selection parameters, not SDK guarantees.
Measure match coverage and audit payloads before choosing the final thresholds.
WITH index_rows AS (
SELECT *
FROM replay_index_v3
WHERE session_month IN ('2026-10', '2026-11')
AND payload IS NULL
AND team_id NOT IN ('<training team ID>')
), labels AS (
SELECT * FROM index_rows WHERE kind = 'json_ld'
), snapshots AS (
SELECT * FROM index_rows WHERE kind = 'full_snapshot'
), pages AS (
SELECT * FROM index_rows WHERE kind = 'page'
), page_urls AS (
SELECT team_id, session_id, window_id, event_ts_ms,
CASE WHEN count(url) = count(*) AND count(DISTINCT url) = 1
THEN min(url) END AS url
FROM pages
GROUP BY team_id, session_id, window_id, event_ts_ms
), snapshot_rows AS (
SELECT DISTINCT team_id, session_id, window_id, event_ts_ms, event_index, url,
block_s3_key, block_byte_start, block_byte_end
FROM snapshots
), snapshot_urls AS (
-- Athena has no ASOF join, so this takes the latest page row at or before each snapshot.
SELECT s.team_id, s.session_id, s.window_id, s.event_ts_ms, s.event_index,
s.block_s3_key, s.block_byte_start, s.block_byte_end,
coalesce(s.url, CASE WHEN s.event_ts_ms - max(p.event_ts_ms) <= 1000
THEN max_by(p.url, p.event_ts_ms) END) AS url
FROM snapshot_rows s
LEFT JOIN page_urls p
ON p.team_id = s.team_id
AND p.session_id = s.session_id
AND p.window_id = s.window_id
AND p.event_ts_ms <= s.event_ts_ms
GROUP BY s.team_id, s.session_id, s.window_id, s.event_ts_ms, s.event_index, s.url,
s.block_s3_key, s.block_byte_start, s.block_byte_end
), incomplete_sessions AS (
SELECT team_id, session_id FROM labels WHERE block_index_truncated
UNION
SELECT team_id, session_id FROM snapshots WHERE block_index_truncated
UNION
SELECT team_id, session_id FROM pages WHERE block_index_truncated
), candidates AS (
SELECT
j.team_id, j.session_id, j.window_id, j.url, j.root_types,
j.event_ts_ms AS label_ts_ms, s.event_ts_ms AS snapshot_ts_ms,
j.block_s3_key AS label_block, j.block_byte_start AS label_start,
j.block_byte_end AS label_end, j.event_index AS label_event_index,
s.block_s3_key AS snapshot_block, s.block_byte_start AS snapshot_start,
s.block_byte_end AS snapshot_end, s.event_index AS snapshot_event_index
FROM labels j
JOIN snapshot_urls s
ON j.team_id = s.team_id
AND j.session_id = s.session_id
AND j.window_id = s.window_id
AND j.url = s.url
AND abs(j.event_ts_ms - s.event_ts_ms) <= 2000
LEFT JOIN incomplete_sessions i
ON i.team_id = j.team_id AND i.session_id = j.session_id
WHERE j.url IS NOT NULL AND j.url <> ''
AND i.team_id IS NULL
)
-- Keep a pair only when no page row with another URL falls between the label and the snapshot.
SELECT DISTINCT c.*
FROM candidates c
LEFT JOIN page_urls p
ON p.team_id = c.team_id
AND p.session_id = c.session_id
AND p.window_id = c.window_id
AND p.event_ts_ms BETWEEN least(c.label_ts_ms, c.snapshot_ts_ms)
AND greatest(c.label_ts_ms, c.snapshot_ts_ms)
AND p.url IS DISTINCT FROM c.url
WHERE p.team_id IS NULL
This query returns candidates, not verified pairs. A matching scrubbed URL and timestamp gap cannot prove that a JSON-LD mutation describes an earlier DOM snapshot. The index cannot prove adjacency across blocks because it omits most event kinds. Missing events, masked Meta events, and URL collisions can also conceal a navigation. Neither an adjacent URL nor a temporal fallback proves the label describes that snapshot. A later query can find late arrivals once both blocks exist; record the input object list to make a dataset run reproducible.
For v1, fetch each distinct block key and inclusive byte range once, then decompress its Snappy JSONL recording lines.
For v2, use the encrypted recording reader to perform the fetch, decryption, and decompression.
Check the window, event kind, and timestamp at each event index before using its payload.
Reject ambiguous matches, including distinct full snapshots that share a timestamp or several plausible snapshots for one label.
A timestamp is not a unique rrweb event ID.
Deduplicate repeated snapshots by content before resolving such ambiguity.
Several JSON-LD scripts can label one snapshot; combine their labels after validation instead of counting separate examples.
Define a label policy for multiple root types and @graph entities before training a single-root classifier.
The query excludes sessions with visible truncated index rows, but cannot detect a missing block or an entirely absent index. For strict examples, inspect the surrounding replay events and reject pairs whose page state cannot be established. Exclude JSON-LD rows without URLs; do not infer those URLs from older SDK navigation events.
Domain and page coverage
Use the url on each json_ld row for site and page coverage.
The SDK captures it in data.href at the same time as the label, after applying replay URL masking and hash settings.
The shared recorder emits a label entry and a URL entry for that event.
The consumer combines them by event index for every version of the separate index.
The anonymizer scrubs it again before extracting index metadata.
This works when navigation events and JSON-LD arrive in separate payloads and needs no session URL cache.
Exclude rows without a usable URL from coverage counts and dataset selection.
Older SDKs do not send this field; URL masking can also omit it.
The index retains those rows, but coverage queries do not infer their URLs from page events.
Every index version omits a page row at the event index of a JSON-LD or full-snapshot row, and moves its URL onto that row.
The original entries inside encrypted v2 block metadata retain both entries.
Count distinct normalized scrubbed URLs as page families. Scrubbing can group similar paths, which helps deduplicate similar pages. These counts do not measure distinct DOM structures. For domain-disjoint datasets, normalize hostnames to registrable domains with a pinned public suffix list, including its private suffix rules. Store a deterministic domain-to-split assignment before sampling examples. Never split individual sessions at random, because sessions from the same site would leak across splits.
The index supplies candidate types, URLs, timestamps, and fetch locations; it does not supply DOM hashes or verified training labels. A dataset builder still needs to:
- Deduplicate index retries, validate candidate pairs, and consolidate their labels.
- Normalize scrubbed URLs and select a representative per page family, with a per-domain cap.
- Deduplicate DOM content globally so copies under different domains cannot leak across splits.
- Balance sampling within the fixed domain splits, then save the selection parameters and source locations in a manifest.
- Report usable examples, sessions, unique page families, domains, and root-type counts for each split, plus rejected and ambiguous candidates.
Keep split assignment stable across resampling and dataset versions. Audit examples where one scrubbed URL has different labels; URL deduplication alone must not silently choose a conflicting label. Exclude cross-split content duplicates or keep their domain groups together before freezing the final split.
The v1 layout supports a bounded-date Parquet scan followed by selective block fetches. V2 readers select monthly index objects, decrypt their entries, and then fetch selected blocks. V3 readers scan the monthly index objects directly, and then fetch selected blocks. No layout provides an S3 point lookup by domain or URL: those filters still scan the selected partitions. Cache the selected index locally for repeated sampling and split experiments. A compacted metadata table can reduce file-listing overhead later without changing SDK capture or replay storage.
Delivery and limits
Each block has a 128 KiB index budget. If it exceeds that budget, its retained entries have block_index_truncated=true.
The recorder still stores all replay events.
The metadata batcher also flushes at 32 MiB of input messages and buffered encrypted index envelopes, checked after each Kafka batch.
This limits accumulation to that threshold plus one input batch; decoded objects and Parquet encoding require additional memory.
The legacy metadata sink writes index partitions sequentially.
The v2 sink writes encrypted index partitions and metadata by session month.
The v3 sink writes plain index partitions and metadata by session month.
Every path commits Kafka offsets only after its storage writes succeed.
A partial upload followed by a retry can produce duplicate rows.
Deduplicate rows by team, session, block key, byte range, event index, and kind before counting them.
Repeated source blocks can also have different storage keys, so dataset preparation still needs content deduplication.
The existing write-error metric includes storage failures.
The following index counters cover every index version.
ml_mirror_replay_index_rows_written_total counts uploaded entries by kind, including retries.
ml_mirror_replay_index_skipped_total counts invalid entries, blocks without a usable session start, and truncated blocks.
These counters measure indexing, not unique sessions or pages.
The consumer can deploy before the producer because index metadata is optional. Deploy the consumer first: an older consumer accepts new metadata but does not write the index. Confirm that the sink's S3 permissions and bucket lifecycle policy cover the new sibling prefix before rollout. This change does not backfill old recordings.
JSON-LD URL rollout
Deploy the anonymizer that scrubs data.href before releasing the SDK change that sends it.
The metadata consumer populates URLs on JSON-LD and full-snapshot index rows for every version.
The separate v2 index encrypts those projected rows before storage.
The existing schema and shared recorder already provide the fields needed by that consumer.
Full-snapshot URL enrichment also works with older SDKs that send Meta events; JSON-LD coverage still requires the SDK URL addition.
Existing events without data.href remain readable and do not contribute to URL-based coverage.