Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(ai-training): document the v4 replay dataset (#117905)

Nicholas Waltzb10128fa756a
diff --git a/products/ai_training/docs/replay-data.md b/products/ai_training/docs/replay-data.md
index 05b6a080a2c3..acf428237d75 100644
--- a/products/ai_training/docs/replay-data.md
+++ b/products/ai_training/docs/replay-data.md
@@ -312,5 +312,281 @@ The database user needs SELECT and UPDATE only on `posthog_aitrainingdeletionreq
The worker starts with `bin/docker-worker-ai-training-privacy` and does not use shared Django signing secrets.
Its process-local signing key is not used for application requests.
The worker skips general migration checks; the outbox table must exist before deployment.
-All processes that enqueue deletion work, including the general-purpose Temporal worker, need the key table setting.
+All processes that enqueue deletion work, including the general-purpose Temporal worker, need a key table setting: `AI_RESEARCH_REPLAY_KEY_TABLE` for v3 or `AI_RESEARCH_V4_KEY_TABLE` for v4. Either one turns queuing on.
Shared Django and Temporal workers cannot delete keys from the key table.
+
+## v4 dataset
+
+The recorder reads the Kafka key and headers of each message before it reads the payload.
+It records a message only when every check passes, in this order:
+
+1. The session passes the gate below.
+2. Capture set the `token`, `session_id` and `distinct_id` headers.
+3. No event ingestion restriction for `session_recordings` drops the message, as in the main lane.
+4. The team records sessions, has a valid retention period, and its organization opts in to AI training.
+5. The session ID is a recent UUIDv7, as in v3.
+6. The recorder's own new-session limit did not block the session, as described below.
+
+The recorder reads the team list from Postgres every five minutes.
+A refresh that takes more than 60 seconds fails.
+After 20 failed refreshes in a row it admits no team until a refresh succeeds, so an opt-out is never ignored for long.
+It also admits no team when its list is older than 20 refresh intervals plus their timeouts, whatever the refresh task does.
+It consumes the replay topic and its overflow topic.
+
+### New-session limit
+
+The recorder protects the dataset from floods of new sessions with the main lane's rules, applied by itself.
+It does not read the main lane's or the v3 mirror's blocklist.
+
+- A session is new when the recorder has neither seen it nor blocked it.
+- Each new session from the replay topic takes one token from its team's token bucket. A session that gets no token is blocked as a whole, and the recorder drops every later message of it from both topics.
+- A new session from the overflow topic takes no token. The recorder only counts it, as the replay overflow lane does.
+- The bucket holds `AI_RESEARCH_V4_NEW_SESSION_BUCKET_CAPACITY` tokens (default 3000) and refills at `AI_RESEARCH_V4_NEW_SESSION_BUCKET_REPLENISH_RATE` per second (default 300), the main lane's production values. Both scale with the sample percentage, because the limit counts only sessions that pass the gate. With `AI_RESEARCH_V4_NEW_SESSION_BLOCKING_ENABLED` set to false, the recorder counts and blocks nothing.
+- Each pod keeps its own buckets in memory, as the main lane does, so two deployments block slightly different sessions.
+- Seen and blocked sessions are stored in v4's own Redis, `ai-research-v4` (`AI_RESEARCH_V4_REDIS_URL`), not in the primary replay Redis, under `@posthog/ai-research-v4/`. They have the main lane's expiry, so a block survives a restart or a rebalance.
+- That Redis evicts the least recently used keys under memory pressure. An evicted seen key makes its session take a second token, and an evicted blocked key lets its session through, so both fail open.
+- A Redis error lets the session through, as in the main lane.
+
+### Gate
+
+```text
+id = lowercase(session_id) if it has the UUID shape, else session_id
+hash = md5(id) as lowercase hex
+value = integer of the first 8 hex digits
+percentage = (value % 10000) / 100
+include = percentage < AI_RESEARCH_V4_SAMPLE_PERCENTAGE
+```
+
+- A percentage of 100 or more includes every session, and 0 or less includes none. The default is 0.
+- The gate is the same function as the session feature recorder's rollout gate, so at the same percentage the two lanes select the same sessions.
+- Capture keys every replay message with the session ID, so the recorder applies the gate to the Kafka key before it reads the payload.
+- Capture keeps the session ID as the SDK sent it. The replay lanes lower-case an ID with the UUID shape (8-4-4-4-12 hex digits), so the gate does too, and every casing of one session gets the same verdict.
+- Raising the percentage keeps every session that a lower percentage includes.
+
+### Keys
+
+v4 copies the v3 key design: one key per team and UTC month, and one key per session.
+
+| Row | `pk` | `sk` | Attributes |
+| -------------- | -------------------------- | ---------------------- | ----------------------------------------------------------------------------------- |
+| Team month key | `team:{team_id}` | `month:{YYYY-MM}` | `team_id` (N), `key_month` (S), `wrapped_key` (B), `expires_at` (N) |
+| Session key | `team:{team_id}:shard:{n}` | `session:{session_id}` | `team_id` (N), `key_month` (S), `sealed_key` (B), `key_nonce` (B), `expires_at` (N) |
+
+- `n` is the v3 shard: the first 8 hex digits of `sha256(session_id)` as an integer, modulo 32.
+- `key_month` is the UTC month in which the recorder first stored data for the team or session. A session keeps its key in later months.
+- KMS wraps a team month key with the encryption context `{"purpose": "ai-research-v4", "team_id": ..., "key_month": ...}`.
+- A session key is sealed under the team month key with HKDF-SHA256 (empty salt, info `ml-session-key-wrap`) and AES-256-GCM. The additional authenticated data is the canonical JSON of `{"key_month", "purpose": "ai-research-v4", "session_id", "team_id"}`.
+- `expires_at` is the start of the fifth month after `key_month`, in epoch seconds: key month 2026-10 expires at 2027-03-01. DynamoDB TTL removes the row after that. A session writes for at most about 15 days after its key month ends, and an object lives at most 38 days, so every key outlives the objects that use it. The margin lets the object windows go back up to 90 days without a change to the key expiry.
+- A row past `expires_at` that DynamoDB has not removed yet is not used.
+- Writers store keys before the objects that use them, as in v3.
+- Only the recorder creates keys. The image fetcher only reads them, and it drops a job whose team month row is missing or deleted.
+
+The recorder derives two more keys from the team month key with HKDF-SHA256 and an empty salt:
+
+- The image content key, with info `ml-image-content-key`, keys the HMAC of an inline image.
+- The image URL key, with info `ml-image-url-key`, keys the HMAC of a remote image URL.
+
+Image and URL references therefore stay the same for a whole month, and cannot be compared across teams or months.
+
+### Deletion
+
+v4 uses the v3 deletion outbox and worker. The same team, recording and person deletion requests shred v4 keys.
+The worker shreds v4 keys only when `AI_RESEARCH_V4_KEY_TABLE` is set, and a request completes only when every configured key table is done.
+It keeps its v4 progress under its own cursor key, `v4_work`.
+It writes a tombstone and never removes a row:
+
+```text
+UpdateItem SET deleted = :true, expires_at = if_not_exists(expires_at, :expiry) REMOVE wrapped_key, sealed_key, key_nonce
+```
+
+| Request | The worker tombstones | Unreadable after the key caches expire |
+| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
+| Session, and person through its sessions | The session row | The session's blocks and its sealed index values. Its `url_fingerprints` can no longer be linked to a URL |
+| Team | The team month rows of the previous, current and next UTC month, then every `month:` row of the team, then every session row in the 32 shards | Everything of the team: blocks, inline images, fetched images and sealed index values |
+
+- A session tombstone does not reach inline or fetched images. Both are sealed under the team month key, and other sessions of the team can refer to the same image. They stay readable with that key until they expire (see [Lifecycle](#lifecycle)), but the deleted session's blocks, which say where the session used them, cannot be read.
+- A tombstone the worker creates for a row that does not exist yet gets an `expires_at` by the key rule, from the row's month for a team month row and from the current month for a session row. A tombstone that replaces a key keeps the key's expiry.
+- The worker reads only `pk` and `sk` and has no KMS access. Its role has Query and UpdateItem on the v4 key table.
+- A conditional put cannot replace a tombstone, so no writer creates a new key for a deleted row. A writer that meets a tombstone drops the data, and the image fetcher drops a job whose team month row is deleted.
+- A process that already holds a key can use it until its cache entry expires: 5 minutes for a session row, one hour for a team month row. What it writes in that time stays unreadable, because the stored row no longer holds the key.
+- v4 has no month deletion. `delete_ai_training_month` shreds v3 keys only, and v4 objects expire on their own (see [Lifecycle](#lifecycle)).
+- Deletion does not remove index rows or the plaintext columns in them. They expire with the bucket lifecycle.
+
+### Lifecycle
+
+Nothing in v4 is kept for good. Each kind of data has a fixed lifetime, and a deletion only makes data unreadable sooner.
+
+| Data | Where | Lifetime | Removed by |
+| ------------------------------ | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- |
+| Session blocks | `rrweb-v4` `blocks/` | 30 days after the write | S3 expiration |
+| Block and image index | `rrweb-v4` `index/` | 30 days after the write | S3 expiration |
+| Inline image packs | `images-v4` `images/packs/` | 38 days after the write | S3 expiration |
+| Fetched images | `images-v4` `images/url/` | 38 days after the last write. The fetcher copies an image onto itself when a job refers to it and the copy is more than 7 days old | S3 expiration |
+| Team month and session keys | `ai-research-replay-v4-keys` | Until the start of the fifth month after `key_month` | DynamoDB TTL |
+| Tombstones | `ai-research-replay-v4-keys` | The same expiry as the row they replace | DynamoDB TTL |
+| Fetch history and robots cache | `ai-research-v4-image-fetch-history` | URL history: 8 days after the end of its month. Robots and tdmrep cache: at least 30 days | DynamoDB TTL |
+| Index records | Kafka `ai_research_v4_index` | 7 days | Topic retention |
+| Fetch jobs, with raw URLs | Kafka `ai_research_v4_image_fetch`, its retry topics and its DLQ | 24 hours | Topic retention |
+| Seen and blocked sessions | `ai-research-v4` Redis, `@posthog/ai-research-v4/` | The main lane's expiry, or earlier under memory pressure | Redis expiry and LRU eviction |
+| Held keys | Recorder and fetcher memory | 5 minutes for a session row, one hour for a team month row | Cache expiry |
+
+- S3 expiration counts from the time an object was written, rounds up to the next midnight UTC and can run late, so plan for objects to stay for up to 2 days more.
+- Blocks expire by write time, not by session start, so a session whose blocks span several write days loses its first blocks first. Nothing marks the rest as partial, and `cold_start` does not cover it. Readers that need whole sessions follow the rule in [Block index](#block-index).
+- The buckets have no versioning, so an expired object leaves no older version behind. S3 removes the parts of an unfinished upload after one day.
+- `blocks/` is stored in Glacier Instant Retrieval from the start. The index prefixes and both image prefixes stay in Standard, because readers read them often.
+- A block can use an image only while the image outlives the block. A pack is written in the same flush as its blocks and expires 8 days later. A fetched image lives at least 30 days after the last block that refers to it, because the recorder sends a job for each referenced URL at least once per day.
+- A key outlives every object it sealed: a session writes for at most about 15 days after its key month ends, and an object lives at most 38 days plus the S3 delay. The margin lets the object windows go up to 90 days without a change to the key expiry. Raise the object windows in `v4-staging.hcl` in posthog-cloud-infra only together with the key expiry.
+- DynamoDB TTL can remove a row days after `expires_at`. Writers and readers treat a row past `expires_at` as unusable, so the delay only costs storage.
+- The key table has no point-in-time recovery, backups, exports, streams or replicas, so no copy brings a shredded key back.
+
+### Envelope
+
+v4 objects use the v3 `AISR03` frame.
+The additional authenticated data is the canonical JSON of `{"v": 4, "context": ...}`, so a v3 reader never opens a v4 object.
+
+| Object | Key | `context` |
+| ---------------------- | -------------- | --------------------------------------------------------------------------------------------------- |
+| Session block | session key | `teamId`, `sessionId`, `keyMonth`, `kind: "rrweb"`, `codec: "brotli"`, `ref` (the block object key) |
+| Inline image in a pack | team month key | `teamId`, `keyMonth`, `kind: "image"`, `codec: "none"`, `ref` (the image reference) |
+| Fetched image | team month key | `teamId`, `keyMonth`, `kind: "image-url"`, `codec: "none"`, `ref` (the image URL reference) |
+| Sealed index value | session key | `teamId`, `sessionId`, `keyMonth`, `kind: "index"`, `ref` (the column name) |
+
+A sealed index value uses the v3 JSON envelope with `"v": 4`.
+`nodejs/src/ingestion/pipelines/sessionreplay/ai-research/v4-keys/encryption-vector-v4.json` pins one frame of each sealed object kind, so a reader in another language can check its own bytes.
+
+### Blocks
+
+A block holds the replay messages of one session from one Kafka partition in one flush, as JSON lines of `[window_id, event]`, compressed with Brotli at quality 9 and a 22-bit window, as in v3.
+The recorder streams each block into the encoder. Read a block as a stream: a Brotli stream does not state its decompressed size.
+
+- posthog-js compresses some fields of an event and marks the event `cv: "2024-10"`. The recorder expands those fields to plain JSON and removes the `cv` key, as the replay player does, because the gzip strings do not compress again. An event that does not decode stays exactly as it arrived.
+- An inline image becomes `image:v4:{team_id}:{YYYY-MM}:{hash}` in the block, where the month is the UTC month of the write.
+- The recorder takes base64 PNG, JPEG, GIF and WebP data URIs in any letter case and with parameters, under the v3 size limits. It also takes the image frames of the posthog-js canvas recorder. Anything else stays inline in the sealed block, for example SVG or line-wrapped base64.
+- A remote image URL becomes `imageurl:v4:{team_id}:{YYYY-MM}:{hash}` in the block and in its fetch job, where the month is the UTC month of the write. The block and the job always carry the same reference, so a reader opens the fetched image directly.
+- A URL that the URL policy refuses becomes `imageurl:v4:declined:{reason}` in the block, for example `declined:credential`, `declined:non_public_host` or `declined:tracking_beacon`. The original URL cannot be recovered. The recorder counts rewrites and refusals in `ai_research_v4_recorder_image_url_rewrites_total`.
+- The rewrite covers every remote URL that the recorder reads from an image slot: image sources, `srcset`, SVG image links, video posters, `<picture>` sources and CSS `url(...)` in style attributes, style sheets and style rules. It also covers `src`, `srcset`, `href` and `poster` in an attribute change on a node the recorder does not know, for example after a restart. `srcset` keeps its descriptors and CSS keeps its `url(...)` syntax. It is not limited by the 512 URLs per message that get fetch jobs.
+- Some URLs stay as they arrived: relative URLs, remote URLs outside image slots (links, frames, text, CSS custom properties and other CSS properties), the `image-set()` candidates the recorder does not choose, URLs inside a compressed value that did not decode or that the decode budget skipped, and URL text written in another form than the recorder read it, for example with a CSS escape. The recorder counts the last kind as `outcome="unrewritten"`.
+- Before the rewrite, the recorder writes again any JSON string that holds an escape capture would not write, for example `\/`, so an escaped URL is found too. A scheme matches in any letter case.
+- If a refused URL is still in a block after the rewrite, the recorder drops the block and counts it as `dropped_blocks_total{outcome="declined_unrewritten"}`, so no refused URL is stored.
+- The recorder reads object keys as the player does: it writes an escaped key, such as `"\u0073rc"`, in its plain form before it inspects the event, and it takes the last copy of a repeated key.
+- The recorder drops a block that it cannot check in full, and counts the reason in `dropped_blocks_total`: an image-slot URL longer than 8 KiB (`url_too_long`), more than 10,000 image-slot URLs (`too_many_urls`) or more than 2 MiB of them (`too_many_url_bytes`), or a mutation with more than 200,000 attribute changes (`too_many_attribute_items`). It sends no fetch job for a dropped block.
+- When the text of a rewritten URL also appears outside an image slot in the same block, it is replaced there too.
+- `hash` is the first 22 characters of the base64url HMAC-SHA256, as in v3.
+
+### Layout
+
+Every prefix that the recorder writes is partitioned by the UTC day of the write.
+Fetched images are partitioned by the key month of their reference.
+
+| Bucket | Prefix | Contents |
+| ----------- | --------------------------------------------- | ------------------------------------------------------------------------------- |
+| `rrweb-v4` | `blocks/dt={day}/` | One object per recorder flush: sealed blocks, each found by byte range |
+| `rrweb-v4` | `index/blocks/dt={day}/` | Parquet, one row per block |
+| `rrweb-v4` | `index/images/dt={day}/` | Parquet, one row per stored inline image |
+| `images-v4` | `images/packs/dt={day}/{team_id}/` | One object per recorder flush and team: sealed images, each found by byte range |
+| `images-v4` | `images/url/month={YYYY-MM}/{team_id}/{hash}` | One sealed fetched image per URL reference |
+
+S3 expires every object a fixed time after it was written: 30 days in `rrweb-v4` and 38 days in `images-v4`. [Lifecycle](#lifecycle) lists every lifetime and why they fit together.
+
+The recorder does not write the two index prefixes itself, as in v3:
+
+- After a flush stores its packs and blocks, the recorder sends one JSON record per index row to `ai_research_v4_index`. The record's keys are the Parquet column names.
+- The recorder commits without waiting for delivery. A row that is not delivered leaves its block or image stored but not in the index.
+- The v4 Parquet sink reads the topic, groups the rows by index and write day, and writes one file per group and flush as `{ms}-{random}.parquet`. A flush runs every 10 minutes, or at 128 MiB of rows or 1,000,000 rows. The sink commits its offsets only after the files are stored.
+- Index files use four column types: `INT32`, `INT64`, `UTF8` and `BOOLEAN`. Integers are signed. The recorder caps an `INT32` value at the type's range and an `INT64` value at ±(2^53 − 1), so every value is a safe JavaScript integer. It ignores an event timestamp outside the JavaScript date range. Byte counts, offsets and times are `INT64`, the other counts `INT32`. `moments` is a repeated group whose item is named `moments`; an Arrow reader sees a list of structs. `url_fingerprints` and `window_ids` are lists of `UTF8` with no null items. Row groups hold 131072 rows, and every column is snappy-compressed. The sink writes no statistics for `urls`, because a sealed value has no useful order.
+- The sink can write a row twice after a restart. The copies have the same `flush_id`, so the rule below does not remove them: readers first drop rows that repeat within one `flush_id`.
+
+### Block index
+
+A flush can write its blocks and index rows and then fail before it commits its Kafka offsets.
+The next attempt for that partition starts at the same committed offset and writes the same messages again.
+A retry can also cover a different range than the attempt it replaces, for example one flush that replays two earlier flushes.
+Every row therefore names its flush and its Kafka position.
+The index rows are published without waiting for delivery, so a retry can also reach the index only in part.
+Every row of one (`flush_id`, `topic`, `partition`) carries the same `flush_end_offset` and `flush_row_count`, and its own `flush_row_index` from 0.
+Readers drop duplicates with one rule per (`topic`, `partition`), applied to all rows before any session or team filter:
+
+1. Drop rows that repeat a (`flush_id`, `flush_row_index`).
+2. Call a flush complete when it has `flush_row_count` distinct rows. Give each flush the range from its `start_offset` to its `flush_end_offset`.
+3. Walk the complete flushes from the highest (`flushed_at_ms`, `flush_id`) to the lowest, and keep a flush only if its range does not overlap the range of a flush already kept.
+4. Then walk the incomplete flushes in the same order, with the same overlap check. A kept incomplete flush is best effort: some of its blocks are missing from the index, but the blocks it lists are valid.
+
+A flush keeps or loses its whole range. When two flushes overlap, the reader keeps only the one it reaches first, even when their ranges differ in size, so it never merges parts of two flushes.
+
+A reader that needs whole sessions selects them by the start time in the session UUIDv7, not by write day:
+
+- Read the write days from the UTC day of `start - 1 day` onward. The recorder admits a session from 1 day before its start until 14 days after it, so no block of a session has an earlier write day.
+- Use a session only while its start is less than 29 days old. Every block lives at least 30 days after its write, so until then no block has expired.
+- A session can get more blocks until its start is 14 days old. A reader that needs final sessions uses only sessions with a start between 14 and 29 days old.
+
+A session row is a fold over its block rows: sort them by `first_ts_ms`, then merge each into the result in order.
+Counts sum, times take the minimum or maximum, flags take the logical or, `host` takes the first value and the viewport takes the one with the highest `viewport_ts_ms`.
+The DOM sizes `dom_nodes` and `dom_text_chars` and `max_written_event_bytes` take the maximum, because they describe a state or a single event, not an amount.
+`url_fingerprints` and `window_ids` take the ordered union, capped at 10 and 32 entries. A session at the cap has that many or more.
+The flush columns describe a row, not a session, so the fold does not carry them.
+`events_before_first_full_snapshot` sums, because the recorder stops counting it once the session has a decoded full snapshot.
+`max_gap_ms` also counts the gap between one block's last event and the next block's first event. When blocks overlap, for example the main and overflow topics, it is an upper bound.
+`active_ms` is the active time that a block's events add to the session.
+The recorder runs the segmentation of `segmentation.ts` over the whole session, not over one block, so the blocks sum to the session's active time wherever the flushes cut it.
+Blocks that overlap in time, such as main and overflow blocks in one flush, do not count the same time twice.
+A block's `active_ms` can include time before its first event when the block continues an active segment of an earlier block.
+This holds for the blocks that one recorder process inspected. After a rebalance or a restart, up to 5 seconds of a segment that spans the handover is lost. When two processes record overlapping parts of one session, the overlap can count twice.
+`fold_session` in `replay-core-v4` implements this fold.
+
+| Group | Columns |
+| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Location | `team_id`, `session_id`, `key_month`, `block_s3_key`, `block_byte_start`, `block_byte_end` |
+| Kafka | `topic`, `partition`, `start_offset` (the flush's first offset on the partition), `end_offset` (the block's last offset), `flush_end_offset` (the last offset the flush covers), `flush_id`, `flushed_at_ms`, `flush_row_count`, `flush_row_index` |
+| Time | `first_ts_ms`, `last_ts_ms`, `event_count`, `byte_size`, `written_jsonl_bytes`, `max_written_event_bytes` |
+| Source | `snapshot_source`, `snapshot_library`, `host` |
+| Playability | `full_snapshots`, `events_before_first_full_snapshot`, `orphan_mutations`, `max_gap_ms`, `dom_nodes`, `dom_text_chars`, `remote_stylesheets`, `inline_stylesheets`, `iframes`, `canvases`, `videos`, `shadow_roots`, `cold_start`, `node_cap_reached`, `decode_failures`, `window_ids` |
+| Activity | `active_ms`, `clicks`, `mouse_interactions`, `rage_clicks`, `dead_clicks`, `keypresses`, `scrolls`, `mouse_moves`, `touches`, `touch_moves`, `selections`, `media_events`, `viewport_resizes`, `navigations`, `console_errors`, `console_warnings`, `network_requests`, `network_failures` |
+| Context | `viewport_width`, `viewport_height`, `viewport_ts_ms` (the time of the viewport event), `touch_input` |
+| Images | `images_extracted`, `image_bytes`, `images_inline`, `image_urls` |
+| Moments | `moments`: a list of `{kind, window_id, ts_ms, event_index}` for full snapshots, navigations, clicks and large DOM changes, and `moments_truncated` |
+| Sealed | `urls`: the visited URLs as a sealed index value, `urls_truncated`, and `url_fingerprints` |
+
+- `full_snapshots` counts only snapshots that decoded.
+- `events_before_first_full_snapshot` counts the session's events before its first decoded full snapshot.
+- `host` is the host of the first page URL. An IP-address host is stored as `[ip_address]`, as v3 does, and a host longer than 253 bytes is stored as null.
+- `navigations` counts URL changes from Meta events and from posthog-js `$url_changed` events, so single-page apps count too.
+- `node_cap_reached` is true when the recorder could not check a window for orphan mutations: the window passed the node cap, a snapshot or mutation did not decode, the window was evicted and came back, or the recorder dropped the session's DOM state to stay within its memory budget.
+- `byte_size` counts the event bytes as they arrived. `written_jsonl_bytes` counts the block's lines as the recorder wrote them, after it expanded compressed values and removed images, and before Brotli. `max_written_event_bytes` is the longest of those lines. Use these two for memory and size limits.
+- `mouse_interactions` counts every mouse interaction event, of every type. `clicks` counts only clicks. `touch_moves` counts touch move events; `touches` also counts touch starts and ends.
+- `decode_failures` counts, once per event, a compressed value that did not decompress or that the decode budget skipped, a full snapshot the recorder could not decode or walk, a mutation that did not decode or held a node too deep to walk, and a style sheet rule whose compressed value failed. A message that did not read at all reaches no block, so this column does not count it. A block with failures has activity and DOM columns that can be too low.
+- `moments` are hints for where to look, not checked baselines. A `full_snapshot` moment means the recorder decoded that snapshot. `full_snapshot_undecoded` marks an event tagged as a full snapshot that it did not decode. The list keeps the first 256 moments, and `moments_truncated` is true when it dropped more.
+- `window_ids` lists the distinct window IDs of the block's events, the same IDs as in `moments`.
+- `urls` holds at most 100 page URLs of up to 2 KiB each and 32 KiB in all. `urls_truncated` is true when the recorder left pages out.
+- `url_fingerprints` holds up to 10 distinct page URLs of the block as the first 16 hexadecimal characters of an HMAC-SHA256 of the URL. The key is derived from the session key with HKDF-SHA256, empty salt and info `ml-url-fingerprint-key`, so fingerprints can be compared only within one session. They count distinct URLs, where `navigations` counts URL changes.
+- `cold_start` is true when the recorder first sees a session after it started, for example after a partition rebalance, or after it dropped the session's DOM state to stay within its memory budget. Its playability columns then describe only the part of the session the recorder saw.
+
+### Image index
+
+`index/images/` has one row per inline image in a pack: `team_id`, `key_month`, `image_ref`, `pack_s3_key`, `byte_start`, `byte_end`, `mime` and `byte_size`.
+
+- Each frame in a pack is sealed under the team month key of the write month, as a fetched image is. A session deletion therefore does not make inline images unreadable (see [Deletion](#deletion)).
+- `image_ref` uses the team month image content key, so the same image has the same reference in every session of the team month. A pack holds one frame per distinct reference, so an image that two sessions show in one flush is stored once. A later flush stores it again.
+- A reader resolves an image of a block by (`team_id`, `image_ref`) and opens it with the team month key of `key_month`.
+
+### Remote images
+
+The recorder sends each remote image URL that the URL policy accepts to the `ai_research_v4_image_fetch` topic in the v3 job shape, keyed by registrable domain. The job carries the reference that replaces the URL in the block.
+It collects URLs from image sources, `srcset`, SVG image links, CSS `url(...)` values and attribute mutations, and skips hidden or 1×1 images as v3 does.
+The topics hold jobs for 24 hours, because jobs carry raw URLs.
+The v4 image fetcher is its own copy of the v3 fetcher under `ai-research/image-fetch-v4/`, with the v3 fetch policy, its own consumer groups, its own `AI_RESEARCH_V4_IMAGE_FETCH_*` settings and a share of the v3 per-host limits. It shares no code or table with v3.
+It applies the recorder's URL policy through `@posthog/ai-research-v4-native`, an addon over `replay_core_v4::url_policy`, so it canonicalizes a URL, groups it by registrable domain and refuses private hosts exactly as the recorder does.
+It stores each image, sealed and unscrubbed, under `images/url/`, and it does not send images to the scrub lane.
+
+The fetcher keeps a crawl history per URL reference, so per team and month, in its own DynamoDB table `ai-research-v4-image-fetch-history`, with the v3 fetch interval and conditional requests. The table also holds its robots.txt and tdmrep cache:
+
+- The first job for a reference fetches the image from its origin.
+- A later job before the v3 refetch time does not contact the origin. When the stored copy is more than 7 days old, the fetcher copies it onto itself. Otherwise it drops the job.
+- A job after the v3 refetch time sends a conditional request with the stored `ETag` or `Last-Modified`. A `304` refreshes the stored copy by a copy onto itself, and a `200` replaces it.
+- The history item expires 8 days after the end of its month, as v3 history does. A reference carries the month of its write, so no job refers to it after that.
+- An item that does not parse, for example one that a newer build wrote, reads as a miss. The fetcher counts it and fetches the image again.
+- The history holds no URL in plaintext. It binds the stored validators to an HMAC of the response URL under the team month URL key, and sends them only when the job's URL has the same HMAC.
+
+A URL is therefore fetched from its origin about once per team and month, as v3 fetches it about once in 30 days.
+Unlike v3, the history is per team: a sealed image is readable only with its team's key, so two teams that refer to one URL each fetch it.
+The recorder sends a job for each referenced URL at least once per day, so the fetcher sees a reference in every 7-day refresh interval.
+The fetcher drops a job whose team month row is missing, expired or deleted.