Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

110 KB
Raw Blame

ML replay data

The Django controls and their tests live in products/ai_training/backend/, owned by AI Research. Application lifecycle hooks use the product facade; the models share the main database to preserve atomic transactions.

AI Research owns the ML mirror, its encryption keys, and its training readers. The mirror collects scrubbed recordings from organizations that consent to AI training. Its storage and deletion implementation is separate from production session replay.

Session versions

The cutoff is Tuesday, 2026-09-15 at 12:00 UTC, which is 13:00 in Europe/London. The session UUIDv7 timestamp selects the version:

  • Before the cutoff: v1 uses HMAC team and session IDs.
  • At or after the cutoff: v2 uses raw team and session IDs and encrypted payloads.
  • At or after Monday, 2026-09-21 at 17:00 UTC (18:00 in Europe/London): v3 keeps the v2 identifiers and keys, and stores its objects in the v3 buckets under rrweb_3/ and the /v3/ dataset paths. Its metadata catalog and evaluation index are plain Parquet.
  • The ML mirror drops a session if its ID is not UUIDv7 or its start year is beyond 9999.

Event timestamps, arrival times, retries, and flushes do not change the version. Both versions can occur in one ingestion batch. Their replay blocks, images, and metadata use separate storage paths. The version applies to the whole session, including a session that crosses the cutoff. An image reference carries the dataset version of the session that collected it, so a v3 session's images are v3 whatever their month, and the image lanes never read a session ID. The three lanes require AI_RESEARCH_REPLAY_S3_BUCKET at startup, because a v3 session or image has no other place to go.

Consent

Ingestion uses the organization's existing is_ai_training_opted_in flag for both versions. The team lookup caches this flag for up to five minutes. Withdrawing consent stops collection after that refresh and preserves previously collected data and keys. Re-enabling consent resumes existing sessions as well as new sessions, using their existing keys. Messages already admitted to the mirror can finish processing. Session start time selects the dataset version and monthly partition only. There is no separate consent model, DynamoDB consent entry, consent timestamp, or initialization command.

Keys and batch processing

The independent DynamoDB table stores session keys, team image keys, deletion markers, and monthly key indexes. A team image key is wrapped by KMS. A session key is sealed under the team image key of its month, so KMS holds one key per team per month, not one key per session. A session key stored before this change carries its own KMS blob, and every reader opens both shapes. HKDF-SHA256 makes the key that seals session keys from the stored team image key, which still seals image data itself. The recording_blob_ingestion_v2_ml_key_scheme_total metric counts session keys by scheme and leaves team image keys out, so v2 reaches zero when no session key predates v3. ML outputs omit distinct IDs, including their hashes and pseudonyms. The metadata consumer projects supported fields before storage, including for messages already in Kafka. A session has one data key. A team has one image key per session start month. KMS wraps each key it holds with an encryption context that binds the team, the session or month, and the purpose. A sealed session key authenticates the same context. A team can change organization while a session is open, so the organization is not part of that context; keys wrapped before this change carry the organization they were wrapped under on their row, and the mirror unwraps them under it. The replay lane that serves playback encrypts a block with XSalsa20-Poly1305, which is a different format: nothing this lane writes uses it. The authenticated payload also binds the dataset kind and, for images, the object or reference being encrypted. The envelope seals the raw payload with AES-256-GCM. Its additional authenticated data is the JSON of {"v": 3, "context": ...} with sorted keys and no whitespace. An envelope takes one of two shapes, and where it is stored decides which. An object body is a binary frame: the ASCII magic AISR03, the length of the additional authenticated data as a 16-bit big-endian integer, those bytes themselves, the 12-byte nonce, then the sealed bytes followed by the 16-byte tag. The frame carries those bytes verbatim, so a reader passes them to AES-GCM as they are rather than rebuilding the canonical JSON and risking a re-serialization mismatch. A reader must still decode that context and check it equals the object it asked for, because one team month key seals every image object in a flush and the context is the only thing that tells them apart: a reader that skips the check authenticates a shard where it expected an index, or one image's location where it expected another's. A reader must also bound the frame before it slices, because the magic and the length sit outside the authenticated data. In the v2 dataset, a parquet column holds a value and not an object body, so metadata and replay-index stay JSON with v, context, nonce (12 bytes, base64) and ciphertext (the sealed bytes followed by the 16-byte tag, base64). A reader there knows the shape from the column and inspects no magic bytes. The v3 dataset does not seal its metadata or its replay index. Both are plain Parquet columns, described in Data layout and readers. The frame block in nodejs/src/ingestion/pipelines/sessionreplay/ml-mirror/keys/encryption-vector.json pins one frame with a brotli body, so a reader in another language can check its own bytes. It carries the whole frame and also each part on its own: the additional authenticated data as text, the nonce, the sealed bytes, the tag, the brotli body, and the decompressed data. The framed context names a codec, which tells a reader how to expand the plaintext. An rrweb block uses brotli at quality 9. Each lane declares its own codec, so this lane never produces snappy. It reads a block rarely and keeps it for months, so it stores fewer bytes instead. An image object uses none, because an image already arrives compressed. A sealed session key row carries sealed_key and key_nonce, and carries no wrapped_key. HKDF-SHA256 makes its 32-byte wrapping key from the stored team image key, with an empty salt and the info string ml-session-key-wrap. AES-256-GCM then seals the session key under that wrapping key. key_nonce holds the 12-byte nonce, and sealed_key holds the sealed bytes followed by the 16-byte tag. Its additional authenticated data is the JSON of {"purpose": "ai-research-session", "team_id": ..., "session_id": ...} with sorted keys and no whitespace, so every reader rebuilds the same bytes. The seal block in nodejs/src/ingestion/pipelines/sessionreplay/ml-mirror/keys/encryption-vector.json pins one seal, so a reader in another language can check its own bytes.

Ingestion processes key state in batches:

  1. Bulk-read session keys and image keys in one pass. A session's start month names its image key, so the batch knows every row before it reads.
  2. Resolve keys in memory while processing the batch.
  3. Write each new key's month index entry, then the key with a conditional put.
  4. Adopt the key a competing writer stored, which the refused put returns, then publish replay blocks or image messages.

A conditional put refuses to recreate a shredded session key. A shredded team image key stops its team month for good, and this is the reason the table holds no team block row. The shred sets the tombstone and removes the key material in one update. A batch that meets a tombstoned image row does not add that month key to its keys, so it never writes the row again, and a conditional put would refuse it in any case because the row exists. A session in that month then finds no month key, so the batch mints no session key and stores none, and it counts the session as month_key_unavailable. Every reader drops the same sessions, because the seal only opens under a month key that no longer exists. The deletion worker sweeps the team once, and that sweep removes rows to save cost. A key stored after the sweep passes a shard is sealed under a month key the deletion already tombstoned, so a second sweep finds nothing that is readable. A team deletion also closes the previous month, the current one and the next one, because a sweep reaches only an image key that already exists. Ingestion admits a session up to ML_SESSION_MAX_AGE_DAYS old, so those three are every month a later session can still open. Consent stops collection after that, not deletion: a team that keeps its opt-in and keeps sending opens a later month again. Kafka offsets advance only after the required writes and publication succeed. Bulk reads use batches of at most 100 keys. Each new key is one conditional put, so no commit in the fleet waits on another, and the month index entries it needs go in together, at most 25 to a request. Batches overlap, so a session first seen in one batch and also present in the next costs a second conditional put, which loses and settles on the stored key. Reads use strongly consistent BatchGetItem requests with bounded retries for unprocessed keys and for a throttled request. A retry stops when the caller's deadline expires.

The mirror runs each Kafka batch through three stages that each hold one batch at a time, in batch order: prepare (steps 1 and 2, with session tracking), anonymize (the scrub), and commit (steps 3 and 4, then offset tracking and any flush). Neighboring batches overlap across stages, so one batch waits on DynamoDB, KMS, Kafka or S3 while another scrubs. The anonymize stage does not admit a batch while an earlier batch is in it. The record step writes to the recorder that is current at commit time, not the one that was current when the batch was read from Kafka, so a flush between those two moments does not lose the batch.

Ingestion holds a usable session key row and image key row in the process, and a KMS plaintext cache reduces repeated decrypt calls. A row with no wrapped key is never held, so a repaired row is seen at once. A tombstone is held, because a shred only ever sets one and a conditional put cannot overwrite a row that exists, so a deleted session stops costing a read and a refused write on every batch. A session key deleted out of band stays usable in a process that already read it, until that entry expires. ROW_CACHE_LIFETIME_MS therefore sets how soon ingestion observes a session deletion. A sealed session key keeps that same bound, because its seal lives on the session row and nothing else holds it. The team image key stays cached far longer, but it opens no session on its own. A team image key is held for an hour, because it is one row per team per month and a longer lifetime costs far fewer reads. A shred removes the durable key, so a process that runs on a held row of either kind writes data that no reader can open. Data written under such a key stays unreadable, because the envelope stores no key, and the stored row is a tombstone with no wrapped key, no seal, and no nonce. Training readers do not use this cache. Each process limits KMS concurrency and request rate; deployment capacity must account for the sum across replicas. Readers check live state before each batch and permit key use for at most five minutes from the start of that read. An expired read must obtain permission again.

The key table has no TTL or point-in-time recovery. Its resource policy denies backups, exports, and enabling continuous backups or Kinesis copies. Do not copy wrapped keys into object storage, logs, workflow payloads, or another persistent cache. A restored key defeats deletion, whether KMS wrapped it or a team image key sealed it.

Deletion

Existing recording, person, team, and organization deletion flows enqueue ML deletion work. Person deletion resolves the combined supplied and profile distinct IDs through the replay ClickHouse index. This lookup also runs when the user does not select replay deletion and includes IDs with no remaining person profile. Only the resulting session IDs enter the ML deletion outbox. The lookup runs in a background task, so person deletion waits only until the task is queued. The task carries the distinct IDs, because the profile they belong to is deleted by then. A failed lookup retries with backoff for about 24 hours, then fails the task, which the task failure metric reports. Team deletion and its deletion outbox request commit in the same database transaction. If the outbox write fails, team deletion fails and can retry. Consent changes do not enqueue deletion requests. The outbox survives removal of the source team or organization. Its team IDs refer to the original environment, without resolving a child environment to its parent.

Scope Effect
Session Remove its wrapped key, its seal, and its nonce, then permanently block that session ID.
Person Resolve its session IDs through replay, then apply session deletion to each result.
Team Remove its session keys and its image keys, and close the current month and the next one.

The person lookup matches any available replay row for the requested IDs that began before the deletion, then deduplicates and paginates sessions. It does not filter out recordings marked deleted or past their replay retention date while their index rows remain. Replay index retention limits this lookup: after those rows expire, person deletion cannot discover the corresponding ML sessions. New sessions from the same person can be collected if consent permits them. Team lookups use 32 session shards with strongly consistent DynamoDB queries. Deleting a resolved session removes all of its events, including events from other identities.

Deletion uses resumable query pages and stores progress in the outbox. A failed request backs off without blocking unrelated requests. Completion follows key removal and the five-minute reader lifetime. Scrubbed images remain available after session or person deletion, but become unreadable after team or month deletion.

This mechanism covers encrypted v2 and v3 objects. It does not erase legacy plaintext objects, plain v3 metadata and index rows, previously downloaded data, derived training artifacts, or a trained model. Legacy dataset retirement needs a separate storage operation before claiming deletion across the entire bucket.

Monthly key deletion

Key creation writes the month index entry with a plain put, then the wrapped key with a conditional put. The index entry comes first, so every stored key has an index entry. An index entry without a key is harmless: the month sweep leaves a tombstone that a later key put respects. The index uses 32 partitions named month:<YYYY-MM>:shard:<0..31> and stores key locations, without copying wrapped keys. Session keys and image keys appear in this index.

Run python manage.py delete_ai_training_month YYYY-MM to remove the keys of that UTC session month. The mirror drops a session whose ID started more than 14 days in the past or more than 1 day in the future, and the command accepts a month from 14 days and one hour after the month ends, so a batch admitted just inside the limit cannot commit a key after its index shard was swept. Neither side reads a shared block item for the month, because every commit in the fleet would contend on that one DynamoDB item. The command uses strongly consistent queries and bounded writes. Rerun the command after an interrupted run; it safely repeats completed pages. Rerun it once for any month that an earlier version of the command deleted, because readers no longer honor the month block that version wrote. A reader that already cached a key can still use it. The matching monthly S3 folders can then be removed from each dataset. Deleting a month does not affect another month's image keys.

Data layout and readers

All v2 S3 datasets use a YYYY-MM directory derived from the session UUIDv7 start timestamp in UTC. A session that crosses a month boundary stays in its start month, including late blocks and image fetches.

Dataset Default path Encryption key
Replay blocks rrweb_2/<month>/ Session
Metadata catalog block-metadata/v2/<month>/ Each row's payload uses its session key
Evaluation index block-metadata-replay-index/v2/<month>/kind=<kind>/ Each block and kind uses its session key
Inline image shards scrubbed-images/v2/<month>/<team>/shards/ Team and session month
Inline image lookups scrubbed-images/v2/<month>/<team>/lookup/<hash>.encrypted Team and session month
Inline image indexes scrubbed-images/v2/<month>/<team>/index/ Team and session month
URL images scrubbed-images/v2/<month>/<team>/url/<hash> Team and session month

A v3 session uses the same layout in the v3 buckets: blocks under rrweb_3/<month>/, and the metadata catalog, the evaluation index and every image path with /v3/ in place of /v2/. The v3 metadata catalog and the v3 evaluation index use no key. Some v3 objects under those two paths are sealed, mostly under 2026-09. A sealed object has a payload column that holds an encrypted JSON envelope, as in v2. A reader checks each object for a payload column in every month. See the structured data index. A reader finds the bucket of a block in its block_url, and resolves an image:v3: or imageurl:v3: reference in the v3 images bucket under the /v3/ paths.

A v2 metadata catalog exposes raw team_id, session_id, format_version, and an encrypted payload. URLs, block locations, and replay indexes are inside that payload. Neither the catalog nor its encrypted payload includes a distinct-ID field. V2 does not write a separate plaintext replay index.

A v3 metadata catalog stores each block as plain Parquet columns: raw team_id and session_id, the block location, timestamps, counts, and the scrubbed first_url and urls. It has no distinct-ID field and no replay index entries. The v3 evaluation index stores its entries as plain Parquet columns with raw team and session IDs and the scrubbed URL. Deletion does not remove v3 metadata or index rows. The sink reads no session key for a v3 row, so it also writes rows for a session that was deleted after the mirror produced them. Those rows hold no personal data, and the recording blocks they point to become unreadable when the session key is shredded and the cached copies of that key expire.

Athena can select v2 catalog rows but cannot decrypt replay fields. It can query every column of a plain v3 object. To leave out sealed v3 objects, include a payload column in the table and filter on payload IS NULL. Training readers must bulk-read live keys and deletion markers before decrypting. If a download exceeds the key read lifetime, readers must check live eligibility again before decryption. Cross-account readers use the full DynamoDB table ARN and the prod-us KMS key ARN. Both accounts must authorize the reader role. Readers have key-read and decrypt permissions; they cannot create keys or change deletion state.

Use metadata block locations and byte ranges to fetch recordings, then decrypt before decompressing. Read metadata from the session start month, including blocks that arrive in later months.

Join analytics on both raw team and session IDs. Remove identifiers from model inputs. Resolve image references before training because they contain team IDs.

Images and Kafka

A reference names its dataset version: image:v2:<team>:<month>:<hash> and imageurl:v2:<team>:<month>:<hash> for a v2 session, image:v3:... and imageurl:v3:... for a v3 session. The version is part of the reference, so the fetch frontier and every dedup cache treat a v3 reference as new even when a v2 session already stored the same image, and the v3 dataset gets its own copy. Images do not deduplicate across teams or session months. Kafka records between the ML lanes travel in cleartext. In S3, recording blocks and images are sealed, and stored scrubbed images use team image keys. V2 metadata and the v2 evaluation index are sealed. V3 metadata and the v3 evaluation index are plain Parquet, apart from the sealed v3 objects that Data layout and readers describes. Consumers reject malformed UUIDv7 session identifiers before reading DynamoDB. Oversized identifiers cannot fail a whole bulk key lookup. Inline images have an encrypted lookup for each reference, published after the shard and its index. Readers fetch that lookup directly; a missing image does not require a scan of the team's image history. The image scrubber hands the images it scrubbed to a write lane every 30 seconds or when its buffer is full, writes them while it scrubs the next batches, and writes the shard groups of one hand-off concurrently. The offsets of a hand-off are stored after its writes complete, in hand-off order, so a failed write stops every later store and the pod replays from the last stored offset. Source deduplication uses the image reference, which names the team and the session month, so one fetch serves every session of that team month. Session deletion does not remove images, so a session cannot lose an image that another session fetched. The v2 image-fetch frontier uses a separate, initially empty DynamoDB history table. Its URL history expires eight days after the end of the session's UTC month. HTTP freshness and cache directives do not change this expiry, because the stored image for a team, month and URL is write-once. A no-store response leaves no cache metadata in that history. Robots.txt and TDM reservation caches keep their shared origin keys and existing expiry rules. It does not inherit v1 seen flags or successful fetch results.

ML Kafka producers write ai_research_ingestion_version: 1 or 2. Retries and dead-letter replay preserve this header and the record bytes. Consumers drop records that still use the sealed envelope shape from before cleartext records, and count them in recording_blob_ingestion_v2_ml_legacy_envelopes_dropped_total. Headerless queued messages mean v1. Unknown versions are rejected, and so is an image reference whose version does not match the header. For a v2 session, the metadata sink resolves each row's session key from the row's own team and session identifiers before it seals the row for Parquet; a row whose key is deleted is dropped. A v3 row needs no key, because the sink writes it as plain Parquet.

Legacy image references and paths remain available for v1 sessions. Their HMAC key must remain stable while that data is in use.

Configuration

New key manager and v2 storage settings use the AI_RESEARCH_REPLAY_* prefix:

  • KEY_TABLE, KMS_KEY_ARN, and AWS_REGION select the key store and wrapping key.
  • KEY_CACHE_MAX, KEY_CACHE_LIFETIME_MS, and KMS_REQUESTS_PER_SECOND bound the KMS plaintext cache and KMS traffic.
  • ROW_CACHE_MAX and ROW_CACHE_LIFETIME_MS bound the stored key row cache. The lifetime applies to a session key row and is capped; a team image key row is held for up to 48 hours. A value that is not a positive integer stops the consumer at startup and names the setting.
  • IMAGE_FETCH_V2_DYNAMODB_TABLE selects the fresh v2 frontier.
  • S3_PREFIX selects v2 replay storage and defaults to rrweb_2.
  • S3_BUCKET names the v3 bucket of a lane. A session that started at or after the v3 cutoff writes there, and v2 keeps its own bucket. The mirror and the metadata sink use the recording bucket, which holds AISR03 frames for recording blocks and Parquet for the v3 metadata catalog and evaluation index. The image scrubber uses the images bucket, which holds AISR03 frames. The mirror, the sink and the image scrubber stop at startup when it is empty.
  • S3_V3_PREFIX selects the block prefix inside the v3 bucket and defaults to rrweb_3. Each dataset version has its own prefix, so a bucket policy grants only the versions it holds.

The v2 producer requires AI_RESEARCH_REPLAY_KEY_TABLE and AI_RESEARCH_REPLAY_KMS_KEY_ARN at startup. Missing values stop startup before it consumes Kafka messages.

Established HMAC settings retain their transition aliases. When both aliases are set, the AI_RESEARCH_REPLAY_* value takes precedence, including an explicit empty value. The wrapped HMAC secret keeps the single name SESSION_RECORDING_ML_PSEUDONYM_WRAPPED_KEY in both the environment and secret store. It has no new alias. Renaming configuration must not rotate that key.

The shared ML server configuration applies the legacy aliases before explicit server overrides. An image scrubber with the key manager enabled must configure SESSION_RECORDING_ML_IMAGE_SCRUB_DLQ_TOPIC before startup. Malformed encrypted images retain their original payload and headers in the dead-letter queue. The scrubber retries failed dead-letter writes and interrupts retry waits during shutdown. The image fetch consumer dead-letters unsupported ingestion version headers while processing other valid records in the batch. A v2 message without key manager configuration still fails the batch because it needs the missing encryption settings.

V2 data uses YYYY-MM directories from the session UUIDv7 start timestamp in UTC. Recording blocks, metadata, image shards, image lookups and URL images retain that month across late arrivals. V2 image references include <team>:<month>:<hash>, so the image-fetch seen history is independent for each month. Robots.txt and TDM reservation caches remain shared by origin across months.

JSON-LD evaluation index

The v2 metadata consumer writes a separate encrypted index for json_ld, full_snapshot, and page entries. The path uses the session's UTC start month, even when events or uploads fall in a later month. Each Parquet row exposes real team_id and session_id values and encrypts the projected entries with the session key. Session, person, team, and month deletion therefore remove access to the index along with its recording. The JSON-LD payload remains in the referenced recording block.

The v3 metadata consumer writes the same three kinds as plain Parquet columns with real team and session IDs. Athena can pair v3 labels with snapshots by URL and time. See the pairing query. Deletion does not remove v3 index rows. It makes the recording blocks that they point to unreadable, as Data layout and readers describes.

Use the real team ID to exclude all teams present in the model's training data before selecting eval examples. The exclusion must cover every training month, not only the eval partition's month. For v2, use the encrypted reader for index entries. For v3, read the Parquet columns directly. Then fetch selected recording blocks to inspect their JSON-LD payloads. Legacy v1 indexes retain their pseudonymized identifiers and daily partitions.

Deletion worker isolation

The deletion task uses the ai_research_privacy Celery queue. In prod-us, only the dedicated ai-research-privacy-worker deployment consumes this queue. Its service account has a dedicated IAM role and cloud-database user. The database user needs SELECT and UPDATE only on posthog_aitrainingdeletionrequest. The worker starts with bin/docker-worker-ai-training-privacy and does not use shared Django signing secrets. Its process-local signing key is not used for application requests. The worker skips general migration checks; the outbox table must exist before deployment. All processes that enqueue deletion work, including the general-purpose Temporal worker, need a key table setting: AI_RESEARCH_REPLAY_KEY_TABLE for v3 or AI_RESEARCH_V4_KEY_TABLE for v4. Either one turns queuing on. Shared Django and Temporal workers cannot delete keys from the key table.

v4 dataset

The recorder reads the Kafka key and headers of each message before it reads the payload. It records a message only when every check passes, in this order:

  1. The session passes the gate below.
  2. Capture set the token, session_id and distinct_id headers.
  3. No event ingestion restriction for session_recordings drops the message, as in the main lane.
  4. The team records sessions, has a valid retention period, and its organization opts in to AI training.
  5. The session ID is a recent UUIDv7, as in v3.
  6. The recorder's own new-session limit did not block the session, as described below.

The recorder reads the team list from Postgres every five minutes. A refresh that takes more than 60 seconds fails. After 20 failed refreshes in a row it admits no team until a refresh succeeds, so an opt-out is never ignored for long. It also admits no team when its list is older than 20 refresh intervals plus their timeouts, whatever the refresh task does. It consumes the replay topic and its overflow topic.

New-session limit

The recorder protects the dataset from floods of new sessions with the main lane's rules, applied by itself. It does not read the main lane's or the v3 mirror's blocklist.

  • A session is new when the recorder has neither seen it nor blocked it.
  • Each new session from the replay topic takes one token from its team's token bucket. A session that gets no token is blocked as a whole, and the recorder drops every later message of it from both topics.
  • A new session from the overflow topic takes no token. The recorder only counts it, as the replay overflow lane does.
  • The bucket holds AI_RESEARCH_V4_NEW_SESSION_BUCKET_CAPACITY tokens (default 3000) and refills at AI_RESEARCH_V4_NEW_SESSION_BUCKET_REPLENISH_RATE per second (default 300), the main lane's production values. Both scale with the sample percentage, because the limit counts only sessions that pass the gate. With AI_RESEARCH_V4_NEW_SESSION_BLOCKING_ENABLED set to false, the recorder counts and blocks nothing.
  • Each pod keeps its own buckets in memory, as the main lane does, so two deployments block slightly different sessions.
  • Seen and blocked sessions are stored in v4's own Redis, ai-research-v4 (AI_RESEARCH_V4_REDIS_URL), not in the primary replay Redis, under @posthog/ai-research-v4/. They have the main lane's expiry, so a block survives a restart or a rebalance.
  • That Redis evicts the least recently used keys under memory pressure. An evicted seen key makes its session take a second token, and an evicted blocked key lets its session through, so both fail open.
  • A Redis error lets the session through, as in the main lane.

Gate

id         = lowercase(session_id) if it has the UUID shape, else session_id
hash       = md5(id) as lowercase hex
value      = integer of the first 8 hex digits
percentage = (value % 10000) / 100
include    = percentage < AI_RESEARCH_V4_SAMPLE_PERCENTAGE
  • A percentage of 100 or more includes every session, and 0 or less includes none. The default is 0.
  • The gate is the same function as the session feature recorder's rollout gate, so at the same percentage the two lanes select the same sessions.
  • Capture keys every replay message with the session ID, so the recorder applies the gate to the Kafka key before it reads the payload.
  • Capture keeps the session ID as the SDK sent it. The replay lanes lower-case an ID with the UUID shape (8-4-4-4-12 hex digits), so the gate does too, and every casing of one session gets the same verdict.
  • Raising the percentage keeps every session that a lower percentage includes.

Keys

v4 copies the v3 key design: one key per team and UTC month, and one key per session.

Row pk sk Attributes
Team month key team:{team_id} month:{YYYY-MM} team_id (N), key_month (S), wrapped_key (B), expires_at (N)
Session key team:{team_id}:shard:{n} session:{session_id} team_id (N), key_month (S), sealed_key (B), key_nonce (B), expires_at (N)
  • n is the v3 shard: the first 8 hex digits of sha256(session_id) as an integer, modulo 32.
  • key_month is the UTC month in which the recorder first stored data for the team or session. A session keeps its key in later months.
  • KMS wraps a team month key with the encryption context {"purpose": "ai-research-v4", "team_id": ..., "key_month": ...}.
  • A session key is sealed under the team month key with HKDF-SHA256 (empty salt, info ml-session-key-wrap) and AES-256-GCM. The additional authenticated data is the canonical JSON of {"key_month", "purpose": "ai-research-v4", "session_id", "team_id"}.
  • expires_at is the start of the fifth month after key_month, in epoch seconds: key month 2026-10 expires at 2027-03-01. DynamoDB TTL removes the row after that. A session writes for at most about 15 days after its key month ends, and an object lives at most 38 days, so every key outlives the objects that use it. The margin lets the object windows go back up to 90 days without a change to the key expiry.
  • A row past expires_at that DynamoDB has not removed yet is not used.
  • Writers store keys before the objects that use them, as in v3.
  • Only the recorder creates keys. The image fetcher only reads them, and it drops a job whose team month row is missing or deleted.

The recorder derives two more keys from the team month key with HKDF-SHA256 and an empty salt:

  • The image content key, with info ml-image-content-key, keys the HMAC of an inline image.
  • The image URL key, with info ml-image-url-key, keys the HMAC of a remote image URL.

Image and URL references therefore stay the same for a whole month, and cannot be compared across teams or months.

Deletion

v4 uses the v3 deletion outbox and worker. The same team, recording and person deletion requests shred v4 keys. The worker shreds v4 keys only when AI_RESEARCH_V4_KEY_TABLE is set, and a request completes only when every configured key table is done. It keeps its v4 progress under its own cursor key, v4_work. It writes a tombstone and never removes a row:

UpdateItem  SET deleted = :true, expires_at = if_not_exists(expires_at, :expiry) REMOVE wrapped_key, sealed_key, key_nonce
Request The worker tombstones Unreadable after the key caches expire
Session, and person through its sessions The session row The session's blocks and its sealed index values. Its url_fingerprints can no longer be linked to a URL
Team The team month rows of the previous, current and next UTC month, then every month: row of the team, then every session row in the 32 shards Everything of the team: blocks, inline images, fetched images and sealed index values
  • A public font file (see Web fonts) belongs to no team, so no deletion reaches it. It is a file that Google serves to anyone, and the blocks that say where a session used it are sealed.
  • A session tombstone does not reach inline or fetched images. Both are sealed under the team month key, and other sessions of the team can refer to the same image. They stay readable with that key until they expire (see Lifecycle), but the deleted session's blocks, which say where the session used them, cannot be read.
  • A tombstone the worker creates for a row that does not exist yet gets an expires_at by the key rule, from the row's month for a team month row and from the current month for a session row. A tombstone that replaces a key keeps the key's expiry.
  • The worker reads only pk and sk and has no KMS access. Its role has Query and UpdateItem on the v4 key table.
  • A conditional put cannot replace a tombstone, so no writer creates a new key for a deleted row. A writer that meets a tombstone drops the data, and the image fetcher drops a job whose team month row is deleted.
  • A process that already holds a key can use it until its cache entry expires: 5 minutes for a session row, one hour for a team month row. What it writes in that time stays unreadable, because the stored row no longer holds the key.
  • v4 has no month deletion. delete_ai_training_month shreds v3 keys only, and v4 objects expire on their own (see Lifecycle).
  • Deletion does not remove index rows or the plaintext columns in them. They expire with the bucket lifecycle.

Lifecycle

Nothing in v4 is kept for good. Each kind of data has a fixed lifetime, and a deletion only makes data unreadable sooner.

Data Where Lifetime Removed by
Session blocks rrweb-v4 blocks/ 30 days after the write S3 expiration
Block and image index rrweb-v4 index/ 30 days after the write S3 expiration
Inline image packs images-v4 images/packs/ 38 days after the write S3 expiration
Fetched images images-v4 images/url/ 38 days after the last write. The fetcher copies an image onto itself when a job refers to it and the copy is more than 7 days old S3 expiration
Fetched fonts images-v4 fonts/url/ The same as fetched images S3 expiration
Team month and session keys ai-research-replay-v4-keys Until the start of the fifth month after key_month DynamoDB TTL
Tombstones ai-research-replay-v4-keys The same expiry as the row they replace DynamoDB TTL
Fetch history and robots cache ai-research-v4-image-fetch-history URL history: 8 days after the end of its month, and a public font's history until its next fetch. Robots and tdmrep cache: at least 30 days DynamoDB TTL
Index records Kafka ai_research_v4_index 7 days Topic retention
Fetch jobs, with raw URLs Kafka ai_research_v4_image_fetch, its retry topics and its DLQ 24 hours Topic retention
Seen and blocked sessions ai-research-v4 Redis, @posthog/ai-research-v4/ The main lane's expiry, or earlier under memory pressure Redis expiry and LRU eviction
Held keys Recorder and fetcher memory 5 minutes for a session row, one hour for a team month row Cache expiry
  • S3 expiration counts from the time an object was written, rounds up to the next midnight UTC and can run late, so plan for objects to stay for up to 2 days more.
  • Blocks expire by write time, not by session start, so a session whose blocks span several write days loses its first blocks first. Nothing marks the rest as partial, and cold_start does not cover it. Readers that need whole sessions follow the rule in Block index.
  • The buckets have no versioning, so an expired object leaves no older version behind. S3 removes the parts of an unfinished upload after one day.
  • blocks/ is stored in Glacier Instant Retrieval from the start. The index prefixes and both image prefixes stay in Standard, because readers read them often.
  • A block can use an image only while the image outlives the block. A pack is written in the same flush as its blocks and expires 8 days later. A fetched image lives at least 30 days after the last block that refers to it. Each day that a block refers to an image, the recorder sends a job for it, or the fetcher's history shows a stored copy that is younger than 7 days at the end of that day.
  • A key outlives every object it sealed: a session writes for at most about 15 days after its key month ends, and an object lives at most 38 days plus the S3 delay. The margin lets the object windows go up to 90 days without a change to the key expiry. Raise the object windows in v4-staging.hcl in posthog-cloud-infra only together with the key expiry.
  • DynamoDB TTL can remove a row days after expires_at. Writers and readers treat a row past expires_at as unusable, so the delay only costs storage.
  • The key table has no point-in-time recovery, backups, exports, streams or replicas, so no copy brings a shredded key back.

Envelope

v4 objects use the v3 AISR03 frame. The additional authenticated data is the canonical JSON of {"v": 4, "context": ...}, so a v3 reader never opens a v4 object.

Object Key context
Session block session key teamId, sessionId, keyMonth, kind: "rrweb", codec: "brotli", ref (the block object key)
Inline image in a pack team month key teamId, keyMonth, kind: "image", codec: "none", ref (the image reference)
Fetched image team month key teamId, keyMonth, kind: "image-url", codec: "none", ref (the image URL reference)
Fetched font team month key teamId, keyMonth, kind: "font-url", codec: "none", ref (the font URL reference)
Sealed index value session key teamId, sessionId, keyMonth, kind: "index", ref (the column name)

A sealed index value uses the v3 JSON envelope with "v": 4. nodejs/src/ingestion/pipelines/sessionreplay/ai-research/v4-keys/encryption-vector-v4.json pins one frame of each sealed object kind, so a reader in another language can check its own bytes.

Blocks

A block holds the replay messages of one session from one Kafka partition in one flush, as JSON lines of [window_id, event], compressed with Brotli at quality 9 and a 22-bit window, as in v3. The recorder streams each block into the encoder. Read a block as a stream: a Brotli stream does not state its decompressed size.

  • posthog-js compresses some fields of an event and marks the event cv: "2024-10". The recorder expands those fields to plain JSON and removes the cv key, as the replay player does, because the gzip strings do not compress again. An event that does not decode stays exactly as it arrived.

  • An inline image becomes image:v4:{team_id}:{YYYY-MM}:{hash} in the block, where the month is the UTC month of the write.

  • The recorder takes PNG, JPEG, GIF and WebP data URIs in any letter case and with parameters, under the v3 size limits. It decodes the JSON escapes of each string before it looks for data URIs, so an escaped letter does not hide one. It also reads the CSS escapes in the scheme, the media type and the parameters, skips the tabs and line breaks there, and reads parameters of any length, as a browser does. Only CSS reads a CSS escape as its character. In an attribute such as src it is text, so a data URI there whose header holds a CSS escape becomes image:v4:declined:undecodable, up to the end of its string. It reads a payload as base64 only when ;base64 ends the header, as a browser does. It reads a base64 payload across line breaks, escaped slashes and percent escapes, and without its padding. It reads a payload that is not base64 as percent-encoded bytes. The image bytes name the format, not the data URI's label, so the image index mime is always one of these four.

  • An inline image that the recorder does not store becomes a marker that names the reason, and its bytes are stored nowhere. A srcset loses such a candidate. The recorder counts these as ai_research_v4_recorder_removed_images_total{reason}.

    • image:v4:declined:avif: the label or the bytes name AVIF. The dataset keeps no AVIF image.
    • image:v4:declined:over_limit: the image is larger than 900 KiB, or its message already holds 4,096 images or 64 MiB of them.
    • image:v4:declined:undecodable: the payload does not decode, or it holds an escape that the recorder does not read, for example a CSS escape in the payload.
    • image:v4:declined:unsupported: the bytes are not PNG, JPEG, GIF or WebP, for example a font labeled image/ttf, an icon or a bitmap.
  • An SVG data URI whose payload starts with markup (<) stays inline, because SVG is text that the text scrub reads in place. Any other payload under an SVG label follows the rules for raster images, because a browser shows raster bytes as an image whatever the label says. A data URI with an empty payload holds no image, so it stays as it is.

  • The recorder reads every argument of a canvas mutation, not only the frames of the posthog-js canvas recorder. A buffer becomes a reference when its bytes are PNG, JPEG, GIF or WebP, and a marker when they are not. The pixels of an ImageData, and the numbers of a Uint8Array or a Uint8ClampedArray, become image:v4:declined:unsupported. A data URI in the src of an HTMLImageElement moves out like any data URI, and a remote URL there stays as it is.

  • A remote image URL becomes imageurl:v4:{team_id}:{YYYY-MM}:{hash} in the block and in its fetch job, where the month is the UTC month of the write. The block and the job always carry the same reference, so a reader opens the fetched image directly.

  • A URL that the URL policy refuses gets no reference and no fetch job. It becomes imageurl:v4:declined:{reason} or fonturl:v4:declined:{reason} in the block, for example declined:non_public_host or declined:tracking_beacon, and the original URL cannot be recovered.

  • A URL that the policy refuses only for a credential keeps its place without its credentials: its userinfo, each credential parameter of its query and fragment, such as X-Amz-Signature or #access_token=, and each path segment that holds a token, such as ;jsessionid=. A relative URL counts as on any host, so /p/{token}/n/ loses its token as an Oracle Object Storage path does. The rest stays as the page wrote it. When the policy still reads a credential after those cuts, as in a path that hides a token behind a dot segment, only the URL's origin stays. When the policy refuses the URL that is left, for example for its host, the slot holds declined:credential. The recorder reads each URL with the URL parser, as the policy does, so it sees a host after IDNA maps it and a path once it is decoded and its dot segments are gone. A URL that the URL parser does not read, or one longer than 2,048 characters, can hold a credential that the recorder cannot see, so it becomes declined:not_absolute or declined:too_long, also where it shows no image. Blocks that an earlier recorder wrote hold declined:credential for every credential URL. The recorder counts rewrites and refusals in ai_research_v4_recorder_image_url_rewrites_total.

  • Each src URL of an @font-face rule becomes a font reference in the same way, as Web fonts describes. ai_research_v4_recorder_image_url_rewrites_total counts both, by kind: image, font or bundled_font.

  • The v4_image_refs module of the replay-core-v4 crate defines each reference and marker shape, for images and for fonts. The recorder writes them with it. A reader that must tell them apart from page text uses its parse function, which accepts only the exact strings the recorder writes.

  • The rewrite covers every remote URL that the recorder reads from an image slot: image sources, srcset, SVG image links, video posters, <picture> sources and CSS url(...) in style attributes, style sheets and style rules, in custom properties too, because a page passes a custom property to an image property through var(). It also covers src, srcset, href and poster in an attribute change on a node the recorder does not know, for example after a restart. For the same reason, a <source> added under such a node is read as a <picture> source, and a text change on such a node loses the credentials of the url()s, the image-set() strings and the @imports in it, because the node can be the text of a <style>. Every other byte of that text stays, so it gets no references and an image-set() keeps each candidate, because it may not be CSS. srcset keeps its descriptors and CSS keeps its url(...) syntax. A class such as bg-[url(https://cdn.example.com/a.png)] and its rule's selector keep the URL as the page wrote it, because a selector is no URL slot. It is not limited by the 512 URLs per message that get fetch jobs.

  • A srcset in an image slot keeps one candidate: the one that Chromium shows at a device pixel ratio of 1, which is the ratio the screenshotter renders at. Of x descriptors, that is the smallest density at or above 1x, or the largest density when all are below 1x. A w descriptor gives a density only against the layout width, which the recorder does not know, so of w descriptors the recorder keeps the v3 choice: the largest at or below 1024w, or the smallest when all are wider. The first candidate wins a tie. Only that candidate gets a fetch job, because the renderer has no network access and shows only fetched images. A srcset that mixes w and x descriptors, or that v3 would read as malformed, keeps no candidate.

  • The kept candidate stays as its reference when the URL policy accepts it, or as it is when it is an inline image that the image scan keeps. A kept candidate that the policy refuses leaves the srcset empty. An inline image that the scan declines leaves the srcset empty, and no other candidate takes its place, because no other candidate has a fetch job. A candidate that leaves becomes no marker.

  • HTML adds the non-empty src of an img as a 1x candidate when its srcset has no 1x candidate and no w descriptor. At a device pixel ratio of 1 the img then shows its src, so the recorder fetches the src and empties the srcset.

  • A visible img that shows its kept candidate gets no fetch job for its src or rr_src, as in v3. They still become references. Any other img with a src gets a fetch job for it. The recorder decides this on the srcset and the src as the event holds them, before the inline images leave, and keeps the same candidate in the stored block.

  • An attribute change that sets the srcset but not the src does not hold the src, so the recorder decides as for an img with no src. If the img has a src that wins over the new srcset, the replay shows that src, which has a fetch job only if an earlier event gave it one.

  • When an attribute change shows a hidden img, the recorder decides this on the srcset and the src after the change.

  • The image scan can still decline an inline kept candidate when its message is past the caps. The srcset is then empty, and the img shows no image.

  • An inline image in a candidate that leaves still moves to the image pack, where no reference names it.

  • A <picture> source whose type names a media type the image fetcher does not store, such as image/avif, gets no fetch job and loses its remote candidates. Its kept candidate stays only when it is inline, such as an SVG data URI. The writer rebuilds each srcset that the inspector read in an image slot, where the inspector read it. A srcset key elsewhere, for example in a custom event's payload, stays as it is. A browser skips a source with no candidates, so the replay falls back to the next source or the img. The recorder counts these as ai_research_v4_recorder_srcset_rewrites_total{outcome="trimmed"|"emptied"}.

  • The writer rewrites each CSS text where the inspector read it, so that an offline render loads only images the block refers to. An image-set() keeps one candidate, because a renderer picks the candidate that fits its pixel ratio and does not fall back when that candidate fails. It keeps the highest resolution of a type the image fetcher stores among the candidates that can load: inline, accepted by the URL policy, or root-relative on a known page. Only that candidate gets a fetch job, and the other candidates leave the set. A set that holds an inline image keeps all its candidates, and each remote one gets its reference.

  • A root-relative CSS url(/…) resolves against the origin of its window's latest page URL, from a Meta event or a custom event that names a page, such as $url_changed, with https in place of the page's scheme. It gets what the absolute URL gets: its reference, or the marker or the URL without its credentials when the URL policy refuses it. A C0 control or a space before the /, which the URL parser strips, does not change this. A path-relative url(…) stays as the page wrote it, so that no part of a page path, which can hold a token, reaches a fetch. rrweb writes snapshot CSS with absolute URLs, but sends rule text, adopted style sheets and style declarations as the page wrote them. Each window resolves its own texts, so the same rule in windows on two origins gets two references. An event resolves at most 1,024 relative URLs, a flush keeps at most 64 MiB of resolved URLs, and a page URL longer than 2,048 characters resolves none.

  • The recorder reads CSS with cssparser, which tokenizes as CSS Syntax 3 does, so names, escapes, comments, strings and bad URLs read as a browser reads them. So background-\69mage, \75 rl(…) and background/**/:url("…"/**/) count, and so does a url( that the end of the text closes. A bad URL such as url(a(1).png), which no browser loads, does not count. An image URL that itself holds an escape gets the reference of the URL it decodes to, as a font URL does, and its whole url() is rewritten. A block whose image-set() holds such a candidate, or a candidate such as cross-fade() with a url() inside, gets unread_reason, because the set keeps its candidate as the page wrote it. So does a block with an escaped URL that decodes to a remote URL that the URL parser does not read.

  • The writer rewrites only whole url() and image-set() functions, found as the inspector finds them, inside JSON strings that hold CSS: the values of _cssText, style (style diffs included), textContent, rule, replace, replaceSync and value. So a class name such as bg-[url(/a.png)], a quoted string or a comment keeps its text, and the class's rule still matches. The recorder counts these as ai_research_v4_recorder_css_rewrites_total{kind="image_set"|"relative_url", outcome="rewritten"}.

  • The recorder reads a protocol-relative URL, such as //cdn.example.com/a.png, as https. It gets the same reference as the https form, and the rewrite replaces the text as the block holds it. A browser reads a backslash as a slash there, so \\cdn.example.com/a.png counts too, and so does a URL such as https:\\cdn.example.com/a.png with any mix of slashes and backslashes after the scheme, or none: on a page of another scheme the browser loads https:cdn.example.com/a.png from that host. The URL parser drops ASCII tabs and line breaks anywhere in a URL, and C0 controls and spaces at either end, so the recorder drops them before it decides that a URL is remote, and /\n/cdn.example.com/a.png and \u0001https://cdn.example.com/a.png count too.

  • Some URLs get no reference and keep their place without their credentials: relative URLs outside CSS, path-relative CSS URLs, root-relative CSS URLs that go past a budget or come before this process saw their window's page URL (after a restart or an eviction), and remote URLs in attributes that show no image. Those attributes are href and src of links, frames, scripts and other elements, action, formaction, data, background, cite, longdesc, manifest, codebase, icon, imagesrcset and the URL lists of ping and attributionsrc, and the data- attributes that lazy loaders use, such as data-src and data-srcset. So do url()s in CSS properties that show no image, such as filter, in SVG presentation attributes such as fill, stroke, filter, clip-path, mask and marker-end, and in @import. A URL with no credential there keeps every byte. On a page of its scheme a browser reads https:a/b, with no two slashes after the scheme, as a path on the page's host, so the recorder looks for a credential in both readings, and a URL whose credential is in what the URL parser alone reads as its host becomes its decline marker. The candidates of an image-set() that does not parse stay as they arrived.

  • The writer writes again any JSON string that holds an escape capture would not write, for example \/. A scheme matches in any letter case.

  • v4 stores pages unscrubbed, so the recorder removes credentials only from URLs where a page can load them: URL attributes and CSS. A copy of a URL elsewhere, such as in alt or title text, a text node, a CSS selector, a custom event, a console log or a network body, keeps its text.

  • The recorder reads object keys as the player does: it writes an escaped key, such as "\u0073rc", in its plain form before it inspects the event, and it takes the last copy of a repeated key.

  • The inspector records where each image slot, srcset and CSS string is, and the writer copies each event once and writes each reference, marker or URL without its credentials there. So a block has no limit on its URLs, and the writer finds no URL by its text. A URL of any length is named; one longer than the fetcher takes, 2,048 characters, becomes imageurl:v4:declined:too_long.

  • A block that the recorder cannot read in full is stored, and its index row names the reason in unread_reason, because a URL in the part it did not read went unchecked and can stay as it arrived, with its credentials: CSS that nests rules or functions 75 deep (too_many_nested_blocks), compressed JSON that nests more than 1,024 deep, which the message read cannot see (too_deep_json), a mutation with more than 200,000 attribute changes (too_many_attribute_items), an event whose compressed strings did not decode or that a decode budget skipped (undecoded_events), a block whose edits do not fit in the flush's 256 MiB edit budget (too_many_edits), and a slot that the writer could not find again (unplaced_slot). The other URLs of such a block keep their references. A block past the edit budget is the exception: it keeps no edits, so all its URLs stay as they arrived, and it gets no fetch jobs, because no reference would name the files. The recorder counts these as unread_blocks_total{outcome}. A block is dropped only when sealing it panics, which dropped_blocks_total{outcome} counts.

  • hash is the first 22 characters of the base64url HMAC-SHA256, as in v3.

Layout

Every prefix that the recorder writes is partitioned by the UTC day of the write. Fetched images and fonts are partitioned by the key month of their reference. A public font file has no month.

Bucket Prefix Contents
rrweb-v4 blocks/dt={day}/ One object per recorder flush: sealed blocks, each found by byte range
rrweb-v4 index/blocks/dt={day}/ Parquet, one row per block
rrweb-v4 index/images/dt={day}/ Parquet, one row per stored inline image
rrweb-v4 index/blocks/_watermarks/ JSON, one object per Kafka partition: how far the block index is complete
rrweb-v4 index/images/_watermarks/ JSON, one object per Kafka partition: how far the image index is complete
rrweb-v4 index/_recorder_watermarks/dt={day}/ JSON, one object per input Kafka partition: how far the recorder is complete
images-v4 images/packs/dt={day}/{team_id}/ One object per recorder flush and team: sealed images, each found by byte range
images-v4 images/url/month={YYYY-MM}/{team_id}/{hash} One sealed fetched image per URL reference
images-v4 fonts/url/month={YYYY-MM}/{team_id}/{hash} One sealed fetched font per font URL reference
images-v4 fonts/url/public/{hash} One unsealed font file per public font reference, for every team

S3 expires every object a fixed time after it was written: 30 days in rrweb-v4 and 38 days in images-v4. Lifecycle lists every lifetime and why they fit together.

The recorder does not write the two index prefixes itself, as in v3:

  • After a flush stores its packs and blocks, the recorder sends one JSON record per index row to ai_research_v4_index. The record's keys are the Parquet column names.
  • The recorder commits without waiting for delivery. A row that is not delivered leaves its block or image stored but not in the index.
  • The v4 Parquet sink reads the topic, groups the rows by index and write day, and writes one file per group and flush as {ms}-{random}.parquet. A flush runs every 10 minutes, or at 128 MiB of rows or 1,000,000 rows. The sink commits its offsets only after the files are stored.
  • Index files use four column types: INT32, INT64, UTF8 and BOOLEAN. Integers are signed. The recorder caps an INT32 value at the type's range and an INT64 value at ±(2^53 − 1), so every value is a safe JavaScript integer. It ignores an event timestamp outside the JavaScript date range. Byte counts, offsets and times are INT64, the other counts INT32. moments is a 3-level LIST (moments, then the repeated group list, then the struct element), so Athena, Arrow readers and DuckDB see a non-null list of structs. Files written before the sink used that layout hold moments as a bare repeated group whose item is named moments. Arrow readers and DuckDB read those as the same list of structs, but Athena reads the column as null, because Trino reads a bare repeated group of more than one field as a missing column. Those files expire with the bucket's lifecycle. url_fingerprints and window_ids are lists of UTF8 with no null items. The three surfacing sets are nullable 3-level LISTs of UTF8 or INT64 with no null items. Row groups hold 131072 rows, and every column is zstd-compressed at level 9. Older files can still be snappy-compressed and in arrival order until they expire; Parquet records the codec per column chunk, so a reader handles both. The sink writes no statistics for urls, because a sealed value has no useful order.
  • The sink can write a row twice after a restart. The copies have the same flush_id, so the rule below does not remove them: readers first drop rows that repeat within one flush_id.

Block index

A flush can write its blocks and index rows and then fail before it commits its Kafka offsets. The recorder keeps running and tries to commit those offsets again later, and it exits only when commits keep failing for 10 minutes. When the partition moves first, or the process restarts, the next attempt for that partition starts at the same committed offset and writes the same messages again. A retry can also cover a different range than the attempt it replaces, for example one flush that replays two earlier flushes. Every row therefore names its flush and its Kafka position. The index rows are published without waiting for delivery, so a retry can also reach the index only in part. flush_id is a random 64-bit number for each attempt, written as 16 lowercase hexadecimal digits, so its text order is its numeric order. Every row of one (flush_id, topic, partition) carries the same flush_end_offset and flush_row_count, and its own flush_row_index from 0. Readers drop duplicates with one rule per (topic, partition), applied to all rows before any session or team filter:

  1. Drop rows that repeat a (flush_id, flush_row_index).
  2. Call a flush complete when it has flush_row_count distinct rows. Give each flush the range from its start_offset to its flush_end_offset.
  3. Walk the complete flushes from the highest (flushed_at_ms, flush_id) to the lowest, and keep a flush only if its range does not overlap the range of a flush already kept.
  4. Then walk the incomplete flushes in the same order, with the same overlap check. A kept incomplete flush is best effort: some of its blocks are missing from the index, but the blocks it lists are valid.

A flush keeps or loses its whole range. When two flushes overlap, the reader keeps only the one it reaches first, even when their ranges differ in size, so it never merges parts of two flushes.

Each file holds its rows in (team_id, session_id, first_ts_ms) order, with a null first_ts_ms last, so the blocks of a session in one file are adjacent and in time order. The order holds only within a file. The blocks of a session spread over many files and write days, and a file is usually one row group, so its min and max statistics do not narrow a lookup.

A reader that needs whole sessions selects them by the start time in the session UUIDv7, not by write day:

  • Read the write days from the UTC day of start - 1 day onward. The recorder admits a session from 1 day before its start until 14 days after it, so no block of a session has an earlier write day.
  • The recorder watermark files give the last write day to read for every block captured before a cutoff. A recorder that lags writes blocks to write days after their capture day.
  • Use a session only while its start is less than 29 days old. Every block lives at least 30 days after its write, so until then no block has expired.
  • A session can get more blocks until its start is 14 days old. A reader that needs final sessions uses only sessions with a start between 14 and 29 days old.

A session row is a fold over its block rows: sort them by first_ts_ms, then merge each into the result in order. Counts sum, times take the minimum or maximum, flags take the logical or, host takes the first value and the viewport takes the one with the highest viewport_ts_ms. The DOM sizes dom_nodes and dom_text_chars and max_written_event_bytes take the maximum, because they describe a state or a single event, not an amount. url_fingerprints and window_ids take the ordered union, capped at 10 and 32 entries. A session at the cap has that many or more. The surfacing sums add, its two maxima take the maximum, and its three sets take the union with no cap. The flush columns describe a row, not a session, so the fold does not carry them. events_before_first_full_snapshot sums, because the recorder stops counting it once the session has a decoded full snapshot. max_gap_ms also counts the gap between one block's last event and the next block's first event. When blocks overlap, for example the main and overflow topics, it is an upper bound. active_ms is the active time that a block's events add to the session. The recorder runs the segmentation of segmentation.ts over the whole session, not over one block, so the blocks sum to the session's active time wherever the flushes cut it. Blocks that overlap in time, such as main and overflow blocks in one flush, do not count the same time twice. A block's active_ms can include time before its first event when the block continues an active segment of an earlier block. This holds for the blocks that one recorder process inspected. After a rebalance or a restart, up to 5 seconds of a segment that spans the handover is lost. When two processes record overlapping parts of one session, the overlap can count twice. fold_session in replay-core-v4 implements this fold.

Group Columns
Location team_id, session_id, key_month, block_s3_key, block_byte_start, block_byte_end
Kafka topic, partition, start_offset (the flush's first offset on the partition), end_offset (the block's last offset), flush_end_offset (the last offset the flush covers), flush_id, flushed_at_ms, flush_row_count, flush_row_index
Time first_ts_ms, last_ts_ms, event_count, byte_size, written_jsonl_bytes, max_written_event_bytes
Source snapshot_source, snapshot_library, host
Playability full_snapshots, events_before_first_full_snapshot, orphan_mutations, max_gap_ms, dom_nodes, dom_text_chars, remote_stylesheets, inline_stylesheets, iframes, canvases, videos, shadow_roots, cold_start, node_cap_reached, decode_failures, unread_reason, window_ids
Activity active_ms, clicks, mouse_interactions, rage_clicks, dead_clicks, keypresses, scrolls, mouse_moves, touches, touch_moves, selections, media_events, viewport_resizes, navigations, console_errors, console_warnings, network_requests, network_failures
Context viewport_width, viewport_height, viewport_ts_ms (the time of the viewport event), touch_input
Images images_extracted, image_bytes, images_inline, image_urls
Moments moments: a list of {kind, window_id, ts_ms, event_index} for full snapshots, navigations, clicks and large DOM changes, and moments_truncated
Sealed urls: the visited URLs as a sealed index value, urls_truncated, and url_fingerprints
Surfacing The surfacing_ columns: what ingestion's surfacing feature recorder records for the block as one batch. See Surfacing features
  • full_snapshots counts only snapshots that decoded.
  • events_before_first_full_snapshot counts the session's events before its first decoded full snapshot.
  • host is the host of the first page URL. An IP-address host is stored as [ip_address], as v3 does, and a host longer than 253 bytes is stored as null.
  • navigations counts URL changes from Meta events and from posthog-js $url_changed events, so single-page apps count too.
  • node_cap_reached is true when the recorder could not check a window for orphan mutations: the window passed the node cap, a snapshot or mutation did not decode, the window was evicted and came back, or the recorder dropped the session's DOM state to stay within its memory budget.
  • images_inline counts the SVG data URIs that stay inline in the block. An image that becomes a marker counts in no column.
  • byte_size counts the event bytes as they arrived. written_jsonl_bytes counts the block's lines as the recorder wrote them, after it expanded compressed values and removed images, and before Brotli. max_written_event_bytes is the longest of those lines. Use these two for memory and size limits.
  • mouse_interactions counts every mouse interaction event, of every type. clicks counts only clicks. touch_moves counts touch move events; touches also counts touch starts and ends.
  • decode_failures counts, once per event, a compressed value that did not decompress or that the decode budget skipped, a full snapshot the recorder could not decode or walk, a mutation that did not decode or held a node too deep to walk, and a style sheet rule whose compressed value failed. A message that did not read at all reaches no block, so this column does not count it. A block with failures has activity and DOM columns that can be too low.
  • unread_reason names why the recorder could not read a block in full, and is null otherwise. A session row keeps the reason of its first such block. A reader must not treat the URLs of such a block as checked.
  • moments are hints for where to look, not checked baselines. A full_snapshot moment means the recorder decoded that snapshot. full_snapshot_undecoded marks an event tagged as a full snapshot that it did not decode. The list keeps the first 256 moments, and moments_truncated is true when it dropped more.
  • window_ids lists the distinct window IDs of the block's events, the same IDs as in moments.
  • urls holds at most 100 page URLs of up to 2 KiB each and 32 KiB in all. urls_truncated is true when the recorder left pages out.
  • url_fingerprints holds up to 10 distinct page URLs of the block as the first 16 hexadecimal characters of an HMAC-SHA256 of the URL. The key is derived from the session key with HKDF-SHA256, empty salt and info ml-url-fingerprint-key, so fingerprints can be compared only within one session. They count distinct URLs, where navigations counts URL changes.
  • cold_start is true when the recorder first sees a session after it started, for example after a partition rebalance, or after it dropped the session's DOM state to stay within its memory budget. Its playability columns then describe only the part of the session the recorder saw.

Surfacing features

The surfacing_ columns let a sampler score whole sessions with the surfacing booster without opening a block. Each block row holds what ingestion's SessionFeatureRecorder (session-feature-recorder.ts) records for one batch, and the recorder computes it with the same definitions. Its state starts again at each block, as ingestion's does at each batch: a block is one session's events from one Kafka partition in one recorder flush, as a batch is. After the prefix, each column has the name of the session_replay_features column it matches, with a unit for the sums that are not counts. A session's values are the fold of its block rows, as _AGGREGATED_STATS_FRAGMENT (posthog/temporal/session_replay/surfacing_scoring_sweep/sql.py) folds the batch rows, and fold_session in replay-core-v4 implements it.

Kind Columns
Counts INT32: surfacing_{name}_count for the 31 counters of session_replay_features, from event_count to selection_copy_count, and surfacing_{token}_path_visit_count for the 14 path tokens, where 404 is not_found
Sums INT64: surfacing_mouse_sum_x_millipx, surfacing_mouse_sum_y_millipx, surfacing_mouse_sum_x_squared_px2, surfacing_mouse_sum_y_squared_px2, surfacing_mouse_distance_traveled_millipx, surfacing_mouse_velocity_sum_px_per_s, surfacing_mouse_velocity_sum_of_squares_px2_per_s2, surfacing_total_scroll_magnitude_millipx, surfacing_inter_action_gap_sum_ms, surfacing_inter_action_gap_sum_of_squares_ms2, surfacing_network_request_duration_sum_us, surfacing_network_request_duration_sum_of_squares_ms2
Maxima INT64: surfacing_max_scroll_y_millipx, surfacing_max_idle_gap_ms
Sets surfacing_visited_url_fingerprints (UTF8), surfacing_click_target_ids and surfacing_form_field_ids (INT64)
  • Ingestion stores a float sum as a double. The index stores it as an integer in the unit at the end of its name, rounded once per block, so that a fold is exact in any order. millipx is 1/1000 of a pixel and us is a microsecond. Ingestion's mouse velocity is in pixels per millisecond, so surfacing_mouse_velocity_sum_px_per_s is 1000 times its sum and surfacing_mouse_velocity_sum_of_squares_px2_per_s2 1000000 times its sum of squares.
  • The sets hold distinct values, sorted. The click targets and form fields are rrweb node IDs. The visited pages are fingerprints in the format of url_fingerprints, under the same key: ingestion stores an md5 of each URL instead. Each set keeps the first 1000 values of a block, as ingestion keeps 1000 per batch. The sweep counts a set with uniqCombinedMerge(12), and the length of the folded set is the exact count.
  • A page URL is the trimmed data.href, or else data.payload.href, of any event, as hrefFrom in rrweb-types.ts reads it. The path token counts compare the segments of its path with a fixed list of words, so they hold no other part of a URL.
  • The recorder skips an event without a positive timestamp, which ingestion's parse step drops before its recorder. The booster's rates divide by the session duration, which a sampler takes from first_ts_ms and last_ts_ms. Those also count events with a timestamp of zero or less.
  • The columns are null in a row from a recorder that predates them, and absent from files written before them. A sampler falls back to the session's live score for such a session.
  • Ingestion corrects the event times of a message by the sender's clock skew, rounded to 5 minutes, and the recorder does not. A block whose messages got different corrections can therefore differ in its time-based values.
  • A field of an unexpected JSON type counts as missing, where JavaScript can coerce it or produce NaN, for example a mouse position without coordinates. Ingestion's parse step lets such fields through.

Watermark files

After a flush, the sink writes one JSON file per index and Kafka partition of ai_research_v4_index, so a reader can learn from the bucket how far the index is complete:

  • index/blocks/_watermarks/partition={NNNN}.json for block rows.
  • index/images/_watermarks/partition={NNNN}.json for image rows.

{NNNN} is the partition number with four digits, for example partition=0007.json. Each file holds one object:

Field Meaning
partition The Kafka partition.
complete_up_to_capture_ms Every row that the sink read from the partition with an earlier capture time is in an index file in S3. It is the value of ml_replay_capture_watermark_timestamp_seconds for the partition, in milliseconds: data="event_metadata" for blocks and data="inline_images" for images.
committed_offset The next offset to read on the partition. The sink stored it for commit after it wrote every row below it.
written_at_ms When the sink wrote the file.

The capture time of a row is the Kafka timestamp of the replay record that it came from, which the recorder sends in the capture-timestamp-ms header. The sink writes a file only for a partition that it still owns, and only after a flush wrote the partition's rows and stored its offset. A row that the sink holds but has not written keeps the value at or below its capture time. The sink rewrites its files once every WATERMARK_FILE_INTERVAL_MS (index-watermark-files.ts), on the first flush or poll after the interval, also when no rows arrive. A value can rise without new rows, because the watermark also counts the rows that it finished in the last FINISHED_WINDOW_MS (capture-watermark.ts). Each write is a plain PUT that replaces the file. A failed write does not fail the flush. The sink counts it in ai_research_v4_parquet_sink_watermark_writes_total, logs it, and writes the file again after the next interval. A write that the sink aborts because it lost the partition counts as outcome="cancelled". S3 expires a file 30 days after its last write, as it does every object in the bucket.

To learn how far the sink has written an index, a reader:

  1. Reads every file under that index's _watermarks/.
  2. Takes the minimum complete_up_to_capture_ms across the partitions.
  3. Treats a file whose written_at_ms is more than about 15 minutes old as a partition that is not progressing: no sink writes it. Its value is still true, but it does not move. A running sink keeps each file it owns at most the rewrite interval plus one poll old.

Some limits apply:

  • A partition that delivers no records keeps its last value, as the gauge does, and the sink keeps rewriting its file. So a quiet partition holds the minimum down.
  • When the sink loses a partition, it aborts the partition's watermark writes that are still in flight, retries included. A write that S3 already received can still land. The new owner writes its first file only after its first flush, so the two overlap only if S3 takes longer to apply that write than the new owner takes to flush. The file then holds an older value until the new owner writes again. That value is still true. A full fence is not possible, because the bucket does not let the sink read the file that it replaces.
  • A partition that the sink has not flushed since it started to write these files has no file. A reader that needs every partition compares the number of files with the partition count of the topic.
  • The value covers only the rows that reached the sink. The recorder sends a row only after its own flush stores the block or image, so a row with an earlier capture time can still be in the recorder. A sink that has read all of its topic does not make the index complete. The recorder watermark files cover the recorder.
  • A reader that lists index/blocks/ or index/images/ for Parquet files must skip _watermarks/. Hive-style readers such as Spark, Athena and pyarrow datasets ignore paths that start with _, and a *.parquet glob does not match the JSON files.

Recorder watermark files

The recorder sets the write day of a flush from its own clock when the flush starts. A recorder that lags therefore puts blocks and index rows in a write day after their capture day. The sink's watermark files cannot show this, so the recorder writes its own files to the rrweb bucket:

  • index/_recorder_watermarks/dt={day}/topic={topic}/partition={NNNN}.json

There is one file per write day, input topic and Kafka partition. The input topics are the replay topic and its overflow topic. {NNNN} is the partition number with four digits. Each file holds one object:

Field Meaning
topic The input topic.
partition The input partition.
complete_up_to_capture_ms Every record of the partition with an earlier capture time has its blocks and images stored in this write day or an earlier one. Each index row of the record was delivered to ai_research_v4_index or failed.
index_end_offsets The high watermark of each partition of ai_research_v4_index, by partition number. The recorder reads them after it takes the value, so each delivered row that the value covers is below them.
input_partitions The partition count of each input topic, by topic name. Any one file tells a reader which files a day must have.
written_at_ms When the recorder wrote the file.

The capture time of a record is its Kafka timestamp, the same time as the capture-timestamp-ms header of its index rows. The recorder sets the value this way:

  • A record that the recorder has read but not finished holds the value at or below its capture time. A record is finished when its flush has stored the blocks and the producer has delivered or failed every index row of the flush.
  • A record that the recorder has not read yet can have an earlier capture time than one that it has read, by up to the delivery timeout of the record's producer. So the value is at least CAPTURE_ORDER_SLACK_MS (watermark.rs) before the capture time of the last record read, unless the partition is read to its end.
  • A partition that the recorder has read to its end moves with the clock, also when no records arrive. Its value can rise to CAPTURE_ORDER_SLACK_MS before the time that the recorder read the end offset, so a quiet overflow partition does not stop a reader. A partition that holds no record counts as read to its end, also when it has no committed offset.
  • After an assignment, the recorder writes no file for the partition until the index delivery timeout plus one minute has passed. By then, each index row that the previous owner sent is delivered or failed.

The recorder writes the files every WATERMARK_FILE_INTERVAL (watermark.rs), from the consume loop, for each partition that it owns. Each write goes to the UTC day of the write and is a plain PUT that replaces the file of that day. So the files of a day change until the day ends, and the last write of the day stays. A file of a running recorder is at most the interval plus one flush old. A file of the current day that is more than about 15 minutes old shows a partition that no consume loop moves forward. A failed write does not stop the recorder. It counts the write in ai_research_v4_recorder_watermark_files_total and writes the file again in the next round. A round that cannot read the Kafka offsets writes no files and counts outcome="kafka_failed" in ai_research_v4_recorder_watermark_rounds_total. S3 expires a file 30 days after its last write, as it does every object in the bucket.

To find the write days that hold every record captured before a cutoff C, a reader:

  1. Starts at the UTC day of C and uses only days that have ended, because the files of the current day still change.
  2. Finds the first day X with a file for every partition of every topic in input_partitions, each with complete_up_to_capture_ms at or after C. Each block and index row of a record captured before C then has a write day of X or earlier. A day with a missing file proves nothing, but a later day can.
  3. Takes the highest value of index_end_offsets for each index partition, over the files of day X.
  4. Waits until, for each index partition, one of the sink's watermark files of that partition has a committed_offset at or past that value. The block and image files of a partition carry its one committed offset, so either file shows it, and a reader of one index does not wait for the other's file to refresh. An index partition whose value is 0 has never received a row, so it needs no sink file. Every index row of those records is then in an index file in S3.
  5. Reads the write days up to and including X.

For example, take a cutoff of 2026-10-27 00:00 and a recorder that is 60 hours behind. The recorder reaches the cutoff at about 2026-10-29 12:00, so 2026-10-29 is the first day whose files reach the cutoff. A reader of the sessions that start from 2026-10-12 then reads the write days from 2026-10-11 to 2026-10-29.

Some limits apply:

  • An index row that the producer failed to deliver stays out of the index, as above. The value counts its record as finished.
  • A write of an earlier owner can land after a newer file of the same day. Its value is older, but it is still true.
  • The value assumes that a record reaches its partition no later than CAPTURE_ORDER_SLACK_MS after its capture time. A record that arrives later can have a capture time before a value that the recorder already wrote.
  • A reader that lists index/ must skip _recorder_watermarks/.

Image index

index/images/ has one row per inline image in a pack: team_id, key_month, image_ref, pack_s3_key, byte_start, byte_end, mime and byte_size.

  • Each frame in a pack is sealed under the team month key of the write month, as a fetched image is. A session deletion therefore does not make inline images unreadable (see Deletion).
  • image_ref uses the team month image content key, so the same image has the same reference in every session of the team month. A pack holds one frame per distinct reference, so an image that two sessions show in one flush is stored once. A later flush stores it again.
  • A reader resolves an image of a block by (team_id, image_ref) and opens it with the team month key of key_month.
  • The sink writes watermark files for this index too, under index/images/_watermarks/.

Remote images

The recorder sends each remote image URL that the URL policy accepts to the ai_research_v4_image_fetch topic in the v3 job shape, keyed by registrable domain. The job carries the reference that replaces the URL in the block. It collects URLs from image sources, the kept candidate of each srcset, SVG image links, CSS url(...) values and attribute mutations, and skips hidden or 1×1 images as v3 does. Unlike v3, it also collects CSS custom properties and relative CSS URLs, and fetches one image-set() candidate of a stored type. The topics hold jobs for 24 hours, because jobs carry raw URLs. The v4 image fetcher is its own copy of the v3 fetcher under ai-research/image-fetch-v4/, with the v3 fetch policy, its own consumer groups and its own AI_RESEARCH_V4_IMAGE_FETCH_* settings. It shares no code or table with v3. It applies the recorder's URL policy through @posthog/ai-research-v4-native, an addon over replay_core_v4::url_policy, so it canonicalizes a URL, groups it by registrable domain and refuses private hosts exactly as the recorder does. It stores each image, sealed and unscrubbed, under images/url/, and it does not send images to the scrub lane. It parses the image URL reference, seals the frame and computes the response URL HMAC with replay_core_v4::fetched_image through the same addon. It stores PNG, JPEG, GIF, WebP and SVG as fetched. It does not rasterize an SVG; redaction downstream of the dataset handles SVG text. The response must have one of these media types and bytes of that format: a signature for a raster image, or an SVG root element for SVG. An SVG may carry a byte order mark, comments, processing instructions and a doctype before its root. The doctype may name no internal subset and no external DTD other than a W3C SVG DTD, so that a parser downstream cannot resolve entities from the document. It refuses AVIF and every other type.

The fetcher keeps a crawl history per URL reference, so per team and month, in its own DynamoDB table ai-research-v4-image-fetch-history, with the v3 fetch interval and conditional requests. The table also holds its robots.txt and tdmrep cache:

  • The first job for a reference fetches the image from its origin.
  • A later job before the v3 refetch time does not contact the origin. When the stored copy is more than 7 days old, the fetcher copies it onto itself. Otherwise it drops the job.
  • A job after the v3 refetch time sends a conditional request with the stored ETag or Last-Modified. A 304 refreshes the stored copy by a copy onto itself, and a 200 replaces it.
  • The history item expires 8 days after the end of its month, as v3 history does. A reference carries the month of its write, so no job refers to it after that.
  • An item that does not parse, for example one that a newer build wrote, reads as a miss. The fetcher counts it and fetches the image again.
  • The history holds no URL in plaintext. It binds the stored validators to an HMAC of the response URL under the team month URL key, and sends them only when the job's URL has the same HMAC.

A URL is therefore fetched from its origin about once per team and month, as v3 fetches it about once in 30 days. Unlike v3, the history is per team: a sealed image is readable only with its team's key, so two teams that refer to one URL each fetch it. A public font file is the exception, as Web fonts says. The recorder sends a job for each referenced URL at least once per day, unless the fetcher's history shows that the fetcher would drop it (see below). So the fetcher sees a reference in every 7-day refresh interval. The fetcher drops a job whose team month row is missing, expired or deleted. A public font job needs no row.

Each recorder pod sends a reference at most once per UTC day. When AI_RESEARCH_V4_IMAGE_FETCH_HISTORY_TABLE is set, the recorder also reads the fetcher's history, as the v3 mirror did:

  • It reads only the image and font references of the flush that it did not send or mark that day, each one once. The read starts with the flush and runs while the flush seals its blocks.
  • It does not send a job that the fetcher would still drop at the end of the day. That is a job before the refetch time of its reference, when the reference has no stored copy or a stored copy less than 7 days old. It marks that reference as sent for the day.
  • It sends every other job, so the fetcher still refreshes each stored copy within its interval.
  • It parses a history row with the same code as the fetcher. A row that the fetcher reads as a miss is a miss for the recorder too.
  • A read that fails or takes more than 5 seconds checks nothing, and its jobs go out. The read does not retry a failed request.
  • ai_research_v4_recorder_image_fetch_history_total counts the references it read by outcome: fresh (marked), due (a refresh or refetch is due today), absent, invalid and error.

replay_core_v4::fetch_history holds the drop rule, the 7-day refresh interval and the row parser. The recorder calls it directly, and the fetcher calls it through @posthog/ai-research-v4-native.

Limits per domain and frontier backlog

The fetcher sends at most AI_RESEARCH_V4_IMAGE_FETCH_MAX_CONCURRENT_PER_REGISTRABLE_DOMAIN requests at once to one registrable domain. The default is 7. An explicit robots.txt crawl delay and the back-off after a failure can lower the rate for a domain. Retry-After can also lower it.

The topic key sends each registrable domain to one partition and one pod, so more pods do not drain one domain faster. A pass ends after AI_RESEARCH_V4_IMAGE_FETCH_REQUEST_BUDGET_MS. A URL that the pass does not reach goes back to the frontier as pass_deadline. A URL whose domain is in back-off goes to a delay topic as not_ready. A URL finishes as url_too_old when it reaches AI_RESEARCH_V4_IMAGE_FETCH_MAX_URL_AGE_MS.

When one frontier partition holds most of the lag, find the cause before you change capacity:

  1. Read ai_research_v4_image_fetch_partition_urls_total for the partition. A high store_deduped rate means that the recorder sends jobs that the fetcher drops. Check that the recorder has AI_RESEARCH_V4_IMAGE_FETCH_HISTORY_TABLE and read ai_research_v4_recorder_image_fetch_history_total.
  2. Read the TopHog metrics by registrable_domain, which the staff query path in /querying-tophog shows:
    • ai_research_v4_image_fetch_filtered_urls_by_registrable_domain with stage=received shows the demand. pass=first counts what the recorder sent, and pass=republished counts what the fetcher sent back to itself. The other stages count unique URLs after duplicates in a batch merge, so they do not separate the two passes.
    • ai_research_v4_image_fetch_attempts_by_registrable_domain shows each outcome.
    • ai_research_v4_image_fetch_block_events_by_registrable_domain and ai_research_v4_image_fetch_blocked_ms_by_registrable_domain show which limit holds the domain.
  3. Act on the limit that holds the domain:
    • For transient_backoff, retry_after or breaker_open, the site fails or asks for less traffic. Do not raise the limit. The URLs wait, and those that stay blocked finish as url_too_old.
    • For pass_deadline with domain_concurrency, the domain has more URLs than its limit can serve in a pass. Raise the limit per domain only for a host that can take the traffic.
    • For configuration_unreachable, the fetcher cannot read robots.txt or tdmrep for the origin. It waits for one hour and tries again.

IngestionSessionReplayMlConsumerNotDraining and IngestionSessionReplayMlConsumerStalled read the lag of the whole topic. The AiResearchV4ImageFetch* alerts in the charts repository cover late completions, rejected input and long batches. The v4 image fetch dashboard in Grafana shows the same signals.

Web fonts

A renderer that has no network needs the page's fonts. rrweb keeps the @font-face rules of a page in its style text, but it does not record font files, so the v4 lane fetches them as it fetches images.

The recorder reads the src URLs of @font-face rules in <style> text, _cssText, style sheet rules and adopted style sheets. Each URL becomes one of these:

The rule's family In the block Fetch job
A family in replay-core-v4/src/bundled_fonts.txt fonturl:v4:bundled:{slug} None
Any other family, for a public font file fonturl:v4:public:{hash} Yes
Any other family fonturl:v4:{team_id}:{YYYY-MM}:{hash} Yes
Any other family, for a URL that the URL policy refuses fonturl:v4:declined:{reason} None
Any other family, for a URL that the policy refuses as credential The URL without its credentials None
  • The renderer ships every family in bundled_fonts.txt. A reader replaces fonturl:v4:bundled:{slug} with its own copy of that family, and picks the face from the rule's font-weight and font-style descriptors. The slug is the family name in lowercase, with - for each space.
  • The list holds only families whose licence lets the renderer carry them. The renderer ships the full font of each family, which holds every glyph of a subset that a page asks for, such as a Google Fonts text= or icon_names= subset.
  • The list holds Google's icon families, Material Icons and Material Symbols, because Google never moves an icon to another code point or ligature. It holds no site's own icon font. Many sites share a generated name such as icomoon, and an icon font generator can number the glyphs again on each build, so the fetcher fetches those.
  • A page can ask Google for one fixed fill or grade of Material Symbols, such as FILL@1. The bundled font has every fill and grade, and shows the default unless the page's CSS sets font-variation-settings. So such a page shows outlined icons where it showed filled ones. The font-weight and font-size of the page still set the weight and optical size.
  • The recorder compares family names without case, and reads a run of spaces as one space. A family it does not know, for example a name that a build tool made, is fetched.
  • A team font reference uses the same hash, image URL key and URL policy as an image URL reference. A public font reference uses the same URL policy. A font job goes to the ai_research_v4_image_fetch topic in the same shape, keyed by registrable domain, so fonts share the image jobs' partitions, politeness limits and crawl history rules. A record holds only font jobs or only image jobs.
  • An Embedded OpenType or SVG font file gets a reference but no fetch job, because Chromium loads neither format.
  • A root-relative font URL, such as src:url(/fonts/a.woff2), resolves as a root-relative image URL does, and gets the font reference of the absolute URL. A path-relative font URL stays as the page wrote it.
  • The recorder reads CSS escapes as a browser does, in the @font-face keyword, the src and font-family descriptors, the url() name and the URL itself. So @\66 ont-face, \73 rc and url(https://cdn.example.com/a\2e woff2) count, and an escaped protocol-relative URL reads as https. A quoted URL drops a backslash before a newline, as a line continuation. Comments around a descriptor name, as in src/**/:, do not hide it. A font URL that holds an escape gets a rewrite of its whole url() to its font reference, as a root-relative font URL does.
  • A URL that is both an image and a font gets the reference of each slot's kind.

The fetcher reads the asset from the reference prefix. For a font it sends Accept: font/woff2, font/woff, font/ttf, font/otf, */*;q=0.1. It takes a response whose media type is a font/* type, a legacy font type such as application/x-font-woff, or application/octet-stream, because many servers label fonts with a generic type. The decoded bytes must start with a WOFF2, WOFF, OpenType, TrueType or font collection signature. It stores the font as fetched under fonts/url/, sealed under the team month key with kind: "font-url".

Google serves each file under https://fonts.gstatic.com/s/ at a versioned path whose bytes never change, and it serves the same file to every page. Such a public font file belongs to no team:

  • Its reference is fonturl:v4:public:{hash}, where hash is the first 22 characters of the base64url SHA-256 of the canonical URL. The hash has no key, because the URL holds no personal data. A person who holds a reference can test it against a Google Fonts URL, which tells them only which public font file a page used.
  • The fetcher stores the file once, unsealed, at fonts/url/public/{hash}, and reads no team month key for it. All teams share one crawl history entry, so Google gets one request per file, and a later job from any team only refreshes the stored file.
  • The fetcher stores the file only when the response came from the URL that the reference names, so a redirect target never becomes the shared file. It counts a refused one as ai_research_v4_image_fetch_stored_images_total{asset="font",outcome="not_public"}.
  • A URL with a query, a port or a path outside /s/ gets a team reference. Google serves a text= subset under /l/, and such a file holds the glyphs of the text that the page asked for.

An older fetcher sends a whole record to its dead letter topic when one job in it has a fonturl reference. Records never mix the two kinds of job, so while a recorder runs ahead of the fetcher, only font jobs are lost. The fetcher's role must also be allowed to write fonts/url/ (v4-staging.hcl in posthog-cloud-infra). Until then, each font write fails and the fetcher tries the job again after a minute.