:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(ai-research): move every inline image out of v4 replay blocks (#126379)
Robbiedbd9ca502b55
diff --git a/products/ai_training/docs/replay-data.md b/products/ai_training/docs/replay-data.mdindex 8ee212f9e1c3..4bc971f1e7f9 100644--- a/products/ai_training/docs/replay-data.md+++ b/products/ai_training/docs/replay-data.md@@ -461,8 +461,14 @@ The recorder streams each block into the encoder. Read a block as a stream: a Br- posthog-js compresses some fields of an event and marks the event `cv: "2024-10"`. The recorder expands those fields to plain JSON and removes the `cv` key, as the replay player does, because the gzip strings do not compress again. An event that does not decode stays exactly as it arrived.- An inline image becomes `image:v4:{team_id}:{YYYY-MM}:{hash}` in the block, where the month is the UTC month of the write.-- The recorder takes base64 PNG, JPEG, GIF and WebP data URIs in any letter case and with parameters, under the v3 size limits. The image bytes name the format, not the data URI's label, so the image index `mime` is always one of these four. A data URI whose bytes have none of their signatures stays inline, for example a font labeled `image/ttf`, an icon or an empty payload.-- The dataset keeps no AVIF image. A base64 data URI that its label or its bytes name as AVIF becomes `image:v4:declined:avif`, and a srcset loses such a candidate. The recorder counts these as `ai_research_v4_recorder_removed_images_total{format="avif"}`. A payload broken by an escape the scan does not read stays inline. It also takes the image frames of the posthog-js canvas recorder. Anything else stays inline in the sealed block, for example SVG or line-wrapped base64.+- The recorder takes PNG, JPEG, GIF and WebP data URIs in any letter case and with parameters, under the v3 size limits. It decodes the JSON escapes of each string before it looks for data URIs, so an escaped letter does not hide one. It reads a base64 payload across line breaks and escaped slashes, and without its padding. It reads a payload that is not base64 as percent-encoded bytes. The image bytes name the format, not the data URI's label, so the image index `mime` is always one of these four.+- An inline image that the recorder does not store becomes a marker that names the reason, and its bytes are stored nowhere. A srcset loses such a candidate. The recorder counts these as `ai_research_v4_recorder_removed_images_total{reason}`.+ - `image:v4:declined:avif`: the label or the bytes name AVIF. The dataset keeps no AVIF image.+ - `image:v4:declined:over_limit`: the image is larger than 900 KiB, or its message already holds 4,096 images or 64 MiB of them.+ - `image:v4:declined:undecodable`: the payload does not decode, or it holds an escape that the recorder does not read, for example a CSS escape.+ - `image:v4:declined:unsupported`: the bytes are not PNG, JPEG, GIF or WebP, for example a font labeled `image/ttf`, an icon or a bitmap.+- An SVG data URI stays inline, because SVG is text that the text scrub reads in place. A data URI with an empty payload holds no image, so it stays as it is.+- The recorder reads every argument of a canvas mutation, not only the frames of the posthog-js canvas recorder. A buffer becomes a reference when its bytes are PNG, JPEG, GIF or WebP, and a marker when they are not. The pixels of an `ImageData`, and the numbers of a `Uint8Array` or a `Uint8ClampedArray`, become `image:v4:declined:unsupported`. A data URI in the `src` of an `HTMLImageElement` moves out like any data URI, and a remote URL there stays as it is.- A remote image URL becomes `imageurl:v4:{team_id}:{YYYY-MM}:{hash}` in the block and in its fetch job, where the month is the UTC month of the write. The block and the job always carry the same reference, so a reader opens the fetched image directly.- A URL that the URL policy refuses becomes `imageurl:v4:declined:{reason}` in the block, for example `declined:credential`, `declined:non_public_host` or `declined:tracking_beacon`. The original URL cannot be recovered. The recorder counts rewrites and refusals in `ai_research_v4_recorder_image_url_rewrites_total`.- The rewrite covers every remote URL that the recorder reads from an image slot: image sources, `srcset`, SVG image links, video posters, `<picture>` sources and CSS `url(...)` in style attributes, style sheets and style rules, in custom properties too, because a page passes a custom property to an image property through `var()`. It also covers `src`, `srcset`, `href` and `poster` in an attribute change on a node the recorder does not know, for example after a restart. `srcset` keeps its descriptors and CSS keeps its `url(...)` syntax. A URL is also replaced where a selector holds it with CSSOM's escapes, by the reference with the same escapes, so a class such as `bg-[url(https://cdn.example.com/a.png)]` and its rule `.bg-\[url\(https\:\/\/cdn\.example\.com\/a\.png\)\]` still match within a block. A reader that replaces references by text must replace the escaped form as well. It is not limited by the 512 URLs per message that get fetch jobs.@@ -564,6 +570,7 @@ This holds for the blocks that one recorder process inspected. After a rebalance- `host` is the host of the first page URL. An IP-address host is stored as `[ip_address]`, as v3 does, and a host longer than 253 bytes is stored as null.- `navigations` counts URL changes from Meta events and from posthog-js `$url_changed` events, so single-page apps count too.- `node_cap_reached` is true when the recorder could not check a window for orphan mutations: the window passed the node cap, a snapshot or mutation did not decode, the window was evicted and came back, or the recorder dropped the session's DOM state to stay within its memory budget.+- `images_inline` counts the SVG data URIs that stay inline in the block. An image that becomes a marker counts in no column.- `byte_size` counts the event bytes as they arrived. `written_jsonl_bytes` counts the block's lines as the recorder wrote them, after it expanded compressed values and removed images, and before Brotli. `max_written_event_bytes` is the longest of those lines. Use these two for memory and size limits.- `mouse_interactions` counts every mouse interaction event, of every type. `clicks` counts only clicks. `touch_moves` counts touch move events; `touches` also counts touch starts and ends.- `decode_failures` counts, once per event, a compressed value that did not decompress or that the decode budget skipped, a full snapshot the recorder could not decode or walk, a mutation that did not decode or held a node too deep to walk, and a style sheet rule whose compressed value failed. A message that did not read at all reaches no block, so this column does not count it. A block with failures has activity and DOM columns that can be too low.diff --git a/rust/ai-research/replay-core-v4/src/bytescan.rs b/rust/ai-research/replay-core-v4/src/bytescan.rsindex 2479dbd14411..59c5f5a03715 100644--- a/rust/ai-research/replay-core-v4/src/bytescan.rs+++ b/rust/ai-research/replay-core-v4/src/bytescan.rs@@ -7,6 +7,8 @@//! callers that need decoded text use [`unescape`], and callers that meet an escaped **key** must fall//! back to a real parse — a `\uXXXX` key could alias a routing key, and byte comparison would miss it.+use std::borrow::Cow;+/// Half-open byte range into the scanned buffer.pub type Span = (usize, usize);@@ -698,6 +700,69 @@ pub fn write_json_string(s: &str, out: &mut Vec<u8>) {out.push(b'"');}+/// The bytes with each JSON string that holds an escape `write_json_string` does not write, such as `\/` or `\u0041`,+/// written again as it writes them. Capture writes events that way, but the JSON inside a `cv` payload arrives as the+/// page wrote it, and a client can send any escape. A URL or a data URI could then hide behind an escape from a reader+/// that matches bytes.+pub fn canonical_strings(bytes: &[u8]) -> Cow<'_, [u8]> {+ if !bytes.contains(&b'\\') {+ return Cow::Borrowed(bytes);+ }+ let mut out: Option<Vec<u8>> = None;+ let mut copied_to = 0;+ let mut position = 0;+ while let Some(start) = bytes[position..]+ .iter()+ .position(|&b| b == b'"')+ .map(|at| position + at)+ {+ let mut end = start + 1;+ let mut canonical = true;+ while end < bytes.len() && bytes[end] != b'"' {+ if bytes[end] == b'\\' {+ match bytes.get(end + 1) {+ Some(b'"' | b'\\' | b'b' | b'f' | b'n' | b'r' | b't') => {}+ Some(b'u') => canonical &= is_written_escape(bytes.get(end + 2..end + 6)),+ _ => canonical = false,+ }+ end += 1;+ }+ end += 1;+ }+ if end >= bytes.len() {+ break;+ }+ if !canonical {+ if let Ok(text) = unescape(bytes, (start, end + 1)) {+ let out = out.get_or_insert_with(|| Vec::with_capacity(bytes.len()));+ out.extend_from_slice(&bytes[copied_to..start]);+ write_json_string(&text, out);+ copied_to = end + 1;+ }+ }+ position = end + 1;+ }+ match out {+ Some(mut out) => {+ out.extend_from_slice(&bytes[copied_to..]);+ Cow::Owned(out)+ }+ None => Cow::Borrowed(bytes),+ }+}++/// Whether `write_json_string` writes this `\u` escape: a control character without a short escape, in lowercase hex.+fn is_written_escape(hex: Option<&[u8]>) -> bool {+ let Some(&[b'0', b'0', high, low]) = hex else {+ return false;+ };+ let digit = |b: u8| matches!(b, b'0'..=b'9' | b'a'..=b'f');+ digit(high)+ && digit(low)+ && high <= b'1'+ && !matches!((high, low), (b'0', b'8' | b'9' | b'a' | b'c' | b'd'))+}+#[cfg(test)]mod tests {use super::*;diff --git a/rust/ai-research/replay-core-v4/src/scan/canvas.rs b/rust/ai-research/replay-core-v4/src/scan/canvas.rsindex 61e3f3ab2571..3e39654e0266 100644--- a/rust/ai-research/replay-core-v4/src/scan/canvas.rs+++ b/rust/ai-research/replay-core-v4/src/scan/canvas.rs@@ -1,125 +1,101 @@-//! Canvas frames. posthog-js records a canvas as one frame shape only: clear the canvas, then draw one encoded image-//! of it. The frame's image moves out of the block like an inline image. Every other canvas command stays as it is.+//! Canvas mutations: the `ImageBitmap(Blob(ArrayBuffer))` frames of posthog-js, and the buffers, `ImageData` and pixel arrays of rrweb's command recorder.+//! The data URI scan reads an `HTMLImageElement` `src`, and a remote `src` holds no image bytes.use crate::bytescan::{self as scan, Span};+const INCREMENTAL_SNAPSHOT: u8 = 3;const CANVAS_MUTATION: u8 = 9;-const CONTEXT_2D: f64 = 0.0;-const FRAME_MIMES: &[&str] = &["image/webp", "image/png", "image/jpeg"];+/// Typed arrays whose numbers can be pixels, as in `ImageData` or WebGL's `texImage2D`: the set the text scrub drops.+const PIXEL_ARRAYS: &[&str] = &["Uint8Array", "Uint8ClampedArray"];-/// The values of `keys` in the object at `span`. `None` when it is not an object, or a key is escaped or repeated.-fn members<const N: usize>(- bytes: &[u8],- span: Span,- keys: [&[u8]; N],-) -> Option<[Option<Span>; N]> {- let mut found = [None; N];- for entry in scan::object_entries(bytes, span).ok()? {- let entry = entry.ok()?;- if entry.key_escaped {- return None;- }- let key = &bytes[entry.key.0..entry.key.1];- if let Some(index) = keys.iter().position(|&k| k == key) {- if found[index].replace(entry.value).is_some() {- return None;- }- }- }- Some(found)+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum CanvasImage {+ Base64(Span),+ Pixels(Span),}-/// The items of the array at `span`, when it has at most `max`.-fn items(bytes: &[u8], span: Span, max: usize) -> Option<Vec<Span>> {- let mut items = Vec::new();- for item in scan::array_items(bytes, span).ok()? {- if items.len() == max {- return None;+impl CanvasImage {+ pub fn span(self) -> Span {+ match self {+ CanvasImage::Base64(span) | CanvasImage::Pixels(span) => span,}- items.push(item.ok()?);}- Some(items)}-fn is_text(bytes: &[u8], span: Option<Span>, text: &str) -> bool {- span.filter(|&s| scan::is_string(bytes, s))- .is_some_and(|s| scan::unescape(bytes, s).is_ok_and(|v| v == text))+fn is_text(bytes: &[u8], span: Span, text: &str) -> bool {+ scan::is_string(bytes, span) && scan::unescape(bytes, span).is_ok_and(|v| v == text)}-fn is_number(bytes: &[u8], span: Span) -> bool {- scan::parse_number(bytes, span).is_some_and(f64::is_finite)+fn is_container(bytes: &[u8], span: Span) -> bool {+ scan::is_array(bytes, span) || scan::is_object(bytes, span)}-fn is_u32(bytes: &[u8], span: Option<Span>) -> bool {- span.and_then(|s| scan::parse_number(bytes, s))- .is_some_and(|n| n.fract() == 0.0 && (0.0..=f64::from(u32::MAX)).contains(&n))-}--fn command(bytes: &[u8], span: Span, property: &str, max_args: usize) -> Option<Vec<Span>> {- let [name, args] = members(bytes, span, [b"property", b"args"])?;- is_text(bytes, name, property).then_some(())?;- items(bytes, args?, max_args)-}--/// The base64 string of a posthog-js snapshot frame and its mime type, matched as v3 `canvas.rs` matches it:-/// `{id, type: 2D, commands: [clearRect(4 numbers), drawImage(ImageBitmap(Blob(ArrayBuffer)), 2 or 4 numbers)]}`.-pub fn frame_image(+/// In byte order. The walk keeps its own stack, so no depth of nested arguments can overflow the call stack.+pub fn canvas_images(bytes: &[u8],kind: Option<u8>,source: Option<u8>,data: Span,-) -> Option<(&'static str, Span)> {- if kind != Some(3) || source != Some(CANVAS_MUTATION) {- return None;- }- let [id, context, commands, width, height] = members(- bytes,- data,- [- b"id",- b"type",- b"commands",- b"displayWidth",- b"displayHeight",- ],- )?;- if !is_u32(bytes, id)- || context.and_then(|c| scan::parse_number(bytes, c)) != Some(CONTEXT_2D)- || [width, height]- .into_iter()- .flatten()- .any(|v| !is_number(bytes, v))- {- return None;+) -> Vec<CanvasImage> {+ let mut images = Vec::new();+ if kind != Some(INCREMENTAL_SNAPSHOT) || source != Some(CANVAS_MUTATION) {+ return images;}- let [clear, draw] = items(bytes, commands?, 2)?[..] else {- return None;- };- let clear = command(bytes, clear, "clearRect", 4)?;- if clear.len() != 4 || !clear.iter().all(|&a| is_number(bytes, a)) {- return None;- }- // posthog-js before 1.391 drew at the origin without a size, so `[bitmap, x, y]`.- let draw = command(bytes, draw, "drawImage", 5)?;- if !matches!(draw.len(), 3 | 5) || !draw[1..].iter().all(|&a| is_number(bytes, a)) {- return None;- }- let [bitmap_type, bitmap_args] = members(bytes, draw[0], [b"rr_type", b"args"])?;- let [blob] = items(bytes, bitmap_args?, 1)?[..] else {- return None;- };- if !is_text(bytes, bitmap_type, "ImageBitmap") {- return None;+ let mut values = vec![data];+ while let Some(value) = values.pop() {+ if scan::is_array(bytes, value) {+ values.extend(+ scan::array_items(bytes, value)+ .into_iter()+ .flatten()+ .flatten()+ .filter(|&item| is_container(bytes, item)),+ );+ } else if scan::is_object(bytes, value) {+ visit_object(bytes, value, &mut images, &mut values);+ }}- let [blob_type, mime, blob_data] = members(bytes, blob, [b"rr_type", b"type", b"data"])?;- let mime = FRAME_MIMES.iter().find(|&&m| is_text(bytes, mime, m))?;- let [buffer] = items(bytes, blob_data?, 1)?[..] else {- return None;+ images.sort_unstable_by_key(|image| image.span().0);+ images+}++fn visit_object(bytes: &[u8], object: Span, images: &mut Vec<CanvasImage>, values: &mut Vec<Span>) {+ let entries: Vec<scan::Entry> = scan::object_entries(bytes, object)+ .into_iter()+ .flatten()+ .flatten()+ .collect();+ // Each copy of a repeated `rr_type` counts, so the walk covers the one the player reads.+ let has_type = |names: &[&str]| {+ entries.iter().any(|entry| {+ &bytes[entry.key.0..entry.key.1] == b"rr_type"+ && names.iter().any(|name| is_text(bytes, entry.value, name))+ })};- if !is_text(bytes, blob_type, "Blob") {- return None;+ let image_data = has_type(&["ImageData"]);+ let pixel_array = has_type(PIXEL_ARRAYS);+ for entry in &entries {+ match &bytes[entry.key.0..entry.key.1] {+ b"base64" if scan::is_string(bytes, entry.value) => {+ images.push(CanvasImage::Base64(entry.value))+ }+ b"args" if (image_data || pixel_array) && scan::is_array(bytes, entry.value) => {+ let mut args = scan::array_items(bytes, entry.value)+ .into_iter()+ .flatten()+ .flatten();+ if let Some(first) = args.next() {+ if scan::is_array(bytes, first) || (image_data && scan::is_object(bytes, first))+ {+ images.push(CanvasImage::Pixels(first));+ } else if is_container(bytes, first) {+ values.push(first);+ }+ }+ values.extend(args.filter(|&arg| is_container(bytes, arg)));+ }+ _ if is_container(bytes, entry.value) => values.push(entry.value),+ _ => {}+ }}- let [buffer_type, base64] = members(bytes, buffer, [b"rr_type", b"base64"])?;- let base64 = base64.filter(|&b| scan::is_string(bytes, b))?;- is_text(bytes, buffer_type, "ArrayBuffer").then_some((mime, base64))}diff --git a/rust/ai-research/replay-core-v4/src/scan/cv.rs b/rust/ai-research/replay-core-v4/src/scan/cv.rsindex f830ba131755..d90a425aad7b 100644--- a/rust/ai-research/replay-core-v4/src/scan/cv.rs+++ b/rust/ai-research/replay-core-v4/src/scan/cv.rs@@ -9,8 +9,8 @@ use crate::bytescan::{self as scan, Span};use crate::compression;use serde::de::IgnoredAny;-use super::canvas;-use super::images::{extract_images, ImageSink};+use super::canvas::{self, CanvasImage};+use super::images::{extract_images, Declined, ImageSink};use super::message::{canonicalize_keys, Event, Message};const FULL_SNAPSHOT: u8 = 2;@@ -387,20 +387,15 @@ impl<'b> Output<'b> {})}- fn scan_to(&mut self, end: usize, sink: &mut ImageSink<'_>) {- match extract_images(&self.bytes[self.copied_to..end], sink) {- Cow::Owned(extracted) => self.changed().extend_from_slice(&extracted),- Cow::Borrowed(same) => {- if let Some(out) = self.out.as_mut() {- out.extend_from_slice(same);- }- }+ fn copy_to(&mut self, end: usize) {+ if let Some(out) = self.out.as_mut() {+ out.extend_from_slice(&self.bytes[self.copied_to..end]);}self.copied_to = end;}- fn replace(&mut self, span: Span, with: &[u8], sink: &mut ImageSink<'_>) {- self.scan_to(span.0, sink);+ fn replace(&mut self, span: Span, with: &[u8]) {+ self.copy_to(span.0);self.changed().extend_from_slice(with);self.copied_to = span.1;}@@ -408,18 +403,29 @@ impl<'b> Output<'b> {/// The event as a v4 block stores it, or `None` when that is the event as it arrived. Inline images become references,/// and the compressed strings of a `cv` event are expanded to JSON, without the `cv` marker. An event whose strings-/// do not decode stays as it arrived, `cv` and images included. The stored event has no raw line break.+/// do not decode keeps its `cv` strings and its images. The stored event has no raw line break, and its strings hold+/// only the escapes that `write_json_string` writes.pub fn extract_event_images(inner: &[u8],event: &Event,sink: &mut ImageSink<'_>,) -> Option<Vec<u8>> {- let stored = stored_event(inner, event, sink);- let bytes = stored- .as_deref()- .unwrap_or(&inner[event.span.0..event.span.1]);- if let Cow::Owned(one_line) = without_line_breaks(bytes) {- return Some(one_line);+ let (mut stored, takes_images) = match &*expansion(inner, event) {+ Expansion::Failed | Expansion::Skipped => (None, false),+ Expansion::Decoded(fields) => (Some(expanded(inner, event, fields)), true),+ Expansion::Plain => (move_out_canvas_images(inner, event, sink), true),+ };+ let arrived = &inner[event.span.0..event.span.1];+ if let Cow::Owned(one_line) = without_line_breaks(stored.as_deref().unwrap_or(arrived)) {+ stored = Some(one_line);+ }+ if let Cow::Owned(canonical) = scan::canonical_strings(stored.as_deref().unwrap_or(arrived)) {+ stored = Some(canonical);+ }+ if takes_images {+ if let Cow::Owned(extracted) = extract_images(stored.as_deref().unwrap_or(arrived), sink) {+ stored = Some(extracted);+ }}stored}@@ -454,30 +460,29 @@ fn without_line_breaks(bytes: &[u8]) -> Cow<'_, [u8]> {Cow::Owned(out)}-fn stored_event(inner: &[u8], event: &Event, sink: &mut ImageSink<'_>) -> Option<Vec<u8>> {- match &*expansion(inner, event) {- Expansion::Failed | Expansion::Skipped => None,- Expansion::Decoded(fields) => Some(expanded(inner, event, fields, sink)),- Expansion::Plain => {- let mut output = Output::new(inner, event.span);- let frame = event- .data- .and_then(|data| canvas::frame_image(inner, event.kind, event.source, data));- if let Some((_, span)) = frame {- let payload = scan::unescape(inner, span).ok();- match payload.and_then(|p| sink.take(p.as_bytes())) {- Some(reference) => {- let mut quoted = Vec::with_capacity(reference.len() + 2);- scan::write_json_string(&reference, &mut quoted);- output.replace(span, "ed, sink);- }- None => sink.left_inline += 1,- }- }- output.scan_to(event.span.1, sink);- output.out- }+fn move_out_canvas_images(+ inner: &[u8],+ event: &Event,+ sink: &mut ImageSink<'_>,+) -> Option<Vec<u8>> {+ let mut output = Output::new(inner, event.span);+ let images = event.data.map_or_else(Vec::new, |data| {+ canvas::canvas_images(inner, event.kind, event.source, data)+ });+ for image in images {+ let replacement = match image {+ CanvasImage::Base64(span) => match scan::unescape(inner, span) {+ Ok(payload) => sink.take_base64(payload.as_bytes()),+ Err(_) => sink.decline(Declined::Undecodable),+ },+ CanvasImage::Pixels(_) => sink.decline(Declined::Unsupported),+ };+ let mut quoted = Vec::with_capacity(replacement.len() + 2);+ scan::write_json_string(&replacement, &mut quoted);+ output.replace(image.span(), "ed);}+ output.copy_to(event.span.1);+ output.out}/// Writes the object at `span` entry by entry. `value` writes an entry's value, or returns false to leave the entry out.@@ -485,8 +490,7 @@ fn write_object(bytes: &[u8],span: Span,out: &mut Vec<u8>,- sink: &mut ImageSink<'_>,- mut value: impl FnMut(&[u8], Span, &mut Vec<u8>, &mut ImageSink<'_>) -> bool,+ mut value: impl FnMut(&[u8], Span, &mut Vec<u8>) -> bool,) {out.push(b'{');let mut first = true;@@ -501,7 +505,7 @@ fn write_object(}out.extend_from_slice(&bytes[entry.key.0 - 1..entry.key.1 + 1]);out.push(b':');- if value(&bytes[entry.key.0..entry.key.1], entry.value, out, sink) {+ if value(&bytes[entry.key.0..entry.key.1], entry.value, out) {first = false;} else {out.truncate(mark);@@ -510,7 +514,7 @@ fn write_object(out.push(b'}');}-fn expanded(inner: &[u8], event: &Event, fields: &[Decoded], sink: &mut ImageSink<'_>) -> Vec<u8> {+fn expanded(inner: &[u8], event: &Event, fields: &[Decoded]) -> Vec<u8> {let mut out = Vec::with_capacity(event.span.1 - event.span.0);let json = |span: Span| {fields@@ -518,41 +522,32 @@ fn expanded(inner: &[u8], event: &Event, fields: &[Decoded], sink: &mut ImageSin.find(|f| f.span == span).map(|f| f.json.as_deref())};- write_object(- inner,- event.span,- &mut out,- sink,- |key, value, out, sink| {- if key == b"cv" {- return false;+ write_object(inner, event.span, &mut out, |key, value, out| {+ if key == b"cv" {+ return false;+ }+ if key == b"data" {+ if let Some(decoded) = json(value) {+ // A full snapshot whose data is empty has no data, as in the player.+ let Some(decoded) = decoded else { return false };+ out.extend_from_slice(decoded);+ return true;}- if key == b"data" {- if let Some(decoded) = json(value) {- // A full snapshot whose data is empty has no data, as in the player.- let Some(decoded) = decoded else { return false };- out.extend_from_slice(&extract_images(decoded, sink));- return true;- }- if scan::is_object(inner, value) {- write_object(inner, value, out, sink, |_, value, out, sink| {- match json(value) {- Some(Some(decoded)) => {- out.extend_from_slice(&extract_images(decoded, sink))- }- Some(None) => return false,- None => out- .extend_from_slice(&extract_images(&inner[value.0..value.1], sink)),- }- true- });- return true;- }+ if scan::is_object(inner, value) {+ write_object(inner, value, out, |_, value, out| {+ match json(value) {+ Some(Some(decoded)) => out.extend_from_slice(decoded),+ Some(None) => return false,+ None => out.extend_from_slice(&inner[value.0..value.1]),+ }+ true+ });+ return true;}- out.extend_from_slice(&extract_images(&inner[value.0..value.1], sink));- true- },- );+ }+ out.extend_from_slice(&inner[value.0..value.1]);+ true+ });out}@@ -633,6 +628,60 @@ mod tests {);}+ #[test]+ fn moves_out_an_image_whose_data_uri_holds_json_escapes() {+ let png = b"\x89PNG\r\n\x1a\npng bytes";+ let payload = STANDARD.encode(png);+ let escaped = |letter: char| format!("{}u{:04x}", '\\', u32::from(letter));+ for uri in [+ format!("{}ata:image/png;base64,{payload}", escaped('d')),+ format!("data{}image/png;base64,{payload}", escaped(':')),+ format!("DATA:{}mage/png;base64,{payload}", escaped('i')),+ ] {+ assert!(uri.contains('\\'), "{uri}");+ let snapshot =+ sdk_cv_text(&format!(r#"{{"node":{{"attributes":{{"src":"{uri}"}}}}}}"#));+ let events = json!([+ {"type": 3, "timestamp": 1, "data": {"source": 0, "adds": [], "removes": [], "texts": [],+ "attributes": [{"id": 3, "attributes": {"src": "ESCAPED_URI"}}]}},+ {"type": 2, "timestamp": 2, "data": snapshot, "cv": "2024-10"},+ ]);+ // Only JSON text can carry the escapes, so the URI goes in after serialization.+ let inner = json!({"event": "$snapshot_items", "properties": {"$session_id": "s", "$snapshot_items": events}})+ .to_string()+ .replace("ESCAPED_URI", &uri);+ let mut message = read_message(+ json!({"distinct_id": "d", "data": inner})+ .to_string()+ .into_bytes(),+ )+ .unwrap();+ message.decode_compressed();+ let mut sink = sink();+ let stored: Vec<Value> = message+ .events+ .iter()+ .map(|e| {+ let stored = extract_event_images(&message.inner, e, &mut sink)+ .unwrap_or_else(|| message.inner[e.span.0..e.span.1].to_vec());+ serde_json::from_slice(&stored).unwrap()+ })+ .collect();++ assert_eq!(sink.images.len(), 1, "{uri}");+ assert_eq!(sink.images[0].bytes, png, "{uri}");+ let reference = json!(sink.images[0].reference);+ assert_eq!(+ stored[0]["data"]["attributes"][0]["attributes"]["src"], reference,+ "{uri}"+ );+ assert_eq!(+ stored[1]["data"]["node"]["attributes"]["src"], reference,+ "{uri}"+ );+ }+ }+#[test]fn leaves_the_events_past_the_message_budget_as_they_arrived() {let data = json!({"node": {"type": 0, "childNodes": [], "id": 1}});@@ -825,59 +874,129 @@ mod tests {}#[test]- fn moves_only_the_posthog_js_canvas_frame() {- let frame = |draw_args: Value| {- json!({"type": 3, "timestamp": 1, "data": {"source": 9, "id": 4, "type": 0, "commands": [- {"property": "clearRect", "args": [0, 0, 8, 8]},- {"property": "drawImage", "args": draw_args},- ]}})- };- // posthog-js encodes the frame in the type the blob names.- let bitmap = |mime: &str, buffer: &str| {- let bytes: &[u8] = match mime {- "image/webp" => b"RIFF\x00\x00\x00\x00WEBPframe",- "image/gif" => b"GIF89aframe",- _ => b"\x89PNG\r\n\x1a\nframe",- };- json!({"rr_type": "ImageBitmap", "args": [{"rr_type": "Blob", "type": mime,- "data": [{"rr_type": buffer, "base64": STANDARD.encode(bytes)}]}]})- };+ fn moves_every_canvas_image_out_or_declines_it() {+ enum Outcome {+ Stored(&'static str),+ Marker(Declined),+ Kept,+ }+ let png = b"\x89PNG\r\n\x1a\nframe".as_slice();+ let buffer =+ |bytes: &[u8]| json!({"rr_type": "ArrayBuffer", "base64": STANDARD.encode(bytes)});+ let bitmap = |mime: &str, bytes: &[u8]| json!({"rr_type": "ImageBitmap", "args": [{"rr_type": "Blob", "type": mime, "data": [buffer(bytes)]}]});+ let commands = |commands: Value| json!({"type": 3, "timestamp": 1, "data": {"source": 9, "id": 4, "type": 0, "commands": commands}});+ let webgl = |property: &str, args: Value| json!({"type": 3, "timestamp": 1, "data": {"source": 9, "id": 5, "type": 1, "property": property, "args": args}});+ let draw =+ |image: Value| commands(json!([{"property": "drawImage", "args": [image, 0, 0]}]));+ let blob = "/args/0/data/0/base64";let cases = [(- frame(json!([bitmap("image/webp", "ArrayBuffer"), 0, 0, 8, 8])),- true,+ commands(json!([{"property": "clearRect", "args": [0, 0, 8, 8]},+ {"property": "drawImage", "args": [bitmap("image/webp", b"RIFF\x00\x00\x00\x00WEBPframe"), 0, 0, 8, 8]}])),+ format!("/data/commands/1/args/0{blob}"),+ Outcome::Stored("image/webp"),+ ),+ (+ commands(json!([{"property": "fillRect", "args": [0, 0, 1, 1]},+ {"property": "drawImage", "args": [bitmap("image/gif", b"GIF89aframe"), 0, 0]}])),+ format!("/data/commands/1/args/0{blob}"),+ Outcome::Stored("image/gif"),+ ),+ (+ webgl(+ "texImage2D",+ json!([3553, 0, 6408, 6408, 5121, bitmap("image/png", png)]),+ ),+ format!("/data/args/5{blob}"),+ Outcome::Stored("image/png"),+ ),+ (+ draw(+ json!({"rr_type": "HTMLImageElement", "src": format!("data:image/png;base64,{}", STANDARD.encode(png))}),+ ),+ "/data/commands/0/args/0/src".to_owned(),+ Outcome::Stored("image/png"),),(- frame(json!([bitmap("image/png", "ArrayBuffer"), 0, 0])),- true,+ draw(bitmap("image/bmp", b"BMbitmap")),+ format!("/data/commands/0/args/0{blob}"),+ Outcome::Marker(Declined::Unsupported),),(- frame(json!([bitmap("image/gif", "ArrayBuffer"), 0, 0])),- false,+ draw(bitmap(+ "image/avif",+ b"\x00\x00\x00\x1cftypavif\x00\x00\x00\x00avifmif1miaf",+ )),+ format!("/data/commands/0/args/0{blob}"),+ Outcome::Marker(Declined::Avif),),(- frame(json!([bitmap("image/png", "Uint8Array"), 0, 0])),- false,+ draw(+ json!({"rr_type": "ImageBitmap", "args": [{"rr_type": "Blob", "type": "image/png",+ "data": [{"rr_type": "ArrayBuffer", "base64": "A"}]}]}),+ ),+ format!("/data/commands/0/args/0{blob}"),+ Outcome::Marker(Declined::Undecodable),),- (frame(json!([bitmap("image/png", "ArrayBuffer"), 0])), false),(- json!({"type": 3, "timestamp": 1, "data": {"source": 9, "id": 4, "type": 0, "commands": [- {"property": "putImageData", "args": [{"rr_type": "ArrayBuffer", "base64": STANDARD.encode(b"pixels")}, 0, 0]},- ]}}),- false,+ commands(+ json!([{"property": "putImageData", "args": [{"rr_type": "ImageData",+ "args": [{"rr_type": "Uint8ClampedArray", "args": [[255, 0, 0, 255]]}, 1, 1]}, 0, 0]}]),+ ),+ "/data/commands/0/args/0/args/0".to_owned(),+ Outcome::Marker(Declined::Unsupported),+ ),+ (+ webgl(+ "texImage2D",+ json!([3553, 0, 6408, 1, 1, 0, 6408, 5121, {"rr_type": "Uint8Array", "args": [[255, 0, 0, 255]]}]),+ ),+ "/data/args/8/args/0".to_owned(),+ Outcome::Marker(Declined::Unsupported),+ ),+ (+ webgl("bufferData", json!([34962, buffer(b"vertices"), 35044])),+ "/data/args/1/base64".to_owned(),+ Outcome::Marker(Declined::Unsupported),+ ),+ (+ draw(json!({"rr_type": "HTMLImageElement", "src": "https://example.com/a.png"})),+ String::new(),+ Outcome::Kept,+ ),+ (+ webgl(+ "uniform4fv",+ json!([{"rr_type": "WebGLUniformLocation", "index": 0},+ {"rr_type": "Float32Array", "args": [[0.5, 0.5, 0.5, 1]]}]),+ ),+ String::new(),+ Outcome::Kept,),];- for (event, moved) in cases {+ for (event, pointer, outcome) in cases {let message = message_with(json!([event]));let mut sink = sink();- let stored = extract_event_images(&message.inner, &message.events[0], &mut sink);- assert_eq!(stored.is_some(), moved, "{event}");- if let Some(stored) = stored {- let stored: Value = serde_json::from_slice(&stored).unwrap();- let blob = &stored["data"]["commands"][1]["args"][0]["args"][0];- assert_eq!(blob["data"][0]["base64"], sink.images[0].reference.as_str());- assert_eq!(blob["type"], sink.images[0].mime.as_str());- }+ let stored = extract_event_images(&message.inner, &message.events[0], &mut sink)+ .map(|stored| serde_json::from_slice::<Value>(&stored).unwrap());+ let replacement = match outcome {+ Outcome::Kept => {+ assert_eq!(stored, None, "{event}");+ continue;+ }+ Outcome::Stored(mime) => {+ assert_eq!(sink.images.len(), 1, "{event}");+ assert_eq!(sink.images[0].mime, mime, "{event}");+ sink.images[0].reference.clone()+ }+ Outcome::Marker(reason) => {+ assert!(sink.images.is_empty(), "{event}");+ reason.marker().to_owned()+ }+ };+ let mut expected = event.clone();+ *expected.pointer_mut(&pointer).unwrap() = json!(replacement);+ assert_eq!(stored, Some(expected), "{event}");}}}diff --git a/rust/ai-research/replay-core-v4/src/scan/images.rs b/rust/ai-research/replay-core-v4/src/scan/images.rsindex c68c0e46cdf0..4d685bb1b406 100644--- a/rust/ai-research/replay-core-v4/src/scan/images.rs+++ b/rust/ai-research/replay-core-v4/src/scan/images.rs@@ -1,26 +1,70 @@-//! Moves inline images out of a v4 block. Each base64 raster data URI becomes an `image:v4` reference, and its-//! decoded bytes go to the flush's image pack. A base64 AVIF data URI becomes [`AVIF_MARKER`], so the block holds no-//! AVIF image.+//! Moves inline images out of a v4 block. Each raster data URI becomes an `image:v4` reference, and its decoded bytes+//! go to the flush's image pack. An image the image scrub cannot read, or one over a cap, becomes a [`Declined`] marker.//!//! The scan is one pass over the colons, for `data:image/` in any case. A data URI looks the same in an attribute, a CSS `url()`, a//! `srcset` candidate or stylesheet text, so the scan needs no knowledge of where it sits. Every other byte is-//! copied as it is. Nothing here protects privacy, so an image the scan cannot take stays inline in the sealed block.+//! copied as it is. An SVG data URI stays inline, because SVG is text that the text scrub reads in place.use std::borrow::Cow;use std::collections::HashSet;-use base64::engine::general_purpose::STANDARD;+use base64::alphabet;+use base64::engine::general_purpose::{GeneralPurpose, GeneralPurposeConfig};+use base64::engine::DecodePaddingMode;use base64::Engine;+use crate::bytescan as scan;use crate::refs::{image_ref, reference_hash};-/// What an AVIF data URI becomes. The dataset keeps no AVIF image, inline or fetched.-pub const AVIF_MARKER: &str = "image:v4:declined:avif";+pub const DECLINED_PREFIX: &str = "image:v4:declined:";++/// Readers of the dataset accept only these exact markers.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum Declined {+ Avif,+ OverLimit,+ Undecodable,+ Unsupported,+}++impl Declined {+ pub const ALL: [Declined; 4] = [+ Declined::Avif,+ Declined::OverLimit,+ Declined::Undecodable,+ Declined::Unsupported,+ ];++ pub const fn label(self) -> &'static str {+ match self {+ Declined::Avif => "avif",+ Declined::OverLimit => "over_limit",+ Declined::Undecodable => "undecodable",+ Declined::Unsupported => "unsupported",+ }+ }++ pub const fn marker(self) -> &'static str {+ match self {+ Declined::Avif => "image:v4:declined:avif",+ Declined::OverLimit => "image:v4:declined:over_limit",+ Declined::Undecodable => "image:v4:declined:undecodable",+ Declined::Unsupported => "image:v4:declined:unsupported",+ }+ }+}++/// Base64 as a browser reads a data URI: the padding is optional, and the bits after the last byte do not count.+const FORGIVING_BASE64: GeneralPurpose = GeneralPurpose::new(+ &alphabet::STANDARD,+ GeneralPurposeConfig::new()+ .with_decode_padding_mode(DecodePaddingMode::Indifferent)+ .with_decode_allow_trailing_bits(true),+);/// The longest media type and parameters a taken data URI has before its comma. A raster type with parameters, such/// as `image/png;charset=utf-8;base64`, fits. The bound keeps the scan linear.const MAX_META: usize = 64;-const MAX_SUBTYPE: usize = 32;#[derive(Debug, Clone, Copy)]pub struct ImageLimits {@@ -56,10 +100,9 @@ pub struct ImageSink<'k> {seen: HashSet<String>,total_bytes: usize,pub images: Vec<Image>,- /// Images left inline: over a cap, not decodable, or not a supported raster type.+ /// SVG data URIs, the only images left inline.pub left_inline: u32,- /// AVIF images replaced by [`AVIF_MARKER`].- pub removed_avif: u32,+ pub declined: [u32; Declined::ALL.len()],}impl<'k> ImageSink<'k> {@@ -78,21 +121,41 @@ impl<'k> ImageSink<'k> {total_bytes: 0,images: Vec::new(),left_inline: 0,- removed_avif: 0,+ declined: [0; Declined::ALL.len()],+ }+ }++ /// The bytes name the media type, because a page can label a data URI with any type: a font as `image/ttf`, or nothing at all.+ pub(crate) fn take(&mut self, bytes: Vec<u8>) -> String {+ match self.store(bytes) {+ Ok(reference) => reference,+ Err(reason) => self.decline(reason),}}- /// Takes a base64 image and returns its reference, with the caps of v3 `ImageCollector::collect`. The bytes name- /// the media type, because a page can label a data URI with any type: a font as `image/ttf`, or nothing at all.- pub(crate) fn take(&mut self, payload: &[u8]) -> Option<String> {+ pub(crate) fn take_base64(&mut self, payload: &[u8]) -> String {if payload.len() > self.limits.max_image_bytes.div_ceil(3) * 4 {- return None;+ return self.decline(Declined::OverLimit);+ }+ match FORGIVING_BASE64.decode(payload) {+ Ok(bytes) => self.take(bytes),+ Err(_) => self.decline(Declined::Undecodable),}- let bytes = STANDARD.decode(payload).ok()?;+ }++ pub(crate) fn decline(&mut self, reason: Declined) -> String {+ self.declined[reason as usize] += 1;+ reason.marker().to_owned()+ }++ fn store(&mut self, bytes: Vec<u8>) -> Result<String, Declined> {if bytes.len() > self.limits.max_image_bytes {- return None;+ return Err(Declined::OverLimit);}- let mime = raster_type(&bytes)?;+ if is_avif_header(&bytes) {+ return Err(Declined::Avif);+ }+ let mime = raster_type(&bytes).ok_or(Declined::Unsupported)?;let reference = image_ref(self.team_id,self.key_month,@@ -100,12 +163,12 @@ impl<'k> ImageSink<'k> {);// An image repeated in one message is stored once and referenced at every place it appears, past the caps too.if self.seen.contains(&reference) {- return Some(reference);+ return Ok(reference);}if self.images.len() >= self.limits.max_images|| self.total_bytes + bytes.len() > self.limits.max_total_bytes{- return None;+ return Err(Declined::OverLimit);}self.seen.insert(reference.clone());self.total_bytes += bytes.len();@@ -114,12 +177,11 @@ impl<'k> ImageSink<'k> {mime: mime.to_owned(),bytes,});- Some(reference)+ Ok(reference)}}/// The raster format of the bytes, by its signature: PNG, JPEG, GIF or WebP, the formats the image fetcher also stores.-/// Anything else stays inline in the sealed block.fn raster_type(bytes: &[u8]) -> Option<&'static str> {if bytes.starts_with(b"\x89PNG\r\n\x1a\n") {Some("image/png")@@ -134,25 +196,6 @@ fn raster_type(bytes: &[u8]) -> Option<&'static str> {}}-/// The media type of a raster subtype, or `None` for a type v3 does not collect. SVG stays inline on purpose.-fn raster_mime(subtype: &str) -> Option<String> {- if subtype.starts_with("svg") || subtype.contains("bmp") || subtype.contains("avif") {- return None;- }- if subtype.is_empty()- || subtype.len() > MAX_SUBTYPE- || !subtype- .bytes()- .all(|b| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'+' | b'-'))- {- return None;- }- Some(match subtype {- "jpg" | "pjpeg" => "image/jpeg".to_owned(),- _ => format!("image/{subtype}"),- })-}-fn is_base64(byte: u8) -> bool {byte.is_ascii_alphanumeric() || matches!(byte, b'+' | b'/' | b'=')}@@ -186,7 +229,7 @@ fn is_descriptor(bytes: &[u8], start: usize) -> bool {}/// Whether the payload may go on past the break at `end`. The run before the break is then only a prefix of the image,-/// so the caller leaves the URI inline. Line-wrapped base64 continues it, and so does an escape inside it: an escaped+/// so the caller reads on with [`wrapped_end`]. Line-wrapped base64 continues it, and so does an escape inside it: an escaped/// slash (`\/` or `\u002F`), or a CSS escape such as `\2f `, which arrives with its backslash doubled. Only the `\"`/// that closes a quoted CSS url ends it for sure.fn continues_past(bytes: &[u8], end: usize) -> bool {@@ -216,63 +259,13 @@ fn continues_past(bytes: &[u8], end: usize) -> bool {enum DataUri {NotImage,- Inline,- /// The reference, and where the payload ends.- Taken(String, usize),- /// An AVIF image, and where its payload ends.- Avif(usize),+ Svg,+ Replaced { replacement: String, end: usize },}/// The longest `ftyp` box the scan reads for its brands. Real files list a handful of brands in far fewer bytes.const MAX_FTYP_BYTES: usize = 256;-/// The first bytes of a base64 payload, decoded across the line breaks, whitespace and escaped slashes that-/// [`wrapped_end`] also reads, so that a break inside the header does not hide it.-fn payload_head(bytes: &[u8], start: usize, decoded: usize) -> Vec<u8> {- let want = decoded.div_ceil(3) * 4;- let mut base64 = Vec::with_capacity(want);- let mut position = start;- while base64.len() < want {- match bytes.get(position..) {- Some([b'\\', b'n' | b'r' | b't', ..]) => position += 2,- Some([b'\\', b'/', ..]) => {- base64.push(b'/');- position += 2;- }- Some([b'\\', b'u', b'0', b'0', b'2', b'f' | b'F', ..]) => {- base64.push(b'/');- position += 6;- }- // A space before a srcset descriptor such as `100w` ends the payload.- Some([b' ', ..]) => {- let next = position + bytes[position..].iter().take_while(|&&b| b == b' ').count();- if is_descriptor(bytes, next) {- break;- }- position = next;- }- Some([b'\t' | b'\n' | b'\r', ..]) => position += 1,- // Padding ends the payload.- Some([b'=', ..]) => {- base64.push(b'=');- break;- }- Some([byte, ..]) if is_base64(*byte) => {- base64.push(*byte);- position += 1;- }- _ => break,- }- }- if base64.last() == Some(&b'=') {- while base64.len() % 4 != 0 {- base64.push(b'=');- }- }- base64.truncate(base64.len() / 4 * 4);- STANDARD.decode(&base64).unwrap_or_default()-}-/// Whether the bytes start with an ISO-BMFF `ftyp` box that names an AVIF brand, as its major brand or as one of its/// compatible brands.fn is_avif_header(bytes: &[u8]) -> bool {@@ -289,16 +282,21 @@ fn is_avif_header(bytes: &[u8]) -> bool {}/// Where a base64 payload that `continues_past` its first break really ends: after the line breaks, whitespace and-/// escaped slashes inside it. `None` when it holds an escape that the scan does not read, so the URI cannot be cut out.-fn wrapped_end(bytes: &[u8], mut end: usize) -> Option<usize> {+/// escapes inside it. Also whether the scan read every escape, which only escaped slashes and line breaks are.+fn wrapped_end(bytes: &[u8], mut end: usize) -> (usize, bool) {+ let mut readable = true;while continues_past(bytes, end) {let before = end;loop {match bytes.get(end..) {+ Some([b'\\', b'"', ..]) => break,Some([b'\\', b'n' | b'r' | b't' | b'/', ..]) => end += 2,Some([b'\\', b'u', b'0', b'0', b'2', b'f' | b'F', ..]) => end += 6,+ Some([b'\\', _, ..]) => {+ readable = false;+ end += 2;+ }Some([b' ' | b'\t' | b'\n' | b'\r', ..]) => end += 1,- Some([b'\\', ..]) => return None,_ => break,}}@@ -307,64 +305,173 @@ fn wrapped_end(bytes: &[u8], mut end: usize) -> Option<usize> {.position(|&b| !is_base64(b)).unwrap_or(bytes.len() - end);if end == before {- return None;+ return (end, false);}}- Some(end)+ (end, readable)+}++fn wrapped_payload(bytes: &[u8], start: usize, end: usize) -> Vec<u8> {+ let mut base64 = Vec::with_capacity(end - start);+ let mut rest = &bytes[start..end];+ loop {+ rest = match rest {+ [b'\\', b'u', b'0', b'0', b'2', b'f' | b'F', tail @ ..] | [b'\\', b'/', tail @ ..] => {+ base64.push(b'/');+ tail+ }+ [b'\\', _, tail @ ..] | [b' ' | b'\t' | b'\n' | b'\r', tail @ ..] => tail,+ [byte, tail @ ..] => {+ base64.push(*byte);+ tail+ }+ [] => return base64,+ };+ }+}++/// Where a data URI ends whose payload is not base64: at the end of its JSON string, at the quote or parenthesis that+/// opened it, or at whitespace when no quote opened it. A URL may hold a bare `'` or `)`, so a quoted URI ends only at its own quote.+fn uri_end(bytes: &[u8], start: usize, from: usize) -> usize {+ let opener = start.checked_sub(1).map(|at| bytes[at]);+ // `\"` opens a quoted CSS url in stylesheet text. A `"` after an even number of backslashes opens the JSON string.+ let backslashes = bytes[..start.saturating_sub(1)]+ .iter()+ .rev()+ .take_while(|&&b| b == b'\\')+ .count();+ let quoted = match opener {+ Some(b'\'') => true,+ Some(b'"') => backslashes % 2 == 1,+ _ => false,+ };+ let close = match opener {+ Some(b'(') => Some(b')'),+ Some(b'\'') => Some(b'\''),+ _ => None,+ };+ let mut at = from;+ while let Some(&byte) = bytes.get(at) {+ match byte {+ b'"' => return at,+ b'\\' => match bytes.get(at + 1) {+ Some(b'"') => return at,+ Some(b'n' | b'r' | b't') if !quoted => return at,+ _ => at += 2,+ },+ b' ' | b'\t' | b'\n' | b'\r' if !quoted => return at,+ _ if Some(byte) == close => return at,+ _ => at += 1,+ }+ }+ bytes.len()+}++fn percent_decoded(raw: &[u8]) -> Option<Vec<u8>> {+ let quoted = [b"\"".as_slice(), raw, b"\""].concat();+ let text = scan::unescape("ed, (0, quoted.len())).ok()?;+ Some(percent_encoding::percent_decode(text.as_bytes()).collect())+}++enum Meta {+ Read(String, usize),+ NoPayload,+ TooLong(String),+}++/// A JSON writer may escape a `/` as `\/` or `\u002F`, so both read as `/`.+fn read_meta(bytes: &[u8], start: usize) -> Meta {+ let mut meta = String::new();+ let mut at = start;+ let mut whitespace = false;+ while at - start <= MAX_META {+ match bytes.get(at..) {+ Some([b',', ..]) => return Meta::Read(meta, at),+ Some([b'\\', b'/', ..]) => {+ meta.push('/');+ at += 2;+ }+ Some([b'\\', b'u', b'0', b'0', b'2', b'f' | b'F', ..]) => {+ meta.push('/');+ at += 6;+ }+ Some([b'"' | b'\\', ..] | []) | None => return Meta::NoPayload,+ Some([byte, ..]) => {+ whitespace |= byte.is_ascii_whitespace();+ meta.push(char::from(byte.to_ascii_lowercase()));+ at += 1;+ }+ }+ }+ if whitespace {+ Meta::NoPayload+ } else {+ Meta::TooLong(meta)+ }}/// Reads the data URI whose `data:` starts at `start`, with the rules of v3 `collectable_data_uri_bytes`: the media/// type and `base64` in any case, and any parameters before the comma.fn read_data_uri(bytes: &[u8], start: usize, sink: &mut ImageSink<'_>) -> DataUri {let meta_start = start + b"data:".len();- let window = &bytes[meta_start..bytes.len().min(meta_start + MAX_META + 1)];- if !window- .get(..b"image/".len())- .is_some_and(|prefix| prefix.eq_ignore_ascii_case(b"image/"))- {- return DataUri::NotImage;- }- // A quote or a backslash cannot be in the media type. A comma after one is outside the JSON string of the URI.- let Some(comma) = window- .iter()- .position(|&b| matches!(b, b',' | b'"' | b'\\'))- .filter(|&at| window[at] == b',')- else {- return DataUri::Inline;+ let (meta, comma) = match read_meta(bytes, meta_start) {+ Meta::Read(meta, comma) if meta.starts_with("image/") => (meta, comma),+ // A browser reads parameters of any length, so the URI can hold an image whose payload the scan cannot find.+ Meta::TooLong(meta) if meta.starts_with("image/") => {+ let end = uri_end(bytes, start, meta_start);+ return DataUri::Replaced {+ replacement: sink.decline(Declined::Undecodable),+ end,+ };+ }+ _ => return DataUri::NotImage,};- let meta = String::from_utf8_lossy(&window[..comma]).to_ascii_lowercase();let media_type = meta.split(';').next().unwrap_or_default();let subtype = &media_type["image/".len()..];+ if subtype.starts_with("svg") {+ return DataUri::Svg;+ }+ let payload_start = comma + 1;if !meta.contains("base64") {- return DataUri::Inline;+ let end = uri_end(bytes, start, payload_start);+ let replacement = match &bytes[payload_start..end] {+ [] => return DataUri::NotImage,+ _ if subtype.starts_with("avif") => sink.decline(Declined::Avif),+ raw => match percent_decoded(raw) {+ Some(image) => sink.take(image),+ None => sink.decline(Declined::Undecodable),+ },+ };+ return DataUri::Replaced { replacement, end };}- let payload_start = meta_start + comma + 1;- let end = bytes[payload_start..]+ let run_end = bytes[payload_start..].iter().position(|&b| !is_base64(b)).map_or(bytes.len(), |length| payload_start + length);- // The label or the bytes can name AVIF, so both count.- if subtype.starts_with("avif")- || is_avif_header(&payload_head(bytes, payload_start, MAX_FTYP_BYTES))- {- return match wrapped_end(bytes, end) {- Some(end) => DataUri::Avif(end),- None => DataUri::Inline,- };- }- if raster_mime(subtype).is_none() {- return DataUri::Inline;- }- if continues_past(bytes, end) {- return DataUri::Inline;- }- match sink.take(&bytes[payload_start..end]) {- Some(reference) => DataUri::Taken(reference, end),- None => DataUri::Inline,+ let (end, readable) = if continues_past(bytes, run_end) {+ wrapped_end(bytes, run_end)+ } else {+ (run_end, true)+ };+ if end == payload_start {+ return DataUri::NotImage;}+ // The label or the bytes can name AVIF, so both count.+ let replacement = if subtype.starts_with("avif") {+ sink.decline(Declined::Avif)+ } else if !readable {+ sink.decline(Declined::Undecodable)+ } else if end == run_end {+ sink.take_base64(&bytes[payload_start..end])+ } else {+ sink.take_base64(&wrapped_payload(bytes, payload_start, end))+ };+ DataUri::Replaced { replacement, end }}-/// Replaces every inline raster image in `bytes` with its reference. Returns the input unchanged when it holds none.+/// Replaces every inline image in `bytes` but SVG with its reference or a marker. Returns the input unchanged when it holds none.+/// The scan matches `data:image/` byte for byte, so a JSON escape of one of its letters hides the image. Give it bytes+/// that [`canonical_strings`](crate::bytescan::canonical_strings) wrote.pub fn extract_images<'b>(bytes: &'b [u8], sink: &mut ImageSink<'_>) -> Cow<'b, [u8]> {let mut out: Option<Vec<u8>> = None;let mut copied_to = 0;@@ -382,18 +489,11 @@ pub fn extract_images<'b>(bytes: &'b [u8], sink: &mut ImageSink<'_>) -> Cow<'b,}match read_data_uri(bytes, start, sink) {DataUri::NotImage => {}- DataUri::Inline => sink.left_inline += 1,- DataUri::Taken(reference, end) => {- let out = out.get_or_insert_with(|| Vec::with_capacity(bytes.len()));- out.extend_from_slice(&bytes[copied_to..start]);- out.extend_from_slice(reference.as_bytes());- copied_to = end;- }- DataUri::Avif(end) => {- sink.removed_avif += 1;+ DataUri::Svg => sink.left_inline += 1,+ DataUri::Replaced { replacement, end } => {let out = out.get_or_insert_with(|| Vec::with_capacity(bytes.len()));out.extend_from_slice(&bytes[copied_to..start]);- out.extend_from_slice(AVIF_MARKER.as_bytes());+ out.extend_from_slice(replacement.as_bytes());copied_to = end;}}@@ -409,6 +509,8 @@ pub fn extract_images<'b>(bytes: &'b [u8], sink: &mut ImageSink<'_>) -> Cow<'b,#[cfg(test)]mod tests {+ use base64::engine::general_purpose::STANDARD;+use super::*;const KEY: [u8; 32] = [3; 32];@@ -439,6 +541,31 @@ mod tests {text.into_bytes()}+ fn declined(reason: Declined) -> [u32; Declined::ALL.len()] {+ let mut counts = [0; Declined::ALL.len()];+ counts[reason as usize] = 1;+ counts+ }++ #[test]+ fn writes_only_the_markers_readers_accept() {+ assert_eq!(+ Declined::ALL.map(Declined::marker),+ [+ "image:v4:declined:avif",+ "image:v4:declined:over_limit",+ "image:v4:declined:undecodable",+ "image:v4:declined:unsupported",+ ]+ );+ for reason in Declined::ALL {+ assert_eq!(+ reason.marker(),+ format!("{DECLINED_PREFIX}{}", reason.label())+ );+ }+ }+#[test]fn moves_images_out_of_every_slot_and_back_without_changing_a_byte() {let png = data_uri("image/png", &png(b"png bytes"));@@ -460,89 +587,207 @@ mod tests {}#[test]- fn leaves_what_it_cannot_take_inline() {- let big = png(&[1u8; 992]);+ fn takes_wrapped_escaped_unpadded_and_percent_encoded_payloads() {+ let image = png(&[0xfb; 62]);+ let base64 = STANDARD.encode(&image);+ assert!(base64.contains('/') && base64.ends_with('='));+ let (head, tail) = base64.split_at(40);+ let percent: String = image+ .iter()+ .map(|&b| {+ if b.is_ascii_alphanumeric() {+ char::from(b).to_string()+ } else {+ format!("%{b:02X}")+ }+ })+ .collect();let cases = [- (data_uri("image/svg+xml", b"<svg/>"), ImageLimits::default()),- (data_uri("image/bmp", b"BM"), ImageLimits::default()),- // The bytes name the type, so a font, an icon, an empty payload or a mislabeled text stays inline.(- data_uri("image/ttf", b"\x00\x01\x00\x00font"),- ImageLimits::default(),+ r#"{"src":""#,+ format!(r"data:image/png;base64,{head}\n{tail}"),+ r#""}"#,),(- data_uri("image/x-icon", b"\x00\x00\x01\x00icon"),- ImageLimits::default(),+ r#"{"src":""#,+ format!(r"data:image/png;base64,{head}\r\n {tail}"),+ r#""}"#,),- ("data:image/gif;base64,".to_owned(), ImageLimits::default()),(- data_uri("image/png", b"not an image"),- ImageLimits::default(),+ r#"{"src":""#,+ format!(r"data:image\/png;base64,{}", base64.replace('/', r"\/")),+ r#""}"#,),- ("data:image/png;utf8,abc".to_owned(), ImageLimits::default()),- // A URI without a comma before its string ends: the next comma separates JSON fields.(- r#"data:image/png;base64","alt":"x"#.to_owned(),- ImageLimits::default(),+ r#"{"src":""#,+ format!("data:image/png;base64,{}", base64.replace('/', r"\u002F")),+ r#""}"#,),(- "data:image/png;base64,iVBOR\\u002Fw".to_owned(),- ImageLimits::default(),+ r#"{"src":""#,+ format!("data:image/png;base64,{}", base64.trim_end_matches('=')),+ r#""}"#,),- // A CSS escape for `/`, with the backslash doubled by JSON.+ (r#"{"src":""#, format!("data:image/png,{percent}"), r#""}"#),(- format!(- "{}\\\\2f {}",- data_uri("image/png", &png(&[7; 52])),- STANDARD.encode([8; 30])- ),- ImageLimits::default(),+ r#"{"style":"background:url("#,+ format!("data:image/png,{percent}"),+ r#") no-repeat"}"#,),- // Line-wrapped base64: the run before the break is only part of the image.(- format!(- "{}\\n{}",- data_uri("image/png", &png(&[7; 52])),- STANDARD.encode([8; 30])- ),- ImageLimits::default(),+ r#"{"textContent":".a{background:url(\""#,+ format!("data:image/png;charset=binary,{percent}"),+ r#"\")}"}"#,),- // A wrapped tail that looks like a srcset descriptor.(- format!("{}\\n123x", data_uri("image/png", &png(&[7; 52]))),- ImageLimits::default(),+ r#"{"srcset":""#,+ format!("data:image/png,{percent}"),+ r#" 2x, https://example.com/a.png 1x"}"#,),+ ];+ for (before, uri, after) in cases {+ let event = format!("{before}{uri}{after}");+ let mut images = sink(ImageLimits::default());+ let out = extract_images(event.as_bytes(), &mut images);+ assert_eq!(images.images.len(), 1, "{event}");+ assert_eq!(images.images[0].bytes, image, "{event}");+ assert_eq!(+ String::from_utf8_lossy(&out),+ format!("{before}{}{after}", images.images[0].reference),+ "{event}"+ );+ }+ }++ #[test]+ fn declines_each_image_the_image_scrub_cannot_read() {+ let big = png(&[1u8; 992]);+ let limits = ImageLimits::default();+ let wrapped = data_uri("image/png", &png(&[7; 52]));+ let cases = [(data_uri("image/png", &big),ImageLimits {max_image_bytes: 999,- ..ImageLimits::default()+ ..limits},+ Declined::OverLimit,),(data_uri("image/png", &big),ImageLimits {max_images: 0,- ..ImageLimits::default()+ ..limits},+ Declined::OverLimit,+ ),+ (+ data_uri("image/png", &big),+ ImageLimits {+ max_total_bytes: 999,+ ..limits+ },+ Declined::OverLimit,+ ),+ (+ "data:image/png;base64,iVBORw0KG".to_owned(),+ limits,+ Declined::Undecodable,+ ),+ (+ "data:image/png;base64,iVBO=Rw0KGgo".to_owned(),+ limits,+ Declined::Undecodable,+ ),+ // A CSS escape for `/`, with the backslash doubled by JSON.+ (+ format!(r"{wrapped}\\2f {}", STANDARD.encode([8; 30])),+ limits,+ Declined::Undecodable,+ ),+ (+ format!(+ "data:image/png;name={};base64,{}",+ "a".repeat(80),+ STANDARD.encode(png(b"x"))+ ),+ limits,+ Declined::Undecodable,+ ),+ (+ data_uri("image/ttf", b"\x00\x01\x00\x00font"),+ limits,+ Declined::Unsupported,+ ),+ (+ data_uri("image/x-icon", b"\x00\x00\x01\x00icon"),+ limits,+ Declined::Unsupported,+ ),+ (+ data_uri("image/bmp", b"BMbitmap"),+ limits,+ Declined::Unsupported,+ ),+ (+ data_uri("image/png", b"not an image"),+ limits,+ Declined::Unsupported,+ ),+ (+ "data:image/png;utf8,abc".to_owned(),+ limits,+ Declined::Unsupported,),];- for (uri, limits) in cases {- let event = format!(r#"{{"src":"{uri}"}}"#);+ for (uri, limits, reason) in cases {+ let event = format!(r#"{{"src":"{uri}","alt":"x"}}"#);let mut images = sink(limits);+ let out = extract_images(event.as_bytes(), &mut images);+ assert_eq!(+ String::from_utf8_lossy(&out),+ format!(r#"{{"src":"{}","alt":"x"}}"#, reason.marker()),+ "{uri}"+ );+ assert_eq!(+ (images.images.len(), images.declined),+ (0, declined(reason)),+ "{uri}"+ );+ }+ }++ #[test]+ fn leaves_svg_and_data_uris_without_a_payload_as_they_are() {+ for event in [+ format!(r#"{{"src":"{}"}}"#, data_uri("image/svg+xml", b"<svg/>")),+ r#"{"src":"data:image/svg+xml,%3Csvg%2F%3E"}"#.to_owned(),+ r#"{"src":"data:image/gif;base64,"}"#.to_owned(),+ // A URI without a comma before its string ends: the next comma separates JSON fields.+ r#"{"src":"data:image/png;base64","alt":"x"}"#.to_owned(),+ r#"{"textContent":"Use data:image/png, data:image/gif, or a link"}"#.to_owned(),+ r#"{"textContent":"Paste a data:image/ URI in the box below and the page shows a preview of the image right here"}"#.to_owned(),+ ] {+ let mut images = sink(ImageLimits::default());assert!(matches!(extract_images(event.as_bytes(), &mut images),Cow::Borrowed(_)),- "{uri}"+ "{event}"+ );+ assert_eq!(images.declined, [0; Declined::ALL.len()], "{event}");+ assert_eq!(+ images.left_inline,+ u32::from(event.contains("svg")),+ "{event}");- assert_eq!((images.images.len(), images.left_inline), (0, 1), "{uri}");}}#[test]fn replaces_every_avif_data_uri_with_the_marker() {+ let marker = Declined::Avif.marker();let avif = STANDARD.encode(b"\x00\x00\x00\x1cftypavif\x00\x00\x00\x00avifmif1miaf");// AVIF as a compatible brand only, under the major brand `mif1`.let compatible = STANDARD.encode(b"\x00\x00\x00\x18ftypmif1\x00\x00\x00\x00miafavif");@@ -561,10 +806,14 @@ mod tests {let out = extract_images(event.as_bytes(), &mut images);assert_eq!(String::from_utf8_lossy(&out),- format!(r#"{{"src":"{AVIF_MARKER}","alt":"x"}}"#),+ format!(r#"{{"src":"{marker}","alt":"x"}}"#),+ "{uri}"+ );+ assert_eq!(+ (images.images.len(), images.declined),+ (0, declined(Declined::Avif)),"{uri}");- assert_eq!((images.images.len(), images.removed_avif), (0, 1), "{uri}");}// A short, padded payload before a srcset descriptor, under another label.let event = format!(r#"{{"srcset":"data:image/png;base64,{avif} 100w"}}"#);@@ -572,7 +821,7 @@ mod tests {let out = extract_images(event.as_bytes(), &mut images);assert_eq!(String::from_utf8_lossy(&out),- format!(r#"{{"srcset":"{AVIF_MARKER} 100w"}}"#)+ format!(r#"{{"srcset":"{marker} 100w"}}"#));}@@ -585,6 +834,7 @@ mod tests {// A label that does not name the bytes: the stored type comes from the bytes.format!("data:image/jpeg;base64,{payload}"),format!("data:image/png+xml;base64,{payload}"),+ format!("data:image/bmp;base64,{payload}"),] {let event = format!(r#"{{"src":"{uri}"}}"#);let mut images = sink(ImageLimits::default());@@ -612,22 +862,34 @@ mod tests {let out = extract_images(event.as_bytes(), &mut images);let reference = &images.images[0].reference;assert_eq!(- String::from_utf8_lossy(&out)- .matches(reference.as_str())- .count(),- 2+ String::from_utf8_lossy(&out),+ format!(+ r#"["{reference}","{}","{reference}"]"#,+ Declined::OverLimit.marker()+ )+ );+ assert_eq!(+ (images.images.len(), images.declined),+ (1, declined(Declined::OverLimit)));- assert_eq!((images.images.len(), images.left_inline), (1, 1));}#[test]fn scans_unterminated_data_uris_in_linear_time() {- // Without a bound, each match searches to the `;` at the end of the buffer for its subtype.- let bytes = format!("{};", "data:image/".repeat(200_000));let started = std::time::Instant::now();+ // Without a bound, each match searches to the `"` at the end of the buffer for its comma.+ let spaced = format!("{}\"", "data:image/ ".repeat(200_000));+ let mut images = sink(ImageLimits::default());+ assert!(matches!(+ extract_images(spaced.as_bytes(), &mut images),+ Cow::Borrowed(_)+ ));+ let unbroken = format!("{};", "data:image/".repeat(200_000));let mut images = sink(ImageLimits::default());- extract_images(bytes.as_bytes(), &mut images);- assert_eq!(images.left_inline, 200_000);+ assert_eq!(+ extract_images(unbroken.as_bytes(), &mut images).as_ref(),+ Declined::Undecodable.marker().as_bytes()+ );assert!(started.elapsed() < std::time::Duration::from_secs(5));}}diff --git a/rust/ai-research/replay-recorder-v4/src/rewrite.rs b/rust/ai-research/replay-recorder-v4/src/rewrite.rsindex b18a88c9b006..461ed6acc513 100644--- a/rust/ai-research/replay-recorder-v4/src/rewrite.rs+++ b/rust/ai-research/replay-recorder-v4/src/rewrite.rs@@ -5,8 +5,8 @@//! A srcset is rebuilt instead: a declined candidate leaves the list, and so does each remote candidate of a picture//! source whose type the fetcher does not store. An inline image, such as an SVG data URI, stays. A browser skips a//! source with no candidates, so it falls back to a source or an img that can load, where a marker in the list would-//! show a broken image. The writer rebuilds each srcset in the bytes it stores, after inline images left and escapes-//! were written again, so the rebuild never depends on a prediction of those bytes.+//! show a broken image. The writer rebuilds each srcset in the bytes it stores, after escapes were written again and+//! inline images left, so the rebuild never depends on a prediction of those bytes.use std::borrow::Cow;use std::cell::Cell;@@ -14,10 +14,12 @@ use std::collections::{BTreeMap, HashMap, HashSet};use aho_corasick::{AhoCorasick, AhoCorasickKind, MatchKind};use metrics::counter;-use replay_core_v4::bytescan::{unescape, write_json_string, Span};+use replay_core_v4::bytescan::{canonical_strings, unescape, write_json_string, Span};use replay_core_v4::inspect::css::{escape_in_identifier, rewrite_functions, CssRewrite};use replay_core_v4::inspect::dom::{for_each_srcset_candidate, Srcset};-use replay_core_v4::scan::images::{extract_images, ImageLimits, ImageSink, AVIF_MARKER};+use replay_core_v4::scan::images::{+ extract_images, Declined, ImageLimits, ImageSink, DECLINED_PREFIX,+};use replay_core_v4::scan::tokens::{Token, Tokens};use replay_core_v4::seal::{image_content_key, image_url_key};use replay_core_v4::url_policy::is_remote;@@ -25,68 +27,6 @@ use replay_core_v4::url_policy::is_remote;use crate::scrub::{Scrub, Scrubber};use crate::urls::url_reference;-/// The bytes with each JSON string that holds an escape `write_json_string` does not write, such as `\/` or `\u0041`,-/// written again as it writes them. Capture writes events that way, but the JSON inside a `cv` payload arrives as the-/// page wrote it, so a URL could hide from the rewrite behind an escape.-pub fn canonical_strings(bytes: &[u8]) -> Cow<'_, [u8]> {- if !bytes.contains(&b'\\') {- return Cow::Borrowed(bytes);- }- let mut out: Option<Vec<u8>> = None;- let mut copied_to = 0;- let mut position = 0;- while let Some(start) = bytes[position..]- .iter()- .position(|&b| b == b'"')- .map(|at| position + at)- {- let mut end = start + 1;- let mut canonical = true;- while end < bytes.len() && bytes[end] != b'"' {- if bytes[end] == b'\\' {- match bytes.get(end + 1) {- Some(b'"' | b'\\' | b'b' | b'f' | b'n' | b'r' | b't') => {}- Some(b'u') => canonical &= is_written_escape(bytes.get(end + 2..end + 6)),- _ => canonical = false,- }- end += 1;- }- end += 1;- }- if end >= bytes.len() {- break;- }- if !canonical {- if let Ok(text) = unescape(bytes, (start, end + 1)) {- let out = out.get_or_insert_with(|| Vec::with_capacity(bytes.len()));- out.extend_from_slice(&bytes[copied_to..start]);- write_json_string(&text, out);- copied_to = end + 1;- }- }- position = end + 1;- }- match out {- Some(mut out) => {- out.extend_from_slice(&bytes[copied_to..]);- Cow::Owned(out)- }- None => Cow::Borrowed(bytes),- }-}--/// Whether `write_json_string` writes this `\u` escape: a control character without a short escape, in lowercase hex.-fn is_written_escape(hex: Option<&[u8]>) -> bool {- let Some(&[b'0', b'0', high, low]) = hex else {- return false;- };- let digit = |b: u8| matches!(b, b'0'..=b'9' | b'a'..=b'f');- digit(high)- && digit(low)- && high <= b'1'- && !matches!((high, low), (b'0', b'8' | b'9' | b'a' | b'c' | b'd'))-}-/// What a declined URL becomes. The reason is the policy's metric label.pub fn declined_marker(reason: &str) -> String {format!("imageurl:v4:declined:{reason}")@@ -296,8 +236,8 @@ pub struct UrlRewrite {css_texts: HashMap<String, usize>,/// Remote URLs that the block names only in picture sources of a type the fetcher does not store.unfetched: HashSet<String>,- /// Each inspected srcset value as the stored block holds it. Only these are rebuilt, so a srcset key outside an- /// image slot, such as in a custom event's payload, stays as it is.+ /// Each inspected srcset value as the stored block holds it, under [`without_image_ids`]. Only these are rebuilt, so+ /// a srcset key outside an image slot, such as in a custom event's payload, stays as it is.stored_srcsets: HashSet<Vec<u8>>,url_targets: HashMap<String, usize>,team_id: i64,@@ -318,7 +258,7 @@ pub struct PassCounters {srcset_rewrites: BTreeMap<&'static str, u64>,css_rewrites: BTreeMap<(&'static str, &'static str), u64>,url_rewrites: BTreeMap<(&'static str, &'static str), u64>,- pub removed_avif: u64,+ pub removed_images: [u64; Declined::ALL.len()],}impl PassCounters {@@ -343,8 +283,10 @@ impl PassCounters {).increment(count);}- counter!("ai_research_v4_recorder_removed_images_total", "format" => "avif")- .increment(self.removed_avif);+ for (reason, &count) in Declined::ALL.iter().zip(&self.removed_images) {+ counter!("ai_research_v4_recorder_removed_images_total", "reason" => reason.label())+ .increment(count);+ }}}@@ -432,7 +374,7 @@ impl UrlRewrite {.filter(|&(_, unfetched)| unfetched).map(|(url, _)| url.to_owned()).collect();- // The writer takes the inline images out of the raw event and then writes its escapes again, so each value+ // The writer writes the escapes of the raw event again and then takes its inline images out, so each value// goes through the same two steps.let content_key = image_content_key(team_key);let stored_srcsets = srcsets@@ -440,8 +382,9 @@ impl UrlRewrite {.map(|srcset| {let mut sink =ImageSink::new(&content_key, team_id, key_month, ImageLimits::default());- let extracted = extract_images(&srcset.raw, &mut sink);- canonical_strings(&extracted).into_owned()+ let canonical = canonical_strings(&srcset.raw);+ let stored = extract_images(&canonical, &mut sink);+ without_image_ids(&stored).into_owned()}).collect();let matcher = (!patterns.is_empty())@@ -566,7 +509,10 @@ impl UrlRewrite {let Some(end) = string_end(bytes, start) else {continue;};- if !self.stored_srcsets.contains(&bytes[start..end]) {+ if !self+ .stored_srcsets+ .contains(without_image_ids(&bytes[start..end]).as_ref())+ {continue;}let Ok(text) = unescape(bytes, (start, end)) else {@@ -877,6 +823,38 @@ fn string_end(bytes: &[u8], start: usize) -> Option<usize> {None}+/// The bytes with each image reference and declined marker cut to its `image:v4:` prefix. The image caps of a message+/// decide whether an image becomes a reference or `over_limit`, and the srcset prediction in [`UrlRewrite::new`]+/// cannot see the other images of that message.+fn without_image_ids(bytes: &[u8]) -> Cow<'_, [u8]> {+ const PREFIX: &[u8] = b"image:v4:";+ let find = |bytes: &[u8]| {+ bytes+ .windows(PREFIX.len())+ .position(|window| window == PREFIX)+ };+ let Some(mut at) = find(bytes) else {+ return Cow::Borrowed(bytes);+ };+ let mut out = Vec::with_capacity(bytes.len());+ let mut rest = bytes;+ loop {+ out.extend_from_slice(&rest[..at + PREFIX.len()]);+ rest = &rest[at + PREFIX.len()..];+ let id_length = rest+ .iter()+ .position(|&b| !(b.is_ascii_alphanumeric() || matches!(b, b'_' | b'-' | b':')))+ .unwrap_or(rest.len());+ rest = &rest[id_length..];+ match find(rest) {+ Some(next) => at = next,+ None => break,+ }+ }+ out.extend_from_slice(rest);+ Cow::Owned(out)+}+/// The srcset with only the candidates the block keeps, each remote URL as its reference, how it was rebuilt, and the/// URL targets of the remote URLs it held. `None` when every candidate stays, because the URL targets then replace it.fn rebuilt_srcset(@@ -902,7 +880,7 @@ fn rebuilt_srcset(}};for_each_srcset_candidate(text, |url, descriptors| {- if url == AVIF_MARKER {+ if url.starts_with(DECLINED_PREFIX) {removed = true;} else if !is_remote(url) {keep(url, descriptors);@@ -927,6 +905,9 @@ fn rebuilt_srcset(#[cfg(test)]mod tests {+ use base64::engine::general_purpose::STANDARD;+ use base64::Engine;+use super::*;const TEAM_KEY: [u8; 32] = [6; 32];@@ -983,6 +964,31 @@ mod tests {assert!(rewrite.finish().safe);}+ #[test]+ fn rebuilds_a_srcset_whose_image_the_message_cap_declined() {+ let png = format!(+ "data:image/png;base64,{}",+ STANDARD.encode(b"\x89PNG\r\n\x1a\nbytes")+ );+ let svg = "data:image/svg+xml,%3Csvg%2F%3E";+ let text = format!("{png} 1x, {svg} 2x");+ let srcset = Srcset {+ raw: serde_json::to_vec(&text).unwrap(),+ text,+ unfetched_type: false,+ };+ let mut rewrite = UrlRewrite::new(&[], &[srcset], &[], 7, "2026-10", &TEAM_KEY);+ // A message past its image cap stores the PNG as `over_limit`, where a fresh sink takes it.+ let stored = format!(+ r#"{{"srcset":"{} 1x, {svg} 2x"}}"#,+ Declined::OverLimit.marker()+ );+ assert_eq!(+ String::from_utf8(rewrite.apply(stored.as_bytes(), true).into_owned()).unwrap(),+ format!(r#"{{"srcset":"{svg} 2x"}}"#)+ );+ }+#[test]fn rewrites_a_url_that_a_json_escape_hid() {let url = "https://cdn.example.com/a.png?token=secret";diff --git a/rust/ai-research/replay-recorder-v4/src/write.rs b/rust/ai-research/replay-recorder-v4/src/write.rsindex 601cda4e5f56..35c5e4ce9c88 100644--- a/rust/ai-research/replay-recorder-v4/src/write.rs+++ b/rust/ai-research/replay-recorder-v4/src/write.rs@@ -19,14 +19,14 @@ use replay_core_v4::inspect::stats::MAX_URL_FINGERPRINTS;use replay_core_v4::keys::batch::{FlushKeys, SessionId, SessionKey as Key};use replay_core_v4::refs::url_fingerprint;use replay_core_v4::scan::cv::extract_event_images;-use replay_core_v4::scan::images::{Image, ImageLimits, ImageSink};+use replay_core_v4::scan::images::{Declined, Image, ImageLimits, ImageSink};use replay_core_v4::seal::{image_content_key, url_fingerprint_key};use tokio::sync::Semaphore;use crate::batch::{Block, SessionKey};use crate::index::{BlockIndexRow, ImageIndexRow};-use crate::rewrite::{canonical_strings, Fallback, RewriteReport, UrlRewrite};+use crate::rewrite::{Fallback, RewriteReport, UrlRewrite};use crate::scrub::Scrub;use crate::store::{BlobStore, StoreError};@@ -134,7 +134,7 @@ fn prepare_block(let mut images = Vec::new();let mut kept = HashSet::new();let mut counts = WriterCounts::default();- let mut removed_avif = 0u64;+ let mut removed_images = [0u64; Declined::ALL.len()];for message in &block.messages {let mut window = Vec::new();write_json_string(@@ -152,15 +152,13 @@ fn prepare_block(out.write_all(b"[")?;out.write_all(&window)?;out.write_all(b",")?;- let canonical = canonical_strings(- extracted- .as_deref()- .unwrap_or(&message.inner[event.span.0..event.span.1]),- );+ let stored = extracted+ .as_deref()+ .unwrap_or(&message.inner[event.span.0..event.span.1]);// The inspector reads image slots in full snapshots and mutations only.let dom_event =event.kind == Some(2) || (event.kind == Some(3) && event.source == Some(0));- let event_bytes = rewrite.apply(&canonical, dom_event);+ let event_bytes = rewrite.apply(stored, dom_event);out.write_all(&event_bytes)?;out.write_all(b"]\n")?;let line = (window.len() + event_bytes.len() + 4) as u64;@@ -174,7 +172,9 @@ fn prepare_block(.map(|i| i.bytes.len() as u64).sum::<u64>();counts.inline += sink.left_inline;- removed_avif += u64::from(sink.removed_avif);+ for (removed, declined) in removed_images.iter_mut().zip(sink.declined) {+ *removed += u64::from(declined);+ }images.extend(sink.images.into_iter()@@ -182,7 +182,7 @@ fn prepare_block();}let mut report = rewrite.finish();- report.counters.removed_avif = removed_avif;+ report.counters.removed_images = removed_images;counts.scrubbed = report.scrubbed;Ok((images, counts, report))}@@ -1059,6 +1059,7 @@ mod tests {"data:image/avif;base64,{}",STANDARD.encode(b"\x00\x00\x00\x1cftypavif\x00\x00\x00\x00avifmif1other"));+ let bitmap = format!("data:image/bmp;base64,{}", STANDARD.encode(b"BMbitmap"));// The dataset keeps no AVIF, so an inline one leaves the list too.let inline_avif = format!("data:image/avif;base64,{}",@@ -1076,7 +1077,8 @@ mod tests {source(12, json!({"type": "image/webp", "srcset": format!("{shared} 1x, {other_avif} 2x")})),// An empty type gives no hint, so the source keeps its candidate.source(9, json!({"type": "", "srcset": format!("{typeless} 1x")})),- source(4, json!({"type": "image/webp", "srcset": format!("{signed} 2x, {webp} 1x")})),+ // An inline image the image scrub cannot read leaves the list as a declined URL does.+ source(4, json!({"type": "image/webp", "srcset": format!("{signed} 2x, {webp} 1x, {bitmap} 3x")})),{"type": 2, "tagName": "img", "attributes": {"src": fallback}, "childNodes": [], "id": 5},]});let img = json!({"type": 2, "tagName": "img", "attributes": {"src": bare}, "childNodes": [], "id": 10});@@ -1141,10 +1143,10 @@ mod tests {#[test]fn rebuilds_a_srcset_as_the_block_holds_it_after_escapes_are_written_again() {- // The `\/` escapes keep the PNG inline, and the writer then writes them as `/`.- let png = STANDARD.encode(png(&[0xff; 30]));- assert!(png.contains('/'));- let inline = format!("data:image/png;base64,{png}");+ // An SVG stays inline, and the writer writes its `\/` escapes as `/`.+ let svg = STANDARD.encode(b"<svg><!--??? --></svg>");+ assert!(svg.contains('/'));+ let inline = format!("data:image/svg+xml;base64,{svg}");let events = format!(r#"[{{"type":3,"timestamp":1,"data":{{"source":0,"adds":[],"removes":[],"texts":[],"attributes":[{{"id":99,"attributes":{{"srcset":"{} 1x, https://10.0.0.7/a.png 2x"}}}}]}}}}]"#,inline.replace('/', "\\/")