Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(ai-research-v4): name inline image types from their bytes (#124856)

Nicholas Waltz38f2242b3648
diff --git a/products/ai_training/docs/replay-data.md b/products/ai_training/docs/replay-data.md
index fcc93e201614..1afa91561ae9 100644
--- a/products/ai_training/docs/replay-data.md
+++ b/products/ai_training/docs/replay-data.md
@@ -461,7 +461,7 @@ The recorder streams each block into the encoder. Read a block as a stream: a Br
- posthog-js compresses some fields of an event and marks the event `cv: "2024-10"`. The recorder expands those fields to plain JSON and removes the `cv` key, as the replay player does, because the gzip strings do not compress again. An event that does not decode stays exactly as it arrived.
- An inline image becomes `image:v4:{team_id}:{YYYY-MM}:{hash}` in the block, where the month is the UTC month of the write.
-- The recorder takes base64 PNG, JPEG, GIF and WebP data URIs in any letter case and with parameters, under the v3 size limits. It also takes the image frames of the posthog-js canvas recorder. Anything else stays inline in the sealed block, for example SVG or line-wrapped base64.
+- The recorder takes base64 PNG, JPEG, GIF and WebP data URIs in any letter case and with parameters, under the v3 size limits. The image bytes name the format, not the data URI's label, so the image index `mime` is always one of these four. A data URI whose bytes have none of their signatures stays inline, for example a font labeled `image/ttf`, an icon or an empty payload. It also takes the image frames of the posthog-js canvas recorder. Anything else stays inline in the sealed block, for example SVG or line-wrapped base64.
- A remote image URL becomes `imageurl:v4:{team_id}:{YYYY-MM}:{hash}` in the block and in its fetch job, where the month is the UTC month of the write. The block and the job always carry the same reference, so a reader opens the fetched image directly.
- A URL that the URL policy refuses becomes `imageurl:v4:declined:{reason}` in the block, for example `declined:credential`, `declined:non_public_host` or `declined:tracking_beacon`. The original URL cannot be recovered. The recorder counts rewrites and refusals in `ai_research_v4_recorder_image_url_rewrites_total`.
- The rewrite covers every remote URL that the recorder reads from an image slot: image sources, `srcset`, SVG image links, video posters, `<picture>` sources and CSS `url(...)` in style attributes, style sheets and style rules. It also covers `src`, `srcset`, `href` and `poster` in an attribute change on a node the recorder does not know, for example after a restart. `srcset` keeps its descriptors and CSS keeps its `url(...)` syntax. It is not limited by the 512 URLs per message that get fetch jobs.
diff --git a/rust/ai-research/replay-core-v4/src/scan/cv.rs b/rust/ai-research/replay-core-v4/src/scan/cv.rs
index c70d46bb4ada..f830ba131755 100644
--- a/rust/ai-research/replay-core-v4/src/scan/cv.rs
+++ b/rust/ai-research/replay-core-v4/src/scan/cv.rs
@@ -463,9 +463,9 @@ fn stored_event(inner: &[u8], event: &Event, sink: &mut ImageSink<'_>) -> Option
let frame = event
.data
.and_then(|data| canvas::frame_image(inner, event.kind, event.source, data));
- if let Some((mime, span)) = frame {
+ if let Some((_, span)) = frame {
let payload = scan::unescape(inner, span).ok();
- match payload.and_then(|p| sink.take(mime, p.as_bytes())) {
+ match payload.and_then(|p| sink.take(p.as_bytes())) {
Some(reference) => {
let mut quoted = Vec::with_capacity(reference.len() + 2);
scan::write_json_string(&reference, &mut quoted);
@@ -597,7 +597,10 @@ mod tests {
#[test]
fn stores_cv_events_expanded_with_their_images_moved_out() {
- let image = format!("data:image/png;base64,{}", STANDARD.encode(b"png bytes"));
+ let image = format!(
+ "data:image/png;base64,{}",
+ STANDARD.encode(b"\x89PNG\r\n\x1a\npng bytes")
+ );
let snapshot = json!({"node": {"attributes": {"src": image}}, "initialOffset": {"top": 0}});
let adds = json!([{"parentId": 1, "node": {"attributes": {"src": image}}}]);
let mut message = message_with(json!([
@@ -801,7 +804,10 @@ mod tests {
#[test]
fn leaves_a_cv_event_that_does_not_decode_as_it_arrived() {
- let image = format!("data:image/png;base64,{}", STANDARD.encode(b"png bytes"));
+ let image = format!(
+ "data:image/png;base64,{}",
+ STANDARD.encode(b"\x89PNG\r\n\x1a\npng bytes")
+ );
let message = message_with(json!([
{"type": 2, "timestamp": 1, "data": format!("not gzip {image}"), "cv": "2024-10"},
// One list that does not decode fails the whole event, as in the player. So does a list sent uncompressed.
@@ -826,9 +832,15 @@ mod tests {
{"property": "drawImage", "args": draw_args},
]}})
};
+ // posthog-js encodes the frame in the type the blob names.
let bitmap = |mime: &str, buffer: &str| {
+ let bytes: &[u8] = match mime {
+ "image/webp" => b"RIFF\x00\x00\x00\x00WEBPframe",
+ "image/gif" => b"GIF89aframe",
+ _ => b"\x89PNG\r\n\x1a\nframe",
+ };
json!({"rr_type": "ImageBitmap", "args": [{"rr_type": "Blob", "type": mime,
- "data": [{"rr_type": buffer, "base64": STANDARD.encode(b"frame bytes")}]}]})
+ "data": [{"rr_type": buffer, "base64": STANDARD.encode(bytes)}]}]})
};
let cases = [
(
diff --git a/rust/ai-research/replay-core-v4/src/scan/images.rs b/rust/ai-research/replay-core-v4/src/scan/images.rs
index 1e8f64641901..e1f94707b548 100644
--- a/rust/ai-research/replay-core-v4/src/scan/images.rs
+++ b/rust/ai-research/replay-core-v4/src/scan/images.rs
@@ -75,19 +75,17 @@ impl<'k> ImageSink<'k> {
}
}
- /// Takes a base64 image and returns its reference, with the checks and caps of v3 `ImageCollector::collect`.
- pub(crate) fn take(&mut self, mime: &str, payload: &[u8]) -> Option<String> {
+ /// Takes a base64 image and returns its reference, with the caps of v3 `ImageCollector::collect`. The bytes name
+ /// the media type, because a page can label a data URI with any type: a font as `image/ttf`, or nothing at all.
+ pub(crate) fn take(&mut self, payload: &[u8]) -> Option<String> {
if payload.len() > self.limits.max_image_bytes.div_ceil(3) * 4 {
return None;
}
let bytes = STANDARD.decode(payload).ok()?;
- // v3 refuses BMP and ISO-BMFF (AVIF, HEIC) bytes under any label.
- if bytes.len() > self.limits.max_image_bytes
- || bytes.starts_with(b"BM")
- || bytes.get(4..8) == Some(b"ftyp")
- {
+ if bytes.len() > self.limits.max_image_bytes {
return None;
}
+ let mime = raster_type(&bytes)?;
let reference = image_ref(
self.team_id,
self.key_month,
@@ -113,6 +111,22 @@ impl<'k> ImageSink<'k> {
}
}
+/// The raster format of the bytes, by its signature: PNG, JPEG, GIF or WebP, the formats the image fetcher also stores.
+/// Anything else stays inline in the sealed block.
+fn raster_type(bytes: &[u8]) -> Option<&'static str> {
+ if bytes.starts_with(b"\x89PNG\r\n\x1a\n") {
+ Some("image/png")
+ } else if bytes.starts_with(b"\xff\xd8\xff") {
+ Some("image/jpeg")
+ } else if bytes.starts_with(b"GIF87a") || bytes.starts_with(b"GIF89a") {
+ Some("image/gif")
+ } else if bytes.starts_with(b"RIFF") && bytes.get(8..12) == Some(b"WEBP") {
+ Some("image/webp")
+ } else {
+ None
+ }
+}
+
/// The media type of a raster subtype, or `None` for a type v3 does not collect. SVG stays inline on purpose.
fn raster_mime(subtype: &str) -> Option<String> {
if subtype.starts_with("svg") || subtype.contains("bmp") || subtype.contains("avif") {
@@ -221,10 +235,9 @@ fn read_data_uri(bytes: &[u8], start: usize, sink: &mut ImageSink<'_>) -> DataUr
};
let meta = String::from_utf8_lossy(&window[..comma]).to_ascii_lowercase();
let media_type = meta.split(';').next().unwrap_or_default();
- let Some(mime) = raster_mime(&media_type["image/".len()..]).filter(|_| meta.contains("base64"))
- else {
+ if raster_mime(&media_type["image/".len()..]).is_none() || !meta.contains("base64") {
return DataUri::Inline;
- };
+ }
let payload_start = meta_start + comma + 1;
let end = bytes[payload_start..]
.iter()
@@ -233,7 +246,7 @@ fn read_data_uri(bytes: &[u8], start: usize, sink: &mut ImageSink<'_>) -> DataUr
if continues_past(bytes, end) {
return DataUri::Inline;
}
- match sink.take(&mime, &bytes[payload_start..end]) {
+ match sink.take(&bytes[payload_start..end]) {
Some(reference) => DataUri::Taken(reference, end),
None => DataUri::Inline,
}
@@ -281,6 +294,15 @@ mod tests {
const KEY: [u8; 32] = [3; 32];
+ /// Image bytes that start with a raster signature, so that the scan takes them.
+ fn png(body: &[u8]) -> Vec<u8> {
+ [b"\x89PNG\r\n\x1a\n".as_slice(), body].concat()
+ }
+
+ fn jpeg(body: &[u8]) -> Vec<u8> {
+ [b"\xff\xd8\xff".as_slice(), body].concat()
+ }
+
fn sink(limits: ImageLimits) -> ImageSink<'static> {
ImageSink::new(&KEY, 7, "2026-10", limits)
}
@@ -300,8 +322,8 @@ mod tests {
#[test]
fn moves_images_out_of_every_slot_and_back_without_changing_a_byte() {
- let png = data_uri("image/png", b"png bytes");
- let jpeg = data_uri("image/JPEG", b"jpeg bytes");
+ let png = data_uri("image/png", &png(b"png bytes"));
+ let jpeg = data_uri("image/JPEG", &jpeg(b"jpeg bytes"));
// The stylesheet text sits inside a JSON string, so its quotes arrive escaped.
let event = format!(
r#"{{"type":2,"data":{{"node":{{"attributes":{{"src":"{png}","srcset":"{jpeg} 1x, {png} 2x","style":"background:url({jpeg})"}},"childNodes":[{{"textContent":".a{{background:url(\"{png}\")}}"}}]}}}}}}"#
@@ -320,10 +342,24 @@ mod tests {
#[test]
fn leaves_what_it_cannot_take_inline() {
- let big = vec![1u8; 1000];
+ let big = png(&[1u8; 992]);
let cases = [
(data_uri("image/svg+xml", b"<svg/>"), ImageLimits::default()),
(data_uri("image/bmp", b"BM"), ImageLimits::default()),
+ // The bytes name the type, so a font, an icon, an empty payload or a mislabeled text stays inline.
+ (
+ data_uri("image/ttf", b"\x00\x01\x00\x00font"),
+ ImageLimits::default(),
+ ),
+ (
+ data_uri("image/x-icon", b"\x00\x00\x01\x00icon"),
+ ImageLimits::default(),
+ ),
+ ("data:image/gif;base64,".to_owned(), ImageLimits::default()),
+ (
+ data_uri("image/png", b"not an image"),
+ ImageLimits::default(),
+ ),
("data:image/png;utf8,abc".to_owned(), ImageLimits::default()),
// A URI without a comma before its string ends: the next comma separates JSON fields.
(
@@ -338,7 +374,7 @@ mod tests {
(
format!(
"{}\\\\2f {}",
- data_uri("image/png", &[7; 60]),
+ data_uri("image/png", &png(&[7; 52])),
STANDARD.encode([8; 30])
),
ImageLimits::default(),
@@ -347,14 +383,14 @@ mod tests {
(
format!(
"{}\\n{}",
- data_uri("image/png", &[7; 60]),
+ data_uri("image/png", &png(&[7; 52])),
STANDARD.encode([8; 30])
),
ImageLimits::default(),
),
// A wrapped tail that looks like a srcset descriptor.
(
- format!("{}\\n123x", data_uri("image/png", &[7; 60])),
+ format!("{}\\n123x", data_uri("image/png", &png(&[7; 52]))),
ImageLimits::default(),
),
(
@@ -388,16 +424,19 @@ mod tests {
#[test]
fn takes_the_data_uris_v3_collects() {
- let payload = STANDARD.encode(b"icon bytes");
+ let payload = STANDARD.encode(png(b"icon bytes"));
for uri in [
format!("data:image/png;charset=utf-8;base64,{payload}"),
format!("DATA:image/PNG;BASE64,{payload}"),
- format!("data:image/x-icon;base64,{payload}"),
+ // A label that does not name the bytes: the stored type comes from the bytes.
+ format!("data:image/jpeg;base64,{payload}"),
+ format!("data:image/png+xml;base64,{payload}"),
] {
let event = format!(r#"{{"src":"{uri}"}}"#);
let mut images = sink(ImageLimits::default());
let out = extract_images(event.as_bytes(), &mut images);
assert_eq!(images.images.len(), 1, "{uri}");
+ assert_eq!(images.images[0].mime, "image/png", "{uri}");
assert_eq!(
out.as_ref(),
format!(r#"{{"src":"{}"}}"#, images.images[0].reference).as_bytes()
@@ -407,10 +446,10 @@ mod tests {
#[test]
fn references_a_repeated_image_after_the_count_cap() {
- let first = data_uri("image/png", b"first");
+ let first = data_uri("image/png", &png(b"first"));
let event = format!(
r#"["{first}","{}","{first}"]"#,
- data_uri("image/png", b"second")
+ data_uri("image/png", &png(b"second"))
);
let mut images = sink(ImageLimits {
max_images: 1,
diff --git a/rust/ai-research/replay-recorder-v4/src/write.rs b/rust/ai-research/replay-recorder-v4/src/write.rs
index cd5932694044..5054b8905605 100644
--- a/rust/ai-research/replay-recorder-v4/src/write.rs
+++ b/rust/ai-research/replay-recorder-v4/src/write.rs
@@ -689,8 +689,13 @@ mod tests {
}
}
- fn uri(bytes: &[u8]) -> String {
- format!("data:image/png;base64,{}", STANDARD.encode(bytes))
+ /// PNG bytes: the signature, then `body`. The image scan takes only bytes of a raster format.
+ fn png(body: &[u8]) -> Vec<u8> {
+ [b"\x89PNG\r\n\x1a\n".as_slice(), body].concat()
+ }
+
+ fn uri(body: &[u8]) -> String {
+ format!("data:image/png;base64,{}", STANDARD.encode(png(body)))
}
/// The block's stored lines, after the inspector named its remote URLs.
@@ -1027,7 +1032,7 @@ mod tests {
let old_ref = replay_core_v4::refs::image_ref(
7,
"2026-10",
- &replay_core_v4::refs::reference_hash(&image_content_key(&[6; 32]), b"old"),
+ &replay_core_v4::refs::reference_hash(&image_content_key(&[6; 32]), &png(b"old")),
);
let object = object_key("2026-10-15", at());
let mut sealed: Vec<SealedBlock> = blocks
@@ -1047,7 +1052,7 @@ mod tests {
sealed[0].counts,
WriterCounts {
extracted: 2,
- bytes: 6,
+ bytes: 2 * 8 + 6,
inline: 0,
..sealed[0].counts
}
@@ -1129,7 +1134,7 @@ mod tests {
let sealed_image = &pack[row.byte_start as usize..=row.byte_end as usize];
assert_eq!(
open_frame(&[6; 32], Version::V4, &image_context, sealed_image).unwrap(),
- b"new"
+ png(b"new")
);
// The team month key seals images, as the fetcher's, so a session key does not open them.
assert!(open_frame(&[5; 32], Version::V4, &image_context, sealed_image).is_err());