Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(ai-research-v4): remove inline avif images from v4 blocks (#124867)

Nicholas Waltz2f91117acbdd
diff --git a/products/ai_training/docs/replay-data.md b/products/ai_training/docs/replay-data.md
index 1afa91561ae9..dc2650abb37c 100644
--- a/products/ai_training/docs/replay-data.md
+++ b/products/ai_training/docs/replay-data.md
@@ -461,11 +461,12 @@ The recorder streams each block into the encoder. Read a block as a stream: a Br
- posthog-js compresses some fields of an event and marks the event `cv: "2024-10"`. The recorder expands those fields to plain JSON and removes the `cv` key, as the replay player does, because the gzip strings do not compress again. An event that does not decode stays exactly as it arrived.
- An inline image becomes `image:v4:{team_id}:{YYYY-MM}:{hash}` in the block, where the month is the UTC month of the write.
-- The recorder takes base64 PNG, JPEG, GIF and WebP data URIs in any letter case and with parameters, under the v3 size limits. The image bytes name the format, not the data URI's label, so the image index `mime` is always one of these four. A data URI whose bytes have none of their signatures stays inline, for example a font labeled `image/ttf`, an icon or an empty payload. It also takes the image frames of the posthog-js canvas recorder. Anything else stays inline in the sealed block, for example SVG or line-wrapped base64.
+- The recorder takes base64 PNG, JPEG, GIF and WebP data URIs in any letter case and with parameters, under the v3 size limits. The image bytes name the format, not the data URI's label, so the image index `mime` is always one of these four. A data URI whose bytes have none of their signatures stays inline, for example a font labeled `image/ttf`, an icon or an empty payload.
+- The dataset keeps no AVIF image. A base64 data URI that its label or its bytes name as AVIF becomes `image:v4:declined:avif`, and a srcset loses such a candidate. The recorder counts these as `ai_research_v4_recorder_removed_images_total{format="avif"}`. A payload broken by an escape the scan does not read stays inline. It also takes the image frames of the posthog-js canvas recorder. Anything else stays inline in the sealed block, for example SVG or line-wrapped base64.
- A remote image URL becomes `imageurl:v4:{team_id}:{YYYY-MM}:{hash}` in the block and in its fetch job, where the month is the UTC month of the write. The block and the job always carry the same reference, so a reader opens the fetched image directly.
- A URL that the URL policy refuses becomes `imageurl:v4:declined:{reason}` in the block, for example `declined:credential`, `declined:non_public_host` or `declined:tracking_beacon`. The original URL cannot be recovered. The recorder counts rewrites and refusals in `ai_research_v4_recorder_image_url_rewrites_total`.
- The rewrite covers every remote URL that the recorder reads from an image slot: image sources, `srcset`, SVG image links, video posters, `<picture>` sources and CSS `url(...)` in style attributes, style sheets and style rules. It also covers `src`, `srcset`, `href` and `poster` in an attribute change on a node the recorder does not know, for example after a restart. `srcset` keeps its descriptors and CSS keeps its `url(...)` syntax. It is not limited by the 512 URLs per message that get fetch jobs.
-- A srcset in an image slot keeps only the candidates the recorder stores. A candidate the URL policy refuses leaves the list instead of becoming a `declined` marker, and the rest keep their descriptors. A `<picture>` source whose `type` names a media type the image fetcher does not store, such as `image/avif`, gets no fetch job and loses its remote candidates. Its inline images, such as SVG data URIs, stay. A browser skips a source with no candidates, so the replay falls back to the next source or the `img`. The recorder counts these as `ai_research_v4_recorder_srcset_rewrites_total{outcome="trimmed"|"emptied"}`. A srcset that an inline image extraction changed falls back to the rewrite of each URL.
+- A srcset in an image slot keeps only the candidates the recorder stores. A candidate the URL policy refuses leaves the list instead of becoming a `declined` marker, and the rest keep their descriptors. A `<picture>` source whose `type` names a media type the image fetcher does not store, such as `image/avif`, gets no fetch job and loses its remote candidates, unless a source of a stored type names the same URL. Its inline images, such as SVG data URIs, stay. The writer rebuilds each srcset that the inspector read in an image slot, in the bytes it stores, after the inline images left and the escapes were written again. It separates the candidates with a comma and a space. A `srcset` key elsewhere, for example in a custom event's payload, stays as it is. A browser skips a source with no candidates, so the replay falls back to the next source or the `img`. The recorder counts these as `ai_research_v4_recorder_srcset_rewrites_total{outcome="trimmed"|"emptied"}`.
- The recorder reads a protocol-relative URL, such as `//cdn.example.com/a.png`, as `https`. It gets the same reference or `declined` marker as the `https` form, and the rewrite replaces the text as the block holds it. A browser reads a backslash as a slash there, so `\\cdn.example.com/a.png` counts too, and so does a URL such as `https:\\cdn.example.com/a.png` with any mix of slashes and backslashes after the scheme, or none: on a page of another scheme the browser loads `https:cdn.example.com/a.png` from that host.
- Some URLs stay as they arrived: relative URLs, remote URLs outside image slots (links, frames, text, CSS custom properties and other CSS properties), the `image-set()` candidates the recorder does not choose, URLs inside a compressed value that did not decode or that the decode budget skipped, and URL text written in another form than the recorder read it, for example with a CSS escape. The recorder counts the last kind as `outcome="unrewritten"`.
- Before the rewrite, the recorder writes again any JSON string that holds an escape capture would not write, for example `\/`, so an escaped URL is found too. A scheme matches in any letter case.
diff --git a/rust/ai-research/replay-core-v4/src/inspect/dom.rs b/rust/ai-research/replay-core-v4/src/inspect/dom.rs
index 2eaeaddca6c0..44d919155b8c 100644
--- a/rust/ai-research/replay-core-v4/src/inspect/dom.rs
+++ b/rust/ai-research/replay-core-v4/src/inspect/dom.rs
@@ -217,7 +217,7 @@ pub struct RemoteUrls {
#[derive(Debug, Default)]
struct SrcsetValues {
- texts: HashMap<String, bool>,
+ values: HashMap<Vec<u8>, (String, bool)>,
bytes: usize,
}
@@ -225,6 +225,8 @@ struct SrcsetValues {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Srcset {
pub text: String,
+ /// The value as a JSON string with its quotes and escapes, as the event holds it.
+ pub raw: Vec<u8>,
/// The value belongs to a `<source>` whose `type` the fetcher does not store, so none of its images is stored.
pub unfetched_type: bool,
}
@@ -248,15 +250,18 @@ impl RemoteUrls {
}
/// A value seen on a source of a stored type and on one of another type keeps its candidates.
- fn insert_srcset(&mut self, text: &str, unfetched_type: bool) {
+ /// Keyed by the value as the event holds it, so the writer finds that value again after it stores the event.
+ fn insert_srcset(&mut self, raw: &[u8], text: &str, unfetched_type: bool) {
let srcsets = self.srcsets.get_or_insert_with(Box::default);
- if let Some(known) = srcsets.texts.get_mut(text) {
+ if let Some((_, known)) = srcsets.values.get_mut(raw) {
*known &= unfetched_type;
- } else if srcsets.texts.len() < MAX_BLOCK_REMOTE_URLS
- && srcsets.bytes + text.len() <= MAX_BLOCK_REMOTE_URL_BYTES
+ } else if srcsets.values.len() < MAX_BLOCK_REMOTE_URLS
+ && srcsets.bytes + raw.len() + text.len() <= MAX_BLOCK_REMOTE_URL_BYTES
{
- srcsets.bytes += text.len();
- srcsets.texts.insert(text.to_owned(), unfetched_type);
+ srcsets.bytes += raw.len() + text.len();
+ srcsets
+ .values
+ .insert(raw.to_vec(), (text.to_owned(), unfetched_type));
}
}
@@ -265,19 +270,20 @@ impl RemoteUrls {
for url in &other.urls {
self.insert(url);
}
- for (text, unfetched_type) in other.srcsets.iter().flat_map(|s| &s.texts) {
- self.insert_srcset(text, *unfetched_type);
+ for (raw, (text, unfetched_type)) in other.srcsets.iter().flat_map(|s| &s.values) {
+ self.insert_srcset(raw, text, *unfetched_type);
}
}
pub fn take_srcsets(&mut self) -> Vec<Srcset> {
self.srcsets
.take()
- .map(|srcsets| srcsets.texts)
+ .map(|srcsets| srcsets.values)
.unwrap_or_default()
.into_iter()
- .map(|(text, unfetched_type)| Srcset {
+ .map(|(raw, (text, unfetched_type))| Srcset {
text,
+ raw,
unfetched_type,
})
.collect()
@@ -417,10 +423,10 @@ impl Walk {
}
fn push_srcset_value(&mut self, bytes: &[u8], value: Option<Span>, unfetched_type: bool) {
- if let Some(srcset) = text(bytes, value) {
+ if let (Some(span), Some(srcset)) = (value, text(bytes, value)) {
self.image_urls
.remote
- .insert_srcset(&srcset, unfetched_type);
+ .insert_srcset(&bytes[span.0..span.1], &srcset, unfetched_type);
}
}
@@ -467,7 +473,7 @@ pub fn srcset_candidates(srcset: &str) -> Vec<(&str, &str)> {
}
/// Streams the candidates of [`srcset_candidates`], so that a message-sized srcset needs no list of them.
-fn for_each_srcset_candidate<'s>(srcset: &'s str, mut found: impl FnMut(&'s str, &'s str)) {
+pub fn for_each_srcset_candidate<'s>(srcset: &'s str, mut found: impl FnMut(&'s str, &'s str)) {
let bytes = srcset.as_bytes();
let mut position = 0;
loop {
@@ -2020,8 +2026,12 @@ mod tests {
}
// Srcset values share the byte budget, past which the writer falls back to replacing each URL.
let mut remote = RemoteUrls::default();
- (0..=most_long).for_each(|i| remote.insert_srcset(&url(i, longest), false));
- assert_eq!(remote.take_srcsets().len(), most_long);
+ // A value counts its raw and its unescaped form.
+ (0..=most_long).for_each(|i| {
+ let value = url(i, longest);
+ remote.insert_srcset(value.as_bytes(), &value, false)
+ });
+ assert_eq!(remote.take_srcsets().len(), most_long / 2);
// A mutation past the attribute cap is not read to its end.
let attributes = |count: usize| {
diff --git a/rust/ai-research/replay-core-v4/src/scan/images.rs b/rust/ai-research/replay-core-v4/src/scan/images.rs
index e1f94707b548..c68c0e46cdf0 100644
--- a/rust/ai-research/replay-core-v4/src/scan/images.rs
+++ b/rust/ai-research/replay-core-v4/src/scan/images.rs
@@ -1,5 +1,6 @@
//! Moves inline images out of a v4 block. Each base64 raster data URI becomes an `image:v4` reference, and its
-//! decoded bytes go to the flush's image pack.
+//! decoded bytes go to the flush's image pack. A base64 AVIF data URI becomes [`AVIF_MARKER`], so the block holds no
+//! AVIF image.
//!
//! The scan is one pass over the colons, for `data:image/` in any case. A data URI looks the same in an attribute, a CSS `url()`, a
//! `srcset` candidate or stylesheet text, so the scan needs no knowledge of where it sits. Every other byte is
@@ -13,6 +14,9 @@ use base64::Engine;
use crate::refs::{image_ref, reference_hash};
+/// What an AVIF data URI becomes. The dataset keeps no AVIF image, inline or fetched.
+pub const AVIF_MARKER: &str = "image:v4:declined:avif";
+
/// The longest media type and parameters a taken data URI has before its comma. A raster type with parameters, such
/// as `image/png;charset=utf-8;base64`, fits. The bound keeps the scan linear.
const MAX_META: usize = 64;
@@ -54,6 +58,8 @@ pub struct ImageSink<'k> {
pub images: Vec<Image>,
/// Images left inline: over a cap, not decodable, or not a supported raster type.
pub left_inline: u32,
+ /// AVIF images replaced by [`AVIF_MARKER`].
+ pub removed_avif: u32,
}
impl<'k> ImageSink<'k> {
@@ -72,6 +78,7 @@ impl<'k> ImageSink<'k> {
total_bytes: 0,
images: Vec::new(),
left_inline: 0,
+ removed_avif: 0,
}
}
@@ -212,6 +219,98 @@ enum DataUri {
Inline,
/// The reference, and where the payload ends.
Taken(String, usize),
+ /// An AVIF image, and where its payload ends.
+ Avif(usize),
+}
+
+/// The longest `ftyp` box the scan reads for its brands. Real files list a handful of brands in far fewer bytes.
+const MAX_FTYP_BYTES: usize = 256;
+
+/// The first bytes of a base64 payload, decoded across the line breaks, whitespace and escaped slashes that
+/// [`wrapped_end`] also reads, so that a break inside the header does not hide it.
+fn payload_head(bytes: &[u8], start: usize, decoded: usize) -> Vec<u8> {
+ let want = decoded.div_ceil(3) * 4;
+ let mut base64 = Vec::with_capacity(want);
+ let mut position = start;
+ while base64.len() < want {
+ match bytes.get(position..) {
+ Some([b'\\', b'n' | b'r' | b't', ..]) => position += 2,
+ Some([b'\\', b'/', ..]) => {
+ base64.push(b'/');
+ position += 2;
+ }
+ Some([b'\\', b'u', b'0', b'0', b'2', b'f' | b'F', ..]) => {
+ base64.push(b'/');
+ position += 6;
+ }
+ // A space before a srcset descriptor such as `100w` ends the payload.
+ Some([b' ', ..]) => {
+ let next = position + bytes[position..].iter().take_while(|&&b| b == b' ').count();
+ if is_descriptor(bytes, next) {
+ break;
+ }
+ position = next;
+ }
+ Some([b'\t' | b'\n' | b'\r', ..]) => position += 1,
+ // Padding ends the payload.
+ Some([b'=', ..]) => {
+ base64.push(b'=');
+ break;
+ }
+ Some([byte, ..]) if is_base64(*byte) => {
+ base64.push(*byte);
+ position += 1;
+ }
+ _ => break,
+ }
+ }
+ if base64.last() == Some(&b'=') {
+ while base64.len() % 4 != 0 {
+ base64.push(b'=');
+ }
+ }
+ base64.truncate(base64.len() / 4 * 4);
+ STANDARD.decode(&base64).unwrap_or_default()
+}
+
+/// Whether the bytes start with an ISO-BMFF `ftyp` box that names an AVIF brand, as its major brand or as one of its
+/// compatible brands.
+fn is_avif_header(bytes: &[u8]) -> bool {
+ if bytes.get(4..8) != Some(b"ftyp") {
+ return false;
+ }
+ let size = u32::from_be_bytes([bytes[0], bytes[1], bytes[2], bytes[3]]) as usize;
+ let ftyp = &bytes[..size.clamp(8, MAX_FTYP_BYTES).min(bytes.len())];
+ let is_avif = |brand: &[u8]| matches!(brand, b"avif" | b"avis");
+ ftyp.get(8..12).is_some_and(is_avif)
+ || ftyp
+ .get(16..)
+ .is_some_and(|brands| brands.chunks_exact(4).any(is_avif))
+}
+
+/// Where a base64 payload that `continues_past` its first break really ends: after the line breaks, whitespace and
+/// escaped slashes inside it. `None` when it holds an escape that the scan does not read, so the URI cannot be cut out.
+fn wrapped_end(bytes: &[u8], mut end: usize) -> Option<usize> {
+ while continues_past(bytes, end) {
+ let before = end;
+ loop {
+ match bytes.get(end..) {
+ Some([b'\\', b'n' | b'r' | b't' | b'/', ..]) => end += 2,
+ Some([b'\\', b'u', b'0', b'0', b'2', b'f' | b'F', ..]) => end += 6,
+ Some([b' ' | b'\t' | b'\n' | b'\r', ..]) => end += 1,
+ Some([b'\\', ..]) => return None,
+ _ => break,
+ }
+ }
+ end += bytes[end..]
+ .iter()
+ .position(|&b| !is_base64(b))
+ .unwrap_or(bytes.len() - end);
+ if end == before {
+ return None;
+ }
+ }
+ Some(end)
}
/// Reads the data URI whose `data:` starts at `start`, with the rules of v3 `collectable_data_uri_bytes`: the media
@@ -235,7 +334,8 @@ fn read_data_uri(bytes: &[u8], start: usize, sink: &mut ImageSink<'_>) -> DataUr
};
let meta = String::from_utf8_lossy(&window[..comma]).to_ascii_lowercase();
let media_type = meta.split(';').next().unwrap_or_default();
- if raster_mime(&media_type["image/".len()..]).is_none() || !meta.contains("base64") {
+ let subtype = &media_type["image/".len()..];
+ if !meta.contains("base64") {
return DataUri::Inline;
}
let payload_start = meta_start + comma + 1;
@@ -243,6 +343,18 @@ fn read_data_uri(bytes: &[u8], start: usize, sink: &mut ImageSink<'_>) -> DataUr
.iter()
.position(|&b| !is_base64(b))
.map_or(bytes.len(), |length| payload_start + length);
+ // The label or the bytes can name AVIF, so both count.
+ if subtype.starts_with("avif")
+ || is_avif_header(&payload_head(bytes, payload_start, MAX_FTYP_BYTES))
+ {
+ return match wrapped_end(bytes, end) {
+ Some(end) => DataUri::Avif(end),
+ None => DataUri::Inline,
+ };
+ }
+ if raster_mime(subtype).is_none() {
+ return DataUri::Inline;
+ }
if continues_past(bytes, end) {
return DataUri::Inline;
}
@@ -277,6 +389,13 @@ pub fn extract_images<'b>(bytes: &'b [u8], sink: &mut ImageSink<'_>) -> Cow<'b,
out.extend_from_slice(reference.as_bytes());
copied_to = end;
}
+ DataUri::Avif(end) => {
+ sink.removed_avif += 1;
+ let out = out.get_or_insert_with(|| Vec::with_capacity(bytes.len()));
+ out.extend_from_slice(&bytes[copied_to..start]);
+ out.extend_from_slice(AVIF_MARKER.as_bytes());
+ copied_to = end;
+ }
}
}
match out {
@@ -422,6 +541,41 @@ mod tests {
}
}
+ #[test]
+ fn replaces_every_avif_data_uri_with_the_marker() {
+ let avif = STANDARD.encode(b"\x00\x00\x00\x1cftypavif\x00\x00\x00\x00avifmif1miaf");
+ // AVIF as a compatible brand only, under the major brand `mif1`.
+ let compatible = STANDARD.encode(b"\x00\x00\x00\x18ftypmif1\x00\x00\x00\x00miafavif");
+ for uri in [
+ format!("data:image/png;base64,{compatible}"),
+ // A break inside the header under another label.
+ format!("data:image/png;base64,{}\\n{}", &avif[..8], &avif[8..]),
+ format!("data:image/avif;base64,{avif}"),
+ // The bytes name AVIF under another label.
+ format!("data:image/png;base64,{avif}"),
+ // A line-wrapped payload is cut out whole.
+ format!("data:image/avif;base64,{}\\n{}", &avif[..16], &avif[16..]),
+ ] {
+ let event = format!(r#"{{"src":"{uri}","alt":"x"}}"#);
+ let mut images = sink(ImageLimits::default());
+ let out = extract_images(event.as_bytes(), &mut images);
+ assert_eq!(
+ String::from_utf8_lossy(&out),
+ format!(r#"{{"src":"{AVIF_MARKER}","alt":"x"}}"#),
+ "{uri}"
+ );
+ assert_eq!((images.images.len(), images.removed_avif), (0, 1), "{uri}");
+ }
+ // A short, padded payload before a srcset descriptor, under another label.
+ let event = format!(r#"{{"srcset":"data:image/png;base64,{avif} 100w"}}"#);
+ let mut images = sink(ImageLimits::default());
+ let out = extract_images(event.as_bytes(), &mut images);
+ assert_eq!(
+ String::from_utf8_lossy(&out),
+ format!(r#"{{"srcset":"{AVIF_MARKER} 100w"}}"#)
+ );
+ }
+
#[test]
fn takes_the_data_uris_v3_collects() {
let payload = STANDARD.encode(png(b"icon bytes"));
diff --git a/rust/ai-research/replay-recorder-v4/src/rewrite.rs b/rust/ai-research/replay-recorder-v4/src/rewrite.rs
index 0830e9c75599..8e16b7024479 100644
--- a/rust/ai-research/replay-recorder-v4/src/rewrite.rs
+++ b/rust/ai-research/replay-recorder-v4/src/rewrite.rs
@@ -5,16 +5,18 @@
//! A srcset is rebuilt instead: a declined candidate leaves the list, and so does each remote candidate of a picture
//! source whose type the fetcher does not store. An inline image, such as an SVG data URI, stays. A browser skips a
//! source with no candidates, so it falls back to a source or an img that can load, where a marker in the list would
-//! show a broken image.
+//! show a broken image. The writer rebuilds each srcset in the bytes it stores, after inline images left and escapes
+//! were written again, so the rebuild never depends on a prediction of those bytes.
use std::borrow::Cow;
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
use aho_corasick::{AhoCorasick, AhoCorasickKind, MatchKind};
use metrics::counter;
use replay_core_v4::bytescan::{unescape, write_json_string};
-use replay_core_v4::inspect::dom::{srcset_candidates, Srcset};
-use replay_core_v4::seal::image_url_key;
+use replay_core_v4::inspect::dom::{for_each_srcset_candidate, Srcset};
+use replay_core_v4::scan::images::{extract_images, ImageLimits, ImageSink, AVIF_MARKER};
+use replay_core_v4::seal::{image_content_key, image_url_key};
use replay_core_v4::url_policy::is_remote;
use crate::urls::url_reference;
@@ -119,9 +121,6 @@ struct Target {
/// The policy's reason, for a declined URL.
declined: Option<&'static str>,
found: bool,
- /// For a srcset, which is a whole JSON string with its quotes: how it was rebuilt, and the URL targets its value
- /// holds. Those URLs leave the block with the old value, so a match counts as theirs.
- srcset: Option<(&'static str, Vec<usize>)>,
}
pub struct UrlRewrite {
@@ -132,6 +131,16 @@ pub struct UrlRewrite {
targets: Vec<Target>,
/// Bytes that `apply` returned still hold a declined URL.
declined_kept: bool,
+ srcset_key: AhoCorasick,
+ /// Remote URLs that the block names only in picture sources of a type the fetcher does not store.
+ unfetched: HashSet<String>,
+ /// Each inspected srcset value as the stored block holds it. Only these are rebuilt, so a srcset key outside an
+ /// image slot, such as in a custom event's payload, stays as it is.
+ stored_srcsets: HashSet<Vec<u8>>,
+ url_targets: HashMap<String, usize>,
+ team_id: i64,
+ key_month: String,
+ url_key: [u8; 32],
}
impl UrlRewrite {
@@ -159,33 +168,39 @@ impl UrlRewrite {
replacement: replacement.into_bytes(),
declined,
found: false,
- srcset: None,
});
}
- let url_targets: HashMap<&str, usize> = urls
+ let url_targets: HashMap<String, usize> = urls
.iter()
.enumerate()
- .map(|(index, url)| (url.as_str(), index))
+ .map(|(index, url)| (url.clone(), index))
.collect();
+ // A URL that a source of a stored type also names stays, as the inspector merges identical srcset values.
+ let mut source_types: HashMap<&str, bool> = HashMap::new();
for srcset in srcsets {
- let Some((value, outcome, covers)) =
- rebuilt_srcset(srcset, &url_targets, team_id, key_month, &url_key)
- else {
- continue;
- };
- // The key keeps the pattern to srcset attributes, so a `src` or `href` with the same value stays.
- let mut pattern = SRCSET_KEY.to_vec();
- write_json_string(&srcset.text, &mut pattern);
- patterns.push(pattern);
- let mut replacement = SRCSET_KEY.to_vec();
- write_json_string(&value, &mut replacement);
- targets.push(Target {
- replacement,
- declined: None,
- found: false,
- srcset: Some((outcome, covers)),
+ for_each_srcset_candidate(&srcset.text, |url, _| {
+ if is_remote(url) {
+ *source_types.entry(url).or_insert(true) &= srcset.unfetched_type;
+ }
});
}
+ let unfetched = source_types
+ .into_iter()
+ .filter(|&(_, unfetched)| unfetched)
+ .map(|(url, _)| url.to_owned())
+ .collect();
+ // The writer takes the inline images out of the raw event and then writes its escapes again, so each value
+ // goes through the same two steps.
+ let content_key = image_content_key(team_key);
+ let stored_srcsets = srcsets
+ .iter()
+ .map(|srcset| {
+ let mut sink =
+ ImageSink::new(&content_key, team_id, key_month, ImageLimits::default());
+ let extracted = extract_images(&srcset.raw, &mut sink);
+ canonical_strings(&extracted).into_owned()
+ })
+ .collect();
let matcher = (!patterns.is_empty())
.then(|| {
// A DFA over long URLs can take hundreds of times their bytes. A contiguous NFA takes about ten.
@@ -215,11 +230,72 @@ impl UrlRewrite {
declined,
targets,
declined_kept: false,
+ srcset_key: AhoCorasick::new([SRCSET_KEY]).expect("one literal pattern builds"),
+ unfetched,
+ stored_srcsets,
+ url_targets,
+ team_id,
+ key_month: key_month.to_owned(),
+ url_key,
+ }
+ }
+
+ /// The bytes with each srcset that holds a candidate the block cannot keep rebuilt from the candidates it keeps.
+ fn rebuild_srcsets<'b>(&mut self, bytes: &'b [u8]) -> Cow<'b, [u8]> {
+ let mut out: Option<Vec<u8>> = None;
+ let mut copied_to = 0;
+ for found in self.srcset_key.find_iter(bytes) {
+ let start = found.end();
+ if start < copied_to || bytes.get(start) != Some(&b'"') {
+ continue;
+ }
+ let Some(end) = string_end(bytes, start) else {
+ continue;
+ };
+ if !self.stored_srcsets.contains(&bytes[start..end]) {
+ continue;
+ }
+ let Ok(text) = unescape(bytes, (start, end)) else {
+ continue;
+ };
+ let Some((value, outcome, covers)) = rebuilt_srcset(
+ &text,
+ &self.unfetched,
+ &self.url_targets,
+ self.team_id,
+ &self.key_month,
+ &self.url_key,
+ ) else {
+ continue;
+ };
+ for url in covers {
+ self.targets[url].found = true;
+ }
+ counter!("ai_research_v4_recorder_srcset_rewrites_total", "outcome" => outcome)
+ .increment(1);
+ let out = out.get_or_insert_with(|| Vec::with_capacity(bytes.len()));
+ out.extend_from_slice(&bytes[copied_to..start]);
+ write_json_string(&value, out);
+ copied_to = end;
+ }
+ match out {
+ Some(mut out) => {
+ out.extend_from_slice(&bytes[copied_to..]);
+ Cow::Owned(out)
+ }
+ None => Cow::Borrowed(bytes),
}
}
- /// The bytes with each URL replaced. Returns the input when it holds none.
+ /// The bytes with each srcset rebuilt and each URL replaced. Returns the input when it holds none.
pub fn apply<'b>(&mut self, bytes: &'b [u8]) -> Cow<'b, [u8]> {
+ match self.rebuild_srcsets(bytes) {
+ Cow::Borrowed(bytes) => self.replace_urls(bytes),
+ Cow::Owned(rebuilt) => Cow::Owned(self.replace_urls(&rebuilt).into_owned()),
+ }
+ }
+
+ fn replace_urls<'b>(&mut self, bytes: &'b [u8]) -> Cow<'b, [u8]> {
let Some(matcher) = &self.matcher else {
return Cow::Borrowed(bytes);
};
@@ -228,12 +304,9 @@ impl UrlRewrite {
for found in matcher.find_iter(bytes) {
let (start, end) = (found.start(), found.end());
let index = found.pattern().as_usize();
- // A srcset pattern holds its quotes, so it matches only a whole JSON string.
- let whole_string = self.targets[index].srcset.is_some();
- if !whole_string
- && (start.checked_sub(1).is_some_and(|before| {
- continues_url(bytes[before]) && !ends_with_escape(bytes, start)
- }) || bytes.get(end).is_some_and(|&after| continues_url(after)))
+ if start.checked_sub(1).is_some_and(|before| {
+ continues_url(bytes[before]) && !ends_with_escape(bytes, start)
+ }) || bytes.get(end).is_some_and(|&after| continues_url(after))
{
continue;
}
@@ -243,11 +316,6 @@ impl UrlRewrite {
out.extend_from_slice(&bytes[copied_to..start]);
out.extend_from_slice(&target.replacement);
copied_to = end;
- if let Some((_, covers)) = self.targets[index].srcset.clone() {
- for url in covers {
- self.targets[url].found = true;
- }
- }
}
let out = match out {
Some(mut out) => {
@@ -268,14 +336,6 @@ impl UrlRewrite {
pub fn finish(self) -> bool {
let mut safe = !self.declined_kept;
for target in self.targets {
- if let Some((outcome, _)) = target.srcset {
- counter!(
- "ai_research_v4_recorder_srcset_rewrites_total",
- "outcome" => if target.found { outcome } else { "unrewritten" },
- )
- .increment(1);
- continue;
- }
safe &= target.found || target.declined.is_none();
let outcome = match (target.found, target.declined) {
(false, _) => "unrewritten",
@@ -295,45 +355,62 @@ impl UrlRewrite {
/// The srcset with only the candidates the block keeps, each remote URL as its reference, how it was rebuilt, and the
/// URL targets of the remote URLs it held. `None` when every candidate stays, because the URL targets then replace it.
+/// Where the JSON string that opens at `start` ends, after its closing quote.
+fn string_end(bytes: &[u8], start: usize) -> Option<usize> {
+ let mut position = start + 1;
+ while let Some(&byte) = bytes.get(position) {
+ match byte {
+ b'\\' => position += 2,
+ b'"' => return Some(position + 1),
+ _ => position += 1,
+ }
+ }
+ None
+}
+
fn rebuilt_srcset(
- srcset: &Srcset,
- url_targets: &HashMap<&str, usize>,
+ text: &str,
+ unfetched: &HashSet<String>,
+ url_targets: &HashMap<String, usize>,
team_id: i64,
key_month: &str,
url_key: &[u8; 32],
) -> Option<(String, &'static str, Vec<usize>)> {
- let mut kept = Vec::new();
+ // Streams the candidates into the new value, so that a long srcset holds no list of them.
+ let mut value = String::with_capacity(text.len());
let mut covers = Vec::new();
let mut removed = false;
- for (url, descriptors) in srcset_candidates(&srcset.text) {
- if !is_remote(url) {
- kept.push((Cow::Borrowed(url), descriptors));
- continue;
+ let mut keep = |url: &str, descriptors: &str| {
+ if !value.is_empty() {
+ value.push_str(", ");
+ }
+ value.push_str(url);
+ if !descriptors.is_empty() {
+ value.push(' ');
+ value.push_str(descriptors);
}
- covers.extend(url_targets.get(url));
- match url_reference(url, team_id, key_month, url_key) {
- Ok((_, reference)) if !srcset.unfetched_type => {
- kept.push((Cow::Owned(reference), descriptors))
+ };
+ for_each_srcset_candidate(text, |url, descriptors| {
+ if url == AVIF_MARKER {
+ removed = true;
+ } else if !is_remote(url) {
+ keep(url, descriptors);
+ } else {
+ covers.extend(url_targets.get(url));
+ match url_reference(url, team_id, key_month, url_key) {
+ Ok((_, reference)) if !unfetched.contains(url) => keep(&reference, descriptors),
+ _ => removed = true,
}
- _ => removed = true,
}
- }
+ });
if !removed {
return None;
}
- let outcome = if kept.is_empty() {
+ let outcome = if value.is_empty() {
"emptied"
} else {
"trimmed"
};
- let value = kept
- .iter()
- .map(|(url, descriptors)| match *descriptors {
- "" => url.to_string(),
- descriptors => format!("{url} {descriptors}"),
- })
- .collect::<Vec<_>>()
- .join(", ");
Some((value, outcome, covers))
}
diff --git a/rust/ai-research/replay-recorder-v4/src/write.rs b/rust/ai-research/replay-recorder-v4/src/write.rs
index 5054b8905605..f5ae41548d11 100644
--- a/rust/ai-research/replay-recorder-v4/src/write.rs
+++ b/rust/ai-research/replay-recorder-v4/src/write.rs
@@ -171,6 +171,8 @@ pub fn prepare_block(
.map(|i| i.bytes.len() as u64)
.sum::<u64>();
counts.inline += sink.left_inline;
+ counter!("ai_research_v4_recorder_removed_images_total", "format" => "avif")
+ .increment(u64::from(sink.removed_avif));
images.extend(
sink.images
.into_iter()
@@ -818,12 +820,26 @@ mod tests {
let svg =
"data:image/svg+xml,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%2F%3E";
let remote_svg = "https://cdn.example.com/a.svg";
+ let shared = "https://cdn.example.com/shared.webp";
+ let other_avif = format!(
+ "data:image/avif;base64,{}",
+ STANDARD.encode(b"\x00\x00\x00\x1cftypavif\x00\x00\x00\x00avifmif1other")
+ );
+ // The dataset keeps no AVIF, so an inline one leaves the list too.
+ let inline_avif = format!(
+ "data:image/avif;base64,{}",
+ STANDARD.encode(b"\x00\x00\x00\x1cftypavif\x00\x00\x00\x00avifmif1miaf")
+ );
let picture = json!({"type": 2, "tagName": "picture", "attributes": {}, "id": 2, "childNodes": [
source(6, json!({"type": "image/svg+xml", "srcset": format!("{svg} 1x")})),
source(7, json!({"type": "image/svg+xml", "srcset": format!("{remote_svg} 1x")})),
- source(3, json!({"type": "image/avif", "srcset": format!("{avif} 1x")})),
+ source(3, json!({"type": "image/avif", "srcset": format!("{avif} 1x, {inline_avif} 2x")})),
// A bare srcset equal to the img's src below: only the srcset empties.
source(8, json!({"type": "image/avif", "srcset": bare})),
+ // Values that read the same once their inline AVIFs leave merge their policies, so the WebP source keeps
+ // the shared candidate.
+ source(11, json!({"type": "image/avif", "srcset": format!("{shared} 1x, {inline_avif} 2x")})),
+ source(12, json!({"type": "image/webp", "srcset": format!("{shared} 1x, {other_avif} 2x")})),
// An empty type gives no hint, so the source keeps its candidate.
source(9, json!({"type": "", "srcset": format!("{typeless} 1x")})),
source(4, json!({"type": "image/webp", "srcset": format!("{signed} 2x, {webp} 1x")})),
@@ -867,6 +883,10 @@ mod tests {
"{lines}"
);
assert!(lines.contains(&reference(fallback)), "{lines}");
+ assert!(
+ lines.contains(&format!(r#""srcset":"{} 1x""#, reference(shared))),
+ "{lines}"
+ );
assert!(
lines.contains(&format!(r#""src":"{}""#, reference(bare))),
"{lines}"
@@ -877,12 +897,45 @@ mod tests {
);
assert!(
!lines.contains("a.avif")
+ && !lines.contains("ftypavif")
+ && !lines.contains("data:image/avif")
&& !lines.contains("X-Amz-Signature")
&& !lines.contains("declined"),
"{lines}"
);
}
+ #[test]
+ fn rebuilds_a_srcset_as_the_block_holds_it_after_escapes_are_written_again() {
+ // The `\/` escapes keep the PNG inline, and the writer then writes them as `/`.
+ let png = STANDARD.encode(png(&[0xff; 30]));
+ assert!(png.contains('/'));
+ let inline = format!("data:image/png;base64,{png}");
+ let events = format!(
+ r#"[{{"type":3,"timestamp":1,"data":{{"source":0,"adds":[],"removes":[],"texts":[],"attributes":[{{"id":99,"attributes":{{"srcset":"{} 1x, https://10.0.0.7/a.png 2x"}}}}]}}}}]"#,
+ inline.replace('/', "\\/")
+ );
+ let lines = stored_lines(&events, &[]);
+ assert!(
+ lines.contains(&format!(r#""srcset":"{inline} 1x""#)),
+ "{lines}"
+ );
+ assert!(
+ !lines.contains("declined") && !lines.contains("10.0.0.7"),
+ "{lines}"
+ );
+
+ // A srcset key outside an image slot, here in a custom event, is no srcset of the DOM and stays as it is.
+ let custom = r#"[{"type":5,"timestamp":1,"data":{"tag":"app","payload":{"srcset":"https://cdn.example.com/a.png 1x, https://10.0.0.7/a.png 2x"}}}]"#;
+ let lines = stored_lines(custom, &[]);
+ assert!(
+ lines.contains(
+ r#""srcset":"https://cdn.example.com/a.png 1x, https://10.0.0.7/a.png 2x""#
+ ),
+ "{lines}"
+ );
+ }
+
#[test]
fn keeps_the_sealed_urls_of_a_row_under_the_message_limit() {
let page = |i: usize| format!("https://app.example.com/{i}/{}", "p".repeat(1000));