Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(ai-research-v4): keep only loadable candidates in image srcsets (#124841)

Nicholas Waltz75960fdb597b
diff --git a/products/ai_training/docs/replay-data.md b/products/ai_training/docs/replay-data.md
index 30ad40bec97d..2202acd7b70d 100644
--- a/products/ai_training/docs/replay-data.md
+++ b/products/ai_training/docs/replay-data.md
@@ -465,6 +465,7 @@ The recorder streams each block into the encoder. Read a block as a stream: a Br
- A remote image URL becomes `imageurl:v4:{team_id}:{YYYY-MM}:{hash}` in the block and in its fetch job, where the month is the UTC month of the write. The block and the job always carry the same reference, so a reader opens the fetched image directly.
- A URL that the URL policy refuses becomes `imageurl:v4:declined:{reason}` in the block, for example `declined:credential`, `declined:non_public_host` or `declined:tracking_beacon`. The original URL cannot be recovered. The recorder counts rewrites and refusals in `ai_research_v4_recorder_image_url_rewrites_total`.
- The rewrite covers every remote URL that the recorder reads from an image slot: image sources, `srcset`, SVG image links, video posters, `<picture>` sources and CSS `url(...)` in style attributes, style sheets and style rules. It also covers `src`, `srcset`, `href` and `poster` in an attribute change on a node the recorder does not know, for example after a restart. `srcset` keeps its descriptors and CSS keeps its `url(...)` syntax. It is not limited by the 512 URLs per message that get fetch jobs.
+- A srcset in an image slot keeps only the candidates the recorder stores. A candidate the URL policy refuses leaves the list instead of becoming a `declined` marker, and the rest keep their descriptors. A `<picture>` source whose `type` names a media type the image fetcher does not store, such as `image/avif`, gets no fetch job and loses its remote candidates. Its inline images, such as SVG data URIs, stay. A browser skips a source with no candidates, so the replay falls back to the next source or the `img`. The recorder counts these as `ai_research_v4_recorder_srcset_rewrites_total{outcome="trimmed"|"emptied"}`. A srcset that an inline image extraction changed falls back to the rewrite of each URL.
- The recorder reads a protocol-relative URL, such as `//cdn.example.com/a.png`, as `https`. It gets the same reference or `declined` marker as the `https` form, and the rewrite replaces the text as the block holds it. A browser reads a backslash as a slash there, so `\\cdn.example.com/a.png` counts too, and so does a URL such as `https:\\cdn.example.com/a.png` with any mix of slashes and backslashes after the scheme, or none: on a page of another scheme the browser loads `https:cdn.example.com/a.png` from that host.
- Some URLs stay as they arrived: relative URLs, remote URLs outside image slots (links, frames, text, CSS custom properties and other CSS properties), the `image-set()` candidates the recorder does not choose, URLs inside a compressed value that did not decode or that the decode budget skipped, and URL text written in another form than the recorder read it, for example with a CSS escape. The recorder counts the last kind as `outcome="unrewritten"`.
- Before the rewrite, the recorder writes again any JSON string that holds an escape capture would not write, for example `\/`, so an escaped URL is found too. A scheme matches in any letter case.
diff --git a/rust/ai-research/replay-core-v4/src/inspect/dom.rs b/rust/ai-research/replay-core-v4/src/inspect/dom.rs
index 1ae79a06beaf..2eaeaddca6c0 100644
--- a/rust/ai-research/replay-core-v4/src/inspect/dom.rs
+++ b/rust/ai-research/replay-core-v4/src/inspect/dom.rs
@@ -5,7 +5,7 @@ use std::collections::{HashMap, HashSet};
use std::ops::Range;
use crate::bytescan::{self as scan, Span};
-use crate::url_policy::{self, MAX_URL_LEN};
+use crate::url_policy::{is_fetched_image_type, is_remote, MAX_URL_LEN};
use super::css;
use super::events::{field, fields, number, text};
@@ -82,7 +82,10 @@ enum Media {
},
SvgImage,
Video,
- PictureSource,
+ /// A `source` under a `picture`. Its `type` can name a media type the fetcher does not store, such as AVIF.
+ PictureSource {
+ unfetched_type: bool,
+ },
/// A `picture`, so that a `source` added under it later is known to be a picture source.
Picture,
}
@@ -204,6 +207,26 @@ pub struct RemoteUrls {
urls: HashSet<String>,
bytes: usize,
pub limit: Option<UrlLimit>,
+ /// Each image-slot srcset value, and whether it belongs to a source of a type the fetcher does not store. The
+ /// writer rebuilds each from the candidates it keeps, and it builds one matcher over them as over `urls`, so they
+ /// share the byte budget of `urls`. Past it this keeps no more, which is safe: the writer still replaces each URL
+ /// in `urls` on its own.
+ /// Boxed, because the session state counts this struct's size and most blocks hold no srcset.
+ srcsets: Option<Box<SrcsetValues>>,
+}
+
+#[derive(Debug, Default)]
+struct SrcsetValues {
+ texts: HashMap<String, bool>,
+ bytes: usize,
+}
+
+/// An image-slot srcset value as the event holds it.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct Srcset {
+ pub text: String,
+ /// The value belongs to a `<source>` whose `type` the fetcher does not store, so none of its images is stored.
+ pub unfetched_type: bool,
}
impl RemoteUrls {
@@ -224,11 +247,40 @@ impl RemoteUrls {
};
}
+ /// A value seen on a source of a stored type and on one of another type keeps its candidates.
+ fn insert_srcset(&mut self, text: &str, unfetched_type: bool) {
+ let srcsets = self.srcsets.get_or_insert_with(Box::default);
+ if let Some(known) = srcsets.texts.get_mut(text) {
+ *known &= unfetched_type;
+ } else if srcsets.texts.len() < MAX_BLOCK_REMOTE_URLS
+ && srcsets.bytes + text.len() <= MAX_BLOCK_REMOTE_URL_BYTES
+ {
+ srcsets.bytes += text.len();
+ srcsets.texts.insert(text.to_owned(), unfetched_type);
+ }
+ }
+
pub fn extend(&mut self, other: RemoteUrls) {
self.limit = self.limit.or(other.limit);
for url in &other.urls {
self.insert(url);
}
+ for (text, unfetched_type) in other.srcsets.iter().flat_map(|s| &s.texts) {
+ self.insert_srcset(text, *unfetched_type);
+ }
+ }
+
+ pub fn take_srcsets(&mut self) -> Vec<Srcset> {
+ self.srcsets
+ .take()
+ .map(|srcsets| srcsets.texts)
+ .unwrap_or_default()
+ .into_iter()
+ .map(|(text, unfetched_type)| Srcset {
+ text,
+ unfetched_type,
+ })
+ .collect()
}
pub fn into_urls(self) -> Vec<String> {
@@ -253,22 +305,6 @@ fn fetchable(url: &str) -> Option<&str> {
(url.len() <= MAX_URL_LEN && is_remote(url)).then_some(url)
}
-/// A URL scheme is not case sensitive, so `HTTPS://` loads as `https://` does. A browser reads any run of slashes and
-/// backslashes after `http:` or `https:` as `//`, and none at all on a page of another scheme, so `https:\\host` and
-/// `https:host` can load from the host too. A protocol-relative URL does as well. The writer must replace each of them
-/// like any other remote URL.
-fn is_remote(url: &str) -> bool {
- let after_scheme = ["https:", "http:"].iter().find_map(|scheme| {
- url.get(..scheme.len())
- .filter(|start| start.eq_ignore_ascii_case(scheme))
- .map(|_| &url[scheme.len()..])
- });
- match after_scheme {
- Some(rest) => !rest.is_empty(),
- None => url_policy::is_protocol_relative(url),
- }
-}
-
impl Urls {
fn push(&mut self, url: &str) {
self.push_remote(url);
@@ -338,7 +374,7 @@ struct Walk {
hidden: Vec<(i64, HiddenSources)>,
/// `source` elements that are a walk's root, with their srcset. Whether each is a picture source depends on the
/// parent that the mutation names.
- root_sources: Vec<(Option<i64>, Option<Span>)>,
+ root_sources: Vec<(Option<i64>, Option<Span>, Option<Span>)>,
}
impl Walk {
@@ -380,6 +416,25 @@ impl Walk {
}
}
+ fn push_srcset_value(&mut self, bytes: &[u8], value: Option<Span>, unfetched_type: bool) {
+ if let Some(srcset) = text(bytes, value) {
+ self.image_urls
+ .remote
+ .insert_srcset(&srcset, unfetched_type);
+ }
+ }
+
+ /// A picture source of a type the fetcher does not store gets no fetch job, and the writer empties its srcset, so
+ /// the browser falls back to the next source or the img.
+ fn push_source(&mut self, bytes: &[u8], srcset: Option<Span>, unfetched_type: bool) {
+ if unfetched_type {
+ self.push_remote(bytes, srcset, true);
+ } else {
+ self.push_srcset(bytes, srcset);
+ }
+ self.push_srcset_value(bytes, srcset, unfetched_type);
+ }
+
fn push_css(&mut self, bytes: &[u8], value: Option<Span>) {
let Some(value) = value else { return };
if let Some(text) = text(bytes, Some(value)) {
@@ -401,9 +456,18 @@ impl Walk {
}
}
-/// The URLs of a srcset, parsed as HTML parses one: a URL is a run of non-whitespace, so it can hold a comma. A comma
-/// ends a candidate only at the end of its URL or after its descriptors.
-fn srcset_urls(srcset: &str, found: &mut impl FnMut(&str)) {
+/// The candidates of a srcset as (URL, descriptors), parsed as HTML parses one: a URL is a run of non-whitespace, so
+/// it can hold a comma. A comma ends a candidate only at the end of its URL or after its descriptors.
+pub fn srcset_candidates(srcset: &str) -> Vec<(&str, &str)> {
+ let mut candidates = Vec::new();
+ for_each_srcset_candidate(srcset, |url, descriptors| {
+ candidates.push((url, descriptors))
+ });
+ candidates
+}
+
+/// Streams the candidates of [`srcset_candidates`], so that a message-sized srcset needs no list of them.
+fn for_each_srcset_candidate<'s>(srcset: &'s str, mut found: impl FnMut(&'s str, &'s str)) {
let bytes = srcset.as_bytes();
let mut position = 0;
loop {
@@ -424,23 +488,33 @@ fn srcset_urls(srcset: &str, found: &mut impl FnMut(&str)) {
position += 1;
}
let url = srcset[start..position].trim_end_matches(',');
- found(url);
if url.len() < position - start {
+ found(url, "");
continue;
}
+ let descriptors = position;
+ let mut end = bytes.len();
let mut in_parens = false;
while let Some(&byte) = bytes.get(position) {
position += 1;
match byte {
b'(' => in_parens = true,
b')' => in_parens = false,
- b',' if !in_parens => break,
+ b',' if !in_parens => {
+ end = position - 1;
+ break;
+ }
_ => {}
}
}
+ found(url, srcset[descriptors..end].trim());
}
}
+fn srcset_urls(srcset: &str, found: &mut impl FnMut(&str)) {
+ for_each_srcset_candidate(srcset, |url, _| found(url));
+}
+
fn style_property_value(bytes: &[u8], span: Span) -> Option<Span> {
if scan::is_array(bytes, span) {
scan::array_items(bytes, span).ok()?.next()?.ok()
@@ -464,6 +538,7 @@ struct Attributes {
hidden: Option<Span>,
width: Option<Span>,
height: Option<Span>,
+ source_type: Option<Span>,
}
impl Attributes {
@@ -481,6 +556,7 @@ impl Attributes {
b"hidden" => &mut self.hidden,
b"width" => &mut self.width,
b"height" => &mut self.height,
+ b"type" => &mut self.source_type,
_ => return None,
})
}
@@ -503,6 +579,7 @@ impl Attributes {
walk.push_remote(bytes, self.src, false);
walk.push_remote(bytes, self.rr_src, false);
walk.push_remote(bytes, self.srcset, true);
+ walk.push_srcset_value(bytes, self.srcset, false);
walk.push_remote(bytes, self.href, false);
walk.push_remote(bytes, self.xlink_href, false);
walk.push_remote(bytes, self.poster, false);
@@ -515,18 +592,22 @@ impl Attributes {
walk.push_url(bytes, self.src);
walk.push_url(bytes, self.rr_src);
walk.push_srcset(bytes, self.srcset);
+ walk.push_srcset_value(bytes, self.srcset, false);
}
Some(Media::Img { .. }) => {
walk.push_remote(bytes, self.src, false);
walk.push_remote(bytes, self.rr_src, false);
walk.push_remote(bytes, self.srcset, true);
+ walk.push_srcset_value(bytes, self.srcset, false);
}
Some(Media::SvgImage) => {
walk.push_url(bytes, self.href);
walk.push_url(bytes, self.xlink_href);
}
Some(Media::Video) => walk.push_url(bytes, self.poster),
- Some(Media::PictureSource) => walk.push_srcset(bytes, self.srcset),
+ Some(Media::PictureSource { unfetched_type }) => {
+ walk.push_source(bytes, self.srcset, unfetched_type)
+ }
_ => {}
}
walk.push_css(bytes, self.style);
@@ -534,6 +615,11 @@ impl Attributes {
}
}
+/// An empty `type` gives the browser no hint, as a missing one does.
+fn is_unfetched_type(bytes: &[u8], source_type: Option<Span>) -> bool {
+ text(bytes, source_type).is_some_and(|t| !t.trim().is_empty() && !is_fetched_image_type(&t))
+}
+
/// A CSS or HTML length read as pixels, as in v3: `1`, `1px` and `1.0` are one pixel, other units are unknown.
fn px_length(text: &str) -> Option<f64> {
let text = text.trim().to_ascii_lowercase();
@@ -650,7 +736,7 @@ struct NodeFrame {
/// Child text, and child `source` elements with their srcset, which count only under a `style` or a `picture`.
/// They wait here because the tag can come after the children.
child_texts: Vec<(Option<i64>, Span)>,
- child_sources: Vec<(Option<i64>, Option<Span>)>,
+ child_sources: Vec<(Option<i64>, Option<Span>, Option<Span>)>,
}
#[derive(Clone, Copy, PartialEq, Eq)]
@@ -748,9 +834,11 @@ fn finish_node(bytes: &[u8], node: NodeFrame, parent: Option<&mut NodeFrame>, wa
}
}
if node.tag == Tag::Picture {
- for &(id, srcset) in &node.child_sources {
- walk.media.extend(id.map(|id| (id, Media::PictureSource)));
- walk.push_srcset(bytes, srcset);
+ for &(id, srcset, source_type) in &node.child_sources {
+ let unfetched_type = is_unfetched_type(bytes, source_type);
+ walk.media
+ .extend(id.map(|id| (id, Media::PictureSource { unfetched_type })));
+ walk.push_source(bytes, srcset, unfetched_type);
}
}
match node.id {
@@ -814,7 +902,11 @@ fn finish_node(bytes: &[u8], node: NodeFrame, parent: Option<&mut NodeFrame>, wa
.extend(node.text.map(|text| (node.id, text)));
}
if node.tag == Tag::Source {
- parent.child_sources.push((node.id, node.attributes.srcset));
+ parent.child_sources.push((
+ node.id,
+ node.attributes.srcset,
+ node.attributes.source_type,
+ ));
}
}
None => {
@@ -822,7 +914,11 @@ fn finish_node(bytes: &[u8], node: NodeFrame, parent: Option<&mut NodeFrame>, wa
walk.push_id(id, None);
}
if node.tag == Tag::Source {
- walk.root_sources.push((node.id, node.attributes.srcset));
+ walk.root_sources.push((
+ node.id,
+ node.attributes.srcset,
+ node.attributes.source_type,
+ ));
}
}
}
@@ -1318,11 +1414,13 @@ fn observe_mutation(
.filter(|(_, media)| *media == Media::Picture)
.map(|&(id, _)| id)
.collect();
- for (parent, (id, srcset)) in sources {
+ for (parent, (id, srcset, source_type)) in sources {
let Some(parent) = parent else { continue };
if pictures.contains(&parent) || dom.media.get(&parent) == Some(&Media::Picture) {
- walk.media.extend(id.map(|id| (id, Media::PictureSource)));
- walk.push_srcset(list.bytes, srcset);
+ let unfetched_type = is_unfetched_type(list.bytes, source_type);
+ walk.media
+ .extend(id.map(|id| (id, Media::PictureSource { unfetched_type })));
+ walk.push_source(list.bytes, srcset, unfetched_type);
}
}
}
@@ -1360,6 +1458,11 @@ fn observe_mutation(
visibility.update(attributes.bytes, &values);
hidden = Some((id, was_hidden, visibility.hides()));
}
+ if let (Media::PictureSource { unfetched_type }, Some(_)) =
+ (&mut *known, values.source_type)
+ {
+ *unfetched_type = is_unfetched_type(attributes.bytes, values.source_type);
+ }
media = Some(*known);
}
match hidden {
@@ -1915,6 +2018,10 @@ mod tests {
] {
assert_eq!(limit(&mut urls.into_iter()), expected);
}
+ // Srcset values share the byte budget, past which the writer falls back to replacing each URL.
+ let mut remote = RemoteUrls::default();
+ (0..=most_long).for_each(|i| remote.insert_srcset(&url(i, longest), false));
+ assert_eq!(remote.take_srcsets().len(), most_long);
// A mutation past the attribute cap is not read to its end.
let attributes = |count: usize| {
diff --git a/rust/ai-research/replay-core-v4/src/inspect/session.rs b/rust/ai-research/replay-core-v4/src/inspect/session.rs
index f177c7bc0a11..436326e4c446 100644
--- a/rust/ai-research/replay-core-v4/src/inspect/session.rs
+++ b/rust/ai-research/replay-core-v4/src/inspect/session.rs
@@ -3,7 +3,7 @@
use std::sync::Arc;
use super::clicks::{ClickKind, ClickTracker};
-use super::dom::{DomInspector, RemoteUrls, UrlLimit};
+use super::dom::{DomInspector, RemoteUrls, Srcset, UrlLimit};
use super::events::{fields, number, Activity, EventCounter};
use super::rrweb::{
is_click, CUSTOM, FULL_SNAPSHOT, INCREMENTAL_SNAPSHOT, META, MOUSE_INTERACTION, MUTATION,
@@ -62,15 +62,18 @@ pub struct Inspected {
pub remote_urls: Vec<String>,
/// Set when `remote_urls` stops short, so that the block must not be stored.
pub url_limit: Option<UrlLimit>,
+ /// The image-slot srcset values of the block, which the writer rebuilds from the candidates it keeps.
+ pub srcsets: Vec<Srcset>,
}
impl BlockInspection {
- pub fn finish(self) -> Inspected {
+ pub fn finish(mut self) -> Inspected {
Inspected {
stats: self.counter.finish(),
moments: self.moments,
image_urls: self.image_urls,
url_limit: self.remote_urls.limit,
+ srcsets: self.remote_urls.take_srcsets(),
remote_urls: self.remote_urls.into_urls(),
}
}
diff --git a/rust/ai-research/replay-core-v4/src/url_policy.rs b/rust/ai-research/replay-core-v4/src/url_policy.rs
index 870675ad1539..f39f95ba33fc 100644
--- a/rust/ai-research/replay-core-v4/src/url_policy.rs
+++ b/rust/ai-research/replay-core-v4/src/url_policy.rs
@@ -346,6 +346,33 @@ pub fn is_protocol_relative(raw: &str) -> bool {
matches!(raw.trim().as_bytes(), [b'/' | b'\\', b'/' | b'\\', ..])
}
+/// Whether the browser loads the URL from a host. A URL scheme is not case sensitive, so `HTTPS://` loads as
+/// `https://` does. A browser reads any run of slashes and backslashes after `http:` or `https:` as `//`, and none at
+/// all on a page of another scheme, so `https:\\host` and `https:host` can load from the host too. A
+/// protocol-relative URL does as well.
+pub fn is_remote(url: &str) -> bool {
+ let after_scheme = ["https:", "http:"].iter().find_map(|scheme| {
+ url.get(..scheme.len())
+ .filter(|start| start.eq_ignore_ascii_case(scheme))
+ .map(|_| &url[scheme.len()..])
+ });
+ match after_scheme {
+ Some(rest) => !rest.is_empty(),
+ None => is_protocol_relative(url),
+ }
+}
+
+/// The media types the image fetcher stores, the `ALLOWED_CONTENT_TYPES` of `image-fetch-v4/image-fetcher.ts`.
+pub const FETCHED_IMAGE_TYPES: [&str; 4] = ["image/png", "image/jpeg", "image/gif", "image/webp"];
+
+/// Whether a `<source type>` names a type the fetcher stores, ignoring parameters such as `codecs`.
+pub fn is_fetched_image_type(source_type: &str) -> bool {
+ let essence = source_type.split(';').next().unwrap_or_default().trim();
+ FETCHED_IMAGE_TYPES
+ .iter()
+ .any(|fetched| fetched.eq_ignore_ascii_case(essence))
+}
+
pub fn try_canonicalize(raw: &str) -> Result<CanonicalUrl, Decline> {
if raw.len() > MAX_URL_LEN {
return Err(Decline::TooLong);
diff --git a/rust/ai-research/replay-recorder-v4/src/batch.rs b/rust/ai-research/replay-recorder-v4/src/batch.rs
index 995d6c137a91..dab9ac95a210 100644
--- a/rust/ai-research/replay-recorder-v4/src/batch.rs
+++ b/rust/ai-research/replay-recorder-v4/src/batch.rs
@@ -8,7 +8,7 @@ use std::time::{Duration, Instant};
use futures::stream::{self, StreamExt};
use metrics::counter;
use replay_core_v4::compression::{GZIP_MAGIC, MAX_DECOMPRESSED_BYTES};
-use replay_core_v4::inspect::dom::UrlLimit;
+use replay_core_v4::inspect::dom::{Srcset, UrlLimit};
use replay_core_v4::inspect::session::{BlockInspection, Moment, SessionInspector};
use replay_core_v4::inspect::stats::BlockStats;
use replay_core_v4::scan::cv::{Expansion, MESSAGE_DECODE_BUDGET};
@@ -96,6 +96,8 @@ pub struct Block {
pub remote_urls: Vec<String>,
/// Set when the inspector stopped short of naming every remote URL, so that the block must not be stored.
pub url_limit: Option<UrlLimit>,
+ /// The image-slot srcset values of the block, which the writer rebuilds from the candidates it keeps.
+ pub srcsets: Vec<Srcset>,
/// The earliest Kafka timestamp of the block's messages, for the capture watermark of the index topic.
pub captured_at_ms: Option<i64>,
}
@@ -508,6 +510,7 @@ fn inspect_session(
image_urls: inspected.image_urls,
remote_urls: inspected.remote_urls,
url_limit: inspected.url_limit,
+ srcsets: inspected.srcsets,
captured_at_ms,
}
})
diff --git a/rust/ai-research/replay-recorder-v4/src/consumer.rs b/rust/ai-research/replay-recorder-v4/src/consumer.rs
index e117d139674e..2c93ed8d8736 100644
--- a/rust/ai-research/replay-recorder-v4/src/consumer.rs
+++ b/rust/ai-research/replay-recorder-v4/src/consumer.rs
@@ -841,6 +841,7 @@ mod tests {
image_urls: Vec::new(),
remote_urls: Vec::new(),
url_limit: None,
+ srcsets: Vec::new(),
captured_at_ms: None,
})
};
diff --git a/rust/ai-research/replay-recorder-v4/src/rewrite.rs b/rust/ai-research/replay-recorder-v4/src/rewrite.rs
index 70a4485d61c0..0830e9c75599 100644
--- a/rust/ai-research/replay-recorder-v4/src/rewrite.rs
+++ b/rust/ai-research/replay-recorder-v4/src/rewrite.rs
@@ -1,13 +1,21 @@
//! Replaces the remote image URLs of a stored block with their `imageurl:v4` references, as v3 replaced them. A URL the
//! policy declines becomes a marker that names the reason, so that no credential, signature or private host stays in
//! the block. The inspector names the URLs: those in image slots, CSS included.
+//!
+//! A srcset is rebuilt instead: a declined candidate leaves the list, and so does each remote candidate of a picture
+//! source whose type the fetcher does not store. An inline image, such as an SVG data URI, stays. A browser skips a
+//! source with no candidates, so it falls back to a source or an img that can load, where a marker in the list would
+//! show a broken image.
use std::borrow::Cow;
+use std::collections::HashMap;
use aho_corasick::{AhoCorasick, AhoCorasickKind, MatchKind};
use metrics::counter;
use replay_core_v4::bytescan::{unescape, write_json_string};
+use replay_core_v4::inspect::dom::{srcset_candidates, Srcset};
use replay_core_v4::seal::image_url_key;
+use replay_core_v4::url_policy::is_remote;
use crate::urls::url_reference;
@@ -104,11 +112,16 @@ fn ends_with_escape(bytes: &[u8], at: usize) -> bool {
short || unicode
}
+const SRCSET_KEY: &[u8] = br#""srcset":"#;
+
struct Target {
replacement: Vec<u8>,
/// The policy's reason, for a declined URL.
declined: Option<&'static str>,
found: bool,
+ /// For a srcset, which is a whole JSON string with its quotes: how it was rebuilt, and the URL targets its value
+ /// holds. Those URLs leave the block with the old value, so a match counts as theirs.
+ srcset: Option<(&'static str, Vec<usize>)>,
}
pub struct UrlRewrite {
@@ -123,10 +136,16 @@ pub struct UrlRewrite {
impl UrlRewrite {
/// `team_key` is the team's month key for `key_month`, the write month, as for the fetch jobs.
- pub fn new(urls: &[String], team_id: i64, key_month: &str, team_key: &[u8; 32]) -> Self {
+ pub fn new(
+ urls: &[String],
+ srcsets: &[Srcset],
+ team_id: i64,
+ key_month: &str,
+ team_key: &[u8; 32],
+ ) -> Self {
let url_key = image_url_key(team_key);
- let mut patterns = Vec::with_capacity(urls.len());
- let mut targets = Vec::with_capacity(urls.len());
+ let mut patterns = Vec::with_capacity(urls.len() + srcsets.len());
+ let mut targets = Vec::with_capacity(urls.len() + srcsets.len());
for url in urls {
// The block holds the URL as a JSON string holds it.
let mut escaped = Vec::with_capacity(url.len() + 2);
@@ -140,6 +159,31 @@ impl UrlRewrite {
replacement: replacement.into_bytes(),
declined,
found: false,
+ srcset: None,
+ });
+ }
+ let url_targets: HashMap<&str, usize> = urls
+ .iter()
+ .enumerate()
+ .map(|(index, url)| (url.as_str(), index))
+ .collect();
+ for srcset in srcsets {
+ let Some((value, outcome, covers)) =
+ rebuilt_srcset(srcset, &url_targets, team_id, key_month, &url_key)
+ else {
+ continue;
+ };
+ // The key keeps the pattern to srcset attributes, so a `src` or `href` with the same value stays.
+ let mut pattern = SRCSET_KEY.to_vec();
+ write_json_string(&srcset.text, &mut pattern);
+ patterns.push(pattern);
+ let mut replacement = SRCSET_KEY.to_vec();
+ write_json_string(&value, &mut replacement);
+ targets.push(Target {
+ replacement,
+ declined: None,
+ found: false,
+ srcset: Some((outcome, covers)),
});
}
let matcher = (!patterns.is_empty())
@@ -183,18 +227,27 @@ impl UrlRewrite {
let mut copied_to = 0;
for found in matcher.find_iter(bytes) {
let (start, end) = (found.start(), found.end());
- if start.checked_sub(1).is_some_and(|before| {
- continues_url(bytes[before]) && !ends_with_escape(bytes, start)
- }) || bytes.get(end).is_some_and(|&after| continues_url(after))
+ let index = found.pattern().as_usize();
+ // A srcset pattern holds its quotes, so it matches only a whole JSON string.
+ let whole_string = self.targets[index].srcset.is_some();
+ if !whole_string
+ && (start.checked_sub(1).is_some_and(|before| {
+ continues_url(bytes[before]) && !ends_with_escape(bytes, start)
+ }) || bytes.get(end).is_some_and(|&after| continues_url(after)))
{
continue;
}
- let target = &mut self.targets[found.pattern().as_usize()];
+ let target = &mut self.targets[index];
target.found = true;
let out = out.get_or_insert_with(|| Vec::with_capacity(bytes.len()));
out.extend_from_slice(&bytes[copied_to..start]);
out.extend_from_slice(&target.replacement);
copied_to = end;
+ if let Some((_, covers)) = self.targets[index].srcset.clone() {
+ for url in covers {
+ self.targets[url].found = true;
+ }
+ }
}
let out = match out {
Some(mut out) => {
@@ -215,6 +268,14 @@ impl UrlRewrite {
pub fn finish(self) -> bool {
let mut safe = !self.declined_kept;
for target in self.targets {
+ if let Some((outcome, _)) = target.srcset {
+ counter!(
+ "ai_research_v4_recorder_srcset_rewrites_total",
+ "outcome" => if target.found { outcome } else { "unrewritten" },
+ )
+ .increment(1);
+ continue;
+ }
safe &= target.found || target.declined.is_none();
let outcome = match (target.found, target.declined) {
(false, _) => "unrewritten",
@@ -232,6 +293,50 @@ impl UrlRewrite {
}
}
+/// The srcset with only the candidates the block keeps, each remote URL as its reference, how it was rebuilt, and the
+/// URL targets of the remote URLs it held. `None` when every candidate stays, because the URL targets then replace it.
+fn rebuilt_srcset(
+ srcset: &Srcset,
+ url_targets: &HashMap<&str, usize>,
+ team_id: i64,
+ key_month: &str,
+ url_key: &[u8; 32],
+) -> Option<(String, &'static str, Vec<usize>)> {
+ let mut kept = Vec::new();
+ let mut covers = Vec::new();
+ let mut removed = false;
+ for (url, descriptors) in srcset_candidates(&srcset.text) {
+ if !is_remote(url) {
+ kept.push((Cow::Borrowed(url), descriptors));
+ continue;
+ }
+ covers.extend(url_targets.get(url));
+ match url_reference(url, team_id, key_month, url_key) {
+ Ok((_, reference)) if !srcset.unfetched_type => {
+ kept.push((Cow::Owned(reference), descriptors))
+ }
+ _ => removed = true,
+ }
+ }
+ if !removed {
+ return None;
+ }
+ let outcome = if kept.is_empty() {
+ "emptied"
+ } else {
+ "trimmed"
+ };
+ let value = kept
+ .iter()
+ .map(|(url, descriptors)| match *descriptors {
+ "" => url.to_string(),
+ descriptors => format!("{url} {descriptors}"),
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+ Some((value, outcome, covers))
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -240,7 +345,7 @@ mod tests {
fn rewrite(urls: &[&str], bytes: &str) -> String {
let urls: Vec<String> = urls.iter().map(|u| (*u).to_owned()).collect();
- let mut rewrite = UrlRewrite::new(&urls, 7, "2026-10", &TEAM_KEY);
+ let mut rewrite = UrlRewrite::new(&urls, &[], 7, "2026-10", &TEAM_KEY);
String::from_utf8(rewrite.apply(bytes.as_bytes()).into_owned()).unwrap()
}
@@ -274,6 +379,7 @@ mod tests {
let event = format!(r#"{{"srcset":"{a} 1x,\n{b} 2x,\t{declined} 3x"}}"#);
let mut rewrite = UrlRewrite::new(
&[a.to_owned(), b.to_owned(), declined.to_owned()],
+ &[],
7,
"2026-10",
&TEAM_KEY,
@@ -293,7 +399,7 @@ mod tests {
let url = "https://cdn.example.com/a.png?token=secret";
let hidden = r#"{"src":"https:\/\/cdn.example.com\/a.png?token=secr\u0065t","alt":"a\nb"}"#;
let canonical = canonical_strings(hidden.as_bytes());
- let mut rewrite = UrlRewrite::new(&[url.to_owned()], 7, "2026-10", &TEAM_KEY);
+ let mut rewrite = UrlRewrite::new(&[url.to_owned()], &[], 7, "2026-10", &TEAM_KEY);
assert_eq!(
String::from_utf8(rewrite.apply(&canonical).into_owned()).unwrap(),
format!(
@@ -309,7 +415,7 @@ mod tests {
Cow::Borrowed(_)
));
// A declined URL the block still holds fails the block.
- let rewrite = UrlRewrite::new(&[url.to_owned()], 7, "2026-10", &TEAM_KEY);
+ let rewrite = UrlRewrite::new(&[url.to_owned()], &[], 7, "2026-10", &TEAM_KEY);
assert!(!rewrite.finish());
}
@@ -336,7 +442,7 @@ mod tests {
}
// A second copy inside a longer URL stays as it is, so the block fails.
let url = "https://bucket.s3.amazonaws.com/a.png?X-Amz-Signature=abc123";
- let mut rewrite = UrlRewrite::new(&[url.to_owned()], 7, "2026-10", &TEAM_KEY);
+ let mut rewrite = UrlRewrite::new(&[url.to_owned()], &[], 7, "2026-10", &TEAM_KEY);
rewrite.apply(format!(r#"{{"src":"{url}","alt":"x{url}"}}"#).as_bytes());
assert!(!rewrite.finish());
}
diff --git a/rust/ai-research/replay-recorder-v4/src/urls.rs b/rust/ai-research/replay-recorder-v4/src/urls.rs
index 99143d4bd782..d014a5a7ba10 100644
--- a/rust/ai-research/replay-recorder-v4/src/urls.rs
+++ b/rust/ai-research/replay-recorder-v4/src/urls.rs
@@ -207,6 +207,7 @@ mod tests {
image_urls: urls.iter().map(|u| (*u).to_owned()).collect(),
remote_urls: Vec::new(),
url_limit: None,
+ srcsets: Vec::new(),
captured_at_ms: None,
};
let mut stats = BlockStats::default();
diff --git a/rust/ai-research/replay-recorder-v4/src/write.rs b/rust/ai-research/replay-recorder-v4/src/write.rs
index b8c7b1a6319f..451a16ce382b 100644
--- a/rust/ai-research/replay-recorder-v4/src/write.rs
+++ b/rust/ai-research/replay-recorder-v4/src/write.rs
@@ -127,6 +127,7 @@ pub fn prepare_block(
let content_key = image_content_key(team_key);
let mut rewrite = UrlRewrite::new(
&block.remote_urls,
+ &block.srcsets,
block.session.team_id,
key_month,
team_key,
@@ -683,6 +684,7 @@ mod tests {
image_urls: Vec::new(),
remote_urls: Vec::new(),
url_limit: None,
+ srcsets: Vec::new(),
captured_at_ms: None,
}
}
@@ -702,6 +704,7 @@ mod tests {
let found = inspection.finish();
inspected.remote_urls = found.remote_urls;
inspected.url_limit = found.url_limit;
+ inspected.srcsets = found.srcsets;
inspected
.remote_urls
.extend(extra.iter().map(|url| (*url).to_owned()));
@@ -793,6 +796,82 @@ mod tests {
assert!(!stored_lines(&event, &[public]).is_empty());
}
+ #[test]
+ fn leaves_a_picture_only_the_sources_that_can_load() {
+ let (avif, webp, signed, fallback) = (
+ "https://cdn.example.com/a.avif",
+ "https://cdn.example.com/a.webp",
+ "https://bucket.s3.amazonaws.com/a.webp?X-Amz-Signature=abc123",
+ "https://cdn.example.com/a.png",
+ );
+ let source = |id: i64, attributes: serde_json::Value| json!({"type": 2, "tagName": "source", "attributes": attributes, "childNodes": [], "id": id});
+ // The scan leaves an SVG inline, so the browser can load it from the block.
+ let (bare, typeless) = (
+ "https://cdn.example.com/b.avif",
+ "https://cdn.example.com/c.webp",
+ );
+ let svg =
+ "data:image/svg+xml,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%2F%3E";
+ let picture = json!({"type": 2, "tagName": "picture", "attributes": {}, "id": 2, "childNodes": [
+ source(6, json!({"type": "image/svg+xml", "srcset": format!("{svg} 1x")})),
+ source(3, json!({"type": "image/avif", "srcset": format!("{avif} 1x")})),
+ // A bare srcset equal to the img's src below: only the srcset empties.
+ source(8, json!({"type": "image/avif", "srcset": bare})),
+ // An empty type gives no hint, so the source keeps its candidate.
+ source(9, json!({"type": "", "srcset": format!("{typeless} 1x")})),
+ source(4, json!({"type": "image/webp", "srcset": format!("{signed} 2x, {webp} 1x")})),
+ {"type": 2, "tagName": "img", "attributes": {"src": fallback}, "childNodes": [], "id": 5},
+ ]});
+ let img = json!({"type": 2, "tagName": "img", "attributes": {"src": bare}, "childNodes": [], "id": 10});
+ let events = json!([{"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": [picture, img]}}}]).to_string();
+
+ let block = block_of("kept", &events);
+ let mut inspector = replay_core_v4::inspect::session::SessionInspector::default();
+ let mut inspection = replay_core_v4::inspect::session::BlockInspection::default();
+ for message in &block.messages {
+ inspector.observe_message(&mut inspection, message);
+ }
+ // The fetcher stores no AVIF, so the AVIF source gets no fetch job.
+ assert!(!inspection.finish().image_urls.iter().any(|u| u == avif));
+
+ let lines = stored_lines(&events, &[]);
+ let reference = |url: &str| {
+ crate::urls::url_reference(
+ url,
+ 7,
+ "2026-10",
+ &replay_core_v4::seal::image_url_key(&[6; 32]),
+ )
+ .unwrap()
+ .1
+ };
+ // The AVIF source has no candidate left and the declined one leaves the list, so the browser takes the WebP.
+ assert!(lines.contains(r#""srcset":"""#), "{lines}");
+ assert!(
+ lines.contains(&format!(r#""srcset":"{svg} 1x""#)),
+ "{lines}"
+ );
+ assert!(
+ lines.contains(&format!(r#""srcset":"{} 1x""#, reference(webp))),
+ "{lines}"
+ );
+ assert!(lines.contains(&reference(fallback)), "{lines}");
+ assert!(
+ lines.contains(&format!(r#""src":"{}""#, reference(bare))),
+ "{lines}"
+ );
+ assert!(
+ lines.contains(&format!(r#""srcset":"{} 1x""#, reference(typeless))),
+ "{lines}"
+ );
+ assert!(
+ !lines.contains("a.avif")
+ && !lines.contains("X-Amz-Signature")
+ && !lines.contains("declined"),
+ "{lines}"
+ );
+ }
+
#[test]
fn keeps_the_sealed_urls_of_a_row_under_the_message_limit() {
let page = |i: usize| format!("https://app.example.com/{i}/{}", "p".repeat(1000));