:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
refactor(ai-research): write each v4 slot in place in one pass (#128227)
Robbie4c641f751f0f
diff --git a/products/ai_training/docs/replay-data.md b/products/ai_training/docs/replay-data.mdindex 08ad44f576e9..2fac8b1ea1b7 100644--- a/products/ai_training/docs/replay-data.md+++ b/products/ai_training/docs/replay-data.md@@ -485,19 +485,19 @@ The recorder streams each block into the encoder. Read a block as a stream: a Br- When an attribute change shows a hidden `img`, the recorder decides this on the srcset and the `src` after the change.- The image scan can still decline an inline kept candidate when its message is past the caps. The srcset is then empty, and the `img` shows no image.- An inline image in a candidate that leaves still moves to the image pack, where no reference names it.-- A `<picture>` source whose `type` names a media type the image fetcher does not store, such as `image/avif`, gets no fetch job and loses its remote candidates. Its kept candidate stays only when it is inline, such as an SVG data URI. Srcsets with the same remote URLs in the same order share one decision, so such a source keeps the candidate of a source of a stored type with the same remote URLs, and an `img` that shows its `src` keeps the candidate of an `img` that shows the candidate. That `img` still shows its `src`. The writer rebuilds each srcset that the inspector read in an image slot, in the bytes it stores, after the inline images left and the escapes were written again. A `srcset` key elsewhere, for example in a custom event's payload, stays as it is. A browser skips a source with no candidates, so the replay falls back to the next source or the `img`. The recorder counts these as `ai_research_v4_recorder_srcset_rewrites_total{outcome="trimmed"|"emptied"}`.-- The writer rewrites CSS text before it replaces the URLs, so that an offline render loads only images the block refers to. An `image-set()` keeps one candidate, because a renderer picks the candidate that fits its pixel ratio and does not fall back when that candidate fails. It keeps the highest resolution of a type the image fetcher stores among the candidates that can load: inline, accepted by the URL policy, or root-relative on a known page. Only that candidate gets a fetch job, and the other remote candidates get references without one, in case the writer does not find the set's text. A set that holds an inline image keeps all its candidates, because the writer moves the inline image out first.-- A root-relative CSS `url(/…)` resolves against the origin of its window's latest page URL, from a Meta event or a custom event that names a page, such as `$url_changed`, with `https` in place of the page's scheme. It gets the reference of the absolute URL, or a `declined` marker. A path-relative `url(…)` stays as the page wrote it, so that no part of a page path, which can hold a token, reaches a fetch. rrweb writes snapshot CSS with absolute URLs, but sends rule text, adopted style sheets and style declarations as the page wrote them. A text that resolves two ways in one block, as in windows on two origins, or that one window resolves and another cannot, keeps no rewrite. An event resolves at most 1,024 relative URLs, a flush keeps at most 64 MiB of resolved URLs, and a page URL longer than 2,048 characters resolves none.+- A `<picture>` source whose `type` names a media type the image fetcher does not store, such as `image/avif`, gets no fetch job and loses its remote candidates. Its kept candidate stays only when it is inline, such as an SVG data URI. The writer rebuilds each srcset that the inspector read in an image slot, where the inspector read it. A `srcset` key elsewhere, for example in a custom event's payload, stays as it is. A browser skips a source with no candidates, so the replay falls back to the next source or the `img`. The recorder counts these as `ai_research_v4_recorder_srcset_rewrites_total{outcome="trimmed"|"emptied"}`.+- The writer rewrites each CSS text where the inspector read it, so that an offline render loads only images the block refers to. An `image-set()` keeps one candidate, because a renderer picks the candidate that fits its pixel ratio and does not fall back when that candidate fails. It keeps the highest resolution of a type the image fetcher stores among the candidates that can load: inline, accepted by the URL policy, or root-relative on a known page. Only that candidate gets a fetch job, and the other candidates leave the set. A set that holds an inline image keeps all its candidates, and each remote one gets its reference.+- A root-relative CSS `url(/…)` resolves against the origin of its window's latest page URL, from a Meta event or a custom event that names a page, such as `$url_changed`, with `https` in place of the page's scheme. It gets the reference of the absolute URL, or a `declined` marker. A path-relative `url(…)` stays as the page wrote it, so that no part of a page path, which can hold a token, reaches a fetch. rrweb writes snapshot CSS with absolute URLs, but sends rule text, adopted style sheets and style declarations as the page wrote them. Each window resolves its own texts, so the same rule in windows on two origins gets two references. An event resolves at most 1,024 relative URLs, a flush keeps at most 64 MiB of resolved URLs, and a page URL longer than 2,048 characters resolves none.-- The recorder reads CSS with `cssparser`, which tokenizes as CSS Syntax 3 does, so names, escapes, comments, strings and bad URLs read as a browser reads them. So `background-\69mage`, `\75 rl(…)` and `background/**/:url("…"/**/)` count, and so does a `url(` that the end of the text closes. A bad URL such as `url(a(1).png)`, which no browser loads, does not count. An image URL that itself holds an escape stays as the page wrote it, as v3 left it.-- The writer rewrites only whole `url()` and `image-set()` functions, found as the inspector finds them, inside JSON strings that hold CSS: the values of `_cssText`, `style` (style diffs included), `textContent`, `rule`, `replace`, `replaceSync` and `value`. So a class name such as `bg-[url(/a.png)]`, a quoted string or a comment keeps its text, and the class's rule still matches. A text node that shows the same function text gets the rewrite too. The recorder counts these as `ai_research_v4_recorder_css_rewrites_total{kind="image_set"|"relative_url", outcome="rewritten"|"unfound"}`.+- The recorder reads CSS with `cssparser`, which tokenizes as CSS Syntax 3 does, so names, escapes, comments, strings and bad URLs read as a browser reads them. So `background-\69mage`, `\75 rl(…)` and `background/**/:url("…"/**/)` count, and so does a `url(` that the end of the text closes. A bad URL such as `url(a(1).png)`, which no browser loads, does not count. An image URL that itself holds an escape gets the reference of the URL it decodes to, as a font URL does, and its whole `url()` is rewritten. A block whose `image-set()` holds such a candidate is dropped, because the set keeps its candidate as the page wrote it.+- The writer rewrites only whole `url()` and `image-set()` functions, found as the inspector finds them, inside JSON strings that hold CSS: the values of `_cssText`, `style` (style diffs included), `textContent`, `rule`, `replace`, `replaceSync` and `value`. So a class name such as `bg-[url(/a.png)]`, a quoted string or a comment keeps its text, and the class's rule still matches. The recorder counts these as `ai_research_v4_recorder_css_rewrites_total{kind="image_set"|"relative_url", outcome="rewritten"}`.- The recorder reads a protocol-relative URL, such as `//cdn.example.com/a.png`, as `https`. It gets the same reference or `declined` marker as the `https` form, and the rewrite replaces the text as the block holds it. A browser reads a backslash as a slash there, so `\\cdn.example.com/a.png` counts too, and so does a URL such as `https:\\cdn.example.com/a.png` with any mix of slashes and backslashes after the scheme, or none: on a page of another scheme the browser loads `https:cdn.example.com/a.png` from that host.-- Some URLs stay as they arrived: relative URLs outside CSS, path-relative CSS URLs, root-relative CSS URLs that resolve two ways in a block, go past a budget, or come before this process saw their window's page URL (after a restart or an eviction), the candidates of an `image-set()` that does not parse, remote URLs outside image slots (links, frames, text and CSS properties that do not show an image), URLs inside a compressed value that did not decode or that the decode budget skipped, and URL text written in another form than the recorder read it, for example with a CSS hex escape. The recorder counts the last kind as `outcome="unrewritten"`.-- Before the rewrite, the recorder writes again any JSON string that holds an escape capture would not write, for example `\/`, so an escaped URL is found too. A scheme matches in any letter case.-- If a refused URL is still in a block after the rewrite, the recorder writes the block again instead of dropping it. The second pass replaces the refused URL wherever its bytes are, also inside a longer URL. For a refused URL the stored bytes do not hold as the recorder read it, for example behind a CSS escape, it replaces every URL token that holds the URL's host with `imageurl:v4:declined:unchecked`, or with `fonturl:v4:declined:unchecked` in a `url()` of an `@font-face` `src`. The recorder counts this as `url_fallbacks_total{reason="declined_kept"|"declined_unfound"}`. It drops the block, as `dropped_blocks_total{outcome="declined_unrewritten"}`, only when the second pass still leaves a refused URL, so no refused URL is stored.+- Some URLs stay as they arrived: relative URLs outside CSS, path-relative CSS URLs, root-relative CSS URLs that go past a budget or come before this process saw their window's page URL (after a restart or an eviction), the candidates of an `image-set()` that does not parse, and remote URLs outside image slots: links, frames, text, CSS properties that do not show an image, and a copy of a slot's accepted URL elsewhere in the block.+- The writer writes again any JSON string that holds an escape capture would not write, for example `\/`. A scheme matches in any letter case.+- A refused URL leaves the block wherever it is. The writer writes its `declined` marker in its slot, and overwrites each copy elsewhere in the block, also inside a longer URL, as a JSON string holds the URL and as a CSS selector escapes it. It does this before it moves inline images out, because that changes a copy that holds one. A selector copy takes the marker escaped, so the selector still matches. Only a block that holds a refused URL pays for that search. The recorder counts the copies as `ai_research_v4_recorder_overwritten_declined_copies_total`.- The recorder reads object keys as the player does: it writes an escaped key, such as `"\u0073rc"`, in its plain form before it inspects the event, and it takes the last copy of a repeated key.-- A block that the recorder cannot check in full is kept. Past 10,000 image-slot URLs (`too_many_urls`), 2 MiB of them (`too_many_url_bytes`) or 200,000 attribute changes in a mutation (`too_many_attribute_items`), the recorder names the URLs it read, and replaces every other remote URL token in the block, links and text included, with `imageurl:v4:declined:unchecked`. A `url()` in the `src` of an `@font-face` rule becomes `fonturl:v4:declined:unchecked` instead, so the renderer reads it as a font it does not have. A replaced token that starts after `(`, `'` or `"` runs to the `)`, `'` or `"` that ends it, a token that starts its JSON string runs to the end of the string, because a space does not end an attribute's URL, and any other token runs to whitespace, so the scrub never splits an escape. It counts these as `url_fallbacks_total{reason}` and the replaced tokens as `scrubbed_urls_total`. A URL of any length is named; one longer than the fetcher takes becomes `imageurl:v4:declined:too_long`. Only a block whose sealing fails is dropped.-- CSS nesting is the exception. A block whose CSS nests rules or functions 75 deep is dropped, as `dropped_blocks_total{outcome="too_many_nested_blocks"}`. The recorder cannot read CSS past that depth, and the scrub replaces only the URL tokens that it finds, so a scrub could keep a URL that nobody checked.+- The inspector records where each image slot, srcset and CSS string is, and the writer copies each event once and writes each reference or marker there. So a block has no limit on its URLs, and the writer finds no URL by its text. A URL of any length is named; one longer than the fetcher takes becomes `imageurl:v4:declined:too_long`.+- A block that the recorder cannot read in full is dropped, because a URL in the part it did not read went unchecked: CSS that nests rules or functions 75 deep (`too_many_nested_blocks`), a mutation with more than 200,000 attribute changes (`too_many_attribute_items`), an event whose compressed strings did not decode or that a decode budget skipped (`undecoded_events`), more than 1 MiB of refused URLs, whose copies the writer searches for (`too_many_declined_urls`), more than 1,000,000 matches of those refused URLs (`too_many_declined_copies`), a block whose edits do not fit in the flush's 256 MiB edit budget (`too_many_edits`), and a slot that the writer could not find again (`unplaced_slot`). The recorder counts these as `dropped_blocks_total{outcome}`.- When the text of a rewritten URL also appears outside an image slot in the same block, it is replaced there too.- `hash` is the first 22 characters of the base64url HMAC-SHA256, as in v3.@@ -802,7 +802,6 @@ The recorder reads the `src` URLs of `@font-face` rules in `<style>` text, `_css| Any other family, for a public font file | `fonturl:v4:public:{hash}` | Yes || Any other family | `fonturl:v4:{team_id}:{YYYY-MM}:{hash}` | Yes || Any other family, for a URL that the URL policy refuses | `fonturl:v4:declined:{reason}` | None |-| Any other family, for a URL the recorder did not check | `fonturl:v4:declined:unchecked` | None |- The renderer ships every family in `bundled_fonts.txt`. A reader replaces `fonturl:v4:bundled:{slug}` with its own copy of that family, and picks the face from the rule's `font-weight` and `font-style` descriptors. The slug is the family name in lowercase, with `-` for each space.- The list holds only families whose licence lets the renderer carry them. The renderer ships the full font of each family, which holds every glyph of a subset that a page asks for, such as a Google Fonts `text=` or `icon_names=` subset.@@ -813,8 +812,7 @@ The recorder reads the `src` URLs of `@font-face` rules in `<style>` text, `_css- An Embedded OpenType or SVG font file gets a reference but no fetch job, because Chromium loads neither format.- A root-relative font URL, such as `src:url(/fonts/a.woff2)`, resolves as a root-relative image URL does, and gets the font reference of the absolute URL. A path-relative font URL stays as the page wrote it.- The recorder reads CSS escapes as a browser does, in the `@font-face` keyword, the `src` and `font-family` descriptors, the `url()` name and the URL itself. So `@\66 ont-face`, `\73 rc` and `url(https://cdn.example.com/a\2e woff2)` count, and an escaped protocol-relative URL reads as `https`. A quoted URL drops a backslash before a newline, as a line continuation. Comments around a descriptor name, as in `src/**/:`, do not hide it. A font URL that holds an escape gets a rewrite of its whole `url()` to its font reference, as a root-relative font URL does.-- When one URL is both an image and a font, the block holds its image reference.-- Font URLs have their own budget per block: 4,096 URLs and 256 KiB. Font URLs never count toward the image URL limits above. Past the font budget, the recorder still names every other URL it reads. It replaces each remote URL in an `@font-face` `src` that it did not name with `fonturl:v4:declined:unchecked`, and leaves links and text as they are. It does the same when a block holds more font URLs with an escape than its CSS rewrites can hold, 10,000 of them or 2 MiB. It counts these as `url_fallbacks_total{reason="too_many_font_urls"|"too_many_font_url_bytes"|"too_many_font_rewrites"}`.+- A URL that is both an image and a font gets the reference of each slot's kind.The fetcher reads the asset from the reference prefix. For a font it sends `Accept: font/woff2, font/woff, font/ttf, font/otf, */*;q=0.1`. It takes a response whose media type is a `font/*` type, a legacy font type such as `application/x-font-woff`, or `application/octet-stream`, because many servers label fonts with a generic type. The decoded bytes must start with a WOFF2, WOFF, OpenType, TrueType or font collection signature. It stores the font as fetched under `fonts/url/`, sealed under the team month key with `kind: "font-url"`.diff --git a/rust/Cargo.lock b/rust/Cargo.lockindex 7abb87bef810..440405416836 100644--- a/rust/Cargo.lock+++ b/rust/Cargo.lock@@ -11162,7 +11162,6 @@ dependencies = ["tokio","tracing","tracing-subscriber",- "url",][[package]]diff --git a/rust/ai-research/replay-core-v4/src/inspect/css.rs b/rust/ai-research/replay-core-v4/src/inspect/css.rsindex c52eea5f30b4..981439e82cfb 100644--- a/rust/ai-research/replay-core-v4/src/inspect/css.rs+++ b/rust/ai-research/replay-core-v4/src/inspect/css.rs@@ -2,6 +2,7 @@//! Font URLs are the `url()`s in the `src` descriptor of an `@font-face` rule. `cssparser` tokenizes as CSS Syntax 3 does, so a name, an escape, a comment or a bad URL reads here as the renderer reads it.use std::borrow::Cow;+use std::ops::Range;use std::sync::atomic::{AtomicUsize, Ordering};use cssparser::{@@ -32,7 +33,11 @@ pub enum CssUrl<'a> {Resolved(&'a str),/// A root-relative `url()` that did not resolve. Its text keeps no rewrite, which another window would apply here.Unresolved(&'a str),- Rewrite(CssRewrite),+ /// `function` is a slice of the scanned text, so that the writer replaces the function where the scan read it.+ Rewrite {+ rewrite: CssRewrite,+ function: &'a str,+ },/// A `src` URL of an `@font-face` rule, with the raw value of that rule's `font-family` descriptor.Font {url: &'a str,@@ -42,6 +47,8 @@ pub enum CssUrl<'a> {ResolvedFont(&'a str),/// Rules or functions nest to [`NESTED_BLOCK_LIMIT`], where `cssparser` enters no block, so a URL there goes unnamed.NestedTooDeep,+ /// A URL that the scan names but cannot find in the text it read, so the writer could not replace it there.+ Unplaced,}#[derive(Debug, Clone, PartialEq, Eq)]@@ -51,6 +58,10 @@ pub struct CssRewrite {pub dropped: Vec<String>,/// The URL that `replacement` holds in place of a relative one, for the writer to replace with the reference of its kind.pub resolved: Option<RemoteUrl>,+ /// Where `replacement` holds its one URL, so that the edit finds the URL without a search.+ pub url_at: Option<Range<usize>>,+ /// The URL as the page wrote it and as its escapes decode, where `resolved` spells it otherwise, to find their copies.+ pub spellings: Vec<String>,}/// The origin that a root-relative URL on `page_url` resolves against, as `https` because the fetcher loads nothing else.@@ -73,6 +84,7 @@ pub struct Resolver<'p> {base: Option<&'p Url>,remaining: usize,budget: Option<&'p AtomicUsize>,+ edit_budget: Option<&'p AtomicUsize>,}impl<'p> Resolver<'p> {@@ -81,9 +93,21 @@ impl<'p> Resolver<'p> {base,remaining: MAX_RESOLVED_URLS,budget,+ edit_budget: None,+ }+ }++ pub fn with_edit_budget(self, edit_budget: Option<&'p AtomicUsize>) -> Self {+ Self {+ edit_budget,+ ..self}}+ pub fn edit_budget(&self) -> Option<&'p AtomicUsize> {+ self.edit_budget+ }+/// Only a root-relative URL, as `/a.png`, resolves. It needs the page's origin and nothing else, so no part of a page/// path, which can hold a token, reaches a fetch, and a `<base>` or a style sheet's own URL cannot change it.fn resolves(&self, source: &str) -> bool {@@ -111,55 +135,6 @@ impl<'p> Resolver<'p> {}}-/// `css` with each `url()` and `image-set()` that `replacement` names replaced, found as the inspector finds them, so-/// that a quoted string, a comment or an escaped selector never matches. `None` when nothing changed.-pub fn rewrite_functions(- css: &str,- replacement: &mut impl FnMut(&str) -> Option<String>,-) -> Option<String> {- let mut out: Option<String> = None;- let mut copied_to = 0;- each_function(&mut Parser::new(css), 0, &mut |input, start, opened| {- if !matches!(opened, Opened::UnquotedUrl(_)) {- skip_block(input);- }- let start = start.byte_index();- let end = input.position().byte_index();- if let Some(replaced) = replacement(&css[start..end]) {- let out = out.get_or_insert_with(|| String::with_capacity(css.len()));- out.push_str(&css[copied_to..start]);- out.push_str(&replaced);- copied_to = end;- }- });- let mut out = out?;- out.push_str(&css[copied_to..]);- Some(out)-}--pub fn rewrite_font_src_functions(- css: &str,- replacement: &mut impl FnMut(&str) -> Option<String>,-) -> Option<String> {- let mut out: Option<String> = None;- let mut copied_to = 0;- for declaration in declarations(css).found {- if !matches!(declaration.kind, DeclarationKind::FontSrc(_)) {- continue;- }- let value = &css[declaration.start..declaration.end];- if let Some(rewritten) = rewrite_functions(value, replacement) {- let out = out.get_or_insert_with(|| String::with_capacity(css.len()));- out.push_str(&css[copied_to..declaration.start]);- out.push_str(&rewritten);- copied_to = declaration.end;- }- }- let mut out = out?;- out.push_str(&css[copied_to..]);- Some(out)-}-/// Calls `found` with each URL and rewrite in `css`: a `style` attribute, `<style>` text or `_cssText`.pub fn urls(css: &str, resolver: &mut Resolver<'_>, found: &mut impl FnMut(CssUrl<'_>)) {let Declarations {@@ -182,36 +157,35 @@ pub fn urls(css: &str, resolver: &mut Resolver<'_>, found: &mut impl FnMut(CssUrfor declaration in &declarations {let value = &css[declaration.start..declaration.end];match declaration.kind {- DeclarationKind::Image => value_urls(value, resolver, EscapedSources::Skipped, found),+ DeclarationKind::Image => value_urls(value, resolver, found),DeclarationKind::FontSrc(face) => {let family = families.get(face).copied().flatten();let resolved_kind = match family.and_then(bundled_font) {Some(slug) => RemoteKind::BundledFont(slug),None => RemoteKind::Font,};- value_urls(- value,- resolver,- EscapedSources::Decoded,- &mut |url| match url {- CssUrl::Image(url) => found(CssUrl::Font { url, family }),- CssUrl::Resolved(url) if resolved_kind == RemoteKind::Font => {- found(CssUrl::ResolvedFont(url))- }- CssUrl::Rewrite(mut rewrite) => {- if let Some(resolved) = &mut rewrite.resolved {- resolved.kind = resolved_kind;- }- found(CssUrl::Rewrite(rewrite));+ value_urls(value, resolver, &mut |url| match url {+ CssUrl::Image(url) => found(CssUrl::Font { url, family }),+ CssUrl::Resolved(url) if resolved_kind == RemoteKind::Font => {+ found(CssUrl::ResolvedFont(url))+ }+ CssUrl::Rewrite {+ mut rewrite,+ function,+ } => {+ if let Some(resolved) = &mut rewrite.resolved {+ resolved.kind = resolved_kind;}- CssUrl::Unresolved(text) => found(CssUrl::Unresolved(text)),- CssUrl::NestedTooDeep => found(CssUrl::NestedTooDeep),- CssUrl::Resolved(_)- | CssUrl::Unshown(_)- | CssUrl::Font { .. }- | CssUrl::ResolvedFont(_) => {}- },- );+ found(CssUrl::Rewrite { rewrite, function });+ }+ CssUrl::Unresolved(text) => found(CssUrl::Unresolved(text)),+ CssUrl::NestedTooDeep => found(CssUrl::NestedTooDeep),+ CssUrl::Unplaced => found(CssUrl::Unplaced),+ CssUrl::Resolved(_)+ | CssUrl::Unshown(_)+ | CssUrl::Font { .. }+ | CssUrl::ResolvedFont(_) => {}+ });}DeclarationKind::FontFamily(_) => {}}@@ -225,7 +199,7 @@ pub fn declaration_image_urls(found: &mut impl FnMut(CssUrl<'_>),) {if can_hold_image(property.trim()) {- value_urls(value, resolver, EscapedSources::Skipped, found);+ value_urls(value, resolver, found);}}@@ -446,13 +420,6 @@ fn is_image_property(property: &str) -> bool {}}-/// An image source that holds a CSS escape is skipped, as v3 skips it. A font has no v3 to match, and its URL must not stay.-#[derive(Clone, Copy, PartialEq, Eq)]-enum EscapedSources {- Skipped,- Decoded,-}-enum Opened<'i> {UnquotedUrl(CowRcStr<'i>),UrlFunction,@@ -504,16 +471,7 @@ fn is_image_set(name: &str) -> bool {name.eq_ignore_ascii_case("image-set") || name.eq_ignore_ascii_case("-webkit-image-set")}-fn skip_block(input: &mut Parser<'_>) {- let _ = input.parse_nested_block(|_| Ok::<_, ParseError<()>>(()));-}--fn value_urls(- value: &str,- resolver: &mut Resolver<'_>,- escaped: EscapedSources,- found: &mut impl FnMut(CssUrl<'_>),-) {+fn value_urls(value: &str, resolver: &mut Resolver<'_>, found: &mut impl FnMut(CssUrl<'_>)) {let mut input = Parser::new(value);input.set_nested_block_limit(NESTED_BLOCK_LIMIT);let whole = each_function(&mut input, 0, &mut |input, start, opened| {@@ -525,9 +483,13 @@ fn value_urls(let Some(source) = source else { return };let function = input.slice_from(start);match collectable(&source) {- Some(url) => url_function_urls(function, url, resolver, found),- None if source.escaped && escaped == EscapedSources::Decoded => {- escaped_url_function_urls(function, &source.url, resolver, found)+ Some(url) => match located(source.written, url) {+ Some(url) => url_function_urls(function, url, resolver, found),+ None => found(CssUrl::Unplaced),+ },+ // A browser loads the URL its escapes decode to, so the block must not keep it as the page wrote it.+ None if source.escaped => {+ escaped_url_function_urls(function, source.written, &source.url, resolver, found)}None => {}}@@ -539,16 +501,19 @@ fn value_urls(struct UrlSource<'i> {url: CowRcStr<'i>,+ /// The URL as the page wrote it, between its quotes or parentheses. A comment in the function can hold a copy of its decoded spelling, so the URL is looked for only here.+ written: &'i str,escaped: bool,}/// The name before the `(` can be escaped, as in `\75 rl(`, and still read as `url`, so only the text after it counts.-fn unquoted_url_source<'i>(url: CowRcStr<'i>, token: &str) -> UrlSource<'i> {+fn unquoted_url_source<'i>(url: CowRcStr<'i>, token: &'i str) -> UrlSource<'i> {let contents = token.split_once('(').map_or(token, |(_, contents)| contents);UrlSource {escaped: contents.contains('\\'),+ written: contents.strip_suffix(')').unwrap_or(contents).trim(),url,}}@@ -559,14 +524,28 @@ fn quoted_url_source<'i>(input: &mut Parser<'i>) -> Option<UrlSource<'i>> {input.skip_whitespace();let start = input.position();let url = input.expect_string_cloned()?;+ let token = input.slice_from(start);Ok::<_, ParseError<()>>(UrlSource {- escaped: input.slice_from(start).contains('\\'),+ escaped: token.contains('\\'),+ written: between_quotes(token),url,})}).ok()}+/// The contents of a string token, whose closing quote is missing when the text ends inside the string.+fn between_quotes(token: &str) -> &str {+ let mut chars = token.chars();+ match chars.next() {+ Some(quote @ ('"' | '\'')) => {+ let rest = chars.as_str();+ rest.strip_suffix(quote).unwrap_or(rest)+ }+ _ => token,+ }+}+/// v3 collects neither a fragment nor a source with an escape in it.fn collectable<'s>(source: &'s UrlSource<'_>) -> Option<&'s str> {let url = source.url.trim();@@ -582,15 +561,21 @@ fn url_function_urls(match resolver.resolve(source) {Some(absolute) => {found(CssUrl::Resolved(&absolute));- found(CssUrl::Rewrite(CssRewrite {- text: function.to_owned(),- replacement: url_function(&absolute),- dropped: Vec::new(),- resolved: Some(RemoteUrl {- url: absolute,- kind: RemoteKind::Image,- }),- }));+ let (replacement, url_at) = url_function(&absolute);+ found(CssUrl::Rewrite {+ rewrite: CssRewrite {+ text: function.to_owned(),+ replacement,+ dropped: Vec::new(),+ resolved: Some(RemoteUrl {+ url: absolute,+ kind: RemoteKind::Image,+ }),+ url_at: Some(url_at),+ spellings: Vec::new(),+ },+ function,+ });}None if is_root_relative(source) => found(CssUrl::Unresolved(function)),None => found(CssUrl::Image(source)),@@ -600,18 +585,19 @@ fn url_function_urls(/// The block does not hold the URL a browser reads from an escaped source, so the rewrite replaces the whole `url()`.fn escaped_url_function_urls(function: &str,+ escaped: &str,decoded: &str,resolver: &mut Resolver<'_>,found: &mut impl FnMut(CssUrl<'_>),) {- let decoded = decoded.trim();- if is_root_relative(decoded) {- return url_function_urls(function, decoded, resolver, found);+ let written = decoded.trim();+ if is_root_relative(written) {+ return url_function_urls(function, written, resolver, found);}- let decoded = if is_protocol_relative(decoded) {- Cow::Owned(format!("https:{decoded}"))+ let decoded = if is_protocol_relative(written) {+ Cow::Owned(format!("https:{written}"))} else {- Cow::Borrowed(decoded)+ Cow::Borrowed(written)};// A serialized URL percent-encodes `"` and holds no backslash, so the quotes of the rewrite hold it.let Some(absolute) = Url::parse(&decoded)@@ -622,24 +608,41 @@ fn escaped_url_function_urls(return;};found(CssUrl::Resolved(&absolute));- found(CssUrl::Rewrite(CssRewrite {- text: function.to_owned(),- replacement: url_function(&absolute),- dropped: Vec::new(),- resolved: Some(RemoteUrl {- url: absolute,- kind: RemoteKind::Image,- }),- }));+ let (replacement, url_at) = url_function(&absolute);+ let mut spellings: Vec<String> = Vec::new();+ for spelling in [escaped, written] {+ if spelling != absolute && !spellings.iter().any(|kept| kept == spelling) {+ spellings.push(spelling.to_owned());+ }+ }+ found(CssUrl::Rewrite {+ rewrite: CssRewrite {+ text: function.to_owned(),+ replacement,+ dropped: Vec::new(),+ resolved: Some(RemoteUrl {+ url: absolute,+ kind: RemoteKind::Image,+ }),+ url_at: Some(url_at),+ spellings,+ },+ function,+ });}fn is_root_relative(source: &str) -> bool {source.starts_with('/') && !is_protocol_relative(source)}+const URL_FUNCTION_OPEN: &str = "url(\"";+/// A serialized URL percent-encodes `"`, so the quotes hold any URL that [`Resolver`] returns.-fn url_function(url: &str) -> String {- format!("url(\"{url}\")")+fn url_function(url: &str) -> (String, Range<usize>) {+ (+ format!("{URL_FUNCTION_OPEN}{url}\")"),+ URL_FUNCTION_OPEN.len()..URL_FUNCTION_OPEN.len() + url.len(),+ )}/// A renderer picks the candidate for its pixel ratio and fails offline on any the fetcher did not take, so the set keeps@@ -662,6 +665,9 @@ fn image_set_urls(let Some(candidates) = candidates else {return;};+ if candidates.iter().any(|candidate| candidate.unplaced) {+ found(CssUrl::Unplaced);+ }let loads = |source: &str| {is_data_url(source)|| (is_remote(source) && try_canonicalize(source).is_ok())@@ -671,7 +677,7 @@ fn image_set_urls(.iter().enumerate().filter_map(|(index, candidate)| {- let source = candidate.source.as_deref().filter(|source| loads(source))?;+ let source = candidate.source.filter(|source| loads(source))?;Some((index, source, candidate.resolution_dppx?))}).fold(None, |best: Option<(usize, &str, f64)>, candidate| {@@ -683,7 +689,7 @@ fn image_set_urls(});let mut dropped = Vec::new();for (index, candidate) in candidates.iter().enumerate() {- let source = candidate.source.as_deref().unwrap_or_default();+ let source = candidate.source.unwrap_or_default();if kept.is_none_or(|kept| kept.0 != index) && is_remote(source) {found(CssUrl::Unshown(source));dropped.push(source.to_owned());@@ -699,26 +705,34 @@ fn image_set_urls(None => found(CssUrl::Image(source)),}// A candidate kept as the page wrote it holds its URL as the writer's URL pass looks for it.- let candidate = match &resolved {- Some(absolute) => format!("{} {}", url_function(absolute), kept.descriptors.trim())- .trim_end()- .to_owned(),- None => kept.text.to_owned(),+ let (candidate, url_in_candidate) = match &resolved {+ Some(absolute) => {+ let (function, url_at) = url_function(absolute);+ let candidate = format!("{function} {}", kept.descriptors.trim());+ (candidate.trim_end().to_owned(), Some(url_at))+ }+ None => (kept.text.to_owned(), range_in(kept.text, source)),};+ let opening = name.len() + 1;// The writer moves an inline image out of the block before it rewrites CSS, so a set that holds one never matches.let holds_data_url = candidates.iter()- .any(|candidate| candidate.source.as_deref().is_some_and(is_data_url));+ .any(|candidate| candidate.source.is_some_and(is_data_url));if !holds_data_url && (candidates.len() > 1 || resolved.is_some()) {- found(CssUrl::Rewrite(CssRewrite {- text: function.to_owned(),- replacement: format!("{name}({candidate})"),- dropped,- resolved: resolved.map(|url| RemoteUrl {- url,- kind: RemoteKind::Image,- }),- }));+ found(CssUrl::Rewrite {+ rewrite: CssRewrite {+ text: function.to_owned(),+ replacement: format!("{name}({candidate})"),+ dropped,+ resolved: resolved.map(|url| RemoteUrl {+ url,+ kind: RemoteKind::Image,+ }),+ url_at: url_in_candidate.map(|at| at.start + opening..at.end + opening),+ spellings: Vec::new(),+ },+ function,+ });}}@@ -730,7 +744,9 @@ fn is_data_url(source: &str) -> bool {struct ImageSetCandidate<'a> {text: &'a str,- source: Option<String>,+ /// A slice of the scanned text, so that the writer finds the URL where the page wrote it.+ source: Option<&'a str>,+ unplaced: bool,descriptors: &'a str,resolution_dppx: Option<f64>,}@@ -758,9 +774,12 @@ fn image_set_candidates<'i>(}} else if readable {let held = std::mem::take(&mut candidates);- for source in held.into_iter().chain(candidate).filter_map(|c| c.source) {- if is_remote(&source) {- found(CssUrl::Unshown(&source));+ for candidate in held.into_iter().chain(candidate) {+ if candidate.unplaced {+ found(CssUrl::Unplaced);+ }+ if let Some(source) = candidate.source.filter(|source| is_remote(source)) {+ found(CssUrl::Unshown(source));}}}@@ -794,21 +813,37 @@ fn image_set_candidate<'i>(input: &mut Parser<'i>) -> Option<ImageSetCandidate<'CandidateStart::Url(url) => Some(unquoted_url_source(url, input.slice_from(start))),CandidateStart::String(url) => Some(UrlSource {escaped: input.slice_from(start).contains('\\'),+ written: between_quotes(input.slice_from(start)),url,}),CandidateStart::UrlFunction => quoted_url_source(input),CandidateStart::Other => None,};let source_end = input.position();+ let written = source.as_ref().and_then(collectable);+ // A set keeps a candidate as the page wrote it, so an escaped remote URL, which the block does not hold as a browser reads it, has no place.+ let escaped_remote = source+ .as_ref()+ .is_some_and(|source| source.escaped && is_remote(source.url.trim()));+ let source = source+ .as_ref()+ .zip(written)+ .and_then(|(source, url)| located(source.written, url));let resolution_dppx = parse_descriptors(input);Some(ImageSetCandidate {text: input.slice_from(start).trim_end(),- source: source.as_ref().and_then(collectable).map(str::to_owned),+ source,+ unplaced: escaped_remote || (written.is_some() && source.is_none()),descriptors: input.slice(source_end..input.position()),resolution_dppx,})}+/// `url` as a slice of `text`. A source with no escape reads as the page wrote it, so it is always there.+fn located<'t>(text: &'t str, url: &str) -> Option<&'t str> {+ text.find(url).map(|at| &text[at..at + url.len()])+}+enum Descriptor {Resolution(f64),Type,@@ -851,6 +886,190 @@ fn resolution_dppx(value: f64, unit: &str) -> Option<f64> {}}+#[derive(Debug, Clone, PartialEq, Eq)]+pub enum CssPart {+ Text(String),+ Url(RemoteUrl),+}++#[derive(Debug, Clone, PartialEq, Eq)]+pub struct CssPiece {+ pub range: Range<usize>,+ pub parts: Vec<CssPart>,+}++/// The parts of one CSS text that the writer replaces, in order, by byte range in the text that [`urls`] read.+#[derive(Debug, Clone, Default, PartialEq, Eq)]+pub struct CssEdit {+ pub pieces: Vec<CssPiece>,+}++impl CssUrl<'_> {+ /// An estimate of the memory that [`CssEditBuilder::observe`] adds to an edit for this URL.+ pub fn edit_bytes(&self) -> usize {+ let piece = std::mem::size_of::<CssPiece>() + std::mem::size_of::<CssPart>();+ match self {+ CssUrl::Image(url) | CssUrl::Unshown(url) | CssUrl::Font { url, .. } => {+ piece + url.len()+ }+ CssUrl::Rewrite { rewrite, .. } => {+ let resolved = rewrite.resolved.as_ref().map_or(0, |url| url.url.len());+ piece + std::mem::size_of::<CssPart>() + rewrite.replacement.len() + resolved+ }+ CssUrl::Resolved(_)+ | CssUrl::ResolvedFont(_)+ | CssUrl::Unresolved(_)+ | CssUrl::NestedTooDeep+ | CssUrl::Unplaced => 0,+ }+ }+}++/// Turns what [`urls`] reports for one text into its [`CssEdit`], so that the writer searches the text for nothing.+pub struct CssEditBuilder<'c> {+ css: &'c str,+ pieces: Vec<CssPiece>,+ /// URLs that the next rewrite holds, such as the kept candidate of an `image-set()`.+ pending: Vec<RemoteUrl>,+ unplaced: bool,+}++impl<'c> CssEditBuilder<'c> {+ pub fn new(css: &'c str) -> Self {+ Self {+ css,+ pieces: Vec::new(),+ pending: Vec::new(),+ unplaced: false,+ }+ }++ pub fn observe(&mut self, found: &CssUrl<'_>) {+ match found {+ CssUrl::Image(url) | CssUrl::Unshown(url) => self.place(url, RemoteKind::Image),+ CssUrl::Font { url, family } => {+ let kind = match family.and_then(bundled_font) {+ Some(slug) => RemoteKind::BundledFont(slug),+ None => RemoteKind::Font,+ };+ self.place(url, kind);+ }+ CssUrl::Rewrite { rewrite, function } => self.rewrite(rewrite, function),+ CssUrl::Unplaced => self.unplaced = true,+ CssUrl::Resolved(_)+ | CssUrl::ResolvedFont(_)+ | CssUrl::Unresolved(_)+ | CssUrl::NestedTooDeep => {}+ }+ }++ fn place(&mut self, url: &str, kind: RemoteKind) {+ let url = url.trim();+ if !is_remote(url) {+ return;+ }+ let named = RemoteUrl {+ url: url.to_owned(),+ kind,+ };+ match range_in(self.css, url) {+ Some(range) => self.pieces.push(CssPiece {+ range,+ parts: vec![CssPart::Url(named)],+ }),+ None => self.pending.push(named),+ }+ }++ /// The rewrite replaces the whole function, so the URLs placed inside it go into its replacement, or out with what it drops.+ fn rewrite(&mut self, rewrite: &CssRewrite, function: &str) {+ let Some(range) = range_in(self.css, function) else {+ self.unplaced = true;+ return;+ };+ let mut urls = std::mem::take(&mut self.pending);+ // The scan reports a function's URLs right before its rewrite, so they are the last pieces placed.+ while let Some(piece) = self+ .pieces+ .pop_if(|piece| range.start <= piece.range.start && piece.range.end <= range.end)+ {+ urls.extend(piece.parts.into_iter().filter_map(|part| match part {+ CssPart::Url(url) => Some(url),+ CssPart::Text(_) => None,+ }));+ }+ urls.extend(rewrite.resolved.clone());+ let url = rewrite.url_at.clone().and_then(|at| {+ let written = rewrite.replacement.get(at.clone())?;+ let url = urls.into_iter().rev().find(|url| url.url == written)?;+ Some((at, url))+ });+ self.pieces.push(CssPiece {+ range,+ parts: parts_around_url(&rewrite.replacement, url),+ });+ }++ /// The edit, or `None` when the text keeps every byte. `Err` when a URL has no place in the text.+ pub fn finish(mut self) -> Result<Option<CssEdit>, Unplaced> {+ self.pieces.sort_by_key(|piece| piece.range.start);+ let overlaps = self+ .pieces+ .windows(2)+ .any(|pair| pair[0].range.end > pair[1].range.start);+ if self.unplaced || !self.pending.is_empty() || overlaps {+ return Err(Unplaced);+ }+ Ok((!self.pieces.is_empty()).then_some(CssEdit {+ pieces: self.pieces,+ }))+ }+}++#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub struct Unplaced;++fn range_in(text: &str, part: &str) -> Option<Range<usize>> {+ let start = (part.as_ptr() as usize).checked_sub(text.as_ptr() as usize)?;+ (start + part.len() <= text.len()).then_some(start..start + part.len())+}++fn parts_around_url(replacement: &str, url: Option<(Range<usize>, RemoteUrl)>) -> Vec<CssPart> {+ let Some((at, url)) = url else {+ return vec![CssPart::Text(replacement.to_owned())];+ };+ let mut parts = Vec::with_capacity(3);+ if at.start > 0 {+ parts.push(CssPart::Text(replacement[..at.start].to_owned()));+ }+ parts.push(CssPart::Url(url));+ if at.end < replacement.len() {+ parts.push(CssPart::Text(replacement[at.end..].to_owned()));+ }+ parts+}++pub fn write_css_edit(+ text: &str,+ edit: &CssEdit,+ reference: &mut impl FnMut(&RemoteUrl) -> String,+) -> String {+ let mut out = String::with_capacity(text.len());+ let mut copied_to = 0;+ for piece in &edit.pieces {+ out.push_str(&text[copied_to..piece.range.start]);+ for part in &piece.parts {+ match part {+ CssPart::Text(literal) => out.push_str(literal),+ CssPart::Url(url) => out.push_str(&reference(url)),+ }+ }+ copied_to = piece.range.end;+ }+ out.push_str(&text[copied_to..]);+ out+}+#[cfg(test)]mod tests {use super::*;@@ -864,13 +1083,14 @@ mod tests {CssUrl::Unshown(url) => format!("unshown {url}"),CssUrl::Resolved(url) => format!("resolved {url}"),CssUrl::Unresolved(text) => format!("unresolved {text}"),- CssUrl::Rewrite(rewrite) => format!(+ CssUrl::Rewrite { rewrite, .. } => format!("rewrite {} => {} dropping {:?}",rewrite.text, rewrite.replacement, rewrite.dropped),CssUrl::Font { url, family } => format!("font {url} {family:?}"),CssUrl::ResolvedFont(url) => format!("resolved font {url}"),CssUrl::NestedTooDeep => "nested too deep".to_owned(),+ CssUrl::Unplaced => "unplaced".to_owned(),})});found@@ -1015,10 +1235,6 @@ mod tests {named,["https://app.example.com/img/a.png", "url(/img/b.png)"]);- let rewritten = rewrite_functions("url(/a.png) image-set( url(/a.png)", &mut |function| {- (function == "url(/a.png)").then(|| "url(x)".to_owned())- });- assert_eq!(rewritten.as_deref(), Some("url(x) image-set( url(/a.png)"));let many: Vec<String> = (1..=MAX_IMAGE_SET_CANDIDATES + 1).map(|i| format!("url(https://cdn.example.com/{i}.png) {i}x")).collect();diff --git a/rust/ai-research/replay-core-v4/src/inspect/dom.rs b/rust/ai-research/replay-core-v4/src/inspect/dom.rsindex 967877c46625..4809fc233247 100644--- a/rust/ai-research/replay-core-v4/src/inspect/dom.rs+++ b/rust/ai-research/replay-core-v4/src/inspect/dom.rs@@ -3,16 +3,17 @@use std::collections::{HashMap, HashSet};use std::ops::Range;-use std::sync::atomic::AtomicUsize;+use std::sync::atomic::{AtomicUsize, Ordering};use std::sync::Arc;use url::Url;use crate::bytescan::{self as scan, Span};use crate::fonts::{bundled_font, is_fetched_font_url};-use crate::url_policy::{is_fetched_image_type, is_remote, MAX_URL_LEN};+use crate::url_policy::{checked_url, is_fetched_image_type, is_remote, Decline, MAX_URL_LEN};-use super::css::{self, CssRewrite, CssUrl, Resolver};+use super::css::{self, CssEditBuilder, CssUrl, Resolver};+use super::edits::{Buffers, Edit, Slot};use super::events::{field, fields, number, text};use super::rrweb::{ADOPTED_STYLE_SHEET, FULL_SNAPSHOT, INCREMENTAL_SNAPSHOT, MUTATION, STYLE_DECLARATION,@@ -32,13 +33,8 @@ pub const MAX_FONT_URLS: usize = 512;/// Attribute changes read per mutation, as many as the nodes tracked. A mutation with more drops its block, because/// the inspector does not name the URLs in the rest.pub const MAX_ATTRIBUTE_ITEMS: usize = MAX_TRACKED_NODES;-/// The distinct remote URLs in a block's image slots, and their bytes. The writer builds one matcher over them, and-/// the build takes memory in proportion to their bytes.-pub const MAX_BLOCK_REMOTE_URLS: usize = 10_000;-pub const MAX_BLOCK_REMOTE_URL_BYTES: usize = 2 * 1024 * 1024;-/// Font URLs have a budget of their own, so that many font faces never stop the naming of image URLs. Past it the writer scrubs the block's unnamed remote URLs, as past an image limit.-pub const MAX_BLOCK_FONT_URLS: usize = 4096;-pub const MAX_BLOCK_FONT_URL_BYTES: usize = 256 * 1024;+/// The writer builds one matcher over a block's refused URLs to overwrite their copies, and its memory follows their bytes.+pub const MAX_BLOCK_DECLINED_URL_BYTES: usize = 1024 * 1024;/// Windows tracked per session. A new window past this evicts the window used least recently.pub const MAX_WINDOWS: usize = 16;/// Evicted windows remembered, so that one that comes back is known to have lost its DOM.@@ -196,48 +192,29 @@ struct WindowDom {page_base: Option<Arc<Url>>,}-/// Why the inspector stopped naming a block's remote image or font URLs. The writer replaces only the URLs it is given, so it scrubs every other remote URL token in the block.+/// Why the inspector could not read a block in full. A URL in the part it did not read went unchecked, so the writer drops the block.#[derive(Debug, Clone, Copy, PartialEq, Eq)]pub enum UrlLimit {- UrlCount,- UrlBytes,AttributeItems,- FontUrlCount,- FontUrlBytes,- /// A font rewrite can hold an escaped remote URL, which no other rewrite replaces.- FontRewrites,NestedBlocks,+ DeclinedUrls,+ Edits,}impl UrlLimit {/// The outcome of the dropped block in the recorder's metrics.pub fn label(self) -> &'static str {match self {- UrlLimit::UrlCount => "too_many_urls",- UrlLimit::UrlBytes => "too_many_url_bytes",UrlLimit::AttributeItems => "too_many_attribute_items",- UrlLimit::FontUrlCount => "too_many_font_urls",- UrlLimit::FontUrlBytes => "too_many_font_url_bytes",- UrlLimit::FontRewrites => "too_many_font_rewrites",UrlLimit::NestedBlocks => "too_many_nested_blocks",+ UrlLimit::DeclinedUrls => "too_many_declined_urls",+ UrlLimit::Edits => "too_many_edits",}}-- /// The scrub replaces only the URL tokens it finds in the text, so it cannot vouch for CSS that the inspector could not read.- pub fn drops_block(self) -> bool {- matches!(self, UrlLimit::NestedBlocks)- }-- pub fn is_font_budget(self) -> bool {- matches!(- self,- UrlLimit::FontUrlCount | UrlLimit::FontUrlBytes | UrlLimit::FontRewrites- )- }}-/// One URL gets one reference, so a URL used as several kinds takes the greatest. A fetched font beats a bundled one because it is the page's own file.-#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]+/// The kind of a remote URL by the slot that holds it. A refused URL named as several kinds takes the greatest.+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]pub enum RemoteKind {BundledFont(&'static str),Font,@@ -250,193 +227,12 @@ pub struct RemoteUrl {pub kind: RemoteKind,}-/// Distinct remote URLs, within a count and a byte budget per kind. Past a budget it keeps no more of that kind, and [`RemoteUrls::url_limit`] says why.-#[derive(Debug, Default)]-pub struct RemoteUrls {- urls: HashMap<String, RemoteKind>,- bytes: usize,- font_urls: u16,- font_bytes: u32,- pub limit: Option<UrlLimit>,- font_limit: Option<UrlLimit>,- /// Each image-slot srcset value, and the slot that holds it. The writer rebuilds each from the candidates it keeps,- /// and it builds one matcher over them as over `urls`, so they share the byte budget of `urls`. Past it this keeps no- /// more, which is safe: the writer still replaces each URL in `urls` on its own.- /// Boxed, because the session state counts this struct's size and most blocks hold no srcset.- srcsets: Option<Box<SrcsetValues>>,- /// Past its count and byte budget this keeps no more, which is safe: each URL a rewrite drops is in `urls` too, and a- /// relative URL stays as the page wrote it.- css_rewrites: Option<Box<CssRewrites>>,-}--#[derive(Debug, Default)]-struct CssRewrites {- values: HashMap<String, Option<CssRewrite>>,- bytes: usize,-}--#[derive(Debug, Default)]-struct SrcsetValues {- values: HashMap<Vec<u8>, (String, Vec<SrcsetSlot>)>,- bytes: usize,-}--/// An image-slot srcset value as the event holds it.+/// A spelling of a refused URL, with the reason for the URL as the policy reads it, which an escaped spelling hides.#[derive(Debug, Clone, PartialEq, Eq)]-pub struct Srcset {- pub text: String,- /// The value as a JSON string with its quotes and escapes, as the event holds it.- pub raw: Vec<u8>,- pub slot: SrcsetSlot,-}--impl RemoteUrls {- fn insert(&mut self, url: &str, kind: RemoteKind) {- if let Some(known) = self.urls.get_mut(url) {- *known = (*known).max(kind);- return;- }- if kind != RemoteKind::Image {- let font_bytes = self.font_bytes as usize + url.len();- if usize::from(self.font_urls) >= MAX_BLOCK_FONT_URLS {- self.font_limit = self.font_limit.or(Some(UrlLimit::FontUrlCount));- } else if font_bytes > MAX_BLOCK_FONT_URL_BYTES {- self.font_limit = self.font_limit.or(Some(UrlLimit::FontUrlBytes));- } else {- self.font_urls += 1;- self.font_bytes = font_bytes as u32;- self.urls.insert(url.to_owned(), kind);- }- return;- }- if self.limit.is_some() {- return;- }- // A URL of any length is named: the policy declines one too long to fetch, and its bytes count toward the- // block's budget.- self.limit = if self.urls.len() - usize::from(self.font_urls) >= MAX_BLOCK_REMOTE_URLS {- Some(UrlLimit::UrlCount)- } else if self.bytes + url.len() > MAX_BLOCK_REMOTE_URL_BYTES {- Some(UrlLimit::UrlBytes)- } else {- self.bytes += url.len();- self.urls.insert(url.to_owned(), kind);- None- };- }-- /// A value in several kinds of slot keeps each kind, so that the writer keeps the candidates that some slot shows- /// and no other. Keyed by the value as the event holds it, so the writer finds that value again after it stores the- /// event.- fn insert_srcset(&mut self, raw: &[u8], text: &str, slot: SrcsetSlot) {- let srcsets = self.srcsets.get_or_insert_with(Box::default);- if let Some((_, slots)) = srcsets.values.get_mut(raw) {- if !slots.contains(&slot) {- slots.push(slot);- }- } else if srcsets.values.len() < MAX_BLOCK_REMOTE_URLS- && srcsets.bytes + raw.len() + text.len() <= MAX_BLOCK_REMOTE_URL_BYTES- {- srcsets.bytes += raw.len() + text.len();- srcsets- .values- .insert(raw.to_vec(), (text.to_owned(), vec![slot]));- }- }-- /// A text that resolves two ways, as a root-relative URL in windows on two origins, keeps no rewrite, so that no- /// window shows another's image. Its URL then stays relative, as the page wrote it.- fn insert_css_rewrite(&mut self, text: String, rewrite: Option<CssRewrite>) {- let rewrites = self.css_rewrites.get_or_insert_with(Box::default);- if let Some(known) = rewrites.values.get_mut(&text) {- if known.as_ref().map(|known| &known.replacement)- != rewrite.as_ref().map(|rewrite| &rewrite.replacement)- {- *known = None;- } else if let (Some(known), Some(resolved)) = (- known.as_mut().and_then(|known| known.resolved.as_mut()),- rewrite.and_then(|rewrite| rewrite.resolved),- ) {- known.kind = known.kind.max(resolved.kind);- }- return;- }- let bytes = text.len()- + rewrite.as_ref().map_or(0, |rewrite| {- rewrite.replacement.len()- + rewrite.dropped.iter().map(String::len).sum::<usize>()- + rewrite- .resolved- .as_ref()- .map_or(0, |resolved| resolved.url.len())- });- if rewrites.values.len() < MAX_BLOCK_REMOTE_URLS- && rewrites.bytes + bytes <= MAX_BLOCK_REMOTE_URL_BYTES- {- rewrites.bytes += bytes;- rewrites.values.insert(text, rewrite);- } else if rewrite- .and_then(|rewrite| rewrite.resolved)- .is_some_and(|resolved| resolved.kind != RemoteKind::Image)- {- self.font_limit = self.font_limit.or(Some(UrlLimit::FontRewrites));- }- }-- pub fn extend(&mut self, other: RemoteUrls) {- // The URLs go in before the other's limit, so that a part that hit a limit still names what it holds.- for (url, &kind) in &other.urls {- self.insert(url, kind);- }- self.limit = self.limit.or(other.limit);- self.font_limit = self.font_limit.or(other.font_limit);- for (raw, (text, slots)) in other.srcsets.iter().flat_map(|s| &s.values) {- for &slot in slots {- self.insert_srcset(raw, text, slot);- }- }- for (text, rewrite) in other.css_rewrites.into_iter().flat_map(|r| r.values) {- self.insert_css_rewrite(text, rewrite);- }- }-- /// Why the writer scrubs the block. Past a font limit, unlike past `limit`, the inspector still names image URLs.- pub fn url_limit(&self) -> Option<UrlLimit> {- self.limit.or(self.font_limit)- }-- pub fn take_css_rewrites(&mut self) -> Vec<CssRewrite> {- self.css_rewrites- .take()- .map(|rewrites| rewrites.values)- .unwrap_or_default()- .into_values()- .flatten()- .collect()- }-- pub fn take_srcsets(&mut self) -> Vec<Srcset> {- self.srcsets- .take()- .map(|srcsets| srcsets.values)- .unwrap_or_default()- .into_iter()- .flat_map(|(raw, (text, slots))| {- slots.into_iter().map(move |slot| Srcset {- text: text.clone(),- raw: raw.clone(),- slot,- })- })- .collect()- }-- pub fn into_urls(self) -> Vec<RemoteUrl> {- self.urls- .into_iter()- .map(|(url, kind)| RemoteUrl { url, kind })- .collect()- }+pub struct DeclinedUrl {+ pub url: String,+ pub kind: RemoteKind,+ pub decline: Decline,}/// Distinct remote image URLs, at most [`MAX_IMAGE_URLS`] per message.@@ -446,9 +242,15 @@ struct Urls {in_message: HashSet<String>,/// Boxed, because the session state counts this struct's size and most messages queue no font file.fonts: Option<Box<FontQueue>>,- /// Every remote URL in an image slot, past the fetch cap and in a hidden img too. The writer replaces each one in- /// the stored block, and the fetch cap limits only the fetch jobs.- remote: RemoteUrls,+ limit: Option<UrlLimit>,+ /// Every slot with a remote URL, past the fetch cap and in a hidden img too, because the fetch cap limits only the fetch jobs.+ edits: Vec<Edit>,+ edit_bytes_taken: usize,+ /// A slot the writer could not find again, so that the block must not be stored.+ unplaced: bool,+ /// Named URLs that the policy refuses, by the largest kind each was named as. The writer overwrites every copy of each one, outside its slot too.+ declined: HashMap<String, (RemoteKind, Decline)>,+ declined_bytes: usize,}#[derive(Default)]@@ -482,7 +284,28 @@ impl Urls {fn push_remote(&mut self, url: &str) {let url = url.trim();if is_remote(url) {- self.remote.insert(url, RemoteKind::Image);+ self.note_declined(url, RemoteKind::Image);+ }+ }++ fn note_declined(&mut self, url: &str, kind: RemoteKind) -> Option<Decline> {+ if let Some((noted, decline)) = self.declined.get_mut(url) {+ *noted = (*noted).max(kind);+ return Some(*decline);+ }+ let decline = checked_url(url).err()?;+ self.add_declined(url.to_owned(), kind, decline);+ Some(decline)+ }++ fn add_declined(&mut self, url: String, kind: RemoteKind, decline: Decline) {+ if let Some((noted, _)) = self.declined.get_mut(&url) {+ *noted = (*noted).max(kind);+ } else if self.declined_bytes + url.len() > MAX_BLOCK_DECLINED_URL_BYTES {+ self.limit = self.limit.or(Some(UrlLimit::DeclinedUrls));+ } else {+ self.declined_bytes += url.len();+ self.declined.insert(url, (kind, decline));}}@@ -491,12 +314,13 @@ impl Urls {if !is_remote(url) {return;}- match family.and_then(bundled_font) {- Some(slug) => self.remote.insert(url, RemoteKind::BundledFont(slug)),- None => {- self.remote.insert(url, RemoteKind::Font);- self.queue_font(url);- }+ let kind = match family.and_then(bundled_font) {+ Some(slug) => RemoteKind::BundledFont(slug),+ None => RemoteKind::Font,+ };+ self.note_declined(url, kind);+ if kind == RemoteKind::Font {+ self.queue_font(url);}}@@ -515,16 +339,102 @@ impl Urls {match found {CssUrl::Image(url) => self.push(url),CssUrl::Unshown(url) => self.push_remote(url),- CssUrl::Resolved(url) => self.push_fetch(url),- CssUrl::Unresolved(text) => self.remote.insert_css_rewrite(text.to_owned(), None),- CssUrl::Rewrite(rewrite) => self- .remote- .insert_css_rewrite(rewrite.text.clone(), Some(rewrite)),+ CssUrl::Resolved(url) => {+ self.note_declined(url, RemoteKind::Image);+ self.push_fetch(url);+ }+ // A bundled font's URL comes only in its rewrite, as no fetch takes it.+ CssUrl::Rewrite { rewrite, .. } => {+ let Some(resolved) = &rewrite.resolved else {+ return;+ };+ if let Some(decline) = self.note_declined(&resolved.url, resolved.kind) {+ for spelling in &rewrite.spellings {+ self.add_declined(spelling.clone(), resolved.kind, decline);+ }+ }+ }+ CssUrl::Unresolved(_) => {}CssUrl::Font { url, family } => self.push_font(url, family),- CssUrl::ResolvedFont(url) => self.queue_font(url),- CssUrl::NestedTooDeep => {- self.remote.limit = self.remote.limit.or(Some(UrlLimit::NestedBlocks))+ CssUrl::ResolvedFont(url) => {+ self.note_declined(url, RemoteKind::Font);+ self.queue_font(url);+ }+ CssUrl::NestedTooDeep => self.limit = self.limit.or(Some(UrlLimit::NestedBlocks)),+ CssUrl::Unplaced => self.unplaced = true,+ }+ }++ /// A block past any limit is dropped, so it takes no more of the budget and builds no more edits.+ fn take_edit_bytes(&mut self, budget: Option<&AtomicUsize>, bytes: usize) -> bool {+ if self.limit.is_some() {+ return false;+ }+ let within_budget = budget.is_none_or(|budget| {+ budget+ .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |left| {+ left.checked_sub(bytes)+ })+ .is_ok()+ });+ if within_budget {+ self.edit_bytes_taken += bytes;+ } else {+ self.limit = Some(UrlLimit::Edits);+ }+ within_budget+ }++ fn record(+ &mut self,+ buffers: &Buffers,+ (bytes, value): (&[u8], Span),+ slot: Slot,+ edit_budget: Option<&AtomicUsize>,+ ) {+ let Some(source) = buffers.source_of(bytes) else {+ self.unplaced = true;+ return;+ };+ if self.take_edit_bytes(edit_budget, std::mem::size_of::<Edit>()) {+ self.edits.push(Edit {+ source,+ value,+ slot,+ });+ }+ }++ /// Reads `css`, the text of the JSON string at `value`, as a style sheet, or as the value of `property`.+ fn read_css(+ &mut self,+ buffers: &Buffers,+ (bytes, value): (&[u8], Span),+ css: &str,+ property: Option<&str>,+ resolver: &mut Resolver<'_>,+ ) {+ let edit_budget = resolver.edit_budget();+ let mut edit = Some(CssEditBuilder::new(css));+ let mut found = |found: CssUrl<'_>| {+ if edit.is_some() && !self.take_edit_bytes(edit_budget, found.edit_bytes()) {+ edit = None;}+ if let Some(edit) = &mut edit {+ edit.observe(&found);+ }+ self.push_css(found);+ };+ match property {+ Some(property) => css::declaration_image_urls(property, css, resolver, &mut found),+ None => css::urls(css, resolver, &mut found),+ }+ match edit.map(CssEditBuilder::finish) {+ Some(Ok(Some(edit))) => {+ self.record(buffers, (bytes, value), Slot::Css(edit), edit_budget)+ }+ Some(Ok(None)) | None => {}+ Some(Err(_)) => self.unplaced = true,}}@@ -536,10 +446,23 @@ impl Urls {for url in walk.fonts.iter().flat_map(|fonts| &fonts.urls) {self.queue_font(url);}- self.remote.extend(walk.remote);+ self.limit = self.limit.or(walk.limit);+ self.edits.extend(walk.edits);+ self.edit_bytes_taken += walk.edit_bytes_taken;+ self.unplaced |= walk.unplaced;+ for (url, (kind, decline)) in walk.declined {+ self.add_declined(url, kind, decline);+ }}}+/// The budgets that the blocks of one flush share, so that together they hold no more than the flush may.+#[derive(Debug, Clone, Copy, Default)]+pub struct FlushBudgets<'b> {+ pub resolved_bytes: Option<&'b AtomicUsize>,+ pub edit_bytes: Option<&'b AtomicUsize>,+}+/// The DOM state of a session's windows in this process.#[derive(Default)]pub struct DomInspector {@@ -565,6 +488,7 @@ pub struct Observed {#[derive(Default)]struct Walk<'p> {resolver: Resolver<'p>,+ buffers: Buffers,/// Each node and its parent. A root's parent is `None`.ids: Vec<(i64, Option<i64>)>,/// The walk met more nodes than the node cap, so `ids` stops short.@@ -596,24 +520,34 @@ impl Walk<'_> {}fn push_url(&mut self, bytes: &[u8], value: Option<Span>) {- if let Some(url) = value+ if let Some((span, url)) = value.filter(|&v| scan::is_string(bytes, v))- .and_then(|v| scan::unescape(bytes, v).ok())+ .and_then(|v| Some((v, scan::unescape(bytes, v).ok()?))){+ self.record_url(bytes, span, &url);self.push_url_text(&url);}}+ fn record_url(&mut self, bytes: &[u8], value: Span, url: &str) {+ if is_remote(url.trim()) {+ let edit_budget = self.resolver.edit_budget();+ self.image_urls+ .record(&self.buffers, (bytes, value), Slot::Url, edit_budget);+ }+ }+fn push_url_text(&mut self, url: &str) {self.image_urls.push(url);}/// The URLs of a slot the fetcher does not take now. The block still holds them, so the writer replaces them.fn push_remote(&mut self, bytes: &[u8], value: Option<Span>, srcset: bool) {- if let Some(text) = text(bytes, value) {+ if let (Some(span), Some(text)) = (value, text(bytes, value)) {if srcset {srcset_urls(&text, &mut |url| self.image_urls.push_remote(url));} else {+ self.record_url(bytes, span, &text);self.image_urls.push_remote(&text);}}@@ -634,10 +568,14 @@ impl Walk<'_> {}fn push_srcset_value(&mut self, bytes: &[u8], value: Option<Span>, slot: SrcsetSlot) {- if let (Some(span), Some(srcset)) = (value, text(bytes, value)) {- self.image_urls- .remote- .insert_srcset(&bytes[span.0..span.1], &srcset, slot);+ if let (Some(span), Some(_)) = (value, text(bytes, value)) {+ let edit_budget = self.resolver.edit_budget();+ self.image_urls.record(+ &self.buffers,+ (bytes, span),+ Slot::Srcset(slot),+ edit_budget,+ );}}@@ -656,22 +594,29 @@ impl Walk<'_> {fn push_css(&mut self, bytes: &[u8], value: Option<Span>) {let Some(value) = value else { return };if let Some(text) = text(bytes, Some(value)) {- css::urls(&text, &mut self.resolver, &mut |found| {- self.image_urls.push_css(found)- });+ self.image_urls.read_css(+ &self.buffers,+ (bytes, value),+ &text,+ None,+ &mut self.resolver,+ );} else if scan::is_object(bytes, value) {let Ok(entries) = scan::object_entries(bytes, value) else {return;};for entry in entries.filter_map(Result::ok).filter(|e| !e.key_escaped) {let property = std::str::from_utf8(&bytes[entry.key.0..entry.key.1]);- let declared = text(bytes, style_property_value(bytes, entry.value));- if let (Ok(property), Some(declared)) = (property, declared) {- css::declaration_image_urls(- property,+ let declared_at = style_property_value(bytes, entry.value);+ let declared = text(bytes, declared_at);+ if let (Ok(property), Some(at), Some(declared)) = (property, declared_at, declared)+ {+ self.image_urls.read_css(+ &self.buffers,+ (bytes, at),&declared,+ Some(property),&mut self.resolver,- &mut |found| self.image_urls.push_css(found),);}}@@ -1202,6 +1147,7 @@ fn walk_snapshot<'p>(};let mut walk = Walk {resolver,+ buffers: Buffers::new(inner, expansion),..Walk::default()};if walk_nodes(bytes, node, &mut walk) {@@ -1435,7 +1381,7 @@ impl DomInspector {event: &Event,window: &str,event_page_url: Option<&str>,- resolve_budget: Option<&AtomicUsize>,+ budgets: FlushBudgets<'_>,stats: &mut BlockStats,) -> Observed {self.clock += 1;@@ -1450,7 +1396,8 @@ impl DomInspector {dom.page_base = css::page_base(page_url).map(Arc::new);}let page_base = dom.page_base.clone();- let resolver = Resolver::new(page_base.as_deref(), resolve_budget);+ let resolver = Resolver::new(page_base.as_deref(), budgets.resolved_bytes)+ .with_edit_budget(budgets.edit_bytes);let mut observed = Observed::default();match (event.kind, event.source) {(Some(FULL_SNAPSHOT), _) => {@@ -1551,9 +1498,25 @@ impl DomInspector {.unwrap_or_default()}- /// Every remote image URL the events since the last call hold, for the writer to replace.- pub fn take_remote_urls(&mut self) -> RemoteUrls {- std::mem::take(&mut self.urls.remote)+ /// The edits the last event needs, whether it held a slot the writer could not find again, and the bytes its edits took from the flush's edit budget.+ pub fn take_edits(&mut self) -> (Vec<Edit>, bool, usize) {+ (+ std::mem::take(&mut self.urls.edits),+ std::mem::take(&mut self.urls.unplaced),+ std::mem::take(&mut self.urls.edit_bytes_taken),+ )+ }++ pub fn take_declined(&mut self) -> Vec<DeclinedUrl> {+ self.urls.declined_bytes = 0;+ std::mem::take(&mut self.urls.declined)+ .into_iter()+ .map(|(url, (kind, decline))| DeclinedUrl { url, kind, decline })+ .collect()+ }++ pub fn take_url_limit(&mut self) -> Option<UrlLimit> {+ self.urls.limit.take()}/// Whether a full snapshot of any window of the session decoded in this process.@@ -1677,6 +1640,7 @@ fn observe_mutation(let mut undecoded = false;let mut walk = Walk {resolver,+ buffers: Buffers::new(inner, expansion),..Walk::default()};let mut adds = Vec::new();@@ -1736,7 +1700,7 @@ fn observe_mutation(let walked: HashSet<i64> = walk.ids.iter().map(|&(id, _)| id).collect();for (index, item) in items(attributes).enumerate() {if index == MAX_ATTRIBUTE_ITEMS {- walk.image_urls.remote.limit = Some(UrlLimit::AttributeItems);+ walk.image_urls.limit = Some(UrlLimit::AttributeItems);break;}let [id, values] = fields(attributes.bytes, item, [b"id", b"attributes"]);@@ -1856,31 +1820,34 @@ fn style_sheet_urls(mut resolver: Resolver<'_>,urls: &mut Urls,) {- let mut push = |found: CssUrl<'_>| urls.push_css(found);+ let buffers = Buffers::new(inner, expansion);let resolver = &mut resolver;+ let mut read_rule = |bytes: &[u8], span: Option<Span>, urls: &mut Urls| {+ if let (Some(span), Some(rule)) = (span, text(bytes, span)) {+ urls.read_css(&buffers, (bytes, span), &rule, None, resolver);+ }+ };match source {Some(STYLE_SHEET_RULE) => {let [adds, replace, replace_sync] =fields(inner, data, [b"adds", b"replace", b"replaceSync"]);if let Some(adds) = adds.and_then(|adds| decoded_list(inner, adds, expansion)) {for item in items(adds) {- if let Some(rule) = text(adds.bytes, field(adds.bytes, item, b"rule")) {- css::urls(&rule, resolver, &mut push);- }+ read_rule(adds.bytes, field(adds.bytes, item, b"rule"), urls);}}for sheet in [replace, replace_sync] {- if let Some(sheet) = text(inner, sheet) {- css::urls(&sheet, resolver, &mut push);- }+ read_rule(inner, sheet, urls);}}Some(STYLE_DECLARATION) => {let set = field(inner, data, b"set");let [property, value] =set.map_or([None; 2], |set| fields(inner, set, [b"property", b"value"]));- if let (Some(property), Some(value)) = (text(inner, property), text(inner, value)) {- css::declaration_image_urls(&property, &value, resolver, &mut push);+ if let (Some(property), Some(at), Some(value)) =+ (text(inner, property), value, text(inner, value))+ {+ urls.read_css(&buffers, (inner, at), &value, Some(&property), resolver);}}Some(ADOPTED_STYLE_SHEET) => {@@ -1898,9 +1865,7 @@ fn style_sheet_urls(bytes: inner,span: rules,}) {- if let Some(rule) = text(inner, field(inner, rule, b"rule")) {- css::urls(&rule, resolver, &mut push);- }+ read_rule(inner, field(inner, rule, b"rule"), urls);}}}@@ -1991,7 +1956,7 @@ mod tests {&message.events[0],"w",None,- None,+ FlushBudgets::default(),&mut stats,);// The srcset has no 1x candidate, so the img shows its `src`, which HTML adds as one.@@ -2038,7 +2003,14 @@ mod tests {let mut dom = DomInspector::default();let mut stats = BlockStats::default();for event in &message.events {- dom.observe(&message.inner, event, "w", None, None, &mut stats);+ dom.observe(+ &message.inner,+ event,+ "w",+ None,+ FlushBudgets::default(),+ &mut stats,+ );}assert_eq!(stats.events_before_first_full_snapshot, 1);assert_eq!(stats.orphan_mutations, 2);@@ -2063,13 +2035,65 @@ mod tests {event,window,Some(PAGE_URL),- None,+ FlushBudgets::default(),&mut stats,);}(dom, stats)}+ /// What the writer replaces, read back from the edits: each URL with its kind, and each function a rewrite replaces whole.+ #[derive(Default)]+ struct Named {+ urls: Vec<(String, RemoteKind)>,+ rewrites: Vec<(String, Vec<RemoteUrl>)>,+ }++ fn named(dom: &mut DomInspector, message: &crate::scan::message::Message) -> Named {+ let (edits, unplaced, _) = dom.take_edits();+ assert!(!unplaced);+ let mut named = Named::default();+ for edit in edits {+ let bytes = match edit.source {+ super::super::edits::EditSource::Payload => message.inner.clone(),+ super::super::edits::EditSource::Field(span) => message+ .events+ .iter()+ .find_map(|event| {+ cv::expansion(&message.inner, event)+ .json(span)+ .flatten()+ .map(<[u8]>::to_vec)+ })+ .unwrap(),+ };+ let text = scan::unescape(&bytes, edit.value).unwrap();+ match edit.slot {+ Slot::Url => named.urls.push((text.trim().to_owned(), RemoteKind::Image)),+ Slot::Srcset(_) => srcset_urls(&text, &mut |url| {+ if is_remote(url) {+ named.urls.push((url.to_owned(), RemoteKind::Image));+ }+ }),+ Slot::Css(edit) => {+ for piece in edit.pieces {+ let urls = piece.parts.iter().filter_map(|part| match part {+ css::CssPart::Url(url) => Some(url.clone()),+ css::CssPart::Text(_) => None,+ });+ if piece.parts.len() == 1 {+ named.urls.extend(urls.map(|url| (url.url, url.kind)));+ } else {+ let urls = urls.collect();+ named.rewrites.push((text[piece.range].to_owned(), urls));+ }+ }+ }+ }+ }+ named+ }+#[test]fn removes_a_whole_subtree_and_follows_a_move() {let message = events(json!([@@ -2240,24 +2264,15 @@ mod tests {]));let (mut dom, _) = observe_all(&message, "w");// A hidden or one-pixel img is not fetched, but the block holds its URL, so the writer replaces it too.- let mut remote = dom.take_remote_urls();- let mut rewrites: Vec<String> = remote- .take_css_rewrites()- .into_iter()- .map(|rewrite| rewrite.text)- .collect();- let remote: Vec<String> = remote- .into_urls()- .into_iter()- .map(|remote| remote.url)- .collect();+ let named = named(&mut dom, &message);+ let mut rewrites: Vec<String> = named.rewrites.into_iter().map(|(text, _)| text).collect();+ let remote: Vec<String> = named.urls.into_iter().map(|(url, _)| url).collect();for name in ["hidden","display-none","pixel","shown-later","lazy",- "set-1x","w_640,q_auto/srcset-comma","srcset-second",] {@@ -2364,11 +2379,11 @@ mod tests {.to_vec();expected.insert(0, "https://app.example.com/fonts/serif.woff2".to_owned());assert_eq!(fetched, expected);- let mut named = dom.take_remote_urls();+ let named = named(&mut dom, &message);let mut resolved: Vec<RemoteUrl> = named- .take_css_rewrites()+ .rewrites.into_iter()- .filter_map(|rewrite| rewrite.resolved)+ .flat_map(|(_, urls)| urls).collect();resolved.sort_by(|a, b| a.url.cmp(&b.url));assert_eq!(@@ -2384,11 +2399,8 @@ mod tests {},]);- let mut remote: Vec<(String, RemoteKind)> = named- .into_urls()- .into_iter()- .map(|remote| (remote.url, remote.kind))- .collect();+ // Each slot names its URL by its own kind, so a URL in two slots has two references.+ let mut remote = named.urls;remote.sort();assert_eq!(remote,@@ -2396,11 +2408,13 @@ mod tests {(font("brand.eot?#iefix"), RemoteKind::Font),(font("brand.woff2"), RemoteKind::Font),(font("icons.woff"), RemoteKind::Font),+ (font("logo.png"), RemoteKind::Font),(font("logo.png"), RemoteKind::Image),(font("open-sans.woff2"),RemoteKind::BundledFont("open-sans")),+ (font("shared.woff2"), RemoteKind::BundledFont("inter")),(font("shared.woff2"), RemoteKind::Font),]);@@ -2426,66 +2440,14 @@ mod tests {}#[test]- fn stops_naming_remote_urls_past_a_block_limit() {- let url = |i: usize, len: usize| {- let mut url = format!("https://cdn.example.com/{i}/");- url.extend(std::iter::repeat_n('a', len - url.len()));- url- };- let limit = |urls: &mut dyn Iterator<Item = String>| {- let mut remote = RemoteUrls::default();- urls.for_each(|u| remote.insert(&u, RemoteKind::Image));- remote.limit- };- let longest = 8 * 1024;- let most_long = MAX_BLOCK_REMOTE_URL_BYTES / longest;- for (urls, expected) in [- (vec![url(0, longest)], None),- // A URL of any length is named, and the policy declines it.- (vec![url(0, longest + 1)], None),- (- (0..MAX_BLOCK_REMOTE_URLS).map(|i| url(i, 40)).collect(),- None,- ),- (- (0..=MAX_BLOCK_REMOTE_URLS).map(|i| url(i, 40)).collect(),- Some(UrlLimit::UrlCount),- ),- ((0..most_long).map(|i| url(i, longest)).collect(), None),- (- (0..=most_long).map(|i| url(i, longest)).collect(),- Some(UrlLimit::UrlBytes),- ),- ] {- assert_eq!(limit(&mut urls.into_iter()), expected);- }- let mut remote = RemoteUrls::default();- (0..=MAX_BLOCK_FONT_URLS).for_each(|i| remote.insert(&url(i, 40), RemoteKind::Font));- remote.insert(&url(MAX_BLOCK_FONT_URLS + 1, 40), RemoteKind::Image);- assert_eq!(remote.limit, None);- assert_eq!(remote.url_limit(), Some(UrlLimit::FontUrlCount));- assert_eq!(remote.into_urls().len(), MAX_BLOCK_FONT_URLS + 1);- let mut remote = RemoteUrls::default();- remote.insert(&url(0, MAX_BLOCK_FONT_URL_BYTES + 1), RemoteKind::Font);- assert_eq!(remote.url_limit(), Some(UrlLimit::FontUrlBytes));-- // Srcset values share the byte budget, past which the writer falls back to replacing each URL.- let mut remote = RemoteUrls::default();- // A value counts its raw and its unescaped form.- (0..=most_long).for_each(|i| {- let value = url(i, longest);- remote.insert_srcset(value.as_bytes(), &value, SrcsetSlot::Alone)- });- assert_eq!(remote.take_srcsets().len(), most_long / 2);-- // A mutation past the attribute cap is not read to its end.+ fn stops_reading_a_mutation_past_the_attribute_cap() {let attributes = |count: usize| {let items = vec![r#"{"id":1,"attributes":{}}"#; count].join(",");let inner = format!(r#"{{"properties":{{"$session_id":"s","$snapshot_items":[{{"type":3,"timestamp":2,"data":{{"source":0,"attributes":[{items}]}}}}]}}}}"#);let message = read_message(json!({ "data": inner }).to_string().into_bytes()).unwrap();- observe_all(&message, "w").0.take_remote_urls().limit+ observe_all(&message, "w").0.take_url_limit()};assert_eq!(attributes(MAX_ATTRIBUTE_ITEMS), None);assert_eq!(@@ -2494,6 +2456,33 @@ mod tests {);}+ #[test]+ fn notes_each_refused_url_once_within_the_declined_budget() {+ let declined = |srcset: String| {+ let img = json!({"parentId": 1, "node": element(2, "img", json!({"srcset": srcset}), json!([]))});+ let message = events(json!([mutation(+ json!([img]),+ json!([]),+ json!([]),+ json!([])+ )]));+ let (mut dom, _) = observe_all(&message, "w");+ (dom.take_declined().len(), dom.take_url_limit())+ };+ let repeated = vec!["https://cdn.example.com/a.png?token=x 1x"; 10_000].join(", ");+ assert_eq!(declined(repeated), (1, None));+ let distinct = (0..2_000)+ .map(|i| {+ format!(+ "https://cdn.example.com/{i}/{}.png?token=x {i}w",+ "a".repeat(600)+ )+ })+ .collect::<Vec<_>>()+ .join(", ");+ assert_eq!(declined(distinct).1, Some(UrlLimit::DeclinedUrls));+ }+#[test]fn caps_distinct_remote_image_urls_per_message() {let img = |i: i64, src: String| json!({"parentId": 1, "node": element(i, "img", json!({"src": src}), json!([]))});@@ -2510,8 +2499,8 @@ mod tests {let (mut dom, _) = observe_all(&message, "w");let found = dom.take_image_urls();assert_eq!(found.len(), MAX_IMAGE_URLS);- // The cap limits the fetch jobs only. The writer replaces every remote URL in the block.- assert_eq!(dom.take_remote_urls().into_urls().len(), 601);+ // The cap limits the fetch jobs only. The writer replaces the URL of every remote slot in the block.+ assert_eq!(named(&mut dom, &message).urls.len(), 1200);assert_eq!(found[..2],[@@ -2552,7 +2541,7 @@ mod tests {&message.events[0],&window.to_string(),None,- None,+ FlushBudgets::default(),&mut stats,);}@@ -2562,7 +2551,7 @@ mod tests {&message.events[1],"0",None,- None,+ FlushBudgets::default(),&mut stats,);assert!(stats.node_cap_reached);@@ -2574,7 +2563,7 @@ mod tests {&message.events[1],&other,None,- None,+ FlushBudgets::default(),&mut stats,);}@@ -2584,7 +2573,7 @@ mod tests {&message.events[1],"0",None,- None,+ FlushBudgets::default(),&mut stats,);assert!(stats.node_cap_reached);diff --git a/rust/ai-research/replay-core-v4/src/inspect/edits.rs b/rust/ai-research/replay-core-v4/src/inspect/edits.rsnew file mode 100644index 000000000000..1168ebbae565--- /dev/null+++ b/rust/ai-research/replay-core-v4/src/inspect/edits.rs@@ -0,0 +1,86 @@+//! Where the writer changes a block's bytes, by position, so that it copies each event once and searches it for nothing.++use std::ops::Range;++use super::css::CssEdit;+use super::srcset::SrcsetSlot;+use crate::bytescan::Span;+use crate::scan::cv::Expansion;++#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]+pub enum EditSource {+ Payload,+ /// A decoded `cv` string, by its span in the payload. The writer decodes it again to the same bytes.+ Field(Span),+}++#[derive(Debug, Clone, PartialEq, Eq)]+pub enum Slot {+ /// A URL attribute. The writer writes the URL's reference, or its decline marker.+ Url,+ Srcset(SrcsetSlot),+ Css(CssEdit),+}++#[derive(Debug, Clone, PartialEq, Eq)]+pub struct Edit {+ pub source: EditSource,+ /// The JSON string the slot holds.+ pub value: Span,+ pub slot: Slot,+}++/// The edits of one event, by its position in its block.+#[derive(Debug, Clone, Default, PartialEq, Eq)]+pub struct EventEdits {+ pub event: u32,+ /// Sorted by source and position, one per JSON string.+ pub edits: Vec<Edit>,+}++impl EventEdits {+ pub fn new(event: u32, mut edits: Vec<Edit>) -> Self {+ edits.sort_by_key(|edit| (edit.source, edit.value.0));+ edits.dedup_by_key(|edit| (edit.source, edit.value));+ Self { event, edits }+ }+}++/// The buffers an event's walk reads, by address, so that a slot names the one it is in.+#[derive(Debug, Clone, Default)]+pub struct Buffers {+ payload: Range<usize>,+ fields: Vec<(Range<usize>, Span)>,+}++fn addresses(bytes: &[u8]) -> Range<usize> {+ let start = bytes.as_ptr() as usize;+ start..start + bytes.len()+}++impl Buffers {+ pub fn new(payload: &[u8], expansion: &Expansion) -> Self {+ let fields = match expansion {+ Expansion::Decoded(fields) => fields+ .iter()+ .filter_map(|field| Some((addresses(field.json.as_deref()?), field.span)))+ .collect(),+ _ => Vec::new(),+ };+ Self {+ payload: addresses(payload),+ fields,+ }+ }++ pub fn source_of(&self, bytes: &[u8]) -> Option<EditSource> {+ let bytes = addresses(bytes);+ if bytes == self.payload {+ return Some(EditSource::Payload);+ }+ self.fields+ .iter()+ .find(|(field, _)| *field == bytes)+ .map(|&(_, span)| EditSource::Field(span))+ }+}diff --git a/rust/ai-research/replay-core-v4/src/inspect/mod.rs b/rust/ai-research/replay-core-v4/src/inspect/mod.rsindex a30406d8786d..1f16854a7587 100644--- a/rust/ai-research/replay-core-v4/src/inspect/mod.rs+++ b/rust/ai-research/replay-core-v4/src/inspect/mod.rs@@ -3,6 +3,7 @@pub mod clicks;pub mod css;pub mod dom;+pub mod edits;pub mod events;pub mod features;pub mod rrweb;diff --git a/rust/ai-research/replay-core-v4/src/inspect/session.rs b/rust/ai-research/replay-core-v4/src/inspect/session.rsindex 2670708c2f63..06acf0d16519 100644--- a/rust/ai-research/replay-core-v4/src/inspect/session.rs+++ b/rust/ai-research/replay-core-v4/src/inspect/session.rs@@ -1,11 +1,14 @@//! Everything the recorder learns about one session, block by block.-use std::sync::atomic::AtomicUsize;+use std::collections::HashMap;+use std::sync::atomic::{AtomicUsize, Ordering};use std::sync::Arc;use super::clicks::{ClickKind, ClickTracker};-use super::css::CssRewrite;-use super::dom::{DomInspector, RemoteUrl, RemoteUrls, Srcset, UrlLimit};+use super::dom::{+ DeclinedUrl, DomInspector, FlushBudgets, RemoteKind, UrlLimit, MAX_BLOCK_DECLINED_URL_BYTES,+};+use super::edits::EventEdits;use super::events::{href_from, number, Activity, DataFields, EventCounter};use super::features::{BlockFeatures, FeatureRecorder, SurfacingFeatures};use super::rrweb::{@@ -13,6 +16,7 @@ use super::rrweb::{};use super::stats::{BlockStats, MAX_WINDOW_IDS};use crate::scan::message::{Event, Message};+use crate::url_policy::Decline;/// Moments kept per block, so a long block cannot grow its index row without bound.pub const MAX_MOMENTS_PER_BLOCK: usize = 256;@@ -53,9 +57,15 @@ pub struct BlockInspection {moments: Vec<Moment>,image_urls: Vec<String>,font_urls: Vec<String>,- remote_urls: RemoteUrls,+ url_limit: Option<UrlLimit>,+ edits: Vec<EventEdits>,+ unplaced: bool,+ declined: HashMap<String, (RemoteKind, Decline)>,+ declined_bytes: usize,events: u32,resolve_budget: Option<Arc<AtomicUsize>>,+ edit_budget: Option<Arc<AtomicUsize>>,+ edit_bytes_taken: usize,}/// What the inspector learned about one block.@@ -67,25 +77,40 @@ pub struct Inspected {/// Remote image URLs the block's DOM referenced, for the image fetcher.pub image_urls: Vec<String>,pub font_urls: Vec<String>,- /// Every remote URL in an image slot or `@font-face` rule of the block, which the writer replaces with its reference.- pub remote_urls: Vec<RemoteUrl>,- /// Set when `remote_urls` stops short, so that the block must not be stored.+ /// Set when the inspector could not read the block in full, so that the block must not be stored.pub url_limit: Option<UrlLimit>,- /// The image-slot srcset values of the block, which the writer rebuilds from the candidates it keeps.- pub srcsets: Vec<Srcset>,- pub css_rewrites: Vec<CssRewrite>,+ /// Where the writer changes the block's events, in event order.+ pub edits: Vec<EventEdits>,+ /// The inspector read a slot that the writer could not find again, so the block must not be stored.+ pub unplaced: bool,+ /// Spellings of named URLs that the policy refuses, by the largest kind each was named as.+ pub declined: Vec<DeclinedUrl>,}impl BlockInspection {- /// A block whose relative CSS URLs draw their resolved bytes from `budget`, which other blocks can share.- pub fn with_resolve_budget(budget: Arc<AtomicUsize>) -> Self {+ /// A block whose resolved CSS URLs and edits draw their bytes from budgets that other blocks can share.+ pub fn with_flush_budgets(+ resolved_bytes: Arc<AtomicUsize>,+ edit_bytes: Arc<AtomicUsize>,+ ) -> Self {Self {- resolve_budget: Some(budget),+ resolve_budget: Some(resolved_bytes),+ edit_budget: Some(edit_bytes),..Self::default()}}- pub fn finish(mut self) -> Inspected {+ fn release_edits(&mut self) {+ if let Some(budget) = &self.edit_budget {+ budget.fetch_add(+ std::mem::take(&mut self.edit_bytes_taken),+ Ordering::Relaxed,+ );+ }+ self.edits = Vec::new();+ }++ pub fn finish(self) -> Inspected {let features = self.features.finish();let mut stats = self.counter.finish();stats.surfacing = SurfacingFeatures::from(&features);@@ -95,10 +120,14 @@ impl BlockInspection {moments: self.moments,image_urls: self.image_urls,font_urls: self.font_urls,- url_limit: self.remote_urls.url_limit(),- srcsets: self.remote_urls.take_srcsets(),- css_rewrites: self.remote_urls.take_css_rewrites(),- remote_urls: self.remote_urls.into_urls(),+ url_limit: self.url_limit,+ edits: self.edits,+ unplaced: self.unplaced,+ declined: self+ .declined+ .into_iter()+ .map(|(url, (kind, decline))| DeclinedUrl { url, kind, decline })+ .collect(),}}}@@ -186,12 +215,36 @@ impl SessionInspector {event,window,page_url.as_deref(),- block.resolve_budget.as_deref(),+ FlushBudgets {+ resolved_bytes: block.resolve_budget.as_deref(),+ edit_bytes: block.edit_budget.as_deref(),+ },&mut block.counter.stats,);block.image_urls.extend(self.dom.take_image_urls());block.font_urls.extend(self.dom.take_font_urls());- block.remote_urls.extend(self.dom.take_remote_urls());+ block.url_limit = block.url_limit.or(self.dom.take_url_limit());+ let (edits, unplaced, edit_bytes) = self.dom.take_edits();+ block.edit_bytes_taken += edit_bytes;+ block.unplaced |= unplaced;+ for declined in self.dom.take_declined() {+ if let Some((kind, _)) = block.declined.get_mut(&declined.url) {+ *kind = (*kind).max(declined.kind);+ } else if block.declined_bytes + declined.url.len() > MAX_BLOCK_DECLINED_URL_BYTES {+ block.url_limit = block.url_limit.or(Some(UrlLimit::DeclinedUrls));+ } else {+ block.declined_bytes += declined.url.len();+ block+ .declined+ .insert(declined.url, (declined.kind, declined.decline));+ }+ }+ if block.url_limit.is_none() && !edits.is_empty() {+ block.edits.push(EventEdits::new(index, edits));+ }+ if block.url_limit.is_some() {+ block.release_edits();+ }if observed.full_snapshot {self.cold = false;}@@ -331,6 +384,22 @@ mod tests {#[test]fn resolves_relative_css_against_its_own_window() {+ let resolved = |edits: &[EventEdits]| -> Vec<String> {+ edits+ .iter()+ .flat_map(|event| &event.edits)+ .filter_map(|edit| match &edit.slot {+ super::super::edits::Slot::Css(css) => Some(css),+ _ => None,+ })+ .flat_map(|css| &css.pieces)+ .flat_map(|piece| &piece.parts)+ .filter_map(|part| match part {+ super::super::css::CssPart::Url(url) => Some(url.url.clone()),+ super::super::css::CssPart::Text(_) => None,+ })+ .collect()+ };let mut session = SessionInspector::default();let mut block = BlockInspection::default();let meta = |host: &str| json!([{"type": 4, "timestamp": 1, "data": {"href": format!("https://{host}/shop/item")}}]);@@ -347,13 +416,55 @@ mod tests {"https://b.example.com/img/a.png"]);- assert!(inspected.css_rewrites.is_empty());+ assert_eq!(+ resolved(&inspected.edits),+ [+ "https://a.example.com/img/a.png",+ "https://b.example.com/img/a.png"+ ]+ );let mut block = BlockInspection::default();let rule = json!([{"type": 3, "timestamp": 3, "data": {"source": 8, "id": 3, "replaceSync": ".b{background:url(/img/b.png)}"}}]);session.observe_message(&mut block, &message_in(Some("unseen"), rule.clone()));session.observe_message(&mut block, &message_in(Some("a"), rule));- assert!(block.finish().css_rewrites.is_empty());+ // A window whose page is not known leaves its relative URL as the page wrote it.+ assert_eq!(+ resolved(&block.finish().edits),+ ["https://a.example.com/img/b.png"]+ );+ }++ #[test]+ fn drops_a_block_whose_edits_the_flush_cannot_hold() {+ let sheet: String = (0..100)+ .map(|i| format!(".a{i}{{background:url(https://cdn.example.com/{i}.png)}}"))+ .collect();+ let styled = message(+ json!([{"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": [+ {"type": 2, "tagName": "style", "attributes": {"_cssText": sheet}, "childNodes": [], "id": 2},+ ]}}}]),+ );+ let inspect_within = |edit_budget: &Arc<AtomicUsize>| {+ let mut block = BlockInspection::with_flush_budgets(+ Arc::new(AtomicUsize::new(usize::MAX)),+ edit_budget.clone(),+ );+ SessionInspector::default().observe_message(&mut block, &styled);+ block.finish()+ };+ let unbounded = Arc::new(AtomicUsize::new(usize::MAX));+ inspect_within(&unbounded);+ let held = usize::MAX - unbounded.load(Ordering::Relaxed);+ let edit_budget = Arc::new(AtomicUsize::new(held + held / 2));+ let blocks = [inspect_within(&edit_budget), inspect_within(&edit_budget)];+ assert_eq!((blocks[0].url_limit, blocks[0].edits.len()), (None, 1));+ assert_eq!(+ (blocks[1].url_limit, blocks[1].edits.len()),+ (Some(UrlLimit::Edits), 0)+ );+ // The dropped block stopped part way through its style sheet and gave back what it took.+ assert_eq!(edit_budget.load(Ordering::Relaxed), held / 2);}#[test]diff --git a/rust/ai-research/replay-core-v4/src/scan/cv.rs b/rust/ai-research/replay-core-v4/src/scan/cv.rsindex 09c91e2273af..077c671d64fe 100644--- a/rust/ai-research/replay-core-v4/src/scan/cv.rs+++ b/rust/ai-research/replay-core-v4/src/scan/cv.rs@@ -7,6 +7,7 @@ use std::io::Read;use crate::bytescan::{self as scan, Span};use crate::compression;+use crate::inspect::edits::EditSource;use serde::de::IgnoredAny;use super::canvas::{self, CanvasImage};@@ -411,11 +412,19 @@ pub fn extract_event_images(inner: &[u8],event: &Event,sink: &mut ImageSink<'_>,+ copy: Option<&mut Copier<'_>>,+ before_images: Option<&mut BeforeImages<'_>>,) -> Option<Vec<u8>> {- let (mut stored, takes_images) = match &*expansion(inner, event) {- Expansion::Failed | Expansion::Skipped => (None, false),- Expansion::Decoded(fields) => (Some(expanded(inner, event, fields)), true),- Expansion::Plain => (move_out_canvas_images(inner, event, sink), true),+ let (mut stored, takes_images) = match (&*expansion(inner, event), copy) {+ (Expansion::Failed | Expansion::Skipped, _) => (None, false),+ (Expansion::Decoded(fields), copy) => (Some(expanded(inner, event, fields, copy)), true),+ // Only a canvas event holds canvas images, and only a DOM event holds slots that the writer changes.+ (Expansion::Plain, Some(copy)) => {+ let mut out = Vec::with_capacity(event.span.1 - event.span.0);+ copy(EditSource::Payload, inner, event.span, &mut out);+ (Some(out), true)+ }+ (Expansion::Plain, None) => (move_out_canvas_images(inner, event, sink), true),};let arrived = &inner[event.span.0..event.span.1];if let Cow::Owned(one_line) = without_line_breaks(stored.as_deref().unwrap_or(arrived)) {@@ -424,6 +433,11 @@ pub fn extract_event_images(if let Cow::Owned(canonical) = scan::canonical_strings(stored.as_deref().unwrap_or(arrived)) {stored = Some(canonical);}+ if let Some(changed) =+ before_images.and_then(|before| before(stored.as_deref().unwrap_or(arrived)))+ {+ stored = Some(changed);+ }if takes_images {if let Cow::Owned(extracted) = extract_images(stored.as_deref().unwrap_or(arrived), sink) {stored = Some(extracted);@@ -516,7 +530,18 @@ fn write_object(out.push(b'}');}-fn expanded(inner: &[u8], event: &Event, fields: &[Decoded]) -> Vec<u8> {+/// Copies a range of the payload or of a decoded string into the stored event, with what the writer changes there.+pub type Copier<'c> = dyn FnMut(EditSource, &[u8], Span, &mut Vec<u8>) + 'c;++/// Changes the stored event once its strings are canonical, before moving its inline images out changes them.+pub type BeforeImages<'c> = dyn FnMut(&[u8]) -> Option<Vec<u8>> + 'c;++fn expanded(+ inner: &[u8],+ event: &Event,+ fields: &[Decoded],+ mut copy: Option<&mut Copier<'_>>,+) -> Vec<u8> {let mut out = Vec::with_capacity(event.span.1 - event.span.0);let json = |span: Span| {fields@@ -524,6 +549,11 @@ fn expanded(inner: &[u8], event: &Event, fields: &[Decoded]) -> Vec<u8> {.find(|f| f.span == span).map(|f| f.json.as_deref())};+ let mut put = |source: EditSource, bytes: &[u8], range: Span, out: &mut Vec<u8>| match &mut copy+ {+ Some(copy) => copy(source, bytes, range, out),+ None => out.extend_from_slice(&bytes[range.0..range.1]),+ };write_object(inner, event.span, &mut out, |key, value, out| {if key == b"cv" {return false;@@ -532,22 +562,24 @@ fn expanded(inner: &[u8], event: &Event, fields: &[Decoded]) -> Vec<u8> {if let Some(decoded) = json(value) {// A full snapshot whose data is empty has no data, as in the player.let Some(decoded) = decoded else { return false };- out.extend_from_slice(decoded);+ put(EditSource::Field(value), decoded, (0, decoded.len()), out);return true;}if scan::is_object(inner, value) {write_object(inner, value, out, |_, value, out| {match json(value) {- Some(Some(decoded)) => out.extend_from_slice(decoded),+ Some(Some(decoded)) => {+ put(EditSource::Field(value), decoded, (0, decoded.len()), out)+ }Some(None) => return false,- None => out.extend_from_slice(&inner[value.0..value.1]),+ None => put(EditSource::Payload, inner, value, out),}true});return true;}}- out.extend_from_slice(&inner[value.0..value.1]);+ put(EditSource::Payload, inner, value, out);true});out@@ -612,8 +644,10 @@ mod tests {.events.iter().map(|e| {- serde_json::from_slice(&extract_event_images(&message.inner, e, &mut sink).unwrap())- .unwrap()+ serde_json::from_slice(+ &extract_event_images(&message.inner, e, &mut sink, None, None).unwrap(),+ )+ .unwrap()}).collect();@@ -664,7 +698,7 @@ mod tests {.events.iter().map(|e| {- let stored = extract_event_images(&message.inner, e, &mut sink)+ let stored = extract_event_images(&message.inner, e, &mut sink, None, None).unwrap_or_else(|| message.inner[e.span.0..e.span.1].to_vec());serde_json::from_slice(&stored).unwrap()})@@ -708,7 +742,7 @@ mod tests {);let mut sink = sink();assert_eq!(- extract_event_images(&message.inner, &message.events[1], &mut sink),+ extract_event_images(&message.inner, &message.events[1], &mut sink, None, None),None);@@ -814,7 +848,7 @@ mod tests {let stored: Vec<Vec<u8>> = message.events.iter()- .map(|e| extract_event_images(&message.inner, e, &mut sink).unwrap())+ .map(|e| extract_event_images(&message.inner, e, &mut sink, None, None).unwrap()).collect();assert!(stored.iter()@@ -845,7 +879,9 @@ mod tests {let mut message =message_with(json!([{"type": 2, "timestamp": 1, "data": compressed, "cv": "2024-10"}]));message.decode_compressed();- let stored = extract_event_images(&message.inner, &message.events[0], &mut sink()).unwrap();+ let stored =+ extract_event_images(&message.inner, &message.events[0], &mut sink(), None, None)+ .unwrap();assert!(String::from_utf8_lossy(&stored).contains(r#""textContent":"a"#));assert_eq!(serde_json::from_slice::<Value>(&stored).unwrap(),@@ -870,7 +906,10 @@ mod tests {]));let mut sink = sink();for event in &message.events {- assert_eq!(extract_event_images(&message.inner, event, &mut sink), None);+ assert_eq!(+ extract_event_images(&message.inner, event, &mut sink, None, None),+ None+ );}assert!(sink.images.is_empty());}@@ -979,8 +1018,9 @@ mod tests {for (event, pointer, outcome) in cases {let message = message_with(json!([event]));let mut sink = sink();- let stored = extract_event_images(&message.inner, &message.events[0], &mut sink)- .map(|stored| serde_json::from_slice::<Value>(&stored).unwrap());+ let stored =+ extract_event_images(&message.inner, &message.events[0], &mut sink, None, None)+ .map(|stored| serde_json::from_slice::<Value>(&stored).unwrap());let replacement = match outcome {Outcome::Kept => {assert_eq!(stored, None, "{event}");diff --git a/rust/ai-research/replay-core-v4/src/url_policy.rs b/rust/ai-research/replay-core-v4/src/url_policy.rsindex 8475d397f528..e3be18721cf7 100644--- a/rust/ai-research/replay-core-v4/src/url_policy.rs+++ b/rust/ai-research/replay-core-v4/src/url_policy.rs@@ -348,6 +348,21 @@ impl Decline {}}+/// The URL as a reference names it, or why the policy declines it. It checks the host and path again as `Url` serializes them.+pub fn checked_url(raw: &str) -> Result<CanonicalUrl, Decline> {+ let url = try_canonicalize(raw)?;+ let path = Url::parse(&url.fetch)+ .map(|u| u.path().to_owned())+ .unwrap_or_default();+ if !is_public_host(&url.host) {+ return Err(Decline::NonPublicHost);+ }+ if is_tracking_beacon(&url.host, &path) {+ return Err(Decline::TrackingBeacon);+ }+ Ok(url)+}+/// Canonicalize a remote image URL into its fetch and dedup forms, or say why it was refused.pub fn canonicalize(raw: &str) -> Option<CanonicalUrl> {try_canonicalize(raw).ok()diff --git a/rust/ai-research/replay-recorder-v4/Cargo.toml b/rust/ai-research/replay-recorder-v4/Cargo.tomlindex d10574b8ab99..41b1cb956849 100644--- a/rust/ai-research/replay-recorder-v4/Cargo.toml+++ b/rust/ai-research/replay-recorder-v4/Cargo.toml@@ -43,7 +43,6 @@ thiserror = { workspace = true }tokio = { workspace = true }tracing = { workspace = true }tracing-subscriber = { workspace = true }-url = { workspace = true }[dev-dependencies]aes-gcm = "0.10"diff --git a/rust/ai-research/replay-recorder-v4/src/batch.rs b/rust/ai-research/replay-recorder-v4/src/batch.rsindex d37789d69bc5..acc4a52d4b57 100644--- a/rust/ai-research/replay-recorder-v4/src/batch.rs+++ b/rust/ai-research/replay-recorder-v4/src/batch.rs@@ -8,8 +8,8 @@ use std::time::{Duration, Instant};use futures::stream::{self, StreamExt};use metrics::counter;use replay_core_v4::compression::{GZIP_MAGIC, MAX_DECOMPRESSED_BYTES};-use replay_core_v4::inspect::css::CssRewrite;-use replay_core_v4::inspect::dom::{RemoteUrl, Srcset, UrlLimit};+use replay_core_v4::inspect::dom::{DeclinedUrl, UrlLimit};+use replay_core_v4::inspect::edits::EventEdits;use replay_core_v4::inspect::session::{BlockInspection, Moment, SessionInspector};use replay_core_v4::inspect::stats::BlockStats;use replay_core_v4::scan::cv::{Expansion, MESSAGE_DECODE_BUDGET};@@ -98,13 +98,11 @@ pub struct Block {/// Remote image URLs the block's DOM referenced.pub image_urls: Vec<String>,pub font_urls: Vec<String>,- /// Every remote URL in an image slot or `@font-face` rule of the block, which the writer replaces with its reference.- pub remote_urls: Vec<RemoteUrl>,- /// Set when the inspector stopped short of naming every remote URL, so that the block must not be stored.+ /// Set when the inspector could not read the block in full, so that the block must not be stored.pub url_limit: Option<UrlLimit>,- /// The image-slot srcset values of the block, which the writer rebuilds from the candidates it keeps.- pub srcsets: Vec<Srcset>,- pub css_rewrites: Vec<CssRewrite>,+ pub edits: Vec<EventEdits>,+ pub declined: Vec<DeclinedUrl>,+ pub unplaced: bool,/// The distinct pages of the block's surfacing features, which the writer stores as fingerprints.pub visited_urls: Vec<String>,/// The earliest Kafka timestamp of the block's messages, for the capture watermark of the index topic.@@ -129,11 +127,16 @@ const FLUSH_EVENTS_BUDGET: usize = 1024 * 1024 * 1024;/// The bytes of resolved CSS URLs one flush keeps. Each carries its page's origin, which many short URLs can outgrow.const FLUSH_RESOLVED_BUDGET: usize = 64 * 1024 * 1024;+/// The memory the blocks of one flush hold for their edits until they are written. A short repeated `url()` in a+/// compressed style sheet takes more memory as an edit than it expands to, so a block past this is dropped.+const FLUSH_EDIT_BUDGET: usize = 256 * 1024 * 1024;+struct DecodeBudget {kept: AtomicUsize,expanded: AtomicUsize,events: AtomicUsize,resolved: Arc<AtomicUsize>,+ edits: Arc<AtomicUsize>,}impl DecodeBudget {@@ -143,6 +146,7 @@ impl DecodeBudget {expanded: AtomicUsize::new(expanded),events: AtomicUsize::new(FLUSH_EVENTS_BUDGET),resolved: Arc::new(AtomicUsize::new(FLUSH_RESOLVED_BUDGET)),+ edits: Arc::new(AtomicUsize::new(FLUSH_EDIT_BUDGET)),}}}@@ -510,7 +514,7 @@ fn inspect_session(// order, so that a mutation never comes before the nodes it changes. Each still counts toward its partition's block.let mut inspections: Vec<BlockInspection> = decoded.iter()- .map(|_| BlockInspection::with_resolve_budget(budget.resolved.clone()))+ .map(|_| BlockInspection::with_flush_budgets(budget.resolved.clone(), budget.edits.clone())).collect();let times: Vec<Vec<f64>> = decoded.iter()@@ -537,10 +541,10 @@ fn inspect_session(moments: inspected.moments,image_urls: inspected.image_urls,font_urls: inspected.font_urls,- remote_urls: inspected.remote_urls,url_limit: inspected.url_limit,- srcsets: inspected.srcsets,- css_rewrites: inspected.css_rewrites,+ edits: inspected.edits,+ declined: inspected.declined,+ unplaced: inspected.unplaced,visited_urls: inspected.features.visited_urls,captured_at_ms,}diff --git a/rust/ai-research/replay-recorder-v4/src/consumer.rs b/rust/ai-research/replay-recorder-v4/src/consumer.rsindex 155ff6921ccf..7dd7faa751c7 100644--- a/rust/ai-research/replay-recorder-v4/src/consumer.rs+++ b/rust/ai-research/replay-recorder-v4/src/consumer.rs@@ -1232,10 +1232,10 @@ mod tests {moments: Vec::new(),image_urls: Vec::new(),font_urls: Vec::new(),- remote_urls: Vec::new(),url_limit: None,- srcsets: Vec::new(),- css_rewrites: Vec::new(),+ edits: Vec::new(),+ declined: Vec::new(),+ unplaced: false,visited_urls: Vec::new(),captured_at_ms: None,})diff --git a/rust/ai-research/replay-recorder-v4/src/lib.rs b/rust/ai-research/replay-recorder-v4/src/lib.rsindex ac5b7d58c7cc..d410de66ff7f 100644--- a/rust/ai-research/replay-recorder-v4/src/lib.rs+++ b/rust/ai-research/replay-recorder-v4/src/lib.rs@@ -9,8 +9,7 @@ pub mod history;pub mod index;pub mod limits;pub mod process;-pub mod rewrite;-pub mod scrub;+pub mod splice;pub mod store;pub mod teams;pub mod urls;diff --git a/rust/ai-research/replay-recorder-v4/src/rewrite.rs b/rust/ai-research/replay-recorder-v4/src/rewrite.rsdeleted file mode 100644index 97a9d164a25c..000000000000--- a/rust/ai-research/replay-recorder-v4/src/rewrite.rs+++ /dev/null@@ -1,1205 +0,0 @@-//! Replaces the remote image URLs of a stored block with their `imageurl:v4` references, as v3 replaced them. A URL the-//! policy declines becomes a marker that names the reason, so that no credential, signature or private host stays in-//! the block. The inspector names the URLs: those in image slots, CSS included.-//!-//! A srcset is rebuilt instead: it keeps only the candidate that the inspector fetched, by [`shown_candidate`] in the-//! value the inspector read and its slot. It keeps none when the slot shows no candidate of that value: the inspector-//! then fetches the img's `src`, and a browser skips a source with no candidates, so it falls back to a source or an img-//! that can load, where a marker in the list would show a broken image. The writer rebuilds each srcset in the bytes it-//! stores, after escapes were written again and inline images left. Moving an inline image out can change how a srcset-//! parses, so the writer finds the inspector's candidate by its [`CandidatePosition`].--use std::borrow::Cow;-use std::cell::Cell;-use std::collections::{BTreeMap, HashMap};--use aho_corasick::{AhoCorasick, AhoCorasickKind, MatchKind};-use metrics::counter;-use replay_core_v4::bytescan::{unescape, write_json_string, Span};-use replay_core_v4::inspect::css::{- escape_in_identifier, rewrite_font_src_functions, rewrite_functions, CssRewrite,-};-use replay_core_v4::inspect::dom::{for_each_srcset_candidate, RemoteKind, RemoteUrl, Srcset};-use replay_core_v4::inspect::srcset::shown_candidate;-use replay_core_v4::refs::{declined_url_ref, UrlAsset};-use replay_core_v4::scan::tokens::{Token, Tokens};-use replay_core_v4::seal::image_url_key;-use replay_core_v4::url_policy::is_remote;-use replay_core_v4::v4_image_refs::{- bundled_font_ref, ImageDecline, FONT_URL_UNCHECKED_MARKER, IMAGE_DECLINED_PREFIX,-};--use crate::scrub::{Scrub, Scrubber};-use crate::urls::url_reference;--/// The text that replaces `url` in the block, and the policy's reason when it declines the URL.-fn named_as(- url: &str,- kind: RemoteKind,- team_id: i64,- key_month: &str,- url_key: &[u8; 32],-) -> (String, Option<&'static str>) {- let asset = match kind {- RemoteKind::Image => UrlAsset::Image,- RemoteKind::Font | RemoteKind::BundledFont(_) => UrlAsset::Font,- };- match (kind, url_reference(url, asset, team_id, key_month, url_key)) {- // The marker names no URL, but a declined URL keeps its reason, so that a copy inside a longer token is replaced too.- (RemoteKind::BundledFont(slug), named) => (- bundled_font_ref(slug),- named.err().map(|decline| decline.label()),- ),- (_, Ok((_, reference))) => (reference, None),- (_, Err(decline)) => (declined_url_ref(asset, decline), Some(decline.label())),- }-}--/// A byte that a URL can go on with. A match next to one is part of a longer URL, so it stays as it is. A comma, a-/// parenthesis or a quote can be in a URL too, but in a srcset or a CSS `url()` it ends one.-fn continues_url(byte: u8) -> bool {- byte.is_ascii_alphanumeric() || b"-._~/?#@!$&*+=%:;[]".contains(&byte)-}--/// Whether the bytes before `at` end with a JSON escape, such as the `\n` of a multi-line srcset. The escape's last-/// letter is not part of the URL that follows it.-fn ends_with_escape(bytes: &[u8], at: usize) -> bool {- let escaped = |letter: usize| {- bytes[..letter]- .iter()- .rev()- .take_while(|&&b| b == b'\\')- .count()- % 2- == 1- };- let short = at >= 2 && b"\"\\/bfnrt".contains(&bytes[at - 1]) && escaped(at - 1);- let unicode = at >= 6- && bytes[at - 5] == b'u'- && bytes[at - 4..at].iter().all(u8::is_ascii_hexdigit)- && escaped(at - 5);- short || unicode-}--const SRCSET_KEY: &[u8] = br#""srcset":"#;--/// Runs a stage of `apply` on bytes that an earlier stage may own, and keeps those bytes when the stage changes nothing.-macro_rules! then_stage {- ($bytes:expr, $writer:ident . $stage:ident) => {- match $bytes {- Cow::Borrowed(bytes) => $writer.$stage(bytes),- Cow::Owned(owned) => {- let changed = match $writer.$stage(&owned) {- Cow::Owned(changed) => Some(changed),- Cow::Borrowed(_) => None,- };- Cow::Owned(changed.unwrap_or(owned))- }- }- };-}-/// A CSS-escaped `:/` or `//` as a JSON string holds it, which starts a URL that a selector names. PHP-style JSON holds-/// `:\/\/` too, so a `//` right after a plain colon does not count.-const ESCAPED_URL_STARTS: [&[u8]; 2] = [br"\\:\\/", br"\\/\\/"];--fn holds_escaped_url(starts: &AhoCorasick, bytes: &[u8]) -> bool {- starts.find_iter(bytes).any(|found| {- found.pattern().as_usize() == 0 || found.start() == 0 || bytes[found.start() - 1] != b':'- })-}--/// The block's URLs as a selector holds them. Most events hold none, so these are built when one first does.-struct EscapedMatchers {- urls: Option<AhoCorasick>,- declined: Option<AhoCorasick>,- replacements: Vec<Vec<u8>>,-}--impl EscapedMatchers {- fn new(targets: &[Target]) -> Self {- let build = |targets: &mut dyn Iterator<Item = &Target>| {- let patterns: Vec<Vec<u8>> = targets- .map(|target| json_string_contents(&escape_in_identifier(&target.url)))- .collect();- (!patterns.is_empty())- .then(|| {- AhoCorasick::builder()- .kind(Some(AhoCorasickKind::ContiguousNFA))- .match_kind(MatchKind::LeftmostLongest)- .build(patterns)- .ok()- })- .flatten()- };- Self {- urls: build(&mut targets.iter()),- declined: build(&mut targets.iter().filter(|target| target.declined.is_some())),- replacements: targets- .iter()- .map(|target| {- json_string_contents(&escape_in_identifier(&String::from_utf8_lossy(- &target.replacement,- )))- })- .collect(),- }- }-}--fn json_string_contents(text: &str) -> Vec<u8> {- let mut escaped = Vec::with_capacity(text.len() + 2);- write_json_string(text, &mut escaped);- escaped.pop();- escaped.remove(0);- escaped-}--struct CssTarget {- /// With its URLs already replaced, so that a long page URL resolved for many short relative URLs never fills memory.- replacement: String,- /// The indexes in `targets` of the URLs that the rewrite removes or replaces, which count as found when it applies.- covers: Vec<usize>,- kind: &'static str,- found: bool,-}--/// The keys whose string value is CSS text. Every string under one counts too, such as a declaration in a style diff.-const CSS_KEYS: [&[u8]; 7] = [- b"_cssText",- b"style",- b"textContent",- b"rule",- b"replace",- b"replaceSync",- b"value",-];--struct CssStrings<'b> {- bytes: &'b [u8],- tokens: Tokens<'b>,- containers_in_css: Vec<bool>,- key: Option<Span>,-}--impl<'b> CssStrings<'b> {- fn new(bytes: &'b [u8]) -> Self {- Self {- bytes,- tokens: Tokens::new(bytes, (0, bytes.len())),- containers_in_css: Vec::new(),- key: None,- }- }-}--impl Iterator for CssStrings<'_> {- type Item = Span;-- fn next(&mut self) -> Option<Span> {- for token in self.tokens.by_ref() {- let in_css = self.containers_in_css.last() == Some(&true)- || self- .key- .is_some_and(|key| CSS_KEYS.contains(&&self.bytes[key.0..key.1]));- match token {- Token::ObjectStart | Token::ArrayStart => {- self.containers_in_css.push(in_css);- self.key = None;- }- Token::ObjectEnd | Token::ArrayEnd => {- self.containers_in_css.pop();- self.key = None;- }- Token::Key { span, escaped } => self.key = (!escaped).then_some(span),- Token::Value(span) => {- self.key = None;- if in_css && self.bytes.get(span.0) == Some(&b'"') {- return Some(span);- }- }- }- }- None- }-}--/// CSS escapes can spell `font-face` in many ways, but never its `@`.-fn holds_at_rule(bytes: &[u8]) -> bool {- bytes.contains(&b'@')-}--/// The metric label of a rewrite: the inspector rewrites a whole `url()` only to resolve a relative URL.-fn css_rewrite_kind(text: &str) -> &'static str {- if text- .get(..4)- .is_some_and(|start| start.eq_ignore_ascii_case("url("))- {- "relative_url"- } else {- "image_set"- }-}--struct Target {- url: String,- kind_label: &'static str,- replacement: Vec<u8>,- /// The policy's reason, for a declined URL.- declined: Option<&'static str>,- found: Cell<bool>,-}--pub struct UrlRewrite {- matcher: Option<AhoCorasick>,- /// The declined URLs alone. `apply` passes over a copy inside a longer URL, so the bytes it returns are scanned- /// again for them.- declined: Option<AhoCorasick>,- targets: Vec<Target>,- /// Bytes that `apply` returned still hold a declined URL.- declined_kept: bool,- srcset_key: AhoCorasick,- escaped_url_starts: AhoCorasick,- escaped: Option<EscapedMatchers>,- css_matcher: Option<AhoCorasick>,- css_targets: Vec<CssTarget>,- css_texts: HashMap<String, usize>,- /// The shown candidate of each inspected srcset value, by the [`remote_urls`] of the value. Only a stored srcset with- /// the same remote URLs is rebuilt, so a srcset key outside an image slot, such as in a custom event's payload, stays- /// as it is. Values with the same remote URLs share an entry, so each keeps the candidate of every one of them.- kept_srcset_candidates: HashMap<Vec<String>, Vec<CandidatePosition>>,- url_targets: HashMap<String, usize>,- team_id: i64,- key_month: String,- url_key: [u8; 32],- fallback: Fallback,- scrubber: Scrubber,- font_source_scrubber: Scrubber,- scrubbed: u64,- /// The JSON key whose value held the first declined URL that stayed, for the log. It names the shape without any URL.- kept_in: Option<String>,- counters: PassCounters,-}--/// The counter increments of one writer pass. The writer emits them only for the pass whose output it keeps, so a block-/// written twice counts once.-#[derive(Debug, Clone, Default, PartialEq, Eq)]-pub struct PassCounters {- srcset_rewrites: BTreeMap<&'static str, u64>,- css_rewrites: BTreeMap<(&'static str, &'static str), u64>,- url_rewrites: BTreeMap<(&'static str, &'static str, &'static str), u64>,- pub removed_images: [u64; ImageDecline::ALL.len()],-}--impl PassCounters {- pub fn emit(&self) {- for (&outcome, &count) in &self.srcset_rewrites {- counter!("ai_research_v4_recorder_srcset_rewrites_total", "outcome" => outcome)- .increment(count);- }- for (&(kind, outcome), &count) in &self.css_rewrites {- counter!(- "ai_research_v4_recorder_css_rewrites_total",- "kind" => kind,- "outcome" => outcome,- )- .increment(count);- }- for (&(kind, outcome, reason), &count) in &self.url_rewrites {- counter!(- "ai_research_v4_recorder_image_url_rewrites_total",- "kind" => kind,- "outcome" => outcome,- "reason" => reason,- )- .increment(count);- }- for (reason, &count) in ImageDecline::ALL.iter().zip(&self.removed_images) {- counter!("ai_research_v4_recorder_removed_images_total", "reason" => reason.label())- .increment(count);- }- }-}--/// How the writer handles what the first pass over a block left unsafe, instead of dropping the block.-#[derive(Debug, Clone, Default, PartialEq, Eq)]-pub struct Fallback {- /// Replace a declined URL wherever its bytes are, also inside a longer URL that would otherwise keep it.- pub overwrite_declined: bool,- pub scrub: Scrub,- /// Replace every remote URL in a `url()` of an `@font-face` `src`, also where `scrub` replaces nothing.- pub scrub_font_sources: bool,- /// The metric label of why the fallback runs.- pub reason: Option<&'static str>,-}--/// What `finish` found about a block.-#[derive(Debug, Clone, Default, PartialEq, Eq)]-pub struct RewriteReport {- /// No URL the policy declines can stay in the stored bytes.- pub safe: bool,- /// A declined URL stayed, as a second copy that the rewrite passed over.- pub declined_kept: bool,- /// The hosts of declined URLs the inspector named but the stored bytes do not hold as it read them. `None` when- /// one of them has no host to look for.- pub unfound_hosts: Option<Vec<String>>,- pub scrubbed: u64,- pub kept_in: Option<String>,- pub counters: PassCounters,-}--impl UrlRewrite {- /// `team_key` is the team's month key for `key_month`, the write month, as for the fetch jobs.- pub fn new(- urls: &[RemoteUrl],- srcsets: &[Srcset],- css_rewrites: &[CssRewrite],- team_id: i64,- key_month: &str,- team_key: &[u8; 32],- ) -> Self {- let url_key = image_url_key(team_key);- let mut patterns = Vec::with_capacity(urls.len() + srcsets.len());- let mut targets = Vec::with_capacity(urls.len() + srcsets.len());- for RemoteUrl { url, kind } in urls {- // The block holds the URL as a JSON string holds it.- patterns.push(json_string_contents(url));- let kind_label = match kind {- RemoteKind::Image => "image",- RemoteKind::Font => "font",- RemoteKind::BundledFont(_) => "bundled_font",- };- let (replacement, declined) = named_as(url, *kind, team_id, key_month, &url_key);- targets.push(Target {- url: url.clone(),- kind_label,- replacement: replacement.into_bytes(),- declined,- found: Cell::new(false),- });- }- let url_targets: HashMap<String, usize> = urls- .iter()- .enumerate()- .map(|(index, remote)| (remote.url.clone(), index))- .collect();- let css_matcher = (!css_rewrites.is_empty())- .then(|| {- AhoCorasick::builder()- .kind(Some(AhoCorasickKind::ContiguousNFA))- .match_kind(MatchKind::LeftmostLongest)- .build(- css_rewrites- .iter()- .map(|rewrite| json_string_contents(&rewrite.text)),- )- .ok()- })- .flatten();- let mut kept_srcset_candidates: HashMap<Vec<String>, Vec<CandidatePosition>> =- HashMap::new();- for srcset in srcsets {- let kept = kept_srcset_candidates- .entry(remote_urls(&srcset.text))- .or_default();- if let Some(position) = shown_position(srcset).filter(|p| !kept.contains(p)) {- kept.push(position);- }- }- let matcher = (!patterns.is_empty())- .then(|| {- // A DFA over long URLs can take hundreds of times their bytes. A contiguous NFA takes about ten.- AhoCorasick::builder()- .kind(Some(AhoCorasickKind::ContiguousNFA))- .match_kind(MatchKind::LeftmostLongest)- .build(&patterns)- .ok()- })- .flatten();- let css_targets: Vec<CssTarget> = css_rewrites- .iter()- .map(|rewrite| {- let mut covers = Vec::new();- let resolved = match &rewrite.resolved {- Some(RemoteUrl {- url: absolute,- kind,- }) => {- let kind = url_targets- .get(absolute)- .map_or(*kind, |&index| urls[index].kind.max(*kind));- let (reference, _) = named_as(absolute, kind, team_id, key_month, &url_key);- rewrite.replacement.replacen(- &format!("\"{absolute}\""),- &format!("\"{reference}\""),- 1,- )- }- None => rewrite.replacement.clone(),- };- let replacement = match &matcher {- Some(matcher) => {- let quoted =- [b"\"".as_slice(), &json_string_contents(&resolved), b"\""].concat();- let replaced = replace_targets(- matcher,- &targets,- false,- None,- "ed,- &mut |index| covers.push(index),- );- unescape(&replaced, (0, replaced.len()))- .map_or_else(|_| resolved.clone(), Cow::into_owned)- }- None => resolved,- };- covers.extend(- rewrite- .dropped- .iter()- .filter_map(|url| url_targets.get(url).copied()),- );- CssTarget {- replacement,- covers,- kind: css_rewrite_kind(&rewrite.text),- found: false,- }- })- .collect();- let declined: Vec<&[u8]> = patterns- .iter()- .zip(&targets)- .filter(|(_, target)| target.declined.is_some())- .map(|(pattern, _)| pattern.as_slice())- .collect();- let declined = (!declined.is_empty())- .then(|| {- AhoCorasick::builder()- .kind(Some(AhoCorasickKind::ContiguousNFA))- .build(declined)- .ok()- })- .flatten();- Self {- matcher,- declined,- targets,- declined_kept: false,- srcset_key: AhoCorasick::new([SRCSET_KEY]).expect("one literal pattern builds"),- escaped_url_starts: AhoCorasick::new(ESCAPED_URL_STARTS)- .expect("two literal patterns build"),- escaped: None,- css_matcher,- css_targets,- css_texts: css_rewrites- .iter()- .enumerate()- .map(|(index, rewrite)| (rewrite.text.clone(), index))- .collect(),- kept_srcset_candidates,- url_targets,- team_id,- key_month: key_month.to_owned(),- url_key,- fallback: Fallback::default(),- scrubber: Scrubber::new(&Scrub::Nothing),- font_source_scrubber: Scrubber::new(&Scrub::Nothing),- scrubbed: 0,- kept_in: None,- counters: PassCounters::default(),- }- }-- pub fn with_fallback(mut self, fallback: Fallback) -> Self {- self.scrubber = Scrubber::new(&fallback.scrub);- self.font_source_scrubber = Scrubber::new(if fallback.scrub_font_sources {- &Scrub::AllRemote- } else {- &fallback.scrub- });- self.fallback = fallback;- self- }-- /// The bytes with each srcset that holds a candidate the block cannot keep rebuilt from the candidates it keeps.- fn rebuild_srcsets<'b>(&mut self, bytes: &'b [u8]) -> Cow<'b, [u8]> {- let mut out: Option<Vec<u8>> = None;- let mut copied_to = 0;- for found in self.srcset_key.find_iter(bytes) {- let start = found.end();- if start < copied_to || bytes.get(start) != Some(&b'"') {- continue;- }- let Some(end) = string_end(bytes, start) else {- continue;- };- let Ok(text) = unescape(bytes, (start, end)) else {- continue;- };- let Some(kept) = self.kept_srcset_candidates.get(&remote_urls(&text)) else {- continue;- };- let Some((value, outcome, covers)) = rebuilt_srcset(- &text,- kept,- &self.url_targets,- self.team_id,- &self.key_month,- &self.url_key,- ) else {- continue;- };- for url in covers {- self.targets[url].found.set(true);- }- *self.counters.srcset_rewrites.entry(outcome).or_default() += 1;- let out = out.get_or_insert_with(|| Vec::with_capacity(bytes.len()));- out.extend_from_slice(&bytes[copied_to..start]);- write_json_string(&value, out);- copied_to = end;- }- match out {- Some(mut out) => {- out.extend_from_slice(&bytes[copied_to..]);- Cow::Owned(out)- }- None => Cow::Borrowed(bytes),- }- }-- /// Rewrites only whole `url()` and `image-set()` functions inside JSON strings that hold CSS, found as the inspector- /// finds them. So a class such as `bg-[url(/a.png)]`, a quoted string or a comment keeps its text.- fn rewrite_css<'b>(&mut self, bytes: &'b [u8]) -> Cow<'b, [u8]> {- let Some(matcher) = self.css_matcher.clone() else {- return Cow::Borrowed(bytes);- };- let mut css_strings: Option<std::iter::Peekable<CssStrings<'_>>> = None;- let mut out: Option<Vec<u8>> = None;- let mut copied_to = 0;- let mut scanned_to = 0;- for found in matcher.find_iter(bytes) {- let start = found.start();- if start < scanned_to {- continue;- }- let strings = css_strings.get_or_insert_with(|| CssStrings::new(bytes).peekable());- while strings.next_if(|string| string.1 <= start).is_some() {}- let Some(&string) = strings.peek().filter(|string| string.0 < start) else {- continue;- };- scanned_to = string.1;- let Ok(css) = unescape(bytes, string) else {- continue;- };- let rewritten = rewrite_functions(&css, &mut |function| {- let &index = self.css_texts.get(function)?;- let target = &mut self.css_targets[index];- target.found = true;- for &url in &target.covers {- self.targets[url].found.set(true);- }- Some(target.replacement.clone())- });- if let Some(rewritten) = rewritten {- let out = out.get_or_insert_with(|| Vec::with_capacity(bytes.len()));- out.extend_from_slice(&bytes[copied_to..string.0]);- write_json_string(&rewritten, out);- copied_to = string.1;- }- }- match out {- Some(mut out) => {- out.extend_from_slice(&bytes[copied_to..]);- Cow::Owned(out)- }- None => Cow::Borrowed(bytes),- }- }-- /// The bytes with each URL replaced, and in a DOM event each inspected srcset rebuilt. Returns the input when it- /// holds none. Only a full snapshot or a mutation holds image slots, so a srcset in any other event, such as a- /// custom event's payload with the same value, is left to the rewrite of each URL.- pub fn apply<'b>(&mut self, bytes: &'b [u8], dom_event: bool) -> Cow<'b, [u8]> {- let rebuilt = if dom_event {- self.rebuild_srcsets(bytes)- } else {- Cow::Borrowed(bytes)- };- let rewritten = then_stage!(rebuilt, self.rewrite_css);- let replaced = then_stage!(rewritten, self.replace_urls);- let replaced = if holds_escaped_url(&self.escaped_url_starts, replaced.as_ref()) {- then_stage!(replaced, self.replace_escaped_urls)- } else {- replaced- };- let replaced = then_stage!(replaced, self.mark_unchecked_fonts);- let (scrubbed, count) = self.scrubber.apply(&replaced);- if count == 0 {- return replaced;- }- self.scrubbed += count;- Cow::Owned(scrubbed.into_owned())- }-- /// Runs before the scrub, which would leave an image marker where the renderer loads a font.- fn mark_unchecked_fonts<'b>(&mut self, bytes: &'b [u8]) -> Cow<'b, [u8]> {- if !self.font_source_scrubber.replaces_tokens() || !holds_at_rule(bytes) {- return Cow::Borrowed(bytes);- }- let mut out: Option<Vec<u8>> = None;- let mut copied_to = 0;- for string in CssStrings::new(bytes) {- if !holds_at_rule(&bytes[string.0..string.1]) {- continue;- }- let Ok(css) = unescape(bytes, string) else {- continue;- };- let rewritten = rewrite_font_src_functions(&css, &mut |function| {- let mut json = Vec::with_capacity(function.len() + 2);- write_json_string(function, &mut json);- (self.font_source_scrubber.apply(&json).1 > 0).then(|| {- self.scrubbed += 1;- format!("url(\"{FONT_URL_UNCHECKED_MARKER}\")")- })- });- if let Some(rewritten) = rewritten {- let out = out.get_or_insert_with(|| Vec::with_capacity(bytes.len()));- out.extend_from_slice(&bytes[copied_to..string.0]);- write_json_string(&rewritten, out);- copied_to = string.1;- }- }- match out {- Some(mut out) => {- out.extend_from_slice(&bytes[copied_to..]);- Cow::Owned(out)- }- None => Cow::Borrowed(bytes),- }- }-- fn replace_escaped_urls<'b>(&mut self, bytes: &'b [u8]) -> Cow<'b, [u8]> {- let escaped = self- .escaped- .get_or_insert_with(|| EscapedMatchers::new(&self.targets));- let Some(matcher) = &escaped.urls else {- return Cow::Borrowed(bytes);- };- let targets = &self.targets;- let out = replace_targets(- matcher,- targets,- self.fallback.overwrite_declined,- Some(&escaped.replacements),- bytes,- &mut |index| targets[index].found.set(true),- );- if escaped- .declined- .as_ref()- .is_some_and(|declined| declined.is_match(out.as_ref()))- {- self.declined_kept = true;- }- out- }-- fn replace_urls<'b>(&mut self, bytes: &'b [u8]) -> Cow<'b, [u8]> {- let Some(matcher) = &self.matcher else {- return Cow::Borrowed(bytes);- };- let targets = &self.targets;- let out = replace_targets(- matcher,- targets,- self.fallback.overwrite_declined,- None,- bytes,- &mut |index| targets[index].found.set(true),- );- if let Some(declined) = &self.declined {- if let Some(kept) = declined.find(out.as_ref()) {- self.declined_kept = true;- if self.kept_in.is_none() {- self.kept_in =- Some(enclosing_key(&out, kept.start()).unwrap_or("?").to_owned());- }- }- }- out- }-- /// Counts the block's URLs. One the block does not hold as the inspector read it, for example with a CSS escape,- /// stays as it arrived. The block is unsafe when a URL the policy declines can stay, unreplaced or as a second copy- /// that `apply` passed over, unless the fallback scrubbed every token that could hold it.- pub fn finish(self) -> RewriteReport {- let covered = |url: &str| match &self.fallback.scrub {- Scrub::AllRemote => true,- Scrub::Hosts(hosts) => {- url_host(url).is_some_and(|host| hosts.binary_search(&host).is_ok())- }- Scrub::Nothing => false,- };- let mut unfound_hosts = Some(Vec::new());- let mut safe = !self.declined_kept;- for target in &self.targets {- if !target.found.get() && target.declined.is_some() && !covered(&target.url) {- safe = false;- match (url_host(&target.url), &mut unfound_hosts) {- (Some(host), Some(hosts)) => hosts.push(host),- _ => unfound_hosts = None,- }- }- }- // Many declined URLs share a host. The second pass binary-searches this list for every target, so it is sorted.- if let Some(hosts) = &mut unfound_hosts {- hosts.sort_unstable();- hosts.dedup();- }- let mut counters = self.counters;- for target in self.css_targets {- let outcome = if target.found { "rewritten" } else { "unfound" };- *counters- .css_rewrites- .entry((target.kind, outcome))- .or_default() += 1;- }- for target in self.targets {- let outcome = match (target.found.get(), target.declined) {- (false, _) => "unrewritten",- (true, None) => "rewritten",- (true, Some(_)) => "declined",- };- *counters- .url_rewrites- .entry((- target.kind_label,- outcome,- target.declined.unwrap_or("none"),- ))- .or_default() += 1;- }- RewriteReport {- safe,- declined_kept: self.declined_kept,- unfound_hosts,- scrubbed: self.scrubbed,- kept_in: self.kept_in,- counters,- }- }-}--fn replace_targets<'b>(- matcher: &AhoCorasick,- targets: &[Target],- overwrite_declined: bool,- escaped_replacements: Option<&[Vec<u8>]>,- bytes: &'b [u8],- replaced: &mut dyn FnMut(usize),-) -> Cow<'b, [u8]> {- let mut out: Option<Vec<u8>> = None;- let mut copied_to = 0;- for found in matcher.find_iter(bytes) {- let (start, end) = (found.start(), found.end());- let index = found.pattern().as_usize();- // An escaped URL goes on with an escaped character, which a JSON string holds as `\\` and the character.- let continues_after = bytes.get(end).is_some_and(|&after| continues_url(after))- || (escaped_replacements.is_some()- && bytes.get(end..end + 2) == Some(br"\\")- && bytes- .get(end + 2)- .is_some_and(|&after| continues_url(after)));- // A declined URL inside a longer URL is replaced too when the fallback asks, so that it cannot stay.- let overwrite = overwrite_declined && targets[index].declined.is_some();- if !overwrite- && (start.checked_sub(1).is_some_and(|before| {- continues_url(bytes[before]) && !ends_with_escape(bytes, start)- }) || continues_after)- {- continue;- }- replaced(index);- let out = out.get_or_insert_with(|| Vec::with_capacity(bytes.len()));- out.extend_from_slice(&bytes[copied_to..start]);- out.extend_from_slice(match escaped_replacements {- Some(replacements) => &replacements[index],- None => &targets[index].replacement,- });- copied_to = end;- }- let out = match out {- Some(mut out) => {- out.extend_from_slice(&bytes[copied_to..]);- Cow::Owned(out)- }- None => Cow::Borrowed(bytes),- };- out-}--/// The key of the JSON object entry whose string value holds `at`, when it is a plain short key.-fn enclosing_key(bytes: &[u8], at: usize) -> Option<&str> {- let mut open = at;- while open > 0 {- open -= 1;- if bytes[open] == b'"' && !(open > 0 && bytes[open - 1] == b'\\') {- break;- }- }- let colon = open.checked_sub(1).filter(|&c| bytes[c] == b':')?;- let key_end = colon.checked_sub(1).filter(|&e| bytes[e] == b'"')?;- let key_start = bytes[..key_end].iter().rposition(|&b| b == b'"')? + 1;- let key = std::str::from_utf8(&bytes[key_start..key_end]).ok()?;- (key.len() <= 40- && key- .bytes()- .all(|b| b.is_ascii_alphanumeric() || b == b'_' || b == b'$' || b == b'-'))- .then_some(key)-}--/// The host of a URL the inspector named, in lowercase, protocol-relative and backslash forms included.-fn url_host(url: &str) -> Option<String> {- let with_scheme;- let url = if replay_core_v4::url_policy::is_protocol_relative(url) {- with_scheme = format!("https:{}", url.trim());- with_scheme.as_str()- } else {- url- };- url::Url::parse(url)- .ok()?- .host_str()- .map(str::to_ascii_lowercase)-}--/// Where the JSON string that opens at `start` ends, after its closing quote.-fn string_end(bytes: &[u8], start: usize) -> Option<usize> {- let mut position = start + 1;- while let Some(&byte) = bytes.get(position) {- match byte {- b'\\' => position += 2,- b'"' => return Some(position + 1),- _ => position += 1,- }- }- None-}--/// The remote URLs of a srcset, in order. The writer moves only inline images out of a srcset, so these name the-/// srcset the same way in the event the inspector read and in the block the writer stores.-fn remote_urls(srcset: &str) -> Vec<String> {- let mut urls = Vec::new();- for_each_srcset_candidate(srcset, |url, _| {- if is_remote(url) {- urls.push(url.to_owned());- }- });- urls-}--/// A srcset candidate by its kind and its index among the candidates of that kind. The writer moves only inline images-/// out of a srcset, so a position names the same candidate in the event the inspector read and in the block the writer-/// stores.-#[derive(Debug, Clone, Copy, PartialEq, Eq)]-enum CandidatePosition {- Remote(usize),- Inline(usize),-}--fn for_each_positioned_candidate<'s>(- srcset: &'s str,- mut found: impl FnMut(CandidatePosition, &'s str, &'s str),-) {- let (mut remote, mut inline) = (0, 0);- for_each_srcset_candidate(srcset, |url, descriptors| {- let position = if is_remote(url) {- remote += 1;- CandidatePosition::Remote(remote - 1)- } else {- inline += 1;- CandidatePosition::Inline(inline - 1)- };- found(position, url, descriptors);- });-}--fn shown_position(srcset: &Srcset) -> Option<CandidatePosition> {- // `shown_candidate` reads only a well-formed srcset, whose candidates `for_each_srcset_candidate` reads in the same order.- let shown_index = shown_candidate(&srcset.text, srcset.slot)?.index;- let (mut index, mut shown) = (0, None);- for_each_positioned_candidate(&srcset.text, |position, _, _| {- if index == shown_index {- shown = Some(position);- }- index += 1;- });- shown-}--/// The srcset with only the candidate the block keeps, a remote URL as its reference, how it was rebuilt, and the URL-/// targets of the remote URLs it held. `None` when the srcset keeps its one candidate, because the URL targets then-/// replace it.-fn rebuilt_srcset(- text: &str,- kept: &[CandidatePosition],- url_targets: &HashMap<String, usize>,- team_id: i64,- key_month: &str,- url_key: &[u8; 32],-) -> Option<(String, &'static str, Vec<usize>)> {- // Streams the candidates into the new value, so that a long srcset holds no list of them.- let mut value = String::with_capacity(text.len());- let mut covers = Vec::new();- let mut removed = false;- let mut keep = |url: &str, descriptors: &str| {- if !value.is_empty() {- value.push_str(", ");- }- value.push_str(url);- if !descriptors.is_empty() {- value.push(' ');- value.push_str(descriptors);- }- };- for_each_positioned_candidate(text, |position, url, descriptors| {- let is_kept = kept.contains(&position);- match position {- CandidatePosition::Remote(_) => {- covers.extend(url_targets.get(url));- match url_reference(url, UrlAsset::Image, team_id, key_month, url_key) {- Ok((_, reference)) if is_kept => keep(&reference, descriptors),- _ => removed = true,- }- }- CandidatePosition::Inline(_) if is_kept && !url.starts_with(IMAGE_DECLINED_PREFIX) => {- keep(url, descriptors)- }- CandidatePosition::Inline(_) => removed = true,- }- });- if !removed {- return None;- }- let outcome = if value.is_empty() {- "emptied"- } else {- "trimmed"- };- Some((value, outcome, covers))-}--#[cfg(test)]-mod tests {- use base64::engine::general_purpose::STANDARD;- use base64::Engine;- use replay_core_v4::bytescan::canonical_strings;- use replay_core_v4::inspect::srcset::SrcsetSlot;-- use super::*;-- const TEAM_KEY: [u8; 32] = [6; 32];-- fn images(urls: &[&str]) -> Vec<RemoteUrl> {- urls.iter()- .map(|url| RemoteUrl {- url: (*url).to_owned(),- kind: RemoteKind::Image,- })- .collect()- }-- fn rewrite(urls: &[&str], bytes: &str) -> String {- let mut rewrite = UrlRewrite::new(&images(urls), &[], &[], 7, "2026-10", &TEAM_KEY);- String::from_utf8(rewrite.apply(bytes.as_bytes(), true).into_owned()).unwrap()- }-- fn reference(url: &str) -> String {- url_reference(- url,- UrlAsset::Image,- 7,- "2026-10",- &image_url_key(&TEAM_KEY),- )- .unwrap()- .1- }-- #[test]- fn keeps_srcset_and_css_syntax_while_it_replaces_each_url() {- let (a, b) = (- "https://cdn.example.com/w_640,q_auto/a.png",- "https://cdn.example.com/b.png",- );- let event = format!(- r#"{{"srcset":"{a} 640w, {b} 2x","style":"background:url({b}) no-repeat","text":".x{{background:url(\"{a}\")}}","other":"{b}?v=2"}}"#- );- let out = rewrite(&[a, b], &event);- let (ra, rb) = (reference(a), reference(b));- assert_eq!(- out,- format!(- r#"{{"srcset":"{ra} 640w, {rb} 2x","style":"background:url({rb}) no-repeat","text":".x{{background:url(\"{ra}\")}}","other":"{b}?v=2"}}"#- )- );- assert!(serde_json::from_str::<serde_json::Value>(&out).is_ok());-- // A srcset over several lines holds each URL after an escape, which is no part of the URL.- let declined = "https://10.0.0.7/c.png";- let event = format!(r#"{{"srcset":"{a} 1x,\n{b} 2x,\t{declined} 3x"}}"#);- let mut rewrite = UrlRewrite::new(- &images(&[a, b, declined]),- &[],- &[],- 7,- "2026-10",- &TEAM_KEY,- );- assert_eq!(- String::from_utf8(rewrite.apply(event.as_bytes(), true).into_owned()).unwrap(),- format!(- r#"{{"srcset":"{ra} 1x,\n{rb} 2x,\timageurl:v4:declined:non_public_host 3x"}}"#- )- );- assert!(rewrite.finish().safe);- }-- #[test]- fn keeps_only_the_srcset_candidate_the_inspector_fetched() {- let png = format!(- "data:image/png;base64,{}",- STANDARD.encode(b"\x89PNG\r\n\x1a\nbytes")- );- let svg = "data:image/svg+xml,%3Csvg%2F%3E";- let (a, b, private) = (- "https://cdn.example.com/a.png",- "https://cdn.example.com/b.png",- "https://10.0.0.7/b.png",- );- let over_limit = ImageDecline::OverLimit.marker();- for (slot, inspected, stored, expected) in [- // A message past its image cap stores the PNG as `over_limit`, where the inspector read a data URI.- (- SrcsetSlot::Alone,- format!("{png} 2x, {svg} 1x"),- format!("{over_limit} 2x, {svg} 1x"),- format!("{svg} 1x"),- ),- (- SrcsetSlot::Alone,- format!("{a} 1x, {b} 2x"),- format!("{a} 1x, {b} 2x"),- format!("{} 1x", reference(a)),- ),- (- SrcsetSlot::BesideImgSrc,- format!("{a} 1x, {b} 2x"),- format!("{a} 1x, {b} 2x"),- format!("{} 1x", reference(a)),- ),- // The img shows its `src`, which HTML adds as a 1x candidate.- (- SrcsetSlot::BesideImgSrc,- format!("{b} 2x"),- format!("{b} 2x"),- String::new(),- ),- // The 2x candidate has no fetch job, so it cannot stand in for the declined 1x one.- (- SrcsetSlot::Alone,- format!("{private} 1x, {b} 2x"),- format!("{private} 1x, {b} 2x"),- String::new(),- ),- (- SrcsetSlot::Alone,- format!("{a} 1x, {b} 640w"),- format!("{a} 1x, {b} 640w"),- String::new(),- ),- // The image scan moves out a data URI in any case, where the inspector read a malformed srcset and fetched- // the img's `src` instead.- (- SrcsetSlot::Alone,- format!(- "DATA:image/png;base64,{} 1x, {b} 2x",- STANDARD.encode(b"\x89PNG\r\n\x1a\nbytes")- ),- format!("image:v4:7:2026-10:0123abcd 1x, {b} 2x"),- String::new(),- ),- ] {- let srcset = Srcset {- raw: serde_json::to_vec(&inspected).unwrap(),- text: inspected.clone(),- slot,- };- let mut rewrite = UrlRewrite::new(&[], &[srcset], &[], 7, "2026-10", &TEAM_KEY);- let stored = format!(r#"{{"srcset":"{stored}"}}"#);- assert_eq!(- String::from_utf8(rewrite.apply(stored.as_bytes(), true).into_owned()).unwrap(),- format!(r#"{{"srcset":"{expected}"}}"#),- "{inspected}"- );- }- }-- #[test]- fn rewrites_a_url_that_a_json_escape_hid() {- let url = "https://cdn.example.com/a.png?token=secret";- let hidden = r#"{"src":"https:\/\/cdn.example.com\/a.png?token=secr\u0065t","alt":"a\nb"}"#;- let canonical = canonical_strings(hidden.as_bytes());- let mut rewrite = UrlRewrite::new(&images(&[url]), &[], &[], 7, "2026-10", &TEAM_KEY);- assert_eq!(- String::from_utf8(rewrite.apply(&canonical, true).into_owned()).unwrap(),- r#"{"src":"imageurl:v4:declined:credential","alt":"a\nb"}"#- );- assert!(rewrite.finish().safe);- // An event capture wrote is left as it is.- let plain = r#"{"text":"a\"b\\c\u0001","n":1}"#;- assert!(matches!(- canonical_strings(plain.as_bytes()),- Cow::Borrowed(_)- ));- // A declined URL the block still holds fails the block.- let rewrite = UrlRewrite::new(&images(&[url]), &[], &[], 7, "2026-10", &TEAM_KEY);- assert!(!rewrite.finish().safe);- }-- #[test]- fn leaves_only_a_marker_where_the_policy_declines_a_url() {- for (url, reason) in [- ("https://user:[email protected]/a.png", "credential"),- (- "https://bucket.s3.amazonaws.com/a.png?X-Amz-Signature=abc123",- "credential",- ),- ("https://10.0.0.7/a.png", "non_public_host"),- ("http://cdn.example.com/a.png", "bad_scheme"),- ("//user:[email protected]/a.png", "credential"),- ("//10.0.0.7/a.png", "non_public_host"),- ("https:/user:[email protected]/a.png", "credential"),- ] {- let out = rewrite(&[url], &format!(r#"{{"src":"{url}"}}"#));- assert_eq!(- out,- format!(r#"{{"src":"imageurl:v4:declined:{reason}"}}"#),- "{url}"- );- }- // A second copy inside a longer URL stays as it is, so the block fails.- let url = "https://bucket.s3.amazonaws.com/a.png?X-Amz-Signature=abc123";- let mut rewrite = UrlRewrite::new(&images(&[url]), &[], &[], 7, "2026-10", &TEAM_KEY);- rewrite.apply(- format!(r#"{{"src":"{url}","alt":"x{url}"}}"#).as_bytes(),- true,- );- let report = rewrite.finish();- assert!(!report.safe);- // The log names where the copy stayed, without the URL.- assert_eq!(report.kept_in.as_deref(), Some("alt"));- }-}diff --git a/rust/ai-research/replay-recorder-v4/src/scrub.rs b/rust/ai-research/replay-recorder-v4/src/scrub.rsdeleted file mode 100644index e96a78b0d018..000000000000--- a/rust/ai-research/replay-recorder-v4/src/scrub.rs+++ /dev/null@@ -1,600 +0,0 @@-//! The fallback for a block whose remote URLs the writer could not all name or replace. Each remote URL token left in the stored bytes becomes [`URL_UNCHECKED_MARKER`], or `fonturl:v4:declined:unchecked` in the `src` of an `@font-face` rule, so no URL the policy would refuse is stored, and the block is kept instead of dropped.-//!-//! The scrub reads the bytes one character at a time through their JSON escapes and CSS escapes, so it never splits an-//! escape and the stored JSON stays valid. A URL that starts after `(`, `'` or `"` ends at `)`, `'` or `"`, because CSS-//! ends it there. A URL that starts its JSON string may be the whole value of an attribute, where a space does not end-//! it, so it runs to the end of the string. Any other URL token ends at whitespace or at the end of its JSON string. The-//! scrub does not know which slot a token sits in, so it also replaces URLs outside image slots, such as links and text.-//! It runs only on the few blocks that need it.--use std::borrow::Cow;--use aho_corasick::{AhoCorasick, AhoCorasickBuilder, MatchKind};-use replay_core_v4::v4_image_refs::URL_UNCHECKED_MARKER;--/// Which URL tokens the scrub replaces.-#[derive(Debug, Clone, Default, PartialEq, Eq)]-pub enum Scrub {- #[default]- Nothing,- /// Every token that `url_policy::is_remote` reads as remote: `http:` or `https:`, or two slashes or backslashes,- /// in any letter case and through any escape.- AllRemote,- /// Every token that holds one of these hosts, in any letter case. The list is sorted, without duplicates.- Hosts(Vec<String>),-}--pub struct Scrubber {- all_remote: bool,- hosts: Option<AhoCorasick>,-}--#[derive(Clone, Copy, PartialEq, Eq)]-enum Kind {- Plain,- /// An unescaped `"`, which ends the JSON string.- StringEnd,- /// A character a CSS escape names. It is part of the URL whatever it is.- CssEscaped,- /// An escaped line break, which a CSS string drops, so it is no character of the URL.- Continuation,-}--/// One character of the stored bytes and the bytes that spell it.-#[derive(Clone, Copy)]-struct Unit {- ch: u32,- len: usize,- kind: Kind,-}--/// The character that ends a URL besides the end of its JSON string.-#[derive(Clone, Copy)]-enum Closer {- Whitespace,- Nothing,- Paren,- Apostrophe,- Quote,-}--enum Start {- Scheme,- Slashes,-}--fn is_space(ch: u32) -> bool {- ch <= u32::from(b' ')-}--fn is_char(ch: u32, byte: u8) -> bool {- ch == u32::from(byte)-}--/// A character that a relative path or a word can hold, so `//` after it starts no URL.-fn continues_word(ch: u32) -> bool {- u8::try_from(ch)- .is_ok_and(|b| b.is_ascii_alphanumeric() || b"-._~/?#@!$&*+=%:;[]\\".contains(&b))-}--/// The character at `at` after its JSON escape, and its length in bytes.-fn json_unit(bytes: &[u8], at: usize) -> (u32, usize) {- let byte = bytes[at];- if byte != b'\\' {- return (u32::from(byte), 1);- }- match bytes.get(at + 1) {- Some(b'u') => bytes- .get(at + 2..at + 6)- .and_then(|hex| std::str::from_utf8(hex).ok())- .and_then(|hex| u32::from_str_radix(hex, 16).ok())- .map_or((u32::from(byte), 1), |code| (code, 6)),- Some(&escaped) => {- let ch = match escaped {- b'n' => b'\n',- b'r' => b'\r',- b't' => b'\t',- b'b' => 0x08,- b'f' => 0x0c,- other => other,- };- (u32::from(ch), 2)- }- None => (u32::from(byte), 1),- }-}--/// The position past the `\n` of a `\r\n` whose `\r` ends at `after`, because CSS reads the pair as one line break.-fn past_crlf(bytes: &[u8], after: usize, ch: u32) -> usize {- if is_char(ch, b'\r') && after < bytes.len() {- let (next, next_len) = json_unit(bytes, after);- if is_char(next, b'\n') {- return after + next_len;- }- }- after-}--/// The character at `at` after its JSON escape and a CSS escape inside it. A CSS hex escape takes the one whitespace-/// after it, as CSS reads it.-fn unit(bytes: &[u8], at: usize) -> Unit {- let (ch, len) = json_unit(bytes, at);- if len == 1 && is_char(ch, b'"') {- return Unit {- ch,- len,- kind: Kind::StringEnd,- };- }- let plain = Unit {- ch,- len,- kind: Kind::Plain,- };- if len == 1 || !is_char(ch, b'\\') {- return plain;- }- let mut end = at + len;- let (mut code, mut digits) = (0u32, 0);- while digits < 6 && end < bytes.len() {- let (next, next_len) = json_unit(bytes, end);- let Some(digit) = char::from_u32(next).and_then(|c| c.to_digit(16)) else {- break;- };- code = code * 16 + digit;- digits += 1;- end += next_len;- }- if digits > 0 {- if end < bytes.len() {- let (next, next_len) = json_unit(bytes, end);- if is_space(next) {- end = past_crlf(bytes, end + next_len, next);- }- }- return Unit {- ch: code,- len: end - at,- kind: Kind::CssEscaped,- };- }- if end >= bytes.len() {- return plain;- }- let (next, next_len) = json_unit(bytes, end);- if is_char(next, b'\n') || is_char(next, b'\r') || is_char(next, 0x0c) {- end = past_crlf(bytes, end + next_len, next);- return Unit {- ch: next,- len: end - at,- kind: Kind::Continuation,- };- }- if is_space(next) || (next_len == 1 && is_char(next, b'"')) {- return plain;- }- Unit {- ch: next,- len: len + next_len,- kind: Kind::CssEscaped,- }-}--fn lower(ch: u32) -> u32 {- u8::try_from(ch).map_or(ch, |b| u32::from(b.to_ascii_lowercase()))-}--/// Whether a remote URL starts at `at`, and how.-fn remote_start(bytes: &[u8], at: usize) -> Option<Start> {- let mut chars = [0u32; 6];- let (mut pos, mut count) = (at, 0);- while count < chars.len() && pos < bytes.len() {- let next = unit(bytes, pos);- if next.kind == Kind::StringEnd {- break;- }- pos += next.len;- if next.kind == Kind::Continuation {- continue;- }- chars[count] = lower(next.ch);- count += 1;- }- let slash = |ch: u32| is_char(ch, b'/') || is_char(ch, b'\\');- if count >= 2 && slash(chars[0]) && slash(chars[1]) {- return Some(Start::Slashes);- }- // Outside CSS, `\\` is two backslashes rather than one escaped backslash, and a browser reads both as slashes.- let (first, first_len) = json_unit(bytes, at);- if slash(first) && at + first_len < bytes.len() && slash(json_unit(bytes, at + first_len).0) {- return Some(Start::Slashes);- }- let http = chars[..4]- .iter()- .zip(b"http")- .all(|(&ch, &b)| is_char(ch, b));- let colon_at = if is_char(chars[4], b's') { 5 } else { 4 };- (count > colon_at && http && is_char(chars[colon_at], b':')).then_some(Start::Scheme)-}--fn closer_after(prev: Option<Unit>) -> Closer {- match prev {- Some(Unit {- ch,- kind: Kind::Plain,- ..- }) if is_char(ch, b'(') => Closer::Paren,- Some(Unit {- ch,- kind: Kind::Plain,- ..- }) if is_char(ch, b'\'') => Closer::Apostrophe,- Some(Unit {- ch,- kind: Kind::Plain,- ..- }) if is_char(ch, b'"') => Closer::Quote,- Some(Unit {- kind: Kind::StringEnd,- ..- }) => Closer::Nothing,- _ => Closer::Whitespace,- }-}--fn ends_url(next: Unit, closer: Closer) -> bool {- match next.kind {- Kind::StringEnd => true,- Kind::CssEscaped | Kind::Continuation => false,- Kind::Plain => match closer {- Closer::Whitespace => is_space(next.ch),- Closer::Nothing => false,- Closer::Paren => is_char(next.ch, b')'),- Closer::Apostrophe => is_char(next.ch, b'\''),- Closer::Quote => is_char(next.ch, b'"'),- },- }-}--/// The spans of the remote URL tokens, in order.-fn url_spans(bytes: &[u8]) -> Vec<(usize, usize)> {- scan(bytes).urls-}--/// The remote URL tokens of the bytes and the insides of their JSON strings, each in order.-struct Scan {- urls: Vec<(usize, usize)>,- strings: Vec<(usize, usize)>,-}--fn scan(bytes: &[u8]) -> Scan {- let mut spans = Vec::new();- let mut strings = Vec::new();- let mut string_start: Option<usize> = None;- let mut prev: Option<Unit> = None;- let mut at = 0;- while at < bytes.len() {- let current = unit(bytes, at);- if current.kind == Kind::StringEnd {- match string_start.take() {- Some(start) => strings.push((start, at)),- None => string_start = Some(at + 1),- }- }- if current.kind == Kind::Continuation {- at += current.len;- continue;- }- let first = lower(current.ch);- let starts = string_start.is_some()- && current.kind != Kind::StringEnd- && (is_char(first, b'h') || is_char(first, b'/') || is_char(first, b'\\'))- && match remote_start(bytes, at) {- Some(Start::Scheme) => true,- Some(Start::Slashes) => !prev.is_some_and(|p| continues_word(p.ch)),- None => false,- };- if !starts {- prev = Some(current);- at += current.len;- continue;- }- let closer = closer_after(prev);- let mut end = at;- while end < bytes.len() {- let next = unit(bytes, end);- if ends_url(next, closer) {- break;- }- end += next.len;- }- spans.push((at, end));- prev = None;- at = end;- }- Scan {- urls: spans,- strings,- }-}--/// The token's characters after their JSON and CSS escapes, in lower case, so an escaped host still matches.-fn decoded(bytes: &[u8], (start, end): (usize, usize)) -> String {- let mut text = String::new();- let mut at = start;- while at < end {- let next = unit(bytes, at);- if next.kind != Kind::Continuation {- text.extend(char::from_u32(lower(next.ch)));- }- at += next.len;- }- text-}--/// Whether the byte before `at` closes a JSON escape that ends a token, such as `\n`, rather than an escaped backslash.-fn escape_ends_at(bytes: &[u8], at: usize) -> bool {- at >= 2- && matches!(bytes[at - 1], b'n' | b'r' | b't' | b'b' | b'f')- && bytes[..at - 1]- .iter()- .rev()- .take_while(|&&b| b == b'\\')- .count()- % 2- == 1-}--/// The span around a host that no URL token holds, such as a bare host in text. It ends at whitespace and quotes, which-/// never sit inside an escape, so the span keeps every escape whole.-fn bare_span(bytes: &[u8], start: usize, end: usize) -> (usize, usize) {- let mut first = start;- while first > 0- && bytes[first - 1] > b' '- && bytes[first - 1] != b'"'- && !escape_ends_at(bytes, first)- {- first -= 1;- }- let mut last = end;- while last < bytes.len() && bytes[last] > b' ' && bytes[last] != b'"' {- if bytes[last] == b'\\' {- let (ch, len) = json_unit(bytes, last);- if len > 1 && (is_space(ch) || is_char(ch, b'"')) {- break;- }- last += len;- } else {- last += 1;- }- }- (first, last.min(bytes.len()))-}--impl Scrubber {- pub fn new(scrub: &Scrub) -> Self {- let hosts = match scrub {- Scrub::Hosts(hosts) if !hosts.is_empty() => {- let mut hosts: Vec<String> =- hosts.iter().map(|host| host.to_ascii_lowercase()).collect();- hosts.sort_unstable();- hosts.dedup();- AhoCorasickBuilder::new()- .ascii_case_insensitive(true)- .match_kind(MatchKind::LeftmostFirst)- .build(&hosts)- .ok()- }- _ => None,- };- Self {- all_remote: *scrub == Scrub::AllRemote,- hosts,- }- }-- pub fn replaces_tokens(&self) -> bool {- self.all_remote || self.hosts.is_some()- }-- /// The bytes with each URL token replaced, and how many were.- pub fn apply<'b>(&self, bytes: &'b [u8]) -> (Cow<'b, [u8]>, u64) {- let spans = if self.all_remote {- url_spans(bytes)- } else if let Some(hosts) = &self.hosts {- Self::host_spans(hosts, bytes)- } else {- Vec::new()- };- if spans.is_empty() {- return (Cow::Borrowed(bytes), 0);- }- let mut out = Vec::with_capacity(bytes.len());- let mut copied_to = 0;- let mut replaced = 0;- for (start, end) in spans {- if end <= copied_to {- continue;- }- if start >= copied_to {- out.extend_from_slice(&bytes[copied_to..start]);- out.extend_from_slice(URL_UNCHECKED_MARKER.as_bytes());- replaced += 1;- }- copied_to = end;- }- out.extend_from_slice(&bytes[copied_to..]);- (Cow::Owned(out), replaced)- }-- /// The URL tokens that hold a host, and a bare span around each host no URL token holds, by start.- fn host_spans(hosts: &AhoCorasick, bytes: &[u8]) -> Vec<(usize, usize)> {- let Scan { urls, strings } = scan(bytes);- let mut spans: Vec<(usize, usize)> = urls- .into_iter()- .filter(|&span| hosts.is_match(decoded(bytes, span).as_bytes()))- .collect();- let inside = |list: &[(usize, usize)], (start, end): (usize, usize)| {- let first = list.partition_point(|&(_, list_end)| list_end < end);- list.get(first)- .filter(|&&(list_start, list_end)| list_start <= start && end <= list_end)- .copied()- };- let mut bare = Vec::new();- let mut covered_to = 0;- for found in hosts.find_iter(bytes) {- let host = (found.start(), found.end());- // A host outside a JSON string is part of a literal such as `null`, and an earlier span may already hold it.- if host.0 < covered_to || inside(&spans, host).is_some() {- continue;- }- let Some((string_start, string_end)) = inside(&strings, host) else {- continue;- };- let (start, end) = bare_span(bytes, host.0, host.1);- let span = (start.max(string_start), end.min(string_end));- covered_to = span.1;- bare.push(span);- }- spans.extend(bare);- spans.sort_unstable();- spans- }-}--#[cfg(test)]-mod tests {- use super::*;-- fn scrub(scrub: Scrub, bytes: &str) -> (String, u64) {- let (out, count) = Scrubber::new(&scrub).apply(bytes.as_bytes());- let out = String::from_utf8(out.into_owned()).unwrap();- serde_json::from_str::<serde_json::Value>(&out).unwrap_or_else(|e| panic!("{out}: {e}"));- (out, count)- }-- #[test]- fn replaces_every_remote_url_token_and_keeps_the_json_around_it() {- let m = "imageurl:v4:declined:unchecked";- for (input, expected) in [- (- r#"{"src":"https://cdn.example.com/a.png?x=1"}"#.to_owned(),- format!(r#"{{"src":"{m}"}}"#),- ),- (- r#"{"srcset":"HTTP://a.example.com/a.png 1x,\nhttps://b.example.com/b.png 2x"}"#- .to_owned(),- format!(r#"{{"srcset":"{m}"}}"#),- ),- // A space inside a URL does not end it, so the query after the space goes too.- (- r#"{"src":"https://cdn.example.com/a b.png?access_token=x"}"#.to_owned(),- format!(r#"{{"src":"{m}"}}"#),- ),- (- r#"{"style":"background:url(\"https://cdn.example.com/a b.png?access_token=x\")"}"#- .to_owned(),- format!(r#"{{"style":"background:url(\"{m}\")"}}"#),- ),- // Inside text, whitespace still ends a URL.- (- r#"{"alt":"see https://a.example.com/a and https://b.example.com/b now"}"#- .to_owned(),- format!(r#"{{"alt":"see {m} and {m} now"}}"#),- ),- (- r#"{"style":"background:url(\"//cdn.example.com/a.png\")"}"#.to_owned(),- format!(r#"{{"style":"background:url(\"{m}\")"}}"#),- ),- (- r#"{"style":"url('https://cdn.example.com/a.png') no-repeat"}"#.to_owned(),- format!(r#"{{"style":"url('{m}') no-repeat"}}"#),- ),- // A CSS escape is part of the URL a browser loads, the whitespace that ends a hex escape included.- (- r#"{"style":"url(https\\3a //cdn.example.com/a.png)"}"#.to_owned(),- format!(r#"{{"style":"url({m})"}}"#),- ),- (- r#"{"style":"url(\\2f\\2f bucket.example.com/a.png?sig=x)"}"#.to_owned(),- format!(r#"{{"style":"url({m})"}}"#),- ),- // A CSS string drops an escaped line break, so a browser reads a URL that one splits whole.- (- r#"{"style":"url(\"htt\\\nps://cdn.example.com/a.png?token=x\")"}"#.to_owned(),- format!(r#"{{"style":"url(\"{m}\")"}}"#),- ),- (- r#"{"style":"url('ht\\\r\ntps://cdn.example.com/a.png?token=x')"}"#.to_owned(),- format!(r#"{{"style":"url('{m}')"}}"#),- ),- (- r#"{"style":"url(\"\\68\r\nttps://cdn.example.com/a.png?token=x\")"}"#.to_owned(),- format!(r#"{{"style":"url(\"{m}\")"}}"#),- ),- // A browser reads a backslash as a slash, and the inspector reads any host after two of them as remote.- (- r#"{"src":"\\\\10.0.0.7/a.png"}"#.to_owned(),- format!(r#"{{"src":"{m}"}}"#),- ),- (- r#"{"src":"\/\/[::1]/a.png"}"#.to_owned(),- format!(r#"{{"src":"{m}"}}"#),- ),- // Outside CSS, an apostrophe or a parenthesis is part of the URL, so the query after it goes too.- (- r#"{"src":"https://cdn.example.com/a'b(c).png?token=x"}"#.to_owned(),- format!(r#"{{"src":"{m}"}}"#),- ),- // An escaped backslash at the end goes whole, so the quote after it still ends the string.- (- r#"{"src":"https://cdn.example.com/a\\"}"#.to_owned(),- format!(r#"{{"src":"{m}"}}"#),- ),- // `http` inside a word, and a path's `//`, start no URL.- (- r#"{"text":"httpd at /a//b"}"#.to_owned(),- r#"{"text":"httpd at /a//b"}"#.to_owned(),- ),- ] {- assert_eq!(scrub(Scrub::AllRemote, &input).0, expected, "{input}");- }- }-- #[test]- fn replaces_the_tokens_that_hold_a_host_in_any_form() {- let m = "imageurl:v4:declined:unchecked";- let (out, count) = scrub(- Scrub::Hosts(vec!["internal.example.com".to_owned()]),- r#"{"style":"url(https\\3a \\2f \\2f Internal.example.com\\2f a.png)","alt":"see internal.example.com/a'b?token=x\nnow","title":"other.example.com"}"#,- );- assert_eq!(- out,- format!(r#"{{"style":"url({m})","alt":"see {m}\nnow","title":"other.example.com"}}"#)- );- assert_eq!(count, 2);- }-- #[test]- fn matches_a_host_only_as_the_browser_reads_it_inside_a_string() {- let m = "imageurl:v4:declined:unchecked";- for (host, input, expected) in [- (- "10.0.0.7",- r#"{"style":"url(https://10\\2e 0.0.7/a.png)"}"#.to_owned(),- format!(r#"{{"style":"url({m})"}}"#),- ),- (- "null",- r#"{"src":"https://null/a.png","alt":null}"#.to_owned(),- format!(r#"{{"src":"{m}","alt":null}}"#),- ),- ] {- assert_eq!(- scrub(Scrub::Hosts(vec![host.to_owned()]), &input).0,- expected,- "{input}"- );- }- }-}diff --git a/rust/ai-research/replay-recorder-v4/src/splice.rs b/rust/ai-research/replay-recorder-v4/src/splice.rsnew file mode 100644index 000000000000..ae2983572b70--- /dev/null+++ b/rust/ai-research/replay-recorder-v4/src/splice.rs@@ -0,0 +1,367 @@+//! The writer's one pass over an event. It copies the event, and where the inspector read each slot, it writes the+//! slot's references and markers, so it searches the event for no URL. A URL the policy refuses must leave the block+//! wherever it is, so the writer also overwrites each copy of one, which only a block that holds one pays for.++use std::borrow::Cow;+use std::collections::{BTreeMap, HashMap, VecDeque};++use aho_corasick::{AhoCorasick, AhoCorasickKind, MatchKind};+use metrics::counter;+use replay_core_v4::bytescan::{unescape, write_json_string, Span};+use replay_core_v4::inspect::css::{escape_in_identifier, write_css_edit, CssEdit};+use replay_core_v4::inspect::dom::{for_each_srcset_candidate, DeclinedUrl, RemoteKind};+use replay_core_v4::inspect::edits::{Edit, EditSource, EventEdits, Slot};+use replay_core_v4::inspect::srcset::{shown_candidate, SrcsetSlot};+use replay_core_v4::refs::{declined_url_ref, UrlAsset};+use replay_core_v4::seal::image_url_key;+use replay_core_v4::url_policy::is_remote;+use replay_core_v4::v4_image_refs::{bundled_font_ref, ImageDecline, IMAGE_DECLINED_PREFIX};++use crate::urls::url_reference;++fn asset_of(kind: RemoteKind) -> UrlAsset {+ match kind {+ RemoteKind::Image => UrlAsset::Image,+ RemoteKind::Font | RemoteKind::BundledFont(_) => UrlAsset::Font,+ }+}++fn kind_label(kind: RemoteKind) -> &'static str {+ match kind {+ RemoteKind::Image => "image",+ RemoteKind::Font => "font",+ RemoteKind::BundledFont(_) => "bundled_font",+ }+}++fn json_string_contents(text: &str) -> Vec<u8> {+ let mut quoted = Vec::with_capacity(text.len() + 2);+ write_json_string(text, &mut quoted);+ quoted[1..quoted.len() - 1].to_vec()+}++#[derive(Clone)]+struct Named {+ text: String,+ /// The policy's reason, for a URL it declines.+ decline: Option<&'static str>,+}++#[derive(Debug, Default, PartialEq, Eq)]+pub struct SpliceCounters {+ srcset_rewrites: BTreeMap<&'static str, u64>,+ css_rewrites: BTreeMap<&'static str, u64>,+ url_rewrites: BTreeMap<(&'static str, &'static str, &'static str), u64>,+ pub removed_images: [u64; ImageDecline::ALL.len()],+}++impl SpliceCounters {+ pub fn emit(&self) {+ for (&outcome, &count) in &self.srcset_rewrites {+ counter!("ai_research_v4_recorder_srcset_rewrites_total", "outcome" => outcome)+ .increment(count);+ }+ for (&kind, &count) in &self.css_rewrites {+ counter!(+ "ai_research_v4_recorder_css_rewrites_total",+ "kind" => kind,+ "outcome" => "rewritten",+ )+ .increment(count);+ }+ for (&(kind, outcome, reason), &count) in &self.url_rewrites {+ counter!(+ "ai_research_v4_recorder_image_url_rewrites_total",+ "kind" => kind,+ "outcome" => outcome,+ "reason" => reason,+ )+ .increment(count);+ }+ for (reason, &count) in ImageDecline::ALL.iter().zip(&self.removed_images) {+ counter!("ai_research_v4_recorder_removed_images_total", "reason" => reason.label())+ .increment(count);+ }+ }+}++/// Refused URLs that hold one another can end many matches at each byte, so a block past this many is dropped.+const MAX_BLOCK_DECLINED_MATCHES: usize = 1_000_000;++/// Overwrites every copy of the block's refused URLs, as a JSON string holds it and as a CSS selector escapes it.+pub struct DeclinedCopies {+ matcher: AhoCorasick,+ markers: Vec<Vec<u8>>,+ longest: usize,+ matches_left: usize,+}++#[derive(Clone, Copy)]+struct Overwrite {+ start: usize,+ end: usize,+ marker: usize,+}++impl DeclinedCopies {+ pub fn new(declined: &[DeclinedUrl]) -> Option<Self> {+ let mut patterns = Vec::new();+ let mut markers = Vec::new();+ for DeclinedUrl { url, kind, decline } in declined {+ let marker = declined_url_ref(asset_of(*kind), *decline);+ // A selector copy takes an escaped marker, so that its colons do not start pseudo-classes.+ for (form, replacement) in [+ (url.clone(), marker.clone()),+ (escape_in_identifier(url), escape_in_identifier(&marker)),+ ] {+ patterns.push(json_string_contents(&form));+ markers.push(json_string_contents(&replacement));+ }+ }+ let matcher = AhoCorasick::builder()+ .kind(Some(AhoCorasickKind::ContiguousNFA))+ .match_kind(MatchKind::Standard)+ .build(&patterns)+ .ok()?;+ let longest = patterns.iter().map(Vec::len).max()?;+ Some(Self {+ matcher,+ markers,+ longest,+ matches_left: MAX_BLOCK_DECLINED_MATCHES,+ })+ }++ /// Copies that overlap share one span, so that a refused URL that starts inside another one is overwritten too.+ pub fn apply<'b>(&mut self, bytes: &'b [u8], overwritten: &mut u64) -> Option<Cow<'b, [u8]>> {+ let mut out = None;+ let mut copied_to = 0;+ let mut pending: VecDeque<Overwrite> = VecDeque::new();+ // Matches come in order of their end, so a later match cannot reach a span that ends `longest` bytes before.+ for found in self.matcher.find_overlapping_iter(bytes) {+ self.matches_left = self.matches_left.checked_sub(1)?;+ let mut span = Overwrite {+ start: found.start(),+ end: found.end(),+ marker: found.pattern().as_usize(),+ };+ while let Some(last) = pending.back().copied().filter(|last| last.end > span.start) {+ pending.pop_back();+ if last.start <= span.start {+ span.start = last.start;+ span.marker = last.marker;+ }+ }+ pending.push_back(span);+ while let Some(first) = pending+ .front()+ .copied()+ .filter(|first| first.end + self.longest <= span.end)+ {+ pending.pop_front();+ self.write(bytes, first, &mut out, &mut copied_to);+ *overwritten += 1;+ }+ }+ for span in pending {+ self.write(bytes, span, &mut out, &mut copied_to);+ *overwritten += 1;+ }+ Some(match out {+ Some(mut out) => {+ out.extend_from_slice(&bytes[copied_to..]);+ Cow::Owned(out)+ }+ None => Cow::Borrowed(bytes),+ })+ }++ fn write(+ &self,+ bytes: &[u8],+ span: Overwrite,+ out: &mut Option<Vec<u8>>,+ copied_to: &mut usize,+ ) {+ let out = out.get_or_insert_with(|| Vec::with_capacity(bytes.len()));+ out.extend_from_slice(&bytes[*copied_to..span.start]);+ out.extend_from_slice(&self.markers[span.marker]);+ *copied_to = span.end;+ }+}++pub struct Splicer<'b> {+ edits: &'b [EventEdits],+ next: usize,+ team_id: i64,+ key_month: String,+ url_key: [u8; 32],+ named: HashMap<(String, RemoteKind), Named>,+ pub counters: SpliceCounters,+}++impl<'b> Splicer<'b> {+ pub fn new(+ edits: &'b [EventEdits],+ team_id: i64,+ key_month: &str,+ team_key: &[u8; 32],+ ) -> Self {+ Self {+ edits,+ next: 0,+ team_id,+ key_month: key_month.to_owned(),+ url_key: image_url_key(team_key),+ named: HashMap::new(),+ counters: SpliceCounters::default(),+ }+ }++ /// The edits of the block's event at `event`. Events come in order.+ pub fn edits_of(&mut self, event: u32) -> Option<&'b [Edit]> {+ while self.edits.get(self.next).is_some_and(|e| e.event < event) {+ self.next += 1;+ }+ let edits = self.edits.get(self.next).filter(|e| e.event == event)?;+ Some(&edits.edits)+ }++ /// Copies `range` of `bytes`, the bytes that `source` names, into `out`, with each edit inside it written in place.+ pub fn copy(+ &mut self,+ edits: &[Edit],+ source: EditSource,+ bytes: &[u8],+ range: Span,+ out: &mut Vec<u8>,+ ) {+ let mut copied_to = range.0;+ let first = edits.partition_point(|edit| (edit.source, edit.value.0) < (source, range.0));+ for edit in edits[first..]+ .iter()+ .take_while(|edit| edit.source == source && edit.value.0 < range.1)+ {+ if edit.value.0 < copied_to || edit.value.1 > range.1 {+ continue;+ }+ let Ok(text) = unescape(bytes, edit.value) else {+ continue;+ };+ if let Some(replacement) = self.replacement(&edit.slot, &text) {+ out.extend_from_slice(&bytes[copied_to..edit.value.0]);+ write_json_string(&replacement, out);+ copied_to = edit.value.1;+ }+ }+ out.extend_from_slice(&bytes[copied_to..range.1]);+ }++ fn named(&mut self, url: &str, kind: RemoteKind) -> Named {+ if let Some(named) = self.named.get(&(url.to_owned(), kind)) {+ return named.clone();+ }+ let asset = asset_of(kind);+ let named = match (+ kind,+ url_reference(url, asset, self.team_id, &self.key_month, &self.url_key),+ ) {+ (RemoteKind::BundledFont(slug), named) => Named {+ text: bundled_font_ref(slug),+ decline: named.err().map(|decline| decline.label()),+ },+ (_, Ok((_, reference))) => Named {+ text: reference,+ decline: None,+ },+ (_, Err(decline)) => Named {+ text: declined_url_ref(asset, decline),+ decline: Some(decline.label()),+ },+ };+ let outcome = if named.decline.is_some() {+ "declined"+ } else {+ "rewritten"+ };+ *self+ .counters+ .url_rewrites+ .entry((kind_label(kind), outcome, named.decline.unwrap_or("none")))+ .or_default() += 1;+ self.named.insert((url.to_owned(), kind), named.clone());+ named+ }++ fn replacement(&mut self, slot: &Slot, text: &str) -> Option<String> {+ match slot {+ Slot::Url => {+ let url = text.trim();+ is_remote(url).then(|| self.named(url, RemoteKind::Image).text)+ }+ Slot::Srcset(slot) => self.rebuilt_srcset(text, *slot),+ Slot::Css(edit) => Some(self.written_css(text, edit)),+ }+ }++ fn written_css(&mut self, text: &str, edit: &CssEdit) -> String {+ for piece in &edit.pieces {+ if piece.parts.len() > 1 {+ let function = &text[piece.range.clone()];+ let kind = if function+ .get(..4)+ .is_some_and(|start| start.eq_ignore_ascii_case("url("))+ {+ "relative_url"+ } else {+ "image_set"+ };+ *self.counters.css_rewrites.entry(kind).or_default() += 1;+ }+ }+ write_css_edit(text, edit, &mut |url| self.named(&url.url, url.kind).text)+ }++ /// The srcset with only the candidate it shows, a remote one as its reference. `None` when it has nothing to change.+ fn rebuilt_srcset(&mut self, text: &str, slot: SrcsetSlot) -> Option<String> {+ let shown = shown_candidate(text, slot).map(|kept| kept.index);+ let mut value = String::with_capacity(text.len());+ let (mut removed, mut remote) = (false, false);+ let mut index = 0;+ for_each_srcset_candidate(text, |url, descriptors| {+ let is_shown = shown == Some(index);+ index += 1;+ let written = if is_remote(url) {+ remote = true;+ is_shown+ .then(|| self.named(url, RemoteKind::Image))+ .filter(|named| named.decline.is_none())+ .map(|named| named.text)+ } else {+ (is_shown && !url.starts_with(IMAGE_DECLINED_PREFIX)).then(|| url.to_owned())+ };+ match written {+ Some(written) => {+ if !value.is_empty() {+ value.push_str(", ");+ }+ value.push_str(&written);+ if !descriptors.is_empty() {+ value.push(' ');+ value.push_str(descriptors);+ }+ }+ None => removed = true,+ }+ });+ if removed {+ let outcome = if value.is_empty() {+ "emptied"+ } else {+ "trimmed"+ };+ *self.counters.srcset_rewrites.entry(outcome).or_default() += 1;+ }+ (removed || remote).then_some(value)+ }+}diff --git a/rust/ai-research/replay-recorder-v4/src/urls.rs b/rust/ai-research/replay-recorder-v4/src/urls.rsindex f5f1cb549c8b..4709a8438072 100644--- a/rust/ai-research/replay-recorder-v4/src/urls.rs+++ b/rust/ai-research/replay-recorder-v4/src/urls.rs@@ -7,9 +7,7 @@ use metrics::counter;use replay_core_v4::keys::batch::{FlushKeys, SessionId};use replay_core_v4::refs::{public_font_url_ref, reference_hash, url_ref, UrlAsset};use replay_core_v4::seal::image_url_key;-use replay_core_v4::url_policy::{- is_public_host, is_tracking_beacon, try_canonicalize, CanonicalUrl, Decline,-};+use replay_core_v4::url_policy::{checked_url, CanonicalUrl, Decline};use serde_json::{json, Value};use replay_core_v4::inspect::stats::BlockStats;@@ -151,16 +149,7 @@ pub fn url_reference(key_month: &str,url_key: &[u8; 32],) -> Result<(CanonicalUrl, String), Decline> {- let url = try_canonicalize(raw)?;- let path = url::Url::parse(&url.fetch)- .map(|u| u.path().to_owned())- .unwrap_or_default();- if !is_public_host(&url.host) {- return Err(Decline::NonPublicHost);- }- if is_tracking_beacon(&url.host, &path) {- return Err(Decline::TrackingBeacon);- }+ let url = checked_url(raw)?;let public = match asset {UrlAsset::Font => public_font_url_ref(&url),UrlAsset::Image => None,@@ -239,6 +228,7 @@ pub fn sent_urls(capacity: u64) -> moka::sync::Cache<(String, String), ()> {#[cfg(test)]mod tests {use replay_core_v4::keys::batch::SessionKey as Key;+ use replay_core_v4::url_policy::try_canonicalize;use super::*;use crate::batch::{Origin, SessionKey, Tp};@@ -276,10 +266,10 @@ mod tests {moments: Vec::new(),image_urls: urls.iter().map(|u| (*u).to_owned()).collect(),font_urls: Vec::new(),- remote_urls: Vec::new(),url_limit: None,- srcsets: Vec::new(),- css_rewrites: Vec::new(),+ edits: Vec::new(),+ declined: Vec::new(),+ unplaced: false,visited_urls: Vec::new(),captured_at_ms: None,}diff --git a/rust/ai-research/replay-recorder-v4/src/write.rs b/rust/ai-research/replay-recorder-v4/src/write.rsindex a99ff2bcc299..3f537d82a03e 100644--- a/rust/ai-research/replay-recorder-v4/src/write.rs+++ b/rust/ai-research/replay-recorder-v4/src/write.rs@@ -3,6 +3,7 @@//! The order matters for a crash between writes. A pack without its blocks is unreferenced, which is harmless. A block//! whose images were never stored would hold references that resolve to nothing. The rows go out only after both.+use std::borrow::Cow;use std::collections::{BTreeMap, HashMap, HashSet};use std::io::{self, Write};use std::sync::Arc;@@ -18,17 +19,15 @@ use replay_core_v4::inspect::stats::BlockStats;use replay_core_v4::inspect::stats::MAX_URL_FINGERPRINTS;use replay_core_v4::keys::batch::{FlushKeys, SessionId, SessionKey as Key};use replay_core_v4::refs::url_fingerprint;-use replay_core_v4::scan::cv::extract_event_images;+use replay_core_v4::scan::cv::{extract_event_images, BeforeImages, Expansion};use replay_core_v4::scan::images::{Image, ImageLimits, ImageSink};use replay_core_v4::seal::{image_content_key, url_fingerprint_key};-use replay_core_v4::v4_image_refs::ImageDecline;use tokio::sync::Semaphore;use crate::batch::{Block, SessionKey};use crate::index::{BlockIndexRow, ImageIndexRow};-use crate::rewrite::{Fallback, RewriteReport, UrlRewrite};-use crate::scrub::Scrub;+use crate::splice::{DeclinedCopies, Splicer};use crate::store::{BlobStore, StoreError};/// Brotli settings for blocks, the same as the v3 lane, which uses the same C encoder. The lane keeps blocks for months@@ -67,13 +66,10 @@ pub struct WriterCounts {/// The bytes of the lines the writer gives the encoder, and the longest line.pub jsonl_bytes: u64,pub max_line_bytes: u64,- /// Why the block must not be stored, as the outcome of its drop: a URL the policy declines stays in it even after- /// the fallback.+ /// Why the block must not be stored, as the outcome of its drop: the inspector could not read some of it.pub dropped: Option<&'static str>,- /// Why the writer used a fallback to keep the block, as a metric label.- pub fallback: Option<&'static str>,- /// The URL tokens the fallback replaced.- pub scrubbed: u64,+ /// Copies of a refused URL that the writer overwrote outside its slot.+ pub overwritten: u64,}/// A block compressed once for all the attempts of its flush, with the images it moved out.@@ -119,23 +115,16 @@ fn prepare_block(block: &Block,team_key: &[u8; 32],key_month: &str,- fallback: &Fallback,out: &mut impl Write,-) -> io::Result<(Vec<Image>, WriterCounts, RewriteReport)> {+) -> io::Result<(Vec<Image>, WriterCounts)> {let content_key = image_content_key(team_key);- let mut rewrite = UrlRewrite::new(- &block.remote_urls,- &block.srcsets,- &block.css_rewrites,- block.session.team_id,- key_month,- team_key,- )- .with_fallback(fallback.clone());+ let mut splicer = Splicer::new(&block.edits, block.session.team_id, key_month, team_key);+ let mut declined = DeclinedCopies::new(&block.declined);+ let mut too_many_matches = false;let mut images = Vec::new();let mut kept = HashSet::new();let mut counts = WriterCounts::default();- let mut removed_images = [0u64; ImageDecline::ALL.len()];+ let mut event_index = 0;for message in &block.messages {let mut window = Vec::new();write_json_string(@@ -149,18 +138,46 @@ fn prepare_block(ImageLimits::default(),);for event in &message.events {- let extracted = extract_event_images(&message.inner, event, &mut sink);+ let mut overwrite =+ |bytes: &[u8]| match declined.as_mut()?.apply(bytes, &mut counts.overwritten) {+ Some(Cow::Owned(overwritten)) => Some(overwritten),+ Some(Cow::Borrowed(_)) => None,+ None => {+ too_many_matches = true;+ None+ }+ };+ let before_images: Option<&mut BeforeImages<'_>> = Some(&mut overwrite);+ let extracted = match splicer.edits_of(event_index) {+ Some(edits) => {+ let mut copy = |source, bytes: &[u8], range, out: &mut Vec<u8>| {+ splicer.copy(edits, source, bytes, range, out)+ };+ extract_event_images(+ &message.inner,+ event,+ &mut sink,+ Some(&mut copy),+ before_images,+ )+ }+ None => extract_event_images(&message.inner, event, &mut sink, None, before_images),+ };+ event_index += 1;+ if too_many_matches {+ let counts = WriterCounts {+ dropped: Some("too_many_declined_copies"),+ ..WriterCounts::default()+ };+ return Ok((Vec::new(), counts));+ }+ let event_bytes = extracted+ .as_deref()+ .unwrap_or(&message.inner[event.span.0..event.span.1]);out.write_all(b"[")?;out.write_all(&window)?;out.write_all(b",")?;- let stored = extracted- .as_deref()- .unwrap_or(&message.inner[event.span.0..event.span.1]);- // The inspector reads image slots in full snapshots and mutations only.- let dom_event =- event.kind == Some(2) || (event.kind == Some(3) && event.source == Some(0));- let event_bytes = rewrite.apply(stored, dom_event);- out.write_all(&event_bytes)?;+ out.write_all(event_bytes)?;out.write_all(b"]\n")?;let line = (window.len() + event_bytes.len() + 4) as u64;counts.jsonl_bytes += line;@@ -173,7 +190,12 @@ fn prepare_block(.map(|i| i.bytes.len() as u64).sum::<u64>();counts.inline += sink.left_inline;- for (removed, declined) in removed_images.iter_mut().zip(sink.declined) {+ for (removed, declined) in splicer+ .counters+ .removed_images+ .iter_mut()+ .zip(sink.declined)+ {*removed += u64::from(declined);}images.extend(@@ -182,98 +204,51 @@ fn prepare_block(.filter(|image| kept.insert(image.reference.clone())),);}- let mut report = rewrite.finish();- report.counters.removed_images = removed_images;- counts.scrubbed = report.scrubbed;- Ok((images, counts, report))+ splicer.counters.emit();+ Ok((images, counts))}-/// A block whose URLs the inspector did not all name has every remote URL token it did not name scrubbed, so that the-/// block is kept without a URL nobody checked.-fn first_fallback(block: &Block) -> Fallback {- match block.url_limit {- // Past the font budget the inspector still named every URL it reads except font sources, so links and text keep theirs.- Some(limit) if limit.is_font_budget() => Fallback {- overwrite_declined: true,- scrub: Scrub::Nothing,- scrub_font_sources: true,- reason: Some(limit.label()),- },- Some(limit) => Fallback {- overwrite_declined: true,- scrub: Scrub::AllRemote,- scrub_font_sources: false,- reason: Some(limit.label()),- },- None => Fallback::default(),+/// Why a block must not be stored: the inspector could not read some of it, so a URL there went unchecked.+fn unread(block: &Block) -> Option<&'static str> {+ if block.unplaced {+ return Some("unplaced_slot");}-}--/// What the second pass does about what the first left unsafe: it overwrites each declined URL wherever it is, and-/// scrubs the tokens that hold the host of a declined URL the stored bytes do not hold as the inspector read it.-fn second_fallback(first: &Fallback, report: &RewriteReport) -> Fallback {- let scrub = match (&first.scrub, &report.unfound_hosts) {- (Scrub::AllRemote, _) | (_, None) => Scrub::AllRemote,- (_, Some(hosts)) if hosts.is_empty() => Scrub::Nothing,- (_, Some(hosts)) => Scrub::Hosts(hosts.clone()),- };- let reason = if report.unfound_hosts.as_ref().is_some_and(Vec::is_empty) {- "declined_kept"- } else {- "declined_unfound"- };- Fallback {- overwrite_declined: true,- scrub,- scrub_font_sources: first.scrub_font_sources,- reason: first.reason.or(Some(reason)),+ if let Some(limit) = block.url_limit {+ return Some(limit.label());}+ // A block stores such an event as it arrived, with its compressed strings unread.+ let undecoded = block+ .messages+ .iter()+ .flat_map(|message| &message.events)+ .any(|event| {+ matches!(+ event.expansion,+ Some(Expansion::Skipped | Expansion::Failed)+ )+ });+ undecoded.then_some("undecoded_events")}-/// Writes a block's lines to an output from `open`. A block that the first pass leaves unsafe is written again to a-/// fresh output with the fallback that makes it safe, so a block is dropped only when even that fails, or when its-/// limit leaves nothing a fallback can make safe. Moves the images out and returns them with the output.+/// Writes a block's lines to an output from `open`, or drops a block that the inspector could not read in full.+/// A dropped block gets a new output, so that none of its lines are stored.pub fn write_block<W: Write>(block: &Block,team_key: &[u8; 32],key_month: &str,mut open: impl FnMut() -> io::Result<W>,) -> io::Result<(W, Vec<Image>, WriterCounts)> {- if let Some(limit) = block.url_limit.filter(|limit| limit.drops_block()) {+ if let Some(reason) = unread(block) {let counts = WriterCounts {- dropped: Some(limit.label()),+ dropped: Some(reason),..WriterCounts::default()};return Ok((open()?, Vec::new(), counts));}- let first = first_fallback(block);- let mut out = open()?;- let (images, mut counts, report) = prepare_block(block, team_key, key_month, &first, &mut out)?;- if report.safe {- report.counters.emit();- counts.fallback = first.reason;- return Ok((out, images, counts));- }- let second = second_fallback(&first, &report);- if let Some(key) = &report.kept_in {- tracing::info!(- team_id = block.session.team_id,- session_id = %block.session.session_id,- key,- "a declined URL stayed inside a longer token"- );- }- // The second pass copies every image again, so the first pass's output and images go before it starts.- drop(out);- drop(images);let mut out = open()?;- let (images, mut counts, report) =- prepare_block(block, team_key, key_month, &second, &mut out)?;- report.counters.emit();- counts.fallback = second.reason;- // Fail closed only when the fallback could not make the block safe either.- if !report.safe {- counts.dropped = Some("declined_unrewritten");+ let (images, counts) = prepare_block(block, team_key, key_month, &mut out)?;+ if counts.dropped.is_some() {+ return Ok((open()?, images, counts));}Ok((out, images, counts))}@@ -325,11 +300,8 @@ pub fn compress_block(};let (lines, images, counts) =write_block(block, team_key, key_month, open).map_err(seal_error)?;- if let Some(reason) = counts.fallback {- counter!("ai_research_v4_recorder_url_fallbacks_total", "reason" => reason).increment(1);- counter!("ai_research_v4_recorder_scrubbed_urls_total", "reason" => reason)- .increment(counts.scrubbed);- }+ counter!("ai_research_v4_recorder_overwritten_declined_copies_total")+ .increment(counts.overwritten);let encoder = lines.into_inner().map_err(|e| seal_error(e.into_error()))?;let body = encoder.into_inner()@@ -733,9 +705,6 @@ mod tests {use super::*;use crate::batch::{Origin, Tp};- use replay_core_v4::inspect::dom::{- RemoteKind, RemoteUrl, MAX_BLOCK_FONT_URLS, MAX_BLOCK_REMOTE_URLS,- };use replay_core_v4::refs::UrlAsset;#[derive(Default)]@@ -813,10 +782,10 @@ mod tests {moments: Vec::new(),image_urls: Vec::new(),font_urls: Vec::new(),- remote_urls: Vec::new(),url_limit: None,- srcsets: Vec::new(),- css_rewrites: Vec::new(),+ edits: Vec::new(),+ declined: Vec::new(),+ unplaced: false,visited_urls: Vec::new(),captured_at_ms: None,}@@ -832,38 +801,40 @@ mod tests {}/// The block's stored lines, after the inspector named its remote URLs.- fn stored_lines(events: &str, extra: &[&str]) -> String {- stored(block_of("kept", events), extra)+ fn stored_lines(events: &str) -> String {+ stored(block_of("kept", events))}- fn stored(inspected: Block, extra: &[&str]) -> String {- let (lines, counts) = written(inspected, extra);+ fn stored(inspected: Block) -> String {+ let (lines, counts) = written(inspected);if counts.dropped.is_some() {return String::new();}lines}- fn written(mut inspected: Block, extra: &[&str]) -> (String, WriterCounts) {+ fn written(block: Block) -> (String, WriterCounts) {+ let inspected = inspected(block);+ let (lines, _, counts) =+ write_block(&inspected, &[6; 32], "2026-10", || Ok(Vec::new())).unwrap();+ (String::from_utf8(lines).unwrap(), counts)+ }++ /// The block with what the session inspector finds in it, as a flush hands it to the writer.+ fn inspected(mut inspected: Block) -> Block {let mut inspector = replay_core_v4::inspect::session::SessionInspector::default();let mut inspection = replay_core_v4::inspect::session::BlockInspection::default();for message in &inspected.messages {inspector.observe_message(&mut inspection, message);}let found = inspection.finish();- inspected.remote_urls = found.remote_urls;+ inspected.image_urls = found.image_urls;+ inspected.font_urls = found.font_urls;inspected.url_limit = found.url_limit;- inspected.srcsets = found.srcsets;- inspected.css_rewrites = found.css_rewrites;+ inspected.edits = found.edits;+ inspected.declined = found.declined;+ inspected.unplaced = found.unplaced;inspected- .remote_urls- .extend(extra.iter().map(|url| RemoteUrl {- url: (*url).to_owned(),- kind: RemoteKind::Image,- }));- let (lines, _, counts) =- write_block(&inspected, &[6; 32], "2026-10", || Ok(Vec::new())).unwrap();- (String::from_utf8(lines).unwrap(), counts)}#[test]@@ -876,7 +847,7 @@ mod tests {let events = json!([{"type": 3, "timestamp": 1, "data": {"source": 0, "adds": [], "removes": [], "texts": [],"attributes": [{"id": 99, "attributes": {"src": signed, "srcset": format!("{public} 2x")}}]}}]).to_string();- let lines = stored_lines(&events, &[]);+ let lines = stored_lines(&events);let reference = crate::urls::url_reference(public,replay_core_v4::refs::UrlAsset::Image,@@ -900,7 +871,7 @@ mod tests {.to_string()};// A URL too long to fetch is declined, and the block is kept.- let long_lines = stored_lines(&long(16 * 1024), &[]);+ let long_lines = stored_lines(&long(16 * 1024));assert!(long_lines.contains("imageurl:v4:declined:too_long"),"{long_lines}"@@ -909,7 +880,7 @@ mod tests {let escaped = events.replace(r#""src""#, r#""\u0073rc""#).replace(r#""srcset""#, r#""src\u0073et""#);- assert_eq!(stored_lines(&escaped, &[]), lines);+ assert_eq!(stored_lines(&escaped), lines);// A browser reads backslashes after the scheme as slashes, so such a URL is replaced like its `https://` form.for backslashed in [r"https:\\user:[email protected]/a.png",@@ -919,7 +890,7 @@ mod tests {let events = json!([{"type": 3, "timestamp": 1, "data": {"source": 0, "adds": [], "removes": [], "texts": [],"attributes": [{"id": 99, "attributes": {"src": backslashed}}]}}]).to_string();- let lines = stored_lines(&events, &[]);+ let lines = stored_lines(&events);assert!(lines.contains("imageurl:v4:declined:credential") && !lines.contains("secret"),"{lines}"@@ -934,27 +905,26 @@ mod tests {r#"[{{"type":3,"timestamp":1,"data":{{"source":0,"adds":[{{"parentId":1,"node":{{"type":2,"tagName":"img","attributes":{{"src":"{public}","src":"{signed}"}},"childNodes":[],"id":5}}}}],"removes":[],"texts":[],"attributes":[]}}}}]"#),] {- let lines = stored_lines(&events, &[]);+ let lines = stored_lines(&events);assert!(lines.contains("imageurl:v4:declined:credential")&& !lines.contains("X-Amz-Signature"),"{lines}");}- // A declined URL the inspector named but the stored bytes do not show as it read it no longer fails the block:- // the writer scrubs every token that holds its host, and keeps the rest.- let event = json!([{"type": 4, "timestamp": 1, "data": {"href": "https://app.example.com/", "img": "https\\3a //bucket.s3.amazonaws.com/avatar.png"}}])- .to_string();- let lines = stored_lines(&event, &[signed]);- assert!(- lines.contains("https://app.example.com/")- && !lines.contains("bucket.s3.amazonaws.com"),- "{lines}"- );- assert!(!stored_lines(&event, &[public]).is_empty());- // A declined URL that a longer URL also holds is overwritten there too, and the block is kept, also for a bundled font, which names no URL.+ // A declined URL that a longer URL also holds is overwritten there too, and the block is kept, also for a bundled font, which names no URL, and for an escaped font URL, as the scan decodes it, of a bundled family too.let bundled_font = json!({"parentId": 1, "node": {"type": 2, "tagName": "style","attributes": {"_cssText": format!("@font-face{{font-family:Roboto;src:url({signed})}}")}, "childNodes": [], "id": 5}});+ let escaped_font = |family: &str, host: &str| {+ json!({"parentId": 1, "node": {"type": 2, "tagName": "style",+ "attributes": {"_cssText": format!("@font-face{{font-family:{family};src:url(https://{host}/f\\2e woff2?X-Amz-Signature=abc123)}}")}, "childNodes": [], "id": 6}})+ };+ let overlapping =+ "https://a.example/?X-Amz-Signature=x&next=https://b.example/?X-Amz-Signature=abc123";+ // Image extraction changes the copy's inline image, so the writer overwrites the copy before it.+ let with_inline_image = "https://cdn.example.com/a?X-Amz-Signature=abc123&x=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==";+ let decoded_font_copy =+ |host: &str| json!({"alt": format!("https://{host}/f.woff2?X-Amz-Signature=abc123")});for (adds, attributes) in [(json!([]),@@ -964,11 +934,44 @@ mod tests {json!([bundled_font]),json!({"alt": format!("{signed}&size=2")}),),+ (+ json!([escaped_font("Brand", "bucket.s3.amazonaws.com")]),+ decoded_font_copy("bucket.s3.amazonaws.com"),+ ),+ // The URL parser lowercases the host, and a copy keeps the spelling the page wrote.+ (+ json!([escaped_font("Brand", "CDN.example.com")]),+ decoded_font_copy("CDN.example.com"),+ ),+ // A text copy of the rule keeps its CSS escapes.+ (+ json!([escaped_font("Brand", "bucket.s3.amazonaws.com")]),+ json!({"alt": "url(https://bucket.s3.amazonaws.com/f\\2e woff2?X-Amz-Signature=abc123)"}),+ ),+ (+ json!([]),+ json!({"src": with_inline_image, "alt": with_inline_image}),+ ),+ (+ json!([escaped_font("Roboto", "bucket.s3.amazonaws.com")]),+ decoded_font_copy("bucket.s3.amazonaws.com"),+ ),+ (+ json!([{"parentId": 1, "node": {"type": 2, "tagName": "div", "attributes": {"style": "background-image:url('\\2f \\2f cdn.example.com/a.png?X-Amz-Signature=abc123')"}, "childNodes": [], "id": 9}}]),+ json!({"alt": "//cdn.example.com/a.png?X-Amz-Signature=abc123"}),+ ),+ (+ json!([+ {"parentId": 1, "node": {"type": 2, "tagName": "img", "attributes": {"src": "https://a.example/?X-Amz-Signature=x&next=https://b.example/"}, "childNodes": [], "id": 7}},+ {"parentId": 1, "node": {"type": 2, "tagName": "img", "attributes": {"src": "https://b.example/?X-Amz-Signature=abc123"}, "childNodes": [], "id": 8}},+ ]),+ json!({"alt": format!("{overlapping} {} {overlapping}", "x".repeat(200))}),+ ),] {let events = json!([{"type": 3, "timestamp": 1, "data": {"source": 0, "adds": adds, "removes": [], "texts": [],"attributes": [{"id": 99, "attributes": attributes}]}}]).to_string();- let lines = stored_lines(&events, &[]);+ let lines = stored_lines(&events);assert!(!lines.is_empty() && !lines.contains("X-Amz-Signature"),"{lines}"@@ -977,70 +980,56 @@ mod tests {}#[test]- fn keeps_a_block_past_the_url_limit_without_a_url_it_did_not_check() {- let images: Vec<_> = (0..=MAX_BLOCK_REMOTE_URLS)+ fn names_every_url_in_place_however_many_a_block_holds() {+ let link = || json!({"type": 2, "tagName": "a", "attributes": {"href": "https://app.example.com/pricing"}, "childNodes": [], "id": 1_000_000});+ let mut images: Vec<_> = (0..=10_000).map(|i| json!({"type": 2, "tagName": "img", "attributes": {"src": format!("https://cdn.example.com/{i}.png")}, "childNodes": [], "id": i + 2})).collect();- let faces: String = (0..=MAX_BLOCK_FONT_URLS)+ images.push(link());+ let faces: String = (0..=4_096).map(|i| format!("@font-face{{font-family:Brand{i};src:url(https://fonts.example.com/{i}.woff2)}}")).collect();- let link = "https://app.example.com/pricing";let fonts = vec![json!({"type": 2, "tagName": "style", "attributes": {"_cssText": faces}, "childNodes": [], "id": 2}),- json!({"type": 2, "tagName": "a", "attributes": {"href": link}, "childNodes": [], "id": 3}),+ link(),json!({"type": 2, "tagName": "style", "attributes": {"_cssText": "@\\66 ont-face{font-family:Escaped;src:url(https://fonts.example.com/escaped.woff2)}"}, "childNodes": [], "id": 4}),];- // Each escaped URL takes a rewrite, so past the rewrite budget its URL is not named, even where a line continuation splits its scheme.- let escaped_faces: String = (0..=MAX_BLOCK_REMOTE_URLS)+ // An escaped URL takes a rewrite of its whole `url()`, also where a line continuation splits its scheme.+ let escaped_faces: String = (0..=10_000).map(|i| format!("@font-face{{font-family:Escaped{i};src:url(https://fonts.example.com/{i}\\2e woff2)}}")).collect();let split_face = "@font-face{font-family:Split;src:url(\"htt\\\nps://fonts.example.com/split.woff2?token=secret\")}";let escaped_fonts = vec![json!({"type": 2, "tagName": "style", "attributes": {"_cssText": escaped_faces}, "childNodes": [], "id": 2}),- json!({"type": 2, "tagName": "a", "attributes": {"href": link}, "childNodes": [], "id": 3}),+ link(),json!({"type": 2, "tagName": "style", "attributes": {"_cssText": split_face}, "childNodes": [], "id": 4}),];- use replay_core_v4::v4_image_refs::{FONT_URL_UNCHECKED_MARKER, URL_UNCHECKED_MARKER};- for (nodes, named, marker, other_marker) in [- (- images,- "imageurl:v4:7:2026-10:",- URL_UNCHECKED_MARKER,- FONT_URL_UNCHECKED_MARKER,- ),- (- fonts,- "fonturl:v4:7:2026-10:",- FONT_URL_UNCHECKED_MARKER,- URL_UNCHECKED_MARKER,- ),- (- escaped_fonts,- "fonturl:v4:7:2026-10:",- FONT_URL_UNCHECKED_MARKER,- URL_UNCHECKED_MARKER,- ),+ for (nodes, named, references, declined) in [+ (images, "imageurl:v4:7:2026-10:", 10_001, 0),+ (fonts, "fonturl:v4:7:2026-10:", 4_098, 0),+ (escaped_fonts, "fonturl:v4:7:2026-10:", 10_001, 1),] {let events = json!([{"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": nodes}}}]).to_string();- let lines = stored_lines(&events, &[]);- assert!(lines.contains(named), "named URLs keep their references");- assert!(- lines.contains(marker) && !lines.contains(other_marker),- "{marker}"+ let lines = stored_lines(&events);+ assert_eq!(lines.matches(named).count(), references);+ assert_eq!(+ lines.matches("fonturl:v4:declined:credential").count(),+ declined);- for raw in ["https://cdn.", "https://fonts.", "token=secret"] {+ for raw in [+ "https://cdn.",+ "https://fonts.",+ "token=secret",+ "declined:unchecked",+ ] {assert!(!lines.contains(raw), "{raw}");}- assert_eq!(- lines.contains(link),- marker == FONT_URL_UNCHECKED_MARKER,- "only the font budget keeps links"- );+ assert!(lines.contains("https://app.example.com/pricing"));}}#[test]- fn drops_a_block_whose_css_nests_too_deep_to_read() {+ fn drops_a_block_the_inspector_could_not_read_in_full() {let deep_face = format!("{}@font-face{{font-family:Deep;src:url(\"https://fonts.example.com/deep.woff2?token=secret\")}}{}","@media all{".repeat(80),@@ -1056,9 +1045,52 @@ mod tests {json!({"type": 2, "tagName": "div", "attributes": {"style": deep_value}, "childNodes": [], "id": 2}),] {let events = json!([{"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": [node]}}}]).to_string();- let (_, counts) = written(block_of("kept", &events), &[]);+ let (_, counts) = written(block_of("kept", &events));assert_eq!(counts.dropped, Some("too_many_nested_blocks"));}+ let refused: Vec<_> = (0..2_000)+ .map(|i| json!({"type": 2, "tagName": "img", "attributes": {"src": format!("https://cdn.example.com/{i}/{}.png?token=x", "a".repeat(600))}, "childNodes": [], "id": i + 2}))+ .collect();+ let events = json!([{"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": refused}}}]).to_string();+ let (_, counts) = written(block_of("kept", &events));+ assert_eq!(counts.dropped, Some("too_many_declined_urls"));+ // Each copy of the longest of these refused URLs holds a match of every one of them.+ let nested: Vec<_> = (1..=1_000)+ .map(|i| json!({"type": 2, "tagName": "img", "attributes": {"src": format!("https://10.0.0.7/{}", "a".repeat(i))}, "childNodes": [], "id": i + 1}))+ .collect();+ let copies = format!("https://10.0.0.7/{} ", "a".repeat(1_000)).repeat(1_001);+ let events = json!([+ {"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": nested}}},+ {"type": 5, "timestamp": 2, "data": {"tag": "copies", "payload": {"text": copies}}},+ ])+ .to_string();+ let (lines, counts) = written(block_of("kept", &events));+ assert_eq!(+ (lines.as_str(), counts.dropped),+ ("", Some("too_many_declined_copies"))+ );+ // cssparser reads a NUL in a string as U+FFFD, so the decoded URL is not in its token, only in the comment.+ let nul_url = "url(\"https://cdn.example.com/a.png?X-Amz-Signature=abc123#\u{0}\"/*https://cdn.example.com/a.png?X-Amz-Signature=abc123#\u{FFFD}*/)";+ // A set keeps its candidate as the page wrote it, so it cannot keep an escaped one.+ for style in [+ format!("background-image:{nul_url}"),+ format!("background-image:image-set({nul_url} 1x)"),+ "background-image:image-set(url('\\2f \\2f cdn.example.com/a.png?token=secret') 1x)"+ .to_owned(),+ ] {+ let node = json!({"type": 2, "tagName": "div", "attributes": {"style": style}, "childNodes": [], "id": 2});+ let events = json!([{"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": [node]}}}]).to_string();+ let (_, counts) = written(block_of("kept", &events));+ assert_eq!(counts.dropped, Some("unplaced_slot"), "{style}");+ }+ // The block would store an event that the flush did not decode as it arrived, with its URLs unread.+ let events = json!([{"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": []}}}]).to_string();+ for expansion in [Expansion::Skipped, Expansion::Failed] {+ let mut block = block_of("kept", &events);+ block.messages[0].events[0].expansion = Some(expansion);+ let (_, counts) = written(block);+ assert_eq!(counts.dropped, Some("undecoded_events"));+ }}#[test]@@ -1077,9 +1109,9 @@ mod tests {{"type": 3, "timestamp": 2, "cv": "2024-10", "data": {"source": 8, "id": 3, "adds": compressed}},{"type": 3, "timestamp": 3, "data": {"source": 0, "adds": [], "removes": [], "texts": [], "attributes": [{"id": 9, "attributes": {"class": "bg-[url(/img/a.png)]"}}, {"id": 10, "attributes": {"class": "bg-[url(https://cdn.example.com/t.png)]"}}]}},{"type": 3, "timestamp": 4, "data": {"source": 8, "id": 3, "replaceSync": ".\\/\\/cdn\\.example\\.com\\/p\\.png\\?X-Amz-Signature\\=abc123{background:url(//cdn.example.com/p.png?X-Amz-Signature=abc123)}"}},- {"type": 3, "timestamp": 5, "data": {"source": 8, "id": 4, "replaceSync": "@font-face{font-family:Brand;src:url(/fonts/brand.woff2)}@font-face{font-family:Roboto;src:url(/fonts/roboto.woff2)}@font-face{font-family:Escaped;src:url(https://fonts.example.com/escaped\\2e woff2)}"}},+ {"type": 3, "timestamp": 5, "data": {"source": 8, "id": 4, "replaceSync": "@font-face{font-family:Brand;src:url(/fonts/brand.woff2)}@font-face{font-family:Roboto;src:url(/fonts/roboto.woff2)}@font-face{font-family:Escaped;src:url(https://fonts.example.com/escaped\\2e woff2)} .e{background:url(https://cdn.example.com/escaped\\2e png)}"}},]);- let lines = stored_lines(&events.to_string(), &[]);+ let lines = stored_lines(&events.to_string());let reference = |url: &str| {crate::urls::url_reference(url,@@ -1114,6 +1146,10 @@ mod tests {.1),r#"src:url(\"fonturl:v4:bundled:roboto\")"#.to_owned(),+ format!(+ r#".e{{background:url(\"{}\")}}"#,+ reference("https://cdn.example.com/escaped.png")+ ),format!(r#"src:url(\"{}\")"#,crate::urls::url_reference(@@ -1135,13 +1171,17 @@ mod tests {] {assert!(lines.contains(kept), "{kept} in {lines}");}- let class_url = reference("https://cdn.example.com/t.png");- for expected in [- format!(r#""class":"bg-[url({class_url})]""#),- format!(r".bg-\\[url\\({}\\)\\]", class_url.replace(':', r"\\:")),+ // A copy of an accepted URL outside an image slot stays as the page wrote it, as a link does.+ for kept in [+ r#""class":"bg-[url(https://cdn.example.com/t.png)]""#,+ r".bg-\\[url\\(https\\:\\/\\/cdn\\.example\\.com\\/t\\.png\\)\\]",] {- assert!(lines.contains(&expected), "{expected} in {lines}");+ assert!(lines.contains(kept), "{kept} in {lines}");}+ assert!(+ lines.contains(r".s-\\[url\\(imageurl\\:v4\\:declined\\:credential\\)\\]"),+ "{lines}"+ );for gone in [":url(/img/a.png)","1x.png",@@ -1155,7 +1195,7 @@ mod tests {}#[test]- fn leaves_a_relative_url_that_resolves_two_ways_as_the_page_wrote_it() {+ fn resolves_a_relative_url_against_the_page_of_its_own_window() {let window = |id: &str, href: &str| {let mut message = block_of("kept",@@ -1175,15 +1215,25 @@ mod tests {window("w1", "http://10.0.0.5/admin"),window("w2", "https://app.example.com/shop"),];- let lines = stored(block, &[]);- // A private host declines its resolved URL, which must not send the block to the host scrub.- for kept in [- r#""href":"http://10.0.0.5/admin""#,- r#""href":"https://app.example.com/shop""#,+ let lines = stored(block);+ let reference = crate::urls::url_reference(+ "https://app.example.com/a.png",+ UrlAsset::Image,+ 7,+ "2026-10",+ &replay_core_v4::seal::image_url_key(&[6; 32]),+ )+ .unwrap()+ .1;+ for expected in [+ r#""href":"http://10.0.0.5/admin""#.to_owned(),+ r#""href":"https://app.example.com/shop""#.to_owned(),+ r#"url(\"imageurl:v4:declined:non_public_host\")"#.to_owned(),+ format!(r#"url(\"{reference}\")"#),] {- assert!(lines.contains(kept), "{kept} in {lines}");+ assert!(lines.contains(&expected), "{expected} in {lines}");}- assert_eq!(lines.matches("url(/a.png)").count(), 2, "{lines}");+ assert!(!lines.contains("url(/a.png)"), "{lines}");}#[test]@@ -1264,7 +1314,7 @@ mod tests {assert!(fetched.iter().any(|u| u == img_src));assert!(fetched.iter().any(|u| u == webp_1x));- let lines = stored_lines(&events, &[]);+ let lines = stored_lines(&events);let reference = |url: &str| {crate::urls::url_reference(url,@@ -1330,7 +1380,7 @@ mod tests {r#"[{{"type":3,"timestamp":1,"data":{{"source":0,"adds":[],"removes":[],"texts":[],"attributes":[{{"id":99,"attributes":{{"srcset":"{} 1x, https://10.0.0.7/a.png 2x"}}}}]}}}}]"#,inline.replace('/', "\\/"));- let lines = stored_lines(&events, &[]);+ let lines = stored_lines(&events);assert!(lines.contains(&format!(r#""srcset":"{inline} 1x""#)),"{lines}"@@ -1341,13 +1391,13 @@ mod tests {);// A srcset key outside an image slot, here in a custom event, is no srcset of the DOM and stays as it is. With- // the same value as an inspected image slot, only the slot is rebuilt, and the custom event gets the rewrite of- // each URL that every copy of a named URL gets.+ // the same value as an inspected image slot, only the slot is rebuilt, and the custom event loses only the URL+ // that the policy refuses.let value = "https://cdn.example.com/a.png 1x, https://10.0.0.7/a.png 2x";let both = format!(r#"[{{"type":3,"timestamp":1,"data":{{"source":0,"adds":[],"removes":[],"texts":[],"attributes":[{{"id":99,"attributes":{{"srcset":"{value}"}}}}]}}}},{{"type":5,"timestamp":2,"data":{{"tag":"app","payload":{{"srcset":"{value}"}}}}}}]"#);- let lines = stored_lines(&both, &[]);+ let lines = stored_lines(&both);let reference = crate::urls::url_reference("https://cdn.example.com/a.png",replay_core_v4::refs::UrlAsset::Image,@@ -1363,13 +1413,13 @@ mod tests {"{dom}");assert!(- custom.contains(&format!(- r#""srcset":"{reference} 1x, imageurl:v4:declined:non_public_host 2x""#- )),+ custom.contains(+ r#""srcset":"https://cdn.example.com/a.png 1x, imageurl:v4:declined:non_public_host 2x""#+ ),"{custom}");let custom = r#"[{"type":5,"timestamp":1,"data":{"tag":"app","payload":{"srcset":"https://cdn.example.com/a.png 1x, https://10.0.0.7/a.png 2x"}}}]"#;- let lines = stored_lines(custom, &[]);+ let lines = stored_lines(custom);assert!(lines.contains(r#""srcset":"https://cdn.example.com/a.png 1x, https://10.0.0.7/a.png 2x""#@@ -1410,21 +1460,11 @@ mod tests {"https://fonts.gstatic.com/s/roboto/v30/KFOmCnqEu92Fr1Mu4mxK.woff2",);let css = format!("@font-face{{font-family:Brand;src:url({font})}}@font-face{{font-family:Inter;src:url({bundled})}}@font-face{{font-family:Brand Display;src:url({public})}}");- let events = json!([{"type": 2, "timestamp": 1, "data": {"node": {"attributes": {"src": fetched, "srcset": format!("{declined} 2x"), "_cssText": css}}}}]);- let mut block = block("kept", events);- block.image_urls = vec![fetched.to_owned()];- block.font_urls = vec![font.to_owned(), public.to_owned()];- let remote = |url: &str, kind| RemoteUrl {- url: url.to_owned(),- kind,- };- block.remote_urls = vec![- remote(fetched, RemoteKind::Image),- remote(declined, RemoteKind::Image),- remote(font, RemoteKind::Font),- remote(bundled, RemoteKind::BundledFont("inter")),- remote(public, RemoteKind::Font),- ];+ let events = json!([{"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": [+ {"type": 2, "tagName": "img", "attributes": {"src": fetched, "srcset": format!("{declined} 2x")}, "childNodes": [], "id": 2},+ {"type": 2, "tagName": "style", "attributes": {"_cssText": css}, "childNodes": [], "id": 3},+ ]}}}]);+ let block = inspected(block("kept", events));let mut keys = FlushKeys::default();let (id, key) = session_key("kept");keys.sessions.insert(id, key);@@ -1453,7 +1493,8 @@ mod tests {);for expected in [format!(r#""src":"{image_reference}""#),- r#""srcset":"imageurl:v4:declined:credential 2x""#.to_owned(),+ // The srcset keeps only the candidate it shows, and the policy refuses that one.+ r#""srcset":"""#.to_owned(),format!("src:url({font_reference})"),"src:url(fonturl:v4:bundled:inter)".to_owned(),format!("src:url({public_reference})"),