Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(ai-research): strip credentials from v4 URLs, with hegel tests (#128485)

Robbied0d934b043fe
diff --git a/products/ai_training/docs/replay-data.md b/products/ai_training/docs/replay-data.md
index 59f3367ad117..7c9058bd23c9 100644
--- a/products/ai_training/docs/replay-data.md
+++ b/products/ai_training/docs/replay-data.md
@@ -473,12 +473,13 @@ The recorder streams each block into the encoder. Read a block as a stream: a Br
- An SVG data URI whose payload starts with markup (`<`) stays inline, because SVG is text that the text scrub reads in place. Any other payload under an SVG label follows the rules for raster images, because a browser shows raster bytes as an image whatever the label says. A data URI with an empty payload holds no image, so it stays as it is.
- The recorder reads every argument of a canvas mutation, not only the frames of the posthog-js canvas recorder. A buffer becomes a reference when its bytes are PNG, JPEG, GIF or WebP, and a marker when they are not. The pixels of an `ImageData`, and the numbers of a `Uint8Array` or a `Uint8ClampedArray`, become `image:v4:declined:unsupported`. A data URI in the `src` of an `HTMLImageElement` moves out like any data URI, and a remote URL there stays as it is.
- A remote image URL becomes `imageurl:v4:{team_id}:{YYYY-MM}:{hash}` in the block and in its fetch job, where the month is the UTC month of the write. The block and the job always carry the same reference, so a reader opens the fetched image directly.
-- A URL that the URL policy refuses becomes `imageurl:v4:declined:{reason}` in the block, for example `declined:credential`, `declined:non_public_host` or `declined:tracking_beacon`. The original URL cannot be recovered. The recorder counts rewrites and refusals in `ai_research_v4_recorder_image_url_rewrites_total`.
+- A URL that the URL policy refuses gets no reference and no fetch job. It becomes `imageurl:v4:declined:{reason}` or `fonturl:v4:declined:{reason}` in the block, for example `declined:non_public_host` or `declined:tracking_beacon`, and the original URL cannot be recovered.
+- A URL that the policy refuses only for a credential keeps its place without its credentials: its userinfo, each credential parameter of its query and fragment, such as `X-Amz-Signature` or `#access_token=`, and each path segment that holds a token, such as `;jsessionid=`. A relative URL counts as on any host, so `/p/{token}/n/` loses its token as an Oracle Object Storage path does. The rest stays as the page wrote it. When the policy still reads a credential after those cuts, as in a path that hides a token behind a dot segment, only the URL's origin stays. When the policy refuses the URL that is left, for example for its host, the slot holds `declined:credential`. The recorder reads each URL with the URL parser, as the policy does, so it sees a host after IDNA maps it and a path once it is decoded and its dot segments are gone. A URL that the URL parser does not read, or one longer than 2,048 characters, can hold a credential that the recorder cannot see, so it becomes `declined:not_absolute` or `declined:too_long`, also where it shows no image. Blocks that an earlier recorder wrote hold `declined:credential` for every credential URL. The recorder counts rewrites and refusals in `ai_research_v4_recorder_image_url_rewrites_total`.
- Each `src` URL of an `@font-face` rule becomes a font reference in the same way, as [Web fonts](#web-fonts) describes. `ai_research_v4_recorder_image_url_rewrites_total` counts both, by `kind`: `image`, `font` or `bundled_font`.
- The `v4_image_refs` module of the `replay-core-v4` crate defines each reference and marker shape, for images and for fonts. The recorder writes them with it. A reader that must tell them apart from page text uses its `parse` function, which accepts only the exact strings the recorder writes.
-- The rewrite covers every remote URL that the recorder reads from an image slot: image sources, `srcset`, SVG image links, video posters, `<picture>` sources and CSS `url(...)` in style attributes, style sheets and style rules, in custom properties too, because a page passes a custom property to an image property through `var()`. It also covers `src`, `srcset`, `href` and `poster` in an attribute change on a node the recorder does not know, for example after a restart. `srcset` keeps its descriptors and CSS keeps its `url(...)` syntax. A URL is also replaced where a selector holds it with CSSOM's escapes, by the reference with the same escapes, so a class such as `bg-[url(https://cdn.example.com/a.png)]` and its rule `.bg-\[url\(https\:\/\/cdn\.example\.com\/a\.png\)\]` still match within a block. A reader that replaces references by text must replace the escaped form as well. It is not limited by the 512 URLs per message that get fetch jobs.
+- The rewrite covers every remote URL that the recorder reads from an image slot: image sources, `srcset`, SVG image links, video posters, `<picture>` sources and CSS `url(...)` in style attributes, style sheets and style rules, in custom properties too, because a page passes a custom property to an image property through `var()`. It also covers `src`, `srcset`, `href` and `poster` in an attribute change on a node the recorder does not know, for example after a restart. For the same reason, a `<source>` added under such a node is read as a `<picture>` source, and a text change on such a node loses the credentials of the `url()`s, the `image-set()` strings and the `@import`s in it, because the node can be the text of a `<style>`. Every other byte of that text stays, so it gets no references and an `image-set()` keeps each candidate, because it may not be CSS. `srcset` keeps its descriptors and CSS keeps its `url(...)` syntax. A class such as `bg-[url(https://cdn.example.com/a.png)]` and its rule's selector keep the URL as the page wrote it, because a selector is no URL slot. It is not limited by the 512 URLs per message that get fetch jobs.
- A srcset in an image slot keeps one candidate: the one that Chromium shows at a device pixel ratio of 1, which is the ratio the screenshotter renders at. Of `x` descriptors, that is the smallest density at or above 1x, or the largest density when all are below 1x. A `w` descriptor gives a density only against the layout width, which the recorder does not know, so of `w` descriptors the recorder keeps the v3 choice: the largest at or below 1024w, or the smallest when all are wider. The first candidate wins a tie. Only that candidate gets a fetch job, because the renderer has no network access and shows only fetched images. A srcset that mixes `w` and `x` descriptors, or that v3 would read as malformed, keeps no candidate.
-- The kept candidate stays only when it can show offline: a URL that the URL policy accepts, or an inline image that the image scan keeps. Otherwise the srcset is empty, and no other candidate takes its place, because no other candidate has a fetch job. A candidate that leaves becomes no `declined` marker.
+- The kept candidate stays as its reference when the URL policy accepts it, or as it is when it is an inline image that the image scan keeps. A kept candidate that the policy refuses leaves the srcset empty. An inline image that the scan declines leaves the srcset empty, and no other candidate takes its place, because no other candidate has a fetch job. A candidate that leaves becomes no marker.
- HTML adds the non-empty `src` of an `img` as a 1x candidate when its srcset has no 1x candidate and no `w` descriptor. At a device pixel ratio of 1 the `img` then shows its `src`, so the recorder fetches the `src` and empties the srcset.
- A visible `img` that shows its kept candidate gets no fetch job for its `src` or `rr_src`, as in v3. They still become references. Any other `img` with a `src` gets a fetch job for it. The recorder decides this on the srcset and the `src` as the event holds them, before the inline images leave, and keeps the same candidate in the stored block.
- An attribute change that sets the srcset but not the `src` does not hold the `src`, so the recorder decides as for an `img` with no `src`. If the `img` has a `src` that wins over the new srcset, the replay shows that `src`, which has a fetch job only if an earlier event gave it one.
@@ -487,18 +488,17 @@ The recorder streams each block into the encoder. Read a block as a stream: a Br
- An inline image in a candidate that leaves still moves to the image pack, where no reference names it.
- A `<picture>` source whose `type` names a media type the image fetcher does not store, such as `image/avif`, gets no fetch job and loses its remote candidates. Its kept candidate stays only when it is inline, such as an SVG data URI. The writer rebuilds each srcset that the inspector read in an image slot, where the inspector read it. A `srcset` key elsewhere, for example in a custom event's payload, stays as it is. A browser skips a source with no candidates, so the replay falls back to the next source or the `img`. The recorder counts these as `ai_research_v4_recorder_srcset_rewrites_total{outcome="trimmed"|"emptied"}`.
- The writer rewrites each CSS text where the inspector read it, so that an offline render loads only images the block refers to. An `image-set()` keeps one candidate, because a renderer picks the candidate that fits its pixel ratio and does not fall back when that candidate fails. It keeps the highest resolution of a type the image fetcher stores among the candidates that can load: inline, accepted by the URL policy, or root-relative on a known page. Only that candidate gets a fetch job, and the other candidates leave the set. A set that holds an inline image keeps all its candidates, and each remote one gets its reference.
-- A root-relative CSS `url(/…)` resolves against the origin of its window's latest page URL, from a Meta event or a custom event that names a page, such as `$url_changed`, with `https` in place of the page's scheme. It gets the reference of the absolute URL, or a `declined` marker. A path-relative `url(…)` stays as the page wrote it, so that no part of a page path, which can hold a token, reaches a fetch. rrweb writes snapshot CSS with absolute URLs, but sends rule text, adopted style sheets and style declarations as the page wrote them. Each window resolves its own texts, so the same rule in windows on two origins gets two references. An event resolves at most 1,024 relative URLs, a flush keeps at most 64 MiB of resolved URLs, and a page URL longer than 2,048 characters resolves none.
+- A root-relative CSS `url(/…)` resolves against the origin of its window's latest page URL, from a Meta event or a custom event that names a page, such as `$url_changed`, with `https` in place of the page's scheme. It gets what the absolute URL gets: its reference, or the marker or the URL without its credentials when the URL policy refuses it. A C0 control or a space before the `/`, which the URL parser strips, does not change this. A path-relative `url(…)` stays as the page wrote it, so that no part of a page path, which can hold a token, reaches a fetch. rrweb writes snapshot CSS with absolute URLs, but sends rule text, adopted style sheets and style declarations as the page wrote them. Each window resolves its own texts, so the same rule in windows on two origins gets two references. An event resolves at most 1,024 relative URLs, a flush keeps at most 64 MiB of resolved URLs, and a page URL longer than 2,048 characters resolves none.
-- The recorder reads CSS with `cssparser`, which tokenizes as CSS Syntax 3 does, so names, escapes, comments, strings and bad URLs read as a browser reads them. So `background-\69mage`, `\75 rl(…)` and `background/**/:url("…"/**/)` count, and so does a `url(` that the end of the text closes. A bad URL such as `url(a(1).png)`, which no browser loads, does not count. An image URL that itself holds an escape gets the reference of the URL it decodes to, as a font URL does, and its whole `url()` is rewritten. A block whose `image-set()` holds such a candidate is dropped, because the set keeps its candidate as the page wrote it.
+- The recorder reads CSS with `cssparser`, which tokenizes as CSS Syntax 3 does, so names, escapes, comments, strings and bad URLs read as a browser reads them. So `background-\69mage`, `\75 rl(…)` and `background/**/:url("…"/**/)` count, and so does a `url(` that the end of the text closes. A bad URL such as `url(a(1).png)`, which no browser loads, does not count. An image URL that itself holds an escape gets the reference of the URL it decodes to, as a font URL does, and its whole `url()` is rewritten. A block whose `image-set()` holds such a candidate, or a candidate such as `cross-fade()` with a `url()` inside, is dropped, because the set keeps its candidate as the page wrote it. A block is also dropped when an escaped URL decodes to a remote URL that the URL parser does not read.
- The writer rewrites only whole `url()` and `image-set()` functions, found as the inspector finds them, inside JSON strings that hold CSS: the values of `_cssText`, `style` (style diffs included), `textContent`, `rule`, `replace`, `replaceSync` and `value`. So a class name such as `bg-[url(/a.png)]`, a quoted string or a comment keeps its text, and the class's rule still matches. The recorder counts these as `ai_research_v4_recorder_css_rewrites_total{kind="image_set"|"relative_url", outcome="rewritten"}`.
-- The recorder reads a protocol-relative URL, such as `//cdn.example.com/a.png`, as `https`. It gets the same reference or `declined` marker as the `https` form, and the rewrite replaces the text as the block holds it. A browser reads a backslash as a slash there, so `\\cdn.example.com/a.png` counts too, and so does a URL such as `https:\\cdn.example.com/a.png` with any mix of slashes and backslashes after the scheme, or none: on a page of another scheme the browser loads `https:cdn.example.com/a.png` from that host.
-- Some URLs stay as they arrived: relative URLs outside CSS, path-relative CSS URLs, root-relative CSS URLs that go past a budget or come before this process saw their window's page URL (after a restart or an eviction), the candidates of an `image-set()` that does not parse, and remote URLs outside image slots: links, frames, text, CSS properties that do not show an image, and a copy of a slot's accepted URL elsewhere in the block.
+- The recorder reads a protocol-relative URL, such as `//cdn.example.com/a.png`, as `https`. It gets the same reference as the `https` form, and the rewrite replaces the text as the block holds it. A browser reads a backslash as a slash there, so `\\cdn.example.com/a.png` counts too, and so does a URL such as `https:\\cdn.example.com/a.png` with any mix of slashes and backslashes after the scheme, or none: on a page of another scheme the browser loads `https:cdn.example.com/a.png` from that host. The URL parser drops ASCII tabs and line breaks anywhere in a URL, and C0 controls and spaces at either end, so the recorder drops them before it decides that a URL is remote, and `/\n/cdn.example.com/a.png` and `\u0001https://cdn.example.com/a.png` count too.
+- Some URLs get no reference and keep their place without their credentials: relative URLs outside CSS, path-relative CSS URLs, root-relative CSS URLs that go past a budget or come before this process saw their window's page URL (after a restart or an eviction), and remote URLs in attributes that show no image. Those attributes are `href` and `src` of links, frames, scripts and other elements, `action`, `formaction`, `data`, `background`, `cite`, `longdesc`, `manifest`, `codebase`, `icon`, `imagesrcset` and the URL lists of `ping` and `attributionsrc`, and the `data-` attributes that lazy loaders use, such as `data-src` and `data-srcset`. So do `url()`s in CSS properties that show no image, such as `filter`, in SVG presentation attributes such as `fill`, `stroke`, `filter`, `clip-path`, `mask` and `marker-end`, and in `@import`. A URL with no credential there keeps every byte. On a page of its scheme a browser reads `https:a/b`, with no two slashes after the scheme, as a path on the page's host, so the recorder looks for a credential in both readings, and a URL whose credential is in what the URL parser alone reads as its host becomes its decline marker. The candidates of an `image-set()` that does not parse stay as they arrived.
- The writer writes again any JSON string that holds an escape capture would not write, for example `\/`. A scheme matches in any letter case.
-- A refused URL leaves the block wherever it is. The writer writes its `declined` marker in its slot, and overwrites each copy elsewhere in the block, also inside a longer URL, as a JSON string holds the URL and as a CSS selector escapes it. It does this before it moves inline images out, because that changes a copy that holds one. A selector copy takes the marker escaped, so the selector still matches. Only a block that holds a refused URL pays for that search. The recorder counts the copies as `ai_research_v4_recorder_overwritten_declined_copies_total`.
+- v4 stores pages unscrubbed, so the recorder removes credentials only from URLs where a page can load them: URL attributes and CSS. A copy of a URL elsewhere, such as in `alt` or `title` text, a text node, a CSS selector, a custom event, a console log or a network body, keeps its text.
- The recorder reads object keys as the player does: it writes an escaped key, such as `"\u0073rc"`, in its plain form before it inspects the event, and it takes the last copy of a repeated key.
-- The inspector records where each image slot, srcset and CSS string is, and the writer copies each event once and writes each reference or marker there. So a block has no limit on its URLs, and the writer finds no URL by its text. A URL of any length is named; one longer than the fetcher takes becomes `imageurl:v4:declined:too_long`.
-- A block that the recorder cannot read in full is dropped, because a URL in the part it did not read went unchecked: CSS that nests rules or functions 75 deep (`too_many_nested_blocks`), a mutation with more than 200,000 attribute changes (`too_many_attribute_items`), an event whose compressed strings did not decode or that a decode budget skipped (`undecoded_events`), more than 1 MiB of refused URLs, whose copies the writer searches for (`too_many_declined_urls`), more than 1,000,000 matches of those refused URLs (`too_many_declined_copies`), a block whose edits do not fit in the flush's 256 MiB edit budget (`too_many_edits`), and a slot that the writer could not find again (`unplaced_slot`). The recorder counts these as `dropped_blocks_total{outcome}`.
-- When the text of a rewritten URL also appears outside an image slot in the same block, it is replaced there too.
+- The inspector records where each image slot, srcset and CSS string is, and the writer copies each event once and writes each reference, marker or URL without its credentials there. So a block has no limit on its URLs, and the writer finds no URL by its text. A URL of any length is named; one longer than the fetcher takes, 2,048 characters, becomes `imageurl:v4:declined:too_long`.
+- A block that the recorder cannot read in full is dropped, because a URL in the part it did not read went unchecked: CSS that nests rules or functions 75 deep (`too_many_nested_blocks`), compressed JSON that nests more than 1,024 deep, which the message read cannot see (`too_deep_json`), a mutation with more than 200,000 attribute changes (`too_many_attribute_items`), an event whose compressed strings did not decode or that a decode budget skipped (`undecoded_events`), a block whose edits do not fit in the flush's 256 MiB edit budget (`too_many_edits`), and a slot that the writer could not find again (`unplaced_slot`). The recorder counts these as `dropped_blocks_total{outcome}`.
- `hash` is the first 22 characters of the base64url HMAC-SHA256, as in v3.
### Layout
@@ -796,12 +796,13 @@ A renderer that has no network needs the page's fonts. rrweb keeps the `@font-fa
The recorder reads the `src` URLs of `@font-face` rules in `<style>` text, `_cssText`, style sheet rules and adopted style sheets. Each URL becomes one of these:
-| The rule's family | In the block | Fetch job |
-| ------------------------------------------------------- | --------------------------------------- | --------- |
-| A family in `replay-core-v4/src/bundled_fonts.txt` | `fonturl:v4:bundled:{slug}` | None |
-| Any other family, for a public font file | `fonturl:v4:public:{hash}` | Yes |
-| Any other family | `fonturl:v4:{team_id}:{YYYY-MM}:{hash}` | Yes |
-| Any other family, for a URL that the URL policy refuses | `fonturl:v4:declined:{reason}` | None |
+| The rule's family | In the block | Fetch job |
+| ----------------------------------------------------------------- | --------------------------------------- | --------- |
+| A family in `replay-core-v4/src/bundled_fonts.txt` | `fonturl:v4:bundled:{slug}` | None |
+| Any other family, for a public font file | `fonturl:v4:public:{hash}` | Yes |
+| Any other family | `fonturl:v4:{team_id}:{YYYY-MM}:{hash}` | Yes |
+| Any other family, for a URL that the URL policy refuses | `fonturl:v4:declined:{reason}` | None |
+| Any other family, for a URL that the policy refuses as credential | The URL without its credentials | None |
- The renderer ships every family in `bundled_fonts.txt`. A reader replaces `fonturl:v4:bundled:{slug}` with its own copy of that family, and picks the face from the rule's `font-weight` and `font-style` descriptors. The slug is the family name in lowercase, with `-` for each space.
- The list holds only families whose licence lets the renderer carry them. The renderer ships the full font of each family, which holds every glyph of a subset that a page asks for, such as a Google Fonts `text=` or `icon_names=` subset.
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
index 90a9275f3396..2e9c7b852ea8 100644
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -2896,7 +2896,7 @@ version = "3.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "faf9468729b8cbcea668e36183cb69d317348c2e08e994829fb56ebfdfbaac34"
dependencies = [
- "windows-sys 0.61.2",
+ "windows-sys 0.48.0",
]
[[package]]
@@ -3876,6 +3876,27 @@ dependencies = [
"parking_lot_core",
]
+[[package]]
+name = "dashu-base"
+version = "0.6.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "861a614e79388a4c72aa66b0bdabc91e3a4f30d53e0c8d962953e207d75884d0"
+dependencies = [
+ "num-modular",
+]
+
+[[package]]
+name = "dashu-int"
+version = "0.6.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "64a0490b40a6a3f6427970035f17f76fd59c01d3cd9f9140809672863a0d64c8"
+dependencies = [
+ "cfg-if",
+ "dashu-base",
+ "num-modular",
+ "static_assertions",
+]
+
[[package]]
name = "data-encoding"
version = "2.10.0"
@@ -4282,7 +4303,7 @@ dependencies = [
"libc",
"option-ext",
"redox_users 0.5.2",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
]
[[package]]
@@ -4629,7 +4650,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [
"libc",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
]
[[package]]
@@ -5773,6 +5794,46 @@ version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
+[[package]]
+name = "hegeltest"
+version = "0.50.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2f7c373a89d726d42180154319cdf2aaa447180fe02e0588025a40f1165e8147"
+dependencies = [
+ "hegeltest-c",
+ "hegeltest-macros",
+ "parking_lot",
+]
+
+[[package]]
+name = "hegeltest-c"
+version = "0.47.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "44ba78cd3a2cfa4f3a06925bda13e0df5bee22aefda13179398a5b23dbed8c03"
+dependencies = [
+ "dashu-int",
+ "hashbrown 0.17.1",
+ "libm",
+ "miniz_oxide",
+ "once_cell",
+ "rand 0.10.2",
+ "rustc-hash 2.1.1",
+ "rustix 1.1.4",
+ "toml 1.1.8+spec-1.1.0",
+ "windows-sys 0.61.2",
+]
+
+[[package]]
+name = "hegeltest-macros"
+version = "0.50.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2d15af95c8c9c3f95806df1827ff30aea97696dc6d5316f19510c6e6f0b6800c"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 2.0.117",
+]
+
[[package]]
name = "hermit-abi"
version = "0.5.2"
@@ -6720,7 +6781,7 @@ checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46"
dependencies = [
"hermit-abi",
"libc",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
]
[[package]]
@@ -8214,7 +8275,7 @@ version = "0.50.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5"
dependencies = [
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
]
[[package]]
@@ -8309,6 +8370,12 @@ dependencies = [
"num-traits",
]
+[[package]]
+name = "num-modular"
+version = "0.6.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bd8e500409e6cd603b03e477c26a6caecdc27ac58979a53e881c75eafc079f44"
+
[[package]]
name = "num-rational"
version = "0.4.2"
@@ -11130,7 +11197,6 @@ name = "replay-recorder-v4"
version = "0.1.0"
dependencies = [
"aes-gcm",
- "aho-corasick",
"arc-swap",
"async-trait",
"aws-config",
@@ -11150,6 +11216,7 @@ dependencies = [
"common-redis",
"envconfig",
"futures",
+ "hegeltest",
"lifecycle",
"metrics",
"moka",
@@ -11585,7 +11652,7 @@ dependencies = [
"errno",
"libc",
"linux-raw-sys 0.12.1",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
]
[[package]]
@@ -11678,7 +11745,7 @@ dependencies = [
"security-framework 3.7.0",
"security-framework-sys",
"webpki-root-certs",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
]
[[package]]
@@ -12087,6 +12154,15 @@ dependencies = [
"serde",
]
+[[package]]
+name = "serde_spanned"
+version = "1.1.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4e7523beb55eece201a2356bee0bbca0d1ab466c14c07703b2e0ee6d42cb0c2c"
+dependencies = [
+ "serde_core",
+]
+
[[package]]
name = "serde_urlencoded"
version = "0.7.1"
@@ -13314,7 +13390,7 @@ dependencies = [
"getrandom 0.4.1",
"once_cell",
"rustix 1.1.4",
- "windows-sys 0.61.2",
+ "windows-sys 0.59.0",
]
[[package]]
@@ -13931,11 +14007,23 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362"
dependencies = [
"serde",
- "serde_spanned",
+ "serde_spanned 0.6.9",
"toml_datetime 0.6.11",
"toml_edit 0.22.27",
]
+[[package]]
+name = "toml"
+version = "1.1.8+spec-1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "20489e00e4d8741d6be680764cc12e270655e375a20d1011e844a9c3379e678d"
+dependencies = [
+ "serde_spanned 1.1.2",
+ "toml_datetime 1.1.2+spec-1.1.0",
+ "toml_parser",
+ "winnow 1.0.4",
+]
+
[[package]]
name = "toml_datetime"
version = "0.6.11"
@@ -13954,6 +14042,15 @@ dependencies = [
"serde_core",
]
+[[package]]
+name = "toml_datetime"
+version = "1.1.2+spec-1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2b86d767906c6c42421dcba507eb9d203e779497710a47782a224bb871653053"
+dependencies = [
+ "serde_core",
+]
+
[[package]]
name = "toml_edit"
version = "0.22.27"
@@ -13962,10 +14059,10 @@ checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a"
dependencies = [
"indexmap 2.13.0",
"serde",
- "serde_spanned",
+ "serde_spanned 0.6.9",
"toml_datetime 0.6.11",
"toml_write",
- "winnow",
+ "winnow 0.7.14",
]
[[package]]
@@ -13977,16 +14074,16 @@ dependencies = [
"indexmap 2.13.0",
"toml_datetime 0.7.5+spec-1.1.0",
"toml_parser",
- "winnow",
+ "winnow 0.7.14",
]
[[package]]
name = "toml_parser"
-version = "1.0.9+spec-1.1.0"
+version = "1.1.5+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "702d4415e08923e7e1ef96cd5727c0dfed80b4d2fa25db9647fe5eb6f7c5a4c4"
+checksum = "baa693a8032d7e1cada7d0041e96126df243179ff061456783ac7f12bda4744c"
dependencies = [
- "winnow",
+ "winnow 1.0.4",
]
[[package]]
@@ -15077,7 +15174,7 @@ version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [
- "windows-sys 0.61.2",
+ "windows-sys 0.48.0",
]
[[package]]
@@ -15322,6 +15419,12 @@ dependencies = [
"memchr",
]
+[[package]]
+name = "winnow"
+version = "1.0.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "23b97319f7b8343df12cc98938e5c3eb436064524c8d2b4e30a1d3a36eecdf81"
+
[[package]]
name = "winreg"
version = "0.50.0"
diff --git a/rust/ai-research/replay-core-v4/src/inspect/css.rs b/rust/ai-research/replay-core-v4/src/inspect/css.rs
index 981439e82cfb..9c4ad5d6336d 100644
--- a/rust/ai-research/replay-core-v4/src/inspect/css.rs
+++ b/rust/ai-research/replay-core-v4/src/inspect/css.rs
@@ -14,8 +14,10 @@ use url::Url;
use super::dom::{RemoteKind, RemoteUrl};
use crate::fonts::bundled_font;
+use crate::refs::{unnamed_url, UrlAsset};
use crate::url_policy::{
- is_fetched_image_type, is_protocol_relative, is_remote, try_canonicalize, MAX_URL_LEN,
+ as_parsed, is_fetched_image_type, is_protocol_relative, is_remote, needs_redaction,
+ parsed_start, try_canonicalize, MAX_URL_LEN,
};
/// Each resolution copies the page's origin, so one event resolves at most this many relative URLs.
@@ -45,6 +47,8 @@ pub enum CssUrl<'a> {
},
/// A [`CssUrl::Resolved`] font URL of a family that the renderer does not ship.
ResolvedFont(&'a str),
+ /// A URL with a credential in a CSS property that shows no image, or in an `@import`, which no fetch takes.
+ Link(&'a str),
/// Rules or functions nest to [`NESTED_BLOCK_LIMIT`], where `cssparser` enters no block, so a URL there goes unnamed.
NestedTooDeep,
/// A URL that the scan names but cannot find in the text it read, so the writer could not replace it there.
@@ -60,8 +64,6 @@ pub struct CssRewrite {
pub resolved: Option<RemoteUrl>,
/// Where `replacement` holds its one URL, so that the edit finds the URL without a search.
pub url_at: Option<Range<usize>>,
- /// The URL as the page wrote it and as its escapes decode, where `resolved` spells it otherwise, to find their copies.
- pub spellings: Vec<String>,
}
/// The origin that a root-relative URL on `page_url` resolves against, as `https` because the fetcher loads nothing else.
@@ -184,10 +186,140 @@ pub fn urls(css: &str, resolver: &mut Resolver<'_>, found: &mut impl FnMut(CssUr
CssUrl::Resolved(_)
| CssUrl::Unshown(_)
| CssUrl::Font { .. }
- | CssUrl::ResolvedFont(_) => {}
+ | CssUrl::ResolvedFont(_)
+ | CssUrl::Link(_) => {}
});
}
DeclarationKind::FontFamily(_) => {}
+ DeclarationKind::Link if may_hold_url(value) => link_urls(value, found),
+ DeclarationKind::Link => {}
+ DeclarationKind::Import => {
+ import_string(value, found);
+ link_urls(value, found);
+ }
+ }
+ }
+}
+
+/// A URL needs the `(` of a `url()` or an `image-set()`, and that name or an escape, so a value with neither holds no URL.
+fn may_hold_url(value: &str) -> bool {
+ let holds = |name: &[u8]| {
+ value
+ .as_bytes()
+ .windows(name.len())
+ .any(|at| at.eq_ignore_ascii_case(name))
+ };
+ value.contains('(') && (value.contains('\\') || holds(b"url") || holds(b"image-set"))
+}
+
+/// The URLs of a text that shows no image, which no fetch takes. Only a URL with a credential changes, so the rest of
+/// the text stays as the page wrote it, with every candidate of an `image-set()`.
+fn link_urls(value: &str, found: &mut impl FnMut(CssUrl<'_>)) {
+ let mut input = Parser::new(value);
+ input.set_nested_block_limit(NESTED_BLOCK_LIMIT);
+ if !link_tokens(&mut input, 0, false, found) {
+ found(CssUrl::NestedTooDeep);
+ }
+}
+
+/// Reads each `url()` at any depth, and each string that an `image-set()` holds as a candidate. Returns false when it
+/// skipped a block too deep to read whole.
+fn link_tokens<'i>(
+ input: &mut Parser<'i>,
+ depth: u8,
+ in_image_set: bool,
+ found: &mut impl FnMut(CssUrl<'_>),
+) -> bool {
+ let mut whole = true;
+ loop {
+ let start = input.position();
+ let token = match input.next_including_whitespace_and_comments() {
+ Err(_) => return whole,
+ Ok(token) => token.clone(),
+ };
+ let opens_block = matches!(
+ token,
+ Token::Function(_)
+ | Token::ParenthesisBlock
+ | Token::SquareBracketBlock
+ | Token::CurlyBracketBlock
+ );
+ if opens_block && depth + 1 >= NESTED_BLOCK_LIMIT {
+ whole = false;
+ continue;
+ }
+ let source = match token {
+ Token::UnquotedUrl(url) => unquoted_url_source(url, input.slice_from(start)),
+ Token::Function(name) if name.eq_ignore_ascii_case("url") => {
+ match quoted_url_source(input) {
+ Some(source) => source,
+ None => continue,
+ }
+ }
+ Token::QuotedString(url) if in_image_set => string_source(url, input.slice_from(start)),
+ _ if opens_block => {
+ let image_set = matches!(&token, Token::Function(name) if is_image_set(name));
+ let _ = input.parse_nested_block(|input| {
+ whole &= link_tokens(input, depth + 1, image_set, found);
+ Ok::<_, ParseError<()>>(())
+ });
+ continue;
+ }
+ _ => continue,
+ };
+ link_url(&source, input.slice_from(start), found);
+ }
+}
+
+/// `token` is the `url()` or string that holds `source`. A browser loads the URL that its escapes decode to, which the
+/// block does not hold as the page wrote it, so then the whole token changes.
+fn link_url<'t>(source: &UrlSource<'t>, token: &'t str, found: &mut impl FnMut(CssUrl<'_>)) {
+ let url = source.url.trim();
+ let Some(kept) = unnamed_url(url, UrlAsset::Image) else {
+ return;
+ };
+ match located(source.written, url).filter(|_| !source.escaped) {
+ Some(url) => found(CssUrl::Link(url)),
+ None => found(redacted(token, &kept)),
+ }
+}
+
+/// `@import "…"` names its sheet with a string, not a `url()`.
+fn import_string(prelude: &str, found: &mut impl FnMut(CssUrl<'_>)) {
+ let mut input = Parser::new(prelude);
+ input.skip_whitespace();
+ let start = input.position();
+ if let Ok(Token::QuotedString(url)) = input.next() {
+ let url = url.clone();
+ let token = input.slice_from(start);
+ link_url(&string_source(url, token), token, found);
+ }
+}
+
+fn string_source<'i>(url: CowRcStr<'i>, token: &'i str) -> UrlSource<'i> {
+ UrlSource {
+ escaped: token.contains('\\'),
+ written: between_quotes(token),
+ url,
+ }
+}
+
+/// The URLs with a credential in a text that may not be CSS, such as the text of a node the window does not know.
+pub fn text_link_urls(text: &str, found: &mut impl FnMut(CssUrl<'_>)) {
+ if may_hold_url(text) {
+ link_urls(text, found);
+ }
+ // An `@import` names its sheet with a string, which holds no `(`, and its keyword can hold an escape.
+ let may_import = text.contains('\\')
+ || text
+ .as_bytes()
+ .windows(7)
+ .any(|at| at.eq_ignore_ascii_case(b"@import"));
+ if may_import {
+ for declaration in &declarations(text).found {
+ if declaration.kind == DeclarationKind::Import {
+ import_string(&text[declaration.start..declaration.end], found);
+ }
}
}
}
@@ -200,29 +332,11 @@ pub fn declaration_image_urls(
) {
if can_hold_image(property.trim()) {
value_urls(value, resolver, found);
+ } else if may_hold_url(value) {
+ link_urls(value, found);
}
}
-/// `text` as CSSOM writes it inside an identifier, as a selector holds the URL of a class like `bg-[url(https://…)]`.
-pub fn escape_in_identifier(text: &str) -> String {
- let mut escaped = String::with_capacity(text.len() * 2);
- for character in text.chars() {
- match character {
- '\0' => escaped.push('\u{fffd}'),
- '\u{1}'..='\u{1f}' | '\u{7f}' => {
- escaped.push_str(&format!("\\{:x} ", u32::from(character)));
- }
- 'a'..='z' | 'A'..='Z' | '0'..='9' | '-' | '_' => escaped.push(character),
- _ if !character.is_ascii() => escaped.push(character),
- _ => {
- escaped.push('\\');
- escaped.push(character);
- }
- }
- }
- escaped
-}
-
fn can_hold_image(property: &str) -> bool {
property.starts_with("--") || is_image_property(property)
}
@@ -232,6 +346,8 @@ enum DeclarationKind {
Image,
FontSrc(usize),
FontFamily(usize),
+ Link,
+ Import,
}
struct Declaration {
@@ -305,7 +421,8 @@ impl<'i> DeclarationParser<'i> for Declarations {
}
let kind = match self.block {
Block::FontFace(face) => font_descriptor(&name, face),
- Block::Style => can_hold_image(&name).then_some(DeclarationKind::Image),
+ Block::Style if can_hold_image(&name) => Some(DeclarationKind::Image),
+ Block::Style => Some(DeclarationKind::Link),
};
if let Some(kind) = kind {
self.found.push(Declaration {
@@ -329,7 +446,15 @@ impl<'i> AtRuleParser<'i> for Declarations {
input: &mut Parser<'i>,
) -> Result<Block, ParseError<()>> {
let font_face = name.eq_ignore_ascii_case("font-face") && input.is_exhausted();
+ let start = input.position().byte_index();
while input.next().is_ok() {}
+ if name.eq_ignore_ascii_case("import") {
+ self.found.push(Declaration {
+ start,
+ end: input.position().byte_index(),
+ kind: DeclarationKind::Import,
+ });
+ }
if !font_face {
return Ok(Block::Style);
}
@@ -489,7 +614,7 @@ fn value_urls(value: &str, resolver: &mut Resolver<'_>, found: &mut impl FnMut(C
},
// A browser loads the URL its escapes decode to, so the block must not keep it as the page wrote it.
None if source.escaped => {
- escaped_url_function_urls(function, source.written, &source.url, resolver, found)
+ escaped_url_function_urls(function, &source.url, resolver, found)
}
None => {}
}
@@ -572,20 +697,35 @@ fn url_function_urls(
kind: RemoteKind::Image,
}),
url_at: Some(url_at),
- spellings: Vec::new(),
},
function,
});
}
- None if is_root_relative(source) => found(CssUrl::Unresolved(function)),
+ None if is_root_relative(source) => match unnamed_url(source, UrlAsset::Image) {
+ Some(kept) => found(redacted(function, &kept)),
+ None => found(CssUrl::Unresolved(function)),
+ },
None => found(CssUrl::Image(source)),
}
}
+fn redacted<'f>(function: &'f str, kept: &str) -> CssUrl<'f> {
+ let replacement = format!("{URL_FUNCTION_OPEN}{}\")", in_css_string(kept));
+ CssUrl::Rewrite {
+ rewrite: CssRewrite {
+ text: function.to_owned(),
+ replacement,
+ dropped: Vec::new(),
+ resolved: None,
+ url_at: None,
+ },
+ function,
+ }
+}
+
/// The block does not hold the URL a browser reads from an escaped source, so the rewrite replaces the whole `url()`.
fn escaped_url_function_urls(
function: &str,
- escaped: &str,
decoded: &str,
resolver: &mut Resolver<'_>,
found: &mut impl FnMut(CssUrl<'_>),
@@ -595,26 +735,24 @@ fn escaped_url_function_urls(
return url_function_urls(function, written, resolver, found);
}
let decoded = if is_protocol_relative(written) {
- Cow::Owned(format!("https:{written}"))
+ Cow::Owned(format!("https:{}", as_parsed(written).trim()))
} else {
Cow::Borrowed(written)
};
- // A serialized URL percent-encodes `"` and holds no backslash, so the quotes of the rewrite hold it.
let Some(absolute) = Url::parse(&decoded)
.ok()
.filter(|url| matches!(url.scheme(), "http" | "https"))
.map(String::from)
else {
+ if is_remote(written) {
+ found(CssUrl::Unplaced);
+ } else if let Some(kept) = unnamed_url(written, UrlAsset::Image) {
+ found(redacted(function, &kept));
+ }
return;
};
found(CssUrl::Resolved(&absolute));
let (replacement, url_at) = url_function(&absolute);
- let mut spellings: Vec<String> = Vec::new();
- for spelling in [escaped, written] {
- if spelling != absolute && !spellings.iter().any(|kept| kept == spelling) {
- spellings.push(spelling.to_owned());
- }
- }
found(CssUrl::Rewrite {
rewrite: CssRewrite {
text: function.to_owned(),
@@ -625,19 +763,38 @@ fn escaped_url_function_urls(
kind: RemoteKind::Image,
}),
url_at: Some(url_at),
- spellings,
},
function,
});
}
fn is_root_relative(source: &str) -> bool {
- source.starts_with('/') && !is_protocol_relative(source)
+ parsed_start(source, 1) == "/" && !is_protocol_relative(source)
}
const URL_FUNCTION_OPEN: &str = "url(\"";
-/// A serialized URL percent-encodes `"`, so the quotes hold any URL that [`Resolver`] returns.
+/// `text` as a quoted CSS string holds it. A serialized URL keeps a `\\` in its query and fragment, which would escape
+/// the closing quote.
+pub fn in_css_string(text: &str) -> Cow<'_, str> {
+ if !text.contains(|ch: char| matches!(ch, '"' | '\\' | '\0'..='\u{1f}' | '\u{7f}')) {
+ return Cow::Borrowed(text);
+ }
+ let mut escaped = String::with_capacity(text.len() + 8);
+ for ch in text.chars() {
+ match ch {
+ '"' | '\\' => {
+ escaped.push('\\');
+ escaped.push(ch);
+ }
+ '\0'..='\u{1f}' | '\u{7f}' => escaped.push_str(&format!("\\{:x} ", u32::from(ch))),
+ _ => escaped.push(ch),
+ }
+ }
+ Cow::Owned(escaped)
+}
+
+/// The writer writes the URL that this `url()` holds with [`in_css_string`].
fn url_function(url: &str) -> (String, Range<usize>) {
(
format!("{URL_FUNCTION_OPEN}{url}\")"),
@@ -690,9 +847,14 @@ fn image_set_urls(
let mut dropped = Vec::new();
for (index, candidate) in candidates.iter().enumerate() {
let source = candidate.source.unwrap_or_default();
- if kept.is_none_or(|kept| kept.0 != index) && is_remote(source) {
+ if kept.is_some_and(|kept| kept.0 == index) {
+ continue;
+ }
+ if is_remote(source) {
found(CssUrl::Unshown(source));
dropped.push(source.to_owned());
+ } else if needs_redaction(source) {
+ found(CssUrl::Link(source));
}
}
let Some((index, source, _)) = kept else {
@@ -729,7 +891,6 @@ fn image_set_urls(
kind: RemoteKind::Image,
}),
url_at: url_in_candidate.map(|at| at.start + opening..at.end + opening),
- spellings: Vec::new(),
},
function,
});
@@ -780,6 +941,10 @@ fn image_set_candidates<'i>(
}
if let Some(source) = candidate.source.filter(|source| is_remote(source)) {
found(CssUrl::Unshown(source));
+ } else if let Some(source) =
+ candidate.source.filter(|source| needs_redaction(source))
+ {
+ found(CssUrl::Link(source));
}
}
}
@@ -794,7 +959,10 @@ enum CandidateStart<'i> {
Url(CowRcStr<'i>),
String(CowRcStr<'i>),
UrlFunction,
- Other,
+ /// Such as `cross-fade()`, which can hold a `url()` that the set keeps as the page wrote it.
+ Other {
+ holds_url: bool,
+ },
}
fn image_set_candidate<'i>(input: &mut Parser<'i>) -> Option<ImageSetCandidate<'i>> {
@@ -807,24 +975,31 @@ fn image_set_candidate<'i>(input: &mut Parser<'i>) -> Option<ImageSetCandidate<'
Ok(Token::Function(name)) if name.eq_ignore_ascii_case("url") => {
CandidateStart::UrlFunction
}
- Ok(_) => CandidateStart::Other,
+ Ok(Token::Function(_)) => CandidateStart::Other {
+ holds_url: input
+ .parse_nested_block(|input| {
+ let mut holds_url = false;
+ let whole = each_function(input, 1, &mut |_, _, _| holds_url = true);
+ Ok::<_, ParseError<()>>(holds_url || !whole)
+ })
+ .unwrap_or(true),
+ },
+ Ok(_) => CandidateStart::Other { holds_url: false },
};
+ let holds_url = matches!(first, CandidateStart::Other { holds_url: true });
let source = match first {
CandidateStart::Url(url) => Some(unquoted_url_source(url, input.slice_from(start))),
- CandidateStart::String(url) => Some(UrlSource {
- escaped: input.slice_from(start).contains('\\'),
- written: between_quotes(input.slice_from(start)),
- url,
- }),
+ CandidateStart::String(url) => Some(string_source(url, input.slice_from(start))),
CandidateStart::UrlFunction => quoted_url_source(input),
- CandidateStart::Other => None,
+ CandidateStart::Other { .. } => None,
};
let source_end = input.position();
let written = source.as_ref().and_then(collectable);
- // A set keeps a candidate as the page wrote it, so an escaped remote URL, which the block does not hold as a browser reads it, has no place.
- let escaped_remote = source
- .as_ref()
- .is_some_and(|source| source.escaped && is_remote(source.url.trim()));
+ // A set keeps a candidate as the page wrote it, so an escaped remote URL, or an escaped relative one with a credential, which the block does not hold as a browser reads it, has no place.
+ let escaped_remote = source.as_ref().is_some_and(|source| {
+ let url = source.url.trim();
+ source.escaped && (is_remote(url) || needs_redaction(url))
+ });
let source = source
.as_ref()
.zip(written)
@@ -833,7 +1008,7 @@ fn image_set_candidate<'i>(input: &mut Parser<'i>) -> Option<ImageSetCandidate<'
Some(ImageSetCandidate {
text: input.slice_from(start).trim_end(),
source,
- unplaced: escaped_remote || (written.is_some() && source.is_none()),
+ unplaced: escaped_remote || holds_url || (written.is_some() && source.is_none()),
descriptors: input.slice(source_end..input.position()),
resolution_dppx,
})
@@ -909,8 +1084,16 @@ impl CssUrl<'_> {
pub fn edit_bytes(&self) -> usize {
let piece = std::mem::size_of::<CssPiece>() + std::mem::size_of::<CssPart>();
match self {
- CssUrl::Image(url) | CssUrl::Unshown(url) | CssUrl::Font { url, .. } => {
- piece + url.len()
+ // The builder keeps only a remote URL, so a relative one takes nothing.
+ CssUrl::Image(url)
+ | CssUrl::Unshown(url)
+ | CssUrl::Font { url, .. }
+ | CssUrl::Link(url) => {
+ if is_remote(url.trim()) || needs_redaction(url) {
+ piece + url.len()
+ } else {
+ 0
+ }
}
CssUrl::Rewrite { rewrite, .. } => {
let resolved = rewrite.resolved.as_ref().map_or(0, |url| url.url.len());
@@ -955,6 +1138,7 @@ impl<'c> CssEditBuilder<'c> {
self.place(url, kind);
}
CssUrl::Rewrite { rewrite, function } => self.rewrite(rewrite, function),
+ CssUrl::Link(url) => self.place(url, RemoteKind::Link),
CssUrl::Unplaced => self.unplaced = true,
CssUrl::Resolved(_)
| CssUrl::ResolvedFont(_)
@@ -963,9 +1147,15 @@ impl<'c> CssEditBuilder<'c> {
}
}
+ /// A URL that no fetch takes is placed only when it holds a credential, so that the rest of the text keeps its bytes.
fn place(&mut self, url: &str, kind: RemoteKind) {
let url = url.trim();
- if !is_remote(url) {
+ let kind = match kind {
+ RemoteKind::Link => RemoteKind::Link,
+ _ if is_remote(url) => kind,
+ _ => RemoteKind::Link,
+ };
+ if kind == RemoteKind::Link && !needs_redaction(url) {
return;
}
let named = RemoteUrl {
@@ -1089,6 +1279,7 @@ mod tests {
),
CssUrl::Font { url, family } => format!("font {url} {family:?}"),
CssUrl::ResolvedFont(url) => format!("resolved font {url}"),
+ CssUrl::Link(url) => format!("link {url}"),
CssUrl::NestedTooDeep => "nested too deep".to_owned(),
CssUrl::Unplaced => "unplaced".to_owned(),
})
diff --git a/rust/ai-research/replay-core-v4/src/inspect/dom.rs b/rust/ai-research/replay-core-v4/src/inspect/dom.rs
index 4809fc233247..0b5a5151c5fa 100644
--- a/rust/ai-research/replay-core-v4/src/inspect/dom.rs
+++ b/rust/ai-research/replay-core-v4/src/inspect/dom.rs
@@ -10,10 +10,10 @@ use url::Url;
use crate::bytescan::{self as scan, Span};
use crate::fonts::{bundled_font, is_fetched_font_url};
-use crate::url_policy::{checked_url, is_fetched_image_type, is_remote, Decline, MAX_URL_LEN};
+use crate::url_policy::{is_fetched_image_type, is_remote, needs_redaction, MAX_URL_LEN};
use super::css::{self, CssEditBuilder, CssUrl, Resolver};
-use super::edits::{Buffers, Edit, Slot};
+use super::edits::{Buffers, Edit, LinkForm, Slot};
use super::events::{field, fields, number, text};
use super::rrweb::{
ADOPTED_STYLE_SHEET, FULL_SNAPSHOT, INCREMENTAL_SNAPSHOT, MUTATION, STYLE_DECLARATION,
@@ -33,8 +33,6 @@ pub const MAX_FONT_URLS: usize = 512;
/// Attribute changes read per mutation, as many as the nodes tracked. A mutation with more drops its block, because
/// the inspector does not name the URLs in the rest.
pub const MAX_ATTRIBUTE_ITEMS: usize = MAX_TRACKED_NODES;
-/// The writer builds one matcher over a block's refused URLs to overwrite their copies, and its memory follows their bytes.
-pub const MAX_BLOCK_DECLINED_URL_BYTES: usize = 1024 * 1024;
/// Windows tracked per session. A new window past this evicts the window used least recently.
pub const MAX_WINDOWS: usize = 16;
/// Evicted windows remembered, so that one that comes back is known to have lost its DOM.
@@ -197,7 +195,7 @@ struct WindowDom {
pub enum UrlLimit {
AttributeItems,
NestedBlocks,
- DeclinedUrls,
+ NestedJson,
Edits,
}
@@ -207,18 +205,20 @@ impl UrlLimit {
match self {
UrlLimit::AttributeItems => "too_many_attribute_items",
UrlLimit::NestedBlocks => "too_many_nested_blocks",
- UrlLimit::DeclinedUrls => "too_many_declined_urls",
+ UrlLimit::NestedJson => "too_deep_json",
UrlLimit::Edits => "too_many_edits",
}
}
}
-/// The kind of a remote URL by the slot that holds it. A refused URL named as several kinds takes the greatest.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum RemoteKind {
BundledFont(&'static str),
Font,
Image,
+ /// A URL that no fetch takes, such as a relative one or one in a CSS property that shows no image. The writer keeps
+ /// it without its credentials.
+ Link,
}
#[derive(Debug, Clone, PartialEq, Eq)]
@@ -227,14 +227,6 @@ pub struct RemoteUrl {
pub kind: RemoteKind,
}
-/// A spelling of a refused URL, with the reason for the URL as the policy reads it, which an escaped spelling hides.
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub struct DeclinedUrl {
- pub url: String,
- pub kind: RemoteKind,
- pub decline: Decline,
-}
-
/// Distinct remote image URLs, at most [`MAX_IMAGE_URLS`] per message.
#[derive(Default)]
struct Urls {
@@ -248,9 +240,15 @@ struct Urls {
edit_bytes_taken: usize,
/// A slot the writer could not find again, so that the block must not be stored.
unplaced: bool,
- /// Named URLs that the policy refuses, by the largest kind each was named as. The writer overwrites every copy of each one, outside its slot too.
- declined: HashMap<String, (RemoteKind, Decline)>,
- declined_bytes: usize,
+}
+
+#[derive(Clone, Copy)]
+enum CssRead<'p> {
+ Sheet,
+ /// The value of the property it names.
+ Declaration(&'p str),
+ /// A text that may not be CSS. See [`css::text_link_urls`].
+ Links,
}
#[derive(Default)]
@@ -267,11 +265,6 @@ fn fetchable(url: &str) -> Option<&str> {
}
impl Urls {
- fn push(&mut self, url: &str) {
- self.push_remote(url);
- self.push_fetch(url);
- }
-
fn push_fetch(&mut self, url: &str) {
let Some(url) = fetchable(url) else { return };
if self.in_message.len() < MAX_IMAGE_URLS && !self.in_message.contains(url) {
@@ -280,46 +273,12 @@ impl Urls {
}
}
- /// Records a URL that the block holds but the fetcher does not take now, such as one of a hidden img.
- fn push_remote(&mut self, url: &str) {
- let url = url.trim();
- if is_remote(url) {
- self.note_declined(url, RemoteKind::Image);
- }
- }
-
- fn note_declined(&mut self, url: &str, kind: RemoteKind) -> Option<Decline> {
- if let Some((noted, decline)) = self.declined.get_mut(url) {
- *noted = (*noted).max(kind);
- return Some(*decline);
- }
- let decline = checked_url(url).err()?;
- self.add_declined(url.to_owned(), kind, decline);
- Some(decline)
- }
-
- fn add_declined(&mut self, url: String, kind: RemoteKind, decline: Decline) {
- if let Some((noted, _)) = self.declined.get_mut(&url) {
- *noted = (*noted).max(kind);
- } else if self.declined_bytes + url.len() > MAX_BLOCK_DECLINED_URL_BYTES {
- self.limit = self.limit.or(Some(UrlLimit::DeclinedUrls));
- } else {
- self.declined_bytes += url.len();
- self.declined.insert(url, (kind, decline));
- }
- }
-
fn push_font(&mut self, url: &str, family: Option<&str>) {
let url = url.trim();
if !is_remote(url) {
return;
}
- let kind = match family.and_then(bundled_font) {
- Some(slug) => RemoteKind::BundledFont(slug),
- None => RemoteKind::Font,
- };
- self.note_declined(url, kind);
- if kind == RemoteKind::Font {
+ if family.and_then(bundled_font).is_none() {
self.queue_font(url);
}
}
@@ -337,29 +296,13 @@ impl Urls {
fn push_css(&mut self, found: CssUrl<'_>) {
match found {
- CssUrl::Image(url) => self.push(url),
- CssUrl::Unshown(url) => self.push_remote(url),
- CssUrl::Resolved(url) => {
- self.note_declined(url, RemoteKind::Image);
- self.push_fetch(url);
- }
- // A bundled font's URL comes only in its rewrite, as no fetch takes it.
- CssUrl::Rewrite { rewrite, .. } => {
- let Some(resolved) = &rewrite.resolved else {
- return;
- };
- if let Some(decline) = self.note_declined(&resolved.url, resolved.kind) {
- for spelling in &rewrite.spellings {
- self.add_declined(spelling.clone(), resolved.kind, decline);
- }
- }
- }
- CssUrl::Unresolved(_) => {}
+ CssUrl::Image(url) | CssUrl::Resolved(url) => self.push_fetch(url),
+ CssUrl::Unshown(_)
+ | CssUrl::Rewrite { .. }
+ | CssUrl::Unresolved(_)
+ | CssUrl::Link(_) => {}
CssUrl::Font { url, family } => self.push_font(url, family),
- CssUrl::ResolvedFont(url) => {
- self.note_declined(url, RemoteKind::Font);
- self.queue_font(url);
- }
+ CssUrl::ResolvedFont(url) => self.queue_font(url),
CssUrl::NestedTooDeep => self.limit = self.limit.or(Some(UrlLimit::NestedBlocks)),
CssUrl::Unplaced => self.unplaced = true,
}
@@ -405,13 +348,12 @@ impl Urls {
}
}
- /// Reads `css`, the text of the JSON string at `value`, as a style sheet, or as the value of `property`.
fn read_css(
&mut self,
buffers: &Buffers,
(bytes, value): (&[u8], Span),
css: &str,
- property: Option<&str>,
+ read: CssRead<'_>,
resolver: &mut Resolver<'_>,
) {
let edit_budget = resolver.edit_budget();
@@ -425,9 +367,12 @@ impl Urls {
}
self.push_css(found);
};
- match property {
- Some(property) => css::declaration_image_urls(property, css, resolver, &mut found),
- None => css::urls(css, resolver, &mut found),
+ match read {
+ CssRead::Sheet => css::urls(css, resolver, &mut found),
+ CssRead::Declaration(property) => {
+ css::declaration_image_urls(property, css, resolver, &mut found)
+ }
+ CssRead::Links => css::text_link_urls(css, &mut found),
}
match edit.map(CssEditBuilder::finish) {
Some(Ok(Some(edit))) => {
@@ -450,9 +395,6 @@ impl Urls {
self.edits.extend(walk.edits);
self.edit_bytes_taken += walk.edit_bytes_taken;
self.unplaced |= walk.unplaced;
- for (url, (kind, decline)) in walk.declined {
- self.add_declined(url, kind, decline);
- }
}
}
@@ -530,7 +472,7 @@ impl Walk<'_> {
}
fn record_url(&mut self, bytes: &[u8], value: Span, url: &str) {
- if is_remote(url.trim()) {
+ if is_remote(url.trim()) || needs_redaction(url) {
let edit_budget = self.resolver.edit_budget();
self.image_urls
.record(&self.buffers, (bytes, value), Slot::Url, edit_budget);
@@ -538,18 +480,34 @@ impl Walk<'_> {
}
fn push_url_text(&mut self, url: &str) {
- self.image_urls.push(url);
+ self.image_urls.push_fetch(url);
}
- /// The URLs of a slot the fetcher does not take now. The block still holds them, so the writer replaces them.
- fn push_remote(&mut self, bytes: &[u8], value: Option<Span>, srcset: bool) {
- if let (Some(span), Some(text)) = (value, text(bytes, value)) {
- if srcset {
- srcset_urls(&text, &mut |url| self.image_urls.push_remote(url));
- } else {
- self.record_url(bytes, span, &text);
- self.image_urls.push_remote(&text);
+ /// A URL that shows no image is a slot only when it holds a credential, so that the writer leaves the rest alone.
+ fn push_link(&mut self, bytes: &[u8], value: Option<Span>, form: LinkForm) {
+ let (Some(span), Some(text)) = (value, text(bytes, value)) else {
+ return;
+ };
+ let holds_credential = match form {
+ LinkForm::Url => needs_redaction(&text),
+ LinkForm::Srcset => {
+ let mut holds = false;
+ for_each_srcset_candidate(&text, |url, _| holds |= needs_redaction(url));
+ holds
}
+ LinkForm::Urls => text.split_ascii_whitespace().any(needs_redaction),
+ };
+ if holds_credential {
+ let edit_budget = self.resolver.edit_budget();
+ self.image_urls
+ .record(&self.buffers, (bytes, span), Slot::Link(form), edit_budget);
+ }
+ }
+
+ /// The URL of a slot the fetcher does not take now. The block still holds it, so the writer replaces it.
+ fn push_remote(&mut self, bytes: &[u8], value: Option<Span>) {
+ if let (Some(span), Some(text)) = (value, text(bytes, value)) {
+ self.record_url(bytes, span, &text);
}
}
@@ -559,7 +517,6 @@ impl Walk<'_> {
let Some(srcset) = text(bytes, value) else {
return false;
};
- srcset_urls(&srcset, &mut |url| self.image_urls.push_remote(url));
let shown = srcset::shown_candidate(&srcset, slot);
if let Some(kept) = shown {
self.push_url_text(kept.url);
@@ -591,6 +548,18 @@ impl Walk<'_> {
self.push_srcset_value(bytes, srcset, slot);
}
+ fn push_css_links(&mut self, bytes: &[u8], value: Option<Span>) {
+ if let (Some(value), Some(text)) = (value, text(bytes, value)) {
+ self.image_urls.read_css(
+ &self.buffers,
+ (bytes, value),
+ &text,
+ CssRead::Links,
+ &mut self.resolver,
+ );
+ }
+ }
+
fn push_css(&mut self, bytes: &[u8], value: Option<Span>) {
let Some(value) = value else { return };
if let Some(text) = text(bytes, Some(value)) {
@@ -598,7 +567,7 @@ impl Walk<'_> {
&self.buffers,
(bytes, value),
&text,
- None,
+ CssRead::Sheet,
&mut self.resolver,
);
} else if scan::is_object(bytes, value) {
@@ -615,7 +584,7 @@ impl Walk<'_> {
&self.buffers,
(bytes, at),
&declared,
- Some(property),
+ CssRead::Declaration(property),
&mut self.resolver,
);
}
@@ -679,10 +648,6 @@ pub fn for_each_srcset_candidate<'s>(srcset: &'s str, mut found: impl FnMut(&'s
}
}
-fn srcset_urls(srcset: &str, found: &mut impl FnMut(&str)) {
- for_each_srcset_candidate(srcset, |url, _| found(url));
-}
-
fn style_property_value(bytes: &[u8], span: Span) -> Option<Span> {
if scan::is_array(bytes, span) {
scan::array_items(bytes, span).ok()?.next()?.ok()
@@ -707,6 +672,66 @@ struct Attributes {
width: Option<Span>,
height: Option<Span>,
source_type: Option<Span>,
+ links: Vec<(Span, LinkForm)>,
+ css_links: Vec<Span>,
+}
+
+/// SVG presentation attributes take a CSS value, and a `url()` in one loads a resource.
+fn is_svg_css_attribute(name: &[u8]) -> bool {
+ const NAMES: &[&[u8]] = &[
+ b"clip-path",
+ b"cursor",
+ b"fill",
+ b"filter",
+ b"marker",
+ b"marker-end",
+ b"marker-mid",
+ b"marker-start",
+ b"mask",
+ b"stroke",
+ ];
+ NAMES.iter().any(|known| known.eq_ignore_ascii_case(name))
+}
+
+/// URL attributes that never hold an image slot, and the `data-` attributes that lazy loaders keep a URL in.
+fn link_attribute(name: &[u8]) -> Option<LinkForm> {
+ const URLS: &[&[u8]] = &[
+ b"action",
+ b"formaction",
+ b"data",
+ b"background",
+ b"cite",
+ b"longdesc",
+ b"manifest",
+ b"codebase",
+ b"icon",
+ ];
+ const DATA_URL_SUFFIXES: &[&[u8]] = &[
+ b"src",
+ b"href",
+ b"url",
+ b"original",
+ b"bg",
+ b"background",
+ b"poster",
+ ];
+ if name == b"imagesrcset" {
+ return Some(LinkForm::Srcset);
+ }
+ if name == b"ping" || name == b"attributionsrc" {
+ return Some(LinkForm::Urls);
+ }
+ if URLS.contains(&name) {
+ return Some(LinkForm::Url);
+ }
+ let data = name.strip_prefix(b"data-")?;
+ if data.ends_with(b"srcset") {
+ return Some(LinkForm::Srcset);
+ }
+ DATA_URL_SUFFIXES
+ .iter()
+ .any(|suffix| data.ends_with(suffix))
+ .then_some(LinkForm::Url)
}
impl Attributes {
@@ -729,13 +754,21 @@ impl Attributes {
})
}
+ fn set(&mut self, name: &[u8], value: Span) {
+ if let Some(slot) = self.slot(name) {
+ *slot = Some(value);
+ } else if let Some(form) = link_attribute(name) {
+ self.links.push((value, form));
+ } else if is_svg_css_attribute(name) {
+ self.css_links.push(value);
+ }
+ }
+
fn read(bytes: &[u8], object: Span) -> Self {
let mut attributes = Self::default();
if let Ok(entries) = scan::object_entries(bytes, object) {
for entry in entries.filter_map(Result::ok).filter(|e| !e.key_escaped) {
- if let Some(slot) = attributes.slot(&bytes[entry.key.0..entry.key.1]) {
- *slot = Some(entry.value);
- }
+ attributes.set(&bytes[entry.key.0..entry.key.1], entry.value);
}
}
attributes
@@ -744,13 +777,12 @@ impl Attributes {
/// The sources of a node the window does not know, after a restart, an eviction or the node cap. What the node
/// shows is not known, so they are not fetched. The block holds them, so the writer replaces them.
fn push_unknown_sources(&self, bytes: &[u8], walk: &mut Walk) {
- walk.push_remote(bytes, self.src, false);
- walk.push_remote(bytes, self.rr_src, false);
- walk.push_remote(bytes, self.srcset, true);
+ walk.push_remote(bytes, self.src);
+ walk.push_remote(bytes, self.rr_src);
walk.push_srcset_value(bytes, self.srcset, SrcsetSlot::Alone);
- walk.push_remote(bytes, self.href, false);
- walk.push_remote(bytes, self.xlink_href, false);
- walk.push_remote(bytes, self.poster, false);
+ walk.push_remote(bytes, self.href);
+ walk.push_remote(bytes, self.xlink_href);
+ walk.push_remote(bytes, self.poster);
}
/// A mutation that does not set `src` leaves it unknown, so the img's srcset counts as alone.
@@ -771,8 +803,8 @@ impl Attributes {
Some(Media::Img { visibility }) if !visibility.hides() => {
let slot = self.img_srcset_slot(bytes);
if walk.push_srcset(bytes, self.srcset, slot) {
- walk.push_remote(bytes, self.src, false);
- walk.push_remote(bytes, self.rr_src, false);
+ walk.push_remote(bytes, self.src);
+ walk.push_remote(bytes, self.rr_src);
} else {
walk.push_url(bytes, self.src);
walk.push_url(bytes, self.rr_src);
@@ -780,20 +812,40 @@ impl Attributes {
walk.push_srcset_value(bytes, self.srcset, slot);
}
Some(Media::Img { .. }) => {
- walk.push_remote(bytes, self.src, false);
- walk.push_remote(bytes, self.rr_src, false);
- walk.push_remote(bytes, self.srcset, true);
+ walk.push_remote(bytes, self.src);
+ walk.push_remote(bytes, self.rr_src);
walk.push_srcset_value(bytes, self.srcset, self.img_srcset_slot(bytes));
}
Some(Media::SvgImage) => {
walk.push_url(bytes, self.href);
walk.push_url(bytes, self.xlink_href);
}
- Some(Media::Video) => walk.push_url(bytes, self.poster),
+ Some(Media::Video) => {
+ walk.push_url(bytes, self.poster);
+ walk.push_link(bytes, self.src, LinkForm::Url);
+ }
Some(Media::PictureSource { unfetched_type }) => {
- walk.push_source(bytes, self.srcset, unfetched_type)
+ walk.push_source(bytes, self.srcset, unfetched_type);
+ walk.push_link(bytes, self.src, LinkForm::Url);
}
- _ => {}
+ Some(Media::Picture) | None => {
+ for value in [
+ self.href,
+ self.xlink_href,
+ self.src,
+ self.rr_src,
+ self.poster,
+ ] {
+ walk.push_link(bytes, value, LinkForm::Url);
+ }
+ walk.push_link(bytes, self.srcset, LinkForm::Srcset);
+ }
+ }
+ for &(value, form) in &self.links {
+ walk.push_link(bytes, Some(value), form);
+ }
+ for &value in &self.css_links {
+ walk.push_css_links(bytes, Some(value));
}
walk.push_css(bytes, self.style);
walk.push_css(bytes, self.css_text);
@@ -992,8 +1044,8 @@ fn walk_nodes(bytes: &[u8], span: Span, walk: &mut Walk) -> bool {
}
}
(Some(FrameKind::Attributes), Some(name)) => {
- if let Some(slot) = nodes.last_mut().and_then(|n| n.attributes.slot(name)) {
- *slot = Some(value);
+ if let Some(node) = nodes.last_mut() {
+ node.attributes.set(name, value);
}
}
_ => {}
@@ -1117,8 +1169,9 @@ fn finish_node(bytes: &[u8], node: NodeFrame, parent: Option<&mut NodeFrame>, wa
enum Snapshot<'p> {
/// The player drops it, because its data or its node is not an object.
Dropped,
- /// It does not decode or is too deep to walk, and the player may still build a DOM from a payload this deep.
Unreadable,
+ /// It nests too deep to walk, and the player may still build a DOM from a payload this deep.
+ Unwalked,
Walked(Box<Walk<'p>>),
}
@@ -1133,7 +1186,7 @@ fn walk_snapshot<'p>(
match expansion.json(data) {
Some(Some(json)) => match scan::locate_value(json, 0) {
Ok(root) => (json, root),
- Err(_) => return Snapshot::Unreadable,
+ Err(_) => return Snapshot::Unwalked,
},
Some(None) => return Snapshot::Dropped,
None => return Snapshot::Unreadable,
@@ -1153,7 +1206,7 @@ fn walk_snapshot<'p>(
if walk_nodes(bytes, node, &mut walk) {
Snapshot::Walked(Box::new(walk))
} else {
- Snapshot::Unreadable
+ Snapshot::Unwalked
}
}
@@ -1434,7 +1487,10 @@ impl DomInspector {
self.seen_snapshot = true;
observed.full_snapshot = true;
}
- Snapshot::Unreadable => {
+ unread @ (Snapshot::Unreadable | Snapshot::Unwalked) => {
+ if matches!(unread, Snapshot::Unwalked) {
+ self.urls.limit = self.urls.limit.or(Some(UrlLimit::NestedJson));
+ }
stats.decode_failures += 1;
*dom = WindowDom {
unverifiable: true,
@@ -1507,14 +1563,6 @@ impl DomInspector {
)
}
- pub fn take_declined(&mut self) -> Vec<DeclinedUrl> {
- self.urls.declined_bytes = 0;
- std::mem::take(&mut self.urls.declined)
- .into_iter()
- .map(|(url, (kind, decline))| DeclinedUrl { url, kind, decline })
- .collect()
- }
-
pub fn take_url_limit(&mut self) -> Option<UrlLimit> {
self.urls.limit.take()
}
@@ -1540,6 +1588,8 @@ struct Lists<'b> {
attributes: Option<List<'b>>,
/// A list is a string that did not decode, so the player drops the whole mutation.
failed: bool,
+ /// A list decoded to JSON that nests too deep to read, which the player still reads.
+ too_deep: bool,
}
fn mutation_lists<'b>(inner: &'b [u8], data: Span, expansion: &'b Expansion) -> Lists<'b> {
@@ -1552,7 +1602,7 @@ fn mutation_lists<'b>(inner: &'b [u8], data: Span, expansion: &'b Expansion) ->
&mut lists.texts,
&mut lists.attributes,
];
- let mut failed = false;
+ let (mut failed, mut too_deep) = (false, false);
for (slot, span) in slots.into_iter().zip(found) {
let Some(span) = span else { continue };
if !scan::is_string(inner, span) {
@@ -1563,13 +1613,14 @@ fn mutation_lists<'b>(inner: &'b [u8], data: Span, expansion: &'b Expansion) ->
match expansion.json(span) {
Some(Some(json)) => match scan::locate_value(json, 0) {
Ok(span) => *slot = Some(List { bytes: json, span }),
- Err(_) => failed = true,
+ Err(_) => too_deep = true,
},
Some(None) => {}
None => failed = true,
}
}
- lists.failed = failed;
+ lists.failed = failed || too_deep;
+ lists.too_deep = too_deep;
lists
}
@@ -1634,6 +1685,10 @@ fn observe_mutation(
return (0, 0, true);
}
let lists = mutation_lists(inner, data, expansion);
+ if lists.too_deep {
+ dom.unverifiable = true;
+ urls.limit = urls.limit.or(Some(UrlLimit::NestedJson));
+ }
if lists.failed {
return (0, 0, true);
}
@@ -1644,6 +1699,8 @@ fn observe_mutation(
..Walk::default()
};
let mut adds = Vec::new();
+ // The node cap allows a mutation to add 200,000 nodes, so the set of their ids is built once.
+ let mut walked: Option<HashSet<i64>> = None;
if let Some(list) = lists.adds {
let mut sources = Vec::new();
for item in items(list) {
@@ -1655,6 +1712,7 @@ fn observe_mutation(
if !walk_nodes(list.bytes, node, &mut walk) {
dom.unverifiable = true;
undecoded = true;
+ walk.image_urls.limit = walk.image_urls.limit.or(Some(UrlLimit::NestedJson));
}
for entry in &mut walk.ids[first..] {
entry.1 = entry.1.or(parent);
@@ -1673,9 +1731,14 @@ fn observe_mutation(
.filter(|(_, media)| *media == Media::Picture)
.map(|&(id, _)| id)
.collect();
+ let walked = walked.get_or_insert_with(|| walk.ids.iter().map(|&(id, _)| id).collect());
for (parent, (id, srcset, source_type)) in sources {
let Some(parent) = parent else { continue };
- if pictures.contains(&parent) || dom.media.get(&parent) == Some(&Media::Picture) {
+ // A parent the window does not know can be a picture, as after a restart.
+ if pictures.contains(&parent)
+ || dom.media.get(&parent) == Some(&Media::Picture)
+ || !dom.knows(parent, walked)
+ {
let unfetched_type = is_unfetched_type(list.bytes, source_type);
walk.media
.extend(id.map(|id| (id, Media::PictureSource { unfetched_type })));
@@ -1687,17 +1750,23 @@ fn observe_mutation(
dom.add_media(&walk.media);
dom.add_hidden_sources(&walk.hidden);
dom.add_style_texts(&walk.style_texts);
+ let walked = walked.unwrap_or_else(|| walk.ids.iter().map(|&(id, _)| id).collect());
if let Some(texts) = lists.texts {
for item in items(texts) {
let [id, value] = fields(texts.bytes, item, [b"id", b"value"]);
- if number(texts.bytes, id).is_some_and(|id| dom.style_texts.contains(&(id as i64))) {
+ let Some(id) = number(texts.bytes, id).map(|id| id as i64) else {
+ continue;
+ };
+ if dom.style_texts.contains(&id) {
walk.push_css(texts.bytes, value);
+ } else if !dom.knows(id, &walked) {
+ // A node the window does not know can be the text of a style element, as after a restart.
+ walk.push_css_links(texts.bytes, value);
}
}
}
let mut attribute_ids = Vec::new();
if let Some(attributes) = lists.attributes {
- let walked: HashSet<i64> = walk.ids.iter().map(|&(id, _)| id).collect();
for (index, item) in items(attributes).enumerate() {
if index == MAX_ATTRIBUTE_ITEMS {
walk.image_urls.limit = Some(UrlLimit::AttributeItems);
@@ -1824,17 +1893,21 @@ fn style_sheet_urls(
let resolver = &mut resolver;
let mut read_rule = |bytes: &[u8], span: Option<Span>, urls: &mut Urls| {
if let (Some(span), Some(rule)) = (span, text(bytes, span)) {
- urls.read_css(&buffers, (bytes, span), &rule, None, resolver);
+ urls.read_css(&buffers, (bytes, span), &rule, CssRead::Sheet, resolver);
}
};
match source {
Some(STYLE_SHEET_RULE) => {
let [adds, replace, replace_sync] =
fields(inner, data, [b"adds", b"replace", b"replaceSync"]);
- if let Some(adds) = adds.and_then(|adds| decoded_list(inner, adds, expansion)) {
- for item in items(adds) {
- read_rule(adds.bytes, field(adds.bytes, item, b"rule"), urls);
+ match adds.map(|adds| decoded_list(inner, adds, expansion)) {
+ Some(Ok(Some(adds))) => {
+ for item in items(adds) {
+ read_rule(adds.bytes, field(adds.bytes, item, b"rule"), urls);
+ }
}
+ Some(Err(limit)) => urls.limit = urls.limit.or(Some(limit)),
+ Some(Ok(None)) | None => {}
}
for sheet in [replace, replace_sync] {
read_rule(inner, sheet, urls);
@@ -1847,7 +1920,13 @@ fn style_sheet_urls(
if let (Some(property), Some(at), Some(value)) =
(text(inner, property), value, text(inner, value))
{
- urls.read_css(&buffers, (inner, at), &value, Some(&property), resolver);
+ urls.read_css(
+ &buffers,
+ (inner, at),
+ &value,
+ CssRead::Declaration(&property),
+ resolver,
+ );
}
}
Some(ADOPTED_STYLE_SHEET) => {
@@ -1874,15 +1953,19 @@ fn style_sheet_urls(
}
/// A list as the player reads it, decoded when the event compressed it. `None` when it is empty or does not decode.
-fn decoded_list<'b>(inner: &'b [u8], span: Span, expansion: &'b Expansion) -> Option<List<'b>> {
+fn decoded_list<'b>(
+ inner: &'b [u8],
+ span: Span,
+ expansion: &'b Expansion,
+) -> Result<Option<List<'b>>, UrlLimit> {
if !scan::is_string(inner, span) {
- return Some(List { bytes: inner, span });
+ return Ok(Some(List { bytes: inner, span }));
}
- let json = expansion.json(span)??;
- Some(List {
- bytes: json,
- span: scan::locate_value(json, 0).ok()?,
- })
+ let Some(Some(json)) = expansion.json(span) else {
+ return Ok(None);
+ };
+ let span = scan::locate_value(json, 0).map_err(|_| UrlLimit::NestedJson)?;
+ Ok(Some(List { bytes: json, span }))
}
fn items(list: List<'_>) -> impl Iterator<Item = Span> + '_ {
@@ -2070,11 +2153,12 @@ mod tests {
let text = scan::unescape(&bytes, edit.value).unwrap();
match edit.slot {
Slot::Url => named.urls.push((text.trim().to_owned(), RemoteKind::Image)),
- Slot::Srcset(_) => srcset_urls(&text, &mut |url| {
+ Slot::Srcset(_) => for_each_srcset_candidate(&text, |url, _| {
if is_remote(url) {
named.urls.push((url.to_owned(), RemoteKind::Image));
}
}),
+ Slot::Link(_) => {}
Slot::Css(edit) => {
for piece in edit.pieces {
let urls = piece.parts.iter().filter_map(|part| match part {
@@ -2456,33 +2540,6 @@ mod tests {
);
}
- #[test]
- fn notes_each_refused_url_once_within_the_declined_budget() {
- let declined = |srcset: String| {
- let img = json!({"parentId": 1, "node": element(2, "img", json!({"srcset": srcset}), json!([]))});
- let message = events(json!([mutation(
- json!([img]),
- json!([]),
- json!([]),
- json!([])
- )]));
- let (mut dom, _) = observe_all(&message, "w");
- (dom.take_declined().len(), dom.take_url_limit())
- };
- let repeated = vec!["https://cdn.example.com/a.png?token=x 1x"; 10_000].join(", ");
- assert_eq!(declined(repeated), (1, None));
- let distinct = (0..2_000)
- .map(|i| {
- format!(
- "https://cdn.example.com/{i}/{}.png?token=x {i}w",
- "a".repeat(600)
- )
- })
- .collect::<Vec<_>>()
- .join(", ");
- assert_eq!(declined(distinct).1, Some(UrlLimit::DeclinedUrls));
- }
-
#[test]
fn caps_distinct_remote_image_urls_per_message() {
let img = |i: i64, src: String| json!({"parentId": 1, "node": element(i, "img", json!({"src": src}), json!([]))});
diff --git a/rust/ai-research/replay-core-v4/src/inspect/edits.rs b/rust/ai-research/replay-core-v4/src/inspect/edits.rs
index 1168ebbae565..37ce4733b41c 100644
--- a/rust/ai-research/replay-core-v4/src/inspect/edits.rs
+++ b/rust/ai-research/replay-core-v4/src/inspect/edits.rs
@@ -16,10 +16,22 @@ pub enum EditSource {
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Slot {
- /// A URL attribute. The writer writes the URL's reference, or its decline marker.
+ /// A URL attribute of an image slot. The writer writes the URL's reference, its decline marker, or the URL without its credentials.
Url,
Srcset(SrcsetSlot),
Css(CssEdit),
+ /// A URL attribute that shows no image, such as a link's `href`. No fetch takes it, so the writer keeps it without
+ /// its credentials.
+ Link(LinkForm),
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum LinkForm {
+ Url,
+ /// Candidates with descriptors, as `imagesrcset` holds them.
+ Srcset,
+ /// URLs apart by whitespace, as `ping` and `attributionsrc` hold them.
+ Urls,
}
#[derive(Debug, Clone, PartialEq, Eq)]
@@ -39,8 +51,12 @@ pub struct EventEdits {
}
impl EventEdits {
+ /// A string that is an image slot and also holds a link, such as a picture source's srcset, keeps the image slot.
pub fn new(event: u32, mut edits: Vec<Edit>) -> Self {
- edits.sort_by_key(|edit| (edit.source, edit.value.0));
+ edits.sort_by_key(|edit| {
+ let link = matches!(edit.slot, Slot::Link(_));
+ (edit.source, edit.value.0, link)
+ });
edits.dedup_by_key(|edit| (edit.source, edit.value));
Self { event, edits }
}
diff --git a/rust/ai-research/replay-core-v4/src/inspect/session.rs b/rust/ai-research/replay-core-v4/src/inspect/session.rs
index 06acf0d16519..23bd4ba3195e 100644
--- a/rust/ai-research/replay-core-v4/src/inspect/session.rs
+++ b/rust/ai-research/replay-core-v4/src/inspect/session.rs
@@ -1,13 +1,10 @@
//! Everything the recorder learns about one session, block by block.
-use std::collections::HashMap;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use super::clicks::{ClickKind, ClickTracker};
-use super::dom::{
- DeclinedUrl, DomInspector, FlushBudgets, RemoteKind, UrlLimit, MAX_BLOCK_DECLINED_URL_BYTES,
-};
+use super::dom::{DomInspector, FlushBudgets, UrlLimit};
use super::edits::EventEdits;
use super::events::{href_from, number, Activity, DataFields, EventCounter};
use super::features::{BlockFeatures, FeatureRecorder, SurfacingFeatures};
@@ -16,7 +13,6 @@ use super::rrweb::{
};
use super::stats::{BlockStats, MAX_WINDOW_IDS};
use crate::scan::message::{Event, Message};
-use crate::url_policy::Decline;
/// Moments kept per block, so a long block cannot grow its index row without bound.
pub const MAX_MOMENTS_PER_BLOCK: usize = 256;
@@ -60,8 +56,6 @@ pub struct BlockInspection {
url_limit: Option<UrlLimit>,
edits: Vec<EventEdits>,
unplaced: bool,
- declined: HashMap<String, (RemoteKind, Decline)>,
- declined_bytes: usize,
events: u32,
resolve_budget: Option<Arc<AtomicUsize>>,
edit_budget: Option<Arc<AtomicUsize>>,
@@ -83,8 +77,6 @@ pub struct Inspected {
pub edits: Vec<EventEdits>,
/// The inspector read a slot that the writer could not find again, so the block must not be stored.
pub unplaced: bool,
- /// Spellings of named URLs that the policy refuses, by the largest kind each was named as.
- pub declined: Vec<DeclinedUrl>,
}
impl BlockInspection {
@@ -123,11 +115,6 @@ impl BlockInspection {
url_limit: self.url_limit,
edits: self.edits,
unplaced: self.unplaced,
- declined: self
- .declined
- .into_iter()
- .map(|(url, (kind, decline))| DeclinedUrl { url, kind, decline })
- .collect(),
}
}
}
@@ -227,18 +214,6 @@ impl SessionInspector {
let (edits, unplaced, edit_bytes) = self.dom.take_edits();
block.edit_bytes_taken += edit_bytes;
block.unplaced |= unplaced;
- for declined in self.dom.take_declined() {
- if let Some((kind, _)) = block.declined.get_mut(&declined.url) {
- *kind = (*kind).max(declined.kind);
- } else if block.declined_bytes + declined.url.len() > MAX_BLOCK_DECLINED_URL_BYTES {
- block.url_limit = block.url_limit.or(Some(UrlLimit::DeclinedUrls));
- } else {
- block.declined_bytes += declined.url.len();
- block
- .declined
- .insert(declined.url, (declined.kind, declined.decline));
- }
- }
if block.url_limit.is_none() && !edits.is_empty() {
block.edits.push(EventEdits::new(index, edits));
}
@@ -437,27 +412,34 @@ mod tests {
#[test]
fn drops_a_block_whose_edits_the_flush_cannot_hold() {
- let sheet: String = (0..100)
- .map(|i| format!(".a{i}{{background:url(https://cdn.example.com/{i}.png)}}"))
- .collect();
- let styled = message(
- json!([{"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": [
- {"type": 2, "tagName": "style", "attributes": {"_cssText": sheet}, "childNodes": [], "id": 2},
- ]}}}]),
- );
- let inspect_within = |edit_budget: &Arc<AtomicUsize>| {
+ let styled = |sheet: String| {
+ message(
+ json!([{"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": [
+ {"type": 2, "tagName": "style", "attributes": {"_cssText": sheet}, "childNodes": [], "id": 2},
+ ]}}}]),
+ )
+ };
+ let inspect_within = |message: &Message, edit_budget: &Arc<AtomicUsize>| {
let mut block = BlockInspection::with_flush_budgets(
Arc::new(AtomicUsize::new(usize::MAX)),
edit_budget.clone(),
);
- SessionInspector::default().observe_message(&mut block, &styled);
+ SessionInspector::default().observe_message(&mut block, message);
block.finish()
};
+ let remote = styled(
+ (0..100)
+ .map(|i| format!(".a{i}{{background:url(https://cdn.example.com/{i}.png)}}"))
+ .collect(),
+ );
let unbounded = Arc::new(AtomicUsize::new(usize::MAX));
- inspect_within(&unbounded);
+ inspect_within(&remote, &unbounded);
let held = usize::MAX - unbounded.load(Ordering::Relaxed);
let edit_budget = Arc::new(AtomicUsize::new(held + held / 2));
- let blocks = [inspect_within(&edit_budget), inspect_within(&edit_budget)];
+ let blocks = [
+ inspect_within(&remote, &edit_budget),
+ inspect_within(&remote, &edit_budget),
+ ];
assert_eq!((blocks[0].url_limit, blocks[0].edits.len()), (None, 1));
assert_eq!(
(blocks[1].url_limit, blocks[1].edits.len()),
@@ -465,6 +447,15 @@ mod tests {
);
// The dropped block stopped part way through its style sheet and gave back what it took.
assert_eq!(edit_budget.load(Ordering::Relaxed), held / 2);
+ // The writer leaves relative URLs as they are, so a sheet of them takes nothing.
+ let relative = styled(
+ (0..100)
+ .map(|i| format!(".r{i}{{background:url(img/{i}.png)}}"))
+ .collect(),
+ );
+ let untouched = Arc::new(AtomicUsize::new(usize::MAX));
+ inspect_within(&relative, &untouched);
+ assert_eq!(untouched.load(Ordering::Relaxed), usize::MAX);
}
#[test]
diff --git a/rust/ai-research/replay-core-v4/src/refs.rs b/rust/ai-research/replay-core-v4/src/refs.rs
index 76d0fb353d96..3ecfb6738c19 100644
--- a/rust/ai-research/replay-core-v4/src/refs.rs
+++ b/rust/ai-research/replay-core-v4/src/refs.rs
@@ -9,7 +9,7 @@ use base64::Engine;
use hmac::{Hmac, Mac};
use sha2::{Digest, Sha256};
-use crate::url_policy::{CanonicalUrl, Decline};
+use crate::url_policy::{redaction, CanonicalUrl, Decline, Redaction, MAX_URL_LEN};
/// 64 bits of the HMAC: enough to tell the pages of one session apart.
const FINGERPRINT_BYTES: usize = 8;
@@ -59,6 +59,19 @@ pub fn url_ref(asset: UrlAsset, team_id: i64, key_month: &str, hash: &str) -> St
}
}
+/// A URL that no reference names, as a block keeps it: without its credentials, or as a decline marker when the
+/// redaction cannot read it. `None` when the block keeps the URL as it is.
+pub fn unnamed_url(raw: &str, asset: UrlAsset) -> Option<String> {
+ match redaction(raw) {
+ Redaction::Clean => None,
+ Redaction::Redacted(kept) => Some(kept),
+ Redaction::Unread if raw.len() > MAX_URL_LEN => {
+ Some(declined_url_ref(asset, Decline::TooLong))
+ }
+ Redaction::Unread => Some(declined_url_ref(asset, Decline::NotAbsolute)),
+ }
+}
+
pub fn declined_url_ref(asset: UrlAsset, reason: Decline) -> String {
match asset {
UrlAsset::Image => url_declined_marker(reason),
diff --git a/rust/ai-research/replay-core-v4/src/scan/cv.rs b/rust/ai-research/replay-core-v4/src/scan/cv.rs
index 077c671d64fe..a3b2c8c968f7 100644
--- a/rust/ai-research/replay-core-v4/src/scan/cv.rs
+++ b/rust/ai-research/replay-core-v4/src/scan/cv.rs
@@ -413,7 +413,6 @@ pub fn extract_event_images(
event: &Event,
sink: &mut ImageSink<'_>,
copy: Option<&mut Copier<'_>>,
- before_images: Option<&mut BeforeImages<'_>>,
) -> Option<Vec<u8>> {
let (mut stored, takes_images) = match (&*expansion(inner, event), copy) {
(Expansion::Failed | Expansion::Skipped, _) => (None, false),
@@ -433,11 +432,6 @@ pub fn extract_event_images(
if let Cow::Owned(canonical) = scan::canonical_strings(stored.as_deref().unwrap_or(arrived)) {
stored = Some(canonical);
}
- if let Some(changed) =
- before_images.and_then(|before| before(stored.as_deref().unwrap_or(arrived)))
- {
- stored = Some(changed);
- }
if takes_images {
if let Cow::Owned(extracted) = extract_images(stored.as_deref().unwrap_or(arrived), sink) {
stored = Some(extracted);
@@ -533,9 +527,6 @@ fn write_object(
/// Copies a range of the payload or of a decoded string into the stored event, with what the writer changes there.
pub type Copier<'c> = dyn FnMut(EditSource, &[u8], Span, &mut Vec<u8>) + 'c;
-/// Changes the stored event once its strings are canonical, before moving its inline images out changes them.
-pub type BeforeImages<'c> = dyn FnMut(&[u8]) -> Option<Vec<u8>> + 'c;
-
fn expanded(
inner: &[u8],
event: &Event,
@@ -645,7 +636,7 @@ mod tests {
.iter()
.map(|e| {
serde_json::from_slice(
- &extract_event_images(&message.inner, e, &mut sink, None, None).unwrap(),
+ &extract_event_images(&message.inner, e, &mut sink, None).unwrap(),
)
.unwrap()
})
@@ -698,7 +689,7 @@ mod tests {
.events
.iter()
.map(|e| {
- let stored = extract_event_images(&message.inner, e, &mut sink, None, None)
+ let stored = extract_event_images(&message.inner, e, &mut sink, None)
.unwrap_or_else(|| message.inner[e.span.0..e.span.1].to_vec());
serde_json::from_slice(&stored).unwrap()
})
@@ -742,7 +733,7 @@ mod tests {
);
let mut sink = sink();
assert_eq!(
- extract_event_images(&message.inner, &message.events[1], &mut sink, None, None),
+ extract_event_images(&message.inner, &message.events[1], &mut sink, None),
None
);
@@ -848,7 +839,7 @@ mod tests {
let stored: Vec<Vec<u8>> = message
.events
.iter()
- .map(|e| extract_event_images(&message.inner, e, &mut sink, None, None).unwrap())
+ .map(|e| extract_event_images(&message.inner, e, &mut sink, None).unwrap())
.collect();
assert!(stored
.iter()
@@ -880,8 +871,7 @@ mod tests {
message_with(json!([{"type": 2, "timestamp": 1, "data": compressed, "cv": "2024-10"}]));
message.decode_compressed();
let stored =
- extract_event_images(&message.inner, &message.events[0], &mut sink(), None, None)
- .unwrap();
+ extract_event_images(&message.inner, &message.events[0], &mut sink(), None).unwrap();
assert!(String::from_utf8_lossy(&stored).contains(r#""textContent":"a"#));
assert_eq!(
serde_json::from_slice::<Value>(&stored).unwrap(),
@@ -907,7 +897,7 @@ mod tests {
let mut sink = sink();
for event in &message.events {
assert_eq!(
- extract_event_images(&message.inner, event, &mut sink, None, None),
+ extract_event_images(&message.inner, event, &mut sink, None),
None
);
}
@@ -1018,9 +1008,8 @@ mod tests {
for (event, pointer, outcome) in cases {
let message = message_with(json!([event]));
let mut sink = sink();
- let stored =
- extract_event_images(&message.inner, &message.events[0], &mut sink, None, None)
- .map(|stored| serde_json::from_slice::<Value>(&stored).unwrap());
+ let stored = extract_event_images(&message.inner, &message.events[0], &mut sink, None)
+ .map(|stored| serde_json::from_slice::<Value>(&stored).unwrap());
let replacement = match outcome {
Outcome::Kept => {
assert_eq!(stored, None, "{event}");
diff --git a/rust/ai-research/replay-core-v4/src/url_policy.rs b/rust/ai-research/replay-core-v4/src/url_policy.rs
index e3be18721cf7..4df293d0744b 100644
--- a/rust/ai-research/replay-core-v4/src/url_policy.rs
+++ b/rust/ai-research/replay-core-v4/src/url_policy.rs
@@ -11,6 +11,7 @@
//! The fetch lane runs in another language. The registrable domain therefore travels with each
//! URL as data, so that lane reads the result and never repeats the rule.
+use std::borrow::Cow;
use std::collections::HashSet;
use std::net::IpAddr;
use std::sync::LazyLock;
@@ -20,8 +21,11 @@ use public_suffix::{EffectiveTLDProvider, DEFAULT_PROVIDER};
use url::Url;
+mod redact;
mod shopify;
+pub use redact::{needs_redaction, redaction, without_credentials, Redaction};
+
const VOLATILE_PARAMS: &[&str] = &["cb", "nocache", "rnd"];
const CREDENTIAL_PARAMS: &[&str] = &[
@@ -371,7 +375,30 @@ pub fn canonicalize(raw: &str) -> Option<CanonicalUrl> {
/// Whether the URL names a host but no scheme, such as `//cdn.example.com/a.png`. The browser loads it with the page's
/// scheme, so it is remote. A browser reads a backslash as a slash in an `https` URL, so `\\host` and `/\host` count too.
pub fn is_protocol_relative(raw: &str) -> bool {
- matches!(raw.trim().as_bytes(), [b'/' | b'\\', b'/' | b'\\', ..])
+ matches!(
+ parsed_start(raw, 8).trim_start().as_bytes(),
+ [b'/' | b'\\', b'/' | b'\\', ..]
+ )
+}
+
+/// The first `len` characters of `raw` as [`as_parsed`] reads them, from no more of `raw` than they need, because a
+/// slot can hold a data URI of many megabytes.
+pub fn parsed_start(raw: &str, len: usize) -> String {
+ raw.trim_start_matches(|ch: char| ch <= ' ')
+ .chars()
+ .filter(|ch| !matches!(ch, '\t' | '\n' | '\r'))
+ .take(len)
+ .collect()
+}
+
+/// The URL parser strips C0 controls and spaces from both ends of a URL and drops every ASCII tab and line break anywhere in it, so `\u{1}https://cdn.example.com` and `/\n/cdn.example.com` load from that host.
+pub fn as_parsed(raw: &str) -> Cow<'_, str> {
+ let raw = raw.trim_matches(|c: char| c <= ' ');
+ if raw.contains(['\t', '\n', '\r']) {
+ Cow::Owned(raw.replace(['\t', '\n', '\r'], ""))
+ } else {
+ Cow::Borrowed(raw)
+ }
}
/// Whether the browser loads the URL from a host. A URL scheme is not case sensitive, so `HTTPS://` loads as
@@ -379,6 +406,8 @@ pub fn is_protocol_relative(raw: &str) -> bool {
/// all on a page of another scheme, so `https:\\host` and `https:host` can load from the host too. A
/// protocol-relative URL does as well.
pub fn is_remote(url: &str) -> bool {
+ let url = parsed_start(url, 7);
+ let url = url.as_str();
let after_scheme = ["https:", "http:"].iter().find_map(|scheme| {
url.get(..scheme.len())
.filter(|start| start.eq_ignore_ascii_case(scheme))
@@ -415,7 +444,7 @@ pub fn try_canonicalize(raw: &str) -> Result<CanonicalUrl, Decline> {
// check that the same URL with a scheme meets.
let with_scheme;
let raw = if is_protocol_relative(raw) {
- with_scheme = format!("https:{}", raw.trim());
+ with_scheme = format!("https:{}", as_parsed(raw).trim());
with_scheme.as_str()
} else {
raw
@@ -452,7 +481,7 @@ pub fn try_canonicalize(raw: &str) -> Result<CanonicalUrl, Decline> {
url.set_host(Some(&host)).map_err(|_| Decline::NoHost)?;
}
let lowercase_path = decoded_lowercase_path(&url)?;
- if has_credential_query(&url)? || has_credential_path(&lowercase_path, &host) {
+ if has_credential_query(&url)? || has_credential_path(&lowercase_path, Some(&host)) {
return Err(Decline::Credential);
}
if matches_beacon_list(&host, &collapse_slashes(&lowercase_path)) {
@@ -512,12 +541,9 @@ fn has_credential_query(url: &Url) -> Result<bool, Decline> {
.decode_utf8()
.map_err(|_| Decline::InvalidQuery)?;
}
- Ok(url.query_pairs().any(|(name, value)| {
- CREDENTIAL_PARAMS
- .iter()
- .any(|credential| credential.eq_ignore_ascii_case(&name))
- || (name.eq_ignore_ascii_case("s") && value.chars().count() == 32)
- }))
+ Ok(url
+ .query_pairs()
+ .any(|(name, value)| is_credential_param(&name, &value)))
}
fn has_valid_percent_encoding(value: &str) -> bool {
@@ -551,40 +577,70 @@ fn decoded_lowercase_path(url: &Url) -> Result<String, Decline> {
.to_ascii_lowercase())
}
-/// `path` is the decoded lowercase path from [`decoded_lowercase_path`].
-fn has_credential_path(path: &str, host: &str) -> bool {
+/// `path` is the decoded lowercase path from [`decoded_lowercase_path`], and `host` is `None` for a relative URL.
+fn has_credential_path(path: &str, host: Option<&str>) -> bool {
let segments: Vec<&str> = path.split('/').collect();
+ let has_supabase_signature = path.contains("/storage/v1/object/sign/")
+ || path.contains("/storage/v1/render/image/sign/");
+ has_supabase_signature || !path_tokens(&segments, host).is_empty()
+}
+
+fn is_oracle_object_storage(host: &str) -> bool {
+ host.starts_with("objectstorage.")
+ && host.ends_with(".oraclecloud.com")
+ && host.split('.').count() == 4
+}
+
+/// Where a path holds a token: a whole segment, or a `;jsessionid=` parameter in one.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum PathToken {
+ Segment(usize),
+ SessionId(usize),
+}
+
+/// `segments` is a decoded lowercase path split at `/`, so the first one is empty. `host` is `None` for a relative URL,
+/// whose host is the page's, so a rule for one host then matches it on any page.
+fn path_tokens(segments: &[&str], host: Option<&str>) -> Vec<PathToken> {
+ let mut tokens = Vec::new();
let first_segment_has_bunny_token = segments
.get(1)
.and_then(|segment| segment.strip_prefix("bcdn_token="))
.and_then(|value| value.split('&').next())
.is_some_and(|token| !token.is_empty());
- let has_oracle_token = host.starts_with("objectstorage.")
- && host.ends_with(".oraclecloud.com")
- && host.split('.').count() == 4
+ if first_segment_has_bunny_token {
+ tokens.push(PathToken::Segment(1));
+ }
+ let has_oracle_token = host.is_none_or(is_oracle_object_storage)
&& segments.get(1) == Some(&"p")
&& segments.get(2).is_some_and(|token| !token.is_empty())
&& segments.get(3) == Some(&"n");
- let has_cloudinary_signature = segments.iter().any(|segment| {
- segment
+ if has_oracle_token {
+ tokens.push(PathToken::Segment(2));
+ }
+ for (index, segment) in segments.iter().enumerate() {
+ let has_cloudinary_signature = segment
.strip_prefix("s--")
.and_then(|value| value.strip_suffix("--"))
- .is_some_and(|token| !token.is_empty())
- });
- let has_supabase_signature = path.contains("/storage/v1/object/sign/")
- || path.contains("/storage/v1/render/image/sign/");
- let has_session_token = segments.iter().any(|segment| {
- segment
+ .is_some_and(|token| !token.is_empty());
+ if has_cloudinary_signature {
+ tokens.push(PathToken::Segment(index));
+ }
+ let has_session_token = segment
.split_once(";jsessionid=")
.and_then(|(_, value)| value.split(';').next())
- .is_some_and(|token| !token.is_empty())
- });
+ .is_some_and(|token| !token.is_empty());
+ if has_session_token {
+ tokens.push(PathToken::SessionId(index));
+ }
+ }
+ tokens
+}
- first_segment_has_bunny_token
- || has_oracle_token
- || has_cloudinary_signature
- || has_supabase_signature
- || has_session_token
+fn is_credential_param(name: &str, value: &str) -> bool {
+ CREDENTIAL_PARAMS
+ .iter()
+ .any(|credential| credential.eq_ignore_ascii_case(name))
+ || (name.eq_ignore_ascii_case("s") && value.chars().count() == 32)
}
fn remove_volatile_params(raw_query: Option<&str>) -> Result<Option<String>, Decline> {
diff --git a/rust/ai-research/replay-core-v4/src/url_policy/redact.rs b/rust/ai-research/replay-core-v4/src/url_policy/redact.rs
new file mode 100644
index 000000000000..7959ea57cc8b
--- /dev/null
+++ b/rust/ai-research/replay-core-v4/src/url_policy/redact.rs
@@ -0,0 +1,540 @@
+//! A URL as the page wrote it, without what the policy reads as a credential. v4 stores pages unscrubbed, so the recorder
+//! removes only these, from the URLs that it does not replace with a reference.
+
+use std::ops::Range;
+use std::sync::LazyLock;
+
+use percent_encoding::percent_decode_str;
+use url::{form_urlencoded, Url};
+
+use super::{
+ as_parsed, decoded_lowercase_path, has_credential_path, has_credential_query,
+ is_credential_param, path_tokens, PathToken, MAX_URL_LEN,
+};
+
+/// The URL parser reads these with an authority, and a backslash in them as a slash. A URL with no scheme resolves
+/// against a page of one of them.
+const SPECIAL_SCHEMES: &[&str] = &["http", "https", "ws", "wss", "ftp", "file"];
+
+/// A relative URL resolves against this, so that the policy reads its path and query as it reads an absolute URL's.
+static RELATIVE_BASE: LazyLock<Url> =
+ LazyLock::new(|| Url::parse("https://relative.invalid/").expect("a valid base"));
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum Redaction {
+ Clean,
+ Redacted(String),
+ /// The URL can hold a credential that the redaction cannot cut out: it is longer than [`MAX_URL_LEN`], the URL parser
+ /// does not read it, or one reading of it takes the credential as its host.
+ Unread,
+}
+
+/// `raw` without its userinfo, the credential parameters of its query and fragment, and the path segments that hold a
+/// token. The rest stays as the page wrote it, apart from the tabs, line breaks and outer spaces that the URL parser
+/// drops. When the policy still reads a credential after those cuts, as in a path that hides a token behind a dot
+/// segment, only the URL's origin stays, and none of it when the origin holds one too.
+pub fn redaction(raw: &str) -> Redaction {
+ // A `data:` URI can be tens of megabytes, so its scheme is read before anything else.
+ if !is_hierarchical(raw) {
+ return Redaction::Clean;
+ }
+ let url = as_parsed(raw);
+ // A text check reads a URL otherwise than the URL parser, which maps a host with IDNA, decodes it and drops dot
+ // segments. So every URL within the fetcher's limit is parsed, and a longer one keeps none of its text.
+ if url.len() > MAX_URL_LEN {
+ return Redaction::Unread;
+ }
+ let read = readings(&url);
+ let Some(first) = read.first() else {
+ return Redaction::Unread;
+ };
+ if !read.iter().any(holds_credential) {
+ return Redaction::Clean;
+ }
+ let Some(parts) = Parts::of(&url) else {
+ return Redaction::Unread;
+ };
+ let is_clean = |text: &str| {
+ let read = readings(text);
+ !read.is_empty() && !read.iter().any(holds_credential)
+ };
+ let cut = parts.without_credentials(&url, page_host(first));
+ if is_clean(&cut) {
+ return Redaction::Redacted(cut);
+ }
+ let origin = parts.origin(&url);
+ if is_clean(&origin) {
+ return Redaction::Redacted(origin);
+ }
+ Redaction::Unread
+}
+
+pub fn needs_redaction(raw: &str) -> bool {
+ redaction(raw) != Redaction::Clean
+}
+
+pub fn without_credentials(raw: &str) -> Option<String> {
+ match redaction(raw) {
+ Redaction::Redacted(kept) => Some(kept),
+ Redaction::Clean | Redaction::Unread => None,
+ }
+}
+
+/// The URLs that a browser can read `url` as. The policy parses a URL with a scheme on its own, where `https:host/a`
+/// names a host, but on a page of the same scheme that URL is the path `host/a` when no two slashes follow the scheme.
+fn readings(url: &str) -> Vec<Url> {
+ let Ok(alone) = Url::parse(url) else {
+ return RELATIVE_BASE.join(url).into_iter().collect();
+ };
+ let after_scheme = alone.scheme().len() + 1;
+ let names_its_host = url
+ .as_bytes()
+ .get(after_scheme..after_scheme + 2)
+ .is_some_and(|slashes| slashes.iter().all(|&byte| matches!(byte, b'/' | b'\\')));
+ let reads_as_a_path =
+ alone.scheme() != "file" && SPECIAL_SCHEMES.contains(&alone.scheme()) && !names_its_host;
+ let on_a_page = reads_as_a_path
+ .then(|| {
+ let mut base = RELATIVE_BASE.clone();
+ base.set_scheme(alone.scheme()).ok()?;
+ base.join(url).ok()
+ })
+ .flatten();
+ std::iter::once(alone).chain(on_a_page).collect()
+}
+
+fn holds_credential(parsed: &Url) -> bool {
+ if !parsed.username().is_empty() || parsed.password().is_some() {
+ return true;
+ }
+ if has_credential_query(parsed).unwrap_or(true) {
+ return true;
+ }
+ match decoded_lowercase_path(parsed) {
+ Ok(path) if has_credential_path(&path, page_host(parsed)) => return true,
+ Ok(_) => {}
+ Err(_) => return true,
+ }
+ // A fragment holds parameters only by convention, as an OAuth callback's `#access_token=` does.
+ parsed.fragment().is_some_and(|fragment| {
+ fragment.contains('=')
+ && form_urlencoded::parse(fragment.as_bytes())
+ .any(|(name, value)| is_credential_param(&name, &value))
+ })
+}
+
+/// The host as the policy reads it, or `None` for a relative URL, whose host is the page's.
+fn page_host(parsed: &Url) -> Option<&str> {
+ parsed
+ .host_str()
+ .filter(|&host| Some(host) != RELATIVE_BASE.host_str())
+ .map(|host| host.strip_suffix('.').unwrap_or(host))
+}
+
+/// A URL of a scheme such as `data:` or `mailto:` holds no path to a host. The URL parser drops tabs and line breaks
+/// in a scheme too, and reads a scheme only from a letter on, so `1:a.png` is a relative path.
+fn is_hierarchical(raw: &str) -> bool {
+ let bytes = raw.as_bytes();
+ let ignored = |byte: &u8| matches!(byte, b'\t' | b'\n' | b'\r');
+ let mut scheme = Vec::new();
+ let mut at = bytes
+ .iter()
+ .position(|&byte| byte > b' ')
+ .unwrap_or(bytes.len());
+ while let Some(&byte) = bytes.get(at) {
+ at += 1;
+ match byte {
+ _ if ignored(&byte) => {}
+ b':' if !scheme.is_empty() => {
+ if SPECIAL_SCHEMES.contains(&String::from_utf8_lossy(&scheme).as_ref()) {
+ return true;
+ }
+ let mut rest = bytes[at..].iter().filter(|byte| !ignored(byte));
+ return rest.next() == Some(&b'/') && rest.next() == Some(&b'/');
+ }
+ b'a'..=b'z' | b'A'..=b'Z' if scheme.len() < 16 => {
+ scheme.push(byte.to_ascii_lowercase())
+ }
+ b'0'..=b'9' | b'+' | b'-' | b'.' if !scheme.is_empty() && scheme.len() < 16 => {
+ scheme.push(byte)
+ }
+ _ => return true,
+ }
+ }
+ true
+}
+
+/// The parts of a URL by byte offset. The query and the fragment start after their `?` and `#`.
+struct Parts {
+ special: bool,
+ authority: Option<Range<usize>>,
+ path: Range<usize>,
+ query: Option<Range<usize>>,
+ fragment: Option<Range<usize>>,
+}
+
+impl Parts {
+ fn of(url: &str) -> Option<Self> {
+ let bytes = url.as_bytes();
+ let byte_at = |at: usize| bytes.get(at).copied();
+ let scheme_end = bytes
+ .iter()
+ .position(|&byte| !(byte.is_ascii_alphanumeric() || matches!(byte, b'+' | b'-' | b'.')))
+ .filter(|&end| end > 0 && bytes[0].is_ascii_alphabetic() && byte_at(end) == Some(b':'));
+ let scheme = scheme_end.map_or_else(String::new, |end| url[..end].to_ascii_lowercase());
+ let special = scheme.is_empty() || SPECIAL_SCHEMES.contains(&scheme.as_str());
+ let is_slash = |byte: u8| byte == b'/' || (special && byte == b'\\');
+ let after_scheme = scheme_end.map_or(0, |end| end + 1);
+ let mut at = after_scheme;
+ while byte_at(at).is_some_and(is_slash) {
+ at += 1;
+ }
+ let has_authority = (special && !scheme.is_empty()) || at - after_scheme >= 2;
+ if !has_authority {
+ if !special {
+ return None;
+ }
+ at = after_scheme;
+ }
+ let authority = has_authority.then(|| {
+ let start = at;
+ while byte_at(at).is_some_and(|byte| !(is_slash(byte) || byte == b'?' || byte == b'#'))
+ {
+ at += 1;
+ }
+ start..at
+ });
+ let path_start = at;
+ while byte_at(at).is_some_and(|byte| byte != b'?' && byte != b'#') {
+ at += 1;
+ }
+ let path = path_start..at;
+ let mut query = None;
+ if byte_at(at) == Some(b'?') {
+ at += 1;
+ let start = at;
+ while byte_at(at).is_some_and(|byte| byte != b'#') {
+ at += 1;
+ }
+ query = Some(start..at);
+ }
+ let fragment = (byte_at(at) == Some(b'#')).then(|| at + 1..bytes.len());
+ Some(Self {
+ special,
+ authority,
+ path,
+ query,
+ fragment,
+ })
+ }
+
+ /// Where the host starts after any userinfo.
+ fn host_start(&self, url: &str, authority: &Range<usize>) -> usize {
+ url[authority.clone()]
+ .rfind('@')
+ .map_or(authority.start, |at| authority.start + at + 1)
+ }
+
+ /// `host` is the host as the policy reads it, which the path's tokens can depend on.
+ fn without_credentials(&self, url: &str, host: Option<&str>) -> String {
+ let mut out = String::with_capacity(url.len());
+ match &self.authority {
+ Some(authority) => {
+ out.push_str(&url[..authority.start]);
+ out.push_str(&url[self.host_start(url, authority)..authority.end]);
+ }
+ None => out.push_str(&url[..self.path.start]),
+ }
+ let mut copied_to = self.path.start;
+ for cut in self.path_cuts(url, host) {
+ out.push_str(&url[copied_to..cut.start]);
+ copied_to = cut.end;
+ }
+ out.push_str(&url[copied_to..self.path.end]);
+ for (mark, part) in [('?', &self.query), ('#', &self.fragment)] {
+ let Some(part) = part.clone() else { continue };
+ let holds_parameters = mark == '?' || url[part.clone()].contains('=');
+ match holds_parameters
+ .then(|| without_credential_params(&url[part.clone()]))
+ .flatten()
+ {
+ Some(kept) if kept.is_empty() => {}
+ Some(kept) => {
+ out.push(mark);
+ out.push_str(&kept);
+ }
+ None => out.push_str(&url[part.start - 1..part.end]),
+ }
+ }
+ out
+ }
+
+ /// The scheme and host without the userinfo, or `/` for a relative URL.
+ fn origin(&self, url: &str) -> String {
+ match &self.authority {
+ Some(authority) => format!(
+ "{}{}/",
+ &url[..authority.start],
+ &url[self.host_start(url, authority)..authority.end]
+ ),
+ None if url.starts_with(['/', '\\']) => "/".to_owned(),
+ None => String::new(),
+ }
+ }
+
+ /// The ranges of the path to drop, in order: each segment that holds a token with the slash before it, and each
+ /// `;jsessionid=` parameter.
+ fn path_cuts(&self, url: &str, host: Option<&str>) -> Vec<Range<usize>> {
+ let bytes = url.as_bytes();
+ let is_slash = |byte: u8| byte == b'/' || (self.special && byte == b'\\');
+ let mut segments = Vec::new();
+ let mut start = self.path.start;
+ for at in self.path.clone() {
+ if is_slash(bytes[at]) {
+ segments.push(start..at);
+ start = at + 1;
+ }
+ }
+ segments.push(start..self.path.end);
+ let decoded: Vec<String> = segments
+ .iter()
+ .map(|segment| {
+ percent_decode_str(&url[segment.clone()])
+ .decode_utf8_lossy()
+ .to_ascii_lowercase()
+ })
+ .collect();
+ let decoded: Vec<&str> = decoded.iter().map(String::as_str).collect();
+ let mut cuts: Vec<Range<usize>> = Vec::new();
+ for token in path_tokens(&decoded, host) {
+ match token {
+ PathToken::Segment(index) => {
+ let segment = segments[index].clone();
+ cuts.push(segment.start.saturating_sub(1).max(self.path.start)..segment.end);
+ }
+ PathToken::SessionId(index) => {
+ cuts.extend(session_id_parameters(url, segments[index].clone()))
+ }
+ }
+ }
+ cuts.sort_by_key(|cut| cut.start);
+ cuts.dedup_by(|later, earlier| {
+ let overlaps = later.start < earlier.end;
+ if overlaps {
+ earlier.end = earlier.end.max(later.end);
+ }
+ overlaps
+ });
+ cuts
+ }
+}
+
+/// Each `;jsessionid=` parameter of `segment` up to the next `;`, or the whole segment when the page percent-encoded
+/// one.
+fn session_id_parameters(url: &str, segment: Range<usize>) -> Vec<Range<usize>> {
+ const NAME: &[u8] = b";jsessionid=";
+ let written = &url.as_bytes()[segment.clone()];
+ let mut cuts = Vec::new();
+ let mut at = 0;
+ while at + NAME.len() <= written.len() {
+ if !written[at..at + NAME.len()].eq_ignore_ascii_case(NAME) {
+ at += 1;
+ continue;
+ }
+ let end = written[at + 1..]
+ .iter()
+ .position(|&byte| byte == b';')
+ .map_or(written.len(), |next| at + 1 + next);
+ cuts.push(segment.start + at..segment.start + end);
+ at = end;
+ }
+ if cuts.is_empty() {
+ cuts.push(segment);
+ }
+ cuts
+}
+
+/// The parameters of `part` that are not credentials, joined by `&`, or `None` when it holds no credential. A name
+/// that does not decode is dropped as well, because the policy cannot tell what it names.
+fn without_credential_params(part: &str) -> Option<String> {
+ let mut dropped = false;
+ let kept: Vec<&str> = part
+ .split('&')
+ .filter(|pair| {
+ let (name, value) = pair.split_once('=').unwrap_or((pair, ""));
+ let credential = match form_decoded(name) {
+ Some(name) => is_credential_param(&name, &lossy_form_decoded(value)),
+ None => true,
+ };
+ dropped |= credential;
+ !credential
+ })
+ .collect();
+ dropped.then(|| kept.join("&"))
+}
+
+fn form_decoded(text: &str) -> Option<String> {
+ let text = text.replace('+', " ");
+ let bytes = text.as_bytes();
+ let valid = bytes.iter().enumerate().all(|(at, &byte)| {
+ byte != b'%'
+ || (bytes.get(at + 1).is_some_and(u8::is_ascii_hexdigit)
+ && bytes.get(at + 2).is_some_and(u8::is_ascii_hexdigit))
+ });
+ if !valid {
+ return None;
+ }
+ percent_decode_str(&text)
+ .decode_utf8()
+ .ok()
+ .map(|decoded| decoded.into_owned())
+}
+
+fn lossy_form_decoded(text: &str) -> String {
+ percent_decode_str(&text.replace('+', " "))
+ .decode_utf8_lossy()
+ .into_owned()
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::url_policy::checked_url;
+ use crate::url_policy::Decline;
+
+ #[test]
+ fn drops_only_the_credentials_of_a_url() {
+ for (raw, kept) in [
+ (
+ "https://cdn.example.com/a.png?w=10&X-Amz-Signature=abc&h=20#top",
+ Some("https://cdn.example.com/a.png?w=10&h=20#top"),
+ ),
+ (
+ "https://cdn.example.com/a.png?token=abc",
+ Some("https://cdn.example.com/a.png"),
+ ),
+ (
+ "https://user:[email protected]/a.png",
+ Some("https://cdn.example.com/a.png"),
+ ),
+ (
+ r"HTTPS:\\user:[email protected]\a.png?Sig=abc",
+ Some(r"HTTPS:\\CDN.example.com\a.png"),
+ ),
+ (
+ " //cdn.example.com/a.png?to\tken=abc&w=1 ",
+ Some("//cdn.example.com/a.png?w=1"),
+ ),
+ ("/api/img?access_token=abc", Some("/api/img")),
+ // A relative URL's host is the page's, which can be an Oracle bucket.
+ (
+ "/p/abc/n/ns/b/bucket/o/a.png",
+ Some("/p/n/ns/b/bucket/o/a.png"),
+ ),
+ ("img/a.png?w=1&token=abc", Some("img/a.png?w=1")),
+ ("1:a.png?token=abc", Some("1:a.png")),
+ (":a.png?token=abc", Some(":a.png")),
+ (
+ "https://app.example.com/cb#access_token=abc&state=x",
+ Some("https://app.example.com/cb#state=x"),
+ ),
+ (
+ "https://app.example.com/a;jsessionid=abc;x=1/b.png",
+ Some("https://app.example.com/a;x=1/b.png"),
+ ),
+ (
+ "https://app.example.com/a;jsessionid=first;jsessionid=abc/b.png",
+ Some("https://app.example.com/a/b.png"),
+ ),
+ (
+ "https://res.example.com/image/upload/s--abc123--/v1/a.png",
+ Some("https://res.example.com/image/upload/v1/a.png"),
+ ),
+ (
+ "https://cdn.example.com/a.png?%ff=1&w=2",
+ Some("https://cdn.example.com/a.png?w=2"),
+ ),
+ // The policy reads these paths decoded and without dot segments, so only the origin stays.
+ (
+ "https://res.example.com/v1%2Fs--abc--/a.png",
+ Some("https://res.example.com/"),
+ ),
+ (
+ "https://cdn.example.com/x/../bcdn_token=abc/a.png",
+ Some("https://cdn.example.com/"),
+ ),
+ (
+ r"https://cdn.example.com\storage\v1\object\sign\abc",
+ Some("https://cdn.example.com/"),
+ ),
+ // IDNA maps fullwidth letters to ASCII, so this host is an Oracle bucket.
+ (
+ "https://objectstorage.r.oraclecloud.com/p/abc/n/ns",
+ Some("https://objectstorage.r.oraclecloud.com/p/n/ns"),
+ ),
+ ("https://cdn.example.com/a.png?w=10", None),
+ ("https://cdn.example.com/a.png#token", None),
+ ("data:image/png;base64,abc?token=x", None),
+ ("mailto:[email protected]?token=x", None),
+ ] {
+ assert_eq!(without_credentials(raw).as_deref(), kept, "{raw}");
+ }
+ }
+
+ #[test]
+ fn leaves_no_url_that_the_policy_refuses_as_a_credential() {
+ for raw in [
+ "https://cdn.example.com/a.png?w=10&X-Amz-Signature=abc&X-Amz-Credential=k",
+ "https://user:[email protected]/a.png",
+ "https://cdn.example.com/a.png?s=0123456789abcdef0123456789abcdef",
+ "https://objectstorage.us-ashburn-1.oraclecloud.com/p/secret/n/ns/b/bucket/o/a.png",
+ "https://objectstorage.us-ashburn-1.oraclecloud.com./p/secret/n/ns/b/bucket/o/a.png",
+ "https://%6fbjectstorage.us-ashburn-1.oraclecloud.com/p/secret/n/ns/b/bucket/o/a.png",
+ "https://cdn.example.com/bcdn_token=secret&expires=1/a.png",
+ "https://cdn.example.com/image/upload%2Fs--secret--%2Fv1/a.png",
+ "https://cdn.example.com/a;jsessionid=first;jsessionid=secret",
+ "https://cdn.example.com/x/./../bcdn_token=secret/a.png",
+ "https://OBJECTSTORAGE.r.oraclecloud.com/p/secret/n/ns",
+ ] {
+ assert_eq!(checked_url(raw).err(), Some(Decline::Credential), "{raw}");
+ let kept = without_credentials(raw).unwrap();
+ assert_ne!(
+ checked_url(&kept).err(),
+ Some(Decline::Credential),
+ "{kept}"
+ );
+ assert!(!kept.contains("secret") && !kept.contains("abc"), "{kept}");
+ }
+ }
+
+ #[test]
+ fn marks_a_url_whose_credential_it_cannot_cut() {
+ let long = format!(
+ "https://cdn.example.com{}?token=abc",
+ "/a".repeat(16 * 1024 * 1024)
+ );
+ assert_eq!(redaction(&long), Redaction::Unread);
+ let padding = "/a".repeat(MAX_URL_LEN);
+ for long in [
+ format!("https://cdn.example.com/bcdn_to\tken=abc{padding}"),
+ format!("/p/abc/n/ns/b/bucket/o{padding}"),
+ format!("https://cdn.example.com{padding}"),
+ ] {
+ assert_eq!(redaction(&long), Redaction::Unread);
+ }
+ assert_eq!(
+ redaction("https://cdn.example.com:99999/a.png?token=abc"),
+ Redaction::Unread
+ );
+ // On a page of their scheme these are paths, so what the URL parser alone reads as a host holds the token.
+ for raw in ["https:bcdn_token=abc/a.png", "https:/bcdn_token=abc/a.png"] {
+ assert_eq!(redaction(raw), Redaction::Unread, "{raw}");
+ }
+ }
+
+ #[test]
+ fn reads_no_more_of_a_data_uri_than_its_scheme() {
+ let data = format!("data:image/png;base64,{}", "A/".repeat(16 * 1024 * 1024));
+ assert_eq!(without_credentials(&data), None);
+ }
+}
diff --git a/rust/ai-research/replay-recorder-v4/Cargo.toml b/rust/ai-research/replay-recorder-v4/Cargo.toml
index 41b1cb956849..43f38a73a35e 100644
--- a/rust/ai-research/replay-recorder-v4/Cargo.toml
+++ b/rust/ai-research/replay-recorder-v4/Cargo.toml
@@ -19,7 +19,6 @@ common-redis = { path = "../../common/redis" }
lifecycle = { path = "../../common/lifecycle" }
replay-core-v4 = { path = "../replay-core-v4" }
-aho-corasick = { workspace = true }
arc-swap = { workspace = true }
async-trait = { workspace = true }
aws-config = { workspace = true }
@@ -47,5 +46,6 @@ tracing-subscriber = { workspace = true }
[dev-dependencies]
aes-gcm = "0.10"
base64 = { workspace = true }
+hegeltest = { version = "0.50.1", features = ["static-engine"] }
serde_json = { workspace = true }
tokio = { workspace = true, features = ["test-util"] }
diff --git a/rust/ai-research/replay-recorder-v4/src/batch.rs b/rust/ai-research/replay-recorder-v4/src/batch.rs
index acc4a52d4b57..7b5270d13a43 100644
--- a/rust/ai-research/replay-recorder-v4/src/batch.rs
+++ b/rust/ai-research/replay-recorder-v4/src/batch.rs
@@ -8,7 +8,7 @@ use std::time::{Duration, Instant};
use futures::stream::{self, StreamExt};
use metrics::counter;
use replay_core_v4::compression::{GZIP_MAGIC, MAX_DECOMPRESSED_BYTES};
-use replay_core_v4::inspect::dom::{DeclinedUrl, UrlLimit};
+use replay_core_v4::inspect::dom::UrlLimit;
use replay_core_v4::inspect::edits::EventEdits;
use replay_core_v4::inspect::session::{BlockInspection, Moment, SessionInspector};
use replay_core_v4::inspect::stats::BlockStats;
@@ -101,7 +101,6 @@ pub struct Block {
/// Set when the inspector could not read the block in full, so that the block must not be stored.
pub url_limit: Option<UrlLimit>,
pub edits: Vec<EventEdits>,
- pub declined: Vec<DeclinedUrl>,
pub unplaced: bool,
/// The distinct pages of the block's surfacing features, which the writer stores as fingerprints.
pub visited_urls: Vec<String>,
@@ -543,7 +542,6 @@ fn inspect_session(
font_urls: inspected.font_urls,
url_limit: inspected.url_limit,
edits: inspected.edits,
- declined: inspected.declined,
unplaced: inspected.unplaced,
visited_urls: inspected.features.visited_urls,
captured_at_ms,
diff --git a/rust/ai-research/replay-recorder-v4/src/consumer.rs b/rust/ai-research/replay-recorder-v4/src/consumer.rs
index 7dd7faa751c7..ab691fe1b46a 100644
--- a/rust/ai-research/replay-recorder-v4/src/consumer.rs
+++ b/rust/ai-research/replay-recorder-v4/src/consumer.rs
@@ -1234,7 +1234,6 @@ mod tests {
font_urls: Vec::new(),
url_limit: None,
edits: Vec::new(),
- declined: Vec::new(),
unplaced: false,
visited_urls: Vec::new(),
captured_at_ms: None,
diff --git a/rust/ai-research/replay-recorder-v4/src/splice.rs b/rust/ai-research/replay-recorder-v4/src/splice.rs
index ae2983572b70..9ae4ed40273a 100644
--- a/rust/ai-research/replay-recorder-v4/src/splice.rs
+++ b/rust/ai-research/replay-recorder-v4/src/splice.rs
@@ -1,27 +1,26 @@
//! The writer's one pass over an event. It copies the event, and where the inspector read each slot, it writes the
-//! slot's references and markers, so it searches the event for no URL. A URL the policy refuses must leave the block
-//! wherever it is, so the writer also overwrites each copy of one, which only a block that holds one pays for.
+//! slot's references, decline markers, or URLs without their credentials, so it searches the event for no URL.
-use std::borrow::Cow;
-use std::collections::{BTreeMap, HashMap, VecDeque};
+use std::collections::{BTreeMap, HashMap};
-use aho_corasick::{AhoCorasick, AhoCorasickKind, MatchKind};
use metrics::counter;
use replay_core_v4::bytescan::{unescape, write_json_string, Span};
-use replay_core_v4::inspect::css::{escape_in_identifier, write_css_edit, CssEdit};
-use replay_core_v4::inspect::dom::{for_each_srcset_candidate, DeclinedUrl, RemoteKind};
-use replay_core_v4::inspect::edits::{Edit, EditSource, EventEdits, Slot};
+use replay_core_v4::inspect::css::{in_css_string, write_css_edit, CssEdit};
+use replay_core_v4::inspect::dom::{for_each_srcset_candidate, RemoteKind};
+use replay_core_v4::inspect::edits::{Edit, EditSource, EventEdits, LinkForm, Slot};
use replay_core_v4::inspect::srcset::{shown_candidate, SrcsetSlot};
-use replay_core_v4::refs::{declined_url_ref, UrlAsset};
+use replay_core_v4::refs::{declined_url_ref, unnamed_url, UrlAsset};
use replay_core_v4::seal::image_url_key;
-use replay_core_v4::url_policy::is_remote;
+use replay_core_v4::url_policy::{
+ checked_url, is_remote, redaction, Decline, Redaction, MAX_URL_LEN,
+};
use replay_core_v4::v4_image_refs::{bundled_font_ref, ImageDecline, IMAGE_DECLINED_PREFIX};
use crate::urls::url_reference;
fn asset_of(kind: RemoteKind) -> UrlAsset {
match kind {
- RemoteKind::Image => UrlAsset::Image,
+ RemoteKind::Image | RemoteKind::Link => UrlAsset::Image,
RemoteKind::Font | RemoteKind::BundledFont(_) => UrlAsset::Font,
}
}
@@ -31,13 +30,20 @@ fn kind_label(kind: RemoteKind) -> &'static str {
RemoteKind::Image => "image",
RemoteKind::Font => "font",
RemoteKind::BundledFont(_) => "bundled_font",
+ RemoteKind::Link => "link",
}
}
-fn json_string_contents(text: &str) -> Vec<u8> {
- let mut quoted = Vec::with_capacity(text.len() + 2);
- write_json_string(text, &mut quoted);
- quoted[1..quoted.len() - 1].to_vec()
+/// `url` without its credentials, or as it is when it holds none.
+fn kept(url: &str) -> String {
+ unnamed_url(url, UrlAsset::Image).unwrap_or_else(|| url.to_owned())
+}
+
+fn push_kept(value: &mut String, url: &str) {
+ match unnamed_url(url, UrlAsset::Image) {
+ Some(kept) => value.push_str(&kept),
+ None => value.push_str(url),
+ }
}
#[derive(Clone)]
@@ -85,111 +91,6 @@ impl SpliceCounters {
}
}
-/// Refused URLs that hold one another can end many matches at each byte, so a block past this many is dropped.
-const MAX_BLOCK_DECLINED_MATCHES: usize = 1_000_000;
-
-/// Overwrites every copy of the block's refused URLs, as a JSON string holds it and as a CSS selector escapes it.
-pub struct DeclinedCopies {
- matcher: AhoCorasick,
- markers: Vec<Vec<u8>>,
- longest: usize,
- matches_left: usize,
-}
-
-#[derive(Clone, Copy)]
-struct Overwrite {
- start: usize,
- end: usize,
- marker: usize,
-}
-
-impl DeclinedCopies {
- pub fn new(declined: &[DeclinedUrl]) -> Option<Self> {
- let mut patterns = Vec::new();
- let mut markers = Vec::new();
- for DeclinedUrl { url, kind, decline } in declined {
- let marker = declined_url_ref(asset_of(*kind), *decline);
- // A selector copy takes an escaped marker, so that its colons do not start pseudo-classes.
- for (form, replacement) in [
- (url.clone(), marker.clone()),
- (escape_in_identifier(url), escape_in_identifier(&marker)),
- ] {
- patterns.push(json_string_contents(&form));
- markers.push(json_string_contents(&replacement));
- }
- }
- let matcher = AhoCorasick::builder()
- .kind(Some(AhoCorasickKind::ContiguousNFA))
- .match_kind(MatchKind::Standard)
- .build(&patterns)
- .ok()?;
- let longest = patterns.iter().map(Vec::len).max()?;
- Some(Self {
- matcher,
- markers,
- longest,
- matches_left: MAX_BLOCK_DECLINED_MATCHES,
- })
- }
-
- /// Copies that overlap share one span, so that a refused URL that starts inside another one is overwritten too.
- pub fn apply<'b>(&mut self, bytes: &'b [u8], overwritten: &mut u64) -> Option<Cow<'b, [u8]>> {
- let mut out = None;
- let mut copied_to = 0;
- let mut pending: VecDeque<Overwrite> = VecDeque::new();
- // Matches come in order of their end, so a later match cannot reach a span that ends `longest` bytes before.
- for found in self.matcher.find_overlapping_iter(bytes) {
- self.matches_left = self.matches_left.checked_sub(1)?;
- let mut span = Overwrite {
- start: found.start(),
- end: found.end(),
- marker: found.pattern().as_usize(),
- };
- while let Some(last) = pending.back().copied().filter(|last| last.end > span.start) {
- pending.pop_back();
- if last.start <= span.start {
- span.start = last.start;
- span.marker = last.marker;
- }
- }
- pending.push_back(span);
- while let Some(first) = pending
- .front()
- .copied()
- .filter(|first| first.end + self.longest <= span.end)
- {
- pending.pop_front();
- self.write(bytes, first, &mut out, &mut copied_to);
- *overwritten += 1;
- }
- }
- for span in pending {
- self.write(bytes, span, &mut out, &mut copied_to);
- *overwritten += 1;
- }
- Some(match out {
- Some(mut out) => {
- out.extend_from_slice(&bytes[copied_to..]);
- Cow::Owned(out)
- }
- None => Cow::Borrowed(bytes),
- })
- }
-
- fn write(
- &self,
- bytes: &[u8],
- span: Overwrite,
- out: &mut Option<Vec<u8>>,
- copied_to: &mut usize,
- ) {
- let out = out.get_or_insert_with(|| Vec::with_capacity(bytes.len()));
- out.extend_from_slice(&bytes[*copied_to..span.start]);
- out.extend_from_slice(&self.markers[span.marker]);
- *copied_to = span.end;
- }
-}
-
pub struct Splicer<'b> {
edits: &'b [EventEdits],
next: usize,
@@ -274,8 +175,16 @@ impl<'b> Splicer<'b> {
text: reference,
decline: None,
},
+ // The policy reads a credential before the host, so what is left of a credential URL must pass the policy on its own.
(_, Err(decline)) => Named {
- text: declined_url_ref(asset, decline),
+ text: match redaction(url) {
+ Redaction::Redacted(kept)
+ if decline == Decline::Credential && checked_url(&kept).is_ok() =>
+ {
+ kept
+ }
+ _ => declined_url_ref(asset, decline),
+ },
decline: Some(decline.label()),
},
};
@@ -289,7 +198,10 @@ impl<'b> Splicer<'b> {
.url_rewrites
.entry((kind_label(kind), outcome, named.decline.unwrap_or("none")))
.or_default() += 1;
- self.named.insert((url.to_owned(), kind), named.clone());
+ // A URL past the fetcher's limit becomes a marker, and caching it would hold the whole URL until the block ends.
+ if url.len() <= MAX_URL_LEN {
+ self.named.insert((url.to_owned(), kind), named.clone());
+ }
named
}
@@ -297,11 +209,39 @@ impl<'b> Splicer<'b> {
match slot {
Slot::Url => {
let url = text.trim();
- is_remote(url).then(|| self.named(url, RemoteKind::Image).text)
+ if is_remote(url) {
+ Some(self.named(url, RemoteKind::Image).text)
+ } else {
+ self.redacted(unnamed_url(text, UrlAsset::Image))
+ }
}
Slot::Srcset(slot) => self.rebuilt_srcset(text, *slot),
Slot::Css(edit) => Some(self.written_css(text, edit)),
+ Slot::Link(LinkForm::Url) => self.redacted(unnamed_url(text, UrlAsset::Image)),
+ Slot::Link(LinkForm::Srcset) => self.redacted(Some(links_without_credentials(text))),
+ Slot::Link(LinkForm::Urls) => {
+ let mut value = String::with_capacity(text.len());
+ for url in text.split_ascii_whitespace() {
+ if !value.is_empty() {
+ value.push(' ');
+ }
+ push_kept(&mut value, url);
+ }
+ self.redacted(Some(value))
+ }
+ }
+ }
+
+ /// Counts a URL that keeps its place without its credentials and gets no reference.
+ fn redacted(&mut self, redacted: Option<String>) -> Option<String> {
+ if redacted.is_some() {
+ *self
+ .counters
+ .url_rewrites
+ .entry((kind_label(RemoteKind::Link), "redacted", "credential"))
+ .or_default() += 1;
}
+ redacted
}
fn written_css(&mut self, text: &str, edit: &CssEdit) -> String {
@@ -319,7 +259,13 @@ impl<'b> Splicer<'b> {
*self.counters.css_rewrites.entry(kind).or_default() += 1;
}
}
- write_css_edit(text, edit, &mut |url| self.named(&url.url, url.kind).text)
+ write_css_edit(text, edit, &mut |url| {
+ let written = match url.kind {
+ RemoteKind::Link => self.redacted(Some(kept(&url.url))).unwrap_or_default(),
+ kind => self.named(&url.url, kind).text,
+ };
+ in_css_string(&written).into_owned()
+ })
}
/// The srcset with only the candidate it shows, a remote one as its reference. `None` when it has nothing to change.
@@ -337,8 +283,12 @@ impl<'b> Splicer<'b> {
.then(|| self.named(url, RemoteKind::Image))
.filter(|named| named.decline.is_none())
.map(|named| named.text)
+ } else if is_shown && !url.starts_with(IMAGE_DECLINED_PREFIX) {
+ let written = kept(url);
+ remote |= written != url;
+ Some(written)
} else {
- (is_shown && !url.starts_with(IMAGE_DECLINED_PREFIX)).then(|| url.to_owned())
+ None
};
match written {
Some(written) => {
@@ -365,3 +315,18 @@ impl<'b> Splicer<'b> {
(removed || remote).then_some(value)
}
}
+
+fn links_without_credentials(srcset: &str) -> String {
+ let mut value = String::with_capacity(srcset.len());
+ for_each_srcset_candidate(srcset, |url, descriptors| {
+ if !value.is_empty() {
+ value.push_str(", ");
+ }
+ push_kept(&mut value, url);
+ if !descriptors.is_empty() {
+ value.push(' ');
+ value.push_str(descriptors);
+ }
+ });
+ value
+}
diff --git a/rust/ai-research/replay-recorder-v4/src/urls.rs b/rust/ai-research/replay-recorder-v4/src/urls.rs
index 4709a8438072..d40d224cf345 100644
--- a/rust/ai-research/replay-recorder-v4/src/urls.rs
+++ b/rust/ai-research/replay-recorder-v4/src/urls.rs
@@ -268,7 +268,6 @@ mod tests {
font_urls: Vec::new(),
url_limit: None,
edits: Vec::new(),
- declined: Vec::new(),
unplaced: false,
visited_urls: Vec::new(),
captured_at_ms: None,
diff --git a/rust/ai-research/replay-recorder-v4/src/write.rs b/rust/ai-research/replay-recorder-v4/src/write.rs
index 3f537d82a03e..53d2cb4aa573 100644
--- a/rust/ai-research/replay-recorder-v4/src/write.rs
+++ b/rust/ai-research/replay-recorder-v4/src/write.rs
@@ -3,7 +3,6 @@
//! The order matters for a crash between writes. A pack without its blocks is unreferenced, which is harmless. A block
//! whose images were never stored would hold references that resolve to nothing. The rows go out only after both.
-use std::borrow::Cow;
use std::collections::{BTreeMap, HashMap, HashSet};
use std::io::{self, Write};
use std::sync::Arc;
@@ -19,7 +18,7 @@ use replay_core_v4::inspect::stats::BlockStats;
use replay_core_v4::inspect::stats::MAX_URL_FINGERPRINTS;
use replay_core_v4::keys::batch::{FlushKeys, SessionId, SessionKey as Key};
use replay_core_v4::refs::url_fingerprint;
-use replay_core_v4::scan::cv::{extract_event_images, BeforeImages, Expansion};
+use replay_core_v4::scan::cv::{extract_event_images, Expansion};
use replay_core_v4::scan::images::{Image, ImageLimits, ImageSink};
use replay_core_v4::seal::{image_content_key, url_fingerprint_key};
@@ -27,7 +26,7 @@ use tokio::sync::Semaphore;
use crate::batch::{Block, SessionKey};
use crate::index::{BlockIndexRow, ImageIndexRow};
-use crate::splice::{DeclinedCopies, Splicer};
+use crate::splice::Splicer;
use crate::store::{BlobStore, StoreError};
/// Brotli settings for blocks, the same as the v3 lane, which uses the same C encoder. The lane keeps blocks for months
@@ -68,8 +67,6 @@ pub struct WriterCounts {
pub max_line_bytes: u64,
/// Why the block must not be stored, as the outcome of its drop: the inspector could not read some of it.
pub dropped: Option<&'static str>,
- /// Copies of a refused URL that the writer overwrote outside its slot.
- pub overwritten: u64,
}
/// A block compressed once for all the attempts of its flush, with the images it moved out.
@@ -119,8 +116,6 @@ fn prepare_block(
) -> io::Result<(Vec<Image>, WriterCounts)> {
let content_key = image_content_key(team_key);
let mut splicer = Splicer::new(&block.edits, block.session.team_id, key_month, team_key);
- let mut declined = DeclinedCopies::new(&block.declined);
- let mut too_many_matches = false;
let mut images = Vec::new();
let mut kept = HashSet::new();
let mut counts = WriterCounts::default();
@@ -138,39 +133,16 @@ fn prepare_block(
ImageLimits::default(),
);
for event in &message.events {
- let mut overwrite =
- |bytes: &[u8]| match declined.as_mut()?.apply(bytes, &mut counts.overwritten) {
- Some(Cow::Owned(overwritten)) => Some(overwritten),
- Some(Cow::Borrowed(_)) => None,
- None => {
- too_many_matches = true;
- None
- }
- };
- let before_images: Option<&mut BeforeImages<'_>> = Some(&mut overwrite);
let extracted = match splicer.edits_of(event_index) {
Some(edits) => {
let mut copy = |source, bytes: &[u8], range, out: &mut Vec<u8>| {
splicer.copy(edits, source, bytes, range, out)
};
- extract_event_images(
- &message.inner,
- event,
- &mut sink,
- Some(&mut copy),
- before_images,
- )
+ extract_event_images(&message.inner, event, &mut sink, Some(&mut copy))
}
- None => extract_event_images(&message.inner, event, &mut sink, None, before_images),
+ None => extract_event_images(&message.inner, event, &mut sink, None),
};
event_index += 1;
- if too_many_matches {
- let counts = WriterCounts {
- dropped: Some("too_many_declined_copies"),
- ..WriterCounts::default()
- };
- return Ok((Vec::new(), counts));
- }
let event_bytes = extracted
.as_deref()
.unwrap_or(&message.inner[event.span.0..event.span.1]);
@@ -300,8 +272,6 @@ pub fn compress_block(
};
let (lines, images, counts) =
write_block(block, team_key, key_month, open).map_err(seal_error)?;
- counter!("ai_research_v4_recorder_overwritten_declined_copies_total")
- .increment(counts.overwritten);
let encoder = lines.into_inner().map_err(|e| seal_error(e.into_error()))?;
let body = encoder
.into_inner()
@@ -784,7 +754,6 @@ mod tests {
font_urls: Vec::new(),
url_limit: None,
edits: Vec::new(),
- declined: Vec::new(),
unplaced: false,
visited_urls: Vec::new(),
captured_at_ms: None,
@@ -832,7 +801,6 @@ mod tests {
inspected.font_urls = found.font_urls;
inspected.url_limit = found.url_limit;
inspected.edits = found.edits;
- inspected.declined = found.declined;
inspected.unplaced = found.unplaced;
inspected
}
@@ -859,43 +827,102 @@ mod tests {
.1;
assert!(
lines.contains(&format!(
- r#""src":"imageurl:v4:declined:credential","srcset":"{reference} 2x""#
+ r#""src":"https://bucket.s3.amazonaws.com/avatar.png","srcset":"{reference} 2x""#
)),
"{lines}"
);
- // A URL past the length the fetcher takes is named and declined.
- let long = |len: usize| {
- let url = format!("https://cdn.example.com/{}", "a".repeat(len - 24));
- json!([{"type": 3, "timestamp": 1, "data": {"source": 0, "adds": [], "removes": [], "texts": [],
- "attributes": [{"id": 99, "attributes": {"src": url}}]}}])
- .to_string()
- };
- // A URL too long to fetch is declined, and the block is kept.
- let long_lines = stored_lines(&long(16 * 1024));
- assert!(
- long_lines.contains("imageurl:v4:declined:too_long"),
- "{long_lines}"
- );
+ // A URL refused for a reason other than a credential keeps none of its text. Neither does a credential URL whose
+ // rest the policy refuses, nor a link that is too long to read with a credential in it.
+ let long = format!("https://cdn.example.com/{}", "a".repeat(16 * 1024 - 24));
+ let long_link = format!("{long}?token=abc123");
+ for (attributes, marker) in [
+ (
+ json!({"src": long}),
+ r#""src":"imageurl:v4:declined:too_long""#,
+ ),
+ (
+ json!({"src": "https://cdn.example.com:99999/a.png?token=abc123"}),
+ r#""src":"imageurl:v4:declined:not_absolute""#,
+ ),
+ (
+ json!({"data-src": long_link}),
+ r#""data-src":"imageurl:v4:declined:too_long""#,
+ ),
+ (
+ json!({"src": "https://user:[email protected]/a.png"}),
+ r#""src":"imageurl:v4:declined:credential""#,
+ ),
+ ] {
+ let lines = stored_lines(
+ &json!([{"type": 3, "timestamp": 1, "data": {"source": 0, "adds": [], "removes": [], "texts": [],
+ "attributes": [{"id": 99, "attributes": attributes}]}}])
+ .to_string(),
+ );
+ assert!(lines.contains(marker), "{marker} in {lines}");
+ }
// Keys written with escapes hide no URL.
let escaped = events
.replace(r#""src""#, r#""\u0073rc""#)
.replace(r#""srcset""#, r#""src\u0073et""#);
assert_eq!(stored_lines(&escaped), lines);
// A browser reads backslashes after the scheme as slashes, so such a URL is replaced like its `https://` form.
+ // The URL parser drops tabs and line breaks anywhere, and C0 controls at either end, so these load from the host too.
for backslashed in [
r"https:\\user:[email protected]/a.png",
r"https:/\user:[email protected]/a.png",
"https:user:[email protected]/a.png",
+ "ht\ttps://user:[email protected]/a.png",
+ "/\n/user:[email protected]/a.png",
+ "\u{1}https://user:[email protected]/a.png",
+ "\u{0}\u{1f}//user:[email protected]/a.png",
] {
let events = json!([{"type": 3, "timestamp": 1, "data": {"source": 0, "adds": [], "removes": [], "texts": [],
"attributes": [{"id": 99, "attributes": {"src": backslashed}}]}}])
.to_string();
let lines = stored_lines(&events);
assert!(
- lines.contains("imageurl:v4:declined:credential") && !lines.contains("secret"),
+ lines.contains("cdn.example.com/a.png") && !lines.contains("secret"),
"{lines}"
);
}
+ // Such a node's text may not be CSS, so only a URL with a credential in it changes. An accepted URL gets no
+ // reference, and an `image-set()` keeps each candidate.
+ let accepted = ".a{background:url(https://cdn.example.com/a.png)}";
+ let set = ".a{background:image-set(url(https://cdn.example.com/a.png) 1x, url(https://cdn.example.com/b.png) 2x)}";
+ let events = json!([{"type": 3, "timestamp": 1, "data": {"source": 0, "removes": [], "attributes": [],
+ "texts": [{"id": 99, "value": format!(".a{{background:url({signed})}}")}, {"id": 96, "value": accepted},
+ {"id": 95, "value": format!("@import \"{signed}\";")}, {"id": 94, "value": set},
+ {"id": 93, "value": format!(".a{{background:image-set(\"{signed}\" 1x)}}")}],
+ "adds": [{"parentId": 98, "node": {"type": 2, "tagName": "source", "attributes": {"srcset": format!("{signed} 1x")}, "childNodes": [], "id": 97}}]}}])
+ .to_string();
+ let lines = stored_lines(&events);
+ assert!(
+ lines.contains(accepted) && lines.contains(set) && !lines.contains("X-Amz-Signature"),
+ "{lines}"
+ );
+ // The URL parser keeps a backslash in a query, so the rewrite escapes it, and the CSS string still ends.
+ let style = r#"background-image:url("https://cdn.example.com/a.png?token=abc123&q=\\")"#;
+ let lines = stored_lines(
+ &json!([{"type": 3, "timestamp": 1, "data": {"source": 0, "adds": [], "removes": [], "texts": [],
+ "attributes": [{"id": 99, "attributes": {"style": style}}]}}])
+ .to_string(),
+ );
+ let escaped =
+ json!(r#"background-image:url("https://cdn.example.com/a.png?q=\\")"#).to_string();
+ assert!(lines.contains(&escaped), "{escaped} in {lines}");
+ // A property set on its own, by a style diff or a style declaration event, loses its credential too.
+ let events = json!([
+ {"type": 3, "timestamp": 1, "data": {"source": 0, "adds": [], "removes": [], "texts": [],
+ "attributes": [{"id": 99, "attributes": {"style": {"filter": format!("url({signed})")}}}]}},
+ {"type": 3, "timestamp": 2, "data": {"source": 13, "id": 3, "set": {"property": "filter", "value": format!("url({signed})")}}},
+ ])
+ .to_string();
+ let lines = stored_lines(&events);
+ assert!(
+ lines.matches("bucket.s3.amazonaws.com/avatar.png").count() == 2
+ && !lines.contains("X-Amz-Signature"),
+ "{lines}"
+ );
// The player loads the last of a repeated attribute, so the inspector names that one.
for events in [
format!(
@@ -907,75 +934,93 @@ mod tests {
] {
let lines = stored_lines(&events);
assert!(
- lines.contains("imageurl:v4:declined:credential")
+ lines.contains(r#""src":"https://bucket.s3.amazonaws.com/avatar.png""#)
&& !lines.contains("X-Amz-Signature"),
"{lines}"
);
}
- // A declined URL that a longer URL also holds is overwritten there too, and the block is kept, also for a bundled font, which names no URL, and for an escaped font URL, as the scan decodes it, of a bundled family too.
- let bundled_font = json!({"parentId": 1, "node": {"type": 2, "tagName": "style",
- "attributes": {"_cssText": format!("@font-face{{font-family:Roboto;src:url({signed})}}")}, "childNodes": [], "id": 5}});
- let escaped_font = |family: &str, host: &str| {
+ // Each slot loses its credential, also an escaped font URL of a bundled family, which names no URL. v4 stores
+ // pages unscrubbed, so a copy of the URL outside a URL slot, here in `alt`, keeps its text.
+ let font = |family: &str, host: &str| {
json!({"parentId": 1, "node": {"type": 2, "tagName": "style",
"attributes": {"_cssText": format!("@font-face{{font-family:{family};src:url(https://{host}/f\\2e woff2?X-Amz-Signature=abc123)}}")}, "childNodes": [], "id": 6}})
};
- let overlapping =
- "https://a.example/?X-Amz-Signature=x&next=https://b.example/?X-Amz-Signature=abc123";
- // Image extraction changes the copy's inline image, so the writer overwrites the copy before it.
- let with_inline_image = "https://cdn.example.com/a?X-Amz-Signature=abc123&x=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==";
- let decoded_font_copy =
- |host: &str| json!({"alt": format!("https://{host}/f.woff2?X-Amz-Signature=abc123")});
- for (adds, attributes) in [
- (
- json!([]),
- json!({"src": signed, "alt": format!("{signed}&size=2")}),
- ),
- (
- json!([bundled_font]),
- json!({"alt": format!("{signed}&size=2")}),
- ),
- (
- json!([escaped_font("Brand", "bucket.s3.amazonaws.com")]),
- decoded_font_copy("bucket.s3.amazonaws.com"),
- ),
- // The URL parser lowercases the host, and a copy keeps the spelling the page wrote.
- (
- json!([escaped_font("Brand", "CDN.example.com")]),
- decoded_font_copy("CDN.example.com"),
- ),
- // A text copy of the rule keeps its CSS escapes.
- (
- json!([escaped_font("Brand", "bucket.s3.amazonaws.com")]),
- json!({"alt": "url(https://bucket.s3.amazonaws.com/f\\2e woff2?X-Amz-Signature=abc123)"}),
- ),
- (
- json!([]),
- json!({"src": with_inline_image, "alt": with_inline_image}),
- ),
- (
- json!([escaped_font("Roboto", "bucket.s3.amazonaws.com")]),
- decoded_font_copy("bucket.s3.amazonaws.com"),
- ),
- (
- json!([{"parentId": 1, "node": {"type": 2, "tagName": "div", "attributes": {"style": "background-image:url('\\2f \\2f cdn.example.com/a.png?X-Amz-Signature=abc123')"}, "childNodes": [], "id": 9}}]),
- json!({"alt": "//cdn.example.com/a.png?X-Amz-Signature=abc123"}),
- ),
- (
- json!([
- {"parentId": 1, "node": {"type": 2, "tagName": "img", "attributes": {"src": "https://a.example/?X-Amz-Signature=x&next=https://b.example/"}, "childNodes": [], "id": 7}},
- {"parentId": 1, "node": {"type": 2, "tagName": "img", "attributes": {"src": "https://b.example/?X-Amz-Signature=abc123"}, "childNodes": [], "id": 8}},
- ]),
- json!({"alt": format!("{overlapping} {} {overlapping}", "x".repeat(200))}),
- ),
+ let style = |style: &str| json!({"parentId": 1, "node": {"type": 2, "tagName": "div", "attributes": {"style": style}, "childNodes": [], "id": 9}});
+ let link = |tag: &str, attributes: Value| json!({"parentId": 1, "node": {"type": 2, "tagName": tag, "attributes": attributes, "childNodes": [], "id": 10}});
+ for adds in [
+ json!([link("img", json!({"src": signed}))]),
+ json!([font("Brand", "bucket.s3.amazonaws.com")]),
+ json!([font("Roboto", "bucket.s3.amazonaws.com")]),
+ json!([style(
+ "background-image:url('\\2f \\2f cdn.example.com/a.png?X-Amz-Signature=abc123')"
+ )]),
+ json!([style(
+ "background-image:url('\\1 //cdn.example.com/a.png?X-Amz-Signature=abc123')"
+ )]),
+ // A browser loads a URL with a tab in it, or in another letter case, from the same place.
+ json!([link(
+ "img",
+ json!({"src": "HTTPS://Bucket.s3.amazonaws.com/x\t.png?X-Amz-Signature=abc123"})
+ )]),
+ json!([link("a", json!({"href": signed}))]),
+ json!([link(
+ "form",
+ json!({"action": "/session/reset?token=abc123"})
+ )]),
+ json!([link(
+ "img",
+ json!({"data-src": signed, "data-srcset": format!("{signed} 2x, /b.png 1x")})
+ )]),
+ json!([link(
+ "img",
+ json!({"src": "/api/avatar.png?X-Amz-Signature=abc123"})
+ )]),
+ json!([style(
+ "cursor:pointer;filter:url(https://cdn.example.com/f.svg?X-Amz-Signature=abc123#f)"
+ )]),
+ json!([link(
+ "rect",
+ json!({"fill": "url(https://cdn.example.com/p.svg?X-Amz-Signature=abc123#g)"})
+ )]),
+ // On a page of its scheme, a browser reads `https:` with no slashes as a path on the page's host.
+ json!([link("a", json!({"href": "https:bcdn_token=abc123/a.png"}))]),
+ // This window has no page URL, so the set's root-relative candidate cannot load, and keeps its place.
+ json!([style(
+ "background-image:image-set(url(/a.png?X-Amz-Signature=abc123) 1x)"
+ )]),
+ json!([{"parentId": 1, "node": {"type": 2, "tagName": "style", "attributes": {"_cssText": "@import url(\"https://cdn.example.com/a.css?X-Amz-Signature=abc123\");@import \"/b.css?token=abc123\";@import \"https://cdn.example.com/c\\2e css?token=abc123\";"}, "childNodes": [], "id": 11}}]),
+ json!([link(
+ "a",
+ json!({"ping": format!("{signed} https://cdn.example.com/b")})
+ )]),
+ json!([link(
+ "img",
+ json!({"attributionsrc": format!("https://cdn.example.com/b {signed}")})
+ )]),
+ // The policy reads a host without its trailing dot, and a path once it is decoded.
+ json!([link(
+ "img",
+ json!({"src": "https://objectstorage.us-ashburn-1.oraclecloud.com./p/X-Amz-Signature/n/ns/b/bucket/o/a.png"})
+ )]),
+ json!([link(
+ "img",
+ json!({"src": "https://cdn.example.com/image/upload%2Fs--X-Amz-Signature--%2Fv1/a.png"})
+ )]),
] {
+ let copy = json!(signed).to_string();
let events = json!([{"type": 3, "timestamp": 1, "data": {"source": 0, "adds": adds, "removes": [], "texts": [],
- "attributes": [{"id": 99, "attributes": attributes}]}}])
+ "attributes": [{"id": 99, "attributes": {"alt": signed}}]}}])
.to_string();
let lines = stored_lines(&events);
+ assert_eq!(lines.matches(&copy).count(), 1, "{lines}");
assert!(
- !lines.is_empty() && !lines.contains("X-Amz-Signature"),
+ !lines.replace(&copy, "").contains("X-Amz-Signature")
+ && !lines.contains("token=abc123"),
"{lines}"
);
+ for line in lines.lines() {
+ serde_json::from_str::<Value>(line).unwrap_or_else(|e| panic!("{e}: {line}"));
+ }
}
}
@@ -1004,18 +1049,24 @@ mod tests {
link(),
json!({"type": 2, "tagName": "style", "attributes": {"_cssText": split_face}, "childNodes": [], "id": 4}),
];
- for (nodes, named, references, declined) in [
- (images, "imageurl:v4:7:2026-10:", 10_001, 0),
- (fonts, "fonturl:v4:7:2026-10:", 4_098, 0),
- (escaped_fonts, "fonturl:v4:7:2026-10:", 10_001, 1),
+ let split_kept = r#"src:url(\"https://fonts.example.com/split.woff2\")"#;
+ for (nodes, named, references, kept) in [
+ (images, "imageurl:v4:7:2026-10:", 10_001, None),
+ (fonts, "fonturl:v4:7:2026-10:", 4_098, None),
+ (
+ escaped_fonts,
+ "fonturl:v4:7:2026-10:",
+ 10_001,
+ Some(split_kept),
+ ),
] {
let events = json!([{"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": nodes}}}]).to_string();
- let lines = stored_lines(&events);
+ let mut lines = stored_lines(&events);
assert_eq!(lines.matches(named).count(), references);
- assert_eq!(
- lines.matches("fonturl:v4:declined:credential").count(),
- declined
- );
+ if let Some(kept) = kept {
+ assert!(lines.contains(kept), "{lines}");
+ lines = lines.replace(kept, "");
+ }
for raw in [
"https://cdn.",
"https://fonts.",
@@ -1048,35 +1099,40 @@ mod tests {
let (_, counts) = written(block_of("kept", &events));
assert_eq!(counts.dropped, Some("too_many_nested_blocks"));
}
- let refused: Vec<_> = (0..2_000)
- .map(|i| json!({"type": 2, "tagName": "img", "attributes": {"src": format!("https://cdn.example.com/{i}/{}.png?token=x", "a".repeat(600))}, "childNodes": [], "id": i + 2}))
- .collect();
- let events = json!([{"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": refused}}}]).to_string();
- let (_, counts) = written(block_of("kept", &events));
- assert_eq!(counts.dropped, Some("too_many_declined_urls"));
- // Each copy of the longest of these refused URLs holds a match of every one of them.
- let nested: Vec<_> = (1..=1_000)
- .map(|i| json!({"type": 2, "tagName": "img", "attributes": {"src": format!("https://10.0.0.7/{}", "a".repeat(i))}, "childNodes": [], "id": i + 1}))
- .collect();
- let copies = format!("https://10.0.0.7/{} ", "a".repeat(1_000)).repeat(1_001);
- let events = json!([
- {"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": nested}}},
- {"type": 5, "timestamp": 2, "data": {"tag": "copies", "payload": {"text": copies}}},
- ])
- .to_string();
- let (lines, counts) = written(block_of("kept", &events));
- assert_eq!(
- (lines.as_str(), counts.dropped),
- ("", Some("too_many_declined_copies"))
- );
+ // posthog-js compresses these, so the message read does not see how deep they nest.
+ let cv = |value: Value| -> String {
+ replay_core_v4::compression::gzip(value.to_string().as_bytes())
+ .unwrap()
+ .iter()
+ .map(|&byte| char::from(byte))
+ .collect()
+ };
+ let img = json!({"type": 2, "tagName": "img", "attributes": {"src": "https://cdn.example.com/a.png?token=secret"}, "childNodes": [], "id": 2});
+ let deep_node = (0..600).fold(img, |node, id| {
+ json!({"type": 2, "tagName": "div", "attributes": {}, "childNodes": [node], "id": id + 3})
+ });
+ let deep_index = (0..1_100).fold(json!(0), |index, _| json!([index]));
+ for event in [
+ json!({"type": 2, "timestamp": 1, "cv": "2024-10", "data": cv(json!({"node": {"type": 0, "id": 1, "childNodes": [deep_node]}}))}),
+ json!({"type": 3, "timestamp": 1, "cv": "2024-10", "data": {"source": 0, "adds": cv(json!([{"parentId": 1, "node": deep_node}])), "removes": "", "texts": "", "attributes": ""}}),
+ json!({"type": 3, "timestamp": 1, "cv": "2024-10", "data": {"source": 8, "id": 3, "adds": cv(json!([{"rule": ".a{background:url(https://cdn.example.com/a.png?token=secret)}", "index": deep_index}]))}}),
+ ] {
+ let (_, counts) = written(block_of("kept", &json!([event]).to_string()));
+ assert_eq!(counts.dropped, Some("too_deep_json"), "{event:.80}");
+ }
// cssparser reads a NUL in a string as U+FFFD, so the decoded URL is not in its token, only in the comment.
let nul_url = "url(\"https://cdn.example.com/a.png?X-Amz-Signature=abc123#\u{0}\"/*https://cdn.example.com/a.png?X-Amz-Signature=abc123#\u{FFFD}*/)";
- // A set keeps its candidate as the page wrote it, so it cannot keep an escaped one.
+ // A set keeps its candidate as the page wrote it, so it cannot keep an escaped one, or one that another image function holds.
for style in [
format!("background-image:{nul_url}"),
format!("background-image:image-set({nul_url} 1x)"),
"background-image:image-set(url('\\2f \\2f cdn.example.com/a.png?token=secret') 1x)"
.to_owned(),
+ "background-image:image-set(url('\\2f a.png?token=secret') 1x)".to_owned(),
+ "background-image:image-set(-webkit-cross-fade(url(https://cdn.example.com/a.png?token=secret), url(https://cdn.example.com/b.png), 50%) 1x)"
+ .to_owned(),
+ // An escaped remote URL that the URL parser does not read may still load in a browser that does.
+ "background-image:url('\\68 ttps://cdn.exa\\1 mple.com/a.png?token=secret')".to_owned(),
] {
let node = json!({"type": 2, "tagName": "div", "attributes": {"style": style}, "childNodes": [], "id": 2});
let events = json!([{"type": 2, "timestamp": 1, "data": {"node": {"type": 0, "id": 1, "childNodes": [node]}}}]).to_string();
@@ -1132,7 +1188,9 @@ mod tests {
r#"image-set(url({}) 2x)"#,
reference("https://cdn.example.com/2x.png")
),
- r#"url(\"imageurl:v4:declined:credential\")"#.to_owned(),
+ r#".c{background:url(\"https://app.example.com/c.png\")}"#.to_owned(),
+ r".s-\\[url\\(https\\:\\/\\/cdn\\.example\\.com\\/s\\.png\\?X-Amz-Signature\\=abc123\\)\\]{background-image:url(https://cdn.example.com/s.png)}".to_owned(),
+ r".\\/\\/cdn\\.example\\.com\\/p\\.png\\?X-Amz-Signature\\=abc123{background:url(//cdn.example.com/p.png)}".to_owned(),
format!(
r#"src:url(\"{}\")"#,
crate::urls::url_reference(
@@ -1178,15 +1236,12 @@ mod tests {
] {
assert!(lines.contains(kept), "{kept} in {lines}");
}
- assert!(
- lines.contains(r".s-\\[url\\(imageurl\\:v4\\:declined\\:credential\\)\\]"),
- "{lines}"
- );
+ // A selector is no URL slot, so it keeps a refused URL as the page wrote it.
+ assert_eq!(lines.matches("X-Amz-Signature").count(), 2, "{lines}");
for gone in [
":url(/img/a.png)",
"1x.png",
"1x,",
- "Signature",
"/fonts/",
"fonts.example.com",
] {
@@ -1234,6 +1289,23 @@ mod tests {
assert!(lines.contains(&expected), "{expected} in {lines}");
}
assert!(!lines.contains("url(/a.png)"), "{lines}");
+ // The URL parser strips a C0 control at either end, so these load from the page's origin too.
+ for css in [
+ ".a{background:url(\"\u{1}/api/img?token=secret\")}",
+ ".a{background:url(\"\\1 /api/img?token=secret\")}",
+ ] {
+ let events = json!([
+ {"type": 4, "timestamp": 1, "data": {"href": "https://app.example.com/shop"}},
+ {"type": 3, "timestamp": 2, "data": {"source": 8, "id": 3, "replaceSync": css}},
+ ])
+ .to_string();
+ let lines = stored_lines(&events);
+ assert!(
+ lines.contains(r#"url(\"https://app.example.com/api/img\")"#)
+ && !lines.contains("secret"),
+ "{css}: {lines}"
+ );
+ }
}
#[test]
@@ -1390,9 +1462,8 @@ mod tests {
"{lines}"
);
- // A srcset key outside an image slot, here in a custom event, is no srcset of the DOM and stays as it is. With
- // the same value as an inspected image slot, only the slot is rebuilt, and the custom event loses only the URL
- // that the policy refuses.
+ // A srcset key outside an image slot, here in a custom event, is no srcset of the DOM and stays as it is, also
+ // with the same value as an inspected image slot.
let value = "https://cdn.example.com/a.png 1x, https://10.0.0.7/a.png 2x";
let both = format!(
r#"[{{"type":3,"timestamp":1,"data":{{"source":0,"adds":[],"removes":[],"texts":[],"attributes":[{{"id":99,"attributes":{{"srcset":"{value}"}}}}]}}}},{{"type":5,"timestamp":2,"data":{{"tag":"app","payload":{{"srcset":"{value}"}}}}}}]"#
@@ -1413,9 +1484,7 @@ mod tests {
"{dom}"
);
assert!(
- custom.contains(
- r#""srcset":"https://cdn.example.com/a.png 1x, imageurl:v4:declined:non_public_host 2x""#
- ),
+ custom.contains(&format!(r#""srcset":"{value}""#)),
"{custom}"
);
let custom = r#"[{"type":5,"timestamp":1,"data":{"tag":"app","payload":{"srcset":"https://cdn.example.com/a.png 1x, https://10.0.0.7/a.png 2x"}}}]"#;
diff --git a/rust/ai-research/replay-recorder-v4/tests/pbt_escapes.rs b/rust/ai-research/replay-recorder-v4/tests/pbt_escapes.rs
new file mode 100644
index 000000000000..4b48e18003c3
--- /dev/null
+++ b/rust/ai-research/replay-recorder-v4/tests/pbt_escapes.rs
@@ -0,0 +1,875 @@
+//! Property tests for the writer on pages that spell URLs in unusual ways: CSS escapes, quotes, comments and copies of
+//! a URL outside its slot. They search for bugs rather than guard against them, so CI does not run them: each bug they
+//! find gets an example test in `src/write.rs`. Run them with
+//! `cargo test -p replay-recorder-v4 --test pbt_escapes -- --ignored`, and set `HEGEL_TEST_CASES` for a longer search.
+
+use hegel::generators as gs;
+use hegel::TestCase;
+use replay_core_v4::inspect::session::{BlockInspection, SessionInspector};
+use replay_core_v4::inspect::stats::BlockStats;
+use replay_core_v4::refs::UrlAsset;
+use replay_core_v4::scan::message::read_message;
+use replay_core_v4::seal::image_url_key;
+use replay_core_v4::url_policy::{redaction, Redaction};
+use replay_recorder_v4::batch::{Block, Origin, SessionKey, Tp};
+use replay_recorder_v4::urls::url_reference;
+use replay_recorder_v4::write::{write_block, WriterCounts};
+use serde_json::{json, Value};
+
+/// The credential of every refused URL, in its query, userinfo, fragment or path. No other part of a page holds it.
+const SECRET: &str = "pbtsecret";
+/// The host label of every accepted URL. No other part of a generated page holds it.
+const ACCEPTED: &str = "pbt-ok";
+const TEAM_KEY: [u8; 32] = [6; 32];
+const KEY_MONTH: &str = "2026-10";
+
+const HOSTS: &[&str] = &[
+ "cdn.example.com",
+ "CDN.Example.COM",
+ "bucket.s3.amazonaws.com",
+];
+/// Where a refused URL holds its credential, after its path.
+const REFUSALS: &[&str] = &[
+ "?token=",
+ "?X-Amz-Signature=",
+ "?sig=",
+ "#access_token=",
+ ";jsessionid=",
+];
+
+/// Past the 1,024 levels of JSON that the inspector reads, as each node takes two.
+const DEEP_NODES: usize = 600;
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+enum Slot {
+ ImgSrc,
+ Srcset,
+ StyleAttribute,
+ StyleSheet,
+ FontFace(&'static str),
+ ImageSet,
+ CrossFade { in_image_set: bool },
+ StyleRule,
+ StyleChange,
+ StyleObjectChange,
+ StyleText,
+ PictureSource,
+ Href,
+ FormAction,
+ DataSrc,
+ Filter,
+ Import,
+ SvgFill,
+ AttributionSrc,
+}
+
+const SLOTS: &[Slot] = &[
+ Slot::ImgSrc,
+ Slot::Srcset,
+ Slot::StyleAttribute,
+ Slot::StyleSheet,
+ Slot::FontFace("Brand"),
+ Slot::FontFace("Roboto"),
+ Slot::ImageSet,
+ Slot::CrossFade {
+ in_image_set: false,
+ },
+ Slot::CrossFade { in_image_set: true },
+ Slot::StyleRule,
+ Slot::StyleChange,
+ Slot::StyleObjectChange,
+ Slot::StyleText,
+ Slot::PictureSource,
+ Slot::Href,
+ Slot::FormAction,
+ Slot::DataSrc,
+ Slot::Filter,
+ Slot::Import,
+ Slot::SvgFill,
+ Slot::AttributionSrc,
+];
+
+impl Slot {
+ fn is_css(self) -> bool {
+ !matches!(
+ self,
+ Slot::ImgSrc
+ | Slot::Srcset
+ | Slot::PictureSource
+ | Slot::Href
+ | Slot::FormAction
+ | Slot::DataSrc
+ | Slot::AttributionSrc
+ )
+ }
+
+ fn is_html_url(self) -> bool {
+ matches!(
+ self,
+ Slot::ImgSrc | Slot::Href | Slot::FormAction | Slot::DataSrc
+ )
+ }
+
+ /// No fetch takes the URL, so the block keeps it without its credentials, and with no reference.
+ fn is_link(self) -> bool {
+ matches!(
+ self,
+ Slot::Href
+ | Slot::FormAction
+ | Slot::DataSrc
+ | Slot::Filter
+ | Slot::Import
+ | Slot::SvgFill
+ | Slot::AttributionSrc
+ )
+ }
+
+ /// After a restart the inspector cannot tell that a style element's text is CSS, so it names no URL there.
+ fn names_an_accepted_url(self, shape: Shape) -> bool {
+ !(self.is_link() || (shape.restart && self == Slot::StyleText))
+ }
+
+ /// The bundled font's reference names no URL.
+ fn names_its_url(self) -> bool {
+ !self.is_link() && self != Slot::FontFace("Roboto")
+ }
+
+ /// A set keeps its candidate as the page wrote it, so it cannot keep an escaped one, or one that another image function holds.
+ fn drops_its_block(self, written: &str) -> bool {
+ match self {
+ Slot::ImageSet => written.contains('\\'),
+ Slot::CrossFade { in_image_set } => in_image_set,
+ _ => false,
+ }
+ }
+}
+
+/// A URL as the page writes it in its slot, and the spellings a copy of it can take elsewhere.
+struct Placed {
+ slot: Slot,
+ written: String,
+ copies: Vec<String>,
+}
+
+#[derive(Debug, Clone, Copy)]
+struct Shape {
+ /// The block holds no full snapshot, as after a restart, so the inspector does not know the nodes that its incremental snapshots name.
+ restart: bool,
+ compressed: bool,
+ deep: bool,
+}
+
+fn draw_index(tc: &TestCase, len: usize) -> usize {
+ tc.draw(gs::integers::<usize>().min_value(0).max_value(len - 1))
+}
+
+fn draw_url(tc: &TestCase, index: usize, refused: bool) -> String {
+ let path = ["a.png", "img/b.png", "f.woff2"][draw_index(tc, 3)];
+ let scheme = ["https://", "//"][draw_index(tc, 2)];
+ if !refused {
+ return format!("{scheme}{ACCEPTED}-{index}.example.com/{path}");
+ }
+ let host = HOSTS[draw_index(tc, HOSTS.len())];
+ match draw_index(tc, 10) {
+ 0 => format!(
+ "/{path}{}{SECRET}",
+ REFUSALS[draw_index(tc, REFUSALS.len())]
+ ),
+ // A scheme starts with a letter, so the URL parser reads these as relative paths on the page's host.
+ 5 => format!(
+ "{}{path}{}{SECRET}",
+ ["1:", "9a:", ":"][draw_index(tc, 3)],
+ REFUSALS[draw_index(tc, REFUSALS.len())]
+ ),
+ // A relative URL's host is the page's, which can be an Oracle bucket.
+ 6 => format!("/p/{SECRET}/n/ns/b/bucket/o/{path}"),
+ // On a page of its scheme a browser reads this as a path, where the URL parser alone reads a host.
+ 7 => format!(
+ "{}bcdn_token={SECRET}&expires=1/{path}",
+ ["https:", "https:/"][draw_index(tc, 2)]
+ ),
+ 1 => format!("{scheme}u:{SECRET}@{host}/{path}"),
+ // The policy finds these tokens by their place in the path, and the Oracle one by its host too.
+ 2 => format!(
+ "{scheme}objectstorage.us-ashburn-1.oraclecloud.com/p/{SECRET}/n/ns/b/bucket/o/{path}"
+ ),
+ 3 => format!("{scheme}{host}/bcdn_token={SECRET}&expires=1/{path}"),
+ 4 => format!("{scheme}{host}/image/upload/s--{SECRET}--/v1/{path}"),
+ _ => format!(
+ "{scheme}{host}/{path}{}{SECRET}",
+ REFUSALS[draw_index(tc, REFUSALS.len())]
+ ),
+ }
+}
+
+/// The bytes of `url` that a spelling must leave alone, so that the secret stays readable as plain text.
+fn secret_range(url: &str) -> std::ops::Range<usize> {
+ url.find(SECRET).map_or(0..0, |at| at..at + SECRET.len())
+}
+
+/// `url` as the URL parser reads it the same: the scheme and host in any letter case, the default port, a dot segment in the path, the first letter of the host percent-encoded, or fullwidth letters in the host, which IDNA maps to ASCII.
+fn parsed_alike(tc: &TestCase, url: &str) -> String {
+ let Some((scheme, rest)) = url.split_once("//") else {
+ return url.to_owned();
+ };
+ let (authority, path) = rest.split_once('/').unwrap();
+ let (userinfo, host) = authority
+ .rsplit_once('@')
+ .map_or(("", authority), |(userinfo, host)| (userinfo, host));
+ let userinfo = if userinfo.is_empty() {
+ String::new()
+ } else {
+ format!("{userinfo}@")
+ };
+ let scheme = if scheme.is_empty() {
+ ""
+ } else {
+ ["https:", "HTTPS:", "hTtPs:"][draw_index(tc, 3)]
+ };
+ let mut host: String = match draw_index(tc, 3) {
+ 0 => host.to_owned(),
+ 1 => host.to_ascii_uppercase(),
+ _ => host
+ .chars()
+ .enumerate()
+ .map(|(at, ch)| {
+ if at % 2 == 0 {
+ ch.to_ascii_uppercase()
+ } else {
+ ch
+ }
+ })
+ .collect(),
+ };
+ match draw_index(tc, 8) {
+ 0 | 1 => {
+ let first = host.remove(0);
+ host.insert_str(0, &format!("%{:02X}", first as u32));
+ }
+ 2 => {
+ host = host
+ .chars()
+ .enumerate()
+ .map(|(at, ch)| match char::from_u32(ch as u32 + 0xFEE0) {
+ Some(fullwidth) if at % 2 == 1 && ch.is_ascii_alphabetic() => fullwidth,
+ _ => ch,
+ })
+ .collect()
+ }
+ _ => {}
+ }
+ if draw_index(tc, 6) == 0 {
+ host.push('.');
+ }
+ let port = if draw_index(tc, 4) == 0 { ":443" } else { "" };
+ let dots = ["", "", "./", "x/../"][draw_index(tc, 4)];
+ format!("{scheme}//{userinfo}{host}{port}/{dots}{path}")
+}
+
+/// `url` with CSS escapes in the part before its query, so the secret stays readable as plain text.
+fn css_escaped(tc: &TestCase, url: &str) -> String {
+ let query_at = url.find('?').unwrap_or(url.len());
+ let secret = secret_range(url);
+ let mut out = String::new();
+ for (at, ch) in url.char_indices() {
+ let escape = at < query_at
+ && !secret.contains(&at)
+ && tc.draw(gs::integers::<u8>().min_value(0).max_value(5)) == 0;
+ if !escape {
+ out.push(ch);
+ continue;
+ }
+ match draw_index(tc, 3) {
+ 1 => out.push_str(&format!("\\{:06X}", ch as u32)),
+ 2 if !ch.is_ascii_hexdigit() => {
+ out.push('\\');
+ out.push(ch);
+ }
+ _ => out.push_str(&format!("\\{:x} ", ch as u32)),
+ }
+ }
+ out
+}
+
+/// `url` as an HTML attribute can hold it and a browser still loads it: the URL parser drops tabs and line breaks
+/// anywhere, and reads a backslash as a slash.
+fn html_spelled(tc: &TestCase, url: &str) -> String {
+ let mut out = url.to_owned();
+ if tc.draw(gs::booleans()) {
+ out = out.replacen("//", ["\\\\", "/\\", "\\/"][draw_index(tc, 3)], 1);
+ }
+ let query_at = out.find('?').unwrap_or(out.len());
+ for _ in 0..draw_index(tc, 3) {
+ let at = draw_index(tc, query_at);
+ let secret = secret_range(&out);
+ if out.is_char_boundary(at) && !(secret.start < at && at < secret.end) {
+ out.insert(at, ['\t', '\n'][draw_index(tc, 2)]);
+ }
+ }
+ out
+}
+
+/// `url` with the C0 controls and spaces that the URL parser strips from both ends. CSS reads a NUL as U+FFFD and ends a string at a line break, so a CSS slot takes neither.
+fn with_stripped_ends(tc: &TestCase, slot: Slot, url: &str) -> String {
+ let ends: &[char] = if slot.is_css() {
+ &[' ', '\t', '\u{1}', '\u{1f}']
+ } else {
+ &[' ', '\t', '\n', '\u{c}', '\u{0}', '\u{1}', '\u{1f}']
+ };
+ let end = || {
+ if draw_index(tc, 4) == 0 {
+ ends[draw_index(tc, ends.len())].to_string()
+ } else {
+ String::new()
+ }
+ };
+ let start = end();
+ format!("{start}{url}{}", end())
+}
+
+/// A spelling of `url` that a browser loads from the same place in `slot`, and that spelling with its CSS escapes decoded.
+fn draw_spelling(tc: &TestCase, slot: Slot, url: &str) -> (String, String) {
+ if !tc.draw(gs::booleans()) {
+ return (url.to_owned(), url.to_owned());
+ }
+ let parsed = parsed_alike(tc, url);
+ let mut spelled = parsed.clone();
+ let css_escapes = slot.is_css() && tc.draw(gs::booleans());
+ if css_escapes {
+ spelled = css_escaped(tc, &spelled);
+ } else if slot.is_html_url() && tc.draw(gs::booleans()) {
+ spelled = html_spelled(tc, &spelled);
+ }
+ let written = with_stripped_ends(tc, slot, &spelled);
+ let decoded = if !css_escapes {
+ written.clone()
+ } else {
+ let start = &written[..written.find(&spelled).unwrap()];
+ let end = &written[start.len() + spelled.len()..];
+ format!("{start}{parsed}{end}")
+ };
+ (written, decoded)
+}
+
+/// `url` as the URL parser writes it, which is what a script reads back from a slot.
+fn serialized(url: &str) -> String {
+ let Some((_, rest)) = url.split_once("//") else {
+ return format!("https://app.example.com{url}");
+ };
+ let (authority, tail) = rest.split_once('/').unwrap();
+ format!("https://{}/{tail}", authority.to_ascii_lowercase())
+}
+
+/// The CSS `url()` of `written`, quoted or not, under any spelling of its name. A comment can follow a quoted string.
+/// One before it makes a bad URL, which a browser never loads.
+fn url_function(tc: &TestCase, written: &str) -> String {
+ let name = ["url", "URL", "\\75 rl", "u\\72l"][draw_index(tc, 4)];
+ let comment = if tc.draw(gs::booleans()) { "/*c*/" } else { "" };
+ let space = if tc.draw(gs::booleans()) { " " } else { "" };
+ let needs_quotes =
+ written.contains(['(', ')', '"', '\'', '\\']) || written.contains(|ch: char| ch <= ' ');
+ match draw_index(tc, 3) {
+ 0 if !needs_quotes => format!("{name}({written})"),
+ 1 => format!("{name}({space}\"{written}\"{comment})"),
+ _ => format!("{name}({space}'{written}'{comment})"),
+ }
+}
+
+/// `data` as posthog-js sends a compressed snapshot or list: gzip bytes, one per character.
+fn compressed(data: &Value) -> String {
+ replay_core_v4::compression::gzip(data.to_string().as_bytes())
+ .unwrap()
+ .iter()
+ .map(|&byte| char::from(byte))
+ .collect()
+}
+
+/// `json` with some slashes and letters of its strings escaped. A draw for each character would make a large page too many choices, so one drawn seed decides them.
+fn with_json_escapes(tc: &TestCase, json: &str) -> String {
+ let rate = [0, 4, 16][draw_index(tc, 3)];
+ let mut state = tc.draw(gs::integers::<u64>()) | 1;
+ let mut escape = || {
+ state ^= state << 13;
+ state ^= state >> 7;
+ state ^= state << 17;
+ rate != 0 && state.is_multiple_of(rate)
+ };
+ let mut out = String::with_capacity(json.len());
+ let mut in_string = false;
+ let mut chars = json.chars();
+ while let Some(ch) = chars.next() {
+ match ch {
+ '"' => {
+ in_string = !in_string;
+ out.push(ch);
+ }
+ '\\' => {
+ out.push(ch);
+ let escaped = chars.next().unwrap();
+ out.push(escaped);
+ if escaped == 'u' {
+ out.extend(chars.by_ref().take(4));
+ }
+ }
+ '/' if in_string && escape() => out.push_str("\\/"),
+ _ if in_string && ch.is_ascii_alphabetic() && escape() => {
+ out.push_str(&format!("\\u{:04x}", ch as u32))
+ }
+ _ => out.push(ch),
+ }
+ }
+ out
+}
+
+/// `text` as a JSON string holds it, the way a network body or a console log copies a slot's value.
+fn json_text(text: &str) -> String {
+ let quoted = serde_json::to_string(text).unwrap();
+ quoted[1..quoted.len() - 1].to_owned()
+}
+
+/// A copy takes a spelling that the inspector records for the URL: as the page wrote it in the slot, as CSS decodes it, or as the URL parser writes it. A JSON copy holds the slot's spelling within its escaped ends.
+fn draw_placed(tc: &TestCase, slots: &[Slot], url: String, refused: bool) -> Placed {
+ let slot = slots[draw_index(tc, slots.len())];
+ let (written, decoded) = draw_spelling(tc, slot, &url);
+ let copies = if refused {
+ let mut spellings = vec![written.clone(), decoded, serialized(&url)];
+ let core = written.trim_matches(|ch: char| ch <= ' ');
+ if json_text(core) == core {
+ spellings.push(json_text(&written));
+ }
+ tc.draw(gs::vecs(gs::sampled_from(spellings)).max_size(2))
+ } else {
+ Vec::new()
+ };
+ Placed {
+ slot,
+ written,
+ copies,
+ }
+}
+
+fn draw_shape(tc: &TestCase) -> Shape {
+ Shape {
+ restart: tc.draw(gs::booleans()),
+ compressed: tc.draw(gs::booleans()),
+ deep: draw_index(tc, 16) == 0,
+ }
+}
+
+fn element(id: &mut i64, tag: &str, attributes: Value) -> Value {
+ *id += 1;
+ json!({"type": 2, "tagName": tag, "attributes": attributes, "childNodes": [], "id": *id})
+}
+
+fn mutation(lists: Value) -> Value {
+ let mut data = json!({"source": 0, "adds": [], "removes": [], "texts": [], "attributes": []});
+ for (name, list) in lists.as_object().unwrap() {
+ data[name] = list.clone();
+ }
+ json!({"type": 3, "timestamp": 2, "data": data})
+}
+
+/// The page's events. Each URL is in its slot and each copy in an `alt` attribute or a text node. A full snapshot
+/// holds the nodes, or after a restart a mutation adds them under a node that the inspector does not know.
+fn events(tc: &TestCase, placed: &[Placed], shape: Shape) -> String {
+ let mut id = 1;
+ let mut added = Vec::new();
+ let mut known = Vec::new();
+ let mut incremental = Vec::new();
+ for Placed {
+ slot,
+ written,
+ copies,
+ } in placed
+ {
+ let function = || url_function(tc, written);
+ match slot {
+ Slot::ImgSrc => added.push(element(&mut id, "img", json!({"src": written}))),
+ Slot::Srcset => added.push(element(
+ &mut id,
+ "img",
+ json!({"srcset": format!("{written} 1x, https://cdn.example.com/other.png 2x")}),
+ )),
+ Slot::StyleAttribute => added.push(element(
+ &mut id,
+ "div",
+ json!({"style": format!("background-image:{}", function())}),
+ )),
+ Slot::StyleSheet => added.push(element(
+ &mut id,
+ "style",
+ json!({"_cssText": format!(".a{{background:{}}}", function())}),
+ )),
+ Slot::FontFace(family) => added.push(element(
+ &mut id,
+ "style",
+ json!({"_cssText": format!("@font-face{{font-family:{family};src:{}}}", function())}),
+ )),
+ Slot::ImageSet => added.push(element(
+ &mut id,
+ "div",
+ json!({"style": format!("background-image:image-set({} 1x)", function())}),
+ )),
+ Slot::CrossFade { in_image_set } => {
+ let fade = format!(
+ "-webkit-cross-fade({}, url(https://cdn.example.com/other.png), 50%)",
+ function()
+ );
+ let image = if *in_image_set {
+ format!("image-set({fade} 1x)")
+ } else {
+ fade
+ };
+ added.push(element(
+ &mut id,
+ "div",
+ json!({"style": format!("background-image:{image}")}),
+ ));
+ }
+ Slot::StyleRule => incremental.push(json!({"type": 3, "timestamp": 2, "data": {"source": 8, "id": 3,
+ "replaceSync": format!(".r{{background:{}}}", function())}})),
+ Slot::StyleChange => {
+ known.push(element(&mut id, "div", json!({})));
+ incremental.push(mutation(json!({"attributes": [{"id": id,
+ "attributes": {"style": format!("background-image:{}", function())}}]})));
+ }
+ Slot::StyleObjectChange => {
+ known.push(element(&mut id, "div", json!({})));
+ incremental.push(mutation(json!({"attributes": [{"id": id,
+ "attributes": {"style": {"background-image": function()}}}]})));
+ }
+ Slot::StyleText => {
+ let mut style = element(&mut id, "style", json!({}));
+ id += 1;
+ style["childNodes"] =
+ json!([{"type": 3, "textContent": "", "isStyle": true, "id": id}]);
+ known.push(style);
+ incremental.push(mutation(json!({"texts": [{"id": id,
+ "value": format!(".t{{background:{}}}", function())}]})));
+ }
+ Slot::Href => added.push(element(&mut id, "a", json!({"href": written}))),
+ Slot::FormAction => added.push(element(&mut id, "form", json!({"action": written}))),
+ Slot::DataSrc => added.push(element(&mut id, "img", json!({"data-src": written}))),
+ Slot::Filter => added.push(element(
+ &mut id,
+ "div",
+ json!({"style": format!("filter:{}", function())}),
+ )),
+ Slot::Import => added.push(element(
+ &mut id,
+ "style",
+ json!({"_cssText": format!("@import {};", function())}),
+ )),
+ Slot::SvgFill => added.push(element(&mut id, "rect", json!({"fill": function()}))),
+ Slot::AttributionSrc => added.push(element(
+ &mut id,
+ "img",
+ json!({"attributionsrc": format!("https://cdn.example.com/x {written}")}),
+ )),
+ Slot::PictureSource => {
+ known.push(element(&mut id, "picture", json!({})));
+ let parent = id;
+ let source = element(
+ &mut id,
+ "source",
+ json!({"srcset": format!("{written} 1x")}),
+ );
+ incremental.push(mutation(
+ json!({"adds": [{"parentId": parent, "nextId": null, "node": source}]}),
+ ));
+ }
+ }
+ for copy in copies {
+ if tc.draw(gs::booleans()) {
+ added.push(element(&mut id, "img", json!({"alt": copy})));
+ } else {
+ id += 1;
+ added.push(json!({"type": 3, "textContent": copy, "id": id}));
+ }
+ }
+ }
+ if shape.deep {
+ added = (0..DEEP_NODES).fold(added, |children, _| {
+ let mut parent = element(&mut id, "div", json!({}));
+ parent["childNodes"] = Value::Array(children);
+ vec![parent]
+ });
+ }
+ let mut events = Vec::new();
+ if shape.restart {
+ let adds: Vec<Value> = added
+ .into_iter()
+ .map(|node| json!({"parentId": 1, "nextId": null, "node": node}))
+ .collect();
+ events.push(if shape.compressed {
+ json!({"type": 3, "timestamp": 1, "cv": "2024-10", "data": {"source": 0, "adds": compressed(&json!(adds)),
+ "removes": compressed(&json!([])), "texts": compressed(&json!([])), "attributes": compressed(&json!([]))}})
+ } else {
+ mutation(json!({"adds": adds}))
+ });
+ } else {
+ events.push(
+ json!({"type": 4, "timestamp": 1, "data": {"href": "https://app.example.com/page"}}),
+ );
+ known.extend(added);
+ let snapshot = json!({"node": {"type": 0, "id": 1, "childNodes": known}});
+ events.push(if shape.compressed {
+ json!({"type": 2, "timestamp": 1, "cv": "2024-10", "data": compressed(&snapshot)})
+ } else {
+ json!({"type": 2, "timestamp": 1, "data": snapshot})
+ });
+ }
+ events.extend(incremental);
+ with_json_escapes(tc, &Value::Array(events).to_string())
+}
+
+fn drop_reasons(placed: &[Placed], shape: Shape) -> Vec<&'static str> {
+ let mut reasons = Vec::new();
+ if placed
+ .iter()
+ .any(|placed| placed.slot.drops_its_block(&placed.written))
+ {
+ reasons.push("unplaced_slot");
+ }
+ if shape.deep {
+ reasons.extend(["too_deep_json", "unreadable_message"]);
+ }
+ reasons
+}
+
+fn block_of(events: &str) -> Option<Block> {
+ let inner = format!(
+ r#"{{"event":"$snapshot_items","properties":{{"$session_id":"s","$window_id":"w1","$snapshot_items":{events}}}}}"#
+ );
+ let message = read_message(
+ json!({"distinct_id": "d", "data": inner})
+ .to_string()
+ .into_bytes(),
+ )
+ .ok()?;
+ let mut inspector = SessionInspector::default();
+ let mut inspection = BlockInspection::default();
+ inspector.observe_message(&mut inspection, &message);
+ let found = inspection.finish();
+ Some(Block {
+ session: SessionKey {
+ team_id: 7,
+ session_id: "s".to_owned(),
+ },
+ origin: Origin {
+ tp: Tp {
+ topic: "main",
+ partition: 0,
+ },
+ start_offset: 100,
+ end_offset: 105,
+ flush_end_offset: 105,
+ },
+ messages: vec![message],
+ stats: BlockStats::default(),
+ moments: Vec::new(),
+ image_urls: found.image_urls,
+ font_urls: found.font_urls,
+ url_limit: found.url_limit,
+ edits: found.edits,
+ unplaced: found.unplaced,
+ visited_urls: Vec::new(),
+ captured_at_ms: None,
+ })
+}
+
+/// The stored lines, or why the block is not stored. The recorder skips a message it cannot read.
+fn stored(tc: &TestCase, events: &str) -> Result<String, &'static str> {
+ let Some(block) = block_of(events) else {
+ tc.event("unreadable_message");
+ return Err("unreadable_message");
+ };
+ let (lines, _, counts): (Vec<u8>, _, WriterCounts) =
+ write_block(&block, &TEAM_KEY, KEY_MONTH, || Ok(Vec::new())).unwrap();
+ tc.event(counts.dropped.unwrap_or("stored"));
+ match counts.dropped {
+ Some(reason) => Err(reason),
+ None => Ok(String::from_utf8(lines).unwrap()),
+ }
+}
+
+/// `lines` with each JSON line written again, so that an escape in a string does not hide what the string holds.
+fn decoded(lines: &str) -> String {
+ lines
+ .lines()
+ .map(|line| {
+ serde_json::from_str::<Value>(line)
+ .map_or_else(|_| line.to_owned(), |value| value.to_string())
+ })
+ .collect::<Vec<_>>()
+ .join("\n")
+}
+
+struct Page {
+ placed: Vec<Placed>,
+ shape: Shape,
+ events: String,
+}
+
+fn draw_page(tc: &TestCase) -> Page {
+ let shape = draw_shape(tc);
+ let refused = tc.draw(gs::integers::<usize>().min_value(0).max_value(3));
+ let accepted = tc.draw(gs::integers::<usize>().min_value(0).max_value(2));
+ // A link keeps an accepted URL as the page wrote it, so only a slot that names one takes one.
+ let image_slots: Vec<Slot> = SLOTS
+ .iter()
+ .copied()
+ .filter(|slot| slot.names_an_accepted_url(shape))
+ .collect();
+ let placed: Vec<Placed> = (0..refused)
+ .map(|index| (index, true))
+ .chain((0..accepted).map(|index| (index, false)))
+ .map(|(index, refused)| {
+ let url = draw_url(tc, index, refused);
+ let slots = if refused { SLOTS } else { &image_slots };
+ draw_placed(tc, slots, url, refused)
+ })
+ .collect();
+ let events = events(tc, &placed, shape);
+ Page {
+ placed,
+ shape,
+ events,
+ }
+}
+
+/// v4 stores pages unscrubbed, so each copy of a URL outside a URL slot keeps its text, and a slot keeps no credential.
+#[hegel::test]
+#[ignore = "a search for bugs, run on demand"]
+fn a_stored_block_holds_a_credential_only_in_copies(tc: TestCase) {
+ let page = draw_page(&tc);
+ let Ok(lines) = stored(&tc, &page.events) else {
+ return;
+ };
+ let mut rest = decoded(&lines);
+ let mut copies: Vec<String> = page
+ .placed
+ .iter()
+ .flat_map(|placed| &placed.copies)
+ .map(|copy| json!(copy).to_string())
+ .collect();
+ copies.sort();
+ for copy in copies.chunk_by(|a, b| a == b) {
+ assert_eq!(
+ rest.matches(&copy[0]).count(),
+ copy.len(),
+ "{}\n---\n{lines}",
+ page.events
+ );
+ rest = rest.replace(&copy[0], "");
+ }
+ let rest = rest
+ .replace("\\t", "")
+ .replace("\\n", "")
+ .to_ascii_lowercase();
+ assert!(!rest.contains(SECRET), "{}\n---\n{lines}", page.events);
+}
+
+/// Every spelling of a refused URL that a browser loads holds a credential that the policy finds, and loses it.
+#[hegel::test]
+#[ignore = "a search for bugs, run on demand"]
+fn a_url_without_its_credentials_holds_no_secret(tc: TestCase) {
+ let url = draw_url(&tc, 0, true);
+ let (written, _) = draw_spelling(&tc, Slot::ImgSrc, &url);
+ let kept = match redaction(&written) {
+ Redaction::Redacted(kept) => kept,
+ // The block keeps a decline marker in its place.
+ Redaction::Unread => return,
+ Redaction::Clean => panic!("{written:?}"),
+ };
+ assert!(
+ !kept
+ .replace(['\t', '\n'], "")
+ .to_ascii_lowercase()
+ .contains(SECRET),
+ "{written:?} => {kept:?}"
+ );
+}
+
+#[hegel::test]
+#[ignore = "a search for bugs, run on demand"]
+fn each_stored_line_is_a_window_and_an_event(tc: TestCase) {
+ let page = draw_page(&tc);
+ if let Ok(lines) = stored(&tc, &page.events) {
+ for line in lines.lines() {
+ let parsed: Value = serde_json::from_str(line)
+ .unwrap_or_else(|error| panic!("{error}: {line}\n---\n{}", page.events));
+ assert!(
+ parsed.as_array().is_some_and(|line| line.len() == 2),
+ "{line}"
+ );
+ }
+ }
+}
+
+#[hegel::test]
+#[ignore = "a search for bugs, run on demand"]
+fn a_stored_block_holds_no_accepted_url_in_an_image_slot(tc: TestCase) {
+ let page = draw_page(&tc);
+ if let Ok(lines) = stored(&tc, &page.events) {
+ assert!(
+ !decoded(&lines).to_ascii_lowercase().contains(ACCEPTED),
+ "{}\n---\n{lines}",
+ page.events
+ );
+ }
+}
+
+#[hegel::test]
+#[ignore = "a search for bugs, run on demand"]
+fn drops_a_block_only_for_a_reason_the_page_holds(tc: TestCase) {
+ let page = draw_page(&tc);
+ if let Err(reason) = stored(&tc, &page.events) {
+ assert!(
+ drop_reasons(&page.placed, page.shape).contains(&reason),
+ "{reason}: {}",
+ page.events
+ );
+ }
+}
+
+#[hegel::test]
+#[ignore = "a search for bugs, run on demand"]
+fn each_spelling_of_a_url_gets_the_reference_its_fetch_job_names(tc: TestCase) {
+ let shape = Shape {
+ deep: false,
+ ..draw_shape(&tc)
+ };
+ let url = draw_url(&tc, 0, false);
+ let slots: Vec<Slot> = SLOTS
+ .iter()
+ .copied()
+ .filter(|slot| !slot.drops_its_block("") && slot.names_an_accepted_url(shape))
+ .collect();
+ let count = tc.draw(gs::integers::<usize>().min_value(2).max_value(4));
+ let placed: Vec<Placed> = (0..count)
+ .map(|_| draw_placed(&tc, &slots, url.clone(), false))
+ .collect();
+ let events = events(&tc, &placed, shape);
+ let Ok(lines) = stored(&tc, &events) else {
+ return;
+ };
+ let (_, reference) = url_reference(
+ &url,
+ UrlAsset::Image,
+ 7,
+ KEY_MONTH,
+ &image_url_key(&TEAM_KEY),
+ )
+ .unwrap();
+ let hash = reference.rsplit(':').next().unwrap();
+ let naming = placed
+ .iter()
+ .filter(|placed| placed.slot.names_its_url())
+ .count();
+ assert_eq!(
+ decoded(&lines).matches(hash).count(),
+ naming,
+ "{events}\n---\n{lines}"
+ );
+}