Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(ai-research-v4): apply the v4 url policy to protocol-relative image urls (#124822)

Nicholas Waltz64aca6166136
diff --git a/products/ai_training/docs/replay-data.md b/products/ai_training/docs/replay-data.md
index acf428237d75..30ad40bec97d 100644
--- a/products/ai_training/docs/replay-data.md
+++ b/products/ai_training/docs/replay-data.md
@@ -465,6 +465,7 @@ The recorder streams each block into the encoder. Read a block as a stream: a Br
- A remote image URL becomes `imageurl:v4:{team_id}:{YYYY-MM}:{hash}` in the block and in its fetch job, where the month is the UTC month of the write. The block and the job always carry the same reference, so a reader opens the fetched image directly.
- A URL that the URL policy refuses becomes `imageurl:v4:declined:{reason}` in the block, for example `declined:credential`, `declined:non_public_host` or `declined:tracking_beacon`. The original URL cannot be recovered. The recorder counts rewrites and refusals in `ai_research_v4_recorder_image_url_rewrites_total`.
- The rewrite covers every remote URL that the recorder reads from an image slot: image sources, `srcset`, SVG image links, video posters, `<picture>` sources and CSS `url(...)` in style attributes, style sheets and style rules. It also covers `src`, `srcset`, `href` and `poster` in an attribute change on a node the recorder does not know, for example after a restart. `srcset` keeps its descriptors and CSS keeps its `url(...)` syntax. It is not limited by the 512 URLs per message that get fetch jobs.
+- The recorder reads a protocol-relative URL, such as `//cdn.example.com/a.png`, as `https`. It gets the same reference or `declined` marker as the `https` form, and the rewrite replaces the text as the block holds it. A browser reads a backslash as a slash there, so `\\cdn.example.com/a.png` counts too, and so does a URL such as `https:\\cdn.example.com/a.png` with any mix of slashes and backslashes after the scheme, or none: on a page of another scheme the browser loads `https:cdn.example.com/a.png` from that host.
- Some URLs stay as they arrived: relative URLs, remote URLs outside image slots (links, frames, text, CSS custom properties and other CSS properties), the `image-set()` candidates the recorder does not choose, URLs inside a compressed value that did not decode or that the decode budget skipped, and URL text written in another form than the recorder read it, for example with a CSS escape. The recorder counts the last kind as `outcome="unrewritten"`.
- Before the rewrite, the recorder writes again any JSON string that holds an escape capture would not write, for example `\/`, so an escaped URL is found too. A scheme matches in any letter case.
- If a refused URL is still in a block after the rewrite, the recorder drops the block and counts it as `dropped_blocks_total{outcome="declined_unrewritten"}`, so no refused URL is stored.
diff --git a/rust/ai-research/replay-core-v4/src/inspect/dom.rs b/rust/ai-research/replay-core-v4/src/inspect/dom.rs
index f79788d476d9..1ae79a06beaf 100644
--- a/rust/ai-research/replay-core-v4/src/inspect/dom.rs
+++ b/rust/ai-research/replay-core-v4/src/inspect/dom.rs
@@ -5,7 +5,7 @@ use std::collections::{HashMap, HashSet};
use std::ops::Range;
use crate::bytescan::{self as scan, Span};
-use crate::url_policy::MAX_URL_LEN;
+use crate::url_policy::{self, MAX_URL_LEN};
use super::css;
use super::events::{field, fields, number, text};
@@ -253,13 +253,20 @@ fn fetchable(url: &str) -> Option<&str> {
(url.len() <= MAX_URL_LEN && is_remote(url)).then_some(url)
}
-/// A URL scheme is not case sensitive, so `HTTPS://` loads as `https://` does.
+/// A URL scheme is not case sensitive, so `HTTPS://` loads as `https://` does. A browser reads any run of slashes and
+/// backslashes after `http:` or `https:` as `//`, and none at all on a page of another scheme, so `https:\\host` and
+/// `https:host` can load from the host too. A protocol-relative URL does as well. The writer must replace each of them
+/// like any other remote URL.
fn is_remote(url: &str) -> bool {
- let starts = |prefix: &str| {
- url.get(..prefix.len())
- .is_some_and(|start| start.eq_ignore_ascii_case(prefix))
- };
- starts("https://") || starts("http://")
+ let after_scheme = ["https:", "http:"].iter().find_map(|scheme| {
+ url.get(..scheme.len())
+ .filter(|start| start.eq_ignore_ascii_case(scheme))
+ .map(|_| &url[scheme.len()..])
+ });
+ match after_scheme {
+ Some(rest) => !rest.is_empty(),
+ None => url_policy::is_protocol_relative(url),
+ }
}
impl Urls {
@@ -1562,7 +1569,7 @@ mod tests {
element(8, "iframe", json!({}), json!([])),
element(9, "canvas", json!({}), json!([])),
element(11, "img", json!({"src": "https://cdn.example.com/a.png", "srcset": "https://cdn.example.com/[email protected] 2x, data:image/png;base64,AA 3x"}), json!([])),
- element(12, "video", json!({"poster": "https://cdn.example.com/p.jpg"}), json!([])),
+ element(12, "video", json!({"poster": "//cdn.example.com/p.jpg"}), json!([])),
{"type": 2, "tagName": "div", "attributes": {}, "childNodes": [], "isShadowHost": true, "id": 10},
])),
]))], "id": 1}})
@@ -1586,7 +1593,7 @@ mod tests {
[
"https://cdn.example.com/a.png",
"https://cdn.example.com/[email protected]",
- "https://cdn.example.com/p.jpg"
+ "//cdn.example.com/p.jpg"
]
);
diff --git a/rust/ai-research/replay-core-v4/src/url_policy.rs b/rust/ai-research/replay-core-v4/src/url_policy.rs
index 0798daa1f68e..870675ad1539 100644
--- a/rust/ai-research/replay-core-v4/src/url_policy.rs
+++ b/rust/ai-research/replay-core-v4/src/url_policy.rs
@@ -292,7 +292,7 @@ pub enum Decline {
/// Longer than [`MAX_URL_LEN`], before or after normalization.
TooLong,
/// Not parseable as an absolute URL. A relative `src` lands here, because the recording does
- /// not carry the base it would need to resolve against.
+ /// not carry the base it would need to resolve against. A protocol-relative URL does not, because it names its host.
NotAbsolute,
/// A scheme we do not fetch. Only `https` passes, so plain `http` lands here too.
BadScheme,
@@ -340,10 +340,25 @@ pub fn canonicalize(raw: &str) -> Option<CanonicalUrl> {
try_canonicalize(raw).ok()
}
+/// Whether the URL names a host but no scheme, such as `//cdn.example.com/a.png`. The browser loads it with the page's
+/// scheme, so it is remote. A browser reads a backslash as a slash in an `https` URL, so `\\host` and `/\host` count too.
+pub fn is_protocol_relative(raw: &str) -> bool {
+ matches!(raw.trim().as_bytes(), [b'/' | b'\\', b'/' | b'\\', ..])
+}
+
pub fn try_canonicalize(raw: &str) -> Result<CanonicalUrl, Decline> {
if raw.len() > MAX_URL_LEN {
return Err(Decline::TooLong);
}
+ // Only `https` passes the scheme check below, so a protocol-relative URL is read as `https`. It then meets every
+ // check that the same URL with a scheme meets.
+ let with_scheme;
+ let raw = if is_protocol_relative(raw) {
+ with_scheme = format!("https:{}", raw.trim());
+ with_scheme.as_str()
+ } else {
+ raw
+ };
let mut url = Url::parse(raw).map_err(|_| Decline::NotAbsolute)?;
// The browser blocks a plain `http` image on an HTTPS page as mixed content, so it never
// rendered for the person we recorded, and a fetch puts the request on the wire in clear text
@@ -556,7 +571,6 @@ mod tests {
"data:image/png;base64,aGk=",
"/relative/a.png",
"not a url",
- "//protocol-relative.example.com/a.png",
] {
assert!(canonicalize(raw).is_none(), "{raw} should not canonicalize");
}
@@ -632,6 +646,23 @@ mod tests {
let c = canonicalize("HTTPS://Example.COM:443/A.png#frag").unwrap();
assert_eq!(c.fetch, "https://example.com/A.png");
assert_eq!(c.host, "example.com");
+ // A protocol-relative URL gets the same canonical form, so it shares the reference of its https form.
+ for raw in [
+ "//Example.COM/A.png",
+ " //example.com/A.png ",
+ r"\\example.com/A.png",
+ r"/\example.com/A.png",
+ r"https:\\example.com/A.png",
+ r"https:/\example.com/A.png",
+ "https:/example.com/A.png",
+ "https:example.com/A.png",
+ ] {
+ assert_eq!(canonicalize(raw).unwrap(), c, "{raw}");
+ }
+ assert_eq!(
+ try_canonicalize("//user:[email protected]/a.png"),
+ Err(Decline::Credential)
+ );
}
#[test]
diff --git a/rust/ai-research/replay-recorder-v4/src/rewrite.rs b/rust/ai-research/replay-recorder-v4/src/rewrite.rs
index b015ff7df02d..70a4485d61c0 100644
--- a/rust/ai-research/replay-recorder-v4/src/rewrite.rs
+++ b/rust/ai-research/replay-recorder-v4/src/rewrite.rs
@@ -323,6 +323,9 @@ mod tests {
),
("https://10.0.0.7/a.png", "non_public_host"),
("http://cdn.example.com/a.png", "bad_scheme"),
+ ("//user:[email protected]/a.png", "credential"),
+ ("//10.0.0.7/a.png", "non_public_host"),
+ ("https:/user:[email protected]/a.png", "credential"),
] {
let out = rewrite(&[url], &format!(r#"{{"src":"{url}"}}"#));
assert_eq!(
diff --git a/rust/ai-research/replay-recorder-v4/src/write.rs b/rust/ai-research/replay-recorder-v4/src/write.rs
index 248e3b256b39..b8c7b1a6319f 100644
--- a/rust/ai-research/replay-recorder-v4/src/write.rs
+++ b/rust/ai-research/replay-recorder-v4/src/write.rs
@@ -753,6 +753,22 @@ mod tests {
.replace(r#""src""#, r#""\u0073rc""#)
.replace(r#""srcset""#, r#""src\u0073et""#);
assert_eq!(stored_lines(&escaped, &[]), lines);
+ // A browser reads backslashes after the scheme as slashes, so such a URL is replaced like its `https://` form.
+ for backslashed in [
+ r"https:\\user:[email protected]/a.png",
+ r"https:/\user:[email protected]/a.png",
+ "https:user:[email protected]/a.png",
+ ] {
+ let events = json!([{"type": 3, "timestamp": 1, "data": {"source": 0, "adds": [], "removes": [], "texts": [],
+ "attributes": [{"id": 99, "attributes": {"src": backslashed}}]}}])
+ .to_string();
+ let lines = stored_lines(&events, &[]);
+ assert!(
+ lines.contains(&crate::rewrite::declined_marker("credential"))
+ && !lines.contains("secret"),
+ "{lines}"
+ );
+ }
// The player loads the last of a repeated attribute, so the inspector names that one.
for events in [
format!(