Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(ai-research): move the v4 image ref grammar into the anonymizer (#126496)

Robbieba539045a5c0
diff --git a/products/ai_training/docs/replay-data.md b/products/ai_training/docs/replay-data.md
index 4bc971f1e7f9..361237b7ad7b 100644
--- a/products/ai_training/docs/replay-data.md
+++ b/products/ai_training/docs/replay-data.md
@@ -471,6 +471,7 @@ The recorder streams each block into the encoder. Read a block as a stream: a Br
- The recorder reads every argument of a canvas mutation, not only the frames of the posthog-js canvas recorder. A buffer becomes a reference when its bytes are PNG, JPEG, GIF or WebP, and a marker when they are not. The pixels of an `ImageData`, and the numbers of a `Uint8Array` or a `Uint8ClampedArray`, become `image:v4:declined:unsupported`. A data URI in the `src` of an `HTMLImageElement` moves out like any data URI, and a remote URL there stays as it is.
- A remote image URL becomes `imageurl:v4:{team_id}:{YYYY-MM}:{hash}` in the block and in its fetch job, where the month is the UTC month of the write. The block and the job always carry the same reference, so a reader opens the fetched image directly.
- A URL that the URL policy refuses becomes `imageurl:v4:declined:{reason}` in the block, for example `declined:credential`, `declined:non_public_host` or `declined:tracking_beacon`. The original URL cannot be recovered. The recorder counts rewrites and refusals in `ai_research_v4_recorder_image_url_rewrites_total`.
+- The `v4_image_refs` module of the `posthog-replay-anonymizer` crate defines each reference and marker shape. The recorder writes them with it. A reader that must tell them apart from page text uses its `parse` function, which accepts only the exact strings the recorder writes.
- The rewrite covers every remote URL that the recorder reads from an image slot: image sources, `srcset`, SVG image links, video posters, `<picture>` sources and CSS `url(...)` in style attributes, style sheets and style rules, in custom properties too, because a page passes a custom property to an image property through `var()`. It also covers `src`, `srcset`, `href` and `poster` in an attribute change on a node the recorder does not know, for example after a restart. `srcset` keeps its descriptors and CSS keeps its `url(...)` syntax. A URL is also replaced where a selector holds it with CSSOM's escapes, by the reference with the same escapes, so a class such as `bg-[url(https://cdn.example.com/a.png)]` and its rule `.bg-\[url\(https\:\/\/cdn\.example\.com\/a\.png\)\]` still match within a block. A reader that replaces references by text must replace the escaped form as well. It is not limited by the 512 URLs per message that get fetch jobs.
- A srcset in an image slot keeps only the candidates the recorder stores. A candidate the URL policy refuses leaves the list instead of becoming a `declined` marker, and the rest keep their descriptors. A `<picture>` source whose `type` names a media type the image fetcher does not store, such as `image/avif`, gets no fetch job and loses its remote candidates, unless a source of a stored type names the same URL. Its inline images, such as SVG data URIs, stay. The writer rebuilds each srcset that the inspector read in an image slot, in the bytes it stores, after the inline images left and the escapes were written again. It separates the candidates with a comma and a space. A `srcset` key elsewhere, for example in a custom event's payload, stays as it is. A browser skips a source with no candidates, so the replay falls back to the next source or the `img`. The recorder counts these as `ai_research_v4_recorder_srcset_rewrites_total{outcome="trimmed"|"emptied"}`.
- The writer rewrites CSS text before it replaces the URLs, so that an offline render loads only images the block refers to. An `image-set()` keeps one candidate, because a renderer picks the candidate that fits its pixel ratio and does not fall back when that candidate fails. It keeps the highest resolution of a type the image fetcher stores among the candidates that can load: inline, accepted by the URL policy, or root-relative on a known page. Only that candidate gets a fetch job, and the other remote candidates get references without one, in case the writer does not find the set's text. A set that holds an inline image keeps all its candidates, because the writer moves the inline image out first.
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
index 7391b08a2c17..2335705ed671 100644
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -9778,7 +9778,7 @@ dependencies = [
[[package]]
name = "posthog-replay-anonymizer"
-version = "0.12.1"
+version = "0.13.0"
dependencies = [
"ahash 0.8.12",
"anyhow",
@@ -11050,6 +11050,7 @@ dependencies = [
"memchr",
"moka",
"percent-encoding",
+ "posthog-replay-anonymizer",
"public-suffix",
"rand 0.8.5",
"serde",
@@ -11097,6 +11098,7 @@ dependencies = [
"lifecycle",
"metrics",
"moka",
+ "posthog-replay-anonymizer",
"rand 0.8.5",
"rdkafka",
"replay-core-v4",
diff --git a/rust/ai-research/replay-core-v4/Cargo.toml b/rust/ai-research/replay-core-v4/Cargo.toml
index 1797eb26310f..0c219e86651e 100644
--- a/rust/ai-research/replay-core-v4/Cargo.toml
+++ b/rust/ai-research/replay-core-v4/Cargo.toml
@@ -31,6 +31,7 @@ md5 = "0.8"
memchr = "2"
moka = { workspace = true, optional = true }
percent-encoding = { workspace = true }
+posthog-replay-anonymizer = { path = "../../replay-anonymizer", default-features = false }
public-suffix = { workspace = true }
rand = { workspace = true, optional = true }
serde = { workspace = true }
diff --git a/rust/ai-research/replay-core-v4/src/refs.rs b/rust/ai-research/replay-core-v4/src/refs.rs
index b5dce802a72d..8e815172bdb0 100644
--- a/rust/ai-research/replay-core-v4/src/refs.rs
+++ b/rust/ai-research/replay-core-v4/src/refs.rs
@@ -1,12 +1,12 @@
-//! Image and URL references, which stand in for an image in a v4 block, and the URL fingerprints of the block index.
+//! The keyed hash of the image and URL references in `posthog_replay_anonymizer::v4_image_refs`, and the URL
+//! fingerprints of the block index.
use base64::engine::general_purpose::URL_SAFE_NO_PAD;
use base64::Engine;
use hmac::{Hmac, Mac};
+use posthog_replay_anonymizer::v4_image_refs::HASH_CHARS;
use sha2::Sha256;
-/// The length v3 truncates a reference hash to: 132 bits of the HMAC.
-const HASH_CHARS: usize = 22;
/// 64 bits of the HMAC: enough to tell the pages of one session apart.
const FINGERPRINT_BYTES: usize = 8;
@@ -20,14 +20,6 @@ pub fn reference_hash(key: &[u8; 32], bytes: &[u8]) -> String {
encoded
}
-pub fn image_ref(team_id: i64, key_month: &str, hash: &str) -> String {
- format!("image:v4:{team_id}:{key_month}:{hash}")
-}
-
-pub fn image_url_ref(team_id: i64, key_month: &str, hash: &str) -> String {
- format!("imageurl:v4:{team_id}:{key_month}:{hash}")
-}
-
/// A page URL as the block index shows it in plaintext: lowercase hex of the HMAC under
/// [`crate::seal::url_fingerprint_key`].
pub fn url_fingerprint(key: &[u8; 32], url: &str) -> String {
diff --git a/rust/ai-research/replay-core-v4/src/scan/cv.rs b/rust/ai-research/replay-core-v4/src/scan/cv.rs
index d90a425aad7b..89445932a52f 100644
--- a/rust/ai-research/replay-core-v4/src/scan/cv.rs
+++ b/rust/ai-research/replay-core-v4/src/scan/cv.rs
@@ -10,7 +10,9 @@ use crate::compression;
use serde::de::IgnoredAny;
use super::canvas::{self, CanvasImage};
-use super::images::{extract_images, Declined, ImageSink};
+use posthog_replay_anonymizer::v4_image_refs::ImageDecline;
+
+use super::images::{extract_images, ImageSink};
use super::message::{canonicalize_keys, Event, Message};
const FULL_SNAPSHOT: u8 = 2;
@@ -473,9 +475,9 @@ fn move_out_canvas_images(
let replacement = match image {
CanvasImage::Base64(span) => match scan::unescape(inner, span) {
Ok(payload) => sink.take_base64(payload.as_bytes()),
- Err(_) => sink.decline(Declined::Undecodable),
+ Err(_) => sink.decline(ImageDecline::Undecodable),
},
- CanvasImage::Pixels(_) => sink.decline(Declined::Unsupported),
+ CanvasImage::Pixels(_) => sink.decline(ImageDecline::Unsupported),
};
let mut quoted = Vec::with_capacity(replacement.len() + 2);
scan::write_json_string(&replacement, &mut quoted);
@@ -877,7 +879,7 @@ mod tests {
fn moves_every_canvas_image_out_or_declines_it() {
enum Outcome {
Stored(&'static str),
- Marker(Declined),
+ Marker(ImageDecline),
Kept,
}
let png = b"\x89PNG\r\n\x1a\nframe".as_slice();
@@ -920,7 +922,7 @@ mod tests {
(
draw(bitmap("image/bmp", b"BMbitmap")),
format!("/data/commands/0/args/0{blob}"),
- Outcome::Marker(Declined::Unsupported),
+ Outcome::Marker(ImageDecline::Unsupported),
),
(
draw(bitmap(
@@ -928,7 +930,7 @@ mod tests {
b"\x00\x00\x00\x1cftypavif\x00\x00\x00\x00avifmif1miaf",
)),
format!("/data/commands/0/args/0{blob}"),
- Outcome::Marker(Declined::Avif),
+ Outcome::Marker(ImageDecline::Avif),
),
(
draw(
@@ -936,7 +938,7 @@ mod tests {
"data": [{"rr_type": "ArrayBuffer", "base64": "A"}]}]}),
),
format!("/data/commands/0/args/0{blob}"),
- Outcome::Marker(Declined::Undecodable),
+ Outcome::Marker(ImageDecline::Undecodable),
),
(
commands(
@@ -944,7 +946,7 @@ mod tests {
"args": [{"rr_type": "Uint8ClampedArray", "args": [[255, 0, 0, 255]]}, 1, 1]}, 0, 0]}]),
),
"/data/commands/0/args/0/args/0".to_owned(),
- Outcome::Marker(Declined::Unsupported),
+ Outcome::Marker(ImageDecline::Unsupported),
),
(
webgl(
@@ -952,12 +954,12 @@ mod tests {
json!([3553, 0, 6408, 1, 1, 0, 6408, 5121, {"rr_type": "Uint8Array", "args": [[255, 0, 0, 255]]}]),
),
"/data/args/8/args/0".to_owned(),
- Outcome::Marker(Declined::Unsupported),
+ Outcome::Marker(ImageDecline::Unsupported),
),
(
webgl("bufferData", json!([34962, buffer(b"vertices"), 35044])),
"/data/args/1/base64".to_owned(),
- Outcome::Marker(Declined::Unsupported),
+ Outcome::Marker(ImageDecline::Unsupported),
),
(
draw(json!({"rr_type": "HTMLImageElement", "src": "https://example.com/a.png"})),
diff --git a/rust/ai-research/replay-core-v4/src/scan/images.rs b/rust/ai-research/replay-core-v4/src/scan/images.rs
index 4d685bb1b406..a1247472bb56 100644
--- a/rust/ai-research/replay-core-v4/src/scan/images.rs
+++ b/rust/ai-research/replay-core-v4/src/scan/images.rs
@@ -1,5 +1,6 @@
//! Moves inline images out of a v4 block. Each raster data URI becomes an `image:v4` reference, and its decoded bytes
-//! go to the flush's image pack. An image the image scrub cannot read, or one over a cap, becomes a [`Declined`] marker.
+//! go to the flush's image pack. An image the image scrub cannot read, or one over a cap, becomes an [`ImageDecline`]
+//! marker.
//!
//! The scan is one pass over the colons, for `data:image/` in any case. A data URI looks the same in an attribute, a CSS `url()`, a
//! `srcset` candidate or stylesheet text, so the scan needs no knowledge of where it sits. Every other byte is
@@ -13,46 +14,10 @@ use base64::engine::general_purpose::{GeneralPurpose, GeneralPurposeConfig};
use base64::engine::DecodePaddingMode;
use base64::Engine;
-use crate::bytescan as scan;
-use crate::refs::{image_ref, reference_hash};
-
-pub const DECLINED_PREFIX: &str = "image:v4:declined:";
-
-/// Readers of the dataset accept only these exact markers.
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub enum Declined {
- Avif,
- OverLimit,
- Undecodable,
- Unsupported,
-}
-
-impl Declined {
- pub const ALL: [Declined; 4] = [
- Declined::Avif,
- Declined::OverLimit,
- Declined::Undecodable,
- Declined::Unsupported,
- ];
-
- pub const fn label(self) -> &'static str {
- match self {
- Declined::Avif => "avif",
- Declined::OverLimit => "over_limit",
- Declined::Undecodable => "undecodable",
- Declined::Unsupported => "unsupported",
- }
- }
+use posthog_replay_anonymizer::v4_image_refs::{image_ref, ImageDecline};
- pub const fn marker(self) -> &'static str {
- match self {
- Declined::Avif => "image:v4:declined:avif",
- Declined::OverLimit => "image:v4:declined:over_limit",
- Declined::Undecodable => "image:v4:declined:undecodable",
- Declined::Unsupported => "image:v4:declined:unsupported",
- }
- }
-}
+use crate::bytescan as scan;
+use crate::refs::reference_hash;
/// Base64 as a browser reads a data URI: the padding is optional, and the bits after the last byte do not count.
const FORGIVING_BASE64: GeneralPurpose = GeneralPurpose::new(
@@ -102,7 +67,7 @@ pub struct ImageSink<'k> {
pub images: Vec<Image>,
/// SVG data URIs, the only images left inline.
pub left_inline: u32,
- pub declined: [u32; Declined::ALL.len()],
+ pub declined: [u32; ImageDecline::ALL.len()],
}
impl<'k> ImageSink<'k> {
@@ -121,7 +86,7 @@ impl<'k> ImageSink<'k> {
total_bytes: 0,
images: Vec::new(),
left_inline: 0,
- declined: [0; Declined::ALL.len()],
+ declined: [0; ImageDecline::ALL.len()],
}
}
@@ -135,27 +100,27 @@ impl<'k> ImageSink<'k> {
pub(crate) fn take_base64(&mut self, payload: &[u8]) -> String {
if payload.len() > self.limits.max_image_bytes.div_ceil(3) * 4 {
- return self.decline(Declined::OverLimit);
+ return self.decline(ImageDecline::OverLimit);
}
match FORGIVING_BASE64.decode(payload) {
Ok(bytes) => self.take(bytes),
- Err(_) => self.decline(Declined::Undecodable),
+ Err(_) => self.decline(ImageDecline::Undecodable),
}
}
- pub(crate) fn decline(&mut self, reason: Declined) -> String {
+ pub(crate) fn decline(&mut self, reason: ImageDecline) -> String {
self.declined[reason as usize] += 1;
reason.marker().to_owned()
}
- fn store(&mut self, bytes: Vec<u8>) -> Result<String, Declined> {
+ fn store(&mut self, bytes: Vec<u8>) -> Result<String, ImageDecline> {
if bytes.len() > self.limits.max_image_bytes {
- return Err(Declined::OverLimit);
+ return Err(ImageDecline::OverLimit);
}
if is_avif_header(&bytes) {
- return Err(Declined::Avif);
+ return Err(ImageDecline::Avif);
}
- let mime = raster_type(&bytes).ok_or(Declined::Unsupported)?;
+ let mime = raster_type(&bytes).ok_or(ImageDecline::Unsupported)?;
let reference = image_ref(
self.team_id,
self.key_month,
@@ -168,7 +133,7 @@ impl<'k> ImageSink<'k> {
if self.images.len() >= self.limits.max_images
|| self.total_bytes + bytes.len() > self.limits.max_total_bytes
{
- return Err(Declined::OverLimit);
+ return Err(ImageDecline::OverLimit);
}
self.seen.insert(reference.clone());
self.total_bytes += bytes.len();
@@ -420,7 +385,7 @@ fn read_data_uri(bytes: &[u8], start: usize, sink: &mut ImageSink<'_>) -> DataUr
Meta::TooLong(meta) if meta.starts_with("image/") => {
let end = uri_end(bytes, start, meta_start);
return DataUri::Replaced {
- replacement: sink.decline(Declined::Undecodable),
+ replacement: sink.decline(ImageDecline::Undecodable),
end,
};
}
@@ -436,10 +401,10 @@ fn read_data_uri(bytes: &[u8], start: usize, sink: &mut ImageSink<'_>) -> DataUr
let end = uri_end(bytes, start, payload_start);
let replacement = match &bytes[payload_start..end] {
[] => return DataUri::NotImage,
- _ if subtype.starts_with("avif") => sink.decline(Declined::Avif),
+ _ if subtype.starts_with("avif") => sink.decline(ImageDecline::Avif),
raw => match percent_decoded(raw) {
Some(image) => sink.take(image),
- None => sink.decline(Declined::Undecodable),
+ None => sink.decline(ImageDecline::Undecodable),
},
};
return DataUri::Replaced { replacement, end };
@@ -458,9 +423,9 @@ fn read_data_uri(bytes: &[u8], start: usize, sink: &mut ImageSink<'_>) -> DataUr
}
// The label or the bytes can name AVIF, so both count.
let replacement = if subtype.starts_with("avif") {
- sink.decline(Declined::Avif)
+ sink.decline(ImageDecline::Avif)
} else if !readable {
- sink.decline(Declined::Undecodable)
+ sink.decline(ImageDecline::Undecodable)
} else if end == run_end {
sink.take_base64(&bytes[payload_start..end])
} else {
@@ -541,31 +506,12 @@ mod tests {
text.into_bytes()
}
- fn declined(reason: Declined) -> [u32; Declined::ALL.len()] {
- let mut counts = [0; Declined::ALL.len()];
+ fn declined(reason: ImageDecline) -> [u32; ImageDecline::ALL.len()] {
+ let mut counts = [0; ImageDecline::ALL.len()];
counts[reason as usize] = 1;
counts
}
- #[test]
- fn writes_only_the_markers_readers_accept() {
- assert_eq!(
- Declined::ALL.map(Declined::marker),
- [
- "image:v4:declined:avif",
- "image:v4:declined:over_limit",
- "image:v4:declined:undecodable",
- "image:v4:declined:unsupported",
- ]
- );
- for reason in Declined::ALL {
- assert_eq!(
- reason.marker(),
- format!("{DECLINED_PREFIX}{}", reason.label())
- );
- }
- }
-
#[test]
fn moves_images_out_of_every_slot_and_back_without_changing_a_byte() {
let png = data_uri("image/png", &png(b"png bytes"));
@@ -671,7 +617,7 @@ mod tests {
max_image_bytes: 999,
..limits
},
- Declined::OverLimit,
+ ImageDecline::OverLimit,
),
(
data_uri("image/png", &big),
@@ -679,7 +625,7 @@ mod tests {
max_images: 0,
..limits
},
- Declined::OverLimit,
+ ImageDecline::OverLimit,
),
(
data_uri("image/png", &big),
@@ -687,23 +633,23 @@ mod tests {
max_total_bytes: 999,
..limits
},
- Declined::OverLimit,
+ ImageDecline::OverLimit,
),
(
"data:image/png;base64,iVBORw0KG".to_owned(),
limits,
- Declined::Undecodable,
+ ImageDecline::Undecodable,
),
(
"data:image/png;base64,iVBO=Rw0KGgo".to_owned(),
limits,
- Declined::Undecodable,
+ ImageDecline::Undecodable,
),
// A CSS escape for `/`, with the backslash doubled by JSON.
(
format!(r"{wrapped}\\2f {}", STANDARD.encode([8; 30])),
limits,
- Declined::Undecodable,
+ ImageDecline::Undecodable,
),
(
format!(
@@ -712,32 +658,32 @@ mod tests {
STANDARD.encode(png(b"x"))
),
limits,
- Declined::Undecodable,
+ ImageDecline::Undecodable,
),
(
data_uri("image/ttf", b"\x00\x01\x00\x00font"),
limits,
- Declined::Unsupported,
+ ImageDecline::Unsupported,
),
(
data_uri("image/x-icon", b"\x00\x00\x01\x00icon"),
limits,
- Declined::Unsupported,
+ ImageDecline::Unsupported,
),
(
data_uri("image/bmp", b"BMbitmap"),
limits,
- Declined::Unsupported,
+ ImageDecline::Unsupported,
),
(
data_uri("image/png", b"not an image"),
limits,
- Declined::Unsupported,
+ ImageDecline::Unsupported,
),
(
"data:image/png;utf8,abc".to_owned(),
limits,
- Declined::Unsupported,
+ ImageDecline::Unsupported,
),
];
for (uri, limits, reason) in cases {
@@ -776,7 +722,7 @@ mod tests {
),
"{event}"
);
- assert_eq!(images.declined, [0; Declined::ALL.len()], "{event}");
+ assert_eq!(images.declined, [0; ImageDecline::ALL.len()], "{event}");
assert_eq!(
images.left_inline,
u32::from(event.contains("svg")),
@@ -787,7 +733,7 @@ mod tests {
#[test]
fn replaces_every_avif_data_uri_with_the_marker() {
- let marker = Declined::Avif.marker();
+ let marker = ImageDecline::Avif.marker();
let avif = STANDARD.encode(b"\x00\x00\x00\x1cftypavif\x00\x00\x00\x00avifmif1miaf");
// AVIF as a compatible brand only, under the major brand `mif1`.
let compatible = STANDARD.encode(b"\x00\x00\x00\x18ftypmif1\x00\x00\x00\x00miafavif");
@@ -811,7 +757,7 @@ mod tests {
);
assert_eq!(
(images.images.len(), images.declined),
- (0, declined(Declined::Avif)),
+ (0, declined(ImageDecline::Avif)),
"{uri}"
);
}
@@ -865,12 +811,12 @@ mod tests {
String::from_utf8_lossy(&out),
format!(
r#"["{reference}","{}","{reference}"]"#,
- Declined::OverLimit.marker()
+ ImageDecline::OverLimit.marker()
)
);
assert_eq!(
(images.images.len(), images.declined),
- (1, declined(Declined::OverLimit))
+ (1, declined(ImageDecline::OverLimit))
);
}
@@ -888,7 +834,7 @@ mod tests {
let mut images = sink(ImageLimits::default());
assert_eq!(
extract_images(unbroken.as_bytes(), &mut images).as_ref(),
- Declined::Undecodable.marker().as_bytes()
+ ImageDecline::Undecodable.marker().as_bytes()
);
assert!(started.elapsed() < std::time::Duration::from_secs(5));
}
diff --git a/rust/ai-research/replay-core-v4/src/url_policy.rs b/rust/ai-research/replay-core-v4/src/url_policy.rs
index fe0fe5495a5f..ba50b1f9500e 100644
--- a/rust/ai-research/replay-core-v4/src/url_policy.rs
+++ b/rust/ai-research/replay-core-v4/src/url_policy.rs
@@ -321,16 +321,22 @@ impl Decline {
/// The metric label. Stable, because a dashboard and an alert both key on it.
pub fn label(self) -> &'static str {
- match self {
- Decline::TooLong => "too_long",
- Decline::NotAbsolute => "not_absolute",
- Decline::BadScheme => "bad_scheme",
- Decline::BadPort => "bad_port",
- Decline::NoHost => "no_host",
- Decline::NonPublicHost => "non_public_host",
- Decline::Credential => "credential",
- Decline::InvalidQuery => "invalid_query",
- Decline::TrackingBeacon => "tracking_beacon",
+ posthog_replay_anonymizer::Decline::from(self).label()
+ }
+}
+
+impl From<Decline> for posthog_replay_anonymizer::Decline {
+ fn from(decline: Decline) -> Self {
+ match decline {
+ Decline::TooLong => Self::TooLong,
+ Decline::NotAbsolute => Self::NotAbsolute,
+ Decline::BadScheme => Self::BadScheme,
+ Decline::BadPort => Self::BadPort,
+ Decline::NoHost => Self::NoHost,
+ Decline::NonPublicHost => Self::NonPublicHost,
+ Decline::Credential => Self::Credential,
+ Decline::InvalidQuery => Self::InvalidQuery,
+ Decline::TrackingBeacon => Self::TrackingBeacon,
}
}
}
diff --git a/rust/ai-research/replay-core-v4/tests/v4_vector.rs b/rust/ai-research/replay-core-v4/tests/v4_vector.rs
index 924ebdd121fa..3fb0b197d50a 100644
--- a/rust/ai-research/replay-core-v4/tests/v4_vector.rs
+++ b/rust/ai-research/replay-core-v4/tests/v4_vector.rs
@@ -1,9 +1,10 @@
use base64::engine::general_purpose::STANDARD;
use base64::Engine;
+use posthog_replay_anonymizer::v4_image_refs::{image_ref, image_url_ref};
use replay_core_v4::envelope::{
open_frame, seal_frame_with_nonce, seal_json_envelope_with_nonce, Codec, Context, Version,
};
-use replay_core_v4::refs::{image_ref, image_url_ref, reference_hash, url_fingerprint};
+use replay_core_v4::refs::{reference_hash, url_fingerprint};
use replay_core_v4::seal::{
image_content_key, image_url_key, open_session_key, seal_session_key_with_nonce,
url_fingerprint_key, SealedKey, WrappingContext,
diff --git a/rust/ai-research/replay-recorder-v4/Cargo.toml b/rust/ai-research/replay-recorder-v4/Cargo.toml
index d10574b8ab99..5586a2d1ed91 100644
--- a/rust/ai-research/replay-recorder-v4/Cargo.toml
+++ b/rust/ai-research/replay-recorder-v4/Cargo.toml
@@ -17,6 +17,7 @@ common-kafka = { path = "../../common/kafka" }
common-metrics = { path = "../../common/metrics" }
common-redis = { path = "../../common/redis" }
lifecycle = { path = "../../common/lifecycle" }
+posthog-replay-anonymizer = { path = "../../replay-anonymizer", default-features = false }
replay-core-v4 = { path = "../replay-core-v4" }
aho-corasick = { workspace = true }
diff --git a/rust/ai-research/replay-recorder-v4/src/rewrite.rs b/rust/ai-research/replay-recorder-v4/src/rewrite.rs
index 461ed6acc513..b7f89da8d7c0 100644
--- a/rust/ai-research/replay-recorder-v4/src/rewrite.rs
+++ b/rust/ai-research/replay-recorder-v4/src/rewrite.rs
@@ -14,12 +14,13 @@ use std::collections::{BTreeMap, HashMap, HashSet};
use aho_corasick::{AhoCorasick, AhoCorasickKind, MatchKind};
use metrics::counter;
+use posthog_replay_anonymizer::v4_image_refs::{
+ url_declined_marker, ImageDecline, IMAGE_DECLINED_PREFIX,
+};
use replay_core_v4::bytescan::{canonical_strings, unescape, write_json_string, Span};
use replay_core_v4::inspect::css::{escape_in_identifier, rewrite_functions, CssRewrite};
use replay_core_v4::inspect::dom::{for_each_srcset_candidate, Srcset};
-use replay_core_v4::scan::images::{
- extract_images, Declined, ImageLimits, ImageSink, DECLINED_PREFIX,
-};
+use replay_core_v4::scan::images::{extract_images, ImageLimits, ImageSink};
use replay_core_v4::scan::tokens::{Token, Tokens};
use replay_core_v4::seal::{image_content_key, image_url_key};
use replay_core_v4::url_policy::is_remote;
@@ -27,11 +28,6 @@ use replay_core_v4::url_policy::is_remote;
use crate::scrub::{Scrub, Scrubber};
use crate::urls::url_reference;
-/// What a declined URL becomes. The reason is the policy's metric label.
-pub fn declined_marker(reason: &str) -> String {
- format!("imageurl:v4:declined:{reason}")
-}
-
/// A byte that a URL can go on with. A match next to one is part of a longer URL, so it stays as it is. A comma, a
/// parenthesis or a quote can be in a URL too, but in a srcset or a CSS `url()` it ends one.
fn continues_url(byte: u8) -> bool {
@@ -258,7 +254,7 @@ pub struct PassCounters {
srcset_rewrites: BTreeMap<&'static str, u64>,
css_rewrites: BTreeMap<(&'static str, &'static str), u64>,
url_rewrites: BTreeMap<(&'static str, &'static str), u64>,
- pub removed_images: [u64; Declined::ALL.len()],
+ pub removed_images: [u64; ImageDecline::ALL.len()],
}
impl PassCounters {
@@ -283,7 +279,7 @@ impl PassCounters {
)
.increment(count);
}
- for (reason, &count) in Declined::ALL.iter().zip(&self.removed_images) {
+ for (reason, &count) in ImageDecline::ALL.iter().zip(&self.removed_images) {
counter!("ai_research_v4_recorder_removed_images_total", "reason" => reason.label())
.increment(count);
}
@@ -333,7 +329,7 @@ impl UrlRewrite {
patterns.push(json_string_contents(url));
let (replacement, declined) = match url_reference(url, team_id, key_month, &url_key) {
Ok((_, reference)) => (reference, None),
- Err(decline) => (declined_marker(decline.label()), Some(decline.label())),
+ Err(decline) => (url_declined_marker(decline.into()), Some(decline.label())),
};
targets.push(Target {
url: url.clone(),
@@ -406,7 +402,7 @@ impl UrlRewrite {
let reference = match url_reference(absolute, team_id, key_month, &url_key)
{
Ok((_, reference)) => reference,
- Err(decline) => declined_marker(decline.label()),
+ Err(decline) => url_declined_marker(decline.into()),
};
rewrite.replacement.replacen(
&format!("\"{absolute}\""),
@@ -880,7 +876,7 @@ fn rebuilt_srcset(
}
};
for_each_srcset_candidate(text, |url, descriptors| {
- if url.starts_with(DECLINED_PREFIX) {
+ if url.starts_with(IMAGE_DECLINED_PREFIX) {
removed = true;
} else if !is_remote(url) {
keep(url, descriptors);
@@ -957,8 +953,7 @@ mod tests {
assert_eq!(
String::from_utf8(rewrite.apply(event.as_bytes(), true).into_owned()).unwrap(),
format!(
- r#"{{"srcset":"{ra} 1x,\n{rb} 2x,\t{} 3x"}}"#,
- declined_marker("non_public_host")
+ r#"{{"srcset":"{ra} 1x,\n{rb} 2x,\timageurl:v4:declined:non_public_host 3x"}}"#
)
);
assert!(rewrite.finish().safe);
@@ -981,7 +976,7 @@ mod tests {
// A message past its image cap stores the PNG as `over_limit`, where a fresh sink takes it.
let stored = format!(
r#"{{"srcset":"{} 1x, {svg} 2x"}}"#,
- Declined::OverLimit.marker()
+ ImageDecline::OverLimit.marker()
);
assert_eq!(
String::from_utf8(rewrite.apply(stored.as_bytes(), true).into_owned()).unwrap(),
@@ -997,10 +992,7 @@ mod tests {
let mut rewrite = UrlRewrite::new(&[url.to_owned()], &[], &[], 7, "2026-10", &TEAM_KEY);
assert_eq!(
String::from_utf8(rewrite.apply(&canonical, true).into_owned()).unwrap(),
- format!(
- r#"{{"src":"{}","alt":"a\nb"}}"#,
- declined_marker("credential")
- )
+ r#"{"src":"imageurl:v4:declined:credential","alt":"a\nb"}"#
);
assert!(rewrite.finish().safe);
// An event capture wrote is left as it is.
@@ -1031,7 +1023,7 @@ mod tests {
let out = rewrite(&[url], &format!(r#"{{"src":"{url}"}}"#));
assert_eq!(
out,
- format!(r#"{{"src":"{}"}}"#, declined_marker(reason)),
+ format!(r#"{{"src":"imageurl:v4:declined:{reason}"}}"#),
"{url}"
);
}
diff --git a/rust/ai-research/replay-recorder-v4/src/scrub.rs b/rust/ai-research/replay-recorder-v4/src/scrub.rs
index 58b609cff08a..3eeaffd2dd3f 100644
--- a/rust/ai-research/replay-recorder-v4/src/scrub.rs
+++ b/rust/ai-research/replay-recorder-v4/src/scrub.rs
@@ -1,6 +1,6 @@
//! The fallback for a block whose remote URLs the writer could not all name or replace. Each remote URL token left in
-//! the stored bytes becomes [`UNCHECKED_MARKER`], so no URL the policy would refuse is stored, and the block is kept
-//! instead of dropped.
+//! the stored bytes becomes [`URL_UNCHECKED_MARKER`], so no URL the policy would refuse is stored, and the block is
+//! kept instead of dropped.
//!
//! The scrub reads the bytes one character at a time through their JSON escapes and CSS escapes, so it never splits an
//! escape and the stored JSON stays valid. A URL token ends at whitespace or at the end of its JSON string. A URL that
@@ -11,9 +11,7 @@
use std::borrow::Cow;
use aho_corasick::{AhoCorasick, AhoCorasickBuilder, MatchKind};
-
-/// What a scrubbed URL token becomes.
-pub const UNCHECKED_MARKER: &str = "imageurl:v4:declined:unchecked";
+use posthog_replay_anonymizer::v4_image_refs::URL_UNCHECKED_MARKER;
/// Which URL tokens the scrub replaces.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
@@ -384,7 +382,7 @@ impl Scrubber {
}
if start >= copied_to {
out.extend_from_slice(&bytes[copied_to..start]);
- out.extend_from_slice(UNCHECKED_MARKER.as_bytes());
+ out.extend_from_slice(URL_UNCHECKED_MARKER.as_bytes());
replaced += 1;
}
copied_to = end;
@@ -441,7 +439,7 @@ mod tests {
#[test]
fn replaces_every_remote_url_token_and_keeps_the_json_around_it() {
- let m = UNCHECKED_MARKER;
+ let m = "imageurl:v4:declined:unchecked";
for (input, expected) in [
(
r#"{"src":"https://cdn.example.com/a.png?x=1"}"#.to_owned(),
@@ -500,7 +498,7 @@ mod tests {
#[test]
fn replaces_the_tokens_that_hold_a_host_in_any_form() {
- let m = UNCHECKED_MARKER;
+ let m = "imageurl:v4:declined:unchecked";
let (out, count) = scrub(
Scrub::Hosts(vec!["internal.example.com".to_owned()]),
r#"{"style":"url(https\\3a \\2f \\2f Internal.example.com\\2f a.png)","alt":"see internal.example.com/a'b?token=x\nnow","title":"other.example.com"}"#,
@@ -514,7 +512,7 @@ mod tests {
#[test]
fn matches_a_host_only_as_the_browser_reads_it_inside_a_string() {
- let m = UNCHECKED_MARKER;
+ let m = "imageurl:v4:declined:unchecked";
for (host, input, expected) in [
(
"10.0.0.7",
diff --git a/rust/ai-research/replay-recorder-v4/src/urls.rs b/rust/ai-research/replay-recorder-v4/src/urls.rs
index 070d23dcea15..d10f266f3bcc 100644
--- a/rust/ai-research/replay-recorder-v4/src/urls.rs
+++ b/rust/ai-research/replay-recorder-v4/src/urls.rs
@@ -4,8 +4,9 @@ use std::collections::{BTreeMap, HashSet};
use std::time::Duration;
use metrics::counter;
+use posthog_replay_anonymizer::v4_image_refs::image_url_ref;
use replay_core_v4::keys::batch::{FlushKeys, SessionId};
-use replay_core_v4::refs::{image_url_ref, reference_hash};
+use replay_core_v4::refs::reference_hash;
use replay_core_v4::seal::image_url_key;
use replay_core_v4::url_policy::{
is_public_host, is_tracking_beacon, try_canonicalize, CanonicalUrl, Decline,
diff --git a/rust/ai-research/replay-recorder-v4/src/write.rs b/rust/ai-research/replay-recorder-v4/src/write.rs
index 35c5e4ce9c88..1eb6852c69a5 100644
--- a/rust/ai-research/replay-recorder-v4/src/write.rs
+++ b/rust/ai-research/replay-recorder-v4/src/write.rs
@@ -10,6 +10,7 @@ use std::sync::Arc;
use chrono::{DateTime, Utc};
use futures::stream::{self, StreamExt, TryStreamExt};
use metrics::counter;
+use posthog_replay_anonymizer::v4_image_refs::ImageDecline;
use rand::Rng;
use replay_core_v4::bytescan::write_json_string;
use replay_core_v4::envelope::{seal_frame, seal_json_envelope, Codec, Context, Version};
@@ -19,7 +20,7 @@ use replay_core_v4::inspect::stats::MAX_URL_FINGERPRINTS;
use replay_core_v4::keys::batch::{FlushKeys, SessionId, SessionKey as Key};
use replay_core_v4::refs::url_fingerprint;
use replay_core_v4::scan::cv::extract_event_images;
-use replay_core_v4::scan::images::{Declined, Image, ImageLimits, ImageSink};
+use replay_core_v4::scan::images::{Image, ImageLimits, ImageSink};
use replay_core_v4::seal::{image_content_key, url_fingerprint_key};
use tokio::sync::Semaphore;
@@ -134,7 +135,7 @@ fn prepare_block(
let mut images = Vec::new();
let mut kept = HashSet::new();
let mut counts = WriterCounts::default();
- let mut removed_images = [0u64; Declined::ALL.len()];
+ let mut removed_images = [0u64; ImageDecline::ALL.len()];
for message in &block.messages {
let mut window = Vec::new();
write_json_string(
@@ -849,8 +850,7 @@ mod tests {
.1;
assert!(
lines.contains(&format!(
- r#""src":"{}","srcset":"{reference} 2x""#,
- crate::rewrite::declined_marker("credential")
+ r#""src":"imageurl:v4:declined:credential","srcset":"{reference} 2x""#
)),
"{lines}"
);
@@ -864,7 +864,7 @@ mod tests {
// A URL too long to fetch is declined, and the block is kept.
let long_lines = stored_lines(&long(16 * 1024), &[]);
assert!(
- long_lines.contains(&crate::rewrite::declined_marker("too_long")),
+ long_lines.contains("imageurl:v4:declined:too_long"),
"{long_lines}"
);
// Keys written with escapes hide no URL.
@@ -883,8 +883,7 @@ mod tests {
.to_string();
let lines = stored_lines(&events, &[]);
assert!(
- lines.contains(&crate::rewrite::declined_marker("credential"))
- && !lines.contains("secret"),
+ lines.contains("imageurl:v4:declined:credential") && !lines.contains("secret"),
"{lines}"
);
}
@@ -899,7 +898,7 @@ mod tests {
] {
let lines = stored_lines(&events, &[]);
assert!(
- lines.contains(&crate::rewrite::declined_marker("credential"))
+ lines.contains("imageurl:v4:declined:credential")
&& !lines.contains("X-Amz-Signature"),
"{lines}"
);
@@ -939,7 +938,7 @@ mod tests {
lines.contains("imageurl:v4:7:2026-10:"),
"named URLs keep their references"
);
- assert!(lines.contains(crate::scrub::UNCHECKED_MARKER));
+ assert!(lines.contains("imageurl:v4:declined:unchecked"));
assert!(!lines.contains("https://"));
}
@@ -980,10 +979,7 @@ mod tests {
r#"image-set(url({}) 2x)"#,
reference("https://cdn.example.com/2x.png")
),
- format!(
- r#"url(\"{}\")"#,
- crate::rewrite::declined_marker("credential")
- ),
+ r#"url(\"imageurl:v4:declined:credential\")"#.to_owned(),
] {
assert!(lines.contains(&expected), "{expected} in {lines}");
}
@@ -1184,8 +1180,7 @@ mod tests {
);
assert!(
custom.contains(&format!(
- r#""srcset":"{reference} 1x, {} 2x""#,
- crate::rewrite::declined_marker("non_public_host")
+ r#""srcset":"{reference} 1x, imageurl:v4:declined:non_public_host 2x""#
)),
"{custom}"
);
@@ -1243,10 +1238,7 @@ mod tests {
lines.contains(&format!(r#""src":"{reference}""#)),
"{lines}"
);
- assert!(lines.contains(&format!(
- r#""srcset":"{} 2x""#,
- crate::rewrite::declined_marker("credential")
- )));
+ assert!(lines.contains(r#""srcset":"imageurl:v4:declined:credential 2x""#));
assert!(!lines.contains("cdn.example.com"));
}
@@ -1348,7 +1340,7 @@ mod tests {
keys.sessions.insert(id, key);
}
keys.team_months.insert(7, [6; 32]);
- let old_ref = replay_core_v4::refs::image_ref(
+ let old_ref = posthog_replay_anonymizer::v4_image_refs::image_ref(
7,
"2026-10",
&replay_core_v4::refs::reference_hash(&image_content_key(&[6; 32]), &png(b"old")),
diff --git a/rust/replay-anonymizer/Cargo.toml b/rust/replay-anonymizer/Cargo.toml
index a34ef9186d5f..c3fea1242595 100644
--- a/rust/replay-anonymizer/Cargo.toml
+++ b/rust/replay-anonymizer/Cargo.toml
@@ -1,6 +1,6 @@
[package]
name = "posthog-replay-anonymizer"
-version = "0.12.1"
+version = "0.13.0"
edition = "2021"
description = "PII scrubber for rrweb session-replay events: text/URL redaction, image collection for an out-of-band scrub, and canvas/cv neutralization"
license = "MIT"
diff --git a/rust/replay-anonymizer/README.md b/rust/replay-anonymizer/README.md
index 5793ad1201f8..7b1809aee0b6 100644
--- a/rust/replay-anonymizer/README.md
+++ b/rust/replay-anonymizer/README.md
@@ -6,6 +6,7 @@ Consumers:
- `replay-anonymizer-node` (this workspace) wraps it as a Neon addon for the Node ingestion workers.
- The MLHog training pipelines consume it from crates.io.
+- The v4 recorder crates (`replay-core-v4`, `replay-recorder-v4`) write their image references and decline markers with the `v4_image_refs` module, by path and with `default-features = false`. A reader of v4 blocks parses those strings with `v4_image_refs::parse`, so the writer and the reader share one definition.
Behavior is pinned by the JSON fixtures under `tests/fixtures/`, shared with the Node addon's Jest suite. Scrubbing operates on untrusted input; the public entry points contain panics and convert them to errors, so callers fail closed (drop the message) without their own `catch_unwind`. Under `panic = "abort"` that backstop cannot run — builds that must fail closed need `panic = "unwind"`.
diff --git a/rust/replay-anonymizer/src/lib.rs b/rust/replay-anonymizer/src/lib.rs
index 95fe7dc553e6..c50708bbb690 100644
--- a/rust/replay-anonymizer/src/lib.rs
+++ b/rust/replay-anonymizer/src/lib.rs
@@ -19,10 +19,10 @@
//! The stable crates.io surface is what this page documents: [`AllowLists`], [`Ctx`], the event
//! entry points ([`anonymize_message`], [`anonymize_event`], [`anonymize_line`] and friends), the
//! byte-buffer snapshot pipeline ([`anonymize_kafka_payload`] and friends), the rrweb routing
-//! constants in [`event`], and — behind the default `typed-parse` feature — the typed rrweb parse
-//! in [`typed`] ([`parse_scrubbed_event`]). Everything `#[doc(hidden)]` stays `pub` only for this
-//! workspace (the Node addon, the parity tests) — it is internal and may change or disappear in
-//! any release.
+//! constants in [`event`], the v4 image references and markers in [`v4_image_refs`], and — behind
+//! the default `typed-parse` feature — the typed rrweb parse in [`typed`]
+//! ([`parse_scrubbed_event`]). Everything `#[doc(hidden)]` stays `pub` only for this workspace
+//! (the Node addon, the parity tests) — it is internal and may change or disappear in any release.
pub mod allow_lists;
#[doc(hidden)]
@@ -67,6 +67,7 @@ pub mod url;
pub mod url_collect;
#[doc(hidden)]
pub mod url_policy;
+pub mod v4_image_refs;
#[doc(hidden)]
pub mod value;
diff --git a/rust/replay-anonymizer/src/url_policy.rs b/rust/replay-anonymizer/src/url_policy.rs
index 955b2749195c..9b04c3dd0d84 100644
--- a/rust/replay-anonymizer/src/url_policy.rs
+++ b/rust/replay-anonymizer/src/url_policy.rs
@@ -312,6 +312,18 @@ pub enum Decline {
}
impl Decline {
+ pub const ALL: [Decline; 9] = [
+ Decline::TooLong,
+ Decline::NotAbsolute,
+ Decline::BadScheme,
+ Decline::BadPort,
+ Decline::NoHost,
+ Decline::NonPublicHost,
+ Decline::Credential,
+ Decline::InvalidQuery,
+ Decline::TrackingBeacon,
+ ];
+
/// True for a URL the policy refuses because nobody wants it fetched, not because it is
/// malformed or unsafe. A consumer that reads jobs from a queue drops only that job for such a
/// decline, and rejects the whole record for every other one.
@@ -319,7 +331,8 @@ impl Decline {
matches!(self, Decline::TrackingBeacon)
}
- /// The metric label. Stable, because a dashboard and an alert both key on it.
+ /// The metric label, and the reason in a v4 URL decline marker. Stable, because a dashboard, an
+ /// alert and a reader of v4 blocks all key on it.
pub fn label(self) -> &'static str {
match self {
Decline::TooLong => "too_long",
diff --git a/rust/replay-anonymizer/src/v4_image_refs.rs b/rust/replay-anonymizer/src/v4_image_refs.rs
new file mode 100644
index 000000000000..3f6dad05c43f
--- /dev/null
+++ b/rust/replay-anonymizer/src/v4_image_refs.rs
@@ -0,0 +1,360 @@
+//! The references and decline markers that stand in for an image in a v4 replay block. A captured
+//! page can hold text that looks like a reference, so [`parse`] accepts only the exact strings the
+//! v4 recorder writes.
+
+use std::fmt;
+
+use crate::url_policy::Decline;
+
+/// Starts a reference whose hash names the image bytes. The recorder stores the bytes.
+pub const IMAGE_PREFIX: &str = "image:v4:";
+/// Starts a reference whose hash names the image URL. The image fetcher stores the bytes.
+pub const IMAGE_URL_PREFIX: &str = "imageurl:v4:";
+/// The length v3 truncates a reference hash to: 132 bits of the HMAC, in URL-safe base64.
+pub const HASH_CHARS: usize = 22;
+/// What a remote URL becomes when the recorder replaces it without a check by the URL policy.
+pub const URL_UNCHECKED_MARKER: &str = "imageurl:v4:declined:unchecked";
+pub const IMAGE_DECLINED_PREFIX: &str = "image:v4:declined:";
+
+const DECLINED: &str = "declined:";
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub enum ImageDecline {
+ /// The dataset keeps no AVIF image, inline or fetched.
+ Avif,
+ OverLimit,
+ Undecodable,
+ /// Bytes of another format than PNG, JPEG, GIF or WebP, the formats the image scrub reads.
+ Unsupported,
+}
+
+impl ImageDecline {
+ pub const ALL: [ImageDecline; 4] = [
+ ImageDecline::Avif,
+ ImageDecline::OverLimit,
+ ImageDecline::Undecodable,
+ ImageDecline::Unsupported,
+ ];
+
+ /// The metric label, which also ends the marker.
+ pub const fn label(self) -> &'static str {
+ match self {
+ ImageDecline::Avif => "avif",
+ ImageDecline::OverLimit => "over_limit",
+ ImageDecline::Undecodable => "undecodable",
+ ImageDecline::Unsupported => "unsupported",
+ }
+ }
+
+ pub const fn marker(self) -> &'static str {
+ match self {
+ ImageDecline::Avif => "image:v4:declined:avif",
+ ImageDecline::OverLimit => "image:v4:declined:over_limit",
+ ImageDecline::Undecodable => "image:v4:declined:undecodable",
+ ImageDecline::Unsupported => "image:v4:declined:unsupported",
+ }
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub enum RefKind {
+ Image,
+ ImageUrl,
+}
+
+impl RefKind {
+ pub const fn prefix(self) -> &'static str {
+ match self {
+ RefKind::Image => IMAGE_PREFIX,
+ RefKind::ImageUrl => IMAGE_URL_PREFIX,
+ }
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub struct Ref<'a> {
+ pub kind: RefKind,
+ /// Positive, because a PostHog team id starts at 1.
+ pub team_id: i64,
+ /// The UTC month of the write, as `YYYY-MM`.
+ pub key_month: &'a str,
+ pub hash: &'a str,
+}
+
+/// A marker for an image that the dataset does not keep. No bytes stand behind it.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub enum Declined {
+ Image(ImageDecline),
+ Url(Decline),
+ UrlUnchecked,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub enum Parsed<'a> {
+ Ref(Ref<'a>),
+ Declined(Declined),
+}
+
+impl fmt::Display for Ref<'_> {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ write!(
+ f,
+ "{}{}:{}:{}",
+ self.kind.prefix(),
+ self.team_id,
+ self.key_month,
+ self.hash
+ )
+ }
+}
+
+impl fmt::Display for Declined {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ match self {
+ Declined::Image(reason) => f.write_str(reason.marker()),
+ Declined::Url(reason) => write!(f, "{IMAGE_URL_PREFIX}{DECLINED}{}", reason.label()),
+ Declined::UrlUnchecked => f.write_str(URL_UNCHECKED_MARKER),
+ }
+ }
+}
+
+impl fmt::Display for Parsed<'_> {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ match self {
+ Parsed::Ref(reference) => reference.fmt(f),
+ Parsed::Declined(declined) => declined.fmt(f),
+ }
+ }
+}
+
+pub fn image_ref(team_id: i64, key_month: &str, hash: &str) -> String {
+ Ref {
+ kind: RefKind::Image,
+ team_id,
+ key_month,
+ hash,
+ }
+ .to_string()
+}
+
+pub fn image_url_ref(team_id: i64, key_month: &str, hash: &str) -> String {
+ Ref {
+ kind: RefKind::ImageUrl,
+ team_id,
+ key_month,
+ hash,
+ }
+ .to_string()
+}
+
+pub fn url_declined_marker(reason: Decline) -> String {
+ Declined::Url(reason).to_string()
+}
+
+/// Reads a reference or a marker. Any other string is `None`, also one that differs from a string
+/// the recorder writes only in letter case, a leading zero, whitespace or a trailing byte.
+pub fn parse(s: &str) -> Option<Parsed<'_>> {
+ if let Some(declined) = parse_declined(s) {
+ return Some(Parsed::Declined(declined));
+ }
+ let (kind, rest) = match s.strip_prefix(IMAGE_PREFIX) {
+ Some(rest) => (RefKind::Image, rest),
+ None => (RefKind::ImageUrl, s.strip_prefix(IMAGE_URL_PREFIX)?),
+ };
+ let mut fields = rest.split(':');
+ let (team_id, key_month, hash) = (fields.next()?, fields.next()?, fields.next()?);
+ if fields.next().is_some() || !is_key_month(key_month) || !is_hash(hash) {
+ return None;
+ }
+ Some(Parsed::Ref(Ref {
+ kind,
+ team_id: parse_team_id(team_id)?,
+ key_month,
+ hash,
+ }))
+}
+
+fn parse_declined(s: &str) -> Option<Declined> {
+ if s == URL_UNCHECKED_MARKER {
+ return Some(Declined::UrlUnchecked);
+ }
+ if let Some(reason) = ImageDecline::ALL
+ .into_iter()
+ .find(|reason| reason.marker() == s)
+ {
+ return Some(Declined::Image(reason));
+ }
+ let label = s.strip_prefix(IMAGE_URL_PREFIX)?.strip_prefix(DECLINED)?;
+ Decline::ALL
+ .into_iter()
+ .find(|reason| reason.label() == label)
+ .map(Declined::Url)
+}
+
+/// Digits alone, as `i64` writes a positive number: no sign and no leading zero.
+fn parse_team_id(digits: &str) -> Option<i64> {
+ if digits.starts_with('0') || !digits.bytes().all(|b| b.is_ascii_digit()) {
+ return None;
+ }
+ digits.parse().ok()
+}
+
+fn is_key_month(month: &str) -> bool {
+ let [y0, y1, y2, y3, b'-', m0, m1] = *month.as_bytes() else {
+ return false;
+ };
+ [y0, y1, y2, y3].iter().all(u8::is_ascii_digit)
+ && matches!((m0, m1), (b'0', b'1'..=b'9') | (b'1', b'0'..=b'2'))
+}
+
+fn is_hash(hash: &str) -> bool {
+ hash.len() == HASH_CHARS
+ && hash
+ .bytes()
+ .all(|b| b.is_ascii_alphanumeric() || b == b'-' || b == b'_')
+}
+
+#[cfg(test)]
+mod tests {
+ use std::collections::HashSet;
+
+ use super::*;
+
+ const HASH: &str = "AbCdEfGhIjKlMnOpQr-_09";
+
+ #[test]
+ fn every_string_the_recorder_writes_round_trips() {
+ let reference = |kind| {
+ Parsed::Ref(Ref {
+ kind,
+ team_id: 7,
+ key_month: "2026-10",
+ hash: HASH,
+ })
+ };
+ let image = |reason| Parsed::Declined(Declined::Image(reason));
+ let url = |reason| Parsed::Declined(Declined::Url(reason));
+ let vectors = [
+ (
+ "image:v4:7:2026-10:AbCdEfGhIjKlMnOpQr-_09",
+ reference(RefKind::Image),
+ ),
+ (
+ "imageurl:v4:7:2026-10:AbCdEfGhIjKlMnOpQr-_09",
+ reference(RefKind::ImageUrl),
+ ),
+ ("image:v4:declined:avif", image(ImageDecline::Avif)),
+ (
+ "image:v4:declined:over_limit",
+ image(ImageDecline::OverLimit),
+ ),
+ (
+ "image:v4:declined:undecodable",
+ image(ImageDecline::Undecodable),
+ ),
+ (
+ "image:v4:declined:unsupported",
+ image(ImageDecline::Unsupported),
+ ),
+ (
+ "imageurl:v4:declined:unchecked",
+ Parsed::Declined(Declined::UrlUnchecked),
+ ),
+ ("imageurl:v4:declined:too_long", url(Decline::TooLong)),
+ (
+ "imageurl:v4:declined:not_absolute",
+ url(Decline::NotAbsolute),
+ ),
+ ("imageurl:v4:declined:bad_scheme", url(Decline::BadScheme)),
+ ("imageurl:v4:declined:bad_port", url(Decline::BadPort)),
+ ("imageurl:v4:declined:no_host", url(Decline::NoHost)),
+ (
+ "imageurl:v4:declined:non_public_host",
+ url(Decline::NonPublicHost),
+ ),
+ ("imageurl:v4:declined:credential", url(Decline::Credential)),
+ (
+ "imageurl:v4:declined:invalid_query",
+ url(Decline::InvalidQuery),
+ ),
+ (
+ "imageurl:v4:declined:tracking_beacon",
+ url(Decline::TrackingBeacon),
+ ),
+ ];
+ for (written, parsed) in vectors {
+ assert_eq!(parse(written), Some(parsed), "{written}");
+ assert_eq!(parsed.to_string(), written);
+ }
+ assert_eq!(image_ref(7, "2026-10", HASH), vectors[0].0);
+ assert_eq!(image_url_ref(7, "2026-10", HASH), vectors[1].0);
+
+ let every_marker: HashSet<String> = ImageDecline::ALL
+ .map(Declined::Image)
+ .into_iter()
+ .chain(Decline::ALL.map(Declined::Url))
+ .chain([Declined::UrlUnchecked])
+ .map(|declined| declined.to_string())
+ .collect();
+ let vector_markers: HashSet<String> = vectors
+ .iter()
+ .filter(|(_, parsed)| matches!(parsed, Parsed::Declined(_)))
+ .map(|(written, _)| written.to_string())
+ .collect();
+ assert_eq!(every_marker, vector_markers);
+ for reason in ImageDecline::ALL {
+ assert_eq!(
+ reason.marker(),
+ format!("{IMAGE_DECLINED_PREFIX}{}", reason.label())
+ );
+ }
+ }
+
+ #[test]
+ fn rejects_near_misses() {
+ let near_misses = [
+ String::new(),
+ "image:v4:".to_owned(),
+ format!("image:v3:7:2026-10:{HASH}"),
+ format!("Image:v4:7:2026-10:{HASH}"),
+ format!("image:V4:7:2026-10:{HASH}"),
+ format!("image:v4:07:2026-10:{HASH}"),
+ format!("image:v4:0:2026-10:{HASH}"),
+ format!("image:v4:-7:2026-10:{HASH}"),
+ format!("image:v4:+7:2026-10:{HASH}"),
+ format!("image:v4:9223372036854775808:2026-10:{HASH}"),
+ format!("image:v4::2026-10:{HASH}"),
+ format!("image:v4:7:2026-13:{HASH}"),
+ format!("image:v4:7:2026-00:{HASH}"),
+ format!("image:v4:7:2026-1:{HASH}"),
+ format!("image:v4:7:226-10:{HASH}"),
+ format!("image:v4:7:2026_10:{HASH}"),
+ format!("image:v4:7:2026-10-01:{HASH}"),
+ format!("image:v4:7:{HASH}"),
+ format!("image:v4:7:2026-10:{}", &HASH[1..]),
+ format!("image:v4:7:2026-10:{HASH}A"),
+ format!("image:v4:7:2026-10:{HASH}="),
+ format!("image:v4:7:2026-10:{HASH}:7"),
+ "image:v4:7:2026-10:AbCdEfGhIjKlMnOpQr+/09".to_owned(),
+ "image:v4:7:2026-10:AbCdEfGhIjKlMnOpQré09".to_owned(),
+ format!(" image:v4:7:2026-10:{HASH}"),
+ format!("image:v4:7:2026-10:{HASH}\n"),
+ format!("imageurl:v4:7:2026-10:{HASH}%20"),
+ "image:v4:declined".to_owned(),
+ "image:v4:declined:".to_owned(),
+ "image:v4:declined:AVIF".to_owned(),
+ "image:v4:declined:avif ".to_owned(),
+ "image:v4:declined:credential".to_owned(),
+ "image:v4:declined:unchecked".to_owned(),
+ "imageurl:v4:declined:avif".to_owned(),
+ "imageurl:v4:declined:".to_owned(),
+ "imageurl:v4:declined:Credential".to_owned(),
+ "imageurl:v4:declined:credential:x".to_owned(),
+ "imageurl:v4:declined:session_token".to_owned(),
+ "imageurl:v3:declined:credential".to_owned(),
+ ];
+ for near_miss in near_misses {
+ assert_eq!(parse(&near_miss), None, "{near_miss:?}");
+ }
+ }
+}