:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(security): log out-of-band interaction domains in the rust guards (#118875)
Tom Piccirelloa706fcc2eda1
diff --git a/.agents/security.md b/.agents/security.mdindex 0e0cd304d9ac..f223befb1930 100644--- a/.agents/security.md+++ b/.agents/security.md@@ -220,8 +220,14 @@ The email guard always has a team.The request guard has one only when the fetch runs under `fetchAttribution`, as a CDP destination fetch and a recording export's stylesheet fetch do, and logs `team_id: null` for any other caller.`noteOobDomain` logs one line per host, team and guard every 15 minutes, because a retrying destination would otherwise log per attempt.-The check runs on cloud only, and it runs before each guard's own DNS resolution, because the-lookup alone confirms the path to whoever runs the collaborator.+The Rust guards call `common_dns::oob_domains::note_oob_domain` and emit `security.oob_domain_seen` under three guard names.+They read the hand-written mirror in `rust/common/dns/src/oob_domains.rs`, which a parity test holds against the JSON.+`note_oob_domain` takes the team as an `Option`, logs one line per host, team and guard every 15 minutes, and omits the `team_id` field rather than printing an empty one.+`cymbal_sourcemap` names the team on the source and sourcemap fetches, and omits it on a redirect hop, because one HTTP client serves every team.+`batch_import_s3_endpoint` always names the team, which the job model carries.+`common_dns_resolver` never names one: it fires inside the shared reqwest and Smithy resolvers, which no caller passes a team to.++Every check runs on cloud only, and each runs before its guard's own DNS resolution, because the lookup alone confirms the path to whoever runs the collaborator.It stops the default scanner configuration. It does not stop a determined attacker, who points aCNAME at the collaborator or runs their own server. The IP checks and the address pinning remaindiff --git a/.github/workflows/ci-rust.yml b/.github/workflows/ci-rust.ymlindex de4578854e38..3cf0b6033051 100644--- a/.github/workflows/ci-rust.yml+++ b/.github/workflows/ci-rust.yml@@ -86,6 +86,9 @@ jobs:- 'ee/migrations/**'- 'docker-compose.dev.yml'- 'docker-compose.base.yml'+ # The rust mirror of the out-of-band domain list has a parity test+ # against the shared json.+ - 'posthog/security/crosslang/**'# Hypercache contract: Python serializer changes may break Rust deserialization- 'posthog/api/feature_flag.py'- 'posthog/models/feature_flag/flags_cache.py'diff --git a/posthog/security/crosslang/README.md b/posthog/security/crosslang/README.mdindex 82afbc71341c..cfa59365110a 100644--- a/posthog/security/crosslang/README.md+++ b/posthog/security/crosslang/README.md@@ -18,7 +18,8 @@ These do not belong here:## How to edit the list1. Edit this file first.-2. Update each mirror. A parity test in that language fails until you do.+2. Update each mirror. The Rust mirror is `OOB_DOMAINS` in `rust/common/dns/src/oob_domains.rs`.+ A parity test in that language fails until you do.3. The matcher compares the canonical host, exactly or on a label boundary. Write the registrabledomain only, lowercase, with no leading dot.diff --git a/posthog/test/repo_invariants/oob_denylist_coverage_baseline.txt b/posthog/test/repo_invariants/oob_denylist_coverage_baseline.txtindex 335c4662fb2d..7dd0cca54780 100644--- a/posthog/test/repo_invariants/oob_denylist_coverage_baseline.txt+++ b/posthog/test/repo_invariants/oob_denylist_coverage_baseline.txt@@ -13,5 +13,4 @@ products/tasks/backend/logic/services/agent_command.pyproducts/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/destinations_load/writers/postgres.pyproducts/warehouse_sources/backend/temporal/data_imports/sources/salesforce/auth.pyproducts/workflows/backend/providers/ses.py-rust/common/dns/src/lib.rsrust/cymbal/src/modes/processing/stages/resolution/remote/dns.rsdiff --git a/posthog/test/repo_invariants/test_oob_denylist_coverage.py b/posthog/test/repo_invariants/test_oob_denylist_coverage.pyindex 37ec8fc15c21..7aeeeef8c307 100644--- a/posthog/test/repo_invariants/test_oob_denylist_coverage.py+++ b/posthog/test/repo_invariants/test_oob_denylist_coverage.py@@ -3,8 +3,8 @@Every host guard in this repo is IP-based, and an out-of-band interaction domain resolves to anordinary public IP, so a new resolver that skips the check silently restores the gap.-The check proves a file references the matcher. It cannot prove the call runs before the resolve,-so placement stays a review matter.+The check proves a file calls the matcher. It cannot prove the call runs before the resolve, so+placement stays a review matter."""import sys@@ -36,14 +36,14 @@ RESOLUTION_MARKERS = ("trust_dns",)+# Each marker ends at the open paren, so only a call clears a file. A bare module name would+# also match an import, a `mod` declaration or a comment, none of which run the check.CHECK_MARKERS = (- "matched_oob_domain",- "note_oob_domain",- "OOB_DOMAINS",- "oob_domains",+ "matched_oob_domain(",+ "note_oob_domain(",# The TypeScript mirror names the same two helpers in camel case.- "matchedOobDomain",- "noteOobDomain",+ "matchedOobDomain(",+ "noteOobDomain(",)SKIPPED_PARTS = ("node_modules", "__pycache__", "target", "dist", "migrations")@@ -54,7 +54,7 @@ def _is_test(path: Path) -> bool:def _resolves_hostnames(text: str) -> bool:- """Whether the file resolves a hostname and never names the out-of-band check."""+ """Whether the file resolves a hostname and never calls the out-of-band check."""return any(marker in text for marker in RESOLUTION_MARKERS) and not any(marker in text for marker in CHECK_MARKERS)@@ -71,6 +71,14 @@ def collect() -> list[str]:return sorted(found)+def test_naming_the_check_without_calling_it_does_not_exempt_a_resolver() -> None:+ resolve = "(host, 0).to_socket_addrs()"+ assert _resolves_hostnames(f"use common_dns::oob_domains;\n{resolve}")+ assert _resolves_hostnames(f"pub mod oob_domains;\n{resolve}")+ assert not _resolves_hostnames(f'note_oob_domain(host, "g", None);\n{resolve}')+ assert not _resolves_hostnames(f"noteOobDomain(host, 'g', ctx);\n{resolve}")++def test_resolvers_match_the_baseline() -> None:scanned = collect()recorded = sorted(line for line in BASELINE.read_text().splitlines() if line.strip())diff --git a/rust/AGENTS.md b/rust/AGENTS.mdindex 236a8abd0066..a16de21004db 100644--- a/rust/AGENTS.md+++ b/rust/AGENTS.md@@ -20,3 +20,10 @@ A pull request builds no Rust image unless it carries a `smoke_test:<image>` labAdd a label when the change can break the image build but not `cargo build`: a new binary or image entry, a `build.rs` or system dependency change, a new workspace crate, or a Dockerfile edit.Add one too when the PR needs a deployable image.Master builds and deploys after merge, and a broken image stops that deploy.++## Out-of-band interaction domains++Call `common_dns::oob_domains::note_oob_domain(host, "<guard>", team_id)` before a crate resolves or dials a host that a customer supplied.+Pass `Some(team_id)` when the caller knows the project, and `None` when it does not, such as in shared infrastructure.+`OOB_DOMAINS` in `common/dns/src/oob_domains.rs` mirrors `posthog/security/crosslang/oob_domains.json`, so edit the json first and the mirror second.+The parity test in `common-dns` fails while the two differ.diff --git a/rust/Cargo.lock b/rust/Cargo.lockindex 5bc2258f4946..3229a54de865 100644--- a/rust/Cargo.lock+++ b/rust/Cargo.lock@@ -2991,7 +2991,10 @@ dependencies = ["aws-smithy-runtime-api","futures","reqwest 0.12.28",+ "serde_json","tokio",+ "tracing",+ "tracing-subscriber",][[package]]diff --git a/rust/affected-services/determinator-rules.toml b/rust/affected-services/determinator-rules.tomlindex 286986663f7c..e18ef1f9d0e6 100644--- a/rust/affected-services/determinator-rules.toml+++ b/rust/affected-services/determinator-rules.toml@@ -87,6 +87,12 @@ globs = []mark-changed = "all"+# The out-of-band domain list: common-dns mirrors it, and a parity test in that crate+# fails when the two differ. Must come BEFORE the catch-all rule below.+[[path-rule]]+globs = ["../posthog/security/crosslang/**"]+mark-changed = ["common-dns"]+# Catch-all for files outside the rust workspace (Python, frontend, CI, docs,# etc.) — these are not crate source and should not trigger rust rebuilds.# Must come AFTER all specific rules for files outside the workspace.diff --git a/rust/batch-import-worker/Cargo.toml b/rust/batch-import-worker/Cargo.tomlindex 6a863e1c7954..db2438c56b65 100644--- a/rust/batch-import-worker/Cargo.toml+++ b/rust/batch-import-worker/Cargo.toml@@ -52,6 +52,7 @@ moka = { workspace = true }posthog-rs = { workspace = true }[dev-dependencies]+common-dns = { path = "../common/dns", features = ["test-util"] }httpmock = { workspace = true }flate2 = { workspace = true }mockito = { workspace = true }diff --git a/rust/batch-import-worker/src/job/config.rs b/rust/batch-import-worker/src/job/config.rsindex a0553a27756e..0584237aa285 100644--- a/rust/batch-import-worker/src/job/config.rs+++ b/rust/batch-import-worker/src/job/config.rs@@ -246,6 +246,7 @@ impl SourceConfig {context: Arc<AppContext>,is_restarting: bool,job_id: Uuid,+ team_id: i32,) -> Result<Box<dyn DataSource>, Error> {let staging_dir = context.config.staging_dir();let staging_max_bytes = context.config.staging_dir_max_bytes;@@ -277,6 +278,7 @@ impl SourceConfig {.create_source(secrets,context.config.managed_migrations_role_arn.as_deref(),+ team_id,).await?,)),@@ -285,6 +287,7 @@ impl SourceConfig {.create_gzip_source(secrets,context.config.managed_migrations_role_arn.as_deref(),+ team_id,staging_dir,staging_max_bytes,staging_backend,@@ -412,13 +415,34 @@ impl UrlListConfig {impl S3SourceConfig {/// Reject endpoint URLs that point to private/internal IPs./// Catches literal IP addresses that would bypass DNS-level SSRF protection.- fn validate_endpoint_url(url: &str, allow_internal_ips: bool) -> Result<(), Error> {+ ///+ /// A listed out-of-band interaction domain is logged above the escape hatch below, so a+ /// probe is recorded whatever `allow_internal_ips` says. That check refuses nothing.+ fn validate_endpoint_url(+ url: &str,+ allow_internal_ips: bool,+ team_id: i32,+ ) -> Result<(), Error> {+ // Parsed once: the detection below runs on every endpoint, while a parse failure is only+ // an error on the path that goes on to check the address.+ let parsed = url::Url::parse(url);++ if let Ok(ref parsed) = parsed {+ if let Some(url::Host::Domain(host)) = parsed.host() {+ common_dns::oob_domains::note_oob_domain(+ host,+ "batch_import_s3_endpoint",+ Some(team_id),+ );+ }+ }+if allow_internal_ips {return Ok(());}- let parsed = url::Url::parse(url)- .map_err(|e| Error::msg(format!("Invalid endpoint URL '{}': {}", url, e)))?;+ let parsed =+ parsed.map_err(|e| Error::msg(format!("Invalid endpoint URL '{}': {}", url, e)))?;let host = parsed.host()@@ -613,6 +637,7 @@ impl S3SourceConfig {&self,secrets: &JobSecrets,managed_migrations_role_arn: Option<&str>,+ team_id: i32,) -> Result<aws_sdk_s3::config::Builder, Error> {let mut builder = aws_sdk_s3::config::Builder::new().region(Region::new(self.region.clone()))@@ -648,7 +673,7 @@ impl S3SourceConfig {};if let Some(ref url) = self.endpoint_url {- Self::validate_endpoint_url(url, self.allow_internal_ips)?;+ Self::validate_endpoint_url(url, self.allow_internal_ips, team_id)?;builder = builder.endpoint_url(url).force_path_style(true);if !self.allow_internal_ips {@@ -668,9 +693,10 @@ impl S3SourceConfig {&self,secrets: &JobSecrets,managed_migrations_role_arn: Option<&str>,+ team_id: i32,) -> Result<aws_sdk_s3::Client, Error> {let builder = self- .build_s3_config(secrets, managed_migrations_role_arn)+ .build_s3_config(secrets, managed_migrations_role_arn, team_id).await?;Ok(aws_sdk_s3::Client::from_conf(builder.build()))}@@ -679,9 +705,10 @@ impl S3SourceConfig {&self,secrets: &JobSecrets,managed_migrations_role_arn: Option<&str>,+ team_id: i32,) -> Result<S3Source, Error> {let client = self- .build_client(secrets, managed_migrations_role_arn)+ .build_client(secrets, managed_migrations_role_arn, team_id).await?;Ok(S3Source::new(@@ -691,17 +718,19 @@ impl S3SourceConfig {))}+ #[allow(clippy::too_many_arguments)]pub async fn create_gzip_source(&self,secrets: &JobSecrets,managed_migrations_role_arn: Option<&str>,+ team_id: i32,staging_dir: PathBuf,staging_max_bytes: u64,staging_backend: Option<Arc<dyn StagingBackend>>,staged_plaintext_max_bytes: u64,) -> Result<GzipS3Source, Error> {let client = self- .build_client(secrets, managed_migrations_role_arn)+ .build_client(secrets, managed_migrations_role_arn, team_id).await?;let remote_staging = staging_backend.map(|backend| RemoteStaging {@@ -813,6 +842,7 @@ impl JobConfig {#[cfg(test)]mod tests {use super::*;+ use common_dns::test_util::capture_oob_logs;fn create_test_job_config() -> JobConfig {JobConfig {@@ -1148,7 +1178,7 @@ mod tests {serde_json::json!({"endpoint_url": "https://acct123.r2.cloudflarestorage.com"}),);let err = config- .build_s3_config(&empty_secrets(), None)+ .build_s3_config(&empty_secrets(), None, 1).await.unwrap_err().to_string();@@ -1159,7 +1189,7 @@ mod tests {async fn test_build_s3_config_rejects_role_auth_without_managed_migrations_role() {let config = role_auth_config(serde_json::json!({}));let err = config- .build_s3_config(&empty_secrets(), None)+ .build_s3_config(&empty_secrets(), None, 1).await.unwrap_err().to_string();@@ -1171,7 +1201,7 @@ mod tests {let mut config = role_auth_config(serde_json::json!({}));config.external_id = None;let err = config- .build_s3_config(&empty_secrets(), None)+ .build_s3_config(&empty_secrets(), None, 1).await.unwrap_err().to_string();@@ -1192,7 +1222,7 @@ mod tests {secrets.insert("ak".to_string(), Value::String("key-id".to_string()));secrets.insert("sk".to_string(), Value::String("key-secret".to_string()));let secrets = JobSecrets { secrets };- assert!(config.build_s3_config(&secrets, None).await.is_ok());+ assert!(config.build_s3_config(&secrets, None, 1).await.is_ok());}#[tokio::test]@@ -1206,7 +1236,7 @@ mod tests {}"#;let config: S3SourceConfig = serde_json::from_str(json).unwrap();let err = config- .build_s3_config(&empty_secrets(), None)+ .build_s3_config(&empty_secrets(), None, 1).await.unwrap_err().to_string();@@ -1225,7 +1255,7 @@ mod tests {for patch in cases {let config = role_auth_config(patch.clone());let err = config- .build_s3_config(&empty_secrets(), None)+ .build_s3_config(&empty_secrets(), None, 1).await.unwrap_err().to_string();@@ -1248,7 +1278,7 @@ mod tests {}"#;let config: S3SourceConfig = serde_json::from_str(json).unwrap();let err = config- .build_s3_config(&empty_secrets(), None)+ .build_s3_config(&empty_secrets(), None, 1).await.unwrap_err().to_string();@@ -1260,7 +1290,7 @@ mod tests {let json = r#"{"bucket": "b", "prefix": "p", "region": "us-east-1"}"#;let config: S3SourceConfig = serde_json::from_str(json).unwrap();let err = config- .build_s3_config(&empty_secrets(), None)+ .build_s3_config(&empty_secrets(), None, 1).await.unwrap_err().to_string();@@ -1282,7 +1312,7 @@ mod tests {#[test]fn test_validate_endpoint_url_rejects_metadata_service() {- let result = S3SourceConfig::validate_endpoint_url("http://169.254.169.254", false);+ let result = S3SourceConfig::validate_endpoint_url("http://169.254.169.254", false, 1);assert!(result.is_err());assert!(result.unwrap_err()@@ -1292,19 +1322,19 @@ mod tests {#[test]fn test_validate_endpoint_url_rejects_private_ip() {- let result = S3SourceConfig::validate_endpoint_url("http://10.0.0.1:9000", false);+ let result = S3SourceConfig::validate_endpoint_url("http://10.0.0.1:9000", false, 1);assert!(result.is_err());}#[test]fn test_validate_endpoint_url_rejects_loopback() {- let result = S3SourceConfig::validate_endpoint_url("http://127.0.0.1:9000", false);+ let result = S3SourceConfig::validate_endpoint_url("http://127.0.0.1:9000", false, 1);assert!(result.is_err());}#[test]fn test_validate_endpoint_url_rejects_ipv6_loopback() {- let result = S3SourceConfig::validate_endpoint_url("http://[::1]:9000", false);+ let result = S3SourceConfig::validate_endpoint_url("http://[::1]:9000", false, 1);assert!(result.is_err());}@@ -1313,20 +1343,51 @@ mod tests {let result = S3SourceConfig::validate_endpoint_url("https://acct123.r2.cloudflarestorage.com",false,+ 1,);assert!(result.is_ok());}#[test]fn test_validate_endpoint_url_accepts_internal_when_allowed() {- let result = S3SourceConfig::validate_endpoint_url("http://localhost:9000", true);+ let result = S3SourceConfig::validate_endpoint_url("http://localhost:9000", true, 1);assert!(result.is_ok());}#[test]fn test_validate_endpoint_url_rejects_invalid_url() {- let result = S3SourceConfig::validate_endpoint_url("not-a-url", false);+ let result = S3SourceConfig::validate_endpoint_url("not-a-url", false, 1);assert!(result.is_err());+ // The out-of-band domain check parses the url first, and must leave this input alone.+ assert!(S3SourceConfig::validate_endpoint_url("not-a-url", true, 1).is_ok());+ }++ #[test]+ fn test_validate_endpoint_url_logs_an_out_of_band_domain_and_still_accepts_it() {+ for (url, listed) in [+ ("https://bucket.oast.live:9000", true),+ ("https://notoast.me:9000", false),+ ] {+ // The check sits above the internal-ip escape hatch, so it runs under both values.+ for allow_internal_ips in [false, true] {+ let logs = capture_oob_logs(|| {+ assert!(+ S3SourceConfig::validate_endpoint_url(url, allow_internal_ips, 4242)+ .is_ok(),+ "detection must not refuse {url}"+ );+ });+ let seen = logs.contains("security.oob_domain_seen")+ && logs.contains("batch_import_s3_endpoint");+ assert_eq!(seen, listed, "unexpected log for {url}: {logs}");+ if listed {+ assert!(+ logs.contains("team_id=4242"),+ "the log line names the project that configured the endpoint: {logs}"+ );+ }+ }+ }}#[test]diff --git a/rust/batch-import-worker/src/job/mod.rs b/rust/batch-import-worker/src/job/mod.rsindex 509b4810febe..5502c9ca305a 100644--- a/rust/batch-import-worker/src/job/mod.rs+++ b/rust/batch-import-worker/src/job/mod.rs@@ -618,7 +618,13 @@ impl Job {let source = model.import_config.source- .construct(&model.secrets, context.clone(), is_restarting, model.id)+ .construct(+ &model.secrets,+ context.clone(),+ is_restarting,+ model.id,+ model.team_id,+ ).await.with_context(|| "Failed to construct data source for job".to_string())?;diff --git a/rust/batch-import-worker/src/trial/job.rs b/rust/batch-import-worker/src/trial/job.rsindex 2eecf8ee5498..f923192afb69 100644--- a/rust/batch-import-worker/src/trial/job.rs+++ b/rust/batch-import-worker/src/trial/job.rs@@ -84,7 +84,13 @@ impl TrialJob {let source = model.import_config.source- .construct(&model.secrets, context.clone(), is_restarting, model.id)+ .construct(+ &model.secrets,+ context.clone(),+ is_restarting,+ model.id,+ model.team_id,+ ).await.with_context(|| "Failed to construct data source for trial job".to_string())?;source.prepare_for_job().await?;diff --git a/rust/batch-import-worker/tests/s3_gzip_minio_integration_test.rs b/rust/batch-import-worker/tests/s3_gzip_minio_integration_test.rsindex 47f8d543a4c7..15f0a6645688 100644--- a/rust/batch-import-worker/tests/s3_gzip_minio_integration_test.rs+++ b/rust/batch-import-worker/tests/s3_gzip_minio_integration_test.rs@@ -108,7 +108,7 @@ async fn test_s3_gzip_minio_integration() {let _staging = tempfile::TempDir::new().expect("create staging dir");let source = s3_config- .create_gzip_source(&secrets, None, _staging.path().to_path_buf(), 0, None, 0)+ .create_gzip_source(&secrets, None, 1, _staging.path().to_path_buf(), 0, None, 0).await.expect("create gzip source");source.prepare_for_job().await.expect("prepare for job");diff --git a/rust/common/dns/Cargo.toml b/rust/common/dns/Cargo.tomlindex 354f95d86e14..e9ef98a19aad 100644--- a/rust/common/dns/Cargo.toml+++ b/rust/common/dns/Cargo.toml@@ -8,9 +8,15 @@ workspace = true[features]smithy = ["aws-smithy-runtime-api"]+test-util = ["dep:tracing-subscriber"][dependencies]aws-smithy-runtime-api = { workspace = true, optional = true }futures = { workspace = true }reqwest = { workspace = true }-tokio = { workspace = true }\ No newline at end of file+tokio = { workspace = true }+tracing = { workspace = true }+tracing-subscriber = { workspace = true, optional = true }++[dev-dependencies]+serde_json = { workspace = true }diff --git a/rust/common/dns/src/lib.rs b/rust/common/dns/src/lib.rsindex dc0b25b252fd..5f9fea79ad76 100644--- a/rust/common/dns/src/lib.rs+++ b/rust/common/dns/src/lib.rs@@ -6,6 +6,11 @@ use futures::FutureExt;use reqwest::dns::{Addrs, Name, Resolve, Resolving};use tokio::task::spawn_blocking;+pub mod oob_domains;++#[cfg(feature = "test-util")]+pub mod test_util;+pub struct NoPublicIPv4Error;impl std::error::Error for NoPublicIPv4Error {}@@ -75,6 +80,10 @@ pub struct PublicIPv4Resolver {}impl Resolve for PublicIPv4Resolver {fn resolve(&self, name: Name) -> Resolving {+ // The last net before the lookup leaves the process. A caller that reaches this resolver+ // through no host guard, such as the batch import url list, is still recorded here.+ oob_domains::note_oob_domain(name.as_str(), "common_dns_resolver", None);+// Closure to call the system's resolver (blocking call) through the ToSocketAddrs trait.let resolve_host = move || (name.as_str(), 0).to_socket_addrs();@@ -128,6 +137,9 @@ impl aws_smithy_runtime_api::client::dns::ResolveDns for PublicIPv4SmithyResolve) -> aws_smithy_runtime_api::client::dns::DnsFuture<'a> {use aws_smithy_runtime_api::client::dns::{DnsFuture, ResolveDnsError};+ // Same net as the reqwest resolver above, for the S3 client path.+ oob_domains::note_oob_domain(name, "common_dns_resolver", None);+let name = name.to_string();DnsFuture::new(async move {let resolve_host =diff --git a/rust/common/dns/src/oob_domains.rs b/rust/common/dns/src/oob_domains.rsnew file mode 100644index 000000000000..f261b71b808a--- /dev/null+++ b/rust/common/dns/src/oob_domains.rs@@ -0,0 +1,236 @@+//! Out-of-band interaction domains, and the check the outbound guards run on a host.+//!+//! Every other host guard here is IP-based, and an out-of-band interaction domain resolves to+//! an ordinary public IP, so the lookup alone confirms the path to whoever runs the collaborator.+//! The guards log a match. Nothing is refused yet.++use std::collections::HashMap;+#[cfg(feature = "test-util")]+use std::sync::atomic::{AtomicBool, Ordering};+use std::sync::{Mutex, OnceLock};+use std::time::{Duration, Instant};++use tracing::info;++/// Mirror of posthog/security/crosslang/oob_domains.json. Edit that file first, then this list.+/// The parity test mirror_matches_the_shared_json fails when they differ.+pub const OOB_DOMAINS: &[&str] = &[+ "1u.ms",+ "burpcollaborator.net",+ "ceye.io",+ "dnslog.cn",+ "interact.sh",+ "oast.fun",+ "oast.live",+ "oast.me",+ "oast.online",+ "oast.pro",+ "oast.site",+ "oastify.com",+ "rbndr.us",+];++const MAX_DNS_NAME_LEN: usize = 253;++/// The listed domain this host belongs to, or None.+///+/// Pass a host that `url` parsed, never a raw url. The `url` crate applies IDNA, so an+/// internationalized name arrives in the ASCII form this list holds.+pub fn matched_oob_domain(host: &str) -> Option<&'static str> {+ if host.is_empty() || host.contains('@') || host.contains('/') {+ // A match on a raw url or a credential would put that string in the log line.+ return None;+ }+ if host.strip_suffix('.').unwrap_or(host).len() > MAX_DNS_NAME_LEN {+ // The url crate parses a name of any length, and a match keeps the host in the dedupe map.+ return None;+ }+ // The resolver treats the absolute form ("oast.live.") as the same name.+ let canonical = host.trim_end_matches('.').to_ascii_lowercase();+ OOB_DOMAINS+ .iter()+ .copied()+ .find(|domain| is_same_or_subdomain(&canonical, domain))+}++/// Same set as `is_cloud` in posthog/cloud_utils.py and `isCloud` in the node env utils.+const CLOUD_DEPLOYMENTS: &[&str] = &["EU", "US", "DEV", "E2E"];++/// A test cannot use `set_var`: it races libc environment reads from DNS lookups on other threads.+#[cfg(feature = "test-util")]+static FORCE_CLOUD: AtomicBool = AtomicBool::new(false);++/// Whether this deployment is PostHog Cloud. The python and node guards read the same variable.+///+/// Read per call rather than cached, because it is only reached once a host already matched, which+/// is rare.+fn is_cloud() -> bool {+ #[cfg(feature = "test-util")]+ if FORCE_CLOUD.load(Ordering::Relaxed) {+ return true;+ }+ is_cloud_deployment(std::env::var("CLOUD_DEPLOYMENT").ok().as_deref())+}++fn is_cloud_deployment(value: Option<&str>) -> bool {+ value.is_some_and(|value| {+ CLOUD_DEPLOYMENTS+ .iter()+ .any(|cloud| value.eq_ignore_ascii_case(cloud))+ })+}++#[cfg(feature = "test-util")]+pub fn force_cloud() {+ FORCE_CLOUD.store(true, Ordering::Relaxed);+}++/// One line per host, team and guard in this window. A resolver sees a host once per+/// connection, so an undeduped count tracks traffic rather than probing.+const DEDUPE_WINDOW: Duration = Duration::from_secs(15 * 60);+const DEDUPE_MAX_ENTRIES: usize = 10_000;++type SeenMap = HashMap<(&'static str, Option<i32>, String), Instant>;+static SEEN: OnceLock<Mutex<SeenMap>> = OnceLock::new();++fn seen_map() -> std::sync::MutexGuard<'static, SeenMap> {+ match SEEN.get_or_init(|| Mutex::new(HashMap::new())).lock() {+ Ok(seen) => seen,+ // A poisoned lock must not cost us the detection, and must not panic a guard either.+ Err(poisoned) => poisoned.into_inner(),+ }+}++/// Forget every recorded sighting, so one test cannot suppress the next one's log line.+#[cfg(feature = "test-util")]+pub fn reset_dedupe() {+ seen_map().clear();+}++fn first_sight_in_window(+ guard: &'static str,+ team_id: Option<i32>,+ host: &str,+ now: Instant,+) -> bool {+ let mut seen = seen_map();+ // The team is part of the key so that a sighting for one project does not suppress the+ // line for another.+ let key = (guard, team_id, host.to_owned());+ if let Some(at) = seen.get(&key) {+ if now.duration_since(*at) < DEDUPE_WINDOW {+ return false;+ }+ }+ if seen.len() >= DEDUPE_MAX_ENTRIES {+ // Cheaper than an LRU, and the window makes a stale entry harmless.+ seen.retain(|_, at| now.duration_since(*at) < DEDUPE_WINDOW);+ if seen.len() >= DEDUPE_MAX_ENTRIES {+ seen.clear();+ }+ }+ seen.insert(key, now);+ true+}++/// Log that a guard holds a listed host. Detection only: the caller still proceeds.+///+/// A guard that knows the project passes its id. The shared resolvers and a redirect hop do not+/// have one, so they pass `None` and the line carries no `team_id`.+pub fn note_oob_domain(host: &str, guard: &'static str, team_id: Option<i32>) {+ // Match first: almost every host misses, and the miss costs a few string comparisons, which+ // keeps the environment read and the lock off the common path.+ let Some(domain) = matched_oob_domain(host) else {+ return;+ };+ if !is_cloud() {+ return;+ }+ if !first_sight_in_window(guard, team_id, host, Instant::now()) {+ return;+ }+ // `Option` records the inner value and omits the field for `None`, so a line either names a+ // team or carries no `team_id` field at all.+ info!(team_id, host, domain, guard, "security.oob_domain_seen");+}++/// Whether the host is the domain itself, or sits under it on a label boundary. A plain suffix+/// test would also match "notoast.me", which belongs to somebody else.+fn is_same_or_subdomain(host: &str, domain: &str) -> bool {+ match host.len().checked_sub(domain.len()) {+ Some(0) => host == domain,+ Some(offset) => host.ends_with(domain) && host.as_bytes()[offset - 1] == b'.',+ None => false,+ }+}++#[cfg(test)]+mod tests {+ use super::{first_sight_in_window, is_cloud_deployment, matched_oob_domain};+ use std::time::Instant;++ #[test]+ fn only_a_cloud_deployment_value_counts_as_cloud() {+ for (value, expected) in [+ (Some("US"), true),+ (Some("EU"), true),+ (Some("DEV"), true),+ (Some("E2E"), true),+ (Some("us"), true),+ (Some("local"), false),+ (Some("TEST"), false),+ (Some(""), false),+ (None, false),+ ] {+ assert_eq!(is_cloud_deployment(value), expected, "value: {value:?}");+ }+ }++ #[test]+ fn matches_a_listed_domain_exactly_or_on_a_label_boundary() {+ for (host, expected) in [+ ("oast.live", Some("oast.live")),+ ("x.oast.live", Some("oast.live")),+ ("a.b.oast.pro", Some("oast.pro")),+ ("burpcollaborator.net", Some("burpcollaborator.net")),+ ("sub.oastify.com", Some("oastify.com")),+ ("OAST.ME", Some("oast.me")),+ ("oast.me.", Some("oast.me")),+ ("notoast.me", None),+ ("myoast.live", None),+ ("oast.me.evil.example", None),+ ("example.com", None),+ ("", None),+ ("1.2.3.4", None),+ ("https://oast.live/a.js", None),+ ("user:[email protected]", None),+ ] {+ assert_eq!(matched_oob_domain(host), expected, "host: {host}");+ }+ }++ #[test]+ fn a_host_longer_than_a_dns_name_never_matches() {+ let longest = format!("{}.oast.me", "a".repeat(245));+ assert_eq!(longest.len(), 253);+ assert_eq!(matched_oob_domain(&longest), Some("oast.me"));+ assert_eq!(matched_oob_domain(&format!("{longest}.")), Some("oast.me"));+ assert_eq!(matched_oob_domain(&format!("a{longest}")), None);+ assert_eq!(+ matched_oob_domain(&format!("x.oast.me{}", ".".repeat(300))),+ None+ );+ }++ #[test]+ fn one_team_does_not_dedupe_another_teams_sighting() {+ let now = Instant::now();+ let host = "one-team-does-not-dedupe.oast.me";+ assert!(first_sight_in_window("g", Some(1), host, now));+ // An actor can probe several projects with the same host, so a key without the team+ // would report the first project and hide the rest for the window.+ assert!(first_sight_in_window("g", Some(2), host, now));+ assert!(first_sight_in_window("g", None, host, now));+ assert!(!first_sight_in_window("g", Some(1), host, now));+ }+}diff --git a/rust/common/dns/src/test_util.rs b/rust/common/dns/src/test_util.rsnew file mode 100644index 000000000000..76dc4584f33b--- /dev/null+++ b/rust/common/dns/src/test_util.rs@@ -0,0 +1,48 @@+//! Test support for the out-of-band domain guards. Enabled by the `test-util` feature, so it is+//! built only by the crates that test a guard.++use std::sync::{Arc, Mutex};++use tracing_subscriber::fmt::MakeWriter;++#[derive(Clone)]+pub struct Sink(Arc<Mutex<Vec<u8>>>);++impl std::io::Write for Sink {+ fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {+ self.0.lock().expect("sink lock").extend_from_slice(buf);+ Ok(buf.len())+ }++ fn flush(&mut self) -> std::io::Result<()> {+ Ok(())+ }+}++impl<'a> MakeWriter<'a> for Sink {+ type Writer = Sink;++ fn make_writer(&'a self) -> Self::Writer {+ self.clone()+ }+}++/// Run `work` as cloud with the tracing output captured, and return that output.+///+/// `note_oob_domain` logs on cloud only, so a guard test has to look like cloud. The override is+/// set for the rest of the process rather than restored, because the tests in a binary run in+/// parallel and a restore would race one that is still inside `work`.+pub fn capture_oob_logs(work: impl FnOnce()) -> String {+ crate::oob_domains::force_cloud();+ // Each capture starts clean, so one case cannot dedupe away the next one's line.+ crate::oob_domains::reset_dedupe();++ let sink = Sink(Arc::new(Mutex::new(Vec::new())));+ let subscriber = tracing_subscriber::fmt()+ .with_writer(sink.clone())+ .with_ansi(false)+ .finish();+ tracing::subscriber::with_default(subscriber, work);+ let captured = sink.0.lock().expect("sink lock").clone();+ String::from_utf8(captured).expect("log output is utf-8")+}diff --git a/rust/common/dns/tests/oob_domains_parity.rs b/rust/common/dns/tests/oob_domains_parity.rsnew file mode 100644index 000000000000..9b6fddeebaf0--- /dev/null+++ b/rust/common/dns/tests/oob_domains_parity.rs@@ -0,0 +1,50 @@+//! The list lives outside this workspace, so point the test at it with `OOB_DOMAINS_JSON`.+//! Otherwise it walks up from the manifest dir looking for the file.++use std::collections::BTreeSet;+use std::path::PathBuf;++use common_dns::oob_domains::OOB_DOMAINS;++const RELATIVE_PATH: &str = "posthog/security/crosslang/oob_domains.json";++fn domains_json() -> PathBuf {+ if let Ok(path) = std::env::var("OOB_DOMAINS_JSON") {+ let path = PathBuf::from(path);+ if path.is_file() {+ return path;+ }+ panic!(+ "OOB_DOMAINS_JSON is set to {}, which is not a file",+ path.display()+ );+ }+ let mut current = PathBuf::from(env!("CARGO_MANIFEST_DIR"));+ loop {+ let candidate = current.join(RELATIVE_PATH);+ if candidate.is_file() {+ return candidate;+ }+ if !current.pop() {+ panic!("Could not locate {RELATIVE_PATH} - set OOB_DOMAINS_JSON");+ }+ }+}++#[test]+fn mirror_matches_the_shared_json() {+ let raw = std::fs::read_to_string(domains_json()).expect("the domain list is readable");+ let parsed: serde_json::Value = serde_json::from_str(&raw).expect("the domain list is json");+ let from_file: BTreeSet<&str> = parsed["domains"]+ .as_array()+ .expect("the domain list holds a domains array")+ .iter()+ .map(|value| value.as_str().expect("every domain is a string"))+ .collect();+ let from_mirror: BTreeSet<&str> = OOB_DOMAINS.iter().copied().collect();++ assert_eq!(+ from_file, from_mirror,+ "OOB_DOMAINS has drifted from {RELATIVE_PATH}. Edit the json first, then the mirror."+ );+}diff --git a/rust/cymbal/Cargo.toml b/rust/cymbal/Cargo.tomlindex 1d8385337e17..e18ee1de2f5a 100644--- a/rust/cymbal/Cargo.toml+++ b/rust/cymbal/Cargo.toml@@ -60,6 +60,7 @@ common-temporal = { path = "../common/temporal" }[dev-dependencies]+common-dns = { path = "../common/dns", features = ["test-util"] }common-compression = { path = "../common/compression" }# For constructing IO-flavored RedisErrors in redis_heal classifier testsredis = { workspace = true }diff --git a/rust/cymbal/src/core/symbolication/symbol_store/sourcemap.rs b/rust/cymbal/src/core/symbolication/symbol_store/sourcemap.rsindex 19f5274e47cd..fde51db366e2 100644--- a/rust/cymbal/src/core/symbolication/symbol_store/sourcemap.rs+++ b/rust/cymbal/src/core/symbolication/symbol_store/sourcemap.rs@@ -54,7 +54,19 @@ const MAX_REDIRECTS: usize = 10;////// Hostnames are passed through, to be vetted at connect time by whichever layer owns that:/// `PublicIPv4Resolver` normally, or the egress proxy when one is configured.-fn ensure_fetchable_host(url: &Url, allow_internal_ips: bool) -> Result<(), JsResolveErr> {+///+/// An out-of-band interaction domain is logged on the way past. That check is detection only,+/// and it sits above the escape hatch below so a probe is recorded whatever+/// `allow_internal_ips` says.+fn ensure_fetchable_host(+ url: &Url,+ allow_internal_ips: bool,+ team_id: Option<i32>,+) -> Result<(), JsResolveErr> {+ if let Some(host) = url.host_str() {+ common_dns::oob_domains::note_oob_domain(host, "cymbal_sourcemap", team_id);+ }+if allow_internal_ips {return Ok(());}@@ -337,7 +349,8 @@ impl SourcemapProvider {"exceeded {MAX_REDIRECTS} redirects")));}- match ensure_fetchable_host(attempt.url(), allow_internal_ips) {+ // One client serves every team, so a redirect hop cannot name a project.+ match ensure_fetchable_host(attempt.url(), allow_internal_ips, None) {Ok(()) => attempt.follow(),Err(e) => attempt.error(e),}@@ -392,6 +405,7 @@ impl Fetcher for SourcemapProvider {r,self.max_response_bytes,self.allow_internal_ips,+ team_id,).await?;@@ -416,6 +430,7 @@ impl Fetcher for SourcemapProvider {sourcemap_url.clone(),self.max_response_bytes,self.allow_internal_ips,+ team_id,).await?}@@ -549,11 +564,12 @@ async fn find_sourcemap_url(start: Url,max_response_bytes: usize,allow_internal_ips: bool,+ team_id: i32,) -> Result<JsSourcePeek, ResolveError> {// The frame's source url comes straight off an ingested event, so vet it before we connect.// Every other url in here is derived from the response, and so keeps this validated host,// except an absolute sourcemap url - `fetch_source_map` vets that one itself.- ensure_fetchable_host(&start, allow_internal_ips)?;+ ensure_fetchable_host(&start, allow_internal_ips, Some(team_id))?;debug!("Fetching script source from {}", start);@@ -659,10 +675,11 @@ async fn fetch_source_map(url: Url,max_response_bytes: usize,allow_internal_ips: bool,+ team_id: i32,) -> Result<String, ResolveError> {// A `SourceMap` header or `//# sourceMappingURL=` comment can name an absolute url on any// host, so this needs vetting independently of the source url it came from.- ensure_fetchable_host(&url, allow_internal_ips)?;+ ensure_fetchable_host(&url, allow_internal_ips, Some(team_id))?;metrics::counter!(SOURCEMAP_BODY_FETCHES).increment(1);let res = client.get(url).send().await.map_err(JsResolveErr::from)?;@@ -830,6 +847,7 @@ fn assert_is_sourcemap(data: &str) -> Result<(), ResolveError> {#[cfg(test)]mod test {use crate::error::FrameError;+ use common_dns::test_util::capture_oob_logs;use httpmock::MockServer;use sourcemap::SourceMapBuilder;@@ -943,7 +961,7 @@ mod test {let client = reqwest::Client::new();let url = server.url("/static/chunk-PGUQKT6S.js").parse().unwrap();- let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true)+ let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true, 1).await.unwrap();@@ -972,7 +990,7 @@ mod test {let client = reqwest::Client::new();let url = server.url("/static/chunk.js").parse().unwrap();- let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true)+ let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true, 1).await.unwrap();@@ -1000,7 +1018,7 @@ mod test {let client = reqwest::Client::new();let url = server.url("/static/chunk.js").parse().unwrap();- let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true)+ let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true, 1).await.unwrap();@@ -1028,7 +1046,7 @@ mod test {let client = reqwest::Client::new();let url = server.url("/static/chunk.js?v=193bf1").parse().unwrap();- let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true)+ let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true, 1).await.unwrap();@@ -1055,7 +1073,7 @@ mod test {let client = reqwest::Client::new();let url = server.url("/static/chunk.js?v=193bf1").parse().unwrap();- let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true)+ let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true, 1).await.unwrap();@@ -1081,7 +1099,7 @@ mod test {let client = reqwest::Client::new();let url = server.url("/static/js/chunk.js").parse().unwrap();- let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true)+ let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true, 1).await.unwrap();@@ -1111,7 +1129,7 @@ mod test {let client = reqwest::Client::new();let url = source_server.url("/static/chunk.js").parse().unwrap();- let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true)+ let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true, 1).await.unwrap();@@ -1336,7 +1354,7 @@ mod test {.url("/static/inline_sourcemap_example.js").parse().unwrap();- let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true)+ let peek = find_sourcemap_url(&client, url, TEST_MAX_RESPONSE_BYTES, true, 1).await.unwrap();@@ -1369,6 +1387,64 @@ mod test {));}+ #[test]+ fn a_fullwidth_at_sign_never_reaches_the_matcher_as_a_host() {+ // The matcher refuses a string holding @ or /, but only the ascii forms. It never sees the+ // fullwidth ones, because Url rejects the whole input first. This pins that, so the guard+ // is not quietly weakened by a future url crate that starts accepting them.+ for raw in [+ "https://user@abc.oast.live/app.js",+ "https://user/abc.oast.live/app.js",+ ] {+ assert!(raw.parse::<Url>().is_err(), "expected {raw} to be refused");+ }+ }++ #[test]+ fn ensure_fetchable_host_logs_an_out_of_band_domain_and_still_fetches_it() {+ for (raw, host, listed) in [+ ("https://x.oast.live/app.js", "x.oast.live", true),+ // IDNA maps the fullwidth form onto the ascii name the resolver queries.+ // A subdomain, so the logged host differs from the logged domain and the assertion+ // below cannot pass on the domain alone.+ ("https://x.oast.me/app.js", "x.oast.me", true),+ ("https://notoast.me/app.js", "notoast.me", false),+ ] {+ let url: Url = raw.parse().unwrap();+ assert_eq!(+ url.host_str(),+ Some(host),+ "expected {raw} to parse to the host {host}"+ );++ // The check sits above the internal-ip escape hatch, so it runs under both values.+ for allow_internal_ips in [false, true] {+ // A redirect hop passes None, so both shapes have to render correctly.+ for team_id in [Some(4242), None] {+ let logs = capture_oob_logs(|| {+ assert!(+ ensure_fetchable_host(&url, allow_internal_ips, team_id).is_ok(),+ "detection must not refuse {raw}"+ );+ });+ let seen = logs.contains("security.oob_domain_seen")+ && logs.contains("cymbal_sourcemap");+ assert_eq!(seen, listed, "unexpected log for {raw}: {logs}");+ if listed {+ assert!(logs.contains(host), "the log line names the host: {logs}");+ assert_eq!(+ logs.contains("team_id=4242"),+ team_id.is_some(),+ "the log line names the team it was given: {logs}"+ );+ // An absent team drops the field rather than printing a Rust Option.+ assert!(!logs.contains("team_id=None"), "{logs}");+ }+ }+ }+ }+ }+#[test]fn ensure_fetchable_host_blocks_internal_ip_literals() {// Literal IPs never reach PublicIPv4Resolver, so this guard is the only thing@@ -1390,14 +1466,14 @@ mod test {let url: Url = raw.parse().unwrap();assert!(matches!(- ensure_fetchable_host(&url, false),+ ensure_fetchable_host(&url, false, None),Err(JsResolveErr::BlockedUrl(_))),"expected {raw} to be blocked");// The local-development escape hatch still lets these through.assert!(- ensure_fetchable_host(&url, true).is_ok(),+ ensure_fetchable_host(&url, true, None).is_ok(),"expected {raw} to be allowed when internal IPs are permitted");}@@ -1415,7 +1491,7 @@ mod test {);assert!(matches!(- ensure_fetchable_host(&url, false),+ ensure_fetchable_host(&url, false, None),Err(JsResolveErr::BlockedUrl(_))),"expected {raw} to be blocked, normalized to {url}"@@ -1433,7 +1509,7 @@ mod test {for raw in allowed {let url: Url = raw.parse().unwrap();assert!(- ensure_fetchable_host(&url, false).is_ok(),+ ensure_fetchable_host(&url, false, None).is_ok(),"expected {raw} to pass the guard");}