:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(replay): drop surfacing features that overflow float32 before scoring (#124935)
Tue Haulund65834f51b124
diff --git a/posthog/temporal/session_replay/surfacing_scoring_sweep/features.py b/posthog/temporal/session_replay/surfacing_scoring_sweep/features.pyindex 2ed284d98b81..efe107822f8b 100644--- a/posthog/temporal/session_replay/surfacing_scoring_sweep/features.py+++ b/posthog/temporal/session_replay/surfacing_scoring_sweep/features.py@@ -54,6 +54,9 @@ ID_COLUMNS: tuple[str, ...] = ("team_id", "session_id", "distinct_id", "min_firs_NUMERIC = "iuf"_FLOAT = "f"+# XGBoost casts features to float32, so a finite float64 above this still reaches it as inf.+_FLOAT32_MAX = float(np.finfo(np.float32).max)+@dataclass(frozen=True)class FeatureSpec:@@ -244,10 +247,10 @@ def validate_features(df: pd.DataFrame, *, feature_names: tuple[str, ...]) -> Nodef out_of_contract_row_mask(df: pd.DataFrame, *, feature_names: tuple[str, ...]) -> pd.Series:"""Boolean mask of rows whose feature values break the trained contract.- Flags +/-inf (a feature-engineering bug — division returning inf) and- values outside `FEATURE_RANGES` bounds. NaN passes — XGBoost handles it- natively, and our SQL deliberately produces NULL (→ NaN) when- denominators are zero.+ Flags +/-inf (a feature-engineering bug — division returning inf), values+ that overflow float32, and values outside `FEATURE_RANGES` bounds. NaN+ passes — XGBoost handles it natively, and our SQL deliberately produces+ NULL (→ NaN) when denominators are zero.Replay events are client-controlled, so a single crafted session canproduce an out-of-range derived feature. Flagging per-row lets callers@@ -262,8 +265,8 @@ def out_of_contract_row_mask(df: pd.DataFrame, *, feature_names: tuple[str, ...]series: pd.Series = df.loc[:, name]spec = FEATURE_RANGES[name]if series.dtype.kind == "f":- arr = series.to_numpy()- mask |= ~(np.isfinite(arr) | np.isnan(arr))+ # NaN compares False, so it passes; +/-inf compares True.+ mask |= np.abs(series.to_numpy()) > _FLOAT32_MAXif spec.min_value is not None:mask |= series.lt(spec.min_value).fillna(False)if spec.max_value is not None:diff --git a/posthog/temporal/tests/session_replay/surfacing_scoring_sweep/test_features.py b/posthog/temporal/tests/session_replay/surfacing_scoring_sweep/test_features.pyindex 3aa9b3a721d9..e124786b85cd 100644--- a/posthog/temporal/tests/session_replay/surfacing_scoring_sweep/test_features.py+++ b/posthog/temporal/tests/session_replay/surfacing_scoring_sweep/test_features.py@@ -107,6 +107,9 @@ class TestOutOfContractRowMask:("selection_copy_count", -1),("event_rate", float("inf")),("event_rate", float("-inf")),+ # Finite in float64 but inf once XGBoost casts to float32.+ ("event_rate", 1e39),+ ("mouse_mean_x", -1e39),],)def test_out_of_contract_value_flags_only_its_row(@@ -118,6 +121,8 @@ class TestOutOfContractRowMask:) -> None:good_row = feature_frame.iloc[0].copy()df = pd.DataFrame([feature_frame.iloc[0], good_row]).reset_index(drop=True)+ # ClickHouse returns Float64; the float32 fixture would turn 1e39 into inf before the mask sees it.+ df[column] = df[column].astype(np.float64)df.loc[0, column] = bad_valuemask = out_of_contract_row_mask(df, feature_names=feature_names_for_tests)