πΈ Rybbit - open-source and privacy-friendly alternative to Google Analytics that is 10x more intuitive.
fix(tracker): refine flood detection logic for residential addresses
Bill Yang534f3f424350
diff --git a/server/src/services/tracker/botBlocking/README.md b/server/src/services/tracker/botBlocking/README.mdindex 100d2921a..fd290bbaa 100644--- a/server/src/services/tracker/botBlocking/README.md+++ b/server/src/services/tracker/botBlocking/README.md@@ -131,7 +131,7 @@ Reasons come back in two lists, and the distinction is structural rather than aFive rules deserve their own note:-- **Site flood** (`site_flood_oneshot_cohort_10m`, `site_flood_actor_1d`) is the only site-relative logic in the layer. Every other threshold is absolute, so a fleet of disposable identities β one event each, a fresh address per hit, a real browser β can push a four-events-an-hour site to four thousand and trip nothing (measured: 14% caught in such hours versus 48% at rest). The gate is the site's 10-minute volume against its own padded weekly median, computed every 15 minutes from ClickHouse by `siteBaseline.ts` and shared through Redis; a site with no baseline or under a week old is never in flood. The gate alone never convicts. Inside a flood, a device cohort convicts when it is at least a quarter of the site's traffic, at least 60% distinct actors, and at least 95% direct β a launch is many devices, several events per visitor and referrers set, and a campaign spreads over real devices so no cohort reaches a quarter. An actor convicts inside a flood at 200 events a day from a hosting address, 1,000 otherwise.+- **Site flood** (`site_flood_oneshot_cohort_10m`, `site_flood_actor_1d`) is the only site-relative logic in the layer. Every other threshold is absolute, so a fleet of disposable identities β one event each, a fresh address per hit, a real browser β can push a four-events-an-hour site to four thousand and trip nothing (measured: 14% caught in such hours versus 48% at rest). The gate is the site's 10-minute volume against its own padded weekly median, computed every 15 minutes from ClickHouse by `siteBaseline.ts` and shared through Redis; a site with no baseline or under a week old is never in flood. The gate alone never convicts. Inside a flood, a device cohort convicts when it is at least a quarter of the site's traffic, at least 60% distinct actors, and at least 95% direct β a launch is many devices, several events per visitor and referrers set, and a campaign spreads over real devices so no cohort reaches a quarter. An actor convicts inside a flood at 200 events a day from a hosting address only β a residential address never convicts on volume, because a person working in a small internal tool all day is that site's whole flood (two were blocked this way on the rule's first day).- **Hosting actor volume** (`hosting_actor_events_1d`) convicts a hosting/datacenter address at 1,000 events a day on its own. The supporting-only daily rule below exists because a residential or office address can be hundreds of people; a datacenter address is not. SASE egress (Cato, Zscaler, Netskope) is exempt by ASN, on top of the many-user-agents guard.- **Cohort version uniformity** convicts, and is the only rule that can catch a paced distributed crawler β one that stays under every per-identity threshold by design. Its key must carry the browser family: without it, Chrome, Safari and Firefox version numbers pool into one distribution, which inflates the distinct-version count and depresses the modal share, i.e. manufactures exactly the two conditions the rule fires on. Its bucket is an hour: the crawler paces itself, and in a one-minute bucket its busiest cohort never came near the volume gate, so the rule did not fire once in two weeks of production.- **Events per actor per day** is supporting-only and scores 1. Its actor is the exact request IP, deliberately *not* the /24 identity bucket the per-visitor rules use β bucketing merges a datacenter range or a SASE egress pool into one actor, and a day is long enough for that to reach any threshold on legitimate traffic. It is additionally dropped outright for an address showing more than three user agents in five minutes, which is what a shared gateway looks like.diff --git a/server/src/services/tracker/botBlocking/anomalyScorer.test.ts b/server/src/services/tracker/botBlocking/anomalyScorer.test.tsindex db701e42a..2f0e2f038 100644--- a/server/src/services/tracker/botBlocking/anomalyScorer.test.ts+++ b/server/src/services/tracker/botBlocking/anomalyScorer.test.ts@@ -633,9 +633,12 @@ describe("site flood (in-process fallback)", () => {expect(result!.convictingReasons.map(reason => reason.rule)).toContain("site_flood_actor_1d");});- it("holds a residential address to the ordinary daily threshold inside a flood", async () => {+ it("never convicts a residential address on volume inside a flood", async () => {+ // A power user of a small internal tool: one home address, all day, a+ // few events a minute, and the site's entire "flood" against a near-zero+ // baseline. Two real people were blocked this way on the rule's first day.let result;- for (let index = 0; index < 250; index++) {+ for (let index = 0; index < 1200; index++) {result = await observeTrackingAnomaly({...baseInput,siteBaseline: quietSite,diff --git a/server/src/services/tracker/botBlocking/anomalyScorer.ts b/server/src/services/tracker/botBlocking/anomalyScorer.tsindex 6f8915dac..c49bf9cb9 100644--- a/server/src/services/tracker/botBlocking/anomalyScorer.ts+++ b/server/src/services/tracker/botBlocking/anomalyScorer.ts@@ -97,6 +97,14 @@ const DAY = 24 * 60 * MINUTE;* flood gate alone never convicts; a cohort or an actor still has to show a* shape no organic audience produces.*+ * The actor rule inside a flood is hosting-only. On the first day in+ * production a residential version of it (1,000 events a day) convicted two+ * real people: power users of small internal tools, clicking and typing+ * through edit pages at a few events a minute, who *were* their site's whole+ * "flood" against a near-zero baseline. A human at a keyboard can produce a+ * thousand events a day on a site they work in; a datacenter address cannot+ * be one.+ ** The gate is the site's 10-minute volume at twenty times its padded weekly* median and at least 100 events. Sites without a baseline, or younger than a* week, never pass it.@@ -108,7 +116,6 @@ const FLOOD_COHORT_MIN_ACTOR_RATIO = 0.6;const FLOOD_COHORT_MIN_DIRECT_SHARE = 0.95;const FLOOD_COHORT_MIN_SITE_SHARE = 0.25;const FLOOD_ACTOR_EVENTS_1D_HOSTING = 200;-const FLOOD_ACTOR_EVENTS_1D_THRESHOLD = ACTOR_EVENTS_1D_THRESHOLD;const TEN_MINUTES = 10 * MINUTE;/**@@ -1037,13 +1044,13 @@ function computeAnomalyResult(counters: AnomalyCounters, context: AnomalyContext});}- if (sharedEgressGuard && !saseExempt) {+ if (context.isHostingAsn && sharedEgressGuard && !saseExempt) {addReason(convictingReasons,"site_flood_actor_1d",4,counters.actorEvents1d,- context.isHostingAsn ? FLOOD_ACTOR_EVENTS_1D_HOSTING : FLOOD_ACTOR_EVENTS_1D_THRESHOLD,+ FLOOD_ACTOR_EVENTS_1D_HOSTING,86400);}