Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
docs(depth): ADR-0011, the 3.4 entry, and the three deviations main lost
Beenu Arora1321ddce9983
diff --git a/DEPTH_PROGRESS.md b/DEPTH_PROGRESS.mdindex 849c039af..22d438e75 100644--- a/DEPTH_PROGRESS.md+++ b/DEPTH_PROGRESS.md@@ -64,9 +64,9 @@ is written. Next free at capture: migration `095`, ADR `0010`| migration `107` | 9.4 — `107_case_war_rooms.sql` | claimed || migration `108` | 9.5 — `108_hunt_reports.sql` | claimed || migration `109` | 9.6 — `109_push_devices.sql` | claimed |-| migration `110` | 3.4 — `110_entity_baselines.sql` | claimed |+| ~~migration `110`~~ | ~~3.4 — `110_entity_baselines.sql`~~ | **released**, ADR-0011 put the baselines in the Redis fusion already runs, so there is no table || migration `111` | 3.7 — `111_detection_coverage.sql` | claimed |-| ADR `0011` | 3.4 — behavioural baselines in CORE | claimed |+| ADR `0011` | 3.4 — behavioural baselines in CORE | **landed**, `docs/decisions/0011-behavioural-baselines-in-core.md`, accepted |Highest migration on `main` at claim time was `101_audit_immutable_nullify.sql`,so `095`--`101` were taken by sibling lanes between the capture above and this@@ -305,6 +305,117 @@ Instead, every selector field, `group-by` and `distinct_by` is checkedagainst the emitted-field namespace with **no ceiling**, which caught threeinvented fields on the first run of the gate.+### D9 -- Five of the 45 "unreachable" rules were never unreachable++`scripts/check_detection_fields.py` carries its own copy of the matcher's+operator suffixes, under a comment saying it mirrors+`detection_matcher.OPERATORS`. It did not. `neq` had been added to the+matcher and not to the gate, so `approver_role_neq: "codeowner"` — which the+engine reads correctly as `approver_role` with `!=` — was reported as a rule+naming a field called `approver_role_neq`. Five rules were counted against+the ratchet for a defect that had already been fixed.++The same comparison in the other direction found a worse one: the gate+listed `not_startswith_any` and `not_endswith`, which the **matcher** did not+implement. Three shipped rules use `path_not_startswith_any`, so the matcher+read the whole string as a field name and all three could never fire — while+the gate, stripping a suffix nothing implemented, reported them reachable.+Both operators are now implemented in the vendored matcher and the canonical+one, the fixture synthesiser knows them, and+`test_the_gate_and_the_matcher_agree_on_every_operator` compares the two+lists in both directions.++### D10 -- 18 rules are retired rather than fixed, and why each one is++The plan says of one family: "build them where cheap, otherwise quarantine+with a reason". Applied to all three families, that leaves 18 rules that need+work belonging to another item, recorded individually in+`scripts/enrichment_decisions.py`:++| kind | n | needs |+|---|---|---|+| `needs-connector-field` | 9 | a field the vendor has and no connector surfaces — depth plan 2.2 |+| `needs-external-source` | 6 | a registry, a credential report or a factor record — 2.5 and beyond |+| `needs-inventory` | 3 | an inventory this platform does not hold (DC machine accounts, privileged OAuth scopes) |++`scripts/check_enrichment_decisions.py` is what stops this being a way to+move the ratchet: it fails if a retired rule's fields are all resolvable.+Proved by retiring a working rule and watching it fail.++The line that mattered most was refusing a plausible substitute.+`domain_age_days` wants a domain's registration age from a registry; the+first-seen store could have answered "when this deployment first saw it" and+the number would have looked right in a diff. It would also have fired the+rule on most of the internet. Five such age-of-a-thing fields are named in a+test that forbids the first-seen store from ever growing them.++### D11 -- Identity privilege reads a table a tenant must populate++The `*_priv` booleans resolve from `identity_nodes.privilege_tier >= 2`, the+column migration 018 created for the question. The table's only writer is+`POST /api/v1/identity-graph/nodes`, so a tenant that has imported no+directory gets **no key at all** and those 18 rules stay silent — they are+reachable "when the tenant has identity data", the same standard this gate+already applies to a vendor-payload field. That is weaker than "reachable on+any deployment" and is stated rather than glossed. Depth plan 2.5's posture+collectors are what will populate it automatically.++Four booleans in the same family are deliberately **not** computed —+`scope_priv`, `act_as_user_priv`, `gpo_link_priv` and `account_is_dc` — and+are among the 18 above. Each has no subject field any source emits, and+guessing a subject would produce a boolean that is confidently wrong rather+than absent.++### D12 -- The "behavioural baseline" family named the wrong piece of work++`check_detection_fields.py` classified an unreachable field into a family so+the headline number would be actionable. Its behavioural pattern was+`_baseline|_deviation|^active_|_is_first_`, and `^active_` is not a+behavioural shape at all. It matched the only two rules left on the ratchet:++| rule | field | what it actually needs |+|---|---|---|+| `det-endpoint-146` | `active_state` | a systemd unit state from a host agent |+| `det-identity-015` | `active_p0_or_p1_incident` | the tenant's ITSM incident queue |++So the gate reported "2 rules need depth plan 3.4" when the true answer was+**zero**, and building 3.4 exactly as specified would not have moved the+number by one. A gate pointing at the wrong piece of work is worse than one+pointing at none, because the named thing gets built and the next reader+concludes the fix failed.++The pattern now matches the shape `baselines.py` actually produces+(`^baseline_|_baseline$|_deviation$`), and a new `external system context`+family carries the ITSM rule. `MAX_UNREACHABLE` stays at **2** — this is a+reclassification and not a fix, and lowering a ceiling on the strength of+relabelling is the move this file exists to refuse.++Two consequences worth stating. The behavioural family is now genuinely+empty, so 3.4 closes no unreachable rule and is not credited with any; it+earns its place by making 3.5's rules possible, which is a different claim.+And D9's lesson repeated one level up: a gate that keeps its own description+of another component drifts from it, and here the drift was in the+classifier rather than in the operator list.++### D13 -- D9, D10 and D11 were written, merged, and lost on `main`++PR #1226 (depth 3.3) added three deviation sections. None of them is on+`main`. The item log still cites "(gate operator drift, D9)" and "18 retired+with a reason (D10)", so the tracker has spent two days pointing at+explanations a reader could not find, and the same merge truncated 3.2's+checklist paragraph mid-sentence and left a stale unchecked duplicate+beneath it.++Nothing was wrong with the branch; the loss happened in a conflict+resolution that took `main`'s side of this file wholesale. That is the+documented failure mode for a file several lanes write to at once, and the+documented remedy — keep both sides' paragraphs, never take one side whole —+is the one that would have prevented it.++All three are restored here verbatim from the merge commit rather than+rewritten from memory, which is why D11 keeps its original number and this+phase's own new deviations take D12 and D13.+## Phase 1: Verdict quality you can publish- [x] **1.1** Balanced, labelled verdict corpus@@ -349,7 +460,23 @@ invented fields on the first run of the gate.`MAX_UNREACHABLE` 45 → 2, and the two that remain both need the 3.4baseline, so they are named rather than hidden. Two matcher operators thatthree shipped rules already used were implemented.-- [ ] **3.4** Behavioural baselines in CORE+- [x] **3.4** Behavioural baselines in CORE — the scoring core moved into+ fusion as `services/fusion/app/services/baselines.py` rather than UEBA+ moving into CORE, measured and recorded in ADR-0011: the UEBA image is+ 145.4 MB on disk with a 63.3 MiB import floor, against zero new+ containers and a measured 2,412 bytes of Redis per principal (200+ principals over 10,000 events, `used_memory` read before and after+ against a real Redis 7). All three families the plan names are there —+ per-principal rarity of actions, resources, client families and+ locations; peer-group deviation; volume per principal and per OAuth app —+ each a typed `AnomalySignal`, never an alert on its own. Wired into+ `DetectionEngine.evaluate_async` beside the 3.3 first-seen store, so a+ CORE install computes them with no `full` service running. 21 tests. The+ reachability gate's "behavioural baseline" family was **narrowed**: it+ matched `^active_`, which swept in a systemd unit state and an ITSM+ lookup and reported both as blocked on a baseline that would never have+ helped either (D12). `MAX_UNREACHABLE` stays at 2, now with accurate+ reasons.- [ ] **3.5** Threat-and-anomaly condition rules- [ ] **3.6** 395 → at least 800 cloud, identity, SaaS and code conditions- [ ] **3.7** Detection engineering agent (gap-closure Phase 9 / parity 6.2)diff --git a/docs/decisions/0011-behavioural-baselines-in-core.md b/docs/decisions/0011-behavioural-baselines-in-core.mdnew file mode 100644index 000000000..101b966b6--- /dev/null+++ b/docs/decisions/0011-behavioural-baselines-in-core.md@@ -0,0 +1,148 @@+# ADR-0011: behavioural baselines in CORE are a fusion module; the UEBA service stays in `full`++- **Status:** accepted+- **Date:** 2026-10-11+- **Related:** [ADR-0010](./0010-entity-and-session-context-in-core.md), which made the same shape of decision for entity and session context; [ADR-0007](./0007-connectors-and-actions-in-core.md), which established that a capability configured and pointed at nothing is the failure to avoid; [ADR-0006](./0006-llm-gateway-in-core.md), which established the measurement method+- **Implements:** depth plan 3.4++## Context++Depth plan 3.4 states the problem and names both options:++> Behavioural scoring needs the `full`-profile UEBA service today. Move its+> scoring core into fusion as a module, or add UEBA to CORE behind an ADR+> that measures it.++The reason it has to move at all is the plan's own "Done when" for Phase 3:++> a CORE install fires a threat-and-anomaly rule on a replayed fixture with+> no `full` service running.++Depth plan 3.5 pairs anomaly signals with threat conditions into ordinary+rules. If the anomaly half of every such rule is only computable in `full`,+then on the profile `make up` starts — the one every quickstart, every+install script and every evaluator uses — those rules read `None` on their+baseline clause and can never fire. That is precisely the shape ADR-0007+closed: content that exists, is counted, and is pointed at nothing.++## What was measured, rather than assumed++Same method as ADR-0006, ADR-0007 and ADR-0010: image size from+`docker image inspect`, resident memory from `docker stats --no-stream`.++Host: macOS, Docker Engine 29.5.2, 15.58 GiB available to the engine.+Date 2026-10-11, `arm64`.++### Option B — add the UEBA service to CORE++| Measure | Value | How |+|---|---|---|+| Image on disk | **145.4 MB** | `docker image inspect ghcr.io/beenuar/aisoc-ueba:latest` |+| Resident at import | **63.3 MiB** | container running `import app.main`, then `docker stats --no-stream` |++The resident figure is a **floor, not an idle cost**. It is the footprint of+importing the application; a running instance adds a SQLAlchemy pool, an+`aiokafka` consumer and its buffers on top. It is recorded as a floor rather+than extrapolated, because the honest version of "we did not run the full+service against live dependencies" is to say so.++The service also brings a second cost that is not memory: UEBA owns its own+Alembic chain and its own tables. CORE would gain a migration chain whose+failure mode is already documented in this repository — four Alembic chains+once shared one `alembic_version` table and two services came up healthy+with zero tables while every command reported success.++### Option A — the scoring core as a fusion module++Zero new containers and zero new images. Fusion already runs a Redis for the+windowed engine's sliding windows and for the Phase 3.3 first-seen store, so+the counting state has a home that costs no new process.++What it does cost is Redis memory, measured directly rather than estimated —+`used_memory` before and after driving `BaselineStore.derived_fields` against+a real Redis 7 container:++| Measure | Value |+|---|---|+| Workload | 200 principals × 50 events = 10,000 events, six attributes each |+| Redis keys created | 1,020 |+| `used_memory` delta | **471 KiB** |+| Per principal | **2,412 bytes** |++At 10,000 principals — larger than most tenants a CORE install serves — that+projects to roughly **24 MB of Redis**, against the 63.3 MiB floor plus an+image, a migration chain and a Kafka consumer group for the alternative.++## Decision++**Take option A.** The behavioural scoring core moves into fusion as+`services/fusion/app/services/baselines.py`. The UEBA service stays in the+`full` profile, unchanged.++The scoring itself is three lines of Welford arithmetic. What makes UEBA a+`full`-profile service is the table, the Alembic chain and the Kafka consumer+around that arithmetic — not the statistics. Moving the statistics is cheap;+moving the scaffolding is what the measurement above prices.++### What CORE gets, and what it does not++CORE gets the three signal families depth plan 3.4 names: per-principal+rarity of actions, resources, client families and locations; peer-group+deviation; and volume anomalies per principal and per OAuth application.+Each is a typed signal that rules read, never an alert on its own.++CORE does **not** get UEBA's per-entity history, its API surface, or its+`ueba.anomalies` topic. A deployment that wants an entity's full behavioural+record over time still runs `full`. Fusion's existing `ueba_signal.py`+consumer is untouched, so a `full` deployment keeps folding the service's+richer score into alert confidence exactly as before — the two coexist+rather than compete.++## Consequences++**The arithmetic now exists twice**, in `services/ueba/app/services/baseline.py`+and in the fusion module. That is a real cost and it is accepted deliberately:+the alternative is a shared package that both services depend on, which makes+CORE's detection path depend on a `full`-profile service's release cadence for+three lines of maths. The fusion copy is pure and has no SQLAlchemy in it,+which is what lets it carry the one correctness property the UEBA copy had to+learn the hard way — see below.++**One bug class is designed out rather than copied.** `_welford_update` in the+UEBA service once edited its argument in place; its caller passed the object+SQLAlchemy had loaded, SQLAlchemy compares loaded-against-new to decide+whether a JSONB column is dirty, found them equal, and omitted the column from+every `UPDATE`. No baseline in that service advanced past its first event, and+the symptom was `features_unscoreable`, which reads as "this entity is too+quiet to score" rather than as a bug. The fusion `welford_add` returns a new+dict and mutates nothing, and a test asserts exactly that.++**`None` means unknown and never normal.** Carried across from the UEBA copy,+which already draws this distinction, because it is the property that decides+whether the module is useful: service accounts and batch jobs converge on a+constant stream, their standard deviation collapses to zero, and every+subsequent value then sits exactly zero deviations from the mean. A `0.0`+there would make the most automated account in the estate the least+suspicious one it is possible to be.++**A short history produces no key at all**, for the same reason the first-seen+store reports nothing during a Redis outage. The first event a principal ever+produces is 100% of their history, so on a fresh install every value is novel;+publishing that would fire every novelty rule on every event until the+baselines warmed.++**The value set per principal is bounded.** A resource name is+attacker-chosen, so an unbounded hash is a memory-exhaustion primitive aimed+at the detection path. Past `MAX_DISTINCT_VALUES` the hash stops admitting new+members while the total still advances, so the baseline degrades into+"stopped learning new values for this principal" rather than into a wrong+denominator or an outage.++**Peer groups are only as good as the group key.** The plan names the Phase+2.5 posture snapshot as the source. Until that lands, the group is taken from+whatever the identity provider already puts on the event — `department`,+`group`, `job_title` — which is what Workspace, Entra and Okta all emit. An+event naming no group yields no peer signal rather than a group of one, and a+group below `MIN_PEER_MEMBERS` distinct principals yields none either: a group+of one is that principal's own baseline wearing a group's name, and comparing+them reports every event as perfectly typical.diff --git a/docs/decisions/README.md b/docs/decisions/README.mdindex 163ed6ca9..c649d8232 100644--- a/docs/decisions/README.md+++ b/docs/decisions/README.md@@ -24,6 +24,7 @@ ADRs are immutable once accepted. If a decision changes, write a new ADR that su| [ADR-0008](./0008-scim-trust-boundary.md) | SCIM is a third-party write surface into identity, and is scoped by its credential | accepted | 2026-09-27 || [ADR-0009](./0009-auto-close-without-an-earned-grant.md) | Whether a tenant with no closure policy may auto-close | proposed | 2026-10-04 || [ADR-0010](./0010-entity-and-session-context-in-core.md) | Entity and session context in CORE is a Postgres store; Neo4j stays in `full` | accepted | 2026-10-11 |+| [ADR-0011](./0011-behavioural-baselines-in-core.md) | Behavioural baselines in CORE are a fusion module; the UEBA service stays in `full` | accepted | 2026-10-11 |## Authoring conventionsdiff --git a/services/fusion/app/services/baselines.py b/services/fusion/app/services/baselines.pyindex 774a2c978..a3a14fd50 100644--- a/services/fusion/app/services/baselines.py+++ b/services/fusion/app/services/baselines.py@@ -261,9 +261,7 @@ class BaselineStore:# -- rarity ------------------------------------------------------------ async def observe_value(- self, tenant: str, subject: str, attribute: str, value: str- ) -> tuple[int, int] | None:+ async def observe_value(self, tenant: str, subject: str, attribute: str, value: str) -> tuple[int, int] | None:"""Count one sighting; return (times seen, total for this attribute).None means the store could not answer. The pair is returned rather@@ -307,9 +305,7 @@ class BaselineStore:# -- volume ------------------------------------------------------------ async def observe_volume(- self, tenant: str, subject: str, kind: str, *, now: float | None = None- ) -> float | None:+ async def observe_volume(self, tenant: str, subject: str, kind: str, *, now: float | None = None) -> float | None:"""Count one event into this hour; return the z-score of the *previous* hour.Scoring the hour in progress would compare a partial count against@@ -348,9 +344,7 @@ class BaselineStore:# -- peer group -------------------------------------------------------- async def observe_peer(- self, tenant: str, group: str, subject: str, value: float- ) -> float | None:+ async def observe_peer(self, tenant: str, group: str, subject: str, value: float) -> float | None:"""Fold one member's observation into the group; return its deviation.The member count is tracked as a set rather than inferred from the@@ -380,9 +374,7 @@ class BaselineStore:# -- the engine's entry point ----------------------------------------- async def derived_fields(- self, tenant: str, event: dict[str, Any], *, now: float | None = None- ) -> dict[str, Any]:+ async def derived_fields(self, tenant: str, event: dict[str, Any], *, now: float | None = None) -> dict[str, Any]:"""The baseline fields this event produces, for the detection engine.Mirrors `FirstSeenStore.derived_fields` deliberately: one awaited@@ -433,9 +425,7 @@ class BaselineStore:return out- async def signals(- self, tenant: str, event: dict[str, Any], *, now: float | None = None- ) -> list[AnomalySignal]:+ async def signals(self, tenant: str, event: dict[str, Any], *, now: float | None = None) -> list[AnomalySignal]:"""The same observations as typed signals, for consumers that want them.Depth plan 3.4 requires each anomaly to be *a typed signal, not an@@ -455,9 +445,7 @@ class BaselineStore:value = _first_present(event, PEER_GROUP_FIELDS) or ""else:kind, attribute = "volume", field- value = principal if field == "baseline_principal_volume_z" else (- _first_present(event, OAUTH_APP_FIELDS) or ""- )+ value = principal if field == "baseline_principal_volume_z" else (_first_present(event, OAUTH_APP_FIELDS) or "")out.append(AnomalySignal(kind=kind,diff --git a/services/fusion/tests/test_baselines.py b/services/fusion/tests/test_baselines.pynew file mode 100644index 000000000..1d90176df--- /dev/null+++ b/services/fusion/tests/test_baselines.py@@ -0,0 +1,316 @@+"""Per-principal behavioural baselines, and the three ways they refuse.++Depth plan 3.4. The assertions below are about behaviour under the+conditions that actually occur — a principal with no history, one with+enough, a service account whose variance has collapsed, a tenant boundary,+and Redis being unavailable — rather than about a helper returning a float.++Three of them are the ones worth having, because each pins a refusal that a+plausible-looking implementation would get wrong in the unsafe direction:++* **A short history contributes no key.** The first event a principal ever+ produces is 100% of their history, so every value looks rare. Reporting+ that would make every new account fire every novelty rule.+* **A degenerate distribution scores `None`, not `0.0`.** Service accounts+ converge on a constant and their standard deviation collapses, so a+ z-score of zero would make the most automated account in the estate the+ least suspicious one it is possible to be.+* **An unreadable store contributes no key.** Same reason as the first-seen+ store: absent means the clause does not match, which is quiet.+"""++from __future__ import annotations++import importlib.util+from pathlib import Path+from typing import Any++import pytest+from app.services.baselines import (+ DERIVED_FIELDS,+ MIN_PEER_MEMBERS,+ MIN_VOLUME_SAMPLES,+ RARITY_FIELDS,+ BaselineStore,+ resolve_principal,+ welford_add,+ z_score,+)++_REPO = Path(__file__).resolve().parents[3]+++class _FakeRedis:+ """The hash and set operations the store actually calls."""++ def __init__(self) -> None:+ self.hashes: dict[str, dict[str, str]] = {}+ self.sets: dict[str, set[str]] = {}++ async def hincrby(self, key: str, field: str, amount: int = 1) -> int:+ bucket = self.hashes.setdefault(key, {})+ value = int(bucket.get(field, "0")) + amount+ bucket[field] = str(value)+ return value++ async def hget(self, key: str, field: str) -> Any:+ return self.hashes.get(key, {}).get(field)++ async def hset(self, key: str, field: str, value: str) -> int:+ self.hashes.setdefault(key, {})[field] = value+ return 1++ async def hgetall(self, key: str) -> dict[str, str]:+ return dict(self.hashes.get(key, {}))++ async def hlen(self, key: str) -> int:+ return len(self.hashes.get(key, {}))++ async def sadd(self, key: str, member: str) -> int:+ self.sets.setdefault(key, set()).add(member)+ return 1++ async def scard(self, key: str) -> int:+ return len(self.sets.get(key, set()))++ async def expire(self, key: str, ttl: int) -> bool:+ return True+++class _NoRedis:+ def __getattr__(self, _name: str) -> Any:+ async def _boom(*_args: Any, **_kwargs: Any) -> Any:+ raise ConnectionError("redis is down")++ return _boom+++async def _warm(store: BaselineStore, tenant: str, event: dict[str, Any], times: int) -> None:+ """Give a principal enough history that rarity is reportable."""+ for _ in range(times):+ await store.derived_fields(tenant, event, now=1000.0)+++# -- rarity -----------------------------------------------------------------+++async def test_a_principal_with_too_little_history_gets_no_rarity() -> None:+ """Every value is novel on day one; saying so would fire on everything."""+ store = BaselineStore(_FakeRedis())+ fields = await store.derived_fields("t-1", {"user": "alice", "event_name": "GetObject"}, now=1000.0)+ assert "baseline_action_rarity" not in fields+++async def test_a_routine_action_scores_near_zero_and_a_novel_one_near_one() -> None:+ store = BaselineStore(_FakeRedis(), min_observations=20)+ routine = {"user": "alice", "event_name": "GetObject"}+ await _warm(store, "t-1", routine, 40)++ common = await store.derived_fields("t-1", routine, now=1000.0)+ assert common["baseline_action_rarity"] < 0.05++ novel = await store.derived_fields("t-1", {"user": "alice", "event_name": "CreateAccessKey"}, now=1000.0)+ # 1 sighting out of 42 observations: rare, but bounded by how much+ # history exists rather than asserted as a flat 1.0.+ assert 0.9 < novel["baseline_action_rarity"] < 1.0+++async def test_rarity_is_per_principal_not_per_tenant() -> None:+ """Alice's routine is not Bob's, which is the whole point of a baseline."""+ store = BaselineStore(_FakeRedis(), min_observations=20)+ await _warm(store, "t-1", {"user": "alice", "event_name": "CreateAccessKey"}, 40)+ await _warm(store, "t-1", {"user": "bob", "event_name": "GetObject"}, 40)++ for_bob = await store.derived_fields("t-1", {"user": "bob", "event_name": "CreateAccessKey"}, now=1000.0)+ assert for_bob["baseline_action_rarity"] > 0.9+++async def test_tenants_do_not_share_baselines() -> None:+ store = BaselineStore(_FakeRedis(), min_observations=20)+ await _warm(store, "t-1", {"user": "alice", "event_name": "GetObject"}, 40)+ other = await store.derived_fields("t-2", {"user": "alice", "event_name": "GetObject"}, now=1000.0)+ assert "baseline_action_rarity" not in other+++async def test_the_value_set_per_principal_is_bounded() -> None:+ """An attacker-chosen resource name must not be a memory primitive."""+ store = BaselineStore(_FakeRedis(), min_observations=1)+ for index in range(700):+ await store.derived_fields("t-1", {"user": "alice", "resource": f"obj-{index}"}, now=1000.0)+ key = "aisoc:bl:t-1:alice:baseline_resource_rarity"+ assert len(store._redis.hashes[key]) <= 514 # cap, plus the total field+++# -- volume -----------------------------------------------------------------+++async def test_the_hour_in_progress_is_not_scored() -> None:+ """A partial count against complete ones calls every event at :01 odd."""+ store = BaselineStore(_FakeRedis())+ first = await store.derived_fields("t-1", {"user": "alice"}, now=3600.0)+ assert "baseline_principal_volume_z" not in first+++async def test_a_volume_spike_scores_once_the_hour_completes() -> None:+ store = BaselineStore(_FakeRedis())+ hour = 3600.0+ # Quiet hours of 8-12 events. The counts must *vary*: a principal who+ # does exactly ten things an hour has zero variance, and the degenerate+ # guard then refuses to score them at all — correctly, and it is what+ # the first draft of this fixture accidentally measured.+ for index in range(MIN_VOLUME_SAMPLES + 2):+ for _ in range(8 + (index % 5)):+ await store.derived_fields("t-1", {"user": "alice"}, now=hour * (index + 1))+ # A loud hour, then the next hour's first event scores the loud one.+ for _ in range(400):+ await store.derived_fields("t-1", {"user": "alice"}, now=hour * 40)+ scored = await store.derived_fields("t-1", {"user": "alice"}, now=hour * 41)+ assert scored["baseline_principal_volume_z"] > 3.0+++async def test_an_oauth_app_is_baselined_separately_from_the_user() -> None:+ """A rogue grant acts as a user, so the user's own volume looks normal."""+ store = BaselineStore(_FakeRedis())+ event = {"user": "alice", "oauth_app": "backup-tool"}+ hour = 3600.0+ for index in range(MIN_VOLUME_SAMPLES + 2):+ await store.derived_fields("t-1", event, now=hour * (index + 1))+ key = "aisoc:bl:t-1:backup-tool:vol:oauth_app"+ assert key in store._redis.hashes+++# -- peer groups ------------------------------------------------------------+++async def test_a_group_smaller_than_the_floor_yields_no_peer_signal() -> None:+ """A group of one is that principal's own baseline wearing a group name."""+ store = BaselineStore(_FakeRedis())+ fields = await store.derived_fields("t-1", {"user": "alice", "department": "finance"}, now=1000.0)+ assert "baseline_peer_deviation" not in fields+++async def test_the_peer_floor_counts_members_not_observations() -> None:+ """One noisy principal must not satisfy the minimum on its own."""+ store = BaselineStore(_FakeRedis())+ for _ in range(MIN_PEER_MEMBERS * 10):+ await store.derived_fields("t-1", {"user": "alice", "department": "finance"}, now=1000.0)+ fields = await store.derived_fields("t-1", {"user": "alice", "department": "finance"}, now=1000.0)+ assert "baseline_peer_deviation" not in fields+++# -- the refusals -----------------------------------------------------------+++def test_a_degenerate_distribution_is_unknown_and_not_normal() -> None:+ """A service account's variance collapses; 0.0 would read as all-clear."""+ stats: dict[str, float] = {}+ for _ in range(50):+ stats = welford_add(stats, 10.0)+ assert z_score(stats, 9999.0, min_samples=5) is None+++def test_a_short_history_is_unknown() -> None:+ stats: dict[str, float] = {}+ for value in (1.0, 2.0, 3.0):+ stats = welford_add(stats, value)+ assert z_score(stats, 100.0, min_samples=12) is None+++def test_welford_mutates_nothing() -> None:+ """The ueba copy edited in place, so no baseline there ever advanced."""+ stats = {"count": 1.0, "mean": 5.0, "m2": 0.0}+ before = dict(stats)+ welford_add(stats, 9.0)+ assert stats == before+++async def test_an_unreadable_store_contributes_no_key() -> None:+ store = BaselineStore(_NoRedis())+ assert await store.derived_fields("t-1", {"user": "alice", "event_name": "x"}, now=1000.0) == {}+++async def test_an_event_naming_no_principal_contributes_no_key() -> None:+ """A baseline with no subject is not a baseline."""+ store = BaselineStore(_FakeRedis())+ assert await store.derived_fields("t-1", {"event_name": "GetObject"}, now=1000.0) == {}+++async def test_a_tenantless_event_is_not_recorded() -> None:+ """Without a tenant the key is shared, which is a cross-tenant read."""+ redis = _FakeRedis()+ store = BaselineStore(redis)+ assert await store.derived_fields("", {"user": "alice", "event_name": "x"}, now=1000.0) == {}+ assert redis.hashes == {}+++def test_a_principal_is_resolved_case_insensitively() -> None:+ assert resolve_principal({"user": " Alice "}) == "alice"+ assert resolve_principal({"actor_email": "[email protected]"}) == "[email protected]"+ assert resolve_principal({"event_name": "GetObject"}) is None+++# -- typed signals ----------------------------------------------------------+++async def test_each_observation_is_a_typed_signal() -> None:+ """3.4 requires a typed signal, not an alert — so the kind must survive."""+ store = BaselineStore(_FakeRedis(), min_observations=20)+ routine = {"user": "alice", "event_name": "GetObject"}+ await _warm(store, "t-1", routine, 40)+ signals = await store.signals("t-1", routine, now=1000.0)+ assert signals+ assert {s.kind for s in signals} <= {"rarity", "volume", "peer"}+ assert all(s.subject == "alice" for s in signals)+++# -- drift ------------------------------------------------------------------+++def test_the_reachability_gate_mirrors_this_module() -> None:+ """A new baseline field must be added in both places or it is invisible.++ The direction that bites is the one the matcher's operator list got+ wrong (deviation D9): the gate kept its own copy, the module grew a+ field, and the gate went on reporting rules unreachable that were not.+ Asserting equality rather than containment catches both directions.+ """+ spec = importlib.util.spec_from_file_location("cdf_baselines", _REPO / "scripts" / "check_detection_fields.py")+ assert spec and spec.loader+ gate = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(gate)+ assert gate._BASELINE_DERIVED_FIELDS == DERIVED_FIELDS+++def test_every_rarity_field_is_declared_as_derived() -> None:+ assert set(RARITY_FIELDS) <= DERIVED_FIELDS+++def test_the_narrowed_family_no_longer_claims_unrelated_fields() -> None:+ """`^active_` swept in a systemd unit state and an ITSM lookup.++ Both were reported as blocked on a baseline that would never have+ helped either, so the family read "2 rules need 3.4" when the answer+ was zero. Pinned so the convenient-looking broad pattern cannot return.+ """+ spec = importlib.util.spec_from_file_location("cdf_family", _REPO / "scripts" / "check_detection_fields.py")+ assert spec and spec.loader+ gate = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(gate)+ assert gate._family("active_state") != "behavioural baseline"+ assert gate._family("active_p0_or_p1_incident") == "external system context"+ for field in DERIVED_FIELDS:+ assert gate._family(field) == "behavioural baseline"