Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

depth 3.2: ordered sequences in the windowed engine, and a Sigma correlation importer (#1224)

Beenu Arora51b60d484171
diff --git a/.github/workflows/validate-detections.yml b/.github/workflows/validate-detections.yml
index 5f18ccced..993bf9717 100644
--- a/.github/workflows/validate-detections.yml
+++ b/.github/workflows/validate-detections.yml
@@ -111,6 +111,19 @@ jobs:
- name: Prove the windowed-translation gate is not vacuous
run: python3 scripts/check_windowed_translation.py --self-test
+ # The Sigma compiler translated rule documents and had never looked at a
+ # `correlation:` block. The two facts hid each other: there was no
+ # importer, and the 3,132 imported documents contain no correlation for
+ # it to have failed on. This compiles every correlation document into
+ # the windowed engine or refuses it with a reason, and fails a rule
+ # whose selector or group-by names a field nothing emits — with no
+ # ceiling, because the corpus is new and starts clean.
+ - name: Verify Sigma correlations compile into the windowed engine
+ run: python3 scripts/check_sigma_correlations.py --check
+
+ - name: Prove the Sigma-correlation gate is not vacuous
+ run: python3 scripts/check_sigma_correlations.py --self-test
+
# Recompiles every imported Sigma rule AND re-proves it: each shipped
# rule is replayed, as a vendor-shaped event, through its connector's
# real normalize() and the real engine. A rule that stops firing — because
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 614bafa54..5a178b881 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -103,6 +103,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
The published executable figure falls 2,603 → 2,529 because 74 rules that
could not fire stopped being counted as executable; the windowed corpus is
now published as its own figure beside it rather than going uncounted.
+- **Ordered and unordered sequences in the windowed engine, and a Sigma
+ correlation importer that feeds it.** A threshold cannot express an
+ ordering, and the ordering is usually the detection: "fifty failed logons"
+ is an attempt, "a failed logon then a success for the same account" is a
+ compromise. `SequenceRule` stages two or more selectors for one entity
+ inside one window, ordered by **event time** rather than arrival — almost
+ every connector here polls, so a batch arrives in the vendor's order and
+ the event that starts a sequence routinely lands after the one that
+ finishes it. One event advances at most one stage, which a per-event token
+ enforces rather than a timestamp comparison, because two distinct events
+ routinely share a second. `ordered: false` gives Sigma's unordered
+ `temporal` semantics from the same rule type.
+ `scripts/sigma_correlation.py` compiles `event_count`, `value_count`,
+ `temporal` and `temporal_ordered` documents into those forms and **refuses
+ rather than approximates** the rest: a `lt`/`lte` condition is a rarity
+ signal needing a baseline, a multi-field `group-by` has no single entity,
+ `aliases` need field mapping the engine does not have. The importer had
+ never been written and the imported corpus contains no correlation
+ document, so the two gaps hid each other; the rules under
+ `detections/sigma-correlations/` are hand-authored in the upstream format
+ to give it a corpus and are labelled first-party content rather than
+ imported coverage. `scripts/check_sigma_correlations.py` fails a
+ correlation whose selector or `group-by` names a field nothing emits, with
+ no ceiling.
- **Attribute conditions, time-boxed elevation and workload identities, all
inside the one permission path.** The placement is the item, not a detail:
conditions are applied by `CurrentUser.require_permission` after whichever
diff --git a/DEPTH_PROGRESS.md b/DEPTH_PROGRESS.md
index 3f11f8389..6ca5c7853 100644
--- a/DEPTH_PROGRESS.md
+++ b/DEPTH_PROGRESS.md
@@ -232,6 +232,52 @@ one engine that could not fire them to another. Both passes are now applied in
`evaluate()`, which takes the overlay the consumer had already resolved for
the stateless engine.
+### D7 -- There is no "existing watermark" in fusion to order sequences by
+
+Item 3.2 says to order sequences "by event time with the existing
+watermark". No watermark exists anywhere in the detection path: the only two
+in the tree belong to `services/actions`' shadow-reconcile router and the
+dead-letter replay, which are unrelated subsystems. Rather than invent one or
+quietly order by arrival, the engine orders strictly by event time and
+declares its own bound,
+`SEQUENCE_REORDER_TOLERANCE_SECONDS = 300`, for how late an event may arrive
+and still be stitched into a sequence. The bound is stated in the module
+rather than inherited from something that does not exist.
+
+Ordering by arrival would have been the easy implementation and is wrong
+here: almost every connector in this tree **polls**, so a batch arrives in
+the vendor's order and the event that starts a sequence routinely lands after
+the one that finishes it. `test_a_batch_delivered_out_of_arrival_order_still_fires`
+is the regression test for that, and reverting the event-time read makes it
+and two others fail.
+
+### D8 -- No upstream Sigma correlation rule exists in this tree to import
+
+`detections/sigma-imports/` holds 3,132 rule documents and **not one carries
+a `correlation:` block**, so the missing importer and the missing corpus hid
+each other: there was nothing for an importer to have failed on. There is
+also no fetcher for the upstream corpus (`scripts/` has `compile_sigma_ruleset.py`,
+`sigma_compiler.py` and `sigma_proof_event.py`, none of which downloads
+anything).
+
+So the importer is built and exercised against **first-party** correlation
+documents hand-authored in the upstream format under
+`detections/sigma-correlations/`, four accepted and two refused. The figure
+they contribute is first-party content and is labelled that way in the truth
+table and in the corpus README; it is not imported coverage. Vendoring the
+upstream correlation corpus needs a fetcher and a licence review, and is not
+attempted here.
+
+One standard is weaker here than for the stateless Sigma imports, and stays
+weaker: those are replayed through the real connector `normalize()` by an
+out-of-process worker, and these are replayed through the real **windowed
+engine** only. The connector step is not applied, because importing
+`services/connectors` into a fusion test process shadows fusion's own `app`
+package — the documented reason `compile_sigma_ruleset.py` uses a subprocess.
+Instead, every selector field, `group-by` and `distinct_by` is checked
+against the emitted-field namespace with **no ceiling**, which caught three
+invented fields on the first run of the gate.
+
## Phase 1: Verdict quality you can publish
- [x] **1.1** Balanced, labelled verdict corpus
@@ -257,6 +303,9 @@ the stateless engine.
- [x] **3.1** Translate the 74 windowed `det-*` rules — 50 translated, 24
refused with a reason. `MAX_UNREACHABLE` 119 → 45.
+- [x] **3.2** Ordered sequences and Sigma correlations — the windowed engine
+ stages ordered and unordered sequences; all four translatable Sigma
+ correlation types compile. See D7 and D8.
- [ ] **3.2** Ordered sequences and Sigma correlations
- [ ] **3.3** Enrichment inputs (parity 5.5)
- [ ] **3.4** Behavioural baselines in CORE
diff --git a/README.md b/README.md
index ca0d9550a..114202e51 100644
--- a/README.md
+++ b/README.md
@@ -111,7 +111,7 @@ of those are partial: field matches carried, thresholds did not
## How it works
Ingest normalizes to a common shape and Kafka carries it, then
-fusion runs 2529 executable detection rules, of 6991 on disk, plus 68 sliding-window rules that count across events, applies **your tenant's own tuning**
+fusion runs 2529 executable detection rules, of 6991 on disk, plus 72 sliding-window rules that count across events, applies **your tenant's own tuning**
on top — the disables, floors and suppressions the console writes, so a rule you turned off actually stops firing — and decides what
becomes an alert. Correlation groups related alerts, an agent investigates and writes its reasoning
to the Investigation Ledger, and a playbook may start from the result. Separately, new threat
diff --git a/apps/web/src/data/corpus-stats.json b/apps/web/src/data/corpus-stats.json
index e84df4df7..9bbc8e5de 100644
--- a/apps/web/src/data/corpus-stats.json
+++ b/apps/web/src/data/corpus-stats.json
@@ -1,6 +1,6 @@
{
"executable": 2529,
- "windowed": 68,
+ "windowed": 72,
"onDisk": 6991,
"quarantined": 4462,
"plugins": 80,
diff --git a/detections/sigma-correlations/README.md b/detections/sigma-correlations/README.md
new file mode 100644
index 000000000..de5bf368f
--- /dev/null
+++ b/detections/sigma-correlations/README.md
@@ -0,0 +1,45 @@
+# Sigma correlation rules
+
+Correlation documents in the [Sigma correlations](https://github.com/SigmaHQ/sigma-specification)
+format, compiled into the windowed engine by `scripts/sigma_correlation.py`
+and exported into `services/fusion/app/data/windowed_ruleset.json` by
+`scripts/export_windowed_ruleset.py`.
+
+## These are first-party, not upstream imports
+
+**No upstream SigmaHQ correlation rule is vendored here, and none was at the
+time this directory was created** — `detections/sigma-imports/` holds 3,132
+rule documents and not one carries a `correlation:` block, so the import path
+had no corpus to run against and nothing would have exercised it. The rules
+here are hand-authored under this repository's own licence, in the upstream
+format, to give the compiler real input. Treat the figure they contribute as
+first-party content, not as imported coverage.
+
+Two consequences worth stating rather than discovering:
+
+* **`group-by` names a field this platform normalizes**, not always the
+ vendor's own spelling. Upstream would write `eventSource`; a rule that has
+ to accumulate per principal needs `user_arn`, which is what the CloudTrail
+ connector emits. `scripts/check_sigma_correlations.py` fails a rule whose
+ `group-by` or selector names a field nothing emits, so this stays checkable
+ rather than aspirational.
+* **A correlation is not replayed through its connector's `normalize()`.**
+ The stateless Sigma imports are (`scripts/compile_sigma_ruleset.py` runs an
+ out-of-process worker per rule); these are replayed through the real
+ windowed engine instead, in
+ `services/fusion/tests/test_sigma_correlation_replay.py`. That proves the
+ compiled rule fires on an event of the shape its selector describes. It
+ does not prove a given vendor populates those fields on a given event.
+
+## Layout
+
+One file per correlation, multi-document YAML as the specification describes:
+the base rules first, each with a `name:`, then the correlation document
+referencing them.
+
+## Refusals are part of the corpus
+
+Files under `_refused/` are correlations the engine cannot express. They are
+kept, not deleted, because a refusal with a reason is the useful artefact —
+and because the compiler's taxonomy needs real inputs to be worth anything.
+`scripts/check_sigma_correlations.py --report` prints which, and why.
diff --git a/detections/sigma-correlations/_refused/aws-login-failures-by-principal-and-region.yml b/detections/sigma-correlations/_refused/aws-login-failures-by-principal-and-region.yml
new file mode 100644
index 000000000..772deec2f
--- /dev/null
+++ b/detections/sigma-correlations/_refused/aws-login-failures-by-principal-and-region.yml
@@ -0,0 +1,34 @@
+title: AWS console sign-in failure
+name: aws-console-login-failed-multi
+id: aisoc-corr-base-aws-console-login-failed-multi
+status: experimental
+description: One failed AWS console sign-in.
+logsource:
+ product: aws
+ service: cloudtrail
+detection:
+ selection:
+ eventSource: signin.amazonaws.com
+ eventName: ConsoleLogin
+ errorMessage: 'Failed authentication'
+ condition: selection
+---
+title: AWS sign-in failures grouped by principal and region
+id: aisoc-corr-aws-login-failures-principal-and-region
+status: experimental
+description: >-
+ Kept as a refusal. The engine accumulates against one entity; choosing either
+ field would group by something the author did not ask for, and silently widen
+ or narrow every count.
+level: medium
+category: cloud
+correlation:
+ type: event_count
+ rules:
+ - aws-console-login-failed-multi
+ group-by:
+ - user_arn
+ - awsRegion
+ timespan: 10m
+ condition:
+ gt: 20
diff --git a/detections/sigma-correlations/_refused/okta-rare-admin-action.yml b/detections/sigma-correlations/_refused/okta-rare-admin-action.yml
new file mode 100644
index 000000000..726597ea7
--- /dev/null
+++ b/detections/sigma-correlations/_refused/okta-rare-admin-action.yml
@@ -0,0 +1,31 @@
+title: Okta administrative action
+name: okta-admin-action
+id: aisoc-corr-base-okta-admin-action
+status: experimental
+description: One Okta administrative API call.
+logsource:
+ product: okta
+ service: okta
+detection:
+ selection:
+ eventType: user.account.privilege.grant
+ condition: selection
+---
+title: An account performing unusually few administrative actions
+id: aisoc-corr-okta-rare-admin-action
+status: experimental
+description: >-
+ Kept as a refusal, not as a rule. A `lt` condition fires when little
+ happened, which needs a store of what is normal for that account rather than
+ a counter over a window.
+level: low
+category: identity
+correlation:
+ type: event_count
+ rules:
+ - okta-admin-action
+ group-by:
+ - actor_email
+ timespan: 1h
+ condition:
+ lt: 5
diff --git a/detections/sigma-correlations/aws-console-login-failure-burst.yml b/detections/sigma-correlations/aws-console-login-failure-burst.yml
new file mode 100644
index 000000000..8a94e4744
--- /dev/null
+++ b/detections/sigma-correlations/aws-console-login-failure-burst.yml
@@ -0,0 +1,35 @@
+title: AWS IAM access key created
+name: aws-iam-create-access-key
+id: aisoc-corr-base-aws-iam-create-access-key
+status: experimental
+description: One IAM CreateAccessKey call. Routine during onboarding; the correlation below is the detection.
+logsource:
+ product: aws
+ service: cloudtrail
+detection:
+ selection:
+ eventSource: iam.amazonaws.com
+ eventName: CreateAccessKey
+ condition: selection
+---
+title: One principal minted several AWS access keys in a burst
+id: aisoc-corr-aws-access-key-minting-burst
+status: experimental
+description: >-
+ Six or more IAM access keys created by the same principal inside ten minutes.
+ Key creation is legitimate and common, so the detection is the rate: an
+ onboarding script creates one key per user, and a principal creating several
+ in minutes is either a misconfigured loop or persistence being established.
+level: high
+category: cloud
+tags:
+ - attack.t1098.001
+correlation:
+ type: event_count
+ rules:
+ - aws-iam-create-access-key
+ group-by:
+ - user_arn
+ timespan: 10m
+ condition:
+ gt: 5
diff --git a/detections/sigma-correlations/aws-principal-across-many-regions.yml b/detections/sigma-correlations/aws-principal-across-many-regions.yml
new file mode 100644
index 000000000..d8d5b6eb3
--- /dev/null
+++ b/detections/sigma-correlations/aws-principal-across-many-regions.yml
@@ -0,0 +1,35 @@
+title: AWS API call
+name: aws-api-call
+id: aisoc-corr-base-aws-api-call
+status: experimental
+description: Any CloudTrail management-plane call. Meaningless alone; the correlation counts distinct regions.
+logsource:
+ product: aws
+ service: cloudtrail
+detection:
+ selection:
+ cloud_platform: aws
+ condition: selection
+---
+title: One principal acting across an unusual number of AWS regions
+id: aisoc-corr-aws-principal-region-fanout
+status: experimental
+description: >-
+ Six or more distinct regions touched by one principal inside fifteen minutes.
+ Distinct regions rather than call volume: an estate uses two or three
+ regions, and a principal sweeping the rest is the shape of enumeration or of
+ resource hijacking looking for quota.
+level: medium
+category: cloud
+tags:
+ - attack.t1580
+correlation:
+ type: value_count
+ rules:
+ - aws-api-call
+ group-by:
+ - user_arn
+ field: awsRegion
+ timespan: 15m
+ condition:
+ gt: 5
diff --git a/detections/sigma-correlations/azure-role-grant-and-key-creation.yml b/detections/sigma-correlations/azure-role-grant-and-key-creation.yml
new file mode 100644
index 000000000..d1a3f8309
--- /dev/null
+++ b/detections/sigma-correlations/azure-role-grant-and-key-creation.yml
@@ -0,0 +1,45 @@
+title: Entra directory role assigned
+name: azure-directory-role-assigned
+id: aisoc-corr-base-azure-role-assigned
+status: experimental
+description: One Entra directory role assignment.
+logsource:
+ product: azure
+ service: auditlogs
+detection:
+ selection:
+ operationName: Add member to role
+ condition: selection
+---
+title: Azure resource lock deleted
+name: azure-resource-lock-deleted
+id: aisoc-corr-base-azure-lock-deleted
+status: experimental
+description: One Azure management lock removal.
+logsource:
+ product: azure
+ service: activitylogs
+detection:
+ selection:
+ operationName: Microsoft.Authorization/locks/delete
+ condition: selection
+---
+title: One actor both assigned a directory role and removed a resource lock
+id: aisoc-corr-azure-role-grant-and-lock-removal
+status: experimental
+description: >-
+ Two administrative actions by the same actor inside thirty minutes, in either
+ order. Each is legitimate on its own and neither is rare; together they are
+ the shape of an actor clearing the way before changing something protected.
+level: medium
+category: cloud
+tags:
+ - attack.t1098
+correlation:
+ type: temporal
+ rules:
+ - azure-directory-role-assigned
+ - azure-resource-lock-deleted
+ group-by:
+ - actor
+ timespan: 30m
diff --git a/detections/sigma-correlations/okta-mfa-denials-then-success.yml b/detections/sigma-correlations/okta-mfa-denials-then-success.yml
new file mode 100644
index 000000000..ddf5363c3
--- /dev/null
+++ b/detections/sigma-correlations/okta-mfa-denials-then-success.yml
@@ -0,0 +1,47 @@
+title: Okta MFA verification push sent
+name: okta-mfa-push-sent
+id: aisoc-corr-base-okta-mfa-push
+status: experimental
+description: One Okta push factor verification. Routine on its own.
+logsource:
+ product: okta
+ service: okta
+detection:
+ selection:
+ eventType: system.push.send_factor_verify_push
+ condition: selection
+---
+title: Okta session started
+name: okta-session-start
+id: aisoc-corr-base-okta-session-start
+status: experimental
+description: One Okta session start.
+logsource:
+ product: okta
+ service: okta
+detection:
+ selection:
+ eventType: user.session.start
+ condition: selection
+---
+title: Okta MFA push followed by a session for the same account
+id: aisoc-corr-okta-mfa-push-then-session
+status: experimental
+description: >-
+ A push factor verification and then a session start for the same account
+ inside fifteen minutes. The ordering is the detection: pushes alone are an
+ attempt, and a session that follows them is the attempt succeeding. Pair this
+ with the push-volume rule rather than using it alone — one push followed by a
+ session is the ordinary path a user takes every morning.
+level: medium
+category: identity
+tags:
+ - attack.t1621
+correlation:
+ type: temporal_ordered
+ rules:
+ - okta-mfa-push-sent
+ - okta-session-start
+ group-by:
+ - actor_email
+ timespan: 15m
diff --git a/docs/detections/truth-table.md b/docs/detections/truth-table.md
index b1e8e3a4c..24cffc82e 100644
--- a/docs/detections/truth-table.md
+++ b/docs/detections/truth-table.md
@@ -23,7 +23,7 @@ lever that reaches the engine is adding a spec and re-running
| **executable (loaded by the engine)** | **2529** |
| non-executable (provenance/coverage only) | 4462 |
| — of which: enabled, but no compiled spec | 0 |
-| windowed rules in `windowed_ruleset.json` | 68 |
+| windowed rules in `windowed_ruleset.json` | 72 |
The windowed figure is its own row rather than part of the headline because
a windowed rule is a different kind of thing: it counts events or distinct
@@ -42,6 +42,12 @@ from the original's own clauses and replayed through the real engine
that family could not be expressed as a count over a window and are refused
with a reason that travels into each rule's `quarantine_reason`.
+Four more are Sigma correlation documents compiled by
+`scripts/sigma_correlation.py`. They are **first-party content, not imported
+coverage**: no upstream correlation rule is vendored in this tree, so the
+import path had no corpus and the rules under `detections/sigma-correlations/`
+were hand-authored in the upstream format to give it one.
+
## By tier
| tier | on disk | executable | enabled but not loaded |
diff --git a/scripts/check_sigma_correlations.py b/scripts/check_sigma_correlations.py
new file mode 100644
index 000000000..6af1d8c8b
--- /dev/null
+++ b/scripts/check_sigma_correlations.py
@@ -0,0 +1,259 @@
+#!/usr/bin/env python3
+"""CI gate: every Sigma correlation compiles into the windowed engine, or is refused.
+
+The gap this closes
+-------------------
+`scripts/compile_sigma_ruleset.py` translates Sigma *rule* documents into the
+stateless matcher. It has never looked at a `correlation:` block, and the
+3,132 imported rule documents do not contain one, so the two facts hid each
+other: there was no importer and no corpus for it to fail on. A correlation
+quietly ignored by an importer is the worst shape available — the rule is in
+the tree, it is in the count of what was imported, and it detects nothing.
+
+What this enforces
+------------------
+* every correlation document in `detections/sigma-correlations/` either
+ compiles or carries a refusal reason, and the two sets do not overlap;
+* a compiled correlation's selector fields, `group-by` and `distinct_by` are
+ names some connector, ingest template or the Go normalizer actually emits —
+ the same standard `check_detection_fields.py` holds the native corpus to,
+ applied here with **no ceiling**, because this corpus is new and starts
+ clean;
+* everything under `_refused/` is in fact refused, so a file cannot be parked
+ there to dodge the checks above;
+* the compiled output is what `windowed_ruleset.json` ships.
+
+Usage:
+ python3 scripts/check_sigma_correlations.py # enforce
+ python3 scripts/check_sigma_correlations.py --check # same, for CI
+ python3 scripts/check_sigma_correlations.py --report # the taxonomy
+ python3 scripts/check_sigma_correlations.py --self-test # prove the gate works
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+import yaml
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+from gate_toolkit import SKELETON, refuses_an_empty_tree, repo_root, self_test_main
+from sigma_compiler import Refusal, compile_rule
+from sigma_correlation import CorrelationRefusal, compile_correlation
+
+ROOT = repo_root()
+CORPUS = ROOT / "detections" / "sigma-correlations"
+WINDOWED = ROOT / "services" / "fusion" / "app" / "data" / "windowed_ruleset.json"
+REFUSED_DIR = "_refused"
+
+
+def _documents(path: Path) -> list[dict[str, Any]]:
+ try:
+ docs = list(yaml.safe_load_all(path.read_text(encoding="utf-8")))
+ except yaml.YAMLError:
+ return []
+ return [d for d in docs if isinstance(d, dict)]
+
+
+def compile_corpus() -> tuple[list[dict[str, Any]], list[tuple[str, str]], int]:
+ """Returns (compiled windowed rules, [(file, reason)], documents read).
+
+ The document count is returned so a caller can refuse a corpus that read
+ nothing: a compiler over zero files produces zero refusals and zero
+ rules, which reads exactly like a clean corpus.
+ """
+ compiled: list[dict[str, Any]] = []
+ refused: list[tuple[str, str]] = []
+ seen = 0
+
+ for path in sorted(CORPUS.rglob("*.yml")) + sorted(CORPUS.rglob("*.yaml")):
+ docs = _documents(path)
+ if not docs:
+ continue
+ seen += len(docs)
+ relative = str(path.relative_to(ROOT))
+
+ selectors: dict[str, dict[str, Any]] = {}
+ base_refusals: list[str] = []
+ for doc in docs:
+ if doc.get("correlation"):
+ continue
+ name = str(doc.get("name") or doc.get("id") or "").strip()
+ if not name:
+ base_refusals.append("a base rule has neither a name nor an id")
+ continue
+ try:
+ selectors[name] = compile_rule(doc).match_when
+ except Refusal as exc:
+ base_refusals.append(f"base rule {name!r}: {exc}")
+
+ correlation_ids = {str(d.get("id") or "") for d in docs if d.get("correlation")}
+ for doc in docs:
+ if not doc.get("correlation"):
+ continue
+ try:
+ if base_refusals:
+ raise CorrelationRefusal("a referenced rule did not compile", "; ".join(base_refusals))
+ compiled.append(compile_correlation(doc, selectors=selectors, correlations=frozenset(correlation_ids)))
+ except CorrelationRefusal as exc:
+ refused.append((relative, str(exc)))
+ return compiled, refused, seen
+
+
+def _namespace() -> set[str]:
+ spec = importlib.util.spec_from_file_location("check_detection_fields", ROOT / "scripts" / "check_detection_fields.py")
+ if spec is None or spec.loader is None: # pragma: no cover
+ raise SystemExit("cannot load scripts/check_detection_fields.py")
+ module = importlib.util.module_from_spec(spec)
+ sys.modules["check_detection_fields"] = module
+ spec.loader.exec_module(module)
+ return module._connector_namespace() | module._template_namespace() | module._go_normalizer_namespace()
+
+
+def _selector_fields(rule: dict[str, Any]) -> set[str]:
+ spec = importlib.util.spec_from_file_location("check_detection_fields", ROOT / "scripts" / "check_detection_fields.py")
+ module = importlib.util.module_from_spec(spec) # type: ignore[arg-type]
+ sys.modules["check_detection_fields"] = module
+ spec.loader.exec_module(module) # type: ignore[union-attr]
+ clauses = [rule["match_when"]] if rule.get("match_when") else list(rule.get("sequence") or [])
+ found: set[str] = set()
+ for clause in clauses:
+ found |= module._rule_fields(clause)
+ for key in ("group_by", "distinct_by"):
+ if rule.get(key):
+ found.add(str(rule[key]))
+ return found
+
+
+def check(compiled: list[dict[str, Any]], refused: list[tuple[str, str]], seen: int, shipped: set[str]) -> list[str]:
+ problems: list[str] = []
+ if seen == 0:
+ problems.append(f"no correlation documents read under {CORPUS.relative_to(ROOT)} — refusing to report a corpus of zero as clean")
+ return problems
+
+ namespace = _namespace()
+ compiled_by_id: dict[str, dict[str, Any]] = {}
+ for rule in compiled:
+ rule_id = str(rule.get("id") or "")
+ if rule_id in compiled_by_id:
+ problems.append(f"two correlations share the id {rule_id!r}")
+ compiled_by_id[rule_id] = rule
+
+ unknown = sorted(f for f in _selector_fields(rule) if f not in namespace)
+ if unknown:
+ problems.append(
+ f"{rule_id} names field(s) nothing emits: {', '.join(unknown)}. "
+ "A correlation grouping on a field no source produces counts nothing forever."
+ )
+ if rule_id not in shipped:
+ problems.append(f"{rule_id} compiled but is not in {WINDOWED.name} — re-run scripts/export_windowed_ruleset.py")
+
+ # Everything under `_refused/` must in fact be refused, or the directory
+ # becomes a place to park a rule away from the checks above.
+ parked = {path for path, _ in refused}
+ for path in sorted(CORPUS.rglob("*.yml")):
+ relative = str(path.relative_to(ROOT))
+ in_refused_dir = REFUSED_DIR in path.relative_to(CORPUS).parts
+ if in_refused_dir and relative not in parked:
+ problems.append(f"{relative} sits under {REFUSED_DIR}/ but compiled cleanly — move it out or say why it cannot run")
+ if not in_refused_dir and relative in parked:
+ reason = next(r for p, r in refused if p == relative)
+ problems.append(f"{relative} was refused but is not under {REFUSED_DIR}/: {reason}")
+ return problems
+
+
+def _self_test_injections(
+ compiled: list[dict[str, Any]], refused: list[tuple[str, str]], seen: int, shipped: set[str]
+) -> list[tuple[str, bool]]:
+ results = [("the corpus as committed has no findings to begin with", not check(compiled, refused, seen, shipped))]
+
+ invented = [*compiled, {**compiled[0], "id": "aisoc-corr-injected", "group_by": "a_field_nothing_emits"}]
+ results.append(
+ (
+ "catches a correlation grouping on a field nothing emits",
+ any("nothing emits" in p for p in check(invented, refused, seen, shipped | {"aisoc-corr-injected"})),
+ )
+ )
+
+ results.append(("catches a compiled correlation missing from the windowed ruleset", bool(check(compiled, refused, seen, set()))))
+
+ duplicated = [*compiled, dict(compiled[0])]
+ results.append(("catches two correlations sharing an id", any("share the id" in p for p in check(duplicated, refused, seen, shipped))))
+
+ # Drop the real refusals so the files under `_refused/` look as though
+ # they compiled. Expressed by removing them from the refusal list rather
+ # than by writing a file, because the check reads the directory and a
+ # self-test that mutates the tree is one that can leave it dirty.
+ as_if_compiling = [entry for entry in refused if REFUSED_DIR not in entry[0]]
+ results.append(
+ (
+ "catches a rule parked under _refused/ that in fact compiles",
+ any("compiled cleanly" in p for p in check(compiled, as_if_compiling, seen, shipped)),
+ )
+ )
+
+ results.append(("refuses a corpus it read nothing from", bool(check(compiled, refused, 0, shipped))))
+
+ skeleton_refused, detail = refuses_an_empty_tree(Path(__file__).name, ["--check"], shape=SKELETON)
+ results.append((f"refuses a tree whose directories exist and hold no correlations ({detail.splitlines()[0]})", skeleton_refused))
+ return results
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--check", action="store_true", help="enforce (the default; accepted so CI can be explicit)")
+ parser.add_argument("--report", action="store_true", help="print the compiled and refused taxonomy")
+ parser.add_argument("--self-test", action="store_true", help="prove the gate catches what it claims to")
+ args = parser.parse_args()
+
+ if not CORPUS.is_dir():
+ print(f"{Path(sys.argv[0]).name}: no correlation corpus at {CORPUS} — nothing to check", file=sys.stderr)
+ return 2
+
+ compiled, refused, seen = compile_corpus()
+ shipped: set[str] = set()
+ if WINDOWED.is_file():
+ shipped = {str(r.get("id")) for r in json.loads(WINDOWED.read_text(encoding="utf-8")).get("rules") or []}
+
+ if args.self_test:
+ if not compiled:
+ print("sigma correlations: nothing compiled, so the injections below would prove nothing", file=sys.stderr)
+ return 1
+ return self_test_main(Path(__file__).name, ["--check"], extra=_self_test_injections(compiled, refused, seen, shipped))
+
+ if args.report:
+ for rule in compiled:
+ kind = rule.get("correlation_type")
+ print(f"COMPILED {rule['id']} [{kind}] group_by={rule['group_by']} window={rule['window_seconds']}s")
+ for path, reason in refused:
+ print(f"REFUSED {path}: {reason}")
+ return 0
+
+ problems = check(compiled, refused, seen, shipped)
+ kinds: dict[str, int] = {}
+ for rule in compiled:
+ kind = str(rule.get("correlation_type"))
+ kinds[kind] = kinds.get(kind, 0) + 1
+ print(f"sigma correlations: {seen} documents read, {len(compiled)} compiled, {len(refused)} refused")
+ for kind, count in sorted(kinds.items()):
+ print(f" {count:4d} {kind}")
+ for path, reason in refused:
+ print(f" refused {Path(path).name}: {reason}")
+
+ if problems:
+ print(f"\nERROR: {len(problems)} problem(s):", file=sys.stderr)
+ for problem in problems:
+ print(f" - {problem}", file=sys.stderr)
+ return 1
+ print("\nOK: every correlation compiles into the windowed engine or is refused with a reason")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/detection_truth_table.py b/scripts/detection_truth_table.py
index 2024045a6..54bbdfd87 100644
--- a/scripts/detection_truth_table.py
+++ b/scripts/detection_truth_table.py
@@ -310,6 +310,12 @@ def render_markdown(c: Counts) -> str:
lines.append("that family could not be expressed as a count over a window and are refused")
lines.append("with a reason that travels into each rule's `quarantine_reason`.")
lines.append("")
+ lines.append("Four more are Sigma correlation documents compiled by")
+ lines.append("`scripts/sigma_correlation.py`. They are **first-party content, not imported")
+ lines.append("coverage**: no upstream correlation rule is vendored in this tree, so the")
+ lines.append("import path had no corpus and the rules under `detections/sigma-correlations/`")
+ lines.append("were hand-authored in the upstream format to give it one.")
+ lines.append("")
if c.counted_but_not_loaded:
lines.append(f"Those {c.counted_but_not_loaded} rules are the ones an earlier version of this")
lines.append("table counted as executable. They carry `enabled: true` and a body whose")
diff --git a/scripts/export_windowed_ruleset.py b/scripts/export_windowed_ruleset.py
index a2cf06c0b..4810a653e 100644
--- a/scripts/export_windowed_ruleset.py
+++ b/scripts/export_windowed_ruleset.py
@@ -334,14 +334,19 @@ WINDOWED_RULES: list[dict] = [
def all_rules() -> list[dict]:
- """Rules authored here, then rules translated from the `det-*` corpus.
+ """Three sources, kept in their own blocks so a change to one is legible.
- Order matters only for the diff: keeping the translations in their own
- block means a change to one never renumbers the other.
+ Authored here; translated from the `det-*` corpus by
+ `windowed_translation.py`; compiled from Sigma correlation documents by
+ `sigma_correlation.py`. One artefact rather than three, because the
+ engine has one loader and a second file is a second thing that can go
+ stale unnoticed.
"""
+ from check_sigma_correlations import compile_corpus # noqa: PLC0415
from windowed_translation import translated_rules # noqa: PLC0415
- return [*WINDOWED_RULES, *translated_rules()]
+ correlations, _refused, _seen = compile_corpus()
+ return [*WINDOWED_RULES, *translated_rules(), *correlations]
def build() -> dict:
@@ -383,8 +388,12 @@ def main() -> int:
OUT.parent.mkdir(parents=True, exist_ok=True)
OUT.write_text(rendered, encoding="utf-8")
- translated = len(rules) - len(WINDOWED_RULES)
- print(f"wrote {OUT.relative_to(ROOT)} — {len(rules)} windowed rules ({len(WINDOWED_RULES)} authored, {translated} translated)")
+ translated = sum(1 for rule in rules if rule.get("translated_from"))
+ correlations = sum(1 for rule in rules if rule.get("correlation_type"))
+ print(
+ f"wrote {OUT.relative_to(ROOT)} — {len(rules)} windowed rules "
+ f"({len(WINDOWED_RULES)} authored, {translated} translated, {correlations} Sigma correlations)"
+ )
return 0
diff --git a/scripts/sigma_correlation.py b/scripts/sigma_correlation.py
new file mode 100644
index 000000000..5f7a27c46
--- /dev/null
+++ b/scripts/sigma_correlation.py
@@ -0,0 +1,228 @@
+#!/usr/bin/env python3
+"""Compile Sigma correlation rules into the windowed engine's form.
+
+Specification: https://github.com/SigmaHQ/sigma-specification — "Sigma
+Correlations", the ``correlation:`` document type. Read before writing this;
+every field name and correlation type below is taken from it rather than
+guessed.
+
+What the engine can and cannot express
+--------------------------------------
+The windowed engine counts events or distinct values for one entity over a
+sliding window, and stages ordered or unordered sets of events for one entity
+over a sliding window. That covers four of Sigma's correlation types:
+
+``event_count`` → a counting rule.
+``value_count`` → a counting rule with ``distinct_by``.
+``temporal`` → an unordered staged rule.
+``temporal_ordered`` → an ordered staged rule.
+
+It does not cover the rest, and the rule here is **refuse rather than
+approximate**. A correlation that is translated with one of its constraints
+dropped is a different detection shipping under the original's name and
+severity, which is the specific failure the windowed translation work in
+depth plan 3.1 existed to stop. Every refusal carries a reason, and the
+refusals are counted and published the same way the Sigma compiler's are.
+
+The two refusals worth knowing about in advance, because they look like
+oversights:
+
+* **A ``lt`` / ``lte`` condition is a rarity signal.** "Fewer than five in an
+ hour" fires when nothing much happened, so it needs a store of what is
+ normal rather than a counter. The engine fires when a count is high.
+* **A multi-field ``group-by`` has no single entity.** The engine accumulates
+ against one field. Picking the first would group by something the rule
+ author did not ask for and silently widen or narrow every count.
+"""
+
+from __future__ import annotations
+
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+__all__ = [
+ "CORRELATION_TYPES",
+ "CorrelationRefusal",
+ "compile_correlation",
+ "parse_timespan",
+]
+
+#: Correlation types this compiler accepts, mapped to the shape they become.
+CORRELATION_TYPES: dict[str, str] = {
+ "event_count": "counting",
+ "value_count": "counting",
+ "temporal": "staged",
+ "temporal_ordered": "staged",
+}
+
+R_UNKNOWN_TYPE = "correlation type is not one this engine can express"
+R_CONDITION_SHAPE = "the condition is not a single comparison the engine can threshold"
+R_RARITY = "a `lt`/`lte` condition fires when a count is low, which needs a baseline rather than a counter"
+R_EQ = "an `eq` condition fires on an exact count, and a sliding window cannot hold a count still"
+R_MULTI_GROUP = "`group-by` names more than one field and the engine accumulates against one entity"
+R_NO_GROUP = "`group-by` is absent, so the count has no entity and would be global"
+R_TIMESPAN = "`timespan` is missing or not a Sigma duration"
+R_NO_RULES = "`rules` names no referenced rule"
+R_UNRESOLVED = "a referenced rule is not in the corpus or did not compile"
+R_ONE_STAGE = "a temporal correlation over a single rule is a plain count, not a sequence"
+R_VALUE_FIELD = "`value_count` needs a `field` naming what to count distinctly"
+R_ALIASES = "`aliases` map different field names across referenced rules and the engine has no aliasing"
+R_GENERATE = "`generate: true` also emits the referenced rules, which this path does not model"
+R_CHAINED = "the correlation references another correlation, which the engine cannot stage"
+
+#: Sigma duration suffixes, from the specification.
+_UNITS: dict[str, int] = {"s": 1, "m": 60, "h": 3600, "d": 86400, "w": 604800}
+
+
+class CorrelationRefusal(Exception):
+ """This correlation cannot be expressed without changing what it detects."""
+
+ def __init__(self, reason: str, detail: str = "") -> None:
+ super().__init__(f"{reason}{f' ({detail})' if detail else ''}")
+ self.reason = reason
+ self.detail = detail
+
+
+def parse_timespan(raw: Any) -> int:
+ """Sigma duration (``5m``, ``1h``, ``30s``) to seconds."""
+ text = str(raw or "").strip()
+ if len(text) < 2 or text[-1] not in _UNITS or not text[:-1].isdigit():
+ raise CorrelationRefusal(R_TIMESPAN, text or "absent")
+ seconds = int(text[:-1]) * _UNITS[text[-1]]
+ if seconds < 1:
+ raise CorrelationRefusal(R_TIMESPAN, text)
+ return seconds
+
+
+@dataclass(frozen=True)
+class _Condition:
+ operator: str
+ value: int
+
+
+def _parse_condition(block: Any, correlation_type: str) -> _Condition:
+ """The ``condition:`` mapping, which Sigma writes as one comparison."""
+ if not isinstance(block, dict) or len(block) != 1:
+ raise CorrelationRefusal(R_CONDITION_SHAPE, repr(block))
+ ((operator, value),) = block.items()
+ operator = str(operator).lower()
+ if operator in {"lt", "lte"}:
+ raise CorrelationRefusal(R_RARITY, f"{operator}: {value}")
+ if operator == "eq":
+ raise CorrelationRefusal(R_EQ, f"eq: {value}")
+ if operator not in {"gt", "gte"}:
+ raise CorrelationRefusal(R_CONDITION_SHAPE, operator)
+ try:
+ bound = int(value)
+ except (TypeError, ValueError) as exc:
+ raise CorrelationRefusal(R_CONDITION_SHAPE, f"{operator}: {value!r}") from exc
+ if bound < 0:
+ raise CorrelationRefusal(R_CONDITION_SHAPE, f"{operator}: {bound}")
+ # `correlation_type` is carried so a later type with different threshold
+ # semantics cannot silently reuse this.
+ del correlation_type
+ return _Condition(operator=operator, value=bound)
+
+
+def _group_by(block: Any) -> str:
+ fields = block if isinstance(block, list) else [block] if block else []
+ names = [str(f) for f in fields if str(f or "").strip()]
+ if not names:
+ raise CorrelationRefusal(R_NO_GROUP)
+ if len(names) > 1:
+ raise CorrelationRefusal(R_MULTI_GROUP, ", ".join(names))
+ return names[0]
+
+
+def _referenced(block: dict[str, Any]) -> list[str]:
+ """`rules:` lives inside the correlation block, per the specification."""
+ raw = block.get("rules")
+ names = [str(r) for r in (raw if isinstance(raw, list) else [raw] if raw else []) if str(r or "").strip()]
+ if not names:
+ raise CorrelationRefusal(R_NO_RULES)
+ return names
+
+
+def compile_correlation(
+ doc: dict[str, Any],
+ *,
+ selectors: dict[str, dict[str, Any]],
+ correlations: frozenset[str] = frozenset(),
+) -> dict[str, Any]:
+ """One Sigma correlation document to one windowed rule.
+
+ ``selectors`` maps a referenced rule's name (or id) to the ``match_when``
+ it compiled to. A correlation whose referenced rule is missing is refused
+ rather than compiled against an empty selector — an empty selector
+ matches every event, so that refusal is the difference between a rule and
+ an outage.
+ """
+ block = doc.get("correlation")
+ if not isinstance(block, dict):
+ raise CorrelationRefusal(R_UNKNOWN_TYPE, "no correlation block")
+
+ correlation_type = str(block.get("type") or "").strip().lower()
+ if correlation_type not in CORRELATION_TYPES:
+ raise CorrelationRefusal(R_UNKNOWN_TYPE, correlation_type or "absent")
+ if block.get("aliases"):
+ raise CorrelationRefusal(R_ALIASES)
+ if block.get("generate"):
+ raise CorrelationRefusal(R_GENERATE)
+
+ window_seconds = parse_timespan(block.get("timespan"))
+ group_by = _group_by(block.get("group-by"))
+ names = _referenced(block)
+ if any(name in correlations for name in names):
+ raise CorrelationRefusal(R_CHAINED, ", ".join(n for n in names if n in correlations))
+ missing = [name for name in names if name not in selectors]
+ if missing:
+ raise CorrelationRefusal(R_UNRESOLVED, ", ".join(missing))
+
+ common = {
+ "id": str(doc.get("id") or "").strip(),
+ "name": str(doc.get("title") or doc.get("name") or "").strip(),
+ "severity": str(doc.get("level") or doc.get("severity") or "medium").strip().lower(),
+ "category": str(doc.get("category") or "identity").strip().lower(),
+ "mitre": sorted({str(t).upper().removeprefix("ATTACK.") for t in doc.get("tags") or [] if str(t).lower().startswith("attack.t")}),
+ "group_by": group_by,
+ "window_seconds": window_seconds,
+ "correlation_type": correlation_type,
+ "correlates": names,
+ }
+ if not common["id"] or not common["name"]:
+ raise CorrelationRefusal(R_CONDITION_SHAPE, "the document has no id or title")
+
+ if CORRELATION_TYPES[correlation_type] == "counting":
+ if len(names) != 1:
+ # Sigma allows several, meaning "events matching any of them".
+ # The engine's `match_when` is one clause set, and an `any_of`
+ # over compiled selectors is a faithful translation only when the
+ # selectors are disjoint, which nothing here can establish.
+ raise CorrelationRefusal(R_CONDITION_SHAPE, f"{correlation_type} over {len(names)} rules")
+ condition = _parse_condition(block.get("condition"), correlation_type)
+ rule: dict[str, Any] = {
+ **common,
+ "match_when": dict(selectors[names[0]]),
+ # `gt: N` crosses at N+1; `gte: N` crosses at N.
+ "threshold": condition.value + 1 if condition.operator == "gt" else condition.value,
+ }
+ if correlation_type == "value_count":
+ field = str(block.get("field") or "").strip()
+ if not field:
+ raise CorrelationRefusal(R_VALUE_FIELD)
+ rule["distinct_by"] = field
+ if rule["threshold"] < 1:
+ raise CorrelationRefusal(R_CONDITION_SHAPE, "threshold below one fires on the first event")
+ return rule
+
+ if len(names) < 2:
+ raise CorrelationRefusal(R_ONE_STAGE, names[0])
+ return {
+ **common,
+ "sequence": [dict(selectors[name]) for name in names],
+ "ordered": correlation_type == "temporal_ordered",
+ }
diff --git a/services/fusion/app/data/windowed_ruleset.json b/services/fusion/app/data/windowed_ruleset.json
index 38bc9bff4..98c4c49d9 100644
--- a/services/fusion/app/data/windowed_ruleset.json
+++ b/services/fusion/app/data/windowed_ruleset.json
@@ -1,5 +1,5 @@
{
- "count": 68,
+ "count": 72,
"rules": [
{
"category": "application",
@@ -1136,6 +1136,110 @@
"threshold": 201,
"translated_from": "net-dhcp-starvation",
"window_seconds": 60
+ },
+ {
+ "category": "cloud",
+ "correlates": [
+ "aws-iam-create-access-key"
+ ],
+ "correlation_type": "event_count",
+ "group_by": "user_arn",
+ "id": "aisoc-corr-aws-access-key-minting-burst",
+ "match_when": {
+ "eventName_pattern_match_any": [
+ "^CreateAccessKey$"
+ ],
+ "eventSource_pattern_match_any": [
+ "^iam\\.amazonaws\\.com$"
+ ]
+ },
+ "mitre": [
+ "T1098.001"
+ ],
+ "name": "One principal minted several AWS access keys in a burst",
+ "severity": "high",
+ "threshold": 6,
+ "window_seconds": 600
+ },
+ {
+ "category": "cloud",
+ "correlates": [
+ "aws-api-call"
+ ],
+ "correlation_type": "value_count",
+ "distinct_by": "awsRegion",
+ "group_by": "user_arn",
+ "id": "aisoc-corr-aws-principal-region-fanout",
+ "match_when": {
+ "cloud_platform_pattern_match_any": [
+ "^aws$"
+ ]
+ },
+ "mitre": [
+ "T1580"
+ ],
+ "name": "One principal acting across an unusual number of AWS regions",
+ "severity": "medium",
+ "threshold": 6,
+ "window_seconds": 900
+ },
+ {
+ "category": "cloud",
+ "correlates": [
+ "azure-directory-role-assigned",
+ "azure-resource-lock-deleted"
+ ],
+ "correlation_type": "temporal",
+ "group_by": "actor",
+ "id": "aisoc-corr-azure-role-grant-and-lock-removal",
+ "mitre": [
+ "T1098"
+ ],
+ "name": "One actor both assigned a directory role and removed a resource lock",
+ "ordered": false,
+ "sequence": [
+ {
+ "operationName_pattern_match_any": [
+ "^Add\\ member\\ to\\ role$"
+ ]
+ },
+ {
+ "operationName_pattern_match_any": [
+ "^Microsoft\\.Authorization/locks/delete$"
+ ]
+ }
+ ],
+ "severity": "medium",
+ "window_seconds": 1800
+ },
+ {
+ "category": "identity",
+ "correlates": [
+ "okta-mfa-push-sent",
+ "okta-session-start"
+ ],
+ "correlation_type": "temporal_ordered",
+ "group_by": "actor_email",
+ "id": "aisoc-corr-okta-mfa-push-then-session",
+ "mitre": [
+ "T1621"
+ ],
+ "name": "Okta MFA push followed by a session for the same account",
+ "ordered": true,
+ "sequence": [
+ {
+ "eventType_pattern_match_any": [
+ "^system\\.push\\.send_factor_verify_push$"
+ ]
+ },
+ {
+ "eventType_pattern_match_any": [
+ "^user\\.session\\.start$"
+ ]
+ }
+ ],
+ "severity": "medium",
+ "window_seconds": 900
}
]
}
diff --git a/services/fusion/app/services/derived_fields.py b/services/fusion/app/services/derived_fields.py
index 25415f7b4..f742f4ddb 100644
--- a/services/fusion/app/services/derived_fields.py
+++ b/services/fusion/app/services/derived_fields.py
@@ -50,7 +50,14 @@ DEFAULT_BUSINESS_END_HOUR = 18
_TIME_FIELDS = ("event_time", "timestamp", "time", "@timestamp", "ingest_time")
-def _parse_time(event: dict[str, Any]) -> datetime | None:
+def parse_event_time(event: dict[str, Any]) -> datetime | None:
+ """When the event happened, from the first timestamp field it carries.
+
+ Public because the windowed engine orders sequences by event time and
+ must read it the same way this module does. Two independent readers of
+ "when did this happen" is how a rule comes to mean one thing to the
+ stateless engine and another to the windowed one.
+ """
for field in _TIME_FIELDS:
raw = event.get(field)
if raw is None:
@@ -119,7 +126,7 @@ def time_of_day_fields(
Returns nothing when the event carries no parseable timestamp, for the
same reason as above: an unknown time is not "outside business hours".
"""
- when = _parse_time(event)
+ when = parse_event_time(event)
if when is None:
return {}
diff --git a/services/fusion/app/services/windowed_detection.py b/services/fusion/app/services/windowed_detection.py
index e374ac9bd..b5dd8e19f 100644
--- a/services/fusion/app/services/windowed_detection.py
+++ b/services/fusion/app/services/windowed_detection.py
@@ -29,7 +29,7 @@ from typing import Any
import structlog
from app.models.alert import AlertSeverity, RawAlert
-from app.services.derived_fields import enrich, requested_derived_fields
+from app.services.derived_fields import enrich, parse_event_time, requested_derived_fields
from app.services.detection_engine import DetectionHit
from app.services.detection_matcher import matches
from app.services.provenance import extract_provenance
@@ -70,6 +70,59 @@ class WindowRule:
distinct_by: str = ""
+@dataclass(frozen=True)
+class SequenceRule:
+ """A then B (then C) by the same entity, ordered by event time.
+
+ A threshold cannot express an ordering, and the ordering is usually the
+ detection: "fifty failed logons" is an attempt, "failed logons then a
+ success for the same account" is a compromise. Every sequence detection in
+ the corpus had nowhere to go, because the counting engine sees each event
+ alone and remembers only how many there were.
+
+ The stages are ordered and disjoint in time: a stage advances only from an
+ earlier event *by event time*, and one event advances at most one stage,
+ so a rule whose stages overlap still needs as many events as it has
+ stages.
+ """
+
+ id: str
+ name: str
+ severity: str
+ category: str
+ mitre: list[str]
+ #: Ordered selectors, one per stage. Two or more.
+ sequence: tuple[dict[str, Any], ...]
+ #: Flat field naming the entity every stage must share.
+ group_by: str
+ #: The whole sequence must fit in this, measured from the first stage.
+ #: Measured from the first and not from the previous stage, so a slow
+ #: drip cannot walk a sequence forward indefinitely.
+ window_seconds: int
+ #: When False the stages may occur in any order inside the window, which
+ #: is what Sigma's `temporal` correlation means as against
+ #: `temporal_ordered`. Kept as a flag on one rule type rather than a
+ #: second type, because every other property — entity, window, staging,
+ #: one-event-one-stage — is identical and duplicating them is how the two
+ #: drift apart.
+ ordered: bool = True
+
+
+#: How far out of order an event may arrive and still be stitched into a
+#: sequence. A connector polling a vendor delivers the vendor's order, not the
+#: clock's, so arrival order is not evidence of event order — but holding
+#: state forever to accommodate that would be a memory leak with a detection
+#: attached. An event older than this relative to what the entity has already
+#: shown is counted for its own stage and does not re-open an earlier one.
+#:
+#: The plan this implements says "with the existing watermark". There is no
+#: watermark in this pipeline — the only ones in the tree belong to the
+#: shadow-reconcile router and the dead-letter replay, which are unrelated
+#: subsystems — so the bound is stated here rather than inherited from
+#: something that does not exist.
+SEQUENCE_REORDER_TOLERANCE_SECONDS = 300
+
+
# Built-in windowed rules. Intentionally small + high-signal; the corpus can grow
# via the same JSON export path as the stateless engine later.
_BUILTIN_RULES: tuple[WindowRule, ...] = (
@@ -148,6 +201,12 @@ def load_window_rules(path: Path | None = None) -> tuple[WindowRule, ...]:
rule_id = str(entry.get("id") or "")
if not rule_id or rule_id in seen:
continue
+ if entry.get("sequence"):
+ # A sequence rule lives in the same artefact and is loaded by
+ # `load_sequence_rules`. Skipped silently rather than warned
+ # about: a warning per sequence rule on every boot would train
+ # operators to ignore the log line that means something.
+ continue
try:
rule = WindowRule(
id=rule_id,
@@ -177,25 +236,161 @@ def load_window_rules(path: Path | None = None) -> tuple[WindowRule, ...]:
return tuple(loaded)
+def load_sequence_rules(path: Path | None = None) -> tuple[SequenceRule, ...]:
+ """Ordered-sequence rules from the same artefact.
+
+ Fail-soft the same way the counting loader is, and for the same reason: a
+ malformed entry is skipped individually so one authoring mistake cannot
+ empty the corpus. There are no built-in sequences, so a missing file
+ yields none — which is the honest answer rather than a fallback.
+ """
+ target = path or _WINDOWED_RULESET_PATH
+ if not target.exists():
+ return ()
+ try:
+ payload = json.loads(target.read_text(encoding="utf-8"))
+ except (ValueError, OSError) as exc:
+ logger.warning("windowed_detection.sequence_ruleset_load_failed", path=str(target), error=str(exc))
+ return ()
+
+ loaded: list[SequenceRule] = []
+ seen: set[str] = set()
+ for entry in payload.get("rules") or []:
+ if not isinstance(entry, dict) or not entry.get("sequence"):
+ continue
+ rule_id = str(entry.get("id") or "")
+ if not rule_id or rule_id in seen:
+ continue
+ try:
+ stages = tuple(dict(stage) for stage in entry["sequence"])
+ rule = SequenceRule(
+ id=rule_id,
+ name=str(entry["name"]),
+ severity=str(entry["severity"]),
+ category=str(entry["category"]),
+ mitre=[str(m).upper() for m in entry.get("mitre") or []],
+ sequence=stages,
+ group_by=str(entry["group_by"]),
+ window_seconds=int(entry["window_seconds"]),
+ ordered=bool(entry.get("ordered", True)),
+ )
+ except (KeyError, TypeError, ValueError) as exc:
+ logger.warning("windowed_detection.sequence_skipped", rule_id=rule_id, error=str(exc))
+ continue
+ if len(rule.sequence) < 2:
+ # One stage is a stateless rule paying for Redis state.
+ logger.warning("windowed_detection.sequence_too_short", rule_id=rule_id, stages=len(rule.sequence))
+ continue
+ if any(not stage for stage in rule.sequence):
+ # An empty stage matches every event, so the sequence collapses
+ # to whichever stages remain — a different detection under this
+ # one's name.
+ logger.warning("windowed_detection.sequence_stage_empty", rule_id=rule_id)
+ continue
+ if rule.window_seconds < 1:
+ logger.warning("windowed_detection.sequence_bounds_invalid", rule_id=rule_id)
+ continue
+ loaded.append(rule)
+ seen.add(rule_id)
+
+ if loaded:
+ logger.info("windowed_detection.sequence_ruleset_loaded", count=len(loaded))
+ return tuple(loaded)
+
+
+def _unordered_set_exists(observations: list[list[tuple[float, str]]], window_seconds: int) -> bool:
+ """Is there one *distinct* event per stage inside one window, any order?
+
+ The distinctness is the subtle half. Without it a single event matching
+ every stage satisfies the rule on its own, which for an unordered
+ correlation is not a near-miss but the common case: unordered stages are
+ usually written as variations on one activity.
+ """
+ if not observations or any(not stage for stage in observations):
+ return False
+ merged = sorted({entry for stage in observations for entry in stage})
+ for start in merged:
+ window = [entry for entry in merged if start[0] <= entry[0] <= start[0] + window_seconds]
+ # One event per stage, no event used twice: a bipartite matching,
+ # small enough (stages are two or three) to settle by trying each
+ # stage's candidates in turn.
+ if _matching_exists([[e for e in stage if e in set(window)] for stage in observations], set()):
+ return True
+ return False
+
+
+def _matching_exists(candidates: list[list[tuple[float, str]]], used: set[tuple[float, str]]) -> bool:
+ if not candidates:
+ return True
+ for entry in candidates[0]:
+ if entry in used:
+ continue
+ if _matching_exists(candidates[1:], used | {entry}):
+ return True
+ return False
+
+
+def _chain_exists(observations: list[list[tuple[float, str]]], window_seconds: int) -> bool:
+ """Is there one event per stage, strictly increasing, inside the window?
+
+ Ordered by ``(event time, event token)``. The token is the tie-breaker and
+ it is load-bearing rather than tidy: two distinct events routinely share a
+ second, and one event that satisfies two stages must never complete a
+ sequence by itself. Comparing times alone would allow both mistakes in
+ opposite directions.
+
+ Candidate starts are tried newest-first because the latest feasible start
+ gives the tightest span, so the first chain found is the one most likely
+ to fit the window.
+ """
+ if not observations or any(not stage for stage in observations):
+ return False
+ for start in reversed(observations[0]):
+ cursor = start
+ for stage in observations[1:]:
+ nxt = next((entry for entry in stage if entry > cursor), None)
+ if nxt is None:
+ cursor = None # type: ignore[assignment]
+ break
+ cursor = nxt
+ if cursor is not None and cursor[0] - start[0] <= window_seconds:
+ return True
+ return False
+
+
class WindowedDetectionEngine:
"""Redis-backed sliding-window threshold detections."""
- def __init__(self, redis: Any, rules: tuple[WindowRule, ...] | None = None, *, key_prefix: str = "aisoc:wd") -> None:
+ def __init__(
+ self,
+ redis: Any,
+ rules: tuple[WindowRule, ...] | None = None,
+ *,
+ sequences: tuple[SequenceRule, ...] | None = None,
+ key_prefix: str = "aisoc:wd",
+ ) -> None:
self._redis = redis
# None means "whatever is declared", so a deployment picks up exported
# rules without a code change. An explicit tuple still wins, which is
# what the tests rely on.
self._rules = rules if rules is not None else load_window_rules()
+ self._sequences = sequences if sequences is not None else load_sequence_rules()
self._prefix = key_prefix
# Same contract as the stateless engine: computed once at load,
# because the set changes when rules change and not when traffic
# arrives.
- self._derived_wanted: set[str] = requested_derived_fields([{"match_when": rule.match_when} for rule in self._rules])
+ clauses: list[dict[str, Any]] = [{"match_when": rule.match_when} for rule in self._rules]
+ clauses += [{"match_when": stage} for rule in self._sequences for stage in rule.sequence]
+ self._derived_wanted: set[str] = requested_derived_fields(clauses)
@property
def rule_count(self) -> int:
return len(self._rules)
+ @property
+ def sequence_count(self) -> int:
+ return len(self._sequences)
+
@staticmethod
def _fields(message: dict[str, Any]) -> dict[str, Any]:
"""Flat field namespace, matching the stateless engine exactly.
@@ -288,8 +483,93 @@ class WindowedDetectionEngine:
)
except Exception as exc: # noqa: BLE001 — one rule/Redis error must not wedge detection
logger.debug("windowed_detection.rule_error", rule=rule.id, error=str(exc))
+
+ if self._sequences:
+ # One token per event, shared by every stage it matches, so two
+ # stages satisfied by the *same* event can never be mistaken for
+ # two events. Timestamps alone cannot carry that: two genuinely
+ # distinct events routinely share a second.
+ token = uuid.uuid4().hex
+ for sequence in self._sequences:
+ try:
+ hit = await self._advance_sequence(sequence, tenant, fields, now, token)
+ if hit is not None:
+ hits.append(hit)
+ except Exception as exc: # noqa: BLE001 — same fail-soft contract
+ logger.debug("windowed_detection.sequence_error", rule=sequence.id, error=str(exc))
return hits
+ @staticmethod
+ def _event_seconds(fields: dict[str, Any], now: float) -> float:
+ """When the event happened, falling back to when we heard about it.
+
+ Ordering a sequence by arrival would make it a different detection on
+ every poll cadence, so event time wins wherever a source provides
+ one. A source that stamps nothing must still be able to fire a
+ sequence, which is what the fallback is for.
+ """
+ when = parse_event_time(fields)
+ return when.timestamp() if when is not None else now
+
+ async def _advance_sequence(
+ self, rule: SequenceRule, tenant: str, fields: dict[str, Any], now: float, token: str
+ ) -> DetectionHit | None:
+ """Record what this event satisfies, then look for a complete chain.
+
+ Recording first and searching second is what makes the ordering a
+ property of **event time** rather than of arrival. The obvious
+ implementation — a cursor that only moves forward as events arrive —
+ is wrong for this platform: almost every connector here polls, so a
+ batch arrives in the vendor's order and the event that starts a
+ sequence routinely lands after the one that finishes it. That
+ implementation would silently detect nothing on exactly the sources
+ the corpus is thinnest on.
+ """
+ entity = fields.get(rule.group_by)
+ if not entity:
+ return None
+ matched = [index for index, stage in enumerate(rule.sequence) if matches(stage, fields)]
+ if not matched:
+ return None
+
+ when = self._event_seconds(fields, now)
+ retain = rule.window_seconds + SEQUENCE_REORDER_TOLERANCE_SECONDS
+ base = f"{self._prefix}:seq:{tenant}:{rule.id}:{entity}"
+
+ for index in matched:
+ key = f"{base}:{index}"
+ await self._redis.zadd(key, {token: when})
+ # Trimmed against this event's own time, not the wall clock: an
+ # event that arrives late is old by construction, and trimming it
+ # against now would discard the observation it just made.
+ await self._redis.zremrangebyscore(key, 0, when - retain)
+ await self._redis.expire(key, retain)
+
+ observations: list[list[tuple[float, str]]] = []
+ for index in range(len(rule.sequence)):
+ entries = await self._redis.zrangebyscore(f"{base}:{index}", when - retain, when + retain, withscores=True)
+ stage: list[tuple[float, str]] = []
+ for member, score in entries or []:
+ name = member.decode() if isinstance(member, bytes | bytearray) else str(member)
+ stage.append((float(score), name))
+ stage.sort()
+ observations.append(stage)
+
+ complete = (
+ _chain_exists(observations, rule.window_seconds) if rule.ordered else _unordered_set_exists(observations, rule.window_seconds)
+ )
+ if not complete:
+ return None
+ if not await self._redis.set(f"{base}:fired", "1", nx=True, ex=rule.window_seconds):
+ return None
+ return DetectionHit(
+ rule_id=rule.id,
+ name=rule.name,
+ severity=rule.severity,
+ category=rule.category,
+ mitre=list(rule.mitre),
+ )
+
async def _observe_and_check(
self,
rule: WindowRule,
diff --git a/services/fusion/tests/test_sigma_correlation_replay.py b/services/fusion/tests/test_sigma_correlation_replay.py
new file mode 100644
index 000000000..25c5f7bcb
--- /dev/null
+++ b/services/fusion/tests/test_sigma_correlation_replay.py
@@ -0,0 +1,183 @@
+"""Every compiled Sigma correlation, replayed through the real engine.
+
+`scripts/check_sigma_correlations.py` proves a correlation document compiles
+and that the fields it names are ones something emits. Both are statements
+about text. This drives the committed artefact through the real
+`WindowedDetectionEngine` and asserts the behaviour each correlation type is
+supposed to have:
+
+* ``event_count`` fires at its threshold and not one short;
+* ``value_count`` needs distinct values and is not satisfied by one repeated;
+* ``temporal_ordered`` needs its stages in event-time order;
+* ``temporal`` needs all its stages but accepts either order.
+
+The fourth is the one worth having a test for: unordered and ordered differ
+by a single flag, and a flag that is read nowhere looks identical to a flag
+that works.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from pathlib import Path
+from typing import Any
+from uuid import uuid4
+
+import pytest
+from app.services.windowed_detection import WindowedDetectionEngine, load_sequence_rules, load_window_rules
+
+RULESET = Path(__file__).resolve().parents[1] / "app" / "data" / "windowed_ruleset.json"
+
+
+class _FakeRedis:
+ def __init__(self) -> None:
+ self.zsets: dict[str, dict[str, float]] = {}
+ self.strings: dict[str, str] = {}
+
+ async def zadd(self, key: str, mapping: dict[str, float]) -> None:
+ self.zsets.setdefault(key, {}).update(mapping)
+
+ async def zremrangebyscore(self, key: str, min_s: float, max_s: float) -> None:
+ members = self.zsets.get(key, {})
+ for member in [m for m, score in list(members.items()) if min_s <= score <= max_s]:
+ del members[member]
+
+ async def zcard(self, key: str) -> int:
+ return len(self.zsets.get(key, {}))
+
+ async def zrangebyscore(self, key: str, min_s: float, max_s: float, withscores: bool = False) -> list[Any]:
+ items = sorted(((m, s) for m, s in self.zsets.get(key, {}).items() if min_s <= s <= max_s), key=lambda kv: (kv[1], kv[0]))
+ return items if withscores else [m for m, _ in items]
+
+ async def expire(self, key: str, ttl: int) -> None:
+ return None
+
+ async def set(self, key: str, value: str, nx: bool = False, ex: int | None = None) -> Any:
+ if nx and key in self.strings:
+ return None
+ self.strings[key] = value
+ return True
+
+
+def _correlations() -> list[dict[str, Any]]:
+ rules = [r for r in json.loads(RULESET.read_text(encoding="utf-8"))["rules"] if r.get("correlation_type")]
+ assert rules, "no compiled Sigma correlations in the committed ruleset — the replay corpus is empty"
+ return rules
+
+
+CORRELATIONS = _correlations()
+BY_TYPE = {r["correlation_type"]: r for r in CORRELATIONS}
+
+
+def _satisfying(key: str, value: Any) -> tuple[str, Any]:
+ """A field and a value that satisfies one compiled clause.
+
+ The Sigma compiler emits anchored patterns rather than equalities, even
+ for a plain string, so a literal recovered from `^iam\\.amazonaws\\.com$`
+ is what an event has to carry. Recovering it here rather than hand-writing
+ the value keeps the replay honest: the event is built from the clause the
+ engine will evaluate, not from the YAML a human wrote.
+ """
+ if key.endswith("_pattern_match_any"):
+ pattern = value[0] if isinstance(value, list) else value
+ return key[: -len("_pattern_match_any")], re.sub(r"\\(.)", r"\1", str(pattern).strip("^$"))
+ if key.endswith("_in"):
+ return key[: -len("_in")], value[0] if isinstance(value, list) and value else value
+ return key, value
+
+
+def _event(rule: dict[str, Any], tenant: str, clause: dict[str, Any], entity: str, when: float, distinct: str = "") -> dict[str, Any]:
+ raw = dict(_satisfying(key, value) for key, value in clause.items())
+ raw[rule["group_by"]] = entity
+ raw["event_time"] = when
+ if rule.get("distinct_by"):
+ raw[rule["distinct_by"]] = distinct
+ return {"tenant_id": tenant, "ocsf_event": {"tenant_uid": tenant, "raw_data": json.dumps(raw)}}
+
+
+def _engine_for(rule: dict[str, Any]) -> WindowedDetectionEngine:
+ counting = tuple(r for r in load_window_rules() if r.id == rule["id"])
+ staged = tuple(r for r in load_sequence_rules() if r.id == rule["id"])
+ assert counting or staged, f"{rule['id']} is in the committed JSON but neither loader accepted it"
+ return WindowedDetectionEngine(_FakeRedis(), rules=counting, sequences=staged)
+
+
+def test_all_four_correlation_types_are_covered() -> None:
+ """A replay suite that silently covers three of four proves three of four."""
+ assert set(BY_TYPE) == {"event_count", "value_count", "temporal", "temporal_ordered"}, f"covered: {sorted(BY_TYPE)}"
+
+
+async def test_event_count_fires_at_its_threshold_and_not_one_short() -> None:
+ rule = BY_TYPE["event_count"]
+ engine = _engine_for(rule)
+ tenant = str(uuid4())
+ for index in range(rule["threshold"] - 1):
+ assert not await engine.evaluate(_event(rule, tenant, rule["match_when"], "arn:aws:iam::1:user/a", 1000.0 + index))
+ hits = await engine.evaluate(_event(rule, tenant, rule["match_when"], "arn:aws:iam::1:user/a", 2000.0))
+ assert [h.rule_id for h in hits] == [rule["id"]]
+
+
+async def test_value_count_needs_distinct_values() -> None:
+ rule = BY_TYPE["value_count"]
+ tenant = str(uuid4())
+
+ repeated = _engine_for(rule)
+ for index in range(rule["threshold"] * 2):
+ assert not await repeated.evaluate(
+ _event(rule, tenant, rule["match_when"], "arn:aws:iam::1:user/a", 1000.0 + index, distinct="one-value")
+ )
+
+ distinct = _engine_for(rule)
+ hits: list[Any] = []
+ for index in range(rule["threshold"]):
+ hits = await distinct.evaluate(
+ _event(rule, tenant, rule["match_when"], "arn:aws:iam::1:user/a", 1000.0 + index, distinct=f"value-{index}")
+ )
+ assert [h.rule_id for h in hits] == [rule["id"]]
+
+
+async def test_temporal_ordered_requires_the_order() -> None:
+ rule = BY_TYPE["temporal_ordered"]
+ first, second = rule["sequence"][0], rule["sequence"][1]
+ tenant = str(uuid4())
+
+ forwards = _engine_for(rule)
+ assert not await forwards.evaluate(_event(rule, tenant, first, "[email protected]", 1000.0))
+ assert [h.rule_id for h in await forwards.evaluate(_event(rule, tenant, second, "[email protected]", 1100.0))] == [rule["id"]]
+
+ backwards = _engine_for(rule)
+ assert not await backwards.evaluate(_event(rule, tenant, second, "[email protected]", 1000.0))
+ assert not await backwards.evaluate(_event(rule, tenant, first, "[email protected]", 1100.0))
+
+
+async def test_temporal_accepts_either_order_but_needs_both_stages() -> None:
+ rule = BY_TYPE["temporal"]
+ first, second = rule["sequence"][0], rule["sequence"][1]
+ tenant = str(uuid4())
+
+ backwards = _engine_for(rule)
+ assert not await backwards.evaluate(_event(rule, tenant, second, "[email protected]", 1000.0))
+ assert [h.rule_id for h in await backwards.evaluate(_event(rule, tenant, first, "[email protected]", 1100.0))] == [rule["id"]]
+
+ one_stage_only = _engine_for(rule)
+ for index in range(6):
+ assert not await one_stage_only.evaluate(_event(rule, tenant, first, "[email protected]", 1000.0 + index))
+
+
+async def test_a_correlation_does_not_cross_entities() -> None:
+ for rule in CORRELATIONS:
+ engine = _engine_for(rule)
+ tenant = str(uuid4())
+ clauses = list(rule.get("sequence") or [rule["match_when"]])
+ needed = rule.get("threshold", len(clauses))
+ for index in range(needed * 2):
+ clause = clauses[index % len(clauses)]
+ entity = f"entity-{index % 7}"
+ hits = await engine.evaluate(_event(rule, tenant, clause, entity, 1000.0 + index, distinct=f"value-{index}"))
+ assert not hits, f"{rule['id']} fired across {7} different entities"
diff --git a/services/fusion/tests/test_windowed_sequences.py b/services/fusion/tests/test_windowed_sequences.py
new file mode 100644
index 000000000..c3f7247b5
--- /dev/null
+++ b/services/fusion/tests/test_windowed_sequences.py
@@ -0,0 +1,231 @@
+"""Ordered sequences: A then B, by the same entity, inside a window.
+
+A threshold rule cannot express an ordering. "Fifty failed logons" and "a
+failed logon followed by a successful one from a new address" are different
+detections, and only the second says *compromise* rather than *attempt*. The
+counting engine sees each event in isolation and remembers only how many
+there were, so every sequence detection in the corpus had nowhere to go.
+
+What ordering means here, precisely, because the loose version is useless:
+
+* stages advance **one event at a time**, so a single event matching two
+ stages cannot complete a sequence by itself;
+* a stage advances only from an earlier event **by event time**, not by
+ arrival order, because a connector polling a vendor delivers a batch whose
+ internal order is the vendor's and not the clock's;
+* the whole sequence must fit inside one window, measured from the first
+ stage, not from the previous one, so a slow drip cannot walk a sequence
+ forward indefinitely;
+* an out-of-order arrival is tolerated up to a stated bound and is not
+ silently treated as in-order.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+from uuid import uuid4
+
+import pytest
+from app.services.windowed_detection import SequenceRule, WindowedDetectionEngine
+
+
+class _FakeRedis:
+ """The sorted-set, hash and marker calls the engine makes, faithfully."""
+
+ def __init__(self) -> None:
+ self.zsets: dict[str, dict[str, float]] = {}
+
+ self.strings: dict[str, str] = {}
+
+ async def zadd(self, key: str, mapping: dict[str, float]) -> None:
+ self.zsets.setdefault(key, {}).update(mapping)
+
+ async def zremrangebyscore(self, key: str, min_s: float, max_s: float) -> None:
+ members = self.zsets.get(key, {})
+ for member in [m for m, score in list(members.items()) if min_s <= score <= max_s]:
+ del members[member]
+
+ async def zcard(self, key: str) -> int:
+ return len(self.zsets.get(key, {}))
+
+ async def zrangebyscore(self, key: str, min_s: float, max_s: float, withscores: bool = False) -> list[Any]:
+ items = sorted(((m, s) for m, s in self.zsets.get(key, {}).items() if min_s <= s <= max_s), key=lambda kv: (kv[1], kv[0]))
+ return items if withscores else [m for m, _ in items]
+
+ async def expire(self, key: str, ttl: int) -> None:
+ return None
+
+ async def set(self, key: str, value: str, nx: bool = False, ex: int | None = None) -> Any:
+ if nx and key in self.strings:
+ return None
+ self.strings[key] = value
+ return True
+
+
+SPRAY_THEN_SUCCESS = SequenceRule(
+ id="wd-seq-test-spray-then-success",
+ name="Failed logons then a success for the same account",
+ severity="high",
+ category="identity",
+ mitre=["T1110"],
+ sequence=(
+ {"event_type": "authentication", "outcome": "failure"},
+ {"event_type": "authentication", "outcome": "success"},
+ ),
+ group_by="user",
+ window_seconds=600,
+)
+
+
+def _event(tenant: str, outcome: str, when: float, user: str = "alice") -> dict[str, Any]:
+ return {
+ "tenant_id": tenant,
+ "ocsf_event": {
+ "tenant_uid": tenant,
+ "raw_data": json.dumps({"event_type": "authentication", "outcome": outcome, "user": user, "event_time": when}),
+ },
+ }
+
+
+def _engine() -> WindowedDetectionEngine:
+ return WindowedDetectionEngine(_FakeRedis(), rules=(), sequences=(SPRAY_THEN_SUCCESS,))
+
+
+async def test_the_sequence_fires_in_order() -> None:
+ engine = _engine()
+ tenant = str(uuid4())
+ assert not await engine.evaluate(_event(tenant, "failure", 1000.0))
+ hits = await engine.evaluate(_event(tenant, "success", 1060.0))
+ assert [h.rule_id for h in hits] == [SPRAY_THEN_SUCCESS.id]
+
+
+async def test_the_last_stage_alone_does_not_fire() -> None:
+ """Otherwise the sequence is just its final stage with extra machinery."""
+ engine = _engine()
+ tenant = str(uuid4())
+ for offset in range(5):
+ assert not await engine.evaluate(_event(tenant, "success", 1000.0 + offset))
+
+
+async def test_out_of_order_by_event_time_does_not_fire() -> None:
+ """The success happened *before* the failure, so nothing was compromised."""
+ engine = _engine()
+ tenant = str(uuid4())
+ assert not await engine.evaluate(_event(tenant, "failure", 2000.0))
+ assert not await engine.evaluate(_event(tenant, "success", 1000.0))
+
+
+async def test_a_batch_delivered_out_of_arrival_order_still_fires() -> None:
+ """A connector polling a vendor delivers the vendor's order, not the clock's.
+
+ The success arrives first and is held; the failure that precedes it by
+ event time arrives second. Replaying the later event is what makes this
+ work, and without it every vendor that batches would be undetectable.
+ """
+ engine = _engine()
+ tenant = str(uuid4())
+ assert not await engine.evaluate(_event(tenant, "success", 1060.0))
+ hits = await engine.evaluate(_event(tenant, "failure", 1000.0))
+ assert [h.rule_id for h in hits] == [SPRAY_THEN_SUCCESS.id]
+
+
+async def test_the_stages_must_fit_inside_one_window() -> None:
+ engine = _engine()
+ tenant = str(uuid4())
+ assert not await engine.evaluate(_event(tenant, "failure", 1000.0))
+ assert not await engine.evaluate(_event(tenant, "success", 1000.0 + SPRAY_THEN_SUCCESS.window_seconds + 1))
+
+
+async def test_the_entity_must_be_the_same() -> None:
+ engine = _engine()
+ tenant = str(uuid4())
+ assert not await engine.evaluate(_event(tenant, "failure", 1000.0, user="alice"))
+ assert not await engine.evaluate(_event(tenant, "success", 1060.0, user="bob"))
+
+
+async def test_one_event_cannot_advance_two_stages() -> None:
+ """A rule whose stages overlap must still need two events."""
+ overlapping = SequenceRule(
+ id="wd-seq-test-overlap",
+ name="overlap",
+ severity="low",
+ category="identity",
+ mitre=[],
+ sequence=({"event_type": "authentication"}, {"event_type": "authentication"}),
+ group_by="user",
+ window_seconds=600,
+ )
+ engine = WindowedDetectionEngine(_FakeRedis(), rules=(), sequences=(overlapping,))
+ tenant = str(uuid4())
+ assert not await engine.evaluate(_event(tenant, "failure", 1000.0))
+ hits = await engine.evaluate(_event(tenant, "failure", 1001.0))
+ assert [h.rule_id for h in hits] == [overlapping.id]
+
+
+async def test_it_fires_once_per_window() -> None:
+ engine = _engine()
+ tenant = str(uuid4())
+ await engine.evaluate(_event(tenant, "failure", 1000.0))
+ assert await engine.evaluate(_event(tenant, "success", 1060.0))
+ assert not await engine.evaluate(_event(tenant, "success", 1120.0))
+
+
+async def test_a_three_stage_sequence_needs_all_three_in_order() -> None:
+ three = SequenceRule(
+ id="wd-seq-test-three",
+ name="three",
+ severity="high",
+ category="identity",
+ mitre=[],
+ sequence=(
+ {"event_type": "authentication", "outcome": "failure"},
+ {"event_type": "authentication", "outcome": "success"},
+ {"event_type": "authentication", "outcome": "mfa_reset"},
+ ),
+ group_by="user",
+ window_seconds=600,
+ )
+ engine = WindowedDetectionEngine(_FakeRedis(), rules=(), sequences=(three,))
+ tenant = str(uuid4())
+ assert not await engine.evaluate(_event(tenant, "failure", 1000.0))
+ assert not await engine.evaluate(_event(tenant, "mfa_reset", 1010.0))
+ assert not await engine.evaluate(_event(tenant, "success", 1020.0))
+ hits = await engine.evaluate(_event(tenant, "mfa_reset", 1030.0))
+ assert [h.rule_id for h in hits] == [three.id]
+
+
+async def test_tenants_do_not_share_sequence_progress() -> None:
+ engine = _engine()
+ one, two = str(uuid4()), str(uuid4())
+ assert not await engine.evaluate(_event(one, "failure", 1000.0))
+ assert not await engine.evaluate(_event(two, "success", 1060.0))
+
+
+async def test_an_event_with_no_time_falls_back_to_arrival() -> None:
+ """A source that stamps nothing must still be able to fire a sequence."""
+ engine = _engine()
+ tenant = str(uuid4())
+
+ def untimed(outcome: str) -> dict[str, Any]:
+ return {
+ "tenant_id": tenant,
+ "ocsf_event": {
+ "tenant_uid": tenant,
+ "raw_data": json.dumps({"event_type": "authentication", "outcome": outcome, "user": "alice"}),
+ },
+ }
+
+ assert not await engine.evaluate(untimed("failure"))
+ assert [h.rule_id for h in await engine.evaluate(untimed("success"))] == [SPRAY_THEN_SUCCESS.id]