Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
depth 3.2: ordered sequences in the windowed engine, and a Sigma correlation importer (#1224)
Beenu Arora51b60d484171
diff --git a/.github/workflows/validate-detections.yml b/.github/workflows/validate-detections.ymlindex 5f18ccced..993bf9717 100644--- a/.github/workflows/validate-detections.yml+++ b/.github/workflows/validate-detections.yml@@ -111,6 +111,19 @@ jobs:- name: Prove the windowed-translation gate is not vacuousrun: python3 scripts/check_windowed_translation.py --self-test+ # The Sigma compiler translated rule documents and had never looked at a+ # `correlation:` block. The two facts hid each other: there was no+ # importer, and the 3,132 imported documents contain no correlation for+ # it to have failed on. This compiles every correlation document into+ # the windowed engine or refuses it with a reason, and fails a rule+ # whose selector or group-by names a field nothing emits — with no+ # ceiling, because the corpus is new and starts clean.+ - name: Verify Sigma correlations compile into the windowed engine+ run: python3 scripts/check_sigma_correlations.py --check++ - name: Prove the Sigma-correlation gate is not vacuous+ run: python3 scripts/check_sigma_correlations.py --self-test+# Recompiles every imported Sigma rule AND re-proves it: each shipped# rule is replayed, as a vendor-shaped event, through its connector's# real normalize() and the real engine. A rule that stops firing — becausediff --git a/CHANGELOG.md b/CHANGELOG.mdindex 614bafa54..5a178b881 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -103,6 +103,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0The published executable figure falls 2,603 → 2,529 because 74 rules thatcould not fire stopped being counted as executable; the windowed corpus isnow published as its own figure beside it rather than going uncounted.+- **Ordered and unordered sequences in the windowed engine, and a Sigma+ correlation importer that feeds it.** A threshold cannot express an+ ordering, and the ordering is usually the detection: "fifty failed logons"+ is an attempt, "a failed logon then a success for the same account" is a+ compromise. `SequenceRule` stages two or more selectors for one entity+ inside one window, ordered by **event time** rather than arrival — almost+ every connector here polls, so a batch arrives in the vendor's order and+ the event that starts a sequence routinely lands after the one that+ finishes it. One event advances at most one stage, which a per-event token+ enforces rather than a timestamp comparison, because two distinct events+ routinely share a second. `ordered: false` gives Sigma's unordered+ `temporal` semantics from the same rule type.+ `scripts/sigma_correlation.py` compiles `event_count`, `value_count`,+ `temporal` and `temporal_ordered` documents into those forms and **refuses+ rather than approximates** the rest: a `lt`/`lte` condition is a rarity+ signal needing a baseline, a multi-field `group-by` has no single entity,+ `aliases` need field mapping the engine does not have. The importer had+ never been written and the imported corpus contains no correlation+ document, so the two gaps hid each other; the rules under+ `detections/sigma-correlations/` are hand-authored in the upstream format+ to give it a corpus and are labelled first-party content rather than+ imported coverage. `scripts/check_sigma_correlations.py` fails a+ correlation whose selector or `group-by` names a field nothing emits, with+ no ceiling.- **Attribute conditions, time-boxed elevation and workload identities, allinside the one permission path.** The placement is the item, not a detail:conditions are applied by `CurrentUser.require_permission` after whicheverdiff --git a/DEPTH_PROGRESS.md b/DEPTH_PROGRESS.mdindex 3f11f8389..6ca5c7853 100644--- a/DEPTH_PROGRESS.md+++ b/DEPTH_PROGRESS.md@@ -232,6 +232,52 @@ one engine that could not fire them to another. Both passes are now applied in`evaluate()`, which takes the overlay the consumer had already resolved forthe stateless engine.+### D7 -- There is no "existing watermark" in fusion to order sequences by++Item 3.2 says to order sequences "by event time with the existing+watermark". No watermark exists anywhere in the detection path: the only two+in the tree belong to `services/actions`' shadow-reconcile router and the+dead-letter replay, which are unrelated subsystems. Rather than invent one or+quietly order by arrival, the engine orders strictly by event time and+declares its own bound,+`SEQUENCE_REORDER_TOLERANCE_SECONDS = 300`, for how late an event may arrive+and still be stitched into a sequence. The bound is stated in the module+rather than inherited from something that does not exist.++Ordering by arrival would have been the easy implementation and is wrong+here: almost every connector in this tree **polls**, so a batch arrives in+the vendor's order and the event that starts a sequence routinely lands after+the one that finishes it. `test_a_batch_delivered_out_of_arrival_order_still_fires`+is the regression test for that, and reverting the event-time read makes it+and two others fail.++### D8 -- No upstream Sigma correlation rule exists in this tree to import++`detections/sigma-imports/` holds 3,132 rule documents and **not one carries+a `correlation:` block**, so the missing importer and the missing corpus hid+each other: there was nothing for an importer to have failed on. There is+also no fetcher for the upstream corpus (`scripts/` has `compile_sigma_ruleset.py`,+`sigma_compiler.py` and `sigma_proof_event.py`, none of which downloads+anything).++So the importer is built and exercised against **first-party** correlation+documents hand-authored in the upstream format under+`detections/sigma-correlations/`, four accepted and two refused. The figure+they contribute is first-party content and is labelled that way in the truth+table and in the corpus README; it is not imported coverage. Vendoring the+upstream correlation corpus needs a fetcher and a licence review, and is not+attempted here.++One standard is weaker here than for the stateless Sigma imports, and stays+weaker: those are replayed through the real connector `normalize()` by an+out-of-process worker, and these are replayed through the real **windowed+engine** only. The connector step is not applied, because importing+`services/connectors` into a fusion test process shadows fusion's own `app`+package — the documented reason `compile_sigma_ruleset.py` uses a subprocess.+Instead, every selector field, `group-by` and `distinct_by` is checked+against the emitted-field namespace with **no ceiling**, which caught three+invented fields on the first run of the gate.+## Phase 1: Verdict quality you can publish- [x] **1.1** Balanced, labelled verdict corpus@@ -257,6 +303,9 @@ the stateless engine.- [x] **3.1** Translate the 74 windowed `det-*` rules — 50 translated, 24refused with a reason. `MAX_UNREACHABLE` 119 → 45.+- [x] **3.2** Ordered sequences and Sigma correlations — the windowed engine+ stages ordered and unordered sequences; all four translatable Sigma+ correlation types compile. See D7 and D8.- [ ] **3.2** Ordered sequences and Sigma correlations- [ ] **3.3** Enrichment inputs (parity 5.5)- [ ] **3.4** Behavioural baselines in COREdiff --git a/README.md b/README.mdindex ca0d9550a..114202e51 100644--- a/README.md+++ b/README.md@@ -111,7 +111,7 @@ of those are partial: field matches carried, thresholds did not## How it worksIngest normalizes to a common shape and Kafka carries it, then-fusion runs 2529 executable detection rules, of 6991 on disk, plus 68 sliding-window rules that count across events, applies **your tenant's own tuning**+fusion runs 2529 executable detection rules, of 6991 on disk, plus 72 sliding-window rules that count across events, applies **your tenant's own tuning**on top — the disables, floors and suppressions the console writes, so a rule you turned off actually stops firing — and decides whatbecomes an alert. Correlation groups related alerts, an agent investigates and writes its reasoningto the Investigation Ledger, and a playbook may start from the result. Separately, new threatdiff --git a/apps/web/src/data/corpus-stats.json b/apps/web/src/data/corpus-stats.jsonindex e84df4df7..9bbc8e5de 100644--- a/apps/web/src/data/corpus-stats.json+++ b/apps/web/src/data/corpus-stats.json@@ -1,6 +1,6 @@{"executable": 2529,- "windowed": 68,+ "windowed": 72,"onDisk": 6991,"quarantined": 4462,"plugins": 80,diff --git a/detections/sigma-correlations/README.md b/detections/sigma-correlations/README.mdnew file mode 100644index 000000000..de5bf368f--- /dev/null+++ b/detections/sigma-correlations/README.md@@ -0,0 +1,45 @@+# Sigma correlation rules++Correlation documents in the [Sigma correlations](https://github.com/SigmaHQ/sigma-specification)+format, compiled into the windowed engine by `scripts/sigma_correlation.py`+and exported into `services/fusion/app/data/windowed_ruleset.json` by+`scripts/export_windowed_ruleset.py`.++## These are first-party, not upstream imports++**No upstream SigmaHQ correlation rule is vendored here, and none was at the+time this directory was created** — `detections/sigma-imports/` holds 3,132+rule documents and not one carries a `correlation:` block, so the import path+had no corpus to run against and nothing would have exercised it. The rules+here are hand-authored under this repository's own licence, in the upstream+format, to give the compiler real input. Treat the figure they contribute as+first-party content, not as imported coverage.++Two consequences worth stating rather than discovering:++* **`group-by` names a field this platform normalizes**, not always the+ vendor's own spelling. Upstream would write `eventSource`; a rule that has+ to accumulate per principal needs `user_arn`, which is what the CloudTrail+ connector emits. `scripts/check_sigma_correlations.py` fails a rule whose+ `group-by` or selector names a field nothing emits, so this stays checkable+ rather than aspirational.+* **A correlation is not replayed through its connector's `normalize()`.**+ The stateless Sigma imports are (`scripts/compile_sigma_ruleset.py` runs an+ out-of-process worker per rule); these are replayed through the real+ windowed engine instead, in+ `services/fusion/tests/test_sigma_correlation_replay.py`. That proves the+ compiled rule fires on an event of the shape its selector describes. It+ does not prove a given vendor populates those fields on a given event.++## Layout++One file per correlation, multi-document YAML as the specification describes:+the base rules first, each with a `name:`, then the correlation document+referencing them.++## Refusals are part of the corpus++Files under `_refused/` are correlations the engine cannot express. They are+kept, not deleted, because a refusal with a reason is the useful artefact —+and because the compiler's taxonomy needs real inputs to be worth anything.+`scripts/check_sigma_correlations.py --report` prints which, and why.diff --git a/detections/sigma-correlations/_refused/aws-login-failures-by-principal-and-region.yml b/detections/sigma-correlations/_refused/aws-login-failures-by-principal-and-region.ymlnew file mode 100644index 000000000..772deec2f--- /dev/null+++ b/detections/sigma-correlations/_refused/aws-login-failures-by-principal-and-region.yml@@ -0,0 +1,34 @@+title: AWS console sign-in failure+name: aws-console-login-failed-multi+id: aisoc-corr-base-aws-console-login-failed-multi+status: experimental+description: One failed AWS console sign-in.+logsource:+ product: aws+ service: cloudtrail+detection:+ selection:+ eventSource: signin.amazonaws.com+ eventName: ConsoleLogin+ errorMessage: 'Failed authentication'+ condition: selection+---+title: AWS sign-in failures grouped by principal and region+id: aisoc-corr-aws-login-failures-principal-and-region+status: experimental+description: >-+ Kept as a refusal. The engine accumulates against one entity; choosing either+ field would group by something the author did not ask for, and silently widen+ or narrow every count.+level: medium+category: cloud+correlation:+ type: event_count+ rules:+ - aws-console-login-failed-multi+ group-by:+ - user_arn+ - awsRegion+ timespan: 10m+ condition:+ gt: 20diff --git a/detections/sigma-correlations/_refused/okta-rare-admin-action.yml b/detections/sigma-correlations/_refused/okta-rare-admin-action.ymlnew file mode 100644index 000000000..726597ea7--- /dev/null+++ b/detections/sigma-correlations/_refused/okta-rare-admin-action.yml@@ -0,0 +1,31 @@+title: Okta administrative action+name: okta-admin-action+id: aisoc-corr-base-okta-admin-action+status: experimental+description: One Okta administrative API call.+logsource:+ product: okta+ service: okta+detection:+ selection:+ eventType: user.account.privilege.grant+ condition: selection+---+title: An account performing unusually few administrative actions+id: aisoc-corr-okta-rare-admin-action+status: experimental+description: >-+ Kept as a refusal, not as a rule. A `lt` condition fires when little+ happened, which needs a store of what is normal for that account rather than+ a counter over a window.+level: low+category: identity+correlation:+ type: event_count+ rules:+ - okta-admin-action+ group-by:+ - actor_email+ timespan: 1h+ condition:+ lt: 5diff --git a/detections/sigma-correlations/aws-console-login-failure-burst.yml b/detections/sigma-correlations/aws-console-login-failure-burst.ymlnew file mode 100644index 000000000..8a94e4744--- /dev/null+++ b/detections/sigma-correlations/aws-console-login-failure-burst.yml@@ -0,0 +1,35 @@+title: AWS IAM access key created+name: aws-iam-create-access-key+id: aisoc-corr-base-aws-iam-create-access-key+status: experimental+description: One IAM CreateAccessKey call. Routine during onboarding; the correlation below is the detection.+logsource:+ product: aws+ service: cloudtrail+detection:+ selection:+ eventSource: iam.amazonaws.com+ eventName: CreateAccessKey+ condition: selection+---+title: One principal minted several AWS access keys in a burst+id: aisoc-corr-aws-access-key-minting-burst+status: experimental+description: >-+ Six or more IAM access keys created by the same principal inside ten minutes.+ Key creation is legitimate and common, so the detection is the rate: an+ onboarding script creates one key per user, and a principal creating several+ in minutes is either a misconfigured loop or persistence being established.+level: high+category: cloud+tags:+ - attack.t1098.001+correlation:+ type: event_count+ rules:+ - aws-iam-create-access-key+ group-by:+ - user_arn+ timespan: 10m+ condition:+ gt: 5diff --git a/detections/sigma-correlations/aws-principal-across-many-regions.yml b/detections/sigma-correlations/aws-principal-across-many-regions.ymlnew file mode 100644index 000000000..d8d5b6eb3--- /dev/null+++ b/detections/sigma-correlations/aws-principal-across-many-regions.yml@@ -0,0 +1,35 @@+title: AWS API call+name: aws-api-call+id: aisoc-corr-base-aws-api-call+status: experimental+description: Any CloudTrail management-plane call. Meaningless alone; the correlation counts distinct regions.+logsource:+ product: aws+ service: cloudtrail+detection:+ selection:+ cloud_platform: aws+ condition: selection+---+title: One principal acting across an unusual number of AWS regions+id: aisoc-corr-aws-principal-region-fanout+status: experimental+description: >-+ Six or more distinct regions touched by one principal inside fifteen minutes.+ Distinct regions rather than call volume: an estate uses two or three+ regions, and a principal sweeping the rest is the shape of enumeration or of+ resource hijacking looking for quota.+level: medium+category: cloud+tags:+ - attack.t1580+correlation:+ type: value_count+ rules:+ - aws-api-call+ group-by:+ - user_arn+ field: awsRegion+ timespan: 15m+ condition:+ gt: 5diff --git a/detections/sigma-correlations/azure-role-grant-and-key-creation.yml b/detections/sigma-correlations/azure-role-grant-and-key-creation.ymlnew file mode 100644index 000000000..d1a3f8309--- /dev/null+++ b/detections/sigma-correlations/azure-role-grant-and-key-creation.yml@@ -0,0 +1,45 @@+title: Entra directory role assigned+name: azure-directory-role-assigned+id: aisoc-corr-base-azure-role-assigned+status: experimental+description: One Entra directory role assignment.+logsource:+ product: azure+ service: auditlogs+detection:+ selection:+ operationName: Add member to role+ condition: selection+---+title: Azure resource lock deleted+name: azure-resource-lock-deleted+id: aisoc-corr-base-azure-lock-deleted+status: experimental+description: One Azure management lock removal.+logsource:+ product: azure+ service: activitylogs+detection:+ selection:+ operationName: Microsoft.Authorization/locks/delete+ condition: selection+---+title: One actor both assigned a directory role and removed a resource lock+id: aisoc-corr-azure-role-grant-and-lock-removal+status: experimental+description: >-+ Two administrative actions by the same actor inside thirty minutes, in either+ order. Each is legitimate on its own and neither is rare; together they are+ the shape of an actor clearing the way before changing something protected.+level: medium+category: cloud+tags:+ - attack.t1098+correlation:+ type: temporal+ rules:+ - azure-directory-role-assigned+ - azure-resource-lock-deleted+ group-by:+ - actor+ timespan: 30mdiff --git a/detections/sigma-correlations/okta-mfa-denials-then-success.yml b/detections/sigma-correlations/okta-mfa-denials-then-success.ymlnew file mode 100644index 000000000..ddf5363c3--- /dev/null+++ b/detections/sigma-correlations/okta-mfa-denials-then-success.yml@@ -0,0 +1,47 @@+title: Okta MFA verification push sent+name: okta-mfa-push-sent+id: aisoc-corr-base-okta-mfa-push+status: experimental+description: One Okta push factor verification. Routine on its own.+logsource:+ product: okta+ service: okta+detection:+ selection:+ eventType: system.push.send_factor_verify_push+ condition: selection+---+title: Okta session started+name: okta-session-start+id: aisoc-corr-base-okta-session-start+status: experimental+description: One Okta session start.+logsource:+ product: okta+ service: okta+detection:+ selection:+ eventType: user.session.start+ condition: selection+---+title: Okta MFA push followed by a session for the same account+id: aisoc-corr-okta-mfa-push-then-session+status: experimental+description: >-+ A push factor verification and then a session start for the same account+ inside fifteen minutes. The ordering is the detection: pushes alone are an+ attempt, and a session that follows them is the attempt succeeding. Pair this+ with the push-volume rule rather than using it alone — one push followed by a+ session is the ordinary path a user takes every morning.+level: medium+category: identity+tags:+ - attack.t1621+correlation:+ type: temporal_ordered+ rules:+ - okta-mfa-push-sent+ - okta-session-start+ group-by:+ - actor_email+ timespan: 15mdiff --git a/docs/detections/truth-table.md b/docs/detections/truth-table.mdindex b1e8e3a4c..24cffc82e 100644--- a/docs/detections/truth-table.md+++ b/docs/detections/truth-table.md@@ -23,7 +23,7 @@ lever that reaches the engine is adding a spec and re-running| **executable (loaded by the engine)** | **2529** || non-executable (provenance/coverage only) | 4462 || — of which: enabled, but no compiled spec | 0 |-| windowed rules in `windowed_ruleset.json` | 68 |+| windowed rules in `windowed_ruleset.json` | 72 |The windowed figure is its own row rather than part of the headline becausea windowed rule is a different kind of thing: it counts events or distinct@@ -42,6 +42,12 @@ from the original's own clauses and replayed through the real enginethat family could not be expressed as a count over a window and are refusedwith a reason that travels into each rule's `quarantine_reason`.+Four more are Sigma correlation documents compiled by+`scripts/sigma_correlation.py`. They are **first-party content, not imported+coverage**: no upstream correlation rule is vendored in this tree, so the+import path had no corpus and the rules under `detections/sigma-correlations/`+were hand-authored in the upstream format to give it one.+## By tier| tier | on disk | executable | enabled but not loaded |diff --git a/scripts/check_sigma_correlations.py b/scripts/check_sigma_correlations.pynew file mode 100644index 000000000..6af1d8c8b--- /dev/null+++ b/scripts/check_sigma_correlations.py@@ -0,0 +1,259 @@+#!/usr/bin/env python3+"""CI gate: every Sigma correlation compiles into the windowed engine, or is refused.++The gap this closes+-------------------+`scripts/compile_sigma_ruleset.py` translates Sigma *rule* documents into the+stateless matcher. It has never looked at a `correlation:` block, and the+3,132 imported rule documents do not contain one, so the two facts hid each+other: there was no importer and no corpus for it to fail on. A correlation+quietly ignored by an importer is the worst shape available — the rule is in+the tree, it is in the count of what was imported, and it detects nothing.++What this enforces+------------------+* every correlation document in `detections/sigma-correlations/` either+ compiles or carries a refusal reason, and the two sets do not overlap;+* a compiled correlation's selector fields, `group-by` and `distinct_by` are+ names some connector, ingest template or the Go normalizer actually emits —+ the same standard `check_detection_fields.py` holds the native corpus to,+ applied here with **no ceiling**, because this corpus is new and starts+ clean;+* everything under `_refused/` is in fact refused, so a file cannot be parked+ there to dodge the checks above;+* the compiled output is what `windowed_ruleset.json` ships.++Usage:+ python3 scripts/check_sigma_correlations.py # enforce+ python3 scripts/check_sigma_correlations.py --check # same, for CI+ python3 scripts/check_sigma_correlations.py --report # the taxonomy+ python3 scripts/check_sigma_correlations.py --self-test # prove the gate works+"""++from __future__ import annotations++import argparse+import importlib.util+import json+import sys+from pathlib import Path+from typing import Any++import yaml++sys.path.insert(0, str(Path(__file__).resolve().parent))++from gate_toolkit import SKELETON, refuses_an_empty_tree, repo_root, self_test_main+from sigma_compiler import Refusal, compile_rule+from sigma_correlation import CorrelationRefusal, compile_correlation++ROOT = repo_root()+CORPUS = ROOT / "detections" / "sigma-correlations"+WINDOWED = ROOT / "services" / "fusion" / "app" / "data" / "windowed_ruleset.json"+REFUSED_DIR = "_refused"+++def _documents(path: Path) -> list[dict[str, Any]]:+ try:+ docs = list(yaml.safe_load_all(path.read_text(encoding="utf-8")))+ except yaml.YAMLError:+ return []+ return [d for d in docs if isinstance(d, dict)]+++def compile_corpus() -> tuple[list[dict[str, Any]], list[tuple[str, str]], int]:+ """Returns (compiled windowed rules, [(file, reason)], documents read).++ The document count is returned so a caller can refuse a corpus that read+ nothing: a compiler over zero files produces zero refusals and zero+ rules, which reads exactly like a clean corpus.+ """+ compiled: list[dict[str, Any]] = []+ refused: list[tuple[str, str]] = []+ seen = 0++ for path in sorted(CORPUS.rglob("*.yml")) + sorted(CORPUS.rglob("*.yaml")):+ docs = _documents(path)+ if not docs:+ continue+ seen += len(docs)+ relative = str(path.relative_to(ROOT))++ selectors: dict[str, dict[str, Any]] = {}+ base_refusals: list[str] = []+ for doc in docs:+ if doc.get("correlation"):+ continue+ name = str(doc.get("name") or doc.get("id") or "").strip()+ if not name:+ base_refusals.append("a base rule has neither a name nor an id")+ continue+ try:+ selectors[name] = compile_rule(doc).match_when+ except Refusal as exc:+ base_refusals.append(f"base rule {name!r}: {exc}")++ correlation_ids = {str(d.get("id") or "") for d in docs if d.get("correlation")}+ for doc in docs:+ if not doc.get("correlation"):+ continue+ try:+ if base_refusals:+ raise CorrelationRefusal("a referenced rule did not compile", "; ".join(base_refusals))+ compiled.append(compile_correlation(doc, selectors=selectors, correlations=frozenset(correlation_ids)))+ except CorrelationRefusal as exc:+ refused.append((relative, str(exc)))+ return compiled, refused, seen+++def _namespace() -> set[str]:+ spec = importlib.util.spec_from_file_location("check_detection_fields", ROOT / "scripts" / "check_detection_fields.py")+ if spec is None or spec.loader is None: # pragma: no cover+ raise SystemExit("cannot load scripts/check_detection_fields.py")+ module = importlib.util.module_from_spec(spec)+ sys.modules["check_detection_fields"] = module+ spec.loader.exec_module(module)+ return module._connector_namespace() | module._template_namespace() | module._go_normalizer_namespace()+++def _selector_fields(rule: dict[str, Any]) -> set[str]:+ spec = importlib.util.spec_from_file_location("check_detection_fields", ROOT / "scripts" / "check_detection_fields.py")+ module = importlib.util.module_from_spec(spec) # type: ignore[arg-type]+ sys.modules["check_detection_fields"] = module+ spec.loader.exec_module(module) # type: ignore[union-attr]+ clauses = [rule["match_when"]] if rule.get("match_when") else list(rule.get("sequence") or [])+ found: set[str] = set()+ for clause in clauses:+ found |= module._rule_fields(clause)+ for key in ("group_by", "distinct_by"):+ if rule.get(key):+ found.add(str(rule[key]))+ return found+++def check(compiled: list[dict[str, Any]], refused: list[tuple[str, str]], seen: int, shipped: set[str]) -> list[str]:+ problems: list[str] = []+ if seen == 0:+ problems.append(f"no correlation documents read under {CORPUS.relative_to(ROOT)} — refusing to report a corpus of zero as clean")+ return problems++ namespace = _namespace()+ compiled_by_id: dict[str, dict[str, Any]] = {}+ for rule in compiled:+ rule_id = str(rule.get("id") or "")+ if rule_id in compiled_by_id:+ problems.append(f"two correlations share the id {rule_id!r}")+ compiled_by_id[rule_id] = rule++ unknown = sorted(f for f in _selector_fields(rule) if f not in namespace)+ if unknown:+ problems.append(+ f"{rule_id} names field(s) nothing emits: {', '.join(unknown)}. "+ "A correlation grouping on a field no source produces counts nothing forever."+ )+ if rule_id not in shipped:+ problems.append(f"{rule_id} compiled but is not in {WINDOWED.name} — re-run scripts/export_windowed_ruleset.py")++ # Everything under `_refused/` must in fact be refused, or the directory+ # becomes a place to park a rule away from the checks above.+ parked = {path for path, _ in refused}+ for path in sorted(CORPUS.rglob("*.yml")):+ relative = str(path.relative_to(ROOT))+ in_refused_dir = REFUSED_DIR in path.relative_to(CORPUS).parts+ if in_refused_dir and relative not in parked:+ problems.append(f"{relative} sits under {REFUSED_DIR}/ but compiled cleanly — move it out or say why it cannot run")+ if not in_refused_dir and relative in parked:+ reason = next(r for p, r in refused if p == relative)+ problems.append(f"{relative} was refused but is not under {REFUSED_DIR}/: {reason}")+ return problems+++def _self_test_injections(+ compiled: list[dict[str, Any]], refused: list[tuple[str, str]], seen: int, shipped: set[str]+) -> list[tuple[str, bool]]:+ results = [("the corpus as committed has no findings to begin with", not check(compiled, refused, seen, shipped))]++ invented = [*compiled, {**compiled[0], "id": "aisoc-corr-injected", "group_by": "a_field_nothing_emits"}]+ results.append(+ (+ "catches a correlation grouping on a field nothing emits",+ any("nothing emits" in p for p in check(invented, refused, seen, shipped | {"aisoc-corr-injected"})),+ )+ )++ results.append(("catches a compiled correlation missing from the windowed ruleset", bool(check(compiled, refused, seen, set()))))++ duplicated = [*compiled, dict(compiled[0])]+ results.append(("catches two correlations sharing an id", any("share the id" in p for p in check(duplicated, refused, seen, shipped))))++ # Drop the real refusals so the files under `_refused/` look as though+ # they compiled. Expressed by removing them from the refusal list rather+ # than by writing a file, because the check reads the directory and a+ # self-test that mutates the tree is one that can leave it dirty.+ as_if_compiling = [entry for entry in refused if REFUSED_DIR not in entry[0]]+ results.append(+ (+ "catches a rule parked under _refused/ that in fact compiles",+ any("compiled cleanly" in p for p in check(compiled, as_if_compiling, seen, shipped)),+ )+ )++ results.append(("refuses a corpus it read nothing from", bool(check(compiled, refused, 0, shipped))))++ skeleton_refused, detail = refuses_an_empty_tree(Path(__file__).name, ["--check"], shape=SKELETON)+ results.append((f"refuses a tree whose directories exist and hold no correlations ({detail.splitlines()[0]})", skeleton_refused))+ return results+++def main() -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--check", action="store_true", help="enforce (the default; accepted so CI can be explicit)")+ parser.add_argument("--report", action="store_true", help="print the compiled and refused taxonomy")+ parser.add_argument("--self-test", action="store_true", help="prove the gate catches what it claims to")+ args = parser.parse_args()++ if not CORPUS.is_dir():+ print(f"{Path(sys.argv[0]).name}: no correlation corpus at {CORPUS} — nothing to check", file=sys.stderr)+ return 2++ compiled, refused, seen = compile_corpus()+ shipped: set[str] = set()+ if WINDOWED.is_file():+ shipped = {str(r.get("id")) for r in json.loads(WINDOWED.read_text(encoding="utf-8")).get("rules") or []}++ if args.self_test:+ if not compiled:+ print("sigma correlations: nothing compiled, so the injections below would prove nothing", file=sys.stderr)+ return 1+ return self_test_main(Path(__file__).name, ["--check"], extra=_self_test_injections(compiled, refused, seen, shipped))++ if args.report:+ for rule in compiled:+ kind = rule.get("correlation_type")+ print(f"COMPILED {rule['id']} [{kind}] group_by={rule['group_by']} window={rule['window_seconds']}s")+ for path, reason in refused:+ print(f"REFUSED {path}: {reason}")+ return 0++ problems = check(compiled, refused, seen, shipped)+ kinds: dict[str, int] = {}+ for rule in compiled:+ kind = str(rule.get("correlation_type"))+ kinds[kind] = kinds.get(kind, 0) + 1+ print(f"sigma correlations: {seen} documents read, {len(compiled)} compiled, {len(refused)} refused")+ for kind, count in sorted(kinds.items()):+ print(f" {count:4d} {kind}")+ for path, reason in refused:+ print(f" refused {Path(path).name}: {reason}")++ if problems:+ print(f"\nERROR: {len(problems)} problem(s):", file=sys.stderr)+ for problem in problems:+ print(f" - {problem}", file=sys.stderr)+ return 1+ print("\nOK: every correlation compiles into the windowed engine or is refused with a reason")+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/scripts/detection_truth_table.py b/scripts/detection_truth_table.pyindex 2024045a6..54bbdfd87 100644--- a/scripts/detection_truth_table.py+++ b/scripts/detection_truth_table.py@@ -310,6 +310,12 @@ def render_markdown(c: Counts) -> str:lines.append("that family could not be expressed as a count over a window and are refused")lines.append("with a reason that travels into each rule's `quarantine_reason`.")lines.append("")+ lines.append("Four more are Sigma correlation documents compiled by")+ lines.append("`scripts/sigma_correlation.py`. They are **first-party content, not imported")+ lines.append("coverage**: no upstream correlation rule is vendored in this tree, so the")+ lines.append("import path had no corpus and the rules under `detections/sigma-correlations/`")+ lines.append("were hand-authored in the upstream format to give it one.")+ lines.append("")if c.counted_but_not_loaded:lines.append(f"Those {c.counted_but_not_loaded} rules are the ones an earlier version of this")lines.append("table counted as executable. They carry `enabled: true` and a body whose")diff --git a/scripts/export_windowed_ruleset.py b/scripts/export_windowed_ruleset.pyindex a2cf06c0b..4810a653e 100644--- a/scripts/export_windowed_ruleset.py+++ b/scripts/export_windowed_ruleset.py@@ -334,14 +334,19 @@ WINDOWED_RULES: list[dict] = [def all_rules() -> list[dict]:- """Rules authored here, then rules translated from the `det-*` corpus.+ """Three sources, kept in their own blocks so a change to one is legible.- Order matters only for the diff: keeping the translations in their own- block means a change to one never renumbers the other.+ Authored here; translated from the `det-*` corpus by+ `windowed_translation.py`; compiled from Sigma correlation documents by+ `sigma_correlation.py`. One artefact rather than three, because the+ engine has one loader and a second file is a second thing that can go+ stale unnoticed."""+ from check_sigma_correlations import compile_corpus # noqa: PLC0415from windowed_translation import translated_rules # noqa: PLC0415- return [*WINDOWED_RULES, *translated_rules()]+ correlations, _refused, _seen = compile_corpus()+ return [*WINDOWED_RULES, *translated_rules(), *correlations]def build() -> dict:@@ -383,8 +388,12 @@ def main() -> int:OUT.parent.mkdir(parents=True, exist_ok=True)OUT.write_text(rendered, encoding="utf-8")- translated = len(rules) - len(WINDOWED_RULES)- print(f"wrote {OUT.relative_to(ROOT)} — {len(rules)} windowed rules ({len(WINDOWED_RULES)} authored, {translated} translated)")+ translated = sum(1 for rule in rules if rule.get("translated_from"))+ correlations = sum(1 for rule in rules if rule.get("correlation_type"))+ print(+ f"wrote {OUT.relative_to(ROOT)} — {len(rules)} windowed rules "+ f"({len(WINDOWED_RULES)} authored, {translated} translated, {correlations} Sigma correlations)"+ )return 0diff --git a/scripts/sigma_correlation.py b/scripts/sigma_correlation.pynew file mode 100644index 000000000..5f7a27c46--- /dev/null+++ b/scripts/sigma_correlation.py@@ -0,0 +1,228 @@+#!/usr/bin/env python3+"""Compile Sigma correlation rules into the windowed engine's form.++Specification: https://github.com/SigmaHQ/sigma-specification — "Sigma+Correlations", the ``correlation:`` document type. Read before writing this;+every field name and correlation type below is taken from it rather than+guessed.++What the engine can and cannot express+--------------------------------------+The windowed engine counts events or distinct values for one entity over a+sliding window, and stages ordered or unordered sets of events for one entity+over a sliding window. That covers four of Sigma's correlation types:++``event_count`` → a counting rule.+``value_count`` → a counting rule with ``distinct_by``.+``temporal`` → an unordered staged rule.+``temporal_ordered`` → an ordered staged rule.++It does not cover the rest, and the rule here is **refuse rather than+approximate**. A correlation that is translated with one of its constraints+dropped is a different detection shipping under the original's name and+severity, which is the specific failure the windowed translation work in+depth plan 3.1 existed to stop. Every refusal carries a reason, and the+refusals are counted and published the same way the Sigma compiler's are.++The two refusals worth knowing about in advance, because they look like+oversights:++* **A ``lt`` / ``lte`` condition is a rarity signal.** "Fewer than five in an+ hour" fires when nothing much happened, so it needs a store of what is+ normal rather than a counter. The engine fires when a count is high.+* **A multi-field ``group-by`` has no single entity.** The engine accumulates+ against one field. Picking the first would group by something the rule+ author did not ask for and silently widen or narrow every count.+"""++from __future__ import annotations++import sys+from dataclasses import dataclass+from pathlib import Path+from typing import Any++sys.path.insert(0, str(Path(__file__).resolve().parent))++__all__ = [+ "CORRELATION_TYPES",+ "CorrelationRefusal",+ "compile_correlation",+ "parse_timespan",+]++#: Correlation types this compiler accepts, mapped to the shape they become.+CORRELATION_TYPES: dict[str, str] = {+ "event_count": "counting",+ "value_count": "counting",+ "temporal": "staged",+ "temporal_ordered": "staged",+}++R_UNKNOWN_TYPE = "correlation type is not one this engine can express"+R_CONDITION_SHAPE = "the condition is not a single comparison the engine can threshold"+R_RARITY = "a `lt`/`lte` condition fires when a count is low, which needs a baseline rather than a counter"+R_EQ = "an `eq` condition fires on an exact count, and a sliding window cannot hold a count still"+R_MULTI_GROUP = "`group-by` names more than one field and the engine accumulates against one entity"+R_NO_GROUP = "`group-by` is absent, so the count has no entity and would be global"+R_TIMESPAN = "`timespan` is missing or not a Sigma duration"+R_NO_RULES = "`rules` names no referenced rule"+R_UNRESOLVED = "a referenced rule is not in the corpus or did not compile"+R_ONE_STAGE = "a temporal correlation over a single rule is a plain count, not a sequence"+R_VALUE_FIELD = "`value_count` needs a `field` naming what to count distinctly"+R_ALIASES = "`aliases` map different field names across referenced rules and the engine has no aliasing"+R_GENERATE = "`generate: true` also emits the referenced rules, which this path does not model"+R_CHAINED = "the correlation references another correlation, which the engine cannot stage"++#: Sigma duration suffixes, from the specification.+_UNITS: dict[str, int] = {"s": 1, "m": 60, "h": 3600, "d": 86400, "w": 604800}+++class CorrelationRefusal(Exception):+ """This correlation cannot be expressed without changing what it detects."""++ def __init__(self, reason: str, detail: str = "") -> None:+ super().__init__(f"{reason}{f' ({detail})' if detail else ''}")+ self.reason = reason+ self.detail = detail+++def parse_timespan(raw: Any) -> int:+ """Sigma duration (``5m``, ``1h``, ``30s``) to seconds."""+ text = str(raw or "").strip()+ if len(text) < 2 or text[-1] not in _UNITS or not text[:-1].isdigit():+ raise CorrelationRefusal(R_TIMESPAN, text or "absent")+ seconds = int(text[:-1]) * _UNITS[text[-1]]+ if seconds < 1:+ raise CorrelationRefusal(R_TIMESPAN, text)+ return seconds+++@dataclass(frozen=True)+class _Condition:+ operator: str+ value: int+++def _parse_condition(block: Any, correlation_type: str) -> _Condition:+ """The ``condition:`` mapping, which Sigma writes as one comparison."""+ if not isinstance(block, dict) or len(block) != 1:+ raise CorrelationRefusal(R_CONDITION_SHAPE, repr(block))+ ((operator, value),) = block.items()+ operator = str(operator).lower()+ if operator in {"lt", "lte"}:+ raise CorrelationRefusal(R_RARITY, f"{operator}: {value}")+ if operator == "eq":+ raise CorrelationRefusal(R_EQ, f"eq: {value}")+ if operator not in {"gt", "gte"}:+ raise CorrelationRefusal(R_CONDITION_SHAPE, operator)+ try:+ bound = int(value)+ except (TypeError, ValueError) as exc:+ raise CorrelationRefusal(R_CONDITION_SHAPE, f"{operator}: {value!r}") from exc+ if bound < 0:+ raise CorrelationRefusal(R_CONDITION_SHAPE, f"{operator}: {bound}")+ # `correlation_type` is carried so a later type with different threshold+ # semantics cannot silently reuse this.+ del correlation_type+ return _Condition(operator=operator, value=bound)+++def _group_by(block: Any) -> str:+ fields = block if isinstance(block, list) else [block] if block else []+ names = [str(f) for f in fields if str(f or "").strip()]+ if not names:+ raise CorrelationRefusal(R_NO_GROUP)+ if len(names) > 1:+ raise CorrelationRefusal(R_MULTI_GROUP, ", ".join(names))+ return names[0]+++def _referenced(block: dict[str, Any]) -> list[str]:+ """`rules:` lives inside the correlation block, per the specification."""+ raw = block.get("rules")+ names = [str(r) for r in (raw if isinstance(raw, list) else [raw] if raw else []) if str(r or "").strip()]+ if not names:+ raise CorrelationRefusal(R_NO_RULES)+ return names+++def compile_correlation(+ doc: dict[str, Any],+ *,+ selectors: dict[str, dict[str, Any]],+ correlations: frozenset[str] = frozenset(),+) -> dict[str, Any]:+ """One Sigma correlation document to one windowed rule.++ ``selectors`` maps a referenced rule's name (or id) to the ``match_when``+ it compiled to. A correlation whose referenced rule is missing is refused+ rather than compiled against an empty selector — an empty selector+ matches every event, so that refusal is the difference between a rule and+ an outage.+ """+ block = doc.get("correlation")+ if not isinstance(block, dict):+ raise CorrelationRefusal(R_UNKNOWN_TYPE, "no correlation block")++ correlation_type = str(block.get("type") or "").strip().lower()+ if correlation_type not in CORRELATION_TYPES:+ raise CorrelationRefusal(R_UNKNOWN_TYPE, correlation_type or "absent")+ if block.get("aliases"):+ raise CorrelationRefusal(R_ALIASES)+ if block.get("generate"):+ raise CorrelationRefusal(R_GENERATE)++ window_seconds = parse_timespan(block.get("timespan"))+ group_by = _group_by(block.get("group-by"))+ names = _referenced(block)+ if any(name in correlations for name in names):+ raise CorrelationRefusal(R_CHAINED, ", ".join(n for n in names if n in correlations))+ missing = [name for name in names if name not in selectors]+ if missing:+ raise CorrelationRefusal(R_UNRESOLVED, ", ".join(missing))++ common = {+ "id": str(doc.get("id") or "").strip(),+ "name": str(doc.get("title") or doc.get("name") or "").strip(),+ "severity": str(doc.get("level") or doc.get("severity") or "medium").strip().lower(),+ "category": str(doc.get("category") or "identity").strip().lower(),+ "mitre": sorted({str(t).upper().removeprefix("ATTACK.") for t in doc.get("tags") or [] if str(t).lower().startswith("attack.t")}),+ "group_by": group_by,+ "window_seconds": window_seconds,+ "correlation_type": correlation_type,+ "correlates": names,+ }+ if not common["id"] or not common["name"]:+ raise CorrelationRefusal(R_CONDITION_SHAPE, "the document has no id or title")++ if CORRELATION_TYPES[correlation_type] == "counting":+ if len(names) != 1:+ # Sigma allows several, meaning "events matching any of them".+ # The engine's `match_when` is one clause set, and an `any_of`+ # over compiled selectors is a faithful translation only when the+ # selectors are disjoint, which nothing here can establish.+ raise CorrelationRefusal(R_CONDITION_SHAPE, f"{correlation_type} over {len(names)} rules")+ condition = _parse_condition(block.get("condition"), correlation_type)+ rule: dict[str, Any] = {+ **common,+ "match_when": dict(selectors[names[0]]),+ # `gt: N` crosses at N+1; `gte: N` crosses at N.+ "threshold": condition.value + 1 if condition.operator == "gt" else condition.value,+ }+ if correlation_type == "value_count":+ field = str(block.get("field") or "").strip()+ if not field:+ raise CorrelationRefusal(R_VALUE_FIELD)+ rule["distinct_by"] = field+ if rule["threshold"] < 1:+ raise CorrelationRefusal(R_CONDITION_SHAPE, "threshold below one fires on the first event")+ return rule++ if len(names) < 2:+ raise CorrelationRefusal(R_ONE_STAGE, names[0])+ return {+ **common,+ "sequence": [dict(selectors[name]) for name in names],+ "ordered": correlation_type == "temporal_ordered",+ }diff --git a/services/fusion/app/data/windowed_ruleset.json b/services/fusion/app/data/windowed_ruleset.jsonindex 38bc9bff4..98c4c49d9 100644--- a/services/fusion/app/data/windowed_ruleset.json+++ b/services/fusion/app/data/windowed_ruleset.json@@ -1,5 +1,5 @@{- "count": 68,+ "count": 72,"rules": [{"category": "application",@@ -1136,6 +1136,110 @@"threshold": 201,"translated_from": "net-dhcp-starvation","window_seconds": 60+ },+ {+ "category": "cloud",+ "correlates": [+ "aws-iam-create-access-key"+ ],+ "correlation_type": "event_count",+ "group_by": "user_arn",+ "id": "aisoc-corr-aws-access-key-minting-burst",+ "match_when": {+ "eventName_pattern_match_any": [+ "^CreateAccessKey$"+ ],+ "eventSource_pattern_match_any": [+ "^iam\\.amazonaws\\.com$"+ ]+ },+ "mitre": [+ "T1098.001"+ ],+ "name": "One principal minted several AWS access keys in a burst",+ "severity": "high",+ "threshold": 6,+ "window_seconds": 600+ },+ {+ "category": "cloud",+ "correlates": [+ "aws-api-call"+ ],+ "correlation_type": "value_count",+ "distinct_by": "awsRegion",+ "group_by": "user_arn",+ "id": "aisoc-corr-aws-principal-region-fanout",+ "match_when": {+ "cloud_platform_pattern_match_any": [+ "^aws$"+ ]+ },+ "mitre": [+ "T1580"+ ],+ "name": "One principal acting across an unusual number of AWS regions",+ "severity": "medium",+ "threshold": 6,+ "window_seconds": 900+ },+ {+ "category": "cloud",+ "correlates": [+ "azure-directory-role-assigned",+ "azure-resource-lock-deleted"+ ],+ "correlation_type": "temporal",+ "group_by": "actor",+ "id": "aisoc-corr-azure-role-grant-and-lock-removal",+ "mitre": [+ "T1098"+ ],+ "name": "One actor both assigned a directory role and removed a resource lock",+ "ordered": false,+ "sequence": [+ {+ "operationName_pattern_match_any": [+ "^Add\\ member\\ to\\ role$"+ ]+ },+ {+ "operationName_pattern_match_any": [+ "^Microsoft\\.Authorization/locks/delete$"+ ]+ }+ ],+ "severity": "medium",+ "window_seconds": 1800+ },+ {+ "category": "identity",+ "correlates": [+ "okta-mfa-push-sent",+ "okta-session-start"+ ],+ "correlation_type": "temporal_ordered",+ "group_by": "actor_email",+ "id": "aisoc-corr-okta-mfa-push-then-session",+ "mitre": [+ "T1621"+ ],+ "name": "Okta MFA push followed by a session for the same account",+ "ordered": true,+ "sequence": [+ {+ "eventType_pattern_match_any": [+ "^system\\.push\\.send_factor_verify_push$"+ ]+ },+ {+ "eventType_pattern_match_any": [+ "^user\\.session\\.start$"+ ]+ }+ ],+ "severity": "medium",+ "window_seconds": 900}]}diff --git a/services/fusion/app/services/derived_fields.py b/services/fusion/app/services/derived_fields.pyindex 25415f7b4..f742f4ddb 100644--- a/services/fusion/app/services/derived_fields.py+++ b/services/fusion/app/services/derived_fields.py@@ -50,7 +50,14 @@ DEFAULT_BUSINESS_END_HOUR = 18_TIME_FIELDS = ("event_time", "timestamp", "time", "@timestamp", "ingest_time")-def _parse_time(event: dict[str, Any]) -> datetime | None:+def parse_event_time(event: dict[str, Any]) -> datetime | None:+ """When the event happened, from the first timestamp field it carries.++ Public because the windowed engine orders sequences by event time and+ must read it the same way this module does. Two independent readers of+ "when did this happen" is how a rule comes to mean one thing to the+ stateless engine and another to the windowed one.+ """for field in _TIME_FIELDS:raw = event.get(field)if raw is None:@@ -119,7 +126,7 @@ def time_of_day_fields(Returns nothing when the event carries no parseable timestamp, for thesame reason as above: an unknown time is not "outside business hours"."""- when = _parse_time(event)+ when = parse_event_time(event)if when is None:return {}diff --git a/services/fusion/app/services/windowed_detection.py b/services/fusion/app/services/windowed_detection.pyindex e374ac9bd..b5dd8e19f 100644--- a/services/fusion/app/services/windowed_detection.py+++ b/services/fusion/app/services/windowed_detection.py@@ -29,7 +29,7 @@ from typing import Anyimport structlogfrom app.models.alert import AlertSeverity, RawAlert-from app.services.derived_fields import enrich, requested_derived_fields+from app.services.derived_fields import enrich, parse_event_time, requested_derived_fieldsfrom app.services.detection_engine import DetectionHitfrom app.services.detection_matcher import matchesfrom app.services.provenance import extract_provenance@@ -70,6 +70,59 @@ class WindowRule:distinct_by: str = ""+@dataclass(frozen=True)+class SequenceRule:+ """A then B (then C) by the same entity, ordered by event time.++ A threshold cannot express an ordering, and the ordering is usually the+ detection: "fifty failed logons" is an attempt, "failed logons then a+ success for the same account" is a compromise. Every sequence detection in+ the corpus had nowhere to go, because the counting engine sees each event+ alone and remembers only how many there were.++ The stages are ordered and disjoint in time: a stage advances only from an+ earlier event *by event time*, and one event advances at most one stage,+ so a rule whose stages overlap still needs as many events as it has+ stages.+ """++ id: str+ name: str+ severity: str+ category: str+ mitre: list[str]+ #: Ordered selectors, one per stage. Two or more.+ sequence: tuple[dict[str, Any], ...]+ #: Flat field naming the entity every stage must share.+ group_by: str+ #: The whole sequence must fit in this, measured from the first stage.+ #: Measured from the first and not from the previous stage, so a slow+ #: drip cannot walk a sequence forward indefinitely.+ window_seconds: int+ #: When False the stages may occur in any order inside the window, which+ #: is what Sigma's `temporal` correlation means as against+ #: `temporal_ordered`. Kept as a flag on one rule type rather than a+ #: second type, because every other property — entity, window, staging,+ #: one-event-one-stage — is identical and duplicating them is how the two+ #: drift apart.+ ordered: bool = True+++#: How far out of order an event may arrive and still be stitched into a+#: sequence. A connector polling a vendor delivers the vendor's order, not the+#: clock's, so arrival order is not evidence of event order — but holding+#: state forever to accommodate that would be a memory leak with a detection+#: attached. An event older than this relative to what the entity has already+#: shown is counted for its own stage and does not re-open an earlier one.+#:+#: The plan this implements says "with the existing watermark". There is no+#: watermark in this pipeline — the only ones in the tree belong to the+#: shadow-reconcile router and the dead-letter replay, which are unrelated+#: subsystems — so the bound is stated here rather than inherited from+#: something that does not exist.+SEQUENCE_REORDER_TOLERANCE_SECONDS = 300++# Built-in windowed rules. Intentionally small + high-signal; the corpus can grow# via the same JSON export path as the stateless engine later._BUILTIN_RULES: tuple[WindowRule, ...] = (@@ -148,6 +201,12 @@ def load_window_rules(path: Path | None = None) -> tuple[WindowRule, ...]:rule_id = str(entry.get("id") or "")if not rule_id or rule_id in seen:continue+ if entry.get("sequence"):+ # A sequence rule lives in the same artefact and is loaded by+ # `load_sequence_rules`. Skipped silently rather than warned+ # about: a warning per sequence rule on every boot would train+ # operators to ignore the log line that means something.+ continuetry:rule = WindowRule(id=rule_id,@@ -177,25 +236,161 @@ def load_window_rules(path: Path | None = None) -> tuple[WindowRule, ...]:return tuple(loaded)+def load_sequence_rules(path: Path | None = None) -> tuple[SequenceRule, ...]:+ """Ordered-sequence rules from the same artefact.++ Fail-soft the same way the counting loader is, and for the same reason: a+ malformed entry is skipped individually so one authoring mistake cannot+ empty the corpus. There are no built-in sequences, so a missing file+ yields none — which is the honest answer rather than a fallback.+ """+ target = path or _WINDOWED_RULESET_PATH+ if not target.exists():+ return ()+ try:+ payload = json.loads(target.read_text(encoding="utf-8"))+ except (ValueError, OSError) as exc:+ logger.warning("windowed_detection.sequence_ruleset_load_failed", path=str(target), error=str(exc))+ return ()++ loaded: list[SequenceRule] = []+ seen: set[str] = set()+ for entry in payload.get("rules") or []:+ if not isinstance(entry, dict) or not entry.get("sequence"):+ continue+ rule_id = str(entry.get("id") or "")+ if not rule_id or rule_id in seen:+ continue+ try:+ stages = tuple(dict(stage) for stage in entry["sequence"])+ rule = SequenceRule(+ id=rule_id,+ name=str(entry["name"]),+ severity=str(entry["severity"]),+ category=str(entry["category"]),+ mitre=[str(m).upper() for m in entry.get("mitre") or []],+ sequence=stages,+ group_by=str(entry["group_by"]),+ window_seconds=int(entry["window_seconds"]),+ ordered=bool(entry.get("ordered", True)),+ )+ except (KeyError, TypeError, ValueError) as exc:+ logger.warning("windowed_detection.sequence_skipped", rule_id=rule_id, error=str(exc))+ continue+ if len(rule.sequence) < 2:+ # One stage is a stateless rule paying for Redis state.+ logger.warning("windowed_detection.sequence_too_short", rule_id=rule_id, stages=len(rule.sequence))+ continue+ if any(not stage for stage in rule.sequence):+ # An empty stage matches every event, so the sequence collapses+ # to whichever stages remain — a different detection under this+ # one's name.+ logger.warning("windowed_detection.sequence_stage_empty", rule_id=rule_id)+ continue+ if rule.window_seconds < 1:+ logger.warning("windowed_detection.sequence_bounds_invalid", rule_id=rule_id)+ continue+ loaded.append(rule)+ seen.add(rule_id)++ if loaded:+ logger.info("windowed_detection.sequence_ruleset_loaded", count=len(loaded))+ return tuple(loaded)+++def _unordered_set_exists(observations: list[list[tuple[float, str]]], window_seconds: int) -> bool:+ """Is there one *distinct* event per stage inside one window, any order?++ The distinctness is the subtle half. Without it a single event matching+ every stage satisfies the rule on its own, which for an unordered+ correlation is not a near-miss but the common case: unordered stages are+ usually written as variations on one activity.+ """+ if not observations or any(not stage for stage in observations):+ return False+ merged = sorted({entry for stage in observations for entry in stage})+ for start in merged:+ window = [entry for entry in merged if start[0] <= entry[0] <= start[0] + window_seconds]+ # One event per stage, no event used twice: a bipartite matching,+ # small enough (stages are two or three) to settle by trying each+ # stage's candidates in turn.+ if _matching_exists([[e for e in stage if e in set(window)] for stage in observations], set()):+ return True+ return False+++def _matching_exists(candidates: list[list[tuple[float, str]]], used: set[tuple[float, str]]) -> bool:+ if not candidates:+ return True+ for entry in candidates[0]:+ if entry in used:+ continue+ if _matching_exists(candidates[1:], used | {entry}):+ return True+ return False+++def _chain_exists(observations: list[list[tuple[float, str]]], window_seconds: int) -> bool:+ """Is there one event per stage, strictly increasing, inside the window?++ Ordered by ``(event time, event token)``. The token is the tie-breaker and+ it is load-bearing rather than tidy: two distinct events routinely share a+ second, and one event that satisfies two stages must never complete a+ sequence by itself. Comparing times alone would allow both mistakes in+ opposite directions.++ Candidate starts are tried newest-first because the latest feasible start+ gives the tightest span, so the first chain found is the one most likely+ to fit the window.+ """+ if not observations or any(not stage for stage in observations):+ return False+ for start in reversed(observations[0]):+ cursor = start+ for stage in observations[1:]:+ nxt = next((entry for entry in stage if entry > cursor), None)+ if nxt is None:+ cursor = None # type: ignore[assignment]+ break+ cursor = nxt+ if cursor is not None and cursor[0] - start[0] <= window_seconds:+ return True+ return False++class WindowedDetectionEngine:"""Redis-backed sliding-window threshold detections."""- def __init__(self, redis: Any, rules: tuple[WindowRule, ...] | None = None, *, key_prefix: str = "aisoc:wd") -> None:+ def __init__(+ self,+ redis: Any,+ rules: tuple[WindowRule, ...] | None = None,+ *,+ sequences: tuple[SequenceRule, ...] | None = None,+ key_prefix: str = "aisoc:wd",+ ) -> None:self._redis = redis# None means "whatever is declared", so a deployment picks up exported# rules without a code change. An explicit tuple still wins, which is# what the tests rely on.self._rules = rules if rules is not None else load_window_rules()+ self._sequences = sequences if sequences is not None else load_sequence_rules()self._prefix = key_prefix# Same contract as the stateless engine: computed once at load,# because the set changes when rules change and not when traffic# arrives.- self._derived_wanted: set[str] = requested_derived_fields([{"match_when": rule.match_when} for rule in self._rules])+ clauses: list[dict[str, Any]] = [{"match_when": rule.match_when} for rule in self._rules]+ clauses += [{"match_when": stage} for rule in self._sequences for stage in rule.sequence]+ self._derived_wanted: set[str] = requested_derived_fields(clauses)@propertydef rule_count(self) -> int:return len(self._rules)+ @property+ def sequence_count(self) -> int:+ return len(self._sequences)+@staticmethoddef _fields(message: dict[str, Any]) -> dict[str, Any]:"""Flat field namespace, matching the stateless engine exactly.@@ -288,8 +483,93 @@ class WindowedDetectionEngine:)except Exception as exc: # noqa: BLE001 — one rule/Redis error must not wedge detectionlogger.debug("windowed_detection.rule_error", rule=rule.id, error=str(exc))++ if self._sequences:+ # One token per event, shared by every stage it matches, so two+ # stages satisfied by the *same* event can never be mistaken for+ # two events. Timestamps alone cannot carry that: two genuinely+ # distinct events routinely share a second.+ token = uuid.uuid4().hex+ for sequence in self._sequences:+ try:+ hit = await self._advance_sequence(sequence, tenant, fields, now, token)+ if hit is not None:+ hits.append(hit)+ except Exception as exc: # noqa: BLE001 — same fail-soft contract+ logger.debug("windowed_detection.sequence_error", rule=sequence.id, error=str(exc))return hits+ @staticmethod+ def _event_seconds(fields: dict[str, Any], now: float) -> float:+ """When the event happened, falling back to when we heard about it.++ Ordering a sequence by arrival would make it a different detection on+ every poll cadence, so event time wins wherever a source provides+ one. A source that stamps nothing must still be able to fire a+ sequence, which is what the fallback is for.+ """+ when = parse_event_time(fields)+ return when.timestamp() if when is not None else now++ async def _advance_sequence(+ self, rule: SequenceRule, tenant: str, fields: dict[str, Any], now: float, token: str+ ) -> DetectionHit | None:+ """Record what this event satisfies, then look for a complete chain.++ Recording first and searching second is what makes the ordering a+ property of **event time** rather than of arrival. The obvious+ implementation — a cursor that only moves forward as events arrive —+ is wrong for this platform: almost every connector here polls, so a+ batch arrives in the vendor's order and the event that starts a+ sequence routinely lands after the one that finishes it. That+ implementation would silently detect nothing on exactly the sources+ the corpus is thinnest on.+ """+ entity = fields.get(rule.group_by)+ if not entity:+ return None+ matched = [index for index, stage in enumerate(rule.sequence) if matches(stage, fields)]+ if not matched:+ return None++ when = self._event_seconds(fields, now)+ retain = rule.window_seconds + SEQUENCE_REORDER_TOLERANCE_SECONDS+ base = f"{self._prefix}:seq:{tenant}:{rule.id}:{entity}"++ for index in matched:+ key = f"{base}:{index}"+ await self._redis.zadd(key, {token: when})+ # Trimmed against this event's own time, not the wall clock: an+ # event that arrives late is old by construction, and trimming it+ # against now would discard the observation it just made.+ await self._redis.zremrangebyscore(key, 0, when - retain)+ await self._redis.expire(key, retain)++ observations: list[list[tuple[float, str]]] = []+ for index in range(len(rule.sequence)):+ entries = await self._redis.zrangebyscore(f"{base}:{index}", when - retain, when + retain, withscores=True)+ stage: list[tuple[float, str]] = []+ for member, score in entries or []:+ name = member.decode() if isinstance(member, bytes | bytearray) else str(member)+ stage.append((float(score), name))+ stage.sort()+ observations.append(stage)++ complete = (+ _chain_exists(observations, rule.window_seconds) if rule.ordered else _unordered_set_exists(observations, rule.window_seconds)+ )+ if not complete:+ return None+ if not await self._redis.set(f"{base}:fired", "1", nx=True, ex=rule.window_seconds):+ return None+ return DetectionHit(+ rule_id=rule.id,+ name=rule.name,+ severity=rule.severity,+ category=rule.category,+ mitre=list(rule.mitre),+ )+async def _observe_and_check(self,rule: WindowRule,diff --git a/services/fusion/tests/test_sigma_correlation_replay.py b/services/fusion/tests/test_sigma_correlation_replay.pynew file mode 100644index 000000000..25c5f7bcb--- /dev/null+++ b/services/fusion/tests/test_sigma_correlation_replay.py@@ -0,0 +1,183 @@+"""Every compiled Sigma correlation, replayed through the real engine.++`scripts/check_sigma_correlations.py` proves a correlation document compiles+and that the fields it names are ones something emits. Both are statements+about text. This drives the committed artefact through the real+`WindowedDetectionEngine` and asserts the behaviour each correlation type is+supposed to have:++* ``event_count`` fires at its threshold and not one short;+* ``value_count`` needs distinct values and is not satisfied by one repeated;+* ``temporal_ordered`` needs its stages in event-time order;+* ``temporal`` needs all its stages but accepts either order.++The fourth is the one worth having a test for: unordered and ordered differ+by a single flag, and a flag that is read nowhere looks identical to a flag+that works.+"""++from __future__ import annotations++import json+import re+from pathlib import Path+from typing import Any+from uuid import uuid4++import pytest+from app.services.windowed_detection import WindowedDetectionEngine, load_sequence_rules, load_window_rules++RULESET = Path(__file__).resolve().parents[1] / "app" / "data" / "windowed_ruleset.json"+++class _FakeRedis:+ def __init__(self) -> None:+ self.zsets: dict[str, dict[str, float]] = {}+ self.strings: dict[str, str] = {}++ async def zadd(self, key: str, mapping: dict[str, float]) -> None:+ self.zsets.setdefault(key, {}).update(mapping)++ async def zremrangebyscore(self, key: str, min_s: float, max_s: float) -> None:+ members = self.zsets.get(key, {})+ for member in [m for m, score in list(members.items()) if min_s <= score <= max_s]:+ del members[member]++ async def zcard(self, key: str) -> int:+ return len(self.zsets.get(key, {}))++ async def zrangebyscore(self, key: str, min_s: float, max_s: float, withscores: bool = False) -> list[Any]:+ items = sorted(((m, s) for m, s in self.zsets.get(key, {}).items() if min_s <= s <= max_s), key=lambda kv: (kv[1], kv[0]))+ return items if withscores else [m for m, _ in items]++ async def expire(self, key: str, ttl: int) -> None:+ return None++ async def set(self, key: str, value: str, nx: bool = False, ex: int | None = None) -> Any:+ if nx and key in self.strings:+ return None+ self.strings[key] = value+ return True+++def _correlations() -> list[dict[str, Any]]:+ rules = [r for r in json.loads(RULESET.read_text(encoding="utf-8"))["rules"] if r.get("correlation_type")]+ assert rules, "no compiled Sigma correlations in the committed ruleset — the replay corpus is empty"+ return rules+++CORRELATIONS = _correlations()+BY_TYPE = {r["correlation_type"]: r for r in CORRELATIONS}+++def _satisfying(key: str, value: Any) -> tuple[str, Any]:+ """A field and a value that satisfies one compiled clause.++ The Sigma compiler emits anchored patterns rather than equalities, even+ for a plain string, so a literal recovered from `^iam\\.amazonaws\\.com$`+ is what an event has to carry. Recovering it here rather than hand-writing+ the value keeps the replay honest: the event is built from the clause the+ engine will evaluate, not from the YAML a human wrote.+ """+ if key.endswith("_pattern_match_any"):+ pattern = value[0] if isinstance(value, list) else value+ return key[: -len("_pattern_match_any")], re.sub(r"\\(.)", r"\1", str(pattern).strip("^$"))+ if key.endswith("_in"):+ return key[: -len("_in")], value[0] if isinstance(value, list) and value else value+ return key, value+++def _event(rule: dict[str, Any], tenant: str, clause: dict[str, Any], entity: str, when: float, distinct: str = "") -> dict[str, Any]:+ raw = dict(_satisfying(key, value) for key, value in clause.items())+ raw[rule["group_by"]] = entity+ raw["event_time"] = when+ if rule.get("distinct_by"):+ raw[rule["distinct_by"]] = distinct+ return {"tenant_id": tenant, "ocsf_event": {"tenant_uid": tenant, "raw_data": json.dumps(raw)}}+++def _engine_for(rule: dict[str, Any]) -> WindowedDetectionEngine:+ counting = tuple(r for r in load_window_rules() if r.id == rule["id"])+ staged = tuple(r for r in load_sequence_rules() if r.id == rule["id"])+ assert counting or staged, f"{rule['id']} is in the committed JSON but neither loader accepted it"+ return WindowedDetectionEngine(_FakeRedis(), rules=counting, sequences=staged)+++def test_all_four_correlation_types_are_covered() -> None:+ """A replay suite that silently covers three of four proves three of four."""+ assert set(BY_TYPE) == {"event_count", "value_count", "temporal", "temporal_ordered"}, f"covered: {sorted(BY_TYPE)}"+++async def test_event_count_fires_at_its_threshold_and_not_one_short() -> None:+ rule = BY_TYPE["event_count"]+ engine = _engine_for(rule)+ tenant = str(uuid4())+ for index in range(rule["threshold"] - 1):+ assert not await engine.evaluate(_event(rule, tenant, rule["match_when"], "arn:aws:iam::1:user/a", 1000.0 + index))+ hits = await engine.evaluate(_event(rule, tenant, rule["match_when"], "arn:aws:iam::1:user/a", 2000.0))+ assert [h.rule_id for h in hits] == [rule["id"]]+++async def test_value_count_needs_distinct_values() -> None:+ rule = BY_TYPE["value_count"]+ tenant = str(uuid4())++ repeated = _engine_for(rule)+ for index in range(rule["threshold"] * 2):+ assert not await repeated.evaluate(+ _event(rule, tenant, rule["match_when"], "arn:aws:iam::1:user/a", 1000.0 + index, distinct="one-value")+ )++ distinct = _engine_for(rule)+ hits: list[Any] = []+ for index in range(rule["threshold"]):+ hits = await distinct.evaluate(+ _event(rule, tenant, rule["match_when"], "arn:aws:iam::1:user/a", 1000.0 + index, distinct=f"value-{index}")+ )+ assert [h.rule_id for h in hits] == [rule["id"]]+++async def test_temporal_ordered_requires_the_order() -> None:+ rule = BY_TYPE["temporal_ordered"]+ first, second = rule["sequence"][0], rule["sequence"][1]+ tenant = str(uuid4())++ forwards = _engine_for(rule)+ assert not await forwards.evaluate(_event(rule, tenant, first, "[email protected]", 1000.0))+ assert [h.rule_id for h in await forwards.evaluate(_event(rule, tenant, second, "[email protected]", 1100.0))] == [rule["id"]]++ backwards = _engine_for(rule)+ assert not await backwards.evaluate(_event(rule, tenant, second, "[email protected]", 1000.0))+ assert not await backwards.evaluate(_event(rule, tenant, first, "[email protected]", 1100.0))+++async def test_temporal_accepts_either_order_but_needs_both_stages() -> None:+ rule = BY_TYPE["temporal"]+ first, second = rule["sequence"][0], rule["sequence"][1]+ tenant = str(uuid4())++ backwards = _engine_for(rule)+ assert not await backwards.evaluate(_event(rule, tenant, second, "[email protected]", 1000.0))+ assert [h.rule_id for h in await backwards.evaluate(_event(rule, tenant, first, "[email protected]", 1100.0))] == [rule["id"]]++ one_stage_only = _engine_for(rule)+ for index in range(6):+ assert not await one_stage_only.evaluate(_event(rule, tenant, first, "[email protected]", 1000.0 + index))+++async def test_a_correlation_does_not_cross_entities() -> None:+ for rule in CORRELATIONS:+ engine = _engine_for(rule)+ tenant = str(uuid4())+ clauses = list(rule.get("sequence") or [rule["match_when"]])+ needed = rule.get("threshold", len(clauses))+ for index in range(needed * 2):+ clause = clauses[index % len(clauses)]+ entity = f"entity-{index % 7}"+ hits = await engine.evaluate(_event(rule, tenant, clause, entity, 1000.0 + index, distinct=f"value-{index}"))+ assert not hits, f"{rule['id']} fired across {7} different entities"diff --git a/services/fusion/tests/test_windowed_sequences.py b/services/fusion/tests/test_windowed_sequences.pynew file mode 100644index 000000000..c3f7247b5--- /dev/null+++ b/services/fusion/tests/test_windowed_sequences.py@@ -0,0 +1,231 @@+"""Ordered sequences: A then B, by the same entity, inside a window.++A threshold rule cannot express an ordering. "Fifty failed logons" and "a+failed logon followed by a successful one from a new address" are different+detections, and only the second says *compromise* rather than *attempt*. The+counting engine sees each event in isolation and remembers only how many+there were, so every sequence detection in the corpus had nowhere to go.++What ordering means here, precisely, because the loose version is useless:++* stages advance **one event at a time**, so a single event matching two+ stages cannot complete a sequence by itself;+* a stage advances only from an earlier event **by event time**, not by+ arrival order, because a connector polling a vendor delivers a batch whose+ internal order is the vendor's and not the clock's;+* the whole sequence must fit inside one window, measured from the first+ stage, not from the previous one, so a slow drip cannot walk a sequence+ forward indefinitely;+* an out-of-order arrival is tolerated up to a stated bound and is not+ silently treated as in-order.+"""++from __future__ import annotations++import json+from typing import Any+from uuid import uuid4++import pytest+from app.services.windowed_detection import SequenceRule, WindowedDetectionEngine+++class _FakeRedis:+ """The sorted-set, hash and marker calls the engine makes, faithfully."""++ def __init__(self) -> None:+ self.zsets: dict[str, dict[str, float]] = {}++ self.strings: dict[str, str] = {}++ async def zadd(self, key: str, mapping: dict[str, float]) -> None:+ self.zsets.setdefault(key, {}).update(mapping)++ async def zremrangebyscore(self, key: str, min_s: float, max_s: float) -> None:+ members = self.zsets.get(key, {})+ for member in [m for m, score in list(members.items()) if min_s <= score <= max_s]:+ del members[member]++ async def zcard(self, key: str) -> int:+ return len(self.zsets.get(key, {}))++ async def zrangebyscore(self, key: str, min_s: float, max_s: float, withscores: bool = False) -> list[Any]:+ items = sorted(((m, s) for m, s in self.zsets.get(key, {}).items() if min_s <= s <= max_s), key=lambda kv: (kv[1], kv[0]))+ return items if withscores else [m for m, _ in items]++ async def expire(self, key: str, ttl: int) -> None:+ return None++ async def set(self, key: str, value: str, nx: bool = False, ex: int | None = None) -> Any:+ if nx and key in self.strings:+ return None+ self.strings[key] = value+ return True+++SPRAY_THEN_SUCCESS = SequenceRule(+ id="wd-seq-test-spray-then-success",+ name="Failed logons then a success for the same account",+ severity="high",+ category="identity",+ mitre=["T1110"],+ sequence=(+ {"event_type": "authentication", "outcome": "failure"},+ {"event_type": "authentication", "outcome": "success"},+ ),+ group_by="user",+ window_seconds=600,+)+++def _event(tenant: str, outcome: str, when: float, user: str = "alice") -> dict[str, Any]:+ return {+ "tenant_id": tenant,+ "ocsf_event": {+ "tenant_uid": tenant,+ "raw_data": json.dumps({"event_type": "authentication", "outcome": outcome, "user": user, "event_time": when}),+ },+ }+++def _engine() -> WindowedDetectionEngine:+ return WindowedDetectionEngine(_FakeRedis(), rules=(), sequences=(SPRAY_THEN_SUCCESS,))+++async def test_the_sequence_fires_in_order() -> None:+ engine = _engine()+ tenant = str(uuid4())+ assert not await engine.evaluate(_event(tenant, "failure", 1000.0))+ hits = await engine.evaluate(_event(tenant, "success", 1060.0))+ assert [h.rule_id for h in hits] == [SPRAY_THEN_SUCCESS.id]+++async def test_the_last_stage_alone_does_not_fire() -> None:+ """Otherwise the sequence is just its final stage with extra machinery."""+ engine = _engine()+ tenant = str(uuid4())+ for offset in range(5):+ assert not await engine.evaluate(_event(tenant, "success", 1000.0 + offset))+++async def test_out_of_order_by_event_time_does_not_fire() -> None:+ """The success happened *before* the failure, so nothing was compromised."""+ engine = _engine()+ tenant = str(uuid4())+ assert not await engine.evaluate(_event(tenant, "failure", 2000.0))+ assert not await engine.evaluate(_event(tenant, "success", 1000.0))+++async def test_a_batch_delivered_out_of_arrival_order_still_fires() -> None:+ """A connector polling a vendor delivers the vendor's order, not the clock's.++ The success arrives first and is held; the failure that precedes it by+ event time arrives second. Replaying the later event is what makes this+ work, and without it every vendor that batches would be undetectable.+ """+ engine = _engine()+ tenant = str(uuid4())+ assert not await engine.evaluate(_event(tenant, "success", 1060.0))+ hits = await engine.evaluate(_event(tenant, "failure", 1000.0))+ assert [h.rule_id for h in hits] == [SPRAY_THEN_SUCCESS.id]+++async def test_the_stages_must_fit_inside_one_window() -> None:+ engine = _engine()+ tenant = str(uuid4())+ assert not await engine.evaluate(_event(tenant, "failure", 1000.0))+ assert not await engine.evaluate(_event(tenant, "success", 1000.0 + SPRAY_THEN_SUCCESS.window_seconds + 1))+++async def test_the_entity_must_be_the_same() -> None:+ engine = _engine()+ tenant = str(uuid4())+ assert not await engine.evaluate(_event(tenant, "failure", 1000.0, user="alice"))+ assert not await engine.evaluate(_event(tenant, "success", 1060.0, user="bob"))+++async def test_one_event_cannot_advance_two_stages() -> None:+ """A rule whose stages overlap must still need two events."""+ overlapping = SequenceRule(+ id="wd-seq-test-overlap",+ name="overlap",+ severity="low",+ category="identity",+ mitre=[],+ sequence=({"event_type": "authentication"}, {"event_type": "authentication"}),+ group_by="user",+ window_seconds=600,+ )+ engine = WindowedDetectionEngine(_FakeRedis(), rules=(), sequences=(overlapping,))+ tenant = str(uuid4())+ assert not await engine.evaluate(_event(tenant, "failure", 1000.0))+ hits = await engine.evaluate(_event(tenant, "failure", 1001.0))+ assert [h.rule_id for h in hits] == [overlapping.id]+++async def test_it_fires_once_per_window() -> None:+ engine = _engine()+ tenant = str(uuid4())+ await engine.evaluate(_event(tenant, "failure", 1000.0))+ assert await engine.evaluate(_event(tenant, "success", 1060.0))+ assert not await engine.evaluate(_event(tenant, "success", 1120.0))+++async def test_a_three_stage_sequence_needs_all_three_in_order() -> None:+ three = SequenceRule(+ id="wd-seq-test-three",+ name="three",+ severity="high",+ category="identity",+ mitre=[],+ sequence=(+ {"event_type": "authentication", "outcome": "failure"},+ {"event_type": "authentication", "outcome": "success"},+ {"event_type": "authentication", "outcome": "mfa_reset"},+ ),+ group_by="user",+ window_seconds=600,+ )+ engine = WindowedDetectionEngine(_FakeRedis(), rules=(), sequences=(three,))+ tenant = str(uuid4())+ assert not await engine.evaluate(_event(tenant, "failure", 1000.0))+ assert not await engine.evaluate(_event(tenant, "mfa_reset", 1010.0))+ assert not await engine.evaluate(_event(tenant, "success", 1020.0))+ hits = await engine.evaluate(_event(tenant, "mfa_reset", 1030.0))+ assert [h.rule_id for h in hits] == [three.id]+++async def test_tenants_do_not_share_sequence_progress() -> None:+ engine = _engine()+ one, two = str(uuid4()), str(uuid4())+ assert not await engine.evaluate(_event(one, "failure", 1000.0))+ assert not await engine.evaluate(_event(two, "success", 1060.0))+++async def test_an_event_with_no_time_falls_back_to_arrival() -> None:+ """A source that stamps nothing must still be able to fire a sequence."""+ engine = _engine()+ tenant = str(uuid4())++ def untimed(outcome: str) -> dict[str, Any]:+ return {+ "tenant_id": tenant,+ "ocsf_event": {+ "tenant_uid": tenant,+ "raw_data": json.dumps({"event_type": "authentication", "outcome": outcome, "user": "alice"}),+ },+ }++ assert not await engine.evaluate(untimed("failure"))+ assert [h.rule_id for h in await engine.evaluate(untimed("success"))] == [SPRAY_THEN_SUCCESS.id]