Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
#!/usr/bin/env python3
"""The `det-*` rules that need a window, and what became of each one.
Why this file exists
--------------------
``scripts/check_detection_fields.py`` attributes 74 engine-loaded rules to a
"windowed evaluator" family. The windowed engine has existed since Wave 2 and
the exporter beside this file has shipped rules into it, but the two corpora
were disjoint: not one ``wd-*`` id corresponded to a ``det-*`` rule, so
building the engine moved none of these rules and the unreachable count never
changed. Each of the 74 still names a counter no connector emits
(``fail_count``, ``events_per_minute``, ``distinct_secrets_per_minute``),
reads ``None`` on its first clause, and can never fire — while being loaded by
the engine and counted toward the published executable total.
This module records one decision per rule, and the decision is the unit of
work: either the rule is translated into windowed form, or it is refused with
a reason. Nothing may simply disappear. ``scripts/check_windowed_translation.py``
enforces that, and ``scripts/export_detection_ruleset.py`` reads it so a
decided rule leaves the stateless corpus it could never fire in.
How a translation is derived
----------------------------
**From the rule's own clauses, never invented.** A rule like
{"event_type": "user.session.start", "outcome": "FAILURE",
"fail_count_gt": 20, "time_window_minutes_lt": 10}
already carries everything a windowed rule needs: the selector is what is left
after the counting and window clauses are removed, the threshold is the ``_gt``
bound plus one, and the window is the time clause. :func:`derive` computes all
three, so a translated rule cannot drift from the number its author chose, and
``check_windowed_translation.py`` re-derives them to prove it.
The one thing the clauses do not always carry is the entity the count
accumulates against. ``request_count_per_src`` names it; ``fail_count`` does
not. Where the name carries no entity the table below supplies one, and the
choice is a field the rule's own source emits rather than a plausible-looking
name — a ``group_by`` nothing emits is the same defect in a new place, because
``fields.get(group_by)`` returns ``None`` and the engine skips the event.
Why some rules are refused
--------------------------
The windowed engine is deliberately narrow: match, group by one entity, count
events or distinct values, threshold, window. It computes no sums, no means,
no variances, and it fires when a count is **high**. A rule that needs any of
those cannot be expressed here, and half-translating it — keeping the count
and dropping the mean — ships something that detects a different thing under
the original's name and severity. Those are refused with the reason stated, and
the refusal travels into the rule's YAML and its marketplace entry so a reader
of the catalogue sees why the rule does not run.
Refusal is not deletion. A refused rule keeps its id, its YAML and its place
in the corpus; it stops being counted as executable, which it never was.
"""
from __future__ import annotations
import re
import sys
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
sys.path.insert(0, str(Path(__file__).resolve().parent))
__all__ = [
"AGGREGATE",
"DECISIONS",
"MULTI_DISTINCT",
"NOT_WINDOWED",
"OPERAND_NOT_EMITTED",
"RARITY",
"REFUSAL_KINDS",
"Covered",
"Refusal",
"ShapeError",
"Translation",
"WindowShape",
"decided_keys",
"derive",
"parse_shape",
"retirement_reason",
"translated_rules",
]
# --------------------------------------------------------------------------
# Decisions
# --------------------------------------------------------------------------
@dataclass(frozen=True)
class Translation:
"""This rule becomes a `wd-*` rule.
``group_by`` and ``distinct_by`` are the only parts a human chooses; the
threshold, the window and the residual selector are derived from the
``det-*`` spec by :func:`derive`.
"""
group_by: str
distinct_by: str = ""
#: Set only where the default `wd-<slug>` id would collide with a rule the
#: windowed corpus already ships.
wd_id: str = ""
#: Why this entity field, when the counter's name does not name one. Also
#: the place to record which connector key it comes from.
entity_note: str = ""
@dataclass(frozen=True)
class Refusal:
"""This rule cannot be expressed as a count over a window.
``kind`` groups the reasons so the gate can report them as families rather
than as 19 unrelated sentences, and so a later phase can find the ones it
unblocks.
"""
kind: str
reason: str
@dataclass(frozen=True)
class Covered:
"""A windowed rule the corpus already ships detects this.
Distinct from a refusal: the detection exists and runs, so retiring the
`det-*` loses nothing. The link is what stops it being re-authored.
"""
by: str
reason: str
#: Refusal kinds. Named so the gate's summary and a later phase's search agree.
AGGREGATE = "aggregate" # needs a sum, mean or variance over the window
NOT_WINDOWED = "not-windowed" # the field is a per-event property, not a counter
RARITY = "rarity" # fires when a count is LOW; needs the baseline/first-seen store
MULTI_DISTINCT = "multi-distinct" # two independent distinct counts in one rule
OPERAND_NOT_EMITTED = "operand-not-emitted" # nothing surfaces the thing to count distinctly
REFUSAL_KINDS = frozenset({AGGREGATE, NOT_WINDOWED, RARITY, MULTI_DISTINCT, OPERAND_NOT_EMITTED})
#: One entry per rule the reachability gate attributes to the windowed family,
#: keyed ``<category>/<slug>`` the way `detections/rule-ids.lock.json` is.
#:
#: Ordered by category then slug so a diff reads as content rather than as
#: churn.
DECISIONS: dict[str, Translation | Refusal | Covered] = {
# ---------------------------------------------------------------- application
"application/rate-limit-burst": Translation(group_by="src_ip"),
"application/api-mass-enumeration-ids": Refusal(
OPERAND_NOT_EMITTED,
"counts distinct object ids per source, and no WAF or application source "
"in this tree surfaces the requested object id as a field — the engine "
"would count nothing. Revisit when the activity projection lands a "
"`resource` field (depth plan 2.2).",
),
"application/saas-mass-share-public": Translation(
group_by="actor_email",
entity_note="google_workspace.normalize emits actor_email; `user` is not in its output",
),
"application/app-account-enum-reset": Translation(group_by="src_ip"),
"application/app-docker-insecure-registry-flag": Refusal(
NOT_WINDOWED,
"`insecure_registries_count` is the length of a list inside one daemon "
"config event, not a count over a window. The reachability gate reads it "
"as windowed because the name ends in `_count`.",
),
"application/app-salesforce-mass-export": Refusal(
NOT_WINDOWED,
"`row_count` is a per-event field of one export record, not a count over a window. The Salesforce connector does not surface it.",
),
"application/app-box-public-link-mass-create": Translation(
group_by="actor_email",
entity_note="box.normalize emits actor_email",
),
"application/app-servicenow-record-export-bulk": Refusal(
NOT_WINDOWED,
"`row_count` is a per-event field of one export record, not a count over a window. The ServiceNow connector does not surface it.",
),
# ---------------------------------------------------------------------- cloud
"cloud/aws-iam-enumeration": Translation(
group_by="user_arn",
distinct_by="event_name",
entity_note="aws_cloudtrail.normalize emits both user_arn and event_name",
),
"cloud/aws-s3-mass-getobject": Translation(group_by="user_arn"),
"cloud/azure-resource-mass-delete": Translation(
group_by="actor",
entity_note="azure_activity.normalize emits actor; `principal` is not in its output",
),
"cloud/gcp-secret-manager-mass-export": Translation(
group_by="actor",
entity_note="gcp_cloud_audit.normalize emits actor",
),
"cloud/aws-iam-create-user-burst": Translation(group_by="user_arn"),
"cloud/aws-iam-account-summary-recon": Translation(group_by="user_arn"),
"cloud/aws-iam-list-keys-recon": Refusal(
OPERAND_NOT_EMITTED,
"counts distinct *target* users of ListAccessKeys. CloudTrail carries "
"that in `requestParameters.UserName`, and the matcher has no dotted-path "
"traversal, so the connector's `user_name` is the caller rather than the "
"target. Counting the caller would be a different detection.",
),
"cloud/aws-s3-mass-deleteobject": Translation(group_by="user_arn"),
"cloud/aws-secretsmanager-mass-getsecret": Refusal(
OPERAND_NOT_EMITTED,
"counts distinct secrets, which CloudTrail carries in "
"`requestParameters.SecretId`; the connector surfaces no secret "
"identifier, so the engine has nothing to be distinct about.",
),
"cloud/azure-ad-bulk-user-creation": Translation(group_by="actor"),
"cloud/azure-ad-bulk-license-assignment": Translation(group_by="actor"),
"cloud/azure-arm-locks-removed-bulk": Translation(group_by="actor"),
"cloud/azure-arm-mass-resource-modify": Translation(group_by="actor"),
"cloud/azure-keyvault-secret-mass-get": Translation(
group_by="actor",
distinct_by="resource",
entity_note="azure_activity.normalize emits `resource`, which for a Key Vault operation is the secret",
),
"cloud/gcp-iam-service-account-key-mass-create": Translation(group_by="actor"),
"cloud/gcp-iam-impersonation-burst": Refusal(
OPERAND_NOT_EMITTED,
"counts distinct impersonation *targets*. The GCP audit connector emits "
"the caller (`actor`) and no target service account, so nothing "
"identifies what is being impersonated.",
),
"cloud/gcp-gcs-mass-delete": Translation(group_by="actor"),
"cloud/gcp-bq-mass-export": Translation(group_by="actor"),
"cloud/k8s-secret-mass-get": Translation(
group_by="k8s_user",
distinct_by="k8s_object_name",
entity_note="kubernetes_audit.normalize emits both; k8s_object_name is the secret's name",
),
"cloud/cloud-multi-cross-cloud-mass-egress": Refusal(
AGGREGATE,
"`bytes_out_per_minute` is a sum of a numeric field over the window. The "
"engine counts occurrences and distinct values; it adds nothing up, so "
"a gigabyte in one transfer and a gigabyte in a thousand are the same "
"count to it.",
),
# ----------------------------------------------------------------- data-exfil
"data-exfil/large-egress-personal-cloud": Refusal(
NOT_WINDOWED,
"carries a `time_window_minutes_lt` clause and no counter at all, so "
"there is nothing to accumulate. `bytes_out` is a per-event proxy field; "
"the window clause is vestigial and is what makes the rule unreachable. "
"Deleting that one clause is a content decision, not a translation.",
),
"data-exfil/egress-spike-baseline": Refusal(
AGGREGATE,
"needs a sum of `bytes_out` per source and a z-score against a learned "
"baseline. Both are out of scope for a counting engine (the baseline "
"arrives with depth plan 3.4).",
),
"data-exfil/salesforce-bulk-export": Refusal(
NOT_WINDOWED,
"`row_count` is a per-event field of one export record, not a count over a window. The Salesforce connector does not surface it.",
),
"data-exfil/db-mass-row-read": Refusal(
AGGREGATE,
"needs a sum of rows read per session and a z-score against a learned baseline; the engine computes neither.",
),
"data-exfil/printer-mass-print-confidential": Refusal(
AGGREGATE,
"`page_count_per_user` sums pages across print jobs. Counting jobs "
"instead would fire on 500 one-page prints and miss one 1,500-page "
"print, which inverts what the rule is for.",
),
"data-exfil/container-mass-image-export": Translation(
group_by="actor",
entity_note="the registry-push sources in this tree normalize the pusher to actor",
),
"data-exfil/secrets-mass-fetch": Translation(
group_by="actor",
entity_note="vendor-neutral rule; `actor` is the field every cloud connector here emits",
),
# ------------------------------------------------------------------- endpoint
"endpoint/ransomware-file-extension-change": Translation(
group_by="process_name",
entity_note="EDR connectors normalize the acting binary to process_name",
),
# ------------------------------------------------------------------- identity
"identity/brute-force-login": Translation(
group_by="actor_email",
entity_note="okta.normalize emits actor_email as the account; `user` is not in its output",
),
"identity/password-spray": Translation(
group_by="src_ip",
distinct_by="actor_email",
wd_id="wd-okta-password-spray",
entity_note=("id overridden because `wd-password-spray` already ships and counts events rather than distinct accounts"),
),
"identity/mfa-fatigue": Translation(
group_by="actor_email",
wd_id="wd-okta-mfa-push-bombing",
entity_note="id overridden because `wd-mfa-fatigue` already ships against a different source shape",
),
"identity/shared-account-login-spike": Translation(group_by="actor_email", distinct_by="src_ip"),
"identity/kerberoasting": Refusal(
OPERAND_NOT_EMITTED,
"counts distinct service principal names requested per account. Windows "
"4769 carries the SPN in `ServiceName`, which no source in this tree "
"lifts, so there is nothing to be distinct about.",
),
"identity/ident-password-spray-many-targets": Refusal(
AGGREGATE,
"pairs a distinct-user count with a `tries_per_user` ceiling, which is a "
"mean inside the window. Keeping only the distinct half would widen the "
"rule into a brute-force detection `wd-bruteforce-auth` already covers.",
),
"identity/ident-password-spray-distributed": Translation(group_by="user", distinct_by="src_ip"),
"identity/ident-credstuffing-known-breached-list": Translation(group_by="src_ip"),
"identity/ident-mfa-fatigue-burst": Translation(group_by="user"),
"identity/ident-account-locked-many-times": Translation(group_by="user"),
"identity/ident-totp-skewed-clock": Translation(group_by="user"),
"identity/ident-priv-role-burst-granted": Translation(group_by="actor"),
"identity/ident-priv-role-removed-bulk": Translation(group_by="actor"),
"identity/ident-orphan-svc-acct-many-keys": Refusal(
NOT_WINDOWED,
"`active_keys_per_account` is the state of an account at one moment, "
"reported by an inventory event. A window counts what happened, not what "
"is true; this needs the posture snapshot (depth plan 2.5).",
),
"identity/ident-ad-machine-account-quota-abuse": Translation(group_by="user"),
"identity/ident-kerberos-failure-spike-from-host": Translation(group_by="hostname"),
"identity/ident-okta-admin-app-assigned-many-users": Translation(group_by="actor"),
"identity/ident-okta-mass-deactivate-users": Translation(group_by="actor"),
"identity/ident-ping-admin-role-burst": Translation(group_by="actor"),
"identity/ident-oidc-many-clients-from-single-actor": Translation(group_by="actor"),
"identity/ident-github-org-collaborator-removed-bulk": Translation(group_by="actor"),
# `repo` is the key the audit-log entry actually carries (the connector's
# own tests and GitHub's REST reference use it); `repository` was a
# guessed name, and a distinct-by over a missing field never counts
# (issue #1545).
"identity/ident-github-deploy-key-added-org-wide": Translation(group_by="actor", distinct_by="repo"),
# -------------------------------------------------------------------- network
"network/c2-beacon-high-frequency": Refusal(
AGGREGATE,
"`interval_consistency` is the regularity of inter-arrival times across "
"the window — a variance, not a count. Keeping only the connection count "
"turns a beacon detection into a chatty-host detection.",
),
"network/dns-data-exfiltration": Refusal(
AGGREGATE,
"`avg_subdomain_len` is a mean over the window. Dropping it leaves "
"'100 DNS queries in 30 minutes', which every workstation clears.",
),
"network/port-scan-internal": Refusal(
MULTI_DISTINCT,
"requires distinct destination ports *and* distinct destinations per "
"source. The engine holds one distinct operand per rule; splitting this "
"into two rules would fire twice on one scan and once on neither half.",
),
"network/ssh-failed-from-internet": Translation(group_by="src_ip"),
"network/smb-lateral-movement": Translation(group_by="src", distinct_by="dst_ip"),
"network/dns-fast-flux": Refusal(
OPERAND_NOT_EMITTED,
"counts distinct A records, which live as a list inside a single DNS "
"response rather than one per event. The engine counts distinct values "
"of one field per event, so it cannot see inside the answer set.",
),
"network/icmp-tunnel": Refusal(
AGGREGATE,
"`avg_payload_bytes` is a mean over the window, and `packet_count` is a per-flow sum. Counting flow records measures neither.",
),
"network/egress-to-rare-asn": Refusal(
RARITY,
"fires when a destination ASN has been seen *fewer* than five times in "
"thirty days. The engine fires when a count is high; a rarity signal "
"needs the first-seen store (depth plan 3.3) and the baseline module "
"(3.4).",
),
"network/ldap-bind-anomaly": Translation(group_by="src_ip"),
"network/ddos-syn-flood": Translation(group_by="dst_ip"),
"network/internal-host-spamming-mail": Translation(group_by="src_ip"),
"network/net-dns-nxdomain-spike": Translation(group_by="src_ip"),
"network/net-dns-any-query-spike": Translation(group_by="src_ip"),
"network/net-dns-tunnel-pattern-len": Refusal(
NOT_WINDOWED,
"`subdomain_hex_ratio` is a property of one query name, not a count over "
"a window; the reachability gate reads it as windowed because the name "
"ends in `_ratio`. It is cheap to derive from the query name and belongs "
"with the derived fields, not here.",
),
"network/net-dns-fast-flux": Refusal(
NOT_WINDOWED,
"`answer_count` is the number of records in one DNS response, a per-event field, not a count over a window.",
),
"network/net-dhcp-starvation": Translation(
group_by="src_ip",
entity_note="the relay is identified by the source address on the DHCP event",
),
}
# --------------------------------------------------------------------------
# Derivation
# --------------------------------------------------------------------------
#: Clause fields that declare the window rather than the count.
_TIME_UNITS: dict[str, int] = {
"time_window_minutes": 60,
"time_window_hours": 3600,
"time_window_days": 86400,
}
#: Suffixes on the counter's own name that imply a window when no time clause
#: is present. `events_per_minute_gt: 10` is a rate, and the rate names its
#: own denominator.
_NAME_WINDOWS: tuple[tuple[str, int], ...] = (
("_per_second", 1),
("_per_minute", 60),
("_per_hour", 3600),
("_per_day", 86400),
)
#: Operator suffixes a counting clause may carry, longest first.
_BOUNDS: tuple[str, ...] = ("gte", "gt", "lte", "lt")
#: Matches the counter half of a clause, mirroring the "windowed evaluator"
#: family in `scripts/check_detection_fields.py`. Kept as its own copy rather
#: than imported because that module is a gate with a ratchet and importing it
#: here would make the exporter depend on a CI threshold.
_COUNTER = re.compile(r"_count$|^count_|_per_|time_window|_window_|_5min|_ratio$")
class ShapeError(ValueError):
"""The spec does not have the shape a windowed rule is derived from."""
@dataclass(frozen=True)
class WindowShape:
"""The windowed half of a `det-*` rule, recovered from its clauses."""
threshold: int
window_seconds: int
residual: dict[str, Any]
#: Clause keys consumed to produce the above, so the gate can prove that
#: everything else survived into the translated rule untouched.
consumed: tuple[str, ...] = field(default_factory=tuple)
def _split_bound(key: str) -> tuple[str, str]:
for bound in _BOUNDS:
if key.endswith("_" + bound):
return key[: -len(bound) - 1], bound
return key, ""
def parse_shape(match_when: dict[str, Any]) -> WindowShape:
"""Recover threshold, window and residual selector from a `det-*` clause set.
Raises :class:`ShapeError` when the rule carries no counter, two counters,
or a counter with no upper bound — each of which means the decision table
should hold a :class:`Refusal` rather than a :class:`Translation`.
"""
counters: list[tuple[str, str, Any]] = []
times: list[tuple[str, str, Any]] = []
for key, value in match_when.items():
base, bound = _split_bound(key)
if base in _TIME_UNITS:
times.append((key, base, value))
elif _COUNTER.search(base):
counters.append((key, bound, value))
if not counters:
raise ShapeError("no counting clause")
if len(counters) > 1:
raise ShapeError(f"{len(counters)} counting clauses; a windowed rule holds one")
if len(times) > 1:
raise ShapeError(f"{len(times)} window clauses; a windowed rule holds one")
counter_key, bound, raw_bound = counters[0]
if bound == "gt":
threshold = int(raw_bound) + 1
elif bound == "gte":
threshold = int(raw_bound)
else:
raise ShapeError(f"counter {counter_key!r} has no lower bound — it fires when the count is low")
if times:
time_key, unit, raw_window = times[0]
window_seconds = int(float(raw_window) * _TIME_UNITS[unit])
else:
time_key = ""
window_seconds = 0
base, _ = _split_bound(counter_key)
for suffix, seconds in _NAME_WINDOWS:
if suffix in base:
window_seconds = seconds
break
if "_5min" in base:
window_seconds = 300
if not window_seconds:
raise ShapeError(f"counter {counter_key!r} declares no window and carries no rate suffix")
consumed = tuple(k for k in (counter_key, time_key) if k)
residual = {k: v for k, v in match_when.items() if k not in consumed}
if threshold < 1 or window_seconds < 1:
raise ShapeError(f"derived threshold {threshold} / window {window_seconds}s is not a window")
return WindowShape(threshold=threshold, window_seconds=window_seconds, residual=residual, consumed=consumed)
def derive(category: str, spec: dict[str, Any], decision: Translation) -> dict[str, Any]:
"""Build the `wd-*` rule for one `det-*` spec."""
shape = parse_shape(spec["match_when"])
rule = {
"id": decision.wd_id or f"wd-{spec['slug']}",
"name": spec["name"],
"severity": spec["severity"],
"category": category,
"mitre": [str(m).upper() for m in spec.get("mitre") or []],
"match_when": shape.residual,
"group_by": decision.group_by,
"threshold": shape.threshold,
"window_seconds": shape.window_seconds,
"translated_from": spec["slug"],
}
if decision.distinct_by:
rule["distinct_by"] = decision.distinct_by
return rule
def decided_keys() -> frozenset[str]:
"""`<category>/<slug>` for every rule this module has decided."""
return frozenset(DECISIONS)
def _load_specs() -> dict[str, tuple[str, dict[str, Any]]]:
from detection_specs_index import all_specs # noqa: PLC0415
return {f"{category}/{spec['slug']}": (category, spec) for category, spec in all_specs()}
def translated_rules() -> list[dict[str, Any]]:
"""Every `wd-*` rule derived from a decided `det-*` spec, in table order."""
specs = _load_specs()
out: list[dict[str, Any]] = []
for key, decision in DECISIONS.items():
if not isinstance(decision, Translation):
continue
if key not in specs:
raise SystemExit(
f"windowed_translation: {key!r} is decided but no spec declares it. "
"A slug was renamed or removed; update DECISIONS in the same change."
)
category, spec = specs[key]
out.append(derive(category, spec, decision))
return out
def retirement_reason(key: str) -> str:
"""Catalogue-facing sentence for a decided rule, or '' when undecided."""
decision = DECISIONS.get(key)
if isinstance(decision, Translation):
wd_id = decision.wd_id or f"wd-{key.split('/', 1)[1]}"
return f"superseded by the windowed rule {wd_id}; this stateless form named a counter no source emits"
if isinstance(decision, Covered):
return f"already detected by the windowed rule {decision.by}; {decision.reason}"
if isinstance(decision, Refusal):
return f"not executable ({decision.kind}): {decision.reason}"
return ""