Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
Give every connector type a deliberate OCSF class, so routine telemetry is not promoted as a vendor finding (#1195)
Beenu Aroraf9ecdbf72264
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex f00adb24b..e827e755b 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -649,6 +649,16 @@ jobs:run: python3 scripts/check_connector_profiles.py --self-test- name: Connector profile drift gaterun: python3 scripts/check_connector_profiles.py+ # One layer in from the name: the profile gate asks whether a connector+ # reaches a normalization path, this asks which OCSF class it reaches+ # when it gets there. The class uid divided by a thousand is the+ # category, and fusion promotes category 2 unconditionally — so 78 of+ # the 84 connectors landing on the 2001 default meant every permitted+ # DNS lookup and every routine Windows event became an alert.+ - name: OCSF class coverage gate — self-test+ run: python3 scripts/check_ocsf_class_coverage.py --self-test+ - name: OCSF class coverage gate+ run: python3 scripts/check_ocsf_class_coverage.py --check# One layer in again: a connector's events reach a profile, and the# catalogue says what each of its event types *means*. Grown from the# fixtures, so adding a fixture that carries a new event type failsdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 325125314..d90385114 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -234,6 +234,63 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0both classified and excused, and any drift between `schemas/` and thevendored copy the binary carries.+### Fixed++- **Routine telemetry was promoted to an alert as though a vendor had judged+ it.** Ingest mapped 78 of the 84 declared connector types onto OCSF class+ 2001 Security Finding, because `canonicalProfile()` used it as the default+ and only five connectors had an override. 2001 is category 2, and+ `should_promote()` promotes category 2 unconditionally — so every permitted+ DNS lookup from Cisco Umbrella, every `Allow` from Zscaler, every accepted+ VPC flow record, every routine Windows event and every Workspace or M365+ audit entry became an alert carrying severity `info`. Measured on the tree+ before the fix: **14 of 14** telemetry sources promoted an info-severity+ event.++ `services/ingest/internal/normalizer/ocsf_classes.go` now carries a decision+ for every connector type the registry declares: the OCSF class its stream+ actually is, or the reason it stays on the Security Finding default. Which+ one each source gets was read from the endpoint its connector calls rather+ than from the vendor's product category, and the two disagree often enough+ that guessing would have silenced four sources — Netskope reads+ `/api/v2/events/data/alert`, Proofpoint reads `/v2/siem/messages/blocked`,+ Abnormal reads `/v1/threats`, and Falco drops anything below its configured+ priority, so all four return judgements and keep category 2.++ Ten classes are added, each uid, caption and category read from+ `schema.ocsf.io` for both the 1.9.0 the tree targets and the 1.1.0 the+ normalizer's `metadata.version` declares, because emitting a newer class+ under the older version string would be a false claim about a public+ standard: Process Activity, File System Activity, DNS Activity, HTTP+ Activity, Email Activity, Account Change, User Access Management, Group+ Management, Web Resources Activity and Datastore Activity, plus Email URL+ Activity and File Hosting Activity where the schema's own description named+ the product. Connector types on the generic mapping: **78 → 42**, all 42+ with a recorded reason and none without a decision. (depth plan 2.1)++- **A Sentinel incident was labelled a vulnerability finding.** The+ `microsoft_sentinel` profile declared `class_uid: 2002` beside+ `class_name: "Security Finding"`; 2002 is Vulnerability Finding in every+ OCSF version this service targets. Both are category 2, so promotion hid it,+ but every Sentinel event in the lake disagreed with itself and a query+ filtering on 2002 returned incidents alongside Qualys and Tenable findings.+ Found by the new gate, which fails on a profile whose class name contradicts+ its own uid.++### Added++- `scripts/check_ocsf_class_coverage.py`, wired into `ci.yml` with its+ self-test ahead of it. A declared connector with neither a class nor a+ recorded reason fails, as does an entry carrying both, a class uid whose+ declared category disagrees with `uid/1000`, a non-finding class whose+ severity map cannot reach the promote floor of 4, and a profile whose class+ name contradicts its uid. It reports the number of connector types on the+ generic mapping and holds it at a ceiling that may shrink but not grow. The+ self-test injects each of those eleven defects and requires the gate to+ catch each one, and requires an empty tree to be refused rather than+ reported clean.++## [17.1.0] - 2026-10-07diff --git a/scripts/check_ocsf_class_coverage.py b/scripts/check_ocsf_class_coverage.pynew file mode 100644index 000000000..7ebb09032--- /dev/null+++ b/scripts/check_ocsf_class_coverage.py@@ -0,0 +1,600 @@+#!/usr/bin/env python3+"""Every connector type names an OCSF class, or says why it does not.++Why this exists+---------------+``should_promote()`` in ``services/fusion/app/services/promoter.py`` promotes+an event to an alert when it is OCSF **category 2** (Findings) *or* its+``severity_id`` is at least 4. The category is the class uid divided by a+thousand, so the class a connector's events carry decides whether its routine+telemetry reaches an analyst's queue or stays in the lake.++Before the table this gate reads, 72 of the 84 declared connector types had no+class of their own. They fell to the ``2001`` Security Finding default inside+``canonicalProfile()`` — category 2, therefore **always promoted** — and that+default was invisible: nothing named it, nothing counted it, and no reviewer+saw a list of the sources it covered.++For the connectors whose ``fetch_alerts`` really does return the vendor's own+findings, 2001 is right. For the ones returning raw telemetry it meant every+permitted DNS lookup, every allowed proxy request, every accepted VPC flow+record and every routine Windows event became an alert carrying severity+``info`` — fusion's alert-reduction property inverted, for the sources that+emit the most volume. Reproduced on the pre-fix tree by+``TestRoutineTelemetryWasPromotedBeforeTheClassTable`` in the normalizer's own+suite.++What it checks+--------------+Two directions, because a one-directional check is this repository's dominant+gate defect — it compares A against B, never B against A, so drift in the+direction things actually change slips past while the gate prints OK.++ PY -> GO every ``connector_id`` the connectors service declares has+ either a class mapping or a recorded reason it stays on the+ default. Neither is the failure this gate exists for.+ GO -> PY every entry in the Go table names a connector that exists, no+ entry carries both a class and a reason, and no entry names a+ class uid absent from the service's own closed set.+ SCHEMA every class in that closed set declares the category its uid+ implies, so a typo cannot change which promotion branch a+ connector's events take.+ PROMOTION every class a connector is mapped to can actually promote: it+ is category 2, or the profile carrying it ships a severity map+ that reaches the floor of 4. A new class with an empty severity+ map is archived to the lake and never alerts, silently, which is+ exactly how the borrowed ``splunk_enterprise`` profile broke 26+ connectors.++It also reports the **count of connector types on the generic mapping**, which+is the figure the depth plan's Phase 2 is graded against, and holds it with+``--check`` against the committed number so it can only go down by a decision+somebody made rather than drift up by accident.++Usage+-----+ python3 scripts/check_ocsf_class_coverage.py # gate+ python3 scripts/check_ocsf_class_coverage.py --check # + drift on the count+ python3 scripts/check_ocsf_class_coverage.py --json+ python3 scripts/check_ocsf_class_coverage.py --self-test # prove it bites++``--repo-root`` overrides the tree under inspection. The resolved root, every+file read and every count are printed before the verdict, and an empty read is+a hard error rather than a quiet zero.+"""++from __future__ import annotations++import argparse+import ast+import json+import re+import sys+from pathlib import Path++# `scripts/` is on sys.path when this runs as a program but not when a test+# loads it by path with importlib. gate_toolkit sits beside it either way.+sys.path.insert(0, str(Path(__file__).resolve().parent))++from gate_toolkit import repo_root++CLASSES_REL = Path("services/ingest/internal/normalizer/ocsf_classes.go")+NORMALIZER_REL = Path("services/ingest/internal/normalizer/normalizer.go")+CONNECTORS_REL = Path("services/connectors/app/connectors")++#: Connector types on the generic mapping, as committed. The figure the depth+#: plan grades Phase 2 against, so it lives in code with a drift gate rather+#: than in prose that goes stale silently.+#:+#: 72 at capture (commit ee33b02f): 84 declared, and only 12 with a class of+#: their own — 8 through a hand-written profile and 4 more through the+#: five-entry override map this table replaced. It may fall as connectors are+#: classified. It may **not** rise: a new connector arrives with a decision or+#: the gate fails, which is the whole point of counting it.+#:+#: The number that must stay at zero is `unclassified`, which is a hard+#: failure rather than a ceiling: every one of the 42 below carries a reason.+MAX_GENERIC = 42++#: OCSF categories whose events fusion promotes unconditionally.+FINDINGS_CATEGORY = 2++#: The severity_id floor the promoter applies to everything else.+PROMOTE_SEVERITY_FLOOR = 4+++class GateError(RuntimeError):+ """An input could not be read. Never downgraded to a passing result."""+++# --------------------------------------------------------------------------+# Go side+# --------------------------------------------------------------------------+def _go_block(src: str, header: str) -> str:+ """The brace-balanced literal following `header`."""+ start = src.find(header)+ if start == -1:+ raise GateError(f"could not find `{header}`")+ depth, i = 0, src.index("{", start)+ for j in range(i, len(src)):+ if src[j] == "{":+ depth += 1+ elif src[j] == "}":+ depth -= 1+ if depth == 0:+ return src[i : j + 1]+ raise GateError(f"unbalanced braces after `{header}`")+++def parse_class_consts(src: str) -> dict[str, int]:+ """Go constant name -> uid, from the const block."""+ return {name: int(uid) for name, uid in re.findall(r"^\t(class\w+)\s*=\s*(\d+)$", src, re.MULTILINE)}+++def parse_classes(src: str, consts: dict[str, int]) -> dict[int, tuple[str, int]]:+ """uid -> (caption, declared category), from `ocsfClasses`."""+ block = _go_block(src, "var ocsfClasses = map[int]ocsfClass{")+ out: dict[int, tuple[str, int]] = {}+ for key, uid_ref, caption, category in re.findall(r'^\t(class\w+):\s*\{(class\w+),\s*"([^"]+)",\s*(\d+)\},?$', block, re.MULTILINE):+ if key not in consts:+ raise GateError(f"ocsfClasses keys off {key!r}, which the const block does not declare")+ if uid_ref != key:+ raise GateError(f"ocsfClasses row {key!r} carries uid {uid_ref!r}; a row must describe its own key")+ out[consts[key]] = (caption, int(category))+ return out+++def parse_connector_classes(src: str, consts: dict[str, int]) -> dict[str, dict]:+ """connector type -> {"class_uid": int|None, "reason": str|None}."""+ block = _go_block(src, "var connectorOCSFClass = map[string]connectorClass{")+ out: dict[str, dict] = {}+ for name, body in re.findall(r'^\t"([a-z0-9_]+)":\s*\{([^}]*)\},?$', block, re.MULTILINE):+ class_match = re.search(r"classUID:\s*(class\w+)", body)+ reason_match = re.search(r'genericReason:\s*"([^"]*)"', body)+ class_uid = None+ if class_match:+ key = class_match.group(1)+ if key not in consts:+ raise GateError(f"connector {name!r} names class constant {key!r}, which is not declared")+ class_uid = consts[key]+ out[name] = {"class_uid": class_uid, "reason": reason_match.group(1) if reason_match else None}+ return out+++def parse_profile_classes(src: str) -> dict[str, dict]:+ """profile key -> {"class_uid", "class_name", "severity_promotes"}.++ ``severity_promotes`` answers the question the promoter asks: can any+ value in this profile's severity map reach the floor of 4? A profile whose+ map is empty, or tops out at 3, cannot promote a non-finding event however+ severe the vendor thought it was. Shared-ladder references are resolved+ rather than treated as unknown, because the whole point of naming the+ shared ladder was that a reader can see what it contains.+ """+ block = _go_block(src, "var connectorProfiles = map[string]connectorProfile{")+ shared = {+ name: [int(v) for v in re.findall(r":\s*(\d+)", _go_block(src, f"var {name} = map[string]int{{"))]+ for name in re.findall(r"var (_\w*[Ss]everityMap) = map\[string\]int\{", src)+ }+ out: dict[str, dict] = {}+ for match in re.finditer(r'^\t"([a-z0-9_]+)":\s*\{$', block, re.MULTILINE):+ key = match.group(1)+ entry = _go_block(block, match.group(0))+ class_match = re.search(r"classUID:\s*(\d+)", entry)+ if not class_match:+ raise GateError(f"profile {key!r} declares no classUID")+ name_match = re.search(r'className:\s*"([^"]+)"', entry)+ ref = re.search(r"severityMap:\s*(_\w+),", entry)+ if ref:+ values = shared.get(ref.group(1))+ if values is None:+ raise GateError(f"profile {key!r} references severity map {ref.group(1)!r}, which is not declared")+ else:+ sev_block = re.search(r"severityMap:\s*map\[string\]int\{(.*?)\}", entry, re.DOTALL)+ values = [int(v) for v in re.findall(r":\s*(\d+)", sev_block.group(1))] if sev_block else []+ out[key] = {+ "class_uid": int(class_match.group(1)),+ "class_name": name_match.group(1) if name_match else None,+ "severity_promotes": any(v >= PROMOTE_SEVERITY_FLOOR for v in values),+ }+ return out+++def parse_canonical_severity_promotes(src: str) -> bool:+ """Whether the ladder every canonical envelope uses can reach the floor."""+ block = _go_block(src, "var _canonicalSeverityMap = map[string]int{")+ return any(int(v) >= PROMOTE_SEVERITY_FLOOR for v in re.findall(r":\s*(\d+)", block))+++# --------------------------------------------------------------------------+# Python side+# --------------------------------------------------------------------------+def parse_declared_connectors(directory: Path) -> set[str]:+ """Every `connector_id` the connectors service declares."""+ out: set[str] = set()+ for path in sorted(directory.glob("*.py")):+ if path.name in {"__init__.py", "base.py"}:+ continue+ tree = ast.parse(path.read_text(encoding="utf-8"))+ for cls in (n for n in ast.walk(tree) if isinstance(n, ast.ClassDef)):+ for stmt in cls.body:+ # The value is bound in each branch rather than read off+ # `stmt` afterwards: past the if/elif the name is still an+ # `ast.stmt`, which has no `.value`, so the later read was+ # four unchecked attribute accesses that happened to be safe.+ if isinstance(stmt, ast.Assign):+ target = next((getattr(t, "id", None) for t in stmt.targets), None)+ value: ast.expr | None = stmt.value+ elif isinstance(stmt, ast.AnnAssign):+ target = getattr(stmt.target, "id", None)+ value = stmt.value+ else:+ continue+ if target == "connector_id" and isinstance(value, ast.Constant) and isinstance(value.value, str) and value.value:+ out.add(value.value)+ return out+++# --------------------------------------------------------------------------+# Gate+# --------------------------------------------------------------------------+def evaluate(+ declared: set[str],+ classes: dict[int, tuple[str, int]],+ connector_classes: dict[str, dict],+ profiles: dict[str, dict],+ canonical_severity_promotes: bool,+ max_generic: int,+ check_count: bool,+) -> tuple[list[tuple[str, str]], dict]:+ failures: list[tuple[str, str]] = []++ # SCHEMA. A uid whose declared category disagrees with uid/1000 would+ # change which promotion branch a connector's events take.+ for uid, (caption, category) in sorted(classes.items()):+ if uid // 1000 != category:+ failures.append(+ (+ "class-category-mismatch",+ f"class {uid} ({caption}) declares category {category}, but uid/1000 is {uid // 1000}; "+ "fusion derives the category from the uid, so the two cannot differ",+ )+ )++ # GO -> PY. Every entry names something real and says exactly one thing.+ for name, entry in sorted(connector_classes.items()):+ if name not in declared:+ failures.append(("class-names-nothing", f"connectorOCSFClass names {name!r}, which no connector declares"))+ if entry["class_uid"] is None and not entry["reason"]:+ failures.append(("class-entry-empty", f"{name!r} has an entry carrying neither a class nor a reason"))+ if entry["class_uid"] is not None and entry["reason"]:+ failures.append(+ (+ "class-entry-ambiguous",+ f"{name!r} declares both a class and a generic reason; one of them is not what happens",+ )+ )+ if entry["class_uid"] is not None and entry["class_uid"] not in classes:+ failures.append(+ (+ "class-uid-unknown",+ f"{name!r} maps to class uid {entry['class_uid']}, which is not in the service's closed set",+ )+ )+ if entry["reason"] and len(entry["reason"]) < 15:+ failures.append(+ (+ "class-reason-thin",+ f"{name!r} stays generic for the reason {entry['reason']!r}, which does not say enough "+ "for a reviewer to disagree with it",+ )+ )++ # PY -> GO. The direction the defect lived in: a declared connector with+ # no decision at all, reaching the default silently.+ generic: list[str] = []+ for name in sorted(declared):+ decision = connector_classes.get(name)+ if decision is None:+ failures.append(+ (+ "connector-unclassified",+ f"connector {name!r} has no OCSF class and no recorded reason: its events take the 2001 "+ "Security Finding default, which is category 2 and therefore always promoted, and nothing "+ "says that was a decision",+ )+ )+ generic.append(name)+ elif decision["class_uid"] is None:+ generic.append(name)++ # PROMOTION. A class a connector is mapped to must be able to promote.+ for name, entry in sorted(connector_classes.items()):+ uid = entry["class_uid"]+ if uid is None or uid not in classes:+ continue+ if uid // 1000 == FINDINGS_CATEGORY:+ continue+ # A connector with its own profile carries that profile's ladder; one+ # on the canonical path carries the shared ladder.+ promotes = profiles[name]["severity_promotes"] if name in profiles else canonical_severity_promotes+ if not promotes:+ caption, _ = classes[uid]+ failures.append(+ (+ "class-cannot-promote",+ f"{name!r} maps to {uid} ({caption}), category {uid // 1000}, and the severity map it carries "+ f"never reaches {PROMOTE_SEVERITY_FLOOR}: every one of its events would be archived to the lake "+ "and never alert, silently",+ )+ )++ # A profile declaring a class the closed set does not know is the same+ # defect one layer over, and the profiles predate this table.+ for key, profile in sorted(profiles.items()):+ if profile["class_uid"] not in classes:+ failures.append(+ (+ "profile-class-unknown",+ f"profile {key!r} declares class {profile['class_uid']}, which is not in the closed set; "+ "the gate cannot tell whether it can promote",+ )+ )+ continue+ # A profile carries the uid *and* the caption, written by hand and+ # side by side, which is how `microsoft_sentinel` came to declare uid+ # 2002 beside "Security Finding" — the caption of 2001. Both are+ # category 2, so promotion hid it, and every Sentinel event in the+ # lake disagreed with itself while a query filtering on 2002 pulled+ # incidents in among vulnerability findings.+ caption, _ = classes[profile["class_uid"]]+ if profile["class_name"] and profile["class_name"] != caption:+ failures.append(+ (+ "profile-class-name-mismatch",+ f"profile {key!r} declares class {profile['class_uid']} beside the name "+ f"{profile['class_name']!r}; the schema calls that uid {caption!r}, so one of the two is "+ "wrong and both travel onto every event",+ )+ )++ unclassified = sorted(n for n in generic if n not in connector_classes)+ stats = {+ "declared": len(declared),+ "with_class": sum(1 for e in connector_classes.values() if e["class_uid"] is not None and e["reason"] is None),+ "generic": len(generic),+ "generic_connectors": generic,+ "generic_with_reason": sorted(set(generic) - set(unclassified)),+ "unclassified": unclassified,+ "classes": len(classes),+ }++ if check_count and len(generic) > max_generic:+ failures.append(+ (+ "generic-count-grew",+ f"{len(generic)} connector types are on the generic mapping, above the committed ceiling of "+ f"{max_generic}. The ceiling is a debt balance, not a target: classify the new connector or "+ "record why it stays generic, rather than raising the number",+ )+ )+ return failures, stats+++def load(root: Path) -> dict:+ classes_go = root / CLASSES_REL+ normalizer = root / NORMALIZER_REL+ connectors_dir = root / CONNECTORS_REL+ for path in (classes_go, normalizer, connectors_dir):+ if not path.exists():+ raise GateError(f"expected input does not exist: {path}")++ classes_src = classes_go.read_text(encoding="utf-8")+ normalizer_src = normalizer.read_text(encoding="utf-8")+ consts = parse_class_consts(classes_src)+ data = {+ "declared": parse_declared_connectors(connectors_dir),+ "classes": parse_classes(classes_src, consts),+ "connector_classes": parse_connector_classes(classes_src, consts),+ "profiles": parse_profile_classes(normalizer_src),+ "canonical_severity_promotes": parse_canonical_severity_promotes(normalizer_src),+ }+ # An empty parse means the format moved, not that the tree is clean.+ for name in ("declared", "classes", "connector_classes", "profiles"):+ if not data[name]:+ raise GateError(f"parsed zero {name} — refusing to report a clean tree from an empty read")+ if not data["canonical_severity_promotes"]:+ raise GateError(+ "_canonicalSeverityMap never reaches the promote floor — every canonical-envelope connector on a "+ "non-finding class would be unpromotable, so refusing to report on the mapping at all"+ )+ return data+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ parser.add_argument("--repo-root", type=Path, default=repo_root())+ parser.add_argument("--json", action="store_true")+ parser.add_argument("--check", action="store_true", help="also fail when the generic-mapping count exceeds MAX_GENERIC")+ parser.add_argument("--self-test", action="store_true", help="prove the gate detects what it claims to")+ args = parser.parse_args(argv)++ if args.self_test:+ return self_test(args.repo_root.resolve())++ root = args.repo_root.resolve()+ try:+ data = load(root)+ except GateError as exc:+ print(f"check_ocsf_class_coverage: FAILED to read the tree: {exc}", file=sys.stderr)+ return 2++ failures, stats = evaluate(**data, max_generic=MAX_GENERIC, check_count=args.check)++ if args.json:+ print(+ json.dumps(+ {+ "repo_root": str(root),+ "max_generic": MAX_GENERIC,+ **stats,+ "failures": [{"code": c, "detail": d} for c, d in failures],+ },+ indent=2,+ sort_keys=True,+ )+ )+ return 1 if failures else 0++ print(f"repo root {root}")+ print(f"class table {CLASSES_REL} ({stats['classes']} OCSF classes, verified against schema.ocsf.io)")+ print(f"normalizer {NORMALIZER_REL} ({len(data['profiles'])} profiles)")+ print(f"connectors {CONNECTORS_REL} ({stats['declared']} declared connector ids)")+ print()+ print(f"class coverage {stats['with_class']} with an OCSF class of their own")+ print(f" {stats['generic']} on the generic 2001 mapping (ceiling {MAX_GENERIC})")+ print(+ f" of those, {len(stats['generic_with_reason'])} with a recorded reason "+ f"and {len(stats['unclassified'])} with no decision at all"+ )+ if stats["generic_connectors"]:+ print(f" {', '.join(stats['generic_connectors'])}")+ print()+ if failures:+ print(f"FAIL — {len(failures)} finding(s):")+ for code, detail in failures:+ print(f" [{code}] {detail}")+ return 1+ print("OK — every declared connector type names an OCSF class or records why it stays")+ print(" generic, every class can promote, and every uid matches its category.")+ return 0+++# --------------------------------------------------------------------------+# Self-test+# --------------------------------------------------------------------------+def self_test(root: Path) -> int:+ """Inject each defect the gate claims to catch and require it to bite."""+ try:+ base = load(root)+ except GateError as exc:+ print(f"self-test: cannot read the tree: {exc}", file=sys.stderr)+ return 2++ clean, _ = evaluate(**base, max_generic=MAX_GENERIC, check_count=True)+ if clean:+ print("self-test: the unmodified tree already fails; fix that first", file=sys.stderr)+ for code, detail in clean:+ print(f" [{code}] {detail}", file=sys.stderr)+ return 1++ def mutate(**overrides) -> dict:+ data = {k: (v.copy() if hasattr(v, "copy") else v) for k, v in base.items()}+ for key, fn in overrides.items():+ fn(data[key])+ return data++ cases: list[tuple[str, str, dict, int]] = [+ (+ "PY -> GO: a declared connector with no class and no recorded reason",+ "connector-unclassified",+ mutate(declared=lambda s: s.add("acme_telemetry")),+ MAX_GENERIC,+ ),+ (+ "GO -> PY: a table entry naming a connector that does not exist",+ "class-names-nothing",+ mutate(connector_classes=lambda d: d.update({"acme_xdr_9000": {"class_uid": 4001, "reason": None}})),+ MAX_GENERIC,+ ),+ (+ "GO -> PY: an entry carrying both a class and a reason",+ "class-entry-ambiguous",+ mutate(connector_classes=lambda d: d.update({"okta": {"class_uid": 3002, "reason": "also stays generic somehow"}})),+ MAX_GENERIC,+ ),+ (+ "GO -> PY: an entry carrying neither",+ "class-entry-empty",+ mutate(connector_classes=lambda d: d.update({"okta": {"class_uid": None, "reason": None}})),+ MAX_GENERIC + 1,+ ),+ (+ "GO -> PY: a class uid outside the service's closed set",+ "class-uid-unknown",+ mutate(connector_classes=lambda d: d.update({"okta": {"class_uid": 9999, "reason": None}})),+ MAX_GENERIC,+ ),+ (+ "GO -> PY: a reason too thin to review",+ "class-reason-thin",+ mutate(connector_classes=lambda d: d.update({"splunk": {"class_uid": None, "reason": "noisy"}})),+ MAX_GENERIC + 1,+ ),+ (+ "SCHEMA: a class whose declared category disagrees with its uid",+ "class-category-mismatch",+ mutate(classes=lambda d: d.update({4003: ("DNS Activity", 6)})),+ MAX_GENERIC,+ ),+ (+ "PROMOTION: a non-finding class whose severity map cannot reach the floor",+ "class-cannot-promote",+ mutate(profiles=lambda d: d.update({"okta": {"class_uid": 3002, "class_name": "Authentication", "severity_promotes": False}})),+ MAX_GENERIC,+ ),+ (+ "PROMOTION: a profile declaring a class the closed set does not know",+ "profile-class-unknown",+ mutate(profiles=lambda d: d.update({"splunk": {"class_uid": 7777, "class_name": "Invented", "severity_promotes": True}})),+ MAX_GENERIC,+ ),+ (+ "SCHEMA: a profile whose class name contradicts its own class uid",+ "profile-class-name-mismatch",+ mutate(+ profiles=lambda d: d.update({"splunk": {"class_uid": 2002, "class_name": "Security Finding", "severity_promotes": True}})+ ),+ MAX_GENERIC,+ ),+ (+ "COUNT: the generic-mapping count rises above the committed ceiling",+ "generic-count-grew",+ mutate(declared=lambda s: s.add("acme_telemetry")),+ MAX_GENERIC,+ ),+ ]++ print(f"self-test against {root}")+ print(f"clean tree: 0 failures at MAX_GENERIC={MAX_GENERIC} (the baseline every case below perturbs)\n")+ ok = True+ for description, expected_code, data, ceiling in cases:+ found, _ = evaluate(**data, max_generic=ceiling, check_count=True)+ codes = {code for code, _ in found}+ caught = expected_code in codes+ ok &= caught+ print(f" {'PASS' if caught else 'FAIL'} {description}")+ print(f" expected [{expected_code}] got {sorted(codes) or 'nothing'}")++ # An empty tree must be refused, not reported clean.+ empty_refused = True+ try:+ load(Path("/nonexistent-tree-for-the-self-test"))+ empty_refused = False+ except GateError:+ # Refusing an empty tree is the behaviour under test, so the+ # exception is the pass and there is nothing to handle.+ pass+ ok &= empty_refused+ print(f" {'PASS' if empty_refused else 'FAIL'} TREE: a tree with no inputs is refused rather than reported clean")++ print()+ if not ok:+ print("self-test FAILED: the gate did not catch what it claims to catch")+ return 1+ print(f"self-test OK: {len(cases)} injected defects plus the empty-tree case, each caught")+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/scripts/competitor_names.toml b/scripts/competitor_names.tomlindex fa7828817..81aea490f 100644--- a/scripts/competitor_names.toml+++ b/scripts/competitor_names.toml@@ -264,3 +264,8 @@ file = "apps/docs/src/pages/index.tsx"start = "const COMPARE_HEADERS"end = "function compareCellClass"reason = "Capability comparison table rendered on the documentation portal landing page."++[[allow]]+paths = ["services/ingest/internal/normalizer/ocsf_classes.go"]+names = ["Torq", "Tines", "Wazuh"]+reason = "The OCSF class decision for every declared connector type. Each name is a connector id this platform ingests under, beside the reason its stream stays on Security Finding — all three of these send a judgement the vendor's own ruleset or workflow already made, which is why they are not reclassified to a raw-activity class. Dropping a name here would return that connector to the generic mapping, which promotes every record to an alert; the same first-party integration reference the connector modules and the bundled catalog already carry."diff --git a/services/ingest/internal/normalizer/normalizer.go b/services/ingest/internal/normalizer/normalizer.goindex 4946023fd..f23b57d7f 100644--- a/services/ingest/internal/normalizer/normalizer.go+++ b/services/ingest/internal/normalizer/normalizer.go@@ -112,9 +112,17 @@ var connectorProfiles = map[string]connectorProfile{"Critical": 5, "High": 4, "Medium": 3, "Low": 2, "Informational": 1,},},+ // 2001 and not 2002: the profile declared uid 2002 beside the caption+ // "Security Finding", and 2002 is Vulnerability Finding in every OCSF+ // version this service targets. A Sentinel incident is a detection, not+ // a vulnerability. Nothing downstream read the uid closely enough to+ // notice — both are category 2, so promotion was unaffected — but the+ // class_uid and class_name on every Sentinel event in the lake+ // contradicted each other, and a lake query filtering on 2002 would have+ // returned Sentinel incidents alongside Qualys and Tenable findings."microsoft_sentinel": {product: OcsfProduct{Name: "Sentinel", VendorName: "Microsoft"},- classUID: 2002,+ classUID: 2001,className: "Security Finding",fieldMap: map[string]string{"TimeGenerated": "time",@@ -569,7 +577,7 @@ var connectorTypeAliases = map[string]string{// them safe: the connector's own connector_name is the long form (`qradar` is// "IBM QRadar", `chronicle` is "Google Chronicle", `syslog_cef` is// "Syslog / CEF"). Resolution happens once, at the top of Normalize, so the-// profile lookup, the alias map, canonicalClassByConnector and the product+// profile lookup, the alias map, connectorOCSFClass and the product// identity on the canonical path all agree on one name.//// `palo_alto_cortex` was the one entry that named a vendor rather than a@@ -606,20 +614,6 @@ func canonicalConnectorType(connectorType string) string {return connectorType}-// canonicalClassByConnector overrides the default Security Finding class for-// connector types whose canonical alerts are better modeled as another OCSF-// class (identity providers -> Authentication 3002).-var canonicalClassByConnector = map[string]struct {- classUID int- className string-}{- "okta": {3002, "Authentication"},- "azure_entra": {3002, "Authentication"},- "auth0": {3002, "Authentication"},- "duo_security": {3002, "Authentication"},- "onepassword": {3002, "Authentication"},-}-func isCanonicalEnvelope(p map[string]interface{}) bool {if p == nil {return false@@ -704,10 +698,20 @@ func unpromotableWarning(classUID int, severityID interface{}) string {)}+// canonicalProfile builds the profile for a connector-normalized envelope.+//+// The class comes from `connectorOCSFClass` in ocsf_classes.go, which carries+// a decision for every connector the registry declares — either the OCSF+// class its telemetry is, or a recorded reason it stays on the Security+// Finding default. That default is category 2, which fusion promotes+// unconditionally, so a source left on it silently turns every routine record+// it emits into an alert. The table replaced a five-entry override map; see+// the file header for the promotion mechanism and+// `scripts/check_ocsf_class_coverage.py` for the gate that keeps it complete.func canonicalProfile(connectorType string) connectorProfile {- classUID, className := 2001, "Security Finding"- if override, ok := canonicalClassByConnector[connectorType]; ok {- classUID, className = override.classUID, override.className+ classUID, className := classSecurityFinding, "Security Finding"+ if class, ok := ocsfClassForConnector(connectorType); ok {+ classUID, className = class.uid, class.caption}name := connectorTypeif name == "" {diff --git a/services/ingest/internal/normalizer/normalizer_test.go b/services/ingest/internal/normalizer/normalizer_test.goindex 50d78493b..8895f11a7 100644--- a/services/ingest/internal/normalizer/normalizer_test.go+++ b/services/ingest/internal/normalizer/normalizer_test.go@@ -757,10 +757,15 @@ func TestConsoleVocabularyNormalizesAsTheDeclaredConnector(t *testing.T) {if gotAlt.OcsfEvent["severity_id"] != 5 {t.Errorf("severity_id = %v via %q, want 5 for critical", gotAlt.OcsfEvent["severity_id"], alternate)}- // Category 2 is what should_promote() requires without a severity- // floor; a connector whose events cannot promote is the defect.- if gotAlt.OcsfEvent["category_uid"] != 2 {- t.Errorf("category_uid = %v via %q, want 2", gotAlt.OcsfEvent["category_uid"], alternate)+ // The property this stands for is that the event can reach an+ // analyst. It was written as `category_uid == 2` when every+ // canonical envelope defaulted to 2001; once connectorOCSFClass+ // gave telemetry sources their own class, that spelling stopped+ // describing the requirement and started describing one way of+ // meeting it. should_promote() takes either branch.+ if !promotable(gotAlt.OcsfEvent) {+ t.Errorf("a critical event via %q cannot promote: category_uid=%v severity_id=%v",+ alternate, gotAlt.OcsfEvent["category_uid"], gotAlt.OcsfEvent["severity_id"])}})}@@ -789,8 +794,11 @@ func TestConsoleVocabularyIsAcceptedInStrictModeOnlyWhenItResolves(t *testing.T)t.Errorf("strict mode rejected %q, which folds onto a declared connector: %v", alternate, err)continue}- if ev.OcsfEvent["category_uid"] != 2 {- t.Errorf("%q: category_uid = %v, want 2", alternate, ev.OcsfEvent["category_uid"])+ // As above: the requirement is that a high-severity event resolves to+ // something promotable, not that it resolves to category 2.+ if !promotable(ev.OcsfEvent) {+ t.Errorf("%q: a high event cannot promote: category_uid=%v severity_id=%v",+ alternate, ev.OcsfEvent["category_uid"], ev.OcsfEvent["severity_id"])}}if _, err := strict.Normalize(&RawEvent{diff --git a/services/ingest/internal/normalizer/ocsf_classes.go b/services/ingest/internal/normalizer/ocsf_classes.gonew file mode 100644index 000000000..76975e8ea--- /dev/null+++ b/services/ingest/internal/normalizer/ocsf_classes.go@@ -0,0 +1,350 @@+package normalizer++// OCSF event classes, and which one each connector type's telemetry is.+//+// Every uid, caption and category below was read from the published schema+// before it was used here, not recalled:+//+// https://schema.ocsf.io/api/1.9.0/classes (the version the tree targets)+// https://schema.ocsf.io/api/1.1.0/classes (the version metadata.version declares)+//+// Both were checked because the two numbers are not the same thing. The+// normalizer stamps `metadata.version: 1.1.0` as its contract with+// connectors, while the evidence bundle declares 1.9.0+// (`docs/architecture/evidence-bundles.md`), and emitting a class that exists+// only in the newer schema under the older version string would be a false+// claim about a public standard. Every class named here carries the same uid,+// caption and category in both, so the question does not arise — but it was+// asked, and it is why this file lists two URLs rather than one.+//+// Why a connector's class is load-bearing rather than cosmetic+// ------------------------------------------------------------+// `should_promote()` in services/fusion/app/services/promoter.py promotes an+// event to an alert when it is OCSF category 2 (Findings) **or** its+// severity_id is at least 4. The category is the class uid divided by a+// thousand, so the class a connector's events carry decides whether its+// routine telemetry reaches an analyst's queue or stays in the lake.+//+// Before this table, every connector without a hand-written profile landed on+// 2001 Security Finding through `canonicalProfile` — category 2, therefore+// always promoted. For the connectors whose fetch_alerts really does return+// vendor findings that is right. For the ones returning raw telemetry it+// meant every permitted DNS lookup, every allowed proxy request, every+// accepted VPC flow record and every routine Windows event became an alert+// carrying severity "info": the alert-reduction property fusion exists to+// provide, inverted, for the sources that emit the most volume.+//+// The split mirrors the one `ai_runtime` and `ai_guardrail` already make in+// normalizer.go, for the same stated reason: routine activity at a category+// the promoter leaves alone, judged findings at category 2.+//+// How a connector's class was chosen+// ----------------------------------+// By what its `fetch_alerts` actually returns, read from the connector rather+// than inferred from the vendor's product category — the two disagree often+// enough that guessing would have demoted four sources that only ever return+// findings. Netskope's connector reads `/api/v2/events/data/alert` and+// Proofpoint's reads `/v2/siem/messages/blocked`, so both return judgements+// and keep category 2; Zscaler's reads web logs with `action: "any"` and+// Cisco Umbrella's reads `/activity`, so both are telemetry and take their+// activity class. Where a stream is genuinely mixed the class covering the+// security-relevant majority is used and the entry says so.++// OCSF class uids this service may emit. Named rather than spelled inline so+// the gate reads one table and a reader can see the category each implies+// (uid/1000), which is what decides promotion.+const (+ // Category 1 — System Activity.+ classFileSystemActivity = 1001+ classProcessActivity = 1007++ // Category 2 — Findings. Promoted unconditionally.+ classSecurityFinding = 2001+ classVulnerabilityFinding = 2002++ // Category 3 — Identity & Access Management.+ classAccountChange = 3001+ classAuthentication = 3002+ classUserAccessManagement = 3005+ classGroupManagement = 3006++ // Category 4 — Network Activity.+ classNetworkActivity = 4001+ classHTTPActivity = 4002+ classDNSActivity = 4003+ classEmailActivity = 4009+ classEmailURLActivity = 4012++ // Category 6 — Application Activity.+ classWebResourcesActivity = 6001+ classAPIActivity = 6003+ classDatastoreActivity = 6005+ classFileHostingActivity = 6006+)++// ocsfClass is one row of the published schema, carried here so the mapping+// below names a class rather than an integer and so a wrong caption cannot+// ship: the Go test asserts uid/1000 equals the declared category for every+// row and pins every caption against the schema.+type ocsfClass struct {+ uid int+ caption string+ category int+}++// ocsfClasses is the closed set. A mapping or a webhook template naming a uid+// absent from here fails the Go test and the gate, which is what stops an+// invented class uid from reaching the lake.+var ocsfClasses = map[int]ocsfClass{+ classFileSystemActivity: {classFileSystemActivity, "File System Activity", 1},+ classProcessActivity: {classProcessActivity, "Process Activity", 1},+ classSecurityFinding: {classSecurityFinding, "Security Finding", 2},+ classVulnerabilityFinding: {classVulnerabilityFinding, "Vulnerability Finding", 2},+ classAccountChange: {classAccountChange, "Account Change", 3},+ classAuthentication: {classAuthentication, "Authentication", 3},+ classUserAccessManagement: {classUserAccessManagement, "User Access Management", 3},+ classGroupManagement: {classGroupManagement, "Group Management", 3},+ classNetworkActivity: {classNetworkActivity, "Network Activity", 4},+ classHTTPActivity: {classHTTPActivity, "HTTP Activity", 4},+ classDNSActivity: {classDNSActivity, "DNS Activity", 4},+ classEmailActivity: {classEmailActivity, "Email Activity", 4},+ classEmailURLActivity: {classEmailURLActivity, "Email URL Activity", 4},+ classWebResourcesActivity: {classWebResourcesActivity, "Web Resources Activity", 6},+ classAPIActivity: {classAPIActivity, "API Activity", 6},+ classDatastoreActivity: {classDatastoreActivity, "Datastore Activity", 6},+ classFileHostingActivity: {classFileHostingActivity, "File Hosting Activity", 6},+}++// classesWithNoProducerYet records a declared class that nothing in the tree+// emits, and why it is declared ahead of its producer.+//+// All three are *per-event* classes inside streams this table can only+// describe per connector: a file syscall inside auditd's stream, a privilege+// grant inside an IdP's sign-in stream, a group membership change inside a+// directory's account stream. Choosing one of them as a whole connector's+// class would misdescribe the majority of that connector's records, which is+// the opposite of what this table is for. The event catalogue (depth plan+// 2.3) is the per-event discriminator that reaches them.+//+// Declaring them anyway is deliberate: they are the vocabulary the catalogue+// will map onto, verified against the schema now rather than invented later.+// `TestEveryDeclaredClassIsReachableOrRecorded` requires every declared class+// to be reached or listed here, and requires a listed class that *has*+// acquired a producer to be removed — so the list can only shrink.+var classesWithNoProducerYet = map[int]string{+ classFileSystemActivity: "a file syscall inside auditd's and osquery's streams, whose majority shape is process activity",+ classUserAccessManagement: "a privilege grant inside an IdP's stream, whose majority shape is authentication; " ++ "Vault's audit log is the nearest whole-connector fit and is a secrets datastore, not a privilege manager",+ classGroupManagement: "a group membership change inside a directory's stream, whose majority shape is account change",+}++// connectorClass is the decision recorded for one connector type: either the+// OCSF class its canonical events carry, or the reason it is left on the+// Security Finding default.+//+// `genericReason` is not an escape hatch for "nobody looked" — it is the+// opposite. A connector reaching the default silently is the state this table+// exists to end; one reaching it with a sentence saying why is a decision+// somebody can disagree with in review.+type connectorClass struct {+ classUID int+ genericReason string+}++// connectorOCSFClass carries a decision for every connector type+// services/connectors declares. Exactly one field per entry: the gate+// (`scripts/check_ocsf_class_coverage.py`) fails on a declared connector with+// neither, and on an entry carrying both.+var connectorOCSFClass = map[string]connectorClass{+ // ---------------------------------------------------------------+ // Category 3 — identity. Sign-ins and directory changes.+ // ---------------------------------------------------------------+ "okta": {classUID: classAuthentication},+ "azure_entra": {classUID: classAuthentication},+ "auth0": {classUID: classAuthentication},+ "duo_security": {classUID: classAuthentication},+ "onepassword": {classUID: classAuthentication},+ // JumpCloud's directory insights are account lifecycle first and+ // authentication second: its event_type vocabulary is dominated by user+ // and group create, update and delete, with `success` carrying the auth+ // outcome. Account Change is the superset those records fit.+ "jumpcloud": {classUID: classAccountChange},++ // ---------------------------------------------------------------+ // Category 4 — network and email telemetry. High volume, mostly benign.+ // ---------------------------------------------------------------+ // Umbrella reads /activity: one record per DNS or proxy transaction,+ // with `verdict` carrying the decision. At 2001 every permitted lookup+ // in the estate became an alert.+ "cisco_umbrella": {classUID: classDNSActivity},+ // Secure web gateway and WAF logs: one record per web request, with the+ // policy action in the record. Zscaler's connector asks for+ // `action: "any"`, so an Allow arrives alongside a Quarantine.+ "zscaler": {classUID: classHTTPActivity},+ "imperva": {classUID: classHTTPActivity},+ // Cloudflare Zero Trust ships two modes from one connector — WAF+ // firewall events and Access application-request decisions. Both are a+ // decision taken on an HTTP request reaching an application.+ "cloudflare_zt": {classUID: classHTTPActivity},+ // Flow records and network-monitoring telemetry.+ "aws_vpc_flow": {classUID: classNetworkActivity},+ "zeek_suricata": {classUID: classNetworkActivity},+ // Sublime reads /v1/messages, which is every message rather than only+ // the flagged ones, and defaults an unrecognised verdict to info.+ "sublime_security": {classUID: classEmailActivity},+ // Mimecast reads the TTP URL Protect log, which is one record per URL+ // click with a verdict — not a message stream, which is why it carries+ // the narrower class.+ "mimecast": {classUID: classEmailURLActivity},++ // ---------------------------------------------------------------+ // Category 1 — endpoint telemetry. The EDR products that return the+ // vendor's own detections are findings and are listed further down.+ // ---------------------------------------------------------------+ // auditd and the Windows Security channel are syscall and event-log+ // streams. Process creation is the dominant shape in both and the one+ // the detection corpus reads; their file-operation records are the+ // reason classFileSystemActivity is declared and not yet reached.+ "auditd": {classUID: classProcessActivity},+ "windows_event": {classUID: classProcessActivity},+ // osquery results through FleetDM and osctrl are scheduled-query+ // differentials spanning processes, files, packages and host state. The+ // security-relevant majority is process and file telemetry; the+ // promotion consequence is right for the whole stream either way,+ // because a routine inventory row should not be an alert.+ "fleetdm": {classUID: classProcessActivity},+ "osctrl": {classUID: classProcessActivity},++ // ---------------------------------------------------------------+ // Category 6 — application and data activity.+ // ---------------------------------------------------------------+ // Cloud control-plane audit logs: one API call per record.+ "gcp_cloud_audit": {classUID: classAPIActivity},+ "azure_activity": {classUID: classAPIActivity},+ "oci": {classUID: classAPIActivity},+ "kubernetes_audit": {classUID: classAPIActivity},+ // Cloudflare's connector reads /accounts/{id}/audit_logs — the account+ // control plane, not the HTTP request logs its vendor is better known+ // for. Reading the connector rather than the brand is what keeps this+ // out of HTTP Activity.+ "cloudflare": {classUID: classAPIActivity},+ // Tailscale's connector reads /tailnet/{t}/audit: ACL and device+ // administration, not packets.+ "tailscale": {classUID: classAPIActivity},+ // SaaS audit trails. Workspace and M365 carry sign-in, admin, file and+ // mail events in one stream; API Activity holds the whole stream+ // honestly rather than claiming a precision the per-event discriminator+ // does not yet exist for.+ "google_workspace": {classUID: classAPIActivity},+ "m365_audit": {classUID: classAPIActivity},+ "slack_audit": {classUID: classAPIActivity},+ "salesforce": {classUID: classAPIActivity},+ "servicenow": {classUID: classAPIActivity},+ "jira": {classUID: classAPIActivity},+ "confluence_audit": {classUID: classAPIActivity},+ // Source-control audit logs.+ "github": {classUID: classAPIActivity},+ "gitlab": {classUID: classAPIActivity},+ // Provider admin audit logs: who created a key, who changed a seat.+ "llm_usage": {classUID: classAPIActivity},+ // The schema's own description of File Hosting Activity names Box and+ // the services Dropbox competes with, which is why these two are not at+ // File System Activity: that class is a process acting on a local file.+ "box": {classUID: classFileHostingActivity},+ "dropbox": {classUID: classFileHostingActivity},+ // Warehouse query and login history.+ "snowflake": {classUID: classDatastoreActivity},+ // Vault's audit records are an operation against a secret path — a+ // read, write or delete on a datastore — not a privilege update, which+ // is what User Access Management describes.+ "vault": {classUID: classDatastoreActivity},+ // The AI gateway's runtime log is a request per model or tool call+ // against a served resource.+ "ai_gateway": {classUID: classWebResourcesActivity},++ // ---------------------------------------------------------------+ // Vulnerability findings — category 2, so still promoted+ // unconditionally, but a class the lake can tell apart from a+ // detection.+ // ---------------------------------------------------------------+ "qualys": {classUID: classVulnerabilityFinding},+ "tenable_io": {classUID: classVulnerabilityFinding},+ "snyk": {classUID: classVulnerabilityFinding},++ // ---------------------------------------------------------------+ // Deliberately 2001 Security Finding. Each returns the vendor's own+ // judged findings, so category 2 and unconditional promotion is the+ // correct outcome rather than an oversight. The reason is recorded so a+ // reader can tell these apart from a connector nobody classified.+ // ---------------------------------------------------------------+ "crowdstrike": {genericReason: "EDR detections: already judged by the vendor"},+ "sentinelone": {genericReason: "EDR threats: already judged by the vendor"},+ "carbon_black": {genericReason: "EDR alerts: already judged by the vendor"},+ "cortex_xdr": {genericReason: "XDR incidents: already judged by the vendor"},+ "trend_vision_one": {genericReason: "XDR workbench alerts: already judged by the vendor"},+ "azure_defender": {genericReason: "Defender alerts: already judged by the vendor"},+ "aws_guardduty": {genericReason: "GuardDuty findings: already judged by the vendor"},+ "aws_security_hub": {genericReason: "Security Hub findings: already judged by the vendor"},+ "gcp_scc": {genericReason: "Security Command Center findings: already judged by the vendor"},+ "wiz": {genericReason: "cloud security issues: already judged by the vendor"},+ "orca": {genericReason: "cloud security alerts: already judged by the vendor"},+ "lacework": {genericReason: "cloud security events: already judged by the vendor"},+ "prisma_cloud": {genericReason: "cloud security alerts: already judged by the vendor"},+ "sysdig": {genericReason: "runtime security events: already judged by the vendor"},+ "darktrace": {genericReason: "NDR model breaches: already judged by the vendor"},+ "greynoise": {genericReason: "threat-intel verdicts on an address already observed"},+ "wazuh": {genericReason: "agent alerts above a rule-level threshold: judged by the vendor's ruleset"},+ // Falco's connector drops anything below the configured priority, so+ // what reaches ingest is a rule hit the operator asked to see.+ "falco": {genericReason: "rule hits above a configured priority: the rule is the judgement"},+ // Netskope's connector reads /api/v2/events/data/alert, not the web+ // traffic log, so what arrives is already an alert.+ "netskope": {genericReason: "CASB alerts, not the web traffic log: already judged by the vendor"},+ // Proofpoint's connector reads /v2/siem/messages/blocked and Abnormal's+ // reads /v1/threats and /v1/cases — the judged half of each product.+ "proofpoint": {genericReason: "messages the gateway blocked: already judged by the vendor"},+ "abnormal_security": {genericReason: "threats and cases, not the message stream: already judged by the vendor"},+ // A forwarded or reported message carries a human's judgement, which is+ // the whole reason it was forwarded.+ "email_inbox": {genericReason: "messages a person reported as suspicious: the report is the judgement"},+ // The connector ships a curated allow-list of security-relevant API+ // calls, so the selection has already happened upstream of ingest.+ "aws_cloudtrail": {genericReason: "a curated allow-list of security-relevant API calls: the connector has already decided"},+ "opsgenie": {genericReason: "alerts raised by another system and forwarded"},+ "pagerduty": {genericReason: "incidents raised by another system and forwarded"},+ "tines": {genericReason: "SOAR story results: already judged by the sending workflow"},+ "torq": {genericReason: "SOAR workflow results: already judged by the sending workflow"},+ // CEF and LEEF carry the sending device's own class and severity. One+ // class for every CEF sender would be a claim about devices this+ // connector cannot see.+ "syslog_cef": {genericReason: "CEF and LEEF carry the sending device's own class, which this connector cannot narrow"},++ // SIEMs forward notables and correlation-rule hits, which are findings+ // by construction: the search that produced them is the judgement.+ "splunk": {genericReason: "SIEM notables: the correlation search is the judgement"},+ "microsoft_sentinel": {genericReason: "SIEM incidents: the analytics rule is the judgement"},+ "qradar": {genericReason: "SIEM offenses: the rule chain is the judgement"},+ "elastic": {genericReason: "SIEM detection alerts: the rule is the judgement"},+ "chronicle": {genericReason: "SIEM detections: the rule is the judgement"},+ "cortex_xsiam": {genericReason: "SIEM incidents: the correlation is the judgement"},+ "exabeam": {genericReason: "SIEM notable sessions: the model is the judgement"},+ "securonix": {genericReason: "SIEM violations: the policy is the judgement"},+ "devo": {genericReason: "SIEM alerts: the rule is the judgement"},+ "sumo_logic": {genericReason: "SIEM signals: the rule is the judgement"},+ "rapid7_insightidr": {genericReason: "SIEM investigations: the detection rule is the judgement"},+ "trellix_helix": {genericReason: "SIEM alerts: the rule is the judgement"},+ "datadog": {genericReason: "monitor alerts: the monitor is the judgement"},+ "datadog_cloud_siem": {genericReason: "Cloud SIEM signals: the rule is the judgement"},+}++// ocsfClassForConnector returns the class a connector type's canonical events+// carry, and whether the table names one. A connector with a recorded+// `genericReason`, or no entry at all, returns false and keeps the default.+func ocsfClassForConnector(connectorType string) (ocsfClass, bool) {+ entry, ok := connectorOCSFClass[connectorType]+ if !ok || entry.classUID == 0 {+ return ocsfClass{}, false+ }+ class, known := ocsfClasses[entry.classUID]+ return class, known+}diff --git a/services/ingest/internal/normalizer/ocsf_classes_test.go b/services/ingest/internal/normalizer/ocsf_classes_test.gonew file mode 100644index 000000000..c0fdc0272--- /dev/null+++ b/services/ingest/internal/normalizer/ocsf_classes_test.go@@ -0,0 +1,378 @@+package normalizer++import (+ "os"+ "path/filepath"+ "regexp"+ "strconv"+ "testing"+)++// promotable mirrors should_promote() in+// services/fusion/app/services/promoter.py: OCSF category 2 (Findings) is+// promoted unconditionally, and anything else needs severity_id >= 4.+//+// Duplicated here rather than shared because the two services do not share a+// runtime — the same reasoning the `findingsCategory` constants in+// normalizer.go already carry, and TestUnpromotableWarningMatchesFusionPolicy+// is what keeps the pair honest.+func promotable(ocsf map[string]interface{}) bool {+ if category, ok := ocsf["category_uid"].(int); ok && category == findingsCategory {+ return true+ }+ severity, ok := ocsf["severity_id"].(int)+ return ok && severity >= promoteSeverityFloor+}++func normalizeCanonical(t *testing.T, connectorType string, payload map[string]interface{}) map[string]interface{} {+ t.Helper()+ full := map[string]interface{}{"source": connectorType, "raw_event": map[string]interface{}{"id": "1"}}+ for k, v := range payload {+ full[k] = v+ }+ ev, err := newLenientNormalizer().Normalize(&RawEvent{+ ConnectorID: "c-1",+ ConnectorType: connectorType,+ TenantID: "11111111-1111-1111-1111-111111111111",+ ReceivedAt: "2026-10-07T00:00:00Z",+ Payload: full,+ })+ if err != nil {+ t.Fatalf("Normalize(%q): %v", connectorType, err)+ }+ return ev.OcsfEvent+}++// The defect the class table exists for, stated as the behaviour that must+// now hold. On the pre-fix tree every one of these connectors reached the+// 2001 Security Finding default inside canonicalProfile(), which is category+// 2 and therefore promoted unconditionally — so a permitted DNS lookup, an+// allowed proxy request, an accepted VPC flow record and a routine Windows+// event each became an alert carrying severity "info".+//+// This test fails on that tree, by design: it is the negative control for the+// mapping as much as the assertion for it.+func TestRoutineTelemetryIsNotPromotedAsAFinding(t *testing.T) {+ // Each case is the shape the named connector's own normalize() emits for+ // its most common, least interesting record. Severity strings are the+ // ones those connectors actually produce for a benign record: Umbrella's+ // non-blocked verdict, Zscaler's Allow, a parsed ACCEPT flow line, a+ // Windows event id outside _HIGH_EVENT_IDS.+ cases := []struct {+ connector string+ why string+ }{+ {"cisco_umbrella", "one record per DNS lookup, verdict not blocked"},+ {"zscaler", "one record per web request, action Allow"},+ {"aws_vpc_flow", "one record per accepted flow"},+ {"zeek_suricata", "network session telemetry"},+ {"windows_event", "a Windows event id outside the high-risk set"},+ {"auditd", "a syscall record with no matching audit key"},+ {"google_workspace", "a routine Workspace activity"},+ {"m365_audit", "a routine M365 audit record"},+ {"snowflake", "a completed warehouse query"},+ {"github", "a routine organisation audit entry"},+ {"box", "a routine file view in a file-hosting account"},+ {"vault", "a secret read against a path"},+ {"cloudflare", "a routine account audit entry"},+ {"mimecast", "a URL click the gateway did not block"},+ }+ for _, tc := range cases {+ t.Run(tc.connector, func(t *testing.T) {+ ocsf := normalizeCanonical(t, tc.connector, map[string]interface{}{+ "severity": "info",+ "title": "routine activity",+ })+ if promotable(ocsf) {+ t.Errorf(+ "an info-severity event from %q (%s) is promotable: class_uid=%v category_uid=%v severity_id=%v. "++ "Every record this source emits would become an alert",+ tc.connector, tc.why, ocsf["class_uid"], ocsf["category_uid"], ocsf["severity_id"],+ )+ }+ })+ }+}++// The other half, and the reason the mapping is per connector rather than a+// blanket demotion: a source whose fetch_alerts returns the vendor's own+// judged findings must still promote on class, so a Medium EDR detection or a+// Medium SIEM notable is never silently dropped.+func TestVendorFindingsStillPromoteOnClass(t *testing.T) {+ for _, connector := range []string{+ "crowdstrike", "sentinelone", "carbon_black", "aws_guardduty", "wiz",+ "splunk", "qradar", "elastic", "microsoft_sentinel", "aws_cloudtrail",+ // Verified against the endpoint each connector reads: an alert+ // stream, a blocked-message stream and a priority-filtered rule+ // stream are judgements, and demoting them would have silenced a+ // medium-severity vendor finding.+ "netskope", "proofpoint", "abnormal_security", "falco", "email_inbox",+ } {+ t.Run(connector, func(t *testing.T) {+ ocsf := normalizeCanonical(t, connector, map[string]interface{}{+ "severity": "medium",+ "title": "vendor finding",+ })+ if !promotable(ocsf) {+ t.Errorf("a medium finding from %q is not promotable: class_uid=%v severity_id=%v",+ connector, ocsf["class_uid"], ocsf["severity_id"])+ }+ })+ }+}++// The gate counts a connector with a recorded reason as "on the generic+// mapping", and that count is the figure Phase 2 is graded against. It only+// means what it says if a recorded reason really does resolve to 2001 — a+// reason sitting beside a profile that moved the class would make the number+// describe the table rather than the events.+func TestARecordedReasonResolvesToSecurityFinding(t *testing.T) {+ for connector, entry := range connectorOCSFClass {+ if entry.genericReason == "" {+ continue+ }+ t.Run(connector, func(t *testing.T) {+ ocsf := normalizeCanonical(t, connector, map[string]interface{}{+ "severity": "medium",+ "title": "vendor finding",+ })+ if ocsf["class_uid"] != classSecurityFinding {+ t.Errorf("%q records a reason for staying on the Security Finding default but resolves to "++ "class_uid %v; the generic count would describe the table rather than the events",+ connector, ocsf["class_uid"])+ }+ })+ }+}++// Demoting a telemetry source out of category 2 only works if its severe+// events still get through on the other branch. A class with a severity map+// that cannot reach the floor is the `splunk_enterprise` defect wearing a new+// class uid: archived to the lake, never an alert, silently.+func TestMappedTelemetryClassesStillPromoteOnSeverity(t *testing.T) {+ for connector, entry := range connectorOCSFClass {+ if entry.classUID == 0 || entry.classUID/1000 == findingsCategory {+ continue+ }+ t.Run(connector, func(t *testing.T) {+ for _, severity := range []string{"high", "critical"} {+ ocsf := normalizeCanonical(t, connector, map[string]interface{}{+ "severity": severity,+ "title": "something worth a human",+ })+ if !promotable(ocsf) {+ t.Errorf("a %s event from %q cannot promote: class_uid=%v severity_id=%v",+ severity, connector, ocsf["class_uid"], ocsf["severity_id"])+ }+ }+ })+ }+}++// Every connector the registry declares carries a decision. The registry is+// read off disk rather than listed here, because a hardcoded list drifts from+// it and then agrees with itself forever — the failure this whole area keeps+// producing.+func TestEveryDeclaredConnectorHasAnOCSFDecision(t *testing.T) {+ for id := range declaredConnectorIDs(t) {+ entry, ok := connectorOCSFClass[id]+ if !ok {+ t.Errorf("connector %q has no entry in connectorOCSFClass: its events take the 2001 default, "++ "which is always promoted, and nothing records that as a decision", id)+ continue+ }+ if entry.classUID == 0 && entry.genericReason == "" {+ t.Errorf("connector %q has an entry carrying neither a class nor a reason", id)+ }+ if entry.classUID != 0 && entry.genericReason != "" {+ t.Errorf("connector %q declares both a class and a generic reason; one of them is not what happens", id)+ }+ }+}++// A class uid whose declared category disagrees with uid/1000 would change+// which promotion branch a connector's events take without anyone editing the+// mapping, because fusion derives the category from the uid alone.+func TestOCSFClassCategoryIsDerivedFromItsUID(t *testing.T) {+ if len(ocsfClasses) == 0 {+ t.Fatal("ocsfClasses is empty — refusing to pass on an empty table")+ }+ for uid, class := range ocsfClasses {+ if class.uid != uid {+ t.Errorf("ocsfClasses[%d] carries uid %d; a row must describe its own key", uid, class.uid)+ }+ if uid/1000 != class.category {+ t.Errorf("class %d (%s) declares category %d, but uid/1000 is %d",+ uid, class.caption, class.category, uid/1000)+ }+ }+}++// Nothing may map to a uid outside the closed set. Inventing one is how an+// event reaches the lake carrying a class no schema defines.+func TestConnectorClassesNameAKnownClass(t *testing.T) {+ for connector, entry := range connectorOCSFClass {+ if entry.classUID == 0 {+ continue+ }+ if _, ok := ocsfClasses[entry.classUID]; !ok {+ t.Errorf("connector %q maps to class uid %d, which ocsfClasses does not declare", connector, entry.classUID)+ }+ }+}++// The uids and captions above were read from schema.ocsf.io before use. This+// pins the pairing so a later edit cannot quietly rename a class into one the+// schema spells differently — the caption travels into `class_name` on every+// event and into the lake.+func TestClassCaptionsMatchThePublishedSchema(t *testing.T) {+ published := map[int]string{+ 1001: "File System Activity",+ 1007: "Process Activity",+ 2001: "Security Finding",+ 2002: "Vulnerability Finding",+ 3001: "Account Change",+ 3002: "Authentication",+ 3005: "User Access Management",+ 3006: "Group Management",+ 4001: "Network Activity",+ 4002: "HTTP Activity",+ 4003: "DNS Activity",+ 4009: "Email Activity",+ 4012: "Email URL Activity",+ 6001: "Web Resources Activity",+ 6003: "API Activity",+ 6005: "Datastore Activity",+ 6006: "File Hosting Activity",+ }+ for uid, caption := range published {+ class, ok := ocsfClasses[uid]+ if !ok {+ t.Errorf("class %d (%s) is no longer declared", uid, caption)+ continue+ }+ if class.caption != caption {+ t.Errorf("class %d: caption %q, schema says %q", uid, class.caption, caption)+ }+ }+ for uid := range ocsfClasses {+ if _, ok := published[uid]; !ok {+ t.Errorf("class %d is declared but was not checked against the published schema; "++ "add it to this table only after reading schema.ocsf.io", uid)+ }+ }+}++// A declared class nothing emits is the "exists but nothing calls it" shape+// one layer down: a reader sees the vocabulary and assumes the lake contains+// it. Every class must therefore be reached by a connector mapping, a+// hand-written profile or a webhook template, or be listed in+// `classesWithNoProducerYet` with the reason.+//+// The list is a ratchet in both directions. A class with no producer and no+// entry fails, and so does an entry for a class that has since acquired one —+// otherwise the record outlives the gap it describes, which is how a caveat+// becomes a lie.+func TestEveryDeclaredClassIsReachableOrRecorded(t *testing.T) {+ produced := map[int]string{}+ for connector, entry := range connectorOCSFClass {+ if entry.classUID != 0 {+ produced[entry.classUID] = "connector " + connector+ }+ }+ for key, profile := range connectorProfiles {+ produced[profile.classUID] = "profile " + key+ }+ for name, uid := range templateClassUIDs(t) {+ produced[uid] = "template " + name+ }++ for uid, class := range ocsfClasses {+ by, reached := produced[uid]+ reason, recorded := classesWithNoProducerYet[uid]+ switch {+ case reached && recorded:+ t.Errorf("class %d (%s) is produced by %s but is still listed as having no producer (%q); "++ "remove the entry so the list keeps shrinking", uid, class.caption, by, reason)+ case !reached && !recorded:+ t.Errorf("class %d (%s) is declared and nothing emits it; map a connector to it or record why "++ "it is declared ahead of its producer", uid, class.caption)+ case !reached && len(reason) < 30:+ t.Errorf("class %d (%s) is recorded as having no producer for the reason %q, which does not say "++ "enough for a reviewer to disagree with it", uid, class.caption, reason)+ }+ }++ for uid := range classesWithNoProducerYet {+ if _, declared := ocsfClasses[uid]; !declared {+ t.Errorf("classesWithNoProducerYet names class %d, which ocsfClasses does not declare", uid)+ }+ }+}++// Webhook templates are the second producer of class uids — the push path to+// the connectors' pull path — and nothing checked them against the closed+// set. A template naming a uid no schema defines reaches the lake just as+// easily as a connector would.+func TestWebhookTemplatesNameAKnownClass(t *testing.T) {+ for name, uid := range templateClassUIDs(t) {+ if _, ok := ocsfClasses[uid]; !ok {+ t.Errorf("template %s declares class_uid %d, which ocsfClasses does not declare", name, uid)+ }+ }+}++// templateClassUIDs reads the class each webhook template stamps, off disk.+// The templates are embedded into the binary from this directory, so reading+// them here is reading what ships.+func templateClassUIDs(t *testing.T) map[string]int {+ t.Helper()+ entries, err := os.ReadDir("templates")+ if err != nil {+ t.Fatalf("cannot read templates: %v", err)+ }+ classUIDRe := regexp.MustCompile(`(?m)^class_uid:\s*(\d+)`)+ out := map[string]int{}+ for _, entry := range entries {+ if entry.IsDir() || filepath.Ext(entry.Name()) != ".yaml" {+ continue+ }+ src, err := os.ReadFile(filepath.Join("templates", entry.Name()))+ if err != nil {+ t.Fatalf("cannot read template %s: %v", entry.Name(), err)+ }+ if m := classUIDRe.FindSubmatch(src); m != nil {+ uid, convErr := strconv.Atoi(string(m[1]))+ if convErr != nil {+ t.Fatalf("template %s: class_uid %q is not an integer", entry.Name(), m[1])+ }+ out[entry.Name()] = uid+ }+ }+ if len(out) == 0 {+ t.Fatal("parsed zero template class uids — refusing to pass on an empty read")+ }+ return out+}++// A connector with a hand-written profile takes that profile's class, not the+// table's. Declaring a class for one would be a decision nothing acts on, and+// a reader would believe it.+func TestClassTableDoesNotContradictAHandWrittenProfile(t *testing.T) {+ for connector, entry := range connectorOCSFClass {+ if entry.classUID == 0 {+ continue+ }+ profile, hasProfile := connectorProfiles[connector]+ if !hasProfile {+ if aliased, isAlias := connectorTypeAliases[connector]; isAlias {+ profile, hasProfile = connectorProfiles[aliased]+ }+ }+ if hasProfile && profile.classUID != entry.classUID {+ t.Errorf("connector %q maps to class %d in the table but its profile declares %d; "++ "a flat payload takes the profile and a canonical envelope takes the table, so the two must agree",+ connector, entry.classUID, profile.classUID)+ }+ }+}