Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

Give every connector type a deliberate OCSF class, so routine telemetry is not promoted as a vendor finding (#1195)

Beenu Aroraf9ecdbf72264
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index f00adb24b..e827e755b 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -649,6 +649,16 @@ jobs:
run: python3 scripts/check_connector_profiles.py --self-test
- name: Connector profile drift gate
run: python3 scripts/check_connector_profiles.py
+ # One layer in from the name: the profile gate asks whether a connector
+ # reaches a normalization path, this asks which OCSF class it reaches
+ # when it gets there. The class uid divided by a thousand is the
+ # category, and fusion promotes category 2 unconditionally — so 78 of
+ # the 84 connectors landing on the 2001 default meant every permitted
+ # DNS lookup and every routine Windows event became an alert.
+ - name: OCSF class coverage gate — self-test
+ run: python3 scripts/check_ocsf_class_coverage.py --self-test
+ - name: OCSF class coverage gate
+ run: python3 scripts/check_ocsf_class_coverage.py --check
# One layer in again: a connector's events reach a profile, and the
# catalogue says what each of its event types *means*. Grown from the
# fixtures, so adding a fixture that carries a new event type fails
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 325125314..d90385114 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -234,6 +234,63 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
both classified and excused, and any drift between `schemas/` and the
vendored copy the binary carries.
+### Fixed
+
+- **Routine telemetry was promoted to an alert as though a vendor had judged
+ it.** Ingest mapped 78 of the 84 declared connector types onto OCSF class
+ 2001 Security Finding, because `canonicalProfile()` used it as the default
+ and only five connectors had an override. 2001 is category 2, and
+ `should_promote()` promotes category 2 unconditionally — so every permitted
+ DNS lookup from Cisco Umbrella, every `Allow` from Zscaler, every accepted
+ VPC flow record, every routine Windows event and every Workspace or M365
+ audit entry became an alert carrying severity `info`. Measured on the tree
+ before the fix: **14 of 14** telemetry sources promoted an info-severity
+ event.
+
+ `services/ingest/internal/normalizer/ocsf_classes.go` now carries a decision
+ for every connector type the registry declares: the OCSF class its stream
+ actually is, or the reason it stays on the Security Finding default. Which
+ one each source gets was read from the endpoint its connector calls rather
+ than from the vendor's product category, and the two disagree often enough
+ that guessing would have silenced four sources — Netskope reads
+ `/api/v2/events/data/alert`, Proofpoint reads `/v2/siem/messages/blocked`,
+ Abnormal reads `/v1/threats`, and Falco drops anything below its configured
+ priority, so all four return judgements and keep category 2.
+
+ Ten classes are added, each uid, caption and category read from
+ `schema.ocsf.io` for both the 1.9.0 the tree targets and the 1.1.0 the
+ normalizer's `metadata.version` declares, because emitting a newer class
+ under the older version string would be a false claim about a public
+ standard: Process Activity, File System Activity, DNS Activity, HTTP
+ Activity, Email Activity, Account Change, User Access Management, Group
+ Management, Web Resources Activity and Datastore Activity, plus Email URL
+ Activity and File Hosting Activity where the schema's own description named
+ the product. Connector types on the generic mapping: **78 → 42**, all 42
+ with a recorded reason and none without a decision. (depth plan 2.1)
+
+- **A Sentinel incident was labelled a vulnerability finding.** The
+ `microsoft_sentinel` profile declared `class_uid: 2002` beside
+ `class_name: "Security Finding"`; 2002 is Vulnerability Finding in every
+ OCSF version this service targets. Both are category 2, so promotion hid it,
+ but every Sentinel event in the lake disagreed with itself and a query
+ filtering on 2002 returned incidents alongside Qualys and Tenable findings.
+ Found by the new gate, which fails on a profile whose class name contradicts
+ its own uid.
+
+### Added
+
+- `scripts/check_ocsf_class_coverage.py`, wired into `ci.yml` with its
+ self-test ahead of it. A declared connector with neither a class nor a
+ recorded reason fails, as does an entry carrying both, a class uid whose
+ declared category disagrees with `uid/1000`, a non-finding class whose
+ severity map cannot reach the promote floor of 4, and a profile whose class
+ name contradicts its uid. It reports the number of connector types on the
+ generic mapping and holds it at a ceiling that may shrink but not grow. The
+ self-test injects each of those eleven defects and requires the gate to
+ catch each one, and requires an empty tree to be refused rather than
+ reported clean.
+
+
## [17.1.0] - 2026-10-07
diff --git a/scripts/check_ocsf_class_coverage.py b/scripts/check_ocsf_class_coverage.py
new file mode 100644
index 000000000..7ebb09032
--- /dev/null
+++ b/scripts/check_ocsf_class_coverage.py
@@ -0,0 +1,600 @@
+#!/usr/bin/env python3
+"""Every connector type names an OCSF class, or says why it does not.
+
+Why this exists
+---------------
+``should_promote()`` in ``services/fusion/app/services/promoter.py`` promotes
+an event to an alert when it is OCSF **category 2** (Findings) *or* its
+``severity_id`` is at least 4. The category is the class uid divided by a
+thousand, so the class a connector's events carry decides whether its routine
+telemetry reaches an analyst's queue or stays in the lake.
+
+Before the table this gate reads, 72 of the 84 declared connector types had no
+class of their own. They fell to the ``2001`` Security Finding default inside
+``canonicalProfile()`` — category 2, therefore **always promoted** — and that
+default was invisible: nothing named it, nothing counted it, and no reviewer
+saw a list of the sources it covered.
+
+For the connectors whose ``fetch_alerts`` really does return the vendor's own
+findings, 2001 is right. For the ones returning raw telemetry it meant every
+permitted DNS lookup, every allowed proxy request, every accepted VPC flow
+record and every routine Windows event became an alert carrying severity
+``info`` — fusion's alert-reduction property inverted, for the sources that
+emit the most volume. Reproduced on the pre-fix tree by
+``TestRoutineTelemetryWasPromotedBeforeTheClassTable`` in the normalizer's own
+suite.
+
+What it checks
+--------------
+Two directions, because a one-directional check is this repository's dominant
+gate defect — it compares A against B, never B against A, so drift in the
+direction things actually change slips past while the gate prints OK.
+
+ PY -> GO every ``connector_id`` the connectors service declares has
+ either a class mapping or a recorded reason it stays on the
+ default. Neither is the failure this gate exists for.
+ GO -> PY every entry in the Go table names a connector that exists, no
+ entry carries both a class and a reason, and no entry names a
+ class uid absent from the service's own closed set.
+ SCHEMA every class in that closed set declares the category its uid
+ implies, so a typo cannot change which promotion branch a
+ connector's events take.
+ PROMOTION every class a connector is mapped to can actually promote: it
+ is category 2, or the profile carrying it ships a severity map
+ that reaches the floor of 4. A new class with an empty severity
+ map is archived to the lake and never alerts, silently, which is
+ exactly how the borrowed ``splunk_enterprise`` profile broke 26
+ connectors.
+
+It also reports the **count of connector types on the generic mapping**, which
+is the figure the depth plan's Phase 2 is graded against, and holds it with
+``--check`` against the committed number so it can only go down by a decision
+somebody made rather than drift up by accident.
+
+Usage
+-----
+ python3 scripts/check_ocsf_class_coverage.py # gate
+ python3 scripts/check_ocsf_class_coverage.py --check # + drift on the count
+ python3 scripts/check_ocsf_class_coverage.py --json
+ python3 scripts/check_ocsf_class_coverage.py --self-test # prove it bites
+
+``--repo-root`` overrides the tree under inspection. The resolved root, every
+file read and every count are printed before the verdict, and an empty read is
+a hard error rather than a quiet zero.
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import json
+import re
+import sys
+from pathlib import Path
+
+# `scripts/` is on sys.path when this runs as a program but not when a test
+# loads it by path with importlib. gate_toolkit sits beside it either way.
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+from gate_toolkit import repo_root
+
+CLASSES_REL = Path("services/ingest/internal/normalizer/ocsf_classes.go")
+NORMALIZER_REL = Path("services/ingest/internal/normalizer/normalizer.go")
+CONNECTORS_REL = Path("services/connectors/app/connectors")
+
+#: Connector types on the generic mapping, as committed. The figure the depth
+#: plan grades Phase 2 against, so it lives in code with a drift gate rather
+#: than in prose that goes stale silently.
+#:
+#: 72 at capture (commit ee33b02f): 84 declared, and only 12 with a class of
+#: their own — 8 through a hand-written profile and 4 more through the
+#: five-entry override map this table replaced. It may fall as connectors are
+#: classified. It may **not** rise: a new connector arrives with a decision or
+#: the gate fails, which is the whole point of counting it.
+#:
+#: The number that must stay at zero is `unclassified`, which is a hard
+#: failure rather than a ceiling: every one of the 42 below carries a reason.
+MAX_GENERIC = 42
+
+#: OCSF categories whose events fusion promotes unconditionally.
+FINDINGS_CATEGORY = 2
+
+#: The severity_id floor the promoter applies to everything else.
+PROMOTE_SEVERITY_FLOOR = 4
+
+
+class GateError(RuntimeError):
+ """An input could not be read. Never downgraded to a passing result."""
+
+
+# --------------------------------------------------------------------------
+# Go side
+# --------------------------------------------------------------------------
+def _go_block(src: str, header: str) -> str:
+ """The brace-balanced literal following `header`."""
+ start = src.find(header)
+ if start == -1:
+ raise GateError(f"could not find `{header}`")
+ depth, i = 0, src.index("{", start)
+ for j in range(i, len(src)):
+ if src[j] == "{":
+ depth += 1
+ elif src[j] == "}":
+ depth -= 1
+ if depth == 0:
+ return src[i : j + 1]
+ raise GateError(f"unbalanced braces after `{header}`")
+
+
+def parse_class_consts(src: str) -> dict[str, int]:
+ """Go constant name -> uid, from the const block."""
+ return {name: int(uid) for name, uid in re.findall(r"^\t(class\w+)\s*=\s*(\d+)$", src, re.MULTILINE)}
+
+
+def parse_classes(src: str, consts: dict[str, int]) -> dict[int, tuple[str, int]]:
+ """uid -> (caption, declared category), from `ocsfClasses`."""
+ block = _go_block(src, "var ocsfClasses = map[int]ocsfClass{")
+ out: dict[int, tuple[str, int]] = {}
+ for key, uid_ref, caption, category in re.findall(r'^\t(class\w+):\s*\{(class\w+),\s*"([^"]+)",\s*(\d+)\},?$', block, re.MULTILINE):
+ if key not in consts:
+ raise GateError(f"ocsfClasses keys off {key!r}, which the const block does not declare")
+ if uid_ref != key:
+ raise GateError(f"ocsfClasses row {key!r} carries uid {uid_ref!r}; a row must describe its own key")
+ out[consts[key]] = (caption, int(category))
+ return out
+
+
+def parse_connector_classes(src: str, consts: dict[str, int]) -> dict[str, dict]:
+ """connector type -> {"class_uid": int|None, "reason": str|None}."""
+ block = _go_block(src, "var connectorOCSFClass = map[string]connectorClass{")
+ out: dict[str, dict] = {}
+ for name, body in re.findall(r'^\t"([a-z0-9_]+)":\s*\{([^}]*)\},?$', block, re.MULTILINE):
+ class_match = re.search(r"classUID:\s*(class\w+)", body)
+ reason_match = re.search(r'genericReason:\s*"([^"]*)"', body)
+ class_uid = None
+ if class_match:
+ key = class_match.group(1)
+ if key not in consts:
+ raise GateError(f"connector {name!r} names class constant {key!r}, which is not declared")
+ class_uid = consts[key]
+ out[name] = {"class_uid": class_uid, "reason": reason_match.group(1) if reason_match else None}
+ return out
+
+
+def parse_profile_classes(src: str) -> dict[str, dict]:
+ """profile key -> {"class_uid", "class_name", "severity_promotes"}.
+
+ ``severity_promotes`` answers the question the promoter asks: can any
+ value in this profile's severity map reach the floor of 4? A profile whose
+ map is empty, or tops out at 3, cannot promote a non-finding event however
+ severe the vendor thought it was. Shared-ladder references are resolved
+ rather than treated as unknown, because the whole point of naming the
+ shared ladder was that a reader can see what it contains.
+ """
+ block = _go_block(src, "var connectorProfiles = map[string]connectorProfile{")
+ shared = {
+ name: [int(v) for v in re.findall(r":\s*(\d+)", _go_block(src, f"var {name} = map[string]int{{"))]
+ for name in re.findall(r"var (_\w*[Ss]everityMap) = map\[string\]int\{", src)
+ }
+ out: dict[str, dict] = {}
+ for match in re.finditer(r'^\t"([a-z0-9_]+)":\s*\{$', block, re.MULTILINE):
+ key = match.group(1)
+ entry = _go_block(block, match.group(0))
+ class_match = re.search(r"classUID:\s*(\d+)", entry)
+ if not class_match:
+ raise GateError(f"profile {key!r} declares no classUID")
+ name_match = re.search(r'className:\s*"([^"]+)"', entry)
+ ref = re.search(r"severityMap:\s*(_\w+),", entry)
+ if ref:
+ values = shared.get(ref.group(1))
+ if values is None:
+ raise GateError(f"profile {key!r} references severity map {ref.group(1)!r}, which is not declared")
+ else:
+ sev_block = re.search(r"severityMap:\s*map\[string\]int\{(.*?)\}", entry, re.DOTALL)
+ values = [int(v) for v in re.findall(r":\s*(\d+)", sev_block.group(1))] if sev_block else []
+ out[key] = {
+ "class_uid": int(class_match.group(1)),
+ "class_name": name_match.group(1) if name_match else None,
+ "severity_promotes": any(v >= PROMOTE_SEVERITY_FLOOR for v in values),
+ }
+ return out
+
+
+def parse_canonical_severity_promotes(src: str) -> bool:
+ """Whether the ladder every canonical envelope uses can reach the floor."""
+ block = _go_block(src, "var _canonicalSeverityMap = map[string]int{")
+ return any(int(v) >= PROMOTE_SEVERITY_FLOOR for v in re.findall(r":\s*(\d+)", block))
+
+
+# --------------------------------------------------------------------------
+# Python side
+# --------------------------------------------------------------------------
+def parse_declared_connectors(directory: Path) -> set[str]:
+ """Every `connector_id` the connectors service declares."""
+ out: set[str] = set()
+ for path in sorted(directory.glob("*.py")):
+ if path.name in {"__init__.py", "base.py"}:
+ continue
+ tree = ast.parse(path.read_text(encoding="utf-8"))
+ for cls in (n for n in ast.walk(tree) if isinstance(n, ast.ClassDef)):
+ for stmt in cls.body:
+ # The value is bound in each branch rather than read off
+ # `stmt` afterwards: past the if/elif the name is still an
+ # `ast.stmt`, which has no `.value`, so the later read was
+ # four unchecked attribute accesses that happened to be safe.
+ if isinstance(stmt, ast.Assign):
+ target = next((getattr(t, "id", None) for t in stmt.targets), None)
+ value: ast.expr | None = stmt.value
+ elif isinstance(stmt, ast.AnnAssign):
+ target = getattr(stmt.target, "id", None)
+ value = stmt.value
+ else:
+ continue
+ if target == "connector_id" and isinstance(value, ast.Constant) and isinstance(value.value, str) and value.value:
+ out.add(value.value)
+ return out
+
+
+# --------------------------------------------------------------------------
+# Gate
+# --------------------------------------------------------------------------
+def evaluate(
+ declared: set[str],
+ classes: dict[int, tuple[str, int]],
+ connector_classes: dict[str, dict],
+ profiles: dict[str, dict],
+ canonical_severity_promotes: bool,
+ max_generic: int,
+ check_count: bool,
+) -> tuple[list[tuple[str, str]], dict]:
+ failures: list[tuple[str, str]] = []
+
+ # SCHEMA. A uid whose declared category disagrees with uid/1000 would
+ # change which promotion branch a connector's events take.
+ for uid, (caption, category) in sorted(classes.items()):
+ if uid // 1000 != category:
+ failures.append(
+ (
+ "class-category-mismatch",
+ f"class {uid} ({caption}) declares category {category}, but uid/1000 is {uid // 1000}; "
+ "fusion derives the category from the uid, so the two cannot differ",
+ )
+ )
+
+ # GO -> PY. Every entry names something real and says exactly one thing.
+ for name, entry in sorted(connector_classes.items()):
+ if name not in declared:
+ failures.append(("class-names-nothing", f"connectorOCSFClass names {name!r}, which no connector declares"))
+ if entry["class_uid"] is None and not entry["reason"]:
+ failures.append(("class-entry-empty", f"{name!r} has an entry carrying neither a class nor a reason"))
+ if entry["class_uid"] is not None and entry["reason"]:
+ failures.append(
+ (
+ "class-entry-ambiguous",
+ f"{name!r} declares both a class and a generic reason; one of them is not what happens",
+ )
+ )
+ if entry["class_uid"] is not None and entry["class_uid"] not in classes:
+ failures.append(
+ (
+ "class-uid-unknown",
+ f"{name!r} maps to class uid {entry['class_uid']}, which is not in the service's closed set",
+ )
+ )
+ if entry["reason"] and len(entry["reason"]) < 15:
+ failures.append(
+ (
+ "class-reason-thin",
+ f"{name!r} stays generic for the reason {entry['reason']!r}, which does not say enough "
+ "for a reviewer to disagree with it",
+ )
+ )
+
+ # PY -> GO. The direction the defect lived in: a declared connector with
+ # no decision at all, reaching the default silently.
+ generic: list[str] = []
+ for name in sorted(declared):
+ decision = connector_classes.get(name)
+ if decision is None:
+ failures.append(
+ (
+ "connector-unclassified",
+ f"connector {name!r} has no OCSF class and no recorded reason: its events take the 2001 "
+ "Security Finding default, which is category 2 and therefore always promoted, and nothing "
+ "says that was a decision",
+ )
+ )
+ generic.append(name)
+ elif decision["class_uid"] is None:
+ generic.append(name)
+
+ # PROMOTION. A class a connector is mapped to must be able to promote.
+ for name, entry in sorted(connector_classes.items()):
+ uid = entry["class_uid"]
+ if uid is None or uid not in classes:
+ continue
+ if uid // 1000 == FINDINGS_CATEGORY:
+ continue
+ # A connector with its own profile carries that profile's ladder; one
+ # on the canonical path carries the shared ladder.
+ promotes = profiles[name]["severity_promotes"] if name in profiles else canonical_severity_promotes
+ if not promotes:
+ caption, _ = classes[uid]
+ failures.append(
+ (
+ "class-cannot-promote",
+ f"{name!r} maps to {uid} ({caption}), category {uid // 1000}, and the severity map it carries "
+ f"never reaches {PROMOTE_SEVERITY_FLOOR}: every one of its events would be archived to the lake "
+ "and never alert, silently",
+ )
+ )
+
+ # A profile declaring a class the closed set does not know is the same
+ # defect one layer over, and the profiles predate this table.
+ for key, profile in sorted(profiles.items()):
+ if profile["class_uid"] not in classes:
+ failures.append(
+ (
+ "profile-class-unknown",
+ f"profile {key!r} declares class {profile['class_uid']}, which is not in the closed set; "
+ "the gate cannot tell whether it can promote",
+ )
+ )
+ continue
+ # A profile carries the uid *and* the caption, written by hand and
+ # side by side, which is how `microsoft_sentinel` came to declare uid
+ # 2002 beside "Security Finding" — the caption of 2001. Both are
+ # category 2, so promotion hid it, and every Sentinel event in the
+ # lake disagreed with itself while a query filtering on 2002 pulled
+ # incidents in among vulnerability findings.
+ caption, _ = classes[profile["class_uid"]]
+ if profile["class_name"] and profile["class_name"] != caption:
+ failures.append(
+ (
+ "profile-class-name-mismatch",
+ f"profile {key!r} declares class {profile['class_uid']} beside the name "
+ f"{profile['class_name']!r}; the schema calls that uid {caption!r}, so one of the two is "
+ "wrong and both travel onto every event",
+ )
+ )
+
+ unclassified = sorted(n for n in generic if n not in connector_classes)
+ stats = {
+ "declared": len(declared),
+ "with_class": sum(1 for e in connector_classes.values() if e["class_uid"] is not None and e["reason"] is None),
+ "generic": len(generic),
+ "generic_connectors": generic,
+ "generic_with_reason": sorted(set(generic) - set(unclassified)),
+ "unclassified": unclassified,
+ "classes": len(classes),
+ }
+
+ if check_count and len(generic) > max_generic:
+ failures.append(
+ (
+ "generic-count-grew",
+ f"{len(generic)} connector types are on the generic mapping, above the committed ceiling of "
+ f"{max_generic}. The ceiling is a debt balance, not a target: classify the new connector or "
+ "record why it stays generic, rather than raising the number",
+ )
+ )
+ return failures, stats
+
+
+def load(root: Path) -> dict:
+ classes_go = root / CLASSES_REL
+ normalizer = root / NORMALIZER_REL
+ connectors_dir = root / CONNECTORS_REL
+ for path in (classes_go, normalizer, connectors_dir):
+ if not path.exists():
+ raise GateError(f"expected input does not exist: {path}")
+
+ classes_src = classes_go.read_text(encoding="utf-8")
+ normalizer_src = normalizer.read_text(encoding="utf-8")
+ consts = parse_class_consts(classes_src)
+ data = {
+ "declared": parse_declared_connectors(connectors_dir),
+ "classes": parse_classes(classes_src, consts),
+ "connector_classes": parse_connector_classes(classes_src, consts),
+ "profiles": parse_profile_classes(normalizer_src),
+ "canonical_severity_promotes": parse_canonical_severity_promotes(normalizer_src),
+ }
+ # An empty parse means the format moved, not that the tree is clean.
+ for name in ("declared", "classes", "connector_classes", "profiles"):
+ if not data[name]:
+ raise GateError(f"parsed zero {name} — refusing to report a clean tree from an empty read")
+ if not data["canonical_severity_promotes"]:
+ raise GateError(
+ "_canonicalSeverityMap never reaches the promote floor — every canonical-envelope connector on a "
+ "non-finding class would be unpromotable, so refusing to report on the mapping at all"
+ )
+ return data
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ parser.add_argument("--repo-root", type=Path, default=repo_root())
+ parser.add_argument("--json", action="store_true")
+ parser.add_argument("--check", action="store_true", help="also fail when the generic-mapping count exceeds MAX_GENERIC")
+ parser.add_argument("--self-test", action="store_true", help="prove the gate detects what it claims to")
+ args = parser.parse_args(argv)
+
+ if args.self_test:
+ return self_test(args.repo_root.resolve())
+
+ root = args.repo_root.resolve()
+ try:
+ data = load(root)
+ except GateError as exc:
+ print(f"check_ocsf_class_coverage: FAILED to read the tree: {exc}", file=sys.stderr)
+ return 2
+
+ failures, stats = evaluate(**data, max_generic=MAX_GENERIC, check_count=args.check)
+
+ if args.json:
+ print(
+ json.dumps(
+ {
+ "repo_root": str(root),
+ "max_generic": MAX_GENERIC,
+ **stats,
+ "failures": [{"code": c, "detail": d} for c, d in failures],
+ },
+ indent=2,
+ sort_keys=True,
+ )
+ )
+ return 1 if failures else 0
+
+ print(f"repo root {root}")
+ print(f"class table {CLASSES_REL} ({stats['classes']} OCSF classes, verified against schema.ocsf.io)")
+ print(f"normalizer {NORMALIZER_REL} ({len(data['profiles'])} profiles)")
+ print(f"connectors {CONNECTORS_REL} ({stats['declared']} declared connector ids)")
+ print()
+ print(f"class coverage {stats['with_class']} with an OCSF class of their own")
+ print(f" {stats['generic']} on the generic 2001 mapping (ceiling {MAX_GENERIC})")
+ print(
+ f" of those, {len(stats['generic_with_reason'])} with a recorded reason "
+ f"and {len(stats['unclassified'])} with no decision at all"
+ )
+ if stats["generic_connectors"]:
+ print(f" {', '.join(stats['generic_connectors'])}")
+ print()
+ if failures:
+ print(f"FAIL — {len(failures)} finding(s):")
+ for code, detail in failures:
+ print(f" [{code}] {detail}")
+ return 1
+ print("OK — every declared connector type names an OCSF class or records why it stays")
+ print(" generic, every class can promote, and every uid matches its category.")
+ return 0
+
+
+# --------------------------------------------------------------------------
+# Self-test
+# --------------------------------------------------------------------------
+def self_test(root: Path) -> int:
+ """Inject each defect the gate claims to catch and require it to bite."""
+ try:
+ base = load(root)
+ except GateError as exc:
+ print(f"self-test: cannot read the tree: {exc}", file=sys.stderr)
+ return 2
+
+ clean, _ = evaluate(**base, max_generic=MAX_GENERIC, check_count=True)
+ if clean:
+ print("self-test: the unmodified tree already fails; fix that first", file=sys.stderr)
+ for code, detail in clean:
+ print(f" [{code}] {detail}", file=sys.stderr)
+ return 1
+
+ def mutate(**overrides) -> dict:
+ data = {k: (v.copy() if hasattr(v, "copy") else v) for k, v in base.items()}
+ for key, fn in overrides.items():
+ fn(data[key])
+ return data
+
+ cases: list[tuple[str, str, dict, int]] = [
+ (
+ "PY -> GO: a declared connector with no class and no recorded reason",
+ "connector-unclassified",
+ mutate(declared=lambda s: s.add("acme_telemetry")),
+ MAX_GENERIC,
+ ),
+ (
+ "GO -> PY: a table entry naming a connector that does not exist",
+ "class-names-nothing",
+ mutate(connector_classes=lambda d: d.update({"acme_xdr_9000": {"class_uid": 4001, "reason": None}})),
+ MAX_GENERIC,
+ ),
+ (
+ "GO -> PY: an entry carrying both a class and a reason",
+ "class-entry-ambiguous",
+ mutate(connector_classes=lambda d: d.update({"okta": {"class_uid": 3002, "reason": "also stays generic somehow"}})),
+ MAX_GENERIC,
+ ),
+ (
+ "GO -> PY: an entry carrying neither",
+ "class-entry-empty",
+ mutate(connector_classes=lambda d: d.update({"okta": {"class_uid": None, "reason": None}})),
+ MAX_GENERIC + 1,
+ ),
+ (
+ "GO -> PY: a class uid outside the service's closed set",
+ "class-uid-unknown",
+ mutate(connector_classes=lambda d: d.update({"okta": {"class_uid": 9999, "reason": None}})),
+ MAX_GENERIC,
+ ),
+ (
+ "GO -> PY: a reason too thin to review",
+ "class-reason-thin",
+ mutate(connector_classes=lambda d: d.update({"splunk": {"class_uid": None, "reason": "noisy"}})),
+ MAX_GENERIC + 1,
+ ),
+ (
+ "SCHEMA: a class whose declared category disagrees with its uid",
+ "class-category-mismatch",
+ mutate(classes=lambda d: d.update({4003: ("DNS Activity", 6)})),
+ MAX_GENERIC,
+ ),
+ (
+ "PROMOTION: a non-finding class whose severity map cannot reach the floor",
+ "class-cannot-promote",
+ mutate(profiles=lambda d: d.update({"okta": {"class_uid": 3002, "class_name": "Authentication", "severity_promotes": False}})),
+ MAX_GENERIC,
+ ),
+ (
+ "PROMOTION: a profile declaring a class the closed set does not know",
+ "profile-class-unknown",
+ mutate(profiles=lambda d: d.update({"splunk": {"class_uid": 7777, "class_name": "Invented", "severity_promotes": True}})),
+ MAX_GENERIC,
+ ),
+ (
+ "SCHEMA: a profile whose class name contradicts its own class uid",
+ "profile-class-name-mismatch",
+ mutate(
+ profiles=lambda d: d.update({"splunk": {"class_uid": 2002, "class_name": "Security Finding", "severity_promotes": True}})
+ ),
+ MAX_GENERIC,
+ ),
+ (
+ "COUNT: the generic-mapping count rises above the committed ceiling",
+ "generic-count-grew",
+ mutate(declared=lambda s: s.add("acme_telemetry")),
+ MAX_GENERIC,
+ ),
+ ]
+
+ print(f"self-test against {root}")
+ print(f"clean tree: 0 failures at MAX_GENERIC={MAX_GENERIC} (the baseline every case below perturbs)\n")
+ ok = True
+ for description, expected_code, data, ceiling in cases:
+ found, _ = evaluate(**data, max_generic=ceiling, check_count=True)
+ codes = {code for code, _ in found}
+ caught = expected_code in codes
+ ok &= caught
+ print(f" {'PASS' if caught else 'FAIL'} {description}")
+ print(f" expected [{expected_code}] got {sorted(codes) or 'nothing'}")
+
+ # An empty tree must be refused, not reported clean.
+ empty_refused = True
+ try:
+ load(Path("/nonexistent-tree-for-the-self-test"))
+ empty_refused = False
+ except GateError:
+ # Refusing an empty tree is the behaviour under test, so the
+ # exception is the pass and there is nothing to handle.
+ pass
+ ok &= empty_refused
+ print(f" {'PASS' if empty_refused else 'FAIL'} TREE: a tree with no inputs is refused rather than reported clean")
+
+ print()
+ if not ok:
+ print("self-test FAILED: the gate did not catch what it claims to catch")
+ return 1
+ print(f"self-test OK: {len(cases)} injected defects plus the empty-tree case, each caught")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/competitor_names.toml b/scripts/competitor_names.toml
index fa7828817..81aea490f 100644
--- a/scripts/competitor_names.toml
+++ b/scripts/competitor_names.toml
@@ -264,3 +264,8 @@ file = "apps/docs/src/pages/index.tsx"
start = "const COMPARE_HEADERS"
end = "function compareCellClass"
reason = "Capability comparison table rendered on the documentation portal landing page."
+
+[[allow]]
+paths = ["services/ingest/internal/normalizer/ocsf_classes.go"]
+names = ["Torq", "Tines", "Wazuh"]
+reason = "The OCSF class decision for every declared connector type. Each name is a connector id this platform ingests under, beside the reason its stream stays on Security Finding — all three of these send a judgement the vendor's own ruleset or workflow already made, which is why they are not reclassified to a raw-activity class. Dropping a name here would return that connector to the generic mapping, which promotes every record to an alert; the same first-party integration reference the connector modules and the bundled catalog already carry."
diff --git a/services/ingest/internal/normalizer/normalizer.go b/services/ingest/internal/normalizer/normalizer.go
index 4946023fd..f23b57d7f 100644
--- a/services/ingest/internal/normalizer/normalizer.go
+++ b/services/ingest/internal/normalizer/normalizer.go
@@ -112,9 +112,17 @@ var connectorProfiles = map[string]connectorProfile{
"Critical": 5, "High": 4, "Medium": 3, "Low": 2, "Informational": 1,
},
},
+ // 2001 and not 2002: the profile declared uid 2002 beside the caption
+ // "Security Finding", and 2002 is Vulnerability Finding in every OCSF
+ // version this service targets. A Sentinel incident is a detection, not
+ // a vulnerability. Nothing downstream read the uid closely enough to
+ // notice — both are category 2, so promotion was unaffected — but the
+ // class_uid and class_name on every Sentinel event in the lake
+ // contradicted each other, and a lake query filtering on 2002 would have
+ // returned Sentinel incidents alongside Qualys and Tenable findings.
"microsoft_sentinel": {
product: OcsfProduct{Name: "Sentinel", VendorName: "Microsoft"},
- classUID: 2002,
+ classUID: 2001,
className: "Security Finding",
fieldMap: map[string]string{
"TimeGenerated": "time",
@@ -569,7 +577,7 @@ var connectorTypeAliases = map[string]string{
// them safe: the connector's own connector_name is the long form (`qradar` is
// "IBM QRadar", `chronicle` is "Google Chronicle", `syslog_cef` is
// "Syslog / CEF"). Resolution happens once, at the top of Normalize, so the
-// profile lookup, the alias map, canonicalClassByConnector and the product
+// profile lookup, the alias map, connectorOCSFClass and the product
// identity on the canonical path all agree on one name.
//
// `palo_alto_cortex` was the one entry that named a vendor rather than a
@@ -606,20 +614,6 @@ func canonicalConnectorType(connectorType string) string {
return connectorType
}
-// canonicalClassByConnector overrides the default Security Finding class for
-// connector types whose canonical alerts are better modeled as another OCSF
-// class (identity providers -> Authentication 3002).
-var canonicalClassByConnector = map[string]struct {
- classUID int
- className string
-}{
- "okta": {3002, "Authentication"},
- "azure_entra": {3002, "Authentication"},
- "auth0": {3002, "Authentication"},
- "duo_security": {3002, "Authentication"},
- "onepassword": {3002, "Authentication"},
-}
-
func isCanonicalEnvelope(p map[string]interface{}) bool {
if p == nil {
return false
@@ -704,10 +698,20 @@ func unpromotableWarning(classUID int, severityID interface{}) string {
)
}
+// canonicalProfile builds the profile for a connector-normalized envelope.
+//
+// The class comes from `connectorOCSFClass` in ocsf_classes.go, which carries
+// a decision for every connector the registry declares — either the OCSF
+// class its telemetry is, or a recorded reason it stays on the Security
+// Finding default. That default is category 2, which fusion promotes
+// unconditionally, so a source left on it silently turns every routine record
+// it emits into an alert. The table replaced a five-entry override map; see
+// the file header for the promotion mechanism and
+// `scripts/check_ocsf_class_coverage.py` for the gate that keeps it complete.
func canonicalProfile(connectorType string) connectorProfile {
- classUID, className := 2001, "Security Finding"
- if override, ok := canonicalClassByConnector[connectorType]; ok {
- classUID, className = override.classUID, override.className
+ classUID, className := classSecurityFinding, "Security Finding"
+ if class, ok := ocsfClassForConnector(connectorType); ok {
+ classUID, className = class.uid, class.caption
}
name := connectorType
if name == "" {
diff --git a/services/ingest/internal/normalizer/normalizer_test.go b/services/ingest/internal/normalizer/normalizer_test.go
index 50d78493b..8895f11a7 100644
--- a/services/ingest/internal/normalizer/normalizer_test.go
+++ b/services/ingest/internal/normalizer/normalizer_test.go
@@ -757,10 +757,15 @@ func TestConsoleVocabularyNormalizesAsTheDeclaredConnector(t *testing.T) {
if gotAlt.OcsfEvent["severity_id"] != 5 {
t.Errorf("severity_id = %v via %q, want 5 for critical", gotAlt.OcsfEvent["severity_id"], alternate)
}
- // Category 2 is what should_promote() requires without a severity
- // floor; a connector whose events cannot promote is the defect.
- if gotAlt.OcsfEvent["category_uid"] != 2 {
- t.Errorf("category_uid = %v via %q, want 2", gotAlt.OcsfEvent["category_uid"], alternate)
+ // The property this stands for is that the event can reach an
+ // analyst. It was written as `category_uid == 2` when every
+ // canonical envelope defaulted to 2001; once connectorOCSFClass
+ // gave telemetry sources their own class, that spelling stopped
+ // describing the requirement and started describing one way of
+ // meeting it. should_promote() takes either branch.
+ if !promotable(gotAlt.OcsfEvent) {
+ t.Errorf("a critical event via %q cannot promote: category_uid=%v severity_id=%v",
+ alternate, gotAlt.OcsfEvent["category_uid"], gotAlt.OcsfEvent["severity_id"])
}
})
}
@@ -789,8 +794,11 @@ func TestConsoleVocabularyIsAcceptedInStrictModeOnlyWhenItResolves(t *testing.T)
t.Errorf("strict mode rejected %q, which folds onto a declared connector: %v", alternate, err)
continue
}
- if ev.OcsfEvent["category_uid"] != 2 {
- t.Errorf("%q: category_uid = %v, want 2", alternate, ev.OcsfEvent["category_uid"])
+ // As above: the requirement is that a high-severity event resolves to
+ // something promotable, not that it resolves to category 2.
+ if !promotable(ev.OcsfEvent) {
+ t.Errorf("%q: a high event cannot promote: category_uid=%v severity_id=%v",
+ alternate, ev.OcsfEvent["category_uid"], ev.OcsfEvent["severity_id"])
}
}
if _, err := strict.Normalize(&RawEvent{
diff --git a/services/ingest/internal/normalizer/ocsf_classes.go b/services/ingest/internal/normalizer/ocsf_classes.go
new file mode 100644
index 000000000..76975e8ea
--- /dev/null
+++ b/services/ingest/internal/normalizer/ocsf_classes.go
@@ -0,0 +1,350 @@
+package normalizer
+
+// OCSF event classes, and which one each connector type's telemetry is.
+//
+// Every uid, caption and category below was read from the published schema
+// before it was used here, not recalled:
+//
+// https://schema.ocsf.io/api/1.9.0/classes (the version the tree targets)
+// https://schema.ocsf.io/api/1.1.0/classes (the version metadata.version declares)
+//
+// Both were checked because the two numbers are not the same thing. The
+// normalizer stamps `metadata.version: 1.1.0` as its contract with
+// connectors, while the evidence bundle declares 1.9.0
+// (`docs/architecture/evidence-bundles.md`), and emitting a class that exists
+// only in the newer schema under the older version string would be a false
+// claim about a public standard. Every class named here carries the same uid,
+// caption and category in both, so the question does not arise — but it was
+// asked, and it is why this file lists two URLs rather than one.
+//
+// Why a connector's class is load-bearing rather than cosmetic
+// ------------------------------------------------------------
+// `should_promote()` in services/fusion/app/services/promoter.py promotes an
+// event to an alert when it is OCSF category 2 (Findings) **or** its
+// severity_id is at least 4. The category is the class uid divided by a
+// thousand, so the class a connector's events carry decides whether its
+// routine telemetry reaches an analyst's queue or stays in the lake.
+//
+// Before this table, every connector without a hand-written profile landed on
+// 2001 Security Finding through `canonicalProfile` — category 2, therefore
+// always promoted. For the connectors whose fetch_alerts really does return
+// vendor findings that is right. For the ones returning raw telemetry it
+// meant every permitted DNS lookup, every allowed proxy request, every
+// accepted VPC flow record and every routine Windows event became an alert
+// carrying severity "info": the alert-reduction property fusion exists to
+// provide, inverted, for the sources that emit the most volume.
+//
+// The split mirrors the one `ai_runtime` and `ai_guardrail` already make in
+// normalizer.go, for the same stated reason: routine activity at a category
+// the promoter leaves alone, judged findings at category 2.
+//
+// How a connector's class was chosen
+// ----------------------------------
+// By what its `fetch_alerts` actually returns, read from the connector rather
+// than inferred from the vendor's product category — the two disagree often
+// enough that guessing would have demoted four sources that only ever return
+// findings. Netskope's connector reads `/api/v2/events/data/alert` and
+// Proofpoint's reads `/v2/siem/messages/blocked`, so both return judgements
+// and keep category 2; Zscaler's reads web logs with `action: "any"` and
+// Cisco Umbrella's reads `/activity`, so both are telemetry and take their
+// activity class. Where a stream is genuinely mixed the class covering the
+// security-relevant majority is used and the entry says so.
+
+// OCSF class uids this service may emit. Named rather than spelled inline so
+// the gate reads one table and a reader can see the category each implies
+// (uid/1000), which is what decides promotion.
+const (
+ // Category 1 — System Activity.
+ classFileSystemActivity = 1001
+ classProcessActivity = 1007
+
+ // Category 2 — Findings. Promoted unconditionally.
+ classSecurityFinding = 2001
+ classVulnerabilityFinding = 2002
+
+ // Category 3 — Identity & Access Management.
+ classAccountChange = 3001
+ classAuthentication = 3002
+ classUserAccessManagement = 3005
+ classGroupManagement = 3006
+
+ // Category 4 — Network Activity.
+ classNetworkActivity = 4001
+ classHTTPActivity = 4002
+ classDNSActivity = 4003
+ classEmailActivity = 4009
+ classEmailURLActivity = 4012
+
+ // Category 6 — Application Activity.
+ classWebResourcesActivity = 6001
+ classAPIActivity = 6003
+ classDatastoreActivity = 6005
+ classFileHostingActivity = 6006
+)
+
+// ocsfClass is one row of the published schema, carried here so the mapping
+// below names a class rather than an integer and so a wrong caption cannot
+// ship: the Go test asserts uid/1000 equals the declared category for every
+// row and pins every caption against the schema.
+type ocsfClass struct {
+ uid int
+ caption string
+ category int
+}
+
+// ocsfClasses is the closed set. A mapping or a webhook template naming a uid
+// absent from here fails the Go test and the gate, which is what stops an
+// invented class uid from reaching the lake.
+var ocsfClasses = map[int]ocsfClass{
+ classFileSystemActivity: {classFileSystemActivity, "File System Activity", 1},
+ classProcessActivity: {classProcessActivity, "Process Activity", 1},
+ classSecurityFinding: {classSecurityFinding, "Security Finding", 2},
+ classVulnerabilityFinding: {classVulnerabilityFinding, "Vulnerability Finding", 2},
+ classAccountChange: {classAccountChange, "Account Change", 3},
+ classAuthentication: {classAuthentication, "Authentication", 3},
+ classUserAccessManagement: {classUserAccessManagement, "User Access Management", 3},
+ classGroupManagement: {classGroupManagement, "Group Management", 3},
+ classNetworkActivity: {classNetworkActivity, "Network Activity", 4},
+ classHTTPActivity: {classHTTPActivity, "HTTP Activity", 4},
+ classDNSActivity: {classDNSActivity, "DNS Activity", 4},
+ classEmailActivity: {classEmailActivity, "Email Activity", 4},
+ classEmailURLActivity: {classEmailURLActivity, "Email URL Activity", 4},
+ classWebResourcesActivity: {classWebResourcesActivity, "Web Resources Activity", 6},
+ classAPIActivity: {classAPIActivity, "API Activity", 6},
+ classDatastoreActivity: {classDatastoreActivity, "Datastore Activity", 6},
+ classFileHostingActivity: {classFileHostingActivity, "File Hosting Activity", 6},
+}
+
+// classesWithNoProducerYet records a declared class that nothing in the tree
+// emits, and why it is declared ahead of its producer.
+//
+// All three are *per-event* classes inside streams this table can only
+// describe per connector: a file syscall inside auditd's stream, a privilege
+// grant inside an IdP's sign-in stream, a group membership change inside a
+// directory's account stream. Choosing one of them as a whole connector's
+// class would misdescribe the majority of that connector's records, which is
+// the opposite of what this table is for. The event catalogue (depth plan
+// 2.3) is the per-event discriminator that reaches them.
+//
+// Declaring them anyway is deliberate: they are the vocabulary the catalogue
+// will map onto, verified against the schema now rather than invented later.
+// `TestEveryDeclaredClassIsReachableOrRecorded` requires every declared class
+// to be reached or listed here, and requires a listed class that *has*
+// acquired a producer to be removed — so the list can only shrink.
+var classesWithNoProducerYet = map[int]string{
+ classFileSystemActivity: "a file syscall inside auditd's and osquery's streams, whose majority shape is process activity",
+ classUserAccessManagement: "a privilege grant inside an IdP's stream, whose majority shape is authentication; " +
+ "Vault's audit log is the nearest whole-connector fit and is a secrets datastore, not a privilege manager",
+ classGroupManagement: "a group membership change inside a directory's stream, whose majority shape is account change",
+}
+
+// connectorClass is the decision recorded for one connector type: either the
+// OCSF class its canonical events carry, or the reason it is left on the
+// Security Finding default.
+//
+// `genericReason` is not an escape hatch for "nobody looked" — it is the
+// opposite. A connector reaching the default silently is the state this table
+// exists to end; one reaching it with a sentence saying why is a decision
+// somebody can disagree with in review.
+type connectorClass struct {
+ classUID int
+ genericReason string
+}
+
+// connectorOCSFClass carries a decision for every connector type
+// services/connectors declares. Exactly one field per entry: the gate
+// (`scripts/check_ocsf_class_coverage.py`) fails on a declared connector with
+// neither, and on an entry carrying both.
+var connectorOCSFClass = map[string]connectorClass{
+ // ---------------------------------------------------------------
+ // Category 3 — identity. Sign-ins and directory changes.
+ // ---------------------------------------------------------------
+ "okta": {classUID: classAuthentication},
+ "azure_entra": {classUID: classAuthentication},
+ "auth0": {classUID: classAuthentication},
+ "duo_security": {classUID: classAuthentication},
+ "onepassword": {classUID: classAuthentication},
+ // JumpCloud's directory insights are account lifecycle first and
+ // authentication second: its event_type vocabulary is dominated by user
+ // and group create, update and delete, with `success` carrying the auth
+ // outcome. Account Change is the superset those records fit.
+ "jumpcloud": {classUID: classAccountChange},
+
+ // ---------------------------------------------------------------
+ // Category 4 — network and email telemetry. High volume, mostly benign.
+ // ---------------------------------------------------------------
+ // Umbrella reads /activity: one record per DNS or proxy transaction,
+ // with `verdict` carrying the decision. At 2001 every permitted lookup
+ // in the estate became an alert.
+ "cisco_umbrella": {classUID: classDNSActivity},
+ // Secure web gateway and WAF logs: one record per web request, with the
+ // policy action in the record. Zscaler's connector asks for
+ // `action: "any"`, so an Allow arrives alongside a Quarantine.
+ "zscaler": {classUID: classHTTPActivity},
+ "imperva": {classUID: classHTTPActivity},
+ // Cloudflare Zero Trust ships two modes from one connector — WAF
+ // firewall events and Access application-request decisions. Both are a
+ // decision taken on an HTTP request reaching an application.
+ "cloudflare_zt": {classUID: classHTTPActivity},
+ // Flow records and network-monitoring telemetry.
+ "aws_vpc_flow": {classUID: classNetworkActivity},
+ "zeek_suricata": {classUID: classNetworkActivity},
+ // Sublime reads /v1/messages, which is every message rather than only
+ // the flagged ones, and defaults an unrecognised verdict to info.
+ "sublime_security": {classUID: classEmailActivity},
+ // Mimecast reads the TTP URL Protect log, which is one record per URL
+ // click with a verdict — not a message stream, which is why it carries
+ // the narrower class.
+ "mimecast": {classUID: classEmailURLActivity},
+
+ // ---------------------------------------------------------------
+ // Category 1 — endpoint telemetry. The EDR products that return the
+ // vendor's own detections are findings and are listed further down.
+ // ---------------------------------------------------------------
+ // auditd and the Windows Security channel are syscall and event-log
+ // streams. Process creation is the dominant shape in both and the one
+ // the detection corpus reads; their file-operation records are the
+ // reason classFileSystemActivity is declared and not yet reached.
+ "auditd": {classUID: classProcessActivity},
+ "windows_event": {classUID: classProcessActivity},
+ // osquery results through FleetDM and osctrl are scheduled-query
+ // differentials spanning processes, files, packages and host state. The
+ // security-relevant majority is process and file telemetry; the
+ // promotion consequence is right for the whole stream either way,
+ // because a routine inventory row should not be an alert.
+ "fleetdm": {classUID: classProcessActivity},
+ "osctrl": {classUID: classProcessActivity},
+
+ // ---------------------------------------------------------------
+ // Category 6 — application and data activity.
+ // ---------------------------------------------------------------
+ // Cloud control-plane audit logs: one API call per record.
+ "gcp_cloud_audit": {classUID: classAPIActivity},
+ "azure_activity": {classUID: classAPIActivity},
+ "oci": {classUID: classAPIActivity},
+ "kubernetes_audit": {classUID: classAPIActivity},
+ // Cloudflare's connector reads /accounts/{id}/audit_logs — the account
+ // control plane, not the HTTP request logs its vendor is better known
+ // for. Reading the connector rather than the brand is what keeps this
+ // out of HTTP Activity.
+ "cloudflare": {classUID: classAPIActivity},
+ // Tailscale's connector reads /tailnet/{t}/audit: ACL and device
+ // administration, not packets.
+ "tailscale": {classUID: classAPIActivity},
+ // SaaS audit trails. Workspace and M365 carry sign-in, admin, file and
+ // mail events in one stream; API Activity holds the whole stream
+ // honestly rather than claiming a precision the per-event discriminator
+ // does not yet exist for.
+ "google_workspace": {classUID: classAPIActivity},
+ "m365_audit": {classUID: classAPIActivity},
+ "slack_audit": {classUID: classAPIActivity},
+ "salesforce": {classUID: classAPIActivity},
+ "servicenow": {classUID: classAPIActivity},
+ "jira": {classUID: classAPIActivity},
+ "confluence_audit": {classUID: classAPIActivity},
+ // Source-control audit logs.
+ "github": {classUID: classAPIActivity},
+ "gitlab": {classUID: classAPIActivity},
+ // Provider admin audit logs: who created a key, who changed a seat.
+ "llm_usage": {classUID: classAPIActivity},
+ // The schema's own description of File Hosting Activity names Box and
+ // the services Dropbox competes with, which is why these two are not at
+ // File System Activity: that class is a process acting on a local file.
+ "box": {classUID: classFileHostingActivity},
+ "dropbox": {classUID: classFileHostingActivity},
+ // Warehouse query and login history.
+ "snowflake": {classUID: classDatastoreActivity},
+ // Vault's audit records are an operation against a secret path — a
+ // read, write or delete on a datastore — not a privilege update, which
+ // is what User Access Management describes.
+ "vault": {classUID: classDatastoreActivity},
+ // The AI gateway's runtime log is a request per model or tool call
+ // against a served resource.
+ "ai_gateway": {classUID: classWebResourcesActivity},
+
+ // ---------------------------------------------------------------
+ // Vulnerability findings — category 2, so still promoted
+ // unconditionally, but a class the lake can tell apart from a
+ // detection.
+ // ---------------------------------------------------------------
+ "qualys": {classUID: classVulnerabilityFinding},
+ "tenable_io": {classUID: classVulnerabilityFinding},
+ "snyk": {classUID: classVulnerabilityFinding},
+
+ // ---------------------------------------------------------------
+ // Deliberately 2001 Security Finding. Each returns the vendor's own
+ // judged findings, so category 2 and unconditional promotion is the
+ // correct outcome rather than an oversight. The reason is recorded so a
+ // reader can tell these apart from a connector nobody classified.
+ // ---------------------------------------------------------------
+ "crowdstrike": {genericReason: "EDR detections: already judged by the vendor"},
+ "sentinelone": {genericReason: "EDR threats: already judged by the vendor"},
+ "carbon_black": {genericReason: "EDR alerts: already judged by the vendor"},
+ "cortex_xdr": {genericReason: "XDR incidents: already judged by the vendor"},
+ "trend_vision_one": {genericReason: "XDR workbench alerts: already judged by the vendor"},
+ "azure_defender": {genericReason: "Defender alerts: already judged by the vendor"},
+ "aws_guardduty": {genericReason: "GuardDuty findings: already judged by the vendor"},
+ "aws_security_hub": {genericReason: "Security Hub findings: already judged by the vendor"},
+ "gcp_scc": {genericReason: "Security Command Center findings: already judged by the vendor"},
+ "wiz": {genericReason: "cloud security issues: already judged by the vendor"},
+ "orca": {genericReason: "cloud security alerts: already judged by the vendor"},
+ "lacework": {genericReason: "cloud security events: already judged by the vendor"},
+ "prisma_cloud": {genericReason: "cloud security alerts: already judged by the vendor"},
+ "sysdig": {genericReason: "runtime security events: already judged by the vendor"},
+ "darktrace": {genericReason: "NDR model breaches: already judged by the vendor"},
+ "greynoise": {genericReason: "threat-intel verdicts on an address already observed"},
+ "wazuh": {genericReason: "agent alerts above a rule-level threshold: judged by the vendor's ruleset"},
+ // Falco's connector drops anything below the configured priority, so
+ // what reaches ingest is a rule hit the operator asked to see.
+ "falco": {genericReason: "rule hits above a configured priority: the rule is the judgement"},
+ // Netskope's connector reads /api/v2/events/data/alert, not the web
+ // traffic log, so what arrives is already an alert.
+ "netskope": {genericReason: "CASB alerts, not the web traffic log: already judged by the vendor"},
+ // Proofpoint's connector reads /v2/siem/messages/blocked and Abnormal's
+ // reads /v1/threats and /v1/cases — the judged half of each product.
+ "proofpoint": {genericReason: "messages the gateway blocked: already judged by the vendor"},
+ "abnormal_security": {genericReason: "threats and cases, not the message stream: already judged by the vendor"},
+ // A forwarded or reported message carries a human's judgement, which is
+ // the whole reason it was forwarded.
+ "email_inbox": {genericReason: "messages a person reported as suspicious: the report is the judgement"},
+ // The connector ships a curated allow-list of security-relevant API
+ // calls, so the selection has already happened upstream of ingest.
+ "aws_cloudtrail": {genericReason: "a curated allow-list of security-relevant API calls: the connector has already decided"},
+ "opsgenie": {genericReason: "alerts raised by another system and forwarded"},
+ "pagerduty": {genericReason: "incidents raised by another system and forwarded"},
+ "tines": {genericReason: "SOAR story results: already judged by the sending workflow"},
+ "torq": {genericReason: "SOAR workflow results: already judged by the sending workflow"},
+ // CEF and LEEF carry the sending device's own class and severity. One
+ // class for every CEF sender would be a claim about devices this
+ // connector cannot see.
+ "syslog_cef": {genericReason: "CEF and LEEF carry the sending device's own class, which this connector cannot narrow"},
+
+ // SIEMs forward notables and correlation-rule hits, which are findings
+ // by construction: the search that produced them is the judgement.
+ "splunk": {genericReason: "SIEM notables: the correlation search is the judgement"},
+ "microsoft_sentinel": {genericReason: "SIEM incidents: the analytics rule is the judgement"},
+ "qradar": {genericReason: "SIEM offenses: the rule chain is the judgement"},
+ "elastic": {genericReason: "SIEM detection alerts: the rule is the judgement"},
+ "chronicle": {genericReason: "SIEM detections: the rule is the judgement"},
+ "cortex_xsiam": {genericReason: "SIEM incidents: the correlation is the judgement"},
+ "exabeam": {genericReason: "SIEM notable sessions: the model is the judgement"},
+ "securonix": {genericReason: "SIEM violations: the policy is the judgement"},
+ "devo": {genericReason: "SIEM alerts: the rule is the judgement"},
+ "sumo_logic": {genericReason: "SIEM signals: the rule is the judgement"},
+ "rapid7_insightidr": {genericReason: "SIEM investigations: the detection rule is the judgement"},
+ "trellix_helix": {genericReason: "SIEM alerts: the rule is the judgement"},
+ "datadog": {genericReason: "monitor alerts: the monitor is the judgement"},
+ "datadog_cloud_siem": {genericReason: "Cloud SIEM signals: the rule is the judgement"},
+}
+
+// ocsfClassForConnector returns the class a connector type's canonical events
+// carry, and whether the table names one. A connector with a recorded
+// `genericReason`, or no entry at all, returns false and keeps the default.
+func ocsfClassForConnector(connectorType string) (ocsfClass, bool) {
+ entry, ok := connectorOCSFClass[connectorType]
+ if !ok || entry.classUID == 0 {
+ return ocsfClass{}, false
+ }
+ class, known := ocsfClasses[entry.classUID]
+ return class, known
+}
diff --git a/services/ingest/internal/normalizer/ocsf_classes_test.go b/services/ingest/internal/normalizer/ocsf_classes_test.go
new file mode 100644
index 000000000..c0fdc0272
--- /dev/null
+++ b/services/ingest/internal/normalizer/ocsf_classes_test.go
@@ -0,0 +1,378 @@
+package normalizer
+
+import (
+ "os"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "testing"
+)
+
+// promotable mirrors should_promote() in
+// services/fusion/app/services/promoter.py: OCSF category 2 (Findings) is
+// promoted unconditionally, and anything else needs severity_id >= 4.
+//
+// Duplicated here rather than shared because the two services do not share a
+// runtime — the same reasoning the `findingsCategory` constants in
+// normalizer.go already carry, and TestUnpromotableWarningMatchesFusionPolicy
+// is what keeps the pair honest.
+func promotable(ocsf map[string]interface{}) bool {
+ if category, ok := ocsf["category_uid"].(int); ok && category == findingsCategory {
+ return true
+ }
+ severity, ok := ocsf["severity_id"].(int)
+ return ok && severity >= promoteSeverityFloor
+}
+
+func normalizeCanonical(t *testing.T, connectorType string, payload map[string]interface{}) map[string]interface{} {
+ t.Helper()
+ full := map[string]interface{}{"source": connectorType, "raw_event": map[string]interface{}{"id": "1"}}
+ for k, v := range payload {
+ full[k] = v
+ }
+ ev, err := newLenientNormalizer().Normalize(&RawEvent{
+ ConnectorID: "c-1",
+ ConnectorType: connectorType,
+ TenantID: "11111111-1111-1111-1111-111111111111",
+ ReceivedAt: "2026-10-07T00:00:00Z",
+ Payload: full,
+ })
+ if err != nil {
+ t.Fatalf("Normalize(%q): %v", connectorType, err)
+ }
+ return ev.OcsfEvent
+}
+
+// The defect the class table exists for, stated as the behaviour that must
+// now hold. On the pre-fix tree every one of these connectors reached the
+// 2001 Security Finding default inside canonicalProfile(), which is category
+// 2 and therefore promoted unconditionally — so a permitted DNS lookup, an
+// allowed proxy request, an accepted VPC flow record and a routine Windows
+// event each became an alert carrying severity "info".
+//
+// This test fails on that tree, by design: it is the negative control for the
+// mapping as much as the assertion for it.
+func TestRoutineTelemetryIsNotPromotedAsAFinding(t *testing.T) {
+ // Each case is the shape the named connector's own normalize() emits for
+ // its most common, least interesting record. Severity strings are the
+ // ones those connectors actually produce for a benign record: Umbrella's
+ // non-blocked verdict, Zscaler's Allow, a parsed ACCEPT flow line, a
+ // Windows event id outside _HIGH_EVENT_IDS.
+ cases := []struct {
+ connector string
+ why string
+ }{
+ {"cisco_umbrella", "one record per DNS lookup, verdict not blocked"},
+ {"zscaler", "one record per web request, action Allow"},
+ {"aws_vpc_flow", "one record per accepted flow"},
+ {"zeek_suricata", "network session telemetry"},
+ {"windows_event", "a Windows event id outside the high-risk set"},
+ {"auditd", "a syscall record with no matching audit key"},
+ {"google_workspace", "a routine Workspace activity"},
+ {"m365_audit", "a routine M365 audit record"},
+ {"snowflake", "a completed warehouse query"},
+ {"github", "a routine organisation audit entry"},
+ {"box", "a routine file view in a file-hosting account"},
+ {"vault", "a secret read against a path"},
+ {"cloudflare", "a routine account audit entry"},
+ {"mimecast", "a URL click the gateway did not block"},
+ }
+ for _, tc := range cases {
+ t.Run(tc.connector, func(t *testing.T) {
+ ocsf := normalizeCanonical(t, tc.connector, map[string]interface{}{
+ "severity": "info",
+ "title": "routine activity",
+ })
+ if promotable(ocsf) {
+ t.Errorf(
+ "an info-severity event from %q (%s) is promotable: class_uid=%v category_uid=%v severity_id=%v. "+
+ "Every record this source emits would become an alert",
+ tc.connector, tc.why, ocsf["class_uid"], ocsf["category_uid"], ocsf["severity_id"],
+ )
+ }
+ })
+ }
+}
+
+// The other half, and the reason the mapping is per connector rather than a
+// blanket demotion: a source whose fetch_alerts returns the vendor's own
+// judged findings must still promote on class, so a Medium EDR detection or a
+// Medium SIEM notable is never silently dropped.
+func TestVendorFindingsStillPromoteOnClass(t *testing.T) {
+ for _, connector := range []string{
+ "crowdstrike", "sentinelone", "carbon_black", "aws_guardduty", "wiz",
+ "splunk", "qradar", "elastic", "microsoft_sentinel", "aws_cloudtrail",
+ // Verified against the endpoint each connector reads: an alert
+ // stream, a blocked-message stream and a priority-filtered rule
+ // stream are judgements, and demoting them would have silenced a
+ // medium-severity vendor finding.
+ "netskope", "proofpoint", "abnormal_security", "falco", "email_inbox",
+ } {
+ t.Run(connector, func(t *testing.T) {
+ ocsf := normalizeCanonical(t, connector, map[string]interface{}{
+ "severity": "medium",
+ "title": "vendor finding",
+ })
+ if !promotable(ocsf) {
+ t.Errorf("a medium finding from %q is not promotable: class_uid=%v severity_id=%v",
+ connector, ocsf["class_uid"], ocsf["severity_id"])
+ }
+ })
+ }
+}
+
+// The gate counts a connector with a recorded reason as "on the generic
+// mapping", and that count is the figure Phase 2 is graded against. It only
+// means what it says if a recorded reason really does resolve to 2001 — a
+// reason sitting beside a profile that moved the class would make the number
+// describe the table rather than the events.
+func TestARecordedReasonResolvesToSecurityFinding(t *testing.T) {
+ for connector, entry := range connectorOCSFClass {
+ if entry.genericReason == "" {
+ continue
+ }
+ t.Run(connector, func(t *testing.T) {
+ ocsf := normalizeCanonical(t, connector, map[string]interface{}{
+ "severity": "medium",
+ "title": "vendor finding",
+ })
+ if ocsf["class_uid"] != classSecurityFinding {
+ t.Errorf("%q records a reason for staying on the Security Finding default but resolves to "+
+ "class_uid %v; the generic count would describe the table rather than the events",
+ connector, ocsf["class_uid"])
+ }
+ })
+ }
+}
+
+// Demoting a telemetry source out of category 2 only works if its severe
+// events still get through on the other branch. A class with a severity map
+// that cannot reach the floor is the `splunk_enterprise` defect wearing a new
+// class uid: archived to the lake, never an alert, silently.
+func TestMappedTelemetryClassesStillPromoteOnSeverity(t *testing.T) {
+ for connector, entry := range connectorOCSFClass {
+ if entry.classUID == 0 || entry.classUID/1000 == findingsCategory {
+ continue
+ }
+ t.Run(connector, func(t *testing.T) {
+ for _, severity := range []string{"high", "critical"} {
+ ocsf := normalizeCanonical(t, connector, map[string]interface{}{
+ "severity": severity,
+ "title": "something worth a human",
+ })
+ if !promotable(ocsf) {
+ t.Errorf("a %s event from %q cannot promote: class_uid=%v severity_id=%v",
+ severity, connector, ocsf["class_uid"], ocsf["severity_id"])
+ }
+ }
+ })
+ }
+}
+
+// Every connector the registry declares carries a decision. The registry is
+// read off disk rather than listed here, because a hardcoded list drifts from
+// it and then agrees with itself forever — the failure this whole area keeps
+// producing.
+func TestEveryDeclaredConnectorHasAnOCSFDecision(t *testing.T) {
+ for id := range declaredConnectorIDs(t) {
+ entry, ok := connectorOCSFClass[id]
+ if !ok {
+ t.Errorf("connector %q has no entry in connectorOCSFClass: its events take the 2001 default, "+
+ "which is always promoted, and nothing records that as a decision", id)
+ continue
+ }
+ if entry.classUID == 0 && entry.genericReason == "" {
+ t.Errorf("connector %q has an entry carrying neither a class nor a reason", id)
+ }
+ if entry.classUID != 0 && entry.genericReason != "" {
+ t.Errorf("connector %q declares both a class and a generic reason; one of them is not what happens", id)
+ }
+ }
+}
+
+// A class uid whose declared category disagrees with uid/1000 would change
+// which promotion branch a connector's events take without anyone editing the
+// mapping, because fusion derives the category from the uid alone.
+func TestOCSFClassCategoryIsDerivedFromItsUID(t *testing.T) {
+ if len(ocsfClasses) == 0 {
+ t.Fatal("ocsfClasses is empty — refusing to pass on an empty table")
+ }
+ for uid, class := range ocsfClasses {
+ if class.uid != uid {
+ t.Errorf("ocsfClasses[%d] carries uid %d; a row must describe its own key", uid, class.uid)
+ }
+ if uid/1000 != class.category {
+ t.Errorf("class %d (%s) declares category %d, but uid/1000 is %d",
+ uid, class.caption, class.category, uid/1000)
+ }
+ }
+}
+
+// Nothing may map to a uid outside the closed set. Inventing one is how an
+// event reaches the lake carrying a class no schema defines.
+func TestConnectorClassesNameAKnownClass(t *testing.T) {
+ for connector, entry := range connectorOCSFClass {
+ if entry.classUID == 0 {
+ continue
+ }
+ if _, ok := ocsfClasses[entry.classUID]; !ok {
+ t.Errorf("connector %q maps to class uid %d, which ocsfClasses does not declare", connector, entry.classUID)
+ }
+ }
+}
+
+// The uids and captions above were read from schema.ocsf.io before use. This
+// pins the pairing so a later edit cannot quietly rename a class into one the
+// schema spells differently — the caption travels into `class_name` on every
+// event and into the lake.
+func TestClassCaptionsMatchThePublishedSchema(t *testing.T) {
+ published := map[int]string{
+ 1001: "File System Activity",
+ 1007: "Process Activity",
+ 2001: "Security Finding",
+ 2002: "Vulnerability Finding",
+ 3001: "Account Change",
+ 3002: "Authentication",
+ 3005: "User Access Management",
+ 3006: "Group Management",
+ 4001: "Network Activity",
+ 4002: "HTTP Activity",
+ 4003: "DNS Activity",
+ 4009: "Email Activity",
+ 4012: "Email URL Activity",
+ 6001: "Web Resources Activity",
+ 6003: "API Activity",
+ 6005: "Datastore Activity",
+ 6006: "File Hosting Activity",
+ }
+ for uid, caption := range published {
+ class, ok := ocsfClasses[uid]
+ if !ok {
+ t.Errorf("class %d (%s) is no longer declared", uid, caption)
+ continue
+ }
+ if class.caption != caption {
+ t.Errorf("class %d: caption %q, schema says %q", uid, class.caption, caption)
+ }
+ }
+ for uid := range ocsfClasses {
+ if _, ok := published[uid]; !ok {
+ t.Errorf("class %d is declared but was not checked against the published schema; "+
+ "add it to this table only after reading schema.ocsf.io", uid)
+ }
+ }
+}
+
+// A declared class nothing emits is the "exists but nothing calls it" shape
+// one layer down: a reader sees the vocabulary and assumes the lake contains
+// it. Every class must therefore be reached by a connector mapping, a
+// hand-written profile or a webhook template, or be listed in
+// `classesWithNoProducerYet` with the reason.
+//
+// The list is a ratchet in both directions. A class with no producer and no
+// entry fails, and so does an entry for a class that has since acquired one —
+// otherwise the record outlives the gap it describes, which is how a caveat
+// becomes a lie.
+func TestEveryDeclaredClassIsReachableOrRecorded(t *testing.T) {
+ produced := map[int]string{}
+ for connector, entry := range connectorOCSFClass {
+ if entry.classUID != 0 {
+ produced[entry.classUID] = "connector " + connector
+ }
+ }
+ for key, profile := range connectorProfiles {
+ produced[profile.classUID] = "profile " + key
+ }
+ for name, uid := range templateClassUIDs(t) {
+ produced[uid] = "template " + name
+ }
+
+ for uid, class := range ocsfClasses {
+ by, reached := produced[uid]
+ reason, recorded := classesWithNoProducerYet[uid]
+ switch {
+ case reached && recorded:
+ t.Errorf("class %d (%s) is produced by %s but is still listed as having no producer (%q); "+
+ "remove the entry so the list keeps shrinking", uid, class.caption, by, reason)
+ case !reached && !recorded:
+ t.Errorf("class %d (%s) is declared and nothing emits it; map a connector to it or record why "+
+ "it is declared ahead of its producer", uid, class.caption)
+ case !reached && len(reason) < 30:
+ t.Errorf("class %d (%s) is recorded as having no producer for the reason %q, which does not say "+
+ "enough for a reviewer to disagree with it", uid, class.caption, reason)
+ }
+ }
+
+ for uid := range classesWithNoProducerYet {
+ if _, declared := ocsfClasses[uid]; !declared {
+ t.Errorf("classesWithNoProducerYet names class %d, which ocsfClasses does not declare", uid)
+ }
+ }
+}
+
+// Webhook templates are the second producer of class uids — the push path to
+// the connectors' pull path — and nothing checked them against the closed
+// set. A template naming a uid no schema defines reaches the lake just as
+// easily as a connector would.
+func TestWebhookTemplatesNameAKnownClass(t *testing.T) {
+ for name, uid := range templateClassUIDs(t) {
+ if _, ok := ocsfClasses[uid]; !ok {
+ t.Errorf("template %s declares class_uid %d, which ocsfClasses does not declare", name, uid)
+ }
+ }
+}
+
+// templateClassUIDs reads the class each webhook template stamps, off disk.
+// The templates are embedded into the binary from this directory, so reading
+// them here is reading what ships.
+func templateClassUIDs(t *testing.T) map[string]int {
+ t.Helper()
+ entries, err := os.ReadDir("templates")
+ if err != nil {
+ t.Fatalf("cannot read templates: %v", err)
+ }
+ classUIDRe := regexp.MustCompile(`(?m)^class_uid:\s*(\d+)`)
+ out := map[string]int{}
+ for _, entry := range entries {
+ if entry.IsDir() || filepath.Ext(entry.Name()) != ".yaml" {
+ continue
+ }
+ src, err := os.ReadFile(filepath.Join("templates", entry.Name()))
+ if err != nil {
+ t.Fatalf("cannot read template %s: %v", entry.Name(), err)
+ }
+ if m := classUIDRe.FindSubmatch(src); m != nil {
+ uid, convErr := strconv.Atoi(string(m[1]))
+ if convErr != nil {
+ t.Fatalf("template %s: class_uid %q is not an integer", entry.Name(), m[1])
+ }
+ out[entry.Name()] = uid
+ }
+ }
+ if len(out) == 0 {
+ t.Fatal("parsed zero template class uids — refusing to pass on an empty read")
+ }
+ return out
+}
+
+// A connector with a hand-written profile takes that profile's class, not the
+// table's. Declaring a class for one would be a decision nothing acts on, and
+// a reader would believe it.
+func TestClassTableDoesNotContradictAHandWrittenProfile(t *testing.T) {
+ for connector, entry := range connectorOCSFClass {
+ if entry.classUID == 0 {
+ continue
+ }
+ profile, hasProfile := connectorProfiles[connector]
+ if !hasProfile {
+ if aliased, isAlias := connectorTypeAliases[connector]; isAlias {
+ profile, hasProfile = connectorProfiles[aliased]
+ }
+ }
+ if hasProfile && profile.classUID != entry.classUID {
+ t.Errorf("connector %q maps to class %d in the table but its profile declares %d; "+
+ "a flat payload takes the profile and a canonical envelope takes the table, so the two must agree",
+ connector, entry.classUID, profile.classUID)
+ }
+ }
+}