Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
An event classification catalogue, read at boot and grown from the connector fixtures (#1209)
Beenu Arora7cf5fad60f36
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 0cc054fd9..f00adb24b 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -649,6 +649,19 @@ jobs:run: python3 scripts/check_connector_profiles.py --self-test- name: Connector profile drift gaterun: python3 scripts/check_connector_profiles.py+ # One layer in again: a connector's events reach a profile, and the+ # catalogue says what each of its event types *means*. Grown from the+ # fixtures, so adding a fixture that carries a new event type fails+ # here until somebody decides what it is — a catalogue that only grows+ # when someone remembers is a catalogue that stops growing. It also+ # refuses a source whose fixtures yield nothing, because a gate over an+ # empty corpus passes for the wrong reason.+ - name: Event catalogue gate — self-test+ run: |+ python3 -m pip install --quiet pyyaml+ python3 scripts/check_event_catalog.py --self-test+ - name: Event catalogue gate+ run: python3 scripts/check_event_catalog.py# The activity projection lives in five hand-written lists: the Go, the# lake's CREATE TABLE, the lake's migration, the fusion writer's column# tuple and the graph schema. Three of the disagreements between themdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 64e208e0b..325125314 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -182,6 +182,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0`KNOWN_ABSENT` table so that building it will fail the gate until the proseis corrected. (#PR)+### Added++- **An event classification catalogue, read by ingest at boot.**+ `schemas/event_catalog/<source>.yaml` maps each vendor's own event type+ onto a normalized action, a sensitivity on the five-tier ladder, and an+ optional ATT&CK hint. Ten sources, 44 event types: AWS CloudTrail, GCP+ Cloud Audit, Azure Activity, Entra, Okta, Google Workspace, M365, GitHub,+ Slack and Kubernetes audit. Every classified event carries the answer, so+ the lake, the detection matcher and triage read one classification rather+ than each re-deriving a meaning from an event name.++ **A sensitivity deliberately does not change a severity or whether fusion+ promotes.** Sensitivity is a property of the event *class*; the vendor's+ own severity on the record and the OCSF class still decide promotion.+ Letting `critical` force an alert would make a one-line edit to a data+ file able to flood a queue, and the promotion contract belongs where it+ already is. Pinned by a test that a critical-sensitivity and an+ info-sensitivity event from the same source agree on `severity_id`,+ `severity`, `class_uid` and `category_uid`.++ An event type a catalogued source has never seen is the one case worth an+ operator's attention, and it arrives as a `normalization_warnings` entry+ on the event naming the file to edit. A source with no catalogue, and a+ record carrying nothing at the declared path, are both silent — warning on+ either would drown the actionable case.++ The catalogue is embedded into the ingest binary from a byte-identical+ vendored copy, because `go:embed` cannot reach above its own package and+ reading a directory from disk at run time is a failure this service has+ already shipped: the webhook templates were read from `/app/templates`, a+ path the Dockerfile never populated, so every deployment answered 503+ behind a startup warning nothing was watching.++- **Recorded Okta and Slack vendor payloads.** Two of the ten sources the+ catalogue covers had no vendor-payload fixture anywhere in the connectors+ suite — only schema and conformance checks, which never see a record. A+ gate grown from fixtures would have passed over both forever while+ reporting OK. `tests/fixtures/okta/sample_event.json` (7 records, 6 event+ types) and `tests/fixtures/slack_audit/sample_event.json` (6 records, 6+ event types) follow each vendor's published log schema and are driven+ through the real connectors' `normalize()`.++- `scripts/check_event_catalog.py`, wired into `ci.yml` with its self-test+ ahead of it. Every event type appearing in a catalogue's declared fixtures+ must be classified or listed as deliberately unclassified with a reason,+ so adding a fixture that carries a new event type fails CI until somebody+ decides what it means. It also fails a source whose declared fixtures+ yield **no** event type, a sensitivity outside the five tiers, a missing+ normalized action, a malformed ATT&CK id, a thin excuse, an event type+ both classified and excused, and any drift between `schemas/` and the+ vendored copy the binary carries.++## [17.1.0] - 2026-10-07### Securitydiff --git a/schemas/event_catalog/README.md b/schemas/event_catalog/README.mdnew file mode 100644index 000000000..34532e8bf--- /dev/null+++ b/schemas/event_catalog/README.md@@ -0,0 +1,65 @@+# Event classification catalogue++One file per source. Each maps a vendor's own event type onto a normalized+action, a sensitivity, and an optional ATT&CK hint.++```yaml+source: aws_cloudtrail+event_type_path: eventName # where the vendor's event type lives+fixture_files: # the corpus the gate grows this from+ - services/connectors/tests/test_aws_cloudtrail.py+events:+ CreateAccessKey:+ action: create.access.key+ sensitivity: high+ attack: [T1098.001]+unclassified:+ SomeEventType: why it is deliberately not classified+```++## Why a data file rather than code++The question "how sensitive is this event type" has a different answer per+vendor, per event, and sometimes per customer, and it changes whenever a+vendor adds an event. A regex over event names would be wrong in both+directions and invisible when it was. A reviewed table is wrong in exactly+the places somebody can point at.++It also has to be the *same* table everywhere. Ingest reads it at boot and+stamps the classification onto every event, so the lake, the detection+matcher and triage see one answer rather than three re-derivations.++## Sensitivity++The five-tier ladder the rest of the platform uses — `info`, `low`,+`medium`, `high`, `critical` — and it means **how security-relevant this+event type is**, not how severe this particular occurrence was. A vendor's+own severity on the record still governs promotion; sensitivity is a+property of the *class* of event and is what makes "show me everything above+medium that a non-human actor did" answerable without enumerating verbs.++`critical` is reserved for event types that are an incident on their own+reading — disabling the audit trail, destroying it, granting the top-level+administrative role. Everything an estate does hourly is `info`.++## How this grows++`scripts/check_event_catalog.py` extracts every event type that appears in+the fixtures each catalogue declares, and fails when one is neither+classified nor listed under `unclassified` with a reason. Adding a connector+fixture that carries a new event type therefore fails CI until somebody+decides what it means, which is the point: a catalogue that only grows when+someone remembers is a catalogue that stops growing.++A source whose declared fixtures yield **no** event types also fails. A gate+over an empty corpus is a gate that passes for the wrong reason, and two of+the ten sources here had no vendor-payload fixture at all until this+catalogue was written.++## What is deliberately not here++A sensitivity does not change an event's severity or whether fusion promotes+it. Making `critical` sensitivity force an alert would mean this file could+silently flood a queue, and the promotion contract belongs with the OCSF+class and the vendor's severity where it already is. The classification is+attached to the event for detections, hunts and triage to read.diff --git a/schemas/event_catalog/aws_cloudtrail.yaml b/schemas/event_catalog/aws_cloudtrail.yamlnew file mode 100644index 000000000..311a7b9e7--- /dev/null+++ b/schemas/event_catalog/aws_cloudtrail.yaml@@ -0,0 +1,36 @@+# AWS CloudTrail. Event types are the API call names in `eventName`.+#+# Sensitivity reflects what the call class means, not what any one+# occurrence scored: DescribeInstances is reconnaissance-shaped and happens+# thousands of times an hour in a healthy account, DeleteTrail is an+# incident on its own reading.+source: aws_cloudtrail+event_type_path: eventName+fixture_files:+ - services/connectors/tests/test_aws_cloudtrail.py+events:+ ConsoleLogin:+ action: console.login+ sensitivity: medium+ attack: [T1078.004]+ CreateAccessKey:+ action: create.access.key+ sensitivity: high+ attack: [T1098.001]+ DeleteTrail:+ # Destroying the audit trail is the one event where the absence of+ # everything after it is the finding.+ action: delete.trail+ sensitivity: critical+ attack: [T1562.008]+ PutEventSelectors:+ # Narrowing what the trail records is the quieter half of the same+ # technique, which is why it sits one tier below rather than at info.+ action: put.event.selectors+ sensitivity: high+ attack: [T1562.008]+ DescribeInstances:+ action: describe.instances+ sensitivity: info+ attack: [T1580]+unclassified: {}diff --git a/schemas/event_catalog/azure_activity.yaml b/schemas/event_catalog/azure_activity.yamlnew file mode 100644index 000000000..cae5b0a42--- /dev/null+++ b/schemas/event_catalog/azure_activity.yaml@@ -0,0 +1,16 @@+# Azure Activity Log. Event types are the ARM operation names in+# `operationName.value`, which are `Provider/resourceType/action`.+source: azure_activity+event_type_path: operationName.value+fixture_files:+ - services/connectors/tests/test_azure_connectors.py+events:+ Microsoft.Authorization/roleAssignments/write:+ action: authorization.role.assignment.write+ sensitivity: high+ attack: [T1098.003]+ Microsoft.Storage/storageAccounts/read:+ action: storage.account.read+ sensitivity: info+ attack: [T1580]+unclassified: {}diff --git a/schemas/event_catalog/azure_entra.yaml b/schemas/event_catalog/azure_entra.yamlnew file mode 100644index 000000000..e2274665c--- /dev/null+++ b/schemas/event_catalog/azure_entra.yaml@@ -0,0 +1,18 @@+# Microsoft Entra ID audit logs. Event types are the+# `activityDisplayName` strings, which are prose rather than identifiers —+# Microsoft's own vocabulary, kept verbatim so a lookup against a live+# tenant's logs matches.+source: azure_entra+event_type_path: activityDisplayName+fixture_files:+ - services/connectors/tests/test_azure_connectors.py+events:+ Add member to role:+ action: add.member.to.role+ sensitivity: high+ attack: [T1098.003]+ Add user:+ action: add.user+ sensitivity: medium+ attack: [T1136.003]+unclassified: {}diff --git a/schemas/event_catalog/gcp_cloud_audit.yaml b/schemas/event_catalog/gcp_cloud_audit.yamlnew file mode 100644index 000000000..9dab8749c--- /dev/null+++ b/schemas/event_catalog/gcp_cloud_audit.yaml@@ -0,0 +1,27 @@+# GCP Cloud Audit Logs. Event types are the `methodName` on the proto+# payload: either a bare admin method or the fully-qualified service method.+source: gcp_cloud_audit+event_type_path: methodName+fixture_files:+ - services/connectors/tests/test_gcp_connectors.py+events:+ SetIamPolicy:+ # Rewrites who can do what on a resource, which is the single call an+ # escalation most often ends with.+ action: set.iam.policy+ sensitivity: high+ attack: [T1098.003]+ google.compute.v1.instances.start:+ action: compute.instances.start+ sensitivity: low+ google.compute.v1.instances.detachDisk:+ # Detaching a disk is how a volume is moved to an instance the+ # attacker controls, and it is otherwise rare.+ action: compute.instances.detach.disk+ sensitivity: medium+ attack: [T1530]+ google.storage.buckets.list:+ action: storage.buckets.list+ sensitivity: info+ attack: [T1580]+unclassified: {}diff --git a/schemas/event_catalog/github.yaml b/schemas/event_catalog/github.yamlnew file mode 100644index 000000000..2d7b5a010--- /dev/null+++ b/schemas/event_catalog/github.yaml@@ -0,0 +1,25 @@+# GitHub organisation audit log. Event types are the `action` values,+# namespaced `area.verb`.+source: github+event_type_path: action+fixture_requires: [actor]+fixture_files:+ - services/connectors/tests/test_saas_connectors.py+events:+ org.update_member:+ action: org.update.member+ sensitivity: high+ attack: [T1098]+ team.destroy:+ action: team.destroy+ sensitivity: medium+ secret_scanning.disable:+ # Turning off the control that finds leaked credentials, in the system+ # that holds the code they are leaked in.+ action: secret.scanning.disable+ sensitivity: critical+ attack: [T1562.001]+ issues.opened:+ action: issues.opened+ sensitivity: info+unclassified: {}diff --git a/schemas/event_catalog/google_workspace.yaml b/schemas/event_catalog/google_workspace.yamlnew file mode 100644index 000000000..43ddf9a1f--- /dev/null+++ b/schemas/event_catalog/google_workspace.yaml@@ -0,0 +1,32 @@+# Google Workspace Reports API. Event types are the `name` of each entry in+# the activity's `events` list; one activity can carry several.+source: google_workspace+event_type_path: events[].name+fixture_files:+ - services/connectors/tests/test_saas_connectors.py+events:+ GRANT_ADMIN_PRIVILEGE:+ action: grant.admin.privilege+ sensitivity: critical+ attack: [T1098]+ suspicious_login:+ # Google has already judged this one, which is why it is not merely a+ # sign-in.+ action: suspicious.login+ sensitivity: high+ attack: [T1078]+ login_failure:+ # On its own this is noise; in volume it is a spray. Sensitivity is a+ # property of the event class, and the windowed engine is what reads+ # the volume.+ action: login.failure+ sensitivity: low+ attack: [T1110]+ change_acl_editors:+ action: change.acl.editors+ sensitivity: medium+ attack: [T1222]+ view:+ action: view+ sensitivity: info+unclassified: {}diff --git a/schemas/event_catalog/kubernetes_audit.yaml b/schemas/event_catalog/kubernetes_audit.yamlnew file mode 100644index 000000000..26676881b--- /dev/null+++ b/schemas/event_catalog/kubernetes_audit.yaml@@ -0,0 +1,33 @@+# Kubernetes apiserver audit. The event type is the `verb`, and on its own+# it says nothing about what was acted on — `create` on a ConfigMap and+# `create` on a ClusterRoleBinding are not the same event.+#+# The catalogue classifies the verb, which is what this source's event type+# *is*, and the resource is carried separately on the event. `impersonate`+# is the exception that needs no resource to be interesting.+source: kubernetes_audit+event_type_path: verb+fixture_requires: [objectRef]+fixture_files:+ - services/connectors/tests/test_kubernetes_audit.py+events:+ get:+ action: get+ sensitivity: info+ list:+ action: list+ sensitivity: info+ attack: [T1613]+ create:+ action: create+ sensitivity: medium+ patch:+ action: patch+ sensitivity: medium+ impersonate:+ # The only verb that is a privilege decision by itself: the request is+ # asking the apiserver to act as somebody else.+ action: impersonate+ sensitivity: critical+ attack: [T1134]+unclassified: {}diff --git a/schemas/event_catalog/m365_audit.yaml b/schemas/event_catalog/m365_audit.yamlnew file mode 100644index 000000000..d6c7a0005--- /dev/null+++ b/schemas/event_catalog/m365_audit.yaml@@ -0,0 +1,28 @@+# Microsoft 365 Management Activity API. Event types are the `Operation`+# values, which span every workload in one stream.+source: m365_audit+event_type_path: Operation+fixture_files:+ - services/connectors/tests/test_saas_connectors.py+events:+ UserLoggedIn:+ action: user.logged.in+ sensitivity: low+ attack: [T1078]+ Add member to role:+ action: add.member.to.role+ sensitivity: high+ attack: [T1098.003]+ New-InboxRule:+ # An inbox rule created by an attacker is how a compromised mailbox+ # stays quiet: replies to the real owner are filed away unread.+ action: new.inbox.rule+ sensitivity: high+ attack: [T1114.003]+ FileAccessed:+ action: file.accessed+ sensitivity: info+ FilePreviewed:+ action: file.previewed+ sensitivity: info+unclassified: {}diff --git a/schemas/event_catalog/okta.yaml b/schemas/event_catalog/okta.yamlnew file mode 100644index 000000000..b86302650--- /dev/null+++ b/schemas/event_catalog/okta.yaml@@ -0,0 +1,38 @@+# Okta System Log. Event types are the dotted `eventType` strings, which+# Okta namespaces by subject — `user.session.*`, `system.api_token.*`,+# `security.*`.+source: okta+event_type_path: eventType+fixture_files:+ - services/connectors/tests/fixtures/okta/sample_event.json+events:+ user.session.start:+ # Both a success and a failure arrive under this one type; the outcome+ # is on the record, which is why the class sits at low rather than+ # being split into two entries the vendor does not have.+ action: user.session.start+ sensitivity: low+ attack: [T1078]+ user.account.lock:+ action: user.account.lock+ sensitivity: high+ attack: [T1110]+ user.account.privilege.grant:+ action: user.account.privilege.grant+ sensitivity: critical+ attack: [T1098]+ system.api_token.create:+ # A token outlives the session that minted it and carries the+ # creator's authority with no second factor behind it.+ action: system.api.token.create+ sensitivity: high+ attack: [T1098.001]+ user.mfa.factor.deactivate:+ action: user.mfa.factor.deactivate+ sensitivity: high+ attack: [T1556.006]+ security.request.blocked:+ action: security.request.blocked+ sensitivity: high+ attack: [T1078]+unclassified: {}diff --git a/schemas/event_catalog/slack_audit.yaml b/schemas/event_catalog/slack_audit.yamlnew file mode 100644index 000000000..f9b1cefb9--- /dev/null+++ b/schemas/event_catalog/slack_audit.yaml@@ -0,0 +1,35 @@+# Slack Audit Logs API. Event types are the `action` values, which Slack+# names as past-tense verbs over an entity.+source: slack_audit+event_type_path: action+fixture_requires: [actor]+fixture_files:+ - services/connectors/tests/fixtures/slack_audit/sample_event.json+events:+ user_login:+ action: user.login+ sensitivity: low+ attack: [T1078]+ file_downloaded:+ action: file.downloaded+ sensitivity: info+ attack: [T1213]+ app_installed:+ # A third-party app arrives holding the scopes the installer consented+ # to, which is how data leaves a workspace without anyone downloading+ # anything.+ action: app.installed+ sensitivity: high+ attack: [T1550.001]+ user_session_reset_by_admin:+ action: user.session.reset.by.admin+ sensitivity: medium+ public_channel_created:+ action: public.channel.created+ sensitivity: info+ organization_export_started:+ # Every message and file in the workspace, in one archive.+ action: organization.export.started+ sensitivity: critical+ attack: [T1213]+unclassified: {}diff --git a/scripts/check_event_catalog.py b/scripts/check_event_catalog.pynew file mode 100644index 000000000..a312a8b79--- /dev/null+++ b/scripts/check_event_catalog.py@@ -0,0 +1,564 @@+#!/usr/bin/env python3+"""Every event type in a connector's fixtures is classified, or excused.++Why this exists+---------------+`schemas/event_catalog/<source>.yaml` says what a vendor's event types mean:+a normalized action, a sensitivity on the platform's five-tier ladder, and an+optional ATT&CK hint. Ingest reads it at boot and stamps the answer onto every+event, so the lake, the detection matcher and triage share one classification+rather than each re-deriving a meaning from an event name.++A catalogue that only grows when somebody remembers is a catalogue that stops+growing. So this gate extracts every event type that appears in the fixtures+each catalogue declares, and fails when one is neither classified nor listed+under ``unclassified`` with a reason. Adding a connector fixture carrying a+new event type therefore fails CI until somebody decides what it means.++Three things it refuses that a simpler version of itself would not+------------------------------------------------------------------+**A source whose fixtures yield no event types.** That is a gate passing for+the wrong reason — the shape this repository keeps finding, where the clean+result and the broken glob print the same word. Two of the ten sources here+had no vendor-payload fixture at all before this catalogue was written, and+a gate that shrugged at that would have certified them forever.++**A vendored copy that has drifted.** `go:embed` cannot reach above its own+package, so the binary carries a copy under+``services/ingest/internal/eventcatalog/catalog/``. Reading the directory+from disk at run time instead is a failure this service has already shipped:+the webhook templates were read from a path the Dockerfile never populated.+The two directories are held byte-identical in both directions, so neither+can gain, lose or alter a file alone.++**A classification outside the ladder.** Severity is five tiers everywhere in+this platform. A sixth would be invisible until a query filtered on it and+found nothing.++Usage+-----+ python3 scripts/check_event_catalog.py+ python3 scripts/check_event_catalog.py --json+ python3 scripts/check_event_catalog.py --self-test++``--repo-root`` overrides the tree. Every input is named before the verdict+and an empty read is a hard error, never a quiet pass.+"""++from __future__ import annotations++import argparse+import ast+import json+import re+import sys+from pathlib import Path+from typing import Any++sys.path.insert(0, str(Path(__file__).resolve().parent))++import yaml+from gate_toolkit import repo_root++CATALOG_REL = Path("schemas/event_catalog")+VENDORED_REL = Path("services/ingest/internal/eventcatalog/catalog")+LOADER_REL = Path("services/ingest/internal/eventcatalog/catalog.go")+CONNECTORS_REL = Path("services/connectors/app/connectors")++#: The five tiers, lowest first. Identical to `eventcatalog.Sensitivities` in+#: the Go, which this gate reads back out of the loader so the two cannot+#: diverge without failing here.+SENSITIVITIES = ("info", "low", "medium", "high", "critical")++#: An ATT&CK technique or sub-technique id. Nothing resolves these against+#: the corpus here — `scripts/` already owns that check for detection content+#: — but a malformed id is cheap to catch and travels onto every event.+ATTACK_ID = re.compile(r"^T\d{4}(\.\d{3})?$")+++class GateError(RuntimeError):+ """An input could not be read. Never downgraded to a passing result."""+++# --------------------------------------------------------------------------+# Catalogue+# --------------------------------------------------------------------------+def load_catalogs(directory: Path) -> dict[str, dict]:+ """source -> parsed catalogue, keyed by the `source:` field, not the name."""+ out: dict[str, dict] = {}+ for path in sorted(directory.glob("*.yaml")):+ parsed = yaml.safe_load(path.read_text(encoding="utf-8"))+ if not isinstance(parsed, dict):+ raise GateError(f"{path} does not parse to a mapping")+ source = parsed.get("source")+ if not source:+ raise GateError(f"{path} declares no `source`")+ if source in out:+ raise GateError(f"two catalogue files declare source {source!r}")+ parsed["__file__"] = path.name+ out[source] = parsed+ return out+++# --------------------------------------------------------------------------+# Fixture corpus+# --------------------------------------------------------------------------+def _records_from_json(path: Path):+ data = json.loads(path.read_text(encoding="utf-8"))+ stack = [data]+ while stack:+ item = stack.pop()+ if isinstance(item, dict):+ yield item+ stack.extend(item.values())+ elif isinstance(item, list):+ stack.extend(item)+++def _records_from_python(path: Path):+ """Every dict literal in a Python test file, as a plain dict.++ Only constant keys and values survive; a nested dict or list becomes a+ real nested structure so a declared path like ``operationName.value`` or+ ``events[].name`` resolves the same way it does at run time. Anything+ built by a call or a comprehension is skipped rather than guessed at.+ """++ def literal(node):+ if isinstance(node, ast.Constant):+ return node.value+ if isinstance(node, ast.Dict):+ out = {}+ for key, value in zip(node.keys, node.values, strict=False):+ if isinstance(key, ast.Constant) and isinstance(key.value, str):+ out[key.value] = literal(value)+ return out+ if isinstance(node, (ast.List, ast.Tuple)):+ return [literal(item) for item in node.elts]+ return None++ tree = ast.parse(path.read_text(encoding="utf-8"))+ for node in ast.walk(tree):+ if isinstance(node, ast.Dict):+ value = literal(node)+ if value:+ yield value+++def extract_at_path(record: dict, declared_path: str) -> str | None:+ """Read the event type out of one record, using the declared path.++ One list hop (``events[].name``) is supported, which is the only shape+ any source here needs. Mirrors ``eventcatalog.EventType`` in the Go; the+ two are small enough that a shared implementation would cost more than+ it saves, and the fixtures exercise both.+ """+ # Annotated rather than inferred: it starts as the record but walks into+ # whatever each segment finds, so `dict` -- what mypy takes from the first+ # assignment -- is wrong from the second one onward.+ cur: Any = record+ for segment in declared_path.split("."):+ list_hop = segment.endswith("[]")+ key = segment[:-2] if list_hop else segment+ if not isinstance(cur, dict):+ return None+ cur = cur.get(key)+ if list_hop:+ if not isinstance(cur, list) or not cur:+ return None+ cur = cur[0]+ if isinstance(cur, str) and cur.strip():+ return cur.strip()+ return None+++def event_types_in_fixtures(root: Path, catalog: dict) -> tuple[set[str], list[str]]:+ """(event types found, files that could not be read)."""+ declared_path = catalog["event_type_path"]+ requires = catalog.get("fixture_requires") or []+ found: set[str] = set()+ missing: list[str] = []++ for relative in catalog.get("fixture_files") or []:+ path = root / relative+ if not path.exists():+ missing.append(relative)+ continue+ records = _records_from_json(path) if path.suffix == ".json" else _records_from_python(path)+ for record in records:+ if not isinstance(record, dict):+ continue+ # A discriminator, where the source declares one. `action`+ # appears on records from several sources and in unrelated test+ # dictionaries, so without this the GitHub catalogue would be+ # asked to classify a VPC flow log's `ACCEPT`.+ if requires and not all(key in record for key in requires):+ continue+ value = extract_at_path(record, declared_path)+ if value:+ found.add(value)+ return found, missing+++# --------------------------------------------------------------------------+# Go side+# --------------------------------------------------------------------------+def parse_go_sensitivities(src: str) -> list[str]:+ match = re.search(r"var Sensitivities = \[\]Sensitivity\{(.*?)\}", src, re.DOTALL)+ if not match:+ raise GateError("could not find the Sensitivities slice in the loader")+ by_const = dict(re.findall(r'^\tSensitivity(\w+)\s+Sensitivity = "(\w+)"$', src, re.MULTILINE))+ return [by_const[name] for name in re.findall(r"Sensitivity(\w+)", match.group(1)) if name in by_const]+++def parse_declared_connectors(directory: Path) -> set[str]:+ out: set[str] = set()+ for path in sorted(directory.glob("*.py")):+ if path.name in {"__init__.py", "base.py"}:+ continue+ for match in re.finditer(r'(?m)^\s*connector_id(?:\s*:\s*\w+)?\s*=\s*"([^"]+)"', path.read_text(encoding="utf-8")):+ out.add(match.group(1))+ return out+++# --------------------------------------------------------------------------+# Gate+# --------------------------------------------------------------------------+def evaluate(+ catalogs: dict[str, dict],+ fixture_types: dict[str, set[str]],+ missing_fixtures: dict[str, list[str]],+ vendored_bytes: dict[str, bytes],+ source_bytes: dict[str, bytes],+ go_sensitivities: list[str],+ declared_connectors: set[str],+) -> tuple[list[tuple[str, str]], dict]:+ failures: list[tuple[str, str]] = []++ # LADDER. The Go's closed set and this gate's must be the same five.+ if list(go_sensitivities) != list(SENSITIVITIES):+ failures.append(+ (+ "ladder-mismatch",+ f"the loader declares the sensitivity ladder as {go_sensitivities}, and this gate checks "+ f"against {list(SENSITIVITIES)}; a value the gate accepts and the loader rejects fails at boot",+ )+ )++ # VENDORED COPY. Both directions: neither directory may gain, lose or+ # alter a file alone.+ for name in sorted(set(source_bytes) | set(vendored_bytes)):+ if name not in vendored_bytes:+ failures.append(("vendored-missing", f"{name} is in {CATALOG_REL} and not in {VENDORED_REL}; the binary would not carry it"))+ elif name not in source_bytes:+ failures.append(+ ("vendored-orphan", f"{name} is in {VENDORED_REL} and not in {CATALOG_REL}; the binary carries a file nobody edits")+ )+ elif source_bytes[name] != vendored_bytes[name]:+ failures.append(+ (+ "vendored-drift",+ f"{name} differs between {CATALOG_REL} and {VENDORED_REL}; the file a reviewer edits is not the one the binary loads",+ )+ )++ for source, catalog in sorted(catalogs.items()):+ file_name = catalog["__file__"]++ # The source must be a connector type that exists.+ if source not in declared_connectors:+ failures.append(("source-names-nothing", f"{file_name} declares source {source!r}, which no connector declares"))++ # Declared fixtures must exist, or the corpus is a promise.+ for relative in missing_fixtures.get(source, []):+ failures.append(("fixture-missing", f"{file_name} declares the fixture {relative}, which does not exist"))++ events = catalog.get("events") or {}+ unclassified = catalog.get("unclassified") or {}++ for event_type, classification in sorted(events.items()):+ if not isinstance(classification, dict):+ failures.append(("entry-malformed", f"{file_name}: {event_type!r} is not a mapping"))+ continue+ if not classification.get("action"):+ failures.append(("action-missing", f"{file_name}: {event_type!r} declares no normalized action"))+ sensitivity = classification.get("sensitivity")+ if sensitivity not in SENSITIVITIES:+ failures.append(+ (+ "sensitivity-invalid",+ f"{file_name}: {event_type!r} declares sensitivity {sensitivity!r}, which is not one of {', '.join(SENSITIVITIES)}",+ )+ )+ for technique in classification.get("attack") or []:+ if not ATTACK_ID.match(str(technique)):+ failures.append(("attack-malformed", f"{file_name}: {event_type!r} hints at {technique!r}, which is not an ATT&CK id"))++ for event_type, reason in sorted(unclassified.items()):+ if event_type in events:+ failures.append(+ (+ "entry-both",+ f"{file_name}: {event_type!r} is both classified and listed as deliberately unclassified",+ )+ )+ if not isinstance(reason, str) or len(reason) < 15:+ failures.append(+ (+ "excuse-thin",+ f"{file_name}: {event_type!r} is excused with {reason!r}, which does not say enough for a "+ "reviewer to disagree with it",+ )+ )++ # The corpus itself. An empty one is a pass for the wrong reason.+ found = fixture_types.get(source, set())+ if not found:+ failures.append(+ (+ "fixture-corpus-empty",+ f"{file_name} declares {len(catalog.get('fixture_files') or [])} fixture file(s) and no event "+ f"type was found at {catalog['event_type_path']!r} in any of them. This gate would pass over "+ "this source forever without checking anything",+ )+ )+ continue++ for event_type in sorted(found - set(events) - set(unclassified)):+ failures.append(+ (+ "event-type-unclassified",+ f"{source}: the event type {event_type!r} appears in the fixtures and is neither classified "+ f"nor listed as deliberately unclassified in {file_name}",+ )+ )++ stats = {+ "sources": sorted(catalogs),+ "classified": sum(len(c.get("events") or {}) for c in catalogs.values()),+ "excused": sum(len(c.get("unclassified") or {}) for c in catalogs.values()),+ "fixture_types": {source: sorted(types) for source, types in sorted(fixture_types.items())},+ "by_sensitivity": {+ tier: sum(+ 1+ for c in catalogs.values()+ for e in (c.get("events") or {}).values()+ if isinstance(e, dict) and e.get("sensitivity") == tier+ )+ for tier in SENSITIVITIES+ },+ }+ return failures, stats+++def load(root: Path) -> dict:+ catalog_dir = root / CATALOG_REL+ vendored_dir = root / VENDORED_REL+ loader = root / LOADER_REL+ connectors = root / CONNECTORS_REL+ for path in (catalog_dir, vendored_dir, loader, connectors):+ if not path.exists():+ raise GateError(f"expected input does not exist: {path}")++ catalogs = load_catalogs(catalog_dir)+ fixture_types: dict[str, set[str]] = {}+ missing_fixtures: dict[str, list[str]] = {}+ for source, catalog in catalogs.items():+ found, missing = event_types_in_fixtures(root, catalog)+ fixture_types[source] = found+ missing_fixtures[source] = missing++ data = {+ "catalogs": catalogs,+ "fixture_types": fixture_types,+ "missing_fixtures": missing_fixtures,+ "source_bytes": {p.name: p.read_bytes() for p in sorted(catalog_dir.glob("*.yaml"))},+ "vendored_bytes": {p.name: p.read_bytes() for p in sorted(vendored_dir.glob("*.yaml"))},+ "go_sensitivities": parse_go_sensitivities(loader.read_text(encoding="utf-8")),+ "declared_connectors": parse_declared_connectors(connectors),+ }+ for name in ("catalogs", "source_bytes", "vendored_bytes", "go_sensitivities", "declared_connectors"):+ if not data[name]:+ raise GateError(f"parsed zero {name} — refusing to report a clean tree from an empty read")+ return data+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ parser.add_argument("--repo-root", type=Path, default=repo_root())+ parser.add_argument("--json", action="store_true")+ parser.add_argument("--self-test", action="store_true", help="prove the gate detects what it claims to")+ args = parser.parse_args(argv)++ if args.self_test:+ return self_test(args.repo_root.resolve())++ root = args.repo_root.resolve()+ try:+ data = load(root)+ except GateError as exc:+ print(f"check_event_catalog: FAILED to read the tree: {exc}", file=sys.stderr)+ return 2++ failures, stats = evaluate(**data)++ if args.json:+ report = {"repo_root": str(root), **stats, "failures": [{"code": c, "detail": d} for c, d in failures]}+ print(json.dumps(report, indent=2, sort_keys=True))+ return 1 if failures else 0++ print(f"repo root {root}")+ print(f"catalogue {CATALOG_REL} ({len(stats['sources'])} sources, {stats['classified']} event types)")+ print(f"vendored copy {VENDORED_REL} ({len(data['vendored_bytes'])} files, byte-compared both ways)")+ print(f"loader {LOADER_REL} (ladder: {', '.join(data['go_sensitivities'])})")+ print()+ for source in stats["sources"]:+ found = stats["fixture_types"][source]+ catalog = data["catalogs"][source]+ print(f" {source:18s} {len(catalog.get('events') or {}):>3} classified {len(found):>3} in fixtures")+ print()+ tiers = " ".join(f"{tier}={count}" for tier, count in stats["by_sensitivity"].items())+ print(f"by sensitivity {tiers}")+ print()+ if failures:+ print(f"FAIL — {len(failures)} finding(s):")+ for code, detail in failures:+ print(f" [{code}] {detail}")+ return 1+ print("OK — every event type in every declared fixture is classified or excused,")+ print(" every sensitivity is on the ladder, and the vendored copy matches.")+ return 0+++def self_test(root: Path) -> int:+ """Inject each defect the gate claims to catch and require it to bite."""+ try:+ base = load(root)+ except GateError as exc:+ print(f"self-test: cannot read the tree: {exc}", file=sys.stderr)+ return 2++ clean, _ = evaluate(**base)+ if clean:+ print("self-test: the unmodified tree already fails; fix that first", file=sys.stderr)+ for code, detail in clean:+ print(f" [{code}] {detail}", file=sys.stderr)+ return 1++ def mutate(**overrides) -> dict:+ import copy++ data = copy.deepcopy(base)+ for key, fn in overrides.items():+ fn(data[key])+ return data++ any_source = sorted(base["catalogs"])[0]+ any_file = sorted(base["source_bytes"])[0]++ cases = [+ (+ "CORPUS: a fixture event type that is neither classified nor excused",+ "event-type-unclassified",+ mutate(fixture_types=lambda d: d[any_source].add("SomethingNobodyClassified")),+ ),+ (+ "CORPUS: a source whose fixtures yield nothing, so the gate would check nothing",+ "fixture-corpus-empty",+ mutate(fixture_types=lambda d: d.__setitem__(any_source, set())),+ ),+ (+ "CORPUS: a declared fixture file that does not exist",+ "fixture-missing",+ mutate(missing_fixtures=lambda d: d.__setitem__(any_source, ["services/connectors/tests/not_there.py"])),+ ),+ (+ "LADDER: a sensitivity outside the five tiers",+ "sensitivity-invalid",+ mutate(catalogs=lambda d: d[any_source]["events"].__setitem__("X", {"action": "x", "sensitivity": "severe"})),+ ),+ (+ "LADDER: the loader and the gate disagreeing about the ladder",+ "ladder-mismatch",+ mutate(go_sensitivities=lambda lst: lst.append("catastrophic")),+ ),+ (+ "ENTRY: a classification with no normalized action",+ "action-missing",+ mutate(catalogs=lambda d: d[any_source]["events"].__setitem__("Y", {"sensitivity": "low"})),+ ),+ (+ "ENTRY: a malformed ATT&CK hint",+ "attack-malformed",+ mutate(+ catalogs=lambda d: d[any_source]["events"].__setitem__("Z", {"action": "z", "sensitivity": "low", "attack": ["TA0001"]})+ ),+ ),+ (+ "ENTRY: an excuse too thin to review",+ "excuse-thin",+ mutate(catalogs=lambda d: d[any_source].__setitem__("unclassified", {"Q": "noisy"})),+ ),+ (+ "ENTRY: an event type both classified and excused",+ "entry-both",+ mutate(+ catalogs=lambda d: d[any_source].__setitem__(+ "unclassified", {next(iter(d[any_source]["events"])): "a reason long enough to pass the length check"}+ )+ ),+ ),+ (+ "SOURCE: a catalogue for a connector that does not exist",+ "source-names-nothing",+ mutate(catalogs=lambda d: d.__setitem__("acme_xdr_9000", {**d[any_source], "__file__": "acme.yaml"})),+ ),+ (+ "VENDORED: the embedded copy differs from the file a reviewer edits",+ "vendored-drift",+ mutate(vendored_bytes=lambda d: d.__setitem__(any_file, b"# drifted\n")),+ ),+ (+ "VENDORED: a catalogue file the binary does not carry",+ "vendored-missing",+ mutate(source_bytes=lambda d: d.__setitem__("new_source.yaml", b"source: x\n")),+ ),+ (+ "VENDORED: an embedded file nobody edits",+ "vendored-orphan",+ mutate(vendored_bytes=lambda d: d.__setitem__("orphan.yaml", b"source: x\n")),+ ),+ ]++ print(f"self-test against {root}")+ print("clean tree: 0 failures (the baseline every case below perturbs)\n")+ ok = True+ for description, expected_code, data in cases:+ found, _ = evaluate(**data)+ codes = {code for code, _ in found}+ caught = expected_code in codes+ ok &= caught+ print(f" {'PASS' if caught else 'FAIL'} {description}")+ print(f" expected [{expected_code}] got {sorted(codes) or 'nothing'}")++ empty_refused = True+ try:+ load(Path("/nonexistent-tree-for-the-self-test"))+ empty_refused = False+ except GateError:+ # Refusing an empty tree is the behaviour under test, so the+ # exception is the pass and there is nothing to handle.+ pass+ ok &= empty_refused+ print(f" {'PASS' if empty_refused else 'FAIL'} TREE: a tree with no inputs is refused rather than reported clean")++ print()+ if not ok:+ print("self-test FAILED: the gate did not catch what it claims to catch")+ return 1+ print(f"self-test OK: {len(cases)} injected defects plus the empty-tree case, each caught")+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/services/connectors/tests/connectors/test_event_catalog_fixtures.py b/services/connectors/tests/connectors/test_event_catalog_fixtures.pynew file mode 100644index 000000000..27f778d9c--- /dev/null+++ b/services/connectors/tests/connectors/test_event_catalog_fixtures.py@@ -0,0 +1,146 @@+"""Vendor-shaped fixtures for the two identity and SaaS sources that had none.++Okta and Slack are two of the ten sources the event catalogue covers, and+before this file neither had a single recorded vendor payload anywhere in the+connectors suite — their only coverage was the schema and conformance checks+that never see a record. The catalogue gate is grown from fixtures, so a+source with no fixture is a source the gate passes over forever while+reporting OK.++These drive the real connectors' ``normalize()`` over the recorded shapes, so+the fixtures are exercised rather than merely present, and the event type the+catalogue classifies is asserted to survive the connector.++Shapes follow each vendor's published log schema:++ Okta System Log+ https://developer.okta.com/docs/reference/api/system-log/+ Slack Audit Logs API+ https://docs.slack.dev/admins/audit-logs-api+"""++from __future__ import annotations++import ipaddress+import json+from pathlib import Path++import pytest+import yaml+from app.connectors.okta import OktaConnector+from app.connectors.slack_audit import SlackAuditConnector++_FIXTURES = Path(__file__).resolve().parent.parent / "fixtures"+_CATALOG = Path(__file__).resolve().parents[4] / "schemas" / "event_catalog"+++def _fixture(name: str) -> list[dict]:+ return json.loads((_FIXTURES / name / "sample_event.json").read_text())+++def _catalog(name: str) -> dict:+ return yaml.safe_load((_CATALOG / f"{name}.yaml").read_text())+++# ── Okta ───────────────────────────────────────────────────────────────────++[email protected](scope="module")+def okta_events() -> list[dict]:+ return _fixture("okta")+++def test_okta_normalizes_every_recorded_shape(okta_events: list[dict]) -> None:+ connector = OktaConnector(domain="https://example.okta.com", api_token="t")+ for raw in okta_events:+ out = connector.normalize(raw)+ assert out["source"] == "okta"+ assert out["event_type"] == raw["eventType"]+ assert out["raw_event"] is raw, "the vendor record must survive under raw_event, never `raw`"+ assert out["severity"] in {"info", "low", "medium", "high", "critical"}+++def test_okta_outcome_and_lockout_drive_severity(okta_events: list[dict]) -> None:+ """The two escalations the connector actually makes, on recorded records+ rather than on a payload written to match the code."""+ connector = OktaConnector(domain="https://example.okta.com", api_token="t")+ by_type: dict[str, list[dict]] = {}+ for raw in okta_events:+ by_type.setdefault(raw["eventType"], []).append(connector.normalize(raw))++ failures = [o for o in by_type["user.session.start"] if "INVALID_CREDENTIALS" in (o["description"] or "")]+ assert failures and all(o["severity"] == "medium" for o in failures)+ assert all(o["severity"] == "high" for o in by_type["user.account.lock"])+ assert all(o["severity"] == "high" for o in by_type["security.request.blocked"])+++def test_every_okta_fixture_event_type_is_in_the_catalogue(okta_events: list[dict]) -> None:+ catalog = _catalog("okta")+ known = set(catalog["events"]) | set(catalog.get("unclassified") or {})+ assert {raw["eventType"] for raw in okta_events} <= known+++# ── Slack ──────────────────────────────────────────────────────────────────++[email protected](scope="module")+def slack_events() -> list[dict]:+ return _fixture("slack_audit")+++def test_slack_normalizes_every_recorded_shape(slack_events: list[dict]) -> None:+ connector = SlackAuditConnector(access_token="t")+ for raw in slack_events:+ out = connector.normalize(raw)+ assert out["source"] == "slack_audit"+ assert out["title"].endswith(raw["action"])+ assert out["raw_event"] is raw+ assert out["alert_id"] == raw["id"]+++def test_every_slack_fixture_event_type_is_in_the_catalogue(slack_events: list[dict]) -> None:+ catalog = _catalog("slack_audit")+ known = set(catalog["events"]) | set(catalog.get("unclassified") or {})+ assert {raw["action"] for raw in slack_events} <= known+++# ── The fixtures themselves ────────────────────────────────────────────────++[email protected](("source", "key"), [("okta", "eventType"), ("slack_audit", "action")])+def test_a_fixture_carries_more_than_one_event_type(source: str, key: str) -> None:+ """A single-shape fixture makes the catalogue gate pass on one entry and+ say nothing about the rest, which is the vacuous case these exist to+ avoid."""+ records = _fixture(source)+ assert len({r[key] for r in records}) >= 5++[email protected](("source", "ip_keys"), [("okta", {"ipAddress"}), ("slack_audit", {"ip_address"})])+def test_a_fixture_uses_documentation_addresses_only(source: str, ip_keys: set[str]) -> None:+ """A recorded fixture must not carry a real routable address.++ Fixtures are read as examples by tests, hunts and anyone learning the+ shape, and an address that resolves somewhere is an address somebody+ eventually queries. Checked on the keys that actually hold addresses+ rather than by scanning the blob — a first version of this test regexed+ the whole JSON and failed on the version in `Chrome/124.0.0.0`.+ """+ addresses: list[str] = []++ def walk(node: object) -> None:+ if isinstance(node, dict):+ for key, value in node.items():+ if key in ip_keys and isinstance(value, str):+ addresses.append(value)+ walk(value)+ elif isinstance(node, list):+ for item in node:+ walk(item)++ walk(_fixture(source))+ assert addresses, f"{source} fixture carries no address at {ip_keys}; the check would be vacuous"+ documentation = (ipaddress.ip_network(net) for net in ("192.0.2.0/24", "198.51.100.0/24", "203.0.113.0/24"))+ allowed = list(documentation)+ routable = [a for a in addresses if not any(ipaddress.ip_address(a) in net for net in allowed)]+ assert not routable, f"{source} fixture carries non-documentation addresses: {routable}"diff --git a/services/connectors/tests/fixtures/okta/sample_event.json b/services/connectors/tests/fixtures/okta/sample_event.jsonnew file mode 100644index 000000000..e4a3c5b4e--- /dev/null+++ b/services/connectors/tests/fixtures/okta/sample_event.json@@ -0,0 +1,90 @@+[+ {+ "uuid": "a1b2c3d4-0000-4000-8000-000000000001",+ "published": "2026-10-07T09:14:02.413Z",+ "eventType": "user.session.start",+ "displayMessage": "User login to Okta",+ "severity": "INFO",+ "actor": {+ "id": "00u1a2b3c4d5e6f7g8h9",+ "type": "User",+ "alternateId": "[email protected]",+ "displayName": "Alice Example"+ },+ "client": {+ "ipAddress": "203.0.113.10",+ "userAgent": {+ "rawUserAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",+ "os": "Mac OS X",+ "browser": "CHROME"+ },+ "zone": "null",+ "device": "Computer"+ },+ "outcome": { "result": "SUCCESS" },+ "target": [+ { "id": "0oa1a2b3c4d5e6f7g8h9", "type": "AppInstance", "alternateId": "AWS Account Federation", "displayName": "AWS" }+ ]+ },+ {+ "uuid": "a1b2c3d4-0000-4000-8000-000000000002",+ "published": "2026-10-07T09:15:40.001Z",+ "eventType": "user.session.start",+ "displayMessage": "User login to Okta",+ "severity": "WARN",+ "actor": { "id": "00u1a2b3c4d5e6f7g8h9", "type": "User", "alternateId": "[email protected]", "displayName": "Alice Example" },+ "client": { "ipAddress": "198.51.100.44" },+ "outcome": { "result": "FAILURE", "reason": "INVALID_CREDENTIALS" }+ },+ {+ "uuid": "a1b2c3d4-0000-4000-8000-000000000003",+ "published": "2026-10-07T09:20:11.500Z",+ "eventType": "user.account.lock",+ "displayMessage": "Max sign in attempts exceeded",+ "severity": "WARN",+ "actor": { "id": "00u1a2b3c4d5e6f7g8h9", "type": "User", "alternateId": "[email protected]", "displayName": "Alice Example" },+ "client": { "ipAddress": "198.51.100.44" },+ "outcome": { "result": "SUCCESS" }+ },+ {+ "uuid": "a1b2c3d4-0000-4000-8000-000000000004",+ "published": "2026-10-07T09:31:07.220Z",+ "eventType": "user.account.privilege.grant",+ "displayMessage": "Grant user privilege",+ "severity": "INFO",+ "actor": { "id": "00uADMIN0000000000", "type": "User", "alternateId": "[email protected]", "displayName": "Admin Example" },+ "client": { "ipAddress": "203.0.113.11" },+ "outcome": { "result": "SUCCESS" },+ "target": [{ "id": "00u1a2b3c4d5e6f7g8h9", "type": "User", "alternateId": "[email protected]" }]+ },+ {+ "uuid": "a1b2c3d4-0000-4000-8000-000000000005",+ "published": "2026-10-07T09:42:55.900Z",+ "eventType": "system.api_token.create",+ "displayMessage": "Create API token",+ "severity": "INFO",+ "actor": { "id": "00uADMIN0000000000", "type": "User", "alternateId": "[email protected]", "displayName": "Admin Example" },+ "client": { "ipAddress": "203.0.113.11" },+ "outcome": { "result": "SUCCESS" }+ },+ {+ "uuid": "a1b2c3d4-0000-4000-8000-000000000006",+ "published": "2026-10-07T09:50:02.000Z",+ "eventType": "user.mfa.factor.deactivate",+ "displayMessage": "Deactivate factor for user",+ "severity": "INFO",+ "actor": { "id": "00uSYSTEM000000000", "type": "SystemPrincipal", "displayName": "Okta System" },+ "client": { "ipAddress": "203.0.113.12" },+ "outcome": { "result": "SUCCESS" }+ },+ {+ "uuid": "a1b2c3d4-0000-4000-8000-000000000007",+ "published": "2026-10-07T10:02:19.100Z",+ "eventType": "security.request.blocked",+ "displayMessage": "Request from suspicious actor blocked",+ "severity": "WARN",+ "actor": { "id": "unknown", "type": "User", "displayName": "unknown" },+ "client": { "ipAddress": "198.51.100.200" },+ "outcome": { "result": "DENY", "reason": "Suspicious activity" }+ }+]diff --git a/services/connectors/tests/fixtures/slack_audit/sample_event.json b/services/connectors/tests/fixtures/slack_audit/sample_event.jsonnew file mode 100644index 000000000..fdb6d2da0--- /dev/null+++ b/services/connectors/tests/fixtures/slack_audit/sample_event.json@@ -0,0 +1,57 @@+[+ {+ "id": "0123a45b-6c7d-8e90-1234-56789abcdef0",+ "date_create": 1791367200,+ "action": "user_login",+ "actor": {+ "type": "user",+ "user": { "id": "W123AB456", "name": "alice", "email": "[email protected]", "team": "T1234ABCD" }+ },+ "entity": { "type": "user", "user": { "id": "W123AB456", "name": "alice", "email": "[email protected]" } },+ "context": {+ "location": { "type": "enterprise", "id": "E1701", "name": "Example", "domain": "example" },+ "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",+ "ip_address": "203.0.113.21"+ }+ },+ {+ "id": "0123a45b-6c7d-8e90-1234-56789abcdef1",+ "date_create": 1791367800,+ "action": "file_downloaded",+ "actor": { "type": "user", "user": { "id": "W123AB456", "name": "alice", "email": "[email protected]" } },+ "entity": { "type": "file", "file": { "id": "F0123ABCD", "name": "q4-forecast.xlsx", "filetype": "xlsx", "title": "Q4 forecast" } },+ "context": { "ua": "Slack/4.37.101 CFNetwork/1494.0.7 Darwin/23.4.0", "ip_address": "203.0.113.21" }+ },+ {+ "id": "0123a45b-6c7d-8e90-1234-56789abcdef2",+ "date_create": 1791368400,+ "action": "app_installed",+ "actor": { "type": "user", "user": { "id": "WADMIN0001", "name": "admin", "email": "[email protected]" } },+ "entity": { "type": "app", "app": { "id": "A0123ABCD", "name": "Unvetted Exporter", "is_distributed": false, "scopes": ["files:read", "channels:history"] } },+ "context": { "ua": "Mozilla/5.0", "ip_address": "203.0.113.22" }+ },+ {+ "id": "0123a45b-6c7d-8e90-1234-56789abcdef3",+ "date_create": 1791369000,+ "action": "user_session_reset_by_admin",+ "actor": { "type": "user", "user": { "id": "WADMIN0001", "name": "admin", "email": "[email protected]" } },+ "entity": { "type": "user", "user": { "id": "W123AB456", "name": "alice" } },+ "context": { "ua": "Mozilla/5.0", "ip_address": "203.0.113.22" }+ },+ {+ "id": "0123a45b-6c7d-8e90-1234-56789abcdef4",+ "date_create": 1791369600,+ "action": "public_channel_created",+ "actor": { "type": "user", "user": { "id": "W123AB456", "name": "alice" } },+ "entity": { "type": "channel", "channel": { "id": "C0123ABCD", "name": "incident-2026-10-07", "privacy": "public", "is_shared": false } },+ "context": { "ua": "Mozilla/5.0", "ip_address": "203.0.113.21" }+ },+ {+ "id": "0123a45b-6c7d-8e90-1234-56789abcdef5",+ "date_create": 1791370200,+ "action": "organization_export_started",+ "actor": { "type": "user", "user": { "id": "WOWNER0001", "name": "owner", "email": "[email protected]" } },+ "entity": { "type": "workspace", "workspace": { "id": "T1234ABCD", "name": "Example", "domain": "example" } },+ "context": { "ua": "Mozilla/5.0", "ip_address": "203.0.113.23" }+ }+]diff --git a/services/ingest/internal/eventcatalog/catalog.go b/services/ingest/internal/eventcatalog/catalog.gonew file mode 100644index 000000000..75f5b7e5d--- /dev/null+++ b/services/ingest/internal/eventcatalog/catalog.go@@ -0,0 +1,275 @@+// Package eventcatalog loads the event classification catalogue at boot and+// answers "what does this vendor event type mean" for every normalized event.+//+// The catalogue is a reviewed data file per source, mapping the vendor's own+// event type onto a normalized action, a sensitivity on the platform's+// five-tier ladder, and an optional ATT&CK hint. The authoritative copies+// live at `schemas/event_catalog/<source>.yaml`; the copies under `catalog/`+// here are byte-identical and are what the binary carries.+//+// Why a vendored copy rather than reading `schemas/` at run time+// --------------------------------------------------------------+// `go:embed` cannot reach above its own package, and the alternative —+// reading the directory from disk at a configurable path — is a failure this+// service has already shipped once: the webhook templates were read from+// `/app/templates`, a path the Dockerfile never populated, so every+// deployment answered 503 for every template behind a startup warning+// nothing was watching (see templates.go). Embedding removes the class+// rather than the instance, and `scripts/check_event_catalog.py` holds the+// two directories byte-identical in both directions so the copy cannot+// drift from the file a reviewer edits.+//+// Why a sensitivity does not change a severity+// --------------------------------------------+// Sensitivity says how security-relevant an event *type* is. The vendor's+// own severity on the record, and the OCSF class, still decide whether+// fusion promotes it. Letting a `critical` sensitivity force an alert would+// make this data file able to flood a queue from a one-line edit, and the+// promotion contract belongs where it already is. The classification rides+// on the event for detections, hunts and triage to read.+package eventcatalog++import (+ "embed"+ "fmt"+ "io/fs"+ "path"+ "sort"+ "strings"++ "gopkg.in/yaml.v3"+)++//go:embed catalog/*.yaml+var files embed.FS++const catalogDir = "catalog"++// Sensitivity is how security-relevant an event type is, on the same+// five-tier ladder the rest of the platform uses.+type Sensitivity string++const (+ SensitivityInfo Sensitivity = "info"+ SensitivityLow Sensitivity = "low"+ SensitivityMedium Sensitivity = "medium"+ SensitivityHigh Sensitivity = "high"+ SensitivityCritical Sensitivity = "critical"+)++// Sensitivities is the closed set, lowest first. A value outside it is a+// load error rather than a silently-kept string: the ladder is five tiers+// everywhere in this platform and a sixth would be invisible until a query+// filtered on it and found nothing.+var Sensitivities = []Sensitivity{+ SensitivityInfo, SensitivityLow, SensitivityMedium, SensitivityHigh, SensitivityCritical,+}++// Classification is one catalogue entry.+type Classification struct {+ Action string `yaml:"action"`+ Sensitivity Sensitivity `yaml:"sensitivity"`+ ATTACK []string `yaml:"attack,omitempty"`+}++// sourceFile is the on-disk shape of one `<source>.yaml`.+type sourceFile struct {+ Source string `yaml:"source"`+ // EventTypePath is where the vendor's event type lives on its own+ // record, in the dotted form `a.b` with `a[].b` for a list.+ EventTypePath string `yaml:"event_type_path"`+ // FixtureRequires are keys a record must also carry for the gate to+ // read it as this source's event. `action` alone appears on records+ // from several sources and in unrelated test dictionaries.+ FixtureRequires []string `yaml:"fixture_requires,omitempty"`+ FixtureFiles []string `yaml:"fixture_files"`+ Events map[string]Classification `yaml:"events"`+ // Unclassified is event type -> why it is deliberately not classified.+ // Read here as well as by the gate so a loaded catalogue can say "this+ // type is known and deliberately unclassified", which is a different+ // answer from "never seen".+ Unclassified map[string]string `yaml:"unclassified"`+}++// Catalog is every source's classifications, loaded.+type Catalog struct {+ bySource map[string]map[string]Classification+ unclassified map[string]map[string]string+ paths map[string]string+}++// Load reads the embedded catalogue. It returns an error rather than a+// partial catalogue: a source that failed to parse would answer "never seen"+// for every one of its event types, which is indistinguishable from a source+// nobody has written a catalogue for.+func Load() (*Catalog, error) {+ entries, err := fs.ReadDir(files, catalogDir)+ if err != nil {+ return nil, fmt.Errorf("reading the embedded catalogue: %w", err)+ }+ c := &Catalog{+ bySource: map[string]map[string]Classification{},+ unclassified: map[string]map[string]string{},+ paths: map[string]string{},+ }+ for _, entry := range entries {+ if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".yaml") {+ continue+ }+ raw, err := files.ReadFile(path.Join(catalogDir, entry.Name()))+ if err != nil {+ return nil, fmt.Errorf("reading %s: %w", entry.Name(), err)+ }+ var parsed sourceFile+ if err := yaml.Unmarshal(raw, &parsed); err != nil {+ return nil, fmt.Errorf("parsing %s: %w", entry.Name(), err)+ }+ if parsed.Source == "" {+ return nil, fmt.Errorf("%s declares no source", entry.Name())+ }+ if parsed.EventTypePath == "" {+ return nil, fmt.Errorf("%s declares no event_type_path, so nothing can find its event type", entry.Name())+ }+ if existing, dup := c.paths[parsed.Source]; dup {+ return nil, fmt.Errorf("two catalogue files declare source %q (%s and %s)", parsed.Source, existing, entry.Name())+ }+ for eventType, classification := range parsed.Events {+ if !validSensitivity(classification.Sensitivity) {+ return nil, fmt.Errorf(+ "%s: %q declares sensitivity %q, which is not one of the five tiers",+ entry.Name(), eventType, classification.Sensitivity,+ )+ }+ if classification.Action == "" {+ return nil, fmt.Errorf("%s: %q declares no normalized action", entry.Name(), eventType)+ }+ }+ c.bySource[parsed.Source] = parsed.Events+ c.unclassified[parsed.Source] = parsed.Unclassified+ c.paths[parsed.Source] = parsed.EventTypePath+ }+ if len(c.bySource) == 0 {+ return nil, fmt.Errorf("the embedded catalogue is empty — refusing to load a catalogue that classifies nothing")+ }+ return c, nil+}++func validSensitivity(value Sensitivity) bool {+ for _, s := range Sensitivities {+ if s == value {+ return true+ }+ }+ return false+}++// Lookup returns the classification for one source's event type.+//+// The second return distinguishes the three answers that matter: a hit, a+// type the catalogue lists as deliberately unclassified, and one the+// catalogue has never seen. Only the third is worth a warning on the event.+func (c *Catalog) Lookup(source, eventType string) (Classification, Status) {+ if c == nil {+ return Classification{}, StatusNoCatalog+ }+ events, ok := c.bySource[source]+ if !ok {+ return Classification{}, StatusNoCatalog+ }+ if classification, hit := events[eventType]; hit {+ return classification, StatusClassified+ }+ if _, excused := c.unclassified[source][eventType]; excused {+ return Classification{}, StatusDeliberatelyUnclassified+ }+ return Classification{}, StatusUnknown+}++// Status is which of the three answers Lookup gave.+type Status string++const (+ StatusClassified Status = "classified"+ // StatusDeliberatelyUnclassified: somebody looked and decided.+ StatusDeliberatelyUnclassified Status = "unclassified"+ // StatusUnknown: this source has a catalogue and this type is not in+ // it. The only one worth telling an operator about, because it is how+ // the catalogue learns it is behind the vendor.+ StatusUnknown Status = "unknown"+ // StatusNoCatalog: no catalogue for this source at all.+ StatusNoCatalog Status = "no_catalog"+)++// EventTypePath is where a source keeps its event type, as declared.+func (c *Catalog) EventTypePath(source string) (string, bool) {+ if c == nil {+ return "", false+ }+ p, ok := c.paths[source]+ return p, ok+}++// Sources is every source with a catalogue, sorted.+func (c *Catalog) Sources() []string {+ if c == nil {+ return nil+ }+ out := make([]string, 0, len(c.bySource))+ for source := range c.bySource {+ out = append(out, source)+ }+ sort.Strings(out)+ return out+}++// Size is the total number of classified event types, for the boot log.+func (c *Catalog) Size() int {+ if c == nil {+ return 0+ }+ total := 0+ for _, events := range c.bySource {+ total += len(events)+ }+ return total+}++// EventType reads the vendor's event type out of a record, using the path+// the source declared.+//+// Supports one list hop (`events[].name`), which is the only shape any+// source here needs: Workspace bundles several events under one activity.+// A deeper grammar would be a parser nothing in the corpus exercises.+func EventType(record map[string]any, declaredPath string) (string, bool) {+ if record == nil || declaredPath == "" {+ return "", false+ }+ var cur any = record+ for _, segment := range strings.Split(declaredPath, ".") {+ listHop := strings.HasSuffix(segment, "[]")+ key := strings.TrimSuffix(segment, "[]")++ asMap, ok := cur.(map[string]any)+ if !ok {+ return "", false+ }+ cur = asMap[key]++ if listHop {+ items, ok := cur.([]any)+ if !ok || len(items) == 0 {+ return "", false+ }+ // The first element drives the classification, matching the+ // connectors, which already pick `events[0]` to drive severity+ // and the title.+ cur = items[0]+ }+ }+ value, ok := cur.(string)+ if !ok {+ return "", false+ }+ value = strings.TrimSpace(value)+ return value, value != ""+}diff --git a/services/ingest/internal/eventcatalog/catalog/aws_cloudtrail.yaml b/services/ingest/internal/eventcatalog/catalog/aws_cloudtrail.yamlnew file mode 100644index 000000000..311a7b9e7--- /dev/null+++ b/services/ingest/internal/eventcatalog/catalog/aws_cloudtrail.yaml@@ -0,0 +1,36 @@+# AWS CloudTrail. Event types are the API call names in `eventName`.+#+# Sensitivity reflects what the call class means, not what any one+# occurrence scored: DescribeInstances is reconnaissance-shaped and happens+# thousands of times an hour in a healthy account, DeleteTrail is an+# incident on its own reading.+source: aws_cloudtrail+event_type_path: eventName+fixture_files:+ - services/connectors/tests/test_aws_cloudtrail.py+events:+ ConsoleLogin:+ action: console.login+ sensitivity: medium+ attack: [T1078.004]+ CreateAccessKey:+ action: create.access.key+ sensitivity: high+ attack: [T1098.001]+ DeleteTrail:+ # Destroying the audit trail is the one event where the absence of+ # everything after it is the finding.+ action: delete.trail+ sensitivity: critical+ attack: [T1562.008]+ PutEventSelectors:+ # Narrowing what the trail records is the quieter half of the same+ # technique, which is why it sits one tier below rather than at info.+ action: put.event.selectors+ sensitivity: high+ attack: [T1562.008]+ DescribeInstances:+ action: describe.instances+ sensitivity: info+ attack: [T1580]+unclassified: {}diff --git a/services/ingest/internal/eventcatalog/catalog/azure_activity.yaml b/services/ingest/internal/eventcatalog/catalog/azure_activity.yamlnew file mode 100644index 000000000..cae5b0a42--- /dev/null+++ b/services/ingest/internal/eventcatalog/catalog/azure_activity.yaml@@ -0,0 +1,16 @@+# Azure Activity Log. Event types are the ARM operation names in+# `operationName.value`, which are `Provider/resourceType/action`.+source: azure_activity+event_type_path: operationName.value+fixture_files:+ - services/connectors/tests/test_azure_connectors.py+events:+ Microsoft.Authorization/roleAssignments/write:+ action: authorization.role.assignment.write+ sensitivity: high+ attack: [T1098.003]+ Microsoft.Storage/storageAccounts/read:+ action: storage.account.read+ sensitivity: info+ attack: [T1580]+unclassified: {}diff --git a/services/ingest/internal/eventcatalog/catalog/azure_entra.yaml b/services/ingest/internal/eventcatalog/catalog/azure_entra.yamlnew file mode 100644index 000000000..e2274665c--- /dev/null+++ b/services/ingest/internal/eventcatalog/catalog/azure_entra.yaml@@ -0,0 +1,18 @@+# Microsoft Entra ID audit logs. Event types are the+# `activityDisplayName` strings, which are prose rather than identifiers —+# Microsoft's own vocabulary, kept verbatim so a lookup against a live+# tenant's logs matches.+source: azure_entra+event_type_path: activityDisplayName+fixture_files:+ - services/connectors/tests/test_azure_connectors.py+events:+ Add member to role:+ action: add.member.to.role+ sensitivity: high+ attack: [T1098.003]+ Add user:+ action: add.user+ sensitivity: medium+ attack: [T1136.003]+unclassified: {}diff --git a/services/ingest/internal/eventcatalog/catalog/gcp_cloud_audit.yaml b/services/ingest/internal/eventcatalog/catalog/gcp_cloud_audit.yamlnew file mode 100644index 000000000..9dab8749c--- /dev/null+++ b/services/ingest/internal/eventcatalog/catalog/gcp_cloud_audit.yaml@@ -0,0 +1,27 @@+# GCP Cloud Audit Logs. Event types are the `methodName` on the proto+# payload: either a bare admin method or the fully-qualified service method.+source: gcp_cloud_audit+event_type_path: methodName+fixture_files:+ - services/connectors/tests/test_gcp_connectors.py+events:+ SetIamPolicy:+ # Rewrites who can do what on a resource, which is the single call an+ # escalation most often ends with.+ action: set.iam.policy+ sensitivity: high+ attack: [T1098.003]+ google.compute.v1.instances.start:+ action: compute.instances.start+ sensitivity: low+ google.compute.v1.instances.detachDisk:+ # Detaching a disk is how a volume is moved to an instance the+ # attacker controls, and it is otherwise rare.+ action: compute.instances.detach.disk+ sensitivity: medium+ attack: [T1530]+ google.storage.buckets.list:+ action: storage.buckets.list+ sensitivity: info+ attack: [T1580]+unclassified: {}diff --git a/services/ingest/internal/eventcatalog/catalog/github.yaml b/services/ingest/internal/eventcatalog/catalog/github.yamlnew file mode 100644index 000000000..2d7b5a010--- /dev/null+++ b/services/ingest/internal/eventcatalog/catalog/github.yaml@@ -0,0 +1,25 @@+# GitHub organisation audit log. Event types are the `action` values,+# namespaced `area.verb`.+source: github+event_type_path: action+fixture_requires: [actor]+fixture_files:+ - services/connectors/tests/test_saas_connectors.py+events:+ org.update_member:+ action: org.update.member+ sensitivity: high+ attack: [T1098]+ team.destroy:+ action: team.destroy+ sensitivity: medium+ secret_scanning.disable:+ # Turning off the control that finds leaked credentials, in the system+ # that holds the code they are leaked in.+ action: secret.scanning.disable+ sensitivity: critical+ attack: [T1562.001]+ issues.opened:+ action: issues.opened+ sensitivity: info+unclassified: {}diff --git a/services/ingest/internal/eventcatalog/catalog/google_workspace.yaml b/services/ingest/internal/eventcatalog/catalog/google_workspace.yamlnew file mode 100644index 000000000..43ddf9a1f--- /dev/null+++ b/services/ingest/internal/eventcatalog/catalog/google_workspace.yaml@@ -0,0 +1,32 @@+# Google Workspace Reports API. Event types are the `name` of each entry in+# the activity's `events` list; one activity can carry several.+source: google_workspace+event_type_path: events[].name+fixture_files:+ - services/connectors/tests/test_saas_connectors.py+events:+ GRANT_ADMIN_PRIVILEGE:+ action: grant.admin.privilege+ sensitivity: critical+ attack: [T1098]+ suspicious_login:+ # Google has already judged this one, which is why it is not merely a+ # sign-in.+ action: suspicious.login+ sensitivity: high+ attack: [T1078]+ login_failure:+ # On its own this is noise; in volume it is a spray. Sensitivity is a+ # property of the event class, and the windowed engine is what reads+ # the volume.+ action: login.failure+ sensitivity: low+ attack: [T1110]+ change_acl_editors:+ action: change.acl.editors+ sensitivity: medium+ attack: [T1222]+ view:+ action: view+ sensitivity: info+unclassified: {}diff --git a/services/ingest/internal/eventcatalog/catalog/kubernetes_audit.yaml b/services/ingest/internal/eventcatalog/catalog/kubernetes_audit.yamlnew file mode 100644index 000000000..26676881b--- /dev/null+++ b/services/ingest/internal/eventcatalog/catalog/kubernetes_audit.yaml@@ -0,0 +1,33 @@+# Kubernetes apiserver audit. The event type is the `verb`, and on its own+# it says nothing about what was acted on — `create` on a ConfigMap and+# `create` on a ClusterRoleBinding are not the same event.+#+# The catalogue classifies the verb, which is what this source's event type+# *is*, and the resource is carried separately on the event. `impersonate`+# is the exception that needs no resource to be interesting.+source: kubernetes_audit+event_type_path: verb+fixture_requires: [objectRef]+fixture_files:+ - services/connectors/tests/test_kubernetes_audit.py+events:+ get:+ action: get+ sensitivity: info+ list:+ action: list+ sensitivity: info+ attack: [T1613]+ create:+ action: create+ sensitivity: medium+ patch:+ action: patch+ sensitivity: medium+ impersonate:+ # The only verb that is a privilege decision by itself: the request is+ # asking the apiserver to act as somebody else.+ action: impersonate+ sensitivity: critical+ attack: [T1134]+unclassified: {}diff --git a/services/ingest/internal/eventcatalog/catalog/m365_audit.yaml b/services/ingest/internal/eventcatalog/catalog/m365_audit.yamlnew file mode 100644index 000000000..d6c7a0005--- /dev/null+++ b/services/ingest/internal/eventcatalog/catalog/m365_audit.yaml@@ -0,0 +1,28 @@+# Microsoft 365 Management Activity API. Event types are the `Operation`+# values, which span every workload in one stream.+source: m365_audit+event_type_path: Operation+fixture_files:+ - services/connectors/tests/test_saas_connectors.py+events:+ UserLoggedIn:+ action: user.logged.in+ sensitivity: low+ attack: [T1078]+ Add member to role:+ action: add.member.to.role+ sensitivity: high+ attack: [T1098.003]+ New-InboxRule:+ # An inbox rule created by an attacker is how a compromised mailbox+ # stays quiet: replies to the real owner are filed away unread.+ action: new.inbox.rule+ sensitivity: high+ attack: [T1114.003]+ FileAccessed:+ action: file.accessed+ sensitivity: info+ FilePreviewed:+ action: file.previewed+ sensitivity: info+unclassified: {}diff --git a/services/ingest/internal/eventcatalog/catalog/okta.yaml b/services/ingest/internal/eventcatalog/catalog/okta.yamlnew file mode 100644index 000000000..b86302650--- /dev/null+++ b/services/ingest/internal/eventcatalog/catalog/okta.yaml@@ -0,0 +1,38 @@+# Okta System Log. Event types are the dotted `eventType` strings, which+# Okta namespaces by subject — `user.session.*`, `system.api_token.*`,+# `security.*`.+source: okta+event_type_path: eventType+fixture_files:+ - services/connectors/tests/fixtures/okta/sample_event.json+events:+ user.session.start:+ # Both a success and a failure arrive under this one type; the outcome+ # is on the record, which is why the class sits at low rather than+ # being split into two entries the vendor does not have.+ action: user.session.start+ sensitivity: low+ attack: [T1078]+ user.account.lock:+ action: user.account.lock+ sensitivity: high+ attack: [T1110]+ user.account.privilege.grant:+ action: user.account.privilege.grant+ sensitivity: critical+ attack: [T1098]+ system.api_token.create:+ # A token outlives the session that minted it and carries the+ # creator's authority with no second factor behind it.+ action: system.api.token.create+ sensitivity: high+ attack: [T1098.001]+ user.mfa.factor.deactivate:+ action: user.mfa.factor.deactivate+ sensitivity: high+ attack: [T1556.006]+ security.request.blocked:+ action: security.request.blocked+ sensitivity: high+ attack: [T1078]+unclassified: {}diff --git a/services/ingest/internal/eventcatalog/catalog/slack_audit.yaml b/services/ingest/internal/eventcatalog/catalog/slack_audit.yamlnew file mode 100644index 000000000..f9b1cefb9--- /dev/null+++ b/services/ingest/internal/eventcatalog/catalog/slack_audit.yaml@@ -0,0 +1,35 @@+# Slack Audit Logs API. Event types are the `action` values, which Slack+# names as past-tense verbs over an entity.+source: slack_audit+event_type_path: action+fixture_requires: [actor]+fixture_files:+ - services/connectors/tests/fixtures/slack_audit/sample_event.json+events:+ user_login:+ action: user.login+ sensitivity: low+ attack: [T1078]+ file_downloaded:+ action: file.downloaded+ sensitivity: info+ attack: [T1213]+ app_installed:+ # A third-party app arrives holding the scopes the installer consented+ # to, which is how data leaves a workspace without anyone downloading+ # anything.+ action: app.installed+ sensitivity: high+ attack: [T1550.001]+ user_session_reset_by_admin:+ action: user.session.reset.by.admin+ sensitivity: medium+ public_channel_created:+ action: public.channel.created+ sensitivity: info+ organization_export_started:+ # Every message and file in the workspace, in one archive.+ action: organization.export.started+ sensitivity: critical+ attack: [T1213]+unclassified: {}diff --git a/services/ingest/internal/eventcatalog/catalog_test.go b/services/ingest/internal/eventcatalog/catalog_test.gonew file mode 100644index 000000000..56e33e893--- /dev/null+++ b/services/ingest/internal/eventcatalog/catalog_test.go@@ -0,0 +1,196 @@+package eventcatalog++import (+ "testing"+)++func load(t *testing.T) *Catalog {+ t.Helper()+ c, err := Load()+ if err != nil {+ t.Fatalf("Load: %v", err)+ }+ return c+}++// The catalogue is embedded, so a missing or malformed file is a build-time+// or boot-time fact rather than a runtime surprise. A load that returned a+// partial catalogue would answer "never seen" for every type in the file+// that failed, which is indistinguishable from a source nobody classified.+func TestTheEmbeddedCatalogueLoads(t *testing.T) {+ c := load(t)+ if len(c.Sources()) == 0 {+ t.Fatal("no sources loaded")+ }+ if c.Size() == 0 {+ t.Fatal("no event types loaded")+ }+ t.Logf("%d sources, %d event types", len(c.Sources()), c.Size())+}++// The ten sources depth plan 2.2's corpus names. Listed here rather than+// derived from the directory, because the point is that *these* are covered+// — a derived list would agree with whatever shipped.+func TestTheSourcesThePlanNamesAreCovered(t *testing.T) {+ c := load(t)+ present := map[string]bool{}+ for _, source := range c.Sources() {+ present[source] = true+ }+ for _, source := range []string{+ "aws_cloudtrail", "gcp_cloud_audit", "azure_activity", "azure_entra", "okta",+ "google_workspace", "m365_audit", "github", "slack_audit", "kubernetes_audit",+ } {+ if !present[source] {+ t.Errorf("no catalogue for %q", source)+ }+ }+}++// Lookup distinguishes three answers, and only one of them is worth telling+// an operator about. Folding them together would either spam a warning for+// every event from an unclassified source or hide the one case that means+// the catalogue has fallen behind the vendor.+func TestLookupDistinguishesTheThreeAnswers(t *testing.T) {+ c := load(t)++ classification, status := c.Lookup("aws_cloudtrail", "DeleteTrail")+ if status != StatusClassified {+ t.Fatalf("DeleteTrail: status = %q, want classified", status)+ }+ if classification.Sensitivity != SensitivityCritical {+ t.Errorf("DeleteTrail: sensitivity = %q, want critical", classification.Sensitivity)+ }+ if len(classification.ATTACK) == 0 {+ t.Error("DeleteTrail: no ATT&CK hint")+ }++ if _, status := c.Lookup("aws_cloudtrail", "SomethingNobodyHasSeen"); status != StatusUnknown {+ t.Errorf("an unseen type in a catalogued source: status = %q, want unknown", status)+ }+ if _, status := c.Lookup("a_source_with_no_catalogue", "anything"); status != StatusNoCatalog {+ t.Errorf("an uncatalogued source: status = %q, want no_catalog", status)+ }+}++// A nil catalogue is a usable one. The normalizer's own test constructors+// build a Normalizer directly, so the classification path has to tolerate+// having no catalogue rather than panicking in a test nobody wrote.+func TestANilCatalogueIsSafe(t *testing.T) {+ var c *Catalog+ if _, status := c.Lookup("aws_cloudtrail", "DeleteTrail"); status != StatusNoCatalog {+ t.Errorf("status = %q, want no_catalog", status)+ }+ if _, ok := c.EventTypePath("aws_cloudtrail"); ok {+ t.Error("a nil catalogue claimed to know an event-type path")+ }+ if c.Size() != 0 || c.Sources() != nil {+ t.Error("a nil catalogue reported contents")+ }+}++// Every sensitivity in the loaded catalogue is one of the five tiers, and+// Load is what enforces it — a sixth tier would be invisible until a query+// filtered on it and found nothing.+func TestEverySensitivityIsOnTheLadder(t *testing.T) {+ c := load(t)+ allowed := map[Sensitivity]bool{}+ for _, s := range Sensitivities {+ allowed[s] = true+ }+ for _, source := range c.Sources() {+ for eventType, classification := range c.bySource[source] {+ if !allowed[classification.Sensitivity] {+ t.Errorf("%s/%s: sensitivity %q is not on the ladder", source, eventType, classification.Sensitivity)+ }+ if classification.Action == "" {+ t.Errorf("%s/%s: no normalized action", source, eventType)+ }+ }+ }+}++// The declared path is how the classification finds the event type on a+// vendor record. These are the shapes the ten sources actually use.+func TestEventTypeReadsTheDeclaredPath(t *testing.T) {+ cases := []struct {+ name string+ record map[string]any+ path string+ want string+ found bool+ }{+ {+ name: "a bare key",+ record: map[string]any{"eventName": "CreateAccessKey"},+ path: "eventName",+ want: "CreateAccessKey",+ found: true,+ },+ {+ name: "a nested key, as Azure spells its operation",+ record: map[string]any{"operationName": map[string]any{"value": "Microsoft.Authorization/roleAssignments/write"}},+ path: "operationName.value",+ want: "Microsoft.Authorization/roleAssignments/write",+ found: true,+ },+ {+ name: "a list hop, as Workspace bundles several events per activity",+ record: map[string]any{+ "events": []any{map[string]any{"name": "GRANT_ADMIN_PRIVILEGE"}, map[string]any{"name": "view"}},+ },+ path: "events[].name",+ want: "GRANT_ADMIN_PRIVILEGE",+ found: true,+ },+ {+ name: "an empty list is not an event type",+ record: map[string]any{"events": []any{}},+ path: "events[].name",+ found: false,+ },+ {+ name: "a path that is not there",+ record: map[string]any{"somethingElse": "x"},+ path: "eventName",+ found: false,+ },+ {+ name: "a value that is not a string",+ record: map[string]any{"verb": 7},+ path: "verb",+ found: false,+ },+ {+ name: "whitespace is not an event type",+ record: map[string]any{"verb": " "},+ path: "verb",+ found: false,+ },+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ got, found := EventType(tc.record, tc.path)+ if found != tc.found {+ t.Fatalf("found = %v, want %v", found, tc.found)+ }+ if found && got != tc.want {+ t.Errorf("got %q, want %q", got, tc.want)+ }+ })+ }+}++// Every catalogue declares the path its own fixtures are read at, and the+// loader is what the classification uses. A path declared in the YAML that+// the loader does not expose would make the classification silently find+// nothing for that source.+func TestEverySourceDeclaresItsEventTypePath(t *testing.T) {+ c := load(t)+ for _, source := range c.Sources() {+ path, ok := c.EventTypePath(source)+ if !ok || path == "" {+ t.Errorf("%s declares no event-type path", source)+ }+ }+}diff --git a/services/ingest/internal/normalizer/classify.go b/services/ingest/internal/normalizer/classify.gonew file mode 100644index 000000000..90b088527--- /dev/null+++ b/services/ingest/internal/normalizer/classify.go@@ -0,0 +1,89 @@+package normalizer++import (+ "fmt"++ "github.com/beenuar/aisoc/services/ingest/internal/eventcatalog"+)++// Event classification, from the catalogue loaded at boot.+//+// The catalogue maps a vendor's own event type onto a normalized action, a+// sensitivity and an optional ATT&CK hint. This attaches the answer to the+// event so the lake, the detection matcher and triage read one+// classification rather than each re-deriving a meaning from the event name.+//+// It deliberately changes nothing else on the event. Sensitivity is a+// property of the event *class*; the vendor's severity on the record and the+// OCSF class still decide whether fusion promotes it. A `critical`+// sensitivity that forced an alert would let a one-line edit to a data file+// flood a queue.++// eventClassification is what lands on the event.+type eventClassification struct {+ // Source and EventType are carried so a reader of the lake row can+ // check the classification against the catalogue without the raw+ // payload, and so a hunt can group by the vendor's own vocabulary.+ Source string `json:"source"`+ EventType string `json:"event_type"`+ Action string `json:"action"`+ Sensitivity string `json:"sensitivity"`+ ATTACK []string `json:"attack,omitempty"`+}++// classify returns the classification for this event, or a warning+// explaining why there is none.+//+// Three outcomes, and the distinction is the point:+//+// - a hit, which lands on the event;+// - a type the catalogue lists as deliberately unclassified, or a source+// with no catalogue at all, both of which are silent: somebody has+// already decided, or nobody has started, and neither is news;+// - a type this source's catalogue has never seen, which is a warning on+// the event. That is the only case an operator can act on, and it is how+// the catalogue learns it has fallen behind the vendor.+func (n *Normalizer) classify(connectorType string, payload map[string]interface{}) (*eventClassification, string) {+ declaredPath, hasCatalog := n.catalog.EventTypePath(connectorType)+ if !hasCatalog {+ return nil, ""+ }++ // The vendor's record, which for a canonical envelope is nested under+ // `raw_event`. The key is `raw_event` and never `raw`: 26 connectors+ // once emitted `raw`, missed the canonical-envelope check and fell+ // through to a borrowed vendor profile.+ record := payload+ if nested, ok := payload["raw_event"].(map[string]interface{}); ok {+ record = nested+ }++ eventType, found := eventcatalog.EventType(record, declaredPath)+ if !found {+ // The source has a catalogue and this record carries nothing at+ // the declared path. Not a warning: several connectors emit shapes+ // that genuinely have no event type, and warning on each would+ // make the field unreadable.+ return nil, ""+ }++ classification, status := n.catalog.Lookup(connectorType, eventType)+ switch status {+ case eventcatalog.StatusClassified:+ return &eventClassification{+ Source: connectorType,+ EventType: eventType,+ Action: classification.Action,+ Sensitivity: string(classification.Sensitivity),+ ATTACK: classification.ATTACK,+ }, ""+ case eventcatalog.StatusUnknown:+ return nil, fmt.Sprintf(+ "event type %q is not in the %s event catalogue: it is neither classified nor recorded as "++ "deliberately unclassified, so nothing downstream knows how sensitive it is "++ "(schemas/event_catalog/%s.yaml)",+ eventType, connectorType, connectorType,+ )+ }+ return nil, ""+}diff --git a/services/ingest/internal/normalizer/classify_test.go b/services/ingest/internal/normalizer/classify_test.gonew file mode 100644index 000000000..433742c0e--- /dev/null+++ b/services/ingest/internal/normalizer/classify_test.go@@ -0,0 +1,191 @@+package normalizer++import (+ "strings"+ "testing"++ "github.com/beenuar/aisoc/services/ingest/internal/config"+ "github.com/beenuar/aisoc/services/ingest/internal/eventcatalog"+)++// A normalizer with the real catalogue loaded, which is what New() builds.+// The other constructors in this package build a Normalizer directly and so+// carry no catalogue; that is deliberate and the nil path is asserted below.+func newClassifyingNormalizer(t *testing.T) *Normalizer {+ t.Helper()+ catalog, err := eventcatalog.Load()+ if err != nil {+ t.Fatalf("loading the catalogue: %v", err)+ }+ return &Normalizer{cfg: &config.Config{NormalizerMode: "lenient"}, version: "test", catalog: catalog}+}++func normalizeWithVendorRecord(t *testing.T, connector string, vendor map[string]interface{}) *NormalizedEvent {+ t.Helper()+ ev, err := newClassifyingNormalizer(t).Normalize(&RawEvent{+ ConnectorID: "c-1",+ ConnectorType: connector,+ TenantID: "11111111-1111-1111-1111-111111111111",+ ReceivedAt: "2026-10-07T00:00:00Z",+ Payload: map[string]interface{}{+ "source": connector,+ "raw_event": vendor,+ "severity": "medium",+ "title": "an event",+ },+ })+ if err != nil {+ t.Fatalf("Normalize(%q): %v", connector, err)+ }+ return ev+}++// The classification lands on the event, carrying the vendor's own event+// type beside the normalized answer so a lake row can be checked against the+// catalogue without the raw payload.+func TestAClassifiedEventCarriesItsClassification(t *testing.T) {+ ev := normalizeWithVendorRecord(t, "aws_cloudtrail", map[string]interface{}{"eventName": "DeleteTrail"})+ classification, ok := ev.OcsfEvent["event_classification"].(*eventClassification)+ if !ok {+ t.Fatalf("no classification on the event: %T", ev.OcsfEvent["event_classification"])+ }+ if classification.Source != "aws_cloudtrail" || classification.EventType != "DeleteTrail" {+ t.Errorf("classification = %+v", classification)+ }+ if classification.Sensitivity != "critical" {+ t.Errorf("sensitivity = %q, want critical", classification.Sensitivity)+ }+ if classification.Action != "delete.trail" {+ t.Errorf("action = %q", classification.Action)+ }+ if len(classification.ATTACK) == 0 {+ t.Error("no ATT&CK hint carried through")+ }+ for _, w := range ev.NormalizationWarnings {+ if strings.Contains(w, "event catalogue") {+ t.Errorf("a classified event carried a catalogue warning: %s", w)+ }+ }+}++// The three shapes the declared paths cover, driven through the normalizer+// rather than the loader, so the `raw_event` unwrapping is exercised too.+func TestClassificationReadsEachSourcesOwnEventTypeShape(t *testing.T) {+ cases := []struct {+ connector string+ vendor map[string]interface{}+ want string+ }{+ {"okta", map[string]interface{}{"eventType": "system.api_token.create"}, "system.api_token.create"},+ {+ "azure_activity",+ map[string]interface{}{"operationName": map[string]interface{}{"value": "Microsoft.Authorization/roleAssignments/write"}},+ "Microsoft.Authorization/roleAssignments/write",+ },+ {+ "google_workspace",+ map[string]interface{}{"events": []interface{}{map[string]interface{}{"name": "GRANT_ADMIN_PRIVILEGE"}}},+ "GRANT_ADMIN_PRIVILEGE",+ },+ {"kubernetes_audit", map[string]interface{}{"verb": "impersonate"}, "impersonate"},+ {"slack_audit", map[string]interface{}{"action": "organization_export_started"}, "organization_export_started"},+ }+ for _, tc := range cases {+ t.Run(tc.connector, func(t *testing.T) {+ ev := normalizeWithVendorRecord(t, tc.connector, tc.vendor)+ classification, ok := ev.OcsfEvent["event_classification"].(*eventClassification)+ if !ok {+ t.Fatalf("no classification for %q", tc.connector)+ }+ if classification.EventType != tc.want {+ t.Errorf("event_type = %q, want %q", classification.EventType, tc.want)+ }+ })+ }+}++// An event type a catalogued source has never seen is the one case worth+// telling an operator about: it is how the catalogue learns it has fallen+// behind the vendor. The warning rides on the event and names the file to+// edit.+func TestAnUnknownEventTypeWarnsOnTheEvent(t *testing.T) {+ ev := normalizeWithVendorRecord(t, "aws_cloudtrail", map[string]interface{}{"eventName": "SomeNewApiCallAwsShippedLastWeek"})+ if _, present := ev.OcsfEvent["event_classification"]; present {+ t.Error("an unknown event type produced a classification")+ }+ var warning string+ for _, w := range ev.NormalizationWarnings {+ if strings.Contains(w, "event catalogue") {+ warning = w+ }+ }+ if warning == "" {+ t.Fatal("an unknown event type produced no warning; nothing would ever grow the catalogue")+ }+ if !strings.Contains(warning, "SomeNewApiCallAwsShippedLastWeek") {+ t.Errorf("the warning does not name the event type: %s", warning)+ }+ if !strings.Contains(warning, "schemas/event_catalog/aws_cloudtrail.yaml") {+ t.Errorf("the warning does not name the file to edit: %s", warning)+ }+}++// A source with no catalogue at all must be silent. Warning on every event+// from the seventy-four connectors with no catalogue would make the+// warnings field unreadable and the one actionable case invisible in it.+func TestASourceWithNoCatalogueIsSilent(t *testing.T) {+ ev := normalizeWithVendorRecord(t, "zscaler", map[string]interface{}{"action": "Allow"})+ if _, present := ev.OcsfEvent["event_classification"]; present {+ t.Error("a source with no catalogue produced a classification")+ }+ for _, w := range ev.NormalizationWarnings {+ if strings.Contains(w, "event catalogue") {+ t.Errorf("a source with no catalogue warned: %s", w)+ }+ }+}++// A record from a catalogued source that carries nothing at the declared+// path is also silent: several connectors emit shapes with no event type at+// all, and warning on each would drown the actionable case.+func TestARecordWithNoEventTypeIsSilent(t *testing.T) {+ ev := normalizeWithVendorRecord(t, "aws_cloudtrail", map[string]interface{}{"somethingElse": "x"})+ for _, w := range ev.NormalizationWarnings {+ if strings.Contains(w, "event catalogue") {+ t.Errorf("a record with no event type warned: %s", w)+ }+ }+}++// The classification must not change what the event is or whether it+// promotes. A `critical` sensitivity that raised a severity would let a+// one-line edit to a data file flood an alert queue, and the promotion+// contract belongs with the OCSF class and the vendor's severity.+func TestTheClassificationChangesNeitherSeverityNorClass(t *testing.T) {+ critical := normalizeWithVendorRecord(t, "aws_cloudtrail", map[string]interface{}{"eventName": "DeleteTrail"})+ informational := normalizeWithVendorRecord(t, "aws_cloudtrail", map[string]interface{}{"eventName": "DescribeInstances"})++ for _, field := range []string{"severity_id", "severity", "class_uid", "category_uid"} {+ if critical.OcsfEvent[field] != informational.OcsfEvent[field] {+ t.Errorf("%s differs between a critical-sensitivity and an info-sensitivity event (%v vs %v); "++ "the catalogue must not move promotion",+ field, critical.OcsfEvent[field], informational.OcsfEvent[field])+ }+ }+}++// New() loads the catalogue, which is what makes "ingest reads it at boot"+// true rather than a claim. A failure there is fatal, so this asserts the+// successful path reaches a normalizer that can classify.+func TestNewLoadsTheCatalogueAtBoot(t *testing.T) {+ n, err := New(&config.Config{NormalizerMode: "lenient"})+ if err != nil {+ t.Fatalf("New: %v", err)+ }+ if n.catalog == nil {+ t.Fatal("New() returned a normalizer with no catalogue")+ }+ if n.catalog.Size() == 0 {+ t.Fatal("New() loaded an empty catalogue")+ }+}diff --git a/services/ingest/internal/normalizer/normalizer.go b/services/ingest/internal/normalizer/normalizer.goindex 1abb04329..4946023fd 100644--- a/services/ingest/internal/normalizer/normalizer.go+++ b/services/ingest/internal/normalizer/normalizer.go@@ -12,6 +12,7 @@ import ("github.com/beenuar/aisoc/services/ingest/internal/attck""github.com/beenuar/aisoc/services/ingest/internal/config""github.com/beenuar/aisoc/services/ingest/internal/enrichment"+ "github.com/beenuar/aisoc/services/ingest/internal/eventcatalog""github.com/google/uuid""github.com/rs/zerolog/log")@@ -78,6 +79,7 @@ type Normalizer struct {version stringshodan *enrichment.ShodanEnrichervulnCorrel *enrichment.VulnCorrelator+ catalog *eventcatalog.CatalogipEnricher *activity.IPEnricher// VulnMatches is a channel where VULNERABILITY_MATCH events are published.@@ -409,9 +411,24 @@ func New(cfg *config.Config) (*Normalizer, error) {log.Warn().Err(err).Msg("ATT&CK corpus unavailable; technique enrichment disabled")}+ // The event catalogue, at boot. A parse failure is fatal rather than+ // logged: a half-loaded catalogue answers "never seen" for every type+ // in the file that failed, which is indistinguishable from a source+ // nobody has classified, and the warning that said so would have+ // scrolled past weeks earlier.+ catalog, err := eventcatalog.Load()+ if err != nil {+ return nil, fmt.Errorf("event catalogue: %w", err)+ }+ log.Info().+ Int("sources", len(catalog.Sources())).+ Int("event_types", catalog.Size()).+ Msg("event classification catalogue loaded")+n := &Normalizer{cfg: cfg,version: "1.1.0",+ catalog: catalog,ipEnricher: activity.NewIPEnricher(cfg.EnrichmentServiceURL,cfg.ServiceToken,@@ -827,6 +844,13 @@ func (n *Normalizer) Normalize(raw *RawEvent) (*NormalizedEvent, error) {warnings = append(warnings, w)}+ // The event classification, from the catalogue read at boot.+ if classification, warning := n.classify(connectorType, raw.Payload); classification != nil {+ ocsf["event_classification"] = classification+ } else if warning != "" {+ warnings = append(warnings, warning)+ }+// The activity projection: who acted, what kind of actor they were, what// they did, to which resource, from where, with what outcome.//