Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

An event classification catalogue, read at boot and grown from the connector fixtures (#1209)

Beenu Arora7cf5fad60f36
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 0cc054fd9..f00adb24b 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -649,6 +649,19 @@ jobs:
run: python3 scripts/check_connector_profiles.py --self-test
- name: Connector profile drift gate
run: python3 scripts/check_connector_profiles.py
+ # One layer in again: a connector's events reach a profile, and the
+ # catalogue says what each of its event types *means*. Grown from the
+ # fixtures, so adding a fixture that carries a new event type fails
+ # here until somebody decides what it is — a catalogue that only grows
+ # when someone remembers is a catalogue that stops growing. It also
+ # refuses a source whose fixtures yield nothing, because a gate over an
+ # empty corpus passes for the wrong reason.
+ - name: Event catalogue gate — self-test
+ run: |
+ python3 -m pip install --quiet pyyaml
+ python3 scripts/check_event_catalog.py --self-test
+ - name: Event catalogue gate
+ run: python3 scripts/check_event_catalog.py
# The activity projection lives in five hand-written lists: the Go, the
# lake's CREATE TABLE, the lake's migration, the fusion writer's column
# tuple and the graph schema. Three of the disagreements between them
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 64e208e0b..325125314 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -182,6 +182,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`KNOWN_ABSENT` table so that building it will fail the gate until the prose
is corrected. (#PR)
+### Added
+
+- **An event classification catalogue, read by ingest at boot.**
+ `schemas/event_catalog/<source>.yaml` maps each vendor's own event type
+ onto a normalized action, a sensitivity on the five-tier ladder, and an
+ optional ATT&CK hint. Ten sources, 44 event types: AWS CloudTrail, GCP
+ Cloud Audit, Azure Activity, Entra, Okta, Google Workspace, M365, GitHub,
+ Slack and Kubernetes audit. Every classified event carries the answer, so
+ the lake, the detection matcher and triage read one classification rather
+ than each re-deriving a meaning from an event name.
+
+ **A sensitivity deliberately does not change a severity or whether fusion
+ promotes.** Sensitivity is a property of the event *class*; the vendor's
+ own severity on the record and the OCSF class still decide promotion.
+ Letting `critical` force an alert would make a one-line edit to a data
+ file able to flood a queue, and the promotion contract belongs where it
+ already is. Pinned by a test that a critical-sensitivity and an
+ info-sensitivity event from the same source agree on `severity_id`,
+ `severity`, `class_uid` and `category_uid`.
+
+ An event type a catalogued source has never seen is the one case worth an
+ operator's attention, and it arrives as a `normalization_warnings` entry
+ on the event naming the file to edit. A source with no catalogue, and a
+ record carrying nothing at the declared path, are both silent — warning on
+ either would drown the actionable case.
+
+ The catalogue is embedded into the ingest binary from a byte-identical
+ vendored copy, because `go:embed` cannot reach above its own package and
+ reading a directory from disk at run time is a failure this service has
+ already shipped: the webhook templates were read from `/app/templates`, a
+ path the Dockerfile never populated, so every deployment answered 503
+ behind a startup warning nothing was watching.
+
+- **Recorded Okta and Slack vendor payloads.** Two of the ten sources the
+ catalogue covers had no vendor-payload fixture anywhere in the connectors
+ suite — only schema and conformance checks, which never see a record. A
+ gate grown from fixtures would have passed over both forever while
+ reporting OK. `tests/fixtures/okta/sample_event.json` (7 records, 6 event
+ types) and `tests/fixtures/slack_audit/sample_event.json` (6 records, 6
+ event types) follow each vendor's published log schema and are driven
+ through the real connectors' `normalize()`.
+
+- `scripts/check_event_catalog.py`, wired into `ci.yml` with its self-test
+ ahead of it. Every event type appearing in a catalogue's declared fixtures
+ must be classified or listed as deliberately unclassified with a reason,
+ so adding a fixture that carries a new event type fails CI until somebody
+ decides what it means. It also fails a source whose declared fixtures
+ yield **no** event type, a sensitivity outside the five tiers, a missing
+ normalized action, a malformed ATT&CK id, a thin excuse, an event type
+ both classified and excused, and any drift between `schemas/` and the
+ vendored copy the binary carries.
+
+
## [17.1.0] - 2026-10-07
### Security
diff --git a/schemas/event_catalog/README.md b/schemas/event_catalog/README.md
new file mode 100644
index 000000000..34532e8bf
--- /dev/null
+++ b/schemas/event_catalog/README.md
@@ -0,0 +1,65 @@
+# Event classification catalogue
+
+One file per source. Each maps a vendor's own event type onto a normalized
+action, a sensitivity, and an optional ATT&CK hint.
+
+```yaml
+source: aws_cloudtrail
+event_type_path: eventName # where the vendor's event type lives
+fixture_files: # the corpus the gate grows this from
+ - services/connectors/tests/test_aws_cloudtrail.py
+events:
+ CreateAccessKey:
+ action: create.access.key
+ sensitivity: high
+ attack: [T1098.001]
+unclassified:
+ SomeEventType: why it is deliberately not classified
+```
+
+## Why a data file rather than code
+
+The question "how sensitive is this event type" has a different answer per
+vendor, per event, and sometimes per customer, and it changes whenever a
+vendor adds an event. A regex over event names would be wrong in both
+directions and invisible when it was. A reviewed table is wrong in exactly
+the places somebody can point at.
+
+It also has to be the *same* table everywhere. Ingest reads it at boot and
+stamps the classification onto every event, so the lake, the detection
+matcher and triage see one answer rather than three re-derivations.
+
+## Sensitivity
+
+The five-tier ladder the rest of the platform uses — `info`, `low`,
+`medium`, `high`, `critical` — and it means **how security-relevant this
+event type is**, not how severe this particular occurrence was. A vendor's
+own severity on the record still governs promotion; sensitivity is a
+property of the *class* of event and is what makes "show me everything above
+medium that a non-human actor did" answerable without enumerating verbs.
+
+`critical` is reserved for event types that are an incident on their own
+reading — disabling the audit trail, destroying it, granting the top-level
+administrative role. Everything an estate does hourly is `info`.
+
+## How this grows
+
+`scripts/check_event_catalog.py` extracts every event type that appears in
+the fixtures each catalogue declares, and fails when one is neither
+classified nor listed under `unclassified` with a reason. Adding a connector
+fixture that carries a new event type therefore fails CI until somebody
+decides what it means, which is the point: a catalogue that only grows when
+someone remembers is a catalogue that stops growing.
+
+A source whose declared fixtures yield **no** event types also fails. A gate
+over an empty corpus is a gate that passes for the wrong reason, and two of
+the ten sources here had no vendor-payload fixture at all until this
+catalogue was written.
+
+## What is deliberately not here
+
+A sensitivity does not change an event's severity or whether fusion promotes
+it. Making `critical` sensitivity force an alert would mean this file could
+silently flood a queue, and the promotion contract belongs with the OCSF
+class and the vendor's severity where it already is. The classification is
+attached to the event for detections, hunts and triage to read.
diff --git a/schemas/event_catalog/aws_cloudtrail.yaml b/schemas/event_catalog/aws_cloudtrail.yaml
new file mode 100644
index 000000000..311a7b9e7
--- /dev/null
+++ b/schemas/event_catalog/aws_cloudtrail.yaml
@@ -0,0 +1,36 @@
+# AWS CloudTrail. Event types are the API call names in `eventName`.
+#
+# Sensitivity reflects what the call class means, not what any one
+# occurrence scored: DescribeInstances is reconnaissance-shaped and happens
+# thousands of times an hour in a healthy account, DeleteTrail is an
+# incident on its own reading.
+source: aws_cloudtrail
+event_type_path: eventName
+fixture_files:
+ - services/connectors/tests/test_aws_cloudtrail.py
+events:
+ ConsoleLogin:
+ action: console.login
+ sensitivity: medium
+ attack: [T1078.004]
+ CreateAccessKey:
+ action: create.access.key
+ sensitivity: high
+ attack: [T1098.001]
+ DeleteTrail:
+ # Destroying the audit trail is the one event where the absence of
+ # everything after it is the finding.
+ action: delete.trail
+ sensitivity: critical
+ attack: [T1562.008]
+ PutEventSelectors:
+ # Narrowing what the trail records is the quieter half of the same
+ # technique, which is why it sits one tier below rather than at info.
+ action: put.event.selectors
+ sensitivity: high
+ attack: [T1562.008]
+ DescribeInstances:
+ action: describe.instances
+ sensitivity: info
+ attack: [T1580]
+unclassified: {}
diff --git a/schemas/event_catalog/azure_activity.yaml b/schemas/event_catalog/azure_activity.yaml
new file mode 100644
index 000000000..cae5b0a42
--- /dev/null
+++ b/schemas/event_catalog/azure_activity.yaml
@@ -0,0 +1,16 @@
+# Azure Activity Log. Event types are the ARM operation names in
+# `operationName.value`, which are `Provider/resourceType/action`.
+source: azure_activity
+event_type_path: operationName.value
+fixture_files:
+ - services/connectors/tests/test_azure_connectors.py
+events:
+ Microsoft.Authorization/roleAssignments/write:
+ action: authorization.role.assignment.write
+ sensitivity: high
+ attack: [T1098.003]
+ Microsoft.Storage/storageAccounts/read:
+ action: storage.account.read
+ sensitivity: info
+ attack: [T1580]
+unclassified: {}
diff --git a/schemas/event_catalog/azure_entra.yaml b/schemas/event_catalog/azure_entra.yaml
new file mode 100644
index 000000000..e2274665c
--- /dev/null
+++ b/schemas/event_catalog/azure_entra.yaml
@@ -0,0 +1,18 @@
+# Microsoft Entra ID audit logs. Event types are the
+# `activityDisplayName` strings, which are prose rather than identifiers —
+# Microsoft's own vocabulary, kept verbatim so a lookup against a live
+# tenant's logs matches.
+source: azure_entra
+event_type_path: activityDisplayName
+fixture_files:
+ - services/connectors/tests/test_azure_connectors.py
+events:
+ Add member to role:
+ action: add.member.to.role
+ sensitivity: high
+ attack: [T1098.003]
+ Add user:
+ action: add.user
+ sensitivity: medium
+ attack: [T1136.003]
+unclassified: {}
diff --git a/schemas/event_catalog/gcp_cloud_audit.yaml b/schemas/event_catalog/gcp_cloud_audit.yaml
new file mode 100644
index 000000000..9dab8749c
--- /dev/null
+++ b/schemas/event_catalog/gcp_cloud_audit.yaml
@@ -0,0 +1,27 @@
+# GCP Cloud Audit Logs. Event types are the `methodName` on the proto
+# payload: either a bare admin method or the fully-qualified service method.
+source: gcp_cloud_audit
+event_type_path: methodName
+fixture_files:
+ - services/connectors/tests/test_gcp_connectors.py
+events:
+ SetIamPolicy:
+ # Rewrites who can do what on a resource, which is the single call an
+ # escalation most often ends with.
+ action: set.iam.policy
+ sensitivity: high
+ attack: [T1098.003]
+ google.compute.v1.instances.start:
+ action: compute.instances.start
+ sensitivity: low
+ google.compute.v1.instances.detachDisk:
+ # Detaching a disk is how a volume is moved to an instance the
+ # attacker controls, and it is otherwise rare.
+ action: compute.instances.detach.disk
+ sensitivity: medium
+ attack: [T1530]
+ google.storage.buckets.list:
+ action: storage.buckets.list
+ sensitivity: info
+ attack: [T1580]
+unclassified: {}
diff --git a/schemas/event_catalog/github.yaml b/schemas/event_catalog/github.yaml
new file mode 100644
index 000000000..2d7b5a010
--- /dev/null
+++ b/schemas/event_catalog/github.yaml
@@ -0,0 +1,25 @@
+# GitHub organisation audit log. Event types are the `action` values,
+# namespaced `area.verb`.
+source: github
+event_type_path: action
+fixture_requires: [actor]
+fixture_files:
+ - services/connectors/tests/test_saas_connectors.py
+events:
+ org.update_member:
+ action: org.update.member
+ sensitivity: high
+ attack: [T1098]
+ team.destroy:
+ action: team.destroy
+ sensitivity: medium
+ secret_scanning.disable:
+ # Turning off the control that finds leaked credentials, in the system
+ # that holds the code they are leaked in.
+ action: secret.scanning.disable
+ sensitivity: critical
+ attack: [T1562.001]
+ issues.opened:
+ action: issues.opened
+ sensitivity: info
+unclassified: {}
diff --git a/schemas/event_catalog/google_workspace.yaml b/schemas/event_catalog/google_workspace.yaml
new file mode 100644
index 000000000..43ddf9a1f
--- /dev/null
+++ b/schemas/event_catalog/google_workspace.yaml
@@ -0,0 +1,32 @@
+# Google Workspace Reports API. Event types are the `name` of each entry in
+# the activity's `events` list; one activity can carry several.
+source: google_workspace
+event_type_path: events[].name
+fixture_files:
+ - services/connectors/tests/test_saas_connectors.py
+events:
+ GRANT_ADMIN_PRIVILEGE:
+ action: grant.admin.privilege
+ sensitivity: critical
+ attack: [T1098]
+ suspicious_login:
+ # Google has already judged this one, which is why it is not merely a
+ # sign-in.
+ action: suspicious.login
+ sensitivity: high
+ attack: [T1078]
+ login_failure:
+ # On its own this is noise; in volume it is a spray. Sensitivity is a
+ # property of the event class, and the windowed engine is what reads
+ # the volume.
+ action: login.failure
+ sensitivity: low
+ attack: [T1110]
+ change_acl_editors:
+ action: change.acl.editors
+ sensitivity: medium
+ attack: [T1222]
+ view:
+ action: view
+ sensitivity: info
+unclassified: {}
diff --git a/schemas/event_catalog/kubernetes_audit.yaml b/schemas/event_catalog/kubernetes_audit.yaml
new file mode 100644
index 000000000..26676881b
--- /dev/null
+++ b/schemas/event_catalog/kubernetes_audit.yaml
@@ -0,0 +1,33 @@
+# Kubernetes apiserver audit. The event type is the `verb`, and on its own
+# it says nothing about what was acted on — `create` on a ConfigMap and
+# `create` on a ClusterRoleBinding are not the same event.
+#
+# The catalogue classifies the verb, which is what this source's event type
+# *is*, and the resource is carried separately on the event. `impersonate`
+# is the exception that needs no resource to be interesting.
+source: kubernetes_audit
+event_type_path: verb
+fixture_requires: [objectRef]
+fixture_files:
+ - services/connectors/tests/test_kubernetes_audit.py
+events:
+ get:
+ action: get
+ sensitivity: info
+ list:
+ action: list
+ sensitivity: info
+ attack: [T1613]
+ create:
+ action: create
+ sensitivity: medium
+ patch:
+ action: patch
+ sensitivity: medium
+ impersonate:
+ # The only verb that is a privilege decision by itself: the request is
+ # asking the apiserver to act as somebody else.
+ action: impersonate
+ sensitivity: critical
+ attack: [T1134]
+unclassified: {}
diff --git a/schemas/event_catalog/m365_audit.yaml b/schemas/event_catalog/m365_audit.yaml
new file mode 100644
index 000000000..d6c7a0005
--- /dev/null
+++ b/schemas/event_catalog/m365_audit.yaml
@@ -0,0 +1,28 @@
+# Microsoft 365 Management Activity API. Event types are the `Operation`
+# values, which span every workload in one stream.
+source: m365_audit
+event_type_path: Operation
+fixture_files:
+ - services/connectors/tests/test_saas_connectors.py
+events:
+ UserLoggedIn:
+ action: user.logged.in
+ sensitivity: low
+ attack: [T1078]
+ Add member to role:
+ action: add.member.to.role
+ sensitivity: high
+ attack: [T1098.003]
+ New-InboxRule:
+ # An inbox rule created by an attacker is how a compromised mailbox
+ # stays quiet: replies to the real owner are filed away unread.
+ action: new.inbox.rule
+ sensitivity: high
+ attack: [T1114.003]
+ FileAccessed:
+ action: file.accessed
+ sensitivity: info
+ FilePreviewed:
+ action: file.previewed
+ sensitivity: info
+unclassified: {}
diff --git a/schemas/event_catalog/okta.yaml b/schemas/event_catalog/okta.yaml
new file mode 100644
index 000000000..b86302650
--- /dev/null
+++ b/schemas/event_catalog/okta.yaml
@@ -0,0 +1,38 @@
+# Okta System Log. Event types are the dotted `eventType` strings, which
+# Okta namespaces by subject — `user.session.*`, `system.api_token.*`,
+# `security.*`.
+source: okta
+event_type_path: eventType
+fixture_files:
+ - services/connectors/tests/fixtures/okta/sample_event.json
+events:
+ user.session.start:
+ # Both a success and a failure arrive under this one type; the outcome
+ # is on the record, which is why the class sits at low rather than
+ # being split into two entries the vendor does not have.
+ action: user.session.start
+ sensitivity: low
+ attack: [T1078]
+ user.account.lock:
+ action: user.account.lock
+ sensitivity: high
+ attack: [T1110]
+ user.account.privilege.grant:
+ action: user.account.privilege.grant
+ sensitivity: critical
+ attack: [T1098]
+ system.api_token.create:
+ # A token outlives the session that minted it and carries the
+ # creator's authority with no second factor behind it.
+ action: system.api.token.create
+ sensitivity: high
+ attack: [T1098.001]
+ user.mfa.factor.deactivate:
+ action: user.mfa.factor.deactivate
+ sensitivity: high
+ attack: [T1556.006]
+ security.request.blocked:
+ action: security.request.blocked
+ sensitivity: high
+ attack: [T1078]
+unclassified: {}
diff --git a/schemas/event_catalog/slack_audit.yaml b/schemas/event_catalog/slack_audit.yaml
new file mode 100644
index 000000000..f9b1cefb9
--- /dev/null
+++ b/schemas/event_catalog/slack_audit.yaml
@@ -0,0 +1,35 @@
+# Slack Audit Logs API. Event types are the `action` values, which Slack
+# names as past-tense verbs over an entity.
+source: slack_audit
+event_type_path: action
+fixture_requires: [actor]
+fixture_files:
+ - services/connectors/tests/fixtures/slack_audit/sample_event.json
+events:
+ user_login:
+ action: user.login
+ sensitivity: low
+ attack: [T1078]
+ file_downloaded:
+ action: file.downloaded
+ sensitivity: info
+ attack: [T1213]
+ app_installed:
+ # A third-party app arrives holding the scopes the installer consented
+ # to, which is how data leaves a workspace without anyone downloading
+ # anything.
+ action: app.installed
+ sensitivity: high
+ attack: [T1550.001]
+ user_session_reset_by_admin:
+ action: user.session.reset.by.admin
+ sensitivity: medium
+ public_channel_created:
+ action: public.channel.created
+ sensitivity: info
+ organization_export_started:
+ # Every message and file in the workspace, in one archive.
+ action: organization.export.started
+ sensitivity: critical
+ attack: [T1213]
+unclassified: {}
diff --git a/scripts/check_event_catalog.py b/scripts/check_event_catalog.py
new file mode 100644
index 000000000..a312a8b79
--- /dev/null
+++ b/scripts/check_event_catalog.py
@@ -0,0 +1,564 @@
+#!/usr/bin/env python3
+"""Every event type in a connector's fixtures is classified, or excused.
+
+Why this exists
+---------------
+`schemas/event_catalog/<source>.yaml` says what a vendor's event types mean:
+a normalized action, a sensitivity on the platform's five-tier ladder, and an
+optional ATT&CK hint. Ingest reads it at boot and stamps the answer onto every
+event, so the lake, the detection matcher and triage share one classification
+rather than each re-deriving a meaning from an event name.
+
+A catalogue that only grows when somebody remembers is a catalogue that stops
+growing. So this gate extracts every event type that appears in the fixtures
+each catalogue declares, and fails when one is neither classified nor listed
+under ``unclassified`` with a reason. Adding a connector fixture carrying a
+new event type therefore fails CI until somebody decides what it means.
+
+Three things it refuses that a simpler version of itself would not
+------------------------------------------------------------------
+**A source whose fixtures yield no event types.** That is a gate passing for
+the wrong reason — the shape this repository keeps finding, where the clean
+result and the broken glob print the same word. Two of the ten sources here
+had no vendor-payload fixture at all before this catalogue was written, and
+a gate that shrugged at that would have certified them forever.
+
+**A vendored copy that has drifted.** `go:embed` cannot reach above its own
+package, so the binary carries a copy under
+``services/ingest/internal/eventcatalog/catalog/``. Reading the directory
+from disk at run time instead is a failure this service has already shipped:
+the webhook templates were read from a path the Dockerfile never populated.
+The two directories are held byte-identical in both directions, so neither
+can gain, lose or alter a file alone.
+
+**A classification outside the ladder.** Severity is five tiers everywhere in
+this platform. A sixth would be invisible until a query filtered on it and
+found nothing.
+
+Usage
+-----
+ python3 scripts/check_event_catalog.py
+ python3 scripts/check_event_catalog.py --json
+ python3 scripts/check_event_catalog.py --self-test
+
+``--repo-root`` overrides the tree. Every input is named before the verdict
+and an empty read is a hard error, never a quiet pass.
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import json
+import re
+import sys
+from pathlib import Path
+from typing import Any
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+import yaml
+from gate_toolkit import repo_root
+
+CATALOG_REL = Path("schemas/event_catalog")
+VENDORED_REL = Path("services/ingest/internal/eventcatalog/catalog")
+LOADER_REL = Path("services/ingest/internal/eventcatalog/catalog.go")
+CONNECTORS_REL = Path("services/connectors/app/connectors")
+
+#: The five tiers, lowest first. Identical to `eventcatalog.Sensitivities` in
+#: the Go, which this gate reads back out of the loader so the two cannot
+#: diverge without failing here.
+SENSITIVITIES = ("info", "low", "medium", "high", "critical")
+
+#: An ATT&CK technique or sub-technique id. Nothing resolves these against
+#: the corpus here — `scripts/` already owns that check for detection content
+#: — but a malformed id is cheap to catch and travels onto every event.
+ATTACK_ID = re.compile(r"^T\d{4}(\.\d{3})?$")
+
+
+class GateError(RuntimeError):
+ """An input could not be read. Never downgraded to a passing result."""
+
+
+# --------------------------------------------------------------------------
+# Catalogue
+# --------------------------------------------------------------------------
+def load_catalogs(directory: Path) -> dict[str, dict]:
+ """source -> parsed catalogue, keyed by the `source:` field, not the name."""
+ out: dict[str, dict] = {}
+ for path in sorted(directory.glob("*.yaml")):
+ parsed = yaml.safe_load(path.read_text(encoding="utf-8"))
+ if not isinstance(parsed, dict):
+ raise GateError(f"{path} does not parse to a mapping")
+ source = parsed.get("source")
+ if not source:
+ raise GateError(f"{path} declares no `source`")
+ if source in out:
+ raise GateError(f"two catalogue files declare source {source!r}")
+ parsed["__file__"] = path.name
+ out[source] = parsed
+ return out
+
+
+# --------------------------------------------------------------------------
+# Fixture corpus
+# --------------------------------------------------------------------------
+def _records_from_json(path: Path):
+ data = json.loads(path.read_text(encoding="utf-8"))
+ stack = [data]
+ while stack:
+ item = stack.pop()
+ if isinstance(item, dict):
+ yield item
+ stack.extend(item.values())
+ elif isinstance(item, list):
+ stack.extend(item)
+
+
+def _records_from_python(path: Path):
+ """Every dict literal in a Python test file, as a plain dict.
+
+ Only constant keys and values survive; a nested dict or list becomes a
+ real nested structure so a declared path like ``operationName.value`` or
+ ``events[].name`` resolves the same way it does at run time. Anything
+ built by a call or a comprehension is skipped rather than guessed at.
+ """
+
+ def literal(node):
+ if isinstance(node, ast.Constant):
+ return node.value
+ if isinstance(node, ast.Dict):
+ out = {}
+ for key, value in zip(node.keys, node.values, strict=False):
+ if isinstance(key, ast.Constant) and isinstance(key.value, str):
+ out[key.value] = literal(value)
+ return out
+ if isinstance(node, (ast.List, ast.Tuple)):
+ return [literal(item) for item in node.elts]
+ return None
+
+ tree = ast.parse(path.read_text(encoding="utf-8"))
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Dict):
+ value = literal(node)
+ if value:
+ yield value
+
+
+def extract_at_path(record: dict, declared_path: str) -> str | None:
+ """Read the event type out of one record, using the declared path.
+
+ One list hop (``events[].name``) is supported, which is the only shape
+ any source here needs. Mirrors ``eventcatalog.EventType`` in the Go; the
+ two are small enough that a shared implementation would cost more than
+ it saves, and the fixtures exercise both.
+ """
+ # Annotated rather than inferred: it starts as the record but walks into
+ # whatever each segment finds, so `dict` -- what mypy takes from the first
+ # assignment -- is wrong from the second one onward.
+ cur: Any = record
+ for segment in declared_path.split("."):
+ list_hop = segment.endswith("[]")
+ key = segment[:-2] if list_hop else segment
+ if not isinstance(cur, dict):
+ return None
+ cur = cur.get(key)
+ if list_hop:
+ if not isinstance(cur, list) or not cur:
+ return None
+ cur = cur[0]
+ if isinstance(cur, str) and cur.strip():
+ return cur.strip()
+ return None
+
+
+def event_types_in_fixtures(root: Path, catalog: dict) -> tuple[set[str], list[str]]:
+ """(event types found, files that could not be read)."""
+ declared_path = catalog["event_type_path"]
+ requires = catalog.get("fixture_requires") or []
+ found: set[str] = set()
+ missing: list[str] = []
+
+ for relative in catalog.get("fixture_files") or []:
+ path = root / relative
+ if not path.exists():
+ missing.append(relative)
+ continue
+ records = _records_from_json(path) if path.suffix == ".json" else _records_from_python(path)
+ for record in records:
+ if not isinstance(record, dict):
+ continue
+ # A discriminator, where the source declares one. `action`
+ # appears on records from several sources and in unrelated test
+ # dictionaries, so without this the GitHub catalogue would be
+ # asked to classify a VPC flow log's `ACCEPT`.
+ if requires and not all(key in record for key in requires):
+ continue
+ value = extract_at_path(record, declared_path)
+ if value:
+ found.add(value)
+ return found, missing
+
+
+# --------------------------------------------------------------------------
+# Go side
+# --------------------------------------------------------------------------
+def parse_go_sensitivities(src: str) -> list[str]:
+ match = re.search(r"var Sensitivities = \[\]Sensitivity\{(.*?)\}", src, re.DOTALL)
+ if not match:
+ raise GateError("could not find the Sensitivities slice in the loader")
+ by_const = dict(re.findall(r'^\tSensitivity(\w+)\s+Sensitivity = "(\w+)"$', src, re.MULTILINE))
+ return [by_const[name] for name in re.findall(r"Sensitivity(\w+)", match.group(1)) if name in by_const]
+
+
+def parse_declared_connectors(directory: Path) -> set[str]:
+ out: set[str] = set()
+ for path in sorted(directory.glob("*.py")):
+ if path.name in {"__init__.py", "base.py"}:
+ continue
+ for match in re.finditer(r'(?m)^\s*connector_id(?:\s*:\s*\w+)?\s*=\s*"([^"]+)"', path.read_text(encoding="utf-8")):
+ out.add(match.group(1))
+ return out
+
+
+# --------------------------------------------------------------------------
+# Gate
+# --------------------------------------------------------------------------
+def evaluate(
+ catalogs: dict[str, dict],
+ fixture_types: dict[str, set[str]],
+ missing_fixtures: dict[str, list[str]],
+ vendored_bytes: dict[str, bytes],
+ source_bytes: dict[str, bytes],
+ go_sensitivities: list[str],
+ declared_connectors: set[str],
+) -> tuple[list[tuple[str, str]], dict]:
+ failures: list[tuple[str, str]] = []
+
+ # LADDER. The Go's closed set and this gate's must be the same five.
+ if list(go_sensitivities) != list(SENSITIVITIES):
+ failures.append(
+ (
+ "ladder-mismatch",
+ f"the loader declares the sensitivity ladder as {go_sensitivities}, and this gate checks "
+ f"against {list(SENSITIVITIES)}; a value the gate accepts and the loader rejects fails at boot",
+ )
+ )
+
+ # VENDORED COPY. Both directions: neither directory may gain, lose or
+ # alter a file alone.
+ for name in sorted(set(source_bytes) | set(vendored_bytes)):
+ if name not in vendored_bytes:
+ failures.append(("vendored-missing", f"{name} is in {CATALOG_REL} and not in {VENDORED_REL}; the binary would not carry it"))
+ elif name not in source_bytes:
+ failures.append(
+ ("vendored-orphan", f"{name} is in {VENDORED_REL} and not in {CATALOG_REL}; the binary carries a file nobody edits")
+ )
+ elif source_bytes[name] != vendored_bytes[name]:
+ failures.append(
+ (
+ "vendored-drift",
+ f"{name} differs between {CATALOG_REL} and {VENDORED_REL}; the file a reviewer edits is not the one the binary loads",
+ )
+ )
+
+ for source, catalog in sorted(catalogs.items()):
+ file_name = catalog["__file__"]
+
+ # The source must be a connector type that exists.
+ if source not in declared_connectors:
+ failures.append(("source-names-nothing", f"{file_name} declares source {source!r}, which no connector declares"))
+
+ # Declared fixtures must exist, or the corpus is a promise.
+ for relative in missing_fixtures.get(source, []):
+ failures.append(("fixture-missing", f"{file_name} declares the fixture {relative}, which does not exist"))
+
+ events = catalog.get("events") or {}
+ unclassified = catalog.get("unclassified") or {}
+
+ for event_type, classification in sorted(events.items()):
+ if not isinstance(classification, dict):
+ failures.append(("entry-malformed", f"{file_name}: {event_type!r} is not a mapping"))
+ continue
+ if not classification.get("action"):
+ failures.append(("action-missing", f"{file_name}: {event_type!r} declares no normalized action"))
+ sensitivity = classification.get("sensitivity")
+ if sensitivity not in SENSITIVITIES:
+ failures.append(
+ (
+ "sensitivity-invalid",
+ f"{file_name}: {event_type!r} declares sensitivity {sensitivity!r}, which is not one of {', '.join(SENSITIVITIES)}",
+ )
+ )
+ for technique in classification.get("attack") or []:
+ if not ATTACK_ID.match(str(technique)):
+ failures.append(("attack-malformed", f"{file_name}: {event_type!r} hints at {technique!r}, which is not an ATT&CK id"))
+
+ for event_type, reason in sorted(unclassified.items()):
+ if event_type in events:
+ failures.append(
+ (
+ "entry-both",
+ f"{file_name}: {event_type!r} is both classified and listed as deliberately unclassified",
+ )
+ )
+ if not isinstance(reason, str) or len(reason) < 15:
+ failures.append(
+ (
+ "excuse-thin",
+ f"{file_name}: {event_type!r} is excused with {reason!r}, which does not say enough for a "
+ "reviewer to disagree with it",
+ )
+ )
+
+ # The corpus itself. An empty one is a pass for the wrong reason.
+ found = fixture_types.get(source, set())
+ if not found:
+ failures.append(
+ (
+ "fixture-corpus-empty",
+ f"{file_name} declares {len(catalog.get('fixture_files') or [])} fixture file(s) and no event "
+ f"type was found at {catalog['event_type_path']!r} in any of them. This gate would pass over "
+ "this source forever without checking anything",
+ )
+ )
+ continue
+
+ for event_type in sorted(found - set(events) - set(unclassified)):
+ failures.append(
+ (
+ "event-type-unclassified",
+ f"{source}: the event type {event_type!r} appears in the fixtures and is neither classified "
+ f"nor listed as deliberately unclassified in {file_name}",
+ )
+ )
+
+ stats = {
+ "sources": sorted(catalogs),
+ "classified": sum(len(c.get("events") or {}) for c in catalogs.values()),
+ "excused": sum(len(c.get("unclassified") or {}) for c in catalogs.values()),
+ "fixture_types": {source: sorted(types) for source, types in sorted(fixture_types.items())},
+ "by_sensitivity": {
+ tier: sum(
+ 1
+ for c in catalogs.values()
+ for e in (c.get("events") or {}).values()
+ if isinstance(e, dict) and e.get("sensitivity") == tier
+ )
+ for tier in SENSITIVITIES
+ },
+ }
+ return failures, stats
+
+
+def load(root: Path) -> dict:
+ catalog_dir = root / CATALOG_REL
+ vendored_dir = root / VENDORED_REL
+ loader = root / LOADER_REL
+ connectors = root / CONNECTORS_REL
+ for path in (catalog_dir, vendored_dir, loader, connectors):
+ if not path.exists():
+ raise GateError(f"expected input does not exist: {path}")
+
+ catalogs = load_catalogs(catalog_dir)
+ fixture_types: dict[str, set[str]] = {}
+ missing_fixtures: dict[str, list[str]] = {}
+ for source, catalog in catalogs.items():
+ found, missing = event_types_in_fixtures(root, catalog)
+ fixture_types[source] = found
+ missing_fixtures[source] = missing
+
+ data = {
+ "catalogs": catalogs,
+ "fixture_types": fixture_types,
+ "missing_fixtures": missing_fixtures,
+ "source_bytes": {p.name: p.read_bytes() for p in sorted(catalog_dir.glob("*.yaml"))},
+ "vendored_bytes": {p.name: p.read_bytes() for p in sorted(vendored_dir.glob("*.yaml"))},
+ "go_sensitivities": parse_go_sensitivities(loader.read_text(encoding="utf-8")),
+ "declared_connectors": parse_declared_connectors(connectors),
+ }
+ for name in ("catalogs", "source_bytes", "vendored_bytes", "go_sensitivities", "declared_connectors"):
+ if not data[name]:
+ raise GateError(f"parsed zero {name} — refusing to report a clean tree from an empty read")
+ return data
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ parser.add_argument("--repo-root", type=Path, default=repo_root())
+ parser.add_argument("--json", action="store_true")
+ parser.add_argument("--self-test", action="store_true", help="prove the gate detects what it claims to")
+ args = parser.parse_args(argv)
+
+ if args.self_test:
+ return self_test(args.repo_root.resolve())
+
+ root = args.repo_root.resolve()
+ try:
+ data = load(root)
+ except GateError as exc:
+ print(f"check_event_catalog: FAILED to read the tree: {exc}", file=sys.stderr)
+ return 2
+
+ failures, stats = evaluate(**data)
+
+ if args.json:
+ report = {"repo_root": str(root), **stats, "failures": [{"code": c, "detail": d} for c, d in failures]}
+ print(json.dumps(report, indent=2, sort_keys=True))
+ return 1 if failures else 0
+
+ print(f"repo root {root}")
+ print(f"catalogue {CATALOG_REL} ({len(stats['sources'])} sources, {stats['classified']} event types)")
+ print(f"vendored copy {VENDORED_REL} ({len(data['vendored_bytes'])} files, byte-compared both ways)")
+ print(f"loader {LOADER_REL} (ladder: {', '.join(data['go_sensitivities'])})")
+ print()
+ for source in stats["sources"]:
+ found = stats["fixture_types"][source]
+ catalog = data["catalogs"][source]
+ print(f" {source:18s} {len(catalog.get('events') or {}):>3} classified {len(found):>3} in fixtures")
+ print()
+ tiers = " ".join(f"{tier}={count}" for tier, count in stats["by_sensitivity"].items())
+ print(f"by sensitivity {tiers}")
+ print()
+ if failures:
+ print(f"FAIL — {len(failures)} finding(s):")
+ for code, detail in failures:
+ print(f" [{code}] {detail}")
+ return 1
+ print("OK — every event type in every declared fixture is classified or excused,")
+ print(" every sensitivity is on the ladder, and the vendored copy matches.")
+ return 0
+
+
+def self_test(root: Path) -> int:
+ """Inject each defect the gate claims to catch and require it to bite."""
+ try:
+ base = load(root)
+ except GateError as exc:
+ print(f"self-test: cannot read the tree: {exc}", file=sys.stderr)
+ return 2
+
+ clean, _ = evaluate(**base)
+ if clean:
+ print("self-test: the unmodified tree already fails; fix that first", file=sys.stderr)
+ for code, detail in clean:
+ print(f" [{code}] {detail}", file=sys.stderr)
+ return 1
+
+ def mutate(**overrides) -> dict:
+ import copy
+
+ data = copy.deepcopy(base)
+ for key, fn in overrides.items():
+ fn(data[key])
+ return data
+
+ any_source = sorted(base["catalogs"])[0]
+ any_file = sorted(base["source_bytes"])[0]
+
+ cases = [
+ (
+ "CORPUS: a fixture event type that is neither classified nor excused",
+ "event-type-unclassified",
+ mutate(fixture_types=lambda d: d[any_source].add("SomethingNobodyClassified")),
+ ),
+ (
+ "CORPUS: a source whose fixtures yield nothing, so the gate would check nothing",
+ "fixture-corpus-empty",
+ mutate(fixture_types=lambda d: d.__setitem__(any_source, set())),
+ ),
+ (
+ "CORPUS: a declared fixture file that does not exist",
+ "fixture-missing",
+ mutate(missing_fixtures=lambda d: d.__setitem__(any_source, ["services/connectors/tests/not_there.py"])),
+ ),
+ (
+ "LADDER: a sensitivity outside the five tiers",
+ "sensitivity-invalid",
+ mutate(catalogs=lambda d: d[any_source]["events"].__setitem__("X", {"action": "x", "sensitivity": "severe"})),
+ ),
+ (
+ "LADDER: the loader and the gate disagreeing about the ladder",
+ "ladder-mismatch",
+ mutate(go_sensitivities=lambda lst: lst.append("catastrophic")),
+ ),
+ (
+ "ENTRY: a classification with no normalized action",
+ "action-missing",
+ mutate(catalogs=lambda d: d[any_source]["events"].__setitem__("Y", {"sensitivity": "low"})),
+ ),
+ (
+ "ENTRY: a malformed ATT&CK hint",
+ "attack-malformed",
+ mutate(
+ catalogs=lambda d: d[any_source]["events"].__setitem__("Z", {"action": "z", "sensitivity": "low", "attack": ["TA0001"]})
+ ),
+ ),
+ (
+ "ENTRY: an excuse too thin to review",
+ "excuse-thin",
+ mutate(catalogs=lambda d: d[any_source].__setitem__("unclassified", {"Q": "noisy"})),
+ ),
+ (
+ "ENTRY: an event type both classified and excused",
+ "entry-both",
+ mutate(
+ catalogs=lambda d: d[any_source].__setitem__(
+ "unclassified", {next(iter(d[any_source]["events"])): "a reason long enough to pass the length check"}
+ )
+ ),
+ ),
+ (
+ "SOURCE: a catalogue for a connector that does not exist",
+ "source-names-nothing",
+ mutate(catalogs=lambda d: d.__setitem__("acme_xdr_9000", {**d[any_source], "__file__": "acme.yaml"})),
+ ),
+ (
+ "VENDORED: the embedded copy differs from the file a reviewer edits",
+ "vendored-drift",
+ mutate(vendored_bytes=lambda d: d.__setitem__(any_file, b"# drifted\n")),
+ ),
+ (
+ "VENDORED: a catalogue file the binary does not carry",
+ "vendored-missing",
+ mutate(source_bytes=lambda d: d.__setitem__("new_source.yaml", b"source: x\n")),
+ ),
+ (
+ "VENDORED: an embedded file nobody edits",
+ "vendored-orphan",
+ mutate(vendored_bytes=lambda d: d.__setitem__("orphan.yaml", b"source: x\n")),
+ ),
+ ]
+
+ print(f"self-test against {root}")
+ print("clean tree: 0 failures (the baseline every case below perturbs)\n")
+ ok = True
+ for description, expected_code, data in cases:
+ found, _ = evaluate(**data)
+ codes = {code for code, _ in found}
+ caught = expected_code in codes
+ ok &= caught
+ print(f" {'PASS' if caught else 'FAIL'} {description}")
+ print(f" expected [{expected_code}] got {sorted(codes) or 'nothing'}")
+
+ empty_refused = True
+ try:
+ load(Path("/nonexistent-tree-for-the-self-test"))
+ empty_refused = False
+ except GateError:
+ # Refusing an empty tree is the behaviour under test, so the
+ # exception is the pass and there is nothing to handle.
+ pass
+ ok &= empty_refused
+ print(f" {'PASS' if empty_refused else 'FAIL'} TREE: a tree with no inputs is refused rather than reported clean")
+
+ print()
+ if not ok:
+ print("self-test FAILED: the gate did not catch what it claims to catch")
+ return 1
+ print(f"self-test OK: {len(cases)} injected defects plus the empty-tree case, each caught")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/services/connectors/tests/connectors/test_event_catalog_fixtures.py b/services/connectors/tests/connectors/test_event_catalog_fixtures.py
new file mode 100644
index 000000000..27f778d9c
--- /dev/null
+++ b/services/connectors/tests/connectors/test_event_catalog_fixtures.py
@@ -0,0 +1,146 @@
+"""Vendor-shaped fixtures for the two identity and SaaS sources that had none.
+
+Okta and Slack are two of the ten sources the event catalogue covers, and
+before this file neither had a single recorded vendor payload anywhere in the
+connectors suite — their only coverage was the schema and conformance checks
+that never see a record. The catalogue gate is grown from fixtures, so a
+source with no fixture is a source the gate passes over forever while
+reporting OK.
+
+These drive the real connectors' ``normalize()`` over the recorded shapes, so
+the fixtures are exercised rather than merely present, and the event type the
+catalogue classifies is asserted to survive the connector.
+
+Shapes follow each vendor's published log schema:
+
+ Okta System Log
+ https://developer.okta.com/docs/reference/api/system-log/
+ Slack Audit Logs API
+ https://docs.slack.dev/admins/audit-logs-api
+"""
+
+from __future__ import annotations
+
+import ipaddress
+import json
+from pathlib import Path
+
+import pytest
+import yaml
+from app.connectors.okta import OktaConnector
+from app.connectors.slack_audit import SlackAuditConnector
+
+_FIXTURES = Path(__file__).resolve().parent.parent / "fixtures"
+_CATALOG = Path(__file__).resolve().parents[4] / "schemas" / "event_catalog"
+
+
+def _fixture(name: str) -> list[dict]:
+ return json.loads((_FIXTURES / name / "sample_event.json").read_text())
+
+
+def _catalog(name: str) -> dict:
+ return yaml.safe_load((_CATALOG / f"{name}.yaml").read_text())
+
+
+# ── Okta ───────────────────────────────────────────────────────────────────
+
+
[email protected](scope="module")
+def okta_events() -> list[dict]:
+ return _fixture("okta")
+
+
+def test_okta_normalizes_every_recorded_shape(okta_events: list[dict]) -> None:
+ connector = OktaConnector(domain="https://example.okta.com", api_token="t")
+ for raw in okta_events:
+ out = connector.normalize(raw)
+ assert out["source"] == "okta"
+ assert out["event_type"] == raw["eventType"]
+ assert out["raw_event"] is raw, "the vendor record must survive under raw_event, never `raw`"
+ assert out["severity"] in {"info", "low", "medium", "high", "critical"}
+
+
+def test_okta_outcome_and_lockout_drive_severity(okta_events: list[dict]) -> None:
+ """The two escalations the connector actually makes, on recorded records
+ rather than on a payload written to match the code."""
+ connector = OktaConnector(domain="https://example.okta.com", api_token="t")
+ by_type: dict[str, list[dict]] = {}
+ for raw in okta_events:
+ by_type.setdefault(raw["eventType"], []).append(connector.normalize(raw))
+
+ failures = [o for o in by_type["user.session.start"] if "INVALID_CREDENTIALS" in (o["description"] or "")]
+ assert failures and all(o["severity"] == "medium" for o in failures)
+ assert all(o["severity"] == "high" for o in by_type["user.account.lock"])
+ assert all(o["severity"] == "high" for o in by_type["security.request.blocked"])
+
+
+def test_every_okta_fixture_event_type_is_in_the_catalogue(okta_events: list[dict]) -> None:
+ catalog = _catalog("okta")
+ known = set(catalog["events"]) | set(catalog.get("unclassified") or {})
+ assert {raw["eventType"] for raw in okta_events} <= known
+
+
+# ── Slack ──────────────────────────────────────────────────────────────────
+
+
[email protected](scope="module")
+def slack_events() -> list[dict]:
+ return _fixture("slack_audit")
+
+
+def test_slack_normalizes_every_recorded_shape(slack_events: list[dict]) -> None:
+ connector = SlackAuditConnector(access_token="t")
+ for raw in slack_events:
+ out = connector.normalize(raw)
+ assert out["source"] == "slack_audit"
+ assert out["title"].endswith(raw["action"])
+ assert out["raw_event"] is raw
+ assert out["alert_id"] == raw["id"]
+
+
+def test_every_slack_fixture_event_type_is_in_the_catalogue(slack_events: list[dict]) -> None:
+ catalog = _catalog("slack_audit")
+ known = set(catalog["events"]) | set(catalog.get("unclassified") or {})
+ assert {raw["action"] for raw in slack_events} <= known
+
+
+# ── The fixtures themselves ────────────────────────────────────────────────
+
+
[email protected](("source", "key"), [("okta", "eventType"), ("slack_audit", "action")])
+def test_a_fixture_carries_more_than_one_event_type(source: str, key: str) -> None:
+ """A single-shape fixture makes the catalogue gate pass on one entry and
+ say nothing about the rest, which is the vacuous case these exist to
+ avoid."""
+ records = _fixture(source)
+ assert len({r[key] for r in records}) >= 5
+
+
[email protected](("source", "ip_keys"), [("okta", {"ipAddress"}), ("slack_audit", {"ip_address"})])
+def test_a_fixture_uses_documentation_addresses_only(source: str, ip_keys: set[str]) -> None:
+ """A recorded fixture must not carry a real routable address.
+
+ Fixtures are read as examples by tests, hunts and anyone learning the
+ shape, and an address that resolves somewhere is an address somebody
+ eventually queries. Checked on the keys that actually hold addresses
+ rather than by scanning the blob — a first version of this test regexed
+ the whole JSON and failed on the version in `Chrome/124.0.0.0`.
+ """
+ addresses: list[str] = []
+
+ def walk(node: object) -> None:
+ if isinstance(node, dict):
+ for key, value in node.items():
+ if key in ip_keys and isinstance(value, str):
+ addresses.append(value)
+ walk(value)
+ elif isinstance(node, list):
+ for item in node:
+ walk(item)
+
+ walk(_fixture(source))
+ assert addresses, f"{source} fixture carries no address at {ip_keys}; the check would be vacuous"
+ documentation = (ipaddress.ip_network(net) for net in ("192.0.2.0/24", "198.51.100.0/24", "203.0.113.0/24"))
+ allowed = list(documentation)
+ routable = [a for a in addresses if not any(ipaddress.ip_address(a) in net for net in allowed)]
+ assert not routable, f"{source} fixture carries non-documentation addresses: {routable}"
diff --git a/services/connectors/tests/fixtures/okta/sample_event.json b/services/connectors/tests/fixtures/okta/sample_event.json
new file mode 100644
index 000000000..e4a3c5b4e
--- /dev/null
+++ b/services/connectors/tests/fixtures/okta/sample_event.json
@@ -0,0 +1,90 @@
+[
+ {
+ "uuid": "a1b2c3d4-0000-4000-8000-000000000001",
+ "published": "2026-10-07T09:14:02.413Z",
+ "eventType": "user.session.start",
+ "displayMessage": "User login to Okta",
+ "severity": "INFO",
+ "actor": {
+ "id": "00u1a2b3c4d5e6f7g8h9",
+ "type": "User",
+ "alternateId": "[email protected]",
+ "displayName": "Alice Example"
+ },
+ "client": {
+ "ipAddress": "203.0.113.10",
+ "userAgent": {
+ "rawUserAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
+ "os": "Mac OS X",
+ "browser": "CHROME"
+ },
+ "zone": "null",
+ "device": "Computer"
+ },
+ "outcome": { "result": "SUCCESS" },
+ "target": [
+ { "id": "0oa1a2b3c4d5e6f7g8h9", "type": "AppInstance", "alternateId": "AWS Account Federation", "displayName": "AWS" }
+ ]
+ },
+ {
+ "uuid": "a1b2c3d4-0000-4000-8000-000000000002",
+ "published": "2026-10-07T09:15:40.001Z",
+ "eventType": "user.session.start",
+ "displayMessage": "User login to Okta",
+ "severity": "WARN",
+ "actor": { "id": "00u1a2b3c4d5e6f7g8h9", "type": "User", "alternateId": "[email protected]", "displayName": "Alice Example" },
+ "client": { "ipAddress": "198.51.100.44" },
+ "outcome": { "result": "FAILURE", "reason": "INVALID_CREDENTIALS" }
+ },
+ {
+ "uuid": "a1b2c3d4-0000-4000-8000-000000000003",
+ "published": "2026-10-07T09:20:11.500Z",
+ "eventType": "user.account.lock",
+ "displayMessage": "Max sign in attempts exceeded",
+ "severity": "WARN",
+ "actor": { "id": "00u1a2b3c4d5e6f7g8h9", "type": "User", "alternateId": "[email protected]", "displayName": "Alice Example" },
+ "client": { "ipAddress": "198.51.100.44" },
+ "outcome": { "result": "SUCCESS" }
+ },
+ {
+ "uuid": "a1b2c3d4-0000-4000-8000-000000000004",
+ "published": "2026-10-07T09:31:07.220Z",
+ "eventType": "user.account.privilege.grant",
+ "displayMessage": "Grant user privilege",
+ "severity": "INFO",
+ "actor": { "id": "00uADMIN0000000000", "type": "User", "alternateId": "[email protected]", "displayName": "Admin Example" },
+ "client": { "ipAddress": "203.0.113.11" },
+ "outcome": { "result": "SUCCESS" },
+ "target": [{ "id": "00u1a2b3c4d5e6f7g8h9", "type": "User", "alternateId": "[email protected]" }]
+ },
+ {
+ "uuid": "a1b2c3d4-0000-4000-8000-000000000005",
+ "published": "2026-10-07T09:42:55.900Z",
+ "eventType": "system.api_token.create",
+ "displayMessage": "Create API token",
+ "severity": "INFO",
+ "actor": { "id": "00uADMIN0000000000", "type": "User", "alternateId": "[email protected]", "displayName": "Admin Example" },
+ "client": { "ipAddress": "203.0.113.11" },
+ "outcome": { "result": "SUCCESS" }
+ },
+ {
+ "uuid": "a1b2c3d4-0000-4000-8000-000000000006",
+ "published": "2026-10-07T09:50:02.000Z",
+ "eventType": "user.mfa.factor.deactivate",
+ "displayMessage": "Deactivate factor for user",
+ "severity": "INFO",
+ "actor": { "id": "00uSYSTEM000000000", "type": "SystemPrincipal", "displayName": "Okta System" },
+ "client": { "ipAddress": "203.0.113.12" },
+ "outcome": { "result": "SUCCESS" }
+ },
+ {
+ "uuid": "a1b2c3d4-0000-4000-8000-000000000007",
+ "published": "2026-10-07T10:02:19.100Z",
+ "eventType": "security.request.blocked",
+ "displayMessage": "Request from suspicious actor blocked",
+ "severity": "WARN",
+ "actor": { "id": "unknown", "type": "User", "displayName": "unknown" },
+ "client": { "ipAddress": "198.51.100.200" },
+ "outcome": { "result": "DENY", "reason": "Suspicious activity" }
+ }
+]
diff --git a/services/connectors/tests/fixtures/slack_audit/sample_event.json b/services/connectors/tests/fixtures/slack_audit/sample_event.json
new file mode 100644
index 000000000..fdb6d2da0
--- /dev/null
+++ b/services/connectors/tests/fixtures/slack_audit/sample_event.json
@@ -0,0 +1,57 @@
+[
+ {
+ "id": "0123a45b-6c7d-8e90-1234-56789abcdef0",
+ "date_create": 1791367200,
+ "action": "user_login",
+ "actor": {
+ "type": "user",
+ "user": { "id": "W123AB456", "name": "alice", "email": "[email protected]", "team": "T1234ABCD" }
+ },
+ "entity": { "type": "user", "user": { "id": "W123AB456", "name": "alice", "email": "[email protected]" } },
+ "context": {
+ "location": { "type": "enterprise", "id": "E1701", "name": "Example", "domain": "example" },
+ "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
+ "ip_address": "203.0.113.21"
+ }
+ },
+ {
+ "id": "0123a45b-6c7d-8e90-1234-56789abcdef1",
+ "date_create": 1791367800,
+ "action": "file_downloaded",
+ "actor": { "type": "user", "user": { "id": "W123AB456", "name": "alice", "email": "[email protected]" } },
+ "entity": { "type": "file", "file": { "id": "F0123ABCD", "name": "q4-forecast.xlsx", "filetype": "xlsx", "title": "Q4 forecast" } },
+ "context": { "ua": "Slack/4.37.101 CFNetwork/1494.0.7 Darwin/23.4.0", "ip_address": "203.0.113.21" }
+ },
+ {
+ "id": "0123a45b-6c7d-8e90-1234-56789abcdef2",
+ "date_create": 1791368400,
+ "action": "app_installed",
+ "actor": { "type": "user", "user": { "id": "WADMIN0001", "name": "admin", "email": "[email protected]" } },
+ "entity": { "type": "app", "app": { "id": "A0123ABCD", "name": "Unvetted Exporter", "is_distributed": false, "scopes": ["files:read", "channels:history"] } },
+ "context": { "ua": "Mozilla/5.0", "ip_address": "203.0.113.22" }
+ },
+ {
+ "id": "0123a45b-6c7d-8e90-1234-56789abcdef3",
+ "date_create": 1791369000,
+ "action": "user_session_reset_by_admin",
+ "actor": { "type": "user", "user": { "id": "WADMIN0001", "name": "admin", "email": "[email protected]" } },
+ "entity": { "type": "user", "user": { "id": "W123AB456", "name": "alice" } },
+ "context": { "ua": "Mozilla/5.0", "ip_address": "203.0.113.22" }
+ },
+ {
+ "id": "0123a45b-6c7d-8e90-1234-56789abcdef4",
+ "date_create": 1791369600,
+ "action": "public_channel_created",
+ "actor": { "type": "user", "user": { "id": "W123AB456", "name": "alice" } },
+ "entity": { "type": "channel", "channel": { "id": "C0123ABCD", "name": "incident-2026-10-07", "privacy": "public", "is_shared": false } },
+ "context": { "ua": "Mozilla/5.0", "ip_address": "203.0.113.21" }
+ },
+ {
+ "id": "0123a45b-6c7d-8e90-1234-56789abcdef5",
+ "date_create": 1791370200,
+ "action": "organization_export_started",
+ "actor": { "type": "user", "user": { "id": "WOWNER0001", "name": "owner", "email": "[email protected]" } },
+ "entity": { "type": "workspace", "workspace": { "id": "T1234ABCD", "name": "Example", "domain": "example" } },
+ "context": { "ua": "Mozilla/5.0", "ip_address": "203.0.113.23" }
+ }
+]
diff --git a/services/ingest/internal/eventcatalog/catalog.go b/services/ingest/internal/eventcatalog/catalog.go
new file mode 100644
index 000000000..75f5b7e5d
--- /dev/null
+++ b/services/ingest/internal/eventcatalog/catalog.go
@@ -0,0 +1,275 @@
+// Package eventcatalog loads the event classification catalogue at boot and
+// answers "what does this vendor event type mean" for every normalized event.
+//
+// The catalogue is a reviewed data file per source, mapping the vendor's own
+// event type onto a normalized action, a sensitivity on the platform's
+// five-tier ladder, and an optional ATT&CK hint. The authoritative copies
+// live at `schemas/event_catalog/<source>.yaml`; the copies under `catalog/`
+// here are byte-identical and are what the binary carries.
+//
+// Why a vendored copy rather than reading `schemas/` at run time
+// --------------------------------------------------------------
+// `go:embed` cannot reach above its own package, and the alternative —
+// reading the directory from disk at a configurable path — is a failure this
+// service has already shipped once: the webhook templates were read from
+// `/app/templates`, a path the Dockerfile never populated, so every
+// deployment answered 503 for every template behind a startup warning
+// nothing was watching (see templates.go). Embedding removes the class
+// rather than the instance, and `scripts/check_event_catalog.py` holds the
+// two directories byte-identical in both directions so the copy cannot
+// drift from the file a reviewer edits.
+//
+// Why a sensitivity does not change a severity
+// --------------------------------------------
+// Sensitivity says how security-relevant an event *type* is. The vendor's
+// own severity on the record, and the OCSF class, still decide whether
+// fusion promotes it. Letting a `critical` sensitivity force an alert would
+// make this data file able to flood a queue from a one-line edit, and the
+// promotion contract belongs where it already is. The classification rides
+// on the event for detections, hunts and triage to read.
+package eventcatalog
+
+import (
+ "embed"
+ "fmt"
+ "io/fs"
+ "path"
+ "sort"
+ "strings"
+
+ "gopkg.in/yaml.v3"
+)
+
+//go:embed catalog/*.yaml
+var files embed.FS
+
+const catalogDir = "catalog"
+
+// Sensitivity is how security-relevant an event type is, on the same
+// five-tier ladder the rest of the platform uses.
+type Sensitivity string
+
+const (
+ SensitivityInfo Sensitivity = "info"
+ SensitivityLow Sensitivity = "low"
+ SensitivityMedium Sensitivity = "medium"
+ SensitivityHigh Sensitivity = "high"
+ SensitivityCritical Sensitivity = "critical"
+)
+
+// Sensitivities is the closed set, lowest first. A value outside it is a
+// load error rather than a silently-kept string: the ladder is five tiers
+// everywhere in this platform and a sixth would be invisible until a query
+// filtered on it and found nothing.
+var Sensitivities = []Sensitivity{
+ SensitivityInfo, SensitivityLow, SensitivityMedium, SensitivityHigh, SensitivityCritical,
+}
+
+// Classification is one catalogue entry.
+type Classification struct {
+ Action string `yaml:"action"`
+ Sensitivity Sensitivity `yaml:"sensitivity"`
+ ATTACK []string `yaml:"attack,omitempty"`
+}
+
+// sourceFile is the on-disk shape of one `<source>.yaml`.
+type sourceFile struct {
+ Source string `yaml:"source"`
+ // EventTypePath is where the vendor's event type lives on its own
+ // record, in the dotted form `a.b` with `a[].b` for a list.
+ EventTypePath string `yaml:"event_type_path"`
+ // FixtureRequires are keys a record must also carry for the gate to
+ // read it as this source's event. `action` alone appears on records
+ // from several sources and in unrelated test dictionaries.
+ FixtureRequires []string `yaml:"fixture_requires,omitempty"`
+ FixtureFiles []string `yaml:"fixture_files"`
+ Events map[string]Classification `yaml:"events"`
+ // Unclassified is event type -> why it is deliberately not classified.
+ // Read here as well as by the gate so a loaded catalogue can say "this
+ // type is known and deliberately unclassified", which is a different
+ // answer from "never seen".
+ Unclassified map[string]string `yaml:"unclassified"`
+}
+
+// Catalog is every source's classifications, loaded.
+type Catalog struct {
+ bySource map[string]map[string]Classification
+ unclassified map[string]map[string]string
+ paths map[string]string
+}
+
+// Load reads the embedded catalogue. It returns an error rather than a
+// partial catalogue: a source that failed to parse would answer "never seen"
+// for every one of its event types, which is indistinguishable from a source
+// nobody has written a catalogue for.
+func Load() (*Catalog, error) {
+ entries, err := fs.ReadDir(files, catalogDir)
+ if err != nil {
+ return nil, fmt.Errorf("reading the embedded catalogue: %w", err)
+ }
+ c := &Catalog{
+ bySource: map[string]map[string]Classification{},
+ unclassified: map[string]map[string]string{},
+ paths: map[string]string{},
+ }
+ for _, entry := range entries {
+ if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".yaml") {
+ continue
+ }
+ raw, err := files.ReadFile(path.Join(catalogDir, entry.Name()))
+ if err != nil {
+ return nil, fmt.Errorf("reading %s: %w", entry.Name(), err)
+ }
+ var parsed sourceFile
+ if err := yaml.Unmarshal(raw, &parsed); err != nil {
+ return nil, fmt.Errorf("parsing %s: %w", entry.Name(), err)
+ }
+ if parsed.Source == "" {
+ return nil, fmt.Errorf("%s declares no source", entry.Name())
+ }
+ if parsed.EventTypePath == "" {
+ return nil, fmt.Errorf("%s declares no event_type_path, so nothing can find its event type", entry.Name())
+ }
+ if existing, dup := c.paths[parsed.Source]; dup {
+ return nil, fmt.Errorf("two catalogue files declare source %q (%s and %s)", parsed.Source, existing, entry.Name())
+ }
+ for eventType, classification := range parsed.Events {
+ if !validSensitivity(classification.Sensitivity) {
+ return nil, fmt.Errorf(
+ "%s: %q declares sensitivity %q, which is not one of the five tiers",
+ entry.Name(), eventType, classification.Sensitivity,
+ )
+ }
+ if classification.Action == "" {
+ return nil, fmt.Errorf("%s: %q declares no normalized action", entry.Name(), eventType)
+ }
+ }
+ c.bySource[parsed.Source] = parsed.Events
+ c.unclassified[parsed.Source] = parsed.Unclassified
+ c.paths[parsed.Source] = parsed.EventTypePath
+ }
+ if len(c.bySource) == 0 {
+ return nil, fmt.Errorf("the embedded catalogue is empty — refusing to load a catalogue that classifies nothing")
+ }
+ return c, nil
+}
+
+func validSensitivity(value Sensitivity) bool {
+ for _, s := range Sensitivities {
+ if s == value {
+ return true
+ }
+ }
+ return false
+}
+
+// Lookup returns the classification for one source's event type.
+//
+// The second return distinguishes the three answers that matter: a hit, a
+// type the catalogue lists as deliberately unclassified, and one the
+// catalogue has never seen. Only the third is worth a warning on the event.
+func (c *Catalog) Lookup(source, eventType string) (Classification, Status) {
+ if c == nil {
+ return Classification{}, StatusNoCatalog
+ }
+ events, ok := c.bySource[source]
+ if !ok {
+ return Classification{}, StatusNoCatalog
+ }
+ if classification, hit := events[eventType]; hit {
+ return classification, StatusClassified
+ }
+ if _, excused := c.unclassified[source][eventType]; excused {
+ return Classification{}, StatusDeliberatelyUnclassified
+ }
+ return Classification{}, StatusUnknown
+}
+
+// Status is which of the three answers Lookup gave.
+type Status string
+
+const (
+ StatusClassified Status = "classified"
+ // StatusDeliberatelyUnclassified: somebody looked and decided.
+ StatusDeliberatelyUnclassified Status = "unclassified"
+ // StatusUnknown: this source has a catalogue and this type is not in
+ // it. The only one worth telling an operator about, because it is how
+ // the catalogue learns it is behind the vendor.
+ StatusUnknown Status = "unknown"
+ // StatusNoCatalog: no catalogue for this source at all.
+ StatusNoCatalog Status = "no_catalog"
+)
+
+// EventTypePath is where a source keeps its event type, as declared.
+func (c *Catalog) EventTypePath(source string) (string, bool) {
+ if c == nil {
+ return "", false
+ }
+ p, ok := c.paths[source]
+ return p, ok
+}
+
+// Sources is every source with a catalogue, sorted.
+func (c *Catalog) Sources() []string {
+ if c == nil {
+ return nil
+ }
+ out := make([]string, 0, len(c.bySource))
+ for source := range c.bySource {
+ out = append(out, source)
+ }
+ sort.Strings(out)
+ return out
+}
+
+// Size is the total number of classified event types, for the boot log.
+func (c *Catalog) Size() int {
+ if c == nil {
+ return 0
+ }
+ total := 0
+ for _, events := range c.bySource {
+ total += len(events)
+ }
+ return total
+}
+
+// EventType reads the vendor's event type out of a record, using the path
+// the source declared.
+//
+// Supports one list hop (`events[].name`), which is the only shape any
+// source here needs: Workspace bundles several events under one activity.
+// A deeper grammar would be a parser nothing in the corpus exercises.
+func EventType(record map[string]any, declaredPath string) (string, bool) {
+ if record == nil || declaredPath == "" {
+ return "", false
+ }
+ var cur any = record
+ for _, segment := range strings.Split(declaredPath, ".") {
+ listHop := strings.HasSuffix(segment, "[]")
+ key := strings.TrimSuffix(segment, "[]")
+
+ asMap, ok := cur.(map[string]any)
+ if !ok {
+ return "", false
+ }
+ cur = asMap[key]
+
+ if listHop {
+ items, ok := cur.([]any)
+ if !ok || len(items) == 0 {
+ return "", false
+ }
+ // The first element drives the classification, matching the
+ // connectors, which already pick `events[0]` to drive severity
+ // and the title.
+ cur = items[0]
+ }
+ }
+ value, ok := cur.(string)
+ if !ok {
+ return "", false
+ }
+ value = strings.TrimSpace(value)
+ return value, value != ""
+}
diff --git a/services/ingest/internal/eventcatalog/catalog/aws_cloudtrail.yaml b/services/ingest/internal/eventcatalog/catalog/aws_cloudtrail.yaml
new file mode 100644
index 000000000..311a7b9e7
--- /dev/null
+++ b/services/ingest/internal/eventcatalog/catalog/aws_cloudtrail.yaml
@@ -0,0 +1,36 @@
+# AWS CloudTrail. Event types are the API call names in `eventName`.
+#
+# Sensitivity reflects what the call class means, not what any one
+# occurrence scored: DescribeInstances is reconnaissance-shaped and happens
+# thousands of times an hour in a healthy account, DeleteTrail is an
+# incident on its own reading.
+source: aws_cloudtrail
+event_type_path: eventName
+fixture_files:
+ - services/connectors/tests/test_aws_cloudtrail.py
+events:
+ ConsoleLogin:
+ action: console.login
+ sensitivity: medium
+ attack: [T1078.004]
+ CreateAccessKey:
+ action: create.access.key
+ sensitivity: high
+ attack: [T1098.001]
+ DeleteTrail:
+ # Destroying the audit trail is the one event where the absence of
+ # everything after it is the finding.
+ action: delete.trail
+ sensitivity: critical
+ attack: [T1562.008]
+ PutEventSelectors:
+ # Narrowing what the trail records is the quieter half of the same
+ # technique, which is why it sits one tier below rather than at info.
+ action: put.event.selectors
+ sensitivity: high
+ attack: [T1562.008]
+ DescribeInstances:
+ action: describe.instances
+ sensitivity: info
+ attack: [T1580]
+unclassified: {}
diff --git a/services/ingest/internal/eventcatalog/catalog/azure_activity.yaml b/services/ingest/internal/eventcatalog/catalog/azure_activity.yaml
new file mode 100644
index 000000000..cae5b0a42
--- /dev/null
+++ b/services/ingest/internal/eventcatalog/catalog/azure_activity.yaml
@@ -0,0 +1,16 @@
+# Azure Activity Log. Event types are the ARM operation names in
+# `operationName.value`, which are `Provider/resourceType/action`.
+source: azure_activity
+event_type_path: operationName.value
+fixture_files:
+ - services/connectors/tests/test_azure_connectors.py
+events:
+ Microsoft.Authorization/roleAssignments/write:
+ action: authorization.role.assignment.write
+ sensitivity: high
+ attack: [T1098.003]
+ Microsoft.Storage/storageAccounts/read:
+ action: storage.account.read
+ sensitivity: info
+ attack: [T1580]
+unclassified: {}
diff --git a/services/ingest/internal/eventcatalog/catalog/azure_entra.yaml b/services/ingest/internal/eventcatalog/catalog/azure_entra.yaml
new file mode 100644
index 000000000..e2274665c
--- /dev/null
+++ b/services/ingest/internal/eventcatalog/catalog/azure_entra.yaml
@@ -0,0 +1,18 @@
+# Microsoft Entra ID audit logs. Event types are the
+# `activityDisplayName` strings, which are prose rather than identifiers —
+# Microsoft's own vocabulary, kept verbatim so a lookup against a live
+# tenant's logs matches.
+source: azure_entra
+event_type_path: activityDisplayName
+fixture_files:
+ - services/connectors/tests/test_azure_connectors.py
+events:
+ Add member to role:
+ action: add.member.to.role
+ sensitivity: high
+ attack: [T1098.003]
+ Add user:
+ action: add.user
+ sensitivity: medium
+ attack: [T1136.003]
+unclassified: {}
diff --git a/services/ingest/internal/eventcatalog/catalog/gcp_cloud_audit.yaml b/services/ingest/internal/eventcatalog/catalog/gcp_cloud_audit.yaml
new file mode 100644
index 000000000..9dab8749c
--- /dev/null
+++ b/services/ingest/internal/eventcatalog/catalog/gcp_cloud_audit.yaml
@@ -0,0 +1,27 @@
+# GCP Cloud Audit Logs. Event types are the `methodName` on the proto
+# payload: either a bare admin method or the fully-qualified service method.
+source: gcp_cloud_audit
+event_type_path: methodName
+fixture_files:
+ - services/connectors/tests/test_gcp_connectors.py
+events:
+ SetIamPolicy:
+ # Rewrites who can do what on a resource, which is the single call an
+ # escalation most often ends with.
+ action: set.iam.policy
+ sensitivity: high
+ attack: [T1098.003]
+ google.compute.v1.instances.start:
+ action: compute.instances.start
+ sensitivity: low
+ google.compute.v1.instances.detachDisk:
+ # Detaching a disk is how a volume is moved to an instance the
+ # attacker controls, and it is otherwise rare.
+ action: compute.instances.detach.disk
+ sensitivity: medium
+ attack: [T1530]
+ google.storage.buckets.list:
+ action: storage.buckets.list
+ sensitivity: info
+ attack: [T1580]
+unclassified: {}
diff --git a/services/ingest/internal/eventcatalog/catalog/github.yaml b/services/ingest/internal/eventcatalog/catalog/github.yaml
new file mode 100644
index 000000000..2d7b5a010
--- /dev/null
+++ b/services/ingest/internal/eventcatalog/catalog/github.yaml
@@ -0,0 +1,25 @@
+# GitHub organisation audit log. Event types are the `action` values,
+# namespaced `area.verb`.
+source: github
+event_type_path: action
+fixture_requires: [actor]
+fixture_files:
+ - services/connectors/tests/test_saas_connectors.py
+events:
+ org.update_member:
+ action: org.update.member
+ sensitivity: high
+ attack: [T1098]
+ team.destroy:
+ action: team.destroy
+ sensitivity: medium
+ secret_scanning.disable:
+ # Turning off the control that finds leaked credentials, in the system
+ # that holds the code they are leaked in.
+ action: secret.scanning.disable
+ sensitivity: critical
+ attack: [T1562.001]
+ issues.opened:
+ action: issues.opened
+ sensitivity: info
+unclassified: {}
diff --git a/services/ingest/internal/eventcatalog/catalog/google_workspace.yaml b/services/ingest/internal/eventcatalog/catalog/google_workspace.yaml
new file mode 100644
index 000000000..43ddf9a1f
--- /dev/null
+++ b/services/ingest/internal/eventcatalog/catalog/google_workspace.yaml
@@ -0,0 +1,32 @@
+# Google Workspace Reports API. Event types are the `name` of each entry in
+# the activity's `events` list; one activity can carry several.
+source: google_workspace
+event_type_path: events[].name
+fixture_files:
+ - services/connectors/tests/test_saas_connectors.py
+events:
+ GRANT_ADMIN_PRIVILEGE:
+ action: grant.admin.privilege
+ sensitivity: critical
+ attack: [T1098]
+ suspicious_login:
+ # Google has already judged this one, which is why it is not merely a
+ # sign-in.
+ action: suspicious.login
+ sensitivity: high
+ attack: [T1078]
+ login_failure:
+ # On its own this is noise; in volume it is a spray. Sensitivity is a
+ # property of the event class, and the windowed engine is what reads
+ # the volume.
+ action: login.failure
+ sensitivity: low
+ attack: [T1110]
+ change_acl_editors:
+ action: change.acl.editors
+ sensitivity: medium
+ attack: [T1222]
+ view:
+ action: view
+ sensitivity: info
+unclassified: {}
diff --git a/services/ingest/internal/eventcatalog/catalog/kubernetes_audit.yaml b/services/ingest/internal/eventcatalog/catalog/kubernetes_audit.yaml
new file mode 100644
index 000000000..26676881b
--- /dev/null
+++ b/services/ingest/internal/eventcatalog/catalog/kubernetes_audit.yaml
@@ -0,0 +1,33 @@
+# Kubernetes apiserver audit. The event type is the `verb`, and on its own
+# it says nothing about what was acted on — `create` on a ConfigMap and
+# `create` on a ClusterRoleBinding are not the same event.
+#
+# The catalogue classifies the verb, which is what this source's event type
+# *is*, and the resource is carried separately on the event. `impersonate`
+# is the exception that needs no resource to be interesting.
+source: kubernetes_audit
+event_type_path: verb
+fixture_requires: [objectRef]
+fixture_files:
+ - services/connectors/tests/test_kubernetes_audit.py
+events:
+ get:
+ action: get
+ sensitivity: info
+ list:
+ action: list
+ sensitivity: info
+ attack: [T1613]
+ create:
+ action: create
+ sensitivity: medium
+ patch:
+ action: patch
+ sensitivity: medium
+ impersonate:
+ # The only verb that is a privilege decision by itself: the request is
+ # asking the apiserver to act as somebody else.
+ action: impersonate
+ sensitivity: critical
+ attack: [T1134]
+unclassified: {}
diff --git a/services/ingest/internal/eventcatalog/catalog/m365_audit.yaml b/services/ingest/internal/eventcatalog/catalog/m365_audit.yaml
new file mode 100644
index 000000000..d6c7a0005
--- /dev/null
+++ b/services/ingest/internal/eventcatalog/catalog/m365_audit.yaml
@@ -0,0 +1,28 @@
+# Microsoft 365 Management Activity API. Event types are the `Operation`
+# values, which span every workload in one stream.
+source: m365_audit
+event_type_path: Operation
+fixture_files:
+ - services/connectors/tests/test_saas_connectors.py
+events:
+ UserLoggedIn:
+ action: user.logged.in
+ sensitivity: low
+ attack: [T1078]
+ Add member to role:
+ action: add.member.to.role
+ sensitivity: high
+ attack: [T1098.003]
+ New-InboxRule:
+ # An inbox rule created by an attacker is how a compromised mailbox
+ # stays quiet: replies to the real owner are filed away unread.
+ action: new.inbox.rule
+ sensitivity: high
+ attack: [T1114.003]
+ FileAccessed:
+ action: file.accessed
+ sensitivity: info
+ FilePreviewed:
+ action: file.previewed
+ sensitivity: info
+unclassified: {}
diff --git a/services/ingest/internal/eventcatalog/catalog/okta.yaml b/services/ingest/internal/eventcatalog/catalog/okta.yaml
new file mode 100644
index 000000000..b86302650
--- /dev/null
+++ b/services/ingest/internal/eventcatalog/catalog/okta.yaml
@@ -0,0 +1,38 @@
+# Okta System Log. Event types are the dotted `eventType` strings, which
+# Okta namespaces by subject — `user.session.*`, `system.api_token.*`,
+# `security.*`.
+source: okta
+event_type_path: eventType
+fixture_files:
+ - services/connectors/tests/fixtures/okta/sample_event.json
+events:
+ user.session.start:
+ # Both a success and a failure arrive under this one type; the outcome
+ # is on the record, which is why the class sits at low rather than
+ # being split into two entries the vendor does not have.
+ action: user.session.start
+ sensitivity: low
+ attack: [T1078]
+ user.account.lock:
+ action: user.account.lock
+ sensitivity: high
+ attack: [T1110]
+ user.account.privilege.grant:
+ action: user.account.privilege.grant
+ sensitivity: critical
+ attack: [T1098]
+ system.api_token.create:
+ # A token outlives the session that minted it and carries the
+ # creator's authority with no second factor behind it.
+ action: system.api.token.create
+ sensitivity: high
+ attack: [T1098.001]
+ user.mfa.factor.deactivate:
+ action: user.mfa.factor.deactivate
+ sensitivity: high
+ attack: [T1556.006]
+ security.request.blocked:
+ action: security.request.blocked
+ sensitivity: high
+ attack: [T1078]
+unclassified: {}
diff --git a/services/ingest/internal/eventcatalog/catalog/slack_audit.yaml b/services/ingest/internal/eventcatalog/catalog/slack_audit.yaml
new file mode 100644
index 000000000..f9b1cefb9
--- /dev/null
+++ b/services/ingest/internal/eventcatalog/catalog/slack_audit.yaml
@@ -0,0 +1,35 @@
+# Slack Audit Logs API. Event types are the `action` values, which Slack
+# names as past-tense verbs over an entity.
+source: slack_audit
+event_type_path: action
+fixture_requires: [actor]
+fixture_files:
+ - services/connectors/tests/fixtures/slack_audit/sample_event.json
+events:
+ user_login:
+ action: user.login
+ sensitivity: low
+ attack: [T1078]
+ file_downloaded:
+ action: file.downloaded
+ sensitivity: info
+ attack: [T1213]
+ app_installed:
+ # A third-party app arrives holding the scopes the installer consented
+ # to, which is how data leaves a workspace without anyone downloading
+ # anything.
+ action: app.installed
+ sensitivity: high
+ attack: [T1550.001]
+ user_session_reset_by_admin:
+ action: user.session.reset.by.admin
+ sensitivity: medium
+ public_channel_created:
+ action: public.channel.created
+ sensitivity: info
+ organization_export_started:
+ # Every message and file in the workspace, in one archive.
+ action: organization.export.started
+ sensitivity: critical
+ attack: [T1213]
+unclassified: {}
diff --git a/services/ingest/internal/eventcatalog/catalog_test.go b/services/ingest/internal/eventcatalog/catalog_test.go
new file mode 100644
index 000000000..56e33e893
--- /dev/null
+++ b/services/ingest/internal/eventcatalog/catalog_test.go
@@ -0,0 +1,196 @@
+package eventcatalog
+
+import (
+ "testing"
+)
+
+func load(t *testing.T) *Catalog {
+ t.Helper()
+ c, err := Load()
+ if err != nil {
+ t.Fatalf("Load: %v", err)
+ }
+ return c
+}
+
+// The catalogue is embedded, so a missing or malformed file is a build-time
+// or boot-time fact rather than a runtime surprise. A load that returned a
+// partial catalogue would answer "never seen" for every type in the file
+// that failed, which is indistinguishable from a source nobody classified.
+func TestTheEmbeddedCatalogueLoads(t *testing.T) {
+ c := load(t)
+ if len(c.Sources()) == 0 {
+ t.Fatal("no sources loaded")
+ }
+ if c.Size() == 0 {
+ t.Fatal("no event types loaded")
+ }
+ t.Logf("%d sources, %d event types", len(c.Sources()), c.Size())
+}
+
+// The ten sources depth plan 2.2's corpus names. Listed here rather than
+// derived from the directory, because the point is that *these* are covered
+// — a derived list would agree with whatever shipped.
+func TestTheSourcesThePlanNamesAreCovered(t *testing.T) {
+ c := load(t)
+ present := map[string]bool{}
+ for _, source := range c.Sources() {
+ present[source] = true
+ }
+ for _, source := range []string{
+ "aws_cloudtrail", "gcp_cloud_audit", "azure_activity", "azure_entra", "okta",
+ "google_workspace", "m365_audit", "github", "slack_audit", "kubernetes_audit",
+ } {
+ if !present[source] {
+ t.Errorf("no catalogue for %q", source)
+ }
+ }
+}
+
+// Lookup distinguishes three answers, and only one of them is worth telling
+// an operator about. Folding them together would either spam a warning for
+// every event from an unclassified source or hide the one case that means
+// the catalogue has fallen behind the vendor.
+func TestLookupDistinguishesTheThreeAnswers(t *testing.T) {
+ c := load(t)
+
+ classification, status := c.Lookup("aws_cloudtrail", "DeleteTrail")
+ if status != StatusClassified {
+ t.Fatalf("DeleteTrail: status = %q, want classified", status)
+ }
+ if classification.Sensitivity != SensitivityCritical {
+ t.Errorf("DeleteTrail: sensitivity = %q, want critical", classification.Sensitivity)
+ }
+ if len(classification.ATTACK) == 0 {
+ t.Error("DeleteTrail: no ATT&CK hint")
+ }
+
+ if _, status := c.Lookup("aws_cloudtrail", "SomethingNobodyHasSeen"); status != StatusUnknown {
+ t.Errorf("an unseen type in a catalogued source: status = %q, want unknown", status)
+ }
+ if _, status := c.Lookup("a_source_with_no_catalogue", "anything"); status != StatusNoCatalog {
+ t.Errorf("an uncatalogued source: status = %q, want no_catalog", status)
+ }
+}
+
+// A nil catalogue is a usable one. The normalizer's own test constructors
+// build a Normalizer directly, so the classification path has to tolerate
+// having no catalogue rather than panicking in a test nobody wrote.
+func TestANilCatalogueIsSafe(t *testing.T) {
+ var c *Catalog
+ if _, status := c.Lookup("aws_cloudtrail", "DeleteTrail"); status != StatusNoCatalog {
+ t.Errorf("status = %q, want no_catalog", status)
+ }
+ if _, ok := c.EventTypePath("aws_cloudtrail"); ok {
+ t.Error("a nil catalogue claimed to know an event-type path")
+ }
+ if c.Size() != 0 || c.Sources() != nil {
+ t.Error("a nil catalogue reported contents")
+ }
+}
+
+// Every sensitivity in the loaded catalogue is one of the five tiers, and
+// Load is what enforces it — a sixth tier would be invisible until a query
+// filtered on it and found nothing.
+func TestEverySensitivityIsOnTheLadder(t *testing.T) {
+ c := load(t)
+ allowed := map[Sensitivity]bool{}
+ for _, s := range Sensitivities {
+ allowed[s] = true
+ }
+ for _, source := range c.Sources() {
+ for eventType, classification := range c.bySource[source] {
+ if !allowed[classification.Sensitivity] {
+ t.Errorf("%s/%s: sensitivity %q is not on the ladder", source, eventType, classification.Sensitivity)
+ }
+ if classification.Action == "" {
+ t.Errorf("%s/%s: no normalized action", source, eventType)
+ }
+ }
+ }
+}
+
+// The declared path is how the classification finds the event type on a
+// vendor record. These are the shapes the ten sources actually use.
+func TestEventTypeReadsTheDeclaredPath(t *testing.T) {
+ cases := []struct {
+ name string
+ record map[string]any
+ path string
+ want string
+ found bool
+ }{
+ {
+ name: "a bare key",
+ record: map[string]any{"eventName": "CreateAccessKey"},
+ path: "eventName",
+ want: "CreateAccessKey",
+ found: true,
+ },
+ {
+ name: "a nested key, as Azure spells its operation",
+ record: map[string]any{"operationName": map[string]any{"value": "Microsoft.Authorization/roleAssignments/write"}},
+ path: "operationName.value",
+ want: "Microsoft.Authorization/roleAssignments/write",
+ found: true,
+ },
+ {
+ name: "a list hop, as Workspace bundles several events per activity",
+ record: map[string]any{
+ "events": []any{map[string]any{"name": "GRANT_ADMIN_PRIVILEGE"}, map[string]any{"name": "view"}},
+ },
+ path: "events[].name",
+ want: "GRANT_ADMIN_PRIVILEGE",
+ found: true,
+ },
+ {
+ name: "an empty list is not an event type",
+ record: map[string]any{"events": []any{}},
+ path: "events[].name",
+ found: false,
+ },
+ {
+ name: "a path that is not there",
+ record: map[string]any{"somethingElse": "x"},
+ path: "eventName",
+ found: false,
+ },
+ {
+ name: "a value that is not a string",
+ record: map[string]any{"verb": 7},
+ path: "verb",
+ found: false,
+ },
+ {
+ name: "whitespace is not an event type",
+ record: map[string]any{"verb": " "},
+ path: "verb",
+ found: false,
+ },
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ got, found := EventType(tc.record, tc.path)
+ if found != tc.found {
+ t.Fatalf("found = %v, want %v", found, tc.found)
+ }
+ if found && got != tc.want {
+ t.Errorf("got %q, want %q", got, tc.want)
+ }
+ })
+ }
+}
+
+// Every catalogue declares the path its own fixtures are read at, and the
+// loader is what the classification uses. A path declared in the YAML that
+// the loader does not expose would make the classification silently find
+// nothing for that source.
+func TestEverySourceDeclaresItsEventTypePath(t *testing.T) {
+ c := load(t)
+ for _, source := range c.Sources() {
+ path, ok := c.EventTypePath(source)
+ if !ok || path == "" {
+ t.Errorf("%s declares no event-type path", source)
+ }
+ }
+}
diff --git a/services/ingest/internal/normalizer/classify.go b/services/ingest/internal/normalizer/classify.go
new file mode 100644
index 000000000..90b088527
--- /dev/null
+++ b/services/ingest/internal/normalizer/classify.go
@@ -0,0 +1,89 @@
+package normalizer
+
+import (
+ "fmt"
+
+ "github.com/beenuar/aisoc/services/ingest/internal/eventcatalog"
+)
+
+// Event classification, from the catalogue loaded at boot.
+//
+// The catalogue maps a vendor's own event type onto a normalized action, a
+// sensitivity and an optional ATT&CK hint. This attaches the answer to the
+// event so the lake, the detection matcher and triage read one
+// classification rather than each re-deriving a meaning from the event name.
+//
+// It deliberately changes nothing else on the event. Sensitivity is a
+// property of the event *class*; the vendor's severity on the record and the
+// OCSF class still decide whether fusion promotes it. A `critical`
+// sensitivity that forced an alert would let a one-line edit to a data file
+// flood a queue.
+
+// eventClassification is what lands on the event.
+type eventClassification struct {
+ // Source and EventType are carried so a reader of the lake row can
+ // check the classification against the catalogue without the raw
+ // payload, and so a hunt can group by the vendor's own vocabulary.
+ Source string `json:"source"`
+ EventType string `json:"event_type"`
+ Action string `json:"action"`
+ Sensitivity string `json:"sensitivity"`
+ ATTACK []string `json:"attack,omitempty"`
+}
+
+// classify returns the classification for this event, or a warning
+// explaining why there is none.
+//
+// Three outcomes, and the distinction is the point:
+//
+// - a hit, which lands on the event;
+// - a type the catalogue lists as deliberately unclassified, or a source
+// with no catalogue at all, both of which are silent: somebody has
+// already decided, or nobody has started, and neither is news;
+// - a type this source's catalogue has never seen, which is a warning on
+// the event. That is the only case an operator can act on, and it is how
+// the catalogue learns it has fallen behind the vendor.
+func (n *Normalizer) classify(connectorType string, payload map[string]interface{}) (*eventClassification, string) {
+ declaredPath, hasCatalog := n.catalog.EventTypePath(connectorType)
+ if !hasCatalog {
+ return nil, ""
+ }
+
+ // The vendor's record, which for a canonical envelope is nested under
+ // `raw_event`. The key is `raw_event` and never `raw`: 26 connectors
+ // once emitted `raw`, missed the canonical-envelope check and fell
+ // through to a borrowed vendor profile.
+ record := payload
+ if nested, ok := payload["raw_event"].(map[string]interface{}); ok {
+ record = nested
+ }
+
+ eventType, found := eventcatalog.EventType(record, declaredPath)
+ if !found {
+ // The source has a catalogue and this record carries nothing at
+ // the declared path. Not a warning: several connectors emit shapes
+ // that genuinely have no event type, and warning on each would
+ // make the field unreadable.
+ return nil, ""
+ }
+
+ classification, status := n.catalog.Lookup(connectorType, eventType)
+ switch status {
+ case eventcatalog.StatusClassified:
+ return &eventClassification{
+ Source: connectorType,
+ EventType: eventType,
+ Action: classification.Action,
+ Sensitivity: string(classification.Sensitivity),
+ ATTACK: classification.ATTACK,
+ }, ""
+ case eventcatalog.StatusUnknown:
+ return nil, fmt.Sprintf(
+ "event type %q is not in the %s event catalogue: it is neither classified nor recorded as "+
+ "deliberately unclassified, so nothing downstream knows how sensitive it is "+
+ "(schemas/event_catalog/%s.yaml)",
+ eventType, connectorType, connectorType,
+ )
+ }
+ return nil, ""
+}
diff --git a/services/ingest/internal/normalizer/classify_test.go b/services/ingest/internal/normalizer/classify_test.go
new file mode 100644
index 000000000..433742c0e
--- /dev/null
+++ b/services/ingest/internal/normalizer/classify_test.go
@@ -0,0 +1,191 @@
+package normalizer
+
+import (
+ "strings"
+ "testing"
+
+ "github.com/beenuar/aisoc/services/ingest/internal/config"
+ "github.com/beenuar/aisoc/services/ingest/internal/eventcatalog"
+)
+
+// A normalizer with the real catalogue loaded, which is what New() builds.
+// The other constructors in this package build a Normalizer directly and so
+// carry no catalogue; that is deliberate and the nil path is asserted below.
+func newClassifyingNormalizer(t *testing.T) *Normalizer {
+ t.Helper()
+ catalog, err := eventcatalog.Load()
+ if err != nil {
+ t.Fatalf("loading the catalogue: %v", err)
+ }
+ return &Normalizer{cfg: &config.Config{NormalizerMode: "lenient"}, version: "test", catalog: catalog}
+}
+
+func normalizeWithVendorRecord(t *testing.T, connector string, vendor map[string]interface{}) *NormalizedEvent {
+ t.Helper()
+ ev, err := newClassifyingNormalizer(t).Normalize(&RawEvent{
+ ConnectorID: "c-1",
+ ConnectorType: connector,
+ TenantID: "11111111-1111-1111-1111-111111111111",
+ ReceivedAt: "2026-10-07T00:00:00Z",
+ Payload: map[string]interface{}{
+ "source": connector,
+ "raw_event": vendor,
+ "severity": "medium",
+ "title": "an event",
+ },
+ })
+ if err != nil {
+ t.Fatalf("Normalize(%q): %v", connector, err)
+ }
+ return ev
+}
+
+// The classification lands on the event, carrying the vendor's own event
+// type beside the normalized answer so a lake row can be checked against the
+// catalogue without the raw payload.
+func TestAClassifiedEventCarriesItsClassification(t *testing.T) {
+ ev := normalizeWithVendorRecord(t, "aws_cloudtrail", map[string]interface{}{"eventName": "DeleteTrail"})
+ classification, ok := ev.OcsfEvent["event_classification"].(*eventClassification)
+ if !ok {
+ t.Fatalf("no classification on the event: %T", ev.OcsfEvent["event_classification"])
+ }
+ if classification.Source != "aws_cloudtrail" || classification.EventType != "DeleteTrail" {
+ t.Errorf("classification = %+v", classification)
+ }
+ if classification.Sensitivity != "critical" {
+ t.Errorf("sensitivity = %q, want critical", classification.Sensitivity)
+ }
+ if classification.Action != "delete.trail" {
+ t.Errorf("action = %q", classification.Action)
+ }
+ if len(classification.ATTACK) == 0 {
+ t.Error("no ATT&CK hint carried through")
+ }
+ for _, w := range ev.NormalizationWarnings {
+ if strings.Contains(w, "event catalogue") {
+ t.Errorf("a classified event carried a catalogue warning: %s", w)
+ }
+ }
+}
+
+// The three shapes the declared paths cover, driven through the normalizer
+// rather than the loader, so the `raw_event` unwrapping is exercised too.
+func TestClassificationReadsEachSourcesOwnEventTypeShape(t *testing.T) {
+ cases := []struct {
+ connector string
+ vendor map[string]interface{}
+ want string
+ }{
+ {"okta", map[string]interface{}{"eventType": "system.api_token.create"}, "system.api_token.create"},
+ {
+ "azure_activity",
+ map[string]interface{}{"operationName": map[string]interface{}{"value": "Microsoft.Authorization/roleAssignments/write"}},
+ "Microsoft.Authorization/roleAssignments/write",
+ },
+ {
+ "google_workspace",
+ map[string]interface{}{"events": []interface{}{map[string]interface{}{"name": "GRANT_ADMIN_PRIVILEGE"}}},
+ "GRANT_ADMIN_PRIVILEGE",
+ },
+ {"kubernetes_audit", map[string]interface{}{"verb": "impersonate"}, "impersonate"},
+ {"slack_audit", map[string]interface{}{"action": "organization_export_started"}, "organization_export_started"},
+ }
+ for _, tc := range cases {
+ t.Run(tc.connector, func(t *testing.T) {
+ ev := normalizeWithVendorRecord(t, tc.connector, tc.vendor)
+ classification, ok := ev.OcsfEvent["event_classification"].(*eventClassification)
+ if !ok {
+ t.Fatalf("no classification for %q", tc.connector)
+ }
+ if classification.EventType != tc.want {
+ t.Errorf("event_type = %q, want %q", classification.EventType, tc.want)
+ }
+ })
+ }
+}
+
+// An event type a catalogued source has never seen is the one case worth
+// telling an operator about: it is how the catalogue learns it has fallen
+// behind the vendor. The warning rides on the event and names the file to
+// edit.
+func TestAnUnknownEventTypeWarnsOnTheEvent(t *testing.T) {
+ ev := normalizeWithVendorRecord(t, "aws_cloudtrail", map[string]interface{}{"eventName": "SomeNewApiCallAwsShippedLastWeek"})
+ if _, present := ev.OcsfEvent["event_classification"]; present {
+ t.Error("an unknown event type produced a classification")
+ }
+ var warning string
+ for _, w := range ev.NormalizationWarnings {
+ if strings.Contains(w, "event catalogue") {
+ warning = w
+ }
+ }
+ if warning == "" {
+ t.Fatal("an unknown event type produced no warning; nothing would ever grow the catalogue")
+ }
+ if !strings.Contains(warning, "SomeNewApiCallAwsShippedLastWeek") {
+ t.Errorf("the warning does not name the event type: %s", warning)
+ }
+ if !strings.Contains(warning, "schemas/event_catalog/aws_cloudtrail.yaml") {
+ t.Errorf("the warning does not name the file to edit: %s", warning)
+ }
+}
+
+// A source with no catalogue at all must be silent. Warning on every event
+// from the seventy-four connectors with no catalogue would make the
+// warnings field unreadable and the one actionable case invisible in it.
+func TestASourceWithNoCatalogueIsSilent(t *testing.T) {
+ ev := normalizeWithVendorRecord(t, "zscaler", map[string]interface{}{"action": "Allow"})
+ if _, present := ev.OcsfEvent["event_classification"]; present {
+ t.Error("a source with no catalogue produced a classification")
+ }
+ for _, w := range ev.NormalizationWarnings {
+ if strings.Contains(w, "event catalogue") {
+ t.Errorf("a source with no catalogue warned: %s", w)
+ }
+ }
+}
+
+// A record from a catalogued source that carries nothing at the declared
+// path is also silent: several connectors emit shapes with no event type at
+// all, and warning on each would drown the actionable case.
+func TestARecordWithNoEventTypeIsSilent(t *testing.T) {
+ ev := normalizeWithVendorRecord(t, "aws_cloudtrail", map[string]interface{}{"somethingElse": "x"})
+ for _, w := range ev.NormalizationWarnings {
+ if strings.Contains(w, "event catalogue") {
+ t.Errorf("a record with no event type warned: %s", w)
+ }
+ }
+}
+
+// The classification must not change what the event is or whether it
+// promotes. A `critical` sensitivity that raised a severity would let a
+// one-line edit to a data file flood an alert queue, and the promotion
+// contract belongs with the OCSF class and the vendor's severity.
+func TestTheClassificationChangesNeitherSeverityNorClass(t *testing.T) {
+ critical := normalizeWithVendorRecord(t, "aws_cloudtrail", map[string]interface{}{"eventName": "DeleteTrail"})
+ informational := normalizeWithVendorRecord(t, "aws_cloudtrail", map[string]interface{}{"eventName": "DescribeInstances"})
+
+ for _, field := range []string{"severity_id", "severity", "class_uid", "category_uid"} {
+ if critical.OcsfEvent[field] != informational.OcsfEvent[field] {
+ t.Errorf("%s differs between a critical-sensitivity and an info-sensitivity event (%v vs %v); "+
+ "the catalogue must not move promotion",
+ field, critical.OcsfEvent[field], informational.OcsfEvent[field])
+ }
+ }
+}
+
+// New() loads the catalogue, which is what makes "ingest reads it at boot"
+// true rather than a claim. A failure there is fatal, so this asserts the
+// successful path reaches a normalizer that can classify.
+func TestNewLoadsTheCatalogueAtBoot(t *testing.T) {
+ n, err := New(&config.Config{NormalizerMode: "lenient"})
+ if err != nil {
+ t.Fatalf("New: %v", err)
+ }
+ if n.catalog == nil {
+ t.Fatal("New() returned a normalizer with no catalogue")
+ }
+ if n.catalog.Size() == 0 {
+ t.Fatal("New() loaded an empty catalogue")
+ }
+}
diff --git a/services/ingest/internal/normalizer/normalizer.go b/services/ingest/internal/normalizer/normalizer.go
index 1abb04329..4946023fd 100644
--- a/services/ingest/internal/normalizer/normalizer.go
+++ b/services/ingest/internal/normalizer/normalizer.go
@@ -12,6 +12,7 @@ import (
"github.com/beenuar/aisoc/services/ingest/internal/attck"
"github.com/beenuar/aisoc/services/ingest/internal/config"
"github.com/beenuar/aisoc/services/ingest/internal/enrichment"
+ "github.com/beenuar/aisoc/services/ingest/internal/eventcatalog"
"github.com/google/uuid"
"github.com/rs/zerolog/log"
)
@@ -78,6 +79,7 @@ type Normalizer struct {
version string
shodan *enrichment.ShodanEnricher
vulnCorrel *enrichment.VulnCorrelator
+ catalog *eventcatalog.Catalog
ipEnricher *activity.IPEnricher
// VulnMatches is a channel where VULNERABILITY_MATCH events are published.
@@ -409,9 +411,24 @@ func New(cfg *config.Config) (*Normalizer, error) {
log.Warn().Err(err).Msg("ATT&CK corpus unavailable; technique enrichment disabled")
}
+ // The event catalogue, at boot. A parse failure is fatal rather than
+ // logged: a half-loaded catalogue answers "never seen" for every type
+ // in the file that failed, which is indistinguishable from a source
+ // nobody has classified, and the warning that said so would have
+ // scrolled past weeks earlier.
+ catalog, err := eventcatalog.Load()
+ if err != nil {
+ return nil, fmt.Errorf("event catalogue: %w", err)
+ }
+ log.Info().
+ Int("sources", len(catalog.Sources())).
+ Int("event_types", catalog.Size()).
+ Msg("event classification catalogue loaded")
+
n := &Normalizer{
cfg: cfg,
version: "1.1.0",
+ catalog: catalog,
ipEnricher: activity.NewIPEnricher(
cfg.EnrichmentServiceURL,
cfg.ServiceToken,
@@ -827,6 +844,13 @@ func (n *Normalizer) Normalize(raw *RawEvent) (*NormalizedEvent, error) {
warnings = append(warnings, w)
}
+ // The event classification, from the catalogue read at boot.
+ if classification, warning := n.classify(connectorType, raw.Payload); classification != nil {
+ ocsf["event_classification"] = classification
+ } else if warning != "" {
+ warnings = append(warnings, warning)
+ }
+
// The activity projection: who acted, what kind of actor they were, what
// they did, to which resource, from where, with what outcome.
//