Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
Resolve identity on the generic normalizer path, and make connector names resolve both ways (#808)
Beenu Arora8cafcd97e184
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex cb579fc29..c5adf2ff6 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -280,6 +280,20 @@ jobs:# commit the result.- name: Verify connector count is in sync (Phase 1.1)run: python scripts/generate_connector_count.py --check+ # The count says how many connectors exist; this says whether the name+ # each one is registered under still reaches a normalization profile.+ # The connectors service declares `crowdstrike` and `okta` while the+ # ingest profiles were keyed `crowdstrike_falcon` and `okta_system_log`,+ # so events arriving under the advertised names — including the README's+ # own push example — missed the lookup and fell to the generic fallback.+ #+ # The self-test runs first and on the same tree: it injects drift in each+ # direction and requires the gate to catch each one, so a refactor that+ # quietly stops checking a direction fails here rather than printing OK.+ - name: Connector profile drift gate — self-test+ run: python3 scripts/check_connector_profiles.py --self-test+ - name: Connector profile drift gate+ run: python3 scripts/check_connector_profiles.py# The count claim above says how many connectors exist; this says# whether a reader can find out how to use them. 35 of 84 had no page# at all and the sidebar listed 34 of 96, so most connectordiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9536dac52..e43933a4e 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -77,6 +77,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added+- **`scripts/check_connector_profiles.py` — a connector-type drift gate that+ reads in both directions.** Nothing compared the profile keys in+ `services/ingest/internal/normalizer/normalizer.go` against the identifiers+ the connectors service declares, and they had drifted apart in both+ directions at once. The gate enumerates four name spaces — the Go profiles+ and type aliases, the 84 declared `connector_id`s, the `ConnectorType`+ union, and every `connector_type` appearing in a documented example — and+ fails when a name in any of them resolves to nothing.++ It also checks that a documented example reaches a *profile*, not merely a+ declared connector: a reader pastes a flat payload, which never carries the+ canonical envelope, so only a profile or alias can give the event a vendor+ identity. That is the specific check that catches the README defect.++ `--self-test` injects twelve defects, one per direction and per failure+ code, and requires the gate to catch each; it runs in CI on the same tree+ immediately before the gate itself. The resolved repo root, every file read+ and every count are printed before the verdict, and an input that is missing+ or parses empty is a hard error rather than a quiet pass — a gate that+ reports OK about a tree it never opened is worse than no gate.+- **Competitor product names removed from the docs portal, the benchmark pageand the archived plan subtree, and a CI gate added to keep them out.** AiSOCnames no competitor product, but two published comparison tables and the@@ -292,6 +313,61 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **An event ingested under a connector type with no profile became an alert+ with no host, no user and no source IP.** `_canonicalAliases` in+ `services/ingest/internal/normalizer/normalizer.go` already resolved+ `actor.user.name`, `device.name` and `src_endpoint.ip` from the spellings+ connectors actually use, but the pass that applied it was gated behind the+ canonical-envelope branch. Anything reaching the generic fallback — every+ push through `/v1/ingest/batch`, which sends a flat payload and so never+ matches that branch — resolved `title` and `external_id` and nothing else.++ Identity is what the rest of the platform is built on: entity extraction,+ the Investigation Rail's pivots, the `{tenant}:{entity}:{tactic}`+ correlation key, the entity graph and UEBA all key off those three fields.+ An alert still appeared, which is what made it look like it had worked, but+ it carried no entity chips and correlated into the `unknown` bucket. The+ alias pass now runs for every profile and fills only destinations the field+ map left empty, so a vendor profile's own mapping still wins.++ Three defects in the same family, found while fixing it:++ - **A nested vendor object was written whole into a scalar identity slot.**+ Fourteen of the 84 registered connectors emit `actor`, and four emit it as+ the vendor's object rather than a name, so `actor.user.name` became a map+ — an entity that renders as a map and correlates as garbage. Identity+ resolution now takes strings only and digs one level (`actor.name`,+ `actor.displayName`, `user.name`) for the scalar underneath.+ - **A vendor profile's severity ladder is spelled the way its vendor spells+ it.** `crowdstrike_falcon`'s is capitalised, and a pushed payload is+ whatever the caller wrote, so a lowercase `high` scored 0 and rendered as+ Unknown. The shared five-tier ladder is consulted when the profile's own+ has no entry for the value; `critical` stays its own tier.+ - **A vendor profile left `message` empty for a pushed payload**, because it+ maps its vendor's field name, so the promoter generated+ "Security Finding from <product>" over the title the caller actually sent.++- **`connector_type` values the product advertises did not reach their+ profile.** The connectors service declares `crowdstrike` and `okta`; the+ ingest profiles were keyed `crowdstrike_falcon` and `okta_system_log`. The+ README's own push example used `crowdstrike`, so the example a new user+ copies missed the lookup, fell to the generic fallback and lost its vendor+ attribution — 80 of the 84 declared connectors had no profile entry under+ the name they are registered with.++ A `connectorTypeAliases` map resolves both spellings rather than renaming+ either. The longer names are load-bearing elsewhere — `packages/types`'+ `ConnectorType` union, the CLI's default, the graph extractor, the actions+ credential resolver — so renaming one side would have broken the other.++- **Two registered connectors reached no normalization path at all.**+ `auditd` emitted `source` without `raw_event`, so it failed the canonical+ envelope check and lost the host it carries; it now emits both.+ `email_inbox` returns the message envelope shaped for+ `email-forwarded.yaml`, so it matched neither branch and strict mode+ rejected it outright; it now has a profile mirroring that template, the way+ `ai_runtime` mirrors `ai-runtime.yaml`.+- **The agent recommended evidence acquisition the platform could notperform.** `capture_forensics` was an `ActionType` with no executoranywhere, and `services/agents/app/agents/investigation_agent.py` proposesdiff --git a/README.md b/README.mdindex 7aaf9a460..cced87168 100644--- a/README.md+++ b/README.md@@ -94,8 +94,9 @@ Sentinel, Elastic, CrowdStrike, Okta, AWS (GuardDuty / CloudTrail / SecurityHub), Wiz, and Kubernetes audit logs. The full list is in the[connector docs](https://beenuar.github.io/AiSOC/docs/connectors/api-coverage).-A connector without a vendor profile still ingests through a generic mapping —-events flow, but entity extraction is weaker.+A connector without a vendor profile still ingests through a generic mapping,+which resolves host, user and source IP from the usual spellings so the alert+is still pivotable. A vendor profile adds that vendor's own field names on top.## How it worksdiff --git a/docs/architecture/README.md b/docs/architecture/README.mdindex 4ffb7fbfc..6c2fa756b 100644--- a/docs/architecture/README.md+++ b/docs/architecture/README.md@@ -25,8 +25,18 @@ curl -X POST http://localhost:8081/v1/ingest/batch \**2. Ingest normalizes it.** [`services/ingest`](../../services/ingest) mapsthe vendor payload onto a common OCSF-shaped envelope using a per-connector-profile. A connector with no profile falls through to a vendor-neutral-generic profile — the event still flows, but entity extraction is weaker.+profile. A connector with no profile falls through to a vendor-neutral generic+profile, which resolves `actor.user.name`, `device.name` and `src_endpoint.ip`+from the spellings connectors actually use (`user` / `username` / `actor`,+`host` / `hostname`, `src_ip` / `source_ip` / `client_ip`) in a fixed order.+The event keeps the entities the rest of the platform pivots on; what a vendor+profile adds is that vendor's own field names and its OCSF class.++`connector_type` is matched against the profile keys, and the identifiers the+connectors service declares are the ones to use — `crowdstrike`, not+`crowdstrike_falcon`. Both resolve, and+[`scripts/check_connector_profiles.py`](../../scripts/check_connector_profiles.py)+fails CI if any name in either direction stops resolving.**3. It enters the event spine.** Ingest publishes to the Kafka topic`aisoc.raw_events`. This is the boundary that makes everything downstreamdiff --git a/scripts/check_connector_profiles.py b/scripts/check_connector_profiles.pynew file mode 100644index 000000000..11bc82e1d--- /dev/null+++ b/scripts/check_connector_profiles.py@@ -0,0 +1,590 @@+#!/usr/bin/env python3+"""Connector-type drift gate: names must resolve, in both directions.++Why this exists+---------------+``connectorProfiles`` in ``services/ingest/internal/normalizer/normalizer.go``+decides how a connector's events become OCSF, and it is keyed by+``connector_type``. The connectors service declares its own identifiers in+``services/connectors/app/connectors/``. Nothing compared the two, and they+drifted: the service declares ``crowdstrike`` and ``okta`` while the profiles+were keyed ``crowdstrike_falcon`` and ``okta_system_log``, so events arriving+under the names the product advertises missed the lookup entirely. The README's+own push example was one of them.++A missing profile is not cosmetic. In strict mode the normalizer rejects an+unknown connector type outright. In lenient mode it falls to the generic+profile, and before the identity aliases were wired into that path the+resulting alert carried no host, no user and no source IP — so entity+extraction, the Investigation Rail's pivots, the ``{tenant}:{entity}:{tactic}``+correlation key, the entity graph and UEBA all had nothing to work with. The+alert still appeared, which is what made it look like it had worked.++What it checks, in both directions+----------------------------------+The repository's dominant bug shape is a one-directional check: it compares A+against B, never B against A, so drift in the direction things actually change+slips through while the check prints OK. Every rule below is therefore stated+as a resolution requirement on a specific name, and both name spaces are+enumerated.++ GO -> PY every ``connectorProfiles`` key and every ``connectorTypeAliases``+ entry must name something that exists.+ PY -> GO every declared ``connector_id`` must reach a normalization path+ that survives strict mode: its own profile, an alias to one, or+ the canonical ``raw_event`` + ``source`` envelope.+ DOCS -> * every ``connector_type`` in a documented example must resolve.+ MECHANISM the identity aliases the generic path depends on must still be+ declared, and the Go tests that prove the behaviour must exist.+ Coverage without the behaviour behind it is the vacuous case.++Usage+-----+ python3 scripts/check_connector_profiles.py # gate+ python3 scripts/check_connector_profiles.py --json+ python3 scripts/check_connector_profiles.py --self-test # prove it bites++``--repo-root`` overrides the tree under inspection. The resolved root, every+file read and every count are printed before the verdict, and a missing or+empty input is a hard error rather than a quiet zero: a gate that resolves its+own location and reports OK about a tree it never opened is worse than no gate.+"""++from __future__ import annotations++import argparse+import ast+import json+import re+import sys+from pathlib import Path++NORMALIZER_REL = Path("services/ingest/internal/normalizer/normalizer.go")+NORMALIZER_TEST_REL = Path("services/ingest/internal/normalizer/normalizer_test.go")+CONNECTORS_REL = Path("services/connectors/app/connectors")+TEMPLATES_REL = Path("services/ingest/internal/normalizer/templates")++# Profile keys that are push-path template names rather than connector ids.+# Each must name a webhook template that exists, so this cannot become a+# dumping ground for keys nobody can account for.+PUSH_ONLY_PROFILES: dict[str, str] = {+ "ai_runtime": "ai-runtime.yaml",+ "ai_guardrail": "ai-finding.yaml",+}++# Legacy profile keys that the wider platform uses as a connector_type even+# though the connectors service spells the id differently. They stay because+# packages/types' ConnectorType union, the CLI default, the graph extractor and+# the actions credential resolver all key off them; the connectorTypeAliases+# map is what makes the declared id resolve to the same profile. Each entry+# must be reachable via an alias or still declared in the union, which is+# checked against those files rather than asserted here.+LEGACY_PROFILE_KEYS = {"crowdstrike_falcon", "okta_system_log", "splunk_enterprise"}++# packages/types' ConnectorType union is a third name space, and it predates+# the connectors registry. These members name no connector the service+# declares and no profile: they are the console's older vocabulary+# (``google_chronicle`` for ``chronicle``, ``ibm_qradar`` for ``qradar``) plus+# four generic transports that were never connectors at all. They are recorded+# rather than fixed here because the union is shared with the console.+#+# This is a ratchet: it may shrink, never grow. A new unresolvable union member+# fails the gate.+KNOWN_UNION_ONLY_TYPES = {+ "custom_webhook",+ "google_chronicle",+ "http_pull",+ "ibm_qradar",+ "kafka",+ "palo_alto_cortex",+ "slack",+ "syslog",+ "teams",+ "vectra_ai",+}++TS_TYPES_REL = Path("packages/types/src/connector.ts")++# Identity destinations the generic path depends on. Losing any one of these+# is the defect this gate exists for.+REQUIRED_IDENTITY_DESTINATIONS = ("actor.user.name", "device.name", "src_endpoint.ip")++# Behavioural proof. Coverage arithmetic alone would still pass if someone+# deleted the alias pass, so the gate requires the tests that exercise it.+REQUIRED_GO_TESTS = (+ "TestGenericProfileResolvesIdentityFields",+ "TestDeclaredConnectorIdResolvesToVendorProfile",+ "TestIdentityAliasResolutionIsDeterministic",+)++_GO_BLOCK = "var {name} = map[string]{vtype}{{"+++class GateError(RuntimeError):+ """An input could not be read. Never downgraded to a passing result."""+++# --------------------------------------------------------------------------+# Go side+# --------------------------------------------------------------------------+def _go_block(src: str, header: str, open_after: str | None = None) -> str:+ """The brace-balanced literal following `header`.++ `open_after` exists for a slice of anonymous structs, where the first brace+ after the header opens the type definition rather than the literal: the+ value starts at the `}{` that closes the struct and opens the composite.+ """+ start = src.find(header)+ if start == -1:+ raise GateError(f"could not find `{header}` in the normalizer")+ if open_after is not None:+ marker = src.find(open_after, start)+ if marker == -1:+ raise GateError(f"could not find `{open_after}` after `{header}`")+ start = marker+ depth, i = 0, src.index("{", start)+ for j in range(i, len(src)):+ if src[j] == "{":+ depth += 1+ elif src[j] == "}":+ depth -= 1+ if depth == 0:+ return src[i : j + 1]+ raise GateError(f"unbalanced braces after `{header}`")+++def parse_profile_keys(src: str) -> set[str]:+ block = _go_block(src, "var connectorProfiles = map[string]connectorProfile{")+ # Top-level keys only: nested fieldMap entries are indented deeper.+ return set(re.findall(r'^\t"([^"]+)":\s*\{', block, re.MULTILINE))+++def parse_type_aliases(src: str) -> dict[str, str]:+ # Absent is a valid state to report on, not a parse error: a tree with no+ # alias map is exactly the tree whose declared ids reach no profile, and+ # the gate should say so rather than abort before it gets there.+ if "var connectorTypeAliases = map[string]string{" not in src:+ return {}+ block = _go_block(src, "var connectorTypeAliases = map[string]string{")+ return dict(re.findall(r'"([^"]+)":\s*"([^"]+)"', block))+++def parse_identity_destinations(src: str) -> dict[str, list[str]]:+ block = _go_block(src, "var _canonicalAliases = []struct {", open_after="}{")+ out: dict[str, list[str]] = {}+ for dst, sources in re.findall(r'\{"([^"]+)",\s*\[\]string\{([^}]*)\}\}', block):+ out[dst] = re.findall(r'"([^"]+)"', sources)+ return out+++# --------------------------------------------------------------------------+# Python side+# --------------------------------------------------------------------------+def _class_connector_id(tree: ast.Module) -> str | None:+ for cls in (n for n in ast.walk(tree) if isinstance(n, ast.ClassDef)):+ for stmt in cls.body:+ if (+ isinstance(stmt, ast.Assign)+ and any(getattr(t, "id", None) == "connector_id" for t in stmt.targets)+ and isinstance(stmt.value, ast.Constant)+ and isinstance(stmt.value.value, str)+ and stmt.value.value+ ):+ return stmt.value.value+ if (+ isinstance(stmt, ast.AnnAssign)+ and getattr(stmt.target, "id", None) == "connector_id"+ and isinstance(stmt.value, ast.Constant)+ and isinstance(stmt.value.value, str)+ and stmt.value.value+ ):+ return stmt.value.value+ return None+++def _emits_canonical_envelope(tree: ast.Module) -> bool:+ """Whether the connector's normalized event carries `raw_event` + `source`.++ Every string key contributed by a dict literal, a subscript assignment or+ an envelope helper inside the normalize path counts. Restricting this to+ keys of a directly-returned dict literal reported 13 connectors as+ non-canonical when a runtime probe of all 84 found 2 — eleven build the+ envelope through a helper or a spread, and a gate that fails eleven+ correct connectors gets switched off.+ """++ def in_normalize_path(name: str) -> bool:+ return name == "normalize" or name.startswith(("_envelope", "_normalize", "_to_", "_build"))++ keys: set[str] = set()+ functions = (ast.FunctionDef, ast.AsyncFunctionDef)+ for fn in (n for n in ast.walk(tree) if isinstance(n, functions) and in_normalize_path(n.name)):+ for node in ast.walk(fn):+ if isinstance(node, ast.Dict):+ keys.update(k.value for k in node.keys if isinstance(k, ast.Constant) and isinstance(k.value, str))+ elif isinstance(node, ast.Subscript) and isinstance(node.slice, ast.Constant) and isinstance(node.slice.value, str):+ keys.add(node.slice.value)+ return {"raw_event", "source"} <= keys+++def parse_connectors(directory: Path) -> dict[str, bool]:+ """connector_id -> emits a canonical envelope."""+ out: dict[str, bool] = {}+ for path in sorted(directory.glob("*.py")):+ if path.name in {"__init__.py", "base.py"}:+ continue+ tree = ast.parse(path.read_text(encoding="utf-8"))+ connector_id = _class_connector_id(tree)+ if connector_id:+ out[connector_id] = _emits_canonical_envelope(tree)+ return out+++# --------------------------------------------------------------------------+# Docs side+# --------------------------------------------------------------------------+DOC_GLOBS = ("README.md", "docs/**/*.md", "apps/docs/docs/**/*.md", "apps/docs/docs/**/*.mdx")+DOC_PATTERN = re.compile(r"""["']?connector_type["']?\s*[:=]\s*["']([a-z0-9_]+)["']""")+++def parse_doc_examples(root: Path) -> dict[str, list[str]]:+ """connector_type -> the documents that show it."""+ found: dict[str, list[str]] = {}+ for pattern in DOC_GLOBS:+ for path in sorted(root.glob(pattern)):+ if "node_modules" in path.parts or "plans" in path.parts:+ continue+ for name in DOC_PATTERN.findall(path.read_text(encoding="utf-8", errors="replace")):+ found.setdefault(name, []).append(str(path.relative_to(root)))+ return found+++# --------------------------------------------------------------------------+# Gate+# --------------------------------------------------------------------------+def parse_union_types(path: Path) -> set[str]:+ """Members of packages/types' ConnectorType union."""+ src = path.read_text(encoding="utf-8")+ match = re.search(r"export type ConnectorType\s*=(.*?);", src, re.DOTALL)+ if not match:+ raise GateError(f"could not find the ConnectorType union in {path}")+ return set(re.findall(r'"([^"]+)"', match.group(1)))+++def evaluate(+ profile_keys: set[str],+ type_aliases: dict[str, str],+ identity: dict[str, list[str]],+ connectors: dict[str, bool],+ doc_examples: dict[str, list[str]],+ go_tests: set[str],+ template_names: set[str],+ union_types: set[str],+) -> list[tuple[str, str]]:+ failures: list[tuple[str, str]] = []+ declared = set(connectors)+ # A name is resolvable if it is a declared connector, a profile key, or an+ # alias that leads to one.+ resolvable = declared | profile_keys | set(type_aliases)++ # GO -> PY. Every profile key must name something real.+ for key in sorted(profile_keys):+ if key in declared:+ continue+ if key in PUSH_ONLY_PROFILES:+ template = PUSH_ONLY_PROFILES[key]+ if template not in template_names:+ failures.append(+ ("profile-template-missing", f"profile {key!r} is allowed as a push-only template key, but {template} does not exist")+ )+ continue+ if key in LEGACY_PROFILE_KEYS:+ if key not in set(type_aliases.values()) and key not in union_types:+ failures.append(+ (+ "legacy-profile-unreachable",+ f"profile {key!r} is a legacy name: no connector declares it, no alias points at it, "+ "and the ConnectorType union no longer carries it — nothing can reach this profile",+ )+ )+ continue+ failures.append(("profile-names-nothing", f"profile {key!r} names a connector type that does not exist"))++ # GO -> PY. Aliases must start somewhere real and land somewhere real.+ for source, target in sorted(type_aliases.items()):+ if source not in declared:+ failures.append(("alias-source-undeclared", f"alias {source!r} -> {target!r}: no connector declares {source!r}"))+ if target not in profile_keys:+ failures.append(("alias-target-missing", f"alias {source!r} -> {target!r}: {target!r} is not a profile key"))+ if source in profile_keys:+ failures.append(("alias-shadowed", f"alias {source!r} is shadowed by its own profile entry and never fires"))++ # PY -> GO. Every declared connector must survive strict mode.+ for connector_id, canonical in sorted(connectors.items()):+ if connector_id in profile_keys or connector_id in type_aliases or canonical:+ continue+ failures.append(+ (+ "strict-mode-unreachable",+ f"connector {connector_id!r} has no profile, no alias and emits no canonical envelope: "+ "strict mode rejects it and lenient mode resolves only a title",+ )+ )++ # TS -> *. The console's union is the third name space, and drift here is+ # how `splunk_enterprise` came to exist on one side only. The ratchet may+ # shrink; a new unresolvable member fails.+ for name in sorted(union_types - resolvable - KNOWN_UNION_ONLY_TYPES):+ failures.append(("union-type-unresolvable", f"ConnectorType union member {name!r} names no connector, profile or alias"))+ for name in sorted(KNOWN_UNION_ONLY_TYPES & resolvable):+ failures.append(+ (+ "union-ratchet-stale",+ f"{name!r} now resolves; remove it from KNOWN_UNION_ONLY_TYPES so the ratchet keeps shrinking",+ )+ )++ # DOCS. A copied example that resolves to nothing is the defect that taught+ # us to write this gate.+ for name, where in sorted(doc_examples.items()):+ locations = ", ".join(sorted(set(where)))+ if name not in resolvable:+ failures.append(("doc-example-unresolvable", f"documented connector_type {name!r} resolves to nothing (in {locations})"))+ continue+ # Being a declared connector id is not enough for a documented example.+ # A reader pastes a flat payload, which carries no `raw_event`, so the+ # canonical branch never runs and only a profile or an alias can give+ # the event a vendor identity. This is the exact shape of the original+ # defect: the README said `crowdstrike`, the profile was keyed+ # `crowdstrike_falcon`, and the example a new user copied fell through+ # to the generic fallback while still producing an alert.+ if name not in profile_keys and name not in type_aliases:+ failures.append(+ (+ "doc-example-no-profile",+ f"documented connector_type {name!r} has no profile and no alias, so the flat payload "+ f"shown in {locations} falls through to the generic fallback",+ )+ )++ # MECHANISM. The aliases the generic path leans on, and the tests that+ # prove it actually leans on them.+ for destination in REQUIRED_IDENTITY_DESTINATIONS:+ if not identity.get(destination):+ failures.append(+ ("identity-alias-missing", f"_canonicalAliases declares no sources for {destination!r}; the generic path cannot resolve it")+ )+ for test in REQUIRED_GO_TESTS:+ if test not in go_tests:+ failures.append(("behaviour-test-missing", f"{test} is gone; coverage arithmetic would still pass with the alias pass deleted"))+ return failures+++def load(root: Path) -> dict:+ normalizer = root / NORMALIZER_REL+ normalizer_test = root / NORMALIZER_TEST_REL+ connectors_dir = root / CONNECTORS_REL+ templates_dir = root / TEMPLATES_REL+ ts_types = root / TS_TYPES_REL+ for path in (normalizer, normalizer_test, connectors_dir, templates_dir, ts_types):+ if not path.exists():+ raise GateError(f"expected input does not exist: {path}")++ src = normalizer.read_text(encoding="utf-8")+ data = {+ "profile_keys": parse_profile_keys(src),+ "type_aliases": parse_type_aliases(src),+ "identity": parse_identity_destinations(src),+ "connectors": parse_connectors(connectors_dir),+ "doc_examples": parse_doc_examples(root),+ "go_tests": set(re.findall(r"^func (Test\w+)\(", normalizer_test.read_text(encoding="utf-8"), re.MULTILINE)),+ "template_names": {p.name for p in templates_dir.glob("*.yaml")},+ "union_types": parse_union_types(ts_types),+ }+ # An empty parse means the format moved, not that the tree is clean.+ for name in ("profile_keys", "connectors", "identity", "go_tests", "template_names", "union_types"):+ if not data[name]:+ raise GateError(f"parsed zero {name} — refusing to report a clean tree from an empty read")+ return data+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ parser.add_argument("--repo-root", type=Path, default=Path(__file__).resolve().parent.parent)+ parser.add_argument("--json", action="store_true")+ parser.add_argument("--self-test", action="store_true", help="prove the gate detects injected drift in each direction")+ args = parser.parse_args(argv)++ if args.self_test:+ return self_test(args.repo_root.resolve())++ root = args.repo_root.resolve()+ try:+ data = load(root)+ except GateError as exc:+ print(f"check_connector_profiles: FAILED to read the tree: {exc}", file=sys.stderr)+ return 2++ failures = evaluate(**data)+ declared = data["connectors"]+ with_profile = sorted(c for c in declared if c in data["profile_keys"] or c in data["type_aliases"])+ canonical_only = sorted(c for c in declared if c not in with_profile and declared[c])+ generic_only = sorted(c for c in declared if c not in with_profile and not declared[c])++ if args.json:+ print(+ json.dumps(+ {+ "repo_root": str(root),+ "declared_connectors": len(declared),+ "profile_keys": sorted(data["profile_keys"]),+ "type_aliases": data["type_aliases"],+ "with_profile": with_profile,+ "canonical_envelope_only": canonical_only,+ "generic_fallback_only": generic_only,+ "doc_examples": data["doc_examples"],+ "failures": [{"code": c, "detail": d} for c, d in failures],+ },+ indent=2,+ sort_keys=True,+ )+ )+ return 1 if failures else 0++ # Name the tree and the inputs before the verdict.+ print(f"repo root {root}")+ print(f"normalizer {NORMALIZER_REL} ({len(data['profile_keys'])} profiles, {len(data['type_aliases'])} type aliases)")+ print(f"connectors {CONNECTORS_REL} ({len(declared)} declared connector ids)")+ print(f"templates {TEMPLATES_REL} ({len(data['template_names'])} webhook templates)")+ print(f"console types {TS_TYPES_REL} ({len(data['union_types'])} ConnectorType union members)")+ occurrences = sum(len(v) for v in data["doc_examples"].values())+ print(f"doc examples {len(data['doc_examples'])} distinct connector_type values across {occurrences} occurrences")+ print()+ print(+ f"resolution path {len(with_profile)} via profile/alias, "+ f"{len(canonical_only)} via canonical envelope, {len(generic_only)} generic-only"+ )+ if generic_only:+ print(f" generic-only: {', '.join(generic_only)}")+ print()+ if failures:+ print(f"FAIL — {len(failures)} connector-type name(s) do not resolve:")+ for code, detail in failures:+ print(f" [{code}] {detail}")+ return 1+ print("OK — every profile key names a real connector type, every declared connector")+ print(" reaches a normalization path, and every documented example resolves.")+ return 0+++# --------------------------------------------------------------------------+# Self-test+# --------------------------------------------------------------------------+def self_test(root: Path) -> int:+ """Inject drift in each direction and require the gate to catch each one.++ A gate is only worth its runtime if it fails when it should. Each case+ mutates the real parsed inputs and asserts the specific code fires, so a+ refactor that quietly stops checking one direction fails here rather than+ printing OK forever.+ """+ try:+ base = load(root)+ except GateError as exc:+ print(f"self-test: cannot read the tree: {exc}", file=sys.stderr)+ return 2++ if evaluate(**base):+ print("self-test: the unmodified tree already fails; fix that first", file=sys.stderr)+ for code, detail in evaluate(**base):+ print(f" [{code}] {detail}", file=sys.stderr)+ return 1++ def mutate(**overrides) -> dict:+ data = {k: (v.copy() if hasattr(v, "copy") else v) for k, v in base.items()}+ for key, fn in overrides.items():+ fn(data[key])+ return data++ cases: list[tuple[str, str, dict]] = [+ (+ "GO -> PY: a profile key naming a connector that does not exist",+ "profile-names-nothing",+ mutate(profile_keys=lambda s: s.add("acme_xdr_9000")),+ ),+ (+ "PY -> GO: a declared connector with no profile and no canonical envelope",+ "strict-mode-unreachable",+ mutate(connectors=lambda d: d.update({"brand_new_connector": False})),+ ),+ (+ "DOCS: a documented example whose connector_type resolves to nothing",+ "doc-example-unresolvable",+ mutate(doc_examples=lambda d: d.update({"crowdstrik": ["README.md"]})),+ ),+ (+ "DOCS: a documented example naming a real connector with no profile behind it",+ "doc-example-no-profile",+ mutate(doc_examples=lambda d: d.update({"confluence_audit": ["README.md"]})),+ ),+ (+ "GO -> PY: an alias pointing at a profile that does not exist",+ "alias-target-missing",+ mutate(type_aliases=lambda d: d.update({"okta": "okta_nonexistent"})),+ ),+ (+ "GO -> PY: an alias starting from a connector nobody declares",+ "alias-source-undeclared",+ mutate(type_aliases=lambda d: d.update({"not_a_connector": "splunk"})),+ ),+ (+ "MECHANISM: the identity aliases the generic path depends on are removed",+ "identity-alias-missing",+ mutate(identity=lambda d: d.pop("actor.user.name", None)),+ ),+ (+ "MECHANISM: the Go test proving the behaviour is deleted",+ "behaviour-test-missing",+ mutate(go_tests=lambda s: s.discard("TestGenericProfileResolvesIdentityFields")),+ ),+ (+ "GO -> PY: a push-only profile key whose template is gone",+ "profile-template-missing",+ mutate(template_names=lambda s: s.discard("ai-runtime.yaml")),+ ),+ (+ "TS -> *: a union member naming no connector, profile or alias",+ "union-type-unresolvable",+ mutate(union_types=lambda s: s.add("acme_console_only")),+ ),+ (+ "TS -> *: the union ratchet left stale after a name starts resolving",+ "union-ratchet-stale",+ mutate(connectors=lambda d: d.update({"vectra_ai": True})),+ ),+ (+ "GO -> PY: the last route to a legacy profile key disappears",+ "legacy-profile-unreachable",+ mutate(union_types=lambda s: s.discard("splunk_enterprise")),+ ),+ ]++ print(f"self-test against {root}")+ print("clean tree: 0 failures (the baseline every case below perturbs)\n")+ ok = True+ for description, expected_code, data in cases:+ codes = {code for code, _ in evaluate(**data)}+ caught = expected_code in codes+ ok &= caught+ print(f" {'PASS' if caught else 'FAIL'} {description}")+ print(f" expected [{expected_code}] got {sorted(codes) or 'nothing'}")+ print()+ if not ok:+ print("self-test FAILED: the gate did not catch drift it claims to catch")+ return 1+ print(f"self-test OK: {len(cases)} injected defects, each caught by its own code")+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/services/connectors/app/connectors/auditd.py b/services/connectors/app/connectors/auditd.pyindex 74dc0948d..5cf1ca9ac 100644--- a/services/connectors/app/connectors/auditd.py+++ b/services/connectors/app/connectors/auditd.py@@ -678,6 +678,11 @@ class AuditdConnector(BaseConnector):return {"source": self.connector_id,+ # The ingest normalizer treats `raw_event` + `source` as the+ # canonical connector envelope. Without it this event missed that+ # check and fell to the generic fallback, which took the host below+ # with it.+ "raw_event": raw,"category": "endpoint","external_id": raw.get("event_id"),"title": title,diff --git a/services/ingest/internal/normalizer/normalizer.go b/services/ingest/internal/normalizer/normalizer.goindex b8b4ab2d6..62c2f664f 100644--- a/services/ingest/internal/normalizer/normalizer.go+++ b/services/ingest/internal/normalizer/normalizer.go@@ -254,6 +254,33 @@ var connectorProfiles = map[string]connectorProfile{// as a lowercase string; the map below mirrors KubernetesAuditConnector// so on-prem file_tail and webhook events end up with the same// severity_id.+ // email_inbox — the pull side of the forwarded-email path.+ //+ // EmailInboxConnector.normalize() deliberately returns the message+ // envelope unchanged, because it is shaped for the email-forwarded.yaml+ // template rather than for the canonical connector envelope. That left it+ // as the one registered connector whose events reach neither a profile nor+ // the canonical branch, so strict mode rejected them outright and lenient+ // mode resolved a title and nothing else.+ //+ // The field map mirrors email-forwarded.yaml so the pull and push paths+ // produce the same OCSF shape, the same way ai_runtime mirrors+ // ai-runtime.yaml. Keep the two aligned.+ "email_inbox": {+ product: OcsfProduct{Name: "Forwarded Email", VendorName: "Email"},+ classUID: 2001,+ className: "Security Finding",+ fieldMap: map[string]string{+ "subject": "finding.title",+ "body": "finding.desc",+ "message_id": "finding.uid",+ "from": "actor.user.email_addr",+ "to": "target_user.email_addr",+ },+ severityMap: map[string]int{+ "critical": 5, "high": 4, "medium": 3, "normal": 3, "low": 2, "info": 1, "informational": 1,+ },+ },"kubernetes_audit": {product: OcsfProduct{Name: "Kubernetes Audit", VendorName: "Kubernetes"},classUID: 6003,@@ -353,13 +380,53 @@ var _canonicalFieldMap = map[string]string{// so a missing actor does not merely blank a column — it collapses the// entity segment to "unknown" and every one of that connector's alerts// correlates into the same bucket.+//+// The dotted sources dig one level for a scalar. Replaying the registry's 84+// connectors through normalize() found 14 emitting `actor`, and four of those+// emit it as the vendor's nested object rather than a name. A bare key alone+// would hand that object to setNestedField and write a map into a slot the+// entity extractor reads as a name, so each bare key is followed by the+// scalar-bearing paths underneath it.var _canonicalAliases = []struct {dst stringsources []string}{- {"actor.user.name", []string{"actor", "username", "user", "user_name"}},- {"device.name", []string{"hostname", "host", "device_name"}},- {"src_endpoint.ip", []string{"src_ip", "source_ip", "client_ip"}},+ {"actor.user.name", []string{"actor", "actor.name", "actor.displayName", "actor.username", "username", "user", "user.name", "user.username", "user_name"}},+ {"device.name", []string{"hostname", "host", "host.name", "device_name", "device", "device.name", "device.hostname"}},+ {"src_endpoint.ip", []string{"src_ip", "source_ip", "client_ip", "src_endpoint.ip", "client.ipAddress"}},+ // Not identity, but the same fill-the-blank rule and the same reason to+ // run everywhere. A vendor profile names the field its vendor's rows use+ // — crowdstrike_falcon reads event_simpleName, not title — so a pushed+ // payload that reaches a vendor profile left `message` empty and the+ // promoter fell back to generating "Security Finding from <product>",+ // discarding the title the caller actually sent. These two are the same+ // pair _canonicalFieldMap declares; listing them here extends their reach+ // to the vendor profiles without giving them a second mechanism.+ {"message", []string{"title"}},+ {"finding.uid", []string{"external_id"}},+}++// connectorTypeAliases maps a connector identifier the connectors service+// actually declares onto the profile key that carries its vendor field map.+//+// Two vocabularies grew up either side of the wire. services/connectors+// declares `crowdstrike` and `okta`; this file keyed their profiles+// `crowdstrike_falcon` and `okta_system_log`, and so does the rest of the+// platform — packages/types' ConnectorType union, the CLI's default, the+// graph extractor and the actions credential resolver all use the longer+// names. Renaming either side breaks the other, so both resolve instead.+//+// Without this an event whose connector_type is the name the product itself+// advertises — including the one the README tells a new user to paste — missed+// the profile lookup and fell to the generic fallback, taking generic vendor+// attribution with it.+//+// `splunk` is deliberately absent: it has its own profile for the canonical+// envelope SplunkConnector emits, which is a different shape from the raw+// rows `splunk_enterprise` maps.+var connectorTypeAliases = map[string]string{+ "crowdstrike": "crowdstrike_falcon",+ "okta": "okta_system_log",}// canonicalClassByConnector overrides the default Security Finding class for@@ -453,6 +520,11 @@ func (n *Normalizer) Normalize(raw *RawEvent) (*NormalizedEvent, error) {} else {var ok boolprofile, ok = connectorProfiles[raw.ConnectorType]+ if !ok {+ if aliased, isAlias := connectorTypeAliases[raw.ConnectorType]; isAlias {+ profile, ok = connectorProfiles[aliased]+ }+ }if !ok {if n.cfg.NormalizerMode == "strict" {return nil, fmt.Errorf("unknown connector type: %s", raw.ConnectorType)@@ -495,27 +567,45 @@ func (n *Normalizer) Normalize(raw *RawEvent) (*NormalizedEvent, error) {}}- // Canonical envelopes additionally resolve identity aliases in declared- // order, so a connector that spells the actor `username` is not silently- // anonymised. Only applies where the profile uses the canonical map;- // hand-written vendor profiles already name their own fields.- if isCanonical {- for _, alias := range _canonicalAliases {- if getNestedField(ocsf, alias.dst) != nil {- continue- }- for _, src := range alias.sources {- if val := getNestedField(raw.Payload, src); val != nil {- setNestedField(ocsf, alias.dst, val)- break- }- }+ // Resolve identity aliases in declared order for every profile, filling+ // only destinations the field map left empty.+ //+ // This used to run for canonical envelopes alone, which left the generic+ // fallback resolving nothing but title and external_id. Everything+ // downstream is keyed on identity — entity extraction, the Investigation+ // Rail's pivots, the {tenant}:{entity}:{tactic} correlation key, the+ // entity graph, UEBA — so an event that arrives through the fallback+ // became an alert with no host, no user and no IP, and nobody could pivot+ // from it. The alert appears, which is what makes it look like it worked.+ //+ // Filling only empty destinations is what keeps this safe to run over the+ // hand-written vendor profiles too: a profile that names its own field for+ // a slot always wins, and the aliases reach only the slots it left blank.+ for _, alias := range _canonicalAliases {+ if getNestedField(ocsf, alias.dst) != nil {+ continue+ }+ if val, found := firstIdentityString(raw.Payload, alias.sources); found {+ setNestedField(ocsf, alias.dst, val)}}- // Map severity+ // Map severity. A vendor profile's own ladder wins; the shared five-tier+ // ladder is consulted only when that ladder has no entry for the value.+ //+ // The fallback matters because a profile's ladder is spelled the way its+ // vendor spells it — crowdstrike_falcon's is capitalised — while a pushed+ // payload is written by whoever is pushing, and the README's is lowercase.+ // Without the fallback a "high" that the ladder spells "High" scores 0 and+ // renders as Unknown. Case folding alone would not do: the shared ladder+ // keeps `critical` a distinct fifth tier, so a vendor-native critical maps+ // to critical rather than collapsing into high.if sevField, ok := raw.Payload["severity"].(string); ok {- if sevID, found := profile.severityMap[sevField]; found {+ sevID, found := profile.severityMap[sevField]+ if !found {+ sevID, found = _canonicalSeverityMap[strings.ToLower(strings.TrimSpace(sevField))]+ }+ if found {ocsf["severity_id"] = sevIDocsf["severity"] = sevField} else {@@ -711,6 +801,27 @@ func setNestedField(m map[string]interface{}, path string, val interface{}) {setNestedField(nested, parts[1], val)}+// firstIdentityString returns the first non-empty string among the given dotted+// payload paths, in the order given, and whether one was found.+//+// The string requirement is the point. An identity destination — actor.user.name,+// device.name, src_endpoint.ip — is a scalar the entity extractor turns into a+// pivotable chip and the correlator folds into {tenant}:{entity}:{tactic}.+// Several connectors pass the vendor's nested actor object through under the+// same key a scalar would use, and writing that object into the slot produces+// an entity that renders as a map and correlates as garbage. Skipping it lets+// the next path in the list — the scalar one underneath — resolve instead.+func firstIdentityString(payload map[string]interface{}, paths []string) (string, bool) {+ for _, p := range paths {+ if s, ok := getNestedField(payload, p).(string); ok {+ if trimmed := strings.TrimSpace(s); trimmed != "" {+ return trimmed, true+ }+ }+ }+ return "", false+}+// firstString returns the first non-empty string value among the given payload// keys, or "" if none are present.func firstString(payload map[string]interface{}, keys ...string) string {diff --git a/services/ingest/internal/normalizer/normalizer_test.go b/services/ingest/internal/normalizer/normalizer_test.goindex a8e4be18c..7cc534b8c 100644--- a/services/ingest/internal/normalizer/normalizer_test.go+++ b/services/ingest/internal/normalizer/normalizer_test.go@@ -402,8 +402,10 @@ func TestCanonicalEnvelopeResolvesHostAndIPAliases(t *testing.T) {}}-// Hand-written vendor profiles name their own fields and must not have the-// canonical aliases applied on top of them.+// A hand-written vendor profile's own field mapping always wins. The aliases+// run over every profile now, so this pins the precedence rather than the+// absence: they may fill a destination the vendor map left empty, they may+// never overwrite one it filled.func TestAliasesDoNotApplyToRawVendorProfiles(t *testing.T) {n := newTestNormalizer()ev, err := n.Normalize(&RawEvent{@@ -425,3 +427,231 @@ func TestAliasesDoNotApplyToRawVendorProfiles(t *testing.T) {t.Errorf("actor.user.name = %v, want the vendor profile's own mapping", got)}}++// The generic fallback must resolve identity, not just a title.+//+// This is the README's own push example: a flat payload, a connector type with+// no profile of its own. Before, the fallback's field map held `title` and+// `external_id` and nothing else, and the alias pass was gated behind the+// canonical-envelope branch — so the event became an alert with no host, no+// user and no IP. Entity extraction, the Investigation Rail's pivots, the+// {tenant}:{entity}:{tactic} correlation key, the entity graph and UEBA all+// read those three fields, so the alert arrived with nothing to pivot from.+// The alert appearing at all is what made it look like it had worked.+func TestGenericProfileResolvesIdentityFields(t *testing.T) {+ n := newLenientNormalizer()+ ev, err := n.Normalize(&RawEvent{+ ConnectorID: "edr-1",+ ConnectorType: "acme_xdr", // deliberately absent from connectorProfiles+ TenantID: "11111111-1111-1111-1111-111111111111",+ ReceivedAt: "2026-09-23T00:00:00Z",+ Payload: map[string]interface{}{+ "severity": "high",+ "title": "Encoded PowerShell from Office",+ "host": "WIN-FIN-01",+ "user": "alice",+ "src_ip": "10.20.30.40",+ },+ })+ if err != nil {+ t.Fatalf("Normalize returned error: %v", err)+ }+ for _, tc := range []struct{ path, want string }{+ {"device.name", "WIN-FIN-01"},+ {"actor.user.name", "alice"},+ {"src_endpoint.ip", "10.20.30.40"},+ } {+ if got := nested(t, ev.OcsfEvent, tc.path); got != tc.want {+ t.Errorf("%s = %v, want %q — the generic fallback must resolve identity", tc.path, got, tc.want)+ }+ }+ // Still promotable and still vendor-neutral, which the fallback already got right.+ if ev.OcsfEvent["class_uid"] != 2001 {+ t.Errorf("class_uid = %v, want 2001", ev.OcsfEvent["class_uid"])+ }+ if ev.OcsfEvent["severity_id"] != 4 {+ t.Errorf("severity_id = %v, want 4", ev.OcsfEvent["severity_id"])+ }+}++// The connector identifier the product advertises must reach the profile that+// carries its vendor field map.+//+// services/connectors declares `crowdstrike`; this file keyed the profile+// `crowdstrike_falcon`. The README tells a new user to push the former, so the+// example fell through to the generic fallback and was attributed to nothing+// in particular. Both names resolve now, because the longer one is load-bearing+// elsewhere — the ConnectorType union, the CLI default, the graph extractor.+func TestDeclaredConnectorIdResolvesToVendorProfile(t *testing.T) {+ for _, tc := range []struct {+ connectorType string+ wantVendor string+ wantClass int+ }{+ {"crowdstrike", "CrowdStrike", 2001},+ {"crowdstrike_falcon", "CrowdStrike", 2001},+ {"okta", "Okta", 3002},+ {"okta_system_log", "Okta", 3002},+ } {+ t.Run(tc.connectorType, func(t *testing.T) {+ n := newLenientNormalizer()+ ev, err := n.Normalize(&RawEvent{+ ConnectorID: "c-1",+ ConnectorType: tc.connectorType,+ TenantID: "11111111-1111-1111-1111-111111111111",+ ReceivedAt: "2026-09-23T00:00:00Z",+ // Flat, human-authored, lowercase severity — a pushed payload,+ // not a vendor row.+ Payload: map[string]interface{}{+ "severity": "high",+ "title": "Encoded PowerShell from Office",+ "host": "WIN-FIN-01",+ "user": "alice",+ "src_ip": "10.20.30.40",+ },+ })+ if err != nil {+ t.Fatalf("Normalize returned error: %v", err)+ }+ meta := ev.OcsfEvent["metadata"].(map[string]interface{})+ product := meta["product"].(OcsfProduct)+ if product.VendorName != tc.wantVendor {+ t.Errorf("VendorName = %q, want %q — the declared id must reach the vendor profile", product.VendorName, tc.wantVendor)+ }+ if ev.OcsfEvent["class_uid"] != tc.wantClass {+ t.Errorf("class_uid = %v, want %d", ev.OcsfEvent["class_uid"], tc.wantClass)+ }+ // Reaching a vendor profile must not cost the identity fields the+ // generic fallback would have resolved.+ if got := nested(t, ev.OcsfEvent, "device.name"); got != "WIN-FIN-01" {+ t.Errorf("device.name = %v, want WIN-FIN-01", got)+ }+ if got := nested(t, ev.OcsfEvent, "actor.user.name"); got != "alice" {+ t.Errorf("actor.user.name = %v, want alice", got)+ }+ if got := nested(t, ev.OcsfEvent, "src_endpoint.ip"); got != "10.20.30.40" {+ t.Errorf("src_endpoint.ip = %v, want 10.20.30.40", got)+ }+ // ...nor the severity. crowdstrike_falcon's ladder is capitalised+ // and a pushed payload is lowercase; without the shared-ladder+ // fallback this scores 0 and renders as Unknown.+ if ev.OcsfEvent["severity_id"] != 4 {+ t.Errorf("severity_id = %v, want 4 for %q", ev.OcsfEvent["severity_id"], "high")+ }+ // ...nor the caller's own title. A vendor profile maps its+ // vendor's field name, so `message` was left empty and the+ // promoter generated "Security Finding from <product>" over the+ // top of the title the caller sent.+ if got := nested(t, ev.OcsfEvent, "message"); got != "Encoded PowerShell from Office" {+ t.Errorf("message = %v, want the caller's own title", got)+ }+ })+ }+}++// critical is the fifth tier, not a louder high. A vendor-native critical that+// collapses into high silently downgrades the most urgent thing in the queue.+func TestCriticalSeverityDoesNotCollapseIntoHigh(t *testing.T) {+ for _, connectorType := range []string{"acme_xdr", "crowdstrike"} {+ t.Run(connectorType, func(t *testing.T) {+ n := newLenientNormalizer()+ ev, err := n.Normalize(&RawEvent{+ ConnectorID: "c-1",+ ConnectorType: connectorType,+ TenantID: "11111111-1111-1111-1111-111111111111",+ ReceivedAt: "2026-09-23T00:00:00Z",+ Payload: map[string]interface{}{"title": "t", "severity": "critical"},+ })+ if err != nil {+ t.Fatalf("Normalize returned error: %v", err)+ }+ if ev.OcsfEvent["severity_id"] != 5 {+ t.Errorf("severity_id = %v, want 5 (critical is its own tier)", ev.OcsfEvent["severity_id"])+ }+ })+ }+}++// An identity destination is a scalar the entity extractor turns into a chip+// and the correlator folds into its key. Several connectors pass the vendor's+// nested actor object through under the same key a scalar would use; writing+// that object into the slot yields an entity that renders as a map.+func TestIdentityAliasesSkipNonScalarsAndDigForTheName(t *testing.T) {+ n := newLenientNormalizer()+ ev, err := n.Normalize(&RawEvent{+ ConnectorID: "c-1",+ ConnectorType: "acme_xdr",+ TenantID: "11111111-1111-1111-1111-111111111111",+ ReceivedAt: "2026-09-23T00:00:00Z",+ Payload: map[string]interface{}{+ "title": "t",+ // The vendor's object under the key a scalar would use.+ "actor": map[string]interface{}{"id": "u-1", "name": "alice"},+ "device": map[string]interface{}{"name": "WIN-01"},+ },+ })+ if err != nil {+ t.Fatalf("Normalize returned error: %v", err)+ }+ got := nested(t, ev.OcsfEvent, "actor.user.name")+ if _, isStr := got.(string); !isStr {+ t.Fatalf("actor.user.name = %#v (%T), want a string — a map here is an unpivotable entity", got, got)+ }+ if got != "alice" {+ t.Errorf("actor.user.name = %v, want alice dug out of the nested actor", got)+ }+ if got := nested(t, ev.OcsfEvent, "device.name"); got != "WIN-01" {+ t.Errorf("device.name = %v, want WIN-01", got)+ }+}++// Go randomises map iteration, so alias precedence has to come from a slice.+// Resolving the same payload repeatedly must yield the same answer every time;+// a map-ordered implementation passes this roughly one run in two.+func TestIdentityAliasResolutionIsDeterministic(t *testing.T) {+ payload := func() map[string]interface{} {+ return map[string]interface{}{+ "title": "t",+ // Every alias source for actor.user.name carries a distinct value,+ // so any ordering instability shows up as a different winner.+ "actor": "by-actor", "username": "by-username",+ "user": "by-user", "user_name": "by-user-name",+ "hostname": "by-hostname", "host": "by-host", "device_name": "by-device-name",+ "src_ip": "1.1.1.1", "source_ip": "2.2.2.2", "client_ip": "3.3.3.3",+ }+ }+ const runs = 200+ for i := 0; i < runs; i++ {+ n := newLenientNormalizer()+ ev, err := n.Normalize(&RawEvent{+ ConnectorID: "c-1", ConnectorType: "acme_xdr",+ TenantID: "11111111-1111-1111-1111-111111111111",+ ReceivedAt: "2026-09-23T00:00:00Z", Payload: payload(),+ })+ if err != nil {+ t.Fatalf("Normalize returned error: %v", err)+ }+ if got := nested(t, ev.OcsfEvent, "actor.user.name"); got != "by-actor" {+ t.Fatalf("run %d: actor.user.name = %v, want by-actor (first declared source)", i, got)+ }+ if got := nested(t, ev.OcsfEvent, "device.name"); got != "by-hostname" {+ t.Fatalf("run %d: device.name = %v, want by-hostname", i, got)+ }+ if got := nested(t, ev.OcsfEvent, "src_endpoint.ip"); got != "1.1.1.1" {+ t.Fatalf("run %d: src_endpoint.ip = %v, want 1.1.1.1", i, got)+ }+ }+}++// Every connectorTypeAliases entry must point at a profile that exists, and+// must not shadow a connector type that already has its own profile.+func TestConnectorTypeAliasesPointAtRealProfiles(t *testing.T) {+ for alias, target := range connectorTypeAliases {+ if _, ok := connectorProfiles[target]; !ok {+ t.Errorf("alias %q -> %q, but %q is not in connectorProfiles", alias, target, target)+ }+ if _, ok := connectorProfiles[alias]; ok {+ t.Errorf("alias %q shadows its own profile entry; the alias is dead code", alias)+ }+ }+}