Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
docs: correct synthetic telemetry source list to match actual corpus
Beenu Arora4ad7e6e93c5e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c9d38ef6d..e3d51d453 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -140,16 +140,18 @@ the v5.2 eval harness:- **Synthetic telemetry corpus**(`services/agents/tests/eval_data/synthetic_telemetry.jsonl`,`scripts/generate_eval_incidents.py`) — 361 backing events spanning 14- log sources (Sysmon, Windows Security, M365 audit, Azure sign-in, Okta,- CloudTrail, Linux auditd, journald, EDR, firewall, DNS, VPN, DB audit,- IdP), wired to all 200 incidents. Each event is a templated dictionary- with `{user}/{host}/{ip}/{campaign}` placeholders resolved against the- incident it backs, and carries the fields a real connector pivots on- (process tree, principal, source IP, log source, event ID).+ log sources (Sysmon, Windows Security, M365 audit, Azure sign-in,+ CloudTrail, Linux auditd, journald, EDR, DNS, web access, Kubernetes+ audit, GitHub audit, VPN, DB audit), wired to all 200 incidents. Each+ event is a templated dictionary with `{user}/{host}/{ip}/{campaign}`+ placeholders resolved against the incident it backs, and carries the+ fields a real connector pivots on (process tree, principal, source IP,+ log source, event ID).- **Telemetry event factories + recursive resolver**(`scripts/generate_eval_incidents.py`) — `_sysmon`, `_winsec`, `_m365`,- `_cloudtrail`, `_okta`, `_auditd`, `_journald`, etc. produce base event- shapes; a recursive resolver walks nested dicts and substitutes incident+ `_azure_signin`, `_cloudtrail`, `_auditd`, `_journald`, `_edr`, `_dns`,+ `_web`, `_k8s`, `_github`, `_vpn`, `_db` produce base event shapes; a+ recursive resolver walks nested dicts and substitutes incidentcontext. The 55 templates in `_TEMPLATES` each now carry a`template_id`, a `template_index`, and a tuple of telemetry events.- **Schema + coverage gate** (`services/agents/tests/test_synthetic_telemetry.py`)diff --git a/README.md b/README.mdindex ffc635509..d3077be8c 100644--- a/README.md+++ b/README.md@@ -26,7 +26,7 @@ AiSOC is a single self-hostable stack that ingests security events, correlates tThree properties distinguish it from closed-source AI SOC vendors:1. **Agent decisions are logged.** The Investigation Ledger stores the LLM prompt, the response, the evidence cited, and the downstream tool calls for every step of every run. Replays are available later.-2. **The substrate has a public eval harness in CI.** Five suites gate every PR targeting `main` / `develop`: a 200-incident synthetic dataset drawn from 55 distinct templates drives the MITRE-tactic, investigation-completeness, and response-quality gates (each reporting both a per-case mean and a per-template macro so a single broken template can't hide behind 199 working duplicates); a separately generated 1,000-alert noisy stream drives the alert-reduction gate; and a schema/coverage gate validates `synthetic_telemetry.jsonl` — the companion corpus of ~360 backing events across 14 log sources (Sysmon, Windows Security, M365 audit, Azure sign-in, Okta, CloudTrail, Linux auditd, journald, EDR, firewall, DNS, VPN, DB audit, IdP) that connector and Sigma PRs can wire against. Alert reduction is a real measurement against the fixed alert stream; the three rubric-based suites are substrate self-consistency gates over deterministic templates. The [benchmark page](apps/docs/docs/benchmark.md) explains exactly which is which.+2. **The substrate has a public eval harness in CI.** Five suites gate every PR targeting `main` / `develop`: a 200-incident synthetic dataset drawn from 55 distinct templates drives the MITRE-tactic, investigation-completeness, and response-quality gates (each reporting both a per-case mean and a per-template macro so a single broken template can't hide behind 199 working duplicates); a separately generated 1,000-alert noisy stream drives the alert-reduction gate; and a schema/coverage gate validates `synthetic_telemetry.jsonl` — the companion corpus of ~360 backing events across 14 log sources (Sysmon, Windows Security, M365 audit, Azure sign-in, CloudTrail, Linux auditd, journald, EDR, DNS, web access, Kubernetes audit, GitHub audit, VPN, DB audit) that connector and Sigma PRs can wire against. Alert reduction is a real measurement against the fixed alert stream; the three rubric-based suites are substrate self-consistency gates over deterministic templates. The [benchmark page](apps/docs/docs/benchmark.md) explains exactly which is which.3. **It runs entirely on your infrastructure.** No callbacks to a vendor cloud and no data exfiltration for "model improvement."The orchestrator is a ~600-line LangGraph in [`services/agents/`](services/agents/). It is small enough to read end-to-end, swap models in, and patch.diff --git a/apps/docs/docs/benchmark.md b/apps/docs/docs/benchmark.mdindex 58665af58..ca0b441da 100644--- a/apps/docs/docs/benchmark.md+++ b/apps/docs/docs/benchmark.md@@ -1,7 +1,7 @@---sidebar_position: 4title: Public Eval Harness-description: AiSOC's open, deterministic regression harness. 200 synthetic incidents drawn from 55 distinct templates with backing telemetry (Sysmon, M365, CloudTrail, Okta, Linux auditd, …). Per-case and per-template CI gates over the substrate. Honest about what it measures — and what it doesn't.+description: AiSOC's open, deterministic regression harness. 200 synthetic incidents drawn from 55 distinct templates with backing telemetry (Sysmon, M365, CloudTrail, Azure sign-in, Linux auditd, …). Per-case and per-template CI gates over the substrate. Honest about what it measures — and what it doesn't.---# AiSOC Public Eval Harness@@ -20,9 +20,9 @@ description: AiSOC's open, deterministic regression harness. 200 synthetic incid> **What's new (v1.4):**>> 1. Every synthetic incident now ships with a backing **synthetic telemetry-> corpus** — Sysmon / Windows-Security / M365 audit / CloudTrail / Okta /-> Azure sign-in / Linux auditd / journald / EDR / firewall / DNS / VPN / DB-> audit events — written to+> corpus** — Sysmon / Windows Security / M365 audit / Azure sign-in /+> CloudTrail / Linux auditd / journald / EDR / DNS / web access /+> Kubernetes audit / GitHub audit / VPN / DB audit events — written to> [`synthetic_telemetry.jsonl`](https://github.com/beenuar/AiSOC/blob/main/services/agents/tests/eval_data/synthetic_telemetry.jsonl).> Connector and Sigma PRs now have something concrete to wire against. See> [Synthetic telemetry corpus](#5-synthetic-telemetry-corpus) below.@@ -94,9 +94,9 @@ These numbers move with the codebase. The current snapshot lives at### Per-case vs. per-template metricsThe 200-case dataset is built by drawing each case from one of **55 distinct-templates** (Sysmon process-injection, M365 admin-impersonation, CloudTrail-GuardDuty findings, Okta MFA-bombing, Linux auditd reverse-shell, …) and-swapping the `{user}/{host}/{ip}/{campaign}` slot in each one. That gives the+templates** (Sysmon process hollowing, M365 OAuth-consent phish, CloudTrail+EC2 IMDS credential theft, Azure AD impossible travel, Linux SUID abuse, …)+and swapping the `{user}/{host}/{ip}/{campaign}` slot in each one. That gives thesubstrate a wider blast radius to regress against without inflating thegenerator. Two metrics are reported for every scoring suite:@@ -274,16 +274,16 @@ without having to make up their own. The corpus currently covers 14 sources:| `windows_security` | Windows Security log (logon, privilege use, account changes) | `Computer`, `TargetUserName`, `EventID` || `m365_audit` | Microsoft 365 unified audit log | `UserId`, `Operation`, `Workload`, `ClientIP` || `azure_signin` | Azure AD / Entra sign-in log | `userPrincipalName`, `appDisplayName`, `ipAddress` |-| `okta` | Okta system log | `actor.alternateId`, `eventType`, `outcome` |-| `cloudtrail` | AWS CloudTrail management events | `eventName`, `userIdentity`, `sourceIPAddress` |-| `linux_auditd` | Linux auditd records (execve, syscall) | `host`, `auid`, `exe`, `proctitle` |-| `linux_journald` | Linux journald / syslog | `_HOSTNAME`, `MESSAGE`, `SYSLOG_IDENTIFIER` |-| `edr` | Generic EDR detection (CrowdStrike / SentinelOne shape) | `device.hostname`, `process.command_line` |-| `firewall` | Perimeter firewall (deny / allow / threat) | `src_ip`, `dst_ip`, `action`, `rule` |-| `dns` | DNS resolver / sinkhole | `query`, `client_ip`, `response_code` |-| `vpn` | VPN concentrator (auth and tunnel events) | `username`, `client_ip`, `assigned_ip` |-| `db_audit` | Database audit trail (Postgres / Oracle / SQL Server shapes) | `db_user`, `client_host`, `statement` |-| `idp` | Generic IdP / federated auth | `actor`, `event`, `target` |+| `cloudtrail` | AWS CloudTrail management events | `eventName`, `userIdentity`, `awsRegion` |+| `linux_auditd` | Linux auditd records (execve, syscall) | `type`, `syscall`, `auid`, `exe` |+| `linux_journald` | Linux journald / syslog | `_SYSTEMD_UNIT`, `MESSAGE`, `_HOSTNAME` |+| `edr` | Generic EDR detection (CrowdStrike / SentinelOne shape) | `rule`, `severity`, `device.hostname` |+| `dns` | DNS resolver / sinkhole | `query_name`, `query_type`, `client_ip` |+| `web_access` | Web access / WAF / proxy log | `http_method`, `url`, `status_code` |+| `k8s_audit` | Kubernetes audit log (`audit.k8s.io/v1`) | `verb`, `objectRef.resource`, `user.username` |+| `github_audit` | GitHub audit log (org / repo / app events) | `action`, `actor`, `repo` |+| `vpn` | VPN concentrator (auth and tunnel events) | `action`, `user`, `client_ip` |+| `db_audit` | Database audit trail (Postgres / Oracle / SQL Server shapes) | `user`, `operation`, `query` |Each event has its `{user}/{host}/{ip}/{campaign}` placeholders resolvedagainst the parent incident, so an event for `INC-EVAL-044` carries the same