Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
docs(audit): Phase 0 reality audit (claim-to-gate matrix, no code) (#415)
Beenu Aroraa071cd521c09
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 67d0eddd6..b007e5197 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Added++- **World-class program Phase 0 — reality audit** (no product code). `docs/audit/REALITY_REPORT.md` classifies every headline `README.md` claim against the code (`production` / `functional-untested` / `template-fallback` / `demo-only` / `stub`) and ranks Overclaims, Load-bearing untested paths, and Circular gates. `docs/audit/CLAIM_TO_GATE_MATRIX.md` maps 27 claims to their CI gate or `NO GATE` (9 GATED / 11 PARTIAL / 7 NO GATE), each with a binding "Closes in" phase. `docs/audit/PROGRESS.md` tracks the 12-phase program. Tracking doc: `AISOC_CURSOR_PROMPT_V2.md`.+## [7.5.0] — 2026-06-29v8.0-milestone and trust-readiness release. Folds in the **AiSOC missingdiff --git a/ROADMAP.md b/ROADMAP.mdindex 3ea4f677f..a2e23d261 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -18,6 +18,24 @@This document captures the planned direction for AiSOC across major versions. All v4 deliverables and items deferred beyond v4 are listed here.+## World-Class Hardening Program (2026-07, in flight)++A proof-first, security-first program to make every README claim gate-backed, close four existential agent-security holes, and build the ingest-time-graph + multi-model-router moat. Executed one phase per PR with a mandatory CI gate each. Live status: [`docs/audit/PROGRESS.md`](docs/audit/PROGRESS.md). Baseline audit: [`docs/audit/REALITY_REPORT.md`](docs/audit/REALITY_REPORT.md) and [`docs/audit/CLAIM_TO_GATE_MATRIX.md`](docs/audit/CLAIM_TO_GATE_MATRIX.md).++- [x] Phase 0 — Reality audit (claim-to-gate matrix, ranked overclaims/untested-paths/circular-gates)+- [ ] Phase 1 — Four existential holes (prompt injection, memory poisoning, cross-store tenant isolation, data-exfiltration/redaction, cost DoS, vault)+- [ ] Phase 2 — Supply chain + truth (SAST/deps/containers/secrets/IaC gates, signed attested releases, hard-fail insecure prod defaults, README honesty)+- [ ] Phase 3 — Integration / E2E / chaos / DR (real-container spine test, backup-restore, chaos, upgrade)+- [ ] Phase 4 — Real evals + detection content truth table (third-party-labeled corpus, hallucination/calibration/abstention, model matrix)+- [ ] Phase 5 — Data spine correctness (schema registry, DLQ, idempotency, watermarking, backfill/replay, lineage)+- [ ] Phase 6 — Performance + cost (throughput harness, $/TB, storage-consolidation ADR)+- [ ] Phase 7 — Ingest-time graph + multi-model router (posture, effective permissions, bi-temporal, fusion-time ContextBundle, determinism contract)+- [ ] Phase 8 — LLMOps (prompt registry, model pinning, provider fallback, response caching, structured-output validation)+- [ ] Phase 9 — Autonomy safety (rollback, post-action verification, dry-run default, break-glass, approval SLA, scorecard)+- [ ] Phase 10 — Connector + content quality (conformance suite, published matrix, detection lifecycle)+- [ ] Phase 11 — API / SDK / release engineering (oasdiff, SDK contract drift, versioning, release discipline)+- [ ] Phase 12 — Observability + governance (SLOs, single trace, GOVERNANCE.md, DCO, committers, neutral home, trust docs)+## v4.0 — Shipped### AI multi-agent investigatordiff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.mdnew file mode 100644index 000000000..a2cfed64f--- /dev/null+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@ -0,0 +1,45 @@+# Claim-to-Gate Matrix++Every marketing/capability claim in `README.md` mapped to the CI job that proves it, or `NO GATE`.++Rule (enforced from Phase 2 onward by `scripts/check_claim_gate_matrix.py`, wired into `.github/workflows/security.yml`): **no row may say `NO GATE`.** A `NO GATE` row is either a claim to delete or a gate to build. Rows marked `PARTIAL` name the gap that a later phase closes.++Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL` (some aspect gated, gap named) · `NO GATE` (unproven).++| Claim | Source | Gate (workflow :: job) | Status | Closes in |+|---|---|---|---|---|+| Offline sandbox demo runs with no key/network in < 5 s | README L38 | `readme-gates.yml :: sandbox-offline` (ubuntu+macos matrix) | GATED | - |+| `pnpm aisoc:demo` boots the real stack | README L40, L45 | `compose-smoke.yml`, `compose-smoke-nightly.yml` (cold) | PARTIAL (health-probe only, no time-to-first-investigation assertion) | Phase 3.4 |+| 69 connectors | README L116, L168 | `ci.yml :: python-lint` (`generate_connector_count.py --check`) | GATED (count) | - |+| Connectors: schema-driven config + vault-encrypted secrets | README L168 | `ci.yml :: python-services-test` (`test_schemas.py`); vault tests | PARTIAL (schema + vault gated; live API conformance, rate-limit, checkpoint durability ungated) | Phase 10 |+| Connectors: live Test connection | README L168 | none | NO GATE | Phase 10 |+| Investigation Ledger stores every step | README L61, L169 | `ci.yml :: api tests` (`audit_hash`, audit immutability) | PARTIAL (write path gated; UI replay only in hermetic e2e) | Phase 3.2 |+| Public eval harness gates every PR | README L62, L77 | `ci.yml :: p1-eval` | GATED (but suites are self-consistency; see reality report) | Phase 4 |+| Alert-reduction is a real measurement | README L62 | `ci.yml :: p1-eval` (`alert_reduction`) | PARTIAL (gates an in-test fusion re-impl, not `services/fusion`) | Phase 4 |+| Runs entirely on your infrastructure / no data exfiltration | README L63 | none | NO GATE | Phase 1.4 |+| Detection-as-Code rejects candidates that regress MITRE accuracy | README L170 | `ci.yml :: p1-eval` (w2-dac baseline) | PARTIAL (circular: candidate `rule_body` never evaluated) | Phase 4 |+| 800+ native detection rules | README L78, L170 | `validate-detections.yml` (strict fixture replay) | GATED | - |+| 6000+ imported detection rules | README L78 | `validate-detections.yml` (parse/provenance) | PARTIAL (97% quarantined/non-executable; heatmap is tag-based) | Phase 4 Tier 3 |+| L0-L4 automation maturity gates every action | README L171 | `ci.yml` autonomy-policy drift test | PARTIAL (thresholds gated; rollback/post-verify/dry-run ungated) | Phase 9 |+| Hunt-as-Code + `/hunt` | README L172 | `ci.yml :: p1-eval` (`hunt_corpus`) | GATED | - |+| Weekly benchmark scoreboard runs live against `main` | README L173 | `wet-eval.yml` (weekly) | NO GATE (no-ops without secret; live-agent tables are placeholders) | Phase 4 Tier 1 |+| MCP server exposes 13 tools | README L179 | `ci.yml :: mcp` | GATED | - |+| Plugin SDK Python/TS/Go | README L79, L193 | `ci.yml :: sdk-*` | PARTIAL (build/test gated; contract-drift vs `docs/openapi.yaml` ungated) | Phase 11 |+| Prompt-injection resistance | (implied by agent claims) | none (test excluded from CI file list) | NO GATE | Phase 1.1 |+| Cross-tenant isolation (Postgres) | (implied by multi-tenant) | `cross-tenant-rbac.yml` (nightly, 3 endpoints) + `ci.yml` | PARTIAL (Postgres only, compiled-SQL not live DB) | Phase 1.3 |+| Cross-tenant isolation (Qdrant/Neo4j/Redis/ClickHouse/Kafka) | (implied by multi-tenant) | none | NO GATE | Phase 1.3 |+| SAST | README badge (CodeQL) | `codeql.yml` | GATED | - |+| Dependency CVE scanning | (implied by security) | `security-audit.yml` | GATED | - |+| OpenSSF Scorecard | README badge L13 | `scorecard.yml` | GATED | - |+| Container image / IaC / secret scanning (Trivy/checkov/tfsec/gitleaks/Semgrep) | (implied by security) | none | NO GATE | Phase 2 |+| Signed / attested release artifacts | (implied by "run next to crown jewels") | `release.yml` (SPDX source SBOM), `build-extensions.yml` (extension cosign) | PARTIAL (images unsigned; no per-image SBOM/SLSA) | Phase 2 |+| Insecure defaults hard-fail in production | (implied by self-host) | none (warn-only) | NO GATE | Phase 2 |+| OpenAPI stability for 3 SDKs + MCP | (implied by SDKs) | `check-openapi.yml` (drift only) | NO GATE (no breaking-change semantics) | Phase 11 |++## Summary++- GATED: 9+- PARTIAL: 11+- NO GATE: 7++The Definition of Done requires zero `NO GATE` and zero unclosed `PARTIAL`. Each row's "Closes in" column is the binding commitment.diff --git a/docs/audit/REALITY_REPORT.md b/docs/audit/REALITY_REPORT.mdnew file mode 100644index 000000000..c1c7531a7--- /dev/null+++ b/docs/audit/REALITY_REPORT.md@@ -0,0 +1,88 @@+# AiSOC Reality Report++Phase 0 of the world-class program (`AISOC_CURSOR_PROMPT_V2.md`). This document classifies every headline claim in `README.md` against the code that is supposed to back it. It changes no code.++- Repo state audited: `main` @ `982ef2ca`, `VERSION` 7.5.0.+- Method: source-level reconciliation (read the code, not the docs) across the agent, eval harness, data spine, connectors, SOAR, storage, supply chain, and governance surfaces.++## Status legend++- `production` — unit tests **and** an integration test against a real dependency, gated in CI.+- `functional-untested` — works, no gate.+- `template-fallback` — the "AI" path silently degrades to a curated template/regex without an LLM key, and the template is what CI exercises.+- `demo-only` — only reachable via the seeder or a demo profile.+- `stub` — route/page exists, logic does not.++## Claim-by-claim classification++### "What AiSOC is" — the three differentiators (`README.md` L59-65)++| Claim | Code path | Status | Test evidence | Gated in CI? |+|---|---|---|---|---|+| Agent decisions are logged: Ledger stores prompt, response, evidence, tool calls per step | `services/agents/app/investigator/ledger.py` (records `model_used`, per-event `agent`, `input_hash`/`output_hash`); tamper-evident hash chain `services/api/app/services/audit_hash.py` (migration `043`) | functional-untested | `services/api/tests/test_audit_*`; ledger write covered | Partial — write path via `ci.yml` API tests; UI replay only in hermetic `e2e.yml` |+| Public eval harness in CI: five suites gate every PR; alert reduction is real; three rubric suites are self-consistency; fifth validates telemetry corpus | `scripts/run_evals.py` + `services/agents/tests/test_*` | production (honestly labelled) | `p1-eval` job in `.github/workflows/ci.yml` | Yes — but see Circular Gates below |+| Runs entirely on your infrastructure; no callbacks to a vendor cloud and no data exfiltration | Default cloud-LLM path `services/agents/app/security/llm_resolver.py`; no redaction before egress | functional-untested (OVERCLAIM) | none — no egress-blocked test, no PII-in-payload test | No |+| Orchestrator is a ~600-line LangGraph | `services/agents/app/orchestrator/` | functional-untested | approximate size claim, not gated | No |++### "What's in the box" (`README.md` L164-173)++| Claim | Code path | Status | Test evidence | Gated in CI? |+|---|---|---|---|---|+| 69 click-and-connect connectors, schema-driven config, live Test connection, vault-encrypted secrets | `services/connectors/app/connectors/` (69 modules), `base.py` (`schema()`, `test_connection()`, `fetch_alerts()`), `services/api/app/security/credential_vault.py` | functional-untested (count + schema production; live-connection untested) | `services/connectors/tests/test_schemas.py` runs vs every registered connector; vault tests in api/connectors/actions | Count gated (`scripts/generate_connector_count.py --check`); schema conformance gated; live API conformance NOT gated |+| Investigation Rail + replayable Ledger — every prompt/tool call/evidence/rationale stored, replayable in UI | `apps/web` Investigation Rail; `alert_rail.py`, `narrative_projection.py` | functional-untested | rail projection unit tests | Replay UI NOT gated (e2e hermetic) |+| Detection-as-Code lifecycle — propose to promote; CI rejects any candidate that regresses MITRE accuracy | `services/api/app/api/v1/endpoints/detection_proposals.py` | template-fallback (CIRCULAR) | eval gate never receives `rule_body` | Gated but circular — see below |+| 800+ native Sigma rules in `detections/` | `detections/<category>/` = 861 native (endpoint 304 / cloud 226 / identity 160 / network 81 / application 70 / data-exfil 20) | production (content) | `validate-detections.yml` strict fixture replay; 862 positive + 862 negative fixtures | Yes for native fixtures |+| L0-L4 automation maturity — gate every action on per-action confidence + blast-radius | `services/actions/app/services/maturity.py`; thresholds `services/agents/app/policy/guardrails.py` | functional-untested | thresholds + autonomy-policy drift test | Threshold sync gated; rollback/verify NOT gated |+| Hunt-as-Code + NL `/hunt` workbench | `services/api/app/api/v1/endpoints/saved_hunts.py`; `apps/web/src/app/(app)/hunt/` | functional-untested | `hunt_corpus` eval suite | Yes (corpus coverage) |+| Public weekly benchmark scoreboard — same harness, weekly against `main` | `.github/workflows/wet-eval.yml`; `apps/docs/docs/benchmark-scoreboard.mdx` | functional-untested (OVERCLAIM) | wet-eval no-ops without `WET_EVAL_OPENAI_KEY`; live-agent tables are unfilled placeholders | Effectively No for live-agent numbers |++### "How AiSOC compares" table (`README.md` L71-81)++| Claim | Reality | Status |+|---|---|---|+| Detection content: 800 native + 6000+ imported | 861 native + 6113 imported, of which ~5921 (`_quarantine`, `enabled: false`) are non-executable; ATT&CK heatmap counts metadata tags, not firing rules | OVERCLAIM (imported count implies working coverage) |+| Public substrate eval harness: CI-gated, reproducible, synthetic telemetry corpus + per-template macros | Accurate and honestly labelled in `apps/docs/docs/benchmark.md` | production |+| Agent decision audit trail: public Investigation Ledger | Real | functional-untested |++### MCP + SDK (`README.md` L177-193)++| Claim | Code path | Status | Gated in CI? |+|---|---|---|---|+| MCP server exposes 13 tools | `services/mcp/` | functional-untested | Yes (`ci.yml` MCP job: type-check/test/build) |+| Plugin SDK Python/TypeScript/Go | `packages/sdk-{py,ts,go}` | functional-untested | Build/test gated; contract-drift vs `docs/openapi.yaml` NOT gated |++## Overclaims (ranked)++1. **"No data exfiltration / runs entirely on your infrastructure."** The default investigation path uses a cloud LLM (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY`) and reasons over raw evidence. There is no PII pseudonymization (`services/agents/app/privacy/` does not exist), so usernames, hostnames, internal IPs, file paths, and command lines are sent verbatim to a third-party provider. The claim is only true in the local-model / air-gapped configuration, which is not the default and has no egress-blocked CI proof. Fix in Phase 1.4 + Phase 2.+2. **"6000+ imported detection rules."** ~5921 of 6113 imported rules live under `_quarantine/` (`enabled: false`) because their upstream query language (SPL / YARA-L / CAR pseudocode) does not execute on the engine. The coverage heatmap (`scripts/build_marketplace.py::coverage_block`) counts MITRE tags on rule metadata, not rules that fire. Fix in Phase 4 Tier 3 + Phase 10.+3. **"Detection-as-Code ... CI rejects any candidate that regresses MITRE accuracy."** True in letter, misleading in spirit: the gate never evaluates the proposed rule (see Circular Gates). Fix in Phase 4.+4. **"Public weekly benchmark scoreboard — same harness, weekly against `main`."** The weekly wet-eval no-ops without a secret and its live-agent result tables are still `<!-- placeholder -->`. Published scoreboard numbers are substrate self-consistency, not live-agent accuracy. Fix in Phase 4 Tier 1.+5. **"800 native Sigma rules."** They are AiSOC-native YAML (a bespoke `match_when` schema), not Sigma format; on-disk count is 861. Minor imprecision. Fix in Phase 2 README honesty pass.++## Load-bearing untested paths (ranked)++1. **The Kafka spine end-to-end.** No CI proves raw event -> OCSF normalize -> enrich -> fuse -> alert row -> WS frame -> agent -> ledger. `compose-smoke.yml` boots the stack but only probes `/health` + web 200. Everything else in `ci.yml` is offline/mocked. Fix in Phase 3.1.+2. **Prompt-injection defenses.** `services/agents/app/investigator/prompt_sanitizer.py` (envelope wrap + injection redaction) and `services/agents/app/llm/contract.py` (fail-closed input contract) are real and wired in, but `services/agents/tests/test_prompt_sanitizer.py` is not in the agents job's hardcoded file list in `.github/workflows/ci.yml`, so the defense is effectively ungated. Fix in Phase 1.1.+3. **Cross-tenant isolation outside Postgres.** `services/threatintel/app/storage/qdrant.py` has zero tenant scoping (global collections, no `tenant_id` filter). Neo4j/Redis/ClickHouse/Kafka have no isolation tests. `cross-tenant-rbac.yml` is nightly, Postgres-only, 3 endpoints, and asserts against compiled SQL (no live DB). Fix in Phase 1.3.+4. **SOAR rollback + post-action verification.** `rollback()` is real only for the `aws_sg` path in `services/actions/app/executors/network.py`; other vendors return `True` without a reverse call, and the live-actions layer omits rollback entirely. There is no post-action verification that a containment actually took effect. Fix in Phase 9.+5. **Backup/restore.** `backup.sh` / `restore.sh` exist; nothing tests them and no RTO/RPO is published. Fix in Phase 3.3.+6. **Approval SLA timers.** `services/slack-bot/app/services/approval_timeout.py` is in-memory; a restart wipes pending approvals (fail-safe default is reject, but the timer state is lost). Fix in Phase 9.+7. **Migration down-paths.** No per-service `downgrade base` round-trip test. Fix in Phase 3.1.++## Circular gates (ranked)++1. **DAC promotion gate.** `services/api/app/api/v1/endpoints/detection_proposals.py::_run_eval_subprocess` shells out to `scripts/run_evals.py` **without passing the proposed `rule_body`**. Approval (`/decide`) requires `eval_result.passed`, but "passed" means the repo's global substrate MITRE accuracy did not regress vs. a stored baseline — a value entirely independent of the rule being proposed. A bad rule passes its own exam. Fix in Phase 4.+2. **The gated eval suites are self-consistency.** Of the PR-gated suites, `mitre_accuracy`, `investigation_completeness`, and `response_quality` judge templated output against the data that generated it. `mitre_accuracy` runs an offline keyword extractor (`extract_tactics_from_text`), **not the live LangGraph agent** — so no live-agent accuracy is gated on any PR. This is disclosed in `apps/docs/docs/benchmark.md`, but it remains the hard ceiling on the project's credibility. Fix in Phase 4 Tier 1.+3. **`alert_reduction` gates a re-implementation, not the product.** The only non-tautological accuracy-style suite runs against an in-test fusion re-implementation, not `services/fusion`, so it does not gate the real fusion path. `detection_fp_rate` (cross-fire FP) is the one suite that tests shipped rule content. Fix in Phase 4.++## Internal inconsistencies found (fix in Phase 2 / Phase 12)++- **License disagreement.** `README.md` L229 and `LICENSE` say MIT; `.github/LICENSES.md` L3 says "AiSOC ships under Apache-2.0" and marks native detections Apache-2.0.+- **Referenced-but-missing CI guard.** `docs/decisions/0002-compliance-claims.md` asserts a CI check at `scripts/audit_compliance_claims.py` exists and fails the build on unqualified framework names. That script does not exist and no workflow references it.+- **Stale internal reference.** `AGENTS.md` and `packages/aisoc-sandbox/src/aisoc_sandbox/investigation.py` reference `services/.../llm_safety.py`; the real file is `services/agents/app/investigator/prompt_sanitizer.py`.+- **AGENTS.md eval claim is wrong.** It states only `mitre_accuracy` measures the live agent; per the code, `mitre_accuracy` is an offline keyword extractor and no PR-gated suite runs the live agent.++## Prompt premises that are already satisfied (build the delta, do not rebuild)++- Phase 1.1 assumes "nothing in the repo defending prompt injection." Reality: `prompt_sanitizer.py` + `llm/contract.py` exist. The genuine deltas are per-run nonces, a `PromptInjectionGuard` (ledger flag + auto-demote to L0), tool-call provenance/allowlisting, an adversarial eval, and CI gating.+- Phase 7 assumes the graph is built per-case at investigation time from events only. Reality: `services/ingest/internal/graph/` builds a Neo4j entity graph at ingest (17 node labels / 15 edge types). The genuine deltas are posture collection (`collect_posture()`), feeding the effective-permissions resolver (snapshot loader is a stub), bi-temporal `valid_from`/`valid_to`, a fusion-time `ContextBundle`, and a unified deterministic->ML->LLM router with tier attribution.