Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(phase-D3): live-vendor connector smoke (mock-server conformance) (#470)

Beenu Arora27c52b9a7a31
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 181c7d5f5..feefd4c9c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **Phase D3 — live-vendor connector smoke (mock-server conformance).** The contract test proved each connector *declares* the async runtime methods; this goes further. New `services/connectors/tests/connectors/test_live_vendor_smoke.py` stands up a mock HTTP server (respx) returning realistic vendor payloads and drives each connector's **real** `test_connection()` + paginated `fetch_alerts()` HTTP path end-to-end, asserting a successful probe and that pulled events normalize to a valid five-tier severity — catching the wrong-endpoint-path / normalize-KeyErrors-on-real-shape failures a bare contract test misses. Covers the Phase D1/D2 connectors (QRadar, Exabeam, Securonix, Devo, Netskope, Windows/Sysmon, Zeek/Suricata, syslog/CEF, LLM-usage). Moves two claim-to-gate rows PARTIAL → GATED ("Connectors: schema-driven config + vault-encrypted secrets" and "Connectors: live Test connection"), retiring the Phase 10b deferral (31 GATED / 8 PARTIAL / 1 NO GATE).
- **Phase D2 — AI/LLM-usage governance + tiered lake storage.** Three pieces. (1) **AI/LLM-usage audit connector** (`services/connectors/app/connectors/llm_usage.py`) pulls OpenAI + Anthropic organization audit logs — API key creation, role grants, logging/MFA changes, project deletes — and emits the dotted `event_type` (`openai.api_key.created`, `anthropic.member.added`) that the detections match. (2) **Eight native `llm-*` detection rules** (`scripts/detection_specs_part3_application.py`): LLM API/admin-key created, owner granted, audit logging disabled (critical), MFA disabled, service-account created, project archived — regenerated into the corpus (825 executable rules) and re-exported to the fusion live-detection ruleset; verified firing end-to-end. (3) **Hot/warm/cold lake tiering** (`services/api/clickhouse/tiering/`): an opt-in ClickHouse storage policy + `002_tiering.sql` that rebinds `aisoc.raw_events` to a `tiered` policy and moves data to a cold (object/NAS) volume at 30 days, deleting at 90 — the Phase 6 tiering wired now that the lake is populated. Verified end-to-end on ClickHouse 23.8 (policy loads, table rebinds, TTL applied); a static config gate (`test_storage_tiering.py`) catches drift. Registry 77→78; connector-count + conformance-matrix + marketplace regenerated. Claim-to-gate matrix +1 GATED (29 GATED / 10 PARTIAL / 1 NO GATE).
- **Phase D1 — eight new connectors close the biggest coverage gaps (SIEM / NDR / edge / endpoint).** Following the connector convention (schema + registry + `plugins/<id>/plugin.yaml` + docs + `marketplace:sync`), adds: **IBM QRadar** (offenses; magnitude→severity), **Exabeam** (notable risk-scored sessions), **Securonix** (incidents; priority→severity), **Devo** (triggered alerts) — the four SIEMs the reality audit flagged as missing; **Netskope** (SASE/SWG DLP/malware/anomaly alerts, malware/DLP floored at `high`); **Windows Event / Sysmon** (WEF collector spool; severity from channel + Event ID — log clears, service installs, process-injection surfaces floored); **Zeek / Suricata NDR** (Suricata `eve.json` priority + Zeek `notice` types); and a first-class **generic syslog / CEF listener** (parses the ArcSight CEF header + extension, CEF severity 0–10 → five-tier, non-CEF lines ingested at `info`). Every connector maps onto the exact five-tier ladder and passes the schema + runtime conformance gates. Registry now 69→77 connectors; connector-count + conformance-matrix + marketplace index regenerated. 40 new unit tests (severity mapping across the ladder, CEF parser, mocked pulls); full connectors suite 791 passed at 67.37% coverage. Claim-to-gate matrix +1 GATED (28 GATED / 10 PARTIAL / 1 NO GATE).
- **Phase C3 — autopilot/copilot posture with a visible autonomy scorecard (defaults to copilot).** The per-action guardrail editor already let operators scope autonomy by action; C3 adds the whole-SOC posture view a CISO asks for. New `apps/web/src/components/settings/AutonomyScorecard.tsx` computes an honest posture from the *configured* policy (not fabricated runtime stats): **Copilot** (the safe default — high/critical-blast actions always require a human) vs **Autopilot** (flips only when a high/critical-blast action is configured to auto-execute), plus the distribution of actions by blast radius and auto-exec/override counts. Rendered atop the existing `AutonomyPolicyPanel`. The compute is a pure, unit-tested function; 6 vitest tests (`AutonomyScorecard.test.tsx`). Combined with the Phase B2 `AISOC_MATURITY_TIER` gate (which enforces copilot at the dispatch layer), the platform is copilot-by-default end-to-end. Claim-to-gate matrix +1 GATED (27 GATED / 10 PARTIAL / 1 NO GATE).
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index e7f4e4d7d..a81f36c0c 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -11,8 +11,8 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| Offline sandbox demo runs with no key/network in < 5 s | README L38 | `readme-gates.yml :: sandbox-offline` (ubuntu+macos matrix) | GATED | - |
| `pnpm aisoc:demo` boots the real stack | README L40, L45 | `compose-smoke.yml` (cold health) + `integration.yml :: spine` (Phase A3 — default `docker compose up` ships connectors + graph-at-ingest by default; asserts cold-boot → ingest → lake + detection engine → alert row with the spine test's measured raw-event→alert latency) | GATED | - |
| 69 connectors | README L116, L168 | `ci.yml :: python-lint` (`generate_connector_count.py --check`) | GATED (count) | - |
-| Connectors: schema-driven config + vault-encrypted secrets | README L168 | `ci.yml` connectors matrix (`test_schemas.py` + `test_conformance.py` — secret-shaped fields must be `type=secret`); vault tests | PARTIAL (schema + vault + secret-field-marking + runtime-contract conformance gated; live-vendor API smoke, rate-limit, checkpoint durability deferred) | Phase 10b |
-| Connectors: live Test connection | README L168 | `ci.yml` connectors matrix (`test_conformance.py` — every connector implements the async `test_connection` contract) + `connector_conformance.py --check` published matrix | PARTIAL (contract conformance gated; live-vendor sandbox smoke deferred) | Phase 10b |
+| Connectors: schema-driven config + vault-encrypted secrets | README L168 | `ci.yml` connectors matrix (`test_schemas.py` + `test_conformance.py` — secret-shaped fields must be `type=secret`); vault tests + `test_live_vendor_smoke.py` (Phase D3 — mock-server drives each connector's real HTTP client + normalize) | GATED | - |
+| Connectors: live Test connection | README L168 | `ci.yml` connectors matrix (`test_conformance.py` — every connector implements the async `test_connection` contract) + `connector_conformance.py --check` published matrix + `test_live_vendor_smoke.py` (Phase D3 — mock-server conformance: `test_connection()` + paginated `fetch_alerts()` HTTP path exercised against realistic vendor payloads) | GATED | - |
| Investigation Ledger stores every step | README L61, L169 | `ci.yml :: api tests` (`audit_hash`, audit immutability) | PARTIAL (write path gated; UI replay only in hermetic e2e) | Phase 3.2 |
| Public eval harness gates every PR | README L62, L77 | `ci.yml :: p1-eval` | GATED (but suites are self-consistency; see reality report) | Phase 4 |
| Alert-reduction is a real measurement | README L62 | `ci.yml :: p1-eval` (`alert_reduction`) | PARTIAL (gates an in-test fusion re-impl, not `services/fusion`) | Phase 4 |
@@ -51,8 +51,8 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
## Summary
-- GATED: 29
-- PARTIAL: 10
+- GATED: 31
+- PARTIAL: 8
- NO GATE: 1 (progressively closed through Phases 2–11 and A1–A4: insecure-defaults, secret/IaC scanning, signed releases, cross-store isolation, DAC candidate-rule + imported-count honesty, connector live-test, OpenAPI breaking-change, ClickHouse lake population (A1), live-stream detection (A2), default cold-boot stack (A3), and the behavioral-model fusion (A4). The **last** NO GATE row — wet-eval live-agent scoreboard tables — closes in Phase 4c/E1, which needs a budgeted live-agent run)
The ratchet is enforced by `scripts/check_claim_gate_matrix.py` (wired into `security.yml`): the NO GATE count may only decrease.
diff --git a/services/connectors/tests/connectors/test_live_vendor_smoke.py b/services/connectors/tests/connectors/test_live_vendor_smoke.py
new file mode 100644
index 000000000..1a87a7e10
--- /dev/null
+++ b/services/connectors/tests/connectors/test_live_vendor_smoke.py
@@ -0,0 +1,134 @@
+"""Phase D3 — live-vendor mock-server smoke conformance.
+
+The contract test (`test_conformance.py`) proves each connector *declares* the
+async runtime methods. This smoke goes further: it stands up a mock HTTP server
+(respx) returning realistic vendor payloads and drives each connector's real
+`test_connection()` + `fetch_alerts()` HTTP path end-to-end, asserting a
+successful probe and that pulled events normalize to a valid five-tier severity.
+It exercises the httpx client, pagination, and normalize together — the failure
+mode a bare contract test misses (e.g. a wrong endpoint path or a normalize that
+KeyErrors on the real response shape).
+
+Focus is the Phase D1/D2 connectors (the newest, least battle-tested); the
+matrix ensures each is registered + conformant.
+"""
+
+from __future__ import annotations
+
+import httpx
+import pytest
+import respx
+from app.connectors.devo import DevoConnector
+from app.connectors.exabeam import ExabeamConnector
+from app.connectors.llm_usage import LlmUsageConnector
+from app.connectors.netskope import NetskopeConnector
+from app.connectors.qradar import QRadarConnector
+from app.connectors.securonix import SecuronixConnector
+from app.connectors.syslog_cef import SyslogCefConnector
+from app.connectors.windows_event import WindowsEventConnector
+from app.connectors.zeek_suricata import ZeekSuricataConnector
+
+pytestmark = pytest.mark.asyncio
+
+_LADDER = {"info", "low", "medium", "high", "critical"}
+
+
+def _assert_events(events: list[dict], connector_id: str) -> None:
+ assert events, f"{connector_id}: fetch_alerts returned no events from mock server"
+ for e in events:
+ assert e.get("source") == connector_id
+ assert e.get("severity") in _LADDER, f"{connector_id}: bad severity {e.get('severity')!r}"
+
+
+async def test_qradar_smoke():
+ respx.get(url__regex=r"https://q/api/siem/offenses.*").mock(
+ return_value=httpx.Response(200, json=[{"id": 7, "magnitude": 9, "description": "Suspicious login", "status": "OPEN"}])
+ )
+ c = QRadarConnector(console_url="https://q", sec_token="t")
+ assert (await c.test_connection())["success"] is True
+ _assert_events(await c.fetch_alerts(300), "qradar")
+
+
+async def test_exabeam_smoke():
+ respx.get("https://e/uba/api/ping").respond(200, json={"ok": True})
+ respx.get(url__regex=r"https://e/uba/api/users/sequences/notable.*").respond(
+ 200, json={"sessions": [{"username": "alice", "riskScore": 120, "sessionId": "s1", "numOfReasons": 4}]}
+ )
+ c = ExabeamConnector(base_url="https://e", api_key="k", api_secret="s")
+ assert (await c.test_connection())["success"] is True
+ _assert_events(await c.fetch_alerts(300), "exabeam")
+
+
+async def test_securonix_smoke():
+ payload = {"result": {"data": {"incidentItems": [{"incidentId": "1", "priority": "High", "entity": "bob", "entityType": "Users"}]}}}
+ respx.get(url__regex=r"https://s/ws/incident/get.*").respond(200, json=payload)
+ c = SecuronixConnector(tenant_url="https://s", api_token="t")
+ assert (await c.test_connection())["success"] is True
+ _assert_events(await c.fetch_alerts(300), "securonix")
+
+
+async def test_devo_smoke():
+ respx.get(url__regex=r"https://d/alerts/v1/alerts.*").respond(
+ 200, json={"object": [{"alertId": "a1", "severity": 4, "summary": "brute force", "context": {"srcIp": "1.2.3.4"}}]}
+ )
+ c = DevoConnector(api_url="https://d/alerts", api_token="t")
+ assert (await c.test_connection())["success"] is True
+ _assert_events(await c.fetch_alerts(300), "devo")
+
+
+async def test_netskope_smoke():
+ respx.get(url__regex=r"https://n/api/v2/events/data/alert.*").respond(
+ 200, json={"result": [{"alert_type": "malware", "severity": "low", "user": "alice", "srcip": "10.0.0.1", "_id": "x"}]}
+ )
+ c = NetskopeConnector(tenant_url="https://n", api_token="t")
+ assert (await c.test_connection())["success"] is True
+ events = await c.fetch_alerts(300)
+ _assert_events(events, "netskope")
+ assert events[0]["severity"] == "high" # malware floored
+
+
+async def test_windows_event_smoke():
+ respx.get(url__regex=r"https://w/spool.*").respond(
+ 200, json={"events": [{"EventID": 1102, "Channel": "Security", "Computer": "DC01", "EventData": {}}]}
+ )
+ c = WindowsEventConnector(collector_url="https://w/spool", api_token="t")
+ assert (await c.test_connection())["success"] is True
+ _assert_events(await c.fetch_alerts(300), "windows_event")
+
+
+async def test_zeek_suricata_smoke():
+ respx.get(url__regex=r"https://x/spool.*").respond(
+ 200, json={"events": [{"flow_id": 1, "src_ip": "1.1.1.1", "dest_ip": "2.2.2.2", "alert": {"severity": 1, "signature": "ET"}}]}
+ )
+ c = ZeekSuricataConnector(engine="suricata", spool_url="https://x/spool", api_token="t")
+ assert (await c.test_connection())["success"] is True
+ _assert_events(await c.fetch_alerts(300), "zeek_suricata")
+
+
+async def test_syslog_cef_smoke():
+ respx.get(url__regex=r"https://y/spool.*").respond(
+ 200, json={"messages": ["CEF:0|Fortinet|FortiGate|7.0|13|IPS|8|src=10.0.0.5 dst=8.8.8.8"]}
+ )
+ c = SyslogCefConnector(spool_url="https://y/spool", api_token="t")
+ assert (await c.test_connection())["success"] is True
+ _assert_events(await c.fetch_alerts(300), "syslog_cef")
+
+
+async def test_llm_usage_smoke():
+ respx.get(url__regex=r"https://api.openai.com/v1/organization/audit_logs.*").respond(
+ 200, json={"data": [{"id": "l1", "type": "api_key.created", "actor": {"email": "[email protected]"}}]}
+ )
+ c = LlmUsageConnector(provider="openai", api_key="k")
+ assert (await c.test_connection())["success"] is True
+ events = await c.fetch_alerts(300)
+ _assert_events(events, "llm_usage")
+ assert events[0]["event_type"] == "openai.api_key.created"