Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(mcp): triage verdicts, replay reports and a dry-run-only action preview (#921)
Beenu Aroraae384feedce0
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9d8d866f5..ee667833c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -98,6 +98,49 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added+- **AiSOC's own MCP server gains triage verdicts, replay reports and a+ dry-run-only action preview** (gap-closure Phase 5.6). Five new tools, 13 to+ 18.++ The Investigation Ledger half of this phase was **already built**:+ `aisoc_list_investigations`, `aisoc_get_investigation`,+ `aisoc_replay_decision` and `aisoc_explain_step` have shipped for months.+ Checking before building is the whole point, so it is recorded rather than+ re-implemented.++ New: `aisoc_get_triage_verdict` answers "what did AiSOC decide about this+ alert, and should I believe it" without 30 unrelated columns competing for+ the answer, keeps `confidence` (0 to 100) and `ai_score` (0 to 1) apart by+ name with the scale stated, and says in words when nothing has triaged the+ alert yet, because an absent verdict is not a benign verdict.+ `aisoc_list_replay_reports` and `aisoc_get_replay_report` expose the one+ number in this product measured on real data, serving the stored artefact+ rather than re-rendering it so a withheld headline stays withheld.++ **`aisoc_preview_action` previews and cannot perform.** The path it requests+ is a module constant naming the dry-run route; `/dispatch` appears nowhere+ in the server's source, and a test asserts that by reading the source rather+ than by driving the handler, so a second action tool added later is caught+ too. The API route behind it already forces `dry_run: true` server-side+ whatever the body says. An MCP key is credential material that lives in an+ editor's configuration file, and the distance between "preview" and+ "perform" should not be one careless string.++ **Every tool now publishes MCP behaviour annotations.** A client that takes+ annotations seriously cannot tell "read-only" from "nobody said", and the+ safe reading of silence is "not read-only", so a server publishing nothing+ forces every operator to vouch for every tool by name. Seventeen of the+ eighteen are read-only; `aisoc_run_investigation` is annotated as not+ read-only rather than quietly marked otherwise, because that is the+ direction of dishonesty a client cannot detect.++ **The published tool count was ungated.** "13 tools" was written into six+ documents and nothing compared any of them to the registry, while the claim+ matrix named a CI job that pins the tool *set* and had never read a+ document. `tests/published-count.test.ts` now compares all seven figures+ against `ALL_TOOLS.length` in both directions, so a stale figure and a+ deleted claim both fail.+- **An MCP client, so an investigation can reach the tools a tenant alreadyruns, read-only and untrusted by default** (gap-closure Phase 5).diff --git a/GAP_CLOSURE_PROGRESS.md b/GAP_CLOSURE_PROGRESS.mdindex 5e11892e0..148a6afae 100644--- a/GAP_CLOSURE_PROGRESS.md+++ b/GAP_CLOSURE_PROGRESS.md@@ -364,12 +364,12 @@ in D20.- [x] **5.3 Read-only by default.** A tool is callable only if the tenant allowlisted it *and* the server did not annotate it `destructiveHint: true` or `readOnlyHint: false`. Both halves, because they are different assertions: the allowlist is the tenant's, the annotation is the vendor's, and either saying no is a no. The allowlist defaults empty, so a newly registered server offers nothing. A destructive tool is refused outright and the refusal names governed dispatch; see D22 for why "or not at all" is the honest half of the plan's sentence today.- [x] **5.4 Untrusted by default.** Socket-level byte cap, per-run nonce fence, injection guard, an inline first-party boundary sentence, and a ledger row for every call, refusal and failure. The SSRF guard and air-gap policy run immediately before the socket, not at save time; see D23.- [x] **5.5 Tests and docs.** 40 tests across three files against a real `FastMCP` server over the SDK's memory transport, so nothing mocks `ClientSession` or `McpClient`. `apps/docs/docs/operations/mcp-client.md` marks all five vendor servers unverified against a live vendor, and says the tool names given are examples to be replaced with what the vendor's own `tools/list` publishes.-- [ ] **5.6 AiSOC's own MCP server** gains read tools for triage verdicts, the ledger and replay reports, plus a dry-run-only action preview.+- [x] **5.6 AiSOC's own MCP server.** Extended, not duplicated: `services/mcp` goes from 13 tools to 18. **The ledger half already existed** (`aisoc_list_investigations`, `aisoc_get_investigation`, `aisoc_replay_decision`, `aisoc_explain_step`), which is recorded rather than re-implemented. New: `aisoc_get_triage_verdict`, `aisoc_list_replay_reports`, `aisoc_get_replay_report`, `aisoc_list_actions` and `aisoc_preview_action`. The preview is dry-run only and cannot become anything else; see D24. Every tool now publishes MCP behaviour annotations, and the published tool count is gated for the first time; see D25.**Done when:** against a mock MCP server, an investigation calls an allowlistedread tool, refuses a destructive one, and the ledger records both.-**Met for 5.1 to 5.5.** `services/agents/tests/test_mcp_investigation.py::test_done_when_an_investigation_calls_a_read_tool_and_refuses_a_destructive_one`+**Met.** `services/agents/tests/test_mcp_investigation.py::test_done_when_an_investigation_calls_a_read_tool_and_refuses_a_destructive_one`runs exactly that: a real `FastMCP` server publishing `get_detections`(read-only) and `isolate_host` (destructive), both allowlisted by the operator**on purpose** so the annotation is what refuses rather than the allowlist; the@@ -524,6 +524,13 @@ Nothing yet beyond this kickoff. Each entry below will name its PR.fail against eight injected regressions rather than merely observed passing.See D21 to D23.+- [x] **Phase 5.6, AiSOC's own MCP server.** `services/mcp` 13 tools to 18,+ suite 124 to 135. Every tool now publishes MCP behaviour annotations, and+ the published tool count is compared against the registry across seven+ documents for the first time. Claim matrix 184 rows to 186, GATED 176 to+ 178, and the existing "MCP server exposes 13 tools" row was corrected+ rather than only renumbered, because the gate it named had never read a+ document. See D24 and D25.### D9. A shadow verdict must not reach `alerts.disposition`, and the guard belongs in the SQLRecorded because the obvious implementation of shadow mode is wrong in a way@@ -1066,3 +1073,74 @@ which would have read like the real check to every future reader while ageinginto a false one, and would have made the agents-side guard look like abelt-and-braces duplicate that a later cleanup could reasonably delete.>>>>>>> 54de0d37 (feat(agents): reach a tenant's own MCP servers, read-only and untrusted by default)++### D24. Two thirds of 5.6 already existed, and the third needed a boundary in code rather than in prose++5.6 names three read surfaces and one preview. Checking first, which is the+thing this repository's history says to do before building anything:++**The Investigation Ledger half was already shipped.**+`aisoc_list_investigations`, `aisoc_get_investigation`, `aisoc_replay_decision`+and `aisoc_explain_step` have been in `services/mcp/src/tools/investigations.ts`+for months. Nothing was rebuilt and nothing was wrapped.++**The triage-verdict half half-existed.** `aisoc_get_alert` already returns+every verdict field, so the new tool is a focused projection rather than a new+capability. It earns its place by handling two things the raw record does not:+`confidence` (an integer 0 to 100) and `ai_score` (a float 0 to 1) are kept+apart by name with the scale stated, because a consumer that treats one as the+other renders `2100%`, which this tree has shipped once; and a null verdict is+reported in words as "nothing has triaged this yet" rather than left as a null+field an agent reads as "no threat found".++**Replay reports had no tool at all.** Phase 1.4's surfaces exist; nothing+exposed them over MCP. The report is served **as stored**, not re-rendered,+so a withheld headline (below 30 malicious cases, Phase 1.3 refuses to print a+number) stays withheld. Re-rendering would eventually mean two definitions of+what a replay report says, and the one an agent quotes would not be the one+the operator exported.++**The preview is the part that needed care.** It is the only tool in this+server that touches the response surface, and an MCP key is credential+material that lives in an editor's configuration file. So the boundary is in+code: `DRY_RUN_PATH` is a module constant and the only action path the module+names, `/live-actions/dispatch` appears nowhere in `src/`, and the test asserts+that by **reading the source** rather than by driving the handler, so a second+action tool added later is caught rather than only the one under test. The+schema is strict, so a caller-supplied `dry_run: false` is a validation error+rather than a field forwarded to the API. The enforcing control is still+upstream, where the route forces `dry_run: true` whatever the body says; this+is the second lock, on the side an agent can see.++Proven able to fail by pointing the constant at the dispatch path, which reds+both the source scan and the request assertion.++### D25. The server published no annotations, which is the gap its own client complains about++Phase 5.1's client refuses an MCP tool whose server sets `destructiveHint:+true` or `readOnlyHint: false`, and treats an absent annotation as no claim+rather than as a claim to be read-only. AiSOC's own MCP server published+**none at all**, so by its own client's rules every one of its 13 tools was a+tool an operator would have had to vouch for by name.++Fixed by declaring them on all 18 and advertising them in `ListTools`. The+test pins the set of not-read-only tools to exactly `aisoc_run_investigation`,+in both directions: a future write tool cannot arrive annotated read-only, and+a read tool cannot drift into looking state-changing. `readOnlyHint` is+required as a boolean rather than optional, because an omission is+indistinguishable from a tool nobody thought about.++`aisoc_run_investigation` is the one honest exception. It starts an agent run:+it writes ledger rows, spends model budget and can reach whatever the tenant+configured. Annotating it read-only would be the direction of dishonesty a+client cannot detect.++**And the published tool count was ungated.** "13 tools" was written into six+documents and nothing compared any of them to `ALL_TOOLS`. The claim matrix+carried a row saying the count was gated by `ci.yml :: mcp`; that job runs the+registry tests, which pin the tool *set*, and had never read a document. The+row's Status column was therefore true about a different claim than the one it+stated. `tests/published-count.test.ts` now compares all seven figures against+the registry in both directions, proven by staling one figure and by deleting+another. `plans/cyble-aisoc/platform/README.md` also carries the old figure+and is deliberately left alone: plan files are never edited.diff --git a/ROADMAP.md b/ROADMAP.mdindex 289bf1df5..b22b59fb1 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -45,9 +45,9 @@ A proof-first, security-first program to make every README claim gate-backed, cl- **Phase E (prove it):** E1 CI-gated benchmark scoreboard tied to a deterministic live-agent MITRE-accuracy run.<<<<<<< HEAD-The claim-to-gate matrix stands at **186 rows — 178 GATED / 8 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test** and the ratchet (`MAX_NO_GATE=0`) forbids any regression. The 8 remaining PARTIAL rows are honest, named deferrals; none was relabelled without building the gate it names. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).+The claim-to-gate matrix stands at **188 rows — 180 GATED / 8 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test** and the ratchet (`MAX_NO_GATE=0`) forbids any regression. The 8 remaining PARTIAL rows are honest, named deferrals; none was relabelled without building the gate it names. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).=======-The claim-to-gate matrix stands at **186 rows — 178 GATED / 8 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test** and the ratchet (`MAX_NO_GATE=0`) forbids any regression. The 8 remaining PARTIAL rows are honest, named deferrals; none was relabelled without building the gate it names. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).+The claim-to-gate matrix stands at **188 rows — 180 GATED / 8 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test** and the ratchet (`MAX_NO_GATE=0`) forbids any regression. The 8 remaining PARTIAL rows are honest, named deferrals; none was relabelled without building the gate it names. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).>>>>>>> 54de0d37 (feat(agents): reach a tenant's own MCP servers, read-only and untrusted by default)## v4.0 — Shippeddiff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.mdindex ca6333c5d..a94bf67a5 100644--- a/apps/docs/docs/compliance/evidence-pack.md+++ b/apps/docs/docs/compliance/evidence-pack.md@@ -34,9 +34,9 @@ and says so.| Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |<<<<<<< HEAD-The last row is the one worth reading first. It is the honest index: 178 rows+The last row is the one worth reading first. It is the honest index: 180 rows=======-The last row is the one worth reading first. It is the honest index: 178 rows+The last row is the one worth reading first. It is the honest index: 180 rows>>>>>>> 54de0d37 (feat(agents): reach a tenant's own MCP servers, read-only and untrusted by default)`GATED`, 8 `PARTIAL` with the specific gap named, and a ratchet that refusesto let a row sit at `NO GATE`.diff --git a/apps/docs/docs/integrations/mcp.md b/apps/docs/docs/integrations/mcp.mdindex 8e08935eb..8225d4b77 100644--- a/apps/docs/docs/integrations/mcp.md+++ b/apps/docs/docs/integrations/mcp.md@@ -88,7 +88,7 @@ If the assistant asks for permission to call `aisoc_*` tools, that's expected## Tools exposed-The server advertises **13 tools** to your assistant. Discovery tools list things, deep-dive tools fetch one thing, the lake-query pair lets agents run governed SELECTs over the warm tier, and the replay tools expose the agent's own reasoning:+The server advertises **18 tools** to your assistant. Discovery tools list things, deep-dive tools fetch one thing, the lake-query pair lets agents run governed SELECTs over the warm tier, and the replay tools expose the agent's own reasoning:```mermaidgraph LR@@ -98,18 +98,23 @@ graph LRA3[aisoc_query_detections]A4[aisoc_list_investigations]A5[aisoc_lake_schema]+ A6[aisoc_list_replay_reports]+ A7[aisoc_list_actions]endsubgraph Deep-diveB1[aisoc_get_alert]B2[aisoc_get_case]B3[aisoc_get_detection_rule]B4[aisoc_get_investigation]+ B5[aisoc_get_triage_verdict]+ B6[aisoc_get_replay_report]endsubgraph Lake queryD1[aisoc_lake_query]endsubgraph Action / replayC1[aisoc_run_investigation]+ C4[aisoc_preview_action]C2[aisoc_replay_decision]C3[aisoc_explain_step]end@@ -127,14 +132,33 @@ graph LR| `aisoc_get_detection_rule` | Inspect a single rule (logic, fixtures, FP notes). || `aisoc_list_investigations` | Page through agent investigation runs. || `aisoc_get_investigation` | Run summary (status, duration, agents involved, cost). |+| `aisoc_get_triage_verdict` | What the agent decided about one alert: verdict, recommended actions, confidence with its rationale, and the analyst disposition. Says so explicitly when nothing has triaged the alert yet, because an absent verdict is not a benign one. |+| `aisoc_list_replay_reports` | Replay evaluations: runs that graded AiSOC triage against this tenant's own analysts' past decisions. |+| `aisoc_get_replay_report` | One replay report, served as stored rather than re-rendered. A withheld headline stays withheld. |+| `aisoc_list_actions` | Response actions this deployment can perform, and the vendor behind each. Listing one neither performs nor schedules it. || `aisoc_lake_schema` | Discover allowlisted tables and column names in the warm tier — call this *before* `aisoc_lake_query` so the agent doesn't guess column names. || `aisoc_lake_query` | Run a read-only SELECT against the warm tier (lake). Per-tenant RLS, row caps, and the `lake:query` permission are enforced server-side. |-| **`aisoc_run_investigation`** | Kick off the agent on a case and stream events back. |+| **`aisoc_run_investigation`** | Kick off the agent on a case and stream events back. The one tool here that is not read-only. |+| **`aisoc_preview_action`** | Preview a response action: blast radius, reversibility, verification probe, approval tier. **Dry run only**, see below. || **`aisoc_replay_decision`** | Walk the agent ledger step-by-step (recon, forensic, responder, reporter). || **`aisoc_explain_step`** | Why-did-the-agent-do-this for a single step: prompt, response, tool I/O. |`aisoc_replay_decision` and `aisoc_explain_step` read the investigation ledger, so the prompt, the response and the tool I/O behind any single agent step are retrievable from chat rather than only from the console.+### Annotations, and what is read-only++Every tool carries MCP behaviour annotations (`readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`). A client that takes annotations seriously cannot tell "read-only" from "nobody said", and the safe reading of silence is "not read-only", so publishing them is what lets such a client decide for itself instead of making an operator vouch for each tool by name.++**Seventeen of the eighteen are read-only.** The exception is `aisoc_run_investigation`, which starts an agent run: it writes ledger rows, spends model budget and can reach whatever the tenant has configured. It is annotated as such rather than quietly marked read-only, because that is the direction of dishonesty a client cannot detect. No tool is annotated destructive, because none is.++### Actions are previewed, never performed++`aisoc_preview_action` is the only tool that touches the response surface, and it cannot perform an action. The path it requests is a module constant naming `/api/v1/live-actions/dry-run`; the sibling `/dispatch`, which performs an action against a live vendor, appears nowhere in the server's source, and a test asserts that by reading the source rather than by driving the handler, so a second action tool added later is caught too. The API route behind it forces `dry_run: true` server-side whatever the body says.++The boundary is deliberate. An MCP key is credential material that ends up in an editor's configuration file, and the distance between "preview" and "perform" should not be one careless string. Performing an action needs the AiSOC console or API with an approval.++A preview returns the capability contract: blast radius, whether the action is reversible, the verification probe that would confirm it worked, and the approval tier it would need. That is what lets an assistant tell an analyst what containing a host would actually mean before anybody clicks anything.+## ConfigurationAll flags can be set via env vars; the CLI flag wins if both are present.diff --git a/apps/docs/docs/intro.md b/apps/docs/docs/intro.mdindex 9b93ff48d..6489b5204 100644--- a/apps/docs/docs/intro.md+++ b/apps/docs/docs/intro.md@@ -92,7 +92,7 @@ for the full list.- **AI-vs-AI adversary eval** — deterministic attacker-LLM mutator generates adversarial incidents to test detection resilience under synonym swap, leetspeak, zero-width injection, and fragmentation attacks.- **Marketplace** — 15 first-party plugins, 50+ playbooks, 6,900+ detections (filtered by tier: stable / beta / imported / community), surfaced in-app via [`marketplace/index.json`](https://github.com/beenuar/AiSOC/tree/main/marketplace).- **SDKs** — Python, TypeScript, and Go SDKs for client and plugin development; Ed25519-signed publishing.-- **Model Context Protocol** — `@aisoc/mcp` exposes 13 tools to Claude, Cursor, Continue, and Cody so analysts can replay agent decisions and run governed warm-tier SELECTs from inside their IDE ([MCP integration](./integrations/mcp)).+- **Model Context Protocol** — `@aisoc/mcp` exposes 18 tools to Claude, Cursor, Continue, and Cody so analysts can replay agent decisions and run governed warm-tier SELECTs from inside their IDE ([MCP integration](./integrations/mcp)).## Architecture Overviewdiff --git a/apps/docs/static/llms-full.txt b/apps/docs/static/llms-full.txtindex 5e7ae3e26..eecd987ef 100644--- a/apps/docs/static/llms-full.txt+++ b/apps/docs/static/llms-full.txt@@ -146,18 +146,23 @@ AiSOC to MCP-aware assistants including Claude Desktop, Cursor, Continue.dev andCody. It speaks stdio JSON-RPC and translates tool calls into AiSOC REST calls;the API key stays on the machine running the server.-It advertises 13 tools:+It advertises 18 tools:- `aisoc_explain_step`- `aisoc_get_alert`- `aisoc_get_case`- `aisoc_get_detection_rule`- `aisoc_get_investigation`+- `aisoc_get_replay_report`+- `aisoc_get_triage_verdict`- `aisoc_lake_query`- `aisoc_lake_schema`+- `aisoc_list_actions`- `aisoc_list_alerts`- `aisoc_list_cases`- `aisoc_list_investigations`+- `aisoc_list_replay_reports`+- `aisoc_preview_action`- `aisoc_query_detections`- `aisoc_replay_decision`- `aisoc_run_investigation`diff --git a/apps/docs/static/llms.txt b/apps/docs/static/llms.txtindex e835c0c10..00e82eabf 100644--- a/apps/docs/static/llms.txt+++ b/apps/docs/static/llms.txt@@ -53,7 +53,7 @@ Key facts a summary should get right:## Integrating with AiSOC-- [MCP server](https://beenuar.github.io/AiSOC/docs/integrations/mcp): connect Claude Desktop, Cursor, Continue.dev or Cody to AiSOC. Exposes 13 tools.+- [MCP server](https://beenuar.github.io/AiSOC/docs/integrations/mcp): connect Claude Desktop, Cursor, Continue.dev or Cody to AiSOC. Exposes 18 tools.- [API reference](https://beenuar.github.io/AiSOC/docs/api/rest): the REST surface.- [Python plugin SDK](https://beenuar.github.io/AiSOC/docs/plugins/python-sdk): build a connector or plugin in Python.- [Go plugin SDK](https://beenuar.github.io/AiSOC/docs/plugins/go-sdk): build a connector or plugin in Go.diff --git a/docs/architecture/SYSTEM_DESIGN.md b/docs/architecture/SYSTEM_DESIGN.mdindex 8ec52dfc3..d39b4b9f7 100644--- a/docs/architecture/SYSTEM_DESIGN.md+++ b/docs/architecture/SYSTEM_DESIGN.md@@ -102,7 +102,7 @@ This document describes the end-to-end architecture of the AiSOC platform after| `services/osquery-extensions` | Go 1.25 | Out-of-band osquery extensions (custom virtual tables + decorators) | Loaded by `osquery-tls`-managed agents || `services/slack-bot` | Python 3.11 | ChatOps surface: posts approval prompts, `/aisoc` slash command | Verifies inbound interactions with HMAC-signed Slack request signatures || `services/realtime` | Node 22 | WebSocket fan-out + Web Push for the web console + Responder PWA | n/a |-| `services/mcp` | Node 22 (TypeScript) | Model Context Protocol stdio server, 13 tools for IDE-side agents (Claude / Cursor / Continue / Cody) — discovery, deep-dive, action/replay, and the warm-tier lake query pair (`aisoc_lake_schema`, `aisoc_lake_query`) | n/a |+| `services/mcp` | Node 22 (TypeScript) | Model Context Protocol stdio server, 18 tools for IDE-side agents (Claude / Cursor / Continue / Cody) — discovery, deep-dive, action/replay, and the warm-tier lake query pair (`aisoc_lake_schema`, `aisoc_lake_query`) | n/a || `apps/web` | Next.js 14 | Server components + SWR + Responder PWA route group + benchmark scoreboard | n/a |---diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.mdindex e2071e530..0433f69fa 100644--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@ -23,7 +23,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| L0-L4 automation maturity gates every action | README L171 | `ci.yml` autonomy-policy drift test + `test_autonomy_safety.py` (dry-run-default, rollback-capability contract, verification-mandate, scorecard) + `test_governed_dispatch.py` (Phase B2 — `decide()` wired into the live-actions `/dispatch` path: above-tier→dry-run, dry-run-off→queued, L0→blocked, executor never invoked when queued/blocked) | GATED | - || Hunt-as-Code + `/hunt` | README L172 | `ci.yml :: p1-eval` (`hunt_corpus`) | GATED | - || Weekly benchmark scoreboard runs live against `main` | README L173 | `ci.yml` agents job (Phase E1 — `scripts/check_scoreboard.py` runs the deterministic live-agent MITRE-accuracy eval over the 200-incident corpus and fails if the published `scoreboard.json` substrate row drifts > 0.02, the schema is invalid, or a substrate row is mislabelled) + `wet-eval.yml` (weekly funded LLM rows) | GATED | - |-| MCP server exposes 13 tools | README L179 | `ci.yml :: mcp` | GATED | - |+| MCP server exposes 18 tools | `services/mcp/README.md`; `apps/docs/docs/integrations/mcp.md`; `apps/docs/docs/intro.md`; `docs/architecture/SYSTEM_DESIGN.md` | `ci.yml :: mcp` (`tests/tools.test.ts` pins the exact tool set, and `tests/published-count.test.ts` compares the figure in each of the seven documents that publish it against `ALL_TOOLS.length`, failing in **both** directions: a figure that disagrees, and a document that has stopped carrying one, so deleting the claim is a finding rather than a pass. A floor refuses to let an empty registry make every document agree with zero) | GATED | The row previously read 13 and named `ci.yml :: mcp` as the gate. That job pins the tool *set* and had never read a document, so six published figures could have drifted with nothing to say so || Plugin SDK Python/TS/Go | README L79, L193 | `ci.yml :: sdk-*` | PARTIAL (build/test gated; OpenAPI breaking-change now gated via `openapi-breaking.yml`, so a spec change that would break the generated SDKs is caught; per-language generated-client contract-drift is 11b) | Phase 11b || Prompt-injection resistance | (implied by agent claims) | `ci.yml :: python-test` (agents) runs `test_prompt_sanitizer.py`, `test_prompt_envelope.py` and `tests/adversarial/` — a 32-payload corpus grouped by attacker goal, enforcing a recall floor **and** a false-positive ceiling on real security content that reads adversarially. Action-trigger and exfiltration payloads are individually required rather than averaged, because an injected containment turns the SOC into a denial-of-service tool pointed at its own estate. Phase 3 adds a second, harder corpus at incident level (`test_injection_incidents.py`) whose payloads are written to fit the field they arrive in, enforcing a floor, a ceiling and an exact ratchet naming every current blind spot by id | GATED | obfuscation is an arms race and is recorded rather than individually required; no payload is xfail at present, the homoglyph and newline-flood payloads having been closed by confusable folding and by a rule reaching them. The two corpora measure the same guard and are **not comparable**: one asks whether the guard recognises a string, the other whether it recognises it where an attacker can put it, and the second scores materially lower. Neither measures a model || Prompt-injection payloads are graded against a clean twin, so a rate is attributable to the payload | `services/agents/tests/adversarial/injection_incidents.py` | `ci.yml :: python-test` (agents) runs `test_injection_incidents.py` (every pair differs in exactly **one** telemetry field, asserted by a structural diff rather than by inspection; the payload is present at that field in the injected twin and absent from the clean one; where a base incident lacks the field both twins gain the scaffold record so the injected twin never carries structure the clean twin lacks. A guard detection requires a signal at the injected field **and** no signal at that field in the clean twin, proven in the direction that broke by contaminating a base incident and asserting the pair stops counting) | GATED | The corpus is synthetic and labelled so at every record. It grades placement and phrasing, not whether a payload would be delivered: that a DNS name can carry an instruction is asserted, that an attacker controls that DNS name is assumed |@@ -194,6 +194,8 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| Every MCP call, refusal and failure is recorded in the Investigation Ledger | `apps/docs/docs/operations/mcp-client.md` (What ends up in the ledger) | `ci.yml` agents job (`test_mcp_investigation.py`: the real `_LedgerWriter` calling the real `ledger.record_event`, captured at the database boundary rather than replaced above it, so the run id, resolved tenant, sequence numbers and payload shape are the ones that would reach Postgres; sequence numbers are asserted monotonic and above the graph range because `ON CONFLICT (run_id, seq) DO NOTHING` drops a collision silently; an unresolvable tenant is asserted to write nothing while the call still succeeds) + `scripts/check_mcp_client_policy.py` (the call, refusal and failure paths must each write a ledger kind) | GATED | A ledger row carries argument *names*, byte counts and the injection verdict, never argument values or result text: copying a third party's reply into the audit trail would put untrusted content somewhere with none of the containment. Cost is `not_applicable_no_model_call` rather than `0`, since an MCP call spends a vendor's compute and no model tokens || MCP server URLs pass the SSRF guard and the air-gap policy, and stdio servers stay disabled | `apps/docs/docs/operations/mcp-client.md` (Network and deployment policy) | `ci.yml` agents job (`test_mcp_client.py::TestTransportPolicy`: a loopback URL is refused before the transport exists, air-gap mode refuses a public server and permits an internal one, stdio is off with no environment set, enabling it is not sufficient without the command allowlist, the command is matched whole rather than by prefix or basename, and a shell metacharacter in an argument is refused) + `scripts/check_mcp_client_policy.py` (`validate_outbound_url` must be called at a lower line number than `streamablehttp_client` inside `session()`; proven able to fail by deleting the guard) | GATED | The registry also refuses an obviously unsafe URL at save time, but that check does not resolve DNS and is not the control: a name that resolved publicly at save time can resolve to link-local an hour later, so the enforcing check is the one immediately before the socket opens || The agents service requests the MCP registry at a path the API service serves | `services/agents/app/mcp/registry_client.py` | `ci.yml` agents job (`test_mcp_registry_client.py::test_the_registry_url_matches_where_the_route_is_mounted`: the app mount, the v1 router prefix and the endpoint router prefix are read out of the API service's own source with `ast` and composed the way the service composes them, then compared against the URL the client builds. Proven able to fail by emptying `API_PREFIX`, which names both the requested path and the served set) | GATED | The same defect D16 records: Phase 1.2 shipped a client missing `/api/v1`, so every request would have 404'd while three suites stayed green, because the test asserted the URL the code produced rather than the URL the service serves. This service cannot import that one, so reading the other tree's source is the only comparison in the direction that drifts |+| No MCP tool AiSOC ships can perform a response action; the action tool is dry-run only | `services/mcp/README.md` (Actions are previewed, never performed); `apps/docs/docs/integrations/mcp.md` | `ci.yml :: mcp` (`tests/actions.test.ts`: the **source** of `services/mcp/src/` is read and no string may name `/live-actions/dispatch`, so a second action tool added later is caught rather than only the one under test; the preview handler is driven against a recording client and asserted to make exactly one request, to the dry-run path, carrying `dry_run: true`; and its strict schema is asserted to reject a caller-supplied `dry_run: false`. Proven able to fail by pointing the module constant at the dispatch path, which reds both the source scan and the request assertion) | GATED | The enforcing control is upstream: `POST /api/v1/live-actions/dry-run` forces `dry_run: true` server-side whatever the body says and proxies only that path. This row is the second lock, on the side an agent can see, because an MCP key is credential material that lives in an editor's configuration file |+| Every tool AiSOC's MCP server publishes carries honest behaviour annotations | `services/mcp/README.md` (Tools exposed); `apps/docs/docs/integrations/mcp.md` (Annotations, and what is read-only) | `ci.yml :: mcp` (`tests/actions.test.ts`: every tool in `ALL_TOOLS` must declare `readOnlyHint` as a boolean rather than leaving it absent, no tool may be annotated destructive, and the set of tools annotated **not** read-only is pinned to exactly `aisoc_run_investigation`, so a future write tool cannot arrive annotated read-only and a read tool cannot drift into looking state-changing) | GATED | The annotations are what a client reads to decide whether it may call a tool at all. AiSOC's own MCP client refuses a tool whose server sets `destructiveHint: true` or `readOnlyHint: false`, and treats an absent annotation as no claim, so a server publishing nothing forces every operator to vouch for every tool by name |## Summary@@ -201,7 +203,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`## Summary-- GATED: 178+- GATED: 180- PARTIAL: 8- NO GATE: 0 (**every claim is backed by a failing test.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The remaining PARTIAL rows are honest, named deferrals — each states the specific gap and what closes it — not unproven claims.)diff --git a/docs/audit/REALITY_REPORT.md b/docs/audit/REALITY_REPORT.mdindex da9890fa4..21f25b0ad 100644--- a/docs/audit/REALITY_REPORT.md+++ b/docs/audit/REALITY_REPORT.md@@ -48,7 +48,7 @@ Phase 0 of the world-class program (`AISOC_CURSOR_PROMPT_V2.md`). This document| Claim | Code path | Status | Gated in CI? ||---|---|---|---|-| MCP server exposes 13 tools | `services/mcp/` | functional-untested | Yes (`ci.yml` MCP job: type-check/test/build) |+| MCP server exposes 18 tools | `services/mcp/` | functional-untested | Yes (`ci.yml` MCP job: type-check/test/build) || Plugin SDK Python/TypeScript/Go | `packages/sdk-{py,ts,go}` | functional-untested | Build/test gated; contract-drift vs `docs/openapi.yaml` NOT gated |## Overclaims (ranked)diff --git a/docs/design/landing-page-brief.md b/docs/design/landing-page-brief.mdindex 29de3f666..38cdcf8e2 100644--- a/docs/design/landing-page-brief.md+++ b/docs/design/landing-page-brief.md@@ -889,7 +889,7 @@ Condensed teardown lives at `docs/design/landing-page-references.md`.| Marketplace (7,117 items) | `marketplace/index.json` · `scripts/build_marketplace.py` | Detection engineer | V4 | connectors | Shipped || Credential vault (Fernet + HMAC) | `services/api/app/services/vault.py` · `apps/docs/docs/operations/credentials.md` | CISO | V4, P3 | deploy, pillars (P3) | Shipped || Plugin SDKs (Python, TypeScript, Go) | `packages/plugin-sdk-py/` · `packages/plugin-sdk-ts/` · `packages/plugin-sdk-go/` | Detection engineer | V4 | features-operate | Shipped |-| MCP server (13 tools) | `services/mcp/` | Detection engineer | V4 | features-operate | Shipped |+| MCP server (18 tools) | `services/mcp/` | Detection engineer | V4 | features-operate | Shipped || Cursor extension | `services/mcp/cursor-extension/` (T7.1) | Detection engineer | V4 | features-operate | Shipped (scaffold) || Render one-click deploy | `render.yaml` · `infra/render/` | Analyst, manager | V5 (deploy) | deploy | Shipped || Fly.io deploy script | `infra/fly/` | Manager | V5 | deploy | Shipped |diff --git a/services/mcp/README.md b/services/mcp/README.mdindex 8e9be05e1..5ffe18536 100644--- a/services/mcp/README.md+++ b/services/mcp/README.md@@ -89,7 +89,9 @@ Print the resolved paths for your own machine with `node dist/index.js install -## Tools exposed-The server advertises **13 tools**. Discovery tools list things, deep-dive tools fetch one thing, the lake pair runs governed SELECTs over the warm tier, and the replay tools expose the agent's own reasoning.+The server advertises **18 tools**. Discovery tools list things, deep-dive tools fetch one thing, the lake pair runs governed SELECTs over the warm tier, and the replay tools expose the agent's own reasoning.++Every tool carries MCP behaviour annotations (`readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`), so a client that takes them seriously can decide what it may call without an operator vouching for each tool by name. **Seventeen of the eighteen are read-only.** The exception is `aisoc_run_investigation`, which starts an agent run, and it is annotated as such rather than being quietly marked read-only. No tool is annotated destructive, because none is.| Tool | Required arguments | Purpose ||---|---|---|@@ -97,18 +99,29 @@ The server advertises **13 tools**. Discovery tools list things, deep-dive tools| `aisoc_list_cases` | — | Page through cases with filters (status, owner, priority). || `aisoc_query_detections` | — | Search detection rules by name, MITRE technique, or tag. || `aisoc_list_investigations` | — | Page through agent investigation runs. |+| `aisoc_list_replay_reports` | — | Replay evaluations: runs that graded AiSOC triage against this tenant's own analysts' past decisions. |+| `aisoc_list_actions` | — | Response actions this deployment can perform, and the vendor behind each. Listing one neither performs nor schedules it. || `aisoc_lake_schema` | — | Discover allowlisted tables and columns in the warm tier. Call this *before* `aisoc_lake_query` so the agent does not guess column names. || `aisoc_get_alert` | `alert_id` | Full alert detail including enrichments and matched detections. || `aisoc_get_case` | `case_id` | Full case detail including timeline and linked alerts. || `aisoc_get_detection_rule` | `rule_id` | Inspect a single rule (logic, fixtures, false-positive notes). || `aisoc_get_investigation` | `run_id` | Run summary (status, duration, agents involved, cost). |+| `aisoc_get_triage_verdict` | `alert_id` | What the agent decided about one alert: verdict, recommended actions, confidence with its rationale, and the analyst disposition. Says so explicitly when nothing has triaged it yet. |+| `aisoc_get_replay_report` | `evaluation_id` | One replay report, served as stored rather than re-rendered. A withheld headline stays withheld. || `aisoc_lake_query` | `sql` | Read-only SELECT against the warm tier. Per-tenant RLS, row caps, and the `lake:query` permission are enforced server-side. |-| `aisoc_run_investigation` | `case_id` | Start the agent on a case and stream events back. |+| `aisoc_run_investigation` | `case_id` | Start the agent on a case and stream events back. The one tool here that is not read-only. |+| `aisoc_preview_action` | `capability` | Preview a response action: blast radius, reversibility, verification probe, approval tier. **Dry run only.** || `aisoc_replay_decision` | `run_id` | Walk the agent ledger step by step (recon, forensic, responder, reporter). || `aisoc_explain_step` | `run_id`, `step` | Prompt, response, and tool I/O for a single step. |Tools are advertised in that order deliberately: an agent reading the listing top-to-bottom meets the cheap discovery surface before the expensive one.+### Actions are previewed, never performed++`aisoc_preview_action` is the only tool that touches the response surface, and it cannot perform an action. The path it requests is a module constant naming `/api/v1/live-actions/dry-run`; the sibling `/dispatch`, which performs an action against a live vendor, appears nowhere in `src/`, and `tests/actions.test.ts` asserts that by reading the source rather than by driving the handler, so a second action tool added later is caught too. The API route behind it also forces `dry_run: true` server-side whatever the body says.++That boundary is deliberate. An MCP key is read-scoped credential material that ends up in an editor's configuration file, and the distance between "preview" and "perform" should not be one careless string. Performing an action needs the AiSOC console or API with an approval.+## ConfigurationEvery flag has an environment-variable equivalent. The CLI flag wins when both are set.diff --git a/services/mcp/src/server.ts b/services/mcp/src/server.tsindex b3f9a18f0..235040b6c 100644--- a/services/mcp/src/server.ts+++ b/services/mcp/src/server.ts@@ -87,6 +87,10 @@ export function buildServer(cfg: ServerConfig, log: Logger): Server {string,unknown>,+ // Advertised because a client that takes annotations seriously+ // cannot tell "read-only" from "nobody said" without them, and the+ // safe reading of silence is "not read-only".+ annotations: t.metadata.annotations,})),};});diff --git a/services/mcp/src/tools/actions.ts b/services/mcp/src/tools/actions.tsnew file mode 100644index 000000000..3c004f5bd--- /dev/null+++ b/services/mcp/src/tools/actions.ts@@ -0,0 +1,151 @@+/**+ * Response actions, previewed and never performed.+ *+ * Gap-closure Phase 5.6.+ *+ * This is the one place in this server where a tool touches the response+ * surface, so the boundary is drawn in code rather than in a docstring.+ *+ * `DRY_RUN_PATH` is a module constant and is the only action path this+ * module names. `/dispatch` appears nowhere in `src/`, and+ * `tests/actions.test.ts` asserts that by reading the source: an MCP tool+ * that could reach dispatch would let any agent with a read-scoped key+ * isolate a host, and the distance between "preview" and "perform" is one+ * careless string.+ *+ * The API route behind it is already dry-run-only. It forces+ * `dry_run: true` server-side whatever the body says, and it proxies only+ * `/dry-run`, never `/dispatch`. That is the enforcing control; this module+ * is the second lock, on the side an agent can see.+ *+ * What a preview is for: the contract. Every AiSOC capability declares its+ * blast radius, whether it can be reversed, the verification probe that+ * confirms it worked, and the approval tier it needs. A preview returns+ * that assessment without a vendor hearing about it, so an agent can tell+ * an analyst what containing this host would actually mean before anybody+ * clicks anything.+ */+import { z } from "zod";++import { zodToJsonSchema } from "./alerts.js";+import type { ToolDefinition } from "./types.js";+import { json } from "./types.js";++/**+ * The only action path this server will ever request.+ *+ * Named once, asserted by test. The sibling route `/dispatch` performs the+ * action against a live vendor and is deliberately unreachable from here.+ */+export const DRY_RUN_PATH = "/api/v1/live-actions/dry-run";++/** Read-only discovery of what the tenant's deployment can do. */+export const ACTION_CATALOGUE_PATH = "/api/v1/live-actions";++// ---------------------------------------------------------------------------+// aisoc_list_actions+// ---------------------------------------------------------------------------++const ListActionsSchema = z+ .object({+ capability: z+ .string()+ .max(64)+ .optional()+ .describe("Filter to one verb, e.g. `isolate_host`."),+ vendor_id: z.string().max(64).optional().describe("Filter to one vendor's implementations."),+ })+ .strict();++export const listActionsTool: ToolDefinition<typeof ListActionsSchema> = {+ metadata: {+ name: "aisoc_list_actions",+ description:+ "List the response actions this deployment can perform, with the vendor behind each. Read-only: listing an action neither performs nor schedules it. Use `aisoc_preview_action` to see what one would do.",+ inputSchema: zodToJsonSchema(ListActionsSchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },+ },+ schema: ListActionsSchema,+ async handle(ctx, args) {+ const data = await ctx.client.get<unknown>(ACTION_CATALOGUE_PATH, {+ query: { capability: args.capability, vendor_id: args.vendor_id },+ });+ return json({+ actions: data,+ note: "Nothing here has been performed. `aisoc_preview_action` is the only action tool this server exposes and it is dry-run only.",+ });+ },+};++// ---------------------------------------------------------------------------+// aisoc_preview_action+// ---------------------------------------------------------------------------++const PreviewActionSchema = z+ .object({+ capability: z+ .string()+ .min(1)+ .max(64)+ .describe("The verb to preview, e.g. `isolate_host`. List them with `aisoc_list_actions`."),+ parameters: z+ .record(z.string(), z.unknown())+ .default({})+ .describe("The arguments the action would be given, e.g. {\"hostname\": \"WIN-DC-01\"}."),+ vendor_id: z+ .string()+ .max(64)+ .optional()+ .describe("Which vendor implementation to assess. Omit to let the deployment choose."),+ confidence: z+ .number()+ .min(0)+ .max(1)+ .optional()+ .describe("The confidence an agent would attach to this action. The approval tier depends on it."),+ })+ .strict();++export const previewActionTool: ToolDefinition<typeof PreviewActionSchema> = {+ metadata: {+ name: "aisoc_preview_action",+ description:+ "Preview a response action without touching a vendor: blast radius, whether it is reversible, the verification probe, and the approval tier it would need. Dry run only. This tool cannot perform an action and there is no tool here that can.",+ inputSchema: zodToJsonSchema(PreviewActionSchema),+ annotations: {+ // Read-only is the honest annotation and the reason is structural+ // rather than a promise: the only path this module names is the+ // dry-run one, and the route behind it forces `dry_run: true`+ // server-side whatever the body says.+ readOnlyHint: true,+ destructiveHint: false,+ // Not idempotent as a claim about the world, which it does not touch,+ // but about the answer: the contract assessment depends on tenant+ // policy and earned autonomy, both of which move.+ idempotentHint: false,+ // The assessment is computed from the deployment's own contracts and+ // policy. No vendor is contacted, which is what makes it a preview.+ openWorldHint: false,+ },+ },+ schema: PreviewActionSchema,+ async handle(ctx, args) {+ const body: Record<string, unknown> = {+ capability: args.capability,+ parameters: args.parameters,+ // Sent as well as enforced upstream. Belt and braces on the field that+ // decides whether a customer's estate is touched.+ dry_run: true,+ };+ if (args.vendor_id !== undefined) body.vendor_id = args.vendor_id;+ if (args.confidence !== undefined) body.confidence = args.confidence;++ const data = await ctx.client.post<unknown>(DRY_RUN_PATH, body);+ return json({+ preview: data,+ executed: false,+ note:+ "Nothing was performed and no vendor was contacted. Performing this action needs the AiSOC console or API with an approval, and is deliberately not reachable from any MCP tool.",+ });+ },+};diff --git a/services/mcp/src/tools/alerts.ts b/services/mcp/src/tools/alerts.tsindex 536502c63..d2e558075 100644--- a/services/mcp/src/tools/alerts.ts+++ b/services/mcp/src/tools/alerts.ts@@ -89,6 +89,7 @@ export const listAlertsTool: ToolDefinition<typeof ListAlertsSchema> = {description:"List security alerts in the connected AiSOC tenant. Filter by severity, status, category, or free-text search. Results are paginated; default page_size 25.",inputSchema: zodToJsonSchema(ListAlertsSchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },},schema: ListAlertsSchema,async handle(ctx, args) {@@ -134,6 +135,7 @@ export const getAlertTool: ToolDefinition<typeof GetAlertSchema> = {description:"Fetch the full record for a single alert: AI summary, MITRE mappings, affected assets, and current case linkage.",inputSchema: zodToJsonSchema(GetAlertSchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },},schema: GetAlertSchema,async handle(ctx, args) {diff --git a/services/mcp/src/tools/cases.ts b/services/mcp/src/tools/cases.tsindex 36b6b2ce4..d1677abad 100644--- a/services/mcp/src/tools/cases.ts+++ b/services/mcp/src/tools/cases.ts@@ -75,6 +75,7 @@ export const listCasesTool: ToolDefinition<typeof ListCasesSchema> = {description:"List security cases (incidents). Filter by status, priority, or assigned-to-me. Cases are higher-level groupings of related alerts.",inputSchema: zodToJsonSchema(ListCasesSchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },},schema: ListCasesSchema,async handle(ctx, args) {@@ -117,6 +118,7 @@ export const getCaseTool: ToolDefinition<typeof GetCaseSchema> = {description:"Fetch a case by id. Optionally include the timeline of comments, status changes, and analyst actions.",inputSchema: zodToJsonSchema(GetCaseSchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },},schema: GetCaseSchema,async handle(ctx, args) {@@ -151,6 +153,11 @@ export const runInvestigationTool: ToolDefinition<typeof RunInvestigationSchema>description:"Kick off the AiSOC multi-agent investigator on a case. Returns a run_id immediately; use `aisoc_replay_decision` to fetch the resulting decision ledger.",inputSchema: zodToJsonSchema(RunInvestigationSchema),+ // Starts a multi-agent run: it writes ledger rows, spends model+ // budget and can reach whatever the tenant has configured. Not+ // read-only, and not destructive either: it creates a record of an+ // investigation rather than changing anything about the estate.+ annotations: { readOnlyHint: false, destructiveHint: false, idempotentHint: false, openWorldHint: true },},schema: RunInvestigationSchema,async handle(ctx, args) {diff --git a/services/mcp/src/tools/detections.ts b/services/mcp/src/tools/detections.tsindex 289c6478c..96ea6e6e3 100644--- a/services/mcp/src/tools/detections.ts+++ b/services/mcp/src/tools/detections.ts@@ -97,6 +97,7 @@ export const queryDetectionsTool: ToolDefinition<typeof QueryDetectionsSchema> =description:"Search the detection-rule library by free text, category, language, severity, or MITRE technique. Returns a compact summary; pull the full body with `aisoc_get_detection_rule`.",inputSchema: zodToJsonSchema(QueryDetectionsSchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },},schema: QueryDetectionsSchema,async handle(ctx, args) {@@ -168,6 +169,7 @@ export const getDetectionRuleTool: ToolDefinition<typeof GetDetectionRuleSchema>description:"Fetch the full body of a single detection rule, including the raw rule text, MITRE mappings, hit stats, and false-positive rate.",inputSchema: zodToJsonSchema(GetDetectionRuleSchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },},schema: GetDetectionRuleSchema,async handle(ctx, args) {diff --git a/services/mcp/src/tools/index.ts b/services/mcp/src/tools/index.tsindex 4f729668a..f69058d87 100644--- a/services/mcp/src/tools/index.ts+++ b/services/mcp/src/tools/index.ts@@ -8,6 +8,7 @@* tools so an agent reading the listing top-to-bottom builds an* intuition of how to navigate the surface.*/+import { listActionsTool, previewActionTool } from "./actions.js";import { getAlertTool, listAlertsTool } from "./alerts.js";import {getCaseTool,@@ -25,6 +26,8 @@ import {replayDecisionTool,} from "./investigations.js";import { lakeQueryTool, lakeSchemaTool } from "./lake.js";+import { getReplayReportTool, listReplayReportsTool } from "./replay.js";+import { getTriageVerdictTool } from "./triage.js";import type { ToolDefinition } from "./types.js";export const ALL_TOOLS: ToolDefinition[] = [@@ -33,20 +36,28 @@ export const ALL_TOOLS: ToolDefinition[] = [listCasesTool,queryDetectionsTool,listInvestigationsTool,+ listReplayReportsTool,+ listActionsTool,lakeSchemaTool,// Deep-divegetAlertTool,getCaseTool,getDetectionRuleTool,getInvestigationTool,+ getTriageVerdictTool,+ getReplayReportTool,// Lake query (warm tier — gated by lake:query permission server-side).// Listed near the bottom because it's the most expensive surface and// the schema tool above is the recommended discovery path; agents that// read the listing top-to-bottom should reach for SELECT only after// they've seen the structured tools.lakeQueryTool,- // Action / replay+ // Action / replay. `previewActionTool` is the only tool here that touches+ // the response surface, and it is dry-run only: it names the dry-run path+ // and nothing in `src/` names `/dispatch`. `tests/actions.test.ts` asserts+ // that against the source rather than trusting this comment.runInvestigationTool,+ previewActionTool,replayDecisionTool,explainStepTool,];diff --git a/services/mcp/src/tools/investigations.ts b/services/mcp/src/tools/investigations.tsindex 919415223..ecb7affc0 100644--- a/services/mcp/src/tools/investigations.ts+++ b/services/mcp/src/tools/investigations.ts@@ -99,6 +99,7 @@ export const listInvestigationsTool: ToolDefinition<typeof ListInvestigationsSchdescription:"List recent investigation runs for the connected tenant. Each run is a single agent execution against a case.",inputSchema: zodToJsonSchema(ListInvestigationsSchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },},schema: ListInvestigationsSchema,async handle(ctx, args) {@@ -129,6 +130,7 @@ export const getInvestigationTool: ToolDefinition<typeof GetInvestigationSchema>description:"Fetch a single investigation run with summary stats (status, model, tokens, cost, event/artifact counts).",inputSchema: zodToJsonSchema(GetInvestigationSchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },},schema: GetInvestigationSchema,async handle(ctx, args) {@@ -168,6 +170,7 @@ export const replayDecisionTool: ToolDefinition<typeof ReplayDecisionSchema> = {description:"Walk the agent decision ledger for an investigation run. Returns each step (recon, forensic, responder, reporter, tool-calls, errors) with summaries and timing. Use `aisoc_explain_step` for a deep dive on a single step.",inputSchema: zodToJsonSchema(ReplayDecisionSchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },},schema: ReplayDecisionSchema,async handle(ctx, args) {@@ -209,6 +212,7 @@ export const explainStepTool: ToolDefinition<typeof ExplainStepSchema> = {description:"Why-did-the-agent-do-this view for a single decision step. Returns the focal event, the previous and next events for context, and any inlined artifacts (LLM prompts/responses, tool I/O) attached to that step.",inputSchema: zodToJsonSchema(ExplainStepSchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },},schema: ExplainStepSchema,async handle(ctx, args) {diff --git a/services/mcp/src/tools/lake.ts b/services/mcp/src/tools/lake.tsindex fabf57fac..505fba2b3 100644--- a/services/mcp/src/tools/lake.ts+++ b/services/mcp/src/tools/lake.ts@@ -212,6 +212,10 @@ export const lakeQueryTool: ToolDefinition<typeof LakeQuerySchema> = {+ "(ClickHouse). Tenant isolation, table allowlisting, and LIMIT clamping "+ "are enforced server-side. Use `aisoc_lake_schema` to discover columns.",inputSchema: zodToJsonSchema(LakeQuerySchema),+ // SELECT only, rewritten with a tenant predicate server-side, so it+ // reads and cannot write. Not idempotent as a claim about results:+ // the same query over a live lake returns more rows later.+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: false, openWorldHint: false },},schema: LakeQuerySchema,async handle(ctx, args) {@@ -318,6 +322,7 @@ export const lakeSchemaTool: ToolDefinition<typeof LakeSchemaSchema> = {"List columns + types for the lake's allowlisted tables. Use this "+ "before calling `aisoc_lake_query` so you don't guess column names.",inputSchema: zodToJsonSchema(LakeSchemaSchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },},schema: LakeSchemaSchema,async handle(ctx, args) {diff --git a/services/mcp/src/tools/replay.ts b/services/mcp/src/tools/replay.tsnew file mode 100644index 000000000..7c55c2d70--- /dev/null+++ b/services/mcp/src/tools/replay.ts@@ -0,0 +1,143 @@+/**+ * Replay evaluations: how AiSOC scored against a tenant's own analysts.+ *+ * Gap-closure Phase 5.6, over the surfaces Phase 1.4 shipped.+ *+ * A replay grades AiSOC's triage against decisions the tenant's analysts+ * already made on their own closed findings. It is the one number in this+ * product measured on real data rather than on a synthetic corpus, which+ * makes it the number an agent asking "can I trust this thing" should read+ * first, and also the one most worth getting the caveats right on.+ *+ * Two of those caveats are carried in the payload rather than left to the+ * reader:+ *+ * The headline accuracy is **withheld** below 30 malicious cases, and the+ * report says so in a sentence instead of printing a figure. This tool+ * passes that sentence through untouched. Substituting a number, or a zero,+ * or a dash, would turn a deliberate refusal into a measurement.+ *+ * Every rate travels with the count it was computed over. A rate with no+ * denominator reads "not measured" rather than as a value, and an agent+ * reporting 100% over two cases is worse than one reporting nothing.+ */+import { z } from "zod";++import { zodToJsonSchema } from "./alerts.js";+import type { ToolDefinition } from "./types.js";+import { json, text } from "./types.js";++interface EvaluationSummary {+ id: string;+ vendor: string;+ status: string;+ findings_read: number;+ findings_graded: number;+ created_at: string;+ completed_at: string | null;+ [key: string]: unknown;+}++interface EvaluationDetail extends EvaluationSummary {+ report_markdown: string | null;+ score: Record<string, unknown> | null;+ error: string | null;+}++// ---------------------------------------------------------------------------+// aisoc_list_replay_reports+// ---------------------------------------------------------------------------++const ListReplaySchema = z+ .object({+ limit: z.number().int().min(1).max(200).default(20).describe("How many evaluations to return, newest first."),+ })+ .strict();++export const listReplayReportsTool: ToolDefinition<typeof ListReplaySchema> = {+ metadata: {+ name: "aisoc_list_replay_reports",+ description:+ "List this tenant's replay evaluations: runs that graded AiSOC triage against the tenant's own analysts' past decisions. Returns the vendor, the window and how many findings were read and graded.",+ inputSchema: zodToJsonSchema(ListReplaySchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },+ },+ schema: ListReplaySchema,+ async handle(ctx, args) {+ const rows = await ctx.client.get<EvaluationSummary[]>("/api/v1/evaluations/replay", {+ query: { limit: args.limit },+ });+ return json({+ count: rows.length,+ evaluations: rows,+ note:+ rows.length === 0+ ? "This tenant has run no replay evaluations. That is an absence of measurement, not a measurement of zero."+ : "Read one with `aisoc_get_replay_report`. Each rate in a report travels with the count it was computed over.",+ });+ },+};++// ---------------------------------------------------------------------------+// aisoc_get_replay_report+// ---------------------------------------------------------------------------++const GetReplaySchema = z+ .object({+ evaluation_id: z.string().uuid().describe("Evaluation UUID, from `aisoc_list_replay_reports`."),+ format: z+ .enum(["markdown", "json"])+ .default("markdown")+ .describe(+ "`markdown` returns the stored report as written, including the sentence that replaces a withheld headline. `json` returns the structured score.",+ ),+ })+ .strict();++export const getReplayReportTool: ToolDefinition<typeof GetReplaySchema> = {+ metadata: {+ name: "aisoc_get_replay_report",+ description:+ "Fetch one replay evaluation report. Returns the stored artefact rather than re-rendering it, so the numbers are the ones that were published. A withheld headline stays withheld.",+ inputSchema: zodToJsonSchema(GetReplaySchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },+ },+ schema: GetReplaySchema,+ async handle(ctx, args) {+ const detail = await ctx.client.get<EvaluationDetail>(+ `/api/v1/evaluations/replay/${args.evaluation_id}`,+ );++ if (detail.status !== "completed") {+ return json({+ evaluation_id: detail.id,+ status: detail.status,+ error: detail.error,+ note:+ "This evaluation did not complete, so it has no report. A partial replay is not a smaller measurement: the findings it did grade were not a chosen sample.",+ });+ }++ if (args.format === "json") {+ return json({+ evaluation_id: detail.id,+ vendor: detail.vendor,+ findings_read: detail.findings_read,+ findings_graded: detail.findings_graded,+ score: detail.score,+ note: "Every rate in `score` carries the count it was computed over. A rate with no count was not measured.",+ });+ }++ // The stored Markdown, untouched. Re-rendering it here would eventually+ // mean two definitions of what a replay report says, and the one an+ // agent quotes would not be the one the operator exported.+ const report = detail.report_markdown ?? "";+ return text(report, {+ evaluation_id: detail.id,+ vendor: detail.vendor,+ findings_read: detail.findings_read,+ findings_graded: detail.findings_graded,+ });+ },+};diff --git a/services/mcp/src/tools/triage.ts b/services/mcp/src/tools/triage.tsnew file mode 100644index 000000000..d4214a6d7--- /dev/null+++ b/services/mcp/src/tools/triage.ts@@ -0,0 +1,105 @@+/**+ * The agent's verdict on one alert, and how much of it was measured.+ *+ * Gap-closure Phase 5.6.+ *+ * `aisoc_get_alert` already returns the whole alert record, verdict fields+ * included, so this is not a new capability. It is a focused one: an agent+ * asking "what did AiSOC decide about this, and should I believe it?" gets+ * the verdict, the confidence, the reasoning behind the confidence and the+ * disposition, without 30 unrelated columns competing for the answer.+ *+ * Two fields are easy to read wrongly and are handled explicitly.+ *+ * `confidence` is an integer 0 to 100 and `ai_score` is a float. They are+ * different scales measuring different things, and a consumer that treats+ * one as the other renders `2100%`, which this repository has shipped once+ * already. The payload keeps them apart by name and states the scale.+ *+ * A null verdict is not a benign verdict. An alert nothing has triaged yet+ * has `ai_summary: null`, and reporting that as "no threat found" is the+ * absence-of-evidence error with a confident face on. `triaged` is an+ * explicit boolean and the payload says which case it is in words.+ */+import { z } from "zod";++import { zodToJsonSchema } from "./alerts.js";+import type { ToolDefinition } from "./types.js";+import { json } from "./types.js";++interface AlertVerdictFields {+ id: string;+ title: string;+ severity: string;+ status: string;+ disposition: string | null;+ ai_score: number | null;+ ai_summary: string | null;+ ai_recommendations: unknown[];+ confidence: number | null;+ confidence_label: string | null;+ confidence_rationale: unknown[] | null;+ mitre_tactics: string[];+ mitre_techniques: string[];+ case_id: string | null;+ [key: string]: unknown;+}++const GetTriageVerdictSchema = z+ .object({+ alert_id: z.string().uuid().describe("Alert UUID, from `aisoc_list_alerts`."),+ })+ .strict();++export const getTriageVerdictTool: ToolDefinition<typeof GetTriageVerdictSchema> = {+ metadata: {+ name: "aisoc_get_triage_verdict",+ description:+ "What the AiSOC agent decided about one alert: verdict summary, recommended actions, confidence with its rationale, and the analyst disposition if one was recorded. Says so explicitly when nothing has triaged the alert yet.",+ inputSchema: zodToJsonSchema(GetTriageVerdictSchema),+ annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },+ },+ schema: GetTriageVerdictSchema,+ async handle(ctx, args) {+ const alert = await ctx.client.get<AlertVerdictFields>(`/api/v1/alerts/${args.alert_id}`);++ const triaged = alert.ai_summary !== null && alert.ai_summary !== "";+ return json({+ alert_id: alert.id,+ title: alert.title,+ severity: alert.severity,+ status: alert.status,+ triaged,+ // Said in words as well as in the boolean, because the next reader of+ // this payload is a language model and a false boolean is easy to+ // skim past on the way to a null summary.+ note: triaged+ ? "This alert has been triaged. The summary below is the agent's, not an analyst's."+ : "Nothing has triaged this alert yet. An absent verdict is not a benign verdict: do not report this as 'no threat found'.",+ verdict: {+ summary: alert.ai_summary,+ recommended_actions: alert.ai_recommendations ?? [],+ disposition: alert.disposition,+ disposition_note:+ "`disposition` is the analyst's field, set from the feedback endpoint. Null means no analyst has ruled on it.",+ },+ confidence: {+ // Two different scales. Kept apart by name, with the scale stated,+ // because a consumer that treats one as the other renders 2100%.+ score_0_to_100: alert.confidence,+ band: alert.confidence_label,+ rationale: alert.confidence_rationale ?? [],+ ai_score_0_to_1: alert.ai_score,+ },+ mitre: {+ tactics: alert.mitre_tactics ?? [],+ techniques: alert.mitre_techniques ?? [],+ },+ case_id: alert.case_id,+ next_step:+ alert.case_id === null+ ? "No case is attached. `aisoc_run_investigation` needs a case."+ : "Use `aisoc_list_investigations` with this case_id to see what the agent actually did, then `aisoc_replay_decision` for the ledger.",+ });+ },+};diff --git a/services/mcp/src/tools/types.ts b/services/mcp/src/tools/types.tsindex cfdbb7f9f..d16f81b8f 100644--- a/services/mcp/src/tools/types.ts+++ b/services/mcp/src/tools/types.ts@@ -14,6 +14,33 @@ import type { z } from "zod";import type { AisocClient } from "../client.js";import type { Logger } from "../config.js";+/**+ * MCP tool behaviour hints, as the protocol defines them.+ *+ * These are the fields a *client* reads to decide whether it may call a+ * tool at all. AiSOC's own MCP client refuses a tool whose server sets+ * `destructiveHint: true` or `readOnlyHint: false`, and an absent+ * annotation is not read as a claim to be read-only, so a server that+ * publishes nothing here forces every operator to vouch for every tool by+ * name. Declaring them is how this server becomes usable by a client that+ * takes the annotations seriously, including ours.+ *+ * They have to be honest in both directions. A read tool marked+ * state-changing gets refused for nothing; a state-changing tool marked+ * read-only is a lie a client cannot detect, which is the direction that+ * costs something.+ */+export interface ToolAnnotations {+ /** The tool does not modify anything. */+ readOnlyHint?: boolean;+ /** The tool may perform a destructive update. Meaningless when read-only. */+ destructiveHint?: boolean;+ /** Repeating the call with the same arguments has no additional effect. */+ idempotentHint?: boolean;+ /** The tool reaches systems outside this deployment. */+ openWorldHint?: boolean;+}+/** Minimal MCP "tool" descriptor — name + JSON schema for ListToolsResult. */export interface ToolMetadata {/** Tool ID surfaced to the agent. Convention: `aisoc_<verb>_<resource>`. */@@ -22,6 +49,13 @@ export interface ToolMetadata {description: string;/** JSON Schema for the input arguments (pre-converted from zod). */inputSchema: Record<string, unknown>;+ /**+ * Behaviour hints. Required on every tool in this registry: a test+ * enforces it, because an omission is indistinguishable from a tool+ * nobody thought about, and the client-side default for a missing+ * `readOnlyHint` is "not read-only".+ */+ annotations: ToolAnnotations;}/**diff --git a/services/mcp/tests/actions.test.ts b/services/mcp/tests/actions.test.tsnew file mode 100644index 000000000..0e37b25ae--- /dev/null+++ b/services/mcp/tests/actions.test.ts@@ -0,0 +1,162 @@+/**+ * The action boundary: preview, never perform.+ *+ * Gap-closure Phase 5.6.+ *+ * One tool in this server touches the response surface. An MCP tool that+ * could reach `/dispatch` would let any agent holding a read-scoped key+ * isolate a host, and the distance between "preview" and "perform" is one+ * careless string in a path.+ *+ * So the boundary is asserted three ways, and the first is the one that+ * survives a refactor: the source is read, and `/dispatch` must appear+ * nowhere under `src/`. A test that only drove the handler would keep+ * passing if somebody added a second tool beside it.+ */+import { readFileSync, readdirSync } from "node:fs";+import { join } from "node:path";+import { fileURLToPath } from "node:url";++import { describe, expect, it } from "vitest";++import { ACTION_CATALOGUE_PATH, DRY_RUN_PATH, previewActionTool } from "../src/tools/actions.js";+import { ALL_TOOLS } from "../src/tools/index.js";++const SRC = fileURLToPath(new URL("../src", import.meta.url));++function everySourceFile(dir: string): string[] {+ const out: string[] = [];+ for (const entry of readdirSync(dir, { withFileTypes: true })) {+ const path = join(dir, entry.name);+ if (entry.isDirectory()) out.push(...everySourceFile(path));+ else if (entry.name.endsWith(".ts")) out.push(path);+ }+ return out;+}++interface RecordedCall {+ method: "get" | "post";+ path: string;+ body?: unknown;+ query?: Record<string, unknown>;+}++function recordingClient(calls: RecordedCall[], reply: unknown = { ok: true }) {+ return {+ async get(path: string, opts?: { query?: Record<string, unknown> }) {+ calls.push({ method: "get", path, query: opts?.query });+ return reply;+ },+ async post(path: string, body: unknown) {+ calls.push({ method: "post", path, body });+ return reply;+ },+ };+}++const log = { debug() {}, info() {}, warn() {}, error() {} };++describe("the action boundary", () => {+ it("names no dispatch path anywhere in the source", () => {+ // The structural assertion. Reading the source rather than the handler+ // means a second action tool added later is caught too.+ const offenders: string[] = [];+ for (const file of everySourceFile(SRC)) {+ const body = readFileSync(file, "utf8");+ // The word appears in prose explaining why it is absent, so match the+ // path segment an HTTP call would actually carry.+ if (/["'`][^"'`]*\/live-actions\/dispatch/.test(body)) {+ offenders.push(file);+ }+ }+ expect(offenders).toEqual([]);+ });++ it("requests only the dry-run path", async () => {+ const calls: RecordedCall[] = [];+ await previewActionTool.handle(+ { client: recordingClient(calls) as never, log: log as never },+ { capability: "isolate_host", parameters: { hostname: "WIN-DC-01" } },+ );+ expect(calls).toHaveLength(1);+ expect(calls[0].method).toBe("post");+ expect(calls[0].path).toBe(DRY_RUN_PATH);+ expect(DRY_RUN_PATH.endsWith("/dry-run")).toBe(true);+ });++ it("sends dry_run true as well as relying on the route to force it", async () => {+ const calls: RecordedCall[] = [];+ await previewActionTool.handle(+ { client: recordingClient(calls) as never, log: log as never },+ { capability: "isolate_host", parameters: { hostname: "WIN-DC-01" } },+ );+ expect((calls[0].body as Record<string, unknown>).dry_run).toBe(true);+ });++ it("cannot be talked into a live run through its own arguments", () => {+ // The schema is strict, so an argument the tool does not declare is a+ // validation error rather than a field forwarded to the API.+ const parsed = previewActionTool.schema.safeParse({+ capability: "isolate_host",+ parameters: {},+ dry_run: false,+ execute: true,+ });+ expect(parsed.success).toBe(false);+ });++ it("reports that nothing was performed", async () => {+ const calls: RecordedCall[] = [];+ const result = await previewActionTool.handle(+ { client: recordingClient(calls) as never, log: log as never },+ { capability: "isolate_host", parameters: { hostname: "WIN-DC-01" } },+ );+ expect(result.kind).toBe("json");+ const data = (result as { kind: "json"; data: Record<string, unknown> }).data;+ expect(data.executed).toBe(false);+ expect(String(data.note)).toContain("Nothing was performed");+ });++ it("lists actions without performing one", async () => {+ const calls: RecordedCall[] = [];+ const { listActionsTool } = await import("../src/tools/actions.js");+ await listActionsTool.handle({ client: recordingClient(calls) as never, log: log as never }, {});+ expect(calls).toHaveLength(1);+ expect(calls[0].method).toBe("get");+ expect(calls[0].path).toBe(ACTION_CATALOGUE_PATH);+ });+});++describe("tool annotations", () => {+ it("declares annotations on every tool", () => {+ // A client that takes annotations seriously cannot tell "read-only"+ // from "nobody said", and the safe reading of silence is "not+ // read-only". An omission here makes a read tool look state-changing+ // to every such client, including AiSOC's own.+ for (const tool of ALL_TOOLS) {+ expect(tool.metadata.annotations, tool.metadata.name).toBeDefined();+ expect(typeof tool.metadata.annotations.readOnlyHint, tool.metadata.name).toBe("boolean");+ }+ });++ it("marks no tool destructive, because none is", () => {+ for (const tool of ALL_TOOLS) {+ expect(tool.metadata.annotations.destructiveHint, tool.metadata.name).toBe(false);+ }+ });++ it("marks the one tool that writes as not read-only", () => {+ // `aisoc_run_investigation` starts an agent run: it writes ledger rows,+ // spends model budget and can reach whatever the tenant configured.+ // Annotating it read-only would be a lie a client cannot detect, which+ // is the direction that costs something.+ const writers = ALL_TOOLS.filter((t) => t.metadata.annotations.readOnlyHint === false).map(+ (t) => t.metadata.name,+ );+ expect(writers).toEqual(["aisoc_run_investigation"]);+ });++ it("marks the action preview read-only, since it performs nothing", () => {+ expect(previewActionTool.metadata.annotations.readOnlyHint).toBe(true);+ });+});diff --git a/services/mcp/tests/published-count.test.ts b/services/mcp/tests/published-count.test.tsnew file mode 100644index 000000000..cf6e5047e--- /dev/null+++ b/services/mcp/tests/published-count.test.ts@@ -0,0 +1,65 @@+/**+ * The published tool count must be the count this server publishes.+ *+ * Gap-closure Phase 5.6.+ *+ * "13 tools" was written into six documents across the repository and+ * nothing compared any of them to `ALL_TOOLS`. That is this repository's+ * most familiar drift shape: a figure copied into prose goes stale+ * silently, and the reader who trusts it is the one doing an evaluation.+ *+ * The claim matrix carried a row saying the count was gated by `ci.yml ::+ * mcp`, and it was not: that job runs the registry tests, which pin the+ * *set* of tool names, and nothing there had ever read a document.+ *+ * Both directions:+ *+ * document -> registry a figure that disagrees with `ALL_TOOLS` fails+ * registry -> document a document that has stopped carrying a figure+ * fails as a stale entry, rather than passing+ * because nothing matched+ */+import { readFileSync } from "node:fs";+import { fileURLToPath } from "node:url";++import { describe, expect, it } from "vitest";++import { ALL_TOOLS } from "../src/tools/index.js";++const REPO_ROOT = fileURLToPath(new URL("../../..", import.meta.url));++/** Each document, and the phrase that carries its figure. */+const PUBLISHED: Array<{ path: string; pattern: RegExp }> = [+ { path: "services/mcp/README.md", pattern: /advertises \*\*(\d+) tools\*\*/ },+ { path: "apps/docs/docs/integrations/mcp.md", pattern: /advertises \*\*(\d+) tools\*\*/ },+ { path: "apps/docs/docs/intro.md", pattern: /`@aisoc\/mcp` exposes (\d+) tools/ },+ { path: "docs/architecture/SYSTEM_DESIGN.md", pattern: /stdio server, (\d+) tools/ },+ { path: "docs/audit/REALITY_REPORT.md", pattern: /MCP server exposes (\d+) tools/ },+ { path: "docs/audit/CLAIM_TO_GATE_MATRIX.md", pattern: /MCP server exposes (\d+) tools/ },+ { path: "docs/design/landing-page-brief.md", pattern: /MCP server \((\d+) tools\)/ },+];++describe("the published tool count", () => {+ it("matches the registry in every document that publishes it", () => {+ const actual = ALL_TOOLS.length;+ // A floor, so a broken import that yields an empty registry cannot make+ // every document agree with zero.+ expect(actual).toBeGreaterThan(10);++ const mismatches: string[] = [];+ for (const { path, pattern } of PUBLISHED) {+ const body = readFileSync(new URL(path, `file://${REPO_ROOT}`), "utf8");+ const match = pattern.exec(body);+ if (match === null) {+ // registry -> document. Deleting the claim is a finding, not a pass.+ mismatches.push(`${path}: no published tool count found (pattern ${pattern})`);+ continue;+ }+ // document -> registry.+ if (Number(match[1]) !== actual) {+ mismatches.push(`${path}: publishes ${match[1]}, the registry holds ${actual}`);+ }+ }+ expect(mismatches).toEqual([]);+ });+});diff --git a/services/mcp/tests/tools.test.ts b/services/mcp/tests/tools.test.tsindex 7edbbfcd9..922152854 100644--- a/services/mcp/tests/tools.test.ts+++ b/services/mcp/tests/tools.test.ts@@ -33,6 +33,15 @@ const EXPECTED_TOOLS = [// we ship and lives in the action cluster."aisoc_lake_schema","aisoc_lake_query",+ // Gap-closure Phase 5.6. Triage verdicts and replay reports are read+ // tools over surfaces that already existed; the two action tools are the+ // only ones that touch the response surface, and the preview is dry-run+ // only (see tests/actions.test.ts).+ "aisoc_get_triage_verdict",+ "aisoc_list_replay_reports",+ "aisoc_get_replay_report",+ "aisoc_list_actions",+ "aisoc_preview_action",] as const;describe("tool registry", () => {@@ -115,9 +124,13 @@ describe("tool registry", () => {it("places action/replay tools last", () => {const names = ALL_TOOLS.map((t) => t.metadata.name);- const last3 = names.slice(-3);- expect(last3).toEqual([+ // Phase 5.6 added `aisoc_preview_action` to this cluster, so the tail is+ // four rather than three. Asserted as a slice rather than as "contains"+ // because the ordering is the point: an agent reading the listing+ // top-to-bottom should meet discovery before anything that acts.+ expect(names.slice(-4)).toEqual(["aisoc_run_investigation",+ "aisoc_preview_action","aisoc_replay_decision","aisoc_explain_step",]);