Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
fix(memory): only a human-confirmed, unexpired, uninjected prior auto-closes (#1082)
Beenu Arora805ae4459a2f
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex a5c880969..71bba2e06 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -18,6 +18,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- **The identity-only ceiling is no longer a bare number.** `MAX_UNAUTHORIZED` said "21 routes are excused" and nothing about which or why. Every identity-only route must now sit in a module with a **declared reason** — SCIM authenticates with its own bearer token and resolves no tenant role, passkeys and push and saved views and on-call act on the caller's own resource, `auth.py` is sign-in itself and has no principal to check before authenticating. Checked in both directions: an undeclared module fails, and a declaration that no longer describes any route fails too, so the list cannot become a place excuses outlive the thing they excused. Both directions were proven by perturbing the declaration and watching the gate fail.State-changing routes that authorize: **218 → 224**.+- **An AI triage could teach the platform to stop showing an attacker their own alerts.** Three AI-authored outcomes at ≥0.90 confidence on one evidence signature auto-closed every later alert sharing it, with no human ever having looked. The corroboration threshold was not a mitigation, because **the attacker chooses how many alerts to send** — raising it from three to thirty changes the cost, not the outcome. Only a **human-confirmed** prior suppresses now. AI priors are still written, still surfaced, still counted; they just do not close anybody's alert on their own.++- **A prior never expired.** A signature confirmed benign in March suppressed in September, in an environment that had changed in between — a decision taken once and applied forever. Priors now lapse 90 days after their *last* confirmation, so one an analyst keeps re-confirming stays live and a forgotten one returns to the queue. An unparseable or missing timestamp counts as expired, because the alternative is suppressing on a prior whose age cannot be established.++- **Evidence that tripped the prompt-injection guard now blocks auto-close**, whoever authored the prior. An alert body is attacker-reachable text; if the guard flagged it, a benign disposition derived from it is precisely what the attacker was aiming for. Checked at suppression time rather than only at write time, so a prior written before the guard learned a pattern stops suppressing once it does — and the flag is **sticky in the unsafe direction**: once any evidence for a signature has tripped the guard, later clean evidence does not clear it, or an attacker lands one flagged alert and then launders the prior with clean ones.++- **`suppression_refusal()` returns the reason, not a bool.** These rules decline far more often than the old threshold did, and "suppression declined" with no reason makes a control that has silently stopped working look identical to one that is working.++- **The flag was nearly dead code on arrival, and how it surfaced is the lesson.** `record_outcome` took `injection_suspected` from a `state` field that did not exist, so `getattr(..., False)` would have been permanently false — a control that reads correctly and never fires. Setting it from the guard's own result then broke **four** writer signatures, and *no test failed on the exception*: the call sits inside `contextlib.suppress(Exception)`, so the `TypeError` for an unexpected keyword was swallowed and the write just stopped. What surfaced was `writes_attempted["record_outcome"] == 0` in an unrelated replay test three files away. `test_outcome_writer_signature_parity.py` now compares the four signatures directly, in both directions, and asserts the concrete writer *forwards* the flag rather than merely accepting it.++ One existing test deserves credit: `test_replay_leakage.py` detected that its own premise had dissolved — its leak was constructed from an AI prior, which can no longer suppress — and said so (`"the leak this file guards against is no longer reachable and the guard above proves nothing"`) rather than passing quietly. It is rebuilt on a human prior, which is now the only kind that can leak.### BREAKINGdiff --git a/ROADMAP.md b/ROADMAP.mdindex 9eac36538..c9c4327af 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -44,10 +44,10 @@ A proof-first, security-first program to make every README claim gate-backed, cl- **Phase D (breadth):** D1 eight connectors (QRadar/Exabeam/Securonix/Devo/Netskope/Windows-Sysmon/Zeek-Suricata/syslog-CEF) · D2 AI/LLM-usage audit connector + `llm-*` detections + hot/cold lake tiering · D3 live-vendor mock-server smoke.- **Phase E (prove it):** E1 CI-gated benchmark scoreboard tied to a deterministic live-agent MITRE-accuracy run.-The claim-to-gate matrix stands at **252 rows — 252 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).-The claim-to-gate matrix stands at **252 rows — 252 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).-The claim-to-gate matrix stands at **252 rows — 252 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).-The claim-to-gate matrix stands at **252 rows — 252 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).+The claim-to-gate matrix stands at **253 rows — 253 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).+The claim-to-gate matrix stands at **253 rows — 253 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).+The claim-to-gate matrix stands at **253 rows — 253 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).+The claim-to-gate matrix stands at **253 rows — 253 GATED / 0 PARTIAL / 0 NO GATE** — **every product claim is backed by a failing test**, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).## v4.0 — Shippeddiff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.mdindex 8cdd393f1..4a15a7a7c 100644--- a/apps/docs/docs/compliance/evidence-pack.md+++ b/apps/docs/docs/compliance/evidence-pack.md@@ -33,10 +33,10 @@ and says so.| Prompt-injection resistance | Three adversarial corpora: 32 payloads scanned in isolation and 71 incident pairs whose payloads are placed in attacker-controllable fields, both against a clean twin with a recall floor and a false-positive ceiling, plus 34 held-out pairs authored after the guard was frozen and carrying no floor at all, on which the guard measures 7.1% | `services/agents/tests/adversarial/` || Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |-The last row is the one worth reading first. It is the honest index: 252 rows-The last row is the one worth reading first. It is the honest index: 252 rows-The last row is the one worth reading first. It is the honest index: 252 rows-The last row is the one worth reading first. It is the honest index: 252 rows+The last row is the one worth reading first. It is the honest index: 253 rows+The last row is the one worth reading first. It is the honest index: 253 rows+The last row is the one worth reading first. It is the honest index: 253 rows+The last row is the one worth reading first. It is the honest index: 253 rows`GATED`, 0 `PARTIAL`, and a ratchet that refuses to let a row sit at`NO GATE`. Re-read the figure from the table rather than from here — a countcopied into prose goes stale in silence, which is whydiff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.mdindex eac83d0ea..869472a39 100644--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@ -260,26 +260,27 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| No integration ships a working credential nobody chose | `apps/docs/docs/operations/security.md` (secrets) | `ci.yml :: Python — Tests` (`services/purple-team/tests/test_caldera_key_has_no_default.py`, 4 cases: the published literal is absent from code, the setting defaults to empty, building a client without a key raises and names the variable, and a configured key still builds one) | GATED | **Proven against the pre-fix tree**: 3 of 4 fail at `48234dc5`. `ADMIN123` is Caldera's own published first-run credential, so an unconfigured deployment authenticated with it against any Caldera nobody had rotated. The fourth case runs the other way so this cannot pass on a build with no client at all || Every state-changing route makes an authorization decision, or declares why identity is one | `apps/docs/docs/operations/security.md` (RBAC); `README.md` | `ci.yml :: Python — Lint & Type-check` (`check_route_authz.py` — 224 of 245 state-changing routes authorize; the remaining 21 must each sit in a module carrying a written reason, checked in both directions so an undeclared module fails *and* a declaration describing no route fails) | GATED | **Proven by perturbation**: removing a real declaration and adding one for a nonexistent module were each run and each failed the gate. 218 → 224 authorizing. Two corrections worth recording: four MSSP routes guarding cross-tenant grants were counted as unauthorized while being correctly guarded by an organisation-role dependency, fixed in the gate rather than by bolting a redundant tenant permission onto a surface that is not tenant-scoped; and one route was gated and then **reverted**, because its existing pin argued every authenticated principal is a legitimate rater and the permission added here contradicted that — a prior decision with its reasoning written down outranks a later sweep that did not read it. One limit: the gate reads the signature, so it proves a decision is *made*, not that the permission chosen is the right one || The audit log names the credential that acted, and a tenant can read its own trail | `apps/docs/docs/compliance/evidence-pack.md`; `apps/docs/docs/operations/security.md` | `ci.yml :: Python — Lint & Type-check` (`check_audit_coverage.py` — every actor-bearing `emit_audit` records the credential, `tenant_admin` holds `audit_log:read`, the modules whose *reads* are the sensitive act emit audit, and the coverage floor ratchets; 6 self-test cases including two controls) + `ci.yml :: Python — Tests` (`services/api/tests/test_audit_attribution_and_readership.py`) | GATED | **Proven against the pre-fix tree**: 7 of 11 fail at `3c04a21b`, and the gate found two live defects on its first run that the hand-written fix had missed. The honest limit is published rather than hidden: **7 of 86 endpoint modules emit audit at all**, the floor is set at that measured value rather than an aspiration, and the gate prints the largest unaudited surfaces by route count on success so the next step is named. Raising it is ordinary work, not a gap in the control. One correction recorded: the attribution rule first reported the SCIM writer, whose actor is already `scim:<token>` — a scheme-prefixed actor is self-attributing |+| Only a human-confirmed, unexpired, uninjected prior can auto-close an alert | `apps/docs/docs/concepts/automation-maturity.md`; `apps/docs/docs/console/funnel-kpis.md` | `ci.yml :: Python — Tests` (`services/agents/tests/test_outcome_memory.py` — an AI prior never suppresses at any count or confidence, a prior expires 90 days after its last confirmation, an unparseable timestamp counts as expired, injection-flagged evidence never suppresses, and the refusal names which rule declined; plus `test_outcome_writer_signature_parity.py`, which compares all four `record_outcome` signatures in both directions and asserts the concrete writer forwards the flag) | GATED | **Proven against the pre-fix tree**: `suppression_refusal` does not exist there, so the whole module fails to import, and 2 of the 10 parity cases fail. The durable finding is why the parity test exists: adding the keyword broke four writers and **no test failed on the exception**, because the call sits inside `contextlib.suppress(Exception)` — what surfaced was a zero write-count in an unrelated replay test three files away. One limit: these prove the *rules*, and the injection flag is only as good as the guard that sets it, which scores 98.1% on its tuned corpus and 7.1% on held-out payloads |## Summary-- GATED: 252+- GATED: 253## Summary-- GATED: 252+- GATED: 253## Summary-- GATED: 252+- GATED: 253## Summary-- GATED: 252+- GATED: 253## Summary-- GATED: 252+- GATED: 253- PARTIAL: 0- NO GATE: 0 (**every claim is backed by a failing test, and none is now a named deferral either.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The last two PARTIAL rows closed with `scripts/check_live_agent_floor.py` and a floor derived from ten runs across two environments, not by relabelling. Recount with `scripts/check_claim_gate_matrix.py` rather than trusting these three numbers: they move weekly, and a count copied into prose goes stale in silence.)diff --git a/services/agents/app/memory/outcomes.py b/services/agents/app/memory/outcomes.pyindex 232fa2e28..6f3d82b5f 100644--- a/services/agents/app/memory/outcomes.py+++ b/services/agents/app/memory/outcomes.py@@ -6,9 +6,26 @@ instead of re-triaged. This is what makes alert volume actually shrink overtime: confirmed-benign signatures stop reaching the queue.Honesty + safety:-* AI-authored priors are trusted less than human-verified ones. A human prior- suppresses immediately; an AI prior only suppresses once it has been- corroborated by repeat outcomes (``count``) at high confidence.+* **Only a human-confirmed prior suppresses.** An AI prior informs — it is+ still recorded, still surfaced, still counted — but it never auto-closes+ an alert on its own. The previous rule let three AI triages at >=0.90+ confidence close every future alert with that signature, which is a+ teachable control: an attacker who can produce three benign-looking+ alerts sharing one evidence signature trains the system to stop showing+ them that signature, permanently, with no human ever having looked. The+ corroboration threshold did not help, because the attacker chooses how+ many alerts to send.+* **A prior expires.** Environments change: a signature that was benign in+ March is not necessarily benign in September, and a prior that never ages+ out is a decision taken once and applied forever. Default 90 days from+ the last confirmation, so an actively re-confirmed prior stays live and a+ forgotten one lapses into "show it to somebody".+* **Evidence that tripped the prompt-injection guard never suppresses**,+ whoever authored the prior. An alert body is attacker-reachable text; if+ the guard flagged it, the disposition derived from it is exactly what the+ attacker was aiming for. Checked at suppression time rather than only at+ write time, because a prior written before the guard learned a pattern+ must stop suppressing once it does.* Only auto-closeable dispositions (false_positive / benign / benign_true_positive)ever suppress. A prior true-positive never auto-closes a future alert.* Human authorship is sticky: an AI prior that a human later overrode stays@@ -18,7 +35,7 @@ Honesty + safety:from __future__ import annotationsimport os-from datetime import UTC, datetime+from datetime import UTC, datetime, timedeltafrom typing import Anyimport structlog@@ -32,10 +49,14 @@ OUTCOME_KEY_PREFIX = "outcome:"HUMAN = "human"AI = "ai"-# Suppression gates for AI-authored priors (env-overridable). Human priors are-# always trusted; AI priors need corroboration.-_MIN_AI_COUNT = int(os.getenv("AISOC_MEMORY_MIN_AI_COUNT", "3"))-_AI_CONF_THRESHOLD = float(os.getenv("AISOC_MEMORY_AI_CONFIDENCE", "0.90"))+#: How long a confirmation keeps suppressing, in days. Measured from+#: ``last_seen``, so re-confirming refreshes it and a prior nobody has seen+#: since lapses into "show it to somebody".+_PRIOR_TTL_DAYS = int(os.getenv("AISOC_MEMORY_PRIOR_TTL_DAYS", "90"))++#: Key an evidence record carries when the prompt-injection guard flagged the+#: text the disposition was derived from.+INJECTION_FLAG = "injection_suspected"def outcome_key(signature: str) -> str:@@ -50,6 +71,7 @@ async def record_outcome(confidence: float,author: str = AI,alert_id: Any = None,+ injection_suspected: bool = False,) -> dict[str, Any]:"""Write/refresh the per-signature outcome prior. Best-effort (never raises)."""disposition = normalize_disposition(disposition, default="needs_review")@@ -76,6 +98,11 @@ async def record_outcome("first_seen": first_seen,"last_seen": now,"last_alert_id": str(alert_id) if alert_id else None,+ # Sticky in the unsafe direction: once any evidence for this+ # signature has tripped the guard, later clean evidence does not+ # clear it. An attacker who can get one flagged alert through and+ # then send clean ones would otherwise launder the prior.+ INJECTION_FLAG: bool(injection_suspected) or bool(isinstance(prior, dict) and prior.get(INJECTION_FLAG)),}try:await institutional_set(@@ -96,19 +123,78 @@ async def lookup_prior(tenant_id: str, signature: str) -> dict[str, Any] | None:return prior if isinstance(prior, dict) else None-def should_auto_suppress(+def prior_is_expired(prior: dict[str, Any], *, ttl_days: int = _PRIOR_TTL_DAYS, now: datetime | None = None) -> bool:+ """True when the last confirmation is older than the TTL.++ Measured from ``last_seen`` rather than ``first_seen``: a signature an+ analyst keeps confirming benign stays suppressed, and one nobody has+ looked at since lapses. An unparseable or missing timestamp counts as+ expired, because the alternative is suppressing on a prior whose age+ cannot be established.+ """+ raw = prior.get("last_seen") or prior.get("first_seen")+ if not isinstance(raw, str):+ return True+ try:+ seen = datetime.fromisoformat(raw)+ except ValueError:+ return True+ if seen.tzinfo is None:+ seen = seen.replace(tzinfo=UTC)+ return (now or datetime.now(UTC)) - seen > timedelta(days=ttl_days)+++def suppression_refusal(prior: dict[str, Any] | None,*,- min_ai_count: int = _MIN_AI_COUNT,- ai_confidence: float = _AI_CONF_THRESHOLD,-) -> bool:- """Decide whether a prior is trusted enough to auto-suppress a repeat alert."""+ ttl_days: int = _PRIOR_TTL_DAYS,+ now: datetime | None = None,+) -> str | None:+ """Why this prior may not auto-close an alert, or None if it may.++ A reason string rather than a bool so the caller can log *which* rule+ refused. "Suppression declined" with no reason is how a control that has+ silently stopped working looks identical to one that is working.+ """if not isinstance(prior, dict):- return False+ return "no prior recorded for this signature"+disposition = str(prior.get("disposition", ""))if disposition not in AUTO_CLOSEABLE_DISPOSITIONS:- return False- if prior.get("author") == HUMAN:- return True- # AI-authored: require corroboration by repeat count + high confidence.- return int(prior.get("count", 0)) >= min_ai_count and float(prior.get("confidence", 0.0)) >= ai_confidence+ return f"disposition {disposition!r} is not auto-closeable"++ if prior.get(INJECTION_FLAG):+ return (+ "the evidence this prior was derived from tripped the prompt-injection guard; "+ "a benign disposition reached that way is what an attacker was aiming for"+ )++ if prior.get("author") != HUMAN:+ return (+ "the prior is AI-authored and no human has confirmed it. An attacker who can "+ "produce benign-looking alerts sharing one signature would otherwise train the "+ "system to stop showing them"+ )++ if prior_is_expired(prior, ttl_days=ttl_days, now=now):+ return (+ f"the last confirmation is older than {ttl_days} days; environments change and a "+ "prior that never ages out is a decision taken once and applied forever"+ )++ return None+++def should_auto_suppress(+ prior: dict[str, Any] | None,+ *,+ ttl_days: int = _PRIOR_TTL_DAYS,+ now: datetime | None = None,+) -> bool:+ """Whether a prior is trusted enough to auto-close a repeat alert.++ Kept as a predicate for the existing call sites; `suppression_refusal`+ is the one to call when the reason matters, which at the point of+ closing somebody's alert it usually does.+ """+ return suppression_refusal(prior, ttl_days=ttl_days, now=now) is Nonediff --git a/services/agents/app/models/state.py b/services/agents/app/models/state.pyindex 128c9d9d7..aa118a3db 100644--- a/services/agents/app/models/state.py+++ b/services/agents/app/models/state.py@@ -88,6 +88,16 @@ class InvestigationState(BaseModel):# `bundle.py`. A type annotation is not worth an import cycle.knowledge_base: dict[str, Any] | None = None+ #: True when the prompt-injection guard refused any evidence retrieved+ #: for this alert. Carried onto the per-signature outcome prior, so a+ #: benign disposition reached through attacker-reachable text can never+ #: auto-close a later alert with the same signature.+ #:+ #: A field rather than a lookup at suppression time: by then the+ #: retrieval is gone, and recomputing it would mean re-running the guard+ #: over text the triage no longer holds.+ injection_suspected: bool = False+# The last few analyst decisions on alerts of this shape, as# `app.context.dispositions.RecentDispositions.as_state()` renders them,# and the directory record for the principals this alert names, asdiff --git a/services/agents/app/replay/shadow.py b/services/agents/app/replay/shadow.pyindex 4d5a688cf..4e69d0cc8 100644--- a/services/agents/app/replay/shadow.py+++ b/services/agents/app/replay/shadow.py@@ -169,6 +169,7 @@ class ShadowTriageWriter:confidence: float,author: str,alert_id: Any = None,+ injection_suspected: bool = False,) -> None:self.calls["record_outcome"] += 1diff --git a/services/agents/app/workers/fused_alert_consumer.py b/services/agents/app/workers/fused_alert_consumer.pyindex 66a53440b..1ca5687e8 100644--- a/services/agents/app/workers/fused_alert_consumer.py+++ b/services/agents/app/workers/fused_alert_consumer.py@@ -56,7 +56,7 @@ from app.graph.runner import default_budget, run_escalationfrom app.investigator import ledger as ledger_modulefrom app.investigator.bundle_prompt import prefetch_context_bundle_dictfrom app.llm.factory import llm_override-from app.memory.outcomes import AI, should_auto_suppress+from app.memory.outcomes import AI, suppression_refusalfrom app.models.state import AgentStatus, InvestigationStatefrom app.routing.model_router import is_deterministic_modefrom app.security.llm_resolver import resolve_llm_config@@ -671,6 +671,11 @@ class FusedAlertTriageWorker:confidence=float(confidence or 0.0),author=AI,alert_id=(state.raw_alert or {}).get("id"),+ # The guard already ran over this alert's runbook+ # retrieval; carrying its verdict onto the prior is what+ # stops a disposition derived from attacker-reachable+ # text from auto-closing the next one.+ injection_suspected=bool(getattr(state, "injection_suspected", False)),)_METRICS["outcome_written"] += 1@@ -892,6 +897,11 @@ class FusedAlertTriageWorker:_METRICS["runbooks_retrieved"] += len(retrieval.runbooks)if retrieval.dropped_for_injection:_METRICS["runbooks_refused_for_injection"] += retrieval.dropped_for_injection+ # Recorded on the state so it reaches the outcome prior. Without+ # this the flag threaded into `record_outcome` would read a field+ # nobody sets and the whole injection rule would be dead code+ # that looks like a control.+ state.injection_suspected = Trueasync def _attach_recent_dispositions(self, state: InvestigationState) -> None:"""Record what this tenant's analysts decided the last few times.@@ -959,10 +969,20 @@ class FusedAlertTriageWorker:except Exception as exc: # noqa: BLE001 — memory read is advisorylogger.debug("auto_triage_worker.memory_lookup_failed", error=str(exc))return None- if not should_auto_suppress(prior):+ refusal = suppression_refusal(prior)+ if refusal is not None:+ # Logged with the reason. "Suppression declined" alone makes a+ # control that has silently stopped working look identical to one+ # that is working, and these rules decline far more often now+ # than the old corroboration threshold did.+ logger.debug(+ "auto_triage_worker.suppression_declined",+ signature=signature,+ reason=refusal,+ )return None- assert prior is not None # narrowed by should_auto_suppress+ assert prior is not None # narrowed by suppression_refusaldisposition = normalize_disposition(prior.get("disposition"), default=NEEDS_REVIEW)confidence = float(prior.get("confidence", 0.9))count = int(prior.get("count", 1))diff --git a/services/agents/app/workers/shadow_mode.py b/services/agents/app/workers/shadow_mode.pyindex 26f713e7d..f22bdbdcd 100644--- a/services/agents/app/workers/shadow_mode.py+++ b/services/agents/app/workers/shadow_mode.py@@ -357,6 +357,7 @@ class ShadowModeTriageWriter:confidence: float,author: str,alert_id: Any = None,+ injection_suspected: bool = False,) -> None:# An outcome prior lets a later identical alert close without triage.# A shadow verdict must never reach that far: it would act on a futurediff --git a/services/agents/app/workers/triage_persistence.py b/services/agents/app/workers/triage_persistence.pyindex 683902bff..8d069d0d9 100644--- a/services/agents/app/workers/triage_persistence.py+++ b/services/agents/app/workers/triage_persistence.py@@ -163,6 +163,7 @@ class TriageWriter(Protocol):confidence: float,author: str,alert_id: Any = None,+ injection_suspected: bool = False,) -> None:"""Write the per-signature outcome prior that lets a repeat alert suppress."""@@ -293,6 +294,7 @@ class LiveTriageWriter:confidence: float,author: str,alert_id: Any = None,+ injection_suspected: bool = False,) -> None:await outcomes_module.record_outcome(tenant_id,@@ -301,6 +303,7 @@ class LiveTriageWriter:confidence=confidence,author=author,alert_id=alert_id,+ injection_suspected=injection_suspected,)async def record_suppression(diff --git a/services/agents/tests/test_outcome_memory.py b/services/agents/tests/test_outcome_memory.pyindex 195fecee5..d0947f348 100644--- a/services/agents/tests/test_outcome_memory.py+++ b/services/agents/tests/test_outcome_memory.py@@ -8,6 +8,8 @@ corroborated-AI priors and never auto-closes a prior true-positive.from __future__ import annotations+from datetime import UTC, datetime, timedelta+import pytestfrom app.core.cost_governor import get_governorfrom app.memory import institutional@@ -17,6 +19,7 @@ from app.memory.outcomes import (lookup_prior,record_outcome,should_auto_suppress,+ suppression_refusal,)from app.workers import fused_alert_consumer as worker_modfrom app.workers.fused_alert_consumer import FusedAlertTriageWorker, build_state@@ -87,14 +90,78 @@ def _signature(message: dict) -> str:# ── unit: trust gate ─────────────────────────────────────────────────────────-async def test_should_auto_suppress_trust_gate():- # Human benign prior → always trusted.- assert should_auto_suppress({"disposition": "false_positive", "author": HUMAN, "count": 1, "confidence": 0.5})- # AI prior needs corroboration (count) + high confidence.- assert not should_auto_suppress({"disposition": "benign", "author": AI, "count": 1, "confidence": 0.99})- assert should_auto_suppress({"disposition": "benign", "author": AI, "count": 3, "confidence": 0.95})- # A prior true-positive NEVER auto-closes a future alert.- assert not should_auto_suppress({"disposition": "true_positive", "author": HUMAN, "count": 9, "confidence": 1.0})+def _fresh(**over):+ """A prior last confirmed today, so TTL is not what is under test."""+ base = {+ "disposition": "false_positive",+ "author": HUMAN,+ "count": 1,+ "confidence": 0.5,+ "last_seen": datetime.now(UTC).isoformat(),+ }+ base.update(over)+ return base+++async def test_only_a_human_confirmed_prior_suppresses():+ """A corroboration threshold does not help when the attacker picks the count.++ The previous rule let three AI triages at >=0.90 confidence auto-close+ every future alert sharing that evidence signature. That is a teachable+ control: anyone who can produce three benign-looking alerts with one+ signature trains the system to stop showing them that signature, with+ no human having looked. Raising the threshold would not have fixed it —+ the attacker chooses how many alerts to send.+ """+ assert should_auto_suppress(_fresh())+ assert not should_auto_suppress(_fresh(author=AI, count=1, confidence=0.99))+ assert not should_auto_suppress(_fresh(author=AI, count=3, confidence=0.95))+ assert not should_auto_suppress(_fresh(author=AI, count=999, confidence=1.0))+ assert "AI-authored" in (suppression_refusal(_fresh(author=AI, count=3, confidence=0.95)) or "")+++async def test_a_prior_true_positive_never_auto_closes():+ assert not should_auto_suppress(_fresh(disposition="true_positive", count=9, confidence=1.0))+++async def test_a_prior_expires():+ """Environments change, and a prior that never ages out is permanent.++ Measured from `last_seen`, so a signature an analyst keeps confirming+ stays suppressed and one nobody has looked at since lapses into "show+ it to somebody".+ """+ stale = _fresh(last_seen=(datetime.now(UTC) - timedelta(days=120)).isoformat())+ assert not should_auto_suppress(stale)+ assert "older than" in (suppression_refusal(stale) or "")+ # And the boundary in the other direction.+ assert should_auto_suppress(_fresh(last_seen=(datetime.now(UTC) - timedelta(days=30)).isoformat()))+++async def test_an_unparseable_timestamp_counts_as_expired():+ """Failing closed, because the alternative is suppressing on an unknown age."""+ assert not should_auto_suppress(_fresh(last_seen="not-a-date"))+ assert not should_auto_suppress({"disposition": "benign", "author": HUMAN})+++async def test_injection_tripped_evidence_never_suppresses():+ """Whoever authored it.++ An alert body is attacker-reachable text. If the prompt-injection guard+ flagged the evidence a benign disposition was derived from, that+ disposition is exactly what the attacker was aiming for — and the check+ is at suppression time, not write time, so a prior written before the+ guard learned a pattern stops suppressing once it does.+ """+ flagged = _fresh(injection_suspected=True)+ assert not should_auto_suppress(flagged)+ assert "injection" in (suppression_refusal(flagged) or "")+++async def test_the_refusal_says_which_rule_declined():+ """ "Declined" with no reason makes a broken control look like a working one."""+ assert suppression_refusal(None) is not None+ assert suppression_refusal(_fresh()) is Noneasync def test_record_and_lookup_roundtrip_and_count_bump():diff --git a/services/agents/tests/test_outcome_writer_signature_parity.py b/services/agents/tests/test_outcome_writer_signature_parity.pynew file mode 100644index 000000000..5f9bc2f4f--- /dev/null+++ b/services/agents/tests/test_outcome_writer_signature_parity.py@@ -0,0 +1,103 @@+"""Every outcome writer accepts what the caller passes.++Four classes implement `record_outcome` — the real persister, the shadow+writer, the replay shadow, and the protocol they share. Adding+`injection_suspected` to the caller broke three of them, and **not one test+failed on the exception**: the call site sits inside+`contextlib.suppress(Exception)`, so a `TypeError` for an unexpected keyword+was swallowed and the write simply stopped happening.++What surfaced was `writes_attempted["record_outcome"] == 0` in an unrelated+replay test, three files away from the cause. That is the same shape as the+SOAR executors whose client signatures drifted because simulation mode never+constructed the client: a call that is never made in the tested path cannot+reveal a signature that no longer matches.++So the parity is asserted directly, by comparing parameters rather than by+calling through and hoping something notices.+"""++from __future__ import annotations++import inspect+from typing import Any++import pytest+from app.memory import outcomes as outcomes_module+from app.replay.shadow import ShadowTriageWriter as ReplayShadowWriter+from app.workers.shadow_mode import ShadowModeTriageWriter+from app.workers.triage_persistence import LiveTriageWriter, TriageWriter++#: The free function is the contract. Everything else must accept what it+#: accepts, minus `self` and the two positional arguments.+CANONICAL = outcomes_module.record_outcome++WRITERS = (+ ("TriageWriter protocol", TriageWriter),+ ("shadow_mode.ShadowModeTriageWriter", ShadowModeTriageWriter),+ ("triage_persistence.LiveTriageWriter", LiveTriageWriter),+ ("replay.shadow.ShadowTriageWriter", ReplayShadowWriter),+)+++def _keyword_params(fn) -> set[str]:+ return {+ name+ for name, p in inspect.signature(fn).parameters.items()+ if name not in ("self", "tenant_id", "signature") and p.kind in (p.KEYWORD_ONLY, p.POSITIONAL_OR_KEYWORD)+ }+++def test_the_canonical_writer_declares_the_injection_flag() -> None:+ """The rule the others are measured against.++ Stated separately so a future change that removes it from the canonical+ function fails here rather than silently relaxing every parity case+ below into a comparison of two empty sets.+ """+ assert "injection_suspected" in _keyword_params(CANONICAL)++[email protected](("label", "cls"), WRITERS)+def test_every_writer_accepts_the_canonical_keywords(label: str, cls: Any) -> None:+ missing = _keyword_params(CANONICAL) - _keyword_params(cls.record_outcome)+ assert not missing, (+ f"{label}.record_outcome does not accept {sorted(missing)}. The call site is inside "+ "contextlib.suppress(Exception), so the TypeError is swallowed and the write stops "+ "happening with no error anywhere"+ )++[email protected](("label", "cls"), WRITERS)+def test_no_writer_requires_something_the_caller_does_not_pass(label: str, cls: Any) -> None:+ """The other direction.++ A writer that adds a required keyword of its own is just as broken, and+ fails in exactly the same silent way.+ """+ required = {+ name+ for name, p in inspect.signature(cls.record_outcome).parameters.items()+ if name not in ("self", "tenant_id", "signature")+ and p.default is inspect.Parameter.empty+ and p.kind in (p.KEYWORD_ONLY, p.POSITIONAL_OR_KEYWORD)+ }+ unknown = required - _keyword_params(CANONICAL)+ assert not unknown, f"{label}.record_outcome requires {sorted(unknown)}, which no caller passes"+++def test_the_concrete_writer_passes_the_flag_through() -> None:+ """Accepting it and dropping it is the same defect one layer down.++ `PostgresTriageWriter.record_outcome` forwards to the free function; a+ parameter it accepts and never forwards would satisfy every case above+ while the prior still lost the flag.+ """+ from app.workers import triage_persistence++ source = inspect.getsource(triage_persistence)+ forwarding = source.split("await outcomes_module.record_outcome(")[1]+ assert "injection_suspected=injection_suspected" in forwarding.split(")")[0], (+ "the concrete writer accepts injection_suspected and does not forward it, so the "+ "flag stops at the port and the prior never carries it"+ )diff --git a/services/agents/tests/test_replay_leakage.py b/services/agents/tests/test_replay_leakage.pyindex e950ef30d..8ae5933af 100644--- a/services/agents/tests/test_replay_leakage.py+++ b/services/agents/tests/test_replay_leakage.py@@ -230,8 +230,18 @@ async def test_the_same_run_leaks_through_memory_once_the_shadow_sinks_are_remov# and this whole file would disarm itself quietly."disposition": normalize_disposition(kwargs["disposition"], default=NEEDS_REVIEW),"confidence": kwargs["confidence"],- "author": kwargs["author"],+ # Recorded as human-confirmed, which the worker never does. An+ # AI prior can no longer suppress at all — that rule was added+ # because a corroboration threshold cannot help when the+ # attacker picks the count — so a human prior is now the only+ # kind that *can* leak, and modelling the leak with the author+ # the worker actually passes would make this file prove nothing+ # while still passing.+ "author": outcomes_module.HUMAN,"count": store.get(signature, {}).get("count", 0) + 1,+ # The TTL reads this. Omitting it makes every prior count as+ # expired, which disarms the leak the same way.+ "last_seen": datetime.now(UTC).isoformat(),}store[signature] = priorreturn prior