Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
"""Shadow mode on the live queue: triage runs, and nothing it says is acted on.
Gap-closure Phase 2.1.
Phase 1 built shadow execution for replay, over a customer's *closed* history:
:class:`app.replay.shadow.ShadowTriageWriter` implements every write the
triage path makes and performs none of them. This is the same seam pointed at
live alerts, and it differs in exactly one way that matters.
A replay writes nothing at all, because its findings are already closed and
anything it wrote would be noise in a store somebody else reads. A live shadow
run has to write two things: the verdict, so an analyst can see what the agent
thought, and a decision row, so agreement can be measured when that analyst
closes the alert. What it must not do is *act*: no disposition set on the
alert, no auto-closure, no writeback to the source SIEM, no approval queued,
no outcome prior recorded, no escalation into the investigation graph.
The one that is easy to get wrong
=================================
``ledger.persist_auto_triage`` writes ``alerts.disposition``. That column is
the analyst's verdict, set from the feedback endpoint, and it is also the
column reconciliation reads to learn what the analyst decided. Letting a
shadow verdict land in it would mean the agent filled in the answer it is
about to be graded against, and every alert an analyst did not explicitly
re-dispose would score as perfect agreement.
That is the Phase 1 leakage lesson arriving by a different route, and it is
the reason ``persist_auto_triage`` grew a ``shadow`` flag rather than this
module simply forwarding the call. Under that flag the ledger still writes the
run, the event and the ``ai_score`` / ``ai_summary`` /
``ai_recommendations`` columns, which are the agent's own fields, and leaves
``disposition``, ``status`` and ``resolved_at`` untouched. The analyst decides
from a blank field, which is the only way their closure is independent
evidence.
Cost is not suppressed
======================
:attr:`ShadowModeTriageWriter.persists_cost` is ``True``. A shadow run places
real model calls against a real key, and a tenant measuring for a month must
see that spend. Replay declines it because a replay is a measurement the
tenant asked for and should not be billed twice for; a shadow run is the
product running.
Deduplication is not suppressed either
======================================
:meth:`~ShadowModeTriageWriter.cache_verdict` forwards. Production dedups an
alert flood through the cost governor, so a shadow run that did not would cost
more than the thing it is modelling and would measure a pipeline the tenant
will never operate. Replay declines it for a reason that does not apply here:
in a replay the cache is a third route by which the test window answers
itself, and on a live queue there is no test window.
Which way it fails
==================
:class:`LiveShadowModePolicy` distinguishes "no database configured" from
"database configured and unreadable", the same distinction
``app.services.tenant_policy`` draws in the actions service and for the same
reason. No database means no shadow table, no tenant could have enabled
anything, and nowhere to record a decision: shadow is off. A configured
database that will not answer means a tenant may well have asked for shadow
and we cannot see it, so the run is treated as shadow. Acting on an alert a
tenant asked us only to observe is the worse of the two errors, and the other
one is a few seconds of writeback that resumes on the next read.
"""
from __future__ import annotations
import time
import uuid
from typing import Any, Protocol, runtime_checkable
import structlog
from app.investigator import ledger as ledger_module
__all__ = [
"UNCLASSIFIED",
"LiveShadowModePolicy",
"ShadowDecision",
"ShadowModePolicy",
"ShadowModeTriageWriter",
"alert_class_of",
"record_shadow_decision",
]
logger = structlog.get_logger(__name__)
#: Where an alert with no category lands. A null class would drop the alert
#: out of every per-class aggregate, so a tenant whose detection content sets
#: no category would appear to have no evidence at all rather than one bucket
#: of it.
UNCLASSIFIED = "unclassified"
#: Matches every class. A tenant starting out does not yet know which classes
#: their queue contains, which is most of what the first weeks of measurement
#: are for.
WILDCARD = "*"
#: Policy changes are rare and this is read once per alert, so a short TTL
#: keeps the common case to zero round trips while still picking up a change
#: within seconds. Same value and same reasoning as the actions service's
#: tenant policy cache.
_CACHE_TTL_SECONDS = 30.0
_cache: dict[tuple[str, str], tuple[float, bool]] = {}
def alert_class_of(raw_alert: dict[str, Any] | None) -> str:
"""The class this alert is measured under.
``alerts.category`` is what the console and the detection corpus already
group by, so shadow mode uses it rather than inventing a parallel
taxonomy that would have to be explained and kept in step.
"""
category = str((raw_alert or {}).get("category") or "").strip().lower()
return category or UNCLASSIFIED
@runtime_checkable
class ShadowModePolicy(Protocol):
"""Whether this tenant is measuring this alert class rather than acting on it."""
async def is_shadow(self, tenant_ref: str, alert_class: str) -> bool:
"""True when this alert is to be measured rather than acted on."""
class LiveShadowModePolicy:
"""Reads ``aisoc_shadow_mode``. Treats an unreadable policy as shadow."""
__slots__ = ()
async def is_shadow(self, tenant_ref: str, alert_class: str) -> bool:
key = (str(tenant_ref), alert_class)
cached = _cache.get(key)
now = time.monotonic()
if cached is not None and now - cached[0] < _CACHE_TTL_SECONDS:
return cached[1]
pool = await ledger_module.get_pool()
if pool is None:
# No database: nothing could have been enabled and there is
# nowhere to record a decision. Not cached, because a pool that
# appears later should take effect immediately.
return False
try:
async with pool.acquire() as conn:
tenant_id = await ledger_module._resolve_tenant_id(conn, str(tenant_ref))
if tenant_id is None:
# An unknown tenant cannot have configured anything, and a
# decision row for it would violate the foreign key.
logger.warning("shadow_mode.unknown_tenant", tenant_ref=str(tenant_ref)[:120])
return False
await ledger_module._set_rls_context(conn, tenant_id)
row = await conn.fetchrow(
"""
SELECT bool_or(enabled) AS enabled
FROM aisoc_shadow_mode
WHERE tenant_id = $1 AND alert_class = ANY($2::text[])
""",
tenant_id,
[alert_class, WILDCARD],
)
except Exception as exc: # noqa: BLE001 - never act on an unreadable policy
logger.error(
"shadow_mode.read_failed",
tenant_ref=str(tenant_ref)[:120],
alert_class=alert_class[:120],
error=str(exc)[:300],
treating_as="shadow",
)
return True
enabled = bool(row and row["enabled"])
_cache[key] = (now, enabled)
return enabled
def clear_cache() -> None:
"""Drop the shadow-mode cache. Used by tests and after a policy update."""
_cache.clear()
class ShadowDecision:
"""One verdict produced in shadow, on its way to the decision ledger.
A plain object rather than a dataclass mirroring the table: the columns it
fills are written out once in :func:`record_shadow_decision`, and a second
field list here would be the copy that drifts.
"""
__slots__ = (
"alert_class",
"alert_id",
"confidence",
"evidence_signature",
"external_id",
"model",
"rule_id",
"source",
"tenant_ref",
"verdict",
)
def __init__(
self,
*,
tenant_ref: str,
alert_class: str,
alert_id: Any = None,
external_id: str | None = None,
rule_id: str | None = None,
source: str | None = None,
model: str | None = None,
verdict: str | None = None,
confidence: float | None = None,
evidence_signature: str | None = None,
) -> None:
self.tenant_ref = tenant_ref
self.alert_class = alert_class
self.alert_id = alert_id
self.external_id = external_id
self.rule_id = rule_id
self.source = source
self.model = model
self.verdict = verdict
self.confidence = confidence
self.evidence_signature = evidence_signature
async def record_shadow_decision(decision: ShadowDecision) -> bool:
"""Append the verdict to ``aisoc_shadow_decisions``. Returns whether it landed.
Idempotent on ``(tenant_id, alert_id)``. A broker redelivery re-triages the
same alert, and two rows carrying the same opinion would count as two
agreements: the sample would grow toward the promotion threshold without
any new evidence behind it. The conflict updates rather than doing nothing,
so a re-triage that produced a *different* verdict replaces the old one
instead of leaving a stale opinion in the evidence.
The analyst half is never touched on conflict. A decision that has already
been graded keeps its grade.
"""
pool = await ledger_module.get_pool()
if pool is None:
return False
alert_uuid = ledger_module._coerce_uuid(decision.alert_id)
if alert_uuid is None and not decision.external_id:
# Nothing to reconcile against later. Recording it would add a row
# that can never become evidence and can never be matched to a
# closure, which reads on the scorecard as an agent that never gets
# graded rather than as a wiring fault.
logger.warning("shadow_mode.decision_unlinkable", tenant_ref=str(decision.tenant_ref)[:120])
return False
try:
async with pool.acquire() as conn:
tenant_id = await ledger_module._resolve_tenant_id(conn, str(decision.tenant_ref))
if tenant_id is None:
logger.warning("shadow_mode.decision_unknown_tenant", tenant_ref=str(decision.tenant_ref)[:120])
return False
await ledger_module._set_rls_context(conn, tenant_id)
await conn.execute(
"""
INSERT INTO aisoc_shadow_decisions
(id, tenant_id, alert_id, external_id, alert_class, rule_id,
source, model, verdict, confidence, evidence_signature, decided_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, now())
ON CONFLICT (tenant_id, alert_id) WHERE alert_id IS NOT NULL
DO UPDATE SET
verdict = EXCLUDED.verdict,
confidence = EXCLUDED.confidence,
model = EXCLUDED.model,
evidence_signature = EXCLUDED.evidence_signature,
decided_at = EXCLUDED.decided_at
""",
uuid.uuid4(),
tenant_id,
alert_uuid,
decision.external_id,
decision.alert_class or UNCLASSIFIED,
decision.rule_id,
decision.source,
decision.model,
decision.verdict,
float(decision.confidence) if decision.confidence is not None else None,
decision.evidence_signature,
)
except Exception as exc: # noqa: BLE001 - a durable verdict outranks its measurement
logger.warning(
"shadow_mode.decision_write_failed",
tenant_ref=str(decision.tenant_ref)[:120],
error=str(exc)[:300],
)
return False
return True
class ShadowModeTriageWriter:
"""A :class:`app.workers.triage_persistence.TriageWriter` that records and never acts.
Wraps the live sink rather than reimplementing it. The two calls that are
forwarded are forwarded to the real thing, so a shadow run and a live run
differ by which methods are declined and by one flag, not by two code
paths that have to be kept in step.
"""
def __init__(self, delegate: Any, decision: ShadowDecision) -> None:
self._delegate = delegate
self._decision = decision
#: Whether the decision row landed. Read by the worker's metrics so a
#: silently failing measurement shows up as a number rather than as a
#: scorecard that never fills in.
self.recorded = False
@property
def escalation_allowed(self) -> bool:
# Escalation runs the investigation graph, which enriches, proposes and
# writes a node per step. That is acting.
return False
@property
def persists_cost(self) -> bool:
# Real calls against a real key. A tenant measuring for a month must
# see the spend.
return True
@property
def fires_playbooks(self) -> bool:
# Shadow mode is measuring whether the agent *would* be right, before a
# tenant grants it autonomy. A playbook that ran would be the agent
# acting on a verdict nobody has accepted yet, which is the one thing
# the mode exists to withhold.
return False
@property
def uses_dedup_cache(self) -> bool:
# Production behaviour. A tenant measuring a real stream should see the
# same deduplication a live run would do, or the shadow scorecard
# describes a different workload from the one it is predicting.
return True
async def persist_auto_triage(self, **fields: Any) -> None:
"""Record the verdict without letting it stand in for the analyst's."""
fields["auto_closed"] = False
fields["shadow"] = True
await self._delegate.persist_auto_triage(**fields)
self._decision.verdict = str(fields.get("verdict") or "") or None
confidence = fields.get("confidence")
self._decision.confidence = float(confidence) if confidence is not None else None
raw = fields.get("raw_alert") or {}
if self._decision.rule_id is None:
self._decision.rule_id = _opt_text(raw.get("rule_id"))
if self._decision.source is None:
self._decision.source = _opt_text(raw.get("connector_type") or raw.get("source"))
if self._decision.external_id is None:
self._decision.external_id = _opt_text(raw.get("external_id"))
self.recorded = await record_shadow_decision(self._decision)
async def record_outcome(
self,
tenant_id: str,
signature: str,
*,
disposition: str,
confidence: float,
author: str,
alert_id: Any = None,
injection_suspected: bool = False,
) -> None:
# An outcome prior lets a later identical alert close without triage.
# A shadow verdict must never reach that far: it would act on a future
# alert through a store rather than directly, and it would put the
# agent's own opinion into the evidence for the next one.
return None
async def record_suppression(
self,
*,
tenant_ref: str,
signature: str,
alert_id: Any,
disposition: str,
prior_author: str,
) -> None:
return None
async def raise_approval(self, **fields: Any) -> Any:
# An approval is a proposal to act, put in front of a human. Under
# measurement there is nothing to approve.
return None
async def write_back_disposition(
self,
*,
tenant_id: str,
alert_id: str,
disposition: str,
confidence: float | None = None,
rationale: str = "",
) -> dict[str, Any] | None:
return None
def cache_verdict(
self,
governor: Any,
tenant_id: str,
fingerprint: str,
verdict: dict[str, Any],
*,
usd: float,
tokens: int,
) -> None:
self._delegate.cache_verdict(governor, tenant_id, fingerprint, verdict, usd=usd, tokens=tokens)
def _opt_text(value: Any) -> str | None:
"""A trimmed string, or ``None`` for absent and empty alike."""
text = str(value).strip() if value is not None else ""
return text or None