Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

fix(api,agents,mcp): four surfaces that reported success while doing nothing (#1280, #1275, #1276, #1274) (#1289)

Beenu Arorade54e6426ee7
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4eef9a5ad..ff0d53cda 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,63 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### BREAKING
+
+- **`PATCH /api/v1/alerts/{id}` now refuses a field it cannot apply.** The
+ request model set no `extra=`, so pydantic v2's default `extra="ignore"`
+ dropped anything outside `status`, `priority`, `tags`, `assignee` and
+ `case_id` and still answered **200** with the unchanged row — a success
+ response for a write that did not happen (#1280). The dropped fields were
+ not nonsense: `disposition`, `severity`, `confidence` and `snoozed_until`
+ are all real `alerts` columns and three of them are writable elsewhere,
+ which is why callers sent them. A caller closing an alert with a
+ disposition got a 200 and no disposition.
+
+ The endpoint now answers **422**, naming the field *and* the route that
+ does apply it — `POST /feedback/alert-override` for a disposition,
+ `POST /alerts/{id}/escalate` for severity, `POST /alerts/{id}/snooze` for
+ a snooze. The accepted surface is unchanged, including the `assignee`
+ alias the queue releases alerts with; only previously-ignored fields are
+ affected. The published spec gains `additionalProperties: false` on
+ `AlertUpdateRequest`, and `alertsApi.update` in the console is typed to
+ the five fields rather than `Partial<Alert>`, so a rejected call now fails
+ to compile instead of failing at runtime.
+
+### Fixed
+
+- **The copilot stopped inventing investigations, and stopped blaming a
+ missing API key for a model that timed out** (#1275). The template
+ fallback returned one of five rotating paragraphs asserting findings
+ about an estate it had never read — "this IP was seen in 3 other alerts",
+ "the attacker dwell time appears short (< 2 hours)" — and ignored the
+ question entirely, so a ransomware alert came back described as
+ "credential-access activity ... LOLBin pattern". That text is gone. The
+ two fallback conditions are also now distinguished: "no model is
+ configured" and "the configured model did not answer" used to produce one
+ message ending "Configure an LLM key", which is the wrong instruction on a
+ deployment running the bundled local model, where the usual cause is
+ contention with the auto-triage backlog. `CopilotChatResponse` carries an
+ additive `template_reason`; `source` is unchanged.
+- **Auto-triage records the prompt behind its verdict** (#1276). The path
+ that handles every fused alert wrote a verdict and nothing to check it
+ against: its hardcoded `INSERT` named neither `input_hash` nor
+ `output_hash`, `duration_ms` was the literal `0`, and no prompt was kept
+ anywhere — so the most common AI decision in the platform was the one that
+ could not be replayed, while `aisoc_explain_step` is documented as
+ returning "the prompt, response, and tools used". The model call is now a
+ `llm_call` event at seq 0 carrying both hashes and its measured duration,
+ with the literal prompt and response attached to the verdict event as
+ artifacts, which is what `/investigations/{run}/explain` inlines.
+ `graph_step` rows carry a measured duration instead of a hardcoded zero.
+- **The MCP `aisoc_run_hunt` tool works on a default install** (#1274). It
+ declared `tenant_id: z.string().uuid()`, and zod 4 enforces RFC 4122
+ version and variant bits that `00000000-0000-0000-0000-000000000001` —
+ the tenant every `make up` creates — does not set, so the tool was
+ uncallable on a stock deployment. The parameter is removed rather than
+ loosened: the tenant comes from the authenticated principal, and the API
+ handler was already dropping the field, so the isolation had been an
+ accident of schema strictness rather than a control.
+
### Fixed
- **The autonomy settings page announced an autopilot posture no deployment
diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.ts
index ad4517452..2e953b90b 100644
--- a/apps/web/src/lib/api.ts
+++ b/apps/web/src/lib/api.ts
@@ -1640,6 +1640,30 @@ export interface AlertFilters {
sort?: 'newest' | 'priority';
}
+/**
+ * The five fields `PATCH /api/v1/alerts/{id}` can actually apply.
+ *
+ * This was `Partial<Alert>`, which let the console send any of the row's
+ * ~40 columns. The server ignored everything outside this set and still
+ * answered 200, so `{ disposition }` or `{ severity }` compiled, ran and
+ * changed nothing (issue #1280). The server now refuses them; narrowing
+ * the type here makes the rejected call fail to compile instead, and
+ * `AlertUpdateRequest` on the Python side is the one definition this
+ * mirrors.
+ *
+ * Severity goes through `POST /alerts/{id}/escalate`, a disposition
+ * through `POST /feedback/alert-override`, and a snooze through
+ * `POST /alerts/{id}/snooze`.
+ */
+export interface AlertUpdate {
+ status?: AlertStatus;
+ priority?: number;
+ tags?: string[];
+ /** `null` releases the alert back to the pool. */
+ assignee?: string | null;
+ case_id?: string | null;
+}
+
export const alertsApi = {
list: async (filters: AlertFilters = {}) => {
const raw = await request<{
@@ -1682,7 +1706,7 @@ export const alertsApi = {
return normalizeAlert(raw);
},
- update: async (id: string, data: Partial<Alert>) => {
+ update: async (id: string, data: AlertUpdate) => {
const raw = await request<unknown>(`/api/v1/alerts/${id}`, {
method: 'PATCH',
body: JSON.stringify(data),
diff --git a/docs/openapi.yaml b/docs/openapi.yaml
index 16054a826..aaa9c8d41 100644
--- a/docs/openapi.yaml
+++ b/docs/openapi.yaml
@@ -20976,6 +20976,7 @@ components:
format: uuid
- type: 'null'
title: Case Id
+ additionalProperties: false
type: object
title: AlertUpdateRequest
ApiKeyOut:
diff --git a/services/agents/app/agents/auto_triage_agent.py b/services/agents/app/agents/auto_triage_agent.py
index db2b78a4c..9bbcdf10f 100644
--- a/services/agents/app/agents/auto_triage_agent.py
+++ b/services/agents/app/agents/auto_triage_agent.py
@@ -432,14 +432,31 @@ async def run_auto_triage(state: InvestigationState) -> InvestigationState:
# constructing outside would have failed the whole graph run over a
# configuration problem the deterministic path handles fine.
llm = make_chat_model("triage", temperature=0.0, max_tokens=512, json_output=True)
+ system_prompt = prompt_text("triage.system") + "\n\n" + system_rule(nonce)
response = await safe_ainvoke(
llm,
[
- SystemMessage(content=prompt_text("triage.system") + "\n\n" + system_rule(nonce)),
+ SystemMessage(content=system_prompt),
HumanMessage(content=alert_context),
],
)
raw_text = response.content
+ # Recorded before parsing, so a verdict that fails to parse still
+ # leaves behind what the model was asked and what it said. The
+ # ledger carries this onto the run (issue #1276): auto-triage is
+ # the path that handles every fused alert, and it was the one
+ # path whose decisions could not be replayed.
+ state.record_llm_exchange(
+ agent="auto_triage",
+ purpose="triage.classify",
+ model=getattr(llm, "model_name", None) or getattr(llm, "model", None),
+ prompt=[
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": alert_context},
+ ],
+ response=str(raw_text),
+ duration_ms=round((time.monotonic() - t0) * 1000),
+ )
result = _parse_llm_response(raw_text)
except Exception as exc:
# Issue #571: do NOT swallow + return a null-verdict RUNNING state.
diff --git a/services/agents/app/api/copilot.py b/services/agents/app/api/copilot.py
index a06acec1e..a3fe38375 100644
--- a/services/agents/app/api/copilot.py
+++ b/services/agents/app/api/copilot.py
@@ -8,14 +8,14 @@ Endpoints (all under ``/api/v1/copilot``):
POST /chat — one-shot chat (creates / continues conv.)
POST /chat/stream — streaming NDJSON variant
-Falls back to synthetic deterministic replies when ``OPENAI_API_KEY`` is
-unset so the demo path never breaks.
+When no model answers, the reply says so and names which of the two
+reasons applies — nothing is configured, or what is configured did not
+return. It does not describe the tenant's data, because nothing read it.
"""
from __future__ import annotations
import asyncio
-import itertools
import json
import uuid
from collections.abc import AsyncIterator
@@ -71,6 +71,11 @@ class CopilotChatResponse(BaseModel):
#: The console must label a ``template`` reply rather than presenting it as
#: analysis of the user's environment.
source: Literal["llm", "template"] = "llm"
+ #: Why no model answered, when ``source`` is ``template``. Additive
+ #: rather than a third ``source`` value, so existing clients branching
+ #: on ``template`` keep working while a new one can tell "nothing is
+ #: configured" from "what is configured did not answer" (issue #1275).
+ template_reason: Literal["no_model_configured", "model_call_failed"] | None = None
notice: str | None = None
#: Which factual claims in `reply` cite a record, and which cite
#: nothing (parity 3.6). The console renders the label beside the
@@ -113,39 +118,49 @@ def _tenant_of(principal: TenantPrincipal) -> uuid.UUID:
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=str(exc)) from exc
-_SYNTHETIC_REPLIES = [
- (
- "I've analysed the alert context. The activity matches T1078 (Valid Accounts) combined"
- " with T1021.002 (SMB/Windows Admin Shares) lateral movement. Recommend isolating the"
- " host and reviewing recent authentication logs."
- ),
- (
- "Based on the indicators, this looks like credential-access activity. The parent process"
- " chain suggests a LOLBin pattern. Consider adding a detection rule for this specific"
- " chain."
- ),
- (
- "The entity risk score is elevated due to multiple failed authentications followed by a"
- " successful login from an unusual geolocation. I recommend triggering a step-up MFA"
- " challenge."
- ),
- (
- "Correlation across the last 24 hours shows this IP was seen in 3 other alerts. The MITRE"
- " mapping points to T1110 (Brute Force). Blocking the IP at the perimeter is the fastest"
- " remediation."
- ),
- (
- "I've reviewed the case timeline. The attacker dwell time appears short (< 2 hours),"
- " suggesting this may be an automated credential-stuffing campaign rather than a targeted"
- " intrusion."
- ),
-]
-
-_reply_cycle = itertools.cycle(_SYNTHETIC_REPLIES)
-
-
-def _synthetic_reply(user_msg: str) -> str:
- return next(_reply_cycle)
+#: Why no model answered. These are two different operator problems and
+#: the old code could not tell them apart (issue #1275).
+NO_MODEL = "no_model_configured"
+CALL_FAILED = "model_call_failed"
+
+
+def _fallback_reply(reason: str, *, model: str | None = None, detail: str | None = None) -> str:
+ """What to say when no model answered.
+
+ This used to return one of five rotating paragraphs asserting findings
+ about an estate it had never seen — "this IP was seen in 3 other
+ alerts", "the attacker dwell time appears short (< 2 hours)", "multiple
+ failed authentications followed by a successful login from an unusual
+ geolocation". None of it was computed from anything. The reporter hit
+ exactly that: a ransomware alert described back to them as
+ "credential-access activity ... LOLBin pattern", because the function
+ ignored its argument and returned the next item in the cycle.
+
+ A canned paragraph that reads like analysis is worse than no answer,
+ and the console appending a disclaimer underneath does not fix the
+ body. So the reply is now the explanation, and it makes no claim about
+ the tenant's data.
+ """
+ if reason == NO_MODEL:
+ return (
+ "I can't answer this: no language model is configured for this deployment.\n\n"
+ "Nothing about your alerts, cases or detections was looked at, so treat this as "
+ "an unanswered question rather than a finding.\n\n"
+ "To enable the copilot, configure a model — either a hosted provider key or the "
+ "bundled local model — and ask again."
+ )
+
+ named = f" ({model})" if model else ""
+ because = f"\n\nThe call reported: {detail}" if detail else ""
+ return (
+ f"I can't answer this: the configured model{named} did not return a reply.{because}\n\n"
+ "Nothing about your alerts, cases or detections was looked at, so treat this as an "
+ "unanswered question rather than a finding.\n\n"
+ "This is a model or gateway problem, not a missing credential — a model is configured. "
+ "On a CPU-only deployment the usual cause is contention with the auto-triage backlog, "
+ "which holds the same local model. Retrying usually works; if it does not, check the "
+ "model service's own logs."
+ )
# ---------------------------------------------------------------------------
@@ -157,17 +172,48 @@ def _title_from_message(msg: str) -> str:
return msg[:60] + ("…" if len(msg) > 60 else "")
+def _notice_for(reason: str | None) -> str | None:
+ """The one-line banner the console puts above a template reply.
+
+ One string served both fallback conditions and ended "Configure an LLM
+ key to get a real investigation". On a deployment running the bundled
+ local model that sentence names a credential the deployment does not
+ use, so an operator whose model had merely timed out was sent to fix
+ something that was not broken.
+ """
+ if reason == NO_MODEL:
+ return "No language model is configured, so nothing answered this question. This is not analysis of your environment."
+ if reason == CALL_FAILED:
+ return (
+ "A model is configured but the call did not return, so nothing answered this "
+ "question. This is not analysis of your environment, and it is not a missing "
+ "API key."
+ )
+ return None
+
+
async def _get_openai_reply(
conversation: dict[str, Any],
user_message: str,
-) -> tuple[str, str]:
- """Return ``(reply_text, source)`` where source is ``llm`` or ``template``.
+) -> tuple[str, str, str | None]:
+ """Return ``(reply_text, source, reason)``.
+
+ ``source`` is ``llm`` or ``template``; ``reason`` is ``None`` on the
+ model path and otherwise names *why* no model answered.
The caller must surface ``template`` to the user. This function silently
returned a canned paragraph as a normal 200 whenever the key was missing or
any exception fired, so an analyst read "this IP was seen in 3 other
alerts" as real analysis of their environment. The frontend had an honest
fallback of its own that never fired, because the backend reported success.
+
+ The two fallback conditions are now distinguished (issue #1275). They
+ used to collapse into one message telling the operator to configure an
+ API key — advice that is simply wrong when a local model is configured
+ and timed out, which is the common case on CPU. This repository has the
+ same lesson recorded from the RBA banner that named fusion while fusion
+ was healthy: a diagnostic pointing at the wrong subsystem sends someone
+ to fix something that is not broken, and is worse than a vague one.
"""
from app.llm.factory import resolve_api_key, resolve_model_alias
@@ -176,7 +222,7 @@ async def _get_openai_reply(
# goes to the bundled gateway the bearer has to be the gateway's master key.
api_key = resolve_api_key(model) or ""
if not api_key:
- return _synthetic_reply(user_message), "template"
+ return _fallback_reply(NO_MODEL), "template", NO_MODEL
try:
from app.llm.contract import safe_chat_completions_request
@@ -204,10 +250,14 @@ async def _get_openai_reply(
url=chat_completions_url(model),
max_tokens=512,
)
- return body["choices"][0]["message"]["content"], "llm"
+ return body["choices"][0]["message"]["content"], "llm", None
except Exception as exc:
- logger.warning("copilot.openai_error", error=str(exc))
- return _synthetic_reply(user_message), "template"
+ # `str(exc)` is empty for several httpx timeout classes, which is
+ # how the report ended up with `copilot.openai_error` carrying no
+ # message at all. The class name always says something.
+ detail = str(exc).strip() or type(exc).__name__
+ logger.warning("copilot.openai_error", error=detail, model=model)
+ return _fallback_reply(CALL_FAILED, model=model, detail=detail), "template", CALL_FAILED
# ---------------------------------------------------------------------------
@@ -266,7 +316,7 @@ async def chat(
}
history = {"messages": [*(existing.messages if existing else []), user_msg]}
- reply_text, reply_source = await _get_openai_reply(history, req.message)
+ reply_text, reply_source, reply_reason = await _get_openai_reply(history, req.message)
assistant_msg: dict[str, Any] = {
"id": str(uuid.uuid4()),
@@ -295,14 +345,9 @@ async def chat(
conversationId=stored.id,
reply=CopilotMessage(**assistant_msg),
source=reply_source,
+ template_reason=reply_reason,
grounding=grounding,
- notice=(
- "This reply came from a built-in template, not a language model. "
- "It is generic guidance and is not analysis of your environment. "
- "Configure an LLM key to get a real investigation."
- if reply_source == "template"
- else None
- ),
+ notice=_notice_for(reply_reason),
)
@@ -328,7 +373,7 @@ async def chat_stream(
}
history = {"messages": [*(existing.messages if existing else []), user_msg]}
- reply_text, reply_source = await _get_openai_reply(history, req.message)
+ reply_text, reply_source, reply_reason = await _get_openai_reply(history, req.message)
msg_id = str(uuid.uuid4())
assistant_msg: dict[str, Any] = {
"id": msg_id,
@@ -354,7 +399,18 @@ async def chat_stream(
async def _stream() -> AsyncIterator[bytes]:
# Provenance first: a consumer must be able to label the answer before
# it starts rendering tokens, not after.
- yield (json.dumps({"source": reply_source, "delta": "", "done": False}) + "\n").encode()
+ yield (
+ json.dumps(
+ {
+ "source": reply_source,
+ "template_reason": reply_reason,
+ "notice": _notice_for(reply_reason),
+ "delta": "",
+ "done": False,
+ }
+ )
+ + "\n"
+ ).encode()
words = reply_text.split(" ")
for i, word in enumerate(words):
chunk = word + (" " if i < len(words) - 1 else "")
diff --git a/services/agents/app/graph/runner.py b/services/agents/app/graph/runner.py
index 4ef86f32e..07d024899 100644
--- a/services/agents/app/graph/runner.py
+++ b/services/agents/app/graph/runner.py
@@ -20,6 +20,7 @@ from __future__ import annotations
import asyncio
import os
+import time
from dataclasses import dataclass
import structlog
@@ -97,6 +98,12 @@ async def _run(
seq = seq_start
timed_out = False
budget_reason: str | None = None
+ # Wall clock for the step about to be streamed. Every `graph_step` row
+ # carried `duration_ms: 0` because the argument was simply never passed
+ # (issue #1276), so the ledger reported that each node of an escalation
+ # took no time — a confident zero for a measurement nobody made, which
+ # is the shape this project treats as worse than an absent one.
+ step_started = time.monotonic()
try:
async with asyncio.timeout(budget.max_seconds):
async for step in graph.astream(state_dict):
@@ -106,6 +113,9 @@ async def _run(
seq += 1
if isinstance(out, dict):
merged.update(out)
+ now = time.monotonic()
+ elapsed_ms = round((now - step_started) * 1000)
+ step_started = now
if tenant_uuid is not None:
await ledger_module.record_event(
run_id=state.run_id,
@@ -115,6 +125,7 @@ async def _run(
agent=str(node),
summary=f"graph node '{node}' completed",
payload={"node": str(node)},
+ duration_ms=elapsed_ms,
)
# Checked **inside** the loop, once per streamed step, after
diff --git a/services/agents/app/investigator/ledger.py b/services/agents/app/investigator/ledger.py
index 50843b2ff..b4dd4a77f 100644
--- a/services/agents/app/investigator/ledger.py
+++ b/services/agents/app/investigator/ledger.py
@@ -408,6 +408,23 @@ async def complete_run(
)
+def _inserted_a_row(status: Any) -> bool:
+ """True when an asyncpg command status reports a row was written.
+
+ ``conn.execute`` returns the tag, e.g. ``"INSERT 0 1"`` for a write and
+ ``"INSERT 0 0"`` when ``ON CONFLICT DO NOTHING`` suppressed it. That is
+ the only signal distinguishing a first write from a replay, and the
+ artifacts have no unique key of their own to lean on.
+ """
+ try:
+ return int(str(status).rsplit(" ", 1)[-1]) > 0
+ except (ValueError, IndexError):
+ # An unrecognised tag means "do not know". Writing nothing is the
+ # safe side: a missing transcript is visibly missing, a duplicated
+ # one silently doubles every replayed run.
+ return False
+
+
async def fail_stale_runs(
*,
older_than_seconds: float,
@@ -511,6 +528,7 @@ async def persist_auto_triage(
groundedness: float | None = None,
ungrounded: bool | None = None,
shadow: bool = False,
+ llm_exchanges: list[dict[str, Any]] | None = None,
) -> bool:
"""Durably record an auto-triage outcome (issue #571) in ONE transaction:
@@ -571,16 +589,56 @@ async def persist_auto_triage(
json.dumps(raw_alert or {}),
f"kafka:auto_triage:{tier}",
)
- await conn.execute(
+ # The model call that produced the verdict, at seq 0 so it
+ # precedes it (issue #1276). This path recorded no prompt at
+ # all, so the decision taken on every fused alert was the one
+ # decision in the platform that could not be replayed.
+ #
+ # seq 0 rather than renumbering: seq 1 is the verdict's key
+ # under `ON CONFLICT (run_id, seq)`, the run id is
+ # deterministic, and moving the verdict would let a replay
+ # write a second one. Escalation graph steps start at seq 2
+ # (`run_escalation(..., seq_start=1)`), so 0 is the free slot
+ # and it is also the one the manual path puts its first
+ # prompt in.
+ for exchange in (llm_exchanges or [])[:1]:
+ await conn.execute(
+ """
+ INSERT INTO investigation_events
+ (id, run_id, tenant_id, seq, ts, kind, agent, summary,
+ payload, input_hash, output_hash, duration_ms, created_at)
+ VALUES
+ ($1, $2, $3, 0, now(), 'llm_call', $4, $5, $6::jsonb, $7, $8, $9, now())
+ ON CONFLICT (run_id, seq) DO NOTHING
+ """,
+ uuid.uuid4(),
+ run_id,
+ tenant_id,
+ str(exchange.get("agent") or "auto_triage"),
+ f"model call: {exchange.get('purpose') or 'triage'}"[:8000],
+ json.dumps(exchange),
+ exchange.get("input_hash"),
+ exchange.get("output_hash"),
+ int(exchange.get("duration_ms") or 0),
+ )
+
+ verdict_event_id = uuid.uuid4()
+ # The hashes travel onto the verdict too. An auditor asking
+ # "what produced this verdict" reads the verdict row, and a
+ # null `input_hash` there is indistinguishable from a run that
+ # never called a model — which is a real state on the
+ # deterministic path and must stay tellable apart.
+ first = (llm_exchanges or [{}])[0]
+ written = await conn.execute(
"""
INSERT INTO investigation_events
(id, run_id, tenant_id, seq, ts, kind, agent, summary,
- payload, duration_ms, created_at)
+ payload, input_hash, output_hash, duration_ms, created_at)
VALUES
- ($1, $2, $3, 1, now(), 'triage_verdict', $4, $5, $6::jsonb, 0, now())
+ ($1, $2, $3, 1, now(), 'triage_verdict', $4, $5, $6::jsonb, $7, $8, $9, now())
ON CONFLICT (run_id, seq) DO NOTHING
""",
- uuid.uuid4(),
+ verdict_event_id,
run_id,
tenant_id,
f"auto_triage:{tier}",
@@ -595,7 +653,53 @@ async def persist_auto_triage(
"auto_closed": auto_closed,
}
),
+ first.get("input_hash"),
+ first.get("output_hash"),
+ int(first.get("duration_ms") or 0),
)
+
+ # The literal transcript, attached to the verdict event.
+ # `GET /investigations/{run}/explain?step=1` inlines the
+ # artifacts of the focal event, which is how the MCP
+ # `aisoc_explain_step` tool keeps its documented promise to
+ # return "the prompt, response, and tools used".
+ #
+ # Written inline rather than through `record_artifact`
+ # because that acquires its own connection, and a transcript
+ # committed separately from the verdict it explains can
+ # survive a rollback of the verdict.
+ #
+ # Skipped entirely when the verdict row already existed.
+ # `investigation_artifacts` has no natural key to conflict
+ # on, so without this a replay of a deterministic run id
+ # would append a second copy of the transcript on every
+ # pass while the events it explains stayed at one.
+ if not _inserted_a_row(written):
+ llm_exchanges = []
+ for exchange in llm_exchanges or []:
+ for kind, text in (
+ ("llm_prompt", exchange.get("prompt")),
+ ("llm_response", exchange.get("response")),
+ ):
+ if not text:
+ continue
+ blob = str(text)
+ await conn.execute(
+ """
+ INSERT INTO investigation_artifacts
+ (id, run_id, event_id, tenant_id, kind, content,
+ sha256, size_bytes, created_at)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now())
+ """,
+ uuid.uuid4(),
+ run_id,
+ verdict_event_id,
+ tenant_id,
+ kind,
+ blob,
+ hashlib.sha256(blob.encode("utf-8")).hexdigest(),
+ len(blob.encode("utf-8")),
+ )
await conn.execute(
# `total_cost_usd` is measured cost only. It used to carry
# a list-price guess keyed on a gateway alias, so a local
diff --git a/services/agents/app/models/state.py b/services/agents/app/models/state.py
index 0e85fa9ec..a2ca84500 100644
--- a/services/agents/app/models/state.py
+++ b/services/agents/app/models/state.py
@@ -4,6 +4,8 @@ Agent state models for LangGraph workflows.
from __future__ import annotations
+import hashlib
+import json
from datetime import datetime
from enum import Enum
from typing import Any
@@ -11,6 +13,14 @@ from uuid import UUID, uuid4
from pydantic import BaseModel, Field
+#: Ceiling on the literal prompt / response kept beside a hash. Matches
+#: `InvestigatorState`'s, so the two paths truncate at the same point.
+_LLM_LITERAL_LIMIT = 8000
+
+
+def _stable_hash(value: str) -> str:
+ return hashlib.sha256(value.encode("utf-8")).hexdigest()
+
class AgentTask(str, Enum):
TRIAGE = "triage"
@@ -169,6 +179,19 @@ class InvestigationState(BaseModel):
# pivoted five times. This is what the depth gate grades.
investigation_depth: dict[str, Any] | None = None
+ # The literal model calls this run made, in order (issue #1276).
+ #
+ # The auto-triage ledger recorded a verdict and nothing a reader could
+ # check it against: no prompt, and `input_hash`/`output_hash` null on
+ # every step. A verdict with no recorded prompt cannot be replayed, and
+ # replay is what `aisoc_explain_step` is documented to provide.
+ #
+ # The convention is the manual path's, not a second one: the hash is
+ # over the full text and is always present, while the literal copy is
+ # truncated, because these payloads carry customer data. Written by
+ # `record_llm_exchange`; read by `ledger.persist_auto_triage`.
+ llm_exchanges: list[dict[str, Any]] = Field(default_factory=list)
+
# Metadata
iteration_count: int = 0
max_iterations: int = 10
@@ -180,5 +203,42 @@ class InvestigationState(BaseModel):
def add_finding(self, finding: str) -> None:
self.findings.append(finding)
+ def record_llm_exchange(
+ self,
+ *,
+ agent: str,
+ purpose: str,
+ model: str | None,
+ prompt: str | list[dict[str, Any]],
+ response: str,
+ duration_ms: int,
+ ) -> dict[str, Any]:
+ """Record one model call so the verdict it produced can be replayed.
+
+ Mirrors ``InvestigatorState.log_llm_prompt`` /
+ ``log_llm_response`` on the manual path: a SHA-256 over the full
+ text always, a truncated literal for a human reader. The two
+ differ only in shape — the manual path appends two audit entries,
+ this one records a single call with both of its hashes, because
+ the ledger row it becomes carries ``input_hash`` and
+ ``output_hash`` on the same event.
+ """
+ prompt_text = prompt if isinstance(prompt, str) else json.dumps(prompt, sort_keys=True, default=str)
+ exchange = {
+ "agent": agent,
+ "purpose": purpose,
+ "model": model,
+ "prompt": prompt_text[:_LLM_LITERAL_LIMIT],
+ "prompt_truncated": len(prompt_text) > _LLM_LITERAL_LIMIT,
+ "response": response[:_LLM_LITERAL_LIMIT],
+ "response_truncated": len(response) > _LLM_LITERAL_LIMIT,
+ "input_hash": _stable_hash(prompt_text),
+ "output_hash": _stable_hash(response),
+ "duration_ms": int(duration_ms),
+ "timestamp": datetime.utcnow().isoformat(),
+ }
+ self.llm_exchanges.append(exchange)
+ return exchange
+
def to_dict(self) -> dict:
return self.model_dump(mode="json")
diff --git a/services/agents/app/workers/fused_alert_consumer.py b/services/agents/app/workers/fused_alert_consumer.py
index b9a0af14b..8d172d9df 100644
--- a/services/agents/app/workers/fused_alert_consumer.py
+++ b/services/agents/app/workers/fused_alert_consumer.py
@@ -1289,6 +1289,10 @@ class FusedAlertTriageWorker:
# only inside a findings string.
groundedness=state.groundedness,
ungrounded=(state.groundedness is not None and state.groundedness < _groundedness_floor()),
+ # The prompt and response behind the verdict (issue #1276).
+ # Empty on the deterministic path, which is the honest value
+ # there — no model was asked, so there is nothing to replay.
+ llm_exchanges=list(state.llm_exchanges),
)
@staticmethod
diff --git a/services/agents/tests/test_auto_triage_ledger_records_the_prompt.py b/services/agents/tests/test_auto_triage_ledger_records_the_prompt.py
new file mode 100644
index 000000000..98abdbf90
--- /dev/null
+++ b/services/agents/tests/test_auto_triage_ledger_records_the_prompt.py
@@ -0,0 +1,369 @@
+"""Auto-triage records what it asked the model, not only what it decided.
+
+Issue #1276. The launch claim is that investigations are replayable "with
+prompts, tool calls and evidence captured", and the MCP tool
+``aisoc_explain_step`` is documented as returning "the prompt, response,
+and tools used". Both held on the manual path and neither held on the one
+that handles every fused alert.
+
+Three concrete gaps, all reproduced below:
+
+* ``run_auto_triage`` discarded the prompt. ``log_llm_prompt`` exists and
+ works, but its four callers are all manual agents — no worker called it,
+ so the most common AI decision in the platform left nothing to replay.
+* ``persist_auto_triage``'s hardcoded ``INSERT`` named neither
+ ``input_hash`` nor ``output_hash``, so both landed NULL, and
+ ``duration_ms`` was the literal ``0``.
+* ``graph/runner.py`` called ``record_event`` without a duration, so every
+ ``graph_step`` row also reported zero.
+
+A zero duration and a null hash are not missing data in the harmless
+sense: they read as measurements. "This step took no time" is a claim.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+from uuid import uuid4
+
+import pytest
+from app.models.state import InvestigationState
+
+
+def _state() -> InvestigationState:
+ return InvestigationState(incident_id=uuid4(), tenant_id=uuid4())
+
+
+class TestTheExchangeIsRecorded:
+ def test_a_model_call_leaves_a_replayable_record(self) -> None:
+ state = _state()
+ state.record_llm_exchange(
+ agent="auto_triage",
+ purpose="triage.classify",
+ model="llama3.2:3b",
+ prompt=[{"role": "system", "content": "classify this"}],
+ response='{"verdict": "false_positive"}',
+ duration_ms=1234,
+ )
+
+ assert len(state.llm_exchanges) == 1
+ recorded = state.llm_exchanges[0]
+ assert "classify this" in recorded["prompt"]
+ assert recorded["response"] == '{"verdict": "false_positive"}'
+ assert recorded["duration_ms"] == 1234
+ assert recorded["model"] == "llama3.2:3b"
+
+ def test_both_hashes_are_present_and_differ(self) -> None:
+ """The two NULL columns from the report, from the other end."""
+ state = _state()
+ recorded = state.record_llm_exchange(
+ agent="auto_triage",
+ purpose="triage.classify",
+ model=None,
+ prompt="ask",
+ response="answer",
+ duration_ms=1,
+ )
+
+ assert len(recorded["input_hash"]) == 64
+ assert len(recorded["output_hash"]) == 64
+ assert recorded["input_hash"] != recorded["output_hash"]
+
+ def test_the_hash_covers_the_whole_prompt_not_the_truncated_copy(self) -> None:
+ """The convention the manual path uses, and the reason for it: the
+ literal is bounded because it carries customer data, so a hash over
+ the bounded copy would verify the wrong thing."""
+ long_prompt = "x" * 9000
+ a = _state().record_llm_exchange(agent="a", purpose="p", model=None, prompt=long_prompt, response="r", duration_ms=0)
+ b = _state().record_llm_exchange(agent="a", purpose="p", model=None, prompt=long_prompt + "DIFFERENT", response="r", duration_ms=0)
+
+ assert a["prompt"] == b["prompt"], "both literals truncate to the same 8000 chars"
+ assert a["input_hash"] != b["input_hash"], "the hashes must still tell them apart"
+ assert a["prompt_truncated"] is True
+
+ def test_a_run_that_called_no_model_records_nothing(self) -> None:
+ """The deterministic path must stay distinguishable from a model
+ call whose prompt went missing. An empty list is the honest value."""
+ assert _state().llm_exchanges == []
+
+ def test_the_record_survives_serialisation_through_the_graph(self) -> None:
+ """LangGraph round-trips the state as a dict."""
+ state = _state()
+ state.record_llm_exchange(agent="auto_triage", purpose="p", model=None, prompt="ask", response="answer", duration_ms=5)
+ assert state.to_dict()["llm_exchanges"][0]["response"] == "answer"
+
+
+class TestTheWorkerPathActuallyCallsIt:
+ """The mechanism existing is not the claim. ``log_llm_prompt`` existed,
+ worked and was unit-tested the whole time — and its four callers were
+ all manual agents, so nothing on the path that handles every fused
+ alert ever reached it. Drive the real ``run_auto_triage``."""
+
+ @pytest.mark.asyncio
+ async def test_a_real_triage_run_records_its_prompt(self, monkeypatch: Any) -> None:
+ from types import SimpleNamespace
+
+ from app.agents import auto_triage_agent as ata
+
+ payload = json.dumps({"verdict": "false_positive", "confidence": 0.93, "rationale": "scanner"})
+ seen: list[Any] = []
+
+ async def _fake_ainvoke(_llm: Any, messages: Any) -> Any:
+ seen.append(messages)
+ return SimpleNamespace(content=payload)
+
+ monkeypatch.setattr(ata, "make_chat_model", lambda *a, **k: object())
+ monkeypatch.setattr(ata, "safe_ainvoke", _fake_ainvoke)
+
+ state = InvestigationState(
+ incident_id=uuid4(),
+ tenant_id=uuid4(),
+ alert_summary="Port scan from 10.0.0.5",
+ raw_alert={"severity": "medium", "hostname": "WEB-01"},
+ )
+ out = await ata.run_auto_triage(state)
+
+ assert out.verdict == "false_positive"
+ assert len(out.llm_exchanges) == 1, "the verdict was reached with no recorded prompt"
+
+ recorded = out.llm_exchanges[0]
+ # What was recorded is what was sent, not a reconstruction of it.
+ sent = json.dumps([{"role": m.type, "content": m.content} for m in seen[0]], default=str)
+ assert "WEB-01" in recorded["prompt"], "the alert context is missing from the record"
+ assert "WEB-01" in sent
+ assert recorded["response"] == payload
+ assert recorded["duration_ms"] >= 0
+ assert len(recorded["input_hash"]) == 64
+
+ @pytest.mark.asyncio
+ async def test_a_response_that_fails_to_parse_still_leaves_a_record(self, monkeypatch: Any) -> None:
+ """The case where a transcript is worth most: a parse failure whose
+ message is a character offset into text nobody kept."""
+ from types import SimpleNamespace
+
+ from app.agents import auto_triage_agent as ata
+
+ async def _fake_ainvoke(_llm: Any, _messages: Any) -> Any:
+ return SimpleNamespace(content="not json at all")
+
+ monkeypatch.setattr(ata, "make_chat_model", lambda *a, **k: object())
+ monkeypatch.setattr(ata, "safe_ainvoke", _fake_ainvoke)
+
+ state = InvestigationState(incident_id=uuid4(), tenant_id=uuid4(), alert_summary="x")
+ try:
+ await ata.run_auto_triage(state)
+ except Exception:
+ # Swallowed on purpose: the subject here is what the agent
+ # *recorded* before the unparseable reply took it down, asserted
+ # below. Whether it raises or falls back is a different question
+ # and a different test; letting either outcome through keeps this
+ # one from failing for a reason it is not about.
+ pass
+
+ assert state.llm_exchanges, "nothing recorded for the call that failed to parse"
+ assert state.llm_exchanges[0]["response"] == "not json at all"
+
+
+class TestTheLedgerWritesIt:
+ """Drives the real ``persist_auto_triage`` against a connection that
+ records what it was asked to execute. The assertion is on the SQL and
+ the bound parameters, because the defect *was* the column list — a
+ double that accepted any INSERT would reproduce the bug happily."""
+
+ @staticmethod
+ async def _run(exchanges: list[dict[str, Any]] | None) -> list[tuple[str, tuple]]:
+ from app.investigator import ledger as ledger_module
+
+ executed: list[tuple[str, tuple]] = []
+ tenant = uuid4()
+
+ class _Conn:
+ async def execute(self, sql: str, *args: Any) -> str:
+ executed.append((sql, args))
+ return "INSERT 0 1"
+
+ async def fetchval(self, *_a: Any, **_k: Any) -> Any:
+ return tenant
+
+ def transaction(self) -> Any:
+ return _Txn()
+
+ class _Txn:
+ async def __aenter__(self) -> None:
+ return None
+
+ async def __aexit__(self, *_a: Any) -> bool:
+ return False
+
+ class _Acquire:
+ async def __aenter__(self) -> _Conn:
+ return _Conn()
+
+ async def __aexit__(self, *_a: Any) -> bool:
+ return False
+
+ class _Pool:
+ def acquire(self) -> _Acquire:
+ return _Acquire()
+
+ async def _pool() -> _Pool:
+ return _Pool()
+
+ async def _resolve(_conn: Any, _ref: str) -> Any:
+ return tenant
+
+ # Through `MonkeyPatch` rather than a hand-rolled save/assign/restore.
+ # Assigning the module attribute directly is a rebind CodeQL can see,
+ # and it makes `py/import-of-mutable-attribute` fire on
+ # `test_ledger_tenant.py`, which imports `_resolve_tenant_id` by value
+ # and would therefore keep the original — a real hazard, just not one
+ # this test was trying to create.
+ with pytest.MonkeyPatch.context() as mp:
+ mp.setattr(ledger_module, "get_pool", _pool)
+ mp.setattr(ledger_module, "_resolve_tenant_id", _resolve)
+ await ledger_module.persist_auto_triage(
+ run_id=uuid4(),
+ alert_id=str(uuid4()),
+ tenant_ref=str(tenant),
+ alert_summary="s",
+ raw_alert={},
+ tier="L1",
+ verdict="false_positive",
+ confidence=0.9,
+ rationale="because",
+ llm_exchanges=exchanges,
+ )
+ return executed
+
+ @pytest.mark.asyncio
+ async def test_the_verdict_insert_names_both_hash_columns(self) -> None:
+ executed = await self._run([])
+ verdict_sql = [s for s, _ in executed if "triage_verdict" in s]
+ assert verdict_sql, "no verdict row was written"
+ assert "input_hash" in verdict_sql[0]
+ assert "output_hash" in verdict_sql[0]
+
+ @pytest.mark.asyncio
+ async def test_the_verdict_no_longer_hardcodes_a_zero_duration(self) -> None:
+ executed = await self._run([])
+ verdict_sql = [s for s, _ in executed if "triage_verdict" in s][0]
+ # The original literal was `..., $6::jsonb, 0, now())`.
+ assert ", 0, now())" not in verdict_sql
+
+ @pytest.mark.asyncio
+ async def test_the_prompt_is_written_as_its_own_step(self) -> None:
+ exchange = {
+ "agent": "auto_triage",
+ "purpose": "triage.classify",
+ "prompt": "the literal prompt",
+ "response": "the literal response",
+ "input_hash": "a" * 64,
+ "output_hash": "b" * 64,
+ "duration_ms": 742,
+ }
+ executed = await self._run([exchange])
+
+ llm_steps = [(s, a) for s, a in executed if "'llm_call'" in s]
+ assert len(llm_steps) == 1, "the model call left no step of its own"
+ sql, args = llm_steps[0]
+ assert ", 0, now()," in sql or " 0, now(), 'llm_call'" in sql, "the prompt must precede the verdict"
+ assert "a" * 64 in args
+ assert "b" * 64 in args
+ assert 742 in args
+ assert any("the literal prompt" in json.dumps(a) for a in args if isinstance(a, str))
+
+ @pytest.mark.asyncio
+ async def test_the_transcript_is_attached_for_explain_step(self) -> None:
+ """`/investigations/{run}/explain` inlines the artifacts of the
+ focal event, which is how `aisoc_explain_step` returns a literal
+ transcript rather than a hash."""
+ executed = await self._run(
+ [
+ {
+ "prompt": "the literal prompt",
+ "response": "the literal response",
+ "input_hash": "a" * 64,
+ "output_hash": "b" * 64,
+ "duration_ms": 1,
+ }
+ ]
+ )
+
+ artifacts = [a for s, a in executed if "investigation_artifacts" in s]
+ kinds = {a[4] for a in artifacts}
+ assert kinds == {"llm_prompt", "llm_response"}
+ assert any("the literal prompt" in str(a) for a in artifacts[0])
+
+ @pytest.mark.asyncio
+ async def test_a_run_with_no_model_call_writes_no_transcript(self) -> None:
+ """No invented artifact for the deterministic path."""
+ executed = await self._run([])
+ assert [s for s, _ in executed if "investigation_artifacts" in s] == []
+ assert [s for s, _ in executed if "'llm_call'" in s] == []
+
+
+class TestReplayDoesNotDuplicateTheTranscript:
+ def test_a_suppressed_insert_is_detected(self) -> None:
+ """`investigation_artifacts` has no key to conflict on, so the
+ artifact write is gated on whether the verdict row was new. The
+ run id is deterministic, so this path is taken on every replay."""
+ from app.investigator.ledger import _inserted_a_row
+
+ assert _inserted_a_row("INSERT 0 1") is True
+ assert _inserted_a_row("INSERT 0 0") is False
+
+ def test_an_unreadable_tag_writes_nothing(self) -> None:
+ """A missing transcript is visibly missing; a duplicated one
+ silently doubles on every replay."""
+ from app.investigator.ledger import _inserted_a_row
+
+ assert _inserted_a_row(None) is False
+ assert _inserted_a_row("weird") is False
+
+
+class TestGraphStepsCarryTheirDuration:
+ """`record_event` defaults ``duration_ms`` to 0 and the call site never
+ passed it, so every escalation step recorded as instantaneous. Driven
+ through the real ``_run`` against a graph that takes measurable time —
+ reading the source would pass on a call that passes a literal zero."""
+
+ @pytest.mark.asyncio
+ async def test_a_step_that_took_time_does_not_record_zero(self, monkeypatch: Any) -> None:
+ import asyncio
+
+ from app.graph import runner as runner_mod
+
+ recorded: list[dict[str, Any]] = []
+
+ async def _record_event(**kwargs: Any) -> None:
+ recorded.append(kwargs)
+
+ async def _resolve(_ref: str) -> Any:
+ return uuid4()
+
+ async def _start_run(**_kwargs: Any) -> Any:
+ return uuid4()
+
+ monkeypatch.setattr(runner_mod.ledger_module, "record_event", _record_event)
+ monkeypatch.setattr(runner_mod.ledger_module, "resolve_tenant", _resolve)
+ monkeypatch.setattr(runner_mod.ledger_module, "start_run", _start_run)
+
+ class _Graph:
+ async def astream(self, _state: dict[str, Any]) -> Any:
+ await asyncio.sleep(0.05)
+ yield {"triage": {}}
+ await asyncio.sleep(0.05)
+ yield {"respond": {}}
+
+ await runner_mod._run(
+ _Graph(),
+ _state(),
+ budget=runner_mod.default_budget(),
+ persist=True,
+ seq_start=1,
+ )
+
+ steps = [e for e in recorded if e.get("kind") == "graph_step"]
+ assert len(steps) == 2, f"expected two graph steps, got {len(steps)}"
+ assert all(s["duration_ms"] > 0 for s in steps), [s["duration_ms"] for s in steps]
diff --git a/services/agents/tests/test_copilot_fallback_is_honest.py b/services/agents/tests/test_copilot_fallback_is_honest.py
new file mode 100644
index 000000000..324bef534
--- /dev/null
+++ b/services/agents/tests/test_copilot_fallback_is_honest.py
@@ -0,0 +1,167 @@
+"""The copilot says what happened instead of inventing an investigation.
+
+Issue #1275. Two defects on the same code path.
+
+**The fallback text named the wrong cause.** One string served both
+fallback conditions and ended "Configure an LLM key to get a real
+investigation". That branch is reached on *any* provider failure,
+including a timeout from the bundled local model, which needs no API key
+at all — so an operator whose model had merely timed out was told to go
+and configure a credential their deployment does not use. The same lesson
+is already recorded here from the RBA banner that blamed fusion while
+fusion was healthy.
+
+**The canned body was fabricated security analysis.** ``_synthetic_reply``
+ignored its argument and returned the next of five rotating paragraphs,
+each asserting specific findings about an estate it had never read — "this
+IP was seen in 3 other alerts", "attacker dwell time appears short
+(< 2 hours)". The reporter watched a ransomware alert come back described
+as "credential-access activity ... LOLBin pattern". The console appends a
+disclaimer above it, which does not make the body true.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import pytest
+from app.api import copilot
+from app.llm import contract as llm_contract
+from app.llm import factory as llm_factory
+
+
+class TestTheTwoReasonsAreDistinguished:
+ """Driven through the real `_get_openai_reply`, because the defect was
+ that both conditions arrived at one string — a test calling the two
+ message builders directly would never have noticed."""
+
+ @pytest.mark.asyncio
+ async def test_no_model_configured_is_reported_as_such(self, monkeypatch: Any) -> None:
+ monkeypatch.setattr(llm_factory, "resolve_api_key", lambda _m: "")
+
+ _text, source, reason = await copilot._get_openai_reply({"messages": []}, "what happened?")
+
+ assert source == "template"
+ assert reason == copilot.NO_MODEL
+
+ @pytest.mark.asyncio
+ async def test_a_configured_model_that_fails_is_not_called_a_missing_key(self, monkeypatch: Any) -> None:
+ """The exact situation from the report: a model *is* configured and
+ the call timed out."""
+ monkeypatch.setattr(llm_factory, "resolve_api_key", lambda _m: "sk-present")
+ monkeypatch.setattr(llm_factory, "chat_completions_url", lambda _m=None: "http://local/v1/chat/completions")
+
+ async def _boom(**_kwargs: Any) -> Any:
+ raise TimeoutError()
+
+ monkeypatch.setattr(llm_contract, "safe_chat_completions_request", _boom)
+
+ text, source, reason = await copilot._get_openai_reply({"messages": []}, "what happened?")
+
+ assert source == "template"
+ assert reason == copilot.CALL_FAILED
+ assert "not a missing credential" in text
+ assert "not a missing API key" in (copilot._notice_for(reason) or "")
+
+ @pytest.mark.asyncio
+ async def test_an_empty_exception_message_still_names_the_failure(self, monkeypatch: Any) -> None:
+ """`str(exc)` is empty for several httpx timeout classes, which is
+ why the report saw `copilot.openai_error` with no message."""
+ monkeypatch.setattr(llm_factory, "resolve_api_key", lambda _m: "sk-present")
+ monkeypatch.setattr(llm_factory, "chat_completions_url", lambda _m=None: "http://local/v1/chat/completions")
+
+ async def _boom(**_kwargs: Any) -> Any:
+ raise TimeoutError()
+
+ monkeypatch.setattr(llm_contract, "safe_chat_completions_request", _boom)
+
+ text, _source, _reason = await copilot._get_openai_reply({"messages": []}, "q")
+ assert "TimeoutError" in text
+
+ def test_the_two_notices_differ(self) -> None:
+ assert copilot._notice_for(copilot.NO_MODEL) != copilot._notice_for(copilot.CALL_FAILED)
+
+ def test_only_the_unconfigured_notice_mentions_configuring_one(self) -> None:
+ """The whole defect in one assertion."""
+ failed = copilot._notice_for(copilot.CALL_FAILED) or ""
+ assert "not a missing API key" in failed
+ assert "A model is configured" in failed
+
+ def test_a_model_answer_carries_no_notice(self) -> None:
+ assert copilot._notice_for(None) is None
+
+
+class TestTheFallbackBodyAssertsNothingAboutTheEstate:
+ """Every one of these strings was in a reply the product shipped as a
+ 200, under a `source` field most callers never read."""
+
+ @pytest.mark.parametrize(
+ "fabricated",
+ [
+ "seen in 3 other alerts",
+ "dwell time",
+ "LOLBin",
+ "unusual geolocation",
+ "T1078",
+ "T1021.002",
+ "T1110",
+ "isolating the host",
+ "step-up MFA",
+ "credential-stuffing",
+ ],
+ )
+ def test_no_fabricated_finding_survives_in_any_fallback(self, fabricated: str) -> None:
+ bodies = [
+ copilot._fallback_reply(copilot.NO_MODEL),
+ copilot._fallback_reply(copilot.CALL_FAILED, model="llama3.2:3b", detail="timeout"),
+ ]
+ for body in bodies:
+ assert fabricated.lower() not in body.lower()
+
+ def test_the_module_no_longer_carries_the_canned_corpus(self) -> None:
+ """The paragraphs are gone, not merely unreferenced. A constant left
+ in place is one edit away from being used again."""
+ assert not hasattr(copilot, "_SYNTHETIC_REPLIES")
+ assert not hasattr(copilot, "_synthetic_reply")
+
+ def test_the_reply_states_plainly_that_nothing_was_read(self) -> None:
+ for body in (
+ copilot._fallback_reply(copilot.NO_MODEL),
+ copilot._fallback_reply(copilot.CALL_FAILED),
+ ):
+ assert "unanswered question rather than a finding" in body
+
+ def test_the_failed_call_names_the_model_and_the_error(self) -> None:
+ body = copilot._fallback_reply(copilot.CALL_FAILED, model="llama3.2:3b", detail="ReadTimeout")
+ assert "llama3.2:3b" in body
+ assert "ReadTimeout" in body
+
+ def test_it_points_at_the_contention_the_reporter_actually_hit(self) -> None:
+ """CPU-only deployments share one local model between interactive
+ copilot and the auto-triage backlog. Naming it is the difference
+ between a retry and an afternoon of debugging."""
+ body = copilot._fallback_reply(copilot.CALL_FAILED)
+ assert "auto-triage" in body
+
+ def test_the_same_question_does_not_get_a_different_canned_answer(self) -> None:
+ """`_synthetic_reply` cycled, so asking twice produced two
+ unrelated 'analyses' of one alert."""
+ first = copilot._fallback_reply(copilot.NO_MODEL)
+ second = copilot._fallback_reply(copilot.NO_MODEL)
+ assert first == second
+
+
+class TestTheWireContractCarriesTheReason:
+ def test_the_response_model_declares_the_reason(self) -> None:
+ fields = copilot.CopilotChatResponse.model_fields
+ assert "template_reason" in fields
+
+ def test_it_is_optional_so_existing_clients_are_unaffected(self) -> None:
+ """Additive, not a third `source` value — the console branches on
+ `source === 'template'` and must keep working."""
+ built = copilot.CopilotChatResponse(
+ conversationId="c",
+ reply=copilot.CopilotMessage(role="assistant", content="x"),
+ )
+ assert built.template_reason is None
+ assert built.source == "llm"
diff --git a/services/agents/tests/test_response_provenance.py b/services/agents/tests/test_response_provenance.py
index 5ce4cbce2..c6ac0f989 100644
--- a/services/agents/tests/test_response_provenance.py
+++ b/services/agents/tests/test_response_provenance.py
@@ -45,9 +45,13 @@ class TestCopilotProvenance:
@pytest.mark.asyncio
async def test_no_api_key_is_reported_as_template(self, monkeypatch: pytest.MonkeyPatch):
monkeypatch.delenv("OPENAI_API_KEY", raising=False)
- text, source = await copilot_mod._get_openai_reply({"messages": []}, "what happened?")
+ text, source, reason = await copilot_mod._get_openai_reply({"messages": []}, "what happened?")
assert source == "template"
assert text, "the fallback should still say something useful"
+ # The reason is carried alongside the source (issue #1275): "nothing
+ # is configured" and "what is configured did not answer" used to be
+ # the same answer, and only one of them is fixed by a credential.
+ assert reason == copilot_mod.NO_MODEL
@pytest.mark.asyncio
async def test_llm_failure_is_reported_as_template(self, monkeypatch: pytest.MonkeyPatch):
@@ -58,8 +62,9 @@ class TestCopilotProvenance:
raise RuntimeError("provider unreachable")
monkeypatch.setattr("app.llm.contract.safe_chat_completions_request", _boom, raising=False)
- _text, source = await copilot_mod._get_openai_reply({"messages": []}, "hello")
+ _text, source, reason = await copilot_mod._get_openai_reply({"messages": []}, "hello")
assert source == "template"
+ assert reason == copilot_mod.CALL_FAILED
def test_response_model_defaults_to_llm_but_can_carry_template(self):
"""The field must exist on the wire, not just in the handler."""
diff --git a/services/api/app/api/v1/endpoints/alerts.py b/services/api/app/api/v1/endpoints/alerts.py
index af397ce7f..02c06d517 100644
--- a/services/api/app/api/v1/endpoints/alerts.py
+++ b/services/api/app/api/v1/endpoints/alerts.py
@@ -8,7 +8,7 @@ from datetime import UTC, datetime
from typing import Annotated, Any, Literal
from fastapi import APIRouter, Depends, Header, HTTPException, Query, Request, status
-from pydantic import BaseModel, ConfigDict, Field
+from pydantic import BaseModel, ConfigDict, Field, model_validator
from sqlalchemy import and_, func, select, update
from sqlalchemy.exc import IntegrityError
@@ -232,8 +232,26 @@ class AlertListResponse(BaseModel):
facets: AlertFacetCounts = Field(default_factory=AlertFacetCounts)
+#: Fields this endpoint cannot apply, mapped to the route that can.
+#:
+#: Each key is a real ``Alert`` column, which is why they arrive here and
+#: why dropping them silently was expensive: the caller is asking for
+#: something the platform does support, just not through this verb.
+_ALERT_UPDATE_ELSEWHERE = {
+ "disposition": "POST /api/v1/feedback/alert-override",
+ "corrected_verdict": "POST /api/v1/feedback/alert-override",
+ "verdict": "POST /api/v1/feedback/alert-override",
+ "severity": "POST /api/v1/alerts/{alert_id}/escalate",
+ "snoozed_until": "POST /api/v1/alerts/{alert_id}/snooze",
+}
+
+
class AlertUpdateRequest(BaseModel):
- model_config = ConfigDict(populate_by_name=True)
+ #: ``extra="forbid"``, because the default is ``ignore`` and this is a
+ #: write endpoint. An ignored field on a read is a tolerated unknown;
+ #: on a write it is a 200 reporting a change that did not happen, and
+ #: nothing in the response body distinguishes the two.
+ model_config = ConfigDict(populate_by_name=True, extra="forbid")
status: str | None = None
priority: int | None = None
@@ -243,6 +261,29 @@ class AlertUpdateRequest(BaseModel):
assigned_to_id: uuid.UUID | None = Field(default=None, alias="assignee")
case_id: uuid.UUID | None = None
+ @model_validator(mode="before")
+ @classmethod
+ def _name_the_route_that_applies_it(cls, data: Any) -> Any:
+ """Refuse an unapplicable field by name, and say where it does apply.
+
+ ``extra="forbid"`` alone answers "Extra inputs are not permitted",
+ which tells an operator their call was wrong without telling them
+ what is right. Three of these fields have a route that works.
+ """
+ if not isinstance(data, dict):
+ return data
+
+ accepted = {"status", "priority", "tags", "assigned_to_id", "assignee", "case_id"}
+ unknown = [str(key) for key in data if str(key) not in accepted]
+ if not unknown:
+ return data
+
+ parts = []
+ for key in sorted(unknown):
+ route = _ALERT_UPDATE_ELSEWHERE.get(key)
+ parts.append(f"{key} (use {route})" if route else key)
+ raise ValueError(f"PATCH /alerts cannot apply: {', '.join(parts)}. It accepts only: {', '.join(sorted(accepted))}.")
+
class AlertStatsResponse(BaseModel):
total: int
diff --git a/services/api/tests/test_alert_update_refuses_unknown_fields.py b/services/api/tests/test_alert_update_refuses_unknown_fields.py
new file mode 100644
index 000000000..2d498f5d3
--- /dev/null
+++ b/services/api/tests/test_alert_update_refuses_unknown_fields.py
@@ -0,0 +1,114 @@
+"""``PATCH /alerts/{id}`` refuses a field it cannot apply (issue #1280).
+
+``AlertUpdateRequest`` set no ``extra=``, so pydantic v2's default
+``extra="ignore"`` applied and every key outside the declared five was
+dropped before the handler ran. The response was a 200 carrying the
+unchanged row, which is indistinguishable from a successful write.
+
+The fields that reach this endpoint are not nonsense — ``disposition``,
+``severity``, ``confidence`` and ``snoozed_until`` are all real ``Alert``
+columns, three of them writable through a different route. That is what
+made the silent drop expensive: a caller closing an alert with a
+disposition got a 200 and no disposition, and nothing in the response
+distinguished "applied" from "ignored".
+"""
+
+from __future__ import annotations
+
+import uuid
+
+import pytest
+from app.api.v1.endpoints.alerts import AlertUpdateRequest
+from pydantic import ValidationError
+
+
+def _message(exc: ValidationError) -> str:
+ return str(exc)
+
+
+class TestAnUnapplicableFieldIsRefused:
+ def test_the_reported_payload_is_refused_rather_than_half_applied(self) -> None:
+ """The exact body from the report: status lands, disposition does not."""
+ with pytest.raises(ValidationError) as caught:
+ AlertUpdateRequest(status="resolved", disposition="false_positive")
+
+ assert "disposition" in _message(caught.value)
+
+ @pytest.mark.parametrize(
+ "field",
+ ["disposition", "severity", "confidence", "resolution", "snoozed_until"],
+ )
+ def test_every_silently_dropped_field_is_refused(self, field: str) -> None:
+ """The report named one; the drop list was wider than one."""
+ with pytest.raises(ValidationError) as caught:
+ AlertUpdateRequest(**{field: "x"})
+
+ assert field in _message(caught.value)
+
+ def test_a_typo_in_a_real_field_is_refused_rather_than_ignored(self) -> None:
+ """`statuss` used to be a 200 that changed nothing."""
+ with pytest.raises(ValidationError):
+ AlertUpdateRequest(statuss="resolved")
+
+
+class TestTheRefusalNamesSomewhereToGo:
+ """A 422 that only says "not permitted" tells an operator their call
+ was wrong without telling them what is right, and three of these
+ fields have a route that does apply them."""
+
+ @pytest.mark.parametrize(
+ ("field", "route"),
+ [
+ ("disposition", "/feedback/alert-override"),
+ ("severity", "/escalate"),
+ ("snoozed_until", "/snooze"),
+ ],
+ )
+ def test_a_field_with_a_home_elsewhere_names_that_route(self, field: str, route: str) -> None:
+ with pytest.raises(ValidationError) as caught:
+ AlertUpdateRequest(**{field: "x"})
+
+ assert route in _message(caught.value)
+
+ def test_a_field_with_no_home_lists_what_is_accepted(self) -> None:
+ """No route is invented for a field that has none."""
+ with pytest.raises(ValidationError) as caught:
+ AlertUpdateRequest(nonsense=1)
+
+ message = _message(caught.value)
+ assert "status" in message
+ assert "assignee" in message
+
+
+class TestTheAcceptedSurfaceIsUnchanged:
+ """`extra="forbid"` is a behaviour change on a published endpoint, so
+ the controls matter as much as the refusal. Every shape the console
+ and the SDK already send must still be accepted."""
+
+ def test_the_five_declared_fields_are_accepted(self) -> None:
+ parsed = AlertUpdateRequest(
+ status="triaged",
+ priority=80,
+ tags=["a"],
+ assigned_to_id=None,
+ case_id=None,
+ )
+ assert parsed.status == "triaged"
+ assert parsed.priority == 80
+
+ def test_the_assignee_alias_is_not_extra(self) -> None:
+ """A declared alias is not an unknown field. The queue releases an
+ alert with `{"assignee": null}` and that must keep working."""
+ parsed = AlertUpdateRequest(assignee=None)
+ assert "assigned_to_id" in parsed.model_fields_set
+
+ def test_assigning_by_alias_still_binds_the_value(self) -> None:
+ who = uuid.uuid4()
+ assert AlertUpdateRequest(assignee=who).assigned_to_id == who
+
+ def test_the_canonical_field_name_still_works(self) -> None:
+ who = uuid.uuid4()
+ assert AlertUpdateRequest(assigned_to_id=who).assigned_to_id == who
+
+ def test_an_empty_body_is_still_a_no_op_rather_than_an_error(self) -> None:
+ assert AlertUpdateRequest().model_fields_set == set()
diff --git a/services/mcp/src/tools/hunt.ts b/services/mcp/src/tools/hunt.ts
index c15eb59e0..938ce07db 100644
--- a/services/mcp/src/tools/hunt.ts
+++ b/services/mcp/src/tools/hunt.ts
@@ -23,6 +23,21 @@
*
* **It is read-only.** A hunt searches; it does not act. The annotation
* says so and no write path exists here.
+ *
+ * **It takes no tenant.** It used to declare
+ * `tenant_id: z.string().uuid()` and forward it in the body (issue
+ * #1274). Two things were wrong with that, and the smaller one is what
+ * got noticed: zod 4's `.uuid()` enforces RFC 4122 version and variant
+ * bits, which the tenant every default install is bootstrapped into
+ * (`00000000-0000-0000-0000-000000000001`) does not set — so the tool
+ * could not be called at all on a stock deployment.
+ *
+ * The larger one is that a tenant supplied by a *model* is not a
+ * boundary. `AgentHuntRequest` declares only `hypothesis`, so pydantic
+ * was silently dropping the field: the isolation was an accident of
+ * schema strictness rather than a decision. Relaxing the validator to
+ * `z.guid()` would have made the bad input pass and left that in place.
+ * The tenant comes from the credential, here as everywhere else.
*/
import { z } from "zod";
@@ -46,7 +61,6 @@ const RunHuntSchema = z
.describe(
"What you suspect, in plain language. For example: 'a service account signed in from a country it has never used before'.",
),
- tenant_id: z.string().uuid().describe("The tenant whose recorded events to search."),
})
.strict();
@@ -69,7 +83,6 @@ export const runHuntTool: ToolDefinition<typeof RunHuntSchema> = {
async handle(ctx, args) {
const reply = await ctx.client.post<HuntReply>("/api/v1/agents/hunt", {
hypothesis: args.hypothesis,
- tenant_id: args.tenant_id,
});
const matches = reply.matches ?? [];
diff --git a/services/mcp/tests/tenant-is-not-an-argument.test.ts b/services/mcp/tests/tenant-is-not-an-argument.test.ts
new file mode 100644
index 000000000..0c9f879b6
--- /dev/null
+++ b/services/mcp/tests/tenant-is-not-an-argument.test.ts
@@ -0,0 +1,159 @@
+/**
+ * The tenant is never a tool argument (issue #1274).
+ *
+ * `aisoc_run_hunt` declared `tenant_id: z.string().uuid()` and forwarded
+ * it in the request body. Two separate things were wrong with that.
+ *
+ * The one the report hit: zod 4's `.uuid()` checks RFC 4122 version and
+ * variant bits, and the tenant every default install is bootstrapped into
+ * (`00000000-0000-0000-0000-000000000001`) sets neither. So the tool was
+ * unusable on a stock deployment — the model could not produce a value the
+ * schema would accept.
+ *
+ * The one that matters more: a tenant a *model* supplies is not a control.
+ * `AgentHuntRequest` declares only `hypothesis` and the handler's docstring
+ * says the tenant "comes from the authenticated principal and is never read
+ * from the body", so today's protection is an accident of pydantic dropping
+ * an undeclared key — not a boundary. Loosening the validator to `z.guid()`
+ * would have made the bad input pass and left the shape in place.
+ *
+ * The assertion is therefore structural and repo-wide, not a fix to one
+ * tool: no tool on this server takes a tenant from its caller.
+ */
+import { readFileSync, readdirSync } from "node:fs";
+import { join } from "node:path";
+import { fileURLToPath } from "node:url";
+
+import { describe, expect, it } from "vitest";
+import { z } from "zod";
+
+import { ALL_TOOLS } from "../src/tools/index.js";
+import { runHuntTool } from "../src/tools/hunt.js";
+
+const SRC = fileURLToPath(new URL("../src", import.meta.url));
+
+/** `bootstrap_admin.DEFAULT_TENANT_ID` — what a stock `make up` creates. */
+const DEFAULT_TENANT = "00000000-0000-0000-0000-000000000001";
+
+function everySourceFile(dir: string): string[] {
+ const out: string[] = [];
+ for (const entry of readdirSync(dir, { withFileTypes: true })) {
+ const path = join(dir, entry.name);
+ if (entry.isDirectory()) out.push(...everySourceFile(path));
+ else if (entry.name.endsWith(".ts")) out.push(path);
+ }
+ return out;
+}
+
+interface RecordedCall {
+ path: string;
+ body?: unknown;
+}
+
+function recordingClient(calls: RecordedCall[], reply: unknown) {
+ return {
+ async get(path: string) {
+ calls.push({ path });
+ return reply;
+ },
+ async post(path: string, body: unknown) {
+ calls.push({ path, body });
+ return reply;
+ },
+ };
+}
+
+const log = { debug() {}, info() {}, warn() {}, error() {} };
+
+describe("the reported symptom", () => {
+ it("confirms zod rejects the default tenant that z.guid() accepts", () => {
+ // Not an assertion about our code — it pins the upstream behaviour the
+ // report rests on, so a zod bump that changes it shows up here rather
+ // than as a mystery. The version-and-variant bits are all zero.
+ expect(z.string().uuid().safeParse(DEFAULT_TENANT).success).toBe(false);
+ expect(z.guid().safeParse(DEFAULT_TENANT).success).toBe(true);
+
+ // A conventional v4 uuid passes both, which is why this went unnoticed:
+ // it works on any tenant except the one every first run lands in.
+ const v4 = "550e8400-e29b-41d4-a716-446655440000";
+ expect(z.string().uuid().safeParse(v4).success).toBe(true);
+ });
+});
+
+describe("no tool takes a tenant from its caller", () => {
+ it("declares no tenant field in any tool's input schema", () => {
+ const offenders: string[] = [];
+ for (const tool of ALL_TOOLS) {
+ const schema = tool.metadata.inputSchema as {
+ properties?: Record<string, unknown>;
+ };
+ for (const key of Object.keys(schema.properties ?? {})) {
+ if (/tenant/i.test(key)) offenders.push(`${tool.metadata.name}.${key}`);
+ }
+ }
+ expect(offenders).toEqual([]);
+ });
+
+ it("sends no tenant in any request body built from tool arguments", async () => {
+ // The schema and the wire are two different claims. A tool could drop
+ // the parameter and still stamp a tenant into the body from somewhere.
+ const calls: RecordedCall[] = [];
+ await runHuntTool.handle(
+ {
+ client: recordingClient(calls, {
+ hypothesis: "h",
+ checked: true,
+ matches: [],
+ refusals: [],
+ unavailable_reason: null,
+ }),
+ log,
+ } as never,
+ { hypothesis: "a service account signed in from a new country" } as never,
+ );
+
+ expect(calls).toHaveLength(1);
+ expect(calls[0].path).toBe("/api/v1/agents/hunt");
+ expect(Object.keys(calls[0].body as object)).toEqual(["hypothesis"]);
+ });
+
+ it("names no tenant_id key anywhere a request body is built", () => {
+ // Structural, so a tool added later is covered without editing this
+ // test. Prose and response-shape types legitimately mention the word,
+ // so match an object key being assigned, which is what a body literal
+ // and a zod schema field both look like.
+ const offenders: string[] = [];
+ for (const file of everySourceFile(SRC)) {
+ const body = readFileSync(file, "utf8");
+ for (const line of body.split("\n")) {
+ // Skip comments — `lake.ts` explains at length that the predicate is
+ // injected server-side, which is the behaviour this test wants.
+ const code = line.replace(/^\s*(\/\/|\*|\/\*).*$/, "");
+ if (/\btenant_id\s*:/.test(code) && !/\btenant_id\s*:\s*string\s*\|\s*null/.test(code)) {
+ offenders.push(`${file.replace(SRC, "src")}: ${line.trim()}`);
+ }
+ }
+ }
+ expect(offenders).toEqual([]);
+ });
+});
+
+describe("the hunt tool still works", () => {
+ it("accepts a hypothesis alone", () => {
+ expect(runHuntTool.schema.safeParse({ hypothesis: "x" }).success).toBe(true);
+ });
+
+ it("refuses a caller-supplied tenant rather than ignoring it", () => {
+ // `.strict()` is deliberate. Silently dropping the key is how the API
+ // side ended up with a protection nobody had decided on.
+ const parsed = runHuntTool.schema.safeParse({
+ hypothesis: "x",
+ tenant_id: DEFAULT_TENANT,
+ });
+ expect(parsed.success).toBe(false);
+ });
+
+ it("still requires a non-empty hypothesis", () => {
+ expect(runHuntTool.schema.safeParse({ hypothesis: "" }).success).toBe(false);
+ });
+});