Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
fix(agents,api): bound the manual investigation path and stop the LLM panel claiming Live while every call fails (#1284)
Beenu Arorae10afaa8a53e
diff --git a/.env.example b/.env.exampleindex fba11f0d3..1b818a2d5 100644--- a/.env.example+++ b/.env.example@@ -245,6 +245,14 @@ LITELLM_MASTER_KEY=sk-aisoc-local# default deployment.LLM_GATEWAY_URL=http://litellm:4000/v1#+# How long the gateway waits on ONE generation before giving up, in seconds.+# 60 used to be a literal inside infra/litellm/config.yaml, so raising it meant+# editing a mounted file. The bundled 3B model doing CPU-only inference can+# exceed 60s on a long generation, and every call then fails while Settings →+# Deployment & AI still reports the provider as configured. That panel now says+# "Degraded" when recent calls are failing — this is the knob to turn.+AISOC_LITELLM_REQUEST_TIMEOUT=60+## ENABLE LIVE LLM:# (a) Through the gateway — the default, and already wired. Put your real# provider key in OPENAI_API_KEY above (the gateway reads it) and bring updiff --git a/apps/docs/docs/deployment/env-vars.md b/apps/docs/docs/deployment/env-vars.mdindex 8193af4f0..a28f91f93 100644--- a/apps/docs/docs/deployment/env-vars.md+++ b/apps/docs/docs/deployment/env-vars.md@@ -274,6 +274,11 @@ Source: [`services/agents/app/`](https://github.com/beenuar/AiSOC/tree/main/serv| `OPENAI_API_KEY` | — | Provider key. With the bundled gateway running, the *gateway* uses this to reach the upstream model; AiSOC itself authenticates with `LITELLM_MASTER_KEY`. || `LLM_GATEWAY_URL` | `http://litellm:4000/v1` (set by compose) | In-network URL of the bundled LiteLLM gateway. Both resolvers read it for any `aisoc-<role>` alias — an alias resolves nowhere else. A concrete `AISOC_MODEL_PIN_<ROLE>` is left pointing at its provider. || `LITELLM_MASTER_KEY` | `sk-aisoc-local` in compose | Bearer AiSOC sends when it routes to the gateway itself. Resolved *with* the URL, so a provider key is never sent to the gateway (it rejects one). |+| `AISOC_LITELLM_REQUEST_TIMEOUT` | `60` (seconds, set by compose) | How long the gateway waits on one generation before giving up. Raise it when the bundled local model is doing CPU-only inference and long generations are timing out; the Settings → Deployment & AI panel reports **Degraded** when recent calls are failing, which is the signal to turn this up. Compose must always pass a value — an unset variable resolves to `None` inside the gateway, which then serves `/health/liveliness` 200 while failing every completion. |+| `AISOC_INVESTIGATION_MAX_SECONDS` | `120` | Wall-clock budget for one investigation, on **both** the auto-triage path and the console's manual "Investigate with agent". A run that exceeds it is closed as `failed` with the reason, in the run store and in the `investigation_runs` row. |+| `AISOC_INVESTIGATION_STALE_AFTER_SECONDS` | `3 × AISOC_INVESTIGATION_MAX_SECONDS`, floor 300 | When a background sweep closes an `investigation_runs` row abandoned by a process that exited mid-run. Derived from the budget above so the two cannot disagree; set it only if you need a different margin. |+| `AISOC_INVESTIGATION_REAP_INTERVAL_SECONDS` | `60` | How often that sweep runs. |+| `AISOC_INVESTIGATION_REAPER_DISABLE` | unset (sweep on) | Set to `1` to turn the sweep off. Leaving it off means a run orphaned by a restart stays `running` in the ledger forever. || `OPENAI_BASE_URL` / `LLM_BASE_URL` | — | Explicit override. Outranks `LLM_GATEWAY_URL` for every role; you own the key pairing when you set it. || `AISOC_MODEL_PIN_<ROLE>` | `aisoc-<role>` | Per-role model. Set a concrete provider model to call a provider directly with no gateway. Roles: `triage`, `recon`, `investigation`, `copilot`, `summary`, `report`, `nl`. || `OPENAI_MODEL` | `aisoc-summary` | BYOK / "explain this alert" path **only** — never a task role's model. Ships as an alias because the default deployment routes through the gateway, which 400s anything `infra/litellm/config.yaml` does not define. |diff --git a/apps/web/src/components/settings/SettingsView.llmhealth.test.tsx b/apps/web/src/components/settings/SettingsView.llmhealth.test.tsxnew file mode 100644index 000000000..96292e3ed--- /dev/null+++ b/apps/web/src/components/settings/SettingsView.llmhealth.test.tsx@@ -0,0 +1,188 @@+/**+ * The LLM pill must not read "Live" while every call is failing (issue #1241).+ *+ * `LlmCard` derived its badge from `effective_path` alone — a *configuration*+ * value decided by whether a key is set and whether the air-gap policy would+ * permit the host. A deployment whose every completion times out therefore+ * rendered the emerald "Live" pill, which is the panel asserting something it+ * had never checked.+ *+ * `recent_health` carries the outcome of the calls the product actually made.+ * Three states, and `unknown` matters as much as `degraded`: a pod that has+ * observed nothing must not be rendered as healthy, and must not be rendered+ * as broken either.+ */+import { describe, expect, it, vi, beforeEach } from 'vitest';+import { render, screen } from '@testing-library/react';+import userEvent from '@testing-library/user-event';++const swrData = vi.hoisted(() => new Map<string, unknown>());+const swrErrors = vi.hoisted(() => new Map<string, unknown>());+const swrLoadingKeys = vi.hoisted(() => new Set<string>());+const swrMutate = vi.hoisted(() => vi.fn());++vi.mock('swr', () => ({+ __esModule: true,+ default: (key: unknown) => {+ const k = typeof key === 'string' ? key : JSON.stringify(key);+ return {+ data: swrData.get(k),+ error: swrErrors.get(k),+ isLoading: swrLoadingKeys.has(k),+ mutate: swrMutate,+ };+ },+}));++const noop = vi.hoisted(() => vi.fn());+vi.mock('@/lib/api', () => ({+ __esModule: true,+ deploymentApi: {+ getAirgapStatus: noop,+ getLlmStatus: noop,+ getLlmCredential: noop,+ upsertLlmCredential: noop,+ deleteLlmCredential: noop,+ },+ connectorsApi: { list: noop, statuses: noop },+ ApiError: class ApiError extends Error {+ status: number;+ constructor(message: string, status = 500) {+ super(message);+ this.status = status;+ }+ },+}));++vi.mock('react-hot-toast', () => ({+ __esModule: true,+ default: { success: vi.fn(), error: vi.fn() },+ toast: { success: vi.fn(), error: vi.fn() },+}));++import { SettingsView } from './SettingsView';+import type { LlmStatus } from '@/lib/api';++const AIRGAP_KEY = 'settings:airgap-status';+const LLM_KEY = 'settings:llm-status';+const CRED_KEY = 'settings:llm-credential';++const airgapFixture = {+ enabled: false,+ allowlist: [],+ implicit_private_suffixes: ['.local', '.internal'],+ policy: 'Air-gap disabled; all egress permitted.',+};++function llmFixture(overrides: Partial<LlmStatus> = {}): LlmStatus {+ return {+ provider: 'local-litellm',+ model: 'aisoc-triage',+ base_url: 'http://litellm:4000/v1',+ host: 'litellm',+ key_set: true,+ airgap_enabled: false,+ airgap_compliant: true,+ is_local: true,+ effective_path: 'live',+ policy_note: 'LLM calls are routed to a local provider (local-litellm).',+ recent_health: 'healthy',+ recent_calls: 6,+ recent_failures: 0,+ recent_note: 'The most recent LLM call succeeded (0 of the last 6 failed).',+ ...overrides,+ };+}++async function renderWith(llm: LlmStatus) {+ swrData.set(AIRGAP_KEY, airgapFixture);+ swrData.set(LLM_KEY, llm);+ swrData.set(CRED_KEY, undefined);+ render(<SettingsView />);+ await userEvent.click(+ await screen.findByRole('button', { name: /deployment.*ai/i }),+ );+}++beforeEach(() => {+ swrData.clear();+ swrErrors.clear();+ swrLoadingKeys.clear();+});++describe('LLM provider pill reflects whether the provider answers', () => {+ it('reads Degraded, not Live, when recent calls are failing', async () => {+ await renderWith(+ llmFixture({+ recent_health: 'degraded',+ recent_calls: 5,+ recent_failures: 5,+ recent_note:+ 'The most recent LLM call failed (ReadTimeout); 5 of the last 5 failed. ' ++ 'If the provider is a local model on CPU, raise AISOC_LITELLM_REQUEST_TIMEOUT.',+ }),+ );++ expect(await screen.findByText('Degraded')).toBeInTheDocument();+ expect(screen.queryByText('Live')).not.toBeInTheDocument();+ });++ it('tells the operator why, rather than only that something is wrong', async () => {+ await renderWith(+ llmFixture({+ recent_health: 'degraded',+ recent_calls: 5,+ recent_failures: 5,+ recent_note:+ 'The most recent LLM call failed (ReadTimeout); 5 of the last 5 failed. ' ++ 'If the provider is a local model on CPU, raise AISOC_LITELLM_REQUEST_TIMEOUT.',+ }),+ );++ expect(await screen.findByText(/5 of the last 5 failed/)).toBeInTheDocument();+ expect(screen.getByText(/AISOC_LITELLM_REQUEST_TIMEOUT/)).toBeInTheDocument();+ });++ it('still reads Live when the provider is answering', async () => {+ await renderWith(llmFixture());+ expect(await screen.findByText('Live')).toBeInTheDocument();+ expect(screen.queryByText('Degraded')).not.toBeInTheDocument();+ });++ it('says calls are unobserved rather than claiming health it has not seen', async () => {+ await renderWith(+ llmFixture({+ recent_health: 'unknown',+ recent_calls: 0,+ recent_failures: 0,+ recent_note:+ 'No LLM call has been observed by this pod recently, so whether the provider ' ++ 'answers is unmeasured. This reports configuration only.',+ }),+ );++ // Configuration is live, so the path badge stays — but the panel must not+ // present an unobserved provider as a verified-working one.+ expect(await screen.findByText('Live')).toBeInTheDocument();+ expect(screen.getByText(/No LLM call has been observed/)).toBeInTheDocument();+ });++ it('keeps the fallback badge winning over the health signal', async () => {+ // A deployment with no key is on the deterministic path: no call is even+ // attempted, so "degraded" would be the wrong story to lead with.+ await renderWith(+ llmFixture({+ key_set: false,+ effective_path: 'fallback',+ recent_health: 'unknown',+ recent_calls: 0,+ recent_failures: 0,+ recent_note: 'No LLM call has been observed by this pod recently.',+ }),+ );++ expect(await screen.findByText('Fallback path')).toBeInTheDocument();+ expect(screen.queryByText('Live')).not.toBeInTheDocument();+ expect(screen.queryByText('Degraded')).not.toBeInTheDocument();+ });+});diff --git a/apps/web/src/components/settings/SettingsView.tsx b/apps/web/src/components/settings/SettingsView.tsxindex d5f290a66..158b80f39 100644--- a/apps/web/src/components/settings/SettingsView.tsx+++ b/apps/web/src/components/settings/SettingsView.tsx@@ -1655,6 +1655,14 @@ function LlmCard({const providerLabel = PROVIDER_LABEL[llm.provider] ?? llm.provider;const isFallback = llm.effective_path === 'fallback';const blockedByAirgap = llm.airgap_enabled && !llm.airgap_compliant;+ // `effective_path` is configuration: it answers "would a call be attempted",+ // not "did one come back". Deciding the pill from it alone is why a+ // deployment timing out on every call rendered emerald Live (issue #1241).+ //+ // Fallback still wins: on the deterministic path no call is attempted, so+ // there is no provider outcome to report and "Degraded" would blame a+ // provider that was never asked.+ const isDegraded = !isFallback && llm.recent_health === 'degraded';return (<section@@ -1678,6 +1686,8 @@ function LlmCard({)}{isFallback ? (<StatusPill tone="amber" label="Fallback path" />+ ) : isDegraded ? (+ <StatusPill tone="red" label="Degraded" />) : (<StatusPill tone="emerald" label="Live" />)}@@ -1712,8 +1722,29 @@ function LlmCard({</div>) : null}+ {isDegraded ? (+ <div+ role="alert"+ className="mt-4 rounded-lg border border-red-500/40 bg-red-500/10 p-3 text-xs text-red-200"+ >+ <p className="font-semibold text-red-100">+ The provider is configured but recent calls are failing.+ </p>+ <p className="mt-1">{llm.recent_note}</p>+ </div>+ ) : null}+<dl className="mt-4 grid grid-cols-1 gap-3 sm:grid-cols-2"><KeyValue label="Provider">{providerLabel}</KeyValue>+ {/* Shown whenever the alert block above is not already carrying it, so+ the evidence behind the badge is visible in every state — including+ `unknown`. A panel that shows a badge and hides what it rests on is+ how "Live" came to mean "configured" without anyone noticing. */}+ {isDegraded ? null : (+ <KeyValue label="Recent calls">+ <span className="text-xs text-gray-300">{llm.recent_note}</span>+ </KeyValue>+ )}<KeyValue label="Model">{llm.model ? (<code className="font-mono text-xs text-gray-100">{llm.model}</code>diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.tsindex 230a74131..5ba9502e7 100644--- a/apps/web/src/lib/api.ts+++ b/apps/web/src/lib/api.ts@@ -6155,6 +6155,8 @@ export type LlmProvider =export type LlmEffectivePath = 'live' | 'fallback';+export type LlmRecentHealth = 'healthy' | 'degraded' | 'unknown';+export interface LlmStatus {/** Stable provider id classified from the configured base URL. */provider: LlmProvider;@@ -6185,6 +6187,22 @@ export interface LlmStatus {effective_path: LlmEffectivePath;/** Operator-readable explanation of the current state. */policy_note: string;+ /**+ * Whether the provider is actually *answering*, from the outcomes of the+ * calls this pod already made — not a probe, and not a latency figure.+ *+ * Everything above describes configuration, which is why a deployment+ * timing out on every call used to render the emerald "Live" pill. Three+ * states, and ``unknown`` is a real one: a pod that has observed no call+ * has not earned a green badge.+ */+ recent_health: LlmRecentHealth;+ /** Observed chat-completions calls in the recent window (this pod only). */+ recent_calls: number;+ /** How many of those failed. */+ recent_failures: number;+ /** Operator-readable summary of the window, including what to do about it. */+ recent_note: string;}// ─── BYOK per-tenant LLM credentials (WS-H2) ────────────────────────────────diff --git a/docker-compose.yml b/docker-compose.ymlindex 1c6796675..bb4a6cd3d 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -933,6 +933,20 @@ services:# config). Empty is fine — the local model answers instead.OPENAI_API_KEY: ${OPENAI_API_KEY:-}ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}+ # How long the gateway waits on one generation before giving up, in+ # seconds. 60 was a bare literal inside the mounted config, so a+ # deployment whose model needs longer — CPU-only inference on a larger+ # prompt is the case — timed out on every call and could only be fixed by+ # editing a tracked file and restarting (issue #1241). The default is+ # unchanged; the knob is new. Raise it with+ # AISOC_LITELLM_REQUEST_TIMEOUT=300 in .env.+ #+ # The `:-60` default is load-bearing, not cosmetic: the gateway resolves+ # this key with `get_secret`, which yields None for an unset variable,+ # and a gateway in that state serves /health/liveliness 200 while failing+ # every completion. `test_litellm_config.py` fails if the default is+ # removed.+ AISOC_LITELLM_REQUEST_TIMEOUT: ${AISOC_LITELLM_REQUEST_TIMEOUT:-60}ports:- "${AISOC_BIND_ADDR:-127.0.0.1}:4000:4000"healthcheck:diff --git a/infra/litellm/config.yaml b/infra/litellm/config.yamlindex 8d7a3d741..eb442096c 100644--- a/infra/litellm/config.yaml+++ b/infra/litellm/config.yaml@@ -147,7 +147,34 @@ litellm_settings:require_auth_for_metrics_endpoint: false# Never wait forever on a stalled provider — fall through to AiSOC's own# deterministic path instead of hanging the alert drawer.- request_timeout: 60+ #+ # This read `request_timeout: 60`, the one bare literal in a file where every+ # other value resolves from the environment, so raising it meant editing a+ # mounted file and rebuilding. 60s is not a universal number: the bundled 3B+ # model doing CPU-only inference on a long generation can exceed it, and+ # every call then times out while the Settings panel still reports the+ # provider Live (issue #1241). The default is unchanged — the knob is new.+ #+ # Measured against `ghcr.io/berriai/litellm:main-stable`, because two+ # plausible-looking alternatives do not work:+ #+ # * LiteLLM's own `REQUEST_TIMEOUT` environment variable is read and+ # float-coerced in `litellm/constants.py`, but it does **not** reach the+ # proxy's router path: a gateway started with `REQUEST_TIMEOUT=5` and no+ # key here was still waiting on a blackholed provider after 200s.+ # * This key *does* reach it. `get_secret` resolves `os.environ/...` to a+ # string (it coerces only booleans), LiteLLM assigns that straight onto+ # `litellm.request_timeout`, and the value lands: with the variable at 3,+ # the gateway answered 408 with `timeout value=3.0, time taken=3.0`.+ #+ # The variable must therefore be SET. `get_secret` returns None for an unset+ # one, and a gateway in that state answers 200 on `/health/liveliness` while+ # failing every completion with `float() argument must be a string or a real+ # number, not 'NoneType'`. `docker-compose.yml` supplies the default, and+ # `test_litellm_config.py` fails if that default is ever removed — the config+ # is already hard-dependent on its environment in the same way (with+ # `AISOC_LLM_MODEL_FAST` unset this file does not start at all).+ request_timeout: os.environ/AISOC_LITELLM_REQUEST_TIMEOUT# ── Router: retries + cross-model fallbacks ───────────────────────────────────# model_pins.py documents that "the LiteLLM gateway owns any model-level@@ -157,6 +184,13 @@ litellm_settings:router_settings:# Retry a transient failure (rate limit / 5xx / timeout) on the same alias# before falling back.+ #+ # A literal on purpose, and not environment-driven like the timeout above.+ # LiteLLM hands `router_settings` straight to `Router.__init__` with no type+ # coercion (`router_params[k] = v`), and the retry path evaluates+ # `num_retries > 0`, which raises `TypeError` against a string at precisely+ # the moment a retry is needed. The knob that matters for slow hardware is+ # the timeout, and that one is environment-driven.num_retries: 2# Cross-model failover: if an alias's primary model keeps failing, route the# request to an adjacent/cheaper alias so quality degrades gracefully insteaddiff --git a/scripts/check_tenant_query_predicates.py b/scripts/check_tenant_query_predicates.pyindex 4e8ba6d16..9eb21f5a7 100644--- a/scripts/check_tenant_query_predicates.py+++ b/scripts/check_tenant_query_predicates.py@@ -238,6 +238,14 @@ RATCHET: dict[str, str] = {"iterate, and a tenant missing from that list would have approvals that never ""expire, which is the silent hang the sweep exists to prevent"),+ "services/agents/app/investigator/ledger.py::fail_stale_runs::investigation_runs": (+ "the stale-run reaper, the same shape as the approval and webhook sweeps below: a run is "+ "abandoned when the process that owned it is gone, so there is no caller and no tenant to "+ "filter by, and a per-tenant pass needs a list of tenants whose omissions are runs that "+ "render as 'running' forever. Admitted by the second arm of the policy in "+ "008_investigation_ledger.sql (`OR current_tenant_id() IS NULL`); the function says so, so a "+ "future policy that drops that arm turns this into a visible no-op rather than a silent one"+ ),"services/api/app/services/outbound_webhooks.py::due_deliveries::OutboundDelivery": ("the outbound-webhook retry sweep, the same shape as the approval and wait sweeps above: a ""per-tenant pass needs a list of tenants, and a tenant missing from it has events that are "@@ -428,7 +436,7 @@ RATCHET: dict[str, str] = {# ceiling with a spare slot in it and the list held 41. The ceiling now# equals the list, so the next unscoped statement has to be argued for# rather than absorbed — which is what this number is for.-MAX_RATCHET = 43+MAX_RATCHET = 44# ---------------------------------------------------------------------------diff --git a/services/agents/app/api/investigate.py b/services/agents/app/api/investigate.pyindex 80bb7752b..fdd643328 100644--- a/services/agents/app/api/investigate.py+++ b/services/agents/app/api/investigate.py@@ -26,7 +26,8 @@ from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, WebSockefrom fastapi.responses import HTMLResponse, PlainTextResponsefrom pydantic import BaseModel-from app.investigator import InvestigatorOrchestrator+from app.graph.runner import default_budget+from app.investigator import InvestigatorOrchestrator, ledgerfrom app.orchestrator.router import RouterOrchestratorfrom app.security.tenant_scope import (TENANT_HEADER,@@ -211,6 +212,53 @@ async def _emit_event(run_id: str, tenant_id: str, event: dict[str, Any]) -> Non# Background task: runs investigation and streams steps to realtime service# ---------------------------------------------------------------------------+#: Statuses that mean this run is over. The console's ``CaseWorkspace.tsx``+#: branches on ``idle``/``starting``/``running``/``failed``/``completed`` and the+#: WebSocket tail below breaks on ``completed``/``failed``, so a run that ends in+#: any other word is a run no surface can finish rendering. A timed-out run is+#: therefore ``failed`` carrying the reason, not a fifth vocabulary entry.+_TERMINAL_STATUSES = frozenset({"completed", "failed"})+++async def _fail_run(run_id: str, run_uuid: UUID, tenant_ref: str, reason: str) -> None:+ """Close a run as ``failed`` in **both** places a consumer reads it.++ The console polls ``GET /api/v1/investigations/{run_id}``, which serves the+ process dict; the Investigation Ledger renders the ``investigation_runs``+ row. Writing one and not the other swaps a spinner that never resolves for+ a ledger entry that never ends, which is the same defect wearing the other+ surface.++ Never downgrades a run that already finished: a ``done`` event followed by+ a slow generator teardown must not be relabelled a failure.+ """+ entry = _runs.get(run_id)+ if entry is not None:+ if entry.get("status") in _TERMINAL_STATUSES:+ return+ entry.update(+ {+ "status": "failed",+ "error": reason,+ "completed_at": datetime.utcnow().isoformat(),+ }+ )++ # Best-effort, exactly like every other ledger write: no database+ # configured is not an error, and a failed bookkeeping write must not+ # replace the reason the run ended with a reason it did not.+ try:+ tenant_uuid = await ledger.resolve_tenant(tenant_ref)+ if tenant_uuid is not None:+ await ledger.complete_run(+ run_id=run_uuid,+ tenant_id=tenant_uuid,+ status="failed",+ error=reason,+ )+ except Exception as exc: # noqa: BLE001 — bookkeeping never masks the outcome+ logger.warning("investigation.ledger_close_failed", run_id=run_id, error=str(exc))+async def _run_and_store(run_id: str, case_id: str, req: InvestigateRequest) -> None:audit_log: list[dict[str, Any]] = []@@ -220,67 +268,97 @@ async def _run_and_store(run_id: str, case_id: str, req: InvestigateRequest) ->run_uuid = UUID(run_id)except (ValueError, TypeError):run_uuid = uuid4()++ # The same wall-clock budget the auto-triage path runs under — read at call+ # time from ``AISOC_INVESTIGATION_MAX_SECONDS`` rather than restated here,+ # because two constants describing one deadline is how the shorter one wins+ # silently. Without it this path had none at all: `ChatOpenAI` is built with+ # no timeout, so a merely-slow model raises nothing and the run stays+ # `running` for the life of the process (issue #1242).+ max_seconds = default_budget().max_seconds+ deadline = asyncio.timeout(max_seconds)try:- # Use the streaming orchestrator so we can emit events progressively.- # ``_investigate_stream`` picks investigator vs. router at call time- # based on ``AISOC_INVESTIGATE_USE_ROUTER``.- async for event in _investigate_stream(- case_id=case_id,- alert_summary=req.alert_summary,- raw_alert=req.raw_alert,- tenant_id=req.tenant_id,- run_id=run_uuid,- ):- if event.get("type") == "step":- audit_log.append(event)- # Update the in-memory run so pollers see progress- _runs[run_id]["audit_log"] = audit_log- # Broadcast to realtime → WebSocket clients- await _emit_event(run_id, req.tenant_id, event)-- elif event.get("type") == "done":- state_data = event.get("state", {})- _runs[run_id].update(- {- "status": "completed",- "report_md": state_data.get("report_md", ""),- "report_html": state_data.get("report_html", ""),- "audit_log": audit_log,- "recon": state_data.get("recon", {}),- "forensic": state_data.get("forensic", {}),- "responder": state_data.get("responder", {}),- "completed_at": datetime.utcnow().isoformat(),- "error": None,- }- )- await _emit_event(- run_id,- req.tenant_id,- {- "kind": "completed",- "agent": "orchestrator",- "summary": "Investigation completed",- "data": {"status": "completed"},- },- )+ async with deadline:+ # Use the streaming orchestrator so we can emit events progressively.+ # ``_investigate_stream`` picks investigator vs. router at call time+ # based on ``AISOC_INVESTIGATE_USE_ROUTER``.+ async for event in _investigate_stream(+ case_id=case_id,+ alert_summary=req.alert_summary,+ raw_alert=req.raw_alert,+ tenant_id=req.tenant_id,+ run_id=run_uuid,+ ):+ if event.get("type") == "step":+ audit_log.append(event)+ # Update the in-memory run so pollers see progress+ _runs[run_id]["audit_log"] = audit_log+ # Broadcast to realtime → WebSocket clients+ await _emit_event(run_id, req.tenant_id, event)++ elif event.get("type") == "done":+ state_data = event.get("state", {})+ _runs[run_id].update(+ {+ "status": "completed",+ "report_md": state_data.get("report_md", ""),+ "report_html": state_data.get("report_html", ""),+ "audit_log": audit_log,+ "recon": state_data.get("recon", {}),+ "forensic": state_data.get("forensic", {}),+ "responder": state_data.get("responder", {}),+ "completed_at": datetime.utcnow().isoformat(),+ "error": None,+ }+ )+ await _emit_event(+ run_id,+ req.tenant_id,+ {+ "kind": "completed",+ "agent": "orchestrator",+ "summary": "Investigation completed",+ "data": {"status": "completed"},+ },+ )- elif event.get("type") == "error":- err_msg = event.get("error", "Unknown error")- _runs[run_id].update({"status": "failed", "error": err_msg})- await _emit_event(- run_id,- req.tenant_id,- {- "kind": "error",- "agent": "orchestrator",- "summary": err_msg,- "data": {"status": "failed"},- },- )+ elif event.get("type") == "error":+ err_msg = event.get("error", "Unknown error")+ _runs[run_id].update({"status": "failed", "error": err_msg})+ await _emit_event(+ run_id,+ req.tenant_id,+ {+ "kind": "error",+ "agent": "orchestrator",+ "summary": err_msg,+ "data": {"status": "failed"},+ },+ )++ except TimeoutError as exc:+ # `asyncio.timeout` converts the cancellation it issued into+ # `TimeoutError`; an inner `TimeoutError` (an httpx read timeout, say)+ # arrives here too and is *not* the budget. `expired()` tells them+ # apart, so the operator-facing reason names the thing that actually+ # ran out.+ if deadline.expired():+ reason = (+ f"Investigation exceeded its wall-clock budget of {max_seconds}s and was stopped. "+ "Raise AISOC_INVESTIGATION_MAX_SECONDS if the configured model needs longer."+ )+ logger.warning("investigation.budget_timeout", run_id=run_id, max_seconds=max_seconds)+ else:+ reason = str(exc) or "Investigation timed out"+ logger.error("investigation_bg_task timed out", run_id=run_id, error=reason)+ await _fail_run(run_id, run_uuid, req.tenant_id, reason)except Exception as exc: # noqa: BLE001logger.error("investigation_bg_task failed", run_id=run_id, error=str(exc))- _runs[run_id].update({"status": "failed", "error": str(exc)})+ # Through the same helper as the deadline: an exception escaping the+ # stream means the orchestrator's own `except` arm did not run, so+ # nothing else is going to close the ledger row either.+ await _fail_run(run_id, run_uuid, req.tenant_id, str(exc))# ---------------------------------------------------------------------------diff --git a/services/agents/app/investigator/ledger.py b/services/agents/app/investigator/ledger.pyindex 99c1da8fe..50843b2ff 100644--- a/services/agents/app/investigator/ledger.py+++ b/services/agents/app/investigator/ledger.py@@ -408,6 +408,78 @@ async def complete_run()+async def fail_stale_runs(+ *,+ older_than_seconds: float,+ reason: str,+ limit: int = 500,+) -> list[uuid.UUID]:+ """Close ``investigation_runs`` rows abandoned by a process that is gone.++ The wall-clock deadline in :func:`app.api.investigate._run_and_store` ends+ a run whose process is still alive. A restart, an OOM kill or an ordinary+ rollout leaves the row at ``'running'`` with nothing left to finish it:+ the in-memory run store is gone, so no poller, no WebSocket tail and no+ exception handler will ever write a terminal status. The Investigation+ Ledger then renders a run that never ends.++ Safe on several replicas and safe to re-run: the predicate is part of the+ ``UPDATE``, so two reapers racing the same row close it once and the loser+ simply matches nothing.++ Safe against reaping a *live* run only because the deadline exists — a run+ cannot legitimately outlast its own budget any more, so ``older_than_seconds``+ is set well beyond it (see :mod:`app.investigator.run_reaper`). Shortening+ that margin without the deadline in place would cancel healthy work.++ No RLS context is set: this is a cross-tenant maintenance sweep, and+ migration ``008_investigation_ledger.sql`` writes the policy as+ ``USING (tenant_id = current_tenant_id() OR current_tenant_id() IS NULL)``,+ whose second arm is what admits it. Named here because a future policy that+ drops that arm would turn this into a silent no-op rather than an error.++ Returns the ids closed (empty when there was nothing to do, no database is+ configured, or the sweep failed — the caller logs and retries).+ """+ pool = await get_pool()+ if pool is None:+ return []++ try:+ async with pool.acquire() as conn:+ rows = await conn.fetch(+ """+ UPDATE investigation_runs+ SET status = 'failed',+ error = COALESCE(NULLIF(error, ''), $2),+ completed_at = now()+ WHERE id IN (+ SELECT id+ FROM investigation_runs+ WHERE status = 'running'+ AND started_at < now() - make_interval(secs => $1::double precision)+ ORDER BY started_at+ LIMIT $3+ )+ RETURNING id+ """,+ float(older_than_seconds),+ reason,+ int(limit),+ )+ closed = [row["id"] for row in rows]+ if closed:+ logger.warning(+ "ledger.stale_runs_closed",+ count=len(closed),+ older_than_seconds=older_than_seconds,+ )+ return closed+ except Exception as exc: # noqa: BLE001 — the caller backs off and retries+ logger.warning("ledger.fail_stale_runs_failed", error=str(exc))+ return []++def _coerce_uuid(value: Any) -> uuid.UUID | None:try:return uuid.UUID(str(value))diff --git a/services/agents/app/investigator/run_reaper.py b/services/agents/app/investigator/run_reaper.pynew file mode 100644index 000000000..c23b4ceb1--- /dev/null+++ b/services/agents/app/investigator/run_reaper.py@@ -0,0 +1,158 @@+"""The loop that closes investigation runs orphaned by a dead process.++Why this exists alongside the deadline+--------------------------------------+``app.api.investigate._run_and_store`` now runs under the shared wall-clock+budget, so a run that hangs inside a live process ends as ``failed`` and says+why. That covers nothing at all once the process is gone: the run store is a+module-level dict, so a restart, an OOM kill or an ordinary rollout drops every+in-flight run on the floor while its ``investigation_runs`` row stays+``'running'``. Nothing else in the service will ever write a terminal status to+it, and the Investigation Ledger renders exactly that row.++How long is "stale"+-------------------+Derived from the same budget the deadline uses, never restated. A run cannot+legitimately outlast ``AISOC_INVESTIGATION_MAX_SECONDS`` any more, so a row+still ``running`` several multiples past it belongs to a process that is not+coming back. The multiplier plus a floor is the margin against clock skew and+a replica that is merely slow to write its terminal status; an operator who+needs a different one sets ``AISOC_INVESTIGATION_STALE_AFTER_SECONDS``.++Three properties, matching ``app.playbook.sweeper``+---------------------------------------------------+**Safe on several replicas.** The staleness predicate lives inside the+``UPDATE``, so two reapers racing one row close it once.++**Says so when it cannot do its job.** Every failed pass is logged at+``warning`` with the reason, and consecutive failures back off rather than+hammering a store that is down.++**Off only by explicit opt-out.** The failure mode of not running it is the+defect it exists to fix, not the safe state.+"""++from __future__ import annotations++import asyncio+import contextlib+import logging+import os++from app.graph.runner import default_budget+from app.investigator import ledger++logger = logging.getLogger("aisoc.investigator.run_reaper")++#: How often to look. Orphans only appear at process death, so this is a query+#: against an indexed ``status`` column on a table that is empty of stale rows+#: almost always; a minute of latency on a run that has already been abandoned+#: for several multiples of its budget costs nothing.+DEFAULT_INTERVAL_SECONDS = float(os.getenv("AISOC_INVESTIGATION_REAP_INTERVAL_SECONDS", "60"))++#: Multiples of the run budget before a row counts as abandoned.+_STALE_BUDGET_MULTIPLE = 3.0++#: Floor for the staleness margin, so a deployment that sets a very small+#: budget for its own testing does not start reaping runs that a replica is+#: still writing the terminal status for.+_MIN_STALE_SECONDS = 300.0++#: Ceiling for the backoff a failing store earns.+_MAX_BACKOFF_SECONDS = 600.0++#: What the closed row says. An operator reading the ledger needs to know this+#: was a reap rather than a verdict the agent reached.+STALE_REASON = "Investigation abandoned: the process running it exited before writing a result."+++def stale_after_seconds() -> float:+ """Seconds a run may sit at ``running`` before it counts as abandoned.++ Derived from ``AISOC_INVESTIGATION_MAX_SECONDS`` so the reaper and the+ deadline cannot disagree about how long a run is allowed to take — two+ independently written numbers for one budget is how the shorter one wins+ silently. ``AISOC_INVESTIGATION_STALE_AFTER_SECONDS`` overrides it outright+ for an operator who wants a different margin.+ """+ raw = os.getenv("AISOC_INVESTIGATION_STALE_AFTER_SECONDS", "").strip()+ if raw:+ try:+ return float(raw)+ except ValueError:+ logger.warning(+ "run_reaper: AISOC_INVESTIGATION_STALE_AFTER_SECONDS=%r is not a number; deriving from the run budget",+ raw,+ )+ return max(default_budget().max_seconds * _STALE_BUDGET_MULTIPLE, _MIN_STALE_SECONDS)+++def enabled() -> bool:+ """On unless explicitly disabled, like the playbook sweeper."""+ return os.getenv("AISOC_INVESTIGATION_REAPER_DISABLE", "").strip().lower() not in {"1", "true", "yes", "on"}+++async def reap_once() -> int:+ """One pass. Returns how many rows were closed."""+ closed = await ledger.fail_stale_runs(+ older_than_seconds=stale_after_seconds(),+ reason=STALE_REASON,+ )+ if closed:+ logger.info("run_reaper: closed %d abandoned investigation run(s)", len(closed))+ return len(closed)+++async def run_forever(*, interval_seconds: float | None = None) -> None:+ """Reap on a fixed cadence until cancelled."""+ interval = interval_seconds or DEFAULT_INTERVAL_SECONDS+ consecutive_failures = 0+ while True:+ try:+ await reap_once()+ consecutive_failures = 0+ delay = interval+ except asyncio.CancelledError:+ raise+ except Exception as exc: # noqa: BLE001 — a failed pass must not end the loop+ consecutive_failures += 1+ delay = min(interval * (2**consecutive_failures), _MAX_BACKOFF_SECONDS)+ logger.warning(+ "run_reaper: pass %d failed (%s); next attempt in %.0fs",+ consecutive_failures,+ exc,+ delay,+ )+ await asyncio.sleep(delay)+++def start(app: object) -> asyncio.Task | None:+ """Start the reaper as a background task on the app, or say why not."""+ if not enabled():+ logger.info("run_reaper: disabled by AISOC_INVESTIGATION_REAPER_DISABLE")+ return None+ task = asyncio.create_task(run_forever())+ task.add_done_callback(_log_exit)+ with contextlib.suppress(AttributeError):+ app.state.investigation_reaper_task = task # type: ignore[attr-defined]+ logger.info(+ "run_reaper: started, every %.0fs, closing runs older than %.0fs",+ DEFAULT_INTERVAL_SECONDS,+ stale_after_seconds(),+ )+ return task+++def _log_exit(task: asyncio.Task) -> None:+ """Retrieve the task's outcome so a dead reaper is in the log.++ The same pairing the triage worker, the fusion consumer and the playbook+ sweeper carry. A background task whose exception is never retrieved dies+ quietly, and the only symptom is orphaned runs accumulating — which looks+ like an agent problem rather than a missing reaper.+ """+ if task.cancelled():+ return+ exc = task.exception()+ if exc is not None:+ logger.error("run_reaper: exited on %s; abandoned runs will stay 'running'", exc)diff --git a/services/agents/app/main.py b/services/agents/app/main.pyindex 4fc0f628a..3755932a3 100644--- a/services/agents/app/main.py+++ b/services/agents/app/main.py@@ -34,6 +34,7 @@ __all__ = ["score_agent_quality"]from app.hunt import scheduler as hunt_schedulerfrom app.hunt import store as hunt_storefrom app.investigator import ledger as investigation_ledger+from app.investigator import run_reaper as investigation_run_reaperfrom app.llm.factory import preflight_llmfrom app.playbook import PlaybookStorefrom app.playbook import sweeper as playbook_sweeper@@ -148,6 +149,16 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:except Exception as exc: # noqa: BLE001 — never block API startuplogger.warning("playbook_sweeper.start_failed", error=str(exc))+ # Issue #1242 — close investigation runs orphaned by a process that died.+ # The wall-clock deadline on the manual path only ends a run whose process+ # is still alive; the run store is a module dict, so a restart leaves the+ # `investigation_runs` row at 'running' with nothing left to finish it.+ app.state.investigation_reaper_task = None+ try:+ investigation_run_reaper.start(app)+ except Exception as exc: # noqa: BLE001 — never block API startup+ logger.warning("investigation_run_reaper.start_failed", error=str(exc))+# Phase 2.6 — flip /readyz to 200 once startup work is done.app.state.mark_ready()@@ -168,6 +179,9 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:if getattr(app.state, "playbook_sweeper_task", None) is not None:app.state.playbook_sweeper_task.cancel()+ if getattr(app.state, "investigation_reaper_task", None) is not None:+ app.state.investigation_reaper_task.cancel()+# Stop the hunt scheduler before draining DB pools so in-flight runs# can flush their writes.try:diff --git a/services/agents/tests/test_investigation_run_deadline.py b/services/agents/tests/test_investigation_run_deadline.pynew file mode 100644index 000000000..4611d722a--- /dev/null+++ b/services/agents/tests/test_investigation_run_deadline.py@@ -0,0 +1,221 @@+"""A manual investigation must reach a terminal status (issue #1242).++The auto-triage path has had a wall-clock budget since issue #569:+``app.graph.runner._run`` wraps the graph in ``asyncio.timeout(budget.max_seconds)``+and logs ``investigation.budget_timeout``. The console's path is a different+one — ``POST /api/v1/cases/{id}/investigate`` reaches+:func:`app.api.investigate.launch_investigation`, which writes+``status: "running"`` and hands :func:`app.api.investigate._run_and_store` to+``BackgroundTasks`` — and it had no deadline at all. ``_run_and_store`` wrote a+terminal status only on a ``done`` event, an ``error`` event, or a raised+exception, so a model that merely answers slowly (``ChatOpenAI`` is built with+no timeout) left both the in-memory run entry and the ``investigation_runs``+row at ``running`` forever.++Three properties are pinned here, because the fix is wrong if any one is+missing:++**The deadline exists.** A stream that never completes must still end in a+terminal status, not hang until the process dies.++**It is the same budget, not a second one.** Two constants describing one+deadline is a shape this repository has already been bitten by, so the value+comes from ``app.graph.runner.default_budget()`` — i.e. the single+``AISOC_INVESTIGATION_MAX_SECONDS`` knob the auto-triage path reads.++**It reaches the database.** The console polls the API for the run entry but+the Investigation Ledger renders ``investigation_runs``. A timeout that only+updates the process dict leaves the ledger showing a run that never ends.+"""++from __future__ import annotations++import asyncio+import sys+from collections.abc import AsyncIterator+from pathlib import Path+from typing import Any+from uuid import UUID, uuid4++import pytest++_AGENTS_ROOT = Path(__file__).resolve().parents[1]+if str(_AGENTS_ROOT) not in sys.path:+ sys.path.insert(0, str(_AGENTS_ROOT))++from app.api import investigate as investigate_mod # noqa: E402++#: The budget the tests drive the code with. Short enough that a wall-clock+#: assertion is cheap, long enough that it is clearly a deadline firing rather+#: than the coroutine never having started.+_TEST_BUDGET_SECONDS = 0.25++#: How long the test itself is willing to wait. Comfortably above the budget,+#: so a failure here means "no deadline fired", never "the runner was slow".+_TEST_PATIENCE_SECONDS = 15.0++#: Terminal statuses the console already renders. ``CaseWorkspace.tsx`` branches+#: on ``idle``/``starting``/``running``/``failed``/``completed`` and the+#: WebSocket tail in ``investigate.py`` breaks on ``completed``/``failed``, so a+#: new vocabulary word here would be a run that no surface can finish.+_TERMINAL = {"completed", "failed"}+++class _HungStream:+ """A stream that starts, emits one step, and then never completes.++ This is the shape of the defect: not an exception and not an empty+ generator, but a model call that is still outstanding. ``_run_and_store``+ treats everything short of ``done``/``error`` as progress.+ """++ def __init__(self) -> None:+ self.cancelled = False++ async def __call__(self, **_kwargs: Any) -> AsyncIterator[dict[str, Any]]:+ yield {"type": "step", "kind": "llm_request", "agent": "recon", "summary": "asking the model"}+ try:+ await asyncio.sleep(3600)+ except asyncio.CancelledError:+ self.cancelled = True+ raise+ yield {"type": "done", "state": {}} # pragma: no cover — unreachable+++def hung_stream(monkeypatch: pytest.MonkeyPatch) -> _HungStream:+ """Replace the orchestrator stream with one that never finishes."""+ stream = _HungStream()+ monkeypatch.setattr(investigate_mod, "_investigate_stream", stream, raising=True)+ # The realtime fan-out is a best-effort HTTP POST; keep the test off the+ # network so a 2s connect timeout per step cannot be mistaken for the+ # deadline under test.+ monkeypatch.setattr(investigate_mod, "_emit_event", _noop_emit, raising=True)+ return stream+++async def _noop_emit(*_args: Any, **_kwargs: Any) -> None:+ return None+++def budget(monkeypatch: pytest.MonkeyPatch) -> float:+ monkeypatch.setenv("AISOC_INVESTIGATION_MAX_SECONDS", str(_TEST_BUDGET_SECONDS))+ return _TEST_BUDGET_SECONDS+++def run_entry(monkeypatch: pytest.MonkeyPatch) -> tuple[str, dict[str, Any]]:+ """Seed ``_runs`` exactly as ``launch_investigation`` does, isolated."""+ runs: dict[str, dict[str, Any]] = {}+ monkeypatch.setattr(investigate_mod, "_runs", runs, raising=True)+ run_id = str(uuid4())+ runs[run_id] = {"run_id": run_id, "case_id": "case-1242", "status": "running"}+ return run_id, runs[run_id]+++async def _drive(run_id: str) -> None:+ """Run the background task the endpoint schedules, under the test's patience."""+ req = investigate_mod.InvestigateRequest(+ alert_summary="Impossible travel on a privileged account",+ raw_alert={"user": "svc-deploy"},+ tenant_id="default",+ )+ try:+ await asyncio.wait_for(+ investigate_mod._run_and_store(run_id, "case-1242", req),+ timeout=_TEST_PATIENCE_SECONDS,+ )+ except TimeoutError:+ pytest.fail(+ f"_run_and_store did not return within {_TEST_PATIENCE_SECONDS}s against a stream that "+ f"never completes, with AISOC_INVESTIGATION_MAX_SECONDS={_TEST_BUDGET_SECONDS}. "+ "The manual investigation path has no wall-clock deadline, so the run stays 'running' "+ "forever and the console shows a spinner that never resolves (issue #1242)."+ )+++async def test_hung_stream_still_reaches_a_terminal_status(+ budget: float,+ hung_stream: _HungStream,+ run_entry: tuple[str, dict[str, Any]],+) -> None:+ run_id, entry = run_entry++ await _drive(run_id)++ assert entry["status"] in _TERMINAL, (+ f"run ended at status {entry['status']!r}; the console only renders {sorted(_TERMINAL)} as finished states"+ )+ assert entry.get("error"), "a run that timed out must say so, not finish silently"+ assert hung_stream.cancelled, "the outstanding model call must be cancelled, not left running"+++async def test_deadline_is_the_shared_budget_not_a_second_constant(+ budget: float,+ hung_stream: _HungStream,+ run_entry: tuple[str, dict[str, Any]],+) -> None:+ """``AISOC_INVESTIGATION_MAX_SECONDS`` is the only knob, on both paths.++ Asserted through the env var rather than by reading a module attribute:+ a second hardcoded number would satisfy an attribute check and still+ ignore the operator's setting.+ """+ run_id, entry = run_entry++ loop = asyncio.get_running_loop()+ started = loop.time()+ await _drive(run_id)+ elapsed = loop.time() - started++ assert elapsed < _TEST_PATIENCE_SECONDS / 2, (+ f"run took {elapsed:.1f}s against a {budget}s budget — the deadline is not reading AISOC_INVESTIGATION_MAX_SECONDS"+ )+ assert str(budget) in str(entry.get("error", "")), (+ f"the failure message {entry.get('error')!r} does not quote the configured budget "+ f"({budget}s), so an operator cannot tell which deadline fired"+ )+++async def test_timeout_writes_the_terminal_status_to_the_ledger_row(+ budget: float,+ hung_stream: _HungStream,+ run_entry: tuple[str, dict[str, Any]],+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """The ``investigation_runs`` row must finish too, not just the process dict.++ ``InvestigationLedger.tsx`` renders that row. ``complete_run`` fires from+ the orchestrator's ``done`` and ``except`` arms only, and neither runs when+ the task is cancelled from outside — ``CancelledError`` is a+ ``BaseException``, so the orchestrator's ``except Exception`` does not see+ it.+ """+ run_id, _entry = run_entry+ completions: list[dict[str, Any]] = []+ tenant_uuid = uuid4()++ async def _fake_resolve_tenant(tenant_ref: str) -> UUID:+ assert tenant_ref == "default"+ return tenant_uuid++ async def _fake_complete_run(**kwargs: Any) -> None:+ completions.append(kwargs)++ monkeypatch.setattr(investigate_mod.ledger, "resolve_tenant", _fake_resolve_tenant, raising=True)+ monkeypatch.setattr(investigate_mod.ledger, "complete_run", _fake_complete_run, raising=True)++ await _drive(run_id)++ assert completions, (+ "no ledger write on timeout — investigation_runs.status stays 'running' and the Investigation Ledger shows a run that never ends"+ )+ (call,) = completions+ assert call["status"] == "failed", f"ledger row closed as {call['status']!r}"+ assert call["tenant_id"] == tenant_uuid+ assert str(call["run_id"]) == run_id, "the ledger row closed must be the one the endpoint opened"+ assert call.get("error"), "the ledger row must carry the reason"diff --git a/services/agents/tests/test_investigation_run_reaper.py b/services/agents/tests/test_investigation_run_reaper.pynew file mode 100644index 000000000..d15e1b5aa--- /dev/null+++ b/services/agents/tests/test_investigation_run_reaper.py@@ -0,0 +1,277 @@+"""Runs orphaned by a dead process must still reach a terminal status (#1242).++The wall-clock deadline added to ``app.api.investigate._run_and_store`` ends a+run that hangs inside a live process. It covers nothing once the process is+gone: the run store is a module-level dict, so a restart, an OOM kill or an+ordinary rollout drops every in-flight run while its ``investigation_runs`` row+stays ``'running'`` — and the Investigation Ledger renders that row.++Two layers, tested separately on purpose:++* the policy (how long is stale, and that the number is derived from the same+ budget rather than written a second time) — plain unit tests;+* the SQL, against a **real** Postgres carrying the production schema. A fake+ pool would answer whatever query it was handed, which is exactly the double+ that is more capable than the real thing: it cannot tell you that+ ``make_interval`` binds, that the conditional ``UPDATE`` is race-safe, or+ that the RLS policy on the table lets a cross-tenant sweep see anything at+ all. Set ``AISOC_TEST_LEDGER_DSN`` to run that half.+"""++from __future__ import annotations++import os+import sys+import uuid+from pathlib import Path+from typing import Any++import pytest++_AGENTS_ROOT = Path(__file__).resolve().parents[1]+if str(_AGENTS_ROOT) not in sys.path:+ sys.path.insert(0, str(_AGENTS_ROOT))++from app.investigator import ledger, run_reaper # noqa: E402++LEDGER_DSN = os.environ.get("AISOC_TEST_LEDGER_DSN", "").strip()+++# ---------------------------------------------------------------------------+# Policy: one budget, not two+# ---------------------------------------------------------------------------+++def test_stale_margin_is_derived_from_the_run_budget(monkeypatch: pytest.MonkeyPatch) -> None:+ """Not a second hardcoded deadline — the reaper tracks the same knob.++ A margin written independently of the budget is how the two disagree, and+ a reaper that is shorter than the deadline cancels healthy work.+ """+ monkeypatch.delenv("AISOC_INVESTIGATION_STALE_AFTER_SECONDS", raising=False)++ monkeypatch.setenv("AISOC_INVESTIGATION_MAX_SECONDS", "600")+ assert run_reaper.stale_after_seconds() == 1800.0++ monkeypatch.setenv("AISOC_INVESTIGATION_MAX_SECONDS", "900")+ assert run_reaper.stale_after_seconds() == 2700.0+++def test_stale_margin_is_always_longer_than_the_deadline(monkeypatch: pytest.MonkeyPatch) -> None:+ """The safety property: never reap a run that is still allowed to be running."""+ from app.graph.runner import default_budget++ monkeypatch.delenv("AISOC_INVESTIGATION_STALE_AFTER_SECONDS", raising=False)+ for budget in ("0.25", "1", "120", "600", "3600"):+ monkeypatch.setenv("AISOC_INVESTIGATION_MAX_SECONDS", budget)+ assert run_reaper.stale_after_seconds() > default_budget().max_seconds, (+ f"with a {budget}s budget the reaper would close runs that are still within it"+ )+++def test_operator_override_wins(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("AISOC_INVESTIGATION_MAX_SECONDS", "120")+ monkeypatch.setenv("AISOC_INVESTIGATION_STALE_AFTER_SECONDS", "7200")+ assert run_reaper.stale_after_seconds() == 7200.0+++def test_a_junk_override_falls_back_rather_than_crashing_the_loop(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("AISOC_INVESTIGATION_MAX_SECONDS", "600")+ monkeypatch.setenv("AISOC_INVESTIGATION_STALE_AFTER_SECONDS", "every-ten-minutes")+ assert run_reaper.stale_after_seconds() == 1800.0+++def test_reaper_is_on_unless_explicitly_disabled(monkeypatch: pytest.MonkeyPatch) -> None:+ """Not running it is the defect, so off is the opt-in."""+ monkeypatch.delenv("AISOC_INVESTIGATION_REAPER_DISABLE", raising=False)+ assert run_reaper.enabled() is True+ monkeypatch.setenv("AISOC_INVESTIGATION_REAPER_DISABLE", "1")+ assert run_reaper.enabled() is False+++async def test_reap_once_passes_the_derived_margin_through(monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[dict[str, Any]] = []++ async def _fake(**kwargs: Any) -> list[uuid.UUID]:+ calls.append(kwargs)+ return [uuid.uuid4()]++ monkeypatch.setattr(ledger, "fail_stale_runs", _fake, raising=True)+ monkeypatch.delenv("AISOC_INVESTIGATION_STALE_AFTER_SECONDS", raising=False)+ monkeypatch.setenv("AISOC_INVESTIGATION_MAX_SECONDS", "600")++ assert await run_reaper.reap_once() == 1+ (call,) = calls+ assert call["older_than_seconds"] == 1800.0+ assert "abandoned" in call["reason"].lower(), "the ledger row must say it was reaped, not carry a verdict the agent never reached"+++async def test_a_failing_pass_does_not_kill_the_loop(monkeypatch: pytest.MonkeyPatch) -> None:+ """An unreachable store is transient; the sweeper backs off and retries."""++ async def _boom(**_kwargs: Any) -> list[uuid.UUID]:+ raise ConnectionError("database is down")++ monkeypatch.setattr(ledger, "fail_stale_runs", _boom, raising=True)+ with pytest.raises(ConnectionError):+ await run_reaper.reap_once()+++# ---------------------------------------------------------------------------+# The SQL, against the real schema+# ---------------------------------------------------------------------------++pytestmark_integration = pytest.mark.skipif(+ not LEDGER_DSN,+ reason="set AISOC_TEST_LEDGER_DSN to a Postgres carrying the AiSOC schema to run the ledger sweep",+)+++async def ledger_db(monkeypatch: pytest.MonkeyPatch):+ """Point the ledger at a real Postgres and give each test a clean tenant."""+ import asyncpg++ monkeypatch.setenv("DATABASE_URL", LEDGER_DSN)+ await ledger.close_pool()++ tenant_id = uuid.uuid4()+ conn = await asyncpg.connect(LEDGER_DSN)+ try:+ await conn.execute(+ "INSERT INTO tenants (id, name, slug) VALUES ($1, $2, $3)",+ tenant_id,+ f"reaper-{tenant_id.hex[:8]}",+ f"reaper-{tenant_id.hex[:8]}",+ )+ finally:+ await conn.close()++ yield tenant_id++ await ledger.close_pool()+ conn = await asyncpg.connect(LEDGER_DSN)+ try:+ await conn.execute("DELETE FROM investigation_runs WHERE tenant_id = $1", tenant_id)+ await conn.execute("DELETE FROM tenants WHERE id = $1", tenant_id)+ finally:+ await conn.close()+++async def _insert_run(dsn: str, tenant_id: uuid.UUID, *, age_seconds: float, status: str) -> uuid.UUID:+ import asyncpg++ run_id = uuid.uuid4()+ conn = await asyncpg.connect(dsn)+ try:+ await conn.execute(+ """+ INSERT INTO investigation_runs (id, tenant_id, case_id, status, started_at, created_at)+ VALUES ($1, $2, $3, $4, now() - make_interval(secs => $5::double precision), now())+ """,+ run_id,+ tenant_id,+ "case-reaper",+ status,+ float(age_seconds),+ )+ finally:+ await conn.close()+ return run_id+++async def _status_of(dsn: str, run_id: uuid.UUID) -> tuple[str, str | None, bool]:+ import asyncpg++ conn = await asyncpg.connect(dsn)+ try:+ row = await conn.fetchrow(+ "SELECT status, error, completed_at IS NOT NULL AS closed FROM investigation_runs WHERE id = $1",+ run_id,+ )+ finally:+ await conn.close()+ return row["status"], row["error"], row["closed"]+++@pytestmark_integration+async def test_sweep_closes_an_abandoned_run_and_leaves_a_live_one_alone(ledger_db) -> None:+ tenant_id = ledger_db+ abandoned = await _insert_run(LEDGER_DSN, tenant_id, age_seconds=4000, status="running")+ in_flight = await _insert_run(LEDGER_DSN, tenant_id, age_seconds=5, status="running")+ already_done = await _insert_run(LEDGER_DSN, tenant_id, age_seconds=4000, status="completed")++ closed = await ledger.fail_stale_runs(older_than_seconds=1800, reason=run_reaper.STALE_REASON)++ assert abandoned in closed+ assert in_flight not in closed, "a run still inside its budget was reaped"+ assert already_done not in closed, "a finished run was reopened and re-closed"++ status, error, has_completed_at = await _status_of(LEDGER_DSN, abandoned)+ assert status == "failed"+ assert error == run_reaper.STALE_REASON+ assert has_completed_at, "a closed run with no completed_at renders as 'in progress' in the ledger"++ assert (await _status_of(LEDGER_DSN, in_flight))[0] == "running"+ assert (await _status_of(LEDGER_DSN, already_done))[0] == "completed"+++@pytestmark_integration+async def test_sweep_is_idempotent_and_safe_to_race(ledger_db) -> None:+ """Two replicas sweeping the same row close it once.++ The staleness predicate is inside the ``UPDATE``, so the loser matches+ nothing rather than overwriting the first writer's reason.+ """+ tenant_id = ledger_db+ run_id = await _insert_run(LEDGER_DSN, tenant_id, age_seconds=4000, status="running")++ first = await ledger.fail_stale_runs(older_than_seconds=1800, reason=run_reaper.STALE_REASON)+ second = await ledger.fail_stale_runs(older_than_seconds=1800, reason="a different reason")++ assert run_id in first+ assert run_id not in second, "a second pass re-closed a row that was already terminal"+ assert (await _status_of(LEDGER_DSN, run_id))[1] == run_reaper.STALE_REASON+++@pytestmark_integration+async def test_sweep_sees_every_tenant(ledger_db) -> None:+ """A maintenance sweep that only saw one tenant would leave the rest hanging.++ This is the assertion that would fail if the RLS policy on+ ``investigation_runs`` ever dropped its ``current_tenant_id() IS NULL``+ arm — the thing the sweep silently depends on.+ """+ import asyncpg++ tenant_a = ledger_db+ tenant_b = uuid.uuid4()+ conn = await asyncpg.connect(LEDGER_DSN)+ try:+ await conn.execute(+ "INSERT INTO tenants (id, name, slug) VALUES ($1, $2, $3)",+ tenant_b,+ f"reaper-b-{tenant_b.hex[:8]}",+ f"reaper-b-{tenant_b.hex[:8]}",+ )+ finally:+ await conn.close()++ try:+ run_a = await _insert_run(LEDGER_DSN, tenant_a, age_seconds=4000, status="running")+ run_b = await _insert_run(LEDGER_DSN, tenant_b, age_seconds=4000, status="running")++ closed = await ledger.fail_stale_runs(older_than_seconds=1800, reason=run_reaper.STALE_REASON)++ assert run_a in closed+ assert run_b in closed, "the sweep only saw one tenant's abandoned runs"+ finally:+ conn = await asyncpg.connect(LEDGER_DSN)+ try:+ await conn.execute("DELETE FROM investigation_runs WHERE tenant_id = $1", tenant_b)+ await conn.execute("DELETE FROM tenants WHERE id = $1", tenant_b)+ finally:+ await conn.close()diff --git a/services/agents/tests/test_litellm_config.py b/services/agents/tests/test_litellm_config.pyindex 0cb9ed5a1..53b638fba 100644--- a/services/agents/tests/test_litellm_config.py+++ b/services/agents/tests/test_litellm_config.py@@ -87,3 +87,61 @@ def test_model_pins_roles_match_gateway_aliases_exactly():# Ties the code's logical roles to the gateway config: every pinned role has# an ``aisoc-<role>`` alias in the config, and vice versa — no drift either way.assert {f"aisoc-{role}" for role in all_roles()} == EXPECTED_ALIASES+++# ---------------------------------------------------------------------------+# Per-request timeout (issue #1241)+# ---------------------------------------------------------------------------++TIMEOUT_ENV_VAR = "AISOC_LITELLM_REQUEST_TIMEOUT"+COMPOSE_PATH = REPO_ROOT / "docker-compose.yml"+++def test_request_timeout_is_operator_adjustable_without_editing_this_file():+ """60s was a bare literal in a file every other value reads from the env.++ The bundled 3B model on CPU-only inference can exceed it, and every call+ then times out while Settings still reports the provider Live. Raising it+ must not require editing a mounted config and rebuilding.+ """+ timeout = _load().get("litellm_settings", {}).get("request_timeout")+ assert timeout is not None, "request_timeout must be declared — without it the gateway's own 6000s default applies"+ assert str(timeout) == f"os.environ/{TIMEOUT_ENV_VAR}", (+ f"request_timeout is {timeout!r}; an operator cannot change that without editing this file"+ )+++def test_compose_supplies_a_default_for_the_timeout_variable():+ """The default is load-bearing, not cosmetic.++ LiteLLM resolves ``os.environ/...`` with ``get_secret``, which returns+ ``None`` for an unset variable and assigns that straight onto+ ``litellm.request_timeout``. Measured against+ ``ghcr.io/berriai/litellm:main-stable``, a gateway in that state answers+ 200 on ``/health/liveliness`` and fails *every* completion with+ ``float() argument must be a string or a real number, not 'NoneType'`` —+ a healthy-looking gateway serving nothing, which is the shape of the defect+ this knob exists to fix. So the variable must always have a value on the+ supported path.+ """+ compose = COMPOSE_PATH.read_text(encoding="utf-8")+ assert f"{TIMEOUT_ENV_VAR}: ${{{TIMEOUT_ENV_VAR}:-" in compose, (+ f"docker-compose.yml must pass {TIMEOUT_ENV_VAR} to the litellm service with a `:-` default; "+ "without one an unset variable silently breaks every completion"+ )+++def test_num_retries_stays_a_literal():+ """Not every setting can be env-driven, and this one must not be.++ LiteLLM hands ``router_settings`` to ``Router.__init__`` with no type+ coercion, and the retry path evaluates ``num_retries > 0`` — which raises+ ``TypeError`` against the string an ``os.environ/...`` reference resolves+ to, at exactly the moment a retry is needed. Pinned so a later "make+ everything configurable" pass cannot quietly break failover.+ """+ num_retries = _load().get("router_settings", {}).get("num_retries")+ assert isinstance(num_retries, int), (+ f"num_retries must stay an int literal, got {num_retries!r}: "+ "LiteLLM does not coerce router_settings and `num_retries > 0` raises on a string"+ )diff --git a/services/api/app/api/v1/endpoints/llm_status.py b/services/api/app/api/v1/endpoints/llm_status.pyindex 2968a17a8..3e82b5433 100644--- a/services/api/app/api/v1/endpoints/llm_status.py+++ b/services/api/app/api/v1/endpoints/llm_status.py@@ -66,6 +66,7 @@ from app.core.airgap import is_host_allowed_for_airgapfrom app.core.config import settingsfrom app.models.llm_credential import TenantLlmCredentialfrom app.security.credential_vault import CredentialVaultError, get_vault+from app.services import llm_healthfrom app.services.llm_runtime import probe as probe_runtimefrom app.services.model_aliases import gateway_url, is_gateway_alias, resolve_api_key@@ -253,6 +254,14 @@ def _compute_status("effective_path": effective_path,"policy_note": policy_note,"source": source,+ # Everything above describes *configuration*: which provider this pod+ # would call and whether the policy would let it. Whether the provider+ # answers is a different question, and nothing here used to ask it — so+ # a deployment timing out on every call rendered the emerald Live pill+ # (issue #1241). These three come from the outcomes of the calls the+ # product already made; see app/services/llm_health.py for why they are+ # not a probe, not a latency figure, and why `unknown` is a real answer.+ **llm_health.snapshot(),}diff --git a/services/api/app/services/llm_health.py b/services/api/app/services/llm_health.pynew file mode 100644index 000000000..72a910e31--- /dev/null+++ b/services/api/app/services/llm_health.py@@ -0,0 +1,122 @@+"""Did the LLM actually answer? (issue #1241)++``GET /api/v1/llm/status`` reports *configuration* — which provider this pod is+pointed at, and whether the air-gap policy would let a call leave. It places no+call, so a deployment whose every completion times out still renders the+emerald **Live** pill in Settings → Deployment & AI. The panel's own header+claims the indicator "cannot drift from runtime behaviour", which is true of+which provider is configured and false of whether it answers.++This module is the missing half: a bounded, in-process record of how the calls+the product is *already making* turned out.++What it deliberately is not+---------------------------+**Not a probe.** Nothing here places a request. A synthetic completion on every+page load would spend the tenant's budget to answer a question the real traffic+has already answered, and on a local 3B model it is not cheap.++**Not a latency number.** No figure for how long a call takes is published+anywhere, because none has been measured for this path. ``aisoc_run_costs``+carries a latency sum, but it is written by ``services/agents`` for completed+runs only and records nothing at all about a call that timed out — which is the+case that matters here.++**Not a ratio.** The verdict is the outcome of the most recent observed call.+A ratio over a short window is noise, and a ratio over a long one keeps+reporting healthy for a provider that died five minutes ago. "The last call we+made did not come back" is the question an operator is asking when they look at+that badge. The counts travel alongside so a blip is distinguishable from an+outage.++Scope, stated rather than implied+---------------------------------+Per-process. Behind several replicas this reports what *this* pod has seen, and+a pod that has served no AI traffic reports ``unknown`` rather than a green it+has not earned. ``unknown`` is a first-class answer here: an absence of evidence+rendered as health is the same defect in a different costume.+"""++from __future__ import annotations++import threading+import time+from collections import deque+from typing import Any, Literal++#: How many outcomes to keep. Enough to say "3 of the last 12 failed" and small+#: enough that the whole thing is a few hundred bytes per process.+_WINDOW_SIZE = 20++#: Outcomes older than this stop counting. A provider that was failing an hour+#: ago says nothing about the one answering now, and an operator reading a red+#: badge needs it to be about the present.+_WINDOW_SECONDS = 900.0++RecentHealth = Literal["healthy", "degraded", "unknown"]++_lock = threading.Lock()+_outcomes: deque[tuple[float, bool, str]] = deque(maxlen=_WINDOW_SIZE)+++def record_success() -> None:+ """Note that a chat-completions call returned."""+ with _lock:+ _outcomes.append((time.monotonic(), True, ""))+++def record_failure(kind: str) -> None:+ """Note that a chat-completions call did not return a usable response.++ ``kind`` is the exception class name (``ReadTimeout``, ``ConnectError``,+ ``HTTPStatusError``…) — never the message, which can carry a URL with a key+ in it.+ """+ with _lock:+ _outcomes.append((time.monotonic(), False, kind))+++def reset() -> None:+ """Drop the window. For tests and for a deliberate operator re-baseline."""+ with _lock:+ _outcomes.clear()+++def snapshot() -> dict[str, Any]:+ """The three fields ``/llm/status`` adds, derived from real outcomes only."""+ cutoff = time.monotonic() - _WINDOW_SECONDS+ with _lock:+ recent = [entry for entry in _outcomes if entry[0] >= cutoff]++ if not recent:+ return {+ "recent_health": "unknown",+ "recent_calls": 0,+ "recent_failures": 0,+ "recent_note": (+ "No LLM call has been observed by this pod recently, so whether the provider "+ "answers is unmeasured. This reports configuration only."+ ),+ }++ failures = [entry for entry in recent if not entry[1]]+ last_ok = recent[-1][1]+ last_kind = recent[-1][2]++ if last_ok:+ note = f"The most recent LLM call succeeded ({len(failures)} of the last {len(recent)} failed)."+ health: RecentHealth = "healthy"+ else:+ note = (+ f"The most recent LLM call failed ({last_kind or 'error'}); "+ f"{len(failures)} of the last {len(recent)} failed. "+ "If the provider is a local model on CPU, raise AISOC_LITELLM_REQUEST_TIMEOUT."+ )+ health = "degraded"++ return {+ "recent_health": health,+ "recent_calls": len(recent),+ "recent_failures": len(failures),+ "recent_note": note,+ }diff --git a/services/api/app/services/llm_safety.py b/services/api/app/services/llm_safety.pyindex cdb62b993..16ede0c33 100644--- a/services/api/app/services/llm_safety.py+++ b/services/api/app/services/llm_safety.py@@ -66,6 +66,7 @@ from app._vendor.llm_contract_rules import (from app._vendor.llm_contract_rules import (validate_messages as validate_messages,)+from app.services import llm_health# These names are re-exports: callers do# `from app.services.llm_safety import LLMContractViolation`, and the rules@@ -110,6 +111,14 @@ async def safe_chat_completions_request(a call site can move between services without rewriting. ``client`` isaccepted so callers that already hold a pooled ``AsyncClient`` — and tests— do not each construct one.++ Every outcome from here is recorded in :mod:`app.services.llm_health`, which+ is what lets ``GET /api/v1/llm/status`` say whether the provider is actually+ answering instead of only which one is configured (issue #1241). This is the+ one function the service's LLM call sites share, so recording here needs no+ probe and no second mechanism. A contract violation is **not** recorded: it+ raises above, before the network call, and is AiSOC refusing to send rather+ than the provider failing to answer."""validated = LLMInputContract.validate(messages)@@ -118,12 +127,20 @@ async def safe_chat_completions_request(headers.update(extra_headers)payload: dict[str, Any] = {"model": model, "messages": validated, **extra_body}- if client is not None:- response = await client.post(url, headers=headers, json=payload, timeout=timeout)- response.raise_for_status()- return dict(response.json())-- async with httpx.AsyncClient(timeout=timeout) as owned:- response = await owned.post(url, headers=headers, json=payload)- response.raise_for_status()- return dict(response.json())+ try:+ if client is not None:+ response = await client.post(url, headers=headers, json=payload, timeout=timeout)+ response.raise_for_status()+ result = dict(response.json())+ else:+ async with httpx.AsyncClient(timeout=timeout) as owned:+ response = await owned.post(url, headers=headers, json=payload)+ response.raise_for_status()+ result = dict(response.json())+ except Exception as exc:+ # Re-raised unchanged — every caller's error handling is untouched. The+ # class name only; a message can carry a URL with a key in it.+ llm_health.record_failure(type(exc).__name__)+ raise+ llm_health.record_success()+ return resultdiff --git a/services/api/tests/test_llm_status_degraded.py b/services/api/tests/test_llm_status_degraded.pynew file mode 100644index 000000000..3cdc7bbd2--- /dev/null+++ b/services/api/tests/test_llm_status_degraded.py@@ -0,0 +1,176 @@+"""The LLM panel must not report Live while every call is failing (issue #1241).++``GET /api/v1/llm/status`` places no call. ``_compute_status`` decides+``effective_path`` from ``key_set`` and the air-gap policy alone, so a+deployment whose every completion times out renders the emerald **Live** pill+in Settings → Deployment & AI. The panel's own header says the indicator+"cannot drift from runtime behaviour", which is true of *which provider is+configured* and false of *whether it answers*.++What is added is a recent-outcome signal, not a probe: ``safe_chat_completions_request``+is the single chokepoint every LLM call in this service already goes through,+so the calls the product is making anyway are what the status reports. No+synthetic completion on page load, and no latency number — the honest answer to+"how fast is it" is one nobody has measured here.++The three states are deliberate, and ``unknown`` is the important one: a pod+that has placed no call has observed nothing, and saying so is different from+saying the provider is healthy. Reporting a fabricated green for an absence of+data is the defect in a different costume.+"""++from __future__ import annotations++import httpx+import pytest+from app.api.v1.endpoints.llm_status import llm_status+from app.services import llm_health+from app.services.llm_safety import safe_chat_completions_request++_MESSAGES = [{"role": "user", "content": "Summarise this alert for an analyst."}]++[email protected](autouse=True)+def _clean(monkeypatch: pytest.MonkeyPatch):+ """A live-looking configuration, and an empty outcome window."""+ for var in (+ "LLM_BASE_URL",+ "LLM_MODEL",+ "OPENAI_BASE_URL",+ "OPENAI_MODEL",+ "AISOC_LLM_MODEL",+ "LLM_GATEWAY_URL",+ "LITELLM_MASTER_KEY",+ ):+ monkeypatch.delenv(var, raising=False)+ monkeypatch.setenv("OPENAI_API_KEY", "sk-test")+ monkeypatch.delenv("LLM_API_KEY", raising=False)+ llm_health.reset()+ yield+ llm_health.reset()+++async def _call_through_the_real_chokepoint(handler) -> None:+ """Drive the real ``safe_chat_completions_request`` over a mock transport.++ Deliberately not a direct call to the recorder: the property under test is+ that the function the endpoints already use records what happened, and a+ test that records the outcome itself would pass against a recorder nothing+ calls.+ """+ async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:+ await safe_chat_completions_request(+ api_key="sk-test",+ model="aisoc-triage",+ messages=_MESSAGES,+ url="http://litellm:4000/v1/chat/completions",+ client=client,+ )+++def _ok(_request: httpx.Request) -> httpx.Response:+ return httpx.Response(200, json={"choices": [{"message": {"content": "ok"}}]})+++def _timeout(request: httpx.Request) -> httpx.Response:+ raise httpx.ReadTimeout("timed out", request=request)+++# ---------------------------------------------------------------------------+# The status surface+# ---------------------------------------------------------------------------+++def test_no_observed_calls_reports_unknown_not_healthy() -> None:+ """An absence of evidence is reported as such, never as a green light."""+ status = llm_status()+ assert status["recent_health"] == "unknown"+ assert status["effective_path"] == "live", "configuration is still live — only the outcome is unknown"+++async def test_a_timing_out_provider_is_reported_degraded() -> None:+ """The headline case: configured, reachable-looking, and answering nothing."""+ for _ in range(3):+ with pytest.raises(httpx.ReadTimeout):+ await _call_through_the_real_chokepoint(_timeout)++ status = llm_status()+ assert status["recent_health"] == "degraded", "every recent call timed out and the panel would still render the emerald Live pill"+ assert status["recent_failures"] == 3+ assert "timeout" in str(status["recent_note"]).lower() or "fail" in str(status["recent_note"]).lower(), (+ f"the operator is told nothing actionable: {status['recent_note']!r}"+ )+++async def test_a_working_provider_is_reported_healthy() -> None:+ await _call_through_the_real_chokepoint(_ok)+ status = llm_status()+ assert status["recent_health"] == "healthy"+ assert status["recent_failures"] == 0+++async def test_recovery_clears_the_degraded_state() -> None:+ """A provider that comes back must not stay red — this is a window, not a latch."""+ for _ in range(3):+ with pytest.raises(httpx.ReadTimeout):+ await _call_through_the_real_chokepoint(_timeout)+ assert llm_status()["recent_health"] == "degraded"++ for _ in range(5):+ await _call_through_the_real_chokepoint(_ok)++ assert llm_status()["recent_health"] == "healthy", "a recovered provider stayed red"+++async def test_an_http_error_counts_as_a_failure_too() -> None:+ """A 500 from the gateway is a failed call, not just a timeout."""++ def _five_hundred(_request: httpx.Request) -> httpx.Response:+ return httpx.Response(500, json={"error": "upstream exploded"})++ for _ in range(3):+ with pytest.raises(httpx.HTTPStatusError):+ await _call_through_the_real_chokepoint(_five_hundred)++ assert llm_status()["recent_health"] == "degraded"+++def test_status_publishes_no_latency_figure() -> None:+ """No measured latency exists for this path, so none is published.++ Guards against a future 'helpful' average that would be the fabricated+ number the issue warns about.+ """+ status = llm_status()+ assert not [k for k in status if "latency" in k or "ms" in k.lower()], (+ f"status surfaced a latency-shaped field nobody measured: {sorted(status)}"+ )+++# ---------------------------------------------------------------------------+# The recorder itself+# ---------------------------------------------------------------------------+++async def test_a_contract_violation_is_not_counted_as_a_provider_failure() -> None:+ """Refusing to send is AiSOC working correctly, not the provider failing.++ ``LLMInputContract`` aborts before the network call. Counting that as a+ provider outage would make the panel blame the model for a prompt the+ product deliberately refused to send.+ """+ from app.services.llm_safety import LLMContractViolation++ raw_ocsf = '{"class_uid": 2001, "activity_id": 1, "metadata": {"product": {"name": "x"}, "version": "1.0"}}'+ with pytest.raises(LLMContractViolation):+ async with httpx.AsyncClient(transport=httpx.MockTransport(_ok)) as client:+ await safe_chat_completions_request(+ api_key="sk-test",+ model="aisoc-triage",+ messages=[{"role": "user", "content": raw_ocsf}],+ client=client,+ )++ status = llm_status()+ assert status["recent_health"] == "unknown", "a refused prompt was recorded as a provider outcome; nothing was ever sent"+ assert status["recent_failures"] == 0diff --git a/tests/test_env_placeholder_gate.py b/tests/test_env_placeholder_gate.pyindex 99be4246e..e7f50b6e1 100644--- a/tests/test_env_placeholder_gate.py+++ b/tests/test_env_placeholder_gate.py@@ -114,6 +114,12 @@ DELIBERATE_DEFAULTS: dict[str, str] = {"OPENAI_MODEL": "a logical task alias the gateway defines","LITELLM_MASTER_KEY": "the local gateway's own key; the gateway is in-network only","LLM_GATEWAY_URL": "the compose hostname of the gateway",+ "AISOC_LITELLM_REQUEST_TIMEOUT": (+ "a duration in seconds, and the value the gateway actually runs with. It "+ "was a literal inside infra/litellm/config.yaml, so raising it meant "+ "editing a mounted file; it is surfaced here because CPU-only inference "+ "on the bundled 3B model is the deployment that has to raise it"+ ),"TAXII_FEEDS": "a real public MITRE ATT&CK TAXII endpoint","CISA_KEV_ENABLED": "a boolean","ABUSE_CH_ENABLED": "a boolean",