Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

fix(agents,api): bound the manual investigation path and stop the LLM panel claiming Live while every call fails (#1284)

Beenu Arorae10afaa8a53e
diff --git a/.env.example b/.env.example
index fba11f0d3..1b818a2d5 100644
--- a/.env.example
+++ b/.env.example
@@ -245,6 +245,14 @@ LITELLM_MASTER_KEY=sk-aisoc-local
# default deployment.
LLM_GATEWAY_URL=http://litellm:4000/v1
#
+# How long the gateway waits on ONE generation before giving up, in seconds.
+# 60 used to be a literal inside infra/litellm/config.yaml, so raising it meant
+# editing a mounted file. The bundled 3B model doing CPU-only inference can
+# exceed 60s on a long generation, and every call then fails while Settings →
+# Deployment & AI still reports the provider as configured. That panel now says
+# "Degraded" when recent calls are failing — this is the knob to turn.
+AISOC_LITELLM_REQUEST_TIMEOUT=60
+#
# ENABLE LIVE LLM:
# (a) Through the gateway — the default, and already wired. Put your real
# provider key in OPENAI_API_KEY above (the gateway reads it) and bring up
diff --git a/apps/docs/docs/deployment/env-vars.md b/apps/docs/docs/deployment/env-vars.md
index 8193af4f0..a28f91f93 100644
--- a/apps/docs/docs/deployment/env-vars.md
+++ b/apps/docs/docs/deployment/env-vars.md
@@ -274,6 +274,11 @@ Source: [`services/agents/app/`](https://github.com/beenuar/AiSOC/tree/main/serv
| `OPENAI_API_KEY` | — | Provider key. With the bundled gateway running, the *gateway* uses this to reach the upstream model; AiSOC itself authenticates with `LITELLM_MASTER_KEY`. |
| `LLM_GATEWAY_URL` | `http://litellm:4000/v1` (set by compose) | In-network URL of the bundled LiteLLM gateway. Both resolvers read it for any `aisoc-<role>` alias — an alias resolves nowhere else. A concrete `AISOC_MODEL_PIN_<ROLE>` is left pointing at its provider. |
| `LITELLM_MASTER_KEY` | `sk-aisoc-local` in compose | Bearer AiSOC sends when it routes to the gateway itself. Resolved *with* the URL, so a provider key is never sent to the gateway (it rejects one). |
+| `AISOC_LITELLM_REQUEST_TIMEOUT` | `60` (seconds, set by compose) | How long the gateway waits on one generation before giving up. Raise it when the bundled local model is doing CPU-only inference and long generations are timing out; the Settings → Deployment & AI panel reports **Degraded** when recent calls are failing, which is the signal to turn this up. Compose must always pass a value — an unset variable resolves to `None` inside the gateway, which then serves `/health/liveliness` 200 while failing every completion. |
+| `AISOC_INVESTIGATION_MAX_SECONDS` | `120` | Wall-clock budget for one investigation, on **both** the auto-triage path and the console's manual "Investigate with agent". A run that exceeds it is closed as `failed` with the reason, in the run store and in the `investigation_runs` row. |
+| `AISOC_INVESTIGATION_STALE_AFTER_SECONDS` | `3 × AISOC_INVESTIGATION_MAX_SECONDS`, floor 300 | When a background sweep closes an `investigation_runs` row abandoned by a process that exited mid-run. Derived from the budget above so the two cannot disagree; set it only if you need a different margin. |
+| `AISOC_INVESTIGATION_REAP_INTERVAL_SECONDS` | `60` | How often that sweep runs. |
+| `AISOC_INVESTIGATION_REAPER_DISABLE` | unset (sweep on) | Set to `1` to turn the sweep off. Leaving it off means a run orphaned by a restart stays `running` in the ledger forever. |
| `OPENAI_BASE_URL` / `LLM_BASE_URL` | — | Explicit override. Outranks `LLM_GATEWAY_URL` for every role; you own the key pairing when you set it. |
| `AISOC_MODEL_PIN_<ROLE>` | `aisoc-<role>` | Per-role model. Set a concrete provider model to call a provider directly with no gateway. Roles: `triage`, `recon`, `investigation`, `copilot`, `summary`, `report`, `nl`. |
| `OPENAI_MODEL` | `aisoc-summary` | BYOK / "explain this alert" path **only** — never a task role's model. Ships as an alias because the default deployment routes through the gateway, which 400s anything `infra/litellm/config.yaml` does not define. |
diff --git a/apps/web/src/components/settings/SettingsView.llmhealth.test.tsx b/apps/web/src/components/settings/SettingsView.llmhealth.test.tsx
new file mode 100644
index 000000000..96292e3ed
--- /dev/null
+++ b/apps/web/src/components/settings/SettingsView.llmhealth.test.tsx
@@ -0,0 +1,188 @@
+/**
+ * The LLM pill must not read "Live" while every call is failing (issue #1241).
+ *
+ * `LlmCard` derived its badge from `effective_path` alone — a *configuration*
+ * value decided by whether a key is set and whether the air-gap policy would
+ * permit the host. A deployment whose every completion times out therefore
+ * rendered the emerald "Live" pill, which is the panel asserting something it
+ * had never checked.
+ *
+ * `recent_health` carries the outcome of the calls the product actually made.
+ * Three states, and `unknown` matters as much as `degraded`: a pod that has
+ * observed nothing must not be rendered as healthy, and must not be rendered
+ * as broken either.
+ */
+import { describe, expect, it, vi, beforeEach } from 'vitest';
+import { render, screen } from '@testing-library/react';
+import userEvent from '@testing-library/user-event';
+
+const swrData = vi.hoisted(() => new Map<string, unknown>());
+const swrErrors = vi.hoisted(() => new Map<string, unknown>());
+const swrLoadingKeys = vi.hoisted(() => new Set<string>());
+const swrMutate = vi.hoisted(() => vi.fn());
+
+vi.mock('swr', () => ({
+ __esModule: true,
+ default: (key: unknown) => {
+ const k = typeof key === 'string' ? key : JSON.stringify(key);
+ return {
+ data: swrData.get(k),
+ error: swrErrors.get(k),
+ isLoading: swrLoadingKeys.has(k),
+ mutate: swrMutate,
+ };
+ },
+}));
+
+const noop = vi.hoisted(() => vi.fn());
+vi.mock('@/lib/api', () => ({
+ __esModule: true,
+ deploymentApi: {
+ getAirgapStatus: noop,
+ getLlmStatus: noop,
+ getLlmCredential: noop,
+ upsertLlmCredential: noop,
+ deleteLlmCredential: noop,
+ },
+ connectorsApi: { list: noop, statuses: noop },
+ ApiError: class ApiError extends Error {
+ status: number;
+ constructor(message: string, status = 500) {
+ super(message);
+ this.status = status;
+ }
+ },
+}));
+
+vi.mock('react-hot-toast', () => ({
+ __esModule: true,
+ default: { success: vi.fn(), error: vi.fn() },
+ toast: { success: vi.fn(), error: vi.fn() },
+}));
+
+import { SettingsView } from './SettingsView';
+import type { LlmStatus } from '@/lib/api';
+
+const AIRGAP_KEY = 'settings:airgap-status';
+const LLM_KEY = 'settings:llm-status';
+const CRED_KEY = 'settings:llm-credential';
+
+const airgapFixture = {
+ enabled: false,
+ allowlist: [],
+ implicit_private_suffixes: ['.local', '.internal'],
+ policy: 'Air-gap disabled; all egress permitted.',
+};
+
+function llmFixture(overrides: Partial<LlmStatus> = {}): LlmStatus {
+ return {
+ provider: 'local-litellm',
+ model: 'aisoc-triage',
+ base_url: 'http://litellm:4000/v1',
+ host: 'litellm',
+ key_set: true,
+ airgap_enabled: false,
+ airgap_compliant: true,
+ is_local: true,
+ effective_path: 'live',
+ policy_note: 'LLM calls are routed to a local provider (local-litellm).',
+ recent_health: 'healthy',
+ recent_calls: 6,
+ recent_failures: 0,
+ recent_note: 'The most recent LLM call succeeded (0 of the last 6 failed).',
+ ...overrides,
+ };
+}
+
+async function renderWith(llm: LlmStatus) {
+ swrData.set(AIRGAP_KEY, airgapFixture);
+ swrData.set(LLM_KEY, llm);
+ swrData.set(CRED_KEY, undefined);
+ render(<SettingsView />);
+ await userEvent.click(
+ await screen.findByRole('button', { name: /deployment.*ai/i }),
+ );
+}
+
+beforeEach(() => {
+ swrData.clear();
+ swrErrors.clear();
+ swrLoadingKeys.clear();
+});
+
+describe('LLM provider pill reflects whether the provider answers', () => {
+ it('reads Degraded, not Live, when recent calls are failing', async () => {
+ await renderWith(
+ llmFixture({
+ recent_health: 'degraded',
+ recent_calls: 5,
+ recent_failures: 5,
+ recent_note:
+ 'The most recent LLM call failed (ReadTimeout); 5 of the last 5 failed. ' +
+ 'If the provider is a local model on CPU, raise AISOC_LITELLM_REQUEST_TIMEOUT.',
+ }),
+ );
+
+ expect(await screen.findByText('Degraded')).toBeInTheDocument();
+ expect(screen.queryByText('Live')).not.toBeInTheDocument();
+ });
+
+ it('tells the operator why, rather than only that something is wrong', async () => {
+ await renderWith(
+ llmFixture({
+ recent_health: 'degraded',
+ recent_calls: 5,
+ recent_failures: 5,
+ recent_note:
+ 'The most recent LLM call failed (ReadTimeout); 5 of the last 5 failed. ' +
+ 'If the provider is a local model on CPU, raise AISOC_LITELLM_REQUEST_TIMEOUT.',
+ }),
+ );
+
+ expect(await screen.findByText(/5 of the last 5 failed/)).toBeInTheDocument();
+ expect(screen.getByText(/AISOC_LITELLM_REQUEST_TIMEOUT/)).toBeInTheDocument();
+ });
+
+ it('still reads Live when the provider is answering', async () => {
+ await renderWith(llmFixture());
+ expect(await screen.findByText('Live')).toBeInTheDocument();
+ expect(screen.queryByText('Degraded')).not.toBeInTheDocument();
+ });
+
+ it('says calls are unobserved rather than claiming health it has not seen', async () => {
+ await renderWith(
+ llmFixture({
+ recent_health: 'unknown',
+ recent_calls: 0,
+ recent_failures: 0,
+ recent_note:
+ 'No LLM call has been observed by this pod recently, so whether the provider ' +
+ 'answers is unmeasured. This reports configuration only.',
+ }),
+ );
+
+ // Configuration is live, so the path badge stays — but the panel must not
+ // present an unobserved provider as a verified-working one.
+ expect(await screen.findByText('Live')).toBeInTheDocument();
+ expect(screen.getByText(/No LLM call has been observed/)).toBeInTheDocument();
+ });
+
+ it('keeps the fallback badge winning over the health signal', async () => {
+ // A deployment with no key is on the deterministic path: no call is even
+ // attempted, so "degraded" would be the wrong story to lead with.
+ await renderWith(
+ llmFixture({
+ key_set: false,
+ effective_path: 'fallback',
+ recent_health: 'unknown',
+ recent_calls: 0,
+ recent_failures: 0,
+ recent_note: 'No LLM call has been observed by this pod recently.',
+ }),
+ );
+
+ expect(await screen.findByText('Fallback path')).toBeInTheDocument();
+ expect(screen.queryByText('Live')).not.toBeInTheDocument();
+ expect(screen.queryByText('Degraded')).not.toBeInTheDocument();
+ });
+});
diff --git a/apps/web/src/components/settings/SettingsView.tsx b/apps/web/src/components/settings/SettingsView.tsx
index d5f290a66..158b80f39 100644
--- a/apps/web/src/components/settings/SettingsView.tsx
+++ b/apps/web/src/components/settings/SettingsView.tsx
@@ -1655,6 +1655,14 @@ function LlmCard({
const providerLabel = PROVIDER_LABEL[llm.provider] ?? llm.provider;
const isFallback = llm.effective_path === 'fallback';
const blockedByAirgap = llm.airgap_enabled && !llm.airgap_compliant;
+ // `effective_path` is configuration: it answers "would a call be attempted",
+ // not "did one come back". Deciding the pill from it alone is why a
+ // deployment timing out on every call rendered emerald Live (issue #1241).
+ //
+ // Fallback still wins: on the deterministic path no call is attempted, so
+ // there is no provider outcome to report and "Degraded" would blame a
+ // provider that was never asked.
+ const isDegraded = !isFallback && llm.recent_health === 'degraded';
return (
<section
@@ -1678,6 +1686,8 @@ function LlmCard({
)}
{isFallback ? (
<StatusPill tone="amber" label="Fallback path" />
+ ) : isDegraded ? (
+ <StatusPill tone="red" label="Degraded" />
) : (
<StatusPill tone="emerald" label="Live" />
)}
@@ -1712,8 +1722,29 @@ function LlmCard({
</div>
) : null}
+ {isDegraded ? (
+ <div
+ role="alert"
+ className="mt-4 rounded-lg border border-red-500/40 bg-red-500/10 p-3 text-xs text-red-200"
+ >
+ <p className="font-semibold text-red-100">
+ The provider is configured but recent calls are failing.
+ </p>
+ <p className="mt-1">{llm.recent_note}</p>
+ </div>
+ ) : null}
+
<dl className="mt-4 grid grid-cols-1 gap-3 sm:grid-cols-2">
<KeyValue label="Provider">{providerLabel}</KeyValue>
+ {/* Shown whenever the alert block above is not already carrying it, so
+ the evidence behind the badge is visible in every state — including
+ `unknown`. A panel that shows a badge and hides what it rests on is
+ how "Live" came to mean "configured" without anyone noticing. */}
+ {isDegraded ? null : (
+ <KeyValue label="Recent calls">
+ <span className="text-xs text-gray-300">{llm.recent_note}</span>
+ </KeyValue>
+ )}
<KeyValue label="Model">
{llm.model ? (
<code className="font-mono text-xs text-gray-100">{llm.model}</code>
diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.ts
index 230a74131..5ba9502e7 100644
--- a/apps/web/src/lib/api.ts
+++ b/apps/web/src/lib/api.ts
@@ -6155,6 +6155,8 @@ export type LlmProvider =
export type LlmEffectivePath = 'live' | 'fallback';
+export type LlmRecentHealth = 'healthy' | 'degraded' | 'unknown';
+
export interface LlmStatus {
/** Stable provider id classified from the configured base URL. */
provider: LlmProvider;
@@ -6185,6 +6187,22 @@ export interface LlmStatus {
effective_path: LlmEffectivePath;
/** Operator-readable explanation of the current state. */
policy_note: string;
+ /**
+ * Whether the provider is actually *answering*, from the outcomes of the
+ * calls this pod already made — not a probe, and not a latency figure.
+ *
+ * Everything above describes configuration, which is why a deployment
+ * timing out on every call used to render the emerald "Live" pill. Three
+ * states, and ``unknown`` is a real one: a pod that has observed no call
+ * has not earned a green badge.
+ */
+ recent_health: LlmRecentHealth;
+ /** Observed chat-completions calls in the recent window (this pod only). */
+ recent_calls: number;
+ /** How many of those failed. */
+ recent_failures: number;
+ /** Operator-readable summary of the window, including what to do about it. */
+ recent_note: string;
}
// ─── BYOK per-tenant LLM credentials (WS-H2) ────────────────────────────────
diff --git a/docker-compose.yml b/docker-compose.yml
index 1c6796675..bb4a6cd3d 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -933,6 +933,20 @@ services:
# config). Empty is fine — the local model answers instead.
OPENAI_API_KEY: ${OPENAI_API_KEY:-}
ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
+ # How long the gateway waits on one generation before giving up, in
+ # seconds. 60 was a bare literal inside the mounted config, so a
+ # deployment whose model needs longer — CPU-only inference on a larger
+ # prompt is the case — timed out on every call and could only be fixed by
+ # editing a tracked file and restarting (issue #1241). The default is
+ # unchanged; the knob is new. Raise it with
+ # AISOC_LITELLM_REQUEST_TIMEOUT=300 in .env.
+ #
+ # The `:-60` default is load-bearing, not cosmetic: the gateway resolves
+ # this key with `get_secret`, which yields None for an unset variable,
+ # and a gateway in that state serves /health/liveliness 200 while failing
+ # every completion. `test_litellm_config.py` fails if the default is
+ # removed.
+ AISOC_LITELLM_REQUEST_TIMEOUT: ${AISOC_LITELLM_REQUEST_TIMEOUT:-60}
ports:
- "${AISOC_BIND_ADDR:-127.0.0.1}:4000:4000"
healthcheck:
diff --git a/infra/litellm/config.yaml b/infra/litellm/config.yaml
index 8d7a3d741..eb442096c 100644
--- a/infra/litellm/config.yaml
+++ b/infra/litellm/config.yaml
@@ -147,7 +147,34 @@ litellm_settings:
require_auth_for_metrics_endpoint: false
# Never wait forever on a stalled provider — fall through to AiSOC's own
# deterministic path instead of hanging the alert drawer.
- request_timeout: 60
+ #
+ # This read `request_timeout: 60`, the one bare literal in a file where every
+ # other value resolves from the environment, so raising it meant editing a
+ # mounted file and rebuilding. 60s is not a universal number: the bundled 3B
+ # model doing CPU-only inference on a long generation can exceed it, and
+ # every call then times out while the Settings panel still reports the
+ # provider Live (issue #1241). The default is unchanged — the knob is new.
+ #
+ # Measured against `ghcr.io/berriai/litellm:main-stable`, because two
+ # plausible-looking alternatives do not work:
+ #
+ # * LiteLLM's own `REQUEST_TIMEOUT` environment variable is read and
+ # float-coerced in `litellm/constants.py`, but it does **not** reach the
+ # proxy's router path: a gateway started with `REQUEST_TIMEOUT=5` and no
+ # key here was still waiting on a blackholed provider after 200s.
+ # * This key *does* reach it. `get_secret` resolves `os.environ/...` to a
+ # string (it coerces only booleans), LiteLLM assigns that straight onto
+ # `litellm.request_timeout`, and the value lands: with the variable at 3,
+ # the gateway answered 408 with `timeout value=3.0, time taken=3.0`.
+ #
+ # The variable must therefore be SET. `get_secret` returns None for an unset
+ # one, and a gateway in that state answers 200 on `/health/liveliness` while
+ # failing every completion with `float() argument must be a string or a real
+ # number, not 'NoneType'`. `docker-compose.yml` supplies the default, and
+ # `test_litellm_config.py` fails if that default is ever removed — the config
+ # is already hard-dependent on its environment in the same way (with
+ # `AISOC_LLM_MODEL_FAST` unset this file does not start at all).
+ request_timeout: os.environ/AISOC_LITELLM_REQUEST_TIMEOUT
# ── Router: retries + cross-model fallbacks ───────────────────────────────────
# model_pins.py documents that "the LiteLLM gateway owns any model-level
@@ -157,6 +184,13 @@ litellm_settings:
router_settings:
# Retry a transient failure (rate limit / 5xx / timeout) on the same alias
# before falling back.
+ #
+ # A literal on purpose, and not environment-driven like the timeout above.
+ # LiteLLM hands `router_settings` straight to `Router.__init__` with no type
+ # coercion (`router_params[k] = v`), and the retry path evaluates
+ # `num_retries > 0`, which raises `TypeError` against a string at precisely
+ # the moment a retry is needed. The knob that matters for slow hardware is
+ # the timeout, and that one is environment-driven.
num_retries: 2
# Cross-model failover: if an alias's primary model keeps failing, route the
# request to an adjacent/cheaper alias so quality degrades gracefully instead
diff --git a/scripts/check_tenant_query_predicates.py b/scripts/check_tenant_query_predicates.py
index 4e8ba6d16..9eb21f5a7 100644
--- a/scripts/check_tenant_query_predicates.py
+++ b/scripts/check_tenant_query_predicates.py
@@ -238,6 +238,14 @@ RATCHET: dict[str, str] = {
"iterate, and a tenant missing from that list would have approvals that never "
"expire, which is the silent hang the sweep exists to prevent"
),
+ "services/agents/app/investigator/ledger.py::fail_stale_runs::investigation_runs": (
+ "the stale-run reaper, the same shape as the approval and webhook sweeps below: a run is "
+ "abandoned when the process that owned it is gone, so there is no caller and no tenant to "
+ "filter by, and a per-tenant pass needs a list of tenants whose omissions are runs that "
+ "render as 'running' forever. Admitted by the second arm of the policy in "
+ "008_investigation_ledger.sql (`OR current_tenant_id() IS NULL`); the function says so, so a "
+ "future policy that drops that arm turns this into a visible no-op rather than a silent one"
+ ),
"services/api/app/services/outbound_webhooks.py::due_deliveries::OutboundDelivery": (
"the outbound-webhook retry sweep, the same shape as the approval and wait sweeps above: a "
"per-tenant pass needs a list of tenants, and a tenant missing from it has events that are "
@@ -428,7 +436,7 @@ RATCHET: dict[str, str] = {
# ceiling with a spare slot in it and the list held 41. The ceiling now
# equals the list, so the next unscoped statement has to be argued for
# rather than absorbed — which is what this number is for.
-MAX_RATCHET = 43
+MAX_RATCHET = 44
# ---------------------------------------------------------------------------
diff --git a/services/agents/app/api/investigate.py b/services/agents/app/api/investigate.py
index 80bb7752b..fdd643328 100644
--- a/services/agents/app/api/investigate.py
+++ b/services/agents/app/api/investigate.py
@@ -26,7 +26,8 @@ from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, WebSocke
from fastapi.responses import HTMLResponse, PlainTextResponse
from pydantic import BaseModel
-from app.investigator import InvestigatorOrchestrator
+from app.graph.runner import default_budget
+from app.investigator import InvestigatorOrchestrator, ledger
from app.orchestrator.router import RouterOrchestrator
from app.security.tenant_scope import (
TENANT_HEADER,
@@ -211,6 +212,53 @@ async def _emit_event(run_id: str, tenant_id: str, event: dict[str, Any]) -> Non
# Background task: runs investigation and streams steps to realtime service
# ---------------------------------------------------------------------------
+#: Statuses that mean this run is over. The console's ``CaseWorkspace.tsx``
+#: branches on ``idle``/``starting``/``running``/``failed``/``completed`` and the
+#: WebSocket tail below breaks on ``completed``/``failed``, so a run that ends in
+#: any other word is a run no surface can finish rendering. A timed-out run is
+#: therefore ``failed`` carrying the reason, not a fifth vocabulary entry.
+_TERMINAL_STATUSES = frozenset({"completed", "failed"})
+
+
+async def _fail_run(run_id: str, run_uuid: UUID, tenant_ref: str, reason: str) -> None:
+ """Close a run as ``failed`` in **both** places a consumer reads it.
+
+ The console polls ``GET /api/v1/investigations/{run_id}``, which serves the
+ process dict; the Investigation Ledger renders the ``investigation_runs``
+ row. Writing one and not the other swaps a spinner that never resolves for
+ a ledger entry that never ends, which is the same defect wearing the other
+ surface.
+
+ Never downgrades a run that already finished: a ``done`` event followed by
+ a slow generator teardown must not be relabelled a failure.
+ """
+ entry = _runs.get(run_id)
+ if entry is not None:
+ if entry.get("status") in _TERMINAL_STATUSES:
+ return
+ entry.update(
+ {
+ "status": "failed",
+ "error": reason,
+ "completed_at": datetime.utcnow().isoformat(),
+ }
+ )
+
+ # Best-effort, exactly like every other ledger write: no database
+ # configured is not an error, and a failed bookkeeping write must not
+ # replace the reason the run ended with a reason it did not.
+ try:
+ tenant_uuid = await ledger.resolve_tenant(tenant_ref)
+ if tenant_uuid is not None:
+ await ledger.complete_run(
+ run_id=run_uuid,
+ tenant_id=tenant_uuid,
+ status="failed",
+ error=reason,
+ )
+ except Exception as exc: # noqa: BLE001 — bookkeeping never masks the outcome
+ logger.warning("investigation.ledger_close_failed", run_id=run_id, error=str(exc))
+
async def _run_and_store(run_id: str, case_id: str, req: InvestigateRequest) -> None:
audit_log: list[dict[str, Any]] = []
@@ -220,67 +268,97 @@ async def _run_and_store(run_id: str, case_id: str, req: InvestigateRequest) ->
run_uuid = UUID(run_id)
except (ValueError, TypeError):
run_uuid = uuid4()
+
+ # The same wall-clock budget the auto-triage path runs under — read at call
+ # time from ``AISOC_INVESTIGATION_MAX_SECONDS`` rather than restated here,
+ # because two constants describing one deadline is how the shorter one wins
+ # silently. Without it this path had none at all: `ChatOpenAI` is built with
+ # no timeout, so a merely-slow model raises nothing and the run stays
+ # `running` for the life of the process (issue #1242).
+ max_seconds = default_budget().max_seconds
+ deadline = asyncio.timeout(max_seconds)
try:
- # Use the streaming orchestrator so we can emit events progressively.
- # ``_investigate_stream`` picks investigator vs. router at call time
- # based on ``AISOC_INVESTIGATE_USE_ROUTER``.
- async for event in _investigate_stream(
- case_id=case_id,
- alert_summary=req.alert_summary,
- raw_alert=req.raw_alert,
- tenant_id=req.tenant_id,
- run_id=run_uuid,
- ):
- if event.get("type") == "step":
- audit_log.append(event)
- # Update the in-memory run so pollers see progress
- _runs[run_id]["audit_log"] = audit_log
- # Broadcast to realtime → WebSocket clients
- await _emit_event(run_id, req.tenant_id, event)
-
- elif event.get("type") == "done":
- state_data = event.get("state", {})
- _runs[run_id].update(
- {
- "status": "completed",
- "report_md": state_data.get("report_md", ""),
- "report_html": state_data.get("report_html", ""),
- "audit_log": audit_log,
- "recon": state_data.get("recon", {}),
- "forensic": state_data.get("forensic", {}),
- "responder": state_data.get("responder", {}),
- "completed_at": datetime.utcnow().isoformat(),
- "error": None,
- }
- )
- await _emit_event(
- run_id,
- req.tenant_id,
- {
- "kind": "completed",
- "agent": "orchestrator",
- "summary": "Investigation completed",
- "data": {"status": "completed"},
- },
- )
+ async with deadline:
+ # Use the streaming orchestrator so we can emit events progressively.
+ # ``_investigate_stream`` picks investigator vs. router at call time
+ # based on ``AISOC_INVESTIGATE_USE_ROUTER``.
+ async for event in _investigate_stream(
+ case_id=case_id,
+ alert_summary=req.alert_summary,
+ raw_alert=req.raw_alert,
+ tenant_id=req.tenant_id,
+ run_id=run_uuid,
+ ):
+ if event.get("type") == "step":
+ audit_log.append(event)
+ # Update the in-memory run so pollers see progress
+ _runs[run_id]["audit_log"] = audit_log
+ # Broadcast to realtime → WebSocket clients
+ await _emit_event(run_id, req.tenant_id, event)
+
+ elif event.get("type") == "done":
+ state_data = event.get("state", {})
+ _runs[run_id].update(
+ {
+ "status": "completed",
+ "report_md": state_data.get("report_md", ""),
+ "report_html": state_data.get("report_html", ""),
+ "audit_log": audit_log,
+ "recon": state_data.get("recon", {}),
+ "forensic": state_data.get("forensic", {}),
+ "responder": state_data.get("responder", {}),
+ "completed_at": datetime.utcnow().isoformat(),
+ "error": None,
+ }
+ )
+ await _emit_event(
+ run_id,
+ req.tenant_id,
+ {
+ "kind": "completed",
+ "agent": "orchestrator",
+ "summary": "Investigation completed",
+ "data": {"status": "completed"},
+ },
+ )
- elif event.get("type") == "error":
- err_msg = event.get("error", "Unknown error")
- _runs[run_id].update({"status": "failed", "error": err_msg})
- await _emit_event(
- run_id,
- req.tenant_id,
- {
- "kind": "error",
- "agent": "orchestrator",
- "summary": err_msg,
- "data": {"status": "failed"},
- },
- )
+ elif event.get("type") == "error":
+ err_msg = event.get("error", "Unknown error")
+ _runs[run_id].update({"status": "failed", "error": err_msg})
+ await _emit_event(
+ run_id,
+ req.tenant_id,
+ {
+ "kind": "error",
+ "agent": "orchestrator",
+ "summary": err_msg,
+ "data": {"status": "failed"},
+ },
+ )
+
+ except TimeoutError as exc:
+ # `asyncio.timeout` converts the cancellation it issued into
+ # `TimeoutError`; an inner `TimeoutError` (an httpx read timeout, say)
+ # arrives here too and is *not* the budget. `expired()` tells them
+ # apart, so the operator-facing reason names the thing that actually
+ # ran out.
+ if deadline.expired():
+ reason = (
+ f"Investigation exceeded its wall-clock budget of {max_seconds}s and was stopped. "
+ "Raise AISOC_INVESTIGATION_MAX_SECONDS if the configured model needs longer."
+ )
+ logger.warning("investigation.budget_timeout", run_id=run_id, max_seconds=max_seconds)
+ else:
+ reason = str(exc) or "Investigation timed out"
+ logger.error("investigation_bg_task timed out", run_id=run_id, error=reason)
+ await _fail_run(run_id, run_uuid, req.tenant_id, reason)
except Exception as exc: # noqa: BLE001
logger.error("investigation_bg_task failed", run_id=run_id, error=str(exc))
- _runs[run_id].update({"status": "failed", "error": str(exc)})
+ # Through the same helper as the deadline: an exception escaping the
+ # stream means the orchestrator's own `except` arm did not run, so
+ # nothing else is going to close the ledger row either.
+ await _fail_run(run_id, run_uuid, req.tenant_id, str(exc))
# ---------------------------------------------------------------------------
diff --git a/services/agents/app/investigator/ledger.py b/services/agents/app/investigator/ledger.py
index 99c1da8fe..50843b2ff 100644
--- a/services/agents/app/investigator/ledger.py
+++ b/services/agents/app/investigator/ledger.py
@@ -408,6 +408,78 @@ async def complete_run(
)
+async def fail_stale_runs(
+ *,
+ older_than_seconds: float,
+ reason: str,
+ limit: int = 500,
+) -> list[uuid.UUID]:
+ """Close ``investigation_runs`` rows abandoned by a process that is gone.
+
+ The wall-clock deadline in :func:`app.api.investigate._run_and_store` ends
+ a run whose process is still alive. A restart, an OOM kill or an ordinary
+ rollout leaves the row at ``'running'`` with nothing left to finish it:
+ the in-memory run store is gone, so no poller, no WebSocket tail and no
+ exception handler will ever write a terminal status. The Investigation
+ Ledger then renders a run that never ends.
+
+ Safe on several replicas and safe to re-run: the predicate is part of the
+ ``UPDATE``, so two reapers racing the same row close it once and the loser
+ simply matches nothing.
+
+ Safe against reaping a *live* run only because the deadline exists — a run
+ cannot legitimately outlast its own budget any more, so ``older_than_seconds``
+ is set well beyond it (see :mod:`app.investigator.run_reaper`). Shortening
+ that margin without the deadline in place would cancel healthy work.
+
+ No RLS context is set: this is a cross-tenant maintenance sweep, and
+ migration ``008_investigation_ledger.sql`` writes the policy as
+ ``USING (tenant_id = current_tenant_id() OR current_tenant_id() IS NULL)``,
+ whose second arm is what admits it. Named here because a future policy that
+ drops that arm would turn this into a silent no-op rather than an error.
+
+ Returns the ids closed (empty when there was nothing to do, no database is
+ configured, or the sweep failed — the caller logs and retries).
+ """
+ pool = await get_pool()
+ if pool is None:
+ return []
+
+ try:
+ async with pool.acquire() as conn:
+ rows = await conn.fetch(
+ """
+ UPDATE investigation_runs
+ SET status = 'failed',
+ error = COALESCE(NULLIF(error, ''), $2),
+ completed_at = now()
+ WHERE id IN (
+ SELECT id
+ FROM investigation_runs
+ WHERE status = 'running'
+ AND started_at < now() - make_interval(secs => $1::double precision)
+ ORDER BY started_at
+ LIMIT $3
+ )
+ RETURNING id
+ """,
+ float(older_than_seconds),
+ reason,
+ int(limit),
+ )
+ closed = [row["id"] for row in rows]
+ if closed:
+ logger.warning(
+ "ledger.stale_runs_closed",
+ count=len(closed),
+ older_than_seconds=older_than_seconds,
+ )
+ return closed
+ except Exception as exc: # noqa: BLE001 — the caller backs off and retries
+ logger.warning("ledger.fail_stale_runs_failed", error=str(exc))
+ return []
+
+
def _coerce_uuid(value: Any) -> uuid.UUID | None:
try:
return uuid.UUID(str(value))
diff --git a/services/agents/app/investigator/run_reaper.py b/services/agents/app/investigator/run_reaper.py
new file mode 100644
index 000000000..c23b4ceb1
--- /dev/null
+++ b/services/agents/app/investigator/run_reaper.py
@@ -0,0 +1,158 @@
+"""The loop that closes investigation runs orphaned by a dead process.
+
+Why this exists alongside the deadline
+--------------------------------------
+``app.api.investigate._run_and_store`` now runs under the shared wall-clock
+budget, so a run that hangs inside a live process ends as ``failed`` and says
+why. That covers nothing at all once the process is gone: the run store is a
+module-level dict, so a restart, an OOM kill or an ordinary rollout drops every
+in-flight run on the floor while its ``investigation_runs`` row stays
+``'running'``. Nothing else in the service will ever write a terminal status to
+it, and the Investigation Ledger renders exactly that row.
+
+How long is "stale"
+-------------------
+Derived from the same budget the deadline uses, never restated. A run cannot
+legitimately outlast ``AISOC_INVESTIGATION_MAX_SECONDS`` any more, so a row
+still ``running`` several multiples past it belongs to a process that is not
+coming back. The multiplier plus a floor is the margin against clock skew and
+a replica that is merely slow to write its terminal status; an operator who
+needs a different one sets ``AISOC_INVESTIGATION_STALE_AFTER_SECONDS``.
+
+Three properties, matching ``app.playbook.sweeper``
+---------------------------------------------------
+**Safe on several replicas.** The staleness predicate lives inside the
+``UPDATE``, so two reapers racing one row close it once.
+
+**Says so when it cannot do its job.** Every failed pass is logged at
+``warning`` with the reason, and consecutive failures back off rather than
+hammering a store that is down.
+
+**Off only by explicit opt-out.** The failure mode of not running it is the
+defect it exists to fix, not the safe state.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import logging
+import os
+
+from app.graph.runner import default_budget
+from app.investigator import ledger
+
+logger = logging.getLogger("aisoc.investigator.run_reaper")
+
+#: How often to look. Orphans only appear at process death, so this is a query
+#: against an indexed ``status`` column on a table that is empty of stale rows
+#: almost always; a minute of latency on a run that has already been abandoned
+#: for several multiples of its budget costs nothing.
+DEFAULT_INTERVAL_SECONDS = float(os.getenv("AISOC_INVESTIGATION_REAP_INTERVAL_SECONDS", "60"))
+
+#: Multiples of the run budget before a row counts as abandoned.
+_STALE_BUDGET_MULTIPLE = 3.0
+
+#: Floor for the staleness margin, so a deployment that sets a very small
+#: budget for its own testing does not start reaping runs that a replica is
+#: still writing the terminal status for.
+_MIN_STALE_SECONDS = 300.0
+
+#: Ceiling for the backoff a failing store earns.
+_MAX_BACKOFF_SECONDS = 600.0
+
+#: What the closed row says. An operator reading the ledger needs to know this
+#: was a reap rather than a verdict the agent reached.
+STALE_REASON = "Investigation abandoned: the process running it exited before writing a result."
+
+
+def stale_after_seconds() -> float:
+ """Seconds a run may sit at ``running`` before it counts as abandoned.
+
+ Derived from ``AISOC_INVESTIGATION_MAX_SECONDS`` so the reaper and the
+ deadline cannot disagree about how long a run is allowed to take — two
+ independently written numbers for one budget is how the shorter one wins
+ silently. ``AISOC_INVESTIGATION_STALE_AFTER_SECONDS`` overrides it outright
+ for an operator who wants a different margin.
+ """
+ raw = os.getenv("AISOC_INVESTIGATION_STALE_AFTER_SECONDS", "").strip()
+ if raw:
+ try:
+ return float(raw)
+ except ValueError:
+ logger.warning(
+ "run_reaper: AISOC_INVESTIGATION_STALE_AFTER_SECONDS=%r is not a number; deriving from the run budget",
+ raw,
+ )
+ return max(default_budget().max_seconds * _STALE_BUDGET_MULTIPLE, _MIN_STALE_SECONDS)
+
+
+def enabled() -> bool:
+ """On unless explicitly disabled, like the playbook sweeper."""
+ return os.getenv("AISOC_INVESTIGATION_REAPER_DISABLE", "").strip().lower() not in {"1", "true", "yes", "on"}
+
+
+async def reap_once() -> int:
+ """One pass. Returns how many rows were closed."""
+ closed = await ledger.fail_stale_runs(
+ older_than_seconds=stale_after_seconds(),
+ reason=STALE_REASON,
+ )
+ if closed:
+ logger.info("run_reaper: closed %d abandoned investigation run(s)", len(closed))
+ return len(closed)
+
+
+async def run_forever(*, interval_seconds: float | None = None) -> None:
+ """Reap on a fixed cadence until cancelled."""
+ interval = interval_seconds or DEFAULT_INTERVAL_SECONDS
+ consecutive_failures = 0
+ while True:
+ try:
+ await reap_once()
+ consecutive_failures = 0
+ delay = interval
+ except asyncio.CancelledError:
+ raise
+ except Exception as exc: # noqa: BLE001 — a failed pass must not end the loop
+ consecutive_failures += 1
+ delay = min(interval * (2**consecutive_failures), _MAX_BACKOFF_SECONDS)
+ logger.warning(
+ "run_reaper: pass %d failed (%s); next attempt in %.0fs",
+ consecutive_failures,
+ exc,
+ delay,
+ )
+ await asyncio.sleep(delay)
+
+
+def start(app: object) -> asyncio.Task | None:
+ """Start the reaper as a background task on the app, or say why not."""
+ if not enabled():
+ logger.info("run_reaper: disabled by AISOC_INVESTIGATION_REAPER_DISABLE")
+ return None
+ task = asyncio.create_task(run_forever())
+ task.add_done_callback(_log_exit)
+ with contextlib.suppress(AttributeError):
+ app.state.investigation_reaper_task = task # type: ignore[attr-defined]
+ logger.info(
+ "run_reaper: started, every %.0fs, closing runs older than %.0fs",
+ DEFAULT_INTERVAL_SECONDS,
+ stale_after_seconds(),
+ )
+ return task
+
+
+def _log_exit(task: asyncio.Task) -> None:
+ """Retrieve the task's outcome so a dead reaper is in the log.
+
+ The same pairing the triage worker, the fusion consumer and the playbook
+ sweeper carry. A background task whose exception is never retrieved dies
+ quietly, and the only symptom is orphaned runs accumulating — which looks
+ like an agent problem rather than a missing reaper.
+ """
+ if task.cancelled():
+ return
+ exc = task.exception()
+ if exc is not None:
+ logger.error("run_reaper: exited on %s; abandoned runs will stay 'running'", exc)
diff --git a/services/agents/app/main.py b/services/agents/app/main.py
index 4fc0f628a..3755932a3 100644
--- a/services/agents/app/main.py
+++ b/services/agents/app/main.py
@@ -34,6 +34,7 @@ __all__ = ["score_agent_quality"]
from app.hunt import scheduler as hunt_scheduler
from app.hunt import store as hunt_store
from app.investigator import ledger as investigation_ledger
+from app.investigator import run_reaper as investigation_run_reaper
from app.llm.factory import preflight_llm
from app.playbook import PlaybookStore
from app.playbook import sweeper as playbook_sweeper
@@ -148,6 +149,16 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
except Exception as exc: # noqa: BLE001 — never block API startup
logger.warning("playbook_sweeper.start_failed", error=str(exc))
+ # Issue #1242 — close investigation runs orphaned by a process that died.
+ # The wall-clock deadline on the manual path only ends a run whose process
+ # is still alive; the run store is a module dict, so a restart leaves the
+ # `investigation_runs` row at 'running' with nothing left to finish it.
+ app.state.investigation_reaper_task = None
+ try:
+ investigation_run_reaper.start(app)
+ except Exception as exc: # noqa: BLE001 — never block API startup
+ logger.warning("investigation_run_reaper.start_failed", error=str(exc))
+
# Phase 2.6 — flip /readyz to 200 once startup work is done.
app.state.mark_ready()
@@ -168,6 +179,9 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
if getattr(app.state, "playbook_sweeper_task", None) is not None:
app.state.playbook_sweeper_task.cancel()
+ if getattr(app.state, "investigation_reaper_task", None) is not None:
+ app.state.investigation_reaper_task.cancel()
+
# Stop the hunt scheduler before draining DB pools so in-flight runs
# can flush their writes.
try:
diff --git a/services/agents/tests/test_investigation_run_deadline.py b/services/agents/tests/test_investigation_run_deadline.py
new file mode 100644
index 000000000..4611d722a
--- /dev/null
+++ b/services/agents/tests/test_investigation_run_deadline.py
@@ -0,0 +1,221 @@
+"""A manual investigation must reach a terminal status (issue #1242).
+
+The auto-triage path has had a wall-clock budget since issue #569:
+``app.graph.runner._run`` wraps the graph in ``asyncio.timeout(budget.max_seconds)``
+and logs ``investigation.budget_timeout``. The console's path is a different
+one — ``POST /api/v1/cases/{id}/investigate`` reaches
+:func:`app.api.investigate.launch_investigation`, which writes
+``status: "running"`` and hands :func:`app.api.investigate._run_and_store` to
+``BackgroundTasks`` — and it had no deadline at all. ``_run_and_store`` wrote a
+terminal status only on a ``done`` event, an ``error`` event, or a raised
+exception, so a model that merely answers slowly (``ChatOpenAI`` is built with
+no timeout) left both the in-memory run entry and the ``investigation_runs``
+row at ``running`` forever.
+
+Three properties are pinned here, because the fix is wrong if any one is
+missing:
+
+**The deadline exists.** A stream that never completes must still end in a
+terminal status, not hang until the process dies.
+
+**It is the same budget, not a second one.** Two constants describing one
+deadline is a shape this repository has already been bitten by, so the value
+comes from ``app.graph.runner.default_budget()`` — i.e. the single
+``AISOC_INVESTIGATION_MAX_SECONDS`` knob the auto-triage path reads.
+
+**It reaches the database.** The console polls the API for the run entry but
+the Investigation Ledger renders ``investigation_runs``. A timeout that only
+updates the process dict leaves the ledger showing a run that never ends.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import sys
+from collections.abc import AsyncIterator
+from pathlib import Path
+from typing import Any
+from uuid import UUID, uuid4
+
+import pytest
+
+_AGENTS_ROOT = Path(__file__).resolve().parents[1]
+if str(_AGENTS_ROOT) not in sys.path:
+ sys.path.insert(0, str(_AGENTS_ROOT))
+
+from app.api import investigate as investigate_mod # noqa: E402
+
+#: The budget the tests drive the code with. Short enough that a wall-clock
+#: assertion is cheap, long enough that it is clearly a deadline firing rather
+#: than the coroutine never having started.
+_TEST_BUDGET_SECONDS = 0.25
+
+#: How long the test itself is willing to wait. Comfortably above the budget,
+#: so a failure here means "no deadline fired", never "the runner was slow".
+_TEST_PATIENCE_SECONDS = 15.0
+
+#: Terminal statuses the console already renders. ``CaseWorkspace.tsx`` branches
+#: on ``idle``/``starting``/``running``/``failed``/``completed`` and the
+#: WebSocket tail in ``investigate.py`` breaks on ``completed``/``failed``, so a
+#: new vocabulary word here would be a run that no surface can finish.
+_TERMINAL = {"completed", "failed"}
+
+
+class _HungStream:
+ """A stream that starts, emits one step, and then never completes.
+
+ This is the shape of the defect: not an exception and not an empty
+ generator, but a model call that is still outstanding. ``_run_and_store``
+ treats everything short of ``done``/``error`` as progress.
+ """
+
+ def __init__(self) -> None:
+ self.cancelled = False
+
+ async def __call__(self, **_kwargs: Any) -> AsyncIterator[dict[str, Any]]:
+ yield {"type": "step", "kind": "llm_request", "agent": "recon", "summary": "asking the model"}
+ try:
+ await asyncio.sleep(3600)
+ except asyncio.CancelledError:
+ self.cancelled = True
+ raise
+ yield {"type": "done", "state": {}} # pragma: no cover — unreachable
+
+
+def hung_stream(monkeypatch: pytest.MonkeyPatch) -> _HungStream:
+ """Replace the orchestrator stream with one that never finishes."""
+ stream = _HungStream()
+ monkeypatch.setattr(investigate_mod, "_investigate_stream", stream, raising=True)
+ # The realtime fan-out is a best-effort HTTP POST; keep the test off the
+ # network so a 2s connect timeout per step cannot be mistaken for the
+ # deadline under test.
+ monkeypatch.setattr(investigate_mod, "_emit_event", _noop_emit, raising=True)
+ return stream
+
+
+async def _noop_emit(*_args: Any, **_kwargs: Any) -> None:
+ return None
+
+
+def budget(monkeypatch: pytest.MonkeyPatch) -> float:
+ monkeypatch.setenv("AISOC_INVESTIGATION_MAX_SECONDS", str(_TEST_BUDGET_SECONDS))
+ return _TEST_BUDGET_SECONDS
+
+
+def run_entry(monkeypatch: pytest.MonkeyPatch) -> tuple[str, dict[str, Any]]:
+ """Seed ``_runs`` exactly as ``launch_investigation`` does, isolated."""
+ runs: dict[str, dict[str, Any]] = {}
+ monkeypatch.setattr(investigate_mod, "_runs", runs, raising=True)
+ run_id = str(uuid4())
+ runs[run_id] = {"run_id": run_id, "case_id": "case-1242", "status": "running"}
+ return run_id, runs[run_id]
+
+
+async def _drive(run_id: str) -> None:
+ """Run the background task the endpoint schedules, under the test's patience."""
+ req = investigate_mod.InvestigateRequest(
+ alert_summary="Impossible travel on a privileged account",
+ raw_alert={"user": "svc-deploy"},
+ tenant_id="default",
+ )
+ try:
+ await asyncio.wait_for(
+ investigate_mod._run_and_store(run_id, "case-1242", req),
+ timeout=_TEST_PATIENCE_SECONDS,
+ )
+ except TimeoutError:
+ pytest.fail(
+ f"_run_and_store did not return within {_TEST_PATIENCE_SECONDS}s against a stream that "
+ f"never completes, with AISOC_INVESTIGATION_MAX_SECONDS={_TEST_BUDGET_SECONDS}. "
+ "The manual investigation path has no wall-clock deadline, so the run stays 'running' "
+ "forever and the console shows a spinner that never resolves (issue #1242)."
+ )
+
+
+async def test_hung_stream_still_reaches_a_terminal_status(
+ budget: float,
+ hung_stream: _HungStream,
+ run_entry: tuple[str, dict[str, Any]],
+) -> None:
+ run_id, entry = run_entry
+
+ await _drive(run_id)
+
+ assert entry["status"] in _TERMINAL, (
+ f"run ended at status {entry['status']!r}; the console only renders {sorted(_TERMINAL)} as finished states"
+ )
+ assert entry.get("error"), "a run that timed out must say so, not finish silently"
+ assert hung_stream.cancelled, "the outstanding model call must be cancelled, not left running"
+
+
+async def test_deadline_is_the_shared_budget_not_a_second_constant(
+ budget: float,
+ hung_stream: _HungStream,
+ run_entry: tuple[str, dict[str, Any]],
+) -> None:
+ """``AISOC_INVESTIGATION_MAX_SECONDS`` is the only knob, on both paths.
+
+ Asserted through the env var rather than by reading a module attribute:
+ a second hardcoded number would satisfy an attribute check and still
+ ignore the operator's setting.
+ """
+ run_id, entry = run_entry
+
+ loop = asyncio.get_running_loop()
+ started = loop.time()
+ await _drive(run_id)
+ elapsed = loop.time() - started
+
+ assert elapsed < _TEST_PATIENCE_SECONDS / 2, (
+ f"run took {elapsed:.1f}s against a {budget}s budget — the deadline is not reading AISOC_INVESTIGATION_MAX_SECONDS"
+ )
+ assert str(budget) in str(entry.get("error", "")), (
+ f"the failure message {entry.get('error')!r} does not quote the configured budget "
+ f"({budget}s), so an operator cannot tell which deadline fired"
+ )
+
+
+async def test_timeout_writes_the_terminal_status_to_the_ledger_row(
+ budget: float,
+ hung_stream: _HungStream,
+ run_entry: tuple[str, dict[str, Any]],
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """The ``investigation_runs`` row must finish too, not just the process dict.
+
+ ``InvestigationLedger.tsx`` renders that row. ``complete_run`` fires from
+ the orchestrator's ``done`` and ``except`` arms only, and neither runs when
+ the task is cancelled from outside — ``CancelledError`` is a
+ ``BaseException``, so the orchestrator's ``except Exception`` does not see
+ it.
+ """
+ run_id, _entry = run_entry
+ completions: list[dict[str, Any]] = []
+ tenant_uuid = uuid4()
+
+ async def _fake_resolve_tenant(tenant_ref: str) -> UUID:
+ assert tenant_ref == "default"
+ return tenant_uuid
+
+ async def _fake_complete_run(**kwargs: Any) -> None:
+ completions.append(kwargs)
+
+ monkeypatch.setattr(investigate_mod.ledger, "resolve_tenant", _fake_resolve_tenant, raising=True)
+ monkeypatch.setattr(investigate_mod.ledger, "complete_run", _fake_complete_run, raising=True)
+
+ await _drive(run_id)
+
+ assert completions, (
+ "no ledger write on timeout — investigation_runs.status stays 'running' and the Investigation Ledger shows a run that never ends"
+ )
+ (call,) = completions
+ assert call["status"] == "failed", f"ledger row closed as {call['status']!r}"
+ assert call["tenant_id"] == tenant_uuid
+ assert str(call["run_id"]) == run_id, "the ledger row closed must be the one the endpoint opened"
+ assert call.get("error"), "the ledger row must carry the reason"
diff --git a/services/agents/tests/test_investigation_run_reaper.py b/services/agents/tests/test_investigation_run_reaper.py
new file mode 100644
index 000000000..d15e1b5aa
--- /dev/null
+++ b/services/agents/tests/test_investigation_run_reaper.py
@@ -0,0 +1,277 @@
+"""Runs orphaned by a dead process must still reach a terminal status (#1242).
+
+The wall-clock deadline added to ``app.api.investigate._run_and_store`` ends a
+run that hangs inside a live process. It covers nothing once the process is
+gone: the run store is a module-level dict, so a restart, an OOM kill or an
+ordinary rollout drops every in-flight run while its ``investigation_runs`` row
+stays ``'running'`` — and the Investigation Ledger renders that row.
+
+Two layers, tested separately on purpose:
+
+* the policy (how long is stale, and that the number is derived from the same
+ budget rather than written a second time) — plain unit tests;
+* the SQL, against a **real** Postgres carrying the production schema. A fake
+ pool would answer whatever query it was handed, which is exactly the double
+ that is more capable than the real thing: it cannot tell you that
+ ``make_interval`` binds, that the conditional ``UPDATE`` is race-safe, or
+ that the RLS policy on the table lets a cross-tenant sweep see anything at
+ all. Set ``AISOC_TEST_LEDGER_DSN`` to run that half.
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+import uuid
+from pathlib import Path
+from typing import Any
+
+import pytest
+
+_AGENTS_ROOT = Path(__file__).resolve().parents[1]
+if str(_AGENTS_ROOT) not in sys.path:
+ sys.path.insert(0, str(_AGENTS_ROOT))
+
+from app.investigator import ledger, run_reaper # noqa: E402
+
+LEDGER_DSN = os.environ.get("AISOC_TEST_LEDGER_DSN", "").strip()
+
+
+# ---------------------------------------------------------------------------
+# Policy: one budget, not two
+# ---------------------------------------------------------------------------
+
+
+def test_stale_margin_is_derived_from_the_run_budget(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Not a second hardcoded deadline — the reaper tracks the same knob.
+
+ A margin written independently of the budget is how the two disagree, and
+ a reaper that is shorter than the deadline cancels healthy work.
+ """
+ monkeypatch.delenv("AISOC_INVESTIGATION_STALE_AFTER_SECONDS", raising=False)
+
+ monkeypatch.setenv("AISOC_INVESTIGATION_MAX_SECONDS", "600")
+ assert run_reaper.stale_after_seconds() == 1800.0
+
+ monkeypatch.setenv("AISOC_INVESTIGATION_MAX_SECONDS", "900")
+ assert run_reaper.stale_after_seconds() == 2700.0
+
+
+def test_stale_margin_is_always_longer_than_the_deadline(monkeypatch: pytest.MonkeyPatch) -> None:
+ """The safety property: never reap a run that is still allowed to be running."""
+ from app.graph.runner import default_budget
+
+ monkeypatch.delenv("AISOC_INVESTIGATION_STALE_AFTER_SECONDS", raising=False)
+ for budget in ("0.25", "1", "120", "600", "3600"):
+ monkeypatch.setenv("AISOC_INVESTIGATION_MAX_SECONDS", budget)
+ assert run_reaper.stale_after_seconds() > default_budget().max_seconds, (
+ f"with a {budget}s budget the reaper would close runs that are still within it"
+ )
+
+
+def test_operator_override_wins(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_INVESTIGATION_MAX_SECONDS", "120")
+ monkeypatch.setenv("AISOC_INVESTIGATION_STALE_AFTER_SECONDS", "7200")
+ assert run_reaper.stale_after_seconds() == 7200.0
+
+
+def test_a_junk_override_falls_back_rather_than_crashing_the_loop(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_INVESTIGATION_MAX_SECONDS", "600")
+ monkeypatch.setenv("AISOC_INVESTIGATION_STALE_AFTER_SECONDS", "every-ten-minutes")
+ assert run_reaper.stale_after_seconds() == 1800.0
+
+
+def test_reaper_is_on_unless_explicitly_disabled(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Not running it is the defect, so off is the opt-in."""
+ monkeypatch.delenv("AISOC_INVESTIGATION_REAPER_DISABLE", raising=False)
+ assert run_reaper.enabled() is True
+ monkeypatch.setenv("AISOC_INVESTIGATION_REAPER_DISABLE", "1")
+ assert run_reaper.enabled() is False
+
+
+async def test_reap_once_passes_the_derived_margin_through(monkeypatch: pytest.MonkeyPatch) -> None:
+ calls: list[dict[str, Any]] = []
+
+ async def _fake(**kwargs: Any) -> list[uuid.UUID]:
+ calls.append(kwargs)
+ return [uuid.uuid4()]
+
+ monkeypatch.setattr(ledger, "fail_stale_runs", _fake, raising=True)
+ monkeypatch.delenv("AISOC_INVESTIGATION_STALE_AFTER_SECONDS", raising=False)
+ monkeypatch.setenv("AISOC_INVESTIGATION_MAX_SECONDS", "600")
+
+ assert await run_reaper.reap_once() == 1
+ (call,) = calls
+ assert call["older_than_seconds"] == 1800.0
+ assert "abandoned" in call["reason"].lower(), "the ledger row must say it was reaped, not carry a verdict the agent never reached"
+
+
+async def test_a_failing_pass_does_not_kill_the_loop(monkeypatch: pytest.MonkeyPatch) -> None:
+ """An unreachable store is transient; the sweeper backs off and retries."""
+
+ async def _boom(**_kwargs: Any) -> list[uuid.UUID]:
+ raise ConnectionError("database is down")
+
+ monkeypatch.setattr(ledger, "fail_stale_runs", _boom, raising=True)
+ with pytest.raises(ConnectionError):
+ await run_reaper.reap_once()
+
+
+# ---------------------------------------------------------------------------
+# The SQL, against the real schema
+# ---------------------------------------------------------------------------
+
+pytestmark_integration = pytest.mark.skipif(
+ not LEDGER_DSN,
+ reason="set AISOC_TEST_LEDGER_DSN to a Postgres carrying the AiSOC schema to run the ledger sweep",
+)
+
+
+async def ledger_db(monkeypatch: pytest.MonkeyPatch):
+ """Point the ledger at a real Postgres and give each test a clean tenant."""
+ import asyncpg
+
+ monkeypatch.setenv("DATABASE_URL", LEDGER_DSN)
+ await ledger.close_pool()
+
+ tenant_id = uuid.uuid4()
+ conn = await asyncpg.connect(LEDGER_DSN)
+ try:
+ await conn.execute(
+ "INSERT INTO tenants (id, name, slug) VALUES ($1, $2, $3)",
+ tenant_id,
+ f"reaper-{tenant_id.hex[:8]}",
+ f"reaper-{tenant_id.hex[:8]}",
+ )
+ finally:
+ await conn.close()
+
+ yield tenant_id
+
+ await ledger.close_pool()
+ conn = await asyncpg.connect(LEDGER_DSN)
+ try:
+ await conn.execute("DELETE FROM investigation_runs WHERE tenant_id = $1", tenant_id)
+ await conn.execute("DELETE FROM tenants WHERE id = $1", tenant_id)
+ finally:
+ await conn.close()
+
+
+async def _insert_run(dsn: str, tenant_id: uuid.UUID, *, age_seconds: float, status: str) -> uuid.UUID:
+ import asyncpg
+
+ run_id = uuid.uuid4()
+ conn = await asyncpg.connect(dsn)
+ try:
+ await conn.execute(
+ """
+ INSERT INTO investigation_runs (id, tenant_id, case_id, status, started_at, created_at)
+ VALUES ($1, $2, $3, $4, now() - make_interval(secs => $5::double precision), now())
+ """,
+ run_id,
+ tenant_id,
+ "case-reaper",
+ status,
+ float(age_seconds),
+ )
+ finally:
+ await conn.close()
+ return run_id
+
+
+async def _status_of(dsn: str, run_id: uuid.UUID) -> tuple[str, str | None, bool]:
+ import asyncpg
+
+ conn = await asyncpg.connect(dsn)
+ try:
+ row = await conn.fetchrow(
+ "SELECT status, error, completed_at IS NOT NULL AS closed FROM investigation_runs WHERE id = $1",
+ run_id,
+ )
+ finally:
+ await conn.close()
+ return row["status"], row["error"], row["closed"]
+
+
+@pytestmark_integration
+async def test_sweep_closes_an_abandoned_run_and_leaves_a_live_one_alone(ledger_db) -> None:
+ tenant_id = ledger_db
+ abandoned = await _insert_run(LEDGER_DSN, tenant_id, age_seconds=4000, status="running")
+ in_flight = await _insert_run(LEDGER_DSN, tenant_id, age_seconds=5, status="running")
+ already_done = await _insert_run(LEDGER_DSN, tenant_id, age_seconds=4000, status="completed")
+
+ closed = await ledger.fail_stale_runs(older_than_seconds=1800, reason=run_reaper.STALE_REASON)
+
+ assert abandoned in closed
+ assert in_flight not in closed, "a run still inside its budget was reaped"
+ assert already_done not in closed, "a finished run was reopened and re-closed"
+
+ status, error, has_completed_at = await _status_of(LEDGER_DSN, abandoned)
+ assert status == "failed"
+ assert error == run_reaper.STALE_REASON
+ assert has_completed_at, "a closed run with no completed_at renders as 'in progress' in the ledger"
+
+ assert (await _status_of(LEDGER_DSN, in_flight))[0] == "running"
+ assert (await _status_of(LEDGER_DSN, already_done))[0] == "completed"
+
+
+@pytestmark_integration
+async def test_sweep_is_idempotent_and_safe_to_race(ledger_db) -> None:
+ """Two replicas sweeping the same row close it once.
+
+ The staleness predicate is inside the ``UPDATE``, so the loser matches
+ nothing rather than overwriting the first writer's reason.
+ """
+ tenant_id = ledger_db
+ run_id = await _insert_run(LEDGER_DSN, tenant_id, age_seconds=4000, status="running")
+
+ first = await ledger.fail_stale_runs(older_than_seconds=1800, reason=run_reaper.STALE_REASON)
+ second = await ledger.fail_stale_runs(older_than_seconds=1800, reason="a different reason")
+
+ assert run_id in first
+ assert run_id not in second, "a second pass re-closed a row that was already terminal"
+ assert (await _status_of(LEDGER_DSN, run_id))[1] == run_reaper.STALE_REASON
+
+
+@pytestmark_integration
+async def test_sweep_sees_every_tenant(ledger_db) -> None:
+ """A maintenance sweep that only saw one tenant would leave the rest hanging.
+
+ This is the assertion that would fail if the RLS policy on
+ ``investigation_runs`` ever dropped its ``current_tenant_id() IS NULL``
+ arm — the thing the sweep silently depends on.
+ """
+ import asyncpg
+
+ tenant_a = ledger_db
+ tenant_b = uuid.uuid4()
+ conn = await asyncpg.connect(LEDGER_DSN)
+ try:
+ await conn.execute(
+ "INSERT INTO tenants (id, name, slug) VALUES ($1, $2, $3)",
+ tenant_b,
+ f"reaper-b-{tenant_b.hex[:8]}",
+ f"reaper-b-{tenant_b.hex[:8]}",
+ )
+ finally:
+ await conn.close()
+
+ try:
+ run_a = await _insert_run(LEDGER_DSN, tenant_a, age_seconds=4000, status="running")
+ run_b = await _insert_run(LEDGER_DSN, tenant_b, age_seconds=4000, status="running")
+
+ closed = await ledger.fail_stale_runs(older_than_seconds=1800, reason=run_reaper.STALE_REASON)
+
+ assert run_a in closed
+ assert run_b in closed, "the sweep only saw one tenant's abandoned runs"
+ finally:
+ conn = await asyncpg.connect(LEDGER_DSN)
+ try:
+ await conn.execute("DELETE FROM investigation_runs WHERE tenant_id = $1", tenant_b)
+ await conn.execute("DELETE FROM tenants WHERE id = $1", tenant_b)
+ finally:
+ await conn.close()
diff --git a/services/agents/tests/test_litellm_config.py b/services/agents/tests/test_litellm_config.py
index 0cb9ed5a1..53b638fba 100644
--- a/services/agents/tests/test_litellm_config.py
+++ b/services/agents/tests/test_litellm_config.py
@@ -87,3 +87,61 @@ def test_model_pins_roles_match_gateway_aliases_exactly():
# Ties the code's logical roles to the gateway config: every pinned role has
# an ``aisoc-<role>`` alias in the config, and vice versa — no drift either way.
assert {f"aisoc-{role}" for role in all_roles()} == EXPECTED_ALIASES
+
+
+# ---------------------------------------------------------------------------
+# Per-request timeout (issue #1241)
+# ---------------------------------------------------------------------------
+
+TIMEOUT_ENV_VAR = "AISOC_LITELLM_REQUEST_TIMEOUT"
+COMPOSE_PATH = REPO_ROOT / "docker-compose.yml"
+
+
+def test_request_timeout_is_operator_adjustable_without_editing_this_file():
+ """60s was a bare literal in a file every other value reads from the env.
+
+ The bundled 3B model on CPU-only inference can exceed it, and every call
+ then times out while Settings still reports the provider Live. Raising it
+ must not require editing a mounted config and rebuilding.
+ """
+ timeout = _load().get("litellm_settings", {}).get("request_timeout")
+ assert timeout is not None, "request_timeout must be declared — without it the gateway's own 6000s default applies"
+ assert str(timeout) == f"os.environ/{TIMEOUT_ENV_VAR}", (
+ f"request_timeout is {timeout!r}; an operator cannot change that without editing this file"
+ )
+
+
+def test_compose_supplies_a_default_for_the_timeout_variable():
+ """The default is load-bearing, not cosmetic.
+
+ LiteLLM resolves ``os.environ/...`` with ``get_secret``, which returns
+ ``None`` for an unset variable and assigns that straight onto
+ ``litellm.request_timeout``. Measured against
+ ``ghcr.io/berriai/litellm:main-stable``, a gateway in that state answers
+ 200 on ``/health/liveliness`` and fails *every* completion with
+ ``float() argument must be a string or a real number, not 'NoneType'`` —
+ a healthy-looking gateway serving nothing, which is the shape of the defect
+ this knob exists to fix. So the variable must always have a value on the
+ supported path.
+ """
+ compose = COMPOSE_PATH.read_text(encoding="utf-8")
+ assert f"{TIMEOUT_ENV_VAR}: ${{{TIMEOUT_ENV_VAR}:-" in compose, (
+ f"docker-compose.yml must pass {TIMEOUT_ENV_VAR} to the litellm service with a `:-` default; "
+ "without one an unset variable silently breaks every completion"
+ )
+
+
+def test_num_retries_stays_a_literal():
+ """Not every setting can be env-driven, and this one must not be.
+
+ LiteLLM hands ``router_settings`` to ``Router.__init__`` with no type
+ coercion, and the retry path evaluates ``num_retries > 0`` — which raises
+ ``TypeError`` against the string an ``os.environ/...`` reference resolves
+ to, at exactly the moment a retry is needed. Pinned so a later "make
+ everything configurable" pass cannot quietly break failover.
+ """
+ num_retries = _load().get("router_settings", {}).get("num_retries")
+ assert isinstance(num_retries, int), (
+ f"num_retries must stay an int literal, got {num_retries!r}: "
+ "LiteLLM does not coerce router_settings and `num_retries > 0` raises on a string"
+ )
diff --git a/services/api/app/api/v1/endpoints/llm_status.py b/services/api/app/api/v1/endpoints/llm_status.py
index 2968a17a8..3e82b5433 100644
--- a/services/api/app/api/v1/endpoints/llm_status.py
+++ b/services/api/app/api/v1/endpoints/llm_status.py
@@ -66,6 +66,7 @@ from app.core.airgap import is_host_allowed_for_airgap
from app.core.config import settings
from app.models.llm_credential import TenantLlmCredential
from app.security.credential_vault import CredentialVaultError, get_vault
+from app.services import llm_health
from app.services.llm_runtime import probe as probe_runtime
from app.services.model_aliases import gateway_url, is_gateway_alias, resolve_api_key
@@ -253,6 +254,14 @@ def _compute_status(
"effective_path": effective_path,
"policy_note": policy_note,
"source": source,
+ # Everything above describes *configuration*: which provider this pod
+ # would call and whether the policy would let it. Whether the provider
+ # answers is a different question, and nothing here used to ask it — so
+ # a deployment timing out on every call rendered the emerald Live pill
+ # (issue #1241). These three come from the outcomes of the calls the
+ # product already made; see app/services/llm_health.py for why they are
+ # not a probe, not a latency figure, and why `unknown` is a real answer.
+ **llm_health.snapshot(),
}
diff --git a/services/api/app/services/llm_health.py b/services/api/app/services/llm_health.py
new file mode 100644
index 000000000..72a910e31
--- /dev/null
+++ b/services/api/app/services/llm_health.py
@@ -0,0 +1,122 @@
+"""Did the LLM actually answer? (issue #1241)
+
+``GET /api/v1/llm/status`` reports *configuration* — which provider this pod is
+pointed at, and whether the air-gap policy would let a call leave. It places no
+call, so a deployment whose every completion times out still renders the
+emerald **Live** pill in Settings → Deployment & AI. The panel's own header
+claims the indicator "cannot drift from runtime behaviour", which is true of
+which provider is configured and false of whether it answers.
+
+This module is the missing half: a bounded, in-process record of how the calls
+the product is *already making* turned out.
+
+What it deliberately is not
+---------------------------
+**Not a probe.** Nothing here places a request. A synthetic completion on every
+page load would spend the tenant's budget to answer a question the real traffic
+has already answered, and on a local 3B model it is not cheap.
+
+**Not a latency number.** No figure for how long a call takes is published
+anywhere, because none has been measured for this path. ``aisoc_run_costs``
+carries a latency sum, but it is written by ``services/agents`` for completed
+runs only and records nothing at all about a call that timed out — which is the
+case that matters here.
+
+**Not a ratio.** The verdict is the outcome of the most recent observed call.
+A ratio over a short window is noise, and a ratio over a long one keeps
+reporting healthy for a provider that died five minutes ago. "The last call we
+made did not come back" is the question an operator is asking when they look at
+that badge. The counts travel alongside so a blip is distinguishable from an
+outage.
+
+Scope, stated rather than implied
+---------------------------------
+Per-process. Behind several replicas this reports what *this* pod has seen, and
+a pod that has served no AI traffic reports ``unknown`` rather than a green it
+has not earned. ``unknown`` is a first-class answer here: an absence of evidence
+rendered as health is the same defect in a different costume.
+"""
+
+from __future__ import annotations
+
+import threading
+import time
+from collections import deque
+from typing import Any, Literal
+
+#: How many outcomes to keep. Enough to say "3 of the last 12 failed" and small
+#: enough that the whole thing is a few hundred bytes per process.
+_WINDOW_SIZE = 20
+
+#: Outcomes older than this stop counting. A provider that was failing an hour
+#: ago says nothing about the one answering now, and an operator reading a red
+#: badge needs it to be about the present.
+_WINDOW_SECONDS = 900.0
+
+RecentHealth = Literal["healthy", "degraded", "unknown"]
+
+_lock = threading.Lock()
+_outcomes: deque[tuple[float, bool, str]] = deque(maxlen=_WINDOW_SIZE)
+
+
+def record_success() -> None:
+ """Note that a chat-completions call returned."""
+ with _lock:
+ _outcomes.append((time.monotonic(), True, ""))
+
+
+def record_failure(kind: str) -> None:
+ """Note that a chat-completions call did not return a usable response.
+
+ ``kind`` is the exception class name (``ReadTimeout``, ``ConnectError``,
+ ``HTTPStatusError``…) — never the message, which can carry a URL with a key
+ in it.
+ """
+ with _lock:
+ _outcomes.append((time.monotonic(), False, kind))
+
+
+def reset() -> None:
+ """Drop the window. For tests and for a deliberate operator re-baseline."""
+ with _lock:
+ _outcomes.clear()
+
+
+def snapshot() -> dict[str, Any]:
+ """The three fields ``/llm/status`` adds, derived from real outcomes only."""
+ cutoff = time.monotonic() - _WINDOW_SECONDS
+ with _lock:
+ recent = [entry for entry in _outcomes if entry[0] >= cutoff]
+
+ if not recent:
+ return {
+ "recent_health": "unknown",
+ "recent_calls": 0,
+ "recent_failures": 0,
+ "recent_note": (
+ "No LLM call has been observed by this pod recently, so whether the provider "
+ "answers is unmeasured. This reports configuration only."
+ ),
+ }
+
+ failures = [entry for entry in recent if not entry[1]]
+ last_ok = recent[-1][1]
+ last_kind = recent[-1][2]
+
+ if last_ok:
+ note = f"The most recent LLM call succeeded ({len(failures)} of the last {len(recent)} failed)."
+ health: RecentHealth = "healthy"
+ else:
+ note = (
+ f"The most recent LLM call failed ({last_kind or 'error'}); "
+ f"{len(failures)} of the last {len(recent)} failed. "
+ "If the provider is a local model on CPU, raise AISOC_LITELLM_REQUEST_TIMEOUT."
+ )
+ health = "degraded"
+
+ return {
+ "recent_health": health,
+ "recent_calls": len(recent),
+ "recent_failures": len(failures),
+ "recent_note": note,
+ }
diff --git a/services/api/app/services/llm_safety.py b/services/api/app/services/llm_safety.py
index cdb62b993..16ede0c33 100644
--- a/services/api/app/services/llm_safety.py
+++ b/services/api/app/services/llm_safety.py
@@ -66,6 +66,7 @@ from app._vendor.llm_contract_rules import (
from app._vendor.llm_contract_rules import (
validate_messages as validate_messages,
)
+from app.services import llm_health
# These names are re-exports: callers do
# `from app.services.llm_safety import LLMContractViolation`, and the rules
@@ -110,6 +111,14 @@ async def safe_chat_completions_request(
a call site can move between services without rewriting. ``client`` is
accepted so callers that already hold a pooled ``AsyncClient`` — and tests
— do not each construct one.
+
+ Every outcome from here is recorded in :mod:`app.services.llm_health`, which
+ is what lets ``GET /api/v1/llm/status`` say whether the provider is actually
+ answering instead of only which one is configured (issue #1241). This is the
+ one function the service's LLM call sites share, so recording here needs no
+ probe and no second mechanism. A contract violation is **not** recorded: it
+ raises above, before the network call, and is AiSOC refusing to send rather
+ than the provider failing to answer.
"""
validated = LLMInputContract.validate(messages)
@@ -118,12 +127,20 @@ async def safe_chat_completions_request(
headers.update(extra_headers)
payload: dict[str, Any] = {"model": model, "messages": validated, **extra_body}
- if client is not None:
- response = await client.post(url, headers=headers, json=payload, timeout=timeout)
- response.raise_for_status()
- return dict(response.json())
-
- async with httpx.AsyncClient(timeout=timeout) as owned:
- response = await owned.post(url, headers=headers, json=payload)
- response.raise_for_status()
- return dict(response.json())
+ try:
+ if client is not None:
+ response = await client.post(url, headers=headers, json=payload, timeout=timeout)
+ response.raise_for_status()
+ result = dict(response.json())
+ else:
+ async with httpx.AsyncClient(timeout=timeout) as owned:
+ response = await owned.post(url, headers=headers, json=payload)
+ response.raise_for_status()
+ result = dict(response.json())
+ except Exception as exc:
+ # Re-raised unchanged — every caller's error handling is untouched. The
+ # class name only; a message can carry a URL with a key in it.
+ llm_health.record_failure(type(exc).__name__)
+ raise
+ llm_health.record_success()
+ return result
diff --git a/services/api/tests/test_llm_status_degraded.py b/services/api/tests/test_llm_status_degraded.py
new file mode 100644
index 000000000..3cdc7bbd2
--- /dev/null
+++ b/services/api/tests/test_llm_status_degraded.py
@@ -0,0 +1,176 @@
+"""The LLM panel must not report Live while every call is failing (issue #1241).
+
+``GET /api/v1/llm/status`` places no call. ``_compute_status`` decides
+``effective_path`` from ``key_set`` and the air-gap policy alone, so a
+deployment whose every completion times out renders the emerald **Live** pill
+in Settings → Deployment & AI. The panel's own header says the indicator
+"cannot drift from runtime behaviour", which is true of *which provider is
+configured* and false of *whether it answers*.
+
+What is added is a recent-outcome signal, not a probe: ``safe_chat_completions_request``
+is the single chokepoint every LLM call in this service already goes through,
+so the calls the product is making anyway are what the status reports. No
+synthetic completion on page load, and no latency number — the honest answer to
+"how fast is it" is one nobody has measured here.
+
+The three states are deliberate, and ``unknown`` is the important one: a pod
+that has placed no call has observed nothing, and saying so is different from
+saying the provider is healthy. Reporting a fabricated green for an absence of
+data is the defect in a different costume.
+"""
+
+from __future__ import annotations
+
+import httpx
+import pytest
+from app.api.v1.endpoints.llm_status import llm_status
+from app.services import llm_health
+from app.services.llm_safety import safe_chat_completions_request
+
+_MESSAGES = [{"role": "user", "content": "Summarise this alert for an analyst."}]
+
+
[email protected](autouse=True)
+def _clean(monkeypatch: pytest.MonkeyPatch):
+ """A live-looking configuration, and an empty outcome window."""
+ for var in (
+ "LLM_BASE_URL",
+ "LLM_MODEL",
+ "OPENAI_BASE_URL",
+ "OPENAI_MODEL",
+ "AISOC_LLM_MODEL",
+ "LLM_GATEWAY_URL",
+ "LITELLM_MASTER_KEY",
+ ):
+ monkeypatch.delenv(var, raising=False)
+ monkeypatch.setenv("OPENAI_API_KEY", "sk-test")
+ monkeypatch.delenv("LLM_API_KEY", raising=False)
+ llm_health.reset()
+ yield
+ llm_health.reset()
+
+
+async def _call_through_the_real_chokepoint(handler) -> None:
+ """Drive the real ``safe_chat_completions_request`` over a mock transport.
+
+ Deliberately not a direct call to the recorder: the property under test is
+ that the function the endpoints already use records what happened, and a
+ test that records the outcome itself would pass against a recorder nothing
+ calls.
+ """
+ async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
+ await safe_chat_completions_request(
+ api_key="sk-test",
+ model="aisoc-triage",
+ messages=_MESSAGES,
+ url="http://litellm:4000/v1/chat/completions",
+ client=client,
+ )
+
+
+def _ok(_request: httpx.Request) -> httpx.Response:
+ return httpx.Response(200, json={"choices": [{"message": {"content": "ok"}}]})
+
+
+def _timeout(request: httpx.Request) -> httpx.Response:
+ raise httpx.ReadTimeout("timed out", request=request)
+
+
+# ---------------------------------------------------------------------------
+# The status surface
+# ---------------------------------------------------------------------------
+
+
+def test_no_observed_calls_reports_unknown_not_healthy() -> None:
+ """An absence of evidence is reported as such, never as a green light."""
+ status = llm_status()
+ assert status["recent_health"] == "unknown"
+ assert status["effective_path"] == "live", "configuration is still live — only the outcome is unknown"
+
+
+async def test_a_timing_out_provider_is_reported_degraded() -> None:
+ """The headline case: configured, reachable-looking, and answering nothing."""
+ for _ in range(3):
+ with pytest.raises(httpx.ReadTimeout):
+ await _call_through_the_real_chokepoint(_timeout)
+
+ status = llm_status()
+ assert status["recent_health"] == "degraded", "every recent call timed out and the panel would still render the emerald Live pill"
+ assert status["recent_failures"] == 3
+ assert "timeout" in str(status["recent_note"]).lower() or "fail" in str(status["recent_note"]).lower(), (
+ f"the operator is told nothing actionable: {status['recent_note']!r}"
+ )
+
+
+async def test_a_working_provider_is_reported_healthy() -> None:
+ await _call_through_the_real_chokepoint(_ok)
+ status = llm_status()
+ assert status["recent_health"] == "healthy"
+ assert status["recent_failures"] == 0
+
+
+async def test_recovery_clears_the_degraded_state() -> None:
+ """A provider that comes back must not stay red — this is a window, not a latch."""
+ for _ in range(3):
+ with pytest.raises(httpx.ReadTimeout):
+ await _call_through_the_real_chokepoint(_timeout)
+ assert llm_status()["recent_health"] == "degraded"
+
+ for _ in range(5):
+ await _call_through_the_real_chokepoint(_ok)
+
+ assert llm_status()["recent_health"] == "healthy", "a recovered provider stayed red"
+
+
+async def test_an_http_error_counts_as_a_failure_too() -> None:
+ """A 500 from the gateway is a failed call, not just a timeout."""
+
+ def _five_hundred(_request: httpx.Request) -> httpx.Response:
+ return httpx.Response(500, json={"error": "upstream exploded"})
+
+ for _ in range(3):
+ with pytest.raises(httpx.HTTPStatusError):
+ await _call_through_the_real_chokepoint(_five_hundred)
+
+ assert llm_status()["recent_health"] == "degraded"
+
+
+def test_status_publishes_no_latency_figure() -> None:
+ """No measured latency exists for this path, so none is published.
+
+ Guards against a future 'helpful' average that would be the fabricated
+ number the issue warns about.
+ """
+ status = llm_status()
+ assert not [k for k in status if "latency" in k or "ms" in k.lower()], (
+ f"status surfaced a latency-shaped field nobody measured: {sorted(status)}"
+ )
+
+
+# ---------------------------------------------------------------------------
+# The recorder itself
+# ---------------------------------------------------------------------------
+
+
+async def test_a_contract_violation_is_not_counted_as_a_provider_failure() -> None:
+ """Refusing to send is AiSOC working correctly, not the provider failing.
+
+ ``LLMInputContract`` aborts before the network call. Counting that as a
+ provider outage would make the panel blame the model for a prompt the
+ product deliberately refused to send.
+ """
+ from app.services.llm_safety import LLMContractViolation
+
+ raw_ocsf = '{"class_uid": 2001, "activity_id": 1, "metadata": {"product": {"name": "x"}, "version": "1.0"}}'
+ with pytest.raises(LLMContractViolation):
+ async with httpx.AsyncClient(transport=httpx.MockTransport(_ok)) as client:
+ await safe_chat_completions_request(
+ api_key="sk-test",
+ model="aisoc-triage",
+ messages=[{"role": "user", "content": raw_ocsf}],
+ client=client,
+ )
+
+ status = llm_status()
+ assert status["recent_health"] == "unknown", "a refused prompt was recorded as a provider outcome; nothing was ever sent"
+ assert status["recent_failures"] == 0
diff --git a/tests/test_env_placeholder_gate.py b/tests/test_env_placeholder_gate.py
index 99be4246e..e7f50b6e1 100644
--- a/tests/test_env_placeholder_gate.py
+++ b/tests/test_env_placeholder_gate.py
@@ -114,6 +114,12 @@ DELIBERATE_DEFAULTS: dict[str, str] = {
"OPENAI_MODEL": "a logical task alias the gateway defines",
"LITELLM_MASTER_KEY": "the local gateway's own key; the gateway is in-network only",
"LLM_GATEWAY_URL": "the compose hostname of the gateway",
+ "AISOC_LITELLM_REQUEST_TIMEOUT": (
+ "a duration in seconds, and the value the gateway actually runs with. It "
+ "was a literal inside infra/litellm/config.yaml, so raising it meant "
+ "editing a mounted file; it is surfaced here because CPU-only inference "
+ "on the bundled 3B model is the deployment that has to raise it"
+ ),
"TAXII_FEEDS": "a real public MITRE ATT&CK TAXII endpoint",
"CISA_KEV_ENABLED": "a boolean",
"ABUSE_CH_ENABLED": "a boolean",