Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

fix(chaos): store-outage legs grade real behaviour — readiness surfaces, refusal-reading probes, per-store contracts (#1446) (#1471)

Beenu Arora412fc0f0b42b
diff --git a/.github/workflows/chaos.yml b/.github/workflows/chaos.yml
index 1e957f118..481857131 100644
--- a/.github/workflows/chaos.yml
+++ b/.github/workflows/chaos.yml
@@ -92,9 +92,68 @@ jobs:
strategy:
# One store's result must not cancel the others — the interesting
# output is which of the six behave and which do not.
+ #
+ # Each leg names the readiness surface that OWNS the store and the
+ # dependents the expectation declares (#1446). The previous shape
+ # probed one API URL for all six stores and two fixed dependents, and
+ # every one of its findings was an artifact of that contract rather
+ # than a fact about a service: `/api/v1/health/ready` did not exist
+ # (404 → every key "reads None"), `/api/v1/health` did not exist
+ # either (404 → "api stopped serving" on every leg), the `ingest`
+ # URL pointed at the *enrichment* service's host port (8080; ingest
+ # publishes 8081), and the recovery poll re-read the same 404
+ # forever ("never reconnected"). clickhouse and neo4j are behind
+ # `--profile full`, so those legs graded a container that was never
+ # started and exited 2 blaming Docker.
+ #
+ # Dependent URLs point at readiness-class endpoints: a 503 there
+ # means "alive but cannot do my job", which `_responding` counts as
+ # not serving — the required outcome for a block store's dependents.
+ # Legs whose owning service does not yet export the store's key
+ # (fusion: lake_writer/kafka; ingest: kafka/graph) stay red with the
+ # finding naming that service — that red is the job of this
+ # workflow, and the follow-ups live on #1446.
fail-fast: false
matrix:
- store: [clickhouse, neo4j, qdrant, redis, kafka, postgres]
+ include:
+ - store: postgres
+ profile: core
+ readiness_url: http://127.0.0.1:8000/api/v1/health/ready
+ dependent_urls: >-
+ api=http://127.0.0.1:8000/api/v1/health/ready
+ fusion=http://127.0.0.1:8003/health
+ agents=http://127.0.0.1:8001/health
+ actions=http://127.0.0.1:8002/health
+ - store: redis
+ profile: core
+ readiness_url: http://127.0.0.1:8000/api/v1/health/ready
+ dependent_urls: >-
+ api=http://127.0.0.1:8000/api/v1/health/ready
+ agents=http://127.0.0.1:8001/health
+ - store: qdrant
+ profile: core
+ readiness_url: http://127.0.0.1:8005/health
+ dependent_urls: >-
+ threatintel=http://127.0.0.1:8005/health
+ - store: kafka
+ profile: core
+ readiness_url: http://127.0.0.1:8081/health
+ dependent_urls: >-
+ ingest=http://127.0.0.1:8081/health
+ fusion=http://127.0.0.1:8003/health
+ realtime=http://127.0.0.1:8086/health
+ - store: clickhouse
+ profile: full
+ readiness_url: http://127.0.0.1:8003/health
+ dependent_urls: >-
+ fusion=http://127.0.0.1:8003/health
+ api=http://127.0.0.1:8000/api/v1/health/ready
+ - store: neo4j
+ profile: full
+ readiness_url: http://127.0.0.1:8081/health
+ dependent_urls: >-
+ ingest=http://127.0.0.1:8081/health
+ api=http://127.0.0.1:8000/api/v1/health/ready
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
@@ -112,24 +171,42 @@ jobs:
- name: Bring the stack up
id: boot
+ env:
+ PROFILE: ${{ matrix.profile }}
run: |
set -euo pipefail
- docker compose up -d --wait --wait-timeout 300 || {
- echo "::error::the stack did not come up; the outage below would grade nothing"
- docker compose ps
- exit 1
- }
+ # The full profile is how clickhouse and neo4j exist at all: they
+ # are `--profile full` services, and the flags are what turn the
+ # lake writer and graph writer on in fusion and ingest. Grading a
+ # full-profile store on the core stack stops a container that was
+ # never started.
+ if [ "$PROFILE" = "full" ]; then
+ AISOC_LAKE_WRITER_ENABLED=true AISOC_GRAPH_ENABLED=true \
+ docker compose --profile full up -d --wait --wait-timeout 300 || {
+ echo "::error::the stack did not come up; the outage below would grade nothing"
+ docker compose --profile full ps
+ exit 1
+ }
+ else
+ docker compose up -d --wait --wait-timeout 300 || {
+ echo "::error::the stack did not come up; the outage below would grade nothing"
+ docker compose ps
+ exit 1
+ }
+ fi
- name: Take ${{ matrix.store }} down and grade what the services say
env:
STORE: ${{ matrix.store }}
+ READINESS_URL: ${{ matrix.readiness_url }}
+ DEPENDENT_URLS: ${{ matrix.dependent_urls }}
run: |
set -euo pipefail
- python3 scripts/chaos/store_outage.py \
- --store "$STORE" \
- --readiness-url http://127.0.0.1:8000/api/v1/health/ready \
- --dependent-url "api=http://127.0.0.1:8000/api/v1/health" \
- --dependent-url "ingest=http://127.0.0.1:8080/health"
+ args=(--store "$STORE" --readiness-url "$READINESS_URL")
+ for pair in $DEPENDENT_URLS; do
+ args+=(--dependent-url "$pair")
+ done
+ python3 scripts/chaos/store_outage.py "${args[@]}"
- name: Logs, on failure
if: failure()
diff --git a/.security/allowlist.yml b/.security/allowlist.yml
index cebe36e7b..9ca156561 100644
--- a/.security/allowlist.yml
+++ b/.security/allowlist.yml
@@ -255,7 +255,18 @@ ceilings:
# is the reason `Semgrep (ratcheted)` stays unrequired, and it is
# unchanged by pinning the rules; requiring the job still waits on that
# +/-1 being attributed and eliminated.
- total: 111
+ # 111 -> 112 (error unchanged at 44). A fifth and sixth
+ # `dynamic-urllib-use-detected`, both in scripts/chaos/store_outage.py
+ # (#1446), which probes the liveness/readiness routes the chaos legs
+ # grade. Net +1 against the ceiling: the rewrite replaced one flagged
+ # call and added one. Both call sites refuse any URL that does not start
+ # with the fixed localhost probe prefixes — the same shape as the four
+ # findings above — so the `file://` read the rule warns about is not
+ # reachable; the rule fires on any non-literal argument to `urlopen`.
+ # Counted rather than suppressed, per the note at the bottom of this
+ # file. Measured with semgrep 1.86.0 on the vendored packs
+ # (rules_sha256 below): 112 total / 44 error.
+ total: 112
error: 44
measured_with: "1.86.0"
rules_sha256: "5af76658e7e28b96b1099d2c995d7c85b3b9a4f96795a39df3d02aa7d295cb99"
diff --git a/docs/openapi.yaml b/docs/openapi.yaml
index e872c797d..33bdec9cd 100644
--- a/docs/openapi.yaml
+++ b/docs/openapi.yaml
@@ -10792,6 +10792,67 @@ paths:
$ref: '#/components/schemas/HTTPValidationError'
security:
- HTTPBearer: []
+ /api/v1/health:
+ get:
+ tags:
+ - health
+ summary: Health Liveness
+ description: 'Liveness: the process answers HTTP. Nothing else is claimed.
+
+
+ Deliberately touches no store — a liveness probe that fails when a
+
+ dependency is down invites the orchestrator to restart a healthy
+
+ process, which converts an outage into an outage plus a restart storm.
+
+ Whether the service can do its *job* is ``/ready``''s question.'
+ operationId: health_liveness_api_v1_health_get
+ responses:
+ '200':
+ description: Successful Response
+ content:
+ application/json:
+ schema:
+ additionalProperties: true
+ type: object
+ title: Response Health Liveness Api V1 Health Get
+ /api/v1/health/ready:
+ get:
+ tags:
+ - health
+ summary: Health Ready
+ description: 'Readiness: can this service do its job, named per dependency.
+
+
+ ``database`` is **block**: the API is the system of record''s front door,
+
+ so without Postgres the status code goes 503 and a load balancer should
+
+ stop sending traffic here. The body still names every dependency —
+
+ failing *with the reason* is the whole point, and the chaos grader reads
+
+ the keys out of the 503 body.
+
+
+ ``cache`` is **degrade**: Redis is a cache and every read falls through
+
+ to its source, so its loss is reported (`cache: false`) while the status
+
+ stays 200 and the service keeps serving. Failing readiness over a cache
+
+ would pull the API out of the load balancer to fix a cache.'
+ operationId: health_ready_api_v1_health_ready_get
+ responses:
+ '200':
+ description: Successful Response
+ content:
+ application/json:
+ schema:
+ additionalProperties: true
+ type: object
+ title: Response Health Ready Api V1 Health Ready Get
/api/v1/insights/soc:
get:
tags:
diff --git a/scripts/chaos/store_outage.py b/scripts/chaos/store_outage.py
index 8e9744f1b..c9b188d6a 100644
--- a/scripts/chaos/store_outage.py
+++ b/scripts/chaos/store_outage.py
@@ -39,9 +39,11 @@ restart.
from __future__ import annotations
import argparse
+import http.server
import json
import subprocess
import sys
+import threading
import time
import urllib.error
import urllib.request
@@ -297,6 +299,16 @@ _ALLOWED_PROBE_PREFIXES = ("http://127.0.0.1:", "http://localhost:", "https://12
def _probe(url: str, timeout: float = 4.0) -> dict[str, Any] | None:
+ """Read a readiness body, *including* from an HTTP error response.
+
+ A block-store dependency failing readiness with a 503 **and a JSON body
+ naming why** is exactly the correct behaviour — the API answers 503 with
+ ``{"database": false, ...}`` while Postgres is down. ``urlopen`` raises on
+ any 4xx/5xx, so without reading ``HTTPError``'s body the grader scored
+ that correct behaviour as silence ("``database`` reads None"). Connection
+ refused, timeouts and non-JSON bodies still read as ``None``: those are
+ genuinely "nothing was said".
+ """
if not url.startswith(_ALLOWED_PROBE_PREFIXES):
raise ValueError(
f"refusing to probe {url!r}: this harness only reads readiness endpoints on the "
@@ -305,10 +317,35 @@ def _probe(url: str, timeout: float = 4.0) -> dict[str, Any] | None:
try:
with urllib.request.urlopen(url, timeout=timeout) as response: # noqa: S310 - prefix-checked above
return dict(json.loads(response.read().decode("utf-8")))
+ except urllib.error.HTTPError as err:
+ try:
+ return dict(json.loads(err.read().decode("utf-8")))
+ except (ValueError, OSError):
+ return None
except (urllib.error.URLError, TimeoutError, ValueError, OSError):
return None
+def _responding(url: str, timeout: float = 4.0) -> bool:
+ """Is a dependent *serving its job*? Only a 2xx counts.
+
+ Deliberately stricter than :func:`_probe`: dependents are probed at their
+ readiness endpoints, where a 503 means "alive but cannot do my job" — for
+ a block store that is the required outcome, and counting it as responding
+ would grade the correct refusal as a finding.
+ """
+ if not url.startswith(_ALLOWED_PROBE_PREFIXES):
+ raise ValueError(
+ f"refusing to probe {url!r}: this harness only reads readiness endpoints on the "
+ f"local stack it is taking down, which must start with one of {_ALLOWED_PROBE_PREFIXES}"
+ )
+ try:
+ with urllib.request.urlopen(url, timeout=timeout) as response: # noqa: S310 - prefix-checked above
+ return 200 <= response.status < 300
+ except (urllib.error.URLError, TimeoutError, ValueError, OSError):
+ return False
+
+
def _self_test() -> int:
"""Every branch of the grader, without containers.
@@ -397,10 +434,69 @@ def _self_test() -> int:
print(f" self-test [FAIL] {store.store} declares unknown behaviour {store.behaviour!r}")
failures += 1
+ failures += _probe_self_test()
+
print(f"store_outage: self-test {'OK' if not failures else 'FAILED'} — {len(STORES)} stores declared")
return 1 if failures else 0
+def _probe_self_test() -> int:
+ """The probe split, against a real local HTTP server (stdlib, no containers).
+
+ The behaviours under test are the ones the live run depends on:
+
+ * ``_probe`` reads the JSON body of a **503** — a block store failing
+ readiness with a reason is correct behaviour, not silence.
+ * ``_responding`` refuses that same 503 — a dependent that is alive but
+ cannot do its job is not serving.
+ """
+
+ class _Handler(http.server.BaseHTTPRequestHandler):
+ def do_GET(self) -> None: # noqa: N802 - stdlib handler contract
+ if self.path == "/ready-200":
+ body = json.dumps({"status": "ready", "database": True}).encode()
+ self.send_response(200)
+ elif self.path == "/ready-503":
+ body = json.dumps({"status": "not_ready", "database": False}).encode()
+ self.send_response(503)
+ else:
+ body = b"not json"
+ self.send_response(503)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(body)))
+ self.end_headers()
+ self.wfile.write(body)
+
+ def log_message(self, *_args: Any) -> None: # keep the self-test quiet
+ return None
+
+ server = http.server.HTTPServer(("127.0.0.1", 0), _Handler)
+ thread = threading.Thread(target=server.serve_forever, daemon=True)
+ thread.start()
+ base = f"http://127.0.0.1:{server.server_address[1]}"
+ failures = 0
+ try:
+ cases = [
+ ("_probe reads a 200 body", _probe(f"{base}/ready-200") == {"status": "ready", "database": True}),
+ (
+ "_probe reads the JSON body of a 503 (a refusal with a reason is not silence)",
+ _probe(f"{base}/ready-503") == {"status": "not_ready", "database": False},
+ ),
+ ("_probe returns None for a non-JSON error body", _probe(f"{base}/ready-junk") is None),
+ ("_responding accepts a 200", _responding(f"{base}/ready-200") is True),
+ ("_responding refuses a 503 (alive but not doing its job is not serving)", _responding(f"{base}/ready-503") is False),
+ ("_responding refuses a dead socket", _responding("http://127.0.0.1:1/ready") is False),
+ ]
+ for name, ok in cases:
+ print(f" self-test [{'ok' if ok else 'FAIL'}] {name}")
+ if not ok:
+ failures += 1
+ finally:
+ server.shutdown()
+ thread.join(timeout=5)
+ return failures
+
+
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--self-test", action="store_true", help="prove the grader separates the outcomes")
@@ -416,6 +512,13 @@ def main(argv: list[str] | None = None) -> int:
parser.error("--store and --readiness-url are required unless --self-test")
expectation = next(s for s in STORES if s.store == args.store)
+ if _resolve_container(expectation.container) is None:
+ print(
+ f"store_outage: container {expectation.container!r} is not running in this stack — "
+ "the store is probably behind a compose profile this job did not start "
+ "(clickhouse and neo4j live behind `--profile full`). Grading nothing is not a pass."
+ )
+ return 2
if not _container_action(expectation.container, "stop"):
print(f"store_outage: could not stop container {expectation.container!r} — docker unavailable?")
return 2
@@ -426,7 +529,7 @@ def main(argv: list[str] | None = None) -> int:
responding: dict[str, bool] = {}
for pair in args.dependent_url:
name, _, url = pair.partition("=")
- responding[name] = _probe(url) is not None
+ responding[name] = _responding(url)
finally:
_container_action(expectation.container, "start")
diff --git a/scripts/check_route_auth.py b/scripts/check_route_auth.py
index ef2b0b465..cba03766e 100644
--- a/scripts/check_route_auth.py
+++ b/scripts/check_route_auth.py
@@ -98,6 +98,14 @@ PUBLIC_ROUTES: dict[str, str] = {
"services/agents/app/api/router.py::health": "liveness probe returning a static status document",
"services/agents/app/main.py::health": "liveness probe returning a static status document",
"services/api/app/main.py::health_check": "liveness probe; the readiness detail behind it is already gated",
+ "services/api/app/api/v1/endpoints/health.py::health_liveness": (
+ "liveness probe (#1446): the process answers HTTP — touches no store, returns no tenant data, "
+ "and an LB or kubelet has no credential to offer"
+ ),
+ "services/api/app/api/v1/endpoints/health.py::health_ready": (
+ "readiness probe (#1446): booleans about the service's own dependencies, named per store so a "
+ "gone dependency is distinguishable from a healthy idle one; no tenant data and no caller input"
+ ),
"services/connectors/app/api/router.py::health": "liveness probe, deliberately kept off the guarded connectors router",
"services/fusion/app/api/router.py::health": "liveness probe returning a static status document",
"services/honeytokens/app/main.py::health": "liveness probe returning a static status document",
diff --git a/services/api/app/api/v1/endpoints/health.py b/services/api/app/api/v1/endpoints/health.py
index 83ff7a787..3b7152f34 100644
--- a/services/api/app/api/v1/endpoints/health.py
+++ b/services/api/app/api/v1/endpoints/health.py
@@ -73,16 +73,20 @@ rule in ``connector_freshness``.
from __future__ import annotations
+import asyncio
+import contextlib
import logging
from datetime import UTC, datetime, timedelta
from typing import Annotated, Any
-from fastapi import APIRouter, Depends
+from fastapi import APIRouter, Depends, Response, status
+from redis.asyncio import from_url as redis_from_url
from sqlalchemy import and_, func, select, text
from app.api.v1.deps import AuthUser, CurrentUser, DBSession, require_permission
from app.api.v1.endpoints.metrics import PipelineHealth, PipelineStage
from app.core.config import settings
+from app.db.database import engine
from app.db.rls import TenantDBSession
from app.models.alert import Alert
from app.models.connector import Connector
@@ -1033,3 +1037,95 @@ async def replay_dead_letters(
Defaults to a dry run. Pass `execute: true` once the preview is clean.
"""
return await run_replay(db, tenant_id=user.tenant_id, requested_by=user.user_id, request=request)
+
+
+# ---------------------------------------------------------------------------
+# Liveness and readiness (#1446)
+# ---------------------------------------------------------------------------
+#
+# Both unauthenticated: an LB or probe has no credential, and the bodies
+# carry only booleans about the service's own dependencies.
+#
+# The split follows the consumer doctrine: *liveness* says the process is
+# serving HTTP (restarting it would not help anything), *readiness* says the
+# service can do its job right now and names each dependency it checked —
+# including the healthy ones, because a dependency that is gone while the
+# body omits it is indistinguishable from a healthy idle one. The chaos
+# store-outage grader (scripts/chaos/store_outage.py) reads exactly these
+# keys; until these routes existed it probed a 404 and every leg's first
+# finding was an artifact of that rather than a fact about the service.
+
+
+async def _database_reachable() -> bool:
+ """One ``SELECT 1`` through the real engine, hard-capped.
+
+ The cap matters more than it looks: the chaos grader reads readiness with
+ a 4-second budget *while a store is down*, so a probe that waits out a
+ full connect timeout turns "down and we say so" into "down and we say
+ nothing".
+ """
+ try:
+ async with asyncio.timeout(1.5):
+ async with engine.connect() as conn:
+ await conn.execute(text("SELECT 1"))
+ return True
+ except Exception: # noqa: BLE001 — any failure is the answer, not an error
+ return False
+
+
+async def _cache_reachable() -> bool:
+ """One Redis PING, hard-capped, on a throwaway connection."""
+ client = None
+ try:
+ async with asyncio.timeout(1.5):
+ client = redis_from_url(
+ str(settings.REDIS_URL),
+ socket_connect_timeout=1.0,
+ socket_timeout=1.0,
+ )
+ return bool(await client.ping())
+ except Exception: # noqa: BLE001
+ return False
+ finally:
+ if client is not None:
+ with contextlib.suppress(Exception):
+ await client.aclose()
+
+
+async def health_liveness() -> dict[str, Any]:
+ """Liveness: the process answers HTTP. Nothing else is claimed.
+
+ Deliberately touches no store — a liveness probe that fails when a
+ dependency is down invites the orchestrator to restart a healthy
+ process, which converts an outage into an outage plus a restart storm.
+ Whether the service can do its *job* is ``/ready``'s question.
+ """
+ return {"status": "alive", "service": "aisoc-api"}
+
+
+async def health_ready(response: Response) -> dict[str, Any]:
+ """Readiness: can this service do its job, named per dependency.
+
+ ``database`` is **block**: the API is the system of record's front door,
+ so without Postgres the status code goes 503 and a load balancer should
+ stop sending traffic here. The body still names every dependency —
+ failing *with the reason* is the whole point, and the chaos grader reads
+ the keys out of the 503 body.
+
+ ``cache`` is **degrade**: Redis is a cache and every read falls through
+ to its source, so its loss is reported (`cache: false`) while the status
+ stays 200 and the service keeps serving. Failing readiness over a cache
+ would pull the API out of the load balancer to fix a cache.
+ """
+ database, cache = await asyncio.gather(_database_reachable(), _cache_reachable())
+ ready = database
+ if not ready:
+ response.status_code = status.HTTP_503_SERVICE_UNAVAILABLE
+ return {
+ "status": "ready" if ready else "not_ready",
+ "service": "aisoc-api",
+ "database": database,
+ "cache": cache,
+ }
diff --git a/services/api/tests/test_health_ready_dependencies.py b/services/api/tests/test_health_ready_dependencies.py
new file mode 100644
index 000000000..6e2ee18d3
--- /dev/null
+++ b/services/api/tests/test_health_ready_dependencies.py
@@ -0,0 +1,98 @@
+"""``/api/v1/health`` and ``/api/v1/health/ready`` (#1446).
+
+The chaos store-outage grader reads these two surfaces: readiness must name
+every dependency it checked (a store that is gone while the body omits it is
+indistinguishable from a healthy idle one), and the status code must track the
+service's own job — 503 only for the block dependency (Postgres), 200 with the
+key flipped for the degrade one (Redis). The grader reads the 503 *body*, so
+the keys must be present on the failure path too.
+
+Probes are monkeypatched here: what this file locks down is the route
+semantics, not connectivity — the live halves are exercised by the chaos
+workflow against real containers (redis and postgres legs, #1446).
+"""
+
+from __future__ import annotations
+
+import pytest
+from app.api.v1.endpoints import health
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+
+def _client() -> TestClient:
+ app = FastAPI()
+ app.include_router(health.router, prefix="/api/v1")
+ return TestClient(app)
+
+
+async def _up() -> bool:
+ return True
+
+
+async def _down() -> bool:
+ return False
+
+
+def test_liveness_answers_without_touching_any_store(monkeypatch: pytest.MonkeyPatch) -> None:
+ """A liveness probe that fails when a dependency is down invites the
+ orchestrator to restart a healthy process. Poisoning both probes proves
+ the route cannot reach them."""
+
+ def _boom() -> bool:
+ raise AssertionError("liveness touched a store probe")
+
+ monkeypatch.setattr(health, "_database_reachable", _boom)
+ monkeypatch.setattr(health, "_cache_reachable", _boom)
+
+ response = _client().get("/api/v1/health")
+
+ assert response.status_code == 200
+ assert response.json() == {"status": "alive", "service": "aisoc-api"}
+
+
+def test_ready_names_every_dependency_when_healthy(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(health, "_database_reachable", _up)
+ monkeypatch.setattr(health, "_cache_reachable", _up)
+
+ response = _client().get("/api/v1/health/ready")
+
+ assert response.status_code == 200
+ assert response.json() == {
+ "status": "ready",
+ "service": "aisoc-api",
+ "database": True,
+ "cache": True,
+ }
+
+
+def test_cache_loss_degrades_without_failing_readiness(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Redis is a cache: its loss is reported while the service keeps serving.
+ Failing readiness here would pull the API out of the load balancer to fix
+ a cache."""
+ monkeypatch.setattr(health, "_database_reachable", _up)
+ monkeypatch.setattr(health, "_cache_reachable", _down)
+
+ response = _client().get("/api/v1/health/ready")
+
+ assert response.status_code == 200
+ body = response.json()
+ assert body["status"] == "ready"
+ assert body["cache"] is False
+ assert body["database"] is True
+
+
+def test_database_loss_fails_readiness_with_the_reason_in_the_body(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Postgres is block: 503 so the load balancer stops sending traffic —
+ and the body still names every key, because the chaos grader (and any
+ operator) reads the reason out of the failure response."""
+ monkeypatch.setattr(health, "_database_reachable", _down)
+ monkeypatch.setattr(health, "_cache_reachable", _up)
+
+ response = _client().get("/api/v1/health/ready")
+
+ assert response.status_code == 503
+ body = response.json()
+ assert body["status"] == "not_ready"
+ assert body["database"] is False
+ assert body["cache"] is True
diff --git a/services/threatintel/app/main.py b/services/threatintel/app/main.py
index 88a4c66d5..99e7adda0 100644
--- a/services/threatintel/app/main.py
+++ b/services/threatintel/app/main.py
@@ -10,6 +10,7 @@ AiSOC — open-source AI Security Operations Center (MIT License)
from __future__ import annotations
+import asyncio
from collections.abc import AsyncGenerator
from contextlib import asynccontextmanager
from functools import partial
@@ -340,17 +341,37 @@ app.include_router(indicators_router)
@app.get("/health")
async def health() -> dict:
- """Health and status endpoint."""
+ """Health and status endpoint, naming every dependency it checked.
+
+ ``vector_store`` is the Qdrant similarity leg (#1446): a **degrade**
+ dependency — IOC ingestion continues without it and enrichment responses
+ say the similarity leg was skipped — so its loss flips the key while the
+ status stays ``ok``. The chaos store-outage grader reads exactly this
+ key; before it existed, Qdrant being down was invisible here, which is
+ indistinguishable from a healthy idle store. Each probe is hard-capped
+ well inside the grader's 4-second read budget, because a probe that
+ waits out a full connect timeout turns "down and we say so" into "down
+ and we say nothing".
+ """
try:
- redis: aioredis.Redis = app.state.redis
- await redis.ping()
+ async with asyncio.timeout(1.5):
+ redis: aioredis.Redis = app.state.redis
+ await redis.ping()
redis_ok = True
except Exception:
redis_ok = False
+ try:
+ async with asyncio.timeout(1.5):
+ await app.state.qdrant_client.get_collections()
+ vector_ok = True
+ except Exception:
+ vector_ok = False
+
return {
"status": "ok",
"redis": redis_ok,
+ "vector_store": vector_ok,
"scheduler": app.state.scheduler._scheduler.running if hasattr(app.state, "scheduler") else False,
"airgap": airgap_status(),
}