Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
fix(chaos): store-outage legs grade real behaviour — readiness surfaces, refusal-reading probes, per-store contracts (#1446) (#1471)
Beenu Arora412fc0f0b42b
diff --git a/.github/workflows/chaos.yml b/.github/workflows/chaos.ymlindex 1e957f118..481857131 100644--- a/.github/workflows/chaos.yml+++ b/.github/workflows/chaos.yml@@ -92,9 +92,68 @@ jobs:strategy:# One store's result must not cancel the others — the interesting# output is which of the six behave and which do not.+ #+ # Each leg names the readiness surface that OWNS the store and the+ # dependents the expectation declares (#1446). The previous shape+ # probed one API URL for all six stores and two fixed dependents, and+ # every one of its findings was an artifact of that contract rather+ # than a fact about a service: `/api/v1/health/ready` did not exist+ # (404 → every key "reads None"), `/api/v1/health` did not exist+ # either (404 → "api stopped serving" on every leg), the `ingest`+ # URL pointed at the *enrichment* service's host port (8080; ingest+ # publishes 8081), and the recovery poll re-read the same 404+ # forever ("never reconnected"). clickhouse and neo4j are behind+ # `--profile full`, so those legs graded a container that was never+ # started and exited 2 blaming Docker.+ #+ # Dependent URLs point at readiness-class endpoints: a 503 there+ # means "alive but cannot do my job", which `_responding` counts as+ # not serving — the required outcome for a block store's dependents.+ # Legs whose owning service does not yet export the store's key+ # (fusion: lake_writer/kafka; ingest: kafka/graph) stay red with the+ # finding naming that service — that red is the job of this+ # workflow, and the follow-ups live on #1446.fail-fast: falsematrix:- store: [clickhouse, neo4j, qdrant, redis, kafka, postgres]+ include:+ - store: postgres+ profile: core+ readiness_url: http://127.0.0.1:8000/api/v1/health/ready+ dependent_urls: >-+ api=http://127.0.0.1:8000/api/v1/health/ready+ fusion=http://127.0.0.1:8003/health+ agents=http://127.0.0.1:8001/health+ actions=http://127.0.0.1:8002/health+ - store: redis+ profile: core+ readiness_url: http://127.0.0.1:8000/api/v1/health/ready+ dependent_urls: >-+ api=http://127.0.0.1:8000/api/v1/health/ready+ agents=http://127.0.0.1:8001/health+ - store: qdrant+ profile: core+ readiness_url: http://127.0.0.1:8005/health+ dependent_urls: >-+ threatintel=http://127.0.0.1:8005/health+ - store: kafka+ profile: core+ readiness_url: http://127.0.0.1:8081/health+ dependent_urls: >-+ ingest=http://127.0.0.1:8081/health+ fusion=http://127.0.0.1:8003/health+ realtime=http://127.0.0.1:8086/health+ - store: clickhouse+ profile: full+ readiness_url: http://127.0.0.1:8003/health+ dependent_urls: >-+ fusion=http://127.0.0.1:8003/health+ api=http://127.0.0.1:8000/api/v1/health/ready+ - store: neo4j+ profile: full+ readiness_url: http://127.0.0.1:8081/health+ dependent_urls: >-+ ingest=http://127.0.0.1:8081/health+ api=http://127.0.0.1:8000/api/v1/health/readysteps:- uses: actions/checkout@v7- uses: actions/setup-python@v7@@ -112,24 +171,42 @@ jobs:- name: Bring the stack upid: boot+ env:+ PROFILE: ${{ matrix.profile }}run: |set -euo pipefail- docker compose up -d --wait --wait-timeout 300 || {- echo "::error::the stack did not come up; the outage below would grade nothing"- docker compose ps- exit 1- }+ # The full profile is how clickhouse and neo4j exist at all: they+ # are `--profile full` services, and the flags are what turn the+ # lake writer and graph writer on in fusion and ingest. Grading a+ # full-profile store on the core stack stops a container that was+ # never started.+ if [ "$PROFILE" = "full" ]; then+ AISOC_LAKE_WRITER_ENABLED=true AISOC_GRAPH_ENABLED=true \+ docker compose --profile full up -d --wait --wait-timeout 300 || {+ echo "::error::the stack did not come up; the outage below would grade nothing"+ docker compose --profile full ps+ exit 1+ }+ else+ docker compose up -d --wait --wait-timeout 300 || {+ echo "::error::the stack did not come up; the outage below would grade nothing"+ docker compose ps+ exit 1+ }+ fi- name: Take ${{ matrix.store }} down and grade what the services sayenv:STORE: ${{ matrix.store }}+ READINESS_URL: ${{ matrix.readiness_url }}+ DEPENDENT_URLS: ${{ matrix.dependent_urls }}run: |set -euo pipefail- python3 scripts/chaos/store_outage.py \- --store "$STORE" \- --readiness-url http://127.0.0.1:8000/api/v1/health/ready \- --dependent-url "api=http://127.0.0.1:8000/api/v1/health" \- --dependent-url "ingest=http://127.0.0.1:8080/health"+ args=(--store "$STORE" --readiness-url "$READINESS_URL")+ for pair in $DEPENDENT_URLS; do+ args+=(--dependent-url "$pair")+ done+ python3 scripts/chaos/store_outage.py "${args[@]}"- name: Logs, on failureif: failure()diff --git a/.security/allowlist.yml b/.security/allowlist.ymlindex cebe36e7b..9ca156561 100644--- a/.security/allowlist.yml+++ b/.security/allowlist.yml@@ -255,7 +255,18 @@ ceilings:# is the reason `Semgrep (ratcheted)` stays unrequired, and it is# unchanged by pinning the rules; requiring the job still waits on that# +/-1 being attributed and eliminated.- total: 111+ # 111 -> 112 (error unchanged at 44). A fifth and sixth+ # `dynamic-urllib-use-detected`, both in scripts/chaos/store_outage.py+ # (#1446), which probes the liveness/readiness routes the chaos legs+ # grade. Net +1 against the ceiling: the rewrite replaced one flagged+ # call and added one. Both call sites refuse any URL that does not start+ # with the fixed localhost probe prefixes — the same shape as the four+ # findings above — so the `file://` read the rule warns about is not+ # reachable; the rule fires on any non-literal argument to `urlopen`.+ # Counted rather than suppressed, per the note at the bottom of this+ # file. Measured with semgrep 1.86.0 on the vendored packs+ # (rules_sha256 below): 112 total / 44 error.+ total: 112error: 44measured_with: "1.86.0"rules_sha256: "5af76658e7e28b96b1099d2c995d7c85b3b9a4f96795a39df3d02aa7d295cb99"diff --git a/docs/openapi.yaml b/docs/openapi.yamlindex e872c797d..33bdec9cd 100644--- a/docs/openapi.yaml+++ b/docs/openapi.yaml@@ -10792,6 +10792,67 @@ paths:$ref: '#/components/schemas/HTTPValidationError'security:- HTTPBearer: []+ /api/v1/health:+ get:+ tags:+ - health+ summary: Health Liveness+ description: 'Liveness: the process answers HTTP. Nothing else is claimed.+++ Deliberately touches no store — a liveness probe that fails when a++ dependency is down invites the orchestrator to restart a healthy++ process, which converts an outage into an outage plus a restart storm.++ Whether the service can do its *job* is ``/ready``''s question.'+ operationId: health_liveness_api_v1_health_get+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ additionalProperties: true+ type: object+ title: Response Health Liveness Api V1 Health Get+ /api/v1/health/ready:+ get:+ tags:+ - health+ summary: Health Ready+ description: 'Readiness: can this service do its job, named per dependency.+++ ``database`` is **block**: the API is the system of record''s front door,++ so without Postgres the status code goes 503 and a load balancer should++ stop sending traffic here. The body still names every dependency —++ failing *with the reason* is the whole point, and the chaos grader reads++ the keys out of the 503 body.+++ ``cache`` is **degrade**: Redis is a cache and every read falls through++ to its source, so its loss is reported (`cache: false`) while the status++ stays 200 and the service keeps serving. Failing readiness over a cache++ would pull the API out of the load balancer to fix a cache.'+ operationId: health_ready_api_v1_health_ready_get+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ additionalProperties: true+ type: object+ title: Response Health Ready Api V1 Health Ready Get/api/v1/insights/soc:get:tags:diff --git a/scripts/chaos/store_outage.py b/scripts/chaos/store_outage.pyindex 8e9744f1b..c9b188d6a 100644--- a/scripts/chaos/store_outage.py+++ b/scripts/chaos/store_outage.py@@ -39,9 +39,11 @@ restart.from __future__ import annotationsimport argparse+import http.serverimport jsonimport subprocessimport sys+import threadingimport timeimport urllib.errorimport urllib.request@@ -297,6 +299,16 @@ _ALLOWED_PROBE_PREFIXES = ("http://127.0.0.1:", "http://localhost:", "https://12def _probe(url: str, timeout: float = 4.0) -> dict[str, Any] | None:+ """Read a readiness body, *including* from an HTTP error response.++ A block-store dependency failing readiness with a 503 **and a JSON body+ naming why** is exactly the correct behaviour — the API answers 503 with+ ``{"database": false, ...}`` while Postgres is down. ``urlopen`` raises on+ any 4xx/5xx, so without reading ``HTTPError``'s body the grader scored+ that correct behaviour as silence ("``database`` reads None"). Connection+ refused, timeouts and non-JSON bodies still read as ``None``: those are+ genuinely "nothing was said".+ """if not url.startswith(_ALLOWED_PROBE_PREFIXES):raise ValueError(f"refusing to probe {url!r}: this harness only reads readiness endpoints on the "@@ -305,10 +317,35 @@ def _probe(url: str, timeout: float = 4.0) -> dict[str, Any] | None:try:with urllib.request.urlopen(url, timeout=timeout) as response: # noqa: S310 - prefix-checked abovereturn dict(json.loads(response.read().decode("utf-8")))+ except urllib.error.HTTPError as err:+ try:+ return dict(json.loads(err.read().decode("utf-8")))+ except (ValueError, OSError):+ return Noneexcept (urllib.error.URLError, TimeoutError, ValueError, OSError):return None+def _responding(url: str, timeout: float = 4.0) -> bool:+ """Is a dependent *serving its job*? Only a 2xx counts.++ Deliberately stricter than :func:`_probe`: dependents are probed at their+ readiness endpoints, where a 503 means "alive but cannot do my job" — for+ a block store that is the required outcome, and counting it as responding+ would grade the correct refusal as a finding.+ """+ if not url.startswith(_ALLOWED_PROBE_PREFIXES):+ raise ValueError(+ f"refusing to probe {url!r}: this harness only reads readiness endpoints on the "+ f"local stack it is taking down, which must start with one of {_ALLOWED_PROBE_PREFIXES}"+ )+ try:+ with urllib.request.urlopen(url, timeout=timeout) as response: # noqa: S310 - prefix-checked above+ return 200 <= response.status < 300+ except (urllib.error.URLError, TimeoutError, ValueError, OSError):+ return False++def _self_test() -> int:"""Every branch of the grader, without containers.@@ -397,10 +434,69 @@ def _self_test() -> int:print(f" self-test [FAIL] {store.store} declares unknown behaviour {store.behaviour!r}")failures += 1+ failures += _probe_self_test()+print(f"store_outage: self-test {'OK' if not failures else 'FAILED'} — {len(STORES)} stores declared")return 1 if failures else 0+def _probe_self_test() -> int:+ """The probe split, against a real local HTTP server (stdlib, no containers).++ The behaviours under test are the ones the live run depends on:++ * ``_probe`` reads the JSON body of a **503** — a block store failing+ readiness with a reason is correct behaviour, not silence.+ * ``_responding`` refuses that same 503 — a dependent that is alive but+ cannot do its job is not serving.+ """++ class _Handler(http.server.BaseHTTPRequestHandler):+ def do_GET(self) -> None: # noqa: N802 - stdlib handler contract+ if self.path == "/ready-200":+ body = json.dumps({"status": "ready", "database": True}).encode()+ self.send_response(200)+ elif self.path == "/ready-503":+ body = json.dumps({"status": "not_ready", "database": False}).encode()+ self.send_response(503)+ else:+ body = b"not json"+ self.send_response(503)+ self.send_header("Content-Type", "application/json")+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ def log_message(self, *_args: Any) -> None: # keep the self-test quiet+ return None++ server = http.server.HTTPServer(("127.0.0.1", 0), _Handler)+ thread = threading.Thread(target=server.serve_forever, daemon=True)+ thread.start()+ base = f"http://127.0.0.1:{server.server_address[1]}"+ failures = 0+ try:+ cases = [+ ("_probe reads a 200 body", _probe(f"{base}/ready-200") == {"status": "ready", "database": True}),+ (+ "_probe reads the JSON body of a 503 (a refusal with a reason is not silence)",+ _probe(f"{base}/ready-503") == {"status": "not_ready", "database": False},+ ),+ ("_probe returns None for a non-JSON error body", _probe(f"{base}/ready-junk") is None),+ ("_responding accepts a 200", _responding(f"{base}/ready-200") is True),+ ("_responding refuses a 503 (alive but not doing its job is not serving)", _responding(f"{base}/ready-503") is False),+ ("_responding refuses a dead socket", _responding("http://127.0.0.1:1/ready") is False),+ ]+ for name, ok in cases:+ print(f" self-test [{'ok' if ok else 'FAIL'}] {name}")+ if not ok:+ failures += 1+ finally:+ server.shutdown()+ thread.join(timeout=5)+ return failures++def main(argv: list[str] | None = None) -> int:parser = argparse.ArgumentParser(description=__doc__)parser.add_argument("--self-test", action="store_true", help="prove the grader separates the outcomes")@@ -416,6 +512,13 @@ def main(argv: list[str] | None = None) -> int:parser.error("--store and --readiness-url are required unless --self-test")expectation = next(s for s in STORES if s.store == args.store)+ if _resolve_container(expectation.container) is None:+ print(+ f"store_outage: container {expectation.container!r} is not running in this stack — "+ "the store is probably behind a compose profile this job did not start "+ "(clickhouse and neo4j live behind `--profile full`). Grading nothing is not a pass."+ )+ return 2if not _container_action(expectation.container, "stop"):print(f"store_outage: could not stop container {expectation.container!r} — docker unavailable?")return 2@@ -426,7 +529,7 @@ def main(argv: list[str] | None = None) -> int:responding: dict[str, bool] = {}for pair in args.dependent_url:name, _, url = pair.partition("=")- responding[name] = _probe(url) is not None+ responding[name] = _responding(url)finally:_container_action(expectation.container, "start")diff --git a/scripts/check_route_auth.py b/scripts/check_route_auth.pyindex ef2b0b465..cba03766e 100644--- a/scripts/check_route_auth.py+++ b/scripts/check_route_auth.py@@ -98,6 +98,14 @@ PUBLIC_ROUTES: dict[str, str] = {"services/agents/app/api/router.py::health": "liveness probe returning a static status document","services/agents/app/main.py::health": "liveness probe returning a static status document","services/api/app/main.py::health_check": "liveness probe; the readiness detail behind it is already gated",+ "services/api/app/api/v1/endpoints/health.py::health_liveness": (+ "liveness probe (#1446): the process answers HTTP — touches no store, returns no tenant data, "+ "and an LB or kubelet has no credential to offer"+ ),+ "services/api/app/api/v1/endpoints/health.py::health_ready": (+ "readiness probe (#1446): booleans about the service's own dependencies, named per store so a "+ "gone dependency is distinguishable from a healthy idle one; no tenant data and no caller input"+ ),"services/connectors/app/api/router.py::health": "liveness probe, deliberately kept off the guarded connectors router","services/fusion/app/api/router.py::health": "liveness probe returning a static status document","services/honeytokens/app/main.py::health": "liveness probe returning a static status document",diff --git a/services/api/app/api/v1/endpoints/health.py b/services/api/app/api/v1/endpoints/health.pyindex 83ff7a787..3b7152f34 100644--- a/services/api/app/api/v1/endpoints/health.py+++ b/services/api/app/api/v1/endpoints/health.py@@ -73,16 +73,20 @@ rule in ``connector_freshness``.from __future__ import annotations+import asyncio+import contextlibimport loggingfrom datetime import UTC, datetime, timedeltafrom typing import Annotated, Any-from fastapi import APIRouter, Depends+from fastapi import APIRouter, Depends, Response, status+from redis.asyncio import from_url as redis_from_urlfrom sqlalchemy import and_, func, select, textfrom app.api.v1.deps import AuthUser, CurrentUser, DBSession, require_permissionfrom app.api.v1.endpoints.metrics import PipelineHealth, PipelineStagefrom app.core.config import settings+from app.db.database import enginefrom app.db.rls import TenantDBSessionfrom app.models.alert import Alertfrom app.models.connector import Connector@@ -1033,3 +1037,95 @@ async def replay_dead_letters(Defaults to a dry run. Pass `execute: true` once the preview is clean."""return await run_replay(db, tenant_id=user.tenant_id, requested_by=user.user_id, request=request)+++# ---------------------------------------------------------------------------+# Liveness and readiness (#1446)+# ---------------------------------------------------------------------------+#+# Both unauthenticated: an LB or probe has no credential, and the bodies+# carry only booleans about the service's own dependencies.+#+# The split follows the consumer doctrine: *liveness* says the process is+# serving HTTP (restarting it would not help anything), *readiness* says the+# service can do its job right now and names each dependency it checked —+# including the healthy ones, because a dependency that is gone while the+# body omits it is indistinguishable from a healthy idle one. The chaos+# store-outage grader (scripts/chaos/store_outage.py) reads exactly these+# keys; until these routes existed it probed a 404 and every leg's first+# finding was an artifact of that rather than a fact about the service.+++async def _database_reachable() -> bool:+ """One ``SELECT 1`` through the real engine, hard-capped.++ The cap matters more than it looks: the chaos grader reads readiness with+ a 4-second budget *while a store is down*, so a probe that waits out a+ full connect timeout turns "down and we say so" into "down and we say+ nothing".+ """+ try:+ async with asyncio.timeout(1.5):+ async with engine.connect() as conn:+ await conn.execute(text("SELECT 1"))+ return True+ except Exception: # noqa: BLE001 — any failure is the answer, not an error+ return False+++async def _cache_reachable() -> bool:+ """One Redis PING, hard-capped, on a throwaway connection."""+ client = None+ try:+ async with asyncio.timeout(1.5):+ client = redis_from_url(+ str(settings.REDIS_URL),+ socket_connect_timeout=1.0,+ socket_timeout=1.0,+ )+ return bool(await client.ping())+ except Exception: # noqa: BLE001+ return False+ finally:+ if client is not None:+ with contextlib.suppress(Exception):+ await client.aclose()+++async def health_liveness() -> dict[str, Any]:+ """Liveness: the process answers HTTP. Nothing else is claimed.++ Deliberately touches no store — a liveness probe that fails when a+ dependency is down invites the orchestrator to restart a healthy+ process, which converts an outage into an outage plus a restart storm.+ Whether the service can do its *job* is ``/ready``'s question.+ """+ return {"status": "alive", "service": "aisoc-api"}++[email protected]("/ready")+async def health_ready(response: Response) -> dict[str, Any]:+ """Readiness: can this service do its job, named per dependency.++ ``database`` is **block**: the API is the system of record's front door,+ so without Postgres the status code goes 503 and a load balancer should+ stop sending traffic here. The body still names every dependency —+ failing *with the reason* is the whole point, and the chaos grader reads+ the keys out of the 503 body.++ ``cache`` is **degrade**: Redis is a cache and every read falls through+ to its source, so its loss is reported (`cache: false`) while the status+ stays 200 and the service keeps serving. Failing readiness over a cache+ would pull the API out of the load balancer to fix a cache.+ """+ database, cache = await asyncio.gather(_database_reachable(), _cache_reachable())+ ready = database+ if not ready:+ response.status_code = status.HTTP_503_SERVICE_UNAVAILABLE+ return {+ "status": "ready" if ready else "not_ready",+ "service": "aisoc-api",+ "database": database,+ "cache": cache,+ }diff --git a/services/api/tests/test_health_ready_dependencies.py b/services/api/tests/test_health_ready_dependencies.pynew file mode 100644index 000000000..6e2ee18d3--- /dev/null+++ b/services/api/tests/test_health_ready_dependencies.py@@ -0,0 +1,98 @@+"""``/api/v1/health`` and ``/api/v1/health/ready`` (#1446).++The chaos store-outage grader reads these two surfaces: readiness must name+every dependency it checked (a store that is gone while the body omits it is+indistinguishable from a healthy idle one), and the status code must track the+service's own job — 503 only for the block dependency (Postgres), 200 with the+key flipped for the degrade one (Redis). The grader reads the 503 *body*, so+the keys must be present on the failure path too.++Probes are monkeypatched here: what this file locks down is the route+semantics, not connectivity — the live halves are exercised by the chaos+workflow against real containers (redis and postgres legs, #1446).+"""++from __future__ import annotations++import pytest+from app.api.v1.endpoints import health+from fastapi import FastAPI+from fastapi.testclient import TestClient+++def _client() -> TestClient:+ app = FastAPI()+ app.include_router(health.router, prefix="/api/v1")+ return TestClient(app)+++async def _up() -> bool:+ return True+++async def _down() -> bool:+ return False+++def test_liveness_answers_without_touching_any_store(monkeypatch: pytest.MonkeyPatch) -> None:+ """A liveness probe that fails when a dependency is down invites the+ orchestrator to restart a healthy process. Poisoning both probes proves+ the route cannot reach them."""++ def _boom() -> bool:+ raise AssertionError("liveness touched a store probe")++ monkeypatch.setattr(health, "_database_reachable", _boom)+ monkeypatch.setattr(health, "_cache_reachable", _boom)++ response = _client().get("/api/v1/health")++ assert response.status_code == 200+ assert response.json() == {"status": "alive", "service": "aisoc-api"}+++def test_ready_names_every_dependency_when_healthy(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr(health, "_database_reachable", _up)+ monkeypatch.setattr(health, "_cache_reachable", _up)++ response = _client().get("/api/v1/health/ready")++ assert response.status_code == 200+ assert response.json() == {+ "status": "ready",+ "service": "aisoc-api",+ "database": True,+ "cache": True,+ }+++def test_cache_loss_degrades_without_failing_readiness(monkeypatch: pytest.MonkeyPatch) -> None:+ """Redis is a cache: its loss is reported while the service keeps serving.+ Failing readiness here would pull the API out of the load balancer to fix+ a cache."""+ monkeypatch.setattr(health, "_database_reachable", _up)+ monkeypatch.setattr(health, "_cache_reachable", _down)++ response = _client().get("/api/v1/health/ready")++ assert response.status_code == 200+ body = response.json()+ assert body["status"] == "ready"+ assert body["cache"] is False+ assert body["database"] is True+++def test_database_loss_fails_readiness_with_the_reason_in_the_body(monkeypatch: pytest.MonkeyPatch) -> None:+ """Postgres is block: 503 so the load balancer stops sending traffic —+ and the body still names every key, because the chaos grader (and any+ operator) reads the reason out of the failure response."""+ monkeypatch.setattr(health, "_database_reachable", _down)+ monkeypatch.setattr(health, "_cache_reachable", _up)++ response = _client().get("/api/v1/health/ready")++ assert response.status_code == 503+ body = response.json()+ assert body["status"] == "not_ready"+ assert body["database"] is False+ assert body["cache"] is Truediff --git a/services/threatintel/app/main.py b/services/threatintel/app/main.pyindex 88a4c66d5..99e7adda0 100644--- a/services/threatintel/app/main.py+++ b/services/threatintel/app/main.py@@ -10,6 +10,7 @@ AiSOC — open-source AI Security Operations Center (MIT License)from __future__ import annotations+import asynciofrom collections.abc import AsyncGeneratorfrom contextlib import asynccontextmanagerfrom functools import partial@@ -340,17 +341,37 @@ app.include_router(indicators_router)@app.get("/health")async def health() -> dict:- """Health and status endpoint."""+ """Health and status endpoint, naming every dependency it checked.++ ``vector_store`` is the Qdrant similarity leg (#1446): a **degrade**+ dependency — IOC ingestion continues without it and enrichment responses+ say the similarity leg was skipped — so its loss flips the key while the+ status stays ``ok``. The chaos store-outage grader reads exactly this+ key; before it existed, Qdrant being down was invisible here, which is+ indistinguishable from a healthy idle store. Each probe is hard-capped+ well inside the grader's 4-second read budget, because a probe that+ waits out a full connect timeout turns "down and we say so" into "down+ and we say nothing".+ """try:- redis: aioredis.Redis = app.state.redis- await redis.ping()+ async with asyncio.timeout(1.5):+ redis: aioredis.Redis = app.state.redis+ await redis.ping()redis_ok = Trueexcept Exception:redis_ok = False+ try:+ async with asyncio.timeout(1.5):+ await app.state.qdrant_client.get_collections()+ vector_ok = True+ except Exception:+ vector_ok = False+return {"status": "ok","redis": redis_ok,+ "vector_store": vector_ok,"scheduler": app.state.scheduler._scheduler.running if hasattr(app.state, "scheduler") else False,"airgap": airgap_status(),}