Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

10.2 KB
Raw Blame

AiSOC fix log — October 2026 session

Every issue diagnosed and fixed on the live deployment (10.200.0.69), in the order encountered, with root cause and verification evidence. Branch: fix/aisoc-live-session-2026-10 on https://github.com/alexmateescu/AiSOC.

1. Cross-service proxy calls sent no credentials (401 → instant logout)

Symptom: clicking "Investigate with agent" (and other agent-backed buttons) bounced the user to /login instantly. The web client's 401 self-heal (handleUnauthorized in apps/web/src/lib/api.ts) wipes localStorage and redirects on ANY 401 — the session was fine; a backend hop was failing.

Root cause: the core API's proxies to the agents service (_agents_proxy in cases.py, the proxies in playbooks.py and graphql/query.py) forwarded no Authorization header. The agents guard (require_console_or_service_auth) fails closed → 403/401 inside the proxy → re-raised as the endpoint's own 401 → client logout.

Fix: every proxy call site now attaches Authorization: Bearer <AISOC_AGENTS_SERVICE_TOKEN|AISOC_SERVICE_TOKEN> plus X-AiSOC-Tenant-ID for tenant declaration. Files: services/api/app/api/v1/endpoints/cases.py, playbooks.py, services/api/app/graphql/query.py.

Verified: full flow replayed inside the network: login → investigate → poll run to completed, with negative controls (no creds → 401 still, guards intact).

2. Agent tools → core API lake queries 401

Symptom: agents logs showed investigation_tool.failed HTTPStatusError for process_activity, entity_timeline, fleet_ioc_hunt, etc.

Root cause: services/agents/app/tools/investigation.py called POST /api/v1/graph/investigate/query with only X-Tenant-ID. That route authenticates via get_current_user, which accepts only JWTs or DB-backed aisoc_ API keys — a tenant header is not a credential.

Fix: tools now forward Authorization: Bearer $AISOC_AGENTS_API_KEY (minted DB-backed key, scopes least-privilege: lake:query et al.); services/agents/app/context/bundle.py got the same treatment. The key is injected via .env → compose (AISOC_AGENTS_API_KEY: ${AISOC_...:-}), never committed. Note: service tokens are NOT valid on these routes — earlier config wired the service token into the API-key variable, guaranteeing 401.

Verified: pivot probe from inside aisoc-agents → 200, available: true.

3. Case status PATCH 422 (legacy vocabulary)

Symptom: PATCH /api/v1/cases/{id} with status="in_progress" → 422 Invalid status transition / unprocessable.

Root cause: backend enforces a canonical status ladder (new→triaged→investigating→contained→resolved→closed); the UI and stored rows carry legacy labels (open, in_progress, pending, …). The Literal type rejected legacy tokens before any normalization could run.

Fix: services/api/app/api/v1/endpoints/cases.py: _STATUS_ALIASES + _normalize_status() map legacy→canonical before validation; the CaseStatus Literal lists the aliases too; transition rule relaxed from "adjacent rung only" to forward-monotonic (forward skips legal, backwards 422 — reopening needs the explicit reopen action).

Verified: same PATCH → 200; backwards transition still 422.

4. Missing status text labels in the UI

Symptom: status dropdown options rendered blank (selectable, no text).

Root cause: vocabulary rename swept the dropdown values but not the Record<vocab, …> lookup maps — STATUS_LABEL / STATUS_DOT in CaseWorkspace.tsx were keyed only on one vocabulary; undefined renders as nothing, silently.

Fix: both maps extended to resolve canonical AND legacy keys. Same sweep applied to the alerts side: triaged added to the AlertStatus union in packages/types/src/alert.ts and to the queue filter chips array in AlertsView.tsx (a DB-valid status missing from the chips array is invisible in the UI even though the API accepts it).

5. Investigation panel / tab empty although runs completed

Symptom: "investigation started, steps will appear below" — nothing ever appeared. Ledger tab showed history; investigation tab stayed blank.

Root cause (three stacked layers):

  1. Poll auth: API→agents status polls lacked X-AiSOC-Tenant-ID → 403 on every poll. Fixed with header + service token (§1 pattern).
  2. List endpoint gap: agents implements NO GET /cases/{id}/investigations (only /investigations/{run_id}). The API's list route swallowed the 404 and returned {"runs": []} forever → UI never discovered the run. Fixed with a postgres-ledger fallback in the API list route (watch varchar-vs-uuid coercion in case_id.in_() — a 500 of its own caught in verification).
  3. No restore-on-mount: the Investigation tab's run state lives only in React memory. Any refresh (or viewing a case investigated earlier/background) fell back to the idle prompt. Fixed with a cancel-safe mount effect that fetches the runs list, attaches the latest terminal run, hydrates detail + report.md.

Verified: GET /cases/{id}/investigations → 200 with 8 runs; detail probe → full recon/forensic/responder/audit_log payload.

6. LLM timeout on T4 (runs "failed" silently)

Symptom: litellm 408 Timeout passed=60.0 storms; agents investigation.budget_timeout; panel stayed empty although the ledger said completed.

Root cause: qwen3:8b is a thinking model; ollama 0.6.7 cannot disable thinking for it; reasoning chains on multi-KB SOC prompts exceed 60 s on the Tesla T4 — every LLM call timed out (2 retries × 60 s) while the run machinery kept limping.

Fix: ollama pull qwen2.5:7b-instruct (no-thinking, same size class); .env AISOC_LLM_MODEL_FAST/_DEEP → ollama_chat/qwen2.5:7b-instruct; docker compose up -d litellm agents. Model changes are .env-only — no rebuild needed, aliases resolve at runtime.

Verified: smoke probe latency 0.2–6 s vs 60 s timeouts; investigation runs complete with findings and reports.

7. React error #31 on the investigation tab

Symptom: clicking Investigate threw Minified React error #31 … object with keys {priority, action, rationale, risk}.

Root cause: the Response card rendered responder.recommended_actions items raw, assuming strings — the LLM emits objects {priority, action, rationale, risk}.

Fix: card coerces: action text + P{priority} prefix + risk badge + rationale sub-text; strings still supported.

8. Overview timeline permanently blank

Symptom: Overview tab timeline stuck on "Quiet so far".

Root cause: the case-detail GET returns no timeline key at all; a dedicated /cases/{id}/timeline endpoint (synthesising opened/status/ investigation events) existed and worked (200) — the workspace simply never called it, rendering caseRecord.timeline ?? [].

Fix: mount-fetch casesApi.getTimeline(caseId), merged with inline events deduped by event id.

9. Alert status not following agent verdicts

Symptom: alerts triaged 19 h ago still shown as new.

Root cause: two-column trap — verdicts land in alerts.disposition, the badge reads alerts.status, and the persist path never followed. An escalate verdict deliberately leaves status open for a human (by design); but benign/FP auto-closes and any real verdict should at least lift new→triaged.

Fix: services/agents/app/investigator/ledger.py persist_auto_triage now lifts new → triaged whenever a real (non-shadow) verdict is persisted; further-along statuses never downgraded. One-off backfill UPDATE applied for pre-existing stale rows.

10. Approvals dispatch shipped empty permission sets (502)

Symptom: POST /approvals/{id}/decide → 502; approval decided in DB but actions service refused: "approver lacks 'actions:execute:high'" — even for platform admins.

Root cause: _dispatch_decision read user.permissions / user.roles, which don't exist on CurrentUser (it carries scopes / resolved_permissions / role) — getattr(..., []) silently dispatched an empty grant set.

Fix: resolve through the same three-way precedence CurrentUser uses (scopes → resolved_permissions → ROLE_PERMISSIONS[role]); tenant_id added to the principal.

11. Bulk redisposition 422 on apply

Symptom: analyst bulk override recorded fine (phase 1) but apply → 422.

Root cause: apply REQUIRES confirmation_token (sha256 over sorted alert_ids + disposition); the UI omitted it.

Fix: computeRedispositionToken() in apps/web/src/lib/api.ts mirrors the server canonical form; pure-JS SHA-256 fallback because the console runs over plain HTTP where crypto.subtle is undefined.

12. Release-assignee silent no-op

Symptom: "Release" button toasted success but the alert stayed assigned.

Root cause: key-name mismatch (UI sends assignee, model field is assigned_to_id — pydantic silently drops unknown keys) stacked with is not None making explicit JSON null indistinguishable from absent.

Fix: Field(alias="assignee") + write triggered on model_fields_set key-presence. Verified both ways: clearing clears, absent preserves.

13. Lake infrastructure never started

Symptom: every lake pivot failed with clickhouse:9000 name resolution / LakeQueryError.

Root cause: docker compose up deliberately skips optional infra — aisoc-clickhouse / aisoc-opensearch / aisoc-ingest were defined but had never been started on this box.

Fix: docker compose up -d clickhouse opensearch (first boot runs init SQL ~2 min; ClickHouse auth = user aisoc + CLICKHOUSE_PASSWORD from .env, not the compose default). Ingest worker (aisoc-ingest) was already running. ClickHouse now answers; raw_events currently 0 rows — ingestion backfill is a known open item, not a bug in the fixed chain.

Environment note (GPU)

docker-compose.yml ollama service gained an nvidia GPU reservation block (Tesla T4) — required for the 11434 inference endpoint to keep GPU access under compose v2 scheduler semantics.

Verification snapshot (deployed state)

  • api health 200, web 200, agents 200, clickhouse up, opensearch healthy
  • PATCH case status (legacy vocab) → 200
  • investigations list → 200 with runs; detail → full payload
  • litellm smoke: 0.2–6 s responses
  • negative controls intact: no-creds → 401 everywhere