Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
AiSOC fix log — October 2026 session
Every issue diagnosed and fixed on the live deployment (10.200.0.69), in the
order encountered, with root cause and verification evidence. Branch:
fix/aisoc-live-session-2026-10 on https://github.com/alexmateescu/AiSOC.
1. Cross-service proxy calls sent no credentials (401 → instant logout)
Symptom: clicking "Investigate with agent" (and other agent-backed
buttons) bounced the user to /login instantly. The web client's 401 self-heal
(handleUnauthorized in apps/web/src/lib/api.ts) wipes localStorage and
redirects on ANY 401 — the session was fine; a backend hop was failing.
Root cause: the core API's proxies to the agents service
(_agents_proxy in cases.py, the proxies in playbooks.py and
graphql/query.py) forwarded no Authorization header. The agents guard
(require_console_or_service_auth) fails closed → 403/401 inside the proxy →
re-raised as the endpoint's own 401 → client logout.
Fix: every proxy call site now attaches
Authorization: Bearer <AISOC_AGENTS_SERVICE_TOKEN|AISOC_SERVICE_TOKEN> plus
X-AiSOC-Tenant-ID for tenant declaration. Files:
services/api/app/api/v1/endpoints/cases.py, playbooks.py,
services/api/app/graphql/query.py.
Verified: full flow replayed inside the network: login → investigate → poll run to completed, with negative controls (no creds → 401 still, guards intact).
2. Agent tools → core API lake queries 401
Symptom: agents logs showed investigation_tool.failed HTTPStatusError
for process_activity, entity_timeline, fleet_ioc_hunt, etc.
Root cause: services/agents/app/tools/investigation.py called
POST /api/v1/graph/investigate/query with only X-Tenant-ID. That route
authenticates via get_current_user, which accepts only JWTs or DB-backed
aisoc_ API keys — a tenant header is not a credential.
Fix: tools now forward Authorization: Bearer $AISOC_AGENTS_API_KEY
(minted DB-backed key, scopes least-privilege: lake:query et al.);
services/agents/app/context/bundle.py got the same treatment. The key is
injected via .env → compose (AISOC_AGENTS_API_KEY: ${AISOC_...:-}),
never committed. Note: service tokens are NOT valid on these routes — earlier
config wired the service token into the API-key variable, guaranteeing 401.
Verified: pivot probe from inside aisoc-agents → 200,
available: true.
3. Case status PATCH 422 (legacy vocabulary)
Symptom: PATCH /api/v1/cases/{id} with status="in_progress" → 422
Invalid status transition / unprocessable.
Root cause: backend enforces a canonical status ladder
(new→triaged→investigating→contained→resolved→closed); the UI and stored
rows carry legacy labels (open, in_progress, pending, …). The Literal
type rejected legacy tokens before any normalization could run.
Fix: services/api/app/api/v1/endpoints/cases.py:
_STATUS_ALIASES + _normalize_status() map legacy→canonical before
validation; the CaseStatus Literal lists the aliases too; transition rule
relaxed from "adjacent rung only" to forward-monotonic (forward skips
legal, backwards 422 — reopening needs the explicit reopen action).
Verified: same PATCH → 200; backwards transition still 422.
4. Missing status text labels in the UI
Symptom: status dropdown options rendered blank (selectable, no text).
Root cause: vocabulary rename swept the dropdown values but not the
Record<vocab, …> lookup maps — STATUS_LABEL / STATUS_DOT in
CaseWorkspace.tsx were keyed only on one vocabulary; undefined renders as
nothing, silently.
Fix: both maps extended to resolve canonical AND legacy keys. Same sweep
applied to the alerts side: triaged added to the AlertStatus union in
packages/types/src/alert.ts and to the queue filter chips array in
AlertsView.tsx (a DB-valid status missing from the chips array is invisible
in the UI even though the API accepts it).
5. Investigation panel / tab empty although runs completed
Symptom: "investigation started, steps will appear below" — nothing ever appeared. Ledger tab showed history; investigation tab stayed blank.
Root cause (three stacked layers):
- Poll auth: API→agents status polls lacked
X-AiSOC-Tenant-ID→ 403 on every poll. Fixed with header + service token (§1 pattern). - List endpoint gap: agents implements NO
GET /cases/{id}/investigations(only/investigations/{run_id}). The API's list route swallowed the 404 and returned{"runs": []}forever → UI never discovered the run. Fixed with a postgres-ledger fallback in the API list route (watch varchar-vs-uuid coercion incase_id.in_()— a 500 of its own caught in verification). - No restore-on-mount: the Investigation tab's run state lives only in React memory. Any refresh (or viewing a case investigated earlier/background) fell back to the idle prompt. Fixed with a cancel-safe mount effect that fetches the runs list, attaches the latest terminal run, hydrates detail + report.md.
Verified: GET /cases/{id}/investigations → 200 with 8 runs; detail
probe → full recon/forensic/responder/audit_log payload.
6. LLM timeout on T4 (runs "failed" silently)
Symptom: litellm 408 Timeout passed=60.0 storms; agents
investigation.budget_timeout; panel stayed empty although the ledger said
completed.
Root cause: qwen3:8b is a thinking model; ollama 0.6.7 cannot disable
thinking for it; reasoning chains on multi-KB SOC prompts exceed 60 s on the
Tesla T4 — every LLM call timed out (2 retries × 60 s) while the run
machinery kept limping.
Fix: ollama pull qwen2.5:7b-instruct (no-thinking, same size class);
.env AISOC_LLM_MODEL_FAST/_DEEP → ollama_chat/qwen2.5:7b-instruct;
docker compose up -d litellm agents. Model changes are .env-only — no
rebuild needed, aliases resolve at runtime.
Verified: smoke probe latency 0.2–6 s vs 60 s timeouts; investigation runs complete with findings and reports.
7. React error #31 on the investigation tab
Symptom: clicking Investigate threw Minified React error #31 … object with keys {priority, action, rationale, risk}.
Root cause: the Response card rendered responder.recommended_actions
items raw, assuming strings — the LLM emits objects
{priority, action, rationale, risk}.
Fix: card coerces: action text + P{priority} prefix + risk badge +
rationale sub-text; strings still supported.
8. Overview timeline permanently blank
Symptom: Overview tab timeline stuck on "Quiet so far".
Root cause: the case-detail GET returns no timeline key at all; a
dedicated /cases/{id}/timeline endpoint (synthesising opened/status/
investigation events) existed and worked (200) — the workspace simply never
called it, rendering caseRecord.timeline ?? [].
Fix: mount-fetch casesApi.getTimeline(caseId), merged with inline
events deduped by event id.
9. Alert status not following agent verdicts
Symptom: alerts triaged 19 h ago still shown as new.
Root cause: two-column trap — verdicts land in alerts.disposition, the
badge reads alerts.status, and the persist path never followed. An
escalate verdict deliberately leaves status open for a human (by design);
but benign/FP auto-closes and any real verdict should at least lift
new→triaged.
Fix: services/agents/app/investigator/ledger.py persist_auto_triage
now lifts new → triaged whenever a real (non-shadow) verdict is persisted;
further-along statuses never downgraded. One-off backfill UPDATE applied for
pre-existing stale rows.
10. Approvals dispatch shipped empty permission sets (502)
Symptom: POST /approvals/{id}/decide → 502; approval decided in DB but
actions service refused: "approver lacks 'actions:execute:high'" — even for
platform admins.
Root cause: _dispatch_decision read user.permissions / user.roles,
which don't exist on CurrentUser (it carries scopes /
resolved_permissions / role) — getattr(..., []) silently dispatched an
empty grant set.
Fix: resolve through the same three-way precedence CurrentUser uses
(scopes → resolved_permissions → ROLE_PERMISSIONS[role]); tenant_id added
to the principal.
11. Bulk redisposition 422 on apply
Symptom: analyst bulk override recorded fine (phase 1) but apply → 422.
Root cause: apply REQUIRES confirmation_token (sha256 over sorted
alert_ids + disposition); the UI omitted it.
Fix: computeRedispositionToken() in apps/web/src/lib/api.ts mirrors
the server canonical form; pure-JS SHA-256 fallback because the console runs
over plain HTTP where crypto.subtle is undefined.
12. Release-assignee silent no-op
Symptom: "Release" button toasted success but the alert stayed assigned.
Root cause: key-name mismatch (UI sends assignee, model field is
assigned_to_id — pydantic silently drops unknown keys) stacked with
is not None making explicit JSON null indistinguishable from absent.
Fix: Field(alias="assignee") + write triggered on
model_fields_set key-presence. Verified both ways: clearing clears,
absent preserves.
13. Lake infrastructure never started
Symptom: every lake pivot failed with clickhouse:9000 name resolution /
LakeQueryError.
Root cause: docker compose up deliberately skips optional infra —
aisoc-clickhouse / aisoc-opensearch / aisoc-ingest were defined but had never
been started on this box.
Fix: docker compose up -d clickhouse opensearch (first boot runs init
SQL ~2 min; ClickHouse auth = user aisoc + CLICKHOUSE_PASSWORD from
.env, not the compose default). Ingest worker (aisoc-ingest) was already
running. ClickHouse now answers; raw_events currently 0 rows — ingestion
backfill is a known open item, not a bug in the fixed chain.
Environment note (GPU)
docker-compose.yml ollama service gained an nvidia GPU reservation block
(Tesla T4) — required for the 11434 inference endpoint to keep GPU access
under compose v2 scheduler semantics.
Verification snapshot (deployed state)
- api health 200, web 200, agents 200, clickhouse up, opensearch healthy
- PATCH case status (legacy vocab) → 200
- investigations list → 200 with runs; detail → full payload
- litellm smoke: 0.2–6 s responses
- negative controls intact: no-creds → 401 everywhere