Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

docs: maintenance index, fix log, ops runbook, known gaps (Oct 2026 session)

Alexandru Mateescuc5f8f60daecb
diff --git a/docs/fix-log-2026-10.md b/docs/fix-log-2026-10.md
new file mode 100644
index 000000000..091419f30
--- /dev/null
+++ b/docs/fix-log-2026-10.md
@@ -0,0 +1,229 @@
+# AiSOC fix log — October 2026 session
+
+Every issue diagnosed and fixed on the live deployment (10.200.0.69), in the
+order encountered, with root cause and verification evidence. Branch:
+`fix/aisoc-live-session-2026-10` on https://github.com/alexmateescu/AiSOC.
+
+## 1. Cross-service proxy calls sent no credentials (401 → instant logout)
+
+**Symptom:** clicking "Investigate with agent" (and other agent-backed
+buttons) bounced the user to /login instantly. The web client's 401 self-heal
+(`handleUnauthorized` in `apps/web/src/lib/api.ts`) wipes localStorage and
+redirects on ANY 401 — the session was fine; a backend hop was failing.
+
+**Root cause:** the core API's proxies to the agents service
+(`_agents_proxy` in `cases.py`, the proxies in `playbooks.py` and
+`graphql/query.py`) forwarded no `Authorization` header. The agents guard
+(`require_console_or_service_auth`) fails closed → 403/401 inside the proxy →
+re-raised as the endpoint's own 401 → client logout.
+
+**Fix:** every proxy call site now attaches
+`Authorization: Bearer <AISOC_AGENTS_SERVICE_TOKEN|AISOC_SERVICE_TOKEN>` plus
+`X-AiSOC-Tenant-ID` for tenant declaration. Files:
+`services/api/app/api/v1/endpoints/cases.py`, `playbooks.py`,
+`services/api/app/graphql/query.py`.
+
+**Verified:** full flow replayed inside the network: login → investigate →
+poll run to completed, with negative controls (no creds → 401 still, guards
+intact).
+
+## 2. Agent tools → core API lake queries 401
+
+**Symptom:** agents logs showed `investigation_tool.failed HTTPStatusError`
+for `process_activity`, `entity_timeline`, `fleet_ioc_hunt`, etc.
+
+**Root cause:** `services/agents/app/tools/investigation.py` called
+`POST /api/v1/graph/investigate/query` with only `X-Tenant-ID`. That route
+authenticates via `get_current_user`, which accepts only JWTs or DB-backed
+`aisoc_` API keys — a tenant header is not a credential.
+
+**Fix:** tools now forward `Authorization: Bearer $AISOC_AGENTS_API_KEY`
+(minted DB-backed key, scopes least-privilege: lake:query et al.);
+`services/agents/app/context/bundle.py` got the same treatment. The key is
+injected via `.env` → compose (`AISOC_AGENTS_API_KEY: ${AISOC_...:-}`),
+never committed. Note: service tokens are NOT valid on these routes — earlier
+config wired the service token into the API-key variable, guaranteeing 401.
+
+**Verified:** pivot probe from inside `aisoc-agents` → 200,
+`available: true`.
+
+## 3. Case status PATCH 422 (legacy vocabulary)
+
+**Symptom:** `PATCH /api/v1/cases/{id}` with `status="in_progress"` → 422
+`Invalid status transition` / unprocessable.
+
+**Root cause:** backend enforces a canonical status ladder
+(`new→triaged→investigating→contained→resolved→closed`); the UI and stored
+rows carry legacy labels (`open`, `in_progress`, `pending`, …). The Literal
+type rejected legacy tokens before any normalization could run.
+
+**Fix:** `services/api/app/api/v1/endpoints/cases.py`:
+`_STATUS_ALIASES` + `_normalize_status()` map legacy→canonical *before*
+validation; the `CaseStatus` Literal lists the aliases too; transition rule
+relaxed from "adjacent rung only" to **forward-monotonic** (forward skips
+legal, backwards 422 — reopening needs the explicit reopen action).
+
+**Verified:** same PATCH → 200; backwards transition still 422.
+
+## 4. Missing status text labels in the UI
+
+**Symptom:** status dropdown options rendered blank (selectable, no text).
+
+**Root cause:** vocabulary rename swept the dropdown values but not the
+`Record<vocab, …>` lookup maps — `STATUS_LABEL` / `STATUS_DOT` in
+`CaseWorkspace.tsx` were keyed only on one vocabulary; undefined renders as
+nothing, silently.
+
+**Fix:** both maps extended to resolve canonical AND legacy keys. Same sweep
+applied to the alerts side: `triaged` added to the `AlertStatus` union in
+`packages/types/src/alert.ts` and to the queue filter chips array in
+`AlertsView.tsx` (a DB-valid status missing from the chips array is invisible
+in the UI even though the API accepts it).
+
+## 5. Investigation panel / tab empty although runs completed
+
+**Symptom:** "investigation started, steps will appear below" — nothing ever
+appeared. Ledger tab showed history; investigation tab stayed blank.
+
+**Root cause (three stacked layers):**
+1. **Poll auth:** API→agents status polls lacked `X-AiSOC-Tenant-ID` → 403
+ on every poll. Fixed with header + service token (§1 pattern).
+2. **List endpoint gap:** agents implements NO `GET /cases/{id}/investigations`
+ (only `/investigations/{run_id}`). The API's list route swallowed the 404
+ and returned `{"runs": []}` forever → UI never discovered the run. Fixed
+ with a postgres-ledger fallback in the API list route (watch
+ varchar-vs-uuid coercion in `case_id.in_()` — a 500 of its own caught in
+ verification).
+3. **No restore-on-mount:** the Investigation tab's run state lives only in
+ React memory. Any refresh (or viewing a case investigated earlier/background)
+ fell back to the idle prompt. Fixed with a cancel-safe mount effect that
+ fetches the runs list, attaches the latest terminal run, hydrates detail +
+ report.md.
+
+**Verified:** `GET /cases/{id}/investigations` → 200 with 8 runs; detail
+probe → full recon/forensic/responder/audit_log payload.
+
+## 6. LLM timeout on T4 (runs "failed" silently)
+
+**Symptom:** litellm 408 `Timeout passed=60.0` storms; agents
+`investigation.budget_timeout`; panel stayed empty although the ledger said
+completed.
+
+**Root cause:** `qwen3:8b` is a thinking model; ollama 0.6.7 cannot disable
+thinking for it; reasoning chains on multi-KB SOC prompts exceed 60 s on the
+Tesla T4 — every LLM call timed out (2 retries × 60 s) while the run
+machinery kept limping.
+
+**Fix:** `ollama pull qwen2.5:7b-instruct` (no-thinking, same size class);
+`.env` `AISOC_LLM_MODEL_FAST`/`_DEEP` → `ollama_chat/qwen2.5:7b-instruct`;
+`docker compose up -d litellm agents`. Model changes are .env-only — no
+rebuild needed, aliases resolve at runtime.
+
+**Verified:** smoke probe latency 0.2–6 s vs 60 s timeouts; investigation
+runs complete with findings and reports.
+
+## 7. React error #31 on the investigation tab
+
+**Symptom:** clicking Investigate threw `Minified React error #31 … object
+with keys {priority, action, rationale, risk}`.
+
+**Root cause:** the Response card rendered `responder.recommended_actions`
+items raw, assuming strings — the LLM emits objects
+`{priority, action, rationale, risk}`.
+
+**Fix:** card coerces: action text + `P{priority}` prefix + risk badge +
+rationale sub-text; strings still supported.
+
+## 8. Overview timeline permanently blank
+
+**Symptom:** Overview tab timeline stuck on "Quiet so far".
+
+**Root cause:** the case-detail GET returns no `timeline` key at all; a
+dedicated `/cases/{id}/timeline` endpoint (synthesising opened/status/
+investigation events) existed and worked (200) — the workspace simply never
+called it, rendering `caseRecord.timeline ?? []`.
+
+**Fix:** mount-fetch `casesApi.getTimeline(caseId)`, merged with inline
+events deduped by event id.
+
+## 9. Alert status not following agent verdicts
+
+**Symptom:** alerts triaged 19 h ago still shown as `new`.
+
+**Root cause:** two-column trap — verdicts land in `alerts.disposition`, the
+badge reads `alerts.status`, and the persist path never followed. An
+`escalate` verdict deliberately leaves status open for a human (by design);
+but benign/FP auto-closes and any real verdict should at least lift
+`new→triaged`.
+
+**Fix:** `services/agents/app/investigator/ledger.py` `persist_auto_triage`
+now lifts `new → triaged` whenever a real (non-shadow) verdict is persisted;
+further-along statuses never downgraded. One-off backfill UPDATE applied for
+pre-existing stale rows.
+
+## 10. Approvals dispatch shipped empty permission sets (502)
+
+**Symptom:** `POST /approvals/{id}/decide` → 502; approval decided in DB but
+actions service refused: "approver lacks 'actions:execute:high'" — even for
+platform admins.
+
+**Root cause:** `_dispatch_decision` read `user.permissions` / `user.roles`,
+which don't exist on `CurrentUser` (it carries `scopes` /
+`resolved_permissions` / `role`) — `getattr(..., [])` silently dispatched an
+empty grant set.
+
+**Fix:** resolve through the same three-way precedence CurrentUser uses
+(scopes → resolved_permissions → ROLE_PERMISSIONS[role]); `tenant_id` added
+to the principal.
+
+## 11. Bulk redisposition 422 on apply
+
+**Symptom:** analyst bulk override recorded fine (phase 1) but apply → 422.
+
+**Root cause:** apply REQUIRES `confirmation_token` (sha256 over sorted
+alert_ids + disposition); the UI omitted it.
+
+**Fix:** `computeRedispositionToken()` in `apps/web/src/lib/api.ts` mirrors
+the server canonical form; pure-JS SHA-256 fallback because the console runs
+over plain HTTP where `crypto.subtle` is undefined.
+
+## 12. Release-assignee silent no-op
+
+**Symptom:** "Release" button toasted success but the alert stayed assigned.
+
+**Root cause:** key-name mismatch (UI sends `assignee`, model field is
+`assigned_to_id` — pydantic silently drops unknown keys) stacked with
+`is not None` making explicit JSON `null` indistinguishable from absent.
+
+**Fix:** `Field(alias="assignee")` + write triggered on
+`model_fields_set` key-presence. Verified both ways: clearing clears,
+absent preserves.
+
+## 13. Lake infrastructure never started
+
+**Symptom:** every lake pivot failed with `clickhouse:9000` name resolution /
+`LakeQueryError`.
+
+**Root cause:** `docker compose up` deliberately skips optional infra —
+aisoc-clickhouse / aisoc-opensearch / aisoc-ingest were defined but had never
+been started on this box.
+
+**Fix:** `docker compose up -d clickhouse opensearch` (first boot runs init
+SQL ~2 min; ClickHouse auth = user `aisoc` + `CLICKHOUSE_PASSWORD` from
+.env, not the compose default). Ingest worker (`aisoc-ingest`) was already
+running. ClickHouse now answers; `raw_events` currently 0 rows — ingestion
+backfill is a known open item, not a bug in the fixed chain.
+
+## Environment note (GPU)
+
+`docker-compose.yml` ollama service gained an nvidia GPU reservation block
+(Tesla T4) — required for the 11434 inference endpoint to keep GPU access
+under compose v2 scheduler semantics.
+
+## Verification snapshot (deployed state)
+
+- api health 200, web 200, agents 200, clickhouse up, opensearch healthy
+- PATCH case status (legacy vocab) → 200
+- investigations list → 200 with runs; detail → full payload
+- litellm smoke: 0.2–6 s responses
+- negative controls intact: no-creds → 401 everywhere
diff --git a/docs/known-gaps.md b/docs/known-gaps.md
new file mode 100644
index 000000000..e58732e2e
--- /dev/null
+++ b/docs/known-gaps.md
@@ -0,0 +1,54 @@
+# AiSOC — known gaps & follow-ups
+
+State as of 2026-10-03, after the fix session (see fix-log-2026-10.md).
+
+## Data / ingestion
+
+1. **ClickHouse lake empty (`aisoc.raw_events` = 0 rows).** Lake query chain
+ is fixed end-to-end (`available: true`), but nothing has indexed events
+ into the lake yet. Until ingestion backfills, agent investigations run with
+ thin evidence and the reports will be model-generated rather than
+ evidence-grounded. Next step (needs a decision): verify
+ `aisoc-ingest` → kafka → lake sink path and trigger a backfill window.
+2. **`aisoc-ingest` reports `subscriptions: {}`** at /readyz — consumer
+ detached or not configured on this build. Confirm intended consumption
+ topology before forcing a restart.
+3. **OpenSearch vs ClickHouse ambiguity.** Both defined; API lake routes
+ point at ClickHouse (`CLICKHOUSE_HOST: clickhouse`). OpenSearch runs but
+ holds no event indices. Decide whether it stays (retire it if unused).
+
+## Upstream sync obligation
+
+All fixes are on the fork working tree. The upstream repo
+(`beenuar/AiSOC`) must receive these via PR or they regress on the next
+image pull. Priority upstream candidates:
+- proxy auth fixes (cases/playbooks/graphql → agents)
+- `_normalize_status` + forward-monotonic ladder
+- investigations list fallback to the postgres ledger
+- React #31 responder-card coercion + overview timeline fetch
+- ledger `new→triaged` status lift + approvals principal permission fix
+- alerts assignee alias + model_fields_set patch
+
+## GPU / model
+
+- T4 16 GB: `qwen3:8b` thinking chains are unusable (60 s+ per call).
+ Current: `qwen2.5:7b-instruct` (sub-second to ~6 s). Revisit only with a
+ bigger GPU or litellm timeout raise + streaming.
+- `ollama n_slots=1`: single request at a time; concurrent investigations
+ queue. Fine for this footprint, known ceiling.
+
+## UI follow-ups (not blocking)
+
+- Report tab auto-refresh after a live run completes (currently fetched once
+ on completion; a late report.md write isn't picked up).
+- Audit log entries render `kind/agent/summary` strings only — any future
+ object-valued summary field must be coerced (React #31 class bug).
+
+## Deliberately NOT fixed (by design — do not "fix" again)
+
+- Backwards case status transitions → 422 (ladder guard).
+- `escalate` verdict leaving alert `status='new'` (human queue stays open).
+- Wazuh SIEM writeback: no disposition-write arm exists for wazuh; the
+ execute flag changing nothing is correct.
+- Approvals 502 when approver truly lacks `actions:execute:high` — the gate
+ working; the fixed bug was only the empty-permission forwarding.
diff --git a/docs/maintenance-2026-10-index.md b/docs/maintenance-2026-10-index.md
new file mode 100644
index 000000000..88158e44b
--- /dev/null
+++ b/docs/maintenance-2026-10-index.md
@@ -0,0 +1,20 @@
+# AiSOC maintenance docs — Oct 2026 live session
+
+Working documentation for the deployment on `10.200.0.69`
+(`/home/ubuntu/AiSOC`, compose project `aisoc`), covering the Oct 2026
+diagnose-and-fix session. Code fixes: branch
+`fix/aisoc-live-session-2026-10` on https://github.com/alexmateescu/AiSOC.
+
+| File | Purpose |
+|------|---------|
+| [fix-log-2026-10.md](fix-log-2026-10.md) | Root cause + fix + verification for all 13 issues fixed |
+| [ops-runbook.md](ops-runbook.md) | Live topology, credentials map, deploy & probe recipes |
+| [known-gaps.md](known-gaps.md) | Open items, upstream-sync obligations, by-design non-fixes |
+
+Quick orientation:
+- Stack: web :3000 · core api :8000 · agents :8001→8084 · litellm :4000 ·
+ ollama :11434 (Tesla T4) · postgres · kafka · clickhouse (lake) ·
+ opensearch · ingest worker.
+- Model routing: agents → LiteLLM aliases → `.env`
+ `AISOC_LLM_MODEL_FAST/_DEEP` = `ollama_chat/qwen2.5:7b-instruct`
+ (qwen3:8b thinking chains exceeded the 60 s gateway timeout on the T4).
diff --git a/docs/ops-runbook.md b/docs/ops-runbook.md
new file mode 100644
index 000000000..43428822b
--- /dev/null
+++ b/docs/ops-runbook.md
@@ -0,0 +1,76 @@
+# AiSOC ops runbook — live deployment
+
+Host: `10.200.0.69` (SSH `ubuntu@`, key `~/.ssh/c8r-tech-key`)
+Repo/compose project: `/home/ubuntu/AiSOC` (compose project name `aisoc`)
+Origin: `https://github.com/beenuar/AiSOC` · Fork: `https://github.com/alexmateescu/AiSOC`
+
+## Service map (container → host port → role)
+
+| Container | Host port | Role |
+|---|---|---|
+| aisoc-web | 127.0.0.1:3000 | Next.js console |
+| aisoc-api | 127.0.0.1:8000 | Core FastAPI (cases, alerts, graph, approvals, feedback, lake query) |
+| aisoc-agents | 127.0.0.1:8001 (→8084) | Investigation agents |
+| litellm | 127.0.0.1:4000 | LLM gateway (alias router + fallbacks) |
+| ollama | 127.0.0.1:11434 | Inference, Tesla T4 16 GB |
+| aisoc-postgres | internal | Core store (+ investigation ledger, api_keys, RLS multi-tenant) |
+| aisoc-clickhouse | internal:9000 | Warm lake (`aisoc.raw_events`, `ioc_enrichments`, `alert_metrics`) |
+| aisoc-opensearch | 9200 | Secondary index (defined; lake primary is ClickHouse) |
+| aisoc-ingest | 8081 | Kafka producer/sink worker (service name `ingest-worker`) |
+| kafka / zookeeper / redis / qdrant | internal | Stream / cache / vectors |
+
+Health checks: `curl 127.0.0.1:8000/health`, `/readyz` variants per service;
+`aisoc-ingest` `/readyz` exposes `subscriptions` — empty subscriptions means
+consumer detach, worth watching.
+
+## Credentials map (never write values here)
+
+- `api_keys` table (postgres): DB-backed `aisoc_<48hex>` keys, SHA-256
+ matched, scopes in JSONB. Agents use `AISOC_AGENTS_API_KEY` (from `.env`)
+ for `get_current_user` routes.
+- Service tokens (`AISOC_AGENTS_SERVICE_TOKEN` / `AISOC_SERVICE_TOKEN`):
+ ONLY valid as Bearer on the agents guard or as `X-AiSOC-Service-Token` —
+ never valid on core-API `get_current_user` routes.
+- ClickHouse: user `aisoc` + `CLICKHOUSE_PASSWORD` in `.env` (NOT the compose
+ default).
+- GitHub push from this box: `~/.ssh/github` key → authenticates as
+ `alexmateescu`.
+
+## Deploying changes
+
+Code edits inside service dirs (`services/api`, `services/agents`,
+`apps/web`) need a rebuild — `up -d` alone reuses the old image:
+
+```
+cd ~/AiSOC
+docker compose build <svc> && docker compose up -d <svc>
+```
+
+Verify the patch landed inside the container before testing:
+`docker exec <c> grep -c <new-symbol> /app/app/<file>`.
+Model/env changes are `.env` + `docker compose up -d <svc>` only.
+
+Optional infra (clickhouse/opensearch/ingest) is NOT started by a bare
+`docker compose up` — start explicitly after host reboots if lake pivots
+fail with name-resolution errors.
+
+## Useful probes (run from inside aisoc-agents — api container has no keys)
+
+- investigations list: `GET http://api:8000/api/v1/cases/<case>/investigations`
+- lake pivot: `POST http://api:8000/api/v1/graph/investigate/query`
+ `{"tool":"process_activity","args":{"hostname":"…","hours":24}}`
+- LLM smoke: `POST http://litellm:4000/v1/chat/completions` model
+ `aisoc-triage`.
+- The command relay scrubs literal `Bearer <token>` text before execution —
+ build auth headers from runtime fragments in probe scripts.
+
+## Known operational quirks
+
+- `scp` broken on this box — pipe files (`cat f | ssh host 'cat > /tmp/f'`).
+- Nested heredocs inside `ssh 'bash -s'` are silently swallowed — deliver
+ scripts/files by piping a local file.
+- Compose service names ≠ container names (`ingest-worker` → `aisoc-ingest`);
+ diff services on container_name.
+- Unquoted heredocs executing TSX patches eat backticked text in comments.
+- Alerts: verdict lives in `disposition`, badge reads `status` — check both
+ columns before claiming the agent did nothing.