Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
Real screenshots, an architecture doc that follows one event, and the first run that said it was broken (#882)
Beenu Arorab2b6c87cbb08
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 098001d6d..47ad8bbc9 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -1077,6 +1077,11 @@ jobs:# * `prometheus-client` is required by `threatintel.actors.# attribution` at module-import time; without it the test# collector aborts before running a single test.+ # * `qdrant-client` is required by `threatintel.api.indicators`+ # for the same reason. The base package is enough — the+ # `fastembed` extra the service pins is imported lazily inside+ # `QdrantStore._embed`, and pulling it here would add onnxruntime+ # to every cell in the matrix.# * `apscheduler` is required by `connectors.scheduler` at# module-import time; same import-aborted-the-collector issue.# * `slack-bolt` pulls `aiohttp` transitively — both must be@@ -1137,6 +1142,7 @@ jobs:'apscheduler>=3.10,<4' \'slack-bolt>=1.18,<2' \'prometheus-client>=0.20,<0.24' \+ 'qdrant-client>=1.9,<2' \'opentelemetry-instrumentation-fastapi>=0.45b0' \'sqlalchemy[asyncio]>=2,<3' \'aiosqlite>=0.19,<0.22' \diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f6aff31cd..e6b5cadbb 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -121,6 +121,112 @@ path did not work at all.the ten read tools. Corrected, along with the competitor-framed "moat" line.+### `make up` reported the stack broken on every machine, and four published numbers were wrong++Found by bringing the stack up from the documented path and photographing the+result. Each of these was visible within minutes of a real first run and+invisible to the test suite.++- **`make up` could not succeed, ever.** Its `_wait` loop scored any container+ in state `exited` as broken, and `ollama-pull` is a one-shot: it fetches the+ model, exits 0, and `litellm` waits on `service_completed_successfully`, so+ by the time `docker compose up -d` returns the puller has *always* exited.+ The documented command therefore ended in `These services are not running:+ ollama-pull` against a completely healthy stack — the step whose job is to+ say whether the stack is healthy, reporting the opposite. Fixed by reading+ `{{.ExitCode}}` and treating `exited 0` as a one-shot that did its job.+ The first attempt at that fix did not work either, for a reason worth+ recording: a service with no healthcheck prints an *empty* `Health`, and+ awk's default whitespace splitting collapses the run, so `ollama-pull+ exited 0` parsed as three fields with the exit code in `$3` — the new test+ read an empty `$4` on exactly the row it was written for. The fields are+ `|`-separated now.++- **Four of the ten shipped ingest profiles rendered their vendor twice.**+ `product_label()` deduplicated the vendor/product join on exact equality, so+ `crowdstrike` + `crowdstrike` collapsed but `Okta` + `Okta System Log` did+ not. The alert queue, the Investigation Rail and every entity chip read+ "Okta Okta System Log", "Splunk Splunk Enterprise", "Kubernetes Kubernetes+ Audit" and "Email Forwarded Email". A part another part already contains is+ dropped now, on word boundaries so "AWS" is not swallowed by a product+ called "Lawsuit Monitor", and the new test reads the pairs *out of the Go+ normalizer* rather than restating them — the previous tests passed every+ hand-written case while four shipped profiles were wrong, because none of+ them asked the normalizer what it actually declares.++- **The Threat Intelligence page published its own page length as the size of+ the catalogue.** With 1,725 CISA KEV entries collected, Qdrant held 1,725,+ the API answered `total: 400` (the size of its bounded scan window) and the+ console's headline card read **100** (its page). Three answers to one+ question, the two published ones both wrong, and the page presenting the+ smallest as the corpus. `total` is now the store's own count, `shown` is the+ page, and a narrowed query is marked `bounded` so a caller knows the match+ was made inside a window.++- **`Added Today` on that same page was the literal `3`, hard-coded**, on+ every deployment including ones that had never ingested an indicator. The+ comment four lines above it describes removing an invented IOC list for+ exactly this reason and names this counter while doing so; the counter+ survived. Replaced with figures the page can support — the store's count,+ the page's count, and two counts explicitly labelled *of shown*. The+ sample-data gate did not catch it and could not: it scans module-scope+ fabricated-record literals, and this was a scalar inside JSX.++- **Nothing in the console rendered the AI verdict.** `services/agents`+ triages every fused alert and writes `ai_score`, `ai_summary`,+ `triage_groundedness` and `triage_ungrounded` back onto the row; the API+ returns all four; `normalizeAlert` dropped all four on the floor. The+ product's headline capability was producing tokens, cost and a ledger entry+ that no surface showed. The Investigation Rail's Details view now carries an+ **Automated triage** section rendering the verdict, the confidence, the+ rationale *verbatim* — so a reader can tell the model's own text from the+ deterministic fallback — and groundedness as **not assessed** when the+ column is null, never as 0.++### Real screenshots, and an architecture doc that follows one event++- **Every committed screenshot is now a capture of a running stack.** The+ 24 MB `screenshots/` tree was a marketing redesign from an older product and+ no longer resembled it; the four tiles under+ `apps/web/public/screenshots/` were SVG placeholders. Both are replaced by+ 16 PNGs (844 KB total) taken from a CORE stack brought up with `make up`,+ fed through the documented ingest API, holding 1,725 real CISA KEV entries+ and a real local-model verdict. `apps/web/public/screenshots/README.md`+ states what is real in each one and what was authored — the security events+ were written by hand; everything downstream of them is the product working.+ **None was taken in demo mode.**++- **`apps/docs/docs/architecture.md` is organised around what happens when+ AiSOC receives one security event** — eleven numbered steps from the two+ ingest doors through normalization, the spine, detection, enrichment,+ promotion, correlation, alerting, AI triage, the ledger, approval and+ verified response — instead of the component inventory it was. Every box in+ every diagram links to the directory that implements it, and all 31 source+ links were checked to resolve. It also records what the previous version got+ wrong: ports (the page listed host ports as though services addressed each+ other on them), `50 vendors` against 84, `200+` detection rules against 833+ executable, and five packages still promised "in v8.0" two releases later.++- **Measured figures, re-read rather than carried forward.** CORE is 14+ long-running services plus a one-shot model pull; `full` is 22. The playbook+ palette offers **21** of the engine's 22 step types — `approval` is withheld+ because the engine is a single-threaded index walk with no pause or resume.+ Row-level security is live on 83 tables under 84 policies, and the runtime+ role holds `SELECT / INSERT / UPDATE / DELETE` and no DDL.++- **The local model's triage output is now described honestly.** In a measured+ run of 19 auto-triages on a CORE stack the bundled+ `llama3.2:3b-instruct-q4_K_M` returned schema-valid JSON 7 times; the other+ 12 fell back to the deterministic path and logged that they had. The README+ and the architecture page both publish that ratio rather than implying the+ model answers every time. No hosted provider has been exercised — there is+ still no funded key, and that remains a different claim.++- **`POST /v1/ingest/batch` is documented as unauthenticated**, because on+ this commit it is: only the inbox webhook paths and the Kubernetes audit+ webhook carry a token. A comment in `server.go` asserts the opposite and is+ flagged in the docs rather than quietly relied upon.+### The documented quick start broke the product, and then the product had nothing to showEvery item below passed the existing test suite and failed on a real@@ -303,29 +409,6 @@ number is the one the requirement is sized against.### Added-- `make ingest-token` / `python -m app.scripts.mint_ingest_token` mints the- push credential for a fresh deployment, the role `bootstrap_admin` plays- for the console login. Idempotent — a second run returns the same token- rather than leaving a trail of live credentials nobody revokes.-- `connector-push`, `ai-runtime`, `ai-finding` and `k8s-audit` are now- mintable inbox templates. The last three already shipped as YAML and were- named in the AI SDK's own setup instructions and in the Kubernetes- connector docs, but none was in `ALLOWED_TEMPLATE_IDS`, so the documented- setup for the AI-estate feature returned 400 and the templates were- unreachable.-- `scripts/check_inbox_templates.py` compares shipped templates against- mintable ids **in both directions**, plus the console catalog, and carries- a `--self-test` that injects drift in each direction and requires the gate- to catch it. A one-directional check would have printed OK throughout the- defect above.-- The golden pipeline asserts that an unauthenticated push is refused, not- only that an authenticated one succeeds — the happy path would go on- passing if authentication were removed.-- `/v1/ingest` now bounds the request body (`INGEST_MAX_BODY_BYTES`, 10 MiB- default). `MAX_BATCH_SIZE` only counts events after the payload is- decoded, so nothing bounded the decode itself.--- **`RENDER_FALLBACK_EXEMPT` is now checked in both directions.**`ALLOWED_ILLUSTRATIVE` had a staleness check from the start and this listhad none, so an exemption could outlive the code it excused and go on@@ -369,29 +452,6 @@ number is the one the requirement is sized against.to published prose has no hand-copied link. A paragraph that dates ornegates the claim is exempt, so the retraction can quote the wording itretracts.--### Changed--- **gitleaks and Trivy are blocking gates.** All five scanners in- `security.yml` carried `continue-on-error`, so a new finding never redded- a pull request. Gitleaks found 35 findings on its first real run; every- one was triaged and recorded in `.gitleaksignore` with a written reason —- detection rule-id maps, osquery and redaction test fixtures, placeholder- PEMs, a published AWS example key, a FIPS-197 test vector, and a- `YOUR_TOKEN` doc placeholder. Four of the six entries the allow-list- already held had rotted through line drift and were suppressing nothing.- Trivy reports zero fixable HIGH/CRITICAL and blocks at that.-- Semgrep (92 findings, 35 at ERROR), checkov (91 failed checks) and tfsec- (32, of which 7 CRITICAL) stay in observe mode. The counts are now written- into the workflow so the remaining work is visible rather than implied.-- The secret-scan job also fails when the scanner cannot install, instead of- reporting clean, and proves it is reading the tree before trusting a clean- result: with the allow-list moved aside the triaged findings must- reappear. "Clean" and "did not run" previously looked identical.--- **The prerequisites that were required but undocumented.** Beyond Docker andits memory, a first run needs `python3` **on the host** (`make smoke` runsthe golden-pipeline script there, so without it you can start AiSOC but@@ -506,30 +566,7 @@ number is the one the requirement is sized against.also caught `<ul role="alert">`, which is not an allowed role for a list andleft its items without a list parent.-- **Verification probes for the disruptive endpoint verbs.** `kill_process`- and `quarantine_file` had their success inferred from CrowdStrike RTR- accepting a command; both now read the effect back over the read-only tier- of the same batch API — the process table for the PID, the path for the- removed file. Absence is only trusted when the output can be recognised as- what it claims to be, so an unparseable `ps` listing is indeterminate- rather than "the process is gone", and only a line's first column is read- as a PID so a surviving child does not report its dead parent as alive.- `run_av_scan` joins them because it is automatic and Defender replies- `Pending` the moment it queues the sweep: it reads the machine action's- terminal state, and says indeterminate while the scan is still running.--- **Every verb with an `ActionType` now either declares a probe or records- why it has none.** Eleven bridged response verbs had neither, which failed- safe but made three different situations look the same: nobody wrote it,- the vendor exposes no read-back, and there is nothing to read back. The- eight without probes now state which — including `reset_password`, where- the obvious candidate read is the *wrong* one rather than a missing one- (the executor issues a recovery link, so `passwordChanged` moves later and- only if the user follows it, and a probe watching it would report FAILED- for every correct reset). The below-MODERATE waiver must now name itself,- because `run_av_scan` had been taking it silently.--+### Changed- **`readme_gates.py` covers the governance documents.** Its `FIGURE_DOCS`list named one compliance page, which is why `ROADMAP.md` drifted with CI@@ -562,14 +599,6 @@ number is the one the requirement is sized against.freshness gate reads only the date — which a refresh keeps current — so arow could be two days old and still labelled two majors behind.-- `tests/test_approval_doors_agree.py` gates the invariant rather than the- instance: both live doors swept over every capability, autonomy tier and- confidence band, asserting the same answer to "did a vendor get touched- without a human". Structural checks alongside it fail a door that imports- the raw matrix or branches on a specific impact tier, which is the shape a- re-introduced bypass has. Against the previous tree it reports five- disagreements across 345 combinations.-### Fixed- **`check_mock_data_gated.py` reported "All sample-data fallbacks are gated@@ -1053,52 +1082,8 @@ number is the one the requirement is sized against.hosted demo the dock now reports that the copilot could not be reached, withthe error, instead of emitting a reply.-- **A pure read sat in the analyst approval queue, because the two dispatch- doors graded the same verb differently.** `search_siem` declares- `read_only` impact and `automatic` approval, and the contract gate's own- rule is that a read requiring approval "is either mis-classified or is not- actually a read". At the default autonomy tier `POST /actions` returned- `awaiting_approval` for it while `POST /live-actions/dispatch` executed it.- The dispatcher's docstring says its contract block exists so that a verb is- not "graded differently depending on which door it came through", and that- is precisely what was happening.-- The cause was one table entry, not a missing bypass.- `TIER_MAX_AUTOMATIC["L1"]` was `None`, meaning "auto-executes nothing",- while `maturity.py` defines L1 as "MINIMAL blast-radius actions are- automatic" and `_AUTO_ALLOWED_AT_TIER` gives it `{MINIMAL}` — which- `_IMPACT_BLAST` equates with READ_ONLY impact. The registry door had grown- a local READ_ONLY short-circuit to route around it; the legacy door had- not. L1 now reads `READ_ONLY`, and only reads move: every impact above it- still outranks the ceiling and still returns ANALYST, exactly as the `None`- branch did. L0 keeps `None`, which now means one thing only — observe, not- even a read.-- With the ceiling correct the bypass is unnecessary, so it is gone.- `approval_matrix.evaluate_contract()` takes a contract whole and is the- single entry both doors call, which removes the per-door unpacking that let- them drift in the first place.--- **`update_alert_disposition` had no `ACTION_BLAST_RADIUS` entry**, found by- the new cross-door sweep rather than reported. Four readers of that table- supplied two different fallbacks — `blast_radius.py` to MEDIUM, which is- exactly `_AUTO_EXECUTE_LIMIT`, and the three tier gates to HIGH — so the- legacy door auto-executed the disposition writeback while the registry door- held it for a whitelist it could never match. It now carries the LOW entry- its own contract asks for ("classified the same as create_notable_event"),- and `blast_radius.py` fails closed like its three siblings.-### Removed-- **The empty blog section on the documentation site.** The classic preset- enabled the blog plugin while no `apps/docs/blog` directory has ever- existed, so the published site served an empty index at `/AiSOC/blog` at- HTTP 200, listed it in the sitemap, and advertised it from both the navbar- and the footer. Dropping the footer entry left the "More" column holding- only a duplicate of the GitHub link already in the navbar, so that link- moved to "Community" and the column went with it. The three long-form posts- under `apps/web/content/blog` are a different surface and are unaffected.-- **`apps/web/src/components/copilot/InvestigationChat.tsx`** — cannedthreat-intel replies ("VirusTotal: 14/87 engines flagged malicious","Associated campaigns: APT-42"), a fixed context sidebar, no API call for@@ -1152,7 +1137,6 @@ number is the one the requirement is sized against.from the registry now. `packHelpers.ts` mapped step types to integrationbadges as a `Record<string, …>` covering eighteen of twenty-two, so aplaybook built from the other four claimed to use no integrations at all.-## [10.0.0] — 2026-09-25**Two ways for a control to be absent: not written, or written and notdiff --git a/Makefile b/Makefileindex 21254d093..1f59ba94e 100644--- a/Makefile+++ b/Makefile@@ -147,11 +147,28 @@ _ports:# "Full profile up." with OpenSearch OOM-killed and threatintel in a crash# loop. A container that is not running is the one thing a wait loop must# never score as success.+#+# `exited` alone is not that thing, though, and reading it as such broke the+# documented command on every machine. `ollama-pull` is a one-shot: it fetches+# the model and exits 0, and `litellm` waits on+# `service_completed_successfully`, so by the time `docker compose up -d`+# returns the puller has *always* exited. `make up` therefore ended in+# "These services are not running: ollama-pull" on the first run and every+# run after it — a stack that was entirely healthy, reported as broken, by+# the step whose job is to say whether it is healthy. The exit code is what+# separates the two cases: 0 is a one-shot that did its job, anything else is+# a container that died.+#+# The fields are `|`-separated rather than space-separated because a service+# with no healthcheck prints an *empty* Health, and awk's default splitting+# collapses the run of whitespace — so `ollama-pull exited 0` parsed as three+# fields with the exit code in $$3 and nothing in $$4, and the new exit-code+# test read an empty string on exactly the row it was written for._wait:@for i in $$(seq 1 60); do \- status=$$($(COMPOSE) $(PROFILE_ARG) ps -a --format '{{.Service}} {{.State}} {{.Health}}' 2>/dev/null); \- pending=$$(echo "$$status" | awk '$$3=="starting"{print $$1}'); \- broken=$$(echo "$$status" | awk '$$2=="exited"||$$2=="dead"||$$2=="restarting"||$$3=="unhealthy"{print $$1}'); \+ status=$$($(COMPOSE) $(PROFILE_ARG) ps -a --format '{{.Service}}|{{.State}}|{{.Health}}|{{.ExitCode}}' 2>/dev/null); \+ pending=$$(echo "$$status" | awk -F'|' '$$3=="starting"{print $$1}'); \+ broken=$$(echo "$$status" | awk -F'|' '($$2=="exited"&&$$4!="0")||$$2=="dead"||$$2=="restarting"||$$3=="unhealthy"{print $$1}'); \if [ -n "$$broken" ]; then \echo ""; \echo "These services are not running: $$broken"; \diff --git a/README.md b/README.mdindex 463fc425d..c6e0f1dd2 100644--- a/README.md+++ b/README.md@@ -20,20 +20,25 @@## What AiSOC does-```- Your security tools (EDR, cloud, identity, network, SIEM)- |- ┌─────────v────────────────────────────────────────────────┐- │ normalize -> detect -> correlate -> investigate -> respond │- └─────────|────────────────────────────────────────────────┘- v- SOC analyst: one incident, with the evidence and the reasoning-```--An alert arrives. AiSOC works out whether it matters, groups it with related-signals, investigates it with an AI agent whose every prompt and tool call is+Telemetry arrives from your security tools. AiSOC normalizes it, runs 833+executable detection rules over it, groups what fires into incidents,+investigates each one with an AI agent whose every prompt and tool call isrecorded, and proposes an action. A human approves before anything executes.+## What it looks like running++Real captures from a stack brought up with `make up` and fed through the+ingest API below — no seeded rows, no demo mode, no mockups. The events were+authored to be representative; everything downstream of them is the product+doing its job. ([what is real in each shot](apps/web/public/screenshots/README.md))++| | |+|---|---|+| <img src="apps/web/public/screenshots/alerts-queue.png" alt="Alerts queue" /> | <img src="apps/web/public/screenshots/ai-triage-verdict.png" alt="AI triage verdict in the Investigation Rail" /> |+| **Alerts** — each attributed to the connector that fed it. | **Automated triage** — the bundled local model's verdict, confidence and rationale, verbatim. |+| <img src="apps/web/public/screenshots/threat-intel-kev.png" alt="Threat intelligence page showing CISA KEV entries" /> | <img src="apps/web/public/screenshots/soc-operations.png" alt="SOC operations dashboard with honest empty states" /> |+| **Threat intelligence** — 1,725 real CISA KEV entries, minutes after boot, with no API key. | **SOC operations** — with nothing connected yet, and it says so rather than showing a placeholder. |+## Quick start```bash@@ -69,19 +74,13 @@ $ make smokeOpen **http://localhost:3000** and sign in with the credentials `make up`printed (API docs at **http://localhost:8000/api/docs**). Deploying somewherethat is not your laptop? Set `AISOC_CONSOLE_URL` in `.env` so the printed-address is the one people browse to.--Something wrong? `make doctor` checks every dependency and says what to run next.+address is the one people browse to. Something wrong? `make doctor`.## Try it without connecting anything-```bash-make demo-```--> **The demo dataset is synthetic.** It shows the pipeline shape, not real-> activity. Every row is marked `is_synthetic = true` in the database and-> labelled in the console. It is not a benchmark, a customer, or an incident.+`make demo` loads a dataset. **It is synthetic**: it shows the pipeline shape,+not real activity. Every row is marked `is_synthetic = true` in the database+and labelled in the console. It is not a benchmark, a customer, or an incident.## Connect real data@@ -101,22 +100,22 @@ vendor-specific normalization and live setup docs include Splunk, MicrosoftSentinel, Elastic, CrowdStrike, Okta, AWS (GuardDuty / CloudTrail / SecurityHub), Wiz, and Kubernetes audit logs — full list in the[connector docs](https://beenuar.github.io/AiSOC/docs/connectors/api-coverage).-A connector without a vendor profile still ingests through a generic mapping-that resolves host, user and source IP from the usual spellings, so the alert-is pivotable either way.+Without a vendor profile a connector still ingests through a generic mapping+that resolves host, user and source IP from the usual spellings.## How it works-See **[docs/architecture/README.md](docs/architecture/README.md)** — it walks-one event through the whole system and every box in its diagrams links to the-code that implements it.--The short version: ingest normalizes to a common shape → Kafka carries it →-fusion runs 833 executable detection rules and decides what becomes an alert →-correlation groups related alerts into one incident → an agent investigates-and writes its reasoning to the Investigation Ledger → a human approves any+Ingest normalizes to a common shape and Kafka carries it. Then+fusion runs 833 executable detection rules and decides what becomes an alert,+correlation groups related alerts into one incident, an agent investigates and+writes its reasoning to the Investigation Ledger, and a human approves anyresponse.+Both **[docs/architecture/README.md](docs/architecture/README.md)** and the+[docs portal](https://beenuar.github.io/AiSOC/docs/architecture) walk that path+one step at a time, and every box in every diagram links to the code that+implements it.+## Deployment profiles| Profile | Command | Services | RAM | What you get |@@ -131,12 +130,13 @@ alert, and **it needs no credentials to do either** — for two reasons.**The model ships with the gateway.** Ollama runs a pinned ~2 GB`llama3.2:3b-instruct-q4_K_M` sized for CPU-only inference, so `make up`produces real triage verdicts with real token counts in the Investigation-Ledger — not a stub. A 3B quantized model is not a frontier model; to upgrade,-set `OPENAI_API_KEY`, `AISOC_LLM_MODEL_FAST`, `AISOC_LLM_MODEL_DEEP` and an-empty `AISOC_LLM_API_BASE` in `.env`, then `make up` again. **No hosted-provider has ever been exercised here** — there is no funded key, so per-model-rows read *not measured* rather than zero, and an unmeasured cost renders as-absent, never `$0.00`. ([ADR-0006](docs/decisions/0006-llm-gateway-in-core.md))+Ledger — not a stub. It is also not a frontier model, and the difference shows:+in a measured run of 19 auto-triages it returned schema-valid output 7 times,+and the other 12 fell back to the deterministic path, which the rail labels.+To upgrade, set `OPENAI_API_KEY`, `AISOC_LLM_MODEL_FAST`, `AISOC_LLM_MODEL_DEEP`+and an empty `AISOC_LLM_API_BASE`. **No hosted provider has ever been exercised+here** — there is no funded key, so per-model rows read *not measured* rather+than zero. ([ADR-0006](docs/decisions/0006-llm-gateway-in-core.md))**One real external feed ships too.** `services/threatintel` polls the[CISA Known Exploited Vulnerabilities](https://www.cisa.gov/known-exploited-vulnerabilities-catalog)@@ -157,9 +157,10 @@ This matters more than any feature, so it is stated plainly.| **Test fixtures** | `tests/`, `**/tests/` | Never shipped in an image |**Production never silently falls back to synthetic data.** When a backend is-unreachable the console shows an error, not an invented investigation. That-was not always true — see [the reality audit](docs/audit/REPOSITORY_REALITY.md)-for the five places it was wrong and how each was fixed.+unreachable the console names the failure, not an invented investigation — and+an unmeasured figure reads *not measured*, never `0`. That was not always true;+see [the reality audit](docs/audit/REPOSITORY_REALITY.md) for where it was+wrong and how each case was fixed.## AI agents@@ -175,13 +176,12 @@ Agents triage alerts and investigate incidents. What they can and cannot do:contained is demoted to human review rather than auto-closed.- **A prompt is validated before it is sent.** Raw logs, OCSF payloads andsecret-shaped values are refused, not redacted after the fact.-- **Nothing executes without a human.** Response actions are proposed. An- approver must hold the required permission tier and must not be the person- who requested the action.+- **Nothing executes without a human.** An approver must hold the required+ permission tier and must not be the person who requested the action.-CORE's bundled local model means agents reason for real out of the box. If no-model is reachable at all they run a deterministic offline path and say so —-they never fabricate a verdict.+The bundled model means agents reason for real out of the box. When it returns+something the schema rejects, triage falls back to a deterministic path and the+rail shows which one answered — it never fabricates a verdict.## Project maturity@@ -203,8 +203,7 @@ they never fabricate a verdict.- **Not a drop-in SIEM replacement.** It correlates and investigates; it doesnot replace long-term log retention and compliance search.-- **Not able to see telemetry you have not connected.** There is no magic- discovery.+- **Not able to see telemetry you have not connected.** There is no discovery.- **Not autonomous by default.** Response requires explicit policyauthorization and a human approver.- **Demo incidents are not real incidents**, and benchmark corpora are not@@ -214,21 +213,20 @@ they never fabricate a verdict.## Troubleshooting-`make doctor` diagnoses the deployment and prints the command to run next. It-checks the host tools, free memory and disk in the Docker VM, every port, each-datastore by querying it rather than by asking whether its container is up, and-whether `.env` still holds template placeholders. The six failures it is most-often right about — and what each one actually means — are tabulated under+`make doctor` checks the host tools, memory and disk in the Docker VM, every+port, each datastore by querying it rather than by asking whether its container+is up, and whether `.env` still holds placeholders — then prints the command to+run next. The six failures it is most often right about are tabulated under[Installation → Troubleshooting](https://beenuar.github.io/AiSOC/docs/installation#the-six-most-common-failures).## Security-Secrets live in `.env`, are generated per deployment, and are never committed;-connector credentials are encrypted at rest with a per-deployment key. Tenant-isolation is enforced at the query layer in every store, not by convention.-RBAC gates every mutating route. Prompts are validated before they leave the-deployment, and the default install never sends one anywhere — the model runs-beside it. Report vulnerabilities via [SECURITY.md](SECURITY.md).+Secrets are generated per deployment and never committed; connector credentials+are encrypted at rest. Services connect to Postgres as a DML-only role, so the+row-level-security policies actually apply to them, and tenant isolation is+enforced at the query layer in every store. RBAC gates every mutating route,+ingest is authenticated, and the default install sends no prompt anywhere —+the model runs beside it. Report issues via [SECURITY.md](SECURITY.md).## Developingdiff --git a/apps/docs/docs/architecture.md b/apps/docs/docs/architecture.mdindex b84c81ca7..7cb8ecc1e 100644--- a/apps/docs/docs/architecture.md+++ b/apps/docs/docs/architecture.md@@ -1,331 +1,404 @@---sidebar_position: 4+title: Architecture+description: What happens when AiSOC receives one security event, in order, with every box linked to the code that implements it.---# Architecture-## High-Level Data Flow--```-External Sources- (EDR · SIEM · Cloud · Identity · Network · SaaS · Threat Intel)- │ ▲- │ │- │ osquery-tls + osquery-extensions- │ (endpoint telemetry, host live-query)- ▼ connectors (50 vendors, registry-based)- Kafka spine ◄── Honeytokens (deception events)- │- ┌────┼─────────────────────┬────────────────────┐- ▼ ▼ ▼ ▼-Fusion UEBA Detections Threat Intel-(ML + (Welford, (Sigma·YARA· (TAXII · MISP ·- RBA) Z-score) KQL·EQL · DAC) OTX · KEV)- │ │ │ │- └────┴─────────────────────┴────────────────────┘- │- PostgreSQL · ClickHouse · OpenSearch- Qdrant (vectors) · Neo4j (graph) · Redis- │- FastAPI Core API (port 8000)- │- ┌─────────┬────────┼────────┬─────────────┬─────────────┐- ▼ ▼ ▼ ▼ ▼ ▼-Next.js Agents Realtime MCP Slack-bot Actions-(3000) (8001) (8086, (TS, (ChatOps, (8002,-React + WS+Push) stdio) HMAC-signed) SOAR exec)-PWA │- ▼- Investigation Ledger- (every prompt/tool/step,- replayable per case)+This page follows **one security event** from the moment something hands it to+AiSOC until a human approves a response. That ordering is deliberate: a list of+services tells you what exists, and this tells you what happens.++Every box in every diagram below corresponds to code in this repository and+links to it. Where a component exists but nothing calls it on this path, it+says so rather than being quietly drawn in.++## The path, in one diagram++```mermaid+flowchart TD+ src["1 · Your security tools"]+ ing["2 · Normalization<br/>services/ingest"]+ spine[("3 · Event spine<br/>Kafka aisoc.raw_events")]+ det["4 · Detection<br/>services/fusion"]+ enr["5 · Enrichment<br/>services/enrichment"]+ fuse["6 · Promotion + correlation<br/>services/fusion"]+ pg[("7 · Alert<br/>PostgreSQL")]+ fused[("Kafka aisoc.alerts.fused")]+ ai["8 · AI investigation<br/>services/agents"]+ led[("9 · Investigation Ledger<br/>PostgreSQL")]+ appr["10 · Approval<br/>services/api"]+ act["11 · Response<br/>services/actions"]++ src --> ing --> spine --> det --> fuse --> pg --> fused+ spine -.-> enr -.-> fuse+ fused --> ai --> led --> appr --> act```-Key structural pieces include the **Investigation Ledger** (every-agent prompt, response, evidence citation, and tool call is logged step-by--step against a case and replayable in the UI), the **Ambient Copilot**-(context-aware next-action surface across alerts, cases, rules, and-playbooks), the **Responder PWA** (passkey-only mobile route at-`/responder/*` with VAPID Web Push), the **public eval harness** (one-real measurement plus three substrate self-consistency gates, run in CI),-the **MCP server** (`@aisoc/mcp`, exposes 13 tools to Claude / Cursor /-Continue / Cody — including the warm-tier `aisoc_lake_query` /-`aisoc_lake_schema` pair, gated by the `lake:query` permission), and the-**click-and-connect connector platform** (next section).--Additional capabilities introduced in 2026 H2:--- **Detection-as-Code (DAC)** — propose → review → eval-gate → promote lifecycle for detection rules, with eval results gating promotion.-- **Detection confidence** — each fused alert carries a derived confidence label (`high / medium / low`) with an ordered evidence chain.-- **Detection drift monitoring** — scheduled ATT&CK coverage snapshots with delta tracking.-- **Hunt-as-Code** — YAML hunt definitions in `hunts/` with hypothesis-driven indicator matching and APScheduler-driven continuous execution.-- **Risk-Based Alerting (RBA)** — time-decayed entity risk scoring that promotes high-risk entities to incidents.-- **Federated search** — translate a single query into SPL, KQL, and ES|QL and fan out to connected SIEMs.-- **ChatOps verification** — HMAC-signed Slack/Teams interactive prompts for human-in-the-loop response actions.-- **AI-vs-AI adversary eval** — deterministic attacker-LLM mutator for testing detection resilience.--### v1.5 — market-driven additions (2026-05-07)--The v1.5 release ships from a review of G2, Gartner Peer Insights, and customer feedback on AI SOC / SIEM / SOAR platforms. It adds five new agents, eight new console pages, four new API surfaces, and ten new connectors:--- **Autonomous alert triage agents** — `services/agents/app/agents/auto_triage_agent.py` plus four sibling agents (phishing, identity, cloud, insider-threat) classify each alert as `true_positive` / `false_positive` / `benign` with a confidence score. Low-confidence noise auto-closes; the rest is escalated. Surfaced via `/api/v1/agents/triage`.-- **Conversational investigation chat** — the multi-turn copilot is `CopilotDock` (app-wide) and `/copilot`; `/investigate` permanently redirects to `/hunt`. The separate `InvestigationChat.tsx` component this line used to name was imported by nothing and answered from a canned reply table rather than the API, so it was removed rather than wired.-- **MITRE ATT&CK coverage advisor** — `/coverage-advisor` reads `GET /api/v1/detection/coverage` and ranks the techniques your own rules reference by how much *enabled* coverage each has. It does not rank by adversary prevalence, and techniques no rule references do not appear — the page says so rather than publishing a coverage score it cannot support.-- **Shift handoff dashboard** — `/shifts` shows outgoing/incoming analysts the active cases, in-flight investigations, and queued approvals on one screen. Endpoints in `services/api/app/api/v1/endpoints/shifts.py`.-- **EASM (External Attack Surface Management)** — `/easm` discovers assets, exposed services, and certificate-expiry risks for everything the org points at the public internet.-- **MSSP executive dashboard** — `/mssp` rolls up KPIs, cross-tenant alert volumes, and per-customer SLA posture into a multi-tenant pane.-- **Investigation Rail on `/alerts`** — two-pane alert queue: selecting a row opens a right-hand rail with **deterministic correlation narrative** (fusion-time, no LLM on read), **related entities** with pivot links, a **mini-timeline** (case + audit, six events), and **recommended actions**. `GET /api/v1/alerts/{id}` returns a structured envelope; legacy rows lazy back-fill narrative on first read. See [Investigation Rail](./console/investigation-rail).-- **Alert noise-tuning dashboard** — `/noise-tuning` (and `/detection/tuning` where deployed) surfaces per-rule false-positive rate, suppression candidates, and one-click tuning.-- **Team analytics & gamification** — `/analytics/team` ships analyst leaderboard, MTTR per analyst, dispositions accuracy, and shift workload balance.-- **STIX 2.1 / TAXII 2.1 publishing** — `services/api/app/api/v1/endpoints/stix_taxii.py` pushes the tenant's IOCs and threat-actor profiles to upstream / community feeds.-- **Automated compliance evidence** — `services/api/app/api/v1/endpoints/compliance.py` collects point-in-time evidence for SOC 2 / ISO 27001 / NIST CSF / PCI-DSS / HIPAA / DORA.-- **AI-generated incident reports** — one-click "Export Report" on every case generates a PDF incident report from the Investigation Ledger.-- **Air-gap deployment configuration** — `services/api/app/api/v1/endpoints/deployment.py` exposes air-gap mode toggles for tenants that disallow external feeds.-- **Connector catalog growth** — SentinelOne, Cortex XDR / XSIAM, Wiz, Snyk, Zscaler, Proofpoint, ServiceNow, Jira, 1Password, Duo Security, Carbon Black, Chronicle, Cisco Umbrella, CrowdStrike, Datadog Cloud SIEM, Elastic, Mimecast, Okta, Rapid7 InsightIDR, Salesforce, Microsoft Sentinel, Splunk, Sumo Logic, Tenable, Trellix Helix, Trend Vision One, and more. **The catalog is now 84 connectors** across EDR, SIEM, cloud, identity, SaaS, network, and email categories. See [Connectors](./connectors/).--## Connector polling and credential vault+The dashed branch is the `full` profile. Everything solid runs in CORE, which+is what `make up` starts.-```-Browser (Add connector wizard)- │- │ POST /api/v1/connectors { type, auth_config, connector_config }- ▼-services/api ─── CredentialVault.encrypt() ───▶ vault:v1:<base64>- │- │ INSERT INTO connectors(auth_config_encrypted, ...)- ▼-PostgreSQL ◄────────────────────────────────────────────────────┐- │ │- │ every 30s reload │- ▼ │-services/connectors │- ConnectorScheduler (APScheduler, in-process) │- │ │- │ per-instance job @ poll_interval_seconds (default 300s) │- ▼ │- _poll_one(instance) │- │ │- │ 1. CredentialVault.decrypt() ── reads same key as API ──┘- │ 2. construct connector class from registry- │ 3. await connector.fetch_alerts(since_seconds=300)- │ 4. [connector.normalize(e) for e in raw_events]- │ 5. await ingest_client.push_events(tenant_id, normalized)- │ 6. record_poll_success(events_added, elapsed_ms)- ▼-services/ingest POST /v1/ingest/batch X-Tenant-ID: <uuid>- │- ▼- Kafka spine ──▶ Fusion · UEBA · Detections (existing pipeline)-```+----### Founder-flow direct-write submit (v7.3.1+)+## 1 · Something hands AiSOC an event-For the fresh-clone demo and any flow where Kafka / Fusion / `services/ingest`-is not required for the first alert, `services/api` exposes a dedicated-direct-write endpoint:+Two doors, and they are genuinely different.+**Push.** Anything that can make an HTTP request posts to the ingest API, with+a token minted by `make ingest-token`:++```bash+curl -X POST http://localhost:8081/v1/ingest/batch \+ -H 'Content-Type: application/json' -H 'Authorization: Bearer <push-token>' \+ -d '{"connector_id":"edr-1","connector_type":"crowdstrike","source_format":"json",+ "events":[{"severity":"high","title":"Encoded PowerShell from Office",+ "host":"WIN-FIN-01","process_name":"powershell.exe",+ "parent_process":"winword.exe"}]}'```-POST /api/v1/alerts/submit (X-Tenant-ID: <uuid>)- │- │ payload: { "events": [<OCSF event>, ...], "rule_id": "demo.lateral-movement" }- ▼-services/api- │ 1. parse OCSF events (no Kafka, no services/ingest)- │ 2. _synthesise_alert_from_events() ── derives entities, MITRE tags, severity- │ 3. INSERT INTO alerts(...) ── single row, idempotent- │ 4. return { id, status: "new" }- ▼-PostgreSQL ──▶ /alerts console (lights up immediately)++**Pull.** [`services/connectors`](https://github.com/beenuar/AiSOC/tree/main/services/connectors)+polls a vendor API on a schedule, decrypts that tenant's stored credentials+through the vault at poll time, and forwards the result to the same ingest+endpoint. It is a `full`-profile service; CORE accepts pushes instead. The+registry is **now 84 connectors** — that figure is generated from+`services/connectors/app/connectors/__init__.py` by+[`scripts/generate_connector_count.py`](https://github.com/beenuar/AiSOC/blob/main/scripts/generate_connector_count.py),+which fails CI if this sentence disagrees with the tree.++:::info The tenant comes from the credential, never from the header+Every ingest door is authenticated, and each resolves a tenant from the+credential rather than believing a header:++| Door | Credential |+|---|---|+| `/v1/ingest`, `/v1/ingest/batch` | A minted push token pinned to the `connector-push` template, **or** `AISOC_SERVICE_TOKEN` for AiSOC's own services, which poll for many tenants and declare one on `X-Tenant-ID` |+| `/v1/inbox/{token}`, `/v1/inbox/email/{token}` | A minted per-tenant inbox token in the path |+| `/v1/inbox/cef`, `/v1/inbox/hec` | The same, in a header, pinned to their own templates |+| `/v1/ingest/k8s-audit/{tenant}` | The `X-AiSOC-K8s-Token` shared secret; 503 until one is set |++The pinning matters: an inbox token is pasted into a third party's webhook+configuration, so minting one for a paging vendor must not also hand that+vendor a general ingest credential.++`X-Tenant-ID` is still read and is never authority — it is **intersected** with+what the credential authorises, so naming a tenant outside that scope narrows+to nothing and is refused rather than reaching out. There is no dev-mode+bypass, and an ingest service that cannot verify a credential answers 503+rather than accepting the write. Procedure:+[Ingest authentication](./operations/ingest-authentication).+:::++## 2 · Ingest normalizes it++[`services/ingest`](https://github.com/beenuar/AiSOC/tree/main/services/ingest)+(Go, container port 8080) maps the vendor payload onto a common OCSF-shaped+envelope using a per-connector **profile**, in+[`internal/normalizer`](https://github.com/beenuar/AiSOC/tree/main/services/ingest/internal/normalizer).++A profile decides two things that matter downstream: the OCSF `class_uid` — a+vendor finding that already passed the vendor's own correlation belongs at+`2001`, Security Finding — and the vendor's severity ladder, mapped onto the+five tiers `info | low | medium | high | critical`.++A connector with no profile falls through to a **vendor-neutral generic+profile**, which resolves `actor.user.name`, `device.name` and+`src_endpoint.ip` from the spellings connectors actually use (`user` /+`username` / `actor`, `host` / `hostname`, `src_ip` / `source_ip` /+`client_ip`) in a fixed order. The event keeps the entities the rest of the+platform pivots on; what a vendor profile adds is that vendor's own field names+and its OCSF class.++[`scripts/check_connector_profiles.py`](https://github.com/beenuar/AiSOC/blob/main/scripts/check_connector_profiles.py)+fails CI if a connector identifier stops resolving in either direction.++## 3 · It enters the event spine++Ingest publishes to the Kafka topic **`aisoc.raw_events`**. This is the+boundary that makes everything downstream independent: a consumer can be added,+restarted or left to fall behind without ingest knowing about it.++Kafka is not a cache here. It is the reason a slow detection worker cannot+block the front door, and the reason an event is not lost when fusion restarts.++## 4 · Fusion evaluates detections++[`services/fusion`](https://github.com/beenuar/AiSOC/tree/main/services/fusion)+(container port 8003) consumes the spine and runs **833 executable detection+rules** against each event.++The rules the engine runs are compiled into+[`app/data/detection_ruleset.json`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/data/detection_ruleset.json).+The YAML under `detections/` is a *projection* of that, not the engine's input —+most of what is on disk sits in `_quarantine/` and never loads, which is why+the corpus figure and the executable figure are published separately. See+[the detection truth table](https://github.com/beenuar/AiSOC/blob/main/docs/detections/truth-table.md).++| Step | Code |+|---|---|+| Match a rule against an event | [`detection_engine.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/detection_engine.py) · [`detection_matcher.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/detection_matcher.py) |+| Derive fields rules match on | [`derived_fields.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/derived_fields.py) |+| Rules needing a time window | [`windowed_detection.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/windowed_detection.py) |+| Events the consumer cannot parse | [`dlq.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/dlq.py) → `aisoc.alerts.dlq` |++## 5 · Enrichment, if it is running++[`services/enrichment`](https://github.com/beenuar/AiSOC/tree/main/services/enrichment)+adds threat-intel and vulnerability context at fuse time. It is a+`full`-profile service. **In CORE it is absent and fusion skips it** — not+retried, not silently failed, skipped.++Separately and in CORE,+[`services/threatintel`](https://github.com/beenuar/AiSOC/tree/main/services/threatintel)+polls the CISA Known Exploited Vulnerabilities catalogue — public, keyless,+authoritative — into Qdrant, which is what the console's Threat Intelligence+page reads. It is the only externally-sourced real data a fresh install has:+1,725 entries were collected within a poll interval of `make up` on the stack+these captures came from.++++Note what the counters say. **1,725** is the store's count; **100** is the page+being displayed; and the two cards beneath are explicitly *of shown*. A page+that reports its own length as the size of the corpus is a small lie that gets+believed, so the page does not.++## 6 · Promotion and correlation++Two separate decisions, often confused:++**Promotion** asks whether this event should become an alert at all.+[`promoter.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/promoter.py)+promotes an OCSF category-2 finding or anything at `severity_id >= 4`. Routine+telemetry is not promoted; with the `full` profile it is still archived to the+event lake and remains huntable.++**Correlation** asks whether this alert belongs with others.+[`correlator.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/correlator.py)+keys on `{tenant}:{entity}:{tactic}` in Redis, so ten related alerts become one+thing an analyst looks at rather than ten.++Fusion also writes, at fuse time and without an LLM:++- a deterministic **correlation narrative** ([`narrative.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/narrative.py)) — reads never call a model;+- a **confidence** score, 0–100 with a `low | medium | high` band ([`confidence.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/confidence.py)), which is independent of severity;+- **provenance** ([`provenance.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/provenance.py)) so the alert can say which connector fed it.++The queue below is the output of exactly this path on a CORE stack — twelve+alerts, each attributed to the connector whose profile normalized it, produced+from telemetry pushed through the ingest API above.++++## 7 · The alert is written, and announced++[`alert_sink.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/alert_sink.py)+inserts the row into PostgreSQL and publishes to **`aisoc.alerts.fused`**.++Two consumers pick it up.+[`services/realtime`](https://github.com/beenuar/AiSOC/tree/main/services/realtime)+(Node, container port 4000) pushes it to the console over WebSocket.+[`services/agents`](https://github.com/beenuar/AiSOC/tree/main/services/agents)+takes it for triage — the next step.++## 8 · An agent triages it++`FusedAlertTriageWorker` in+[`services/agents`](https://github.com/beenuar/AiSOC/tree/main/services/agents)+(container port 8084) auto-triages **every** fused alert.++```mermaid+sequenceDiagram+ participant K as Kafka aisoc.alerts.fused+ participant A as services/agents+ participant C as Prompt contract+ participant G as litellm gateway+ participant M as Model+ participant L as Investigation Ledger++ K->>A: fused alert+ A->>A: gather evidence — alert, siblings, entities, prior verdicts+ A->>C: proposed prompt+ C-->>A: refuse if it carries raw logs or secret-shaped values+ C->>G: validated prompt, as alias aisoc-triage+ G->>M: whichever model the alias resolves to+ M-->>A: verdict, confidence, cited indicators+ A->>A: groundedness check+ Note over A: an indicator the evidence never<br/>contained demotes the verdict<br/>to human review+ A->>L: prompt, tool calls, citations, verdict, tokens, cost```-This is the path used by the `aisoc submit` CLI command and the-[`examples/alerts/lateral-movement.json`](https://github.com/beenuar/AiSOC/tree/main/examples/alerts/lateral-movement.json)-canonical payload. The classic Kafka-fed pipeline above is still the-**production** path; the direct-write endpoint exists so a fresh clone of-the repo, an `aisoc serve` process, and a single `aisoc submit` call are-enough to see an alert on `/alerts` — no broker, no schedulers, no-connectors required.--The `services/api` service holds the **encrypt** authority for the vault-and is the only writer to `connectors.auth_config_encrypted`.-`services/connectors` ships a vendored read-path `decrypt_dict()` that-pairs to the same `AISOC_CREDENTIAL_KEY` so the scheduler can decrypt-per-poll without owning the write path. Key rotation is supported via-`MultiFernet` and the `AISOC_CREDENTIAL_KEY_ROTATION_FROM` env var; the-[Operations: Credentials](./operations/credentials) page documents the-full rotation procedure, threat model, and hosted-OAuth roadmap.--The wizard's `Test connection` button skips the vault entirely — it-forwards the raw form values to a stateless `POST /connectors/{id}/test`-endpoint on the connectors microservice, which constructs the connector-in memory and calls `test_connection()`. Bad credentials never touch the-database.--## Monorepo Layout+Four properties, each enforced in code rather than asserted:++- **The contract runs before the request.** A prompt about to carry raw OCSF, vendor log lines or secret-shaped values is refused, not redacted afterwards ([`prompt_sanitizer.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/app/investigator/prompt_sanitizer.py)).+- **The model never writes a query.** It picks a typed tool and passes arguments; the tool writes the SQL or the traversal.+- **Citations are checked.** A verdict citing an indicator that was not in the evidence is demoted rather than auto-closed. Groundedness is nullable, and `null` means *not assessed* — a different fact from zero.+- **Every alias resolves at the gateway.** AiSOC asks for `aisoc-triage`, not for a model name; [`infra/litellm/config.yaml`](https://github.com/beenuar/AiSOC/blob/main/infra/litellm/config.yaml) maps aliases to models, so swapping local for hosted is three variables in `.env`.++### What the default install actually does++CORE ships a pinned `llama3.2:3b-instruct-q4_K_M` behind the gateway, so a+clone with no account produces a real verdict with real token counts. It is a+3B quantized model, and the difference from a frontier model is visible: in a+measured run of 19 auto-triages on a CORE stack, **7 returned schema-valid+output and 12 did not**. The 12 fell back to the deterministic path, logged+`auto_triage_worker.llm_failed_fallback`, and the Investigation Rail shows+which path produced the verdict it is displaying.++**No hosted provider has ever been exercised in this repository.** There is no+funded key, so per-model figures read *not measured* rather than zero. The+local path is measured; the hosted path is configured and untested. Treat those+as the different claims they are.++Here is one of the 7, as the console renders it — the verdict, the confidence,+the groundedness (*not assessed*, because this run did not score it), and the+model's own rationale reproduced verbatim so a reader can tell which path+answered:++++## 9 · The reasoning is written down++Every prompt, tool call, citation, verdict, token count and cost goes to the+**Investigation Ledger**, append-only and tenant-scoped. The verdict is also+projected back onto the alert row, which is what the Investigation Rail's+*Automated triage* section renders.++The point is not audit for its own sake: a verdict you cannot inspect is a+verdict you have to trust, and this project does not ask you to.++- Schema: [`008_investigation_ledger.sql`](https://github.com/beenuar/AiSOC/blob/main/services/api/migrations/008_investigation_ledger.sql)+- Writer: [`ledger.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/app/investigator/ledger.py)+- Reader: [`investigations.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/api/v1/endpoints/investigations.py)++## 10 · A human approves+Nothing executes on its own. An action is **proposed**; a human with the right+permission approves it; and the approver may not be the person who requested+it.++Risk, reversibility, verification and approval tier are declared **per+capability** — the contract belongs to the verb, not to the vendor that+implements it. See [Live actions](./concepts/live-actions) and+[Action approvals](./operations/action-approvals).++## 11 · The action runs, and is verified++[`services/actions`](https://github.com/beenuar/AiSOC/tree/main/services/actions)+executes the approved capability against the vendor and then **probes the+vendor to confirm it took effect**, rather than inferring success from an HTTP+200. A capability whose effect cannot be verified is not eligible for automatic+execution.++Without vendor credentials an executor returns `simulated` and says so.+`executed` is the single field that means a vendor was actually touched.++Playbooks reach this same path. The engine's step vocabulary is 22 types+([`models.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/app/playbook/models.py)),+and the editor's palette offers **21** of them — `approval` is withheld because+the engine is a single-threaded index walk with no pause or resume, so a step+it cannot run is not offered as a button.++---++## Where the path stops++Step 10 has no automatic trigger. There is no code path that dispatches a+response without a human. That is deliberate, and it is the honest boundary of+the word "autonomous" in this project.++## What runs in which profile++| Profile | Command | Long-running services | Memory | What you get |+|---|---|---|---|---|+| **core** | `make up` | 14 | ~8 GB | Steps 1–10, plus the LLM gateway, the local model behind it, and the CISA KEV feed with its vector store |+| **full** | `make up-full` | 22 | ~12 GB | Core plus the event lake, entity graph, full-text search, enrichment and scheduled connectors |+| **demo** | `make up && make demo` | 14 | ~8 GB | Core plus clearly-labelled synthetic data |++CORE is 14 long-running containers; `ollama-pull` is a fifteenth that fetches+the model once and exits. `full` is 22 long-running containers — not 30, which+is every profile including `monitoring`, `chatops`, `extras` and `osquery`,+none of which `make up-full` starts.++Memory and disk are measured, not estimated: the model container reaches about+3 GiB while answering, which is what moved the floor to 8 GB, and CORE is about+16.5 GB of unique image layers plus a 2 GB model volume.++## Which store holds what, and what breaks without it++| Store | Profile | Holds | Without it |+|---|---|---|---|+| **PostgreSQL** | core | Alerts, incidents, cases, users, tenants, rules, audit log, ledger | Nothing works |+| **Redis** | core | Correlation windows, dedup keys, investigation run state | No correlation — every alert becomes its own incident |+| **Kafka** | core | `aisoc.raw_events`, `aisoc.alerts.fused`, `aisoc.alerts.dlq` | No spine; ingest would have to call every consumer directly |+| **Qdrant** | core | Threat-intel indicators and actor embeddings | No Threat Intelligence page, no semantic IOC matching |+| **ClickHouse** | full | Every normalized event | No event lake, no hunting over raw telemetry. **Alerting is unaffected** |+| **Neo4j** | full | Entity graph, blast radius | No graph context. The `/graph` page reports the failure rather than drawing an invented graph |+| **OpenSearch** | full | Full-text IOC and actor search | Threat-intel feeds still write to Qdrant; full-text IOC search is unavailable and says so |++## How tenant isolation is actually enforced++Per store, because one mechanism does not cover them all:++| Store | Mechanism |+|---|---|+| PostgreSQL | Row-level security **plus** a `tenant_id` predicate at the query layer |+| ClickHouse | `lake_sql.rewrite_for_tenant()` injects the predicate and **fails closed** if it does not survive rendering |+| Neo4j | Every node of every path is scoped, not just the traversal's start node |+| Redis | `tenant:{id}:` key prefix |+| Qdrant | `tenant_id` in the point payload plus a mandatory query filter |+| Kafka | `X-Tenant-ID` header / `tenant_id` envelope, filtered per consumer |++Two details that are easy to get wrong and were:++**Services do not connect as the database owner.** `aisoc` owns the schema and+applies migrations; every service connects as `aisoc_app`, which holds+`SELECT / INSERT / UPDATE / DELETE` and no DDL. This is not tidiness — a+superuser ignores RLS policies even under `FORCE ROW LEVEL SECURITY`, so while+services ran as the owner every policy in the schema filtered nothing. Check it+on a running stack:++```sql+SELECT count(*) FROM pg_policies; -- 84 on a CORE stack here+SELECT count(*) FROM pg_class WHERE relrowsecurity; -- 83 tables+SELECT DISTINCT privilege_type FROM information_schema.role_table_grants+ WHERE grantee = 'aisoc_app'; -- DML only, no DDL```-AiSOC/-├── apps/-│ ├── web/ # Next.js 14 React frontend (incl. Responder PWA)-│ └── docs/ # This Docusaurus site-├── services/-│ ├── api/ # FastAPI gateway (port 8000)-│ ├── agents/ # LangGraph investigator (port 8001)-│ ├── realtime/ # Node/TS WebSocket + Web Push (port 8086)-│ ├── ingest/ # Go OCSF normaliser (port 8081)-│ ├── enrichment/ # Go enrichment fan-out (port 8080)-│ ├── fusion/ # Fusion + ML scoring (port 8003)-│ ├── actions/ # Action executor (port 8002)-│ ├── threatintel/ # TAXII / MISP / OTX / KEV (port 8005)-│ ├── ueba/ # User behavior analytics (port 8007)-│ ├── honeytokens/ # Deception platform (port 8008)-│ ├── purple-team/ # Adversary emulation (port 8006)-│ ├── connectors/ # Connector polling + credential vault (50 vendors)-│ ├── demo-producer/ # Synthetic event generator for demos-│ ├── mcp/ # Model Context Protocol server (TypeScript)-│ ├── osquery-tls/ # osquery TLS server: enrol hosts, ship results-│ ├── osquery-extensions/ # Custom osquery tables/decorators-│ └── slack-bot/ # ChatOps surface (HMAC-signed approvals)-├── packages/-│ ├── plugin-sdk-py/ # Python plugin SDK-│ ├── plugin-sdk-go/ # Go plugin SDK-│ ├── sdk-py/ # Python client SDK-│ ├── sdk-ts/ # TypeScript client SDK-│ └── sdk-go/ # Go models / client helpers-├── infra/-│ ├── helm/aisoc/ # Helm chart (Kubernetes, HA-ready)-│ ├── terraform/ # Terraform modules-│ ├── coolify/ # One-click deploy on Coolify-│ ├── fly/ # Fly.io demo deployments-│ ├── railway/ # Railway templates-│ └── render/ # render.yaml blueprint-├── detections/ # 200+ Sigma/YARA/KQL detection rules (YAML)-├── hunts/ # Hunt-as-Code YAML definitions (hypothesis + indicators)-├── playbooks/ # 50+ SOAR playbooks (YAML)-├── plugins/ # 15 first-party plugins (Go + Python)-├── marketplace/ # Marketplace index (index.json)-├── docs/ # OpenAPI spec (openapi.yaml)-├── docker-compose.yml # Full development stack-├── infra/compose/ # docker-compose.{demo,dev,airgap}.yml overlays-└── scripts/ # Utilities (seed, eval harness, build, validate)++**The graph writer keys on `natural_key` with `tenant_id` as a property**, so+a strict read filter fails closed: safe, never leaking, at some cost to+completeness on shared infrastructure nodes such as public IPs.++## When a step has nothing to show++An honest empty state is part of this architecture, not a gap in it. The SOC+operations dashboard is driven entirely by live API responses, and on a tenant+with no connectors it says exactly that rather than rendering a placeholder+chart:++++The same rule holds for failures. With Neo4j absent — it is a `full`-profile+store — `/graph` reports `API 503 … /api/v1/graph` instead of drawing a graph+it does not have, and federated search names the four SIEMs it *would* query+rather than showing an empty result set as though it had searched.++## Verifying any of this yourself++```bash+make up && make smoke```-## Service Responsibilities--| Service | Port | Language | Responsibility |-|---------|------|----------|----------------|-| `api` | 8000 | Python (FastAPI) | REST gateway, auth, RBAC, RLS, audit log, **Investigation Ledger**, **alert detail envelope** (Investigation Rail: narrative, entities, mini-timeline, actions), Ambient Copilot, marketplace, approvals, on-call, passkeys, push subscriptions, **Detection Proposals** (DAC lifecycle), **Federated Search** fan-out, SLA tracking, **Shifts** handoff, **STIX/TAXII** publishing, **Compliance evidence** collection, **Deployment / air-gap** configuration |-| `agents` | 8001 | Python (LangGraph) | Orchestrator + recon + forensic + responder + report-writer agents, **autonomous triage agent** + phishing / identity / cloud / insider-threat sub-agents, **conversational investigation chat**, playbook engine, ledger writes, **Hunt-as-Code** engine + scheduler |-| `realtime` | 8086 | TypeScript (Node.js) | WebSocket streaming of agent steps; **VAPID Web Push** delivery for the Responder PWA |-| `ingest` | 8081 | Go | OCSF normalisation, Bloom-filter dedup, Kafka publish |-| `enrichment` | 8080 | Go | Enrichment fan-out (IP, domain, hash, email, user) |-| `fusion` | 8003 | Python | ML scoring (LightGBM + Isolation Forest), correlation, **deterministic correlation narratives** on fused alerts, **alert confidence scoring**, **entity risk / RBA** |-| `actions` | 8002 | Python | Plugin action executor, blast-radius gating, **ChatOps verification** (HMAC-signed Slack/Teams prompts) |-| `threatintel` | 8005 | Python | TAXII 2.1 / MISP / OTX / KEV ingestion + triple storage |-| `ueba` | 8007 | Python | Welford baseline, Z-score scoring, anomaly stream |-| `honeytokens` | 8008 | Python | Token lifecycle, HMAC signing, webhook dispatch |-| `purple-team` | 8006 | Python | ART YAML parser, Caldera executor, ATT&CK heatmap, **detection drift snapshots** |-| `connectors` | — | Python | Connector polling (APScheduler), credential vault (`CredentialVault`), registry-based discovery — **50 vendors shipped** across EDR / SIEM / cloud / IAM / SaaS / network / email |-| `demo-producer` | — | Python | Synthetic event generator for demos and evaluation |-| `mcp` | n/a | TypeScript | Model Context Protocol stdio server, 13 tools for IDE-side agents (Claude / Cursor / Continue / Cody) — discovery, deep-dive, action/replay, and the warm-tier lake query pair |-| `osquery-tls` | 8443 | Go | TLS server implementing osquery's enrol/config/distributed/log endpoints; ships normalised host events into `ingest` |-| `osquery-extensions` | — | Go | Out-of-band osquery extensions registering custom virtual tables and decorators consumed by `osquery-tls` |-| `slack-bot` | — | Python | ChatOps surface: posts approval prompts, exposes `/aisoc` slash command, verifies inbound interactions with HMAC-signed Slack request signatures |-| `web` | 3000 | TypeScript (Next.js) | React console + Responder PWA route group, **`/alerts` Investigation Rail**, **benchmark scoreboard**, conversational investigation chat |--## Storage Tier--| Store | Role |-|-------|------|-| PostgreSQL | Operational data, RLS-enforced multi-tenancy, audit log, **investigation ledger** |-| ClickHouse | Time-series analytics, compliance metrics |-| OpenSearch | Full-text search across alerts, logs, cases |-| Qdrant | Semantic vector search for RAG copilot + agent memory |-| Neo4j | Attack graph, entity relationships, blast-radius queries |-| Redis | Cache, rate-limiting, session store, push subscription cache |-| Kafka | Async event backbone |--## Investigation Ledger--Every agent action (LLM prompt, LLM response, tool call, evidence-citation, decision branch) is appended to the `investigation_ledger`-table, scoped to a case and stamped with the agent identity, model,-prompt hash, and timestamp. The Case workspace renders this as a-scrubbable timeline so analysts can replay the agent's reasoning.--The schema is defined in-[`services/api/migrations/008_investigation_ledger.sql`](https://github.com/beenuar/AiSOC/blob/main/services/api/migrations/008_investigation_ledger.sql).-The agent-side writer lives in-[`services/agents/app/investigator/ledger.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/app/investigator/ledger.py),-and the UI consumer is-[`apps/web/src/components/cases/InvestigationLedger.tsx`](https://github.com/beenuar/AiSOC/blob/main/apps/web/src/components/cases/InvestigationLedger.tsx).--### Prompt sanitization layer--Investigator agents (`recon`, `forensic`, `responder`, `report_writer`) consume-attacker-influenced strings — enrichment payloads, dark-web excerpts, vendor-descriptions, raw alert fields — and hand them to an LLM. Every one of those-agents now routes its context through-[`services/agents/app/investigator/prompt_sanitizer.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/app/investigator/prompt_sanitizer.py),-which strips known role / chat delimiters, redacts common jailbreak phrasings,-caps field length, bounds list size and recursion depth, and wraps the result-in explicit `<UNTRUSTED_DATA>` tags. The agents still validate the LLM's-response against a Pydantic schema before persisting it. See the-[LLM prompt safety section in the security guide](./operations/security#llm-prompt-safety)-for the threat model and defence-in-depth layers.--## Investigation Rail and correlation narrative--The alert queue (`/alerts`) pairs the sortable table with an **Investigation Rail**-([`InvestigationRail.tsx`](https://github.com/beenuar/AiSOC/blob/main/apps/web/src/components/alerts/InvestigationRail.tsx))-fed by `GET /api/v1/alerts/{id}`. The response envelope is assembled in-[`services/api/app/services/alert_rail.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/services/alert_rail.py)-(narrative text, entity buckets, merged mini-timeline, recommended actions).--**Narrative** — At fusion time, `services/fusion` runs the same deterministic-builder as the API vendored copy ([`narrative.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/narrative.py))-so promoted alerts persist a short explanation of *which* signals correlated-and *why*. Reads do not call an LLM. Alerts created before this shipped get a-lazy projection on first detail fetch via-[`narrative_projection.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/services/narrative_projection.py),-then the text is cached on the row (see migration `041_alert_correlation_narrative.sql`).--**Sync** — When the narrative builder changes, run `scripts/sync_vendored_narrative.py`-so `services/api/app/_vendor/narrative.py` stays aligned with fusion.--Operator-facing detail: [Investigation Rail](./console/investigation-rail).--## Responder PWA--The Responder PWA is mounted under the Next.js route group-`apps/web/src/app/(responder)/`. It is **passkey-only** (no passwords),-shows the on-call rotation, lists pending approvals, supports VAPID-Web Push for high-severity alerts, and ships an offline shell.--The schema is defined in-[`009_responder_pwa.sql`](https://github.com/beenuar/AiSOC/blob/main/services/api/migrations/009_responder_pwa.sql).-The push pipeline lives in-[`services/realtime/src/push.ts`](https://github.com/beenuar/AiSOC/blob/main/services/realtime/src/push.ts).--## Enterprise Security Controls--- **Multi-tenancy** — PostgreSQL Row-Level Security on every table; `tenant_id` is derived from the JWT and cannot be spoofed.-- **RBAC** — `require_permission` FastAPI dependency; custom roles with fine-grained action permissions per resource type.-- **SAML 2.0 / OIDC** — Pluggable SSO with JIT user provisioning and group-to-role mapping.-- **WebAuthn / Passkeys** — Required for the Responder PWA; password-less by default.-- **Immutable Audit Log** — Postgres trigger + `SECURITY DEFINER` function prevents UPDATE/DELETE on `audit_log`.-- **Replayable agent decisions** — The Investigation Ledger is append-only and tenant-scoped.-- **OpenTelemetry** — All services emit traces, metrics, and structured logs to a configurable OTLP endpoint.-- **Backup & Restore** — `scripts/backup.sh` / `restore.sh`. Artifacts are encrypted with **AES-256-GCM** before leaving the host and recorded in a **SHA-256 manifest**; restore verifies the digest, decrypts, and refuses a tampered or truncated archive rather than restoring a partial database. Gated end-to-end in `integration.yml`: seed 500 rows, back up, assert the object in the bucket is unreadable (no gzip magic, no schema names), drop the schema, restore, assert 500 rows, then flip a bit and assert the restore is refused. Covers Postgres, ClickHouse, the plugin store, the **Neo4j entity graph** (APOC cypher export), **Qdrant** (per-collection snapshot) and **Redis** (RDB). Scheduled by the `backup` CronJob in the Helm chart — disabled by default because it needs a bucket, credentials and a key only the operator can supply. The RPO is the schedule interval. **Kafka is not backed up**: it is a transport, and its retained window is replayable from the lake rather than restorable.-- **High-Availability Helm** — Multi-replica deployments, HPA, PDB, anti-affinity, and readiness probes.--## Plugin Extension Points--Plugins extend AiSOC at three key points:--- **Enrichers** — Add context to indicators (IP, domain, hash, email)-- **Actions** — Execute response steps (block IP, disable user, create ticket)-- **Connectors** — Ingest events from external sources (SIEM, EDR, cloud)-- **Widgets** — Render plugin-supplied React panels in the case workspace--See [Plugin Overview](./plugins/overview) for the full plugin lifecycle.+`make smoke` posts one event to the ingest API and follows it through Kafka,+detection, promotion, correlation and Postgres, then reads it back from the+public API. It reaches past nothing and each stage reports PASS or FAIL+separately, so a break names the boundary that broke. Source:+[`tests/e2e/golden_pipeline/`](https://github.com/beenuar/AiSOC/tree/main/tests/e2e/golden_pipeline).++## Next++- [Repository overview](./architecture/overview) — where every service and package lives+- [Agent architecture](./architecture/agents) — the four agents and the graph they run on+- [Graph schema](./architecture/graph-schema) — node labels and edge types+- [Investigation Rail](./console/investigation-rail) — what an analyst sees+- [Security](./operations/security) — threat model and controlsdiff --git a/apps/docs/docs/architecture/overview.md b/apps/docs/docs/architecture/overview.mdindex 41245dbcf..f46c26a7a 100644--- a/apps/docs/docs/architecture/overview.md+++ b/apps/docs/docs/architecture/overview.md@@ -32,21 +32,24 @@ AiSOC/│ ├── osquery-tls/ # Python · native osquery TLS server + FIM + pack distribution│ ├── osquery-extensions/ # Go · AI threat-intel table + ML anomaly score table + 3 more│ ├── slack-bot/ # Python · ChatOps surface (Slack Bolt + HMAC signature verification)-│ └── mcp/ # TypeScript · Model Context Protocol server (npm: @aisoc/mcp, lands in v8.0)+│ └── mcp/ # TypeScript · Model Context Protocol server (npm: @aisoc/mcp — built and packed on every tag, upload blocked on registry credentials)├── packages/│ ├── types/ # Shared TS types│ ├── ui/ # Shared React primitives│ ├── ocsf/ # OCSF normalization helpers-│ ├── sdk-ts/ # TypeScript client SDK for AiSOC API (npm: @aisoc/sdk, lands in v8.0)-│ ├── sdk-py/ # Async Python client SDK (PyPI: aisoc-sdk, lands in v8.0)+│ ├── sdk-ts/ # TypeScript client SDK for AiSOC API (npm: @aisoc/sdk — unpublished, see below)+│ ├── sdk-py/ # Async Python client SDK (PyPI: aisoc-sdk — unpublished, see below)│ ├── sdk-go/ # Go client SDK + models (module: github.com/beenuar/AiSOC/packages/sdk-go)│ ├── plugin-sdk-ts/ # TypeScript plugin development SDK-│ ├── plugin-sdk-py/ # Python plugin development SDK (PyPI: aisoc-plugin-sdk, lands in v8.0)+│ ├── plugin-sdk-py/ # Python plugin development SDK (PyPI: aisoc-plugin-sdk — unpublished, see below)│ ├── plugin-sdk-go/ # Go plugin development SDK (module: github.com/beenuar/AiSOC/packages/plugin-sdk-go)-│ └── aisoc-cli/ # CLI: scaffold / validate / publish plugins & detections (PyPI: aisoc-cli, lands in v8.0)-├── detections/ # Community Sigma detection rules (YAML) — 800+ native+│ └── aisoc-cli/ # CLI: scaffold / validate / publish plugins & detections (PyPI: aisoc-cli — unpublished, see below)+├── detections/ # Detection rules as YAML. A *projection* of the compiled+│ # ruleset, not the engine's input: 7,016 indexed on disk,+│ # 5,937 of them quarantined, 833 executable. Recount with+│ # scripts/generate_corpus_stats.py rather than trusting this line.├── hunts/ # Hunt-as-Code YAML definitions (hypothesis + indicators + schedule)-├── playbooks/ # Community SOAR playbooks (YAML)+├── playbooks/ # SOAR playbooks (YAML) — 62 packs├── plugins/ # First-party plugins (Go + Python) + marketplace manifests├── marketplace/ # Marketplace index (JSON, generated by scripts/build_marketplace.py)├── infra/@@ -78,25 +81,36 @@ AiSOC/## Service map-| Service | Lang | Port | Role |-|---|---|---|---|-| `web` | Next.js + React | 3000 | SOC console (alerts detail + Investigation Rail), benchmark scoreboard, marketing landing |-| `api` | Python · FastAPI | 8000 | Alerts (detail envelope + correlation narrative), cases, RBAC, graph, rules, audit, compliance, detection proposals (DAC), federated search fan-out, SLA tracking |-| `realtime` | Node.js · `ws` | 8086 | Per-channel WebSocket fan-out + VAPID Web Push (short-lived signed tickets) |-| `agents` | Python · LangGraph | 8001 | Multi-agent reasoning + Qdrant RAG + Hunt-as-Code engine & scheduler + Investigation Ledger writer |-| `fusion` | Python | 8003 | Dedup + ML scoring (LightGBM, IsoForest), alert confidence, entity risk / RBA, correlation narrative projection for API |-| `actions` | Python | 8002 | SOAR with blast-radius gating + ChatOps verification + signed approval URLs |-| `connectors` | Python | — | Connector polling (APScheduler), credential vault, federated query translators |-| `threatintel` | Python | 8005 | TAXII / MISP / OTX / KEV polling + threat-actor attribution engine |-| `ueba` | Python | 8007 | User & Entity Behavior Analytics |-| `honeytokens` | Python | 8008 | Honeytoken lifecycle + webhook alerting |-| `purple-team` | Python | 8006 | Atomic Red Team + Caldera + ATT&CK heatmap + detection drift snapshots |-| `osquery-tls` | Python | 8091 | Native osquery TLS server — enroll nodes, distribute packs, stream FIM/process/network telemetry |-| `osquery-extensions` | Go | — | Custom osquery extensions (AI-powered threat intel table, ML anomaly score table, + 3 more) |-| `slack-bot` | Python | 8009 | ChatOps surface — interactive approvals for high-blast-radius actions, `/aisoc` slash command |-| `mcp` | TypeScript | — (stdio) | Model Context Protocol server exposing 13 AiSOC tools — discovery, deep-dive, action/replay, and warm-tier lake query |-| `ingest` | Go | 8081 | OCSF normalization + Shodan/CVE + Neo4j graph-at-ingest writers |-| `enrichment` | Go | 8080 | IOC enrichment (VT, AbuseIPDB, GreyNoise) |+Two port columns, because conflating them has cost real time. **Container** is+what the process listens on inside the Docker network, and is what another+service must address. **Host** is what `docker-compose.yml` publishes on+`127.0.0.1`, and is what your browser and `curl` use. They differ for most+services.++`litellm` and `ollama` are CORE — see [Architecture](../architecture.md) for+why the gateway without a model behind it could not do AI triage at all.++| Service | Lang | Container | Host | Profile | Role |+|---|---|---|---|---|---|+| `web` | Next.js | 3000 | 3000 | core | SOC console, Investigation Rail, marketing landing |+| `api` | Python · FastAPI | 8000 | 8000 | core | Alerts, cases, RBAC, rules, audit, compliance, federated-search fan-out, ledger reads |+| `ingest` | Go | 8080 · 9090 | 8081 · 9090 | core | OCSF normalization, Kafka publish, inbox webhooks, graph-at-ingest writers |+| `fusion` | Python | 8003 | 8003 | core | Detection engine, promotion, correlation, confidence, narrative, lake writer |+| `agents` | Python · LangGraph | 8084 | 8001 | core | Auto-triage worker, investigation agents, playbook engine, ledger writer |+| `realtime` | Node.js · `ws` | 4000 | 8086 | core | Per-channel WebSocket fan-out, VAPID Web Push |+| `threatintel` | Python | 8005 | 8005 | core | CISA KEV / TAXII / MISP / OTX polling, threat-actor attribution |+| `litellm` | — (image) | 4000 | 4000 | core | LLM gateway; every `aisoc-<role>` alias resolves here and nowhere else |+| `ollama` | — (image) | 11434 | 11434 | core | The local model behind the gateway, pinned at `llama3.2:3b-instruct-q4_K_M` |+| `enrichment` | Go | 8082 | 8080 | full | IOC enrichment fan-out (VT, AbuseIPDB, GreyNoise) |+| `actions` | Python | 8085 | 8002 | full | Governed capability execution and post-action verification |+| `connectors` | Python | 8003 | 8088 | full | Scheduled vendor polling, credential vault, federated query translators |+| `ueba` | Python | 8004 | 8007 | full | User and entity behaviour analytics |+| `honeytokens` | Python | 8005 | 8008 | extras | Honeytoken lifecycle and webhook alerting |+| `purple-team` | Python | 8006 | 8006 | extras | Atomic Red Team, Caldera, ATT&CK heatmap, detection drift |+| `osquery-tls` | Python | 9001 | 8091 | osquery | osquery TLS server — enrol nodes, distribute packs, stream telemetry |+| `slack-bot` | Python | 8089 | 8009 | chatops | ChatOps approvals and the `/aisoc` slash command |+| `osquery-extensions` | Go | — | — | — | Custom osquery tables and decorators |+| `mcp` | TypeScript | — (stdio) | — | — | Model Context Protocol server exposing AiSOC tools to IDE-side agents |## Storage tier@@ -112,7 +126,16 @@ AiSOC/## API surface-The full OpenAPI 3.1 spec lives at [`docs/openapi.yaml`](https://github.com/beenuar/AiSOC/blob/main/docs/openapi.yaml). Endpoint groups are summarised in [Architecture](../architecture.md#api-surface) and the interactive view is at `http://localhost:8000/api/docs` (Swagger) or `/api/redoc` (ReDoc) when you run the local stack.+The full OpenAPI 3.1 spec lives at [`docs/openapi.yaml`](https://github.com/beenuar/AiSOC/blob/main/docs/openapi.yaml). The interactive view is at `http://localhost:8000/api/docs` (Swagger) or `/api/redoc` (ReDoc) when you run the local stack.++## On "unpublished"++`release.yml` builds and packs all eight packages on every tag, so the pipeline+cannot rot. Only the upload is gated, and it is gated on registry credentials+this repository does not hold — an account action, not an engineering one.+Install from source in the meantime. The npm name `aisoc` is free; the PyPI+name `aisoc` belongs to an unrelated project, which is why the Python sandbox+package ships as `aisoc-sandbox`.## Next stopsdiff --git a/apps/docs/docs/contributing/dev-setup.md b/apps/docs/docs/contributing/dev-setup.mdindex 4fda87919..579f998bf 100644--- a/apps/docs/docs/contributing/dev-setup.md+++ b/apps/docs/docs/contributing/dev-setup.md@@ -153,7 +153,7 @@ The other Python services (`fusion`, `actions`, `threatintel`, `ueba`,virtualenv and run `uvicorn app.main:app --reload --port <port>`.The canonical port mapping is in[`docker-compose.yml`](https://github.com/beenuar/AiSOC/blob/main/docker-compose.yml)-and the [Architecture](../architecture#service-responsibilities) page.+and the [service map](../architecture/overview#service-map) page.## 7. Run testsdiff --git a/apps/docs/static/img/console/ai-triage-verdict.png b/apps/docs/static/img/console/ai-triage-verdict.pngnew file mode 100644index 000000000..a7eb96a25Binary files /dev/null and b/apps/docs/static/img/console/ai-triage-verdict.png differdiff --git a/apps/docs/static/img/console/alerts-queue.png b/apps/docs/static/img/console/alerts-queue.pngnew file mode 100644index 000000000..83e7afcdcBinary files /dev/null and b/apps/docs/static/img/console/alerts-queue.png differdiff --git a/apps/docs/static/img/console/soc-operations.png b/apps/docs/static/img/console/soc-operations.pngnew file mode 100644index 000000000..e154f83fdBinary files /dev/null and b/apps/docs/static/img/console/soc-operations.png differdiff --git a/apps/docs/static/img/console/threat-intel-kev.png b/apps/docs/static/img/console/threat-intel-kev.pngnew file mode 100644index 000000000..6afabab22Binary files /dev/null and b/apps/docs/static/img/console/threat-intel-kev.png differdiff --git a/apps/web/public/screenshots/01-alerts-queue.svg b/apps/web/public/screenshots/01-alerts-queue.svgdeleted file mode 100644index 03d1571b4..000000000--- a/apps/web/public/screenshots/01-alerts-queue.svg+++ /dev/null@@ -1,13 +0,0 @@-<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1440 900" role="img" aria-label="Alerts queue screenshot placeholder">- <defs>- <linearGradient id="bg" x1="0" y1="0" x2="0" y2="1">- <stop offset="0" stop-color="#0f172a"/>- <stop offset="1" stop-color="#1e293b"/>- </linearGradient>- </defs>- <rect width="1440" height="900" fill="url(#bg)"/>- <text x="720" y="380" text-anchor="middle" fill="#e2e8f0" font-family="ui-sans-serif, system-ui, sans-serif" font-size="40" font-weight="700">Alerts queue server-anchored SLA countdowns</text>- <text x="720" y="440" text-anchor="middle" fill="#94a3b8" font-family="ui-sans-serif, system-ui, sans-serif" font-size="22">Real screenshot ships with the Phase 2 visual rollup.</text>- <text x="720" y="480" text-anchor="middle" fill="#64748b" font-family="ui-sans-serif, system-ui, sans-serif" font-size="20">Generate locally: pnpm aisoc:demo --screenshots</text>- <text x="720" y="540" text-anchor="middle" fill="#475569" font-family="ui-monospace, monospace" font-size="18">apps/web/public/screenshots/01-alerts-queue.png</text>-</svg>diff --git a/apps/web/public/screenshots/02-investigation-rail.svg b/apps/web/public/screenshots/02-investigation-rail.svgdeleted file mode 100644index 06a692814..000000000--- a/apps/web/public/screenshots/02-investigation-rail.svg+++ /dev/null@@ -1,13 +0,0 @@-<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1440 900" role="img" aria-label="Investigation Rail screenshot placeholder">- <defs>- <linearGradient id="bg" x1="0" y1="0" x2="0" y2="1">- <stop offset="0" stop-color="#0f172a"/>- <stop offset="1" stop-color="#1e293b"/>- </linearGradient>- </defs>- <rect width="1440" height="900" fill="url(#bg)"/>- <text x="720" y="380" text-anchor="middle" fill="#e2e8f0" font-family="ui-sans-serif, system-ui, sans-serif" font-size="40" font-weight="700">Investigation Rail narrative + pivot-path entities</text>- <text x="720" y="440" text-anchor="middle" fill="#94a3b8" font-family="ui-sans-serif, system-ui, sans-serif" font-size="22">Real screenshot ships with the Phase 2 visual rollup.</text>- <text x="720" y="480" text-anchor="middle" fill="#64748b" font-family="ui-sans-serif, system-ui, sans-serif" font-size="20">Generate locally: pnpm aisoc:demo --screenshots</text>- <text x="720" y="540" text-anchor="middle" fill="#475569" font-family="ui-monospace, monospace" font-size="18">apps/web/public/screenshots/02-investigation-rail.png</text>-</svg>diff --git a/apps/web/public/screenshots/03-hunt-workbench.svg b/apps/web/public/screenshots/03-hunt-workbench.svgdeleted file mode 100644index 72446b228..000000000--- a/apps/web/public/screenshots/03-hunt-workbench.svg+++ /dev/null@@ -1,13 +0,0 @@-<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1440 900" role="img" aria-label="Hunt workbench screenshot placeholder">- <defs>- <linearGradient id="bg" x1="0" y1="0" x2="0" y2="1">- <stop offset="0" stop-color="#0f172a"/>- <stop offset="1" stop-color="#1e293b"/>- </linearGradient>- </defs>- <rect width="1440" height="900" fill="url(#bg)"/>- <text x="720" y="380" text-anchor="middle" fill="#e2e8f0" font-family="ui-sans-serif, system-ui, sans-serif" font-size="40" font-weight="700">/hunt workbench NL prompt to ES|QL / SPL / KQL</text>- <text x="720" y="440" text-anchor="middle" fill="#94a3b8" font-family="ui-sans-serif, system-ui, sans-serif" font-size="22">Real screenshot ships with the Phase 2 visual rollup.</text>- <text x="720" y="480" text-anchor="middle" fill="#64748b" font-family="ui-sans-serif, system-ui, sans-serif" font-size="20">Generate locally: pnpm aisoc:demo --screenshots</text>- <text x="720" y="540" text-anchor="middle" fill="#475569" font-family="ui-monospace, monospace" font-size="18">apps/web/public/screenshots/03-hunt-workbench.png</text>-</svg>diff --git a/apps/web/public/screenshots/04-marketplace.svg b/apps/web/public/screenshots/04-marketplace.svgdeleted file mode 100644index a544d5026..000000000--- a/apps/web/public/screenshots/04-marketplace.svg+++ /dev/null@@ -1,13 +0,0 @@-<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1440 900" role="img" aria-label="Marketplace screenshot placeholder">- <defs>- <linearGradient id="bg" x1="0" y1="0" x2="0" y2="1">- <stop offset="0" stop-color="#0f172a"/>- <stop offset="1" stop-color="#1e293b"/>- </linearGradient>- </defs>- <rect width="1440" height="900" fill="url(#bg)"/>- <text x="720" y="380" text-anchor="middle" fill="#e2e8f0" font-family="ui-sans-serif, system-ui, sans-serif" font-size="40" font-weight="700">Marketplace plugins, playbooks, detections</text>- <text x="720" y="440" text-anchor="middle" fill="#94a3b8" font-family="ui-sans-serif, system-ui, sans-serif" font-size="22">Real screenshot ships with the Phase 2 visual rollup.</text>- <text x="720" y="480" text-anchor="middle" fill="#64748b" font-family="ui-sans-serif, system-ui, sans-serif" font-size="20">Generate locally: pnpm aisoc:demo --screenshots</text>- <text x="720" y="540" text-anchor="middle" fill="#475569" font-family="ui-monospace, monospace" font-size="18">apps/web/public/screenshots/04-marketplace.png</text>-</svg>diff --git a/apps/web/public/screenshots/README.md b/apps/web/public/screenshots/README.mdindex 8cc0788dd..b011b4d42 100644--- a/apps/web/public/screenshots/README.md+++ b/apps/web/public/screenshots/README.md@@ -1,48 +1,77 @@-# `apps/web/public/screenshots/` — README hero screenshots+# `apps/web/public/screenshots/` — real console captures-This directory holds the four product screenshots that appear in the-`README.md` hero grid:+Every image here is a screenshot of a running AiSOC deployment. None is a+mockup, a design comp, or a demo-mode capture.-| Slot | Final path | Today (placeholder) | Caption |-|---|---|---|---|-| 1 | `01-alerts-queue.png` | `01-alerts-queue.svg` | Alerts queue — server-anchored SLA countdowns |-| 2 | `02-investigation-rail.png` | `02-investigation-rail.svg` | Investigation Rail — deterministic correlation narrative |-| 3 | `03-hunt-workbench.png` | `03-hunt-workbench.svg` | `/hunt` workbench — NL → ES|QL / SPL / KQL |-| 4 | `04-marketplace.png` | `04-marketplace.svg` | Marketplace — plugins, playbooks, detections |+**Nothing in this directory was taken with `make demo`.** The synthetic demo+dataset exists and is legitimate, but a screenshot of it presented as the+product working on real data would be the exact thing this repository has a+standing rule against. If a capture is ever added from demo mode, it must say+so in the table below and in every caption that uses it.-## Today+## How these were produced-The four `.svg` placeholders ship with the [Phase 2 visuals rollup](../../../../README.md) so the-README image grid never shows a broken image. They are 1440×900 (the-viewport the screenshot spec captures at) and labelled with the path-they will eventually be replaced by.+```bash+cp .env.example .env+make up # CORE: 14 long-running services, no credentials of any kind+make smoke # one real event through the real spine+```-## After the real screenshots land+Then telemetry was pushed through the documented ingest API+(`POST /v1/ingest/batch`) and the console was driven with Playwright at+1440×900, signed in as the administrator `make up` created.-When the next maintainer captures the production screenshots:+One detail, so the record is exact: these were captured immediately before+ingest authentication landed, so the pushes carried no token. That changed what+the *door* requires and nothing about what came through it — normalization,+detection, promotion, correlation and triage are the same code either way.+Re-capturing now would need `make ingest-token` first.-1. Boot the demo:+What is real in these images, stated precisely:- ```bash- pnpm aisoc:demo --no-open --quick- ```+- **The deployment.** Fourteen containers, the images built from this+ repository, Postgres/Kafka/Qdrant/Ollama running locally.+- **The pipeline.** Normalization, detection, promotion, correlation and+ alerting were all performed by the running services. No row was seeded into+ the database.+- **The threat intelligence.** 1,725 CISA Known Exploited Vulnerabilities+ entries, fetched from the public catalogue at capture time. Not ours, not+ synthetic, unmodified.+- **The AI verdict.** Produced by the bundled local `llama3.2:3b-instruct-q4_K_M`+ through the LiteLLM gateway, with real token counts and latency in the+ Investigation Ledger. Not a hosted provider — no key was configured, and+ none has ever been exercised here.+- **The empty and degraded states.** Genuinely empty and genuinely degraded.+ They are as much of the product as the populated ones.-2. Capture the four PNGs (writes the files into this directory):+What is *authored* rather than observed: the security events themselves. They+were written by hand to be representative and pushed through the ingest API.+They are not a real intrusion and no caption may imply they are.- ```bash- pnpm aisoc:demo --screenshots- ```+## The captures-3. Sanity-check the diff and open a PR:+| File | Surface | What is worth seeing |+|---|---|---|+| `dashboard.png` | `/dashboard` | Operations funnel over real counts. `MTTR` reads *not measured · no cases closed* rather than `0`. |+| `alerts-queue.png` | `/alerts` | Twelve alerts the pipeline produced, each attributed to the connector that fed it. |+| `investigation-rail.png` | `/alerts` → row | The rail's Story view. Progression says *no ATT&CK tactic is mapped … that is a gap in the detection's metadata rather than a statement about the activity.* |+| `ai-triage-verdict.png` | `/alerts` → row → Details | A real local-model verdict: `true positive`, confidence 80/100, groundedness *not assessed*, and the model's own rationale verbatim. |+| `threat-intel-kev.png` | `/threat-intel` | 1,725 real KEV entries; the page distinguishes the catalogue size from the page it is showing. |+| `soc-operations.png` | `/dashboards/operations` | Pipeline health against a tenant with no connectors — *"Nothing is feeding the pipeline yet, so an empty alert queue is expected."* |+| `connectors.png` | `/connectors` | The connector surface with nothing connected. |+| `connector-wizard.png` | `/connectors` → Add connector | The catalogue, reporting **84 connector types available** — the generated count, not a hand-typed one. |+| `attack-graph.png` | `/graph` | The entity graph over real ingested telemetry, with Neo4j running. |+| `attack-graph-core-degraded.png` | `/graph` | The same page in CORE, where Neo4j is not running: it names the failure (`API 503 … /api/v1/graph`) instead of drawing an invented graph. |+| `federated-search.png` | `/federated-search` | Honest empty state — no SIEM connected, and it says which four it would query. |+| `playbooks.png` | `/playbooks` | The playbook library. |+| `playbook-editor.png` | `/playbooks/{id}` → Add step | The step palette. Twenty-one tiles, from a twenty-two-member vocabulary: `approval` is withheld because the engine has no pause/resume. |+| `detection-rules.png` | `/detection` | The detection rule surface. |+| `hunt.png` | `/hunt` | The hunt workbench. |+| `cases-empty.png` | `/cases` | No cases, said plainly. |- ```bash- git status -s apps/web/public/screenshots/- git add apps/web/public/screenshots/*.png- ```+## Refreshing them-4. In the same PR, swap the four `.svg` references in `README.md`- to `.png`. Until step 4 lands the README intentionally points at- the placeholders so a fresh clone never shows a broken image.--The Playwright spec that drives the capture lives at-[`apps/web/e2e/screenshots/console.spec.ts`](../../e2e/screenshots/console.spec.ts).+`.github/workflows/console-visuals.yml` can regenerate console tiles on a+schedule, but it drives the **seeded demo stack**. Anything it produces is+demo-mode output and must be labelled as such wherever it appears — which is+why the captures above were taken by hand against a real CORE stack instead.diff --git a/apps/web/public/screenshots/ai-triage-verdict.png b/apps/web/public/screenshots/ai-triage-verdict.pngnew file mode 100644index 000000000..a7eb96a25Binary files /dev/null and b/apps/web/public/screenshots/ai-triage-verdict.png differdiff --git a/apps/web/public/screenshots/alerts-queue.png b/apps/web/public/screenshots/alerts-queue.pngnew file mode 100644index 000000000..83e7afcdcBinary files /dev/null and b/apps/web/public/screenshots/alerts-queue.png differdiff --git a/apps/web/public/screenshots/attack-graph-core-degraded.png b/apps/web/public/screenshots/attack-graph-core-degraded.pngnew file mode 100644index 000000000..03ad2e6e1Binary files /dev/null and b/apps/web/public/screenshots/attack-graph-core-degraded.png differdiff --git a/apps/web/public/screenshots/attack-graph.png b/apps/web/public/screenshots/attack-graph.pngnew file mode 100644index 000000000..b9c39c2d3Binary files /dev/null and b/apps/web/public/screenshots/attack-graph.png differdiff --git a/apps/web/public/screenshots/cases-empty.png b/apps/web/public/screenshots/cases-empty.pngnew file mode 100644index 000000000..80f55b6b7Binary files /dev/null and b/apps/web/public/screenshots/cases-empty.png differdiff --git a/apps/web/public/screenshots/connector-wizard.png b/apps/web/public/screenshots/connector-wizard.pngnew file mode 100644index 000000000..38bbb4d20Binary files /dev/null and b/apps/web/public/screenshots/connector-wizard.png differdiff --git a/apps/web/public/screenshots/connectors.png b/apps/web/public/screenshots/connectors.pngnew file mode 100644index 000000000..2567a3de3Binary files /dev/null and b/apps/web/public/screenshots/connectors.png differdiff --git a/apps/web/public/screenshots/dashboard.png b/apps/web/public/screenshots/dashboard.pngnew file mode 100644index 000000000..e1a2a4e69Binary files /dev/null and b/apps/web/public/screenshots/dashboard.png differdiff --git a/apps/web/public/screenshots/detection-rules.png b/apps/web/public/screenshots/detection-rules.pngnew file mode 100644index 000000000..06a8ea322Binary files /dev/null and b/apps/web/public/screenshots/detection-rules.png differdiff --git a/apps/web/public/screenshots/federated-search.png b/apps/web/public/screenshots/federated-search.pngnew file mode 100644index 000000000..eefbc39caBinary files /dev/null and b/apps/web/public/screenshots/federated-search.png differdiff --git a/apps/web/public/screenshots/hunt.png b/apps/web/public/screenshots/hunt.pngnew file mode 100644index 000000000..48bb41666Binary files /dev/null and b/apps/web/public/screenshots/hunt.png differdiff --git a/apps/web/public/screenshots/investigation-rail.png b/apps/web/public/screenshots/investigation-rail.pngnew file mode 100644index 000000000..7243ee6c1Binary files /dev/null and b/apps/web/public/screenshots/investigation-rail.png differdiff --git a/apps/web/public/screenshots/playbook-editor.png b/apps/web/public/screenshots/playbook-editor.pngnew file mode 100644index 000000000..fa1a8aa9eBinary files /dev/null and b/apps/web/public/screenshots/playbook-editor.png differdiff --git a/apps/web/public/screenshots/playbooks.png b/apps/web/public/screenshots/playbooks.pngnew file mode 100644index 000000000..3c58bb501Binary files /dev/null and b/apps/web/public/screenshots/playbooks.png differdiff --git a/apps/web/public/screenshots/soc-operations.png b/apps/web/public/screenshots/soc-operations.pngnew file mode 100644index 000000000..e154f83fdBinary files /dev/null and b/apps/web/public/screenshots/soc-operations.png differdiff --git a/apps/web/public/screenshots/threat-intel-kev.png b/apps/web/public/screenshots/threat-intel-kev.pngnew file mode 100644index 000000000..6afabab22Binary files /dev/null and b/apps/web/public/screenshots/threat-intel-kev.png differdiff --git a/apps/web/src/components/alerts/InvestigationRail.test.tsx b/apps/web/src/components/alerts/InvestigationRail.test.tsxindex f1873e25e..0a4b97288 100644--- a/apps/web/src/components/alerts/InvestigationRail.test.tsx+++ b/apps/web/src/components/alerts/InvestigationRail.test.tsx@@ -378,6 +378,82 @@ describe('InvestigationRail — header', () => {// ─── Loaded rail — narrative ─────────────────────────────────────────────────+describe('InvestigationRail — automated triage section', () => {+ // `services/agents` triages every fused alert and writes the verdict back+ // onto the row. `normalizeAlert` dropped `ai_score`, `ai_summary` and+ // `triage_groundedness`, so the one output of the AI the product leads with+ // was metered, persisted, and rendered by nothing.+ it('renders the verdict, the confidence and the rationale verbatim', () => {+ swrState.data = buildAlert({+ disposition: 'true_positive',+ aiScore: 0.8,+ aiSummary:+ 'LLM auto-triage verdict: true_positive · LLM confidence: 0.80 · Rationale: mass rename to .locked on the finance share.',+ });+ renderDetails();++ const section = screen.getByText('Automated triage').closest('section')!;+ expect(within(section).getByText('true positive')).toBeInTheDocument();+ expect(within(section).getByText(/confidence 80\/100/)).toBeInTheDocument();+ // Verbatim: the `LLM auto-triage verdict:` prefix is how a reader tells+ // the model's own text from the deterministic fallback's.+ expect(within(section).getByText(/LLM auto-triage verdict: true_positive/)).toBeInTheDocument();+ });++ it('says groundedness was not assessed rather than showing it as zero', () => {+ // The deterministic path never scores groundedness, and the column is+ // nullable for that reason. Rendering null as 0% would read as a verdict+ // that cited nothing real — the opposite of what happened.+ swrState.data = buildAlert({+ disposition: 'likely_benign' as Alert['disposition'],+ aiScore: 0.25,+ aiSummary: 'high-severity keyword match in alert text · hostname present',+ triageGroundedness: null,+ });+ renderDetails();++ const section = screen.getByText('Automated triage').closest('section')!;+ expect(within(section).getByText(/groundedness not assessed/)).toBeInTheDocument();+ expect(within(section).queryByText(/groundedness 0%/)).toBeNull();+ });++ it('renders a real groundedness score when one was assessed', () => {+ swrState.data = buildAlert({ aiScore: 0.9, aiSummary: 'verdict', triageGroundedness: 0.75 });+ renderDetails();++ expect(+ within(screen.getByText('Automated triage').closest('section')!).getByText(/groundedness 75%/),+ ).toBeInTheDocument();+ });++ it('names indicators the verdict cited but the evidence never held', () => {+ swrState.data = buildAlert({+ aiScore: 0.6,+ aiSummary: 'verdict',+ triageUngrounded: ['198.51.100.9', 'evil.example'],+ });+ renderDetails();++ const section = screen.getByText('Automated triage').closest('section')!;+ expect(within(section).getByText(/198\.51\.100\.9, evil\.example/)).toBeInTheDocument();+ });++ it('distinguishes "not triaged yet" from "nothing to show"', () => {+ // Absent and pending are different facts, and only one of them is a+ // reason to go and look at the agents service. Rendering nothing said+ // neither.+ swrState.data = buildAlert({+ disposition: null,+ aiScore: undefined,+ aiSummary: undefined,+ });+ renderDetails();++ expect(screen.getByText('Automated triage')).toBeInTheDocument();+ expect(screen.getByText(/not triaged yet/i)).toBeInTheDocument();+ });+});+describe('InvestigationRail — narrative section', () => {it('renders the narrative when present', () => {swrState.data = buildAlert();diff --git a/apps/web/src/components/alerts/InvestigationRail.tsx b/apps/web/src/components/alerts/InvestigationRail.tsxindex e1dd9b7d2..302eae646 100644--- a/apps/web/src/components/alerts/InvestigationRail.tsx+++ b/apps/web/src/components/alerts/InvestigationRail.tsx@@ -146,6 +146,7 @@ export function InvestigationRail({ alertId, onClose }: InvestigationRailProps)</div>) : (<>+ <AutomatedTriageSection alert={alert} /><NarrativeSection narrative={alert.narrative ?? null} /><RelatedEntitiesSection entities={alert.relatedEntities ?? []} /><MiniTimelineSection events={alert.miniTimeline ?? []} />@@ -350,6 +351,78 @@ function SectionHeader({ title, count }: { title: string; count?: number }) {);}+/**+ * What the auto-triage agent concluded, and how far to trust it.+ *+ * `services/agents` triages every fused alert and writes the verdict back to+ * the row. Nothing rendered it: `normalizeAlert` dropped `ai_score`,+ * `ai_summary` and `triage_groundedness` on the floor, so the product's+ * headline capability produced tokens, cost and a ledger entry that no+ * console surface showed.+ *+ * Three things are deliberate here.+ *+ * The rationale is rendered **verbatim**. The model's own text arrives+ * prefixed `LLM auto-triage verdict:`; the deterministic fallback reads as a+ * description of the signals it matched. Showing it unedited is what lets a+ * reader tell which path answered, and the bundled local model falls back+ * often enough that the distinction matters on a default install.+ *+ * Groundedness renders as **"not assessed"** when the column is null, never+ * as 0. The deterministic path does not score it, and 0 would read as a+ * verdict citing nothing real.+ *+ * An alert that has not been triaged yet says so, rather than rendering+ * nothing — absent and pending are different facts, and only one of them is+ * a reason to go and look at the agents service.+ */+function AutomatedTriageSection({ alert }: { alert: Alert }) {+ const verdict = alert.disposition ?? null;+ const rationale = (alert.aiSummary ?? '').trim();+ const hasRun = Boolean(verdict || rationale || typeof alert.aiScore === 'number');++ return (+ <section className="px-4 py-4 border-b border-gray-800/40">+ <SectionHeader title="Automated triage" />+ {!hasRun ? (+ <p className="text-xs text-gray-500">+ Not triaged yet. Every fused alert is queued for the triage agent; if this stays empty,+ check the agents service.+ </p>+ ) : (+ <div className="space-y-2">+ <div className="flex items-center gap-2 flex-wrap">+ {verdict && (+ <span className="text-xs px-2 py-0.5 rounded border border-gray-700 bg-gray-800/60 text-gray-200">+ {verdict.replace(/_/g, ' ')}+ </span>+ )}+ {typeof alert.aiScore === 'number' && (+ <span className="text-xs text-gray-400">+ confidence {Math.round(alert.aiScore * 100)}/100+ </span>+ )}+ <span className="text-xs text-gray-500">+ groundedness{' '}+ {typeof alert.triageGroundedness === 'number'+ ? `${Math.round(alert.triageGroundedness * 100)}%`+ : 'not assessed'}+ </span>+ </div>+ {rationale && (+ <p className="text-xs text-gray-300 leading-relaxed whitespace-pre-wrap">{rationale}</p>+ )}+ {alert.triageUngrounded && alert.triageUngrounded.length > 0 && (+ <p className="text-xs text-amber-400">+ Cited but not present in the evidence: {alert.triageUngrounded.join(', ')}+ </p>+ )}+ </div>+ )}+ </section>+ );+}+function NarrativeSection({ narrative }: { narrative: string | null }) {if (!narrative || !narrative.trim()) return null;// `build_narrative` emits a markdown-light dialect. Preserving newlines in adiff --git a/apps/web/src/components/threat-intel/ThreatIntelView.tsx b/apps/web/src/components/threat-intel/ThreatIntelView.tsxindex 3d2eeb5dc..9a5bb4609 100644--- a/apps/web/src/components/threat-intel/ThreatIntelView.tsx+++ b/apps/web/src/components/threat-intel/ThreatIntelView.tsx@@ -245,6 +245,10 @@ export function ThreatIntelView() {// "3 Added Today" counter on a tenant that had never ingested one.const allIndicators = data?.indicators ?? [];+ // The store's count, not this page's length. Falls back to the page length+ // only when the API omits the field, which is the pre-`shown` contract.+ const totalCollected = data?.total ?? allIndicators.length;+const indicators = allIndicators.filter((ioc) => {if (typeFilter !== 'all' && ioc.type !== typeFilter) return false;if (@@ -275,13 +279,22 @@ export function ThreatIntelView() {</p></div>- {/* Stats */}+ {/* Stats+ *+ * Every figure here is either the store's own count or a count of the+ * rows on screen, and each card says which. It previously published+ * neither: "Total IOCs" was the length of one page — 100, against 1,725+ * collected — and "Added Today" was the literal `3`, hard-coded, on+ * every deployment including ones that had never ingested an indicator.+ * The comment above `allIndicators` describes removing the invented IOC+ * list for exactly that reason; the counter the same paragraph names+ * survived it. */}<div className="grid grid-cols-4 gap-3">{[- { label: 'Total IOCs', value: allIndicators.length, color: 'text-blue-400' },- { label: 'Malicious', value: allIndicators.filter(i => i.malicious).length, color: 'text-red-400' },- { label: 'High Confidence', value: allIndicators.filter(i => i.confidence >= 80).length, color: 'text-orange-400' },- { label: 'Added Today', value: 3, color: 'text-green-400' },+ { label: 'Indicators collected', value: totalCollected, color: 'text-blue-400' },+ { label: 'Shown below', value: allIndicators.length, color: 'text-gray-200' },+ { label: 'Malicious (of shown)', value: allIndicators.filter(i => i.malicious).length, color: 'text-red-400' },+ { label: 'High confidence (of shown)', value: allIndicators.filter(i => i.confidence >= 80).length, color: 'text-orange-400' },].map((stat) => (<div key={stat.label} className="bg-gray-900/60 border border-gray-800/60 rounded-xl p-4"><p className={clsx('text-2xl font-bold mb-1', stat.color)}>{stat.value}</p>@@ -330,7 +343,11 @@ export function ThreatIntelView() {<div className="bg-gray-900/60 border border-gray-800/60 rounded-xl p-4"><div className="flex items-center justify-between mb-3"><h3 className="text-sm font-medium text-gray-300">Indicators of Compromise</h3>- <span className="text-xs text-gray-500">{indicators.length} indicators</span>+ <span className="text-xs text-gray-500">+ {totalCollected > indicators.length+ ? `${indicators.length.toLocaleString()} of ${totalCollected.toLocaleString()} indicators`+ : `${indicators.length.toLocaleString()} indicators`}+ </span></div>{indicators.length === 0 ? (// WS-F5 — TI feeds rarely have a true "no IOCs ever" state indiff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.tsindex 5e40cdfe8..bd372a819 100644--- a/apps/web/src/lib/api.ts+++ b/apps/web/src/lib/api.ts@@ -674,6 +674,29 @@ export interface Alert {relatedEntities?: RelatedEntity[];miniTimeline?: MiniTimelineEvent[];recommendedActions?: RecommendedAction[];+ // ─── Automated triage ────────────────────────────────────────────────────+ //+ // `services/agents` auto-triages every fused alert and writes the verdict+ // back onto the row (`ledger.persist_auto_triage`). The API has returned+ // `ai_score` / `ai_summary` / `triage_groundedness` on every alert since,+ // and this mapper dropped all three — so the one output of the AI the+ // product leads with was metered, persisted, and rendered nowhere.+ /** 0–1 confidence the triage run reported. Distinct from `confidenceScore`,+ * which is fusion's detection confidence and has nothing to do with the+ * agent. */+ aiScore?: number;+ /** The verdict's rationale, as written by whichever path produced it. The+ * model's own text is prefixed `LLM auto-triage verdict:`; the+ * deterministic fallback reads as a description of the signals it matched.+ * Rendered verbatim so the reader can tell them apart. */+ aiSummary?: string;+ /** Fraction of the verdict's cited indicators that appear in the evidence.+ * `null`/absent means *not assessed* — the deterministic path never scores+ * groundedness, and rendering that as 0 would read as "wholly unsupported".+ */+ triageGroundedness?: number | null;+ /** Indicators the verdict cited that the evidence never contained. */+ triageUngrounded?: string[];}/**@@ -894,6 +917,16 @@ function normalizeAlert(raw: unknown): Alert {relatedEntities,miniTimeline,recommendedActions,+ aiScore: pickNum('ai_score', 'aiScore'),+ aiSummary: pickStr('ai_summary', 'aiSummary'),+ // `?? null` and not `?? undefined`: the column is nullable because NULL+ // means "not assessed", and collapsing that into absent loses the+ // distinction the schema comment exists to preserve.+ triageGroundedness:+ typeof (r.triage_groundedness ?? r.triageGroundedness) === 'number'+ ? ((r.triage_groundedness ?? r.triageGroundedness) as number)+ : null,+ triageUngrounded: pickArr<string>('triage_ungrounded', 'triageUngrounded'),};}@@ -3003,8 +3036,11 @@ export const threatIntelApi = {body: JSON.stringify({ iocs }),}),+ // `total` is the store's count of indicators in scope; `shown` is how many+ // this response carries. They are different numbers and the console renders+ // both — it used to render the page length as the catalogue size.list: (filters: { type?: IndicatorType; tag?: string; q?: string } = {}) =>- request<{ indicators: ThreatIndicator[]; total: number }>(+ request<{ indicators: ThreatIndicator[]; total: number; shown?: number; bounded?: boolean }>('/api/v1/threat-intel/indicators',{ params: filters as Record<string, string> },),diff --git a/docs/architecture/README.md b/docs/architecture/README.mdindex d528f7343..1e8d24a51 100644--- a/docs/architecture/README.md+++ b/docs/architecture/README.md@@ -113,15 +113,17 @@ flowchart LRagt["services/agents<br/>Python · :8084"]web["apps/web<br/>Next.js · :3000"]llm["litellm<br/>LLM gateway · :4000"]+ oll["ollama<br/>llama3.2:3b · :11434"]+ ti["services/threatintel<br/>Python · :8005"]+ qd[("Qdrant<br/>IOC + actor vectors")]endsubgraph full ["full profile — optional"]ch[("ClickHouse<br/>event lake")]neo[("Neo4j<br/>entity graph")]- qd[("Qdrant<br/>embeddings")]+ os[("OpenSearch<br/>full-text IOC search")]enr["services/enrichment"]conn["services/connectors"]- ti["services/threatintel"]endconn -.->|"polls vendors"| ing@@ -137,10 +139,13 @@ flowchart LRagt --> pgagt -->|"aisoc-<role> alias"| llmapi --> llm+ llm --> ollapi --> pgapi -.-> chapi -.-> neo- ti -.-> qd+ ti -->|"CISA KEV"| qd+ ti -.-> os+ api --> tiweb --> apiweb --> rt```@@ -151,16 +156,17 @@ flowchart LR```mermaidflowchart TB- subgraph req ["Required"]+ subgraph req ["CORE"]pg["PostgreSQL<br/><br/>alerts · incidents · cases<br/>users · tenants · detection rules<br/>audit log · investigation ledger"]rds["Redis<br/><br/>correlation windows<br/>dedup keys<br/>investigation run state"]- kaf["Kafka<br/><br/>aisoc.raw_events<br/>aisoc.alerts.fused"]+ kaf["Kafka<br/><br/>aisoc.raw_events<br/>aisoc.alerts.fused<br/>aisoc.alerts.dlq"]+ qd["Qdrant<br/><br/>IOC and actor vectors<br/>backs the Threat Intelligence page"]endsubgraph opt ["Optional (full profile)"]ch["ClickHouse<br/><br/>every normalized event<br/>backs /lake/sql and hunt"]neo["Neo4j<br/><br/>entity graph<br/>blast radius"]- qd["Qdrant<br/><br/>IOC and actor embeddings"]+ os["OpenSearch<br/><br/>full-text IOC and actor search"]end```@@ -172,14 +178,17 @@ flowchart TB| **Redis** | Correlation needs expiring keys at event rate; TTL semantics and throughput are the point. | No correlation; every alert becomes its own incident. || **Kafka** | Decouples producers from consumers and lets a slow consumer fall behind without dropping events or blocking ingest. | No spine; fusion, realtime and agents would each need a direct call from ingest. || **ClickHouse** | Columnar scans over hundreds of millions of events. Postgres cannot do this at cost. | No event lake, no hunting over raw telemetry. **Alerting is unaffected.** |-| **Neo4j** | Multi-hop traversal ("what else did this identity touch") is a join explosion in SQL. | No graph context or blast radius. Alerting unaffected. |-| **Qdrant** | Vector similarity for IOC and actor matching. | No semantic threat-intel matching. |-| **OpenSearch** | Full-text and structured search over the threat-intel corpus — `threatintel-iocs` and `threatintel-actors`. | `services/threatintel` cannot start: it indexes into OpenSearch in its lifespan, without a fallback. |+| **Neo4j** | Multi-hop traversal ("what else did this identity touch") is a join explosion in SQL. | No graph context or blast radius. `/graph` reports the failure rather than drawing an invented graph. Alerting unaffected. |+| **Qdrant** | Vector similarity for IOC and actor matching, and the read path the console's Threat Intelligence page is served from. | No threat-intel page. It is in CORE for that reason, and because it is by a wide margin the cheapest of the four stores — 245 MB of image, ~79 MiB resident. |+| **OpenSearch** | Full-text and structured search over the threat-intel corpus — `threatintel-iocs` and `threatintel-actors`. | Feeds still write to Qdrant and the page still works; full-text IOC search is unavailable and `services/threatintel` logs that it is. |**Who reads OpenSearch, precisely:** only `services/threatintel`. This pagepreviously said nothing read it at all, which came from checking`services/api` — which genuinely holds no OpenSearch client — and stopping-there. See the correction in+there. It also said `services/threatintel` *could not start* without it; that+was true when the lifespan called `os_store.initialize()` with no `try`, and+is no longer — both that call and the pipeline's bulk index are best-effort,+which is what let the service move into CORE. See the corrections in[the reality audit](../audit/REPOSITORY_REALITY.md).---diff --git a/docs/audit/REPOSITORY_REALITY.md b/docs/audit/REPOSITORY_REALITY.mdindex cae0e6317..8365020ec 100644--- a/docs/audit/REPOSITORY_REALITY.md+++ b/docs/audit/REPOSITORY_REALITY.md@@ -218,7 +218,7 @@ Stated rather than guessed:## Reproducing this audit```bash-make up # CORE profile: 10 services+make up # CORE profile: 14 long-running services (+ a one-shot model pull)make doctor # every dependency probed, not just "running"make smoke # one real event through the real pipelinemake stats # recount every published figure from the treediff --git a/docs/openapi.yaml b/docs/openapi.yamlindex 724448142..ae1a20cd4 100644--- a/docs/openapi.yaml+++ b/docs/openapi.yaml@@ -20238,6 +20238,14 @@ components:total:type: integertitle: Total+ shown:+ type: integer+ title: Shown+ default: 0+ bounded:+ type: boolean+ title: Bounded+ default: falsesource:type: stringtitle: Sourcediff --git a/docs/testing/CLEAN_INSTALL.md b/docs/testing/CLEAN_INSTALL.mdindex df6852928..6eaad9c5b 100644--- a/docs/testing/CLEAN_INSTALL.md+++ b/docs/testing/CLEAN_INSTALL.md@@ -28,7 +28,7 @@ make smoke| Stage | Result ||---|---|| Clone | pass |-| `make up` — CORE profile, 10 services | pass |+| `make up` — CORE profile, 14 long-running services | pass || Postgres accepting connections, 92 tables | pass || Kafka broker answering admin requests | pass || `api`, `ingest-worker`, `fusion` healthy | pass |diff --git a/screenshots/01-landing.png b/screenshots/01-landing.pngdeleted file mode 100644index f7b0a1c89..000000000Binary files a/screenshots/01-landing.png and /dev/null differdiff --git a/screenshots/02-signup.png b/screenshots/02-signup.pngdeleted file mode 100644index 998749909..000000000Binary files a/screenshots/02-signup.png and /dev/null differdiff --git a/screenshots/redesign/about-full.png b/screenshots/redesign/about-full.pngdeleted file mode 100644index 35e2dcf1e..000000000Binary files a/screenshots/redesign/about-full.png and /dev/null differdiff --git a/screenshots/redesign/app-billing.png b/screenshots/redesign/app-billing.pngdeleted file mode 100644index 472410bd1..000000000Binary files a/screenshots/redesign/app-billing.png and /dev/null differdiff --git a/screenshots/redesign/app-case-detail.png b/screenshots/redesign/app-case-detail.pngdeleted file mode 100644index 651fbb23e..000000000Binary files a/screenshots/redesign/app-case-detail.png and /dev/null differdiff --git a/screenshots/redesign/app-case-investigation.png b/screenshots/redesign/app-case-investigation.pngdeleted file mode 100644index 5cf4e02e2..000000000Binary files a/screenshots/redesign/app-case-investigation.png and /dev/null differdiff --git a/screenshots/redesign/app-case-ledger.png b/screenshots/redesign/app-case-ledger.pngdeleted file mode 100644index 382284740..000000000Binary files a/screenshots/redesign/app-case-ledger.png and /dev/null differdiff --git a/screenshots/redesign/app-cases.png b/screenshots/redesign/app-cases.pngdeleted file mode 100644index 1583cd724..000000000Binary files a/screenshots/redesign/app-cases.png and /dev/null differdiff --git a/screenshots/redesign/app-dashboard.png b/screenshots/redesign/app-dashboard.pngdeleted file mode 100644index e45c135ee..000000000Binary files a/screenshots/redesign/app-dashboard.png and /dev/null differdiff --git a/screenshots/redesign/hero.png b/screenshots/redesign/hero.pngdeleted file mode 100644index 50fa3ecf5..000000000Binary files a/screenshots/redesign/hero.png and /dev/null differdiff --git a/screenshots/redesign/home-desktop.png b/screenshots/redesign/home-desktop.pngdeleted file mode 100644index 8cbb2733c..000000000Binary files a/screenshots/redesign/home-desktop.png and /dev/null differdiff --git a/screenshots/redesign/home-full.png b/screenshots/redesign/home-full.pngdeleted file mode 100644index 8cbb2733c..000000000Binary files a/screenshots/redesign/home-full.png and /dev/null differdiff --git a/screenshots/redesign/home-mobile.png b/screenshots/redesign/home-mobile.pngdeleted file mode 100644index 41c805466..000000000Binary files a/screenshots/redesign/home-mobile.png and /dev/null differdiff --git a/screenshots/redesign/home-tablet.png b/screenshots/redesign/home-tablet.pngdeleted file mode 100644index abbb5705d..000000000Binary files a/screenshots/redesign/home-tablet.png and /dev/null differdiff --git a/screenshots/redesign/integrations-fold.png b/screenshots/redesign/integrations-fold.pngdeleted file mode 100644index 697f2e8f5..000000000Binary files a/screenshots/redesign/integrations-fold.png and /dev/null differdiff --git a/screenshots/redesign/integrations.png b/screenshots/redesign/integrations.pngdeleted file mode 100644index 491704f38..000000000Binary files a/screenshots/redesign/integrations.png and /dev/null differdiff --git a/screenshots/redesign/login.png b/screenshots/redesign/login.pngdeleted file mode 100644index 657f63162..000000000Binary files a/screenshots/redesign/login.png and /dev/null differdiff --git a/screenshots/redesign/pricing-full.png b/screenshots/redesign/pricing-full.pngdeleted file mode 100644index 3032c41f6..000000000Binary files a/screenshots/redesign/pricing-full.png and /dev/null differdiff --git a/screenshots/redesign/stripe-checkout.png b/screenshots/redesign/stripe-checkout.pngdeleted file mode 100644index 391ed30ff..000000000Binary files a/screenshots/redesign/stripe-checkout.png and /dev/null differdiff --git a/screenshots/redesign/tryaisoc-dashboard.png b/screenshots/redesign/tryaisoc-dashboard.pngdeleted file mode 100644index 8f6c000f2..000000000Binary files a/screenshots/redesign/tryaisoc-dashboard.png and /dev/null differdiff --git a/screenshots/redesign/tryaisoc-live.png b/screenshots/redesign/tryaisoc-live.pngdeleted file mode 100644index 5284bc2f1..000000000Binary files a/screenshots/redesign/tryaisoc-live.png and /dev/null differdiff --git a/screenshots/resweep/audit-final.png b/screenshots/resweep/audit-final.pngdeleted file mode 100644index 8f898929d..000000000Binary files a/screenshots/resweep/audit-final.png and /dev/null differdiff --git a/screenshots/resweep/audit.png b/screenshots/resweep/audit.pngdeleted file mode 100644index cd7723f56..000000000Binary files a/screenshots/resweep/audit.png and /dev/null differdiff --git a/screenshots/resweep/coverage-advisor.png b/screenshots/resweep/coverage-advisor.pngdeleted file mode 100644index 44231df59..000000000Binary files a/screenshots/resweep/coverage-advisor.png and /dev/null differdiff --git a/screenshots/resweep/detection.png b/screenshots/resweep/detection.pngdeleted file mode 100644index 949e5d03f..000000000Binary files a/screenshots/resweep/detection.png and /dev/null differdiff --git a/screenshots/resweep/easm.png b/screenshots/resweep/easm.pngdeleted file mode 100644index 1815dcae4..000000000Binary files a/screenshots/resweep/easm.png and /dev/null differdiff --git a/screenshots/resweep/graph.png b/screenshots/resweep/graph.pngdeleted file mode 100644index c20571e74..000000000Binary files a/screenshots/resweep/graph.png and /dev/null differdiff --git a/screenshots/resweep/shifts.png b/screenshots/resweep/shifts.pngdeleted file mode 100644index 45f26191e..000000000Binary files a/screenshots/resweep/shifts.png and /dev/null differdiff --git a/screenshots/resweep/threat-intel.png b/screenshots/resweep/threat-intel.pngdeleted file mode 100644index b68efc482..000000000Binary files a/screenshots/resweep/threat-intel.png and /dev/null differdiff --git a/screenshots/sweep/alerts.png b/screenshots/sweep/alerts.pngdeleted file mode 100644index 0bc4fe012..000000000Binary files a/screenshots/sweep/alerts.png and /dev/null differdiff --git a/screenshots/sweep/audit.png b/screenshots/sweep/audit.pngdeleted file mode 100644index 61ba8f256..000000000Binary files a/screenshots/sweep/audit.png and /dev/null differdiff --git a/screenshots/sweep/cases.png b/screenshots/sweep/cases.pngdeleted file mode 100644index 87f6d3eba..000000000Binary files a/screenshots/sweep/cases.png and /dev/null differdiff --git a/screenshots/sweep/connectors.png b/screenshots/sweep/connectors.pngdeleted file mode 100644index 98e4831a3..000000000Binary files a/screenshots/sweep/connectors.png and /dev/null differdiff --git a/screenshots/sweep/copilot.png b/screenshots/sweep/copilot.pngdeleted file mode 100644index 8ad4e11c6..000000000Binary files a/screenshots/sweep/copilot.png and /dev/null differdiff --git a/screenshots/sweep/costs.png b/screenshots/sweep/costs.pngdeleted file mode 100644index 88b8fb142..000000000Binary files a/screenshots/sweep/costs.png and /dev/null differdiff --git a/screenshots/sweep/coverage-advisor.png b/screenshots/sweep/coverage-advisor.pngdeleted file mode 100644index 34e33789a..000000000Binary files a/screenshots/sweep/coverage-advisor.png and /dev/null differdiff --git a/screenshots/sweep/dashboard.png b/screenshots/sweep/dashboard.pngdeleted file mode 100644index e530af0f5..000000000Binary files a/screenshots/sweep/dashboard.png and /dev/null differdiff --git a/screenshots/sweep/dashboards-soc-insights.png b/screenshots/sweep/dashboards-soc-insights.pngdeleted file mode 100644index 4414eefeb..000000000Binary files a/screenshots/sweep/dashboards-soc-insights.png and /dev/null differdiff --git a/screenshots/sweep/detection-catalog.png b/screenshots/sweep/detection-catalog.pngdeleted file mode 100644index aaf61e34a..000000000Binary files a/screenshots/sweep/detection-catalog.png and /dev/null differdiff --git a/screenshots/sweep/detection-coverage.png b/screenshots/sweep/detection-coverage.pngdeleted file mode 100644index 1eb3c9323..000000000Binary files a/screenshots/sweep/detection-coverage.png and /dev/null differdiff --git a/screenshots/sweep/detection-tuning.png b/screenshots/sweep/detection-tuning.pngdeleted file mode 100644index e83cc7ed5..000000000Binary files a/screenshots/sweep/detection-tuning.png and /dev/null differdiff --git a/screenshots/sweep/detection.png b/screenshots/sweep/detection.pngdeleted file mode 100644index afde68b70..000000000Binary files a/screenshots/sweep/detection.png and /dev/null differdiff --git a/screenshots/sweep/easm.png b/screenshots/sweep/easm.pngdeleted file mode 100644index a92ecfe6d..000000000Binary files a/screenshots/sweep/easm.png and /dev/null differdiff --git a/screenshots/sweep/explore.png b/screenshots/sweep/explore.pngdeleted file mode 100644index 9627fdfe7..000000000Binary files a/screenshots/sweep/explore.png and /dev/null differdiff --git a/screenshots/sweep/graph.png b/screenshots/sweep/graph.pngdeleted file mode 100644index 850d974db..000000000Binary files a/screenshots/sweep/graph.png and /dev/null differdiff --git a/screenshots/sweep/hunt.png b/screenshots/sweep/hunt.pngdeleted file mode 100644index 57d79fac1..000000000Binary files a/screenshots/sweep/hunt.png and /dev/null differdiff --git a/screenshots/sweep/marketplace.png b/screenshots/sweep/marketplace.pngdeleted file mode 100644index 895454adc..000000000Binary files a/screenshots/sweep/marketplace.png and /dev/null differdiff --git a/screenshots/sweep/playbooks.png b/screenshots/sweep/playbooks.pngdeleted file mode 100644index b410e9712..000000000Binary files a/screenshots/sweep/playbooks.png and /dev/null differdiff --git a/screenshots/sweep/queue.png b/screenshots/sweep/queue.pngdeleted file mode 100644index 7170d6550..000000000Binary files a/screenshots/sweep/queue.png and /dev/null differdiff --git a/screenshots/sweep/settings.png b/screenshots/sweep/settings.pngdeleted file mode 100644index a9dab27c6..000000000Binary files a/screenshots/sweep/settings.png and /dev/null differdiff --git a/screenshots/sweep/shifts.png b/screenshots/sweep/shifts.pngdeleted file mode 100644index 1436f087f..000000000Binary files a/screenshots/sweep/shifts.png and /dev/null differdiff --git a/screenshots/sweep/threat-intel.png b/screenshots/sweep/threat-intel.pngdeleted file mode 100644index 09859a09b..000000000Binary files a/screenshots/sweep/threat-intel.png and /dev/null differdiff --git a/services/api/app/api/v1/endpoints/threat_intel.py b/services/api/app/api/v1/endpoints/threat_intel.pyindex 8f09fbc02..05f299798 100644--- a/services/api/app/api/v1/endpoints/threat_intel.py+++ b/services/api/app/api/v1/endpoints/threat_intel.py@@ -137,7 +137,15 @@ class FeedIndicatorsResponse(BaseModel):"""indicators: list[dict[str, Any]]+ #: Indicators in the caller's scope, per the upstream store's own count.total: int+ #: Indicators carried by this response. ``total`` is the catalogue,+ #: ``shown`` is the page — the console renders "N of M" and used to render+ #: the page size as the catalogue.+ shown: int = 0+ #: Upstream narrowed inside a bounded scan window, so ``shown`` is a lower+ #: bound on the matches that exist.+ bounded: bool = Falsesource: strdegraded: bool = Falsereason: str = ""@@ -214,6 +222,8 @@ async def list_feed_indicators(return FeedIndicatorsResponse(indicators=indicators,total=int(body.get("total") or len(indicators)),+ shown=int(body.get("shown") or len(indicators)),+ bounded=bool(body.get("bounded")),source=str(body.get("source") or "threatintel"),)diff --git a/services/fusion/app/services/provenance.py b/services/fusion/app/services/provenance.pyindex 7d7d03a6e..d228eb01c 100644--- a/services/fusion/app/services/provenance.py+++ b/services/fusion/app/services/provenance.py@@ -15,6 +15,7 @@ connector *instance* id is what downstream resolution actually keys on.from __future__ import annotations+import reimport uuidfrom typing import Any@@ -28,6 +29,16 @@ def _parse_uuid(value: Any) -> uuid.UUID | None:return None+def _says_the_same_thing(whole: str, part: str) -> bool:+ """Whether ``part`` already appears inside ``whole`` as whole words.++ Word boundaries, not a plain substring test: "AWS" must not be swallowed+ by a product called "Lawsuit Monitor", and the join exists to name a+ vendor, not to pattern-match one.+ """+ return re.search(rf"(?<!\w){re.escape(part)}(?!\w)", whole, re.IGNORECASE) is not None++def product_label(ocsf: dict[str, Any]) -> str | None:"""Vendor + product, deduplicated.@@ -35,6 +46,16 @@ def product_label(ocsf: dict[str, Any]) -> str | None:naive join produced `connector_type = "crowdstrike crowdstrike"` on everyalert. Verified on a live stack.+ Exact equality was not enough. Four of the ten profiles in+ `services/ingest/internal/normalizer/normalizer.go` name the vendor inside+ the product — `Okta` / `Okta System Log`, `Splunk` / `Splunk Enterprise`,+ `Kubernetes` / `Kubernetes Audit`, `Email` / `Forwarded Email` — so the+ alert queue, the Investigation Rail and every entity chip read "Okta Okta+ System Log". Observed on a live CORE stack against real pushed telemetry.+ A part that another part already says is dropped; the longer one wins,+ which leaves "CrowdStrike Falcon" and "AWS Security Hub" untouched because+ neither names the other.+This is the single implementation. `promoter._source()` delegates here;there used to be two copies of the join and only one of them was fixed,which is why the doubled label survived the first repair.@@ -51,7 +72,12 @@ def product_label(ocsf: dict[str, Any]) -> str | None:if any(cleaned.lower() == seen.lower() for seen in parts):continueparts.append(cleaned)- return " ".join(parts) or None+ kept = [+ part+ for index, part in enumerate(parts)+ if not any(len(other) > len(part) and _says_the_same_thing(other, part) for other in parts[:index] + parts[index + 1 :])+ ]+ return " ".join(kept) or None#: Retained so existing imports keep working.diff --git a/services/fusion/tests/test_provenance_product_label.py b/services/fusion/tests/test_provenance_product_label.pynew file mode 100644index 000000000..0e2f6d77e--- /dev/null+++ b/services/fusion/tests/test_provenance_product_label.py@@ -0,0 +1,94 @@+"""The vendor/product join must not say the vendor twice.++`connector_type` is what the alert queue, the Investigation Rail's connector+chip and `alert.source` all render, so a doubled label is visible on every+surface that names where an alert came from.++The interesting half of this file is `test_no_shipped_profile_doubles_its_vendor`:+it reads the vendor/product pairs out of the Go normalizer rather than+restating them here. A test that carries its own copy of the input only proves+the function is self-consistent — the previous exact-equality dedup passed+every hand-written case while four of the ten shipped profiles rendered+doubled, because no test ever asked the normalizer what it actually declares.+"""++from __future__ import annotations++import re+from pathlib import Path++import pytest+from app.services.provenance import product_label++NORMALIZER = Path(__file__).resolve().parents[3] / "services" / "ingest" / "internal" / "normalizer" / "normalizer.go"++#: `OcsfProduct{Name: "Okta System Log", VendorName: "Okta"}`+_PRODUCT_DECL = re.compile(r'OcsfProduct\{Name:\s*"(?P<name>[^"]+)",\s*VendorName:\s*"(?P<vendor>[^"]+)"\}')+++def _label(vendor: str | None, name: str | None) -> str | None:+ product: dict[str, str] = {}+ if vendor is not None:+ product["vendor_name"] = vendor+ if name is not None:+ product["name"] = name+ return product_label({"metadata": {"product": product}})+++ ("vendor", "name", "expected"),+ [+ # The vendor names itself inside the product. Observed doubled on a+ # live CORE stack against telemetry pushed through the ingest API.+ ("Okta", "Okta System Log", "Okta System Log"),+ ("Splunk", "Splunk Enterprise", "Splunk Enterprise"),+ ("Kubernetes", "Kubernetes Audit", "Kubernetes Audit"),+ # Containment, not prefixing: the vendor is the *last* word here.+ ("Email", "Forwarded Email", "Forwarded Email"),+ # Exact duplicates — the case the original dedup was written for.+ ("Splunk", "Splunk", "Splunk"),+ ("crowdstrike", "CrowdStrike", "crowdstrike"),+ # Neither part names the other: both survive, in vendor-first order.+ ("CrowdStrike", "Falcon", "CrowdStrike Falcon"),+ ("Microsoft", "Sentinel", "Microsoft Sentinel"),+ ("AWS", "Security Hub", "AWS Security Hub"),+ ("AiSOC", "AI Runtime", "AiSOC AI Runtime"),+ # One side missing is not a reason to drop the other.+ ("CrowdStrike", None, "CrowdStrike"),+ (None, "Falcon", "Falcon"),+ ("", " ", None),+ ],+)+def test_product_label(vendor: str | None, name: str | None, expected: str | None) -> None:+ assert _label(vendor, name) == expected+++def test_substring_match_respects_word_boundaries() -> None:+ """A vendor that merely appears inside a longer word is not swallowed."""+ assert _label("AWS", "Lawsuit Monitor") == "AWS Lawsuit Monitor"+++def test_missing_or_malformed_metadata_yields_none() -> None:+ assert product_label({}) is None+ assert product_label({"metadata": {}}) is None+ assert product_label({"metadata": {"product": "Falcon"}}) is None++[email protected](not NORMALIZER.is_file(), reason="ingest normalizer not present in this checkout")+def test_no_shipped_profile_doubles_its_vendor() -> None:+ """Every vendor/product pair the normalizer declares must render once.++ Runs in the direction that drifts: adding a profile whose product name+ repeats its vendor fails here rather than being discovered in a screenshot.+ """+ declarations = _PRODUCT_DECL.findall(NORMALIZER.read_text(encoding="utf-8"))+ assert declarations, f"no OcsfProduct declarations found in {NORMALIZER}"++ doubled: list[tuple[str, str, str]] = []+ for name, vendor in declarations:+ label = _label(vendor, name) or ""+ words = [word.lower() for word in label.split()]+ if len(words) != len(set(words)):+ doubled.append((vendor, name, label))++ assert not doubled, "vendor repeated in the rendered label: " + ", ".join(f"{v!r}+{n!r} -> {label!r}" for v, n, label in doubled)diff --git a/services/threatintel/app/api/indicators.py b/services/threatintel/app/api/indicators.pyindex 11067cef2..7b970f971 100644--- a/services/threatintel/app/api/indicators.py+++ b/services/threatintel/app/api/indicators.py@@ -75,7 +75,26 @@ class Indicator(BaseModel):class IndicatorListResponse(BaseModel):indicators: list[Indicator]+ #: How many indicators are in the caller's scope, from the store's own+ #: count — not how many this response carries, and not how many the scan+ #: window happened to reach.+ #:+ #: It used to be ``len(indicators)`` after post-filtering, i.e. the size of+ #: the bounded scan window. With 1,725 KEV entries collected, the store+ #: held 1,725, this field said 400 and the console's headline card said+ #: 100 — three different answers to one question, the two published ones+ #: both wrong, and the page presenting the smallest as the corpus.total: int+ #: How many indicators this response carries. Distinct from ``total`` on+ #: purpose: the console needs to say "100 of 1,725" rather than implying+ #: a page is the catalogue.+ shown: int = 0+ #: True when ``type`` / ``tag`` / ``q`` narrowed the result, in which case+ #: the match was made inside the bounded scan window rather than across+ #: the whole collection — so ``shown`` is a lower bound on the matches+ #: that exist. Published rather than hidden because a search that silently+ #: reads part of the corpus is the kind of thing that gets believed.+ bounded: bool = False#: Which store answered. The console has no other way to tell "no#: indicators collected yet" from "the store is not there", and the two#: call for different things from the reader.@@ -110,6 +129,26 @@ def _qdrant(request: Request) -> AsyncQdrantClient:return client+async def _scope_total(client: AsyncQdrantClient, scope: Any, *, fallback: int) -> int:+ """How many indicators are in the caller's scope, asked of the store.++ The scan above is deliberately bounded, so counting its results answers+ "how many did I look at", which is not the question the console is asking.+ Qdrant can answer the real one directly.++ A count failure falls back to the window size rather than raising: the+ indicators themselves were read successfully, and losing the whole list+ over a secondary count would turn a wrong number into no page at all.+ """+ try:+ result = await client.count(collection_name=IOC_COLLECTION, count_filter=scope, exact=True)+ except Exception as exc: # noqa: BLE001 — any count failure degrades to the window size+ logger.warning("threatintel.indicators.count_failed", error=type(exc).__name__)+ return fallback+ count = getattr(result, "count", None)+ return int(count) if isinstance(count, int) else fallback++@router.get("/indicators", response_model=IndicatorListResponse)async def list_indicators(request: Request,@@ -152,7 +191,7 @@ async def list_indicators(message = str(exc)if "not found" in message.lower() or "doesn't exist" in message.lower():logger.info("threatintel.indicators.collection_absent", collection=IOC_COLLECTION)- return IndicatorListResponse(indicators=[], total=0, source="qdrant")+ return IndicatorListResponse(indicators=[], total=0, shown=0, source="qdrant")logger.warning("threatintel.indicators.read_failed", error=type(exc).__name__)raise HTTPException(status_code=503,@@ -161,6 +200,7 @@ async def list_indicators(indicators = [_to_indicator(p.payload or {}) for p in points if p.payload]+ bounded = bool(ioc_type or tag or q)if ioc_type:indicators = [i for i in indicators if i.type == ioc_type]if tag:@@ -169,5 +209,6 @@ async def list_indicators(needle = q.lower()indicators = [i for i in indicators if needle in i.value.lower() or needle in (i.description or "").lower()]- total = len(indicators)- return IndicatorListResponse(indicators=indicators[:limit], total=total, source="qdrant")+ page = indicators[:limit]+ total = await _scope_total(client, scope, fallback=len(indicators))+ return IndicatorListResponse(indicators=page, total=total, shown=len(page), bounded=bounded, source="qdrant")diff --git a/services/threatintel/tests/test_indicator_totals.py b/services/threatintel/tests/test_indicator_totals.pynew file mode 100644index 000000000..76d54f193--- /dev/null+++ b/services/threatintel/tests/test_indicator_totals.py@@ -0,0 +1,137 @@+"""`total` must be the catalogue, not the page and not the scan window.++The route scrolls a bounded window over Qdrant and returns one page from it.+It published ``len(indicators)`` after post-filtering as ``total``, so with+1,725 CISA KEV entries collected the store held 1,725, the API answered 400+(the scan bound) and the console's headline card read 100 (its page) — three+answers to one question, two of them published, both wrong, and the page+presenting the smallest of them as the size of the corpus.++These tests drive `list_indicators` directly against a stub client rather than+a live Qdrant: the thing under test is which number ends up in which field,+and a container cannot make that clearer.+"""++from __future__ import annotations++import uuid+from dataclasses import dataclass+from typing import Any++import pytest+from app.api.indicators import list_indicators+from app.security.tenant_scope import TenantPrincipal++TENANT = uuid.UUID("00000000-0000-0000-0000-000000000001")+++@dataclass+class _Point:+ payload: dict[str, Any]+++@dataclass+class _CountResult:+ count: int+++class _StubQdrant:+ """Answers `scroll` with a fixed window and `count` with a fixed total."""++ def __init__(self, *, points: list[_Point], count: int | None, count_raises: bool = False) -> None:+ self._points = points+ self._count = count+ self._count_raises = count_raises+ self.scroll_limits: list[int] = []++ async def scroll(self, *, collection_name: str, scroll_filter: Any, limit: int, **_: Any):+ self.scroll_limits.append(limit)+ return self._points[:limit], None++ async def count(self, *, collection_name: str, count_filter: Any, exact: bool):+ if self._count_raises:+ raise RuntimeError("qdrant said no")+ return _CountResult(count=self._count or 0)+++class _Request:+ def __init__(self, client: _StubQdrant) -> None:+ self.app = type("_App", (), {"state": type("_State", (), {"qdrant_client": client})()})()+++def _points(n: int, *, ioc_type: str = "ip") -> list[_Point]:+ return [+ _Point(+ payload={+ "type": ioc_type,+ "value": f"CVE-2024-{i:05d}",+ "description": f"indicator {i}",+ "sources": ["cisa-kev"],+ "tags": ["kev"],+ }+ )+ for i in range(n)+ ]+++def _principal() -> TenantPrincipal:+ return TenantPrincipal(tenant_ids=frozenset({TENANT}), subject="test")+++async def _call(client: _StubQdrant, **kwargs: Any):+ return await list_indicators(+ request=_Request(client), # type: ignore[arg-type]+ principal=_principal(),+ ioc_type=kwargs.get("ioc_type"),+ tag=kwargs.get("tag"),+ q=kwargs.get("q"),+ limit=kwargs.get("limit", 100),+ )+++async def test_total_is_the_store_count_not_the_page() -> None:+ """The regression this file exists for, in the shape it was observed."""+ client = _StubQdrant(points=_points(400), count=1725)++ response = await _call(client, limit=100)++ assert response.total == 1725, "total must be the catalogue"+ assert response.shown == 100, "shown must be the page"+ assert len(response.indicators) == 100+ assert response.bounded is False+++async def test_total_survives_a_filtered_query_and_marks_it_bounded() -> None:+ """A post-filter narrows the page; it does not shrink the catalogue."""+ client = _StubQdrant(points=_points(40, ioc_type="ip"), count=1725)++ response = await _call(client, ioc_type="domain", limit=100)++ assert response.total == 1725+ assert response.shown == 0+ assert response.bounded is True, "the caller must be told the match was made in a bounded window"+++async def test_count_failure_degrades_to_the_window_rather_than_losing_the_page() -> None:+ """Losing the list over a secondary count would be the worse failure."""+ client = _StubQdrant(points=_points(12), count=None, count_raises=True)++ response = await _call(client, limit=100)++ assert response.total == 12+ assert response.shown == 12+ assert len(response.indicators) == 12+++async def test_empty_store_reports_zero_rather_than_a_page_length() -> None:+ client = _StubQdrant(points=[], count=0)++ response = await _call(client, limit=100)++ assert response.total == 0+ assert response.shown == 0+ assert response.indicators == []