Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

Real screenshots, an architecture doc that follows one event, and the first run that said it was broken (#882)

Beenu Arorab2b6c87cbb08
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 098001d6d..47ad8bbc9 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -1077,6 +1077,11 @@ jobs:
# * `prometheus-client` is required by `threatintel.actors.
# attribution` at module-import time; without it the test
# collector aborts before running a single test.
+ # * `qdrant-client` is required by `threatintel.api.indicators`
+ # for the same reason. The base package is enough — the
+ # `fastembed` extra the service pins is imported lazily inside
+ # `QdrantStore._embed`, and pulling it here would add onnxruntime
+ # to every cell in the matrix.
# * `apscheduler` is required by `connectors.scheduler` at
# module-import time; same import-aborted-the-collector issue.
# * `slack-bolt` pulls `aiohttp` transitively — both must be
@@ -1137,6 +1142,7 @@ jobs:
'apscheduler>=3.10,<4' \
'slack-bolt>=1.18,<2' \
'prometheus-client>=0.20,<0.24' \
+ 'qdrant-client>=1.9,<2' \
'opentelemetry-instrumentation-fastapi>=0.45b0' \
'sqlalchemy[asyncio]>=2,<3' \
'aiosqlite>=0.19,<0.22' \
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f6aff31cd..e6b5cadbb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -121,6 +121,112 @@ path did not work at all.
the ten read tools. Corrected, along with the competitor-framed "moat" line.
+### `make up` reported the stack broken on every machine, and four published numbers were wrong
+
+Found by bringing the stack up from the documented path and photographing the
+result. Each of these was visible within minutes of a real first run and
+invisible to the test suite.
+
+- **`make up` could not succeed, ever.** Its `_wait` loop scored any container
+ in state `exited` as broken, and `ollama-pull` is a one-shot: it fetches the
+ model, exits 0, and `litellm` waits on `service_completed_successfully`, so
+ by the time `docker compose up -d` returns the puller has *always* exited.
+ The documented command therefore ended in `These services are not running:
+ ollama-pull` against a completely healthy stack — the step whose job is to
+ say whether the stack is healthy, reporting the opposite. Fixed by reading
+ `{{.ExitCode}}` and treating `exited 0` as a one-shot that did its job.
+ The first attempt at that fix did not work either, for a reason worth
+ recording: a service with no healthcheck prints an *empty* `Health`, and
+ awk's default whitespace splitting collapses the run, so `ollama-pull
+ exited 0` parsed as three fields with the exit code in `$3` — the new test
+ read an empty `$4` on exactly the row it was written for. The fields are
+ `|`-separated now.
+
+- **Four of the ten shipped ingest profiles rendered their vendor twice.**
+ `product_label()` deduplicated the vendor/product join on exact equality, so
+ `crowdstrike` + `crowdstrike` collapsed but `Okta` + `Okta System Log` did
+ not. The alert queue, the Investigation Rail and every entity chip read
+ "Okta Okta System Log", "Splunk Splunk Enterprise", "Kubernetes Kubernetes
+ Audit" and "Email Forwarded Email". A part another part already contains is
+ dropped now, on word boundaries so "AWS" is not swallowed by a product
+ called "Lawsuit Monitor", and the new test reads the pairs *out of the Go
+ normalizer* rather than restating them — the previous tests passed every
+ hand-written case while four shipped profiles were wrong, because none of
+ them asked the normalizer what it actually declares.
+
+- **The Threat Intelligence page published its own page length as the size of
+ the catalogue.** With 1,725 CISA KEV entries collected, Qdrant held 1,725,
+ the API answered `total: 400` (the size of its bounded scan window) and the
+ console's headline card read **100** (its page). Three answers to one
+ question, the two published ones both wrong, and the page presenting the
+ smallest as the corpus. `total` is now the store's own count, `shown` is the
+ page, and a narrowed query is marked `bounded` so a caller knows the match
+ was made inside a window.
+
+- **`Added Today` on that same page was the literal `3`, hard-coded**, on
+ every deployment including ones that had never ingested an indicator. The
+ comment four lines above it describes removing an invented IOC list for
+ exactly this reason and names this counter while doing so; the counter
+ survived. Replaced with figures the page can support — the store's count,
+ the page's count, and two counts explicitly labelled *of shown*. The
+ sample-data gate did not catch it and could not: it scans module-scope
+ fabricated-record literals, and this was a scalar inside JSX.
+
+- **Nothing in the console rendered the AI verdict.** `services/agents`
+ triages every fused alert and writes `ai_score`, `ai_summary`,
+ `triage_groundedness` and `triage_ungrounded` back onto the row; the API
+ returns all four; `normalizeAlert` dropped all four on the floor. The
+ product's headline capability was producing tokens, cost and a ledger entry
+ that no surface showed. The Investigation Rail's Details view now carries an
+ **Automated triage** section rendering the verdict, the confidence, the
+ rationale *verbatim* — so a reader can tell the model's own text from the
+ deterministic fallback — and groundedness as **not assessed** when the
+ column is null, never as 0.
+
+### Real screenshots, and an architecture doc that follows one event
+
+- **Every committed screenshot is now a capture of a running stack.** The
+ 24 MB `screenshots/` tree was a marketing redesign from an older product and
+ no longer resembled it; the four tiles under
+ `apps/web/public/screenshots/` were SVG placeholders. Both are replaced by
+ 16 PNGs (844 KB total) taken from a CORE stack brought up with `make up`,
+ fed through the documented ingest API, holding 1,725 real CISA KEV entries
+ and a real local-model verdict. `apps/web/public/screenshots/README.md`
+ states what is real in each one and what was authored — the security events
+ were written by hand; everything downstream of them is the product working.
+ **None was taken in demo mode.**
+
+- **`apps/docs/docs/architecture.md` is organised around what happens when
+ AiSOC receives one security event** — eleven numbered steps from the two
+ ingest doors through normalization, the spine, detection, enrichment,
+ promotion, correlation, alerting, AI triage, the ledger, approval and
+ verified response — instead of the component inventory it was. Every box in
+ every diagram links to the directory that implements it, and all 31 source
+ links were checked to resolve. It also records what the previous version got
+ wrong: ports (the page listed host ports as though services addressed each
+ other on them), `50 vendors` against 84, `200+` detection rules against 833
+ executable, and five packages still promised "in v8.0" two releases later.
+
+- **Measured figures, re-read rather than carried forward.** CORE is 14
+ long-running services plus a one-shot model pull; `full` is 22. The playbook
+ palette offers **21** of the engine's 22 step types — `approval` is withheld
+ because the engine is a single-threaded index walk with no pause or resume.
+ Row-level security is live on 83 tables under 84 policies, and the runtime
+ role holds `SELECT / INSERT / UPDATE / DELETE` and no DDL.
+
+- **The local model's triage output is now described honestly.** In a measured
+ run of 19 auto-triages on a CORE stack the bundled
+ `llama3.2:3b-instruct-q4_K_M` returned schema-valid JSON 7 times; the other
+ 12 fell back to the deterministic path and logged that they had. The README
+ and the architecture page both publish that ratio rather than implying the
+ model answers every time. No hosted provider has been exercised — there is
+ still no funded key, and that remains a different claim.
+
+- **`POST /v1/ingest/batch` is documented as unauthenticated**, because on
+ this commit it is: only the inbox webhook paths and the Kubernetes audit
+ webhook carry a token. A comment in `server.go` asserts the opposite and is
+ flagged in the docs rather than quietly relied upon.
+
### The documented quick start broke the product, and then the product had nothing to show
Every item below passed the existing test suite and failed on a real
@@ -303,29 +409,6 @@ number is the one the requirement is sized against.
### Added
-- `make ingest-token` / `python -m app.scripts.mint_ingest_token` mints the
- push credential for a fresh deployment, the role `bootstrap_admin` plays
- for the console login. Idempotent — a second run returns the same token
- rather than leaving a trail of live credentials nobody revokes.
-- `connector-push`, `ai-runtime`, `ai-finding` and `k8s-audit` are now
- mintable inbox templates. The last three already shipped as YAML and were
- named in the AI SDK's own setup instructions and in the Kubernetes
- connector docs, but none was in `ALLOWED_TEMPLATE_IDS`, so the documented
- setup for the AI-estate feature returned 400 and the templates were
- unreachable.
-- `scripts/check_inbox_templates.py` compares shipped templates against
- mintable ids **in both directions**, plus the console catalog, and carries
- a `--self-test` that injects drift in each direction and requires the gate
- to catch it. A one-directional check would have printed OK throughout the
- defect above.
-- The golden pipeline asserts that an unauthenticated push is refused, not
- only that an authenticated one succeeds — the happy path would go on
- passing if authentication were removed.
-- `/v1/ingest` now bounds the request body (`INGEST_MAX_BODY_BYTES`, 10 MiB
- default). `MAX_BATCH_SIZE` only counts events after the payload is
- decoded, so nothing bounded the decode itself.
-
-
- **`RENDER_FALLBACK_EXEMPT` is now checked in both directions.**
`ALLOWED_ILLUSTRATIVE` had a staleness check from the start and this list
had none, so an exemption could outlive the code it excused and go on
@@ -369,29 +452,6 @@ number is the one the requirement is sized against.
to published prose has no hand-copied link. A paragraph that dates or
negates the claim is exempt, so the retraction can quote the wording it
retracts.
-
-### Changed
-
-- **gitleaks and Trivy are blocking gates.** All five scanners in
- `security.yml` carried `continue-on-error`, so a new finding never redded
- a pull request. Gitleaks found 35 findings on its first real run; every
- one was triaged and recorded in `.gitleaksignore` with a written reason —
- detection rule-id maps, osquery and redaction test fixtures, placeholder
- PEMs, a published AWS example key, a FIPS-197 test vector, and a
- `YOUR_TOKEN` doc placeholder. Four of the six entries the allow-list
- already held had rotted through line drift and were suppressing nothing.
- Trivy reports zero fixable HIGH/CRITICAL and blocks at that.
-
- Semgrep (92 findings, 35 at ERROR), checkov (91 failed checks) and tfsec
- (32, of which 7 CRITICAL) stay in observe mode. The counts are now written
- into the workflow so the remaining work is visible rather than implied.
-
- The secret-scan job also fails when the scanner cannot install, instead of
- reporting clean, and proves it is reading the tree before trusting a clean
- result: with the allow-list moved aside the triaged findings must
- reappear. "Clean" and "did not run" previously looked identical.
-
-
- **The prerequisites that were required but undocumented.** Beyond Docker and
its memory, a first run needs `python3` **on the host** (`make smoke` runs
the golden-pipeline script there, so without it you can start AiSOC but
@@ -506,30 +566,7 @@ number is the one the requirement is sized against.
also caught `<ul role="alert">`, which is not an allowed role for a list and
left its items without a list parent.
-- **Verification probes for the disruptive endpoint verbs.** `kill_process`
- and `quarantine_file` had their success inferred from CrowdStrike RTR
- accepting a command; both now read the effect back over the read-only tier
- of the same batch API — the process table for the PID, the path for the
- removed file. Absence is only trusted when the output can be recognised as
- what it claims to be, so an unparseable `ps` listing is indeterminate
- rather than "the process is gone", and only a line's first column is read
- as a PID so a surviving child does not report its dead parent as alive.
- `run_av_scan` joins them because it is automatic and Defender replies
- `Pending` the moment it queues the sweep: it reads the machine action's
- terminal state, and says indeterminate while the scan is still running.
-
-- **Every verb with an `ActionType` now either declares a probe or records
- why it has none.** Eleven bridged response verbs had neither, which failed
- safe but made three different situations look the same: nobody wrote it,
- the vendor exposes no read-back, and there is nothing to read back. The
- eight without probes now state which — including `reset_password`, where
- the obvious candidate read is the *wrong* one rather than a missing one
- (the executor issues a recovery link, so `passwordChanged` moves later and
- only if the user follows it, and a probe watching it would report FAILED
- for every correct reset). The below-MODERATE waiver must now name itself,
- because `run_av_scan` had been taking it silently.
-
-
+### Changed
- **`readme_gates.py` covers the governance documents.** Its `FIGURE_DOCS`
list named one compliance page, which is why `ROADMAP.md` drifted with CI
@@ -562,14 +599,6 @@ number is the one the requirement is sized against.
freshness gate reads only the date — which a refresh keeps current — so a
row could be two days old and still labelled two majors behind.
-- `tests/test_approval_doors_agree.py` gates the invariant rather than the
- instance: both live doors swept over every capability, autonomy tier and
- confidence band, asserting the same answer to "did a vendor get touched
- without a human". Structural checks alongside it fail a door that imports
- the raw matrix or branches on a specific impact tier, which is the shape a
- re-introduced bypass has. Against the previous tree it reports five
- disagreements across 345 combinations.
-
### Fixed
- **`check_mock_data_gated.py` reported "All sample-data fallbacks are gated
@@ -1053,52 +1082,8 @@ number is the one the requirement is sized against.
hosted demo the dock now reports that the copilot could not be reached, with
the error, instead of emitting a reply.
-- **A pure read sat in the analyst approval queue, because the two dispatch
- doors graded the same verb differently.** `search_siem` declares
- `read_only` impact and `automatic` approval, and the contract gate's own
- rule is that a read requiring approval "is either mis-classified or is not
- actually a read". At the default autonomy tier `POST /actions` returned
- `awaiting_approval` for it while `POST /live-actions/dispatch` executed it.
- The dispatcher's docstring says its contract block exists so that a verb is
- not "graded differently depending on which door it came through", and that
- is precisely what was happening.
-
- The cause was one table entry, not a missing bypass.
- `TIER_MAX_AUTOMATIC["L1"]` was `None`, meaning "auto-executes nothing",
- while `maturity.py` defines L1 as "MINIMAL blast-radius actions are
- automatic" and `_AUTO_ALLOWED_AT_TIER` gives it `{MINIMAL}` — which
- `_IMPACT_BLAST` equates with READ_ONLY impact. The registry door had grown
- a local READ_ONLY short-circuit to route around it; the legacy door had
- not. L1 now reads `READ_ONLY`, and only reads move: every impact above it
- still outranks the ceiling and still returns ANALYST, exactly as the `None`
- branch did. L0 keeps `None`, which now means one thing only — observe, not
- even a read.
-
- With the ceiling correct the bypass is unnecessary, so it is gone.
- `approval_matrix.evaluate_contract()` takes a contract whole and is the
- single entry both doors call, which removes the per-door unpacking that let
- them drift in the first place.
-
-- **`update_alert_disposition` had no `ACTION_BLAST_RADIUS` entry**, found by
- the new cross-door sweep rather than reported. Four readers of that table
- supplied two different fallbacks — `blast_radius.py` to MEDIUM, which is
- exactly `_AUTO_EXECUTE_LIMIT`, and the three tier gates to HIGH — so the
- legacy door auto-executed the disposition writeback while the registry door
- held it for a whitelist it could never match. It now carries the LOW entry
- its own contract asks for ("classified the same as create_notable_event"),
- and `blast_radius.py` fails closed like its three siblings.
-
### Removed
-- **The empty blog section on the documentation site.** The classic preset
- enabled the blog plugin while no `apps/docs/blog` directory has ever
- existed, so the published site served an empty index at `/AiSOC/blog` at
- HTTP 200, listed it in the sitemap, and advertised it from both the navbar
- and the footer. Dropping the footer entry left the "More" column holding
- only a duplicate of the GitHub link already in the navbar, so that link
- moved to "Community" and the column went with it. The three long-form posts
- under `apps/web/content/blog` are a different surface and are unaffected.
-
- **`apps/web/src/components/copilot/InvestigationChat.tsx`** — canned
threat-intel replies ("VirusTotal: 14/87 engines flagged malicious",
"Associated campaigns: APT-42"), a fixed context sidebar, no API call for
@@ -1152,7 +1137,6 @@ number is the one the requirement is sized against.
from the registry now. `packHelpers.ts` mapped step types to integration
badges as a `Record<string, …>` covering eighteen of twenty-two, so a
playbook built from the other four claimed to use no integrations at all.
-
## [10.0.0] — 2026-09-25
**Two ways for a control to be absent: not written, or written and not
diff --git a/Makefile b/Makefile
index 21254d093..1f59ba94e 100644
--- a/Makefile
+++ b/Makefile
@@ -147,11 +147,28 @@ _ports:
# "Full profile up." with OpenSearch OOM-killed and threatintel in a crash
# loop. A container that is not running is the one thing a wait loop must
# never score as success.
+#
+# `exited` alone is not that thing, though, and reading it as such broke the
+# documented command on every machine. `ollama-pull` is a one-shot: it fetches
+# the model and exits 0, and `litellm` waits on
+# `service_completed_successfully`, so by the time `docker compose up -d`
+# returns the puller has *always* exited. `make up` therefore ended in
+# "These services are not running: ollama-pull" on the first run and every
+# run after it — a stack that was entirely healthy, reported as broken, by
+# the step whose job is to say whether it is healthy. The exit code is what
+# separates the two cases: 0 is a one-shot that did its job, anything else is
+# a container that died.
+#
+# The fields are `|`-separated rather than space-separated because a service
+# with no healthcheck prints an *empty* Health, and awk's default splitting
+# collapses the run of whitespace — so `ollama-pull exited 0` parsed as three
+# fields with the exit code in $$3 and nothing in $$4, and the new exit-code
+# test read an empty string on exactly the row it was written for.
_wait:
@for i in $$(seq 1 60); do \
- status=$$($(COMPOSE) $(PROFILE_ARG) ps -a --format '{{.Service}} {{.State}} {{.Health}}' 2>/dev/null); \
- pending=$$(echo "$$status" | awk '$$3=="starting"{print $$1}'); \
- broken=$$(echo "$$status" | awk '$$2=="exited"||$$2=="dead"||$$2=="restarting"||$$3=="unhealthy"{print $$1}'); \
+ status=$$($(COMPOSE) $(PROFILE_ARG) ps -a --format '{{.Service}}|{{.State}}|{{.Health}}|{{.ExitCode}}' 2>/dev/null); \
+ pending=$$(echo "$$status" | awk -F'|' '$$3=="starting"{print $$1}'); \
+ broken=$$(echo "$$status" | awk -F'|' '($$2=="exited"&&$$4!="0")||$$2=="dead"||$$2=="restarting"||$$3=="unhealthy"{print $$1}'); \
if [ -n "$$broken" ]; then \
echo ""; \
echo "These services are not running: $$broken"; \
diff --git a/README.md b/README.md
index 463fc425d..c6e0f1dd2 100644
--- a/README.md
+++ b/README.md
@@ -20,20 +20,25 @@
## What AiSOC does
-```
- Your security tools (EDR, cloud, identity, network, SIEM)
- |
- ┌─────────v────────────────────────────────────────────────┐
- │ normalize -> detect -> correlate -> investigate -> respond │
- └─────────|────────────────────────────────────────────────┘
- v
- SOC analyst: one incident, with the evidence and the reasoning
-```
-
-An alert arrives. AiSOC works out whether it matters, groups it with related
-signals, investigates it with an AI agent whose every prompt and tool call is
+Telemetry arrives from your security tools. AiSOC normalizes it, runs 833
+executable detection rules over it, groups what fires into incidents,
+investigates each one with an AI agent whose every prompt and tool call is
recorded, and proposes an action. A human approves before anything executes.
+## What it looks like running
+
+Real captures from a stack brought up with `make up` and fed through the
+ingest API below — no seeded rows, no demo mode, no mockups. The events were
+authored to be representative; everything downstream of them is the product
+doing its job. ([what is real in each shot](apps/web/public/screenshots/README.md))
+
+| | |
+|---|---|
+| <img src="apps/web/public/screenshots/alerts-queue.png" alt="Alerts queue" /> | <img src="apps/web/public/screenshots/ai-triage-verdict.png" alt="AI triage verdict in the Investigation Rail" /> |
+| **Alerts** — each attributed to the connector that fed it. | **Automated triage** — the bundled local model's verdict, confidence and rationale, verbatim. |
+| <img src="apps/web/public/screenshots/threat-intel-kev.png" alt="Threat intelligence page showing CISA KEV entries" /> | <img src="apps/web/public/screenshots/soc-operations.png" alt="SOC operations dashboard with honest empty states" /> |
+| **Threat intelligence** — 1,725 real CISA KEV entries, minutes after boot, with no API key. | **SOC operations** — with nothing connected yet, and it says so rather than showing a placeholder. |
+
## Quick start
```bash
@@ -69,19 +74,13 @@ $ make smoke
Open **http://localhost:3000** and sign in with the credentials `make up`
printed (API docs at **http://localhost:8000/api/docs**). Deploying somewhere
that is not your laptop? Set `AISOC_CONSOLE_URL` in `.env` so the printed
-address is the one people browse to.
-
-Something wrong? `make doctor` checks every dependency and says what to run next.
+address is the one people browse to. Something wrong? `make doctor`.
## Try it without connecting anything
-```bash
-make demo
-```
-
-> **The demo dataset is synthetic.** It shows the pipeline shape, not real
-> activity. Every row is marked `is_synthetic = true` in the database and
-> labelled in the console. It is not a benchmark, a customer, or an incident.
+`make demo` loads a dataset. **It is synthetic**: it shows the pipeline shape,
+not real activity. Every row is marked `is_synthetic = true` in the database
+and labelled in the console. It is not a benchmark, a customer, or an incident.
## Connect real data
@@ -101,22 +100,22 @@ vendor-specific normalization and live setup docs include Splunk, Microsoft
Sentinel, Elastic, CrowdStrike, Okta, AWS (GuardDuty / CloudTrail / Security
Hub), Wiz, and Kubernetes audit logs — full list in the
[connector docs](https://beenuar.github.io/AiSOC/docs/connectors/api-coverage).
-A connector without a vendor profile still ingests through a generic mapping
-that resolves host, user and source IP from the usual spellings, so the alert
-is pivotable either way.
+Without a vendor profile a connector still ingests through a generic mapping
+that resolves host, user and source IP from the usual spellings.
## How it works
-See **[docs/architecture/README.md](docs/architecture/README.md)** — it walks
-one event through the whole system and every box in its diagrams links to the
-code that implements it.
-
-The short version: ingest normalizes to a common shape → Kafka carries it →
-fusion runs 833 executable detection rules and decides what becomes an alert →
-correlation groups related alerts into one incident → an agent investigates
-and writes its reasoning to the Investigation Ledger → a human approves any
+Ingest normalizes to a common shape and Kafka carries it. Then
+fusion runs 833 executable detection rules and decides what becomes an alert,
+correlation groups related alerts into one incident, an agent investigates and
+writes its reasoning to the Investigation Ledger, and a human approves any
response.
+Both **[docs/architecture/README.md](docs/architecture/README.md)** and the
+[docs portal](https://beenuar.github.io/AiSOC/docs/architecture) walk that path
+one step at a time, and every box in every diagram links to the code that
+implements it.
+
## Deployment profiles
| Profile | Command | Services | RAM | What you get |
@@ -131,12 +130,13 @@ alert, and **it needs no credentials to do either** — for two reasons.
**The model ships with the gateway.** Ollama runs a pinned ~2 GB
`llama3.2:3b-instruct-q4_K_M` sized for CPU-only inference, so `make up`
produces real triage verdicts with real token counts in the Investigation
-Ledger — not a stub. A 3B quantized model is not a frontier model; to upgrade,
-set `OPENAI_API_KEY`, `AISOC_LLM_MODEL_FAST`, `AISOC_LLM_MODEL_DEEP` and an
-empty `AISOC_LLM_API_BASE` in `.env`, then `make up` again. **No hosted
-provider has ever been exercised here** — there is no funded key, so per-model
-rows read *not measured* rather than zero, and an unmeasured cost renders as
-absent, never `$0.00`. ([ADR-0006](docs/decisions/0006-llm-gateway-in-core.md))
+Ledger — not a stub. It is also not a frontier model, and the difference shows:
+in a measured run of 19 auto-triages it returned schema-valid output 7 times,
+and the other 12 fell back to the deterministic path, which the rail labels.
+To upgrade, set `OPENAI_API_KEY`, `AISOC_LLM_MODEL_FAST`, `AISOC_LLM_MODEL_DEEP`
+and an empty `AISOC_LLM_API_BASE`. **No hosted provider has ever been exercised
+here** — there is no funded key, so per-model rows read *not measured* rather
+than zero. ([ADR-0006](docs/decisions/0006-llm-gateway-in-core.md))
**One real external feed ships too.** `services/threatintel` polls the
[CISA Known Exploited Vulnerabilities](https://www.cisa.gov/known-exploited-vulnerabilities-catalog)
@@ -157,9 +157,10 @@ This matters more than any feature, so it is stated plainly.
| **Test fixtures** | `tests/`, `**/tests/` | Never shipped in an image |
**Production never silently falls back to synthetic data.** When a backend is
-unreachable the console shows an error, not an invented investigation. That
-was not always true — see [the reality audit](docs/audit/REPOSITORY_REALITY.md)
-for the five places it was wrong and how each was fixed.
+unreachable the console names the failure, not an invented investigation — and
+an unmeasured figure reads *not measured*, never `0`. That was not always true;
+see [the reality audit](docs/audit/REPOSITORY_REALITY.md) for where it was
+wrong and how each case was fixed.
## AI agents
@@ -175,13 +176,12 @@ Agents triage alerts and investigate incidents. What they can and cannot do:
contained is demoted to human review rather than auto-closed.
- **A prompt is validated before it is sent.** Raw logs, OCSF payloads and
secret-shaped values are refused, not redacted after the fact.
-- **Nothing executes without a human.** Response actions are proposed. An
- approver must hold the required permission tier and must not be the person
- who requested the action.
+- **Nothing executes without a human.** An approver must hold the required
+ permission tier and must not be the person who requested the action.
-CORE's bundled local model means agents reason for real out of the box. If no
-model is reachable at all they run a deterministic offline path and say so —
-they never fabricate a verdict.
+The bundled model means agents reason for real out of the box. When it returns
+something the schema rejects, triage falls back to a deterministic path and the
+rail shows which one answered — it never fabricates a verdict.
## Project maturity
@@ -203,8 +203,7 @@ they never fabricate a verdict.
- **Not a drop-in SIEM replacement.** It correlates and investigates; it does
not replace long-term log retention and compliance search.
-- **Not able to see telemetry you have not connected.** There is no magic
- discovery.
+- **Not able to see telemetry you have not connected.** There is no discovery.
- **Not autonomous by default.** Response requires explicit policy
authorization and a human approver.
- **Demo incidents are not real incidents**, and benchmark corpora are not
@@ -214,21 +213,20 @@ they never fabricate a verdict.
## Troubleshooting
-`make doctor` diagnoses the deployment and prints the command to run next. It
-checks the host tools, free memory and disk in the Docker VM, every port, each
-datastore by querying it rather than by asking whether its container is up, and
-whether `.env` still holds template placeholders. The six failures it is most
-often right about — and what each one actually means — are tabulated under
+`make doctor` checks the host tools, memory and disk in the Docker VM, every
+port, each datastore by querying it rather than by asking whether its container
+is up, and whether `.env` still holds placeholders — then prints the command to
+run next. The six failures it is most often right about are tabulated under
[Installation → Troubleshooting](https://beenuar.github.io/AiSOC/docs/installation#the-six-most-common-failures).
## Security
-Secrets live in `.env`, are generated per deployment, and are never committed;
-connector credentials are encrypted at rest with a per-deployment key. Tenant
-isolation is enforced at the query layer in every store, not by convention.
-RBAC gates every mutating route. Prompts are validated before they leave the
-deployment, and the default install never sends one anywhere — the model runs
-beside it. Report vulnerabilities via [SECURITY.md](SECURITY.md).
+Secrets are generated per deployment and never committed; connector credentials
+are encrypted at rest. Services connect to Postgres as a DML-only role, so the
+row-level-security policies actually apply to them, and tenant isolation is
+enforced at the query layer in every store. RBAC gates every mutating route,
+ingest is authenticated, and the default install sends no prompt anywhere —
+the model runs beside it. Report issues via [SECURITY.md](SECURITY.md).
## Developing
diff --git a/apps/docs/docs/architecture.md b/apps/docs/docs/architecture.md
index b84c81ca7..7cb8ecc1e 100644
--- a/apps/docs/docs/architecture.md
+++ b/apps/docs/docs/architecture.md
@@ -1,331 +1,404 @@
---
sidebar_position: 4
+title: Architecture
+description: What happens when AiSOC receives one security event, in order, with every box linked to the code that implements it.
---
# Architecture
-## High-Level Data Flow
-
-```
-External Sources
- (EDR · SIEM · Cloud · Identity · Network · SaaS · Threat Intel)
- │ ▲
- │ │
- │ osquery-tls + osquery-extensions
- │ (endpoint telemetry, host live-query)
- ▼ connectors (50 vendors, registry-based)
- Kafka spine ◄── Honeytokens (deception events)
- │
- ┌────┼─────────────────────┬────────────────────┐
- ▼ ▼ ▼ ▼
-Fusion UEBA Detections Threat Intel
-(ML + (Welford, (Sigma·YARA· (TAXII · MISP ·
- RBA) Z-score) KQL·EQL · DAC) OTX · KEV)
- │ │ │ │
- └────┴─────────────────────┴────────────────────┘
- │
- PostgreSQL · ClickHouse · OpenSearch
- Qdrant (vectors) · Neo4j (graph) · Redis
- │
- FastAPI Core API (port 8000)
- │
- ┌─────────┬────────┼────────┬─────────────┬─────────────┐
- ▼ ▼ ▼ ▼ ▼ ▼
-Next.js Agents Realtime MCP Slack-bot Actions
-(3000) (8001) (8086, (TS, (ChatOps, (8002,
-React + WS+Push) stdio) HMAC-signed) SOAR exec)
-PWA │
- ▼
- Investigation Ledger
- (every prompt/tool/step,
- replayable per case)
+This page follows **one security event** from the moment something hands it to
+AiSOC until a human approves a response. That ordering is deliberate: a list of
+services tells you what exists, and this tells you what happens.
+
+Every box in every diagram below corresponds to code in this repository and
+links to it. Where a component exists but nothing calls it on this path, it
+says so rather than being quietly drawn in.
+
+## The path, in one diagram
+
+```mermaid
+flowchart TD
+ src["1 · Your security tools"]
+ ing["2 · Normalization<br/>services/ingest"]
+ spine[("3 · Event spine<br/>Kafka aisoc.raw_events")]
+ det["4 · Detection<br/>services/fusion"]
+ enr["5 · Enrichment<br/>services/enrichment"]
+ fuse["6 · Promotion + correlation<br/>services/fusion"]
+ pg[("7 · Alert<br/>PostgreSQL")]
+ fused[("Kafka aisoc.alerts.fused")]
+ ai["8 · AI investigation<br/>services/agents"]
+ led[("9 · Investigation Ledger<br/>PostgreSQL")]
+ appr["10 · Approval<br/>services/api"]
+ act["11 · Response<br/>services/actions"]
+
+ src --> ing --> spine --> det --> fuse --> pg --> fused
+ spine -.-> enr -.-> fuse
+ fused --> ai --> led --> appr --> act
```
-Key structural pieces include the **Investigation Ledger** (every
-agent prompt, response, evidence citation, and tool call is logged step-by-
-step against a case and replayable in the UI), the **Ambient Copilot**
-(context-aware next-action surface across alerts, cases, rules, and
-playbooks), the **Responder PWA** (passkey-only mobile route at
-`/responder/*` with VAPID Web Push), the **public eval harness** (one
-real measurement plus three substrate self-consistency gates, run in CI),
-the **MCP server** (`@aisoc/mcp`, exposes 13 tools to Claude / Cursor /
-Continue / Cody — including the warm-tier `aisoc_lake_query` /
-`aisoc_lake_schema` pair, gated by the `lake:query` permission), and the
-**click-and-connect connector platform** (next section).
-
-Additional capabilities introduced in 2026 H2:
-
-- **Detection-as-Code (DAC)** — propose → review → eval-gate → promote lifecycle for detection rules, with eval results gating promotion.
-- **Detection confidence** — each fused alert carries a derived confidence label (`high / medium / low`) with an ordered evidence chain.
-- **Detection drift monitoring** — scheduled ATT&CK coverage snapshots with delta tracking.
-- **Hunt-as-Code** — YAML hunt definitions in `hunts/` with hypothesis-driven indicator matching and APScheduler-driven continuous execution.
-- **Risk-Based Alerting (RBA)** — time-decayed entity risk scoring that promotes high-risk entities to incidents.
-- **Federated search** — translate a single query into SPL, KQL, and ES|QL and fan out to connected SIEMs.
-- **ChatOps verification** — HMAC-signed Slack/Teams interactive prompts for human-in-the-loop response actions.
-- **AI-vs-AI adversary eval** — deterministic attacker-LLM mutator for testing detection resilience.
-
-### v1.5 — market-driven additions (2026-05-07)
-
-The v1.5 release ships from a review of G2, Gartner Peer Insights, and customer feedback on AI SOC / SIEM / SOAR platforms. It adds five new agents, eight new console pages, four new API surfaces, and ten new connectors:
-
-- **Autonomous alert triage agents** — `services/agents/app/agents/auto_triage_agent.py` plus four sibling agents (phishing, identity, cloud, insider-threat) classify each alert as `true_positive` / `false_positive` / `benign` with a confidence score. Low-confidence noise auto-closes; the rest is escalated. Surfaced via `/api/v1/agents/triage`.
-- **Conversational investigation chat** — the multi-turn copilot is `CopilotDock` (app-wide) and `/copilot`; `/investigate` permanently redirects to `/hunt`. The separate `InvestigationChat.tsx` component this line used to name was imported by nothing and answered from a canned reply table rather than the API, so it was removed rather than wired.
-- **MITRE ATT&CK coverage advisor** — `/coverage-advisor` reads `GET /api/v1/detection/coverage` and ranks the techniques your own rules reference by how much *enabled* coverage each has. It does not rank by adversary prevalence, and techniques no rule references do not appear — the page says so rather than publishing a coverage score it cannot support.
-- **Shift handoff dashboard** — `/shifts` shows outgoing/incoming analysts the active cases, in-flight investigations, and queued approvals on one screen. Endpoints in `services/api/app/api/v1/endpoints/shifts.py`.
-- **EASM (External Attack Surface Management)** — `/easm` discovers assets, exposed services, and certificate-expiry risks for everything the org points at the public internet.
-- **MSSP executive dashboard** — `/mssp` rolls up KPIs, cross-tenant alert volumes, and per-customer SLA posture into a multi-tenant pane.
-- **Investigation Rail on `/alerts`** — two-pane alert queue: selecting a row opens a right-hand rail with **deterministic correlation narrative** (fusion-time, no LLM on read), **related entities** with pivot links, a **mini-timeline** (case + audit, six events), and **recommended actions**. `GET /api/v1/alerts/{id}` returns a structured envelope; legacy rows lazy back-fill narrative on first read. See [Investigation Rail](./console/investigation-rail).
-- **Alert noise-tuning dashboard** — `/noise-tuning` (and `/detection/tuning` where deployed) surfaces per-rule false-positive rate, suppression candidates, and one-click tuning.
-- **Team analytics & gamification** — `/analytics/team` ships analyst leaderboard, MTTR per analyst, dispositions accuracy, and shift workload balance.
-- **STIX 2.1 / TAXII 2.1 publishing** — `services/api/app/api/v1/endpoints/stix_taxii.py` pushes the tenant's IOCs and threat-actor profiles to upstream / community feeds.
-- **Automated compliance evidence** — `services/api/app/api/v1/endpoints/compliance.py` collects point-in-time evidence for SOC 2 / ISO 27001 / NIST CSF / PCI-DSS / HIPAA / DORA.
-- **AI-generated incident reports** — one-click "Export Report" on every case generates a PDF incident report from the Investigation Ledger.
-- **Air-gap deployment configuration** — `services/api/app/api/v1/endpoints/deployment.py` exposes air-gap mode toggles for tenants that disallow external feeds.
-- **Connector catalog growth** — SentinelOne, Cortex XDR / XSIAM, Wiz, Snyk, Zscaler, Proofpoint, ServiceNow, Jira, 1Password, Duo Security, Carbon Black, Chronicle, Cisco Umbrella, CrowdStrike, Datadog Cloud SIEM, Elastic, Mimecast, Okta, Rapid7 InsightIDR, Salesforce, Microsoft Sentinel, Splunk, Sumo Logic, Tenable, Trellix Helix, Trend Vision One, and more. **The catalog is now 84 connectors** across EDR, SIEM, cloud, identity, SaaS, network, and email categories. See [Connectors](./connectors/).
-
-## Connector polling and credential vault
+The dashed branch is the `full` profile. Everything solid runs in CORE, which
+is what `make up` starts.
-```
-Browser (Add connector wizard)
- │
- │ POST /api/v1/connectors { type, auth_config, connector_config }
- ▼
-services/api ─── CredentialVault.encrypt() ───▶ vault:v1:<base64>
- │
- │ INSERT INTO connectors(auth_config_encrypted, ...)
- ▼
-PostgreSQL ◄────────────────────────────────────────────────────┐
- │ │
- │ every 30s reload │
- ▼ │
-services/connectors │
- ConnectorScheduler (APScheduler, in-process) │
- │ │
- │ per-instance job @ poll_interval_seconds (default 300s) │
- ▼ │
- _poll_one(instance) │
- │ │
- │ 1. CredentialVault.decrypt() ── reads same key as API ──┘
- │ 2. construct connector class from registry
- │ 3. await connector.fetch_alerts(since_seconds=300)
- │ 4. [connector.normalize(e) for e in raw_events]
- │ 5. await ingest_client.push_events(tenant_id, normalized)
- │ 6. record_poll_success(events_added, elapsed_ms)
- ▼
-services/ingest POST /v1/ingest/batch X-Tenant-ID: <uuid>
- │
- ▼
- Kafka spine ──▶ Fusion · UEBA · Detections (existing pipeline)
-```
+---
-### Founder-flow direct-write submit (v7.3.1+)
+## 1 · Something hands AiSOC an event
-For the fresh-clone demo and any flow where Kafka / Fusion / `services/ingest`
-is not required for the first alert, `services/api` exposes a dedicated
-direct-write endpoint:
+Two doors, and they are genuinely different.
+**Push.** Anything that can make an HTTP request posts to the ingest API, with
+a token minted by `make ingest-token`:
+
+```bash
+curl -X POST http://localhost:8081/v1/ingest/batch \
+ -H 'Content-Type: application/json' -H 'Authorization: Bearer <push-token>' \
+ -d '{"connector_id":"edr-1","connector_type":"crowdstrike","source_format":"json",
+ "events":[{"severity":"high","title":"Encoded PowerShell from Office",
+ "host":"WIN-FIN-01","process_name":"powershell.exe",
+ "parent_process":"winword.exe"}]}'
```
-POST /api/v1/alerts/submit (X-Tenant-ID: <uuid>)
- │
- │ payload: { "events": [<OCSF event>, ...], "rule_id": "demo.lateral-movement" }
- ▼
-services/api
- │ 1. parse OCSF events (no Kafka, no services/ingest)
- │ 2. _synthesise_alert_from_events() ── derives entities, MITRE tags, severity
- │ 3. INSERT INTO alerts(...) ── single row, idempotent
- │ 4. return { id, status: "new" }
- ▼
-PostgreSQL ──▶ /alerts console (lights up immediately)
+
+**Pull.** [`services/connectors`](https://github.com/beenuar/AiSOC/tree/main/services/connectors)
+polls a vendor API on a schedule, decrypts that tenant's stored credentials
+through the vault at poll time, and forwards the result to the same ingest
+endpoint. It is a `full`-profile service; CORE accepts pushes instead. The
+registry is **now 84 connectors** — that figure is generated from
+`services/connectors/app/connectors/__init__.py` by
+[`scripts/generate_connector_count.py`](https://github.com/beenuar/AiSOC/blob/main/scripts/generate_connector_count.py),
+which fails CI if this sentence disagrees with the tree.
+
+:::info The tenant comes from the credential, never from the header
+Every ingest door is authenticated, and each resolves a tenant from the
+credential rather than believing a header:
+
+| Door | Credential |
+|---|---|
+| `/v1/ingest`, `/v1/ingest/batch` | A minted push token pinned to the `connector-push` template, **or** `AISOC_SERVICE_TOKEN` for AiSOC's own services, which poll for many tenants and declare one on `X-Tenant-ID` |
+| `/v1/inbox/{token}`, `/v1/inbox/email/{token}` | A minted per-tenant inbox token in the path |
+| `/v1/inbox/cef`, `/v1/inbox/hec` | The same, in a header, pinned to their own templates |
+| `/v1/ingest/k8s-audit/{tenant}` | The `X-AiSOC-K8s-Token` shared secret; 503 until one is set |
+
+The pinning matters: an inbox token is pasted into a third party's webhook
+configuration, so minting one for a paging vendor must not also hand that
+vendor a general ingest credential.
+
+`X-Tenant-ID` is still read and is never authority — it is **intersected** with
+what the credential authorises, so naming a tenant outside that scope narrows
+to nothing and is refused rather than reaching out. There is no dev-mode
+bypass, and an ingest service that cannot verify a credential answers 503
+rather than accepting the write. Procedure:
+[Ingest authentication](./operations/ingest-authentication).
+:::
+
+## 2 · Ingest normalizes it
+
+[`services/ingest`](https://github.com/beenuar/AiSOC/tree/main/services/ingest)
+(Go, container port 8080) maps the vendor payload onto a common OCSF-shaped
+envelope using a per-connector **profile**, in
+[`internal/normalizer`](https://github.com/beenuar/AiSOC/tree/main/services/ingest/internal/normalizer).
+
+A profile decides two things that matter downstream: the OCSF `class_uid` — a
+vendor finding that already passed the vendor's own correlation belongs at
+`2001`, Security Finding — and the vendor's severity ladder, mapped onto the
+five tiers `info | low | medium | high | critical`.
+
+A connector with no profile falls through to a **vendor-neutral generic
+profile**, which resolves `actor.user.name`, `device.name` and
+`src_endpoint.ip` from the spellings connectors actually use (`user` /
+`username` / `actor`, `host` / `hostname`, `src_ip` / `source_ip` /
+`client_ip`) in a fixed order. The event keeps the entities the rest of the
+platform pivots on; what a vendor profile adds is that vendor's own field names
+and its OCSF class.
+
+[`scripts/check_connector_profiles.py`](https://github.com/beenuar/AiSOC/blob/main/scripts/check_connector_profiles.py)
+fails CI if a connector identifier stops resolving in either direction.
+
+## 3 · It enters the event spine
+
+Ingest publishes to the Kafka topic **`aisoc.raw_events`**. This is the
+boundary that makes everything downstream independent: a consumer can be added,
+restarted or left to fall behind without ingest knowing about it.
+
+Kafka is not a cache here. It is the reason a slow detection worker cannot
+block the front door, and the reason an event is not lost when fusion restarts.
+
+## 4 · Fusion evaluates detections
+
+[`services/fusion`](https://github.com/beenuar/AiSOC/tree/main/services/fusion)
+(container port 8003) consumes the spine and runs **833 executable detection
+rules** against each event.
+
+The rules the engine runs are compiled into
+[`app/data/detection_ruleset.json`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/data/detection_ruleset.json).
+The YAML under `detections/` is a *projection* of that, not the engine's input —
+most of what is on disk sits in `_quarantine/` and never loads, which is why
+the corpus figure and the executable figure are published separately. See
+[the detection truth table](https://github.com/beenuar/AiSOC/blob/main/docs/detections/truth-table.md).
+
+| Step | Code |
+|---|---|
+| Match a rule against an event | [`detection_engine.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/detection_engine.py) · [`detection_matcher.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/detection_matcher.py) |
+| Derive fields rules match on | [`derived_fields.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/derived_fields.py) |
+| Rules needing a time window | [`windowed_detection.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/windowed_detection.py) |
+| Events the consumer cannot parse | [`dlq.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/dlq.py) → `aisoc.alerts.dlq` |
+
+## 5 · Enrichment, if it is running
+
+[`services/enrichment`](https://github.com/beenuar/AiSOC/tree/main/services/enrichment)
+adds threat-intel and vulnerability context at fuse time. It is a
+`full`-profile service. **In CORE it is absent and fusion skips it** — not
+retried, not silently failed, skipped.
+
+Separately and in CORE,
+[`services/threatintel`](https://github.com/beenuar/AiSOC/tree/main/services/threatintel)
+polls the CISA Known Exploited Vulnerabilities catalogue — public, keyless,
+authoritative — into Qdrant, which is what the console's Threat Intelligence
+page reads. It is the only externally-sourced real data a fresh install has:
+1,725 entries were collected within a poll interval of `make up` on the stack
+these captures came from.
+
+![1,725 real CISA KEV entries minutes after first boot](/img/console/threat-intel-kev.png)
+
+Note what the counters say. **1,725** is the store's count; **100** is the page
+being displayed; and the two cards beneath are explicitly *of shown*. A page
+that reports its own length as the size of the corpus is a small lie that gets
+believed, so the page does not.
+
+## 6 · Promotion and correlation
+
+Two separate decisions, often confused:
+
+**Promotion** asks whether this event should become an alert at all.
+[`promoter.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/promoter.py)
+promotes an OCSF category-2 finding or anything at `severity_id >= 4`. Routine
+telemetry is not promoted; with the `full` profile it is still archived to the
+event lake and remains huntable.
+
+**Correlation** asks whether this alert belongs with others.
+[`correlator.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/correlator.py)
+keys on `{tenant}:{entity}:{tactic}` in Redis, so ten related alerts become one
+thing an analyst looks at rather than ten.
+
+Fusion also writes, at fuse time and without an LLM:
+
+- a deterministic **correlation narrative** ([`narrative.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/narrative.py)) — reads never call a model;
+- a **confidence** score, 0–100 with a `low | medium | high` band ([`confidence.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/confidence.py)), which is independent of severity;
+- **provenance** ([`provenance.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/provenance.py)) so the alert can say which connector fed it.
+
+The queue below is the output of exactly this path on a CORE stack — twelve
+alerts, each attributed to the connector whose profile normalized it, produced
+from telemetry pushed through the ingest API above.
+
+![The alerts queue on a live CORE stack](/img/console/alerts-queue.png)
+
+## 7 · The alert is written, and announced
+
+[`alert_sink.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/alert_sink.py)
+inserts the row into PostgreSQL and publishes to **`aisoc.alerts.fused`**.
+
+Two consumers pick it up.
+[`services/realtime`](https://github.com/beenuar/AiSOC/tree/main/services/realtime)
+(Node, container port 4000) pushes it to the console over WebSocket.
+[`services/agents`](https://github.com/beenuar/AiSOC/tree/main/services/agents)
+takes it for triage — the next step.
+
+## 8 · An agent triages it
+
+`FusedAlertTriageWorker` in
+[`services/agents`](https://github.com/beenuar/AiSOC/tree/main/services/agents)
+(container port 8084) auto-triages **every** fused alert.
+
+```mermaid
+sequenceDiagram
+ participant K as Kafka aisoc.alerts.fused
+ participant A as services/agents
+ participant C as Prompt contract
+ participant G as litellm gateway
+ participant M as Model
+ participant L as Investigation Ledger
+
+ K->>A: fused alert
+ A->>A: gather evidence — alert, siblings, entities, prior verdicts
+ A->>C: proposed prompt
+ C-->>A: refuse if it carries raw logs or secret-shaped values
+ C->>G: validated prompt, as alias aisoc-triage
+ G->>M: whichever model the alias resolves to
+ M-->>A: verdict, confidence, cited indicators
+ A->>A: groundedness check
+ Note over A: an indicator the evidence never<br/>contained demotes the verdict<br/>to human review
+ A->>L: prompt, tool calls, citations, verdict, tokens, cost
```
-This is the path used by the `aisoc submit` CLI command and the
-[`examples/alerts/lateral-movement.json`](https://github.com/beenuar/AiSOC/tree/main/examples/alerts/lateral-movement.json)
-canonical payload. The classic Kafka-fed pipeline above is still the
-**production** path; the direct-write endpoint exists so a fresh clone of
-the repo, an `aisoc serve` process, and a single `aisoc submit` call are
-enough to see an alert on `/alerts` — no broker, no schedulers, no
-connectors required.
-
-The `services/api` service holds the **encrypt** authority for the vault
-and is the only writer to `connectors.auth_config_encrypted`.
-`services/connectors` ships a vendored read-path `decrypt_dict()` that
-pairs to the same `AISOC_CREDENTIAL_KEY` so the scheduler can decrypt
-per-poll without owning the write path. Key rotation is supported via
-`MultiFernet` and the `AISOC_CREDENTIAL_KEY_ROTATION_FROM` env var; the
-[Operations: Credentials](./operations/credentials) page documents the
-full rotation procedure, threat model, and hosted-OAuth roadmap.
-
-The wizard's `Test connection` button skips the vault entirely — it
-forwards the raw form values to a stateless `POST /connectors/{id}/test`
-endpoint on the connectors microservice, which constructs the connector
-in memory and calls `test_connection()`. Bad credentials never touch the
-database.
-
-## Monorepo Layout
+Four properties, each enforced in code rather than asserted:
+
+- **The contract runs before the request.** A prompt about to carry raw OCSF, vendor log lines or secret-shaped values is refused, not redacted afterwards ([`prompt_sanitizer.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/app/investigator/prompt_sanitizer.py)).
+- **The model never writes a query.** It picks a typed tool and passes arguments; the tool writes the SQL or the traversal.
+- **Citations are checked.** A verdict citing an indicator that was not in the evidence is demoted rather than auto-closed. Groundedness is nullable, and `null` means *not assessed* — a different fact from zero.
+- **Every alias resolves at the gateway.** AiSOC asks for `aisoc-triage`, not for a model name; [`infra/litellm/config.yaml`](https://github.com/beenuar/AiSOC/blob/main/infra/litellm/config.yaml) maps aliases to models, so swapping local for hosted is three variables in `.env`.
+
+### What the default install actually does
+
+CORE ships a pinned `llama3.2:3b-instruct-q4_K_M` behind the gateway, so a
+clone with no account produces a real verdict with real token counts. It is a
+3B quantized model, and the difference from a frontier model is visible: in a
+measured run of 19 auto-triages on a CORE stack, **7 returned schema-valid
+output and 12 did not**. The 12 fell back to the deterministic path, logged
+`auto_triage_worker.llm_failed_fallback`, and the Investigation Rail shows
+which path produced the verdict it is displaying.
+
+**No hosted provider has ever been exercised in this repository.** There is no
+funded key, so per-model figures read *not measured* rather than zero. The
+local path is measured; the hosted path is configured and untested. Treat those
+as the different claims they are.
+
+Here is one of the 7, as the console renders it — the verdict, the confidence,
+the groundedness (*not assessed*, because this run did not score it), and the
+model's own rationale reproduced verbatim so a reader can tell which path
+answered:
+
+![A local-model triage verdict in the Investigation Rail](/img/console/ai-triage-verdict.png)
+
+## 9 · The reasoning is written down
+
+Every prompt, tool call, citation, verdict, token count and cost goes to the
+**Investigation Ledger**, append-only and tenant-scoped. The verdict is also
+projected back onto the alert row, which is what the Investigation Rail's
+*Automated triage* section renders.
+
+The point is not audit for its own sake: a verdict you cannot inspect is a
+verdict you have to trust, and this project does not ask you to.
+
+- Schema: [`008_investigation_ledger.sql`](https://github.com/beenuar/AiSOC/blob/main/services/api/migrations/008_investigation_ledger.sql)
+- Writer: [`ledger.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/app/investigator/ledger.py)
+- Reader: [`investigations.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/api/v1/endpoints/investigations.py)
+
+## 10 · A human approves
+Nothing executes on its own. An action is **proposed**; a human with the right
+permission approves it; and the approver may not be the person who requested
+it.
+
+Risk, reversibility, verification and approval tier are declared **per
+capability** — the contract belongs to the verb, not to the vendor that
+implements it. See [Live actions](./concepts/live-actions) and
+[Action approvals](./operations/action-approvals).
+
+## 11 · The action runs, and is verified
+
+[`services/actions`](https://github.com/beenuar/AiSOC/tree/main/services/actions)
+executes the approved capability against the vendor and then **probes the
+vendor to confirm it took effect**, rather than inferring success from an HTTP
+200. A capability whose effect cannot be verified is not eligible for automatic
+execution.
+
+Without vendor credentials an executor returns `simulated` and says so.
+`executed` is the single field that means a vendor was actually touched.
+
+Playbooks reach this same path. The engine's step vocabulary is 22 types
+([`models.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/app/playbook/models.py)),
+and the editor's palette offers **21** of them — `approval` is withheld because
+the engine is a single-threaded index walk with no pause or resume, so a step
+it cannot run is not offered as a button.
+
+---
+
+## Where the path stops
+
+Step 10 has no automatic trigger. There is no code path that dispatches a
+response without a human. That is deliberate, and it is the honest boundary of
+the word "autonomous" in this project.
+
+## What runs in which profile
+
+| Profile | Command | Long-running services | Memory | What you get |
+|---|---|---|---|---|
+| **core** | `make up` | 14 | ~8 GB | Steps 1–10, plus the LLM gateway, the local model behind it, and the CISA KEV feed with its vector store |
+| **full** | `make up-full` | 22 | ~12 GB | Core plus the event lake, entity graph, full-text search, enrichment and scheduled connectors |
+| **demo** | `make up && make demo` | 14 | ~8 GB | Core plus clearly-labelled synthetic data |
+
+CORE is 14 long-running containers; `ollama-pull` is a fifteenth that fetches
+the model once and exits. `full` is 22 long-running containers — not 30, which
+is every profile including `monitoring`, `chatops`, `extras` and `osquery`,
+none of which `make up-full` starts.
+
+Memory and disk are measured, not estimated: the model container reaches about
+3 GiB while answering, which is what moved the floor to 8 GB, and CORE is about
+16.5 GB of unique image layers plus a 2 GB model volume.
+
+## Which store holds what, and what breaks without it
+
+| Store | Profile | Holds | Without it |
+|---|---|---|---|
+| **PostgreSQL** | core | Alerts, incidents, cases, users, tenants, rules, audit log, ledger | Nothing works |
+| **Redis** | core | Correlation windows, dedup keys, investigation run state | No correlation — every alert becomes its own incident |
+| **Kafka** | core | `aisoc.raw_events`, `aisoc.alerts.fused`, `aisoc.alerts.dlq` | No spine; ingest would have to call every consumer directly |
+| **Qdrant** | core | Threat-intel indicators and actor embeddings | No Threat Intelligence page, no semantic IOC matching |
+| **ClickHouse** | full | Every normalized event | No event lake, no hunting over raw telemetry. **Alerting is unaffected** |
+| **Neo4j** | full | Entity graph, blast radius | No graph context. The `/graph` page reports the failure rather than drawing an invented graph |
+| **OpenSearch** | full | Full-text IOC and actor search | Threat-intel feeds still write to Qdrant; full-text IOC search is unavailable and says so |
+
+## How tenant isolation is actually enforced
+
+Per store, because one mechanism does not cover them all:
+
+| Store | Mechanism |
+|---|---|
+| PostgreSQL | Row-level security **plus** a `tenant_id` predicate at the query layer |
+| ClickHouse | `lake_sql.rewrite_for_tenant()` injects the predicate and **fails closed** if it does not survive rendering |
+| Neo4j | Every node of every path is scoped, not just the traversal's start node |
+| Redis | `tenant:{id}:` key prefix |
+| Qdrant | `tenant_id` in the point payload plus a mandatory query filter |
+| Kafka | `X-Tenant-ID` header / `tenant_id` envelope, filtered per consumer |
+
+Two details that are easy to get wrong and were:
+
+**Services do not connect as the database owner.** `aisoc` owns the schema and
+applies migrations; every service connects as `aisoc_app`, which holds
+`SELECT / INSERT / UPDATE / DELETE` and no DDL. This is not tidiness — a
+superuser ignores RLS policies even under `FORCE ROW LEVEL SECURITY`, so while
+services ran as the owner every policy in the schema filtered nothing. Check it
+on a running stack:
+
+```sql
+SELECT count(*) FROM pg_policies; -- 84 on a CORE stack here
+SELECT count(*) FROM pg_class WHERE relrowsecurity; -- 83 tables
+SELECT DISTINCT privilege_type FROM information_schema.role_table_grants
+ WHERE grantee = 'aisoc_app'; -- DML only, no DDL
```
-AiSOC/
-├── apps/
-│ ├── web/ # Next.js 14 React frontend (incl. Responder PWA)
-│ └── docs/ # This Docusaurus site
-├── services/
-│ ├── api/ # FastAPI gateway (port 8000)
-│ ├── agents/ # LangGraph investigator (port 8001)
-│ ├── realtime/ # Node/TS WebSocket + Web Push (port 8086)
-│ ├── ingest/ # Go OCSF normaliser (port 8081)
-│ ├── enrichment/ # Go enrichment fan-out (port 8080)
-│ ├── fusion/ # Fusion + ML scoring (port 8003)
-│ ├── actions/ # Action executor (port 8002)
-│ ├── threatintel/ # TAXII / MISP / OTX / KEV (port 8005)
-│ ├── ueba/ # User behavior analytics (port 8007)
-│ ├── honeytokens/ # Deception platform (port 8008)
-│ ├── purple-team/ # Adversary emulation (port 8006)
-│ ├── connectors/ # Connector polling + credential vault (50 vendors)
-│ ├── demo-producer/ # Synthetic event generator for demos
-│ ├── mcp/ # Model Context Protocol server (TypeScript)
-│ ├── osquery-tls/ # osquery TLS server: enrol hosts, ship results
-│ ├── osquery-extensions/ # Custom osquery tables/decorators
-│ └── slack-bot/ # ChatOps surface (HMAC-signed approvals)
-├── packages/
-│ ├── plugin-sdk-py/ # Python plugin SDK
-│ ├── plugin-sdk-go/ # Go plugin SDK
-│ ├── sdk-py/ # Python client SDK
-│ ├── sdk-ts/ # TypeScript client SDK
-│ └── sdk-go/ # Go models / client helpers
-├── infra/
-│ ├── helm/aisoc/ # Helm chart (Kubernetes, HA-ready)
-│ ├── terraform/ # Terraform modules
-│ ├── coolify/ # One-click deploy on Coolify
-│ ├── fly/ # Fly.io demo deployments
-│ ├── railway/ # Railway templates
-│ └── render/ # render.yaml blueprint
-├── detections/ # 200+ Sigma/YARA/KQL detection rules (YAML)
-├── hunts/ # Hunt-as-Code YAML definitions (hypothesis + indicators)
-├── playbooks/ # 50+ SOAR playbooks (YAML)
-├── plugins/ # 15 first-party plugins (Go + Python)
-├── marketplace/ # Marketplace index (index.json)
-├── docs/ # OpenAPI spec (openapi.yaml)
-├── docker-compose.yml # Full development stack
-├── infra/compose/ # docker-compose.{demo,dev,airgap}.yml overlays
-└── scripts/ # Utilities (seed, eval harness, build, validate)
+
+**The graph writer keys on `natural_key` with `tenant_id` as a property**, so
+a strict read filter fails closed: safe, never leaking, at some cost to
+completeness on shared infrastructure nodes such as public IPs.
+
+## When a step has nothing to show
+
+An honest empty state is part of this architecture, not a gap in it. The SOC
+operations dashboard is driven entirely by live API responses, and on a tenant
+with no connectors it says exactly that rather than rendering a placeholder
+chart:
+
+![SOC operations on a tenant with nothing connected](/img/console/soc-operations.png)
+
+The same rule holds for failures. With Neo4j absent — it is a `full`-profile
+store — `/graph` reports `API 503 … /api/v1/graph` instead of drawing a graph
+it does not have, and federated search names the four SIEMs it *would* query
+rather than showing an empty result set as though it had searched.
+
+## Verifying any of this yourself
+
+```bash
+make up && make smoke
```
-## Service Responsibilities
-
-| Service | Port | Language | Responsibility |
-|---------|------|----------|----------------|
-| `api` | 8000 | Python (FastAPI) | REST gateway, auth, RBAC, RLS, audit log, **Investigation Ledger**, **alert detail envelope** (Investigation Rail: narrative, entities, mini-timeline, actions), Ambient Copilot, marketplace, approvals, on-call, passkeys, push subscriptions, **Detection Proposals** (DAC lifecycle), **Federated Search** fan-out, SLA tracking, **Shifts** handoff, **STIX/TAXII** publishing, **Compliance evidence** collection, **Deployment / air-gap** configuration |
-| `agents` | 8001 | Python (LangGraph) | Orchestrator + recon + forensic + responder + report-writer agents, **autonomous triage agent** + phishing / identity / cloud / insider-threat sub-agents, **conversational investigation chat**, playbook engine, ledger writes, **Hunt-as-Code** engine + scheduler |
-| `realtime` | 8086 | TypeScript (Node.js) | WebSocket streaming of agent steps; **VAPID Web Push** delivery for the Responder PWA |
-| `ingest` | 8081 | Go | OCSF normalisation, Bloom-filter dedup, Kafka publish |
-| `enrichment` | 8080 | Go | Enrichment fan-out (IP, domain, hash, email, user) |
-| `fusion` | 8003 | Python | ML scoring (LightGBM + Isolation Forest), correlation, **deterministic correlation narratives** on fused alerts, **alert confidence scoring**, **entity risk / RBA** |
-| `actions` | 8002 | Python | Plugin action executor, blast-radius gating, **ChatOps verification** (HMAC-signed Slack/Teams prompts) |
-| `threatintel` | 8005 | Python | TAXII 2.1 / MISP / OTX / KEV ingestion + triple storage |
-| `ueba` | 8007 | Python | Welford baseline, Z-score scoring, anomaly stream |
-| `honeytokens` | 8008 | Python | Token lifecycle, HMAC signing, webhook dispatch |
-| `purple-team` | 8006 | Python | ART YAML parser, Caldera executor, ATT&CK heatmap, **detection drift snapshots** |
-| `connectors` | — | Python | Connector polling (APScheduler), credential vault (`CredentialVault`), registry-based discovery — **50 vendors shipped** across EDR / SIEM / cloud / IAM / SaaS / network / email |
-| `demo-producer` | — | Python | Synthetic event generator for demos and evaluation |
-| `mcp` | n/a | TypeScript | Model Context Protocol stdio server, 13 tools for IDE-side agents (Claude / Cursor / Continue / Cody) — discovery, deep-dive, action/replay, and the warm-tier lake query pair |
-| `osquery-tls` | 8443 | Go | TLS server implementing osquery's enrol/config/distributed/log endpoints; ships normalised host events into `ingest` |
-| `osquery-extensions` | — | Go | Out-of-band osquery extensions registering custom virtual tables and decorators consumed by `osquery-tls` |
-| `slack-bot` | — | Python | ChatOps surface: posts approval prompts, exposes `/aisoc` slash command, verifies inbound interactions with HMAC-signed Slack request signatures |
-| `web` | 3000 | TypeScript (Next.js) | React console + Responder PWA route group, **`/alerts` Investigation Rail**, **benchmark scoreboard**, conversational investigation chat |
-
-## Storage Tier
-
-| Store | Role |
-|-------|------|
-| PostgreSQL | Operational data, RLS-enforced multi-tenancy, audit log, **investigation ledger** |
-| ClickHouse | Time-series analytics, compliance metrics |
-| OpenSearch | Full-text search across alerts, logs, cases |
-| Qdrant | Semantic vector search for RAG copilot + agent memory |
-| Neo4j | Attack graph, entity relationships, blast-radius queries |
-| Redis | Cache, rate-limiting, session store, push subscription cache |
-| Kafka | Async event backbone |
-
-## Investigation Ledger
-
-Every agent action (LLM prompt, LLM response, tool call, evidence
-citation, decision branch) is appended to the `investigation_ledger`
-table, scoped to a case and stamped with the agent identity, model,
-prompt hash, and timestamp. The Case workspace renders this as a
-scrubbable timeline so analysts can replay the agent's reasoning.
-
-The schema is defined in
-[`services/api/migrations/008_investigation_ledger.sql`](https://github.com/beenuar/AiSOC/blob/main/services/api/migrations/008_investigation_ledger.sql).
-The agent-side writer lives in
-[`services/agents/app/investigator/ledger.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/app/investigator/ledger.py),
-and the UI consumer is
-[`apps/web/src/components/cases/InvestigationLedger.tsx`](https://github.com/beenuar/AiSOC/blob/main/apps/web/src/components/cases/InvestigationLedger.tsx).
-
-### Prompt sanitization layer
-
-Investigator agents (`recon`, `forensic`, `responder`, `report_writer`) consume
-attacker-influenced strings — enrichment payloads, dark-web excerpts, vendor
-descriptions, raw alert fields — and hand them to an LLM. Every one of those
-agents now routes its context through
-[`services/agents/app/investigator/prompt_sanitizer.py`](https://github.com/beenuar/AiSOC/blob/main/services/agents/app/investigator/prompt_sanitizer.py),
-which strips known role / chat delimiters, redacts common jailbreak phrasings,
-caps field length, bounds list size and recursion depth, and wraps the result
-in explicit `<UNTRUSTED_DATA>` tags. The agents still validate the LLM's
-response against a Pydantic schema before persisting it. See the
-[LLM prompt safety section in the security guide](./operations/security#llm-prompt-safety)
-for the threat model and defence-in-depth layers.
-
-## Investigation Rail and correlation narrative
-
-The alert queue (`/alerts`) pairs the sortable table with an **Investigation Rail**
-([`InvestigationRail.tsx`](https://github.com/beenuar/AiSOC/blob/main/apps/web/src/components/alerts/InvestigationRail.tsx))
-fed by `GET /api/v1/alerts/{id}`. The response envelope is assembled in
-[`services/api/app/services/alert_rail.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/services/alert_rail.py)
-(narrative text, entity buckets, merged mini-timeline, recommended actions).
-
-**Narrative** — At fusion time, `services/fusion` runs the same deterministic
-builder as the API vendored copy ([`narrative.py`](https://github.com/beenuar/AiSOC/blob/main/services/fusion/app/services/narrative.py))
-so promoted alerts persist a short explanation of *which* signals correlated
-and *why*. Reads do not call an LLM. Alerts created before this shipped get a
-lazy projection on first detail fetch via
-[`narrative_projection.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/services/narrative_projection.py),
-then the text is cached on the row (see migration `041_alert_correlation_narrative.sql`).
-
-**Sync** — When the narrative builder changes, run `scripts/sync_vendored_narrative.py`
-so `services/api/app/_vendor/narrative.py` stays aligned with fusion.
-
-Operator-facing detail: [Investigation Rail](./console/investigation-rail).
-
-## Responder PWA
-
-The Responder PWA is mounted under the Next.js route group
-`apps/web/src/app/(responder)/`. It is **passkey-only** (no passwords),
-shows the on-call rotation, lists pending approvals, supports VAPID
-Web Push for high-severity alerts, and ships an offline shell.
-
-The schema is defined in
-[`009_responder_pwa.sql`](https://github.com/beenuar/AiSOC/blob/main/services/api/migrations/009_responder_pwa.sql).
-The push pipeline lives in
-[`services/realtime/src/push.ts`](https://github.com/beenuar/AiSOC/blob/main/services/realtime/src/push.ts).
-
-## Enterprise Security Controls
-
-- **Multi-tenancy** — PostgreSQL Row-Level Security on every table; `tenant_id` is derived from the JWT and cannot be spoofed.
-- **RBAC** — `require_permission` FastAPI dependency; custom roles with fine-grained action permissions per resource type.
-- **SAML 2.0 / OIDC** — Pluggable SSO with JIT user provisioning and group-to-role mapping.
-- **WebAuthn / Passkeys** — Required for the Responder PWA; password-less by default.
-- **Immutable Audit Log** — Postgres trigger + `SECURITY DEFINER` function prevents UPDATE/DELETE on `audit_log`.
-- **Replayable agent decisions** — The Investigation Ledger is append-only and tenant-scoped.
-- **OpenTelemetry** — All services emit traces, metrics, and structured logs to a configurable OTLP endpoint.
-- **Backup & Restore** — `scripts/backup.sh` / `restore.sh`. Artifacts are encrypted with **AES-256-GCM** before leaving the host and recorded in a **SHA-256 manifest**; restore verifies the digest, decrypts, and refuses a tampered or truncated archive rather than restoring a partial database. Gated end-to-end in `integration.yml`: seed 500 rows, back up, assert the object in the bucket is unreadable (no gzip magic, no schema names), drop the schema, restore, assert 500 rows, then flip a bit and assert the restore is refused. Covers Postgres, ClickHouse, the plugin store, the **Neo4j entity graph** (APOC cypher export), **Qdrant** (per-collection snapshot) and **Redis** (RDB). Scheduled by the `backup` CronJob in the Helm chart — disabled by default because it needs a bucket, credentials and a key only the operator can supply. The RPO is the schedule interval. **Kafka is not backed up**: it is a transport, and its retained window is replayable from the lake rather than restorable.
-- **High-Availability Helm** — Multi-replica deployments, HPA, PDB, anti-affinity, and readiness probes.
-
-## Plugin Extension Points
-
-Plugins extend AiSOC at three key points:
-
-- **Enrichers** — Add context to indicators (IP, domain, hash, email)
-- **Actions** — Execute response steps (block IP, disable user, create ticket)
-- **Connectors** — Ingest events from external sources (SIEM, EDR, cloud)
-- **Widgets** — Render plugin-supplied React panels in the case workspace
-
-See [Plugin Overview](./plugins/overview) for the full plugin lifecycle.
+`make smoke` posts one event to the ingest API and follows it through Kafka,
+detection, promotion, correlation and Postgres, then reads it back from the
+public API. It reaches past nothing and each stage reports PASS or FAIL
+separately, so a break names the boundary that broke. Source:
+[`tests/e2e/golden_pipeline/`](https://github.com/beenuar/AiSOC/tree/main/tests/e2e/golden_pipeline).
+
+## Next
+
+- [Repository overview](./architecture/overview) — where every service and package lives
+- [Agent architecture](./architecture/agents) — the four agents and the graph they run on
+- [Graph schema](./architecture/graph-schema) — node labels and edge types
+- [Investigation Rail](./console/investigation-rail) — what an analyst sees
+- [Security](./operations/security) — threat model and controls
diff --git a/apps/docs/docs/architecture/overview.md b/apps/docs/docs/architecture/overview.md
index 41245dbcf..f46c26a7a 100644
--- a/apps/docs/docs/architecture/overview.md
+++ b/apps/docs/docs/architecture/overview.md
@@ -32,21 +32,24 @@ AiSOC/
│ ├── osquery-tls/ # Python · native osquery TLS server + FIM + pack distribution
│ ├── osquery-extensions/ # Go · AI threat-intel table + ML anomaly score table + 3 more
│ ├── slack-bot/ # Python · ChatOps surface (Slack Bolt + HMAC signature verification)
-│ └── mcp/ # TypeScript · Model Context Protocol server (npm: @aisoc/mcp, lands in v8.0)
+│ └── mcp/ # TypeScript · Model Context Protocol server (npm: @aisoc/mcp — built and packed on every tag, upload blocked on registry credentials)
├── packages/
│ ├── types/ # Shared TS types
│ ├── ui/ # Shared React primitives
│ ├── ocsf/ # OCSF normalization helpers
-│ ├── sdk-ts/ # TypeScript client SDK for AiSOC API (npm: @aisoc/sdk, lands in v8.0)
-│ ├── sdk-py/ # Async Python client SDK (PyPI: aisoc-sdk, lands in v8.0)
+│ ├── sdk-ts/ # TypeScript client SDK for AiSOC API (npm: @aisoc/sdk — unpublished, see below)
+│ ├── sdk-py/ # Async Python client SDK (PyPI: aisoc-sdk — unpublished, see below)
│ ├── sdk-go/ # Go client SDK + models (module: github.com/beenuar/AiSOC/packages/sdk-go)
│ ├── plugin-sdk-ts/ # TypeScript plugin development SDK
-│ ├── plugin-sdk-py/ # Python plugin development SDK (PyPI: aisoc-plugin-sdk, lands in v8.0)
+│ ├── plugin-sdk-py/ # Python plugin development SDK (PyPI: aisoc-plugin-sdk — unpublished, see below)
│ ├── plugin-sdk-go/ # Go plugin development SDK (module: github.com/beenuar/AiSOC/packages/plugin-sdk-go)
-│ └── aisoc-cli/ # CLI: scaffold / validate / publish plugins & detections (PyPI: aisoc-cli, lands in v8.0)
-├── detections/ # Community Sigma detection rules (YAML) — 800+ native
+│ └── aisoc-cli/ # CLI: scaffold / validate / publish plugins & detections (PyPI: aisoc-cli — unpublished, see below)
+├── detections/ # Detection rules as YAML. A *projection* of the compiled
+│ # ruleset, not the engine's input: 7,016 indexed on disk,
+│ # 5,937 of them quarantined, 833 executable. Recount with
+│ # scripts/generate_corpus_stats.py rather than trusting this line.
├── hunts/ # Hunt-as-Code YAML definitions (hypothesis + indicators + schedule)
-├── playbooks/ # Community SOAR playbooks (YAML)
+├── playbooks/ # SOAR playbooks (YAML) — 62 packs
├── plugins/ # First-party plugins (Go + Python) + marketplace manifests
├── marketplace/ # Marketplace index (JSON, generated by scripts/build_marketplace.py)
├── infra/
@@ -78,25 +81,36 @@ AiSOC/
## Service map
-| Service | Lang | Port | Role |
-|---|---|---|---|
-| `web` | Next.js + React | 3000 | SOC console (alerts detail + Investigation Rail), benchmark scoreboard, marketing landing |
-| `api` | Python · FastAPI | 8000 | Alerts (detail envelope + correlation narrative), cases, RBAC, graph, rules, audit, compliance, detection proposals (DAC), federated search fan-out, SLA tracking |
-| `realtime` | Node.js · `ws` | 8086 | Per-channel WebSocket fan-out + VAPID Web Push (short-lived signed tickets) |
-| `agents` | Python · LangGraph | 8001 | Multi-agent reasoning + Qdrant RAG + Hunt-as-Code engine & scheduler + Investigation Ledger writer |
-| `fusion` | Python | 8003 | Dedup + ML scoring (LightGBM, IsoForest), alert confidence, entity risk / RBA, correlation narrative projection for API |
-| `actions` | Python | 8002 | SOAR with blast-radius gating + ChatOps verification + signed approval URLs |
-| `connectors` | Python | — | Connector polling (APScheduler), credential vault, federated query translators |
-| `threatintel` | Python | 8005 | TAXII / MISP / OTX / KEV polling + threat-actor attribution engine |
-| `ueba` | Python | 8007 | User & Entity Behavior Analytics |
-| `honeytokens` | Python | 8008 | Honeytoken lifecycle + webhook alerting |
-| `purple-team` | Python | 8006 | Atomic Red Team + Caldera + ATT&CK heatmap + detection drift snapshots |
-| `osquery-tls` | Python | 8091 | Native osquery TLS server — enroll nodes, distribute packs, stream FIM/process/network telemetry |
-| `osquery-extensions` | Go | — | Custom osquery extensions (AI-powered threat intel table, ML anomaly score table, + 3 more) |
-| `slack-bot` | Python | 8009 | ChatOps surface — interactive approvals for high-blast-radius actions, `/aisoc` slash command |
-| `mcp` | TypeScript | — (stdio) | Model Context Protocol server exposing 13 AiSOC tools — discovery, deep-dive, action/replay, and warm-tier lake query |
-| `ingest` | Go | 8081 | OCSF normalization + Shodan/CVE + Neo4j graph-at-ingest writers |
-| `enrichment` | Go | 8080 | IOC enrichment (VT, AbuseIPDB, GreyNoise) |
+Two port columns, because conflating them has cost real time. **Container** is
+what the process listens on inside the Docker network, and is what another
+service must address. **Host** is what `docker-compose.yml` publishes on
+`127.0.0.1`, and is what your browser and `curl` use. They differ for most
+services.
+
+`litellm` and `ollama` are CORE — see [Architecture](../architecture.md) for
+why the gateway without a model behind it could not do AI triage at all.
+
+| Service | Lang | Container | Host | Profile | Role |
+|---|---|---|---|---|---|
+| `web` | Next.js | 3000 | 3000 | core | SOC console, Investigation Rail, marketing landing |
+| `api` | Python · FastAPI | 8000 | 8000 | core | Alerts, cases, RBAC, rules, audit, compliance, federated-search fan-out, ledger reads |
+| `ingest` | Go | 8080 · 9090 | 8081 · 9090 | core | OCSF normalization, Kafka publish, inbox webhooks, graph-at-ingest writers |
+| `fusion` | Python | 8003 | 8003 | core | Detection engine, promotion, correlation, confidence, narrative, lake writer |
+| `agents` | Python · LangGraph | 8084 | 8001 | core | Auto-triage worker, investigation agents, playbook engine, ledger writer |
+| `realtime` | Node.js · `ws` | 4000 | 8086 | core | Per-channel WebSocket fan-out, VAPID Web Push |
+| `threatintel` | Python | 8005 | 8005 | core | CISA KEV / TAXII / MISP / OTX polling, threat-actor attribution |
+| `litellm` | — (image) | 4000 | 4000 | core | LLM gateway; every `aisoc-<role>` alias resolves here and nowhere else |
+| `ollama` | — (image) | 11434 | 11434 | core | The local model behind the gateway, pinned at `llama3.2:3b-instruct-q4_K_M` |
+| `enrichment` | Go | 8082 | 8080 | full | IOC enrichment fan-out (VT, AbuseIPDB, GreyNoise) |
+| `actions` | Python | 8085 | 8002 | full | Governed capability execution and post-action verification |
+| `connectors` | Python | 8003 | 8088 | full | Scheduled vendor polling, credential vault, federated query translators |
+| `ueba` | Python | 8004 | 8007 | full | User and entity behaviour analytics |
+| `honeytokens` | Python | 8005 | 8008 | extras | Honeytoken lifecycle and webhook alerting |
+| `purple-team` | Python | 8006 | 8006 | extras | Atomic Red Team, Caldera, ATT&CK heatmap, detection drift |
+| `osquery-tls` | Python | 9001 | 8091 | osquery | osquery TLS server — enrol nodes, distribute packs, stream telemetry |
+| `slack-bot` | Python | 8089 | 8009 | chatops | ChatOps approvals and the `/aisoc` slash command |
+| `osquery-extensions` | Go | — | — | — | Custom osquery tables and decorators |
+| `mcp` | TypeScript | — (stdio) | — | — | Model Context Protocol server exposing AiSOC tools to IDE-side agents |
## Storage tier
@@ -112,7 +126,16 @@ AiSOC/
## API surface
-The full OpenAPI 3.1 spec lives at [`docs/openapi.yaml`](https://github.com/beenuar/AiSOC/blob/main/docs/openapi.yaml). Endpoint groups are summarised in [Architecture](../architecture.md#api-surface) and the interactive view is at `http://localhost:8000/api/docs` (Swagger) or `/api/redoc` (ReDoc) when you run the local stack.
+The full OpenAPI 3.1 spec lives at [`docs/openapi.yaml`](https://github.com/beenuar/AiSOC/blob/main/docs/openapi.yaml). The interactive view is at `http://localhost:8000/api/docs` (Swagger) or `/api/redoc` (ReDoc) when you run the local stack.
+
+## On "unpublished"
+
+`release.yml` builds and packs all eight packages on every tag, so the pipeline
+cannot rot. Only the upload is gated, and it is gated on registry credentials
+this repository does not hold — an account action, not an engineering one.
+Install from source in the meantime. The npm name `aisoc` is free; the PyPI
+name `aisoc` belongs to an unrelated project, which is why the Python sandbox
+package ships as `aisoc-sandbox`.
## Next stops
diff --git a/apps/docs/docs/contributing/dev-setup.md b/apps/docs/docs/contributing/dev-setup.md
index 4fda87919..579f998bf 100644
--- a/apps/docs/docs/contributing/dev-setup.md
+++ b/apps/docs/docs/contributing/dev-setup.md
@@ -153,7 +153,7 @@ The other Python services (`fusion`, `actions`, `threatintel`, `ueba`,
virtualenv and run `uvicorn app.main:app --reload --port <port>`.
The canonical port mapping is in
[`docker-compose.yml`](https://github.com/beenuar/AiSOC/blob/main/docker-compose.yml)
-and the [Architecture](../architecture#service-responsibilities) page.
+and the [service map](../architecture/overview#service-map) page.
## 7. Run tests
diff --git a/apps/docs/static/img/console/ai-triage-verdict.png b/apps/docs/static/img/console/ai-triage-verdict.png
new file mode 100644
index 000000000..a7eb96a25
Binary files /dev/null and b/apps/docs/static/img/console/ai-triage-verdict.png differ
diff --git a/apps/docs/static/img/console/alerts-queue.png b/apps/docs/static/img/console/alerts-queue.png
new file mode 100644
index 000000000..83e7afcdc
Binary files /dev/null and b/apps/docs/static/img/console/alerts-queue.png differ
diff --git a/apps/docs/static/img/console/soc-operations.png b/apps/docs/static/img/console/soc-operations.png
new file mode 100644
index 000000000..e154f83fd
Binary files /dev/null and b/apps/docs/static/img/console/soc-operations.png differ
diff --git a/apps/docs/static/img/console/threat-intel-kev.png b/apps/docs/static/img/console/threat-intel-kev.png
new file mode 100644
index 000000000..6afabab22
Binary files /dev/null and b/apps/docs/static/img/console/threat-intel-kev.png differ
diff --git a/apps/web/public/screenshots/01-alerts-queue.svg b/apps/web/public/screenshots/01-alerts-queue.svg
deleted file mode 100644
index 03d1571b4..000000000
--- a/apps/web/public/screenshots/01-alerts-queue.svg
+++ /dev/null
@@ -1,13 +0,0 @@
-<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1440 900" role="img" aria-label="Alerts queue screenshot placeholder">
- <defs>
- <linearGradient id="bg" x1="0" y1="0" x2="0" y2="1">
- <stop offset="0" stop-color="#0f172a"/>
- <stop offset="1" stop-color="#1e293b"/>
- </linearGradient>
- </defs>
- <rect width="1440" height="900" fill="url(#bg)"/>
- <text x="720" y="380" text-anchor="middle" fill="#e2e8f0" font-family="ui-sans-serif, system-ui, sans-serif" font-size="40" font-weight="700">Alerts queue  server-anchored SLA countdowns</text>
- <text x="720" y="440" text-anchor="middle" fill="#94a3b8" font-family="ui-sans-serif, system-ui, sans-serif" font-size="22">Real screenshot ships with the Phase 2 visual rollup.</text>
- <text x="720" y="480" text-anchor="middle" fill="#64748b" font-family="ui-sans-serif, system-ui, sans-serif" font-size="20">Generate locally: pnpm aisoc:demo --screenshots</text>
- <text x="720" y="540" text-anchor="middle" fill="#475569" font-family="ui-monospace, monospace" font-size="18">apps/web/public/screenshots/01-alerts-queue.png</text>
-</svg>
diff --git a/apps/web/public/screenshots/02-investigation-rail.svg b/apps/web/public/screenshots/02-investigation-rail.svg
deleted file mode 100644
index 06a692814..000000000
--- a/apps/web/public/screenshots/02-investigation-rail.svg
+++ /dev/null
@@ -1,13 +0,0 @@
-<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1440 900" role="img" aria-label="Investigation Rail screenshot placeholder">
- <defs>
- <linearGradient id="bg" x1="0" y1="0" x2="0" y2="1">
- <stop offset="0" stop-color="#0f172a"/>
- <stop offset="1" stop-color="#1e293b"/>
- </linearGradient>
- </defs>
- <rect width="1440" height="900" fill="url(#bg)"/>
- <text x="720" y="380" text-anchor="middle" fill="#e2e8f0" font-family="ui-sans-serif, system-ui, sans-serif" font-size="40" font-weight="700">Investigation Rail  narrative + pivot-path entities</text>
- <text x="720" y="440" text-anchor="middle" fill="#94a3b8" font-family="ui-sans-serif, system-ui, sans-serif" font-size="22">Real screenshot ships with the Phase 2 visual rollup.</text>
- <text x="720" y="480" text-anchor="middle" fill="#64748b" font-family="ui-sans-serif, system-ui, sans-serif" font-size="20">Generate locally: pnpm aisoc:demo --screenshots</text>
- <text x="720" y="540" text-anchor="middle" fill="#475569" font-family="ui-monospace, monospace" font-size="18">apps/web/public/screenshots/02-investigation-rail.png</text>
-</svg>
diff --git a/apps/web/public/screenshots/03-hunt-workbench.svg b/apps/web/public/screenshots/03-hunt-workbench.svg
deleted file mode 100644
index 72446b228..000000000
--- a/apps/web/public/screenshots/03-hunt-workbench.svg
+++ /dev/null
@@ -1,13 +0,0 @@
-<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1440 900" role="img" aria-label="Hunt workbench screenshot placeholder">
- <defs>
- <linearGradient id="bg" x1="0" y1="0" x2="0" y2="1">
- <stop offset="0" stop-color="#0f172a"/>
- <stop offset="1" stop-color="#1e293b"/>
- </linearGradient>
- </defs>
- <rect width="1440" height="900" fill="url(#bg)"/>
- <text x="720" y="380" text-anchor="middle" fill="#e2e8f0" font-family="ui-sans-serif, system-ui, sans-serif" font-size="40" font-weight="700">/hunt workbench  NL prompt to ES|QL / SPL / KQL</text>
- <text x="720" y="440" text-anchor="middle" fill="#94a3b8" font-family="ui-sans-serif, system-ui, sans-serif" font-size="22">Real screenshot ships with the Phase 2 visual rollup.</text>
- <text x="720" y="480" text-anchor="middle" fill="#64748b" font-family="ui-sans-serif, system-ui, sans-serif" font-size="20">Generate locally: pnpm aisoc:demo --screenshots</text>
- <text x="720" y="540" text-anchor="middle" fill="#475569" font-family="ui-monospace, monospace" font-size="18">apps/web/public/screenshots/03-hunt-workbench.png</text>
-</svg>
diff --git a/apps/web/public/screenshots/04-marketplace.svg b/apps/web/public/screenshots/04-marketplace.svg
deleted file mode 100644
index a544d5026..000000000
--- a/apps/web/public/screenshots/04-marketplace.svg
+++ /dev/null
@@ -1,13 +0,0 @@
-<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1440 900" role="img" aria-label="Marketplace screenshot placeholder">
- <defs>
- <linearGradient id="bg" x1="0" y1="0" x2="0" y2="1">
- <stop offset="0" stop-color="#0f172a"/>
- <stop offset="1" stop-color="#1e293b"/>
- </linearGradient>
- </defs>
- <rect width="1440" height="900" fill="url(#bg)"/>
- <text x="720" y="380" text-anchor="middle" fill="#e2e8f0" font-family="ui-sans-serif, system-ui, sans-serif" font-size="40" font-weight="700">Marketplace  plugins, playbooks, detections</text>
- <text x="720" y="440" text-anchor="middle" fill="#94a3b8" font-family="ui-sans-serif, system-ui, sans-serif" font-size="22">Real screenshot ships with the Phase 2 visual rollup.</text>
- <text x="720" y="480" text-anchor="middle" fill="#64748b" font-family="ui-sans-serif, system-ui, sans-serif" font-size="20">Generate locally: pnpm aisoc:demo --screenshots</text>
- <text x="720" y="540" text-anchor="middle" fill="#475569" font-family="ui-monospace, monospace" font-size="18">apps/web/public/screenshots/04-marketplace.png</text>
-</svg>
diff --git a/apps/web/public/screenshots/README.md b/apps/web/public/screenshots/README.md
index 8cc0788dd..b011b4d42 100644
--- a/apps/web/public/screenshots/README.md
+++ b/apps/web/public/screenshots/README.md
@@ -1,48 +1,77 @@
-# `apps/web/public/screenshots/` — README hero screenshots
+# `apps/web/public/screenshots/` — real console captures
-This directory holds the four product screenshots that appear in the
-`README.md` hero grid:
+Every image here is a screenshot of a running AiSOC deployment. None is a
+mockup, a design comp, or a demo-mode capture.
-| Slot | Final path | Today (placeholder) | Caption |
-|---|---|---|---|
-| 1 | `01-alerts-queue.png` | `01-alerts-queue.svg` | Alerts queue — server-anchored SLA countdowns |
-| 2 | `02-investigation-rail.png` | `02-investigation-rail.svg` | Investigation Rail — deterministic correlation narrative |
-| 3 | `03-hunt-workbench.png` | `03-hunt-workbench.svg` | `/hunt` workbench — NL → ES&#124;QL / SPL / KQL |
-| 4 | `04-marketplace.png` | `04-marketplace.svg` | Marketplace — plugins, playbooks, detections |
+**Nothing in this directory was taken with `make demo`.** The synthetic demo
+dataset exists and is legitimate, but a screenshot of it presented as the
+product working on real data would be the exact thing this repository has a
+standing rule against. If a capture is ever added from demo mode, it must say
+so in the table below and in every caption that uses it.
-## Today
+## How these were produced
-The four `.svg` placeholders ship with the [Phase 2 visuals rollup](../../../../README.md) so the
-README image grid never shows a broken image. They are 1440×900 (the
-viewport the screenshot spec captures at) and labelled with the path
-they will eventually be replaced by.
+```bash
+cp .env.example .env
+make up # CORE: 14 long-running services, no credentials of any kind
+make smoke # one real event through the real spine
+```
-## After the real screenshots land
+Then telemetry was pushed through the documented ingest API
+(`POST /v1/ingest/batch`) and the console was driven with Playwright at
+1440×900, signed in as the administrator `make up` created.
-When the next maintainer captures the production screenshots:
+One detail, so the record is exact: these were captured immediately before
+ingest authentication landed, so the pushes carried no token. That changed what
+the *door* requires and nothing about what came through it — normalization,
+detection, promotion, correlation and triage are the same code either way.
+Re-capturing now would need `make ingest-token` first.
-1. Boot the demo:
+What is real in these images, stated precisely:
- ```bash
- pnpm aisoc:demo --no-open --quick
- ```
+- **The deployment.** Fourteen containers, the images built from this
+ repository, Postgres/Kafka/Qdrant/Ollama running locally.
+- **The pipeline.** Normalization, detection, promotion, correlation and
+ alerting were all performed by the running services. No row was seeded into
+ the database.
+- **The threat intelligence.** 1,725 CISA Known Exploited Vulnerabilities
+ entries, fetched from the public catalogue at capture time. Not ours, not
+ synthetic, unmodified.
+- **The AI verdict.** Produced by the bundled local `llama3.2:3b-instruct-q4_K_M`
+ through the LiteLLM gateway, with real token counts and latency in the
+ Investigation Ledger. Not a hosted provider — no key was configured, and
+ none has ever been exercised here.
+- **The empty and degraded states.** Genuinely empty and genuinely degraded.
+ They are as much of the product as the populated ones.
-2. Capture the four PNGs (writes the files into this directory):
+What is *authored* rather than observed: the security events themselves. They
+were written by hand to be representative and pushed through the ingest API.
+They are not a real intrusion and no caption may imply they are.
- ```bash
- pnpm aisoc:demo --screenshots
- ```
+## The captures
-3. Sanity-check the diff and open a PR:
+| File | Surface | What is worth seeing |
+|---|---|---|
+| `dashboard.png` | `/dashboard` | Operations funnel over real counts. `MTTR` reads *not measured · no cases closed* rather than `0`. |
+| `alerts-queue.png` | `/alerts` | Twelve alerts the pipeline produced, each attributed to the connector that fed it. |
+| `investigation-rail.png` | `/alerts` → row | The rail's Story view. Progression says *no ATT&CK tactic is mapped … that is a gap in the detection's metadata rather than a statement about the activity.* |
+| `ai-triage-verdict.png` | `/alerts` → row → Details | A real local-model verdict: `true positive`, confidence 80/100, groundedness *not assessed*, and the model's own rationale verbatim. |
+| `threat-intel-kev.png` | `/threat-intel` | 1,725 real KEV entries; the page distinguishes the catalogue size from the page it is showing. |
+| `soc-operations.png` | `/dashboards/operations` | Pipeline health against a tenant with no connectors — *"Nothing is feeding the pipeline yet, so an empty alert queue is expected."* |
+| `connectors.png` | `/connectors` | The connector surface with nothing connected. |
+| `connector-wizard.png` | `/connectors` → Add connector | The catalogue, reporting **84 connector types available** — the generated count, not a hand-typed one. |
+| `attack-graph.png` | `/graph` | The entity graph over real ingested telemetry, with Neo4j running. |
+| `attack-graph-core-degraded.png` | `/graph` | The same page in CORE, where Neo4j is not running: it names the failure (`API 503 … /api/v1/graph`) instead of drawing an invented graph. |
+| `federated-search.png` | `/federated-search` | Honest empty state — no SIEM connected, and it says which four it would query. |
+| `playbooks.png` | `/playbooks` | The playbook library. |
+| `playbook-editor.png` | `/playbooks/{id}` → Add step | The step palette. Twenty-one tiles, from a twenty-two-member vocabulary: `approval` is withheld because the engine has no pause/resume. |
+| `detection-rules.png` | `/detection` | The detection rule surface. |
+| `hunt.png` | `/hunt` | The hunt workbench. |
+| `cases-empty.png` | `/cases` | No cases, said plainly. |
- ```bash
- git status -s apps/web/public/screenshots/
- git add apps/web/public/screenshots/*.png
- ```
+## Refreshing them
-4. In the same PR, swap the four `.svg` references in `README.md`
- to `.png`. Until step 4 lands the README intentionally points at
- the placeholders so a fresh clone never shows a broken image.
-
-The Playwright spec that drives the capture lives at
-[`apps/web/e2e/screenshots/console.spec.ts`](../../e2e/screenshots/console.spec.ts).
+`.github/workflows/console-visuals.yml` can regenerate console tiles on a
+schedule, but it drives the **seeded demo stack**. Anything it produces is
+demo-mode output and must be labelled as such wherever it appears — which is
+why the captures above were taken by hand against a real CORE stack instead.
diff --git a/apps/web/public/screenshots/ai-triage-verdict.png b/apps/web/public/screenshots/ai-triage-verdict.png
new file mode 100644
index 000000000..a7eb96a25
Binary files /dev/null and b/apps/web/public/screenshots/ai-triage-verdict.png differ
diff --git a/apps/web/public/screenshots/alerts-queue.png b/apps/web/public/screenshots/alerts-queue.png
new file mode 100644
index 000000000..83e7afcdc
Binary files /dev/null and b/apps/web/public/screenshots/alerts-queue.png differ
diff --git a/apps/web/public/screenshots/attack-graph-core-degraded.png b/apps/web/public/screenshots/attack-graph-core-degraded.png
new file mode 100644
index 000000000..03ad2e6e1
Binary files /dev/null and b/apps/web/public/screenshots/attack-graph-core-degraded.png differ
diff --git a/apps/web/public/screenshots/attack-graph.png b/apps/web/public/screenshots/attack-graph.png
new file mode 100644
index 000000000..b9c39c2d3
Binary files /dev/null and b/apps/web/public/screenshots/attack-graph.png differ
diff --git a/apps/web/public/screenshots/cases-empty.png b/apps/web/public/screenshots/cases-empty.png
new file mode 100644
index 000000000..80f55b6b7
Binary files /dev/null and b/apps/web/public/screenshots/cases-empty.png differ
diff --git a/apps/web/public/screenshots/connector-wizard.png b/apps/web/public/screenshots/connector-wizard.png
new file mode 100644
index 000000000..38bbb4d20
Binary files /dev/null and b/apps/web/public/screenshots/connector-wizard.png differ
diff --git a/apps/web/public/screenshots/connectors.png b/apps/web/public/screenshots/connectors.png
new file mode 100644
index 000000000..2567a3de3
Binary files /dev/null and b/apps/web/public/screenshots/connectors.png differ
diff --git a/apps/web/public/screenshots/dashboard.png b/apps/web/public/screenshots/dashboard.png
new file mode 100644
index 000000000..e1a2a4e69
Binary files /dev/null and b/apps/web/public/screenshots/dashboard.png differ
diff --git a/apps/web/public/screenshots/detection-rules.png b/apps/web/public/screenshots/detection-rules.png
new file mode 100644
index 000000000..06a8ea322
Binary files /dev/null and b/apps/web/public/screenshots/detection-rules.png differ
diff --git a/apps/web/public/screenshots/federated-search.png b/apps/web/public/screenshots/federated-search.png
new file mode 100644
index 000000000..eefbc39ca
Binary files /dev/null and b/apps/web/public/screenshots/federated-search.png differ
diff --git a/apps/web/public/screenshots/hunt.png b/apps/web/public/screenshots/hunt.png
new file mode 100644
index 000000000..48bb41666
Binary files /dev/null and b/apps/web/public/screenshots/hunt.png differ
diff --git a/apps/web/public/screenshots/investigation-rail.png b/apps/web/public/screenshots/investigation-rail.png
new file mode 100644
index 000000000..7243ee6c1
Binary files /dev/null and b/apps/web/public/screenshots/investigation-rail.png differ
diff --git a/apps/web/public/screenshots/playbook-editor.png b/apps/web/public/screenshots/playbook-editor.png
new file mode 100644
index 000000000..fa1a8aa9e
Binary files /dev/null and b/apps/web/public/screenshots/playbook-editor.png differ
diff --git a/apps/web/public/screenshots/playbooks.png b/apps/web/public/screenshots/playbooks.png
new file mode 100644
index 000000000..3c58bb501
Binary files /dev/null and b/apps/web/public/screenshots/playbooks.png differ
diff --git a/apps/web/public/screenshots/soc-operations.png b/apps/web/public/screenshots/soc-operations.png
new file mode 100644
index 000000000..e154f83fd
Binary files /dev/null and b/apps/web/public/screenshots/soc-operations.png differ
diff --git a/apps/web/public/screenshots/threat-intel-kev.png b/apps/web/public/screenshots/threat-intel-kev.png
new file mode 100644
index 000000000..6afabab22
Binary files /dev/null and b/apps/web/public/screenshots/threat-intel-kev.png differ
diff --git a/apps/web/src/components/alerts/InvestigationRail.test.tsx b/apps/web/src/components/alerts/InvestigationRail.test.tsx
index f1873e25e..0a4b97288 100644
--- a/apps/web/src/components/alerts/InvestigationRail.test.tsx
+++ b/apps/web/src/components/alerts/InvestigationRail.test.tsx
@@ -378,6 +378,82 @@ describe('InvestigationRail — header', () => {
// ─── Loaded rail — narrative ─────────────────────────────────────────────────
+describe('InvestigationRail — automated triage section', () => {
+ // `services/agents` triages every fused alert and writes the verdict back
+ // onto the row. `normalizeAlert` dropped `ai_score`, `ai_summary` and
+ // `triage_groundedness`, so the one output of the AI the product leads with
+ // was metered, persisted, and rendered by nothing.
+ it('renders the verdict, the confidence and the rationale verbatim', () => {
+ swrState.data = buildAlert({
+ disposition: 'true_positive',
+ aiScore: 0.8,
+ aiSummary:
+ 'LLM auto-triage verdict: true_positive · LLM confidence: 0.80 · Rationale: mass rename to .locked on the finance share.',
+ });
+ renderDetails();
+
+ const section = screen.getByText('Automated triage').closest('section')!;
+ expect(within(section).getByText('true positive')).toBeInTheDocument();
+ expect(within(section).getByText(/confidence 80\/100/)).toBeInTheDocument();
+ // Verbatim: the `LLM auto-triage verdict:` prefix is how a reader tells
+ // the model's own text from the deterministic fallback's.
+ expect(within(section).getByText(/LLM auto-triage verdict: true_positive/)).toBeInTheDocument();
+ });
+
+ it('says groundedness was not assessed rather than showing it as zero', () => {
+ // The deterministic path never scores groundedness, and the column is
+ // nullable for that reason. Rendering null as 0% would read as a verdict
+ // that cited nothing real — the opposite of what happened.
+ swrState.data = buildAlert({
+ disposition: 'likely_benign' as Alert['disposition'],
+ aiScore: 0.25,
+ aiSummary: 'high-severity keyword match in alert text · hostname present',
+ triageGroundedness: null,
+ });
+ renderDetails();
+
+ const section = screen.getByText('Automated triage').closest('section')!;
+ expect(within(section).getByText(/groundedness not assessed/)).toBeInTheDocument();
+ expect(within(section).queryByText(/groundedness 0%/)).toBeNull();
+ });
+
+ it('renders a real groundedness score when one was assessed', () => {
+ swrState.data = buildAlert({ aiScore: 0.9, aiSummary: 'verdict', triageGroundedness: 0.75 });
+ renderDetails();
+
+ expect(
+ within(screen.getByText('Automated triage').closest('section')!).getByText(/groundedness 75%/),
+ ).toBeInTheDocument();
+ });
+
+ it('names indicators the verdict cited but the evidence never held', () => {
+ swrState.data = buildAlert({
+ aiScore: 0.6,
+ aiSummary: 'verdict',
+ triageUngrounded: ['198.51.100.9', 'evil.example'],
+ });
+ renderDetails();
+
+ const section = screen.getByText('Automated triage').closest('section')!;
+ expect(within(section).getByText(/198\.51\.100\.9, evil\.example/)).toBeInTheDocument();
+ });
+
+ it('distinguishes "not triaged yet" from "nothing to show"', () => {
+ // Absent and pending are different facts, and only one of them is a
+ // reason to go and look at the agents service. Rendering nothing said
+ // neither.
+ swrState.data = buildAlert({
+ disposition: null,
+ aiScore: undefined,
+ aiSummary: undefined,
+ });
+ renderDetails();
+
+ expect(screen.getByText('Automated triage')).toBeInTheDocument();
+ expect(screen.getByText(/not triaged yet/i)).toBeInTheDocument();
+ });
+});
+
describe('InvestigationRail — narrative section', () => {
it('renders the narrative when present', () => {
swrState.data = buildAlert();
diff --git a/apps/web/src/components/alerts/InvestigationRail.tsx b/apps/web/src/components/alerts/InvestigationRail.tsx
index e1dd9b7d2..302eae646 100644
--- a/apps/web/src/components/alerts/InvestigationRail.tsx
+++ b/apps/web/src/components/alerts/InvestigationRail.tsx
@@ -146,6 +146,7 @@ export function InvestigationRail({ alertId, onClose }: InvestigationRailProps)
</div>
) : (
<>
+ <AutomatedTriageSection alert={alert} />
<NarrativeSection narrative={alert.narrative ?? null} />
<RelatedEntitiesSection entities={alert.relatedEntities ?? []} />
<MiniTimelineSection events={alert.miniTimeline ?? []} />
@@ -350,6 +351,78 @@ function SectionHeader({ title, count }: { title: string; count?: number }) {
);
}
+/**
+ * What the auto-triage agent concluded, and how far to trust it.
+ *
+ * `services/agents` triages every fused alert and writes the verdict back to
+ * the row. Nothing rendered it: `normalizeAlert` dropped `ai_score`,
+ * `ai_summary` and `triage_groundedness` on the floor, so the product's
+ * headline capability produced tokens, cost and a ledger entry that no
+ * console surface showed.
+ *
+ * Three things are deliberate here.
+ *
+ * The rationale is rendered **verbatim**. The model's own text arrives
+ * prefixed `LLM auto-triage verdict:`; the deterministic fallback reads as a
+ * description of the signals it matched. Showing it unedited is what lets a
+ * reader tell which path answered, and the bundled local model falls back
+ * often enough that the distinction matters on a default install.
+ *
+ * Groundedness renders as **"not assessed"** when the column is null, never
+ * as 0. The deterministic path does not score it, and 0 would read as a
+ * verdict citing nothing real.
+ *
+ * An alert that has not been triaged yet says so, rather than rendering
+ * nothing — absent and pending are different facts, and only one of them is
+ * a reason to go and look at the agents service.
+ */
+function AutomatedTriageSection({ alert }: { alert: Alert }) {
+ const verdict = alert.disposition ?? null;
+ const rationale = (alert.aiSummary ?? '').trim();
+ const hasRun = Boolean(verdict || rationale || typeof alert.aiScore === 'number');
+
+ return (
+ <section className="px-4 py-4 border-b border-gray-800/40">
+ <SectionHeader title="Automated triage" />
+ {!hasRun ? (
+ <p className="text-xs text-gray-500">
+ Not triaged yet. Every fused alert is queued for the triage agent; if this stays empty,
+ check the agents service.
+ </p>
+ ) : (
+ <div className="space-y-2">
+ <div className="flex items-center gap-2 flex-wrap">
+ {verdict && (
+ <span className="text-xs px-2 py-0.5 rounded border border-gray-700 bg-gray-800/60 text-gray-200">
+ {verdict.replace(/_/g, ' ')}
+ </span>
+ )}
+ {typeof alert.aiScore === 'number' && (
+ <span className="text-xs text-gray-400">
+ confidence {Math.round(alert.aiScore * 100)}/100
+ </span>
+ )}
+ <span className="text-xs text-gray-500">
+ groundedness{' '}
+ {typeof alert.triageGroundedness === 'number'
+ ? `${Math.round(alert.triageGroundedness * 100)}%`
+ : 'not assessed'}
+ </span>
+ </div>
+ {rationale && (
+ <p className="text-xs text-gray-300 leading-relaxed whitespace-pre-wrap">{rationale}</p>
+ )}
+ {alert.triageUngrounded && alert.triageUngrounded.length > 0 && (
+ <p className="text-xs text-amber-400">
+ Cited but not present in the evidence: {alert.triageUngrounded.join(', ')}
+ </p>
+ )}
+ </div>
+ )}
+ </section>
+ );
+}
+
function NarrativeSection({ narrative }: { narrative: string | null }) {
if (!narrative || !narrative.trim()) return null;
// `build_narrative` emits a markdown-light dialect. Preserving newlines in a
diff --git a/apps/web/src/components/threat-intel/ThreatIntelView.tsx b/apps/web/src/components/threat-intel/ThreatIntelView.tsx
index 3d2eeb5dc..9a5bb4609 100644
--- a/apps/web/src/components/threat-intel/ThreatIntelView.tsx
+++ b/apps/web/src/components/threat-intel/ThreatIntelView.tsx
@@ -245,6 +245,10 @@ export function ThreatIntelView() {
// "3 Added Today" counter on a tenant that had never ingested one.
const allIndicators = data?.indicators ?? [];
+ // The store's count, not this page's length. Falls back to the page length
+ // only when the API omits the field, which is the pre-`shown` contract.
+ const totalCollected = data?.total ?? allIndicators.length;
+
const indicators = allIndicators.filter((ioc) => {
if (typeFilter !== 'all' && ioc.type !== typeFilter) return false;
if (
@@ -275,13 +279,22 @@ export function ThreatIntelView() {
</p>
</div>
- {/* Stats */}
+ {/* Stats
+ *
+ * Every figure here is either the store's own count or a count of the
+ * rows on screen, and each card says which. It previously published
+ * neither: "Total IOCs" was the length of one page — 100, against 1,725
+ * collected — and "Added Today" was the literal `3`, hard-coded, on
+ * every deployment including ones that had never ingested an indicator.
+ * The comment above `allIndicators` describes removing the invented IOC
+ * list for exactly that reason; the counter the same paragraph names
+ * survived it. */}
<div className="grid grid-cols-4 gap-3">
{[
- { label: 'Total IOCs', value: allIndicators.length, color: 'text-blue-400' },
- { label: 'Malicious', value: allIndicators.filter(i => i.malicious).length, color: 'text-red-400' },
- { label: 'High Confidence', value: allIndicators.filter(i => i.confidence >= 80).length, color: 'text-orange-400' },
- { label: 'Added Today', value: 3, color: 'text-green-400' },
+ { label: 'Indicators collected', value: totalCollected, color: 'text-blue-400' },
+ { label: 'Shown below', value: allIndicators.length, color: 'text-gray-200' },
+ { label: 'Malicious (of shown)', value: allIndicators.filter(i => i.malicious).length, color: 'text-red-400' },
+ { label: 'High confidence (of shown)', value: allIndicators.filter(i => i.confidence >= 80).length, color: 'text-orange-400' },
].map((stat) => (
<div key={stat.label} className="bg-gray-900/60 border border-gray-800/60 rounded-xl p-4">
<p className={clsx('text-2xl font-bold mb-1', stat.color)}>{stat.value}</p>
@@ -330,7 +343,11 @@ export function ThreatIntelView() {
<div className="bg-gray-900/60 border border-gray-800/60 rounded-xl p-4">
<div className="flex items-center justify-between mb-3">
<h3 className="text-sm font-medium text-gray-300">Indicators of Compromise</h3>
- <span className="text-xs text-gray-500">{indicators.length} indicators</span>
+ <span className="text-xs text-gray-500">
+ {totalCollected > indicators.length
+ ? `${indicators.length.toLocaleString()} of ${totalCollected.toLocaleString()} indicators`
+ : `${indicators.length.toLocaleString()} indicators`}
+ </span>
</div>
{indicators.length === 0 ? (
// WS-F5 — TI feeds rarely have a true "no IOCs ever" state in
diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.ts
index 5e40cdfe8..bd372a819 100644
--- a/apps/web/src/lib/api.ts
+++ b/apps/web/src/lib/api.ts
@@ -674,6 +674,29 @@ export interface Alert {
relatedEntities?: RelatedEntity[];
miniTimeline?: MiniTimelineEvent[];
recommendedActions?: RecommendedAction[];
+ // ─── Automated triage ────────────────────────────────────────────────────
+ //
+ // `services/agents` auto-triages every fused alert and writes the verdict
+ // back onto the row (`ledger.persist_auto_triage`). The API has returned
+ // `ai_score` / `ai_summary` / `triage_groundedness` on every alert since,
+ // and this mapper dropped all three — so the one output of the AI the
+ // product leads with was metered, persisted, and rendered nowhere.
+ /** 0–1 confidence the triage run reported. Distinct from `confidenceScore`,
+ * which is fusion's detection confidence and has nothing to do with the
+ * agent. */
+ aiScore?: number;
+ /** The verdict's rationale, as written by whichever path produced it. The
+ * model's own text is prefixed `LLM auto-triage verdict:`; the
+ * deterministic fallback reads as a description of the signals it matched.
+ * Rendered verbatim so the reader can tell them apart. */
+ aiSummary?: string;
+ /** Fraction of the verdict's cited indicators that appear in the evidence.
+ * `null`/absent means *not assessed* — the deterministic path never scores
+ * groundedness, and rendering that as 0 would read as "wholly unsupported".
+ */
+ triageGroundedness?: number | null;
+ /** Indicators the verdict cited that the evidence never contained. */
+ triageUngrounded?: string[];
}
/**
@@ -894,6 +917,16 @@ function normalizeAlert(raw: unknown): Alert {
relatedEntities,
miniTimeline,
recommendedActions,
+ aiScore: pickNum('ai_score', 'aiScore'),
+ aiSummary: pickStr('ai_summary', 'aiSummary'),
+ // `?? null` and not `?? undefined`: the column is nullable because NULL
+ // means "not assessed", and collapsing that into absent loses the
+ // distinction the schema comment exists to preserve.
+ triageGroundedness:
+ typeof (r.triage_groundedness ?? r.triageGroundedness) === 'number'
+ ? ((r.triage_groundedness ?? r.triageGroundedness) as number)
+ : null,
+ triageUngrounded: pickArr<string>('triage_ungrounded', 'triageUngrounded'),
};
}
@@ -3003,8 +3036,11 @@ export const threatIntelApi = {
body: JSON.stringify({ iocs }),
}),
+ // `total` is the store's count of indicators in scope; `shown` is how many
+ // this response carries. They are different numbers and the console renders
+ // both — it used to render the page length as the catalogue size.
list: (filters: { type?: IndicatorType; tag?: string; q?: string } = {}) =>
- request<{ indicators: ThreatIndicator[]; total: number }>(
+ request<{ indicators: ThreatIndicator[]; total: number; shown?: number; bounded?: boolean }>(
'/api/v1/threat-intel/indicators',
{ params: filters as Record<string, string> },
),
diff --git a/docs/architecture/README.md b/docs/architecture/README.md
index d528f7343..1e8d24a51 100644
--- a/docs/architecture/README.md
+++ b/docs/architecture/README.md
@@ -113,15 +113,17 @@ flowchart LR
agt["services/agents<br/>Python · :8084"]
web["apps/web<br/>Next.js · :3000"]
llm["litellm<br/>LLM gateway · :4000"]
+ oll["ollama<br/>llama3.2:3b · :11434"]
+ ti["services/threatintel<br/>Python · :8005"]
+ qd[("Qdrant<br/>IOC + actor vectors")]
end
subgraph full ["full profile — optional"]
ch[("ClickHouse<br/>event lake")]
neo[("Neo4j<br/>entity graph")]
- qd[("Qdrant<br/>embeddings")]
+ os[("OpenSearch<br/>full-text IOC search")]
enr["services/enrichment"]
conn["services/connectors"]
- ti["services/threatintel"]
end
conn -.->|"polls vendors"| ing
@@ -137,10 +139,13 @@ flowchart LR
agt --> pg
agt -->|"aisoc-&lt;role&gt; alias"| llm
api --> llm
+ llm --> oll
api --> pg
api -.-> ch
api -.-> neo
- ti -.-> qd
+ ti -->|"CISA KEV"| qd
+ ti -.-> os
+ api --> ti
web --> api
web --> rt
```
@@ -151,16 +156,17 @@ flowchart LR
```mermaid
flowchart TB
- subgraph req ["Required"]
+ subgraph req ["CORE"]
pg["PostgreSQL<br/><br/>alerts · incidents · cases<br/>users · tenants · detection rules<br/>audit log · investigation ledger"]
rds["Redis<br/><br/>correlation windows<br/>dedup keys<br/>investigation run state"]
- kaf["Kafka<br/><br/>aisoc.raw_events<br/>aisoc.alerts.fused"]
+ kaf["Kafka<br/><br/>aisoc.raw_events<br/>aisoc.alerts.fused<br/>aisoc.alerts.dlq"]
+ qd["Qdrant<br/><br/>IOC and actor vectors<br/>backs the Threat Intelligence page"]
end
subgraph opt ["Optional (full profile)"]
ch["ClickHouse<br/><br/>every normalized event<br/>backs /lake/sql and hunt"]
neo["Neo4j<br/><br/>entity graph<br/>blast radius"]
- qd["Qdrant<br/><br/>IOC and actor embeddings"]
+ os["OpenSearch<br/><br/>full-text IOC and actor search"]
end
```
@@ -172,14 +178,17 @@ flowchart TB
| **Redis** | Correlation needs expiring keys at event rate; TTL semantics and throughput are the point. | No correlation; every alert becomes its own incident. |
| **Kafka** | Decouples producers from consumers and lets a slow consumer fall behind without dropping events or blocking ingest. | No spine; fusion, realtime and agents would each need a direct call from ingest. |
| **ClickHouse** | Columnar scans over hundreds of millions of events. Postgres cannot do this at cost. | No event lake, no hunting over raw telemetry. **Alerting is unaffected.** |
-| **Neo4j** | Multi-hop traversal ("what else did this identity touch") is a join explosion in SQL. | No graph context or blast radius. Alerting unaffected. |
-| **Qdrant** | Vector similarity for IOC and actor matching. | No semantic threat-intel matching. |
-| **OpenSearch** | Full-text and structured search over the threat-intel corpus — `threatintel-iocs` and `threatintel-actors`. | `services/threatintel` cannot start: it indexes into OpenSearch in its lifespan, without a fallback. |
+| **Neo4j** | Multi-hop traversal ("what else did this identity touch") is a join explosion in SQL. | No graph context or blast radius. `/graph` reports the failure rather than drawing an invented graph. Alerting unaffected. |
+| **Qdrant** | Vector similarity for IOC and actor matching, and the read path the console's Threat Intelligence page is served from. | No threat-intel page. It is in CORE for that reason, and because it is by a wide margin the cheapest of the four stores — 245 MB of image, ~79 MiB resident. |
+| **OpenSearch** | Full-text and structured search over the threat-intel corpus — `threatintel-iocs` and `threatintel-actors`. | Feeds still write to Qdrant and the page still works; full-text IOC search is unavailable and `services/threatintel` logs that it is. |
**Who reads OpenSearch, precisely:** only `services/threatintel`. This page
previously said nothing read it at all, which came from checking
`services/api` — which genuinely holds no OpenSearch client — and stopping
-there. See the correction in
+there. It also said `services/threatintel` *could not start* without it; that
+was true when the lifespan called `os_store.initialize()` with no `try`, and
+is no longer — both that call and the pipeline's bulk index are best-effort,
+which is what let the service move into CORE. See the corrections in
[the reality audit](../audit/REPOSITORY_REALITY.md).
---
diff --git a/docs/audit/REPOSITORY_REALITY.md b/docs/audit/REPOSITORY_REALITY.md
index cae0e6317..8365020ec 100644
--- a/docs/audit/REPOSITORY_REALITY.md
+++ b/docs/audit/REPOSITORY_REALITY.md
@@ -218,7 +218,7 @@ Stated rather than guessed:
## Reproducing this audit
```bash
-make up # CORE profile: 10 services
+make up # CORE profile: 14 long-running services (+ a one-shot model pull)
make doctor # every dependency probed, not just "running"
make smoke # one real event through the real pipeline
make stats # recount every published figure from the tree
diff --git a/docs/openapi.yaml b/docs/openapi.yaml
index 724448142..ae1a20cd4 100644
--- a/docs/openapi.yaml
+++ b/docs/openapi.yaml
@@ -20238,6 +20238,14 @@ components:
total:
type: integer
title: Total
+ shown:
+ type: integer
+ title: Shown
+ default: 0
+ bounded:
+ type: boolean
+ title: Bounded
+ default: false
source:
type: string
title: Source
diff --git a/docs/testing/CLEAN_INSTALL.md b/docs/testing/CLEAN_INSTALL.md
index df6852928..6eaad9c5b 100644
--- a/docs/testing/CLEAN_INSTALL.md
+++ b/docs/testing/CLEAN_INSTALL.md
@@ -28,7 +28,7 @@ make smoke
| Stage | Result |
|---|---|
| Clone | pass |
-| `make up` — CORE profile, 10 services | pass |
+| `make up` — CORE profile, 14 long-running services | pass |
| Postgres accepting connections, 92 tables | pass |
| Kafka broker answering admin requests | pass |
| `api`, `ingest-worker`, `fusion` healthy | pass |
diff --git a/screenshots/01-landing.png b/screenshots/01-landing.png
deleted file mode 100644
index f7b0a1c89..000000000
Binary files a/screenshots/01-landing.png and /dev/null differ
diff --git a/screenshots/02-signup.png b/screenshots/02-signup.png
deleted file mode 100644
index 998749909..000000000
Binary files a/screenshots/02-signup.png and /dev/null differ
diff --git a/screenshots/redesign/about-full.png b/screenshots/redesign/about-full.png
deleted file mode 100644
index 35e2dcf1e..000000000
Binary files a/screenshots/redesign/about-full.png and /dev/null differ
diff --git a/screenshots/redesign/app-billing.png b/screenshots/redesign/app-billing.png
deleted file mode 100644
index 472410bd1..000000000
Binary files a/screenshots/redesign/app-billing.png and /dev/null differ
diff --git a/screenshots/redesign/app-case-detail.png b/screenshots/redesign/app-case-detail.png
deleted file mode 100644
index 651fbb23e..000000000
Binary files a/screenshots/redesign/app-case-detail.png and /dev/null differ
diff --git a/screenshots/redesign/app-case-investigation.png b/screenshots/redesign/app-case-investigation.png
deleted file mode 100644
index 5cf4e02e2..000000000
Binary files a/screenshots/redesign/app-case-investigation.png and /dev/null differ
diff --git a/screenshots/redesign/app-case-ledger.png b/screenshots/redesign/app-case-ledger.png
deleted file mode 100644
index 382284740..000000000
Binary files a/screenshots/redesign/app-case-ledger.png and /dev/null differ
diff --git a/screenshots/redesign/app-cases.png b/screenshots/redesign/app-cases.png
deleted file mode 100644
index 1583cd724..000000000
Binary files a/screenshots/redesign/app-cases.png and /dev/null differ
diff --git a/screenshots/redesign/app-dashboard.png b/screenshots/redesign/app-dashboard.png
deleted file mode 100644
index e45c135ee..000000000
Binary files a/screenshots/redesign/app-dashboard.png and /dev/null differ
diff --git a/screenshots/redesign/hero.png b/screenshots/redesign/hero.png
deleted file mode 100644
index 50fa3ecf5..000000000
Binary files a/screenshots/redesign/hero.png and /dev/null differ
diff --git a/screenshots/redesign/home-desktop.png b/screenshots/redesign/home-desktop.png
deleted file mode 100644
index 8cbb2733c..000000000
Binary files a/screenshots/redesign/home-desktop.png and /dev/null differ
diff --git a/screenshots/redesign/home-full.png b/screenshots/redesign/home-full.png
deleted file mode 100644
index 8cbb2733c..000000000
Binary files a/screenshots/redesign/home-full.png and /dev/null differ
diff --git a/screenshots/redesign/home-mobile.png b/screenshots/redesign/home-mobile.png
deleted file mode 100644
index 41c805466..000000000
Binary files a/screenshots/redesign/home-mobile.png and /dev/null differ
diff --git a/screenshots/redesign/home-tablet.png b/screenshots/redesign/home-tablet.png
deleted file mode 100644
index abbb5705d..000000000
Binary files a/screenshots/redesign/home-tablet.png and /dev/null differ
diff --git a/screenshots/redesign/integrations-fold.png b/screenshots/redesign/integrations-fold.png
deleted file mode 100644
index 697f2e8f5..000000000
Binary files a/screenshots/redesign/integrations-fold.png and /dev/null differ
diff --git a/screenshots/redesign/integrations.png b/screenshots/redesign/integrations.png
deleted file mode 100644
index 491704f38..000000000
Binary files a/screenshots/redesign/integrations.png and /dev/null differ
diff --git a/screenshots/redesign/login.png b/screenshots/redesign/login.png
deleted file mode 100644
index 657f63162..000000000
Binary files a/screenshots/redesign/login.png and /dev/null differ
diff --git a/screenshots/redesign/pricing-full.png b/screenshots/redesign/pricing-full.png
deleted file mode 100644
index 3032c41f6..000000000
Binary files a/screenshots/redesign/pricing-full.png and /dev/null differ
diff --git a/screenshots/redesign/stripe-checkout.png b/screenshots/redesign/stripe-checkout.png
deleted file mode 100644
index 391ed30ff..000000000
Binary files a/screenshots/redesign/stripe-checkout.png and /dev/null differ
diff --git a/screenshots/redesign/tryaisoc-dashboard.png b/screenshots/redesign/tryaisoc-dashboard.png
deleted file mode 100644
index 8f6c000f2..000000000
Binary files a/screenshots/redesign/tryaisoc-dashboard.png and /dev/null differ
diff --git a/screenshots/redesign/tryaisoc-live.png b/screenshots/redesign/tryaisoc-live.png
deleted file mode 100644
index 5284bc2f1..000000000
Binary files a/screenshots/redesign/tryaisoc-live.png and /dev/null differ
diff --git a/screenshots/resweep/audit-final.png b/screenshots/resweep/audit-final.png
deleted file mode 100644
index 8f898929d..000000000
Binary files a/screenshots/resweep/audit-final.png and /dev/null differ
diff --git a/screenshots/resweep/audit.png b/screenshots/resweep/audit.png
deleted file mode 100644
index cd7723f56..000000000
Binary files a/screenshots/resweep/audit.png and /dev/null differ
diff --git a/screenshots/resweep/coverage-advisor.png b/screenshots/resweep/coverage-advisor.png
deleted file mode 100644
index 44231df59..000000000
Binary files a/screenshots/resweep/coverage-advisor.png and /dev/null differ
diff --git a/screenshots/resweep/detection.png b/screenshots/resweep/detection.png
deleted file mode 100644
index 949e5d03f..000000000
Binary files a/screenshots/resweep/detection.png and /dev/null differ
diff --git a/screenshots/resweep/easm.png b/screenshots/resweep/easm.png
deleted file mode 100644
index 1815dcae4..000000000
Binary files a/screenshots/resweep/easm.png and /dev/null differ
diff --git a/screenshots/resweep/graph.png b/screenshots/resweep/graph.png
deleted file mode 100644
index c20571e74..000000000
Binary files a/screenshots/resweep/graph.png and /dev/null differ
diff --git a/screenshots/resweep/shifts.png b/screenshots/resweep/shifts.png
deleted file mode 100644
index 45f26191e..000000000
Binary files a/screenshots/resweep/shifts.png and /dev/null differ
diff --git a/screenshots/resweep/threat-intel.png b/screenshots/resweep/threat-intel.png
deleted file mode 100644
index b68efc482..000000000
Binary files a/screenshots/resweep/threat-intel.png and /dev/null differ
diff --git a/screenshots/sweep/alerts.png b/screenshots/sweep/alerts.png
deleted file mode 100644
index 0bc4fe012..000000000
Binary files a/screenshots/sweep/alerts.png and /dev/null differ
diff --git a/screenshots/sweep/audit.png b/screenshots/sweep/audit.png
deleted file mode 100644
index 61ba8f256..000000000
Binary files a/screenshots/sweep/audit.png and /dev/null differ
diff --git a/screenshots/sweep/cases.png b/screenshots/sweep/cases.png
deleted file mode 100644
index 87f6d3eba..000000000
Binary files a/screenshots/sweep/cases.png and /dev/null differ
diff --git a/screenshots/sweep/connectors.png b/screenshots/sweep/connectors.png
deleted file mode 100644
index 98e4831a3..000000000
Binary files a/screenshots/sweep/connectors.png and /dev/null differ
diff --git a/screenshots/sweep/copilot.png b/screenshots/sweep/copilot.png
deleted file mode 100644
index 8ad4e11c6..000000000
Binary files a/screenshots/sweep/copilot.png and /dev/null differ
diff --git a/screenshots/sweep/costs.png b/screenshots/sweep/costs.png
deleted file mode 100644
index 88b8fb142..000000000
Binary files a/screenshots/sweep/costs.png and /dev/null differ
diff --git a/screenshots/sweep/coverage-advisor.png b/screenshots/sweep/coverage-advisor.png
deleted file mode 100644
index 34e33789a..000000000
Binary files a/screenshots/sweep/coverage-advisor.png and /dev/null differ
diff --git a/screenshots/sweep/dashboard.png b/screenshots/sweep/dashboard.png
deleted file mode 100644
index e530af0f5..000000000
Binary files a/screenshots/sweep/dashboard.png and /dev/null differ
diff --git a/screenshots/sweep/dashboards-soc-insights.png b/screenshots/sweep/dashboards-soc-insights.png
deleted file mode 100644
index 4414eefeb..000000000
Binary files a/screenshots/sweep/dashboards-soc-insights.png and /dev/null differ
diff --git a/screenshots/sweep/detection-catalog.png b/screenshots/sweep/detection-catalog.png
deleted file mode 100644
index aaf61e34a..000000000
Binary files a/screenshots/sweep/detection-catalog.png and /dev/null differ
diff --git a/screenshots/sweep/detection-coverage.png b/screenshots/sweep/detection-coverage.png
deleted file mode 100644
index 1eb3c9323..000000000
Binary files a/screenshots/sweep/detection-coverage.png and /dev/null differ
diff --git a/screenshots/sweep/detection-tuning.png b/screenshots/sweep/detection-tuning.png
deleted file mode 100644
index e83cc7ed5..000000000
Binary files a/screenshots/sweep/detection-tuning.png and /dev/null differ
diff --git a/screenshots/sweep/detection.png b/screenshots/sweep/detection.png
deleted file mode 100644
index afde68b70..000000000
Binary files a/screenshots/sweep/detection.png and /dev/null differ
diff --git a/screenshots/sweep/easm.png b/screenshots/sweep/easm.png
deleted file mode 100644
index a92ecfe6d..000000000
Binary files a/screenshots/sweep/easm.png and /dev/null differ
diff --git a/screenshots/sweep/explore.png b/screenshots/sweep/explore.png
deleted file mode 100644
index 9627fdfe7..000000000
Binary files a/screenshots/sweep/explore.png and /dev/null differ
diff --git a/screenshots/sweep/graph.png b/screenshots/sweep/graph.png
deleted file mode 100644
index 850d974db..000000000
Binary files a/screenshots/sweep/graph.png and /dev/null differ
diff --git a/screenshots/sweep/hunt.png b/screenshots/sweep/hunt.png
deleted file mode 100644
index 57d79fac1..000000000
Binary files a/screenshots/sweep/hunt.png and /dev/null differ
diff --git a/screenshots/sweep/marketplace.png b/screenshots/sweep/marketplace.png
deleted file mode 100644
index 895454adc..000000000
Binary files a/screenshots/sweep/marketplace.png and /dev/null differ
diff --git a/screenshots/sweep/playbooks.png b/screenshots/sweep/playbooks.png
deleted file mode 100644
index b410e9712..000000000
Binary files a/screenshots/sweep/playbooks.png and /dev/null differ
diff --git a/screenshots/sweep/queue.png b/screenshots/sweep/queue.png
deleted file mode 100644
index 7170d6550..000000000
Binary files a/screenshots/sweep/queue.png and /dev/null differ
diff --git a/screenshots/sweep/settings.png b/screenshots/sweep/settings.png
deleted file mode 100644
index a9dab27c6..000000000
Binary files a/screenshots/sweep/settings.png and /dev/null differ
diff --git a/screenshots/sweep/shifts.png b/screenshots/sweep/shifts.png
deleted file mode 100644
index 1436f087f..000000000
Binary files a/screenshots/sweep/shifts.png and /dev/null differ
diff --git a/screenshots/sweep/threat-intel.png b/screenshots/sweep/threat-intel.png
deleted file mode 100644
index 09859a09b..000000000
Binary files a/screenshots/sweep/threat-intel.png and /dev/null differ
diff --git a/services/api/app/api/v1/endpoints/threat_intel.py b/services/api/app/api/v1/endpoints/threat_intel.py
index 8f09fbc02..05f299798 100644
--- a/services/api/app/api/v1/endpoints/threat_intel.py
+++ b/services/api/app/api/v1/endpoints/threat_intel.py
@@ -137,7 +137,15 @@ class FeedIndicatorsResponse(BaseModel):
"""
indicators: list[dict[str, Any]]
+ #: Indicators in the caller's scope, per the upstream store's own count.
total: int
+ #: Indicators carried by this response. ``total`` is the catalogue,
+ #: ``shown`` is the page — the console renders "N of M" and used to render
+ #: the page size as the catalogue.
+ shown: int = 0
+ #: Upstream narrowed inside a bounded scan window, so ``shown`` is a lower
+ #: bound on the matches that exist.
+ bounded: bool = False
source: str
degraded: bool = False
reason: str = ""
@@ -214,6 +222,8 @@ async def list_feed_indicators(
return FeedIndicatorsResponse(
indicators=indicators,
total=int(body.get("total") or len(indicators)),
+ shown=int(body.get("shown") or len(indicators)),
+ bounded=bool(body.get("bounded")),
source=str(body.get("source") or "threatintel"),
)
diff --git a/services/fusion/app/services/provenance.py b/services/fusion/app/services/provenance.py
index 7d7d03a6e..d228eb01c 100644
--- a/services/fusion/app/services/provenance.py
+++ b/services/fusion/app/services/provenance.py
@@ -15,6 +15,7 @@ connector *instance* id is what downstream resolution actually keys on.
from __future__ import annotations
+import re
import uuid
from typing import Any
@@ -28,6 +29,16 @@ def _parse_uuid(value: Any) -> uuid.UUID | None:
return None
+def _says_the_same_thing(whole: str, part: str) -> bool:
+ """Whether ``part`` already appears inside ``whole`` as whole words.
+
+ Word boundaries, not a plain substring test: "AWS" must not be swallowed
+ by a product called "Lawsuit Monitor", and the join exists to name a
+ vendor, not to pattern-match one.
+ """
+ return re.search(rf"(?<!\w){re.escape(part)}(?!\w)", whole, re.IGNORECASE) is not None
+
+
def product_label(ocsf: dict[str, Any]) -> str | None:
"""Vendor + product, deduplicated.
@@ -35,6 +46,16 @@ def product_label(ocsf: dict[str, Any]) -> str | None:
naive join produced `connector_type = "crowdstrike crowdstrike"` on every
alert. Verified on a live stack.
+ Exact equality was not enough. Four of the ten profiles in
+ `services/ingest/internal/normalizer/normalizer.go` name the vendor inside
+ the product — `Okta` / `Okta System Log`, `Splunk` / `Splunk Enterprise`,
+ `Kubernetes` / `Kubernetes Audit`, `Email` / `Forwarded Email` — so the
+ alert queue, the Investigation Rail and every entity chip read "Okta Okta
+ System Log". Observed on a live CORE stack against real pushed telemetry.
+ A part that another part already says is dropped; the longer one wins,
+ which leaves "CrowdStrike Falcon" and "AWS Security Hub" untouched because
+ neither names the other.
+
This is the single implementation. `promoter._source()` delegates here;
there used to be two copies of the join and only one of them was fixed,
which is why the doubled label survived the first repair.
@@ -51,7 +72,12 @@ def product_label(ocsf: dict[str, Any]) -> str | None:
if any(cleaned.lower() == seen.lower() for seen in parts):
continue
parts.append(cleaned)
- return " ".join(parts) or None
+ kept = [
+ part
+ for index, part in enumerate(parts)
+ if not any(len(other) > len(part) and _says_the_same_thing(other, part) for other in parts[:index] + parts[index + 1 :])
+ ]
+ return " ".join(kept) or None
#: Retained so existing imports keep working.
diff --git a/services/fusion/tests/test_provenance_product_label.py b/services/fusion/tests/test_provenance_product_label.py
new file mode 100644
index 000000000..0e2f6d77e
--- /dev/null
+++ b/services/fusion/tests/test_provenance_product_label.py
@@ -0,0 +1,94 @@
+"""The vendor/product join must not say the vendor twice.
+
+`connector_type` is what the alert queue, the Investigation Rail's connector
+chip and `alert.source` all render, so a doubled label is visible on every
+surface that names where an alert came from.
+
+The interesting half of this file is `test_no_shipped_profile_doubles_its_vendor`:
+it reads the vendor/product pairs out of the Go normalizer rather than
+restating them here. A test that carries its own copy of the input only proves
+the function is self-consistent — the previous exact-equality dedup passed
+every hand-written case while four of the ten shipped profiles rendered
+doubled, because no test ever asked the normalizer what it actually declares.
+"""
+
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+import pytest
+from app.services.provenance import product_label
+
+NORMALIZER = Path(__file__).resolve().parents[3] / "services" / "ingest" / "internal" / "normalizer" / "normalizer.go"
+
+#: `OcsfProduct{Name: "Okta System Log", VendorName: "Okta"}`
+_PRODUCT_DECL = re.compile(r'OcsfProduct\{Name:\s*"(?P<name>[^"]+)",\s*VendorName:\s*"(?P<vendor>[^"]+)"\}')
+
+
+def _label(vendor: str | None, name: str | None) -> str | None:
+ product: dict[str, str] = {}
+ if vendor is not None:
+ product["vendor_name"] = vendor
+ if name is not None:
+ product["name"] = name
+ return product_label({"metadata": {"product": product}})
+
+
+ ("vendor", "name", "expected"),
+ [
+ # The vendor names itself inside the product. Observed doubled on a
+ # live CORE stack against telemetry pushed through the ingest API.
+ ("Okta", "Okta System Log", "Okta System Log"),
+ ("Splunk", "Splunk Enterprise", "Splunk Enterprise"),
+ ("Kubernetes", "Kubernetes Audit", "Kubernetes Audit"),
+ # Containment, not prefixing: the vendor is the *last* word here.
+ ("Email", "Forwarded Email", "Forwarded Email"),
+ # Exact duplicates — the case the original dedup was written for.
+ ("Splunk", "Splunk", "Splunk"),
+ ("crowdstrike", "CrowdStrike", "crowdstrike"),
+ # Neither part names the other: both survive, in vendor-first order.
+ ("CrowdStrike", "Falcon", "CrowdStrike Falcon"),
+ ("Microsoft", "Sentinel", "Microsoft Sentinel"),
+ ("AWS", "Security Hub", "AWS Security Hub"),
+ ("AiSOC", "AI Runtime", "AiSOC AI Runtime"),
+ # One side missing is not a reason to drop the other.
+ ("CrowdStrike", None, "CrowdStrike"),
+ (None, "Falcon", "Falcon"),
+ ("", " ", None),
+ ],
+)
+def test_product_label(vendor: str | None, name: str | None, expected: str | None) -> None:
+ assert _label(vendor, name) == expected
+
+
+def test_substring_match_respects_word_boundaries() -> None:
+ """A vendor that merely appears inside a longer word is not swallowed."""
+ assert _label("AWS", "Lawsuit Monitor") == "AWS Lawsuit Monitor"
+
+
+def test_missing_or_malformed_metadata_yields_none() -> None:
+ assert product_label({}) is None
+ assert product_label({"metadata": {}}) is None
+ assert product_label({"metadata": {"product": "Falcon"}}) is None
+
+
[email protected](not NORMALIZER.is_file(), reason="ingest normalizer not present in this checkout")
+def test_no_shipped_profile_doubles_its_vendor() -> None:
+ """Every vendor/product pair the normalizer declares must render once.
+
+ Runs in the direction that drifts: adding a profile whose product name
+ repeats its vendor fails here rather than being discovered in a screenshot.
+ """
+ declarations = _PRODUCT_DECL.findall(NORMALIZER.read_text(encoding="utf-8"))
+ assert declarations, f"no OcsfProduct declarations found in {NORMALIZER}"
+
+ doubled: list[tuple[str, str, str]] = []
+ for name, vendor in declarations:
+ label = _label(vendor, name) or ""
+ words = [word.lower() for word in label.split()]
+ if len(words) != len(set(words)):
+ doubled.append((vendor, name, label))
+
+ assert not doubled, "vendor repeated in the rendered label: " + ", ".join(f"{v!r}+{n!r} -> {label!r}" for v, n, label in doubled)
diff --git a/services/threatintel/app/api/indicators.py b/services/threatintel/app/api/indicators.py
index 11067cef2..7b970f971 100644
--- a/services/threatintel/app/api/indicators.py
+++ b/services/threatintel/app/api/indicators.py
@@ -75,7 +75,26 @@ class Indicator(BaseModel):
class IndicatorListResponse(BaseModel):
indicators: list[Indicator]
+ #: How many indicators are in the caller's scope, from the store's own
+ #: count — not how many this response carries, and not how many the scan
+ #: window happened to reach.
+ #:
+ #: It used to be ``len(indicators)`` after post-filtering, i.e. the size of
+ #: the bounded scan window. With 1,725 KEV entries collected, the store
+ #: held 1,725, this field said 400 and the console's headline card said
+ #: 100 — three different answers to one question, the two published ones
+ #: both wrong, and the page presenting the smallest as the corpus.
total: int
+ #: How many indicators this response carries. Distinct from ``total`` on
+ #: purpose: the console needs to say "100 of 1,725" rather than implying
+ #: a page is the catalogue.
+ shown: int = 0
+ #: True when ``type`` / ``tag`` / ``q`` narrowed the result, in which case
+ #: the match was made inside the bounded scan window rather than across
+ #: the whole collection — so ``shown`` is a lower bound on the matches
+ #: that exist. Published rather than hidden because a search that silently
+ #: reads part of the corpus is the kind of thing that gets believed.
+ bounded: bool = False
#: Which store answered. The console has no other way to tell "no
#: indicators collected yet" from "the store is not there", and the two
#: call for different things from the reader.
@@ -110,6 +129,26 @@ def _qdrant(request: Request) -> AsyncQdrantClient:
return client
+async def _scope_total(client: AsyncQdrantClient, scope: Any, *, fallback: int) -> int:
+ """How many indicators are in the caller's scope, asked of the store.
+
+ The scan above is deliberately bounded, so counting its results answers
+ "how many did I look at", which is not the question the console is asking.
+ Qdrant can answer the real one directly.
+
+ A count failure falls back to the window size rather than raising: the
+ indicators themselves were read successfully, and losing the whole list
+ over a secondary count would turn a wrong number into no page at all.
+ """
+ try:
+ result = await client.count(collection_name=IOC_COLLECTION, count_filter=scope, exact=True)
+ except Exception as exc: # noqa: BLE001 — any count failure degrades to the window size
+ logger.warning("threatintel.indicators.count_failed", error=type(exc).__name__)
+ return fallback
+ count = getattr(result, "count", None)
+ return int(count) if isinstance(count, int) else fallback
+
+
@router.get("/indicators", response_model=IndicatorListResponse)
async def list_indicators(
request: Request,
@@ -152,7 +191,7 @@ async def list_indicators(
message = str(exc)
if "not found" in message.lower() or "doesn't exist" in message.lower():
logger.info("threatintel.indicators.collection_absent", collection=IOC_COLLECTION)
- return IndicatorListResponse(indicators=[], total=0, source="qdrant")
+ return IndicatorListResponse(indicators=[], total=0, shown=0, source="qdrant")
logger.warning("threatintel.indicators.read_failed", error=type(exc).__name__)
raise HTTPException(
status_code=503,
@@ -161,6 +200,7 @@ async def list_indicators(
indicators = [_to_indicator(p.payload or {}) for p in points if p.payload]
+ bounded = bool(ioc_type or tag or q)
if ioc_type:
indicators = [i for i in indicators if i.type == ioc_type]
if tag:
@@ -169,5 +209,6 @@ async def list_indicators(
needle = q.lower()
indicators = [i for i in indicators if needle in i.value.lower() or needle in (i.description or "").lower()]
- total = len(indicators)
- return IndicatorListResponse(indicators=indicators[:limit], total=total, source="qdrant")
+ page = indicators[:limit]
+ total = await _scope_total(client, scope, fallback=len(indicators))
+ return IndicatorListResponse(indicators=page, total=total, shown=len(page), bounded=bounded, source="qdrant")
diff --git a/services/threatintel/tests/test_indicator_totals.py b/services/threatintel/tests/test_indicator_totals.py
new file mode 100644
index 000000000..76d54f193
--- /dev/null
+++ b/services/threatintel/tests/test_indicator_totals.py
@@ -0,0 +1,137 @@
+"""`total` must be the catalogue, not the page and not the scan window.
+
+The route scrolls a bounded window over Qdrant and returns one page from it.
+It published ``len(indicators)`` after post-filtering as ``total``, so with
+1,725 CISA KEV entries collected the store held 1,725, the API answered 400
+(the scan bound) and the console's headline card read 100 (its page) — three
+answers to one question, two of them published, both wrong, and the page
+presenting the smallest of them as the size of the corpus.
+
+These tests drive `list_indicators` directly against a stub client rather than
+a live Qdrant: the thing under test is which number ends up in which field,
+and a container cannot make that clearer.
+"""
+
+from __future__ import annotations
+
+import uuid
+from dataclasses import dataclass
+from typing import Any
+
+import pytest
+from app.api.indicators import list_indicators
+from app.security.tenant_scope import TenantPrincipal
+
+TENANT = uuid.UUID("00000000-0000-0000-0000-000000000001")
+
+
+@dataclass
+class _Point:
+ payload: dict[str, Any]
+
+
+@dataclass
+class _CountResult:
+ count: int
+
+
+class _StubQdrant:
+ """Answers `scroll` with a fixed window and `count` with a fixed total."""
+
+ def __init__(self, *, points: list[_Point], count: int | None, count_raises: bool = False) -> None:
+ self._points = points
+ self._count = count
+ self._count_raises = count_raises
+ self.scroll_limits: list[int] = []
+
+ async def scroll(self, *, collection_name: str, scroll_filter: Any, limit: int, **_: Any):
+ self.scroll_limits.append(limit)
+ return self._points[:limit], None
+
+ async def count(self, *, collection_name: str, count_filter: Any, exact: bool):
+ if self._count_raises:
+ raise RuntimeError("qdrant said no")
+ return _CountResult(count=self._count or 0)
+
+
+class _Request:
+ def __init__(self, client: _StubQdrant) -> None:
+ self.app = type("_App", (), {"state": type("_State", (), {"qdrant_client": client})()})()
+
+
+def _points(n: int, *, ioc_type: str = "ip") -> list[_Point]:
+ return [
+ _Point(
+ payload={
+ "type": ioc_type,
+ "value": f"CVE-2024-{i:05d}",
+ "description": f"indicator {i}",
+ "sources": ["cisa-kev"],
+ "tags": ["kev"],
+ }
+ )
+ for i in range(n)
+ ]
+
+
+def _principal() -> TenantPrincipal:
+ return TenantPrincipal(tenant_ids=frozenset({TENANT}), subject="test")
+
+
+async def _call(client: _StubQdrant, **kwargs: Any):
+ return await list_indicators(
+ request=_Request(client), # type: ignore[arg-type]
+ principal=_principal(),
+ ioc_type=kwargs.get("ioc_type"),
+ tag=kwargs.get("tag"),
+ q=kwargs.get("q"),
+ limit=kwargs.get("limit", 100),
+ )
+
+
+async def test_total_is_the_store_count_not_the_page() -> None:
+ """The regression this file exists for, in the shape it was observed."""
+ client = _StubQdrant(points=_points(400), count=1725)
+
+ response = await _call(client, limit=100)
+
+ assert response.total == 1725, "total must be the catalogue"
+ assert response.shown == 100, "shown must be the page"
+ assert len(response.indicators) == 100
+ assert response.bounded is False
+
+
+async def test_total_survives_a_filtered_query_and_marks_it_bounded() -> None:
+ """A post-filter narrows the page; it does not shrink the catalogue."""
+ client = _StubQdrant(points=_points(40, ioc_type="ip"), count=1725)
+
+ response = await _call(client, ioc_type="domain", limit=100)
+
+ assert response.total == 1725
+ assert response.shown == 0
+ assert response.bounded is True, "the caller must be told the match was made in a bounded window"
+
+
+async def test_count_failure_degrades_to_the_window_rather_than_losing_the_page() -> None:
+ """Losing the list over a secondary count would be the worse failure."""
+ client = _StubQdrant(points=_points(12), count=None, count_raises=True)
+
+ response = await _call(client, limit=100)
+
+ assert response.total == 12
+ assert response.shown == 12
+ assert len(response.indicators) == 12
+
+
+async def test_empty_store_reports_zero_rather_than_a_page_length() -> None:
+ client = _StubQdrant(points=[], count=0)
+
+ response = await _call(client, limit=100)
+
+ assert response.total == 0
+ assert response.shown == 0
+ assert response.indicators == []