Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
Make the documented quick start produce a working stack (#879)
Beenu Arora439673c40b55
diff --git a/.env.example b/.env.exampleindex f612be8a6..0c36515fc 100644--- a/.env.example+++ b/.env.example@@ -2,18 +2,26 @@# Copy this file to .env and fill in your values## ╔════════════════════════════════════════════════════════════════════════════╗-# ║ REQUIRED FOR LOCAL DEV — set these three before `aisoc serve`: ║+# ║ YOU DO NOT HAVE TO EDIT THIS FILE TO GET A WORKING STACK. ║# ║ ║-# ║ 1. OPENAI_API_KEY your LLM provider API key (see "AI / LLM") ║-# ║ 2. POSTGRES_PASSWORD dev Postgres password (the default below works ║-# ║ out of the box; change for shared envs) ║-# ║ 3. AISOC_CREDENTIAL_KEY Fernet key for the connector credential vault. ║-# ║ REQUIRED. Generate one (see below) — there is ║-# ║ no default; the app will fail to start without ║-# ║ it. NEVER reuse a key from anywhere public. ║+# ║ make up creates .env from this template and fills the three ║+# ║ generated secrets — AISOC_CREDENTIAL_KEY, SECRET_KEY, ║+# ║ AISOC_SERVICE_TOKEN — with real random values. ║+# ║ make env does only that, if you want to look before you start. ║# ║ ║-# ║ Everything else has sensible defaults for `docker compose -f ║-# ║ infra/compose/docker-compose.dev.yml up -d`. ║+# ║ Those three ship EMPTY here rather than as `replace-me-…` prose. A ║+# ║ placeholder is not a neutral default: a non-empty AISOC_CREDENTIAL_KEY ║+# ║ that is not a valid Fernet key made every connector save answer HTTP 500, ║+# ║ so copying this template used to break the product while skipping the copy ║+# ║ left it working. `scripts/check_env_placeholders.py` now fails on any ║+# ║ placeholder shipped here, and `make doctor` reports them in your .env. ║+# ║ ║+# ║ Optional, and worth setting: ║+# ║ OPENAI_API_KEY a hosted provider key for better AI triage. CORE ║+# ║ ships a local model, so this is an upgrade, not a ║+# ║ requirement — see "AI / LLM" below. ║+# ║ POSTGRES_PASSWORD the dev default works out of the box; change it for ║+# ║ anything shared. ║# ╚════════════════════════════════════════════════════════════════════════════╝# ─── Image Pinning ────────────────────────────────────────────────────────────@@ -24,9 +32,22 @@AISOC_VERSION=latest# ─── Core Configuration ───────────────────────────────────────────────────────+# `development` enables the dev auth bypass in services/api/app/api/v1/dev_auth.py.+# Set `production` for anything network-reachable; docker-compose.yml reads this+# value, so changing it here is enough.ENVIRONMENT=developmentLOG_LEVEL=info-SECRET_KEY=change-this-to-a-random-secret-key-at-least-32-chars++# HS256 signing key for console sessions, shared by the API and the connectors+# service. Left empty here and generated by `scripts/ensure_env.py` — see the+# note on AISOC_CREDENTIAL_KEY below for why a placeholder is not used.+SECRET_KEY=++# Base URL `make bootstrap` prints as the sign-in address, and the origin the+# console is reached on. Leave it at the default for a laptop; set it to the+# address operators actually browse to for anything else, or the printed+# credential arrives with the wrong URL beside it.+AISOC_CONSOLE_URL=http://localhost:3000# Shared HS256 secret used ONLY for short-lived realtime WS/SSE tickets# (Issue #239). The API mints tickets at `POST /api/v1/realtime/ticket` and the@@ -47,12 +68,23 @@ AISOC_REALTIME_JWT_SECRET=AISOC_REALTIME_TICKET_TTL_SECONDS=60# Application-layer encryption key for connector credentials and other-# secret payloads stored in Postgres. Generate with:+# secret payloads stored in Postgres.+#+# LEAVE THIS EMPTY. `make up`, `make env` and `./install.sh` all run+# `scripts/ensure_env.py`, which writes a freshly generated key here the first+# time. Generating it by hand works too:# python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"-# REQUIRED. There is no built-in default — generate your own key with the-# command above and paste it here. Treat this value like a database password:-# never commit it, never reuse it across environments, never share it.-AISOC_CREDENTIAL_KEY=replace-me-with-a-freshly-generated-fernet-key+#+# It ships empty rather than as a `replace-me-…` string on purpose. A non-empty+# value that is not a valid Fernet key is *worse* than no value: the vault takes+# its development path only when the key is empty, so a placeholder reached+# `Fernet()`, raised, and turned every connector save into an HTTP 500 —+# minutes after `cp .env.example .env`, with nothing linking the two. Empty is a+# value the vault understands.+#+# Treat a real value like a database password: never commit it, never reuse it+# across environments, never share it.+AISOC_CREDENTIAL_KEY=# Comma-separated list of previous credential keys to support zero-downtime# rotation. Leave blank unless you are mid-rotation.AISOC_CREDENTIAL_KEY_ROTATION_FROM=@@ -104,7 +136,15 @@ OPENSEARCH_URL=http://localhost:9200QDRANT_URL=http://localhost:6333# ─── AI / LLM ─────────────────────────────────────────────────────────────────-OPENAI_API_KEY=sk-your-openai-api-key-here+# Empty, and empty is a working configuration: CORE ships Ollama with a small+# local model, so a default install does real AI triage with no key at all. Set+# a hosted provider key here to upgrade the quality — the gateway picks it up+# and routes the aliases to the hosted model instead.+#+# It is empty rather than `sk-your-…-here` because a bogus key is not the same+# as no key: the gateway would send it upstream, get a 401, and the deployment+# would look configured while producing nothing.+OPENAI_API_KEY=# OPENAI_MODEL affects the "explain this alert" / BYOK path only. Task workloads# (triage, recon, investigation, copilot, summary, report, NL) request a logical# alias instead — see the LLM gateway block below.@@ -256,13 +296,16 @@ NEXT_PUBLIC_WS_URL=ws://localhost:8086# ─── Service-to-service auth ──────────────────────────────────────────────────# Shared bearer token for backend services that are NOT reachable from the-# browser (purple-team, honeytokens, ueba). Those surfaces execute privileged-# work — purple-team can start adversary-emulation operations against live-# hosts — and they fail closed: with no token configured and AISOC_DEV_MODE+# browser (connectors, purple-team, honeytokens, ueba). Those surfaces execute+# privileged work — purple-team can start adversary-emulation operations against+# live hosts — and they fail closed: with no token configured and AISOC_DEV_MODE# unset, they answer 503 rather than serving the route openly.#-# Leave blank for local `docker compose up` (AISOC_DEV_MODE covers it). Set a-# real value for anything network-reachable.+# Left empty here and generated by `scripts/ensure_env.py`. It is not optional+# in practice: the API presents it when it proxies the connector catalog and+# "Test connection" to the connectors service, so a blank value on one side and+# a configured secret on the other is a 401 the wizard used to report as the+# bare string "Connection test failed".AISOC_SERVICE_TOKEN=# Optional per-service overrides, if you'd rather not share one secret:# AISOC_PURPLE_TEAM_SERVICE_TOKEN=diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f281da2ab..558812cbb 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,186 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### The documented quick start broke the product, and then the product had nothing to show++Every item below passed the existing test suite and failed on a real+deployment. They are grouped by what a new user actually hit, in the order+they hit it.++#### Following the README broke the credential vault++- **`.env.example` shipped `AISOC_CREDENTIAL_KEY=replace-me-with-a-freshly-generated-fernet-key`,+ and `cp .env.example .env` is step two of the quick start.** The vault takes+ its friendly ephemeral-development-key path only when the key is *empty*; a+ non-empty invalid key reaches `Fernet()` and raises, which the connector+ endpoints turn into HTTP 500 `credential vault unavailable`. Nothing failed+ at boot. So **not** copying the template produced a working vault and+ following the documented instructions produced a broken one, discovered+ minutes later at the connector wizard with nothing linking the two.++ Fixed at setup rather than in the vault: `make up` now runs+ `scripts/ensure_env.py`, which creates `.env` and writes a real random value+ for `AISOC_CREDENTIAL_KEY`, `SECRET_KEY` and `AISOC_SERVICE_TOKEN`. It is+ idempotent and never rotates a value an operator already set, and it uses+ the standard library so it does not put a `pip install` in front of+ `make up`. Teaching the vault to tolerate a placeholder was the alternative+ and would have been worse: a deployment whose saved connector credentials+ silently do not survive a restart. The three secrets now ship **empty** in+ the template as well, so a hand-copied `.env` degrades to the documented+ development path instead of a hard 500.++- **`make doctor`'s placeholder check matched neither placeholder the+ repository shipped.** It grepped+ `^[A-Z_]*(SECRET|PASSWORD|KEY)=(change_me|changeme|)$` while the template+ carried `replace-me-…` and `change-this-…` — a gate built to catch shipped+ placeholders that was blind to every shipped placeholder, and that reported+ clean on the one `.env` that broke the vault. The check now delegates to+ `scripts/check_env_placeholders.py`, and `tests/test_env_placeholder_gate.py`+ compares the detector against the template in both directions: every+ non-empty value in `.env.example` must be either recognised as a placeholder+ or declared in the test as a deliberate working default. A new placeholder+ cannot be added without one of the two failing.++#### The connector wizard could not complete++Three independent causes, all silent, all fixed:++- **The API was never given the credential it presents to the connectors+ service.** `_service_token()` reads `AISOC_CONNECTORS_SERVICE_TOKEN` then+ `AISOC_SERVICE_TOKEN`; compose interpolated the shared token into+ `connectors`, `ueba`, `honeytokens` and `purple-team` and **not into the one+ service that has to send it**, and with no `env_file` anywhere a value in+ `.env` could not reach it by any route. Every catalog call was answered 401+ and fell back to the bundled copy with `degraded=True`. Measured after the+ fix: the catalog is `live` with **84** connectors instead of the bundled 26.++- **"Test connection" sent no `Authorization` header at all.** The catalog call+ beside it had been given one and this one had not, so it was answered 401 on+ every invocation — and 401 was the one status the ladder did not branch on,+ so it fell through and returned a body with no `success` key. The wizard+ reads `result.success`, found it absent, and rendered the bare string+ **"Connection test failed"** for an internal service-auth misconfiguration,+ while its own help text promised "Credentials are tested against the+ upstream API". The upstream API was never reached. It now sends the same+ headers as the catalog call, branches on 401/403 explicitly, refuses any+ unhandled 4xx rather than returning it as a verdict, and guarantees a+ `success` key. An operator now sees: *"connectors service returned HTTP 401+ to this API's service credential. AiSOC's API could not authenticate to its+ own connectors service, so your credentials were never sent upstream. Set+ `AISOC_SERVICE_TOKEN` …"*. The wizard also reads FastAPI's `detail` out of+ the response body instead of showing only the status line.++- **The connectors service answered 503 to everything on a default stack.**+ Its `SECRET_KEY` compose default is one of `INSECURE_SECRET_DEFAULTS`, so+ `resolve_console_secret()` returns `""`; with `AISOC_SERVICE_TOKEN` also+ blank it has no credential material and fails closed, correctly. The+ dev-mode escape that exists for exactly that case was set on `actions`,+ `ueba`, `honeytokens`, `purple-team` and `slack-bot`, and not on+ `connectors`. Now set — and once `make env` generates a real token, dev mode+ stops applying at all.++#### CORE had no real data and no AI++- **`services/threatintel` and `qdrant` moved from the `full` profile into+ CORE.** The CISA Known Exploited Vulnerabilities catalog is authoritative,+ public and needs no API key, and has been wired as a scheduled handler the+ whole time in a profile nobody starting out runs. Meanwhile the console+ shipped a `/threat-intel` page whose endpoint existed in no profile, and that+ page was recently caught rendering five invented IOCs. The missing feed and+ the fabrication were one hole. A fresh `make up` now populates the console+ with **1,723 real KEV entries** within a minute of boot, with no credentials.+ New: `GET /api/v1/threat-intel/indicators` on the API, proxying a new+ indicators route on `services/threatintel` backed by Qdrant.++- **A daily feed would not have polled for a day.** `FeedScheduler.register`+ handed APScheduler an `IntervalTrigger` and nothing else, and an interval+ trigger schedules its *first* run one whole interval after start — 86400+ seconds for KEV. A fresh install had a healthy container, a registered feed,+ a created collection, and an empty page for twenty-four hours, with nothing+ anywhere reporting a fault. First poll is now jittered into the first twenty+ seconds.++- **CISA's edge returns 403 to whole networks regardless of user agent.**+ Observed with curl, httpx and a browser UA. Without a fallback every+ deployment on such a network gets permanently zero indicators and no error a+ user would see. The canonical `cisa.gov` URL stays primary; CISA's own+ `cisagov/kev-data` GitHub repository — same publisher, identical schema — is+ the fallback, and the log line names which source answered.++- **OpenSearch and Neo4j are no longer required by `threatintel`.** Its+ lifespan called `os_store.initialize()` with no `try`, which made a+ `full`-profile store a hard dependency of a CORE service. Both that call and+ the pipeline's bulk index are best-effort now: with OpenSearch you+ additionally get full-text IOC search; without it the feeds still write to+ Qdrant, which is what the console reads.++- **Ollama moved from the air-gapped overlay into CORE**, with its pinned+ `llama3.2:3b-instruct-q4_K_M` (~2 GB, CPU-only). The gateway had already+ moved into CORE and the models it pointed at had not, so a default install+ could route a key nobody had. `make up` now produces real triage verdicts+ from a real model with real token counts. `litellm` waits on the model pull+ completing rather than on Ollama being healthy, because a gateway that is up+ before the weights exist answers the first request — the one a new user+ makes — with "model not found". The aliases in `infra/litellm/config.yaml`+ now read their backend from the environment, so moving to a hosted provider+ is three variables in `.env` rather than an edit to a mounted config file.++- **Every auto-triage run on a default install was dead-lettered.**+ `record_auto_triage` bound the two cost columns directly while+ `complete_run` beside it used `COALESCE`; both are `NOT NULL`, and+ `estimated_cost_usd` is `None` for every run against a local model because+ there is no list price for `ollama_chat/…`. The UPDATE violated the+ constraint, the transaction rolled back, the worker retried three times and+ dead-lettered the alert — a real LLM call, real tokens, and no verdict in+ the console. Found only because a model shipped in CORE made it the normal+ case rather than an edge case.++- **A small model that stops mid-JSON no longer loses its verdict.** Measured+ against the bundled model, triage responses arrive with the closing quote+ and brace absent and `finish_reason: "stop"` — every field the caller reads+ present and correct, and the whole response discarded. The parser now closes+ what the model opened, and nothing more: a fragment too damaged to read+ still raises, so the deterministic fallback stays reachable rather than a+ verdict being invented from an empty object. A failed parse now logs a+ bounded excerpt of what the model actually said, which previously could only+ be discovered by reproducing the prompt by hand.++#### Other first-run friction++- **`ENVIRONMENT` was a bare literal in `docker-compose.yml`**, so the value+ `.env.example` documents was ignored and setting `production` changed+ nothing — including the dev auth bypass in `dev_auth.py`, which could not be+ switched off from `.env` at all. Now interpolated, along with `LOG_LEVEL`.+- **`AISOC_CONSOLE_URL` appeared in no `.env.example` entry, no compose+ service and no doc**, so every deployment that is not a laptop printed the+ wrong sign-in address beside a password shown exactly once. Now wired and+ documented.+- **`python3`, `bash` and the Docker disk requirement were undocumented** while+ `make smoke` — the README's headline proof — runs a Python script on the+ host. `make doctor` now checks for both interpreters, and the measured+ requirements are published.+- **`handler.go` rendered its over-size error with `string(rune(…))`**, so a+ limit of 1000 told the caller their batch exceeded a maximum of `Ϩ`.+- **`apps/docs/docs/operations/security.md` named an init script that does not+ exist** (`zz_runtime_role_password.sh`; the file is+ `20_runtime_role_password.sh`).++#### Measured, not estimated++CORE's published figures moved because the footprint was measured before and+after on the same machine:++| | Before | After |+|---|---|---|+| Services | 11 | 14 (plus a one-shot model pull) |+| Images, unique layers | 8.11 GB | 16.46 GB |+| Model weights (named volume) | — | 2.02 GB |+| Resident memory, whole stack | 1.72 GiB | 4.84 GiB |++The README's `~6.5 GB` becomes **8 GB of memory and 20 GB of disk**. Ollama is+9.6 MiB resident when idle and 2.96 GiB while serving a request; the second+number is the one the requirement is sized against.+### Added- **`scripts/generate_corpus_stats.py`** — generatesdiff --git a/Makefile b/Makefileindex e14a5716f..bdaf2386b 100644--- a/Makefile+++ b/Makefile@@ -6,12 +6,14 @@# documentation without a red build.## Profiles (see docs/architecture/README.md):-# core — default. postgres, redis, kafka, ingest, fusion, api, web, agents.-# The smallest deployment that can take an event and produce an-# alert. ~6GB RAM.+# core — default. postgres, redis, kafka, ingest, fusion, api, web, agents,+# the LLM gateway and the local model behind it, and the threat-intel+# feed with its vector store. The smallest deployment that can take an+# event, produce an alert, and triage it with a real model — with no+# credentials at all. ~8GB RAM.# full — core plus the event lake (ClickHouse), entity graph (Neo4j),-# vector store (Qdrant), search (OpenSearch), enrichment, connectors-# and the LLM gateway. ~12GB RAM.+# full-text search (OpenSearch), enrichment and scheduled connectors.+# ~12GB RAM.# demo — core plus clearly-labelled synthetic data..DEFAULT_GOAL := help@@ -23,7 +25,7 @@ PROFILE ?=# `--profile full` when PROFILE=full, nothing otherwise.PROFILE_ARG := $(if $(PROFILE),--profile $(PROFILE),)-.PHONY: help install up up-full down restart status doctor smoke demo logs clean \+.PHONY: help install env up up-full down restart status doctor smoke demo logs clean \bootstrap test test-unit test-integration test-e2e stats papers \papers-install demo-script@@ -31,6 +33,7 @@ help:@echo ""@echo " Getting started"@echo " make install Install prerequisites, configure, start, and verify"+ @echo " make env Create .env and generate its secrets (run by make up)"@echo " make up Start the CORE stack (postgres, kafka, ingest, fusion, api, web)"@echo " make up-full Start CORE plus lake, graph, vector, search and enrichment"@echo " make bootstrap Create the first administrator and print its password once"@@ -56,12 +59,22 @@ help:install:./install.sh+# Creates .env from .env.example and writes a real random value for each secret+# that is empty or still a placeholder. Idempotent: a second run leaves values+# it already generated alone, so `up` can depend on it unconditionally.+#+# This runs *before* compose because compose only interpolates `.env`; there is+# no `env_file` directive anywhere in docker-compose.yml. A value that is wrong+# at `up` time is baked into every container's environment until the next `up`.+env:+ @$(PYTHON) scripts/ensure_env.py+# The port check runs before compose, not after it fails. `docker compose up`# reports a conflict as `Bind for 127.0.0.1:5432 failed: port is already# allocated` against whichever container lost the race, which names neither the# process holding the port nor what to do about it. Half the stack is running# by then, so the error also arrives after a minute of unrelated output.-up: _ports+up: env _ports$(COMPOSE) up -d@echo ""@echo "Waiting for services to become healthy…"@@ -101,7 +114,7 @@ bootstrap:# 8123 stopped the stack after eight containers had already started, with# `Bind for 0.0.0.0:8123 failed: port is already allocated` naming neither the# process holding it nor what to do.-up-full: _ports+up-full: env _portsAISOC_LAKE_WRITER_ENABLED=true AISOC_GRAPH_ENABLED=true $(COMPOSE) --profile full up -d@$(MAKE) --no-print-directory _wait PROFILE=full@echo ""diff --git a/README.md b/README.mdindex d4e291f01..13bb80a5f 100644--- a/README.md+++ b/README.md@@ -23,11 +23,9 @@```Your security tools (EDR, cloud, identity, network, SIEM)|- v- ┌──────────────────────────────────────────────────────┐+ ┌─────────v────────────────────────────────────────────────┐│ normalize -> detect -> correlate -> investigate -> respond │- └──────────────────────────────────────────────────────┘- |+ └─────────|────────────────────────────────────────────────┘vSOC analyst: one incident, with the evidence and the reasoning```@@ -40,36 +38,40 @@ recorded, and proposes an action. A human approves before anything executes.```bashgit clone https://github.com/beenuar/AiSOC && cd AiSOC-cp .env.example .envmake up```-`make up` finishes by creating an administrator and printing its password.-That password is generated on your machine, shown once, and stored nowhere —-copy it before the terminal scrolls. If you lose it, mint a new one with+Needs Docker Compose v2 with **8 GB memory and 20 GB free disk in the Docker+VM**, plus `python3` (3.9+) and `bash` — `make doctor` checks all of it, and+[Installation](https://beenuar.github.io/AiSOC/docs/installation#requirements)+says what each number was measured against. The first run downloads a ~2 GB+language model into a named volume; only `make clean` fetches it again.++`make up` also creates `.env` and generates the three secrets in it — the+credential-vault key, the session signing key, and the service-to-service+token — then creates an administrator and prints its password. That password+is generated on your machine, shown once, and stored nowhere: copy it before+the terminal scrolls, or mint a new one with`make bootstrap ARGS=--reset-password`.-Then **prove it actually works** — this is the part that matters:--```bash-make smoke-```--That posts one real event to the ingest API and follows it through Kafka,-detection, correlation and Postgres, then reads the resulting alert back out-of the public API. Every stage reports PASS or FAIL:+Then **prove it actually works** — this is the part that matters. `make smoke`+posts one real event to the ingest API, follows it through Kafka, detection,+correlation and Postgres, and reads the resulting alert back out of the public+API. Every stage reports PASS or FAIL:```+$ make smoke[PASS] raw telemetry accepted by ingest[PASS] event traversed the spine and became an alert[PASS] alert is retrievable by id from the API```Open **http://localhost:3000** and sign in with the credentials `make up`-printed (API docs at **http://localhost:8000/api/docs**).+printed (API docs at **http://localhost:8000/api/docs**). Deploying somewhere+that is not your laptop? Set `AISOC_CONSOLE_URL` in `.env` so the printed+address is the one people browse to.-Something wrong? `make doctor` checks every dependency and tells you what to-run next. Requires Docker with ~6.5 GB of RAM.+Something wrong? `make doctor` checks every dependency and says what to run next.## Try it without connecting anything@@ -77,10 +79,9 @@ run next. Requires Docker with ~6.5 GB of RAM.make demo```-> **The demo dataset is synthetic.** It exists to show the pipeline shape, not-> to represent real activity. Every row it writes is marked-> `is_synthetic = true` in the database and labelled in the console. It is not-> a benchmark, a customer, or a real incident.+> **The demo dataset is synthetic.** It shows the pipeline shape, not real+> activity. Every row is marked `is_synthetic = true` in the database and+> labelled in the console. It is not a benchmark, a customer, or an incident.## Connect real data@@ -98,12 +99,11 @@ Or pull, by configuring one of **84 click-and-connect data connectors** in**Settings → Connectors** (needs the `full` profile). Those withvendor-specific normalization and live setup docs include Splunk, MicrosoftSentinel, Elastic, CrowdStrike, Okta, AWS (GuardDuty / CloudTrail / Security-Hub), Wiz, and Kubernetes audit logs. The full list is in the+Hub), Wiz, and Kubernetes audit logs — full list in the[connector docs](https://beenuar.github.io/AiSOC/docs/connectors/api-coverage).--A connector without a vendor profile still ingests through a generic mapping,-which resolves host, user and source IP from the usual spellings so the alert-is still pivotable. A vendor profile adds that vendor's own field names on top.+A connector without a vendor profile still ingests through a generic mapping+that resolves host, user and source IP from the usual spellings, so the alert+is pivotable either way.## How it works@@ -121,23 +121,28 @@ response.| Profile | Command | Services | RAM | What you get ||---|---|---|---|---|-| **core** | `make up` | 11 | ~6.5 GB | The full alerting pipeline: ingest → detect → correlate → alert → triage → console, plus the LLM gateway |-| **full** | `make up-full` | 21 | ~12 GB | Core plus event lake, entity graph, vector store, enrichment, scheduled connectors |-| **demo** | `make up && make demo` | 11 | ~6.5 GB | Core plus labelled synthetic data |--CORE is not a cut-down toy — it is the smallest deployment that takes a real-event and produces a real alert.--**What CORE can and cannot do about AI.** The LLM gateway ships in CORE, so a-provider key in `.env` reaches a model with no profile change and no second-command. That route is proven against a **local** model — one real completion,-measured, through Ollama on an operator's own hardware. **No hosted provider-has ever been exercised**: this project has no funded key, so its published-per-model rows read *not measured* rather than zero. Until you supply a key-there is no AI — every alert takes the deterministic path and the console says-so. The gateway is also the only thing that can report what a call cost, so an-unmeasured cost renders as absent, never as `$0.00`. Reasoning:-[ADR-0006](docs/decisions/0006-llm-gateway-in-core.md).+| **core** | `make up` | 14 | ~8 GB | The full alerting pipeline: ingest → detect → correlate → alert → triage → console, plus the LLM gateway, a local model, and the CISA KEV threat feed |+| **full** | `make up-full` | 22 | ~12 GB | Core plus event lake, entity graph, full-text search, enrichment, scheduled connectors |+| **demo** | `make up && make demo` | 14 | ~8 GB | Core plus labelled synthetic data |++CORE is the smallest deployment that takes a real event and produces a real+alert, and **it needs no credentials to do either** — for two reasons.++**The model ships with the gateway.** Ollama runs a pinned ~2 GB+`llama3.2:3b-instruct-q4_K_M` sized for CPU-only inference, so `make up`+produces real triage verdicts with real token counts in the Investigation+Ledger — not a stub. A 3B quantized model is not a frontier model; to upgrade,+set `OPENAI_API_KEY`, `AISOC_LLM_MODEL_FAST`, `AISOC_LLM_MODEL_DEEP` and an+empty `AISOC_LLM_API_BASE` in `.env`, then `make up` again. **No hosted+provider has ever been exercised here** — there is no funded key, so per-model+rows read *not measured* rather than zero, and an unmeasured cost renders as+absent, never `$0.00`. ([ADR-0006](docs/decisions/0006-llm-gateway-in-core.md))++**One real external feed ships too.** `services/threatintel` polls the+[CISA Known Exploited Vulnerabilities](https://www.cisa.gov/known-exploited-vulnerabilities-catalog)+catalog — authoritative, public, no API key — into the console's Threat+Intelligence page: the one thing in a fresh install that is neither synthetic+nor yours.## Real vs synthetic data@@ -146,6 +151,7 @@ This matters more than any feature, so it is stated plainly.| Kind | Where | How you can tell ||---|---|---|| **Real** | Your connectors and the ingest API | `is_synthetic = false` (the default) |+| **Real, and not yours** | The CISA KEV feed on the Threat Intelligence page | Every row carries `source: cisa-kev`; it is the public catalog, unmodified || **Demo** | `make demo` | `is_synthetic = true`, labelled in the console || **Benchmark** | `services/agents/tests/eval_data/` | Every published row carries `substrate: true` || **Test fixtures** | `tests/`, `**/tests/` | Never shipped in an image |@@ -173,8 +179,9 @@ Agents triage alerts and investigate incidents. What they can and cannot do:approver must hold the required permission tier and must not be the personwho requested the action.-Without a model provider key, agents run a deterministic offline path and say-so. They do not fabricate a verdict.+CORE's bundled local model means agents reason for real out of the box. If no+model is reachable at all they run a deterministic offline path and say so —+they never fabricate a verdict.## Project maturity@@ -207,25 +214,21 @@ so. They do not fabricate a verdict.## Troubleshooting-`make doctor` diagnoses the deployment and prints the command to run next.-The five most common failures:--| Symptom | Cause | Fix |-|---|---|---|-| `port is already allocated` | Something else on 5432/6379/9092/3000 | `make doctor` names the port; stop it or edit the host port |-| Kafka reports healthy but nothing flows | Docker VM out of disk — Kafka corrupts its log dir and still passes its healthcheck | `docker system prune -af`, then `make clean && make up` |-| `make smoke` fails at "became an alert" | fusion is down or not consuming | `docker compose logs fusion \| tail -60` |-| Console loads but is empty | No data yet — this is correct | `make smoke`, or `make demo` |-| Services restart-loop on 8 GB machines | Not enough RAM for `full` | Use `make up` (core) |+`make doctor` diagnoses the deployment and prints the command to run next. It+checks the host tools, free memory and disk in the Docker VM, every port, each+datastore by querying it rather than by asking whether its container is up, and+whether `.env` still holds template placeholders. The six failures it is most+often right about — and what each one actually means — are tabulated under+[Installation → Troubleshooting](https://beenuar.github.io/AiSOC/docs/installation#the-six-most-common-failures).## Security-Secrets live in `.env` and are never committed; connector credentials are-encrypted at rest with a per-deployment key. Tenant isolation is enforced at-the query layer in every store, not by convention. RBAC gates every mutating-route. Prompts are validated before they leave the deployment, and you can-bring your own model key or run entirely local models. Report vulnerabilities-via [SECURITY.md](SECURITY.md).+Secrets live in `.env`, are generated per deployment, and are never committed;+connector credentials are encrypted at rest with a per-deployment key. Tenant+isolation is enforced at the query layer in every store, not by convention.+RBAC gates every mutating route. Prompts are validated before they leave the+deployment, and the default install never sends one anywhere — the model runs+beside it. Report vulnerabilities via [SECURITY.md](SECURITY.md).## Developing@@ -238,10 +241,9 @@ make stats # recount every figure this README publishesGuides: [add a connector](https://beenuar.github.io/AiSOC/docs/plugins/hello-plugin) ·[add a detection](https://beenuar.github.io/AiSOC/docs/detections/hello-hunt) ·[plugin lifecycle](https://beenuar.github.io/AiSOC/docs/plugins/lifecycle) ·-[contributing](CONTRIBUTING.md)--The connector and detection-rule counts here are recounted from the tree by-`scripts/project_stats.py`, which CI fails if this README disagrees with it.+[contributing](CONTRIBUTING.md). The connector and detection-rule counts above+are recounted from the tree by `scripts/project_stats.py`, which CI fails if+this README disagrees with it.## Roadmap · Contributing · Licensediff --git a/apps/docs/docs/deployment/env-vars.md b/apps/docs/docs/deployment/env-vars.mdindex 1c1825526..78dd0716e 100644--- a/apps/docs/docs/deployment/env-vars.md+++ b/apps/docs/docs/deployment/env-vars.md@@ -31,10 +31,15 @@ The API uses bare environment variable names (no prefix). Booleans accept `true`| Variable | Default | Description ||----------|---------|-------------|-| `SECRET_KEY` | `change-me-in-production-at-least-32-chars` | **Required in production.** Signs primary access/refresh JWTs. Generate with `openssl rand -hex 32`. |+| `SECRET_KEY` | `change-me-in-production-at-least-32-chars` | **Required in production.** Signs primary access/refresh JWTs. `make env` generates one into `.env`; the connectors service must see the same value. Generate by hand with `openssl rand -hex 32`. || `ACCESS_TOKEN_EXPIRE_MINUTES` | `30` | Lifetime of an access token || `REFRESH_TOKEN_EXPIRE_DAYS` | `7` | Lifetime of a refresh token || `ALGORITHM` | `HS256` | JWT signing algorithm |+| `AISOC_CREDENTIAL_KEY` | _empty_ | Fernet key encrypting connector credentials at rest. Empty means "not configured" and the API generates an **ephemeral** key per process, so saved credentials do not survive a restart. Any other invalid value raises, and every connector save answers HTTP 500 — which is why `.env.example` ships it empty rather than as a placeholder. `make env` writes a real one. |+| `AISOC_CREDENTIAL_KEY_ROTATION_FROM` | _empty_ | Comma-separated previous keys, accepted for decryption only, for zero-downtime rotation. |+| `AISOC_SERVICE_TOKEN` | _empty_ | Shared bearer for service-to-service calls. The API **sends** it when proxying the connector catalog and "Test connection" to `services/connectors`, which **verifies** it — so a value set on one side only is a 401 the wizard reports as a failed connection test. `make env` generates one and compose passes it to both. |+| `AISOC_CONNECTORS_SERVICE_TOKEN` | _empty_ | Per-service override for the above, if you would rather not share one secret. |+| `AISOC_CONSOLE_URL` | `http://localhost:3000` | Base URL `make bootstrap` prints as the sign-in address. Set it to the address operators actually browse to — any non-localhost deployment otherwise prints the wrong one beside a credential shown exactly once. |### Migration runnerdiff --git a/apps/docs/docs/installation.md b/apps/docs/docs/installation.mdindex 114d92c79..655dfde26 100644--- a/apps/docs/docs/installation.md+++ b/apps/docs/docs/installation.md@@ -31,6 +31,36 @@ When the installer finishes, the CORE stack is running atwas printed once during the run, and one real event has been pushedthrough the pipeline and read back out of the API as an alert.+## Requirements++What CORE (`make up`) actually needs, and what each figure was measured+against rather than estimated from.++| | CORE | Why |+|---|---|---|+| Docker memory | **8 GB** | Steady-state RSS across the fourteen CORE containers is well under this. The headroom is for the bundled local model, whose process grows to roughly the model size (~2 GB) while a request is in flight. |+| Docker disk | **20 GB** | Images plus the ~2 GB model plus room for Kafka, Postgres and Qdrant volumes to grow. Below roughly 2 GB free, Kafka corrupts its log directory **and still passes its healthcheck** — `make doctor` fails on this before it can happen. |+| `python3` | **3.9+** | `make up` runs `scripts/ensure_env.py` on the host to create `.env` and generate its secrets, and `make smoke` — the headline proof — is a host-side Python script. CI runs 3.11. |+| `bash` | any | Every script under `scripts/` has a bash shebang. macOS's bundled 3.2 is sufficient. |+| `docker compose` | **v2** | The v1 `docker-compose` binary is not supported; profiles and `service_completed_successfully` both need v2. |++`make up-full` adds ClickHouse, Neo4j, OpenSearch and the enrichment and+connector services, and wants ~12 GB of memory.++None of this needs a credential. There is no account to create, no API key to+obtain, and no paid service in the default path.++### The six most common failures++| Symptom | Cause | Fix |+|---|---|---|+| `port is already allocated` | Something else on 5432/6379/9092/3000 | `make doctor` names the process or container holding it; stop it, or change the host port |+| Kafka reports healthy but nothing flows | Docker VM out of disk — Kafka corrupts its log dir and its healthcheck still passes | `docker system prune -af`, then `make clean && make up` |+| `make smoke` fails at "became an alert" | fusion is down or not consuming | `docker compose logs fusion \| tail -60` |+| Console loads but is empty | No data yet — this is correct | `make smoke`, or `make demo` |+| Services restart-loop on 8 GB machines | Not enough memory for `full` | Use `make up` (CORE) |+| `credential vault unavailable` on saving a connector | `AISOC_CREDENTIAL_KEY` in `.env` is set to something that is not a valid Fernet key | `make env` generates a real one, then `docker compose up -d api` |+## Supported platforms| OS | Package manager | Tested versions |diff --git a/apps/docs/docs/operations/security.md b/apps/docs/docs/operations/security.mdindex 7da7a5eb9..408f647fa 100644--- a/apps/docs/docs/operations/security.md+++ b/apps/docs/docs/operations/security.md@@ -147,7 +147,7 @@ A view executes its underlying reads as the **view's owner** unless it is declar#### What an operator still has to do-The compose stacks and CI need nothing: `infra/postgres/initdb/zz_runtime_role_password.sh` runs inside the postgres image's first-boot init, after the migration chain and before the container reports healthy, and sets the runtime role's password from `AISOC_APP_DB_PASSWORD`. `depends_on: service_healthy` makes that ordering a guarantee rather than a race.+The compose stacks and CI need nothing: `infra/postgres/initdb/20_runtime_role_password.sh` runs inside the postgres image's first-boot init, after the migration chain and before the container reports healthy, and sets the runtime role's password from `AISOC_APP_DB_PASSWORD`. `depends_on: service_healthy` makes that ordering a guarantee rather than a race.Three cases are **not** automatic:diff --git a/apps/docs/docs/quickstart.md b/apps/docs/docs/quickstart.mdindex 2af367bf2..680a554cb 100644--- a/apps/docs/docs/quickstart.md+++ b/apps/docs/docs/quickstart.md@@ -234,32 +234,37 @@ does not.| Profile | Services || --- | --- |-| *(default — CORE)* | **postgres** (5432) · **redis** (6379) · **zookeeper** · **kafka** (9092) · **litellm** (4000, LLM gateway) · **ingest-worker** (8081, Go) · **fusion** (8003) · **api** (8000) · **agents** (8001) · **realtime** (8086) · **web** (3000) |-| `full` | **clickhouse** (8123/9000) · **neo4j** (7474/7687) · **qdrant** (6333) · **opensearch** (9200) · **threatintel** (8005) · **ueba** (8007) · **connectors** (8088) · **enrichment** (8080, Go) · **kafka-ui** (8090) · **actions** (8002) |+| *(default — CORE)* | **postgres** (5432) · **redis** (6379) · **zookeeper** · **kafka** (9092) · **qdrant** (6333) · **litellm** (4000, LLM gateway) · **ollama** (11434, local model) · **ollama-pull** (one-shot model fetch) · **ingest-worker** (8081, Go) · **fusion** (8003) · **api** (8000) · **agents** (8001) · **threatintel** (8005) · **realtime** (8086) · **web** (3000) |+| `full` | **clickhouse** (8123/9000) · **neo4j** (7474/7687) · **opensearch** (9200) · **ueba** (8007) · **connectors** (8088) · **enrichment** (8080, Go) · **kafka-ui** (8090) · **actions** (8002) || `extras` | **honeytokens** (8008) · **purple-team** (8006) || `chatops` | **slack-bot** (8009) · **actions** (8002) || `monitoring` | **prometheus** (9091) · **grafana** (3001) · **alertmanager** (9094) · **tempo** (3200) · **otel-collector** (4317/4318) || `osquery` | **osquery-tls** (8091) |-CORE is 11 services and `full` is 21 — `actions` is in both `full` and+CORE is 14 long-running services — 15 counting `ollama-pull`, which fetches+the model once and exits — and `full` is 22; `actions` is in both `full` and`chatops`, so it is counted once. Named profiles compose, so`docker compose --profile full --profile extras up -d` is valid; it just doesnot wait for health the way `make up-full` does.-**What this gives you about AI.** Every command above starts the CORE profile,-which includes the `litellm` LLM gateway. AiSOC asks for logical task aliases-(`aisoc-triage`, `aisoc-investigation`, …) and the gateway is the only thing-that resolves them, so it has to be running for AI triage to happen at all —-it was a `full`-profile service until 2026-09, which meant the default install-could not do AI triage even with the key you set+**What this gives you about AI.** CORE contains both halves: the `litellm`+gateway, which is the only thing that resolves AiSOC's task aliases+(`aisoc-triage`, `aisoc-investigation`, …), and `ollama` running a pinned+~2 GB `llama3.2:3b-instruct-q4_K_M` behind it. Neither was there to begin with,+and moving only the first was only half the fix: a gateway with no model still+left the default install routing a key nobody had([ADR-0006](https://github.com/beenuar/AiSOC/blob/main/docs/decisions/0006-llm-gateway-in-core.md)).-- **With a provider key** in `.env`: alerts are triaged by the AI, and the- cost dashboard reports what each call actually cost, because the gateway- reports it.-- **Without one**: the gateway still boots and the stack is healthy, but no- LLM call is made. Every alert is triaged by the deterministic path and the- console labels it as such. Nothing pretends the AI ran.+- **Out of the box, no key**: alerts are triaged by a real model. Real+ generated text, real token counts in the Investigation Ledger. A 3B+ quantized model is not a frontier model, and the console does not claim it+ is.+- **With a provider key**: set `OPENAI_API_KEY`, `AISOC_LLM_MODEL_FAST`,+ `AISOC_LLM_MODEL_DEEP` and an empty `AISOC_LLM_API_BASE` in `.env`, then+ `make up` again. The cost dashboard then reports what each call actually+ cost, because the gateway reports it.+- **If no model is reachable at all**: every alert takes the deterministic+ path and the console labels it as such. Nothing pretends the AI ran.### Database migrations — nothing to rundiff --git a/apps/web/src/components/connectors/AddConnectorModal.tsx b/apps/web/src/components/connectors/AddConnectorModal.tsxindex af0951f9c..078de581b 100644--- a/apps/web/src/components/connectors/AddConnectorModal.tsx+++ b/apps/web/src/components/connectors/AddConnectorModal.tsx@@ -48,6 +48,34 @@ import {type TroubleshootResponse,} from '@/lib/api';+/**+ * Turn a thrown request error into something an operator can act on.+ *+ * `ApiError.message` is the status line — `API 502 Bad Gateway — /api/v1/…`.+ * Everything that identifies the fault lives in `body`, which FastAPI sends as+ * `{"detail": "…"}`. This matters most for the failure that is not the+ * customer's: when the API cannot authenticate to AiSOC's own connectors+ * service, the detail names the status, the service and the variable to set,+ * and the status line alone names none of them.+ */+function describeApiError(err: unknown, fallback: string): string {+ if (err instanceof ApiError) {+ if (err.body) {+ try {+ const parsed = JSON.parse(err.body) as { detail?: unknown };+ if (typeof parsed.detail === 'string' && parsed.detail.trim()) {+ return parsed.detail;+ }+ } catch {+ // Not JSON — a proxy error page or a truncated body. Fall through to+ // the status line, which at least names the status.+ }+ }+ return err.message;+ }+ return err instanceof Error ? err.message : fallback;+}+// ─── Field rendering helpers ────────────────────────────────────────────────/**@@ -1385,7 +1413,12 @@ export function AddConnectorModal({toast.error(result.error ?? result.message ?? 'Connection test failed');}} catch (err) {- const msg = err instanceof Error ? err.message : 'Test request failed';+ // `ApiError.message` is only "API 502 Bad Gateway — /api/v1/connectors/test".+ // The reason the operator can act on — which service refused, with what+ // status, and what to set — is in the response body, so a bare `.message`+ // here reproduced the dead end this panel is meant to have stopped+ // showing. Prefer the API's `detail`, fall back to the status line.+ const msg = describeApiError(err, 'Test request failed');setTestResult({ success: false, error: msg });toast.error(msg);} finally {@@ -1419,7 +1452,10 @@ export function AddConnectorModal({setCreatedConnector(created);setStep('verify');} catch (err) {- const msg = err instanceof Error ? err.message : 'Failed to save connector';+ // Same reasoning as handleTest: a save refused by the credential vault+ // answers 500 with a detail that names AISOC_CREDENTIAL_KEY, and the+ // status line names nothing.+ const msg = describeApiError(err, 'Failed to save connector');toast.error(msg);} finally {setSaving(false);diff --git a/docker-compose.yml b/docker-compose.ymlindex ae41360d7..e91da436a 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -62,6 +62,9 @@ volumes:qdrant_data:neo4j_data:neo4j_logs:+ # ~2 GB of model weights. Named so `make down` keeps them and only+ # `make clean` re-downloads.+ ollama_data:services:@@ -266,8 +269,14 @@ services:mem_reservation: 1gqdrant:- # full profile — Vector store for threat-intel embeddings (services/threatintel).- profiles: ["full"]+ # CORE — the store `services/threatintel` writes collected indicators to,+ # and the one the console's /threat-intel page is served from. It moved in+ # with threatintel: see that service's comment for the reasoning.+ #+ # Measured: 245 MB image, 79 MiB RSS with the two threat-intel collections+ # created. It is by a wide margin the cheapest of the four stores — the+ # next one up, OpenSearch, is 2 GB of image and ~1.05 GB resident — which+ # is why it is the one CORE takes.image: qdrant/qdrant:v1.7.0container_name: aisoc-qdrantports:@@ -276,6 +285,16 @@ services:- qdrant_data:/qdrant/storagenetworks:- aisoc+ healthcheck:+ # threatintel creates its collections at startup, so it needs the store+ # answering rather than merely existing. The image ships neither curl nor+ # wget; `/proc/net/tcp` is what is left, and a listener on 6333 (hex+ # 18BD) is the thing being asserted.+ test: ["CMD-SHELL", "grep -q ':18BD' /proc/net/tcp || exit 1"]+ interval: 10s+ timeout: 5s+ retries: 12+ start_period: 20sneo4j:# full profile — Entity graph. Backs graph traversals and blast radius, not alert generation.@@ -361,8 +380,34 @@ services:# match whatever the connectors service verifies with, or the console# token the API forwards there fails its signature check.SECRET_KEY: ${SECRET_KEY:-dev_secret_key_change_in_production}- ENVIRONMENT: development- LOG_LEVEL: info+ # The credential this service presents when it proxies the connector+ # catalog and "Test connection" to the connectors service.+ # `_service_token()` in app/api/v1/endpoints/connectors.py reads+ # AISOC_CONNECTORS_SERVICE_TOKEN then AISOC_SERVICE_TOKEN — and compose+ # interpolated the shared token into connectors, ueba, honeytokens and+ # purple-team but not into the one service that has to *send* it. There+ # is no `env_file` here either, so a value in .env could not reach it by+ # any route. The catalog call was answered 401 on every request and fell+ # back to the bundled copy; the fix that made the proxy read the right+ # variable shipped against a variable nothing ever set.+ AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}+ AISOC_CONNECTORS_SERVICE_TOKEN: ${AISOC_CONNECTORS_SERVICE_TOKEN:-}+ CONNECTORS_SERVICE_URL: ${CONNECTORS_SERVICE_URL:-http://connectors:8003}+ # Where GET /api/v1/threat-intel/indicators is served from. The console+ # has called that path since the IOC inbox shipped and nothing answered+ # it; threatintel is in CORE now and this is the wire.+ THREATINTEL_SERVICE_URL: ${THREATINTEL_SERVICE_URL:-http://threatintel:8005}+ AISOC_THREATINTEL_SERVICE_TOKEN: ${AISOC_THREATINTEL_SERVICE_TOKEN:-}+ # Printed by `make bootstrap` as the sign-in address and used as the+ # console's public origin. Read from os.environ by+ # app/scripts/bootstrap_admin.py, which no compose service set — so any+ # deployment that is not a laptop printed the wrong address.+ AISOC_CONSOLE_URL: ${AISOC_CONSOLE_URL:-http://localhost:3000}+ # Was a bare literal, so `.env`'s documented ENVIRONMENT was ignored and+ # `production` changed nothing — including the dev auth bypass in+ # app/api/v1/dev_auth.py, which could not be switched off from .env at all.+ ENVIRONMENT: ${ENVIRONMENT:-development}+ LOG_LEVEL: ${LOG_LEVEL:-info}# Issue #478 — LiteLLM gateway. This service's live LLM endpoints# (translation, hunts, KB, phishing, NL-detection) route here by default:# `app.services.model_aliases.resolve_base_url` reads LLM_GATEWAY_URL for@@ -575,6 +620,62 @@ services:# exported on /metrics (scraped as job `aisoc-litellm`). Lightweight and inert# unless a service is pointed at it via OPENAI_BASE_URL — see the note on the# `agents` service above.+ ollama:+ # CORE — the model behind the gateway.+ #+ # The gateway shipped in CORE and the models it pointed at did not: every+ # alias resolved to a hosted provider, so the default install could route a+ # key it did not have and otherwise did no AI at all. This closes that: a+ # clone with no account produces a real triage verdict, from a real model,+ # with real token counts.+ #+ # Promoted from infra/compose/docker-compose.airgap.yml, where this exact+ # pairing has been proven end to end. The model is pinned rather than+ # floating on `latest` for the same reason every other image here is.+ #+ # Measured: 6.92 GB image — by some distance the largest thing in CORE —+ # 2.02 GB of weights in the volume, and 9.6 MiB resident when idle.+ #+ # Quoting the idle figure alone would be the misleading half of the truth:+ # the model is mapped in on first inference and the container measured+ # 2.96 GiB while serving. That is what moved the README's memory figure to+ # 8 GB, not the idle number.+ image: ollama/ollama:0.6.7+ container_name: aisoc-ollama+ volumes:+ - ollama_data:/root/.ollama+ ports:+ - "127.0.0.1:11434:11434"+ healthcheck:+ test: ["CMD-SHELL", "ollama list >/dev/null 2>&1 || exit 1"]+ interval: 10s+ timeout: 5s+ retries: 18+ start_period: 30s+ networks:+ - aisoc+ restart: unless-stopped++ ollama-pull:+ # One-shot: pull the pinned model, then exit. A cache hit is a no-op, so+ # this costs one `ollama list` on every `up` after the first.+ #+ # litellm waits on `service_completed_successfully` here rather than on+ # ollama's health: a gateway that is up before the weights exist answers the+ # first triage request with "model not found", and the first request is the+ # one a new user makes.+ image: ollama/ollama:0.6.7+ container_name: aisoc-ollama-pull+ depends_on:+ ollama:+ condition: service_healthy+ entrypoint: ["ollama", "pull", "${AISOC_LLM_LOCAL_MODEL:-llama3.2:3b-instruct-q4_K_M}"]+ environment:+ OLLAMA_HOST: http://ollama:11434+ networks:+ - aisoc+ restart: "no"+litellm:# CORE, deliberately — see docs/decisions/0006-llm-gateway-in-core.md.#@@ -599,13 +700,30 @@ services:command: ["--config", "/etc/litellm/config.yaml", "--port", "4000"]volumes:- ./infra/litellm/config.yaml:/etc/litellm/config.yaml:ro+ depends_on:+ ollama-pull:+ condition: service_completed_successfullyenvironment:# Clients authenticate to the gateway with this key (AiSOC sends it as its# OPENAI_API_KEY when OPENAI_BASE_URL points here). Override in .env.LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY:-sk-aisoc-local}+ # Which backend every alias resolves to. infra/litellm/config.yaml reads+ # these through `os.environ/...` instead of naming a model inline, so the+ # local→hosted upgrade is three variables in .env rather than an edit to a+ # mounted config file.+ #+ # The defaults are the bundled local model, which is what makes AI triage+ # work in a default `make up` with no account and no key. Set+ # OPENAI_API_KEY plus these two to openai/gpt-4o-mini and openai/gpt-4o —+ # and clear AISOC_LLM_API_BASE — to route to a hosted provider instead.+ AISOC_LLM_MODEL_FAST: ${AISOC_LLM_MODEL_FAST:-ollama_chat/llama3.2:3b-instruct-q4_K_M}+ AISOC_LLM_MODEL_DEEP: ${AISOC_LLM_MODEL_DEEP:-ollama_chat/llama3.2:3b-instruct-q4_K_M}+ # `-` and not `:-`, deliberately: an operator moving to a hosted provider+ # sets AISOC_LLM_API_BASE= (empty) in .env, and `:-` would helpfully undo+ # that by substituting the Ollama address back in.+ AISOC_LLM_API_BASE: ${AISOC_LLM_API_BASE-http://ollama:11434}# Upstream provider keys the aliases resolve against (os.environ/... in the- # config). Empty is fine — the gateway still boots, serves /health and- # /metrics, and AiSOC falls back to its deterministic path.+ # config). Empty is fine — the local model answers instead.OPENAI_API_KEY: ${OPENAI_API_KEY:-}ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}ports:@@ -727,9 +845,6 @@ services:condition: service_healthyports:- "127.0.0.1:8088:8003"- # Phase A3 — the connector runtime is part of the default stack now (was- # profile-gated). It idles harmlessly with no configured instances; a cold- # `up` therefore ships the full ingest→detect→respond spine out of the box.environment:# app/scripts/serve.py binds PORT, defaulting to 8087. The mapping above# publishes container port 8003, so leaving PORT unset put the listener@@ -742,6 +857,17 @@ services:# API image instead of the 84 connectors the registry actually has.SECRET_KEY: ${SECRET_KEY:-dev_secret_key_change_in_production}AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}+ # The third way this service answered 503 to everything. `SECRET_KEY`'s+ # compose default is one of INSECURE_SECRET_DEFAULTS, so+ # `resolve_console_secret()` returns "" for it; with AISOC_SERVICE_TOKEN+ # also blank, `require_console_or_service_auth` has no credential+ # material at all and fails closed — correctly. The dev-mode escape that+ # exists for exactly this case was set on actions, ueba, honeytokens,+ # purple-team and slack-bot, and not on connectors. Dev mode only applies+ # when nothing is configured, so once `make env` generates a real+ # AISOC_SERVICE_TOKEN this flag stops having any effect.+ AISOC_DEV_MODE: ${AISOC_DEV_MODE:-1}+ ENVIRONMENT: ${ENVIRONMENT:-development}KAFKA_BOOTSTRAP_SERVERS: kafka:29092REDIS_URL: redis://:redis_dev_secret@redis:6379/6DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc@@ -764,8 +890,19 @@ services:restart: unless-stoppedthreatintel:- # full profile — IOC harvesting and actor attribution. Needs Qdrant.- profiles: ["full"]+ # CORE — the only thing in the default install that puts *real*, externally+ # sourced security data in front of a new user on day one.+ #+ # `app/clients/cisa_kev.py` fetches the CISA Known Exploited Vulnerabilities+ # catalog: authoritative, public, and needing no API key. It has been wired+ # as a scheduled handler the whole time and sat in a profile nobody starting+ # out runs. Meanwhile the console shipped a /threat-intel page calling+ # `/api/v1/threat-intel/indicators`, which had nothing behind it in CORE and+ # was for a while filled with five invented IOCs. The missing feed and the+ # fabrication were the same hole: there was no real data to show, so+ # something invented some.+ #+ # Measured: 1.4 GB image, 219 MiB RSS steady state.build:context: ./services/threatinteldockerfile: Dockerfile@@ -777,13 +914,16 @@ services:condition: service_healthykafka:condition: service_healthy- # OpenSearch is a hard dependency, not an optional one: the lifespan- # calls os_store.initialize() without a try/except, so starting before- # it is up kills the container and leaves `restart: unless-stopped` to- # paper over the race. `service_healthy` is the condition that actually- # expresses that; `service_started` only says the container exists.- opensearch:+ qdrant:condition: service_healthy+ # OpenSearch is deliberately NOT a dependency any more. It was one because+ # the lifespan called os_store.initialize() without a try/except, so an+ # unreachable node killed the container — which made a `full`-profile store+ # a hard requirement of a CORE service. Both that call and the pipeline's+ # bulk index are best-effort now: with OpenSearch present you additionally+ # get full-text IOC search, and without it the feeds still write to Qdrant,+ # which is what the console reads. `make up-full` starts it and this+ # service picks it up on its next restart.ports:- "127.0.0.1:8005:8005"environment:@@ -799,6 +939,10 @@ services:NEO4J_URI: bolt://neo4j:7687NEO4J_USER: neo4jNEO4J_PASSWORD: neo4j_dev_secret+ # Same credential the API presents when it proxies /threat-intel/indicators.+ SECRET_KEY: ${SECRET_KEY:-dev_secret_key_change_in_production}+ AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}+ AISOC_DEV_MODE: ${AISOC_DEV_MODE:-1}MISP_URL: ${MISP_URL:-}MISP_API_KEY: ${MISP_API_KEY:-}OTX_API_KEY: ${OTX_API_KEY:-}diff --git a/docs/architecture/README.md b/docs/architecture/README.mdindex 45b8b78c5..d528f7343 100644--- a/docs/architecture/README.md+++ b/docs/architecture/README.md@@ -251,17 +251,30 @@ One architecture, three profiles of it — not three architectures.| Profile | Command | Services | RAM | What you get ||---|---|---|---|---|-| **core** | `make up` | 11 | ~6.5 GB | Ingest → detect → correlate → alert → triage → console, plus the LLM gateway. The full alerting pipeline. |-| **full** | `make up-full` | 21 | ~12 GB | Core plus event lake, entity graph, vector store, enrichment, connectors. |-| **demo** | `make up && make demo` | 11 | ~6.5 GB | Core plus clearly-labelled synthetic data. |+| **core** | `make up` | 14 | ~8 GB | Ingest → detect → correlate → alert → triage → console, plus the LLM gateway, the local model behind it, and the CISA KEV threat feed with its vector store. |+| **full** | `make up-full` | 22 | ~12 GB | Core plus event lake, entity graph, full-text search, enrichment, scheduled connectors. |+| **demo** | `make up && make demo` | 14 | ~8 GB | Core plus clearly-labelled synthetic data. |-The LLM gateway moved into CORE this release, so a provider key works without-a profile change. `full` is 21 services, not the 30 published here previously:-30 is `full` plus the `monitoring`, `chatops`, `extras` and `osquery`-profiles, which `make up-full` does not start.+`full` is 22 services, not the 30 published here previously: 30 is `full` plus+the `monitoring`, `chatops`, `extras` and `osquery` profiles, which+`make up-full` does not start. (The CORE count is fourteen long-running+containers; `ollama-pull` is a fifteenth that runs once and exits.)CORE is not a toy. It is the smallest deployment that can take a real event-and produce a real alert, which is the thing the product is for.+and produce a real alert, which is the thing the product is for — **and it does+that, and triages the result with a real model, with no credentials**. Three+services moved into CORE to make that last clause true:++* **`litellm`** — the gateway. Every `aisoc-<role>` alias resolves here and+ nowhere else, so a CORE deployment *with* a provider key could not use it.+* **`ollama` (+ a one-shot `ollama-pull`)** — the model behind the gateway,+ pinned at `llama3.2:3b-instruct-q4_K_M` (~2 GB, CPU-only). Promoted from the+ air-gapped overlay, where this pairing was already proven end to end. Without+ it the gateway in CORE could route a key nobody had.+* **`threatintel` + `qdrant`** — the CISA Known Exploited Vulnerabilities feed,+ which is public and keyless, and the store it writes to. OpenSearch and Neo4j+ stay in `full`: both sinks are best-effort, so their absence costs full-text+ IOC search and the actor graph, not the feed.**Two flags travel with the `full` profile.** The lake writer and thegraph writer target stores that exist only there, so in CORE they default offdiff --git a/docs/decisions/0006-llm-gateway-in-core.md b/docs/decisions/0006-llm-gateway-in-core.mdindex a5aed06af..3ee7f8cde 100644--- a/docs/decisions/0006-llm-gateway-in-core.md+++ b/docs/decisions/0006-llm-gateway-in-core.md@@ -114,3 +114,64 @@ against the tree and was wrong: `make up-full` runs `--profile full`, which isdoes not start. Corrected rather than left, since the table is being editedanyway and a published number that is not true is the thing this repositorykeeps having to find.++## Addendum — the gateway without a model was half the fix++*Added when `ollama` was promoted into CORE.*++This ADR moved the gateway and stopped there, and the consequence list above+says so in plain sight: "AI triage works in CORE the moment an operator+supplies a provider key". Read against the problem statement — an evaluator+following the documented path cannot see the product's central claim work —+that is the same failure one step further along. The evaluator now has a+gateway that routes correctly to a provider they do not have an account with.+`make up` still produced no AI.++The missing half was a model, and one was already in the tree: the air-gapped+overlay (`infra/compose/docker-compose.airgap.yml`) has run Ollama with a+pinned `llama3.2:3b-instruct-q4_K_M` — ~2 GB, quantized, sized for CPU-only+inference in 8 GB of RAM — and the full triage path has been proven end to end+against it. It was reachable only by choosing an overlay named for air-gapped+deployments, which nobody evaluating the product on a laptop is looking for.++**Decision: promote `ollama` and its one-shot `ollama-pull` into CORE.** A+default `make up` now does real AI triage — real tokens, real generated text,+a real cost figure off the gateway's headers — with no account and no key.++Three details worth recording, because each was a choice:++- **`litellm` waits on `ollama-pull` completing, not on `ollama` being+ healthy.** A gateway that is up before the weights exist answers the first+ triage request with "model not found", and the first request is the one a new+ user makes.+- **The aliases read their backend from the environment** rather than naming a+ model inline, so `infra/litellm/config.yaml` no longer has to be edited to+ move to a hosted provider — it is `OPENAI_API_KEY` plus two model variables+ in `.env`. The mounted config stays the place to go for *per-alias*+ divergence, which is what it is actually good at.+- **A 3B quantized model is not a frontier model,** and the README says so+ rather than implying the local default is equivalent. The upgrade is+ signposted in the same paragraph as the claim.++### What it costs, measured++Same method as the gateway's numbers above, on the same machine, with the+whole of CORE running.++| | Before | After |+|---|---|---|+| CORE services | 11 | **14** (plus a one-shot `ollama-pull`) |+| Images, unique layers | 8.11 GB | **16.46 GB** |+| Images, sum of sizes | 11.47 GB | **20.03 GB** |+| Model weights (named volume) | — | **2.02 GB** |+| Resident memory, whole stack | 1.72 GiB | **4.84 GiB** |++Per added service: `ollama` **6.92 GB** image / **9.6 MiB** idle,+`threatintel` **1.40 GB** / **219 MiB**, `qdrant` **245 MB** / **79 MiB**.++Ollama's 6.92 GB image is the single largest thing in CORE and its 9.6 MiB+idle figure is the misleading half of the truth: the model is mapped in on+first inference and the container measured **2.96 GiB** while serving a+request. That, not the idle number, is why the README's memory figure moved+from ~6.5 GB to 8 GB and its disk figure to 20 GB. The trade is real and the+numbers are published so a self-hoster can judge it rather than discover it.diff --git a/docs/openapi.yaml b/docs/openapi.yamlindex b18afc18a..724448142 100644--- a/docs/openapi.yaml+++ b/docs/openapi.yaml@@ -11964,6 +11964,77 @@ paths:$ref: '#/components/schemas/HTTPValidationError'security:- HTTPBearer: []+ /api/v1/threat-intel/indicators:+ get:+ tags:+ - threat-intel+ summary: List Feed Indicators+ description: 'Indicators the feed scheduler has collected, from `services/threatintel`.+++ The console has called this path since the IOC inbox shipped and nothing++ served it — the page 404''d, and for a while rendered five invented IOCs++ instead. `services/threatintel` and its Qdrant store are in CORE now, and++ the CISA Known Exploited Vulnerabilities catalog is public and keyless, so++ a default install has real indicators to show within a poll interval.+++ Tenant scope is enforced upstream: this proxy asserts the caller''s tenant++ in the same header the connectors proxy uses, and the threatintel route++ filters on it plus the `shared` sentinel that public feed intel is written++ under.'+ operationId: list_feed_indicators_api_v1_threat_intel_indicators_get+ security:+ - HTTPBearer: []+ parameters:+ - name: type+ in: query+ required: false+ schema:+ anyOf:+ - type: string+ - type: 'null'+ description: filter to one indicator type+ title: Type+ description: filter to one indicator type+ - name: tag+ in: query+ required: false+ schema:+ anyOf:+ - type: string+ - type: 'null'+ title: Tag+ - name: q+ in: query+ required: false+ schema:+ anyOf:+ - type: string+ - type: 'null'+ description: substring match on value or description+ title: Q+ description: substring match on value or description+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/FeedIndicatorsResponse'+ '422':+ description: Validation Error+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/HTTPValidationError'/api/v1/threat-intel/iocs:get:tags:@@ -20156,6 +20227,44 @@ components:- name- feed_typetitle: FeedCreate+ FeedIndicatorsResponse:+ properties:+ indicators:+ items:+ additionalProperties: true+ type: object+ type: array+ title: Indicators+ total:+ type: integer+ title: Total+ source:+ type: string+ title: Source+ degraded:+ type: boolean+ title: Degraded+ default: false+ reason:+ type: string+ title: Reason+ default: ''+ type: object+ required:+ - indicators+ - total+ - source+ title: FeedIndicatorsResponse+ description: 'What the console''s `/threat-intel` page renders.+++ ``source`` and ``degraded`` travel with the payload for the same reason++ they do on the connector catalog: an empty list because nothing has been++ collected yet and an empty list because the service is not deployed are++ different facts, and a reader cannot tell them apart from the rows alone.'FeedOut:properties:name:diff --git a/infra/litellm/config.yaml b/infra/litellm/config.yamlindex 0d70a1eae..3933d1475 100644--- a/infra/litellm/config.yaml+++ b/infra/litellm/config.yaml@@ -18,75 +18,95 @@# OPENAI_BASE_URL=http://litellm:4000/v1# OPENAI_API_KEY=$LITELLM_MASTER_KEY#-# Air-gapped / fully-local deployments: swap the commented Ollama/vLLM blocks in-# for the `openai/*` lines — the alias names stay identical, so AiSOC is unaware-# of the switch. When no live model is reachable AiSOC still degrades to its-# deterministic offline path (the `deterministic` floor in model_pins.py).+# ── WHICH MODEL ANSWERS, AND HOW TO CHANGE IT ─────────────────────────────────+#+# Every alias below takes its backend from the environment rather than naming+# one inline, and docker-compose.yml supplies the defaults. Out of the box those+# defaults point at the **Ollama service bundled in CORE**, running a pinned+# ~2 GB llama3.2:3b-instruct-q4_K_M sized for CPU-only inference in 8 GB of RAM.+# That is why a default `make up` can do real AI triage with no API key and no+# account: the local model is the floor, not a stub.+#+# It is a 3B quantized model. It is genuinely doing the work — real tokens, real+# cost accounting, real generated text — and it is not as good as a frontier+# model. **To upgrade**, put a provider key in `.env` and set the two model+# variables beside it:+#+# OPENAI_API_KEY=sk-...+# AISOC_LLM_MODEL_FAST=openai/gpt-4o-mini+# AISOC_LLM_MODEL_DEEP=openai/gpt-4o+# AISOC_LLM_API_BASE=+#+# then `make up` again. No code change, no profile change, no edit to this file.+# Air-gapped deployments keep the defaults and never set a key.+#+# When no live model is reachable at all AiSOC still degrades to its+# deterministic offline path (the `deterministic` floor in model_pins.py) and+# reports zero model calls and zero tokens rather than pretending to triage.model_list:# ── Auto-triage: high volume, cheap+fast. ─────────────────────────────────────- model_name: aisoc-triagelitellm_params:- model: openai/gpt-4o-mini+ model: os.environ/AISOC_LLM_MODEL_FAST+ api_base: os.environ/AISOC_LLM_API_BASEapi_key: os.environ/OPENAI_API_KEY# ── Recon / enrichment reasoning. ─────────────────────────────────────────────- model_name: aisoc-reconlitellm_params:- model: openai/gpt-4o-mini+ model: os.environ/AISOC_LLM_MODEL_FAST+ api_base: os.environ/AISOC_LLM_API_BASEapi_key: os.environ/OPENAI_API_KEY# ── Deep investigation: the multi-step reasoning path — worth a stronger model.- model_name: aisoc-investigationlitellm_params:- model: openai/gpt-4o+ model: os.environ/AISOC_LLM_MODEL_DEEP+ api_base: os.environ/AISOC_LLM_API_BASEapi_key: os.environ/OPENAI_API_KEY# ── Contextual copilot: interactive analyst Q&A. ──────────────────────────────- model_name: aisoc-copilotlitellm_params:- model: openai/gpt-4o-mini+ model: os.environ/AISOC_LLM_MODEL_FAST+ api_base: os.environ/AISOC_LLM_API_BASEapi_key: os.environ/OPENAI_API_KEY# ── Summaries: alert/incident condensation. ───────────────────────────────────- model_name: aisoc-summarylitellm_params:- model: openai/gpt-4o-mini+ model: os.environ/AISOC_LLM_MODEL_FAST+ api_base: os.environ/AISOC_LLM_API_BASEapi_key: os.environ/OPENAI_API_KEY# ── Reports: analyst-facing write-ups — quality over cost. ────────────────────- model_name: aisoc-reportlitellm_params:- model: openai/gpt-4o+ model: os.environ/AISOC_LLM_MODEL_DEEP+ api_base: os.environ/AISOC_LLM_API_BASEapi_key: os.environ/OPENAI_API_KEY# ── Natural-language generation: NL→query / NL→detection translation. ─────────- model_name: aisoc-nllitellm_params:- model: openai/gpt-4o-mini+ model: os.environ/AISOC_LLM_MODEL_FAST+ api_base: os.environ/AISOC_LLM_API_BASEapi_key: os.environ/OPENAI_API_KEY- # ── Local / air-gapped example (Ollama) ───────────────────────────────────────- # Duplicate any alias above with a local backend to keep the call on-prem. The- # alias name MUST stay the same so AiSOC is unaware of the swap:- #- # - model_name: aisoc-triage- # litellm_params:- # model: ollama/llama3.1- # api_base: http://ollama:11434+ # ── Other backends ────────────────────────────────────────────────────────────+ # The two variables above take any LiteLLM model string, so most swaps need no+ # edit here at all:#- # ── Self-hosted vLLM (OpenAI-compatible) example ──────────────────────────────- # - model_name: aisoc-investigation- # litellm_params:- # model: openai/Qwen2.5-32B-Instruct- # api_base: http://vllm:8000/v1- # api_key: os.environ/VLLM_API_KEY+ # self-hosted vLLM AISOC_LLM_MODEL_DEEP=openai/Qwen2.5-32B-Instruct+ # AISOC_LLM_API_BASE=http://vllm:8000/v1+ # Anthropic AISOC_LLM_MODEL_DEEP=anthropic/claude-3-5-sonnet-latest+ # ANTHROPIC_API_KEY=... (and clear AISOC_LLM_API_BASE)+ # a bigger local AISOC_LLM_MODEL_DEEP=ollama_chat/llama3.1:8b+ # (pull it first: docker compose exec ollama ollama pull llama3.1:8b)#- # ── Anthropic example ─────────────────────────────────────────────────────────- # - model_name: aisoc-report- # litellm_params:- # model: anthropic/claude-3-5-sonnet-latest- # api_key: os.environ/ANTHROPIC_API_KEY+ # Per-alias divergence — a different model for one workload only — is the case+ # that does need an edit: replace that alias's `os.environ/...` with a literal.# ── Gateway behaviour ─────────────────────────────────────────────────────────litellm_settings:diff --git a/install.sh b/install.shindex adf7fad83..1f5527916 100755--- a/install.sh+++ b/install.sh@@ -732,6 +732,22 @@ ensure_env_file() {# in that file), so .env isn't actually load-bearing for the demo. But# several apps and scripts do read .env, so we make sure it exists with# the example defaults to avoid spurious "key not found" warnings.+ #+ # A plain `cp` is what this used to do, and a plain `cp` is what broke the+ # vault: `.env.example` shipped a non-empty placeholder for+ # AISOC_CREDENTIAL_KEY, which fails Fernet validation and turns every+ # connector save into an HTTP 500. scripts/ensure_env.py does the copy *and*+ # writes real random values for the secrets that need generating. It is+ # idempotent, so it is safe to run over an existing .env.+ if [ -x "$(command -v python3 || true)" ] && [ -f "$REPO_ROOT/scripts/ensure_env.py" ]; then+ if python3 "$REPO_ROOT/scripts/ensure_env.py" --env "$REPO_ROOT/.env" --example "$REPO_ROOT/.env.example"; then+ ok "$REPO_ROOT/.env is configured"+ info " (Optional: set OPENAI_API_KEY in $REPO_ROOT/.env to upgrade from the bundled local model.)"+ return 0+ fi+ warn "scripts/ensure_env.py failed; falling back to a plain copy."+ fi+if [ -f "$REPO_ROOT/.env" ]; thenok ".env already exists at $REPO_ROOT/.env"return 0@@ -739,7 +755,7 @@ ensure_env_file() {if [ -f "$REPO_ROOT/.env.example" ]; thencp "$REPO_ROOT/.env.example" "$REPO_ROOT/.env"ok "Created $REPO_ROOT/.env from .env.example"- info " (Optional: edit $REPO_ROOT/.env to add your OpenAI/Anthropic API key for richer agent runs.)"+ warn "Generated secrets were not filled in (no python3). Run: make env"elsewarn "No .env.example found in repo; skipping .env creation."fidiff --git a/scripts/check_env_placeholders.py b/scripts/check_env_placeholders.pynew file mode 100644index 000000000..059864b8e--- /dev/null+++ b/scripts/check_env_placeholders.py@@ -0,0 +1,150 @@+#!/usr/bin/env python3+"""Find values in a dotenv file that are placeholders rather than settings.++Why this exists as a script rather than a grep+----------------------------------------------+``scripts/doctor.sh`` used to carry the check inline::++ grep -qE '^[A-Z_]*(SECRET|PASSWORD|KEY)=(change_me|changeme|)$' .env++That pattern matches neither placeholder this repository actually ships.+``.env.example`` sets ``AISOC_CREDENTIAL_KEY`` to+``replace-me-with-a-freshly-generated-fernet-key`` and ``SECRET_KEY`` to+``change-this-to-a-random-secret-key-at-least-32-chars``; the grep looks for+``change_me``, ``changeme`` or an empty value. So the one gate built to catch a+shipped placeholder was blind to every shipped placeholder, and the documented+quick start — ``cp .env.example .env`` — handed the API an invalid Fernet key+that only surfaced as an HTTP 500 at the connector wizard, minutes later and+several screens away from the cause.++The lesson is not "write a better regex". It is that the detector and the file+it inspects were two separate hand-maintained lists that nothing compared.+``tests/test_env_placeholder_gate.py`` compares them: every non-empty value in+``.env.example`` must be either flagged here or named in that test's list of+deliberate working defaults, so a new placeholder cannot be added without one+of the two failing.++An *empty* value is deliberately not a placeholder. Most of the optional+credentials in ``.env.example`` ship empty and empty is their correct value —+flagging them would produce a warning on a correctly configured deployment,+which is how an operator learns to ignore a check.+"""++from __future__ import annotations++import argparse+import re+import sys+from pathlib import Path++# `scripts/` is on sys.path when this file is run as a program, but not when a+# test loads it by path with importlib. gate_toolkit sits beside it either way.+sys.path.insert(0, str(Path(__file__).resolve().parent))++from gate_toolkit import repo_root, self_test_if_requested++self_test_if_requested(__file__)++#: The shapes a placeholder takes in this repository. Matched case-insensitively+#: against the *value*, anywhere in it — a placeholder is prose, so anchoring+#: would only make the pattern brittle.+#:+#: Each entry is here because something in the tree matches it, or because it is+#: the obvious spelling a contributor would reach for next. Keep it that way:+#: a pattern nothing matches is untested.+PLACEHOLDER_PATTERNS: tuple[str, ...] = (+ r"replace[-_ ]?me", # AISOC_CREDENTIAL_KEY+ r"change[-_ ]?me", # the historical spelling the old grep looked for+ r"change[-_ ]?this", # SECRET_KEY+ r"fill[-_ ]?(me|this)",+ r"\byour[-_][a-z0-9]", # OPENAI_API_KEY=sk-your-openai-api-key-here+ r"[-_]here$", # …-api-key-here+ r"^<.+>$", # <paste-your-token>+ r"placeholder",+ r"\btodo\b",+ r"\bxxxx+",+)++_COMPILED = tuple(re.compile(p, re.IGNORECASE) for p in PLACEHOLDER_PATTERNS)++#: `KEY=value`, tolerating leading whitespace and `export `.+_ASSIGNMENT = re.compile(r"^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=(.*)$")+++def is_placeholder(value: str) -> bool:+ """True when ``value`` is something the reader was meant to replace."""+ stripped = value.strip().strip("\"'")+ if not stripped:+ return False+ return any(pattern.search(stripped) for pattern in _COMPILED)+++def parse_env(text: str) -> list[tuple[int, str, str]]:+ """Return ``(line_number, key, value)`` for every assignment in ``text``."""+ entries: list[tuple[int, str, str]] = []+ for lineno, line in enumerate(text.splitlines(), start=1):+ if line.lstrip().startswith("#"):+ continue+ match = _ASSIGNMENT.match(line)+ if match:+ entries.append((lineno, match.group(1), match.group(2)))+ return entries+++def findings(path: Path) -> list[tuple[int, str, str]]:+ """Every placeholder assignment in the dotenv file at ``path``."""+ return [(lineno, key, value.strip()) for lineno, key, value in parse_env(path.read_text(encoding="utf-8")) if is_placeholder(value)]+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ parser.add_argument(+ "paths",+ nargs="*",+ type=Path,+ help="dotenv files to inspect (default: .env.example at the repository root)",+ )+ parser.add_argument(+ "--allow-placeholders",+ action="store_true",+ help=(+ "exit 0 even when placeholders are found. This is how the template itself is "+ "inspected: .env.example is *supposed* to contain them, and the caller only "+ "wants the list."+ ),+ )+ args = parser.parse_args(argv)++ paths = args.paths or [repo_root() / ".env.example"]++ missing = [p for p in paths if not p.is_file()]+ if missing:+ for path in missing:+ print(f"check_env_placeholders: ERROR — {path} does not exist", file=sys.stderr)+ return 1++ found = 0+ for path in paths:+ for lineno, key, value in findings(path):+ found += 1+ print(f"{path}:{lineno}: {key} is still a placeholder ({value!r})")++ if not found:+ print(f"check_env_placeholders: OK — no placeholder values in {', '.join(str(p) for p in paths)}")+ return 0++ if args.allow_placeholders:+ return 0++ print(+ "\nThese are template values, not settings. A non-empty placeholder is worse than an "+ "empty one: AISOC_CREDENTIAL_KEY fails Fernet validation and the API answers HTTP 500 "+ "on every connector save, while an empty value takes the documented development path.\n"+ "Generate real values with: make env",+ file=sys.stderr,+ )+ return 1+++if __name__ == "__main__":+ sys.exit(main())diff --git a/scripts/doctor.sh b/scripts/doctor.shindex 9cb65e331..4caa713c7 100755--- a/scripts/doctor.sh+++ b/scripts/doctor.sh@@ -83,6 +83,25 @@ elsepass "docker compose v2 available"fi+# Docker is not the only host requirement and the other two were undocumented.+# `make smoke` — the README's headline proof — is a Python script run on the+# host, not in a container, and `make up` generates `.env` with another one. An+# operator without python3 gets a traceback from the command the quick start+# tells them to trust.+if have python3; then+ pass "python3 available ($(python3 --version 2>&1 | awk '{print $2}'))"+else+ fail "python3 is not installed — make smoke and make env run on the host" "https://www.python.org/downloads/ (3.9+; CI uses 3.11)"+fi++# Every script under scripts/ has a bash shebang and uses bash-only syntax.+# macOS ships bash 3.2, which is enough for all of them.+if have bash; then+ pass "bash available"+else+ fail "bash is not installed — every script under scripts/ needs it" "install bash from your package manager"+fi+# Skipped under --ports-only: `make up` runs that mode as a pre-flight, and the# disk probe starts a container. Advisories about free space do not belong in# the path between typing `make up` and the stack starting.@@ -90,22 +109,29 @@ if [ "$PORTS_ONLY" = "0" ] && docker info >/dev/null 2>&1; then# Disk exhaustion inside the Docker VM corrupts Kafka's log dir and the# broker then reports healthy while refusing every request. That exact# failure cost hours, so it is checked before anything else.+ #+ # The thresholds are measured, not guessed: CORE is 16.5GB of unique image+ # layers plus a 2GB model volume, so 20GB is the floor at which a first+ # `make up` completes with room for Postgres, Kafka and Qdrant to grow.avail_raw="$(docker run --rm --entrypoint sh alpine:3 -c 'df -P /' 2>/dev/null | awk 'NR==2{print $4}')"if [ -n "${avail_raw:-}" ]; thenavail_gb=$((avail_raw / 1024 / 1024))if [ "$avail_gb" -lt 5 ]; thenfail "docker has ${avail_gb}GB free — Kafka will corrupt its log dir below ~2GB" "docker system prune -af && docker volume prune -f"- elif [ "$avail_gb" -lt 15 ]; then- warn "docker has ${avail_gb}GB free (15GB+ recommended for the full profile)" "docker system prune -af"+ elif [ "$avail_gb" -lt 20 ]; then+ warn "docker has ${avail_gb}GB free — CORE needs ~20GB (16.5GB of images plus a 2GB model)" "docker system prune -af"elsepass "docker disk space: ${avail_gb}GB free"fifi+ # CORE measured 4.84GiB resident with the local model loaded, so 8GB is the+ # floor rather than the old 6: the model is mapped in on first inference and+ # the ollama container grows to roughly the model size while serving.mem_bytes="$(docker info --format '{{.MemTotal}}' 2>/dev/null || echo 0)"mem_gb=$((mem_bytes / 1024 / 1024 / 1024))- if [ "$mem_gb" -gt 0 ] && [ "$mem_gb" -lt 6 ]; then- fail "docker has ${mem_gb}GB RAM — CORE needs 6GB, full needs 12GB" "raise the memory limit in Docker Desktop → Settings → Resources"+ if [ "$mem_gb" -gt 0 ] && [ "$mem_gb" -lt 8 ]; then+ fail "docker has ${mem_gb}GB RAM — CORE needs 8GB (the bundled local model reaches ~3GB while answering), full needs 12GB" "raise the memory limit in Docker Desktop → Settings → Resources"elif [ "$mem_gb" -gt 0 ]; thenpass "docker memory: ${mem_gb}GB"fi@@ -189,11 +215,32 @@ fihead2 "Configuration"if [ -f .env ]; thenpass ".env present"- if grep -qE '^[A-Z_]*(SECRET|PASSWORD|KEY)=(change_me|changeme|)$' .env 2>/dev/null; then- warn ".env has unset or placeholder secrets" "./install.sh regenerates them, or edit .env by hand"+ # Delegated to scripts/check_env_placeholders.py rather than grepped here.+ # The grep this replaced was+ # ^[A-Z_]*(SECRET|PASSWORD|KEY)=(change_me|changeme|)$+ # which matched neither placeholder the repository actually shipped, so the+ # one check built to catch them reported clean on the exact .env that broke+ # the vault. A detector and the file it inspects cannot be two hand-kept+ # lists; tests/test_env_placeholder_gate.py now compares them.+ if have python3; then+ if placeholders="$(python3 scripts/check_env_placeholders.py .env 2>/dev/null)" && [ -z "${placeholders##*OK*}" ]; then+ pass ".env has no placeholder values"+ else+ fail ".env still contains template placeholders:"$'\n'"$(printf '%s\n' "$placeholders" | sed 's/^/ /')" \+ "make env # generates real values for the secrets that need them"+ fi+ else+ warn "cannot check .env for placeholders: python3 is not installed" "install python3 — make smoke and make env need it too"+ fi+ # A generated secret that is still empty is not fatal (the vault falls back+ # to an ephemeral development key) but it does mean saved connector+ # credentials will not survive a restart, which is worth saying out loud.+ if have python3 && ! python3 scripts/ensure_env.py --check >/dev/null 2>&1; then+ warn ".env has generated secrets that are still unset — connector credentials will not survive a restart" \+ "make env"fielse- warn ".env not found — compose will fall back to built-in dev defaults" "cp .env.example .env"+ warn ".env not found — compose will fall back to built-in dev defaults" "make env"fi# ── 4. Datastores — probed, not just 'running' ──────────────────────────────diff --git a/scripts/ensure_env.py b/scripts/ensure_env.pynew file mode 100644index 000000000..8a9de5bda--- /dev/null+++ b/scripts/ensure_env.py@@ -0,0 +1,177 @@+#!/usr/bin/env python3+"""Create `.env` and fill its generated secrets with real values.++The problem this solves+-----------------------+``README.md`` documents three commands: ``git clone``, ``cp .env.example .env``,+``make up``. The middle one used to break the deployment.++``.env.example`` shipped+``AISOC_CREDENTIAL_KEY=replace-me-with-a-freshly-generated-fernet-key``.+``docker-compose.yml`` interpolates that straight into the API container, and+``app.security.credential_vault`` takes its friendly ephemeral-development-key+path only when the key is **empty** — a non-empty invalid key reaches+``Fernet()`` and raises, which the connector endpoints turn into HTTP 500+``credential vault unavailable``. Nothing failed at boot, so the operator met+the failure minutes later at the connector wizard, with no line connecting it+back to the ``cp``. Skipping the documented ``cp`` produced a *working* vault+and following the README produced a broken one.++The fix chosen, and why+-----------------------+Generate real values at setup time rather than teaching the vault to tolerate a+placeholder. Tolerating it would mean a deployment whose connector credentials+silently do not survive a restart — a worse failure than the loud one, because+it is invisible until a customer's saved credential stops decrypting. The+placeholder is a *setup* defect and setup is where it is fixed.++Two layers, because either alone still leaves a hole:++1. This script, run by ``make up`` / ``make env`` / ``install.sh``, writes a+ freshly generated value for each secret below whenever the current one is+ empty or a placeholder. The documented quick start therefore ends with a+ vault that works *and* persists.+2. ``.env.example`` now ships these three **empty** rather than as prose. An+ operator who hand-copies the template and never runs the generator gets the+ documented development path (an ephemeral key, logged as a warning) instead+ of a hard 500. Empty is a value the vault already understands; the+ placeholder was not.++Secrets are written, never printed: the output says which names were filled.+"""++from __future__ import annotations++import argparse+import base64+import os+import re+import secrets+import sys+from collections.abc import Callable+from pathlib import Path++# `scripts/` is on sys.path when this file is run as a program, but not when a+# test loads it by path with importlib. gate_toolkit sits beside it either way.+sys.path.insert(0, str(Path(__file__).resolve().parent))++from check_env_placeholders import is_placeholder, parse_env+from gate_toolkit import repo_root, self_test_if_requested++# `--check` is a verdict, so it answers `--self-test` like every other verdict+# in this tree: over a repository with no content there is neither a `.env` nor+# a template to copy one from, and the only honest answer is to refuse.+self_test_if_requested(__file__)+++def _fernet_key() -> str:+ """A 32-byte urlsafe-base64 key, the format `cryptography.fernet` requires.++ Generated from the standard library rather than by importing+ ``cryptography``: this runs on the *host*, before any container is built,+ and requiring a pip install to create a config file would put a dependency+ in front of ``make up``. The format is fully specified — 32 random bytes,+ urlsafe base64 — so there is nothing for the library to add.+ """+ return base64.urlsafe_b64encode(os.urandom(32)).decode("ascii")+++#: ``name -> (generator, why it exists)``. The comment is written into `.env`+#: above a key that had to be appended, so an operator reading the file later+#: can tell a generated secret from one they set.+GENERATED: dict[str, tuple[Callable[[], str], str]] = {+ "AISOC_CREDENTIAL_KEY": (+ _fernet_key,+ "Fernet key for the connector credential vault. Rotate via AISOC_CREDENTIAL_KEY_ROTATION_FROM.",+ ),+ "SECRET_KEY": (+ lambda: secrets.token_hex(32),+ "HS256 signing key for console sessions. Shared by the API and the connectors service.",+ ),+ "AISOC_SERVICE_TOKEN": (+ lambda: secrets.token_urlsafe(32),+ "Shared bearer for service-to-service calls (API -> connectors, ueba, honeytokens, purple-team).",+ ),+}+++def _assignment(key: str) -> re.Pattern[str]:+ return re.compile(rf"^(\s*(?:export\s+)?{re.escape(key)}\s*=)(.*)$", re.MULTILINE)+++def needs_filling(text: str, key: str) -> bool:+ """True when ``key`` is absent, empty, or still a placeholder in ``text``."""+ for _lineno, name, value in parse_env(text):+ if name != key:+ continue+ stripped = value.strip()+ return not stripped or is_placeholder(stripped)+ return True+++def fill(text: str, key: str, value: str, why: str) -> str:+ """Return ``text`` with ``key`` set to ``value``, appending it if absent."""+ pattern = _assignment(key)+ if pattern.search(text):+ return pattern.sub(lambda m: f"{m.group(1)}{value}", text, count=1)+ suffix = "" if text.endswith("\n") else "\n"+ return f"{text}{suffix}\n# {why}\n# Generated by scripts/ensure_env.py — keep it secret, keep it out of git.\n{key}={value}\n"+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description="Create .env and generate its secrets.")+ parser.add_argument("--env", type=Path, default=None, help="path to .env (default: repository root)")+ parser.add_argument("--example", type=Path, default=None, help="path to .env.example")+ parser.add_argument(+ "--check",+ action="store_true",+ help="report what would change and exit non-zero if anything would; write nothing",+ )+ args = parser.parse_args(argv)++ root = repo_root()+ env_path = args.env or root / ".env"+ example_path = args.example or root / ".env.example"++ created = False+ if not env_path.exists():+ if not example_path.is_file():+ print(f"ensure_env: ERROR — neither {env_path} nor {example_path} exists", file=sys.stderr)+ return 1+ if args.check:+ print(f"ensure_env: {env_path} does not exist (would be created from {example_path.name})")+ return 1+ env_path.write_text(example_path.read_text(encoding="utf-8"), encoding="utf-8")+ created = True++ text = env_path.read_text(encoding="utf-8")+ pending = [key for key in GENERATED if needs_filling(text, key)]++ if args.check:+ if pending:+ print("ensure_env: these are unset or still placeholders: " + ", ".join(pending))+ return 1+ print("ensure_env: OK — every generated secret has a real value")+ return 0++ for key in pending:+ generator, why = GENERATED[key]+ text = fill(text, key, generator(), why)++ if pending or created:+ env_path.write_text(text, encoding="utf-8")+ # 0600: this file now holds real secrets. The template it came from is+ # world-readable and nothing narrowed the mode after the copy.+ env_path.chmod(0o600)++ if created:+ print(f"ensure_env: created {env_path.name} from {example_path.name}")+ if pending:+ print("ensure_env: generated " + ", ".join(pending))+ elif not created:+ print("ensure_env: .env already has a real value for every generated secret — unchanged")+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/services/agents/app/agents/auto_triage_agent.py b/services/agents/app/agents/auto_triage_agent.pyindex aeac7a1c2..e12c18212 100644--- a/services/agents/app/agents/auto_triage_agent.py+++ b/services/agents/app/agents/auto_triage_agent.py@@ -18,6 +18,7 @@ from __future__ import annotationsimport jsonimport os+import reimport timefrom typing import Any@@ -188,6 +189,60 @@ def _build_alert_context(state: InvestigationState) -> str:return f"{sanitize_text(memory)}\n\n{telemetry}"+def _close_truncated_json(fragment: str) -> str:+ """Close an object the model started and did not finish.++ Small local models stop mid-object more or less routinely — with+ ``finish_reason: "stop"``, not a token limit, so there is nothing to raise+ by giving them more room. Measured against the model CORE ships+ (``llama3.2:3b-instruct-q4_K_M``), a triage response arrived as::++ {+ "verdict": "true_positive",+ "confidence": 0.8,+ "rationale": "…uncertainty remains due to the lack of IOCs.++ with the closing quote and brace simply absent. Every field the caller+ reads was present and correct; the response was discarded and the alert+ fell through to deterministic triage.++ This closes any open string and any unclosed brackets, and does nothing+ else. It cannot invent a field: a fragment that never reached ``verdict``+ still parses to an object without one, and the caller's+ ``normalize_disposition(..., default=TRUE_POSITIVE)`` fails safe to the+ conservative verdict exactly as it does for a malformed response today.+ """+ in_string = False+ escaped = False+ stack: list[str] = []+ for ch in fragment:+ if escaped:+ escaped = False+ continue+ if ch == "\\" and in_string:+ escaped = True+ continue+ if ch == '"':+ in_string = not in_string+ continue+ if in_string:+ continue+ if ch in "{[":+ stack.append("}" if ch == "{" else "]")+ elif ch in "}]" and stack:+ stack.pop()++ repaired = fragment+ if in_string:+ # Drop a dangling escape before closing, or the quote is consumed by it.+ if escaped:+ repaired = repaired[:-1]+ repaired += '"'+ # A trailing comma or bare key left by the cut is not recoverable; strip it.+ repaired = re.sub(r",\s*$", "", repaired)+ return repaired + "".join(reversed(stack))++def _parse_llm_response(text: str) -> dict[str, Any]:"""Extract the JSON verdict from the LLM response, tolerating markdown fences."""cleaned = text.strip()@@ -203,6 +258,10 @@ def _parse_llm_response(text: str) -> dict[str, Any]:end = cleaned.rfind("}") + 1if start >= 0 and end > start:data = json.loads(cleaned[start:end])+ elif start >= 0:+ # An object was opened and never closed. Repair once, then give up+ # and let the caller fall back to deterministic triage.+ data = json.loads(_close_truncated_json(cleaned[start:]))else:raise@@ -267,7 +326,15 @@ async def run_auto_triage(state: InvestigationState) -> InvestigationState:# Issue #571: do NOT swallow + return a null-verdict RUNNING state.# Raise a typed error so the caller falls back to deterministic triage# (or marks the alert needs_review) instead of completing with no verdict.- logger.error("Auto-triage LLM call failed", error=str(exc))+ #+ # The response excerpt is logged with it. A parse failure whose message+ # is a character offset into text nobody kept is not diagnosable: the+ # only way to find out what a model actually emitted was to reproduce+ # the prompt by hand against the gateway. Bounded at 400 characters,+ # and it is the model's own words about an alert this service already+ # logs the summary of.+ excerpt = str(locals().get("raw_text") or "")[:400].replace("\n", "\\n")+ logger.error("Auto-triage LLM call failed", error=str(exc), response_excerpt=excerpt)state.add_finding(f"Auto-triage LLM error: {exc}")_metrics["total_processed"] += 1raise AutoTriageError(str(exc)) from excdiff --git a/services/agents/app/investigator/ledger.py b/services/agents/app/investigator/ledger.pyindex 9ec9d7ba6..37cef0f52 100644--- a/services/agents/app/investigator/ledger.py+++ b/services/agents/app/investigator/ledger.py@@ -518,13 +518,30 @@ async def persist_auto_triage(# a list-price guess keyed on a gateway alias, so a local# deployment that spent nothing accrued a per-alert dollar# figure here and everywhere reading from here.+ #+ # COALESCE on both sums, matching `complete_run` above.+ # Both columns are NOT NULL, and `None` is the ordinary+ # value of `estimated_cost_usd` whenever no estimate was+ # made — which is every run against a local model, because+ # there is no list price for `ollama_chat/...`. Binding it+ # directly made the UPDATE violate the constraint, so the+ # whole transaction rolled back, the worker retried three+ # times and dead-lettered the alert. With a model shipped in+ # CORE that is not an edge case: it was *every* auto-triage+ # run on a default install, and the console showed no+ # verdict for an LLM call that had genuinely happened.+ #+ # Leaving the column at its DEFAULT 0 is not a claim that+ # the run cost nothing — `estimated_call_count` and+ # `unpriced_call_count` are what carry that distinction, and+ # they are written from the same tuple."""UPDATE investigation_runsSET status = 'completed', iterations = $2,total_tokens = $3,- total_cost_usd = $4,+ total_cost_usd = COALESCE($4, total_cost_usd),measured_call_count = $5,- estimated_cost_usd = $6,+ estimated_cost_usd = COALESCE($6, estimated_cost_usd),estimated_call_count = $7,unpriced_call_count = $8,completed_at = now()diff --git a/services/agents/tests/test_auto_triage_truncated_json.py b/services/agents/tests/test_auto_triage_truncated_json.pynew file mode 100644index 000000000..9f128d744--- /dev/null+++ b/services/agents/tests/test_auto_triage_truncated_json.py@@ -0,0 +1,99 @@+"""A small local model that stops mid-object must not lose its verdict.++CORE ships a model now — `llama3.2:3b-instruct-q4_K_M`, ~2 GB, CPU-only — so+the auto-triage parser's tolerance stopped being a nicety. Measured against+that model on the real triage prompt, a response came back as::++ {+ "verdict": "true_positive",+ "confidence": 0.8,+ "rationale": "…some uncertainty remains due to the lack of IOCs.++`finish_reason` was `"stop"`, not a token limit — the model simply ended+without closing the string or the object. Every field the caller reads was+present and correct. `json.loads` raised, the brace-slice fallback found no+closing `}` to slice to, and the alert fell through to deterministic triage:+a real LLM call, real tokens billed, and its verdict thrown away.++The repair closes what was opened and nothing more. These tests pin that it+cannot become a verdict generator — the case it must still refuse is as+important as the ones it must fix.+"""++from __future__ import annotations++import json++import pytest+from app.agents.auto_triage_agent import _close_truncated_json, _parse_llm_response+from app.agents.dispositions import TRUE_POSITIVE+++class TestRepair:+ """The closer, on its own."""++ @pytest.mark.parametrize(+ ("fragment", "expected"),+ [+ (+ '{\n "verdict": "true_positive",\n "confidence": 0.8,\n "rationale": "no IOCs present.',+ {"verdict": "true_positive", "confidence": 0.8, "rationale": "no IOCs present."},+ ),+ ('{"verdict": "benign", "confidence": 0.9', {"verdict": "benign", "confidence": 0.9}),+ ('{"verdict": "benign", "confidence": 0.9,', {"verdict": "benign", "confidence": 0.9}),+ ('{"verdict": "benign", "findings": ["a", "b"', {"verdict": "benign", "findings": ["a", "b"]}),+ ('{"verdict": "benign"}', {"verdict": "benign"}),+ ],+ ids=["mid-string", "after-value", "trailing-comma", "open-array", "already-closed"],+ )+ def test_it_closes_what_the_model_opened(self, fragment: str, expected: dict) -> None:+ assert json.loads(_close_truncated_json(fragment)) == expected++ def test_a_backslash_at_the_cut_does_not_eat_the_closing_quote(self) -> None:+ """`"path C:\\` + `"` is an escaped quote, not a terminator."""+ assert json.loads(_close_truncated_json('{"rationale": "path C:\\\\')) == {"rationale": "path C:\\"}++ def test_a_quote_inside_the_rationale_is_not_mistaken_for_the_end(self) -> None:+ fragment = '{"verdict": "benign", "rationale": "the user ran \\"whoami\\" interactively'+ assert json.loads(_close_truncated_json(fragment))["rationale"] == 'the user ran "whoami" interactively'+++class TestParser:+ """The parser's behaviour end to end."""++ def test_a_truncated_verdict_is_recovered_rather_than_discarded(self) -> None:+ raw = '{\n "verdict": "true_positive",\n "confidence": 0.8,\n "rationale": "Encoded PowerShell under WINWORD.EXE.'+ result = _parse_llm_response(raw)+ assert result["verdict"] == TRUE_POSITIVE+ assert result["confidence"] == 0.8+ assert result["rationale"].startswith("Encoded PowerShell")++ def test_a_fragment_too_damaged_to_read_still_raises(self) -> None:+ """The deterministic fallback must still be reachable.++ Repairing this into `{}` would hand the caller a verdict-less object,+ which `normalize_disposition` would turn into a conservative+ `true_positive` — a fabricated verdict wearing a real one's clothes.+ Raising sends it to deterministic triage, which is honest about what+ produced it.+ """+ with pytest.raises(json.JSONDecodeError):+ _parse_llm_response('{"confi')++ def test_prose_with_no_object_at_all_still_raises(self) -> None:+ with pytest.raises(json.JSONDecodeError):+ _parse_llm_response("I am unable to assess this alert.")++ def test_a_well_formed_response_is_unaffected(self) -> None:+ raw = '{"verdict": "benign", "confidence": 0.95, "rationale": "scheduled scan"}'+ assert _parse_llm_response(raw) == {"verdict": "benign", "confidence": 0.95, "rationale": "scheduled scan"}++ def test_a_fenced_response_is_still_unfenced_first(self) -> None:+ raw = '```json\n{"verdict": "benign", "confidence": 0.5, "rationale": "x"}\n```'+ assert _parse_llm_response(raw)["verdict"] == "benign"++ def test_a_fenced_and_truncated_response_is_both_unfenced_and_closed(self) -> None:+ raw = '```json\n{"verdict": "needs_review", "confidence": 0.4, "rationale": "not enough'+ result = _parse_llm_response(raw)+ assert result["verdict"] == "needs_review"+ assert result["rationale"] == "not enough"diff --git a/services/api/app/api/v1/endpoints/connectors.py b/services/api/app/api/v1/endpoints/connectors.pyindex 75eaf02b2..cbcf5d06b 100644--- a/services/api/app/api/v1/endpoints/connectors.py+++ b/services/api/app/api/v1/endpoints/connectors.py@@ -515,10 +515,32 @@ async def _validate_connector_type(connector_type: str, tenant_id: uuid.UUID | s)+#: The remedy for a vault that will not encrypt. `.env.example` used to ship+#: ``AISOC_CREDENTIAL_KEY=replace-me-…``, which is non-empty and not a valid+#: Fernet key, so the documented ``cp .env.example .env`` produced this 500 on+#: every connector save while *not* copying the template produced a working+#: vault. The template now ships the key empty and `make env` generates a real+#: one; naming the command here means an operator who still hits this has the+#: fix in the message rather than in a changelog.+_VAULT_HINT = "Run `make env` to generate a valid AISOC_CREDENTIAL_KEY, then restart the API."++#: What an operator is told when the connectors service refuses this proxy's+#: credential. It names the status and the service because the alternative —+#: what the wizard showed for as long as this was broken — is the bare string+#: "Connection test failed", which points at the customer's Splunk instead of+#: at the deployment's own configuration.+_SERVICE_AUTH_HINT = (+ "AiSOC's API could not authenticate to its own connectors service, so your credentials were "+ "never sent upstream. Set AISOC_SERVICE_TOKEN to the same value in .env for both services "+ "(`make env` generates one) and restart, or check SECRET_KEY matches across them."+)++async def _proxy_test_connection(connector_type: str,auth_config: dict[str, Any],connector_config: dict[str, Any],+ tenant_id: uuid.UUID | str | None,) -> dict[str, Any]:"""POST plaintext credentials at the stateless test endpoint.@@ -528,6 +550,18 @@ async def _proxy_test_connection(service on the public internet, terminate TLS in front of it andadd a shared-secret header (mirroring how the realtime push proxy inthis codebase does it).++ ``tenant_id`` is required, not defaulted, for the same reason as in+ :func:`_fetch_catalog`: every route on the connectors service sits behind+ ``require_console_or_service_auth``, which refuses a service token that+ does not declare the tenant it acts for. This call used to send no+ ``headers=`` at all — the catalog call beside it was fixed and this one was+ not — so it was answered 401 on every single invocation, and 401 was the+ one status the ladder below did not branch on. The function then fell+ through to ``resp.json()`` and returned a body with no ``success`` key,+ which the wizard renders as "Connection test failed" while its own help+ text promises "Credentials are tested against the upstream API". The+ upstream API was never reached."""if not _CONNECTOR_TYPE_RE.match(connector_type):raise HTTPException(@@ -541,7 +575,7 @@ async def _proxy_test_connection(}try:async with httpx.AsyncClient(timeout=_CATALOG_TIMEOUT) as client:- resp = await client.post(url, json=payload)+ resp = await client.post(url, json=payload, headers=_catalog_headers(tenant_id))except httpx.HTTPError as exc:logger.warning("connectors_service.test.unreachable err=%s",@@ -549,7 +583,11 @@ async def _proxy_test_connection()raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE,- detail="connectors service is unavailable; cannot test connection",+ detail=(+ "AiSOC's connectors service is not reachable, so your credentials were never sent "+ "upstream. It ships in the `full` profile: start it with `make up-full`, or "+ "`docker compose up -d connectors` to add just this one."+ ),) from exc# 404 from the microservice = unknown connector_type. We've already@@ -561,6 +599,27 @@ async def _proxy_test_connection(status_code=status.HTTP_503_SERVICE_UNAVAILABLE,detail=f"connector '{connector_type}' is no longer available in the connectors service",)+ # 401/403 is this deployment's own service-to-service auth, never the+ # vendor's. Saying so is the difference between an operator checking+ # AISOC_SERVICE_TOKEN and an operator re-issuing a perfectly good Splunk+ # token because the product told them their credentials failed.+ if resp.status_code in (status.HTTP_401_UNAUTHORIZED, status.HTTP_403_FORBIDDEN):+ logger.error(+ "connectors_service.test.service_auth_rejected status=%s connector_type=%s",+ resp.status_code,+ # Sanitised inline rather than through `_safe_connector_type`.+ # CodeQL's taint tracker does not follow the helper across the+ # function boundary but does recognise this replace chain, so the+ # helper form raises py/log-injection (#924) on a value that was+ # already `_CONNECTOR_TYPE_RE`-validated at the top of this+ # function. Inline keeps the property visible to the scanner and+ # to the next reader.+ str(connector_type).replace("\r", "").replace("\n", " ")[:100],+ )+ raise HTTPException(+ status_code=status.HTTP_502_BAD_GATEWAY,+ detail=f"connectors service returned HTTP {resp.status_code} to this API's service credential. {_SERVICE_AUTH_HINT}",+ )if resp.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY:# Forward the schema-mismatch detail so the wizard can highlight# the offending field.@@ -571,7 +630,19 @@ async def _proxy_test_connection(if resp.status_code >= 500:raise HTTPException(status_code=status.HTTP_502_BAD_GATEWAY,- detail="connectors service failed while testing connection",+ detail=f"connectors service failed while testing connection (HTTP {resp.status_code})",+ )+ # Anything else in the 4xx range is still not a verdict about the vendor.+ # The ladder above used to end at >=500, so every unhandled status fell+ # through to the JSON decode below and returned whatever shape the body+ # happened to have — which for an error body is one with no `success` key.+ if resp.status_code >= 400:+ raise HTTPException(+ status_code=status.HTTP_502_BAD_GATEWAY,+ detail=(+ f"connectors service returned an unexpected HTTP {resp.status_code} while testing "+ f"this connector; your credentials were not sent upstream"+ ),)body: dict[str, Any]@@ -584,6 +655,15 @@ async def _proxy_test_connection() from excif not isinstance(body, dict):body = {"success": False, "error": "malformed response"}+ if "success" not in body:+ # A 2xx with no verdict is not a pass. The wizard branches on+ # `result.success`, so an absent key reads as failure with no message —+ # exactly the dead end this function used to produce on a 401.+ body = {+ **body,+ "success": False,+ "error": body.get("detail") or "connectors service returned a response with no success verdict",+ }return body@@ -697,6 +777,7 @@ async def test_connection(request.connector_type,request.auth_config,request.connector_config,+ current_user.tenant_id,)@@ -736,7 +817,7 @@ async def create_connector(except CredentialVaultError as exc:raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,- detail=f"credential vault unavailable: {exc}",+ detail=f"credential vault unavailable: {exc}. {_VAULT_HINT}",) from excconnector = Connector(@@ -811,7 +892,7 @@ async def update_connector(except CredentialVaultError as exc:raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,- detail=f"credential vault unavailable: {exc}",+ detail=f"credential vault unavailable: {exc}. {_VAULT_HINT}",) from excupdates[field] = val@@ -983,6 +1064,7 @@ async def test_existing_connector(connector.connector_type,decrypted_auth,connector.connector_config or {},+ current_user.tenant_id,)health_status = "healthy" if verdict.get("success") else "unhealthy"diff --git a/services/api/app/api/v1/endpoints/threat_intel.py b/services/api/app/api/v1/endpoints/threat_intel.pyindex 62e822846..8f09fbc02 100644--- a/services/api/app/api/v1/endpoints/threat_intel.py+++ b/services/api/app/api/v1/endpoints/threat_intel.py@@ -19,21 +19,32 @@ See ``app.core.security.ROLE_PERMISSIONS`` for the authoritative map.from __future__ import annotations+import logging+import osimport uuidfrom datetime import datetimefrom typing import Annotated, Any+import httpxfrom fastapi import APIRouter, Depends, HTTPException, Query, statusfrom pydantic import BaseModel, ConfigDict, Fieldfrom sqlalchemy import selectfrom sqlalchemy.ext.asyncio import AsyncSessionfrom app.api.v1.deps import AuthUser, require_permission+from app.core.config import settingsfrom app.db.database import get_dbfrom app.models.threat_intel import ThreatActor, ThreatIntelFeed, ThreatIntelIOC+logger = logging.getLogger(__name__)+router = APIRouter(prefix="/threat-intel", tags=["threat-intel"])+#: Header a trusted service uses to declare which tenant it is acting for.+#: Must match ``TENANT_HEADER`` in+#: services/threatintel/app/security/tenant_scope.py.+_TENANT_HEADER = "X-AiSOC-Tenant-ID"+# ---------------------------------------------------------------------------# Pydantic schemas@@ -111,6 +122,102 @@ class FeedOut(FeedCreate):model_config = ConfigDict(from_attributes=True)+# ---------------------------------------------------------------------------+# Collected feed indicators (proxied from services/threatintel)+# ---------------------------------------------------------------------------+++class FeedIndicatorsResponse(BaseModel):+ """What the console's `/threat-intel` page renders.++ ``source`` and ``degraded`` travel with the payload for the same reason+ they do on the connector catalog: an empty list because nothing has been+ collected yet and an empty list because the service is not deployed are+ different facts, and a reader cannot tell them apart from the rows alone.+ """++ indicators: list[dict[str, Any]]+ total: int+ source: str+ degraded: bool = False+ reason: str = ""++[email protected]("/indicators", response_model=FeedIndicatorsResponse)+async def list_feed_indicators(+ current_user: Annotated[AuthUser, Depends(require_permission("threat_intel:read"))],+ # `alias` preserves the `?type=` the console sends while keeping the+ # builtin `type()` callable in this scope — it is used in the except arm.+ ioc_type: str | None = Query(None, alias="type", description="filter to one indicator type"),+ tag: str | None = Query(None),+ q: str | None = Query(None, description="substring match on value or description"),+) -> FeedIndicatorsResponse:+ """Indicators the feed scheduler has collected, from `services/threatintel`.++ The console has called this path since the IOC inbox shipped and nothing+ served it — the page 404'd, and for a while rendered five invented IOCs+ instead. `services/threatintel` and its Qdrant store are in CORE now, and+ the CISA Known Exploited Vulnerabilities catalog is public and keyless, so+ a default install has real indicators to show within a poll interval.++ Tenant scope is enforced upstream: this proxy asserts the caller's tenant+ in the same header the connectors proxy uses, and the threatintel route+ filters on it plus the `shared` sentinel that public feed intel is written+ under.+ """+ base = (getattr(settings, "THREATINTEL_SERVICE_URL", "") or "").strip().rstrip("/")+ if not base:+ return FeedIndicatorsResponse(+ indicators=[],+ total=0,+ source="unconfigured",+ degraded=True,+ reason="THREATINTEL_SERVICE_URL is not set on this deployment",+ )++ params = {k: v for k, v in (("type", ioc_type), ("tag", tag), ("q", q)) if v}+ headers: dict[str, str] = {_TENANT_HEADER: str(current_user.tenant_id)}+ token = (os.getenv("AISOC_THREATINTEL_SERVICE_TOKEN") or os.getenv("AISOC_SERVICE_TOKEN") or "").strip()+ if token:+ headers["Authorization"] = f"Bearer {token}"++ timeout = httpx.Timeout(settings.THREATINTEL_SERVICE_TIMEOUT_SECONDS, connect=5.0)+ try:+ async with httpx.AsyncClient(timeout=timeout) as client:+ resp = await client.get(f"{base}/api/v1/threat-intel/indicators", params=params, headers=headers)+ resp.raise_for_status()+ body = resp.json()+ except httpx.HTTPError as exc:+ upstream_status = getattr(getattr(exc, "response", None), "status_code", None)+ logger.warning(+ "threatintel.indicators.unreachable error_type=%s status=%s",+ type(exc).__name__,+ upstream_status,+ )+ return FeedIndicatorsResponse(+ indicators=[],+ total=0,+ source="unavailable",+ degraded=True,+ reason=(+ f"the threat-intel service did not answer ({type(exc).__name__}"+ + (f", HTTP {upstream_status}" if upstream_status else "")+ + "); no indicators can be listed"+ ),+ )++ if not isinstance(body, dict):+ return FeedIndicatorsResponse(indicators=[], total=0, source="unavailable", degraded=True, reason="malformed upstream response")++ raw = body.get("indicators")+ indicators = [i for i in raw if isinstance(i, dict)] if isinstance(raw, list) else []+ return FeedIndicatorsResponse(+ indicators=indicators,+ total=int(body.get("total") or len(indicators)),+ source=str(body.get("source") or "threatintel"),+ )++# ---------------------------------------------------------------------------# IOC endpoints# ---------------------------------------------------------------------------diff --git a/services/api/app/core/config.py b/services/api/app/core/config.pyindex 7949f0561..c4e9d2da8 100644--- a/services/api/app/core/config.py+++ b/services/api/app/core/config.py@@ -197,6 +197,15 @@ class Settings(BaseSettings):CONNECTORS_SERVICE_URL: str = "http://connectors:8003"CONNECTORS_SERVICE_TIMEOUT_SECONDS: float = 15.0+ # Internal URL of ``services/threatintel``, which polls the public feeds+ # (CISA KEV needs no key) and holds what they collected. The console's+ # /threat-intel page reads through ``GET /threat-intel/indicators`` here;+ # empty disables the proxy and that route reports it rather than rendering+ # an empty list, because an empty list and an absent service look the same+ # to a reader and call for different things.+ THREATINTEL_SERVICE_URL: str = "http://threatintel:8005"+ THREATINTEL_SERVICE_TIMEOUT_SECONDS: float = 10.0+# Public ingest base URL — surfaced in the wizard's "Reveal push URL"# response so operators get a copy-pasteable curl example. Empty# falls back to a relative path; production deployments shoulddiff --git a/services/api/tests/test_connectors_endpoint.py b/services/api/tests/test_connectors_endpoint.pyindex 94c9364db..efa8ec805 100644--- a/services/api/tests/test_connectors_endpoint.py+++ b/services/api/tests/test_connectors_endpoint.py@@ -301,7 +301,7 @@ async def test_proxy_test_connection_passthrough_success() -> None:upstream_body = {"success": True, "connector": "splunk", "version": "9.0"}resp = _mock_response(200, upstream_body)with _patched_client(post_resp=resp):- result = await _proxy_test_connection("splunk", {"token": "x"}, {"host": "y"})+ result = await _proxy_test_connection("splunk", {"token": "x"}, {"host": "y"}, _TENANT)assert result == upstream_body@@ -311,7 +311,7 @@ async def test_proxy_test_connection_passthrough_connector_failure() -> None:upstream_body = {"success": False, "connector": "splunk", "error": "401 Unauthorized"}resp = _mock_response(200, upstream_body)with _patched_client(post_resp=resp):- result = await _proxy_test_connection("splunk", {"token": "bad"}, {})+ result = await _proxy_test_connection("splunk", {"token": "bad"}, {}, _TENANT)# Critically, this is NOT raised — the wizard wants to render the# connector's own error message.assert result["success"] is False@@ -324,7 +324,7 @@ async def test_proxy_test_connection_422_passes_detail() -> None:resp = _mock_response(422, {"detail": "missing field 'tenant_id'"})with _patched_client(post_resp=resp):with pytest.raises(HTTPException) as exc_info:- await _proxy_test_connection("azure_entra", {}, {})+ await _proxy_test_connection("azure_entra", {}, {}, _TENANT)assert exc_info.value.status_code == 422assert "tenant_id" in exc_info.value.detail@@ -340,7 +340,7 @@ async def test_proxy_test_connection_404_becomes_503() -> None:resp = _mock_response(404, {"detail": "not found"})with _patched_client(post_resp=resp):with pytest.raises(HTTPException) as exc_info:- await _proxy_test_connection("splunk", {}, {})+ await _proxy_test_connection("splunk", {}, {}, _TENANT)assert exc_info.value.status_code == 503@@ -350,7 +350,7 @@ async def test_proxy_test_connection_5xx_becomes_502() -> None:resp = _mock_response(500, {"detail": "boom"})with _patched_client(post_resp=resp):with pytest.raises(HTTPException) as exc_info:- await _proxy_test_connection("splunk", {}, {})+ await _proxy_test_connection("splunk", {}, {}, _TENANT)assert exc_info.value.status_code == 502@@ -367,7 +367,7 @@ async def test_proxy_test_connection_unreachable_becomes_503() -> None:return_value=client_cm,):with pytest.raises(HTTPException) as exc_info:- await _proxy_test_connection("splunk", {}, {})+ await _proxy_test_connection("splunk", {}, {}, _TENANT)assert exc_info.value.status_code == 503@@ -376,6 +376,107 @@ async def test_proxy_test_connection_non_dict_body_normalised() -> None:"""Defend against connectors microservice returning a non-dict 200."""resp = _mock_response(200, "ok")with _patched_client(post_resp=resp):- result = await _proxy_test_connection("splunk", {}, {})+ result = await _proxy_test_connection("splunk", {}, {}, _TENANT)assert isinstance(result, dict)assert result["success"] is False+++# ---------------------------------------------- the proxy's own credential+#+# The catalog call was given an Authorization header and a tenant assertion;+# the test-connection call beside it was not. Both talk to the same+# `require_console_or_service_auth`-guarded router, so "Test connection" was+# answered 401 on every invocation in every deployment that had a service+# token configured — and 401 was the one status the ladder did not branch on,+# so the function returned an error body with no `success` key and the wizard+# rendered the bare string "Connection test failed".+++def _captured_post(post_resp: MagicMock):+ """Patch the client and hand back the mock whose `.post` was called."""+ client_instance = MagicMock()+ client_instance.post = AsyncMock(return_value=post_resp)+ client_cm = MagicMock()+ client_cm.__aenter__ = AsyncMock(return_value=client_instance)+ client_cm.__aexit__ = AsyncMock(return_value=None)+ return (+ patch("app.api.v1.endpoints.connectors.httpx.AsyncClient", return_value=client_cm),+ client_instance,+ )+++async def test_proxy_test_connection_sends_the_service_credential(monkeypatch: pytest.MonkeyPatch) -> None:+ """The headers the catalog call sends, sent by this call too."""+ monkeypatch.setenv("AISOC_SERVICE_TOKEN", "a-real-token")+ monkeypatch.delenv("AISOC_CONNECTORS_SERVICE_TOKEN", raising=False)+ patcher, client = _captured_post(_mock_response(200, {"success": True}))+ with patcher:+ await _proxy_test_connection("splunk", {"token": "x"}, {}, _TENANT)++ headers = client.post.await_args.kwargs["headers"]+ assert headers["Authorization"] == "Bearer a-real-token"+ # A service token identifies a service, not a tenant; the connectors+ # service answers 403 to one that does not declare the tenant it acts for.+ assert headers["X-AiSOC-Tenant-ID"] == str(_TENANT)++[email protected]("upstream", [401, 403])+async def test_service_auth_rejection_names_the_status_and_the_service(upstream: int) -> None:+ """An operator must be able to tell this from a bad vendor credential.++ The failure is in AiSOC's own configuration and the remedy is+ AISOC_SERVICE_TOKEN — nothing the customer's Splunk token can fix.+ """+ resp = _mock_response(upstream, {"detail": "Not authenticated"})+ with _patched_client(post_resp=resp):+ with pytest.raises(HTTPException) as exc_info:+ await _proxy_test_connection("splunk", {"token": "x"}, {}, _TENANT)++ assert exc_info.value.status_code == 502+ detail = exc_info.value.detail+ assert str(upstream) in detail, "the operator cannot act on a status they are not told"+ assert "connectors service" in detail+ assert "AISOC_SERVICE_TOKEN" in detail+ assert "never sent upstream" in detail, "the wizard promises the upstream API is reached; say when it was not"+++async def test_an_unhandled_4xx_does_not_fall_through_as_a_verdict() -> None:+ """The ladder ended at >=500, so 400/405/429 returned the error body itself."""+ resp = _mock_response(429, {"detail": "slow down"})+ with _patched_client(post_resp=resp):+ with pytest.raises(HTTPException) as exc_info:+ await _proxy_test_connection("splunk", {}, {}, _TENANT)+ assert exc_info.value.status_code == 502+ assert "429" in exc_info.value.detail+++async def test_a_200_with_no_verdict_is_not_a_pass() -> None:+ """`if (result.success)` in the wizard reads an absent key as failure.++ Returning the body unchanged left it with neither `success` nor `error`,+ which is how the modal came to show a message with no cause in it.+ """+ resp = _mock_response(200, {"connector": "splunk"})+ with _patched_client(post_resp=resp):+ result = await _proxy_test_connection("splunk", {}, {}, _TENANT)+ assert result["success"] is False+ assert result["error"], "a failure the wizard renders must carry a reason"+++async def test_connectors_service_absent_says_how_to_start_it() -> None:+ """CORE does not ship the connectors service; the 503 must say so."""+ client_instance = MagicMock()+ client_instance.post = AsyncMock(side_effect=httpx.ConnectError("conn refused"))+ client_cm = MagicMock()+ client_cm.__aenter__ = AsyncMock(return_value=client_instance)+ client_cm.__aexit__ = AsyncMock(return_value=None)+ with patch("app.api.v1.endpoints.connectors.httpx.AsyncClient", return_value=client_cm):+ with pytest.raises(HTTPException) as exc_info:+ await _proxy_test_connection("splunk", {}, {}, _TENANT)+ assert exc_info.value.status_code == 503+ assert "up-full" in exc_info.value.detail or "docker compose up -d connectors" in exc_info.value.detaildiff --git a/services/ingest/internal/handler/handler.go b/services/ingest/internal/handler/handler.goindex de9b99c94..33fc971aa 100644--- a/services/ingest/internal/handler/handler.go+++ b/services/ingest/internal/handler/handler.go@@ -7,6 +7,7 @@ import ("fmt""net/http""sort"+ "strconv""time""github.com/beenuar/aisoc/services/ingest/internal/config"@@ -133,8 +134,11 @@ func (h *Handler) IngestEvents(w http.ResponseWriter, r *http.Request) {}if len(req.Events) > h.cfg.MaxBatchSize {+ // strconv, not string(rune(…)): the conversion produces the *code point*+ // with that value, so a limit of 1000 rendered as "Ϩ" and the caller was+ // told their batch exceeded a maximum of one Greek letter.writeError(w, http.StatusRequestEntityTooLarge,- "batch size exceeds maximum of "+string(rune(h.cfg.MaxBatchSize)))+ "batch size exceeds maximum of "+strconv.Itoa(h.cfg.MaxBatchSize))return}diff --git a/services/threatintel/app/api/indicators.py b/services/threatintel/app/api/indicators.pynew file mode 100644index 000000000..11067cef2--- /dev/null+++ b/services/threatintel/app/api/indicators.py@@ -0,0 +1,173 @@+"""Read the indicators this service has collected.++Why this route exists+---------------------+The console ships a `/threat-intel` page that calls+``GET /api/v1/threat-intel/indicators``. Nothing implemented it. The page was+recently caught rendering five invented IOCs and a hard-coded "3 Added Today"+counter on tenants that had never ingested one; removing the fabrication left+the page correct and empty, because the real feed was in the `full` profile and+the endpoint it wanted did not exist in any profile. The missing feed and the+fabrication were one hole.++``services/threatintel`` and Qdrant now ship in CORE, and the CISA Known+Exploited Vulnerabilities catalog is authoritative, public and needs no API+key — so a default ``make up`` collects real indicators. This route serves+them, and the API proxies it.++Why Qdrant and not OpenSearch+-----------------------------+``ThreatIntelPipeline`` writes to three sinks. OpenSearch is the full-text one+and stays in the `full` profile; Qdrant is the one CORE has, it already carries+the complete IOC payload on every point, and ``tenant_scope_filter`` already+expresses the read scope this route needs — public feed intel under the+``shared`` sentinel, private intel under its owning tenant. Serving from the+store that is actually present is what makes the page work in CORE.++Scrolling a vector store is an unusual list path, and it is bounded on purpose:+``limit`` is capped, and this is a browse surface, not an export.+"""++from __future__ import annotations++from typing import Annotated, Any++import structlog+from fastapi import APIRouter, Depends, HTTPException, Query, Request+from pydantic import BaseModel, Field+from qdrant_client import AsyncQdrantClient++from app.security.tenant_scope import TenantPrincipal, require_console_or_service_auth+from app.storage.qdrant import IOC_COLLECTION, tenant_scope_filter++logger = structlog.get_logger(__name__)++router = APIRouter(+ prefix="/api/v1/threat-intel",+ tags=["threat-intel"],+ dependencies=[Depends(require_console_or_service_auth)],+)++#: Ceiling on one page. The console renders a browse list; anything wanting+#: the whole catalog should read the feed at source.+MAX_LIMIT = 500+++class Indicator(BaseModel):+ """One indicator, in the shape the console's IOC inbox renders.++ Deliberately permissive about the source payload: feeds disagree about+ which fields they carry, and a missing `description` is not a reason to+ drop a real IOC from the list.+ """++ type: str+ value: str+ description: str | None = None+ confidence: int = 50+ malicious: bool = True+ sources: list[str] = Field(default_factory=list)+ tags: list[str] = Field(default_factory=list)+ first_seen: str | None = None+ last_seen: str | None = None+ tlp: str | None = None+++class IndicatorListResponse(BaseModel):+ indicators: list[Indicator]+ total: int+ #: Which store answered. The console has no other way to tell "no+ #: indicators collected yet" from "the store is not there", and the two+ #: call for different things from the reader.+ source: str+++def _to_indicator(payload: dict[str, Any]) -> Indicator:+ raw_tags = payload.get("tags")+ tags = [str(t) for t in raw_tags] if isinstance(raw_tags, list) else []+ source = payload.get("source")+ return Indicator(+ type=str(payload.get("type") or "unknown"),+ value=str(payload.get("value") or ""),+ description=payload.get("description") or payload.get("vulnerability_name") or None,+ confidence=int(payload.get("confidence") or 50),+ # A feed entry is an indicator of compromise; CISA KEV in particular+ # lists vulnerabilities under active exploitation. `false` here would+ # be a claim the feed does not make.+ malicious=bool(payload.get("malicious", True)),+ sources=[str(source)] if source else [],+ tags=tags,+ first_seen=payload.get("first_seen") or payload.get("date_added"),+ last_seen=payload.get("last_seen") or payload.get("date_added"),+ tlp=payload.get("tlp"),+ )+++def _qdrant(request: Request) -> AsyncQdrantClient:+ client = getattr(request.app.state, "qdrant_client", None)+ if client is None:+ raise HTTPException(status_code=503, detail="vector store is not configured on this deployment")+ return client++[email protected]("/indicators", response_model=IndicatorListResponse)+async def list_indicators(+ request: Request,+ principal: Annotated[TenantPrincipal, Depends(require_console_or_service_auth)],+ # `alias` keeps the query string `?type=` the console already sends while+ # the parameter itself is named something that does not shadow `type()` —+ # which this function calls, in the except arm below.+ ioc_type: str | None = Query(default=None, alias="type", description="filter to one indicator type"),+ tag: str | None = Query(default=None),+ q: str | None = Query(default=None, description="substring match on value or description"),+ limit: int = Query(default=100, ge=1, le=MAX_LIMIT),+) -> IndicatorListResponse:+ """Indicators collected by the feed scheduler, scoped to the caller.++ ``tenant_scope_filter`` is what enforces the scope: a caller sees their own+ tenant's points plus the ``shared`` sentinel that public feed intel is+ written under. A principal with no tenants is an empty scope, never a+ global one.+ """+ client = _qdrant(request)++ tenant_ids = principal.ordered_ids()+ scope = tenant_scope_filter(str(tenant_ids[0]) if tenant_ids else None)++ try:+ points, _next_page = await client.scroll(+ collection_name=IOC_COLLECTION,+ scroll_filter=scope,+ # Over-read so post-filtering (`q`, `tag`, `type`) still fills a+ # page. Bounded at 4x so a wide filter cannot walk the collection.+ limit=min(limit * 4, MAX_LIMIT * 4),+ with_payload=True,+ with_vectors=False,+ )+ except Exception as exc:+ # An absent collection is "nothing collected yet", not an error; the+ # scheduler creates it on the first successful poll. Anything else is+ # reported rather than rendered as an empty list, because an empty list+ # is indistinguishable from a working store with no data.+ message = str(exc)+ if "not found" in message.lower() or "doesn't exist" in message.lower():+ logger.info("threatintel.indicators.collection_absent", collection=IOC_COLLECTION)+ return IndicatorListResponse(indicators=[], total=0, source="qdrant")+ logger.warning("threatintel.indicators.read_failed", error=type(exc).__name__)+ raise HTTPException(+ status_code=503,+ detail="the threat-intel vector store did not answer; no indicators can be listed",+ ) from exc++ indicators = [_to_indicator(p.payload or {}) for p in points if p.payload]++ if ioc_type:+ indicators = [i for i in indicators if i.type == ioc_type]+ if tag:+ indicators = [i for i in indicators if tag in i.tags]+ if q:+ needle = q.lower()+ indicators = [i for i in indicators if needle in i.value.lower() or needle in (i.description or "").lower()]++ total = len(indicators)+ return IndicatorListResponse(indicators=indicators[:limit], total=total, source="qdrant")diff --git a/services/threatintel/app/clients/cisa_kev.py b/services/threatintel/app/clients/cisa_kev.pyindex 2ae4c3ba9..179f55588 100644--- a/services/threatintel/app/clients/cisa_kev.py+++ b/services/threatintel/app/clients/cisa_kev.py@@ -13,8 +13,27 @@ import structloglogger = structlog.get_logger(__name__)+#: The canonical catalog, published by CISA on cisa.gov. Tried first, always._KEV_URL = "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json"+#: CISA's own GitHub organisation publishes the identical file, and it is the+#: fallback rather than a third-party copy for exactly that reason — same+#: publisher, same schema (`catalogVersion`, `count`, `vulnerabilities[]`),+#: verified byte-for-byte in shape against the canonical URL.+#:+#: This exists because cisa.gov sits behind an edge that returns+#: `403 Access Denied` to whole networks regardless of user agent — observed+#: from a plain laptop, with curl, httpx and a browser UA all refused. Without+#: a fallback, every deployment on such a network gets a healthy threatintel+#: container, a registered feed, and permanently zero indicators: a silent+#: failure that looks exactly like "nothing has been published yet".+_KEV_FALLBACK_URL = "https://raw.githubusercontent.com/cisagov/kev-data/develop/known_exploited_vulnerabilities.json"++#: cisa.gov's edge also refuses an absent or obviously-robotic user agent on+#: some paths. Identifying the client is both politer and more likely to work+#: than defaulting to `python-httpx/x.y`.+_USER_AGENT = "AiSOC/1.0 (+https://github.com/beenuar/AiSOC)"+class CisaKevClient:"""@@ -22,22 +41,39 @@ class CisaKevClient:each entry into a normalized IOC-style dict for storage."""- def __init__(self, url: str = _KEV_URL) -> None:+ def __init__(self, url: str = _KEV_URL, fallback_url: str | None = _KEV_FALLBACK_URL) -> None:self._url = url+ self._fallback_url = fallback_urlasync def fetch(self) -> list[dict[str, Any]]:- """Download and return the full KEV catalog as a list of entries."""- async with httpx.AsyncClient(timeout=60.0) as client:- try:- resp = await client.get(self._url, follow_redirects=True)- resp.raise_for_status()- body = resp.json()- entries = body.get("vulnerabilities", [])- logger.info("CISA KEV fetched", count=len(entries))+ """Download and return the full KEV catalog as a list of entries.++ Returns ``[]`` only when *every* source failed, and logs at ``error``+ when it does — a caller cannot distinguish an empty catalog from an+ unreachable one by the return value, so the log line has to carry it.+ """+ sources = [("cisa.gov", self._url)]+ if self._fallback_url:+ sources.append(("cisagov/kev-data", self._fallback_url))++ headers = {"User-Agent": _USER_AGENT, "Accept": "application/json"}+ async with httpx.AsyncClient(timeout=60.0, headers=headers) as client:+ for label, url in sources:+ try:+ resp = await client.get(url, follow_redirects=True)+ resp.raise_for_status()+ entries = resp.json().get("vulnerabilities", [])+ except Exception as exc:+ logger.warning("CISA KEV source unavailable", source=label, error=str(exc))+ continue+ logger.info("CISA KEV fetched", source=label, count=len(entries))return entries- except Exception as exc:- logger.error("CISA KEV fetch failed", error=str(exc))- return []++ logger.error(+ "CISA KEV fetch failed from every source; the threat-intel page will stay empty",+ sources=[label for label, _ in sources],+ )+ return []def to_ioc(self, entry: dict[str, Any]) -> dict[str, Any]:"""Convert a KEV entry to a normalized IOC dict."""diff --git a/services/threatintel/app/feeds/pipeline.py b/services/threatintel/app/feeds/pipeline.pyindex 0639a2eaf..128aa350b 100644--- a/services/threatintel/app/feeds/pipeline.py+++ b/services/threatintel/app/feeds/pipeline.py@@ -87,7 +87,18 @@ class ThreatIntelPipeline:return {"total": len(iocs), "new": 0, "duplicate": duplicates}# --- Write to storage sinks ---- indexed = await self._os.bulk_index_iocs(new_iocs)+ #+ # Every sink is best-effort, OpenSearch included. It was the one+ # unguarded call here, which made a store that ships in the `full`+ # profile a hard requirement of a service that now runs in CORE: an+ # exception from it aborted the batch before Qdrant — the CORE store,+ # and the one the console reads through — was written at all. A feed+ # that can reach one of its sinks should write to that one.+ indexed = 0+ try:+ indexed = await self._os.bulk_index_iocs(new_iocs)+ except Exception as exc:+ logger.warning("OpenSearch IOC index failed", error=str(exc))try:await self._qdrant.upsert_iocs(new_iocs)@@ -119,7 +130,11 @@ class ThreatIntelPipeline:if not actors:return {"total": 0, "new": 0}- indexed = await self._os.bulk_index_actors(actors)+ indexed = 0+ try:+ indexed = await self._os.bulk_index_actors(actors)+ except Exception as exc:+ logger.warning("OpenSearch actor index failed", error=str(exc))try:await self._qdrant.upsert_actors(actors)diff --git a/services/threatintel/app/feeds/scheduler.py b/services/threatintel/app/feeds/scheduler.pyindex cd94dccd7..50d3af0e8 100644--- a/services/threatintel/app/feeds/scheduler.py+++ b/services/threatintel/app/feeds/scheduler.py@@ -9,6 +9,8 @@ AiSOC — open-source AI Security Operations Center (MIT License)from __future__ import annotations+import random+from datetime import UTC, datetime, timedeltafrom typing import TYPE_CHECKINGimport structlog@@ -20,6 +22,15 @@ if TYPE_CHECKING:logger = structlog.get_logger(__name__)+#: Window the first poll of each feed is scattered across, in seconds.+#:+#: Not zero, because several feeds registering at once would otherwise fire+#: simultaneously into the same Redis bloom filter and the same vector store on+#: every boot. Not large, because a new install has nothing on its threat-intel+#: page until the first poll returns and that page is one of the first a new+#: user opens.+FIRST_POLL_JITTER_SECONDS = 20+class FeedScheduler:"""@@ -40,7 +51,17 @@ class FeedScheduler:handler,interval_seconds: int,) -> None:- """Register a feed polling function."""+ """Register a feed polling function, polled now and then on interval.++ ``next_run_time`` is explicit because an ``IntervalTrigger`` on its own+ schedules its *first* run one full interval after the scheduler starts.+ CISA KEV polls daily, so a fresh ``make up`` would have shown an empty+ Threat Intelligence page for twenty-four hours — with every component+ running, healthy, and correct. That is indistinguishable from the+ feature not working, and it is the first page a new user looks at for+ evidence the product does anything on its own.+ """+ first_run = datetime.now(UTC) + timedelta(seconds=random.uniform(1, FIRST_POLL_JITTER_SECONDS)) # noqa: S311 - scheduling jitter, not a security decisionself._scheduler.add_job(func=handler,trigger=IntervalTrigger(seconds=interval_seconds),@@ -49,11 +70,13 @@ class FeedScheduler:replace_existing=True,max_instances=1,coalesce=True,+ next_run_time=first_run,)logger.info("Registered feed",feed=feed_name,interval_seconds=interval_seconds,+ first_poll_at=first_run.isoformat(),)def start(self) -> None:diff --git a/services/threatintel/app/main.py b/services/threatintel/app/main.pyindex b6aa6e8f7..6bea31faa 100644--- a/services/threatintel/app/main.py+++ b/services/threatintel/app/main.py@@ -28,6 +28,7 @@ from app._health import install_health_routesfrom app.actors.attribution import ThreatActorAttributionEnginefrom app.airgap import airgap_status, is_host_allowed_for_airgapfrom app.api.actor_attribution import router as actor_attribution_router+from app.api.indicators import router as indicators_routerfrom app.clients.cisa_kev import CisaKevClientfrom app.clients.misp import MispClientfrom app.clients.otx import OtxClient@@ -147,7 +148,19 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:qdrant_store = QdrantStore(qdrant_client)neo4j_store = Neo4jStore(neo4j_driver)- await os_store.initialize()+ # Both stores are best-effort. OpenSearch used to be the one unguarded+ # call in this lifespan, which made it a hard dependency of the whole+ # service: it ships in the `full` profile, this service now ships in CORE,+ # and an unreachable OpenSearch killed the container before a single feed+ # was registered. Qdrant is the CORE store and is what the console reads+ # through; OpenSearch adds full-text search on top when it is there.+ try:+ await os_store.initialize()+ except Exception as exc:+ logger.warning(+ "OpenSearch init failed — full-text IOC search is unavailable; feeds still write to Qdrant",+ error=str(exc),+ )try:await qdrant_store.initialize()except Exception as exc:@@ -245,6 +258,9 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:app.state.pipeline = pipelineapp.state.redis = redisapp.state.os_store = os_store+ # Read path for GET /api/v1/threat-intel/indicators. Qdrant is the sink+ # CORE has, so it is the one the console's IOC list is served from.+ app.state.qdrant_client = qdrant_client# Threat actor attribution engine — shares the os_store so the IOC# component of the score can match against collected threat intel.@@ -291,6 +307,10 @@ app.mount("/metrics", metrics_app)# Threat actor attribution router (v0)app.include_router(actor_attribution_router)+# What the feeds have collected. The API proxies this to serve the console's+# /threat-intel page, which had no backend at all in CORE.+app.include_router(indicators_router)+@app.get("/health")async def health() -> dict:diff --git a/services/threatintel/app/storage/qdrant.py b/services/threatintel/app/storage/qdrant.pyindex 17d5edf3b..d189abbb2 100644--- a/services/threatintel/app/storage/qdrant.py+++ b/services/threatintel/app/storage/qdrant.py@@ -24,6 +24,12 @@ logger = structlog.get_logger(__name__)_IOC_COLLECTION = "threatintel_iocs"_ACTOR_COLLECTION = "threatintel_actors"++#: Public aliases. The read route in ``app.api.indicators`` needs to name the+#: collection it scrolls, and importing a private name across modules is how+#: two spellings of the same string end up in the tree.+IOC_COLLECTION = _IOC_COLLECTION+ACTOR_COLLECTION = _ACTOR_COLLECTION_EMBEDDING_DIM = 1536 # text-embedding-3-small / ada-002# Sentinel tenant for globally-shared feed intel (STIX/TAXII/MISP), which isdiff --git a/services/threatintel/tests/test_feed_scheduler.py b/services/threatintel/tests/test_feed_scheduler.pynew file mode 100644index 000000000..5ece81579--- /dev/null+++ b/services/threatintel/tests/test_feed_scheduler.py@@ -0,0 +1,91 @@+"""A feed that polls daily must still poll on the day it is installed.++`FeedScheduler.register` used to hand APScheduler an `IntervalTrigger` and+nothing else. An interval trigger schedules its *first* run one whole interval+after the scheduler starts, and CISA KEV's interval is 86400 seconds — so a+fresh `make up` had a healthy threatintel container, a registered feed, a+created Qdrant collection, and an empty Threat Intelligence page for the next+twenty-four hours.++Nothing was broken in a way any probe could see. The container was up, the log+said "Registered feed", and the page was correct: there genuinely were no+indicators. That is the shape of defect this file exists to catch, and it is+the same one that made every connector's poll job register as PAUSED.+"""++from __future__ import annotations++from datetime import UTC, datetime, timedelta++import pytest+from app.feeds.scheduler import FIRST_POLL_JITTER_SECONDS, FeedScheduler++DAILY = 86_400+++async def _noop() -> None:+ """Stand-in for a feed handler; never invoked, the job is never run."""+++def scheduler() -> FeedScheduler:+ return FeedScheduler(pipeline=None) # type: ignore[arg-type] # register() never touches it+++def _job(scheduler: FeedScheduler, feed: str):+ return scheduler._scheduler.get_job(feed)+++def test_a_daily_feed_polls_within_the_first_minute(scheduler: FeedScheduler) -> None:+ before = datetime.now(UTC)+ scheduler.register(feed_name="cisa-kev", handler=_noop, interval_seconds=DAILY)++ job = _job(scheduler, "cisa-kev")+ assert job is not None, "register() did not add a job"+ assert job.next_run_time is not None, (+ "next_run_time is None, which APScheduler treats as a PAUSED job — the feed would never poll at all"+ )++ delay = (job.next_run_time - before).total_seconds()+ assert 0 <= delay <= FIRST_POLL_JITTER_SECONDS + 1, (+ f"first poll is {delay:.0f}s away. An IntervalTrigger alone would put it {DAILY}s away, "+ "which is a full day of an empty Threat Intelligence page on a working install."+ )+++def test_the_recurring_interval_is_still_the_one_asked_for(scheduler: FeedScheduler) -> None:+ """Firing early must not turn a daily feed into a busy loop."""+ scheduler.register(feed_name="cisa-kev", handler=_noop, interval_seconds=DAILY)+ trigger = _job(scheduler, "cisa-kev").trigger+ assert trigger.interval == timedelta(seconds=DAILY)+++def test_feeds_do_not_all_fire_at_the_same_instant(scheduler: FeedScheduler) -> None:+ """Jitter: several feeds registering at boot share one bloom filter and one+ vector store, and a synchronised stampede on every restart is avoidable."""+ for name in ("cisa-kev", "misp", "otx", "taxii:enterprise-attack"):+ scheduler.register(feed_name=name, handler=_noop, interval_seconds=DAILY)++ times = {_job(scheduler, n).next_run_time for n in ("cisa-kev", "misp", "otx", "taxii:enterprise-attack")}+ assert len(times) > 1, "every feed was scheduled for the identical instant — the jitter is not being applied"+++async def test_re_registering_a_running_feed_replaces_rather_than_duplicates() -> None:+ """`replace_existing` only takes effect against a started scheduler.++ Before `start()`, APScheduler holds additions in a pending list and does+ not look for an id collision there, so two `register` calls really do+ produce two jobs. That is harmless — the lifespan registers each feed once,+ before starting — but it is worth pinning which of the two states the flag+ protects, so nobody later concludes from a pre-start duplicate that the+ flag is not working.+ """+ scheduler = FeedScheduler(pipeline=None) # type: ignore[arg-type]+ scheduler.start()+ try:+ scheduler.register(feed_name="cisa-kev", handler=_noop, interval_seconds=DAILY)+ scheduler.register(feed_name="cisa-kev", handler=_noop, interval_seconds=DAILY)+ assert len(scheduler._scheduler.get_jobs()) == 1+ finally:+ scheduler.stop()diff --git a/tests/test_env_placeholder_gate.py b/tests/test_env_placeholder_gate.pynew file mode 100644index 000000000..df9ffbf73--- /dev/null+++ b/tests/test_env_placeholder_gate.py@@ -0,0 +1,225 @@+"""The placeholder detector and the file it inspects must not drift apart.++`scripts/doctor.sh` carried this check inline for a long time::++ grep -qE '^[A-Z_]*(SECRET|PASSWORD|KEY)=(change_me|changeme|)$' .env++and `.env.example` shipped+``AISOC_CREDENTIAL_KEY=replace-me-with-a-freshly-generated-fernet-key`` and+``SECRET_KEY=change-this-to-a-random-secret-key-at-least-32-chars``. The grep+matched neither. So the gate built to catch a shipped placeholder reported+clean on the one ``.env`` that broke the product — following the documented+``cp .env.example .env`` produced an API that answered HTTP 500 on every+connector save, while *skipping* the documented step produced a working one.++Two lists maintained by hand and compared by nothing. These tests compare them,+in both directions:++* every value ``.env.example`` ships is either flagged by the detector or named+ below as a deliberate working default, so a new placeholder cannot be added+ without teaching the detector about it;+* the historical placeholder strings still match, so the detector cannot be+ narrowed back to the shape that missed them;+* the three secrets ``scripts/ensure_env.py`` generates are shipped **empty**,+ because empty is the only value the credential vault treats as "not+ configured" and therefore the only safe thing to ship.+"""++from __future__ import annotations++import importlib.util+import sys+from pathlib import Path++import pytest++REPO = Path(__file__).resolve().parents[1]+ENV_EXAMPLE = REPO / ".env.example"+SCRIPTS = REPO / "scripts"++sys.path.insert(0, str(SCRIPTS))+++def _load(name: str):+ spec = importlib.util.spec_from_file_location(name, SCRIPTS / f"{name}.py")+ assert spec and spec.loader+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return module+++check_env_placeholders = _load("check_env_placeholders")+ensure_env = _load("ensure_env")+++#: Non-empty values `.env.example` ships that are **settings**, not prose for+#: the reader to replace. Each one is a value the stack actually runs with, so+#: flagging it would put a permanent warning on a correct deployment.+#:+#: Adding a row here is the deliberate act that says "this is a real default".+#: Anything not here and not caught by the detector fails the partition test+#: below, which is the whole point: a new placeholder has nowhere to hide.+DELIBERATE_DEFAULTS: dict[str, str] = {+ "AISOC_VERSION": "the image tag compose pulls; `latest` tracks main",+ "ENVIRONMENT": "selects the dev auth bypass; documented and read by compose",+ "LOG_LEVEL": "a log level, not a secret",+ "AISOC_CONSOLE_URL": "the address `make bootstrap` prints; localhost is right for a laptop",+ "AISOC_REALTIME_TICKET_TTL_SECONDS": "a duration",+ "CONNECTORS_SERVICE_URL": "the compose hostname of the connectors service",+ "CONNECTORS_SERVICE_TIMEOUT_SECONDS": "a duration",+ "POSTGRES_PASSWORD": "dev Postgres password; works out of the box, documented as changeable",+ "AISOC_APP_DB_PASSWORD": "dev password for the runtime DB role; same reasoning",+ "DATABASE_URL": "a DSN built from the two passwords above",+ "DATABASE_MIGRATION_URL": "the owner-role DSN used only by the migration runner",+ "REDIS_URL": "dev Redis DSN",+ "CLICKHOUSE_URL": "dev ClickHouse DSN",+ "KAFKA_BOOTSTRAP_SERVERS": "a host:port",+ "OPENSEARCH_URL": "a URL",+ "QDRANT_URL": "a URL",+ "OPENAI_MODEL": "a logical task alias the gateway defines",+ "LITELLM_MASTER_KEY": "the local gateway's own key; the gateway is in-network only",+ "LLM_GATEWAY_URL": "the compose hostname of the gateway",+ "TAXII_FEEDS": "a real public MITRE ATT&CK TAXII endpoint",+ "CISA_KEV_ENABLED": "a boolean",+ "ABUSE_CH_ENABLED": "a boolean",+ "CYBLE_VISION_BASE_URL": "a real vendor API base URL",+ "CYBLE_VISION_FEEDS": "a feed name list",+ "ANOMALI_BASE_URL": "a real vendor API base URL",+ "CROWDSTRIKE_BASE_URL": "a real vendor API base URL",+ "SPLUNK_PORT": "the Splunk management port",+ "SPLUNK_SCHEME": "a URL scheme",+ "SPLUNK_VERIFY_SSL": "a boolean, and the secure default",+ "AWS_REGION": "a region",+ "NEXT_PUBLIC_API_URL": "a localhost URL",+ "NEXT_PUBLIC_REALTIME_URL": "a localhost URL",+ "NEXT_PUBLIC_WS_URL": "a localhost URL",+}++#: The exact strings the old grep failed to match. Pinned as a regression test:+#: the detector may be broadened, never narrowed back past these.+HISTORICAL_PLACEHOLDERS = (+ "replace-me-with-a-freshly-generated-fernet-key",+ "change-this-to-a-random-secret-key-at-least-32-chars",+ "sk-your-openai-api-key-here",+ "change_me",+ "changeme",+)+++def _example_entries() -> list[tuple[int, str, str]]:+ entries = check_env_placeholders.parse_env(ENV_EXAMPLE.read_text(encoding="utf-8"))+ assert entries, ".env.example parsed to zero assignments — the parser or the file is wrong"+ return entries+++# ── The two lists, compared ──────────────────────────────────────────────────+++def test_every_shipped_value_is_either_a_placeholder_or_a_declared_default() -> None:+ """The partition that stops the two lists drifting.++ A contributor adding ``FOO_TOKEN=put-yours-here`` to `.env.example` has to+ make the detector match it or declare it a real default. There is no third+ option, which is what the old arrangement silently allowed.+ """+ undeclared: list[str] = []+ for lineno, key, value in _example_entries():+ if not value.strip():+ continue # empty is the correct value for every optional credential+ if check_env_placeholders.is_placeholder(value):+ continue+ if key in DELIBERATE_DEFAULTS:+ continue+ undeclared.append(f" .env.example:{lineno} {key}={value.strip()}")++ assert not undeclared, (+ "these .env.example values are neither recognised as placeholders nor declared as real defaults:\n"+ + "\n".join(undeclared)+ + "\n\nIf it is prose for the reader to replace, add a pattern to "+ "scripts/check_env_placeholders.py::PLACEHOLDER_PATTERNS.\n"+ "If it is a value the stack runs with, add it to DELIBERATE_DEFAULTS in this file with a reason."+ )++[email protected]("value", HISTORICAL_PLACEHOLDERS)+def test_the_detector_still_catches_every_placeholder_this_repo_has_shipped(value: str) -> None:+ assert check_env_placeholders.is_placeholder(value), (+ f"{value!r} shipped in .env.example and the doctor check missed it. "+ "Narrowing the detector past this point re-opens the original defect."+ )+++def test_a_declared_default_is_not_flagged() -> None:+ """The other direction: a real value must not produce a warning.++ A check that cries wolf on a correct deployment teaches operators to ignore+ it, which is how the original grep's `=$` arm — matching every legitimately+ empty optional credential — made the whole check noise.+ """+ for value in ("aisoc_dev_secret", "http://localhost:3000", "sk-aisoc-local", "true", "us-east-1"):+ assert not check_env_placeholders.is_placeholder(value), f"{value!r} is a working value and must not be reported as a placeholder"+++def test_empty_is_not_a_placeholder() -> None:+ assert not check_env_placeholders.is_placeholder("")+ assert not check_env_placeholders.is_placeholder(" ")+++# ── What the template is allowed to ship ─────────────────────────────────────++[email protected]("key", sorted(ensure_env.GENERATED))+def test_generated_secrets_ship_empty(key: str) -> None:+ """Empty, not prose — the distinction the credential vault actually makes.++ `get_vault()` falls back to an ephemeral development key only when the key+ is empty. A non-empty invalid key reaches `Fernet()` and raises, and the+ connector routes turn that into HTTP 500. So the template shipping+ ``AISOC_CREDENTIAL_KEY=replace-me-…`` meant the documented quick start+ produced a broken vault and skipping it produced a working one.+ """+ values = [value.strip() for _lineno, name, value in _example_entries() if name == key]+ assert values, f"{key} is generated by ensure_env.py but is not present in .env.example"+ assert values[-1] == "", (+ f".env.example ships {key}={values[-1]!r}. It must ship empty: the vault treats empty as "+ "'not configured' and takes its documented development path, while any other invalid value "+ "raises and surfaces as HTTP 500 at the connector wizard."+ )+++def test_ensure_env_fills_exactly_the_empty_and_placeholder_secrets(tmp_path: Path) -> None:+ """End-to-end over the real template: copy, generate, and check the result."""+ env = tmp_path / ".env"+ assert ensure_env.main(["--env", str(env), "--example", str(ENV_EXAMPLE)]) == 0++ values = {name: value.strip() for _lineno, name, value in check_env_placeholders.parse_env(env.read_text(encoding="utf-8"))}+ for key in ensure_env.GENERATED:+ assert values.get(key), f"ensure_env left {key} empty"+ assert not check_env_placeholders.is_placeholder(values[key])++ # The credential key has to satisfy Fernet, which is the whole point.+ from cryptography.fernet import Fernet++ Fernet(values["AISOC_CREDENTIAL_KEY"].encode("ascii"))++ # Idempotent: a second run must not rotate a key an operator is already using.+ before = env.read_text(encoding="utf-8")+ assert ensure_env.main(["--env", str(env), "--example", str(ENV_EXAMPLE)]) == 0+ assert env.read_text(encoding="utf-8") == before, "a second `make env` rotated a secret that was already set"+++def test_ensure_env_replaces_a_placeholder_it_finds(tmp_path: Path) -> None:+ """The upgrade path: an operator who already has the broken `.env`."""+ env = tmp_path / ".env"+ env.write_text("AISOC_CREDENTIAL_KEY=replace-me-with-a-freshly-generated-fernet-key\n", encoding="utf-8")+ assert ensure_env.main(["--env", str(env), "--example", str(ENV_EXAMPLE)]) == 0++ values = {name: value.strip() for _lineno, name, value in check_env_placeholders.parse_env(env.read_text(encoding="utf-8"))}+ from cryptography.fernet import Fernet++ Fernet(values["AISOC_CREDENTIAL_KEY"].encode("ascii"))+++def test_the_checker_refuses_a_file_that_is_not_there(tmp_path: Path) -> None:+ """A gate that reports OK over a tree it never opened is worse than none."""+ assert check_env_placeholders.main([str(tmp_path / "nope.env")]) == 1diff --git a/tests/test_full_profile_gate.py b/tests/test_full_profile_gate.pyindex 2b3499140..f4111bf0a 100644--- a/tests/test_full_profile_gate.py+++ b/tests/test_full_profile_gate.py@@ -126,14 +126,55 @@ def test_service_healthy_dependencies_have_a_healthcheck_to_wait_on() -> None:assert not missing, "; ".join(missing)-def test_threatintel_waits_for_a_ready_opensearch() -> None:- dep = (_services()["threatintel"].get("depends_on") or {}).get("opensearch")- assert isinstance(dep, dict) and dep.get("condition") == "service_healthy", (- "threatintel aborts its lifespan when OpenSearch does not answer, so "- "`service_started` leaves restart:unless-stopped to paper over a crash loop."+# ── A CORE service may not require a store that only the full profile starts ─+#+# This used to assert the opposite: that threatintel waited on a *healthy*+# OpenSearch, because its lifespan called `os_store.initialize()` with no+# try/except and starting early killed the container.+#+# threatintel is in CORE now — it is what puts real CISA KEV data in front of a+# new user — and OpenSearch is not. Keeping the dependency would have made+# `make up` start a 2 GB JVM for a full-text index CORE does not read. So the+# unguarded call was fixed instead, which is the stronger property: the service+# no longer aborts, rather than being sequenced so it does not have to. These+# two tests pin both halves — the manifest must not reintroduce the dependency,+# and the code must keep the call guarded, because either alone would let the+# crash loop back in.+def test_no_core_service_depends_on_a_full_profile_service() -> None:+ services = _services()+ core = {name for name, svc in services.items() if not svc.get("profiles")}+ offenders: list[str] = []+ for name in sorted(core):+ for dep in services[name].get("depends_on") or {}:+ if (services.get(dep) or {}).get("profiles"):+ offenders.append(f"{name} (CORE) depends on {dep}, which only starts under {services[dep]['profiles']}")+ assert not offenders, (+ "\n".join(offenders) + "\n\nCompose starts a profiled dependency anyway, so this does not fail loudly — "+ "it silently enlarges CORE by whatever that service costs.")+def test_threatintels_optional_stores_are_guarded_in_code() -> None:+ """The manifest check above is only safe because these calls cannot raise out."""+ lifespan = (REPO / "services/threatintel/app/main.py").read_text()+ pipeline = (REPO / "services/threatintel/app/feeds/pipeline.py").read_text()++ assert "await os_store.initialize()" in lifespan+ init_block = lifespan.split("await os_store.initialize()")[0]+ assert init_block.rstrip().endswith("try:"), (+ "services/threatintel/app/main.py calls os_store.initialize() outside a try — an unreachable "+ "OpenSearch then kills a CORE service on boot."+ )++ for call in ("self._os.bulk_index_iocs(new_iocs)", "self._os.bulk_index_actors(actors)"):+ assert call in pipeline, f"{call} moved; re-point this gate"+ before = pipeline.split(call)[0]+ assert "try:" in before.rsplit("\n\n", 1)[-1], (+ f"{call} is not inside a try block. It is the first sink written, so an exception there "+ "aborts the batch before Qdrant — the store CORE actually reads — is touched at all."+ )++# ── The API reads CLICKHOUSE_HOST, and nothing under services/api reads a URL ─## The api service was handed `CLICKHOUSE_URL` alone. No module under