Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
Merge remote-tracking branch 'origin/main' into fix/ingest-auth-and-blocking-scanners
Beenu Arora596ae126eacd
diff --cc CHANGELOG.mdindex 585f90987,c3e3a3d0a..f7ccd6e2d--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -7,74 -7,8 +7,75 @@@ and this project adheres to [Semantic V## [Unreleased]+### BREAKING++- **`POST /v1/ingest` and `POST /v1/ingest/batch` now require a credential,+ and the tenant comes from the credential rather than from the+ `X-Tenant-ID` header.** The endpoint the README tells users to push+ telemetry to, and the one `make smoke` exercises, authenticated nothing:+ it read `X-Tenant-ID`, believed it, and wrote events for whatever tenant+ the caller named. Anyone who could reach the port could write alerts into+ any tenant by typing that tenant's UUID. Compose binds the port to+ `127.0.0.1`, which contains it locally, but ingesting real telemetry means+ exposing it and nothing said so at the moment the operator did. The+ comment at `services/ingest/internal/server/server.go` asserting that+ "`/v1/ingest` is token-authenticated per request" had been false since it+ was written; it is true now.++ Rather than invent a second mechanism this extends the one the service+ already had. `/v1/inbox/*` resolves a minted token from+ `tenant_inbox_tokens` to a tenant, with optional HMAC-SHA256 over+ `X-Signature`; `/v1/ingest` now resolves the same kind of token, pinned to+ a new `connector-push` template exactly as `/v1/inbox/cef` is pinned to+ `cef-syslog` — an inbox token is pasted into a third party's webhook+ config, so minting one for PagerDuty must not also hand PagerDuty a+ general ingest credential. A second shape covers AiSOC's own services:+ `services/connectors` polls on behalf of many tenants and cannot hold a+ per-tenant token, so it presents `AISOC_SERVICE_TOKEN` and declares the+ tenant on `X-Tenant-ID`, which is then checked against the tenants table+ before it becomes a scope.++ `X-Tenant-ID` is still read and is never authority. It is **intersected**+ with what the credential authorises, so naming an outside tenant narrows+ to nothing and is refused rather than reaching out — the same+ intersection-only rule `app/security/tenant_scope.py` already implements+ for the Python services, ported to Go in+ `services/ingest/internal/ingestauth`. There is no dev-mode bypass:+ `AISOC_DEV_MODE` does not reach this path, and an ingest service that+ cannot verify a credential answers 503 rather than accepting the write.++ **Every deployment must act.** Mint a push token per external pusher with+ `make ingest-token`, and set `AISOC_SERVICE_TOKEN` on both `ingest` and+ `connectors` if you use pull connectors — without it, connector polling is+ refused, loudly, at startup on one side and per request on the other. Full+ procedure: `apps/docs/docs/operations/ingest-authentication.md`.+### Added+- `make ingest-token` / `python -m app.scripts.mint_ingest_token` mints the+ push credential for a fresh deployment, the role `bootstrap_admin` plays+ for the console login. Idempotent — a second run returns the same token+ rather than leaving a trail of live credentials nobody revokes.+- `connector-push`, `ai-runtime`, `ai-finding` and `k8s-audit` are now+ mintable inbox templates. The last three already shipped as YAML and were+ named in the AI SDK's own setup instructions and in the Kubernetes+ connector docs, but none was in `ALLOWED_TEMPLATE_IDS`, so the documented+ setup for the AI-estate feature returned 400 and the templates were+ unreachable.+- `scripts/check_inbox_templates.py` compares shipped templates against+ mintable ids **in both directions**, plus the console catalog, and carries+ a `--self-test` that injects drift in each direction and requires the gate+ to catch it. A one-directional check would have printed OK throughout the+ defect above.+- The golden pipeline asserts that an unauthenticated push is refused, not+ only that an authenticated one succeeds — the happy path would go on+ passing if authentication were removed.+- `/v1/ingest` now bounds the request body (`INGEST_MAX_BODY_BYTES`, 10 MiB+ default). `MAX_BATCH_SIZE` only counts events after the payload is+ decoded, so nothing bounded the decode itself.++++- **`scripts/generate_corpus_stats.py`** — generates`apps/web/src/data/corpus-stats.json` + `corpusStats.ts` from the compiledengine ruleset, the generated detection truth table, and the marketplace@@@ -95,29 -29,19 +96,41 @@@to published prose has no hand-copied link. A paragraph that dates ornegates the claim is exempt, so the retraction can quote the wording itretracts.++### Changed++- **gitleaks and Trivy are blocking gates.** All five scanners in+ `security.yml` carried `continue-on-error`, so a new finding never redded+ a pull request. Gitleaks found 35 findings on its first real run; every+ one was triaged and recorded in `.gitleaksignore` with a written reason —+ detection rule-id maps, osquery and redaction test fixtures, placeholder+ PEMs, a published AWS example key, a FIPS-197 test vector, and a+ `YOUR_TOKEN` doc placeholder. Four of the six entries the allow-list+ already held had rotted through line drift and were suppressing nothing.+ Trivy reports zero fixable HIGH/CRITICAL and blocks at that.++ Semgrep (92 findings, 35 at ERROR), checkov (91 failed checks) and tfsec+ (32, of which 7 CRITICAL) stay in observe mode. The counts are now written+ into the workflow so the remaining work is visible rather than implied.++ The secret-scan job also fails when the scanner cannot install, instead of+ reporting clean, and proves it is reading the tree before trusting a clean+ result: with the allow-list moved aside the triaged findings must+ reappear. "Clean" and "did not run" previously looked identical.+++ - **The prerequisites that were required but undocumented.** Beyond Docker and+ its memory, a first run needs `python3` **on the host** (`make smoke` runs+ the golden-pipeline script there, so without it you can start AiSOC but+ cannot prove it works), `bash` (`make up` gates on `scripts/doctor.sh+ --ports-only` before it calls compose), and roughly 18 GB of free Docker+ disk — previously implied only by a troubleshooting row noting that Kafka+ corrupts its log directory and *still passes its healthcheck* when the+ daemon runs out of space. Node 22 and pnpm 8, which the installers require,+ were also unlisted. All are now in the quick-start prerequisites table with+ the command each one gates.+-### Changed+- **`readme_gates.py` covers the governance documents.** Its `FIGURE_DOCS`list named one compliance page, which is why `ROADMAP.md` drifted with CIgreen. `ROADMAP.md` and `RELEASES.md` are now on the list, the matrix