Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

Merge remote-tracking branch 'origin/main' into fix/ingest-auth-and-blocking-scanners

Beenu Arora596ae126eacd
diff --cc CHANGELOG.md
index 585f90987,c3e3a3d0a..f7ccd6e2d
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,74 -7,8 +7,75 @@@ and this project adheres to [Semantic V
## [Unreleased]
+### BREAKING
+
+- **`POST /v1/ingest` and `POST /v1/ingest/batch` now require a credential,
+ and the tenant comes from the credential rather than from the
+ `X-Tenant-ID` header.** The endpoint the README tells users to push
+ telemetry to, and the one `make smoke` exercises, authenticated nothing:
+ it read `X-Tenant-ID`, believed it, and wrote events for whatever tenant
+ the caller named. Anyone who could reach the port could write alerts into
+ any tenant by typing that tenant's UUID. Compose binds the port to
+ `127.0.0.1`, which contains it locally, but ingesting real telemetry means
+ exposing it and nothing said so at the moment the operator did. The
+ comment at `services/ingest/internal/server/server.go` asserting that
+ "`/v1/ingest` is token-authenticated per request" had been false since it
+ was written; it is true now.
+
+ Rather than invent a second mechanism this extends the one the service
+ already had. `/v1/inbox/*` resolves a minted token from
+ `tenant_inbox_tokens` to a tenant, with optional HMAC-SHA256 over
+ `X-Signature`; `/v1/ingest` now resolves the same kind of token, pinned to
+ a new `connector-push` template exactly as `/v1/inbox/cef` is pinned to
+ `cef-syslog` — an inbox token is pasted into a third party's webhook
+ config, so minting one for PagerDuty must not also hand PagerDuty a
+ general ingest credential. A second shape covers AiSOC's own services:
+ `services/connectors` polls on behalf of many tenants and cannot hold a
+ per-tenant token, so it presents `AISOC_SERVICE_TOKEN` and declares the
+ tenant on `X-Tenant-ID`, which is then checked against the tenants table
+ before it becomes a scope.
+
+ `X-Tenant-ID` is still read and is never authority. It is **intersected**
+ with what the credential authorises, so naming an outside tenant narrows
+ to nothing and is refused rather than reaching out — the same
+ intersection-only rule `app/security/tenant_scope.py` already implements
+ for the Python services, ported to Go in
+ `services/ingest/internal/ingestauth`. There is no dev-mode bypass:
+ `AISOC_DEV_MODE` does not reach this path, and an ingest service that
+ cannot verify a credential answers 503 rather than accepting the write.
+
+ **Every deployment must act.** Mint a push token per external pusher with
+ `make ingest-token`, and set `AISOC_SERVICE_TOKEN` on both `ingest` and
+ `connectors` if you use pull connectors — without it, connector polling is
+ refused, loudly, at startup on one side and per request on the other. Full
+ procedure: `apps/docs/docs/operations/ingest-authentication.md`.
+
### Added
+- `make ingest-token` / `python -m app.scripts.mint_ingest_token` mints the
+ push credential for a fresh deployment, the role `bootstrap_admin` plays
+ for the console login. Idempotent — a second run returns the same token
+ rather than leaving a trail of live credentials nobody revokes.
+- `connector-push`, `ai-runtime`, `ai-finding` and `k8s-audit` are now
+ mintable inbox templates. The last three already shipped as YAML and were
+ named in the AI SDK's own setup instructions and in the Kubernetes
+ connector docs, but none was in `ALLOWED_TEMPLATE_IDS`, so the documented
+ setup for the AI-estate feature returned 400 and the templates were
+ unreachable.
+- `scripts/check_inbox_templates.py` compares shipped templates against
+ mintable ids **in both directions**, plus the console catalog, and carries
+ a `--self-test` that injects drift in each direction and requires the gate
+ to catch it. A one-directional check would have printed OK throughout the
+ defect above.
+- The golden pipeline asserts that an unauthenticated push is refused, not
+ only that an authenticated one succeeds — the happy path would go on
+ passing if authentication were removed.
+- `/v1/ingest` now bounds the request body (`INGEST_MAX_BODY_BYTES`, 10 MiB
+ default). `MAX_BATCH_SIZE` only counts events after the payload is
+ decoded, so nothing bounded the decode itself.
+
+
++
- **`scripts/generate_corpus_stats.py`** — generates
`apps/web/src/data/corpus-stats.json` + `corpusStats.ts` from the compiled
engine ruleset, the generated detection truth table, and the marketplace
@@@ -95,29 -29,19 +96,41 @@@
to published prose has no hand-copied link. A paragraph that dates or
negates the claim is exempt, so the retraction can quote the wording it
retracts.
+
+### Changed
+
+- **gitleaks and Trivy are blocking gates.** All five scanners in
+ `security.yml` carried `continue-on-error`, so a new finding never redded
+ a pull request. Gitleaks found 35 findings on its first real run; every
+ one was triaged and recorded in `.gitleaksignore` with a written reason —
+ detection rule-id maps, osquery and redaction test fixtures, placeholder
+ PEMs, a published AWS example key, a FIPS-197 test vector, and a
+ `YOUR_TOKEN` doc placeholder. Four of the six entries the allow-list
+ already held had rotted through line drift and were suppressing nothing.
+ Trivy reports zero fixable HIGH/CRITICAL and blocks at that.
+
+ Semgrep (92 findings, 35 at ERROR), checkov (91 failed checks) and tfsec
+ (32, of which 7 CRITICAL) stay in observe mode. The counts are now written
+ into the workflow so the remaining work is visible rather than implied.
+
+ The secret-scan job also fails when the scanner cannot install, instead of
+ reporting clean, and proves it is reading the tree before trusting a clean
+ result: with the allow-list moved aside the triaged findings must
+ reappear. "Clean" and "did not run" previously looked identical.
+
+
+ - **The prerequisites that were required but undocumented.** Beyond Docker and
+ its memory, a first run needs `python3` **on the host** (`make smoke` runs
+ the golden-pipeline script there, so without it you can start AiSOC but
+ cannot prove it works), `bash` (`make up` gates on `scripts/doctor.sh
+ --ports-only` before it calls compose), and roughly 18 GB of free Docker
+ disk — previously implied only by a troubleshooting row noting that Kafka
+ corrupts its log directory and *still passes its healthcheck* when the
+ daemon runs out of space. Node 22 and pnpm 8, which the installers require,
+ were also unlisted. All are now in the quick-start prerequisites table with
+ the command each one gates.
+
-### Changed
+
- **`readme_gates.py` covers the governance documents.** Its `FIGURE_DOCS`
list named one compliance page, which is why `ROADMAP.md` drifted with CI
green. `ROADMAP.md` and `RELEASES.md` are now on the list, the matrix