Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
ci(security): stop two scanner jobs absorbing a failed scan, require checkov (#1066)
Beenu Arorabb17ab84dcef
diff --git a/.github/required-checks.json b/.github/required-checks.jsonindex 177507161..e5244735a 100644--- a/.github/required-checks.json+++ b/.github/required-checks.json@@ -22,33 +22,46 @@"The Security workflow's scanner jobs were hard gates that blocked nothing:","Trivy fails outright on a fixable HIGH/CRITICAL and had been red on main for","nine consecutive runs with nothing to stop a merge, because only",- "security-audit was required. Three of the six are promoted here. All six",+ "security-audit was required. Four of the six are promoted here. All six","were checked for the defect that makes a required check worthless rather","than assumed safe: security.yml declares no paths filter and no job-level","if:, and across 40 pull_request runs every one of the six reported a real","conclusion and none reported skipped - GitHub counts a skipped check as","passing, so requiring one that skips on pull requests would recreate the","defect this repository has now fixed twice.",- "Two are deliberately NOT required, and the reason is the same shape one",- "level down. 'Semgrep (ratcheted)' and 'IaC scan (checkov, ratcheted)' each",- "carry a continue-on-error step whose outcome no later step reads, so the",- "job can report success over a step that failed and the Actions API cannot",- "see it. check_required_check_substance.py --static refuses both by name;",- "adding either here fails the gate rather than strengthening anything. The",- "remedy it prints is to give the step an id and add a blocking step reading",- "steps.<id>.outcome == 'failure', or to drop continue-on-error.",- "Semgrep is additionally non-deterministic on an unchanged tree - the",- "measurement is recorded in .security/allowlist.yml, where four pull",- "requests on one base within thirteen minutes split 102/40 and 101/39 over",- "the identical 387 rules and 2267 files - so requiring it would red pull",- "requests that cannot have changed a finding and teach re-running until",- "green, which is how a required check stops meaning anything."+ "'Semgrep (ratcheted)' and 'IaC scan (checkov, ratcheted)' each used to",+ "carry a continue-on-error step whose outcome no later step read, so the",+ "job could report success over a step that failed and the Actions API",+ "cannot see it. Both are now closed the way the gate's own message",+ "prescribes: the scan step carries an id and a blocking step reads",+ "steps.<id>.outcome == 'failure'. That step sits AHEAD of the ratchet,",+ "because a scan that dies part-way still writes a short report and the",+ "ratchet's shrink-only arm would answer it by asking for the ceiling to be",+ "lowered - baking the broken scan in. checkov additionally had its pip",+ "install moved out of the absorbed step into a blocking one, and its",+ "trailing '|| true' replaced by an explicit exit-code rule: 0 and 1 are a",+ "scan that ran, anything above is a scan that did not.",+ "'IaC scan (checkov, ratcheted)' is promoted here on the strength of that",+ "fix.",+ "'Semgrep (ratcheted)' is deliberately STILL NOT required, and that is a",+ "decision rather than an oversight. Do not add it here without reading",+ "this paragraph: closing its absorbed-failure hole and requiring it are",+ "separate questions, and only the first was answered yes. It is",+ "non-deterministic on an unchanged tree - four pull requests on one base,",+ "changing nothing but a lockfile, run within thirteen minutes, split",+ "102/40 and 101/39 over the identical 387 rules and 2267 files, with the",+ "obvious mechanisms eliminated and the whole measurement recorded in",+ ".security/allowlist.yml. Requiring it would red roughly one pull request",+ "in twenty for a reason no author can act on, and the only remedy is a",+ "re-run - which teaches people to re-run until green rather than to read",+ "the finding, and that is how a required check stops meaning anything."],"branch": "main","enforced_from": "2026-09-29T12:00:00Z","contexts": ["Backup → destroy → restore (Postgres via S3)","Claim-to-gate matrix (ratchet)",+ "IaC scan (checkov, ratcheted)","IaC scan (tfsec, ratcheted)","No competitor product named","No required check can skip its own assertions",diff --git a/.github/workflows/security.yml b/.github/workflows/security.ymlindex b006eda6e..76ac85023 100644--- a/.github/workflows/security.yml+++ b/.github/workflows/security.yml@@ -28,6 +28,26 @@ name: Security# They now live in the allow-list, where a run that disagrees with them# fails and prints the number to write.#+# A ratcheted scanner runs its scan under `continue-on-error`, because the+# scan's own exit status is not the verdict — the count is. That is only safe+# while something *reads* the outcome, and for Semgrep and checkov nothing+# did, so either job could report success over a scan that never finished.+# `continue-on-error` is defined as rewriting a failed step's conclusion to+# `success`, so the Actions API cannot show this and no amount of run history+# would have found it; `scripts/check_required_check_substance.py --static`+# refuses the shape instead. Both now give the scan step an `id` and put a+# blocking step that reads `steps.<id>.outcome == 'failure'` *ahead of* the+# ratchet — ahead, because a scan that dies part-way still leaves a short+# report and the ratchet's shrink-only arm would answer it with "lower the+# ceiling to N", which would bake the broken scan in.+#+# Four of the six jobs here are required checks on `main`: the claim-matrix+# ratchet, gitleaks, Trivy and checkov. `Semgrep (ratcheted)` is deliberately+# not, and the reason is measured rather than cautious — it disagrees with+# itself on an unchanged tree about one run in twenty. See+# `.security/allowlist.yml` and `.github/required-checks.json`, both of which+# record the decision so it is not quietly reversed.+## GitHub native push-protection remains the always-on hard secret gate.on:@@ -139,7 +159,13 @@ jobs:# `--error` is gone: it exits non-zero on any finding, which would make# the exit status carry the same information as the count and force# `continue-on-error` back onto the ratchet step below with it.+ #+ # `continue-on-error` here is deliberate and now *read*, by the step+ # below. Without `--error`, semgrep exits 0 whether or not it finds+ # anything, so a non-zero status means the scan itself failed — an+ # unfetchable rule config, a crash — and not a finding.- name: Scan+ id: scancontinue-on-error: truerun: |semgrep --json --output semgrep.json --disable-version-check \@@ -150,6 +176,22 @@ jobs:--config p/react \--exclude plans/ \services/ apps/ packages/ scripts/+ # Blocking, and this is the step that makes the one above safe.+ # `continue-on-error` rewrites a failed step's *conclusion* to success,+ # so without something reading its `outcome` the job reports green over+ # a scan that died — the shape+ # `scripts/check_required_check_substance.py` refuses by name, and the+ # reason this check could not be required.+ #+ # Ahead of the ratchet on purpose. A scan that failed part-way can still+ # leave a short `semgrep.json`, and the ratchet's shrink-only arm would+ # read that short count and print "lower it to N" — advice that would+ # bake a broken scan into the ceiling.+ - name: The scan itself did not fail+ if: steps.scan.outcome == 'failure'+ run: |+ echo "::error title=semgrep::semgrep exited non-zero. Without \`--error\` that is the scan failing, not a finding, so the count below would describe a partial run."+ exit 1# Blocking. No `continue-on-error`: a missing or truncated report means# the scan did not finish, and the gate says so rather than passing.- name: Ratchet the finding count@@ -225,14 +267,58 @@ jobs:- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0with:python-version: "3.11"+ # Its own step, and blocking. The install used to sit inside the scan+ # step under `continue-on-error`, so a failed `pip install` was+ # absorbed and the only symptom was the ratchet below complaining+ # about a report that was never written.+ - name: Install checkov+ run: pip install --quiet checkov==3.2.334+ # `continue-on-error` here is deliberate and now *read*, by the step+ # below. It replaces a trailing `|| true`, which discarded every exit+ # code checkov has — the one that means "I found things" and the one+ # that means "I could not run" alike.- name: checkov (terraform + helm)+ id: checkovcontinue-on-error: truerun: |- pip install --quiet checkov==3.2.334# `--output-file-path .` writes ./results_json.json; checkov treats# a directory argument that way rather than honouring a filename.+ status=0checkov -d infra/terraform -d infra/helm --quiet --compact \- -o json --output-file-path . || true+ -o json --output-file-path . || status=$?+ # Measured against the pinned 3.2.334: 0 = no failed checks, 1 =+ # failed checks, 2 = a usage error. 1 is the normal state here —+ # there are 219 of them — and deciding on it is the ratchet's job,+ # not this step's. Anything above 1 is checkov failing to run.+ #+ # `--soft-fail` would be the obvious way to write this and is the+ # wrong one: measured, it collapses 1 to 0 but leaves 2 alone,+ # which buys nothing, and it is silent about the case that matters+ # most — a `-d` naming a directory that does not exist exits 0+ # under every combination, logging an ERROR and scanning what is+ # left. That one is caught downstream instead, by the ratchet's+ # shrink-only arm, because a vanished directory shows up as a+ # count below the ceiling.+ if [ "$status" -gt 1 ]; then+ echo "::error title=iac-checkov::checkov exited $status — that is the scan failing to run, not a finding"+ exit "$status"+ fi+ # Blocking, and this is the step that makes the one above safe.+ # `continue-on-error` rewrites a failed step's *conclusion* to success,+ # so without something reading its `outcome` the job reports green over+ # a scan that died — the shape+ # `scripts/check_required_check_substance.py` refuses by name, and the+ # reason this check could not be required.+ #+ # Ahead of the ratchet on purpose. A scan that failed part-way can+ # still leave a short `results_json.json`, and the ratchet's+ # shrink-only arm would read that short count and print "lower it to+ # N" — advice that would bake a broken scan into the ceiling.+ - name: The scan itself did not fail+ if: steps.checkov.outcome == 'failure'+ run: |+ echo "::error title=iac-checkov::checkov did not complete, so the count below would describe a partial scan"+ exit 1- name: Ratchet the checkov finding countrun: python3 scripts/check_scanner_ratchet.py --tool checkov --report results_json.jsondiff --git a/.security/allowlist.yml b/.security/allowlist.ymlindex c89939588..944949d46 100644--- a/.security/allowlist.yml+++ b/.security/allowlist.yml@@ -90,6 +90,24 @@ ceilings:# one is to retain `semgrep.json` as a run artefact and diff two runs of# the same tree. Nothing here does that yet.#+ # Third consequence, and the one most likely to be undone by someone+ # tidying up: this is why `Semgrep (ratcheted)` is **not** a required+ # check on `main` while the other ratcheted scanners are. The job used to+ # carry a second, unrelated defect — a `continue-on-error` scan step whose+ # outcome nothing read, so the job reported success over a scan that had+ # died — and that one is now fixed, in the same change that promoted+ # `IaC scan (checkov, ratcheted)` to a required check after fixing the+ # identical hole there. Semgrep was deliberately left unrequired anyway.+ # The split above is the reason: at one run in twenty disagreeing with+ # itself, requiring this job would red a pull request whose author cannot+ # have caused it and cannot do anything about it except press re-run, and+ # a required check whose documented remedy is "run it again" teaches+ # people to re-run until green instead of reading what it said. Fixing+ # the absorbed failure and requiring the job are separate questions;+ # only the first was answered yes. Requiring it becomes reasonable once+ # the +/-1 is attributed and eliminated, not before.+ # The same reasoning is recorded in `.github/required-checks.json`.+ ## 102 -> 103 (error unchanged at 40). The added WARNING is a third# `dynamic-urllib-use-detected`, on scripts/check_required_check_substance.py,# which asks the Actions API which steps each required check actuallydiff --git a/CHANGELOG.md b/CHANGELOG.mdindex fafcfa4f9..a588e0e54 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Fixed++- **Two scanner jobs in `security.yml` could report success over a scan that failed, and that was the only thing keeping one of them off branch protection.** `Semgrep (ratcheted)` and `IaC scan (checkov, ratcheted)` each ran their scan under `continue-on-error: true` with no later step reading the outcome. `continue-on-error` is *defined* as rewriting a failed step's conclusion to `success`, so the Actions API shows a clean job and no amount of run history would have found it — which is why `scripts/check_required_check_substance.py` refuses the shape statically instead, and why it named both by name when they were considered for promotion alongside the three scanner checks that were required yesterday.++ Both now carry the shape the gate's own failure message prescribes: the scan step has an `id`, and a blocking step reads `steps.<id>.outcome == 'failure'`. That step sits **ahead of** the ratchet rather than after it, for a reason that was measured rather than assumed — a scan that dies part-way still writes a report, so the ratchet would read the short count and print "lower it to N", which is precisely the advice that would bake a broken scan into the ceiling. Reproduced with the pinned semgrep 1.86.0: an unfetchable rule config exits 7 **and still writes `semgrep.json`**.++ checkov needed two further corrections. Its `pip install` lived *inside* the absorbed step, so a failed install was swallowed and the only symptom was the ratchet complaining about a report that was never written; it is now a blocking step of its own. And its trailing or-true suffix discarded every exit code checkov has, so even with an `id` the outcome could never have been `failure`. That is now an explicit rule over the exit code, measured against the pinned 3.2.334: 0 = no failed checks, 1 = failed checks (the normal state, and the ratchet's decision to make), 2 = a usage error and the only one that is the scan failing to run. `--soft-fail` was measured and rejected as the way to write it — it collapses 1 into 0 but leaves 2 alone, buying nothing, and it is silent on the case that matters most, a `-d` naming a directory that does not exist, which exits 0 under every combination and is caught downstream by the ratchet's shrink-only arm instead.++ The whole workflow tree was swept for the same shape, not just these two: 60 workflow files, 5 `continue-on-error` steps, and the only unguarded ones were these two. The other three are the dependency audits in `security-audit.yml`, which already read all three outcomes in one blocking step. No job-level `continue-on-error` exists anywhere.++### Changed++- **`IaC scan (checkov, ratcheted)` is now a required status check on `main`**, bringing the protected set to 28. It reported a real conclusion on all 40 sampled `pull_request` runs with zero skips — GitHub counts a skipped check as passing, so that is checked rather than assumed — and the absorbed-failure hole above was the only thing disqualifying it. `.github/required-checks.json` was updated in the same change, because `check_required_check_substance.py` cross-checks the committed manifest against the live protection API whenever it holds a token that can read it.++- **`Semgrep (ratcheted)` stays deliberately unrequired even though its hole is now closed**, and the reason is measured rather than cautious: it disagrees with itself on an unchanged tree. Four pull requests on one base, changing nothing but a lockfile, run within thirteen minutes, split 102/40 and 101/39 over the identical 387 rules and 2,267 files. Requiring it would red roughly one pull request in twenty for a reason no author can act on, and the only remedy is a re-run — which teaches people to re-run until green rather than to read what the check said. Fixing the absorbed failure and requiring the job are separate questions and only the first was answered yes; the distinction is recorded in `.github/required-checks.json`, `.security/allowlist.yml` and the `security.yml` header so it is not quietly reversed.+## [13.0.1] - 2026-09-29### Fixeddiff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.mdindex a48eaa243..10def806d 100644--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@ -42,7 +42,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| SAST | README badge (CodeQL) | `codeql.yml` (the scan) + `codeql-alert-gate.yml` (the invariant: `scripts/check_codeql_alerts.py` fails on any open CodeQL alert at any severity, `note` included, and refuses a ref whose declared languages were not analysed at the triggering commit — `analyze` never failed a build on findings, so this row named a scan and not a gate) | GATED | - || Dependency CVE scanning | (implied by security) | `security-audit.yml` | GATED | - || OpenSSF Scorecard | README badge L13 | `scorecard.yml` | GATED | - |-| Container image / IaC / secret scanning (Trivy/checkov/tfsec/gitleaks/Semgrep) | (implied by security) | `security.yml` (claim-matrix ratchet, gitleaks and Trivy fail outright; Semgrep, checkov and tfsec fail on a changed finding count via `check_scanner_ratchet.py` against the ceilings in `.security/allowlist.yml`) + `ci.yml :: Python — Lint & Type-check` (the same gate's allow-list-schema and scanner-coverage half, which needs no scanner). **Three of the six are required checks on `main`** — `Secret scan (gitleaks)`, `Trivy filesystem` and `IaC scan (tfsec, ratcheted)` — because a hard gate that is not a required context blocks nothing: Trivy fails outright on a fixable HIGH/CRITICAL and had been red on `main` for nine consecutive runs with nothing to stop a merge. All six were checked for the defect that makes a required check worthless rather than assumed safe — `security.yml` declares no `paths:` filter and no job-level `if:`, and across 40 `pull_request` runs each of the six reported a real conclusion and **none reported `skipped`**, which GitHub counts as passing. **Two are deliberately not required**: `Semgrep (ratcheted)` and `IaC scan (checkov, ratcheted)` each carry a `continue-on-error` step whose outcome no later step reads, so the job can report success over a failed step and the Actions API cannot see it — `check_required_check_substance.py --static` refuses both by name, so adding either would fail the grading-integrity gate rather than strengthen anything. Semgrep is additionally non-deterministic on an unchanged tree (four pull requests on one base within thirteen minutes split 102/40 and 101/39 over the identical 387 rules and 2,267 files, recorded in `.security/allowlist.yml`), so requiring it would red pull requests that cannot have changed a finding | GATED (this row previously said the three observe scanners "report-and-ratchet" against `.security/allowlist.yml`, and neither half was true: nothing compared the printed counts to anything, and the allow-list had **no reader anywhere in the tree**. The ratchet now fails on more findings than the ceiling, on fewer (shrink-only, printing the measured number), on zero against a non-zero ceiling — a scanner that did not run — on a scanner version that is not the one the ceiling was measured with — for all three, which this row claimed before it was true: the arm read `if version and measured_with and ...` and tfsec's JSON carries no version field, so its counter returned `None` and the `measured_with` in the allow-list was never compared against anything. A version the gate cannot establish is now a failure rather than a skip, tfsec's is read from the binary's own `--version` (a pinned URL says what was asked for; a cached binary would still satisfy it), and a report version and a `--tool-version` that disagree fail in both directions. `_versions_pinned` also could not see a curled binary's pin, which silently exempted tfsec from the test that keeps pin and ceiling in step — on an expired or unreasoned suppression, and on a ceiling/scanner set mismatch in either direction. All three scanners are pinned exactly, because for a ratchet the scanner version is the behaviour. GitHub push-protection remains the always-on hard secret gate) | — |+| Container image / IaC / secret scanning (Trivy/checkov/tfsec/gitleaks/Semgrep) | (implied by security) | `security.yml` (claim-matrix ratchet, gitleaks and Trivy fail outright; Semgrep, checkov and tfsec fail on a changed finding count via `check_scanner_ratchet.py` against the ceilings in `.security/allowlist.yml`) + `ci.yml :: Python — Lint & Type-check` (the same gate's allow-list-schema and scanner-coverage half, which needs no scanner). **Four of the six are required checks on `main`** — `Secret scan (gitleaks)`, `Trivy filesystem`, `IaC scan (tfsec, ratcheted)` and `IaC scan (checkov, ratcheted)` — because a hard gate that is not a required context blocks nothing: Trivy fails outright on a fixable HIGH/CRITICAL and had been red on `main` for nine consecutive runs with nothing to stop a merge. All six were checked for the defect that makes a required check worthless rather than assumed safe — `security.yml` declares no `paths:` filter and no job-level `if:`, and across 40 `pull_request` runs each of the six reported a real conclusion and **none reported `skipped`**, which GitHub counts as passing. `Semgrep (ratcheted)` and `IaC scan (checkov, ratcheted)` each used to carry a `continue-on-error` step whose outcome no later step read, so either job could report success over a scan that failed, where the Actions API cannot see it; **both are now closed** — the scan step carries an `id` and a blocking step reads `steps.<id>.outcome == 'failure'`, placed *ahead* of the ratchet because a scan that dies part-way still writes a short report and the ratchet's shrink-only arm would answer it with "lower the ceiling to N". checkov's `pip install` also moved out of the absorbed step into a blocking one of its own, and the trailing or-true suffix that used to discard checkov's exit code became an explicit rule over that code — 0 and 1 are a scan that ran, anything above is a scan that did not. `--soft-fail` was measured and rejected as the way to write that: it collapses 1 into 0 but leaves 2 alone, and is silent on a `-d` naming a directory that does not exist, which exits 0 under every combination. **`Semgrep (ratcheted)` stays deliberately unrequired even so, and the reason is measured rather than cautious**: it is non-deterministic on an unchanged tree — four pull requests on one base within thirteen minutes split 102/40 and 101/39 over the identical 387 rules and 2,267 files, recorded in `.security/allowlist.yml` — so requiring it would red roughly one pull request in twenty for a reason no author can act on, and the only remedy is a re-run, which trains people to re-run rather than to read. Closing its absorbed-failure hole and requiring it are separate questions, and only the first was answered yes | GATED (this row previously said the three observe scanners "report-and-ratchet" against `.security/allowlist.yml`, and neither half was true: nothing compared the printed counts to anything, and the allow-list had **no reader anywhere in the tree**. The ratchet now fails on more findings than the ceiling, on fewer (shrink-only, printing the measured number), on zero against a non-zero ceiling — a scanner that did not run — on a scanner version that is not the one the ceiling was measured with — for all three, which this row claimed before it was true: the arm read `if version and measured_with and ...` and tfsec's JSON carries no version field, so its counter returned `None` and the `measured_with` in the allow-list was never compared against anything. A version the gate cannot establish is now a failure rather than a skip, tfsec's is read from the binary's own `--version` (a pinned URL says what was asked for; a cached binary would still satisfy it), and a report version and a `--tool-version` that disagree fail in both directions. `_versions_pinned` also could not see a curled binary's pin, which silently exempted tfsec from the test that keeps pin and ceiling in step — on an expired or unreasoned suppression, and on a ceiling/scanner set mismatch in either direction. All three scanners are pinned exactly, because for a ratchet the scanner version is the behaviour. GitHub push-protection remains the always-on hard secret gate) | — || Signed / attested release artifacts | (implied by "run next to crown jewels") | `release.yml` + `publish-images.yml` (cosign keyless signatures, CycloneDX SBOM attestation, BuildKit SLSA provenance + SPDX SBOM per image; SHA-pinned actions), `build-extensions.yml` (extension cosign) | GATED | - || Insecure defaults hard-fail in production | (implied by self-host) | `ci.yml` api tests (`test_security_defaults.py::test_enforce_*`; `enforce_secure_defaults` raises at boot in production) | GATED | - || OpenAPI stability for 3 SDKs + MCP | (implied by SDKs) | `check-openapi.yml` (drift) + `openapi-breaking.yml` (`scripts/openapi_diff.py` — PR spec vs base fails on removed endpoint/schema/field, type change, tightened request, or dropped enum value). A deliberate break goes through the `breaking-change-approved` PR label, which still runs the detector and records every break it permits plus the CHANGELOG `### BREAKING` note that justified it; `--allow-breaking` requires that note as an argument, so the bypass cannot be wired up without the record. Four of the tests assert the *workflow* reaches the flag — it previously had no caller at all | GATED | - |