Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

ci(security): stop two scanner jobs absorbing a failed scan, require checkov (#1066)

Beenu Arorabb17ab84dcef
diff --git a/.github/required-checks.json b/.github/required-checks.json
index 177507161..e5244735a 100644
--- a/.github/required-checks.json
+++ b/.github/required-checks.json
@@ -22,33 +22,46 @@
"The Security workflow's scanner jobs were hard gates that blocked nothing:",
"Trivy fails outright on a fixable HIGH/CRITICAL and had been red on main for",
"nine consecutive runs with nothing to stop a merge, because only",
- "security-audit was required. Three of the six are promoted here. All six",
+ "security-audit was required. Four of the six are promoted here. All six",
"were checked for the defect that makes a required check worthless rather",
"than assumed safe: security.yml declares no paths filter and no job-level",
"if:, and across 40 pull_request runs every one of the six reported a real",
"conclusion and none reported skipped - GitHub counts a skipped check as",
"passing, so requiring one that skips on pull requests would recreate the",
"defect this repository has now fixed twice.",
- "Two are deliberately NOT required, and the reason is the same shape one",
- "level down. 'Semgrep (ratcheted)' and 'IaC scan (checkov, ratcheted)' each",
- "carry a continue-on-error step whose outcome no later step reads, so the",
- "job can report success over a step that failed and the Actions API cannot",
- "see it. check_required_check_substance.py --static refuses both by name;",
- "adding either here fails the gate rather than strengthening anything. The",
- "remedy it prints is to give the step an id and add a blocking step reading",
- "steps.<id>.outcome == 'failure', or to drop continue-on-error.",
- "Semgrep is additionally non-deterministic on an unchanged tree - the",
- "measurement is recorded in .security/allowlist.yml, where four pull",
- "requests on one base within thirteen minutes split 102/40 and 101/39 over",
- "the identical 387 rules and 2267 files - so requiring it would red pull",
- "requests that cannot have changed a finding and teach re-running until",
- "green, which is how a required check stops meaning anything."
+ "'Semgrep (ratcheted)' and 'IaC scan (checkov, ratcheted)' each used to",
+ "carry a continue-on-error step whose outcome no later step read, so the",
+ "job could report success over a step that failed and the Actions API",
+ "cannot see it. Both are now closed the way the gate's own message",
+ "prescribes: the scan step carries an id and a blocking step reads",
+ "steps.<id>.outcome == 'failure'. That step sits AHEAD of the ratchet,",
+ "because a scan that dies part-way still writes a short report and the",
+ "ratchet's shrink-only arm would answer it by asking for the ceiling to be",
+ "lowered - baking the broken scan in. checkov additionally had its pip",
+ "install moved out of the absorbed step into a blocking one, and its",
+ "trailing '|| true' replaced by an explicit exit-code rule: 0 and 1 are a",
+ "scan that ran, anything above is a scan that did not.",
+ "'IaC scan (checkov, ratcheted)' is promoted here on the strength of that",
+ "fix.",
+ "'Semgrep (ratcheted)' is deliberately STILL NOT required, and that is a",
+ "decision rather than an oversight. Do not add it here without reading",
+ "this paragraph: closing its absorbed-failure hole and requiring it are",
+ "separate questions, and only the first was answered yes. It is",
+ "non-deterministic on an unchanged tree - four pull requests on one base,",
+ "changing nothing but a lockfile, run within thirteen minutes, split",
+ "102/40 and 101/39 over the identical 387 rules and 2267 files, with the",
+ "obvious mechanisms eliminated and the whole measurement recorded in",
+ ".security/allowlist.yml. Requiring it would red roughly one pull request",
+ "in twenty for a reason no author can act on, and the only remedy is a",
+ "re-run - which teaches people to re-run until green rather than to read",
+ "the finding, and that is how a required check stops meaning anything."
],
"branch": "main",
"enforced_from": "2026-09-29T12:00:00Z",
"contexts": [
"Backup → destroy → restore (Postgres via S3)",
"Claim-to-gate matrix (ratchet)",
+ "IaC scan (checkov, ratcheted)",
"IaC scan (tfsec, ratcheted)",
"No competitor product named",
"No required check can skip its own assertions",
diff --git a/.github/workflows/security.yml b/.github/workflows/security.yml
index b006eda6e..76ac85023 100644
--- a/.github/workflows/security.yml
+++ b/.github/workflows/security.yml
@@ -28,6 +28,26 @@ name: Security
# They now live in the allow-list, where a run that disagrees with them
# fails and prints the number to write.
#
+# A ratcheted scanner runs its scan under `continue-on-error`, because the
+# scan's own exit status is not the verdict — the count is. That is only safe
+# while something *reads* the outcome, and for Semgrep and checkov nothing
+# did, so either job could report success over a scan that never finished.
+# `continue-on-error` is defined as rewriting a failed step's conclusion to
+# `success`, so the Actions API cannot show this and no amount of run history
+# would have found it; `scripts/check_required_check_substance.py --static`
+# refuses the shape instead. Both now give the scan step an `id` and put a
+# blocking step that reads `steps.<id>.outcome == 'failure'` *ahead of* the
+# ratchet — ahead, because a scan that dies part-way still leaves a short
+# report and the ratchet's shrink-only arm would answer it with "lower the
+# ceiling to N", which would bake the broken scan in.
+#
+# Four of the six jobs here are required checks on `main`: the claim-matrix
+# ratchet, gitleaks, Trivy and checkov. `Semgrep (ratcheted)` is deliberately
+# not, and the reason is measured rather than cautious — it disagrees with
+# itself on an unchanged tree about one run in twenty. See
+# `.security/allowlist.yml` and `.github/required-checks.json`, both of which
+# record the decision so it is not quietly reversed.
+#
# GitHub native push-protection remains the always-on hard secret gate.
on:
@@ -139,7 +159,13 @@ jobs:
# `--error` is gone: it exits non-zero on any finding, which would make
# the exit status carry the same information as the count and force
# `continue-on-error` back onto the ratchet step below with it.
+ #
+ # `continue-on-error` here is deliberate and now *read*, by the step
+ # below. Without `--error`, semgrep exits 0 whether or not it finds
+ # anything, so a non-zero status means the scan itself failed — an
+ # unfetchable rule config, a crash — and not a finding.
- name: Scan
+ id: scan
continue-on-error: true
run: |
semgrep --json --output semgrep.json --disable-version-check \
@@ -150,6 +176,22 @@ jobs:
--config p/react \
--exclude plans/ \
services/ apps/ packages/ scripts/
+ # Blocking, and this is the step that makes the one above safe.
+ # `continue-on-error` rewrites a failed step's *conclusion* to success,
+ # so without something reading its `outcome` the job reports green over
+ # a scan that died — the shape
+ # `scripts/check_required_check_substance.py` refuses by name, and the
+ # reason this check could not be required.
+ #
+ # Ahead of the ratchet on purpose. A scan that failed part-way can still
+ # leave a short `semgrep.json`, and the ratchet's shrink-only arm would
+ # read that short count and print "lower it to N" — advice that would
+ # bake a broken scan into the ceiling.
+ - name: The scan itself did not fail
+ if: steps.scan.outcome == 'failure'
+ run: |
+ echo "::error title=semgrep::semgrep exited non-zero. Without \`--error\` that is the scan failing, not a finding, so the count below would describe a partial run."
+ exit 1
# Blocking. No `continue-on-error`: a missing or truncated report means
# the scan did not finish, and the gate says so rather than passing.
- name: Ratchet the finding count
@@ -225,14 +267,58 @@ jobs:
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.11"
+ # Its own step, and blocking. The install used to sit inside the scan
+ # step under `continue-on-error`, so a failed `pip install` was
+ # absorbed and the only symptom was the ratchet below complaining
+ # about a report that was never written.
+ - name: Install checkov
+ run: pip install --quiet checkov==3.2.334
+ # `continue-on-error` here is deliberate and now *read*, by the step
+ # below. It replaces a trailing `|| true`, which discarded every exit
+ # code checkov has — the one that means "I found things" and the one
+ # that means "I could not run" alike.
- name: checkov (terraform + helm)
+ id: checkov
continue-on-error: true
run: |
- pip install --quiet checkov==3.2.334
# `--output-file-path .` writes ./results_json.json; checkov treats
# a directory argument that way rather than honouring a filename.
+ status=0
checkov -d infra/terraform -d infra/helm --quiet --compact \
- -o json --output-file-path . || true
+ -o json --output-file-path . || status=$?
+ # Measured against the pinned 3.2.334: 0 = no failed checks, 1 =
+ # failed checks, 2 = a usage error. 1 is the normal state here —
+ # there are 219 of them — and deciding on it is the ratchet's job,
+ # not this step's. Anything above 1 is checkov failing to run.
+ #
+ # `--soft-fail` would be the obvious way to write this and is the
+ # wrong one: measured, it collapses 1 to 0 but leaves 2 alone,
+ # which buys nothing, and it is silent about the case that matters
+ # most — a `-d` naming a directory that does not exist exits 0
+ # under every combination, logging an ERROR and scanning what is
+ # left. That one is caught downstream instead, by the ratchet's
+ # shrink-only arm, because a vanished directory shows up as a
+ # count below the ceiling.
+ if [ "$status" -gt 1 ]; then
+ echo "::error title=iac-checkov::checkov exited $status — that is the scan failing to run, not a finding"
+ exit "$status"
+ fi
+ # Blocking, and this is the step that makes the one above safe.
+ # `continue-on-error` rewrites a failed step's *conclusion* to success,
+ # so without something reading its `outcome` the job reports green over
+ # a scan that died — the shape
+ # `scripts/check_required_check_substance.py` refuses by name, and the
+ # reason this check could not be required.
+ #
+ # Ahead of the ratchet on purpose. A scan that failed part-way can
+ # still leave a short `results_json.json`, and the ratchet's
+ # shrink-only arm would read that short count and print "lower it to
+ # N" — advice that would bake a broken scan into the ceiling.
+ - name: The scan itself did not fail
+ if: steps.checkov.outcome == 'failure'
+ run: |
+ echo "::error title=iac-checkov::checkov did not complete, so the count below would describe a partial scan"
+ exit 1
- name: Ratchet the checkov finding count
run: python3 scripts/check_scanner_ratchet.py --tool checkov --report results_json.json
diff --git a/.security/allowlist.yml b/.security/allowlist.yml
index c89939588..944949d46 100644
--- a/.security/allowlist.yml
+++ b/.security/allowlist.yml
@@ -90,6 +90,24 @@ ceilings:
# one is to retain `semgrep.json` as a run artefact and diff two runs of
# the same tree. Nothing here does that yet.
#
+ # Third consequence, and the one most likely to be undone by someone
+ # tidying up: this is why `Semgrep (ratcheted)` is **not** a required
+ # check on `main` while the other ratcheted scanners are. The job used to
+ # carry a second, unrelated defect — a `continue-on-error` scan step whose
+ # outcome nothing read, so the job reported success over a scan that had
+ # died — and that one is now fixed, in the same change that promoted
+ # `IaC scan (checkov, ratcheted)` to a required check after fixing the
+ # identical hole there. Semgrep was deliberately left unrequired anyway.
+ # The split above is the reason: at one run in twenty disagreeing with
+ # itself, requiring this job would red a pull request whose author cannot
+ # have caused it and cannot do anything about it except press re-run, and
+ # a required check whose documented remedy is "run it again" teaches
+ # people to re-run until green instead of reading what it said. Fixing
+ # the absorbed failure and requiring the job are separate questions;
+ # only the first was answered yes. Requiring it becomes reasonable once
+ # the +/-1 is attributed and eliminated, not before.
+ # The same reasoning is recorded in `.github/required-checks.json`.
+ #
# 102 -> 103 (error unchanged at 40). The added WARNING is a third
# `dynamic-urllib-use-detected`, on scripts/check_required_check_substance.py,
# which asks the Actions API which steps each required check actually
diff --git a/CHANGELOG.md b/CHANGELOG.md
index fafcfa4f9..a588e0e54 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Fixed
+
+- **Two scanner jobs in `security.yml` could report success over a scan that failed, and that was the only thing keeping one of them off branch protection.** `Semgrep (ratcheted)` and `IaC scan (checkov, ratcheted)` each ran their scan under `continue-on-error: true` with no later step reading the outcome. `continue-on-error` is *defined* as rewriting a failed step's conclusion to `success`, so the Actions API shows a clean job and no amount of run history would have found it — which is why `scripts/check_required_check_substance.py` refuses the shape statically instead, and why it named both by name when they were considered for promotion alongside the three scanner checks that were required yesterday.
+
+ Both now carry the shape the gate's own failure message prescribes: the scan step has an `id`, and a blocking step reads `steps.<id>.outcome == 'failure'`. That step sits **ahead of** the ratchet rather than after it, for a reason that was measured rather than assumed — a scan that dies part-way still writes a report, so the ratchet would read the short count and print "lower it to N", which is precisely the advice that would bake a broken scan into the ceiling. Reproduced with the pinned semgrep 1.86.0: an unfetchable rule config exits 7 **and still writes `semgrep.json`**.
+
+ checkov needed two further corrections. Its `pip install` lived *inside* the absorbed step, so a failed install was swallowed and the only symptom was the ratchet complaining about a report that was never written; it is now a blocking step of its own. And its trailing or-true suffix discarded every exit code checkov has, so even with an `id` the outcome could never have been `failure`. That is now an explicit rule over the exit code, measured against the pinned 3.2.334: 0 = no failed checks, 1 = failed checks (the normal state, and the ratchet's decision to make), 2 = a usage error and the only one that is the scan failing to run. `--soft-fail` was measured and rejected as the way to write it — it collapses 1 into 0 but leaves 2 alone, buying nothing, and it is silent on the case that matters most, a `-d` naming a directory that does not exist, which exits 0 under every combination and is caught downstream by the ratchet's shrink-only arm instead.
+
+ The whole workflow tree was swept for the same shape, not just these two: 60 workflow files, 5 `continue-on-error` steps, and the only unguarded ones were these two. The other three are the dependency audits in `security-audit.yml`, which already read all three outcomes in one blocking step. No job-level `continue-on-error` exists anywhere.
+
+### Changed
+
+- **`IaC scan (checkov, ratcheted)` is now a required status check on `main`**, bringing the protected set to 28. It reported a real conclusion on all 40 sampled `pull_request` runs with zero skips — GitHub counts a skipped check as passing, so that is checked rather than assumed — and the absorbed-failure hole above was the only thing disqualifying it. `.github/required-checks.json` was updated in the same change, because `check_required_check_substance.py` cross-checks the committed manifest against the live protection API whenever it holds a token that can read it.
+
+- **`Semgrep (ratcheted)` stays deliberately unrequired even though its hole is now closed**, and the reason is measured rather than cautious: it disagrees with itself on an unchanged tree. Four pull requests on one base, changing nothing but a lockfile, run within thirteen minutes, split 102/40 and 101/39 over the identical 387 rules and 2,267 files. Requiring it would red roughly one pull request in twenty for a reason no author can act on, and the only remedy is a re-run — which teaches people to re-run until green rather than to read what the check said. Fixing the absorbed failure and requiring the job are separate questions and only the first was answered yes; the distinction is recorded in `.github/required-checks.json`, `.security/allowlist.yml` and the `security.yml` header so it is not quietly reversed.
+
## [13.0.1] - 2026-09-29
### Fixed
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index a48eaa243..10def806d 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -42,7 +42,7 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| SAST | README badge (CodeQL) | `codeql.yml` (the scan) + `codeql-alert-gate.yml` (the invariant: `scripts/check_codeql_alerts.py` fails on any open CodeQL alert at any severity, `note` included, and refuses a ref whose declared languages were not analysed at the triggering commit — `analyze` never failed a build on findings, so this row named a scan and not a gate) | GATED | - |
| Dependency CVE scanning | (implied by security) | `security-audit.yml` | GATED | - |
| OpenSSF Scorecard | README badge L13 | `scorecard.yml` | GATED | - |
-| Container image / IaC / secret scanning (Trivy/checkov/tfsec/gitleaks/Semgrep) | (implied by security) | `security.yml` (claim-matrix ratchet, gitleaks and Trivy fail outright; Semgrep, checkov and tfsec fail on a changed finding count via `check_scanner_ratchet.py` against the ceilings in `.security/allowlist.yml`) + `ci.yml :: Python — Lint & Type-check` (the same gate's allow-list-schema and scanner-coverage half, which needs no scanner). **Three of the six are required checks on `main`** — `Secret scan (gitleaks)`, `Trivy filesystem` and `IaC scan (tfsec, ratcheted)` — because a hard gate that is not a required context blocks nothing: Trivy fails outright on a fixable HIGH/CRITICAL and had been red on `main` for nine consecutive runs with nothing to stop a merge. All six were checked for the defect that makes a required check worthless rather than assumed safe — `security.yml` declares no `paths:` filter and no job-level `if:`, and across 40 `pull_request` runs each of the six reported a real conclusion and **none reported `skipped`**, which GitHub counts as passing. **Two are deliberately not required**: `Semgrep (ratcheted)` and `IaC scan (checkov, ratcheted)` each carry a `continue-on-error` step whose outcome no later step reads, so the job can report success over a failed step and the Actions API cannot see it — `check_required_check_substance.py --static` refuses both by name, so adding either would fail the grading-integrity gate rather than strengthen anything. Semgrep is additionally non-deterministic on an unchanged tree (four pull requests on one base within thirteen minutes split 102/40 and 101/39 over the identical 387 rules and 2,267 files, recorded in `.security/allowlist.yml`), so requiring it would red pull requests that cannot have changed a finding | GATED (this row previously said the three observe scanners "report-and-ratchet" against `.security/allowlist.yml`, and neither half was true: nothing compared the printed counts to anything, and the allow-list had **no reader anywhere in the tree**. The ratchet now fails on more findings than the ceiling, on fewer (shrink-only, printing the measured number), on zero against a non-zero ceiling — a scanner that did not run — on a scanner version that is not the one the ceiling was measured with — for all three, which this row claimed before it was true: the arm read `if version and measured_with and ...` and tfsec's JSON carries no version field, so its counter returned `None` and the `measured_with` in the allow-list was never compared against anything. A version the gate cannot establish is now a failure rather than a skip, tfsec's is read from the binary's own `--version` (a pinned URL says what was asked for; a cached binary would still satisfy it), and a report version and a `--tool-version` that disagree fail in both directions. `_versions_pinned` also could not see a curled binary's pin, which silently exempted tfsec from the test that keeps pin and ceiling in step — on an expired or unreasoned suppression, and on a ceiling/scanner set mismatch in either direction. All three scanners are pinned exactly, because for a ratchet the scanner version is the behaviour. GitHub push-protection remains the always-on hard secret gate) | — |
+| Container image / IaC / secret scanning (Trivy/checkov/tfsec/gitleaks/Semgrep) | (implied by security) | `security.yml` (claim-matrix ratchet, gitleaks and Trivy fail outright; Semgrep, checkov and tfsec fail on a changed finding count via `check_scanner_ratchet.py` against the ceilings in `.security/allowlist.yml`) + `ci.yml :: Python — Lint & Type-check` (the same gate's allow-list-schema and scanner-coverage half, which needs no scanner). **Four of the six are required checks on `main`** — `Secret scan (gitleaks)`, `Trivy filesystem`, `IaC scan (tfsec, ratcheted)` and `IaC scan (checkov, ratcheted)` — because a hard gate that is not a required context blocks nothing: Trivy fails outright on a fixable HIGH/CRITICAL and had been red on `main` for nine consecutive runs with nothing to stop a merge. All six were checked for the defect that makes a required check worthless rather than assumed safe — `security.yml` declares no `paths:` filter and no job-level `if:`, and across 40 `pull_request` runs each of the six reported a real conclusion and **none reported `skipped`**, which GitHub counts as passing. `Semgrep (ratcheted)` and `IaC scan (checkov, ratcheted)` each used to carry a `continue-on-error` step whose outcome no later step read, so either job could report success over a scan that failed, where the Actions API cannot see it; **both are now closed** — the scan step carries an `id` and a blocking step reads `steps.<id>.outcome == 'failure'`, placed *ahead* of the ratchet because a scan that dies part-way still writes a short report and the ratchet's shrink-only arm would answer it with "lower the ceiling to N". checkov's `pip install` also moved out of the absorbed step into a blocking one of its own, and the trailing or-true suffix that used to discard checkov's exit code became an explicit rule over that code — 0 and 1 are a scan that ran, anything above is a scan that did not. `--soft-fail` was measured and rejected as the way to write that: it collapses 1 into 0 but leaves 2 alone, and is silent on a `-d` naming a directory that does not exist, which exits 0 under every combination. **`Semgrep (ratcheted)` stays deliberately unrequired even so, and the reason is measured rather than cautious**: it is non-deterministic on an unchanged tree — four pull requests on one base within thirteen minutes split 102/40 and 101/39 over the identical 387 rules and 2,267 files, recorded in `.security/allowlist.yml` — so requiring it would red roughly one pull request in twenty for a reason no author can act on, and the only remedy is a re-run, which trains people to re-run rather than to read. Closing its absorbed-failure hole and requiring it are separate questions, and only the first was answered yes | GATED (this row previously said the three observe scanners "report-and-ratchet" against `.security/allowlist.yml`, and neither half was true: nothing compared the printed counts to anything, and the allow-list had **no reader anywhere in the tree**. The ratchet now fails on more findings than the ceiling, on fewer (shrink-only, printing the measured number), on zero against a non-zero ceiling — a scanner that did not run — on a scanner version that is not the one the ceiling was measured with — for all three, which this row claimed before it was true: the arm read `if version and measured_with and ...` and tfsec's JSON carries no version field, so its counter returned `None` and the `measured_with` in the allow-list was never compared against anything. A version the gate cannot establish is now a failure rather than a skip, tfsec's is read from the binary's own `--version` (a pinned URL says what was asked for; a cached binary would still satisfy it), and a report version and a `--tool-version` that disagree fail in both directions. `_versions_pinned` also could not see a curled binary's pin, which silently exempted tfsec from the test that keeps pin and ceiling in step — on an expired or unreasoned suppression, and on a ceiling/scanner set mismatch in either direction. All three scanners are pinned exactly, because for a ratchet the scanner version is the behaviour. GitHub push-protection remains the always-on hard secret gate) | — |
| Signed / attested release artifacts | (implied by "run next to crown jewels") | `release.yml` + `publish-images.yml` (cosign keyless signatures, CycloneDX SBOM attestation, BuildKit SLSA provenance + SPDX SBOM per image; SHA-pinned actions), `build-extensions.yml` (extension cosign) | GATED | - |
| Insecure defaults hard-fail in production | (implied by self-host) | `ci.yml` api tests (`test_security_defaults.py::test_enforce_*`; `enforce_secure_defaults` raises at boot in production) | GATED | - |
| OpenAPI stability for 3 SDKs + MCP | (implied by SDKs) | `check-openapi.yml` (drift) + `openapi-breaking.yml` (`scripts/openapi_diff.py` — PR spec vs base fails on removed endpoint/schema/field, type change, tightened request, or dropped enum value). A deliberate break goes through the `breaking-change-approved` PR label, which still runs the detector and records every break it permits plus the CHANGELOG `### BREAKING` note that justified it; `--allow-breaking` requires that note as an argument, so the bypass cannot be wired up without the record. Four of the tests assert the *workflow* reaches the flag — it previously had no caller at all | GATED | - |