Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(gates): give the scanner allow-list a reader and the scanners a ratchet (#987)

Beenu Aroraf55be02ff6bc
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 336f02f1c..a6ae869c6 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -570,6 +570,16 @@ jobs:
run: python3 scripts/check_replay_contract_parity.py --self-test
- name: Replay contract parity gate
run: python3 scripts/check_replay_contract_parity.py
+ # The finding-count comparison itself lives in the three scanner jobs in
+ # security.yml, where the reports are. This runs the half that needs no
+ # scanner — the allow-list's own schema, its expiry dates, and whether
+ # the set of ceilings still matches the set of scanners the workflow
+ # runs — in a job that is a required check, so a ceiling left behind by
+ # a deleted scanner cannot hide in a job nobody waits for.
+ - name: Scanner ratchet gate — self-test
+ run: python3 scripts/check_scanner_ratchet.py --self-test
+ - name: Scanner ratchet gate
+ run: python3 scripts/check_scanner_ratchet.py
# The three sdk-* jobs below compile and test each SDK in isolation, and
# none of them compares a client to the API it claims to speak to.
# packages/sdk-go passed all three over a file of type declarations with
diff --git a/.github/workflows/security.yml b/.github/workflows/security.yml
index 9556fa020..f851447b7 100644
--- a/.github/workflows/security.yml
+++ b/.github/workflows/security.yml
@@ -15,15 +15,18 @@ name: Security
# the pinned release had been deleted upstream, so the download
# 404'd, the scan ran a binary that was not there, and
# continue-on-error turned the whole thing green.
-# - OBSERVE gates (report, do not fail yet): Semgrep, checkov, tfsec.
-# These are not observe-mode because nobody has looked. They were run
-# against this tree and the counts are: Semgrep 92 (35 at ERROR),
-# checkov 91 failed checks under infra/terraform, tfsec 32 (7 CRITICAL,
-# 5 HIGH). Blocking on 215 untriaged findings would red every pull
+# - RATCHETED gates (report every finding, fail on a new one): Semgrep,
+# checkov, tfsec. Blocking on 215 untriaged findings would red every pull
# request on day one and the first response would be to turn the gate
-# back off, so each needs its own triage pass first. The count is stated
-# here rather than left as "we'll get to it" so the size of the
-# remaining work is visible.
+# back off, so none of the three fails on its existing findings. What
+# they now fail on is a *change* in the count, in either direction,
+# against the ceilings in `.security/allowlist.yml` — enforced by
+# `scripts/check_scanner_ratchet.py`, which is the first thing in this
+# repository ever to open that file. The counts used to live in this
+# comment, where nothing compared them to anything: a pull request adding
+# twenty findings and one removing twenty produced the same green tick.
+# They now live in the allow-list, where a run that disagrees with them
+# fails and prints the number to write.
#
# GitHub native push-protection remains the always-on hard secret gate.
@@ -110,14 +113,21 @@ jobs:
- name: Scan working tree (respects .gitleaksignore)
run: gitleaks dir . --redact --exit-code 1 --no-banner
- # ── OBSERVE: SAST beyond CodeQL ───────────────────────────────────────────
+ # ── RATCHETED: SAST beyond CodeQL ─────────────────────────────────────────
#
- # 92 findings against this tree (35 ERROR, 51 WARNING, 3 MEDIUM, 3 INFO),
- # measured with semgrep 1.86.0 and the five configs below. Too many to
- # block on before they are triaged, and `--error` fails on any of them, so
- # this stays report-only with the number written down.
+ # 92 findings against this tree (35 ERROR, 51 WARNING, 3 MEDIUM, 3 INFO).
+ # Too many to block on before they are triaged, and `--error` fails on any
+ # of them — so the scan itself stays non-blocking and the *count* is what
+ # blocks, against the ceiling in `.security/allowlist.yml`.
+ #
+ # The version is pinned exactly. For a ratchet the scanner version is the
+ # behaviour: an unpinned `pip install semgrep` re-baselines the count on
+ # upstream's release schedule, which is indistinguishable from a change in
+ # this tree. The gate compares the version the report declares against the
+ # one the ceiling was measured with and fails if they differ, so a bump
+ # here is an explicit re-measure rather than a silent one.
semgrep:
- name: Semgrep (observe)
+ name: Semgrep (ratcheted)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
@@ -125,11 +135,14 @@ jobs:
with:
python-version: "3.11"
- name: Install semgrep
- run: pip install --quiet semgrep
+ run: pip install --quiet semgrep==1.86.0
+ # `--error` is gone: it exits non-zero on any finding, which would make
+ # the exit status carry the same information as the count and force
+ # `continue-on-error` back onto the ratchet step below with it.
- name: Scan
continue-on-error: true
run: |
- semgrep --error --disable-version-check \
+ semgrep --json --output semgrep.json --disable-version-check \
--config p/security-audit \
--config p/owasp-top-ten \
--config p/python \
@@ -137,6 +150,10 @@ jobs:
--config p/react \
--exclude plans/ \
services/ apps/ packages/ scripts/
+ # Blocking. No `continue-on-error`: a missing or truncated report means
+ # the scan did not finish, and the gate says so rather than passing.
+ - name: Ratchet the finding count
+ run: python3 scripts/check_scanner_ratchet.py --tool semgrep --report semgrep.json
# ── HARD: filesystem vuln scan (complements dependency SCA) ───────────────
trivy-fs:
@@ -180,15 +197,28 @@ jobs:
set -euo pipefail
/tmp/trivy fs --severity HIGH,CRITICAL --ignore-unfixed --skip-dirs plans --exit-code 1 .
- # ── OBSERVE: IaC misconfig scan ───────────────────────────────────────────
+ # ── RATCHETED: IaC misconfig scan ─────────────────────────────────────────
+ #
+ # checkov reports 91 failed checks against infra/terraform (164 passing);
+ # tfsec reports 32 findings there (7 CRITICAL, 5 HIGH, 2 MEDIUM, 18 LOW).
+ # Neither scan fails on those — promoting them outright would red every
+ # pull request over pre-existing infrastructure defaults that no single
+ # change introduced. What fails is a count that no longer matches the
+ # ceiling in `.security/allowlist.yml`, in either direction.
#
- # checkov 3.2.x reports 91 failed checks against infra/terraform (164
- # passing); tfsec 1.28.x reports 32 findings there (7 CRITICAL, 5 HIGH,
- # 2 MEDIUM, 18 LOW). Both stay report-only until those are triaged —
- # promoting them now would red every pull request over pre-existing
- # infrastructure defaults that no single change introduced.
- iac-scan:
- name: IaC scan (checkov + tfsec, observe)
+ # tfsec runs as a pinned binary rather than through the marketplace action,
+ # for the two reasons the Trivy job above already records: the action's
+ # version tags proved unreliable, and the ratchet needs the JSON report on
+ # disk. `--soft-fail` keeps the scan's own exit status out of it so the
+ # count is the only thing that decides.
+ # One job each, not one job for both. They shared a job while neither
+ # blocked, and the moment the ratchet made the steps hard the first
+ # scanner to breach its ceiling ended the job before the second one ran —
+ # so a real count for tfsec was unobtainable whenever checkov was red.
+ # A gate that can stop another gate from measuring is a gate that reports
+ # on whichever half happened to run first.
+ iac-checkov:
+ name: IaC scan (checkov, ratcheted)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
@@ -198,11 +228,34 @@ jobs:
- name: checkov (terraform + helm)
continue-on-error: true
run: |
- pip install --quiet checkov
- checkov -d infra/terraform --quiet --compact || true
- checkov -d infra/helm --quiet --compact || true
- - name: tfsec (terraform)
- continue-on-error: true
- uses: aquasecurity/tfsec-action@b466648d6e39e7c75324f25d83891162a721f2d6 # v1.0.3
+ pip install --quiet checkov==3.2.334
+ # `--output-file-path .` writes ./results_json.json; checkov treats
+ # a directory argument that way rather than honouring a filename.
+ checkov -d infra/terraform -d infra/helm --quiet --compact \
+ -o json --output-file-path . || true
+ - name: Ratchet the checkov finding count
+ run: python3 scripts/check_scanner_ratchet.py --tool checkov --report results_json.json
+
+ iac-tfsec:
+ name: IaC scan (tfsec, ratcheted)
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+ - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
- working_directory: infra/terraform
+ python-version: "3.11"
+ - name: Install tfsec
+ run: |
+ set -euo pipefail
+ VERSION=1.28.13
+ url="https://github.com/aquasecurity/tfsec/releases/download/v${VERSION}/tfsec-linux-amd64"
+ if ! curl -sSL --fail --retry 5 --retry-delay 5 --retry-all-errors "$url" -o /tmp/tfsec; then
+ echo "::error title=tfsec::could not download $url — check whether v${VERSION} still exists and bump the pin"
+ exit 1
+ fi
+ chmod +x /tmp/tfsec
+ /tmp/tfsec --version
+ - name: tfsec (terraform)
+ run: /tmp/tfsec infra/terraform --format json --out tfsec.json --soft-fail
+ - name: Ratchet the tfsec finding count
+ run: python3 scripts/check_scanner_ratchet.py --tool tfsec --report tfsec.json
diff --git a/.security/allowlist.yml b/.security/allowlist.yml
index 96a9a4612..b59ec8ca7 100644
--- a/.security/allowlist.yml
+++ b/.security/allowlist.yml
@@ -1,20 +1,66 @@
-# Security scanner allowlist (Phase 2)
+# Security scanner ceilings and suppressions.
#
-# Suppressions for the Semgrep / Trivy / checkov / tfsec scanners in
-# .github/workflows/security.yml. Every entry MUST carry a reason and an
-# expiry date; expired entries fail CI once the scanners flip from observe to
-# blocking (tracked Phase 2 continuation). This is separate from the
-# dependency-CVE ignore list at scripts/security_audit_ignores.txt.
+# Read by scripts/check_scanner_ratchet.py, which runs as a blocking step in
+# each of the three observe-mode scanner jobs in .github/workflows/security.yml.
+# Until that gate was written this file had no reader at all: the workflow
+# described its scanners as "report-and-ratchet" and named this path as where
+# the triaged baseline lives, and nothing opened it. An allow-list nobody
+# reads is worse than no allow-list, because it looks like a control.
#
-# Schema:
+# ── ceilings ────────────────────────────────────────────────────────────────
+#
+# A scanner may report at most this many findings. The ratchet is shrink-only:
+# reporting fewer also fails, with the measured number in the message, so the
+# ceiling comes down in the same change that removes the finding rather than
+# leaving room for the count to drift back up for free. Reporting zero against
+# a non-zero ceiling fails too — a scanner that previously found dozens and
+# now finds none did not run, and "clean" must never look like "did not run".
+#
+# `measured_with` is compared against the version the report declares. Counts
+# move when rules move, so a ceiling compared against a different scanner is
+# not a comparison; the scanners are pinned exactly in the workflow for the
+# same reason, because for a ratchet the scanner version is the behaviour.
+#
+# To lower a ceiling: fix findings, read the measured count out of the gate's
+# failure message, and write it here in the same pull request.
+
+ceilings:
+ # Every number here was measured by this gate in CI, not carried over.
+ # The counts the workflow header used to publish were 92/35 for semgrep and
+ # 91 for checkov, and both were wrong the first time the ratchet compared
+ # them: semgrep came back 101/40 because the tree has grown since anyone
+ # looked, and checkov came back 219 because the 91 covered infra/terraform
+ # while the job had always scanned infra/helm as well and reported only the
+ # first number. That is the whole argument for the ratchet in one run — a
+ # count nothing compares to anything drifts, and nobody finds out.
+ semgrep:
+ total: 101
+ error: 40
+ measured_with: "1.86.0"
+ checkov:
+ failed: 219
+ measured_with: "3.2.334"
+ tfsec:
+ total: 32
+ critical: 7
+ high: 5
+ measured_with: "1.28.13"
+
+# ── suppressions ────────────────────────────────────────────────────────────
+#
+# Individual findings accepted as risk. Separate from the dependency-CVE
+# ignore list at scripts/security_audit_ignores.txt.
+#
+# Schema, enforced by the gate:
# entries:
-# - tool: semgrep|trivy|checkov|tfsec
+# - tool: semgrep|checkov|tfsec
# id: <rule/check id>
# path: <optional path glob>
-# reason: <why this is acceptable> # required, non-empty
-# expires: YYYY-MM-DD # required, <= 90 days out
+# reason: <why this is acceptable> # required, non-empty
+# expires: YYYY-MM-DD # required, at most 90 days out
#
-# Keep this list empty until a scanner is ratcheted to blocking; add triaged
-# baseline findings here at that time, never blanket-suppress.
+# An expired entry fails the build. Re-triage it or fix the finding; do not
+# extend the date. Prefer lowering a ceiling over adding an entry here —
+# a suppression hides a finding, a ceiling counts it.
entries: []
diff --git a/scripts/check_scanner_ratchet.py b/scripts/check_scanner_ratchet.py
new file mode 100644
index 000000000..ead73331d
--- /dev/null
+++ b/scripts/check_scanner_ratchet.py
@@ -0,0 +1,447 @@
+#!/usr/bin/env python3
+"""The observe-mode scanners get a ratchet, and `.security/allowlist.yml` gets a reader.
+
+Why this exists
+---------------
+
+``.github/workflows/security.yml`` describes its three observe-mode scanners
+as "report-and-ratchet" and names ``.security/allowlist.yml`` as where the
+triaged baseline lives. Both halves of that sentence were untrue.
+
+There was no ratchet. Semgrep, checkov and tfsec each ran under
+``continue-on-error: true`` — checkov additionally with a trailing ``|| true``
+— printed a number, and nothing compared that number to anything. A pull
+request that added twenty findings and one that removed twenty produced the
+same green tick, so the counts written into the workflow header ("Semgrep 92,
+35 at ERROR; checkov 91 failed; tfsec 32, 7 CRITICAL, 5 HIGH") were a
+measurement taken once and then left to rot.
+
+And ``.security/allowlist.yml`` had **no reader anywhere in the tree**. A
+repository-wide search for its path returned two documents talking *about* it
+and not one line of code that opens it. Its header declares a schema, requires
+a reason and an expiry on every entry, and says expired entries "fail CI" —
+enforced by nothing. An allow-list nobody reads is worse than no allow-list:
+it looks like a control, so the next person assumes the triage happened.
+
+``scripts/check_gate_coverage.py`` would have caught this had the orphan been
+a *script*. It inventories ``scripts/`` against the workflows. A control file
+in YAML with no caller is the same defect one file type over, and nothing was
+looking for it.
+
+What it checks
+--------------
+
+``regression``
+ A scanner reports more findings than its recorded ceiling. This is the
+ property the word "ratchet" promises and the only one that protects the
+ tree from accumulating new findings.
+
+``stale-ceiling``
+ A scanner reports *fewer* findings than its ceiling. Shrink-only: the
+ ceiling must come down in the same change that removes the finding, or
+ the number drifts back up for free and the ratchet is decorative. The
+ failure prints the measured count, so one CI run tells you the value to
+ write.
+
+``vacuous``
+ Zero findings against a non-zero ceiling. A scanner that reports nothing
+ where it previously reported dozens did not run. This repository has the
+ scar: the Trivy pin 404'd, ``tar`` failed, a binary that was not there was
+ invoked, and ``continue-on-error`` reported green. "Clean" and "did not
+ run" must never look the same.
+
+``version-drift``
+ The scanner version that produced this report is not the version the
+ ceiling was measured with. Counts move when rules move, so a ceiling
+ compared against a different scanner is not a comparison. Failing here
+ turns a silent re-baseline into an explicit, actionable one.
+
+``allowlist``
+ Every entry carries a non-empty reason and an expiry no more than 90 days
+ out, names a tool that is actually run, and has not expired. The file's
+ own header has required this since it was written; this is the first
+ thing to enforce it.
+
+``coverage`` (both directions)
+ Every scanner ``security.yml`` invokes has a ceiling, **and** every
+ ceiling names a scanner ``security.yml`` invokes. One direction alone is
+ the failure shape this repository keeps rediscovering: a gate that
+ compares A against B and never B against A passes while drift
+ accumulates in the direction things actually change. Here the drift that
+ matters is a scanner being removed from the workflow while its ceiling
+ stays behind, which would leave a ratchet guarding nothing.
+
+What it reads
+-------------
+
+``.security/allowlist.yml`` for the ceilings and the entries, parsed by a
+strict reader over the documented subset rather than by PyYAML — the gate has
+to run in ``Python — Lint & Type-check``, which installs ruff and mypy and
+nothing else. The reader refuses anything it does not recognise rather than
+skipping it, and ``scripts/tests/test_check_scanner_ratchet.py`` asserts it
+agrees with ``yaml.safe_load`` on the real file so the two cannot diverge.
+
+``.github/workflows/security.yml`` for which scanners are actually invoked.
+
+The scanner's own JSON report, when ``--tool``/``--report`` are given.
+
+Usage
+-----
+
+::
+
+ python3 scripts/check_scanner_ratchet.py # allow-list + coverage
+ python3 scripts/check_scanner_ratchet.py --list # print the ceilings
+ python3 scripts/check_scanner_ratchet.py --tool semgrep --report semgrep.json
+ python3 scripts/check_scanner_ratchet.py --self-test
+
+Exit codes: 0 clean, 1 findings, 2 the check itself could not run.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from datetime import UTC, date, datetime
+from pathlib import Path
+from typing import Any
+
+# `scripts/` is on sys.path when this file is run as a program, but not when a
+# test loads it by path with importlib. gate_toolkit sits beside it either way.
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+from gate_toolkit import repo_root, self_test_if_requested
+
+self_test_if_requested(__file__)
+
+ALLOWLIST_REL = Path(".security") / "allowlist.yml"
+WORKFLOW_REL = Path(".github") / "workflows" / "security.yml"
+
+#: How far out an allow-list expiry may sit. The file's own header states 90
+#: days; this is that sentence, enforced.
+MAX_EXPIRY_DAYS = 90
+
+#: The scanners this gate knows how to count. A tool invoked by the workflow
+#: that is missing here fails `coverage` rather than being skipped, because a
+#: scanner nobody counts is the state this gate exists to end.
+KNOWN_TOOLS = ("semgrep", "checkov", "tfsec")
+
+
+class AllowlistError(ValueError):
+ """The allow-list could not be read as the schema its header declares."""
+
+
+# ── the strict reader ────────────────────────────────────────────────────────
+#
+# The documented subset is: top-level keys; two levels of nesting under
+# `ceilings:`; scalars that are integers or quoted strings; and `entries:` as
+# either `[]` or a list of flat `- key: value` mappings. Anything else raises.
+
+
+def _scalar(raw: str, where: str) -> Any:
+ text = raw.strip()
+ if text.startswith(('"', "'")) and text.endswith(text[0]) and len(text) >= 2:
+ return text[1:-1]
+ if re.fullmatch(r"-?\d+", text):
+ return int(text)
+ if text in {"true", "false"}:
+ return text == "true"
+ if text == "[]":
+ return []
+ raise AllowlistError(f"{where}: cannot read {raw!r} — quote strings, and use plain integers for counts")
+
+
+def parse_allowlist(text: str) -> dict[str, Any]:
+ """Read the documented subset of the allow-list. Refuse the rest."""
+ root: dict[str, Any] = {}
+ stack: list[tuple[int, Any]] = [(-1, root)]
+ for lineno, raw in enumerate(text.splitlines(), start=1):
+ if not raw.strip() or raw.lstrip().startswith("#"):
+ continue
+ indent = len(raw) - len(raw.lstrip(" "))
+ line = raw.strip()
+ where = f"{ALLOWLIST_REL}:{lineno}"
+
+ if "\t" in raw:
+ raise AllowlistError(f"{where}: tab indentation is not readable as YAML")
+
+ while stack and indent <= stack[-1][0]:
+ stack.pop()
+ if not stack:
+ raise AllowlistError(f"{where}: indentation does not nest under anything")
+ parent = stack[-1][1]
+
+ if line.startswith("- "):
+ if not isinstance(parent, list):
+ raise AllowlistError(f"{where}: list item under a mapping")
+ item: dict[str, Any] = {}
+ parent.append(item)
+ key, _, value = line[2:].partition(":")
+ if not _:
+ raise AllowlistError(f"{where}: list items must be `- key: value` mappings")
+ item[key.strip()] = _scalar(value, where)
+ # A list item's own keys sit at the column the `-` opened, plus
+ # two: `- tool: x` then ` id: y`. Both belong to the same item.
+ stack.append((indent + 1, item))
+ continue
+
+ key, sep, value = line.partition(":")
+ if not sep:
+ raise AllowlistError(f"{where}: expected `key: value`, got {line!r}")
+ key = key.strip()
+ if not value.strip():
+ child: dict[str, Any] | list[Any] = [] if key == "entries" else {}
+ if isinstance(parent, dict):
+ parent[key] = child
+ else:
+ raise AllowlistError(f"{where}: mapping key inside a list item without a value")
+ stack.append((indent, child))
+ continue
+ if isinstance(parent, dict):
+ parent[key] = _scalar(value, where)
+ else:
+ raise AllowlistError(f"{where}: `{key}` is not inside a mapping")
+ return root
+
+
+# ── counting each scanner's report ───────────────────────────────────────────
+
+
+def _semgrep_counts(doc: Any) -> tuple[dict[str, int], str | None]:
+ results = doc.get("results") or []
+ severities = [str((r.get("extra") or {}).get("severity", "")).upper() for r in results]
+ return (
+ {"total": len(results), "error": sum(1 for s in severities if s == "ERROR")},
+ doc.get("version"),
+ )
+
+
+def _checkov_counts(doc: Any) -> tuple[dict[str, int], str | None]:
+ # checkov emits a list when several frameworks ran and a bare object when
+ # one did. Both shapes reach this gate, so both are read.
+ blocks = doc if isinstance(doc, list) else [doc]
+ failed = 0
+ version: str | None = None
+ for block in blocks:
+ if not isinstance(block, dict):
+ continue
+ version = block.get("check_version") or block.get("summary", {}).get("checkov_version") or version
+ summary = block.get("summary") or {}
+ if "failed" in summary:
+ failed += int(summary["failed"] or 0)
+ else:
+ failed += len((block.get("results") or {}).get("failed_checks") or [])
+ return {"failed": failed}, version
+
+
+def _tfsec_counts(doc: Any) -> tuple[dict[str, int], str | None]:
+ results = doc.get("results") or []
+ severities = [str(r.get("severity", "")).upper() for r in results]
+ return (
+ {
+ "total": len(results),
+ "critical": sum(1 for s in severities if s == "CRITICAL"),
+ "high": sum(1 for s in severities if s == "HIGH"),
+ },
+ None,
+ )
+
+
+_COUNTERS = {"semgrep": _semgrep_counts, "checkov": _checkov_counts, "tfsec": _tfsec_counts}
+
+
+def count_findings(tool: str, doc: Any) -> tuple[dict[str, int], str | None]:
+ """Findings by bucket, plus the scanner version the report declares."""
+ counter = _COUNTERS.get(tool)
+ if counter is None:
+ raise ValueError(f"no counter for {tool!r}; known: {', '.join(sorted(_COUNTERS))}")
+ return counter(doc)
+
+
+# ── what the workflow actually runs ──────────────────────────────────────────
+
+
+def tools_invoked(workflow_text: str) -> set[str]:
+ """Which scanners ``security.yml`` runs, read from its `run:`/`uses:` lines."""
+ found: set[str] = set()
+ for line in workflow_text.splitlines():
+ stripped = line.strip()
+ if stripped.startswith("#"):
+ continue
+ for tool in KNOWN_TOOLS:
+ # An invocation, not a mention: the tool name at the head of a
+ # shell word, or in an action reference.
+ if re.search(rf"(^|[\s|&;/]){re.escape(tool)}([\s@-]|$)", stripped):
+ found.add(tool)
+ return found
+
+
+def _versions_pinned(workflow_text: str) -> dict[str, str]:
+ """The exact version each scanner is installed at, where one is pinned."""
+ pins: dict[str, str] = {}
+ for match in re.finditer(r"pip install[^\n]*?([a-z0-9_-]+)==([0-9][^\s'\"]*)", workflow_text):
+ pins[match.group(1)] = match.group(2)
+ return pins
+
+
+# ── the checks ───────────────────────────────────────────────────────────────
+
+
+def check_allowlist_entries(entries: list[Any], today: date) -> list[str]:
+ problems: list[str] = []
+ for index, entry in enumerate(entries):
+ where = f"{ALLOWLIST_REL} entries[{index}]"
+ if not isinstance(entry, dict):
+ problems.append(f"{where}: not a mapping")
+ continue
+ tool = entry.get("tool")
+ if tool not in KNOWN_TOOLS:
+ problems.append(f"{where}: tool {tool!r} is not one of {', '.join(KNOWN_TOOLS)}")
+ if not str(entry.get("id") or "").strip():
+ problems.append(f"{where}: no rule id")
+ if not str(entry.get("reason") or "").strip():
+ problems.append(f"{where}: no reason — a suppression nobody justified is a suppression nobody can review")
+ expires = str(entry.get("expires") or "")
+ try:
+ when = datetime.strptime(expires, "%Y-%m-%d").date()
+ except ValueError:
+ problems.append(f"{where}: expires {expires!r} is not YYYY-MM-DD")
+ continue
+ if when < today:
+ problems.append(f"{where}: expired on {when} — re-triage it or fix the finding, do not extend the date")
+ elif (when - today).days > MAX_EXPIRY_DAYS:
+ problems.append(f"{where}: expires {when}, more than {MAX_EXPIRY_DAYS} days out")
+ return problems
+
+
+def check_coverage(ceilings: dict[str, Any], invoked: set[str]) -> list[str]:
+ problems: list[str] = []
+ for tool in sorted(invoked - set(ceilings)):
+ problems.append(
+ f"{WORKFLOW_REL} runs {tool} and {ALLOWLIST_REL} declares no ceiling for it — an uncounted scanner is an observe-mode scanner"
+ )
+ for tool in sorted(set(ceilings) - invoked):
+ problems.append(f"{ALLOWLIST_REL} declares a ceiling for {tool} but {WORKFLOW_REL} does not run it — the ratchet guards nothing")
+ return problems
+
+
+def check_report(tool: str, ceiling: dict[str, Any], counts: dict[str, int], version: str | None) -> list[str]:
+ problems: list[str] = []
+ buckets = {k: v for k, v in ceiling.items() if isinstance(v, int)}
+ if not buckets:
+ return [f"{ALLOWLIST_REL}: ceilings.{tool} declares no integer bucket to compare against"]
+
+ measured_with = str(ceiling.get("measured_with") or "")
+ if version and measured_with and version != measured_with:
+ problems.append(
+ f"{tool}: this run used {version}, the ceiling was measured with {measured_with}. "
+ "Counts move when rules move — re-measure and update both, do not compare across versions."
+ )
+
+ for bucket, limit in sorted(buckets.items()):
+ actual = counts.get(bucket)
+ if actual is None:
+ problems.append(f"{tool}: ceiling names bucket {bucket!r}, the report has no such count")
+ continue
+ if actual > limit:
+ problems.append(f"{tool}: {actual} {bucket} findings against a ceiling of {limit} — this change adds {actual - limit}")
+ elif actual == 0 and limit > 0:
+ problems.append(
+ f"{tool}: 0 {bucket} findings against a ceiling of {limit}. A scanner that previously "
+ "reported findings and now reports none did not run — check the install step."
+ )
+ elif actual < limit:
+ problems.append(
+ f"{tool}: {actual} {bucket} findings against a ceiling of {limit}. "
+ f"Lower it to {actual} in this change, or the number drifts back up for free."
+ )
+ return problems
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description="Ratchet the observe-mode security scanners.")
+ parser.add_argument("--tool", choices=KNOWN_TOOLS, help="compare one scanner's report against its ceiling")
+ parser.add_argument("--report", type=Path, help="the scanner's JSON output")
+ parser.add_argument("--list", action="store_true", help="print the declared ceilings and exit")
+ parser.add_argument("--repo-root", type=Path, default=None)
+ args = parser.parse_args(argv)
+
+ root = args.repo_root.resolve() if args.repo_root else repo_root()
+ allowlist_path = root / ALLOWLIST_REL
+ workflow_path = root / WORKFLOW_REL
+
+ if not allowlist_path.is_file():
+ print(f"ERROR: {ALLOWLIST_REL} not found under {root}", file=sys.stderr)
+ return 2
+ if not workflow_path.is_file():
+ print(f"ERROR: {WORKFLOW_REL} not found under {root}", file=sys.stderr)
+ return 2
+
+ try:
+ doc = parse_allowlist(allowlist_path.read_text(encoding="utf-8"))
+ except AllowlistError as exc:
+ print(f"ERROR: {exc}", file=sys.stderr)
+ return 2
+
+ ceilings = doc.get("ceilings")
+ if not isinstance(ceilings, dict) or not ceilings:
+ print(f"ERROR: {ALLOWLIST_REL} declares no ceilings — nothing to ratchet", file=sys.stderr)
+ return 2
+ entries = doc.get("entries")
+ if entries is None or not isinstance(entries, list):
+ print(f"ERROR: {ALLOWLIST_REL} has no `entries:` list", file=sys.stderr)
+ return 2
+
+ workflow_text = workflow_path.read_text(encoding="utf-8")
+ invoked = tools_invoked(workflow_text)
+ if not invoked:
+ print(f"ERROR: parsed no scanner invocations out of {WORKFLOW_REL}", file=sys.stderr)
+ return 2
+
+ if args.list:
+ pins = _versions_pinned(workflow_text)
+ for tool in sorted(ceilings):
+ buckets = ", ".join(f"{k}={v}" for k, v in sorted(ceilings[tool].items()) if isinstance(v, int))
+ print(f" {tool:<8} {buckets} measured_with={ceilings[tool].get('measured_with', '—')} workflow pin={pins.get(tool, '—')}")
+ return 0
+
+ problems = check_allowlist_entries(entries, datetime.now(UTC).date())
+ problems += check_coverage(ceilings, invoked)
+
+ scanned = f"{len(ceilings)} ceiling(s), {len(entries)} allow-list entr(ies), {len(invoked)} scanner(s) invoked"
+
+ if args.tool:
+ if args.report is None:
+ print("ERROR: --tool needs --report", file=sys.stderr)
+ return 2
+ if not args.report.is_file():
+ print(f"ERROR: {args.report} not found — the scanner wrote no report, so there is nothing to ratchet", file=sys.stderr)
+ return 2
+ try:
+ report = json.loads(args.report.read_text(encoding="utf-8"))
+ except json.JSONDecodeError as exc:
+ print(f"ERROR: {args.report} is not JSON ({exc}) — the scanner did not finish", file=sys.stderr)
+ return 2
+ ceiling = ceilings.get(args.tool)
+ if not isinstance(ceiling, dict):
+ print(f"ERROR: no ceiling declared for {args.tool}", file=sys.stderr)
+ return 2
+ counts, version = count_findings(args.tool, report)
+ problems += check_report(args.tool, ceiling, counts, version)
+ measured = ", ".join(f"{k}={v}" for k, v in sorted(counts.items()))
+ scanned = f"{args.tool} {version or 'version unreported'}: {measured}"
+
+ if problems:
+ print(f"scanner-ratchet: {len(problems)} finding(s) — {scanned}", file=sys.stderr)
+ for problem in problems:
+ print(f" {problem}", file=sys.stderr)
+ return 1
+
+ print(f"scanner-ratchet: OK — {scanned}")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/tests/test_check_scanner_ratchet.py b/scripts/tests/test_check_scanner_ratchet.py
new file mode 100644
index 000000000..90774950c
--- /dev/null
+++ b/scripts/tests/test_check_scanner_ratchet.py
@@ -0,0 +1,254 @@
+"""The scanner ratchet fails in every direction it claims to.
+
+A ratchet that only catches findings going up is half a ratchet: the count
+drifts back down for free on an upstream rule change and the ceiling silently
+stops meaning anything. A ratchet that credits a scanner which did not run is
+worse than none. Both directions, and the vacuous case, are pinned here.
+
+The strict allow-list reader is checked against ``yaml.safe_load`` on the real
+file rather than trusted. It exists because the gate has to run in a job that
+installs ruff and mypy and nothing else, and a hand-written reader that
+disagrees with the parser everyone else uses would be its own defect.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import json
+import sys
+from datetime import date, timedelta
+from pathlib import Path
+
+import pytest
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+GATE = REPO_ROOT / "scripts" / "check_scanner_ratchet.py"
+ALLOWLIST = REPO_ROOT / ".security" / "allowlist.yml"
+
+
+def _load():
+ spec = importlib.util.spec_from_file_location("check_scanner_ratchet", GATE)
+ assert spec is not None and spec.loader is not None
+ module = importlib.util.module_from_spec(spec)
+ sys.modules[spec.name] = module
+ spec.loader.exec_module(module)
+ return module
+
+
+gate = _load()
+
+
+# ── the strict reader ────────────────────────────────────────────────────────
+
+
+def test_reader_agrees_with_pyyaml_on_the_real_file():
+ yaml = pytest.importorskip("yaml", reason="cross-check needs PyYAML; the gate itself never imports it")
+ text = ALLOWLIST.read_text(encoding="utf-8")
+ assert gate.parse_allowlist(text) == yaml.safe_load(text)
+
+
+def test_reader_refuses_what_it_cannot_represent():
+ with pytest.raises(gate.AllowlistError):
+ gate.parse_allowlist("ceilings:\n semgrep:\n total: not-a-number\n")
+ with pytest.raises(gate.AllowlistError):
+ gate.parse_allowlist("ceilings:\n\tsemgrep: 1\n")
+ with pytest.raises(gate.AllowlistError):
+ gate.parse_allowlist("ceilings\n")
+
+
+def test_reader_handles_the_entry_shape_the_schema_documents():
+ doc = gate.parse_allowlist(
+ 'entries:\n - tool: "semgrep"\n id: "python.lang.security.audit"\n reason: "accepted"\n expires: "2030-01-01"\n'
+ )
+ assert doc["entries"] == [{"tool": "semgrep", "id": "python.lang.security.audit", "reason": "accepted", "expires": "2030-01-01"}]
+
+
+# ── counting ─────────────────────────────────────────────────────────────────
+
+
+def test_semgrep_counts_split_error_from_total():
+ counts, version = gate.count_findings(
+ "semgrep",
+ {
+ "version": "1.86.0",
+ "results": [
+ {"extra": {"severity": "ERROR"}},
+ {"extra": {"severity": "WARNING"}},
+ {"extra": {"severity": "ERROR"}},
+ ],
+ },
+ )
+ assert counts == {"total": 3, "error": 2}
+ assert version == "1.86.0"
+
+
+def test_checkov_counts_sum_across_frameworks():
+ counts, _ = gate.count_findings(
+ "checkov",
+ [
+ {"check_type": "terraform", "summary": {"failed": 7}},
+ {"check_type": "helm", "summary": {"failed": 2}},
+ ],
+ )
+ assert counts == {"failed": 9}
+
+
+def test_checkov_counts_read_a_single_framework_object():
+ counts, _ = gate.count_findings("checkov", {"check_type": "terraform", "summary": {"failed": 4}})
+ assert counts == {"failed": 4}
+
+
+def test_tfsec_counts_split_by_severity():
+ counts, _ = gate.count_findings(
+ "tfsec",
+ {"results": [{"severity": "CRITICAL"}, {"severity": "HIGH"}, {"severity": "LOW"}]},
+ )
+ assert counts == {"total": 3, "critical": 1, "high": 1}
+
+
+def test_tfsec_null_results_is_zero_not_a_crash():
+ counts, _ = gate.count_findings("tfsec", {"results": None})
+ assert counts == {"total": 0, "critical": 0, "high": 0}
+
+
+# ── the three directions ─────────────────────────────────────────────────────
+
+CEILING = {"total": 10, "error": 3, "measured_with": "1.86.0"}
+
+
+def test_more_findings_than_the_ceiling_fails():
+ problems = gate.check_report("semgrep", CEILING, {"total": 11, "error": 3}, "1.86.0")
+ assert any("adds 1" in p for p in problems)
+
+
+def test_fewer_findings_than_the_ceiling_also_fails_and_names_the_new_value():
+ problems = gate.check_report("semgrep", CEILING, {"total": 9, "error": 3}, "1.86.0")
+ assert any("Lower it to 9" in p for p in problems)
+
+
+def test_zero_findings_against_a_real_ceiling_is_a_scanner_that_did_not_run():
+ problems = gate.check_report("semgrep", CEILING, {"total": 0, "error": 0}, "1.86.0")
+ assert any("did not run" in p for p in problems)
+
+
+def test_matching_counts_pass():
+ assert gate.check_report("semgrep", CEILING, {"total": 10, "error": 3}, "1.86.0") == []
+
+
+def test_a_different_scanner_version_is_not_a_comparison():
+ problems = gate.check_report("semgrep", CEILING, {"total": 10, "error": 3}, "1.99.0")
+ assert any("measured with 1.86.0" in p for p in problems)
+
+
+def test_a_bucket_the_report_does_not_carry_fails_rather_than_being_skipped():
+ problems = gate.check_report("semgrep", CEILING, {"total": 10}, "1.86.0")
+ assert any("no such count" in p for p in problems)
+
+
+# ── allow-list hygiene ───────────────────────────────────────────────────────
+
+TODAY = date(2026, 9, 28)
+
+
+def _entry(**overrides):
+ base = {
+ "tool": "semgrep",
+ "id": "python.lang.security.audit.eval-detected",
+ "reason": "the expression is a fixed literal, not user input",
+ "expires": (TODAY + timedelta(days=30)).isoformat(),
+ }
+ base.update(overrides)
+ return base
+
+
+def test_a_well_formed_entry_passes():
+ assert gate.check_allowlist_entries([_entry()], TODAY) == []
+
+
+def test_an_entry_with_no_reason_fails():
+ problems = gate.check_allowlist_entries([_entry(reason="")], TODAY)
+ assert any("no reason" in p for p in problems)
+
+
+def test_an_expired_entry_fails():
+ problems = gate.check_allowlist_entries([_entry(expires="2020-01-01")], TODAY)
+ assert any("expired" in p for p in problems)
+
+
+def test_an_expiry_further_out_than_the_cap_fails():
+ far = (TODAY + timedelta(days=gate.MAX_EXPIRY_DAYS + 1)).isoformat()
+ problems = gate.check_allowlist_entries([_entry(expires=far)], TODAY)
+ assert any("days out" in p for p in problems)
+
+
+def test_an_entry_naming_a_tool_nothing_runs_fails():
+ problems = gate.check_allowlist_entries([_entry(tool="bandit")], TODAY)
+ assert any("not one of" in p for p in problems)
+
+
+# ── coverage, in both directions ─────────────────────────────────────────────
+
+
+def test_a_scanner_with_no_ceiling_fails():
+ problems = gate.check_coverage({"semgrep": {}}, {"semgrep", "checkov"})
+ assert any("runs checkov" in p and "declares no ceiling" in p for p in problems)
+
+
+def test_a_ceiling_for_a_scanner_nothing_runs_fails():
+ problems = gate.check_coverage({"semgrep": {}, "tfsec": {}}, {"semgrep"})
+ assert any("guards nothing" in p for p in problems)
+
+
+def test_coverage_is_clean_when_both_sides_agree():
+ assert gate.check_coverage({"semgrep": {}, "checkov": {}}, {"semgrep", "checkov"}) == []
+
+
+# ── the workflow reader ──────────────────────────────────────────────────────
+
+
+def test_the_real_workflow_invokes_every_tool_the_allowlist_ceils():
+ workflow = (REPO_ROOT / ".github" / "workflows" / "security.yml").read_text(encoding="utf-8")
+ invoked = gate.tools_invoked(workflow)
+ ceilings = gate.parse_allowlist(ALLOWLIST.read_text(encoding="utf-8"))["ceilings"]
+ assert set(ceilings) == invoked
+
+
+def test_a_tool_only_mentioned_in_a_comment_is_not_an_invocation():
+ assert gate.tools_invoked("# we should add semgrep one day\n") == set()
+
+
+def test_every_ceiling_pins_the_version_the_workflow_installs():
+ """A ceiling measured with one version and compared against another is not
+ a comparison, so the pin and the ``measured_with`` have to agree."""
+ workflow = (REPO_ROOT / ".github" / "workflows" / "security.yml").read_text(encoding="utf-8")
+ pins = gate._versions_pinned(workflow)
+ ceilings = gate.parse_allowlist(ALLOWLIST.read_text(encoding="utf-8"))["ceilings"]
+ for tool, pinned in pins.items():
+ if tool in ceilings:
+ assert ceilings[tool].get("measured_with") == pinned, f"{tool} pinned at {pinned}"
+
+
+# ── end to end, through the command line ─────────────────────────────────────
+
+
+def test_the_gate_reds_on_a_regressed_report(tmp_path):
+ report = tmp_path / "semgrep.json"
+ report.write_text(
+ json.dumps({"version": "1.86.0", "results": [{"extra": {"severity": "ERROR"}}] * 200}),
+ encoding="utf-8",
+ )
+ assert gate.main(["--tool", "semgrep", "--report", str(report)]) == 1
+
+
+def test_a_missing_report_is_a_scan_that_did_not_finish(tmp_path):
+ assert gate.main(["--tool", "semgrep", "--report", str(tmp_path / "absent.json")]) == 2
+
+
+def test_a_truncated_report_is_a_scan_that_did_not_finish(tmp_path):
+ report = tmp_path / "semgrep.json"
+ report.write_text('{"results": [', encoding="utf-8")
+ assert gate.main(["--tool", "semgrep", "--report", str(report)]) == 2
+
+
+def test_the_gate_passes_on_the_real_tree_with_no_report():
+ assert gate.main([]) == 0