Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(gates): give the scanner allow-list a reader and the scanners a ratchet (#987)
Beenu Aroraf55be02ff6bc
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 336f02f1c..a6ae869c6 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -570,6 +570,16 @@ jobs:run: python3 scripts/check_replay_contract_parity.py --self-test- name: Replay contract parity gaterun: python3 scripts/check_replay_contract_parity.py+ # The finding-count comparison itself lives in the three scanner jobs in+ # security.yml, where the reports are. This runs the half that needs no+ # scanner — the allow-list's own schema, its expiry dates, and whether+ # the set of ceilings still matches the set of scanners the workflow+ # runs — in a job that is a required check, so a ceiling left behind by+ # a deleted scanner cannot hide in a job nobody waits for.+ - name: Scanner ratchet gate — self-test+ run: python3 scripts/check_scanner_ratchet.py --self-test+ - name: Scanner ratchet gate+ run: python3 scripts/check_scanner_ratchet.py# The three sdk-* jobs below compile and test each SDK in isolation, and# none of them compares a client to the API it claims to speak to.# packages/sdk-go passed all three over a file of type declarations withdiff --git a/.github/workflows/security.yml b/.github/workflows/security.ymlindex 9556fa020..f851447b7 100644--- a/.github/workflows/security.yml+++ b/.github/workflows/security.yml@@ -15,15 +15,18 @@ name: Security# the pinned release had been deleted upstream, so the download# 404'd, the scan ran a binary that was not there, and# continue-on-error turned the whole thing green.-# - OBSERVE gates (report, do not fail yet): Semgrep, checkov, tfsec.-# These are not observe-mode because nobody has looked. They were run-# against this tree and the counts are: Semgrep 92 (35 at ERROR),-# checkov 91 failed checks under infra/terraform, tfsec 32 (7 CRITICAL,-# 5 HIGH). Blocking on 215 untriaged findings would red every pull+# - RATCHETED gates (report every finding, fail on a new one): Semgrep,+# checkov, tfsec. Blocking on 215 untriaged findings would red every pull# request on day one and the first response would be to turn the gate-# back off, so each needs its own triage pass first. The count is stated-# here rather than left as "we'll get to it" so the size of the-# remaining work is visible.+# back off, so none of the three fails on its existing findings. What+# they now fail on is a *change* in the count, in either direction,+# against the ceilings in `.security/allowlist.yml` — enforced by+# `scripts/check_scanner_ratchet.py`, which is the first thing in this+# repository ever to open that file. The counts used to live in this+# comment, where nothing compared them to anything: a pull request adding+# twenty findings and one removing twenty produced the same green tick.+# They now live in the allow-list, where a run that disagrees with them+# fails and prints the number to write.## GitHub native push-protection remains the always-on hard secret gate.@@ -110,14 +113,21 @@ jobs:- name: Scan working tree (respects .gitleaksignore)run: gitleaks dir . --redact --exit-code 1 --no-banner- # ── OBSERVE: SAST beyond CodeQL ───────────────────────────────────────────+ # ── RATCHETED: SAST beyond CodeQL ─────────────────────────────────────────#- # 92 findings against this tree (35 ERROR, 51 WARNING, 3 MEDIUM, 3 INFO),- # measured with semgrep 1.86.0 and the five configs below. Too many to- # block on before they are triaged, and `--error` fails on any of them, so- # this stays report-only with the number written down.+ # 92 findings against this tree (35 ERROR, 51 WARNING, 3 MEDIUM, 3 INFO).+ # Too many to block on before they are triaged, and `--error` fails on any+ # of them — so the scan itself stays non-blocking and the *count* is what+ # blocks, against the ceiling in `.security/allowlist.yml`.+ #+ # The version is pinned exactly. For a ratchet the scanner version is the+ # behaviour: an unpinned `pip install semgrep` re-baselines the count on+ # upstream's release schedule, which is indistinguishable from a change in+ # this tree. The gate compares the version the report declares against the+ # one the ceiling was measured with and fails if they differ, so a bump+ # here is an explicit re-measure rather than a silent one.semgrep:- name: Semgrep (observe)+ name: Semgrep (ratcheted)runs-on: ubuntu-lateststeps:- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7@@ -125,11 +135,14 @@ jobs:with:python-version: "3.11"- name: Install semgrep- run: pip install --quiet semgrep+ run: pip install --quiet semgrep==1.86.0+ # `--error` is gone: it exits non-zero on any finding, which would make+ # the exit status carry the same information as the count and force+ # `continue-on-error` back onto the ratchet step below with it.- name: Scancontinue-on-error: truerun: |- semgrep --error --disable-version-check \+ semgrep --json --output semgrep.json --disable-version-check \--config p/security-audit \--config p/owasp-top-ten \--config p/python \@@ -137,6 +150,10 @@ jobs:--config p/react \--exclude plans/ \services/ apps/ packages/ scripts/+ # Blocking. No `continue-on-error`: a missing or truncated report means+ # the scan did not finish, and the gate says so rather than passing.+ - name: Ratchet the finding count+ run: python3 scripts/check_scanner_ratchet.py --tool semgrep --report semgrep.json# ── HARD: filesystem vuln scan (complements dependency SCA) ───────────────trivy-fs:@@ -180,15 +197,28 @@ jobs:set -euo pipefail/tmp/trivy fs --severity HIGH,CRITICAL --ignore-unfixed --skip-dirs plans --exit-code 1 .- # ── OBSERVE: IaC misconfig scan ───────────────────────────────────────────+ # ── RATCHETED: IaC misconfig scan ─────────────────────────────────────────+ #+ # checkov reports 91 failed checks against infra/terraform (164 passing);+ # tfsec reports 32 findings there (7 CRITICAL, 5 HIGH, 2 MEDIUM, 18 LOW).+ # Neither scan fails on those — promoting them outright would red every+ # pull request over pre-existing infrastructure defaults that no single+ # change introduced. What fails is a count that no longer matches the+ # ceiling in `.security/allowlist.yml`, in either direction.#- # checkov 3.2.x reports 91 failed checks against infra/terraform (164- # passing); tfsec 1.28.x reports 32 findings there (7 CRITICAL, 5 HIGH,- # 2 MEDIUM, 18 LOW). Both stay report-only until those are triaged —- # promoting them now would red every pull request over pre-existing- # infrastructure defaults that no single change introduced.- iac-scan:- name: IaC scan (checkov + tfsec, observe)+ # tfsec runs as a pinned binary rather than through the marketplace action,+ # for the two reasons the Trivy job above already records: the action's+ # version tags proved unreliable, and the ratchet needs the JSON report on+ # disk. `--soft-fail` keeps the scan's own exit status out of it so the+ # count is the only thing that decides.+ # One job each, not one job for both. They shared a job while neither+ # blocked, and the moment the ratchet made the steps hard the first+ # scanner to breach its ceiling ended the job before the second one ran —+ # so a real count for tfsec was unobtainable whenever checkov was red.+ # A gate that can stop another gate from measuring is a gate that reports+ # on whichever half happened to run first.+ iac-checkov:+ name: IaC scan (checkov, ratcheted)runs-on: ubuntu-lateststeps:- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7@@ -198,11 +228,34 @@ jobs:- name: checkov (terraform + helm)continue-on-error: truerun: |- pip install --quiet checkov- checkov -d infra/terraform --quiet --compact || true- checkov -d infra/helm --quiet --compact || true- - name: tfsec (terraform)- continue-on-error: true- uses: aquasecurity/tfsec-action@b466648d6e39e7c75324f25d83891162a721f2d6 # v1.0.3+ pip install --quiet checkov==3.2.334+ # `--output-file-path .` writes ./results_json.json; checkov treats+ # a directory argument that way rather than honouring a filename.+ checkov -d infra/terraform -d infra/helm --quiet --compact \+ -o json --output-file-path . || true+ - name: Ratchet the checkov finding count+ run: python3 scripts/check_scanner_ratchet.py --tool checkov --report results_json.json++ iac-tfsec:+ name: IaC scan (tfsec, ratcheted)+ runs-on: ubuntu-latest+ steps:+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7+ - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0with:- working_directory: infra/terraform+ python-version: "3.11"+ - name: Install tfsec+ run: |+ set -euo pipefail+ VERSION=1.28.13+ url="https://github.com/aquasecurity/tfsec/releases/download/v${VERSION}/tfsec-linux-amd64"+ if ! curl -sSL --fail --retry 5 --retry-delay 5 --retry-all-errors "$url" -o /tmp/tfsec; then+ echo "::error title=tfsec::could not download $url — check whether v${VERSION} still exists and bump the pin"+ exit 1+ fi+ chmod +x /tmp/tfsec+ /tmp/tfsec --version+ - name: tfsec (terraform)+ run: /tmp/tfsec infra/terraform --format json --out tfsec.json --soft-fail+ - name: Ratchet the tfsec finding count+ run: python3 scripts/check_scanner_ratchet.py --tool tfsec --report tfsec.jsondiff --git a/.security/allowlist.yml b/.security/allowlist.ymlindex 96a9a4612..b59ec8ca7 100644--- a/.security/allowlist.yml+++ b/.security/allowlist.yml@@ -1,20 +1,66 @@-# Security scanner allowlist (Phase 2)+# Security scanner ceilings and suppressions.#-# Suppressions for the Semgrep / Trivy / checkov / tfsec scanners in-# .github/workflows/security.yml. Every entry MUST carry a reason and an-# expiry date; expired entries fail CI once the scanners flip from observe to-# blocking (tracked Phase 2 continuation). This is separate from the-# dependency-CVE ignore list at scripts/security_audit_ignores.txt.+# Read by scripts/check_scanner_ratchet.py, which runs as a blocking step in+# each of the three observe-mode scanner jobs in .github/workflows/security.yml.+# Until that gate was written this file had no reader at all: the workflow+# described its scanners as "report-and-ratchet" and named this path as where+# the triaged baseline lives, and nothing opened it. An allow-list nobody+# reads is worse than no allow-list, because it looks like a control.#-# Schema:+# ── ceilings ────────────────────────────────────────────────────────────────+#+# A scanner may report at most this many findings. The ratchet is shrink-only:+# reporting fewer also fails, with the measured number in the message, so the+# ceiling comes down in the same change that removes the finding rather than+# leaving room for the count to drift back up for free. Reporting zero against+# a non-zero ceiling fails too — a scanner that previously found dozens and+# now finds none did not run, and "clean" must never look like "did not run".+#+# `measured_with` is compared against the version the report declares. Counts+# move when rules move, so a ceiling compared against a different scanner is+# not a comparison; the scanners are pinned exactly in the workflow for the+# same reason, because for a ratchet the scanner version is the behaviour.+#+# To lower a ceiling: fix findings, read the measured count out of the gate's+# failure message, and write it here in the same pull request.++ceilings:+ # Every number here was measured by this gate in CI, not carried over.+ # The counts the workflow header used to publish were 92/35 for semgrep and+ # 91 for checkov, and both were wrong the first time the ratchet compared+ # them: semgrep came back 101/40 because the tree has grown since anyone+ # looked, and checkov came back 219 because the 91 covered infra/terraform+ # while the job had always scanned infra/helm as well and reported only the+ # first number. That is the whole argument for the ratchet in one run — a+ # count nothing compares to anything drifts, and nobody finds out.+ semgrep:+ total: 101+ error: 40+ measured_with: "1.86.0"+ checkov:+ failed: 219+ measured_with: "3.2.334"+ tfsec:+ total: 32+ critical: 7+ high: 5+ measured_with: "1.28.13"++# ── suppressions ────────────────────────────────────────────────────────────+#+# Individual findings accepted as risk. Separate from the dependency-CVE+# ignore list at scripts/security_audit_ignores.txt.+#+# Schema, enforced by the gate:# entries:-# - tool: semgrep|trivy|checkov|tfsec+# - tool: semgrep|checkov|tfsec# id: <rule/check id># path: <optional path glob>-# reason: <why this is acceptable> # required, non-empty-# expires: YYYY-MM-DD # required, <= 90 days out+# reason: <why this is acceptable> # required, non-empty+# expires: YYYY-MM-DD # required, at most 90 days out#-# Keep this list empty until a scanner is ratcheted to blocking; add triaged-# baseline findings here at that time, never blanket-suppress.+# An expired entry fails the build. Re-triage it or fix the finding; do not+# extend the date. Prefer lowering a ceiling over adding an entry here —+# a suppression hides a finding, a ceiling counts it.entries: []diff --git a/scripts/check_scanner_ratchet.py b/scripts/check_scanner_ratchet.pynew file mode 100644index 000000000..ead73331d--- /dev/null+++ b/scripts/check_scanner_ratchet.py@@ -0,0 +1,447 @@+#!/usr/bin/env python3+"""The observe-mode scanners get a ratchet, and `.security/allowlist.yml` gets a reader.++Why this exists+---------------++``.github/workflows/security.yml`` describes its three observe-mode scanners+as "report-and-ratchet" and names ``.security/allowlist.yml`` as where the+triaged baseline lives. Both halves of that sentence were untrue.++There was no ratchet. Semgrep, checkov and tfsec each ran under+``continue-on-error: true`` — checkov additionally with a trailing ``|| true``+— printed a number, and nothing compared that number to anything. A pull+request that added twenty findings and one that removed twenty produced the+same green tick, so the counts written into the workflow header ("Semgrep 92,+35 at ERROR; checkov 91 failed; tfsec 32, 7 CRITICAL, 5 HIGH") were a+measurement taken once and then left to rot.++And ``.security/allowlist.yml`` had **no reader anywhere in the tree**. A+repository-wide search for its path returned two documents talking *about* it+and not one line of code that opens it. Its header declares a schema, requires+a reason and an expiry on every entry, and says expired entries "fail CI" —+enforced by nothing. An allow-list nobody reads is worse than no allow-list:+it looks like a control, so the next person assumes the triage happened.++``scripts/check_gate_coverage.py`` would have caught this had the orphan been+a *script*. It inventories ``scripts/`` against the workflows. A control file+in YAML with no caller is the same defect one file type over, and nothing was+looking for it.++What it checks+--------------++``regression``+ A scanner reports more findings than its recorded ceiling. This is the+ property the word "ratchet" promises and the only one that protects the+ tree from accumulating new findings.++``stale-ceiling``+ A scanner reports *fewer* findings than its ceiling. Shrink-only: the+ ceiling must come down in the same change that removes the finding, or+ the number drifts back up for free and the ratchet is decorative. The+ failure prints the measured count, so one CI run tells you the value to+ write.++``vacuous``+ Zero findings against a non-zero ceiling. A scanner that reports nothing+ where it previously reported dozens did not run. This repository has the+ scar: the Trivy pin 404'd, ``tar`` failed, a binary that was not there was+ invoked, and ``continue-on-error`` reported green. "Clean" and "did not+ run" must never look the same.++``version-drift``+ The scanner version that produced this report is not the version the+ ceiling was measured with. Counts move when rules move, so a ceiling+ compared against a different scanner is not a comparison. Failing here+ turns a silent re-baseline into an explicit, actionable one.++``allowlist``+ Every entry carries a non-empty reason and an expiry no more than 90 days+ out, names a tool that is actually run, and has not expired. The file's+ own header has required this since it was written; this is the first+ thing to enforce it.++``coverage`` (both directions)+ Every scanner ``security.yml`` invokes has a ceiling, **and** every+ ceiling names a scanner ``security.yml`` invokes. One direction alone is+ the failure shape this repository keeps rediscovering: a gate that+ compares A against B and never B against A passes while drift+ accumulates in the direction things actually change. Here the drift that+ matters is a scanner being removed from the workflow while its ceiling+ stays behind, which would leave a ratchet guarding nothing.++What it reads+-------------++``.security/allowlist.yml`` for the ceilings and the entries, parsed by a+strict reader over the documented subset rather than by PyYAML — the gate has+to run in ``Python — Lint & Type-check``, which installs ruff and mypy and+nothing else. The reader refuses anything it does not recognise rather than+skipping it, and ``scripts/tests/test_check_scanner_ratchet.py`` asserts it+agrees with ``yaml.safe_load`` on the real file so the two cannot diverge.++``.github/workflows/security.yml`` for which scanners are actually invoked.++The scanner's own JSON report, when ``--tool``/``--report`` are given.++Usage+-----++::++ python3 scripts/check_scanner_ratchet.py # allow-list + coverage+ python3 scripts/check_scanner_ratchet.py --list # print the ceilings+ python3 scripts/check_scanner_ratchet.py --tool semgrep --report semgrep.json+ python3 scripts/check_scanner_ratchet.py --self-test++Exit codes: 0 clean, 1 findings, 2 the check itself could not run.+"""++from __future__ import annotations++import argparse+import json+import re+import sys+from datetime import UTC, date, datetime+from pathlib import Path+from typing import Any++# `scripts/` is on sys.path when this file is run as a program, but not when a+# test loads it by path with importlib. gate_toolkit sits beside it either way.+sys.path.insert(0, str(Path(__file__).resolve().parent))++from gate_toolkit import repo_root, self_test_if_requested++self_test_if_requested(__file__)++ALLOWLIST_REL = Path(".security") / "allowlist.yml"+WORKFLOW_REL = Path(".github") / "workflows" / "security.yml"++#: How far out an allow-list expiry may sit. The file's own header states 90+#: days; this is that sentence, enforced.+MAX_EXPIRY_DAYS = 90++#: The scanners this gate knows how to count. A tool invoked by the workflow+#: that is missing here fails `coverage` rather than being skipped, because a+#: scanner nobody counts is the state this gate exists to end.+KNOWN_TOOLS = ("semgrep", "checkov", "tfsec")+++class AllowlistError(ValueError):+ """The allow-list could not be read as the schema its header declares."""+++# ── the strict reader ────────────────────────────────────────────────────────+#+# The documented subset is: top-level keys; two levels of nesting under+# `ceilings:`; scalars that are integers or quoted strings; and `entries:` as+# either `[]` or a list of flat `- key: value` mappings. Anything else raises.+++def _scalar(raw: str, where: str) -> Any:+ text = raw.strip()+ if text.startswith(('"', "'")) and text.endswith(text[0]) and len(text) >= 2:+ return text[1:-1]+ if re.fullmatch(r"-?\d+", text):+ return int(text)+ if text in {"true", "false"}:+ return text == "true"+ if text == "[]":+ return []+ raise AllowlistError(f"{where}: cannot read {raw!r} — quote strings, and use plain integers for counts")+++def parse_allowlist(text: str) -> dict[str, Any]:+ """Read the documented subset of the allow-list. Refuse the rest."""+ root: dict[str, Any] = {}+ stack: list[tuple[int, Any]] = [(-1, root)]+ for lineno, raw in enumerate(text.splitlines(), start=1):+ if not raw.strip() or raw.lstrip().startswith("#"):+ continue+ indent = len(raw) - len(raw.lstrip(" "))+ line = raw.strip()+ where = f"{ALLOWLIST_REL}:{lineno}"++ if "\t" in raw:+ raise AllowlistError(f"{where}: tab indentation is not readable as YAML")++ while stack and indent <= stack[-1][0]:+ stack.pop()+ if not stack:+ raise AllowlistError(f"{where}: indentation does not nest under anything")+ parent = stack[-1][1]++ if line.startswith("- "):+ if not isinstance(parent, list):+ raise AllowlistError(f"{where}: list item under a mapping")+ item: dict[str, Any] = {}+ parent.append(item)+ key, _, value = line[2:].partition(":")+ if not _:+ raise AllowlistError(f"{where}: list items must be `- key: value` mappings")+ item[key.strip()] = _scalar(value, where)+ # A list item's own keys sit at the column the `-` opened, plus+ # two: `- tool: x` then ` id: y`. Both belong to the same item.+ stack.append((indent + 1, item))+ continue++ key, sep, value = line.partition(":")+ if not sep:+ raise AllowlistError(f"{where}: expected `key: value`, got {line!r}")+ key = key.strip()+ if not value.strip():+ child: dict[str, Any] | list[Any] = [] if key == "entries" else {}+ if isinstance(parent, dict):+ parent[key] = child+ else:+ raise AllowlistError(f"{where}: mapping key inside a list item without a value")+ stack.append((indent, child))+ continue+ if isinstance(parent, dict):+ parent[key] = _scalar(value, where)+ else:+ raise AllowlistError(f"{where}: `{key}` is not inside a mapping")+ return root+++# ── counting each scanner's report ───────────────────────────────────────────+++def _semgrep_counts(doc: Any) -> tuple[dict[str, int], str | None]:+ results = doc.get("results") or []+ severities = [str((r.get("extra") or {}).get("severity", "")).upper() for r in results]+ return (+ {"total": len(results), "error": sum(1 for s in severities if s == "ERROR")},+ doc.get("version"),+ )+++def _checkov_counts(doc: Any) -> tuple[dict[str, int], str | None]:+ # checkov emits a list when several frameworks ran and a bare object when+ # one did. Both shapes reach this gate, so both are read.+ blocks = doc if isinstance(doc, list) else [doc]+ failed = 0+ version: str | None = None+ for block in blocks:+ if not isinstance(block, dict):+ continue+ version = block.get("check_version") or block.get("summary", {}).get("checkov_version") or version+ summary = block.get("summary") or {}+ if "failed" in summary:+ failed += int(summary["failed"] or 0)+ else:+ failed += len((block.get("results") or {}).get("failed_checks") or [])+ return {"failed": failed}, version+++def _tfsec_counts(doc: Any) -> tuple[dict[str, int], str | None]:+ results = doc.get("results") or []+ severities = [str(r.get("severity", "")).upper() for r in results]+ return (+ {+ "total": len(results),+ "critical": sum(1 for s in severities if s == "CRITICAL"),+ "high": sum(1 for s in severities if s == "HIGH"),+ },+ None,+ )+++_COUNTERS = {"semgrep": _semgrep_counts, "checkov": _checkov_counts, "tfsec": _tfsec_counts}+++def count_findings(tool: str, doc: Any) -> tuple[dict[str, int], str | None]:+ """Findings by bucket, plus the scanner version the report declares."""+ counter = _COUNTERS.get(tool)+ if counter is None:+ raise ValueError(f"no counter for {tool!r}; known: {', '.join(sorted(_COUNTERS))}")+ return counter(doc)+++# ── what the workflow actually runs ──────────────────────────────────────────+++def tools_invoked(workflow_text: str) -> set[str]:+ """Which scanners ``security.yml`` runs, read from its `run:`/`uses:` lines."""+ found: set[str] = set()+ for line in workflow_text.splitlines():+ stripped = line.strip()+ if stripped.startswith("#"):+ continue+ for tool in KNOWN_TOOLS:+ # An invocation, not a mention: the tool name at the head of a+ # shell word, or in an action reference.+ if re.search(rf"(^|[\s|&;/]){re.escape(tool)}([\s@-]|$)", stripped):+ found.add(tool)+ return found+++def _versions_pinned(workflow_text: str) -> dict[str, str]:+ """The exact version each scanner is installed at, where one is pinned."""+ pins: dict[str, str] = {}+ for match in re.finditer(r"pip install[^\n]*?([a-z0-9_-]+)==([0-9][^\s'\"]*)", workflow_text):+ pins[match.group(1)] = match.group(2)+ return pins+++# ── the checks ───────────────────────────────────────────────────────────────+++def check_allowlist_entries(entries: list[Any], today: date) -> list[str]:+ problems: list[str] = []+ for index, entry in enumerate(entries):+ where = f"{ALLOWLIST_REL} entries[{index}]"+ if not isinstance(entry, dict):+ problems.append(f"{where}: not a mapping")+ continue+ tool = entry.get("tool")+ if tool not in KNOWN_TOOLS:+ problems.append(f"{where}: tool {tool!r} is not one of {', '.join(KNOWN_TOOLS)}")+ if not str(entry.get("id") or "").strip():+ problems.append(f"{where}: no rule id")+ if not str(entry.get("reason") or "").strip():+ problems.append(f"{where}: no reason — a suppression nobody justified is a suppression nobody can review")+ expires = str(entry.get("expires") or "")+ try:+ when = datetime.strptime(expires, "%Y-%m-%d").date()+ except ValueError:+ problems.append(f"{where}: expires {expires!r} is not YYYY-MM-DD")+ continue+ if when < today:+ problems.append(f"{where}: expired on {when} — re-triage it or fix the finding, do not extend the date")+ elif (when - today).days > MAX_EXPIRY_DAYS:+ problems.append(f"{where}: expires {when}, more than {MAX_EXPIRY_DAYS} days out")+ return problems+++def check_coverage(ceilings: dict[str, Any], invoked: set[str]) -> list[str]:+ problems: list[str] = []+ for tool in sorted(invoked - set(ceilings)):+ problems.append(+ f"{WORKFLOW_REL} runs {tool} and {ALLOWLIST_REL} declares no ceiling for it — an uncounted scanner is an observe-mode scanner"+ )+ for tool in sorted(set(ceilings) - invoked):+ problems.append(f"{ALLOWLIST_REL} declares a ceiling for {tool} but {WORKFLOW_REL} does not run it — the ratchet guards nothing")+ return problems+++def check_report(tool: str, ceiling: dict[str, Any], counts: dict[str, int], version: str | None) -> list[str]:+ problems: list[str] = []+ buckets = {k: v for k, v in ceiling.items() if isinstance(v, int)}+ if not buckets:+ return [f"{ALLOWLIST_REL}: ceilings.{tool} declares no integer bucket to compare against"]++ measured_with = str(ceiling.get("measured_with") or "")+ if version and measured_with and version != measured_with:+ problems.append(+ f"{tool}: this run used {version}, the ceiling was measured with {measured_with}. "+ "Counts move when rules move — re-measure and update both, do not compare across versions."+ )++ for bucket, limit in sorted(buckets.items()):+ actual = counts.get(bucket)+ if actual is None:+ problems.append(f"{tool}: ceiling names bucket {bucket!r}, the report has no such count")+ continue+ if actual > limit:+ problems.append(f"{tool}: {actual} {bucket} findings against a ceiling of {limit} — this change adds {actual - limit}")+ elif actual == 0 and limit > 0:+ problems.append(+ f"{tool}: 0 {bucket} findings against a ceiling of {limit}. A scanner that previously "+ "reported findings and now reports none did not run — check the install step."+ )+ elif actual < limit:+ problems.append(+ f"{tool}: {actual} {bucket} findings against a ceiling of {limit}. "+ f"Lower it to {actual} in this change, or the number drifts back up for free."+ )+ return problems+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description="Ratchet the observe-mode security scanners.")+ parser.add_argument("--tool", choices=KNOWN_TOOLS, help="compare one scanner's report against its ceiling")+ parser.add_argument("--report", type=Path, help="the scanner's JSON output")+ parser.add_argument("--list", action="store_true", help="print the declared ceilings and exit")+ parser.add_argument("--repo-root", type=Path, default=None)+ args = parser.parse_args(argv)++ root = args.repo_root.resolve() if args.repo_root else repo_root()+ allowlist_path = root / ALLOWLIST_REL+ workflow_path = root / WORKFLOW_REL++ if not allowlist_path.is_file():+ print(f"ERROR: {ALLOWLIST_REL} not found under {root}", file=sys.stderr)+ return 2+ if not workflow_path.is_file():+ print(f"ERROR: {WORKFLOW_REL} not found under {root}", file=sys.stderr)+ return 2++ try:+ doc = parse_allowlist(allowlist_path.read_text(encoding="utf-8"))+ except AllowlistError as exc:+ print(f"ERROR: {exc}", file=sys.stderr)+ return 2++ ceilings = doc.get("ceilings")+ if not isinstance(ceilings, dict) or not ceilings:+ print(f"ERROR: {ALLOWLIST_REL} declares no ceilings — nothing to ratchet", file=sys.stderr)+ return 2+ entries = doc.get("entries")+ if entries is None or not isinstance(entries, list):+ print(f"ERROR: {ALLOWLIST_REL} has no `entries:` list", file=sys.stderr)+ return 2++ workflow_text = workflow_path.read_text(encoding="utf-8")+ invoked = tools_invoked(workflow_text)+ if not invoked:+ print(f"ERROR: parsed no scanner invocations out of {WORKFLOW_REL}", file=sys.stderr)+ return 2++ if args.list:+ pins = _versions_pinned(workflow_text)+ for tool in sorted(ceilings):+ buckets = ", ".join(f"{k}={v}" for k, v in sorted(ceilings[tool].items()) if isinstance(v, int))+ print(f" {tool:<8} {buckets} measured_with={ceilings[tool].get('measured_with', '—')} workflow pin={pins.get(tool, '—')}")+ return 0++ problems = check_allowlist_entries(entries, datetime.now(UTC).date())+ problems += check_coverage(ceilings, invoked)++ scanned = f"{len(ceilings)} ceiling(s), {len(entries)} allow-list entr(ies), {len(invoked)} scanner(s) invoked"++ if args.tool:+ if args.report is None:+ print("ERROR: --tool needs --report", file=sys.stderr)+ return 2+ if not args.report.is_file():+ print(f"ERROR: {args.report} not found — the scanner wrote no report, so there is nothing to ratchet", file=sys.stderr)+ return 2+ try:+ report = json.loads(args.report.read_text(encoding="utf-8"))+ except json.JSONDecodeError as exc:+ print(f"ERROR: {args.report} is not JSON ({exc}) — the scanner did not finish", file=sys.stderr)+ return 2+ ceiling = ceilings.get(args.tool)+ if not isinstance(ceiling, dict):+ print(f"ERROR: no ceiling declared for {args.tool}", file=sys.stderr)+ return 2+ counts, version = count_findings(args.tool, report)+ problems += check_report(args.tool, ceiling, counts, version)+ measured = ", ".join(f"{k}={v}" for k, v in sorted(counts.items()))+ scanned = f"{args.tool} {version or 'version unreported'}: {measured}"++ if problems:+ print(f"scanner-ratchet: {len(problems)} finding(s) — {scanned}", file=sys.stderr)+ for problem in problems:+ print(f" {problem}", file=sys.stderr)+ return 1++ print(f"scanner-ratchet: OK — {scanned}")+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/scripts/tests/test_check_scanner_ratchet.py b/scripts/tests/test_check_scanner_ratchet.pynew file mode 100644index 000000000..90774950c--- /dev/null+++ b/scripts/tests/test_check_scanner_ratchet.py@@ -0,0 +1,254 @@+"""The scanner ratchet fails in every direction it claims to.++A ratchet that only catches findings going up is half a ratchet: the count+drifts back down for free on an upstream rule change and the ceiling silently+stops meaning anything. A ratchet that credits a scanner which did not run is+worse than none. Both directions, and the vacuous case, are pinned here.++The strict allow-list reader is checked against ``yaml.safe_load`` on the real+file rather than trusted. It exists because the gate has to run in a job that+installs ruff and mypy and nothing else, and a hand-written reader that+disagrees with the parser everyone else uses would be its own defect.+"""++from __future__ import annotations++import importlib.util+import json+import sys+from datetime import date, timedelta+from pathlib import Path++import pytest++REPO_ROOT = Path(__file__).resolve().parents[2]+GATE = REPO_ROOT / "scripts" / "check_scanner_ratchet.py"+ALLOWLIST = REPO_ROOT / ".security" / "allowlist.yml"+++def _load():+ spec = importlib.util.spec_from_file_location("check_scanner_ratchet", GATE)+ assert spec is not None and spec.loader is not None+ module = importlib.util.module_from_spec(spec)+ sys.modules[spec.name] = module+ spec.loader.exec_module(module)+ return module+++gate = _load()+++# ── the strict reader ────────────────────────────────────────────────────────+++def test_reader_agrees_with_pyyaml_on_the_real_file():+ yaml = pytest.importorskip("yaml", reason="cross-check needs PyYAML; the gate itself never imports it")+ text = ALLOWLIST.read_text(encoding="utf-8")+ assert gate.parse_allowlist(text) == yaml.safe_load(text)+++def test_reader_refuses_what_it_cannot_represent():+ with pytest.raises(gate.AllowlistError):+ gate.parse_allowlist("ceilings:\n semgrep:\n total: not-a-number\n")+ with pytest.raises(gate.AllowlistError):+ gate.parse_allowlist("ceilings:\n\tsemgrep: 1\n")+ with pytest.raises(gate.AllowlistError):+ gate.parse_allowlist("ceilings\n")+++def test_reader_handles_the_entry_shape_the_schema_documents():+ doc = gate.parse_allowlist(+ 'entries:\n - tool: "semgrep"\n id: "python.lang.security.audit"\n reason: "accepted"\n expires: "2030-01-01"\n'+ )+ assert doc["entries"] == [{"tool": "semgrep", "id": "python.lang.security.audit", "reason": "accepted", "expires": "2030-01-01"}]+++# ── counting ─────────────────────────────────────────────────────────────────+++def test_semgrep_counts_split_error_from_total():+ counts, version = gate.count_findings(+ "semgrep",+ {+ "version": "1.86.0",+ "results": [+ {"extra": {"severity": "ERROR"}},+ {"extra": {"severity": "WARNING"}},+ {"extra": {"severity": "ERROR"}},+ ],+ },+ )+ assert counts == {"total": 3, "error": 2}+ assert version == "1.86.0"+++def test_checkov_counts_sum_across_frameworks():+ counts, _ = gate.count_findings(+ "checkov",+ [+ {"check_type": "terraform", "summary": {"failed": 7}},+ {"check_type": "helm", "summary": {"failed": 2}},+ ],+ )+ assert counts == {"failed": 9}+++def test_checkov_counts_read_a_single_framework_object():+ counts, _ = gate.count_findings("checkov", {"check_type": "terraform", "summary": {"failed": 4}})+ assert counts == {"failed": 4}+++def test_tfsec_counts_split_by_severity():+ counts, _ = gate.count_findings(+ "tfsec",+ {"results": [{"severity": "CRITICAL"}, {"severity": "HIGH"}, {"severity": "LOW"}]},+ )+ assert counts == {"total": 3, "critical": 1, "high": 1}+++def test_tfsec_null_results_is_zero_not_a_crash():+ counts, _ = gate.count_findings("tfsec", {"results": None})+ assert counts == {"total": 0, "critical": 0, "high": 0}+++# ── the three directions ─────────────────────────────────────────────────────++CEILING = {"total": 10, "error": 3, "measured_with": "1.86.0"}+++def test_more_findings_than_the_ceiling_fails():+ problems = gate.check_report("semgrep", CEILING, {"total": 11, "error": 3}, "1.86.0")+ assert any("adds 1" in p for p in problems)+++def test_fewer_findings_than_the_ceiling_also_fails_and_names_the_new_value():+ problems = gate.check_report("semgrep", CEILING, {"total": 9, "error": 3}, "1.86.0")+ assert any("Lower it to 9" in p for p in problems)+++def test_zero_findings_against_a_real_ceiling_is_a_scanner_that_did_not_run():+ problems = gate.check_report("semgrep", CEILING, {"total": 0, "error": 0}, "1.86.0")+ assert any("did not run" in p for p in problems)+++def test_matching_counts_pass():+ assert gate.check_report("semgrep", CEILING, {"total": 10, "error": 3}, "1.86.0") == []+++def test_a_different_scanner_version_is_not_a_comparison():+ problems = gate.check_report("semgrep", CEILING, {"total": 10, "error": 3}, "1.99.0")+ assert any("measured with 1.86.0" in p for p in problems)+++def test_a_bucket_the_report_does_not_carry_fails_rather_than_being_skipped():+ problems = gate.check_report("semgrep", CEILING, {"total": 10}, "1.86.0")+ assert any("no such count" in p for p in problems)+++# ── allow-list hygiene ───────────────────────────────────────────────────────++TODAY = date(2026, 9, 28)+++def _entry(**overrides):+ base = {+ "tool": "semgrep",+ "id": "python.lang.security.audit.eval-detected",+ "reason": "the expression is a fixed literal, not user input",+ "expires": (TODAY + timedelta(days=30)).isoformat(),+ }+ base.update(overrides)+ return base+++def test_a_well_formed_entry_passes():+ assert gate.check_allowlist_entries([_entry()], TODAY) == []+++def test_an_entry_with_no_reason_fails():+ problems = gate.check_allowlist_entries([_entry(reason="")], TODAY)+ assert any("no reason" in p for p in problems)+++def test_an_expired_entry_fails():+ problems = gate.check_allowlist_entries([_entry(expires="2020-01-01")], TODAY)+ assert any("expired" in p for p in problems)+++def test_an_expiry_further_out_than_the_cap_fails():+ far = (TODAY + timedelta(days=gate.MAX_EXPIRY_DAYS + 1)).isoformat()+ problems = gate.check_allowlist_entries([_entry(expires=far)], TODAY)+ assert any("days out" in p for p in problems)+++def test_an_entry_naming_a_tool_nothing_runs_fails():+ problems = gate.check_allowlist_entries([_entry(tool="bandit")], TODAY)+ assert any("not one of" in p for p in problems)+++# ── coverage, in both directions ─────────────────────────────────────────────+++def test_a_scanner_with_no_ceiling_fails():+ problems = gate.check_coverage({"semgrep": {}}, {"semgrep", "checkov"})+ assert any("runs checkov" in p and "declares no ceiling" in p for p in problems)+++def test_a_ceiling_for_a_scanner_nothing_runs_fails():+ problems = gate.check_coverage({"semgrep": {}, "tfsec": {}}, {"semgrep"})+ assert any("guards nothing" in p for p in problems)+++def test_coverage_is_clean_when_both_sides_agree():+ assert gate.check_coverage({"semgrep": {}, "checkov": {}}, {"semgrep", "checkov"}) == []+++# ── the workflow reader ──────────────────────────────────────────────────────+++def test_the_real_workflow_invokes_every_tool_the_allowlist_ceils():+ workflow = (REPO_ROOT / ".github" / "workflows" / "security.yml").read_text(encoding="utf-8")+ invoked = gate.tools_invoked(workflow)+ ceilings = gate.parse_allowlist(ALLOWLIST.read_text(encoding="utf-8"))["ceilings"]+ assert set(ceilings) == invoked+++def test_a_tool_only_mentioned_in_a_comment_is_not_an_invocation():+ assert gate.tools_invoked("# we should add semgrep one day\n") == set()+++def test_every_ceiling_pins_the_version_the_workflow_installs():+ """A ceiling measured with one version and compared against another is not+ a comparison, so the pin and the ``measured_with`` have to agree."""+ workflow = (REPO_ROOT / ".github" / "workflows" / "security.yml").read_text(encoding="utf-8")+ pins = gate._versions_pinned(workflow)+ ceilings = gate.parse_allowlist(ALLOWLIST.read_text(encoding="utf-8"))["ceilings"]+ for tool, pinned in pins.items():+ if tool in ceilings:+ assert ceilings[tool].get("measured_with") == pinned, f"{tool} pinned at {pinned}"+++# ── end to end, through the command line ─────────────────────────────────────+++def test_the_gate_reds_on_a_regressed_report(tmp_path):+ report = tmp_path / "semgrep.json"+ report.write_text(+ json.dumps({"version": "1.86.0", "results": [{"extra": {"severity": "ERROR"}}] * 200}),+ encoding="utf-8",+ )+ assert gate.main(["--tool", "semgrep", "--report", str(report)]) == 1+++def test_a_missing_report_is_a_scan_that_did_not_finish(tmp_path):+ assert gate.main(["--tool", "semgrep", "--report", str(tmp_path / "absent.json")]) == 2+++def test_a_truncated_report_is_a_scan_that_did_not_finish(tmp_path):+ report = tmp_path / "semgrep.json"+ report.write_text('{"results": [', encoding="utf-8")+ assert gate.main(["--tool", "semgrep", "--report", str(report)]) == 2+++def test_the_gate_passes_on_the_real_tree_with_no_report():+ assert gate.main([]) == 0