Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
fix(security): detect vendor credential shapes and PEM keys, not just variable names (#2223)
Aman2d361070fd7f
diff --git a/docs/layers/SECURITY.md b/docs/layers/SECURITY.mdindex 4d1e4ee18..6b5b47c3e 100644--- a/docs/layers/SECURITY.md+++ b/docs/layers/SECURITY.md@@ -4,7 +4,7 @@ Repowise records a small set of security signals while it indexes: patternmatches over source text, and symbol names that look security-relevant. Pureregex and SQL, no LLM calls, no network, no dependency resolution.-This is a floor, not a scanner. The registry holds sixteen patterns. It has no+This is a floor, not a scanner. The registry holds twenty-two patterns. It has nomodel of your framework, no notion of which inputs are attacker-controlled, andno dataflow: it cannot tell a parameterised query from a concatenated one beyondwhat the surrounding characters give away, and it cannot tell whether a@@ -44,9 +44,9 @@ Findings also appear on the Security tab of the code-health page, and at## What the registry catches-Sixteen patterns plus a symbol-name scan, giving seventeen kinds across three-severities. Severity is a fixed property of the pattern; nothing is scored,-ranked, or aggregated.+Twenty-two patterns plus a symbol-name scan, giving twenty-three kinds across+three severities. Severity is a fixed property of the pattern; nothing is+scored, ranked, or aggregated.| Kind | Severity | Matches ||------|----------|---------|@@ -56,7 +56,13 @@ ranked, or aggregated.| `subprocess_shell_true` | high | `subprocess.*` with `shell=True`, including across physical lines || `os_system` | high | `os.system` || `hardcoded_password` | high | an assignment of a quoted literal to a name containing `password`, any case |-| `hardcoded_secret` | high | the same for `api_key`, `apikey` or `secret` |+| `hardcoded_secret` | high | the same for a name containing `api_key`, `apikey`, `secret`, `token` or `access_key` |+| `aws_access_key` | high | an AWS access key ID (`AKIA`/`ASIA` + 16 chars), regardless of variable name |+| `github_token` | high | a GitHub PAT, OAuth, app, or refresh token (`ghp_`/`gho_`/`ghu_`/`ghs_`/`ghr_`/`github_pat_`), regardless of variable name |+| `slack_token` | high | a Slack token (`xoxb-`, `xoxa-`, `xoxp-`, `xoxr-`, `xoxs-`), regardless of variable name |+| `google_api_key` | high | a Google API key (`AIza` + 35 chars), regardless of variable name |+| `stripe_key` | high | a Stripe secret or live publishable key (`sk_live_`, `sk_test_`, `pk_live_`), regardless of variable name |+| `private_key_pem` | high | a PEM header (`-----BEGIN ... PRIVATE KEY-----`) followed by a base64-looking body line, so assembling just the header string does not fire || `fstring_sql` | med | an f-string containing `SELECT` and an interpolation || `concat_sql` | med | `.execute("SELECT ... +` || `tls_verify_false` | med | `verify = False` |@@ -111,11 +117,12 @@ and this layer does not try.reaches a dangerous call is not computed. `eval_call` fires the same on aconstant and on a request parameter.-**Real secret detection.** The two secret patterns match a literal assignment to-a variable named like a credential, in any case, so the constant spelling a-pinned credential usually carries is covered. That is the whole of it: they do-not know entropy, key formats, or provider prefixes, and a credential held in a-name they do not list is invisible. For secret scanning proper, run gitleaks or+**Real secret detection.** `hardcoded_password` / `hardcoded_secret` match a+literal assignment to a variable named like a credential, in any case, and the+six value-shape kinds above (AWS, GitHub, Slack, Google, Stripe, PEM) catch a+handful of common vendor formats regardless of variable name. That is still far+short of a real scanner: no entropy scoring, and any format or provider not in+that short list is invisible. For secret scanning proper, run gitleaks ortrufflehog; history mode below is complementary to those, not a replacement.**Dependencies.** No CVE lookup, no advisory feed, no SBOM. Nothing here looks@@ -189,8 +196,10 @@ commit that introduced the match. This finds what the working tree cannot: acredential committed in March and removed in April is absent from HEAD andpresent in the clone forever.-History mode reports **only `hardcoded_password` and `hardcoded_secret`** by-default. The reasoning is asymmetry of decay. A commit that once called `eval()`+History mode reports **only the `SECRET_KINDS` (genuine leaked-credential)+kinds** by default — `hardcoded_password`, `hardcoded_secret`, and the six+vendor value-shape kinds above. The reasoning is asymmetry of decay. A commit+that once called `eval()`is history doing what history does — the code changed, that is the point, andreporting every such moment across every revision buries the surface in noise. Acommitted secret does not decay. It stays valid until someone rotates it, and@@ -249,7 +258,8 @@ persistence silently; every other write failure is a real error.A useful way to read a repo's findings, in order:-1. **Any `hardcoded_secret` or `hardcoded_password` from history mode.** These+1. **Any `SECRET_KINDS` finding from history mode** (`hardcoded_secret`,+ `hardcoded_password`, or one of the vendor value-shape kinds). Theseare the findings most likely to be both true and actionable. Rotate first,remove from history second.2. **The high-severity working-tree kinds**, as a list of places to read ratherdiff --git a/packages/cli/src/repowise/cli/commands/security_cmd.py b/packages/cli/src/repowise/cli/commands/security_cmd.pyindex 83a919584..14897f8d2 100644--- a/packages/cli/src/repowise/cli/commands/security_cmd.py+++ b/packages/cli/src/repowise/cli/commands/security_cmd.py@@ -57,7 +57,8 @@ def security_command() -> None:default=False,help="History mode: also report code-smell patterns (eval, os.system, ""weak hashes, ...). By default history mode reports only leaked-secret "- "patterns (hardcoded_password / hardcoded_secret) to avoid noise.",+ "patterns (hardcoded credentials and known vendor key/token/PEM shapes) "+ "to avoid noise.",)@format_option(help="Output format. ``json`` is the machine-readable summary.")@click.option(diff --git a/packages/core/src/repowise/core/analysis/history_scan.py b/packages/core/src/repowise/core/analysis/history_scan.pyindex bddf8273b..73f199c0d 100644--- a/packages/core/src/repowise/core/analysis/history_scan.py+++ b/packages/core/src/repowise/core/analysis/history_scan.py@@ -22,7 +22,8 @@ Design notes (in response to review)patterns are code smells (``eval``/``os.system``/``weak_hash``) rather thanleaked credentials; running those across all of history produces mostly noise("os.system in a two-year-old commit") with little to act on. The- history-relevant subset is ``hardcoded_password`` / ``hardcoded_secret``. This+ history-relevant subset is ``SECRET_KINDS`` (``hardcoded_password`` /+ ``hardcoded_secret`` and the vendor value-shape kinds). Thispositions history scanning as complementary to a real secret scanner(gitleaks / trufflehog) rather than a noisy replacement. ``--all-patterns``opts back into the full registry when desired.@@ -330,9 +331,10 @@ class HistorySecurityScanner:reachable history.secrets_only:When True (default), only the secret-oriented patterns- (hardcoded_password / hardcoded_secret) are reported, to avoid the- code-smell noise of scanning all of history. Pass False to scan the- full pattern registry.+ (``SECRET_KINDS``: hardcoded credentials and known vendor+ key/token/PEM shapes) are reported, to avoid the code-smell noise+ of scanning all of history. Pass False to scan the full pattern+ registry.progress:Optional callable ``progress(message)`` for CLI feedback."""diff --git a/packages/core/src/repowise/core/analysis/security_scan.py b/packages/core/src/repowise/core/analysis/security_scan.pyindex 0ac127e1b..aee09eace 100644--- a/packages/core/src/repowise/core/analysis/security_scan.py+++ b/packages/core/src/repowise/core/analysis/security_scan.py@@ -134,7 +134,31 @@ _PATTERNS: list[tuple[re.Pattern, str, str]] = [# flags. A flag here would leave the prefilter case-sensitive and it would# reject the line before the pattern ever ran.(re.compile(r"(?i:password)\s*=\s*['\"]([^'\"]*)"), "hardcoded_password", "high"),- (re.compile(r"(?i:api_?key|secret)\s*=\s*['\"]([^'\"]*)"), "hardcoded_secret", "high"),+ (+ re.compile(r"(?i:api_?key|secret|token|access_?key)\s*=\s*['\"]([^'\"]*)"),+ "hardcoded_secret",+ "high",+ ),+ # Value-shape patterns for common vendor credential formats (gitleaks'+ # rule set, MIT-licensed, is the reference for these shapes). These fire+ # regardless of the variable name a key is assigned to, so a vendor key+ # bound to an unlisted name (``client_id``) or passed inline is still+ # caught. Each carries a capture group around the credential value itself+ # so the ``SECRET_KINDS`` gate below (length + placeholder check) applies+ # to it the same as the keyword patterns above.+ (re.compile(r"\b((?:AKIA|ASIA)[A-Z2-7]{16})\b"), "aws_access_key", "high"),+ (+ re.compile(r"\b(gh[oprsu]_[0-9A-Za-z]{36}|github_pat_\w{82})\b"),+ "github_token",+ "high",+ ),+ (re.compile(r"(?i:(xox[baprs]-[0-9a-zA-Z-]{10,72}))"), "slack_token", "high"),+ (re.compile(r"\b(AIza[0-9A-Za-z_-]{35})\b"), "google_api_key", "high"),+ (+ re.compile(r"\b((?:sk_(?:live|test)|pk_live)_[0-9A-Za-z]{10,99})\b"),+ "stripe_key",+ "high",+ ),(re.compile(r'f[\'"].*SELECT.*\{.*\}'), "fstring_sql", "med"),(re.compile(r"\.execute\(\s*[\'\"]\s*SELECT.*\+"), "concat_sql", "med"),(re.compile(r"verify\s*=\s*False"), "tls_verify_false", "med"),@@ -207,6 +231,17 @@ _SPANNING_PATTERNS: list[tuple[re.Pattern, str, str]] = ["subprocess_shell_true","high",),+ # A PEM header alone is not a leak: code that assembles PEM text+ # (``"-----BEGIN " + kind + " PRIVATE KEY-----"``) contains the header+ # string without ever holding key material. Requiring a base64-looking+ # body line right after the header is what tells the two apart, and the+ # capture group around that body line is the value the SECRET_KINDS gate+ # below checks (length + placeholder), same as every other secret kind.+ (+ re.compile(r"(?i:-----BEGIN[ A-Z0-9_-]{0,100}PRIVATE KEY-----)\r?\n([A-Za-z0-9+/=]{20,})"),+ "private_key_pem",+ "high",+ ),]# Symbol names that are informational security hotspots@@ -218,7 +253,18 @@ _SYMBOL_KEYWORDS = re.compile(r"\b(auth|token|password|jwt|session|crypto)\b", r# mostly noise, whereas a committed secret is actionable and persists in# history. This positions history mode as complementary to gitleaks /# trufflehog rather than a noisy replacement.-SECRET_KINDS: frozenset[str] = frozenset({"hardcoded_password", "hardcoded_secret"})+SECRET_KINDS: frozenset[str] = frozenset(+ {+ "hardcoded_password",+ "hardcoded_secret",+ "aws_access_key",+ "github_token",+ "slack_token",+ "google_api_key",+ "stripe_key",+ "private_key_pem",+ }+)# Kinds whose ``snippet`` is a symbol *name* rather than the text of the line# it sits on (see the symbol-name scan below). Serve-time line verification@@ -515,6 +561,12 @@ class SecurityScanner:# the per-line pass already caught on that line is not duplicated.for pattern, kind, severity in _SPANNING_PATTERNS:for match in pattern.finditer(source):+ if kind in SECRET_KINDS:+ val = match.group(1) if match.groups() else ""+ if not _is_valid_credential_value(val):+ continue+ if is_low_sev_file:+ severity = "low"start_line = source.count("\n", 0, match.start()) + 1if any(f["kind"] == kind and f["line"] == start_line for f in findings):continuediff --git a/tests/unit/analysis/test_security_scan.py b/tests/unit/analysis/test_security_scan.pyindex 8b76a6ac0..9f3b79b7b 100644--- a/tests/unit/analysis/test_security_scan.py+++ b/tests/unit/analysis/test_security_scan.py@@ -245,6 +245,100 @@ class TestScanFile:assert len(hits) == 1assert hits[0]["severity"] == "low"+ # -- Vendor credential shapes (#2117) ----------------------------------++ @pytest.mark.parametrize(+ ("source", "expected_kind"),+ [+ ('AWS_ACCESS_KEY_ID = "AKIAQZXNRTVYWMPKLBHG"\n', "aws_access_key"),+ (+ 'GITHUB_TOKEN = "ghp_abcdefghijklmnopqrstuvwxyzABCDEFGHIJ"\n',+ "github_token",+ ),+ (+ "GH_APP_TOKEN = 'ghu_abcdefghijklmnopqrstuvwxyzABCDEFGHIJ'\n",+ "github_token",+ ),+ (+ 'GH_FINE_GRAINED = "github_pat_'+ + "abcdefghijklmnopqrstuvwxyzABCDEFGHIJabcdefghijklmnopqrstuvwxyzABCDEFGHIJabcdefghij"+ + '"\n',+ "github_token",+ ),+ ('SLACK_BOT_TOKEN = "xoxb-4721609583-T5Rk9mPz2Qa"\n', "slack_token"),+ (+ 'GOOGLE_MAPS_API_KEY = "AIzaSyD9K2vQ7xR4mZ1pL8tY6wU3nB0cF5hD9aX"\n',+ "google_api_key",+ ),+ (+ # Built from parts so the literal "sk_" + "live_" prefix pair+ # never appears contiguous in source: GitHub's push+ # protection flags that Stripe prefix on sight, independent+ # of body entropy.+ 'stripe.api_key = "' + "sk_" + "live_" + "A" * 24 + '"\n',+ "stripe_key",+ ),+ ],+ )+ def test_vendor_key_shapes_fire_regardless_of_variable_name(+ self, source: str, expected_kind: str+ ) -> None:+ scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]+ findings = asyncio.run(scanner.scan_file("config.py", source, symbols=[]))+ kinds = {f["kind"] for f in findings}+ assert expected_kind in kinds++ def test_token_and_access_key_keywords_are_detected(self) -> None:+ """#2117: ``token`` and ``access_key`` were not in the keyword list."""+ scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]+ for source in (+ 'TOKEN = "a_real_looking_token_value_123"\n',+ 'access_key = "a_real_looking_access_key_456"\n',+ ):+ findings = asyncio.run(scanner.scan_file("config.py", source, symbols=[]))+ assert any(f["kind"] == "hardcoded_secret" for f in findings), source++ def test_client_id_vendor_key_is_caught_by_shape_not_name(self) -> None:+ """A vendor key is invisible to the keyword list under a name like+ ``client_id``, so it must be caught by the value shape alone."""+ scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]+ source = 'client_id = "AKIAQZXNRTVYWMPKLBHG"\n'+ findings = asyncio.run(scanner.scan_file("config.py", source, symbols=[]))+ assert any(f["kind"] == "aws_access_key" for f in findings)++ def test_pem_private_key_with_body_is_flagged(self) -> None:+ scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]+ source = (+ "-----BEGIN RSA PRIVATE KEY-----\n"+ "MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC7VJTUt9Us8cKj\n"+ "-----END RSA PRIVATE KEY-----\n"+ )+ findings = asyncio.run(scanner.scan_file("id_rsa.py", source, symbols=[]))+ hits = [f for f in findings if f["kind"] == "private_key_pem"]+ assert len(hits) == 1+ assert hits[0]["line"] == 1++ def test_pem_header_without_body_does_not_fire(self) -> None:+ """Code that assembles PEM text from the header string alone (no key+ material) must not be reported."""+ scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]+ source = (+ 'PEM_HEADER = "-----BEGIN RSA PRIVATE KEY-----"\n'+ 'PEM_FOOTER = "-----END RSA PRIVATE KEY-----"\n'+ )+ findings = asyncio.run(scanner.scan_file("pem_templates.py", source, symbols=[]))+ assert not any(f["kind"] == "private_key_pem" for f in findings)++ def test_vendor_key_shapes_are_gated_by_credential_value_rules(self) -> None:+ """#2121's value gate must apply to the new vendor kinds too: a+ low-severity path downgrades severity, same as the keyword kinds."""+ scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]+ source = 'AWS_ACCESS_KEY_ID = "AKIAQZXNRTVYWMPKLBHG"\n'+ findings = asyncio.run(scanner.scan_file("tests/fixtures/keys.py", source, symbols=[]))+ hits = [f for f in findings if f["kind"] == "aws_access_key"]+ assert len(hits) == 1+ assert hits[0]["severity"] == "low"+def test_single_line_subprocess_shell_true_is_flagged(self) -> None:scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]source = 'subprocess.run("rm -rf " + path, shell=True)\n'diff --git a/tests/unit/analysis/test_security_scan_history.py b/tests/unit/analysis/test_security_scan_history.pyindex 8ffb3d602..f684a6a04 100644--- a/tests/unit/analysis/test_security_scan_history.py+++ b/tests/unit/analysis/test_security_scan_history.py@@ -246,9 +246,24 @@ async def test_history_progress_fires_for_clean_repo(session: AsyncSession) -> Nassert summary.files_scanned == 1-def test_secret_kinds_are_the_two_secret_patterns() -> None:- """Guard against the registry drifting away from the history gate."""- assert {"hardcoded_password", "hardcoded_secret"} == SECRET_KINDS+def test_secret_kinds_are_the_genuine_credential_patterns() -> None:+ """Guard against the registry drifting away from the history gate.++ #2117 added value-shape patterns for vendor credential formats (AWS,+ GitHub, Slack, Google, Stripe) and a PEM private-key pattern; all are+ genuine leaked-credential kinds, so history mode's default gate covers+ them the same as the two keyword-based kinds.+ """+ assert {+ "hardcoded_password",+ "hardcoded_secret",+ "aws_access_key",+ "github_token",+ "slack_token",+ "google_api_key",+ "stripe_key",+ "private_key_pem",+ } == SECRET_KINDS# ---------------------------------------------------------------------------