Skip to content

Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.

fix(security): detect vendor credential shapes and PEM keys, not just variable names (#2223)

Aman2d361070fd7f
diff --git a/docs/layers/SECURITY.md b/docs/layers/SECURITY.md
index 4d1e4ee18..6b5b47c3e 100644
--- a/docs/layers/SECURITY.md
+++ b/docs/layers/SECURITY.md
@@ -4,7 +4,7 @@ Repowise records a small set of security signals while it indexes: pattern
matches over source text, and symbol names that look security-relevant. Pure
regex and SQL, no LLM calls, no network, no dependency resolution.
-This is a floor, not a scanner. The registry holds sixteen patterns. It has no
+This is a floor, not a scanner. The registry holds twenty-two patterns. It has no
model of your framework, no notion of which inputs are attacker-controlled, and
no dataflow: it cannot tell a parameterised query from a concatenated one beyond
what the surrounding characters give away, and it cannot tell whether a
@@ -44,9 +44,9 @@ Findings also appear on the Security tab of the code-health page, and at
## What the registry catches
-Sixteen patterns plus a symbol-name scan, giving seventeen kinds across three
-severities. Severity is a fixed property of the pattern; nothing is scored,
-ranked, or aggregated.
+Twenty-two patterns plus a symbol-name scan, giving twenty-three kinds across
+three severities. Severity is a fixed property of the pattern; nothing is
+scored, ranked, or aggregated.
| Kind | Severity | Matches |
|------|----------|---------|
@@ -56,7 +56,13 @@ ranked, or aggregated.
| `subprocess_shell_true` | high | `subprocess.*` with `shell=True`, including across physical lines |
| `os_system` | high | `os.system` |
| `hardcoded_password` | high | an assignment of a quoted literal to a name containing `password`, any case |
-| `hardcoded_secret` | high | the same for `api_key`, `apikey` or `secret` |
+| `hardcoded_secret` | high | the same for a name containing `api_key`, `apikey`, `secret`, `token` or `access_key` |
+| `aws_access_key` | high | an AWS access key ID (`AKIA`/`ASIA` + 16 chars), regardless of variable name |
+| `github_token` | high | a GitHub PAT, OAuth, app, or refresh token (`ghp_`/`gho_`/`ghu_`/`ghs_`/`ghr_`/`github_pat_`), regardless of variable name |
+| `slack_token` | high | a Slack token (`xoxb-`, `xoxa-`, `xoxp-`, `xoxr-`, `xoxs-`), regardless of variable name |
+| `google_api_key` | high | a Google API key (`AIza` + 35 chars), regardless of variable name |
+| `stripe_key` | high | a Stripe secret or live publishable key (`sk_live_`, `sk_test_`, `pk_live_`), regardless of variable name |
+| `private_key_pem` | high | a PEM header (`-----BEGIN ... PRIVATE KEY-----`) followed by a base64-looking body line, so assembling just the header string does not fire |
| `fstring_sql` | med | an f-string containing `SELECT` and an interpolation |
| `concat_sql` | med | `.execute("SELECT ... +` |
| `tls_verify_false` | med | `verify = False` |
@@ -111,11 +117,12 @@ and this layer does not try.
reaches a dangerous call is not computed. `eval_call` fires the same on a
constant and on a request parameter.
-**Real secret detection.** The two secret patterns match a literal assignment to
-a variable named like a credential, in any case, so the constant spelling a
-pinned credential usually carries is covered. That is the whole of it: they do
-not know entropy, key formats, or provider prefixes, and a credential held in a
-name they do not list is invisible. For secret scanning proper, run gitleaks or
+**Real secret detection.** `hardcoded_password` / `hardcoded_secret` match a
+literal assignment to a variable named like a credential, in any case, and the
+six value-shape kinds above (AWS, GitHub, Slack, Google, Stripe, PEM) catch a
+handful of common vendor formats regardless of variable name. That is still far
+short of a real scanner: no entropy scoring, and any format or provider not in
+that short list is invisible. For secret scanning proper, run gitleaks or
trufflehog; history mode below is complementary to those, not a replacement.
**Dependencies.** No CVE lookup, no advisory feed, no SBOM. Nothing here looks
@@ -189,8 +196,10 @@ commit that introduced the match. This finds what the working tree cannot: a
credential committed in March and removed in April is absent from HEAD and
present in the clone forever.
-History mode reports **only `hardcoded_password` and `hardcoded_secret`** by
-default. The reasoning is asymmetry of decay. A commit that once called `eval()`
+History mode reports **only the `SECRET_KINDS` (genuine leaked-credential)
+kinds** by default — `hardcoded_password`, `hardcoded_secret`, and the six
+vendor value-shape kinds above. The reasoning is asymmetry of decay. A commit
+that once called `eval()`
is history doing what history does — the code changed, that is the point, and
reporting every such moment across every revision buries the surface in noise. A
committed secret does not decay. It stays valid until someone rotates it, and
@@ -249,7 +258,8 @@ persistence silently; every other write failure is a real error.
A useful way to read a repo's findings, in order:
-1. **Any `hardcoded_secret` or `hardcoded_password` from history mode.** These
+1. **Any `SECRET_KINDS` finding from history mode** (`hardcoded_secret`,
+ `hardcoded_password`, or one of the vendor value-shape kinds). These
are the findings most likely to be both true and actionable. Rotate first,
remove from history second.
2. **The high-severity working-tree kinds**, as a list of places to read rather
diff --git a/packages/cli/src/repowise/cli/commands/security_cmd.py b/packages/cli/src/repowise/cli/commands/security_cmd.py
index 83a919584..14897f8d2 100644
--- a/packages/cli/src/repowise/cli/commands/security_cmd.py
+++ b/packages/cli/src/repowise/cli/commands/security_cmd.py
@@ -57,7 +57,8 @@ def security_command() -> None:
default=False,
help="History mode: also report code-smell patterns (eval, os.system, "
"weak hashes, ...). By default history mode reports only leaked-secret "
- "patterns (hardcoded_password / hardcoded_secret) to avoid noise.",
+ "patterns (hardcoded credentials and known vendor key/token/PEM shapes) "
+ "to avoid noise.",
)
@format_option(help="Output format. ``json`` is the machine-readable summary.")
@click.option(
diff --git a/packages/core/src/repowise/core/analysis/history_scan.py b/packages/core/src/repowise/core/analysis/history_scan.py
index bddf8273b..73f199c0d 100644
--- a/packages/core/src/repowise/core/analysis/history_scan.py
+++ b/packages/core/src/repowise/core/analysis/history_scan.py
@@ -22,7 +22,8 @@ Design notes (in response to review)
patterns are code smells (``eval``/``os.system``/``weak_hash``) rather than
leaked credentials; running those across all of history produces mostly noise
("os.system in a two-year-old commit") with little to act on. The
- history-relevant subset is ``hardcoded_password`` / ``hardcoded_secret``. This
+ history-relevant subset is ``SECRET_KINDS`` (``hardcoded_password`` /
+ ``hardcoded_secret`` and the vendor value-shape kinds). This
positions history scanning as complementary to a real secret scanner
(gitleaks / trufflehog) rather than a noisy replacement. ``--all-patterns``
opts back into the full registry when desired.
@@ -330,9 +331,10 @@ class HistorySecurityScanner:
reachable history.
secrets_only:
When True (default), only the secret-oriented patterns
- (hardcoded_password / hardcoded_secret) are reported, to avoid the
- code-smell noise of scanning all of history. Pass False to scan the
- full pattern registry.
+ (``SECRET_KINDS``: hardcoded credentials and known vendor
+ key/token/PEM shapes) are reported, to avoid the code-smell noise
+ of scanning all of history. Pass False to scan the full pattern
+ registry.
progress:
Optional callable ``progress(message)`` for CLI feedback.
"""
diff --git a/packages/core/src/repowise/core/analysis/security_scan.py b/packages/core/src/repowise/core/analysis/security_scan.py
index 0ac127e1b..aee09eace 100644
--- a/packages/core/src/repowise/core/analysis/security_scan.py
+++ b/packages/core/src/repowise/core/analysis/security_scan.py
@@ -134,7 +134,31 @@ _PATTERNS: list[tuple[re.Pattern, str, str]] = [
# flags. A flag here would leave the prefilter case-sensitive and it would
# reject the line before the pattern ever ran.
(re.compile(r"(?i:password)\s*=\s*['\"]([^'\"]*)"), "hardcoded_password", "high"),
- (re.compile(r"(?i:api_?key|secret)\s*=\s*['\"]([^'\"]*)"), "hardcoded_secret", "high"),
+ (
+ re.compile(r"(?i:api_?key|secret|token|access_?key)\s*=\s*['\"]([^'\"]*)"),
+ "hardcoded_secret",
+ "high",
+ ),
+ # Value-shape patterns for common vendor credential formats (gitleaks'
+ # rule set, MIT-licensed, is the reference for these shapes). These fire
+ # regardless of the variable name a key is assigned to, so a vendor key
+ # bound to an unlisted name (``client_id``) or passed inline is still
+ # caught. Each carries a capture group around the credential value itself
+ # so the ``SECRET_KINDS`` gate below (length + placeholder check) applies
+ # to it the same as the keyword patterns above.
+ (re.compile(r"\b((?:AKIA|ASIA)[A-Z2-7]{16})\b"), "aws_access_key", "high"),
+ (
+ re.compile(r"\b(gh[oprsu]_[0-9A-Za-z]{36}|github_pat_\w{82})\b"),
+ "github_token",
+ "high",
+ ),
+ (re.compile(r"(?i:(xox[baprs]-[0-9a-zA-Z-]{10,72}))"), "slack_token", "high"),
+ (re.compile(r"\b(AIza[0-9A-Za-z_-]{35})\b"), "google_api_key", "high"),
+ (
+ re.compile(r"\b((?:sk_(?:live|test)|pk_live)_[0-9A-Za-z]{10,99})\b"),
+ "stripe_key",
+ "high",
+ ),
(re.compile(r'f[\'"].*SELECT.*\{.*\}'), "fstring_sql", "med"),
(re.compile(r"\.execute\(\s*[\'\"]\s*SELECT.*\+"), "concat_sql", "med"),
(re.compile(r"verify\s*=\s*False"), "tls_verify_false", "med"),
@@ -207,6 +231,17 @@ _SPANNING_PATTERNS: list[tuple[re.Pattern, str, str]] = [
"subprocess_shell_true",
"high",
),
+ # A PEM header alone is not a leak: code that assembles PEM text
+ # (``"-----BEGIN " + kind + " PRIVATE KEY-----"``) contains the header
+ # string without ever holding key material. Requiring a base64-looking
+ # body line right after the header is what tells the two apart, and the
+ # capture group around that body line is the value the SECRET_KINDS gate
+ # below checks (length + placeholder), same as every other secret kind.
+ (
+ re.compile(r"(?i:-----BEGIN[ A-Z0-9_-]{0,100}PRIVATE KEY-----)\r?\n([A-Za-z0-9+/=]{20,})"),
+ "private_key_pem",
+ "high",
+ ),
]
# Symbol names that are informational security hotspots
@@ -218,7 +253,18 @@ _SYMBOL_KEYWORDS = re.compile(r"\b(auth|token|password|jwt|session|crypto)\b", r
# mostly noise, whereas a committed secret is actionable and persists in
# history. This positions history mode as complementary to gitleaks /
# trufflehog rather than a noisy replacement.
-SECRET_KINDS: frozenset[str] = frozenset({"hardcoded_password", "hardcoded_secret"})
+SECRET_KINDS: frozenset[str] = frozenset(
+ {
+ "hardcoded_password",
+ "hardcoded_secret",
+ "aws_access_key",
+ "github_token",
+ "slack_token",
+ "google_api_key",
+ "stripe_key",
+ "private_key_pem",
+ }
+)
# Kinds whose ``snippet`` is a symbol *name* rather than the text of the line
# it sits on (see the symbol-name scan below). Serve-time line verification
@@ -515,6 +561,12 @@ class SecurityScanner:
# the per-line pass already caught on that line is not duplicated.
for pattern, kind, severity in _SPANNING_PATTERNS:
for match in pattern.finditer(source):
+ if kind in SECRET_KINDS:
+ val = match.group(1) if match.groups() else ""
+ if not _is_valid_credential_value(val):
+ continue
+ if is_low_sev_file:
+ severity = "low"
start_line = source.count("\n", 0, match.start()) + 1
if any(f["kind"] == kind and f["line"] == start_line for f in findings):
continue
diff --git a/tests/unit/analysis/test_security_scan.py b/tests/unit/analysis/test_security_scan.py
index 8b76a6ac0..9f3b79b7b 100644
--- a/tests/unit/analysis/test_security_scan.py
+++ b/tests/unit/analysis/test_security_scan.py
@@ -245,6 +245,100 @@ class TestScanFile:
assert len(hits) == 1
assert hits[0]["severity"] == "low"
+ # -- Vendor credential shapes (#2117) ----------------------------------
+
+ @pytest.mark.parametrize(
+ ("source", "expected_kind"),
+ [
+ ('AWS_ACCESS_KEY_ID = "AKIAQZXNRTVYWMPKLBHG"\n', "aws_access_key"),
+ (
+ 'GITHUB_TOKEN = "ghp_abcdefghijklmnopqrstuvwxyzABCDEFGHIJ"\n',
+ "github_token",
+ ),
+ (
+ "GH_APP_TOKEN = 'ghu_abcdefghijklmnopqrstuvwxyzABCDEFGHIJ'\n",
+ "github_token",
+ ),
+ (
+ 'GH_FINE_GRAINED = "github_pat_'
+ + "abcdefghijklmnopqrstuvwxyzABCDEFGHIJabcdefghijklmnopqrstuvwxyzABCDEFGHIJabcdefghij"
+ + '"\n',
+ "github_token",
+ ),
+ ('SLACK_BOT_TOKEN = "xoxb-4721609583-T5Rk9mPz2Qa"\n', "slack_token"),
+ (
+ 'GOOGLE_MAPS_API_KEY = "AIzaSyD9K2vQ7xR4mZ1pL8tY6wU3nB0cF5hD9aX"\n',
+ "google_api_key",
+ ),
+ (
+ # Built from parts so the literal "sk_" + "live_" prefix pair
+ # never appears contiguous in source: GitHub's push
+ # protection flags that Stripe prefix on sight, independent
+ # of body entropy.
+ 'stripe.api_key = "' + "sk_" + "live_" + "A" * 24 + '"\n',
+ "stripe_key",
+ ),
+ ],
+ )
+ def test_vendor_key_shapes_fire_regardless_of_variable_name(
+ self, source: str, expected_kind: str
+ ) -> None:
+ scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]
+ findings = asyncio.run(scanner.scan_file("config.py", source, symbols=[]))
+ kinds = {f["kind"] for f in findings}
+ assert expected_kind in kinds
+
+ def test_token_and_access_key_keywords_are_detected(self) -> None:
+ """#2117: ``token`` and ``access_key`` were not in the keyword list."""
+ scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]
+ for source in (
+ 'TOKEN = "a_real_looking_token_value_123"\n',
+ 'access_key = "a_real_looking_access_key_456"\n',
+ ):
+ findings = asyncio.run(scanner.scan_file("config.py", source, symbols=[]))
+ assert any(f["kind"] == "hardcoded_secret" for f in findings), source
+
+ def test_client_id_vendor_key_is_caught_by_shape_not_name(self) -> None:
+ """A vendor key is invisible to the keyword list under a name like
+ ``client_id``, so it must be caught by the value shape alone."""
+ scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]
+ source = 'client_id = "AKIAQZXNRTVYWMPKLBHG"\n'
+ findings = asyncio.run(scanner.scan_file("config.py", source, symbols=[]))
+ assert any(f["kind"] == "aws_access_key" for f in findings)
+
+ def test_pem_private_key_with_body_is_flagged(self) -> None:
+ scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]
+ source = (
+ "-----BEGIN RSA PRIVATE KEY-----\n"
+ "MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC7VJTUt9Us8cKj\n"
+ "-----END RSA PRIVATE KEY-----\n"
+ )
+ findings = asyncio.run(scanner.scan_file("id_rsa.py", source, symbols=[]))
+ hits = [f for f in findings if f["kind"] == "private_key_pem"]
+ assert len(hits) == 1
+ assert hits[0]["line"] == 1
+
+ def test_pem_header_without_body_does_not_fire(self) -> None:
+ """Code that assembles PEM text from the header string alone (no key
+ material) must not be reported."""
+ scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]
+ source = (
+ 'PEM_HEADER = "-----BEGIN RSA PRIVATE KEY-----"\n'
+ 'PEM_FOOTER = "-----END RSA PRIVATE KEY-----"\n'
+ )
+ findings = asyncio.run(scanner.scan_file("pem_templates.py", source, symbols=[]))
+ assert not any(f["kind"] == "private_key_pem" for f in findings)
+
+ def test_vendor_key_shapes_are_gated_by_credential_value_rules(self) -> None:
+ """#2121's value gate must apply to the new vendor kinds too: a
+ low-severity path downgrades severity, same as the keyword kinds."""
+ scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]
+ source = 'AWS_ACCESS_KEY_ID = "AKIAQZXNRTVYWMPKLBHG"\n'
+ findings = asyncio.run(scanner.scan_file("tests/fixtures/keys.py", source, symbols=[]))
+ hits = [f for f in findings if f["kind"] == "aws_access_key"]
+ assert len(hits) == 1
+ assert hits[0]["severity"] == "low"
+
def test_single_line_subprocess_shell_true_is_flagged(self) -> None:
scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]
source = 'subprocess.run("rm -rf " + path, shell=True)\n'
diff --git a/tests/unit/analysis/test_security_scan_history.py b/tests/unit/analysis/test_security_scan_history.py
index 8ffb3d602..f684a6a04 100644
--- a/tests/unit/analysis/test_security_scan_history.py
+++ b/tests/unit/analysis/test_security_scan_history.py
@@ -246,9 +246,24 @@ async def test_history_progress_fires_for_clean_repo(session: AsyncSession) -> N
assert summary.files_scanned == 1
-def test_secret_kinds_are_the_two_secret_patterns() -> None:
- """Guard against the registry drifting away from the history gate."""
- assert {"hardcoded_password", "hardcoded_secret"} == SECRET_KINDS
+def test_secret_kinds_are_the_genuine_credential_patterns() -> None:
+ """Guard against the registry drifting away from the history gate.
+
+ #2117 added value-shape patterns for vendor credential formats (AWS,
+ GitHub, Slack, Google, Stripe) and a PEM private-key pattern; all are
+ genuine leaked-credential kinds, so history mode's default gate covers
+ them the same as the two keyword-based kinds.
+ """
+ assert {
+ "hardcoded_password",
+ "hardcoded_secret",
+ "aws_access_key",
+ "github_token",
+ "slack_token",
+ "google_api_key",
+ "stripe_key",
+ "private_key_pem",
+ } == SECRET_KINDS
# ---------------------------------------------------------------------------