Skip to content

Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.

fix(security): report a sink only where it can run (#3534)

Raghav Chamadiyaf81791aa5246
diff --git a/docs/layers/SECURITY.md b/docs/layers/SECURITY.md
index fddfc89fd..e1b9c7576 100644
--- a/docs/layers/SECURITY.md
+++ b/docs/layers/SECURITY.md
@@ -49,8 +49,8 @@ material, or under a directory named `test`, `tests`, `__tests__`, `__test__`,
| Kind | Severity | Matches |
|------|----------|---------|
| `eval_call` | high | `eval(...)`, including a receiver chain (`vm.eval(`) |
-| `exec_call` | high | `exec(...)`. Outside Python, only a call through a name bound to `child_process` (`exec`, `execFile`, `execSync`) |
-| `pickle_loads` | high | `pickle.loads` |
+| `exec_call` | high | `exec(...)`. Outside Python, only a call through a name bound to `child_process` (`exec`, `execSync`). `execFile` runs no shell and does not fire |
+| `pickle_loads` | high | `pickle.loads`, except a round trip of an own object (`pickle.loads(pickle.dumps(x))`) |
| `subprocess_shell_true` | high | `subprocess.*` with `shell=True`, also across lines |
| `os_system` | high | `os.system` |
| `hardcoded_password` | high | a quoted literal assigned to a name containing `password`, any case |
@@ -66,14 +66,14 @@ material, or under a directory named `test`, `tests`, `__tests__`, `__test__`,
| `private_key_pem` | high | a `-----BEGIN ... PRIVATE KEY-----` header followed by a base64 body line (an escaped `\n` counts) |
| `public_env_secret` | high | `API_KEY`/`SECRET`/`TOKEN`/`PASSWORD` behind `NEXT_PUBLIC_` or `VITE_`, excluding `..._ANON_...` |
| `new_function_call` | high | `new Function(...)` |
-| `fstring_sql` | med | a Python f-string with `SELECT` and an interpolation |
+| `fstring_sql` | med | a Python f-string query (`SELECT` with `FROM`) that interpolates a value after a comparison or `LIKE` (`WHERE id = {uid}`). A table, column, fragment or placeholder list cannot be bound, so interpolating one does not fire |
| `concat_sql` | med | `.execute("SELECT ... +` |
| `template_literal_sql` | med | a JS/TS template literal with `SELECT`+`FROM` or `UPDATE`+`SET` and an interpolation |
-| `tls_verify_false` | med | `verify = False` |
+| `tls_verify_false` | med | `verify=False` passed to an HTTP client call or set as `verify`, in a file that imports an HTTP client library (`requests`, `httpx`, `aiohttp`, `urllib3`, boto3). A call through a plain name must go through that import or a value built from it |
| `reject_unauthorized_false` | med | `rejectUnauthorized: false` |
| `unsafe_inner_html` | med | `__html:` (React's `dangerouslySetInnerHTML`) assigned a non-literal |
-| `weak_hash` | low | the words `md5` or `sha1` |
-| `security_sensitive_symbol` | low | a symbol named with `auth`, `token`, `password`, `jwt`, `session` or `crypto` |
+| `weak_hash` | low | MD5 or SHA-1 in use in code: a call (`hashlib.md5(`), a type (`MD5.Create()`) or the algorithm named to a hashing call (`createHash("sha1")`). A call passing `usedforsecurity=False` does not fire |
+| `security_sensitive_symbol` | low | a symbol named `auth`, `password`, `jwt` or `crypto`. A bare `session` or `token` is usually a database session or a lexer token and does not fire |
`security_sensitive_symbol` is informational: it marks where security code
lives, not a problem.
@@ -95,13 +95,18 @@ and the in-change commit scan report only these by default.
comment is still a leak.
- **Prose files** (`.md`, `.mdx`, `.rst`, `.txt`, `.adoc`) are scanned for the
secret kinds only.
+- **JavaScript kinds stay in JavaScript.** `unsafe_inner_html`,
+ `template_literal_sql`, `new_function_call`, `reject_unauthorized_false` and
+ `public_env_secret` do not fire in Python files, where such a line can only be
+ quoted text.
- **Placeholders are skipped**: a keyword value that is elided or templated
- (`...`, `<your key>`, `xxx`, `example`, `placeholder`, `dummy`, `fake`,
+ (`...`, `<your key>`, `xxx`, `example`, `placeholder`, `dummy`, `fake`, `redacted`,
`changeme`, `{{ }}`, `${}`), shorter than eight characters, a short
single-case word, or a snake_case key name. A value a vendor kind already
reports is not reported twice.
-- **`weak_hash`, the SQL kinds, the TLS kinds and `unsafe_inner_html`** run on
- raw source, comments included.
+- **`concat_sql`, `template_literal_sql` and `unsafe_inner_html`** run on raw
+ source, comments included. `fstring_sql` and `weak_hash` need their match to
+ start in code, so a comment or a fixture string quoting one does not fire.
### Working tree versus history
@@ -143,7 +148,7 @@ its first four characters and `****`. The raw value is never stored.
2. High working-tree kinds, as places to read. The layer found a call; you
decide whether it is reachable.
3. `security_sensitive_symbol`, as a map of where auth and crypto code lives.
-4. Low findings last. `weak_hash` is a word match.
+4. Low findings last. `weak_hash` cannot tell a checksum from a password hash.
A low finding count means the registry did not match. On most repos, in most
languages, that is all it means.
@@ -248,7 +253,8 @@ result with that in mind.
eight vendor shapes are invisible unless you add custom patterns. Use gitleaks
or trufflehog for secret scanning proper.
- **No dependency or CVE scanning.**
-- **False positives happen.** `weak_hash` fires on `md5` in a comment.
+- **False positives happen.** `weak_hash` fires on an MD5 checksum unless the
+ call passes `usedforsecurity=False`.
`unsafe_inner_html` ships at `med` because it cannot tell a name pointing at
a pinned constant from a dynamic value.
- **The five JS/TS kinds were checked by hand** on a 17-repository, 1,109-file
diff --git a/packages/cli/src/repowise/cli/commands/augment_cmd/_shared.py b/packages/cli/src/repowise/cli/commands/augment_cmd/_shared.py
index 2d3674686..448880b7c 100644
--- a/packages/cli/src/repowise/cli/commands/augment_cmd/_shared.py
+++ b/packages/cli/src/repowise/cli/commands/augment_cmd/_shared.py
@@ -91,7 +91,7 @@ def _ledger_key(surface: str, category: str, text: str) -> str:
"""
import hashlib
- digest = hashlib.sha1(text.encode("utf-8", "replace")).hexdigest()[:12]
+ digest = hashlib.sha1(text.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:12]
return f"{surface}:{category}:{digest}"
diff --git a/packages/cli/src/repowise/cli/commands/augment_cmd/command.py b/packages/cli/src/repowise/cli/commands/augment_cmd/command.py
index 8eed8b4de..332eee5ac 100644
--- a/packages/cli/src/repowise/cli/commands/augment_cmd/command.py
+++ b/packages/cli/src/repowise/cli/commands/augment_cmd/command.py
@@ -364,7 +364,7 @@ def _claim_emission(event: str, context: str) -> bool:
import time
try:
- key = hashlib.sha1(f"{event}\x00{context}".encode()).hexdigest()[:16]
+ key = hashlib.sha1(f"{event}\x00{context}".encode(), usedforsecurity=False).hexdigest()[:16]
marker = Path(tempfile.gettempdir()) / f".repowise-augment-{key}"
now = time.time()
try:
diff --git a/packages/cli/src/repowise/cli/commands/augment_cmd/read_state.py b/packages/cli/src/repowise/cli/commands/augment_cmd/read_state.py
index 5daabf7ef..2a341ca59 100644
--- a/packages/cli/src/repowise/cli/commands/augment_cmd/read_state.py
+++ b/packages/cli/src/repowise/cli/commands/augment_cmd/read_state.py
@@ -78,7 +78,7 @@ def _session_state_path(repo_path: Path, session_id: str = "") -> Path:
"""This session's state file. One per session id — see :data:`_STATE_DIR`."""
import hashlib
- digest = hashlib.sha1(session_id.encode("utf-8", "replace")).hexdigest()[:16]
+ digest = hashlib.sha1(session_id.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:16]
return Path(repo_path) / ".repowise" / _STATE_DIR / f"{digest}.json"
diff --git a/packages/cli/src/repowise/cli/commands/augment_cmd/reread.py b/packages/cli/src/repowise/cli/commands/augment_cmd/reread.py
index 6cdeeb5e2..d1cbebe01 100644
--- a/packages/cli/src/repowise/cli/commands/augment_cmd/reread.py
+++ b/packages/cli/src/repowise/cli/commands/augment_cmd/reread.py
@@ -89,7 +89,7 @@ def content_digest(content: str) -> str:
"""
import hashlib
- return hashlib.sha1(content.encode("utf-8", "replace")).hexdigest()[:12]
+ return hashlib.sha1(content.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:12]
def _range_label(offset: object, limit: object) -> str:
diff --git a/packages/cli/src/repowise/cli/hook_ledger.py b/packages/cli/src/repowise/cli/hook_ledger.py
index 75de7f10d..1f8a1de0a 100644
--- a/packages/cli/src/repowise/cli/hook_ledger.py
+++ b/packages/cli/src/repowise/cli/hook_ledger.py
@@ -96,7 +96,9 @@ def emitting_build() -> str:
# The directory rather than ``__file__``: two modules in one install
# must not read as two installs.
- install = hashlib.sha1(os.path.dirname(__file__).encode("utf-8", "replace")).hexdigest()
+ install = hashlib.sha1(
+ os.path.dirname(__file__).encode("utf-8", "replace"), usedforsecurity=False
+ ).hexdigest()
_BUILD = f"{__version__}+{install[:8]}"
return _BUILD
diff --git a/packages/core/src/repowise/core/analysis/actions/model.py b/packages/core/src/repowise/core/analysis/actions/model.py
index c55812d0b..80b3d9a4d 100644
--- a/packages/core/src/repowise/core/analysis/actions/model.py
+++ b/packages/core/src/repowise/core/analysis/actions/model.py
@@ -203,7 +203,7 @@ class Action:
def action_id(self) -> str:
identity = self.identity or f"{self.target_path}:{self.target_symbol or ''}"
key = f"{self.rule}:{identity}"
- return "act_" + hashlib.sha1(key.encode("utf-8")).hexdigest()[:16]
+ return "act_" + hashlib.sha1(key.encode("utf-8"), usedforsecurity=False).hexdigest()[:16]
def as_dict(self) -> dict[str, Any]:
return {
@@ -256,4 +256,4 @@ class RuleOutcome:
def fingerprint(*parts: object) -> str:
raw = "|".join(str(p) for p in parts)
- return hashlib.sha1(raw.encode("utf-8")).hexdigest()[:12]
+ return hashlib.sha1(raw.encode("utf-8"), usedforsecurity=False).hexdigest()[:12]
diff --git a/packages/core/src/repowise/core/analysis/security_gate.py b/packages/core/src/repowise/core/analysis/security_gate.py
index a99d9f237..4a694670d 100644
--- a/packages/core/src/repowise/core/analysis/security_gate.py
+++ b/packages/core/src/repowise/core/analysis/security_gate.py
@@ -94,7 +94,7 @@ _BINARY_SNIFF = 8000
#: is absent on purpose: it needs parsed symbols and never gates.
_RULE_TEXT: dict[str, str] = {
"eval_call": "A call to eval.",
- "exec_call": "A call to exec (outside Python, in a file that names child_process).",
+ "exec_call": "A call to exec (outside Python, child_process exec or execSync).",
"pickle_loads": "pickle.loads, which runs code from the data it loads.",
"subprocess_shell_true": "A subprocess call with shell=True.",
"os_system": "os.system, which runs its argument through a shell.",
@@ -109,10 +109,13 @@ _RULE_TEXT: dict[str, str] = {
"azure_devops_pat": "An Azure DevOps personal access token.",
"git_url_credentials": "A password or token embedded in a git remote URL.",
"private_key_pem": "A PEM private key with its body.",
- "fstring_sql": "An f-string holding SQL and an interpolation.",
+ "fstring_sql": "An f-string query that interpolates a value where a bound parameter belongs.",
"concat_sql": "SQL built by string concatenation.",
- "tls_verify_false": "TLS certificate verification turned off (verify=False).",
- "weak_hash": "The words md5 or sha1.",
+ "tls_verify_false": "TLS certificate verification turned off (verify=False on an HTTP client).",
+ "weak_hash": (
+ "MD5 or SHA-1 used in code: a call, a type, or the algorithm named to a hashing "
+ "call. A call passing usedforsecurity=False is exempt."
+ ),
"unsafe_inner_html": "__html set from a non-literal value.",
"template_literal_sql": "A template literal holding SQL and an interpolation.",
"public_env_secret": "A secret-shaped name behind a NEXT_PUBLIC_ or VITE_ prefix.",
diff --git a/packages/core/src/repowise/core/analysis/security_scan.py b/packages/core/src/repowise/core/analysis/security_scan.py
index c9a9a834d..33ba204b6 100644
--- a/packages/core/src/repowise/core/analysis/security_scan.py
+++ b/packages/core/src/repowise/core/analysis/security_scan.py
@@ -103,6 +103,7 @@ _CREDENTIAL_SUBSTRING_PLACEHOLDERS: tuple[str, ...] = (
"...",
"\u2026", # a typographic ellipsis
"fixture",
+ "redacted",
)
# Broader than ``is_test_related_path`` on purpose. That one asks whether a file
@@ -177,7 +178,9 @@ _CALL_KINDS = frozenset(kind for _, kind, _ in _CALL_PATTERNS)
# The dangerous call in JavaScript comes from ``child_process``, so only a call
# through a name bound to that module counts: a namespace (``cp.exec(``), a named
# import (``exec(`` after ``import { exec } from "child_process"``) or the
-# ``require`` result itself. A file's own function named ``exec``, or a regex's
+# ``require`` result itself. Only ``exec`` and ``execSync`` run a shell;
+# ``execFile`` passes its arguments as argv, like a Python ``subprocess.run``
+# without ``shell=True``, which does not fire either. A file's own function named ``exec``, or a regex's
# ``.exec`` next to an unrelated ``spawn`` import, is not a shell. ``eval`` needs
# no such gate: it is a genuine global there.
_CP_MODULE = r"""['"](?:node:)?child_process['"]"""
@@ -191,9 +194,9 @@ _CP_NAMED = re.compile(
rf"|(?:const|let|var)\s*\{{([^}}]*)\}}\s*=\s*require\(\s*{_CP_MODULE}\s*\)"
)
_CP_REQUIRE_CALL = re.compile(
- rf"require\(\s*{_CP_MODULE}\s*\)\s*\.\s*exec(?:File)?(?:Sync)?\s*\("
+ rf"require\(\s*{_CP_MODULE}\s*\)\s*\.\s*exec(?:Sync)?\s*\("
)
-_EXEC_EXPORT = re.compile(r"exec(?:File)?(?:Sync)?")
+_EXEC_EXPORT = re.compile(r"exec(?:Sync)?")
def _child_process_exec_calls(source: str, masked: str) -> list[int]:
@@ -214,7 +217,7 @@ def _child_process_exec_calls(source: str, masked: str) -> list[int]:
offsets = [m.start() for m in _CP_REQUIRE_CALL.finditer(source) if not masked[m.start()].isspace()]
names = [re.escape(n) for n in namespaces]
if names:
- receiver = rf"(?<![\w$.])(?:{'|'.join(names)})\s*\.\s*exec(?:File)?(?:Sync)?\s*\("
+ receiver = rf"(?<![\w$.])(?:{'|'.join(names)})\s*\.\s*exec(?:Sync)?\s*\("
offsets += [m.start() for m in re.finditer(receiver, masked)]
if functions:
bare = rf"(?<![\w$.])(?:{'|'.join(map(re.escape, functions))})\s*\("
@@ -223,7 +226,9 @@ def _child_process_exec_calls(source: str, masked: str) -> list[int]:
_PATTERNS: list[tuple[re.Pattern, str, str]] = [
*_CALL_PATTERNS,
- (re.compile(r"pickle\.loads"), "pickle_loads", "high"),
+ # Not ``pickle.loads(pickle.dumps(x))``: a round trip of the program's own
+ # object, which is how a copy or a picklability test is written.
+ (re.compile(r"pickle\.loads(?!\s*\(\s*pickle\.dumps\s*\()"), "pickle_loads", "high"),
(re.compile(r"subprocess\..*shell\s*=\s*True"), "subprocess_shell_true", "high"),
(re.compile(r"os\.system"), "os_system", "high"),
# Case-insensitive: a credential pinned in source is usually written as a
@@ -313,7 +318,27 @@ _PATTERNS: list[tuple[re.Pattern, str, str]] = [
(re.compile(r'f[\'"].*SELECT.*\{.*\}'), "fstring_sql", "med"),
(re.compile(r"\.execute\(\s*[\'\"]\s*SELECT.*\+"), "concat_sql", "med"),
(re.compile(r"verify\s*=\s*False"), "tls_verify_false", "med"),
- (re.compile(r"\bmd5\b|\bsha1\b"), "weak_hash", "low"),
+ # A weak algorithm put to use, not the word: a call (``md5(``, ``hashlib.sha1(``,
+ # ``DigestUtils.md5Hex(``, ``MD5_Init(``), a type or module function
+ # (``MD5.Create()``, ``md5.Sum(``, ``Sha1::new()``, ``Digest::SHA1``), ``hashlib.md5``
+ # handed over as a digest, or the algorithm named to a hashing call
+ # (``createHash("md5")``, ``getInstance("SHA-1")``). A ``sha1`` variable
+ # holding a commit id, or the word in a comment, is neither.
+ (
+ re.compile(
+ r"(?<![\w$])(?i:md5|sha1)(?i:_?(?:hex|file|hash|digest|init|update|final))?\s*\("
+ r"|(?<![\w$])(?i:md5|sha1)\s*(?:\.|::)\s*"
+ r"(?:New|Sum|Create|new|hexdigest|digest|compute)\b"
+ r"|\b(?:MD5|SHA1)(?:CryptoServiceProvider|Managed)\b"
+ r"|\bDigest::(?:MD5|SHA1)\b"
+ r"|\bhashlib\s*\.\s*(?:md5|sha1)\b"
+ r"|\b(?:new|createHash|createHmac|getInstance|hash|hash_hmac|hash_file|digest|Create)"
+ r"\s*\(\s*['\"](?i:md5|sha-?1)['\"]"
+ r"|\bdigestmod\s*=\s*['\"](?i:md5|sha1)['\"]"
+ ),
+ "weak_hash",
+ "low",
+ ),
# -- JS/TS patterns (#1935 Tier 1) -------------------------------------
# Measured on the same 17-repo, 1109-file corpus as the exec_call/secret
# fixes in #1947. Three of these five needed tightening before they were
@@ -401,8 +426,10 @@ _SPANNING_PATTERNS: list[tuple[re.Pattern, str, str]] = [
),
]
-# Symbol names that are informational security hotspots
-_SYMBOL_KEYWORDS = re.compile(r"\b(auth|token|password|jwt|session|crypto)\b", re.IGNORECASE)
+# Symbol names that are informational security hotspots. Not ``session`` or
+# ``token``: a symbol named only that is a database session, a pytest fixture or
+# a lexer token far more often than auth code.
+_SYMBOL_KEYWORDS = re.compile(r"\b(auth|password|jwt|crypto)\b", re.IGNORECASE)
# Kinds that name a dangerous call. They are matched against source with
# comments and string literals blanked, so a docstring, a comment or a regex
@@ -418,6 +445,207 @@ _MASKED_KINDS = frozenset(
}
)
+# JavaScript sinks. Python has no JSX, template literals, ``new Function`` or
+# Node TLS options, and no bundler reads a ``.py`` file into a client, so a
+# Python file holding one of these is quoting it, most often as a test fixture.
+_JS_ONLY_KINDS = frozenset(
+ {
+ "unsafe_inner_html",
+ "template_literal_sql",
+ "new_function_call",
+ "reject_unauthorized_false",
+ "public_env_secret",
+ }
+)
+
+# ``verify=False`` turns TLS off only when an HTTP client receives it, so the
+# file must import an HTTP client library, and a call through a plain name must
+# go through that import or a value built from it (``s = requests.Session()``).
+# Any other function is free to name a keyword ``verify``. Ceiling: a project's
+# own wrapper that forwards ``verify``, imported from another module, is not seen.
+_HTTP_LIBS = frozenset(
+ {"requests", "httpx", "aiohttp", "urllib3", "boto3", "botocore", "aiobotocore", "aioboto3"}
+)
+_HTTP_CALLS = frozenset(
+ {
+ "get",
+ "post",
+ "put",
+ "patch",
+ "delete",
+ "head",
+ "options",
+ "request",
+ "send",
+ "stream",
+ "urlopen",
+ "Client",
+ "AsyncClient",
+ "Session",
+ "HTTPTransport",
+ "AsyncHTTPTransport",
+ "client",
+ "resource",
+ }
+)
+
+# ``fstring_sql`` asks for a query (``SELECT`` with its ``FROM``) that
+# interpolates a value where a bound parameter belongs: after a comparison or
+# ``LIKE``. A table, a column, a fragment or a placeholder list cannot be bound,
+# so building those into the text is the only way to write them, and a value
+# interpolated there has no parameter fix to offer. Ceiling: ``IN ({ids})`` of
+# joined values, and an identifier taken from a request, are not seen.
+_SQL_FROM = re.compile(r"\bFROM\b")
+_SQL_VALUE = re.compile(r"""(?:[=<>]|\b(?i:LIKE|ILIKE))\s*['"%]*\{(?!\{)""")
+# A string literal continuing the one before it (implicit concatenation).
+_STRING_CONTINUATION = re.compile(r"""\A\s*[rRbBfFuU]{0,2}['"]""")
+
+
+def _interpolates_sql_value(line: str, start: int, lines: list[str], lineno: int) -> bool:
+ """Whether the f-string query opening at *start* on *line* puts a value in its text.
+
+ Only the literal is read, and the one opening the next line when the query
+ runs on into it.
+ """
+ triple = line[start + 1 : start + 4]
+ if triple in ('"""', "'''"):
+ # A triple-quoted query runs to its closing delimiter, often lines on.
+ text = "\n".join(lines[lineno - 1 : lineno + 29])
+ end = text.find(triple, start + 4)
+ query = text[text.find("SELECT", start) : end if end != -1 else len(text)]
+ return bool(_SQL_FROM.search(query) and _SQL_VALUE.search(query))
+ body = start + 2 # past ``f`` and the opening quote
+ query = line[line.find("SELECT", start) : _string_end(line, body)]
+ following = lines[lineno] if lineno < len(lines) else ""
+ if (cont := _STRING_CONTINUATION.match(following)) is not None:
+ query += "\n" + following[cont.end() : _string_end(following, cont.end())]
+ return bool(_SQL_FROM.search(query) and _SQL_VALUE.search(query))
+
+
+def _dotted(node: ast.AST) -> str:
+ """``a.b.c`` for a name or attribute chain, else ``""``."""
+ if isinstance(node, ast.Name):
+ return node.id
+ if isinstance(node, ast.Attribute) and (base := _dotted(node.value)):
+ return f"{base}.{node.attr}"
+ return ""
+
+
+def _root(node: ast.AST) -> ast.AST:
+ """The expression a chain of calls and attributes starts from."""
+ while isinstance(node, (ast.Call, ast.Attribute, ast.Await)):
+ node = node.func if isinstance(node, ast.Call) else node.value
+ return node
+
+
+def _http_bindings(tree: ast.AST) -> set[str]:
+ """Names and ``self.x`` targets bound to an HTTP client library or a value built from it."""
+ bound: set[str] = set()
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Import):
+ for alias in node.names:
+ if alias.name.split(".")[0] in _HTTP_LIBS:
+ bound.add(alias.asname or alias.name.split(".")[0])
+ elif (
+ isinstance(node, ast.ImportFrom)
+ and node.level == 0
+ and (node.module or "").split(".")[0] in _HTTP_LIBS
+ ):
+ bound.update(alias.asname or alias.name for alias in node.names)
+ if not bound:
+ return bound
+ # Twice, so a value built from a bound value (``c = s.client()``) is bound too.
+ for _ in range(2):
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Assign):
+ pairs = [(t, node.value) for t in node.targets]
+ elif isinstance(node, (ast.AnnAssign, ast.NamedExpr)) and node.value is not None:
+ pairs = [(node.target, node.value)]
+ elif isinstance(node, (ast.With, ast.AsyncWith)):
+ pairs = [(i.optional_vars, i.context_expr) for i in node.items if i.optional_vars]
+ else:
+ continue
+ for target, value in pairs:
+ if _dotted(_root(value)) in bound and (name := _dotted(target)):
+ bound.add(name)
+ return bound
+
+
+def _tls_verify_off_lines(source: str) -> set[int] | None:
+ """Lines where ``verify=False`` reaches an HTTP client; None when *source* does not parse."""
+ try:
+ tree = ast.parse(source)
+ except SyntaxError:
+ return None
+ bound = _http_bindings(tree)
+ if not bound:
+ return set()
+ found: set[int] = set()
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Call):
+ func = node.func
+ if isinstance(func, ast.Attribute):
+ name, receiver = func.attr, _root(func.value)
+ # A plain name must be bound; ``self.x`` and call results are
+ # taken on trust once the file uses an HTTP client library.
+ reaches = (
+ _dotted(receiver) in bound
+ or _dotted(func.value) in bound
+ or not isinstance(receiver, ast.Name)
+ or receiver.id in ("self", "cls")
+ )
+ elif isinstance(func, ast.Name):
+ name, reaches = func.id, func.id in bound
+ else:
+ continue
+ if name in _HTTP_CALLS and reaches:
+ found.update(
+ kw.lineno
+ for kw in node.keywords
+ if kw.arg == "verify"
+ and isinstance(kw.value, ast.Constant)
+ and kw.value.value is False
+ )
+ elif (
+ isinstance(node, ast.Assign)
+ and isinstance(node.value, ast.Constant)
+ and node.value.value is False
+ ):
+ # ``session.verify = False``, or a ``verify = False`` setting handed
+ # to a client later.
+ found.update(
+ target.lineno
+ for target in node.targets
+ if (isinstance(target, ast.Attribute) and target.attr == "verify")
+ or (isinstance(target, ast.Name) and target.id == "verify")
+ )
+ return found
+
+
+# Python's own declaration that a digest is a checksum or a cache key, not a
+# security control (``hashlib.sha1(data, usedforsecurity=False)``).
+_NOT_FOR_SECURITY = re.compile(r"\busedforsecurity\s*=\s*False\b")
+
+
+def _call_text(masked: list[str], lineno: int, start: int, limit: int = 600) -> str:
+ """The code from *start* on line *lineno* through the first call's closing paren.
+
+ Read from *masked* (strings and comments blanked, offsets kept), so a ``)``
+ or the keyword inside a string neither ends the call nor counts as code.
+ Bounded by *limit* characters.
+ """
+ code = "\n".join(masked[lineno - 1 : lineno + 9])[start : start + limit]
+ depth = 0
+ for i, char in enumerate(code):
+ if char == "(":
+ depth += 1
+ elif char == ")":
+ depth -= 1
+ if depth <= 0:
+ return code[: i + 1]
+ return code
+
+
# Prose is documentation, not executable code. Only secret kinds scan it,
# because a key pasted into a README is still a leak.
_PROSE_EXTENSIONS = tuple(DOC_EXTENSIONS)
@@ -914,6 +1142,8 @@ def scan_source(file_path: str, source: str, symbols: Iterable[Any] = ()) -> lis
# alike to every lookaround in ``_ANY_PATTERN``, so a file the whole-source
# search rejects has no matching line, and most files are such.
is_low_sev_file = _is_low_severity_path(file_path)
+ is_python = file_path.lower().endswith((".py", ".pyi"))
+ tls_lines: list[set[int] | None] = []
scan_lines = lines if _ANY_PATTERN.search(source) else []
for lineno, line in enumerate(scan_lines, start=1):
if not _ANY_PATTERN.search(line):
@@ -925,8 +1155,28 @@ def scan_source(file_path: str, source: str, symbols: Iterable[Any] = ()) -> lis
continue
if is_prose and kind not in SECRET_KINDS:
continue
+ if is_python and kind in _JS_ONLY_KINDS:
+ continue
if kind in _MASKED_KINDS:
match = pattern.search(masked_line(lineno, line))
+ if match and kind == "tls_verify_false" and is_python:
+ if not tls_lines:
+ tls_lines.append(_tls_verify_off_lines(source))
+ if tls_lines[0] is not None and lineno not in tls_lines[0]:
+ match = None
+ elif kind == "fstring_sql":
+ # The ``f`` must open a literal in code, not sit inside another
+ # string, and the query must interpolate a value.
+ match = _match_starting_in(pattern, line, masked_line(lineno, line))
+ if match and not _interpolates_sql_value(line, match.start(), lines, lineno):
+ match = None
+ elif kind == "weak_hash":
+ # Written in code, and not declared a non-security use.
+ match = _match_starting_in(pattern, line, masked_line(lineno, line))
+ if match and _NOT_FOR_SECURITY.search(
+ _call_text(mask()[1], lineno, match.start())
+ ):
+ match = None
elif kind in _KEYWORD_KINDS:
# ``password = "..."`` in a comment or docstring is an example.
match = _match_starting_in(pattern, line, masked_line(lineno, line))
diff --git a/packages/core/src/repowise/core/sessions/efficacy.py b/packages/core/src/repowise/core/sessions/efficacy.py
index 74f6fde95..e5c4be5d1 100644
--- a/packages/core/src/repowise/core/sessions/efficacy.py
+++ b/packages/core/src/repowise/core/sessions/efficacy.py
@@ -234,7 +234,7 @@ def ledger_key(surface: str, category: str, text: str) -> str:
the emitted text is what lets a transcript firing find the row the live
hook wrote for it.
"""
- digest = hashlib.sha1(text.encode("utf-8", "replace")).hexdigest()[:12]
+ digest = hashlib.sha1(text.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:12]
return f"{surface}:{category}:{digest}"
diff --git a/tests/unit/analysis/test_security_gate.py b/tests/unit/analysis/test_security_gate.py
index e7125886d..75dcd6489 100644
--- a/tests/unit/analysis/test_security_gate.py
+++ b/tests/unit/analysis/test_security_gate.py
@@ -29,6 +29,8 @@ from repowise.core.ci.baseline import read_baseline, read_entries
KEY = "AKIA" + "QZXNRTVYWMPKLBHG"
PEM_BODY = "MIIEpAIBAAKCAQEAu1SU1LfVLPHCozMxH2Mo4lgOEePzNm0tRgeLezV6ffAt0gunVTLw"
+# A key shape that can sit inside base64, so a finding lands on a body line.
+STRAY = "AKIAQ2Z7RT4WXY5BCDFG" # pragma: allowlist secret
def _git(cwd, *args: str) -> str:
@@ -338,11 +340,11 @@ def test_only_high_entropy_shapes_hash_their_material():
@pytest.mark.parametrize(
("line", "raw"),
[
- ("PASSWORD = 'Tr0ub4dor\"TAILSECRET9876' # md5\n", "TAILSECRET9876"),
- ('secret = "token = \'RAWVALUE12345678\'" # md5\n', "RAWVALUE12345678"),
- ('ENV["NEXT_PUBLIC_API_KEY"] = "RAWVALUE12345678" # md5\n', "RAWVALUE12345678"),
- ("ENV NEXT_PUBLIC_API_KEY RAWVALUE12345678 md5\n", "RAWVALUE12345678"),
- ('k = process.env.NEXT_PUBLIC_API_KEY ?? "RAWVALUE12345678" // md5\n', "RAWVALUE12345678"),
+ ("PASSWORD = 'Tr0ub4dor\"TAILSECRET9876'; md5(b)\n", "TAILSECRET9876"),
+ ('secret = "token = \'RAWVALUE12345678\'"; md5(b)\n', "RAWVALUE12345678"),
+ ('ENV["NEXT_PUBLIC_API_KEY"] = "RAWVALUE12345678"; md5(b)\n', "RAWVALUE12345678"),
+ ("ENV NEXT_PUBLIC_API_KEY RAWVALUE12345678 md5(b)\n", "RAWVALUE12345678"),
+ ('k = process.env.NEXT_PUBLIC_API_KEY ?? "RAWVALUE12345678"; md5(b)\n', "RAWVALUE12345678"),
],
)
def test_no_raw_value_escapes_the_mask(line, raw):
@@ -355,9 +357,9 @@ def test_no_raw_value_escapes_the_mask(line, raw):
@pytest.mark.parametrize(
"source",
[
- f"key: |\n -----BEGIN RSA PRIVATE KEY-----\n {PEM_BODY}md5\n -----END RSA PRIVATE KEY-----\n",
- f"-----BEGIN RSA PRIVATE KEY-----\n{PEM_BODY}\n{PEM_BODY[::-1]}md5\n",
- f"KEY=-----BEGIN RSA PRIVATE KEY----- {PEM_BODY} md5 -----END RSA PRIVATE KEY-----\n",
+ f"key: |\n -----BEGIN RSA PRIVATE KEY-----\n {PEM_BODY}+{STRAY}\n -----END RSA PRIVATE KEY-----\n",
+ f"-----BEGIN RSA PRIVATE KEY-----\n{PEM_BODY}\n{PEM_BODY[::-1]}+{STRAY}\n",
+ f"KEY=-----BEGIN RSA PRIVATE KEY----- {PEM_BODY} {STRAY} -----END RSA PRIVATE KEY-----\n",
],
)
def test_pem_body_lines_never_reach_a_snippet(source):
@@ -434,9 +436,9 @@ def test_sarif_suppresses_the_baseline_and_drops_history_lines():
@pytest.mark.parametrize(
("line", "raw"),
[
- ("ENV NEXT_PUBLIC_API_KEY p@ssw0rd!Xy#Zq9 md5\n", "p@ssw0rd!Xy#Zq9"),
- ('ENV["NEXT_PUBLIC_TOKEN"] = "Ab$9!kLm@2#Qr%zz" # md5\n', "Ab$9!kLm@2#Qr%zz"),
- ('x = NEXT_PUBLIC_SECRET ?? "p@ss:w0rd!2024x" // md5\n', "p@ss:w0rd!2024x"),
+ ("ENV NEXT_PUBLIC_API_KEY p@ssw0rd!Xy#Zq9 md5(b)\n", "p@ssw0rd!Xy#Zq9"),
+ ('ENV["NEXT_PUBLIC_TOKEN"] = "Ab$9!kLm@2#Qr%zz"; md5(b)\n', "Ab$9!kLm@2#Qr%zz"),
+ ('x = NEXT_PUBLIC_SECRET ?? "p@ss:w0rd!2024x"; md5(b)\n', "p@ss:w0rd!2024x"),
("password = \"ab'SuperSecretPassword123\"; eval(x)\n", "SuperSecretPassword123"),
],
)
diff --git a/tests/unit/analysis/test_security_scan.py b/tests/unit/analysis/test_security_scan.py
index 1df7b9900..5a708172b 100644
--- a/tests/unit/analysis/test_security_scan.py
+++ b/tests/unit/analysis/test_security_scan.py
@@ -75,6 +75,18 @@ async def _rows(sf) -> list[tuple]:
class TestScanFile:
+ @pytest.mark.parametrize(
+ ("source", "expected"),
+ [
+ ("obj = pickle.loads(request.body)\n", True),
+ ("obj = pickle.loads(blob); copy = pickle.dumps(obj)\n", True),
+ ("assert pickle.loads(pickle.dumps(config)) == config\n", False),
+ ("gb2 = pickle.loads(pickle.dumps(gb, protocol=pickle.HIGHEST_PROTOCOL))\n", False),
+ ],
+ )
+ def test_pickle_round_trip_is_not_a_load(self, source: str, expected: bool) -> None:
+ assert bool([f for f in scan_source("a.py", source) if f["kind"] == "pickle_loads"]) is expected
+
def test_line_patterns_fire_on_real_source(self) -> None:
scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]
findings = asyncio.run(scanner.scan_file("a.py", SNIPPY.decode(), symbols=[]))
@@ -173,13 +185,12 @@ class TestScanFile:
[
('import { exec } from "child_process";\nexec(cmd);\n', {"exec_call"}),
('const cp = require("child_process");\ncp.execSync(cmd);\n', {"exec_call"}),
- (
- 'import { execFile } from "node:child_process";\nexecFile(bin, args);\n',
- {"exec_call"},
- ),
+ # execFile passes argv and runs no shell, like subprocess without shell=True.
+ ('import { execFile } from "node:child_process";\nexecFile(bin, args);\n', set()),
('import { exec as run } from "child_process";\nrun(cmd);\n', {"exec_call"}),
('const { execSync: sh } = require("child_process");\nsh(cmd);\n', {"exec_call"}),
- ('import * as cp from "node:child_process";\ncp.execFile(bin);\n', {"exec_call"}),
+ ('import * as cp from "node:child_process";\ncp.execFile(bin);\n', set()),
+ ('import * as cp from "node:child_process";\ncp.exec(cmd);\n', {"exec_call"}),
('require("child_process").exec(cmd);\n', {"exec_call"}),
# A regex's ``.exec`` in a file that also spawns is not bound to the module.
('require("child_process");\nconst m = /a/.exec(s);\n', set()),
@@ -285,6 +296,8 @@ class TestScanFile:
'api_key = "your-api-key-here"\n',
'api_key = "<YOUR_API_KEY>"\n',
'password = "fixture-value"\n',
+ 'api_key = "«redacted:sk-repo»"\n',
+ 'OPENAI_API_KEY = "[REDACTED]-k3y-0f-mine"\n',
],
)
def test_credential_placeholder_filtering(self, source: str) -> None:
@@ -293,6 +306,25 @@ class TestScanFile:
findings = asyncio.run(scanner.scan_file("config.py", source, symbols=[]))
assert not any(f["kind"] in {"hardcoded_password", "hardcoded_secret"} for f in findings)
+ @pytest.mark.parametrize(
+ ("name", "expected"),
+ [
+ ("auth", True),
+ ("Password", True),
+ ("jwt", True),
+ ("crypto", True),
+ # A database session, a pytest fixture, a lexer token.
+ ("session", False),
+ ("SESSION", False),
+ ("Token", False),
+ ("get_token", False),
+ ],
+ )
+ def test_security_sensitive_symbol_names(self, name: str, expected: bool) -> None:
+ sym = SimpleNamespace(name=name, start_line=3)
+ kinds = {f["kind"] for f in scan_source("m.py", "", [sym])}
+ assert ("security_sensitive_symbol" in kinds) is expected
+
@pytest.mark.parametrize(
"file_path",
[
@@ -549,7 +581,7 @@ class TestScanFile:
@pytest.mark.parametrize(
("source", "expected"),
[
- ("requests.get(url, verify=False)\n", True),
+ ("import requests\nrequests.get(url, verify=False)\n", True),
('"tls_verify_false": "TLS verification turned off (verify=False).",\n', False),
("# verify=False disables certificate checks\n", False),
],
@@ -559,6 +591,111 @@ class TestScanFile:
findings = asyncio.run(scanner.scan_file("client.py", source, symbols=[]))
assert bool([f for f in findings if f["kind"] == "tls_verify_false"]) is expected
+ @pytest.mark.parametrize(
+ ("source", "expected"),
+ [
+ ("import httpx\nhttpx.Client(\n base_url=url,\n verify=False,\n)\n", True),
+ ('import boto3\nboto3.client("s3", verify=False)\n', True),
+ ('import boto3\nc = boto3.session.Session().client("s3", verify=False)\n', True),
+ ("from requests import get\nget(url, verify=False)\n", True),
+ ("import requests\ns = requests.Session()\ns.get(url, verify=False)\n", True),
+ ("import requests as r\nwith r.Session() as s:\n s.post(url, verify=False)\n", True),
+ ("import requests\nclass C:\n def go(self):\n self.http.get(u, verify=False)\n", True),
+ ("import requests\ns = requests.Session()\ns.verify = False\n", True),
+ ("import requests\nclass Backend:\n verify = False\n", True),
+ # A file with no HTTP client library: whatever ``get`` or ``client`` is, not TLS.
+ ("rows = db.get(key, verify=False)\n", False),
+ ('c = client("s3", verify=False)\n', False),
+ ("session.verify = False\n", False),
+ ("class Backend:\n verify = False\n", False),
+ # An HTTP library imported, but the receiver is some other object.
+ ("import requests\nrows = db.get(key, verify=False)\n", False),
+ ("import requests\nGitRepo(io, git_commit_verify=False)\n", False),
+ ("import requests\nmock_get.assert_called_with(url, verify=False)\n", False),
+ # A function of the project's own whose keyword means something else.
+ ("import httpx\nrows = load_fix_first(session, repo_id, verify=False)\n", False),
+ ],
+ )
+ def test_tls_verify_false_needs_an_http_client(self, source: str, expected: bool) -> None:
+ findings = scan_source("client.py", source)
+ assert bool([f for f in findings if f["kind"] == "tls_verify_false"]) is expected
+
+ def test_tls_verify_false_keeps_the_line_match_when_python_does_not_parse(self) -> None:
+ findings = scan_source("client.py", "def broken(:\nload(x, verify=False)\n")
+ assert [f["line"] for f in findings if f["kind"] == "tls_verify_false"] == [2]
+
+ @pytest.mark.parametrize(
+ ("source", "expected"),
+ [
+ ("rows = con.execute(f\"SELECT id FROM users WHERE name = '{name}'\")\n", True),
+ ('sql = (\n f"SELECT {cols}, name "\n f"FROM users WHERE id = {uid}"\n)\n', True),
+ ("q = f\"SELECT * FROM docs WHERE title LIKE '%{term}%'\"\n", True),
+ ('q = f"SELECT * FROM t WHERE age >= {age} ORDER BY 1"\n', True),
+ # Identifiers, fragments and placeholder lists, which cannot be bound.
+ ('found = con.execute(f"SELECT MAX(at) FROM {table}").fetchone()\n', False),
+ ('q = f"SELECT ref FROM omissions WHERE ref IN ({placeholders})"\n', False),
+ ('q = f"SELECT {column}, COUNT(*) FROM episodes{where} GROUP BY {column}"\n', False),
+ ('q = f"SELECT id FROM t WHERE id = :id", params={"id": 1}\n', False),
+ ('q = f"SELECT a FROM t WHERE b = {{literal}}"\n', False),
+ ('q = f"""SELECT id FROM users WHERE id = {uid}"""\n', True),
+ ('q = f"""SELECT {cols}\n FROM users\n WHERE name = \'{name}\'\n"""\n', True),
+ ('q = f"""SELECT id FROM {table}"""\nx = {"a": 1}\n', False),
+ # A message about SQL, with no query in it.
+ ('reason = f"SELECT * inside a {kind}: the shape changes"\n', False),
+ # Python source quoted inside a string, as a test fixture holds it.
+ ("src = 'q = f\"SELECT * FROM t WHERE id = {uid}\"'\n", False),
+ ],
+ )
+ def test_fstring_sql_needs_a_value_written_into_a_query(self, source: str, expected: bool) -> None:
+ findings = scan_source("db.py", source)
+ assert bool([f for f in findings if f["kind"] == "fstring_sql"]) is expected
+
+ @pytest.mark.parametrize(
+ ("line", "kind"),
+ [
+ ("<div dangerouslySetInnerHTML={{ __html: markup }} />", "unsafe_inner_html"),
+ ("const q = `SELECT * FROM users WHERE id = ${id}`;", "template_literal_sql"),
+ ("const f = new Function(body);", "new_function_call"),
+ ("const a = new https.Agent({ rejectUnauthorized: false });", "reject_unauthorized_false"),
+ ("const k = process.env.NEXT_PUBLIC_API_KEY;", "public_env_secret"),
+ ],
+ )
+ def test_javascript_kinds_fire_in_javascript_not_python(self, line: str, kind: str) -> None:
+ assert kind in {f["kind"] for f in scan_source("web/app.tsx", line + "\n")}
+ quoted = f"CASE = {line!r}\n"
+ assert kind not in {f["kind"] for f in scan_source("src/check.py", quoted)}
+
+ @pytest.mark.parametrize(
+ ("path", "source", "expected"),
+ [
+ ("a.py", "digest = hashlib.md5(data).hexdigest()\n", True),
+ ("a.py", "mac = hmac.new(key, msg, hashlib.sha1)\n", True),
+ ("a.py", 'h = hashlib.new("md5")\n', True),
+ ("a.py", "from hashlib import sha1\nkey = sha1(raw)\n", True),
+ ("a.js", 'const h = crypto.createHash("md5");\n', True),
+ ("A.java", 'MessageDigest md = MessageDigest.getInstance("SHA-1");\n', True),
+ ("a.go", "sum := md5.Sum(data)\n", True),
+ ("A.cs", "using var h = MD5.Create();\n", True),
+ ("a.rb", "Digest::SHA1.hexdigest(s)\n", True),
+ ("a.rs", "let d = md5::compute(bytes);\n", True),
+ ("a.php", "$h = md5($password);\n", True),
+ # A commit id named for its algorithm, and the word in prose.
+ ("a.py", 'sha1 = git("rev-parse", "HEAD")\n', False),
+ ("a.py", "assert repo.head_commit == sha1\n", False),
+ ("a.py", "#: A commit id (sha1 or sha256).\n", False),
+ ("a.py", '"weak_hash": "The words md5 or sha1.",\n', False),
+ ("a.py", "_commit(repo, \"h.py\", \"hashlib.md5(b'')\")\n", False),
+ # Declared a checksum, not a security control.
+ ("a.py", "key = hashlib.sha1(text.encode(), usedforsecurity=False).hexdigest()\n", False),
+ ("a.py", "key = hashlib.sha1(\n text.encode(), usedforsecurity=False\n)\n", False),
+ ("a.py", 'key = hashlib.sha1(f"({a})".encode(), usedforsecurity=False)\n', False),
+ ("a.py", 'key = hashlib.sha1(b"usedforsecurity=False")\n', True),
+ ("a.py", "key = hashlib.sha1(\n text.encode()\n)\nok = sha256(b, usedforsecurity=False)\n", True),
+ ],
+ )
+ def test_weak_hash_is_an_algorithm_in_use(self, path: str, source: str, expected: bool) -> None:
+ assert bool([f for f in scan_source(path, source) if f["kind"] == "weak_hash"]) is expected
+
class TestPersistSecurityFindings:
"""The persist.py wiring: source_map in, idempotent rows out."""
@@ -921,7 +1058,7 @@ class TestEverySnippetIsMasked:
@pytest.mark.parametrize(
("source", "kind"),
[
- (f'API_KEY = "{KEY}" # md5\n', "weak_hash"),
+ (f'API_KEY = "{KEY}"; h = md5(b)\n', "weak_hash"),
(f'os.system(cmd); secret = "{KEY}"\n', "os_system"),
(f'x = eval(y); secret = "{KEY}"\n', "eval_call"),
(f'subprocess.run(cmd, secret = "{KEY}",\n shell=True)\n', "subprocess_shell_true"),
@@ -997,9 +1134,10 @@ class TestEverySnippetIsMasked:
_assert_no_raw(hit["snippet"], body[::-1])
def test_stray_hit_on_a_pem_body_line_is_masked(self) -> None:
- body = "MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgE+md5/AoIBAQC7VJTUt9Us8cKj"
+ # pragma: allowlist nextline secret
+ body = "MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgE+AKIAQ2Z7RT4WXY5BCDFG/AoIBAQC7VJTUt9Us8cKj"
source = f"-----BEGIN RSA PRIVATE KEY-----\n{body}\n-----END RSA PRIVATE KEY-----\n"
- (hit,) = [f for f in scan_source("id_rsa", source) if f["kind"] == "weak_hash"]
+ (hit,) = [f for f in scan_source("id_rsa", source) if f["kind"] == "aws_access_key"]
assert hit["line"] == 2
_assert_no_raw(hit["snippet"], body)