Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
fix(security): report a sink only where it can run (#3534)
Raghav Chamadiyaf81791aa5246
diff --git a/docs/layers/SECURITY.md b/docs/layers/SECURITY.mdindex fddfc89fd..e1b9c7576 100644--- a/docs/layers/SECURITY.md+++ b/docs/layers/SECURITY.md@@ -49,8 +49,8 @@ material, or under a directory named `test`, `tests`, `__tests__`, `__test__`,| Kind | Severity | Matches ||------|----------|---------|| `eval_call` | high | `eval(...)`, including a receiver chain (`vm.eval(`) |-| `exec_call` | high | `exec(...)`. Outside Python, only a call through a name bound to `child_process` (`exec`, `execFile`, `execSync`) |-| `pickle_loads` | high | `pickle.loads` |+| `exec_call` | high | `exec(...)`. Outside Python, only a call through a name bound to `child_process` (`exec`, `execSync`). `execFile` runs no shell and does not fire |+| `pickle_loads` | high | `pickle.loads`, except a round trip of an own object (`pickle.loads(pickle.dumps(x))`) || `subprocess_shell_true` | high | `subprocess.*` with `shell=True`, also across lines || `os_system` | high | `os.system` || `hardcoded_password` | high | a quoted literal assigned to a name containing `password`, any case |@@ -66,14 +66,14 @@ material, or under a directory named `test`, `tests`, `__tests__`, `__test__`,| `private_key_pem` | high | a `-----BEGIN ... PRIVATE KEY-----` header followed by a base64 body line (an escaped `\n` counts) || `public_env_secret` | high | `API_KEY`/`SECRET`/`TOKEN`/`PASSWORD` behind `NEXT_PUBLIC_` or `VITE_`, excluding `..._ANON_...` || `new_function_call` | high | `new Function(...)` |-| `fstring_sql` | med | a Python f-string with `SELECT` and an interpolation |+| `fstring_sql` | med | a Python f-string query (`SELECT` with `FROM`) that interpolates a value after a comparison or `LIKE` (`WHERE id = {uid}`). A table, column, fragment or placeholder list cannot be bound, so interpolating one does not fire || `concat_sql` | med | `.execute("SELECT ... +` || `template_literal_sql` | med | a JS/TS template literal with `SELECT`+`FROM` or `UPDATE`+`SET` and an interpolation |-| `tls_verify_false` | med | `verify = False` |+| `tls_verify_false` | med | `verify=False` passed to an HTTP client call or set as `verify`, in a file that imports an HTTP client library (`requests`, `httpx`, `aiohttp`, `urllib3`, boto3). A call through a plain name must go through that import or a value built from it || `reject_unauthorized_false` | med | `rejectUnauthorized: false` || `unsafe_inner_html` | med | `__html:` (React's `dangerouslySetInnerHTML`) assigned a non-literal |-| `weak_hash` | low | the words `md5` or `sha1` |-| `security_sensitive_symbol` | low | a symbol named with `auth`, `token`, `password`, `jwt`, `session` or `crypto` |+| `weak_hash` | low | MD5 or SHA-1 in use in code: a call (`hashlib.md5(`), a type (`MD5.Create()`) or the algorithm named to a hashing call (`createHash("sha1")`). A call passing `usedforsecurity=False` does not fire |+| `security_sensitive_symbol` | low | a symbol named `auth`, `password`, `jwt` or `crypto`. A bare `session` or `token` is usually a database session or a lexer token and does not fire |`security_sensitive_symbol` is informational: it marks where security codelives, not a problem.@@ -95,13 +95,18 @@ and the in-change commit scan report only these by default.comment is still a leak.- **Prose files** (`.md`, `.mdx`, `.rst`, `.txt`, `.adoc`) are scanned for thesecret kinds only.+- **JavaScript kinds stay in JavaScript.** `unsafe_inner_html`,+ `template_literal_sql`, `new_function_call`, `reject_unauthorized_false` and+ `public_env_secret` do not fire in Python files, where such a line can only be+ quoted text.- **Placeholders are skipped**: a keyword value that is elided or templated- (`...`, `<your key>`, `xxx`, `example`, `placeholder`, `dummy`, `fake`,+ (`...`, `<your key>`, `xxx`, `example`, `placeholder`, `dummy`, `fake`, `redacted`,`changeme`, `{{ }}`, `${}`), shorter than eight characters, a shortsingle-case word, or a snake_case key name. A value a vendor kind alreadyreports is not reported twice.-- **`weak_hash`, the SQL kinds, the TLS kinds and `unsafe_inner_html`** run on- raw source, comments included.+- **`concat_sql`, `template_literal_sql` and `unsafe_inner_html`** run on raw+ source, comments included. `fstring_sql` and `weak_hash` need their match to+ start in code, so a comment or a fixture string quoting one does not fire.### Working tree versus history@@ -143,7 +148,7 @@ its first four characters and `****`. The raw value is never stored.2. High working-tree kinds, as places to read. The layer found a call; youdecide whether it is reachable.3. `security_sensitive_symbol`, as a map of where auth and crypto code lives.-4. Low findings last. `weak_hash` is a word match.+4. Low findings last. `weak_hash` cannot tell a checksum from a password hash.A low finding count means the registry did not match. On most repos, in mostlanguages, that is all it means.@@ -248,7 +253,8 @@ result with that in mind.eight vendor shapes are invisible unless you add custom patterns. Use gitleaksor trufflehog for secret scanning proper.- **No dependency or CVE scanning.**-- **False positives happen.** `weak_hash` fires on `md5` in a comment.+- **False positives happen.** `weak_hash` fires on an MD5 checksum unless the+ call passes `usedforsecurity=False`.`unsafe_inner_html` ships at `med` because it cannot tell a name pointing ata pinned constant from a dynamic value.- **The five JS/TS kinds were checked by hand** on a 17-repository, 1,109-filediff --git a/packages/cli/src/repowise/cli/commands/augment_cmd/_shared.py b/packages/cli/src/repowise/cli/commands/augment_cmd/_shared.pyindex 2d3674686..448880b7c 100644--- a/packages/cli/src/repowise/cli/commands/augment_cmd/_shared.py+++ b/packages/cli/src/repowise/cli/commands/augment_cmd/_shared.py@@ -91,7 +91,7 @@ def _ledger_key(surface: str, category: str, text: str) -> str:"""import hashlib- digest = hashlib.sha1(text.encode("utf-8", "replace")).hexdigest()[:12]+ digest = hashlib.sha1(text.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:12]return f"{surface}:{category}:{digest}"diff --git a/packages/cli/src/repowise/cli/commands/augment_cmd/command.py b/packages/cli/src/repowise/cli/commands/augment_cmd/command.pyindex 8eed8b4de..332eee5ac 100644--- a/packages/cli/src/repowise/cli/commands/augment_cmd/command.py+++ b/packages/cli/src/repowise/cli/commands/augment_cmd/command.py@@ -364,7 +364,7 @@ def _claim_emission(event: str, context: str) -> bool:import timetry:- key = hashlib.sha1(f"{event}\x00{context}".encode()).hexdigest()[:16]+ key = hashlib.sha1(f"{event}\x00{context}".encode(), usedforsecurity=False).hexdigest()[:16]marker = Path(tempfile.gettempdir()) / f".repowise-augment-{key}"now = time.time()try:diff --git a/packages/cli/src/repowise/cli/commands/augment_cmd/read_state.py b/packages/cli/src/repowise/cli/commands/augment_cmd/read_state.pyindex 5daabf7ef..2a341ca59 100644--- a/packages/cli/src/repowise/cli/commands/augment_cmd/read_state.py+++ b/packages/cli/src/repowise/cli/commands/augment_cmd/read_state.py@@ -78,7 +78,7 @@ def _session_state_path(repo_path: Path, session_id: str = "") -> Path:"""This session's state file. One per session id — see :data:`_STATE_DIR`."""import hashlib- digest = hashlib.sha1(session_id.encode("utf-8", "replace")).hexdigest()[:16]+ digest = hashlib.sha1(session_id.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:16]return Path(repo_path) / ".repowise" / _STATE_DIR / f"{digest}.json"diff --git a/packages/cli/src/repowise/cli/commands/augment_cmd/reread.py b/packages/cli/src/repowise/cli/commands/augment_cmd/reread.pyindex 6cdeeb5e2..d1cbebe01 100644--- a/packages/cli/src/repowise/cli/commands/augment_cmd/reread.py+++ b/packages/cli/src/repowise/cli/commands/augment_cmd/reread.py@@ -89,7 +89,7 @@ def content_digest(content: str) -> str:"""import hashlib- return hashlib.sha1(content.encode("utf-8", "replace")).hexdigest()[:12]+ return hashlib.sha1(content.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:12]def _range_label(offset: object, limit: object) -> str:diff --git a/packages/cli/src/repowise/cli/hook_ledger.py b/packages/cli/src/repowise/cli/hook_ledger.pyindex 75de7f10d..1f8a1de0a 100644--- a/packages/cli/src/repowise/cli/hook_ledger.py+++ b/packages/cli/src/repowise/cli/hook_ledger.py@@ -96,7 +96,9 @@ def emitting_build() -> str:# The directory rather than ``__file__``: two modules in one install# must not read as two installs.- install = hashlib.sha1(os.path.dirname(__file__).encode("utf-8", "replace")).hexdigest()+ install = hashlib.sha1(+ os.path.dirname(__file__).encode("utf-8", "replace"), usedforsecurity=False+ ).hexdigest()_BUILD = f"{__version__}+{install[:8]}"return _BUILDdiff --git a/packages/core/src/repowise/core/analysis/actions/model.py b/packages/core/src/repowise/core/analysis/actions/model.pyindex c55812d0b..80b3d9a4d 100644--- a/packages/core/src/repowise/core/analysis/actions/model.py+++ b/packages/core/src/repowise/core/analysis/actions/model.py@@ -203,7 +203,7 @@ class Action:def action_id(self) -> str:identity = self.identity or f"{self.target_path}:{self.target_symbol or ''}"key = f"{self.rule}:{identity}"- return "act_" + hashlib.sha1(key.encode("utf-8")).hexdigest()[:16]+ return "act_" + hashlib.sha1(key.encode("utf-8"), usedforsecurity=False).hexdigest()[:16]def as_dict(self) -> dict[str, Any]:return {@@ -256,4 +256,4 @@ class RuleOutcome:def fingerprint(*parts: object) -> str:raw = "|".join(str(p) for p in parts)- return hashlib.sha1(raw.encode("utf-8")).hexdigest()[:12]+ return hashlib.sha1(raw.encode("utf-8"), usedforsecurity=False).hexdigest()[:12]diff --git a/packages/core/src/repowise/core/analysis/security_gate.py b/packages/core/src/repowise/core/analysis/security_gate.pyindex a99d9f237..4a694670d 100644--- a/packages/core/src/repowise/core/analysis/security_gate.py+++ b/packages/core/src/repowise/core/analysis/security_gate.py@@ -94,7 +94,7 @@ _BINARY_SNIFF = 8000#: is absent on purpose: it needs parsed symbols and never gates._RULE_TEXT: dict[str, str] = {"eval_call": "A call to eval.",- "exec_call": "A call to exec (outside Python, in a file that names child_process).",+ "exec_call": "A call to exec (outside Python, child_process exec or execSync).","pickle_loads": "pickle.loads, which runs code from the data it loads.","subprocess_shell_true": "A subprocess call with shell=True.","os_system": "os.system, which runs its argument through a shell.",@@ -109,10 +109,13 @@ _RULE_TEXT: dict[str, str] = {"azure_devops_pat": "An Azure DevOps personal access token.","git_url_credentials": "A password or token embedded in a git remote URL.","private_key_pem": "A PEM private key with its body.",- "fstring_sql": "An f-string holding SQL and an interpolation.",+ "fstring_sql": "An f-string query that interpolates a value where a bound parameter belongs.","concat_sql": "SQL built by string concatenation.",- "tls_verify_false": "TLS certificate verification turned off (verify=False).",- "weak_hash": "The words md5 or sha1.",+ "tls_verify_false": "TLS certificate verification turned off (verify=False on an HTTP client).",+ "weak_hash": (+ "MD5 or SHA-1 used in code: a call, a type, or the algorithm named to a hashing "+ "call. A call passing usedforsecurity=False is exempt."+ ),"unsafe_inner_html": "__html set from a non-literal value.","template_literal_sql": "A template literal holding SQL and an interpolation.","public_env_secret": "A secret-shaped name behind a NEXT_PUBLIC_ or VITE_ prefix.",diff --git a/packages/core/src/repowise/core/analysis/security_scan.py b/packages/core/src/repowise/core/analysis/security_scan.pyindex c9a9a834d..33ba204b6 100644--- a/packages/core/src/repowise/core/analysis/security_scan.py+++ b/packages/core/src/repowise/core/analysis/security_scan.py@@ -103,6 +103,7 @@ _CREDENTIAL_SUBSTRING_PLACEHOLDERS: tuple[str, ...] = ("...","\u2026", # a typographic ellipsis"fixture",+ "redacted",)# Broader than ``is_test_related_path`` on purpose. That one asks whether a file@@ -177,7 +178,9 @@ _CALL_KINDS = frozenset(kind for _, kind, _ in _CALL_PATTERNS)# The dangerous call in JavaScript comes from ``child_process``, so only a call# through a name bound to that module counts: a namespace (``cp.exec(``), a named# import (``exec(`` after ``import { exec } from "child_process"``) or the-# ``require`` result itself. A file's own function named ``exec``, or a regex's+# ``require`` result itself. Only ``exec`` and ``execSync`` run a shell;+# ``execFile`` passes its arguments as argv, like a Python ``subprocess.run``+# without ``shell=True``, which does not fire either. A file's own function named ``exec``, or a regex's# ``.exec`` next to an unrelated ``spawn`` import, is not a shell. ``eval`` needs# no such gate: it is a genuine global there._CP_MODULE = r"""['"](?:node:)?child_process['"]"""@@ -191,9 +194,9 @@ _CP_NAMED = re.compile(rf"|(?:const|let|var)\s*\{{([^}}]*)\}}\s*=\s*require\(\s*{_CP_MODULE}\s*\)")_CP_REQUIRE_CALL = re.compile(- rf"require\(\s*{_CP_MODULE}\s*\)\s*\.\s*exec(?:File)?(?:Sync)?\s*\("+ rf"require\(\s*{_CP_MODULE}\s*\)\s*\.\s*exec(?:Sync)?\s*\(")-_EXEC_EXPORT = re.compile(r"exec(?:File)?(?:Sync)?")+_EXEC_EXPORT = re.compile(r"exec(?:Sync)?")def _child_process_exec_calls(source: str, masked: str) -> list[int]:@@ -214,7 +217,7 @@ def _child_process_exec_calls(source: str, masked: str) -> list[int]:offsets = [m.start() for m in _CP_REQUIRE_CALL.finditer(source) if not masked[m.start()].isspace()]names = [re.escape(n) for n in namespaces]if names:- receiver = rf"(?<![\w$.])(?:{'|'.join(names)})\s*\.\s*exec(?:File)?(?:Sync)?\s*\("+ receiver = rf"(?<![\w$.])(?:{'|'.join(names)})\s*\.\s*exec(?:Sync)?\s*\("offsets += [m.start() for m in re.finditer(receiver, masked)]if functions:bare = rf"(?<![\w$.])(?:{'|'.join(map(re.escape, functions))})\s*\("@@ -223,7 +226,9 @@ def _child_process_exec_calls(source: str, masked: str) -> list[int]:_PATTERNS: list[tuple[re.Pattern, str, str]] = [*_CALL_PATTERNS,- (re.compile(r"pickle\.loads"), "pickle_loads", "high"),+ # Not ``pickle.loads(pickle.dumps(x))``: a round trip of the program's own+ # object, which is how a copy or a picklability test is written.+ (re.compile(r"pickle\.loads(?!\s*\(\s*pickle\.dumps\s*\()"), "pickle_loads", "high"),(re.compile(r"subprocess\..*shell\s*=\s*True"), "subprocess_shell_true", "high"),(re.compile(r"os\.system"), "os_system", "high"),# Case-insensitive: a credential pinned in source is usually written as a@@ -313,7 +318,27 @@ _PATTERNS: list[tuple[re.Pattern, str, str]] = [(re.compile(r'f[\'"].*SELECT.*\{.*\}'), "fstring_sql", "med"),(re.compile(r"\.execute\(\s*[\'\"]\s*SELECT.*\+"), "concat_sql", "med"),(re.compile(r"verify\s*=\s*False"), "tls_verify_false", "med"),- (re.compile(r"\bmd5\b|\bsha1\b"), "weak_hash", "low"),+ # A weak algorithm put to use, not the word: a call (``md5(``, ``hashlib.sha1(``,+ # ``DigestUtils.md5Hex(``, ``MD5_Init(``), a type or module function+ # (``MD5.Create()``, ``md5.Sum(``, ``Sha1::new()``, ``Digest::SHA1``), ``hashlib.md5``+ # handed over as a digest, or the algorithm named to a hashing call+ # (``createHash("md5")``, ``getInstance("SHA-1")``). A ``sha1`` variable+ # holding a commit id, or the word in a comment, is neither.+ (+ re.compile(+ r"(?<![\w$])(?i:md5|sha1)(?i:_?(?:hex|file|hash|digest|init|update|final))?\s*\("+ r"|(?<![\w$])(?i:md5|sha1)\s*(?:\.|::)\s*"+ r"(?:New|Sum|Create|new|hexdigest|digest|compute)\b"+ r"|\b(?:MD5|SHA1)(?:CryptoServiceProvider|Managed)\b"+ r"|\bDigest::(?:MD5|SHA1)\b"+ r"|\bhashlib\s*\.\s*(?:md5|sha1)\b"+ r"|\b(?:new|createHash|createHmac|getInstance|hash|hash_hmac|hash_file|digest|Create)"+ r"\s*\(\s*['\"](?i:md5|sha-?1)['\"]"+ r"|\bdigestmod\s*=\s*['\"](?i:md5|sha1)['\"]"+ ),+ "weak_hash",+ "low",+ ),# -- JS/TS patterns (#1935 Tier 1) -------------------------------------# Measured on the same 17-repo, 1109-file corpus as the exec_call/secret# fixes in #1947. Three of these five needed tightening before they were@@ -401,8 +426,10 @@ _SPANNING_PATTERNS: list[tuple[re.Pattern, str, str]] = [),]-# Symbol names that are informational security hotspots-_SYMBOL_KEYWORDS = re.compile(r"\b(auth|token|password|jwt|session|crypto)\b", re.IGNORECASE)+# Symbol names that are informational security hotspots. Not ``session`` or+# ``token``: a symbol named only that is a database session, a pytest fixture or+# a lexer token far more often than auth code.+_SYMBOL_KEYWORDS = re.compile(r"\b(auth|password|jwt|crypto)\b", re.IGNORECASE)# Kinds that name a dangerous call. They are matched against source with# comments and string literals blanked, so a docstring, a comment or a regex@@ -418,6 +445,207 @@ _MASKED_KINDS = frozenset(})+# JavaScript sinks. Python has no JSX, template literals, ``new Function`` or+# Node TLS options, and no bundler reads a ``.py`` file into a client, so a+# Python file holding one of these is quoting it, most often as a test fixture.+_JS_ONLY_KINDS = frozenset(+ {+ "unsafe_inner_html",+ "template_literal_sql",+ "new_function_call",+ "reject_unauthorized_false",+ "public_env_secret",+ }+)++# ``verify=False`` turns TLS off only when an HTTP client receives it, so the+# file must import an HTTP client library, and a call through a plain name must+# go through that import or a value built from it (``s = requests.Session()``).+# Any other function is free to name a keyword ``verify``. Ceiling: a project's+# own wrapper that forwards ``verify``, imported from another module, is not seen.+_HTTP_LIBS = frozenset(+ {"requests", "httpx", "aiohttp", "urllib3", "boto3", "botocore", "aiobotocore", "aioboto3"}+)+_HTTP_CALLS = frozenset(+ {+ "get",+ "post",+ "put",+ "patch",+ "delete",+ "head",+ "options",+ "request",+ "send",+ "stream",+ "urlopen",+ "Client",+ "AsyncClient",+ "Session",+ "HTTPTransport",+ "AsyncHTTPTransport",+ "client",+ "resource",+ }+)++# ``fstring_sql`` asks for a query (``SELECT`` with its ``FROM``) that+# interpolates a value where a bound parameter belongs: after a comparison or+# ``LIKE``. A table, a column, a fragment or a placeholder list cannot be bound,+# so building those into the text is the only way to write them, and a value+# interpolated there has no parameter fix to offer. Ceiling: ``IN ({ids})`` of+# joined values, and an identifier taken from a request, are not seen.+_SQL_FROM = re.compile(r"\bFROM\b")+_SQL_VALUE = re.compile(r"""(?:[=<>]|\b(?i:LIKE|ILIKE))\s*['"%]*\{(?!\{)""")+# A string literal continuing the one before it (implicit concatenation).+_STRING_CONTINUATION = re.compile(r"""\A\s*[rRbBfFuU]{0,2}['"]""")+++def _interpolates_sql_value(line: str, start: int, lines: list[str], lineno: int) -> bool:+ """Whether the f-string query opening at *start* on *line* puts a value in its text.++ Only the literal is read, and the one opening the next line when the query+ runs on into it.+ """+ triple = line[start + 1 : start + 4]+ if triple in ('"""', "'''"):+ # A triple-quoted query runs to its closing delimiter, often lines on.+ text = "\n".join(lines[lineno - 1 : lineno + 29])+ end = text.find(triple, start + 4)+ query = text[text.find("SELECT", start) : end if end != -1 else len(text)]+ return bool(_SQL_FROM.search(query) and _SQL_VALUE.search(query))+ body = start + 2 # past ``f`` and the opening quote+ query = line[line.find("SELECT", start) : _string_end(line, body)]+ following = lines[lineno] if lineno < len(lines) else ""+ if (cont := _STRING_CONTINUATION.match(following)) is not None:+ query += "\n" + following[cont.end() : _string_end(following, cont.end())]+ return bool(_SQL_FROM.search(query) and _SQL_VALUE.search(query))+++def _dotted(node: ast.AST) -> str:+ """``a.b.c`` for a name or attribute chain, else ``""``."""+ if isinstance(node, ast.Name):+ return node.id+ if isinstance(node, ast.Attribute) and (base := _dotted(node.value)):+ return f"{base}.{node.attr}"+ return ""+++def _root(node: ast.AST) -> ast.AST:+ """The expression a chain of calls and attributes starts from."""+ while isinstance(node, (ast.Call, ast.Attribute, ast.Await)):+ node = node.func if isinstance(node, ast.Call) else node.value+ return node+++def _http_bindings(tree: ast.AST) -> set[str]:+ """Names and ``self.x`` targets bound to an HTTP client library or a value built from it."""+ bound: set[str] = set()+ for node in ast.walk(tree):+ if isinstance(node, ast.Import):+ for alias in node.names:+ if alias.name.split(".")[0] in _HTTP_LIBS:+ bound.add(alias.asname or alias.name.split(".")[0])+ elif (+ isinstance(node, ast.ImportFrom)+ and node.level == 0+ and (node.module or "").split(".")[0] in _HTTP_LIBS+ ):+ bound.update(alias.asname or alias.name for alias in node.names)+ if not bound:+ return bound+ # Twice, so a value built from a bound value (``c = s.client()``) is bound too.+ for _ in range(2):+ for node in ast.walk(tree):+ if isinstance(node, ast.Assign):+ pairs = [(t, node.value) for t in node.targets]+ elif isinstance(node, (ast.AnnAssign, ast.NamedExpr)) and node.value is not None:+ pairs = [(node.target, node.value)]+ elif isinstance(node, (ast.With, ast.AsyncWith)):+ pairs = [(i.optional_vars, i.context_expr) for i in node.items if i.optional_vars]+ else:+ continue+ for target, value in pairs:+ if _dotted(_root(value)) in bound and (name := _dotted(target)):+ bound.add(name)+ return bound+++def _tls_verify_off_lines(source: str) -> set[int] | None:+ """Lines where ``verify=False`` reaches an HTTP client; None when *source* does not parse."""+ try:+ tree = ast.parse(source)+ except SyntaxError:+ return None+ bound = _http_bindings(tree)+ if not bound:+ return set()+ found: set[int] = set()+ for node in ast.walk(tree):+ if isinstance(node, ast.Call):+ func = node.func+ if isinstance(func, ast.Attribute):+ name, receiver = func.attr, _root(func.value)+ # A plain name must be bound; ``self.x`` and call results are+ # taken on trust once the file uses an HTTP client library.+ reaches = (+ _dotted(receiver) in bound+ or _dotted(func.value) in bound+ or not isinstance(receiver, ast.Name)+ or receiver.id in ("self", "cls")+ )+ elif isinstance(func, ast.Name):+ name, reaches = func.id, func.id in bound+ else:+ continue+ if name in _HTTP_CALLS and reaches:+ found.update(+ kw.lineno+ for kw in node.keywords+ if kw.arg == "verify"+ and isinstance(kw.value, ast.Constant)+ and kw.value.value is False+ )+ elif (+ isinstance(node, ast.Assign)+ and isinstance(node.value, ast.Constant)+ and node.value.value is False+ ):+ # ``session.verify = False``, or a ``verify = False`` setting handed+ # to a client later.+ found.update(+ target.lineno+ for target in node.targets+ if (isinstance(target, ast.Attribute) and target.attr == "verify")+ or (isinstance(target, ast.Name) and target.id == "verify")+ )+ return found+++# Python's own declaration that a digest is a checksum or a cache key, not a+# security control (``hashlib.sha1(data, usedforsecurity=False)``).+_NOT_FOR_SECURITY = re.compile(r"\busedforsecurity\s*=\s*False\b")+++def _call_text(masked: list[str], lineno: int, start: int, limit: int = 600) -> str:+ """The code from *start* on line *lineno* through the first call's closing paren.++ Read from *masked* (strings and comments blanked, offsets kept), so a ``)``+ or the keyword inside a string neither ends the call nor counts as code.+ Bounded by *limit* characters.+ """+ code = "\n".join(masked[lineno - 1 : lineno + 9])[start : start + limit]+ depth = 0+ for i, char in enumerate(code):+ if char == "(":+ depth += 1+ elif char == ")":+ depth -= 1+ if depth <= 0:+ return code[: i + 1]+ return code++# Prose is documentation, not executable code. Only secret kinds scan it,# because a key pasted into a README is still a leak._PROSE_EXTENSIONS = tuple(DOC_EXTENSIONS)@@ -914,6 +1142,8 @@ def scan_source(file_path: str, source: str, symbols: Iterable[Any] = ()) -> lis# alike to every lookaround in ``_ANY_PATTERN``, so a file the whole-source# search rejects has no matching line, and most files are such.is_low_sev_file = _is_low_severity_path(file_path)+ is_python = file_path.lower().endswith((".py", ".pyi"))+ tls_lines: list[set[int] | None] = []scan_lines = lines if _ANY_PATTERN.search(source) else []for lineno, line in enumerate(scan_lines, start=1):if not _ANY_PATTERN.search(line):@@ -925,8 +1155,28 @@ def scan_source(file_path: str, source: str, symbols: Iterable[Any] = ()) -> liscontinueif is_prose and kind not in SECRET_KINDS:continue+ if is_python and kind in _JS_ONLY_KINDS:+ continueif kind in _MASKED_KINDS:match = pattern.search(masked_line(lineno, line))+ if match and kind == "tls_verify_false" and is_python:+ if not tls_lines:+ tls_lines.append(_tls_verify_off_lines(source))+ if tls_lines[0] is not None and lineno not in tls_lines[0]:+ match = None+ elif kind == "fstring_sql":+ # The ``f`` must open a literal in code, not sit inside another+ # string, and the query must interpolate a value.+ match = _match_starting_in(pattern, line, masked_line(lineno, line))+ if match and not _interpolates_sql_value(line, match.start(), lines, lineno):+ match = None+ elif kind == "weak_hash":+ # Written in code, and not declared a non-security use.+ match = _match_starting_in(pattern, line, masked_line(lineno, line))+ if match and _NOT_FOR_SECURITY.search(+ _call_text(mask()[1], lineno, match.start())+ ):+ match = Noneelif kind in _KEYWORD_KINDS:# ``password = "..."`` in a comment or docstring is an example.match = _match_starting_in(pattern, line, masked_line(lineno, line))diff --git a/packages/core/src/repowise/core/sessions/efficacy.py b/packages/core/src/repowise/core/sessions/efficacy.pyindex 74f6fde95..e5c4be5d1 100644--- a/packages/core/src/repowise/core/sessions/efficacy.py+++ b/packages/core/src/repowise/core/sessions/efficacy.py@@ -234,7 +234,7 @@ def ledger_key(surface: str, category: str, text: str) -> str:the emitted text is what lets a transcript firing find the row the livehook wrote for it."""- digest = hashlib.sha1(text.encode("utf-8", "replace")).hexdigest()[:12]+ digest = hashlib.sha1(text.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:12]return f"{surface}:{category}:{digest}"diff --git a/tests/unit/analysis/test_security_gate.py b/tests/unit/analysis/test_security_gate.pyindex e7125886d..75dcd6489 100644--- a/tests/unit/analysis/test_security_gate.py+++ b/tests/unit/analysis/test_security_gate.py@@ -29,6 +29,8 @@ from repowise.core.ci.baseline import read_baseline, read_entriesKEY = "AKIA" + "QZXNRTVYWMPKLBHG"PEM_BODY = "MIIEpAIBAAKCAQEAu1SU1LfVLPHCozMxH2Mo4lgOEePzNm0tRgeLezV6ffAt0gunVTLw"+# A key shape that can sit inside base64, so a finding lands on a body line.+STRAY = "AKIAQ2Z7RT4WXY5BCDFG" # pragma: allowlist secretdef _git(cwd, *args: str) -> str:@@ -338,11 +340,11 @@ def test_only_high_entropy_shapes_hash_their_material():@pytest.mark.parametrize(("line", "raw"),[- ("PASSWORD = 'Tr0ub4dor\"TAILSECRET9876' # md5\n", "TAILSECRET9876"),- ('secret = "token = \'RAWVALUE12345678\'" # md5\n', "RAWVALUE12345678"),- ('ENV["NEXT_PUBLIC_API_KEY"] = "RAWVALUE12345678" # md5\n', "RAWVALUE12345678"),- ("ENV NEXT_PUBLIC_API_KEY RAWVALUE12345678 md5\n", "RAWVALUE12345678"),- ('k = process.env.NEXT_PUBLIC_API_KEY ?? "RAWVALUE12345678" // md5\n', "RAWVALUE12345678"),+ ("PASSWORD = 'Tr0ub4dor\"TAILSECRET9876'; md5(b)\n", "TAILSECRET9876"),+ ('secret = "token = \'RAWVALUE12345678\'"; md5(b)\n', "RAWVALUE12345678"),+ ('ENV["NEXT_PUBLIC_API_KEY"] = "RAWVALUE12345678"; md5(b)\n', "RAWVALUE12345678"),+ ("ENV NEXT_PUBLIC_API_KEY RAWVALUE12345678 md5(b)\n", "RAWVALUE12345678"),+ ('k = process.env.NEXT_PUBLIC_API_KEY ?? "RAWVALUE12345678"; md5(b)\n', "RAWVALUE12345678"),],)def test_no_raw_value_escapes_the_mask(line, raw):@@ -355,9 +357,9 @@ def test_no_raw_value_escapes_the_mask(line, raw):@pytest.mark.parametrize("source",[- f"key: |\n -----BEGIN RSA PRIVATE KEY-----\n {PEM_BODY}md5\n -----END RSA PRIVATE KEY-----\n",- f"-----BEGIN RSA PRIVATE KEY-----\n{PEM_BODY}\n{PEM_BODY[::-1]}md5\n",- f"KEY=-----BEGIN RSA PRIVATE KEY----- {PEM_BODY} md5 -----END RSA PRIVATE KEY-----\n",+ f"key: |\n -----BEGIN RSA PRIVATE KEY-----\n {PEM_BODY}+{STRAY}\n -----END RSA PRIVATE KEY-----\n",+ f"-----BEGIN RSA PRIVATE KEY-----\n{PEM_BODY}\n{PEM_BODY[::-1]}+{STRAY}\n",+ f"KEY=-----BEGIN RSA PRIVATE KEY----- {PEM_BODY} {STRAY} -----END RSA PRIVATE KEY-----\n",],)def test_pem_body_lines_never_reach_a_snippet(source):@@ -434,9 +436,9 @@ def test_sarif_suppresses_the_baseline_and_drops_history_lines():@pytest.mark.parametrize(("line", "raw"),[- ("ENV NEXT_PUBLIC_API_KEY p@ssw0rd!Xy#Zq9 md5\n", "p@ssw0rd!Xy#Zq9"),- ('ENV["NEXT_PUBLIC_TOKEN"] = "Ab$9!kLm@2#Qr%zz" # md5\n', "Ab$9!kLm@2#Qr%zz"),- ('x = NEXT_PUBLIC_SECRET ?? "p@ss:w0rd!2024x" // md5\n', "p@ss:w0rd!2024x"),+ ("ENV NEXT_PUBLIC_API_KEY p@ssw0rd!Xy#Zq9 md5(b)\n", "p@ssw0rd!Xy#Zq9"),+ ('ENV["NEXT_PUBLIC_TOKEN"] = "Ab$9!kLm@2#Qr%zz"; md5(b)\n', "Ab$9!kLm@2#Qr%zz"),+ ('x = NEXT_PUBLIC_SECRET ?? "p@ss:w0rd!2024x"; md5(b)\n', "p@ss:w0rd!2024x"),("password = \"ab'SuperSecretPassword123\"; eval(x)\n", "SuperSecretPassword123"),],)diff --git a/tests/unit/analysis/test_security_scan.py b/tests/unit/analysis/test_security_scan.pyindex 1df7b9900..5a708172b 100644--- a/tests/unit/analysis/test_security_scan.py+++ b/tests/unit/analysis/test_security_scan.py@@ -75,6 +75,18 @@ async def _rows(sf) -> list[tuple]:class TestScanFile:+ @pytest.mark.parametrize(+ ("source", "expected"),+ [+ ("obj = pickle.loads(request.body)\n", True),+ ("obj = pickle.loads(blob); copy = pickle.dumps(obj)\n", True),+ ("assert pickle.loads(pickle.dumps(config)) == config\n", False),+ ("gb2 = pickle.loads(pickle.dumps(gb, protocol=pickle.HIGHEST_PROTOCOL))\n", False),+ ],+ )+ def test_pickle_round_trip_is_not_a_load(self, source: str, expected: bool) -> None:+ assert bool([f for f in scan_source("a.py", source) if f["kind"] == "pickle_loads"]) is expected+def test_line_patterns_fire_on_real_source(self) -> None:scanner = SecurityScanner(session=None, repo_id="r1") # type: ignore[arg-type]findings = asyncio.run(scanner.scan_file("a.py", SNIPPY.decode(), symbols=[]))@@ -173,13 +185,12 @@ class TestScanFile:[('import { exec } from "child_process";\nexec(cmd);\n', {"exec_call"}),('const cp = require("child_process");\ncp.execSync(cmd);\n', {"exec_call"}),- (- 'import { execFile } from "node:child_process";\nexecFile(bin, args);\n',- {"exec_call"},- ),+ # execFile passes argv and runs no shell, like subprocess without shell=True.+ ('import { execFile } from "node:child_process";\nexecFile(bin, args);\n', set()),('import { exec as run } from "child_process";\nrun(cmd);\n', {"exec_call"}),('const { execSync: sh } = require("child_process");\nsh(cmd);\n', {"exec_call"}),- ('import * as cp from "node:child_process";\ncp.execFile(bin);\n', {"exec_call"}),+ ('import * as cp from "node:child_process";\ncp.execFile(bin);\n', set()),+ ('import * as cp from "node:child_process";\ncp.exec(cmd);\n', {"exec_call"}),('require("child_process").exec(cmd);\n', {"exec_call"}),# A regex's ``.exec`` in a file that also spawns is not bound to the module.('require("child_process");\nconst m = /a/.exec(s);\n', set()),@@ -285,6 +296,8 @@ class TestScanFile:'api_key = "your-api-key-here"\n','api_key = "<YOUR_API_KEY>"\n','password = "fixture-value"\n',+ 'api_key = "«redacted:sk-repo»"\n',+ 'OPENAI_API_KEY = "[REDACTED]-k3y-0f-mine"\n',],)def test_credential_placeholder_filtering(self, source: str) -> None:@@ -293,6 +306,25 @@ class TestScanFile:findings = asyncio.run(scanner.scan_file("config.py", source, symbols=[]))assert not any(f["kind"] in {"hardcoded_password", "hardcoded_secret"} for f in findings)+ @pytest.mark.parametrize(+ ("name", "expected"),+ [+ ("auth", True),+ ("Password", True),+ ("jwt", True),+ ("crypto", True),+ # A database session, a pytest fixture, a lexer token.+ ("session", False),+ ("SESSION", False),+ ("Token", False),+ ("get_token", False),+ ],+ )+ def test_security_sensitive_symbol_names(self, name: str, expected: bool) -> None:+ sym = SimpleNamespace(name=name, start_line=3)+ kinds = {f["kind"] for f in scan_source("m.py", "", [sym])}+ assert ("security_sensitive_symbol" in kinds) is expected+@pytest.mark.parametrize("file_path",[@@ -549,7 +581,7 @@ class TestScanFile:@pytest.mark.parametrize(("source", "expected"),[- ("requests.get(url, verify=False)\n", True),+ ("import requests\nrequests.get(url, verify=False)\n", True),('"tls_verify_false": "TLS verification turned off (verify=False).",\n', False),("# verify=False disables certificate checks\n", False),],@@ -559,6 +591,111 @@ class TestScanFile:findings = asyncio.run(scanner.scan_file("client.py", source, symbols=[]))assert bool([f for f in findings if f["kind"] == "tls_verify_false"]) is expected+ @pytest.mark.parametrize(+ ("source", "expected"),+ [+ ("import httpx\nhttpx.Client(\n base_url=url,\n verify=False,\n)\n", True),+ ('import boto3\nboto3.client("s3", verify=False)\n', True),+ ('import boto3\nc = boto3.session.Session().client("s3", verify=False)\n', True),+ ("from requests import get\nget(url, verify=False)\n", True),+ ("import requests\ns = requests.Session()\ns.get(url, verify=False)\n", True),+ ("import requests as r\nwith r.Session() as s:\n s.post(url, verify=False)\n", True),+ ("import requests\nclass C:\n def go(self):\n self.http.get(u, verify=False)\n", True),+ ("import requests\ns = requests.Session()\ns.verify = False\n", True),+ ("import requests\nclass Backend:\n verify = False\n", True),+ # A file with no HTTP client library: whatever ``get`` or ``client`` is, not TLS.+ ("rows = db.get(key, verify=False)\n", False),+ ('c = client("s3", verify=False)\n', False),+ ("session.verify = False\n", False),+ ("class Backend:\n verify = False\n", False),+ # An HTTP library imported, but the receiver is some other object.+ ("import requests\nrows = db.get(key, verify=False)\n", False),+ ("import requests\nGitRepo(io, git_commit_verify=False)\n", False),+ ("import requests\nmock_get.assert_called_with(url, verify=False)\n", False),+ # A function of the project's own whose keyword means something else.+ ("import httpx\nrows = load_fix_first(session, repo_id, verify=False)\n", False),+ ],+ )+ def test_tls_verify_false_needs_an_http_client(self, source: str, expected: bool) -> None:+ findings = scan_source("client.py", source)+ assert bool([f for f in findings if f["kind"] == "tls_verify_false"]) is expected++ def test_tls_verify_false_keeps_the_line_match_when_python_does_not_parse(self) -> None:+ findings = scan_source("client.py", "def broken(:\nload(x, verify=False)\n")+ assert [f["line"] for f in findings if f["kind"] == "tls_verify_false"] == [2]++ @pytest.mark.parametrize(+ ("source", "expected"),+ [+ ("rows = con.execute(f\"SELECT id FROM users WHERE name = '{name}'\")\n", True),+ ('sql = (\n f"SELECT {cols}, name "\n f"FROM users WHERE id = {uid}"\n)\n', True),+ ("q = f\"SELECT * FROM docs WHERE title LIKE '%{term}%'\"\n", True),+ ('q = f"SELECT * FROM t WHERE age >= {age} ORDER BY 1"\n', True),+ # Identifiers, fragments and placeholder lists, which cannot be bound.+ ('found = con.execute(f"SELECT MAX(at) FROM {table}").fetchone()\n', False),+ ('q = f"SELECT ref FROM omissions WHERE ref IN ({placeholders})"\n', False),+ ('q = f"SELECT {column}, COUNT(*) FROM episodes{where} GROUP BY {column}"\n', False),+ ('q = f"SELECT id FROM t WHERE id = :id", params={"id": 1}\n', False),+ ('q = f"SELECT a FROM t WHERE b = {{literal}}"\n', False),+ ('q = f"""SELECT id FROM users WHERE id = {uid}"""\n', True),+ ('q = f"""SELECT {cols}\n FROM users\n WHERE name = \'{name}\'\n"""\n', True),+ ('q = f"""SELECT id FROM {table}"""\nx = {"a": 1}\n', False),+ # A message about SQL, with no query in it.+ ('reason = f"SELECT * inside a {kind}: the shape changes"\n', False),+ # Python source quoted inside a string, as a test fixture holds it.+ ("src = 'q = f\"SELECT * FROM t WHERE id = {uid}\"'\n", False),+ ],+ )+ def test_fstring_sql_needs_a_value_written_into_a_query(self, source: str, expected: bool) -> None:+ findings = scan_source("db.py", source)+ assert bool([f for f in findings if f["kind"] == "fstring_sql"]) is expected++ @pytest.mark.parametrize(+ ("line", "kind"),+ [+ ("<div dangerouslySetInnerHTML={{ __html: markup }} />", "unsafe_inner_html"),+ ("const q = `SELECT * FROM users WHERE id = ${id}`;", "template_literal_sql"),+ ("const f = new Function(body);", "new_function_call"),+ ("const a = new https.Agent({ rejectUnauthorized: false });", "reject_unauthorized_false"),+ ("const k = process.env.NEXT_PUBLIC_API_KEY;", "public_env_secret"),+ ],+ )+ def test_javascript_kinds_fire_in_javascript_not_python(self, line: str, kind: str) -> None:+ assert kind in {f["kind"] for f in scan_source("web/app.tsx", line + "\n")}+ quoted = f"CASE = {line!r}\n"+ assert kind not in {f["kind"] for f in scan_source("src/check.py", quoted)}++ @pytest.mark.parametrize(+ ("path", "source", "expected"),+ [+ ("a.py", "digest = hashlib.md5(data).hexdigest()\n", True),+ ("a.py", "mac = hmac.new(key, msg, hashlib.sha1)\n", True),+ ("a.py", 'h = hashlib.new("md5")\n', True),+ ("a.py", "from hashlib import sha1\nkey = sha1(raw)\n", True),+ ("a.js", 'const h = crypto.createHash("md5");\n', True),+ ("A.java", 'MessageDigest md = MessageDigest.getInstance("SHA-1");\n', True),+ ("a.go", "sum := md5.Sum(data)\n", True),+ ("A.cs", "using var h = MD5.Create();\n", True),+ ("a.rb", "Digest::SHA1.hexdigest(s)\n", True),+ ("a.rs", "let d = md5::compute(bytes);\n", True),+ ("a.php", "$h = md5($password);\n", True),+ # A commit id named for its algorithm, and the word in prose.+ ("a.py", 'sha1 = git("rev-parse", "HEAD")\n', False),+ ("a.py", "assert repo.head_commit == sha1\n", False),+ ("a.py", "#: A commit id (sha1 or sha256).\n", False),+ ("a.py", '"weak_hash": "The words md5 or sha1.",\n', False),+ ("a.py", "_commit(repo, \"h.py\", \"hashlib.md5(b'')\")\n", False),+ # Declared a checksum, not a security control.+ ("a.py", "key = hashlib.sha1(text.encode(), usedforsecurity=False).hexdigest()\n", False),+ ("a.py", "key = hashlib.sha1(\n text.encode(), usedforsecurity=False\n)\n", False),+ ("a.py", 'key = hashlib.sha1(f"({a})".encode(), usedforsecurity=False)\n', False),+ ("a.py", 'key = hashlib.sha1(b"usedforsecurity=False")\n', True),+ ("a.py", "key = hashlib.sha1(\n text.encode()\n)\nok = sha256(b, usedforsecurity=False)\n", True),+ ],+ )+ def test_weak_hash_is_an_algorithm_in_use(self, path: str, source: str, expected: bool) -> None:+ assert bool([f for f in scan_source(path, source) if f["kind"] == "weak_hash"]) is expected+class TestPersistSecurityFindings:"""The persist.py wiring: source_map in, idempotent rows out."""@@ -921,7 +1058,7 @@ class TestEverySnippetIsMasked:@pytest.mark.parametrize(("source", "kind"),[- (f'API_KEY = "{KEY}" # md5\n', "weak_hash"),+ (f'API_KEY = "{KEY}"; h = md5(b)\n', "weak_hash"),(f'os.system(cmd); secret = "{KEY}"\n', "os_system"),(f'x = eval(y); secret = "{KEY}"\n', "eval_call"),(f'subprocess.run(cmd, secret = "{KEY}",\n shell=True)\n', "subprocess_shell_true"),@@ -997,9 +1134,10 @@ class TestEverySnippetIsMasked:_assert_no_raw(hit["snippet"], body[::-1])def test_stray_hit_on_a_pem_body_line_is_masked(self) -> None:- body = "MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgE+md5/AoIBAQC7VJTUt9Us8cKj"+ # pragma: allowlist nextline secret+ body = "MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgE+AKIAQ2Z7RT4WXY5BCDFG/AoIBAQC7VJTUt9Us8cKj"source = f"-----BEGIN RSA PRIVATE KEY-----\n{body}\n-----END RSA PRIVATE KEY-----\n"- (hit,) = [f for f in scan_source("id_rsa", source) if f["kind"] == "weak_hash"]+ (hit,) = [f for f in scan_source("id_rsa", source) if f["kind"] == "aws_access_key"]assert hit["line"] == 2_assert_no_raw(hit["snippet"], body)