Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(pii): stop reporting timestamps, matrices, and vectors as PII (#304)
Narendran Raghavan4c3fc87b463d
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 427a87ba..216f8398 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -501,6 +501,34 @@ All notable changes to SkillEvaluator are documented in this file.single commands need longer can raise Claude Code's default with`BASH_DEFAULT_TIMEOUT_MS` in `harbor.runtime_env`.+- PII scans no longer report Luhn-valid timestamps and digests as CRITICAL+ credit cards that fail `validate` and `pii-scan`. A number is skipped when a+ label such as `timestamp_microseconds:`, `event_timestamp:`, `unix_time`,+ `epoch_us`, `sha256:`, or `checksum=` directly precedes it, or when it sits+ inside a longer ASCII identifier or hex token such as `id_1707938736269577`;+ a card after a payment word and an underscore (`card_4532015112830366`,+ `cc_...`), in Markdown underscore emphasis (`_4532015112830366_`), after a string+ escape such as `\n` or a percent-encoded byte such as `%3D`, or written+ directly next to Chinese, Japanese, or Korean text is still reported. Known+ test card numbers, now including other widely published test cards such as+ `4242424242424242` and `5555555555554444`, are excepted however their digits+ are grouped (`4111 1111 1111 1111`). They no longer hide a different card+ number on the same line, so a card listed next to a test card is now+ reported. The exponent of a scientific-notation value such as+ `1.000000e+00` is no longer reported as an international phone number.+ GPS-shaped decimal pairs now need a geospatial cue (GPS, latitude/longitude,+ lat/lng, `latlong`/`lonlat` keys, WGS 84, a `geo:` URI, a Google, Apple,+ OpenStreetMap, or Bing map link, a GeoJSON geometry, `Feature`, or collection+ type or `"geometry"` member, or location, office, address, or degrees, also+ in plural, numeric-suffix, snake_case, or camelCase forms such as+ `locations:`, `lat1`, or `homeLocation`) on the line, on one of the two lines+ before it, or in the file name, so waypoints, poses, quaternions, and+ calibration vectors are no longer reported as location data. Every position+ inside a GeoJSON geometry's `coordinates` array written as JSON, a Python+ dict, or a JS object is still reported. Pairs without a cue, such as+ `coordinates: 37.7749, -122.4194` or `Meet me at 37.7749, -122.4194`, are no+ longer reported either.+- Tier 1 security scans no longer fail a skill for an exact read-only`/etc/passwd` bind mount such as `docker run -v /etc/passwd:/etc/passwd:ro`.SkillSpector 2.11.1 and later tags this mount as likely benign but stilldiff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdxindex b2720329..eb7dfffb 100644--- a/docs/tier1-validation.mdx+++ b/docs/tier1-validation.mdx@@ -206,6 +206,8 @@ For `validate`, the default set is `schema`, `version`, `security`, `pii`, `lice| Credentials and secrets | Database connection strings, hardcoded API keys, cloud keys, GitHub tokens, private keys, JWTs, webhook URLs | Most credential findings are CRITICAL || Network and financial | Public IPs, credit-card numbers, cryptocurrency wallet addresses | Private RFC1918 addresses are excluded |+ Numeric values are judged by their context as well as their shape. A card-shaped number is not reported when a timestamp or digest label (`timestamp_microseconds:`, `event_timestamp:`, `epoch`, `epoch_us`, `unix_time`, `sha256:`, `checksum=`) directly precedes it or when it sits inside a longer ASCII token, such as a hex digest or an identifier like `id_1707938736269577` where an underscore joins it to a letter or digit (unless a payment word such as `card`, `cc`, or `pan` is part of that identifier, as in `card_4532015112830366`). A card in Markdown underscore emphasis (`_4532015112830366_`), after a string escape such as `\n` or `\t` or a percent-encoded byte such as `%3D`, or written directly next to Chinese, Japanese, Korean, or other non-ASCII letters (including the Turkish dotless `ı`) is still reported. Widely published test card numbers (`4111111111111111`, `4242424242424242`, `5555555555554444`, and similar) are excepted however their digits are grouped, without hiding a different card number on the same line. An international phone candidate that starts inside a scientific-notation value such as `1.000000e+00` is ignored. A GPS-shaped decimal pair is reported only with a geospatial cue on its line, on one of the two lines before it, or in the file name. Strong cues are GPS, latitude/longitude, lat/lng, `latlong`/`lonlat` keys, WGS 84, a `geo:` URI, a Google, Apple, OpenStreetMap, or Bing map link, and a GeoJSON geometry, `Feature`, or collection `"type"` or `"geometry"` member. Soft cues are location, office, address, degrees, and `°`. Cue words count in their plural forms, with a numeric suffix, and inside snake_case or camelCase keys and file names (`store_locations:`, `latlngs`, `lat1`, `homeLocation`, `gpsCoords`, `locations.json`). The GeoJSON `type` and `geometry` keys count double-quoted, single-quoted, or bare, so JSON, Python dicts, JS objects, and YAML all carry the cue; a bare type value must use GeoJSON's spelling (`Point`, `MultiPolygon`). Every position inside a GeoJSON geometry's `coordinates` array is cued, however deeply its rings are nested, when the geometry is written with braces (JSON, a Python dict, or a JS object); block-style YAML geometry is cued only through the two-line window. Technical labels outweigh soft cues but not strong ones: `waypoint`, `quaternion`, `pose`, `vector`, `calibration`, `roi`, and `scale` count only on the pair's own line and only before the pair (so `Location: 37.774929, -122.419416 (map scale 1:5000)` is still reported), while `<pixel|camera|world|...> coordinates` also counts on the two lines before it and in the file name. Pairs without a cue are not reported, for example `coordinates: 37.7749, -122.4194`, `Meet me at 37.7749, -122.4194`, or `home: 37.774929, -122.419416`.+Common placeholders such as `YOUR_API_KEY_HERE`, `example`, `test`, `dummy`, and `placeholder`, along with `@example.com` and `@test.com`, are excluded, as are configured allowed paths.A card-shaped number is reported only when it passes the Luhn checksum; all-zero numbers do not. Digits inside a canonical GUID/UUID, such as the COM interface ID `{00000000-0000-0000-C000-000000000046}`, are treated as an identifier unless a payment label (`card_number`, `credit_card`, `cc_no`, `cc_number`, `pan`, and similar) directly precedes the GUID on its line or ends the line before it. Every card-shaped number on a line is checked, so an identifier or non-card number earlier on the line does not hide a card after it.diff --git a/src/skillevaluator/config/pii_patterns.yaml b/src/skillevaluator/config/pii_patterns.yamlindex 4058f33c..7eb85f1d 100644--- a/src/skillevaluator/config/pii_patterns.yaml+++ b/src/skillevaluator/config/pii_patterns.yaml@@ -91,6 +91,8 @@ phone_numbers:- "&id="- "Id="+ # A candidate that starts inside a scientific-notation value, such as the+ # "+00" exponent of "1.000000e+00", is discarded in security.py.- pattern: '\+\d{1,3}[\s.-]?\(?\d{1,4}\)?[\s.-]?\d{1,4}[\s.-]?\d{1,9}'severity: mediumdescription: International phone number@@ -207,14 +209,28 @@ private_keys:# Credit card numbers (less common but high severity)# Note: Matches are post-validated with the Luhn algorithm in security.py.-# Numbers that fail Luhn are silently discarded (not valid card numbers).-# The lookbehind/lookahead reject matches embedded in a longer digit run or-# immediately preceded by a decimal point -- this prevents floating-point-# eval scores like "score": 0.8603333333333333 in JSON dumps from being-# misread as 16-digit card numbers (the 16-digit window 8603333333333333-# happens to pass Luhn).+# Numbers that fail Luhn are silently discarded (not valid card numbers), and+# so are numbers directly after a timestamp or digest label such as+# "timestamp_microseconds:" or "sha256:"; passing Luhn alone is not payment+# evidence. Numeric exceptions below are compared with each candidate's digits,+# so a test card on a line does not hide a different card number next to it.+# The lookbehind/lookahead reject matches embedded in a longer ASCII+# alphanumeric token (a hex digest, or an identifier such as id_1707938736269577+# where an underscore joins the number to a letter or digit) or immediately+# preceded by a decimal point -- this prevents floating-point eval scores like+# "score": 0.8603333333333333 in JSON dumps from being misread as 16-digit card+# numbers (the 16-digit window 8603333333333333 happens to pass Luhn). An+# underscore next to a space or punctuation, as in Markdown emphasis+# (_4532015112830366_ or __4532015112830366__), does not hide a card, and+# neither does a backslash escape such as \n or \t in a JSON, YAML, or code+# string, or a percent-encoded byte such as %3D in a URL. The classes are ASCII+# on purpose: Chinese, Japanese, and Korean prose often puts a number directly+# between letters, and such a card is still reported. They are scoped+# case-sensitive with (?-i:...), because the scanner compiles every pattern+# with IGNORECASE, under which [A-Za-z] also matches non-ASCII letters such as+# the dotless i and the Kelvin sign.credit_cards:- - pattern: '(?<![\d.])[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}(?!\d)'+ - pattern: '(?:(?<=\\(?-i:[abfnrtv]))|(?<=%(?-i:[0-9A-Fa-f]{2}))|(?<!(?-i:[A-Za-z0-9.])))[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}[\s-]?[0-9]{4}(?!(?-i:[A-Za-z0-9]))(?!_(?-i:[A-Za-z0-9]))'severity: criticaldescription: Possible credit card numbersuggestion: "Remove credit card number - use placeholder like 4111-1111-1111-1111"@@ -231,12 +247,23 @@ credit_cards:- "test card"- "test number"- "for testing"- # Well-known test card numbers (full, not prefix substrings)+ # Well-known, publicly documented test card numbers (full, not prefix+ # substrings). Each excepts only a candidate with the same digits, in any+ # grouping. The 15-digit Amex number never matches the 16-digit pattern+ # above; it is kept for a future 15-digit pattern.- "4111111111111111"- "4111-1111-1111-1111"- "5500000000000004"- "5500-0000-0000-0004"- "378282246310005"+ - "4242424242424242"+ - "4000056655665556"+ - "4012888888881881"+ - "5555555555554444"+ - "5105105105105100"+ - "2223003122003222"+ - "6011111111111117"+ - "3530111333300000"# Cryptocurrency addressescrypto_addresses:@@ -264,6 +291,14 @@ mac_addresses:# pairs that pepper technical/API reference docs (normalized ROI floats like# "0.5, 0.7", scale factors like "1.5, 2.0", etc.) while still flagging genuine# location data. The exact-pole forms (90.0 / 180.0) are kept as-is.+# A match also needs a geospatial cue (GPS, latitude/longitude, lat/lng,+# latlong/lonlat keys, WGS 84, a GeoJSON geometry type, a geo: URI, a map link+# such as google.com/maps or maps.apple.com, or location/office/address/degrees,+# including plural, numeric-suffix, snake_case, and camelCase forms) on its+# line, on one of the two lines before it, or in the file name, or must sit+# inside a GeoJSON geometry's "coordinates" array written with braces (JSON, a+# Python dict, or a JS object), so precise vectors, poses, quaternions,+# calibration values, and waypoints are not reported.gps_coordinates:- pattern: '[-+]?([1-8]?\d\.\d{4,}|90\.0+)[,\s]+[-+]?(180\.0+|((1[0-7]\d)|([1-9]?\d))\.\d{4,})'severity: mediumdiff --git a/src/skillevaluator/validators/security.py b/src/skillevaluator/validators/security.pyindex 20e6f783..7d722d8b 100644--- a/src/skillevaluator/validators/security.py+++ b/src/skillevaluator/validators/security.py@@ -12,10 +12,12 @@ Detects security vulnerabilities and PII/secrets using:from __future__ import annotations+import bisectimport contextlibimport getpassimport hashlibimport io+import itertoolsimport jsonimport mathimport os@@ -3771,6 +3773,67 @@ class SecurityValidator(ValidatorBase):return root.lower() in protected_usernames_GPS_ZERO_PATTERN = re.compile(r"[-+]?0+\.0+[,\s]+[-+]?0+\.0+")+ # Geospatial cues. A strong cue names a geographic coordinate system, a map+ # link (an RFC 5870 ``geo:`` URI or a Google, Apple, OpenStreetMap, or Bing+ # map URL), or a GeoJSON object: a geometry ``"type"``, a ``Feature``,+ # ``FeatureCollection`` or ``GeometryCollection`` ``"type"``, or a+ # ``"geometry": {`` member, which comes before ``"coordinates"`` when keys+ # are sorted. GeoJSON member names and these object types are+ # case-sensitive, so ``"type": "feature"`` in other JSON is not a cue. A soft+ # cue (location, office, address, degrees, or a degree sign) also labels+ # technical data such as ``calibration_location_offset`` or a rotation in+ # degrees, so it does not override a technical label on the same line.+ # Cue words accept their plurals and a numeric suffix (``locations:``,+ # ``latlngs``, ``lat1, lon1``, ``location2``). The GeoJSON ``type`` and+ # ``geometry`` keys may be double-quoted, single-quoted (a Python dict) or+ # bare (a JS object or YAML); a bare type value must use GeoJSON's own+ # spelling (``Point``, ``MultiPolygon``). Map hosts are matched with bounded+ # TLD forms so long lines stay linear.+ _GPS_STRONG_CONTEXT_PATTERN = re.compile(+ r"(?i)\b(?:gps|geolocations?|geospatial|latitudes?|longitudes?|lats?|lons?|lngs?|latlngs?|latlong?s?"+ r"|lonlats?|lnglats?|wgs[- ]?84)\d*\b"+ r"|\bgeo:"+ r"|\bmaps\.google\.|\bgoogle(?:\.[a-z]{2,3}){1,2}/maps\b|\bmaps\.apple\.com\b|\bopenstreetmap\.org\b"+ r"|\bbing\.com/maps\b"+ r"|\btype[\"']?\s*:\s*(?:[\"'](?:(?:multi)?(?:point|linestring|polygon)"+ r"|(?-i:Feature(?:Collection)?|GeometryCollection))[\"']"+ r"|(?-i:(?:Multi)?(?:Point|LineString|Polygon)|Feature(?:Collection)?|GeometryCollection)\b)"+ r"|\b(?-i:geometry)[\"']?\s*:\s*\{"+ )+ _GPS_SOFT_CONTEXT_PATTERN = re.compile(r"(?i)\b(?:locations?|offices?|address(?:es)?|degrees?)\d*\b|[°º]")+ _TECHNICAL_COORDINATE_CONTEXT_PATTERN = re.compile(+ r"(?i)\b(?:texture|uv|pixel|screen|canvas|image|normalized|cartesian|robot(?:ic(?:s)?)?|local|world|camera"+ r"|pose|joint|model|object|viewport|mesh|vertex|clip|device)[ -]+coordinates?\b"+ )+ _TECHNICAL_VECTOR_CONTEXT_PATTERN = re.compile(+ r"(?i)\b(?:quaternion|pose(?:[ _-]?translation)?|waypoint|vector|calibration|roi|scale)s?\b"+ r"|\bcoordinates?\s*=|(?:^|\s)(?:x\s+y|u\s+v)(?:\s|$)"+ )+ _CAMEL_CASE_BOUNDARY = re.compile(r"(?<=[a-z])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z])")+ # Double- or single-quoted strings (one line at most, so a stray quote in+ # prose cannot run on), bare identifiers (JS object keys), and structural+ # characters, for the GeoJSON ``"coordinates"`` array pass.+ _JSON_TOKEN_PATTERN = re.compile(r'"((?:[^"\\\n]|\\.)*)"|\'((?:[^\'\\\n]|\\.)*)\'|([A-Za-z_$][\w$]*)|([{}\[\]:,])')+ _GEOJSON_GEOMETRY_TYPE = re.compile(r"(?i)(?:multi)?(?:point|linestring|polygon)")+ # A bare (unquoted) type value must use GeoJSON's own spelling.+ _GEOJSON_BARE_GEOMETRY_TYPE = re.compile(r"(?:Multi)?(?:Point|LineString|Polygon)")+ _SCIENTIFIC_LITERAL_PATTERN = re.compile(r"(?<![\w.])[-+]?(?:\d+(?:\.\d*)?|\.\d+)[eE][-+]?\d+")+ # ``<identifier>_`` directly before a candidate; group 1 is the identifier.+ _UNDERSCORE_JOINED_PREFIX_PATTERN = re.compile(r"(?-i:([A-Za-z0-9]+(?:_[A-Za-z0-9]+)*)_)\Z")+ _PAYMENT_LABEL_WORDS = frozenset(+ {"card", "cards", "cc", "ccn", "pan", "credit", "debit", "payment", "visa", "mastercard", "amex"}+ )+ # A timestamp or digest label that directly precedes a value+ # (``timestamp_microseconds: ``, ``"sha256": "``, ``checksum=``). An+ # underscore separates words, so snake_case prefixes such as+ # ``event_timestamp:`` and ``"created_at_epoch":`` count as labels.+ _NON_PAYMENT_NUMBER_CONTEXT_PATTERN = re.compile(+ r"(?i)(?:(?<![A-Za-z0-9])(?:(?:timestamp|unix(?:[_ -]?time)?|epoch)"+ r"(?:[_ -]?(?:microseconds?|micros?|us|usec|nanoseconds?|nanos?|ns|nsec))?"+ r"|(?:microseconds?|nanoseconds?)(?:[_ -]?timestamp)?)\b"+ r"|(?<![A-Za-z0-9])(?:md5|sha(?:1|224|256|384|512)|hash|digest|checksum)\b)"+ r"[\"']?(?:\s*[:=]\s*[\"']?|\s+[\"']?)\Z"+ )# Match version/tag as identifier tokens, including separator and camel-case styles.# Plain substrings such as ``conversion`` and ``staging`` are not version labels._VERSION_LABEL_PATTERN = re.compile(@@ -3816,6 +3879,170 @@ class SecurityValidator(ValidatorBase):"""Check if a GPS match contains only near-zero coordinates (Null Island)."""return bool(SecurityValidator._GPS_ZERO_PATTERN.search(line))+ @classmethod+ def _is_numeric_pii_false_positive(cls, category: str, match: re.Match, line: str) -> bool:+ """Return whether a card or phone candidate on *line* is technical data."""+ if category == "credit_cards":+ # A Luhn-valid timestamp or digest is not payment data. Only a label+ # directly before this number counts, so a timestamp field does not+ # hide a card number in a later field on the same line.+ label_start = max(0, match.start() - 100)+ if cls._NON_PAYMENT_NUMBER_CONTEXT_PATTERN.search(line, label_start, match.start()) is not None:+ return True+ # An underscore joining the number to a letter or digit makes it part+ # of an identifier (``id_1707938736269577``), unless a word in that+ # identifier is a payment label (``card_4532015112830366``).+ joined = cls._UNDERSCORE_JOINED_PREFIX_PATTERN.search(line, label_start, match.start())+ if joined is None:+ return False+ words = cls._CAMEL_CASE_BOUNDARY.sub("_", joined.group(1)).lower().split("_")+ return not any(word in cls._PAYMENT_LABEL_WORDS for word in words)++ if category == "phone_numbers":+ # International phone syntax can begin at the ``+00`` exponent of a+ # scientific-notation value and run into the next column. Only a+ # candidate that starts inside such a literal is dropped, so a real+ # phone number elsewhere on the line is still reported.+ window = (max(0, match.start() - 64), match.start() + 64)+ return any(+ literal.start() <= match.start() < literal.end()+ for literal in cls._SCIENTIFIC_LITERAL_PATTERN.finditer(line, *window)+ )++ return False++ @classmethod+ def _cue_text_variants(cls, text: str) -> tuple[str, ...]:+ """Return *text* with underscores as spaces, unsplit and camelCase-split.++ Keys and file names often use snake_case or camelCase (``homeLocation``,+ ``gpsCoords``). The unsplit form keeps GeoJSON ``"type"`` values such as+ ``FeatureCollection`` intact.+ """+ spaced = text.replace("_", " ")+ split = cls._CAMEL_CASE_BOUNDARY.sub(" ", spaced)+ return (spaced,) if split == spaced else (spaced, split)++ @staticmethod+ def _search_cue(pattern: re.Pattern, variants: tuple[str, ...]) -> bool:+ """Return whether *pattern* matches any form of the cue text."""+ return any(pattern.search(text) for text in variants)++ @classmethod+ def _lacks_geospatial_context(+ cls, line: str, context: str, pair_start: int | None = None, label_start: int = 0+ ) -> bool:+ """Return whether GPS-shaped pairs on *line* have no geospatial cue.++ Precise vectors, poses, quaternions, and waypoints have the same shape+ as coordinates, so a pair needs a cue. *context* is the file name, up to+ two lines before *line*, and *line*. Keys and file names often use+ snake_case or camelCase, so underscores and case changes separate words.+ Vector labels such as ``waypoint`` count only before the pair at+ *pair_start*; a GPS cue after a separator can limit that label segment.+ """+ local = cls._cue_text_variants(line)+ # A technical label labels the pair only when it comes before it, so+ # ``Location: 37.77, -122.41 (map scale 1:5000)`` keeps its location cue.+ label_end = len(line) if pair_start is None else pair_start+ label_segment = line[label_start:label_end]+ last_separator = max(label_segment.rfind(";"), label_segment.rfind("|"))+ tail = label_segment[last_separator + 1 :]+ tail_variants = cls._cue_text_variants(tail)+ if cls._search_cue(cls._GPS_SOFT_CONTEXT_PATTERN, tail_variants) or cls._search_cue(+ cls._GPS_STRONG_CONTEXT_PATTERN, tail_variants+ ):+ label_segment = tail+ label = cls._cue_text_variants(label_segment)+ nearby = cls._cue_text_variants(context)+ if cls._search_cue(cls._GPS_STRONG_CONTEXT_PATTERN, local):+ return False+ inherits_technical_label = False+ if label_start and ";" not in line[label_start:label_end] and "|" not in line[label_start:label_end]:+ previous_separator = max(line.rfind(";", 0, label_start), line.rfind("|", 0, label_start))+ previous_label = cls._cue_text_variants(line[previous_separator + 1 : label_start])+ inherits_technical_label = cls._search_cue(+ cls._TECHNICAL_COORDINATE_CONTEXT_PATTERN, previous_label+ ) or cls._search_cue(cls._TECHNICAL_VECTOR_CONTEXT_PATTERN, previous_label)+ if (+ cls._search_cue(cls._TECHNICAL_COORDINATE_CONTEXT_PATTERN, label)+ or cls._search_cue(cls._TECHNICAL_VECTOR_CONTEXT_PATTERN, label)+ or inherits_technical_label+ ):+ return not cls._search_cue(cls._GPS_STRONG_CONTEXT_PATTERN, nearby)+ if cls._search_cue(cls._GPS_SOFT_CONTEXT_PATTERN, local):+ return False+ if cls._search_cue(cls._GPS_STRONG_CONTEXT_PATTERN, nearby):+ return False+ if cls._search_cue(cls._TECHNICAL_COORDINATE_CONTEXT_PATTERN, nearby):+ return True+ return not cls._search_cue(cls._GPS_SOFT_CONTEXT_PATTERN, nearby)++ @classmethod+ def _geojson_coordinate_lines(cls, lines: list[str]) -> frozenset[int]:+ """Return 1-based numbers of lines inside a GeoJSON geometry's ``"coordinates"`` array.++ A pretty-printed ``Polygon`` or ``Multi*`` geometry opens its rings on+ their own lines, so positions can sit far below the ``"type"`` cue. The+ array counts when its object has a geometry ``"type"`` (before or after+ ``"coordinates"``) or is the value of a ``"geometry"`` member. Keys and+ values may be double-quoted (JSON), single-quoted (a Python dict) or+ bare keys (a JS object). One pass over these tokens finds the arrays,+ and nested arrays are merged before their lines are marked, so the scan+ stays linear however deep the nesting; unbalanced brackets in other+ text are tolerated.+ """+ text = "\n".join(lines)+ if "coordinates" not in text:+ return frozenset()+ line_starts = [0, *itertools.accumulate(len(line) + 1 for line in lines)]+ # Each frame is [kind, is_geojson, coordinate_spans, coordinates_owner, start].+ stack: list[list] = []+ spans: list[tuple[int, int]] = []+ last_string: str | None = None+ key: str | None = None+ for token in cls._JSON_TOKEN_PATTERN.finditer(text):+ bare = token.group(3)+ value = token.group(1) if token.group(1) is not None else token.group(2)+ if value is None:+ value = bare+ punct = token.group(4)+ top = stack[-1] if stack else None+ if value is not None:+ type_pattern = cls._GEOJSON_BARE_GEOMETRY_TYPE if bare is not None else cls._GEOJSON_GEOMETRY_TYPE+ if key == "type" and top is not None and top[0] == "{" and type_pattern.fullmatch(value):+ top[1] = True+ last_string, key = value, None+ elif punct == ":":+ key, last_string = last_string, None+ elif punct in "{[":+ owner = top if punct == "[" and key == "coordinates" and top is not None and top[0] == "{" else None+ stack.append([punct, punct == "{" and key == "geometry", [], owner, token.start()])+ key = last_string = None+ elif punct in "}]":+ if not stack:+ continue+ frame = stack.pop()+ if frame[3] is not None:+ frame[3][2].append((frame[4], token.start()))+ if frame[0] == "{" and frame[1]:+ spans.extend(frame[2])+ key = last_string = None+ else:+ key = last_string = None+ # Nested arrays yield nested spans; keep only the outermost of each+ # group so every line is marked once.+ numbers: set[int] = set()+ kept_end = -1+ for start, end in sorted(spans):+ if end <= kept_end:+ continue+ first = bisect.bisect_right(line_starts, max(start, kept_end + 1))+ last = bisect.bisect_right(line_starts, end)+ numbers.update(range(first, last + 1))+ kept_end = end+ return frozenset(numbers)+@classmethoddef _is_version_literal(cls, match: re.Match, line: str) -> bool:"""Return whether an IPv4-shaped match is clearly a release version."""@@ -4012,6 +4239,15 @@ class SecurityValidator(ValidatorBase):protected_usernames = protected_usernames or set()author_emails = author_emails or {}comment_lines = comment_lines if comment_lines is not None else _comment_line_numbers(file_path, lines)+ line_exceptions = exceptions+ excepted_card_digits: frozenset[str] = frozenset()+ if category == "credit_cards":+ # Known test card numbers are compared with each candidate, in any+ # grouping, so one on a line does not hide a different card number.+ numeric_exceptions = {exc for exc in exceptions if re.fullmatch(r"[0-9\s-]*[0-9][0-9\s-]*", exc)}+ excepted_card_digits = frozenset(re.sub(r"[\s-]", "", exc) for exc in numeric_exceptions)+ line_exceptions = [exc for exc in exceptions if exc not in numeric_exceptions]+ geojson_lines: frozenset[int] | None = Nonefor line_num, line in enumerate(lines, 1):if line_num in comment_lines:continue@@ -4021,8 +4257,32 @@ class SecurityValidator(ValidatorBase):scan_line = re.sub(re.escape(author_email), "[email protected]", line, count=1, flags=re.IGNORECASE)matches = list(regex.finditer(scan_line))- if not matches or any(exc in line for exc in exceptions):+ if excepted_card_digits:+ matches = [+ match for match in matches if re.sub(r"[\s-]", "", match.group()) not in excepted_card_digits+ ]+ if not matches or any(exc in line for exc in line_exceptions):continue+ if category in ("credit_cards", "phone_numbers"):+ matches = [match for match in matches if not self._is_numeric_pii_false_positive(category, match, line)]+ if not matches:+ continue+ if category == "gps_coordinates":+ # A position inside a GeoJSON geometry's "coordinates" array is+ # cued however far its rings sit below the "type" line.+ if geojson_lines is None:+ geojson_lines = self._geojson_coordinate_lines(lines)+ if line_num not in geojson_lines:+ context = "\n".join((file_path.name, *lines[max(0, line_num - 3) : line_num]))+ previous_end = 0+ kept_matches = []+ for pair in matches:+ if not self._lacks_geospatial_context(line, context, pair.start(), label_start=previous_end):+ kept_matches.append(pair)+ previous_end = pair.end()+ matches = kept_matches+ if not matches:+ continueif category == "ip_addresses":matches = [match for match in matches if not self._is_version_literal(match, line)]if not matches:diff --git a/tests/validators/test_security.py b/tests/validators/test_security.pyindex 12fbc6d6..351e1b50 100644--- a/tests/validators/test_security.py+++ b/tests/validators/test_security.py@@ -6437,6 +6437,229 @@ class TestFalsePositivePrevention:gps_errors = [f for f in result.errors + result.warnings if "gps" in f.lower() or "coordinate" in f.lower()]assert len(gps_errors) == 0, f"Low-precision pairs should not be flagged: {gps_errors}"+ @pytest.mark.parametrize(+ "technical_value",+ [+ "waypoint = [0.387236, 0.247973, 0.666778]",+ "waypoints: [[0.387236, 0.247973], [0.412345, 0.256789]]",+ "quaternion = [0.707107, 0.000000, 0.707107, 0.000000]",+ "pose_translation = (37.123456, -12.654321, 4.000000)",+ "calibration_pair: 12.345678, 45.678901",+ "coordinates = [0.123456, 0.654321]",+ "coordinates: [0.123456, 0.654321]",+ "long vector = 0.123456, 0.654321",+ "37.123456, -12.654321",+ '{"type": "feature", "weights": [0.123456, 0.654321]}',+ # A bare GeoJSON type must use GeoJSON's spelling, so a point light is not a cue.+ "type: point\nposition: [12.345678, 45.678901]",+ ],+ )+ def test_gps_shape_requires_geospatial_context(self, tmp_path: Path, technical_value: str):+ """Precise vectors and unlabeled decimal pairs are not location PII without a geospatial cue."""+ (tmp_path / "values.txt").write_text(f"{technical_value}\n", encoding="utf-8")++ result = SecurityValidator().validate_pii_only(tmp_path)++ assert not any(finding.check_name == "gps_coordinates" for finding in result.findings)++ @pytest.mark.parametrize(+ "line",+ [+ "Location: 37.774929, -122.419416 (map scale 1:5000)",+ "Office: 37.774929, -122.419416; vector tiles from the same provider",+ "Address: 37.774929, -122.419416 roi=downtown, calibration=none",+ ],+ )+ def test_gps_technical_word_after_pair_does_not_hide_location_label(self, tmp_path: Path, line: str):+ """A technical word that follows the pair does not label it, so the location cue still counts."""+ (tmp_path / "values.txt").write_text(f"{line}\n", encoding="utf-8")++ result = SecurityValidator().validate_pii_only(tmp_path)++ assert any(finding.check_name == "gps_coordinates" for finding in result.findings)++ @pytest.mark.parametrize(+ "line",+ [+ "map scale 1:5000; Location: 37.774929, -122.419416",+ "Camera pose 10.123456, 20.123456; home location: 37.774929, -122.419416",+ ],+ )+ def test_gps_labels_are_applied_per_pair(self, tmp_path: Path, line: str):+ data_file = tmp_path / "values.txt"+ data_file.write_text(f"{line}\n", encoding="utf-8")++ candidates = SecurityValidator()._scan_file_for_pii(data_file)++ assert [item["matched_value"] for item in candidates if item["category"] == "gps_coordinates"] == [+ "37.774929, -122.419416"+ ]++ @pytest.mark.parametrize(+ ("filename", "contents"),+ [+ ("values.txt", "Location data formats\nquaternion = [0.707107, 0.000000, 0.707107, 0.000000]\n"),+ ("values.txt", "calibration_location_offset = [0.123456, 0.654321]\n"),+ ("values.txt", "Pixel coordinates of the office logo\n0.123456, 0.654321\n"),+ ("robotics_coordinates.txt", "Pick location\n0.123456, 0.654321\n"),+ ("mesh.json", '{"type": "Mesh", "coordinates": [0.123456, 0.654321]}\n'),+ # Degrees describe rotations and joint limits too, so they are a soft cue.+ (+ "values.txt",+ "Rotate the gripper 90 degrees about Y:\nquaternion = [0.707107, 0.000000, 0.707107, 0.000000]\n",+ ),+ ("values.txt", "pose (yaw in degrees): 12.345678, 45.678901\n"),+ ("values.txt", "Joint limits are +/-170\u00b0\nwaypoint = [12.345678, 45.678901]\n"),+ ],+ )+ def test_gps_technical_context_outweighs_soft_location_cue(self, tmp_path: Path, filename: str, contents: str):+ """Technical coordinate labels win over location, office, or address wording."""+ (tmp_path / filename).write_text(contents, encoding="utf-8")++ result = SecurityValidator().validate_pii_only(tmp_path)++ assert not any(finding.check_name == "gps_coordinates" for finding in result.findings)++ @pytest.mark.parametrize(+ ("filename", "contents"),+ [+ ("values.txt", "lng pair: 37.7749, -122.4194\n"),+ ("values.txt", "LatLng(37.7749, -122.4194)\n"),+ ("values.yaml", "gps_coordinates: 37.7749,-122.4194\n"),+ ("values.yaml", "latitude_longitude: 37.7749,-122.4194\n"),+ ("location.json", '"coordinates": [37.7749, -122.4194]\n'),+ ("gps_points.txt", "37.7749, -122.4194\n"),+ ("values.txt", "Head office\n\n37.7749, -122.4194\n"),+ ("values.txt", "GPS coordinates\nwaypoint = [37.7749, -122.4194]\n"),+ ("point.json", '{"type": "Point", "coordinates": [-122.4194, 37.7749]}\n'),+ ("feature.json", '{\n "type": "Point",\n "coordinates": [\n -122.4194, 37.7749\n ]\n}\n'),+ ("sorted.json", '{"geometry": {\n "coordinates": [-122.4194, 37.7749],\n "type": "Point"}}\n'),+ ("values.json", '{\n "type": "Feature",\n "bbox": [-122.4194, 37.7749, -122.4094, 37.7849]\n}\n'),+ # Map links, RFC 5870 geo URIs, and combined lat/lon keys are geospatial cues.+ ("SKILL.md", "Open https://www.google.com/maps/@37.774929,-122.419416,17z to find my house\n"),+ ("values.txt", "https://www.google.co.uk/maps/@37.7749,-122.4194,15z\n"),+ ("values.txt", "https://maps.google.com/?q=37.7749,-122.4194\n"),+ ("values.txt", "https://maps.apple.com/?ll=37.774929,-122.419416\n"),+ ("values.txt", "https://www.bing.com/maps?cp=37.7749, -122.4194\n"),+ ("values.txt", "geo:37.774929,-122.419416\n"),+ ("values.yaml", "latlong: 37.774929, -122.419416\n"),+ ("values.yaml", "latlon: 37.774929, -122.419416\n"),+ ("values.yaml", "lonlat: -122.419416, 37.774929\n"),+ ("values.yaml", "lnglat: -122.419416, 37.774929\n"),+ # Plural cue words label lists of locations.+ ("values.yaml", "locations:\n - 37.774929, -122.419416\n"),+ ("values.yaml", "store_locations:\n - 37.774929, -122.419416\n"),+ ("values.txt", "Our offices\n37.774929, -122.419416\n"),+ ("values.txt", "latitudes_longitudes: 37.774929, -122.419416\n"),+ ("locations.json", "[37.774929, -122.419416]\n"),+ ("values.yaml", "geolocations:\n - 37.774929, -122.419416\n"),+ ("map.md", "var latlngs = [[37.774929, -122.419416]];\n"),+ ("values.yaml", "lats: 37.774929, 37.784929\n"),+ # A numeric suffix keeps the cue word.+ ("haversine.py", "lat1, lon1 = 37.774929, -122.419416\n"),+ ("values.yaml", "location1: 37.774929, -122.419416\n"),+ ("data.json", '{"homeLocation2": [37.774929, -122.419416]}\n'),+ # camelCase JSON keys carry the same cue words.+ ("data.json", '{"homeLocation": [37.774929, -122.419416]}\n'),+ ("data.json", '{"officeAddress": "37.774929, -122.419416"}\n'),+ ("data.json", '{"gpsCoords": [37.774929, -122.419416]}\n'),+ ("data.json", '{"lastKnownGps": [37.774929, -122.419416]}\n'),+ # GeoJSON written as a JS object (in a Markdown snippet) or in YAML.+ ("point.md", "const p = {type: 'Point', coordinates: [-122.4194, 37.7749]};\n"),+ ("point.yaml", "type: Point\ncoordinates: [-122.4194, 37.7749]\n"),+ # A strong cue nearby outweighs a technical coordinate label nearby.+ ("world_coordinates.md", "GPS position of the office:\n37.7749, -122.4194\n"),+ ("gps_world_coordinates.txt", "37.7749, -122.4194\n"),+ ("values.txt", "Camera coordinates and GPS position below\n37.7749, -122.4194\n"),+ # Degrees alone remain a cue when no technical label is present.+ ("values.txt", "Coordinates in decimal degrees: 37.774929, -122.419416\n"),+ # Vector labels count only on the pair's own line, so an office cue still applies.+ ("values.txt", "Robot waypoints near the office:\n0.123456, 0.654321\n"),+ ],+ )+ def test_gps_coordinates_with_geospatial_context_still_flagged(self, tmp_path: Path, filename: str, contents: str):+ """A geospatial cue on the line, two lines above, or in the file name keeps the finding."""+ (tmp_path / filename).write_text(contents, encoding="utf-8")++ result = SecurityValidator().validate_pii_only(tmp_path)++ assert any(finding.check_name == "gps_coordinates" for finding in result.findings)++ @pytest.mark.parametrize(+ ("contents", "expected_lines"),+ [+ (+ '{\n "type": "Polygon",\n "coordinates": [\n [\n [-122.4194, 37.7749],\n'+ " [-122.4094, 37.7849],\n [-122.3994, 37.7749],\n [-122.4194, 37.7749]\n"+ " ]\n ]\n}\n",+ [5, 6, 7, 8],+ ),+ (+ '{\n "type": "FeatureCollection",\n "features": [\n {\n "type": "Feature",\n'+ ' "properties": {},\n "geometry": {\n "type": "MultiPolygon",\n'+ ' "coordinates": [\n [\n [\n [-122.4194, 37.7749],\n'+ " [-122.4094, 37.7849]\n ]\n ]\n ]\n }\n }\n ]\n}\n",+ [12, 13],+ ),+ # Sorted keys put "type" after "coordinates".+ (+ '{\n "coordinates": [\n [\n [\n [-122.4194, 37.7749],\n'+ ' [-122.4094, 37.7849]\n ]\n ]\n ],\n "type": "MultiLineString"\n}\n',+ [5, 6],+ ),+ # A single-quoted Python dict.+ (+ "{\n 'type': 'Polygon',\n 'coordinates': [\n [\n [-122.4194, 37.7749],\n"+ " [-122.4094, 37.7849],\n ]\n ],\n}\n",+ [5, 6],+ ),+ # A JS object with bare keys.+ (+ "{\n type: 'MultiLineString',\n coordinates: [\n [\n [-122.4194, 37.7749],\n"+ " [-122.4094, 37.7849]\n ]\n ]\n}\n",+ [5, 6],+ ),+ # A bare type value must use GeoJSON's spelling.+ (+ "{\n type: polygon,\n coordinates: [\n [\n [0.123456, 0.654321],\n"+ " [0.234567, 0.765432]\n ]\n ]\n}\n",+ [],+ ),+ # A non-geographic "type" does not make its coordinates location data.+ (+ '{\n "type": "Mesh",\n "coordinates": [\n [\n [\n'+ " [0.123456, 0.654321],\n [0.234567, 0.765432]\n ]\n ]\n ]\n}\n",+ [],+ ),+ ],+ )+ def test_gps_pretty_printed_geojson_geometry_positions_flagged(+ self, tmp_path: Path, contents: str, expected_lines: list[int]+ ):+ """Every position inside a GeoJSON geometry's coordinates array is cued, however deep it is nested."""+ geojson_file = tmp_path / "area.json"+ geojson_file.write_text(contents, encoding="utf-8")++ candidates = SecurityValidator()._scan_file_for_pii(geojson_file)++ assert [item["line"] for item in candidates if item["category"] == "gps_coordinates"] == expected_lines++ def test_gps_deeply_nested_geojson_inner_position_flagged(self, tmp_path: Path):+ """Nested coordinates arrays are merged before marking lines, and the inner position stays cued."""+ depth = 2000+ geojson_file = tmp_path / "nested.json"+ geojson_file.write_text(+ '{"type": "Point", "coordinates": [\n' * depth + "-122.419416, 37.774929\n" + "]}\n" * depth,+ encoding="utf-8",+ )++ candidates = SecurityValidator()._scan_file_for_pii(geojson_file)++ assert [item["line"] for item in candidates if item["category"] == "gps_coordinates"] == [depth + 1]+ assert SecurityValidator._geojson_coordinate_lines(geojson_file.read_text(encoding="utf-8").splitlines()) == (+ frozenset(range(1, 2 * depth + 2))+ )+# --- Phone number false positives ---def test_phone_bare_ten_digit_run_not_flagged(self, tmp_path: Path):@@ -6467,8 +6690,141 @@ class TestFalsePositivePrevention:f"Formatted phone number should still be flagged: {all_findings}")+ def test_phone_scientific_notation_matrix_not_flagged(self, tmp_path: Path):+ """The exponent of a scientific-notation value is not an international phone prefix."""+ (tmp_path / "matrix.txt").write_text(+ "1.000000e+00 0.000000e+00 0.000000e+00 1.234567e-01\n+1.000000e+00 -2.500000E+03\n",+ encoding="utf-8",+ )++ result = SecurityValidator().validate_pii_only(tmp_path)++ assert not any(finding.check_name == "phone_numbers" for finding in result.findings)++ def test_phone_after_scientific_notation_still_flagged(self, tmp_path: Path):+ """Dropping a candidate inside a scientific literal keeps a real phone later on the line."""+ (tmp_path / "notes.txt").write_text(+ "scale 1.000000e+00 0.000000e+00, support +44 20 7946 0958\n",+ encoding="utf-8",+ )++ result = SecurityValidator().validate_pii_only(tmp_path)++ phones = [+ finding.metadata["matched_value"] for finding in result.findings if finding.check_name == "phone_numbers"+ ]+ assert phones == ["+44 20 7946 0958"]+# --- Credit card false positives ---+ @pytest.mark.parametrize(+ "technical_value",+ [+ "timestamp_microseconds: 1707938736269577",+ "epoch nanoseconds = 1707938736269577",+ '{"unix_time": 1707938736269577}',+ "sha256: 1707938736269577",+ "checksum=1707938736269577",+ "sha256 1707938736269577",+ '"sha256" "1707938736269577"',+ f"sha256={'a' * 24}1707938736269577{'b' * 24}",+ "build id_1707938736269577",+ "1707938736269577_suffix",+ # A snake_case prefix still names a timestamp or digest field.+ "event_timestamp: 1707938736269577",+ '"created_at_epoch": 1707938736269577',+ "file_sha256: 1707938736269577",+ # Short unit suffixes after epoch and unix_time name microsecond epochs.+ "epoch_us: 1707938736269577",+ '"epoch_us": 1707938736269577',+ "epoch_micros: 1707938736269577",+ "unix_time_us: 1707938736269577",+ ],+ )+ def test_credit_card_shape_in_timestamp_or_hash_context_not_flagged(self, tmp_path: Path, technical_value: str):+ """A Luhn-valid timestamp, digest, or identifier is not a blocking card number."""+ (tmp_path / "data.txt").write_text(f"{technical_value}\n", encoding="utf-8")++ result = SecurityValidator().validate_pii_only(tmp_path)++ assert result.passed+ assert not any(finding.check_name == "credit_cards" for finding in result.findings)++ @pytest.mark.parametrize(+ ("contents", "card"),+ [+ ("timestamp_microseconds: 1707938736269577 card_number: 4532015112830366", "4532015112830366"),+ ("known=4111111111111111; card_number=4532015112830366", "4532015112830366"),+ ("Use 4111-1111-1111-1111 or 4532 0151 1283 0366", "4532 0151 1283 0366"),+ ("checksum verified; card: 4532015112830366", "4532015112830366"),+ # Markdown underscore emphasis is not an identifier joining the number to a word.+ ("Use _4532015112830366_ at checkout", "4532015112830366"),+ ("Use __4532015112830366__ at checkout", "4532015112830366"),+ # Letters outside ASCII do not hide a card written directly next to them.+ ("カード番号は4532015112830366です", "4532015112830366"),+ ("我的卡号是4532015112830366。", "4532015112830366"),+ ("카드번호는4532015112830366입니다", "4532015112830366"),+ ("หมายเลขบัตร4532015112830366", "4532015112830366"),+ # IGNORECASE must not let the ASCII class match the dotless i or the Kelvin sign.+ ("Kart numaras\u01314532015112830366", "4532015112830366"),+ ("Ref \u212a4532015112830366", "4532015112830366"),+ # A backslash escape or a percent-encoded byte is a separator, not part of a token.+ ('{"prompt": "Charge this card:\\n4532015112830366\\nExp 12/27"}', "4532015112830366"),+ ('{"input": "Card\\t4532015112830366"}', "4532015112830366"),+ ('prompt: "Pay with\\r4532015112830366 now"', "4532015112830366"),+ ('msg = "Card:\\n4532 0151 1283 0366"', "4532 0151 1283 0366"),+ ("https://pay.acme.io/charge?q=card%3D4532015112830366", "4532015112830366"),+ # A payment word before the underscore names a card, not an identifier.+ ("card_4532015112830366", "4532015112830366"),+ ("CARD_4532015112830366.png", "4532015112830366"),+ ("cc_4532015112830366", "4532015112830366"),+ ("creditCard_4532015112830366", "4532015112830366"),+ ("cardNumber_4532015112830366", "4532015112830366"),+ ("ccNum_4532015112830366", "4532015112830366"),+ ("payment_card_4532015112830366", "4532015112830366"),+ ],+ )+ def test_credit_card_next_to_technical_or_test_value_still_flagged(self, tmp_path: Path, contents: str, card: str):+ """A timestamp label, test card, emphasis, or non-ASCII letter excludes no card number next to it."""+ data_file = tmp_path / "data.txt"+ data_file.write_text(f"{contents}\n", encoding="utf-8")++ result = SecurityValidator().validate_pii_only(tmp_path)++ assert not result.passed+ assert [finding.check_name for finding in result.findings].count("credit_cards") == 1+ # Reports may redact card values, so check the scanner's own candidate.+ candidates = SecurityValidator()._scan_file_for_pii(data_file)+ assert [item["matched_value"] for item in candidates if item["category"] == "credit_cards"] == [card]++ @pytest.mark.parametrize("card", ["4111 1111 1111 1111", "5500 0000 0000 0004", "4111111111111111"])+ def test_known_test_card_in_any_grouping_not_flagged(self, tmp_path: Path, card: str):+ """Known test card numbers are excepted whether written with spaces, dashes, or no separators."""+ (tmp_path / "data.txt").write_text(f"Sandbox card: {card}\n", encoding="utf-8")++ result = SecurityValidator().validate_pii_only(tmp_path)++ assert not any(finding.check_name == "credit_cards" for finding in result.findings)++ @pytest.mark.parametrize(+ "contents",+ [+ "Cards: 4111111111111111, 4242424242424242",+ "Sandbox cards: 378282246310005 and 5555555555554444",+ "Test cards: 4111-1111-1111-1111 / 5555-5555-5555-4444",+ "Visa 4012 8888 8888 1881, Mastercard 5105-1051-0510-5100, Discover 6011111111111117",+ "Debit 4000056655665556, 2-series 2223003122003222, JCB 3530111333300000",+ ],+ )+ def test_line_of_published_test_cards_not_flagged(self, tmp_path: Path, contents: str):+ """Payment docs that list several published test cards on one line pass the gate."""+ (tmp_path / "SKILL.md").write_text(f"{contents}\n", encoding="utf-8")++ result = SecurityValidator().validate_pii_only(tmp_path)++ assert result.passed+ assert not any(finding.check_name == "credit_cards" for finding in result.findings)+def test_credit_card_non_luhn_number_not_flagged(self, tmp_path: Path):"""16-digit numbers that fail Luhn should not be flagged as credit cards."""skill_dir = tmp_path / "cc-nonluhn"