Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(tier1): apply audited SkillSpector baselines (#300)
Narendran Raghavan39d8901104fb
diff --git a/.gitleaks.toml b/.gitleaks.tomlindex f00073bf..85dcf5b8 100644--- a/.gitleaks.toml+++ b/.gitleaks.toml@@ -38,6 +38,13 @@ targetRules = ["generic-api-key"]regexes = ['''^us\.anthropic\.claude-opus-5$''']paths = ['''^tests/validators/test_security\.py$''']+[[allowlists]]+description = "Bedrock model ID in the merge-leftover SkillSpector bridge test copy from commit e8aa35d6a"+condition = "AND"+targetRules = ["generic-api-key"]+regexes = ['''^us\.anthropic\.claude-opus-5$''']+paths = ['''^tests/validators/test_security\.py\.orig$''']+[[allowlists]]description = "Synthetic HTTP hook credential used by static risk tests"condition = "AND"diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 1e8234d4..37b9c362 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -182,6 +182,26 @@ All notable changes to SkillEvaluator are documented in this file.`tier1` leaves a check out and reports it as skipped. A `scanners` row and theJSON report record which scanner versions ran.+- `--skillspector-baseline FILE` on `validate`, `tier1`, and `security-scan`+ applies an audited SkillSpector suppression baseline, so reviewed false+ positives no longer leave the security check failed or `INCOMPLETE`.+ Baselines apply only when passed explicitly; one shipped inside the skill is+ never used, and suppressed findings without a requested baseline still fail+ closed. A rule without a literal run in its `id` (two characters)+ or in its `path` or `message` (four), such as `id: "*"` or `message: "*e*"`, or a version 2 rule or fingerprint without a `reason`, is+ rejected before the scan, as are entries that are not mappings, unparseable+ files, and files beyond SkillSpector's size and YAML expansion limits; an+ empty file is an empty baseline. A baseline kept inside the skill is+ excluded from the scan, including the staged copy used when the skill has+ `evals/` outputs. JSON reports add `results[].skillspector_baseline` with the+ outcome (`applied`, `no_matches`, `fingerprint_version_mismatch`,+ `llm_stage_unmatched`, or `not_applied`), the baseline file name and+ `sha256:` digest (never its local path), scanner versions, and per-stage+ finding counts with the suppressed findings' severities and rule IDs, and Markdown and HTML reports show the outcome as a+ `skillspector_baseline` line. Fingerprints ignored after a SkillSpector+ upgrade, or not matched by the LLM stage, are reported instead of silently+ suppressing nothing; fingerprints only match static-only runs.+### Changed- Shared LLM calls and the Harbor verifier retry HTTP 408, 429, and every 5xxdiff --git a/docs/ci-integration.mdx b/docs/ci-integration.mdxindex 2562f1a9..85ccb869 100644--- a/docs/ci-integration.mdx+++ b/docs/ci-integration.mdx@@ -244,6 +244,7 @@ When the exit code is not enough — dashboards, custom thresholds, badge genera| `benchmark_policy` | object | Persisted `tier2_required` and `tier3_required` evidence policy used for the publication decision || `results[].publication_target` | object | Exact NFC filesystem-entry name, versioned source-tree digest, and digest algorithm used to bind split-tier evidence || `results[].publication_evidence` | object | Validated built-in Tier 1 or Tier 2 producer contract: schema version, producer, tier, and canonical check ID |+| `results[].skillspector_baseline` | object | Single-skill targets with `--skillspector-baseline`, when SkillSpector is installed: `status`, baseline file name and `sha256:` digest (no local path), scanner versions, suppressed (per stage) and active finding counts, and a message. Absent from merged `security-scan` folder reports; `validate` on a folder records it in each skill's report. See [Audited SkillSpector baselines](tier1-validation.mdx#audited-skillspector-baselines) || `tier3.run_id` | string | Run-owned Tier 3 identity; duplicated in the Tier 3 summary for consistency checks || `severity_counts` | object | Totals for `critical`, `high`, `medium`, `low` || `total_errors`, `total_warnings` | number | Aggregate counts across all validators |diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdxindex 4e87a393..24c6bb05 100644--- a/docs/cli-reference.mdx+++ b/docs/cli-reference.mdx@@ -134,6 +134,7 @@ errors, such as an option the workflow does not accept, exit `2`.| `--quality-config PATH` | packaged `quality-default.yaml` | YAML file that replaces the quality check's expected-heading rules. See [Tier 1: Validation](tier1-validation.mdx#the-checks). || `--profile TEXT` | `SKILLEVALUATOR_PROFILE`, then `external` | Validation policy profile || `--previous-version TEXT` | `SKILLEVALUATOR_PREVIOUS_VERSION`, else unset | Previous version for version checks |+| `--skillspector-baseline FILE` | unset | Audited SkillSpector suppression baseline; never discovered from the skill. See [Audited SkillSpector baselines](tier1-validation.mdx#audited-skillspector-baselines). |Also accepts `-r/--report` and `-o/--output-dir` described above.@@ -243,6 +244,7 @@ Static checks; LLM-free by default. Tier 1 gates the exit code and always runs.| `-c, --continue-on-failure` | off | Run the full pipeline without stopping early; record all issues in the reports. Overrides `--fail-fast`, and for folder validation keeps scanning every skill past a CRITICAL finding. || `--llm, --tier1-llm / --no-llm, --no-tier1-llm` | `no-llm` | Enable LLM-backed security analysis (requires a configured public provider — see [Providers & Credentials](configuration.mdx)). || `--llm-verify` | off | Run a second LLM pass to suppress false-positive findings. |+| `--skillspector-baseline FILE` | unset | Audited SkillSpector suppression baseline for the `security` check; never discovered from the skill. See [Audited SkillSpector baselines](tier1-validation.mdx#audited-skillspector-baselines). || `--min-score INTEGER` | `70` | Minimum quality score (0–100) required per skill when the `quality` check runs; policy severity overrides cannot waive this gate. || `--quality-config PATH` | packaged `quality-default.yaml` | YAML file that replaces the quality check's expected-heading rules. See [Tier 1: Validation](tier1-validation.mdx#the-checks). |@@ -342,6 +344,7 @@ skillevaluator security-scan ./my-skill| --- | --- | --- || `--llm / --no-llm` | `no-llm` | Enable LLM security analysis. || `--llm-verify` | off | Use LLM verification to reduce false positives. |+| `--skillspector-baseline FILE` | unset | Audited SkillSpector suppression baseline; never discovered from the skill. See [Audited SkillSpector baselines](tier1-validation.mdx#audited-skillspector-baselines). |Plus the standard report options.diff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdxindex 8b54d252..1bd23836 100644--- a/docs/tier1-validation.mdx+++ b/docs/tier1-validation.mdx@@ -191,6 +191,8 @@ For `validate`, the default set is `schema`, `version`, `security`, `pii`, `liceSkillSpector 2.11.1 and later tags an exact read-only `/etc/passwd` bind mount, such as `docker run -v /etc/passwd:/etc/passwd:ro` or the Podman, `--volume`, and quoted forms, as likely benign but still reports it as a HIGH Credential Access (PE3) finding. SkillEvaluator ignores that finding only when SkillSpector tagged it benign and every `/etc/passwd` on the reported source line is the source or target of such a mount. Writable or implicit-mode mounts, other source or target paths, and reads such as `cat /etc/passwd` still block.SkillSpector 2.13.3 can report separate findings for the mount's source and target; both must pass the same checks. Ignoring a mount never clears incomplete inspection, including refused or truncated LLM responses. A static report can say an LLM provider is available without having requested or applied LLM analysis; provider availability alone does not invalidate the static scan.++ To accept reviewed findings, pass an audited baseline with `--skillspector-baseline`; see [Audited SkillSpector baselines](#audited-skillspector-baselines).</Accordion><Accordion title="pii — PII, credentials, and local identifiers">@@ -400,6 +402,57 @@ Bundled Python scanners resolve next to the SkillEvaluator interpreter before `PScanner path overrides fail closed: a relative, missing, or non-executable override path fails the run rather than silently falling back. Required scanners that are missing or produce malformed output leave the run `INCOMPLETE` and non-green.</Warning>+## Audited SkillSpector baselines++Record reviewed SkillSpector exceptions in a SkillSpector YAML or JSON baseline and pass it with `--skillspector-baseline` to `validate`, `tier1`, or `security-scan`. SkillEvaluator never discovers a baseline on its own, including a `.skillspector-baseline.yaml` shipped inside the skill: applying exceptions is an operator or CI decision. A scan that reports suppressed findings without a requested baseline stays `INCOMPLETE`.++```bash title="Apply a reviewed baseline"+skillevaluator validate ./my-skill \+ --skillspector-baseline ./trusted/skillspector-baseline.yaml+```++Keep the baseline outside the skill under review, for example in a checkout of the base branch or a separate policy repository. A baseline kept inside the skill is only as trusted as the review of that file: whoever edits the skill can edit its exceptions, SkillSpector leaves the selected baseline file out of its own scan so extra text in it is never inspected, and a version 2 rule needs only a non-empty `reason`. In a pull-request pipeline, the checked-out skill is the PR's version, so pass a baseline from the target branch instead.++Prefer narrow rules. Every field a rule sets must match the finding. `id` (the rule ID), `path` (the file), and `message` (the finding text) are all case-insensitive globs, and `reason` is required.++```yaml title="config/skillspector-baseline.yaml"+version: 2+rules:+ - id: "PE5"+ path: "SKILL.md"+ message: "*--privilege[d]*"+ reason: "Reviewed: the diagnostics container needs host device access."+```++The bracket class `[d]` matches the letter `d`, so the rule still matches `--privileged`, but the baseline file does not contain the literal text that triggers the finding. SkillEvaluator rejects a rule unless its `id` has at least two literal letters or digits in a row, or its `path` or `message` has at least four. Wildcards and multi-character classes break a run. This rejects `id: "*"` and also `message: "*e*"`, which matches almost every finding because SkillSpector matches the glob case-insensitively against the finding text. Before the scan it also rejects a version 2 rule or fingerprint without a `reason`, `rules` or `fingerprints` that are not lists or contain entries that are not mappings, a file that is not valid YAML or JSON (including an impossible date such as `2026-06-31`), and a file larger than 2 MiB or whose aliases and merge keys expand beyond SkillSpector's own limits. An empty or comment-only file is an empty baseline, as in SkillSpector. SkillSpector checks the rest of the schema. If it rejects the baseline, it exits with code 2, the security check is `INCOMPLETE`, and the error suggests checking the file with `skillspector scan <skill> --baseline <file>`.++To waive individual findings exactly, generate fingerprints with SkillSpector. The command records every current finding, so review each entry before you commit the file:++```bash title="Generate exact fingerprints"+skillspector baseline ./my-skill --no-llm \+ --reason "Reviewed: <rationale>" \+ -o ./my-skill/config/skillspector-baseline.yaml+```++Fingerprints bind to the scanned file content and to the SkillSpector version that generated them. After a source edit, the affected findings are reported again. After a scanner upgrade, SkillSpector ignores every fingerprint, and SkillEvaluator reports `fingerprint_version_mismatch` with both versions: as an error when blocking findings remain, otherwise as a warning. Rules keep working across scanner versions.++Fingerprints only match static-only runs: `tier1 --no-llm`, or `validate` and `security-scan` without `--llm`. When the LLM security stage runs (`tier1` by default when a provider is configured, or `--llm`), SkillSpector's meta-analysis rewrites each finding's message, explanation, remediation, confidence, or tags before the baseline is applied. Those fields are part of the fingerprint, so fingerprints from `skillspector baseline --no-llm` no longer match, and the LLM stage reports the findings again. SkillEvaluator then reports `llm_stage_unmatched` with both stages' suppressed counts: as an error when blocking findings remain, otherwise as a warning. For runs with LLM analysis, use narrow `id`, `path`, and `message` rules instead; they still match because SkillSpector also checks `message` against the matched text.++A baseline stored inside the skill is excluded from the scan, including when SkillEvaluator scans a staged copy without `evals/` outputs. For a folder of skills, the same baseline applies to each skill, and rule paths are relative to each skill.++For a single-skill target, when SkillSpector is installed, JSON reports record the outcome under `results[].skillspector_baseline`:++- `status`: `applied`, `no_matches`, `fingerprint_version_mismatch`, `llm_stage_unmatched`, or `not_applied`.+- `file` and `digest`: the baseline's file name and a `sha256:` digest of its bytes. The local path is not recorded.+- The baseline and runtime scanner versions, and the rule and fingerprint counts.+- `suppressed_findings` from the static stage and, when the LLM stage completed, `llm_stage_suppressed_findings`; and the active and active blocking finding counts. All are taken from SkillSpector's reports before any `--llm-verify` downgrade or policy severity override.+- `suppressed_by_severity` and `suppressed_rule_ids`: the static stage's suppressed findings by severity and their rule IDs, so CI can refuse a baseline that hides anything blocking. The Markdown and HTML line shows the severity counts.+- `message`: the outcome or the reason the baseline was rejected.++Markdown and HTML reports show the same outcome as a `skillspector_baseline` line in the Security result, with the file name and digest, and a scan whose only findings were suppressed reads "No active security findings; N finding(s) suppressed by the requested baseline" rather than a clean result.++`security-scan` and `tier1 security-scan` on a folder of skills write one merged report without the JSON field. When a skill in the folder passes, its outcome is appended to that skill's "All checks passed" line in the Markdown and HTML reports. `validate` on a folder validates each skill as its own job, so each skill's JSON report records the field. The field keeps the `fingerprint_version_mismatch` and `llm_stage_unmatched` messages even when a later `--llm-verify` downgrade or policy severity override rebuilds the error and warning lists. See the [SkillSpector suppression reference](https://github.com/NVIDIA/SkillSpector/blob/v2.13.3/docs/SUPPRESSION.md) for the full baseline schema.+## Exit behavior and scoring| Exit code | Meaning |diff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.pyindex e51255c9..a2843d88 100644--- a/src/skillevaluator/cli.py+++ b/src/skillevaluator/cli.py@@ -128,6 +128,11 @@ class _QualityConfigPath(click.Path):QUALITY_CONFIG_PATH = _QualityConfigPath()+# Resolved so parallel catalog children receive the same absolute file.+SKILLSPECTOR_BASELINE_PATH = click.Path(exists=True, dir_okay=False, readable=True, resolve_path=True, path_type=Path)+_SKILLSPECTOR_BASELINE_HELP = (+ "Apply an audited SkillSpector YAML/JSON suppression baseline. Never discovered from the scanned skill."+)def _validate_similarity_threshold(_ctx: click.Context, _param: click.Parameter, value: float) -> float:@@ -1220,6 +1225,8 @@ def _catalog_child_argv_from_ctx(argv.append("--no-llm")if params.get("llm_verify"):argv.append("--llm-verify")+ if params.get("skillspector_baseline"):+ argv.extend(["--skillspector-baseline", str(params["skillspector_baseline"])])if not params.get("dedup", True):argv.append("--no-dedup")block_on_dedup = params.get("block_on_dedup")@@ -1777,6 +1784,14 @@ def _print_run_banner(target_path: Path, content_type: str, profile: str | None)help_group=_TIER1_GROUP,help="Run a second LLM pass to suppress false-positive findings.",)+ "--skillspector-baseline",+ type=SKILLSPECTOR_BASELINE_PATH,+ default=None,+ cls=GroupedOption,+ help_group=_TIER1_GROUP,+ help=_SKILLSPECTOR_BASELINE_HELP,+)@click.option("--min-score",type=int,@@ -2053,6 +2068,7 @@ def validate(continue_on_failure: bool,llm: bool,llm_verify: bool,+ skillspector_baseline: Path | None,min_score: int,quality_config: Path | None,profile: str | None,@@ -2284,6 +2300,7 @@ def validate(checks=checks,use_llm=llm,llm_verify=llm_verify,+ skillspector_baseline=skillspector_baseline,min_score=min_score,quality_config=quality_config,previous_version=previous_version,@@ -2616,14 +2633,22 @@ def rubric_eval(target_path: Path, min_score: int, report_formats: tuple[str, ..@_target_argument@click.option("--llm/--no-llm", default=False, show_default=True, help="Enable LLM security analysis.")@click.option("--llm-verify", is_flag=True, help="Use LLM verification to reduce false positives.")+ "--skillspector-baseline", type=SKILLSPECTOR_BASELINE_PATH, default=None, help=_SKILLSPECTOR_BASELINE_HELP+)@_report_optionsdef security_scan(- target_path: Path, llm: bool, llm_verify: bool, report_formats: tuple[str, ...], output_dir: Path+ target_path: Path,+ llm: bool,+ llm_verify: bool,+ skillspector_baseline: Path | None,+ report_formats: tuple[str, ...],+ output_dir: Path,) -> None:"""Scan for security vulnerabilities."""output_dir = _resolve_file_report_output_location(target_path.resolve(), output_dir, report_formats)if not emit_reports(- run_security_scan(target_path, use_llm=llm, llm_verify=llm_verify),+ run_security_scan(target_path, use_llm=llm, llm_verify=llm_verify, skillspector_baseline=skillspector_baseline),report_formats=report_formats,output_dir=output_dir,basename=report_basename("security"),@@ -2872,6 +2897,9 @@ def _workflow_report_options(func):)@click.option("--profile", default=None, help="Validation policy profile [default: external].")@click.option("--previous-version", default=None, help="Previous version for version checks.")+ "--skillspector-baseline", type=SKILLSPECTOR_BASELINE_PATH, default=None, help=_SKILLSPECTOR_BASELINE_HELP+)@_workflow_report_optionsdef _tier1_workflow(skill_path: Path,@@ -2882,6 +2910,7 @@ def _tier1_workflow(quality_config: Path | None,profile: str | None,previous_version: str | None,+ skillspector_baseline: Path | None,report_formats: tuple[str, ...],output_dir: Path,) -> None:@@ -2903,6 +2932,7 @@ def _tier1_workflow(quality_config=quality_config,profile=policy,previous_version=previous_version,+ skillspector_baseline=skillspector_baseline,)# Resolved after the run so configuration errors still leave no output.output_dir = _resolve_file_report_output_location(skill_path.resolve(), output_dir, report_formats)diff --git a/src/skillevaluator/reporting/json_reporter.py b/src/skillevaluator/reporting/json_reporter.pyindex 936766d1..c50f0af5 100644--- a/src/skillevaluator/reporting/json_reporter.py+++ b/src/skillevaluator/reporting/json_reporter.py@@ -332,6 +332,12 @@ class JSONReporter(ReporterBase):if qs:data["quality"] = qs+ # Whether an explicitly requested SkillSpector baseline was applied, so+ # CI can tell an inert baseline from one that matched nothing.+ skillspector_baseline = result.metadata.get("skillspector_baseline")+ if isinstance(skillspector_baseline, dict):+ data["skillspector_baseline"] = dict(skillspector_baseline)+# Tier 3: Agent evaluation dataae = result.metadata.get("agent_eval")if ae:diff --git a/src/skillevaluator/tier1/commands.py b/src/skillevaluator/tier1/commands.pyindex 11374341..7c19f9e3 100644--- a/src/skillevaluator/tier1/commands.py+++ b/src/skillevaluator/tier1/commands.py@@ -212,6 +212,7 @@ def run_validation(fail_fast: bool = False,continue_on_failure: bool = False,on_check: Callable[[str], None] | None = None,+ skillspector_baseline: Path | None = None,skip: str | None = None,) -> list[ValidationResult]:"""Run selected Tier 1 validators and return structured results.@@ -234,6 +235,9 @@ def run_validation(When *policy* is provided, the schema validator applies the policy'saudience-aware author rules; finalized severities for all validators areapplied centrally in :func:`emit_reports` via the policy.++ *skillspector_baseline* is an audited SkillSpector suppression baseline+ applied to the ``security`` check; it is never discovered from the target."""enabled = _enabled_checks(checks)skipped = skipped_checks(skip)@@ -272,7 +276,7 @@ def run_validation(return [_run_result("schema", v.name, v.description, v.validate)]def _security_results() -> list[ValidationResult]:- v = SecurityValidator(use_llm=use_llm, verify_llm=llm_verify)+ v = SecurityValidator(use_llm=use_llm, verify_llm=llm_verify, skillspector_baseline=skillspector_baseline)return [_run_result("security", "Security Scan", v.description, v.validate_security_only)]def _pii_results() -> list[ValidationResult]:@@ -412,8 +416,9 @@ def run_security_scan(*,use_llm: bool = False,llm_verify: bool = False,+ skillspector_baseline: Path | None = None,) -> list[ValidationResult]:- validator = SecurityValidator(use_llm=use_llm, verify_llm=llm_verify)+ validator = SecurityValidator(use_llm=use_llm, verify_llm=llm_verify, skillspector_baseline=skillspector_baseline)return [_as_result("Security Scan",diff --git a/src/skillevaluator/tier_workflows.py b/src/skillevaluator/tier_workflows.pyindex a6bf264b..aa15f462 100644--- a/src/skillevaluator/tier_workflows.py+++ b/src/skillevaluator/tier_workflows.py@@ -72,6 +72,7 @@ def run_tier1_workflow(quality_config: Path | None,profile: ValidationPolicy,previous_version: str | None,+ skillspector_baseline: Path | None = None,skip: str | None = None,) -> list[ValidationResult]:require_skill(skill_path)@@ -99,6 +100,7 @@ def run_tier1_workflow(policy=profile,content_type="skill",continue_on_failure=True,+ skillspector_baseline=skillspector_baseline,skip=skip,)apply_policy(results, profile)diff --git a/src/skillevaluator/validators/base.py b/src/skillevaluator/validators/base.pyindex b585c09f..7d020778 100644--- a/src/skillevaluator/validators/base.py+++ b/src/skillevaluator/validators/base.py@@ -273,9 +273,16 @@ class ValidatorBase(ABC):for detail in skill_result.success_details]+ # A reviewed baseline suppressing findings must stay visible in+ # reports that show only this one-line summary.+ notes = [+ detail.message+ for detail in skill_result.success_details+ if detail.check_name == "skillspector_baseline" or "suppressed_count" in detail.metadata+ ]result.add_success(check_name=skill_name,- message="All checks passed",+ message="All checks passed" + (f"; {'; '.join(notes)}" if notes else ""),checks=check_details,total_checks=len(check_details),)diff --git a/src/skillevaluator/validators/security.py b/src/skillevaluator/validators/security.pyindex f16265cb..14b42c80 100644--- a/src/skillevaluator/validators/security.py+++ b/src/skillevaluator/validators/security.py@@ -14,6 +14,7 @@ from __future__ import annotationsimport contextlibimport getpass+import hashlibimport ioimport jsonimport math@@ -74,6 +75,15 @@ _SKILLSPECTOR_TIMEOUT_ENV = "SKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDS"_AUTHOR_IDENTITY_RE = re.compile(r"^\S[^<>\n]* <(?P<email>[^<>@\s]+@[^<>\s]+)>$")_SKILLSPECTOR_POLICY_EXIT_CODES = frozenset({0, 1})+# SkillSpector maps an unreadable or invalid --baseline file to exit code 2.+_SKILLSPECTOR_INPUT_ERROR_EXIT_CODE = 2+_SKILLSPECTOR_BASELINE_SCHEMA_VERSION = 2+# SkillSpector's baseline loader limits (v2.12.0 and later); a file beyond+# them is rejected by SkillSpector, so the pre-check stops there too.+_SKILLSPECTOR_BASELINE_MAX_BYTES = 2 * 1024 * 1024+_SKILLSPECTOR_BASELINE_MAX_NODES = 100_000+_SKILLSPECTOR_BASELINE_MAX_DEPTH = 64+_SKILLSPECTOR_BASELINE_MAX_SCALAR_CHARS = 64 * 1024_SKILLSPECTOR_STATUSLESS_COMPLETENESS_VERSIONS = {(2, 9, 5), (2, 9, 6)}_SKILLSPECTOR_FINDING_IDENTITY_VERSION = (2, 11, 1)_SKILLSPECTOR_COMPLETENESS_SCHEMA_VERSION = (2, 10, 0)@@ -582,6 +592,277 @@ def _ignore_artifact_dirs(dirpath: str, names: list[str]) -> set[str]:return {n for n in names if n in _SKILLSPECTOR_SCAN_EXCLUDED_DIRS and Path(dirpath, n).is_dir()}+_SKILLSPECTOR_SUPPRESSED_RULE_ID_LIMIT = 50++# Shortest run of literal characters that bounds a rule glob. SkillSpector matches+# message and path globs case-insensitively against free text, so a single+# letter such as ``*e*`` matches almost every finding.+_BASELINE_ID_MIN_LITERAL_RUN = 2+_BASELINE_TEXT_MIN_LITERAL_RUN = 4+++def _baseline_glob_has_literal_scope(value: object, *, min_run: int = 1) -> bool:+ """Return whether a SkillSpector suppression glob has a literal run of ``min_run`` characters.++ Wildcards, negated classes, and multi-character classes match too broadly+ to bound a rule and break a run; a one-character class such as ``[d]`` is a+ literal. Only letters and digits count toward a run.+ """+ if not isinstance(value, str):+ return False+ pattern = value.strip()+ run = 0+ index = 0+ while index < len(pattern):+ character = pattern[index]+ if character in "*?":+ run = 0+ index += 1+ continue+ if character != "[":+ if character.isalnum():+ run += 1+ if run >= min_run:+ return True+ index += 1+ continue+ cursor = index + 1+ negated = cursor < len(pattern) and pattern[cursor] in "!^"+ if negated:+ cursor += 1+ if cursor < len(pattern) and pattern[cursor] == "]":+ cursor += 1+ close = pattern.find("]", cursor)+ if close < 0:+ # fnmatch treats an unclosed bracket as literal text.+ return len(pattern) - index + run >= min_run+ class_body = pattern[index + 1 : close]+ if not negated and len(class_body) == 1 and class_body.isalnum():+ run += 1+ if run >= min_run:+ return True+ else:+ run = 0+ index = close + 1+ return False+++class _SkillSpectorBaselineLimitError(ValueError):+ """A baseline exceeds the size or YAML expansion limits SkillSpector enforces."""+++class _BoundedSkillSpectorBaselineLoader(yaml.SafeLoader):+ """SafeLoader that bounds composition and alias or merge expansion like SkillSpector.++ Plain ``yaml.safe_load`` expands chained merge keys without limit, so a+ small file could take minutes or exhaust memory before any check runs.+ Expansion is counted on the composed node graph, revisiting every alias,+ before any value is constructed.+ """++ def __init__(self, stream: str) -> None:+ super().__init__(stream)+ self._composed_nodes = 0+ self._compose_depth = 0++ def compose_node(self, parent: object, index: object) -> yaml.Node:+ self._composed_nodes += 1+ if self._composed_nodes > _SKILLSPECTOR_BASELINE_MAX_NODES:+ raise _SkillSpectorBaselineLimitError("exceeds the YAML node limit")+ if self._compose_depth >= _SKILLSPECTOR_BASELINE_MAX_DEPTH:+ raise _SkillSpectorBaselineLimitError("exceeds the YAML depth limit")+ self._compose_depth += 1+ try:+ return super().compose_node(parent, index)+ finally:+ self._compose_depth -= 1++ def construct_document(self, node: yaml.Node) -> object:+ counts = {"nodes": 0, "characters": 0}+ active: set[int] = set()++ def visit(item: yaml.Node, depth: int) -> None:+ counts["nodes"] += 1+ if counts["nodes"] > _SKILLSPECTOR_BASELINE_MAX_NODES:+ raise _SkillSpectorBaselineLimitError("exceeds the expanded YAML node limit")+ if depth > _SKILLSPECTOR_BASELINE_MAX_DEPTH:+ raise _SkillSpectorBaselineLimitError("exceeds the expanded YAML depth limit")+ if id(item) in active:+ raise _SkillSpectorBaselineLimitError("contains cyclic YAML aliases")+ if isinstance(item, yaml.ScalarNode):+ if len(item.value) > _SKILLSPECTOR_BASELINE_MAX_SCALAR_CHARS:+ raise _SkillSpectorBaselineLimitError("exceeds the YAML scalar character limit")+ counts["characters"] += len(item.value)+ if counts["characters"] > _SKILLSPECTOR_BASELINE_MAX_BYTES:+ raise _SkillSpectorBaselineLimitError("exceeds the expanded YAML character limit")+ return+ active.add(id(item))+ children = (+ (child for pair in item.value for child in pair) if isinstance(item, yaml.MappingNode) else item.value+ )+ for child in children:+ visit(child, depth + 1)+ active.remove(id(item))++ # Depth is bounded, so this recursion stays shallow; every alias is+ # revisited because counting unique nodes misses merge amplification.+ visit(node, 1)+ return super().construct_document(node)+++def _is_non_empty_text(value: object) -> bool:+ """Return whether a parsed YAML value is a string with visible text."""+ return isinstance(value, str) and bool(value.strip())+++def _inspect_skillspector_baseline(path: Path) -> tuple[dict, str | None]:+ """Return the reportable facts of an operator-selected baseline and any blocking error.++ SkillSpector validates the full schema itself. SkillEvaluator rejects a+ file it cannot read, the cheap schema errors SkillSpector would otherwise+ report only as an opaque exit code, and rules whose globs carry no literal+ text, because SkillSpector accepts a rule such as ``id: "*"`` that silently+ suppresses every finding.++ Only the file name and a content digest are recorded, so reports and+ errors do not leak local paths.+ """+ name = path.name+ state: dict = {+ "file": name,+ "digest": None,+ "loaded": False,+ "status": "not_applied",+ "schema_version": None,+ "baseline_scanner_version": None,+ "runtime_scanner_version": None,+ "rule_count": 0,+ "fingerprint_count": 0,+ "suppressed_findings": None,+ "llm_stage_suppressed_findings": None,+ "active_findings": None,+ "active_blocking_findings": None,+ "suppressed_by_severity": {},+ "suppressed_rule_ids": [],+ "message": None,+ }+ if not path.is_file():+ return state, f"SkillSpector baseline {name} does not exist or is not a file"+ try:+ with path.open("rb") as source:+ content = source.read(_SKILLSPECTOR_BASELINE_MAX_BYTES + 1)+ except OSError as exc:+ return state, f"SkillSpector baseline {name} could not be read ({type(exc).__name__})"+ if len(content) > _SKILLSPECTOR_BASELINE_MAX_BYTES:+ return state, f"SkillSpector baseline {name} exceeds the {_SKILLSPECTOR_BASELINE_MAX_BYTES}-byte limit"+ state["digest"] = f"sha256:{hashlib.sha256(content).hexdigest()}"+ try:+ raw = yaml.load(content.decode("utf-8"), Loader=_BoundedSkillSpectorBaselineLoader)+ except _SkillSpectorBaselineLimitError as exc:+ return state, f"SkillSpector baseline {name} {exc}"+ except (+ RecursionError,+ yaml.YAMLError,+ # SafeLoader scalar constructors raise these for values such as an+ # impossible timestamp (``2026-06-31``) or ``!!int ""``.+ ValueError,+ IndexError,+ KeyError,+ AttributeError,+ TypeError,+ OverflowError,+ ) as exc:+ return state, f"SkillSpector baseline {name} could not be parsed ({type(exc).__name__})"+ if raw is None:+ # Like SkillSpector, an empty or comment-only file is an empty baseline.+ raw = {}+ if not isinstance(raw, dict):+ return state, f"SkillSpector baseline {name} must be a YAML or JSON mapping"++ version = raw.get("version")+ rules = raw.get("rules")+ fingerprints = raw.get("fingerprints")+ scanner_version = raw.get("scanner_version")+ state.update(+ {+ "schema_version": version if isinstance(version, int | str) and not isinstance(version, bool) else None,+ "baseline_scanner_version": scanner_version.strip() if _is_non_empty_text(scanner_version) else None,+ "rule_count": len(rules) if isinstance(rules, list) else 0,+ "fingerprint_count": len(fingerprints) if isinstance(fingerprints, list) else 0,+ }+ )+ for field, value in (("rules", rules), ("fingerprints", fingerprints)):+ if value is not None and not isinstance(value, list):+ return state, f"SkillSpector baseline {name} field '{field}' must be a list"+ rule_list = rules or []+ fingerprint_list = fingerprints or []+ # Shape errors first, so a bare string is not reported as an unbounded+ # rule or a missing reason.+ malformed = [f"rule {index}" for index, rule in enumerate(rule_list) if not isinstance(rule, dict)]+ if malformed:+ return state, f"SkillSpector baseline {', '.join(malformed)} in {name} must be a mapping"+ malformed = [f"fingerprint {index}" for index, entry in enumerate(fingerprint_list) if not isinstance(entry, dict)]+ if malformed:+ return state, (+ f"SkillSpector baseline {', '.join(malformed)} in {name} must be a mapping with 'hash' and 'reason'"+ )+ unbounded = [+ index+ for index, rule in enumerate(rule_list)+ # Field aliases match SkillSpector's rule loader.+ if not any(+ _baseline_glob_has_literal_scope(rule.get(field) or rule.get(alias), min_run=min_run)+ for field, alias, min_run in (+ ("id", "rule_id", _BASELINE_ID_MIN_LITERAL_RUN),+ ("path", "file", _BASELINE_TEXT_MIN_LITERAL_RUN),+ ("message", "message", _BASELINE_TEXT_MIN_LITERAL_RUN),+ )+ )+ ]+ if unbounded:+ noun = "rule" if len(unbounded) == 1 else "rules"+ return state, (+ f"SkillSpector baseline {noun} {', '.join(map(str, unbounded))} in {name} must set an id glob with "+ f"{_BASELINE_ID_MIN_LITERAL_RUN} literal characters in a row, or a path or message glob with "+ f"{_BASELINE_TEXT_MIN_LITERAL_RUN}; a broader rule would suppress nearly every finding"+ )+ if version == _SKILLSPECTOR_BASELINE_SCHEMA_VERSION:+ # SkillSpector rejects a version 2 entry without a reason.+ unexplained = [+ f"rule {index}" for index, rule in enumerate(rule_list) if not _is_non_empty_text(rule.get("reason"))+ ]+ unexplained += [+ f"fingerprint {index}"+ for index, entry in enumerate(fingerprint_list)+ if not _is_non_empty_text(entry.get("reason"))+ ]+ if unexplained:+ return state, (+ f"SkillSpector baseline {name} has no reason for {', '.join(unexplained)}; "+ "a version 2 baseline needs a non-empty reason on every rule and fingerprint"+ )+ return state, None+++def _skillspector_baseline_for_scan(original_root: Path, scan_root: Path, baseline: Path | None) -> Path | None:+ """Point SkillSpector at the staged copy of a baseline that lives inside the skill.++ SkillSpector excludes only the exact file passed to ``--baseline`` from its+ content scan. On a staged copy, passing the original path would leave the+ copied baseline in scope, and its reviewed match text would be reported as+ new findings. Baselines outside the skill, or not copied into the staged+ tree, keep their original path.+ """+ if baseline is None or scan_root == original_root:+ return baseline+ try:+ relative = baseline.relative_to(original_root)+ except ValueError:+ return baseline+ staged_baseline = scan_root / relative+ return staged_baseline if staged_baseline.is_file() else baseline++def _rewrite_path_prefix(value, old: str, new: str):"""Recursively rewrite ``old`` path prefixes in a parsed JSON structure."""if isinstance(value, str):@@ -1002,6 +1283,7 @@ class SecurityValidator(ValidatorBase):use_llm: bool = False,verify_llm: bool = False,submitter_usernames: Iterable[str] | None = None,+ skillspector_baseline: Path | str | None = None,):"""Initialize with optional LLM analysis for deeper inspection.@@ -1014,6 +1296,10 @@ class SecurityValidator(ValidatorBase)::data:`HOME_PATH_SUBMITTER_ENV_VARS`) and the OS login name.Pass an explicit value to wire a CI-provided contributor identityor to make tests deterministic.+ skillspector_baseline: Audited SkillSpector YAML/JSON suppression+ baseline. Matching findings are removed by SkillSpector before+ scoring. Baselines are only applied when passed explicitly; one+ shipped inside the scanned skill is never picked up on its own."""self._pii_patterns: dict | None = Noneself._submitter_override: set[str] | None = (@@ -1025,6 +1311,9 @@ class SecurityValidator(ValidatorBase):self._home_check_disabled_warned = Falseself.use_llm = use_llmself.verify_llm = verify_llm+ self.skillspector_baseline = (+ Path(skillspector_baseline).expanduser().resolve() if skillspector_baseline is not None else None+ )@propertydef pii_patterns(self) -> dict:@@ -1199,6 +1488,10 @@ class SecurityValidator(ValidatorBase):on a temp copy without those outputs. Shipped bytecode remains forartifact-integrity analysis, and reported paths map back to theoriginal location.++ An explicitly selected baseline is passed to both stages with+ ``--baseline``. When it lives inside a staged skill, the staged copy is+ passed instead so SkillSpector excludes the file it actually scans."""result = ValidationResult()@@ -1209,6 +1502,16 @@ class SecurityValidator(ValidatorBase):result.mark_scan_incomplete("skillspector-llm" if self.use_llm else "skillspector")return result+ baseline_state: dict | None = None+ if self.skillspector_baseline is not None:+ baseline_state, baseline_error = _inspect_skillspector_baseline(self.skillspector_baseline)+ if baseline_error is not None:+ baseline_state["message"] = baseline_error+ result.metadata["skillspector_baseline"] = baseline_state+ result.add_error(f"{baseline_error}; security scan did not run")+ result.mark_scan_incomplete("skillspector-llm" if self.use_llm else "skillspector")+ return result+original_root = skill_path.resolve()scan_root = original_rootstaged: tempfile.TemporaryDirectory | None = None@@ -1224,6 +1527,7 @@ class SecurityValidator(ValidatorBase):result.add_warning(f"Could not stage artifact-free skill copy ({exc}); scanning in place.")try:+ baseline = _skillspector_baseline_for_scan(original_root, scan_root, self.skillspector_baseline)# The deterministic stage is authoritative and always runs, even# when LLM enrichment was requested. Enrichment is additive and# cannot erase static findings.@@ -1231,13 +1535,22 @@ class SecurityValidator(ValidatorBase):scan_root,original_root=original_root if staged is not None else None,use_llm=False,+ baseline=baseline,)+ # Merging the enrichment stage overwrites this metadata, so keep+ # each stage's own suppression count for the baseline outcome.+ static_suppressed = result.metadata.get("skillspector_suppressed_count")+ static_summary = result.metadata.get("skillspector_suppressed_summary")+ llm_suppressed = Noneif self.use_llm and not result.is_incomplete:enrichment = self._run_skillspector_once(scan_root,original_root=original_root if staged is not None else None,use_llm=True,+ baseline=baseline,)+ if not enrichment.is_incomplete:+ llm_suppressed = enrichment.metadata.get("skillspector_suppressed_count")# Both stages run the same static analyzers, so a note such as# an accepted image gap can repeat word for word; keep one copy.finding_messages = {finding.to_legacy_string() for finding in enrichment.findings}@@ -1253,6 +1566,15 @@ class SecurityValidator(ValidatorBase):result.merge(enrichment)if not enrichment.is_incomplete:self._deduplicate_skillspector_findings(result)+ if baseline_state is not None:+ result.metadata["skillspector_baseline"] = baseline_state+ self._report_skillspector_baseline(+ result,+ baseline_state,+ static_suppressed=static_suppressed,+ llm_suppressed=llm_suppressed,+ static_summary=static_summary,+ )return resultfinally:if staged is not None:@@ -1264,11 +1586,14 @@ class SecurityValidator(ValidatorBase):*,original_root: Path | None,use_llm: bool,+ baseline: Path | None = None,) -> ValidationResult:"""Run one SkillSpector stage and validate its process/report contract."""result = ValidationResult()stage_name = "skillspector-llm" if use_llm else "skillspector"args = ["scan", str(scan_root), "--format", "json"]+ if baseline is not None:+ args.extend(["--baseline", str(baseline)])if not use_llm:args.append("--no-llm")@@ -1312,6 +1637,16 @@ class SecurityValidator(ValidatorBase):"scanner diagnostics were redacted"))+ if (+ baseline is not None+ and tool_result.error_message is None+ and tool_result.exit_code == _SKILLSPECTOR_INPUT_ERROR_EXIT_CODE+ ):+ detail += (+ f". SkillSpector exits {_SKILLSPECTOR_INPUT_ERROR_EXIT_CODE} when it rejects its input, "+ "including an invalid --baseline file; check the baseline with "+ "`skillspector scan <skill> --baseline <file>`"+ )result.add_error(detail)result.mark_scan_incomplete(stage_name)return result@@ -1350,6 +1685,7 @@ class SecurityValidator(ValidatorBase):use_llm,stage_name,result,+ baseline_requested=baseline is not None,scanned_root=original_root if original_root is not None else scan_root,):result.mark_scan_incomplete(stage_name)@@ -1388,10 +1724,14 @@ class SecurityValidator(ValidatorBase):stage_name: str,result: ValidationResult,*,+ baseline_requested: bool = False,scanned_root: Path | None = None,) -> bool:"""Return whether JSON is a trustworthy SkillSpector findings report.+ Suppressed findings are accepted only when SkillEvaluator passed an+ explicit baseline, and that report must then account for them.+A ``partial`` analysis makes the scan INCOMPLETE unless its only gaps arereferenced PNG or JPEG images that the files under *scanned_root* confirm(:func:`_skillspector_passive_raster_note`); that is a warning instead.@@ -2229,6 +2569,13 @@ class SecurityValidator(ValidatorBase):if isinstance(suppressed, list) and not all(isinstance(item, dict) for item in suppressed):result.add_error("skillspector JSON 'suppressed' entries must be objects; security scan did not complete")return False+ if baseline_requested and (suppressed_count is None or suppressed is None):+ # Without both fields, findings could disappear without a record.+ result.add_error(+ "skillspector JSON report does not account for findings suppressed by the requested baseline; "+ "security scan did not complete"+ )+ return Falsenormalized_suppressed_count = suppressed_count or 0normalized_suppressed = suppressed or []if normalized_suppressed_count != len(normalized_suppressed):@@ -2261,7 +2608,7 @@ class SecurityValidator(ValidatorBase):"security scan did not complete")return False- if normalized_suppressed_count:+ if normalized_suppressed_count and not baseline_requested:result.add_error("skillspector reported unexpected suppressed findings without a requested baseline; ""security scan did not complete"@@ -2272,7 +2619,11 @@ class SecurityValidator(ValidatorBase):normalized_components,uses_report_identities=uses_completeness_schema,uses_finding_identity=uses_finding_identity,- findings_after_filtering=findings_after_filtering,+ # Baseline-suppressed findings do not score; counting them as+ # unseen occurrences would loosen this reconciliation.+ findings_after_filtering=(+ findings_after_filtering - normalized_suppressed_count if findings_after_filtering is not None else None+ ),all_findings_serialized=(findings_after_filtering is Noneor findings_after_filtering == len(issues) + normalized_suppressed_count@@ -2683,15 +3034,17 @@ class SecurityValidator(ValidatorBase):if isinstance(analysis_completeness, dict)else None)- suppressed = data.get("suppressed")- serialized_findings = len(issues) + (len(suppressed) if isinstance(suppressed, list) else 0)+ suppressed_count = len(data["suppressed"]) if isinstance(data.get("suppressed"), list) else 0+ serialized_findings = len(issues) + suppressed_counteffective_score = (self._minimum_skillspector_risk_score(scanned_issues,report_components,uses_report_identities=uses_report_identities,uses_finding_identity=uses_finding_identity,- findings_after_filtering=findings_after_filtering,+ findings_after_filtering=(+ findings_after_filtering - suppressed_count if isinstance(findings_after_filtering, int) else None+ ),reported_score=reported_score,removed_issues=removed_issues,all_findings_serialized=findings_after_filtering == serialized_findings,@@ -2874,6 +3227,7 @@ class SecurityValidator(ValidatorBase):"skillspector_components_count": len(components),"skillspector_has_executable_scripts": sp_metadata.get("has_executable_scripts", False),"skillspector_suppressed_count": suppressed_count,+ "skillspector_suppressed_summary": SecurityValidator._summarize_suppressed(data.get("suppressed")),})@@ -2888,6 +3242,122 @@ class SecurityValidator(ValidatorBase):if score > 50:result.add_message(f"skillspector risk score: {score}/100 ({severity})")+ @staticmethod+ def _summarize_suppressed(suppressed: object) -> dict:+ """Return the severity counts and rule IDs of the findings a baseline suppressed."""+ by_severity: dict[str, int] = {}+ rule_ids: set[str] = set()+ for item in suppressed if isinstance(suppressed, list) else []:+ if not isinstance(item, dict):+ continue+ severity = str(item.get("severity") or "UNKNOWN").upper()+ by_severity[severity] = by_severity.get(severity, 0) + 1+ rule_id = item.get("id")+ if isinstance(rule_id, str) and rule_id:+ rule_ids.add(rule_id[:64])+ return {"by_severity": by_severity, "rule_ids": sorted(rule_ids)[:_SKILLSPECTOR_SUPPRESSED_RULE_ID_LIMIT]}++ @staticmethod+ def _report_skillspector_baseline(+ result: ValidationResult,+ baseline: dict,+ *,+ static_suppressed: object = None,+ llm_suppressed: object = None,+ static_summary: object = None,+ ) -> None:+ """Record whether the requested baseline applied, so an inert one is not silent.++ The diagnostic is added after SkillSpector returns, outside its finding+ stream, so the baseline cannot suppress it. ``static_suppressed`` and+ ``llm_suppressed`` are the counts each stage reported; the LLM count+ is ``None`` when that stage did not run or did not complete.++ Every processed outcome is also a ``skillspector_baseline`` success+ detail, so Markdown and HTML reports show that reviewed exceptions+ were applied, not only the JSON report.+ """+ runtime_version = result.metadata.get("skillspector_version")+ if not isinstance(runtime_version, str):+ # No report was processed; the scan is already incomplete.+ baseline["message"] = (+ "SkillSpector did not return a usable report, so the baseline outcome is unknown; "+ "see the security check errors."+ )+ return+ suppressed = static_suppressed if isinstance(static_suppressed, int) else 0+ summary = static_summary if isinstance(static_summary, dict) else {}+ by_severity = summary.get("by_severity") if isinstance(summary.get("by_severity"), dict) else {}+ baseline["suppressed_by_severity"] = by_severity+ baseline["suppressed_rule_ids"] = summary.get("rule_ids") if isinstance(summary.get("rule_ids"), list) else []+ llm_stage_suppressed = llm_suppressed if isinstance(llm_suppressed, int) else None+ blocking = sum(+ 1+ for finding in result.findings+ if (finding.severity if isinstance(finding.severity, Severity) else Severity(str(finding.severity).lower()))+ in (Severity.CRITICAL, Severity.HIGH)+ )+ baseline.update(+ {+ "loaded": True,+ "runtime_scanner_version": runtime_version,+ "suppressed_findings": suppressed,+ "llm_stage_suppressed_findings": llm_stage_suppressed,+ "active_findings": len(result.findings),+ "active_blocking_findings": blocking,+ }+ )+ counts = (+ f"rules={baseline['rule_count']}, fingerprints={baseline['fingerprint_count']}, suppressed={suppressed}"+ )+ if llm_stage_suppressed is not None:+ counts += f", llm_stage_suppressed={llm_stage_suppressed}"+ if by_severity:+ breakdown = ", ".join(f"{name}={count}" for name, count in sorted(by_severity.items()))+ counts += f", suppressed_by_severity: {breakdown}"+ file_label = f"{baseline['file']} ({baseline['digest']})"++ def report(status: str, message: str, *, problem: bool) -> None:+ baseline["status"] = status+ baseline["message"] = message+ # Recorded for every outcome, so reports that hide errors still show it.+ result.add_success(check_name="skillspector_baseline", message=f"{file_label}: {message}")+ if problem and blocking:+ # Explain remaining blockers without turning a passing scan into a failure.+ result.add_error(message)+ elif problem:+ result.add_warning(message)++ # SkillSpector compares scanner versions as exact strings and ignores+ # every fingerprint on a mismatch; rules still apply.+ if baseline["fingerprint_count"] and baseline["baseline_scanner_version"] != runtime_version:+ report(+ "fingerprint_version_mismatch",+ "SkillSpector baseline fingerprints were ignored: they were generated with SkillSpector "+ f"{baseline['baseline_scanner_version'] or 'unknown'}, but this scan used {runtime_version} "+ f"({counts}). Regenerate the fingerprints with SkillSpector {runtime_version}, or replace them "+ "with narrow reviewed rules.",+ problem=True,+ )+ return+ if llm_stage_suppressed is not None and llm_stage_suppressed < suppressed:+ # SkillSpector's LLM meta-analysis rewrites findings before the+ # baseline is applied, so fingerprints of static findings stop+ # matching and those findings come back from the LLM stage.+ report(+ "llm_stage_unmatched",+ f"SkillSpector baseline suppressed {suppressed} finding(s) in the static stage but only "+ f"{llm_stage_suppressed} in the LLM stage ({counts}). LLM analysis rewrites findings before the "+ "baseline applies, so fingerprints do not match them; use narrow id, path, or message rules for "+ "runs with LLM analysis, or scan without --llm.",+ problem=True,+ )+ return+ if suppressed or llm_stage_suppressed:+ report("applied", f"SkillSpector baseline applied ({counts}).", problem=False)+ return+ report("no_matches", f"SkillSpector baseline loaded but matched no findings ({counts}).", problem=False)+@staticmethoddef _convert_skillspector_issue(issue: dict) -> tuple[Finding, bool]:"""Convert a single skillspector issue dict into a Finding.@@ -2952,7 +3422,15 @@ class SecurityValidator(ValidatorBase):@staticmethoddef _summarize_skillspector_results(issues: list, has_critical_or_high: bool, result: ValidationResult) -> None:"""Add a summary success/message entry for the skillspector scan."""- if not issues:+ suppressed = result.metadata.get("skillspector_suppressed_count")+ if not issues and isinstance(suppressed, int) and suppressed > 0:+ # Suppressions require an explicit baseline; never present them as a clean scan.+ result.add_success(+ check_name="skillspector",+ message=f"No active security findings; {suppressed} finding(s) suppressed by the requested baseline",+ suppressed_count=suppressed,+ )+ elif not issues:result.add_success(check_name="skillspector",message="No security vulnerabilities detected (secrets, API keys, credentials)",diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.jsonindex a8870cb7..d3dd377a 100644--- a/tests/golden/cli_surface.json+++ b/tests/golden/cli_surface.json@@ -1356,6 +1356,14 @@"param_type": "option","type": "boolean"},+ {+ "name": "skillspector_baseline",+ "opts": [+ "--skillspector-baseline"+ ],+ "param_type": "option",+ "type": "file"+ },{"choices": ["cli",@@ -1748,6 +1756,14 @@"param_type": "option","type": "boolean"},+ {+ "name": "skillspector_baseline",+ "opts": [+ "--skillspector-baseline"+ ],+ "param_type": "option",+ "type": "file"+ },{"choices": ["cli",@@ -1849,6 +1865,14 @@"param_type": "option","type": "text"},+ {+ "name": "skillspector_baseline",+ "opts": [+ "--skillspector-baseline"+ ],+ "param_type": "option",+ "type": "file"+ },{"choices": ["cli",@@ -1952,6 +1976,14 @@"param_type": "option","type": "text"},+ {+ "name": "skillspector_baseline",+ "opts": [+ "--skillspector-baseline"+ ],+ "param_type": "option",+ "type": "file"+ },{"choices": ["cli",@@ -3587,6 +3619,14 @@"param_type": "option","type": "boolean"},+ {+ "name": "skillspector_baseline",+ "opts": [+ "--skillspector-baseline"+ ],+ "param_type": "option",+ "type": "file"+ },{"default": "70","name": "min_score",diff --git a/tests/test_ci_workflows.py b/tests/test_ci_workflows.pyindex 3dfb8169..6fee9ad0 100644--- a/tests/test_ci_workflows.py+++ b/tests/test_ci_workflows.py@@ -261,6 +261,10 @@ def test_gitleaks_single_value_allowlists_are_exactly_scoped() -> None:r"^us\.anthropic\.claude-opus-5$",r"^tests/validators/test_security\.py$",),+ "Bedrock model ID in the merge-leftover SkillSpector bridge test copy from commit e8aa35d6a": (+ r"^us\.anthropic\.claude-opus-5$",+ r"^tests/validators/test_security\.py\.orig$",+ ),}for description, (regex, path) in expected.items():entry = allowlists[description]diff --git a/tests/test_commands.py b/tests/test_commands.pyindex cc78160d..579f82b9 100644--- a/tests/test_commands.py+++ b/tests/test_commands.py@@ -61,6 +61,112 @@ def test_validate_passes_explicit_previous_version(monkeypatch, tmp_path: Path)assert captured == ["1.2.0"]+ "command",+ [+ ["validate", "--no-tier3", "--no-dedup", "--no-llm", "--checks", "security"],+ ["security-scan"],+ ["tier1", "security-scan"],+ ["tier1", "--checks", "security", "--no-llm"],+ ["tier1", "validate", "--checks", "security", "--no-llm"],+ ],+ ids=["validate", "security-scan", "tier1-security-scan", "tier1", "tier1-validate"],+)+def test_explicit_skillspector_baseline_reaches_the_security_scan(+ command: list[str], monkeypatch: pytest.MonkeyPatch, tmp_path: Path+) -> None:+ from skillevaluator.models.result import ValidationResult+ from skillevaluator.validators.security import SecurityValidator++ baseline = tmp_path / "approved-baseline.yaml"+ baseline.write_text('version: 2\nrules:\n - id: "PE5"\n reason: "Reviewed"\n', encoding="utf-8")+ seen: list[Path | None] = []++ def _run_skillspector(self: SecurityValidator, _path: Path) -> ValidationResult:+ seen.append(self.skillspector_baseline)+ return ValidationResult()++ monkeypatch.setattr(SecurityValidator, "_run_skillspector", _run_skillspector)+ monkeypatch.chdir(tmp_path)++ result = CliRunner().invoke(+ cli,+ [+ *command,+ str(FIXTURE),+ "--skillspector-baseline",+ baseline.name,+ "-r",+ "json",+ "-o",+ str(tmp_path / "reports"),+ ],+ )++ assert result.exit_code == 0, result.output+ assert seen == [baseline.resolve()]+++ "command",+ [+ ["validate", "--no-tier3", "--no-dedup", "--no-llm"],+ ["tier1", "--no-llm"],+ ],+ ids=["validate", "tier1"],+)+def test_skillspector_baseline_and_skip_both_reach_the_run(+ command: list[str], monkeypatch: pytest.MonkeyPatch, tmp_path: Path+) -> None:+ """``--skip`` on another check must not drop the baseline, and vice versa."""+ from skillevaluator.models.result import ValidationResult+ from skillevaluator.validators.security import SecurityValidator++ baseline = tmp_path / "approved-baseline.yaml"+ baseline.write_text('version: 2\nrules:\n - id: "PE5"\n reason: "Reviewed"\n', encoding="utf-8")+ seen: list[Path | None] = []++ def _run_skillspector(self: SecurityValidator, _path: Path) -> ValidationResult:+ seen.append(self.skillspector_baseline)+ return ValidationResult()++ monkeypatch.setattr(SecurityValidator, "_run_skillspector", _run_skillspector)+ reports = tmp_path / "reports"++ result = CliRunner().invoke(+ cli,+ [+ *command,+ str(FIXTURE),+ "--checks",+ "security,unicode",+ "--skip",+ "unicode",+ "--skillspector-baseline",+ str(baseline),+ "-r",+ "json",+ "-o",+ str(reports),+ ],+ )++ assert result.exit_code == 0, result.output+ assert seen == [baseline.resolve()]+ (report,) = reports.rglob("*.json")+ by_name = {entry["validator"]: entry for entry in json.loads(report.read_text(encoding="utf-8"))["results"]}+ assert by_name["Unicode Smuggling Detection"]["skip_reason"] == "skipped with --skip"+++def test_skillspector_baseline_must_name_an_existing_file(tmp_path: Path) -> None:+ result = CliRunner().invoke(+ cli, ["security-scan", str(FIXTURE), "--skillspector-baseline", str(tmp_path / "missing.yaml")]+ )++ assert result.exit_code == 2+ assert "--skillspector-baseline" in result.output++def test_claude_alias_is_canonicalized_and_deduplicated() -> None:assert parse_agents("claude, claude-code, opencode, claude") == ["claude-code", "opencode"]@@ -1549,6 +1655,23 @@ def test_catalog_child_argv_carries_the_evaluated_source(tmp_path: Path) -> Noneassert "--workers" not in argv+def test_catalog_child_argv_forwards_the_skillspector_baseline(tmp_path: Path) -> None:+ from skillevaluator.cli import _catalog_child_argv_from_ctx, validate++ skill_dir = tmp_path / "catalog" / "simple"+ skill_dir.mkdir(parents=True)+ baseline = tmp_path / "approved-baseline.yaml"+ baseline.write_text('version: 2\nrules:\n - id: "PE5"\n reason: "Reviewed"\n', encoding="utf-8")+ ctx = validate.make_context(+ "validate",+ [str(skill_dir), "--workers", "2", "--skillspector-baseline", str(baseline)],+ )++ argv = _catalog_child_argv_from_ctx(ctx, skill_dir, tmp_path / "out" / "simple")++ assert argv[argv.index("--skillspector-baseline") + 1] == str(baseline.resolve())++def test_catalog_child_argv_forwards_every_validate_option() -> None:"""The allowlist is explicit, so a new option is silently dropped by default.diff --git a/tests/validators/test_artifact_dir_exclusion.py b/tests/validators/test_artifact_dir_exclusion.pyindex 304810cf..ac0a659c 100644--- a/tests/validators/test_artifact_dir_exclusion.py+++ b/tests/validators/test_artifact_dir_exclusion.py@@ -12,6 +12,7 @@ Regression source: a real skill dir with 798MB / 53k artifact files took~25 minutes to validate and reported 270 artifact-only secret findings."""+import hashlibimport jsonimport osimport re@@ -103,23 +104,25 @@ class TestIterScannableFilesPrunes:# =============================================================================-def _clean_spector_result():- return ToolResult(- success=True,- stdout=json.dumps(- {- "risk_assessment": {"score": 0, "severity": "LOW", "recommendation": "SAFE"},- "issues": [],- "metadata": {- "skillspector_version": "1.0.0",- "llm_requested": False,- "llm_available": False,- },- }- ),- stderr="",- exit_code=0,- )+_REVIEWED_BASELINE = (+ 'version: 2\nrules:\n - id: "PE5"\n path: "SKILL.md"\n message: "*--privilege[d]*"\n'+ ' reason: "Reviewed diagnostics container"\n'+)+++def _clean_spector_result(*, suppression_accounting: bool = False):+ report = {+ "risk_assessment": {"score": 0, "severity": "LOW", "recommendation": "SAFE"},+ "issues": [],+ "metadata": {+ "skillspector_version": "1.0.0",+ "llm_requested": False,+ "llm_available": False,+ },+ }+ if suppression_accounting:+ report.update({"suppressed_count": 0, "suppressed": []})+ return ToolResult(success=True, stdout=json.dumps(report), stderr="", exit_code=0)class TestSkillspectorFilteredCopy:@@ -266,6 +269,76 @@ class TestSkillspectorFilteredCopy:assert len(result.findings) == 1assert result.findings[0].file_path == str(skill_with_artifacts.resolve() / "payload.py")+ @pytest.mark.parametrize(+ ("relative", "passes_staged_copy"),+ [+ ("config/skillspector-baseline.yaml", True),+ # Excluded from the staged copy, so it is outside the scan either way.+ ("evals/skillspector-baseline.yaml", False),+ ],+ )+ @patch.object(Tools.skillspector, "_path", "/usr/bin/skillspector")+ @patch.object(Tools.skillspector, "run")+ def test_in_tree_baseline_is_excluded_from_the_staged_scan(+ self, mock_run, skill_with_artifacts, relative, passes_staged_copy+ ):+ """SkillSpector excludes only the --baseline file; a stale copy would be scanned as content."""+ baseline = skill_with_artifacts / relative+ baseline.parent.mkdir(exist_ok=True)+ baseline.write_text(_REVIEWED_BASELINE, encoding="utf-8")+ seen: dict = {}++ def capture(args, **kwargs):+ seen["scanned"] = Path(args[args.index("scan") + 1])+ seen["baseline"] = Path(args[args.index("--baseline") + 1])+ seen["baseline_is_file"] = seen["baseline"].is_file()+ return _clean_spector_result(suppression_accounting=True)++ mock_run.side_effect = capture+ result = SecurityValidator(skillspector_baseline=baseline)._run_skillspector(skill_with_artifacts)++ assert result.status == "passed", result.errors+ assert seen["scanned"] != skill_with_artifacts.resolve()+ expected = seen["scanned"] / relative if passes_staged_copy else baseline.resolve()+ assert seen["baseline"] == expected+ assert seen["baseline_is_file"] is True+ # Reports identify the file the operator supplied without exposing a local path.+ state = result.metadata["skillspector_baseline"]+ assert state["file"] == "skillspector-baseline.yaml"+ assert state["digest"] == f"sha256:{hashlib.sha256(baseline.read_bytes()).hexdigest()}"+ assert "path" not in state+ assert str(skill_with_artifacts) not in json.dumps(state)++ @patch.object(Tools.skillspector, "_path", "/usr/bin/skillspector")+ @patch.object(Tools.skillspector, "run")+ def test_external_baseline_keeps_its_path_for_a_staged_scan(self, mock_run, skill_with_artifacts, tmp_path):+ baseline = tmp_path / "approved" / "skillspector-baseline.yaml"+ baseline.parent.mkdir()+ baseline.write_text(_REVIEWED_BASELINE, encoding="utf-8")+ mock_run.return_value = _clean_spector_result(suppression_accounting=True)++ SecurityValidator(skillspector_baseline=baseline)._run_skillspector(skill_with_artifacts)++ args = mock_run.call_args.args[0]+ assert args[args.index("scan") + 1] != str(skill_with_artifacts.resolve())+ assert args[args.index("--baseline") + 1] == str(baseline.resolve())++ @patch.object(Tools.skillspector, "_path", "/usr/bin/skillspector")+ @patch.object(Tools.skillspector, "run")+ def test_in_tree_baseline_keeps_its_path_when_scanned_in_place(self, mock_run, tmp_path):+ skill = tmp_path / "clean-skill"+ (skill / "config").mkdir(parents=True)+ (skill / "SKILL.md").write_text("---\nname: clean-skill\n---\n")+ baseline = skill / "config" / "skillspector-baseline.yaml"+ baseline.write_text(_REVIEWED_BASELINE, encoding="utf-8")+ mock_run.return_value = _clean_spector_result(suppression_accounting=True)++ SecurityValidator(skillspector_baseline=baseline)._run_skillspector(skill)++ args = mock_run.call_args.args[0]+ assert args[args.index("scan") + 1] == str(skill.resolve())+ assert args[args.index("--baseline") + 1] == str(baseline.resolve())+# =============================================================================# GITLEAKS CONFIG EXCLUDES ARTIFACT DIRSdiff --git a/tests/validators/test_security.py b/tests/validators/test_security.pyindex 63b5d151..8032e0be 100644--- a/tests/validators/test_security.py+++ b/tests/validators/test_security.py@@ -3,6 +3,7 @@"""Tests for security validator."""+import hashlibimport jsonimport osimport re@@ -194,12 +195,20 @@ def _skillspector_json_report(}+def _merge_amplified_yaml(levels: int) -> str:+ """Return YAML whose chained merge keys double the expanded size on every line."""+ lines = ["a0: &a0 {k: 1}"]+ lines += [f"a{n}: &a{n} {{<<: [*a{n - 1}, *a{n - 1}], k{n}: 1}}" for n in range(1, levels + 1)]+ return "\n".join(lines)++def _validate_skillspector_payload(mock_tools,sample_skill_dir: Path,payload: dict,*,exit_code: int = 0,+ validator: SecurityValidator | None = None,) -> ValidationResult:"""Run one deterministic SkillSpector payload through the public validation seam."""mock_tools.skillspector.is_available = True@@ -209,7 +218,7 @@ def _validate_skillspector_payload(stderr="",exit_code=exit_code,)- return SecurityValidator(use_llm=False).validate_security_only(sample_skill_dir)+ return (validator or SecurityValidator(use_llm=False)).validate_security_only(sample_skill_dir)def _passwd_mount_issue(**overrides) -> dict:@@ -1673,6 +1682,521 @@ print(json.dumps([finding.metadata["matched_value"] for finding in result.findinassert result.status == "incomplete"assert any("unexpected suppressed findings" in error.lower() for error in result.errors)+ @patch("skillevaluator.validators.security.Tools")+ def test_requested_skillspector_baseline_accepts_audited_suppressions(+ self, mock_tools, sample_skill_dir: Path, tmp_path: Path+ ) -> None:+ baseline = tmp_path / "approved" / "skillspector-baseline.yaml"+ baseline.parent.mkdir()+ baseline.write_text(+ 'version: 2\nrules:\n - id: "PE5"\n path: "SKILL.md"\n reason: "Reviewed"\n',+ encoding="utf-8",+ )+ payload = _skillspector_json_report()+ payload["suppressed_count"] = 1+ payload["suppressed"] = [+ {"id": "PE5", "severity": "HIGH", "suppressed": True, "suppression_reason": "Reviewed"}+ ]+ payload["analysis_completeness"].update({"findings_before_filtering": 1, "findings_after_filtering": 1})++ result = _validate_skillspector_payload(+ mock_tools,+ sample_skill_dir,+ payload,+ validator=SecurityValidator(use_llm=False, skillspector_baseline=baseline),+ )++ args = mock_tools.skillspector.run.call_args.args[0]+ assert args[args.index("--baseline") + 1] == str(baseline.resolve())+ assert result.status == "passed"+ state = result.metadata["skillspector_baseline"]+ assert state["loaded"] is True+ assert state["status"] == "applied"+ assert (state["rule_count"], state["fingerprint_count"], state["suppressed_findings"]) == (1, 0, 1)+ # CI can refuse a baseline that hides blocking findings.+ assert state["suppressed_by_severity"] == {"HIGH": 1}+ assert state["suppressed_rule_ids"] == ["PE5"]+ rendered = JSONReporter(include_timestamp=False).render_all([result])+ reported = json.loads(rendered)["results"][0]["skillspector_baseline"]+ assert reported["status"] == "applied"+ # Like quality_config provenance, the report names the file and its digest, never a local path.+ assert reported["file"] == "skillspector-baseline.yaml"+ assert reported["digest"] == f"sha256:{hashlib.sha256(baseline.read_bytes()).hexdigest()}"+ assert str(baseline.parent) not in rendered+ # Markdown and HTML reports disclose the suppression too; a clean-looking check would hide it.+ markdown = MarkdownReporter(include_timestamp=False).render_all([result])+ html = HTMLReporter(include_timestamp=False).render_all([result])+ for text in (markdown, html):+ assert "skillspector_baseline" in text+ assert "skillspector-baseline.yaml" in text and "SkillSpector baseline applied" in text+ assert "suppressed by the requested baseline" in text+ assert "No security vulnerabilities detected" not in text+ assert str(baseline.parent) not in text++ @patch("skillevaluator.validators.security.Tools")+ def test_skillspector_baseline_suppressions_do_not_loosen_risk_reconciliation(+ self, mock_tools, sample_skill_dir: Path, tmp_path: Path+ ) -> None:+ """Suppressed rows are not unseen occurrences of the active finding."""+ baseline = tmp_path / "baseline.yaml"+ baseline.write_text('version: 2\nrules:\n - id: "PE5"\n reason: "Reviewed"\n', encoding="utf-8")+ issue = {+ "id": "PI-1",+ "category": "Prompt Injection",+ "pattern": "Instruction override",+ "severity": "HIGH",+ "finding": "Ignore prior instructions",+ "location": {"file": "SKILL.md", "start_line": 8},+ }+ payload = _skillspector_json_report([issue])+ payload["risk_assessment"] = {"score": 0, "severity": "LOW", "recommendation": "SAFE"}+ payload["suppressed_count"] = 3+ payload["suppressed"] = [{"id": "PE5", "suppressed": True} for _ in range(3)]+ payload["analysis_completeness"].update({"findings_before_filtering": 4, "findings_after_filtering": 4})++ result = _validate_skillspector_payload(+ mock_tools,+ sample_skill_dir,+ payload,+ validator=SecurityValidator(use_llm=False, skillspector_baseline=baseline),+ )++ assert result.status == "incomplete"+ assert any("risk score understates" in error for error in result.errors)++ @patch("skillevaluator.validators.security.Tools")+ def test_skillspector_baseline_suppressions_do_not_lower_filtered_risk_floor(+ self, mock_tools, sample_skill_dir: Path, tmp_path: Path+ ) -> None:+ """Suppressed rows must not weaken the score floor kept after generated-artifact issues are removed."""+ baseline = tmp_path / "baseline.yaml"+ baseline.write_text('version: 2\nrules:\n - id: "PE5"\n reason: "Reviewed"\n', encoding="utf-8")+ issues = [+ {+ "id": f"M{index}",+ "severity": "MEDIUM",+ "finding": "retained advisory",+ "location": {"file": "SKILL.md", "start_line": index + 1},+ }+ for index in range(6)+ ]+ issues.append(+ {+ "id": "G1",+ "severity": "LOW",+ "finding": "generated evaluation output",+ "location": {"file": "evals/run-1/output.md", "start_line": 1},+ }+ )+ payload = _skillspector_json_report(issues)+ payload["risk_assessment"] = {"score": 60, "severity": "HIGH", "recommendation": "DO_NOT_INSTALL"}+ payload["suppressed_count"] = 3+ payload["suppressed"] = [{"id": "PE5", "suppressed": True} for _ in range(3)]+ # One finding SkillSpector did not serialize, plus the three suppressed rows.+ payload["analysis_completeness"].update({"findings_before_filtering": 11, "findings_after_filtering": 11})++ result = _validate_skillspector_payload(+ mock_tools,+ sample_skill_dir,+ payload,+ exit_code=1,+ validator=SecurityValidator(use_llm=False, skillspector_baseline=baseline),+ )++ assert any("generated artifact" in message for message in result.messages)+ assert result.metadata["skillspector_baseline"]["status"] == "applied"+ assert result.status == "failed"+ assert any(finding.check_name == "skillspector_risk_score" for finding in result.findings)++ @patch("skillevaluator.validators.security.Tools")+ def test_requested_skillspector_baseline_requires_suppression_accounting(+ self, mock_tools, sample_skill_dir: Path, tmp_path: Path+ ) -> None:+ baseline = tmp_path / "baseline.yaml"+ baseline.write_text('version: 2\nrules:\n - id: "PE5"\n reason: "Reviewed"\n', encoding="utf-8")+ payload = _skillspector_json_report()+ del payload["suppressed_count"], payload["suppressed"]++ result = _validate_skillspector_payload(+ mock_tools,+ sample_skill_dir,+ payload,+ validator=SecurityValidator(use_llm=False, skillspector_baseline=baseline),+ )++ assert result.status == "incomplete"+ assert any("does not account for findings suppressed" in error for error in result.errors)++ @pytest.mark.parametrize(("blocking", "status"), [(True, "failed"), (False, "passed")])+ @patch("skillevaluator.validators.security.Tools")+ def test_skillspector_fingerprint_version_mismatch_is_reported(+ self, mock_tools, blocking: bool, status: str, sample_skill_dir: Path, tmp_path: Path+ ) -> None:+ baseline = tmp_path / "baseline.yaml"+ baseline.write_text(+ 'version: 2\nscanner_version: "2.9.0"\nrules: []\nfingerprints:\n'+ f' - hash: "sha256:{"0" * 64}"\n reason: "Reviewed"\n',+ encoding="utf-8",+ )+ issue = {+ "id": "PI-1",+ "category": "Prompt Injection",+ "pattern": "Instruction override",+ "severity": "HIGH",+ "finding": "Ignore prior instructions",+ "location": {"file": "SKILL.md", "start_line": 8},+ }+ payload = _skillspector_json_report([issue] if blocking else [])++ result = _validate_skillspector_payload(+ mock_tools,+ sample_skill_dir,+ payload,+ exit_code=1 if blocking else 0,+ validator=SecurityValidator(use_llm=False, skillspector_baseline=baseline),+ )++ state = result.metadata["skillspector_baseline"]+ assert state["status"] == "fingerprint_version_mismatch"+ assert (state["baseline_scanner_version"], state["runtime_scanner_version"]) == ("2.9.0", "2.10.0")+ assert state["active_blocking_findings"] == int(blocking)+ assert result.status == status+ # The explanation sits outside SkillSpector's findings, so no baseline entry can hide it.+ reported = result.errors if blocking else result.warnings+ assert any("fingerprints were ignored" in line and "SkillSpector 2.10.0" in line for line in reported)+ assert not any("fingerprints were ignored" in finding.message for finding in result.findings)++ @patch("skillevaluator.validators.security.Tools")+ def test_rule_only_skillspector_baseline_ignores_scanner_version(+ self, mock_tools, sample_skill_dir: Path, tmp_path: Path+ ) -> None:+ baseline = tmp_path / "baseline.yaml"+ baseline.write_text(+ 'version: 2\nscanner_version: "2.9.0"\nrules:\n - id: "PE5"\n reason: "Reviewed"\n',+ encoding="utf-8",+ )++ result = _validate_skillspector_payload(+ mock_tools,+ sample_skill_dir,+ _skillspector_json_report(),+ validator=SecurityValidator(use_llm=False, skillspector_baseline=baseline),+ )++ assert result.status == "passed"+ assert result.metadata["skillspector_baseline"]["status"] == "no_matches"+ assert any("matched no findings" in message for message in result.messages)+ assert not result.warnings+ assert "matched no findings" in MarkdownReporter(include_timestamp=False).render_all([result])++ @pytest.mark.parametrize(+ "content",+ [+ 'version: 2\nrules:\n - id: "*"\n reason: "Too broad"\n',+ 'version: 2\nrules:\n - id: "PE5"\n reason: "Fine"\n - path: "*/*"\n message: "*?*"\n'+ ' reason: "Too broad"\n',+ 'version: 2\nrules:\n - message: "[!a]*"\n reason: "Too broad"\n',+ # A single common letter or digit still matches nearly every finding.+ 'version: 2\nrules:\n - message: "*e*"\n reason: "Too broad"\n',+ 'version: 2\nrules:\n - path: "*s*"\n reason: "Too broad"\n',+ 'version: 2\nrules:\n - id: "*P*"\n reason: "Too broad"\n',+ 'version: 2\nrules:\n - message: "*a*b*c*d*"\n reason: "Too broad"\n',+ "version: 2\nrules:\n - just-a-string\n",+ "- not\n- a mapping\n",+ "version: [2\n",+ # Schema errors SkillSpector reports only as exit code 2.+ 'version: 2\nrules:\n - id: "PE5"\n',+ 'version: 2\nrules:\n - id: "PE5"\n reason: " "\n',+ f'version: 2\nscanner_version: "2.12.0"\nfingerprints:\n - hash: "sha256:{"0" * 64}"\n',+ 'version: 2\nrules:\n PE5:\n reason: "Not a list"\n',+ 'version: 2\nfingerprints: "sha256:abc"\n',+ # SafeLoader raises a plain ValueError for an impossible timestamp.+ 'version: 2\nrules:\n - id: "PE5"\n reason: "Reviewed"\n reviewed_on: 2026-06-31\n',+ 'version: !!int ""\n',+ # Chained merge keys double the expanded size on every line.+ _merge_amplified_yaml(20) + '\nversion: 2\nrules:\n - id: "PE5"\n reason: "Reviewed"\n',+ "a: &a [*a]\nversion: 2\n",+ ],+ )+ @patch("skillevaluator.validators.security.Tools")+ def test_unusable_skillspector_baseline_blocks_the_scan(+ self, mock_tools, content: str, sample_skill_dir: Path, tmp_path: Path+ ) -> None:+ baseline = tmp_path / "baseline.yaml"+ baseline.write_text(content, encoding="utf-8")+ mock_tools.skillspector.is_available = True++ result = SecurityValidator(use_llm=False, skillspector_baseline=baseline).validate_security_only(+ sample_skill_dir+ )++ assert result.status == "incomplete"+ assert any("security scan did not run" in error for error in result.errors)+ assert not any(str(tmp_path) in error for error in result.errors)+ state = result.metadata["skillspector_baseline"]+ assert state["status"] == "not_applied"+ assert state["message"] and "baseline.yaml" in state["message"]+ mock_tools.skillspector.run.assert_not_called()++ @patch("skillevaluator.validators.security.Tools")+ def test_skillspector_baseline_rejection_exit_points_at_the_baseline(+ self, mock_tools, sample_skill_dir: Path, tmp_path: Path+ ) -> None:+ """SkillSpector rejects some baselines only at load time, with exit code 2 and redacted diagnostics."""+ baseline = tmp_path / "baseline.yaml"+ baseline.write_text('version: 2\nrules:\n - id: "PE5"\n reason: "Reviewed"\n', encoding="utf-8")+ mock_tools.skillspector.is_available = True+ mock_tools.skillspector.run.return_value = ToolResult(success=True, stdout="", stderr="", exit_code=2)++ result = SecurityValidator(use_llm=False, skillspector_baseline=baseline).validate_security_only(+ sample_skill_dir+ )++ assert result.status == "incomplete"+ assert any("exit code 2" in error and "--baseline" in error for error in result.errors)+ state = result.metadata["skillspector_baseline"]+ assert state["status"] == "not_applied"+ assert state["message"] and "outcome is unknown" in state["message"]++ @patch("skillevaluator.validators.security.Tools")+ def test_skillspector_exit_2_without_a_baseline_has_no_baseline_hint(+ self, mock_tools, sample_skill_dir: Path+ ) -> None:+ mock_tools.skillspector.is_available = True+ mock_tools.skillspector.run.return_value = ToolResult(success=True, stdout="", stderr="", exit_code=2)++ result = SecurityValidator(use_llm=False).validate_security_only(sample_skill_dir)++ assert result.status == "incomplete"+ assert any("exit code 2" in error for error in result.errors)+ assert not any("--baseline" in error for error in result.errors)+ assert "skillspector_baseline" not in result.metadata++ @patch("skillevaluator.validators.security.Tools")+ def test_missing_skillspector_baseline_blocks_the_scan(+ self, mock_tools, sample_skill_dir: Path, tmp_path: Path+ ) -> None:+ mock_tools.skillspector.is_available = True++ result = SecurityValidator(+ use_llm=False, skillspector_baseline=tmp_path / "missing.yaml"+ ).validate_security_only(sample_skill_dir)++ assert result.status == "incomplete"+ assert any("missing.yaml does not exist" in error for error in result.errors)+ assert not any(str(tmp_path) in error for error in result.errors)+ mock_tools.skillspector.run.assert_not_called()++ @pytest.mark.parametrize(+ "rule",+ [+ {"id": "PE5"},+ {"rule_id": "PE5"},+ {"file": "SKILL.md"},+ {"path": "*SKILL.md"},+ {"message": "*--privilege[d]*"},+ {"id": "*", "message": "*[t]elemetry*"},+ ],+ )+ def test_bounded_skillspector_baseline_rules_are_accepted(self, rule: dict, tmp_path: Path) -> None:+ from skillevaluator.validators.security import _inspect_skillspector_baseline++ baseline = tmp_path / "baseline.yaml"+ baseline.write_text(json.dumps({"version": 2, "rules": [{**rule, "reason": "Reviewed"}]}), encoding="utf-8")++ state, error = _inspect_skillspector_baseline(baseline)++ assert error is None+ assert state["rule_count"] == 1++ @pytest.mark.parametrize(+ ("content", "expected"),+ [+ ("version: 2\nrules:\n - PE5\n", "rule 0 in baseline.yaml must be a mapping"),+ (+ f'version: 2\nscanner_version: "2.13.3"\nfingerprints:\n - "sha256:{"0" * 64}"\n',+ "fingerprint 0 in baseline.yaml must be a mapping with 'hash' and 'reason'",+ ),+ (+ 'version: 2\nrules:\n - id: "PE5"\n reason: "Reviewed"\n reviewed_on: 2026-06-31\n',+ "baseline.yaml could not be parsed (ValueError)",+ ),+ (+ _merge_amplified_yaml(20),+ "baseline.yaml exceeds the expanded YAML node limit",+ ),+ ],+ )+ def test_skillspector_baseline_rejections_name_the_actual_problem(+ self, content: str, expected: str, tmp_path: Path+ ) -> None:+ """A shape, date, or expansion error is reported as such, not as an unbounded rule or a crash."""+ from skillevaluator.validators.security import _inspect_skillspector_baseline++ baseline = tmp_path / "baseline.yaml"+ baseline.write_text(content, encoding="utf-8")++ _state, error = _inspect_skillspector_baseline(baseline)++ assert error is not None and error.endswith(expected), error++ def test_oversized_skillspector_baseline_is_rejected_before_parsing(self, tmp_path: Path) -> None:+ from skillevaluator.validators.security import (+ _SKILLSPECTOR_BASELINE_MAX_BYTES,+ _inspect_skillspector_baseline,+ )++ baseline = tmp_path / "baseline.yaml"+ baseline.write_bytes(b"#" * (_SKILLSPECTOR_BASELINE_MAX_BYTES + 1))++ state, error = _inspect_skillspector_baseline(baseline)++ assert error is not None and "byte limit" in error+ assert state["digest"] is None++ @pytest.mark.parametrize("content", ["", "# no exceptions yet\n"])+ @patch("skillevaluator.validators.security.Tools")+ def test_empty_skillspector_baseline_is_an_empty_baseline(+ self, mock_tools, content: str, sample_skill_dir: Path, tmp_path: Path+ ) -> None:+ """SkillSpector loads an empty or comment-only baseline as one with no exceptions."""+ baseline = tmp_path / "baseline.yaml"+ baseline.write_text(content, encoding="utf-8")++ result = _validate_skillspector_payload(+ mock_tools,+ sample_skill_dir,+ _skillspector_json_report(),+ validator=SecurityValidator(use_llm=False, skillspector_baseline=baseline),+ )++ assert result.status == "passed", result.errors+ state = result.metadata["skillspector_baseline"]+ assert (state["status"], state["rule_count"], state["fingerprint_count"]) == ("no_matches", 0, 0)++ @pytest.mark.parametrize("version", [None, 1])+ def test_legacy_rule_only_skillspector_baseline_needs_no_reason(self, version: int | None, tmp_path: Path) -> None:+ """SkillSpector requires a reason only in version 2 baselines."""+ from skillevaluator.validators.security import _inspect_skillspector_baseline++ content: dict = {"rules": [{"id": "PE5"}]}+ if version is not None:+ content["version"] = version+ baseline = tmp_path / "baseline.yaml"+ baseline.write_text(json.dumps(content), encoding="utf-8")++ state, error = _inspect_skillspector_baseline(baseline)++ assert error is None+ assert (state["schema_version"], state["rule_count"]) == (version, 1)++ @patch("skillevaluator.validators.security.Tools")+ def test_skillspector_baseline_reaches_llm_enrichment_stage(+ self, mock_tools, sample_skill_dir: Path, tmp_path: Path, monkeypatch+ ) -> None:+ baseline = tmp_path / "baseline.yaml"+ baseline.write_text('version: 2\nrules:\n - id: "PE5"\n reason: "Reviewed"\n', encoding="utf-8")+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "nv_build")+ monkeypatch.setenv("NVIDIA_API_KEY", "test-credential")+ static = _skillspector_json_report()+ enriched = _skillspector_json_report(llm_requested=True, llm_available=True)+ mock_tools.skillspector.is_available = True+ mock_tools.skillspector.run.side_effect = [+ ToolResult(success=True, stdout=json.dumps(report), stderr="", exit_code=0) for report in (static, enriched)+ ]++ result = SecurityValidator(use_llm=True, skillspector_baseline=baseline).validate_security_only(+ sample_skill_dir+ )++ assert result.status == "passed", result.errors+ stages = [call.args[0] for call in mock_tools.skillspector.run.call_args_list]+ assert ["--no-llm" in args for args in stages] == [True, False]+ assert all(args[args.index("--baseline") + 1] == str(baseline.resolve()) for args in stages)++ @pytest.mark.parametrize(("llm_suppressed", "status"), [(0, "llm_stage_unmatched"), (1, "applied")])+ @patch("skillevaluator.validators.security.Tools")+ def test_skillspector_baseline_outcome_accounts_for_each_stage(+ self,+ mock_tools,+ llm_suppressed: int,+ status: str,+ sample_skill_dir: Path,+ tmp_path: Path,+ monkeypatch,+ ) -> None:+ """LLM meta-analysis rewrites findings, so static fingerprints can stop matching in that stage."""+ baseline = tmp_path / "baseline.yaml"+ baseline.write_text(+ f'version: 2\nscanner_version: "2.10.0"\nfingerprints:\n - hash: "sha256:{"1" * 64}"\n'+ ' reason: "Reviewed"\n',+ encoding="utf-8",+ )+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "nv_build")+ monkeypatch.setenv("NVIDIA_API_KEY", "test-credential")+ static = _skillspector_json_report()+ static["suppressed_count"] = 1+ static["suppressed"] = [{"id": "PE5", "suppressed": True}]+ static["analysis_completeness"].update({"findings_before_filtering": 1, "findings_after_filtering": 1})+ privileged = {+ "id": "PE5",+ "category": "Privilege Escalation",+ "pattern": "Privileged container",+ "severity": "HIGH",+ "finding": "Container runs with full host privileges (LLM-confirmed)",+ "location": {"file": "SKILL.md", "start_line": 5},+ }+ enriched = _skillspector_json_report(+ [] if llm_suppressed else [privileged], llm_requested=True, llm_available=True+ )+ if llm_suppressed:+ enriched["suppressed_count"] = 1+ enriched["suppressed"] = [{"id": "PE5", "suppressed": True}]+ enriched["analysis_completeness"].update({"findings_before_filtering": 1, "findings_after_filtering": 1})+ mock_tools.skillspector.is_available = True+ mock_tools.skillspector.run.side_effect = [+ ToolResult(success=True, stdout=json.dumps(static), stderr="", exit_code=0),+ ToolResult(success=True, stdout=json.dumps(enriched), stderr="", exit_code=0 if llm_suppressed else 1),+ ]++ result = SecurityValidator(use_llm=True, skillspector_baseline=baseline).validate_security_only(+ sample_skill_dir+ )++ state = result.metadata["skillspector_baseline"]+ assert state["status"] == status+ # The static stage's count is not overwritten by the LLM stage's.+ assert (state["suppressed_findings"], state["llm_stage_suppressed_findings"]) == (1, llm_suppressed)+ if llm_suppressed:+ assert result.status == "passed", result.errors+ else:+ assert result.status == "failed"+ assert state["active_blocking_findings"] == 1+ assert any("only 0 in the LLM stage" in error and "--llm" in error for error in result.errors)+ # The outcome stays visible when the error list is rebuilt later.+ assert any(+ detail.check_name == "skillspector_baseline" and "only 0 in the LLM stage" in detail.message+ for detail in result.success_details+ )+ assert not any("matched no findings" in message for message in result.messages)++ def test_folder_scan_keeps_the_baseline_outcome_in_each_passing_skill_line(self, tmp_path: Path) -> None:+ folder = tmp_path / "skills"+ for name in ("a", "b"):+ (folder / name).mkdir(parents=True)+ (folder / name / "SKILL.md").write_text(f"---\nname: {name}\ndescription: d\n---\n", encoding="utf-8")++ def single(skill_dir: Path) -> ValidationResult:+ outcome = ValidationResult()+ outcome.add_success(check_name="skillspector_baseline", message=f"{skill_dir.name}: baseline applied")+ return outcome++ result = SecurityValidator(use_llm=False)._validate_folder_or_skill(folder, single, "Scanning")++ markdown = MarkdownReporter(include_timestamp=False).render_all([result])+ assert "a: baseline applied" in markdown+ assert "b: baseline applied" in markdown+@patch("skillevaluator.validators.security.Tools")def test_skillspector_2_10_partial_scan_reports_incomplete_before_recommendation(self,