Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(tier1): score tier 1 quality checks via a configurable YAML file (#213)
Weston Plattereb777154ea0d
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 1038f702..6f2edd4d 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -216,6 +216,7 @@ jobs:cd "$RUNNER_TEMP""$venv/bin/python" -c 'import anthropic, harbor, openai, skillevaluator, skillevaluator.model_catalog, skillevaluator.model_commands'"$venv/bin/python" -c 'from importlib.resources import files; assert files("skillevaluator.tier3.harbor").joinpath("templates/eval.py").is_file()'+ "$venv/bin/python" -c 'from importlib.resources import files; from skillevaluator.quality_config import load_quality_config; assert files("skillevaluator.config").joinpath("quality-default.yaml").is_file(); c=load_quality_config(); assert [h["heading"] for h in c["expected_headers"]]==["## Prerequisites","## Limitations","## Troubleshooting"], c'"$venv/bin/skillevaluator" --version"$venv/bin/skillevaluator" --help >/dev/null"$venv/bin/skillevaluator" models --help >/dev/nulldiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7a7b56b7..88b56de0 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,6 +4,24 @@ All notable changes to SkillEvaluator are documented in this file.## Unreleased+### Added++- `--quality-config PATH` on `validate`, `quality-check`, and `tier1` loads the+ deterministic quality check's expected Markdown headings from a YAML file+ instead of the built-in list. Each `expected_headers` entry sets a `heading`+ and optional `alternatives`, `points` (default 5), `message`, and+ `suggestion`; omitted message and suggestion text is generated from the+ headings. Without the flag, the packaged `quality-default.yaml` reproduces+ the previous Prerequisites, Limitations, and Troubleshooting checks and+ wording, so default scores are unchanged. An invalid file (bad YAML or+ unparseable values, unknown keys, duplicate keys or headings, non-heading+ values) is rejected before validation starts with exit code 2 and a message+ naming the file and field, even when `--checks` leaves out `quality`. The JSON report, Markdown report,+ and BENCHMARK.md record which config produced the quality score as+ `quality_config` (`packaged` or `custom`, the custom file's name, a+ `sha256:` digest of the normalized rules, and the `--min-score` pass+ threshold, which is kept out of the digest).+### Changed- README and onboarding docs now offer two first-run paths: a lightweight,diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdxindex e04cc5a5..fe1d4934 100644--- a/docs/cli-reference.mdx+++ b/docs/cli-reference.mdx@@ -101,6 +101,7 @@ autopilot. It is separate from these tier-only workflows.| `--checks TEXT` | All applicable static checks, including dependency auditing | Comma-separated subset of static checks || `--llm / --no-llm` | Enabled when a provider is configured | Enable or skip rubric, LLM security, and finding verification; explicit `--llm` requires valid provider configuration || `--min-score INTEGER` | `70` | Minimum passing quality and rubric score, from 0 to 100 |+| `--quality-config PATH` | packaged `quality-default.yaml` | YAML file that replaces the quality check's expected-heading rules. See [Tier 1: Validation](tier1-validation.mdx#the-checks). || `--profile TEXT` | `SKILLEVALUATOR_PROFILE`, then `external` | Validation policy profile || `--previous-version TEXT` | `SKILLEVALUATOR_PREVIOUS_VERSION`, else unset | Previous version for version checks |@@ -210,6 +211,7 @@ Static checks; LLM-free by default. Tier 1 gates the exit code and always runs.| `--llm, --tier1-llm / --no-llm, --no-tier1-llm` | `no-llm` | Enable LLM-backed security analysis (requires a configured public provider — see [Providers & Credentials](configuration.mdx)). || `--llm-verify` | off | Run a second LLM pass to suppress false-positive findings. || `--min-score INTEGER` | `70` | Minimum quality score (0–100) required to pass when the `quality` check runs. |+| `--quality-config PATH` | packaged `quality-default.yaml` | YAML file that replaces the quality check's expected-heading rules. See [Tier 1: Validation](tier1-validation.mdx#the-checks). |### Tier 2 · Deduplication@@ -273,8 +275,9 @@ skillevaluator quality-check ./my-skill| Flag | Default | Effect || --- | --- | --- || `--min-score INTEGER` | `70` | Minimum score required to pass. |+| `--quality-config PATH` | packaged `quality-default.yaml` | YAML file with expected section headers and rules for grading. |-Plus the standard report options. Scoring weights and grade bands are documented in [Tier 1: Validation](tier1-validation.mdx).+Plus the standard report options. Scoring weights, grade bands, and the `--quality-config` file format are documented in [Tier 1: Validation](tier1-validation.mdx).## rubric-evaldiff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdxindex 8f8253aa..b952f16d 100644--- a/docs/tier1-validation.mdx+++ b/docs/tier1-validation.mdx@@ -240,6 +240,47 @@ For `validate`, the default set is `schema`, `version`, `security`, `pii`, `lice<Tip>If a quality score changes unexpectedly, inspect the finding message rather than adding keywords. The checker rewards usable instructions and evidence in their relevant context, not vocabulary alone.</Tip>++ **Expected headings.** Reliability deducts points when `SKILL.md` lacks an expected heading. The packaged `quality-default.yaml` expects `## Prerequisites` (or `## Requirements`), `## Limitations`, and `## Troubleshooting` (or `## Common Issues` or `## FAQ`), each worth 5 points. Pass `--quality-config PATH` to `quality-check`, `tier1`, or `validate` to use your own list instead. The file replaces the packaged defaults rather than merging with them, so copy any default entries you want to keep.++ ```yaml title="quality.yaml"+ expected_headers:+ - heading: "## Prerequisites"+ alternatives:+ - "## Requirements"+ points: 5+ message: "No prerequisites/requirements documented"+ suggestion: "Document dependencies, API keys, or setup needed"+ - heading: "## Security"+ points: 8+ message: "No security notes documented"+ suggestion: "Add '## Security' describing permissions and data access"+ - heading: "## Examples"+ ```++ ```bash title="Score with a custom heading list"+ skillevaluator quality-check ./my-skill --quality-config quality.yaml+ skillevaluator tier1 ./my-skill --quality-config quality.yaml+ skillevaluator validate ./my-skill --quality-config quality.yaml+ ```++ | Field | Required | Meaning |+ | --- | --- | --- |+ | `heading` | Yes | A single ATX Markdown heading, such as `## Security` |+ | `alternatives` | No | Other headings that satisfy the same entry |+ | `points` | No (default `5`) | Non-negative number deducted from Reliability when no accepted heading is found |+ | `message` | No | Finding message reported when the heading is missing. Defaults to `Missing expected heading: '## Examples'`, listing any alternatives |+ | `suggestion` | No | Fix suggestion attached to the finding. Defaults to `Add a '## Examples' section`, or `Add one of these sections: ...` when there are alternatives |++ The packaged defaults set `message` and `suggestion` explicitly, so their findings keep the original wording.++ An entry is satisfied when the heading or any alternative appears anywhere in `SKILL.md` as text, so `## Security` also matches `### Security Notes` or a heading inside a code block.++ The file is checked before any validation runs. A file that is missing, unreadable, not UTF-8, invalid YAML, has an unknown key (such as `point` instead of `points`), or lists the same heading twice is a usage error: the command exits with code 2 and names the file and the problem. `validate` checks the file even when `--checks` leaves out `quality`.++ ```text title="Example error"+ Error: Invalid value for '--quality-config': quality.yaml: unknown key(s) in expected_headers[0]: point (allowed: alternatives, heading, message, points, suggestion)+ ```</Accordion><Accordion title="rubric-eval — LLM-as-judge (requires a provider)">@@ -332,7 +373,7 @@ Bundled Python scanners resolve next to the SkillEvaluator interpreter before `P| --- | --- || `0` | Success — every gate passed || `1` | Validation failed — a check failed or required scanner evidence is incomplete |-| `2` | Usage error — an unknown flag, a value outside an option's type or choices, or a target path that does not exist |+| `2` | Usage error — an unknown flag, a value outside an option's type or choices, a target path that does not exist, or an invalid `--quality-config` file |Tier 1 checks always gate the exit code inside `validate`. Tier 2 gates when it runs unless `--no-block-on-dedup` is set. Tier 3 is advisory unless `--block-on-agent-eval` is set. The `quality` check passes when the composite score reaches `--min-score` (default `70`). Reports record the effective gate membership for every tier so saved evidence and process status agree.diff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.pyindex fbdb766a..9e039075 100644--- a/src/skillevaluator/cli.py+++ b/src/skillevaluator/cli.py@@ -26,6 +26,7 @@ from skillevaluator.constants import ()from skillevaluator.logging_config import setup_loggingfrom skillevaluator.models.result import ValidationResult+from skillevaluator.quality_config import QualityConfigError, load_quality_configfrom skillevaluator.reporting.console_ui import (ValidateView,ViewProgressReporter,@@ -98,6 +99,24 @@ CUSTOM_GRADING_MODE_CHOICE = _AliasChoice()+class _QualityConfigPath(click.Path):+ """A ``--quality-config`` file, validated while arguments are parsed."""++ def __init__(self) -> None:+ super().__init__(exists=True, dir_okay=False, path_type=Path)++ def convert(self, value, param, ctx):+ path = super().convert(value, param, ctx)+ try:+ load_quality_config(path)+ except QualityConfigError as exc:+ self.fail(str(exc), param, ctx)+ return path+++QUALITY_CONFIG_PATH = _QualityConfigPath()++def _validate_similarity_threshold(_ctx: click.Context, _param: click.Parameter, value: float) -> float:if not math.isfinite(value) or not 0.0 <= value <= 1.0:raise click.BadParameter("must be finite and within [0, 1]")@@ -1079,6 +1098,8 @@ def _catalog_child_argv_from_ctx(ctx: click.Context, skill_dir: Path, output_dirmin_score = params.get("min_score", 70)if min_score != 70:argv.extend(["--min-score", str(min_score)])+ if params.get("quality_config"):+ argv.extend(["--quality-config", str(Path(params["quality_config"]).resolve())])if params.get("external"):argv.append("--external")if params.get("policy_path"):@@ -1589,7 +1610,7 @@ def _print_run_banner(target_path: Path, content_type: str, profile: str | None))@click.option("--quality-config",- type=click.Path(exists=True, dir_okay=False, path_type=Path),+ type=QUALITY_CONFIG_PATH,default=None,cls=GroupedOption,help_group=_TIER1_GROUP,@@ -2324,7 +2345,7 @@ validate.help_group_descriptions = {@click.option("--min-score", type=int, default=70, show_default=True)@click.option("--quality-config",- type=click.Path(exists=True, dir_okay=False, path_type=Path),+ type=QUALITY_CONFIG_PATH,default=None,help="Override the packaged quality-default.yaml with a custom YAML file.",)@@ -2608,6 +2629,12 @@ def _workflow_report_options(func):"--llm/--no-llm", default=None, help="Include rubric, LLM security, and verification [default: when configured].")@click.option("--min-score", type=click.IntRange(0, 100), default=70, show_default=True)+ "--quality-config",+ type=QUALITY_CONFIG_PATH,+ default=None,+ help="Override the packaged quality-default.yaml with a custom YAML file.",+)@click.option("--profile", default=None, help="Validation policy profile [default: external].")@click.option("--previous-version", default=None, help="Previous version for version checks.")@_workflow_report_options@@ -2616,6 +2643,7 @@ def _tier1_workflow(checks: str | None,llm: bool | None,min_score: int,+ quality_config: Path | None,profile: str | None,previous_version: str | None,report_formats: tuple[str, ...],@@ -2634,6 +2662,7 @@ def _tier1_workflow(checks=checks,llm=llm,min_score=min_score,+ quality_config=quality_config,profile=policy,previous_version=previous_version,)diff --git a/src/skillevaluator/config/quality-default.yaml b/src/skillevaluator/config/quality-default.yamlindex b0ceded7..a036afac 100644--- a/src/skillevaluator/config/quality-default.yaml+++ b/src/skillevaluator/config/quality-default.yaml@@ -1,4 +1,22 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0+# Default deterministic quality-check configuration.-limitations:- required: true- heading: "## Limitations"+expected_headers:+ - heading: "## Prerequisites"+ alternatives:+ - "## Requirements"+ points: 5+ message: "No prerequisites/requirements documented"+ suggestion: "Document dependencies, API keys, or setup needed"+ - heading: "## Limitations"+ points: 5+ message: "No limitations documented"+ suggestion: "Add '## Limitations' section with known issues/constraints"+ - heading: "## Troubleshooting"+ alternatives:+ - "## Common Issues"+ - "## FAQ"+ points: 5+ message: "No troubleshooting section documented"+ suggestion: "Add '## Troubleshooting' with Error/Cause/Solution patterns"diff --git a/src/skillevaluator/quality_config.py b/src/skillevaluator/quality_config.pyindex cdf42a0c..38cfcee2 100644--- a/src/skillevaluator/quality_config.py+++ b/src/skillevaluator/quality_config.py@@ -1,30 +1,184 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0+"""Configuration for deterministic quality scoring."""from __future__ import annotations+import hashlib+import json+import refrom importlib.resources import files+from math import isfinitefrom pathlib import Path-from typing import Any+from typing import Literal, TypedDictimport yamlDEFAULT_QUALITY_CONFIG = files("skillevaluator.config").joinpath("quality-default.yaml")+_ATX_HEADING_RE = re.compile(r"^(#{1,6})[ \t]+(.+?)(?:[ \t]+#+)?[ \t]*$")+_TOP_LEVEL_KEYS = frozenset({"expected_headers"})+_EXPECTED_HEADER_KEYS = frozenset({"heading", "alternatives", "points", "message", "suggestion"})class QualityConfigError(ValueError):"""Raised when a quality configuration is invalid."""-def load_quality_config(path: Path | None = None) -> dict[str, Any]:- source = path.read_text(encoding="utf-8") if path else DEFAULT_QUALITY_CONFIG.read_text(encoding="utf-8")+class ExpectedHeader(TypedDict):+ """A Markdown heading expected by the quality validator."""++ heading: str+ alternatives: list[str]+ points: int | float+ message: str+ suggestion: str+++class QualityConfig(TypedDict):+ """Validated deterministic quality configuration."""++ expected_headers: list[ExpectedHeader]+++class QualityConfigProvenance(TypedDict):+ """Which quality configuration produced a score, as recorded in reports."""++ source: Literal["packaged", "custom"]+ file: str | None+ digest: str+++def _normalize_heading(value: object, source: str, location: str) -> str:+ heading_match = _ATX_HEADING_RE.fullmatch(value) if isinstance(value, str) else None+ heading_title = heading_match.group(2).strip() if heading_match else ""+ if heading_match is None or not heading_title:+ raise QualityConfigError(f"{source}: {location} must be a single ATX Markdown heading")+ return f"{heading_match.group(1)} {heading_title}"+++def _reject_unknown_keys(mapping: dict, allowed: frozenset[str], source: str, location: str) -> None:+ unknown = sorted(str(key) for key in mapping if key not in allowed)+ if unknown:+ raise QualityConfigError(+ f"{source}: unknown key(s) in {location}: {', '.join(unknown)} (allowed: {', '.join(sorted(allowed))})"+ )+++class _UniqueKeySafeLoader(yaml.SafeLoader):+ """SafeLoader that rejects repeated mapping keys instead of keeping the last value."""++ def construct_mapping(self, node: yaml.MappingNode, deep: bool = False) -> dict:+ seen: set[object] = set()+ for key_node, _ in node.value:+ if key_node.tag == "tag:yaml.org,2002:merge":+ continue+ key = self.construct_object(key_node, deep=deep)+ try:+ duplicate = key in seen+ except TypeError:+ continue # unhashable keys are rejected by the base constructor+ if duplicate:+ raise yaml.constructor.ConstructorError(+ "while constructing a mapping", node.start_mark, f"duplicate key {key!r}", key_node.start_mark+ )+ seen.add(key)+ return super().construct_mapping(node, deep=deep)+++def _yaml_error_detail(exc: yaml.YAMLError) -> str:+ mark = getattr(exc, "problem_mark", None)+ problem = getattr(exc, "problem", None)+ if mark is None or problem is None:+ return str(exc)+ return f"line {mark.line + 1}, column {mark.column + 1}: {problem}"+++def load_quality_config(path: Path | None = None) -> QualityConfig:+ source = str(path) if path else "packaged quality-default.yaml"try:- raw = yaml.safe_load(source)+ text = path.read_text(encoding="utf-8") if path else DEFAULT_QUALITY_CONFIG.read_text(encoding="utf-8")+ except UnicodeDecodeError as exc:+ raise QualityConfigError(f"{source}: file is not valid UTF-8") from exc+ except OSError as exc:+ raise QualityConfigError(f"{source}: cannot read file: {exc.strerror or exc}") from exc+ try:+ raw = yaml.load(text, Loader=_UniqueKeySafeLoader) # nosec B506except yaml.YAMLError as exc:- raise QualityConfigError(f"{path or 'packaged quality-default.yaml'}: invalid YAML: {exc}") from exc- if not isinstance(raw, dict) or not isinstance(raw.get("limitations", {}), dict):- raise QualityConfigError("quality config: top-level and limitations must be mappings")- limitations = raw["limitations"]- required = limitations.get("required", True)- heading = limitations.get("heading", "## Limitations")- if not isinstance(required, bool) or not isinstance(heading, str) or not heading.strip():- raise QualityConfigError("quality config: limitations.required must be boolean and heading non-empty")- return {"limitations": {"required": required, "heading": heading}}+ raise QualityConfigError(f"{source}: invalid YAML: {_yaml_error_detail(exc)}") from exc+ except RecursionError as exc:+ raise QualityConfigError(f"{source}: invalid YAML: nesting is too deep") from exc+ except (ValueError, TypeError, KeyError, AttributeError) as exc:+ raise QualityConfigError(f"{source}: invalid YAML: cannot parse a value: {exc}") from exc+ if not isinstance(raw, dict) or not isinstance(raw.get("expected_headers"), list):+ raise QualityConfigError(f"{source}: expected_headers must be a list")+ _reject_unknown_keys(raw, _TOP_LEVEL_KEYS, source, "top level")++ expected_headers: list[ExpectedHeader] = []+ seen_headings: dict[str, str] = {}+ for index, rule in enumerate(raw["expected_headers"]):+ location = f"expected_headers[{index}]"+ if not isinstance(rule, dict):+ raise QualityConfigError(f"{source}: {location} must be a mapping")+ _reject_unknown_keys(rule, _EXPECTED_HEADER_KEYS, source, location)++ heading = _normalize_heading(rule.get("heading"), source, f"{location}.heading")+ raw_alternatives = rule.get("alternatives", [])+ if not isinstance(raw_alternatives, list):+ raise QualityConfigError(f"{source}: {location}.alternatives must be a list")+ alternatives = [+ _normalize_heading(value, source, f"{location}.alternatives[{alternative_index}]")+ for alternative_index, value in enumerate(raw_alternatives)+ ]+ points = rule.get("points", 5)+ try:+ valid_points = (+ not isinstance(points, bool) and isinstance(points, (int, float)) and isfinite(points) and points >= 0+ )+ except OverflowError: # isfinite() on an int too large for a float+ valid_points = False+ if not valid_points:+ raise QualityConfigError(f"{source}: {location}.points must be a non-negative number")+ # message and suggestion are optional; omitted ones get generic text naming the headings.+ accepted_display = " or ".join(f"'{accepted}'" for accepted in (heading, *alternatives))+ message = rule.get("message", f"Missing expected heading: {accepted_display}")+ suggestion = rule.get(+ "suggestion",+ f"Add a {accepted_display} section" if not alternatives else f"Add one of these sections: {accepted_display}",+ )+ for field, value in (("message", message), ("suggestion", suggestion)):+ if not isinstance(value, str) or not value.strip():+ raise QualityConfigError(f"{source}: {location}.{field} must be a non-empty string when set")++ for accepted in (heading, *alternatives):+ if accepted in seen_headings:+ raise QualityConfigError(+ f"{source}: duplicate heading '{accepted}' in {location} (already listed in {seen_headings[accepted]})"+ )+ seen_headings[accepted] = location++ expected_headers.append(+ {+ "heading": heading,+ "alternatives": alternatives,+ "points": points,+ "message": message,+ "suggestion": suggestion,+ }+ )++ return {"expected_headers": expected_headers}+++def quality_config_digest(config: QualityConfig) -> str:+ """Return a stable digest of the normalized rules, independent of file formatting."""+ canonical = json.dumps(config, sort_keys=True, separators=(",", ":")).encode()+ return f"sha256:{hashlib.sha256(canonical).hexdigest()}"+++def quality_config_provenance(path: Path | None, config: QualityConfig) -> QualityConfigProvenance:+ """Describe the effective config. Only the file name is kept, so reports do not leak local paths."""+ return {+ "source": "custom" if path else "packaged",+ "file": path.name if path else None,+ "digest": quality_config_digest(config),+ }diff --git a/src/skillevaluator/reporting/benchmark.py b/src/skillevaluator/reporting/benchmark.pyindex 14853e5c..941b6453 100644--- a/src/skillevaluator/reporting/benchmark.py+++ b/src/skillevaluator/reporting/benchmark.py@@ -104,6 +104,7 @@ _PUBLIC_USER_PATH = re.compile(rf")[^\s'\"<>]+)",flags=re.IGNORECASE,)+_SHA256_DIGEST = re.compile(r"sha256:[0-9a-f]{64}")_MARKDOWN_INLINE_SPECIAL = re.compile(r"([\\*_\[\]~])")_MARKDOWN_BLOCK_PREFIX = re.compile(r"^(?:#{1,6}|>|[+*-]|\d+[.)])(?=\s|$)")# The public benchmark gate rejects a non-PASS card whose visible text contains@@ -266,6 +267,13 @@ class BenchmarkReporter(ReporterBase):else:lines.append("- Source digest: not recorded (legacy or unbound result)")+ quality_config = _quality_config_label(results, private_labels)+ if quality_config:+ lines.append(f"- Quality config: {quality_config}")+ quality_min_score = _quality_min_score(results)+ if quality_min_score is not None:+ lines.append(f"- Quality min score: {quality_min_score}")+evaluated_at = _evaluated_at(ae)lines.append(f"- Evaluation date: {_publication_safe_inline(_evaluation_date(evaluated_at))}"@@ -932,6 +940,36 @@ def _evaluation_date(value: str) -> str:return value[:10] if re.fullmatch(r"\d{4}-\d{2}-\d{2}.*", value) else value+def _quality_config_label(results: list[ValidationResult], private_labels: tuple[str, ...]) -> str | None:+ """Return the Tier 1 quality config line, or ``None`` when no QUALITY result recorded one."""+ for result in results:+ config = result.metadata.get("quality_config") if isinstance(result.metadata, dict) else None+ if not isinstance(config, dict):+ continue+ source = config.get("source")+ digest = config.get("digest")+ if source not in {"packaged", "custom"} or not isinstance(digest, str) or not _SHA256_DIGEST.fullmatch(digest):+ continue+ file_name = config.get("file")+ file_label = (+ f" ({_publication_safe_inline(file_name, private_labels)})"+ if source == "custom" and isinstance(file_name, str) and file_name+ else ""+ )+ return f"{source}{file_label}, `{digest}`"+ return None+++def _quality_min_score(results: list[ValidationResult]) -> int | float | None:+ """Return the Tier 1 quality pass threshold, or ``None`` when no QUALITY result recorded one."""+ for result in results:+ config = result.metadata.get("quality_config") if isinstance(result.metadata, dict) else None+ min_score = config.get("min_score") if isinstance(config, dict) else None+ if isinstance(min_score, (int, float)) and not isinstance(min_score, bool) and math.isfinite(min_score):+ return min_score+ return None++def _evaluated_source(results: list[ValidationResult]) -> dict[str, str]:"""Return the validated evaluated-source identity recorded for this run.diff --git a/src/skillevaluator/reporting/json_reporter.py b/src/skillevaluator/reporting/json_reporter.pyindex 2e9fac9d..cab7fc39 100644--- a/src/skillevaluator/reporting/json_reporter.py+++ b/src/skillevaluator/reporting/json_reporter.py@@ -188,6 +188,17 @@ class JSONReporter(ReporterBase):if policy is not None:data["policy"] = policy+ quality_config = next(+ (+ result.metadata.get("quality_config")+ for result in results+ if isinstance(result.metadata.get("quality_config"), dict)+ ),+ None,+ )+ if quality_config is not None:+ data["quality_config"] = quality_config+# BENCHMARK.md records which source tree was evaluated, so the report a# CI job actually parses has to record it too: a consumer cannot verify# a card it never reads. The key is always present, so ``null`` says thediff --git a/src/skillevaluator/reporting/markdown.py b/src/skillevaluator/reporting/markdown.pyindex 40164bf0..9490f8db 100644--- a/src/skillevaluator/reporting/markdown.py+++ b/src/skillevaluator/reporting/markdown.py@@ -234,6 +234,24 @@ class MarkdownReporter(ReporterBase):if policy.get("digest"):lines.append(f"**Policy digest:** `{policy['digest']}`")+ quality_config = next(+ (+ result.metadata.get("quality_config")+ for result in results+ if isinstance(result.metadata.get("quality_config"), dict)+ ),+ None,+ )+ if quality_config is not None:+ source = quality_config.get("source", "packaged")+ file_label = f" (`{quality_config['file']}`)" if quality_config.get("file") else ""+ min_score_label = (+ f", min score {quality_config['min_score']}" if quality_config.get("min_score") is not None else ""+ )+ lines.append(+ f"**Quality config:** {source}{file_label}, `{quality_config.get('digest', '')}`{min_score_label}"+ )+if self.include_timestamp:timestamp = datetime.now(tz=UTC).strftime("%B %d, %Y at %I:%M %p UTC")lines.append(f"**Generated:** {timestamp}")diff --git a/src/skillevaluator/tier_workflows.py b/src/skillevaluator/tier_workflows.pyindex e0f273b1..11c540c6 100644--- a/src/skillevaluator/tier_workflows.py+++ b/src/skillevaluator/tier_workflows.py@@ -69,6 +69,7 @@ def run_tier1_workflow(checks: str | None,llm: bool | None,min_score: int,+ quality_config: Path | None,profile: ValidationPolicy,previous_version: str | None,) -> list[ValidationResult]:@@ -92,6 +93,7 @@ def run_tier1_workflow(use_llm=use_llm,llm_verify=use_llm,min_score=min_score,+ quality_config=quality_config,previous_version=previous_version,policy=profile,content_type="skill",diff --git a/src/skillevaluator/validators/quality_score.py b/src/skillevaluator/validators/quality_score.pyindex 58d2947a..ed27ce81 100644--- a/src/skillevaluator/validators/quality_score.py+++ b/src/skillevaluator/validators/quality_score.py@@ -36,8 +36,8 @@ from skillevaluator.constants import (from skillevaluator.logging_config import get_loggerfrom skillevaluator.models.quality import QualityScoreResultfrom skillevaluator.models.result import Finding, Severity, ValidationResult-from skillevaluator.quality_config import load_quality_configfrom skillevaluator.models.skill import XML_TAG_RE+from skillevaluator.quality_config import load_quality_config, quality_config_provenancefrom skillevaluator.validators.base import ValidatorBasefrom skillevaluator.validators.frontmatter_parser import FRONTMATTER_PATTERNfrom skillevaluator.validators.markdown import markdown_link_targets@@ -133,6 +133,7 @@ class QualityScoreValidator(ValidatorBase):def __init__(self, min_score: int = 70, quality_config: Path | None = None) -> None:self.min_score = min_scoreself.quality_config = load_quality_config(quality_config)+ self.quality_config_provenance = quality_config_provenance(quality_config, self.quality_config)@propertydef name(self) -> str:@@ -214,8 +215,12 @@ class QualityScoreValidator(ValidatorBase):def validate(self, skill_path: Path) -> ValidationResult:"""Validate a skill directory and produce quality scores."""if self._is_skill_directory(skill_path):- return self._validate_single_skill(skill_path)- return self._validate_folder(skill_path)+ result = self._validate_single_skill(skill_path)+ else:+ result = self._validate_folder(skill_path)+ # min_score sits beside the digest rather than in it: the digest names the rules, min_score the pass bar.+ result.metadata["quality_config"] = {**self.quality_config_provenance, "min_score": self.min_score}+ return resultdef _validate_folder(self, root: Path) -> ValidationResult:"""Validate all skills in a folder, aggregating quality results."""@@ -670,29 +675,10 @@ class QualityScoreValidator(ValidatorBase):elif qs.skill_type == "lib-based":self._check_lib_reliability(dim, skill_path)- # Universal checks- if "## Prerequisites" not in content and "## Requirements" not in content:- dim.deduct(- 5,- "info",- "No prerequisites/requirements documented",- "Document dependencies, API keys, or setup needed",- )- limitations = self.quality_config["limitations"]- if limitations["required"] and limitations["heading"] not in content:- dim.deduct(- 5,- "info",- "No limitations documented",- "Add '## Limitations' section with known issues/constraints",- )- if not any(s in content for s in ["## Troubleshooting", "## Common Issues", "## FAQ"]):- dim.deduct(- 5,- "info",- "No troubleshooting section documented",- "Add '## Troubleshooting' with Error/Cause/Solution patterns",- )+ for expected in self.quality_config["expected_headers"]:+ accepted_headings = [expected["heading"], *expected["alternatives"]]+ if not any(heading in content for heading in accepted_headings):+ dim.deduct(expected["points"], "info", expected["message"], expected["suggestion"])def _check_script_reliability(self, dim, skill_path: Path) -> None:no_error_handling = []diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.jsonindex 8d68336d..3d65650b 100644--- a/tests/golden/cli_surface.json+++ b/tests/golden/cli_surface.json@@ -1561,6 +1561,14 @@"param_type": "option","type": "integer range"},+ {+ "name": "quality_config",+ "opts": [+ "--quality-config"+ ],+ "param_type": "option",+ "type": "file"+ },{"name": "profile","opts": [@@ -1648,6 +1656,14 @@"param_type": "option","type": "integer range"},+ {+ "name": "quality_config",+ "opts": [+ "--quality-config"+ ],+ "param_type": "option",+ "type": "file"+ },{"name": "profile","opts": [diff --git a/tests/reporting/test_quality_config_reporting.py b/tests/reporting/test_quality_config_reporting.pynew file mode 100644index 00000000..7be1d36e--- /dev/null+++ b/tests/reporting/test_quality_config_reporting.py@@ -0,0 +1,90 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Every report names the quality config that produced the Tier 1 quality score."""++import json++import pytest++from skillevaluator.models import ValidationResult+from skillevaluator.reporting import BenchmarkReporter, JSONReporter, MarkdownReporter++_DIGEST = "sha256:" + "a" * 64+++def _quality_result(quality_config: dict | None) -> ValidationResult:+ result = ValidationResult(validator_name="QUALITY", validator_description="Quality score")+ if quality_config is not None:+ result.metadata["quality_config"] = quality_config+ return result+++def _benchmark_line(results: list[ValidationResult], field: str = "Quality config") -> str | None:+ card = BenchmarkReporter().render_all(results)+ return next((line for line in card.splitlines() if line.startswith(f"- {field}:")), None)+++def test_json_report_records_quality_config() -> None:+ config = {"source": "custom", "file": "team-quality.yaml", "digest": _DIGEST, "min_score": 80}++ data = json.loads(JSONReporter().render_all([_quality_result(config)]))++ assert data["quality_config"] == config+++def test_markdown_report_records_quality_config() -> None:+ report = MarkdownReporter().render_all(+ [_quality_result({"source": "custom", "file": "team-quality.yaml", "digest": _DIGEST, "min_score": 80})]+ )++ assert f"**Quality config:** custom (`team-quality.yaml`), `{_DIGEST}`, min score 80" in report+++ ("config", "expected"),+ [+ pytest.param(+ {"source": "custom", "file": "team-quality.yaml", "digest": _DIGEST},+ f"- Quality config: custom (team-quality.yaml), `{_DIGEST}`",+ id="custom",+ ),+ pytest.param(+ {"source": "packaged", "file": None, "digest": _DIGEST},+ f"- Quality config: packaged, `{_DIGEST}`",+ id="packaged",+ ),+ pytest.param(+ {"source": "custom", "file": "team_*rules*.yaml", "digest": _DIGEST},+ f"- Quality config: custom (team\\_\\*rules\\*.yaml), `{_DIGEST}`",+ id="markdown-escaped-file-name",+ ),+ ],+)+def test_benchmark_records_quality_config(config: dict, expected: str) -> None:+ assert _benchmark_line([_quality_result(config)]) == expected+++ "config",+ [+ pytest.param(None, id="not-recorded"),+ pytest.param({"source": "custom", "file": "x.yaml", "digest": "sha256:`injected`"}, id="malformed-digest"),+ pytest.param({"source": "other", "file": "x.yaml", "digest": _DIGEST}, id="unknown-source"),+ ],+)+def test_benchmark_omits_missing_or_malformed_quality_config(config: dict | None) -> None:+ assert _benchmark_line([_quality_result(config)]) is None+++def test_benchmark_records_quality_min_score() -> None:+ config = {"source": "packaged", "file": None, "digest": _DIGEST, "min_score": 80}++ assert _benchmark_line([_quality_result(config)], "Quality min score") == "- Quality min score: 80"++[email protected]("min_score", [None, True, "80", float("nan")], ids=["missing", "bool", "string", "nan"])+def test_benchmark_omits_missing_or_malformed_quality_min_score(min_score: object) -> None:+ config = {"source": "packaged", "file": None, "digest": _DIGEST, "min_score": min_score}++ assert _benchmark_line([_quality_result(config)], "Quality min score") is Nonediff --git a/tests/test_commands.py b/tests/test_commands.pyindex ca690f8f..13647772 100644--- a/tests/test_commands.py+++ b/tests/test_commands.py@@ -669,6 +669,230 @@ def test_standalone_tier1_rejects_in_skill_report_output_before_validation(assert not output_dir.exists()+def _quality_check_messages(+ command: list[str],+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+ *extra_args: str,+) -> list[str]:+ shutil.copytree(FIXTURE, tmp_path / "simple")+ private_state = tmp_path / "private-state"+ private_state.mkdir(mode=0o700)+ monkeypatch.setenv("SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE", str(private_state / "key"))+ monkeypatch.chdir(tmp_path)++ result = CliRunner().invoke(+ cli,+ [*command, "simple", "--min-score", "0", "--report", "json", "--output-dir", "out", *extra_args],+ )++ assert result.exit_code == 0, result.output+ report = json.loads((tmp_path / "out" / "skillevaluator-quality.json").read_text(encoding="utf-8"))+ return [finding["message"] for result in report["results"] for finding in result["findings"]]++[email protected]("command", [["quality-check"], ["tier1", "quality-check"]])+def test_quality_check_uses_packaged_default_quality_config(+ command: list[str],+ monkeypatch: pytest.MonkeyPatch,+ tmp_path: Path,+) -> None:+ messages = _quality_check_messages(command, tmp_path, monkeypatch)++ assert "No prerequisites/requirements documented" in messages+ assert "No limitations documented" in messages+ assert "No troubleshooting section documented" in messages++[email protected]("command", [["quality-check"], ["tier1", "quality-check"]])+def test_quality_check_applies_custom_quality_config(+ command: list[str],+ monkeypatch: pytest.MonkeyPatch,+ tmp_path: Path,+) -> None:+ (tmp_path / "quality.yaml").write_text(+ 'expected_headers:\n - heading: "## Security"\n'+ " message: No security notes documented\n suggestion: Add security notes\n",+ encoding="utf-8",+ )++ messages = _quality_check_messages(command, tmp_path, monkeypatch, "--quality-config", "quality.yaml")++ assert "No security notes documented" in messages+ assert "No limitations documented" not in messages++[email protected]("command", [["quality-check"], ["tier1", "quality-check"]])+def test_quality_check_rejects_missing_quality_config(command: list[str], tmp_path: Path) -> None:+ result = CliRunner().invoke(+ cli, [*command, str(FIXTURE), "--quality-config", str(tmp_path / "missing.yaml")]+ )++ assert result.exit_code == 2+ assert "does not exist" in result.output+++def _validate_quality_messages(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *extra_args: str) -> list[str]:+ shutil.copytree(FIXTURE, tmp_path / "simple")+ private_state = tmp_path / "private-state"+ private_state.mkdir(mode=0o700)+ monkeypatch.setenv("SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE", str(private_state / "key"))+ monkeypatch.chdir(tmp_path)++ result = CliRunner().invoke(+ cli,+ [+ "validate",+ "--no-tier3",+ "simple",+ "--no-llm",+ "--no-tier2",+ "--tier1-checks",+ "schema,quality",+ "--min-score",+ "0",+ "--report",+ "json",+ "--output-dir",+ "out",+ *extra_args,+ ],+ )++ assert result.exit_code == 0, result.output+ (report_path,) = (tmp_path / "out").glob("skillevaluator-output-*.json")+ report = json.loads(report_path.read_text(encoding="utf-8"))+ return [finding["message"] for result in report["results"] for finding in result["findings"]]+++def test_validate_uses_packaged_default_quality_config(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:+ messages = _validate_quality_messages(tmp_path, monkeypatch)++ assert "No prerequisites/requirements documented" in messages+ assert "No limitations documented" in messages+ assert "No troubleshooting section documented" in messages+++def test_validate_applies_custom_quality_config(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:+ (tmp_path / "quality.yaml").write_text(+ 'expected_headers:\n - heading: "## Security"\n'+ " message: No security notes documented\n suggestion: Add security notes\n",+ encoding="utf-8",+ )++ messages = _validate_quality_messages(tmp_path, monkeypatch, "--quality-config", "quality.yaml")++ assert "No security notes documented" in messages+ assert "No limitations documented" not in messages+++def _tier1_quality_messages(+ command: list[str],+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+ *extra_args: str,+) -> list[str]:+ shutil.copytree(FIXTURE, tmp_path / "simple")+ private_state = tmp_path / "private-state"+ private_state.mkdir(mode=0o700)+ monkeypatch.setenv("SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE", str(private_state / "key"))+ monkeypatch.chdir(tmp_path)++ result = CliRunner().invoke(+ cli,+ [+ *command,+ "simple",+ "--checks",+ "schema,quality",+ "--no-llm",+ "--min-score",+ "0",+ "--report",+ "json",+ "--output-dir",+ "out",+ *extra_args,+ ],+ )++ assert result.exit_code == 0, result.output+ report = json.loads((tmp_path / "out" / "skillevaluator-tier1.json").read_text(encoding="utf-8"))+ return [finding["message"] for result in report["results"] for finding in result["findings"]]++[email protected]("command", [["tier1"], ["tier1", "validate"]])+def test_tier1_workflow_uses_packaged_default_quality_config(+ command: list[str],+ monkeypatch: pytest.MonkeyPatch,+ tmp_path: Path,+) -> None:+ messages = _tier1_quality_messages(command, tmp_path, monkeypatch)++ assert "No prerequisites/requirements documented" in messages+ assert "No limitations documented" in messages+ assert "No troubleshooting section documented" in messages++[email protected]("command", [["tier1"], ["tier1", "validate"]])+def test_tier1_workflow_applies_custom_quality_config(+ command: list[str],+ monkeypatch: pytest.MonkeyPatch,+ tmp_path: Path,+) -> None:+ (tmp_path / "quality.yaml").write_text(+ 'expected_headers:\n - heading: "## Security"\n'+ " message: No security notes documented\n suggestion: Add security notes\n",+ encoding="utf-8",+ )++ messages = _tier1_quality_messages(command, tmp_path, monkeypatch, "--quality-config", "quality.yaml")++ assert "No security notes documented" in messages+ assert "No limitations documented" not in messages+++_INVALID_QUALITY_CONFIG_COMMANDS = [+ ["quality-check"],+ ["tier1", "quality-check"],+ ["tier1"],+ ["tier1", "validate"],+ ["validate", "--no-tier3", "--no-llm", "--no-tier2", "--checks", "schema,quality"],+ # The config is validated up front even when the quality check is not selected.+ ["validate", "--no-tier3", "--no-llm", "--no-tier2", "--checks", "schema"],+]++[email protected]("command", _INVALID_QUALITY_CONFIG_COMMANDS)+ ("config", "expected"),+ [+ ("not: [valid\n", "invalid YAML: line 2, column 1:"),+ (+ 'expected_headers:\n - heading: "## Limitations"\n point: 10\n'+ " message: Missing\n suggestion: Add it\n",+ "unknown key(s) in expected_headers[0]: point",+ ),+ ],+)+def test_cli_rejects_invalid_quality_config_as_usage_error(+ command: list[str],+ config: str,+ expected: str,+ tmp_path: Path,+) -> None:+ config_path = tmp_path / "quality.yaml"+ config_path.write_text(config, encoding="utf-8")++ result = CliRunner().invoke(cli, [*command, str(FIXTURE), "--quality-config", str(config_path)])++ output = _plain_text(result.output)+ assert result.exit_code == 2, result.output+ assert "Invalid value for '--quality-config'" in output+ assert expected in output+ assert "Traceback" not in output++def test_standalone_tier2_rejects_in_skill_report_output_before_validation(monkeypatch: pytest.MonkeyPatch,tmp_path: Path,diff --git a/tests/test_quality_config.py b/tests/test_quality_config.pyindex ab1f6965..55af12b0 100644--- a/tests/test_quality_config.py+++ b/tests/test_quality_config.py@@ -1,27 +1,248 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0+from pathlib import Path-from skillevaluator.quality_config import load_quality_config+import pytest++from skillevaluator.quality_config import (+ QualityConfigError,+ load_quality_config,+ quality_config_digest,+ quality_config_provenance,+)def test_packaged_default_quality_config_is_available() -> None:config = load_quality_config()- assert config["limitations"] == {- "required": True,- "heading": "## Limitations",- }+ assert config["expected_headers"] == [+ {+ "heading": "## Prerequisites",+ "alternatives": ["## Requirements"],+ "points": 5,+ "message": "No prerequisites/requirements documented",+ "suggestion": "Document dependencies, API keys, or setup needed",+ },+ {+ "heading": "## Limitations",+ "alternatives": [],+ "points": 5,+ "message": "No limitations documented",+ "suggestion": "Add '## Limitations' section with known issues/constraints",+ },+ {+ "heading": "## Troubleshooting",+ "alternatives": ["## Common Issues", "## FAQ"],+ "points": 5,+ "message": "No troubleshooting section documented",+ "suggestion": "Add '## Troubleshooting' with Error/Cause/Solution patterns",+ },+ ]+ # Upstream deducted the int 5, so reports keep "deduction": 5 rather than 5.0.+ assert all(type(rule["points"]) is int for rule in config["expected_headers"])def test_quality_config_can_be_overridden(tmp_path: Path) -> None:config_path = tmp_path / "quality.yaml"config_path.write_text(- 'limitations:\n required: true\n heading: "## Failure Constellations"\n',+ 'expected_headers:\n - heading: "## Failure Constellations"\n'+ ' alternatives:\n - "## Known Failures"\n'+ " message: No failure constellations documented\n"+ " suggestion: Add '## Failure Constellations'\n"+ ' - heading: "### Security"\n points: 8.5\n'+ " message: No security notes documented\n"+ " suggestion: Add '### Security'\n",+ encoding="utf-8",+ )++ config = load_quality_config(config_path)++ assert config["expected_headers"] == [+ {+ "heading": "## Failure Constellations",+ "alternatives": ["## Known Failures"],+ "points": 5,+ "message": "No failure constellations documented",+ "suggestion": "Add '## Failure Constellations'",+ },+ {+ "heading": "### Security",+ "alternatives": [],+ "points": 8.5,+ "message": "No security notes documented",+ "suggestion": "Add '### Security'",+ },+ ]+++ "config",+ [+ "expected_headers: not-a-list\n",+ "expected_headers:\n - heading: Limitations\n",+ 'expected_headers:\n - heading: "## "\n',+ 'expected_headers:\n - heading: "## Limitations"\n alternatives: not-a-list\n',+ 'expected_headers:\n - heading: "## Limitations"\n points: -1\n',+ 'expected_headers:\n - heading: "## Limitations"\n message: ""\n',+ 'expected_headers:\n - heading: "## Limitations"\n suggestion:\n',+ 'expected_headers:\n - heading: "## Limitations"\n message: [Missing]\n',+ ],+)+def test_quality_config_rejects_invalid_expected_headers(tmp_path: Path, config: str) -> None:+ config_path = tmp_path / "quality.yaml"+ config_path.write_text(config, encoding="utf-8")++ with pytest.raises(QualityConfigError, match="expected_headers"):+ load_quality_config(config_path)+++def test_quality_config_generates_message_and_suggestion_when_omitted(tmp_path: Path) -> None:+ config_path = tmp_path / "quality.yaml"+ config_path.write_text(+ 'expected_headers:\n - heading: "## Security"\n'+ ' - heading: "## Prerequisites"\n alternatives:\n - "## Requirements"\n points: 5\n'+ ' - heading: "## Limitations"\n message: No limitations documented\n',encoding="utf-8",)config = load_quality_config(config_path)- assert config["limitations"] == {- "required": True,- "heading": "## Failure Constellations",+ assert [(rule["message"], rule["suggestion"]) for rule in config["expected_headers"]] == [+ ("Missing expected heading: '## Security'", "Add a '## Security' section"),+ (+ "Missing expected heading: '## Prerequisites' or '## Requirements'",+ "Add one of these sections: '## Prerequisites' or '## Requirements'",+ ),+ ("No limitations documented", "Add a '## Limitations' section"),+ ]+++ ("config", "match"),+ [+ (+ "expected_headers: []\nexpected_header: []\n",+ r"unknown key\(s\) in top level: expected_header",+ ),+ (+ 'expected_headers:\n - heading: "## Limitations"\n point: 10\n'+ " message: Missing\n suggestion: Add it\n",+ r"unknown key\(s\) in expected_headers\[0\]: point",+ ),+ (+ 'expected_headers:\n - heading: "## Limitations"\n message: Missing\n suggestion: Add it\n'+ ' - heading: "## Limits"\n alternatives:\n - "## Limitations"\n'+ " message: Missing\n suggestion: Add it\n",+ r"duplicate heading '## Limitations' in expected_headers\[1\] \(already listed in expected_headers\[0\]\)",+ ),+ ],+)+def test_quality_config_rejects_unknown_keys_and_duplicate_headings(+ tmp_path: Path, config: str, match: str+) -> None:+ config_path = tmp_path / "quality.yaml"+ config_path.write_text(config, encoding="utf-8")++ with pytest.raises(QualityConfigError, match=match):+ load_quality_config(config_path)+++def test_quality_config_errors_name_the_file_and_yaml_position(tmp_path: Path) -> None:+ config_path = tmp_path / "quality.yaml"+ config_path.write_text("expected_headers: [\n", encoding="utf-8")++ with pytest.raises(QualityConfigError) as exc_info:+ load_quality_config(config_path)++ message = str(exc_info.value)+ assert message.startswith(f"{config_path}: invalid YAML: line 2, column 1: ")+ assert "\n" not in message+++def test_quality_config_rejects_non_utf8_file(tmp_path: Path) -> None:+ config_path = tmp_path / "quality.yaml"+ config_path.write_text("expected_headers: []\n", encoding="utf-16")++ with pytest.raises(QualityConfigError, match="not valid UTF-8"):+ load_quality_config(config_path)+++ ("config", "match"),+ [+ (+ 'expected_headers:\n - heading: "## Prerequisites"\nexpected_headers:\n - heading: "## Security"\n',+ r"invalid YAML: line 3, column 1: duplicate key 'expected_headers'",+ ),+ (+ 'expected_headers:\n - heading: "## Limitations"\n points: 5\n points: 10\n',+ r"invalid YAML: line 4, column 5: duplicate key 'points'",+ ),+ ("expected_headers: " + "[" * 10000 + "]" * 10000 + "\n", "invalid YAML: nesting is too deep"),+ (+ 'expected_headers:\n - heading: "## Limitations"\n points: ' + "9" * 400 + "\n",+ r"expected_headers\[0\]\.points must be a non-negative number",+ ),+ (+ 'expected_headers:\n - heading: "## Limitations"\n points: ' + "9" * 5000 + "\n",+ r"invalid YAML: cannot parse a value: Exceeds the limit",+ ),+ (+ 'expected_headers:\n - heading: "## Limitations"\n message: 2026-02-30\n',+ r"invalid YAML: cannot parse a value: day is out of range",+ ),+ (+ 'expected_headers:\n - heading: "## Limitations"\n points: !!int abc\n',+ r"invalid YAML: cannot parse a value: invalid literal for int\(\)",+ ),+ ],+ ids=[+ "duplicate-top-level-key",+ "duplicate-rule-key",+ "deep-nesting",+ "points-overflow",+ "integer-digit-limit",+ "impossible-date",+ "bad-explicit-tag",+ ],+)+def test_quality_config_rejects_duplicate_keys_and_pathological_values(+ tmp_path: Path, config: str, match: str+) -> None:+ config_path = tmp_path / "quality.yaml"+ config_path.write_text(config, encoding="utf-8")++ with pytest.raises(QualityConfigError, match=match):+ load_quality_config(config_path)+++def test_quality_config_digest_ignores_formatting_but_not_rules(tmp_path: Path) -> None:+ compact = tmp_path / "compact.yaml"+ compact.write_text('expected_headers: [{heading: "## Limitations", points: 5}]\n', encoding="utf-8")+ verbose = tmp_path / "verbose.yaml"+ verbose.write_text(+ '# team rules\nexpected_headers:\n - heading: "## Limitations"\n points: 5\n',+ encoding="utf-8",+ )+ changed = tmp_path / "changed.yaml"+ changed.write_text('expected_headers:\n - heading: "## Limitations"\n points: 6\n', encoding="utf-8")++ digest = quality_config_digest(load_quality_config(compact))++ assert digest.startswith("sha256:") and len(digest) == 71+ assert quality_config_digest(load_quality_config(verbose)) == digest+ assert quality_config_digest(load_quality_config(changed)) != digest+++def test_quality_config_provenance_names_custom_file_without_its_directory(tmp_path: Path) -> None:+ config_path = tmp_path / "team-quality.yaml"+ config_path.write_text("expected_headers: []\n", encoding="utf-8")+ config = load_quality_config(config_path)++ assert quality_config_provenance(config_path, config) == {+ "source": "custom",+ "file": "team-quality.yaml",+ "digest": quality_config_digest(config),}+ assert quality_config_provenance(None, load_quality_config())["file"] is Nonediff --git a/tests/validators/test_quality_score.py b/tests/validators/test_quality_score.pyindex d556a4c2..127e3608 100644--- a/tests/validators/test_quality_score.py+++ b/tests/validators/test_quality_score.py@@ -8,6 +8,7 @@ from pathlib import Pathimport pytestfrom skillevaluator.models.quality import QualityScoreResult, score_to_grade+from skillevaluator.quality_config import load_quality_config, quality_config_digestfrom skillevaluator.validators.quality_score import QualityScoreValidator@@ -249,6 +250,98 @@ class TestQualityScoreValidator:assert qs["overall_score"] < 90assert qs["skill_type"] == "guide-only"+ def test_expected_headers_use_configured_deductions(self, tmp_path: Path):+ skill_dir = _write_issue_skill(tmp_path)+ config_path = tmp_path / "quality.yaml"+ config_path.write_text(+ 'expected_headers:\n - heading: "## Limitations"\n points: 2.0\n'+ " message: No limitations documented\n suggestion: Add limitations\n"+ ' - heading: "### Security"\n points: 7.5\n'+ " message: No security notes documented\n suggestion: Add security notes\n",+ encoding="utf-8",+ )++ result = QualityScoreValidator(min_score=0, quality_config=config_path).validate(skill_dir)++ missing_header = next(+ finding for finding in result.findings if finding.message == "No security notes documented"+ )+ assert missing_header.metadata["deduction"] == 7.5+ assert missing_header.suggestion == "Add security notes"+ assert all(finding.message != "No limitations documented" for finding in result.findings)++ def test_expected_header_matches_heading_text_anywhere_in_content(self, tmp_path: Path):+ skill_dir = _write_issue_skill(tmp_path, extra_body="\n### Security Notes\n\nReview permissions.\n")+ config_path = tmp_path / "quality.yaml"+ config_path.write_text(+ 'expected_headers:\n - heading: "## Security"\n'+ " message: No security notes documented\n suggestion: Add security notes\n",+ encoding="utf-8",+ )++ result = QualityScoreValidator(min_score=0, quality_config=config_path).validate(skill_dir)++ assert all(finding.message != "No security notes documented" for finding in result.findings)++ # Matching is a case-sensitive substring test on the raw SKILL.md text, not a+ # parsed-heading match. These cases pin that behavior in both directions.+ @pytest.mark.parametrize(+ ("extra_body", "satisfied"),+ [+ pytest.param("\n## Security\n\nReview permissions.\n", True, id="exact-heading"),+ pytest.param("\n### Security\n\nReview permissions.\n", True, id="deeper-heading-level"),+ pytest.param("\n## Security and more\n\nReview permissions.\n", True, id="longer-heading-text"),+ pytest.param("\n```markdown\n## Security\n```\n", True, id="inside-code-block"),+ pytest.param("\nSee the ## Security notes below.\n", True, id="inside-prose"),+ pytest.param("\n# Security\n\nReview permissions.\n", False, id="shallower-heading-level"),+ pytest.param("\n## security\n\nReview permissions.\n", False, id="different-case"),+ pytest.param("\n##Security\n\nReview permissions.\n", False, id="no-space-after-hashes"),+ pytest.param("\nSecurity\n========\n\nReview permissions.\n", False, id="setext-heading"),+ ],+ )+ def test_expected_header_matches_heading_as_text_anywhere(+ self, tmp_path: Path, extra_body: str, satisfied: bool+ ):+ skill_dir = _write_issue_skill(tmp_path, extra_body=extra_body)+ config_path = tmp_path / "quality.yaml"+ config_path.write_text(+ 'expected_headers:\n - heading: "## Security"\n'+ " message: No security notes documented\n suggestion: Add security notes\n",+ encoding="utf-8",+ )++ result = QualityScoreValidator(min_score=0, quality_config=config_path).validate(skill_dir)++ messages = [finding.message for finding in result.findings]+ assert ("No security notes documented" not in messages) is satisfied++ def test_expected_header_accepts_configured_alternative(self, tmp_path: Path):+ skill_dir = _write_issue_skill(tmp_path)+ skill_md = skill_dir / "SKILL.md"+ skill_md.write_text(+ skill_md.read_text(encoding="utf-8").replace("## Prerequisites", "## Requirements"),+ encoding="utf-8",+ )++ result = QualityScoreValidator(min_score=0).validate(skill_dir)++ assert all(+ finding.message != "No prerequisites/requirements documented" for finding in result.findings+ )++ @pytest.mark.parametrize("alternative", ["## Common Issues", "## FAQ"])+ def test_troubleshooting_header_accepts_configured_alternative(self, tmp_path: Path, alternative: str):+ skill_dir = _write_issue_skill(tmp_path)+ skill_md = skill_dir / "SKILL.md"+ skill_md.write_text(+ skill_md.read_text(encoding="utf-8").replace("## Troubleshooting", alternative),+ encoding="utf-8",+ )++ result = QualityScoreValidator(min_score=0).validate(skill_dir)++ assert all(finding.message != "No troubleshooting section documented" for finding in result.findings)+def test_bad_skill_fails_min_score(self, bad_skill):v = QualityScoreValidator(min_score=70)result = v.validate(bad_skill)@@ -512,6 +605,44 @@ class TestQualityScoreValidator:assert "Quality" in v.nameassert "Correctness" in v.description+ def test_result_records_packaged_quality_config(self, tmp_path: Path):+ skill_dir = _write_issue_skill(tmp_path)++ result = QualityScoreValidator().validate(skill_dir)++ assert result.metadata["quality_config"] == {+ "source": "packaged",+ "file": None,+ "digest": quality_config_digest(load_quality_config()),+ "min_score": 70,+ }++ def test_result_records_custom_quality_config_file_name_only(self, tmp_path: Path):+ skill_dir = _write_issue_skill(tmp_path)+ config_path = tmp_path / "configs" / "team-quality.yaml"+ config_path.parent.mkdir()+ config_path.write_text("expected_headers: []\n", encoding="utf-8")++ result = QualityScoreValidator(min_score=80, quality_config=config_path).validate(skill_dir)++ assert result.metadata["quality_config"] == {+ "source": "custom",+ "file": "team-quality.yaml",+ "digest": quality_config_digest({"expected_headers": []}),+ "min_score": 80,+ }++ def test_folder_result_records_quality_config(self, tmp_path: Path):+ _write_issue_skill(tmp_path)+ config_path = tmp_path / "team-quality.yaml"+ config_path.write_text("expected_headers: []\n", encoding="utf-8")++ result = QualityScoreValidator(min_score=0, quality_config=config_path).validate(tmp_path)++ assert result.metadata["quality_config"]["source"] == "custom"+ assert result.metadata["quality_config"]["file"] == "team-quality.yaml"+ assert result.metadata["quality_config"]["min_score"] == 0+class TestQualityScoreDeterministicContracts:@pytest.mark.parametrize(