Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Fix quality scoring with CommonMark structural parsing (#53)
Christopher Kevin68013dff0273
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c268ca29..83140006 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -69,10 +69,12 @@ All notable changes to SkillEvaluator are documented in this file.configured judge provider, model, source, and whether a dedicated job-wideoverride was applied, separately from agent models. A provider fallback maystill use a different model for an individual judge call.-- Quality scoring now uses boundary-aware and context-aware matching for XML tags,- reserved names, MCP guidance, README references, time references, exclusivity- language, instruction action verbs, and nested Markdown links, avoiding- incidental-word score changes.+- Quality scoring now uses boundary-aware lexical matching and CommonMark-parsed+ structural links instead of hand-written Markdown parsing or regex inference+ of author intent. Deterministic checks no longer infer MCP negation, temporal+ intent, README guidance, or exclusivity from prose;+ use `rubric-eval` for semantic documentation judgments and Tier 3 for+ observed agent behavior.- Tier 3 generated tasks now stage only an entry's declared `files`, preventingundeclared fixtures from the shared `evals/files/` directory from appearingin that task's `/workspace/input/`, while preserving copy-all behavior fordiff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.mdindex 69664187..790b89b9 100644--- a/THIRD_PARTY_NOTICES.md+++ b/THIRD_PARTY_NOTICES.md@@ -6,7 +6,7 @@ records the exact resolved dependency set used for this release.| Dependency group | Packages and licenses || --- | --- |-| Base | Click (BSD-3-Clause), IDNA (BSD-3-Clause), Jinja2 (BSD-3-Clause), Pydantic (MIT), PyYAML (MIT), Rich (MIT) |+| Base | Click (BSD-3-Clause), IDNA (BSD-3-Clause), Jinja2 (BSD-3-Clause), Markdown-It-Py (MIT), Pydantic (MIT), PyYAML (MIT), Rich (MIT) || LLM | Anthropic (MIT), Boto3 (Apache-2.0), LiteLLM (MIT), OpenAI (Apache-2.0) || Tier 3 | Harbor (Apache-2.0) || Security | Bandit (Apache-2.0), pip-audit (Apache-2.0) |diff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdxindex 9ab956f3..fbdc90cd 100644--- a/docs/tier1-validation.mdx+++ b/docs/tier1-validation.mdx@@ -208,7 +208,7 @@ The default set is `schema`, `version`, `security`, `pii`, `license`, `code-inteThe checker detects five skill shapes and applies relevant checks: **script-based** (Python or shell files under `scripts/`), **lib-based** (a Python module containing `__init__.py`), **resource-based** (assets, templates, design-system content, or resources), **guide-only** (documentation without executable or resource content), and **hybrid** (scripts plus library or resource content).- Quality signals use boundary-aware and context-aware matching. A score changes only when the relevant construct is present — for example, a complete reserved-name segment, an actual XML-like tag, active MCP usage without nearby negation, an actionable link to `README.md`, a calendar-year reference, an exclusivity claim, an instruction action verb, or a local Markdown link from a reference file. Incidental substrings, prose inside fenced examples or comments, external URLs, and unrelated troubleshooting text do not satisfy or trigger those checks.+ Quality signals use boundary-aware lexical matching for unambiguous terms and CommonMark parsing for structural link evidence. Links inside code, images, or comments do not count, and external URLs do not satisfy local-reference checks. Deterministic Tier 1 checks do not infer author intent from ambiguous prose such as MCP negation, temporal language, README guidance, or exclusivity claims. Use `rubric-eval` for semantic judgments about the documentation and Tier 3 for observed agent behavior.<Tip>If a quality score changes unexpectedly, inspect the finding message rather than adding keywords. The checker rewards usable instructions and evidence in their relevant context, not vocabulary alone.diff --git a/pyproject.toml b/pyproject.tomlindex 621da630..302b43fa 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -40,6 +40,7 @@ dependencies = ["click>=8.3.3,<9","idna>=3.10,<4","jinja2>=3.1",+ "markdown-it-py>=4.2,<5","pydantic>=2.11.0","pyyaml>=6.0","rich>=13.5.0",diff --git a/src/skillevaluator/validators/frontmatter_parser.py b/src/skillevaluator/validators/frontmatter_parser.pyindex 5004bf97..9c8f1aa4 100644--- a/src/skillevaluator/validators/frontmatter_parser.py+++ b/src/skillevaluator/validators/frontmatter_parser.py@@ -17,7 +17,10 @@ import yamlfrom skillevaluator.validators.base import ValidationResult# Regex pattern for extracting frontmatter between --- markers-FRONTMATTER_PATTERN = re.compile(r"^---\s*\n(.*?)\n---\s*\n?(.*)", re.DOTALL)+FRONTMATTER_PATTERN = re.compile(+ r"^---[^\S\r\n]*\r?\n(.*?)\r?\n---[^\S\r\n]*(?=\r?\n|\Z)(?:\r?\n)?(.*)",+ re.DOTALL,+)@dataclassdiff --git a/src/skillevaluator/validators/markdown.py b/src/skillevaluator/validators/markdown.pynew file mode 100644index 00000000..fc8769d5--- /dev/null+++ b/src/skillevaluator/validators/markdown.py@@ -0,0 +1,180 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Structural Markdown helpers for deterministic validators."""++import re+from collections.abc import Iterable, Iterator+from html import escape+from html.parser import HTMLParser++import yaml+from markdown_it import MarkdownIt+from markdown_it.token import Token++from skillevaluator.validators.frontmatter_parser import FRONTMATTER_PATTERN++_MARKDOWN_PARSER = MarkdownIt("commonmark", {"html": True, "linkify": False})+_TEXT_ONLY_HTML_TAGS = frozenset(+ {"script", "style", "textarea", "title", "xmp", "iframe", "noembed", "noframes", "plaintext"}+)+_INERT_HTML_TAGS = frozenset({"template"})+_NON_NAVIGATION_HTML_TAGS = _TEXT_ONLY_HTML_TAGS | _INERT_HTML_TAGS+_SCRIPT_START_RE = re.compile(r"<script(?=[\t\n\r\f />])", re.IGNORECASE | re.ASCII)+_SCRIPT_END_RE = re.compile(r"</script(?=[\t\n\r\f />])", re.IGNORECASE | re.ASCII)+_SCRIPT_ESCAPED_STATES = frozenset({"escaped", "escaped-dash", "escaped-dash-dash"})+_SCRIPT_DOUBLE_ESCAPED_STATES = frozenset({"double-escaped", "double-escaped-dash", "double-escaped-dash-dash"})+++def _find_script_end(content: str, start: int) -> int | None:+ state = "data"+ position = start+ while position < len(content):+ character = content[position]+ if state == "data":+ if content.startswith("<!--", position):+ state = "escaped-dash-dash"+ position += 4+ continue+ if _SCRIPT_END_RE.match(content, position):+ return position+ elif state in _SCRIPT_ESCAPED_STATES:+ if character == "<":+ if _SCRIPT_END_RE.match(content, position):+ return position+ if _SCRIPT_START_RE.match(content, position):+ state = "double-escaped"+ position += len("<script")+ continue+ state = "escaped"+ elif state == "escaped":+ if character == "-":+ state = "escaped-dash"+ elif state == "escaped-dash":+ state = "escaped-dash-dash" if character == "-" else "escaped"+ elif character == ">":+ state = "data"+ elif character != "-":+ state = "escaped"+ elif state in _SCRIPT_DOUBLE_ESCAPED_STATES:+ if character == "<" and _SCRIPT_END_RE.match(content, position):+ state = "escaped"+ position += len("</script")+ continue+ if character == "<":+ state = "double-escaped"+ elif state == "double-escaped":+ if character == "-":+ state = "double-escaped-dash"+ elif state == "double-escaped-dash":+ state = "double-escaped-dash-dash" if character == "-" else "double-escaped"+ elif character == ">":+ state = "data"+ elif character != "-":+ state = "double-escaped"+ position += 1+ return None+++class _ScriptEndSearch:+ def search(self, content: str, start: int = 0) -> re.Match[str] | None:+ position = _find_script_end(content, start)+ return None if position is None else _SCRIPT_END_RE.match(content, position)+++_SCRIPT_END_SEARCH = _ScriptEndSearch()+++class _AnchorHrefParser(HTMLParser):+ """Collect string href values from HTML anchor start tags."""++ CDATA_CONTENT_ELEMENTS = ("script", "style", "xmp", "iframe", "noembed", "noframes")+ RCDATA_CONTENT_ELEMENTS = ("textarea", "title")++ def __init__(self) -> None:+ super().__init__(convert_charrefs=True)+ self.targets: list[str] = []+ self._non_navigation_tags: list[str] = []++ def set_cdata_mode(self, elem: str, *, escapable: bool = False) -> None:+ super().set_cdata_mode(elem, escapable=escapable)+ if elem == "script":+ self.interesting = _SCRIPT_END_SEARCH++ @property+ def suppresses_navigation(self) -> bool:+ return bool(self._non_navigation_tags)++ def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:+ # HTMLParser has already normalized ASCII tag and attribute names to+ # lowercase. Unicode case folding would incorrectly turn unknown names+ # containing U+017F into reserved HTML names such as ``noframes``.+ normalized_tag = tag+ if self.suppresses_navigation:+ if (+ self._non_navigation_tags[-1] not in _TEXT_ONLY_HTML_TAGS+ and normalized_tag in _NON_NAVIGATION_HTML_TAGS+ ):+ self._non_navigation_tags.append(normalized_tag)+ return+ if normalized_tag in _NON_NAVIGATION_HTML_TAGS:+ self._non_navigation_tags.append(normalized_tag)+ return+ if normalized_tag != "a":+ return+ for name, value in attrs:+ if name == "href" and isinstance(value, str):+ self.targets.append(value)+ break++ def handle_startendtag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:+ normalized_tag = tag+ if normalized_tag in _NON_NAVIGATION_HTML_TAGS:+ self.handle_starttag(tag, attrs)+ if normalized_tag in self.CDATA_CONTENT_ELEMENTS:+ self.set_cdata_mode(normalized_tag)+ elif normalized_tag in self.RCDATA_CONTENT_ELEMENTS:+ self.set_cdata_mode(normalized_tag, escapable=True)+ return+ super().handle_startendtag(tag, attrs)++ def handle_endtag(self, tag: str) -> None:+ if not self._non_navigation_tags or self._non_navigation_tags[-1] == "plaintext":+ return+ if tag == self._non_navigation_tags[-1]:+ self._non_navigation_tags.pop()+++def _walk_tokens(tokens: Iterable[Token]) -> Iterator[Token]:+ """Yield tokens and their descendants in document order."""+ for token in tokens:+ yield token+ if token.children and token.type != "image":+ yield from _walk_tokens(token.children)+++def markdown_link_targets(content: str) -> list[str]:+ """Return parser-normalized Markdown link destinations in document order."""+ frontmatter = FRONTMATTER_PATTERN.match(content)+ markdown_content = content+ if frontmatter:+ try:+ frontmatter_data = yaml.safe_load(frontmatter.group(1))+ except yaml.YAMLError:+ frontmatter_data = None+ if isinstance(frontmatter_data, dict):+ markdown_content = frontmatter.group(2)++ html_parts: list[str] = []+ for token in _walk_tokens(_MARKDOWN_PARSER.parse(markdown_content)):+ if token.type == "link_open":+ href = token.attrGet("href")+ if isinstance(href, str):+ html_parts.append(f'<a href="{escape(href, quote=True)}"></a>')+ elif token.type in {"html_inline", "html_block"}:+ html_parts.append(token.content)++ html_parser = _AnchorHrefParser()+ html_parser.feed("".join(html_parts))+ html_parser.close()+ return html_parser.targetsdiff --git a/src/skillevaluator/validators/quality_score.py b/src/skillevaluator/validators/quality_score.pyindex f7285426..c43105f7 100644--- a/src/skillevaluator/validators/quality_score.py+++ b/src/skillevaluator/validators/quality_score.py@@ -22,6 +22,7 @@ import posixpathimport refrom collections.abc import Iterablefrom pathlib import Path+from urllib.parse import unquote, urlsplitimport yaml@@ -36,90 +37,17 @@ from skillevaluator.models.quality import QualityScoreResultfrom skillevaluator.models.result import Finding, Severity, ValidationResultfrom skillevaluator.models.skill import XML_TAG_REfrom skillevaluator.validators.base import ValidatorBase+from skillevaluator.validators.markdown import markdown_link_targetslogger = get_logger(__name__)_WORD_CHAR = r"A-Za-z0-9_"-_MARKDOWN_LINK_START_RE = re.compile(r"\[[^\]\n]*\]\(\s*")-_MARKDOWN_LINK_CLOSER_RE = re.compile(r"^\s*(?:(?:\"[^\"\n]*\"|'[^'\n]*'|\([^\)\n]*\))\s*)?\)")-_MARKDOWN_REFERENCE_DEFINITION_RE = re.compile(- r"^[ \t]{0,3}\[([^\]\n]+)\]:[ \t]*(?:<([^>\n]*)>|([^ \t\r\n]+))",- re.MULTILINE,-)-_MARKDOWN_REFERENCE_LINK_RE = re.compile(r"\[([^\]\n]+)\]\[([^\]\n]*)\]")-_MARKDOWN_SHORTCUT_REFERENCE_RE = re.compile(r"(?<![!\]])\[([^\]\n]+)\](?![\(\[])")+_URL_EDGE_C0_OR_SPACE_RE = re.compile(r"^[\x00-\x20]+|[\x00-\x20]+$")_ERROR_HANDLING_RE = re.compile(r"\b(?:errors?|exceptions?|invalid|fail(?:s|ed|ure|ures|ing)?|"r"validat(?:e|es|ed|ing|ion|ions))\b",re.IGNORECASE,)-_MCP_RE = re.compile(r"\bmcp\b", re.IGNORECASE)-_NEGATED_MCP_RES = (- re.compile(- r"\b(?:(?:does|do|did|should|must|shall|can|could|would)\s+not|"- r"(?:doesn't|don't|didn't|shouldn't|mustn't|shan't|can't|couldn't|wouldn't)|never)\s+"- r"(?:\w+\s+){0,3}mcp\b",- re.IGNORECASE,- ),- re.compile(- r"\b(?:(?:is|are|was|were)\s+not|(?:isn't|aren't|wasn't|weren't))\s+"- r"(?:(?:currently|actively)\s+){0,2}"- r"(?:using|requiring|depending\s+on|relying\s+on)\s+(?:an?\s+|the\s+)?mcp\b",- re.IGNORECASE,- ),- re.compile(r"\bwithout\s+(?:an?\s+)?mcp\b", re.IGNORECASE),- re.compile(r"\b(?:no|not\s+(?:an?\s+)?)mcp\b", re.IGNORECASE),- re.compile(- r"\bmcp\b(?:\s+(?:use|usage|support|integration|access|capability|server))?\s+(?:"- r"(?:(?:is|are|was|were)\s+not|(?:isn't|aren't|wasn't|weren't))|"- r"(?:(?:should|must|shall|can|could|would)\s+not|"- r"(?:shouldn't|mustn't|shan't|can't|couldn't|wouldn't))\s+be"- r")\s+"- r"(?:used|required|needed|enabled|supported|involved|necessary|available)\b",- re.IGNORECASE,- ),- re.compile(- r"\b(?:avoid(?:s|ed|ing)?|disabl(?:e|es|ed|ing)|exclud(?:e|es|ed|ing))\s+"- r"(?:(?:using|relying\s+on|depending\s+on)\s+)?(?:an?\s+|the\s+)?mcp\b"- r"(?!\s+(?:errors?|failures?|timeouts?|issues?|problems?|disconnects?|outages?))",- re.IGNORECASE,- ),- re.compile(- r"\bmcp\b(?:\s+(?:support|integration|access|capability|server))?\s+"- r"(?:is|are|was|were|remains?|stays?)\s+"- r"(?:disabled|excluded|unavailable|unsupported)\b",- re.IGNORECASE,- ),- re.compile(r"\bmcp[-\s]+free\b", re.IGNORECASE),-)-_MCP_GUIDANCE_RES = (- re.compile(r"\bconnect(?:s|ed|ing|ion|ions)?\b", re.IGNORECASE),- re.compile(r"\breconnect(?:s|ed|ing|ion|ions)?\b", re.IGNORECASE),- re.compile(r"\bretr(?:y|ies|ied|ying)\b", re.IGNORECASE),- re.compile(r"\btimeouts?\b", re.IGNORECASE),- re.compile(r"\bserver\b[^\n.!?]{0,80}\brunning\b", re.IGNORECASE),- re.compile(r"\bapi\b[^\n.!?]{0,40}\bkeys?\b", re.IGNORECASE),-)-_MARKDOWN_H2_RE = re.compile(r"^##[ \t]+(.+?)[ \t]*\r?$", re.MULTILINE)-_MCP_SUPPORT_HEADINGS = frozenset({"troubleshooting", "common issues", "faq"})-_MCP_SUPPORT_SUBJECT_RE = re.compile(r"\b(?:connections?|sessions?|api\s+keys?)\b", re.IGNORECASE)-_TIME_REFERENCE_RE = re.compile(- r"\b(?:before|after|as of|until)\s+(?P<explicit_year>the\s+year\s+)?(?:19\d{2}|2\d{3})\b",- re.IGNORECASE,-)-_NON_TEMPORAL_COUNT_RE = re.compile(- r"^\s+(?:[A-Za-z][A-Za-z0-9_-]*s|bytes?|people|children|data\s+points?)\b",- re.IGNORECASE,-)-_EXCLUSIVITY_RE = re.compile(- r"\breplaces\s+all\s+"- r"(?P<modifiers>(?:(?:[A-Za-z0-9]+(?:-[A-Za-z0-9]+)*)\s+){0,6})"- r"(?:tools|skills|alternatives|solutions|approaches)\b",- re.IGNORECASE,-)-_NON_EXCLUSIVE_REPLACEMENT_MODIFIERS = frozenset(- {"deprecated", "legacy", "obsolete", "old", "removed", "retired", "superseded"}-)def _contains_term(text: str, term: str) -> bool:@@ -152,251 +80,36 @@ def _has_api_documentation(content: str) -> bool:return any(re.search(pattern, content, re.IGNORECASE) for pattern in api_patterns)-def _without_fenced_code(content: str) -> str:- """Mask fenced Markdown code while preserving line boundaries."""- if "```" not in content and "~~~" not in content:- return content-- visible = []- fence_char = ""- fence_length = 0- for line in content.splitlines(keepends=True):- body = line.rstrip("\r\n")- line_ending = line[len(body) :]- leading_spaces = len(body) - len(body.lstrip(" "))- marker = None- if leading_spaces <= 3:- stripped = body[leading_spaces:]- if stripped.startswith(("`", "~")):- char = stripped[0]- run_length = len(stripped) - len(stripped.lstrip(char))- remainder = stripped[run_length:]- if run_length >= 3 and (char != "`" or "`" not in remainder):- marker = (char, run_length, remainder)-- if not fence_char:- if marker is None:- visible.append(line)- continue- fence_char, fence_length, _ = marker- elif marker is not None:- char, run_length, remainder = marker- if char == fence_char and run_length >= fence_length and not remainder.strip():- fence_char = ""- fence_length = 0- visible.append(line_ending)- return "".join(visible)---def _without_html_comments(content: str) -> str:- """Mask HTML comments while preserving line boundaries."""- if "<!--" not in content:- return content-- visible = []- cursor = 0- while True:- comment_start = content.find("<!--", cursor)- if comment_start == -1:- visible.append(content[cursor:])- break- visible.append(content[cursor:comment_start])- closing_marker = content.find("-->", comment_start + 4)- comment_end = len(content) if closing_marker == -1 else closing_marker + 3- visible.append("".join(char if char in "\r\n" else " " for char in content[comment_start:comment_end]))- cursor = comment_end- return "".join(visible)---def _markdown_link_targets(content: str) -> list[str]:- """Extract inline and reference-style Markdown link targets."""- content = _without_html_comments(_without_fenced_code(content))- targets = []- skip_until = 0- for match in _MARKDOWN_LINK_START_RE.finditer(content):- if match.start() < skip_until:- continue- cursor = match.end()- if cursor >= len(content):- continue- if content[cursor] == "<":- end = content.find(">", cursor + 1)- if end != -1 and _MARKDOWN_LINK_CLOSER_RE.match(content[end + 1 :]):- targets.append(content[cursor + 1 : end])- elif end == -1:- skip_until = len(content)- else:- skip_until = end + 1- continue-- chars = []- depth = 0- closed = False- while cursor < len(content):- char = content[cursor]- if char == "\\" and cursor + 1 < len(content):- chars.append(content[cursor + 1])- cursor += 2- continue- if char == "(":- depth += 1- elif char == ")":- if depth == 0:- closed = True- break- depth -= 1- elif char.isspace():- if depth == 0:- closed = _MARKDOWN_LINK_CLOSER_RE.match(content[cursor:]) is not None- break- chars.append(char)- cursor += 1- if chars and depth == 0 and closed:- targets.append("".join(chars))- elif not closed:- skip_until = max(skip_until, cursor)-- definitions: dict[str, str] = {}- definition_label_starts: set[int] = set()- for match in _MARKDOWN_REFERENCE_DEFINITION_RE.finditer(content):- label = re.sub(r"\s+", " ", match.group(1).strip()).casefold()- target = match.group(2) if match.group(2) is not None else match.group(3)- definitions.setdefault(label, target)- definition_label_starts.add(match.start(1))-- for match in _MARKDOWN_REFERENCE_LINK_RE.finditer(content):- label = match.group(2) or match.group(1)- normalized = re.sub(r"\s+", " ", label.strip()).casefold()- if normalized in definitions:- targets.append(definitions[normalized])-- for match in _MARKDOWN_SHORTCUT_REFERENCE_RE.finditer(content):- if match.start(1) in definition_label_starts:- continue- normalized = re.sub(r"\s+", " ", match.group(1).strip()).casefold()- if normalized in definitions:- targets.append(definitions[normalized])- return targets---def _is_other_cli_mcp_comparison(sentence: str, mcp_end: int, skill_name: str | None) -> bool:- """Return whether a sentence contrasts other CLIs' MCP use with this skill."""- if not skill_name:- return False-- before_mcp = sentence[:mcp_end]- if not re.search(- r"\b[a-z0-9]+(?:-[a-z0-9]+)*-cli\b[^.!?]{0,80}\buses?\b[^.!?]{0,80}\bmcp\b",- before_mcp,- re.IGNORECASE,- ):- return False-- after_mcp = sentence[mcp_end:]- if re.search(r"\bnot\s+(?:this|the\s+current)\s+skill\b", after_mcp, re.IGNORECASE):- return True-- name_parts = [part for part in re.split(r"[-_\s]+", skill_name) if part]- aliases = (" ".join(name_parts[index:]) for index in range(max(1, len(name_parts) - 1)))- return any(- re.search(rf"\bnot\s+{re.escape(alias)}\b", after_mcp, re.IGNORECASE)- for alias in aliases- if len(alias.split()) >= 2- )---def _mcp_usage_contexts(content: str, skill_name: str | None = None) -> list[str]:- """Return sentences where MCP is used as a capability rather than negated."""- content = _without_html_comments(_without_fenced_code(content))- contexts = []- seen_contexts = set()- for paragraph in re.split(r"\n\s*\n", content):- for match in _MCP_RE.finditer(paragraph):- sentence_start = max(paragraph.rfind(mark, 0, match.start()) for mark in ".!?") + 1- sentence_ends = [paragraph.find(mark, match.end()) for mark in ".!?"]- sentence_end = min((end for end in sentence_ends if end != -1), default=len(paragraph))- sentence = paragraph[sentence_start:sentence_end]- relative_mcp_start = match.start() - sentence_start- is_negated = any(- negated.start() <= relative_mcp_start < negated.end()- for pattern in _NEGATED_MCP_RES- for negated in pattern.finditer(sentence)- )- is_other_cli_comparison = _is_other_cli_mcp_comparison(- sentence,- match.end() - sentence_start,- skill_name,- )- if not is_negated and not is_other_cli_comparison and sentence not in seen_contexts:- contexts.append(sentence)- seen_contexts.add(sentence)- return contexts---def _markdown_h2_sections(content: str) -> Iterable[tuple[str, str]]:- """Yield H2 headings and bodies without a backtracking multi-line pattern."""- content = _without_html_comments(_without_fenced_code(content))- matches = list(_MARKDOWN_H2_RE.finditer(content))- for index, match in enumerate(matches):- body_start = match.end()- if body_start < len(content) and content[body_start] == "\n":- body_start += 1- body_end = matches[index + 1].start() if index + 1 < len(matches) else len(content)- yield match.group(1).strip(), content[body_start:body_end]---def _has_mcp_guidance(content: str, usage_contexts: list[str]) -> bool:- """Return whether usage or a clearly MCP-related support section has guidance."""- if any(pattern.search(context) for context in usage_contexts for pattern in _MCP_GUIDANCE_RES):- return True- previous_heading = ""- previous_section = ""- for heading, section in _markdown_h2_sections(content):- normalized_heading = heading.casefold()- heading_mentions_mcp = bool(_MCP_RE.search(heading))- is_support_section = normalized_heading in _MCP_SUPPORT_HEADINGS or (- heading_mentions_mcp and any(label in normalized_heading for label in _MCP_SUPPORT_HEADINGS)- )- has_guidance = any(pattern.search(section) for pattern in _MCP_GUIDANCE_RES)- explicitly_mcp_related = heading_mentions_mcp or bool(_MCP_RE.search(section))- follows_mcp_section = bool(- _MCP_RE.search(previous_heading)- and _mcp_usage_contexts(previous_section)- and _MCP_SUPPORT_SUBJECT_RE.search(section)- )- if is_support_section and has_guidance and (explicitly_mcp_related or follows_mcp_section):- return True- previous_heading = heading- previous_section = section- return False---def _has_exclusive_replacement(content: str) -> bool:- """Return whether content claims to replace a category of composable tooling."""- for match in _EXCLUSIVITY_RE.finditer(content):- modifiers = set(re.findall(r"[A-Za-z0-9]+(?:-[A-Za-z0-9]+)*", match.group("modifiers").lower()))- if not modifiers.intersection(_NON_EXCLUSIVE_REPLACEMENT_MODIFIERS):- return True- return False-+def _normalized_local_path(href: str) -> str | None:+ """Return a once-decoded, normalized local file path from a link destination."""+ href = _URL_EDGE_C0_OR_SPACE_RE.sub("", href)+ try:+ parsed = urlsplit(href)+ except ValueError:+ return None+ if parsed.scheme or parsed.netloc:+ return None-def _has_time_reference(content: str) -> bool:- for match in _TIME_REFERENCE_RE.finditer(content):- if match.group("explicit_year") or not _NON_TEMPORAL_COUNT_RE.match(content[match.end() : match.end() + 32]):- return True- return False+ path = unquote(parsed.path).replace("\\", "/")+ if path.startswith("/"):+ return None+ try:+ if urlsplit(path).scheme:+ return None+ except ValueError:+ return None+ if path.endswith(("/", "/.", "/..")):+ return None+ return posixpath.normpath(path)def _has_nested_markdown_reference(content: str) -> bool:"""Return whether a reference document links to another local Markdown document."""- for target in _markdown_link_targets(content):- path = re.split(r"[?#]", target, maxsplit=1)[0].replace("\\", "/")- lowered = path.lower()- if not lowered.endswith(".md"):- continue- if re.match(r"^[a-z][a-z0-9+.-]*:", target, re.IGNORECASE) or path.startswith(("//", "/")):+ for target in markdown_link_targets(content):+ path = _normalized_local_path(target)+ if path is None or not path.casefold().endswith(".md"):continue- if posixpath.normpath(path).lower() == "../skill.md":+ if path.casefold() == "../skill.md":continuereturn Truereturn False@@ -669,44 +382,10 @@ class QualityScoreValidator(ValidatorBase):@staticmethoddef _references_readme(content: str) -> bool:- """Return True if SKILL.md points agents at a README.md.-- Markdown links and explicit instructions to read/open/load the file count.- Merely naming README.md, including negative guidance not to load it, does- not pull the file into agent context.- """- content = _without_html_comments(_without_fenced_code(content))- for link_target in _markdown_link_targets(content):- target = re.split(r"[?#]", link_target, maxsplit=1)[0].replace("\\", "/")- if posixpath.basename(posixpath.normpath(target)).lower() == "readme.md":- return True-- action_re = re.compile(- r"\b(?:read|open|load|consult|review|see|use|follow|refer\s+to)\b"- r"[^\n.!?]{0,40}\breadme\.md\b",- re.IGNORECASE,- )- negated_action_re = re.compile(- r"\b(?:(?:(?:do|does|did|should|must|shall|can|could|would)\s+not|"- r"(?:don't|doesn't|didn't|shouldn't|mustn't|shan't|can't|couldn't|wouldn't))"- r"(?:\s+need\s+to)?|"- r"(?:is|are|was|were)\s+not\s+(?:allowed|permitted|required|expected)\s+to|"- r"cannot|never|not\s+to|need\s+not|no\s+need\s+to)\s+"- r"(?:(?:ever|directly|automatically|accidentally|normally)\s+){0,2}"- r"(?:read|open|load|consult|review|see|use|follow|refer\s+to)\b"- r"[^\n.!?]{0,40}?\breadme\.md\b",- re.IGNORECASE,- )- passive_action_re = re.compile(- r"\breadme\.md\b[^\n.!?]{0,40}"- r"\b(?:should|must|needs?\s+to|is\s+required\s+to)\s+be\s+"- r"(?:read|opened|loaded|consulted|reviewed|followed|used)\b",- re.IGNORECASE,- )- normalized_content = re.sub(r"\s+", " ", content)- for sentence in re.split(r"(?<=[.!?])\s+", normalized_content):- affirmative_content = negated_action_re.sub(" ", sentence)- if action_re.search(affirmative_content) or passive_action_re.search(affirmative_content):+ """Return whether SKILL.md contains a parsed Markdown link to README.md."""+ for link_target in markdown_link_targets(content):+ path = _normalized_local_path(link_target)+ if path is not None and path.casefold() == "readme.md":return Truereturn False@@ -922,21 +601,6 @@ class QualityScoreValidator(ValidatorBase):"Add boundary phrases like 'Do NOT use for...'",)- # Exclusivity language (M5)- exclusivity = [- "always use this skill",- "the only way to",- "do not use any other",- "this skill handles everything",- ]- if _contains_any_term(content, exclusivity) or _has_exclusive_replacement(content):- dim.deduct(- 5,- "info",- "Skill uses exclusivity language that conflicts with composability",- "Skills should work alongside others (composability principle)",- )-if "## Purpose" not in content:dim.deduct(5, "info", "No '## Purpose' section", "Add purpose section to clarify use cases")@@ -1007,16 +671,6 @@ class QualityScoreValidator(ValidatorBase):"Add '## Troubleshooting' with Error/Cause/Solution patterns",)- # MCP connection guidance (M2)- mcp_contexts = _mcp_usage_contexts(content, qs.skill_name)- if mcp_contexts and not _has_mcp_guidance(content, mcp_contexts):- dim.deduct(- 10,- "warning",- "MCP skill lacks connection/error guidance",- "Add MCP troubleshooting: connection verification, retry logic",- )-def _check_script_reliability(self, dim, skill_path: Path) -> None:scripts_dir = skill_path / "scripts"if not scripts_dir.exists():@@ -1158,15 +812,6 @@ class QualityScoreValidator(ValidatorBase):"Use simple, direct language: 'use' not 'utilize'",)- # Time-sensitive info- if _has_time_reference(content):- dim.deduct(- 5,- "info",- "Time-sensitive information detected",- "Avoid dates that become outdated; use 'old patterns' section",- )-# Reference file namingrefs_dir = skill_path / "references"if refs_dir.exists():diff --git a/tests/validators/test_frontmatter_parser.py b/tests/validators/test_frontmatter_parser.pyindex 43f0882a..9baf27b4 100644--- a/tests/validators/test_frontmatter_parser.py+++ b/tests/validators/test_frontmatter_parser.py@@ -5,6 +5,7 @@from pathlib import Path+import pytestfrom pydantic import BaseModelfrom skillevaluator.validators.base import ValidationResult@@ -30,6 +31,18 @@ description: A test fileassert parsed.yaml_data["title"] == "Test"assert parsed.content.strip() == "# Content here"+ @pytest.mark.parametrize("newline", ["\n", "\r\n"], ids=["lf", "crlf"])+ def test_first_body_line_indentation_is_preserved(self, tmp_path: Path, newline: str):+ test_file = tmp_path / "indented-body.mdc"+ content = newline.join(["---", "title: Test", "---", " [code](false.md)", ""])+ test_file.write_bytes(content.encode())++ parsed, result = parse_frontmatter(test_file)++ assert parsed is not None+ assert result.passed+ assert parsed.content == " [code](false.md)\n"+def test_missing_frontmatter(self, tmp_path: Path):"""Test file without frontmatter markers."""test_file = tmp_path / "test.mdc"diff --git a/tests/validators/test_markdown.py b/tests/validators/test_markdown.pynew file mode 100644index 00000000..07e58c1d--- /dev/null+++ b/tests/validators/test_markdown.py@@ -0,0 +1,244 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Tests for structural Markdown helpers used by deterministic validators."""++import pytest++from skillevaluator.validators import markdown as markdown_validator+from skillevaluator.validators.markdown import markdown_link_targets++_CLOSED_NON_NAVIGATION_HTML_TAGS = (+ "script",+ "style",+ "textarea",+ "title",+ "xmp",+ "iframe",+ "noembed",+ "noframes",+ "template",+ "XmP",+)+++ "content",+ [+ "`[example](advanced.md)`",+ " [example](advanced.md)",+ "<span data-doc='[example](advanced.md)'>text</span>",+ "<span href='not-a-link.md'>text</span>",+ "<!-- <a href='hidden.md'>hidden</a> -->",+ "",+ "![diagram][guide]\n\n[guide]: advanced.md",+ "](diagram.png)",+ "> ```markdown\n> [example](advanced.md)\n> ```",+ ],+)+def test_non_navigation_markdown_does_not_produce_links(content: str) -> None:+ assert markdown_link_targets(content) == []+++ ("content", "target"),+ [+ ("[guide](inline.md)", "inline.md"),+ ("[outer [inner]](nested.md)", "nested.md"),+ (r"[escaped \] label](escaped.md)", "escaped.md"),+ ("[reference][guide]\n\n[guide]: reference.md", "reference.md"),+ ("[reference][guide\\]]\n\n[guide\\]]: escaped-ref.md", "escaped-ref.md"),+ ("[guide][]\n\n[guide]: collapsed.md", "collapsed.md"),+ ("[guide]\n\n[guide]: shortcut.md", "shortcut.md"),+ ('[guide](<my file.md> "Title")', "my%20file.md"),+ ("[wrapped\nlabel](multiline.md)", "multiline.md"),+ ],+)+def test_commonmark_links_are_extracted(content: str, target: str) -> None:+ assert markdown_link_targets(content) == [target]+++ ("content", "target"),+ [+ ('<a href="README.md">docs</a>', "README.md"),+ ('<A HREF="upper.md">docs</A>', "upper.md"),+ ("<div>\n<a href='block.md'>docs</a>\n</div>", "block.md"),+ ],+)+def test_html_anchor_links_are_extracted(content: str, target: str) -> None:+ assert markdown_link_targets(content) == [target]+++def test_markdown_and_html_links_preserve_source_order() -> None:+ content = '[first](first.md) <a href="second.md">second</a> [third](third.md)'++ assert markdown_link_targets(content) == ["first.md", "second.md", "third.md"]++[email protected]("tag", _CLOSED_NON_NAVIGATION_HTML_TAGS)+def test_non_navigation_html_suppresses_links_and_then_restores_navigation(tag: str) -> None:+ content = f'prefix <{tag}><a href="hidden.md">[hidden](also-hidden.md)</a></{tag}> '+ content += '<a href="visible.md">visible</a> [last](last.md)'++ assert markdown_link_targets(content) == ["visible.md", "last.md"]++[email protected]("tag", _CLOSED_NON_NAVIGATION_HTML_TAGS)+def test_self_closing_syntax_does_not_close_non_void_html_content(tag: str) -> None:+ content = f'prefix <{tag}/><a href="hidden.md">hidden</a></{tag}> <a href="visible.md">visible</a>'++ assert markdown_link_targets(content) == ["visible.md"]+++def test_nested_template_content_remains_non_navigation_until_outer_close() -> None:+ content = (+ '<template><template><a href="hidden.md">hidden</a></template>'+ '<a href="still-hidden.md">still hidden</a></template>'+ '<a href="visible.md">visible</a>'+ )++ assert markdown_link_targets(content) == ["visible.md"]+++def test_shadowroot_template_without_a_valid_host_remains_non_navigation() -> None:+ content = (+ '<a><template shadowrootmode="open"><a href="hidden.md">hidden</a></template></a>'+ '<a href="visible.md">visible</a>'+ )++ assert markdown_link_targets(content) == ["visible.md"]++[email protected]("mode", ["open", "CLOSED"])+def test_shadowroot_template_content_is_conservatively_non_navigation(mode: str) -> None:+ content = (+ f'<div><template shadowrootmode="{mode}"><a href="shadow.md">shadow link</a></template></div>'+ '<a href="visible.md">visible</a>'+ )++ assert markdown_link_targets(content) == ["visible.md"]+++def test_raw_text_pseudo_start_tag_does_not_require_an_extra_close() -> None:+ content = '<xmp><xmp><a href="hidden.md">hidden</a></xmp><a href="visible.md">visible</a>'++ assert markdown_link_targets(content) == ["visible.md"]++[email protected]("outer", ["xmp", "iframe", "noembed", "noframes"])[email protected]("pseudo_inner", ["script", "style", "textarea", "title"])+def test_text_only_pseudo_start_does_not_hijack_outer_state(outer: str, pseudo_inner: str) -> None:+ content = f'<{outer}><{pseudo_inner}></{outer}><a href="visible.md">visible</a>'++ assert markdown_link_targets(content) == ["visible.md"]++[email protected]("tag", ["xmp", "iframe", "noembed", "noframes"])+def test_comment_syntax_does_not_hide_raw_text_end_tag(tag: str) -> None:+ content = f'<{tag}><!-- </{tag}> --><a href="visible.md">visible</a>'++ assert markdown_link_targets(content) == ["visible.md"]++[email protected]("tag", ["xmp", "iframe", "noembed", "noframes"])+def test_self_closing_raw_text_syntax_keeps_native_text_mode(tag: str) -> None:+ content = f'<{tag}/><!-- </{tag}> --><a href="visible.md">visible</a>'++ assert markdown_link_targets(content) == ["visible.md"]+++def test_mismatched_raw_text_close_does_not_restore_navigation() -> None:+ content = '<iframe><a href="hidden.md">hidden</a></xmp><a href="still-hidden.md">still hidden</a>'++ assert markdown_link_targets(content) == []+++def test_plaintext_html_suppresses_navigation_to_end_of_document() -> None:+ content = (+ '<plaintext><a href="hidden.md">[also hidden](also-hidden.md)</a>'+ '</plaintext> <a href="still-hidden.md">still hidden</a> [last](last.md)'+ )++ assert markdown_link_targets(content) == []+++def test_script_double_escaped_content_does_not_produce_links() -> None:+ content = '<script><!--<script></script><a href="hidden.md">hidden</a>--></script><a href="visible.md">visible</a>'++ assert markdown_link_targets(content) == ["visible.md"]++[email protected]("escape_boundary", ["<!-->", "<!--->"])+def test_script_escaped_dash_dash_greater_than_returns_to_data(escape_boundary: str) -> None:+ content = (+ f'<script>{escape_boundary}<script><a href="hidden.md">hidden</a></script><a href="visible.md">visible</a>'+ )++ assert markdown_link_targets(content) == ["visible.md"]+++ "content",+ [+ '<script><!--<\u017fcript></script><a href="visible.md">visible</a>',+ '<script></\u017fcript><a href="hidden.md">hidden</a></script><a href="visible.md">visible</a>',+ ],+)+def test_script_end_scanner_uses_ascii_case_insensitive_matching(content: str) -> None:+ assert markdown_link_targets(content) == ["visible.md"]+++def test_html_tag_matching_does_not_casefold_non_ascii_characters() -> None:+ content = '<noframe\u017f><a href="inside.md">inside</a></noframe\u017f><a href="outside.md">outside</a>'++ assert markdown_link_targets(content) == ["inside.md", "outside.md"]+++def test_many_inline_html_tokens_are_fed_to_html_parser_once(monkeypatch: pytest.MonkeyPatch) -> None:+ feed_calls = 0+ original_feed = markdown_validator._AnchorHrefParser.feed++ def counted_feed(parser: markdown_validator._AnchorHrefParser, content: str) -> None:+ nonlocal feed_calls+ feed_calls += 1+ original_feed(parser, content)++ monkeypatch.setattr(markdown_validator._AnchorHrefParser, "feed", counted_feed)+ content = "prefix <xmp>" + ' <a href="hidden.md">hidden</a>' * 100++ assert markdown_link_targets(content) == []+ assert feed_calls == 1+++def test_escaped_space_destination_does_not_produce_link() -> None:+ assert markdown_link_targets(r"[guide](my\ file.md)") == []+++def test_yaml_frontmatter_links_are_ignored() -> None:+ content = (+ '---\ndescription: "See [metadata](frontmatter.md) for details."\n---\n\nSee [body](body.md) for details.\n'+ )++ assert markdown_link_targets(content) == ["body.md"]+++def test_thematic_breaks_do_not_hide_markdown_links() -> None:+ content = "---\nSee [guide](advanced.md).\n---\n"++ assert markdown_link_targets(content) == ["advanced.md"]++[email protected]("newline", ["\n", "\r\n"], ids=["lf", "crlf"])+def test_frontmatter_does_not_promote_indented_first_body_line(newline: str) -> None:+ content = newline.join(["---", "title: Test", "---", " [code](false.md)", ""])++ assert markdown_link_targets(content) == []+++def test_malformed_80_kib_input_does_not_produce_links() -> None:+ """Malformed link syntax must not manufacture navigation destinations."""+ content = ("[broken](" * 10_000)[: 80 * 1024]++ assert len(content) == 80 * 1024+ assert markdown_link_targets(content) == []diff --git a/tests/validators/test_quality_score.py b/tests/validators/test_quality_score.pyindex cc9708ee..89f62151 100644--- a/tests/validators/test_quality_score.py+++ b/tests/validators/test_quality_score.py@@ -7,7 +7,6 @@ from pathlib import Pathimport pytest-import skillevaluator.validators.quality_score as quality_score_modulefrom skillevaluator.models.quality import QualityScoreResult, score_to_gradefrom skillevaluator.validators.quality_score import QualityScoreValidator@@ -479,7 +478,7 @@ class TestQualityScoreValidator:assert "Correctness" in v.description-class TestQualityScoreKeywordBoundaries:+class TestQualityScoreDeterministicContracts:@pytest.mark.parametrize("description",[@@ -561,26 +560,14 @@ class TestQualityScoreKeywordBoundaries:assert result.metadata["quality_scores"]["metrics"]["has_error_handling"] is Trueassert "No mention of error handling or validation" not in [finding.message for finding in result.findings]- def test_negated_mcp_mention_does_not_classify_mcp_skill(self, tmp_path: Path):- messages = _finding_messages(_write_issue_skill(tmp_path, extra_body="\nThis skill does not use MCP.\n"))-- assert "MCP skill lacks connection/error guidance" not in messages-- def test_postposed_negated_mcp_mention_does_not_classify_mcp_skill(self, tmp_path: Path):- messages = _finding_messages(_write_issue_skill(tmp_path, extra_body="\nMCP is not used by this skill.\n"))-- assert "MCP skill lacks connection/error guidance" not in messages-@pytest.mark.parametrize("statement",[- "This skill is not using MCP.",- "This workflow isn't currently relying on MCP.",- "MCP usage is not supported.",- "MCP access isn't available.",+ "This skill lacks MCP support.",+ "Use the MCP server to enumerate tools.",],)- def test_copular_negated_mcp_mention_does_not_classify_mcp_skill(self, tmp_path: Path, statement: str):+ def test_mcp_prose_does_not_trigger_deterministic_finding(self, tmp_path: Path, statement: str):messages = _finding_messages(_write_issue_skill(tmp_path, extra_body=f"\n{statement}\n"))assert "MCP skill lacks connection/error guidance" not in messages@@ -588,391 +575,239 @@ class TestQualityScoreKeywordBoundaries:@pytest.mark.parametrize("statement",[- "This skill should not use MCP.",- "MCP should not be used by this skill.",- ],- )- def test_modal_negated_mcp_mention_does_not_classify_mcp_skill(self, tmp_path: Path, statement: str):- messages = _finding_messages(_write_issue_skill(tmp_path, extra_body=f"\n{statement}\n"))-- assert "MCP skill lacks connection/error guidance" not in messages-- @pytest.mark.parametrize(- "statement",- [- "This skill avoids MCP.",- "This skill avoids using MCP.",- "This workflow excludes the MCP integration.",- "MCP is disabled for this skill.",- "MCP support remains unavailable.",- "This workflow is MCP-free.",- ],- )- def test_negative_mcp_capability_does_not_classify_mcp_skill(self, tmp_path: Path, statement: str):- messages = _finding_messages(_write_issue_skill(tmp_path, extra_body=f"\n{statement}\n"))-- assert "MCP skill lacks connection/error guidance" not in messages-- def test_avoiding_mcp_failures_remains_mcp_usage(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(tmp_path, extra_body="\nAvoid MCP failures during validation.\n")- )-- assert "MCP skill lacks connection/error guidance" in messages-- def test_fenced_mcp_example_does_not_classify_mcp_skill(self, tmp_path: Path):- messages = _finding_messages(_write_issue_skill(tmp_path, extra_body="\n```text\nMCP example output\n```\n"))-- assert "MCP skill lacks connection/error guidance" not in messages-- def test_other_cli_mcp_comparison_does_not_classify_current_skill(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(- tmp_path,- name="authenticating-entra-device-code",- extra_body=(- "\nNote: sharepoint-cli, gdrive-cli, and glean-cli use MaaS MCP auth "- "(via nv-discovery-cli), not Entra device code.\n"- ),- )- )-- assert "MCP skill lacks connection/error guidance" not in messages-- def test_current_cli_mcp_comparison_remains_mcp_usage(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(- tmp_path,- name="sharepoint-cli",- extra_body="\nsharepoint-cli uses MaaS MCP auth, not Entra device code.\n",- )- )-- assert "MCP skill lacks connection/error guidance" in messages-- def test_mcp_mention_in_html_comment_does_not_classify_skill(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(- tmp_path,- extra_body=(- "\n<!-- Related skills:\n- managing-sharepoint: SharePoint sites and lists (MaaS MCP)\n-->\n"- ),- )- )-- assert "MCP skill lacks connection/error guidance" not in messages-- def test_mcp_guidance_in_html_comment_does_not_satisfy_active_usage(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(- tmp_path,- extra_body=(- "\nUse the MCP server to enumerate tools.\n\n"- "<!--\n## MCP Troubleshooting\n\n"- "Reconnect the server and retry after a timeout.\n-->\n"- ),- )- )-- assert "MCP skill lacks connection/error guidance" in messages-- def test_interconnect_does_not_satisfy_mcp_connection_guidance(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(tmp_path, extra_body="\nUse the MCP server for GPU interconnect analysis.\n")- )-- assert "MCP skill lacks connection/error guidance" in messages-- def test_unrelated_connection_text_does_not_satisfy_mcp_guidance(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(- tmp_path,- extra_body=(- "\nUse the MCP server to enumerate tools.\n\n"- "## Compilation Cache\n\nConnect to the compilation cache before measuring.\n"- ),- )- )-- assert "MCP skill lacks connection/error guidance" in messages-- def test_unrelated_guidance_in_same_paragraph_does_not_satisfy_mcp_guidance(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(- tmp_path,- extra_body=(- "\n## Usage\n\n"- "- Use the MCP server to enumerate tools.\n"- "- Connect to the compilation cache before measuring.\n"- ),- )- )-- assert "MCP skill lacks connection/error guidance" in messages-- def test_mcp_reconnect_guidance_is_still_accepted(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(- tmp_path,- extra_body="\nUse the MCP server. Reconnect the server if the session expires.\n",- )- )-- assert "MCP skill lacks connection/error guidance" not in messages-- def test_later_explicit_mcp_sentence_can_supply_guidance(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(- tmp_path,- extra_body=(- "\n## Usage\n\n"- "Use the MCP server to enumerate tools. "- "If the MCP server disconnects, reconnect it.\n"- ),- )- )-- assert "MCP skill lacks connection/error guidance" not in messages-- def test_mcp_guidance_in_troubleshooting_section_is_accepted(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(- tmp_path,- extra_body=(- "\n## MCP Usage\n\nUse the MCP server to enumerate tools.\n\n"- "## Troubleshooting\n\nIf the connection drops, reconnect the server and retry.\n"- ),- )- )-- assert "MCP skill lacks connection/error guidance" not in messages-- def test_mcp_guidance_in_explicitly_titled_support_section_is_accepted(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(- tmp_path,- extra_body=(- "\n## MCP Usage\n\nUse the MCP server to enumerate tools.\n\n"- "## MCP Troubleshooting\n\nReconnect the server and retry after a timeout.\n"- ),- )- )-- assert "MCP skill lacks connection/error guidance" not in messages-- def test_unrelated_troubleshooting_does_not_satisfy_mcp_guidance(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(- tmp_path,- extra_body=(- "\n## MCP Usage\n\nUse the MCP server to enumerate tools.\n\n"- "## Troubleshooting\n\nRetry compilation if the local cache is unavailable.\n"- ),- )- )-- assert "MCP skill lacks connection/error guidance" in messages-- def test_unrelated_build_server_troubleshooting_does_not_satisfy_mcp_guidance(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(- tmp_path,- extra_body=(- "\n## MCP Usage\n\nUse the MCP server to enumerate tools.\n\n"- "## Troubleshooting\n\nIf the build server fails, retry compilation.\n"- ),- )- )-- assert "MCP skill lacks connection/error guidance" in messages-- @pytest.mark.parametrize(- "statement",- [- "Unrolling stops after 2048 iterations.",- "Stop after 2048 files.","Retry after 2025 requests.",- "Fail after 2000 data points.",+ "Use this compatibility path after 2025.",],)- def test_four_digit_count_is_not_time_sensitive_information(self, tmp_path: Path, statement: str):+ def test_year_and_count_prose_does_not_trigger_time_finding(self, tmp_path: Path, statement: str):messages = _finding_messages(_write_issue_skill(tmp_path, extra_body=f"\n{statement}\n"))assert "Time-sensitive information detected" not in messages- def test_actual_year_reference_remains_time_sensitive(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(tmp_path, extra_body="\nUse this compatibility path after 2025.\n")- )-- assert "Time-sensitive information detected" in messages-- def test_explicit_year_reference_with_following_plural_remains_time_sensitive(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(tmp_path, extra_body="\nAfter the year 2025, releases use this compatibility path.\n")- )-- assert "Time-sensitive information detected" in messages-- def test_future_year_reference_remains_time_sensitive(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(tmp_path, extra_body="\nUse this compatibility path after 2101.\n")- )-- assert "Time-sensitive information detected" in messages-- def test_replacing_deprecated_calls_is_not_exclusivity(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(tmp_path, extra_body="\nThis release replaces all deprecated launch calls.\n")- )-- assert "Skill uses exclusivity language that conflicts with composability" not in messages-@pytest.mark.parametrize("statement",[- "This release replaces all deprecated tool calls.",- "This migration replaces all legacy build tools.",- "This migration replaces all other deprecated tool calls.",+ "This skill replaces all tools.",+ "This skill replaces all tools retired before v2.",+ "Always use this skill.",+ "This is not the only way to configure the tool.",+ 'Avoid the anti-pattern "do not use any other tool."',+ "`This skill handles everything` is a bad instruction.",],)- def test_replacing_obsolete_implementation_details_is_not_exclusivity(self, tmp_path: Path, statement: str):+ def test_exclusivity_prose_does_not_trigger_deterministic_finding(self, tmp_path: Path, statement: str):messages = _finding_messages(_write_issue_skill(tmp_path, extra_body=f"\n{statement}\n"))assert "Skill uses exclusivity language that conflicts with composability" not in messages- def test_replacing_all_other_tools_remains_exclusivity(self, tmp_path: Path):- messages = _finding_messages(- _write_issue_skill(tmp_path, extra_body="\nThis skill replaces all other build tools.\n")- )-- assert "Skill uses exclusivity language that conflicts with composability" in messages-@pytest.mark.parametrize("statement",[- "This skill replaces all tools.",- "This skill replaces all competing build tools.",- "This skill replaces all alternative skills.",- "This skill replaces all competing third-party tools.",- "This skill replaces all currently available competing build tools.",+ "README.md is human-facing documentation.",+ "Read README.md before publishing.",+ "Do not read README.md before publishing.",],)- def test_replacing_all_tooling_remains_exclusivity(self, tmp_path: Path, statement: str):- messages = _finding_messages(_write_issue_skill(tmp_path, extra_body=f"\n{statement}\n"))-- assert "Skill uses exclusivity language that conflicts with composability" in messages-- def test_readme_negative_guidance_is_not_a_reference(self, tmp_path: Path):- skill_dir = _write_issue_skill(- tmp_path,- extra_body="\nREADME.md is human-facing; do not load it.\n",- )+ def test_readme_prose_does_not_trigger_reference_finding(self, tmp_path: Path, statement: str):+ skill_dir = _write_issue_skill(tmp_path, extra_body=f"\n{statement}\n")(skill_dir / "README.md").write_text("# Human documentation\n")messages = _finding_messages(skill_dir)assert all("SKILL.md references README.md" not in message for message in messages)- def test_readme_load_instruction_is_still_a_reference(self, tmp_path: Path):- skill_dir = _write_issue_skill(tmp_path, extra_body="\nRead README.md before publishing.\n")- (skill_dir / "README.md").write_text("# Human documentation\n")-- messages = _finding_messages(skill_dir)-- assert any("SKILL.md references README.md" in message for message in messages)-- def test_wrapped_readme_negative_guidance_is_not_a_reference(self, tmp_path: Path):- skill_dir = _write_issue_skill(tmp_path, extra_body="\nDo not\nread README.md before publishing.\n")+ def test_markdown_link_to_readme_suffix_is_not_a_readme_reference(self, tmp_path: Path):+ skill_dir = _write_issue_skill(tmp_path, extra_body="\nSee [release notes](NOTREADME.md).\n")(skill_dir / "README.md").write_text("# Human documentation\n")+ (skill_dir / "NOTREADME.md").write_text("# Release notes\n")messages = _finding_messages(skill_dir)assert all("SKILL.md references README.md" not in message for message in messages)- def test_passive_readme_instruction_is_still_a_reference(self, tmp_path: Path):- skill_dir = _write_issue_skill(tmp_path, extra_body="\nREADME.md should be read before publishing.\n")+ @pytest.mark.parametrize(+ "target",+ [+ "https://example.com/README.md",+ "//example.com/README.md",+ "/README.md",+ "../README.md",+ "nested/README.md",+ ],+ )+ def test_nonlocal_readme_links_do_not_trigger_reference_finding(self, tmp_path: Path, target: str):+ skill_dir = _write_issue_skill(tmp_path, extra_body=f"\nSee [documentation]({target}).\n")(skill_dir / "README.md").write_text("# Human documentation\n")messages = _finding_messages(skill_dir)- assert any("SKILL.md references README.md" in message for message in messages)+ assert all("SKILL.md references README.md" not in message for message in messages)- def test_passive_negative_readme_guidance_is_not_a_reference(self, tmp_path: Path):- skill_dir = _write_issue_skill(tmp_path, extra_body="\nREADME.md should not be read by agents.\n")+ @pytest.mark.parametrize(+ "target",+ [+ "README.md/",+ "README.md/.",+ "README.md/child/..",+ "README.md%2F",+ "README.md/%2E",+ "README.md%5C",+ "README.md%2Fchild%2F%2E%2E",+ "README.md/?view=1",+ "README.md/.#overview",+ ],+ )+ def test_directory_form_readme_links_do_not_trigger_reference_finding(self, tmp_path: Path, target: str):+ skill_dir = _write_issue_skill(tmp_path, extra_body=f"\nSee [documentation]({target}).\n")(skill_dir / "README.md").write_text("# Human documentation\n")- messages = _finding_messages(skill_dir)+ result = QualityScoreValidator(min_score=99).validate(skill_dir)- assert all("SKILL.md references README.md" not in message for message in messages)+ assert result.passed+ assert result.metadata["quality_scores"]["overall_score"] == 100.0+ assert all("SKILL.md references README.md" not in finding.message for finding in result.findings)@pytest.mark.parametrize(- "statement",+ "target",[- "Agents should not read README.md.",- "Agents must not load README.md.",- "Agents shouldn't consult README.md.",- "Agents should not ever read README.md.",- "Agents should not refer to README.md.",- "Agents don't need to read README.md.",- "Agents are not required to read README.md.",- "Agents are not allowed to load README.md.",+ "README.md",+ "./README.md",+ "docs/../README.md",+ "README%2Emd",+ "%52EADME.md",+ "README.md?view=1",+ "README.md#overview",+ "README.md?view=1#overview",+ "README.md/../README.md",],)- def test_modal_negative_readme_guidance_is_not_a_reference(self, tmp_path: Path, statement: str):- skill_dir = _write_issue_skill(tmp_path, extra_body=f"\n{statement}\n")+ def test_root_readme_link_variants_trigger_reference_finding(self, tmp_path: Path, target: str):+ skill_dir = _write_issue_skill(tmp_path, extra_body=f"\nSee [documentation]({target}).\n")(skill_dir / "README.md").write_text("# Human documentation\n")messages = _finding_messages(skill_dir)- assert all("SKILL.md references README.md" not in message for message in messages)+ assert "SKILL.md references README.md (pulls human-facing docs into agent context)" in messages- def test_not_only_readme_instruction_remains_a_reference(self, tmp_path: Path):- skill_dir = _write_issue_skill(+ def test_readme_locality_is_reflected_in_score_grade_and_gate(self, tmp_path: Path):+ external_skill = _write_issue_skill(tmp_path,- extra_body="\nAgents should not only read README.md; they should also inspect the release notes.\n",+ name="external-readme-skill",+ extra_body="\nSee [external documentation](https://example.com/README.md).\n",)- (skill_dir / "README.md").write_text("# Human documentation\n")-- messages = _finding_messages(skill_dir)-- assert any("SKILL.md references README.md" in message for message in messages)+ local_skill = _write_issue_skill(+ tmp_path,+ name="local-readme-skill",+ extra_body="\nSee [local documentation](README.md).\n",+ )+ (external_skill / "README.md").write_text("# Human documentation\n")+ (local_skill / "README.md").write_text("# Human documentation\n")++ validator = QualityScoreValidator(min_score=99)+ external_result = validator.validate(external_skill)+ local_result = validator.validate(local_skill)+ external_quality = external_result.metadata["quality_scores"]+ local_quality = local_result.metadata["quality_scores"]++ assert external_result.passed+ assert external_quality["overall_score"] == 100.0+ assert external_quality["grade"] == "A"+ assert all("SKILL.md references README.md" not in finding.message for finding in external_result.findings)++ assert not local_result.passed+ assert local_quality["overall_score"] == 98.2+ assert local_quality["grade"] == "A"+ assert "SKILL.md references README.md (pulls human-facing docs into agent context)" in [+ finding.message for finding in local_result.findings+ ]- def test_affirmative_readme_instruction_after_negative_one_remains_a_reference(self, tmp_path: Path):+ @pytest.mark.parametrize(+ "content_template",+ [+ "`[example]({target})`",+ " [example]({target})",+ "<span data-doc='[example]({target})'>text</span>",+ "",+ "> ```markdown\n> [example]({target})\n> ```",+ ],+ )+ def test_non_navigation_markdown_does_not_trigger_link_findings(+ self,+ tmp_path: Path,+ content_template: str,+ ):skill_dir = _write_issue_skill(tmp_path,- extra_body="\nDo not read README.md for setup; review README.md before publishing.\n",+ extra_body=f"\n\n{content_template.format(target='README.md')}\n",)(skill_dir / "README.md").write_text("# Human documentation\n")+ references = skill_dir / "references"+ references.mkdir()+ (references / "mechanisms.md").write_text(content_template.format(target="advanced.md") + "\n")- messages = _finding_messages(skill_dir)+ messages = set(_finding_messages(skill_dir))- assert any("SKILL.md references README.md" in message for message in messages)+ assert not any(+ "SKILL.md references README.md" in message or "Deeply nested references" in message for message in messages+ )- def test_markdown_link_to_readme_suffix_is_not_a_readme_reference(self, tmp_path: Path):- skill_dir = _write_issue_skill(tmp_path, extra_body="\nSee [release notes](NOTREADME.md).\n")+ @pytest.mark.parametrize(+ "tag",+ ["script", "style", "textarea", "title", "xmp", "iframe", "noembed", "noframes", "template", "plaintext"],+ )+ def test_non_navigation_html_does_not_trigger_link_findings(self, tmp_path: Path, tag: str):+ content_template = f'<{tag}><a href="{{target}}">hidden documentation</a></{tag}>'+ skill_dir = _write_issue_skill(+ tmp_path,+ extra_body=f"\n\n{content_template.format(target='README.md')}\n",+ )(skill_dir / "README.md").write_text("# Human documentation\n")- (skill_dir / "NOTREADME.md").write_text("# Release notes\n")+ references = skill_dir / "references"+ references.mkdir()+ (references / "mechanisms.md").write_text(content_template.format(target="advanced.md") + "\n")- messages = _finding_messages(skill_dir)+ messages = set(_finding_messages(skill_dir))- assert all("SKILL.md references README.md" not in message for message in messages)+ assert not any(+ "SKILL.md references README.md" in message or "Deeply nested references" in message for message in messages+ )- def test_fenced_readme_link_example_is_not_a_readme_reference(self, tmp_path: Path):+ @pytest.mark.parametrize(+ "content_template",+ [+ "[outer [inner]]({target})",+ r"[escaped \] label]({target})",+ '<a href="{target}">documentation</a>',+ '<a href="{target} ">documentation</a>',+ '<pre><a href="{target}">documentation</a></pre>',+ '<noscript><a href="{target}">documentation</a></noscript>',+ ],+ )+ def test_navigation_links_trigger_link_findings(+ self,+ tmp_path: Path,+ content_template: str,+ ):skill_dir = _write_issue_skill(tmp_path,- extra_body="\n```markdown\nSee [human documentation](README.md).\n```\n",+ extra_body=f"\n\n{content_template.format(target='README.md')}\n",)(skill_dir / "README.md").write_text("# Human documentation\n")+ references = skill_dir / "references"+ references.mkdir()+ (references / "mechanisms.md").write_text(content_template.format(target="advanced.md") + "\n")- messages = _finding_messages(skill_dir)+ messages = set(_finding_messages(skill_dir))- assert all("SKILL.md references README.md" not in message for message in messages)+ assert {+ "SKILL.md references README.md (pulls human-facing docs into agent context)",+ "Deeply nested references in mechanisms.md",+ }.issubset(messages)- def test_commented_readme_link_is_not_a_readme_reference(self, tmp_path: Path):+ def test_raw_html_encoded_trailing_space_remains_path_content(self, tmp_path: Path):skill_dir = _write_issue_skill(tmp_path,- extra_body="\n<!-- See [human documentation](README.md). -->\n",+ extra_body='\nSee <a href="README.md%20">other documentation</a>.\n',)(skill_dir / "README.md").write_text("# Human documentation\n")@@ -980,59 +815,94 @@ class TestQualityScoreKeywordBoundaries:assert all("SKILL.md references README.md" not in message for message in messages)- def test_markdown_link_parser_scans_malformed_nested_input_linearly(self):- read_count = 0+ @pytest.mark.parametrize(+ "target",+ [+ "https://agentskills.io/specification.md",+ "../SKILL.md",+ ],+ )+ def test_external_and_parent_markdown_links_are_not_nested_references(self, tmp_path: Path, target: str):+ skill_dir = _write_issue_skill(tmp_path)+ references = skill_dir / "references"+ references.mkdir()+ (references / "mechanisms.md").write_text(f"See [the specification]({target}).\n")- class CountingText(str):- __slots__ = ()+ messages = _finding_messages(skill_dir)- def __getitem__(self, key):- nonlocal read_count- if isinstance(key, int):- read_count += 1- return super().__getitem__(key)+ assert all("Deeply nested references" not in message for message in messages)- content = CountingText("[x](" * 200)+ @pytest.mark.parametrize(+ "target",+ [+ "advanced.md/",+ "advanced.md/.",+ "advanced.md/child/..",+ "advanced.md%2F",+ "advanced.md/%2E",+ "advanced.md%5C",+ "advanced.md%2Fchild%2F%2E%2E",+ "advanced.md/?view=1",+ "advanced.md/.#overview",+ ],+ )+ def test_directory_form_markdown_links_are_not_nested_references(self, tmp_path: Path, target: str):+ skill_dir = _write_issue_skill(tmp_path)+ references = skill_dir / "references"+ references.mkdir()+ (references / "mechanisms.md").write_text(f"See [the documentation directory]({target}).\n")- assert quality_score_module._markdown_link_targets(content) == []- assert read_count <= len(content)+ messages = _finding_messages(skill_dir)- def test_inline_backtick_span_at_line_start_does_not_mask_following_links(self):- content = "```literal```\n[details](advanced.md)\n"+ assert all("Deeply nested references" not in message for message in messages)- assert quality_score_module._markdown_link_targets(content) == ["advanced.md"]+ def test_local_markdown_link_remains_a_nested_reference(self, tmp_path: Path):+ skill_dir = _write_issue_skill(tmp_path)+ references = skill_dir / "references"+ references.mkdir()+ (references / "mechanisms.md").write_text("See [more details](advanced.md).\n")- def test_shortcut_reference_style_link_resolves_definition(self):- content = "See [readme] for details.\n\n[readme]: README.md\n"+ messages = _finding_messages(skill_dir)- assert quality_score_module._markdown_link_targets(content) == ["README.md"]+ assert "Deeply nested references in mechanisms.md" in messages@pytest.mark.parametrize("target",[- "https://agentskills.io/specification.md",- "../SKILL.md",+ "advanced%2Emd",+ "docs/%2E%2E/advanced.md",],)- def test_external_and_parent_markdown_links_are_not_nested_references(self, tmp_path: Path, target: str):+ def test_encoded_local_markdown_paths_remain_nested_references(self, tmp_path: Path, target: str):skill_dir = _write_issue_skill(tmp_path)references = skill_dir / "references"references.mkdir()- (references / "mechanisms.md").write_text(f"See [the specification]({target}).\n")+ (references / "mechanisms.md").write_text(f"See [more details]({target}).\n")messages = _finding_messages(skill_dir)- assert all("Deeply nested references" not in message for message in messages)+ assert "Deeply nested references in mechanisms.md" in messages- def test_local_markdown_link_remains_a_nested_reference(self, tmp_path: Path):+ @pytest.mark.parametrize(+ "target",+ [+ "..%5CSKILL.md",+ "%2F%2Fserver%2Fguide.md",+ "/guide.md",+ "https://example.com/guide.md",+ "mailto:guide.md",+ "custom+scheme:guide.md",+ ],+ )+ def test_nonlocal_and_parent_skill_links_are_not_nested_references(self, tmp_path: Path, target: str):skill_dir = _write_issue_skill(tmp_path)references = skill_dir / "references"references.mkdir()- (references / "mechanisms.md").write_text("See [more details](advanced.md).\n")+ (references / "mechanisms.md").write_text(f"See [documentation]({target}).\n")messages = _finding_messages(skill_dir)- assert "Deeply nested references in mechanisms.md" in messages+ assert all("Deeply nested references" not in message for message in messages)@pytest.mark.parametrize("target",@@ -1040,6 +910,7 @@ class TestQualityScoreKeywordBoundaries:"../other.md","nested/SKILL.md","guide(v2).md",+ "advanced.md/../advanced.md",],)def test_other_local_markdown_paths_remain_nested_references(self, tmp_path: Path, target: str):diff --git a/uv.lock b/uv.lockindex f5a1861f..174cb4ac 100644--- a/uv.lock+++ b/uv.lock@@ -1973,6 +1973,7 @@ dependencies = [{ name = "click" },{ name = "idna" },{ name = "jinja2" },+ { name = "markdown-it-py" },{ name = "pydantic" },{ name = "pyyaml" },{ name = "rich" },@@ -2036,6 +2037,7 @@ requires-dist = [{ name = "idna", specifier = ">=3.10,<4" },{ name = "jinja2", specifier = ">=3.1" },{ name = "litellm", marker = "extra == 'llm'", specifier = ">=1.83.10,<1.89.0.dev0" },+ { name = "markdown-it-py", specifier = ">=4.2,<5" },{ name = "mcp", marker = "extra == 'tier3'", specifier = ">=1.28.1,<2" },{ name = "openai", marker = "extra == 'llm'", specifier = ">=2.21.0" },{ name = "pip-audit", marker = "extra == 'security'", specifier = ">=2.10.0" },