Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier2): parse skill frontmatter with bounded YAML (#422)
Narendran Raghavan610d8de7c468
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ed2b29e0..a2a491bc 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -505,6 +505,46 @@ All notable changes to SkillEvaluator are documented in this file.Missing lift data is shown as `-`; custom-only and legacy agents retain theirarm-difference comparison.+- Tier 2 `similarity-check` and context deduplication now parse Markdown+ frontmatter within fixed structure limits: 100 nesting levels, 20,000 nodes,+ 1,024 alias references, 1,024 items in one mapping or sequence, 65,536+ characters in one value, and integers up to 14,000 bits, with aliases and+ merge keys counted after expansion. In 0.8.0 both used plain+ `yaml.safe_load`, and `similarity-check` converted a `name` or `description`+ that was not a string to text. A 477-byte `SKILL.md` whose frontmatter chains+ YAML aliases produced an 82-million-character description, and a 543-byte+ one that chains merge keys took 180 MB in context deduplication; each further+ level multiplies the cost by ten. Mapping frontmatter over a limit now stops+ `similarity-check` with an error that names the manifest, and context+ deduplication with a CRITICAL `manifest_complexity_limit` finding, before any+ provider call. Text between the `---` lines that is not a mapping is still+ not frontmatter and is never checked against these limits.+ `similarity-check` now excludes a manifest whose `name`, `title`, or+ `description` is a non-empty value that is not a string, such as `name: 2048`+ or `name: yes`, with the coverage reason `non_string_name_or_description`+ instead of converting the value. Its debug diagnostic omits the absolute+ manifest path, keeping private host-directory names out of logs.+ An empty, zero, or false value such as `0`,+ `false`, or `[]` still gets `missing_name_or_description`. Either way, like+ any excluded manifest, it makes the scan incomplete. It stops with an error+ that names the manifest when a value is longer than 16,384 characters, the+ per-field catalog text limit; description mode still also caps+ `name: description` at 16,384 characters. An invalid tagged value such as+ `!!int ""`, an impossible date such as `2024-02-30`, or a decimal integer+ longer than 4,300 digits (Python's default limit for converting text to an+ integer) now counts as invalid frontmatter instead of stopping either check+ with an unexpected error. A repeated key still keeps its last value in both+ checks. `tier3 select-decoys` and group-mode decoy loading+ (`skill_workspace.decoy_manifest`) read the target's and each selected+ skill's frontmatter the same way. A `name` or `description` that is not a+ string, such as `name: yes`, now fails them with `Decoy selection requires a+ skill with a root SKILL.md whose frontmatter is a valid YAML mapping with a+ non-empty string name and description` instead of being converted to text;+ invalid YAML, such as an unquoted `description: Use when: debugging logs`, and+ an empty value get the same message. Frontmatter over a limit, or a value+ longer than 16,384 characters, fails them with an error that names the+ manifest.+- The Tier 3 verdict is now FAIL when a scored dimension is below 0.40 andanother dimension has no score. In 0.8.0 a dimension with no score made theagent NEUTRAL before the 0.40 check ran. Correctness has no score when nodiff --git a/docs/agents-and-sandboxes.mdx b/docs/agents-and-sandboxes.mdxindex b40ce178..21e2f9a2 100644--- a/docs/agents-and-sandboxes.mdx+++ b/docs/agents-and-sandboxes.mdx@@ -788,6 +788,19 @@ helpers stay explicit in `include`; the selector does not infer prerequisites.A collision between a selected skill and an explicitly included skill is anerror.+Selection and group-mode evaluation read the `name` and `description` of the+target and of each selected skill as `similarity-check` does, within the same+[frontmatter structure limits](/skills/skillevaluator/tier2-deduplication#defaults-and-safety-limits).+The command fails, and says the skill needs a root `SKILL.md` whose frontmatter+is a valid YAML mapping with a non-empty string name and description, when that+file or its frontmatter is missing, the frontmatter is invalid YAML or not a+mapping, or either value is empty or not a string. A value that is not a string,+such as `name: yes` or `name: 2048`, is never converted to text. Quote such a+value, as in `name: "yes"`, and quote a value that contains a colon followed by+a space, as in `description: "Use when: debugging logs"`. Frontmatter over a+structure limit, or a value longer than 16,384 characters, fails with an error+that names the manifest, such as `my-skill/SKILL.md`.+The manifest records catalog and model identity, scores, the target manifesthash, and selected skill runtime hashes. Evaluation verifies the hashes anduses one snapshot of the selected skills in both arms; it makes no embeddingdiff --git a/docs/tier2-deduplication.mdx b/docs/tier2-deduplication.mdxindex 9003d3d4..2fce8d78 100644--- a/docs/tier2-deduplication.mdx+++ b/docs/tier2-deduplication.mdx@@ -133,6 +133,15 @@ vocabulary.| `.sh` | Function bodies and comment blocks of at least three consecutive comment lines || Other extensions | Skipped |+Tier 2 parses Markdown frontmatter, the text between a file's opening `---` line+and the next `---` line, only when it is a YAML mapping. It parses a mapping+within the [frontmatter structure limits](#defaults-and-safety-limits), so YAML+aliases and merge keys are never expanded past them. A file whose mapping+frontmatter exceeds a limit fails the scan with a CRITICAL+`manifest_complexity_limit` finding that names the file, before any provider+call. Text between the `---` lines that is not a mapping, such as a list or+prose after a thematic break, is never checked against these limits.+Valid mapping frontmatter is removed from Markdown before chunking. Malformed,empty, or non-mapping frontmatter remains part of the analyzed text. Everyretained chunk keeps its source file and line range, so findings point you to@@ -270,19 +279,38 @@ skills with 2,048-dimensional embeddings need 255,488,000 scalar comparisons:skillevaluator similarity-check ./skills --max-entries 500 --max-scalar-comparisons 256000000```+`similarity-check` reads `name` (`title` for rules and workflows) and+`description` from each manifest's frontmatter, within the same+[frontmatter structure limits](#defaults-and-safety-limits) as context+deduplication. A manifest whose frontmatter exceeds a limit stops the scan with+an error that names it, before any embedding request. So does a `name` or+`description` longer than 16,384 characters, the per-field limit for catalog+text, in every embedding mode. Description mode embeds `name: description`, so+there the two values together must also fit in 16,384 characters. A list,+mapping, number, date, or boolean is never converted to text. Such a value, for+example `name: 2048` or `name: yes`, excludes the manifest with the reason+`non_string_name_or_description`. A missing value, or one that is empty, zero,+or false (`""`, `[]`, `{}`, `0`, `false`, `no`), excludes it with the reason+`missing_name_or_description` instead. Invalid YAML and frontmatter that is not+a mapping have their own reasons. Quote a value such as `name: "2048"` or+`name: "yes"` to keep it a string. Errors name a manifest by its path under the+scan root, or by its directory and file name when one manifest is read on its+own, such as a catalog query target.+The entry budget counts selected manifests even when their frontmatter isinvalid. It applies before any embedding request and never truncates a scan.Every selected manifest is accounted for: reports include `collection_coverage`with `selected_count`, `indexed_count`, and `skipped_count`. Here indexed meansvalid manifests eligible for embedding; a provider or work-limit error can stillstop the index build. Missing frontmatter, malformed YAML, a non-mapping header,-or missing name/title or description produces an exclusion reason. Up to 100-exclusions are listed with collection-relative paths (each bounded to 512-characters); `omitted_exclusions` counts additional diagnostics. An excluded-manifest makes the scan **INCOMPLETE** and blocking, even when the indexed subset-has no matches. Matches in the valid subset remain available. Fix excluded-manifests and rerun; an incomplete collection cannot replace or create a saved-catalog. Unsafe paths and input limits continue to stop discovery outright.+a missing name/title or description, or one that is not a string, produces an+exclusion reason. Up to 100 exclusions are listed with collection-relative paths+(each bounded to 512 characters); `omitted_exclusions` counts additional+diagnostics. An excluded manifest makes the scan **INCOMPLETE** and blocking,+even when the indexed subset has no matches. Matches in the valid subset remain+available. Fix excluded manifests and rerun; an incomplete collection cannot+replace or create a saved catalog. Unsafe paths and input limits continue to+stop discovery outright.For Python callers reusing an `EmbeddingRegistry`, each successful`build_from_directory()` or `build_from_paths()` replaces the entire indexed@@ -449,6 +477,8 @@ you intend to skip Tier 2 entirely.| Minimum content chunk | 80 characters | Drops sections and chunks too small to compare usefully; short paragraphs of an oversized section stay attached to a neighbour || Embedding input cap | 3,000 characters | Caps every Markdown, Python, and shell embedding input; oversized Markdown sections are packed at paragraph boundaries || High-confidence duplicate boundary | `0.70` | Maps actionable duplicate verdicts to HIGH rather than MEDIUM |+| Frontmatter structure | 100 nesting levels, 20,000 nodes, 1,024 alias references, 1,024 items per mapping or sequence, 65,536 characters per value, integers up to 14,000 bits (about 4,200 digits) | Bounds every Markdown mapping frontmatter Tier 2 parses; aliases and merge keys count after expansion. A decimal integer longer than 4,300 digits, Python's default limit for converting text to an integer, counts as invalid YAML, not as a limit failure |+| Inter-skill name and description | 16,384 characters each | Per-field catalog text limit, in every embedding mode; description mode also caps `name: description` at 16,384 characters. A value that is not a string excludes the manifest and makes the scan incomplete || Intra-skill content chunks | 512 | Per-skill ceiling, checked before any embedding request || Intra-skill comparison work | 267,911,168 scalar comparisons | Bounds all-pairs clustering; allows all 512 chunks with embeddings of up to 2,048 dimensions, 418 chunks at 3,072, and 362 at 4,096 || Intra-skill LLM review clusters | 50 | Most candidate clusters sent for chat-LLM review in one skill |@@ -480,7 +510,8 @@ that target as a regular directory. The alias is never followed, so the targetdirectory is scanned once. Absolute links and Windows junctions, broken links,links that escape the root, and aliases that point at another link remainrejected. Explicit limits bound-discovered paths, file counts, per-file and total input bytes, content chunks,+discovered paths, file counts, per-file and total input bytes, frontmatter+structure, content chunks,candidate clusters, catalog size and entry count, vector length, returnedmatches, and pairwise scalar work. Catalog loading also rejects duplicate JSONkeys, unexpected fields, invalid identities, non-finite vectors, and@@ -523,6 +554,8 @@ handling terms before sending confidential skill content to a hosted endpoint.| A directory is a Windows junction, other reparse point, or a symbolic link that does not qualify as a contained alias | Rejected during the walk, before descent || Embedding or chat-LLM analysis is incomplete | Reports an error; does not claim a clean result || Skill exceeds the content-chunk, comparison-work, or LLM-review limit (candidate clusters or prompt size) | Stops before the remaining analysis and reports the scan as incomplete; no duplicate-content finding is added |+ | Markdown mapping frontmatter exceeds a structure limit, for example through chained YAML aliases or merge keys | Fails closed with a CRITICAL `manifest_complexity_limit` finding before any provider call |+ | Text between the `---` lines is a list, prose, or other YAML that is not a mapping | Stays in the analyzed text and is never checked against the frontmatter structure limits || The chat LLM declines to classify a cluster | Reports an advisory MEDIUM `llm_refused` finding for that cluster's sections |</Accordion><Accordion title="Inter-skill analysis">@@ -531,7 +564,9 @@ handling terms before sending confidential skill content to a hosted endpoint.| Direct collection has fewer than two skills | Returns an actionable input error || Catalog source collection is empty | Refuses to save an empty catalog || Catalog path is missing | Returns an error and does not rebuild it silently |- | Candidate lacks a root `SKILL.md` | Rejects the catalog query |+ | Candidate lacks a root `SKILL.md`, or its frontmatter is not a valid YAML mapping with a non-empty string name and description | Rejects the catalog query and says what the frontmatter needs |+ | A manifest's frontmatter exceeds a structure limit, or its name or description is longer than 16,384 characters | Fails with an error that names the manifest, before any embedding request |+ | A manifest's name or description is not a string, such as `name: 2048`, or its frontmatter is not a mapping | Excludes the manifest with a coverage reason, as for invalid YAML, and reports the scan as incomplete || A skill contains an exactly named, single-link regular `AGENTS.md` plus `CLAUDE.md -> AGENTS.md` | Validates and skips the unrelated compatibility alias while discovering `SKILL.md` || The collection contains a relative directory alias such as `.agents -> .claude` whose target is a regular directory in the same collection | Skips the alias without following it, so manifests in the target are discovered once || Candidate matches multiple catalog entries | Reports matches in descending similarity order |diff --git a/src/skillevaluator/deduplication/utils/skill_collector.py b/src/skillevaluator/deduplication/utils/skill_collector.pyindex 7560947e..75d9e6e6 100644--- a/src/skillevaluator/deduplication/utils/skill_collector.py+++ b/src/skillevaluator/deduplication/utils/skill_collector.py@@ -18,6 +18,7 @@ from itertools import islicefrom pathlib import Pathimport yaml+from yaml.events import MappingStartEvent, NodeEventfrom skillevaluator.constants import (CONTENT_DEDUP_EXCLUDED_DIRS,@@ -29,6 +30,11 @@ from skillevaluator.constants import (CONTENT_DEDUP_SCANNABLE_EXTENSIONS,)from skillevaluator.utils.path_security import matches_filesystem_name+from skillevaluator.utils.structured_data import (+ StructuredDataLimitError,+ StructuredDataSyntaxError,+ load_bounded_yaml,+)from skillevaluator.utils.tier2_paths import (contained_directory_alias_target,is_contained_compatibility_alias,@@ -39,6 +45,8 @@ from skillevaluator.validators.frontmatter_parser import FRONTMATTER_PATTERNlogger = logging.getLogger(__name__)_OPEN_SUPPORTS_DIR_FD = os.open in os.supports_dir_fd+# Root tags that load as a dict: none, the non-specific "!", or an explicit !!map.+_PLAIN_MAPPING_TAGS = frozenset({None, "!", "tag:yaml.org,2002:map"})class SkillCollectionError(ValueError):@@ -340,16 +348,48 @@ def _is_link_or_reparse(path: Path, rel_path: str | None = None) -> bool:return stat.S_ISLNK(info.st_mode) or bool(reparse_flag and file_attributes & reparse_flag)+def _frontmatter_root_kind(frontmatter_yaml: str) -> str:+ """Classify the root node of the text between the ``---`` lines.++ Returns ``"mapping"`` for a plain YAML mapping, ``"invalid"`` when the YAML+ is malformed before its root node, and ``"other"`` for anything else. Only+ the parser events up to the root node are read. Text that is not a+ mapping, such as a thematic break followed by a long list or prose, or a+ ``!!set`` that loads as a set, is never frontmatter, so it is not built or+ checked against the structure limits.+ """+ try:+ for event in yaml.parse(frontmatter_yaml, Loader=yaml.SafeLoader):+ if isinstance(event, NodeEvent):+ is_mapping = isinstance(event, MappingStartEvent) and event.tag in _PLAIN_MAPPING_TAGS+ return "mapping" if is_mapping else "other"+ except yaml.YAMLError:+ return "invalid"+ return "other"+++def _frontmatter_root_is_mapping(frontmatter_yaml: str) -> bool:+ """Return whether the text between the ``---`` lines opens a plain YAML mapping."""+ return _frontmatter_root_kind(frontmatter_yaml) == "mapping"++def _strip_valid_frontmatter(raw_text: str) -> tuple[str, int]:- """Strip valid mapping frontmatter and return its original line offset."""+ """Strip valid mapping frontmatter and return its original line offset.++ Raises :class:`StructuredDataLimitError` when mapping frontmatter exceeds a+ nesting, node, alias, merge, scalar, or integer-size limit.+ """match = FRONTMATTER_PATTERN.match(raw_text)if not match:return raw_text, 0frontmatter_yaml, markdown_content = match.groups()+ if not _frontmatter_root_is_mapping(frontmatter_yaml):+ return raw_text, 0try:- data = yaml.safe_load(frontmatter_yaml)- except yaml.YAMLError:+ # Agent clients keep the last value of a repeated frontmatter key.+ data = load_bounded_yaml(frontmatter_yaml, last_key_wins=True)+ except StructuredDataSyntaxError:return raw_text, 0if not data or not isinstance(data, dict):return raw_text, 0@@ -664,7 +704,17 @@ def _collect_files_anchored(content = raw_textline_offset = 0if candidate.extension in {".md", ".mdc"}:- content, line_offset = _strip_valid_frontmatter(raw_text)+ try:+ content, line_offset = _strip_valid_frontmatter(raw_text)+ except StructuredDataLimitError as e:+ raise SkillCollectionError(+ "manifest_complexity_limit",+ f"Frontmatter structured-data complexity limit exceeded: {candidate.rel_path}",+ rel_path=candidate.rel_path,+ suggestion=(+ "Reduce frontmatter nesting, collection sizes, scalar lengths, integer sizes, or YAML aliases."+ ),+ ) from ecollected.append(CollectedFile(diff --git a/src/skillevaluator/embedding/extractor.py b/src/skillevaluator/embedding/extractor.pyindex 42581073..7971f0cc 100644--- a/src/skillevaluator/embedding/extractor.py+++ b/src/skillevaluator/embedding/extractor.py@@ -13,9 +13,7 @@ from __future__ import annotationsimport osimport statfrom dataclasses import dataclass, field-from pathlib import Path--import yaml+from pathlib import Path, PurePosixPathfrom skillevaluator.constants import (CONTENT_DEDUP_MAX_FILE_BYTES,@@ -30,9 +28,19 @@ from skillevaluator.constants import (SKILL_MANIFEST_VARIANTS,WORKFLOWS_MANIFEST_FILE,)-from skillevaluator.deduplication.utils.skill_collector import _SecureReadError, _SecureRoot+from skillevaluator.deduplication.utils.skill_collector import (+ _frontmatter_root_kind,+ _SecureReadError,+ _SecureRoot,+)from skillevaluator.embedding.limits import validate_max_entriesfrom skillevaluator.logging_config import get_logger+from skillevaluator.utils.structured_data import (+ StructuredDataLimitError,+ StructuredDataSyntaxError,+ load_bounded_yaml,+ require_bounded_string,+)from skillevaluator.utils.tier2_paths import (contained_directory_alias_target,is_contained_compatibility_alias,@@ -48,6 +56,12 @@ MAX_DISCOVERED_PATHS = SIMILARITY_MAX_DISCOVERED_PATHSDISCOVERY_EXCLUDED_DIRS = SCAN_EXCLUDED_DIRSMAX_EXTRACTION_EXCLUSIONS = 100MAX_EXCLUSION_PATH_CHARS = 512+# Longest name or description read from frontmatter, in every embedding mode.+# It bounds the memory one field can take and equals the registry's per-field+# catalog text limit (MAX_CATALOG_TEXT_LENGTH). The registry still enforces the+# combined description embedding-text limit and the catalog limits, and the+# Tier 1 length rules stay with Tier 1.+MAX_FRONTMATTER_FIELD_CHARS = 16_384@dataclass@@ -162,6 +176,7 @@ def extract_from_skill(skill_dir: Path, *, budget: _ExtractionBudget | None = Nobudget=budget,secure_root=secure_root,relative_path=Path(variant),+ label=_manifest_label(skill_dir, variant),)except _SecureReadError as exc:raise ValueError(f"Skill root is a symlink, reparse point, or cannot be securely opened: {skill_dir}") from exc@@ -188,6 +203,7 @@ def extract_from_rule(rule_path: Path, *, budget: _ExtractionBudget | None = Nonbudget=budget,secure_root=secure_root,relative_path=Path(rule_path.name),+ label=_manifest_label(rule_path.parent, rule_path.name),)except _SecureReadError as exc:raise ValueError(@@ -212,6 +228,7 @@ def extract_from_workflow(workflow_dir: Path, *, budget: _ExtractionBudget | Nonbudget=budget,secure_root=secure_root,relative_path=Path(WORKFLOWS_MANIFEST_FILE),+ label=_manifest_label(workflow_dir, WORKFLOWS_MANIFEST_FILE),)except _SecureReadError as exc:raise ValueError(@@ -264,6 +281,16 @@ def discover_and_extract_detailed(# ---------------------------------------------------------------------------+def _manifest_label(directory: Path, manifest_name: str) -> str:+ """Name a manifest read on its own by its directory, such as ``my-skill/SKILL.md``.++ The directory is made absolute and ``..`` is removed lexically, without+ following links, so ``Path(".")`` or ``Path("..")`` is named by the real+ directory name.+ """+ return PurePosixPath(Path(os.path.normpath(directory.absolute())).name, manifest_name).as_posix()++def _is_symlink_or_reparse(path: Path) -> bool:"""Return whether *path* redirects traversal through a link or reparse point."""try:@@ -308,8 +335,12 @@ def _extract_from_file(budget: _ExtractionBudget | None = None,secure_root: _SecureRoot | None = None,relative_path: Path | None = None,+ label: str | None = None,) -> ContentEntry | None:- """Read once, parse frontmatter, and build a bounded content entry."""+ """Read once, parse frontmatter, and build a bounded content entry.++ Errors name the manifest by *label*, or else by its path under the scan root.+ """full_text = _read_bounded_manifest(file_path,budget,@@ -324,13 +355,24 @@ def _extract_from_file(return Nonefrontmatter_yaml, _markdown_content = match.groups()+ if label is None:+ label = relative_path.as_posix() if relative_path is not None else file_path.name+ # Only the parser events up to the root node are read first. A root that+ # is not a mapping is never built; it is excluded like an empty mapping,+ # even when the YAML after the root node is also malformed. YAML that is+ # malformed before its root node goes to the loader, which rejects it as+ # invalid YAML.+ root_kind = _frontmatter_root_kind(frontmatter_yaml)try:- data = yaml.safe_load(frontmatter_yaml)- except yaml.YAMLError as exc:+ # Agent clients keep the last value of a repeated frontmatter key.+ data = load_bounded_yaml(frontmatter_yaml, last_key_wins=True) if root_kind != "other" else None+ except StructuredDataSyntaxError as exc:if budget is not None:budget.exclude(file_path, "invalid_yaml")logger.debug("Invalid YAML frontmatter in %s: %s", file_path, exc)return None+ except StructuredDataLimitError as exc:+ raise ValueError(f"Manifest structured-data complexity limit exceeded: {label}") from excif not data or not isinstance(data, dict):if budget is not None:budget.exclude(file_path, "invalid_frontmatter_mapping")@@ -349,10 +391,25 @@ def _extract_from_file(file_path,)return None+ # Never call str() on a frontmatter value: a list or mapping built from+ # YAML aliases renders to a string far larger than the manifest. A value+ # that is not a string makes the entry invalid, with its own reason.+ if not (isinstance(name, str) and isinstance(description, str)):+ if budget is not None:+ budget.exclude(file_path, "non_string_name_or_description")+ logger.debug("Frontmatter %s or %s is not a string", name_field, description_field)+ return None+ try:+ name = require_bounded_string(name, name_field, max_chars=MAX_FRONTMATTER_FIELD_CHARS, allow_empty=True)+ description = require_bounded_string(+ description, description_field, max_chars=MAX_FRONTMATTER_FIELD_CHARS, allow_empty=True+ )+ except ValueError as exc:+ raise ValueError(f"Manifest frontmatter {exc}: {label}") from excreturn ContentEntry(- name=str(name),- description=str(description),+ name=name,+ description=description,path=str(file_path.parent if content_type == CONTENT_TYPE_SKILL else file_path),content_type=content_type,full_text=full_text,diff --git a/src/skillevaluator/tier3/decoys.py b/src/skillevaluator/tier3/decoys.pyindex 44dc1c2b..aa788ec9 100644--- a/src/skillevaluator/tier3/decoys.py+++ b/src/skillevaluator/tier3/decoys.py@@ -110,7 +110,10 @@ def _is_target_path(path: Path, target: Path) -> bool:def _skill(path: Path) -> ContentEntry:entry = extract_from_skill(path)if entry is None or not (path / "SKILL.md").is_file():- raise ValueError(f"Decoy selection requires a skill with SKILL.md and name/description: {path}")+ raise ValueError(+ "Decoy selection requires a skill with a root SKILL.md whose frontmatter is a valid YAML mapping "+ f"with a non-empty string name and description: {path}"+ )_validate_catalog_string(entry.name, "Skill name")return entrydiff --git a/src/skillevaluator/validators/similarity.py b/src/skillevaluator/validators/similarity.pyindex 42541771..34d7450c 100644--- a/src/skillevaluator/validators/similarity.py+++ b/src/skillevaluator/validators/similarity.py@@ -129,7 +129,10 @@ class SimilarityValidator(ValidatorBase):)target = extract_from_skill(skill_path)if target is None:- result.add_error("Catalog comparison requires a skill directory with a root SKILL.md")+ result.add_error(+ "Catalog comparison requires a skill directory with a root SKILL.md whose "+ "frontmatter is a valid YAML mapping with a non-empty string name and description"+ )return result# A catalog built from the tree that holds the target records the# target itself; never report it as its own duplicate. Copiesdiff --git a/tests/deduplication/intra_skill/test_intra_skill_validator.py b/tests/deduplication/intra_skill/test_intra_skill_validator.pyindex 45416730..4cdf2692 100644--- a/tests/deduplication/intra_skill/test_intra_skill_validator.py+++ b/tests/deduplication/intra_skill/test_intra_skill_validator.py@@ -181,6 +181,29 @@ class TestIntraSkillValidatorValidate:assert finding.suggestion is not Noneassert "replace" in finding.suggestion.lower()+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")+ def test_frontmatter_merge_chain_fails_closed_before_embedding(self, mock_embed, tmp_path: Path) -> None:+ # Each level merges the previous mapping ten times, so unbounded YAML+ # builds 10**(level + 1) key-value pairs, about 180 MB at six levels.+ # Four levels keep this test cheap if the bound ever regresses.+ lines = ["m0: &m0 {" + ", ".join(f"k{index}: {index}" for index in range(10)) + "}"]+ lines += [f"m{level}: &m{level} {{<<: [{', '.join([f'*m{level - 1}'] * 10)}]}}" for level in range(1, 5)]+ skill_dir = tmp_path / "skill"+ skill_dir.mkdir()+ (skill_dir / "SKILL.md").write_text(+ "---\nname: skill\n" + "\n".join(lines) + "\n---\n" + _numbered_sections(3), encoding="utf-8"+ )++ result = IntraSkillValidator().validate(skill_dir)++ assert result.passed is False+ assert len(result.findings) == 1+ finding = result.findings[0]+ assert finding.severity == Severity.CRITICAL+ assert finding.check_name == "manifest_complexity_limit"+ assert finding.file_path == "SKILL.md"+ mock_embed.assert_not_called()+@staticmethoddef _assert_work_limit_incomplete(result, work_limit: dict) -> None:"""A tripped work limit is missing coverage, never a duplicate-content finding."""diff --git a/tests/deduplication/utils/test_skill_collector.py b/tests/deduplication/utils/test_skill_collector.pyindex 5387d3c6..62c74ccf 100644--- a/tests/deduplication/utils/test_skill_collector.py+++ b/tests/deduplication/utils/test_skill_collector.py@@ -9,6 +9,7 @@ import loggingimport osimport statimport subprocess+import sysfrom pathlib import Pathfrom types import SimpleNamespacefrom unittest.mock import patch@@ -24,6 +25,15 @@ from skillevaluator.deduplication.utils.skill_collector import ()+def default_int_digit_limit():+ """Pin Python's default 4,300-digit limit for converting text to an int."""+ previous = sys.get_int_max_str_digits()+ sys.set_int_max_str_digits(4_300)+ yield+ sys.set_int_max_str_digits(previous)++class TestCollectedFile:def test_dataclass_fields(self, tmp_path: Path) -> None:cf = CollectedFile(@@ -133,6 +143,104 @@ class TestCollectFiles:assert len(result) == 1assert result[0].extension == ".mdc"+ @pytest.mark.usefixtures("default_int_digit_limit")+ @pytest.mark.parametrize(+ "frontmatter",+ [+ pytest.param(+ "\n".join(+ ["m0: &m0 {" + ", ".join(f"k{index}: {index}" for index in range(10)) + "}"]+ + [f"m{level}: &m{level} {{<<: [{', '.join([f'*m{level - 1}'] * 10)}]}}" for level in range(1, 5)]+ ),+ id="merge-key-chain",+ ),+ pytest.param(+ "\n".join(+ ["a0: &a0 safe"]+ + [f"a{level}: &a{level} [*a{level - 1}, *a{level - 1}]" for level in range(1, 19)]+ + ["description: *a18"]+ ),+ id="alias-dag",+ ),+ pytest.param("extra: " + "[" * 1_500 + "safe" + "]" * 1_500, id="deep-nesting"),+ pytest.param("description: " + "x" * 65_537, id="long-scalar"),+ pytest.param("tags:\n" + "\n".join(f" - tag{index}" for index in range(1_100)), id="long-list"),+ # 3,600 hex digits are 14,400 bits, over the 14,000-bit integer limit.+ pytest.param("version: 0x" + "f" * 3_600, id="big-integer"),+ # 4,250 decimal digits are about 14,118 bits, within Python's 4,300-digit limit.+ pytest.param("version: " + "9" * 4_250, id="big-decimal-integer"),+ ],+ )+ def test_frontmatter_complexity_limit_fails_closed(self, skill_root: Path, frontmatter: str) -> None:+ references = skill_root / "references"+ references.mkdir()+ (references / "guide.md").write_text(f"---\nname: guide\n{frontmatter}\n---\n# Guide\n", encoding="utf-8")++ with pytest.raises(SkillCollectionError, match=r"complexity limit exceeded: references/guide\.md") as exc:+ collect_files(skill_root)++ assert exc.value.check_name == "manifest_complexity_limit"+ assert exc.value.rel_path == "references/guide.md"+ assert "YAML aliases" in exc.value.suggestion++ @pytest.mark.usefixtures("default_int_digit_limit")+ @pytest.mark.parametrize(+ "line",+ [+ pytest.param('version: !!int ""', id="tagged-int"),+ pytest.param("version: !!bool nope", id="tagged-bool"),+ pytest.param("version: !!float x", id="tagged-float"),+ pytest.param("last_updated: 2024-02-30", id="impossible-date"),+ # int() rejects more than 4,300 decimal digits before the bit limit applies.+ pytest.param("version: " + "9" * 5_000, id="decimal-over-digit-limit"),+ ],+ )+ def test_unconstructible_value_keeps_frontmatter_in_the_analyzed_text(self, skill_root: Path, line: str) -> None:+ text = f"---\nname: tagged\n{line}\n---\n# Body\nContent."+ # Bytes keep the LF line endings on every platform, so the exact text compares.+ (skill_root / "SKILL.md").write_bytes(text.encode("utf-8"))++ result = collect_files(skill_root)++ assert result[0].content == text+ assert result[0].line_offset == 0++ @pytest.mark.parametrize(+ "block",+ [+ pytest.param("\n".join(f"- item {index}" for index in range(1_100)), id="thematic-break-list"),+ pytest.param("A line of prose that holds no key or colon.\n" * 1_600, id="thematic-break-prose"),+ pytest.param(+ "\n".join(+ ["- &m0 {" + ", ".join(f"k{index}: {index}" for index in range(10)) + "}"]+ + [f"- &m{level} {{<<: [{', '.join([f'*m{level - 1}'] * 10)}]}}" for level in range(1, 5)]+ ),+ id="sequence-merge-chain",+ ),+ pytest.param("!!set\n" + "\n".join(f"? item{index}" for index in range(1_100)), id="set-root"),+ ],+ )+ def test_block_that_is_not_a_mapping_stays_in_the_analyzed_text(self, skill_root: Path, block: str) -> None:+ # Each block is over a structure limit if parsed as YAML, but text that+ # is not a mapping is not frontmatter, so it is never built.+ references = skill_root / "references"+ references.mkdir()+ text = f"---\n{block}\n---\n# Guide\nContent."+ (references / "guide.md").write_bytes(text.encode("utf-8"))++ result = {item.rel_path: item for item in collect_files(skill_root)}++ assert result["references/guide.md"].content == text+ assert result["references/guide.md"].line_offset == 0++ def test_repeated_frontmatter_key_is_still_stripped(self, skill_root: Path) -> None:+ (skill_root / "SKILL.md").write_bytes(b"---\nname: first\nname: second\n---\n# Body\nContent.")++ result = collect_files(skill_root)++ assert result[0].content == "# Body\nContent."+ assert result[0].line_offset == 4+def test_openclaw_compatibility_alias_is_skipped_and_regular_target_is_collected_once(self,skill_root: Path,diff --git a/tests/embedding/test_collection_coverage.py b/tests/embedding/test_collection_coverage.pyindex 4cfbc50a..7d3a5b08 100644--- a/tests/embedding/test_collection_coverage.py+++ b/tests/embedding/test_collection_coverage.py@@ -207,7 +207,11 @@ def test_failed_build_preserves_committed_index_and_coverage(elif failure == "vector":client.embed.side_effect = lambda texts: [[float("nan"), 0.0] for _ in texts]elif failure == "input":- (second / "gamma" / "SKILL.md").write_text("---\nname: gamma\ndescription: " + "x" * 17000 + "\n---\n")+ # Each field fits the 16,384-character frontmatter field limit, so discovery+ # succeeds; together they exceed the description-mode embedding-text limit.+ (second / "gamma" / "SKILL.md").write_text(+ "---\nname: gamma-" + "g" * 9000 + "\ndescription: " + "x" * 9000 + "\n---\n"+ )with pytest.raises((ValueError, SimilarityConfigError)):registry.build_from_directory(second, "skill", minimum_entries=3 if failure == "minimum" else 1, for_pairwise_scan=failure == "preflight"diff --git a/tests/embedding/test_extractor.py b/tests/embedding/test_extractor.pyindex 0febd62b..ca8183b7 100644--- a/tests/embedding/test_extractor.py+++ b/tests/embedding/test_extractor.py@@ -7,15 +7,18 @@ from __future__ import annotationsimport osimport stat+import sysfrom pathlib import Pathfrom unittest.mock import MagicMockimport pytestimport skillevaluator.embedding.extractor as extractor_module+import skillevaluator.embedding.registry as registry_modulefrom skillevaluator.embedding.extractor import (_is_symlink_or_reparse,discover_and_extract,+ discover_and_extract_detailed,extract_from_rule,extract_from_skill,extract_from_workflow,@@ -74,6 +77,26 @@ def test_discovery_debug_log_uses_root_label_without_host_path(tmp_path: Path, massert root.name in rendered[email protected]("field", ["name", "description"])+def test_non_string_frontmatter_debug_log_omits_host_path(tmp_path: Path, monkeypatch, field: str) -> None:+ skill_dir = tmp_path / "private-ancestor" / "typed-skill"+ skill_dir.mkdir(parents=True)+ values = {"name": "typed-skill", "description": "A test skill"}+ values[field] = "[not, a, string]"+ (skill_dir / "SKILL.md").write_text(+ f"---\nname: {values['name']}\ndescription: {values['description']}\n---\n", encoding="utf-8"+ )+ debug = MagicMock()+ monkeypatch.setattr(extractor_module.logger, "debug", debug)++ assert extract_from_skill(skill_dir) is None++ rendered = "\n".join(call.args[0] % call.args[1:] for call in debug.call_args_list)+ assert "is not a string" in rendered+ assert str(tmp_path) not in rendered+ assert "private-ancestor" not in rendered++def _write_rule(root: Path, filename: str, title: str, description: str) -> Path:rule_file = root / filenamerule_file.parent.mkdir(parents=True, exist_ok=True)@@ -648,3 +671,333 @@ class TestDiscoverAndExtract:with pytest.raises(ValueError, match=r"symlink|reparse|escape"):extract_from_skill(skill_dir)+++def _alias_dag(field: str, *, levels: int, fanout: int) -> str:+ """Return frontmatter whose *field* is a list built from chained YAML aliases."""+ lines = ["seed: &a0 safe"]+ for level in range(1, levels + 1):+ lines.append(f"a{level}: &a{level} [{', '.join([f'*a{level - 1}'] * fanout)}]")+ lines.append("name: safe-name" if field != "name" else f"name: *a{levels}")+ lines.append("description: Safe description" if field != "description" else f"description: *a{levels}")+ return "---\n" + "\n".join(lines) + "\n---\n# Body\n"+++def _merge_chain(*, levels: int, fanout: int) -> str:+ """Return frontmatter whose merge keys repeat earlier mappings at every level."""+ lines = ["m0: &m0 {" + ", ".join(f"k{index}: {index}" for index in range(fanout)) + "}"]+ for level in range(1, levels + 1):+ lines.append(f"m{level}: &m{level} {{<<: [{', '.join([f'*m{level - 1}'] * fanout)}]}}")+ lines += ["name: merge-chain", "description: Merge chain"]+ return "---\n" + "\n".join(lines) + "\n---\n# Body\n"+++# Markdown whose text between two ``---`` lines is not a YAML mapping, so it is+# not frontmatter. Each is over a structure limit if parsed as YAML: a list of+# 1,100 items, a plain scalar of about 70,400 characters, a sequence of mappings+# that chains merge keys, and a 1,100-item ``!!set``, which loads as a set.+NON_MAPPING_BLOCKS = [+ pytest.param("\n".join(f"- item {index}" for index in range(1_100)), id="thematic-break-list"),+ pytest.param("A line of prose that holds no key or colon.\n" * 1_600, id="thematic-break-prose"),+ pytest.param(+ "\n".join(+ ["- &m0 {" + ", ".join(f"k{index}: {index}" for index in range(10)) + "}"]+ + [f"- &m{level} {{<<: [{', '.join([f'*m{level - 1}'] * 10)}]}}" for level in range(1, 5)]+ ),+ id="sequence-merge-chain",+ ),+ pytest.param("!!set\n" + "\n".join(f"? item{index}" for index in range(1_100)), id="set-root"),+]+++def _write_skill_manifest(root: Path, name: str, manifest: str) -> Path:+ skill_dir = root / name+ skill_dir.mkdir(parents=True)+ (skill_dir / "SKILL.md").write_bytes(manifest.encode("utf-8"))+ return skill_dir+++def default_int_digit_limit():+ """Pin Python's default 4,300-digit limit for converting text to an int."""+ previous = sys.get_int_max_str_digits()+ sys.set_int_max_str_digits(4_300)+ yield+ sys.set_int_max_str_digits(previous)+++class TestBoundedFrontmatter:+ """Frontmatter is parsed with structured-data limits and never coerced to text."""++ @pytest.mark.parametrize("field", ["name", "description"])+ def test_alias_amplified_field_fails_closed(self, tmp_path: Path, field: str) -> None:+ # 2**18 list items from under 450 bytes; str() of the list would be megabytes.+ skill_dir = _write_skill_manifest(tmp_path, "alias-dag", _alias_dag(field, levels=18, fanout=2))++ with pytest.raises(ValueError, match=r"structured-data complexity limit exceeded: alias-dag/SKILL\.md$"):+ extract_from_skill(skill_dir)++ def test_merge_key_chain_fails_closed(self, tmp_path: Path) -> None:+ skill_dir = _write_skill_manifest(tmp_path, "merge-chain", _merge_chain(levels=4, fanout=10))++ with pytest.raises(ValueError, match=r"structured-data complexity limit exceeded: merge-chain/SKILL\.md$"):+ extract_from_skill(skill_dir)++ @pytest.mark.usefixtures("default_int_digit_limit")+ @pytest.mark.parametrize(+ "value",+ [+ # 3,600 hex digits are 14,400 bits, over the 14,000-bit integer limit.+ pytest.param("0x" + "f" * 3_600, id="hex"),+ # 4,250 decimal digits are about 14,118 bits, within Python's 4,300-digit limit.+ pytest.param("9" * 4_250, id="decimal"),+ ],+ )+ def test_oversized_integer_fails_closed(self, tmp_path: Path, value: str) -> None:+ skill_dir = _write_skill_manifest(+ tmp_path, "big-int", f"---\nname: big-int\ndescription: Big\nversion: {value}\n---\n"+ )++ with pytest.raises(ValueError, match=r"structured-data complexity limit exceeded: big-int/SKILL\.md$"):+ extract_from_skill(skill_dir)++ def test_deep_nesting_fails_closed_without_recursion_error(self, tmp_path: Path) -> None:+ nested = "[" * 1_500 + "safe" + "]" * 1_500+ skill_dir = _write_skill_manifest(+ tmp_path, "deep", f"---\nname: deep\ndescription: Deep\nextra: {nested}\n---\n# Body\n"+ )++ with pytest.raises(ValueError, match=r"structured-data complexity limit exceeded: deep/SKILL\.md$"):+ extract_from_skill(skill_dir)++ @pytest.mark.parametrize(+ ("field", "value"),+ [+ ("description", "*items"),+ ("description", "{key: value}"),+ ("description", "3.5"),+ ("name", "2048"),+ ("name", "true"),+ ],+ )+ def test_non_string_field_skips_the_manifest_without_coercion(+ self, tmp_path: Path, monkeypatch, field: str, value: str+ ) -> None:+ # Three alias levels stay within the graph limits, yet str() of *items+ # would turn this 278-byte frontmatter into an 8,222-character description.+ lines = ["seed: &a0 safe"]+ lines += [f"a{level}: &a{level} [{', '.join([f'*a{level - 1}'] * 10)}]" for level in (1, 2, 3)]+ lines += ["anchor: &items [*a3]", "name: safe-name", "description: Safe description", f"{field}: {value}"]+ skill_dir = _write_skill_manifest(tmp_path, "typed", "---\n" + "\n".join(lines) + "\n---\n# Body\n")+ monkeypatch.setattr(extractor_module, "ContentEntry", MagicMock(side_effect=AssertionError("entry built")))++ assert extract_from_skill(skill_dir) is None++ @pytest.mark.parametrize("field", ["name", "description"])+ def test_field_length_matches_the_catalog_text_limit(self, tmp_path: Path, field: str) -> None:+ limit = extractor_module.MAX_FRONTMATTER_FIELD_CHARS+ assert limit == registry_module.MAX_CATALOG_TEXT_LENGTH == 16_384+ values = {"name": "safe-name", "description": "Safe description"}++ values[field] = "x" * limit+ accepted = _write_skill_manifest(+ tmp_path, "at-limit", f"---\nname: {values['name']}\ndescription: {values['description']}\n---\n"+ )+ entry = extract_from_skill(accepted)+ assert entry is not None+ assert len(getattr(entry, field)) == limit++ values[field] = "x" * (limit + 1)+ rejected = _write_skill_manifest(+ tmp_path, "over-limit", f"---\nname: {values['name']}\ndescription: {values['description']}\n---\n"+ )+ with pytest.raises(+ ValueError, match=rf"frontmatter {field} exceeds the 16384-character limit: over-limit/SKILL\.md$"+ ):+ extract_from_skill(rejected)++ def test_single_workflow_error_names_its_directory(self, tmp_path: Path) -> None:+ workflow_dir = tmp_path / "long-workflow"+ workflow_dir.mkdir()+ (workflow_dir / "workflow-rules.mdc").write_bytes(+ ("---\ntitle: Workflow\ndescription: " + "x" * 16_385 + "\n---\n").encode("utf-8")+ )++ with pytest.raises(+ ValueError, match=r"description exceeds the 16384-character limit: long-workflow/workflow-rules\.mdc$"+ ):+ extract_from_workflow(workflow_dir)++ def test_single_rule_error_names_its_directory(self, tmp_path: Path) -> None:+ rule_dir = tmp_path / "rules-dir"+ rule_dir.mkdir()+ (rule_dir / "long.mdc").write_bytes(+ ("---\ntitle: " + "x" * 16_385 + "\ndescription: Rule\n---\n").encode("utf-8")+ )++ with pytest.raises(ValueError, match=r"title exceeds the 16384-character limit: rules-dir/long\.mdc$"):+ extract_from_rule(rule_dir / "long.mdc")++ @pytest.mark.parametrize("relative", [".", "nested/.."])+ def test_relative_directory_is_named_by_its_own_name(self, tmp_path: Path, monkeypatch, relative: str) -> None:+ skill_dir = _write_skill_manifest(tmp_path, "here", _merge_chain(levels=4, fanout=10))+ (skill_dir / "nested").mkdir()+ monkeypatch.chdir(skill_dir)++ with pytest.raises(ValueError, match=r"structured-data complexity limit exceeded: here/SKILL\.md$"):+ extract_from_skill(Path(*relative.split("/")))++ def test_repeated_key_keeps_the_last_value_like_agent_clients(self, tmp_path: Path) -> None:+ skill_dir = _write_skill_manifest(+ tmp_path, "repeated", "---\nname: first\nname: second\ndescription: Repeated name\n---\n"+ )++ entry = extract_from_skill(skill_dir)++ assert entry is not None+ assert entry.name == "second"++ @pytest.mark.usefixtures("default_int_digit_limit")+ @pytest.mark.parametrize(+ "line",+ [+ pytest.param('version: !!int ""', id="tagged-int"),+ pytest.param("version: !!bool nope", id="tagged-bool"),+ pytest.param("version: !!float x", id="tagged-float"),+ pytest.param("last_updated: 2024-02-30", id="impossible-date"),+ # int() rejects more than 4,300 decimal digits before the bit limit applies.+ pytest.param("version: " + "9" * 5_000, id="decimal-over-digit-limit"),+ ],+ )+ def test_unconstructible_value_is_invalid_frontmatter(self, tmp_path: Path, line: str) -> None:+ skill_dir = _write_skill_manifest(tmp_path, "tagged", f"---\nname: tagged\ndescription: Tagged\n{line}\n---\n")++ assert extract_from_skill(skill_dir) is None++ @pytest.mark.parametrize("value", ['""', "[]", "{}", "0", "false"])+ def test_empty_field_still_skips_the_manifest(self, tmp_path: Path, value: str) -> None:+ skill_dir = _write_skill_manifest(tmp_path, "empty", f"---\nname: empty\ndescription: {value}\n---\n")++ assert extract_from_skill(skill_dir) is None++ @pytest.mark.parametrize("block", NON_MAPPING_BLOCKS)+ def test_block_that_is_not_a_mapping_skips_the_manifest_unparsed(self, tmp_path: Path, block: str) -> None:+ skill_dir = _write_skill_manifest(tmp_path, "not-a-mapping", f"---\n{block}\n---\n# Body\n")++ assert extract_from_skill(skill_dir) is None++ @pytest.mark.parametrize(+ ("content_type", "relative", "manifest", "message"),+ [+ pytest.param(+ "skill",+ "nested/bomb/SKILL.md",+ _alias_dag("description", levels=18, fanout=2),+ "complexity limit exceeded",+ id="skill",+ ),+ pytest.param(+ "rules",+ "nested/long.mdc",+ "---\ntitle: " + "x" * 16_385 + "\ndescription: Rule\n---\n",+ "title exceeds the 16384-character limit",+ id="rules",+ ),+ pytest.param(+ "workflows",+ "nested/long/workflow-rules.mdc",+ "---\ntitle: Workflow\ndescription: " + "x" * 16_385 + "\n---\n",+ "description exceeds the 16384-character limit",+ id="workflows",+ ),+ ],+ )+ def test_collection_error_names_the_manifest(+ self, tmp_path: Path, write_skill, content_type: str, relative: str, manifest: str, message: str+ ) -> None:+ write_skill(tmp_path, "valid-skill", "A valid skill")+ _write_rule(tmp_path, "valid.mdc", "valid-rule", "A valid rule")+ _write_workflow(tmp_path, "valid-workflow", "valid-workflow", "A valid workflow")+ target = tmp_path.joinpath(*relative.split("/"))+ target.parent.mkdir(parents=True, exist_ok=True)+ target.write_bytes(manifest.encode("utf-8"))++ with pytest.raises(ValueError, match=rf"{message}: {relative}$"):+ discover_and_extract(tmp_path, content_type)++ @pytest.mark.parametrize(+ ("content_type", "relative", "manifest", "expected", "reason"),+ [+ pytest.param(+ "skill",+ "nested/odd/SKILL.md",+ "---\nname: 2048\ndescription: Odd\n---\n",+ "valid-skill",+ "non_string_name_or_description",+ id="skill",+ ),+ pytest.param(+ "skill",+ "nested/odd/SKILL.md",+ "---\n- item\n- item\n---\n# Body\n",+ "valid-skill",+ "invalid_frontmatter_mapping",+ id="skill-list",+ ),+ pytest.param(+ "rules",+ "nested/odd.mdc",+ "---\ntitle: [a, b]\ndescription: Rule\n---\n",+ "valid-rule",+ "non_string_name_or_description",+ id="rules",+ ),+ pytest.param(+ "workflows",+ "nested/odd/workflow-rules.mdc",+ "---\ntitle: Workflow\ndescription: {a: b}\n---\n",+ "valid-workflow",+ "non_string_name_or_description",+ id="workflows",+ ),+ ],+ )+ def test_collection_excludes_a_manifest_whose_field_is_not_a_string(+ self, tmp_path: Path, write_skill, content_type: str, relative: str, manifest: str, expected: str, reason: str+ ) -> None:+ write_skill(tmp_path, "valid-skill", "A valid skill")+ _write_rule(tmp_path, "valid.mdc", "valid-rule", "A valid rule")+ _write_workflow(tmp_path, "valid-workflow", "valid-workflow", "A valid workflow")+ target = tmp_path.joinpath(*relative.split("/"))+ target.parent.mkdir(parents=True, exist_ok=True)+ target.write_bytes(manifest.encode("utf-8"))++ result = discover_and_extract_detailed(tmp_path, content_type)++ assert [entry.name for entry in result.entries] == [expected]+ assert result.selected_count == 2+ assert result.exclusions == [{"path": relative, "reason": reason}]++ @pytest.mark.parametrize(+ ("frontmatter", "reason"),+ [+ pytest.param('name: odd\ndescription: Odd\nversion: !!int ""', "invalid_yaml", id="tagged-int"),+ pytest.param("name: odd\ndescription: Odd\nlast_updated: 2024-02-30", "invalid_yaml", id="impossible-date"),+ # The YAML is malformed before its root node, so it is invalid, not a non-mapping root.+ pytest.param("\tname: odd\ndescription: Odd", "invalid_yaml", id="tab-before-root"),+ pytest.param("- item\n- item", "invalid_frontmatter_mapping", id="list-root"),+ # A list root is never built, so YAML errors after it do not change the reason.+ pytest.param("- item\n- key: a: b", "invalid_frontmatter_mapping", id="list-root-then-invalid"),+ pytest.param("name: ''\ndescription: Odd", "missing_name_or_description", id="empty-name"),+ ],+ )+ def test_collection_names_why_frontmatter_is_excluded(+ self, tmp_path: Path, write_skill, frontmatter: str, reason: str+ ) -> None:+ write_skill(tmp_path, "valid-skill", "A valid skill")+ _write_skill_manifest(tmp_path, "odd", f"---\n{frontmatter}\n---\n# Body\n")++ result = discover_and_extract_detailed(tmp_path, "skill")++ assert [entry.name for entry in result.entries] == ["valid-skill"]+ assert result.exclusions == [{"path": "odd/SKILL.md", "reason": reason}]diff --git a/tests/embedding/test_similarity_limits.py b/tests/embedding/test_similarity_limits.pyindex 0ce7a729..f4a19a23 100644--- a/tests/embedding/test_similarity_limits.py+++ b/tests/embedding/test_similarity_limits.py@@ -291,3 +291,94 @@ def test_loaded_catalog_rejects_over_budget_work_before_normalizing(normalize.assert_not_called()client.embed_single.assert_not_called()++[email protected]("full_body", [False, True])+def test_alias_amplified_frontmatter_fails_the_scan_before_any_embedding_request(+ tmp_path: Path, monkeypatch, full_body: bool+) -> None:+ _collection(tmp_path, 2)+ lines = ["seed: &a0 safe"]+ lines += [f"a{level}: &a{level} [*a{level - 1}, *a{level - 1}]" for level in range(1, 19)]+ lines += ["name: alias-dag", "description: *a18"]+ (tmp_path / "alias-dag").mkdir()+ (tmp_path / "alias-dag" / "SKILL.md").write_text("---\n" + "\n".join(lines) + "\n---\n", encoding="utf-8")+ client = _client()+ monkeypatch.setattr("skillevaluator.validators.similarity.EmbeddingClient", lambda **_kwargs: client)++ (result,) = run_similarity_check(tmp_path, content_type="skill", full_body=full_body)++ assert result.passed is False+ assert result.errors == ["Manifest structured-data complexity limit exceeded: alias-dag/SKILL.md"]+ client.embed.assert_not_called()+ client.embed_single.assert_not_called()+++def _embedded_texts(client: MagicMock) -> list[str]:+ return [text for call in client.embed.call_args_list for text in call.args[0]]+++ ("manifest", "reason"),+ [+ pytest.param("---\nname: yes\ndescription: Odd\n---\n", "non_string_name_or_description", id="boolean-name"),+ pytest.param(+ "---\nname: odd\ndescription: Odd\nlast_updated: 2024-02-30\n---\n", "invalid_yaml", id="impossible-date"+ ),+ ],+)+def test_invalid_manifest_is_excluded_and_the_rest_are_compared(+ tmp_path: Path, monkeypatch, manifest: str, reason: str+) -> None:+ _collection(tmp_path, 2)+ (tmp_path / "odd").mkdir()+ (tmp_path / "odd" / "SKILL.md").write_text(manifest, encoding="utf-8")+ client = _client()+ monkeypatch.setattr("skillevaluator.validators.similarity.EmbeddingClient", lambda **_kwargs: client)++ (result,) = run_similarity_check(tmp_path, content_type="skill")++ assert result.passed is False+ assert result.is_incomplete is True+ (error,) = result.errors+ assert error.startswith("Incomplete collection coverage: 1 selected manifests were excluded")+ assert error.endswith(f"odd/SKILL.md: {reason}")+ assert sorted(_embedded_texts(client)) == ["skill-0: Description 0", "skill-1: Description 1"]++[email protected]("full_body", [False, True])+def test_frontmatter_field_over_the_catalog_text_limit_fails_in_every_mode(+ tmp_path: Path, monkeypatch, full_body: bool+) -> None:+ _collection(tmp_path, 2)+ (tmp_path / "odd").mkdir()+ (tmp_path / "odd" / "SKILL.md").write_text(+ "---\nname: odd\ndescription: " + "x" * 16_385 + "\n---\n", encoding="utf-8"+ )+ client = _client()+ monkeypatch.setattr("skillevaluator.validators.similarity.EmbeddingClient", lambda **_kwargs: client)++ (result,) = run_similarity_check(tmp_path, content_type="skill", full_body=full_body)++ assert result.passed is False+ assert result.errors == ["Manifest frontmatter description exceeds the 16384-character limit: odd/SKILL.md"]+ client.embed.assert_not_called()+ client.embed_single.assert_not_called()+++def test_name_and_description_must_fit_the_embedding_text_limit_together(tmp_path: Path, monkeypatch) -> None:+ # Each field is within the 16,384-character per-field limit, but description+ # mode embeds "name: description", which is over the same limit.+ _collection(tmp_path, 2)+ (tmp_path / "odd").mkdir()+ (tmp_path / "odd" / "SKILL.md").write_text(+ "---\nname: " + "n" * 9_000 + "\ndescription: " + "d" * 9_000 + "\n---\n", encoding="utf-8"+ )+ client = _client()+ monkeypatch.setattr("skillevaluator.validators.similarity.EmbeddingClient", lambda **_kwargs: client)++ (result,) = run_similarity_check(tmp_path, content_type="skill")++ assert result.passed is False+ assert result.errors == ["Description embedding text character limit exceeded (16384) before embedding"]+ client.embed.assert_not_called()diff --git a/tests/tier3/test_decoy_commands.py b/tests/tier3/test_decoy_commands.pyindex 3b8e44aa..4808fbaa 100644--- a/tests/tier3/test_decoy_commands.py+++ b/tests/tier3/test_decoy_commands.py@@ -288,3 +288,50 @@ def test_distinct_case_sensitive_directory_can_hold_output(tmp_path: Path, monkeassert result.exit_code == 0, result.outputassert json.loads(output.read_text(encoding="utf-8")) == manifest+++ ("frontmatter", "message"),+ [+ pytest.param(+ "name: yes\ndescription: Target.",+ "Error: Decoy selection requires a skill with a root SKILL.md whose frontmatter is a valid YAML mapping "+ "with a non-empty string name and description:",+ id="boolean-name",+ ),+ pytest.param(+ "name: target\ndescription: Use when: debugging logs",+ "Error: Decoy selection requires a skill with a root SKILL.md whose frontmatter is a valid YAML mapping "+ "with a non-empty string name and description:",+ id="invalid-yaml",+ ),+ pytest.param(+ "name: target\ndescription: Target.\nextra: " + "[" * 150 + "x" + "]" * 150,+ "Error: Manifest structured-data complexity limit exceeded: target/SKILL.md",+ id="nesting-over-limit",+ ),+ pytest.param(+ "name: target\ndescription: " + "x" * 16_385,+ "Error: Manifest frontmatter description exceeds the 16384-character limit: target/SKILL.md",+ id="long-description",+ ),+ ],+)+def test_target_frontmatter_errors_fail_cleanly_without_writing(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, frontmatter: str, message: str+) -> None:+ from skillevaluator.embedding import client as embedding_client++ client = Mock()+ monkeypatch.setattr(embedding_client, "EmbeddingClient", Mock(return_value=client))+ arguments = _arguments(tmp_path)+ (tmp_path / "skills" / "target" / "SKILL.md").write_bytes(f"---\n{frontmatter}\n---\n# Target\n".encode())+ output = tmp_path / "selection.json"++ result = CliRunner().invoke(cli, [*arguments, "--output", str(output)])++ assert result.exit_code == 1+ assert isinstance(result.exception, SystemExit), result.exception+ assert message in " ".join(result.output.split())+ assert not output.exists()+ client.embed_single.assert_not_called()diff --git a/tests/tier3/test_decoys.py b/tests/tier3/test_decoys.pyindex 8b16d2a7..ea16dec4 100644--- a/tests/tier3/test_decoys.py+++ b/tests/tier3/test_decoys.py@@ -583,3 +583,97 @@ def test_rejects_mixed_content_catalog_before_model_calls(selection: SimpleNameswith pytest.raises(ValueError, match="skill-only"):_select(selection)selection.client.embed_single.assert_not_called()+++# Decoy selection and group-mode loading read SKILL.md frontmatter as similarity-check does.+_STRING_FIELDS_REQUIRED = (+ r"requires a skill with a root SKILL\.md whose frontmatter is a valid YAML mapping "+ r"with a non-empty string name and description: "+)+NON_STRING_FRONTMATTER = [+ pytest.param("name: yes\ndescription: Help with target.", id="boolean-name"),+ pytest.param("name: 2048\ndescription: Help with target.", id="integer-name"),+ pytest.param("name: target\ndescription: [Draft]", id="list-description"),+]+# Each value reads as a string to the author, but the frontmatter is invalid YAML or the value is empty.+UNUSABLE_FRONTMATTER = [+ pytest.param("name: target\ndescription: Use when: debugging logs", id="unquoted-colon"),+ pytest.param("name: target\n\tdescription: Help with target.", id="tab-indent"),+ pytest.param("name: ''\ndescription: Help with target.", id="empty-name"),+ pytest.param("- name: target\n- description: Help with target.", id="list-root"),+]+++def _write_manifest(skill: Path, frontmatter: str) -> None:+ (skill / "SKILL.md").write_bytes(f"---\n{frontmatter}\n---\n\n# {skill.name}\n".encode())+++def _alias_chain(name: str, *, levels: int = 18) -> str:+ """Return frontmatter whose description is a list of 2**levels items built from chained aliases."""+ lines = ["seed: &a0 safe"]+ lines += [f"a{level}: &a{level} [*a{level - 1}, *a{level - 1}]" for level in range(1, levels + 1)]+ return "\n".join([*lines, f"name: {name}", f"description: *a{levels}"])++[email protected]("frontmatter", NON_STRING_FRONTMATTER)+def test_target_name_or_description_that_is_not_a_string_is_rejected(+ selection: SimpleNamespace, frontmatter: str+) -> None:+ _write_manifest(selection.target, frontmatter)++ with pytest.raises(ValueError, match=_STRING_FIELDS_REQUIRED):+ _select(selection)+ selection.client.embed_single.assert_not_called()++[email protected]("frontmatter", UNUSABLE_FRONTMATTER)+def test_target_with_invalid_yaml_or_an_empty_value_gets_the_whole_rule(+ selection: SimpleNamespace, frontmatter: str+) -> None:+ _write_manifest(selection.target, frontmatter)++ with pytest.raises(ValueError, match=_STRING_FIELDS_REQUIRED):+ _select(selection)+ selection.client.embed_single.assert_not_called()+++def test_target_without_skill_md_gets_the_whole_rule(selection: SimpleNamespace) -> None:+ (selection.target / "SKILL.md").unlink()++ with pytest.raises(ValueError, match=_STRING_FIELDS_REQUIRED):+ _select(selection)+ selection.client.embed_single.assert_not_called()+++def test_target_frontmatter_over_a_structure_limit_fails_before_model_calls(selection: SimpleNamespace) -> None:+ _write_manifest(selection.target, _alias_chain("target"))++ with pytest.raises(ValueError, match=r"^Manifest structured-data complexity limit exceeded: target/SKILL\.md$"):+ _select(selection)+ selection.client.embed_single.assert_not_called()+++ ("changed", "frontmatter", "match"),+ [+ pytest.param("target", "name: yes\ndescription: Help with target.", _STRING_FIELDS_REQUIRED, id="target"),+ pytest.param(+ "selected", "name: yes\ndescription: Help with api-debugger.", _STRING_FIELDS_REQUIRED, id="selected"+ ),+ pytest.param(+ "selected",+ _alias_chain("api-debugger"),+ r"^Manifest structured-data complexity limit exceeded: api-debugger/SKILL\.md$",+ id="selected-over-limit",+ ),+ ],+)+def test_loader_reads_frontmatter_as_selection_does(+ selection: SimpleNamespace, tmp_path: Path, changed: str, frontmatter: str, match: str+) -> None:+ path = tmp_path / "decoys.json"+ write_decoy_manifest(_select(selection), path)+ _write_manifest(selection.target if changed == "target" else selection.skills[0], frontmatter)++ with pytest.raises(ValueError, match=match):+ load_decoy_manifest(selection.target, path)diff --git a/tests/validators/test_similarity.py b/tests/validators/test_similarity.pyindex 76fc863e..66968e39 100644--- a/tests/validators/test_similarity.py+++ b/tests/validators/test_similarity.py@@ -413,6 +413,34 @@ class TestLocalCatalogWorkflow:assert any("root SKILL.md" in error for error in result.errors)mock_registry_cls.return_value.query_entry.assert_not_called()+ @pytest.mark.parametrize(+ "frontmatter",+ [+ pytest.param("name: 2048\ndescription: Numeric name", id="integer-name"),+ pytest.param("name: my-skill\ndescription: Use when: debugging logs", id="unquoted-colon"),+ pytest.param("name: ''\ndescription: Empty name", id="empty-name"),+ ],+ )+ @patch("skillevaluator.validators.similarity.EmbeddingRegistry")+ @patch("skillevaluator.validators.similarity.EmbeddingClient")+ def test_catalog_target_with_unusable_frontmatter_explains_the_frontmatter_rule(+ self, _mock_client_cls, mock_registry_cls, tmp_path: Path, frontmatter: str+ ) -> None:+ catalog = tmp_path / "catalog.json"+ catalog.write_text("{}")+ target = tmp_path / "target"+ target.mkdir()+ (target / "SKILL.md").write_text(f"---\n{frontmatter}\n---\n", encoding="utf-8")++ result = SimilarityValidator(content_type="skill", catalog_path=catalog).validate(target)++ assert not result.passed+ assert result.errors == [+ "Catalog comparison requires a skill directory with a root SKILL.md whose "+ "frontmatter is a valid YAML mapping with a non-empty string name and description"+ ]+ mock_registry_cls.return_value.query_entry.assert_not_called()+@patch("skillevaluator.validators.similarity.EmbeddingRegistry")@patch("skillevaluator.validators.similarity.EmbeddingClient")def test_catalog_extracts_only_the_root_skill(self, _mock_client_cls, mock_registry_cls, tmp_path: Path) -> None: