Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(harness): flag frontmatter that harness parsers reject (#522)
Christopher Kevinf22af0b73354
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f40d2a02..92c6687d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -6,6 +6,10 @@ All notable changes to SkillEvaluator are documented in this file.### Added+- `harness` rules for frontmatter parsing: duplicate keys and tab indentation+ (strict YAML parsers drop the skill), a UTF-8 BOM (Codex), no front matter+ (Copilot CLI) and long custom frontmatter that never reaches the model. Tier 1+ `schema` now reports duplicate frontmatter keys (HIGH) and a UTF-8 BOM (MEDIUM).- New `skillevaluator harness-check <skill>` command and a "Harness compatibility"panel after the Tier 1 results of `validate`: one line per harness with its mostserious finding and an outcome label (`not found`, `rejected`, `listingdiff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdxindex 15fc909b..e8d40b72 100644--- a/docs/tier1-validation.mdx+++ b/docs/tier1-validation.mdx@@ -257,6 +257,8 @@ For `validate`, the default set is `schema`, `version`, `security`, `pii`, `lice- The standard hierarchy is `skills/<skill-name>/` or `team-skills/<team>/<skill-name>/`. A nonstandard standalone location is reported as an advisory finding.- Recognized skill-root subdirectories are `agents/`, `references/`, `scripts/`, `assets/`, `evals/`, `tests/`, `tools/`, and `config/`; anything else gets a LOW advisory finding. Extend the set per repository with `SKILLEVALUATOR_SCHEMA_ALLOWED_DIRS` (comma-separated names are added to the defaults, never replace them).- `SKILL.md` should stay at or below 500 lines; larger files receive an advisory finding.+ - Harness portability: patterns the spec allows but common agent harnesses (Claude Code, Codex, OpenCode, OpenClaw, Hermes, Copilot CLI) load differently are reported; override any of them with `SCHEMA.<check_name>` in the validation policy.+ - A duplicate frontmatter key is HIGH (`duplicate_key`; several harnesses drop the skill), and a UTF-8 BOM is MEDIUM (`manifest_bom`).- `metadata.author` is optional under the `default` profile. Use `Name <email@host>` to include a contact address, or a person, team, or organization name such as `NVIDIA cuOpt Team` or `Jane Doe (@jdoe)`. A value that is malformed or unsafe is a HIGH `author_format` finding: a bare email or an email outside `Name <email@host>`, stray angle brackets, control, private-use, or bidirectional formatting characters (joiners such as ZWNJ and ZWJ, which Persian, Arabic and Devanagari names need, are accepted), no letters or digits, or more than 256 characters. A bare address counts even when its domain has no dot (`root@buildhost`), while a social handle such as `Jane Doe (@jdoe)` does not. The `registry` profile requires the `Name <email@host>` form.- `metadata.id` is an optional stable skill ID. When present it must be an exact 1–64 character lowercase ASCII slug (`^[a-z0-9]+(-[a-z0-9]+)*$`, no surrounding whitespace); it may differ from `name`. A collection run reports an ID that more than one discovered skill declares; `validate` on a catalog also lists these in `catalog-summary.json` under `metadata_id_conflicts`. Under `registry` a shared ID blocks: the run exits nonzero, and each sharing skill's entry is `"passed": false` with the conflict repeated under `catalog_findings` and counted in its `severity_counts`. A catalog validates each skill on its own, so that skill's own report was written before the catalog-wide check ran and keeps its single-skill verdict, which the entry carries alongside as `report_verdict` beside `report_scope: "single_skill"`. Uniqueness is checked only across the skills validated together, never against a registry, and a single-skill run does not claim it. The ID does not change local catalog identity, which stays keyed by content type and relative path.- `metadata.domain` accepts one category string or a list of them, such as `domain: [backend, ai-ml]`. Values are trimmed and deduplicated in order; a blank or null scalar means no domain. A list with more than 16 entries, or a non-string, blank, or over-128-character entry, is a HIGH `Field 'metadata.domain'` finding. The 128-character limit also applies to a single string.@@ -272,6 +274,8 @@ For `validate`, the default set is `schema`, `version`, `security`, `pii`, `liceThe profiles ship with the package and come from repeated runs of each harness in which the request sent to the model was captured. The check doesn't change pass/fail: its findings are MEDIUM (`not_loaded`), LOW (`degraded`) or INFO (`advisory`), so it never fails a run unless a policy raises those severities. Install-location rules apply only when the skill sits under a harness root such as `.claude/skills` or `.agents/skills`. Use `--harness-profiles DIR` to try newer profile YAMLs without a release.`validate` also shows a "Harness compatibility" panel after Tier 1, `! harness` in the checks line when a harness drops or degrades the skill, a `harness` line in the Tier 1 summary (kept out of the advisories count) and a note in the result box. `skillevaluator harness-check <skill>` runs only this check and shows the same panel, updated as each harness is checked. It accepts a folder of skills, `--harness`, `-v` and `--json PATH`, always exits 0, and points to `validate --tiers 1` when Tier 1 schema also fails. By default the panel is compact: one line per harness with its most serious finding (not loaded, then degraded, then note; ties go to the rule listed first in the profile) labelled with its outcome (`not found`, `rejected`, `listing shortened`, `delivery truncated` and so on), then `+N more` and the number of notes. `-v` (also on `validate`) lists every finding with its fix and each harness's verified version.++ Frontmatter rules: duplicate keys (Codex, Copilot CLI, OpenClaw and OpenCode drop the skill), tab indentation (Codex, Copilot CLI and OpenClaw), a UTF-8 BOM (Codex), no front matter (Copilot CLI), and long custom frontmatter that never reaches the model.</Accordion><Accordion title="security — vulnerability scanning">diff --git a/src/skillevaluator/harness/overlay.py b/src/skillevaluator/harness/overlay.pyindex 564301aa..98d114fc 100644--- a/src/skillevaluator/harness/overlay.py+++ b/src/skillevaluator/harness/overlay.py@@ -14,10 +14,36 @@ import yamlfrom skillevaluator.constants import SKILL_MANIFEST_VARIANTSfrom skillevaluator.validators.frontmatter_parser import FRONTMATTER_PATTERN+from skillevaluator.validators.schema import _duplicate_frontmatter_keysPROFILES_DIR = Path(__file__).parent / "profiles"HARNESS_STATUSES = ("loads", "degraded", "not_loaded")_EFFECT_RANK = {"advisory": 0, "degraded": 1, "not_loaded": 2}+_CLAUDE_ONLY_FIELDS = (+ "argument-hint",+ "when_to_use",+ "context",+ "agent",+ "model",+ "hooks",+)+_KNOWN_FIELDS = frozenset(+ (+ "name",+ "description",+ "license",+ "compatibility",+ "metadata",+ "allowed-tools",+ "disable-model-invocation",+ "user-invocable",+ "platforms",+ "version",+ "author",+ "tags",+ *_CLAUDE_ONLY_FIELDS,+ )+)_FACT_RE = re.compile(r"\{([a-z0-9_]+)\}")_KNOWN_ROOTS = (".agents/skills",@@ -119,15 +145,35 @@ def _skill_name(skill_path: Path, data: dict[str, Any]) -> str:return name.strip() if isinstance(name, str) and name.strip() else skill_path.name+def _tab_indent_breaks_yaml(fm_yaml: str) -> bool:+ """A tab in a line's indentation that makes the frontmatter invalid YAML (a tab inside a block scalar is fine)."""+ if not any("\t" in line[: len(line) - len(line.lstrip())] for line in fm_yaml.splitlines()):+ return False+ try:+ yaml.safe_load(fm_yaml)+ except yaml.YAMLError:+ return True+ return False++def skill_facts(skill_path: Path) -> dict[str, Any]:"""Compute the facts profile rules refer to."""- *_, data = _read_manifest(skill_path)+ manifest, raw, text, fm_yaml, data = _read_manifest(skill_path)+ description = data.get("description")root, depth = _install_root(skill_path)+ custom = {k: v for k, v in data.items() if str(k) not in _KNOWN_FIELDS}name = _skill_name(skill_path, data)return {"skill_name": name,"install_root": root,"grouping_depth": depth,+ "frontmatter_missing": manifest.is_file() and FRONTMATTER_PATTERN.match(text) is None,+ "description_type": type(description).__name__ if description is not None else "none",+ "frontmatter_bom": raw.startswith(b"\xef\xbb\xbf"),+ "frontmatter_duplicate_keys": bool(_duplicate_frontmatter_keys(fm_yaml)),+ "frontmatter_tab_indent": _tab_indent_breaks_yaml(fm_yaml),+ "custom_fields": ", ".join(map(str, custom)),+ "custom_field_chars": sum(len(v) if isinstance(v, str) else len(yaml.safe_dump(v)) for v in custom.values()),}diff --git a/src/skillevaluator/harness/profiles/claude-code.yaml b/src/skillevaluator/harness/profiles/claude-code.yamlindex 784293d5..f3a72251 100644--- a/src/skillevaluator/harness/profiles/claude-code.yaml+++ b/src/skillevaluator/harness/profiles/claude-code.yaml@@ -8,6 +8,8 @@ roots:listing: {cut: 1536}rules:- {id: CC-ROOT, when: "install_root not in roots", effect: not_loaded, outcome: not found, message: "Claude Code reads only .claude/skills and ~/.claude/skills; .agents/skills is not scanned.", fix: "Install it under .claude/skills or ~/.claude/skills (a symlink from .agents/skills works)."}+ - {id: CC-DUP-KEY, when: "frontmatter_duplicate_keys", effect: advisory, message: "Duplicate keys: Claude Code keeps the last value (four other harnesses drop the skill).", fix: "Remove the duplicate frontmatter key."}+ - {id: CC-FM-STRIPPED, when: "custom_field_chars > 200", effect: advisory, message: "Frontmatter is stripped when the skill is used, so the text in {custom_fields} never reaches the model.", fix: "Move instructions from frontmatter into the SKILL.md body."}notes:- "Follows symlinked skill folders and symlinked SKILL.md files, inside or outside the root."- "Nested SKILL.md, CRLF, BOM, tabs, uppercase names: no effect observed."diff --git a/src/skillevaluator/harness/profiles/codex.yaml b/src/skillevaluator/harness/profiles/codex.yamlindex 478506b9..cfe4333b 100644--- a/src/skillevaluator/harness/profiles/codex.yaml+++ b/src/skillevaluator/harness/profiles/codex.yaml@@ -8,6 +8,8 @@ limits: {scan_depth: 6, skips_hidden_dirs: true, catalog_desc_codepoints: 1024,listing: {cut: 1024}rules:- {id: CX-ROOT, when: "install_root not in roots", effect: not_loaded, outcome: not found, message: "Codex reads .agents/skills and .codex/skills (project and user).", fix: "Install it under .agents/skills or .codex/skills."}+ - {id: CX-STRICT-YAML, when: "frontmatter_tab_indent or frontmatter_duplicate_keys", effect: not_loaded, message: "Skill dropped: invalid YAML (tabs or duplicate keys).", fix: "Use spaces, not tabs, and remove duplicate keys."}+ - {id: CX-BOM, when: "frontmatter_bom", effect: not_loaded, outcome: rejected, message: "Skill dropped: SKILL.md starts with a UTF-8 BOM.", fix: "Save SKILL.md as UTF-8 without a BOM."}notes:- "Explicit $skill injects the whole SKILL.md (145 KB tested)."- "Follows symlinked skill folders (inside or outside the root)."diff --git a/src/skillevaluator/harness/profiles/copilot-cli.yaml b/src/skillevaluator/harness/profiles/copilot-cli.yamlindex 465b4d95..0bc9d994 100644--- a/src/skillevaluator/harness/profiles/copilot-cli.yaml+++ b/src/skillevaluator/harness/profiles/copilot-cli.yaml@@ -6,6 +6,9 @@ roots:user: [~/.copilot/skills, ~/.agents/skills]rules:- {id: CP-ROOT, when: "install_root not in roots", effect: not_loaded, outcome: not found, message: "Copilot reads .github/.agents/.claude skills (project) and ~/.copilot or ~/.agents skills (user).", fix: "Install it under .github/skills, .agents/skills or .claude/skills."}+ - {id: CP-STRICT-YAML, when: "frontmatter_tab_indent or frontmatter_duplicate_keys or description_type not in ['str', 'none']", effect: not_loaded, message: "Skill dropped: Copilot's YAML parsing is strict.", fix: "Use spaces, not tabs, remove duplicate keys, and make the description a string."}+ - {id: CP-NO-FRONTMATTER, when: "frontmatter_missing", effect: not_loaded, outcome: not found, message: "Skill dropped: a SKILL.md with no front matter, or an unclosed one, is not registered ('Skill not found'; canary pair 3/3, drift NOCLOSE canary in 4 releases, Trial I 2/2).", fix: "Add front matter with a name and a description."}+ - {id: CP-FM-STRIPPED, when: "custom_field_chars > 200", effect: advisory, message: "Frontmatter is stripped when the skill is used, so the text in {custom_fields} never reaches the model.", fix: "Move instructions from frontmatter into the SKILL.md body."}notes:- "Docs say skills only in immediate subfolders; runtime finds them 3 grouping levels deep."- "Follows symlinks inside and outside the checkout (SDK docs say inside only)."diff --git a/src/skillevaluator/harness/profiles/hermes.yaml b/src/skillevaluator/harness/profiles/hermes.yamlindex 91ea8711..1c97b708 100644--- a/src/skillevaluator/harness/profiles/hermes.yaml+++ b/src/skillevaluator/harness/profiles/hermes.yaml@@ -9,6 +9,7 @@ roots:limits: {listing_desc_chars: 60}rules:- {id: HM-ROOT, when: "install_root not in roots", effect: not_loaded, outcome: not found, message: "Hermes reads ~/.hermes/skills, and project .hermes/skills or .agents/skills only when trusted.", fix: "Install it under ~/.hermes/skills, or .hermes/skills or .agents/skills in a trusted repo."}+ - {id: HM-DUP-KEY, when: "frontmatter_duplicate_keys", effect: advisory, message: "Duplicate keys: last value wins.", fix: "Remove the duplicate frontmatter key."}notes:- "Precedence: project > ~/.hermes/skills > skills.create_dir > external_dirs (project wins, 3/3)."- "Large bodies are spilled and paged through in small reads."diff --git a/src/skillevaluator/harness/profiles/openclaw.yaml b/src/skillevaluator/harness/profiles/openclaw.yamlindex 5ed5987c..09bd2af5 100644--- a/src/skillevaluator/harness/profiles/openclaw.yaml+++ b/src/skillevaluator/harness/profiles/openclaw.yaml@@ -7,6 +7,7 @@ roots:limits: {max_skills_in_prompt: 150, max_prompt_chars: 18000, compact_desc_chars: 220, max_skill_file_bytes: 256000, read_budget: "16,000-64,000 weighted chars of the skills_read result by model context"}rules:- {id: OW-ROOT, when: "install_root not in roots", effect: not_loaded, outcome: not found, message: "OpenClaw reads skills/, .agents/skills, ~/.agents/skills and ~/.openclaw/skills.", fix: "Install it under skills/ or .agents/skills in the workspace, or ~/.openclaw/skills."}+ - {id: OW-STRICT-YAML, when: "frontmatter_tab_indent or frontmatter_duplicate_keys", effect: not_loaded, outcome: rejected, message: "Skill dropped: invalid YAML (tabs or duplicate keys).", fix: "Use spaces, not tabs, and remove duplicate keys."}notes:- "Precedence: workspace skills/ > .agents/skills > ~/.agents/skills > managed > bundled > extra dirs (project wins, 3/3)."- "Description given as a YAML list is stringified and listed. BOM, CRLF, bare colons tolerated."diff --git a/src/skillevaluator/harness/profiles/opencode.yaml b/src/skillevaluator/harness/profiles/opencode.yamlindex b4025e04..d1078c8f 100644--- a/src/skillevaluator/harness/profiles/opencode.yaml+++ b/src/skillevaluator/harness/profiles/opencode.yaml@@ -8,6 +8,8 @@ limits: {scan_depth: unlimited, desc_cap: none, list_cap: none, tool_output: "2,listing: {}rules:- {id: OC-ROOT, when: "install_root not in roots", effect: not_loaded, outcome: not found, message: "OpenCode reads .opencode, .claude and .agents skill folders.", fix: "Install it under .opencode/skills, .claude/skills or .agents/skills."}+ - {id: OC-DUP-KEY, when: "frontmatter_duplicate_keys", effect: not_loaded, outcome: rejected, message: "Skill dropped: duplicate frontmatter keys.", fix: "Remove the duplicate frontmatter key."}+ - {id: OC-FM-STRIPPED, when: "custom_field_chars > 200", effect: advisory, message: "Frontmatter is stripped when the skill is used, so the text in {custom_fields} never reaches the model.", fix: "Move instructions from frontmatter into the SKILL.md body."}notes:- "Docs (V1) say one level deep and a 1-1024 description; runtime matches the V2 docs: recursive, no length cap."- "Tolerant of tabs, bare colons and BOM. Follows all symlinks tested."diff --git a/src/skillevaluator/validators/schema.py b/src/skillevaluator/validators/schema.pyindex 4236a44a..f2a4b62b 100644--- a/src/skillevaluator/validators/schema.py+++ b/src/skillevaluator/validators/schema.py@@ -181,6 +181,38 @@ def _display_metadata_id(value: Any) -> str:return value+_UTF8_BOM = b"\xef\xbb\xbf"+++def _duplicate_frontmatter_keys(frontmatter_yaml: str) -> list[tuple[str, int]]:+ """Return ``(dotted_key, line)`` for every repeated mapping key (PyYAML keeps the last silently)."""+ try:+ root = yaml.compose(frontmatter_yaml, Loader=yaml.SafeLoader)+ except yaml.YAMLError:+ return []+ duplicates: list[tuple[str, int]] = []+ stack: list[tuple[yaml.Node, str]] = [(root, "")] if root is not None else []+ visited: set[int] = set()+ while stack:+ node, prefix = stack.pop()+ if id(node) in visited:+ continue+ visited.add(id(node))+ if isinstance(node, yaml.MappingNode):+ seen: set[str] = set()+ for key_node, value_node in node.value:+ if not isinstance(key_node, yaml.ScalarNode) or key_node.value == "<<":+ continue+ key = f"{prefix}{key_node.value}"+ if key_node.value in seen:+ duplicates.append((key, key_node.start_mark.line + 2))+ seen.add(key_node.value)+ stack.append((value_node, f"{key}."))+ elif isinstance(node, yaml.SequenceNode):+ stack.extend((item, prefix) for item in node.value)+ return sorted(duplicates, key=lambda item: item[1])++class SchemaValidator(ValidatorBase):"""Validates skill schema and repository structure per SkillEvaluator spec.@@ -380,6 +412,7 @@ class SchemaValidator(ValidatorBase):result.merge(self._validate_name_matches_directory(skill_path, frontmatter))result.merge(self._validate_author(frontmatter, str(skill_md)))result.merge(self._validate_metadata_id(frontmatter, str(skill_md)))+ result.merge(self._validate_portability(skill_md))return result@@ -453,6 +486,23 @@ class SchemaValidator(ValidatorBase):)return result+ for key, line_number in _duplicate_frontmatter_keys(frontmatter_yaml):+ result.add_finding(+ Finding(+ category="SCHEMA",+ severity=self.policy.severity_for("SCHEMA", "duplicate_key", default=Severity.HIGH),+ check_name="duplicate_key",+ message=(+ f"Duplicate frontmatter key '{key}': YAML parsers disagree on it "+ "(Codex, OpenCode, Copilot CLI and OpenClaw drop the skill; others keep the last value)"+ ),+ file_path=file_path,+ line_number=line_number,+ suggestion=f"Keep a single '{key}' entry",+ metadata={"key": key},+ )+ )+try:frontmatter = SkillFrontmatter.model_validate(data)result.add_success(@@ -643,6 +693,49 @@ class SchemaValidator(ValidatorBase):return result+ def _portability_finding(+ self,+ check_name: str,+ default: Severity,+ message: str,+ file_path: str,+ suggestion: str,+ **metadata: object,+ ) -> Finding:+ return Finding(+ category="SCHEMA",+ severity=self.policy.severity_for("SCHEMA", check_name, default=default),+ check_name=check_name,+ message=message,+ file_path=file_path,+ suggestion=suggestion,+ metadata=dict(metadata),+ )++ def _validate_portability(self, skill_md: Path) -> ValidationResult:+ """Flag spec-valid patterns that common agent harnesses load differently or not at all."""+ result = ValidationResult()+ file_path = str(skill_md)+ findings: list[Finding] = []++ try:+ raw = skill_md.read_bytes()+ except OSError:+ raw = b""+ if raw.startswith(_UTF8_BOM):+ findings.append(+ self._portability_finding(+ "manifest_bom",+ Severity.MEDIUM,+ "SKILL.md starts with a UTF-8 byte-order mark; Codex drops the skill",+ file_path,+ "Save SKILL.md as UTF-8 without BOM",+ )+ )+ for finding in findings:+ result.add_finding(finding)+ return result+def _validate_line_count(self, skill_md: Path) -> ValidationResult:"""Warn if SKILL.md exceeds recommended line count."""result = ValidationResult()diff --git a/tests/harness/test_overlay.py b/tests/harness/test_overlay.pyindex 7127852d..637f1ccb 100644--- a/tests/harness/test_overlay.py+++ b/tests/harness/test_overlay.py@@ -62,6 +62,24 @@ def test_clean_skill_loads_everywhere(tmp_path: Path):assert set(_status(_skill(tmp_path)).values()) == {"loads"}+def test_duplicate_key_is_not_loaded_by_strict_parsers(tmp_path: Path):+ status = _status(_skill(tmp_path, extra="description: Use when a second value is given.\n"))++ assert status["codex"] == status["copilot-cli"] == status["openclaw"] == status["opencode"] == "not_loaded"+ assert status["claude-code"] == "loads"+++def test_bom_is_not_loaded_by_codex_only(tmp_path: Path):+ skill = _skill(tmp_path)+ manifest = skill / "SKILL.md"+ manifest.write_bytes(b"\xef\xbb\xbf" + manifest.read_bytes())++ status = _status(skill)++ assert status["codex"] == "not_loaded"+ assert {h for h, s in status.items() if s != "loads"} == {"codex"}++def test_install_root_only_applies_inside_known_harness_roots(tmp_path: Path):skill = tmp_path / ".claude" / "skills" / "demo-skill"skill.parent.mkdir(parents=True)@@ -274,6 +292,10 @@ def test_tier1_summary_gives_harness_its_own_line(tmp_path: Path):assert "harness" not in labels.get("advisories", "")+def _rule_ids(skill: Path, harness: str, config: dict | None = None) -> set[str]:+ return {f["id"] for r in evaluate_harnesses(skill, harness, config=config) for f in r.findings}++def test_native_root_skips_root_rules(tmp_path):skill = tmp_path / ".claude" / "skills" / "demo"skill.mkdir(parents=True)@@ -286,6 +308,15 @@ def test_native_root_skips_root_rules(tmp_path):assert not {i for i in ids({"native_root": True}) if i.endswith("-ROOT")}+def test_copilot_drops_skill_without_front_matter(tmp_path: Path):+ skill = tmp_path / ".github" / "skills" / "bare"+ skill.mkdir(parents=True)+ (skill / "SKILL.md").write_text("# Bare\n\nNo front matter here.\n")++ assert "CP-NO-FRONTMATTER" in _rule_ids(skill, "copilot-cli")+ assert "CP-NO-FRONTMATTER" not in _rule_ids(_skill(tmp_path), "copilot-cli")++def test_harness_note_names_the_skill_not_the_harness():block = {"a": [{"harness": "codex", "status": "not_loaded"}, {"harness": "hermes", "status": "degraded"}],@@ -293,3 +324,19 @@ def test_harness_note_names_the_skill_not_the_harness():}assert harness_note(block) == "2 skills won't load in codex, opencode · skill loads degraded in hermes"+++def test_tab_inside_a_block_scalar_is_not_tab_indentation(tmp_path: Path):+ skill = _rooted(tmp_path)+ (skill / "SKILL.md").write_text(+ "---\nname: demo-skill\ndescription: |\n Use when checking.\n \tindented\n---\nBody\n"+ )++ assert skill_facts(skill)["frontmatter_tab_indent"] is False+++def test_tab_after_spaces_that_breaks_yaml_is_flagged(tmp_path: Path):+ skill = _rooted(tmp_path)+ (skill / "SKILL.md").write_text("---\nname: demo-skill\nmetadata:\n \tauthor: x\n---\nBody\n")++ assert skill_facts(skill)["frontmatter_tab_indent"] is Truediff --git a/tests/validators/test_schema.py b/tests/validators/test_schema.pyindex c5ebbd7f..6ac2bd8b 100644--- a/tests/validators/test_schema.py+++ b/tests/validators/test_schema.py@@ -10,6 +10,7 @@ from pathlib import Pathimport pytest+from skillevaluator.models.result import Severityfrom skillevaluator.models.skill import SkillMetadatafrom skillevaluator.validators.schema import SchemaValidator@@ -1541,3 +1542,86 @@ class TestMetadataIdPreviewBounds:(finding,) = [finding for finding in result.findings if finding.check_name == "metadata_id_format"]assert "must be a string" in finding.messageassert len(finding.line_content) <= 32+++def _write_skill(+ root: Path, frontmatter: str, body: str = "# Portable\n\n## Instructions\n\n1. Step\n\n## Examples\n\nExample.\n"+) -> Path:+ skill_dir = root / "skills" / "portable-skill"+ skill_dir.mkdir(parents=True)+ (skill_dir / "SKILL.md").write_text(f"---\n{frontmatter}---\n\n{body}", encoding="utf-8")+ return skill_dir+++_BASE_FM = (+ "name: portable-skill\n"+ "description: Use when checking harness portability of a skill.\n"+ "metadata:\n author: Test User <[email protected]>\n"+)+++def _checks(result, name: str) -> list:+ return [finding for finding in result.findings if finding.check_name == name]+++class TestSchemaPortabilityChecks:+ """Spec-valid patterns that agent harnesses load differently (Tier 1, harness-neutral)."""++ def test_clean_skill_has_no_portability_findings(self, tmp_path: Path):+ result = SchemaValidator().validate(_write_skill(tmp_path, _BASE_FM))++ assert result.passed, result.errors+ names = {+ "duplicate_key",+ "manifest_bom",+ "manifest_symlink",+ "manifest_size",+ "nested_manifest",+ "description_utf16_length",+ "description_trigger",+ "unknown_field",+ }+ assert not [f for f in result.findings if f.check_name in names]++ def test_duplicate_top_level_key_is_high(self, tmp_path: Path):+ result = SchemaValidator().validate(_write_skill(tmp_path, _BASE_FM + "description: Second value.\n"))++ findings = _checks(result, "duplicate_key")+ assert not result.passed+ assert [f.metadata["key"] for f in findings] == ["description"]+ assert findings[0].severity == "high"+ assert findings[0].line_number == 6++ def test_duplicate_nested_key_is_reported_with_path(self, tmp_path: Path):+ fm = _BASE_FM + " author: Other User <[email protected]>\n"+ result = SchemaValidator().validate(_write_skill(tmp_path, fm))++ assert [f.metadata["key"] for f in _checks(result, "duplicate_key")] == ["metadata.author"]++ def test_same_key_in_different_mappings_is_not_duplicate(self, tmp_path: Path):+ fm = _BASE_FM + " tags:\n - name: a\n - name: b\n"+ result = SchemaValidator().validate(_write_skill(tmp_path, fm))++ assert not _checks(result, "duplicate_key")++ def test_bom_is_medium_and_does_not_fail(self, tmp_path: Path):+ skill_dir = _write_skill(tmp_path, _BASE_FM)+ manifest = skill_dir / "SKILL.md"+ manifest.write_bytes(b"\xef\xbb\xbf" + manifest.read_bytes())++ result = SchemaValidator().validate(skill_dir)++ assert result.passed, result.errors+ assert [f.severity for f in _checks(result, "manifest_bom")] == ["medium"]++ def test_portability_severity_can_be_overridden_by_policy(self, tmp_path: Path):+ from skillevaluator.validators.policy import ValidationPolicy++ skill_dir = _write_skill(tmp_path, _BASE_FM)+ manifest = skill_dir / "SKILL.md"+ manifest.write_bytes(b"\xef\xbb\xbf" + manifest.read_bytes())+ policy = ValidationPolicy(severity_overrides={"SCHEMA.manifest_bom": Severity.HIGH})+ result = SchemaValidator(policy=policy).validate(skill_dir)++ assert not result.passed+ assert [f.severity for f in _checks(result, "manifest_bom")] == ["high"]