Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

feat(harness): flag SKILL.md locations harnesses miss or load twice (#527)

Christopher Kevin559f0d53f1e0
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d0bb49c6..194f3583 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,11 @@ All notable changes to SkillEvaluator are documented in this file.
comment helpers, and needs no Pages configuration or history branch. Future
generated bundles include the template; archived bundles and the released
onboarding CLI retain their existing hosted publisher setup.
+- `harness` rules for where `SKILL.md` lives: casing, nested manifests, symlinked
+ manifests and resources, OpenClaw symlink escapes, Codex scan depth, and files
+ Hermes leaves out of its file list. Tier 1 `schema` now reports a symlinked or
+ nested `SKILL.md` (MEDIUM), and the `skill.md` naming finding explains the
+ macOS/Linux difference.
- `harness` rules for descriptions: missing or empty descriptions, Claude Code and
Codex listing cuts, Copilot CLI's 1,024 UTF-16 unit limit, quotes escaped in the
Copilot CLI and OpenClaw lists, and Hermes's 60-character listing; `-v` shows
diff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdx
index a56ee063..115dff46 100644
--- a/docs/tier1-validation.mdx
+++ b/docs/tier1-validation.mdx
@@ -260,6 +260,7 @@ For `validate`, the default set is `schema`, `version`, `security`, `pii`, `lice
- Harness portability: patterns the spec allows but common agent harnesses (Claude Code, Codex, OpenCode, OpenClaw, Hermes, Copilot CLI) load differently are reported; override any of them with `SCHEMA.<check_name>` in the validation policy.
- A duplicate frontmatter key is HIGH (`duplicate_key`; several harnesses drop the skill), and a UTF-8 BOM is MEDIUM (`manifest_bom`).
- A description within 1,024 characters but over 1,024 UTF-16 units is MEDIUM (`description_utf16_length`), and no "Use when ..." trigger in the first 60 characters is LOW (`description_trigger`).
+ - A symlinked `SKILL.md` (`manifest_symlink`) or a `SKILL.md` nested inside the skill (`nested_manifest`) is MEDIUM.
- `metadata.author` is optional under the `default` profile. Use `Name <email@host>` to include a contact address, or a person, team, or organization name such as `NVIDIA cuOpt Team` or `Jane Doe (@jdoe)`. A value that is malformed or unsafe is a HIGH `author_format` finding: a bare email or an email outside `Name <email@host>`, stray angle brackets, control, private-use, or bidirectional formatting characters (joiners such as ZWNJ and ZWJ, which Persian, Arabic and Devanagari names need, are accepted), no letters or digits, or more than 256 characters. A bare address counts even when its domain has no dot (`root@buildhost`), while a social handle such as `Jane Doe (@jdoe)` does not. The `registry` profile requires the `Name <email@host>` form.
- `metadata.id` is an optional stable skill ID. When present it must be an exact 1–64 character lowercase ASCII slug (`^[a-z0-9]+(-[a-z0-9]+)*$`, no surrounding whitespace); it may differ from `name`. A collection run reports an ID that more than one discovered skill declares; `validate` on a catalog also lists these in `catalog-summary.json` under `metadata_id_conflicts`. Under `registry` a shared ID blocks: the run exits nonzero, and each sharing skill's entry is `"passed": false` with the conflict repeated under `catalog_findings` and counted in its `severity_counts`. A catalog validates each skill on its own, so that skill's own report was written before the catalog-wide check ran and keeps its single-skill verdict, which the entry carries alongside as `report_verdict` beside `report_scope: "single_skill"`. Uniqueness is checked only across the skills validated together, never against a registry, and a single-skill run does not claim it. The ID does not change local catalog identity, which stays keyed by content type and relative path.
- `metadata.domain` accepts one category string or a list of them, such as `domain: [backend, ai-ml]`. Values are trimmed and deduplicated in order; a blank or null scalar means no domain. A list with more than 16 entries, or a non-string, blank, or over-128-character entry, is a HIGH `Field 'metadata.domain'` finding. The 128-character limit also applies to a single string.
@@ -279,6 +280,8 @@ For `validate`, the default set is `schema`, `version`, `security`, `pii`, `lice
Frontmatter rules: duplicate keys (Codex, Copilot CLI, OpenClaw and OpenCode drop the skill), tab indentation (Codex, Copilot CLI and OpenClaw), a UTF-8 BOM (Codex), no front matter (Copilot CLI), and long custom frontmatter that never reaches the model.
Description rules: missing or empty descriptions, Claude Code and Codex listing cuts, Copilot CLI's 1,024 UTF-16 unit limit, quotes escaped in the Copilot CLI and OpenClaw lists, and Hermes's 60-character listing. `-v` also shows the description each harness puts in the model's skill list.
+
+ Location rules: `SKILL.md` casing, nested manifests, symlinked manifests and resources, OpenClaw symlink escapes, Codex scan depth, and files Hermes leaves out of its file list (outside `references/`, `templates/`, `assets/` and `scripts/`).
</Accordion>
<Accordion title="security — vulnerability scanning">
diff --git a/src/skillevaluator/constants.py b/src/skillevaluator/constants.py
index 685247dc..f42bbcf9 100644
--- a/src/skillevaluator/constants.py
+++ b/src/skillevaluator/constants.py
@@ -752,6 +752,10 @@ PLUGIN_CONFIG_MAX_BYTES = 256 * 1024
SCAN_ARTIFACT_DIRS = frozenset({"evals", ".evals", "results", ".results", "versions", ".versions"})
+# Folders no harness reads as part of a skill: VCS data, caches and dependencies. The artifact
+# folders above are not in this set because harnesses scan them like any other folder.
+SKILL_NOT_SHIPPED_DIRS = SCAN_EXCLUDED_DIRS - SCAN_ARTIFACT_DIRS
+
PLUGIN_TREE_PRUNED_DIRS = SCAN_EXCLUDED_DIRS - {"__pycache__"}
PLUGIN_TREE_MAX_DISCOVERED_PATHS = 20_000
diff --git a/src/skillevaluator/harness/overlay.py b/src/skillevaluator/harness/overlay.py
index 7f660a3b..515f6de2 100644
--- a/src/skillevaluator/harness/overlay.py
+++ b/src/skillevaluator/harness/overlay.py
@@ -5,6 +5,7 @@
from __future__ import annotations
import ast
+import os
import re
from dataclasses import dataclass, field
from pathlib import Path
@@ -12,7 +13,7 @@ from typing import Any
import yaml
-from skillevaluator.constants import DESCRIPTION_TRIGGER_PATTERN, SKILL_MANIFEST_VARIANTS
+from skillevaluator.constants import DESCRIPTION_TRIGGER_PATTERN, SKILL_MANIFEST_VARIANTS, SKILL_NOT_SHIPPED_DIRS
from skillevaluator.validators.frontmatter_parser import FRONTMATTER_PATTERN
from skillevaluator.validators.schema import _duplicate_frontmatter_keys
@@ -159,12 +160,68 @@ def _tab_indent_breaks_yaml(fm_yaml: str) -> bool:
return False
+def _shipped_paths(skill_path: Path) -> list[Path]:
+ """Files and symlinks a skill ships, without descending into VCS or cache folders."""
+ found = []
+ for folder, dirs, files in os.walk(skill_path):
+ dirs[:] = [d for d in dirs if d not in SKILL_NOT_SHIPPED_DIRS]
+ here = Path(folder)
+ found.extend(here / name for name in files)
+ found.extend(here / name for name in dirs if (here / name).is_symlink())
+ return sorted(found)
+
+
+def _bundle_files(skill_path: Path, manifest: Path) -> list[Path]:
+ return [path for path in _shipped_paths(skill_path) if path != manifest]
+
+
+_HERMES_LINKED_DIRS = ("references", "templates", "assets", "scripts")
+
+
+def _paths(items: list[str]) -> str:
+ return ", ".join(items[:3]) + (f" +{len(items) - 3} more" if len(items) > 3 else "")
+
+
+def _bundle_facts(skill_path: Path, manifest: Path) -> dict[str, Any]:
+ base = skill_path.resolve()
+ links, escapes, unlisted = [], [], []
+ for path in _bundle_files(skill_path, manifest):
+ rel = path.relative_to(skill_path).as_posix()
+ if path.is_symlink():
+ links.append(rel)
+ target = path.resolve()
+ if target != base and base not in target.parents:
+ escapes.append(rel)
+ continue
+ if rel.split("/", 1)[0] not in _HERMES_LINKED_DIRS:
+ unlisted.append(rel)
+ return {
+ "resource_symlinks": len(links),
+ "resource_symlink_paths": _paths(links),
+ "resource_symlink_escapes": len(escapes),
+ "resource_symlink_escape_paths": _paths(escapes),
+ "hermes_unlisted_files": len(unlisted),
+ "hermes_unlisted_dirs": _paths(list(dict.fromkeys(rel.split("/", 1)[0] for rel in unlisted))),
+ }
+
+
def skill_facts(skill_path: Path) -> dict[str, Any]:
"""Compute the facts profile rules refer to."""
manifest, raw, text, fm_yaml, data = _read_manifest(skill_path)
description = data.get("description")
trigger = _TRIGGER_RE.search(description) if isinstance(description, str) else None
root, depth = _install_root(skill_path)
+ target = "none"
+ if skill_path.is_symlink():
+ resolved = skill_path.resolve()
+ base = (Path(*_link_location_parts(skill_path)).parents[depth] if root else skill_path.parent).resolve()
+ target = "inside_root" if base in resolved.parents else "outside_root"
+ names = {p.name for p in skill_path.iterdir()} if skill_path.is_dir() else set()
+ nested = (
+ [p for p in _shipped_paths(skill_path) if p.name in SKILL_MANIFEST_VARIANTS and p.parent != skill_path]
+ if skill_path.is_dir()
+ else []
+ )
custom = {k: v for k, v in data.items() if str(k) not in _KNOWN_FIELDS}
name = _skill_name(skill_path, data)
return {
@@ -172,6 +229,11 @@ def skill_facts(skill_path: Path) -> dict[str, Any]:
"description_text": description if isinstance(description, str) else "",
"install_root": root,
"grouping_depth": depth,
+ "manifest_name_exact": "SKILL.md" in names,
+ "nested_skill_md": bool(nested),
+ "nested_skill_md_exact": any(p.name == "SKILL.md" and not p.is_symlink() for p in nested),
+ "skill_md_symlink": manifest.is_symlink(),
+ "skill_dir_symlink_target": target,
"frontmatter_missing": manifest.is_file() and FRONTMATTER_PATTERN.match(text) is None,
"description_missing": "description" not in data,
"description_empty": isinstance(description, str) and not description.strip(),
@@ -188,6 +250,7 @@ def skill_facts(skill_path: Path) -> dict[str, Any]:
"custom_field_chars": sum(len(v) if isinstance(v, str) else len(yaml.safe_dump(v)) for v in custom.values()),
"description_over": max(0, (len(description) if isinstance(description, str) else 0) - 1024),
"utf16_over": max(0, (_utf16_units(description)) - 1024),
+ **_bundle_facts(skill_path, manifest),
}
diff --git a/src/skillevaluator/harness/profiles/claude-code.yaml b/src/skillevaluator/harness/profiles/claude-code.yaml
index 6a51dab3..3099d755 100644
--- a/src/skillevaluator/harness/profiles/claude-code.yaml
+++ b/src/skillevaluator/harness/profiles/claude-code.yaml
@@ -11,6 +11,7 @@ rules:
- {id: CC-DESC-CUT, when: "description_codepoints > 1536", effect: degraded, outcome: listing shortened, message: "Description cut at 1,536 characters.", fix: "Keep the description at or under 1,536 characters."}
- {id: CC-DESC-MISSING, when: "description_missing or description_empty or description_type != 'str'", effect: degraded, message: "No usable description: Claude Code shows body text instead.", fix: "Add a non-empty string description."}
- {id: CC-DUP-KEY, when: "frontmatter_duplicate_keys", effect: advisory, message: "Duplicate keys: Claude Code keeps the last value (four other harnesses drop the skill).", fix: "Remove the duplicate frontmatter key."}
+ - {id: CC-CASING, when: "not manifest_name_exact", effect: degraded, message: "Non-canonical name (skill.md/Skill.md): loaded on case-insensitive filesystems (macOS default APFS, 3/3), ignored on case-sensitive Linux (3/3). Works or breaks depending on the OS.", fix: "Rename the file to SKILL.md (exact casing)."}
- {id: CC-FM-STRIPPED, when: "custom_field_chars > 200", effect: advisory, message: "Frontmatter is stripped when the skill is used, so the text in {custom_fields} never reaches the model.", fix: "Move instructions from frontmatter into the SKILL.md body."}
notes:
- "Follows symlinked skill folders and symlinked SKILL.md files, inside or outside the root."
diff --git a/src/skillevaluator/harness/profiles/codex.yaml b/src/skillevaluator/harness/profiles/codex.yaml
index 5b9eabe1..3a13ee90 100644
--- a/src/skillevaluator/harness/profiles/codex.yaml
+++ b/src/skillevaluator/harness/profiles/codex.yaml
@@ -12,6 +12,10 @@ rules:
- {id: CX-DESC-MISSING, when: "description_missing or description_empty or description_type != 'str'", effect: not_loaded, message: "Skill dropped: Codex requires a string description.", fix: "Add a non-empty string description."}
- {id: CX-STRICT-YAML, when: "frontmatter_tab_indent or frontmatter_duplicate_keys", effect: not_loaded, message: "Skill dropped: invalid YAML (tabs or duplicate keys).", fix: "Use spaces, not tabs, and remove duplicate keys."}
- {id: CX-BOM, when: "frontmatter_bom", effect: not_loaded, outcome: rejected, message: "Skill dropped: SKILL.md starts with a UTF-8 BOM.", fix: "Save SKILL.md as UTF-8 without a BOM."}
+ - {id: CX-NESTED, when: "nested_skill_md_exact", effect: degraded, message: "The nested SKILL.md is loaded as a second, separate skill.", fix: "Move the nested skill to its own folder under the skills root, or rename its SKILL.md."}
+ - {id: CX-DEPTH, when: "grouping_depth > 5", effect: not_loaded, outcome: not found, message: "Beyond Codex's scan depth of 6.", fix: "Install the skill at most 5 folders below the skills root."}
+ - {id: CX-SYMLINK-FILE, when: "skill_md_symlink", effect: not_loaded, message: "Symlinked SKILL.md files are skipped (codex#15756).", fix: "Copy SKILL.md into the folder instead of symlinking the file (a symlinked folder works)."}
+ - {id: CX-CASING, when: "not manifest_name_exact", effect: not_loaded, message: "Non-canonical name (skill.md/Skill.md): never loaded, on Linux or macOS (3/3 each).", fix: "Rename the file to SKILL.md (exact casing)."}
notes:
- "Explicit $skill injects the whole SKILL.md (145 KB tested)."
- "Follows symlinked skill folders (inside or outside the root)."
diff --git a/src/skillevaluator/harness/profiles/copilot-cli.yaml b/src/skillevaluator/harness/profiles/copilot-cli.yaml
index 82a964d5..d54dec53 100644
--- a/src/skillevaluator/harness/profiles/copilot-cli.yaml
+++ b/src/skillevaluator/harness/profiles/copilot-cli.yaml
@@ -11,6 +11,8 @@ rules:
- {id: CP-STRICT-YAML, when: "frontmatter_tab_indent or frontmatter_duplicate_keys or description_type not in ['str', 'none']", effect: not_loaded, message: "Skill dropped: Copilot's YAML parsing is strict.", fix: "Use spaces, not tabs, remove duplicate keys, and make the description a string."}
- {id: CP-NO-FRONTMATTER, when: "frontmatter_missing", effect: not_loaded, outcome: not found, message: "Skill dropped: a SKILL.md with no front matter, or an unclosed one, is not registered ('Skill not found'; canary pair 3/3, drift NOCLOSE canary in 4 releases, Trial I 2/2).", fix: "Add front matter with a name and a description."}
- {id: CP-DESC-MISSING, when: "description_missing", effect: degraded, message: "No description: Copilot shows body text instead.", fix: "Add a non-empty string description."}
+ - {id: CP-NESTED, when: "nested_skill_md", effect: advisory, message: "Nested SKILL.md is ignored.", fix: "Move the nested skill to its own folder under the skills root, or rename its SKILL.md."}
+ - {id: CP-CASING, when: "not manifest_name_exact", effect: degraded, message: "Non-canonical name (skill.md/Skill.md): loaded on case-insensitive filesystems (macOS default APFS, 3/3), ignored on case-sensitive Linux (3/3). Works or breaks depending on the OS.", fix: "Rename the file to SKILL.md (exact casing)."}
- {id: CP-FM-STRIPPED, when: "custom_field_chars > 200", effect: advisory, message: "Frontmatter is stripped when the skill is used, so the text in {custom_fields} never reaches the model.", fix: "Move instructions from frontmatter into the SKILL.md body."}
- {id: CP-DESC-ESCAPED, when: "description_has_quote", effect: advisory, outcome: text rewritten, message: "Double quotes in the description are shown to the model as &quot; in the skill list.", fix: "Use single quotes in the description if the exact text matters."}
notes:
diff --git a/src/skillevaluator/harness/profiles/hermes.yaml b/src/skillevaluator/harness/profiles/hermes.yaml
index d0b4083b..41a6c518 100644
--- a/src/skillevaluator/harness/profiles/hermes.yaml
+++ b/src/skillevaluator/harness/profiles/hermes.yaml
@@ -12,7 +12,12 @@ rules:
- {id: HM-ROOT, when: "install_root not in roots", effect: not_loaded, outcome: not found, message: "Hermes reads ~/.hermes/skills, and project .hermes/skills or .agents/skills only when trusted.", fix: "Install it under ~/.hermes/skills, or .hermes/skills or .agents/skills in a trusted repo."}
- {id: HM-DESC-60, when: "trigger_end is None or trigger_end > 60", effect: advisory, outcome: listing shortened, message: "The skill list shows only the first ~60 chars of the description (57 + '...' in 0.16, a word boundary near 59 on main), and the trigger isn't in them.", fix: "Put the trigger (\"Use when …\") in the first 57 characters."}
- {id: HM-DESC-MISSING, when: "description_missing or description_empty", effect: degraded, message: "Listed by name with an empty description.", fix: "Add a non-empty string description."}
+ - {id: HM-NESTED, when: "nested_skill_md_exact", effect: degraded, message: "The nested SKILL.md is loaded as a second, separate skill (hidden folders are walked too).", fix: "Move the nested skill to its own folder under the skills root, or rename its SKILL.md."}
+ - {id: HM-SYMLINK-FILE, when: "skill_md_symlink", effect: not_loaded, message: "Symlinked SKILL.md files are not loaded.", fix: "Copy SKILL.md into the folder instead of symlinking the file (a symlinked folder works)."}
- {id: HM-DUP-KEY, when: "frontmatter_duplicate_keys", effect: advisory, message: "Duplicate keys: last value wins.", fix: "Remove the duplicate frontmatter key."}
+ - {id: HM-CASING, when: "not manifest_name_exact", effect: not_loaded, message: "Non-canonical name (skill.md/Skill.md): never loaded, on Linux or macOS (3/3 each).", fix: "Rename the file to SKILL.md (exact casing)."}
+ - {id: HM-RESOURCE-SYMLINK, when: "resource_symlinks > 0", effect: degraded, message: "Symlinked files ({resource_symlink_paths}) are left out of the skill's file list.", fix: "Copy those files into the skill instead of symlinking them."}
+ - {id: HM-LINKED-LAYOUT, when: "hermes_unlisted_files > 0", effect: advisory, message: "{hermes_unlisted_files} files outside references/, templates/, assets/ and scripts/ ({hermes_unlisted_dirs}) are left out of the skill's file list; the model finds them only if SKILL.md names their paths.", fix: "Move supporting files under references/, templates/, assets/ or scripts/."}
notes:
- "Precedence: project > ~/.hermes/skills > skills.create_dir > external_dirs (project wins, 3/3)."
- "Large bodies are spilled and paged through in small reads."
diff --git a/src/skillevaluator/harness/profiles/openclaw.yaml b/src/skillevaluator/harness/profiles/openclaw.yaml
index 032fbdba..0039f720 100644
--- a/src/skillevaluator/harness/profiles/openclaw.yaml
+++ b/src/skillevaluator/harness/profiles/openclaw.yaml
@@ -9,6 +9,10 @@ rules:
- {id: OW-ROOT, when: "install_root not in roots", effect: not_loaded, outcome: not found, message: "OpenClaw reads skills/, .agents/skills, ~/.agents/skills and ~/.openclaw/skills.", fix: "Install it under skills/ or .agents/skills in the workspace, or ~/.openclaw/skills."}
- {id: OW-DESC-MISSING, when: "description_missing or description_empty", effect: not_loaded, outcome: rejected, message: "Skill dropped: no description.", fix: "Add a non-empty string description."}
- {id: OW-STRICT-YAML, when: "frontmatter_tab_indent or frontmatter_duplicate_keys", effect: not_loaded, outcome: rejected, message: "Skill dropped: invalid YAML (tabs or duplicate keys).", fix: "Use spaces, not tabs, and remove duplicate keys."}
+ - {id: OW-SYMLINK-ESCAPE, when: "skill_dir_symlink_target == 'outside_root'", effect: not_loaded, message: "Symlink target is outside the skills root (even inside the repo): rejected.", fix: "Copy the skill into the skills folder instead of symlinking it from elsewhere."}
+ - {id: OW-SYMLINK-FILE, when: "skill_md_symlink", effect: not_loaded, message: "Symlinked SKILL.md files are not loaded.", fix: "Copy SKILL.md into the folder instead of symlinking the file (a symlinked folder works)."}
+ - {id: OW-NESTED, when: "nested_skill_md", effect: advisory, message: "Nested SKILL.md is ignored.", fix: "Move the nested skill to its own folder under the skills root, or rename its SKILL.md."}
+ - {id: OW-CASING, when: "not manifest_name_exact", effect: degraded, message: "Non-canonical name (skill.md/Skill.md): loaded on case-insensitive filesystems (macOS default APFS, 3/3), ignored on case-sensitive Linux (3/3). Works or breaks depending on the OS.", fix: "Rename the file to SKILL.md (exact casing)."}
- {id: OW-DESC-ESCAPED, when: "description_has_quote", effect: advisory, outcome: text rewritten, message: "Double quotes in the description are shown to the model as &quot; in the skill list.", fix: "Use single quotes in the description if the exact text matters."}
notes:
- "Precedence: workspace skills/ > .agents/skills > ~/.agents/skills > managed > bundled > extra dirs (project wins, 3/3)."
diff --git a/src/skillevaluator/harness/profiles/opencode.yaml b/src/skillevaluator/harness/profiles/opencode.yaml
index 271e44a1..6bc5c0ce 100644
--- a/src/skillevaluator/harness/profiles/opencode.yaml
+++ b/src/skillevaluator/harness/profiles/opencode.yaml
@@ -12,6 +12,8 @@ rules:
- {id: OC-DESC-MISSING, when: "description_missing or description_type != 'str'", effect: not_loaded, message: "Loaded but hidden from the model's skill list (no string description).", fix: "Add a non-empty string description."}
- {id: OC-DESC-EMPTY, when: "description_empty", effect: degraded, message: "Listed by name only.", fix: "Add a non-empty string description."}
- {id: OC-DUP-KEY, when: "frontmatter_duplicate_keys", effect: not_loaded, outcome: rejected, message: "Skill dropped: duplicate frontmatter keys.", fix: "Remove the duplicate frontmatter key."}
+ - {id: OC-NESTED, when: "nested_skill_md", effect: degraded, message: "The nested SKILL.md is loaded as a second, separate skill.", fix: "Move the nested skill to its own folder under the skills root, or rename its SKILL.md."}
+ - {id: OC-CASING, when: "not manifest_name_exact", effect: degraded, message: "Non-canonical name (skill.md/Skill.md): loaded on case-insensitive filesystems (macOS default APFS, 3/3), ignored on case-sensitive Linux (3/3). Works or breaks depending on the OS.", fix: "Rename the file to SKILL.md (exact casing)."}
- {id: OC-FM-STRIPPED, when: "custom_field_chars > 200", effect: advisory, message: "Frontmatter is stripped when the skill is used, so the text in {custom_fields} never reaches the model.", fix: "Move instructions from frontmatter into the SKILL.md body."}
notes:
- "Docs (V1) say one level deep and a 1-1024 description; runtime matches the V2 docs: recursive, no length cap."
diff --git a/src/skillevaluator/validators/schema.py b/src/skillevaluator/validators/schema.py
index 96435a1a..90f6f1f7 100644
--- a/src/skillevaluator/validators/schema.py
+++ b/src/skillevaluator/validators/schema.py
@@ -28,6 +28,8 @@ from skillevaluator.constants import (
SCAN_EXCLUDED_FILES,
SCHEMA_ALLOWED_DIRS_ENV,
SKILL_MANIFEST_FILE,
+ SKILL_MANIFEST_VARIANTS,
+ SKILL_NOT_SHIPPED_DIRS,
)
from skillevaluator.logging_config import get_logger
from skillevaluator.models.result import Finding, Severity
@@ -390,8 +392,9 @@ class SchemaValidator(ValidatorBase):
message=(
"Skill manifest uses non-canonical name 'skill.md' — "
"the agentskills.io spec requires 'SKILL.md' (uppercase). "
- "On case-sensitive filesystems (Linux), this will not be "
- "recognized by spec-compliant tooling"
+ "Codex and Hermes never load it; Claude Code, OpenCode, Copilot CLI "
+ "and OpenClaw load it only on case-insensitive filesystems (macOS), "
+ "so it can work locally and be missing on Linux"
),
file_path=str(skill_md),
suggestion="Rename 'skill.md' to 'SKILL.md' to comply with the agentskills.io spec",
@@ -416,7 +419,7 @@ class SchemaValidator(ValidatorBase):
result.merge(self._validate_name_matches_directory(skill_path, frontmatter))
result.merge(self._validate_author(frontmatter, str(skill_md)))
result.merge(self._validate_metadata_id(frontmatter, str(skill_md)))
- result.merge(self._validate_portability(skill_md, frontmatter))
+ result.merge(self._validate_portability(skill_path, skill_md, frontmatter))
return result
@@ -716,12 +719,25 @@ class SchemaValidator(ValidatorBase):
metadata=dict(metadata),
)
- def _validate_portability(self, skill_md: Path, frontmatter: SkillFrontmatter) -> ValidationResult:
+ def _validate_portability(
+ self, skill_path: Path, skill_md: Path, frontmatter: SkillFrontmatter
+ ) -> ValidationResult:
"""Flag spec-valid patterns that common agent harnesses load differently or not at all."""
result = ValidationResult()
file_path = str(skill_md)
findings: list[Finding] = []
+ if skill_md.is_symlink():
+ findings.append(
+ self._portability_finding(
+ "manifest_symlink",
+ Severity.MEDIUM,
+ "SKILL.md is a symlink; Codex, Hermes and OpenClaw do not load symlinked manifests",
+ file_path,
+ "Replace the symlink with a regular SKILL.md file",
+ )
+ )
+
try:
raw = skill_md.read_bytes()
except OSError:
@@ -736,6 +752,26 @@ class SchemaValidator(ValidatorBase):
"Save SKILL.md as UTF-8 without BOM",
)
)
+ # Same folders the harness check skips: evals/ and versions/ are scanned like any other folder.
+ nested = sorted(
+ path
+ for name in SKILL_MANIFEST_VARIANTS
+ for path in skill_path.rglob(name)
+ if path.parent != skill_path
+ and not any(part in SKILL_NOT_SHIPPED_DIRS for part in path.relative_to(skill_path).parts)
+ )
+ for path in nested:
+ findings.append(
+ self._portability_finding(
+ "nested_manifest",
+ Severity.MEDIUM,
+ f"Nested skill manifest '{path.relative_to(skill_path).as_posix()}' inside this skill; "
+ "Codex, OpenCode and Hermes load it as a separate skill, others ignore it",
+ str(path),
+ "Rename the nested file or move it out of the skill folder",
+ )
+ )
+
description = frontmatter.description
utf16_units = len(description.encode("utf-16-le", "surrogatepass")) // 2
if len(description) <= DESCRIPTION_MAX_LENGTH < utf16_units:
@@ -753,7 +789,11 @@ class SchemaValidator(ValidatorBase):
trigger = _TRIGGER_RE.search(description)
if trigger is None or trigger.end() > DESCRIPTION_TRIGGER_WINDOW:
- where = "no 'Use when ...' trigger" if trigger is None else f"its trigger at characters {trigger.start()}-{trigger.end()}"
+ where = (
+ "no 'Use when ...' trigger"
+ if trigger is None
+ else f"its trigger at characters {trigger.start()}-{trigger.end()}"
+ )
findings.append(
self._portability_finding(
"description_trigger",
diff --git a/tests/harness/test_overlay.py b/tests/harness/test_overlay.py
index bdcbd77c..701dba3d 100644
--- a/tests/harness/test_overlay.py
+++ b/tests/harness/test_overlay.py
@@ -16,6 +16,7 @@ from skillevaluator.models.result import Severity
from skillevaluator.reporting.console_ui import final_check_ticker_row, summarize_tier1
from skillevaluator.tier1.commands import DEFAULT_CHECKS, run_validation
from skillevaluator.validators.harness import HarnessValidator
+from skillevaluator.validators.schema import SchemaValidator
HARNESSES = ["claude-code", "codex", "copilot-cli", "hermes", "openclaw", "opencode"]
BODY = "\n# Demo\n\n## Instructions\n\n1. Step\n\n## Examples\n\nExample.\n"
@@ -98,6 +99,17 @@ def test_copilot_shows_body_for_missing_description_but_drops_non_string(tmp_pat
assert "CP-STRICT-YAML" in _ids(_rows(listed), "copilot-cli")
+def test_symlinked_manifest(tmp_path: Path):
+ skill = _skill(tmp_path)
+ target = tmp_path / "real.md"
+ (skill / "SKILL.md").rename(target)
+ (skill / "SKILL.md").symlink_to(target)
+
+ status = _status(skill)
+
+ assert {h for h, s in status.items() if s == "not_loaded"} == {"codex", "hermes", "openclaw"}
+
+
def test_install_root_only_applies_inside_known_harness_roots(tmp_path: Path):
skill = tmp_path / ".claude" / "skills" / "demo-skill"
skill.parent.mkdir(parents=True)
@@ -320,6 +332,20 @@ def test_listed_description_projection(tmp_path: Path):
assert rows["opencode"]["listed"]["mode"] == rows["openclaw"]["listed"]["mode"] == "full"
+def test_bundle_files_symlinks_and_large_references(tmp_path: Path):
+ skill = _skill(tmp_path)
+ (skill / "references").mkdir()
+ (skill / "references" / "big.md").write_text("line\n" * 20000)
+ outside = tmp_path / "shared.md"
+ outside.write_text("shared")
+ (skill / "references" / "shared.md").symlink_to(outside)
+ facts = skill_facts(skill)
+ rows = _rows(skill)
+
+ assert facts["resource_symlinks"] == facts["resource_symlink_escapes"] == 1
+ assert "HM-RESOURCE-SYMLINK" in _ids(rows, "hermes")
+
+
def test_fix_reaches_findings_and_panels(tmp_path: Path):
skill = _skill(tmp_path, desc="Use when testing emoji. " + "\U0001f600" * 600)
result = HarnessValidator().validate(skill)
@@ -415,3 +441,49 @@ def test_trigger_cut_at_the_listing_window_counts_as_missing(tmp_path: Path):
facts = skill_facts(_skill(tmp_path, desc="x" * 55 + " Use when checking harness compatibility."))
assert facts["trigger_end"] > 60
+
+
+def test_grouped_skill_linked_to_a_sibling_group_stays_inside_the_root(tmp_path: Path):
+ root = tmp_path / ".agents" / "skills"
+ real = root / "group-b" / "demo-skill"
+ real.parent.mkdir(parents=True)
+ _skill(tmp_path / "src").rename(real)
+ link = root / "group-a" / "demo-skill"
+ link.parent.mkdir()
+ link.symlink_to(real, target_is_directory=True)
+
+ assert skill_facts(link)["skill_dir_symlink_target"] == "inside_root"
+
+
+def test_nested_manifest_under_evals_is_reported(tmp_path: Path):
+ skill = _rooted(tmp_path)
+ (skill / "evals" / "fixture").mkdir(parents=True)
+ (skill / "evals" / "fixture" / "SKILL.md").write_text("---\nname: fixture\n---\n", encoding="utf-8")
+
+ assert skill_facts(skill)["nested_skill_md"] is True
+
+
+ ("folder", "reported"),
+ [("evals", True), ("versions", True), ("references", True), ("node_modules", False), (".git", False)],
+)
+def test_schema_and_harness_agree_on_where_a_nested_manifest_counts(tmp_path: Path, folder: str, reported: bool):
+ skill = _rooted(tmp_path)
+ (skill / folder / "fixture").mkdir(parents=True)
+ (skill / folder / "fixture" / "SKILL.md").write_text("---\nname: fixture\n---\n", encoding="utf-8")
+
+ schema = [f for f in SchemaValidator().validate(skill).findings if f.check_name == "nested_manifest"]
+
+ assert skill_facts(skill)["nested_skill_md"] is reported
+ assert bool(schema) is reported
+
+
+def test_lowercase_nested_manifest_is_not_a_second_skill_in_codex(tmp_path: Path):
+ skill = _rooted(tmp_path)
+ (skill / "sub").mkdir()
+ (skill / "sub" / "skill.md").write_text("---\nname: sub\n---\n", encoding="utf-8")
+
+ facts = skill_facts(skill)
+ assert facts["nested_skill_md"] is True
+ assert facts["nested_skill_md_exact"] is False
+ assert _status(skill)["codex"] == "loads"
diff --git a/tests/validators/test_schema.py b/tests/validators/test_schema.py
index c0f23350..a0bea216 100644
--- a/tests/validators/test_schema.py
+++ b/tests/validators/test_schema.py
@@ -1614,6 +1614,43 @@ class TestSchemaPortabilityChecks:
assert result.passed, result.errors
assert [f.severity for f in _checks(result, "manifest_bom")] == ["medium"]
+ def test_symlinked_manifest_is_medium(self, tmp_path: Path):
+ skill_dir = _write_skill(tmp_path, _BASE_FM)
+ target = tmp_path / "real.md"
+ (skill_dir / "SKILL.md").rename(target)
+ (skill_dir / "SKILL.md").symlink_to(target)
+
+ result = SchemaValidator().validate(skill_dir)
+
+ assert result.passed, result.errors
+ assert len(_checks(result, "manifest_symlink")) == 1
+
+ def test_nested_manifest_is_medium(self, tmp_path: Path):
+ skill_dir = _write_skill(tmp_path, _BASE_FM)
+ (skill_dir / "references" / "inner").mkdir(parents=True)
+ (skill_dir / "references" / "inner" / "SKILL.md").write_text("---\nname: inner\n---\n")
+
+ findings = _checks(SchemaValidator().validate(skill_dir), "nested_manifest")
+
+ assert len(findings) == 1
+ assert "references/inner/SKILL.md" in findings[0].message
+
+ def test_nested_manifest_under_evals_is_reported(self, tmp_path: Path):
+ skill_dir = _write_skill(tmp_path, _BASE_FM)
+ (skill_dir / "evals" / "snapshot").mkdir(parents=True)
+ (skill_dir / "evals" / "snapshot" / "SKILL.md").write_text("---\nname: snapshot\n---\n")
+
+ findings = _checks(SchemaValidator().validate(skill_dir), "nested_manifest")
+
+ assert [f.file_path for f in findings] == [str(skill_dir / "evals" / "snapshot" / "SKILL.md")]
+
+ def test_nested_manifest_under_dependency_dir_is_ignored(self, tmp_path: Path):
+ skill_dir = _write_skill(tmp_path, _BASE_FM)
+ (skill_dir / "node_modules" / "pkg").mkdir(parents=True)
+ (skill_dir / "node_modules" / "pkg" / "SKILL.md").write_text("---\nname: pkg\n---\n")
+
+ assert not _checks(SchemaValidator().validate(skill_dir), "nested_manifest")
+
def test_description_over_utf16_limit_is_medium(self, tmp_path: Path):
description = "Use when testing. " + "\U0001f600" * 600
fm = _BASE_FM.replace("Use when checking harness portability of a skill.", description)