Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(harness): flag invocation fields each harness ignores (#534)
Christopher Kevin8fe4bd99bd08
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7daea4f4..4c8e04a9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -12,6 +12,10 @@ All notable changes to SkillEvaluator are documented in this file.comment helpers, and needs no Pages configuration or history branch. Futuregenerated bundles include the template; archived bundles and the releasedonboarding CLI retain their existing hosted publisher setup.+- `harness` rules for invocation fields: `disable-model-invocation`, Codex's+ `agents/openai.yaml` policy, `allowed-tools`, Claude Code-only fields and Hermes+ project trust. Tier 1 `schema` now reports fields outside the agentskills.io+ spec (LOW).- `harness` rules for placeholders: price-like `$N` text that Claude Code andOpenCode replace with arguments (with line numbers), and `$ARGUMENTS` and`${CLAUDE_SKILL_DIR}`, which only Claude Code substitutes.diff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdxindex cef5f2cd..88c496fa 100644--- a/docs/tier1-validation.mdx+++ b/docs/tier1-validation.mdx@@ -262,6 +262,7 @@ For `validate`, the default set is `schema`, `version`, `security`, `pii`, `lice- A description within 1,024 characters but over 1,024 UTF-16 units is MEDIUM (`description_utf16_length`), and no "Use when ..." trigger in the first 60 characters is LOW (`description_trigger`).- A symlinked `SKILL.md` (`manifest_symlink`) or a `SKILL.md` nested inside the skill (`nested_manifest`) is MEDIUM.- `manifest_size` is LOW above 32 KiB, MEDIUM above 50 KiB and HIGH above 256,000 bytes.+ - Fields outside the spec are LOW (`unknown_field`).- `metadata.author` is optional under the `default` profile. Use `Name <email@host>` to include a contact address, or a person, team, or organization name such as `NVIDIA cuOpt Team` or `Jane Doe (@jdoe)`. A value that is malformed or unsafe is a HIGH `author_format` finding: a bare email or an email outside `Name <email@host>`, stray angle brackets, control, private-use, or bidirectional formatting characters (joiners such as ZWNJ and ZWJ, which Persian, Arabic and Devanagari names need, are accepted), no letters or digits, or more than 256 characters. A bare address counts even when its domain has no dot (`root@buildhost`), while a social handle such as `Jane Doe (@jdoe)` does not. The `registry` profile requires the `Name <email@host>` form.- `metadata.id` is an optional stable skill ID. When present it must be an exact 1–64 character lowercase ASCII slug (`^[a-z0-9]+(-[a-z0-9]+)*$`, no surrounding whitespace); it may differ from `name`. A collection run reports an ID that more than one discovered skill declares; `validate` on a catalog also lists these in `catalog-summary.json` under `metadata_id_conflicts`. Under `registry` a shared ID blocks: the run exits nonzero, and each sharing skill's entry is `"passed": false` with the conflict repeated under `catalog_findings` and counted in its `severity_counts`. A catalog validates each skill on its own, so that skill's own report was written before the catalog-wide check ran and keeps its single-skill verdict, which the entry carries alongside as `report_verdict` beside `report_scope: "single_skill"`. Uniqueness is checked only across the skills validated together, never against a registry, and a single-skill run does not claim it. The ID does not change local catalog identity, which stays keyed by content type and relative path.- `metadata.domain` accepts one category string or a list of them, such as `domain: [backend, ai-ml]`. Values are trimmed and deduplicated in order; a blank or null scalar means no domain. A list with more than 16 entries, or a non-string, blank, or over-128-character entry, is a HIGH `Field 'metadata.domain'` finding. The 128-character limit also applies to a single string.@@ -289,6 +290,8 @@ For `validate`, the default set is `schema`, `version`, `security`, `pii`, `liceDelivery rules: Claude Code's body cap, OpenCode's 51,200-byte / 2,000-line reads and lines over 2,000 characters, OpenClaw's whole-or-nothing read against its result caps, Hermes's 100,000-character `skill_view` budget, and Codex's ~10K-token cut when the model reads `SKILL.md` through the shell.Placeholder rules: `$ARGUMENTS` and `${CLAUDE_SKILL_DIR}` are substituted only by Claude Code, and price-like `$N` text is replaced with arguments by Claude Code and OpenCode.++ Invocation rules: `disable-model-invocation` (ignored by Codex, Hermes and OpenCode), Codex's `agents/openai.yaml` implicit-invocation policy (Codex only), `allowed-tools`, Claude Code-only fields, and Hermes project trust.</Accordion><Accordion title="security — vulnerability scanning">diff --git a/src/skillevaluator/constants.py b/src/skillevaluator/constants.pyindex 4d64333e..82162dd3 100644--- a/src/skillevaluator/constants.py+++ b/src/skillevaluator/constants.py@@ -156,6 +156,9 @@ PLUGIN_CONTAINED_MODE = "contained"# Skill manifest file name variants (case-insensitive search)SKILL_MANIFEST_VARIANTS = ("SKILL.md", "skill.md")+# Top-level SKILL.md frontmatter fields defined by the agentskills.io spec.+SPEC_SKILL_FIELDS = frozenset({"name", "description", "license", "compatibility", "metadata", "allowed-tools"})+# SKILL.md size thresholds (bytes) where common agent harnesses start cutting,# refusing, or skipping the file: 16,000-64,000 character whole-file reads,# 50 KiB tool-output truncation, and a 256,000-byte discovery cap.diff --git a/src/skillevaluator/harness/overlay.py b/src/skillevaluator/harness/overlay.pyindex b1843268..871bf838 100644--- a/src/skillevaluator/harness/overlay.py+++ b/src/skillevaluator/harness/overlay.py@@ -237,6 +237,10 @@ def _tab_indent_breaks_yaml(fm_yaml: str) -> bool:return False+def _is_true(value: Any) -> bool:+ return value is True or (isinstance(value, str) and value.strip().lower() == "true")++def _shipped_paths(skill_path: Path) -> list[Path]:"""Files and symlinks a skill ships, without descending into VCS or cache folders."""found = []@@ -341,6 +345,16 @@ def _dollar_n(body: str, first_line: int) -> dict[str, Any]:return {"dollar_literal_count": prices, "dollar_literal_lines": shown, "dollar_placeholder_count": placeholders}+def _codex_implicit_disabled(skill_path: Path) -> bool:+ policy_file = skill_path / "agents" / "openai.yaml"+ try:+ data = yaml.safe_load(policy_file.read_text(encoding="utf-8")) if policy_file.is_file() else None+ except (OSError, UnicodeDecodeError, yaml.YAMLError):+ return False+ policy = data.get("policy") if isinstance(data, dict) else None+ return isinstance(policy, dict) and policy.get("allow_implicit_invocation") is False++def skill_facts(skill_path: Path, catalog: Catalog | None = None) -> dict[str, Any]:"""Compute the facts profile rules refer to (see profiles README).@@ -349,6 +363,7 @@ def skill_facts(skill_path: Path, catalog: Catalog | None = None) -> dict[str, A"""manifest, raw, text, fm_yaml, data = _read_manifest(skill_path)description = data.get("description")+ tools = data.get("allowed-tools")trigger = _TRIGGER_RE.search(description) if isinstance(description, str) else Noneroot, depth = _install_root(skill_path)target = "none"@@ -395,6 +410,12 @@ def skill_facts(skill_path: Path, catalog: Catalog | None = None) -> dict[str, A"frontmatter_bom": raw.startswith(b"\xef\xbb\xbf"),"frontmatter_duplicate_keys": bool(_duplicate_frontmatter_keys(fm_yaml)),"frontmatter_tab_indent": _tab_indent_breaks_yaml(fm_yaml),+ "allowed_tools_shape": "none"+ if tools is None+ else {str: "string", list: "list", dict: "map"}.get(type(tools), "other"),+ "disable_model_invocation": _is_true(data.get("disable-model-invocation")),+ "codex_implicit_disabled": _codex_implicit_disabled(skill_path),+ "claude_only_fields": ", ".join(f for f in _CLAUDE_ONLY_FIELDS if f in data),"custom_fields": ", ".join(map(str, custom)),"custom_field_chars": sum(len(v) if isinstance(v, str) else len(yaml.safe_dump(v)) for v in custom.values()),"skill_md_chars": len(text),diff --git a/src/skillevaluator/harness/profiles/claude-code.yaml b/src/skillevaluator/harness/profiles/claude-code.yamlindex e4cfb545..6a28e6b4 100644--- a/src/skillevaluator/harness/profiles/claude-code.yaml+++ b/src/skillevaluator/harness/profiles/claude-code.yaml@@ -14,9 +14,11 @@ rules:- {id: CC-DESC-MISSING, when: "description_missing or description_empty or description_type != 'str'", effect: degraded, message: "No usable description: Claude Code shows body text instead.", fix: "Add a non-empty string description."}- {id: CC-LIST-BUDGET, when: "catalog_desc_chars > list_budget_chars", effect: degraded, outcome: listing shortened, message: "The {catalog_size} skills in this folder have {catalog_desc_chars} description chars; Claude Code's skill list fits {list_budget_chars} (1% of the context window at 4 chars per token; 8,000 if the window is unknown), so descriptions get shortened.", fix: "Shorten descriptions (trigger first) or install fewer skills here."}- {id: CC-PRECEDENCE, when: "duplicate_name", effect: advisory, message: "Another skill is also named '{skill_name}' ({duplicate_paths}); only one is used, and a user-level copy beats a project copy.", fix: "Give each skill a unique name."}+ - {id: CC-ALLOWED-TOOLS, when: "allowed_tools_shape != 'none'", effect: advisory, message: "allowed-tools only pre-approves tools, never restricts, and applies only when the skill is invoked through the Skill tool (e.g. /name), not when the model reads the file.", fix: "Don't rely on allowed-tools to restrict tools; use the harness's permission settings."}- {id: CC-DUP-KEY, when: "frontmatter_duplicate_keys", effect: advisory, message: "Duplicate keys: Claude Code keeps the last value (four other harnesses drop the skill).", fix: "Remove the duplicate frontmatter key."}- {id: CC-BODY-CAP, when: "skill_md_bytes > 1048576", effect: not_loaded, outcome: rejected, message: "SKILL.md over 1 MB is rejected (bundle string skill_md_too_large).", fix: "Move detail into references/ files and keep SKILL.md under 1 MB."}- {id: CC-CASING, when: "not manifest_name_exact", effect: degraded, message: "Non-canonical name (skill.md/Skill.md): loaded on case-insensitive filesystems (macOS default APFS, 3/3), ignored on case-sensitive Linux (3/3). Works or breaks depending on the OS.", fix: "Rename the file to SKILL.md (exact casing)."}+ - {id: CC-CODEX-POLICY, when: "codex_implicit_disabled and not disable_model_invocation", effect: degraded, message: "agents/openai.yaml implicit-invocation policy is Codex-only: here the model can still use this skill on its own.", fix: "Also set disable-model-invocation: true (Claude Code, Copilot and OpenClaw honour it)."}- {id: CC-FM-STRIPPED, when: "custom_field_chars > 200", effect: advisory, message: "Frontmatter is stripped when the skill is used, so the text in {custom_fields} never reaches the model.", fix: "Move instructions from frontmatter into the SKILL.md body."}notes:- "Follows symlinked skill folders and symlinked SKILL.md files, inside or outside the root."diff --git a/src/skillevaluator/harness/profiles/codex.yaml b/src/skillevaluator/harness/profiles/codex.yamlindex 5b335eb4..f39ae7c7 100644--- a/src/skillevaluator/harness/profiles/codex.yaml+++ b/src/skillevaluator/harness/profiles/codex.yaml@@ -18,9 +18,12 @@ rules:- {id: CX-SYMLINK-FILE, when: "skill_md_symlink", effect: not_loaded, message: "Symlinked SKILL.md files are skipped (codex#15756).", fix: "Copy SKILL.md into the folder instead of symlinking the file (a symlinked folder works)."}- {id: CX-LIST-BUDGET, when: "catalog_desc_chars > list_budget_chars", effect: degraded, outcome: listing shortened, message: "The {catalog_size} skills in this folder have {catalog_desc_chars} description chars; Codex's skill list fits about {list_budget_chars} (2% of the context window at 4 bytes per token; 8,000 chars if unknown), so descriptions get shortened, then removed, then skills left out.", fix: "Shorten descriptions (trigger first) or install fewer skills here."}- {id: CX-DUP-NAME, when: "duplicate_name", effect: advisory, message: "Another skill is also named '{skill_name}' ({duplicate_paths}); Codex lists both, so the model sees two.", fix: "Give each skill a unique name."}+ - {id: CX-ALLOWED-TOOLS, when: "allowed_tools_shape != 'none'", effect: advisory, message: "allowed-tools is ignored by Codex.", fix: "Don't rely on allowed-tools to restrict tools; use the harness's permission settings."}- {id: CX-CASING, when: "not manifest_name_exact", effect: not_loaded, message: "Non-canonical name (skill.md/Skill.md): never loaded, on Linux or macOS (3/3 each).", fix: "Rename the file to SKILL.md (exact casing)."}+ - {id: CX-DMI-IGNORED, when: "disable_model_invocation and not codex_implicit_disabled", effect: degraded, message: "disable-model-invocation is ignored: the model still sees this skill and can use it on its own.", fix: "For Codex, add agents/openai.yaml with policy: {allow_implicit_invocation: false}."}- {id: CX-ARGUMENTS, when: "uses_arguments_placeholder", effect: degraded, message: "$ARGUMENTS is not substituted: the model sees the literal text.", fix: "Describe the expected input in words instead of $ARGUMENTS."}- {id: CX-SKILL-DIR, when: "uses_claude_skill_dir", effect: degraded, message: "${CLAUDE_SKILL_DIR} is not substituted: the model sees the literal text.", fix: "Use paths relative to SKILL.md."}+ - {id: CX-CLAUDE-ONLY, when: "claude_only_fields != ''", effect: advisory, message: "Ignored here (Claude Code only): {claude_only_fields}.", fix: "Fine to keep for Claude Code; don't depend on them elsewhere."}notes:- "Explicit $skill injects the whole SKILL.md (145 KB tested)."- "Follows symlinked skill folders (inside or outside the root)."diff --git a/src/skillevaluator/harness/profiles/copilot-cli.yaml b/src/skillevaluator/harness/profiles/copilot-cli.yamlindex 4e819223..b6ec69e9 100644--- a/src/skillevaluator/harness/profiles/copilot-cli.yaml+++ b/src/skillevaluator/harness/profiles/copilot-cli.yaml@@ -9,15 +9,19 @@ rules:- {id: CP-ROOT, when: "install_root not in roots", effect: not_loaded, outcome: not found, message: "Copilot reads .github/.agents/.claude skills (project) and ~/.copilot or ~/.agents skills (user).", fix: "Install it under .github/skills, .agents/skills or .claude/skills."}- {id: CP-DESC-LIMIT, when: "description_utf16_units > 1024", effect: not_loaded, outcome: rejected, message: "Skill dropped: description is {description_utf16_units} UTF-16 units, {utf16_over} over the 1,024 limit (an emoji counts as 2).", fix: "Cut the description by at least {utf16_over} UTF-16 units."}- {id: CP-STRICT-YAML, when: "frontmatter_tab_indent or frontmatter_duplicate_keys or description_type not in ['str', 'none']", effect: not_loaded, message: "Skill dropped: Copilot's YAML parsing is strict.", fix: "Use spaces, not tabs, remove duplicate keys, and make the description a string."}+ - {id: CP-ALLOWED-TOOLS-MAP, when: "allowed_tools_shape == 'map'", effect: not_loaded, message: "Skill dropped: allowed-tools given as a map.", fix: "Give allowed-tools as a list or a space-separated string."}- {id: CP-NO-FRONTMATTER, when: "frontmatter_missing", effect: not_loaded, outcome: not found, message: "Skill dropped: a SKILL.md with no front matter, or an unclosed one, is not registered ('Skill not found'; canary pair 3/3, drift NOCLOSE canary in 4 releases, Trial I 2/2).", fix: "Add front matter with a name and a description."}- {id: CP-DESC-MISSING, when: "description_missing", effect: degraded, message: "No description: Copilot shows body text instead.", fix: "Add a non-empty string description."}- {id: CP-LIST-BUDGET, when: "catalog_size > 32", effect: advisory, outcome: listing shortened, message: "{catalog_size} skills here: Copilot gives full descriptions to the first 32 only; the rest are listed by name.", fix: "Keep 32 or fewer skills in one place, or make names self-explanatory."}- {id: CP-NESTED, when: "nested_skill_md", effect: advisory, message: "Nested SKILL.md is ignored.", fix: "Move the nested skill to its own folder under the skills root, or rename its SKILL.md."}- {id: CP-RELOAD, when: "always", effect: advisory, message: "New skills appear only after `/skills reload` or a restart; edits to existing skills apply at once."}+ - {id: CP-ALLOWED-TOOLS, when: "allowed_tools_shape in ['string','list']", effect: advisory, message: "allowed-tools is ignored in -p (headless) mode; interactive behaviour unverified.", fix: "Don't rely on allowed-tools to restrict tools; use the harness's permission settings."}- {id: CP-CASING, when: "not manifest_name_exact", effect: degraded, message: "Non-canonical name (skill.md/Skill.md): loaded on case-insensitive filesystems (macOS default APFS, 3/3), ignored on case-sensitive Linux (3/3). Works or breaks depending on the OS.", fix: "Rename the file to SKILL.md (exact casing)."}- {id: CP-DUP-NAME, when: "duplicate_name", effect: advisory, message: "Another skill is also named '{skill_name}' ({duplicate_paths}); Copilot uses one, and the project copy wins.", fix: "Give each skill a unique name."}+ - {id: CP-CODEX-POLICY, when: "codex_implicit_disabled and not disable_model_invocation", effect: degraded, message: "agents/openai.yaml implicit-invocation policy is Codex-only: here the model can still use this skill on its own.", fix: "Also set disable-model-invocation: true (Claude Code, Copilot and OpenClaw honour it)."}- {id: CP-ARGUMENTS, when: "uses_arguments_placeholder", effect: degraded, message: "$ARGUMENTS is not substituted: the model sees the literal text.", fix: "Describe the expected input in words instead of $ARGUMENTS."}- {id: CP-SKILL-DIR, when: "uses_claude_skill_dir", effect: degraded, message: "${CLAUDE_SKILL_DIR} is not substituted: the model sees the literal text.", fix: "Use paths relative to SKILL.md."}+ - {id: CP-CLAUDE-ONLY, when: "claude_only_fields != ''", effect: advisory, message: "Ignored here (Claude Code only): {claude_only_fields}.", fix: "Fine to keep for Claude Code; don't depend on them elsewhere."}- {id: CP-FM-STRIPPED, when: "custom_field_chars > 200", effect: advisory, message: "Frontmatter is stripped when the skill is used, so the text in {custom_fields} never reaches the model.", fix: "Move instructions from frontmatter into the SKILL.md body."}- {id: CP-DESC-ESCAPED, when: "description_has_quote", effect: advisory, outcome: text rewritten, message: "Double quotes in the description are shown to the model as " in the skill list.", fix: "Use single quotes in the description if the exact text matters."}notes:diff --git a/src/skillevaluator/harness/profiles/hermes.yaml b/src/skillevaluator/harness/profiles/hermes.yamlindex 157d1c8a..4b8ac6d6 100644--- a/src/skillevaluator/harness/profiles/hermes.yaml+++ b/src/skillevaluator/harness/profiles/hermes.yaml@@ -10,18 +10,23 @@ limits: {listing_desc_chars: 60}listing: {cut: 60}rules:- {id: HM-ROOT, when: "install_root not in roots", effect: not_loaded, outcome: not found, message: "Hermes reads ~/.hermes/skills, and project .hermes/skills or .agents/skills only when trusted.", fix: "Install it under ~/.hermes/skills, or .hermes/skills or .agents/skills in a trusted repo."}+ - {id: HM-TRUST, when: "install_root in roots.project", effect: advisory, message: "Project skills are invisible until the repo is trusted (`hermes skills trust`, skills.trusted_project_dirs).", fix: "Install under ~/.hermes/skills, or run `hermes skills trust` only in a repo whose skills you would run as your own (trust admits every project skill)."}- {id: HM-DESC-60, when: "trigger_end is None or trigger_end > 60", effect: advisory, outcome: listing shortened, message: "The skill list shows only the first ~60 chars of the description (57 + '...' in 0.16, a word boundary near 59 on main), and the trigger isn't in them.", fix: "Put the trigger (\"Use when …\") in the first 57 characters."}- {id: HM-DESC-MISSING, when: "description_missing or description_empty", effect: degraded, message: "Listed by name with an empty description.", fix: "Add a non-empty string description."}- {id: HM-NESTED, when: "nested_skill_md_exact", effect: degraded, message: "The nested SKILL.md is loaded as a second, separate skill (hidden folders are walked too).", fix: "Move the nested skill to its own folder under the skills root, or rename its SKILL.md."}- {id: HM-SYMLINK-FILE, when: "skill_md_symlink", effect: not_loaded, message: "Symlinked SKILL.md files are not loaded.", fix: "Copy SKILL.md into the folder instead of symlinking the file (a symlinked folder works)."}- {id: HM-DUP-KEY, when: "frontmatter_duplicate_keys", effect: advisory, message: "Duplicate keys: last value wins.", fix: "Remove the duplicate frontmatter key."}+ - {id: HM-ALLOWED-TOOLS, when: "allowed_tools_shape != 'none'", effect: advisory, message: "allowed-tools is ignored by Hermes.", fix: "Don't rely on allowed-tools to restrict tools; use the harness's permission settings."}- {id: HM-CASING, when: "not manifest_name_exact", effect: not_loaded, message: "Non-canonical name (skill.md/Skill.md): never loaded, on Linux or macOS (3/3 each).", fix: "Rename the file to SKILL.md (exact casing)."}- {id: HM-DUP-NAME, when: "duplicate_name", effect: advisory, message: "Another skill is also named '{skill_name}' ({duplicate_paths}); Hermes uses one: project, then ~/.hermes/skills.", fix: "Give each skill a unique name."}- {id: HM-RESOURCE-SYMLINK, when: "resource_symlinks > 0", effect: degraded, message: "Symlinked files ({resource_symlink_paths}) are left out of the skill's file list.", fix: "Copy those files into the skill instead of symlinking them."}- {id: HM-LINKED-LAYOUT, when: "hermes_unlisted_files > 0", effect: advisory, message: "{hermes_unlisted_files} files outside references/, templates/, assets/ and scripts/ ({hermes_unlisted_dirs}) are left out of the skill's file list; the model finds them only if SKILL.md names their paths.", fix: "Move supporting files under references/, templates/, assets/ or scripts/."}- {id: HM-VIEW-BUDGET, when: "hermes_view_chars > 100000", effect: degraded, outcome: instructions not delivered, message: "When the model loads the skill, skill_view returns ~{hermes_view_chars} chars, over the 100,000-char tool-result budget (lower on models under ~167K context): the model gets a 1,500-char preview, not the instructions.", fix: "Keep SKILL.md under ~95,000 characters; move detail into references/ files."}+ - {id: HM-DMI-IGNORED, when: "disable_model_invocation", effect: degraded, message: "disable-model-invocation is ignored: the model still sees this skill and can use it on its own.", fix: "No equivalent setting in Hermes; don't rely on it."}+ - {id: HM-CODEX-POLICY, when: "codex_implicit_disabled and not disable_model_invocation", effect: degraded, message: "agents/openai.yaml implicit-invocation policy is Codex-only: here the model can still use this skill on its own.", fix: "No equivalent setting in Hermes."}- {id: HM-ARGUMENTS, when: "uses_arguments_placeholder", effect: degraded, message: "$ARGUMENTS is not substituted: the model sees the literal text.", fix: "Describe the expected input in words instead of $ARGUMENTS."}- {id: HM-SKILL-DIR, when: "uses_claude_skill_dir", effect: degraded, message: "${CLAUDE_SKILL_DIR} is not substituted: the model sees the literal text.", fix: "Use ${HERMES_SKILL_DIR} or paths relative to SKILL.md."}+ - {id: HM-CLAUDE-ONLY, when: "claude_only_fields != ''", effect: advisory, message: "Ignored here (Claude Code only): {claude_only_fields}.", fix: "Fine to keep for Claude Code; don't depend on them elsewhere."}notes:- "Precedence: project > ~/.hermes/skills > skills.create_dir > external_dirs (project wins, 3/3)."- "Large bodies are spilled and paged through in small reads."diff --git a/src/skillevaluator/harness/profiles/openclaw.yaml b/src/skillevaluator/harness/profiles/openclaw.yamlindex 18e3adf5..dee064a3 100644--- a/src/skillevaluator/harness/profiles/openclaw.yaml+++ b/src/skillevaluator/harness/profiles/openclaw.yaml@@ -18,12 +18,15 @@ rules:- {id: OW-READ-BUDGET, when: "openclaw_read_chars > 16000 and openclaw_read_chars <= 64000 and (context_tokens < 100000 or (context_tokens < 200000 and openclaw_read_chars > 32000))", effect: degraded, outcome: instructions not delivered, message: "SKILL.md reads as {openclaw_read_chars} characters (OpenClaw counts each CJK character as 4 or more), over the read cap for this setup: the read is refused or returns a cut prefix with a truncation notice. The cap depends on the model context (16,000 under 100K tokens, 32,000 under 200K, 64,000 above; custom providers default to 128K). Pass --context-tokens to check your setup.", fix: "Keep SKILL.md under about 15,000 characters (about 3,700 CJK characters); move detail into references/ files."}- {id: OW-COMPACT, when: "description_codepoints > 220 and (catalog_size > 150 or catalog_desc_chars > 18000)", effect: degraded, outcome: listing shortened, message: "{catalog_size} skills / {catalog_desc_chars} description chars here: over 150 skills or 18,000 chars, OpenClaw cuts descriptions to 220 chars.", fix: "Keep the description under 220 characters, trigger first."}- {id: OW-NESTED, when: "nested_skill_md", effect: advisory, message: "Nested SKILL.md is ignored.", fix: "Move the nested skill to its own folder under the skills root, or rename its SKILL.md."}+ - {id: OW-ALLOWED-TOOLS, when: "allowed_tools_shape != 'none'", effect: advisory, message: "allowed-tools is ignored by OpenClaw.", fix: "Don't rely on allowed-tools to restrict tools; use the harness's permission settings."}- {id: OW-CASING, when: "not manifest_name_exact", effect: degraded, message: "Non-canonical name (skill.md/Skill.md): loaded on case-insensitive filesystems (macOS default APFS, 3/3), ignored on case-sensitive Linux (3/3). Works or breaks depending on the OS.", fix: "Rename the file to SKILL.md (exact casing)."}- {id: OW-PROMPT-CAP, when: "catalog_size > 150 or catalog_desc_chars > 18000", effect: advisory, outcome: listing shortened, message: "{catalog_size} skills / {catalog_desc_chars} description chars here: past 150 skills or 18,000 chars OpenClaw also leaves skills out of the list (121 of 225 listed in our test).", fix: "Shorten descriptions (trigger first) or install fewer skills in one folder."}- {id: OW-DUP-NAME, when: "duplicate_name", effect: advisory, message: "Another skill is also named '{skill_name}' ({duplicate_paths}); OpenClaw uses one: workspace skills/, then .agents/skills, then user folders.", fix: "Give each skill a unique name."}- {id: OW-REF-BUDGET, when: "files_over_64000 > 0", effect: advisory, outcome: needs paging, message: "{files_over_64000} files over 64,000 bytes ({files_over_64000_paths}): OpenClaw's read cap is 16,000-64,000 UTF-8 bytes by model context, so each takes several reads.", fix: "Split large reference files into pieces under 16,000 UTF-8 bytes."}+ - {id: OW-CODEX-POLICY, when: "codex_implicit_disabled and not disable_model_invocation", effect: degraded, message: "agents/openai.yaml implicit-invocation policy is Codex-only: here the model can still use this skill on its own.", fix: "Also set disable-model-invocation: true (Claude Code, Copilot and OpenClaw honour it)."}- {id: OW-ARGUMENTS, when: "uses_arguments_placeholder", effect: degraded, message: "$ARGUMENTS is not substituted: the model sees the literal text.", fix: "Describe the expected input in words instead of $ARGUMENTS."}- {id: OW-SKILL-DIR, when: "uses_claude_skill_dir", effect: degraded, message: "${CLAUDE_SKILL_DIR} is not substituted: the model sees the literal text.", fix: "Use paths relative to SKILL.md."}+ - {id: OW-CLAUDE-ONLY, when: "claude_only_fields != ''", effect: advisory, message: "Ignored here (Claude Code only): {claude_only_fields}.", fix: "Fine to keep for Claude Code; don't depend on them elsewhere."}- {id: OW-DESC-ESCAPED, when: "description_has_quote", effect: advisory, outcome: text rewritten, message: "Double quotes in the description are shown to the model as " in the skill list.", fix: "Use single quotes in the description if the exact text matters."}notes:- "Precedence: workspace skills/ > .agents/skills > ~/.agents/skills > managed > bundled > extra dirs (project wins, 3/3)."diff --git a/src/skillevaluator/harness/profiles/opencode.yaml b/src/skillevaluator/harness/profiles/opencode.yamlindex 9c53352e..a02f92aa 100644--- a/src/skillevaluator/harness/profiles/opencode.yaml+++ b/src/skillevaluator/harness/profiles/opencode.yaml@@ -17,11 +17,15 @@ rules:- {id: OC-DOLLAR-N, when: "dollar_literal_count > 0", effect: degraded, outcome: text rewritten, message: "Run as a /command, every $N is replaced and left blank without that argument: {dollar_literal_count} price-like tokens (lines {dollar_literal_lines}) lose their numbers. The skill tool path is unaffected.", fix: "Reword amounts (e.g. USD 4.00) instead of $4.00."}- {id: OC-DUP-NAME, when: "duplicate_name", effect: degraded, message: "Another skill is also named '{skill_name}' ({duplicate_paths}); OpenCode picks a random one each run (concurrent loading, last write wins).", fix: "Give each skill a unique name."}- {id: OC-RELOAD, when: "always", effect: advisory, message: "Skill list and text are cached at startup; edits need a restart."}+ - {id: OC-ALLOWED-TOOLS, when: "allowed_tools_shape != 'none'", effect: advisory, message: "allowed-tools is ignored by OpenCode.", fix: "Don't rely on allowed-tools to restrict tools; use the harness's permission settings."}- {id: OC-CASING, when: "not manifest_name_exact", effect: degraded, message: "Non-canonical name (skill.md/Skill.md): loaded on case-insensitive filesystems (macOS default APFS, 3/3), ignored on case-sensitive Linux (3/3). Works or breaks depending on the OS.", fix: "Rename the file to SKILL.md (exact casing)."}- {id: OC-REF-CUT, when: "files_over_51200 > 0", effect: advisory, outcome: needs paging, message: "{files_over_51200} files are over 51,200 bytes or 2,000 lines ({files_over_51200_paths}); OpenCode's read tool returns at most that per call, so the model must page.", fix: "Split large files into pieces under 51,200 bytes and 2,000 lines."}- {id: OC-LONG-LINES, when: "long_line_count > 0", effect: advisory, outcome: delivery truncated, message: "{long_line_count} lines over 2,000 chars ({long_line_files}) are cut when the model reads them.", fix: "Wrap long lines (tables, minified JSON) under 2,000 characters."}+ - {id: OC-DMI-IGNORED, when: "disable_model_invocation", effect: degraded, message: "disable-model-invocation is ignored: the model still sees this skill and can use it on its own.", fix: "No equivalent setting in OpenCode; don't rely on it."}+ - {id: OC-CODEX-POLICY, when: "codex_implicit_disabled and not disable_model_invocation", effect: degraded, message: "agents/openai.yaml implicit-invocation policy is Codex-only: here the model can still use this skill on its own.", fix: "No equivalent setting in OpenCode."}- {id: OC-ARGUMENTS, when: "uses_arguments_placeholder", effect: degraded, message: "Loaded through the skill tool, $ARGUMENTS is not substituted: the model sees the literal text (a /command run substitutes it).", fix: "Describe the expected input in words instead of $ARGUMENTS."}- {id: OC-SKILL-DIR, when: "uses_claude_skill_dir", effect: degraded, message: "${CLAUDE_SKILL_DIR} is not substituted: the model sees the literal text.", fix: "Use paths relative to SKILL.md."}+ - {id: OC-CLAUDE-ONLY, when: "claude_only_fields != ''", effect: advisory, message: "Ignored here (Claude Code only): {claude_only_fields}.", fix: "Fine to keep for Claude Code; don't depend on them elsewhere."}- {id: OC-FM-STRIPPED, when: "custom_field_chars > 200", effect: advisory, message: "Frontmatter is stripped when the skill is used, so the text in {custom_fields} never reaches the model.", fix: "Move instructions from frontmatter into the SKILL.md body."}notes:- "Docs (V1) say one level deep and a 1-1024 description; runtime matches the V2 docs: recursive, no length cap."diff --git a/src/skillevaluator/validators/schema.py b/src/skillevaluator/validators/schema.pyindex c54bf807..bc78ef8b 100644--- a/src/skillevaluator/validators/schema.py+++ b/src/skillevaluator/validators/schema.py@@ -31,6 +31,7 @@ from skillevaluator.constants import (SKILL_MANIFEST_VARIANTS,SKILL_MD_SIZE_THRESHOLDS,SKILL_NOT_SHIPPED_DIRS,+ SPEC_SKILL_FIELDS,)from skillevaluator.logging_config import get_loggerfrom skillevaluator.models.result import Finding, Severity@@ -823,6 +824,20 @@ class SchemaValidator(ValidatorBase):))+ unknown = sorted(set(frontmatter.model_extra or {}) - SPEC_SKILL_FIELDS)+ if unknown:+ findings.append(+ self._portability_finding(+ "unknown_field",+ Severity.LOW,+ f"Frontmatter field(s) outside the agentskills.io spec: {', '.join(unknown)}; "+ "agent harnesses do not pass them to the model",+ file_path,+ "Remove them or move them under 'metadata'",+ fields=unknown,+ )+ )+for finding in findings:result.add_finding(finding)return resultdiff --git a/tests/harness/test_overlay.py b/tests/harness/test_overlay.pyindex da1fad61..f2a1b24f 100644--- a/tests/harness/test_overlay.py+++ b/tests/harness/test_overlay.py@@ -479,6 +479,35 @@ def test_bundle_files_symlinks_and_large_references(tmp_path: Path):assert "OC-REF-CUT" in _ids(rows, "opencode")+def test_disable_model_invocation_is_ignored_by_three_harnesses(tmp_path: Path):+ rows = _rows(_skill(tmp_path, extra="disable-model-invocation: true\n"))++ flagged = {h for h, r in rows.items() if any(f["id"].endswith("DMI-IGNORED") for f in r["findings"])}+ assert flagged == {"codex", "hermes", "opencode"}+++def test_codex_policy_makes_disable_model_invocation_effective_in_codex(tmp_path: Path):+ skill = _skill(tmp_path, extra="disable-model-invocation: true\n")+ (skill / "agents").mkdir()+ (skill / "agents" / "openai.yaml").write_text("policy:\n allow_implicit_invocation: false\n")+ rows = _rows(skill)++ flagged = {h for h, r in rows.items() if any(f["id"].endswith("DMI-IGNORED") for f in r["findings"])}+ assert flagged == {"hermes", "opencode"}+ assert not any(f["id"].endswith("CODEX-POLICY") for r in rows.values() for f in r["findings"])+ assert rows["codex"]["status"] == "loads"+++def test_codex_implicit_policy_is_codex_only(tmp_path: Path):+ skill = _skill(tmp_path)+ (skill / "agents").mkdir()+ (skill / "agents" / "openai.yaml").write_text("policy:\n allow_implicit_invocation: false\n")+ rows = _rows(skill)++ flagged = {h for h, r in rows.items() if any(f["id"].endswith("CODEX-POLICY") for f in r["findings"])}+ assert flagged == set(HARNESSES) - {"codex"}++def test_claude_placeholders_reach_other_harnesses_literally(tmp_path: Path):rows = _rows(_skill(tmp_path, body=BODY + "Run ${CLAUDE_SKILL_DIR}/run.sh $ARGUMENTS\n"))@@ -488,6 +517,19 @@ def test_claude_placeholders_reach_other_harnesses_literally(tmp_path: Path):assert not rows["claude-code"]["findings"]+def test_gating_and_claude_only_fields(tmp_path: Path):+ extra = 'argument-hint: "<file>"\nplatforms: [macos]\n'+ skill = _skill(tmp_path, extra=extra)+ manifest = skill / "SKILL.md"+ manifest.write_text(+ manifest.read_text().replace(" author:", ' openclaw: {"requires": {"bins": ["pdftotext"]}}\n author:')+ )+ rows = _rows(skill)++ assert "OC-CLAUDE-ONLY" in _ids(rows, "opencode")+ assert not any(f["id"].endswith("CLAUDE-ONLY") for f in rows["claude-code"]["findings"])++def test_every_rule_has_a_fix_and_placeholders_are_filled(tmp_path: Path):for profile in load_profiles("all"):for rule in profile["rules"]:@@ -499,6 +541,18 @@ def test_every_rule_has_a_fix_and_placeholders_are_filled(tmp_path: Path):assert {"OW-COMPACT", "OW-PROMPT-CAP"} <= _ids(rows, "openclaw")+def test_rule_ids_named_in_messages_exist():+ profiles = load_profiles("all")+ known = {rule["id"] for profile in profiles for rule in profile["rules"]}+ prefixes = "|".join(sorted({rule_id.split("-")[0] for rule_id in known}))+ rule_id_pattern = rf"\b(?:{prefixes})-[A-Z0-9]+(?:-[A-Z0-9]+)*"+ for profile in profiles:+ for rule in profile["rules"]:+ text = f"{rule['message']} {rule.get('fix', '')}"+ for mentioned in re.findall(rule_id_pattern, text):+ assert mentioned in known, f"{rule['id']} refers to {mentioned}, which no profile defines"++def test_fix_reaches_findings_and_panels(tmp_path: Path):skill = _skill(tmp_path, desc="Use when testing emoji. " + "\U0001f600" * 600)result = HarnessValidator().validate(skill)diff --git a/tests/validators/test_schema.py b/tests/validators/test_schema.pyindex 623e7042..dd041fd9 100644--- a/tests/validators/test_schema.py+++ b/tests/validators/test_schema.py@@ -1705,6 +1705,14 @@ class TestSchemaPortabilityChecks:assert bool(findings) is expectedassert all(f.severity == "low" for f in findings)+ def test_unknown_top_level_fields_are_low(self, tmp_path: Path):+ fm = _BASE_FM + "owner: team-x\nversion: 1.0.0\nlicense: MIT\nallowed-tools: Read\n"+ result = SchemaValidator().validate(_write_skill(tmp_path, fm))++ findings = _checks(result, "unknown_field")+ assert result.passed, result.errors+ assert [f.metadata["fields"] for f in findings] == [["owner", "version"]]+def test_portability_severity_can_be_overridden_by_policy(self, tmp_path: Path):from skillevaluator.validators.policy import ValidationPolicy