Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(harness): drop internal test notes from finding messages (#565)
Christopher Kevinf8a2988ac7a8
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5cb18207..87e9808c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -247,6 +247,8 @@ All notable changes to SkillEvaluator are documented in this file.### Fixed+- The Copilot CLI `CP-NO-FRONTMATTER` finding and the plugin component findings+ of `harness-check` no longer show internal test notes in their messages.- The behavior judge's FILE CHANGES evidence now covers Codex `apply_patch`calls. Codex passes its patch as a single `input` argument, which the writeevidence helpers did not read, so a Codex patch contributed no FILE CHANGESdiff --git a/src/skillevaluator/harness/plugin.py b/src/skillevaluator/harness/plugin.pyindex 1c80ef09..88ef2571 100644--- a/src/skillevaluator/harness/plugin.py+++ b/src/skillevaluator/harness/plugin.py@@ -51,8 +51,8 @@ OUTCOME_EFFECT = {OUTCOME_TEXT = {"trust_gated": "{n} {kind}: run only after the user trusts the plugin's current content (or with a per-run trust bypass).","deferred": "{n} {kind}: registered; tool schemas reach the model only when it searches for or asks to use them.",- "registered": "{n} {kind}: started or registered, but nothing reached the model in canary runs.",- "not_observed": "{n} {kind}: not registered or delivered in canary runs.",+ "registered": "{n} {kind}: started or registered, but nothing reached the model when tested.",+ "not_observed": "{n} {kind}: not registered or delivered when tested.",}diff --git a/src/skillevaluator/harness/profiles/copilot-cli.yaml b/src/skillevaluator/harness/profiles/copilot-cli.yamlindex 5d712d6e..e4da1b82 100644--- a/src/skillevaluator/harness/profiles/copilot-cli.yaml+++ b/src/skillevaluator/harness/profiles/copilot-cli.yaml@@ -10,7 +10,7 @@ rules:- {id: CP-DESC-LIMIT, when: "description_utf16_units > 1024", effect: not_loaded, outcome: rejected, message: "Skill dropped: description is {description_utf16_units} UTF-16 units, {utf16_over} over the 1,024 limit (an emoji counts as 2).", fix: "Cut the description by at least {utf16_over} UTF-16 units.", confidence: high}- {id: CP-STRICT-YAML, when: "frontmatter_tab_indent or frontmatter_duplicate_keys or description_type not in ['str', 'none']", effect: not_loaded, message: "Skill dropped: Copilot's YAML parsing is strict.", fix: "Use spaces, not tabs, remove duplicate keys, and make the description a string.", confidence: high}- {id: CP-ALLOWED-TOOLS-MAP, when: "allowed_tools_shape == 'map'", effect: not_loaded, message: "Skill dropped: allowed-tools given as a map.", fix: "Give allowed-tools as a list or a space-separated string.", confidence: high}- - {id: CP-NO-FRONTMATTER, when: "frontmatter_missing", effect: not_loaded, outcome: not found, message: "Skill dropped: a SKILL.md with no front matter, or an unclosed one, is not registered ('Skill not found'; canary pair 3/3, drift NOCLOSE canary in 4 releases, Trial I 2/2).", fix: "Add front matter with a name and a description.", confidence: high}+ - {id: CP-NO-FRONTMATTER, when: "frontmatter_missing", effect: not_loaded, outcome: not found, message: "Skill dropped: a SKILL.md with no front matter, or an unclosed one, is not registered ('Skill not found').", fix: "Add front matter with a name and a description.", confidence: high}- {id: CP-DESC-MISSING, when: "description_missing", effect: degraded, message: "No description: Copilot shows body text instead.", fix: "Add a non-empty string description.", confidence: high}- {id: CP-LIST-BUDGET, when: "catalog_size > 32", effect: advisory, outcome: listing shortened, message: "{catalog_size} skills here: Copilot gives full descriptions to the first 32 only; the rest are listed by name.", fix: "Keep 32 or fewer skills in one place, or make names self-explanatory.", confidence: medium}- {id: CP-NESTED, when: "nested_skill_md", effect: advisory, message: "Nested SKILL.md is ignored.", fix: "Move the nested skill to its own folder under the skills root, or rename its SKILL.md.", confidence: medium}