Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Commits on main
fix(tier2): keep catalog formatting within the byte budget (#628)
5bc1fc8 Christopher Kevin committed 5 days ago
docs(tier2): state collection manifest byte limits (#629)
5217916 Christopher Kevin committed 5 days ago
Flag skills whose descriptions are near-identical to a sibling's (#625)
b007048 Christopher Kevin committed 5 days ago
Say SkillSpector ran when a folder run rejects its scan for an unresolved reference (#624)
38a1d9e Christopher Kevin committed 5 days ago
Show each serious finding of a failed Tier 1 check in what to fix (#621)
f83c2ed Christopher Kevin committed 5 days ago
Flag a project skill with a same-named copy in a user-level skills folder (#622)
53ced5d Christopher Kevin committed 5 days ago
Note Claude Code skill fields whose value Claude Code ignores (#623)
2ff4fc3 Christopher Kevin committed 5 days ago
fix(harness): count when_to_use in Claude Code's 1,536-character cut (CC-DESC-CUT) (#620)
ed49c85 Christopher Kevin committed 5 days ago
docs(tier3): show how to install the pinned NeMo Gym commit (#614)
c0593e7 Christopher Kevin committed 5 days ago
fix(harness): Hermes lists both same-named skills (HM-DUP-NAME) (#619)
a925e48 Christopher Kevin committed 5 days ago
docs(ci): pin the recipe's actions to commit SHAs (#618)
c29eaea Christopher Kevin committed 5 days ago
fix(tier3): refresh an explicit Gym policy key that shares the Vertex ADC token (#613)
d5cfecb Christopher Kevin committed 5 days ago
docs(tier3): point the NeMo Gym example at an OpenAI Responses endpoint (#615)
fc02eb3 Christopher Kevin committed 5 days ago
docs(ci): make the jq quality gate fail closed (#617)
f0ebbaa Christopher Kevin committed 5 days ago
fix(tier3): report budget-denied arms as not run instead of empty Harbor failures (#616)
686174a Christopher Kevin committed 5 days ago
fix(harness): flag an empty Copilot CLI description (#612)
ae7e858 Christopher Kevin committed 5 days ago
Move SkillSpector's Anthropic and Bedrock default to Claude Opus 5.5 (#607)
f469462 Christopher Kevin committed 6 days ago
Name Nemotron 3.5 Lightning as the nv_build default in the docs (#606)
ca2d275 Christopher Kevin committed 6 days ago
Move the nv_build default model to Nemotron 3.5 Lightning (#605)
c35c099 Christopher Kevin committed 6 days ago
fix(tier3): mask judge and finding evidence whose quotes are escaped again (#577)
3a70bb2 Narendran Raghavan committed 6 days ago
fix(tier3): pin OpenCode 1.19.5 for managed local installs (#558)
9944bfe Narendran Raghavan committed 6 days ago
fix(tier1): stop reporting link-local addresses as PII IPs (#555)
434df36 Narendran Raghavan committed 6 days ago
fix(tier3): keep case actions from weakening dataset requirements (#588)
53dfb34 Christopher Kevin committed 6 days ago
fix(tier3): load evals/config.yml strictly before the Harbor run (#554)
e9ea45c Narendran Raghavan committed 6 days ago
fix(tier3): stop short host flags from blocking Compose sidecar values (#598)
befb63d Christopher Kevin committed 6 days ago
fix(ci): stop the Tier 1 runner missing a Ctrl-C, and deflake timing-dependent tests (#568)
59b31f6 Christopher Kevin committed 6 days ago
fix(tier1): stop deducting quality points for a missing root version (#553)
1ea050c Narendran Raghavan committed 6 days ago
fix(tier1): keep authored eval files in the Gitleaks secret scan (#552)
a3d316d Narendran Raghavan committed 6 days ago
fix(tier3): keep script names out of template graded text and say which SKILL.md becomes my SKILL.md (#571)
6946f66 Narendran Raghavan committed 6 days ago
fix(tier2): name the skill whose frontmatter stops the CI comparison catalog (#549)
48fe16a Narendran Raghavan committed 6 days ago