Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Commits on main
Recognize Node scripts in Tier 1 skill type and script checks (#474)
fa8e7b5 Christopher Kevin committed 7 days ago
fix(tier3): confine security exemptions and sanitize hidden case IDs (#516)
5d1928b Narendran Raghavan committed 7 days ago
Explain an HTTP 404 from the default LLM model (#514)
93d595e Christopher Kevin committed 7 days ago
fix(tier1): finish the author and metadata.id review round from #499 (#513)
6000e28 Narendran Raghavan committed 7 days ago
fix(security): update SkillSpector to latest release 2.13.4 (#512)
c564469 Narendran Raghavan committed 7 days ago
fix(tier3): narrow the behavior evidence gate and correct its accounting (#511)
5f79aa3 Narendran Raghavan committed 7 days ago
Clarify literal final-response format checks (partial mitigation) (#509)
868de68 Narendran Raghavan committed 7 days ago
fix(tier3): decode Codex array/map wrappers and make the rubric supplement budget configurable (#502)
a29db5e Narendran Raghavan committed 7 days ago
fix(tier3): validate token case IDs and fix two security false positives (#507)
438b430 Narendran Raghavan committed 7 days ago
fix(reporting): stop validate from repeating its header in short terminals (#368)
e40abf2 Christopher Kevin committed 7 days ago
fix(tier1): make author and metadata.id requirements profile-controlled (#499)
be88fa8 Narendran Raghavan committed 7 days ago
docs(changelog): move post-0.9.0 entries to Unreleased (#506)
f075bb8 Narendran Raghavan committed 7 days ago
test(tier3): cover run-directory checks on local directory downloads (#479)
b6399bb Christopher Kevin committed 7 days ago
fix(tier1): keep image-only SkillSpector reports incomplete when reference evidence is unmatched (#476)
a5df6cb Christopher Kevin committed 7 days ago
fix(reporting): give exact LLM setup fixes and count named Tier 1 checks (#367)
1267ad3 Christopher Kevin committed 7 days ago
Add a three-column Get started section to the README (#488)
c38fefe Christopher Kevin committed 7 days ago
fix(tier3): withhold behavior scores built on cut final answers (#503)
c2f5a02 Narendran Raghavan committed 7 days ago
test(tier1): cover rubric rejection sanitization (#478)
49c388e Christopher Kevin committed 7 days ago
docs(ci): say ci demo only works after the setup merges (#477)
8a7b8bd Christopher Kevin committed 7 days ago
feat(cli): show the wordmark in terminals narrower than 115 columns (#369)
01535f5 Christopher Kevin committed 7 days ago
Report Semgrep file-level parse errors as a partial scan (#475)
957e8ff Christopher Kevin committed 7 days ago
docs: define every metric formula, range, units and validation status (#500)
2682dd4 Narendran Raghavan committed 7 days ago
Report budget overruns and unavailable baseline pass rates (#505)
f62ab6e Narendran Raghavan committed 8 days ago
Add security and efficiency qualification challenge pilot (#501)
b9df531 Narendran Raghavan committed 8 days ago
Fix unchecked behavior-step qualification reporting (#504)
82a2ead Narendran Raghavan committed 8 days ago
feat(tier3): add selectable NeMo Gym execution backend (#225)
a2da244 Christopher Kevin committed 8 days ago
fix(tier1): gate unmodeled hooks and explain AE1 reference gaps (#459)
68e52d7 Narendran Raghavan committed 8 days ago
chore(release): pin verified 0.9.0 CI image and bundles
738448a Christopher Kevin committed 8 days ago
Add opt-in coverage-aware grader qualification policies (#497)
f41c5c0 Narendran Raghavan committed 8 days ago