Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Commits on main
fix(container): run the runtime image as an unprivileged user (#163)
d34500f Deepak Jain (DJ) committed 1 month ago
fix(reporting): pin Chart.js and verify script integrity (#162)
5426d87 Deepak Jain (DJ) committed 1 month ago
feat(cli): simplify tier workflows and improve evaluation feedback (#152)
4dcbe37 Christopher Kevin committed 1 month ago
docs: add docs/livestream/blog links to README (#159)
41e5469 Moshe Abramovitch committed 1 month ago
feat(reporting): record the evaluated source identity on benchmark cards (#131)
a8d1614 Max Freedom Pollard committed 2 months ago
fix(embedding): reject non-progressing chunk windows (#142)
cdb4dbb Deepak Jain (DJ) committed 2 months ago
fix(tier3): prevent false positives in network exfiltration checks (#141)
0818709 Karl Weinmeister committed 2 months ago
fix(security): accept fully covered documentation scans (#143)
d4bcb3f Deepak Jain (DJ) committed 2 months ago
feat(cli): parallel catalog validation and fleet rollup (#123)
baf43f5 Mohammed Imran Khan committed 2 months ago
feat(cli): write catalog-summary.json after catalog validate (#121)
2af26f5 Mohammed Imran Khan committed 2 months ago
fix: map Harbor trial folders to eval case ids (#104)
d979722 Mohammed Imran Khan committed 2 months ago
feat(model-catalog): allow plain HTTP for named hosts (#135)
104ccb7 Tomauskasz committed 2 months ago
fix: accept UTF-8 BOM in SKILL.md frontmatter (#99)
121261e Mohammed Imran Khan committed 2 months ago
fix(hygiene): detect unpinned dependencies with environment markers (#107)
f2d8f5d Rio Yu committed 2 months ago
fix: evaluate full SPDX license expressions (#98)
7ddcafb Mohammed Imran Khan committed 2 months ago
fix: flag Windows PII usernames that start with s (#95)
b882b16 Mohammed Imran Khan committed 2 months ago
fix: treat tools/ as an executable directory (#103)
c923d77 Mohammed Imran Khan committed 2 months ago
fix: do not let frontmatter MIT hide a GPL LICENSE file (#97)
3bfba44 Mohammed Imran Khan committed 2 months ago
feat: add SARIF reporter for GitHub Code Scanning (#117)
ea9b239 Mohammed Imran Khan committed 2 months ago
fix: keep --llm-verify file reads inside the skill (#101)
f1a5b4c Mohammed Imran Khan committed 2 months ago
fix: skip gitleaks test dirs, not substring names (#102)
14a888e Mohammed Imran Khan committed 2 months ago
fix(tier3): detect a results directory the Docker daemon cannot see (#80)
31391b5 genkey6 committed 2 months ago