Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Commits on main
docs(readme): link each get-started column to its guide and fold the uv install (#609)
c6d0bd9 Christopher Kevin committed 4 days ago
test(secure-fs): make same-inode read mutation metadata observable (#652)
db1c91d Christopher Kevin committed 4 days ago
fix(tier3): name Docker address-pool exhaustion and document trial network cleanup (#557)
1244b3c Narendran Raghavan committed 4 days ago
fix(reporting): keep successful execution when Tier 3 quality fails in HTML (#634)
b201fb7 Christopher Kevin committed 4 days ago
fix(tier3): redact only credentials in NeMo Gym progress and logs (#636)
5978061 Christopher Kevin committed 4 days ago
test(tier3): model advanced ctime in the MCP source rewrite race (#659)
d2e0253 Christopher Kevin committed 4 days ago
fix(tier3): validate coverage against admitted attempt ordinals (#663)
2241694 Christopher Kevin committed 4 days ago
fix(reports): avoid completed-agent callout without complete evidence (#660)
1b1903f Christopher Kevin committed 4 days ago
docs(tier3): state the live status of the NeMo Gym backend (#645)
a2309c3 Christopher Kevin committed 4 days ago
fix(tier3): require a model when a custom NeMo Gym policy URL is set (#643)
36f4e67 Christopher Kevin committed 4 days ago
fix(reporting): include required case checks in the HTML verdict rule (#651)
16ef5cc Christopher Kevin committed 4 days ago
fix(reporting): retain fully denied run budget context in HTML (#661)
27fd7f4 Christopher Kevin committed 4 days ago
fix(reports): align benchmark headline with completed-agent verdict (#655)
90dc967 Christopher Kevin committed 4 days ago
fix(tier3): handle malformed mcp_servers.toml on NeMo Gym as Harbor does (#656)
defc6bd Christopher Kevin committed 4 days ago
fix(reporting): include required case checks in the BENCHMARK verdict rule (#644)
46336a6 Christopher Kevin committed 4 days ago
Say when to use the starter text-analyzer skill in its description (#638)
a30916f Christopher Kevin committed 4 days ago
docs(tier2): distinguish measured saved-catalog and in-memory scale (#641)
8d614ad Christopher Kevin committed 4 days ago
fix(tier3): complete fully budget-denied runs without execution errors (#658)
9fafcb9 Christopher Kevin committed 4 days ago
Name the missing tier3 extra when evaluate runs on a base install (#627)
75ffd71 Christopher Kevin committed 4 days ago
fix(reporting): describe what the BENCHMARK Overall row compares (#635)
21274fc Christopher Kevin committed 4 days ago
fix(tier3): keep one [verifier.env] table when staging native Gym tasks (#639)
fa6b18e Christopher Kevin committed 5 days ago
docs(specs): add Tier 3 datasets requirement ledger (#646)
8f1caf6 Christopher Kevin committed 5 days ago
docs(specs): add Tier 3 scoring requirement ledger (#647)
2f17963 Christopher Kevin committed 5 days ago
Report the missing tier3 extra from doctor on a base install (#626)
94f6895 Christopher Kevin committed 5 days ago
Pin SkillSpector and verify the Gitleaks download in the CI recipe (#630)
bcbd770 Christopher Kevin committed 5 days ago
docs(agent-guide): start the agent guide with the npx install and an example session (#610)
2ec6364 Christopher Kevin committed 5 days ago
feat(skills): teach the skillevaluator skill to install the CLI when missing (#570)
451b893 Christopher Kevin committed 5 days ago