Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Commits on main
fix(tier3): harden detached Gym cleanup against signals and inspection errors (#602)
78e3614 Christopher Kevin committed 6 days ago
fix(tier3): reject failed skill lookup activation evidence (#590)
dd87206 Christopher Kevin committed 6 days ago
fix(tier3): project Gym OpenCode skills into the workdir (#589)
a7fadd7 Christopher Kevin committed 6 days ago
fix(tier3): retain price provenance and charge coverage (#596)
f718dd3 Christopher Kevin committed 6 days ago
fix(tier3): probe Gym policy endpoint reachability in doctor (#593)
fc7b039 Christopher Kevin committed 6 days ago
fix(tier3): surface declarative-check coverage in all reports (#586)
f4eefdc Christopher Kevin committed 6 days ago
fix(tier3): scope suggestion and finding evidence to the cited case (#587)
86af411 Christopher Kevin committed 6 days ago
fix(tier3): preserve OpenCode bash exit status in trajectories (#585)
caab02a Christopher Kevin committed 6 days ago
fix(tier3): keep failed judging out of skill failure advice (#579)
a313ae1 Christopher Kevin committed 6 days ago
fix(tier3): accept declarative checks in grader validation (#582)
0ae966c Christopher Kevin committed 6 days ago
fix(tier3): complete Gym cleanup progress after teardown (#592)
18b1baa Christopher Kevin committed 6 days ago
fix(bridge): preserve NVIDIA Build token usage (#578)
22509d2 Christopher Kevin committed 6 days ago
fix(tier3): reserve measured usage for in-flight attempts (#595)
eba4338 Christopher Kevin committed 6 days ago
fix(tier3): report final budgets and incomplete run verdicts (#594)
e7414e4 Christopher Kevin committed 6 days ago
fix(reports): distinguish skipped baselines from incomplete execution (#575)
9e332e2 Christopher Kevin committed 6 days ago
fix(codex): preserve credential-prefixed native plugin launches (#584)
fccab03 Christopher Kevin committed 6 days ago
fix(hermes): validate the selected Build credential during preflight (#583)
e7eb419 Christopher Kevin committed 6 days ago
fix(tier3): make native NeMo Gym Claude Code runs reach the policy model and discover skills (#581)
d0113bd Christopher Kevin committed 6 days ago
fix(tier3): preserve pass thresholds in incomplete runs (#580)
11048a4 Christopher Kevin committed 6 days ago
fix(tier1): report a nested SKILL.md once on case-insensitive filesystems (#599)
a8a67e4 Christopher Kevin committed 6 days ago
fix(tier3): clean up run-owned detached Gym servers (#591)
0fbd41e Christopher Kevin committed 6 days ago
fix(cli): keep success metadata readable in PowerShell (#576)
8d6d6c5 Christopher Kevin committed 6 days ago
fix(tier3): keep the staged case ID and custom grader explanations in NeMo Gym results (#572)
c830041 Christopher Kevin committed 6 days ago
docs(readme): mention harness compatibility in the Tier 1 row (#566)
0d5e356 Christopher Kevin committed 6 days ago
fix(harness): drop internal test notes from finding messages (#565)
f8a2988 Christopher Kevin committed 6 days ago
fix(tier3): count token usage per case correctly and keep the terminal token notes accurate (#546)
33b382f Narendran Raghavan committed 6 days ago
fix(tier3): say progress stopped when a plan line is refused, and keep the running spinner moving (#548)
e466e83 Narendran Raghavan committed 6 days ago
fix(tier3): mask credentials at JSON-escaped line starts in judge evidence and keep RAGAS refusals (#547)
3f303b3 Narendran Raghavan committed 6 days ago