Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Commits on main
fix(tier3): retry malformed structured judge responses (#78)
5387261 Christopher Kevin committed 2 months ago
fix: scan Markdown headings for PII instead of skipping them (#96)
df62d82 Mohammed Imran Khan committed 2 months ago
feat(tier3): report pass@k uncertainty and paired outcomes (#63)
009aa30 Adem Vessell committed 2 months ago
fix(tier3): fail fast on rejected credentials (#75)
7119e2a Christopher Kevin committed 2 months ago
fix(tier3): score Harbor single-step null results (#74)
0d5a9b8 Christopher Kevin committed 2 months ago
Merge pull request #76 from NVIDIA/naren/release-v0.2.1
236accd Narendran Raghavan committed 2 months ago
Merge pull request #58 from rookepoole/fix/issue-57-negative-control-insights
e70f0e3 Narendran Raghavan committed 3 months ago
fix(tier3): preserve trusted negative-control evidence
1cd1c07 Christopher Kevin committed 3 months ago
Merge branch 'main' into fix/issue-57-negative-control-insights
283bd22 Christopher Kevin committed 3 months ago
Fix quality scoring with CommonMark structural parsing (#53)
68013df Christopher Kevin committed 3 months ago
Merge branch 'main' into fix/issue-57-negative-control-insights
5c41993 Rooke Poole committed 3 months ago
fix(tier3): align Anthropic endpoints and fail closed on judge errors (#56)
799f9d6 Christopher Kevin committed 3 months ago
fix(reporting): resolve pull request links from checkout (#60)
7a16f6b Daniel Gaskins committed 3 months ago
Merge branch 'main' into fix/issue-57-negative-control-insights
acd41a9 rookepoole committed 3 months ago
Merge pull request #54 from NVIDIA/feat/christopherk/judge-model-override
e8fa154 Narendran Raghavan committed 3 months ago
Merge branch 'main' into fix/issue-57-negative-control-insights
3f3332e Rooke Poole committed 3 months ago
Merge branch 'main' into feat/christopherk/judge-model-override
7666460 Narendran Raghavan committed 3 months ago
Merge pull request #47 from NVIDIA/naren/rollout-benchmark-pr25
8850da0 Narendran Raghavan committed 3 months ago
Merge branch 'main' into fix/issue-57-negative-control-insights
eed4780 Narendran Raghavan committed 3 months ago
Merge branch 'main' into naren/rollout-benchmark-pr25
d39eb52 Narendran Raghavan committed 3 months ago
chore: centralize frontier OpenAI/Anthropic chat defaults (#38)
1cec101 Christopher Kevin committed 3 months ago
Merge branch 'main' into fix/issue-57-negative-control-insights
b6a0c0b Narendran Raghavan committed 3 months ago