Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Commits on main
fix(validators): stop the version-label regex stalling the PII scan on long identifiers (#684)
b72805b Christopher Kevin committed 4 days ago
test(tier3): allow SDK setup time in invalid-region probe test (#683)
a9b1232 Christopher Kevin committed 4 days ago
fix(tier3): count a protected path after a short option and a variable (#679)
2a385d0 Narendran Raghavan committed 4 days ago
fix(tier3): mask five more credential forms in security finding evidence (#562)
79d1cef Narendran Raghavan committed 4 days ago
fix(tier3): retain partial arm evidence in collector JSON (#670)
cc3563e Christopher Kevin committed 4 days ago
fix(tier3): keep correctly unused negative-case advice off the skill (#671)
066d435 Christopher Kevin committed 4 days ago
fix(report): keep Steps per Eval Case chart readable with long case IDs (#675)
64808e9 Christopher Kevin committed 4 days ago
docs(tier1): say a PowerShell backtick before a space is not a quoted split (#674)
abac7c5 Narendran Raghavan committed 4 days ago
test(reporting): import the CHANGELOG helpers without the reporting test modules (#672)
d356390 Narendran Raghavan committed 4 days ago
docs(readme): say what the CI template does in the get-started CI column (#673)
3a8061a Christopher Kevin committed 4 days ago
fix(tier1): keep the missing-reference line check and its warning accurate (#564)
96f03c1 Narendran Raghavan committed 4 days ago
fix(tier1): keep the hook gate failure FAIL beside an incomplete scan under a policy override (#563)
aeb3dbb Narendran Raghavan committed 4 days ago
fix(tier3): end the rich progress table's line off a terminal, and find a case's trials by its recorded case (#668)
c06b5fa Narendran Raghavan committed 4 days ago
fix(reporting): correct Tier 3 failure, verdict and lift wording in reports and docs (#561)
ae8cd10 Narendran Raghavan committed 4 days ago
docs(ci): say the first Tier 1 run installs SkillSpector and Gitleaks (#669)
2d0660a Narendran Raghavan committed 4 days ago
Restore native NVIDIA report presentation and reduce clutter (#560)
186134b Narendran Raghavan committed 4 days ago
fix(tier3): keep per-trial activation diagnostics for failed arms (#664)
940ed63 Christopher Kevin committed 4 days ago
fix(tier3): exclude failed arms from quality headlines (#657)
0ca1b33 Christopher Kevin committed 4 days ago
docs(dev-guide): keep the directory-snapshot race note with the other fixture notes (#667)
301700c Narendran Raghavan committed 4 days ago
test(tier3): model advanced ctime in the restored-mtime summary race (#654)
d41ec54 Christopher Kevin committed 4 days ago
fix(skill): correct consent, local-mode and eval-case issues from #570/#601 review (#666)
c4dc649 Narendran Raghavan committed 4 days ago
docs(skill): make the skillevaluator skill ask before installing, spending or writing cases (#601)
a2a4bac Christopher Kevin committed 4 days ago
test(ci): use the intended global config in the URL-rewrite fixture (#642)
c883dec Christopher Kevin committed 4 days ago
test(secure-fs): make the directory snapshot race independent of clock ticks (#653)
b60c4ab Christopher Kevin committed 4 days ago
fix(cli): validate the folder of a SKILL.md or workflow-rules.mdc argument (#556)
602cbc0 Narendran Raghavan committed 4 days ago
fix(tier3): skip sourced files and longer paths in the keyword write rule (#559)
ea3c0ee Narendran Raghavan committed 4 days ago
Score a missing command named after the skill as the agent's choice (#662)
6f0e2fc Christopher Kevin committed 4 days ago
test(tier3): time the masking linear-time checks in CPU time, in turn (#665)
6fab14c Narendran Raghavan committed 4 days ago
docs(skill): explain gateway keys, gate versus score and incomplete publication (#640)
e45c091 Christopher Kevin committed 4 days ago