Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Commits on main
fix(tier3): keep Codex apply_patch contents in FILE CHANGES evidence (#551)
707657e Narendran Raghavan committed 6 days ago
fix(scanners): enforce pinned SkillSpector during automatic selection (#550)
f5bf6c1 Narendran Raghavan committed 7 days ago
fix(reporting): write reports for non-UTF-8 file names and keep SARIF URIs valid (#536)
550bb73 Narendran Raghavan committed 7 days ago
feat(harness): flag a harness release the profiles were not verified on (#545)
a8436f6 Christopher Kevin committed 7 days ago
feat(harness): take the model setup in harness-check (#544)
f82c259 Christopher Kevin committed 7 days ago
feat(harness): check the plugin components each harness delivers (#543)
468db73 Christopher Kevin committed 7 days ago
fix(notices): list Bandit with the base dependencies (#542)
d813b6b Christopher Kevin committed 7 days ago
fix(reports): say why the default report directory was refused and how to choose another (#541)
98469e6 Christopher Kevin committed 7 days ago
fix(doctor): label scanners that SkillEvaluator provisions itself (#540)
da42ece Christopher Kevin committed 7 days ago
fix(install): bundle uv so a plain pip install can run the Tier 1 security check (#539)
2b03378 Christopher Kevin committed 7 days ago
feat(harness): check which plugin formats each harness accepts (#538)
a3ec452 Christopher Kevin committed 7 days ago
feat(harness): flag Hermes platforms and OpenClaw requirements (#537)
b010cc0 Christopher Kevin committed 7 days ago
feat(harness): flag invocation fields each harness ignores (#534)
8fe4bd9 Christopher Kevin committed 7 days ago
feat(harness): flag placeholders other harnesses pass through literally (#533)
28571e7 Christopher Kevin committed 7 days ago
feat(harness): flag SKILL.md and files over harness delivery limits (#532)
8cbe349 Christopher Kevin committed 7 days ago
docs(readme): add a PyPI badge linking to the published package (#535)
a286c57 Christopher Kevin committed 7 days ago
feat(harness): measure a skill with the skills installed beside it (#531)
f0e1bee Christopher Kevin committed 7 days ago
fix(llm): make the Anthropic refusal fallback model configurable (#494)
67959b2 Lucas Saldanha Werneck committed 7 days ago
feat(harness): flag SKILL.md locations harnesses miss or load twice (#527)
559f0d5 Christopher Kevin committed 7 days ago
fix(tier3): decode freeform Codex apply_patch literals and trust an explicit exit 0 (#530)
73db5cd Narendran Raghavan committed 7 days ago
fix(reporting): show the evidence that decided the BENCHMARK.md verdict (#524)
d7459a4 Narendran Raghavan committed 7 days ago
Scope comment template permissions to the comment job (#529)
f25d578 Christopher Kevin committed 7 days ago
feat(harness): flag descriptions harnesses drop, cut or hide (#526)
c9956ab Christopher Kevin committed 7 days ago
Reuse standard scored PR comments without hosted reports (#528)
c19b5d7 Christopher Kevin committed 7 days ago
feat(harness): flag frontmatter that harness parsers reject (#522)
f22af0b Christopher Kevin committed 7 days ago
feat(harness): add the harness-check command and compatibility panel (#521)
976f91f Christopher Kevin committed 7 days ago
feat(harness): add the advisory Tier 1 harness compatibility check (#520)
ca67b33 Christopher Kevin committed 7 days ago
fix(tier3): apply baseline-timeout tolerance to the printed verdict, report notes and token evidence (#523)
9cdc223 Narendran Raghavan committed 7 days ago
fix(tier3): match home paths in default words, pair shell quotes as the shell does, and read editor views (#517)
dbc8d17 Narendran Raghavan committed 7 days ago