Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Commits on main
feat(tier3): compare a run against a prior skill version (#387)
b61b1ef Christopher Kevin committed 11 days ago
feat(tier3): import production trace exports as reviewed eval cases (#378)
fe60fd3 Christopher Kevin committed 11 days ago
feat(tier3): add a lift history view over retained runs (#388)
98d1279 Christopher Kevin committed 11 days ago
feat(tier3): add validate-grader qualification command and reference suite (#383)
ad238e9 Christopher Kevin committed 11 days ago
feat(tier3): reuse a compatible retained no-skill baseline (#384)
01b1e63 Christopher Kevin committed 11 days ago
feat(tier3): add run budget limits with admission control and partial results (#404)
e6d8c61 Christopher Kevin committed 11 days ago
feat(tier3): add run-level next steps to Tier 3 reports (#377)
0bd8920 Christopher Kevin committed 11 days ago
docs(tier3): document findings.json and advisory severity thresholds (#372)
8e30825 Christopher Kevin committed 11 days ago
feat(reports): name Skill Lift as the headline and label pass@k as supporting reliability (#397)
3fe3763 Christopher Kevin committed 11 days ago
docs: add a reproducible public starter and terminal demo (#374)
06d8bf6 Christopher Kevin committed 11 days ago
feat(tier3): fail cases on unmet goals and security issues, independent of lift (#395)
3cc6779 Christopher Kevin committed 11 days ago
feat(tier3): declare deterministic checks in datasets without a custom grader (#403)
f6304e5 Christopher Kevin committed 11 days ago
fix(reporting): explain each dimension, its evidence and baseline in reports (#376)
3fbb019 Christopher Kevin committed 11 days ago
feat(tier3): import precomputed custom metrics (#375)
2adb700 Christopher Kevin committed 11 days ago
feat(tier3): write deductions as plain-language reasons with evidence (#399)
9feffaa Christopher Kevin committed 11 days ago
feat(reports): show how many cases and attempts each score rests on (#406)
89c9170 Christopher Kevin committed 11 days ago
feat(tier3): report whether a dataset is big enough for its claims (#400)
2a5e19f Christopher Kevin committed 11 days ago
fix(tier3): keep LLM suggestions grounded, distinct and requirement-preserving (#380)
1a40f52 Christopher Kevin committed 11 days ago
feat(tier3): add experimental Hermes agent with native NVIDIA Build routing (#385)
782c800 Christopher Kevin committed 11 days ago
feat(datasets): choose a grounding source for generated datasets and keep authored expectations (#401)
f256e03 Christopher Kevin committed 11 days ago
fix(reporting): publish the same Overall lift in BENCHMARK.md as every other output (#373)
8b25e5e Christopher Kevin committed 11 days ago
feat(datasets): add lossless preview-first dataset migration (#435)
ed1f002 Narendran Raghavan committed 11 days ago
fix(tier3): show the run's Overall lift in tier3 compare when a metric is N/A in one arm (#386)
d81315e Christopher Kevin committed 11 days ago
feat(tier3): report dataset coverage and label one-case starters (#381)
770c599 Christopher Kevin committed 11 days ago
fix(tier3): report missing tools as environment problems, not skill defects (#379)
a41d5a7 Christopher Kevin committed 11 days ago
feat(tier3): report activation independently of task outcomes (#438)
63f064f Narendran Raghavan committed 11 days ago
fix(reporting): align routing summaries and unavailable-score conclusions (#433)
6a5e7c7 Narendran Raghavan committed 11 days ago
fix(cli): preflight before paid autopilot dataset generation (#434)
d616c5e Narendran Raghavan committed 11 days ago
docs(grading): define evidence-grounded qualitative review contract (#432)
f74c52f Narendran Raghavan committed 11 days ago