Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Commits on main
fix(tier1): report dependency advisories as structured findings (#421)
38274cc Narendran Raghavan committed 8 days ago
fix(tier3): stop per-trial apt installs in Docker agent setup (#413)
882ec05 Narendran Raghavan committed 8 days ago
fix(tier3): match home-anchored sensitive paths in the security check (#444)
f4d3f02 Narendran Raghavan committed 8 days ago
feat(tier3): add opt-in tolerance for baseline-only agent timeouts (#430)
0f4395d Narendran Raghavan committed 8 days ago
fix(tier1): report manifest checks as N/A when validate finds no supported artifact (#410)
0445c4d Narendran Raghavan committed 8 days ago
fix(tier2): reject out-of-range or non-string dedup LLM verdict fields (#412)
3cf34ec Narendran Raghavan committed 8 days ago
fix(tier3): apply case-specific skill metric applicability across reports (#425)
8850662 Narendran Raghavan committed 8 days ago
fix(tier3): reuse a run's saved feedback when view regenerates its report (#419)
4892db0 Narendran Raghavan committed 8 days ago
build(tier3): upgrade Harbor to 0.25.0 and preserve runtime contracts (#464)
a72a611 Narendran Raghavan committed 8 days ago
fix(tier3): keep negative controls out of missed-activation advice (#466)
8ba7f78 Narendran Raghavan committed 8 days ago
docs(reports): explain completed-agent results in incomplete multi-agent Tier 3 runs (#417)
b38b71b Narendran Raghavan committed 8 days ago
fix(reporting): show recovered timeout attempts accurately in Tier 3 attempt details (#416)
4a34bd3 Narendran Raghavan committed 8 days ago
feat(tier3): save RAGAS goal-accuracy requests and replies in judge_attempts.json (#429)
335628d Narendran Raghavan committed 8 days ago
fix(tier3): count edits, searches and MCP calls as productive and leave planning calls out of tool efficiency (#428)
44ee844 Narendran Raghavan committed 8 days ago
fix(tier3): keep the Rich progress frame at a fixed height (#420)
3ef5fad Narendran Raghavan committed 8 days ago
fix(grader): canonicalize verifier paths before provenance checks (#467)
03c27bf Narendran Raghavan committed 8 days ago
fix(tier2): parse skill frontmatter with bounded YAML (#422)
610d8de Narendran Raghavan committed 8 days ago
fix(tier2): chunk shell scripts in one pass instead of once per function header (#423)
cfeb583 Narendran Raghavan committed 8 days ago
feat(ci): add LLM security, gateway agents and benchmark PRs (#439)
487910e Narendran Raghavan committed 8 days ago
Support modern MCP SDK transports in protected URL probes (#465)
a0b65a9 Narendran Raghavan committed 8 days ago
fix(reporting): show judge-mixed lift exclusions on Tier 3 evaluator cards (#418)
d55b240 Narendran Raghavan committed 8 days ago
fix(tier3): show N/A for N/A metrics in the terminal Skill Lift and the HTML trial drill-down (#411)
5b2cd25 Narendran Raghavan committed 8 days ago
fix(tier3): report FAIL when a scored dimension is below 0.40 even if another is N/A (#414)
bb6faad Narendran Raghavan committed 8 days ago
ci: connect SkillEvaluator to NVSkills catalog validation (#440)
ce52a09 Christopher Kevin committed 9 days ago
Show LLM verification reasoning alongside findings (#454)
43a6611 Narendran Raghavan committed 9 days ago
docs: point the plugin inventory reference at the module that exists (#448)
7ae25d2 Hyacinth-of-Security committed 10 days ago
fix(reporting): let default Tier 3 runs with a baseline pass publication (#332)
194f7f9 Saibernard committed 10 days ago
fix(ci): apply reviewed baselines and preserve scan completeness evidence (#453)
b0461ba Christopher Kevin committed 10 days ago
fix(tier3): rank baseline-only agents by "baseline" for root dataset sufficiency (#450)
d717f22 Narendran Raghavan committed 10 days ago
fix(tier2): expose collection coverage and transactional rebuilds (#436)
2bd67ca Narendran Raghavan committed 10 days ago