Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Commits on main
fix(reporting): preserve local and CI skill identity (#489)
5aacb5e Narendran Raghavan committed 8 days ago
fix(tier3): contain local Harbor directory transfers (#493)
69584dd Narendran Raghavan committed 8 days ago
fix(tier3): keep generated and validated eval datasets fair to the no-skill baseline (#456)
81b1e27 Narendran Raghavan committed 8 days ago
fix(tier3): reject judge results returned after the deadline (#486)
a0d8fd0 Narendran Raghavan committed 8 days ago
ci: run the Python 3.12 pull-request tests in parallel without coverage (#492)
3523764 Christopher Kevin committed 8 days ago
fix(tier1): require complete missing-reference evidence (#483)
7a37915 Narendran Raghavan committed 8 days ago
fix(tier3): check verifier imports with python3 -I at image build (#470)
bcac5e0 Narendran Raghavan committed 8 days ago
fix(quality): read QUALITY_DEFAULT_MIN_SCORE instead of six 70 literals (#447)
d40b0e2 Hyacinth-of-Security committed 8 days ago
fix(tier3): stop passing apt-get a blank package in generated Dockerfiles (#471)
38aac1f Narendran Raghavan committed 8 days ago
fix(tier3): keep credentials out of security finding evidence (#445)
e939a03 Narendran Raghavan committed 8 days ago
fix(tier3): scope native MCP references to verifier grants (#491)
801fc69 Narendran Raghavan committed 8 days ago
fix(ci): align setup selection and paid-run guidance (#481)
63eaccd Narendran Raghavan committed 8 days ago
fix(rubric): sanitize final rejection diagnostics (#484)
90dfd5a Narendran Raghavan committed 8 days ago
fix(tier1): scope ownership email policy and retain completeness diagnostics (#333)
0434e81 Christopher Kevin committed 8 days ago
fix(reporting): name the skill when the target is the current directory (#480)
eab725c Christopher Kevin committed 8 days ago
fix(llm): leave empty truncated verifier replies unattributed (#485)
b8bbd52 Narendran Raghavan committed 8 days ago
feat(tier3): add fixed-source skill interaction evaluation (#392)
6eb7563 YMuskrat committed 8 days ago
fix(tier3): decode inline and Promise.allSettled Codex wrappers (#235) (#325)
aa09db2 Amine Marzouki committed 8 days ago
Add advisory enforcement mode to Tier 1 reusable workflow (#431)
91ecf1a Prashant Bhavsar committed 8 days ago
feat(quality): configure Reliability error handling in the quality config (#336)
b64e351 Weston Platter committed 8 days ago
fix(tier3): preserve MCP validation and local artifact paths (#472)
cf325c8 Narendran Raghavan committed 8 days ago
fix(ci): skip paid LLM Tier 1, Tier 2 and Tier 3 jobs when no skill changed (#468)
8b76f2a Narendran Raghavan committed 8 days ago
fix(tier1): accept SkillSpector partial reports whose gaps are missing references (#457)
81b2f30 Narendran Raghavan committed 8 days ago
feat(llm): record the answering model on --llm-verify verdicts and Tier 2 duplicate findings (#455)
517b839 Narendran Raghavan committed 8 days ago
fix(reporting): name the skill in single-skill JSON reports and record the evaluator version (#424)
3d93f2e Narendran Raghavan committed 8 days ago
fix(tier1): retry an invalid rubric judge reply once and name the problem (#415)
4afd3d0 Narendran Raghavan committed 8 days ago
fix(reporting): describe a failed Tier 3 arm as incomplete, not as not run (#407)
12b387f Narendran Raghavan committed 8 days ago