Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Commits on main
fix(tier3): stop reading capitalized HTTP prose as an HTTPie upload (#339)
209a72e Narendran Raghavan committed 15 days ago
feat(tier3): add opt-in judge fallback and strict retry policies (#343)
8a851c6 Narendran Raghavan committed 15 days ago
fix(tier2): share a repeated text only when that shortens the review prompt (#346)
c55ffb8 Narendran Raghavan committed 15 days ago
docs(tier3): say that a refused attempt fails the arm's attempt coverage (#347)
2248e69 Narendran Raghavan committed 15 days ago
feat(ci): check benchmark cards against the expected source identity (#340)
52c78cb Narendran Raghavan committed 15 days ago
feat(pii): add evidence-bound PII waivers and skip Proton profile paths (#307)
27645cd Narendran Raghavan committed 15 days ago
fix(security): explain SkillSpector LLM provider outages as retriable incomplete scans (#342)
9bda66d Narendran Raghavan committed 15 days ago
feat(tier3): add `tier3 feedback` chat over a completed run (#341)
ea147c1 Narendran Raghavan committed 15 days ago
fix(tier3): explain failing scores and show baseline trial rows (#305)
e8f2a1d Narendran Raghavan committed 15 days ago
fix(tier3): turn off Claude Code's refusal model fallback in eval trials (#337)
48870f1 Narendran Raghavan committed 15 days ago
fix(ci): use a neutral default validation policy (#334)
03a7c22 Christopher Kevin committed 15 days ago
fix(inference): recover one JSON object wrapped in LLM prose (#297)
61ca935 Narendran Raghavan committed 15 days ago
feat(llm): record the model that wrote rubric scores and Tier 3 insights (#308)
c928d82 Narendran Raghavan committed 15 days ago
feat(reporting): reconcile stable finding IDs across report formats (#310)
47620b6 Narendran Raghavan committed 15 days ago
fix(tier1): scope SkillSpector agent-threat findings to agent control paths (#306)
f7912c3 Narendran Raghavan committed 15 days ago
fix(tier3): score goal_accuracy with RAGAS for eligible OpenAI judges (#286)
db5cc9c Narendran Raghavan committed 15 days ago
fix(tier2): review large repeated-content clusters within the prompt budget (#338)
ed3aad5 Narendran Raghavan committed 15 days ago
fix(tier3): honor --n-concurrent with --stop-on-pass (#275)
3ad2838 Narendran Raghavan committed 15 days ago
fix(pii): stop reporting timestamps, matrices, and vectors as PII (#304)
4c3fc87 Narendran Raghavan committed 16 days ago
fix(tier2): allow contained directory aliases (#282)
fc900e0 Narendran Raghavan committed 16 days ago
fix(tier3): stop apply_patch workspace writes scoring as credential access (#313)
700c0bb Narendran Raghavan committed 16 days ago
fix(tier3): compute judge scores from verdicts and save judge evidence (#303)
80dff4c Narendran Raghavan committed 16 days ago
fix(tier3): flag behavior evidence shortened by the final-response section limit (#259)
ea3f0f2 xh123637 committed 16 days ago
fix(security): explain incomplete SkillSpector scans that hit a runtime limit (#294)
0ae3fdb Narendran Raghavan committed 16 days ago
fix(tier3): retry transient Claude Code and Codex install failures (#277)
a52c82b Narendran Raghavan committed 16 days ago
feat(tier1): apply audited SkillSpector baselines (#300)
39d8901 Narendran Raghavan committed 16 days ago
fix(tier1): avoid false positives for argument hints and GUIDs (#273)
9aa72f9 Narendran Raghavan committed 16 days ago
fix(license): parse SPDX license expressions (#295)
c61fa97 Narendran Raghavan committed 16 days ago
fix(tier3): stop running and queued Harbor jobs on Ctrl+C (#291)
53329d6 Narendran Raghavan committed 16 days ago
fix(tier3): stage over existing trees on older glibc and explain bwrap failures (#267)
8d8dc25 Narendran Raghavan committed 16 days ago