Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Commits on main
ci: run the all-tier skill scan only when skills or its callers change (#441)
fff5332 Christopher Kevin committed 11 days ago
fix(tier3): treat placeholder expected_script and behavior values as no expectation (#402)
6e11779 Christopher Kevin committed 11 days ago
docs(tier3): document per-arm dimension applicability for the no-skill baseline (#398)
be71254 Christopher Kevin committed 11 days ago
feat(reporting): add per-case expectation diagnostics (#382)
d7f8db1 Christopher Kevin committed 11 days ago
fix(quality): derive instruction tokens from parsed frontmatter (#356)
1af2693 YMuskrat committed 11 days ago
fix(tier2): retain trailing shell comments in deduplication (#357)
e76fcc9 YMuskrat committed 11 days ago
feat(ci): evaluate SkillEvaluator skills and publish PR summaries (#409)
62191aa Narendran Raghavan committed 11 days ago
perf(ci): omit Docker-in-Docker from GitLab Tier 2 (#408)
c3b298c Narendran Raghavan committed 11 days ago
feat(ci): add three-tier onboarding and private report feedback (#349)
86f1a1f Narendran Raghavan committed 11 days ago
fix(tier3): end feedback chat URL spans where a path after them can start (#389)
ecd09e5 Narendran Raghavan committed 12 days ago
feat(plugin): recognize native OpenClaw manifests in the private plugin helpers (#391)
c101790 Narendran Raghavan committed 12 days ago
test(tier3): run the Bedrock judge-deadline retry test on a manual clock (#393)
35ef01e Narendran Raghavan committed 12 days ago
Keep report navigation persistent and evidence accessible (#352)
d136a59 Narendran Raghavan committed 14 days ago
docs(changelog): keep 0.8.0 as released and list later changes under Unreleased (#371)
e5c100a Narendran Raghavan committed 14 days ago
fix(reporting): make HTML charts self-contained and accessible (#350)
40d28e4 Narendran Raghavan committed 14 days ago
Polish terminal reports and keep CI progress durable (#353)
f440bbd Narendran Raghavan committed 14 days ago
fix(reporting): align scoped verdicts and trial accounting (#351)
dd27412 Narendran Raghavan committed 14 days ago
fix(tier3): keep the HTTPie method rule where a capitalized HTTP word may still run (#363)
0205909 Narendran Raghavan committed 15 days ago
fix(tier3): mask skill and home paths and keep more text in feedback chat evidence (#364)
4fd19a8 Narendran Raghavan committed 15 days ago
fix(tier3): keep the Bedrock label on strict judge errors that move on (#359)
af50680 Narendran Raghavan committed 15 days ago
fix(tier3): leave inferred timing N/A when a tool step is not followed by an agent step (#365)
db62830 Narendran Raghavan committed 15 days ago
fix(tier2): number shared review blocks from the most repeated text (#361)
164e5fc Narendran Raghavan committed 15 days ago
docs(tier3): say where a refused claude-code attempt's coverage error appears (#360)
7776eb6 Narendran Raghavan committed 15 days ago
fix(reporting): drop PII evidence digests that redaction made stale (#348)
03e8c8b Narendran Raghavan committed 15 days ago
feat(tier3): split measured agent time into LLM and tool seconds (#344)
55d2d04 Narendran Raghavan committed 15 days ago
test(tier3): start bridge worker-count tests from an idle server (#358)
57f899a Narendran Raghavan committed 15 days ago
feat(reporting): complete private runtime reports and saved-run acceptance (#345)
55d8b8d Narendran Raghavan committed 15 days ago