Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
{
"skill_name": "text-analyzer",
"evals": [
{
"id": "text-analyzer-starter-001",
"prompt": "Analyze /workspace/input/input.txt with text-analyzer. Return the word count, line count, character count, sentence count and most frequent word.",
"expected_output": "4 words, 2 lines, 27 characters, 2 sentences; hello is the most frequent word with 2 occurrences.",
"expected_skill": "text-analyzer",
"expected_script": "analyze_text.py",
"assertions": [
"The agent reports the word, line, character and sentence counts and the most frequent word for input.txt.",
"The returned statistics match the input fixture, including its final newline."
]
}
]
}