Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
docs: complete benchmark publication guidance
Narendran Raghavanbad7e02ef7cd
diff --git a/docs/benchmark-rollout.mdx b/docs/benchmark-rollout.mdxindex ffe19381..ab7d7143 100644--- a/docs/benchmark-rollout.mdx+++ b/docs/benchmark-rollout.mdx@@ -8,6 +8,16 @@ redesign, and publication gate is available. Existing `BENCHMARK.md` files aregenerated artifacts; merging the renderer does not rewrite cards alreadypublished in skill repositories.+## Before and after++The legacy card led with dense tier output and mixed verdict evidence into the+details. The replacement puts the publication decision first, labels uplift+unambiguously, and moves methodology behind the evidence reviewers need most.++| Before: legacy card | After: decision-first publication card |+| --- | --- |+|  |  |+## 1. Pin the evaluatorInstall the merged SkillEvaluator release or immutable commit and record it fordiff --git a/docs/reports.mdx b/docs/reports.mdxindex 0b79dc92..2ee7017a 100644--- a/docs/reports.mdx+++ b/docs/reports.mdx@@ -69,19 +69,27 @@ and never clobbers the previous run.When the target is a skill, `validate` always writes `BENCHMARK.md` to theoutput directory — regardless of which `-r` formats you picked, and even when-validation fails. It is a publication card: evaluation date, the active-validation profile, per-tier summaries, and (when Tier 3 ran) agents, dataset-composition, and the dimension-by-dimension results table.+validation fails. It is a decision-first publication card: the overall verdict+and recommendation come first, followed by per-tier evidence, evaluator and+dataset provenance, agent/model details, and a dimension-by-dimension+`Baseline → Skill Uplift` table when Tier 3 ran. Public cards omit validation+profile names and absolute host paths.The card carries one overall verdict — `PASS`, `FAIL`, or `INCOMPLETE`. Itfails closed: if a required scanner produced no trustworthy evidence, theverdict is `INCOMPLETE` and the card explicitly says not to use it to recommendpublication. A Publication Recommendation section appears only on a clean-`PASS`. An advisory Tier 3 run that was skipped doesn't fail the combined-verdict: the card adds an explicit `Tier 3 live evaluation: SKIPPED — <reason>`-line and adjusts the publication recommendation to say the live evaluation-didn't run. Keep the file with the skill and refresh it whenever the skill or-its eval dataset materially changes.+`PASS`. Tier 3 evidence is publication-required by default, so a missing,+skipped, or incomplete live evaluation produces `INCOMPLETE` rather than a+publication recommendation. A catalog may explicitly persist+`benchmark_policy.tier3_required = false`; the card discloses that exception+instead of implying that Tier 3 ran.++Keep the file with the skill and refresh it whenever the skill, eval dataset,+agent/model, evaluator version, scoring policy, attempt policy, or execution+environment materially changes. Follow the+[rollout and backfill runbook](benchmark-rollout.mdx) when regenerating existing+published cards.## Safety guaranteesdiff --git a/tests/test_benchmark_rollout_docs.py b/tests/test_benchmark_rollout_docs.pyindex b74886be..b203d339 100644--- a/tests/test_benchmark_rollout_docs.py+++ b/tests/test_benchmark_rollout_docs.py@@ -15,6 +15,8 @@ def test_rollout_runbook_covers_safe_backfill_and_freshness() -> None:assert "--output-dir ./benchmark-backfill/example-skill" in runbookassert "check_public_benchmarks.py" in runbook+ assert "assets/benchmark-card-before.png" in runbook+ assert "assets/benchmark-card-after.png" in runbookassert "--require-files" in runbookassert "Do not reuse a prior live score" in runbookassert "The generated evaluation date must come from the live run artifact" in runbook