Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

docs: complete benchmark publication guidance

Narendran Raghavanbad7e02ef7cd
diff --git a/docs/benchmark-rollout.mdx b/docs/benchmark-rollout.mdx
index ffe19381..ab7d7143 100644
--- a/docs/benchmark-rollout.mdx
+++ b/docs/benchmark-rollout.mdx
@@ -8,6 +8,16 @@ redesign, and publication gate is available. Existing `BENCHMARK.md` files are
generated artifacts; merging the renderer does not rewrite cards already
published in skill repositories.
+## Before and after
+
+The legacy card led with dense tier output and mixed verdict evidence into the
+details. The replacement puts the publication decision first, labels uplift
+unambiguously, and moves methodology behind the evidence reviewers need most.
+
+| Before: legacy card | After: decision-first publication card |
+| --- | --- |
+| ![Legacy BENCHMARK.md card before the rollout](assets/benchmark-card-before.png) | ![Decision-first BENCHMARK.md card after the rollout](assets/benchmark-card-after.png) |
+
## 1. Pin the evaluator
Install the merged SkillEvaluator release or immutable commit and record it for
diff --git a/docs/reports.mdx b/docs/reports.mdx
index 0b79dc92..2ee7017a 100644
--- a/docs/reports.mdx
+++ b/docs/reports.mdx
@@ -69,19 +69,27 @@ and never clobbers the previous run.
When the target is a skill, `validate` always writes `BENCHMARK.md` to the
output directory — regardless of which `-r` formats you picked, and even when
-validation fails. It is a publication card: evaluation date, the active
-validation profile, per-tier summaries, and (when Tier 3 ran) agents, dataset
-composition, and the dimension-by-dimension results table.
+validation fails. It is a decision-first publication card: the overall verdict
+and recommendation come first, followed by per-tier evidence, evaluator and
+dataset provenance, agent/model details, and a dimension-by-dimension
+`Baseline → Skill Uplift` table when Tier 3 ran. Public cards omit validation
+profile names and absolute host paths.
The card carries one overall verdict — `PASS`, `FAIL`, or `INCOMPLETE`. It
fails closed: if a required scanner produced no trustworthy evidence, the
verdict is `INCOMPLETE` and the card explicitly says not to use it to recommend
publication. A Publication Recommendation section appears only on a clean
-`PASS`. An advisory Tier 3 run that was skipped doesn't fail the combined
-verdict: the card adds an explicit `Tier 3 live evaluation: SKIPPED — <reason>`
-line and adjusts the publication recommendation to say the live evaluation
-didn't run. Keep the file with the skill and refresh it whenever the skill or
-its eval dataset materially changes.
+`PASS`. Tier 3 evidence is publication-required by default, so a missing,
+skipped, or incomplete live evaluation produces `INCOMPLETE` rather than a
+publication recommendation. A catalog may explicitly persist
+`benchmark_policy.tier3_required = false`; the card discloses that exception
+instead of implying that Tier 3 ran.
+
+Keep the file with the skill and refresh it whenever the skill, eval dataset,
+agent/model, evaluator version, scoring policy, attempt policy, or execution
+environment materially changes. Follow the
+[rollout and backfill runbook](benchmark-rollout.mdx) when regenerating existing
+published cards.
## Safety guarantees
diff --git a/tests/test_benchmark_rollout_docs.py b/tests/test_benchmark_rollout_docs.py
index b74886be..b203d339 100644
--- a/tests/test_benchmark_rollout_docs.py
+++ b/tests/test_benchmark_rollout_docs.py
@@ -15,6 +15,8 @@ def test_rollout_runbook_covers_safe_backfill_and_freshness() -> None:
assert "--output-dir ./benchmark-backfill/example-skill" in runbook
assert "check_public_benchmarks.py" in runbook
+ assert "assets/benchmark-card-before.png" in runbook
+ assert "assets/benchmark-card-after.png" in runbook
assert "--require-files" in runbook
assert "Do not reuse a prior live score" in runbook
assert "The generated evaluation date must come from the live run artifact" in runbook