Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

docs(ci): say ci demo only works after the setup merges (#477)

Christopher Kevin8a7b8bd421da
diff --git a/ci/templates/all-tiers/README.md b/ci/templates/all-tiers/README.md
index bacd5c79..ea837b90 100644
--- a/ci/templates/all-tiers/README.md
+++ b/ci/templates/all-tiers/README.md
@@ -302,7 +302,10 @@ a skill. To exercise a caller change with providers before merging, set
`scan-mode: all` in the caller on that PR and read the results in the run's
artifact; that PR gets no automatic comment, because the listener approves only
the default branch's caller. Before merging, restore `auto` and then refresh
-`trusted-caller-blobs` for the final caller.
+`trusted-caller-blobs` for the final caller. `ci demo` cannot exercise an unmerged
+setup: it requires the all-tier scan caller on the default branch and rejects a
+setup that has not merged, and its skill-change PR branches from the default
+branch. Use `ci demo` after the setup merges.
If a provider returns HTTP 404, verify that the selected model accepts chat
requests with your account. Presence in the public model catalog alone does not
diff --git a/docs/advanced-ci.mdx b/docs/advanced-ci.mdx
index 49a3d588..51a34757 100644
--- a/docs/advanced-ci.mdx
+++ b/docs/advanced-ci.mdx
@@ -43,7 +43,8 @@ pin, model or secret change, skips the provider-backed jobs. Their prerequisites
are first checked by the push to the default branch after it merges, a manual
run there, or the next skill change. To check them before merging, set
`scan-mode: all` (GitLab: `SKILL_EVAL_SCAN_MODE: all`) on that pull request and
-restore `auto` before merging.
+restore `auto` before merging. `skillevaluator ci demo` works only after the setup
+has merged.
Reports retain native evidence and an aggregate artifact for 14 days. Feedback
links to the authenticated artifact for that run; download and extract it,
diff --git a/tests/ci/test_all_tier_workflows.py b/tests/ci/test_all_tier_workflows.py
index 2b373a67..3a51cc86 100644
--- a/tests/ci/test_all_tier_workflows.py
+++ b/tests/ci/test_all_tier_workflows.py
@@ -528,6 +528,18 @@ def test_gitlab_feedback_pipeline_guard_is_offered_and_documented():
assert "keep `test` in any `stages` list" in onboarding
+def test_pre_merge_paid_tier_advice_does_not_route_through_ci_demo():
+ # `ci demo` needs the scan caller on the default branch (the onboarding tests
+ # cover the rejection), so the docs must not offer it for an unmerged setup.
+ docs = " ".join((ROOT / "ci/templates/all-tiers/README.md").read_text().split())
+ assert "(`ci demo` opens one)" not in docs
+ assert "`ci demo` cannot exercise an unmerged setup" in docs
+ assert "rejects a setup that has not merged" in docs
+ assert "Use `ci demo` after the setup merges." in docs
+ advanced = " ".join((ROOT / "docs/advanced-ci.mdx").read_text().split())
+ assert "`skillevaluator ci demo` works only after the setup has merged." in advanced
+
+
def _heredoc(script, marker):
return script.split(f"<<'{marker}'\n", 1)[1].split(f"\n{marker}\n", 1)[0] + "\n"