Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

docs: document explicit tier gating

Narendran Raghavanadf3fa06ea29
diff --git a/CHANGELOG.md b/CHANGELOG.md
index aa891065..e4248bdc 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -41,6 +41,10 @@ All notable changes to SkillEvaluator are documented in this file.
### Changed
+- Added explicit `--block-on-dedup` / `--no-block-on-dedup` and
+ `--block-on-agent-eval` / `--no-block-on-agent-eval` controls with
+ backward-compatible defaults, Tier 3 source preflight, and consistent gating
+ metadata across CLI, JSON, Markdown, and HTML reports.
- Reduced pull-request runner use for changes confined to `docs/**` and
`fern/**`: DCO, Gitleaks, and pinned Fern validation still run, while mixed
and non-docs changes retain the complete Linux, macOS, Windows, packaging,
diff --git a/README.md b/README.md
index c6700b83..927de109 100644
--- a/README.md
+++ b/README.md
@@ -128,8 +128,10 @@ avoids managed sandbox charges, not hosted model charges. It is experimental
and only for trusted skills and workspaces; use Docker or cloud for untrusted
code. Start with one agent and a small dataset.
See the [Tier 3 guide](https://docs.nvidia.com/skills/skillevaluator/tier3-live-evaluation#plan-for-cost)
-before scaling a run. Tier 3 results are advisory within `validate`; Tier 1 and
-Tier 2 determine its exit status.
+before scaling a run. Tier 1 always gates `validate`. Tier 2 gates by default;
+`--no-block-on-dedup` keeps its scan and reports but makes its findings
+advisory. Tier 3 is advisory by default; `--block-on-agent-eval` promotes its
+findings, including invalid task-source evidence, into the exit gate.
## Documentation
diff --git a/docs/ci-integration.mdx b/docs/ci-integration.mdx
index 0145042b..229074ac 100644
--- a/docs/ci-integration.mdx
+++ b/docs/ci-integration.mdx
@@ -16,7 +16,7 @@ Turn SkillEvaluator into a merge gate for your skill repository. The recommended
| `2` | Configuration error — bad flags or an unreadable target | Fix the pipeline, not the skill |
| `3` | Runtime error — an unexpected internal failure | Re-run; the skill was never judged. File an issue if it persists |
-Tier 1 checks gate the exit code — and so do Tier 2 deduplication findings when dedup runs (the recommended gate below disables it with `--no-dedup`). Tier 3 live-evaluation results are always advisory inside `validate`: they appear in the reports but never flip a passing run to `1`. That means you can layer Tier 3 on later without destabilizing the gate.
+Tier 1 checks always gate the exit code. Tier 2 deduplication findings gate when dedup runs unless you pass `--no-block-on-dedup`; the recommended keyless gate below instead disables Tier 2 with `--no-dedup`. Tier 3 live-evaluation results are advisory inside `validate` unless you pass `--block-on-agent-eval`. Reports record the effective choice for each tier.
## The recommended gate
@@ -156,11 +156,11 @@ You do not have to turn on the strictest gate on day one. A path that works well
### Add LLM-backed checks
- Add a provider key as a masked CI secret, set `SKILL_EVAL_LLM_PROVIDER`, then add `--llm` (and `--llm-verify` to suppress false positives) and drop `--no-dedup` so Tier 2 overlap checks run too. Note that once dedup runs, its findings gate the exit code alongside Tier 1. See [Providers & Credentials](configuration.mdx) for provider setup.
+ Add a provider key as a masked CI secret, set `SKILL_EVAL_LLM_PROVIDER`, then add `--llm` (and `--llm-verify` to suppress false positives) and drop `--no-dedup` so Tier 2 overlap checks run too. Once dedup runs, its findings gate alongside Tier 1 by default; add `--no-block-on-dedup` if you want evidence without enforcing it yet. See [Providers & Credentials](configuration.mdx) for provider setup.
### Add an advisory Tier 3 job
- Run live agent evaluation as a separate, non-blocking job — either `skillevaluator tier3 evaluate` on its own or `validate --tier3` (`--agent-eval` is a supported compatibility alias and is not currently deprecated; `validate --full` runs Tier 1+2+3 in one shot). Tier 3 results never gate `validate`'s exit code; the standalone `tier3 evaluate` exits non-zero only when the run itself fails to complete. Expose agent credentials as CI secrets in the job environment — the `env:` block of the workflow step — because SkillEvaluator reads operator credentials from the host environment only. With the `nv_build` provider and `--env-mode docker`, a single `NVIDIA_API_KEY` secret covers the evaluator and all three agents. See [Tier 3: Live Evaluation](tier3-live-evaluation.mdx).
+ Run live agent evaluation as a separate job — either `skillevaluator tier3 evaluate` on its own or `validate --tier3` (`--agent-eval` is a supported compatibility alias and is not currently deprecated; `validate --full` runs Tier 1+2+3 in one shot). Attached Tier 3 is advisory by default; add `--block-on-agent-eval` when the team is ready for its findings or invalid source evidence to fail the merge gate. The standalone `tier3 evaluate` exits non-zero when the run itself fails to complete. Expose agent credentials as CI secrets in the job environment — the `env:` block of the workflow step — because SkillEvaluator reads operator credentials from the host environment only. With the `nv_build` provider and `--env-mode docker`, a single `NVIDIA_API_KEY` secret covers the evaluator and all three agents. See [Tier 3: Live Evaluation](tier3-live-evaluation.mdx).
</Steps>
<Note>
diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdx
index c638f6dd..511ed5a1 100644
--- a/docs/cli-reference.mdx
+++ b/docs/cli-reference.mdx
@@ -17,7 +17,7 @@ commands referenced below.
| Command | Purpose | Needs |
| --- | --- | --- |
-| [validate](#validate) | Run Tier 1 checks plus Tier 2 dedup by default (both gate the exit code), optional advisory Tier 3 | Nothing by default |
+| [validate](#validate) | Run Tier 1 checks plus Tier 2 dedup by default, with explicit Tier 2 and Tier 3 gating controls | Nothing by default |
| [quality-check](#quality-check) | Score skill quality across four weighted categories | Nothing |
| [rubric-eval](#rubric-eval) | LLM-as-judge rubric evaluation | LLM provider key |
| [security-scan](#security-scan) | Scan for security vulnerabilities | Nothing (key only with `--llm`/`--llm-verify`) |
@@ -88,15 +88,18 @@ skillevaluator tier3 validate ./my-skill --strict
## validate
-Validate a skill, rule, workflow, or plugin. Tier 1 static, security, and quality checks gate the exit code — and so do Tier 2 deduplication findings when that pass runs (on by default, degrading gracefully without embedding access); Tier 3 live evaluation attaches as an advisory pass when enabled with `--tier3`, `--autopilot`, `--full`, or the supported `--agent-eval` compatibility alias.
+Validate a skill, rule, workflow, or plugin. Tier 1 static, security, and quality checks always gate the exit code. Tier 2 deduplication runs and gates by default, but `--no-block-on-dedup` keeps the scan while making its findings advisory. Tier 3 live evaluation attaches as an advisory pass when enabled with `--tier3`, `--autopilot`, `--full`, or the supported `--agent-eval` compatibility alias; add `--block-on-agent-eval` to make its findings gate.
```bash title="Common invocations"
skillevaluator validate ./my-skill # Tier 1 + Tier 2
skillevaluator validate ./my-skill --llm # add LLM security analysis
skillevaluator validate ./my-skill -r cli,json,html # multiple report formats
skillevaluator validate ./my-skill --no-dedup # skip Tier 2 dedup
+skillevaluator validate ./my-skill --no-block-on-dedup # run Tier 2 as advisory
skillevaluator validate ./my-skill --external # strict publication profile
skillevaluator validate ./my-skill --tier3 -a codex # add advisory Tier 3 eval
+skillevaluator validate ./my-skill --tier3 -a codex \
+ --block-on-agent-eval # gate on Tier 3 too
skillevaluator validate ./my-skill --autopilot # Tier 3, generating evals if missing
skillevaluator validate ./my-skill --full -a codex # everything, one shot
skillevaluator validate ./my-skill --tiers 1,3 # explicit tier selection
@@ -147,6 +150,7 @@ Embedding + LLM dedup; on by default, skips gracefully without a provider key.
| Flag | Default | Effect |
| --- | --- | --- |
| `--dedup, --tier2 / --no-dedup, --no-tier2` | `dedup` | Run Tier 2 intra-skill semantic-overlap checks. On by default; skipped gracefully without public embedding access. Use `--no-tier2` (or `--no-dedup`) to disable. |
+| `--block-on-dedup / --no-block-on-dedup` | blocking | Choose whether Tier 2 findings participate in the `validate` exit gate. The scan and its report entries still run with `--no-block-on-dedup`. |
### Tier 3 · Live Agent Evaluation
@@ -156,7 +160,8 @@ The following flags are forwarded to the live-eval engine **only when Tier 3 is
| Flag | Default | Effect |
| --- | --- | --- |
-| `--tier3, --agent-eval` | off | Also run Tier 3 live agent evaluation (requires `evals/evals.json`). Advisory: it never changes the exit code. `--tier3` is the primary name; `--agent-eval` is a supported compatibility alias and is not currently deprecated. |
+| `--tier3, --agent-eval` | off | Also run Tier 3 live agent evaluation. Accepts a supported YAML/JSON dataset or native Harbor source and validates it before execution. Tier 3 is advisory by default. `--tier3` is the primary name; `--agent-eval` is a supported compatibility alias and is not currently deprecated. |
+| `--block-on-agent-eval / --no-block-on-agent-eval` | advisory | Choose whether Tier 3 findings, including missing or invalid selected task-source evidence, participate in the `validate` exit gate. This flag does not enable Tier 3 by itself. |
| `--autopilot` | off | Generate an evaluation source automatically when missing, then run Tier 3 (implies `--tier3`). |
| `-a, --agents TEXT` | `codex` | Comma-separated Harbor agents to evaluate. |
| `--env-mode` | `docker` | Harbor environment backend (full list under [tier3 evaluate](#tier3-evaluate)). |
diff --git a/docs/quickstart.mdx b/docs/quickstart.mdx
index 3c49be98..c5a5e65e 100644
--- a/docs/quickstart.mdx
+++ b/docs/quickstart.mdx
@@ -92,7 +92,7 @@ Get your first evaluation result in about two minutes, with no API key. This pag
| Gate skill quality and security | `validate` | [Tier 1: Validation](tier1-validation.mdx) | Exit code `0`, no CRITICAL findings |
| Find overlapping or duplicate skills | `similarity-check` | [Tier 2: Deduplication](tier2-deduplication.mdx) | No `EXACT_DUPLICATE` or `HIGH_SIMILARITY` findings |
| Prove the skill helps a real agent | `tier3 evaluate` | [Tier 3: Live Evaluation](tier3-live-evaluation.mdx) | Positive Skill Lift across the five dimensions |
-| Run all three tiers in one shot | `validate --full` | [Tier 3: Live Evaluation](tier3-live-evaluation.mdx) | Exit code `0` plus an auto-generated dataset and advisory Tier 3 results |
+| Run all three tiers in one shot | `validate --full` | [Tier 3: Live Evaluation](tier3-live-evaluation.mdx) | Exit code `0` plus an auto-generated dataset; Tier 3 is advisory unless you add `--block-on-agent-eval` |
| Block bad skills at merge time | `validate` in CI | [Gate Your CI](ci-integration.mdx) | A red check on failing PRs |
| Look up every flag and default | — | [CLI Reference](cli-reference.mdx) | — |
diff --git a/docs/reports.mdx b/docs/reports.mdx
index 0b79dc92..e408160a 100644
--- a/docs/reports.mdx
+++ b/docs/reports.mdx
@@ -280,7 +280,9 @@ For CI and tooling, two JSON entry points matter:
- **`validate --tier3` runs** — the combined
`skillevaluator-output-<timestamp>.json` report embeds the Tier 3 payload
alongside the Tier 1 and Tier 2 results, so one file covers all three
- tiers. Tier 3 findings are advisory there: they never change the exit code.
+ tiers. Each result carries finalized gating metadata. Tier 3 is advisory by
+ default and becomes blocking with `--block-on-agent-eval`; Tier 2 is blocking
+ by default and becomes advisory with `--no-block-on-dedup`.
Both paths speak the same dialect: standalone `tier3 evaluate` and
`validate --tier3` share one HTML renderer for `report.html`, and both embed
diff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdx
index c043a966..dad9d70a 100644
--- a/docs/tier1-validation.mdx
+++ b/docs/tier1-validation.mdx
@@ -19,7 +19,7 @@ Tier 1 is exposed through one umbrella command and five standalone commands. Eac
| `pii-scan` | Detect PII, credentials, secrets, and local identifiers | No (only with `--llm-verify`) |
| `lint-scripts` | Run advisory AST-based checks on Python scripts | No |
-`validate` is the umbrella command: Tier 1 checks gate the exit code (non-zero on failure or incomplete required scanner evidence), Tier 2 dedup runs by default and degrades gracefully without embedding access, and Tier 3 can be attached as an advisory pass with `--tier3`, `--autopilot`, `--full`, or an explicit tier selection such as `--tiers 1,3` (`--agent-eval` remains a supported compatibility alias and is not currently deprecated).
+`validate` is the umbrella command: Tier 1 checks always gate the exit code (non-zero on failure or incomplete required scanner evidence), Tier 2 dedup runs and gates by default while degrading gracefully without embedding access, and Tier 3 can be attached as an advisory pass with `--tier3`, `--autopilot`, `--full`, or an explicit tier selection such as `--tiers 1,3` (`--agent-eval` remains a supported compatibility alias and is not currently deprecated). Use `--no-block-on-dedup` to keep Tier 2 advisory or `--block-on-agent-eval` to promote Tier 3 into the gate.
## Run a validation
@@ -288,7 +288,7 @@ Bundled Python scanners resolve next to the SkillEvaluator interpreter before `P
| `2` | Configuration error — invalid flags, arguments, or target path |
| `3` | Runtime error — an unexpected internal failure |
-Tier 1 checks — and the Tier 2 dedup pass, when it runs — gate the exit code inside `validate`; Tier 3 findings are advisory and never change it. The `quality` check passes when the composite score reaches `--min-score` (default `70`).
+Tier 1 checks always gate the exit code inside `validate`. Tier 2 gates when it runs unless `--no-block-on-dedup` is set. Tier 3 is advisory unless `--block-on-agent-eval` is set. The `quality` check passes when the composite score reaches `--min-score` (default `70`). Reports record the effective gate membership for every tier so saved evidence and process status agree.
`--fail-fast` stops after the first failed check. `-c`/`--continue-on-failure` overrides it, records the full pipeline, and keeps scanning a collection past a CRITICAL finding. Wiring these exit codes into a merge gate — including a complete GitHub Actions recipe — is covered in [Gate Your CI](ci-integration.mdx).
diff --git a/docs/tier2-deduplication.mdx b/docs/tier2-deduplication.mdx
index 9ee70ac7..30f15ede 100644
--- a/docs/tier2-deduplication.mdx
+++ b/docs/tier2-deduplication.mdx
@@ -206,6 +206,11 @@ produce a non-zero exit code; MEDIUM and LOW findings are advisory. Exit
behavior is identical in every output mode, so these commands work as CI gates
— see [Gate Your CI](ci-integration.mdx).
+Inside the combined `validate` pipeline, Tier 2 also gates by default. Pass
+`--no-block-on-dedup` to keep the scan and its findings in every report without
+letting those findings change `validate`'s exit code. Use `--no-dedup` only when
+you intend to skip Tier 2 entirely.
+
## Defaults and safety limits
| Setting | Default | Purpose |
diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdx
index 99a36556..cb50f2f1 100644
--- a/docs/tier3-live-evaluation.mdx
+++ b/docs/tier3-live-evaluation.mdx
@@ -286,13 +286,20 @@ cloud backend for untrusted code.
| Use case | Command | When to use |
| --- | --- | --- |
-| Full validation | `skillevaluator validate ./my-skill --tier3` | Add advisory Tier 3 results after the standard validation stages |
+| Full validation | `skillevaluator validate ./my-skill --tier3` | Add Tier 3 after the standard stages; advisory by default, optionally blocking |
| Focused evaluation | `skillevaluator tier3 evaluate ./my-skill --agents codex` | Iterate on datasets, agents, environments, and grading settings |
-Attached to `validate`, Tier 3 is advisory: its results appear in the combined
-reports but never change the exit code. `--full` is the one-shot variant
-— Tier 1+2+3 with `--autopilot` dataset generation — and `--autopilot` on its
-own enables Tier 3 while generating an eval case when none exists.
+Attached to `validate`, Tier 3 is advisory by default: its results appear in the
+combined reports without changing the exit code. Add `--block-on-agent-eval`
+to make its findings gate, including evidence that the selected YAML/JSON
+dataset or native Harbor source is missing or invalid. `--full` is the one-shot
+variant — Tier 1+2+3 with `--autopilot` dataset generation — and `--autopilot`
+on its own enables Tier 3 while generating an eval case when none exists.
+
+```bash title="Make attached Tier 3 blocking"
+skillevaluator validate ./my-skill --tier3 --agents codex \
+ --block-on-agent-eval
+```
<Note>
`validate` forwards a dedicated Tier 3 flag group (`--agents`, `--env-mode`,