Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
docs(cli): compare direct tier workflows with validate --tiers (#237)
Christopher Kevinda02abd31610
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 98a11ca1..59621a7c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -24,6 +24,10 @@ All notable changes to SkillEvaluator are documented in this file.### Changed+- The CLI reference now compares the direct `tier1`, `tier2`, and `tier3`+ workflows with `validate --tiers`: which tiers run, Tier 1 dependency and LLM+ stages, report files, and exit codes. It also states that `--tiers` must+ include Tier 1, so Tier 2 or Tier 3 alone runs through its direct workflow.- README and onboarding docs now offer two first-run paths: a lightweight,keyless quality check on a bundled skill, or an NVIDIA Build rubric evaluationwith eligible free prototyping access. Installation adds extras as needed,diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdxindex 39a4f9a6..b90ce2ff 100644--- a/docs/cli-reference.mdx+++ b/docs/cli-reference.mdx@@ -95,6 +95,33 @@ a dataset to repair it.`validate PATH` runs all three tiers for skills by default, including Tier 3autopilot. It is separate from these tier-only workflows.+### Direct tiers compared with `validate --tiers`++`validate --tiers` narrows the combined pipeline; it does not run a tier on+its own. Tier 1 always runs and gates the exit code, so `--tiers 2`,+`--tiers 3`, and `--tiers 2,3` are rejected. Use `tier2 PATH` or+`tier3 PATH` to run Tier 2 or Tier 3 without Tier 1.++| | Direct workflows | `validate PATH --tiers ...` |+| --- | --- | --- |+| Tiers run | Only the named tier | Tier 1 plus any selected tiers |+| Tier 1 dependency audit | Included in `tier1 PATH` | Opt-in with `--checks`, which replaces the default list, so name every check you want plus `dependency` ([selection rules](tier1-validation.mdx#narrow-to-specific-checks)) |+| Tier 1 LLM stages | Rubric, LLM security, and verification run when a provider is configured; `--no-llm` skips them | LLM security analysis runs only with `--llm`, and finding verification only with `--llm-verify`; no rubric stage |+| Reports | `tier1` and `tier2` default to `-r cli,json,html` and write `skillevaluator-tier1.*` or `skillevaluator-tier2.*` under `-o`. `tier3` writes its run under `--results-dir` and does not accept `-r` or `-o` | The compact view writes HTML and JSON as `skillevaluator-output-<timestamp>.*`; skill targets also get `BENCHMARK.md` |+| Exit code `1` | `tier1`: a check failed or required evidence is incomplete. `tier2`: HIGH or CRITICAL findings or a blocking analysis error. `tier3`: the run did not complete; a `FAIL` verdict alone exits `0` | Tier 1 failed; Tier 2 findings unless `--no-block-on-dedup`; Tier 3 only with `--block-on-agent-eval` |++For `tier2 PATH`, `tier3 PATH`, and `tier1 PATH --llm`, a missing provider is+a configuration error that exits `1` before any report is written. `tier1 PATH`+without `--llm` runs keyless. `validate` records Tier 2 and Tier 3 as skipped+and still writes its reports.++Direct Tier 3 checks provider, option, and runtime configuration before it+generates a starter case, so a configuration error leaves `evals/` unchanged and+no paid generation or agent run starts. The exception is `--env-mode` `ack`,+`cwsandbox`, `langsmith`, `openshift`, and `wandb`: their backend checks contact+a remote control plane, so they run after the starter case is generated. Usage+errors, such as an option the workflow does not accept, exit `2`.+### Direct Tier 1 options| Flag | Default | Effect |diff --git a/tests/cli/test_validate_defaults.py b/tests/cli/test_validate_defaults.pyindex 5ff8bf53..d953691f 100644--- a/tests/cli/test_validate_defaults.py+++ b/tests/cli/test_validate_defaults.py@@ -56,6 +56,34 @@ def test_skill_validate_selects_tiers_and_prepares_dataset(monkeypatch, tmp_pathassert calls == expected[email protected]("tiers", ["2", "3", "2,3"])+def test_validate_tiers_without_tier1_is_rejected_before_any_tier_runs(monkeypatch, tmp_path, tiers):+ skill = tmp_path / "sample"+ skill.mkdir()+ (skill / "SKILL.md").write_text("---\nname: sample\ndescription: Test skill\n---\n")+ calls = []++ def record(name):+ def run(*_args, **_kwargs):+ calls.append(name)+ return []++ return run++ monkeypatch.setattr(cli_module, "run_validation", record("tier1"))+ monkeypatch.setattr(cli_module, "_run_dedup_or_skip", record("tier2"))+ monkeypatch.setattr(cli_module, "_ensure_autopilot_dataset", record("dataset"))+ monkeypatch.setattr(cli_module, "_run_agent_eval_or_skip", record("tier3"))+ result = CliRunner().invoke(+ cli_module.cli, ["validate", str(skill), "--tiers", tiers, "-o", str(tmp_path / "reports")]+ )+ assert result.exit_code == 1+ assert "Tier 1 always runs" in result.output+ assert calls == []+ assert not (tmp_path / "reports").exists()+ assert not (skill / "evals").exists()++def test_default_autopilot_receives_skill_directory_for_file_input(monkeypatch, tmp_path):from skillevaluator.tier3 import evals_spec