Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(skills): add skillevaluator agent skill (#366)
Christopher Kevin958d7e880456
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2c01638e..69fb45a6 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -13,6 +13,13 @@ All notable changes to SkillEvaluator are documented in this file.### Added+- `skills/skillevaluator/` is an agent skill for operating SkillEvaluator: it+ routes requests to the right tier, readiness check, dataset, report, or CI+ command, and ships Tier 3 evaluation cases for itself. A test keeps every+ command and option it documents in sync with the CLI. The new "Evaluate from+ Your Agent" docs page shows how to install it in Codex, Claude Code, OpenCode,+ OpenClaw, and other skill-aware agents.+- Private runtime runs retain sanitized loading, coverage, MCP, behavior andcomparison views before cleanup. The saved snapshot supports offline reportsin all six existing formats, with explicit missing evidence and omissions.diff --git a/README.md b/README.mdindex 72694b10..cc41cbd0 100644--- a/README.md+++ b/README.md@@ -173,7 +173,8 @@ Cite the paper and record the SkillEvaluator release or commit; see [CITATION.cfSee [the documentation](https://docs.nvidia.com/skills/skillevaluator/) forsetup, tier guides, reports, CI integration, CLI reference, and-[performance dashboard](docs/reports.mdx#performance-dashboard).+[performance dashboard](docs/reports.mdx#performance-dashboard). To let an agent run+SkillEvaluator, install [`skills/skillevaluator/`](skills/skillevaluator/).## Installation and third-party softwarediff --git a/docs/AGENTS.md b/docs/AGENTS.mdindex e8191e8a..2f94f2fa 100644--- a/docs/AGENTS.md+++ b/docs/AGENTS.md@@ -14,6 +14,7 @@ folder holds MDX pages published to <https://docs.nvidia.com/skills/skillevaluat- `quickstart.mdx` — install, one provider and key, and first runs- `installation.mdx` — extras, uv, pip, Docker, requirements- `configuration.mdx` — providers, embeddings, and Tier 3 credential roles+- `agent-skill.mdx` — installing and using the `skills/skillevaluator` agent skill- `tier1-validation.mdx` — Tier 1 checks, profiles, scanner setup, exit codes- `tier2-deduplication.mdx` — Tier 2 dedup commands, catalogs, thresholds, limits- `tier3-live-evaluation.mdx` — Tier 3 live agent evaluation workflowdiff --git a/docs/agent-skill.mdx b/docs/agent-skill.mdxnew file mode 100644index 00000000..dd13078f--- /dev/null+++ b/docs/agent-skill.mdx@@ -0,0 +1,185 @@+---+title: "Evaluate Skills from Your Agent"+description: "Install the SkillEvaluator agent skill in Codex, Claude Code, OpenCode, OpenClaw, or any skill-aware agent, then evaluate skills by asking in plain language."+---++Give your coding agent SkillEvaluator expertise. Install one skill folder, then+ask: *"Is my skill any good?"* The agent picks the right tier, runs the CLI,+reads the report, and tells you what to fix.++No commands to memorize. No flags to look up. The skill ships in the public+repository and stays in sync with the CLI through a test on every change.++## How it works++```mermaid+flowchart LR+ ask["You ask your agent<br/>to evaluate a skill"] --> skill["SkillEvaluator skill<br/>picks tier and commands"]+ skill --> cli["skillevaluator CLI<br/>runs Tier 1, 2 or 3"]+ cli --> answer["Verdict, scores<br/>and next steps"]+ classDef step fill:#f3f9e8,stroke:#76b900,color:#1f2b16,stroke-width:2px+ class ask,skill,cli,answer step+```++The skill teaches the agent how to operate SkillEvaluator. The scores still come+from the CLI, its isolated sandboxes, and its independent judge, so results match+what you get running SkillEvaluator by hand or in CI.++<CardGroup cols={3}>+ <Card title="Right check, first time" icon="route">+ Routes "is it safe", "will it trigger", and "does it help" to the matching tier and command.+ </Card>+ <Card title="Honest results" icon="scale-balanced">+ Reads verdicts, exit codes, and INCOMPLETE runs correctly. No scores invented from partial runs.+ </Card>+ <Card title="Fixes, not just scores" icon="wrench">+ Turns findings and `what_to_fix.json` into concrete edits to your `SKILL.md` and eval cases.+ </Card>+</CardGroup>++## Install++<Steps>++### Install the CLI++```bash+pip install --upgrade skillevaluator # Tier 1 and 2+pip install --upgrade "skillevaluator[all]" # adds Tier 3 live evaluation+```++See [Installation](installation.mdx) for extras and Python versions.++### Add the skill to your agent++```bash+git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git+```++Copy or symlink `SkillEvaluator/skills/skillevaluator` into your agent's skills directory:++<Tabs>+ <Tab title="Codex">+ ```bash+ cp -r SkillEvaluator/skills/skillevaluator .agents/skills/ # this repository+ cp -r SkillEvaluator/skills/skillevaluator ~/.codex/skills/ # every repository+ ```+ </Tab>+ <Tab title="Claude Code">+ ```bash+ cp -r SkillEvaluator/skills/skillevaluator .claude/skills/ # this repository+ cp -r SkillEvaluator/skills/skillevaluator ~/.claude/skills/ # every repository+ ```+ </Tab>+ <Tab title="OpenCode">+ ```bash+ cp -r SkillEvaluator/skills/skillevaluator .opencode/skills/ # this repository+ cp -r SkillEvaluator/skills/skillevaluator ~/.config/opencode/skills/ # every repository+ ```+ </Tab>+ <Tab title="OpenClaw">+ ```bash+ cp -r SkillEvaluator/skills/skillevaluator ~/.openclaw/skills/ # every workspace+ ```+ </Tab>+ <Tab title="Other agents">+ Any agent that reads [Agent Skills](https://agentskills.io/) folders works. Copy the folder+ into its skills directory, for example `.cursor/skills`, `.gemini/skills`, or+ `.config/goose/skills`.+ </Tab>+</Tabs>++### Set a provider key for LLM checks++Static Tier 1 checks need no key. The rubric, Tier 2, and Tier 3 need one provider,+set in the shell where the agent runs:++```bash+export SKILL_EVAL_LLM_PROVIDER=nv_build+export NVIDIA_API_KEY=...+```++OpenAI, Anthropic, and any OpenAI-compatible gateway also work; see+[Providers & Credentials](configuration.mdx). Keys stay in your environment; the+skill tells the agent never to print or store them.++</Steps>++<Tip>+ Start a new agent session after installing so the agent discovers the skill.+</Tip>++## Ask in plain language++| You ask | The agent runs | You get |+| --- | --- | --- |+| "Check my skill before I publish it, no API keys" | `skillevaluator tier1 <skill-dir> --no-llm` | Pass or fail with security, PII, schema, and quality findings |+| "Score the quality of this SKILL.md" | `skillevaluator quality-check` and `rubric-eval` | Quality score and grade, rubric feedback |+| "Does it overlap with my other skills?" | `skillevaluator tier2` | Duplicate and similarity findings |+| "Write eval cases for my skill" | `skillevaluator create-eval-dataset --full`, then `tier3 validate --strict` | A reviewed `evals/evals.json` with positive and negative cases |+| "Does my skill actually make the agent better?" | `skillevaluator tier3 evaluate` with and without the skill | Verdict, five dimension scores, and Skill Lift |+| "Explain this benchmark report" | Reads `result.json`, `tier3-benchmark.md`, `what_to_fix.json` | Plain-language verdict and prioritized fixes |+| "Gate skills in our CI" | `skillevaluator ci init` | A workflow with the right exit-code gates |++## What a live evaluation looks like++For "does my skill help", the agent runs the same tasks twice in isolated+sandboxes, once without your skill and once with it, and an independent judge+scores both.++```text+Tier 3 Evaluation: PASS+Agent: codex Attempts scored: 32/32++ Baseline With skill Lift+Overall 82.5% 94.4% +11.9+Security 87.5% 100.0% +12.5+Correctness 91.3% 98.8% +7.5+Effectiveness 68.6% 95.3% +26.7+```++This is the SkillEvaluator skill evaluated by SkillEvaluator. Its own eval cases+ship in [`skills/skillevaluator/evals/`](https://github.com/NVIDIA/SkillEvaluator/tree/main/skills/skillevaluator/evals).++<Warning>+ Asking one agent to answer with and without a skill and grade itself is not an+ evaluation. Once it has read the skill, the baseline is contaminated, and the+ same model is judging its own work. The skill always routes "does it help" to+ real Tier 3 runs.+</Warning>++## What runs where++<AccordionGroup>+ <Accordion title="Tier 1 and Tier 2: on your machine">+ Static checks, quality, security, PII, and deduplication run locally through the CLI.+ No Docker and no Harbor. LLM-assisted checks need a provider key.+ </Accordion>+ <Accordion title="Tier 3: in a sandbox">+ Docker is the default backend; SkillEvaluator drives Harbor for you. For trusted skills,+ `--env-mode local` runs the agent on the host under Bubblewrap (Linux) or Seatbelt (macOS).+ See [Agents & Sandboxes](agents-and-sandboxes.mdx).+ </Accordion>+ <Accordion title="Incomplete runs">+ If an attempt errors or times out, the run is INCOMPLETE and no final score or lift is+ reported. The skill tells the agent to say so, find the cause, and rerun, not to average+ the partial results.+ </Accordion>+</AccordionGroup>++## Next steps++<CardGroup cols={2}>+ <Card title="Tier 3: Live Evaluation" icon="flask" href="tier3-live-evaluation.mdx">+ How baseline and with-skill runs are scored, and what Skill Lift means.+ </Card>+ <Card title="Eval Datasets" icon="list-check" href="eval-datasets.mdx">+ The `evals.json` contract the agent writes and validates.+ </Card>+ <Card title="Reports & Results" icon="chart-column" href="reports.mdx">+ Every report file the agent reads, and what each field means.+ </Card>+ <Card title="Bring It to Your Repository" icon="code-branch" href="repository-integration.mdx">+ Check every skill change in GitHub or GitLab CI.+ </Card>+</CardGroup>diff --git a/docs/index.mdx b/docs/index.mdxindex 58ee0369..4f0198bd 100644--- a/docs/index.mdx+++ b/docs/index.mdx@@ -105,6 +105,9 @@ How to read all three, and the on-disk results contract, live in [Reports & Resu<Card title="Quickstart" icon="rocket" href="quickstart.mdx">Run a bundled quality check without an LLM, or use NVIDIA Build for a rubric review.</Card>+ <Card title="Evaluate from Your Agent" icon="robot" href="agent-skill.mdx">+ Install the SkillEvaluator skill in Codex, Claude Code, OpenCode, or OpenClaw and evaluate skills by asking.+ </Card><Card title="Gate Your CI" icon="shield" href="ci-integration.mdx">Make SkillEvaluator a merge gate — exit codes, a GitHub Actions recipe, and progressive adoption.</Card>diff --git a/fern/docs.yml b/fern/docs.ymlindex 97125cd5..5903ada9 100644--- a/fern/docs.yml+++ b/fern/docs.yml@@ -38,6 +38,9 @@ navigation:- page: Providers & Credentialspath: ../docs/configuration.mdxslug: configuration+ - page: Evaluate from Your Agent+ path: ../docs/agent-skill.mdx+ slug: agent-skill- section: Evaluate Your Skillskip-slug: truecontents:diff --git a/skills/skillevaluator/SKILL.md b/skills/skillevaluator/SKILL.mdnew file mode 100644index 00000000..db7926c2--- /dev/null+++ b/skills/skillevaluator/SKILL.md@@ -0,0 +1,196 @@+---+name: skillevaluator+description: Run and interpret SkillEvaluator for agent skills. Use for any skill benchmark or eval - validating or scoring a SKILL.md, writing evals.json test cases or evals config, explaining benchmark reports, verdicts and exit codes, or gating skills in CI.+license: Apache-2.0+compatibility: Requires Python 3.12-3.14 and the skillevaluator CLI; live evaluation also needs Docker (or another backend) and an LLM provider key.+metadata:+ author: Christopher Kevin <[email protected]>+ version: "1.0.0"+ domain: [evaluation, ai-ml]+ tags: [skills, evaluation, benchmarking, ci, security]+---++# SkillEvaluator++Run the SkillEvaluator CLI on agent skills and report what the evidence shows:+Tier 1 validation (static, security, quality), Tier 2 deduplication, and Tier 3+live agent evaluation that compares runs with and without the skill.++## Purpose++Give an agent one reliable procedure for evaluating skills with+SkillEvaluator: pick the right tier, check readiness, run it, and report+scores, findings and exit codes truthfully.++## When To Use++Use this skill when the user wants to:++- check whether a skill is well-formed, safe, and good enough to publish+- get a quality score, rubric score, security or PII scan for a skill+- find repeated guidance inside a skill or overlap across a skill collection+- measure whether a skill actually helps an agent (Skill Lift, pass@k)+- write or fix `<skill-dir>/evals/evals.json`, `<skill-dir>/evals/config.yml`, custom graders, or Harbor tasks+- read a SkillEvaluator report, verdict, or exit code+- add SkillEvaluator checks to GitHub Actions or GitLab CI++Do not use it to write the target skill's domain content, to review ordinary+application code, or to evaluate an LLM without a skill. For converting an+existing benchmark into a custom grader, read+[references/custom-graders.md](references/custom-graders.md).++## Prerequisites++- A target directory containing `SKILL.md` (or a folder of skills, `.mdc`+ rules, or a plugin manifest).+- The CLI: `pip install --upgrade "skillevaluator[all]"`, or from a checkout+ `uv run --python 3.13 --extra all skillevaluator ...`. Check with+ `skillevaluator --version`. Extras: `llm` (rubric), `security` (scanners),+ `tier2`, `tier3`, `all`, `dashboard`.+- No key for keyless commands (`quality-check`, `security-scan`, `pii-scan`,+ `lint-scripts`, `tier1 PATH --no-llm`, `validate PATH --tiers 1`).+- A provider for LLM stages, Tier 2 and Tier 3. Simplest:+ `export SKILL_EVAL_LLM_PROVIDER=nv_build` and `export NVIDIA_API_KEY=...`.+ Other providers: [references/providers.md](references/providers.md).+- For Tier 3: an agent (`opencode`, `codex`, or `claude-code`) and a sandbox.+ Harbor is installed with the `tier3` extra; you do not run it yourself.+ The default `--env-mode docker` needs a Docker daemon. Without Docker, use+ `--env-mode local` (experimental): it runs the agent CLI already installed+ on the host under Bubblewrap (Linux) or Seatbelt (macOS); trusted skills only.++Never print, echo, or commit API keys; keep them in the host environment.++## Choose The Command++| User goal | Command | Needs |+| --- | --- | --- |+| Fast keyless quality score | `skillevaluator quality-check <skill-dir>` | nothing |+| Full static gate for one skill | `skillevaluator tier1 <skill-dir> --no-llm` | scanners (auto-installed) |+| Static gate plus LLM rubric and security review | `skillevaluator tier1 <skill-dir>` | provider |+| LLM rubric only | `skillevaluator rubric-eval <skill-dir>` | provider, `llm` extra |+| Security / PII scan only | `skillevaluator security-scan <skill-dir>`, `skillevaluator pii-scan <skill-dir>` | nothing |+| Repeated content inside one skill | `skillevaluator tier2 <skill-dir>` | provider with embeddings |+| Overlap across a collection | `skillevaluator similarity-check <skills-dir>` | embeddings |+| Does the skill help an agent? | `skillevaluator tier3 <skill-dir>` | provider, agent, Docker |+| Same, without Docker | `skillevaluator tier3 evaluate <skill-dir> -a opencode --env-mode local` | provider, agent CLI on host |+| Everything, with one exit code | `skillevaluator validate <skill-dir>` | all of the above |+| CI-safe keyless gate | `skillevaluator validate <skills-dir> --profile default --tiers 1 -r json,markdown,sarif -o reports` | scanners |+| Set up repo CI | `skillevaluator ci init` | GitHub checkout |++Full flag tables: [Tier 1](references/commands-tier1.md),+[validate](references/commands-validate.md),+[Tier 2, Tier 3 and setup](references/commands-tier2-tier3.md).++## Workflow++1. **Locate the target.** Confirm the path has `SKILL.md` and that its+ frontmatter `name` matches the directory name. For a folder of skills, the+ commands evaluate each skill inside it.+2. **Check readiness before paid runs.** Run `skillevaluator health-check --tier 1`+ for scanners, `skillevaluator doctor --tier 2` for providers, and+ `skillevaluator doctor --env-mode docker` (add `--verify-models`) for Tier 3.+ Fix every failing row first.+3. **Start keyless.** Run `skillevaluator tier1 <skill-dir> --no-llm`. Fix+ CRITICAL and HIGH findings before anything else; `lint` findings are advisory.+4. **Add LLM checks** when a provider is set: `skillevaluator tier1 <skill-dir>`+ (rubric, LLM security, verification) and `skillevaluator tier2 <skill-dir>`.+5. **Prepare Tier 3 cases.** If `<skill-dir>/evals/evals.json` is missing, generate one with+ `skillevaluator create-eval-dataset <skill-dir> --full`, then review it:+ positive cases (explicit, implicit, contextual) and at least one negative+ case with `"expected_skill": null`. Required shape: top-level `skill_name`+ and an `evals` array; each case has `id`, `prompt`, `expected_output`, and+ `expected_skill` (optional `assertions`, `files`). Validate with+ `skillevaluator tier3 validate <skill-dir> --strict`. Format:+ [references/tier3-datasets.md](references/tier3-datasets.md); run policy and+ scoring: [references/tier3-running.md](references/tier3-running.md).+6. **Run live evaluation.** `skillevaluator tier3 <skill-dir>` (or+ `skillevaluator tier3 evaluate <skill-dir> -a opencode --env-mode docker --n-attempts 3`).+ Use `--max-cases 1` for a cheap smoke run first; such runs are never+ publication evidence.+7. **Read the results** (see below), open `skillevaluator view <skill-dir>`, and+ for follow-up questions run `skillevaluator tier3 feedback <run-dir>`.+8. **Iterate.** Change the skill, rerun the same command with the same cases,+ agent and model, and compare. Treat a single run's lift as directional.++## Read The Results++Exit codes: `0` success, `1` a gate failed, evidence is `INCOMPLETE`, or the+run could not complete, `2` usage error (bad flag or missing path; fix the+command, not the skill).++- **Quality** (0-100): Correctness 35%, Discoverability 25%, Reliability 25%,+ Efficiency 15%. Grades A>=90, B>=80, C>=70. Passing needs `--min-score`+ (default 70) and no HIGH/CRITICAL quality finding.+- **Rubric** (0-100): nine criteria scored 0-10; passing needs the weighted+ score >= `--min-score` and every criterion >= 7.+- **Tier 2**: HIGH or CRITICAL duplicate findings exit `1` in `tier2 PATH`.+- **Tier 3**: five dimensions (Security, Correctness, Discoverability,+ Effectiveness, Efficiency), each PASS >= 0.50, NEUTRAL 0.40-0.50, FAIL < 0.40.+ Verdict PASS needs every dimension >= 0.50 for at least one agent. Skill+ Lift = with-skill overall - baseline overall; PASS >= +0.05. `INCONCLUSIVE:+ skill never activated` means no positive trial loaded the skill; fix the+ description or prompts, not the scores. `tier3 PATH` exits `0` on a `FAIL`+ verdict; only an incomplete run exits `1`.+- **Missing scores are `NO SCORE` or `None`, never 0.** Do not average them in.+- **Publication**: require `publication_status == "pass"` in the JSON report,+ not just `overall_passed`.++Report anatomy and JSON fields: [references/reports-and-ci.md](references/reports-and-ci.md).++## Examples++```bash+# Keyless first look+skillevaluator quality-check <skill-dir> -r cli -r json -o <report-dir>++# Static gate for a whole collection, complete report even on failures+skillevaluator validate <skills-dir> --profile default --tiers 1 -c -r json,markdown -o reports++# LLM-backed Tier 1 and Tier 2, dedup advisory+skillevaluator validate <skill-dir> --tiers 1,2 --llm --no-block-on-dedup++# Live A/B evaluation with two agents, retaining trajectories for debugging+skillevaluator tier3 evaluate <skill-dir> -a opencode,codex --n-attempts 3 --harbor-keep-jobs++# Baseline only, to calibrate cases before the skill exists+skillevaluator tier3 evaluate <skill-dir> --skip-with-skill+```++## Limitations++- Tier 1 and the quality score are static: they check structure and wording,+ not whether the skill works. Only Tier 3 observes agent behavior.+- LLM judges (rubric, Tier 3 correctness/effectiveness) vary between runs and+ models; compare runs only with the same model, cases, and attempts.+- Tier 3 costs model tokens and sandbox time per case x attempt x agent x 2+ arms; a generated starter case gives limited coverage.+- `validate --tiers` always includes Tier 1; `--tiers 2` or `--tiers 3` alone is+ rejected. Use `tier2 PATH` or `tier3 PATH` to run a tier on its own.+- Agent credentials come only from the host environment; `<skill-dir>/evals/config.yml`+ `runtime_env` rejects provider keys.++## Troubleshooting++| Symptom | Cause | Fix |+| --- | --- | --- |+| Exit `2` | Unknown flag, bad value, or missing path | Check `skillevaluator <command> --help`; prefix paths named like commands with `./` |+| Check shows `INCOMPLETE` | A scanner is missing or failed | Run `skillevaluator health-check --tier 1`; install the `security` extra or the missing scanner |+| "provider is required" / exit `1` before reports | No `SKILL_EVAL_LLM_PROVIDER` or key, or several keys exported | Set exactly one provider and its key; see providers reference |+| Quality below 70 | Missing `## Prerequisites`, `## Limitations`, `## Troubleshooting`, vague description, no examples | Read each finding's suggestion; do not stuff keywords |+| `INCONCLUSIVE: skill never activated` | Description does not match the prompts | Make the description name the tasks; add explicit-bucket cases |+| Docker trials fail to start | Daemon down or outdated engine | `docker info`; rerun `skillevaluator doctor --env-mode docker` |+| Agent model rejected | Default model not valid for that agent | `--agent-model AGENT=MODEL`, verify with `doctor --verify-models` |+| Run incomplete (`Scored attempt coverage is n/m`) | A trial errored or timed out | Rerun; raise `--timeout-multiplier`; inspect with `--harbor-keep-jobs` |++More cases: [references/troubleshooting.md](references/troubleshooting.md).++## Final Response++Report to the user:++- the exact commands run and their exit codes+- scores and verdicts per tier, quoting the report values (not estimates)+- the top findings to fix, each with its file and suggested change+- report paths (`<report-dir>/...`, `<skill-dir>/evals/results/<run-id>/report.html`)+- what was not run (skipped tiers, missing keys, `--max-cases` smoke runs) and+ therefore not provendiff --git a/skills/skillevaluator/evals/config.yml b/skills/skillevaluator/evals/config.ymlnew file mode 100644index 00000000..4e5c19a6--- /dev/null+++ b/skills/skillevaluator/evals/config.yml@@ -0,0 +1,6 @@+schema_version: 1+harbor:+ n_attempts: 2+ pass_threshold: 0.5+ n_concurrent: 4+ timeout_multiplier: 1.5diff --git a/skills/skillevaluator/evals/evals.json b/skills/skillevaluator/evals/evals.jsonnew file mode 100644index 00000000..3dd69e4c--- /dev/null+++ b/skills/skillevaluator/evals/evals.json@@ -0,0 +1,97 @@+{+ "skill_name": "skillevaluator",+ "evals": [+ {+ "id": "skillevaluator-001-explicit-keyless",+ "prompt": "Use the skillevaluator skill: I have a skill at ./skills/pdf-tools and no LLM API key. Give me the exact SkillEvaluator commands to check that it is well-formed, safe, and see its quality score, and tell me what exit code 1 versus 2 would mean.",+ "expected_output": "Recommends keyless commands such as `skillevaluator tier1 ./skills/pdf-tools --no-llm` and/or `skillevaluator quality-check ./skills/pdf-tools` (or `validate --tiers 1`), and explains exit 1 = a gate failed or evidence incomplete, exit 2 = usage error such as a bad flag or missing path.",+ "assertions": [+ "Read the skillevaluator SKILL.md",+ "Recommends a keyless Tier 1 command (tier1 --no-llm, quality-check, or validate --tiers 1) on ./skills/pdf-tools",+ "Explains exit code 1 as a failed check or incomplete evidence and exit code 2 as a usage error",+ "Does not ask for or invent an API key"+ ],+ "expected_skill": "skillevaluator"+ },+ {+ "id": "skillevaluator-002-explicit-tier3",+ "prompt": "With SkillEvaluator, how do I measure whether my skill at ./my-skill actually makes an agent better? Write the full command sequence from readiness check to report, including a cheap smoke run first, and say which number tells me the improvement.",+ "expected_output": "A sequence using doctor (with --env-mode docker), creating/validating evals (create-eval-dataset, tier3 validate), a smoke `tier3 evaluate ... --max-cases 1`, a full `tier3 evaluate` or `tier3 ./my-skill` run with and without the skill, and `view`; identifies Skill Lift (with-skill minus baseline) as the improvement measure.",+ "assertions": [+ "Includes a readiness check with skillevaluator doctor",+ "Includes dataset generation or validation (create-eval-dataset or tier3 validate)",+ "Includes a smoke run limited with --max-cases",+ "Runs tier3 live evaluation comparing with-skill and baseline",+ "Names Skill Lift as with-skill minus baseline"+ ],+ "expected_skill": "skillevaluator"+ },+ {+ "id": "skillevaluator-003-implicit-dataset",+ "prompt": "I need test cases so an agent benchmark can tell whether my 'csv-cleaner' agent skill gets used correctly. Write the evals/evals.json file for it with three cases: one that names the skill, one that only describes the task, and one prompt where the skill should NOT be used.",+ "expected_output": "A valid evals/evals.json with top-level skill_name 'csv-cleaner' and an evals array of three cases each having id, prompt, expected_output (optionally assertions), where the negative case sets expected_skill to null.",+ "assertions": [+ "Output has top-level skill_name and evals keys",+ "Every case has id, prompt, and expected_output",+ "Includes one explicit and one implicit positive case",+ "The negative case sets expected_skill to null"+ ],+ "expected_skill": "skillevaluator"+ },+ {+ "id": "skillevaluator-004-implicit-report",+ "prompt": "My skill benchmark report says 'INCONCLUSIVE: skill never activated' and the Tier 3 command exited 0. Another run says verdict FAIL but also exited 0. What do these mean and what should I change?",+ "expected_output": "Explains that no positive with-skill trial loaded the skill, so the description or prompts should be fixed rather than the scores; explains that tier3 exits 0 on a FAIL verdict and only an incomplete run exits 1, with --block-on-agent-eval in validate to gate on Tier 3.",+ "assertions": [+ "Explains never activated means the skill was not loaded in positive trials",+ "Recommends improving the skill description or eval prompts",+ "Explains that a FAIL verdict alone exits 0 and only an incomplete run exits 1",+ "Mentions --block-on-agent-eval or another way to make Tier 3 gate"+ ],+ "expected_skill": "skillevaluator"+ },+ {+ "id": "skillevaluator-005-contextual-ci",+ "prompt": "Our repo keeps agent skills under ./skills and we use GitHub Actions. Add a workflow step that fails the PR when any skill fails static validation, uploads SARIF, and does not need any API key. Show the command the step should run.",+ "expected_output": "A GitHub Actions step running `skillevaluator validate ./skills --tiers 1` with SARIF/JSON reports to an output dir (e.g. `-r json,markdown,sarif -o reports`), relying on the exit code to fail the job, with no secrets required; optionally mentions `skillevaluator ci init`.",+ "assertions": [+ "Uses skillevaluator validate on ./skills with --tiers 1",+ "Requests a sarif report and an output directory",+ "Relies on the command exit code to fail the job",+ "Does not require an API key for this gate"+ ],+ "expected_skill": "skillevaluator"+ },+ {+ "id": "skillevaluator-006-contextual-config",+ "prompt": "I'm setting up live agent evaluation for ./skills/sql-helper. I want 3 attempts per case, the opencode agent, and the agent needs my OPENAI_API_KEY. Write evals/config.yml and tell me where the API key should go.",+ "expected_output": "An evals/config.yml with schema_version: 1 and harbor.n_attempts: 3 (and opencode agent settings), explicitly NOT placing the API key in config.yml or harbor.runtime_env; the key is exported in the host environment where skillevaluator runs.",+ "assertions": [+ "config.yml sets schema_version 1 and harbor n_attempts 3",+ "Does not put OPENAI_API_KEY in config.yml or runtime_env",+ "Says to export the key in the host environment or CI secrets"+ ],+ "expected_skill": "skillevaluator"+ },+ {+ "id": "skillevaluator-007-negative-write-skill",+ "prompt": "Write a Python function that removes duplicate rows from a list of dictionaries, keeping the first occurrence.",+ "expected_output": "A direct Python implementation; no SkillEvaluator commands or skill evaluation workflow.",+ "assertions": [+ "Provides a working Python function",+ "Does not run or recommend SkillEvaluator"+ ],+ "expected_skill": null+ },+ {+ "id": "skillevaluator-008-negative-general",+ "prompt": "Explain the difference between precision and recall in classification models in two sentences.",+ "expected_output": "A short correct explanation of precision and recall without invoking skill evaluation.",+ "assertions": [+ "Correctly defines precision and recall",+ "Does not run or recommend SkillEvaluator"+ ],+ "expected_skill": null+ }+ ]+}diff --git a/skills/skillevaluator/references/commands-tier1.md b/skills/skillevaluator/references/commands-tier1.mdnew file mode 100644index 00000000..5ba21f55--- /dev/null+++ b/skills/skillevaluator/references/commands-tier1.md@@ -0,0 +1,55 @@+# Tier 1 command reference++Report options and Tier 1 commands. `skillevaluator <command> --help` is authoritative for the installed version.++Global options go before the command: `skillevaluator -v validate <skill-dir>`.++## Reports (Tier 1 and Tier 2 commands)++| Flag | Default | Effect |+| --- | --- | --- |+| `-r, --report` | `cli,json,html` for `tier1 PATH`, `tier2 PATH`, `validate`; `cli` for focused commands | `cli`, `json`, `html`, `markdown`, `sarif`; comma-separated or repeated |+| `-o, --output-dir` | `reports` | Report directory |++Passing `-r` is honored exactly: `-r cli` writes no files. Basenames:+`validate` writes `skillevaluator-output-<timestamp>.*`; `tier1`/`tier2` write+`skillevaluator-tier1.*`/`skillevaluator-tier2.*`; focused commands write+`skillevaluator-<kind>.*` (for example `skillevaluator-quality.json`) and+overwrite earlier runs.++## Direct tier workflows++A bare `tier1`, `tier2`, or `tier3` prints help. Prefix a path with `./` when+its name matches a subcommand (`tier1 <path>/validate`).++`skillevaluator tier1 PATH`: all static checks including `dependency`, plus+rubric, LLM security and finding verification when a provider is configured.++| Flag | Default | Effect |+| --- | --- | --- |+| `--checks TEXT` | all applicable | Subset: `schema`, `version`, `security`, `pii`, `license`, `code-integrity`, `unicode`, `quality`, `lint`, `dependency` |+| `--skip CHECKS` | none | Leave checks out (also `secrets` for Gitleaks); the run cannot certify Tier 1 |+| `--llm / --no-llm` | on when a provider is configured | Explicit `--llm` requires a valid provider |+| `--min-score INTEGER` | `70` | Quality and rubric minimum |+| `--quality-config PATH` | packaged | Replace expected-heading rules |+| `--pii-waivers PATH` | none | Reviewed, expiring PII waivers |+| `--profile TEXT` | `default` | Validation policy |++`skillevaluator tier2 PATH`: intra-skill duplicate analysis of one skill.+`--catalog FILE` adds inter-skill comparison (build it with+`skillevaluator similarity-check <skills-dir> --save-catalog <catalog.json>`);+`--threshold` (0.80), `--similarity-threshold` (0.75), `--full-body`.++`skillevaluator tier3 PATH`: generates one starter case when no source+exists, then runs the provider-native agent in Docker with and without the+skill. Takes every `tier3 evaluate` option. Does not accept `-r`/`-o`.++## Focused Tier 1 commands++| Command | Key flags |+| --- | --- |+| `quality-check PATH` | `--min-score` (70), `--quality-config` |+| `rubric-eval PATH` | `--min-score` (70); every criterion must reach 7/10 |+| `security-scan PATH` | `--llm`, `--llm-verify`, `--skillspector-baseline` |+| `pii-scan PATH` | `--llm-verify`, `--pii-waivers` |+| `lint-scripts PATH` | report options only; advisory |diff --git a/skills/skillevaluator/references/commands-tier2-tier3.md b/skills/skillevaluator/references/commands-tier2-tier3.mdnew file mode 100644index 00000000..4d3b456c--- /dev/null+++ b/skills/skillevaluator/references/commands-tier2-tier3.md@@ -0,0 +1,44 @@+# Tier 2, Tier 3 and setup command reference++`skillevaluator <command> --help` is authoritative.++## Tier 2 commands++| Command | Key flags |+| --- | --- |+| `context-optimization-check PATH` (alias `dedup-scan`) | `--threshold` (0.8), `--model`, `--llm-model` |+| `similarity-check PATH` | `--threshold` (0.75), `--full-body`, `--catalog`, `--save-catalog`, `--max-entries` |++## Tier 3 commands++| Command | Purpose |+| --- | --- |+| `tier3 evaluate PATH` | Live A/B run; add `--autopilot` to generate a missing case |+| `tier3 validate PATH` | Check `<skill-dir>/evals/` (`--strict`, `--json`, `--harbor-contract`) |+| `create-eval-dataset PATH` | Write `<skill-dir>/evals/evals.json` (`--full`, `--no-llm`, `--dry-run`, `--force`, `--prompt`, `--refine`) |+| `init-custom-grader PATH` | BYOG starter (`--mode`, `--language`) |+| `init-harbor-task PATH` | BYOT starter (`--case-id`, `--with-config`) |+| `tier3 select-decoys PATH` | Save neighbor skills for group workspaces |+| `view PATH`, `compare PATH` | Open latest report; compare agents |+| `tier3 feedback RUN_DIR` | Chat about one completed run (`--skill`) |+| `harbor-view JOBS_DIR` | Browse jobs kept with `--harbor-keep-jobs` |+| `dashboard [PATHS]` | Tables over saved reports (`dashboard` extra) |++Key `tier3 evaluate` flags: `-a, --agents` (`opencode`, `codex`,+`claude-code`), `--env-mode` (`docker` default, `local`, `kata`, `gke`,+`daytona`, `e2b`, `modal`, ...), `--n-attempts`, `--pass-threshold`,+`--stop-on-pass`, `--n-concurrent`, `--max-agents`, `--max-cases`, `--model`,+`--agent-model AGENT=MODEL`, `--skip-baseline`, `--skip-with-skill`,+`--grading-mode default|default_plus_custom|custom_only`, `--results-dir`,+`--include-skills`, `--skill-workspace-mode isolated|group`, `--copy-repo`,+`--timeout-multiplier`, `--harbor-keep-jobs`, `--agent-runtime-preflight`,+`--override-cpus`, `--override-memory-mb`, `--progress`, `--verbose`.++## Readiness and setup++| Command | Purpose |+| --- | --- |+| `health-check [--tier 1\|2\|3\|all]` | Offline readiness; `--checks` for Tier 1 scanners |+| `doctor [--tier ...] [--env-mode M] [-a AGENTS] [--verify-models]` | Deeper readiness; `--verify-models` makes live catalog probes |+| `models [--limit N] [--json]` | List provider catalog models |+| `ci init`, `ci status`, `ci demo` | Guided GitHub CI setup |diff --git a/skills/skillevaluator/references/commands-validate.md b/skills/skillevaluator/references/commands-validate.mdnew file mode 100644index 00000000..cc1dd576--- /dev/null+++ b/skills/skillevaluator/references/commands-validate.md@@ -0,0 +1,26 @@+# validate command reference++The combined pipeline. `skillevaluator validate --help` is authoritative.++## validate++`skillevaluator validate PATH` runs Tier 1, Tier 2 and Tier 3 (with+autopilot) for skills. Tier 1 always gates; Tier 2 gates unless+`--no-block-on-dedup`; Tier 3 gates only with `--block-on-agent-eval`. A+folder without a root `SKILL.md` validates each skill inside it.++| Flag | Effect |+| --- | --- |+| `--tiers 1`, `--tiers 1,2`, `--tiers 1,3` | Scope; Tier 1 always included |+| `--type skill\|rules\|workflows\|plugin\|auto` | Force the content type |+| `--checks`, `--skip` | Tier 1 selection (`dependency` is opt-in here; `--checks` replaces the list) |+| `--llm`, `--llm-verify` | LLM security analysis and false-positive suppression |+| `--min-score`, `--quality-config`, `--pii-waivers`, `--skillspector-baseline FILE` | Tier 1 tuning |+| `--previous-version VERSION` | Require a greater `metadata.version` (single skill only) |+| `--fail-fast`, `-c, --continue-on-failure` | Stop early, or record every issue (recommended in CI) |+| `--no-dedup`, `--no-block-on-dedup` | Skip Tier 2, or make it advisory |+| `--no-tier3`, `--no-autopilot`, `--block-on-agent-eval` | Tier 3 scope and gating |+| `-a`, `--env-mode`, `--n-attempts`, `--max-cases`, `--agent-model`, `--harbor-keep-jobs`, ... | Forwarded to Tier 3 |+| `--workers N` | Validate N skills of a folder in parallel |+| `--profile`, `--policy FILE` | Policy profile and overlay |+| `--evaluated-source-repository`, `--evaluated-source-revision` | Provenance for `BENCHMARK.md` |diff --git a/skills/skillevaluator/references/custom-graders.md b/skills/skillevaluator/references/custom-graders.mdnew file mode 100644index 00000000..cc4d8f1d--- /dev/null+++ b/skills/skillevaluator/references/custom-graders.md@@ -0,0 +1,16 @@+# Custom graders and Harbor tasks++Bring your own grader (BYOG) or Harbor task (BYOT).++- `default_plus_custom`: standard dimensions stay authoritative and your+ `custom_metrics` are added. `custom_only`: your grader's `overall` decides.+- Scaffold with `skillevaluator init-custom-grader <skill-dir> --language python`+ or `skillevaluator init-harbor-task <skill-dir> --case-id case-001 --with-config`,+ replace the placeholder logic, then run+ `skillevaluator tier3 validate <skill-dir> --harbor-contract`.+- A grader reads `/logs/agent/trajectory.json`, `/tests/entry.json`, and+ `/workspace/input/`, and writes `/logs/verifier/reward.json` plus+ `/logs/verifier/reward.txt` with values from 0.0 to 1.0.+- Do not name custom metrics `security`, `skill_execution`,+ `skill_efficiency`, `accuracy`, `goal_accuracy`, `behavior_check`,+ `overall`, `details`, `metrics`, `metric_set`, or `entry_id`.diff --git a/skills/skillevaluator/references/providers.md b/skills/skillevaluator/references/providers.mdnew file mode 100644index 00000000..cd071bdd--- /dev/null+++ b/skills/skillevaluator/references/providers.md@@ -0,0 +1,30 @@+# Providers and credentials++Set exactly one provider. If `SKILL_EVAL_LLM_PROVIDER` is unset and exactly+one of `NVIDIA_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` is exported,+it is auto-selected; with more than one, LLM commands fail until the+provider is set.++| Provider | Variables | Notes |+| --- | --- | --- |+| `nv_build` | `NVIDIA_API_KEY` | One key covers Tier 1 LLM, Tier 2 embeddings, and Tier 3 agents in `docker`, `kata`, `local`; default agent `opencode` |+| `openai` | `OPENAI_API_KEY` | Default agent `codex` |+| `anthropic` | `ANTHROPIC_API_KEY` | Default agent `claude-code`; Tier 2 needs a separate embedding provider |+| `bedrock` | AWS credential chain, `AWS_REGION` | Tier 2 needs a separate embedding provider |+| `openai-compatible` | `SKILL_EVAL_LLM_BASE_URL`, `SKILL_EVAL_LLM_API_KEY` | Any gateway; override models whose catalog IDs differ |++Overrides: `SKILL_EVAL_LLM_MODEL` (evaluator and judge),+`LLM_JUDGE_MODEL` (judge only), `SKILL_EVAL_EMBEDDING_MODEL`,+`--agent-model AGENT=MODEL` (Tier 3 agent). OpenCode on a gateway takes+`--agent-model opencode=openai/MODEL`. Codex needs a Responses-API model,+OpenCode Chat Completions, Claude Code Anthropic Messages, all with tool+calls.++Rules:++- Credentials come only from the host environment (or masked CI secrets).+ `harbor.runtime_env` in `<skill-dir>/evals/config.yml` rejects provider keys and base+ URLs with a hard error.+- `skillevaluator models` and `doctor --verify-models` check catalog+ reachability; a public catalog does not prove the inference key works.+- Judge failures make a result unscored (`NO SCORE`), not `0.0`.diff --git a/skills/skillevaluator/references/reports-and-ci.md b/skills/skillevaluator/references/reports-and-ci.mdnew file mode 100644index 00000000..d46dfe05--- /dev/null+++ b/skills/skillevaluator/references/reports-and-ci.md@@ -0,0 +1,47 @@+# Reports and CI++## JSON fields to read++From `skillevaluator-output-<timestamp>.json` (or `skillevaluator-tier1.json`):++| Field | Meaning |+| --- | --- |+| `overall_passed`, `overall_status` | Command gate: `passed`, `failed`, `incomplete` |+| `publication_status` | `pass`, `neutral`, `fail`, `incomplete`; use for publishing |+| `quality_summary[].overall_score`, `.grade` | Quality 0-100 and A-F |+| `severity_counts` | `critical`, `high`, `medium`, `low` totals |+| `finding_groups[]` | One entry per unique finding (`finding_id` `SEF-...`) |+| `tier3.run_id` | Run identity for the Tier 3 section |++`overall_passed` can be true while an advisory tier did not run; require+`publication_status == "pass"` before claiming a skill is publishable.++Example gate on a stricter score:++```bash+report=$(ls -t <report-dir>/skillevaluator-output-*.json | head -1)+jq -e '([.quality_summary[]?.overall_score] | min // 100) >= 80' "$report"+```++`validate` on a skill also writes `BENCHMARK.md`: verdict first, the three+tier states, and `Baseline -> Skill Uplift` columns.++## CI++Guided GitHub setup from a checkout: `skillevaluator ci init` (then+`ci status`). Manual: copy the GitHub or GitLab template from the+repository's `ci/bundles/<version>/` folder.++Recommended keyless gate:++```bash+skillevaluator validate <skills-dir> --profile default --tiers 1 -c \+ -r json,markdown,sarif -o reports --min-score 70+```++Adopt progressively: run with `|| true` (advisory), then gate on Tier 1,+raise `--min-score`, add `--llm --tiers 1,2` with a masked provider secret,+and finally an advisory Tier 3 job (`--block-on-agent-eval` when ready).+Keep keys in CI secrets exposed through the job environment, never in the+repository. Pin policy with `--policy <policy.yaml>` so severity changes are+reviewed like code.diff --git a/skills/skillevaluator/references/tier3-datasets.md b/skills/skillevaluator/references/tier3-datasets.mdnew file mode 100644index 00000000..65c3f77c--- /dev/null+++ b/skills/skillevaluator/references/tier3-datasets.md@@ -0,0 +1,52 @@+# Tier 3 evaluation datasets++Tier 3 runs a real agent on each case with the skill installed and without it (baseline). This file covers the `<skill-dir>/evals/` folder and dataset format.++## The <skill-dir>/evals/ directory++| Path | Purpose |+| --- | --- |+| `<skill-dir>/evals/evals.json` | Required dataset (`.jsonl`, `.yaml`, `.yml` also accepted) |+| `<skill-dir>/evals/config.yml` | Run policy |+| `<skill-dir>/evals/EVAL.md` | Guidance for `create-eval-dataset` (`## Questions`, `## Behaviors`, `## Negative Cases`, `## Notes`) |+| `<skill-dir>/evals/files/` | Fixtures staged into `/workspace/input/` |+| `<skill-dir>/evals/environment/` | `Dockerfile`, `docker-compose.yaml` sidecars, `mcp_servers.toml` |+| `<skill-dir>/evals/harbor/` | Native Harbor tasks (BYOT) |+| `<skill-dir>/evals/grader.py` or `grader.sh` | Custom grader (BYOG) |+| `<skill-dir>/evals/results/` | Output only; never edit or commit |++## Dataset format++```json+{+ "skill_name": "my-skill",+ "evals": [+ {+ "id": "my-skill-001",+ "prompt": "Convert report.docx in /workspace/input to PDF",+ "expected_output": "A PDF of the document saved next to the input",+ "assertions": ["Read my-skill/SKILL.md", "Ran scripts/convert.py"],+ "expected_skill": "my-skill",+ "files": ["files/case-001"]+ },+ {+ "id": "my-skill-002-negative",+ "prompt": "What is the capital of France?",+ "expected_output": "Answers directly without loading my-skill",+ "expected_skill": null+ }+ ]+}+```++Required: `skill_name`, `evals`, and per case `id` (letters, digits, `.`,+`_`, `-`; at most 128 chars, unique), `prompt`, `expected_output`. Optional:+`assertions`, `expected_skill` (`null` for negatives), `expected_script`,+`acceptable_skills`, `files` (paths relative to `<skill-dir>/evals/`; omit it and the+whole `<skill-dir>/evals/files/` is staged). The legacy flat array with `question`,+`ground_truth`, `expected_behavior` still runs with a deprecation warning.++Good datasets cover four buckets: explicit (names the skill), implicit+(describes the task), contextual (task inside a project), negative (should+not trigger). Write `expected_output` as an observable outcome and one+behavior per assertion.diff --git a/skills/skillevaluator/references/tier3-running.md b/skills/skillevaluator/references/tier3-running.mdnew file mode 100644index 00000000..933defd1--- /dev/null+++ b/skills/skillevaluator/references/tier3-running.md@@ -0,0 +1,62 @@+# Running and scoring Tier 3++Run policy, run sequence, scoring and cost control.++## config.yml++```yaml+schema_version: 1+harbor:+ n_attempts: 3+ pass_threshold: 0.5+ n_concurrent: 2+ timeout_multiplier: 1.5+ agents:+ opencode:+ model: your-model-id+skill_workspace:+ mode: isolated # or group, with include: [...]+grading:+ mode: default # default | default_plus_custom | custom_only+```++Unknown keys fail the load. CLI flags override config values.+`pre_agent_setup` runs only with `--skip-baseline`. Never put credentials in+`runtime_env`.++## Run sequence++```bash+skillevaluator doctor --env-mode docker --verify-models+skillevaluator create-eval-dataset <skill-dir> --full+skillevaluator tier3 validate <skill-dir> --strict+skillevaluator tier3 evaluate <skill-dir> -a opencode --max-cases 1 # smoke+skillevaluator tier3 evaluate <skill-dir> -a opencode --n-attempts 3+skillevaluator view <skill-dir>+```++Results land in `<skill-dir>/evals/results/<timestamp>_<pid>_<id>/` with `result.json`,+`run_config.json`, `report.html` and per-agent `lift.json`,+`with-skill/`, `without-skill/`. Use `--results-dir` to write elsewhere.++## Scoring++Standard grading scores five dimensions on 0.0-1.0 and averages them:++| Dimension | Source |+| --- | --- |+| Security | Deterministic trace checks (destructive commands, uploads) |+| Correctness | LLM judge of the answer (`accuracy`) |+| Discoverability | Whether the skill was loaded when expected (`skill_execution`) |+| Effectiveness | LLM judges `goal_accuracy` and `behavior_check` |+| Efficiency | Wasted tool or skill calls (`skill_efficiency`) |++Bands: PASS >= 0.50, NEUTRAL 0.40-0.50, FAIL < 0.40. Verdict PASS needs one+agent with every dimension passing. Lift band: PASS >= +0.05, FAIL <= -0.10.+pass@k counts a case passed when any attempt reaches `--pass-threshold`.++## Cost control++Trials = cases x attempts x agents x 2 arms. Use `--max-cases`,+`--skip-baseline` (no lift), `--stop-on-pass`, and `--n-concurrent` to+control spend. Runs that skip cases are never publication evidence.diff --git a/skills/skillevaluator/references/troubleshooting.md b/skills/skillevaluator/references/troubleshooting.mdnew file mode 100644index 00000000..5e6767d7--- /dev/null+++ b/skills/skillevaluator/references/troubleshooting.md@@ -0,0 +1,21 @@+# Troubleshooting++| Symptom | Cause | Fix |+| --- | --- | --- |+| `Error: No such command` or exit `2` | Wrong command or flag for this version | `skillevaluator --version`, then `skillevaluator <command> --help` |+| `tier1 validate` evaluated the wrong thing | Directory named like a subcommand | Use `<path>/validate` |+| `--tiers 2` rejected | `validate` always includes Tier 1 | Use `skillevaluator tier2 PATH` |+| Scanner `INCOMPLETE` | SkillSpector, Gitleaks or Bandit missing or failed | `skillevaluator health-check --tier 1`; install the `security` extra; an old scanner earlier on `PATH` wins over the pinned one |+| `--skip` used and publication fails | Skipped checks cannot certify Tier 1 | Rerun without `--skip` |+| Schema: directory does not match name | Folder name differs from frontmatter `name` | Rename one so both match (kebab-case) |+| Quality below minimum | Missing expected headings, vague description, no examples, `SKILL.md` over 500 lines or 5,000 tokens | Follow each finding's suggestion; move detail into `references/` |+| Rubric fails although the score is high | One criterion below 7/10 | Read that criterion's feedback in the JSON report |+| Tier 2 needs embeddings | Anthropic and Bedrock have no embeddings | Set `SKILL_EVAL_EMBEDDING_*` for another provider |+| `<skill-dir>/evals/config.yml` rejected | Unknown key or credential in `runtime_env` | Fix the key; move secrets to the host environment |+| `Missing required fields: question` | Legacy entry without `question`, or new entry without `prompt` | Use the agentskills.io shape |+| Skill never activated | Description and prompts do not match | Name the tasks in the description; add explicit cases |+| Agent auth fails | Agent credential not exported where `skillevaluator` runs | Export it on the host; `doctor -a AGENT` |+| `AgentTimeoutError` or rate-limit errors | Slow model or too much concurrency | `--timeout-multiplier 2`, lower `--n-concurrent` |+| Custom Dockerfile build fails | Base image lacks tools the agent or grader needs | Fix `<skill-dir>/evals/environment/Dockerfile`; rerun with `--harbor-keep-jobs` to read the build log |+| Results need debugging | Harbor jobs deleted by default | Rerun with `--harbor-keep-jobs`, then `skillevaluator harbor-view JOBS_DIR` |+| Lift differs between runs | LLM judges and agents are stochastic | Increase `--n-attempts`; compare only identical setups |diff --git a/tests/test_skillevaluator_skill.py b/tests/test_skillevaluator_skill.pynew file mode 100644index 00000000..b481b8fe--- /dev/null+++ b/tests/test_skillevaluator_skill.py@@ -0,0 +1,79 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Keep the repository's SkillEvaluator operator skill in sync with the CLI."""++from __future__ import annotations++import re+import shlex+from pathlib import Path++import click+import pytest++from skillevaluator.cli import cli++SKILL_DIR = Path(__file__).resolve().parents[1] / "skills" / "skillevaluator"+SKILL_FILES = sorted(SKILL_DIR.rglob("*.md"))+INLINE_COMMAND = re.compile(r"`(skillevaluator [^`]+)`")+FENCED_BLOCK = re.compile(r"```(?:bash|sh)\n(.*?)```", re.DOTALL)+++def _commands(text: str) -> list[str]:+ commands = INLINE_COMMAND.findall(text)+ for block in FENCED_BLOCK.findall(text):+ joined = block.replace("\\\n", " ")+ for line in joined.splitlines():+ line = line.split("#", 1)[0].strip()+ if "skillevaluator " in line:+ commands.append(line[line.index("skillevaluator ") :])+ return commands+++def _resolve(command: str) -> tuple[click.Command, list[str]]:+ tokens = shlex.split(command.split("|", 1)[0])[1:]+ node: click.Command = cli+ while tokens and isinstance(node, click.Group):+ token = tokens[0]+ if token.startswith("-"):+ tokens.pop(0)+ continue+ sub = node.get_command(click.Context(node), token)+ if sub is None:+ node = getattr(node, "workflow", None) or node+ break+ node = sub+ tokens.pop(0)+ return node, tokens+++def _all_commands() -> list[tuple[str, str]]:+ return [+ (path.relative_to(SKILL_DIR).as_posix(), cmd)+ for path in SKILL_FILES+ for cmd in _commands(path.read_text())+ if "<command>" not in cmd and cmd.split()[1:] != ["--version"]+ ]+++def test_skill_files_exist() -> None:+ assert (SKILL_DIR / "SKILL.md").is_file()+ assert _all_commands(), "expected documented skillevaluator commands"++[email protected](("source", "command"), _all_commands())+def test_documented_commands_and_options_exist(source: str, command: str) -> None:+ node, rest = _resolve(command)+ assert node is not cli, f"{source}: unknown command in {command!r}"+ known = {"--help", "-h"} | {opt for param in node.params for opt in [*param.opts, *param.secondary_opts]}+ for token in rest:+ if token.startswith("--") or re.fullmatch(r"-[a-zA-Z]", token):+ flag = token.split("=", 1)[0]+ assert flag in known, f"{source}: {flag} is not an option of {node.name!r} in {command!r}"+++def test_reference_links_resolve() -> None:+ for path in SKILL_FILES:+ for target in re.findall(r"\]\(([^)#]+\.md)(?:#[^)]*)?\)", path.read_text()):+ assert (path.parent / target).is_file(), f"{path.name}: broken link {target}"