Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(skill): correct consent, local-mode and eval-case issues from #570/#601 review (#666)

Narendran Raghavanc4dc6499fc50
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 44292f9c..9069ec45 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,8 +15,10 @@ All notable changes to SkillEvaluator are documented in this file.
skill's in the same folder (`*-LOOKALIKE`), with no embedding key. The model may route a
request to the wrong one of the two.
- The `skillevaluator` agent skill now tells the agent to check for the CLI
- and install it when missing (`uv tool install`, `pipx`, pip in a virtual
- environment, or `uvx`), and adds a CLI-missing Tier 3 case.
+ and, with the user's permission, install it when missing (`uv tool install`,
+ `pipx`, pip in a Python 3.12-3.14 virtual environment, or `uvx`), and adds a
+ CLI-missing Tier 3 case. The case's target skill is inlined in its prompt, so
+ the installed skill folder carries no second `SKILL.md`.
- Canonical `comment-skill-checks.yml` reusable workflow and source template for
the standard scored Tier 1 PR comment without hosted reports. It verifies the
existing signed receipts and native evidence, reuses the shared report and
@@ -160,12 +162,16 @@ All notable changes to SkillEvaluator are documented in this file.
### Changed
-- The `skillevaluator` agent skill now asks the user before installing an
- extra, using a credential, spending on a Tier 3 run or using `--env-mode
- local`. It also asks for real tasks and outcomes before writing eval cases,
- explains that Tier 3 needs its own provider key, and reports no verdict for a
- tier that could not run. Its own `evals/evals.json` gains eight cases
- covering these prompts, including consent before cold scanner provisioning.
+- The `skillevaluator` agent skill now asks the user before installing the
+ CLI, an extra or a missing scanner, using a provider key, spending on a Tier 3
+ run or using `--env-mode local` (trusted skills only; it does not reuse the
+ user's agent logins). It also asks for real tasks, outcomes and input files
+ before writing eval cases, explains that Tier 3 needs its own provider key,
+ and reports no verdict for a tier that could not run. Its own
+ `evals/evals.json` gains nine cases (010-018) covering these prompts,
+ including consent before cold scanner provisioning and writing a dataset from
+ the user's requests; the implicit dataset case now expects clarification and
+ accepts a labeled draft.
- The CI guide's recipe installs SkillSpector from the commit behind its release
tag and checks the Gitleaks archive against the published SHA-256 before
unpacking it, matching the repository's own workflows.
@@ -324,9 +330,6 @@ All notable changes to SkillEvaluator are documented in this file.
### Fixed
-- The bundled skill's implicit dataset case expects clarification when real
- requests and outcomes are missing, matching the skill's case-authoring rule.
-
- Budget-limited Tier 3 coverage checks retain admitted attempt ordinals, avoiding false missing/excess errors when a denied slot leaves a gap.
- HTML reports retain budget context for fully budget-denied runs without
restoring untrusted score evidence.
diff --git a/docs/agent-skill.mdx b/docs/agent-skill.mdx
index 49d16c26..70216c8e 100644
--- a/docs/agent-skill.mdx
+++ b/docs/agent-skill.mdx
@@ -162,7 +162,7 @@ you get running SkillEvaluator by hand or in CI.
| "Is my skill any good?" | `skillevaluator tier1 <skill-dir> --no-llm` | Pass or fail, with security, PII, schema, and quality findings |
| "Score the quality of this SKILL.md" | `skillevaluator quality-check` and `rubric-eval` | Quality score and grade, rubric feedback |
| "Does it overlap with my other skills?" | `skillevaluator tier2` | Duplicate and similarity findings |
-| "Write eval cases for my skill" | Ask for real tasks and outcomes, write `<skill-dir>/evals/EVAL.md`, run `skillevaluator create-eval-dataset <skill-dir> --full --prompt <skill-dir>/evals/EVAL.md`, then `tier3 validate <skill-dir> --strict` | A reviewed `evals/evals.json` with positive and negative cases |
+| "Write eval cases for my skill" | Asks for real tasks and outcomes, writes `<skill-dir>/evals/EVAL.md`, then runs `skillevaluator create-eval-dataset <skill-dir> --full --prompt <skill-dir>/evals/EVAL.md`, then `tier3 validate <skill-dir> --strict` | A reviewed `evals/evals.json` with positive and negative cases |
| "Does my skill actually make the agent better?" | `skillevaluator tier3 evaluate`, with and without the skill | Verdict, five dimension scores, and Skill Lift |
| "Gate skills in our CI" | `skillevaluator ci init` | A setup PR with the right checks |
diff --git a/skills/skillevaluator/SKILL.md b/skills/skillevaluator/SKILL.md
index 44ca92f4..6da7fd72 100644
--- a/skills/skillevaluator/SKILL.md
+++ b/skills/skillevaluator/SKILL.md
@@ -43,8 +43,8 @@ existing benchmark into a custom grader, read
- A target directory containing `SKILL.md` (or a folder of skills, `.mdc`
rules, or a plugin manifest).
-- The CLI on Python 3.12-3.14. If it is missing, install it yourself (Workflow
- step 1). Extras: `llm` (rubric), `security` (scanners), `tier2`, `tier3`,
+- The CLI on Python 3.12-3.14. If it is missing, ask before installing it
+ (Workflow step 1). Extras: `llm` (rubric), `security` (scanners), `tier2`, `tier3`,
`all`, `dashboard`. From a source checkout use
`uv run --python 3.13 --extra all skillevaluator ...`.
- No key for keyless commands (`quality-check`, `security-scan`, `pii-scan`,
@@ -58,8 +58,11 @@ existing benchmark into a custom grader, read
- Tier 3 does not use the model of the agent you are running in. It starts
separate agent processes in a sandbox, with and without the skill, and an LLM
judge scores them. It therefore needs a provider key exported on the host even
- when you launch it from inside Claude Code, Codex or OpenCode. One key
- (`NVIDIA_API_KEY` from https://build.nvidia.com) covers the agents and the judge.
+ when you launch it from inside Claude Code, Codex or OpenCode. With `nv_build`
+ in `--env-mode docker` or `local`, one key (`NVIDIA_API_KEY` from
+ https://build.nvidia.com) covers the agents and the judge; other backends and
+ native Claude or GPT agent models need that vendor's key
+ ([references/providers.md](references/providers.md)).
- For Tier 3: an agent (`opencode`, `codex`, or `claude-code`) and a sandbox.
Harbor is installed with the `tier3` extra; you do not run it yourself.
The default `--env-mode docker` needs a Docker daemon. Without Docker, use
@@ -73,10 +76,10 @@ Never print, echo, or commit API keys; keep them in the host environment.
| User goal | Command | Needs |
| --- | --- | --- |
| Fast keyless quality score | `skillevaluator quality-check <skill-dir>` | nothing |
-| Full static gate for one skill | `skillevaluator tier1 <skill-dir> --no-llm` | scanners (auto-installed) |
+| Full static gate for one skill | `skillevaluator tier1 <skill-dir> --no-llm` | scanners (downloaded if missing; ask first) |
| Static gate plus LLM rubric and security review | `skillevaluator tier1 <skill-dir>` | provider |
| LLM rubric only | `skillevaluator rubric-eval <skill-dir>` | provider, `llm` extra |
-| Security / PII scan only | `skillevaluator security-scan <skill-dir>`, `skillevaluator pii-scan <skill-dir>` | nothing |
+| Security / PII scan only | `skillevaluator security-scan <skill-dir>`, `skillevaluator pii-scan <skill-dir>` | SkillSpector for `security-scan` (downloaded if missing; ask first) |
| Repeated content inside one skill | `skillevaluator tier2 <skill-dir>` | provider with embeddings |
| Overlap across a collection | `skillevaluator similarity-check <skills-dir>` | embeddings |
| Does the skill help an agent? | `skillevaluator tier3 <skill-dir>` | provider, agent, Docker |
@@ -91,23 +94,31 @@ Full flag tables: [Tier 1](references/commands-tier1.md),
## Ask The User First
-Run keyless, read-only commands without asking only when their tools are already
-available. Keyless scans can auto-install missing scanners, so get permission
-before launching a command that would provision them. Reuse the user's explicit
+Keyless, read-only commands need no permission when their tools are already
+available. Keyless scans and `health-check --tier 1` download missing scanners,
+so get permission before launching a command that would provision them. Reuse the user's explicit
authorization for the same installation rather than asking again. Otherwise,
stop and ask one short question before any of these:
- **Installing.** A missing CLI, extra (`tier2`, `tier3`) or scanner changes the
user's environment. Name what is missing and the install command, then ask.
-- **Credentials.** Check by name whether a provider variable is exported. If
- none is, list the options (`NVIDIA_API_KEY` from https://build.nvidia.com,
- `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`) and ask which the user wants to set.
+- **Credentials and model spend.** Check by name whether a provider variable is
+ exported. If none is, list the options (`NVIDIA_API_KEY` from
+ https://build.nvidia.com, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`) and ask which
+ the user wants to set. If one is, name the provider and, unless the user asked
+ for that run, ask before the first command that calls it (`tier1` without
+ `--no-llm`, `rubric-eval`, `tier2`, `create-eval-dataset`, Tier 3). If several
+ keys are exported, ask which provider to use and set `SKILL_EVAL_LLM_PROVIDER`.
Never search files for keys and never print one.
- **Spending.** Before a Tier 3 run, state trials = cases x attempts x agents x 2
- and run `--max-cases 1` first.
-- **Running on the host.** `--env-mode local` runs the agent on the user's
- machine with their logins. Offer it only when the skill cannot change real
- systems; otherwise ask the user to start Docker.
+ (fewer with `--skip-baseline` or `--skip-with-skill`), each also scored by
+ judge calls. Propose a `--max-cases 1` smoke run first, ask again before the
+ full run, and pass `--max-attempts-total N` when the user sets a limit.
+- **Running on the host.** `--env-mode local` runs the agent CLI installed on the
+ host under Bubblewrap (Linux) or the weaker Seatbelt (macOS), with network
+ access on by default. It does not use the user's CLI logins: each trial gets a
+ private home directory and only the provider key exported on the host. Offer
+ it only for skills the user trusts; otherwise ask the user to start Docker.
When a tier cannot run, say which requirement is missing and what the user must
do. Report no score, lift or verdict for that tier.
@@ -119,16 +130,19 @@ do. Report no score, lift or verdict for that tier.
authorized, install it in the first way that works, then run the version
check again:
- `uv tool install --python 3.13 "skillevaluator[all]"` (isolated, preferred)
- - `pipx install "skillevaluator[all]"`
+ - `pipx install --python python3.13 "skillevaluator[all]"`
- `python3 -m pip install --upgrade "skillevaluator[all]"` inside an active
- virtual environment
+ virtual environment whose Python is 3.12-3.14 (on an older Python, pip
+ installs an empty `0.0.0a0` placeholder with no `skillevaluator` command)
- with no install, prefix each command with `uvx --python 3.13 --from "skillevaluator[all]"`
- The same permission rule covers isolated installs, `uvx` provisioning and
- upgrades as well as system or shared Python. Never use `sudo pip`. If the
- binary is installed but not on `PATH`, run
- `uv tool update-shell` or add `~/.local/bin` to `PATH`. If a documented
- command or flag is unknown, upgrade: `uv tool upgrade skillevaluator`. If
+ The same permission rule covers isolated installs, `uvx` provisioning,
+ upgrades and `uv tool update-shell` (it edits shell startup files), as well
+ as system or shared Python. Never use `sudo pip`. If the binary is installed
+ but not on `PATH`, add `~/.local/bin` to `PATH` for this session, or ask
+ before running `uv tool update-shell`. If a documented command or flag is
+ unknown, upgrade with the installer that was used (`uv tool upgrade
+ skillevaluator`, `pipx upgrade skillevaluator`, or pip `--upgrade`). If
installation fails (no network, no Python 3.12+), stop and report the
exact error and the install command instead of guessing results.
2. **Locate the target.** Confirm the path has `SKILL.md` and that its
@@ -137,22 +151,31 @@ do. Report no score, lift or verdict for that tier.
3. **Check readiness.** Run `skillevaluator health-check --tier 1`
for scanners, `skillevaluator doctor --tier 2` for providers, and
`skillevaluator doctor --env-mode docker` (add `--verify-models`) for Tier 3.
- Scanner readiness probes can provision a managed scanner runtime. Get
- installation permission before probing or scanning when its tools are missing.
+ `health-check --tier 1` and keyless scans download pinned SkillSpector and
+ Gitleaks when they are missing. Get installation permission before probing
+ or scanning when those tools are missing.
Fix every failing row first.
4. **Start keyless.** Run `skillevaluator tier1 <skill-dir> --no-llm`. Fix
CRITICAL and HIGH findings before anything else; `lint` findings are advisory.
-5. **Add LLM checks** when a provider is set: `skillevaluator tier1 <skill-dir>`
- (rubric, LLM security, verification) and `skillevaluator tier2 <skill-dir>`.
+5. **Add LLM checks** when a provider is set and the user agreed to model
+ spend: `skillevaluator tier1 <skill-dir>` (rubric, LLM security,
+ verification) and `skillevaluator tier2 <skill-dir>`.
6. **Prepare Tier 3 cases.** If `<skill-dir>/evals/evals.json` is missing, ask
the user for three to five real requests the skill should handle, what a
- good result looks like for each, and one request that should not trigger it.
- Write those into `<skill-dir>/evals/EVAL.md`; headings alone do not qualify.
- Do not invent the cases yourself; if the user asks for a draft, label it a
- draft and have them review it. If the skill directory is not yours to change
- (an installed skill, `~/.claude/skills`), copy it and evaluate the copy. For
- skills that act on real systems, phrase prompts as "give the commands, do not
- run them". Generate with
+ good result looks like for each, one request that should not trigger it, and
+ any input files those requests use. Record their answers in
+ `<skill-dir>/evals/EVAL.md`; headings alone do not qualify. Do not invent the
+ cases yourself; if the user asks for a draft, label it a draft and have them
+ review it. `tier3 <skill-dir>` and `validate <skill-dir>` generate one
+ starter case when none exists, so run them only after the user's cases are
+ in place. If the user has not agreed to adding `evals/` to the skill (for
+ example one installed by `npx skills add` or a plugin), copy the whole
+ directory to `<scratch>/skills/<name>/`, keeping the directory name equal to
+ the frontmatter `name`, and evaluate the copy. For skills that act on real
+ systems, keep cases runnable with fixtures or sidecars (`evals/files/`,
+ `evals/environment/`); only when that is impossible, and with the user's
+ agreement, phrase a prompt as "give the commands, do not run them", knowing
+ that Tier 3 then scores the advice, not the execution. Generate with
`skillevaluator create-eval-dataset <skill-dir> --full --prompt <skill-dir>/evals/EVAL.md`,
then review it. The default prefers author guidance, then matching retained
traces; existing datasets need `--force` or `--refine`. Skill-only
@@ -241,11 +264,11 @@ skillevaluator tier3 evaluate <skill-dir> --skip-with-skill
| --- | --- | --- |
| `skillevaluator: command not found` | CLI not installed or not on `PATH` | Workflow step 1: install with `uv tool install`, `pipx`, or `pip` in a virtual environment, or use `uvx` |
| Exit `2` | Unknown flag, bad value, or missing path | Check `skillevaluator <command> --help`; prefix paths named like commands with `./` |
-| Check shows `INCOMPLETE` | A scanner is missing or failed | Run `skillevaluator health-check --tier 1`; install the `security` extra or the missing scanner |
+| Check shows `INCOMPLETE` | A scanner is missing or failed | Run `skillevaluator health-check --tier 1`; ask first, then install the `security` extra or the missing scanner |
| "provider is required" / exit `1` before reports | No `SKILL_EVAL_LLM_PROVIDER` or key, or several keys exported | Set exactly one provider and its key; see providers reference |
| Quality below 70 | Missing `## Prerequisites`, `## Limitations`, `## Troubleshooting`, vague description, no examples | Read each finding's suggestion; do not stuff keywords |
| `INCONCLUSIVE: skill never activated` | Description does not match the prompts | Make the description name the tasks; add explicit-bucket cases |
-| `doctor` prints a traceback ending in `No module named 'harbor'` | The `tier3` extra is not installed | `pip install "skillevaluator[tier3]"` (ask first), then rerun `doctor` |
+| `doctor` says `Tier 3 dependencies are not installed` (0.9.0 and older: a traceback ending in `No module named 'harbor'`) | The `tier3` extra is not installed | Ask first, then reinstall with the extra the way the CLI was installed (`uv tool install --python 3.13 --reinstall "skillevaluator[all]"`, `pipx install --force --python python3.13 "skillevaluator[all]"`, or pip in the same virtual environment), then rerun `doctor` |
| Docker trials fail to start | Daemon down or outdated engine | `docker info`; rerun `skillevaluator doctor --env-mode docker` |
| Agent model rejected | Default model not valid for that agent | `--agent-model AGENT=MODEL`, verify with `doctor --verify-models` |
| Run incomplete (`Scored attempt coverage is n/m`) | A trial errored or timed out | Rerun; raise `--timeout-multiplier`; inspect with `--harbor-keep-jobs` |
diff --git a/skills/skillevaluator/evals/evals.json b/skills/skillevaluator/evals/evals.json
index 972b249f..4a8cd295 100644
--- a/skills/skillevaluator/evals/evals.json
+++ b/skills/skillevaluator/evals/evals.json
@@ -25,30 +25,16 @@
],
"expected_skill": "skillevaluator"
},
- {
- "id": "skillevaluator-009-explicit-cli-missing",
- "prompt": "Use SkillEvaluator to check whether the notes-helper skill in /workspace/input is well-formed and give me its quality score. SkillEvaluator may not be installed on this machine yet. You may install the CLI and any missing scanners needed for this keyless check using isolated or managed installs.",
- "expected_output": "Checks for the skillevaluator CLI (for example `skillevaluator --version`), installs it if missing with an isolated method such as `uv tool install \"skillevaluator[all]\"`, `pipx`, pip inside a virtual environment, or runs it through `uvx`, without sudo; then runs `skillevaluator quality-check` or `skillevaluator tier1 --no-llm` on the notes-helper skill and reports the score and exit code, or reports the exact install error if installation is impossible.",
- "assertions": [
- "Checks whether the skillevaluator CLI is available before relying on it",
- "Installs it without sudo using uv tool, pipx, pip in a virtual environment, or uvx when it is missing",
- "Runs a SkillEvaluator quality or Tier 1 check on the notes-helper skill",
- "Reports the quality score and exit code from the run, or the exact install error, without inventing results"
- ],
- "files": [
- "files/notes-helper"
- ],
- "expected_skill": "skillevaluator"
- },
{
"id": "skillevaluator-003-implicit-dataset",
"prompt": "I need test cases so an agent benchmark can tell whether my 'csv-cleaner' agent skill gets used correctly. Write the evals/evals.json file for it with three cases: one that names the skill, one that only describes the task, and one prompt where the skill should NOT be used.",
- "expected_output": "Asks for three to five real requests csv-cleaner should handle, the observable good result for each, any required input files, and one request where the skill should not be used. Explains that the reviewed cases will go in evals/evals.json, but does not invent or write a finished dataset before the user supplies those details.",
+ "expected_output": "Asks for three to five real requests csv-cleaner should handle, the observable good result for each, and one request where the skill should not be used. Explains that the reviewed cases will go in evals/evals.json. Does not present invented cases as a finished dataset; any draft it offers is labeled a draft for the user to review.",
"assertions": [
- "Asks for real requests csv-cleaner should handle and what a good observable result looks like for each",
- "Asks for one request that should not trigger the skill and any input files the cases need",
+ "Asks for real requests csv-cleaner should handle",
+ "Asks what a good observable result looks like for each request",
+ "Asks for one request that should not trigger the skill",
"Explains that reviewed cases will go in evals/evals.json",
- "Does not invent or write a finished dataset before getting the missing requests and outcomes"
+ "Does not present invented cases as a finished evals/evals.json; any cases it drafts are labeled a draft for the user to review"
],
"expected_skill": "skillevaluator"
},
@@ -108,7 +94,19 @@
"expected_skill": null
},
{
- "id": "skillevaluator-009-implicit-plain-quality",
+ "id": "skillevaluator-009-explicit-cli-missing",
+ "prompt": "Save the skill below as notes-helper/SKILL.md in your working directory, then use SkillEvaluator to check whether it is well-formed and give me its quality score. SkillEvaluator may not be installed on this machine yet. You may install the CLI and any missing scanners needed for this keyless check using isolated or managed installs.\n\n---\nname: notes-helper\ndescription: Summarize meeting notes into decisions and action items. Use when the user pastes meeting notes and asks for a summary, decisions, or owners.\n---\n\n# Notes Helper\n\nTurn raw meeting notes into a short summary with decisions and action items.\n\n## Workflow\n\n1. Read the notes the user provides.\n2. List the decisions that were made.\n3. List action items with an owner and due date when given.\n\n## Example\n\nInput: \"Ana will ship the beta Friday. We chose Postgres.\"\nOutput: Decisions: use Postgres. Actions: Ana - ship beta - Friday.\n",
+ "expected_output": "Checks for the skillevaluator CLI (for example `skillevaluator --version`), installs it if missing with an isolated method such as `uv tool install \"skillevaluator[all]\"`, `pipx`, pip inside a virtual environment, or runs it through `uvx`, without sudo; then runs `skillevaluator quality-check` or `skillevaluator tier1 --no-llm` on the notes-helper skill and reports the score and exit code, or reports the exact install error if installation is impossible.",
+ "assertions": [
+ "Checks whether the skillevaluator CLI is available before relying on it",
+ "Installs it without sudo using uv tool, pipx, pip in a virtual environment, or uvx when it is missing",
+ "Runs a SkillEvaluator quality or Tier 1 check on the saved notes-helper skill",
+ "Reports the quality score and exit code from the run, or the exact install error, without inventing results"
+ ],
+ "expected_skill": "skillevaluator"
+ },
+ {
+ "id": "skillevaluator-010-implicit-plain-quality",
"prompt": "Is my skill any good? It's in ./my-skill. I don't have an API key. If you can't run anything here, tell me exactly what you would run and what the result would and would not prove.",
"expected_output": "Chooses keyless static validation for ./my-skill (`skillevaluator tier1 ./my-skill --no-llm`, `quality-check`, or `validate ./my-skill --tiers 1`), says the result is a pass/fail gate plus a 0-100 quality score, and says static checks cover structure, wording and security but do not show whether the skill helps an agent.",
"assertions": [
@@ -119,8 +117,8 @@
"expected_skill": "skillevaluator"
},
{
- "id": "skillevaluator-010-implicit-written-well",
- "prompt": "Can you check whether the skill in ./skills/release-notes is written well? Tell me what you'd run and what the score is made of.",
+ "id": "skillevaluator-011-implicit-written-well",
+ "prompt": "Can you check whether the skill in ./skills/release-notes is written well? Tell me what you'd run, what the score is made of, and whether any part of it needs an API key.",
"expected_output": "Runs or gives a keyless quality command on ./skills/release-notes (`skillevaluator tier1 ./skills/release-notes --no-llm` or `quality-check`), explains the 0-100 quality score as Correctness 35%, Discoverability 25%, Reliability 25% and Efficiency 15% with a default pass mark of 70, and notes that the LLM rubric is a separate check that needs a provider.",
"assertions": [
"Gives a keyless quality command (tier1 --no-llm or quality-check) on ./skills/release-notes",
@@ -130,9 +128,9 @@
"expected_skill": "skillevaluator"
},
{
- "id": "skillevaluator-011-implicit-helps-agent-blocked",
+ "id": "skillevaluator-012-implicit-helps-agent-blocked",
"prompt": "I want to know whether my skill in ./skills/deploy-helper actually helps an agent do its tasks better. No API key is set on this machine and Docker isn't running. What do you need from me, and what can you do in the meantime?",
- "expected_output": "Says this needs a live with-skill versus baseline evaluation (Tier 3) and cannot run yet. Names what is missing: a provider key exported in the host environment and a running sandbox (Docker, or the experimental local mode). Asks the user to supply those. Offers what can be done now without them: keyless Tier 1 and preparing or validating eval cases. Reports no lift, score or verdict for the live evaluation.",
+ "expected_output": "Says this needs a live with-skill versus baseline evaluation (Tier 3) and cannot run yet. Names what is missing: a provider key exported in the host environment and a running Docker daemon. It does not offer `--env-mode local` for a skill that deploys unless the user trusts it. Asks the user to supply those. Offers what can be done now without them: keyless Tier 1 and preparing or validating eval cases. Reports no lift, score or verdict for the live evaluation.",
"assertions": [
"Says a live with-skill versus baseline evaluation is required to answer the question",
"Names both blockers: no provider key and no running Docker or other sandbox",
@@ -143,19 +141,19 @@
"expected_skill": "skillevaluator"
},
{
- "id": "skillevaluator-012-contextual-own-model",
+ "id": "skillevaluator-013-contextual-own-model",
"prompt": "I'm running you inside my coding agent, which already has a model behind it. When you run the live agent evaluation for my skill, will it just use that model, or do I have to set up another API key? Keep it short.",
- "expected_output": "Says no, the live evaluation does not use the calling agent's model. It starts separate agent processes in a sandbox, with and without the skill, and an LLM judge scores the answers, so a provider credential must be exported in the host environment. One provider key, for example NVIDIA_API_KEY, can cover both the agents and the judge. Keys never go in evals/config.yml.",
+ "expected_output": "Says no, the live evaluation does not use the calling agent's model. It starts separate agent processes in a sandbox, with and without the skill, and an LLM judge scores the answers, so a provider credential must be exported in the host environment. One provider key, for example NVIDIA_API_KEY, can cover both the agents and the judge.",
"assertions": [
"Says the live evaluation does not reuse the model of the agent that launches it",
"Explains that separate agent processes run the cases with and without the skill",
"Explains that an LLM judge also needs a provider",
- "Says the credential is exported in the host environment, not placed in config.yml"
+ "Says a provider credential must be exported in the host environment where skillevaluator runs"
],
"expected_skill": "skillevaluator"
},
{
- "id": "skillevaluator-013-contextual-dataset-questions",
+ "id": "skillevaluator-014-contextual-dataset-questions",
"prompt": "Set up live evaluation cases for my skill at ./skills/invoice-parser. Before you write anything, tell me what you need to know from me.",
"expected_output": "Asks the user for a few real requests the skill should handle and what a good result looks like for each, at least one request that should not trigger the skill, and any input files the cases need. Says the cases will go in evals/evals.json and should cover requests that name the skill, requests that only describe the task, requests inside a project, and a negative case. Does not invent finished cases before getting that input.",
"assertions": [
@@ -168,7 +166,7 @@
"expected_skill": "skillevaluator"
},
{
- "id": "skillevaluator-014-contextual-run-cost",
+ "id": "skillevaluator-015-contextual-run-cost",
"prompt": "I'm about to run the full live evaluation on ./skills/sql-helper: 10 cases, 3 attempts each, two agents. How many agent runs is that, and how do I check the setup cheaply before spending on all of it?",
"expected_output": "Computes 10 cases x 3 attempts x 2 agents x 2 arms (with skill and baseline) = 120 trials. Recommends a smoke run first with `--max-cases 1`, and says a smoke run is not publication evidence. May mention `--stop-on-pass` or `--skip-baseline` as ways to cut spend, noting that skipping the baseline gives no lift.",
"assertions": [
@@ -179,7 +177,7 @@
"expected_skill": "skillevaluator"
},
{
- "id": "skillevaluator-015-negative-write-skill-content",
+ "id": "skillevaluator-016-negative-write-skill-content",
"prompt": "Write a SKILL.md for a new agent skill that formats SQL queries. Include frontmatter, instructions and two examples.",
"expected_output": "A complete SKILL.md for a SQL formatting skill with name and description frontmatter, instructions and two examples. Writing the skill's own content does not call for an evaluation workflow.",
"assertions": [
@@ -190,16 +188,28 @@
"expected_skill": null
},
{
- "id": "skillevaluator-016-contextual-cold-install-consent",
- "prompt": "Check the skill at ./skills/notes-helper with keyless static validation. The skillevaluator CLI is installed, but SkillSpector and Gitleaks have never been installed or cached on this machine. I have not authorized downloading or installing tools. Tell me the next step before you run anything that could install them.",
- "expected_output": "Explains that keyless scanner checks or readiness probes may provision managed scanner runtimes. Names the missing scanners and asks permission before any download, tool installation or auto-provisioning scan/probe. May offer an already-installed keyless quality-only check while waiting; does not claim a completed security scan.",
+ "id": "skillevaluator-017-implicit-cold-install-consent",
+ "prompt": "Run a keyless static check on my skill at ./skills/notes-helper. SkillSpector and Gitleaks have never been installed on this machine, and I want to know before anything gets downloaded.",
+ "expected_output": "Explains that the keyless Tier 1 scan, `security-scan` or `health-check --tier 1` would download the missing SkillSpector and Gitleaks scanners (and that the CLI itself must be installed if it is missing), and asks permission before running any of them. May offer a provisioning-free quality check while waiting; does not claim a completed security scan or verdict.",
"assertions": [
- "Names SkillSpector and Gitleaks as missing scanner requirements",
- "Asks for permission before downloading, installing or launching a scanner/probe that provisions missing tools",
- "Does not run an auto-provisioning scan or readiness probe before permission",
+ "Explains that the keyless Tier 1 scan or health-check would download or provision the missing scanners",
+ "Asks permission before running any command that would install the CLI or provision scanners",
+ "Does not run tier1, security-scan or health-check before permission",
"Does not invent a completed security scan or security verdict"
],
"expected_skill": "skillevaluator"
+ },
+ {
+ "id": "skillevaluator-018-contextual-dataset-from-requests",
+ "prompt": "Set up live evaluation cases for my skill at ./skills/invoice-parser. Real requests: 1) \"Extract the total and due date from invoice.pdf in /workspace/input\" - a good result prints both values. 2) \"Which vendor billed us most last month?\" over the invoices in /workspace/input/invoices - a good result names the vendor and the amount. It should NOT be used for: \"Write a haiku about taxes.\" Write evals/evals.json.",
+ "expected_output": "Writes ./skills/invoice-parser/evals/evals.json with top-level skill_name 'invoice-parser' and an evals array built from the user's two requests and the negative request. Each case has id, prompt, expected_output and expected_skill; the positive cases name the observable results the user gave, and the negative case sets expected_skill to null.",
+ "assertions": [
+ "Output has top-level skill_name 'invoice-parser' and an evals array",
+ "Every case has id, prompt, expected_output and expected_skill",
+ "Uses the user's requests and outcomes instead of inventing new ones",
+ "The negative case sets expected_skill to null"
+ ],
+ "expected_skill": "skillevaluator"
}
]
}
diff --git a/skills/skillevaluator/evals/files/notes-helper/SKILL.md b/skills/skillevaluator/evals/files/notes-helper/SKILL.md
deleted file mode 100644
index 8e480492..00000000
--- a/skills/skillevaluator/evals/files/notes-helper/SKILL.md
+++ /dev/null
@@ -1,19 +0,0 @@
----
-name: notes-helper
-description: Summarize meeting notes into decisions and action items. Use when the user pastes meeting notes and asks for a summary, decisions, or owners.
----
-
-# Notes Helper
-
-Turn raw meeting notes into a short summary with decisions and action items.
-
-## Workflow
-
-1. Read the notes the user provides.
-2. List the decisions that were made.
-3. List action items with an owner and due date when given.
-
-## Example
-
-Input: "Ana will ship the beta Friday. We chose Postgres."
-Output: Decisions: use Postgres. Actions: Ana - ship beta - Friday.
diff --git a/skills/skillevaluator/references/commands-tier2-tier3.md b/skills/skillevaluator/references/commands-tier2-tier3.md
index afd216c0..e7cb7d98 100644
--- a/skills/skillevaluator/references/commands-tier2-tier3.md
+++ b/skills/skillevaluator/references/commands-tier2-tier3.md
@@ -38,7 +38,7 @@ Key `tier3 evaluate` flags: `-a, --agents` (`opencode`, `codex`,
| Command | Purpose |
| --- | --- |
-| `health-check [--tier 1\|2\|3\|all]` | Offline readiness; `--checks` for Tier 1 scanners |
+| `health-check [--tier 1\|2\|3\|all]` | Readiness; downloads pinned SkillSpector and Gitleaks when missing (ask first); `--checks` for Tier 1 scanners |
| `doctor [--tier ...] [--env-mode M] [-a AGENTS] [--verify-models]` | Deeper readiness; `--verify-models` makes live catalog probes |
| `models [--limit N] [--json]` | List provider catalog models |
| `ci init`, `ci status`, `ci demo` | Guided GitHub CI setup |