Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
docs(tier1): say a PowerShell backtick before a space is not a quoted split (#674)
Narendran Raghavanabac7c5f575d
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ffd5bacc..13dc2bb2 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -316,6 +316,10 @@ All notable changes to SkillEvaluator are documented in this file.### Documentation+- The Tier 1 validation guide now says that a PowerShell backtick before a+ space does not count as a quoted or escaped space for the SkillSpector+ missing-reference line check, so a labelled line such as+ ``**Note:** run setup` h*.ps1`` passes when `setup helper.ps1` is left out.- The agent skill guide explicitly installs the CLI before the first check,so its setup flow also works with skill versions that do not bootstrap it.diff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdxindex 49ebeeab..ccba7cd3 100644--- a/docs/tier1-validation.mdx+++ b/docs/tier1-validation.mdx@@ -334,7 +334,7 @@ For `validate`, the default set is `schema`, `version`, `security`, `pii`, `lice- A span between backticks is read as Markdown code, as written and with its backticks removed, not as a command, so a plural written as `` `topic`s `` is read as `topics`. A leading Markdown `* ` list marker is ignored.- A `$` before a space or the end of the line is read as a literal `$`, as a shell reads it, not as an expansion, so a shell prompt `$ python run.py` adds no pattern, and `sh *a$` matches only names that end in `a$`.- A bold or italic label outside code, such as `**Note:**`, `*Tip:*` or `**Step 1:**`, is read as its text, and each of its words, with its `*`, as a pattern that is not split at `:`, `=` or `,`. So as a pattern, `**Note:**` matches only names that hold `note:`, and `**Step 1:**` only names that end in `step` or start with `1:`. A standalone shell wildcard elsewhere, such as `bash *`, still counts, and so does a label inside code.- - On a line with a space or shell delimiter between two of the same quote or after a backslash, as written or decoded as above, such as `setup' 'helper.sh`, `"hi there"` or `setup\ h*.sh`, a `$` before a space and a label are read like any other word, so they still stand for any text: a shell may join words there, which the check does not. A PowerShell backtick before a space does not count, as a backtick also ends Markdown code, so ``$ sh setup` h*.ps1`` does not name `setup helper.ps1`.+ - On a line with a space or shell delimiter between two of the same quote or after a backslash, as written or decoded as above, such as `setup' 'helper.sh`, `"hi there"` or `setup\ h*.sh`, a `$` before a space and a label are read like any other word, so they still stand for any text: a shell may join words there, which the check does not. A PowerShell backtick before a space does not count, as a backtick also ends Markdown code, so ``$ sh setup` h*.ps1`` does not name `setup helper.ps1`. A PowerShell backtick before a space does not count here: in ``**Note:** run setup` h*.ps1`` the label is read as its text, and ``setup` h*.ps1`` does not match a left-out `setup helper.ps1`, which PowerShell would run.- Prose with this syntax is read the same way: a price such as `$5`, a list such as `{a,b}`, `%APPDATA%` or a lone `^` stands for any text, so a line with one stays `INCOMPLETE` whenever anything is left out of the scan, and `approx~10` stays `INCOMPLETE` when a name left out holds an `a`.- The check reads names that are written in the reported line, in the spellings above. It does not read other lines and does not run commands, so it does not detect a name on another line, such as `DIR=node_modules` set on one line and used as `sh $DIR/run.sh` on the next, or a name that a command prints or builds, such as `printf` output in backticks, a list piped into `xargs sh`, `find -exec`, or a PowerShell string joined with `+`. Nor does it join words at a quoted or escaped space as a shell does, so `setup' 'helper.sh` or `setup\ h*.sh` does not name `setup helper.sh`. It is meant to keep an instruction that names a left-out file `INCOMPLETE`, not to catch a skill written to hide one. Without this check, such a skill passes whenever SkillSpector reports no missing reference in it; with it, the skill also passes when every line SkillSpector reports is another missing reference, such as `output/notes.md`. SkillSpector's own analyzers, not this check, assess the instructions it scanned.- When the target is a single file, such as `skillevaluator security-scan path/to/SKILL.md`, SkillSpector scans that file alone, so every other file and directory beside it counts as not inspected. A line in it that names a file beside it, such as `scripts/install.sh`, keeps the scan `INCOMPLETE`.