Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

docs(changelog): move post-0.9.0 entries to Unreleased (#506)

Narendran Raghavanf075bb87acf7
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e3567da1..8807e85c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -40,6 +40,45 @@ All notable changes to SkillEvaluator are documented in this file.
beside it and the failure is reported. After a timeout, the failed run still keeps
the scores of the arms that finished.
+- Tier 1 **Incomplete referenced artifact analysis (AE1)** findings from
+ SkillSpector now say why SkillSpector stopped inspecting the file that a skill
+ names. The suggestion names the file, its disposition and each reason
+ SkillSpector gives, such as
+ `static_parse_limit (static_patterns_tool_misuse)`, with any line and limits,
+ and the finding's metadata records them under `skillspector_evidence` and
+ `skillspector_occurrences`. The suggestion leaves out the run time and the
+ time budget (`observed_seconds`, `limit_seconds`), which differ from run to
+ run, so a finding keeps its `finding_id` from run to run; because the
+ suggestion text is new, each AE1 `finding_id` changes once with this release.
+ Before, the finding said only
+ `Referenced artifact was not completely inspected`, and its reason had to be
+ matched by hand against the incomplete-analysis error, which lists at most
+ three rows. With SkillSpector before v2.12.0 the finding was
+ titled `Unknown (AE1)` and told the user to make the file available although
+ SkillSpector had read part of it. It now carries the rule's name and the
+ reasons from the report's ledger rows for the file. Its check name, and so its
+ SARIF rule ID, changes: a `severity_overrides` entry keyed on
+ `SECURITY.Unknown (AE1)` must now use
+ `SECURITY.Incomplete referenced artifact analysis (AE1)`. The JSON field
+ `skillspector_reference_summary` counts the files and the lines that name them
+ for every result, and a failed check shows the counts in one line of the
+ terminal, Markdown and HTML reports. Severities and scan completeness are
+ unchanged; a gate changes only where an override of `SECURITY.Unknown (AE1)`
+ was what let the check pass.
+
+- Tier 3 `validate-grader --qualification-policy FILE` supports explicit,
+ assessment-only metric/class/category coverage and risk limits, provider-free
+ preflight checks, retained not-run receipts for insufficient labels, final
+ policy decisions and tracked verifier-source Git provenance. Correct expected
+ abstentions demonstrate category coverage without supplying binary scored
+ cases; evaluated report scope must match its selected assessment cases. The
+ experimental example is opt-in and fails bundled-suite class coverage; it establishes no
+ live reliability or release approval. Resource comparison metadata and HTML
+ also explain that observed consumption includes task failures, lower usage
+ alone does not establish better Efficiency or quality, pairing does not certify
+ equal concurrency/budget or judge-model agreement, and reused telemetry is
+ historical. Scoring, raw deltas and reuse eligibility are unchanged.
+
### Changed
- Install Bandit with the base package, so `pip install skillevaluator` or
@@ -47,6 +86,13 @@ All notable changes to SkillEvaluator are documented in this file.
extra. pip-audit, used only by the opt-in dependency audit
(`--check dependency`), stays in the `security` extra. The Bandit install hint
now says to reinstall SkillEvaluator.
+- Root help shows the SkillEvaluator wordmark in more terminals: the
+ one-line wordmark still needs 115 columns, 76-114 columns now get the same
+ letters stacked (SKILL over EVALUATOR), and 60-75 columns get a two-row
+ wordmark. Narrower terminals keep the plain title. A wordmark that the
+ terminal's output encoding cannot print is skipped: gbk and big5hkscs lack
+ the half blocks of the two-row one, so they get the plain title at 60-75
+ columns.
### Documentation
@@ -87,6 +133,13 @@ All notable changes to SkillEvaluator are documented in this file.
instead of being averaged in as a zero.
`SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT` and
`SKILL_EVAL_BEHAVIOR_CHECK_BUDGET` still override both limits.
+- Code Risk Analysis reports Semgrep file-level parse errors as one partial-scan
+ error that names up to 20 unparsed files, relative to the skill and with the
+ line of the first error, instead of the first three truncated messages. The
+ check stays `incomplete` because Semgrep skips the code it cannot parse. The
+ JSON report lists the files in `semgrep_unparsed_files` (up to 1,000, with
+ `semgrep_unparsed_files_omitted` counting the rest); other Semgrep errors are
+ reported as before.
- The supporting Tier 3 Pass@k table displays an unavailable or skipped
baseline rate as `N/A`, preserving an observed baseline rate of `0%`.
- Tier 3 budget summaries disclose measured usage above configured limits,
@@ -126,6 +179,46 @@ All notable changes to SkillEvaluator are documented in this file.
verdict line under it does, so the two agree, including with `--skip` and
`--fail-fast`.
+- A Tier 1 security scan with SkillSpector v2.12.0 or later no longer stops as
+ `INCOMPLETE` ("Security Scan didn't run") with
+ `hooks/hooks.json [static; opaque_content; analyzers=bundled_execution_surface]`
+ when the scanned skill, folder or hooks-only plugin has an ordinary command
+ or HTTP hook, one SkillSpector does not report as sending hook content to a
+ remote host (BH2). SkillSpector cannot model what such a hook does with its
+ input, so it reports the analysis as `partial` although the scan finished,
+ and its findings cannot show that the hook is safe: a hook script that sends
+ its input over a socket, `nc` or `/dev/tcp` gets no blocking finding. The
+ scan now fails with the Critical `skillspector_hook_payload_unverified`,
+ which names the manifest, such as `hooks/hooks.json` or
+ `.claude/settings.json`, and a gate failure that neither a `--policy`
+ severity override nor `--llm-verify` clears. Only removing the hooks passes
+ the check. The other findings and the `--llm` stage are reported as for any
+ complete scan, and a folder scan goes on to the next skill. This applies
+ only to SkillSpector's nonfatal, whole-file row on a manifest whose BH1
+ finding says its hook payload is `unmodeled`, also under a baseline, and
+ only when the report accounts for the row exactly. This covers only
+ manifests and hook evidence the scanner reports, without claiming coverage
+ for inline plugin-manifest hooks or other unreported hook locations.
+ SkillSpector records the same row for a manifest it read only in part:
+ with v2.13.4, whose row counts
+ the records it read, a manifest with a record that is not a parsed hook
+ declaration, such as a skipped handler or a permission rule, stays
+ `INCOMPLETE`; with v2.13.3, only a BH1 with 2,048 or more declarations does,
+ and a partly parsed manifest gets the Critical. The rest of the report is
+ judged as before: missing references and referenced images are still
+ accepted with their warnings under the rules above; the v2.13.3 missing-path
+ reference caveat that is accepted from the report alone does not apply
+ beside the hooks, because it reads the whole report. Any gap those rules do
+ not accept, such as a missing reference in a scan without a known target or
+ in a skill that `validate` or `security-scan` checks on its own inside a
+ folder or plugin, still records the SkillSpector scan as incomplete
+ (`incomplete_scans` in the JSON report, and publication eligibility
+ `Incomplete`), with an error that names only that gap. The check itself is
+ reported as `FAIL`, not `INCOMPLETE`, because the Critical is a confirmed
+ failure. With SkillSpector v2.13.4, a hook with a `"timeout"` key also gets
+ a `static_parse_limit` row on the manifest, so that scan is recorded as
+ incomplete in the same way.
+
## 0.9.0 - 2026-11-13
### Fixed
@@ -138,18 +231,6 @@ All notable changes to SkillEvaluator are documented in this file.
### Added
-- Tier 3 `validate-grader --qualification-policy FILE` supports explicit,
- assessment-only metric/class/category coverage and risk limits, provider-free
- preflight checks, retained not-run receipts for insufficient labels, final
- policy decisions and tracked verifier-source Git provenance. Correct expected
- abstentions demonstrate category coverage without supplying binary scored
- cases; evaluated report scope must match its selected assessment cases. The
- experimental example is opt-in and fails bundled-suite class coverage; it establishes no
- live reliability or release approval. Resource comparison metadata and HTML
- also explain that observed consumption includes task failures, lower usage
- alone does not establish better Efficiency or quality, pairing does not certify
- equal concurrency/budget or judge-model agreement, and reused telemetry is
- historical. Scoring, raw deltas and reuse eligibility are unchanged.
- Dataset fairness validation checks the evaluated skill's own names and nested
graded text, distinguishes negated reads from required reads, and bounds
warning excerpts and repeated-match scanning. Common Unicode, Markdown and
@@ -337,32 +418,6 @@ All notable changes to SkillEvaluator are documented in this file.
and displays the source and review state in terminal, HTML, and BENCHMARK
reports. Run payloads retain the generation record with shortened input hashes.
-- Tier 1 **Incomplete referenced artifact analysis (AE1)** findings from
- SkillSpector now say why SkillSpector stopped inspecting the file that a skill
- names. The suggestion names the file, its disposition and each reason
- SkillSpector gives, such as
- `static_parse_limit (static_patterns_tool_misuse)`, with any line and limits,
- and the finding's metadata records them under `skillspector_evidence` and
- `skillspector_occurrences`. The suggestion leaves out the run time and the
- time budget (`observed_seconds`, `limit_seconds`), which differ from run to
- run, so a finding keeps its `finding_id` from run to run; because the
- suggestion text is new, each AE1 `finding_id` changes once with this release.
- Before, the finding said only
- `Referenced artifact was not completely inspected`, and its reason had to be
- matched by hand against the incomplete-analysis error, which lists at most
- three rows. With SkillSpector before v2.12.0 the finding was
- titled `Unknown (AE1)` and told the user to make the file available although
- SkillSpector had read part of it. It now carries the rule's name and the
- reasons from the report's ledger rows for the file. Its check name, and so its
- SARIF rule ID, changes: a `severity_overrides` entry keyed on
- `SECURITY.Unknown (AE1)` must now use
- `SECURITY.Incomplete referenced artifact analysis (AE1)`. The JSON field
- `skillspector_reference_summary` counts the files and the lines that name them
- for every result, and a failed check shows the counts in one line of the
- terminal, Markdown and HTML reports. Severities and scan completeness are
- unchanged; a gate changes only where an override of `SECURITY.Unknown (AE1)`
- was what let the check pass.
-
- `create-eval-dataset --grounding` selects skill, author-intent, or trace
grounding. Generated datasets record relative input provenance and review
flags for observed answers that differ from authored expectations and
@@ -568,14 +623,6 @@ All notable changes to SkillEvaluator are documented in this file.
include accessible descriptions, links to data tables and visible fallbacks
when canvas initialization is unavailable; Tier 1 reports omit chart assets.
-- Root help shows the SkillEvaluator wordmark in more terminals: the
- one-line wordmark still needs 115 columns, 76-114 columns now get the same
- letters stacked (SKILL over EVALUATOR), and 60-75 columns get a two-row
- wordmark. Narrower terminals keep the plain title. A wordmark that the
- terminal's output encoding cannot print is skipped: gbk and big5hkscs lack
- the half blocks of the two-row one, so they get the plain title at 60-75
- columns.
-
- Separate HTML Tier 2 evidence into within-skill and across-skill comparisons,
with recorded completion, duplicate, affected-path, and catalog counts. Keep
skipped or unavailable comparisons distinct from completed zero-match scans.
@@ -812,14 +859,6 @@ All notable changes to SkillEvaluator are documented in this file.
- The unknown-artifact CLI guidance links to the published content-types documentation route.
-- Code Risk Analysis reports Semgrep file-level parse errors as one partial-scan
- error that names up to 20 unparsed files, relative to the skill and with the
- line of the first error, instead of the first three truncated messages. The
- check stays `incomplete` because Semgrep skips the code it cannot parse. The
- JSON report lists the files in `semgrep_unparsed_files` (up to 1,000, with
- `semgrep_unparsed_files_omitted` counting the rest); other Semgrep errors are
- reported as before.
-
- Security checks preserve authored evaluation files and their findings while
excluding generated outputs. Fully inspected skills with only missing-path
reference caveats retain visible warnings instead of an incomplete-scan
@@ -1153,46 +1192,6 @@ All notable changes to SkillEvaluator are documented in this file.
embedding input it can send doubles with the ceiling, from 512 to 1,024
texts (724 with either 2,048-dimensional default).
-- A Tier 1 security scan with SkillSpector v2.12.0 or later no longer stops as
- `INCOMPLETE` ("Security Scan didn't run") with
- `hooks/hooks.json [static; opaque_content; analyzers=bundled_execution_surface]`
- when the scanned skill, folder or hooks-only plugin has an ordinary command
- or HTTP hook, one SkillSpector does not report as sending hook content to a
- remote host (BH2). SkillSpector cannot model what such a hook does with its
- input, so it reports the analysis as `partial` although the scan finished,
- and its findings cannot show that the hook is safe: a hook script that sends
- its input over a socket, `nc` or `/dev/tcp` gets no blocking finding. The
- scan now fails with the Critical `skillspector_hook_payload_unverified`,
- which names the manifest, such as `hooks/hooks.json` or
- `.claude/settings.json`, and a gate failure that neither a `--policy`
- severity override nor `--llm-verify` clears. Only removing the hooks passes
- the check. The other findings and the `--llm` stage are reported as for any
- complete scan, and a folder scan goes on to the next skill. This applies
- only to SkillSpector's nonfatal, whole-file row on a manifest whose BH1
- finding says its hook payload is `unmodeled`, also under a baseline, and
- only when the report accounts for the row exactly. This covers only
- manifests and hook evidence the scanner reports, without claiming coverage
- for inline plugin-manifest hooks or other unreported hook locations.
- SkillSpector records the same row for a manifest it read only in part:
- with v2.13.4, whose row counts
- the records it read, a manifest with a record that is not a parsed hook
- declaration, such as a skipped handler or a permission rule, stays
- `INCOMPLETE`; with v2.13.3, only a BH1 with 2,048 or more declarations does,
- and a partly parsed manifest gets the Critical. The rest of the report is
- judged as before: missing references and referenced images are still
- accepted with their warnings under the rules above; the v2.13.3 missing-path
- reference caveat that is accepted from the report alone does not apply
- beside the hooks, because it reads the whole report. Any gap those rules do
- not accept, such as a missing reference in a scan without a known target or
- in a skill that `validate` or `security-scan` checks on its own inside a
- folder or plugin, still records the SkillSpector scan as incomplete
- (`incomplete_scans` in the JSON report, and publication eligibility
- `Incomplete`), with an error that names only that gap. The check itself is
- reported as `FAIL`, not `INCOMPLETE`, because the Critical is a confirmed
- failure. With SkillSpector v2.13.4, a hook with a `"timeout"` key also gets
- a `static_parse_limit` row on the manifest, so that scan is recorded as
- incomplete in the same way.
-
- Tier 3 HTML evaluator cards now say when a judge metric's lift left out cases
whose arms were scored by different judge models, for example after a judge
fallback in one arm. The arm scores keep those cases, so a card could show