Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(tier1): stop deducting quality points for a missing root version (#553)

Narendran Raghavan1ea050cc32d3
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6e9df00c..f1d3d789 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -791,6 +791,13 @@ All notable changes to SkillEvaluator are documented in this file.
as in `tier3 validate`: for a skill named read or own, "read `SKILL.md`"
and "its own `SKILL.md`" are renamed as for any other skill.
+- Tier 1 quality no longer deducts 5 Correctness points, or reports
+ `SKILL_SPEC recommended field missing: 'version'`, when a skill has no root
+ `version` field. The version check reads only the optional
+ `metadata.version`, so a skill with that field or with no version scores
+ 1.75 points higher; the bundled `text-analyzer` example now scores 86.2
+ instead of 84.5.
+
## 0.9.0 - 2026-11-13
### Fixed
diff --git a/README.md b/README.md
index 6afbfd54..1698fcfa 100644
--- a/README.md
+++ b/README.md
@@ -106,7 +106,7 @@ uv run --python 3.13 skillevaluator quality-check \
-r cli -r json -o reports/first-quality
```
-No API key, scanner, GPU or Docker is needed. The example scores **84.5/100 (B)**,
+No API key, scanner, GPU or Docker is needed. The example scores **86.2/100 (B)**,
exits `0` and saves `reports/first-quality/skillevaluator-quality.json`. The
passing threshold defaults to 70.
diff --git a/docs/developer-guide.mdx b/docs/developer-guide.mdx
index b29ef1b7..b1df41d5 100644
--- a/docs/developer-guide.mdx
+++ b/docs/developer-guide.mdx
@@ -110,9 +110,12 @@ not reuse historical scores as a fresh run or silently change expectations to
make a check green. A fresh-user review should ask someone unfamiliar with the
tool to explain the passing, regressed and incomplete outputs before showing
the walkthrough's answers, and record what confused them in the pull request.
-The focused test intentionally runs the checkout's evaluator, so a scoring
-change on main can fail it even when the released replay still passes. Review
-both results before changing expectations.
+The focused test intentionally runs the checkout's evaluator. When the
+evaluator is the checkout's own source, the runner checks the `checkout` scores
+in `examples/starter/expected.json` instead of the pinned release's scores. A
+scoring change on main updates those `checkout` scores; the release scores and
+the starter guide change only when the pinned release does. Review both results
+before changing expectations.
## Code style and dependencies
diff --git a/docs/metrics-reference.mdx b/docs/metrics-reference.mdx
index ab1b64ca..e296986d 100644
--- a/docs/metrics-reference.mdx
+++ b/docs/metrics-reference.mdx
@@ -451,7 +451,7 @@ Quality = 0.35 * Correctness
| Area | Weight | What triggers a deduction |
| --- | --- | --- |
-| Correctness | 35% | XML tags in frontmatter, the name or the description (15 each), invalid or reserved name (15 each), missing `version` or `metadata.author` / `metadata.tags` (5 each), Windows-style paths (10), `SKILL.md` linking to `README.md` (5), no examples (5), plus the type-specific checks below |
+| Correctness | 35% | XML tags in frontmatter, the name or the description (15 each), invalid or reserved name (15 each), missing `metadata.author` / `metadata.tags` (5 each), Windows-style paths (10), `SKILL.md` linking to `README.md` (5), no examples (5), plus the type-specific checks below |
| Discoverability | 25% | Description under 20 characters (20), vague words (15), first or second person (15), no trigger wording (10), short or non-conforming skill name (10 each), over 200 characters (5), no `## Purpose` (5), broad description with no negative triggers (5) |
| Reliability | 25% | No mention of error handling (10), missing `## Prerequisites`, `## Limitations`, `## Troubleshooting` (5 each), scripts or library modules with no error handling (5) |
| Efficiency | 15% | Over the 5,000 estimated-token budget (15), instructions with no action verbs (15), deeply nested references (10), high line repetition (10), corporate buzzwords, non-descriptive reference filenames, non-doc files in `references/`, instructions not in list format (5 each) |
diff --git a/docs/quickstart.mdx b/docs/quickstart.mdx
index bbaa1f00..f5c53681 100644
--- a/docs/quickstart.mdx
+++ b/docs/quickstart.mdx
@@ -42,8 +42,8 @@ uv run --python 3.13 skillevaluator quality-check \
```
The terminal shows a score from 0–100, an A–F grade, and individual findings.
-With the current quality checks, the bundled example scores **84.5/100 (B)**
-with **7 findings**. It passes the default minimum score of 70 and exits `0`.
+With the current quality checks, the bundled example scores **86.2/100 (B)**
+with **6 findings**. It passes the default minimum score of 70 and exits `0`.
The same result is saved to:
```text
diff --git a/docs/starter-demo.mdx b/docs/starter-demo.mdx
index e867425b..e8da5ece 100644
--- a/docs/starter-demo.mdx
+++ b/docs/starter-demo.mdx
@@ -46,7 +46,10 @@ same demonstration, run the command above; `quality.txt` and `regressed.txt`
retain the terminal output, and `receipt.json` records the version and time.
Expected scalar values live in
[`examples/starter/expected.json`](https://github.com/NVIDIA/SkillEvaluator/blob/main/examples/starter/expected.json);
-they are checked against new results, not substituted for them. Historical
+they are checked against new results, not substituted for them. The scores
+in this guide are from the pinned 0.8.0 release. The current source can score
+the same skill differently, so a run with your checkout's own evaluator is
+checked against the `checkout` scores in that file instead. Historical
benchmark scores are not evidence for this run.
## Interpret the reports
diff --git a/examples/starter/expected.json b/examples/starter/expected.json
index 68b2c2d1..999a58d2 100644
--- a/examples/starter/expected.json
+++ b/examples/starter/expected.json
@@ -12,5 +12,6 @@
]
},
"quality": {"score": 84.5, "grade": "B", "status": "passed", "exit_code": 0, "publication": "incomplete"},
- "regressed": {"score": 71.0, "grade": "C", "status": "failed", "exit_code": 1, "publication": "fail"}
+ "regressed": {"score": 71.0, "grade": "C", "status": "failed", "exit_code": 1, "publication": "fail"},
+ "checkout": {"quality": {"score": 86.2}, "regressed": {"score": 72.8}}
}
diff --git a/scripts/demo_starter.py b/scripts/demo_starter.py
index 25f0275e..15006fb3 100644
--- a/scripts/demo_starter.py
+++ b/scripts/demo_starter.py
@@ -14,6 +14,7 @@ import sys
from datetime import UTC, datetime
from importlib.metadata import version
from importlib.resources import as_file, files
+from importlib.util import find_spec
from pathlib import Path
FIXTURES = Path(__file__).resolve().parents[1] / "examples" / "starter"
@@ -65,8 +66,26 @@ def check_quality(report: dict, expected: dict) -> None:
)
-def demo(output: Path) -> dict:
+def load_expected() -> dict:
+ """Return the expected results for the evaluator that runs the demo.
+
+ The top-level values are what the release pinned in the starter guide reports.
+ When the evaluator is this checkout's own source, the ``checkout`` values
+ replace them, because the source can score the starter differently until a
+ release includes its changes.
+ """
expected = json.loads((FIXTURES / "expected.json").read_text(encoding="utf-8"))
+ checkout = expected.pop("checkout", {})
+ evaluator = find_spec("skillevaluator")
+ source = (FIXTURES.parents[1] / "src").resolve()
+ if evaluator and evaluator.origin and Path(evaluator.origin).resolve().is_relative_to(source):
+ for label, values in checkout.items():
+ expected[label] |= values
+ return expected
+
+
+def demo(output: Path) -> dict:
+ expected = load_expected()
output.mkdir(parents=True, exist_ok=False)
skill = output / "text-analyzer"
reference = files("skillevaluator.tier3.reference_skills").joinpath("text-analyzer")
diff --git a/src/skillevaluator/validators/quality_score.py b/src/skillevaluator/validators/quality_score.py
index 09896e98..1b26dae1 100644
--- a/src/skillevaluator/validators/quality_score.py
+++ b/src/skillevaluator/validators/quality_score.py
@@ -981,18 +981,6 @@ class QualityScoreValidator(ValidatorBase):
# as absent metadata rather than indexing into it.
metadata = {}
- top_level_fields = {
- "version": 'Semantic version (e.g., "1.0.0")',
- }
- for field_name, desc in top_level_fields.items():
- if field_name not in frontmatter or frontmatter[field_name] is None:
- qs.correctness.deduct(
- 5,
- "warning",
- f"SKILL_SPEC recommended field missing: '{field_name}'",
- f"Add '{field_name}' to frontmatter — {desc}",
- )
-
nested_fields = {
"author": "Author name or team (under metadata:)",
"tags": "Categorization tags (under metadata:, list of 1-5 items)",
diff --git a/tests/test_cli.py b/tests/test_cli.py
index cdf1902b..753402ec 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -68,12 +68,12 @@ def test_quickstart_quality_check_matches_documented_result(
payload = json.loads((reports / "skillevaluator-quality.json").read_text(encoding="utf-8"))
assert len(payload["results"]) == 1
quality_result = payload["results"][0]
- assert quality_result["quality"]["overall_score"] == 84.5
+ assert quality_result["quality"]["overall_score"] == 86.2
assert quality_result["quality"]["grade"] == "B"
- assert len(quality_result["findings"]) == 7
+ assert len(quality_result["findings"]) == 6
assert payload["overall_passed"] is (expected_exit_code == 0)
assert payload["publication_status"] == ("incomplete" if expected_exit_code == 0 else "fail")
- assert "84.5/100" in result.output
+ assert "86.2/100" in result.output
def test_top_level_help_groups_commands_by_workflow() -> None:
diff --git a/tests/test_demo_starter.py b/tests/test_demo_starter.py
index 8f0bc19d..ceae0d4d 100644
--- a/tests/test_demo_starter.py
+++ b/tests/test_demo_starter.py
@@ -4,14 +4,20 @@
import copy
import json
import os
+import shutil
import subprocess
import sys
from pathlib import Path
import pytest
+from scripts import demo_starter
from scripts.demo_starter import FIXTURES, check_quality
ROOT = Path(__file__).resolve().parents[1]
+# What the release wheel pinned in the starter guide reports for the starter and its
+# shortened copy. The checkout's evaluator can score them differently until a release
+# includes its scoring changes.
+PINNED_RELEASE_SCORES = {"quality": 84.5, "regressed": 71.0}
@pytest.mark.parametrize("page", ["docs/starter-demo.mdx", "examples/starter/README.md"])
@@ -42,7 +48,7 @@ def test_starter_runs_real_cli_and_preserves_sources(tmp_path):
assert result.returncode == 0, result.stdout + result.stderr
assert "PASS: starter output" in result.stdout
assert source.read_bytes() == original
- expected = json.loads((FIXTURES / "expected.json").read_text())
+ expected = demo_starter.load_expected()
assert json.loads((output / "analysis.json").read_text()) == expected["analysis"]
for label in ("quality", "regressed"):
report = json.loads((output / label / "skillevaluator-quality.json").read_text())
@@ -63,6 +69,52 @@ def test_starter_runs_real_cli_and_preserves_sources(tmp_path):
assert (output / "receipt.json").read_bytes() == before
+def test_installed_release_is_checked_against_the_pinned_release_scores(tmp_path, monkeypatch):
+ """The documented run checks the pinned release's scores, not the checkout's."""
+ # Demo files outside the evaluator's source tree, as when the release wheel runs them.
+ fixtures = tmp_path / "checkout" / "examples" / "starter"
+ shutil.copytree(FIXTURES, fixtures)
+ monkeypatch.setattr(demo_starter, "FIXTURES", fixtures)
+ expected = json.loads((fixtures / "expected.json").read_text(encoding="utf-8"))
+
+ def release_cli(command, log, expected_exit):
+ if "quality-check" not in command:
+ log.write_text(json.dumps(expected["analysis"]), encoding="utf-8")
+ return
+ reports = Path(command[command.index("-o") + 1])
+ reports.mkdir()
+ report = {
+ "overall_status": expected[log.stem]["status"],
+ "overall_passed": expected_exit == 0,
+ "quality_summary": [
+ {"overall_score": PINNED_RELEASE_SCORES[log.stem], "grade": expected[log.stem]["grade"]}
+ ],
+ "publication_status": expected[log.stem]["publication"],
+ "publication": {"eligible": False, "reasons": ["Live evaluation did not run"]},
+ }
+ (reports / "skillevaluator-quality.json").write_text(json.dumps(report), encoding="utf-8")
+
+ monkeypatch.setattr(demo_starter, "run", release_cli)
+
+ receipt = demo_starter.demo(tmp_path / "starter")
+
+ assert receipt["live_agent_evaluation"] == "not run"
+
+
+ ("page", "template"),
+ [
+ ("docs/starter-demo.mdx", "score {score:g}, grade {grade}"),
+ ("examples/starter/README.md", "scores {score:g}/100 ({grade})"),
+ ],
+)
+def test_starter_docs_show_the_pinned_release_scores(page, template):
+ text = (ROOT / page).read_text(encoding="utf-8")
+ expected = json.loads((FIXTURES / "expected.json").read_text(encoding="utf-8"))
+ for label, score in PINNED_RELEASE_SCORES.items():
+ assert template.format(score=score, grade=expected[label]["grade"]) in text
+
+
@pytest.fixture
def quality_report():
return {
diff --git a/tests/test_quality_policy_gate.py b/tests/test_quality_policy_gate.py
index 63e3814b..6044adda 100644
--- a/tests/test_quality_policy_gate.py
+++ b/tests/test_quality_policy_gate.py
@@ -76,7 +76,7 @@ print(json.dumps({
)
assert json.loads(result.stdout) == {
"passed": [True, True, True, True],
- "scores": [67.8, 67.8, 67.8, 67.8],
+ "scores": [69.5, 69.5, 69.5, 69.5],
"explicit_override_passed": False,
}
@@ -85,7 +85,7 @@ def test_raising_efficiency_severity_keeps_cli_minimum_score_failure(
below_threshold_skill: Path, tmp_path: Path
) -> None:
result = QualityScoreValidator(min_score=70).validate(below_threshold_skill)
- assert result.metadata["quality_scores"]["overall_score"] == 67.8
+ assert result.metadata["quality_scores"]["overall_score"] == 69.5
assert not result.passed
policy = tmp_path / "stricter-policy.yaml"
policy.write_text("severity_overrides:\n QUALITY.quality_efficiency: medium\n", encoding="utf-8")
@@ -111,7 +111,7 @@ def test_raising_efficiency_severity_keeps_cli_minimum_score_failure(
assert stricter.exit_code == 1, stricter.output
output = strip_terminal_controls(stricter.output)
assert "[PASS] All validations passed" not in output
- assert "67.8/100 is below the required 70/100" in output
+ assert "69.5/100 is below the required 70/100" in output
def test_pipeline_explains_minimum_score_failure(below_threshold_skill: Path) -> None:
@@ -123,7 +123,7 @@ def test_pipeline_explains_minimum_score_failure(below_threshold_skill: Path) ->
assert "".join(result.gate_failures[0].split()) in "".join(fix_text.split())
fix = first_fix([result])
assert fix is not None
- assert "67.8/100 is below the required 70/100" in fix[0][0]
+ assert "69.5/100 is below the required 70/100" in fix[0][0]
def test_json_report_includes_minimum_score_failure(below_threshold_skill: Path) -> None:
diff --git a/tests/validators/test_quality_score.py b/tests/validators/test_quality_score.py
index 98933590..61a28417 100644
--- a/tests/validators/test_quality_score.py
+++ b/tests/validators/test_quality_score.py
@@ -406,6 +406,22 @@ class TestQualityScoreValidator:
result = QualityScoreValidator(min_score=0).validate(skill_dir)
assert any("metadata.author" in finding.message for finding in result.findings)
+ assert all("version" not in finding.message.lower() for finding in result.findings)
+
+ @pytest.mark.parametrize(
+ "metadata",
+ ['{version: "1.0.0", author: Test User, tags: [testing]}', "{author: Test User, tags: [testing]}"],
+ ids=["metadata-version", "no-version"],
+ )
+ def test_optional_version_costs_no_correctness_points(self, tmp_path, metadata):
+ """A skill with an optional ``metadata.version``, or with none, is not told to add a root ``version``."""
+ skill_dir = _write_metadata_skill(tmp_path, metadata)
+
+ result = QualityScoreValidator(min_score=0).validate(skill_dir)
+
+ correctness = result.metadata["quality_scores"]["dimensions"]["correctness"]
+ assert all("version" not in issue["message"].lower() for issue in correctness["issues"])
+ assert all("version" not in finding.message.lower() for finding in result.findings)
@pytest.mark.parametrize("metadata", ["[author, tags]", '"author: Test User"', "true", "1.5", "2026-01-01"])
def test_non_mapping_metadata_counts_as_missing_fields(self, tmp_path, metadata):