Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(tier1): report manifest checks as N/A when validate finds no supported artifact (#410)

Narendran Raghavan0445c4dbf399
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 29d855e2..244f8339 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -396,10 +396,29 @@ All notable changes to SkillEvaluator are documented in this file.
- Repository CI can opt in to reviewed, per-skill SkillSpector baselines from
the PR or merge request's target revision. Policy files from the proposed
changes cannot approve their own exceptions.
+
+- `skillevaluator validate` on a target with no skill, rules, workflows, or
+ plugin, such as an ordinary source repository, no longer fails Tier 1 with
+ three `No skills found` errors and exit 1 when every content scan passes.
+ When `--type auto` detects the type as `unknown`, the `schema`, `version`,
+ and `quality` checks are now recorded as not applicable, never as passes:
+ the terminal summary and the HTML report count them as N/A, and the JSON
+ and Markdown reports list them as skipped with a `not applicable` reason.
+ The JSON report of every such run adds a root `tier1_applicability` object.
+ The content scans still run with their unrestricted scope, so a real
+ finding still fails the run; such a run is never eligible for publication.
+ A target folder whose own `SKILL.md`, `workflow-rules.mdc`,
+ `agent_plugin.yaml`/`.yml`, or `.claude-plugin/plugin.json` is mis-cased
+ (such as `Skill.md`) or a dangling symlink still fails its manifest checks,
+ and `--type skill` still validates the target as a skill.
+
+- The unknown-artifact CLI guidance links to the published content-types documentation route.
+
- Security checks preserve authored evaluation files and their findings while
excluding generated outputs. Fully inspected skills with only missing-path
reference caveats retain visible warnings instead of an incomplete-scan
failure; actual coverage gaps and analyzer failures still block.
+
- PII findings clarify that required author attribution is accepted while
duplicate contact addresses need removal, replacement, or a reviewed waiver.
diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdx
index 138123bb..2f8445ce 100644
--- a/docs/cli-reference.mdx
+++ b/docs/cli-reference.mdx
@@ -284,6 +284,8 @@ The [evaluated source provenance](#evaluated-source-provenance) flags also appea
Auto-detection recognizes: a `SKILL.md` for skills, `.mdc` files for rules, `workflow-rules.mdc` for workflows, and — for plugins — a bundle-reference `agent_plugin.yaml`/`.yml` manifest or a contained `.claude-plugin/plugin.json` manifest. Plugins are validated against their public contract; quality, lint, and version checks are skill-only and skipped for plugins.
+When auto-detection finds none of these, the type is `unknown`. Tier 1 then records the `schema`, `version`, and `quality` checks as not applicable (N/A) instead of failing them with `No skills found`, and still runs the content scans. Tier 1 passes when those scans pass; Tier 2 still runs on the target unless `--no-dedup` or `--tiers 1`. The JSON report adds a root `tier1_applicability` object, whichever checks the run selects. Auto-detection also reads the directory names in the target's absolute path, so a repository whose path includes a directory named `skills`, `team-skills`, `team-rules`, `workflows`, or `team-workflows` is detected as that type, not `unknown`. A target folder whose own `SKILL.md`, `workflow-rules.mdc`, `agent_plugin.yaml`/`.yml`, or `.claude-plugin/plugin.json` is mis-cased (such as `Skill.md`) or a dangling symlink still runs the manifest checks, and fails them. A mis-cased manifest below the target is not discovered; validate that folder directly. Pass `--type` to validate the target as a given type anyway. See [Content types](/skills/skillevaluator/tier1-validation#content-types).
+
`validate` also takes the standard `-r/--report` and `-o/--output-dir` options described under [Global conventions](#global-conventions).
### Tier 1 · Static & Security
diff --git a/docs/reports.mdx b/docs/reports.mdx
index 01756550..19f9772d 100644
--- a/docs/reports.mdx
+++ b/docs/reports.mdx
@@ -1404,6 +1404,16 @@ For CI and tooling, two JSON entry points matter:
different sources abort the run before any report file is written, so a
contradictory identity is never published in one format and reported in
another.
+- **Tier 1 applicability**: when `validate --type auto` finds no skill,
+ rules, workflows, or plugin in the target, the JSON report carries a
+ top-level `tier1_applicability` object (`applicability: not_applicable`,
+ `reason_code: unsupported_content_type`, `detected_content_type: unknown`,
+ `supported_content_types`), whichever checks the run selected. Each of
+ `schema`, `version`, and `quality` that the run selected and did not
+ `--skip` is then `skipped`, with a `skip_reason` that starts with
+ `not applicable`. The key is absent when Tier 1 ran those checks, as it
+ does for a detected artifact, a forced `--type`, or a folder with a
+ mis-cased or dangling manifest such as `SKILL.md`.
Both paths speak the same dialect: standalone `tier3 evaluate` and
`validate --tier3` share one HTML renderer for `report.html`, and both embed
diff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdx
index 38cd55c2..04a5659a 100644
--- a/docs/tier1-validation.mdx
+++ b/docs/tier1-validation.mdx
@@ -120,6 +120,18 @@ Reports for any of these runs are selected with `-r`/`--report` (`cli`, `json`,
Plugins are validated against their public contract; the `quality`, `lint`, and `version` checks are skill-only and are skipped for plugins, rules, and workflows.
+When `--type auto` finds none of these signals, for example in an ordinary source repository, the target's type is `unknown`. There is no manifest to check, so Tier 1:
+
+- Records `schema`, `version`, and `quality` as not applicable instead of failing them with `No skills found`. The terminal summary and the HTML report count them as N/A. The JSON and Markdown reports list them as skipped, with a skip reason that starts with `not applicable`. No report counts them as passes.
+- Still runs the content scans (`security`, `pii`, `license`, `code-integrity`, `unicode`, `lint`, and `dependency` when selected), so a real secret or card number still fails the run. The security scan keeps its unrestricted scope.
+- Passes when those scans pass. A misspelled `--checks` name still fails. Tier 2 deduplication still runs on the target unless you pass `--no-dedup` or `--tiers 1`, and can still fail the run.
+- Adds a root `tier1_applicability` object to the JSON report, whichever checks `--checks` and `--skip` select: `applicability` (`not_applicable`), `reason_code` (`unsupported_content_type`), `detected_content_type` (`unknown`), and `supported_content_types`.
+- Is never eligible for publication, because a not-applicable check is not evidence.
+
+Auto-detection also reads the directory names in the target's absolute path. A repository whose path includes a directory named `skills`, `team-skills`, `team-rules`, `workflows`, or `team-workflows`, such as a CI checkout at `/home/runner/work/skills/skills`, is detected as that type, not as `unknown`, and Tier 1 validates it as that type, as before. `--type` forces a type but cannot select `unknown`.
+
+A target folder that itself holds a manifest entry in any letter case (`SKILL.md`, `workflow-rules.mdc`, `agent_plugin.yaml`, `agent_plugin.yml`, or `.claude-plugin/plugin.json`), such as `Skill.md`, or a manifest symlink whose target is missing, is a broken artifact rather than a folder without one, so its manifest checks still run and fail. A `.claude-plugin` folder without a `plugin.json`, such as a marketplace's, does not count. This applies only to the target's own entries: a mis-cased manifest below the target is not discovered; validate that folder directly. Pass `--type skill` (or another type) to validate any target as that type; a missing `SKILL.md` then fails as before. A folder that holds skills below it, such as `.claude/skills/<name>/SKILL.md` or `.agents/skills/<name>/SKILL.md`, is still validated as a collection of skills.
+
For collections, pass a folder containing skills under `skills/` or `team-skills/`. Folder-level validation discovers each live `SKILL.md` below the target while excluding configured names such as `evals`, `results`, `versions`, `.git`, `.venv`, `node_modules`, and `__pycache__`.
```bash title="Validate a skill or a collection"
diff --git a/src/skillevaluator/reporting/console_ui.py b/src/skillevaluator/reporting/console_ui.py
index eed1a27c..b3949646 100644
--- a/src/skillevaluator/reporting/console_ui.py
+++ b/src/skillevaluator/reporting/console_ui.py
@@ -922,6 +922,13 @@ def _is_user_skipped(result: ValidationResult) -> bool:
return isinstance(result.metadata, dict) and bool(result.metadata.get("skipped_by_user"))
+def _is_not_applicable(result: ValidationResult) -> bool:
+ """True when a check was recorded as not applicable to the target instead of run."""
+ return (
+ _is_skipped(result) and not _is_user_skipped(result) and "not applicable" in get_skip_reason(result).casefold()
+ )
+
+
def _is_skipped(result: ValidationResult) -> bool:
"""True when a validator was recorded without executing (OSS conventions)."""
if bool(getattr(result, "skipped", False)):
@@ -1130,6 +1137,11 @@ def summarize_tier1(results: list[ValidationResult], lineup: list[str] | None =
When *lineup* is given, the per-check ticker stays in the completed
section with each check's final state (failed checks in red).
"""
+ # A check recorded as not applicable did not run: it is neither a pass nor
+ # a failure, so it is listed on its own row and left out of the counts.
+ not_applicable = [r for r in results if _is_not_applicable(r)]
+ results = [r for r in results if not _is_not_applicable(r)]
+
user_skipped = [r for r in results if _is_user_skipped(r)]
total = len(results) - len(user_skipped)
failed = [r for r in results if not r.passed and not _is_skipped(r)]
@@ -1163,6 +1175,12 @@ def summarize_tier1(results: list[ValidationResult], lineup: list[str] | None =
if user_skipped:
names = ", ".join(dict.fromkeys(_short_name(r.validator_name) for r in user_skipped))
rows.append(TierRow("skipped", [(names, MUTED), (" with --skip, not checked", FAINT)]))
+ if not_applicable:
+ names = ", ".join(
+ dict.fromkeys(_check_key(r.validator_name) or _short_name(r.validator_name) for r in not_applicable)
+ )
+ reason = _bounded_console_line(get_skip_reason(not_applicable[0]))
+ rows += [TierRow("n/a", [(names, MUTED)]), TierRow("", [(reason, FAINT)], fit="wrap")]
if advisory:
detail = " · ".join(f"{name} {count}" for name, count in sorted(advisory.items(), key=lambda kv: -kv[1]))
diff --git a/src/skillevaluator/reporting/json_reporter.py b/src/skillevaluator/reporting/json_reporter.py
index d878c8e0..63bb338f 100644
--- a/src/skillevaluator/reporting/json_reporter.py
+++ b/src/skillevaluator/reporting/json_reporter.py
@@ -276,6 +276,21 @@ class JSONReporter(ReporterBase):
if rubric_results:
data["rubric_eval"] = rubric_results[0]
+ # Tier 1 applicability: ``--type auto`` found no supported artifact, so
+ # the manifest checks were recorded as not applicable. Every result of
+ # such a run carries the record, whatever --checks selected, so CI can
+ # read this instead of parsing skip reasons.
+ tier1_applicability = next(
+ (
+ result.metadata.get("tier1_applicability")
+ for result in results
+ if isinstance(result.metadata.get("tier1_applicability"), dict)
+ ),
+ None,
+ )
+ if tier1_applicability is not None:
+ data["tier1_applicability"] = tier1_applicability
+
# Tier 3: Agent evaluation summary
if emitted_agent_eval:
data["tier3"] = emitted_agent_eval
diff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2
index 8e408a8e..06a167f9 100644
--- a/src/skillevaluator/reporting/templates/report.html.j2
+++ b/src/skillevaluator/reporting/templates/report.html.j2
@@ -1506,7 +1506,7 @@
{% set contributors = tier1_contributors %}
{% set total_validators = ui_t1_executed %}
{% set passed_count = ui_t1_pass %}
- {% set skipped_count = tier1_display_summary.skipped_count %}
+ {% set skipped_count = ui_t1.get('skipped', tier1_display_summary.skipped_count) %}
{% set failed_count = ui_t1.get('fail', tier1_display_summary.failed_count) %}
{% set incomplete_count = ui_t1.get('incomplete', 0) %}
{% set total_issues = tier1_display_summary.total_issues %}
diff --git a/src/skillevaluator/tier1/commands.py b/src/skillevaluator/tier1/commands.py
index dca05fee..be82a8d3 100644
--- a/src/skillevaluator/tier1/commands.py
+++ b/src/skillevaluator/tier1/commands.py
@@ -5,8 +5,10 @@
from __future__ import annotations
+import os
import time
from collections.abc import Callable
+from copy import deepcopy
from pathlib import Path
from rich.console import Console
@@ -17,8 +19,13 @@ from skillevaluator.constants import (
CONTENT_TYPE_SKILL,
CONTENT_TYPE_UNKNOWN,
CONTENT_TYPE_WORKFLOWS,
+ PLUGIN_CONTAINED_MANIFEST_DIR,
+ PLUGIN_CONTAINED_MANIFEST_FILE,
+ PLUGIN_MANIFEST_FILES,
SCAN_EXCLUDED_DIRS,
SCAN_EXCLUDED_FILES,
+ SKILL_MANIFEST_FILE,
+ WORKFLOWS_MANIFEST_FILE,
)
from skillevaluator.models.result import ValidationResult
from skillevaluator.publication_evidence import (
@@ -112,6 +119,29 @@ _SKIPPED_VALIDATORS = {
"quality": ("Quality Score (4-Dimension Analysis)",),
"lint": ("Script Lint (Advisory)",),
}
+# Checks that read a skill manifest (SKILL.md frontmatter, its version, its
+# quality). A target that ``--type auto`` detects as ``unknown`` and that holds
+# no artifact manifest entry has no manifest, so these are recorded as not
+# applicable rather than run.
+_MANIFEST_CHECKS = ("schema", "version", "quality")
+# Root manifests that auto-detection looks for, compared in any letter case.
+# Detection matches them by exact name with ``exists()``, so a mis-cased name
+# or a dangling symlink leaves the target ``unknown``.
+_ROOT_MANIFEST_NAMES = frozenset(
+ name.casefold() for name in (SKILL_MANIFEST_FILE, WORKFLOWS_MANIFEST_FILE, *PLUGIN_MANIFEST_FILES)
+)
+UNKNOWN_CONTENT_SKIP_REASON = (
+ "not applicable: no supported skill, rules, workflows or plugin artifact detected; use --type to force"
+)
+# Machine-readable form of the applicability decision. Every result of such a
+# run carries it, and the JSON report emits it at the root as
+# ``tier1_applicability``.
+UNKNOWN_CONTENT_APPLICABILITY = {
+ "applicability": "not_applicable",
+ "reason_code": "unsupported_content_type",
+ "detected_content_type": CONTENT_TYPE_UNKNOWN,
+ "supported_content_types": [CONTENT_TYPE_SKILL, CONTENT_TYPE_RULES, CONTENT_TYPE_WORKFLOWS, CONTENT_TYPE_PLUGIN],
+}
REPORTERS = {
"json": JSONReporter,
"html": HTMLReporter,
@@ -127,6 +157,57 @@ def _stamp_tier1_scan_scope(results: list[ValidationResult], target: Path) -> No
result.metadata[TIER1_UNSCANNED_PATHS_KEY] = list(skipped)
+def _entry_names(folder: Path) -> list[str] | None:
+ """Return the entry names of *folder*, or ``None`` when it cannot be listed."""
+ try:
+ with os.scandir(folder) as entries:
+ return [entry.name for entry in entries]
+ except OSError:
+ return None
+
+
+def _has_manifest_entry(target: Path) -> bool:
+ """Return whether *target* is, or directly holds, an artifact manifest entry in any letter case.
+
+ The manifests are ``SKILL.md``, ``workflow-rules.mdc``,
+ ``agent_plugin.yaml``/``.yml`` and ``.claude-plugin/plugin.json``. Entry
+ names are compared instead of calling ``exists()``, so a dangling symlink
+ and a mis-cased name such as ``Skill.md`` count on every filesystem: such
+ a folder is a broken artifact, not a target without one. A directory that
+ cannot be listed also counts, and so does a ``.claude-plugin`` entry that
+ is not a listable directory, so the manifest checks run and report the
+ problem instead of being recorded as not applicable. A ``.claude-plugin``
+ folder without a ``plugin.json``, such as a marketplace's, does not count.
+ """
+ contained_dir = PLUGIN_CONTAINED_MANIFEST_DIR.casefold()
+ contained_file = PLUGIN_CONTAINED_MANIFEST_FILE.casefold()
+ name = target.name.casefold()
+ if os.path.lexists(target) and (
+ name in _ROOT_MANIFEST_NAMES or (name == contained_file and target.parent.name.casefold() == contained_dir)
+ ):
+ return True
+ if not target.is_dir():
+ return False
+ names = _entry_names(target)
+ if names is None:
+ return True
+ for entry in names:
+ folded = entry.casefold()
+ if folded in _ROOT_MANIFEST_NAMES:
+ return True
+ if folded == contained_dir:
+ contained = _entry_names(target / entry)
+ if contained is None or any(item.casefold() == contained_file for item in contained):
+ return True
+ return False
+
+
+def _stamp_tier1_applicability(results: list[ValidationResult]) -> None:
+ """Record on every result that the manifest checks do not apply to the target."""
+ for result in results:
+ result.metadata["tier1_applicability"] = deepcopy(UNKNOWN_CONTENT_APPLICABILITY)
+
+
def _as_result(
name: str,
description: str,
@@ -180,8 +261,10 @@ def _schema_validator_for(content_type: str | None, policy: ValidationPolicy | N
"""Return the schema validator matching the (forced or detected) content type.
Rules, workflows, and plugins use their dedicated schema validators; skill
- and unknown content fall back to the skill :class:`SchemaValidator` (the
- historical default).
+ content, ``None`` and an ``unknown`` folder that holds an artifact
+ manifest entry fall back to the skill :class:`SchemaValidator` (the
+ historical default). ``run_validation`` records the schema check as not
+ applicable for any other ``unknown`` target instead of calling this.
"""
if content_type == CONTENT_TYPE_RULES:
return RulesSchemaValidator()
@@ -233,7 +316,15 @@ def run_validation(
*content_type* (``skill`` | ``rules`` | ``workflows`` | ``plugin`` |
``unknown`` | ``None``) selects the schema validator and gates skill-only
checks (``quality`` and ``lint`` are skipped for rules, workflows, and
- plugins). When *fail_fast* is set, the
+ plugins). For ``unknown`` (``--type auto`` found no supported artifact) the
+ manifest checks ``schema``, ``version`` and ``quality`` are recorded as not
+ applicable instead of run, and every result carries the
+ ``tier1_applicability`` record, while the content scans still run. A target
+ that holds an artifact manifest entry (``SKILL.md``, ``workflow-rules.mdc``,
+ ``agent_plugin.yaml``/``.yml`` or ``.claude-plugin/plugin.json``) in any
+ letter case, including a dangling symlink, is a broken artifact, so its
+ manifest checks still run and fail.
+ ``None`` keeps the historical skill behavior. When *fail_fast* is set, the
run stops after the first failing check. *continue_on_failure* overrides
*fail_fast* and also keeps batch folder validation scanning every skill past
a CRITICAL finding (parity with SkillEvaluator ``--continue-on-failure``).
@@ -251,6 +342,7 @@ def run_validation(
results: list[ValidationResult] = []
initial_target = publication_target_from_path(target_path)
skill_like = content_type in (None, CONTENT_TYPE_SKILL, CONTENT_TYPE_UNKNOWN)
+ manifest_not_applicable = content_type == CONTENT_TYPE_UNKNOWN and not _has_manifest_entry(target_path)
def _run_result(
check_id: str,
@@ -278,6 +370,18 @@ def run_validation(
stamp_tier1_required_checks([result], skill_like=skill_like)
return result
+ def _not_applicable_result(check_id: str) -> ValidationResult:
+ # Recorded, never executed: a skipped result is not publication
+ # evidence, so a run with one of these can never certify Tier 1.
+ result = ValidationResult(
+ validator_name=_SKIPPED_VALIDATORS[check_id][0],
+ validator_description=UNKNOWN_CONTENT_SKIP_REASON,
+ metadata={"skipped": True, "skip_reason": UNKNOWN_CONTENT_SKIP_REASON},
+ )
+ stamp_publication_evidence([result], tier=1, check_id=check_id)
+ stamp_tier1_required_checks([result], skill_like=skill_like)
+ return result
+
def _schema_results() -> list[ValidationResult]:
v = _schema_validator_for(content_type, policy)
return [_run_result("schema", v.name, v.description, v.validate)]
@@ -354,6 +458,13 @@ def run_validation(
for check_name, _builder in [step for step in active if step[0] in skipped]:
active.remove((check_name, _builder))
results.extend(_skipped_result(check_name, name) for name in _SKIPPED_VALIDATORS[check_name])
+ if manifest_not_applicable:
+ # No skill, rules, workflows or plugin layout was detected, so there is
+ # no manifest to validate: each manifest check would only report "No
+ # skills found". The content scans below keep their unrestricted scope.
+ for check_name, _builder in [step for step in active if step[0] in _MANIFEST_CHECKS]:
+ active.remove((check_name, _builder))
+ results.append(_not_applicable_result(check_name))
with continue_on_failure_scope(continue_on_failure):
for step_number, (check_name, builder) in enumerate(active, 1):
if on_check is not None:
@@ -381,6 +492,8 @@ def run_validation(
)
if fail_fast and not continue_on_failure and any(not r.passed for r in results):
+ if manifest_not_applicable:
+ _stamp_tier1_applicability(results)
_stamp_tier1_scan_scope(results, target_path)
finalize_publication_target(results, target_path, initial_target)
return results
@@ -394,6 +507,11 @@ def run_validation(
result.add_error(f"Unknown Tier 1 check(s): {', '.join(sorted(unknown))}")
results.insert(0, result)
+ if manifest_not_applicable:
+ # Recorded whatever --checks/--skip selected, so CI can read the
+ # applicability of the target from any run, including one narrowed to
+ # the content scans.
+ _stamp_tier1_applicability(results)
_stamp_tier1_scan_scope(results, target_path)
finalize_publication_target(results, target_path, initial_target)
return results
diff --git a/tests/test_artifact_applicability.py b/tests/test_artifact_applicability.py
new file mode 100644
index 00000000..e070ee45
--- /dev/null
+++ b/tests/test_artifact_applicability.py
@@ -0,0 +1,534 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Tier 1 on a target with no supported artifact (``--type auto`` detects ``unknown``).
+
+The skill-manifest checks (schema, version, quality) are recorded as not
+applicable instead of failing with "No skills found"; the content scans still
+run with their unrestricted scope, and such a run never certifies Tier 1. A
+folder that holds an artifact manifest entry (``SKILL.md``,
+``workflow-rules.mdc``, ``agent_plugin.yaml``/``.yml`` or
+``.claude-plugin/plugin.json``) in any letter case, or a dangling symlink by
+one of those names, is a broken artifact and still fails the manifest checks.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from pathlib import Path
+
+import pytest
+from click.testing import CliRunner
+
+from skillevaluator.cli import cli
+from skillevaluator.cli_core import detect_content_type
+from skillevaluator.constants import CONTENT_TYPE_UNKNOWN
+from skillevaluator.models.result import ValidationResult
+from skillevaluator.reporting import HTMLReporter, JSONReporter, MarkdownReporter
+from skillevaluator.reporting.base import assess_publication, is_cleanly_skipped, missing_tier1_publication_checks
+from skillevaluator.reporting.console_ui import summarize_tier1
+from skillevaluator.reporting.presentation import build_final_render_context
+from skillevaluator.tier1 import commands as tier1_commands
+from skillevaluator.tier1.commands import run_validation
+
+MANIFEST_VALIDATORS = {
+ "schema": "Schema & Repository Governance",
+ "version": "Semantic Version Validation",
+ "quality": "Quality Score (4-Dimension Analysis)",
+}
+APPLICABILITY = {
+ "applicability": "not_applicable",
+ "reason_code": "unsupported_content_type",
+ "detected_content_type": "unknown",
+ "supported_content_types": ["skill", "rules", "workflows", "plugin"],
+}
+
+
+def _ordinary_repository(root: Path) -> Path:
+ root.mkdir(parents=True, exist_ok=True)
+ (root / "README.md").write_text("# Example service\n", encoding="utf-8")
+ (root / "main.py").write_text("print('hello')\n", encoding="utf-8")
+ docs = root / "docs"
+ docs.mkdir()
+ (docs / "guide.md").write_text("# Guide\n", encoding="utf-8")
+ workflow = root / ".github" / "workflows"
+ workflow.mkdir(parents=True)
+ (workflow / "ci.yml").write_text("name: CI\n", encoding="utf-8")
+ assert detect_content_type(root) == CONTENT_TYPE_UNKNOWN
+ return root
+
+
+def _by_name(results: list[ValidationResult]) -> dict[str, ValidationResult]:
+ return {result.validator_name: result for result in results}
+
+
+def _rendered(rows) -> str:
+ return "\n".join("".join(text for text, _style in row.segments) for row in rows)
+
+
+def _row_text(row) -> str:
+ return "".join(text for text, _style in row.segments)
+
+
+def _broken_skill(root: Path) -> Path:
+ """A folder that is meant to be an artifact but whose manifest the checks cannot read."""
+ root.mkdir(parents=True)
+ (root / "README.md").write_text("# Example tool\n", encoding="utf-8")
+ return root
+
+
+def _validate(*args: str) -> tuple[int, str]:
+ result = CliRunner().invoke(cli, ["validate", *args, "--no-llm"])
+ return result.exit_code, result.output
+
+
+def _json_report(output_dir: Path) -> dict:
+ return json.loads(next(output_dir.glob("*.json")).read_text(encoding="utf-8"))
+
+
+# The code-integrity validators run Bandit and Gitleaks, which a unit test
+# cannot rely on: CI installs no Gitleaks, and the managed install would
+# download it. Tests that select code-integrity replace them with this stub.
+EXTERNAL_SCANNER_VALIDATORS = ("CodeRiskValidator", "SecretsValidator", "HygieneValidator")
+
+
+class _PassingValidator:
+ name = "Built-in validator"
+ description = "Built-in validation"
+
+ def __init__(self, **_kwargs: object) -> None:
+ pass
+
+ @staticmethod
+ def validate(_target: Path) -> ValidationResult:
+ result = ValidationResult(validator_name="Built-in validator")
+ result.add_success("built_in", "Built-in validation completed")
+ return result
+
+ validate_security_only = validate
+ validate_pii_only = validate
+
+
+def _stub_validators(monkeypatch: pytest.MonkeyPatch, *validator_names: str) -> None:
+ for validator_name in validator_names:
+ monkeypatch.setattr(tier1_commands, validator_name, _PassingValidator)
+
+
+def test_unknown_content_records_manifest_checks_as_not_applicable(tmp_path: Path) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+ ran: list[str] = []
+
+ results = run_validation(
+ repository,
+ checks="schema,version,quality,pii,unicode,lint",
+ content_type=CONTENT_TYPE_UNKNOWN,
+ on_check=ran.append,
+ )
+
+ assert ran == ["pii", "unicode", "lint"]
+ by_name = _by_name(results)
+ for check_id, name in MANIFEST_VALIDATORS.items():
+ result = by_name[name]
+ assert is_cleanly_skipped(result)
+ assert "not applicable" in result.metadata["skip_reason"]
+ assert "--type" in result.metadata["skip_reason"]
+ assert result.metadata["tier1_applicability"] == APPLICABILITY
+ assert result.metadata["publication_evidence"]["check_id"] == check_id
+ assert not result.metadata.get("skipped_by_user")
+ for name in ("PII Scan", "Unicode Smuggling Detection"):
+ assert by_name[name].passed
+ assert not by_name[name].metadata.get("skipped")
+ assert not any("No skills found" in error for result in results for error in result.errors)
+
+
+def test_unknown_content_keeps_the_unrestricted_security_scan(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+ scopes: list[str | None] = []
+
+ def fake_security_scan(self, _target: Path) -> ValidationResult:
+ scopes.append(self.content_type)
+ result = ValidationResult(validator_name="Security Scan")
+ result.add_success("skillspector", "No security issues found")
+ return result
+
+ monkeypatch.setattr(tier1_commands.SecurityValidator, "validate_security_only", fake_security_scan)
+
+ results = run_validation(repository, checks="schema,security", content_type=CONTENT_TYPE_UNKNOWN)
+
+ assert scopes == [None]
+ by_name = _by_name(results)
+ assert by_name["Security Scan"].passed
+ assert not by_name["Security Scan"].metadata.get("skipped")
+ assert is_cleanly_skipped(by_name["Schema & Repository Governance"])
+
+
+def test_unknown_content_still_fails_an_unknown_check_name(tmp_path: Path) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+
+ results = run_validation(repository, checks="schema,typo", content_type=CONTENT_TYPE_UNKNOWN)
+
+ assert not results[0].passed
+ assert "Unknown Tier 1 check(s): typo" in results[0].errors
+ assert is_cleanly_skipped(_by_name(results)["Schema & Repository Governance"])
+
+
+def test_no_content_type_keeps_the_historical_skill_validation(tmp_path: Path) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+
+ results = run_validation(repository, checks="schema", content_type=None)
+
+ schema = _by_name(results)["Schema & Repository Governance"]
+ assert not schema.passed
+ assert any("No skills found" in error for error in schema.errors)
+
+
+def test_unknown_content_with_a_mis_cased_manifest_still_fails_the_manifest_checks(tmp_path: Path) -> None:
+ # Case-sensitive filesystems detect this folder as unknown; spec-compliant
+ # agents do not load a Skill.md, so it must fail like any broken skill.
+ folder = _broken_skill(tmp_path / "my-tool")
+ (folder / "Skill.md").write_text(
+ "---\nname: my-tool\ndescription: A tool whose manifest name has the wrong letter case.\n---\n\n# My tool\n",
+ encoding="utf-8",
+ )
+
+ results = run_validation(
+ folder, checks="schema,quality,pii", content_type=CONTENT_TYPE_UNKNOWN, on_check=lambda _name: None
+ )
+
+ by_check = {result.metadata["publication_evidence"]["check_id"]: result for result in results}
+ assert sorted(by_check) == ["pii", "quality", "schema"]
+ for check_id in ("schema", "quality"):
+ assert not by_check[check_id].metadata.get("skipped")
+ assert not by_check[check_id].passed
+ assert not any("tier1_applicability" in result.metadata for result in results)
+ data = json.loads(JSONReporter(include_timestamp=False).render_all(results))
+ assert "tier1_applicability" not in data
+ assert data["overall_status"] == "failed"
+
+
+def _dangling_symlink(link: Path) -> None:
+ link.parent.mkdir(parents=True, exist_ok=True)
+ try:
+ link.symlink_to("missing-target")
+ except (OSError, NotImplementedError) as exc:
+ pytest.skip(f"symlinks unavailable: {exc}")
+
+
+def _assert_manifest_check_failed(results: list[ValidationResult]) -> None:
+ schema = _by_name(results)["Schema & Repository Governance"]
+ assert not schema.metadata.get("skipped")
+ assert not schema.passed
+ assert not any("tier1_applicability" in result.metadata for result in results)
+
+
+# Every manifest auto-detection looks for. A dangling symlink by one of these
+# names is not detected, so the target is ``unknown``, but it is a broken
+# artifact rather than a target without one.
+DANGLING_MANIFESTS = [
+ "SKILL.md",
+ "workflow-rules.mdc",
+ "agent_plugin.yaml",
+ "agent_plugin.yml",
+ ".claude-plugin/plugin.json",
+ ".claude-plugin",
+]
+
+
[email protected]("manifest", DANGLING_MANIFESTS)
+def test_unknown_content_with_a_dangling_manifest_symlink_still_fails_schema(tmp_path: Path, manifest: str) -> None:
+ folder = _broken_skill(tmp_path / "my-tool")
+ _dangling_symlink(folder / manifest)
+
+ results = run_validation(
+ folder, checks="schema,quality,pii", content_type=CONTENT_TYPE_UNKNOWN, on_check=lambda _name: None
+ )
+
+ _assert_manifest_check_failed(results)
+
+
[email protected]("manifest", ["Workflow-Rules.mdc", "Agent_Plugin.YAML", ".Claude-Plugin/Plugin.JSON"])
+def test_unknown_content_with_a_mis_cased_artifact_manifest_still_fails_schema(tmp_path: Path, manifest: str) -> None:
+ # Auto-detection matches these names exactly, so case-sensitive
+ # filesystems report such a folder as unknown.
+ folder = _broken_skill(tmp_path / "my-tool")
+ path = folder / manifest
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text("name: my-tool\n", encoding="utf-8")
+
+ results = run_validation(
+ folder, checks="schema,quality,pii", content_type=CONTENT_TYPE_UNKNOWN, on_check=lambda _name: None
+ )
+
+ _assert_manifest_check_failed(results)
+
+
+def test_unknown_content_with_only_a_marketplace_manifest_is_not_applicable(tmp_path: Path) -> None:
+ # A marketplace repository lists plugins; its .claude-plugin folder holds
+ # no plugin.json, so it is not a broken plugin.
+ repository = _ordinary_repository(tmp_path / "repository")
+ marketplace = repository / ".claude-plugin" / "marketplace.json"
+ marketplace.parent.mkdir()
+ marketplace.write_text('{"name": "example", "plugins": []}\n', encoding="utf-8")
+
+ results = run_validation(
+ repository, checks="schema,pii", content_type=CONTENT_TYPE_UNKNOWN, on_check=lambda _name: None
+ )
+
+ assert is_cleanly_skipped(_by_name(results)["Schema & Repository Governance"])
+ assert all(result.metadata["tier1_applicability"] == APPLICABILITY for result in results)
+
+
[email protected]("manifest", ["SKILL.md", ".claude-plugin/plugin.json"])
+def test_validate_folder_with_a_dangling_manifest_symlink_still_fails(tmp_path: Path, manifest: str) -> None:
+ folder = _broken_skill(tmp_path / "my-tool")
+ _dangling_symlink(folder / manifest)
+ reports = tmp_path / "reports"
+
+ exit_code, output = _validate(
+ str(folder), "--tiers", "1", "--checks", "schema,quality,pii", "-r", "json", "--output-dir", str(reports)
+ )
+
+ assert exit_code == 1, output
+ payload = _json_report(reports)
+ assert payload["overall_status"] == "failed"
+ assert "tier1_applicability" not in payload
+ schema = next(entry for entry in payload["results"] if entry["validator"] == "Schema & Repository Governance")
+ assert schema["status"] == "failed"
+
+
+ ("checks", "skip"),
+ [("pii", None), ("security,pii,code-integrity", None), ("schema,version,quality,pii", "schema,version,quality")],
+)
+def test_unknown_content_records_applicability_for_any_check_selection(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, checks: str, skip: str | None
+) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+ # Only the PII scan runs for real: the security and code-integrity scans
+ # need SkillSpector, Bandit and Gitleaks, which CI does not provide.
+ _stub_validators(monkeypatch, "SecurityValidator", *EXTERNAL_SCANNER_VALIDATORS)
+
+ results = run_validation(
+ repository, checks=checks, skip=skip, content_type=CONTENT_TYPE_UNKNOWN, on_check=lambda _name: None
+ )
+
+ assert all(result.metadata["tier1_applicability"] == APPLICABILITY for result in results)
+ data = json.loads(JSONReporter(include_timestamp=False).render_all(results))
+ assert data["tier1_applicability"] == APPLICABILITY
+ assert data["overall_status"] == "passed"
+ assert data["publication"]["eligible"] is False
+
+
+def test_unknown_content_records_applicability_when_fail_fast_stops_early(tmp_path: Path) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+ (repository / "README.md").write_text("# Example service\n\nBilling: 4532-0151-1283-0366\n", encoding="utf-8")
+ ran: list[str] = []
+
+ results = run_validation(
+ repository,
+ checks="schema,pii,unicode",
+ content_type=CONTENT_TYPE_UNKNOWN,
+ fail_fast=True,
+ on_check=ran.append,
+ )
+
+ assert ran == ["pii"]
+ assert not _by_name(results)["PII Scan"].passed
+ assert all(result.metadata["tier1_applicability"] == APPLICABILITY for result in results)
+
+
+def test_validate_ordinary_repository_narrowed_to_content_scans_records_applicability(tmp_path: Path) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+ reports = tmp_path / "reports"
+
+ exit_code, output = _validate(
+ str(repository), "--tiers", "1", "--checks", "pii", "-r", "json", "--output-dir", str(reports)
+ )
+
+ assert exit_code == 0, output
+ payload = _json_report(reports)
+ assert payload["tier1_applicability"] == APPLICABILITY
+ assert [entry["validator"] for entry in payload["results"]] == ["PII Scan"]
+
+
+def test_not_applicable_results_never_certify_publication(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+
+ _stub_validators(
+ monkeypatch,
+ "SecurityValidator",
+ *EXTERNAL_SCANNER_VALIDATORS,
+ "UnicodeSmuggleValidator",
+ "ScriptLintValidator",
+ "LicenseValidator",
+ )
+
+ results = run_validation(repository, content_type=CONTENT_TYPE_UNKNOWN)
+
+ assert all(result.passed for result in results)
+ assert missing_tier1_publication_checks(results) == ("quality", "schema", "version")
+ assert assess_publication(results, None).status == "incomplete"
+
+
+def test_tier1_summary_counts_not_applicable_checks_apart(tmp_path: Path) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+ lineup = ["schema", "version", "pii", "quality"]
+ results = run_validation(repository, checks=",".join(lineup), content_type=CONTENT_TYPE_UNKNOWN)
+
+ passed, rows = summarize_tier1(results, lineup=lineup)
+ text = _rendered(rows)
+
+ assert passed is True
+ # "1/1 passed" today; "1/1 checks passed" once the per-check count wording
+ # changes. Counting the N/A checks would give 4/4 in either wording.
+ assert re.search(r"\b1/1 (checks )?passed\b", text), text
+ assert not re.search(r"\b4/4 (checks )?passed\b", text), text
+ assert not re.search(r"\b\d+/4 (checks )?passed\b", text), text
+ # The N/A checks are named on the n/a row only: never marked passed or
+ # failed in the check ticker, and absent from every other row.
+ not_applicable_rows = [row for row in rows if row.label == "n/a"]
+ assert [_row_text(row) for row in not_applicable_rows] == ["schema, version, quality"]
+ ticker = next(_row_text(row) for row in rows if row.label == "checks")
+ assert "✓ pii" in ticker
+ for check_id in MANIFEST_VALIDATORS:
+ assert f"✓ {check_id}" not in ticker
+ assert f"✗ {check_id}" not in ticker
+ assert not any(check_id in _row_text(row) for row in rows if row.label not in {"checks", "n/a"})
+ assert "not applicable" in text
+
+ tier = build_final_render_context(results).tiers[1]
+ assert (tier.passed, tier.failed, tier.skipped, tier.not_applicable) == (1, 0, 0, 3)
+ assert tier.summary == "1 of 1 checks passed; 0 failed, 0 incomplete, 0 skipped, 3 N/A"
+
+
+def test_reports_show_only_manifest_checks_as_not_applicable(tmp_path: Path) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+ results = run_validation(repository, checks="schema,quality", content_type=CONTENT_TYPE_UNKNOWN)
+
+ data = json.loads(JSONReporter(include_timestamp=False).render_all(results))
+ assert data["tier1_applicability"] == APPLICABILITY
+ assert data["overall_status"] == "passed"
+ assert data["publication"]["eligible"] is False
+ assert [entry["status"] for entry in data["results"]] == ["skipped", "skipped"]
+
+ html = HTMLReporter(include_timestamp=False).render_all(results)
+ body = re.sub(r"<(style|script)\b.*?</\1>", "", html, flags=re.DOTALL)
+ assert "0 of 2 checks ran, 2 not applicable" in body
+ assert "2 skipped" not in body
+ assert "verdict-chip fail" not in body
+ assert not re.search(r"\b0(\.0)?%", body)
+
+ markdown = MarkdownReporter(include_timestamp=False).render_all(results)
+ assert "| Validators Run | 0 |" in markdown
+ assert "| ✅ Passed | 0 |" in markdown
+ assert "FAILED" not in markdown
+ assert "not applicable" in markdown
+
+
+def test_json_report_omits_tier1_applicability_for_a_detected_artifact() -> None:
+ result = ValidationResult(validator_name="PII Scan")
+ result.add_success("pii", "No PII found")
+
+ data = json.loads(JSONReporter(include_timestamp=False).render_all([result]))
+
+ assert "tier1_applicability" not in data
+
+
+def test_validate_ordinary_repository_passes_and_records_applicability(tmp_path: Path) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+ reports = tmp_path / "reports"
+
+ exit_code, output = _validate(
+ str(repository), "--tiers", "1", "--checks", "schema,quality,pii", "-r", "json", "--output-dir", str(reports)
+ )
+
+ assert exit_code == 0, output
+ assert "No skills found" not in output
+ payload = _json_report(reports)
+ assert payload["overall_status"] == "passed"
+ assert payload["tier1_applicability"] == APPLICABILITY
+ assert payload["publication"]["eligible"] is False
+ statuses = {entry["validator"]: (entry["status"], entry["skip_reason"]) for entry in payload["results"]}
+ assert statuses["PII Scan"] == ("passed", None)
+ for name in ("Schema & Repository Governance", "Quality Score (4-Dimension Analysis)"):
+ status, reason = statuses[name]
+ assert status == "skipped"
+ assert reason.startswith("not applicable")
+
+
+def test_validate_default_tiers_on_ordinary_repository_report_n_a(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+ # The assertions read the final panel, which wraps at the terminal width.
+ monkeypatch.setenv("COLUMNS", "120")
+
+ exit_code, output = _validate(
+ str(repository), "--no-dedup", "--checks", "schema,version,pii", "--output-dir", str(tmp_path / "reports")
+ )
+
+ assert exit_code == 0, output
+ assert "Tier 1: 1 of 1 checks passed; 0 failed, 0 incomplete, 0 skipped, 2 N/A" in output
+ assert "Publication eligibility: Incomplete" in output
+ assert "No skills found" not in output
+
+
+def test_validate_ordinary_repository_with_a_card_number_still_fails(tmp_path: Path) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+ (repository / "README.md").write_text("# Example service\n\nBilling: 4532-0151-1283-0366\n", encoding="utf-8")
+ reports = tmp_path / "reports"
+
+ exit_code, output = _validate(
+ str(repository), "--tiers", "1", "--checks", "schema,pii", "-r", "json", "--output-dir", str(reports)
+ )
+
+ assert exit_code == 1, output
+ payload = _json_report(reports)
+ assert payload["overall_status"] == "failed"
+ pii = next(entry for entry in payload["results"] if entry["validator"] == "PII Scan")
+ assert pii["status"] == "failed"
+ assert any(finding["check_name"] == "credit_cards" for finding in pii["findings"])
+
+
+def test_explicit_skill_type_still_reports_no_skills_found(tmp_path: Path) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+
+ exit_code, output = _validate(
+ str(repository),
+ "--type",
+ "skill",
+ "--tiers",
+ "1",
+ "--checks",
+ "schema,quality",
+ "--output-dir",
+ str(tmp_path / "reports"),
+ )
+
+ assert exit_code == 1
+ assert "No skills found" in output
+
+
+def test_agent_config_skill_roots_still_validate_as_a_catalog(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ repository = _ordinary_repository(tmp_path / "repository")
+ skill_dirs = []
+ for root in (".agents", ".claude"):
+ skill = repository / root / "skills" / "sample"
+ skill.mkdir(parents=True)
+ (skill / "SKILL.md").write_text(
+ "---\nname: sample\ndescription: A sample skill kept under an agent config directory.\n---\n\n# Sample\n",
+ encoding="utf-8",
+ )
+ skill_dirs.append(skill)
+ catalogs: list[list[Path]] = []
+
+ def fake_catalog(_ctx, *, skill_dirs: list[Path], **_kwargs: object) -> None:
+ catalogs.append(sorted(skill_dirs))
+
+ monkeypatch.setattr("skillevaluator.cli._validate_catalog", fake_catalog)
+
+ exit_code, output = _validate(str(repository), "--tiers", "1", "--output-dir", str(tmp_path / "reports"))
+
+ assert exit_code == 0, output
+ assert [[path.resolve() for path in catalog] for catalog in catalogs] == [sorted(p.resolve() for p in skill_dirs)]