Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Fix Tier 2 discovery for contained compatibility alias
Christopher Kevin2a3ace49c551
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e6c80918..91c36285 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -94,3 +94,8 @@ All notable changes to SkillEvaluator are documented in this file.- Tier 2 content collection now prunes configured evaluation and versionartifact directories before enforcing the discovered-path limit, so excludedgenerated results cannot cause false path-count failures.+- Tier 2 scans now validate but do not follow the exact contained+ `CLAUDE.md -> AGENTS.md` compatibility alias, scanning the exactly named,+ independently discovered, single-link regular target once while continuing+ to reject hard-linked selected files, linked manifests, directories, and all+ other file redirects.diff --git a/docs/tier2-deduplication.mdx b/docs/tier2-deduplication.mdxindex 6a9fdb0b..9ee70ac7 100644--- a/docs/tier2-deduplication.mdx+++ b/docs/tier2-deduplication.mdx@@ -218,8 +218,12 @@ behavior is identical in every output mode, so these commands work as CI gates| High-confidence duplicate boundary | `0.70` | Maps actionable duplicate verdicts to HIGH rather than MEDIUM |Tier 2 treats skill content and catalog files as untrusted input. Before any-provider call, it rejects linked roots, linked or escaping files, non-regular-files, and paths outside the verified scan root. Explicit limits bound+provider call, it rejects linked roots, linked directories and manifests,+hard-linked selected files, non-regular files, paths outside the verified scan+root, and all file redirects except the exact contained+`CLAUDE.md -> AGENTS.md` compatibility alias. That alias is validated but never+followed; its exactly named, independently discovered, single-link regular+sibling target is scanned once. Explicit limits bounddiscovered paths, file counts, per-file and total input bytes, content chunks,candidate clusters, catalog size and entry count, vector length, returnedmatches, and pairwise scalar work. Catalog loading also rejects duplicate JSON@@ -255,6 +259,7 @@ handling terms before sending confidential skill content to a hosted endpoint.| Markdown has no headings | Treats the body as a preamble and attempts paragraph splitting when oversized; retains the original section if no eligible split is produced || Unsupported extension | Skips the file || Duplicate sections occur in one file | Reports the relevant headings and line ranges |+ | Skill contains an exactly named, single-link regular `AGENTS.md` plus `CLAUDE.md -> AGENTS.md` | Validates and skips the alias, then scans `AGENTS.md` once || Skill carries large generated snapshots (`evals`, `.evals`, `results`, `.results`, `versions`, `.versions`) | Excluded directories are pruned during traversal, so their contents never count toward the discovered-path limit — a well-used skill with large evals or results snapshots no longer trips it, as it could in earlier releases || A directory is a Windows junction or other reparse point | Rejected during the walk, like symbolic links || Embedding or chat-LLM analysis is incomplete | Reports an error; does not claim a clean result |@@ -266,6 +271,7 @@ handling terms before sending confidential skill content to a hosted endpoint.| Catalog source collection is empty | Refuses to save an empty catalog || Catalog path is missing | Returns an error and does not rebuild it silently || Candidate lacks a root `SKILL.md` | Rejects the catalog query |+ | A skill contains an exactly named, single-link regular `AGENTS.md` plus `CLAUDE.md -> AGENTS.md` | Validates and skips the unrelated compatibility alias while discovering `SKILL.md` || Candidate matches multiple catalog entries | Reports matches in descending similarity order || Provider, endpoint, model, mode, or vector length differs | Rejects the catalog; query with matching settings or rebuild it || Catalog is malformed, non-finite, or oversized | Rejects it before comparison |diff --git a/src/skillevaluator/deduplication/utils/skill_collector.py b/src/skillevaluator/deduplication/utils/skill_collector.pyindex 778aff42..56c6cde4 100644--- a/src/skillevaluator/deduplication/utils/skill_collector.py+++ b/src/skillevaluator/deduplication/utils/skill_collector.py@@ -28,6 +28,7 @@ from skillevaluator.constants import (CONTENT_DEDUP_MAX_TOTAL_BYTES,CONTENT_DEDUP_SCANNABLE_EXTENSIONS,)+from skillevaluator.utils.tier2_paths import is_contained_compatibility_aliasfrom skillevaluator.validators.frontmatter_parser import FRONTMATTER_PATTERNlogger = logging.getLogger(__name__)@@ -157,6 +158,8 @@ class _SecureRoot:opened_info = os.fstat(fd)if not stat.S_ISREG(opened_info.st_mode):raise _SecureReadError("unsafe_path", f"Refusing non-regular file: {relative_path.as_posix()}")+ if getattr(opened_info, "st_nlink", 1) != 1:+ raise _SecureReadError("unsafe_path", f"Refusing hard-linked file: {relative_path.as_posix()}")if opened_info.st_size > max_bytes:raise _SecureReadError("file_size_limit",@@ -493,6 +496,9 @@ def _collect_files_anchored(},)discovered_paths.sort()+ discovered_names_by_parent: dict[Path, set[str]] = {}+ for discovered_path in discovered_paths:+ discovered_names_by_parent.setdefault(discovered_path.parent, set()).add(discovered_path.name)for file_path in discovered_paths:try:@@ -512,6 +518,12 @@ def _collect_files_anchored(continueif _is_link_or_reparse(file_path, rel_path):+ if is_contained_compatibility_alias(+ file_path,+ sibling_names=discovered_names_by_parent.get(file_path.parent, frozenset()),+ ):+ logger.debug("Skipping compatibility alias in favor of regular target: %s", rel_path)+ continueraise SkillCollectionError("unsafe_path",f"Refusing symbolic link or reparse point: {rel_path}",diff --git a/src/skillevaluator/embedding/extractor.py b/src/skillevaluator/embedding/extractor.pyindex c0bfd8e9..4712703d 100644--- a/src/skillevaluator/embedding/extractor.py+++ b/src/skillevaluator/embedding/extractor.py@@ -32,7 +32,7 @@ from skillevaluator.constants import ()from skillevaluator.deduplication.utils.skill_collector import _SecureReadError, _SecureRootfrom skillevaluator.logging_config import get_logger-from skillevaluator.utils.tier2_paths import safe_path_label+from skillevaluator.utils.tier2_paths import is_contained_compatibility_alias, safe_path_labelfrom skillevaluator.validators.frontmatter_parser import FRONTMATTER_PATTERNlogger = get_logger(__name__)@@ -343,12 +343,19 @@ def _iter_discovery_files(root: Path):kept_dirs.append(dirname)dirnames[:] = kept_dirs+ sibling_names = frozenset(filenames)for filename in filenames:discovered_paths += 1if discovered_paths > MAX_DISCOVERED_PATHS:raise ValueError(f"Collection path limit exceeded ({MAX_DISCOVERED_PATHS}) before embedding")file_path = Path(dirpath) / filenameif _is_symlink_or_reparse(file_path):+ if is_contained_compatibility_alias(file_path, sibling_names=sibling_names):+ logger.debug(+ "Skipping compatibility alias in favor of regular target: %s",+ file_path.relative_to(root).as_posix(),+ )+ continueraise ValueError(f"Discovery path contains a symlink or reparse point: {filename}")yield file_pathdiff --git a/src/skillevaluator/utils/tier2_paths.py b/src/skillevaluator/utils/tier2_paths.pyindex 7a0f554c..3a31a0f9 100644--- a/src/skillevaluator/utils/tier2_paths.py+++ b/src/skillevaluator/utils/tier2_paths.py@@ -7,6 +7,7 @@ from __future__ import annotationsimport osimport stat+from collections.abc import Collectionfrom contextlib import suppressfrom pathlib import Pathfrom typing import TYPE_CHECKING@@ -35,6 +36,34 @@ def is_link_or_reparse(path: Path) -> bool:)+def is_contained_compatibility_alias(path: Path, *, sibling_names: Collection[str]) -> bool:+ """Recognize only ``CLAUDE.md -> AGENTS.md`` with a regular sibling target.++ The alias is validated but never followed. Callers can skip it and process+ the independently discovered ``AGENTS.md`` target once. ``sibling_names``+ must come from the same traversal snapshot that produced ``path``.+ """+ if path.name != "CLAUDE.md" or "AGENTS.md" not in sibling_names:+ return False+ try:+ metadata = path.lstat()+ target_text = os.readlink(path) # noqa: PTH115 - the raw target must match exactly+ target_metadata = path.with_name("AGENTS.md").lstat()+ except OSError:+ return False++ reparse_flag = getattr(stat, "FILE_ATTRIBUTE_REPARSE_POINT", 0x400)+ target_attributes = getattr(target_metadata, "st_file_attributes", 0)+ return bool(+ stat.S_ISLNK(metadata.st_mode)+ and getattr(metadata, "st_nlink", 1) == 1+ and target_text == "AGENTS.md"+ and stat.S_ISREG(target_metadata.st_mode)+ and getattr(target_metadata, "st_nlink", 1) == 1+ and not target_attributes & reparse_flag+ )++def paths_refer_to_same_location(left: Path, right: Path) -> bool:"""Compare output paths after resolving aliases, symlinks, and hard links."""with suppress(OSError):@@ -114,6 +143,7 @@ def sanitize_tier2_results(results: list[ValidationResult], *paths: Path | None)__all__ = [+ "is_contained_compatibility_alias","is_link_or_reparse","paths_refer_to_same_location","safe_path_label",diff --git a/tests/deduplication/utils/test_skill_collector.py b/tests/deduplication/utils/test_skill_collector.pyindex d98833f0..2ad7a5a5 100644--- a/tests/deduplication/utils/test_skill_collector.py+++ b/tests/deduplication/utils/test_skill_collector.py@@ -118,6 +118,100 @@ class TestCollectFiles:assert len(result) == 1assert result[0].extension == ".mdc"+ def test_openclaw_compatibility_alias_is_skipped_and_regular_target_is_collected_once(+ self,+ skill_root: Path,+ ) -> None:+ agents = skill_root / "AGENTS.md"+ agents.write_text("# Shared agent instructions\n")+ try:+ (skill_root / "CLAUDE.md").symlink_to(agents.name)+ except OSError as exc:+ pytest.skip(f"symlinks unavailable: {exc}")++ result = collect_files(skill_root)++ assert [item.rel_path for item in result] == ["AGENTS.md"]++ def test_rejects_openclaw_alias_when_regular_target_is_hard_linked(self, skill_root: Path) -> None:+ outside_target = skill_root.parent / "outside-agents.md"+ outside_target.write_text("# Outside agent instructions\n")+ try:+ os.link(outside_target, skill_root / "AGENTS.md")+ (skill_root / "CLAUDE.md").symlink_to("AGENTS.md")+ except OSError as exc:+ pytest.skip(f"hard links or symlinks unavailable: {exc}")++ with pytest.raises(SkillCollectionError) as exc_info:+ collect_files(skill_root)++ assert exc_info.value.check_name == "unsafe_path"+ assert exc_info.value.rel_path == "CLAUDE.md"++ def test_rejects_hard_linked_selected_file(self, skill_root: Path) -> None:+ outside_target = skill_root.parent / "outside-notes.md"+ outside_target.write_text("# Outside notes\n")+ try:+ os.link(outside_target, skill_root / "notes.md")+ except OSError as exc:+ pytest.skip(f"hard links unavailable: {exc}")++ with pytest.raises(SkillCollectionError) as exc_info:+ collect_files(skill_root)++ assert exc_info.value.check_name == "unsafe_path"+ assert exc_info.value.rel_path == "notes.md"++ def test_rejects_openclaw_alias_when_target_entry_case_does_not_match(self, skill_root: Path) -> None:+ (skill_root / "agents.md").write_text("# Lowercase agent instructions\n")+ try:+ (skill_root / "CLAUDE.md").symlink_to("AGENTS.md")+ except OSError as exc:+ pytest.skip(f"symlinks unavailable: {exc}")++ with pytest.raises(SkillCollectionError) as exc_info:+ collect_files(skill_root)++ assert exc_info.value.check_name == "unsafe_path"+ assert exc_info.value.rel_path == "CLAUDE.md"++ def test_rejects_alias_target_created_after_discovery_snapshot(self, skill_root: Path, monkeypatch) -> None:+ try:+ (skill_root / "CLAUDE.md").symlink_to("AGENTS.md")+ except OSError as exc:+ pytest.skip(f"symlinks unavailable: {exc}")++ real_walk = os.walk++ def walk_with_late_target(*args, **kwargs):+ for dirpath, dirnames, filenames in real_walk(*args, **kwargs):+ if Path(dirpath) == skill_root:+ assert "AGENTS.md" not in filenames+ (skill_root / "AGENTS.md").write_text("# Late agent instructions\n")+ yield dirpath, dirnames, filenames++ monkeypatch.setattr(skill_collector.os, "walk", walk_with_late_target)++ with pytest.raises(SkillCollectionError) as exc_info:+ collect_files(skill_root)++ assert exc_info.value.check_name == "unsafe_path"+ assert exc_info.value.rel_path == "CLAUDE.md"++ @pytest.mark.parametrize("target", ["./AGENTS.md", "../AGENTS.md", "missing.md"])+ def test_rejects_non_exact_openclaw_compatibility_alias(self, skill_root: Path, target: str) -> None:+ (skill_root / "AGENTS.md").write_text("# Shared agent instructions\n")+ try:+ (skill_root / "CLAUDE.md").symlink_to(target)+ except OSError as exc:+ pytest.skip(f"symlinks unavailable: {exc}")++ with pytest.raises(SkillCollectionError) as exc_info:+ collect_files(skill_root)++ assert exc_info.value.check_name == "unsafe_path"+ assert exc_info.value.rel_path == "CLAUDE.md"+class TestCollectFilesSafety:@pytest.mark.skipif(os.name != "posix", reason="descriptor-anchored availability is a POSIX-specific contract")diff --git a/tests/embedding/test_extractor.py b/tests/embedding/test_extractor.pyindex be7511ff..4fe560a9 100644--- a/tests/embedding/test_extractor.py+++ b/tests/embedding/test_extractor.py@@ -394,6 +394,106 @@ class TestDiscoverAndExtract:names = {e.name for e in entries}assert names == {"skill-a", "skill-b"}+ def test_openclaw_compatibility_alias_does_not_abort_skill_discovery(self, tmp_path: Path, write_skill) -> None:+ skill_dir = write_skill(tmp_path, "autoreview", "Review changes with multiple agents")+ (skill_dir / "AGENTS.md").write_text("# Shared agent instructions\n")+ try:+ (skill_dir / "CLAUDE.md").symlink_to("AGENTS.md")+ except OSError as exc:+ pytest.skip(f"symlinks unavailable: {exc}")++ entries = discover_and_extract(tmp_path, "skill")++ assert [entry.name for entry in entries] == ["autoreview"]++ def test_rejects_openclaw_alias_when_regular_target_is_hard_linked(+ self,+ tmp_path: Path,+ write_skill,+ ) -> None:+ collection = tmp_path / "skills"+ skill_dir = write_skill(collection, "autoreview", "Review changes with multiple agents")+ outside_target = tmp_path / "outside-agents.md"+ outside_target.write_text("# Outside agent instructions\n")+ try:+ os.link(outside_target, skill_dir / "AGENTS.md")+ (skill_dir / "CLAUDE.md").symlink_to("AGENTS.md")+ except OSError as exc:+ pytest.skip(f"hard links or symlinks unavailable: {exc}")++ with pytest.raises(ValueError, match=r"symlink|reparse|unsafe"):+ discover_and_extract(collection, "skill")++ def test_rejects_hard_linked_skill_manifest(self, tmp_path: Path) -> None:+ collection = tmp_path / "skills"+ skill_dir = collection / "hard-linked-skill"+ skill_dir.mkdir(parents=True)+ outside_manifest = tmp_path / "outside-SKILL.md"+ outside_manifest.write_text(VALID_SKILL_MD)+ try:+ os.link(outside_manifest, skill_dir / "SKILL.md")+ except OSError as exc:+ pytest.skip(f"hard links unavailable: {exc}")++ with pytest.raises(ValueError, match=r"hard-linked|unsafe"):+ discover_and_extract(collection, "skill")++ def test_rejects_openclaw_alias_when_target_entry_case_does_not_match(+ self,+ tmp_path: Path,+ write_skill,+ ) -> None:+ collection = tmp_path / "skills"+ skill_dir = write_skill(collection, "autoreview", "Review changes with multiple agents")+ (skill_dir / "agents.md").write_text("# Lowercase agent instructions\n")+ try:+ (skill_dir / "CLAUDE.md").symlink_to("AGENTS.md")+ except OSError as exc:+ pytest.skip(f"symlinks unavailable: {exc}")++ with pytest.raises(ValueError, match=r"symlink|reparse|unsafe"):+ discover_and_extract(collection, "skill")++ def test_rejects_alias_target_created_after_discovery_snapshot(self, tmp_path: Path, monkeypatch) -> None:+ collection = tmp_path / "skills"+ skill_dir = collection / "autoreview"+ skill_dir.mkdir(parents=True)+ try:+ (skill_dir / "CLAUDE.md").symlink_to("AGENTS.md")+ except OSError as exc:+ pytest.skip(f"symlinks unavailable: {exc}")++ real_walk = os.walk++ def walk_with_late_target(*args, **kwargs):+ for dirpath, dirnames, filenames in real_walk(*args, **kwargs):+ if Path(dirpath) == skill_dir:+ assert "AGENTS.md" not in filenames+ (skill_dir / "AGENTS.md").write_text("# Late agent instructions\n")+ yield dirpath, dirnames, filenames++ monkeypatch.setattr(extractor_module.os, "walk", walk_with_late_target)++ with pytest.raises(ValueError, match=r"symlink|reparse|unsafe"):+ discover_and_extract(collection, "skill")++ @pytest.mark.parametrize("target", ["./AGENTS.md", "../AGENTS.md", "missing.md"])+ def test_rejects_non_exact_openclaw_alias_during_skill_discovery(+ self,+ tmp_path: Path,+ write_skill,+ target: str,+ ) -> None:+ skill_dir = write_skill(tmp_path, "autoreview", "Review changes with multiple agents")+ (skill_dir / "AGENTS.md").write_text("# Shared agent instructions\n")+ try:+ (skill_dir / "CLAUDE.md").symlink_to(target)+ except OSError as exc:+ pytest.skip(f"symlinks unavailable: {exc}")++ with pytest.raises(ValueError, match=r"symlink|reparse|unsafe"):+ discover_and_extract(tmp_path, "skill")+def test_discover_rules_in_folder(self, tmp_path: Path) -> None:rules_dir = tmp_path / "team-rules"rules_dir.mkdir()diff --git a/tests/test_cli.py b/tests/test_cli.pyindex 467bf076..6f34d7bb 100644--- a/tests/test_cli.py+++ b/tests/test_cli.py@@ -134,6 +134,41 @@ def test_similarity_cli_rejects_catalog_build_and_query_together(tmp_path: Path)assert "cannot be used together" in result.output+def test_similarity_cli_handles_pinned_openclaw_compatibility_layout(tmp_path: Path, monkeypatch) -> None:+ collection = tmp_path / "skills"+ autoreview = collection / "autoreview"+ autoreview.mkdir(parents=True)+ (autoreview / "SKILL.md").write_text(+ "---\nname: autoreview\ndescription: Review changes with multiple agents\n---\n"+ )+ (autoreview / "AGENTS.md").write_text("# Shared agent instructions\n")+ try:+ (autoreview / "CLAUDE.md").symlink_to("AGENTS.md")+ except OSError as exc:+ pytest.skip(f"symlinks unavailable: {exc}")++ second = collection / "second-skill"+ second.mkdir()+ (second / "SKILL.md").write_text("---\nname: second-skill\ndescription: Exercise an unrelated workflow\n---\n")++ embedded_texts: list[str] = []++ def embed_without_network(_self, texts: list[str]) -> list[list[float]]:+ embedded_texts.extend(texts)+ return [[1.0, float(index)] for index, _text in enumerate(texts)]++ monkeypatch.setattr("skillevaluator.embedding.client.EmbeddingClient.embed", embed_without_network)++ result = CliRunner().invoke(cli, ["similarity-check", str(collection), "--type", "skill"])++ assert result.exit_code == 0, result.output+ assert "All validations passed" in result.output+ assert sorted(embedded_texts) == [+ "autoreview: Review changes with multiple agents",+ "second-skill: Exercise an unrelated workflow",+ ]++@pytest.mark.parametrize(("command", "runner_path"),[