Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier2): review large repeated-content clusters within the prompt budget (#338)
Narendran Raghavaned3aad5281bf
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 891ec395..7afc799c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -423,6 +423,22 @@ All notable changes to SkillEvaluator are documented in this file.dimensions, 362 at 4,096), which the incomplete-scan error and the JSON`work_limit` record now report as `effective_chunk_limit`, next to theskill's own `chunk_count`.+- Tier 2 context deduplication no longer stops skills with many repeated+ sections at the `llm_cluster_member_limit` work limit. A candidate cluster+ with more than 64 members was reported as incomplete even when its review+ prompt fit the 64,000-character budget. The member limit now equals the+ 512-chunk ceiling, which the chunk check already enforces, so it is only a+ guard and never stops a scan on its own; the per-cluster prompt, total+ prompt, and cluster-count limits decide. When a cluster's request (system+ prompt plus cluster text) would exceed 64,000 characters, each exactly+ repeated text is sent once as `SHARED CONTENT n`. A Markdown section is+ compared without its own heading line, so the same body under different+ headings is shared too. Every occurrence keeps its own entry with its file,+ section, line range, and heading line. Distinct text is never dropped or+ sampled, so a cluster still over the budget stays incomplete with+ `llm_prompt_size_limit`. Duplicate and `llm_refused` finding messages name+ at most 64 member locations and source files and count the rest; the JSON+ `source_files` list stays complete.- Local-mode verifier uploads now copy directly to the absent destination,avoiding an unnecessary rollback snapshot of an empty tests directory.Inputs and rewritten scripts are prepared privately before publication,@@ -617,8 +633,8 @@ All notable changes to SkillEvaluator are documented in this file.- A skill that exceeds a Tier 2 context-deduplication work limit (more than512 content chunks, too much pairwise comparison work, or an LLM-review limit:- more than 50 candidate clusters, more than 64 members in one cluster, or an- oversized per-cluster or total review prompt) is now reported as+ more than 50 candidate clusters or an oversized per-cluster or total+ review prompt) is now reported as**INCOMPLETE** instead of failing with a CRITICAL `chunk_count_limit`,`scalar_comparison_limit`, `llm_cluster_count_limit`,`llm_cluster_member_limit`, `llm_prompt_size_limit`, ordiff --git a/docs/tier2-deduplication.mdx b/docs/tier2-deduplication.mdxindex b57a7ad5..88246076 100644--- a/docs/tier2-deduplication.mdx+++ b/docs/tier2-deduplication.mdx@@ -149,14 +149,29 @@ Duplicate findings map to severities by confidence:| `DUPLICATE` with confidence < 0.70 | MEDIUM — advisory || Short, same-file duplicate dominated by comment or configuration-style text | Capped at LOW — advisory |+Each candidate cluster is reviewed in one chat-LLM request that carries every+member's file, section, line range, and text. When that request (the system+prompt plus the cluster text) would exceed 64,000 characters, text that repeats+exactly is listed once as `SHARED CONTENT n`. A Markdown section's text starts+with its own heading line, so the text compared is the body after that line:+the same body under different headings is listed once too. Each occurrence+keeps its own entry with its file, section, line range, and heading line, plus+a reference to the shared text. Distinct text, including text that differs by a+single character, is always sent in full; nothing is sampled or dropped to fit.+A cluster that is still over 64,000 characters stops the scan as incomplete+(`llm_prompt_size_limit`), as described below. A duplicate or+`llm_refused` finding names up to 64 member locations and source files and+counts the rest; its `source_files` metadata in the JSON report lists every+file.+If chat-LLM analysis fails for a candidate cluster, Tier 2 reports theincomplete analysis as a blocking error rather than treating the content asclean.The same applies when a skill exceeds a work limit on content that wasalready collected: the content-chunk or comparison-work limit, or a limit on-the LLM review (candidate clusters, members in one cluster, or prompt size per-cluster or in total). Tier 2 stops and reports the scan as **INCOMPLETE**, not+the LLM review (the number of candidate clusters, or prompt size per cluster+or in total). Tier 2 stops and reports the scan as **INCOMPLETE**, notas a duplicate-content finding, with an execution error that names the limit;the JSON report records it under `work_limit`. A policy severity overridecannot turn the incomplete scan into a pass. Split the skill or remove repeated@@ -341,6 +356,10 @@ you intend to skip Tier 2 entirely.| High-confidence duplicate boundary | `0.70` | Maps actionable duplicate verdicts to HIGH rather than MEDIUM || Intra-skill content chunks | 512 | Per-skill ceiling, checked before any embedding request || Intra-skill comparison work | 267,911,168 scalar comparisons | Bounds all-pairs clustering; allows all 512 chunks with embeddings of up to 2,048 dimensions, 418 chunks at 3,072, and 362 at 4,096 |+| Intra-skill LLM review clusters | 50 | Most candidate clusters sent for chat-LLM review in one skill |+| Intra-skill cluster members | 512 | Same as the chunk ceiling, so it never stops a scan on its own; the review prompt limits decide |+| Intra-skill review prompt | 64,000 characters | One cluster's request, system prompt included; exactly repeated text (a Markdown section's body after its heading line) is listed once before this limit applies |+| Intra-skill review prompts in total | 1,000,000 characters | All cluster requests for one skill, checked before the first chat-LLM call |Intra-skill clustering compares every pair of chunks, so its work is`n * (n - 1) / 2 * dimensions`. A skill over the chunk ceiling stops before any@@ -408,7 +427,7 @@ handling terms before sending confidential skill content to a hosted endpoint.| Skill contains a relative directory alias such as `.agents -> .claude` whose target is a regular directory in the same skill | Skips the alias without following it, then scans the target directory once || A directory is a Windows junction, other reparse point, or a symbolic link that does not qualify as a contained alias | Rejected during the walk, before descent || Embedding or chat-LLM analysis is incomplete | Reports an error; does not claim a clean result |- | Skill exceeds the content-chunk, comparison-work, or LLM-review limit (candidate clusters, cluster members, or prompt size) | Stops before the remaining analysis and reports the scan as incomplete; no duplicate-content finding is added |+ | Skill exceeds the content-chunk, comparison-work, or LLM-review limit (candidate clusters or prompt size) | Stops before the remaining analysis and reports the scan as incomplete; no duplicate-content finding is added || The chat LLM declines to classify a cluster | Reports an advisory MEDIUM `llm_refused` finding for that cluster's sections |</Accordion><Accordion title="Inter-skill analysis">diff --git a/src/skillevaluator/deduplication/intra_skill/intra_skill_validator.py b/src/skillevaluator/deduplication/intra_skill/intra_skill_validator.pyindex c7850845..2d24571a 100644--- a/src/skillevaluator/deduplication/intra_skill/intra_skill_validator.py+++ b/src/skillevaluator/deduplication/intra_skill/intra_skill_validator.py@@ -25,6 +25,7 @@ from skillevaluator.constants import (CONTENT_DEDUP_SIMILARITY_THRESHOLD,CONTENT_DEDUP_TRIVIAL_DUP_MAX_CHARS,)+from skillevaluator.deduplication.intra_skill import llm_analyzerfrom skillevaluator.deduplication.intra_skill.llm_analyzer import (SYSTEM_PROMPT,analyze_cluster,@@ -48,9 +49,14 @@ logger = logging.getLogger(__name__)# config assignment. Used to recognize repeated config/comment snippets._COMMENT_PREFIXES = ("#", ";", "//", "--", "!")_CONFIG_KV_RE = re.compile(r"^[\w.\-/]+\s*[=:]\s*\S")-CONTENT_DEDUP_MAX_CLUSTER_MEMBERS = 64-CONTENT_DEDUP_MAX_LLM_PROMPT_CHARS = 64_000+# One cluster may hold every chunk of the scan. The per-call and aggregate+# prompt budgets bound paid review; a fixed member cap below the chunk ceiling+# would refuse large repeated-content clusters whose prompt fits.+CONTENT_DEDUP_MAX_CLUSTER_MEMBERS = CONTENT_DEDUP_MAX_CHUNKSCONTENT_DEDUP_MAX_TOTAL_LLM_PROMPT_CHARS = 1_000_000+# Most member locations and source files named in one finding message. The+# rest are counted, so a finding for a 512-member cluster stays a few kilobytes.+_MAX_REPORTED_LOCATIONS = 64# Scan label recorded when a work limit stops duplicate analysis early._WORK_LIMIT_SCAN = "context-deduplication"@@ -112,6 +118,27 @@ def _is_trivial_intra_file_duplicate(cluster: ContentCluster) -> bool:return all(_chunk_is_short_config(member) for member in cluster.members)+def _cluster_location_text(cluster: ContentCluster) -> tuple[str, str]:+ """Return the finding scope and member location list, bounded in length."""+ distinct_files = sorted({c.source_file for c in cluster.members})+ if len(distinct_files) == 1:+ scope = f"within {distinct_files[0]}"+ else:+ scope = f"across {' and '.join(distinct_files[:_MAX_REPORTED_LOCATIONS])}"+ if (omitted_files := len(distinct_files) - _MAX_REPORTED_LOCATIONS) > 0:+ scope += f" and {omitted_files} more file{'s' if omitted_files != 1 else ''}"++ locations = [+ f'"{c.heading}" in {c.source_file} (lines {c.start_line}-{c.end_line})'+ for c in cluster.members[:_MAX_REPORTED_LOCATIONS]+ ]+ location_text = "\n vs ".join(locations)+ if (omitted := len(cluster.members) - _MAX_REPORTED_LOCATIONS) > 0:+ noun = "locations" if omitted != 1 else "location"+ location_text += f"\n and {omitted} more {noun} ({len(cluster.members)} in total)"+ return scope, location_text++def _mark_work_limit_incomplete(result: ValidationResult,check_name: str,@@ -319,7 +346,9 @@ class IntraSkillValidator(ValidatorBase):),(prompt_chars,- CONTENT_DEDUP_MAX_LLM_PROMPT_CHARS,+ # Read at call time: build_user_prompt shares repeated text+ # against this same attribute, so one value sets both.+ llm_analyzer.MAX_PROMPT_CHARS,"llm_prompt_size_limit","an LLM review prompt of {actual} characters exceeds the limit of {limit}.",),@@ -418,15 +447,7 @@ class IntraSkillValidator(ValidatorBase):continuedistinct_files = sorted({c.source_file for c in cluster.members})-- locations = []- for c in cluster.members:- locations.append(f'"{c.heading}" in {c.source_file} (lines {c.start_line}-{c.end_line})')- location_text = "\n vs ".join(locations)- if len(distinct_files) == 1:- scope = f"within {distinct_files[0]}"- else:- scope = f"across {' and '.join(distinct_files)}"+ scope, location_text = _cluster_location_text(cluster)if verdict is None:# Report the refused overlap so the author can review it, butdiff --git a/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py b/src/skillevaluator/deduplication/intra_skill/llm_analyzer.pyindex 70825d4f..a59957e0 100644--- a/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py+++ b/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py@@ -8,6 +8,7 @@ Builds prompts from content clusters, calls LLMClient, maps verdicts to severityfrom __future__ import annotations+from collections import Counterfrom typing import TYPE_CHECKING, Anyfrom skillevaluator.inference import LLMClient, LLMClientError, LLMVerdict@@ -15,9 +16,14 @@ from skillevaluator.models.result import Severityif TYPE_CHECKING:from skillevaluator.deduplication.intra_skill.semantic_clustering import ContentCluster+ from skillevaluator.deduplication.utils.chunker import ContentChunkVALID_VERDICTS = {"DUPLICATE", "INTENTIONAL_DETAIL", "RELATED_BUT_DISTINCT"}+# Per-call budget for one review request: SYSTEM_PROMPT plus the user prompt.+# The validator refuses a cluster whose request is still over it.+MAX_PROMPT_CHARS = 64_000+VERDICT_JSON_SCHEMA: dict[str, Any] = {"type": "object","properties": {@@ -55,15 +61,69 @@ Respond with ONLY a JSON object:def build_user_prompt(cluster: ContentCluster) -> str:- """Build the user prompt from cluster member chunks."""+ """Build the user prompt from cluster member chunks.++ Every member is rendered with its full text. When the request+ (``SYSTEM_PROMPT`` plus this prompt) would exceed ``MAX_PROMPT_CHARS``,+ each exactly repeated text is listed once as ``SHARED CONTENT n``. A+ Markdown section's text opens with its own heading line, so the text+ compared is the body after that line: the same body under different+ headings is listed once too. Every occurrence keeps its own ``CHUNK``+ entry with file, section and lines, its own heading line, and a+ ``[SHARED CONTENT n]`` reference in place of the shared text. Distinct+ text is never dropped or shortened, so a request still over budget stays+ over budget.+ """+ prompt = _render_user_prompt(cluster)+ if len(SYSTEM_PROMPT) + len(prompt) <= MAX_PROMPT_CHARS:+ return prompt+ counts = Counter(_split_own_heading(chunk)[1] for chunk in cluster.members)+ repeated = (text for text, count in counts.items() if count > 1)+ shared = {text: index for index, text in enumerate(repeated, start=1)}+ if not shared:+ return prompt+ compact = _render_user_prompt(cluster, shared)+ return compact if len(compact) < len(prompt) else prompt+++def _split_own_heading(chunk: ContentChunk) -> tuple[str, str]:+ """Return the chunk's own heading line and the text after it.++ A Markdown section chunk opens with its heading line, so the same body+ under different headings differs only in that line. Text that does not+ open with the chunk's heading line, or has nothing after it, is returned+ whole with an empty heading line. When a heading line is returned,+ ``heading_line + "\n" + body`` is exactly the chunk text.+ """+ first_line, newline, body = chunk.text.partition("\n")+ if newline and body.strip() and first_line.rstrip() == chunk.heading:+ return first_line, body+ return "", chunk.text+++def _render_user_prompt(cluster: ContentCluster, shared: dict[str, int] | None = None) -> str:+ """Render every member; texts in ``shared`` are listed once and referenced."""parts: list[str] = []+ if shared:+ parts.append(+ "Identical content is listed once below. In a chunk's content, [SHARED CONTENT n] stands for "+ "the exact text of that shared block. Review every occurrence in its own file, section and "+ "line context. No chunks were omitted."+ )+ parts.extend(f"---SHARED CONTENT {index}---\n{text}" for text, index in shared.items())for i, chunk in enumerate(cluster.members, start=1):+ content = chunk.text+ if shared:+ heading_line, body = _split_own_heading(chunk)+ if body in shared:+ reference = f"[SHARED CONTENT {shared[body]}]"+ content = f"{heading_line}\n{reference}" if heading_line else referenceparts.append(f"---CHUNK {i}---\n"f"File: {chunk.source_file}\n"f"Section: {chunk.heading}\n"f"Lines: {chunk.start_line}-{chunk.end_line}\n"- f"Content:\n{chunk.text}"+ f"Content:\n{content}")parts.append(f"\nEmbedding similarity: {cluster.max_similarity:.3f}")diff --git a/tests/deduplication/intra_skill/test_intra_llm_analyzer.py b/tests/deduplication/intra_skill/test_intra_llm_analyzer.pyindex d1be7d21..ed0379ea 100644--- a/tests/deduplication/intra_skill/test_intra_llm_analyzer.py+++ b/tests/deduplication/intra_skill/test_intra_llm_analyzer.py@@ -9,6 +9,7 @@ from unittest.mock import MagicMockimport pytest+from skillevaluator.deduplication.intra_skill import llm_analyzerfrom skillevaluator.deduplication.intra_skill.llm_analyzer import (SYSTEM_PROMPT,VERDICT_JSON_SCHEMA,@@ -16,11 +17,171 @@ from skillevaluator.deduplication.intra_skill.llm_analyzer import (build_user_prompt,verdict_to_severity,)+from skillevaluator.deduplication.intra_skill.semantic_clustering import ContentClusterfrom skillevaluator.inference import LLMClient, LLMClientError, LLMVerdictfrom skillevaluator.models.result import Severity+def _request_chars(prompt: str) -> int:+ """Characters of one review request, measured as the validator measures it."""+ return len(SYSTEM_PROMPT) + len(prompt)++class TestBuildUserPrompt:+ def test_large_repeated_content_preserves_each_occurrence(self, make_chunk) -> None:+ repeated = "Exact repeated instructions. " * 10+ unique = "Unique instructions for the final section. " * 3+ members = [+ make_chunk(source_file=f"file-{index // 4}.md", heading=f"Section {index}", text=repeated)+ for index in range(500)+ ]+ members.append(make_chunk(source_file="unique.md", heading="Unique", text=unique))+ cluster = ContentCluster(members, 0.95, 0.85, True, {"markdown"})++ prompt = build_user_prompt(cluster)++ assert _request_chars(prompt) <= llm_analyzer.MAX_PROMPT_CHARS+ assert prompt.count(repeated) == 1+ assert prompt.count(unique) == 1+ assert prompt.count("---SHARED CONTENT ") == 1+ assert prompt.count("---CHUNK ") == 501+ assert prompt.count("Content:\n[SHARED CONTENT 1]") == 500+ assert "No chunks were omitted." in prompt+ for index, member in enumerate(members, start=1):+ assert (+ f"---CHUNK {index}---\nFile: {member.source_file}\nSection: {member.heading}\n"+ f"Lines: {member.start_line}-{member.end_line}\n"+ ) in prompt+ assert f"Section: Unique\nLines: 1-5\nContent:\n{unique}" in prompt+ assert prompt.endswith("Classify this content overlap.")++ def test_each_repeated_text_is_shared_once_in_first_seen_order(self, make_chunk) -> None:+ first = "f" * 1_500+ second = "s" * 1_500+ members = [make_chunk(heading=f"Section {index}", text=first if index % 2 else second) for index in range(60)]+ cluster = ContentCluster(members, 0.95, 0.85, False, {"markdown"})++ prompt = build_user_prompt(cluster)++ assert _request_chars(prompt) <= llm_analyzer.MAX_PROMPT_CHARS+ assert prompt.count(first) == prompt.count(second) == 1+ assert prompt.index(f"---SHARED CONTENT 1---\n{second}") < prompt.index(f"---SHARED CONTENT 2---\n{first}")+ assert prompt.count("Content:\n[SHARED CONTENT 1]") == 30+ assert prompt.count("Content:\n[SHARED CONTENT 2]") == 30+ assert "Section: Section 1\nLines: 1-5\nContent:\n[SHARED CONTENT 2]" in prompt++ def test_request_budget_includes_the_system_prompt(self, make_chunk, monkeypatch) -> None:+ repeated = "r" * 2_000+ filler = make_chunk(source_file="unique.md", heading="Unique", text="u" * 80)+ members = [make_chunk(heading=f"Section {index}", text=repeated) for index in range(30)]+ cluster = ContentCluster([*members, filler], 0.95, 0.85, True, {"markdown"})+ budget = llm_analyzer.MAX_PROMPT_CHARS+ with monkeypatch.context() as patched:+ patched.setattr(llm_analyzer, "MAX_PROMPT_CHARS", 10**9)+ filler.text += "u" * (budget - len(SYSTEM_PROMPT) // 2 - len(build_user_prompt(cluster)))+ full_prompt = build_user_prompt(cluster)+ # The user prompt alone fits; the request with the system prompt does not.+ assert len(full_prompt) <= budget < _request_chars(full_prompt)++ prompt = build_user_prompt(cluster)++ assert _request_chars(prompt) <= budget+ assert prompt.count(repeated) == 1+ assert prompt.count(filler.text) == 1+ assert prompt.count("---CHUNK ") == 31++ def test_prompt_within_budget_keeps_every_occurrence_inline(self, make_chunk) -> None:+ repeated = "Exact repeated instructions. " * 10+ members = [make_chunk(heading=f"Section {index}", text=repeated) for index in range(3)]++ prompt = build_user_prompt(ContentCluster(members, 0.95, 0.85, False, {"markdown"}))++ assert prompt.count(f"Content:\n{repeated}") == 3+ assert "SHARED CONTENT" not in prompt++ def test_distinct_text_over_budget_is_never_dropped_or_shared(self, make_chunk) -> None:+ members = [make_chunk(heading=f"Member {index}", text=f"{index}: " + "x" * 80) for index in range(500)]++ prompt = build_user_prompt(ContentCluster(members, 0.95, 0.85, False, {"markdown"}))++ assert _request_chars(prompt) > llm_analyzer.MAX_PROMPT_CHARS+ assert "SHARED CONTENT" not in prompt+ for member in members:+ assert f"Section: {member.heading}\nLines: 1-5\nContent:\n{member.text}" in prompt++ def test_sharing_is_skipped_when_it_does_not_shorten_the_prompt(self, make_chunk) -> None:+ members = [make_chunk(heading=f"Distinct {index}", text=f"{index}: " + "d" * 2_500) for index in range(30)]+ members += [make_chunk(heading=f"Short {index}", text="short repeated line") for index in range(2)]++ prompt = build_user_prompt(ContentCluster(members, 0.95, 0.85, False, {"markdown"}))++ assert _request_chars(prompt) > llm_analyzer.MAX_PROMPT_CHARS+ assert "SHARED CONTENT" not in prompt+ assert prompt.count("Content:\nshort repeated line") == 2++ def test_same_body_under_different_heading_lines_is_shared(self, make_chunk) -> None:+ body = "Describe the supported workflow. " * 4+ members = [make_chunk(heading=f"## Step {index}", text=f"## Step {index}\n{body}") for index in range(500)]+ cluster = ContentCluster(members, 0.95, 0.85, False, {"markdown"})++ prompt = build_user_prompt(cluster)++ assert _request_chars(prompt) <= llm_analyzer.MAX_PROMPT_CHARS+ assert prompt.count(body) == 1+ assert f"---SHARED CONTENT 1---\n{body}\n\n---CHUNK 1---" in prompt+ assert prompt.count("---CHUNK ") == 500+ for index, member in enumerate(members, start=1):+ # The heading line plus the shared body is exactly the member's text.+ assert (+ f"---CHUNK {index}---\nFile: README.md\nSection: {member.heading}\nLines: 1-5\n"+ f"Content:\n{member.heading}\n[SHARED CONTENT 1]\n\n"+ ) in prompt++ def test_heading_line_with_a_carriage_return_is_kept_with_each_occurrence(self, make_chunk) -> None:+ body = "Describe the supported workflow.\r\n" * 4+ members = [+ make_chunk(heading=f"## Step {index}", text=f"## Step {index}\r\n{body}".strip()) for index in range(500)+ ]+ cluster = ContentCluster(members, 0.95, 0.85, False, {"markdown"})++ prompt = build_user_prompt(cluster)++ assert _request_chars(prompt) <= llm_analyzer.MAX_PROMPT_CHARS+ assert prompt.count(body.strip()) == 1+ assert prompt.count("---SHARED CONTENT ") == 1+ for member in members:+ assert f"Content:\n{member.heading}\r\n[SHARED CONTENT 1]" in prompt++ def test_whole_text_matching_another_chunks_body_shares_it(self, make_chunk) -> None:+ body = "Describe the supported workflow. " * 4+ sections = [make_chunk(heading=f"## Step {index}", text=f"## Step {index}\n{body}") for index in range(250)]+ docstrings = [+ make_chunk(source_file="tool.py", heading=f"step_{index}()", text=body, source_format="python")+ for index in range(250)+ ]+ cluster = ContentCluster([*sections, *docstrings], 0.95, 0.85, True, {"markdown", "python"})++ prompt = build_user_prompt(cluster)++ assert _request_chars(prompt) <= llm_analyzer.MAX_PROMPT_CHARS+ assert prompt.count(body) == 1+ assert prompt.count("---SHARED CONTENT ") == 1+ assert prompt.count("[SHARED CONTENT 1]") == 500+ for member in docstrings:+ assert f"Section: {member.heading}\nLines: 1-5\nContent:\n[SHARED CONTENT 1]" in prompt++ def test_text_that_does_not_open_with_its_heading_line_is_compared_whole(self, make_chunk) -> None:+ body = "Describe the supported workflow. " * 4+ # The first line differs per chunk but is not the chunk's heading.+ members = [make_chunk(heading=f"## Step {index}", text=f"Step {index}\n{body}") for index in range(500)]+ cluster = ContentCluster(members, 0.95, 0.85, False, {"markdown"})++ prompt = build_user_prompt(cluster)++ assert _request_chars(prompt) > llm_analyzer.MAX_PROMPT_CHARS+ assert "SHARED CONTENT" not in prompt+ assert prompt.count(body) == 500+def test_contains_chunk_details(self, make_cluster) -> None:cluster = make_cluster()prompt = build_user_prompt(cluster)@@ -112,6 +273,26 @@ class TestAnalyzeCluster:with pytest.raises(LLMClientError, match="unknown verdict"):analyze_cluster(mock_client, make_cluster())+ def test_sends_the_shared_content_prompt_that_the_validator_measures(self, make_chunk) -> None:+ repeated = "Exact repeated instructions. " * 10+ members = [make_chunk(heading=f"Section {index}", text=repeated) for index in range(500)]+ cluster = ContentCluster(members, 0.95, 0.85, False, {"markdown"})+ mock_client = MagicMock(spec=LLMClient)+ mock_client.extract_json_from_response.return_value = {+ "verdict": "DUPLICATE",+ "confidence": 0.9,+ "rationale": "Same instructions in every section",+ "suggestion": "Keep one copy",+ }++ analyze_cluster(mock_client, cluster)++ system_prompt, user_prompt = mock_client.extract_json_from_response.call_args.args+ assert system_prompt == SYSTEM_PROMPT+ assert user_prompt == build_user_prompt(cluster)+ assert user_prompt.count(repeated) == 1+ assert _request_chars(user_prompt) <= llm_analyzer.MAX_PROMPT_CHARS+def test_requests_the_verdict_schema_and_reads_rationale(self, make_cluster) -> None:mock_client = MagicMock(spec=LLMClient)mock_client.extract_json_from_response.return_value = {diff --git a/tests/deduplication/intra_skill/test_intra_skill_validator.py b/tests/deduplication/intra_skill/test_intra_skill_validator.pyindex 65699dda..2c8cfdf9 100644--- a/tests/deduplication/intra_skill/test_intra_skill_validator.py+++ b/tests/deduplication/intra_skill/test_intra_skill_validator.py@@ -13,7 +13,7 @@ from unittest.mock import patchimport pytestfrom skillevaluator import constants-from skillevaluator.deduplication.intra_skill import intra_skill_validator+from skillevaluator.deduplication.intra_skill import intra_skill_validator, llm_analyzerfrom skillevaluator.deduplication.intra_skill.intra_skill_validator import IntraSkillValidator, effective_chunk_limitfrom skillevaluator.deduplication.intra_skill.llm_analyzer import SYSTEM_PROMPT, build_user_promptfrom skillevaluator.deduplication.intra_skill.semantic_clustering import ContentCluster@@ -375,17 +375,17 @@ class TestIntraSkillValidatorValidate:("limit_name", "check_name", "expected_message"),[(- "CONTENT_DEDUP_MAX_CLUSTER_MEMBERS",+ "intra_skill_validator.CONTENT_DEDUP_MAX_CLUSTER_MEMBERS","llm_cluster_member_limit","a content cluster has 2 members, more than the limit of 1 for LLM review.",),(- "CONTENT_DEDUP_MAX_LLM_PROMPT_CHARS",+ "llm_analyzer.MAX_PROMPT_CHARS","llm_prompt_size_limit","an LLM review prompt of {actual} characters exceeds the limit of 1.",),(- "CONTENT_DEDUP_MAX_TOTAL_LLM_PROMPT_CHARS",+ "intra_skill_validator.CONTENT_DEDUP_MAX_TOTAL_LLM_PROMPT_CHARS","llm_total_prompt_size_limit","LLM review prompts total {actual} characters, more than the limit of 1.",),@@ -405,7 +405,7 @@ class TestIntraSkillValidatorValidate:tmp_path: Path,monkeypatch,) -> None:- monkeypatch.setattr(intra_skill_validator, limit_name, 1)+ monkeypatch.setattr(f"skillevaluator.deduplication.intra_skill.{limit_name}", 1)skill_dir = tmp_path / "skill"skill_dir.mkdir()(skill_dir / "SKILL.md").write_text("## Section A\n" + "a" * 200 + "\n## Section B\n" + "a" * 200)@@ -426,6 +426,268 @@ class TestIntraSkillValidatorValidate:assert str(tmp_path) not in result.errors[0]mock_llm.assert_not_called()+ def test_cluster_member_limit_is_the_chunk_ceiling(self) -> None:+ assert intra_skill_validator.CONTENT_DEDUP_MAX_CLUSTER_MEMBERS == constants.CONTENT_DEDUP_MAX_CHUNKS == 512+ assert llm_analyzer.MAX_PROMPT_CHARS == 64_000+ # The validator reads llm_analyzer.MAX_PROMPT_CHARS; a stale copy here+ # would let a patched budget move one threshold and not the other.+ assert not hasattr(intra_skill_validator, "CONTENT_DEDUP_MAX_LLM_PROMPT_CHARS")+ assert intra_skill_validator.CONTENT_DEDUP_MAX_TOTAL_LLM_PROMPT_CHARS == 1_000_000++ @pytest.mark.parametrize("newline", ["\n", "\r\n"], ids=["lf", "crlf"])+ @pytest.mark.parametrize(+ ("headings", "body"),+ [+ pytest.param(["## Repeated"] * 500, "Describe the supported workflow. " * 4, id="500-same-heading"),+ pytest.param(+ [f"## Step {index}" for index in range(500)],+ "Describe the supported workflow. " * 4,+ id="500-distinct-headings",+ ),+ pytest.param(+ [f"## Tool {index}" for index in range(30)],+ "\n\n".join(["Boilerplate paragraph about safety and usage. " * 20] * 3),+ id="30-distinct-headings-long-body",+ ),+ ],+ )+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.analyze_cluster")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.LLMClient")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")+ def test_repeated_section_bodies_complete_through_real_chunking_and_clustering(+ self, mock_embed, _mock_llm, mock_analyze, tmp_path: Path, headings: list[str], body: str, newline: str+ ) -> None:+ skill_dir = tmp_path / "repeated-sections"+ skill_dir.mkdir()+ markdown = "".join(f"{heading}\n{body}\n" for heading in headings)+ (skill_dir / "SKILL.md").write_bytes(markdown.replace("\n", newline).encode("utf-8"))+ mock_embed.return_value.embed.side_effect = _unit_embeddings(2)+ mock_analyze.return_value = LLMVerdict(+ verdict="DUPLICATE", confidence=0.9, reasoning="Repeated instructions", suggestion="Consolidate sections"+ )++ result = IntraSkillValidator().validate(skill_dir)++ assert result.status == "failed"+ assert result.incomplete_scans == []+ assert "work_limit" not in result.metadata+ mock_analyze.assert_called_once()+ cluster = mock_analyze.call_args.args[1]+ assert len(cluster.members) == len(headings)+ assert [member.heading for member in cluster.members] == headings+ assert all(len(member.text) >= constants.CONTENT_DEDUP_MIN_CHUNK_CHARS for member in cluster.members)+ prompt = build_user_prompt(cluster)+ assert len(SYSTEM_PROMPT) + len(prompt) <= llm_analyzer.MAX_PROMPT_CHARS+ assert prompt.count("---CHUNK ") == len(headings)+ # Each section's text is its own heading line plus the shared body.+ shared_body = body.strip().replace("\n", newline)+ assert prompt.count("---SHARED CONTENT ") == 1+ assert prompt.count(shared_body) == 1+ for member in cluster.members:+ assert member.text == f"{member.heading}{newline}{shared_body}"+ assert f"Content:\n{member.heading}{newline}[SHARED CONTENT 1]" in prompt+ [finding] = result.findings+ assert finding.severity == Severity.HIGH+ assert finding.message.startswith("Duplicate content found within SKILL.md:\n")+ assert finding.message.count(" in SKILL.md (lines ") == min(len(headings), 64)+ if len(headings) > 64:+ assert finding.message.endswith(f"\n and {len(headings) - 64} more locations ({len(headings)} in total)")++ @pytest.mark.parametrize("verdict", ["DUPLICATE", "RELATED_BUT_DISTINCT"])+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.analyze_cluster")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.LLMClient")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.build_clusters")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")+ def test_large_member_cluster_is_reviewed_in_full_when_prompt_fits(+ self, mock_embed, mock_clusters, _mock_llm, mock_analyze, tmp_path: Path, verdict: str+ ) -> None:+ skill_dir = tmp_path / "large-cluster"+ skill_dir.mkdir()+ (skill_dir / "SKILL.md").write_text("## A\n" + "a" * 100 + "\n## B\n" + "b" * 100, encoding="utf-8")+ mock_embed.return_value.embed.return_value = [[1.0, 0.0], [1.0, 0.0]]+ members = [+ ContentChunk("SKILL.md", f"Member {index}", index + 1, index + 1, "content " * 10, "markdown")+ for index in range(500)+ ]+ cluster = ContentCluster(members, 1.0, 1.0, False, {"markdown"})+ mock_clusters.return_value = [cluster]+ mock_analyze.return_value = LLMVerdict(+ verdict=verdict, confidence=0.9, reasoning="Reviewed all members", suggestion="Consolidate duplicates"+ )++ result = IntraSkillValidator().validate(skill_dir)++ assert result.incomplete_scans == []+ assert "work_limit" not in result.metadata+ mock_analyze.assert_called_once()+ assert mock_analyze.call_args.args[1] is cluster+ prompt = build_user_prompt(cluster)+ assert len(SYSTEM_PROMPT) + len(prompt) <= llm_analyzer.MAX_PROMPT_CHARS+ assert prompt.count("---CHUNK ") == 500+ for member in members:+ assert f"Section: {member.heading}\n" in prompt+ if verdict == "DUPLICATE":+ assert result.passed is False+ [finding] = result.findings+ assert finding.severity == Severity.HIGH+ assert '"Member 0" in SKILL.md (lines 1-1)' in finding.message+ assert '"Member 63" in SKILL.md (lines 64-64)' in finding.message+ assert '"Member 64"' not in finding.message+ assert finding.message.endswith("\n and 436 more locations (500 in total)")+ assert finding.metadata["source_files"] == ["SKILL.md"]+ else:+ assert result.passed is True+ assert result.findings == []++ @pytest.mark.parametrize(+ ("member_count", "text", "check_name"),+ [+ pytest.param(513, "content " * 10, "llm_cluster_member_limit", id="member-limit"),+ pytest.param(500, "x" * 80, "llm_prompt_size_limit", id="prompt-limit"),+ ],+ )+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.LLMClient")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.build_clusters")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")+ def test_large_cluster_still_respects_member_and_prompt_limits(+ self,+ mock_embed,+ mock_clusters,+ mock_llm,+ tmp_path: Path,+ member_count: int,+ text: str,+ check_name: str,+ ) -> None:+ skill_dir = tmp_path / "cluster-budget-limit"+ skill_dir.mkdir()+ (skill_dir / "SKILL.md").write_text("## A\n" + "a" * 100 + "\n## B\n" + "b" * 100, encoding="utf-8")+ mock_embed.return_value.embed.return_value = [[1.0, 0.0], [1.0, 0.0]]+ # Members differ in their index prefix, so distinct text is never shared.+ members = [+ ContentChunk("SKILL.md", f"Member {index}", index + 1, index + 1, f"{index}: {text}", "markdown")+ for index in range(member_count)+ ]+ cluster = ContentCluster(members, 1.0, 1.0, False, {"markdown"})+ mock_clusters.return_value = [cluster]+ if check_name == "llm_cluster_member_limit":+ actual, limit = member_count, 512+ else:+ actual, limit = len(SYSTEM_PROMPT) + len(build_user_prompt(cluster)), 64_000+ assert "SHARED CONTENT" not in build_user_prompt(cluster)++ result = IntraSkillValidator().validate(skill_dir)++ assert actual > limit+ self._assert_work_limit_incomplete(result, {"check_name": check_name, "actual": actual, "limit": limit})+ mock_llm.assert_not_called()++ @pytest.mark.parametrize("repeated", [True, False], ids=["shared-fits", "distinct-refused"])+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.analyze_cluster")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.LLMClient")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.build_clusters")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")+ def test_one_prompt_budget_sets_sharing_and_refusal(+ self, mock_embed, mock_clusters, mock_llm, mock_analyze, tmp_path: Path, monkeypatch, repeated: bool+ ) -> None:+ budget = 8_000+ monkeypatch.setattr(llm_analyzer, "MAX_PROMPT_CHARS", budget)+ skill_dir = tmp_path / "lowered-budget"+ skill_dir.mkdir()+ (skill_dir / "SKILL.md").write_text("## A\n" + "a" * 100 + "\n## B\n" + "b" * 100, encoding="utf-8")+ mock_embed.return_value.embed.return_value = [[1.0, 0.0], [1.0, 0.0]]+ members = [+ ContentChunk(+ "SKILL.md",+ f"Member {index}",+ index + 1,+ index + 1,+ ("" if repeated else f"{index}: ") + "z" * 300,+ "markdown",+ )+ for index in range(40)+ ]+ cluster = ContentCluster(members, 1.0, 1.0, False, {"markdown"})+ mock_clusters.return_value = [cluster]+ mock_analyze.return_value = LLMVerdict("DUPLICATE", 0.9, "Same content", "Keep one copy")+ prompt = build_user_prompt(cluster)+ prompt_chars = len(SYSTEM_PROMPT) + len(prompt)++ result = IntraSkillValidator().validate(skill_dir)++ if repeated:+ # Shared under the lowered budget, then accepted by that same budget.+ assert "SHARED CONTENT" in prompt+ assert prompt_chars <= budget+ assert result.incomplete_scans == []+ assert "work_limit" not in result.metadata+ mock_analyze.assert_called_once()+ else:+ # Distinct text cannot be shared, so the lowered budget refuses it.+ assert "SHARED CONTENT" not in prompt+ assert prompt_chars > budget+ self._assert_work_limit_incomplete(+ result, {"check_name": "llm_prompt_size_limit", "actual": prompt_chars, "limit": budget}+ )+ mock_llm.assert_not_called()+ mock_analyze.assert_not_called()++ @pytest.mark.parametrize(+ ("member_count", "message_tail"),+ [+ pytest.param(64, '"Section 63" in file-063.md (lines 64-64)', id="64-members"),+ pytest.param(65, "\n and 1 more location (65 in total)", id="65-members"),+ pytest.param(100, "\n and 36 more locations (100 in total)", id="100-members"),+ ],+ )+ @pytest.mark.parametrize("refused", [False, True], ids=["duplicate", "refused"])+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.analyze_cluster")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.LLMClient")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.build_clusters")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")+ def test_finding_message_names_at_most_64_locations_and_files(+ self,+ mock_embed,+ mock_clusters,+ _mock_llm,+ mock_analyze,+ tmp_path: Path,+ refused: bool,+ member_count: int,+ message_tail: str,+ ) -> None:+ skill_dir = tmp_path / "many-files"+ skill_dir.mkdir()+ (skill_dir / "SKILL.md").write_text("## A\n" + "a" * 100 + "\n## B\n" + "b" * 100, encoding="utf-8")+ mock_embed.return_value.embed.return_value = [[1.0, 0.0], [1.0, 0.0]]+ members = [+ ContentChunk(f"file-{index:03}.md", f"Section {index}", index + 1, index + 1, "y" * 100, "markdown")+ for index in range(member_count)+ ]+ mock_clusters.return_value = [ContentCluster(members, 1.0, 1.0, True, {"markdown"})]+ if refused:+ mock_analyze.side_effect = LLMClientRefusedError("declined")+ else:+ mock_analyze.return_value = LLMVerdict("DUPLICATE", 0.9, "Same content", "Keep one copy")++ result = IntraSkillValidator().validate(skill_dir)++ [finding] = result.findings+ assert finding.check_name == ("llm_refused" if refused else "duplicate")+ assert finding.metadata["source_files"] == [member.source_file for member in members]+ scope, _, locations = finding.message.partition(":\n ")+ named_files = scope.split("across ", 1)[1].split(" and ")+ assert named_files[:64] == [member.source_file for member in members[:64]]+ assert locations.count(" (lines ") == min(member_count, 64)+ assert locations.endswith(message_tail)+ if member_count > 64:+ omitted = member_count - 64+ assert scope.endswith(f" and {omitted} more file{'s' if omitted != 1 else ''}")+ assert '"Section 64"' not in locations+ else:+ assert "more file" not in scope+ assert "more location" not in locations+@patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")def test_policy_override_cannot_waive_work_limit(self, mock_embed, tmp_path: Path, monkeypatch) -> None:"""Severity overrides remap findings; they must not turn missing analysis into a pass."""diff --git a/tests/deduplication/plugin/test_private_dedup.py b/tests/deduplication/plugin/test_private_dedup.pyindex e2b37b69..f78a17a8 100644--- a/tests/deduplication/plugin/test_private_dedup.py+++ b/tests/deduplication/plugin/test_private_dedup.py@@ -419,12 +419,13 @@ def test_private_cluster_allowance_keeps_default_skill_behavior(tmp_path, monkeydef test_context_prompt_limit_refuses_llm_before_external_work(tmp_path, monkeypatch):from skillevaluator.deduplication.intra_skill import intra_skill_validator as core+ from skillevaluator.deduplication.intra_skill import llm_analyzerroot = tmp_path / "skill"root.mkdir()(root / "SKILL.md").write_text("## One\n" + "a" * 200 + "\n## Two\n" + "a" * 200)monkeypatch.setattr(core, "EmbeddingClient", lambda **_kwargs: FakeEmbedding())- monkeypatch.setattr(core, "CONTENT_DEDUP_MAX_LLM_PROMPT_CHARS", 100, raising=False)+ monkeypatch.setattr(llm_analyzer, "MAX_PROMPT_CHARS", 100)def forbidden(**_kwargs):raise AssertionError("LLM must not start above the prompt bound")