Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier2): share a repeated text only when that shortens the review prompt (#346)
Narendran Raghavanc55ffb888e4d
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5e92c480..8b05fb58 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -543,11 +543,12 @@ All notable changes to SkillEvaluator are documented in this file.guard and never stops a scan on its own; the per-cluster prompt, totalprompt, and cluster-count limits decide. When a cluster's request (systemprompt plus cluster text) would exceed 64,000 characters, each exactly- repeated text is sent once as `SHARED CONTENT n`. A Markdown section is- compared without its own heading line, so the same body under different- headings is shared too. Every occurrence keeps its own entry with its file,- section, line range, and heading line. Distinct text is never dropped or- sampled, so a cluster still over the budget stays incomplete with+ repeated text is sent once as `SHARED CONTENT n` when that makes the request+ shorter; a short text repeated under long headings stays inline. A Markdown+ section is compared without its own heading line, so the same body under+ different headings is shared too. Every occurrence keeps its own entry with+ its file, section, line range, and heading line. Distinct text is never+ dropped or sampled, so a cluster still over the budget stays incomplete with`llm_prompt_size_limit`. Duplicate and `llm_refused` finding messages nameat most 64 member locations and source files and count the rest; the JSON`source_files` list stays complete.diff --git a/docs/tier2-deduplication.mdx b/docs/tier2-deduplication.mdxindex 4f0d0882..9c441c9d 100644--- a/docs/tier2-deduplication.mdx+++ b/docs/tier2-deduplication.mdx@@ -156,8 +156,10 @@ exactly is listed once as `SHARED CONTENT n`. A Markdown section's text startswith its own heading line, so the text compared is the body after that line:the same body under different headings is listed once too. Each occurrencekeeps its own entry with its file, section, line range, and heading line, plus-a reference to the shared text. Distinct text, including text that differs by a-single character, is always sent in full; nothing is sampled or dropped to fit.+a reference to the shared text. A repeated text is listed once only when that+shortens the request, so a short one, such as a one-line body under long+headings, stays inline. Distinct text, including text that differs by a single+character, is always sent in full; nothing is sampled or dropped to fit.A cluster that is still over 64,000 characters stops the scan as incomplete(`llm_prompt_size_limit`), as described below. A duplicate or`llm_refused` finding names up to 64 member locations and source files and@@ -366,7 +368,7 @@ you intend to skip Tier 2 entirely.| Intra-skill comparison work | 267,911,168 scalar comparisons | Bounds all-pairs clustering; allows all 512 chunks with embeddings of up to 2,048 dimensions, 418 chunks at 3,072, and 362 at 4,096 || Intra-skill LLM review clusters | 50 | Most candidate clusters sent for chat-LLM review in one skill || Intra-skill cluster members | 512 | Same as the chunk ceiling, so it never stops a scan on its own; the review prompt limits decide |-| Intra-skill review prompt | 64,000 characters | One cluster's request, system prompt included; exactly repeated text (a Markdown section's body after its heading line) is listed once before this limit applies |+| Intra-skill review prompt | 64,000 characters | One cluster's request, system prompt included; exactly repeated text (a Markdown section's body after its heading line) is listed once before this limit applies, when that shortens the request || Intra-skill review prompts in total | 1,000,000 characters | All cluster requests for one skill, checked before the first chat-LLM call |Intra-skill clustering compares every pair of chunks, so its work isdiff --git a/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py b/src/skillevaluator/deduplication/intra_skill/llm_analyzer.pyindex a59957e0..8d438e9a 100644--- a/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py+++ b/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py@@ -24,6 +24,9 @@ VALID_VERDICTS = {"DUPLICATE", "INTENTIONAL_DETAIL", "RELATED_BUT_DISTINCT"}# The validator refuses a cluster whose request is still over it.MAX_PROMPT_CHARS = 64_000+# Joins the parts of a user prompt (notice, shared blocks, chunks, footer).+_PART_SEPARATOR = "\n\n"+VERDICT_JSON_SCHEMA: dict[str, Any] = {"type": "object","properties": {@@ -70,16 +73,20 @@ def build_user_prompt(cluster: ContentCluster) -> str:compared is the body after that line: the same body under differentheadings is listed once too. Every occurrence keeps its own ``CHUNK``entry with file, section and lines, its own heading line, and a- ``[SHARED CONTENT n]`` reference in place of the shared text. Distinct- text is never dropped or shortened, so a request still over budget stays- over budget.+ ``[SHARED CONTENT n]`` reference in place of the shared text. A repeated+ text is shared only when its block and references are shorter than its+ inline copies, so a short text repeated under long headings stays+ inline. Distinct text is never dropped or shortened, so a request still+ over budget stays over budget."""prompt = _render_user_prompt(cluster)if len(SYSTEM_PROMPT) + len(prompt) <= MAX_PROMPT_CHARS:return promptcounts = Counter(_split_own_heading(chunk)[1] for chunk in cluster.members)- repeated = (text for text, count in counts.items() if count > 1)- shared = {text: index for index, text in enumerate(repeated, start=1)}+ shared: dict[str, int] = {}+ for text, count in counts.items():+ if count > 1 and _sharing_saves_chars(text, count, len(shared) + 1):+ shared[text] = len(shared) + 1if not shared:return promptcompact = _render_user_prompt(cluster, shared)@@ -101,6 +108,26 @@ def _split_own_heading(chunk: ContentChunk) -> tuple[str, str]:return "", chunk.text+def _shared_block(text: str, index: int) -> str:+ return f"---SHARED CONTENT {index}---\n{text}"+++def _shared_reference(index: int) -> str:+ return f"[SHARED CONTENT {index}]"+++def _sharing_saves_chars(text: str, count: int, index: int) -> bool:+ """Whether listing ``text`` once as shared block ``index`` shortens the prompt.++ Inline, each of the ``count`` occurrences carries the text. Shared, each+ carries a reference instead, and the text appears once in its own block,+ joined to the other prompt parts by ``_PART_SEPARATOR``.+ """+ inline_chars = count * len(text)+ shared_chars = count * len(_shared_reference(index)) + len(_shared_block(text, index)) + len(_PART_SEPARATOR)+ return shared_chars < inline_chars++def _render_user_prompt(cluster: ContentCluster, shared: dict[str, int] | None = None) -> str:"""Render every member; texts in ``shared`` are listed once and referenced."""parts: list[str] = []@@ -110,13 +137,13 @@ def _render_user_prompt(cluster: ContentCluster, shared: dict[str, int] | None ="the exact text of that shared block. Review every occurrence in its own file, section and ""line context. No chunks were omitted.")- parts.extend(f"---SHARED CONTENT {index}---\n{text}" for text, index in shared.items())+ parts.extend(_shared_block(text, index) for text, index in shared.items())for i, chunk in enumerate(cluster.members, start=1):content = chunk.textif shared:heading_line, body = _split_own_heading(chunk)if body in shared:- reference = f"[SHARED CONTENT {shared[body]}]"+ reference = _shared_reference(shared[body])content = f"{heading_line}\n{reference}" if heading_line else referenceparts.append(f"---CHUNK {i}---\n"@@ -128,7 +155,7 @@ def _render_user_prompt(cluster: ContentCluster, shared: dict[str, int] | None =parts.append(f"\nEmbedding similarity: {cluster.max_similarity:.3f}")parts.append("\nClassify this content overlap.")- return "\n\n".join(parts)+ return _PART_SEPARATOR.join(parts)def analyze_cluster(client: LLMClient, cluster: ContentCluster) -> LLMVerdict:diff --git a/tests/deduplication/intra_skill/test_intra_llm_analyzer.py b/tests/deduplication/intra_skill/test_intra_llm_analyzer.pyindex ed0379ea..ac6dde75 100644--- a/tests/deduplication/intra_skill/test_intra_llm_analyzer.py+++ b/tests/deduplication/intra_skill/test_intra_llm_analyzer.py@@ -109,15 +109,82 @@ class TestBuildUserPrompt:for member in members:assert f"Section: {member.heading}\nLines: 1-5\nContent:\n{member.text}" in prompt- def test_sharing_is_skipped_when_it_does_not_shorten_the_prompt(self, make_chunk) -> None:+ @pytest.mark.parametrize(+ "repeated",+ [+ pytest.param("short repeated line", id="block-longer-than-copies"),+ # Sharing this text alone saves 39 characters, fewer than the+ # shared-content notice adds.+ pytest.param("n" * 100, id="saving-smaller-than-notice"),+ ],+ )+ def test_sharing_is_skipped_when_it_does_not_shorten_the_prompt(self, make_chunk, repeated: str) -> None:members = [make_chunk(heading=f"Distinct {index}", text=f"{index}: " + "d" * 2_500) for index in range(30)]- members += [make_chunk(heading=f"Short {index}", text="short repeated line") for index in range(2)]+ members += [make_chunk(heading=f"Short {index}", text=repeated) for index in range(2)]prompt = build_user_prompt(ContentCluster(members, 0.95, 0.85, False, {"markdown"}))assert _request_chars(prompt) > llm_analyzer.MAX_PROMPT_CHARSassert "SHARED CONTENT" not in prompt- assert prompt.count("Content:\nshort repeated line") == 2+ assert prompt.count(f"Content:\n{repeated}") == 2++ @pytest.mark.parametrize(+ ("length", "shared"),+ [pytest.param(61, False, id="saves-nothing"), pytest.param(62, True, id="saves-one-char")],+ )+ def test_repeated_text_is_shared_only_when_that_saves_characters(+ self, make_chunk, length: int, shared: bool+ ) -> None:+ # Two inline copies cost 2 * length characters. As block 2 they cost two+ # 18-character "[SHARED CONTENT 2]" references, the 23-character block+ # header, one copy and a 2-character separator: length + 61.+ big = "f" * 1_500+ text = "t" * length+ members = [make_chunk(heading=f"Big {index}", text=big) for index in range(60)]+ members += [make_chunk(heading=f"Small {index}", text=text) for index in range(2)]++ prompt = build_user_prompt(ContentCluster(members, 0.95, 0.85, False, {"markdown"}))++ assert _request_chars(prompt) <= llm_analyzer.MAX_PROMPT_CHARS+ assert prompt.count(big) == 1+ assert prompt.count("[SHARED CONTENT 1]") == 60+ if shared:+ assert prompt.count(f"---SHARED CONTENT 2---\n{text}\n\n") == 1+ assert prompt.count("Content:\n[SHARED CONTENT 2]") == 2+ else:+ assert "SHARED CONTENT 2" not in prompt+ assert prompt.count(f"Content:\n{text}\n\n") == 2++ def test_short_repeated_bodies_under_long_headings_stay_inline(self, make_chunk) -> None:+ short_body = "See guide."+ long_body = "Boilerplate paragraph about safety and usage. " * 20+ short_members = [+ make_chunk(source_file="SKILL.md", heading=heading, text=f"{heading}\n{short_body}")+ for heading in (+ f"## Configuration option for the workflow runner, number {index:04d} of the set"+ for index in range(270)+ )+ ]+ long_members = [+ make_chunk(source_file="SKILL.md", heading=f"## Big {index}", text=f"## Big {index}\n{long_body}")+ for index in range(25)+ ]+ cluster = ContentCluster([*short_members, *long_members], 0.95, 0.85, False, {"markdown"})+ budget = llm_analyzer.MAX_PROMPT_CHARS+ # Sent in full, or with both repeated bodies shared, the request is over+ # budget: a "[SHARED CONTENT n]" reference is longer than "See guide.".+ assert _request_chars(llm_analyzer._render_user_prompt(cluster)) > budget+ assert _request_chars(llm_analyzer._render_user_prompt(cluster, {short_body: 1, long_body: 2})) > budget++ prompt = build_user_prompt(cluster)++ assert _request_chars(prompt) <= budget+ assert prompt.count("---SHARED CONTENT ") == 1+ assert prompt.count(f"---SHARED CONTENT 1---\n{long_body}\n\n") == 1+ assert prompt.count(long_body) == 1+ assert prompt.count("Content:\n## Big ") == prompt.count("[SHARED CONTENT 1]") == 25+ for member in short_members:+ assert f"Section: {member.heading}\nLines: 1-5\nContent:\n{member.text}\n\n" in promptdef test_same_body_under_different_heading_lines_is_shared(self, make_chunk) -> None:body = "Describe the supported workflow. " * 4diff --git a/tests/deduplication/intra_skill/test_intra_skill_validator.py b/tests/deduplication/intra_skill/test_intra_skill_validator.pyindex 2c8cfdf9..d3169107 100644--- a/tests/deduplication/intra_skill/test_intra_skill_validator.py+++ b/tests/deduplication/intra_skill/test_intra_skill_validator.py@@ -493,6 +493,54 @@ class TestIntraSkillValidatorValidate:if len(headings) > 64:assert finding.message.endswith(f"\n and {len(headings) - 64} more locations ({len(headings)} in total)")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.analyze_cluster")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.LLMClient")+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")+ def test_short_repeated_bodies_under_long_headings_stay_inline_and_the_cluster_completes(+ self, mock_embed, _mock_llm, mock_analyze, tmp_path: Path+ ) -> None:+ skill_dir = tmp_path / "long-headings"+ skill_dir.mkdir()+ short_body = "See guide."+ long_body = "Boilerplate paragraph about safety and usage. " * 20+ headings = [+ f"## Configuration option for the workflow runner, number {index:04d} of the set" for index in range(270)+ ]+ headings += [f"## Big {index}" for index in range(14)]+ bodies = [short_body] * 270 + [long_body] * 14+ markdown = "".join(f"{heading}\n{body}\n" for heading, body in zip(headings, bodies, strict=True))+ (skill_dir / "SKILL.md").write_bytes(markdown.encode("utf-8"))+ mock_embed.return_value.embed.side_effect = _unit_embeddings(2)+ mock_analyze.return_value = LLMVerdict(+ verdict="DUPLICATE", confidence=0.9, reasoning="Repeated instructions", suggestion="Consolidate sections"+ )++ result = IntraSkillValidator().validate(skill_dir)++ assert result.incomplete_scans == []+ assert "work_limit" not in result.metadata+ mock_analyze.assert_called_once()+ cluster = mock_analyze.call_args.args[1]+ assert [member.heading for member in cluster.members] == headings+ budget = llm_analyzer.MAX_PROMPT_CHARS+ shared_body = long_body.strip()+ # Sent in full, or with both repeated bodies shared, the request is over+ # budget: a "[SHARED CONTENT n]" reference is longer than "See guide.".+ assert len(SYSTEM_PROMPT) + len(llm_analyzer._render_user_prompt(cluster)) > budget+ all_shared = llm_analyzer._render_user_prompt(cluster, {short_body: 1, shared_body: 2})+ assert all_shared.count("[SHARED CONTENT 1]") == 270+ assert len(SYSTEM_PROMPT) + len(all_shared) > budget+ prompt = build_user_prompt(cluster)+ assert len(SYSTEM_PROMPT) + len(prompt) <= budget+ assert prompt.count("---SHARED CONTENT ") == 1+ assert prompt.count(shared_body) == 1+ assert prompt.count("[SHARED CONTENT 1]") == 14+ for member in cluster.members[:270]:+ assert member.text == f"{member.heading}\n{short_body}"+ assert f"Content:\n{member.text}\n\n" in prompt+ [finding] = result.findings+ assert finding.severity == Severity.HIGH+@pytest.mark.parametrize("verdict", ["DUPLICATE", "RELATED_BUT_DISTINCT"])@patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.analyze_cluster")@patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.LLMClient")