Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(tier2): number shared review blocks from the most repeated text (#361)

Narendran Raghavan164e5fcbd9e7
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 63bc8d49..1d35f05a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -271,16 +271,18 @@ All notable changes to SkillEvaluator are documented in this file.
512-chunk ceiling, which the chunk check already enforces, so it is only a
guard and never stops a scan on its own; the per-cluster prompt, total
prompt, and cluster-count limits decide. When a cluster's request (system
- prompt plus cluster text) would exceed 64,000 characters, each exactly
- repeated text is sent once as `SHARED CONTENT n` when that makes the request
- shorter; a short text repeated under long headings stays inline. A Markdown
- section is compared without its own heading line, so the same body under
- different headings is shared too. Every occurrence keeps its own entry with
- its file, section, line range, and heading line. Distinct text is never
- dropped or sampled, so a cluster still over the budget stays incomplete with
- `llm_prompt_size_limit`. Duplicate and `llm_refused` finding messages name
- at most 64 member locations and source files and count the rest; the JSON
- `source_files` list stays complete.
+ prompt plus cluster text) would exceed 64,000 characters, exactly repeated
+ texts are sent once as `SHARED CONTENT n`, choosing the ones that make the
+ request shortest. A short text, such as a one-line body, stays inline
+ because its reference would be longer than the text. The most repeated texts
+ get blocks 1-9, whose references are one character shorter than those of
+ later blocks. A Markdown section is compared without its own heading line,
+ so the same body under different headings is shared too. Every occurrence
+ keeps its own entry with its file, section, line range, and heading line.
+ Distinct text is never dropped or sampled, so a cluster still over the
+ budget stays incomplete with `llm_prompt_size_limit`. Duplicate and
+ `llm_refused` finding messages name at most 64 member locations and source
+ files and count the rest; the JSON `source_files` list stays complete.
- On the native Claude API, the Tier 3 judges' `judge_model_used` names the
server-side fallback model when the API serves a repeated request straight
diff --git a/docs/tier2-deduplication.mdx b/docs/tier2-deduplication.mdx
index 9c441c9d..c2bb502e 100644
--- a/docs/tier2-deduplication.mdx
+++ b/docs/tier2-deduplication.mdx
@@ -157,9 +157,14 @@ with its own heading line, so the text compared is the body after that line:
the same body under different headings is listed once too. Each occurrence
keeps its own entry with its file, section, line range, and heading line, plus
a reference to the shared text. A repeated text is listed once only when that
-shortens the request, so a short one, such as a one-line body under long
-headings, stays inline. Distinct text, including text that differs by a single
-character, is always sent in full; nothing is sampled or dropped to fit.
+shortens the request. A short one, such as a one-line body, stays inline because
+its reference would be longer than the text; the heading line stays with every
+occurrence either way, so heading length does not affect this. The most
+repeated texts are numbered first, because a reference to block 10 or later is
+one character longer than one to blocks 1-9, and the texts listed once are the
+ones that shorten the request most. Distinct text, including text that differs
+by a single character, is always sent in full; nothing is sampled or dropped to
+fit.
A cluster that is still over 64,000 characters stops the scan as incomplete
(`llm_prompt_size_limit`), as described below. A duplicate or
`llm_refused` finding names up to 64 member locations and source files and
diff --git a/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py b/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py
index 8d438e9a..9361ff5b 100644
--- a/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py
+++ b/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py
@@ -75,24 +75,60 @@ def build_user_prompt(cluster: ContentCluster) -> str:
entry with file, section and lines, its own heading line, and a
``[SHARED CONTENT n]`` reference in place of the shared text. A repeated
text is shared only when its block and references are shorter than its
- inline copies, so a short text repeated under long headings stays
- inline. Distinct text is never dropped or shortened, so a request still
- over budget stays over budget.
+ inline copies. A short text, such as a one-line body, stays inline
+ because its reference would be longer than the text; the heading line
+ stays with every occurrence either way, so its length does not matter.
+ Blocks are numbered and chosen as described in ``_choose_shared_texts``.
+ Distinct text is never dropped or shortened, so a request still over
+ budget stays over budget.
"""
prompt = _render_user_prompt(cluster)
if len(SYSTEM_PROMPT) + len(prompt) <= MAX_PROMPT_CHARS:
return prompt
- counts = Counter(_split_own_heading(chunk)[1] for chunk in cluster.members)
- shared: dict[str, int] = {}
- for text, count in counts.items():
- if count > 1 and _sharing_saves_chars(text, count, len(shared) + 1):
- shared[text] = len(shared) + 1
+ shared = _choose_shared_texts(Counter(_split_own_heading(chunk)[1] for chunk in cluster.members))
if not shared:
return prompt
compact = _render_user_prompt(cluster, shared)
return compact if len(compact) < len(prompt) else prompt
+def _choose_shared_texts(counts: Counter[str]) -> dict[str, int]:
+ """Return the repeated texts to share, mapped to their block numbers in order.
+
+ A reference to block 10 or later is one character longer than one to
+ blocks 1-9, so the numbering matters. For any set of shared texts, the
+ prompt is shortest when the most repeated texts get the lowest numbers,
+ so candidates are numbered in descending repeat count, ties in first-seen
+ order. Which candidates to share is then chosen exactly, by dynamic
+ programming over that order, to save the most characters: sharing a text
+ that saves only a little can push a more repeated text past block 9 and
+ cost it one character in every reference.
+ """
+ candidates = [(text, count) for text, count in counts.most_common() if count > 1]
+ # best[k]: most characters saved by sharing k of the candidates seen so
+ # far as blocks 1..k, or None when no such k of them all save characters.
+ best: list[int | None] = [0] + [None] * len(candidates)
+ taken: list[list[bool]] = []
+ for seen, (text, count) in enumerate(candidates, start=1):
+ row = [False] * (len(candidates) + 1)
+ for index in range(seen, 0, -1):
+ before = best[index - 1]
+ saving = _sharing_saving(text, count, index)
+ current = best[index]
+ if before is not None and saving > 0 and (current is None or before + saving > current):
+ best[index] = before + saving
+ row[index] = True
+ taken.append(row)
+ # The fewest blocks among the largest savings.
+ blocks = max(range(len(best)), key=lambda k: -1 if best[k] is None else best[k])
+ shared: dict[str, int] = {}
+ for (text, _count), row in zip(reversed(candidates), reversed(taken), strict=True):
+ if row[blocks]:
+ shared[text] = blocks
+ blocks -= 1
+ return dict(sorted(shared.items(), key=lambda item: item[1]))
+
+
def _split_own_heading(chunk: ContentChunk) -> tuple[str, str]:
"""Return the chunk's own heading line and the text after it.
@@ -116,16 +152,18 @@ def _shared_reference(index: int) -> str:
return f"[SHARED CONTENT {index}]"
-def _sharing_saves_chars(text: str, count: int, index: int) -> bool:
- """Whether listing ``text`` once as shared block ``index`` shortens the prompt.
+def _sharing_saving(text: str, count: int, index: int) -> int:
+ """Characters saved by listing ``text`` once as shared block ``index``.
Inline, each of the ``count`` occurrences carries the text. Shared, each
carries a reference instead, and the text appears once in its own block,
- joined to the other prompt parts by ``_PART_SEPARATOR``.
+ joined to the other prompt parts by ``_PART_SEPARATOR``. A negative
+ result means sharing makes the prompt longer.
"""
inline_chars = count * len(text)
- shared_chars = count * len(_shared_reference(index)) + len(_shared_block(text, index)) + len(_PART_SEPARATOR)
- return shared_chars < inline_chars
+ block_chars = len(_shared_block("", index)) + len(text)
+ shared_chars = count * len(_shared_reference(index)) + block_chars + len(_PART_SEPARATOR)
+ return inline_chars - shared_chars
def _render_user_prompt(cluster: ContentCluster, shared: dict[str, int] | None = None) -> str:
diff --git a/tests/deduplication/intra_skill/test_intra_llm_analyzer.py b/tests/deduplication/intra_skill/test_intra_llm_analyzer.py
index ac6dde75..afa08be6 100644
--- a/tests/deduplication/intra_skill/test_intra_llm_analyzer.py
+++ b/tests/deduplication/intra_skill/test_intra_llm_analyzer.py
@@ -55,7 +55,7 @@ class TestBuildUserPrompt:
assert f"Section: Unique\nLines: 1-5\nContent:\n{unique}" in prompt
assert prompt.endswith("Classify this content overlap.")
- def test_each_repeated_text_is_shared_once_in_first_seen_order(self, make_chunk) -> None:
+ def test_equally_repeated_texts_are_numbered_in_first_seen_order(self, make_chunk) -> None:
first = "f" * 1_500
second = "s" * 1_500
members = [make_chunk(heading=f"Section {index}", text=first if index % 2 else second) for index in range(60)]
@@ -70,6 +70,56 @@ class TestBuildUserPrompt:
assert prompt.count("Content:\n[SHARED CONTENT 2]") == 30
assert "Section: Section 1\nLines: 1-5\nContent:\n[SHARED CONTENT 2]" in prompt
+ def test_most_repeated_text_gets_the_shortest_reference(self, make_chunk) -> None:
+ # Nine 62-character texts seen first, each twice, save one character
+ # each as blocks 1-9. Numbered after them, a 40-character body repeated
+ # 400 times would be block 10, and its 19-character references would
+ # cost 400 characters more in all than references to blocks 1-9.
+ pairs = [letter * 62 for letter in "acdefghij"]
+ body = "b" * 40
+ members = [make_chunk(heading=f"Pair {k}.{j}", text=text) for k, text in enumerate(pairs) for j in range(2)]
+ members += [make_chunk(heading=f"Body {j}", text=body) for j in range(400)]
+ filler = make_chunk(source_file="unique.md", heading="Filler", text="u")
+ cluster = ContentCluster([*members, filler], 0.95, 0.85, False, {"markdown"})
+ budget = llm_analyzer.MAX_PROMPT_CHARS
+ first_seen = {text: index for index, text in enumerate([*pairs, body], start=1)}
+ filler.text += "u" * (budget + 200 - _request_chars(llm_analyzer._render_user_prompt(cluster, first_seen)))
+ assert _request_chars(llm_analyzer._render_user_prompt(cluster, first_seen)) == budget + 200
+
+ prompt = build_user_prompt(cluster)
+
+ assert _request_chars(prompt) == budget - 200
+ assert prompt.count(f"---SHARED CONTENT 1---\n{body}\n\n") == 1
+ assert prompt.count("Content:\n[SHARED CONTENT 1]") == 400
+ # Equally repeated, the pairs keep first-seen order; the ninth would be
+ # block 10, where sharing it costs characters, so it stays inline.
+ for index, text in enumerate(pairs[:8], start=2):
+ assert prompt.count(f"---SHARED CONTENT {index}---\n{text}\n\n") == 1
+ assert "SHARED CONTENT 10" not in prompt
+ assert prompt.count(f"Content:\n{pairs[8]}\n\n") == 2
+
+ def test_text_that_would_push_a_more_repeated_text_past_block_9_stays_inline(self, make_chunk) -> None:
+ # The most repeated text saves 14 characters as block 1 and none past
+ # block 9. Sharing it moves the ninth other text to block 10, which
+ # costs that text's 19 references and its header one character each.
+ marginal = "m" * 21
+ others = [letter * 200 for letter in "abcdefghi"]
+ members = [make_chunk(heading=f"Marginal {j}", text=marginal) for j in range(20)]
+ members += [make_chunk(heading=f"Other {k}.{j}", text=text) for k, text in enumerate(others) for j in range(19)]
+ members.append(make_chunk(source_file="unique.md", heading="Filler", text="u" * 20_000))
+ cluster = ContentCluster(members, 0.95, 0.85, False, {"markdown"})
+ greedy = {marginal: 1, **{text: index for index, text in enumerate(others, start=2)}}
+ assert _request_chars(llm_analyzer._render_user_prompt(cluster)) > llm_analyzer.MAX_PROMPT_CHARS
+
+ prompt = build_user_prompt(cluster)
+
+ assert len(llm_analyzer._render_user_prompt(cluster, greedy)) - len(prompt) == 20 - 14
+ assert prompt.count(f"Content:\n{marginal}\n\n") == 20
+ for index, text in enumerate(others, start=1):
+ assert prompt.count(f"---SHARED CONTENT {index}---\n{text}\n\n") == 1
+ assert prompt.count(f"Content:\n[SHARED CONTENT {index}]") == 19
+ assert "SHARED CONTENT 10" not in prompt
+
def test_request_budget_includes_the_system_prompt(self, make_chunk, monkeypatch) -> None:
repeated = "r" * 2_000
filler = make_chunk(source_file="unique.md", heading="Unique", text="u" * 80)
@@ -155,6 +205,61 @@ class TestBuildUserPrompt:
assert "SHARED CONTENT 2" not in prompt
assert prompt.count(f"Content:\n{text}\n\n") == 2
+ @pytest.mark.parametrize(
+ ("length", "shared"),
+ [pytest.param(64, False, id="saves-nothing-as-block-10"), pytest.param(65, True, id="saves-one-char")],
+ )
+ def test_two_digit_block_numbers_cost_one_more_character(self, make_chunk, length: int, shared: bool) -> None:
+ # As block 10, two copies cost two 19-character "[SHARED CONTENT 10]"
+ # references, the 24-character block header, one copy and a
+ # 2-character separator: length + 64. As block 1-9 they would cost
+ # length + 61, so a 64-character text would be shared there.
+ bigs = [letter * 1_500 for letter in "abcdefghi"]
+ text = "t" * length
+ members = [make_chunk(heading=f"Big {k}.{j}", text=big) for k, big in enumerate(bigs) for j in range(5)]
+ members += [make_chunk(heading=f"Small {index}", text=text) for index in range(2)]
+
+ prompt = build_user_prompt(ContentCluster(members, 0.95, 0.85, False, {"markdown"}))
+
+ assert _request_chars(prompt) <= llm_analyzer.MAX_PROMPT_CHARS
+ for index, big in enumerate(bigs, start=1):
+ assert prompt.count(f"---SHARED CONTENT {index}---\n{big}\n\n") == 1
+ if shared:
+ assert prompt.count(f"---SHARED CONTENT 10---\n{text}\n\n") == 1
+ assert prompt.count("Content:\n[SHARED CONTENT 10]") == 2
+ else:
+ assert "SHARED CONTENT 10" not in prompt
+ assert prompt.count(f"Content:\n{text}\n\n") == 2
+
+ @pytest.mark.parametrize(
+ "heading",
+ [
+ pytest.param("## H{}", id="short-headings"),
+ pytest.param("## Configuration option for the workflow runner, number {:04d}", id="long-headings"),
+ ],
+ )
+ def test_short_repeated_body_stays_inline_whatever_its_heading_length(self, make_chunk, heading: str) -> None:
+ # The heading line stays with every occurrence whether or not the body
+ # is shared, so only the body's length and repeat count decide.
+ short_body = "See guide."
+ long_body = "Boilerplate paragraph about safety and usage. " * 20
+ short_members = [
+ make_chunk(heading=heading.format(index), text=f"{heading.format(index)}\n{short_body}")
+ for index in range(200)
+ ]
+ long_members = [
+ make_chunk(heading=f"## Big {index}", text=f"## Big {index}\n{long_body}") for index in range(60)
+ ]
+ cluster = ContentCluster([*short_members, *long_members], 0.95, 0.85, False, {"markdown"})
+ assert _request_chars(llm_analyzer._render_user_prompt(cluster)) > llm_analyzer.MAX_PROMPT_CHARS
+
+ prompt = build_user_prompt(cluster)
+
+ assert prompt.count("---SHARED CONTENT ") == 1
+ assert prompt.count(f"---SHARED CONTENT 1---\n{long_body}\n\n") == 1
+ for member in short_members:
+ assert f"Section: {member.heading}\nLines: 1-5\nContent:\n{member.text}\n\n" in prompt
+
def test_short_repeated_bodies_under_long_headings_stay_inline(self, make_chunk) -> None:
short_body = "See guide."
long_body = "Boilerplate paragraph about safety and usage. " * 20