Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(tier2): reject out-of-range or non-string dedup LLM verdict fields (#412)

Narendran Raghavan3cf34ec3f052
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f5142dbe..29d855e2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -474,6 +474,42 @@ All notable changes to SkillEvaluator are documented in this file.
confirm a guess for the masked part. Redaction now removes a digest that no
longer matches the reported value.
+- Tier 2 context deduplication now checks each chat-LLM verdict before it
+ becomes a finding. When present, the confidence must be a JSON number from 0
+ to 1, and the rationale and suggestion must be strings of at most 8,192
+ characters; a missing confidence still reads as 0, and a missing rationale or
+ suggestion as empty. In 0.8.0 any value that `float()` accepted was used: a
+ confidence of `85` or `true` made a HIGH finding, and the string `"NaN"` made
+ a MEDIUM finding whose `NaN` confidence then stopped the JSON and SARIF
+ reports from rendering. A list or object in a text field reached the finding
+ unchanged. Such a reply now counts as a failed analysis, so the scan is
+ reported as incomplete, like any other malformed reply. The checks apply to
+ every verdict, including `INTENTIONAL_DETAIL` and `RELATED_BUT_DISTINCT`,
+ which never become findings: a skill that passed on 0.8.0 with such a reply,
+ for example one with a `null` suggestion or a `"0.9"` string confidence, is
+ now reported as incomplete, and so is a `DUPLICATE` reply with a `"0.9"`
+ confidence, which made a HIGH finding in 0.8.0.
+
+- Tier 2 context deduplication now accepts more content chunks per skill. With
+ NVIDIA Build's default embedding model (`nvidia/nemotron-3-embed-1b`, 2,048
+ dimensions) and the OpenAI-compatible default
+ (`nvidia/nvidia/nemotron-3-embed-1b`, the same model) the limit is now 724
+ chunks, and with OpenAI's default `text-embedding-3-small` (1,536
+ dimensions) it is 836. All three were 512. In 0.8.0 a skill with more than
+ 512 chunks stopped the scan as incomplete with `chunk_count_limit`, and
+ `validate` exited 1 because Tier 2 gates by default. The chunk ceiling is now
+ 1,024 and the scalar-comparison budget 536,346,624, so embeddings of up to
+ 1,024 dimensions reach all 1,024 chunks. At 3,072 dimensions the limit is now
+ 591, and at 4,096 it is 512, up from 418 and 362. A skill above its width's
+ limit but within 1,024 chunks, such as 725 to 1,024 chunks with either
+ 2,048-dimensional default, now stops after the first embedding batch with
+ `scalar_comparison_limit` instead of before any embedding request with
+ `chunk_count_limit`. It is still reported as incomplete, with the same exit
+ code. The cluster member limit follows the chunk ceiling to 1,024. The most
+ clustering work one skill can need doubles with the budget, and the most
+ embedding input it can send doubles with the ceiling, from 512 to 1,024
+ texts (724 with either 2,048-dimensional default).
+
- Tier 3 HTML evaluator cards now say when a judge metric's lift left out cases
whose arms were scored by different judge models, for example after a judge
fallback in one arm. The arm scores keep those cases, so a card could show
diff --git a/docs/tier2-deduplication.mdx b/docs/tier2-deduplication.mdx
index 2fce8d78..f6c16fcd 100644
--- a/docs/tier2-deduplication.mdx
+++ b/docs/tier2-deduplication.mdx
@@ -212,6 +212,15 @@ no complete object or with more than one counts as a failed analysis. An object
nested inside an array or inside a malformed container is never used as the
verdict, and neither is an object with a repeated key, a `NaN` or `Infinity`
value, a number too large for a float, or more than 128 levels of nesting.
+The verdict must be one of the three names above. When present, the confidence
+must be a JSON number from 0 to 1, and the rationale and suggestion must be
+strings of at most 8,192 characters each. A missing confidence reads as 0, so a
+`DUPLICATE` verdict without one maps to MEDIUM in the table above, and a missing
+rationale or suggestion reads as empty. A reply that breaks any of these rules
+counts as a failed analysis rather than a finding, whatever its verdict: for
+example a confidence of `85` (a percentage), `true`, or `"0.9"` (a string), or
+a suggestion that is `null` or a list. This applies to `INTENTIONAL_DETAIL` and
+`RELATED_BUT_DISTINCT` replies too, although they never become findings.
If chat-LLM analysis fails for a candidate cluster, Tier 2 reports the
incomplete analysis as a blocking error rather than treating the content as
@@ -479,17 +488,21 @@ you intend to skip Tier 2 entirely.
| High-confidence duplicate boundary | `0.70` | Maps actionable duplicate verdicts to HIGH rather than MEDIUM |
| Frontmatter structure | 100 nesting levels, 20,000 nodes, 1,024 alias references, 1,024 items per mapping or sequence, 65,536 characters per value, integers up to 14,000 bits (about 4,200 digits) | Bounds every Markdown mapping frontmatter Tier 2 parses; aliases and merge keys count after expansion. A decimal integer longer than 4,300 digits, Python's default limit for converting text to an integer, counts as invalid YAML, not as a limit failure |
| Inter-skill name and description | 16,384 characters each | Per-field catalog text limit, in every embedding mode; description mode also caps `name: description` at 16,384 characters. A value that is not a string excludes the manifest and makes the scan incomplete |
-| Intra-skill content chunks | 512 | Per-skill ceiling, checked before any embedding request |
-| Intra-skill comparison work | 267,911,168 scalar comparisons | Bounds all-pairs clustering; allows all 512 chunks with embeddings of up to 2,048 dimensions, 418 chunks at 3,072, and 362 at 4,096 |
+| Intra-skill content chunks | 1,024 | Per-skill ceiling, checked before any embedding request |
+| Intra-skill comparison work | 536,346,624 scalar comparisons | Bounds all-pairs clustering; allows all 1,024 chunks with embeddings of up to 1,024 dimensions, 836 chunks at 1,536 (OpenAI's default `text-embedding-3-small`), 724 at 2,048 (NVIDIA Build's default `nvidia/nemotron-3-embed-1b` and the OpenAI-compatible default `nvidia/nvidia/nemotron-3-embed-1b`), 591 at 3,072, and 512 at 4,096 |
| Intra-skill LLM review clusters | 50 | Most candidate clusters sent for chat-LLM review in one skill |
-| Intra-skill cluster members | 512 | Same as the chunk ceiling, so it never stops a scan on its own; the review prompt limits decide |
+| Intra-skill cluster members | 1,024 | Same as the chunk ceiling, so it never stops a scan on its own; the review prompt limits decide |
| Intra-skill review prompt | 64,000 characters | One cluster's request, system prompt included; exactly repeated text (a Markdown section's body after its heading line) is listed once before this limit applies, when that shortens the request |
+| Intra-skill verdict text | 8,192 characters | Longest rationale or suggestion accepted in one chat-LLM verdict; a longer one fails that cluster's analysis |
| Intra-skill review prompts in total | 1,000,000 characters | All cluster requests for one skill, checked before the first chat-LLM call |
Intra-skill clustering compares every pair of chunks, so its work is
`n * (n - 1) / 2 * dimensions`. A skill over the chunk ceiling stops before any
embedding request. A skill within the ceiling whose embeddings are too wide for
its chunk count stops after the first embedding batch, which sets the width.
+With the 2,048-dimensional default model of NVIDIA Build or of the
+OpenAI-compatible provider, for example, a skill with 725 to 1,024 chunks stops
+there.
Either way the scan is reported as **INCOMPLETE**. For the comparison-work
limit (`scalar_comparison_limit`), the execution error and the JSON
`work_limit` record name the effective chunk limit for that embedding width
diff --git a/src/skillevaluator/constants.py b/src/skillevaluator/constants.py
index d7ecdf68..78313947 100644
--- a/src/skillevaluator/constants.py
+++ b/src/skillevaluator/constants.py
@@ -240,6 +240,9 @@ SIMILARITY_DEFAULT_MODEL = EMBEDDING_DEFAULT_NVIDIA
SIMILARITY_CHUNK_SIZE = 512 # tokens per chunk for full-body mode
SIMILARITY_CHUNK_OVERLAP = 64 # token overlap between chunks
+# Longest rationale or suggestion string accepted from one Tier 2 LLM verdict.
+TIER2_LLM_MAX_RESPONSE_SCALAR_CHARS = 8_192
+
# =============================================================================
# CONTEXT DEDUPLICATION CONSTANTS (Phase 1)
@@ -273,13 +276,13 @@ CONTENT_DEDUP_MAX_FILES = 256
CONTENT_DEDUP_MAX_DISCOVERED_PATHS = 4096
CONTENT_DEDUP_MAX_FILE_BYTES = 1 * 1024 * 1024
CONTENT_DEDUP_MAX_TOTAL_BYTES = 8 * 1024 * 1024
-CONTENT_DEDUP_MAX_CHUNKS = 512
+CONTENT_DEDUP_MAX_CHUNKS = 1024
CONTENT_DEDUP_MAX_PAIR_COMPARISONS = CONTENT_DEDUP_MAX_CHUNKS * (CONTENT_DEDUP_MAX_CHUNKS - 1) // 2
# All-pairs clustering does one multiply-add per vector component per chunk
-# pair. The budget lets embeddings of up to 2,048 dimensions reach the chunk
+# pair. The budget lets embeddings of up to 1,024 dimensions reach the chunk
# ceiling; wider embeddings get a lower effective chunk limit derived from it
-# (418 chunks at 3,072 dimensions, 362 at 4,096).
-CONTENT_DEDUP_SCALAR_BUDGET_DIMENSION = 2048
+# (836 chunks at 1,536 dimensions, 724 at 2,048, 591 at 3,072, 512 at 4,096).
+CONTENT_DEDUP_SCALAR_BUDGET_DIMENSION = 1024
CONTENT_DEDUP_MAX_SCALAR_COMPARISONS = CONTENT_DEDUP_MAX_PAIR_COMPARISONS * CONTENT_DEDUP_SCALAR_BUDGET_DIMENSION
CONTENT_DEDUP_MAX_LLM_CLUSTERS = 50
diff --git a/src/skillevaluator/deduplication/intra_skill/intra_skill_validator.py b/src/skillevaluator/deduplication/intra_skill/intra_skill_validator.py
index 2d24571a..b4356c5d 100644
--- a/src/skillevaluator/deduplication/intra_skill/intra_skill_validator.py
+++ b/src/skillevaluator/deduplication/intra_skill/intra_skill_validator.py
@@ -55,7 +55,7 @@ _CONFIG_KV_RE = re.compile(r"^[\w.\-/]+\s*[=:]\s*\S")
CONTENT_DEDUP_MAX_CLUSTER_MEMBERS = CONTENT_DEDUP_MAX_CHUNKS
CONTENT_DEDUP_MAX_TOTAL_LLM_PROMPT_CHARS = 1_000_000
# Most member locations and source files named in one finding message. The
-# rest are counted, so a finding for a 512-member cluster stays a few kilobytes.
+# rest are counted, so a finding for a 1,024-member cluster stays a few kilobytes.
_MAX_REPORTED_LOCATIONS = 64
# Scan label recorded when a work limit stops duplicate analysis early.
diff --git a/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py b/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py
index 9361ff5b..caad964a 100644
--- a/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py
+++ b/src/skillevaluator/deduplication/intra_skill/llm_analyzer.py
@@ -11,7 +11,7 @@ from __future__ import annotations
from collections import Counter
from typing import TYPE_CHECKING, Any
-from skillevaluator.inference import LLMClient, LLMClientError, LLMVerdict
+from skillevaluator.inference import LLMClient, LLMVerdict, parse_bounded_llm_verdict
from skillevaluator.models.result import Severity
if TYPE_CHECKING:
@@ -197,7 +197,13 @@ def _render_user_prompt(cluster: ContentCluster, shared: dict[str, int] | None =
def analyze_cluster(client: LLMClient, cluster: ContentCluster) -> LLMVerdict:
- """Run LLM analysis on a single content cluster."""
+ """Run LLM analysis on a single content cluster.
+
+ The reply is checked before it can become a finding: a verdict outside
+ ``VALID_VERDICTS``, a confidence that is not a number from 0 to 1, or a
+ rationale or suggestion that is not a bounded string raises
+ :class:`~skillevaluator.inference.LLMClientError`.
+ """
user_prompt = build_user_prompt(cluster)
data = client.extract_json_from_response(
SYSTEM_PROMPT,
@@ -205,17 +211,7 @@ def analyze_cluster(client: LLMClient, cluster: ContentCluster) -> LLMVerdict:
response_schema=VERDICT_JSON_SCHEMA,
schema_name="dedup_verdict",
)
-
- verdict = data.get("verdict", "")
- if verdict not in VALID_VERDICTS:
- raise LLMClientError(f"LLM returned unknown verdict '{verdict}'. Expected one of: {VALID_VERDICTS}")
-
- return LLMVerdict(
- verdict=verdict,
- confidence=float(data.get("confidence", 0.0)),
- reasoning=data.get("rationale", data.get("reasoning", "")),
- suggestion=data.get("suggestion", ""),
- )
+ return parse_bounded_llm_verdict(data, valid_verdicts=VALID_VERDICTS, context="Content deduplication LLM")
def verdict_to_severity(verdict: LLMVerdict) -> Severity:
diff --git a/src/skillevaluator/inference/__init__.py b/src/skillevaluator/inference/__init__.py
index 6f234e6d..dff17f5c 100644
--- a/src/skillevaluator/inference/__init__.py
+++ b/src/skillevaluator/inference/__init__.py
@@ -16,6 +16,8 @@ from skillevaluator.inference.types import (
LLMClientTruncatedError,
LLMConfigError,
LLMVerdict,
+ parse_bounded_llm_verdict,
+ require_bounded_llm_string,
)
__all__ = [
@@ -27,4 +29,6 @@ __all__ = [
"LLMClientTruncatedError",
"LLMConfigError",
"LLMVerdict",
+ "parse_bounded_llm_verdict",
+ "require_bounded_llm_string",
]
diff --git a/src/skillevaluator/inference/types.py b/src/skillevaluator/inference/types.py
index 640d54ff..a6da03f4 100644
--- a/src/skillevaluator/inference/types.py
+++ b/src/skillevaluator/inference/types.py
@@ -5,8 +5,12 @@
from __future__ import annotations
+import math
+from collections.abc import Collection, Mapping
from dataclasses import dataclass
+from skillevaluator.constants import TIER2_LLM_MAX_RESPONSE_SCALAR_CHARS
+
class LLMClientError(Exception):
"""Raised when an LLM operation fails (missing key, bad response, etc.)."""
@@ -42,3 +46,65 @@ class LLMVerdict:
confidence: float
reasoning: str
suggestion: str
+
+
+def require_bounded_llm_string(value: object, label: str, *, max_chars: int) -> str:
+ """Return ``value`` when it is a string of at most ``max_chars`` characters.
+
+ Nothing is coerced: a number, list, mapping or ``None`` is an error, so a
+ malformed reply never reaches a finding or a report as a non-string.
+ """
+ if type(value) is not str:
+ raise LLMClientError(f"{label} must be a string")
+ if len(value) > max_chars:
+ raise LLMClientError(f"{label} exceeds the {max_chars}-character limit")
+ return value
+
+
+def _require_unit_interval(value: object, label: str) -> float:
+ """Require a JSON number (not a bool or a string) that is finite and within [0, 1]."""
+ message = f"{label} must be a finite number within [0, 1]"
+ if not isinstance(value, (int, float)) or isinstance(value, bool):
+ raise LLMClientError(message)
+ try:
+ number = float(value)
+ except (OverflowError, TypeError, ValueError) as exc:
+ raise LLMClientError(message) from exc
+ if not math.isfinite(number) or not 0.0 <= number <= 1.0:
+ raise LLMClientError(message)
+ return number
+
+
+def parse_bounded_llm_verdict(data: object, *, valid_verdicts: Collection[str], context: str) -> LLMVerdict:
+ """Turn a parsed Tier 2 LLM reply into a verdict with checked, bounded fields.
+
+ The reply must be a JSON object whose ``verdict`` is one of
+ ``valid_verdicts`` and whose ``confidence`` is a number from 0 to 1. The
+ ``rationale`` (or a legacy ``reasoning``) and ``suggestion`` fields must be
+ strings of at most ``TIER2_LLM_MAX_RESPONSE_SCALAR_CHARS`` characters. A
+ missing confidence reads as 0 and a missing text field as empty. Any other
+ reply raises :class:`LLMClientError`.
+ """
+ if not isinstance(data, Mapping):
+ raise LLMClientError(f"{context} response must be a JSON object")
+
+ verdict = require_bounded_llm_string(
+ data.get("verdict", ""),
+ f"{context} verdict",
+ max_chars=TIER2_LLM_MAX_RESPONSE_SCALAR_CHARS,
+ )
+ if verdict not in valid_verdicts:
+ raise LLMClientError(f"LLM returned unknown verdict '{verdict}'. Expected one of: {set(valid_verdicts)}")
+
+ confidence = _require_unit_interval(data.get("confidence", 0.0), f"{context} confidence")
+ reasoning = require_bounded_llm_string(
+ data.get("rationale", data.get("reasoning", "")),
+ f"{context} rationale",
+ max_chars=TIER2_LLM_MAX_RESPONSE_SCALAR_CHARS,
+ )
+ suggestion = require_bounded_llm_string(
+ data.get("suggestion", ""),
+ f"{context} suggestion",
+ max_chars=TIER2_LLM_MAX_RESPONSE_SCALAR_CHARS,
+ )
+ return LLMVerdict(verdict=verdict, confidence=confidence, reasoning=reasoning, suggestion=suggestion)
diff --git a/tests/deduplication/intra_skill/test_intra_llm_analyzer.py b/tests/deduplication/intra_skill/test_intra_llm_analyzer.py
index afa08be6..f60a5553 100644
--- a/tests/deduplication/intra_skill/test_intra_llm_analyzer.py
+++ b/tests/deduplication/intra_skill/test_intra_llm_analyzer.py
@@ -493,6 +493,130 @@ class TestAnalyzeCluster:
assert result.suggestion == "The suggestion text"
assert result.confidence == 0.77
+ @pytest.mark.parametrize(
+ "confidence",
+ [float("nan"), float("inf"), -0.1, 1.1, 85, True, "0.5", "NaN", None, 10**400],
+ ids=[
+ "nan",
+ "inf",
+ "negative",
+ "above-one",
+ "percent",
+ "bool",
+ "numeric-string",
+ "nan-string",
+ "null",
+ "huge-int",
+ ],
+ )
+ def test_confidence_outside_zero_to_one_is_rejected(self, make_cluster, confidence: object) -> None:
+ mock_client = MagicMock(spec=LLMClient)
+ mock_client.extract_json_from_response.return_value = {
+ "verdict": "DUPLICATE",
+ "confidence": confidence,
+ "rationale": "Reason",
+ "suggestion": "Suggestion",
+ }
+
+ with pytest.raises(LLMClientError, match=r"confidence must be a finite number within \[0, 1\]"):
+ analyze_cluster(mock_client, make_cluster())
+
+ @pytest.mark.parametrize(("confidence", "expected"), [(0, 0.0), (1, 1.0), (0.0, 0.0), (1.0, 1.0)])
+ def test_confidence_bounds_are_accepted_as_floats(self, make_cluster, confidence: object, expected: float) -> None:
+ mock_client = MagicMock(spec=LLMClient)
+ mock_client.extract_json_from_response.return_value = {
+ "verdict": "DUPLICATE",
+ "confidence": confidence,
+ "rationale": "Reason",
+ "suggestion": "Suggestion",
+ }
+
+ result = analyze_cluster(mock_client, make_cluster())
+
+ assert type(result.confidence) is float
+ assert result.confidence == expected
+
+ @pytest.mark.parametrize(
+ ("field", "value", "message"),
+ [
+ ("verdict", ["DUPLICATE"], "verdict must be a string"),
+ ("verdict", None, "verdict must be a string"),
+ ("rationale", ["Reason"], "rationale must be a string"),
+ ("rationale", None, "rationale must be a string"),
+ ("suggestion", {"text": "Suggestion"}, "suggestion must be a string"),
+ ("suggestion", 3, "suggestion must be a string"),
+ ("rationale", "x" * 8_193, "rationale exceeds the 8192-character limit"),
+ ("suggestion", "x" * 8_193, "suggestion exceeds the 8192-character limit"),
+ ],
+ ids=[
+ "verdict-list",
+ "verdict-null",
+ "rationale-list",
+ "rationale-null",
+ "suggestion-object",
+ "suggestion-number",
+ "rationale-too-long",
+ "suggestion-too-long",
+ ],
+ )
+ def test_non_string_or_oversized_reply_fields_are_rejected(
+ self, make_cluster, field: str, value: object, message: str
+ ) -> None:
+ mock_client = MagicMock(spec=LLMClient)
+ reply: dict[str, object] = {
+ "verdict": "DUPLICATE",
+ "confidence": 0.9,
+ "rationale": "Reason",
+ "suggestion": "Suggestion",
+ }
+ reply[field] = value
+ mock_client.extract_json_from_response.return_value = reply
+
+ with pytest.raises(LLMClientError, match=message):
+ analyze_cluster(mock_client, make_cluster())
+
+ def test_legacy_reasoning_field_is_bounded_too(self, make_cluster) -> None:
+ mock_client = MagicMock(spec=LLMClient)
+ mock_client.extract_json_from_response.return_value = {
+ "verdict": "DUPLICATE",
+ "confidence": 0.9,
+ "reasoning": {"text": "Reason"},
+ "suggestion": "Suggestion",
+ }
+
+ with pytest.raises(LLMClientError, match="rationale must be a string"):
+ analyze_cluster(mock_client, make_cluster())
+
+ def test_reply_fields_at_the_length_limit_are_kept_whole(self, make_cluster) -> None:
+ mock_client = MagicMock(spec=LLMClient)
+ mock_client.extract_json_from_response.return_value = {
+ "verdict": "DUPLICATE",
+ "confidence": 0.9,
+ "rationale": "r" * 8_192,
+ "suggestion": "s" * 8_192,
+ }
+
+ result = analyze_cluster(mock_client, make_cluster())
+
+ assert result.reasoning == "r" * 8_192
+ assert result.suggestion == "s" * 8_192
+
+ def test_missing_confidence_and_text_fields_keep_their_defaults(self, make_cluster) -> None:
+ mock_client = MagicMock(spec=LLMClient)
+ mock_client.extract_json_from_response.return_value = {"verdict": "RELATED_BUT_DISTINCT"}
+
+ result = analyze_cluster(mock_client, make_cluster())
+
+ assert result == LLMVerdict("RELATED_BUT_DISTINCT", 0.0, "", "")
+
+ @pytest.mark.parametrize("reply", [["DUPLICATE"], "DUPLICATE", 0.9, None], ids=["list", "string", "number", "null"])
+ def test_reply_that_is_not_a_json_object_is_rejected(self, make_cluster, reply: object) -> None:
+ mock_client = MagicMock(spec=LLMClient)
+ mock_client.extract_json_from_response.return_value = reply
+
+ with pytest.raises(LLMClientError, match="response must be a JSON object"):
+ analyze_cluster(mock_client, make_cluster())
+
class TestVerdictToSeverity:
def test_duplicate_high_confidence(self) -> None:
diff --git a/tests/deduplication/intra_skill/test_intra_skill_validator.py b/tests/deduplication/intra_skill/test_intra_skill_validator.py
index 4cdf2692..26dc9064 100644
--- a/tests/deduplication/intra_skill/test_intra_skill_validator.py
+++ b/tests/deduplication/intra_skill/test_intra_skill_validator.py
@@ -5,6 +5,7 @@
from __future__ import annotations
+import json
import logging
import math
from pathlib import Path
@@ -21,6 +22,7 @@ from skillevaluator.deduplication.utils.chunker import ContentChunk
from skillevaluator.embedding.client import MAX_EMBEDDING_VECTOR_DIMENSION, SimilarityConfigError
from skillevaluator.inference import LLMClientError, LLMClientRefusedError, LLMVerdict
from skillevaluator.models.result import Severity
+from skillevaluator.reporting.json_reporter import JSONReporter
def _numbered_sections(count: int) -> str:
@@ -65,10 +67,11 @@ class TestIntraSkillValidatorProperties:
class TestEffectiveChunkLimit:
- def test_scalar_budget_lets_2048_dimensional_embeddings_reach_the_chunk_ceiling(self) -> None:
- assert constants.CONTENT_DEDUP_MAX_CHUNKS == 512
- assert constants.CONTENT_DEDUP_MAX_PAIR_COMPARISONS == 512 * 511 // 2
- assert constants.CONTENT_DEDUP_MAX_SCALAR_COMPARISONS == 267_911_168
+ def test_scalar_budget_lets_1024_dimensional_embeddings_reach_the_chunk_ceiling(self) -> None:
+ assert constants.CONTENT_DEDUP_MAX_CHUNKS == 1024
+ assert constants.CONTENT_DEDUP_MAX_PAIR_COMPARISONS == 1024 * 1023 // 2
+ assert constants.CONTENT_DEDUP_SCALAR_BUDGET_DIMENSION == 1024
+ assert constants.CONTENT_DEDUP_MAX_SCALAR_COMPARISONS == 536_346_624
assert (
constants.CONTENT_DEDUP_MAX_SCALAR_COMPARISONS
== constants.CONTENT_DEDUP_MAX_PAIR_COMPARISONS * constants.CONTENT_DEDUP_SCALAR_BUDGET_DIMENSION
@@ -76,11 +79,58 @@ class TestEffectiveChunkLimit:
@pytest.mark.parametrize(
("vector_dimension", "expected_limit"),
- [(1, 512), (1024, 512), (1536, 512), (2048, 512), (2049, 511), (3072, 418), (4096, 362)],
+ [
+ (1, 1024),
+ (768, 1024),
+ (1024, 1024),
+ (1025, 1023),
+ (1536, 836),
+ (2048, 724),
+ (3072, 591),
+ (4096, 512),
+ ],
)
def test_limit_depends_on_vector_dimension(self, vector_dimension: int, expected_limit: int) -> None:
assert effective_chunk_limit(vector_dimension) == expected_limit
+ @pytest.mark.parametrize(
+ "environ",
+ [
+ pytest.param({"SKILL_EVAL_EMBEDDING_PROVIDER": "openai", "OPENAI_API_KEY": "test-key"}, id="openai"),
+ pytest.param({"SKILL_EVAL_EMBEDDING_PROVIDER": "nv_build", "NVIDIA_API_KEY": "nvapi-test"}, id="nv-build"),
+ pytest.param(
+ {
+ "SKILL_EVAL_EMBEDDING_PROVIDER": "openai-compatible",
+ "SKILL_EVAL_EMBEDDING_API_KEY": "test-key",
+ "SKILL_EVAL_EMBEDDING_BASE_URL": "https://gateway.example/v1",
+ },
+ id="openai-compatible",
+ ),
+ ],
+ )
+ def test_docs_give_the_chunk_limit_of_each_default_embedding_model(self, environ: dict[str, str]) -> None:
+ """The limits table names the limit a user of a default model gets, not only the ceiling."""
+ from skillevaluator.provider_config import resolve_embedding_provider
+
+ # Native output width of each default model; the embedding client requests no other width.
+ default_model_widths = {
+ "text-embedding-3-small": 1536,
+ "nvidia/nemotron-3-embed-1b": 2048,
+ "nvidia/nvidia/nemotron-3-embed-1b": 2048,
+ }
+ model = resolve_embedding_provider(environ).model
+ vector_dimension = default_model_widths[model]
+ guide = Path(__file__).resolve().parents[3] / "docs" / "tier2-deduplication.mdx"
+ (row,) = [
+ line
+ for line in guide.read_text(encoding="utf-8").splitlines()
+ if line.startswith("| Intra-skill comparison work |")
+ ]
+
+ (width_entry,) = [entry for entry in row.split(", ") if f"`{model}`" in entry]
+ assert width_entry.startswith(f"{effective_chunk_limit(vector_dimension):,} ")
+ assert f" at {vector_dimension:,} (" in width_entry
+
def test_limit_is_the_largest_chunk_count_within_budget_for_every_width(self) -> None:
budget = constants.CONTENT_DEDUP_MAX_SCALAR_COMPARISONS
for vector_dimension in range(1, MAX_EMBEDDING_VECTOR_DIMENSION + 1):
@@ -296,22 +346,24 @@ class TestIntraSkillValidatorValidate:
@patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.build_clusters")
@patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")
- def test_chunk_ceiling_is_reachable_with_2048_dimensional_embeddings(
+ def test_1024_chunks_complete_in_sixteen_embedding_batches(
self, mock_embed, mock_build_clusters, tmp_path: Path
) -> None:
skill_dir = tmp_path / "boundary-skill"
skill_dir.mkdir()
- (skill_dir / "SKILL.md").write_text(_numbered_sections(512))
- mock_embed.return_value.embed.side_effect = _unit_embeddings(2048)
+ (skill_dir / "SKILL.md").write_text(_numbered_sections(1024), encoding="utf-8")
+ mock_embed.return_value.embed.side_effect = _unit_embeddings(1024)
mock_build_clusters.return_value = []
result = IntraSkillValidator().validate(skill_dir)
assert result.passed is True
+ assert result.status == "passed"
assert result.findings == []
- assert len(mock_build_clusters.call_args.args[0]) == 512
+ assert [len(call.args[0]) for call in mock_embed.return_value.embed.call_args_list] == [64] * 16
+ assert len(mock_build_clusters.call_args.args[0]) == 1024
- @pytest.mark.parametrize(("vector_dimension", "chunk_limit"), [(3072, 418), (4096, 362)])
+ @pytest.mark.parametrize(("vector_dimension", "chunk_limit"), [(1536, 836), (2048, 724), (3072, 591), (4096, 512)])
@patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.build_clusters")
@patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")
def test_wider_embeddings_report_their_effective_chunk_limit(
@@ -327,13 +379,15 @@ class TestIntraSkillValidatorValidate:
mock_embed.return_value.embed.side_effect = _unit_embeddings(vector_dimension)
mock_build_clusters.return_value = []
- (skill_dir / "SKILL.md").write_text(_numbered_sections(chunk_limit))
+ (skill_dir / "SKILL.md").write_text(_numbered_sections(chunk_limit), encoding="utf-8")
assert IntraSkillValidator().validate(skill_dir).passed is True
+ assert len(mock_build_clusters.call_args.args[0]) == chunk_limit
chunk_count = chunk_limit + 1
pair_count = chunk_count * (chunk_count - 1) // 2
- (skill_dir / "SKILL.md").write_text(_numbered_sections(chunk_count))
+ (skill_dir / "SKILL.md").write_text(_numbered_sections(chunk_count), encoding="utf-8")
mock_build_clusters.reset_mock()
+ mock_embed.return_value.embed.reset_mock()
result = IntraSkillValidator().validate(skill_dir)
self._assert_work_limit_incomplete(
@@ -343,7 +397,7 @@ class TestIntraSkillValidatorValidate:
"pair_count": pair_count,
"vector_dimension": vector_dimension,
"scalar_work": pair_count * vector_dimension,
- "limit": 267_911_168,
+ "limit": 536_346_624,
"effective_chunk_limit": chunk_limit,
"chunk_count": chunk_count,
},
@@ -352,6 +406,37 @@ class TestIntraSkillValidatorValidate:
f"effective chunk limit is {chunk_limit} for {vector_dimension}-dimensional embeddings; "
f"this skill has {chunk_count} chunks."
) in result.errors[0]
+ # The first batch sets the width; no further batch is requested.
+ assert mock_embed.return_value.embed.call_count == 1
+ mock_build_clusters.assert_not_called()
+
+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.build_clusters")
+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")
+ def test_2048_dimensional_skill_at_the_chunk_ceiling_stops_after_the_first_batch(
+ self, mock_embed, mock_build_clusters, tmp_path: Path
+ ) -> None:
+ """Between the width's limit and the ceiling, the scan stops on comparison work, not chunk count."""
+ skill_dir = tmp_path / "default-width-skill"
+ skill_dir.mkdir()
+ (skill_dir / "SKILL.md").write_text(_numbered_sections(1024), encoding="utf-8")
+ mock_embed.return_value.embed.side_effect = _unit_embeddings(2048)
+
+ result = IntraSkillValidator().validate(skill_dir)
+
+ pair_count = 1024 * 1023 // 2
+ self._assert_work_limit_incomplete(
+ result,
+ {
+ "check_name": "scalar_comparison_limit",
+ "pair_count": pair_count,
+ "vector_dimension": 2048,
+ "scalar_work": pair_count * 2048,
+ "limit": 536_346_624,
+ "effective_chunk_limit": 724,
+ "chunk_count": 1024,
+ },
+ )
+ assert [len(call.args[0]) for call in mock_embed.return_value.embed.call_args_list] == [64]
mock_build_clusters.assert_not_called()
@patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.build_clusters")
@@ -385,14 +470,15 @@ class TestIntraSkillValidatorValidate:
def test_absolute_chunk_ceiling_fails_before_embedding(self, mock_embed, tmp_path: Path) -> None:
skill_dir = tmp_path / "many-headings"
skill_dir.mkdir()
- (skill_dir / "SKILL.md").write_text(_numbered_sections(513))
+ (skill_dir / "SKILL.md").write_text(_numbered_sections(1025), encoding="utf-8")
result = IntraSkillValidator().validate(skill_dir)
self._assert_work_limit_incomplete(
result,
- {"check_name": "chunk_count_limit", "file_path": "SKILL.md", "actual": 513, "limit": 512},
+ {"check_name": "chunk_count_limit", "file_path": "SKILL.md", "actual": 1025, "limit": 1024},
)
+ assert "more than 1024 content chunks" in result.errors[0]
mock_embed.assert_not_called()
@patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.LLMClient")
@@ -478,7 +564,7 @@ class TestIntraSkillValidatorValidate:
mock_llm.assert_not_called()
def test_cluster_member_limit_is_the_chunk_ceiling(self) -> None:
- assert intra_skill_validator.CONTENT_DEDUP_MAX_CLUSTER_MEMBERS == constants.CONTENT_DEDUP_MAX_CHUNKS == 512
+ assert intra_skill_validator.CONTENT_DEDUP_MAX_CLUSTER_MEMBERS == constants.CONTENT_DEDUP_MAX_CHUNKS == 1024
assert llm_analyzer.MAX_PROMPT_CHARS == 64_000
# The validator reads llm_analyzer.MAX_PROMPT_CHARS; a stale copy here
# would let a patched budget move one threshold and not the other.
@@ -641,7 +727,7 @@ class TestIntraSkillValidatorValidate:
@pytest.mark.parametrize(
("member_count", "text", "check_name"),
[
- pytest.param(513, "content " * 10, "llm_cluster_member_limit", id="member-limit"),
+ pytest.param(1025, "content " * 10, "llm_cluster_member_limit", id="member-limit"),
pytest.param(500, "x" * 80, "llm_prompt_size_limit", id="prompt-limit"),
],
)
@@ -670,7 +756,7 @@ class TestIntraSkillValidatorValidate:
cluster = ContentCluster(members, 1.0, 1.0, False, {"markdown"})
mock_clusters.return_value = [cluster]
if check_name == "llm_cluster_member_limit":
- actual, limit = member_count, 512
+ actual, limit = member_count, 1024
else:
actual, limit = len(SYSTEM_PROMPT) + len(build_user_prompt(cluster)), 64_000
assert "SHARED CONTENT" not in build_user_prompt(cluster)
@@ -1027,6 +1113,94 @@ class TestIntraSkillValidatorValidate:
assert "private provider data" not in str(result.errors)
assert "private provider data" not in caplog.text
+ @pytest.mark.parametrize("confidence", ["NaN", "Infinity", 85, True])
+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.LLMClient")
+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")
+ def test_out_of_range_confidence_is_incomplete_without_skill_finding(
+ self, mock_embed, mock_llm, tmp_path: Path, confidence: object
+ ) -> None:
+ skill_dir = tmp_path / "skill"
+ skill_dir.mkdir()
+ (skill_dir / "SKILL.md").write_text(
+ "## Section A\n" + "a" * 200 + "\n## Section B\n" + "b" * 200, encoding="utf-8"
+ )
+ mock_embed.return_value.embed.return_value = [[1.0, 0.0], [1.0, 0.0]]
+ mock_llm.return_value.extract_json_from_response.return_value = {
+ "verdict": "DUPLICATE",
+ "confidence": confidence,
+ "rationale": "Same content",
+ "suggestion": "Remove one",
+ }
+
+ result = IntraSkillValidator().validate(skill_dir)
+
+ assert mock_llm.return_value.extract_json_from_response.call_count == 1
+ assert result.status == "incomplete"
+ assert result.incomplete_scans == ["deduplication-llm"]
+ assert result.findings == []
+ assert result.metadata["llm_analysis"]["clusters_failed"] == 1
+ assert "LLM configuration or response was invalid" in result.errors[0]
+ # Strict JSON (no NaN or Infinity) still renders for the incomplete result.
+ json.loads(JSONReporter().render(result))
+
+ @pytest.mark.parametrize(
+ "reply",
+ [
+ {"verdict": "INTENTIONAL_DETAIL", "confidence": 0.9, "rationale": "Summary vs detail", "suggestion": None},
+ {"verdict": "RELATED_BUT_DISTINCT", "confidence": "0.9", "rationale": "Different", "suggestion": "Keep"},
+ {"verdict": "DUPLICATE", "confidence": "0.9", "rationale": "Same content", "suggestion": "Remove one"},
+ ],
+ ids=["intentional-null-suggestion", "related-string-confidence", "duplicate-string-confidence"],
+ )
+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.LLMClient")
+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")
+ def test_malformed_reply_field_is_incomplete_whatever_the_verdict(
+ self, mock_embed, mock_llm, tmp_path: Path, reply: dict[str, object]
+ ) -> None:
+ # A non-finding verdict passed before these checks, and a "0.9"
+ # DUPLICATE confidence made a HIGH finding; both now fail the analysis.
+ skill_dir = tmp_path / "skill"
+ skill_dir.mkdir()
+ (skill_dir / "SKILL.md").write_text(
+ "## Section A\n" + "a" * 200 + "\n## Section B\n" + "b" * 200, encoding="utf-8"
+ )
+ mock_embed.return_value.embed.return_value = [[1.0, 0.0], [1.0, 0.0]]
+ mock_llm.return_value.extract_json_from_response.return_value = reply
+
+ result = IntraSkillValidator().validate(skill_dir)
+
+ assert result.passed is False
+ assert result.status == "incomplete"
+ assert result.incomplete_scans == ["deduplication-llm"]
+ assert result.findings == []
+ assert result.metadata["llm_analysis"]["clusters_failed"] == 1
+ assert "LLM configuration or response was invalid" in result.errors[0]
+
+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.LLMClient")
+ @patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")
+ def test_duplicate_without_confidence_is_medium_advisory_finding(
+ self, mock_embed, mock_llm, tmp_path: Path
+ ) -> None:
+ skill_dir = tmp_path / "skill"
+ skill_dir.mkdir()
+ (skill_dir / "SKILL.md").write_text(
+ "## Section A\n" + "a" * 200 + "\n## Section B\n" + "b" * 200, encoding="utf-8"
+ )
+ mock_embed.return_value.embed.return_value = [[1.0, 0.0], [1.0, 0.0]]
+ mock_llm.return_value.extract_json_from_response.return_value = {
+ "verdict": "DUPLICATE",
+ "rationale": "Same content",
+ "suggestion": "Remove one",
+ }
+
+ result = IntraSkillValidator().validate(skill_dir)
+
+ assert result.passed is True
+ assert result.incomplete_scans == []
+ assert result.errors == []
+ assert [(f.check_name, f.severity) for f in result.findings] == [("duplicate", Severity.MEDIUM)]
+ assert result.findings[0].metadata["confidence"] == 0.0
+
@patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.analyze_cluster")
@patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.LLMClient")
@patch("skillevaluator.deduplication.intra_skill.intra_skill_validator.EmbeddingClient")
diff --git a/tests/deduplication/intra_skill/test_semantic_clustering.py b/tests/deduplication/intra_skill/test_semantic_clustering.py
index 346eb602..d4851eca 100644
--- a/tests/deduplication/intra_skill/test_semantic_clustering.py
+++ b/tests/deduplication/intra_skill/test_semantic_clustering.py
@@ -137,3 +137,36 @@ class TestBuildClusters:
for index in range(chunk_count)
]
assert build_clusters(chunks) == []
+
+ def test_accepts_the_1024_chunk_ceiling(self, make_chunk) -> None:
+ # One-dimensional vectors keep the 523,776 pair comparisons cheap.
+ chunks = [
+ make_chunk(source_file=f"chunk-{index}.md", heading=f"## Chunk {index}", embedding=[1.0])
+ for index in range(1024)
+ ]
+
+ [cluster] = build_clusters(chunks)
+
+ assert len(cluster.members) == 1024
+ assert cluster.max_similarity == 1.0
+
+ def test_rejects_one_chunk_over_the_ceiling(self, make_chunk) -> None:
+ chunks = [make_chunk(heading=f"## Chunk {index}", embedding=[1.0]) for index in range(1025)]
+
+ with pytest.raises(ValueError, match="Content chunk count exceeds the configured limit"):
+ build_clusters(chunks)
+
+ @pytest.mark.parametrize(("dimension", "chunk_limit"), [(2048, 724), (4096, 512)])
+ def test_rejects_scalar_work_one_chunk_past_the_width_limit(
+ self, make_chunk, dimension: int, chunk_limit: int
+ ) -> None:
+ # The guard reads the width from the first vector and raises before any
+ # comparison, so one shared vector is enough.
+ embedding = [1.0] + [0.0] * (dimension - 1)
+ chunk_count = chunk_limit + 1
+ assert chunk_limit * (chunk_limit - 1) // 2 * dimension <= constants.CONTENT_DEDUP_MAX_SCALAR_COMPARISONS
+ assert chunk_count * chunk_limit // 2 * dimension > constants.CONTENT_DEDUP_MAX_SCALAR_COMPARISONS
+ chunks = [make_chunk(heading=f"## Chunk {index}", embedding=embedding) for index in range(chunk_count)]
+
+ with pytest.raises(ValueError, match="Content scalar comparison work exceeds the configured limit"):
+ build_clusters(chunks)