Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): keep LLM suggestions grounded, distinct and requirement-preserving (#380)
Christopher Kevin1a40f520b883
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6832abb9..90b88c41 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -109,6 +109,15 @@ All notable changes to SkillEvaluator are documented in this file.### Changed+- Tier 3 LLM suggestions and insight recommendations now keep only evidence+ references the run recorded (cited references that don't resolve are+ dropped rather than shown as evidence) and keep one copy of a repeated+ suggestion. Insight recommendations that restate a deterministic one, or+ that loosen, relax, or lower an expectation, assertion, ground truth, or+ pass threshold, are dropped. Both prompts ask for distinct actions and+ label untested causes, such as switching to a different model, as+ hypotheses or experiments.+- Tier 3 reports missing skill runtime dependencies as environment problems,with declaration and custom Dockerfile guidance. Unrecovered environmentfailures leave the trial UNSCORED instead of reducing the skill's score;diff --git a/docs/reports.mdx b/docs/reports.mdxindex e0846849..e305475a 100644--- a/docs/reports.mdx+++ b/docs/reports.mdx@@ -1089,7 +1089,10 @@ Both paths speak the same dialect: standalone `tier3 evaluate` and`validate --tier3` share one HTML renderer for `report.html`, and both embedthe Tier 3 payload at schema version `2.0`. In that payload, conclusions andrecommendations written by the LLM carry `source: "llm"` and name the model-that wrote them as `model`. For a Claude 5.5 or later model on the native+that wrote them as `model`. LLM recommendations that restate a deterministic+one, or that loosen an expectation, assertion, ground truth, or pass threshold,+are dropped, and LLM suggestions keep only evidence references the run+recorded. For a Claude 5.5 or later model on the nativeClaude API this is the model the reply names, which can be the server-siderefusal fallback model rather than the configured model. Otherwise it is theconfigured model, unless a server-side `fallback` block in the reply namesdiff --git a/src/skillevaluator/evaluation/insights_judge.py b/src/skillevaluator/evaluation/insights_judge.pyindex 1cad3492..5b705559 100644--- a/src/skillevaluator/evaluation/insights_judge.py+++ b/src/skillevaluator/evaluation/insights_judge.py@@ -67,6 +67,29 @@ _NEGATIVE_CONTROL_SEMANTIC_PATTERNS = ()+_REQUIREMENT_NOUN = (+ r"(?:expectations?|expected(?:\s+(?:output|answer|result))?|assertions?|ground[\s_-]*truth|"+ r"pass[\s_-]*thresholds?|thresholds?|requirements?|criteria)"+)+_WEAKENING_VERB = (+ r"(?:loosen(?:s|ed|ing)?|relax(?:es|ed|ing)?|weaken(?:s|ed|ing)?|lower(?:s|ed|ing)?|soften(?:s|ed|ing)?)"+)+_REQUIREMENT_DETERMINER = (+ r"(?:the|a|an|my|your|our|their|its|this|that|these|those|any|each|every|all|both|"+ r"[A-Za-z0-9_-]+(?:'s|\u2019s))"+)+_WEAKENING_PATTERN = re.compile(+ rf"\b{_WEAKENING_VERB}\b(?:\s+{_REQUIREMENT_DETERMINER}){{0,3}}\s+{_REQUIREMENT_NOUN}\b",+ re.IGNORECASE,+)+_PASSIVE_WEAKENING_PATTERN = re.compile(+ rf"\b{_REQUIREMENT_NOUN}\b[\w\s,'\u2019_-]{{0,48}}\b"+ rf"(?:is|are|was|were|be|been|being|should\s+be|must\s+be|ought\s+to\s+be|will\s+be)\s+"+ rf"{_WEAKENING_VERB}\b",+ re.IGNORECASE,+)++_SYSTEM_PROMPT = """\You are an expert Skill Quality Reviewer summarising a Tier 3 (live agent)evaluation of an Agent Skill.@@ -128,6 +151,14 @@ an error message, or a comparison).RECOMMENDATIONS (up to {max_recommendations}): concrete, actionable nextsteps for the skill author. Each must be one short imperative sentence.Tag each with a category from this set: {categories}.+- Make each recommendation a distinct action; do not restate a+ deterministic recommendation or another recommendation in other words.+- Never recommend loosening, relaxing, or lowering an expectation,+ assertion, ground truth, or pass threshold so that a failing agent passes.+- Recommend a different agent or model only when the supplied trials compare+ agents; otherwise phrase it as an experiment to try ("Try ... to test+ whether ..."). Phrase any cause the evidence does not establish the same+ way.SEVERITY: tag every conclusion and recommendation with one of:- \"pass\" - positive observation / low-effort recommendation@@ -439,6 +470,20 @@ def _text_case_ids(text: str, known_case_ids: set[str]) -> set[str]:return references+def _weakens_requirements(text: str) -> bool:+ for pattern in (_WEAKENING_PATTERN, _PASSIVE_WEAKENING_PATTERN):+ for match in pattern.finditer(text):+ prefix = text[max(0, match.start() - 48) : match.start()]+ if re.search(r"\b(?:do\s+not|don't|never|avoid)(?:\s+\w+){0,2}\s*$", prefix, re.IGNORECASE):+ continue+ return True+ return False+++def _normalized_text(text: object) -> str:+ return " ".join(str(text or "").lower().split())++def _has_negative_control_semantics(text: str) -> bool:return any(pattern.search(text) for pattern in _NEGATIVE_CONTROL_SEMANTIC_PATTERNS)@@ -562,6 +607,8 @@ class InsightsJudge(LLMClient):})+ deterministic = kwargs.get("deterministic") if isinstance(kwargs.get("deterministic"), dict) else {}+ seen_messages = {_normalized_text(text) for text in deterministic.get("suggestions") or []}recommendations: list[dict] = []for item in (data.get("recommendations") or [])[:INSIGHTS_JUDGE_MAX_RECOMMENDATIONS]:if not isinstance(item, dict):@@ -572,6 +619,11 @@ class InsightsJudge(LLMClient):message = (item.get("message") or "").strip()if not title and not message:continue+ if _weakens_requirements(title) or _weakens_requirements(message):+ continue+ if (key := _normalized_text(message or title)) in seen_messages:+ continue+ seen_messages.add(key)recommendations.append({"claim_type": item["claim_type"],diff --git a/src/skillevaluator/tier3/harbor/report.py b/src/skillevaluator/tier3/harbor/report.pyindex 13bcbb2a..3d4393a0 100644--- a/src/skillevaluator/tier3/harbor/report.py+++ b/src/skillevaluator/tier3/harbor/report.py@@ -244,6 +244,14 @@ def _metric_label(metric: str) -> str:)+def _normalize_evidence_refs(raw_refs: Any) -> list[Any]:+ if isinstance(raw_refs, list):+ return raw_refs+ if isinstance(raw_refs, dict | str):+ return [raw_refs]+ return []++def _extract_findings(rewards: list[dict[str, Any]],*,@@ -295,11 +303,7 @@ def _extract_findings(for t in trials:d = t["detail"]if isinstance(d, dict):- raw_refs = d.get("evidence_refs")- if isinstance(raw_refs, list):- metric_refs.extend(raw_refs)- elif isinstance(raw_refs, dict | str):- metric_refs.append(raw_refs)+ metric_refs.extend(_normalize_evidence_refs(d.get("evidence_refs")))_seen: set[tuple[Any, ...]] = set()_refs: list[dict[str, Any]] = []for raw_ref in metric_refs:@@ -590,13 +594,14 @@ def _build_evidence_ref_lookup(rewards: list[dict[str, Any]]) -> dict[str, dict[metric_detail = _detail_for_finding(reward, metric)if not isinstance(metric_detail, dict):continue- for ref in metric_detail.get("evidence_refs") or []:- if not isinstance(ref, dict):+ for ref in _normalize_evidence_refs(metric_detail.get("evidence_refs")):+ if not isinstance(ref, dict | str):continue- if ref.get("source") or evidence_ref_identity(ref):- key = _compact_evidence_ref(ref)+ resolved = _resolve_evidence_ref(ref, {})+ if resolved.get("source") or evidence_ref_identity(resolved):+ key = _compact_evidence_ref(resolved)if key not in lookup:- lookup[key] = ref+ lookup[key] = resolvedreturn lookup@@ -631,6 +636,18 @@ def _resolve_evidence_ref(ref: Any, lookup: dict[str, dict[str, Any]]) -> dict[sreturn _bounded_evidence_ref({"source": source, "json_pointer": pointer, "kind": "evidence"})+def _recorded_evidence_refs(raw_refs: list[Any], lookup: dict[str, dict[str, Any]]) -> list[dict[str, Any]]:+ """Resolve LLM-cited references, keeping only those the run recorded, once each."""+ resolved: list[dict[str, Any]] = []+ seen: set[str] = set()+ for ref in raw_refs:+ key = _compact_evidence_ref(ref) if isinstance(ref, dict) else str(ref)[:1024]+ if key in lookup and key not in seen:+ seen.add(key)+ resolved.append(_resolve_evidence_ref(key, lookup))+ return resolved++def _generate_suggestions_structured(skill_name: str,findings: list[dict[str, Any]],@@ -646,8 +663,9 @@ def _generate_suggestions_structured((cheap, reliable). After parsing the LLM response each string ref isresolved against a lookup built from ``rewards[*]["details"][metric]["evidence_refs"]``so that the richer dict fields (``kind``, ``path``, ``excerpt``) are- preserved. Unresolvable strings fall back to a minimal dict with- ``kind="evidence"``. Callers that consume ``suggestions_v2[*].evidence_refs``+ preserved. References the run did not record are dropped, so a+ suggestion never cites evidence that does not exist, and repeated+ suggestions are kept once. Callers that consume ``suggestions_v2[*].evidence_refs``(including the SkillEvaluator report template) therefore always receive dicts ratherthan plain strings."""@@ -715,6 +733,10 @@ Based on these results, provide exactly 3-4 specific, actionable suggestions forKeep each suggestion to 1-2 sentences. Be concrete — reference specific files, commands, or sections.If the failures are purely environmental (e.g. CLI not installed in container), say so directly.+Give distinct suggestions; do not repeat one fix in different words.+Do not suggest loosening, removing, or rewriting an eval expectation just so the agent passes; change a case only when the evidence shows the expectation itself is wrong.+When the evidence does not establish a cause (for example, that a different model would do better), start the suggestion with "Hypothesis:" and say how to test it.+Cite only evidence references listed above; references the run did not record are dropped.Respond with ONLY a JSON array of objects {{"suggestion": "...", "dimension": "<metric>", "evidence_refs": ["trajectory.json#/steps/N", ...]}}, no other text."""@@ -732,25 +754,26 @@ Respond with ONLY a JSON array of objects {{"suggestion": "...", "dimension": "<parsed = _extract_json(content) if content else Noneif isinstance(parsed, list):result: list[dict[str, Any]] = []+ seen: set[str] = set()for item in parsed[:4]:if isinstance(item, dict):+ text = str(item.get("suggestion", ""))[:_MAX_FINDING_TEXT_CHARS]raw_refs = list(item.get("evidence_refs") or [])[:_MAX_FINDING_EVIDENCE_REFS]- resolved_refs = [_resolve_evidence_ref(r, ref_lookup) for r in raw_refs]- result.append(- {- "suggestion": str(item.get("suggestion", ""))[:_MAX_FINDING_TEXT_CHARS],- "dimension": str(item.get("dimension", ""))[:256],- "evidence_refs": resolved_refs,- }- )+ entry = {+ "suggestion": text,+ "dimension": str(item.get("dimension", ""))[:256],+ "evidence_refs": _recorded_evidence_refs(raw_refs, ref_lookup),+ }elif isinstance(item, str):- result.append(- {- "suggestion": item[:_MAX_FINDING_TEXT_CHARS],- "dimension": "",- "evidence_refs": [],- }- )+ text = item[:_MAX_FINDING_TEXT_CHARS]+ entry = {"suggestion": text, "dimension": "", "evidence_refs": []}+ else:+ continue+ key = " ".join(text.lower().split())+ if not key or key in seen:+ continue+ seen.add(key)+ result.append(entry)if result:return resultif isinstance(parsed, dict) and "suggestions" in parsed:diff --git a/tests/evaluation/test_judges.py b/tests/evaluation/test_judges.pyindex d142b0a8..007d05ac 100644--- a/tests/evaluation/test_judges.py+++ b/tests/evaluation/test_judges.py@@ -849,3 +849,82 @@ def test_regressed_dimension_reasoning_names_the_weakest_signal() -> None:assert regressed[-1] == "Weakest signal: goal_accuracy 0.25."assert improved[-1] == "Strongest signal: behavior_check 0.81."+++class TestInsightsJudgeRecommendationQuality:+ @staticmethod+ def _recommendation(message: str, title: str = "Next step") -> dict:+ return {+ "claim_type": "general",+ "category": "Improve",+ "title": title,+ "message": message,+ "severity": "warn",+ "evidence_case_ids": ["positive-001"],+ }++ def _parse(self, *messages: str, deterministic: dict | None = None) -> list[str]:+ response = json.dumps(+ {+ "conclusions": [],+ "recommendations": [self._recommendation(message) for message in messages],+ }+ )+ parsed = InsightsJudge().parse_response(+ response,+ canonical={"dataset": [{"id": "positive-001", "expected_skill": "calculator"}], "trials": []},+ deterministic=deterministic or {},+ )+ return [item["message"] for item in parsed["recommendations"]]++ @pytest.mark.parametrize(+ "message",+ [+ "Loosen the expected output of positive-001 so the agent passes.",+ "Relax the assertions for positive-001.",+ "Lower the pass threshold for positive-001 to 0.3.",+ "Soften positive-001's ground truth to accept partial answers.",+ "The pass threshold for positive-001 should be lowered to 0.3.",+ "Expectations for positive-001 ought to be relaxed so the agent passes.",+ ],+ )+ def test_drops_advice_that_weakens_requirements(self, message: str) -> None:+ assert self._parse(message) == []++ def test_keeps_advice_that_clarifies_requirements(self) -> None:+ message = "Make the expected output of positive-001 explicit in evals.json."+ assert self._parse(message) == [message]++ def test_keeps_negated_and_non_requirement_uses_of_weakening_verbs(self) -> None:+ messages = (+ "Do not lower the pass threshold for positive-001; fix the rounding step instead.",+ "Lower ambiguity in positive-001's acceptance criteria by documenting the rounding rule.",+ "Tell the agent to lowercase unit names, as positive-001's expected output does.",+ )+ assert self._parse(*messages) == list(messages)++ def test_does_not_match_a_requirement_across_title_and_message(self) -> None:+ message = "Document the expected JSON shape for positive-001 in SKILL.md."+ recommendation = self._recommendation(message, title="Lower token usage")+ response = json.dumps({"conclusions": [], "recommendations": [recommendation]})+ parsed = InsightsJudge().parse_response(+ response,+ canonical={"dataset": [{"id": "positive-001", "expected_skill": "calculator"}], "trials": []},+ )+ assert [item["message"] for item in parsed["recommendations"]] == [message]++ def test_drops_repeats_of_deterministic_and_earlier_recommendations(self) -> None:+ first = "Document the rounding rule that positive-001 depends on in SKILL.md."+ repeated = "Add a units example for positive-001 to SKILL.md."+ assert self._parse(+ first,+ first.upper(),+ repeated,+ deterministic={"suggestions": ["add a units example for positive-001 to SKILL.md."]},+ ) == [first]++ def test_prompt_requires_distinct_actions_and_experiments_for_untested_causes(self) -> None:+ prompt = InsightsJudge().get_system_prompt()+ assert "do not restate a\n deterministic recommendation" in prompt+ assert "Never recommend loosening, relaxing, or lowering an expectation" in prompt+ assert "Recommend a different agent or model only when the supplied trials compare" in promptdiff --git a/tests/tier3/test_suggestion_grounding.py b/tests/tier3/test_suggestion_grounding.pyindex e9650342..6f5b0c9d 100644--- a/tests/tier3/test_suggestion_grounding.py+++ b/tests/tier3/test_suggestion_grounding.py@@ -426,25 +426,82 @@ def test_suggestions_evidence_refs_resolved_to_full_dict(monkeypatch):assert "excerpt" in refs[0]-def test_suggestions_evidence_refs_unresolvable_string_fallback(monkeypatch):- """Unresolvable string ref is parsed into a minimal dict with kind='evidence'."""+def test_suggestions_drop_evidence_refs_the_run_did_not_record(monkeypatch):+ """A cited ref that the run did not record is dropped instead of rendered as evidence."""monkeypatch.setattr("skillevaluator.tier3.eval_core.llm_judge.call_public_llm",lambda _prompt, **_kw: ('[{"suggestion": "Fix something", "dimension": "goal_accuracy",'- ' "evidence_refs": ["trajectory.json#/steps/999"]}]',+ ' "evidence_refs": ["trajectory.json#/steps/999", "trajectory.json#/steps/14",'+ ' "trajectory.json#/steps/14", {"source": "made-up.json", "json_pointer": "/x"}]}]',None,),)reward = _reward_multi_metric(0.1)findings = report._extract_findings([reward])- objs = report._generate_suggestions_structured("demo", findings, [reward])- assert objs- refs = objs[0]["evidence_refs"]- assert refs and isinstance(refs[0], dict), f"expected dict fallback, got {refs!r}"- assert refs[0]["source"] == "trajectory.json"- assert refs[0]["json_pointer"] == "/steps/999"- assert refs[0]["kind"] == "evidence"+ [obj] = report._generate_suggestions_structured("demo", findings, [reward])+ assert [ref["json_pointer"] for ref in obj["evidence_refs"]] == ["/steps/14"]+ assert obj["evidence_refs"][0]["kind"] == "tool_call"+++ "raw_refs",+ [+ ["trajectory.json#/steps/14"],+ {"source": "trajectory.json", "json_pointer": "/steps/14", "kind": "tool_call", "excerpt": "submit ..."},+ "trajectory.json#/steps/14",+ ],+)+def test_suggestions_resolve_legacy_evidence_ref_shapes(monkeypatch, raw_refs):+ monkeypatch.setattr(+ "skillevaluator.tier3.eval_core.llm_judge.call_public_llm",+ lambda _prompt, **_kw: (+ '[{"suggestion": "Fix evaluator job", "dimension": "goal_accuracy",'+ ' "evidence_refs": ["trajectory.json#/steps/14"]}]',+ None,+ ),+ )+ reward = _reward_multi_metric(0.1)+ reward["details"]["goal_accuracy"]["evidence_refs"] = raw_refs+ findings = report._extract_findings([reward])+ [suggestion] = report._generate_suggestions_structured("demo", findings, [reward])+ assert suggestion["evidence_refs"][0]["source"] == "trajectory.json"+ assert suggestion["evidence_refs"][0]["json_pointer"] == "/steps/14"+ if isinstance(raw_refs, dict):+ assert suggestion["evidence_refs"][0]["kind"] == "tool_call"+ assert suggestion["evidence_refs"][0]["excerpt"] == "submit ..."+++def test_suggestions_keep_one_copy_of_a_repeated_suggestion(monkeypatch):+ monkeypatch.setattr(+ "skillevaluator.tier3.eval_core.llm_judge.call_public_llm",+ lambda _prompt, **_kw: (+ '[{"suggestion": "Install jq in evals/environment/Dockerfile.", "dimension": "goal_accuracy"},'+ ' {"suggestion": "install jq in evals/environment/Dockerfile.", "dimension": "security"},'+ ' "Document the output format in SKILL.md."]',+ None,+ ),+ )+ reward = _reward_multi_metric(0.1)+ objs = report._generate_suggestions_structured("demo", report._extract_findings([reward]), [reward])+ assert [obj["suggestion"] for obj in objs] == [+ "Install jq in evals/environment/Dockerfile.",+ "Document the output format in SKILL.md.",+ ]+++def test_suggestion_prompt_guards_requirements_and_labels_hypotheses(monkeypatch):+ prompts = []+ monkeypatch.setattr(+ "skillevaluator.tier3.eval_core.llm_judge.call_public_llm",+ lambda prompt, **_kw: (prompts.append(prompt), ("[]", None))[1],+ )+ reward = _reward_multi_metric(0.1)+ report._generate_suggestions_structured("demo", report._extract_findings([reward]), [reward])+ [prompt] = prompts+ assert "Do not suggest loosening, removing, or rewriting an eval expectation just so the agent passes" in prompt+ assert 'start the suggestion with "Hypothesis:"' in prompt+ assert "Give distinct suggestions" in promptdef test_suggestions_evidence_refs_lookup_uses_all_metrics(monkeypatch):