Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(tier3): add run-level next steps to Tier 3 reports (#377)
Christopher Kevin0bd8920069bd
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b104f996..c6691c14 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -15,6 +15,17 @@ All notable changes to SkillEvaluator are documented in this file.- Tier 3 datasets can declare deterministic behavior checks for script invocation and success, workspace files, and final-response text.+- Tier 3 reports list up to three run-level **Next steps** under the per-case+ fixes in the terminal, `report.html` and the Markdown report: re-run an+ incomplete run, add attempts for inconsistent cases, add cases or negative+ controls, or compare agents and models. Each step names its target+ (evaluation, dataset or agent), the observed problem and evidence, the action,+ and the check that would show improvement. Steps about agents or models are+ marked as hypotheses to test, steps from incomplete runs as limited, and none+ suggest removing cases or loosening expectations. The rules need no LLM; the+ Tier 3 payload carries them as `next_steps`.++- Document per-agent Tier 3 `findings.json` artifacts and distinguish advisorymetric severity from dimension verdicts; include severity thresholds in theartifact and a muted CLI legend without changing scores or thresholds.diff --git a/docs/reports.mdx b/docs/reports.mdxindex 7a7b4b7e..2992acf4 100644--- a/docs/reports.mdx+++ b/docs/reports.mdx@@ -562,6 +562,39 @@ Artifacts panel shows the HTML report path, the output directory, and — onlywhen a retained Harbor jobs directory exists — a ready-to-paste `harbor-view`command.+### Next steps for the run++Below the per-case fixes, the case summary in the terminal, `report.html`,+and the Markdown report list up to three **Next steps** for the run as a+whole: what a single case cannot show. The steps come from fixed rules, so+they also appear when no LLM is configured. Each step names:++- the **target** to change: the `evaluation` settings (re-run, more+ attempts), the `dataset` (more cases, negative controls), or the `agent`+ (compare agents or models);+- the **problem** observed in this run and the **evidence** behind it (case+ ids, attempt marks, or per-agent pass-rate changes);+- the **action** to take, with the flag or file to change;+- the **check** that would show the change worked.++| Step | Shown when |+|------|------------|+| Incomplete run | Attempts did not produce a score and no per-case run error already says so |+| Inconsistent results | A case passed on some attempts and failed on others (more `--n-attempts`) |+| Single attempt | Every case ran once |+| Agent-dependent effect | Two or more agents differ by 20 or more points in pass-rate change, and the best agent gains at least 10 points |+| More cases | Fewer than five skill cases ran, unless the case fixes already ask for new cases |+| Negative controls | The dataset snapshot has no case with `expected_skill: null` or `should_trigger: false` |+| Agent experiment | Only one agent ran and the skill showed no clear gain |++A step marked `hypothesis` is a cause or remedy to test, not a finding: for+example, that one agent follows the skill differently. A step marked+`limited` rests on an incomplete run, so re-run first. Steps never suggest+removing cases or loosening expectations to raise a score. Baseline-only runs+and runs where the skill never activated get no next steps. The Tier 3 payload+carries them as `next_steps`, with the `kind`, `target`, `problem`,+`evidence`, `action`, `check` and `strength` fields.+### Findings and severityThe terminal Findings panel uses **advisory metric severity**, not dimensiondiff --git a/src/skillevaluator/evaluation/tier3_report.py b/src/skillevaluator/evaluation/tier3_report.pyindex 989b0d8f..1ff1b08a 100644--- a/src/skillevaluator/evaluation/tier3_report.py+++ b/src/skillevaluator/evaluation/tier3_report.py@@ -554,6 +554,13 @@ def agent_eval_result_from_directory(use_llm_judge=use_llm_judge,enforce_embedded_size=enforce_embedded_size,)+ with contextlib.suppress(Exception):+ source = json.loads((run_dir / "result.json").read_text(encoding="utf-8"))+ if isinstance(source, dict) and isinstance(source.get("agents"), dict):+ from skillevaluator.tier3.case_summary import load_case_findings+ from skillevaluator.tier3.next_steps import next_steps++ payload["next_steps"] = next_steps(source, run_dir=run_dir, fixes=load_case_findings(run_dir))return _validation_result_from_payload(payload)@@ -3939,6 +3946,10 @@ def _apply_case_findings(for fix in fixes]payload["suggestions"] = [fix["action"] for fix in fixes if fix.get("action")]+ from skillevaluator.tier3.next_steps import next_steps++ with contextlib.suppress(Exception): # advisory, like the fixes above+ payload["next_steps"] = next_steps(source, run_dir=run_dir, fixes=fixes)def _recommendation_title_from(text: str) -> str:diff --git a/src/skillevaluator/reporting/html.py b/src/skillevaluator/reporting/html.pyindex 93331c2e..e87d2aea 100644--- a/src/skillevaluator/reporting/html.py+++ b/src/skillevaluator/reporting/html.py@@ -1242,6 +1242,13 @@ def _sanitize_tier3_display_payload(value: dict[str, Any]) -> dict[str, Any]:]for key in ("conclusions", "recommendations", "suggestions_v2"):payload[key] = _sanitize_tier3_insight_items(payload.get(key))+ payload["next_steps"] = [+ {+ key: str(step.get(key) or "")+ for key in ("kind", "target", "problem", "evidence", "action", "check", "strength")+ }+ for step in _dict_list(payload.get("next_steps"))+ ]payload["dataset"] = [_sanitize_tier3_dataset_case(item) for item in _dict_list(payload.get("dataset"))]payload["case_diagnostics"] = _sanitize_tier3_case_diagnostics(payload.get("case_diagnostics"))payload["trials"] = [_sanitize_tier3_trial(item) for item in _dict_list(payload.get("trials"))]diff --git a/src/skillevaluator/reporting/markdown.py b/src/skillevaluator/reporting/markdown.pyindex 3e441722..cb6b46e0 100644--- a/src/skillevaluator/reporting/markdown.py+++ b/src/skillevaluator/reporting/markdown.py@@ -493,6 +493,24 @@ class MarkdownReporter(ReporterBase):lines.append(f" - Evidence: [{label}]({url})")lines.append("")+ if next_steps := _dict_items(ae.get("next_steps")):+ lines.append("### Next Steps")+ lines.append("")+ for idx, step in enumerate(next_steps, start=1):+ problem = _markdown_inline_text(step.get("problem"))+ action = _markdown_inline_text(step.get("action"))+ if not problem or not action:+ continue+ target = _markdown_inline_text(step.get("target")) or "evaluation"+ strength = _markdown_inline_text(step.get("strength"))+ label = target if strength in ("", "supported") else f"{target} · {strength}"+ lines.append(f"{idx}. **{problem}** [{label}]: {action}")+ if check := _markdown_inline_text(step.get("check")):+ lines.append(f" - Check: {check}")+ if evidence := _markdown_inline_text(step.get("evidence")):+ lines.append(f" - Evidence: {evidence}")+ lines.append("")+# Results per validatorlines.append("## Results")lines.append("")diff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2index 4354c83a..8853a885 100644--- a/src/skillevaluator/reporting/templates/report.html.j2+++ b/src/skillevaluator/reporting/templates/report.html.j2@@ -2698,6 +2698,18 @@<p class="t3-next-steps-conclusion">No recommendations were generated for this evaluation.</p>{% endif %}{% endif %}+ {% set t3_run_next_steps = tier3.next_steps or [] %}+ {% if t3_run_next_steps %}+ <p class="t3-next-steps-conclusion" id="tier3-run-next-steps">Next steps for the run:</p>+ <ol class="t3-next-steps-list">+ {% for step in t3_run_next_steps %}+ <li>+ <strong>{{ step.problem }}</strong> [{{ step.target }}{% if step.strength and step.strength != 'supported' %} · {{ step.strength }}{% endif %}]:+ {{ step.action }} Check: {{ step.check }}{% if step.evidence %} Evidence: {{ step.evidence }}{% endif %}+ </li>+ {% endfor %}+ </ol>+ {% endif %}{% if (t3_source_errors | length) == 1 %}<p class="t3-next-steps-conclusion">Source validation error: {% if t3_source_errors[0].path %}{{ t3_source_errors[0].path }}: {% endif %}{{ t3_source_errors[0].message }}diff --git a/src/skillevaluator/tier3/next_steps.py b/src/skillevaluator/tier3/next_steps.pynew file mode 100644index 00000000..a85c32f3--- /dev/null+++ b/src/skillevaluator/tier3/next_steps.py@@ -0,0 +1,300 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Run-level Tier 3 next steps: the change or experiment to try next, and what would show it worked.++The per-case "How to improve" list says how to fix individual failed cases.+These steps cover what a single case cannot show: missing attempts, noisy+results, effects that depend on the agent, and gaps in the dataset. Each step+names its target, the observed problem, the evidence from this run, the action,+and the check that would show improvement. The rules are deterministic, so the+same steps appear when no LLM is configured.+"""++from __future__ import annotations++from collections.abc import Callable, Mapping+from pathlib import Path+from typing import TYPE_CHECKING, Any++from rich.text import Text++from skillevaluator.reporting.terminal_styles import AMBER+from skillevaluator.tier3.case_summary import (+ _CLEAR_EFFECT_PP,+ _VARIANTS,+ ERROR,+ FAIL,+ PASS,+ _agent_rates,+ _case_ids,+ _cases,+ _comparable,+ _dataset_entries,+ _details_truncated,+ _execution_unavailable,+ _marks,+ _never_activated,+ _pct,+ _with_skill_skipped,+)++if TYPE_CHECKING:+ from rich.console import Console++# Observed directly in this run.+SUPPORTED = "supported"+# A possible cause or remedy to test before acting on it.+HYPOTHESIS = "hypothesis"+# Observed, but the run is incomplete, so the evidence is partial.+LIMITED = "limited"++MAX_NEXT_STEPS = 3+_MIN_SKILL_CASES = 5+_MAX_EVIDENCE_ITEMS = 4+++def _routing(entry: Mapping[str, Any]) -> bool | None:+ """Mirror the grader: ``should_trigger`` wins, then a truthy ``expected_skill``; neither means unlabeled."""+ if "should_trigger" in entry:+ return bool(entry.get("should_trigger"))+ if "expected_skill" in entry:+ return bool(entry.get("expected_skill"))+ return None+++def _listed(items: list[str]) -> str:+ shown = ", ".join(items[:_MAX_EVIDENCE_ITEMS])+ return shown + (f" (+{len(items) - _MAX_EVIDENCE_ITEMS} more)" if len(items) > _MAX_EVIDENCE_ITEMS else "")+++def _step(+ kind: str, target: str, problem: str, evidence: str, action: str, check: str, strength: str+) -> dict[str, str]:+ return {+ "kind": kind,+ "target": target,+ "problem": problem,+ "evidence": evidence,+ "action": action,+ "check": check,+ "strength": strength,+ }+++def _missing_attempts(agents: Mapping[str, Mapping[str, Any]]) -> tuple[int, list[str], list[str]]:+ missing = 0+ cases: list[str] = []+ unavailable: list[str] = []+ for name, agent in agents.items():+ if _execution_unavailable(agent) or _details_truncated(agent):+ unavailable.append(name)+ for key, label in _VARIANTS:+ for case_id, case in _cases(agent, key).items():+ if n := _marks(case).count(ERROR):+ missing += n+ cases.append(f"{name}/{case_id} ({label})")+ return missing, cases, unavailable+++def _inconsistent_cases(agents: Mapping[str, Mapping[str, Any]]) -> tuple[list[str], int]:+ mixed: list[str] = []+ most_attempts = 0+ for name, agent in agents.items():+ for key, label in _VARIANTS:+ for case_id, case in _cases(agent, key).items():+ planned = _marks(case)+ most_attempts = max(most_attempts, len(planned))+ marks = [m for m in planned if m in (PASS, FAIL)]+ if PASS in marks and FAIL in marks:+ mixed.append(f"{name}/{case_id} {label} {''.join(marks)}")+ return mixed, most_attempts+++def _pass_rate_changes(agents: Mapping[str, Mapping[str, Any]]) -> dict[str, float]:+ changes: dict[str, float] = {}+ for name, agent in agents.items():+ if not _comparable(agent):+ continue+ rates = _agent_rates(agent)+ with_pct, without_pct = _pct(*rates["with_skill"]), _pct(*rates["without_skill"])+ if with_pct is not None and without_pct is not None:+ changes[name] = with_pct - without_pct+ return changes+++def next_steps(+ result: Mapping[str, Any],+ *,+ run_dir: Path,+ fixes: list[dict[str, Any]] | None = None,+) -> list[dict[str, str]]:+ """Return up to three distinct run-level next steps, most blocking first."""+ raw_agents = result.get("agents")+ raw_agents = raw_agents if isinstance(raw_agents, Mapping) else {}+ total_agent_count = len(raw_agents)+ agents = {+ str(name): agent+ for name, agent in raw_agents.items()+ if isinstance(agent, Mapping) and not _never_activated(agent) and not _with_skill_skipped(agent)+ }+ case_ids = _case_ids(agents)+ if not agents or not case_ids:+ return []+ new_case_fixes = any(+ isinstance(fix, Mapping) and fix.get("kind") == "eval case" and fix.get("case") == "all cases"+ for fix in fixes or []+ )+ has_run_error_fix = any(isinstance(fix, Mapping) and fix.get("kind") == "run error" for fix in fixes or [])+ steps: list[dict[str, str]] = []++ missing, missing_cases, unavailable = _missing_attempts(agents)+ partial = bool(missing or unavailable)+ observed = LIMITED if partial else SUPPORTED+ if partial and not has_run_error_fix:+ problem = (+ f"{missing} attempt(s) did not produce a score"+ if missing+ else f"execution is incomplete for {', '.join(unavailable)}"+ )+ steps.append(+ _step(+ "incomplete run",+ "evaluation",+ problem + ", so the verdict and the steps below rest on partial evidence",+ _listed(missing_cases) if missing_cases else _listed(unavailable),+ "Re-run the evaluation before changing the skill or the cases; "+ "if the same attempts fail again, check the agent and gateway logs in the run directory.",+ "Every planned attempt has a score and the verdict is no longer inconclusive.",+ SUPPORTED,+ )+ )++ mixed, most_attempts = _inconsistent_cases(agents)+ if mixed:+ attempt_count = max(3, most_attempts + 2)+ attempt_flags = f"--n-attempts {attempt_count}"+ attempt_policy = result.get("attempt_policy")+ if isinstance(attempt_policy, Mapping) and attempt_policy.get("stop_on_pass") is True:+ attempt_flags += " --no-stop-on-pass"+ steps.append(+ _step(+ "inconsistent results",+ "evaluation",+ f"{len(mixed)} case result(s) passed on some attempts and failed on others",+ _listed(mixed),+ f"Re-run with more attempts per case (`{attempt_flags}`) "+ "before editing the skill or the cases, so one lucky or unlucky attempt does not decide the change.",+ "Each case passes or fails consistently and the verdict keeps its direction.",+ observed,+ )+ )+ elif most_attempts == 1:+ steps.append(+ _step(+ "single attempt",+ "evaluation",+ "each case ran once, so a single attempt decides every pass or fail mark",+ f"{len(case_ids)} case(s), 1 attempt each",+ "Re-run with `--n-attempts 3` to see whether the results repeat before acting on them.",+ "The marks repeat across attempts and the verdict keeps its direction.",+ observed,+ )+ )++ changes = _pass_rate_changes(agents)+ if len(changes) >= 2:+ best = max(changes, key=changes.__getitem__)+ worst = min(changes, key=changes.__getitem__)+ spread = changes[best] - changes[worst]+ if spread >= 2 * _CLEAR_EFFECT_PP and changes[best] >= _CLEAR_EFFECT_PP:+ steps.append(+ _step(+ "agent-dependent effect",+ "agent",+ f"the skill's effect depends on the agent: {best} {changes[best]:+.0f} pp, "+ f"{worst} {changes[worst]:+.0f} pp",+ ", ".join(f"{name} {delta:+.0f} pp" for name, delta in changes.items()),+ f"Test the hypothesis that {worst} follows the skill differently: compare its failing "+ f"trajectories with {best}'s on the same cases, and re-run {worst} alone with more attempts "+ "before changing SKILL.md for every agent.",+ f"{worst}'s pass-rate change moves toward {best}'s without lowering {best}'s.",+ HYPOTHESIS,+ )+ )++ dataset = _dataset_entries(run_dir)+ routed = {case_id: _routing(dataset[case_id]) for case_id in case_ids if case_id in dataset}+ skill_cases = [case_id for case_id in case_ids if routed.get(case_id) is not False]+ if len(skill_cases) < _MIN_SKILL_CASES and not new_case_fixes:+ steps.append(+ _step(+ "more cases",+ "dataset",+ f"only {len(skill_cases)} skill case(s) were evaluated, so one case moves the pass rate by "+ f"{100 / max(len(skill_cases), 1):.0f} pp",+ _listed(skill_cases),+ "Add realistic requests the skill should handle to evals/evals.json, covering inputs and "+ "phrasings the current cases miss (`skillevaluator create-eval-dataset <skill> --full --dry-run` "+ "previews drafts); keep the existing cases and their expectations.",+ f"With {_MIN_SKILL_CASES} or more skill cases the verdict keeps its direction.",+ observed,+ )+ )+ if routed and not any(value is False for value in routed.values()):+ steps.append(+ _step(+ "negative controls",+ "dataset",+ "the dataset has no negative-control case, so the run cannot show that the skill stays out of "+ "unrelated requests",+ f"{len(routed)} case(s), none with `expected_skill: null` or `should_trigger: false`",+ 'Add one or two unrelated requests with `"expected_skill": null` to evals/evals.json; '+ "the agent should complete them without loading the skill.",+ "The negative controls pass with the skill installed.",+ observed,+ )+ )++ if total_agent_count == 1 and len(changes) == 1:+ [(name, delta)] = changes.items()+ if delta < _CLEAR_EFFECT_PP:+ steps.append(+ _step(+ "agent experiment",+ "agent",+ f"only {name} was evaluated and the skill showed no clear gain ({delta:+.0f} pp), so it is "+ "unknown whether the result depends on the agent or model",+ f"{name} {delta:+.0f} pp",+ f"As an experiment, run the same cases with a second agent (`--agents {name},<agent>`) or "+ "another model (`--agent-model AGENT=MODEL`), keeping the dataset unchanged.",+ "A second agent with the same direction points at the skill; "+ "a different direction points at agent-specific guidance.",+ HYPOTHESIS,+ )+ )++ return steps[:MAX_NEXT_STEPS]+++def render_next_steps(+ steps: list[dict[str, str]],+ *,+ console: Console,+ safe: Callable[[object], str],+) -> None:+ """Print the run-level next steps under the case summary."""+ if not steps:+ return+ console.print(Text("Next steps", style="bold"))+ for i, step in enumerate(steps, 1):+ head = Text(f" {i}. ", style=f"bold {AMBER}")+ head.append(safe(step["problem"]))+ label = step["target"] if step["strength"] == SUPPORTED else f"{step['target']} · {step['strength']}"+ head.append(f" [{safe(label)}]", style="dim")+ console.print(head)+ console.print(Text(f" try: {safe(step['action'])}", style="green"))+ console.print(Text(f" check: {safe(step['check'])}", style="dim"))+ if step.get("evidence"):+ console.print(Text(f" evidence: {safe(step['evidence'])}", style="dim"))+ console.print()diff --git a/src/skillevaluator/tier3/result_display.py b/src/skillevaluator/tier3/result_display.pyindex 2d150615..708b1147 100644--- a/src/skillevaluator/tier3/result_display.py+++ b/src/skillevaluator/tier3/result_display.py@@ -956,7 +956,20 @@ def _case_first_summary(result: Mapping[str, Any], *, console: Console, skill_pasave_case_findings(Path(str(run_dir)), fixes)except Exception: # advisory panel: never break the run summarylogging.getLogger(__name__).debug("What-to-fix skipped", exc_info=True)- return render_case_summary(result, console=console, fixes=fixes)+ shown = render_case_summary(result, console=console, fixes=fixes)+ if shown and run_dir and Path(str(run_dir)).is_dir():+ from skillevaluator.tier3.next_steps import next_steps, render_next_steps++ secrets = secret_values_from_environment(os.environ)+ try:+ steps = next_steps(result, run_dir=Path(str(run_dir)), fixes=fixes)+ except Exception: # advisory panel: never break the run summary+ logging.getLogger(__name__).debug("Next steps skipped", exc_info=True)+ steps = []+ render_next_steps(+ steps, console=console, safe=lambda value: redact_progress_detail(value, secret_values=secrets)+ )+ return showndef render_evaluation_result(diff --git a/tests/test_tier3_next_steps.py b/tests/test_tier3_next_steps.pynew file mode 100644index 00000000..e49c47a4--- /dev/null+++ b/tests/test_tier3_next_steps.py@@ -0,0 +1,261 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Run-level Tier 3 next steps: grounded, distinct, and honest about weak evidence."""++import json+from pathlib import Path++from tests.reporting.test_case_findings_report import _run+from tests.test_tier3_case_summary import _agent, _case++from skillevaluator.evaluation.tier3_report import (+ _validation_result_from_payload,+ agent_eval_result_from_directory,+ build_agent_eval_payload,+ render_agent_eval_html_report,+)+from skillevaluator.reporting import MarkdownReporter+from skillevaluator.tier3.next_steps import HYPOTHESIS, LIMITED, SUPPORTED, next_steps+from skillevaluator.tier3.result_display import render_result++_IDS = ("c1", "c2", "c3", "c4", "c5")+++def _dataset(run: Path, *, negative: bool = True, ids: tuple[str, ...] = _IDS) -> None:+ entries = [{"id": case_id, "question": "q", "expected_skill": "demo"} for case_id in ids]+ if negative:+ entries.append({"id": "neg", "question": "unrelated", "expected_skill": None})+ run.mkdir(parents=True, exist_ok=True)+ (run / "dataset_snapshot.json").write_text(json.dumps({"dataset": entries}))+++def _helps(ids: tuple[str, ...] = _IDS, *, negative: bool = True) -> dict:+ with_cases = {case_id: _case(True, True) for case_id in ids}+ without_cases = {case_id: _case(False, False) for case_id in ids}+ if negative:+ with_cases["neg"] = _case(True, True)+ without_cases["neg"] = _case(True, True)+ return _agent(with_cases, without_cases)+++def _kinds(steps: list[dict]) -> list[str]:+ return [step["kind"] for step in steps]+++def test_clean_run_with_enough_cases_and_a_negative_control_needs_no_next_step(tmp_path: Path) -> None:+ _dataset(tmp_path)+ assert next_steps({"agents": {"a": _helps()}}, run_dir=tmp_path) == []+++def test_missing_attempts_come_first_and_limit_the_other_steps(tmp_path: Path) -> None:+ _dataset(tmp_path, ids=("c1", "c2"))+ agent = _helps(("c1", "c2"))+ agent["pass_at_k"]["with_skill"]["cases"]["c1"] = _case(True, missing=1)++ steps = next_steps({"agents": {"a": agent}}, run_dir=tmp_path)++ assert _kinds(steps) == ["incomplete run", "more cases"]+ assert steps[0]["strength"] == SUPPORTED and steps[0]["target"] == "evaluation"+ assert "a/c1 (with-skill)" in steps[0]["evidence"]+ assert steps[1]["strength"] == LIMITED+++def test_run_errors_already_listed_per_case_are_not_repeated(tmp_path: Path) -> None:+ _dataset(tmp_path, ids=("c1", "c2"))+ agent = _helps(("c1", "c2"))+ agent["pass_at_k"]["with_skill"]["cases"]["c1"] = _case(True, missing=1)++ steps = next_steps({"agents": {"a": agent}}, run_dir=tmp_path, fixes=[{"kind": "run error"}])++ assert _kinds(steps) == ["more cases"]+ assert steps[0]["strength"] == LIMITED+++def test_mixed_attempts_ask_for_more_attempts_before_editing_anything(tmp_path: Path) -> None:+ _dataset(tmp_path)+ agent = _helps()+ agent["pass_at_k"]["with_skill"]["cases"]["c2"] = _case(True, False)++ [step] = next_steps({"agents": {"a": agent}}, run_dir=tmp_path)++ assert step["kind"] == "inconsistent results" and step["target"] == "evaluation"+ assert "`--n-attempts 4`" in step["action"]+ assert step["evidence"] == "a/c2 with-skill ✓✗"++ result = {"agents": {"a": agent}, "attempt_policy": {"stop_on_pass": True}}+ [stopped_step] = next_steps(result, run_dir=tmp_path)+ assert "`--n-attempts 4 --no-stop-on-pass`" in stopped_step["action"]+++def test_single_attempt_runs_are_flagged_but_stop_on_pass_skips_are_not(tmp_path: Path) -> None:+ _dataset(tmp_path)+ once = _agent(+ {case_id: _case(True) for case_id in (*_IDS, "neg")},+ {case_id: _case(False) for case_id in _IDS} | {"neg": _case(True)},+ )+ assert _kinds(next_steps({"agents": {"a": once}}, run_dir=tmp_path)) == ["single attempt"]++ skipped = {**_case(True), "attempts_skipped": 2}+ stopped = _agent(+ {case_id: dict(skipped) for case_id in (*_IDS, "neg")},+ {case_id: _case(False, False, False) for case_id in _IDS} | {"neg": dict(skipped)},+ )+ assert next_steps({"agents": {"a": stopped}}, run_dir=tmp_path) == []+++def test_agent_dependent_effect_is_a_hypothesis_to_test(tmp_path: Path) -> None:+ _dataset(tmp_path)+ flat = _agent(+ {case_id: _case(True, True) for case_id in (*_IDS, "neg")},+ {case_id: _case(True, True) for case_id in (*_IDS, "neg")},+ )++ [step] = next_steps({"agents": {"claude-code": _helps(), "codex": flat}}, run_dir=tmp_path)++ assert step["kind"] == "agent-dependent effect" and step["strength"] == HYPOTHESIS+ assert "claude-code +83 pp" in step["problem"] and "codex +0 pp" in step["problem"]+ assert step["action"].startswith("Test the hypothesis that codex")+++def test_never_activated_agent_does_not_trigger_single_agent_experiment(tmp_path: Path) -> None:+ _dataset(tmp_path)+ never = {**_helps(), "skill_activation": {"status": "never_activated"}}+ flat = _agent(+ {case_id: _case(True, True) for case_id in (*_IDS, "neg")},+ {case_id: _case(True, True) for case_id in (*_IDS, "neg")},+ )++ steps = next_steps({"agents": {"claude-code": never, "codex": flat}}, run_dir=tmp_path)++ assert "agent experiment" not in _kinds(steps)+++def test_few_cases_ask_for_more_cases_and_negative_controls_do_not_count(tmp_path: Path) -> None:+ _dataset(tmp_path, ids=("c1", "c2"))++ [step] = next_steps({"agents": {"a": _helps(("c1", "c2"))}}, run_dir=tmp_path)++ assert step["kind"] == "more cases" and step["target"] == "dataset"+ assert step["problem"].startswith("only 2 skill case(s)") and step["evidence"] == "c1, c2"+ assert "keep the existing cases and their expectations" in step["action"]+ rewritten = [{"kind": "eval case", "case": "c1"}]+ assert _kinds(next_steps({"agents": {"a": _helps(("c1", "c2"))}}, run_dir=tmp_path, fixes=rewritten)) == [+ "more cases"+ ]+ saturated = [{"kind": "eval case", "case": "all cases"}]+ assert next_steps({"agents": {"a": _helps(("c1", "c2"))}}, run_dir=tmp_path, fixes=saturated) == []+++def test_missing_negative_control_is_reported_only_with_a_dataset_snapshot(tmp_path: Path) -> None:+ result = {"agents": {"a": _helps(negative=False)}}+ assert next_steps(result, run_dir=tmp_path) == []++ _dataset(tmp_path, negative=False)+ [step] = next_steps(result, run_dir=tmp_path)+ assert step["kind"] == "negative controls" and step["target"] == "dataset"+ assert '"expected_skill": null' in step["action"]+++def test_single_agent_without_a_clear_gain_suggests_an_agent_experiment(tmp_path: Path) -> None:+ _dataset(tmp_path)+ flat = _agent(+ {case_id: _case(True, True) for case_id in (*_IDS, "neg")},+ {case_id: _case(True, True) for case_id in (*_IDS, "neg")},+ )++ [step] = next_steps({"agents": {"opencode": flat}}, run_dir=tmp_path)++ assert step["kind"] == "agent experiment" and step["strength"] == HYPOTHESIS+ assert "`--agents opencode,<agent>`" in step["action"] and "keeping the dataset unchanged" in step["action"]+++def test_steps_are_capped_distinct_and_never_weaken_the_dataset(tmp_path: Path) -> None:+ _dataset(tmp_path, ids=("c1",), negative=False)+ agent = _agent({"c1": _case(True, False, missing=1)}, {"c1": _case(True, False, False)})++ steps = next_steps({"agents": {"opencode": agent}}, run_dir=tmp_path)++ assert _kinds(steps) == ["incomplete run", "inconsistent results", "more cases"]+ text = " ".join(value for step in steps for value in step.values()).lower()+ assert not any(word in text for word in ("remove", "delete", "loosen", "lower the"))+++def test_baseline_only_and_never_activated_runs_get_no_next_steps(tmp_path: Path) -> None:+ baseline_only = {**_helps(), "conditions": {"with_skill": {"execution_status": "skipped"}}}+ never = {**_helps(), "skill_activation": {"status": "never_activated"}}+ assert next_steps({"agents": {"a": baseline_only}}, run_dir=tmp_path) == []+ assert next_steps({"agents": {"a": never}}, run_dir=tmp_path) == []+++def test_cli_prints_next_steps_after_the_case_summary(tmp_path: Path) -> None:+ _dataset(tmp_path, ids=("c1", "c2"))+ result = {"execution_status": "succeeded", "run_dir": str(tmp_path), "agents": {"a": _helps(("c1", "c2"))}}++ output = render_result(result)++ assert output.index("How to improve") < output.index("Next steps")+ assert "1. only 2 skill case(s) were evaluated" in output and "[dataset]" in output+ assert "check: With 5 or more skill cases the verdict keeps its direction." in output+++def test_html_report_shows_next_steps_under_the_case_fixes(tmp_path: Path) -> None:+ run = _run(tmp_path, goal_accuracy=1.0)++ html = render_agent_eval_html_report(tmp_path, run, use_llm_judge=False).read_text(encoding="utf-8")++ preview = " ".join(+ html.split('id="tier3-next-steps-preview"', 1)[1].split("</ol>\n </div>", 1)[0].split()+ )+ assert preview.index("Nothing to fix in the available case evidence.") < preview.index("Next steps for the run:")+ assert (+ "<strong>each case ran once, so a single attempt decides every pass or fail mark</strong> [evaluation]:"+ in preview+ )+ assert "Check: With 5 or more skill cases the verdict keeps its direction. Evidence: case" in preview+++def test_markdown_report_renders_next_step_strength() -> None:+ agent = {+ "num_trials": 1,+ "num_trials_baseline": 1,+ "with_skill": {"goal_accuracy": 0.9},+ "without_skill": {"goal_accuracy": 0.4},+ "lift": {"overall": {"delta": 0.5}},+ }+ payload = build_agent_eval_payload("demo", {"opencode": agent}, use_llm_judge=False)+ payload["next_steps"] = [+ {+ "kind": "agent experiment",+ "target": "agent",+ "problem": "only opencode was evaluated",+ "evidence": "opencode +0 pp",+ "action": "Run the same cases with a second agent.",+ "check": "A second agent shows the same direction.",+ "strength": HYPOTHESIS,+ }+ ]++ markdown = MarkdownReporter(include_timestamp=False).render_all([_validation_result_from_payload(payload)])++ assert "### Next Steps" in markdown+ assert (+ "1. **only opencode was evaluated** [agent · hypothesis]: Run the same cases with a second agent." in markdown+ )+ assert " - Check: A second agent shows the same direction." in markdown+ assert " - Evidence: opencode +0 pp" in markdown+++def test_markdown_report_uses_next_steps_from_saved_run_findings(tmp_path: Path) -> None:+ run = _run(tmp_path, goal_accuracy=1.0)+ render_agent_eval_html_report(tmp_path, run, use_llm_judge=False)++ result = agent_eval_result_from_directory(tmp_path, run, use_llm_judge=False)+ assert result is not None+ assert result.metadata["agent_eval"]["next_steps"]++ markdown = MarkdownReporter(include_timestamp=False).render_all([result])++ assert "### Next Steps" in markdown+ assert "each case ran once, so a single attempt decides every pass or fail mark" in markdown+ assert "Check: With 5 or more skill cases the verdict keeps its direction." in markdown