Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

docs(tier3): document findings.json and advisory severity thresholds (#372)

Christopher Kevin8e30825a3ca0
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 30abaacc..b104f996 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -14,6 +14,11 @@ All notable changes to SkillEvaluator are documented in this file.
setup, example verification and the existing DCO policy.
- Tier 3 datasets can declare deterministic behavior checks for script invocation and success, workspace files, and final-response text.
+
+- Document per-agent Tier 3 `findings.json` artifacts and distinguish advisory
+ metric severity from dimension verdicts; include severity thresholds in the
+ artifact and a muted CLI legend without changing scores or thresholds.
+
- `tier3 import-metrics` accepts versioned, normalized per-case custom scores
from external evaluations and writes clearly labeled JSON/Markdown reports.
Imports validate identities, metric metadata and values, retain declared
diff --git a/docs/reports.mdx b/docs/reports.mdx
index 639e1a74..7a7b4b7e 100644
--- a/docs/reports.mdx
+++ b/docs/reports.mdx
@@ -512,6 +512,10 @@ completed run directly:
- **`<agent>/lift.json`** and **`<agent>/pass_at_k_lift.json`** — per-metric
deltas between the two arms; written only when both arms produced scores
(a `--skip-baseline` or `--skip-with-skill` run has neither).
+- **`<agent>/findings.json`** — bounded per-metric findings and improvement
+ suggestions; written by `tier3 evaluate` only when the detailed view is
+ shown (`--verbose` or when no per-case summary is shown), never by
+ `validate --tier3`; see [Findings and severity](#findings-and-severity).
- **`<agent>/{with-skill,without-skill}/`** — each arm's `summary.json`
(scores, dimensions, pass@k, trial counts) and a `trials/` directory of
per-trial artifacts: the trial's `reward.json`, Harbor's `trajectory.json`
@@ -558,6 +562,93 @@ Artifacts panel shows the HTML report path, the output directory, and — only
when a retained Harbor jobs directory exists — a ready-to-paste `harbor-view`
command.
+### Findings and severity
+
+The terminal Findings panel uses **advisory metric severity**, not dimension
+verdicts: `ok` at 0.80 or above, `warning` from 0.40 to below 0.80, and
+`critical` below 0.40. These labels identify opportunities for improvement;
+a warning does not mean a dimension failed. Boundaries use the underlying
+score, not its rounded terminal display.
+
+| Scale | Meaning and boundaries |
+| --- | --- |
+| Advisory finding severity | Per-metric `ok` ≥ 0.80, `warning` ≥ 0.40 and below 0.80, `critical` below 0.40 |
+| [Dimension bands](#the-five-dimensions) | PASS ≥ 0.50, NEUTRAL ≥ 0.40 and below 0.50, FAIL below 0.40 |
+| Terminal score colors | Green ≥ 0.80, amber ≥ 0.60, red below 0.60 |
+| [Attempt pass threshold](#passk) | Overall attempt score ≥ 0.50 by default; configurable with `--pass-threshold` |
+| [Overall verdict](#skill-lift) | Combines the five dimensions across successful agents, not finding labels |
+| [Skill Lift bands](#skill-lift) | Diagnostic difference between arms, not an absolute metric score |
+
+For a score interpreted on each scale:
+
+| Score | Finding severity | Dimension band |
+| --- | --- | --- |
+| 0.39 | critical | FAIL |
+| 0.40 | warning | NEUTRAL |
+| 0.4545 | warning | NEUTRAL |
+| 0.50 | warning | PASS |
+| 0.6741 | warning | PASS |
+| 0.80 | ok | PASS |
+
+Exactly 0.40 is warning/neutral, exactly 0.50 is pass, and exactly 0.80 is ok.
+For mapped dimensions with multiple inputs, apply the dimension band to the
+combined dimension score, not each input metric. HTML evaluator cards and
+per-trial failure reasons also use the 0.80 gate:
+scores below it can carry reasons. This is the advisory metric scale, not
+the attempt pass threshold.
+
+`tier3 evaluate` writes the findings report only when it shows the detailed
+view (`--verbose`, or when no per-case summary is shown); `validate --tier3`
+never writes it. In that view, it writes `<run>/<agent>/findings.json` for each
+eligible agent with findings. Only the best agent receives generated
+suggestions; other agents use `suggestion_mode: "not_generated"`. The file is
+also absent in a default `tier3 evaluate` run that shows per-case rows without
+`--verbose`, or when execution did not succeed, the skill never activated,
+there are no rewards or usable metric details/scores, or the output path is
+unsafe or symlinked. Stale files are removed at the start of reporting, so old
+advice is not reused. If the best agent has no eligible findings, the report
+returns without writing artifacts for the other agents. Write failures are
+non-fatal.
+
+Scores come from canonical with-skill summary scores when available. Otherwise,
+the report takes the mean over logical trials only when rewards are complete
+(multi-step rewards first combine within each logical trial). If neither source
+is available, that metric is skipped. Reasons and references come from reward
+details and may describe individual trials rather than the aggregate score.
+
+The JSON fields are:
+
+- `skill_name` and `agent`: skill and agent identifiers (512 encoded bytes each).
+- `suggestion_mode`: `remediation` for generated improvement advice,
+ `passing_next_steps` for fallback next steps when no structured remediation
+ was generated, or `not_generated` for other agents (64 encoded bytes).
+- `severity_thresholds`: self-describing advisory metadata:
+ `{"ok_at_or_above": 0.8, "critical_below": 0.4, "scale": "advisory_metric"}`.
+ Older artifacts omit it; readers do not require it and retain the same
+ 0.80/0.40 interpretation.
+- `findings`: at most 134 entries (six standard metrics plus 128 custom metrics).
+ Each has `metric` (256 encoded bytes), `label` (512), `severity` (64),
+ finite numeric `score`, `reasons` (at most four, 512 bytes each; ok findings
+ extract at most two), and `evidence_refs` (at most three).
+- `findings_total`, `findings_shown`, and `findings_truncated`: input finding
+ count, serialized count, and whether any were omitted, including invalid scores.
+- `suggestions`: at most four plain-text suggestions, 4096 encoded bytes each
+ before final redaction; these can include evidence links.
+- `suggestions_v2`: at most four structured suggestions with `suggestion`
+ (4096 encoded bytes), `dimension` (256), `evidence_refs` (at most three),
+ and optional `trial_id` (512). This list can be empty even when fallback
+ `suggestions` are present.
+- `evidence_refs_omitted` and `evidence_refs_omitted_reason`: optional fields
+ added when references are dropped to fit the 2 MiB artifact limit. The former
+ is `true`; the latter is `"findings artifact size limit"`.
+
+Evidence references retain only non-empty `source` (256 encoded bytes),
+`json_pointer` (512), `evidence_id` (512), `kind` (64), `path` (512),
+`label` (256), and `excerpt` (512). Encoded-byte limits count the JSON-escaped
+text, excluding surrounding quotes. Text is truncated when necessary and the
+whole payload is redacted before writing, with a final 2048-character string
+cap. The artifact is bounded for browser loading, not a complete evidence archive.
+
### The five dimensions
With `default` or `default_plus_custom` grading, reports lead with five
diff --git a/src/skillevaluator/tier3/harbor/report.py b/src/skillevaluator/tier3/harbor/report.py
index de77f91a..1ed089c7 100644
--- a/src/skillevaluator/tier3/harbor/report.py
+++ b/src/skillevaluator/tier3/harbor/report.py
@@ -16,6 +16,7 @@ import math
from pathlib import Path
from typing import Any
+from skillevaluator.evaluation.tier3_findings import TRIAL_PASS_THRESHOLD
from skillevaluator.evidence import evidence_ref_identity
from skillevaluator.tier3.eval_core.llm_judge import _ACCURACY_DEDUCTION_SENTENCES, _redact_configured_credentials
from skillevaluator.tier3.harbor import report_data
@@ -35,6 +36,8 @@ from skillevaluator.utils.redaction import redact_sensitive_data, redact_sensiti
logger = logging.getLogger(__name__)
DISPLAY_METRICS = DEFAULT_METRICS
+FINDING_OK_THRESHOLD = TRIAL_PASS_THRESHOLD
+FINDING_CRITICAL_THRESHOLD = 0.4
_MAX_FINDINGS = len(DEFAULT_METRICS) + MAX_CUSTOM_METRICS
_MAX_FINDING_REASONS = 4
_MAX_FINDING_EVIDENCE_REFS = 3
@@ -318,7 +321,7 @@ def _extract_findings(
_seen.add(k)
_refs.append(r)
- if avg_score >= 0.8:
+ if avg_score >= FINDING_OK_THRESHOLD:
reasons = _collect_pass_reasons(metric, trials)
findings.append(
{
@@ -332,7 +335,7 @@ def _extract_findings(
)
else:
reasons = _collect_fail_reasons(metric, trials)
- severity = "critical" if avg_score < 0.4 else "warning"
+ severity = "critical" if avg_score < FINDING_CRITICAL_THRESHOLD else "warning"
finding = {
"metric": metric,
"label": label,
@@ -355,6 +358,13 @@ def _render_findings_body(findings: list[dict[str, Any]]) -> Any:
from skillevaluator.utils.rich_markup import strip_terminal_controls
body = Text()
+ body.append(
+ f" Advisory metric severity: ok ≥ {FINDING_OK_THRESHOLD:.2f} · "
+ f"warning {FINDING_CRITICAL_THRESHOLD:.2f}\u2013<{FINDING_OK_THRESHOLD:.2f} · "
+ f"critical < {FINDING_CRITICAL_THRESHOLD:.2f}. "
+ "Dimension verdicts use pass ≥ 0.50.\n\n",
+ style="dim",
+ )
for finding in findings:
if finding["severity"] == "critical":
icon = "⚠ CRITICAL"
@@ -1003,7 +1013,7 @@ def _prioritized_evidence_rewards(rewards: list[dict[str, Any]]) -> list[dict[st
def _reward_has_failing_signal(reward: dict[str, Any]) -> bool:
for metric in DISPLAY_METRICS:
value = _metric_score(reward, metric)
- if value is not None and value < 0.8:
+ if value is not None and value < FINDING_OK_THRESHOLD:
return True
details = reward.get("details")
if not isinstance(details, dict):
@@ -1013,7 +1023,7 @@ def _reward_has_failing_signal(reward: dict[str, Any]) -> bool:
continue
score = detail.get("score")
numeric_score = extract_custom_metrics({"detail_score": score}).get("detail_score")
- if numeric_score is not None and numeric_score < 0.8:
+ if numeric_score is not None and numeric_score < FINDING_OK_THRESHOLD:
return True
results = detail.get("results")
if isinstance(results, list) and any(
@@ -1122,6 +1132,11 @@ def _bounded_findings_payload(
"skill_name": _bounded_json_text(skill_name, max_encoded_bytes=512),
"agent": _bounded_json_text(agent, max_encoded_bytes=512),
"suggestion_mode": _bounded_json_text(suggestion_mode, max_encoded_bytes=64),
+ "severity_thresholds": {
+ "ok_at_or_above": FINDING_OK_THRESHOLD,
+ "critical_below": FINDING_CRITICAL_THRESHOLD,
+ "scale": "advisory_metric",
+ },
"findings": bounded_findings,
"findings_total": len(findings),
"findings_shown": len(bounded_findings),
diff --git a/tests/evaluation/test_tier3_benchmark_contract.py b/tests/evaluation/test_tier3_benchmark_contract.py
index d7fe01a4..284c0226 100644
--- a/tests/evaluation/test_tier3_benchmark_contract.py
+++ b/tests/evaluation/test_tier3_benchmark_contract.py
@@ -264,11 +264,11 @@ def test_dataset_snapshot_migrates_valid_legacy_routing_summary_in_memory(tmp_pa
@pytest.mark.parametrize("schema_version", ["1.0", "1.1"])
@pytest.mark.parametrize("tamper", ["summary-rule", "dataset", "digest", "algorithm"])
-def test_dataset_snapshot_rejects_tampered_versioned_truth(
- tmp_path: Path, schema_version: str, tamper: str
-) -> None:
+def test_dataset_snapshot_rejects_tampered_versioned_truth(tmp_path: Path, schema_version: str, tamper: str) -> None:
snapshot = json.loads(
- (Path(__file__).parents[1] / "fixtures" / "legacy-routing-dataset-snapshot-1.0.json").read_text(encoding="utf-8")
+ (Path(__file__).parents[1] / "fixtures" / "legacy-routing-dataset-snapshot-1.0.json").read_text(
+ encoding="utf-8"
+ )
)
legacy_summary = dict(snapshot["dataset_summary"])
current_summary = {
diff --git a/tests/tier3/test_findings_severity.py b/tests/tier3/test_findings_severity.py
new file mode 100644
index 00000000..b376b51f
--- /dev/null
+++ b/tests/tier3/test_findings_severity.py
@@ -0,0 +1,89 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+import json
+from pathlib import Path
+
+import pytest
+
+from skillevaluator.evaluation.tier3_report import _load_suggestions_v2
+from skillevaluator.tier3.harbor import report
+
+
+ ("score", "severity"),
+ [
+ (0.39, "critical"),
+ (0.40, "warning"),
+ (0.4545, "warning"),
+ (0.50, "warning"),
+ (0.6741, "warning"),
+ (0.7999, "warning"),
+ (0.80, "ok"),
+ ],
+)
+def test_finding_boundaries(score, severity):
+ findings = report._extract_findings([{"accuracy": score, "details": {"accuracy": {"reason": "Observed result"}}}])
+ assert len(findings) == 1
+ assert findings[0]["severity"] == severity
+ assert findings[0]["score"] == score
+
+
+def _payload(findings=None):
+ return report._bounded_findings_payload(
+ skill_name="example",
+ agent="codex",
+ findings=findings or [],
+ suggestions=[],
+ suggestion_mode="not_generated",
+ suggestions_v2=[],
+ )
+
+
+def test_payload_and_docs_are_in_sync():
+ payload = _payload()
+ assert payload["severity_thresholds"] == {
+ "ok_at_or_above": report.FINDING_OK_THRESHOLD,
+ "critical_below": report.FINDING_CRITICAL_THRESHOLD,
+ "scale": "advisory_metric",
+ }
+ assert payload["findings"] == []
+ assert payload["findings_total"] == payload["findings_shown"] == 0
+ assert payload["findings_truncated"] is False
+ docs = (Path(__file__).resolve().parents[2] / "docs/reports.mdx").read_text()
+ normalized_docs = " ".join(docs.split())
+ for key in {*payload, "evidence_refs_omitted", "evidence_refs_omitted_reason"}:
+ assert f"`{key}`" in docs
+ assert f"`ok` at {report.FINDING_OK_THRESHOLD:.2f} or above" in docs
+ assert f"`critical` below {report.FINDING_CRITICAL_THRESHOLD:.2f}" in docs
+ assert "| Terminal score colors | Green ≥ 0.80, amber ≥ 0.60, red below 0.60 |" in docs
+ assert "`validate --tier3` never writes it" in normalized_docs
+
+
+def test_payload_skips_invalid_scores():
+ payload = _payload([{"score": float("nan")}, {"score": None}])
+ assert payload["findings"] == []
+ assert payload["findings_total"] == 2
+ assert payload["findings_truncated"] is True
+
+
+def test_cli_legend():
+ body = report._render_findings_body([])
+ assert body.plain == (
+ " Advisory metric severity: ok ≥ 0.80 · warning 0.40\u2013<0.80 · critical < 0.40. "
+ "Dimension verdicts use pass ≥ 0.50.\n\n"
+ )
+ assert body.spans[0].style == "dim"
+
+
[email protected]("include_thresholds", [False, True])
+def test_suggestion_reader_accepts_old_and_new_artifacts(tmp_path, include_thresholds):
+ payload = _payload()
+ if not include_thresholds:
+ payload.pop("severity_thresholds")
+ payload["suggestions_v2"] = [{"suggestion": "Check the result", "dimension": "accuracy", "evidence_refs": []}]
+ (tmp_path / "codex").mkdir()
+ (tmp_path / "codex/findings.json").write_text(json.dumps(payload))
+ assert _load_suggestions_v2(tmp_path, {"codex": {}}) == [
+ {"metric": "accuracy", "recommendation": "Check the result", "evidence_refs": []}
+ ]