Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

116 KB
Raw Blame
---
title: "Reports & Results"
description: "Every file SkillEvaluator writes — report formats, file naming, BENCHMARK.md, the Tier 3 results tree, and how to read dimensions, Skill Lift, and pass@k."
---

This page maps everything SkillEvaluator writes to disk and how to read it.
Nothing here requires an API key — it is the output contract for every command,
from a keyless `quality-check` to a full live evaluation.

Two kinds of artifacts exist. Tier 1 and Tier 2 commands render **reports** —
one per format, selected with `-r`. Tier 3 (`tier3 evaluate`) additionally
writes a **results tree**: a timestamped directory of raw scores, trials, and
lift data that `view` and `compare` read back later.

[Harbor](https://github.com/harbor-framework/harbor) is the open-source agent
evaluation framework that runs those Tier 3 trials. SkillEvaluator collects its
job artifacts into the results tree described on this page.

Tier 3 findings describe failed accuracy criteria in plain language and mark
when trajectory evidence is unavailable. Expected and observed text appears
only when both values were recorded in the result.

Tier 3 results include a `dataset_sufficiency` policy assessment in `result.json`.
The status is sufficient only when coverage, pass-rate reliability, and paired
lift-comparison requirements are met; otherwise it is provisional and does not
change the evaluation verdict.

## Report formats

Every Tier 1 and Tier 2 command accepts `-r`/`--report` and `-o`/`--output-dir`.
The flag is repeatable and accepts comma- or space-separated values
(`-r cli,json,html` or `-r cli json html`). By default — with no `-r` at all —
the terminal shows a compact summary and the run writes `html` and `json`
report files. An explicit `-r` is honored exactly, including `-r cli`, which
prints the full terminal report and writes no files. Files land in `reports/`
unless you override it with `-o`. Report output must be outside the source tree
whose publication identity it describes. When the default `reports/` would be
inside the target (for example, after `cd my-skill`), SkillEvaluator safely
reserves the sibling directory `my-skill-reports/` instead. A catalog uses the
same rule and relocates to `<catalog>-reports/`. An explicit in-target `-o` is
rejected; choose an external directory instead. Existing nonempty sibling
directories are reused only when their SkillEvaluator ownership marker is
valid, preventing the default from overwriting an unrelated directory.
For Tier 3, an output directory inside the target repository or an included
skill root must likewise be new, empty, or already carry a valid ownership
marker. This applies to linked-context staging as well as `--copy-repo`. A
nonempty unowned directory is rejected so selecting an authored subtree cannot
silently remove that subtree from the agent's runtime context.

For compatibility, commands launched from a parent directory keep the familiar
layout when the default output is safe to claim: `skillevaluator validate
./my-skill` writes to `./reports/` when that directory is absent, empty, or
already authenticated as SkillEvaluator output. If `./reports/` is nonempty
and unowned, the default relocates to `./my-skill-reports/` instead of writing
into unrelated content. If that sibling is nonempty and unowned too, the run
stops before any check and asks for an explicit `-o`/`--output-dir`.

Ownership is tied to a key stored in your home directory (see
`SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE` in
[Environment Variables](/skills/skillevaluator/environment-variables)). Report folders written
with another key, for example by another account or by a container whose home
directory was discarded, count as unowned. Pass `-o` in those setups.

For `validate`, that compact default is a pipeline view — a per-check ticker
plus a summary. Pass `--verbose` to get the full per-check detail stream in
the terminal instead. Final panels are written to stdout. When output is
redirected or running in CI, tier and check starts, stage changes, measured
completion counts, and completion lines remain visible on stderr. Unchanged
updates are omitted; a running stage emits a heartbeat after 60 seconds without
progress. Counts describe observed work, without estimating a completion
percentage. Interruption tears down the progress display and its worker.

Terminal grades use green for A/B, amber for C, and red for D/F; failed,
incomplete, and unavailable evidence retains its status treatment. Lift that
rounds to zero is neutral. At narrow widths, condition labels, numeric scores,
and their complete bars stack together. The full terminal summary includes
successful quality and lift evidence in Details. Artifact paths use the emitted
filenames and become clickable only on capable interactive terminals; plain,
no-color, redirected, and string-rendered output retains readable paths.

| Format | What it is | Reach for it when |
| --- | --- | --- |
| `cli` | Full Rich terminal report | You want every finding at the keyboard, no files written |
| `json` | Machine-readable JSON (default) | CI needs to parse scores and findings |
| `html` | Standalone HTML report (default) | A human needs to review or archive a run |
| `markdown` | Markdown for PR comments | You want the result posted on a pull request |
| `sarif` | SARIF 2.1.0 JSON | GitHub Code Scanning or other SARIF consumers need inline annotations |
| `junit` | JUnit XML | CI systems that show native test reports need a JUnit artifact |

HTML reports use a shared NVIDIA presentation across all three tiers, with
numbered Insights cards, expandable findings, and tables alongside charts.
Without a saved theme choice, reports containing Tier 3 open in dark mode and
other reports open in light mode, replacing the previous device-theme default.
Saved Light, Dark and System choices take precedence when browser storage is
available; System follows changes to the device preference.
Reports, including charts, work without a network connection.

```bash title="One run, five report files"
skillevaluator validate ./my-skill -r json,html,markdown,sarif,junit -o reports
```

For gating a pipeline on the JSON output, see [Gate Your CI](/skills/skillevaluator/ci-integration).

## Reading the final status

The compact terminal summary, full terminal report, and HTML hero describe the
same command outcome: **Passed required checks**, **Changes required**, or
**Incomplete**. A confirmed failure takes precedence over missing evidence.
Advisory tiers retain their own findings and execution status without changing
the required gate or process exit code.

**Publication eligibility** is a separate, stricter assessment of the recorded
policy, provenance, and evidence coverage. A passed required gate does not imply
publication eligibility. Standalone Tier 3 reports show **Not assessed (Tier 3
only)**. **Best agent score** describes one evaluated agent, rather than approval
of the whole validation run.

When one of several Tier 3 agents fails, the run fails, and a score shown for
it belongs to one completed agent; see
[Completed agents in an incomplete run](#completed-agents-in-an-incomplete-run).

Each tier counts its own passed, failed, incomplete, skipped, and N/A results.
Skipped and explicitly inapplicable checks are excluded from executed-check
denominators. A tier that produced no results is **Not run**; presentation of an
unrequested tier does not create evidence that it ran.

Tier 3 summaries prefer recorded per-condition counters: for example,
**188 scored trials: 101 with skill, 87 baseline**. Condition details retain
expected counts and execution status, so partial execution and early stopping
remain distinguishable. Missing or malformed counters are **unavailable**, and
explicitly skipped arms remain **skipped**. Legacy reports with trial rows but
no counters show **retained rows**, which are observations rather than complete
scored totals. Preview limits and physical retry rows do not replace recorded
attempt counts. These are display projections; canonical JSON and evaluation
payloads are unchanged.

### Tier 3 declarative-check coverage

Runs with declarative checks include a **Declarative-check coverage** section in
terminal summaries, HTML reports, and both benchmark Markdown cards. Run
`result.json` and exported report JSON retain the same structured evidence under
each agent's `declarative_checks`, separated into `with_skill` and
`without_skill` arms. Reports without declarative checks add no section.

Each arm shows checked totals (passed and failed), judged requirements, and
not-checked checks. Evidence entries identify the case, trial, check type,
expected target, verdict, and reason, including passing checks and checks that
were not applicable without the skill. Not checked is not a pass or a failure.

The behavior-check denominator for each reward is **checked + judged**;
not-checked checks are excluded. Reports show both per-reward denominators and
their sum across an arm. That sum is coverage, not a pooled behavior score:
existing trial averaging and scoring remain unchanged. A judge error still
produces no behavior score, even when its recorded denominator is nonzero.
Missing judgment counts are unavailable rather than assumed to be zero.

Coverage uses the reward artifacts already loaded by reporting, not a new
evaluation or a fresh judgment. Failed trials without reward evidence do not
gain invented checks. For retained runs, only available reward evidence can be
reconstructed; existing report truncation notices still apply.

A very large run can make this evidence exceed the result and report size
limits. Then `result.json` and the report keep at most 200 check rows per arm,
failed and not-checked rows first, and leave out the rest. Totals, judged
counts, and per-reward denominators stay exact. Each trimmed arm records
`omitted_checks`, the summary says how many rows were left out, and the report
notes them as `declarative_check_rows` under `report_truncation.omitted`. The
full rows remain in each trial's `reward.json`.

### Tier 2 comparison scopes and evidence

The HTML Tier 2 view separates **Intra-Skill** (redundant content among
its files) from **Inter-Skill** (catalog or collection comparisons). Each
scope shows Checks completed, Duplicate findings, Affected files or skills,
and Catalog entries compared. Completion uses recorded comparison
evidence; a completed check can still fail its gate because it found duplicates.
Refused LLM classifications and file safety errors remain visible findings,
but do not count as confirmed duplicates within a skill.

A completed comparison with no matches shows zero. A skipped catalog comparison,
missing metadata, or an incomplete scan shows the recorded reason and unavailable
counts. Target against catalog comparisons exclude the target's recorded self
entry. Within-skill checks and recorded pairwise scans mark Catalog entries compared
as not applicable. Pairwise scans identify the collection size separately. Affected counts use recorded file or skill paths,
so equal display names do not merge distinct skills. Token or context savings
are not estimated from duplicate counts.

HTML, terminal, and Markdown reports link every recorded Harbor job to its
**Results** and available **Analysis**. HTML uses concise labels from the
recorded agent and condition when available, keeping exact job identities in
Diagnostics. Otherwise it uses the recorded job name or `Job N`. Legacy
single-job and analysis-only links remain usable. Job names alone do not
establish agent or condition ownership.
Links use the existing URL validation and output escaping.
HTML shows these actions in both the Tier 3 tile and Overview when published
URLs are recorded. Keeping local Harbor jobs does not publish them to a hosted
viewer. Saved upload records and persisted run metadata retain their links
when reports are regenerated.

The HTML header shows the SkillEvaluator renderer version independently of the
generation timestamp, beside the product name. The Tier 1 tile shows the
recorded SkillSpector scanner versions. A regenerated report identifies its
current renderer; it does not imply the saved evaluation used that version.
The header retains its native type scale and aligns with the slightly wider
desktop report. On narrow screens it scrolls with the page so that report
navigation and evidence retain usable space.

HTML disclosures show scanner and LLM provenance, plus
sanitized model, evaluator, dataset digest, and evaluated source metadata under
Tier 3 Diagnostics. Missing values remain unavailable. These views use existing
report data and do not read the source tree or contact providers while rendering.
Full downloads and preview omission counts remain available. Recorded native
token and latency details are available in Trials disclosures, alongside the
current resource comparison tables. Both-condition token charts distinguish
cached and uncached prompt tokens without counting cached reads twice.

### Insights, diagnostics, and JSON downloads

Overview combines execution status, scored-attempt coverage and one recorded
next action in a compact summary. Coverage distinguishes scored and unscored
attempts. The native comparison stays open by default, with dimension scores,
the no-skill baseline, recorded lift, Overall and other agents visible. Signal
lists, definitions and detailed reasoning are folded beside those numbers;
partial-comparison labels remain visible. A baseline that ran without an
aggregate score is **baseline quality unavailable**; **baseline not run** is
reserved for a recorded skipped arm. Configured standard dimensions remain
visible with unavailable cells when a run has no measurements; custom-only runs
show their recorded custom metrics. Missing scores never become zero.
Clickable native agent scorecards still open the corresponding Agent Details.
Recorded Harbor Results and Analysis actions remain in the Tier 3 tile and
Overview; missing viewer URLs stay explicitly unavailable. The header, renderer
and scanner version placement, and native body and metric type scales are
preserved.

Insights owns the complete conclusion and recommendation cards, with
their source, verification and evidence links. For saved failed runs, the
renderer replaces recognized repeated error summaries with concise coverage
and actions; authored advice stays verbatim and full errors remain in
Diagnostics. Newly saved failed-run conclusions and recommendations retain
their existing raw error text. Routine configuration and provenance stay in the
closed **Run details & provenance** footer; important execution or comparison
limitations remain visible beside affected results. Rendering a saved run does
not invoke a judge or change its scores.

**Download Full Tier 3 Payload** returns the retained Tier 3 object, including
unavailable values and its recorded verdict. The visible validation outcome
can be stricter because it also accounts for execution failures and required
checks. Presentation defaults do not replace the retained object. Embedded
report JSON uses compact whitespace while preserving the reconstructed snapshot
and its existing escaping and download behavior.

The HTML header's JSON export is a report snapshot, with supplied plugin
static/runtime projections included. Its envelope differs from the CLI's
`-r json` format. A CI report configured with a canonical JSON artifact links
directly to that retained file instead of embedding another report snapshot.
CLI JSON validator rows now include `optional_skip` when recorded as a boolean.
The retained JSON reader round-trips this additive field so an advisory skip
keeps its recorded policy; this does not alter the Tier 3 payload or gate rules.

Plugin evidence appears within its relevant tier. Routine empty observations
share a closed disclosure, and a catalog comparison's skip reason appears once
for its scope. Bounds notices stay with the evidence they qualify and distinguish
omitted rows from shortened text. A shortened historical snapshot cannot
recover the original text; use recorded evidence artifacts when available.

Dataset expectation diagnostics group an identical structured cause only within
the same case attempt and status. Each assertion keeps its requirement ID and
text, recorded status, rubric-check details and evidence excerpts. Members link
to their shared explanation and next diagnostic step; grouping does not remove
assertions or merge different attempts. Shared run errors link to Diagnostics.
Existing preview bounds and omission notices still apply.

### Optional retained native evidence

When a saved Tier 3 payload contains compatible native fields, the HTML report
can display these additional recorded observations:

| Retained evidence | Report location |
| --- | --- |
| Plugin dependency completeness, including recorded MCP resolution detail | Closed disclosures in Agent Details |
| Plugin composition or ablation arm values, uncertainty and any recorded reason | Closed disclosures in Trials |
| Harbor upload or delivery status, errors and retained archive references | Closed disclosure in Diagnostics |
| Per-agent provider safety refusals, mixed-model attempts and comparison basis | Closed disclosures in Agent Details |
| Per-agent recovery notes | Closed disclosures in Diagnostics |

Recorded partial or unreadable plugin coverage, incomplete Harbor delivery, and
provider refusal or mixed-model evidence also receive short visible notices in
Overview, linked to those details. Routine completeness, composition and recovery
evidence stays folded. Display bounds disclose omitted rows or notes and
shortened text. A retained archive reference becomes a link only when an exact,
safe HTTP or HTTPS URL was supplied; paths remain readable references.

These fields appear only when recorded and accepted by the existing safe report
boundary. Their display does not add backend coverage, execute an ablation,
resolve an endpoint, infer a private URL, or change scoring, gates, exclusions
or comparison denominators. Existing public recovery, MCP and plugin comparison
evidence remains available. An MCP warning needs recorded failed-call counts;
a connected-but-unused interpretation needs a recorded connected status and an
explicit zero call count. Missing or null measurements are unavailable, rather
than zero. Hook census and canary runtime summaries still need separate producer
integration and are not supplied by this display compatibility.

### Dataset generation provenance

When a Tier 3 dataset has a generation record, the CLI evaluate summary,
validate's `AGENT_EVAL` section, the HTML overview, and `BENCHMARK.md` show one
dataset line. LLM-generated datasets name their grounding source and review
state. Template-generated datasets show whether templates were requested or
used as a fallback, explain that they are generic starters, and warn readers
to review them before relying on scores. The line adds the number of cases
marked `needs_review`; handwritten datasets and older datasets without
generation metadata have no line. Run reports retain the generation metadata,
with input hashes shortened to their first 12 characters and recorded paths
preserved. The LLM form is `Dataset: generated by <grounding> grounding, review
state <state>`. The template form is
`Dataset: generated from templates (requested|fallback: reason) — cases are
generic starters; review before relying on scores`. Either form appends
`; N cases need review` when applicable.

## File naming

Report filenames follow two patterns, both under the output directory:

| Command | Basename | Overwrites? |
| --- | --- | --- |
| `validate` (full run) | `skillevaluator-output-<timestamp>` | No — each run gets a new, sortable name |
| `quality-check` | `skillevaluator-quality` | Yes — fixed name per format |
| `rubric-eval` | `skillevaluator-rubric` | Yes |
| `security-scan` | `skillevaluator-security` | Yes |
| `pii-scan` | `skillevaluator-pii` | Yes |
| `lint-scripts` | `skillevaluator-script-lint` | Yes |
| `similarity-check` | `skillevaluator-similarity` | Yes |
| `context-optimization-check` | `skillevaluator-context` | Yes |
| `dedup-scan` | `skillevaluator-dedup` | Yes |

The extension matches the format (`.json`, `.html`, `.md`, `.sarif.json`, `.junit.xml`); the `cli` format
prints to the terminal and never writes a file. A full `validate` run's
timestamp is `YYYYMMDDHHMMSS`, so
`-r json` produces something like `skillevaluator-output-20260709141530.json`
and never clobbers the previous run.

## JUnit report

`-r junit` writes one XML file for CI test-report consumers such as GitLab
merge-request test reports, Jenkins, Azure Pipelines, CircleCI, and Buildkite.

| JUnit element | SkillEvaluator source |
| --- | --- |
| `testsuites` | One document per run |
| `testsuite` | One check group (`ValidationResult`) |
| `testcase` | One check, finding, gate failure, or incomplete scanner |
| `failure` | A blocking finding, with `file:line` when recorded; an independent error on a group whose scanners all completed (schema errors, `No SKILL.md found`) as a case named `error`; or a gate failure, such as a minimum score, as a case named `gate_failure` |
| `error` | An `incomplete/<tool>` case for a scanner that did not complete, with `type="incomplete"` and the scanner's own reason when one was recorded (for example `skillspector not installed`); and, on a group with an incomplete scanner, each independent error that is not a gate failure (a crash or unusable scanner output, such as `Bandit did not return valid JSON; scan did not complete`) as a case named `error` with `type="execution"`. Other checks and findings in that group keep their real passed or failed status |
| `skipped` | A check group that did not run and does not fail the gate, as one case with its skip reason. A blocking group that was skipped is a `gate_failure` instead |
| `system-out` | Score and threshold when recorded, and non-failing finding text |

Counts on `testsuites` and each `testsuite` (`tests`, `failures`, `errors`,
`skipped`, `time`) match the cases they contain. A finding case is named
after its check and location, `check_name (path:line)` (or `check_name (path)`
when no line was recorded), so adding or reordering findings does not rename
the others in CI test history. File paths are relative to the Git repository
root that contains the target (the target folder when it is not in a Git
repository), from `validate` and from single-check commands alike. SARIF uses the
same base, resolution rules, and percent-encoding, so a file
name that is not valid UTF-8 stays distinct (`caf%E9.md`). In other text such a
name is backslash-escaped (`caf\xe9.md`), then characters XML 1.0 forbids are
stripped. An incomplete scan is never emitted as an all-passed suite, and a
run that failed on one never reports `failures` and `errors` both zero.

Failures follow the same gate as the exit code. When `validate` marks a group
as blocking and it fails the gate with no failing case of its own, for example
a scored Tier 3 verdict of FAIL or NEUTRAL under `--block-on-agent-eval`, the
suite gets a `gate_failure` case whose message is the cause: the first error,
the Tier 3 verdict, or the skip reason. An advisory group (Tier 3 without
`--block-on-agent-eval`, Tier 2 with `--no-block-on-dedup`) never adds a
`failure` or `error`; its findings, errors, and execution errors are kept in
`system-out` with an `Advisory, does not gate:` prefix, and its incomplete
scanners are `skipped`.

## HTML report

The HTML report is one self-contained file. It opens on the answer and puts
the evidence below it:

- **Header and summary.** Branding, one Theme control and **Download JSON**
  sit above the target name and required-validation result. Publication
  eligibility stays separate, with its reasons available on demand. One set
  of tier cards opens the detailed evidence.
- **Tier summaries.** Each tier shows its own completion and findings.
  Tier 3 keeps available quality metrics in compact tiles; execution status,
  coverage and a recorded next action share the Overview summary. An incomplete
  run with unavailable quality measurements shows that state directly. Full
  Insights cards appear once in Insights; Overview links to them.
- **Findings and evidence.** Every check with its findings and suggested fixes.
  Failed checks come first. Checks that report a raw ID, such as
  `SCRIPT_LINT`, show a plain name with the ID beside it. Scanner diagnostics
  and gate failures are always shown and printed; raw scanner lines that only
  repeat a finding are folded away.
- **Run details.** A closed footer holds routine identity, configuration,
  target paths and provenance. Passive plugin inventories and source details
  also stay folded, with recorded omissions and limitations disclosed.

Clickable tier cards replace the previous left sidebar and sticky mobile tier
bar. Use the tier cards to open a tier. A comprehensive `validate` report
keeps all three tiers visible, including **Not run** and **Not run (not requested)**
when evidence is absent. Targeted reports keep their narrower scope. On small
screens, navigation and controls wrap within the viewport.
Tier 3 retains Overview, Insights, Agent Details, Trials, Dataset and Diagnostics.
For its tabs, Left/Right and Home/End move focus; Enter or Space opens the focused
view. Finding links clear incompatible filters, open their evidence and move focus
to the exact finding. Direct agent links open and focus the named agent panel;
section links open at the start of the section.

The **Theme** control offers System, Light and Dark. System follows live device
changes; an explicit choice persists until System is selected again. Reports use
native theme-aware scrollbars and respect reduced motion. Without JavaScript,
all tier and agent evidence remains readable. Printing exposes supporting evidence
and all retained Dataset case cards, including cases hidden by a screen filter.
It prepares charts synchronously for print and restores disclosure and filter
state when returning to the screen. Existing report preview limits still apply.

Reports use 16-pixel body text with relaxed line spacing, distinct label/value
gaps, and consistent space between cards, findings and table rows. Tier 3 view
tabs wrap on narrow screens; wide evidence tables keep their own horizontal
scroll area. The same reading rhythm applies in Light and Dark themes.

Tier 3 charts include Chart.js 4.5.1 inside the HTML file and work offline;
they make no CDN or source-map requests. Charts initialize when their view or
disclosure becomes visible, after the selected theme is applied, and reuse their
instances across tab and theme changes. Chart data contains only the fields the
charts consume. Each chart describes its measurements and links to the underlying
data table. If JavaScript or canvas initialization is unavailable, a visible
message points readers to the data. Printing prepares all charts, including
those in previously closed disclosures. **Chart software notices** retains the
full MIT licenses for Chart.js and its bundled color library. Tier 1 reports do not
load or embed these chart resources.

A finding is one problem a check reports, with a severity. The same problem
reported on several lines, or in several files or skills, counts once at its
highest severity; each location is an occurrence, including each line a
PII finding lists when one value repeats. When the unique, occurrence and
raw record counts differ, or when some errors or warnings have no finding
behind them, the findings number shows the occurrences, the raw records and
those errors and warnings beneath it, in the same words as the Markdown
report's Total Issues. The workflow scope note for a stage that configuration
skips, such as LLM checks under `--no-llm`, is not counted as a warning there. A finding in a failed check blocks the
verdict; a finding in a passed check is advisory. A quality score below the
configured minimum fails the Quality score check, so it also blocks the
verdict.

The HTML report, the CI index and the Pages history use the same verdict
words: **Passed**, **Changes required**, **Incomplete**, **Skipped** and
**Neutral**. When no check ran, the verdict is Skipped, not Passed. A tier
whose gate does not block, such as deduplication with `--no-block-on-dedup` or
live evaluation without `--block-on-agent-eval`, shows **Failed (advisory)**
when it fails, because it does not require changes. Publication eligibility is
shown separately from the command result. Summary reasons count required and
advisory checks individually, including when a tier contains both. Eligibility is
shown next to the verdict, because a report can pass its checks and still not
be eligible for publication. Tier 3 reports default to dark and other reports to
light; selecting System follows the device preference. Reports always print
dark text on white paper.

## BENCHMARK.md

When the target is a skill, `validate` always writes `BENCHMARK.md` to the
output directory — regardless of which `-r` formats you picked, and even when
validation fails. It is a decision-first publication card: the overall verdict
and recommendation come first, followed by per-tier evidence, evaluator and
dataset provenance, agent/model details, and a dimension-by-dimension
`Baseline → Skill Uplift` table when Tier 3 ran. Public cards omit validation
profile names and absolute host paths.

Table scores are rounded half-up to one decimal place. A score near a dimension
verdict threshold keeps as many extra decimals as it needs to stay in its band,
so `0.4995` is shown as `49.95%`, not as a passing `50.0%`. Uplift is the
difference of the two displayed scores, so a cell such as
`58.4% → 94.8% (+36.4 points)` always adds up. A score outside 0–1 is shown as
`Not available`.

A missing score is never shown as zero. When a standard signal was active for a
successful agent but produced no with-skill score, the dimension reads `Not
scored with the skill — baseline 50.0%; uplift unavailable` (or `Not scored — a
signal this dimension needs produced no score` without a baseline). A dimension
whose signals had nothing to judge in any eval case reads `Not applicable`; it
has no score, so the agent cannot PASS. A
baseline-only run shows `Baseline 67.0% — baseline-only run; no with-skill score
or uplift`.

The collapsed **Scoring Methodology** section lists, for every dimension, the
question it answers in plain words, its weighted signals, where their evidence
comes from (deterministic trace or trajectory checks, or an LLM judge), and
whether this run has a no-skill baseline score for it. It ends with a short key
that separates the overall verdict, dimension scores, the Overall row, uplift,
pass@k, and the Tier 1 Quality Score.

The card carries one overall verdict — `PASS`, `NEUTRAL`, `FAIL`, or
`INCOMPLETE`. It fails closed: if a required scanner produced no trustworthy
evidence, the verdict is `INCOMPLETE` and the card explicitly says not to use
it to recommend publication. A Publication Recommendation section appears only
on a clean `PASS`; `NEUTRAL` means the evidence is complete but at least one
required dimension remains below the pass band.

Tier 1 is always required for publication. Tier 2 and Tier 3 are also required
by default, so a missing, skipped, or incomplete required tier produces
`INCOMPLETE` rather than a publication recommendation. Catalog orchestration
may explicitly persist `benchmark_policy.tier2_required = false` or
`benchmark_policy.tier3_required = false`; the card discloses each exception
instead of implying that the tier ran. An optional tier that is present but
incomplete still makes the card `INCOMPLETE` because malformed evidence cannot
certify publication. A default-passed result object is not sufficient. Tier 1
and Tier 2 evidence must carry a valid `publication_evidence` marker written by
a built-in command wrapper, and every recognized result must record a success,
finding, or positive check count that proves it executed. Tier 1 must also
cover every default check that applies to the content type: schema, security,
PII, license, code integrity, and Unicode, plus version, quality, and lint for
skills. A run narrowed with `--checks` is reported but cannot certify Tier 1. Legacy and custom
results remain visible, and their failures still block publication, but they
cannot satisfy a required tier or create a waiver. Generic per-result `optional`
metadata is not a publication waiver; only the resolved `benchmark_policy` from
recognized evidence can make Tier 2 or Tier 3 optional. An unrecognized `true`
claim can conservatively require a tier, but an unrecognized `false` claim
cannot waive one.

Agent-evaluation payload metadata and its summary are duplicated claims at the
same precedence level and must agree. A conflict between valid booleans at that
level resolves to required evidence. Invalid entries do not count as policy
values; when the level contains no valid boolean, resolution falls through to
the next lower-precedence source, and no valid value anywhere means required.
The consistent payload/summary claim takes precedence over recognized peer result
metadata. Conflicting booleans among peer results likewise resolve to required
evidence so aggregation order cannot create a publication waiver.
Every Tier 1, Tier 2, and Tier 3 result used for publication carries a
`publication_target` with the exact NFC filesystem-entry name, a SHA-256 source digest,
and the versioned `skill-evaluator-source-tree/2` recipe. The recipe binds
author-owned relative paths, node kinds, executable bits, and file contents
while excluding only known generated artifacts. In particular, authored
`evals/` inputs are included. The v2 exclusions are any-depth `.git`, `.venv`,
`__pycache__`, and `node_modules`; root `.evals`, `.results`, `.versions`,
`results`, and `versions`; `evals/results`; and the generated root files
`BENCHMARK.md`, `skill-card.md`, and `skill.oms.sig`. Filesystem case aliases
are treated according to the host filesystem, while a case-distinct authored
name on a case-sensitive filesystem remains covered. A same-named file below
another authored directory is still included. Tier 3 runtime projection and
full `--copy-repo` staging omit the same generated inputs so excluded generated
bytes inside the target cannot affect evaluation evidence. The Tier 1
file-walk checks and Tier 2 skip the generated-artifact names (`results`,
`versions`, `evals`, their dot variants, `BENCHMARK.md`, `skill-card.md`, and
`skill.oms.sig`) at any depth. When one of those names appears below the target
root, the digest would certify content that those checks never read, so the
Tier 1 run records the path and publication stays `INCOMPLETE` until it is
renamed. The root `evals/` directory holds Tier 3 inputs. The Tier 1 file-walk
checks skip it by design, while the Gitleaks secret scan and SkillSpector read
its authored files. Both still skip generated output such as `evals/results`.
Gitleaks also skips every file under a `tests/`, `examples/`, `fixtures/`, or
`mocks/` folder, or the singular form, such as a Harbor task grader in
`evals/harbor/<case>/tests/`.
Split-job results must match this identity exactly. Tier 3 runs that stage any
linked/full repository context or external reference/workspace skills, or task
inputs and custom environments that carry `.git`, `.venv`, `__pycache__`, or
`node_modules` entries, remain runnable but are marked `INCOMPLETE`, because
that projection is not fully determined by this target-only publication
identity. Legacy anonymous results
remain readable, but their publication status is `INCOMPLETE`.

Built-in Tier 1 and Tier 2 results also carry a versioned
`publication_evidence` object with exact `schema_version`, `producer`, `tier`,
and canonical `check_id` fields. Reporters accept only the documented built-in
producer/check combinations. This marker is provenance inside trusted local or
CI artifacts; it is not a cryptographic signature for accepting result files
from an untrusted party.

Publication model, evaluator, environment, and skill identities reject reserved
placeholder text after Unicode confusable-skeleton comparison. This prevents a
visually spoofed placeholder such as one containing a Cyrillic or Greek letter
from becoming publication evidence while preserving ordinary multilingual
identities.

Payload and summary policy claims are accepted only when their containing
evidence matches that exact publication target; evidence from another source
cannot waive a required tier. A skipped Tier 3 payload must satisfy the same
target binding before its policy can waive evidence. A completed Tier 3 payload
also duplicates its run-owned `run_id` in the payload and summary; missing or
contradictory run identity is incomplete publication evidence.
When multiple complete Tier 3 payloads disagree, reporters select the most
conservative effective verdict (`FAIL`, then `NEUTRAL`, then `PASS`) before
applying deterministic tie-breakers.

Execution controls do not rewrite that publication policy. `--no-dedup` and
`--tiers 1,3` select which validators run, while `--no-block-on-dedup` changes
whether Tier 2 affects the command exit code. None of those flags persists a
publication waiver. Without explicit policy metadata, a run that omits Tier 2
therefore exits according to its CLI gate but still produces an `INCOMPLETE`
publication card.

### Evaluated source provenance

The Evaluation Metadata section distinguishes the source tree that was evaluated
from the evaluator build that evaluated it:

```text
- Evaluator version: `0.8.2`
- Evaluated source: `NVIDIA/NVFlare`
- Evaluated source revision: `2263a2ebdab903e87f7e7c0a001d22c3a926a9cf`
- Evaluator container revision: `ghcr.io/nvidia/skillevaluator@sha256:0117bc2e...`
```

Without that separation, two skills evaluated from different repositories by the
same evaluator container publish cards whose only recorded revision is the shared
container tag, and a reader cannot tell which version of a skill was evaluated or
whether the card has gone stale.

The evaluated source identity is supplied by the orchestration input and carried
through the evaluation-result contract unchanged. It is never inferred from
repository state while rendering, because the tree that renders a card is the
evaluator checkout rather than the evaluated skill's source. The supported seams
are:

- `--evaluated-source-repository`, `--evaluated-source-revision` and
  `--evaluator-container-revision`, accepted by both `validate` and
  `tier3 evaluate`, which record the identity on the card and, for a Tier 3 run,
  persist it into that run's `run_config.json`;
- the `evaluated_source` argument to `build_agent_eval_payload` or
  `agent_eval_result_from_directory`;
- an `evaluated_source` object in the run's `run_config.json`. Note that
  SkillEvaluator writes that file itself, so a publication pipeline using this
  route has to add the key to the generated run artifact before the card is
  rendered;
- `metadata["evaluated_source"]` on any validation result, which is how a card
  that publishes without a completed Tier 3 run records its identity.

Where any two carriers supply a field and disagree, rendering fails rather than
silently choosing one, so the order results happen to arrive in can never decide
which source tree a card names. `Evaluated source revision` records a full Git
object id of 40 or 64 hex characters, or a canonical `sha256`/`sha384`/`sha512`
digest of the evaluated skill content at that algorithm's own width when a run
has no upstream commit. A short revision prefix is refused, because it can grow
a second match as a tree grows. `Evaluator container revision` records an image
reference validated component by component: a repository path of at most 255
characters, an optional tag of at most 128 characters, and an optional digest at
its own algorithm's width. The 255 bound measures the path once the registry
host has been split off it. Path components are lower case, as the OCI grammar
requires, while a registry host may use any case and is read as a host only when
it is `localhost`, carries a dot, or carries a port. A bare name is refused
because it identifies a repository rather than the build that ran, and a
published PASS additionally requires the digest, or a full implementation
revision, because a tag can be repointed after the card is published. Fields
the orchestration did not supply render as
`not recorded (not supplied by the orchestration input)`.

Unlike the evaluated source identity, `Evaluator version` names the evaluator
build. It is the version recorded in the Tier 3 result, because that build ran
the live evaluation. When the run has no Tier 3 result that records one, as in
`validate --tiers 1`, `validate` writes its own version there instead of
`not recorded`.

Keep the file with the skill and refresh it whenever the skill, eval dataset,
agent/model, evaluator version, scoring policy, attempt policy, or execution
environment materially changes. Follow the
[rollout and backfill runbook](/skills/skillevaluator/benchmark-rollout) when regenerating existing
published cards.

## Safety guarantees

Report writes are hardened: every file is written atomically (a temporary file
renamed into place), and SkillEvaluator refuses to write through symlinks,
Windows junctions, or paths with parent traversal. A run that is interrupted
mid-write never leaves a half-written report, and a report path can't be used
to redirect output somewhere unexpected.

## Tier 3 results on disk

`tier3 evaluate` does not use `-r`/`-o`. It writes a results tree to the first
of:

1. `--results-dir <dir>` on the command line
2. the `SKILLEVALUATOR_RESULTS_DIR` environment variable
3. `<skill>/evals/results/` (the default)

With an external root (the first two options), each skill gets its own
subdirectory — runs land under `<dir>/<skill-name>/`, not directly under
`<dir>`. Configured roots must be outside the skill. The canonical
`<skill>/evals/results/` path is the only supported in-skill result root; this
keeps generated runs outside the versioned source-tree digest used to combine
publication evidence.

Each run creates a collision-safe directory named by its run ID
(`YYYYMMDD_HHMMSS_<pid>_<12-hex-nonce>`). Where symlinks are supported, a
`latest` symlink points at the newest run; otherwise readers select the newest
completed run directly:

<FileTree>
- evals/
  - results/
    - latest/
    - 20260709_141530_12345_a1b2c3d4e5f6/
      - result.json
      - run_config.json
      - attempt_policy.json
      - report.html
      - tier3-benchmark.md
      - comparison.json
      - codex/
        - lift.json
        - pass_at_k_lift.json
        - with-skill/
          - summary.json
          - trials/
        - without-skill/
          - summary.json
          - trials/
</FileTree>

- **`result.json`** — the bounded run summary: run ID, per-agent scores and
  lift, pass@k aggregates, the resolved run configuration, and execution
  status. When repeated case, security, or failure details would make this
  browser-facing file too large, `result_projection` records exactly which
  fields were omitted and points to the canonical per-agent JSON artifacts by
  relative path, byte size, and SHA-256 digest. The in-process return value
  remains complete; readers rebuilding a report from disk follow those
  artifact references.
- **Baseline provenance** — `result.json` and `run_config.json` include a
  `baseline` block with mode `fresh`, `reused`, or `skipped`. Reused baselines
  record `source_run_id`, `source_evaluated_at`, `source_run_dir`, and
  `source_identity`. A `latest` fallback records bounded `rejected_candidates`
  and a `reuse_reason`. `comparison_identity` separates baseline configuration
  from skill revision. Each agent's `execution_counts` and `measured_usage`
  distinguish fresh execution from reused evidence (not new spend). Missing
  token/cost measurements stay unavailable rather than becoming zero. Reused
  baseline rows remain in the new run; its without-skill summary is marked
  `reused`.
- **`run_config.json`** / **`attempt_policy.json`** — exactly how the run was
  configured: environment, attempts, concurrency, grading mode, pass threshold.
- **`report.html`** — the human-readable report, generated automatically at
  the end of every run.
- **`tier3-benchmark.md`** — a Tier 3-only benchmark written with
  `report.html`. It leads with the
  [completed agent's result](#completed-agents-in-an-incomplete-run) and shows
  publication as **NOT ASSESSED**, because it covers Tier 3 only, even when it
  comes from a `validate` run that also ran Tiers 1 and 2; its name keeps
  publication tooling from reading it as a `BENCHMARK.md` card. At the end of
  every Tier 3 run, from `tier3 evaluate` or a `validate` run that includes
  Tier 3, the two files are written as a pair: if `tier3-benchmark.md` cannot
  be written, `report.html` is removed too, and `result.json` records the
  reason in `warnings` with `report_status: "degraded"`. [`view`](#view)
  regenerates only a missing `report.html`, never `tier3-benchmark.md`.
- **`comparison.json`** — the cross-agent comparison table; written only when
  you ran more than one agent.
- **`<agent>/lift.json`** and **`<agent>/pass_at_k_lift.json`** — per-metric
  deltas between the two arms; written only when both arms produced scores
  (a `--skip-baseline` or `--skip-with-skill` run has neither).
- **`<agent>/findings.json`** — bounded per-metric findings and improvement
  suggestions; written by `tier3 evaluate` only when the detailed view is
  shown (`--verbose` or when no per-case summary is shown), never by
  `validate --tier3`; see [Findings and severity](#findings-and-severity).
- **`<agent>/{with-skill,without-skill}/`** — each arm's `summary.json`
  (scores, dimensions, pass@k, trial counts) and a `trials/` directory of
  per-trial artifacts: the trial's `reward.json`, Harbor's `trajectory.json`
  when available, and redacted, size-capped diagnostic copies such as
  `trial.log`, `exception.txt`, and agent logs (`codex.txt`, `opencode.txt`).
  With standard grading, `reward.json` records `trajectory_source` (the
  trajectory or agent log that was graded) and, under each LLM judge's
  `details`, an `omitted` record showing whether its evidence was truncated to
  fit the [evidence budget](/skills/skillevaluator/tier3-grading#evidence-budgets) or, for
  `behavior_check`, the final-response section limit
  (`SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT`). For `behavior_check`, the
  record's `reason` names each setting that applied, and
  `incomplete_sections`, `required_complete` and `required_missing` say which
  evidence did not reach the judge whole. A `behavior_check` whose required
  evidence is incomplete is reported as an evaluation error with no score,
  which also leaves the trial's `overall_score` null; see
  [evidence budgets](/skills/skillevaluator/tier3-grading#evidence-budgets).
- **`<agent>/baseline_timeout_tolerance.json`** — written only for an agent
  kept scored by `--tolerate-baseline-timeouts`: the baseline attempts that hit
  the agent execution timeout. `result.json` carries the same record as
  `agents.<agent>.baseline_timeout_tolerance`, a refused agent's
  `agents.<agent>.baseline_timeout_tolerance_rejection` code, and the run-level
  `baseline_timeout_tolerance` (tolerated agents and the launch errors they
  explain). See [Baseline timeouts](/skills/skillevaluator/tier3-run-reference#baseline-timeouts).

<Note>
  Read commands (`view`, `compare`, `create-eval-dataset --refine`) honor the
  same precedence, then fall back to the legacy `<skill>/evals/results/`
  location — so old runs stay visible after you configure
  `SKILLEVALUATOR_RESULTS_DIR`. Current timestamped runs are selected only when
  stable regular `run_config.json` and matching atomic `result.json` artifacts
  are present. Unmarked runs created before run-level `result.json` remain
  readable only when `run_config.json` has a recognized historical task source
  and complete agent metadata, and every configured agent has one stable,
  identity-matched summary satisfying the full historical score, dimension,
  pass-at-k, and completion schema. Authenticated summaries from before status
  fields are consumed as successful; coherent status-era failed summaries stay
  visible without contributing scores. Any marker presence prevents an
  incomplete current run from being interpreted as legacy output. The results
  tree is generated output: don't commit it.
</Note>

## Reading a live-eval report

### The terminal summary

A finished `tier3 evaluate` run prints two tables and closes with a compact
**Artifacts** panel. **Results by Evaluator** lists each evaluation signal
with its per-arm scores and Skill Lift; **Results by Dimension** rolls those
signals up into the five dimensions below. A metric the run couldn't score
renders as `NO SCORE` — it is never coerced to 0.0 — and unscored metrics are
omitted from the averages, so `overall_score` can be `null` in the JSON. A
metric that does not apply to an arm renders as `N/A`, and so does its Skill
Lift: for example Behavior Check when the cases have no `assertions` (legacy
`expected_behavior`), Accuracy when they have no `expected_output` (legacy
`ground_truth`), or Skill Execution in a baseline with no skill it could
activate. N/A metrics are left out of the scores and the lift. In a
baseline-only run (`--skip-with-skill`) the with-skill cells read `skipped`,
because that arm never ran, and so does the Skill Lift of every row that is
not N/A. The
Artifacts panel shows the HTML report path, the output directory, and — only
when a retained Harbor jobs directory exists — a ready-to-paste `harbor-view`
command.

### Next steps for the run

Below the per-case fixes, the case summary in the terminal, `report.html`,
and the Markdown report list up to three **Next steps** for the run as a
whole: what a single case cannot show. The steps come from fixed rules, so
they also appear when no LLM is configured. Each step names:

- the **target** to change: the `evaluation` settings (re-run, more
  attempts), the `dataset` (more cases, negative controls), or the `agent`
  (compare agents or models);
- the **problem** observed in this run and the **evidence** behind it (case
  ids, attempt marks, or per-agent pass-rate changes);
- the **action** to take, with the flag or file to change;
- the **check** that would show the change worked.

| Step | Shown when |
|------|------------|
| Incomplete run | Attempts did not produce a score and no per-case run error already says so |
| Inconsistent results | A case passed on some attempts and failed on others (more `--n-attempts`) |
| Single attempt | Every case ran once |
| Agent-dependent effect | Two or more agents differ by 20 or more points in pass-rate change, and the best agent gains at least 10 points |
| More cases | Fewer than five skill cases ran, unless the case fixes already ask for new cases |
| Negative controls | The dataset snapshot has no case with `expected_skill: null` or `should_trigger: false` |
| Agent experiment | Only one agent ran and the skill showed no clear gain |

A step marked `hypothesis` is a cause or remedy to test, not a finding: for
example, that one agent follows the skill differently. A step marked
`limited` rests on an incomplete run, so re-run first. Steps never suggest
removing cases or loosening expectations to raise a score. Baseline-only runs
and runs where the skill never activated get no next steps. The Tier 3 payload
carries them as `next_steps`, with the `kind`, `target`, `problem`,
`evidence`, `action`, `check` and `strength` fields.

### Findings and severity

The terminal Findings panel uses **advisory metric severity**, not dimension
verdicts: `ok` at 0.80 or above, `warning` from 0.40 to below 0.80, and
`critical` below 0.40. These labels identify opportunities for improvement;
a warning does not mean a dimension failed. Boundaries use the underlying
score, not its rounded terminal display.

| Scale | Meaning and boundaries |
| --- | --- |
| Advisory finding severity | Per-metric `ok` ≥ 0.80, `warning` ≥ 0.40 and below 0.80, `critical` below 0.40 |
| [Dimension bands](#the-five-dimensions) | PASS ≥ 0.50, NEUTRAL ≥ 0.40 and below 0.50, FAIL below 0.40 |
| Terminal score colors | Green ≥ 0.80, amber ≥ 0.60, red below 0.60 |
| [Attempt pass threshold](#passk) | Overall attempt score ≥ 0.50 by default; configurable with `--pass-threshold` |
| [Overall verdict](#skill-lift) | Combines the five dimensions across successful agents, not finding labels |
| [Skill Lift bands](#skill-lift) | Diagnostic difference between arms, not an absolute metric score |

For a score interpreted on each scale:

| Score | Finding severity | Dimension band |
| --- | --- | --- |
| 0.39 | critical | FAIL |
| 0.40 | warning | NEUTRAL |
| 0.4545 | warning | NEUTRAL |
| 0.50 | warning | PASS |
| 0.6741 | warning | PASS |
| 0.80 | ok | PASS |

Exactly 0.40 is warning/neutral, exactly 0.50 is pass, and exactly 0.80 is ok.
For mapped dimensions with multiple inputs, apply the dimension band to the
combined dimension score, not each input metric. HTML evaluator cards and
per-trial failure reasons also use the 0.80 gate:
scores below it can carry reasons. This is the advisory metric scale, not
the attempt pass threshold.

`tier3 evaluate` writes the findings report only when it shows the detailed
view (`--verbose`, or when no per-case summary is shown); `validate --tier3`
never writes it. In that view, it writes `<run>/<agent>/findings.json` for each
eligible agent with findings. Only the best agent receives generated
suggestions; other agents use `suggestion_mode: "not_generated"`. The file is
also absent in a default `tier3 evaluate` run that shows per-case rows without
`--verbose`, or when execution did not succeed, the skill never activated,
there are no rewards or usable metric details/scores, or the output path is
unsafe or symlinked. Stale files are removed at the start of reporting, so old
advice is not reused. If the best agent has no eligible findings, the report
returns without writing artifacts for the other agents. Write failures are
non-fatal.

Scores come from canonical with-skill summary scores when available. Otherwise,
the report takes the mean over logical trials only when rewards are complete
(multi-step rewards first combine within each logical trial). If neither source
is available, that metric is skipped. Reasons and references come from reward
details and may describe individual trials rather than the aggregate score.

The JSON fields are:

- `skill_name` and `agent`: skill and agent identifiers (512 encoded bytes each).
- `suggestion_mode`: `remediation` for generated improvement advice,
  `passing_next_steps` for fallback next steps when no structured remediation
  was generated, or `not_generated` for other agents (64 encoded bytes).
- `severity_thresholds`: self-describing advisory metadata:
  `{"ok_at_or_above": 0.8, "critical_below": 0.4, "scale": "advisory_metric"}`.
  Older artifacts omit it; readers do not require it and retain the same
  0.80/0.40 interpretation.
- `findings`: at most 134 entries (six standard metrics plus 128 custom metrics).
  Each has `metric` (256 encoded bytes), `label` (512), `severity` (64),
  finite numeric `score`, `reasons` (at most four, 512 bytes each; ok findings
  extract at most two), and `evidence_refs` (at most three).
- `findings_total`, `findings_shown`, and `findings_truncated`: input finding
  count, serialized count, and whether any were omitted, including invalid scores.
- `suggestions`: at most four plain-text suggestions, 4096 encoded bytes each
  before final redaction; these can include evidence links.
- `suggestions_v2`: at most four structured suggestions with `suggestion`
  (4096 encoded bytes), `dimension` (256), `evidence_refs` (at most three),
  and optional `trial_id` (512). This list can be empty even when fallback
  `suggestions` are present.
- `evidence_refs_omitted` and `evidence_refs_omitted_reason`: optional fields
  added when references are dropped to fit the 2 MiB artifact limit. The former
  is `true`; the latter is `"findings artifact size limit"`.

Evidence references retain only non-empty `source` (256 encoded bytes),
`json_pointer` (512), `evidence_id` (512), `kind` (64), `path` (512),
`label` (256), and `excerpt` (512). Encoded-byte limits count the JSON-escaped
text, excluding surrounding quotes. Text is truncated when necessary and the
whole payload is redacted before writing, with a final 2048-character string
cap. The artifact is bounded for browser loading, not a complete evidence archive.

### The five dimensions

With `default` or `default_plus_custom` grading, reports lead with five
dimensions, each answering one question. The
[Metrics Reference](/skills/skillevaluator/metrics-reference) gives each one's exact
formula, range, units and evidence source, and shows how to recompute it from
the files on this page:

| Dimension | Question answered | Signals and source |
| --- | --- | --- |
| Security | Is it safe to use? | `security`, from deterministic trace checks |
| Correctness | Is the answer correct? | `accuracy`, from the `judge_accuracy` LLM judge |
| Discoverability | Was the right skill loaded when needed? | `skill_execution`, from deterministic trajectory checks |
| Effectiveness | Did the skill help complete the task? | `goal_accuracy` from an LLM judge; `behavior_check` from its judge and directly evaluated declared checks (checks-only cases make no judge call) |
| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency`, from deterministic trajectory checks |

`skill_efficiency` averages skill routing with tool efficiency, the share of
scored tool calls that did productive work. For Claude Code, Codex and OpenCode
tool names, these calls are productive: reading, searching, listing, writing,
editing or patching files (`Write`, `Edit`, `MultiEdit`, `apply_patch`, `LS`,
OpenCode `list`, Codex `list_dir` and `grep_files`, and similar), running
commands and activating a skill. MCP tools named `mcp__server__tool`, as Claude
Code records them, are productive too. Codex records an MCP call under the bare
tool name and OpenCode as `server_tool`, so these calls are classified by their
name like any other tool, and most such names count as wasted. A Codex name
with a namespace, such as `functions.apply_patch`, is classified by the part
after the last dot. Help, version and install commands are wasted, whether
written as text or recorded as an argument list, as is any tool outside these
classes. Only commands are checked for help, version and install words, so a
file path or patch text that mentions `pip install` is not waste.

Planning and bookkeeping calls are neutral and left out of the score:
`TodoWrite`, `update_plan`, task lists (`TaskCreate`, `TaskUpdate`, `TaskList`,
`TaskGet`), `EnterPlanMode` and `ExitPlanMode`, background-shell polling such
as `BashOutput` and `KillShell`, `TaskStop`, `ToolSearch`, and sub-agent calls
such as `Task` and Codex `spawn_agent`, `wait` or `wait_agent`. Codex
`write_stdin` with no input, or only Enter or Ctrl-C, polls its session and is
neutral, as is a `Monitor` watch with no command. A command typed with
`write_stdin` or run by `Monitor` is classified like any other command. The
reason names how many calls were not counted. A trial whose only calls are
neutral did no work: it scores tool efficiency 0.0, so its `skill_efficiency`
is 0.0, the same as a trial with no tool calls.

Tool-efficiency checks classify explicit searches rooted at `/` as wasted work,
whether or not the command names the expected skill or script. This includes
native `Grep`, `Glob`, `LS`, OpenCode `list`, Codex `list_dir` and `grep_files`
calls on `/`, MCP filesystem searches and listings whose `path` or `dir_path`
is `/`, and commands recorded as an argument list, such as Codex
`["bash", "-lc", "find / -name x"]`, which are read as the command they run.
An MCP call counts as a filesystem search or listing when its tool name
contains `search`, `list`, `tree`, `find`, `glob`, `grep` or `directory`, or
when its server name contains `filesystem`. Other MCP tools use `path` for
other things: `/` in a GitHub `get_file_contents` call is the repository root,
and that call is productive. Scoped searches and `/` used as a search pattern
are productive. This rule recognizes common search commands, including ones with
redirections or wrapped in `sh -c`; it does not interpret expansions, comments,
here-documents or other dynamically constructed shell programs. An argument
list runs without a shell, so `$`, backticks and `#` in its own words are
literal.


Every evaluator and dimension score is on a 0.0 to 1.0 scale. The generated
standard grader rejects non-finite values and clamps numeric scores with
`max(0.0, min(1.0, score))`. A failed required judge produces no score instead
of a numeric zero. The fixed dimension bands are **pass** at 0.50 or above,
**neutral** from 0.40 to below 0.50, and **fail** below 0.40.

`run_config.json` records the configured judge provider, model, source, and
catalog-verification status separately from the agent model. `result.json`
embeds that run configuration alongside the scores. This provenance makes the
LLM-judged values traceable, but it is not evidence that the judges are
validated. The repository does not publish inter-rater agreement, calibration
results, or a comparison against human-labeled ground truth.

The **Quality Score** displayed out of 100 in the HTML report is the Tier 1
static-analysis score. It is not an input to the Tier 3 dimensions, overall
score, or Skill Lift.

For standard grading, each arm's Overall is the mean of the dimensions it
scored, and each dimension is the mean of its scored metrics. N/A dimensions
and metrics are excluded rather than counted as 0 or 1.

This per-arm rule is shared by the report, per-attempt pass@k thresholding, Skill
Lift, comparisons, and best-agent selection. Standard artifacts identify it as
`skill-evaluator-dimension-mean-v1` in `score_policy` and in the persisted
attempt policy. Older metric sets and partial historical artifacts keep their
legacy policy rather than being rescored as current data.

Overall comparisons require both the same scoring policy and the same metric
set. A shared policy name alone does not make five-metric and six-metric means
comparable. When two conditions use different metric sets under the same
policy, their individual scores and shared per-metric deltas remain available,
but overall Skill Lift and the pass@k comparison are omitted. Reports containing
agents scored on different standard metric sets retain their individual scores
and mark the overall comparison as incomplete, without selecting a best agent.
A custom-only agent, one whose conditions use only its grader's own overall
(`custom-overall-v1`), is still ranked against standard agents by that score,
and `tier3 compare` shows it beside them. `tier3 compare` refuses standard
agents scored under different policies or metric sets. When a metric is N/A in
only one arm, `tier3 compare` drops it from both arms for the Overall lift, as
`lift.json` does, so its lift can differ from the difference between the two
arm overall scores.

### Prior-version comparison

When requested, `result.json` contains `prior_version_comparison` with `status`
(`available` or `unavailable`), reasons, source run id and timestamp,
`source_skill_identity`, `current_skill_identity`, and an `identical_skill`
flag. Skill identities preserve evaluated-source revision and publication
target digest when recorded. `latest` also records bounded rejected candidates.
The report configuration retains this block for later rendering.

Available comparisons contain per-agent `current_overall`, `prior_overall`,
`delta`, `metrics` (current/prior/delta), optional custom metric deltas, and
`regression` when delta is negative. The top-level regression flag is true
when any available agent comparison regresses. Incompatible or missing scored
evidence for any selected agent makes the whole comparison unavailable, with
no regression flag. Reports show this
comparison next to, never folded into, no-skill Skill Lift. The read-only
`tier3 compare-versions` command recomputes the same comparison from retained
rows without changing either run.

### Lift history and retention

Use `skillevaluator tier3 history ./my-skill` to list retained runs oldest
first, with one row per run and agent. The reader uses the same results-root
precedence and legacy fallback as `compare`: explicit `--results-dir`, then
`SKILLEVALUATOR_RESULTS_DIR`, then `<skill>/evals/results/`. It reads the first
populated candidate root, not a merged history from different roots.

```bash
skillevaluator tier3 history ./my-skill --agent opencode
skillevaluator tier3 history ./my-skill --results-dir ./saved-runs --json
skillevaluator tier3 history ./my-skill --output ./lift-history.csv
skillevaluator tier3 history ./my-skill --output ./lift-history.md
```

Rows show run id and evaluation time, skill revision/content identity, model,
dataset digest and case count, with/without scores, paired Skill Lift, scored
and expected case coverage for both arms, baseline mode, and status. JSON
keeps the full identities; the table abbreviates long content and dataset
digests. Failed, incomplete, corrupt, skipped-baseline, baseline-only, or
unscored rows retain a null `lift_delta`, `lift_status="unavailable"`, and
reasons. A recorded lift of zero remains a real zero. Older summaries expose
whatever identity and score fields they recorded; missing facts stay unknown.

Each agent has its own chronological segments. The P0 baseline comparison
identity excludes skill revisions but includes model, dataset/cases,
environment, judge/scorer, attempts, score policy, and evaluator version.
Configuration changes start a numbered segment, set `config_changed`, and
list mismatch reasons; Markdown exports insert a break line. Known changes
in unavailable rows also break the segment. `lift_change` compares only
available rows with known compatible identities within the same segment;
unknown identities never establish a cross-run trend. Do not interpret
across-segment differences as skill improvement or regression.

History is a summary reader, not an evidence-eligibility check. Unlike the
completed-run views above, it also lists timestamped reservations with missing
or corrupt completion artifacts. This does not make them eligible baseline
sources. The command never launches evaluation or writes into the results
root. `--output` writes only the requested JSON, CSV, or Markdown file outside
the candidate results roots.

New evaluations reserve a unique run directory; they do not prune older run
directories. The default `<skill>/evals/results/` is inside the skill
repository, but a separate skill repository does not automatically ignore that
path. Add it to that repository's `.gitignore` before force-adding selected
summaries. Keep the full local tree if you need trial evidence or reusable
baselines. To retain a compact history with the repository, review
privacy-sensitive metadata and force-add **only** each chosen run's
`result.json` and `run_config.json` summaries (or copy them into a separately
tracked archive). Do not commit raw trajectories, logs, credentials, or the
entire generated results tree. Summary archives preserve history, not the
per-row evidence required for baseline reuse or publication validation.

### Skill Lift

Skill Lift compares Overall on the metrics and cases both arms scored, so it
can differ from subtracting the displayed Overall scores. It uses the same
score units as both arms and ranges from -1.0 to +1.0. Because live agent runs
are noisy, especially at low attempt
counts, small deltas are kept neutral. The verdict bands:

| Lift | Verdict | Read it as |
| --- | --- | --- |
| ≥ +0.05 | Pass | Positive observed lift; publication still requires the dimension and evidence gates |
| between −0.10 and +0.05 | Neutral | Within the neutral band; raise `--n-attempts` for more evidence |
| ≤ −0.10 | Fail | The skill makes the agent worse — investigate before shipping |

These bands color-code Skill Lift as diagnostic evidence. They do not determine
the combined `validate --tier3` or `BENCHMARK.md` verdict. The overall verdict
comes from the five dimensions: **PASS** when at least one successful agent has
all five dimensions at 0.50 or above; **NEUTRAL** when no agent passes but at
least one has no dimension below 0.40; and **FAIL** when every successful agent
has a dimension below 0.40.

A +0.03 lift therefore remains neutral under these bands. Lift requires both
arms; if you ran with `--skip-baseline`, there is no lift to report. In
`custom_only` grading, your own grader defines the score and lift — see
[Custom Graders & Tasks](/skills/skillevaluator/custom-graders).

The HTML **Dimension Scores and Skill Lift** table shows each dimension's
plain-language meaning and evidence source beside its score. For a successful
best agent, a standard dimension whose signals were active but produced no score
is listed as N/A with a **not scored with the skill** note and **No verdict**.
A dimension whose signals had nothing to judge reads **Not applicable**; it has
no score, so the agent cannot PASS. A
collapsed **What each number means** panel under the Tier 3 overview cards
separates the verdict, Overall Score, Skill Lift, pass@k, the Tier 1 Quality
Score, and publication eligibility.

When the HTML report has no lift for an agent or dimension, its label says
why. A recorded skipped baseline, for example after `--skip-baseline`, keeps
**no baseline** in Agent Details. **Overall baseline unavailable** means the
baseline arm ran or failed, or retained baseline evidence exists, but its
aggregate quality score is missing. Its score subtitle remains `baseline N/A`.
Dimension tables similarly distinguish **baseline quality unavailable** from a
recorded arm that was not run. **Agent coverage notes** retain the execution
details described below.
**comparison unavailable** means a baseline score was recorded but cannot be
compared with the with-skill score, for example because the arms used
different score policies or no case kept matching judge models.

For each LLM metric (`accuracy`, `goal_accuracy`, and `behavior_check`), cases
that used different judge models across the arms are excluded from that
metric's lift. This includes model changes across repeated attempts or the
scoring steps of a multi-step task. `lift.json` records the excluded case
counts by metric in `judge_mixed_cases`. Arm scores still describe all scored
cases, while evaluator, dimension, and overall lift use the eligible
comparisons. A reported lift can therefore differ from subtracting the two
displayed arm scores, even reversing their apparent direction. When no complete
comparison with matching judge models remains, the affected metric, dimension,
and overall lift are unavailable rather than zero. The HTML report's evaluator
card says how many cases its lift left out. When it has no lift, the card says
that no comparable result remains if its `comparison_basis` records
`contributing_attempts` as 0, and otherwise that the loaded results cannot
confirm the comparison. Its per-trial evidence labels those cases
**Left out of lift: different judge models**. In the canonical JSON the
evaluator records that count in `lift_excluded_cases`. Its card's
`comparison_basis` lists in `case_ids` the lift's keys for the cases the loaded
rewards identify: each case ID without an `-attemptN` or `_attemptN` suffix,
cut to 256 characters and listed once, at most 64 keys. It also gives the
with-skill `excluded_attempts` and `contributing_attempts`, which are both
`null` unless both arms' rewards loaded completely and identify every left-out
case. The matching evidence entries carry `comparison_excluded: true`. A
never-activated agent has no lift, so it gets none of these fields.

If every positive with-skill trial has confirmed zero activation, the agent's
result is **INCONCLUSIVE: skill never activated**. Reports keep its arm scores
and diagnostic trials but suppress its evaluator, dimension, overall, custom,
and pass@k lift comparisons and skill-quality advice. Negative controls do not
count as activation opportunities; missing or unsupported evidence remains
unknown. Eligibility is determined separately for each agent, so another
agent with observed activation can still contribute a comparison.

Budget-stopped runs are different from evaluated failures. If `budget.not_run`
contains attempts, the canonical run summary verdict is `incomplete`, with a
reason naming the budget and the planned attempts left not run. Terminal,
HTML, `tier3-benchmark.md`, and the full benchmark card show the budget summary.
Admitted attempts retain their scores, but any Skill Lift is labelled as
covering admitted attempts only, not a complete-run claim that the skill helps.
An overrun alone keeps the verdict and adds the budget warning. All reports
use final accounting; rebuilding older runs prefers `result.json.budget` over
a stale `run_config.json` snapshot. Execution status and exit codes are unchanged.

An arm that ran but failed, for example because one baseline attempt hit the
agent timeout, fails the agent and the run unless `--tolerate-baseline-timeouts`
keeps the agent scored (see [Baseline timeouts](/skills/skillevaluator/tier3-run-reference#baseline-timeouts)),
and the other arm keeps its scores. Reports describe such an arm by how many of its attempts scored,
never as not run. In `BENCHMARK.md`, the agent's Overall and dimension cells
read `90.0% — baseline incomplete (1 of 2 attempts scored); uplift unavailable`, or
`baseline unavailable` when no baseline attempt scored. They read
`baseline failed (2 of 2 attempts scored)` when the arm failed with no attempt
missing, and `baseline failed` with no counts when its recorded counts are
unusable. A baseline skipped with `--skip-baseline` still reads
`baseline not run`. An **Agent coverage** list under the table names each
failed arm, how many of its attempts scored, and the first Harbor exception
type it recorded, in words such as `agent timeout error`. When the agent is
marked failed, the entry also says that the agent and the Tier 3 run are
marked failed. The Tier 3 row of the tier status table names the agent and
arm.

The canonical JSON payload lists the same facts in `coverage_notices`, one
entry per failed arm:

- `code`: `baseline_failed` or `with_skill_failed`, the arm whose recorded
  status is `failed`.
- `agent` and `condition` (`without_skill` or `with_skill`).
- `state`: `incomplete` when some attempts scored, `unavailable` when none
  scored, or `failed` when no attempt is missing or the counts are unusable.
- `scored_attempts` and `expected_attempts`: `null`, with `state` `failed`,
  when the arm's counters are missing or inconsistent, for example more scored
  than expected attempts, or no expected attempts after a job failure.
- `failure`: the first Harbor exception type in words, or `null`.
- `message`: the sentence the reports show.

The HTML report keeps a short coverage cue near the Tier 3 result and retains
the complete per-agent/arm messages in Agent Details. An agent with baseline
execution evidence but no aggregate baseline score shows **Overall baseline
unavailable** with `baseline N/A`; a recorded skipped arm keeps **no baseline**.
The full terminal report, from `validate --verbose` or an explicit `-r cli`, prints the notes
under **Agent coverage notes** in the Tier 3 block, also when Tier 3 is
non-blocking (the default). The compact default view of `validate` does not
print them and leaves them to the HTML and JSON reports and `BENCHMARK.md`.
`tier3 evaluate` lists each agent's failed condition with its errors, as
before.

### Measured agent resources

Each agent's canonical JSON payload contains a `resources` object. The HTML
table shows total prompt plus completion tokens, agent execution seconds, LLM
seconds, tool seconds, and LLM milliseconds per token for each arm, telemetry
coverage, and the with-skill-minus-baseline difference. A negative difference
means less use with the skill; the HTML percentage divides it by the baseline.
Routing/tool-productivity efficiency remains a separate score. Missing counters
and incomplete arms produce null totals or deltas (`N/A` in HTML). Resource
deltas require matching case counts and complete measurements in both arms;
they do not measure evaluator tokens, setup time, price, or statistical savings.

`resources.scope` includes failed task outcomes when execution evidence is
complete. `resources.interpretation` explains the limits of these observed
consumption diagnostics: lower use alone does not establish improved task
quality or Efficiency. A task failure can consume fewer resources than a
successful outcome. Matching case counts does not certify equal concurrency
or budget, or actual judge-model agreement. Review task outcomes and run
conditions alongside the raw differences before making an improvement claim.

Baseline reuse checks configuration and evidence compatibility but deliberately
excludes concurrency and budget from its identity. Reused baseline measurements
remain historical observations from the source run, not fresh consumption or
independent evidence in the current run. Reports retain the source and timestamp;
see [Reuse a baseline](/skills/skillevaluator/tier3-run-reference#reuse-a-baseline).

| Measurement | Meaning |
| --- | --- |
| Agent execution seconds (`agent_seconds`) | Harbor's measured agent phase from the trial's `result.json`. Environment setup, agent installation, and verifier time are excluded. |
| LLM seconds (`llm_seconds`) | Model time from the trial's ATIF trajectory. With native ATIF timing (source `atif`), the sum of `llm_duration_ms` when every agent step records it. Otherwise it is inferred from step timestamps (source `step_timestamps`): each gap between steps counts once, and the part not spent in a tool is LLM time, which then also includes harness and queue time. |
| Tool seconds (`tool_seconds`) | Tool execution time. Native timing takes the union of tool-call intervals, so overlapping calls count once. Inferred timing reads the `Wall time` that Codex writes in the header of its shell and `apply_patch` tool output, never from the command output, or the `duration_seconds` Harbor keeps for those calls; parallel calls on one step count their longest wall time. A trajectory without tool calls measures 0 seconds. |
| LLM ms per token (`llm_ms_per_token`) | `1000 × LLM seconds / (prompt + completion tokens)` over the arm's totals. It needs complete LLM-time and token totals and a positive token count. |

Each trial's `timing` object records `llm_seconds`, `tool_seconds`, and
`source`, and `resources.timing_sources` lists the sources each arm used. If
the measurements mix native and inferred timing, `resources.caveats` contains
`mixed_timing_sources` and the HTML report says the LLM and tool seconds are
not like-for-like.

Inference covers trajectories whose tool calls are all Codex shell tools
(`exec_command`, `write_stdin`, `wait`, `wait_command`, `shell`, and
`shell_command`) or `apply_patch`, each with a recorded wall time. A trial that
also calls a tool with no recorded duration, such as web search,
`update_plan`, or an MCP tool, or whose step timestamps are missing, lack a UTC
offset, or run backwards, has no LLM or tool time. Inferred timing also needs
the agent's next step right after each tool step: a user or system step there
can follow idle or user time, so the trial has no timing. Agents without native
timing or Codex `Wall time` headers are timed only when their trajectory has no
tool calls.

Each step of a multi-step task is timed on its own, so the time between Harbor
steps is not counted. Steps marked `is_copied_context` repeat context from an
earlier trajectory and are not timed; a copied step after new steps leaves the
trial without timing. Copied timestamps do not show when the first new response
started, so inferred timing also leaves the trial without timing when the new
steps after copied context start with an agent step.

Trials whose `trajectory.json` was omitted (too large, or redaction or
validation failed) and multi-step trials scored per step have no timing.
Missing timing stays `N/A` (`null` in JSON), never zero, and follows the same
complete-arm and matching-case rules as tokens. Timing is diagnostic and does
not change scores, verdicts, or gates.

### Token usage

The canonical JSON payload also carries a `token_usage` object when at least
one attempt recorded token counts and the report limits leave room for it (see
below). It sums the agent's own tokens per case, per
agent arm, and over all agents, and says how many attempts each sum covers. It
is a measurement: it is not scored and changes no metric, dimension, verdict,
or gate.

- Tokens are prompt plus completion (`total_tokens`). Prompt counts include
  cached tokens, so cached tokens are never added again;
  `uncached_prompt_tokens` is prompt minus cached. A missing cache counter
  counts as 0.
- Attempts that failed before scoring, such as an attempt that hit the agent
  timeout, still used tokens and are counted. So are the scored attempts of an
  arm that failed. `coverage.unscored_attempts` counts the attempts without a
  score. Such an attempt belongs to the case that its `failure.json` names in
  `entry_id`, which the collector records from the task the attempt ran with
  either backend, so a native task whose `[metadata] entry_id` differs from
  its directory keeps its case. A run without that record gives the case whose
  task directory its Harbor `result.json` or `config.json` names, or else the
  case its trial name starts with. `resources` keeps its own totals, which
  need every attempt of an arm scored and measured.
- Nothing is estimated. An attempt with a missing or invalid counter adds no
  tokens. A report trial whose trajectory recorded an invalid counter names
  it, never its value, in `invalid_token_counters`, and its `tokens` leave
  that counter out. Its arm's `coverage.status` is then `partial` (other
  attempts were measured) or `unavailable` (none were), and the totals cover
  only the measured attempts. An arm that did not run is `not_run`, and its
  `totals` are `null`: a skipped arm, or an arm without attempts that the agent
  has no record of or that the payload `mode` leaves out (`with_skill_only` or
  `baseline_only`). The score tables show such a baseline as not run. A sum
  above 9,007,199,254,740,991 (`2^53 - 1`), the largest integer JSON keeps
  exact, is `unavailable` with `total_out_of_range` and `null` totals.
- `coverage` also gives `expected_attempts`, `observed_attempts`,
  `measured_attempts`, and `reason_codes` such as `missing_token_counters`,
  `invalid_token_counters`, and `missing_attempts`. A case expects the
  attempts its pass@k record planned: attempts skipped after a pass and
  attempts a run budget did not admit are not expected.
- `delta.total_tokens` is with skill minus baseline, and
  `delta.total_tokens_change_percent` divides it by the baseline. A change needs
  complete counts in both arms for the same cases and attempts, and both arms
  must have succeeded, because the report withholds the scores of any other arm.
  Otherwise `delta.status` is `unavailable` and `delta.reason_codes` and
  `delta.reason` say why, for example `baseline_not_run`, `baseline_failed`,
  `baseline_status_unknown` (the arm ran attempts without a recognized run
  status), `baseline_tokens_incomplete`, `baseline_total_out_of_range`,
  `case_scope_mismatch` (both arms ran attempts, for different cases or
  numbers of attempts per case), or `agent_scope_mismatch` for the sum over all
  agents.
- Case rows are validated against the evaluation dataset. Every case row uses
  one of the dataset's own case IDs, each at most once. An attempt whose only
  name is Harbor's 32-character cut of a case name is attributed to that case,
  so its missing telemetry stays with the canonical case instead of becoming a
  row of its own. `case_id_validation` records the check: `status` is
  `validated` or `unverified_case_ids`, `dataset_case_ids` counts the dataset's
  IDs, and `unmatched_case_ids` names the rows that could not be matched — for
  example a name that two dataset IDs cut to alike, which is never guessed.
  Such a row keeps its recorded name, is marked `case_id_in_dataset: false`,
  and every report shows it as an unverified case ID rather than as coverage
  of a dataset case. A table that renders no such row — the terminal table,
  which shows no case rows, or one whose row caps or payload pruning dropped
  them — names the unmatched case IDs instead of explaining a mark it does not
  carry. A run with no dataset to check against carries no
  `case_id_validation` and claims nothing about its case IDs.
- A run that reused an earlier run's baseline
  (`tier3 evaluate --reuse-baseline`) has `baseline_reused: true`. Its baseline
  counts come from the trajectories copied from that run: they are the tokens
  the baseline used there, not tokens used in this run, and every token usage
  table says so.

Token usage joins the payload after the report size budget, so it never
displaces trials or other score evidence. It also never pushes the payload past
the embedded size budget or the report limits that every report checks, such as
100,000 JSON values, including after the run's next steps, the case findings
that `tier3 evaluate` adds and `validate`'s applicability fields join it. When
it does not fit, its case rows go first: `token_usage.case_rows_omitted` and
`report_truncation.omitted.token_usage_case_rows` count them, and every report
says so. If the agent sums do not fit either, `token_usage` is left out and
`report_truncation.omitted.token_usage` is 1. If the payload cannot carry even
that record, for example because it exceeds the report limits without token
usage, `token_usage` is left out without a record, and every report marks the
payload as it would without token usage.

`BENCHMARK.md` has a **Token Usage** section after Results at a Glance: one row
per agent and one for all agents, with the case rows in a collapsed table that
shows the first 50 case rows. The Markdown report has the same table under its
Tier 3 section. The HTML report lists the case rows under **Measured Agent
Resources**, at most 64 per agent, read from the full Tier 3 data rather than
its bounded preview. Each table says how many more case rows are in the JSON
payload. The full terminal report (`validate --verbose` or `-r cli`) and
`tier3 evaluate --verbose` print the agent rows only and name the files that
hold the case rows: the HTML, Markdown and JSON reports for `validate`, and
for `tier3 evaluate` the reports the run wrote in its run directory:
`report.html` and `tier3-benchmark.md` for a Harbor run, `report.html` alone
for a NeMo Gym run, and none when the report step failed. When the payload
left the case rows out, they say instead that no report has them.
`tier3 evaluate` also prints the agent rows for a run whose arms all failed,
after the panel that says the evaluation did not complete.

### Failure reasons and per-trial rows

Each evaluator card lists its per-trial evidence by eval case with the
grader's reason, and a failing card (with-skill score below 0.60) opens the
list. A case that scored below 0.80 also quotes a short excerpt of the
offending command or artifact when the grader recorded one. When the attempts
of one case disagree, the lowest-scoring attempt's reason is shown. In the
**Per-Evaluator Drill-Down** table, hover a dotted-underlined score below 0.80
to read the reason for that trial. A metric the trial recorded as N/A reads
`N/A`, and hovering it says why; a score missing for any other reason reads
`—`.

Reasons are built from each trial's `reward.json` `details`, joining up to
three distinct parts from the grader's `reason` or `message`, failing findings
and behavior results, failed rubric criteria (except criteria the grader
excluded from the score), failed named checks, and the security attribution
note. The report redacts the values of configured credential environment
variables and recognizable credential patterns in reasons and excerpts, keeping
line breaks so a masked `key: value` pair ends at its line. It caps drill-down
reasons at 240 characters and excerpts at 300.
Secret detectors record their pattern rather than the matched text, so their
findings show a reason but no excerpt.

Reasons and excerpts are the first details dropped when a report is large. The
HTML page adds them only while its data stays within the page's preview limit
and says how many it left out, so they never displace the overview, dimension
scores or evaluator cards. The JSON download keeps them unless the embedded
payload reaches its 2 MiB limit; it then drops them before any other detail
and records the count as `failure_reasons` under `report_truncation.omitted`.

The drill-down has one row per trial for each agent and condition, with a
**Condition** column (`With skill` or `Without skill`); baseline rows are
listed even when they have no matching with-skill case, and a metric only
baseline rows scored still gets a column. When the HTML preview is bounded, it
keeps 64 with-skill rows across all agents and 64 baseline rows per agent, and a
note under the filter says how many rows it left out. The **Steps per Eval
Case by Agent and Condition** chart plots one series per agent and condition
from every trial, not only the rows the preview keeps, for up to 200 eval
cases; a note says how many more cases the JSON download holds. A case with
missing or partial step telemetry is a gap, not a zero.

### Qualitative assessment review contract

Use the existing **Insights**, dimension explanations, per-trial reasons and
recommendations first. They already connect outcomes to case evidence; an
additional narrative generator is not needed for the examples below. A review
worksheet adds a place to compare an author's expected behavior with the
observed outcome, record uncertainty and review the explanation's faithfulness.
It does not change grading.

The [example review worksheet](assets/qualitative-assessment-review.json) is a
**documentation prototype**, manually authored from three real retained runs.
It is not a CLI-emitted schema, supported import format, custom grader reward,
or new report artifact. Copy its structure into a review record beside your
retained evidence when assessing an explanation:

| Field | Review requirement |
| --- | --- |
| Identity and scope | Name the public skill/case, tier, agent, condition and environment; do not mix a Tier 1 rubric assessment with a Tier 3 execution result. |
| Expected behavior | Cite the dataset or rubric requirement. Keep a reviewer's desired behavior distinct from the recorded requirement. |
| Observations | Cite a source ID, its SHA-256 and a JSON pointer or trajectory step for every factual claim. Include only bounded, sanitized excerpts. |
| Interpretation | Label inferences separately; a judge's assertion is observed judge output, not independent proof of the assertion. |
| Numeric context | Copy the recorded execution status, scores and coverage with their original scale/policy. Preserve null and missing values; never derive a score from prose. |
| Uncertainty | Record missing, invalid, truncated or unavailable evidence and the limits of the sample. An inaccessible source is unavailable to the reviewer even when its hash is known. |
| Next action | State the action justified by the evidence, the evidence needed to verify it, and whether a skill change is warranted. |
| Provenance and review | Distinguish the execution agent, judge/model and assessment author/model. Record unknown provenance explicitly, review findings and outstanding signoff. |

Source IDs in the worksheet identify retained artifacts by hash. The public
calculator dataset includes an immutable repository commit, path and source URL
for the historical input used by the run; its original hash and JSON pointers
remain tied to that revision, even after the bundled dataset migrates. The
original run directories are not distributed here. The
bounded excerpts make this review inspectable, but do not allow readers to
independently reproduce every source claim. Keep originals under their existing
access policy; remove credentials, host paths and private task content before
sharing a review. A hash supports matching an authorized original, not verifying
its authenticity or granting access.

#### Reviewed real examples and findings

These are assistant-authored case reviews against the retained JSON and, for
the successful run, the retained trajectories. They are not representative-user
validation or human signoff. Independent assistant case review checked all three
assessments against their retained sources and found them faithful and actionable;
no claim is made that the worksheet improves user understanding.

| Example | Existing explanation and observed evidence | Review finding and next action |
| --- | --- | --- |
| Successful Tier 3 local OpenCode run, `calculator-001` | Both attempts scored. With skill, the trajectory invokes the calculator skill, runs `python scripts/calc.py "2 + 3 * 4"`, observes `Result: 14`, then answers `14`. The baseline's final message is `/help`. Existing Insights cites `calculator-001` and reports accuracy, goal accuracy and behavior scores of 1.0. | Faithful to this one case: overall is 1.0 with skill and 0.3333 baseline. This is not broad reliability or causal proof. Review more cases and attempts before generalizing. The worksheet adds the expected behavior and trajectory references beside that limit. |
| Failed Tier 3 Docker OpenCode setup, `calculator-001` | Both arms report `AgentSetupTimeoutError` after 360 seconds; zero of two expected attempts scored and both overall scores are null. Existing feedback calls the skill effect inconclusive and recommends a rerun followed by agent/gateway log inspection if repeated. | This establishes an execution blocker, not a measured skill failure. Check setup/install logs and rerun; do not rewrite the calculator from this evidence. The worksheet preserves the null scores and records the root cause as unresolved. |
| Ambiguous Tier 1 rubric judgment, bundled `text-analyzer` | The judge's positive summary and raw diagnostic `judge_score: 92` coexist with invalid criterion evidence: IDs `1`–`9` were returned instead of the required named IDs. Canonical `execution_status` is `failed`, `overall_score` is null and `overall_pass` is false. | The prose is not a valid quality result. Retry with valid named criterion evidence; do not map numeric IDs by position or promote 92 into the canonical score. The source does not record the judge model, so the worksheet leaves it unknown. |

These cases demonstrate a need for an explicit author/reviewer handoff and
evidence limits, rather than missing grading explanations. They do not establish
a need for runtime changes. The prototype retains the existing explanations,
records what the reviewer checked and keeps the proposed next action bounded by
the available evidence.

#### Qualitative-only decision

**Qualitative-only grading is not supported by this milestone.** The worksheet
is additive to current numeric contracts, including the numeric overall required
by [`custom_only`](/skills/skillevaluator/custom-graders#the-reward-contract). A null score means no
valid score was available; it is neither zero nor a pass. Execution failure,
missing evidence and measured task failure remain distinct.

A future scoreless extension requires a separately approved decision covering:

| Decision | Required definition before implementation |
| --- | --- |
| Schema and compatibility | A versioned result/reward contract with explicit scoreless status, evidence and provenance; migration and consumer behavior. |
| Missing score | Distinguish intentionally scoreless assessment from failed, invalid, skipped or unavailable scoring. |
| Aggregation | Define denominators and mixed scored/scoreless runs without coercing missing scores to zero or silently treating them as successful trials. |
| Comparisons and Skill Lift | Define which comparisons remain meaningful; prose cannot supply numeric lift or a paired score. |
| pass@k | Define eligibility and success semantics separately; this worksheet cannot satisfy a numeric pass threshold or create a passed case. |
| Publication | Define evidence completeness, review authority, eligibility and how scoreless results appear in benchmarks. A positive assessment alone does not grant publication eligibility. |

### Per-case expectation diagnostics

The HTML **Dataset** tab expands each case into its authored requirements and
recorded judgments, separately for each agent, attempt and condition. The JSON
payload's `case_diagnostics` carries the same inventory. Requirements retain
their original text (with credentials redacted); behavior IDs use the source
field and zero-based position, while judge `step` numbers remain one-based.
Routing and script checks use their named sub-checks, not the aggregate score.
Expected output is checked separately for goal achievement and accuracy; an
excluded accuracy criterion is listed as inapplicable, never as unmet.

Each judgment has one of five statuses:

| Status | Meaning | Next diagnostic step |
| --- | --- | --- |
| Met | An explicit verdict supports the requirement | No correction indicated by this check |
| Unmet | An explicit verdict reports a violation | Compare the requirement with the observed evidence; inspect execution if it is valid, or correct the test if it is inappropriate |
| Not checked | No unique usable verdict: missing/truncated output, judge or execution error, ambiguous identity, or explicit judge abstention | Inspect the retained execution/judge artifacts and rerun before diagnosing a skill defect |
| Unscored | The grader did not score this requirement's metric | Inspect grader configuration and score output |
| Not applicable | The grader explicitly excluded this check for this condition | Confirm that exclusion is intended; do not count it as a failure |

Reasons and short evidence excerpts accompany the verdicts when recorded.
Excerpts are metric-level context, not proof automatically attributed to every
requirement. A low score alone never assigns ownership to the skill or test.
Negative controls may correctly avoid the skill while completing an unrelated
task; being unrelated is not a reason to remove the case. The no-skill baseline
can mark skill checks inapplicable while still judging the task outcome.
Intentionally skipped conditions do not create diagnostic attempts. Failed or
unknown conditions remain visible even when no trial result was retained.

Behavior judges may return `passed: null` with an explanation when evidence
does not permit a check. The rich behavior result's `expectations` retains all
original behaviors, including those missing from a salvaged response. Existing
conservative scoring is preserved: only explicit passes earn credit, and the
denominator remains the full expected behavior count. Unchecked entries can
therefore lower the score without establishing a verified defect.

Inventories are bounded to 512 cases, 128 requirements per case, 512 attempts
across the report and 8,192 judgments. Any omissions are recorded under
`report_truncation.omitted`. Oversized HTML previews omit whole diagnostic
inventories rather than truncating identity or status fields; JSON Downloads
retain them unless the canonical 2 MiB limit also requires dropping them.
Dropping an oversized diagnostic inventory preserves built failure reasons
whenever those reasons still fit within the canonical limit.
Retained run artifacts remain the source for details omitted by either limit.

### pass@k

pass@k and its pass-rate difference are supporting reliability, not the headline
claim. The report labels the pass@k delta column **Pass-rate difference**.
In HTML, an absent, null or empty case-outcome collection leaves the pass rate
**N/A**, while recorded attempt counts remain available. A nonempty collection
that records no passed cases still shows the measured **0%**. This display rule
does not change pass@k eligibility, scoring or case denominators.
With `--n-attempts k`, each eval case runs k times per arm, and a case
counts as passed when at least one attempt clears the `--pass-threshold`
score using the same overall formula above. A scored `goal_accuracy` must also
meet its default 1.0 pass threshold; a below-threshold attempt fails with a
goal-not-met reason even when its Overall score passes. Scored security must
also be 1.0: a below-threshold result fails with a security-issue reason.
Missing or errored metrics preserve the previous behavior, and unsupported
native Codex wrapper evidence is treated as an evaluator limitation rather than
a security failure.
`behavior_check` is warning-only; a score below 1.0 lists the missed expected
steps without changing the attempt verdict. The existing Overall threshold is
applied after the required checks.

Each arm reports **Required checks passed**, **Goals met**, and
**Security clean** as counts over scored cases or attempts, with the
with-skill-minus-no-skill difference. Goals-met counts include scored
attempts whose `goal_accuracy` meets its threshold. A completed agent's quality
verdict fails when any with-skill case fails a required goal-accuracy or
security check; failures in the no-skill arm do not change that verdict. A
positive Skill Lift remains available for reference, but does not override
that quality failure.
Comparing pass@k across arms (in
`pass_at_k_lift.json`) tells you
whether the skill makes success more *repeatable*, not just whether the average
score moved. Each arm also records a case-level 95% Wilson score interval for
its pass rate; the paired value is the pass-rate difference. When both arms contain the same identified cases,
`pass_at_k_lift.json` records the paired outcome counts — skill-only passes,
baseline-only passes, both-pass, and neither-pass — plus a two-sided exact
McNemar diagnostic over the discordant pairs. Partial or unidentified pairing
is labeled and does not receive that exact-test result.

Each logical attempt counts once: a multi-step native task's step rewards
combine into one attempt score for averages and pass@k. When one condition
mixes standard-metric and custom-only rewards, its overall score and lift use
that logical-attempt overall rather than the mean of the standard dimensions,
which would cover only the standard rows.

`tier3 evaluate` also shows the evidence volume behind the headline: distinct
cases, planned and scored attempts, and execution or judge failures. When a
baseline ran, it includes the same counts for each arm. Its incomplete-coverage
form, such as “27 of 30 attempts scored,” is shown only by `tier3 evaluate`;
report surfaces show a fully scored best agent. A `Limited evidence:` warning
appears when there are fewer than 10 distinct cases or any non-skipped arm
scored fewer attempts than planned.

Case identifiers used as JSON keys must be printable, free of credential
material, and no larger than 512 UTF-8 bytes. A reward with an unsafe identity
is retained as a failed diagnostic but is not scored. For large evaluations,
`summary.json` keeps exact pass, attempt, failure, and pairing totals while
publishing bounded case, attempt, and failure-detail samples. The matching
`*_total`, `*_shown`, and `*_truncated` fields make every omission explicit;
sampling does not change the aggregate score or paired comparison.

The paired record also reports its pass-rate delta and the minimum p-value the
exact test could attain with the observed number of discordant pairs. When that
minimum is above 0.05, the report labels the test resolution-limited at that
threshold. This is a mathematical resolution limit, not evidence of no effect;
with five or fewer discordant pairs, a two-sided exact result cannot fall below
0.05 regardless of direction. The JSON always preserves compact decimal text
for both probabilities. It also records a reduced rational in each `*_exact`
field when the numerator and denominator fit both SkillEvaluator's 4,300-digit
safety cap and Python's active integer-string conversion limit. Otherwise that
field is `null`, `*_omitted` is `true`, and `*_omitted_reason` is
`"decimal_digit_limit"`. The numeric approximation is `null`, rather than a
false zero, if binary floating-point conversion underflows.

These uncertainty fields describe the cases in one configured run. They do not
make heterogeneous cases independent, correct dataset leakage, or promote a
small p-value into proof that a skill generalizes. Keep the dataset, attempt
policy, agent, model, and environment fixed when interpreting them.

### Completed agents in an incomplete run

When you evaluate several agents and one of them fails, the Tier 3 run fails.
The run's `execution_status` is `failed`, the payload's whole-run
`summary.overall_score` is `null`, and the publication assessment does not
change: one agent's result cannot stand for the whole run. The reports still
show the result of the run's best agent when its evidence is complete. The best
agent is the successful agent with the highest with-skill score; an agent that
never activated the skill is chosen only when no other successful agent has a
score.

- The HTML Overview combines that agent's name and score in **Completed Agent
  Score**, beside **Skill Lift** and **Coverage**. The warning remains
  **Completed-agent result; other execution failures need attention**.
- `tier3-benchmark.md` names the agent and gives its score and, when its
  baseline is comparable, its lift, followed by **Needs attention: other
  execution failures remain**. In a `validate` run, `BENCHMARK.md` shows the
  same lines under its unchanged overall verdict.
- When `--stop-on-pass` recovered a timeout of that agent with a later scored
  attempt, the HTML report (in the collapsed **Run details & provenance**
  panel), `tier3-benchmark.md` and, in a `validate` run, `BENCHMARK.md` give
  its number of recovered attempts. Timeouts that other agents recovered are
  not counted. The timed-out attempts stay in the failure history and are not
  scored. The HTML report and `tier3-benchmark.md` also give this count when
  every agent finished.

In a standalone report, the **Tier 3 quality** status follows the run's verdict.
HTML displays a failed run as **Changes required**, even when the completed
agent's own dimension verdict passes. This also applies to
`validate --block-on-agent-eval`. In a default `validate --tier3` report, Tier 3
is advisory and reads **Failed (advisory)**; the command status follows the
required tiers and reads **Passed required checks** when they pass. A successful
run with a `neutral` verdict reads **Neutral**.

The JSON report's `tier3` object and **Download Full Tier 3 Payload** preserve
the recorded verdict, which can be `neutral` when a failed agent's collected
results record the failure. An execution error reported after every agent
scored can instead leave the recorded score-based verdict as `pass` or `fail`.
HTML still displays the failed execution as a failure. Its stricter display
does not rewrite the retained payload; completed-agent details describe that
agent's evidence.

A failed run whose budget left planned attempts not run is the exception: the
JSON report's `tier3` object and **Download Full Tier 3 Payload** record
`incomplete`. The **Tier 3 quality** status of every HTML report and the
command status of the standalone Tier 3 `report.html` read **Incomplete**, as
does the Overview's **Live quality** line when the Overview shows it. The Tier 3
tile and the command status of a `validate` report read as for any other failed
run: the tile reads **Changes required**, or **Failed (advisory)** when Tier 3
is advisory. Both payloads leave `summary.overall_lift` null even when the best
agent has a lift, so the **Skill Lift** card reads N/A and **Run details &
provenance** has no **Best-agent lift** line.

The evidence is complete when the agent succeeded with no execution errors, has
a with-skill score from 0 to 1, and scored every expected attempt of every case
in each arm. `tier3-benchmark.md` shows lift only when the baseline arm also
succeeded and used the same metric set; after `--skip-baseline`, it shows the
score without a baseline or lift. If the best agent's evidence is incomplete,
the reports do not fall back to another agent. When no agent qualifies, for
example because every agent failed or the agents were scored on different
metric sets, no completed-agent score is shown. The benchmark's leading
callout reads **Tier 3: No complete agent result** when no agent qualifies.

When the run has a best agent, the HTML report names it with the score,
and its **Skill Lift** card and the **Best-agent lift** line under
**Run details & provenance** show that agent's lift when it has one, even when
its evidence is incomplete. The JSON report's `tier3` object and full Tier 3
download both keep this lift in `summary.overall_lift`. The exception is a
failed run whose budget left planned attempts not run, described above: there
`summary.overall_lift` is null, the **Skill Lift** card reads N/A and
**Run details & provenance** has no **Best-agent lift** line.
The lift is diagnostic evidence for one agent, not a whole-run lift.

To get a whole-run result, fix the failed agent and rerun the evaluation with
every agent.

## Browse results

### view

Opens the newest `report.html` for a skill in your browser, regenerating it
from the stored run data if the file is missing. The regenerated report reuses
the conclusions, recommendations, and suggestions the run saved in
`result.json`, so it shows the advice the run produced and makes no LLM call.
It also reuses the run's `what_to_fix.json`. When that file is missing or
unusable, `view` writes the rule-based fixes, which can differ from the
recommendations the run saved. The report also takes the run's duration and any
Harbor log links recorded in `result.json`. Feedback saved by 0.6.0 or earlier
has no per-case verdict, so the report adds the verdict and lists the case
fixes as its recommendations and suggestions, without the LLM, as a new report
does. A run that saved no feedback still has its feedback rebuilt, with the LLM
when a provider is configured, and its `what_to_fix.json` rewritten.

```bash title="Open the latest HTML report"
skillevaluator view ./my-skill
```

### compare

Prints a side-by-side terminal table of the most recent stored results per
agent — score and lift per evaluator — across everything under the results
root. Summaries whose execution status is anything other than `succeeded`
are ignored, so a failed run can't masquerade as a score.

```bash title="Compare stored results across agents"
skillevaluator compare ./my-skill
```

Both commands accept `--results-dir` to read from a non-default location.

### tier3 feedback

Opens a terminal chat about one completed run. Answers are grounded in that
run's scores, comparisons, per-case attempts, measured agent resources, and
saved feedback, without running the evaluation or regenerating its LLM feedback.
Pass the run directory itself; see
[tier3 feedback](/skills/skillevaluator/cli-reference#tier3-feedback) for the options and what the
chat sends to the LLM provider.

```bash title="Discuss one run"
skillevaluator tier3 feedback ./my-skill/evals/results/20260709_141530_12345_a1b2c3d4e5f6
```

### harbor-view

Opens retained Harbor job artifacts in Harbor's own trajectory browser — the
deepest debugging view, down to individual agent transcripts. Staged-task and
raw Harbor job directories are transient: they are deleted after collection by
default, on success and failure alike. Pass `--harbor-keep-jobs` to retain
them — the retention outcome is recorded in the results, and the Artifacts
panel prints this command only when a retained jobs directory actually exists.
Retained jobs live under `_harbor-jobs/` inside the run directory.

```bash title="Browse retained trajectories"
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker --harbor-keep-jobs
skillevaluator harbor-view ./my-skill/evals/results/20260709_141530_12345_a1b2c3d4e5f6/_harbor-jobs
```

(`skillevaluator tier3 harbor-view` is the same command under its expert
alias.)

## Performance dashboard

Use the optional [Performance Dashboard](/skills/skillevaluator/performance-dashboard) to filter
saved Tier 3 reports and retained runs, compare paired scores and resource use,
and download the visible tables as CSV. The guide covers installation,
comparison eligibility, measurement limits, and reloading saved inputs.

## Machine-readable contract

For CI and tooling, two JSON entry points matter:

- **Standalone `tier3 evaluate` runs** — parse `result.json` in the run
  directory (or follow `latest`). It carries per-agent scores, dimensions,
  lift, pass@k, trial counts, the attempt policy, and execution status.
  Unscored metrics stay unscored rather than defaulting to 0.0, so
  `overall_score` can be `null`.
- **`validate --tier3` runs** — the combined
  `skillevaluator-output-<timestamp>.json` report embeds the Tier 3 payload
  alongside the Tier 1 and Tier 2 results, so one file covers all three
  tiers. Each built-in Tier 1 or Tier 2 result carries its validated
  `publication_evidence` producer marker alongside finalized gating metadata,
  and the top-level
  `benchmark_policy` records whether Tier 2 and Tier 3 evidence is required for
  publication. `publication_status` is the compact machine-readable verdict;
  `publication` carries eligibility, reasons, and the Tier 3 evidence status.
  This is intentionally separate from `overall_status`, which continues to
  describe the command/process gate. Tier 3 is advisory for the CLI exit code by default and becomes
  blocking with `--block-on-agent-eval`; Tier 2 is blocking for the exit code
  by default and becomes advisory with `--no-block-on-dedup`. Those exit-code
  controls are separate from `benchmark_policy`. A validator's independent
  policy failures, such as a quality score below its required minimum, appear
  in `gate_failures` when present. Check those alongside `findings`; changing
  finding severity does not waive the minimum-score gate. The terminal, HTML
  report, and `BENCHMARK.md` also show these failures.
- **The evaluated source identity**: every JSON report always carries a
  top-level `evaluated_source` object, resolved from every carrier the run
  recorded, so the JSON and `BENCHMARK.md` name the same source tree. It is
  `null` when the orchestration input supplied none. Carriers that name
  different sources abort the run before any report file is written, so a
  contradictory identity is never published in one format and reported in
  another.
- **Tier 1 applicability**: when `validate --type auto` finds no skill,
  rules, workflows, or plugin in the target, the JSON report carries a
  top-level `tier1_applicability` object (`applicability: not_applicable`,
  `reason_code: unsupported_content_type`, `detected_content_type: unknown`,
  `supported_content_types`), whichever checks the run selected. Each of
  `schema`, `version`, and `quality` that the run selected and did not
  `--skip` is then `skipped`, with a `skip_reason` that starts with
  `not applicable`. The key is absent when Tier 1 ran those checks, as it
  does for a detected artifact, a forced `--type`, or a folder with a
  mis-cased or dangling manifest such as `SKILL.md`.

Both paths speak the same dialect: standalone `tier3 evaluate` and
`validate --tier3` share one HTML renderer for `report.html`, and both embed
the Tier 3 payload at schema version `2.0`. In that payload, conclusions and
recommendations written by the LLM carry `source: "llm"` and name the model
that wrote them as `model`. LLM recommendations that restate a deterministic
one, or that loosen an expectation, assertion, ground truth, or pass threshold,
are dropped, and LLM suggestions keep only evidence references the run
recorded. For a Claude 5.5 or later model on the native
Claude API this is the model the reply names, which can be the server-side
refusal fallback model rather than the configured model. Otherwise it is the
configured model, unless a server-side `fallback` block in the reply names
another model.

Exit codes and a ready-made JSON gating recipe live in
[Gate Your CI](/skills/skillevaluator/ci-integration).

## Next steps

<CardGroup cols={2}>
  <Card title="Gate Your CI" icon="shield" href="/skills/skillevaluator/ci-integration">
    Turn the JSON report and exit codes into a merge gate, with a copy-paste
    GitHub Actions workflow.
  </Card>
  <Card title="Tier 3: Live Agent Evaluation" icon="flask" href="/skills/skillevaluator/tier3-live-evaluation">
    Run the evaluation that produces the results tree on this page.
  </Card>
  <Card title="Custom Graders & Tasks" icon="book" href="/skills/skillevaluator/custom-graders">
    Define your own scoring when the default graders can't express success.
  </Card>
  <Card title="CLI Reference" icon="terminal" href="/skills/skillevaluator/cli-reference">
    Every flag for view, compare, and the report options, with exact defaults.
  </Card>
</CardGroup>

<div id="retained-three-tier-ci-evidence">

For the explicit all-tier opt-in and its prerequisites, see [Advanced CI](/skills/skillevaluator/advanced-ci).
The [retained CI evidence reference](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/all-tiers/README.md#retained-three-tier-ci-evidence) preserves the detailed report contract.

</div>