Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Add opt-in coverage-aware grader qualification policies (#497)
Narendran Raghavanf41c5c03b4d7
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9724e4ff..3a2dc84f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -16,6 +16,18 @@ All notable changes to SkillEvaluator are documented in this file.### Added+- Tier 3 `validate-grader --qualification-policy FILE` supports explicit,+ assessment-only metric/class/category coverage and risk limits, provider-free+ preflight checks, retained not-run receipts for insufficient labels, final+ policy decisions and tracked verifier-source Git provenance. Correct expected+ abstentions demonstrate category coverage without supplying binary scored+ cases; evaluated report scope must match its selected assessment cases. The+ experimental example is opt-in and fails bundled-suite class coverage; it establishes no+ live reliability or release approval. Resource comparison metadata and HTML+ also explain that observed consumption includes task failures, lower usage+ alone does not establish better Efficiency or quality, pairing does not certify+ equal concurrency/budget or judge-model agreement, and reused telemetry is+ historical. Scoring, raw deltas and reuse eligibility are unchanged.- Dataset fairness validation checks the evaluated skill's own names and nestedgraded text, distinguishes negated reads from required reads, and boundswarning excerpts and repeated-match scanning. Common Unicode, Markdown anddiff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdxindex f2c6ba5d..9e1774bd 100644--- a/docs/cli-reference.mdx+++ b/docs/cli-reference.mdx@@ -794,7 +794,8 @@ skillevaluator tier3 validate-grader ./suite --repeats 3 -o qualification| `--case ID` | all selected cases | Repeatable case filter || `-o, --output DIR` | grader-validation-output | Empty output directory for JSON and attempt artifacts || `--json` | off | Print full report as JSON |-| `--check-suite` | off | Validate suite and executable labels only, no provider needed |+| `--check-suite` | off | Validate suite and executable labels; with a policy also check expected coverage, no provider/output writes |+| `--qualification-policy FILE` | unset | Explicit assessment-only coverage and risk policy JSON; no default policy || `--concurrency N` | 1 | Maximum concurrent verifier subprocesses || `--timeout SECONDS` | 300 | Positive per-attempt timeout || `--max-false-pass-rate RATE` | unset | Gate any reported metric's false-pass rate |@@ -805,6 +806,18 @@ Rates must be between 0 and 1. Exit 0 reports results (including disagreement),exit 1 means an explicitly supplied gate was exceeded, and exit 2 means invalidusage, suite or provider configuration. Gates use assessment only with split all.+With `--qualification-policy FILE`, exit 1 also means failed or insufficient policy+evidence. Missing required metrics/rates and unmet distinct-case class/category+minima cannot pass; repeats and behavior steps do not add cases. The policy only+accepts `--split assessment` and cannot be combined with standalone `--max-*-rate`+gates; put its criteria in the policy file. These combinations are usage exit 2+before provider access. Offline `ready` means expected-label coverage only,+with scored counts/rates unobserved. Live insufficient preflight retains a+`not_run` receipt with no attempts before provider access. Covered live runs retain the+evaluated decision in JSON stdout and `grader-validation.json`. Policy identity,+canonical hash, effective definition, coverage checks and reasons are retained;+see [policy details](/skills/skillevaluator/grader-validation#explicit-qualification-policy).+## tier3 feedbackChat in the terminal about one completed live-evaluation run: which casesdiff --git a/docs/grader-validation.mdx b/docs/grader-validation.mdxindex 2e2248b9..6fcce51c 100644--- a/docs/grader-validation.mdx+++ b/docs/grader-validation.mdx@@ -16,7 +16,7 @@ Check the bundled reference suite without credentials or model calls:skillevaluator tier3 validate-grader --check-suite```-Configure a [judge provider](configuration.mdx), then grade held-out assessment+Configure a [judge provider](/skills/skillevaluator/configuration), then grade held-out assessmentcases three times each:```bash@@ -183,7 +183,8 @@ because the shipped verifier intentionally keeps Harbor's reward numeric-only.5. Read class counts and denominators before comparing rates. Inspect errors,abstentions, N/A, truncated evidence and repeated-grade variation separately.6. Save the report with its timestamp, selected case IDs, split, package version,- verifier SHA-256, suite content SHA-256, judge contracts, actual models used+ verifier SHA-256, tracked-source Git commit and verifier-file dirty state when+ available, suite content SHA-256, judge contracts, actual models used(including fallback), goal method, thresholds, repeats and concurrency.The suite digest frames each relative filename, byte length and contents,covering suite.json plus all referenced trajectories, even unselected ones.@@ -210,6 +211,117 @@ means invalid usage, suite, output path or provider configuration. Without gatesdisagreement and judge errors are reported rather than changing the exit code.`--check-suite` never requires a provider.+## Explicit qualification policy++`--qualification-policy FILE` adds operator-owned coverage requirements and risk+limits to an **assessment-only** run. It is off by default. `--split tuning` and+`--split all` with a policy are usage errors; development data cannot fill held-out+coverage. `--case` narrows the assessment scope, so the remaining cases must meet+the complete policy themselves. Existing standalone `--max-*-rate` gates remain+available and keep their existing behavior, including ignoring undefined rates.+A policy cannot be combined with any standalone `--max-*-rate` gate. Put all+qualification criteria in the retained policy file; combining them is usage exit 2+before provider access, including with `--check-suite`.++Review and freeze an immutable assessment suite and explicit policy before a run.+The repository's [experimental example](https://github.com/NVIDIA/SkillEvaluator/blob/main/examples/grader-qualification-policy.json)+is an example for review, not a default or a release policy:++```bash+# Offline: validate executable labels and check expected-class/category coverage.+skillevaluator tier3 validate-grader --check-suite --qualification-policy examples/grader-qualification-policy.json++# After suite, policy and cost approval: retain a manually triggered assessment.+skillevaluator tier3 validate-grader ./reviewed-suite --split assessment --repeats 3 --qualification-policy ./reviewed-policy.json -o qualification-receipt+```++The example requires all eight metrics, at least one distinct case in each expected+class per metric and at least one case per category. Its proposed limits are+false-pass 0.10, false-fail 0.10 and unjudgeable 0.05. These availability floors do+not establish statistical sufficiency. **The bundled suite fails the example's+offline coverage check:** security and skill_efficiency have no independently+labelled failing cases. Its bytes and labels are unchanged. Do not weaken labels+to obtain a pass; separately review and version representative evidence.++A policy is a regular UTF-8 JSON file no larger than 256 KiB with these fields:++| Field | Contract |+| --- | --- |+| `schema` | Exactly `skillevaluator.grader-qualification-policy/v1` |+| `policy_id` | 1–128 ASCII letters, digits, underscores, dots or hyphens; starts with a letter or digit |+| `description` | Optional nonempty string, at most 1024 characters |+| `required_metrics` | Nonempty unique list from `security`, `skill_execution`, `skill_efficiency`, `accuracy`, `goal_accuracy`, `behavior_check`, `behavior_steps`, `overall` |+| `min_class_cases` | Exactly `pass` and `fail`, each a positive integer; applies per required metric |+| `min_category_cases` | Known suite categories mapped to positive integers; an empty object is allowed |+| `max_rates` | Exactly `false_pass_rate`, `false_fail_rate`, `unjudgeable_rate`, each a finite number in [0,1] |++Unknown or duplicate object fields, duplicate metrics, booleans used as numbers,+nonfinite numbers, malformed JSON and excessive parser complexity are rejected.+The report's `qualification` stores the complete effective `policy`, `policy_id`+and `policy_sha256`. The SHA-256 covers UTF-8 canonical JSON with sorted keys,+compact separators, ASCII escapes and no nonfinite numbers. Whitespace and object+key ordering do not change the hash; policy content does.++`qualification.scope` names the selected assessment case IDs. Each required metric+has expected and scored distinct-case counts for each class, minimum checks, and+rate checks containing the existing measurement, limit and result. Category+checks separately report `expected_cases`, `observed_cases`, `expected_met` and+`observed_met`. A scored case has at least one observed pass/fail against that metric's expected pass/fail label. Behavior-step+coverage counts a case once per class, even if it has many steps in that class;+one case with both pass and fail steps can count once in each. Aggregate+behavior_check expected classes use the suite's metric threshold. Repeats never+create additional cases. A category's observed count includes cases with a required+known-class observation scored pass/fail, or a required explicitly expected N/A or+insufficient-evidence observation correctly matched. Correct abstentions demonstrate+representative category evidence without creating binary scored-class cases or+rate denominators. Errors, unexpected abstentions and confident verdicts against+expected-abstention labels cannot fill observed category coverage. Missing category+criteria impose no category minimum.++| Phase/status | Meaning |+| --- | --- |+| `preflight` / `ready` | Expected labels meet coverage minima only. Scored class counts, observed category counts and rate checks are null/unobserved; no grader reliability result exists. |+| `preflight` / `insufficient_evidence` | Expected labels cannot meet the policy. No verifier attempts or provider access occur. |+| `evaluated` / `passed` | Expected/scored class and expected/observed category coverage meet all minima, every required rate is available, and no rate exceeds its limit. |+| `evaluated` / `failed` | Coverage and rates are available, but at least one rate exceeds its limit; equality passes. |+| `evaluated` / `insufficient_evidence` | A required metric/rate, expected/scored class coverage or expected/observed category coverage is missing. This takes precedence when an available rate also exceeds a limit; every reason is retained. |++Rates and Wilson intervals retain the attempt-level denominators described above;+no new success or scoring formula is introduced. Evaluated report attempt IDs+and any declared provenance case IDs/split must match the selected assessment+scope; the evaluator rejects broader/subset scope mismatches instead of borrowing+unselected cases' rates. Observation case identities must match their attempt too.+In particular, zero denominators produce unavailable rates, never accepted zero-error results. Structured+`qualification.reasons` identify missing metrics/rates, class/category coverage+shortfalls and exceeded limits; human output names the same causes.++`--check-suite` with a policy prints suite validation and coverage readiness, exits+1 for insufficient expected labels, and writes no output. A live command with+insufficient expected labels first requires an empty output directory, then saves+`grader-validation.json` with `execution_status: not_run`, `attempts: []`, null+provider/model and the preflight decision. A covered live command runs the shipped+verifier and saves its final decision before writing the receipt. `--json` prints+the same complete report as the retained JSON. Policy failure or insufficient+evidence exits 1; malformed policy/options remain usage exit 2. Without a policy,+the command preserves its original gates and split behavior.++Provenance retains suite/verifier digests and contract IDs.+`verifier_git_commit` and `verifier_git_dirty` disclose the source checkout's HEAD+and whether any actual verifier file has staged or unstaged changes, only when+all actual verifier files are tracked in that repository. Installed or untracked+copies and unavailable Git metadata produce null for both fields, even inside an+unrelated checkout. Unrelated dirty files do not make the verifier dirty. Git+metadata is read-only and does not change baseline reuse eligibility.++Run offline schema/executable-label checks on relevant changes. Manually trigger+and retain live qualification when scorer prompts/code, judge/provider/model or+the reviewed assessment suite changes, and before accepting a release candidate.+Review the frozen suite, interpretation and approved cost bound before each paid+run. This feature introduces no paid schedule or release-blocking workflow. This+PR establishes neither live grader reliability nor release approval; those require+separately reviewed evidence and an explicit promotion decision. Raw resource+comparisons remain separate from scored Efficiency and Overall.+## Measured baselineOn 2026-11-07, SkillEvaluator commitdiff --git a/docs/reports.mdx b/docs/reports.mdxindex d0a0b6b9..3b95ce3c 100644--- a/docs/reports.mdx+++ b/docs/reports.mdx@@ -1027,6 +1027,20 @@ and incomplete arms produce null totals or deltas (`N/A` in HTML). Resourcedeltas require matching case counts and complete measurements in both arms;they do not measure evaluator tokens, setup time, price, or statistical savings.+`resources.scope` includes failed task outcomes when execution evidence is+complete. `resources.interpretation` explains the limits of these observed+consumption diagnostics: lower use alone does not establish improved task+quality or Efficiency. A task failure can consume fewer resources than a+successful outcome. Matching case counts does not certify equal concurrency+or budget, or actual judge-model agreement. Review task outcomes and run+conditions alongside the raw differences before making an improvement claim.++Baseline reuse checks configuration and evidence compatibility but deliberately+excludes concurrency and budget from its identity. Reused baseline measurements+remain historical observations from the source run, not fresh consumption or+independent evidence in the current run. Reports retain the source and timestamp;+see [Reuse a baseline](/skills/skillevaluator/tier3-live-evaluation#reuse-a-baseline).+| Measurement | Meaning || --- | --- || Agent execution seconds (`agent_seconds`) | Harbor's measured agent phase from the trial's `result.json`. Environment setup, agent installation, and verifier time are excluded. |diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex b2b70e12..fccd2fc3 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -56,7 +56,7 @@ human-readable dimensions:| Efficiency | Does it route correctly and use tools productively? |In `custom_only` mode, your own grader defines the score instead — see-[Custom Graders & Tasks](custom-graders.mdx).+[Custom Graders & Tasks](/skills/skillevaluator/custom-graders).If every positive with-skill trial has a confirmed activation score of zero,the result is **INCONCLUSIVE: skill never activated**. Tier 3 keeps the arm@@ -289,10 +289,10 @@ depth.| Requirement | How to check | Covered in || --- | --- | --- |-| The `tier3` extra installed (`[all]` includes it) | `skillevaluator health-check` | [Installation](installation.mdx) |-| An evaluator LLM provider key | `skillevaluator doctor --env-mode docker` names any missing key | [Providers & Credentials](configuration.mdx) |-| The agent and its credential | Same `doctor` run, with your agents; `tier3 evaluate` re-validates before any run | [Agents & Sandboxes](agents-and-sandboxes.mdx) |-| Docker running (the default environment) | `docker info` | [Agents & Sandboxes](agents-and-sandboxes.mdx) |+| The `tier3` extra installed (`[all]` includes it) | `skillevaluator health-check` | [Installation](/skills/skillevaluator/installation) |+| An evaluator LLM provider key | `skillevaluator doctor --env-mode docker` names any missing key | [Providers & Credentials](/skills/skillevaluator/configuration) |+| The agent and its credential | Same `doctor` run, with your agents; `tier3 evaluate` re-validates before any run | [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes) |+| Docker running (the default environment) | `docker info` | [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes) |## Harbor 0.25 compatibility@@ -376,7 +376,7 @@ using Harbor 0.25:uploaded tests. SkillEvaluator validates the verifier's Compose model againstits environment allowlist, and rejects `docker-compose.yml` verifierdefinitions, which Harbor ignores. Standard grading modes reject separate- verifiers entirely; see [Custom Graders & Tasks](custom-graders.mdx).+ verifiers entirely; see [Custom Graders & Tasks](/skills/skillevaluator/custom-graders).- Harbor 0.25 rejects a `reward.json` whose top-level values are not allfinite numbers. SkillEvaluator's verifier and wrapped custom graders alreadywrite numeric rewards; a native task's own test script must do the same. When@@ -386,7 +386,7 @@ using Harbor 0.25:## Start with one command-After the [provider setup](configuration.mdx) and runtime prerequisites above:+After the [provider setup](/skills/skillevaluator/configuration) and runtime prerequisites above:```bash title="Run Tier 3"skillevaluator tier3 ./my-skill@@ -439,7 +439,7 @@ before the measured matrix, `--agent-runtime-preflight` adds an agent-onlyexecution of the first staged task. It does not invoke a separatestandard-grader-only route before the evaluation reaches grading.`skillevaluator health-check` is the quicker-CLI-and-backend variant; [Agents & Sandboxes](agents-and-sandboxes.mdx) covers+CLI-and-backend variant; [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes) coversboth in depth.### Generate an eval dataset@@ -460,7 +460,7 @@ positive buckets and adds a negative case only when `evals/EVAL.md` supplies anoff-skill prompt under a supported negative heading. Generation uses yourconfigured evaluator provider by default; `--no-llm` falls back to keyless localtemplates. The dataset format, `EVAL.md` guidance, and refinement options live-in [Eval Datasets](eval-datasets.mdx).+in [Eval Datasets](/skills/skillevaluator/eval-datasets).### Review the cases@@ -595,10 +595,10 @@ skillevaluator compare ./my-skill````view` opens the latest HTML report; `compare` summarizes stored results across-agents and arms. [Reports & Results](reports.mdx) explains everything the run+agents and arms. [Reports & Results](/skills/skillevaluator/reports) explains everything the runwrote and how to interpret it. To ask follow-up questions about one run and getgrounded advice on skill changes, pass its run directory to-[`tier3 feedback`](cli-reference.mdx#tier3-feedback).+[`tier3 feedback`](/skills/skillevaluator/cli-reference#tier3-feedback).</Steps>@@ -640,7 +640,7 @@ can never supply, replace, or reroute them:| Credential | Used for | How to set || --- | --- | --- |-| Evaluator LLM provider | Dataset generation and standard grading | `SKILL_EVAL_LLM_PROVIDER` plus the provider key — see [Providers & Credentials](configuration.mdx) |+| Evaluator LLM provider | Dataset generation and standard grading | `SKILL_EVAL_LLM_PROVIDER` plus the provider key — see [Providers & Credentials](/skills/skillevaluator/configuration) || Live agent credential | The agent actually performing the task | Export it in the host environment where you run `skillevaluator` |```bash title="Agent credentials are host-environment exports, never config entries"@@ -655,7 +655,7 @@ operator-owned credential (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`,`NVIDIA_API_KEY`, the base-URL variables, `AWS_*`) fails the run with a harderror. This prevents a skill from swapping a credential or routing a trustedkey to an attacker-controlled endpoint. [Providers &-Credentials](configuration.mdx) lists every blocked name.+Credentials](/skills/skillevaluator/configuration) lists every blocked name.`runtime_env` also cannot override host launcher, language-runtime, Docker,Compose, proxy, or tracing controls. Configure the selected backend in the@@ -685,7 +685,7 @@ skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker \On cloud env-modes the bridges areunavailable: `opencode` still works with `NVIDIA_API_KEY` directly, but`codex` and `claude-code` fall back to their own native credentials and-explicit models — see [Agents & Sandboxes](agents-and-sandboxes.mdx).+explicit models — see [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes).## Where agents run@@ -726,7 +726,7 @@ through constructor kwargs. GKE requiresfor its default ephemeral launch mode or `instance_id` with`launch_mode=attach`. The complete environment/extra matrix, examples, Dockerresource tuning, local-mode safety controls, and workspace modes live in-[Agents & Sandboxes](agents-and-sandboxes.mdx).+[Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes).<Warning>Local mode executes the agent on your host rather than in a container. It is@@ -762,7 +762,7 @@ skillevaluator validate ./my-skill --agents codex \`--grading-mode`, and the rest) to the live-eval engine whenever Tier 3 isenabled, which is the default for skills. `--agent-eval` remains a supportedalias for `--tier3`; explicit tier selection and disable flags control the-scope. See the [CLI Reference](cli-reference.mdx) for the full group.+scope. See the [CLI Reference](/skills/skillevaluator/cli-reference) for the full group.</Note>## Plan for cost@@ -852,7 +852,7 @@ The high-signal flags, with defaults:| `--progress` | `auto` | Progress rendering: `auto`, `rich` (live TTY view with secret redaction), `plain`, or `off` || `--progress-interval` | mode default | Seconds between progress heartbeats: 1 for `rich`, 60 for `plain` lines in CI and redirected output || `--autopilot` | off | When no evaluation source exists, generate exactly one eval case — with the configured provider, falling back to a deterministic keyless template — never overwriting an existing source |-| `--grading-mode` | `default` | `default`, `default_plus_custom`, or `custom_only` — see [Custom Graders & Tasks](custom-graders.mdx) |+| `--grading-mode` | `default` | `default`, `default_plus_custom`, or `custom_only` — see [Custom Graders & Tasks](/skills/skillevaluator/custom-graders) || `--results-dir` | `evals/results`, or `SKILLEVALUATOR_RESULTS_DIR` when set | Write results under an external root instead of the skill directory; custom in-skill roots are rejected || `--harbor-keep-jobs` | off | Retain Harbor job directories for inspection || `--timeout-multiplier` | `1.0` | Scale Harbor's agent, verifier, and environment-build timeouts; must be a finite number greater than 0 whose scaled timeouts stay finite |@@ -885,7 +885,7 @@ Frames the terminal refused are dropped then, so they are never drawn overoutput printed since.Every `tier3 evaluate` flag, including resource overrides and Dockerfile-modes, is in the [CLI Reference](cli-reference.mdx).+modes, is in the [CLI Reference](/skills/skillevaluator/cli-reference).<Tip>While iterating on a dataset, `--skip-baseline` roughly halves the runtime.@@ -915,6 +915,11 @@ diagnostics. Reports identify the source and timestamp when reuse succeeds.The new run retains its own baseline rows. Fresh execution counts and reusedtrial counts are separate; reused evidence is not another independent attempt.Cost savings depend on the run, including agent, judge, and environment usage.+Baseline compatibility deliberately excludes concurrency and budget. It does+not establish actual judge-model agreement. Reused resource measurements are+historical observations from the source run, not fresh consumption or independent+evidence in this run; consider these limits alongside task outcomes when+interpreting resource differences.### Compare skill versions@@ -1177,7 +1182,7 @@ limit. Larger budgets send longer judge prompts and use more evaluatortokens. The behavior budget never falls below the final-response limit plus4,000 characters, so raising the limit leaves room for the tool history. Set these variables in the host environment;SkillEvaluator forwards them to the verifier. Defaults and accepted values are-in [Judge evidence budgets](environment-variables.mdx#judge-evidence-budgets).+in [Judge evidence budgets](/skills/skillevaluator/environment-variables#judge-evidence-budgets).### Judge requests and retries@@ -1268,11 +1273,11 @@ in [Judge evidence budgets](environment-variables.mdx#judge-evidence-budgets).Retry counts, delays, and the per-judge budget come from the host environmentand are forwarded to the verifier; see-[LLM retries and judge time budget](environment-variables.mdx#llm-retries-and-judge-time-budget).+[LLM retries and judge time budget](/skills/skillevaluator/environment-variables#llm-retries-and-judge-time-budget).Retries and downgrades do not change scoring formulas. A judge that still failsis an evaluation infrastructure failure, not a `0.0` score, and the affectedcondition publishes no scores; see-[Tier 3 judge failure semantics](environment-variables.mdx#tier-3-judge-failure-semantics).+[Tier 3 judge failure semantics](/skills/skillevaluator/environment-variables#tier-3-judge-failure-semantics).### Judge fallback and retry policies@@ -1481,7 +1486,7 @@ the configuration that produced it. An allowed provider fallback may use adifferent model for an individual call.This provenance does not establish that the LLM judgments are correct. The-[grader validation workflow](grader-validation.mdx) runs the shipped verifier+[grader validation workflow](/skills/skillevaluator/grader-validation) runs the shipped verifieragainst independently labelled cases and reports agreement, abstentions,errors and repeat variability. The bundled reference suite is not a publishedclaim of inter-rater agreement or universal judge reliability.@@ -1553,7 +1558,7 @@ resolved `should_trigger` label also leaves both skill metrics N/A.Skill Lift per dimension and overall uses only cases and metrics both armsscored; the `basis` block in `lift.json` records the shared comparison ([Skill-Lift reporting](reports.mdx#skill-lift)). A baseline is never credited or+Lift reporting](/skills/skillevaluator/reports#skill-lift)). A baseline is never credited orpenalized for a skill it was not given.The Security trace check excludes plain `rm -rf` commands whose operands are@@ -1786,14 +1791,19 @@ completion tokens and Harbor's measured agent execution seconds. Cache tokensare included in prompt totals, not subtracted or counted again. Time excludesenvironment setup, agent installation, and evaluator judging. Delta meanswith-skill total minus baseline total: negative values indicate less observed-use. A delta requires complete telemetry, complete successful arms, and matching+use. A delta requires complete telemetry, complete execution evidence, and matchingcase counts including repeated attempts. Missing measurements, skipped baselines,partial runs, and unsupported multi-step timing remain `N/A`; they are not zero.The same table splits each trial's trajectory time into LLM seconds and toolseconds, from native ATIF timing or inferred from step timestamps and Codextool wall times, and says which source each arm used. See-[Reports & Results](reports.mdx) for the timing rules.-These are observations for this run, not a graded efficiency or cost claim.+[Reports & Results](/skills/skillevaluator/reports) for the timing rules.+These observations include failed task outcomes. Lower use alone does not+establish improved task quality or Efficiency: a failed task can use fewer+resources. Pairing does not certify equal concurrency or budget, or actual+judge-model agreement. Reused baseline measurements remain historical rather+than fresh or independent evidence. These diagnostics do not change the graded+Efficiency score or establish a cost claim.Each dimension receives a score and a diagnostic band: **PASS** at 0.50 orabove, **NEUTRAL** from 0.40 to below 0.50, and **FAIL** below 0.40. A supported@@ -1846,10 +1856,10 @@ skipped payloads may leave live-evidence sections empty.Generated `BENCHMARK.md` cards put the verdict first, label each result columnas `Baseline → Skill Uplift`, show all three tier states, and keep methodology-collapsible. See the [rollout and backfill runbook](benchmark-rollout.mdx) before+collapsible. See the [rollout and backfill runbook](/skills/skillevaluator/benchmark-rollout) beforeregenerating already-published cards.-[Reports & Results](reports.mdx) covers the full on-disk layout, the+[Reports & Results](/skills/skillevaluator/reports) covers the full on-disk layout, themachine-readable JSON contract, and how to judge whether a given lift is good.## Troubleshooting@@ -1857,7 +1867,7 @@ machine-readable JSON contract, and how to judge whether a given lift is good.<AccordionGroup><Accordion title="A public LLM provider is required for live evaluation">Set `SKILL_EVAL_LLM_PROVIDER` and the corresponding evaluator-provider key.- See [Providers & Credentials](configuration.mdx).+ See [Providers & Credentials](/skills/skillevaluator/configuration).</Accordion><Accordion title="Agent authentication fails">Export the agent's credential in the host environment where you run@@ -1899,7 +1909,7 @@ machine-readable JSON contract, and how to judge whether a given lift is good.case rather than an infrastructure failure. Review the case prompt and theskill content that the agent read, then rerun, or run the arm with anothermodel through `--agent-model claude-code=MODEL`. See- [Agents & Sandboxes](agents-and-sandboxes.mdx#supported-agents).+ [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes#supported-agents).</Accordion><Accordion title="Results look wrong or a case is hard to diagnose">Re-run with `--harbor-keep-jobs` — Harbor artifacts are transient anddiff --git a/examples/grader-qualification-policy.json b/examples/grader-qualification-policy.jsonnew file mode 100644index 00000000..d1afbe02--- /dev/null+++ b/examples/grader-qualification-policy.json@@ -0,0 +1,30 @@+{+ "schema": "skillevaluator.grader-qualification-policy/v1",+ "policy_id": "experimental-all-metrics-v1",+ "description": "Experimental review example only. Availability floors and proposed risk limits are not statistical sufficiency or release approval. The bundled suite lacks required class coverage.",+ "required_metrics": [+ "security",+ "skill_execution",+ "skill_efficiency",+ "accuracy",+ "goal_accuracy",+ "behavior_check",+ "behavior_steps",+ "overall"+ ],+ "min_class_cases": {"pass": 1, "fail": 1},+ "min_category_cases": {+ "positive": 1,+ "negative": 1,+ "edge": 1,+ "failure_recovery": 1,+ "unavailable_evidence": 1,+ "non_use": 1,+ "baseline": 1+ },+ "max_rates": {+ "false_pass_rate": 0.10,+ "false_fail_rate": 0.10,+ "unjudgeable_rate": 0.05+ }+}diff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2index 11fa07b4..1321afcb 100644--- a/src/skillevaluator/reporting/templates/report.html.j2+++ b/src/skillevaluator/reporting/templates/report.html.j2@@ -3150,7 +3150,7 @@<details class="t3-details" open><summary>Measured Agent Resources</summary><div class="t3-details-body">- <p class="t3-section-hint">Efficiency scores measure routing and productive tool use. These separate measurements show observed agent resource use. Delta = with skill minus baseline; negative means less use. Judge and setup time are excluded. Deltas require complete telemetry and matching case counts in both arms. LLM and tool seconds split the agent's trajectory time; inferred LLM time also includes harness and queue time.</p>+ <p class="t3-section-hint">Efficiency scores measure routing and productive tool use. These separate measurements show observed agent resource use, including failed task outcomes. Lower use alone does not establish improved task quality or Efficiency. Delta = with skill minus baseline; negative means less use. Judge and setup time are excluded. Deltas require complete execution evidence, complete telemetry and matching case counts in both arms. Pairing does not certify matched concurrency or budget, or judge-model agreement. Reused baseline measurements are historical, not fresh or independent evidence. LLM and tool seconds split the agent's trajectory time; inferred LLM time also includes harness and queue time.</p>{% set t3_resource_rows = [('tokens', 'Prompt + completion tokens (cache included)', '{:,.0f}', 'total'),('agent_seconds', 'Agent execution seconds', '{:,.1f} s', 'total'),diff --git a/src/skillevaluator/tier3/grader_policy.py b/src/skillevaluator/tier3/grader_policy.pynew file mode 100644index 00000000..97349017--- /dev/null+++ b/src/skillevaluator/tier3/grader_policy.py@@ -0,0 +1,285 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Explicit operator-owned coverage and risk decisions for assessment receipts."""++from __future__ import annotations++import hashlib+import json+import math+import os+import re+import stat+from collections import Counter, defaultdict+from pathlib import Path++from skillevaluator.utils.structured_data import load_bounded_json++SCHEMA = "skillevaluator.grader-qualification-policy/v1"+MAX_POLICY_BYTES = 256 * 1024+RATES = ("false_pass_rate", "false_fail_rate", "unjudgeable_rate")+CLASSES = ("pass", "fail")+++def load_qualification_policy(path: Path) -> dict:+ """Read a bounded regular JSON file and reject incomplete/unknown policy criteria."""+ from skillevaluator.tier3.grader_validation import CATEGORIES, METRICS, _object++ # Nonblocking open protects against FIFOs where supported. Native Windows+ # has no O_NONBLOCK; check file type first there and still verify the fd.+ if not hasattr(os, "O_NONBLOCK") and not stat.S_ISREG(path.stat().st_mode):+ raise ValueError("qualification policy must be a regular file")+ flags = os.O_RDONLY | getattr(os, "O_NONBLOCK", 0) | getattr(os, "O_BINARY", 0)+ descriptor = os.open(path, flags)+ try:+ info = os.fstat(descriptor)+ if not stat.S_ISREG(info.st_mode) or info.st_size > MAX_POLICY_BYTES:+ raise ValueError("qualification policy must be a regular file of at most 256 KiB")+ data = os.read(descriptor, MAX_POLICY_BYTES + 1)+ finally:+ os.close(descriptor)+ if len(data) > MAX_POLICY_BYTES:+ raise ValueError("qualification policy exceeds 256 KiB")+ try:+ text = data.decode("utf-8")+ policy = load_bounded_json(text)+ except UnicodeError as exc:+ raise ValueError("qualification policy must be UTF-8 JSON") from exc++ def unique_fields(pairs):+ result = {}+ for key, value in pairs:+ if key in result:+ raise ValueError(f"qualification policy has duplicate field: {key}")+ result[key] = value+ return result++ # The existing parser establishes strict syntax/complexity first. Check all+ # object keys too: a last-wins limit would make a reviewed policy ambiguous.+ policy = json.loads(text, object_pairs_hook=unique_fields)+ required = {"schema", "policy_id", "required_metrics", "min_class_cases", "min_category_cases", "max_rates"}+ _object(policy, required | {"description"}, required, "qualification policy")+ if policy["schema"] != SCHEMA:+ raise ValueError(f"qualification policy schema must be {SCHEMA}")+ if not isinstance(policy["policy_id"], str) or not re.fullmatch(+ r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}", policy["policy_id"]+ ):+ raise ValueError("policy_id must be a safe ASCII identifier of at most 128 characters")+ if "description" in policy and (+ not isinstance(policy["description"], str)+ or not policy["description"].strip()+ or len(policy["description"]) > 1024+ ):+ raise ValueError("description must be a nonempty string of at most 1024 characters")+ metrics = policy["required_metrics"]+ allowed_metrics = {*METRICS, "behavior_steps", "overall"}+ if (+ not isinstance(metrics, list)+ or not metrics+ or any(not isinstance(m, str) or m not in allowed_metrics for m in metrics)+ or len(set(metrics)) != len(metrics)+ ):+ raise ValueError("required_metrics must be a nonempty unique list of known metrics")+ _object(policy["min_class_cases"], set(CLASSES), set(CLASSES), "min_class_cases")+ _object(policy["min_category_cases"], CATEGORIES, set(), "min_category_cases")+ for field in ("min_class_cases", "min_category_cases"):+ for key, value in policy[field].items():+ if type(value) is not int or value < 1:+ raise ValueError(f"{field}.{key}: expected positive integer")+ _object(policy["max_rates"], set(RATES), set(RATES), "max_rates")+ for key, value in policy["max_rates"].items():+ if (+ isinstance(value, bool)+ or not isinstance(value, (int, float))+ or not 0 <= value <= 1+ or not math.isfinite(value)+ ):+ raise ValueError(f"max_rates.{key}: expected finite number in [0,1]")+ return policy+++def _expected(cases, thresholds):+ from skillevaluator.tier3.grader_validation import observations++ if any(case["split"] != "assessment" for case in cases):+ raise ValueError("qualification policies require assessment cases only")+ return {case["id"]: observations(case, {}, thresholds) for case in cases}+++def _decision(cases, thresholds, policy, report=None):+ expected = _expected(cases, thresholds)+ evaluated = report is not None+ reasons = []+ metrics = {}+ observed_categories = set()+ observed = defaultdict(set)+ for attempt in (report or {}).get("attempts", []):+ for row in attempt["observations"]:+ observed[attempt["case"], row["metric"], row["judgment"]].add(row["observed"])+ for metric in policy["required_metrics"]:+ classes = {label: set() for label in CLASSES}+ scored = {label: set() for label in CLASSES}+ labelled = False+ for case_id, rows in expected.items():+ for row in rows:+ if row["metric"] != metric:+ continue+ labelled = True+ verdicts = observed[case_id, metric, row["judgment"]]+ binary_scored = bool(verdicts.intersection(CLASSES))+ if row["expected"] in CLASSES:+ classes[row["expected"]].add(case_id)+ if binary_scored:+ scored[row["expected"]].add(case_id)+ observed_categories.add(case_id)+ elif row["expected"] in {"not_applicable", "insufficient_evidence"} and row["expected"] in verdicts:+ observed_categories.add(case_id)+ if not labelled:+ reasons.append(+ {"code": "missing_metric", "metric": metric, "message": f"{metric}: required metric has no labels"}+ )+ checks = {}+ for label, minimum in policy["min_class_cases"].items():+ count, scored_count = len(classes[label]), len(scored[label])+ checks[label] = {+ "minimum": minimum,+ "expected_met": count >= minimum,+ "scored_met": scored_count >= minimum if evaluated else None,+ }+ for phase, actual in (("expected", count), ("scored", scored_count)):+ if (phase == "expected" or evaluated) and actual < minimum:+ reasons.append(+ {+ "code": f"{phase}_class_coverage",+ "metric": metric,+ "class": label,+ "actual": actual,+ "minimum": minimum,+ "message": f"{metric}: {phase} {label} case coverage {actual} below {minimum}",+ }+ )+ rate_checks = None+ if evaluated:+ rate_checks = {}+ summary = report.get("metrics", {}).get(metric, {})+ if not summary:+ reasons.append(+ {+ "code": "missing_metric",+ "metric": metric,+ "message": f"{metric}: required metric has no observed summary",+ }+ )+ for name, limit in policy["max_rates"].items():+ measurement = summary.get(name)+ value = measurement.get("rate") if isinstance(measurement, dict) else None+ available = (+ isinstance(value, (int, float))+ and not isinstance(value, bool)+ and math.isfinite(value)+ and 0 <= value <= 1+ and measurement.get("denominator", 0) > 0+ )+ rate_checks[name] = {+ "limit": limit,+ "measurement": measurement,+ "met": value <= limit if available else None,+ }+ if not available:+ reasons.append(+ {+ "code": "missing_rate",+ "metric": metric,+ "rate": name,+ "message": f"{metric}: {name} unavailable (no scored denominator or measurement)",+ }+ )+ elif value > limit:+ reasons.append(+ {+ "code": "rate_exceeded",+ "metric": metric,+ "rate": name,+ "actual": value,+ "limit": limit,+ "message": f"{metric}: {name} {value:g} exceeds {limit:g}",+ }+ )+ metrics[metric] = {+ "expected_class_counts": {label: len(values) for label, values in classes.items()},+ "scored_class_counts": {label: len(values) for label, values in scored.items()} if evaluated else None,+ "class_checks": checks,+ "rate_checks": rate_checks,+ }+ categories = {}+ counts = Counter(case["category"] for case in cases)+ observed_counts = Counter(case["category"] for case in cases if case["id"] in observed_categories)+ for category, minimum in policy["min_category_cases"].items():+ categories[category] = {+ "minimum": minimum,+ "expected_cases": counts[category],+ "observed_cases": observed_counts[category] if evaluated else None,+ "expected_met": counts[category] >= minimum,+ "observed_met": observed_counts[category] >= minimum if evaluated else None,+ }+ for phase, actual in (("expected", counts[category]), ("observed", observed_counts[category])):+ if (phase == "expected" or evaluated) and actual < minimum:+ reasons.append(+ {+ "code": f"{phase}_category_coverage",+ "category": category,+ "actual": actual,+ "minimum": minimum,+ "message": f"{category}: {phase} category case coverage {actual} below {minimum}",+ }+ )+ insufficient = any(reason["code"] != "rate_exceeded" for reason in reasons)+ canonical = json.dumps(policy, sort_keys=True, separators=(",", ":"), ensure_ascii=True, allow_nan=False).encode()+ return {+ "policy_id": policy["policy_id"],+ "policy_sha256": hashlib.sha256(canonical).hexdigest(),+ "policy": policy,+ "scope": {"split": "assessment", "case_ids": [case["id"] for case in cases]},+ "phase": "evaluated" if evaluated else "preflight",+ "status": "insufficient_evidence"+ if insufficient+ else "failed"+ if reasons+ else "passed"+ if evaluated+ else "ready",+ "metrics": metrics,+ "categories": categories,+ "reasons": reasons,+ }+++def preflight_qualification(cases, thresholds, policy) -> dict:+ """Check label coverage only; scored counts and rate checks remain unobserved."""+ return _decision(cases, thresholds, policy)+++def evaluate_qualification(report, cases, policy) -> dict:+ """Evaluate distinct-case coverage and existing attempt-level rates together."""+ from skillevaluator.tier3.grader_validation import DEFAULT_THRESHOLDS++ # The existing summaries describe all report attempts. Never attach them to+ # a subset's scope, where unselected cases could dilute error rates.+ selected = {case["id"] for case in cases}+ provenance = report.get("provenance", {})+ declared = provenance.get("case_ids", list(selected))+ if (+ not isinstance(declared, list)+ or any(not isinstance(case_id, str) for case_id in declared)+ or len(declared) != len(selected)+ or set(declared) != selected+ or {attempt["case"] for attempt in report.get("attempts", [])} != selected+ or provenance.get("split", "assessment") != "assessment"+ ):+ raise ValueError("qualification report scope must match the selected assessment cases")+ for attempt in report.get("attempts", []):+ if any(row["judgment"].split(":")[0] != attempt["case"] for row in attempt["observations"]):+ raise ValueError("qualification observation scope must match its attempt case")+ thresholds = DEFAULT_THRESHOLDS | provenance.get("thresholds", {})+ return _decision(cases, thresholds, policy, report)diff --git a/src/skillevaluator/tier3/grader_validation.py b/src/skillevaluator/tier3/grader_validation.pyindex d031817e..091bf477 100644--- a/src/skillevaluator/tier3/grader_validation.py+++ b/src/skillevaluator/tier3/grader_validation.py@@ -426,9 +426,65 @@ def _attempt(root, case, repeat, output, thresholds, timeout, provider_env=None)return {"case": case["id"], "repeat": repeat, "returncode": code, "reward": reward, "observations": rows}-def qualify(root, suite, cases, output, repeats, concurrency, timeout, split, provider, provider_env, judge_config):+def _verifier_git_provenance(paths):+ """Disclose source identity only when every actual verifier file is tracked."""+ unknown = {"verifier_git_commit": None, "verifier_git_dirty": None}+ paths = [path.resolve() for path in paths]+ if not paths:+ return unknown++ def git(directory, *args):+ # Fixed commands and explicit paths only; never read remotes or credentials.+ result = subprocess.run(+ ["git", "--no-optional-locks", "-c", "core.fsmonitor=false", "-C", str(directory), *args],+ env={key: value for key, value in os.environ.items() if not key.startswith("GIT_")},+ capture_output=True,+ timeout=2,+ check=False,+ )+ if result.returncode or len(result.stdout) > 256 * 1024:+ raise ValueError("Git source identity unavailable")+ return result.stdout.decode("utf-8").strip()++ try:+ repository = Path(git(paths[0].parent, "rev-parse", "--show-toplevel")).resolve()+ names = [path.relative_to(repository).as_posix() for path in paths]+ tracked = [name for name in git(repository, "ls-files", "-z", "--", *names).split("\0") if name]+ if set(tracked) != set(names):+ return unknown+ commit = git(repository, "rev-parse", "--verify", "HEAD")+ if not re.fullmatch(r"[0-9a-f]{40}|[0-9a-f]{64}", commit):+ return unknown+ dirty = bool(git(repository, "status", "--porcelain", "--untracked-files=no", "--", *names))+ return {"verifier_git_commit": commit, "verifier_git_dirty": dirty}+ except (OSError, ValueError, subprocess.TimeoutExpired):+ return unknown+++def qualify(+ root,+ suite,+ cases,+ output,+ repeats,+ concurrency,+ timeout,+ split,+ provider,+ provider_env,+ judge_config,+ qualification_policy=None,+):+ from skillevaluator.tier3.grader_policy import evaluate_qualification, preflight_qualification+thresholds = DEFAULT_THRESHOLDS | suite.get("thresholds", {})- tasks = [(case, repeat) for case in cases for repeat in range(1, repeats + 1)]+ if qualification_policy is not None and split != "assessment":+ raise ValueError("qualification policies require --split assessment")+ preflight = (+ preflight_qualification(cases, thresholds, qualification_policy) if qualification_policy is not None else None+ )+ not_run = preflight is not None and preflight["status"] != "ready"+ tasks = [] if not_run else [(case, repeat) for case in cases for repeat in range(1, repeats + 1)]with ThreadPoolExecutor(max_workers=concurrency) as executor:attempts = list(executor.map(lambda task: _attempt(root, *task, output, thresholds, timeout, provider_env), tasks)@@ -469,7 +525,7 @@ def qualify(root, suite, cases, output, repeats, concurrency, timeout, split, prsource_root = Path(__file__).resolve().parents[2]verifier_file_sha256 = {}verifier_digest = hashlib.sha256()- verifier_paths = (path.resolve() for path in _VERIFIER_FILES)+ verifier_paths = [path.resolve() for path in _VERIFIER_FILES]for path in sorted(verifier_paths, key=lambda item: item.relative_to(source_root).as_posix()):name = path.relative_to(source_root).as_posix()content = path.read_bytes()@@ -497,6 +553,7 @@ def qualify(root, suite, cases, output, repeats, concurrency, timeout, split, pr"skillevaluator_version": __version__,"verifier_sha256": verifier_digest.hexdigest(),"verifier_file_sha256": verifier_file_sha256,+ **_verifier_git_provenance(verifier_paths),"contracts": {key: sorted({r[key] for r in rewards if key in r})for key in ("judge_contract", "metric_set", "score_policy")@@ -505,8 +562,8 @@ def qualify(root, suite, cases, output, repeats, concurrency, timeout, split, pr"goal_methods": sorted({r.get("details", {}).get("goal_accuracy", {}).get("method") for r in rewards} - {None}),- "provider": provider.provider,- "model": provider.model,+ "provider": provider.provider if provider is not None else None,+ "model": provider.model if provider is not None else None,"judge_config": judge_config,"judge_environment": safe_env,"thresholds": thresholds,@@ -537,6 +594,9 @@ def qualify(root, suite, cases, output, repeats, concurrency, timeout, split, pr}# Mixed splits have no aggregate headline or gates.result["metrics"] = {}+ if qualification_policy is not None:+ result["execution_status"] = "not_run" if not_run else "completed"+ result["qualification"] = preflight if not_run else evaluate_qualification(result, cases, qualification_policy)_write(output / "grader-validation.json", result)return result@@ -566,6 +626,11 @@ def gate_failures(result, gates):@click.option("--max-false-pass-rate", type=click.FloatRange(0, 1))@click.option("--max-false-fail-rate", type=click.FloatRange(0, 1))@click.option("--max-unjudgeable-rate", type=click.FloatRange(0, 1))+ "--qualification-policy",+ type=click.Path(path_type=Path),+ help="Explicit assessment coverage and risk policy JSON file.",+)def validate_grader(suite_dir,repeats,@@ -579,6 +644,7 @@ def validate_grader(max_false_pass_rate,max_false_fail_rate,max_unjudgeable_rate,+ qualification_policy,):"""Qualify the shipped Tier 3 verifier against independently labelled cases.@@ -586,6 +652,23 @@ def validate_grader("""suite_dir = REFERENCE_SUITE if suite_dir is None else suite_dirtry:+ from skillevaluator.tier3.grader_policy import load_qualification_policy, preflight_qualification++ for name, value in {+ "timeout": timeout,+ "max-false-pass-rate": max_false_pass_rate,+ "max-false-fail-rate": max_false_fail_rate,+ "max-unjudgeable-rate": max_unjudgeable_rate,+ }.items():+ if value is not None and not math.isfinite(value):+ raise ValueError(f"--{name} must be finite")+ if qualification_policy is not None and any(+ value is not None for value in (max_false_pass_rate, max_false_fail_rate, max_unjudgeable_rate)+ ):+ raise ValueError("cannot combine --qualification-policy with --max-*-rate gates; put limits in the policy")+ policy = load_qualification_policy(qualification_policy) if qualification_policy is not None else None+ if policy is not None and split != "assessment":+ raise ValueError("qualification policies require --split assessment")suite = load_suite(suite_dir)unknown = set(case_ids) - {case["id"] for case in suite["cases"]}if unknown:@@ -597,22 +680,36 @@ def validate_grader(]if not cases:raise ValueError("no cases selected")+ preflight = (+ preflight_qualification(cases, DEFAULT_THRESHOLDS | suite.get("thresholds", {}), policy)+ if policy is not None+ else None+ )if check_suite:payload = {"status": "valid", "cases": len(cases), "split": split}+ if preflight is not None:+ payload["qualification"] = preflightclick.echo(json.dumps(payload)if as_jsonelse f"Suite valid: {len(cases)} {split} cases; executable labels verified (no LLM calls).")+ if preflight is not None:+ if not as_json:+ _print_qualification(preflight)+ if preflight["status"] != "ready":+ raise click.exceptions.Exit(1)return- provider = resolve_llm_provider()- from skillevaluator.tier3.harbor.runner import _judge_model_config, _provider_environment-- provider_env = _provider_environment(provider)- judge_config = _judge_model_config(provider, provider_env, "default")output = output.resolve()if output.exists() and any(output.iterdir()):raise ValueError("output directory must be empty (prevents stale attempt evidence)")+ provider, provider_env, judge_config = None, {}, {}+ if preflight is None or preflight["status"] == "ready":+ provider = resolve_llm_provider()+ from skillevaluator.tier3.harbor.runner import _judge_model_config, _provider_environment++ provider_env = _provider_environment(provider)+ judge_config = _judge_model_config(provider, provider_env, "default")output.mkdir(parents=True, exist_ok=True)result = qualify(suite_dir.resolve(),@@ -626,6 +723,7 @@ def validate_grader(provider,provider_env,judge_config,+ qualification_policy=policy,)except (ValueError, OSError) as exc:raise click.UsageError(str(exc)) from exc@@ -653,7 +751,16 @@ def validate_grader(Console().print(table)click.echo(f"Split: {split}; report: {output / 'grader-validation.json'}")click.echo(LIMITATIONS)- if failures:- if not as_json:+ if "qualification" in result:+ _print_qualification(result["qualification"])+ policy_failed = "qualification" in result and result["qualification"]["status"] != "passed"+ if failures or policy_failed:+ if failures and not as_json:click.echo(f"Gates exceeded: {', '.join(failures)}", err=True)raise click.exceptions.Exit(1)+++def _print_qualification(decision):+ click.echo(f"Policy {decision['policy_id']}: {decision['status']} ({decision['phase']})")+ for reason in decision["reasons"]:+ click.echo(f" {reason['message']}")diff --git a/src/skillevaluator/tier3/harbor/resource_usage.py b/src/skillevaluator/tier3/harbor/resource_usage.pyindex be0b8e47..8442152f 100644--- a/src/skillevaluator/tier3/harbor/resource_usage.py+++ b/src/skillevaluator/tier3/harbor/resource_usage.py@@ -152,7 +152,15 @@ def resource_comparison(timing_sources = {"with_skill": sources(with_trials), "baseline": sources(baseline_trials)}mixed = len({*timing_sources["with_skill"], *timing_sources["baseline"]}) > 1return {- "scope": "observed agent-only trial totals; prompt + completion tokens (cache included), excluding judge/setup time",+ "scope": (+ "observed agent-only trial totals, including failed task outcomes; "+ "prompt + completion tokens (cache included), excluding judge/setup time"+ ),+ "interpretation": (+ "Observed consumption only. Lower use alone does not establish improved task quality or Efficiency. "+ "Pairing does not certify matched concurrency or budget, or judge-model agreement. "+ "Reused baseline measurements are historical, not fresh or independent evidence."+ ),"pairing": "matching case counts" if paired else "unavailable","delta_definition": "with skill minus baseline; negative means less observed resource use","timing_definition": (diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.jsonindex 2db448e3..548e2c2a 100644--- a/tests/golden/cli_surface.json+++ b/tests/golden/cli_surface.json@@ -4148,6 +4148,14 @@],"param_type": "option","type": "float range"+ },+ {+ "name": "qualification_policy",+ "opts": [+ "--qualification-policy"+ ],+ "param_type": "option",+ "type": "path"}]},diff --git a/tests/reporting/test_resource_comparison.py b/tests/reporting/test_resource_comparison.pyindex 3aef8e70..ac50ff37 100644--- a/tests/reporting/test_resource_comparison.py+++ b/tests/reporting/test_resource_comparison.py@@ -10,10 +10,10 @@ from skillevaluator.reporting import HTMLReporter, JSONReporterfrom skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS-def write_arm(run, arm, tokens, seconds, steps=None):+def write_arm(run, arm, tokens, seconds, steps=None, score=1.0):directory = run / "opencode" / armdirectory.mkdir(parents=True)- scores = dict.fromkeys(DEFAULT_METRICS, 1.0)+ scores = dict.fromkeys(DEFAULT_METRICS, score)(directory / "summary.json").write_text(json.dumps({@@ -116,6 +116,51 @@ def render(tmp_path, run):return result, resources, html, output[email protected]("reused_baseline", [False, True])+def test_failed_task_outcomes_keep_observed_deltas_and_disclosures(tmp_path, reused_baseline):+ run = tmp_path / "run"+ write_arm(run, "with-skill", 10, 3, score=0.0)+ write_arm(run, "without-skill", 20, 5)+ if reused_baseline:+ (run / "run_config.json").write_text(+ json.dumps(+ {+ "baseline": {+ "mode": "reused",+ "source_run_id": "earlier-run",+ "source_evaluated_at": "2026-10-01T00:00:00Z",+ }+ }+ )+ )+ result, resources, html, output = render(tmp_path, run)+ agent = output["tier3"]["agents"]["opencode"]+ assert agent["execution_status"] == "succeeded"+ assert agent["trials"][0]["scores"]["goal_accuracy"] == 0.0+ assert agent["trials"][0]["attempt_verdict"]["passed"] is False+ assert agent["trials_baseline"][0]["scores"]["goal_accuracy"] == 1.0+ assert agent["trials_baseline"][0]["attempt_verdict"]["passed"] is True+ assert agent["resources"]["tokens"]["delta"] == -10+ assert agent["resources"]["agent_seconds"]["delta"] == -2+ assert "-10 (-45%)" in html+ assert "-2.0 s (-40%)" in html+ assert "including failed task outcomes" in resources["scope"]+ assert "including failed task outcomes" in html+ for disclosure in (+ "Lower use alone does not establish improved task quality or Efficiency",+ "Pairing does not certify matched concurrency or budget, or judge-model agreement",+ "Reused baseline measurements are historical",+ ):+ assert disclosure in agent["resources"]["interpretation"]+ assert disclosure in html+ if reused_baseline:+ assert output["tier3"]["token_usage"]["baseline_reused"] is True+ # Reports retained before interpretation metadata existed still disclose its limits.+ del resources["interpretation"]+ legacy_html = HTMLReporter(target_path=str(tmp_path)).render_all([result])+ assert "Lower use alone does not establish improved task quality or Efficiency" in legacy_html++def test_llm_and_tool_seconds_flow_from_trajectories_to_json_and_html(tmp_path):run = tmp_path / "run"write_arm(run, "with-skill", 10, 3, steps=codex_steps(tool=1.5, llm_after=4.5))diff --git a/tests/tier3/test_grader_policy.py b/tests/tier3/test_grader_policy.pynew file mode 100644index 00000000..70da75b6--- /dev/null+++ b/tests/tier3/test_grader_policy.py@@ -0,0 +1,344 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++from __future__ import annotations++import copy+import hashlib+import json+import os++import pytest++from skillevaluator.tier3 import grader_policy as gp+from skillevaluator.tier3 import grader_validation as gv+++def policy(**changes):+ return {+ "schema": "skillevaluator.grader-qualification-policy/v1",+ "policy_id": "experimental-test-v1",+ "required_metrics": ["accuracy"],+ "min_class_cases": {"pass": 1, "fail": 1},+ "min_category_cases": {"positive": 1, "negative": 1},+ "max_rates": {"false_pass_rate": 0.1, "false_fail_rate": 0.1, "unjudgeable_rate": 0.05},+ } | changes+++def cases():+ return [+ {+ "id": "p",+ "split": "assessment",+ "category": "positive",+ "expected": {"overall": "pass", "metrics": {"accuracy": "pass"}},+ },+ {+ "id": "f",+ "split": "assessment",+ "category": "negative",+ "expected": {"overall": "fail", "metrics": {"accuracy": "fail"}},+ },+ ]+++def report(selected, observed=None, repeats=1):+ attempts = []+ for case in selected:+ for repeat in range(repeats):+ rows = gv.observations(case, {}, gv.DEFAULT_THRESHOLDS)+ for row in rows:+ row["observed"] = observed or row["expected"]+ attempts.append({"case": case["id"], "repeat": repeat + 1, "observations": rows})+ metrics = {+ m: gv.summarize([r for a in attempts for r in a["observations"] if r["metric"] == m])+ for m in ("accuracy", "overall", "behavior_check", "behavior_steps")+ }+ return {"attempts": attempts, "metrics": metrics, "provenance": {"thresholds": gv.DEFAULT_THRESHOLDS}}+++def test_policy_identity_and_label_only_readiness(tmp_path):+ path = tmp_path / "policy.json"+ path.write_text(json.dumps(policy(), indent=2))+ loaded = gp.load_qualification_policy(path)+ decision = gp.preflight_qualification(cases(), gv.DEFAULT_THRESHOLDS, loaded)+ assert decision["status"] == "ready"+ assert decision["phase"] == "preflight"+ assert decision["policy"] == policy()+ assert (+ decision["policy_sha256"]+ == hashlib.sha256(+ json.dumps(policy(), sort_keys=True, separators=(",", ":"), ensure_ascii=True, allow_nan=False).encode()+ ).hexdigest()+ )+ assert decision["metrics"]["accuracy"]["expected_class_counts"] == {"pass": 1, "fail": 1}+ assert decision["metrics"]["accuracy"]["scored_class_counts"] is None+ assert decision["metrics"]["accuracy"]["rate_checks"] is None+ assert decision["categories"]["positive"]["observed_cases"] is None+++ "mutation",+ [+ lambda p: p.update(extra=True),+ lambda p: p.update(schema="v2"),+ lambda p: p.update(policy_id="../unsafe"),+ lambda p: p.update(policy_id="é"),+ lambda p: p.update(policy_id="a" * 129),+ lambda p: p.update(description=""),+ lambda p: p.update(description="x" * 1025),+ lambda p: p.update(required_metrics=[]),+ lambda p: p.update(required_metrics=["accuracy", "accuracy"]),+ lambda p: p.update(required_metrics=["unknown"]),+ lambda p: p.update(required_metrics=[{}]),+ lambda p: p.update(min_class_cases={"pass": True, "fail": 1}),+ lambda p: p.update(min_class_cases={"pass": 1}),+ lambda p: p["min_class_cases"].update(fail=0),+ lambda p: p.update(min_category_cases={"unknown": 1}),+ lambda p: p.update(min_category_cases={"positive": False}),+ lambda p: p["max_rates"].update(false_pass_rate=True),+ lambda p: p["max_rates"].update(false_pass_rate=1.1),+ lambda p: p["max_rates"].update(false_pass_rate=10**400),+ lambda p: p["max_rates"].update(false_pass_rate=float("nan")),+ lambda p: p["max_rates"].update(false_pass_rate=float("inf")),+ lambda p: p["max_rates"].pop("false_fail_rate"),+ ],+)+def test_reject_invalid_policy(tmp_path, mutation):+ value = policy()+ mutation(value)+ path = tmp_path / "policy.json"+ path.write_text(json.dumps(value))+ with pytest.raises(ValueError):+ gp.load_qualification_policy(path)++[email protected]("content", ["{", '"scalar"', "[" * 150 + "]" * 150, " " * (256 * 1024 + 1)])+def test_bounded_policy_read(tmp_path, content):+ path = tmp_path / "policy.json"+ path.write_text(content)+ with pytest.raises(ValueError):+ gp.load_qualification_policy(path)+++def test_nonregular_policy(tmp_path):+ with pytest.raises(ValueError):+ gp.load_qualification_policy(tmp_path)++[email protected](not hasattr(os, "mkfifo"), reason="platform has no FIFO creation")+def test_fifo_policy_rejected_without_blocking(tmp_path):+ fifo = tmp_path / "fifo"+ os.mkfifo(fifo)+ with pytest.raises(ValueError):+ gp.load_qualification_policy(fifo)+++def test_missing_required_metric_class_category_and_selected_cases():+ p = policy(required_metrics=["accuracy", "security"], min_category_cases={"baseline": 1})+ decision = gp.preflight_qualification(cases()[:1], gv.DEFAULT_THRESHOLDS, p)+ assert decision["status"] == "insufficient_evidence"+ assert {r["code"] for r in decision["reasons"]} >= {+ "missing_metric",+ "expected_class_coverage",+ "expected_category_coverage",+ }+ assert decision["scope"]["case_ids"] == ["p"]+ assert gp.preflight_qualification(cases(), gv.DEFAULT_THRESHOLDS, policy())["status"] == "ready"+++def test_evaluated_counts_and_rate_equality():+ selected = cases()+ value = report(selected, repeats=50)+ decision = gp.evaluate_qualification(value, selected, policy(max_rates=dict.fromkeys(policy()["max_rates"], 0)))+ assert decision["status"] == "passed"+ assert decision["phase"] == "evaluated"+ assert decision["metrics"]["accuracy"]["scored_class_counts"] == {"pass": 1, "fail": 1}+ for check in decision["metrics"]["accuracy"]["rate_checks"].values():+ assert check["met"] is True+ assert check["measurement"]["wilson_95"] is not None+ stricter = policy(min_class_cases={"pass": 2, "fail": 2})+ assert gp.evaluate_qualification(value, selected, stricter)["status"] == "insufficient_evidence"++[email protected]("name", ["false_pass_rate", "false_fail_rate", "unjudgeable_rate"])+def test_each_exceeded_rate_fails(name):+ value = report(cases())+ value["metrics"]["accuracy"][name] = gv.rate(1, 2)+ p = policy(max_rates=dict.fromkeys(policy()["max_rates"], 0.5))+ assert gp.evaluate_qualification(value, cases(), p)["status"] == "passed"+ p["max_rates"][name] = 0.49+ decision = gp.evaluate_qualification(value, cases(), p)+ assert decision["status"] == "failed"+ assert any(r["code"] == "rate_exceeded" and r["rate"] == name for r in decision["reasons"])++[email protected]("observed", ["error", "insufficient_evidence", "not_applicable"])+def test_unscored_cannot_satisfy_class_or_category(observed):+ decision = gp.evaluate_qualification(report(cases(), observed), cases(), policy())+ assert decision["status"] == "insufficient_evidence"+ assert decision["metrics"]["accuracy"]["scored_class_counts"] == {"pass": 0, "fail": 0}+ assert decision["categories"]["positive"]["observed_cases"] == 0+ assert {r["code"] for r in decision["reasons"]} >= {+ "scored_class_coverage",+ "observed_category_coverage",+ "missing_rate",+ }++[email protected]("missing", ["metric", "rate", "zero"])+def test_missing_metric_or_rate_never_passes(missing):+ value = report(cases())+ if missing == "metric":+ value["metrics"].pop("accuracy")+ elif missing == "rate":+ value["metrics"]["accuracy"].pop("false_pass_rate")+ else:+ value["metrics"]["accuracy"]["false_pass_rate"] = gv.rate(0, 0)+ assert gp.evaluate_qualification(value, cases(), policy())["status"] == "insufficient_evidence"+++def test_many_behavior_steps_are_distinct_cases_per_class():+ selected = cases()[:1]+ selected[0]["expected"]["metrics"] = {"behavior_check": [True, False] * 50}+ value = report(selected, repeats=50)+ p = policy(required_metrics=["behavior_steps"], min_category_cases={})+ decision = gp.evaluate_qualification(value, selected, p)+ assert decision["status"] == "passed"+ assert decision["metrics"]["behavior_steps"]["expected_class_counts"] == {"pass": 1, "fail": 1}+ assert decision["metrics"]["behavior_steps"]["scored_class_counts"] == {"pass": 1, "fail": 1}+ p["min_class_cases"] = {"pass": 2, "fail": 2}+ assert gp.evaluate_qualification(value, selected, p)["status"] == "insufficient_evidence"+++def test_tuning_observations_cannot_fill_assessment_coverage():+ selected = cases()+ tuning = copy.deepcopy(selected[1])+ tuning.update(id="tuning", split="tuning")+ value = report([selected[0], tuning])+ with pytest.raises(ValueError, match="scope"):+ gp.evaluate_qualification(value, selected, policy())+ with pytest.raises(ValueError, match="assessment"):+ gp.preflight_qualification([tuning], gv.DEFAULT_THRESHOLDS, policy())+++def test_duplicate_json_fields_rejected(tmp_path):+ path = tmp_path / "policy.json"+ content = json.dumps(policy()).replace('"false_pass_rate": 0.1', '"false_pass_rate": 0.9, "false_pass_rate": 0.1')+ path.write_text(content)+ with pytest.raises(ValueError, match="duplicate"):+ gp.load_qualification_policy(path)+++def test_behavior_aggregate_expected_class_uses_suite_threshold():+ selected = cases()+ for case in selected:+ case["expected"]["metrics"] = {"behavior_check": [True, False] if case["id"] == "p" else [False, False]}+ decision = gp.preflight_qualification(+ selected, gv.DEFAULT_THRESHOLDS | {"behavior_check": 0.5}, policy(required_metrics=["behavior_check"])+ )+ assert decision["status"] == "ready"+ assert decision["metrics"]["behavior_check"]["expected_class_counts"] == {"pass": 1, "fail": 1}+++def test_experimental_example_intentionally_fails_bundled_label_coverage():+ from pathlib import Path++ example = Path(__file__).resolve().parents[2] / "examples/grader-qualification-policy.json"+ value = gp.load_qualification_policy(example)+ assert set(value["required_metrics"]) == {*gv.METRICS, "behavior_steps", "overall"}+ assert set(value["min_category_cases"]) == gv.CATEGORIES+ selected = [case for case in gv.load_suite(gv.REFERENCE_SUITE)["cases"] if case["split"] == "assessment"]+ decision = gp.preflight_qualification(selected, gv.DEFAULT_THRESHOLDS, value)+ assert decision["status"] == "insufficient_evidence"+ assert {(r["metric"], r["class"]) for r in decision["reasons"]} == {+ ("security", "fail"),+ ("skill_efficiency", "fail"),+ }+++def test_policy_read_without_nonblocking_flag(tmp_path, monkeypatch):+ path = tmp_path / "policy.json"+ path.write_text(json.dumps(policy()))+ monkeypatch.delattr(gp.os, "O_NONBLOCK", raising=False)+ assert gp.load_qualification_policy(path) == policy()++[email protected]("label", ["not_applicable", "insufficient_evidence"])[email protected]("observed", ["matching", "error", "pass", "fail", "other_abstention"])+def test_category_observed_coverage_distinguishes_correct_expected_abstentions(label, observed):+ selected = cases()+ unavailable = {+ "id": "u",+ "split": "assessment",+ "category": "unavailable_evidence",+ "expected": {"overall": "insufficient_evidence", "metrics": {"accuracy": label}},+ }+ selected.append(unavailable)+ value = report(selected)+ verdict = (+ label+ if observed == "matching"+ else ("not_applicable" if label == "insufficient_evidence" else "insufficient_evidence")+ if observed == "other_abstention"+ else observed+ )+ for row in value["attempts"][-1]["observations"]:+ if row["metric"] == "accuracy":+ row["observed"] = verdict+ value["metrics"]["accuracy"] = gv.summarize(+ [row for attempt in value["attempts"] for row in attempt["observations"] if row["metric"] == "accuracy"]+ )+ criteria = policy(min_category_cases={"unavailable_evidence": 1}, max_rates=dict.fromkeys(gp.RATES, 0))+ decision = gp.evaluate_qualification(value, selected, criteria)+ assert decision["status"] == ("passed" if observed == "matching" else "insufficient_evidence")+ category = decision["categories"]["unavailable_evidence"]+ assert category["observed_cases"] == int(observed == "matching")+ assert category["observed_met"] is (observed == "matching")+ assert "scored_cases" not in category and "scored_met" not in category+ assert decision["metrics"]["accuracy"]["scored_class_counts"] == {"pass": 1, "fail": 1}+ for check in decision["metrics"]["accuracy"]["rate_checks"].values():+ assert check["measurement"]["rate"] == 0+++def test_broader_report_cannot_dilute_selected_case_false_passes():+ selected = cases()+ extras = [dict(copy.deepcopy(selected[1]), id=f"extra-{i}") for i in range(20)]+ value = report(selected + extras)+ selected_fail = value["attempts"][1]+ for row in selected_fail["observations"]:+ row["observed"] = "pass"+ value["metrics"]["accuracy"] = gv.summarize(+ [row for attempt in value["attempts"] for row in attempt["observations"] if row["metric"] == "accuracy"]+ )+ assert value["metrics"]["accuracy"]["false_pass_rate"]["rate"] == pytest.approx(1 / 21)+ assert gp.evaluate_qualification(value, selected + extras, policy())["status"] == "passed"+ with pytest.raises(ValueError, match="scope"):+ gp.evaluate_qualification(value, selected, policy())+ exact = report(selected)+ exact["attempts"][1] = selected_fail+ exact["metrics"]["accuracy"] = gv.summarize(+ [row for attempt in exact["attempts"] for row in attempt["observations"] if row["metric"] == "accuracy"]+ )+ decision = gp.evaluate_qualification(exact, selected, policy())+ assert decision["status"] == "failed"+ assert decision["metrics"]["accuracy"]["rate_checks"]["false_pass_rate"]["measurement"]["rate"] == 1+++ "mutation",+ [+ lambda r: r["provenance"].update(case_ids=["p", "f", "extra"]),+ lambda r: r["provenance"].update(case_ids=["p"]),+ lambda r: r["provenance"].update(case_ids=["p", "f", "p"]),+ lambda r: r["provenance"].update(split="tuning"),+ lambda r: r["attempts"].pop(),+ lambda r: r["attempts"][0].update(case="extra"),+ lambda r: r["attempts"][0]["observations"][0].update(judgment="extra:overall"),+ ],+)+def test_report_provenance_and_attempt_scope_must_agree(mutation):+ value = report(cases())+ mutation(value)+ with pytest.raises(ValueError, match="scope"):+ gp.evaluate_qualification(value, cases(), policy())diff --git a/tests/tier3/test_grader_validation.py b/tests/tier3/test_grader_validation.pyindex 6f75201a..03bf6fe8 100644--- a/tests/tier3/test_grader_validation.py+++ b/tests/tier3/test_grader_validation.py@@ -493,6 +493,8 @@ def test_verifier_provenance_preserves_relative_hashes_through_source_alias(name: hashlib.sha256(content).hexdigest() for name, content in contents.items()}assert report["provenance"]["verifier_sha256"] == expected_digest.hexdigest()+ assert report["provenance"]["verifier_git_commit"] is None+ assert report["provenance"]["verifier_git_dirty"] is Noneassert json.loads((output / "grader-validation.json").read_text())["provenance"] == report["provenance"]@@ -640,3 +642,350 @@ def test_real_judge_failure_is_unscored(judge_server, tmp_path):assert report["metrics"]["accuracy"]["false_fail_rate"]["denominator"] == 0assert report["metrics"]["overall"]["errors"] == 1assert report["metrics"]["security"]["confusion_matrix"]["pass"]["pass"] == 1+++def qualification_policy(tmp_path):+ path = tmp_path / "policy.json"+ path.write_text(+ json.dumps(+ {+ "schema": "skillevaluator.grader-qualification-policy/v1",+ "policy_id": "test-overall-v1",+ "required_metrics": ["overall"],+ "min_class_cases": {"pass": 1, "fail": 1},+ "min_category_cases": {},+ "max_rates": {"false_pass_rate": 0.1, "false_fail_rate": 0.1, "unjudgeable_rate": 0.05},+ }+ )+ )+ return path+++def test_policy_offline_readiness_no_provider_or_output(monkeypatch, qualification_policy, tmp_path):+ monkeypatch.setattr(gv, "resolve_llm_provider", lambda: pytest.fail("provider accessed"))+ output = tmp_path / "not-written"+ runner = CliRunner()+ ready = runner.invoke(+ cli,+ [+ "tier3",+ "validate-grader",+ "--qualification-policy",+ str(qualification_policy),+ "--check-suite",+ "--json",+ "-o",+ str(output),+ ],+ )+ assert ready.exit_code == 0, ready.output+ assert json.loads(ready.output)["qualification"]["status"] == "ready"+ insufficient = runner.invoke(+ cli,+ [+ "tier3",+ "validate-grader",+ "--qualification-policy",+ str(qualification_policy),+ "--check-suite",+ "--case",+ "arithmetic",+ "--json",+ "-o",+ str(output),+ ],+ )+ assert insufficient.exit_code == 1, insufficient.output+ assert json.loads(insufficient.output)["qualification"]["status"] == "insufficient_evidence"+ assert not output.exists()+++def test_policy_insufficient_preflight_retains_not_run_receipt(monkeypatch, qualification_policy, tmp_path):+ monkeypatch.setattr(gv, "resolve_llm_provider", lambda: pytest.fail("provider accessed"))+ output = tmp_path / "preflight"+ args = [+ "tier3",+ "validate-grader",+ "--qualification-policy",+ str(qualification_policy),+ "--case",+ "arithmetic",+ "--json",+ "-o",+ str(output),+ ]+ result = CliRunner().invoke(cli, args)+ assert result.exit_code == 1, result.output+ report = json.loads(result.output)+ assert report["attempts"] == []+ assert report["execution_status"] == "not_run"+ assert report["qualification"]["phase"] == "preflight"+ assert report["qualification"]["status"] == "insufficient_evidence"+ assert report["provenance"]["case_ids"] == ["arithmetic"]+ assert report["provenance"]["provider"] is None+ assert json.loads((output / "grader-validation.json").read_text()) == report+ assert len(list(output.iterdir())) == 1+ assert CliRunner().invoke(cli, args).exit_code == 2+++def test_policy_human_coverage_reason(monkeypatch, qualification_policy, tmp_path):+ monkeypatch.setattr(gv, "resolve_llm_provider", lambda: pytest.fail("provider accessed"))+ result = CliRunner().invoke(+ cli,+ [+ "tier3",+ "validate-grader",+ "--qualification-policy",+ str(qualification_policy),+ "--case",+ "arithmetic",+ "-o",+ str(tmp_path / "human"),+ ],+ )+ assert result.exit_code == 1, result.output+ assert "overall" in result.output and "fail" in result.output and "coverage" in result.output++[email protected]("args", [["--split", "tuning"], ["--split", "all"]])+def test_policy_assessment_only(monkeypatch, qualification_policy, args):+ monkeypatch.setattr(gv, "resolve_llm_provider", lambda: pytest.fail("provider accessed"))+ result = CliRunner().invoke(+ cli, ["tier3", "validate-grader", "--qualification-policy", str(qualification_policy), "--check-suite", *args]+ )+ assert result.exit_code == 2+ assert "assessment" in result.output+++def test_malformed_policy_before_provider(monkeypatch, qualification_policy, tmp_path):+ monkeypatch.setattr(gv, "resolve_llm_provider", lambda: pytest.fail("provider accessed"))+ qualification_policy.write_text('{"schema":"wrong"}')+ output = tmp_path / "never-written"+ result = CliRunner().invoke(+ cli, ["tier3", "validate-grader", "--qualification-policy", str(qualification_policy), "-o", str(output)]+ )+ assert result.exit_code == 2+ assert not output.exists()+++def test_policy_real_template_decision_equals_receipt(judge_server, qualification_policy, tmp_path):+ output = tmp_path / "policy-live"+ result = CliRunner().invoke(+ cli,+ [+ "tier3",+ "validate-grader",+ "--qualification-policy",+ str(qualification_policy),+ "--case",+ "arithmetic",+ "--case",+ "wrong-answer",+ "--repeats",+ "1",+ "--json",+ "-o",+ str(output),+ ],+ )+ assert result.exit_code == 1, result.output+ report = json.loads(result.output)+ assert report["qualification"]["phase"] == "evaluated"+ assert report["qualification"]["status"] == "failed"+ assert report["qualification"]["metrics"]["overall"]["scored_class_counts"] == {"pass": 1, "fail": 1}+ assert report["qualification"]["metrics"]["overall"]["rate_checks"]["false_pass_rate"]["measurement"]["rate"] == 1+ assert report == json.loads((output / "grader-validation.json").read_text())+ assert report["provenance"]["verifier_git_commit"] is not None+ assert report["provenance"]["verifier_git_dirty"] is False+++def test_verifier_git_identity_requires_tracked_files(tmp_path):+ import subprocess++ repository = tmp_path / "repository"+ repository.mkdir()++ def git(*args):+ return subprocess.run(+ ["git", "-C", str(repository), *args], check=True, capture_output=True, text=True+ ).stdout.strip()++ git("init")+ tracked = repository / "verifier.py"+ tracked.write_text("original\n")+ git("add", "verifier.py")+ git(+ "-c",+ "user.name=Test",+ "-c",+ "[email protected]",+ "-c",+ "commit.gpgsign=false",+ "commit",+ "-m",+ "test",+ )+ expected = git("rev-parse", "HEAD")+ assert gv._verifier_git_provenance([tracked]) == {"verifier_git_commit": expected, "verifier_git_dirty": False}+ tracked.write_text("changed\n")+ assert gv._verifier_git_provenance([tracked]) == {"verifier_git_commit": expected, "verifier_git_dirty": True}+ untracked = repository / "site-packages" / "verifier.py"+ untracked.parent.mkdir()+ untracked.write_text("installed\n")+ unknown = {"verifier_git_commit": None, "verifier_git_dirty": None}+ assert gv._verifier_git_provenance([untracked]) == unknown+ assert gv._verifier_git_provenance([tracked, untracked]) == unknown+ assert gv._verifier_git_provenance([tmp_path / "outside.py"]) == unknown++[email protected]("judge_error", [False, True])+def test_policy_live_pass_or_insufficient(judge_server, qualification_policy, tmp_path, judge_error):+ value = json.loads(qualification_policy.read_text())+ value["max_rates"] = dict.fromkeys(value["max_rates"], 1.0)+ qualification_policy.write_text(json.dumps(value))+ judge_server["fail"] = judge_error+ output = tmp_path / "live"+ result = CliRunner().invoke(+ cli,+ [+ "tier3",+ "validate-grader",+ "--qualification-policy",+ str(qualification_policy),+ "--case",+ "arithmetic",+ "--case",+ "wrong-answer",+ "--repeats",+ "1",+ "--json",+ "-o",+ str(output),+ ],+ )+ assert result.exit_code == int(judge_error), result.output+ report = json.loads(result.output)+ assert report["qualification"]["status"] == ("insufficient_evidence" if judge_error else "passed")+ assert report == json.loads((output / "grader-validation.json").read_text())+++def test_verifier_git_ignores_unrelated_dirty_files_and_git_environment(tmp_path, monkeypatch):+ import subprocess++ repository = tmp_path / "source"+ repository.mkdir()++ def git(*args):+ return subprocess.run(+ ["git", "-C", str(repository), *args], check=True, capture_output=True, text=True+ ).stdout.strip()++ git("init")+ verifier = repository / "verifier.py"+ verifier.write_text("source\n")+ unrelated = repository / "other.py"+ unrelated.write_text("clean\n")+ git("add", ".")+ git(+ "-c",+ "user.name=Test",+ "-c",+ "[email protected]",+ "-c",+ "commit.gpgsign=false",+ "commit",+ "-m",+ "test",+ )+ expected = git("rev-parse", "HEAD")+ unrelated.write_text("dirty\n")+ monkeypatch.setenv("GIT_DIR", str(tmp_path / "does-not-exist"))+ assert gv._verifier_git_provenance([verifier]) == {"verifier_git_commit": expected, "verifier_git_dirty": False}+++ "args",+ [+ ["--timeout", "nan"],+ ["--timeout", "inf"],+ ["--max-false-pass-rate", "nan"],+ ["--max-false-fail-rate", "nan"],+ ["--max-unjudgeable-rate", "nan"],+ ],+)+def test_nonfinite_options_rejected_before_provider(monkeypatch, args):+ monkeypatch.setattr(gv, "resolve_llm_provider", lambda: pytest.fail("provider accessed"))+ result = CliRunner().invoke(cli, ["tier3", "validate-grader", *args])+ assert result.exit_code == 2, result.output+ assert "finite" in result.output+++def test_policy_cli_without_platform_nonblocking_flag(monkeypatch, qualification_policy):+ monkeypatch.delattr(gv.os, "O_NONBLOCK", raising=False)+ monkeypatch.setattr(gv, "resolve_llm_provider", lambda: pytest.fail("provider accessed"))+ result = CliRunner().invoke(+ cli,+ ["tier3", "validate-grader", "--qualification-policy", str(qualification_policy), "--check-suite", "--json"],+ )+ assert result.exit_code == 0, result.output+ assert json.loads(result.output)["qualification"]["status"] == "ready"+++def test_live_correct_unavailable_case_meets_observed_category_only(judge_server, qualification_policy, tmp_path):+ value = json.loads(qualification_policy.read_text())+ value["min_category_cases"] = {"unavailable_evidence": 1}+ value["max_rates"] = dict.fromkeys(value["max_rates"], 1)+ qualification_policy.write_text(json.dumps(value))+ output = tmp_path / "observed-category"+ result = CliRunner().invoke(+ cli,+ [+ "tier3",+ "validate-grader",+ "--qualification-policy",+ str(qualification_policy),+ "--case",+ "arithmetic",+ "--case",+ "wrong-answer",+ "--case",+ "missing",+ "--repeats",+ "1",+ "--json",+ "-o",+ str(output),+ ],+ )+ assert result.exit_code == 0, result.output+ receipt = json.loads(result.output)+ decision = receipt["qualification"]+ assert decision["status"] == "passed"+ assert decision["categories"]["unavailable_evidence"]["observed_cases"] == 1+ assert decision["categories"]["unavailable_evidence"]["observed_met"] is True+ assert decision["metrics"]["overall"]["scored_class_counts"] == {"pass": 1, "fail": 1}+ assert receipt == json.loads((output / "grader-validation.json").read_text())++[email protected]("flag", ["--max-false-pass-rate", "--max-false-fail-rate", "--max-unjudgeable-rate"])[email protected]("offline", [True, False])+def test_policy_rejects_legacy_gate_flags_before_provider(monkeypatch, qualification_policy, tmp_path, flag, offline):+ monkeypatch.setattr(gv, "resolve_llm_provider", lambda: pytest.fail("provider accessed"))+ output = tmp_path / "not-written"+ args = [+ "tier3",+ "validate-grader",+ "--qualification-policy",+ str(qualification_policy),+ flag,+ "0",+ "-o",+ str(output),+ ]+ if offline:+ args.append("--check-suite")+ result = CliRunner().invoke(cli, args)+ assert result.exit_code == 2, result.output+ assert "cannot combine" in result.output+ assert not output.exists()