Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix: target dependency coverage and Tier 3 result integrity (#196)
Christopher Kevin87274f440401
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5a3b8b73..36c73254 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -85,31 +85,16 @@ All notable changes to SkillEvaluator are documented in this file.### Fixed-- Include schema-valid ATIF message content parts in final answers, conversation- summaries, and judge evidence references in the host and Harbor verifier.- Image and audio parts use markers without exposing their source paths.-- Preserve scored Harbor trials when agent answers mention authentication errors- and per-step token usage is present but aggregate ATIF token metrics are absent- or zero. Explicit runtime exceptions and error events still invalidate trials.-- Safety-classifier refusals, content filters and output-token truncation are- detected instead of being read as empty or malformed output. `--llm-verify`- and rubric evaluation report the cause, and `--llm-verify` keeps the- complete verdicts of a truncated reply. Claude 5.5 or later requests to the- native Claude API use server-side fallback to Claude Opus 4.8. In the Harbor- verifier, Bedrock retries a filtered Claude 5.5 reply on Claude Opus 4.8, and- a refusal or empty judge reply moves on to the next- `LLM_JUDGE_FALLBACK_MODELS` entry.-- The Harbor verifier's last-resort judge model follows the selected provider.- With no judge or chat model configured, it used to request an OpenAI model- from Anthropic, Bedrock, NVIDIA Build and gateways. The unused host-side- `llm_judge.DEFAULT_JUDGE_MODEL` constant is removed.-- Fixed Docker Codex and Claude Code setup wasting its timeout installing- distribution Node/npm before Harbor's own installers. Native, gateway Codex,- and NVIDIA Build agents now provision Node/npm only on the installer branches- that need them, while preserving installed-agent and version-pin checks.-- Preserved multiline skill descriptions when saving and loading Tier 2- embedding catalogs. Descriptions can contain line breaks and tabs, while- catalog identifiers and other control characters remain strictly validated.+- Tier 1 pyproject dependency audits now inspect static target declarations+ rather than the evaluator environment. Exact direct pins (including optional+ groups) are queried without installation or build-script execution;+ unresolved declarations and invalid scanner evidence are incomplete.+- Tier 1 PII scanning now includes `.mdc` rules and workflows.+- Tier 3 negative-case accuracy excludes inapplicable skill identification and+ records its scoring policy. Reports separately expose measured token and+ agent execution-time differences with complete matching-arm coverage, while+ efficiency continues to measure routing and productive tool use.+- Brought the published documentation in line with 0.5.0. It now covers howstandard grading builds judge evidence and rebuilds trajectories fromOpenCode and Codex logs, structured judge output and retries, the evidencediff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.mdindex 4df5ff18..6a952df9 100644--- a/THIRD_PARTY_NOTICES.md+++ b/THIRD_PARTY_NOTICES.md@@ -6,7 +6,7 @@ records the exact resolved dependency set used for this release.| Dependency group | Packages and licenses || --- | --- |-| Base | Click (BSD-3-Clause), IDNA (BSD-3-Clause), Jinja2 (BSD-3-Clause), Markdown-It-Py (MIT), Pydantic (MIT), PyYAML (MIT), Rich (MIT) |+| Base | Click (BSD-3-Clause), IDNA (BSD-3-Clause), Jinja2 (BSD-3-Clause), Markdown-It-Py (MIT), Packaging (Apache-2.0 or BSD-2-Clause), Pydantic (MIT), PyYAML (MIT), Rich (MIT) || LLM | Anthropic (MIT), Boto3 (Apache-2.0), LiteLLM (MIT), OpenAI (Apache-2.0) || Tier 3 | Harbor (Apache-2.0), MCP (MIT), PyJWT (MIT), python-dotenv (BSD-3-Clause) || Security | Bandit (Apache-2.0), pip-audit (Apache-2.0) |diff --git a/docs/reports.mdx b/docs/reports.mdxindex 0a353539..6121ac76 100644--- a/docs/reports.mdx+++ b/docs/reports.mdx@@ -297,6 +297,16 @@ Lift requires both arms; if you ran with `--skip-baseline`, there is no lift toreport. In `custom_only` grading, your own grader defines the score and lift —see [Custom Graders & Tasks](custom-graders.mdx).+### Measured agent resources++Each agent's canonical JSON payload contains a `resources` object. The HTML+table shows total prompt plus completion tokens and agent execution seconds for+each arm, telemetry coverage, and the with-skill-minus-baseline difference.+Routing/tool-productivity efficiency remains a separate score. Missing counters+and incomplete arms produce null totals or deltas (`N/A` in HTML). Resource+deltas require matching case counts and complete measurements in both arms;+they do not measure evaluator tokens, setup time, price, or statistical savings.+### pass@kpass@k is the reliability signal, reported separately from the dimensiondiff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdxindex a4f10c54..8f8253aa 100644--- a/docs/tier1-validation.mdx+++ b/docs/tier1-validation.mdx@@ -177,6 +177,8 @@ For `validate`, the default set is `schema`, `version`, `security`, `pii`, `lice<Accordion title="pii — PII, credentials, and local identifiers">`pii-scan`, and the `pii` check inside `validate`, use configurable regex patterns to detect sensitive values.+ Instruction content in `.mdc` rules and workflow files is included, for both+ direct file inputs and directory scans.| Group | Examples | Notes || --- | --- | --- |@@ -290,7 +292,21 @@ For `validate`, the default set is `schema`, `version`, `security`, `pii`, `lice</Accordion><Accordion title="dependency — CVE audit (opt-in)">- Scans `requirements*.txt` for known Python package vulnerabilities. When `pyproject.toml` is present, `pip-audit --local` audits the active Python environment rather than resolving the dependencies declared in that file. `pip-audit` is the primary scanner; Safety supplies secondary coverage for requirements files when installed separately. The audit may use network-backed vulnerability databases.+ Scans `requirements*.txt` for known Python package vulnerabilities. For+ `pyproject.toml`, reads static PEP 621 dependencies and every optional group,+ then audits their exact version pins without installing the target or+ executing its build scripts. This covers direct declarations only;+ transitive dependencies require a resolved requirements file. Markers are+ evaluated against the scanner's Python/platform and optional-group context;+ nonmatching declarations are reported as skipped. Version ranges, URL/path+ dependencies, dynamic or Poetry metadata, malformed declarations, and+ missing or failed scanner evidence leave the audit `INCOMPLETE`. Supply+ pinned `requirements.txt` for unresolved declarations. `pip-audit` is the+ primary scanner; Safety supplies secondary coverage for requirements files+ when installed separately. The audit may use network-backed vulnerability databases.+ JSON and HTML export data retain `dependency_coverage` per target, including+ requested and actually audited pins. Unreadable declarations have unknown+ declaration counts and an empty audited inventory, not a completed scan.```bash title="Opt-in dependency audit"skillevaluator validate ./my-skill --checks dependencydiff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex 816903bb..d9acd954 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -34,7 +34,7 @@ human-readable dimensions:| Correctness | Does it do what it's supposed to? || Discoverability | Is it loaded when it should be? || Effectiveness | Is it better with the skill than without? |-| Efficiency | Does it use fewer tool calls and tokens? |+| Efficiency | Does it route correctly and use tools productively? |In `custom_only` mode, your own grader defines the score instead — see[Custom Graders & Tasks](custom-graders.mdx).@@ -551,6 +551,12 @@ grades each trial after the agent finishes. `security`, `skill_execution`, and`goal_accuracy`, and `behavior_check` are LLM judges that call the configuredevaluator provider. `custom_only` runs none of them.+When `should_trigger` is false or no `expected_skill` is declared, accuracy+excludes `SKILL_IDENTIFIED` and calculates the score from the other four+criterion verdicts. The reward details record the excluded criterion and+`applicable-accuracy-criteria/v1` scoring policy. Missing criterion evidence is+an error. Positive cases retain their existing scoring contract.+### The graded trajectoryThe verifier grades Harbor's ATIF `trajectory.json`. When that file is missing@@ -682,6 +688,16 @@ a bare exception.`token_efficiency` is a standalone report signal, not a dimension source. Itnever changes a dimension score or the overall verdict.+The report's **Measured Agent Resources** table separately compares prompt plus+completion tokens and Harbor's measured agent execution seconds. Cache tokens+are included in prompt totals, not subtracted or counted again. Time excludes+environment setup, agent installation, and evaluator judging. Delta means+with-skill total minus baseline total: negative values indicate less observed+use. A delta requires complete telemetry, complete successful arms, and matching+case counts including repeated attempts. Missing measurements, skipped baselines,+partial runs, and unsupported multi-step timing remain `N/A`; they are not zero.+These are observations for this run, not a graded efficiency or cost claim.+Each dimension receives a score and a diagnostic band: **PASS** at 0.50 orabove, **NEUTRAL** from 0.40 to below 0.50, and **FAIL** below 0.40. A supportedagent passes only when every dimension passes. The top-level Tier 3 verdict is:diff --git a/pyproject.toml b/pyproject.tomlindex 6c93450f..53976959 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -41,6 +41,7 @@ dependencies = ["idna>=3.10,<4","jinja2>=3.1","markdown-it-py>=4.2,<5",+ "packaging>=24","pydantic>=2.11.0","pyyaml>=6.0","rich>=13.5.0",diff --git a/src/skillevaluator/constants.py b/src/skillevaluator/constants.pyindex f468ce28..6f543e48 100644--- a/src/skillevaluator/constants.py+++ b/src/skillevaluator/constants.py@@ -156,7 +156,7 @@ BANNED_PACKAGES = []# File extensions to scan-SCANNABLE_EXTENSIONS = {".py", ".sh", ".yaml", ".yml", ".json", ".md", ".txt"}+SCANNABLE_EXTENSIONS = {".py", ".sh", ".yaml", ".yml", ".json", ".md", ".mdc", ".txt"}# Directories to skip at any depth during Tier 1 file-tree walks (scan artifact roots, version snapshots, build caches).SCAN_EXCLUDED_DIRS = frozenset(diff --git a/src/skillevaluator/evaluation/tier3_report.py b/src/skillevaluator/evaluation/tier3_report.pyindex a213cf07..26bb5372 100644--- a/src/skillevaluator/evaluation/tier3_report.py+++ b/src/skillevaluator/evaluation/tier3_report.py@@ -1264,6 +1264,14 @@ def _build_agent(trials = _normalize_trials(info.get("rewards") or [], with_metrics)baseline_trials = _normalize_trials(info.get("rewards_baseline") or [], baseline_metrics)_attach_baseline_pairs(trials, baseline_trials, with_metrics)+ from skillevaluator.tier3.harbor.resource_usage import resource_comparison++ resources = resource_comparison(+ trials,+ baseline_trials,+ with_complete=with_quality_available and info.get("rewards_complete") is True,+ baseline_complete=baseline_quality_available and info.get("rewards_baseline_complete") is True,+ )execution_errors = ([str(error) for error in info.get("execution_errors", [])]@@ -1295,6 +1303,7 @@ def _build_agent("num_trials_baseline": int(info.get("num_trials_baseline", len(baseline_trials)) or 0),"trials": trials,"trials_baseline": baseline_trials,+ "resources": resources,"pass_at_k": {"with_skill": info.get("pass_with_skill") or {},"without_skill": info.get("pass_without_skill") or {},@@ -1919,6 +1928,9 @@ def _normalize_trials(rewards: list[dict[str, Any]], metrics: list[str]) -> list"scores": scores,"overall": _complete_mean([overall_score(item) for item in group]) if group_is_consistent else None,}+ seconds = _finite_float(reward.get("_agent_seconds"))+ if not is_multi_row and seconds is not None and seconds >= 0:+ trial["agent_seconds"] = secondstraj = reward.get("_traj")if not is_multi_row and isinstance(traj, dict):steps = _token_counter(traj.get("steps"))diff --git a/src/skillevaluator/reporting/html.py b/src/skillevaluator/reporting/html.pyindex e2ecdc2c..8f4e31bf 100644--- a/src/skillevaluator/reporting/html.py+++ b/src/skillevaluator/reporting/html.py@@ -1049,6 +1049,9 @@ class HTMLReporter(ReporterBase):"metadata": detail.metadata,})+ coverage = result.metadata.get("dependency_coverage")+ if isinstance(coverage, list):+ result_dict["dependency_coverage"] = coverageoutput.append(result_dict)return outputdiff --git a/src/skillevaluator/reporting/json_reporter.py b/src/skillevaluator/reporting/json_reporter.pyindex e8e5eb30..72b6c397 100644--- a/src/skillevaluator/reporting/json_reporter.py+++ b/src/skillevaluator/reporting/json_reporter.py@@ -219,6 +219,10 @@ class JSONReporter(ReporterBase):}# Quality scores (from QualityScoreValidator)+ coverage = result.metadata.get("dependency_coverage")+ if isinstance(coverage, list):+ data["dependency_coverage"] = coverage+qs = result.metadata.get("quality_scores")if qs:data["quality"] = qsdiff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2index c7a7aed7..3d44675b 100644--- a/src/skillevaluator/reporting/templates/report.html.j2+++ b/src/skillevaluator/reporting/templates/report.html.j2@@ -2658,6 +2658,33 @@</details>{% endif %}+ {% if t3_agents %}+ <details class="t3-details" open>+ <summary>Measured Agent Resources</summary>+ <div class="t3-details-body">+ <p class="t3-section-hint">Efficiency scores measure routing and productive tool use. These separate measurements show observed agent resource use. Delta = with skill minus baseline; negative means less use. Judge and setup time are excluded. Deltas require complete telemetry and matching case counts in both arms.</p>+ <div class="table-scroll"><table class="summary-table">+ <thead><tr><th>Agent</th><th>Measurement</th><th>With skill total</th><th>Baseline total</th><th>Delta</th><th>Measured trials (skill / baseline)</th></tr></thead>+ <tbody>+ {% for name, agent in t3_agents.items() %}+ {% if agent.resources is defined %}+ {% for field, label in [('tokens', 'Prompt + completion tokens (cache included)'), ('agent_seconds', 'Agent execution seconds')] %}+ {% set measured = agent.resources[field] %}+ <tr><td>{{ name }}</td><td>{{ label }}</td>+ <td>{{ '%.2f' | format(measured.with_skill.total) if measured.with_skill.total is not none else 'N/A' }}</td>+ <td>{{ '%.2f' | format(measured.baseline.total) if measured.baseline.total is not none else 'N/A' }}</td>+ <td>{{ '%+.2f' | format(measured.delta) if measured.delta is not none else 'N/A' }}</td>+ <td>{{ measured.with_skill.measured_trials }}/{{ measured.with_skill.trials }} / {{ measured.baseline.measured_trials }}/{{ measured.baseline.trials }}</td>+ </tr>+ {% endfor %}+ {% endif %}+ {% endfor %}+ </tbody>+ </table></div>+ </div>+ </details>+ {% endif %}+{% if tier3.trials %}{% set t3_trial_agent_groups = (tier3.trials or []) | groupby('agent') %}diff --git a/src/skillevaluator/tier3/eval_core/llm_judge.py b/src/skillevaluator/tier3/eval_core/llm_judge.pyindex 3c5b3f9b..d9317e94 100644--- a/src/skillevaluator/tier3/eval_core/llm_judge.py+++ b/src/skillevaluator/tier3/eval_core/llm_judge.py@@ -769,23 +769,35 @@ def judge_accuracy(question: str,ground_truth: str,agent_text: str,+ *,+ skill_identification_applicable: bool = True,**kwargs: Any,) -> dict[str, Any]:- """Run the 5-criterion accuracy judge. Returns ``{"score": float, "reason": str, ...}``."""+ """Judge accuracy; exclude skill identification when no skill is expected."""if not ground_truth:return {"score": 1.0, "reason": "No ground_truth -- skipped"}- prompt = ACCURACY_PROMPT.format(- question=question,- ground_truth=ground_truth,- agent_text=agent_text,- )+ prompt = ACCURACY_PROMPT+ if not skill_identification_applicable:+ prompt = prompt.replace(+ "1. SKILL_IDENTIFIED: Does the response reference or use the correct skill for the task?",+ "1. SKILL_IDENTIFIED: Inapplicable: no skill is expected. Set false; exclude it from the score.",+ ).replace("Compute score = count(true) / 5.", "Compute score = count(true among the other four criteria) / 4.")+ # Adjust the rubric before interpolation so task/evidence text stays intact.+ prompt = prompt.format(question=question, ground_truth=ground_truth, agent_text=agent_text)++ def validate_payload(parsed: Any) -> str | None:+ error = _accuracy_payload_error(parsed)+ if not error and not skill_identification_applicable and not _valid_accuracy_criteria(parsed.get("criteria")):+ return "Accuracy requires complete criterion verdicts when skill identification is inapplicable"+ return error+kwargs.setdefault("response_schema", ACCURACY_JSON_SCHEMA)kwargs.setdefault("schema_name", "accuracy_judgment")parsed, error, _provenance = _call_validated_json_judge(prompt,- _accuracy_payload_error,+ validate_payload,call_public_llm,_extract_json,**kwargs,@@ -798,16 +810,23 @@ def judge_accuracy(criteria_valid = _valid_accuracy_criteria(criteria)score = _finite_score(parsed.get("score"))- if score is None:+ if not skill_identification_applicable:+ assert criteria_valid+ score = sum(value is True for key, value in criteria.items() if key != "SKILL_IDENTIFIED") / 4.0+ elif score is None:assert criteria_validyes_count = sum(1 for v in criteria.values() if v is True)score = yes_count / 5.0- return {+ result = {"score": round(score, 4),"reason": _bounded_judge_text(parsed.get("reason", "")),"criteria": criteria if criteria_valid else {},}+ if not skill_identification_applicable:+ result["excluded_criteria"] = ["SKILL_IDENTIFIED"]+ result["scoring_policy"] = "applicable-accuracy-criteria/v1"+ return result# ---------------------------------------------------------------------------diff --git a/src/skillevaluator/tier3/harbor/report_data.py b/src/skillevaluator/tier3/harbor/report_data.pyindex b81b93c6..01e34999 100644--- a/src/skillevaluator/tier3/harbor/report_data.py+++ b/src/skillevaluator/tier3/harbor/report_data.py@@ -22,6 +22,7 @@ from pathlib import Pathfrom typing import Anyfrom skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS, LEGACY_METRICS, metric_set_for_reward+from skillevaluator.tier3.harbor.resource_usage import agent_execution_secondslogger = logging.getLogger(__name__)@@ -789,6 +790,15 @@ def load_agent_data(continueif not reward.get("entry_id"):reward["entry_id"] = trial_dir.name.split("__", 1)[0] if trial_dir.name else "unknown"+ result_file = trial_dir / "result.json"+ if result_file.exists():+ # Timing is optional telemetry. A malformed/oversized timing+ # source makes resources unavailable, not score evidence truncated.+ trial_result = _load_bounded_json(result_file, [], artifact="trial_result")+ if isinstance(trial_result, dict):+ seconds = agent_execution_seconds(trial_result)+ if seconds is not None:+ reward["_agent_seconds"] = secondstrajectory_file = trial_dir / "trajectory.json"if trajectory_file.exists():trajectory = _load_bounded_json(diff --git a/src/skillevaluator/tier3/harbor/resource_usage.py b/src/skillevaluator/tier3/harbor/resource_usage.pynew file mode 100644index 00000000..2db05cd3--- /dev/null+++ b/src/skillevaluator/tier3/harbor/resource_usage.py@@ -0,0 +1,86 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Measured agent resources, separate from routing/tool-productivity scores."""++from __future__ import annotations++import math+from collections import Counter+from datetime import datetime+from typing import Any+++def agent_execution_seconds(result: dict[str, Any]) -> float | None:+ """Read Harbor's measured agent phase; never substitute total job/verifier time."""+ timing = result.get("agent_execution")+ if not isinstance(timing, dict):+ return None+ try:+ started = datetime.fromisoformat(timing["started_at"])+ finished = datetime.fromisoformat(timing["finished_at"])+ if started.utcoffset() is None or finished.utcoffset() is None:+ return None+ seconds = (finished - started).total_seconds()+ except (KeyError, TypeError, ValueError, OverflowError):+ return None+ return seconds if math.isfinite(seconds) and seconds >= 0 else None+++def resource_comparison(+ with_trials: list[dict[str, Any]],+ baseline_trials: list[dict[str, Any]],+ *,+ with_complete: bool,+ baseline_complete: bool,+) -> dict[str, Any]:+ """Compare complete matching batches; missing counters and baseline remain null."""++ def identity(trials: list[dict[str, Any]]) -> Counter | None:+ ids = [trial.get("entry_id") for trial in trials]+ return Counter(ids) if ids and all(isinstance(value, str) and value for value in ids) else None++ paired = with_complete and baseline_complete and identity(with_trials) is not None+ paired = paired and identity(with_trials) == identity(baseline_trials)++ def measure(trials: list[dict[str, Any]], field: str, complete: bool) -> dict[str, Any]:+ values = []+ for trial in trials:+ value = trial.get("agent_seconds")+ if field == "tokens":+ tokens = trial.get("tokens", {})+ if not isinstance(tokens, dict):+ tokens = {}+ prompt, completion = tokens.get("prompt"), tokens.get("completion")+ value = (+ prompt + completion+ if type(prompt) is int and type(completion) is int and min(prompt, completion) >= 0+ else None+ )+ if type(value) in (int, float) and 0 <= value <= (1 << 53) - 1 and math.isfinite(value):+ values.append(value)+ total = sum(values)+ available = (+ complete and bool(trials) and len(values) == len(trials) and math.isfinite(total) and total <= (1 << 53) - 1+ )+ return {+ "status": "complete" if available else "unavailable",+ "measured_trials": len(values),+ "trials": len(trials),+ "total": total if available else None,+ }++ measurements = {}+ for field in ("tokens", "agent_seconds"):+ with_measure = measure(with_trials, field, with_complete)+ baseline_measure = measure(baseline_trials, field, baseline_complete)+ delta = None+ if paired and with_measure["total"] is not None and baseline_measure["total"] is not None:+ delta = with_measure["total"] - baseline_measure["total"]+ measurements[field] = {"with_skill": with_measure, "baseline": baseline_measure, "delta": delta}+ return {+ "scope": "observed agent-only trial totals; prompt + completion tokens (cache included), excluding judge/setup time",+ "pairing": "matching case counts" if paired else "unavailable",+ "delta_definition": "with skill minus baseline; negative means less observed resource use",+ **measurements,+ }diff --git a/src/skillevaluator/tier3/harbor/templates/eval.py b/src/skillevaluator/tier3/harbor/templates/eval.pyindex 0af5d3a1..ba22e079 100644--- a/src/skillevaluator/tier3/harbor/templates/eval.py+++ b/src/skillevaluator/tier3/harbor/templates/eval.py@@ -8234,10 +8234,10 @@ BEHAVIOR_CHECK_JSON_SCHEMA = {}-def judge_accuracy(question, ground_truth, agent_text):+def judge_accuracy(question, ground_truth, agent_text, *, skill_identification_applicable=True):if not ground_truth:return {"score": 1.0, "reason": "No ground_truth -- skipped"}- prompt = f"""You are an expert evaluator for AI agent responses. Evaluate by checking \+ prompt = """You are an expert evaluator for AI agent responses. Evaluate by checking \each criterion below against the expected answer. For each criterion, determine true (satisfied) or false (not satisfied).1. SKILL_IDENTIFIED: Does the response reference or use the correct skill for the task?@@ -8261,9 +8261,26 @@ EXPECTED ANSWER:SELECTED EVIDENCE (final response + produced artifacts; low-relevance steps may be omitted):{agent_text}"""+ if not skill_identification_applicable:+ prompt = prompt.replace(+ "1. SKILL_IDENTIFIED: Does the response reference or use the correct skill for the task?",+ "1. SKILL_IDENTIFIED: Inapplicable: no skill is expected. Set false; exclude it from the score.",+ ).replace("Compute score = count(true) / 5.", "Compute score = count(true among the other four criteria) / 4.")+ prompt = prompt.replace('"SKILL_IDENTIFIED": true', '"SKILL_IDENTIFIED": false').replace(+ '"score": 0.8', '"score": 1.0'+ )+ # Adjust the rubric before interpolation so task/evidence text stays intact.+ prompt = prompt.format(question=question, ground_truth=ground_truth, agent_text=agent_text)++ def validate_payload(parsed):+ error = _accuracy_payload_error(parsed)+ if not error and not skill_identification_applicable and not _valid_accuracy_criteria(parsed.get("criteria")):+ return "Accuracy requires complete criterion verdicts when skill identification is inapplicable"+ return error+parsed, error, _provenance = _call_validated_json_judge(prompt,- _accuracy_payload_error,+ validate_payload,call_public_llm,extract_json,response_schema=ACCURACY_JSON_SCHEMA,@@ -8277,14 +8294,21 @@ SELECTED EVIDENCE (final response + produced artifacts; low-relevance steps maycriteria_valid = _valid_accuracy_criteria(criteria)score = _finite_score(parsed.get("score"))- if score is None:+ if not skill_identification_applicable:+ assert criteria_valid+ score = sum(value is True for key, value in criteria.items() if key != "SKILL_IDENTIFIED") / 4.0+ elif score is None:assert criteria_validscore = sum(1 for v in criteria.values() if v is True) / 5.0- return {+ result = {"score": round(score, 4),"reason": _bounded_judge_text(parsed.get("reason", "")),"criteria": criteria if criteria_valid else {},}+ if not skill_identification_applicable:+ result["excluded_criteria"] = ["SKILL_IDENTIFIED"]+ result["scoring_policy"] = "applicable-accuracy-criteria/v1"+ return result# ── LLM Judge: Goal Accuracy ─────────────────────────────────────────────────@@ -8775,6 +8799,7 @@ def main():question,ground_truth,bundles["accuracy"]["prompt_evidence"],+ skill_identification_applicable=should_trigger and bool(expected_skill),)acc_score = acc_result["score"]details["accuracy"] = acc_resultdiff --git a/src/skillevaluator/validators/dependencies.py b/src/skillevaluator/validators/dependencies.pyindex 7869c5f7..9dd7d8f9 100644--- a/src/skillevaluator/validators/dependencies.py+++ b/src/skillevaluator/validators/dependencies.py@@ -9,8 +9,16 @@ the PyPI Advisory Database, OSV, and PyUp.io Safety DB.from __future__ import annotations+import json+import tempfile+import tomllibfrom pathlib import Path+from packaging.requirements import InvalidRequirement, Requirement+from packaging.utils import canonicalize_name+from packaging.version import Version++from skillevaluator.utils.secure_fs import SecurePathError, SecureRootfrom skillevaluator.utils.tool_runner import Severity, Tools, cvss_to_severity, parse_json_outputfrom skillevaluator.validators.base import ValidationResult, ValidatorBase@@ -20,7 +28,9 @@ class DependencySecurityValidator(ValidatorBase):Uses pip-audit (primary) and Safety (secondary) to detect knownsecurity issues in Python dependencies from requirements.txt,- pyproject.toml, and setup.py files.+ and static PEP 621 pyproject.toml declarations. Pyproject coverage is+ limited to directly declared exact versions; transitive packages require+ a separately supplied requirements file."""def __init__(self, use_safety: bool = True, fail_on_medium: bool = False):@@ -43,11 +53,30 @@ class DependencySecurityValidator(ValidatorBase):def validate(self, skill_path: Path) -> ValidationResult:"""Run dependency audit on skill(s) at path."""- return self._validate_folder_or_skill(+ coverages = []++ def audit_target(target: Path) -> ValidationResult:+ result = self._validate_single_skill(target)+ coverage = result.metadata.get("dependency_coverage")+ if isinstance(coverage, dict):+ relative_target = target.relative_to(skill_path).as_posix()+ coverages.append({"target": relative_target, **coverage})+ result.add_message(+ f"{relative_target}/pyproject.toml coverage: "+ f"{len(coverage['requested_pins'])} requested pin(s), "+ f"{len(coverage['audited_pins'])} audited pin(s); "+ "direct declarations only, transitive dependencies not audited"+ )+ return result++ result = self._validate_folder_or_skill(skill_path,- self._validate_single_skill,+ audit_target,action_description="Auditing dependencies for",)+ if coverages:+ result.metadata["dependency_coverage"] = coverages+ return resultdef _validate_single_skill(self, skill_path: Path) -> ValidationResult:"""Audit dependencies for a single skill directory."""@@ -73,7 +102,9 @@ class DependencySecurityValidator(ValidatorBase):"""Locate dependency files in skill directory."""return {"requirements": list(skill_path.glob("requirements*.txt")),- "pyproject": skill_path / "pyproject.toml" if (skill_path / "pyproject.toml").exists() else None,+ "pyproject": skill_path / "pyproject.toml"+ if (skill_path / "pyproject.toml").exists() or (skill_path / "pyproject.toml").is_symlink()+ else None,}def _audit_requirements(self, req_file: Path) -> ValidationResult:@@ -89,24 +120,150 @@ class DependencySecurityValidator(ValidatorBase):return resultdef _audit_pyproject(self, pyproject: Path) -> ValidationResult:- """Audit pyproject.toml dependencies via local environment."""+ """Audit declared pins without installing the target or running build code."""result = ValidationResult()- result.add_message(f"Auditing {pyproject.name}")+ result.metadata["dependency_coverage"] = {+ "source": "pyproject.toml",+ "scope": "direct declared pins (all optional groups), scanner-platform markers",+ "declarations": None,+ "requested_pins": [],+ "audited_pins": [],+ "transitive_dependencies": "not audited",+ }+ result.add_message(+ "Auditing pyproject.toml: direct declared pins, including optional groups; no transitive audit"+ )- if not Tools.pip_audit.is_available:- result.add_warning(f"pip-audit not installed. {Tools.pip_audit.get_install_hint()}")+ def incomplete(message: str) -> None:+ result.mark_scan_incomplete("pip-audit")+ result.add_warning(f"pyproject.toml: {message}")++ try:+ with SecureRoot(pyproject.parent) as root:+ raw, _ = root.read_bytes(Path(pyproject.name), 1024 * 1024)+ data = tomllib.loads(raw.decode("utf-8"))+ except (OSError, SecurePathError, UnicodeError, tomllib.TOMLDecodeError):+ incomplete("cannot read a bounded regular TOML file")return result- tool_result = Tools.pip_audit.run(- ["--local", "--format", "json", "--progress-spinner", "off"],- cwd=pyproject.parent,- timeout=180,+ project = data.get("project")+ if not isinstance(project, dict):+ incomplete("no static PEP 621 project metadata; supply pinned requirements.txt")+ return result+ dynamic = project.get("dynamic", [])+ if not isinstance(dynamic, list) or any(name in dynamic for name in ("dependencies", "optional-dependencies")):+ incomplete("dynamic dependency metadata is unsupported; supply pinned requirements.txt")+ tool = data.get("tool", {})+ if isinstance(tool, dict) and "poetry" in tool:+ incomplete("Poetry dependency metadata is not audited; supply pinned requirements.txt")++ groups = {"dependencies": project.get("dependencies", [])}+ optional = project.get("optional-dependencies", {})+ if not isinstance(optional, dict):+ incomplete("optional-dependencies must be a table of requirement lists")+ else:+ groups.update({f"optional group {name}": values for name, values in optional.items()})+ pins: set[str] = set()+ declaration_count = 0+ for group, declarations in groups.items():+ if not isinstance(declarations, list):+ incomplete(f"{group} must be a list")+ continue+ for index, declaration in enumerate(declarations):+ declaration_count += 1+ if declaration_count > 4096:+ incomplete("dependency declaration limit (4096) exceeded")+ return result+ # Never include arbitrary declaration text (e.g. URL credentials) in diagnostics.+ label = f"{group}, declaration {index + 1}"+ try:+ if not isinstance(declaration, str) or "\n" in declaration or "\r" in declaration:+ raise InvalidRequirement("invalid declaration")+ requirement = Requirement(declaration)+ except InvalidRequirement:+ incomplete(f"{label}: invalid requirement")+ continue+ if requirement.url:+ incomplete(f"{label}: URL/path dependencies are unsupported")+ continue+ if requirement.marker:+ # Audit every optional group, evaluating `extra` in its group context.+ extra = group.removeprefix("optional group ") if group.startswith("optional group ") else ""+ if not requirement.marker.evaluate({"extra": extra}):+ result.add_message(f"pyproject.toml: {label} skipped (marker does not match scanner platform)")+ continue+ specifiers = list(requirement.specifier)+ if len(specifiers) != 1 or specifiers[0].operator != "==" or "*" in specifiers[0].version:+ incomplete(f"{label}: an exact version pin is required; supply resolved requirements.txt")+ continue+ pins.add(f"{requirement.name}=={specifiers[0].version}")++ result.metadata["dependency_coverage"].update(+ {"declarations": declaration_count, "requested_pins": sorted(pins)})+ if not pins:+ result.add_message("pyproject.toml: no applicable exact pins to audit")+ return result- if tool_result.error_message:- result.add_warning(tool_result.error_message)- else:- self._process_pip_audit(tool_result.stdout, result, pyproject.name)+ if not Tools.pip_audit.is_available:+ incomplete(f"pip-audit not installed. {Tools.pip_audit.get_install_hint()}")+ return result++ with tempfile.TemporaryDirectory(prefix="skillevaluator-dependencies-") as temporary_dir:+ requirements = Path(temporary_dir) / "requirements.txt"+ requirements.write_text("\n".join(sorted(pins)) + "\n", encoding="utf-8")+ tool_result = Tools.pip_audit.run(+ [+ "-r",+ str(requirements),+ "--no-deps",+ "--disable-pip",+ "--format",+ "json",+ "--progress-spinner",+ "off",+ ],+ timeout=180,+ )++ if tool_result.error_message or tool_result.exit_code not in (0, 1):+ incomplete("pip-audit did not complete successfully")+ return result+ try:+ audit = json.loads(tool_result.stdout)+ dependencies = audit if isinstance(audit, list) else audit["dependencies"]+ if not isinstance(dependencies, list):+ raise ValueError("invalid inventory")+ returned_pins = set()+ for dep in dependencies:+ if not isinstance(dep, dict) or not isinstance(dep.get("vulns"), list):+ raise ValueError("invalid dependency")+ returned_pins.add((canonicalize_name(dep["name"]), Version(dep["version"])))+ if any(+ not isinstance(vuln, dict)+ or not isinstance(vuln.get("id"), str)+ or not isinstance(vuln.get("fix_versions", []), list)+ or any(not isinstance(version, str) for version in vuln.get("fix_versions", []))+ or ("severity" in vuln and not isinstance(vuln["severity"], str))+ or not isinstance(vuln.get("aliases", []), list)+ or any(not isinstance(alias, str) for alias in vuln.get("aliases", []))+ for vuln in dep["vulns"]+ ):+ raise ValueError("invalid advisory")+ expected_pins = {+ (canonicalize_name(requirement.name), Version(next(iter(requirement.specifier)).version))+ for pin in pins+ for requirement in [Requirement(pin)]+ }+ if returned_pins != expected_pins:+ raise ValueError("incomplete inventory")+ if tool_result.exit_code == 1 and not any(dep["vulns"] for dep in dependencies):+ raise ValueError("unexpected finding exit")+ except (KeyError, TypeError, ValueError, RecursionError):+ incomplete("pip-audit returned incomplete or invalid target evidence")+ return result+ self._process_pip_audit(tool_result.stdout, result, pyproject.name)+ result.metadata["dependency_coverage"]["audited_pins"] = sorted(pins)return resultdiff --git a/tests/reporting/test_resource_comparison.py b/tests/reporting/test_resource_comparison.pynew file mode 100644index 00000000..5442a7d2--- /dev/null+++ b/tests/reporting/test_resource_comparison.py@@ -0,0 +1,70 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++import json++import pytest++from skillevaluator.evaluation.tier3_report import agent_eval_result_from_directory+from skillevaluator.reporting import HTMLReporter, JSONReporter+from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS+++def write_arm(run, arm, tokens, seconds):+ directory = run / "opencode" / arm+ directory.mkdir(parents=True)+ scores = dict.fromkeys(DEFAULT_METRICS, 1.0)+ (directory / "summary.json").write_text(+ json.dumps(+ {+ "scores": scores,+ "metrics": list(DEFAULT_METRICS),+ "num_trials": 1,+ "expected_attempts": 1,+ "scored_attempts": 1,+ "execution_status": "succeeded",+ }+ )+ )+ trial = directory / "trials" / "case__1"+ trial.mkdir(parents=True)+ (trial / "reward.json").write_text(json.dumps({"entry_id": "case", **scores}))+ (trial / "trajectory.json").write_text(+ json.dumps(+ {+ "steps": [{}],+ "final_metrics": {+ "total_prompt_tokens": tokens,+ "total_completion_tokens": 2,+ "total_cached_tokens": 3,+ },+ }+ )+ )+ (trial / "result.json").write_text(+ json.dumps(+ {"agent_execution": {"started_at": "2026-10-11T00:00:00Z", "finished_at": f"2026-10-11T00:00:0{seconds}Z"}}+ )+ )+ return trial++[email protected]("missing_tokens", [True, False])+def test_resources_flow_from_harbor_artifacts_to_json_and_html(tmp_path, missing_tokens):+ run = tmp_path / "run"+ write_arm(run, "with-skill", 10, 3)+ baseline = write_arm(run, "without-skill", 20, 5)+ if missing_tokens:+ (baseline / "trajectory.json").unlink()+ result = agent_eval_result_from_directory(tmp_path, run, use_llm_judge=False)+ resources = result.metadata["agent_eval"]["agents"]["opencode"]["resources"]+ assert resources["agent_seconds"]["delta"] == -2+ assert resources["tokens"]["delta"] == (None if missing_tokens else -10)+ assert resources["tokens"]["with_skill"]["total"] == 12+ html = HTMLReporter(target_path=str(tmp_path)).render_all([result])+ assert "Measured Agent Resources" in html+ assert "Agent execution seconds" in html+ assert "-2.00" in html+ assert "N/A" in html if missing_tokens else "-10.00" in html+ output = json.loads(JSONReporter().render_all([result]))+ assert "resources" in json.dumps(output)diff --git a/tests/tier3/test_judge_failure_artifacts.py b/tests/tier3/test_judge_failure_artifacts.pyindex c3e34b6d..c9edcb20 100644--- a/tests/tier3/test_judge_failure_artifacts.py+++ b/tests/tier3/test_judge_failure_artifacts.py@@ -454,7 +454,19 @@ def test_verifier_retries_non_string_judge_text_before_collector_and_report(accuracy_calls.append(prompt)if len(accuracy_calls) == 1:return json.dumps({"score": 1.0, "reason": {"nested": "accuracy"}}), None- return json.dumps({"score": 1.0, "reason": "accuracy recovered"}), None+ return json.dumps(+ {+ "score": 1.0,+ "reason": "accuracy recovered",+ "criteria": {+ "SKILL_IDENTIFIED": False,+ "ACTION_CORRECT": True,+ "FACTUALLY_ACCURATE": True,+ "TASK_ADDRESSED": True,+ "ACTIONABLE": True,+ },+ }+ ), Nonedef goal_call(prompt: str, **_kwargs):goal_calls.append(prompt)@@ -498,6 +510,7 @@ def test_verifier_retries_non_string_judge_text_before_collector_and_report(assert len(accuracy_calls) == 2assert len(goal_calls) == 2assert collected["details"]["accuracy"]["reason"] == "accuracy recovered"+ assert collected["details"]["accuracy"]["excluded_criteria"] == ["SKILL_IDENTIFIED"]assert collected["details"]["goal_accuracy"]["reason"] == "goal recovered"assert all(isinstance(reason, str) for finding in findings for reason in finding["reasons"])diff --git a/tests/tier3/test_judge_result_semantics.py b/tests/tier3/test_judge_result_semantics.pyindex c2a2f1b5..3fda9fcc 100644--- a/tests/tier3/test_judge_result_semantics.py+++ b/tests/tier3/test_judge_result_semantics.py@@ -49,6 +49,46 @@ def _load_template_module() -> ModuleType:eval_template = _load_template_module()[email protected]("module", [llm_judge, eval_template])[email protected]("skill_verdict", [True, False])+def test_negative_accuracy_excludes_skill_identification(module, skill_verdict, monkeypatch):+ criteria = dict.fromkeys(_CRITERIA, True)+ criteria["SKILL_IDENTIFIED"] = skill_verdict+ criteria["ACTION_CORRECT"] = False+ calls = []+ monkeypatch.setattr(+ module,+ "call_public_llm",+ _pair_script([(json.dumps({"criteria": criteria, "score": 0.1, "reason": "case"}), None)], calls),+ )+ result = module.judge_accuracy("question", "answer", "evidence", skill_identification_applicable=False)+ assert result["score"] == 0.75+ assert result["excluded_criteria"] == ["SKILL_IDENTIFIED"]+ assert "other four criteria" in calls[0]++[email protected]("module", [llm_judge, eval_template])+def test_negative_accuracy_requires_criterion_evidence(module, monkeypatch):+ calls = []+ monkeypatch.setattr(+ module, "call_public_llm", _pair_script([('{"score":1,"reason":"missing verdicts"}', None)], calls)+ )+ result = module.judge_accuracy("question", "answer", "evidence", skill_identification_applicable=False)+ assert result["score"] is None+ assert result["status"] == "error"++[email protected]("module", [llm_judge, eval_template])+def test_negative_rubric_does_not_rewrite_supplied_evidence(module, monkeypatch):+ calls = []+ criteria = dict.fromkeys(_CRITERIA, True)+ response = json.dumps({"criteria": criteria, "score": 1.0, "reason": "valid"})+ monkeypatch.setattr(module, "call_public_llm", _pair_script([(response, None)], calls))+ evidence = 'Evidence literal: Compute score = count(true) / 5. "SKILL_IDENTIFIED": true, "score": 0.8'+ module.judge_accuracy("question", "answer", evidence, skill_identification_applicable=False)+ assert evidence in calls[0]++@pytest.fixture(params=["shared", "template"])def judge_module(request: pytest.FixtureRequest) -> ModuleType:return llm_judge if request.param == "shared" else eval_templatediff --git a/tests/tier3/test_resource_usage.py b/tests/tier3/test_resource_usage.pynew file mode 100644index 00000000..607c31b1--- /dev/null+++ b/tests/tier3/test_resource_usage.py@@ -0,0 +1,75 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++import pytest++from skillevaluator.tier3.harbor.resource_usage import agent_execution_seconds, resource_comparison+++def trial(entry="case", tokens=100, seconds=2):+ return {"entry_id": entry, "tokens": {"prompt": tokens, "completion": 10}, "agent_seconds": seconds}+++def test_actual_resources_are_independent_and_signed():+ resources = resource_comparison(+ [trial()], [trial(tokens=200, seconds=1)], with_complete=True, baseline_complete=True+ )+ assert resources["tokens"]["delta"] == -100+ assert resources["agent_seconds"]["delta"] == 1+++ "baseline,complete", [([], False), ([trial(entry="other")], True), ([trial(), trial()], True), ([trial()], False)]+)+def test_incomplete_or_unmatched_baseline_does_not_claim_delta(baseline, complete):+ resources = resource_comparison([trial()], baseline, with_complete=True, baseline_complete=complete)+ assert resources["tokens"]["delta"] is None+ assert resources["agent_seconds"]["delta"] is None+++def test_missing_counters_are_not_zero_and_do_not_hide_other_measurements():+ item = trial()+ item.pop("tokens")+ resources = resource_comparison([item], [trial()], with_complete=True, baseline_complete=True)+ assert resources["tokens"]["with_skill"]["total"] is None+ assert resources["tokens"]["with_skill"]["measured_trials"] == 0+ assert resources["tokens"]["delta"] is None+ assert resources["agent_seconds"]["delta"] == 0+++def test_measured_zero_is_preserved():+ resources = resource_comparison([trial(tokens=0, seconds=0)], [trial()], with_complete=True, baseline_complete=True)+ assert resources["agent_seconds"]["with_skill"]["total"] == 0++[email protected]("value", [True, -1, float("inf"), float("nan"), 10**400, None])+def test_invalid_time_counter_cannot_become_a_measurement(value):+ resources = resource_comparison([trial(seconds=value)], [trial()], with_complete=True, baseline_complete=True)+ assert resources["agent_seconds"]["with_skill"]["total"] is None+ assert resources["agent_seconds"]["delta"] is None+++ "timing",+ [+ {},+ {"started_at": None, "finished_at": None},+ {"started_at": "2026-10-11T00:00:02Z", "finished_at": "2026-10-11T00:00:01Z"},+ {"started_at": "2026-10-11T00:00:00", "finished_at": "2026-10-11T00:00:01"},+ ],+)+def test_unknown_or_invalid_time_is_not_zero(timing):+ assert agent_execution_seconds({"agent_execution": timing}) is None+++def test_agent_phase_excludes_build_and_verifier():+ assert (+ agent_execution_seconds(+ {+ "started_at": "2026-10-11T00:00:00Z",+ "finished_at": "2026-10-11T01:00:00Z",+ "agent_execution": {"started_at": "2026-10-11T00:01:00Z", "finished_at": "2026-10-11T00:01:03Z"},+ }+ )+ == 3+ )diff --git a/tests/validators/test_pii_grouping.py b/tests/validators/test_pii_grouping.pyindex 12bc98ba..52fe4fe8 100644--- a/tests/validators/test_pii_grouping.py+++ b/tests/validators/test_pii_grouping.py@@ -38,6 +38,19 @@ def _email_findings(result):class TestPiiGroupingByValue:+ @pytest.mark.parametrize("as_file", [True, False])+ def test_mdc_rule_and_workflow_pii_is_scanned(self, tmp_path, as_file):+ rule = tmp_path / "instructions.mdc"+ rule.write_text("---\ndescription: Contact rule\n---\nContact [email protected].\n")+ target = rule if as_file else tmp_path+ result = SecurityValidator().validate_pii_only(target)+ assert "[email protected]" in {f.metadata.get("matched_value") for f in _email_findings(result)}++ def test_clean_mdc_has_no_pii(self, tmp_path):+ rule = tmp_path / "clean.mdc"+ rule.write_text("Respond briefly and use the configured contact directory.\n")+ assert SecurityValidator().validate_pii_only(rule).passed+def test_repeated_value_collapses_to_one_finding(self, skill_with_repeated_emails):result = SecurityValidator().validate_pii_only(skill_with_repeated_emails)diff --git a/tests/validators/test_target_dependencies.py b/tests/validators/test_target_dependencies.pynew file mode 100644index 00000000..9235e8d5--- /dev/null+++ b/tests/validators/test_target_dependencies.py@@ -0,0 +1,155 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++import json+from pathlib import Path++import pytest++from skillevaluator.reporting import HTMLReporter, JSONReporter+from skillevaluator.utils.tool_runner import ToolResult, Tools+from skillevaluator.validators.dependencies import DependencySecurityValidator+++def audit(tmp_path, text, monkeypatch, *, exit_code=0, output=None, available=True):+ target = tmp_path / "pyproject.toml"+ target.write_text(text)+ calls = []+ monkeypatch.setattr(type(Tools.pip_audit), "is_available", property(lambda _self: available))++ def run(args, **kwargs):+ assert "--local" not in args+ assert "--no-deps" in args and "--disable-pip" in args+ requirements = Path(args[args.index("-r") + 1]).read_text().splitlines()+ calls.extend(requirements)+ dependencies = [+ {"name": pin.split("==")[0].lower(), "version": pin.split("==")[1], "vulns": []} for pin in requirements+ ]+ return ToolResult(+ True, output if output is not None else json.dumps({"dependencies": dependencies}), "", exit_code+ )++ monkeypatch.setattr(Tools.pip_audit, "run", run)+ return DependencySecurityValidator(use_safety=False)._audit_pyproject(target), calls+++def test_audits_target_pins_and_optional_groups(tmp_path, monkeypatch):+ result, calls = audit(+ tmp_path,+ '[project]\ndependencies=["Requests==2.32.5"]\n[project.optional-dependencies]\nextra=["idna==3.10", "idna==3.10"]',+ monkeypatch,+ )+ assert result.status == "passed"+ assert calls == ["Requests==2.32.5", "idna==3.10"]+ assert result.metadata["dependency_coverage"]["audited_pins"] == calls+ assert result.metadata["dependency_coverage"]["transitive_dependencies"] == "not audited"+++ "declaration", ["requests>=2", "requests==2.*", "requests @ https://example.com/pkg.whl", "--local", "bad\n--local"]+)+def test_unresolved_declarations_are_incomplete(tmp_path, monkeypatch, declaration):+ text = "[project]\ndependencies=" + json.dumps([declaration, "idna==3.10"])+ result, calls = audit(tmp_path, text, monkeypatch)+ assert result.status == "incomplete"+ assert calls == ["idna==3.10"]+ assert declaration not in " ".join(result.warnings)+++ "text",+ [+ "[tool.poetry]\nname='example'",+ "[project]\ndynamic=['dependencies']",+ "[project]\ndependencies='invalid'",+ "[project]\noptional-dependencies=[]",+ "[project\n",+ ],+)+def test_invalid_or_dynamic_metadata_is_incomplete(tmp_path, monkeypatch, text):+ result, calls = audit(tmp_path, text, monkeypatch)+ assert result.status == "incomplete"+ assert not calls+++def test_markers_are_explicitly_scoped(tmp_path, monkeypatch):+ result, calls = audit(tmp_path, "[project]\ndependencies=[\"idna==3.10; python_version < '2'\"]", monkeypatch)+ assert result.status == "passed"+ assert not calls+ assert "skipped" in " ".join(result.messages)+++ "output,exit_code",+ [+ ("oops", 0),+ ('{"dependencies":[]}', 0),+ ('{"dependencies":[]}', 2),+ ('{"dependencies":[{"name":"idna","version":"3.10","vulns":[]}]}', 1),+ ],+)+def test_failed_or_incomplete_inventory_is_incomplete(tmp_path, monkeypatch, output, exit_code):+ result, _ = audit(+ tmp_path, '[project]\ndependencies=["idna==3.10"]', monkeypatch, output=output, exit_code=exit_code+ )+ assert result.status == "incomplete"+ assert result.metadata["dependency_coverage"]["requested_pins"] == ["idna==3.10"]+ assert result.metadata["dependency_coverage"]["audited_pins"] == []+++def test_missing_scanner_is_incomplete(tmp_path, monkeypatch):+ result, calls = audit(tmp_path, '[project]\ndependencies=["idna==3.10"]', monkeypatch, available=False)+ assert result.status == "incomplete"+ assert not calls++[email protected]("fields", [{"aliases": [{"cvss": None}]}, {"severity": []}, {"fix_versions": [None]}])+def test_invalid_advisory_fields_are_incomplete(tmp_path, monkeypatch, fields):+ output = json.dumps(+ {"dependencies": [{"name": "idna", "version": "3.10", "vulns": [{"id": "PYSEC-TEST", **fields}]}]}+ )+ result, _ = audit(tmp_path, '[project]\ndependencies=["idna==3.10"]', monkeypatch, output=output, exit_code=1)+ assert result.status == "incomplete"+ assert result.metadata["dependency_coverage"]["audited_pins"] == []+++def test_vulnerable_target_preserves_findings(tmp_path, monkeypatch):+ output = json.dumps(+ {+ "dependencies": [+ {"name": "requests", "version": "2.19.1", "vulns": [{"id": "PYSEC-TEST", "fix_versions": ["2.32.5"]}]}+ ]+ }+ )+ result, _ = audit(tmp_path, '[project]\ndependencies=["requests==2.19.1"]', monkeypatch, output=output, exit_code=1)+ assert result.status == "failed"+ assert "PYSEC-TEST" in " ".join(result.errors)+++def test_symlink_is_not_read(tmp_path):+ external = tmp_path / "outside.toml"+ external.write_text('[project]\ndependencies=["idna==3.10"]')+ link = tmp_path / "pyproject.toml"+ link.symlink_to(external)+ assert DependencySecurityValidator()._audit_pyproject(link).status == "incomplete"+++def test_folder_target_coverage_survives_report_exports(tmp_path, monkeypatch):+ for name in ("first", "second"):+ skill = tmp_path / name+ skill.mkdir()+ (skill / "SKILL.md").write_text(f"---\nname: {name}\ndescription: Control\n---\n")+ validator = DependencySecurityValidator(use_safety=False)++ def audit_target(target):+ result, _ = audit(target, '[project]\ndependencies=["idna==3.10"]', monkeypatch)+ return result++ monkeypatch.setattr(validator, "_validate_single_skill", audit_target)+ result = validator.validate(tmp_path)+ coverage = result.metadata["dependency_coverage"]+ assert {item["target"] for item in coverage} == {"first", "second"}+ assert all(item["audited_pins"] == ["idna==3.10"] for item in coverage)+ json_data = json.loads(JSONReporter().render_all([result]))+ assert json_data["results"][0]["dependency_coverage"] == coverage+ assert HTMLReporter()._results_to_dict([result])[0]["dependency_coverage"] == coveragediff --git a/uv.lock b/uv.lockindex ca4f830d..252a1e86 100644--- a/uv.lock+++ b/uv.lock@@ -422,10 +422,10 @@ resolution-markers = ["sys_platform == 'linux'",]dependencies = [- { name = "license-expression" },- { name = "packageurl-python" },- { name = "py-serializable" },- { name = "sortedcontainers" },+ { name = "license-expression", marker = "sys_platform == 'linux'" },+ { name = "packageurl-python", marker = "sys_platform == 'linux'" },+ { name = "py-serializable", marker = "sys_platform == 'linux'" },+ { name = "sortedcontainers", marker = "sys_platform == 'linux'" },]sdist = { url = "https://files.pythonhosted.org/packages/66/fc/abaad5482f7b59c9a0a9d8f354ce4ce23346d582a0d85730b559562bbeb4/cyclonedx_python_lib-9.1.0.tar.gz", hash = "sha256:86935f2c88a7b47a529b93c724dbd3e903bc573f6f8bd977628a7ca1b5dadea1", size = 1048735, upload-time = "2025-02-27T17:23:40.367Z" }wheels = [@@ -442,11 +442,11 @@ resolution-markers = ["sys_platform != 'emscripten' and sys_platform != 'linux' and sys_platform != 'win32'",]dependencies = [- { name = "license-expression" },- { name = "packageurl-python" },- { name = "py-serializable" },- { name = "sortedcontainers" },- { name = "typing-extensions", marker = "python_full_version < '3.13'" },+ { name = "license-expression", marker = "sys_platform != 'linux'" },+ { name = "packageurl-python", marker = "sys_platform != 'linux'" },+ { name = "py-serializable", marker = "sys_platform != 'linux'" },+ { name = "sortedcontainers", marker = "sys_platform != 'linux'" },+ { name = "typing-extensions", marker = "python_full_version < '3.13' and sys_platform != 'linux'" },]sdist = { url = "https://files.pythonhosted.org/packages/75/c9/5d0ccdd19bc7d8ab803b90695c1706aa2ea8529685d18e682dc2524d2630/cyclonedx_python_lib-11.11.0.tar.gz", hash = "sha256:4b3194db72b613717f2912447e67ab618c75ff7dcac6c4af3c0e9e1ac617c102", size = 1442983, upload-time = "2026-06-17T11:57:49.055Z" }wheels = [@@ -1830,6 +1830,7 @@ dependencies = [{ name = "idna" },{ name = "jinja2" },{ name = "markdown-it-py" },+ { name = "packaging" },{ name = "pydantic" },{ name = "pyyaml" },{ name = "rich" },@@ -1898,6 +1899,7 @@ requires-dist = [{ name = "markdown-it-py", specifier = ">=4.2,<5" },{ name = "mcp", marker = "extra == 'tier3'", specifier = ">=1.28.1,<2" },{ name = "openai", marker = "extra == 'llm'", specifier = ">=2.21.0" },+ { name = "packaging", specifier = ">=24" },{ name = "pip-audit", marker = "extra == 'security'", specifier = ">=2.10.0" },{ name = "pydantic", specifier = ">=2.11.0" },{ name = "pyjwt", extras = ["crypto"], marker = "extra == 'tier3'", specifier = ">=2.13.0" },