Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Add advisory enforcement mode to Tier 1 reusable workflow (#431)
Prashant Bhavsar91ecf1ac83b6
diff --git a/.github/workflows/evaluate-skills.yml b/.github/workflows/evaluate-skills.ymlindex 3cf18519..1fff11ac 100644--- a/.github/workflows/evaluate-skills.yml+++ b/.github/workflows/evaluate-skills.yml@@ -20,6 +20,10 @@ on:description: Minimum Tier 1 quality score from 0 to 100type: numberdefault: 70+ enforcement-mode:+ description: strict fails on non-passing Tier 1 results; advisory uploads evidence without blocking on verdict+ type: string+ default: strictllm:description: Enable additional LLM security, verification, and rubric checkstype: boolean@@ -61,6 +65,16 @@ jobs:env:SCAN_MODE: ${{ inputs.scan-mode == 'auto' && (github.event_name == 'pull_request' && 'changed' || 'all') || inputs.scan-mode }}steps:+ - name: Validate enforcement mode+ env:+ ENFORCEMENT_MODE: ${{ inputs.enforcement-mode }}+ run: |+ python3 -I - <<'PY'+ import os+ mode = os.environ["ENFORCEMENT_MODE"]+ if mode not in {"strict", "advisory"}:+ raise SystemExit("enforcement-mode must be strict or advisory")+ PY- name: Capture the optional hosted-report execution receiptid: provenancecontinue-on-error: true@@ -132,6 +146,7 @@ jobs:docker --config "$RUNNER_TEMP/anonymous-docker-config" pull "$IMAGE"- name: Evaluate selected skillsenv:+ ENFORCEMENT_MODE: ${{ inputs.enforcement-mode }}IMAGE: ${{ inputs.image }}DIFF_BASE_SHA: ${{ github.event.pull_request.base.sha }}DIFF_HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}@@ -160,6 +175,10 @@ jobs:if [ "$LLM_ENABLED" = true ]; thenllm_secret_args=(--env LLM_API_KEY)fi+ enforcement_args=()+ if [ "$ENFORCEMENT_MODE" = advisory ]; then+ enforcement_args=(--enforcement-mode advisory)+ fidocker run --rm --user "$(id -u):$(id -g)" \--read-only --tmpfs /tmp:rw,nosuid,noexec,size=1g \--tmpfs /audit-tmp:rw,nosuid,noexec,mode=1777,size=1g --env TMPDIR=/audit-tmp \@@ -171,7 +190,7 @@ jobs:--env SECURITY_BASELINES --env POLICY_SOURCE_SHA \--env CALLER_WORKFLOW_SHA \--env GITHUB_ACTIONS --env REPORT_IMAGE --env REPORT_URL --env REPORT_RUN_URL --env REPORT_COMMIT \- "$IMAGE"+ "$IMAGE" "${enforcement_args[@]}"- name: Add the result to the workflow summaryif: always()run: |diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9b12b807..20385a5c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -211,6 +211,13 @@ All notable changes to SkillEvaluator are documented in this file.optional, including the legacy alternate field, without changing numeric grades.Unrendered activation diagnostics remain in canonical exports without consumingHTML overview or failure-reason preview budgets.+- The Tier 1 reusable workflow accepts `enforcement-mode: strict|advisory`+ (default `strict`). Opt-in `advisory` keeps complete policy/score failures+ non-blocking while retaining failed verdicts and reports and emitting a+ workflow warning.+ Configuration, scanner completeness, evidence, and runtime failures still+ block. Advisory mode requires supporting workflow and scanner image pins;+ the released 0.8.0 bundle does not accept this input.- Run SkillEvaluator's own `skills/` through the pinned three-tier GitHubcallers, retaining Tier 1 as required and Tier 2/3 as advisory. A separatediff --git a/ci/README.md b/ci/README.mdindex e17554fb..3fea50ff 100644--- a/ci/README.md+++ b/ci/README.md@@ -37,6 +37,7 @@ policy label. Profile names do not control network access or report visibility.| `artifact-name` | Keep `skill-evaluator-tier1` for the optional publisher || `report-base-url` | Optional own Pages URL, only after hosted publication is configured || `scan-workers` | `4`; skills scanned at once, from 1 to 16. `1` scans one skill at a time |+| `enforcement-mode` | `strict` (default) blocks on failed verdicts; `advisory` keeps complete policy/score failures non-blocking but still blocks runtime/evidence/incomplete failures || `security-baselines` | `false`; explicitly apply reviewed per-skill SkillSpector baselines as described below |The runtime discovers exact-case `SKILL.md` files, rejects unsafe paths anddiff --git a/ci/publish_reports.py b/ci/publish_reports.pyindex 22724516..a647b8a6 100644--- a/ci/publish_reports.py+++ b/ci/publish_reports.py@@ -120,6 +120,15 @@ def _scanner_list(value):return isinstance(value, list) and len(value) <= MAX_FILES and all(isinstance(item, str) for item in value)+def advisory_downgraded(summary: dict) -> bool:+ enforcement = summary.get("enforcement")+ return (+ isinstance(enforcement, dict)+ and enforcement.get("mode") == "advisory"+ and enforcement.get("downgraded") is True+ )++def validate_summary(summary: dict) -> None:if not isinstance(summary, dict) or not isinstance(summary.get("skills"), list):raise PublicationError("The report summary lacks its skill list.")@@ -219,6 +228,20 @@ def validate_summary(summary: dict) -> None:)if summary["overall_status"] != expected_status:raise PublicationError("The summary status contradicts its skill results.")+ enforcement = summary.get("enforcement")+ if enforcement is not None:+ if (+ not isinstance(enforcement, dict)+ or enforcement.get("mode") not in {"strict", "advisory"}+ or type(enforcement.get("downgraded")) is not bool+ ):+ raise PublicationError("The summary enforcement metadata is invalid.")+ if enforcement["downgraded"] and not (+ enforcement["mode"] == "advisory"+ and summary["overall_status"] == "failed"+ and summary["overall_passed"] is False+ ):+ raise PublicationError("The summary enforcement metadata contradicts its verdict.")for key in ("total_validators", "total_errors", "total_warnings"):if not _integer(summary.get(key)) or summary[key] != sum(row.get(key, 0) for row in rows):raise PublicationError("The summary aggregates contradict its per-skill result counts.")@@ -951,7 +974,7 @@ def publish(provenance = _validate_workflow_provenance(api, repository, run)artifact = _select_artifact(api, repository, run)summary, reports = read_artifact(api.download_artifact(artifact["id"]))- successful = summary["overall_passed"] or summary["overall_status"] == "skipped"+ successful = summary["overall_passed"] or summary["overall_status"] == "skipped" or advisory_downgraded(summary)if (run["conclusion"] == "success") != successful:raise PublicationError("The workflow conclusion contradicts the report artifact verdict.")if summary["overall_status"] == "skipped" and summary["selection"]["head"] != run["head_sha"]:diff --git a/ci/run_tier1.py b/ci/run_tier1.pyindex 46af0da5..10bf833a 100644--- a/ci/run_tier1.py+++ b/ci/run_tier1.py@@ -286,7 +286,14 @@ def annotation_text(value: str, *, property_value: bool = False) -> str:return escaped.replace(":", "%3A").replace(",", "%2C") if property_value else escaped-def log_result(row: dict, data: dict | None, *, annotations: bool, budget: dict[str, int]) -> None:+def log_result(+ row: dict,+ data: dict | None,+ *,+ annotations: bool,+ budget: dict[str, int],+ enforcement_mode: str = "strict",+) -> None:"""Explain verdicts promptly without replaying arbitrary subprocess logs."""print(f"Skill result: {json.dumps(row['path'])} — {row['overall_status'].upper()}", flush=True)if not data:@@ -303,7 +310,8 @@ def log_result(row: dict, data: dict | None, *, annotations: bool, budget: dict[if result.get("incomplete_scans"):print(f" Incomplete scanners: {json.dumps(result['incomplete_scans'])}", flush=True)for finding in result.get("findings", []):- if finding.get("severity") not in {"critical", "high"} or shown >= 20:+ severity = str(finding.get("severity", "")).lower()+ if severity not in {"critical", "high"} or shown >= 20:continueshown += 1message = str(finding.get("message", "Finding details unavailable."))[:4000]@@ -314,6 +322,7 @@ def log_result(row: dict, data: dict | None, *, annotations: bool, budget: dict[print(f" Suggested fix: {json.dumps(suggestion)}", flush=True)if annotations and result.get("passed") is False and budget["errors"] < 10:budget["errors"] += 1+ annotation_level = "warning" if enforcement_mode == "advisory" else "error"properties = "title=" + annotation_text(str(result.get("validator", "Tier 1")), property_value=True)relative = finding_repository_path(row["path"], finding.get("file_path"))if relative:@@ -321,7 +330,10 @@ def log_result(row: dict, data: dict | None, *, annotations: bool, budget: dict[line = finding.get("line_number")if type(line) is int and line > 0:properties += f",line={line}"- print(f"::error {properties}::{annotation_text(message + ' Suggested fix: ' + suggestion)}", flush=True)+ print(+ f"::{annotation_level} {properties}::{annotation_text(message + ' Suggested fix: ' + suggestion)}",+ flush=True,+ )if result.get("status") in {"incomplete", "failed"}:for message in result.get("legacy", {}).get("errors", [])[:3]:print(f" Scanner error: {json.dumps(str(message)[:4000])}", flush=True)@@ -506,12 +518,26 @@ def evaluate_skill(return fields, data, False-def run(workspace: Path, reports: Path, environ: dict[str, str], *, execute=None, clock=time.monotonic) -> int:- """Run all selected skills; return nonzero on every incomplete or failed run.+def _set_enforcement(summary: dict, *, mode: str, downgraded: bool) -> None:+ summary["enforcement"] = {"mode": mode, "downgraded": downgraded}+++def run(+ workspace: Path,+ reports: Path,+ environ: dict[str, str],+ *,+ execute=None,+ clock=time.monotonic,+ enforcement_mode: str = "strict",+) -> int:+ """Run selected skills; advisory mode only relaxes complete policy verdicts.``execute`` is called like subprocess.run. The default, run_scan, alsoends running scans when the run stops early on an interrupt or error."""+ if enforcement_mode not in {"strict", "advisory"}:+ raise ConfigurationError("enforcement-mode must be strict or advisory.")if reports.is_symlink():raise ConfigurationError("The report directory must not be a symlink.")reports.mkdir(parents=True, exist_ok=True)@@ -612,6 +638,7 @@ def run(workspace: Path, reports: Path, environ: dict[str, str], *, execute=Noneif selection is not None:summary["selection"] = selectionsummary["execution"] = context+ _set_enforcement(summary, mode=enforcement_mode, downgraded=False)_write_summary(reports, summary)print(f"SkillEvaluator configuration failed: {json.dumps(message)}", flush=True)return 1@@ -620,6 +647,7 @@ def run(workspace: Path, reports: Path, environ: dict[str, str], *, execute=Noneif not selected:summary = _summary([])summary.update(overall_status="skipped", overall_passed=False, selection=selection, execution=context)+ _set_enforcement(summary, mode=enforcement_mode, downgraded=False)_write_summary(reports, summary)print("NO SKILLS CHANGED: no skill was evaluated between the selected Git commits.", flush=True)return 0@@ -642,6 +670,7 @@ def run(workspace: Path, reports: Path, environ: dict[str, str], *, execute=Nonedef checkpoint() -> None:summary = _summary(rows)summary.update(selection=selection, execution=context)+ _set_enforcement(summary, mode=enforcement_mode, downgraded=False)_write_json(reports / "summary.json", summary)render_reports(reports, summary, context)@@ -733,6 +762,7 @@ def run(workspace: Path, reports: Path, environ: dict[str, str], *, execute=Noneresults.pop(logged),annotations=environ.get("GITHUB_ACTIONS") == "true",budget=annotation_budget,+ enforcement_mode=enforcement_mode,)elif not final:return@@ -805,8 +835,29 @@ def run(workspace: Path, reports: Path, environ: dict[str, str], *, execute=Nonesummary = _summary(rows)summary["selection"] = selectionsummary["execution"] = context+ downgraded = (+ enforcement_mode == "advisory"+ and summary["overall_status"] == "failed"+ and all(+ row["overall_status"] in {"passed", "failed"}+ and not row["incomplete_scans"]+ and row["exit_code"] in {0, 1}+ and not row.get("error")+ for row in rows+ )+ )+ _set_enforcement(summary, mode=enforcement_mode, downgraded=downgraded)_write_summary(reports, summary)print(f"SkillEvaluator finished: {summary['passed']} of {summary['total']} skills passed.", flush=True)+ # Reports have been validated and rendered. A failed headline can coexist+ # with runtime errors or incomplete scanners, so inspect every skill too.+ if downgraded:+ print(+ "::warning title=Tier 1 (advisory)::Tier 1 did not pass; enforcement-mode=advisory "+ "keeps policy findings non-blocking. See the uploaded report.",+ flush=True,+ )+ return 0return 0 if summary["overall_passed"] else 1@@ -814,9 +865,10 @@ def main() -> int:parser = argparse.ArgumentParser(description=__doc__)parser.add_argument("--workspace", type=Path, default=Path("/workspace"))parser.add_argument("--reports", type=Path, default=Path("/reports"))+ parser.add_argument("--enforcement-mode", choices=("strict", "advisory"), default="strict")args = parser.parse_args()try:- return run(args.workspace, args.reports, dict(os.environ))+ return run(args.workspace, args.reports, dict(os.environ), enforcement_mode=args.enforcement_mode)except (ConfigurationError, OSError):print("SkillEvaluator could not safely initialize the report directory.", flush=True)return 1diff --git a/docs/github-actions.mdx b/docs/github-actions.mdxindex 462eff14..11e4e02e 100644--- a/docs/github-actions.mdx+++ b/docs/github-actions.mdx@@ -50,6 +50,9 @@ Use your own skill folders. Keep `auto` to scan changed skills in PRs;`all` checks the configured collection. If your default branch is not `main`,update the template's push branch.+The 0.8.0 template pins a workflow that does not accept `enforcement-mode`;+leave that input out when using the released template.+### Commit the workflow and open a skill PRMerge the workflow onto your default branch, then open a PR changing a selected@@ -85,6 +88,27 @@ Failed scans retain their available reports too. The HTML separates therequired security or schema check.<AccordionGroup>+ <Accordion title="Use advisory enforcement">+ This unreleased feature requires both a reusable workflow commit and a+ scanner image digest that include `enforcement-mode` support. The 0.8.0+ bundle does not accept this input. Use the reviewed commit SHA of a+ supporting workflow and the published immutable digest of its matching+ image; changing only the input on the 0.8.0 template causes workflow+ validation to fail.++ With compatible pins, omit `enforcement-mode` or set it to `strict` for+ merge gating. Set `enforcement-mode: advisory` for visibility runs:+ complete policy or score failures retain their failed verdict and full+ artifacts, emit a workflow warning, and leave the job successful.+ Configuration errors, incomplete scanners, missing or malformed evidence,+ and runtime failures still fail the job.++ Hosted publication and PR comments run for advisory downgrades recorded as+ `enforcement: {mode: advisory, downgraded: true}`; strict failures still+ require a failing workflow conclusion. An older image without advisory+ support rejects the explicit advisory flag; strict mode continues to work+ with existing images.+ </Accordion><Accordion title="Why is an author email reported as PII?">The required `metadata.author` attribution is accepted. Extra copies incontact fields, body text, or reference documents are checked separately.@@ -227,6 +251,10 @@ or let the [CLI](/skills/skillevaluator/cli-onboarding) handle it.shown in your repository. `min-score` sets the quality threshold; requiredsecurity, schema and other checks remain independent. The reusable scannerruns quality and security checks only.++ Keep `enforcement-mode: strict` for a required check. Advisory checks stay+ green on failed policy or score verdicts, so do not use them as required+ merge gates.</Accordion><Accordion title="The scan finished, but the comment is missing">Confirm setup is on the default branch and the PR is eligible for comments.diff --git a/tests/ci/test_evaluate_workflow.py b/tests/ci/test_evaluate_workflow.pyindex 8c824394..47e32d5e 100644--- a/tests/ci/test_evaluate_workflow.py+++ b/tests/ci/test_evaluate_workflow.py@@ -53,8 +53,57 @@ def test_pr_history_mode_and_attempt_artifact_wiring():assert int(job["timeout-minutes"]) > 50+def test_enforcement_mode_defaults_to_strict():+ workflow = _workflow()+ mode = workflow["on"]["workflow_call"]["inputs"]["enforcement-mode"]+ assert (mode["type"], mode["default"]) == ("string", "strict")+ validate = next(step for step in _job()["steps"] if step.get("name") == "Validate enforcement mode")+ assert validate["env"]["ENFORCEMENT_MODE"] == "${{ inputs.enforcement-mode }}"+ scan = next(step for step in _job()["steps"] if step.get("name") == "Evaluate selected skills")+ assert scan.get("continue-on-error", "false") == "false"++[email protected]("mode,expected", [("strict", 0), ("advisory", 0), ("bogus", 1), ("Advisory", 1), ("", 1)])+def test_enforcement_mode_validation_executes_before_setup(tmp_path, mode, expected):+ step = _job()["steps"][0]+ assert step["name"] == "Validate enforcement mode"+ result = subprocess.run(+ ["bash", "-e", "-o", "pipefail", "-c", step["run"]],+ cwd=tmp_path,+ env={**os.environ, "ENFORCEMENT_MODE": mode},+ capture_output=True,+ )+ assert result.returncode == expected++[email protected]("mode", ["strict", "advisory"])[email protected]("exit_code", [1, 125, 126, 127, 137])+def test_scan_shell_preserves_container_failures(tmp_path, mode, exit_code):+ binary = tmp_path / "docker"+ binary.write_text(f"#!/bin/sh\nexit {exit_code}\n")+ binary.chmod(0o755)+ scan = next(step for step in _job()["steps"] if step.get("name") == "Evaluate selected skills")+ result = subprocess.run(+ ["bash", "-e", "-o", "pipefail", "-c", scan["run"]],+ env={+ **os.environ,+ "PATH": str(tmp_path) + os.pathsep + os.environ["PATH"],+ "RUNNER_TEMP": str(tmp_path),+ "GITHUB_WORKSPACE": str(tmp_path / "checkout"),+ "IMAGE": "ghcr.io/example/scanner@sha256:" + "a" * 64,+ "REPORT_BASE_URL": "",+ "LLM_ENABLED": "false",+ "ENFORCEMENT_MODE": mode,+ },+ capture_output=True,+ text=True,+ )+ assert result.returncode == exit_code++[email protected]("mode", ["strict", "advisory"])@pytest.mark.parametrize("llm", ["true", "false"])-def test_real_workflow_shell_preserves_container_boundary_and_secret_selection(tmp_path, llm):+def test_real_workflow_shell_preserves_container_boundary_and_secret_selection(tmp_path, llm, mode):scan = next(step for step in _job()["steps"] if step.get("name") == "Evaluate selected skills")binary = tmp_path / "docker"# Test double captures argv of the actual workflow shell, not a reimplementation.@@ -72,10 +121,13 @@ def test_real_workflow_shell_preserves_container_boundary_and_secret_selection(t"IMAGE": "ghcr.io/example/scanner@sha256:" + "a" * 64,"REPORT_BASE_URL": "","LLM_ENABLED": llm,+ "ENFORCEMENT_MODE": mode,"LLM_API_KEY": "dummy-do-not-pass-in-static-mode",}subprocess.run(["bash", "-e", "-o", "pipefail", "-c", scan["run"]], env=env, check=True, capture_output=True)args = json.loads(captured.read_text())+ image_index = args.index(env["IMAGE"])+ assert args[image_index + 1 :] == (["--enforcement-mode", "advisory"] if mode == "advisory" else [])passed_environment = [args[index + 1] for index, arg in enumerate(args) if arg == "--env"]assert {"SCAN_MODE",@@ -147,7 +199,7 @@ def test_basic_scan_does_not_require_identity_credentials(tmp_path):workflow = yaml.load((ROOT / ".github/workflows/evaluate-skills.yml").read_text(), Loader=yaml.BaseLoader)job = workflow["jobs"]["tier1"]assert "permissions" not in workflow and "permissions" not in job- step = job["steps"][0]+ step = next(step for step in job["steps"] if step.get("id") == "provenance")assert step["id"] == "provenance"# Run the actual trusted step from a consumer checkout containing shadow modules.for module in ("re", "json", "tempfile"):diff --git a/tests/ci/test_publish_reports.py b/tests/ci/test_publish_reports.pyindex af60d456..4e5c257f 100644--- a/tests/ci/test_publish_reports.py+++ b/tests/ci/test_publish_reports.py@@ -107,6 +107,62 @@ def report():}+def failed_summary(*, advisory_downgraded=False):+ payload = summary()+ payload.update(+ passed=0,+ failed=1,+ incomplete=0,+ overall_passed=False,+ overall_status="failed",+ total_errors=1,+ severity_counts={"critical": 0, "high": 1, "medium": 0, "low": 0},+ )+ payload["skills"][0].update(+ passed=False,+ overall_status="failed",+ exit_code=1,+ total_errors=1,+ severity_counts={"critical": 0, "high": 1, "medium": 0, "low": 0},+ )+ payload["enforcement"] = {"mode": "advisory", "downgraded": advisory_downgraded}+ return payload+++def failed_report():+ payload = report()+ payload.update(+ overall_passed=False,+ overall_status="failed",+ total_errors=1,+ severity_counts={"critical": 0, "high": 1, "medium": 0, "low": 0},+ )+ payload["results"][0].update(+ passed=False,+ status="failed",+ findings=[+ {+ "category": "SCHEMA",+ "severity": "high",+ "check_name": "fixture_failure",+ "message": "Fixture validation failed",+ "file_path": "SKILL.md",+ }+ ],+ summary={+ "files_scanned": 1,+ "checks_performed": 1,+ "errors": 1,+ "warnings": 0,+ "critical_count": 0,+ "high_count": 1,+ "medium_count": 0,+ "low_count": 0,+ },+ )+ return payload++def valid_archive():return archive([@@ -1071,6 +1127,55 @@ def test_failed_workflow_cannot_publish_a_passing_artifact(tmp_path):)+def test_advisory_downgraded_failed_summary_can_publish_with_success_conclusion(tmp_path):+ class AdvisoryAPI(FakeAPI):+ def download_artifact(self, artifact_id):+ if artifact_id == 457:+ return signed_receipt(self.record)+ assert artifact_id == 456+ return archive(+ [+ ("summary.json", failed_summary(advisory_downgraded=True)),+ ("skill-001/skillevaluator-tier1.json", failed_report()),+ ]+ )++ publisher.publish(+ AdvisoryAPI(record={**run_record(), "conclusion": "success"}),+ "owner/consumer",+ 123,+ "https://owner.github.io/consumer",+ tmp_path / "site",+ tmp_path / "output",+ render=fake_render,+ )+++def test_successful_run_with_failed_summary_still_rejected_without_advisory_downgrade_marker(tmp_path):+ class AdvisoryAPI(FakeAPI):+ def download_artifact(self, artifact_id):+ if artifact_id == 457:+ return signed_receipt(self.record)+ assert artifact_id == 456+ return archive(+ [+ ("summary.json", failed_summary(advisory_downgraded=False)),+ ("skill-001/skillevaluator-tier1.json", failed_report()),+ ]+ )++ with pytest.raises(publisher.PublicationError, match="workflow conclusion contradicts"):+ publisher.publish(+ AdvisoryAPI(record={**run_record(), "conclusion": "success"}),+ "owner/consumer",+ 123,+ "https://owner.github.io/consumer",+ tmp_path / "site",+ tmp_path / "output",+ render=fake_render,+ )++@pytest.mark.parametrize("change",[diff --git a/tests/ci/test_run_tier1.py b/tests/ci/test_run_tier1.pyindex 9b419267..37326cf4 100644--- a/tests/ci/test_run_tier1.py+++ b/tests/ci/test_run_tier1.py@@ -103,6 +103,143 @@ def test_selection_deduplicates_overlapping_roots(tmp_path):assert runner.discover_skills(tmp_path, "skills\nskills/a\nskills") == [first, second][email protected]("mode", ["strict", "advisory"])+ "case",+ ["passed", "failed", "incomplete", "mixed", "mixed-error", "missing", "malformed", "crash", "config"],+)+def test_enforcement_only_downgrades_complete_policy_verdicts(tmp_path, capsys, mode, case):+ workspace = tmp_path / "workspace"+ skill(workspace, "skills/a")+ responses = [(0, payload())]+ if case in {"failed", "mixed", "mixed-error"}:+ responses = [(1, payload(False, "failed"))]+ if case in {"incomplete", "mixed"}:+ incomplete = (1, payload(False, "incomplete", ["semgrep"], findings=0))+ if case == "mixed":+ skill(workspace, "skills/b")+ responses.append(incomplete)+ else:+ responses = [incomplete]+ if case == "missing":+ responses = [(1, None)]+ if case == "mixed-error":+ skill(workspace, "skills/b")+ responses.append((1, None))+ if case == "malformed":+ responses = [(1, {"overall_passed": False})]+ if case == "crash":+ responses = [(137, payload(False, "failed"))]+ reports = tmp_path / "reports"+ result = runner.run(+ workspace,+ reports,+ {+ "SKILL_PATHS": "missing" if case == "config" else "skills",+ "LLM_ENABLED": "false",+ "GITHUB_ACTIONS": "true",+ },+ execute=fake_cli(responses, []),+ enforcement_mode=mode,+ )+ expected = 0 if case == "passed" or (mode == "advisory" and case == "failed") else 1+ assert result == expected+ summary = json.loads((reports / "summary.json").read_text())+ assert summary["overall_passed"] is (case == "passed")+ assert summary["enforcement"]["mode"] == mode+ assert summary["enforcement"]["downgraded"] is (mode == "advisory" and case == "failed")+ if case == "failed":+ assert summary["overall_status"] == "failed"+ assert (reports / "skill-001/skillevaluator-tier1.json").is_file()+ assert (reports / "index.html").is_file()+ output = capsys.readouterr().out+ assert ("::warning title=Tier 1 (advisory)::" in output) is (mode == "advisory" and case == "failed")++[email protected]("mode,marker", [("strict", "::error "), ("advisory", "::warning ")])+def test_annotation_level_follows_enforcement_mode(capsys, mode, marker):+ row = {"path": "skills/a", "overall_status": "failed"}+ data = {+ "results": [+ {+ "validator": "Tier 1 fixture validator 1",+ "status": "failed",+ "passed": False,+ "incomplete_scans": [],+ "legacy": {"errors": []},+ "findings": [+ {+ "severity": "HIGH",+ "message": "Fixture validation failed",+ "suggestion": "Fix the fixture",+ "file_path": "SKILL.md",+ "line_number": 1,+ }+ ],+ }+ ]+ }+ runner.log_result(row, data, annotations=True, budget={"errors": 0}, enforcement_mode=mode)+ output = capsys.readouterr().out+ assert marker in output+++def test_advisory_render_failure_still_fails(tmp_path, monkeypatch):+ workspace = tmp_path / "workspace"+ skill(workspace, "skills/a")++ def failed_render(*_args):+ raise OSError("Report disk is full")++ monkeypatch.setattr(runner, "_write_summary", failed_render)+ with pytest.raises(OSError, match="disk is full"):+ runner.run(+ workspace,+ tmp_path / "reports",+ {"SKILL_PATHS": "skills", "LLM_ENABLED": "false"},+ execute=fake_cli([(1, payload(False, "failed"))], []),+ enforcement_mode="advisory",+ )++[email protected]("mode,expected", [("strict", 1), ("advisory", 0)])+def test_cli_enforcement_flag_preserves_failed_evidence(tmp_path, mode, expected):+ workspace = tmp_path / "workspace"+ skill(workspace, "skills/a")+ binary = tmp_path / "skillevaluator"+ binary.write_text(+ f"#!{sys.executable}\nimport pathlib,sys\n"+ "output=pathlib.Path(sys.argv[sys.argv.index('-o')+1])\n"+ f"(output/'skillevaluator-tier1.json').write_text({json.dumps(payload(False, 'failed'))!r})\n"+ "raise SystemExit(1)\n"+ )+ binary.chmod(0o755)+ reports = tmp_path / "reports"+ result = subprocess.run(+ [+ sys.executable,+ str(MODULE_PATH),+ "--workspace",+ str(workspace),+ "--reports",+ str(reports),+ "--enforcement-mode",+ mode,+ ],+ env={+ **os.environ,+ "PATH": str(tmp_path) + os.pathsep + os.environ["PATH"],+ "SKILL_PATHS": "skills",+ "LLM_ENABLED": "false",+ },+ capture_output=True,+ text=True,+ )+ assert result.returncode == expected, result.stderr+ assert json.loads((reports / "summary.json").read_text())["overall_status"] == "failed"+ assert ("::warning title=Tier 1 (advisory)::" in result.stdout) is (mode == "advisory")++def test_contradictory_passing_report_with_incomplete_scanners_has_publishable_error_row(tmp_path):import ci.publish_reports as publisher