Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(reporting): percent-encode % in SARIF artifact URIs (#519)
0z5aa49049c251a5
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ce9658ba..1725341d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -199,6 +199,9 @@ All notable changes to SkillEvaluator are documented in this file.half was never checked. Every clause of an expectation must now name acheckable artifact of its own; one that does not keeps the gate, as a whollyprose expectation already did.+- SARIF `artifactLocation.uri` values percent-encode a `%` in a file name as+ `%25`. A file such as `a%20b.md` no longer shares the URI of `a b.md`, and+ `100%.md` no longer produces an invalid URI.- Accuracy and Behavior judge prompts clarify that explicit final-answer formatsapply to the entire final response, independently of correct values in tooloutput or references. Scoring and thresholds are unchanged; an Accuracy formatdiff --git a/src/skillevaluator/reporting/sarif_reporter.py b/src/skillevaluator/reporting/sarif_reporter.pyindex 06e83d7c..fbd3a2e4 100644--- a/src/skillevaluator/reporting/sarif_reporter.py+++ b/src/skillevaluator/reporting/sarif_reporter.py@@ -122,7 +122,7 @@ def _normalize_artifact_uri(normalized = path.as_posix()else:normalized = path.as_posix()- return quote(normalized, safe="/:@%")+ return quote(normalized, safe="/:@")def _physical_location(diff --git a/tests/reporting/test_reporters.py b/tests/reporting/test_reporters.pyindex fb5d440d..66c76524 100644--- a/tests/reporting/test_reporters.py+++ b/tests/reporting/test_reporters.py@@ -1640,6 +1640,26 @@ class TestSARIFReporter:uri = run["results"][0]["locations"][0]["physicalLocation"]["artifactLocation"]["uri"]assert uri == "skills/demo/SKILL.md"+ def test_physical_location_uri_encodes_percent_signs(self, tmp_path: Path) -> None:+ names = ["notes/a b.md", "notes/a%20b.md", "notes/100%.md"]+ result = ValidationResult(validator_name="QUALITY", validator_description="quality")+ for name in names:+ result.add_finding(+ Finding(+ category="QUALITY",+ severity=Severity.MEDIUM,+ check_name="spacing",+ message="issue",+ file_path=str(tmp_path / name),+ line_number=1,+ )+ )++ sarif = json.loads(SARIFReporter(include_timestamp=False, workspace_root=tmp_path).render_all([result]))+ results = sarif["runs"][0]["results"]+ uris = [item["locations"][0]["physicalLocation"]["artifactLocation"]["uri"] for item in results]+ assert uris == ["notes/a%20b.md", "notes/a%2520b.md", "notes/100%25.md"]+def test_merge_catalog_sarif_combines_into_single_run(self) -> None:child = SARIFReporter(include_timestamp=False)documents = []