Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(security): explain incomplete SkillSpector scans that hit a runtime limit (#294)
Narendran Raghavan0ae3fdbfc019
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b4b0002b..83e17e47 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -802,6 +802,19 @@ All notable changes to SkillEvaluator are documented in this file.killed; a second Ctrl+C skips that wait, and on Windows the task tree isstopped immediately. Progress shows the agents that are stopping and reportsthe run as interrupted instead of failed unexpectedly.+- A Tier 1 security scan that SkillSpector reports as partial no longer fails+ with only "reports incomplete analysis (status 'partial')". The error quotes+ up to three completeness exceptions as `path [phase; reason_code;+ analyzers=...]` and, when there are more, counts the incomplete files per+ reason, for example `(incomplete files by reason: runtime_limit 462,+ obfuscated_instruction_text 4)`. For `runtime_limit`, it explains the fix:+ when many files are cut short, a skill that bundles a large corpus hits the+ same SkillSpector time budget on every rerun and needs that material moved+ out of the skill, while a sub-analysis that stopped at its own budget (named+ `<analyzer>_runtime_limit`, such as the SC8 bytecode walk in SkillSpector+ 2.12.0) is not affected by bundle size and needs a rerun or a newer+ SkillSpector. A report with both kinds of rows gets both explanations. The+ scan still fails closed as `INCOMPLETE`.### Securitydiff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdxindex 1bd23836..b2720329 100644--- a/docs/tier1-validation.mdx+++ b/docs/tier1-validation.mdx@@ -509,6 +509,33 @@ Tier 1 checks always gate the exit code inside `validate`. Tier 2 gates when it<Accordion title="An executable override is rejected">Use an existing, executable absolute path for the relevant `SKILLEVALUATOR_*_PATH` variable.</Accordion>+ <Accordion title="SkillSpector stops at a runtime limit">+ When SkillSpector reports partial analysis, the security scan is+ `INCOMPLETE`. The error quotes up to three of SkillSpector's completeness+ exceptions as `path [phase; reason_code; analyzers=...]`. When there are more,+ it adds how many files each reason affected, for example+ `and 463 more (incomplete files by reason: runtime_limit 462, obfuscated_instruction_text 4)`.++ For `runtime_limit`, the error also says what to do, with one piece of advice+ for each kind of row the report contains:++ - **Scan time budget.** Rows that name regular analyzers, or no analyzer,+ mean SkillSpector ran out of time for the bundle: at its own whole-scan+ deadline (1,080 seconds by default in v2.13.3) or at a shorter stage+ budget such as bundle discovery. A skill that bundles a large corpus, such+ as vendored documents, generated figures or renderer scripts, hits the+ same limit on every rerun. Keep large reference material outside the+ skill and point to it. A plain rerun helps only when a few files are+ affected on a busy machine. If SkillEvaluator's own SkillSpector timeout+ stops the run before SkillSpector's deadline, the error is+ `skillspector timed out after N seconds` instead, with no list of files.+ - **Sub-analysis budget.** Analyzers named `<analyzer>_runtime_limit`, such as+ `static_patterns_supply_chain_bytecode_runtime_limit`, stopped at a time+ budget of their own, so bundle size does not matter. Releases before+ v2.13.0 charge the SC8 bytecode walk for time spent waiting behind other+ analyzers, so it can stop even on a skill with no bytecode. Rerun the scan,+ and update SkillSpector if the limit repeats.+ </Accordion></AccordionGroup>## Next stepsdiff --git a/src/skillevaluator/validators/security.py b/src/skillevaluator/validators/security.pyindex 14b42c80..20e6f783 100644--- a/src/skillevaluator/validators/security.py+++ b/src/skillevaluator/validators/security.py@@ -186,6 +186,9 @@ _SKILLSPECTOR_SEVERITY_POINTS = {"CRITICAL": 50, "HIGH": 25, "MEDIUM": 10, "LOW"_SKILLSPECTOR_EXECUTABLE_MULTIPLIER = 1.3_SKILLSPECTOR_RISK_SCORE_FLOORS_BY_RULE_ID = {"SC8": 51}_SKILLSPECTOR_DIMINISHING_WEIGHTS = (1.0, 0.5, 0.25)+# Ledger exceptions quoted in an incomplete-analysis error, and the length of each quoted field.+_SKILLSPECTOR_INCOMPLETE_EXAMPLES = 3+_SKILLSPECTOR_REPORT_TEXT_MAX_CHARS = 200_SKILLSPECTOR_SCAN_EXCLUDED_DIRS = SCAN_EXCLUDED_DIRS - {"__pycache__"}_SKILLSPECTOR_PROVIDER_MAP = {"anthropic": "anthropic",@@ -914,6 +917,109 @@ def _skillspector_scoring_match_identity(return ("finding", finding) if finding else ("row", index)+def _skillspector_report_text(value: object) -> str | None:+ """Return untrusted report text as one bounded line, or ``None`` when it is not text."""+ if not isinstance(value, str):+ return None+ # Control, format and separator characters (newlines included) become spaces.+ text = "".join(+ " " if unicodedata.category(character)[0] in {"C", "Z"} else character+ for character in value[: _SKILLSPECTOR_REPORT_TEXT_MAX_CHARS + 1]+ ).strip()+ if len(text) > _SKILLSPECTOR_REPORT_TEXT_MAX_CHARS:+ text = f"{text[: _SKILLSPECTOR_REPORT_TEXT_MAX_CHARS - 3]}..."+ return text or None+++def _skillspector_ledger_exception_label(exception: Mapping[str, object]) -> str:+ """Describe one ledger exception as ``path[:line] [phase; reason; analyzers=...]``."""+ location = _skillspector_report_text(exception.get("path")) or "unknown"+ start_line = exception.get("start_line")+ if isinstance(start_line, int) and not isinstance(start_line, bool):+ location = f"{location}:{start_line}"+ context = [text for key in ("phase", "reason_code") if (text := _skillspector_report_text(exception.get(key)))]+ analyzers = exception.get("analyzers")+ if isinstance(analyzers, list) and (+ analyzer_text := _skillspector_report_text(",".join(name for name in analyzers if isinstance(name, str)))+ ):+ context.append(f"analyzers={analyzer_text}")+ return f"{location} [{'; '.join(context)}]" if context else location+++def _skillspector_incomplete_files_by_reason(ledger_exceptions: Iterable[Mapping[str, object]]) -> str:+ """Count distinct files per ledger reason code, most common first."""+ paths_by_reason: dict[str, set[str]] = {}+ for exception in ledger_exceptions:+ reason = _skillspector_report_text(exception.get("reason_code")) or "unknown"+ path = exception.get("path")+ paths_by_reason.setdefault(reason, set()).add(path if isinstance(path, str) else "")+ counts = Counter({reason: len(paths) for reason, paths in paths_by_reason.items()})+ return ", ".join(f"{reason} {count}" for reason, count in counts.most_common())+++def _skillspector_runtime_limit_hint(ledger_exceptions: Iterable[Mapping[str, object]]) -> str | None:+ """Explain how to clear ``runtime_limit`` ledger exceptions, or ``None`` when there are none."""+ scan_limited: set[str] = set()+ budget_limited: set[str] = set()+ for exception in ledger_exceptions:+ if exception.get("reason_code") != "runtime_limit":+ continue+ path = exception.get("path")+ analyzers = exception.get("analyzers")+ # A sub-analysis that stops at its own time budget is reported as+ # "<analyzer>_runtime_limit" (for example the SC8 bytecode walk). That+ # budget is independent of the whole-scan deadline.+ if (+ isinstance(analyzers, list)+ and analyzers+ and all(isinstance(name, str) and name.endswith("_runtime_limit") for name in analyzers)+ ):+ budget_limited.add(path if isinstance(path, str) else "")+ else:+ scan_limited.add(path if isinstance(path, str) else "")+ hints: list[str] = []+ if scan_limited:+ # These rows come from SkillSpector's whole-scan deadline or from a shorter+ # stage budget such as bundle discovery, and the report does not say which,+ # so the hint names no specific limit.+ hints.append(+ f"SkillSpector reached a runtime limit before it finished {len(scan_limited)} file(s). "+ "When many files are affected, the skill most likely bundles more content than SkillSpector can "+ "inspect within its time budgets, so a skill that bundles a large corpus (vendored documents, "+ "generated figures, renderer scripts) hits the same limit on every rerun; keep large reference "+ "material outside the skill and point to it. When only a few files are affected on a busy machine, "+ "a plain rerun may complete."+ )+ if budget_limited:+ # Reported alongside the scan-budget advice too, so every runtime_limit file+ # counted by reason is covered by one of the two explanations.+ hints.append(+ f"A SkillSpector sub-analysis stopped at its own time budget on {len(budget_limited)} file(s). "+ "That budget is separate from the whole-scan limit, so reducing bundled content does not change it. "+ "Rerun the scan; if the limit repeats, update SkillSpector or report the completeness exceptions "+ "to the SkillSpector project."+ )+ return " ".join(hints) or None+++def _skillspector_incomplete_analysis_error(status: str, ledger_exceptions: list[dict]) -> str:+ """Describe a partial SkillSpector analysis with example gaps and, for runtime limits, a remedy."""+ message = f"skillspector JSON field 'analysis_completeness' reports incomplete analysis (status '{status}')"+ if ledger_exceptions:+ shown = ledger_exceptions[:_SKILLSPECTOR_INCOMPLETE_EXAMPLES]+ message = f"{message}: {'; '.join(_skillspector_ledger_exception_label(row) for row in shown)}"+ remaining = len(ledger_exceptions) - len(shown)+ if remaining:+ # A few examples cannot show why hundreds of files were cut short.+ message = (+ f"{message}; and {remaining} more "+ f"(incomplete files by reason: {_skillspector_incomplete_files_by_reason(ledger_exceptions)})"+ )+ message = f"{message}; security scan did not complete"+ hint = _skillspector_runtime_limit_hint(ledger_exceptions)+ return f"{message}. {hint}" if hint else message++# SkillSpector's static pattern analyzers read text, so each records a referenced# PNG or JPEG image as a nonfatal, whole-file limitation and the analysis becomes# ``partial``; see :func:`_skillspector_passive_raster_note`.@@ -2270,8 +2376,7 @@ class SecurityValidator(ValidatorBase):result.add_warning(raster_note)else:result.add_error(- "skillspector JSON field 'analysis_completeness' reports incomplete analysis "- f"(status '{completeness_status}'); security scan did not complete"+ _skillspector_incomplete_analysis_error(completeness_status, ledger_exceptions))result.mark_scan_incomplete(stage_name)diff --git a/tests/validators/test_security.py b/tests/validators/test_security.pyindex 8032e0be..12fbc6d6 100644--- a/tests/validators/test_security.py+++ b/tests/validators/test_security.py@@ -27,6 +27,7 @@ from skillevaluator.validators.security import (_is_jpeg,_is_png,_skillspector_child_env,+ _skillspector_incomplete_analysis_error,_skillspector_llm_env,_skillspector_timeout_seconds,)@@ -295,6 +296,74 @@ def _skillspector_documentation_only_report() -> dict:return payload+_REGULAR_ANALYZERS = ("static_patterns_prompt_injection", "static_patterns_tool_misuse")+_SC8_BUDGET_ANALYZER = "static_patterns_supply_chain_bytecode_runtime_limit"+++def _ledger_exception(+ path: str,+ reason_code: str = "runtime_limit",+ *,+ analyzers: tuple[str, ...] = _REGULAR_ANALYZERS,+ phase: str = "static",+) -> dict:+ """Return one non-fatal SkillSpector ledger exception."""+ return {+ "outcome": "partial",+ "phase": phase,+ "reason_code": reason_code,+ "message": "Inspection reached its configured runtime limit.",+ "path": path,+ "start_line": None,+ "end_line": None,+ "fatal": False,+ "analyzers": list(analyzers),+ }+++def _skillspector_partial_report(ledger_exceptions: list[dict]) -> dict:+ """Return a valid partial report whose only gaps are *ledger_exceptions*."""+ payload = _skillspector_json_report()+ payload["risk_assessment"]["recommendation"] = "CAUTION"+ payload["analysis_completeness"].update(+ {"is_complete": False, "status": "partial", "ledger_exceptions": ledger_exceptions}+ )+ return payload+++def _skillspector_partially_inspected_report(ledger_exceptions: list[dict], *, inspected: tuple[str, ...]) -> dict:+ """Return a partial report whose inventory, counters and analyzer accounting match *ledger_exceptions*.++ Files in *inspected* are fully inspected. Every ledger path is a partially inspected file, and every+ analyzer a ledger row names is degraded on exactly those files.+ """+ payload = _skillspector_partial_report(ledger_exceptions)+ partial_paths = list(dict.fromkeys(row["path"] for row in ledger_exceptions))+ payload["components"] = [{"path": path, "executable": False} for path in (*inspected, *partial_paths)]+ _set_universal_analyzer_work(payload)+ completeness = payload["analysis_completeness"]+ total = len(payload["components"])+ completeness.update(+ {+ "total_components": total,+ "scanned_components": len(inspected),+ "fully_inspected_files": len(inspected),+ "partially_inspected_files": len(partial_paths),+ "coverage_percent": round(len(inspected) / total * 100, 1),+ }+ )+ paths_by_analyzer: dict[str, set[str]] = {}+ for row in ledger_exceptions:+ for analyzer_id in row["analyzers"]:+ paths_by_analyzer.setdefault(analyzer_id, set()).add(row["path"])+ assert set(paths_by_analyzer) <= _SKILLSPECTOR_UNIVERSAL_ANALYZERS+ for status in completeness["analyzer_statuses"]:+ if partial := len(paths_by_analyzer.get(status["analyzer_id"], ())):+ status.update({"status": "degraded", "completed": total - partial, "partial": partial})+ completeness["limitations"].append(f"Analyzer {status['analyzer_id']} status: degraded.")+ return payload++def _user_facing_reports(result: ValidationResult) -> list[str]:"""Render every Tier 1 user-facing report surface for redaction checks."""return [@@ -2338,6 +2407,137 @@ print(json.dumps([finding.metadata["matched_value"] for finding in result.findinassert result.status == "incomplete"assert any("risk_assessment.recommendation" in error for error in result.errors)+ @patch("skillevaluator.validators.security.Tools")+ def test_skillspector_partial_scan_names_incomplete_files_by_reason(+ self,+ mock_tools,+ sample_skill_dir: Path,+ ) -> None:+ """A truncated incomplete-analysis error still says why every file was cut short."""+ ledger = [_ledger_exception(f"references/sect-{index}.html") for index in range(6)]+ # A second analyzer group on the same file must not count the file twice.+ ledger.append(_ledger_exception("references/sect-0.html", analyzers=("static_yara",)))+ ledger.append(_ledger_exception("references/base.css", "obfuscated_instruction_text"))+ payload = _skillspector_partially_inspected_report(ledger, inspected=("SKILL.md",))++ result = _validate_skillspector_payload(mock_tools, sample_skill_dir, payload)++ assert result.status == "incomplete"+ assert len(result.errors) == 1+ analyzers = ",".join(_REGULAR_ANALYZERS)+ assert result.errors[0].startswith(+ "skillspector JSON field 'analysis_completeness' reports incomplete analysis (status 'partial'): "+ f"references/sect-0.html [static; runtime_limit; analyzers={analyzers}]; "+ f"references/sect-1.html [static; runtime_limit; analyzers={analyzers}]; "+ f"references/sect-2.html [static; runtime_limit; analyzers={analyzers}]; "+ "and 5 more (incomplete files by reason: runtime_limit 6, obfuscated_instruction_text 1); "+ "security scan did not complete. "+ "SkillSpector reached a runtime limit before it finished 6 file(s). When many files are affected, "+ "the skill most likely bundles more content than SkillSpector can inspect within its time budgets"+ )+ assert "keep large reference material outside the skill" in result.errors[0]+ assert "a plain rerun may complete" in result.errors[0]++ @patch("skillevaluator.validators.security.Tools")+ def test_skillspector_sub_analysis_time_budget_does_not_blame_bundle_size(+ self,+ mock_tools,+ sample_skill_dir: Path,+ ) -> None:+ """SkillSpector 2.12.0's SC8 bytecode walk can stop at its own budget on a skill without bytecode."""+ payload = _skillspector_partial_report([_ledger_exception("references", analyzers=(_SC8_BUDGET_ANALYZER,))])+ supply_chain = next(+ status+ for status in payload["analysis_completeness"]["analyzer_statuses"]+ if status["analyzer_id"] == "static_patterns_supply_chain"+ )+ supply_chain.update({"status": "degraded", "planned_work": 1, "completed": 0, "partial": 1})+ payload["analysis_completeness"]["limitations"] = ["Analyzer static_patterns_supply_chain status: degraded."]++ result = _validate_skillspector_payload(mock_tools, sample_skill_dir, payload)++ assert result.status == "incomplete"+ assert result.errors == [+ "skillspector JSON field 'analysis_completeness' reports incomplete analysis (status 'partial'): "+ f"references [static; runtime_limit; analyzers={_SC8_BUDGET_ANALYZER}]; "+ "security scan did not complete. "+ "A SkillSpector sub-analysis stopped at its own time budget on 1 file(s). That budget is separate "+ "from the whole-scan limit, so reducing bundled content does not change it. Rerun the scan; if the "+ "limit repeats, update SkillSpector or report the completeness exceptions to the SkillSpector project."+ ]++ def test_skillspector_runtime_limit_hint_counts_only_scan_budget_files(self) -> None:+ ledger = [+ _ledger_exception("SKILL.md", analyzers=(_SC8_BUDGET_ANALYZER,)),+ *(_ledger_exception(f"references/sect-{index}.html") for index in range(3)),+ # A stage budget such as bundle discovery names no analyzer; bundle size still matters.+ _ledger_exception("references", analyzers=(), phase="discovery"),+ ]++ error = _skillspector_incomplete_analysis_error("partial", ledger)++ assert "SkillSpector reached a runtime limit before it finished 4 file(s)." in error+ assert "keep large reference material outside the skill" in error+ # The bytecode-budget file still gets its own rerun or update advice.+ assert error.endswith(+ "a plain rerun may complete. A SkillSpector sub-analysis stopped at its own time budget on 1 file(s). "+ "That budget is separate from the whole-scan limit, so reducing bundled content does not change it. "+ "Rerun the scan; if the limit repeats, update SkillSpector or report the completeness exceptions to "+ "the SkillSpector project."+ )+ assert error.count("runtime limit before it finished") == 1+ # The report does not say which budget ran out, so the hint names no specific deadline.+ assert "seconds" not in error++ def test_skillspector_runtime_limit_hint_covers_every_runtime_limit_file_in_the_count(self) -> None:+ """With both kinds of runtime_limit rows, the two hints add up to the by-reason file count."""+ ledger = [+ *(_ledger_exception(f"references/sect-{index}.html") for index in range(3)),+ _ledger_exception("scripts/build.py", analyzers=(_SC8_BUDGET_ANALYZER,)),+ ]++ error = _skillspector_incomplete_analysis_error("partial", ledger)++ assert "and 1 more (incomplete files by reason: runtime_limit 4);" in error+ assert "SkillSpector reached a runtime limit before it finished 3 file(s)." in error+ assert "A SkillSpector sub-analysis stopped at its own time budget on 1 file(s)." in error++ def test_skillspector_incomplete_analysis_error_without_runtime_limit_has_no_hint(self) -> None:+ ledger = [+ {**_ledger_exception("scripts/render.py", "static_parse_limit"), "start_line": 12},+ _ledger_exception(+ "references/guide.md", "reference_unresolved", analyzers=(), phase="reference_resolution"+ ),+ ]++ assert _skillspector_incomplete_analysis_error("partial", ledger) == (+ "skillspector JSON field 'analysis_completeness' reports incomplete analysis (status 'partial'): "+ f"scripts/render.py:12 [static; static_parse_limit; analyzers={','.join(_REGULAR_ANALYZERS)}]; "+ "references/guide.md [reference_resolution; reference_unresolved]; security scan did not complete"+ )+ # Without ledger exceptions (limitations or counters only) the error is unchanged.+ assert _skillspector_incomplete_analysis_error("partial", []) == (+ "skillspector JSON field 'analysis_completeness' reports incomplete analysis (status 'partial'); "+ "security scan did not complete"+ )++ def test_skillspector_incomplete_analysis_error_bounds_untrusted_report_text(self) -> None:+ ledger = [+ {+ "fatal": False,+ "reason_code": "runtime_limit\n- injected",+ "path": "docs/a\nb.md",+ "analyzers": ["x" * 500, 7],+ },+ {"fatal": False, "reason_code": None, "path": None, "start_line": True},+ ]++ error = _skillspector_incomplete_analysis_error("partial", ledger)++ assert "\n" not in error+ assert f"docs/a b.md [runtime_limit - injected; analyzers={'x' * 197}...]; unknown; " in error+ assert "x" * 198 not in error+@pytest.mark.parametrize(("failure_mode", "expected_error"),(