Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(security): make the SkillSpector timeout configurable (#271)
Narendran Raghavan54fff6b61cb6
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex be2f247c..603dc915 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -420,6 +420,15 @@ All notable changes to SkillEvaluator are documented in this file.`SKILLSPECTOR_PROVIDER=nv_build` gets the same mapping. The key still reachesSkillSpector only as `OPENAI_API_KEY`, with `OPENAI_BASE_URL` set to thepublic Build endpoint.+- SkillSpector runs are no longer stopped after a fixed 300 seconds, which ended+ slow `--llm` security scans, such as those on local models, as `INCOMPLETE`+ before SkillSpector's own scan deadline (1,080 seconds in v2.13.3). Each run+ now has a 1,200-second watchdog by default; set+ `SKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDS` to a positive integer to change+ it (values above 86400 are capped). `SKILLSPECTOR_MAX_WORKFLOW_SECONDS` is now+ forwarded to SkillSpector so its own deadline can be raised as well, and the+ security result records the effective `skillspector_timeout_seconds`+ ([#155](https://github.com/NVIDIA/SkillEvaluator/issues/155)).### Securitydiff --git a/docs/environment-variables.mdx b/docs/environment-variables.mdxindex 27bd7418..1d8b6351 100644--- a/docs/environment-variables.mdx+++ b/docs/environment-variables.mdx@@ -98,7 +98,7 @@ If none is set, LLM-backed commands fail with a configuration error. If more tha### SkillSpector bridge-The `--llm` security analysis runs [SkillSpector](https://github.com/NVIDIA/SkillSpector), which has its own provider environment. SkillEvaluator bridges your configured provider into that invocation with an isolated subprocess environment: only the selected provider's settings, the SkillSpector model settings below, and a small set of basic process variables are forwarded, so ambient credentials for other providers never reach SkillSpector. `SKILLSPECTOR_TEMPERATURE` is not forwarded, because current reasoning models reject a custom temperature. The `nv_build` provider is bridged as `SKILLSPECTOR_PROVIDER=nv_build`, so SkillSpector sizes requests from its NVIDIA Build model registry, while the key travels as `OPENAI_API_KEY` with `OPENAI_BASE_URL` pointing at the public NVIDIA Build endpoint.+The `--llm` security analysis runs [SkillSpector](https://github.com/NVIDIA/SkillSpector), which has its own provider environment. SkillEvaluator bridges your configured provider into that invocation with an isolated subprocess environment: only the selected provider's settings, the SkillSpector model and deadline settings below, and a small set of basic process variables are forwarded, so ambient credentials for other providers never reach SkillSpector. `SKILLSPECTOR_TEMPERATURE` is not forwarded, because current reasoning models reject a custom temperature. The `nv_build` provider is bridged as `SKILLSPECTOR_PROVIDER=nv_build`, so SkillSpector sizes requests from its NVIDIA Build model registry, while the key travels as `OPENAI_API_KEY` with `OPENAI_BASE_URL` pointing at the public NVIDIA Build endpoint.| Variable | Default | Effect || --- | --- | --- |@@ -107,6 +107,7 @@ The `--llm` security analysis runs [SkillSpector](https://github.com/NVIDIA/Skil| `SKILLSPECTOR_MODEL_<SLOT>` | unset | Per-analyzer model override, such as `SKILLSPECTOR_MODEL_META_ANALYZER`. Forwarded as set. || `SKILLSPECTOR_MODEL_REGISTRY` | unset | Path to a complete custom SkillSpector model registry, for model IDs the bundled registry does not list. Forwarded as set. || `SKILLSPECTOR_REASONING_EFFORT` | `medium` for GPT-6 models, else unset | Reasoning effort for every SkillSpector analyzer. Your value wins. SkillEvaluator sets it only when every SkillSpector model, including `SKILLSPECTOR_MODEL_<SLOT>` overrides, is a GPT-6 model, so never for Claude. |+| `SKILLSPECTOR_MAX_WORKFLOW_SECONDS` | SkillSpector default (1080 in v2.13.3) | SkillSpector's own deadline for one scan, in seconds. Forwarded as set to both the static and the LLM stage. When it expires, SkillSpector reports partial analysis and the security scan is `INCOMPLETE`. Keep it below `SKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDS`. |## Embeddings@@ -129,6 +130,7 @@ These variables tune the static and security checks. The check-by-check guide is| `SKILLEVALUATOR_BANDIT_PATH` | auto-discovered | Absolute path to the Bandit executable. || `SKILLEVALUATOR_SEMGREP_PATH` | auto-discovered | Absolute path to the Semgrep executable. || `SKILLEVALUATOR_SKILLSPECTOR_PATH` | auto-discovered | Absolute path to the SkillSpector executable. |+| `SKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDS` | `1200` | Watchdog for each SkillSpector run, in seconds. The static stage and, with `--llm`, the LLM stage each get the full value. A run that exceeds it is stopped and the security scan is `INCOMPLETE`. Set a positive integer; values above `86400` are capped, and other values fall back to the default with a warning. Keep it above `SKILLSPECTOR_MAX_WORKFLOW_SECONDS` so SkillSpector's own deadline ends a slow scan first. The effective value is recorded as `skillspector_timeout_seconds` in the security result. || `SKILLEVALUATOR_SCHEMA_ALLOWED_DIRS` | — | Extra allowed skill-root directory names (comma- or whitespace-separated), e.g. `data,fixtures`. Additive only — names extend the built-in allowlist and never replace it. || `SKILLEVALUATOR_SUBMITTER` | auto-detected | Explicit submitter identity for the home-path PII check. It is combined with `GITHUB_ACTOR`, `USER`, `LOGNAME`, `USERNAME`, the OS login name, and the skill's declared author — every identity that resolves is protected. If none resolves, the home-path check is disabled with a warning. || `SKILLEVALUATOR_PREVIOUS_VERSION` | — | The previously published skill version that the default version check compares against. Without this value or `--previous-version`, the check validates the current version's format but does not enforce an increase. |diff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdxindex c3872483..e511bf44 100644--- a/docs/tier1-validation.mdx+++ b/docs/tier1-validation.mdx@@ -411,6 +411,24 @@ Tier 1 checks always gate the exit code inside `validate`. Tier 2 gates when it<Accordion title="SkillSpector reports incomplete analysis for a valid helper script">SkillSpector releases before v2.13.1 can stop inspecting a valid Python or Perl helper early and record a `static_parse_limit`, `obfuscated_instruction_text`, or `runtime_limit` ledger exception. The security check is then `INCOMPLETE`, often with a High **Incomplete referenced artifact analysis (AE1)** finding at each line that names the helper. Check the installed release with `skillspector --version` and upgrade with `uv tool install git+https://github.com/NVIDIA/[email protected]`. If the exception remains after the upgrade, investigate it like any other incomplete scan.</Accordion>+ <Accordion title="SkillSpector times out">+ Each SkillSpector run (the static stage, and the LLM stage with `--llm`)+ is stopped after 1,200 seconds by default, and the security scan is reported+ `INCOMPLETE` with `skillspector timed out after 1200 seconds`. SkillSpector+ also ends a scan at its own deadline (1,080 seconds by default in v2.13.3) and reports partial+ analysis, which is `INCOMPLETE` as well. Slow or local LLM providers can+ need more time. Raise both limits, keeping SkillEvaluator's watchdog higher:++ ```bash+ export SKILLSPECTOR_MAX_WORKFLOW_SECONDS=1800+ export SKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDS=2400+ skillevaluator validate ./my-skill --tiers 1 --llm+ ```++ Each stage gets the full watchdog, so a scan that stalls in both stages can+ take up to twice the value before it is reported. See+ [Environment Variables](environment-variables.mdx#validation-tier-1).+ </Accordion><Accordion title="An LLM-backed stage fails immediately">Configure a supported provider or select a keyless Tier 1 run with`skillevaluator tier1 ./my-skill --no-llm` ordiff --git a/src/skillevaluator/validators/security.py b/src/skillevaluator/validators/security.pyindex 4851d5ba..af7e5cee 100644--- a/src/skillevaluator/validators/security.py+++ b/src/skillevaluator/validators/security.py@@ -60,6 +60,14 @@ from skillevaluator.validators.base import (logger = get_logger(__name__)+# Watchdog for each SkillSpector subprocess. The default stays above+# SkillSpector's own workflow deadline (1,080 seconds in v2.13.3) so that deadline, not a kill,+# normally ends a slow scan. The cap keeps the value inside the range that+# subprocess waits accept on every platform.+_DEFAULT_SKILLSPECTOR_TIMEOUT_SECONDS = 1200+_MAX_SKILLSPECTOR_TIMEOUT_SECONDS = 86400+_SKILLSPECTOR_TIMEOUT_ENV = "SKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDS"+_AUTHOR_IDENTITY_RE = re.compile(r"^\S[^<>\n]* <(?P<email>[^<>@\s]+@[^<>\s]+)>$")_SKILLSPECTOR_POLICY_EXIT_CODES = frozenset({0, 1})_SKILLSPECTOR_STATUSLESS_COMPLETENESS_VERSIONS = {(2, 9, 5), (2, 9, 6)}@@ -168,6 +176,8 @@ _SKILLSPECTOR_PROCESS_ENV_NAMES = frozenset("PATH","PATHEXT","REQUESTS_CA_BUNDLE",+ # SkillSpector's own workflow deadline, raised together with the watchdog.+ "SKILLSPECTOR_MAX_WORKFLOW_SECONDS","SSL_CERT_DIR","SSL_CERT_FILE","SYSTEMROOT",@@ -398,6 +408,35 @@ def _copy_selected_environment(environ: Mapping[str, str], names: Iterable[str])return {name: environ[name] for name in names if environ.get(name, "").strip()}+def _skillspector_timeout_seconds(environ: Mapping[str, str] | None = None) -> int:+ """Return the SkillSpector subprocess watchdog configured by the operator."""+ source = os.environ if environ is None else environ+ raw = source.get(_SKILLSPECTOR_TIMEOUT_ENV, "").strip()+ if not raw:+ return _DEFAULT_SKILLSPECTOR_TIMEOUT_SECONDS+ try:+ timeout: int | None = int(raw)+ except ValueError:+ timeout = None+ if timeout is None or timeout <= 0:+ logger.warning(+ "Ignoring %s=%r: expected a positive integer number of seconds; using %ss",+ _SKILLSPECTOR_TIMEOUT_ENV,+ raw,+ _DEFAULT_SKILLSPECTOR_TIMEOUT_SECONDS,+ )+ return _DEFAULT_SKILLSPECTOR_TIMEOUT_SECONDS+ if timeout > _MAX_SKILLSPECTOR_TIMEOUT_SECONDS:+ logger.warning(+ "Capping %s=%r at %ss",+ _SKILLSPECTOR_TIMEOUT_ENV,+ raw,+ _MAX_SKILLSPECTOR_TIMEOUT_SECONDS,+ )+ return _MAX_SKILLSPECTOR_TIMEOUT_SECONDS+ return timeout++def _skillspector_process_env(environ: Mapping[str, str] | None = None) -> dict[str, str]:source = os.environ if environ is None else environreturn _copy_selected_environment(source, _SKILLSPECTOR_PROCESS_ENV_NAMES)@@ -861,7 +900,10 @@ class SecurityValidator(ValidatorBase):result.metadata["skillspector_provider_id"] = provider_idelse:child_env = _skillspector_process_env()- tool_result = Tools.skillspector.run(args, timeout=300, env=child_env, replace_env=True)+ # Read from the parent environment: the isolated child environment omits it.+ timeout = _skillspector_timeout_seconds()+ result.metadata["skillspector_timeout_seconds"] = timeout+ tool_result = Tools.skillspector.run(args, timeout=timeout, env=child_env, replace_env=True)try:data = parse_json_output(tool_result.stdout)except (RecursionError, ValueError):diff --git a/tests/validators/test_security.py b/tests/validators/test_security.pyindex 2c1736e8..ccd32c3c 100644--- a/tests/validators/test_security.py+++ b/tests/validators/test_security.py@@ -19,7 +19,12 @@ from skillevaluator.reporting import CLIReporter, HTMLReporter, JSONReporter, Mafrom skillevaluator.utils.tool_runner import ToolResult, Toolsfrom skillevaluator.validators.base import Finding, Severity, ValidationResultfrom skillevaluator.validators.schema import SchemaValidator-from skillevaluator.validators.security import SecurityValidator, _skillspector_child_env, _skillspector_llm_env+from skillevaluator.validators.security import (+ SecurityValidator,+ _skillspector_child_env,+ _skillspector_llm_env,+ _skillspector_timeout_seconds,+)_SKILLSPECTOR_2_9_6_NO_LLM_REPORT = (Path(__file__).parents[1] / "fixtures" / "skillspector-2.9.6-no-llm.json"@@ -4343,6 +4348,87 @@ assert cap is not None and get_max_output_tokens(model) <= cap, (cap, get_max_ouassert result.metadata["skillspector_reasoning_effort"] == "medium"assert result.metadata["skillspector_version"] == "2.10.0"+ @pytest.mark.parametrize(+ ("value", "expected", "warned"),+ (+ (None, 1200, False),+ ("", 1200, False),+ (" ", 1200, False),+ ("900", 900, False),+ (" 2400 ", 2400, False),+ ("86400", 86400, False),+ ("86401", 86400, True),+ ("10000000", 86400, True),+ ("0", 1200, True),+ ("-1", 1200, True),+ ("1.5", 1200, True),+ ("not-a-number", 1200, True),+ ),+ )+ def test_skillspector_timeout_configuration(self, value: str | None, expected: int, warned: bool) -> None:+ environ = {} if value is None else {"SKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDS": value}++ with patch("skillevaluator.validators.security.logger") as mock_logger:+ assert _skillspector_timeout_seconds(environ) == expected++ assert mock_logger.warning.called is warned++ def test_skillspector_stages_use_configured_timeout_and_workflow_deadline(+ self,+ monkeypatch,+ sample_skill_dir: Path,+ ) -> None:+ """Both stages get the operator watchdog, and SkillSpector gets its own deadline."""+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai")+ monkeypatch.setenv("OPENAI_API_KEY", "provider-test-key")+ monkeypatch.setenv("SKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDS", "2400")+ monkeypatch.setenv("SKILLSPECTOR_MAX_WORKFLOW_SECONDS", "1800")+ monkeypatch.delenv("SKILLSPECTOR_PROVIDER", raising=False)+ reports = (_skillspector_json_report(), _skillspector_json_report(llm_requested=True, llm_available=True))++ with (+ patch.object(Tools.skillspector, "_path", "/usr/bin/skillspector"),+ patch.object(+ Tools.skillspector,+ "run",+ side_effect=[+ ToolResult(success=True, stdout=json.dumps(report), stderr="", exit_code=0) for report in reports+ ],+ ) as mock_run,+ ):+ result = SecurityValidator(use_llm=True).validate_security_only(sample_skill_dir)++ assert len(mock_run.call_args_list) == 2+ for call in mock_run.call_args_list:+ assert call.kwargs["timeout"] == 2400+ assert call.kwargs["replace_env"] is True+ assert call.kwargs["env"]["SKILLSPECTOR_MAX_WORKFLOW_SECONDS"] == "1800"+ assert "SKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDS" not in call.kwargs["env"]+ assert result.metadata["skillspector_timeout_seconds"] == 2400++ @patch("skillevaluator.validators.security.Tools")+ def test_skillspector_static_stage_defaults_to_1200_second_timeout(+ self,+ mock_tools,+ monkeypatch,+ sample_skill_dir: Path,+ ) -> None:+ monkeypatch.delenv("SKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDS", raising=False)+ monkeypatch.delenv("SKILLSPECTOR_MAX_WORKFLOW_SECONDS", raising=False)+ mock_tools.skillspector.is_available = True+ mock_tools.skillspector.run.return_value = ToolResult(+ success=True,+ stdout=json.dumps(_skillspector_json_report()),+ stderr="",+ exit_code=0,+ )++ result = SecurityValidator(use_llm=False).validate_security_only(sample_skill_dir)++ assert mock_tools.skillspector.run.call_args.kwargs["timeout"] == 1200+ assert "SKILLSPECTOR_MAX_WORKFLOW_SECONDS" not in mock_tools.skillspector.run.call_args.kwargs["env"]+ assert result.metadata["skillspector_timeout_seconds"] == 1200+@staticmethoddef _scan_with_skillspector_reports(skill_dir: Path, reports: list[dict], *, use_llm: bool) -> ValidationResult:with (