Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): write plain progress heartbeats once a minute (#274)
Narendran Raghavanaba2fa137df8
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c0759702..63ea6bfe 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -463,6 +463,13 @@ All notable changes to SkillEvaluator are documented in this file.`SKILLSPECTOR_PROVIDER=nv_build` gets the same mapping. The key still reachesSkillSpector only as `OPENAI_API_KEY`, with `OPENAI_BASE_URL` set to thepublic Build endpoint.+- Tier 3 plain progress output, which `--progress auto` selects for CI and+ redirected logs, now repeats its `still running` heartbeat every 60 seconds+ instead of every second, so long runs no longer add thousands of identical+ lines per hour. The interactive Rich view still refreshes every second. A+ new `--progress-interval SECONDS` option on `tier3 evaluate` and `tier3 PATH`+ sets the interval explicitly for either view (0.05 to 86400 seconds;+ non-finite values are rejected).- SkillSpector runs are no longer stopped after a fixed 300 seconds, which endedslow `--llm` security scans, such as those on local models, as `INCOMPLETE`before SkillSpector's own scan deadline (1,080 seconds in v2.13.3). Each rundiff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdxindex e57b134e..b8e2a628 100644--- a/docs/cli-reference.mdx+++ b/docs/cli-reference.mdx@@ -253,7 +253,7 @@ Embedding + LLM dedup; on by default, skips gracefully without a provider key.### Tier 3 · Live Agent Evaluation<Note>-The following flags are forwarded to the live-eval engine when Tier 3 is enabled, which is the default for skills. They do not re-enable a stage excluded by `--tiers` or `--no-tier3`. The Dockerfile-, workspace-, resource-, and progress-tuning flags (`--custom-dockerfile-mode`, `--skill-workspace-mode`, the `--override-*` flags, and `--progress`) are not forwardable — use the standalone [tier3 evaluate](#tier3-evaluate) command for those.+The following flags are forwarded to the live-eval engine when Tier 3 is enabled, which is the default for skills. They do not re-enable a stage excluded by `--tiers` or `--no-tier3`. The Dockerfile-, workspace-, resource-, and progress-tuning flags (`--custom-dockerfile-mode`, `--skill-workspace-mode`, the `--override-*` flags, `--progress`, and `--progress-interval`) are not forwardable — use the standalone [tier3 evaluate](#tier3-evaluate) command for those.</Note>| Flag | Default | Effect |@@ -458,6 +458,7 @@ Use `skillevaluator tier3 PATH` for the complete workflow with automatic missing| `--evaluated-source-revision TEXT` | unset | Immutable revision of the evaluated source: a full Git object id, or a `sha256`/`sha384`/`sha512` digest. A short prefix is refused. || `--evaluator-container-revision TEXT` | unset | Digest-pinned evaluator image reference recorded beside the evaluated source. || `--progress [auto\|rich\|plain\|off]` | `auto` | Progress display for the run: `rich` live view, `plain` line output, or `off`. |+| `--progress-interval SECONDS` | mode default | Seconds between progress heartbeats, from `0.05` to `86400`: how often the `rich` view refreshes elapsed times and how often `plain` output repeats a `still running` line for active stages. Defaults to 1 second for `rich` and 60 seconds for `plain`, including `auto` with redirected or CI output. |Without a `--model`/`--agent-model` override, agent execution uses`gpt-6.1-sol` for OpenAI (OpenCode: `openai/gpt-5.6-sol` unlessdiff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex 416a97a6..d1b9e165 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -563,6 +563,7 @@ The high-signal flags, with defaults:| `--stop-on-pass` | off | Stop a case's remaining attempts once one passes || `--agent-runtime-preflight` | off | Add one agent-only execution of the first staged task per agent before the full A/B matrix; `--no-agent-runtime-preflight` overrides a config value that enables it || `--progress` | `auto` | Progress rendering: `auto`, `rich` (live TTY view with secret redaction), `plain`, or `off` |+| `--progress-interval` | mode default | Seconds between progress heartbeats: 1 for `rich`, 60 for `plain` lines in CI and redirected output || `--autopilot` | off | When no evaluation source exists, generate exactly one eval case — with the configured provider, falling back to a deterministic keyless template — never overwriting an existing source || `--grading-mode` | `default` | `default`, `default_plus_custom`, or `custom_only` — see [Custom Graders & Tasks](custom-graders.mdx) || `--results-dir` | `evals/results`, or `SKILLEVALUATOR_RESULTS_DIR` when set | Write results under an external root instead of the skill directory; custom in-skill roots are rejected |diff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.pyindex a260b412..82128eed 100644--- a/src/skillevaluator/cli.py+++ b/src/skillevaluator/cli.py@@ -135,6 +135,19 @@ def _validate_similarity_threshold(_ctx: click.Context, _param: click.Parameter,return value+def _validate_progress_interval(_ctx: click.Context, _param: click.Parameter, value: float | None) -> float | None:+ if value is None:+ return None+ from skillevaluator.tier3.harbor.progress import MAX_REFRESH_INTERVAL_SECONDS, MIN_REFRESH_INTERVAL_SECONDS++ # NaN passes every range comparison, so check finiteness explicitly.+ if not math.isfinite(value) or not MIN_REFRESH_INTERVAL_SECONDS <= value <= MAX_REFRESH_INTERVAL_SECONDS:+ raise click.BadParameter(+ f"must be finite and within [{MIN_REFRESH_INTERVAL_SECONDS:g}, {MAX_REFRESH_INTERVAL_SECONDS:g}] seconds"+ )+ return value++# Heading + intro for the grouped Tier 3 options in ``validate --help``._RUN_GROUP = "Run & Reports"_RUN_GROUP_DESC = "Applies to the whole run: target typing, policy profile, reports, tier selection."@@ -2902,6 +2915,14 @@ def _tier2_workflow(show_default=True,help="Tier 3 progress presentation (auto uses Rich on a TTY and plain lines otherwise).",)+ "--progress-interval",+ type=float,+ default=None,+ metavar="SECONDS",+ callback=_validate_progress_interval,+ help="Seconds between Tier 3 progress heartbeats (defaults to 1 for Rich and 60 for plain/CI output).",+)@click.option("--verbose",is_flag=True,@@ -2939,6 +2960,7 @@ def evaluate(evaluated_source_revision: str | None,evaluator_container_revision: str | None,progress: str,+ progress_interval: float | None,verbose: bool,) -> None:"""Run Tier 3 live agent evaluation."""@@ -3033,7 +3055,11 @@ def evaluate(padding=(0, 1),))- progress_reporter = create_progress_reporter(progress, stream=click.get_text_stream("stderr"))+ progress_reporter = create_progress_reporter(+ progress,+ stream=click.get_text_stream("stderr"),+ refresh_interval=progress_interval,+ )engine_result = service.evaluate(options, progress_reporter=progress_reporter)failure = service.failure_reason(engine_result)display_result = engine_resultdiff --git a/src/skillevaluator/tier3/harbor/progress.py b/src/skillevaluator/tier3/harbor/progress.pyindex cdcca6c7..b3055b80 100644--- a/src/skillevaluator/tier3/harbor/progress.py+++ b/src/skillevaluator/tier3/harbor/progress.py@@ -11,6 +11,7 @@ or discarded by an API caller.from __future__ import annotationsimport logging+import mathimport reimport sysimport threading@@ -40,6 +41,15 @@ _SECRET_ENV_NAME_RE = re.compile(r"(?i)(?:api[_-]?key|access[_-]?key|auth|creden# shorter credential fragments come only from URI userinfo.MIN_EXACT_SECRET_CHARS = 4_CREDENTIAL_URI_USERINFO_RE = re.compile(r"(?i)(?P<scheme>[a-z][a-z0-9+.-]{0,31}://)(?P<userinfo>[^\s/?#]+@)")+# Plain lines are durable CI and log output, so the still-running heartbeat is+# written once a minute; the interactive Rich frame updates elapsed times every+# second.+_PLAIN_REFRESH_INTERVAL_SECONDS = 60.0+_RICH_REFRESH_INTERVAL_SECONDS = 1.0+MIN_REFRESH_INTERVAL_SECONDS = 0.05+# One day stays below every platform's threading.TIMEOUT_MAX, beyond which the+# heartbeat thread's wait would raise OverflowError.+MAX_REFRESH_INTERVAL_SECONDS = 86400.0@dataclass(frozen=True, slots=True)@@ -148,9 +158,18 @@ def secret_values_from_environment(environment: Mapping[str, str]) -> set[str]:class PlainProgressReporter:"""Durable line-oriented reporter for redirected output and CI logs."""- def __init__(self, *, stream: TextIO | None = None, refresh_interval: float = 1.0) -> None:+ def __init__(+ self,+ *,+ stream: TextIO | None = None,+ refresh_interval: float = _PLAIN_REFRESH_INTERVAL_SECONDS,+ ) -> None:+ interval = float(refresh_interval)+ # A NaN wait returns at once, so the heartbeat thread would spin.+ if not math.isfinite(interval):+ raise ValueError("refresh_interval must be a finite number of seconds")self._stream = stream or sys.stderr- self._refresh_interval = max(float(refresh_interval), 0.05)+ self._refresh_interval = min(max(interval, MIN_REFRESH_INTERVAL_SECONDS), MAX_REFRESH_INTERVAL_SECONDS)self._started_at: float | None = Noneself._active: dict[str, str] = {}self._secret_values: set[str] = set()@@ -286,7 +305,12 @@ class PlainProgressReporter:class RichProgressReporter(PlainProgressReporter):"""Interactive Rich Live table selected for terminal output."""- def __init__(self, *, stream: TextIO | None = None, refresh_interval: float = 1.0) -> None:+ def __init__(+ self,+ *,+ stream: TextIO | None = None,+ refresh_interval: float = _RICH_REFRESH_INTERVAL_SECONDS,+ ) -> None:super().__init__(stream=stream, refresh_interval=refresh_interval)from rich.console import Consolefrom rich.live import Live@@ -534,8 +558,17 @@ def safe_progress_reporter(reporter: ProgressReporter | None) -> SafeProgressRepreturn SafeProgressReporter(reporter or NullProgressReporter())-def create_progress_reporter(mode: ProgressMode, *, stream: TextIO | None = None) -> ProgressReporter:- """Create the requested presentation; ``auto`` follows stream TTY state."""+def create_progress_reporter(+ mode: ProgressMode,+ *,+ stream: TextIO | None = None,+ refresh_interval: float | None = None,+) -> ProgressReporter:+ """Create the requested presentation; ``auto`` follows stream TTY state.++ ``refresh_interval`` overrides the heartbeat period, which otherwise+ defaults to one second for Rich and one minute for plain lines.+ """if mode not in {"auto", "rich", "plain", "off"}:raise ValueError("progress must be one of: auto, rich, plain, off")if mode == "off":@@ -547,9 +580,10 @@ def create_progress_reporter(mode: ProgressMode, *, stream: TextIO | None = Noneconsole = Console(file=output)use_rich = console.is_terminal and not console.is_dumb_terminal+ reporter_options = {} if refresh_interval is None else {"refresh_interval": refresh_interval}if use_rich:- return RichProgressReporter(stream=output)- return PlainProgressReporter(stream=output)+ return RichProgressReporter(stream=output, **reporter_options)+ return PlainProgressReporter(stream=output, **reporter_options)__all__ = [diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.jsonindex 7388736d..6a8181f6 100644--- a/tests/golden/cli_surface.json+++ b/tests/golden/cli_surface.json@@ -697,6 +697,14 @@],"param_type": "option"},+ {+ "name": "progress_interval",+ "opts": [+ "--progress-interval"+ ],+ "param_type": "option",+ "type": "float"+ },{"default": "False","is_flag": true,@@ -2650,6 +2658,14 @@],"param_type": "option"},+ {+ "name": "progress_interval",+ "opts": [+ "--progress-interval"+ ],+ "param_type": "option",+ "type": "float"+ },{"default": "False","is_flag": true,@@ -3184,6 +3200,14 @@],"param_type": "option"},+ {+ "name": "progress_interval",+ "opts": [+ "--progress-interval"+ ],+ "param_type": "option",+ "type": "float"+ },{"default": "False","is_flag": true,diff --git a/tests/test_cli.py b/tests/test_cli.pyindex 9965abb9..167d586a 100644--- a/tests/test_cli.py+++ b/tests/test_cli.py@@ -784,6 +784,65 @@ def test_live_eval_progress_is_presentation_only(monkeypatch: pytest.MonkeyPatchassert "progress" not in captured["options"].engine_kwargs()[email protected]("command", [("evaluate",), ("tier3", "evaluate")])[email protected](("interval_args", "expected"), [((), 60.0), (("--progress-interval", "12.5"), 12.5)])+def test_live_eval_progress_interval_configures_reporter(+ monkeypatch: pytest.MonkeyPatch,+ command: tuple[str, ...],+ interval_args: tuple[str, ...],+ expected: float,+) -> None:+ from skillevaluator.evaluation import EvaluationService+ from skillevaluator.tier3.harbor.progress import PlainProgressReporter++ captured: dict[str, object] = {}++ def _fake_evaluate(self, options, *, progress_reporter=None):+ captured["options"] = options+ captured["reporter"] = progress_reporter+ return {"execution_status": "succeeded", "execution_errors": []}++ monkeypatch.setattr(EvaluationService, "evaluate", _fake_evaluate, raising=True)+ result = CliRunner().invoke(+ cli,+ [+ *command,+ str(Path(__file__).parent / "fixtures" / "skills" / "simple"),+ "--progress",+ "plain",+ *interval_args,+ ],+ )++ assert result.exit_code == 0, result.output+ assert isinstance(captured["reporter"], PlainProgressReporter)+ assert captured["reporter"]._refresh_interval == expected+ assert "progress_interval" not in captured["options"].engine_kwargs()++[email protected]("value", ["nan", "inf", "0.01", "86401", "-1"])+def test_live_eval_rejects_invalid_progress_interval(monkeypatch: pytest.MonkeyPatch, value: str) -> None:+ from skillevaluator.evaluation import EvaluationService++ def _unexpected_evaluate(self, options, *, progress_reporter=None):+ raise AssertionError("evaluation must not start with an invalid progress interval")++ monkeypatch.setattr(EvaluationService, "evaluate", _unexpected_evaluate, raising=True)+ result = CliRunner().invoke(+ cli,+ [+ "evaluate",+ str(Path(__file__).parent / "fixtures" / "skills" / "simple"),+ "--progress-interval",+ value,+ ],+ )++ assert result.exit_code == 2, result.output+ assert "Invalid value for '--progress-interval'" in result.output+ assert "must be finite and within [0.05, 86400] seconds" in result.output++_EVALUATED_REPOSITORY = "NVIDIA/NVFlare"_EVALUATED_COMMIT = "2263a2ebdab903e87f7e7c0a001d22c3a926a9cf"_EVALUATOR_CONTAINER = "ghcr.io/nvidia/skillevaluator@sha256:" + "0117bc2e" * 8diff --git a/tests/test_tier3_progress.py b/tests/test_tier3_progress.pyindex d77e1aaf..b772b8c0 100644--- a/tests/test_tier3_progress.py+++ b/tests/test_tier3_progress.py@@ -104,6 +104,61 @@ def test_auto_reporter_selects_terminal_appropriate_presentation(monkeypatch: pyassert isinstance(progress.create_progress_reporter("off", stream=_TTYBuffer()), progress.NullProgressReporter)+class _RecordingStop:+ """Stop event that records the heartbeat wait and ends the loop at once."""++ def __init__(self) -> None:+ self.timeouts: list[float] = []++ def wait(self, timeout: float) -> bool:+ self.timeouts.append(timeout)+ return True+++def test_plain_heartbeat_defaults_to_one_minute_and_rich_to_one_second(monkeypatch: pytest.MonkeyPatch) -> None:+ progress = _progress_module()+ monkeypatch.setenv("TERM", "xterm-256color")+ plain = progress.create_progress_reporter("auto", stream=io.StringIO())+ rich = progress.create_progress_reporter("auto", stream=_TTYBuffer())++ assert type(plain) is progress.PlainProgressReporter+ assert type(rich) is progress.RichProgressReporter+ for reporter, expected in ((plain, 60.0), (rich, 1.0)):+ stop = _RecordingStop()+ reporter._stop = stop+ reporter._heartbeat_loop()+ assert stop.timeouts == [expected]++[email protected]("mode", ["plain", "rich", "auto"])+def test_progress_interval_overrides_plain_and_rich_defaults(monkeypatch: pytest.MonkeyPatch, mode: str) -> None:+ progress = _progress_module()+ monkeypatch.setenv("TERM", "xterm-256color")++ reporter = progress.create_progress_reporter(mode, stream=_TTYBuffer(), refresh_interval=12.5)++ assert reporter._refresh_interval == 12.5++[email protected]("interval", [float("nan"), float("inf"), float("-inf")])+def test_progress_reporter_rejects_non_finite_refresh_interval(interval: float) -> None:+ progress = _progress_module()++ with pytest.raises(ValueError, match="finite number of seconds"):+ progress.create_progress_reporter("plain", stream=io.StringIO(), refresh_interval=interval)+++def test_progress_reporter_bounds_refresh_interval_below_thread_timeout_limit() -> None:+ progress = _progress_module()++ fast = progress.PlainProgressReporter(stream=io.StringIO(), refresh_interval=0.0)+ slow = progress.PlainProgressReporter(stream=io.StringIO(), refresh_interval=1e12)++ assert fast._refresh_interval == progress.MIN_REFRESH_INTERVAL_SECONDS+ assert slow._refresh_interval == progress.MAX_REFRESH_INTERVAL_SECONDS+ assert progress.MAX_REFRESH_INTERVAL_SECONDS < threading.TIMEOUT_MAX++def test_plain_plan_labels_the_maximum_scaled_top_level_agent_timeout() -> None:progress = _progress_module()output = io.StringIO()