Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
chore(release): prepare v0.2.1
Narendran Raghavan0691c64cd989
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 3d96098c..169ca6a9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,6 +4,8 @@ All notable changes to SkillEvaluator are documented in this file.## Unreleased+## 0.2.1 - 2026-08-24+### Added- Added a public benchmark publication gate, regression coverage, and a@@ -14,6 +16,12 @@ All notable changes to SkillEvaluator are documented in this file.- Unified Tier 3 scoring around the canonical five dimensions, persisted animmutable dataset-truth snapshot with provenance metadata, and redesigned`BENCHMARK.md` as a decision-first publication card.+- Updated public OpenAI / Anthropic / Bedrock chat defaults to pinned frontier+ models (`gpt-5.6-sol`, `claude-opus-5`, `us.anthropic.claude-opus-5`),+ centralized in `provider_config`, and documented `gpt-5.4-mini` as the+ lower-cost OpenAI `SKILL_EVAL_LLM_MODEL` alternative. Raised+ dimension/insights judge token budgets to 4096 and widened the gpt-5\*+ temperature guard to bare model IDs.### Fixed@@ -25,6 +33,32 @@ All notable changes to SkillEvaluator are documented in this file.- GitHub Actions pull request reports now link source targets to the checked-outrepository revision instead of the synthetic `<number>/merge` ref, preventingbroken or cross-repository links.+- Tier 3 LLM insights now receive explicit labels and bounded expected-behavior+ context for `expected_skill: null` negative controls, and the judge is+ instructed not to flag unrelated successes without invocation or+ failed-routing evidence.+- Fixed Anthropic API-root normalization across evaluator and Claude Code+ paths, and made required Tier 3 judge failures fail closed instead of+ appearing as numeric zero scores or publishing misleading quality results+ ([#55](https://github.com/NVIDIA/SkillEvaluator/issues/55)).+- Tier 3 now normalizes host-configured `LLM_JUDGE_MODEL` and+ `SKILL_EVAL_JUDGE_MODEL` overrides in Harbor's parent process and forwards+ the selected value through its verifier-only job layer for standard grading.+ This lets native separate-verifier placeholders resolve without injecting+ either name into the evaluated agent's initial environment. Skill-authored+ `runtime_env` and native task `[environment.env]` tables cannot set or alias+ either operator-controlled override.+ Native verifier declarations remain compatible, while the job-level value+ takes precedence during standard grading. Tier 3 results now record the+ configured judge provider, model, source, and whether a dedicated job-wide+ override was applied, separately from agent models. A provider fallback may+ still use a different model for an individual judge call.+- Quality scoring now uses boundary-aware lexical matching and CommonMark-parsed+ structural links instead of hand-written Markdown parsing or regex inference+ of author intent. Deterministic checks no longer infer MCP negation, temporal+ intent, README guidance, or exclusivity from prose;+ use `rubric-eval` for semantic documentation judgments and Tier 3 for+ observed agent behavior.## 0.2.0 - 2026-08-18@@ -38,12 +72,6 @@ All notable changes to SkillEvaluator are documented in this file.### Changed-- Updated public OpenAI / Anthropic / Bedrock chat defaults to pinned frontier- models (`gpt-5.6-sol`, `claude-opus-5`, `us.anthropic.claude-opus-5`),- centralized in `provider_config`, and documented `gpt-5.4-mini` as the- lower-cost OpenAI `SKILL_EVAL_LLM_MODEL` alternative. Raised- dimension/insights judge token budgets to 4096 and widened the gpt-5\*- temperature guard to bare model IDs.- Added explicit `--block-on-dedup` / `--no-block-on-dedup` and`--block-on-agent-eval` / `--no-block-on-agent-eval` controls withbackward-compatible defaults, Tier 3 source preflight, and consistent gating@@ -58,32 +86,10 @@ All notable changes to SkillEvaluator are documented in this file.### Fixed-- Tier 3 LLM insights now receive explicit labels and bounded expected-behavior- context for `expected_skill: null` negative controls, and the judge is- instructed not to flag unrelated successes without invocation or- failed-routing evidence.-- Fixed Anthropic API-root normalization across evaluator and Claude Code- paths, and made required Tier 3 judge failures fail closed instead of- appearing as numeric zero scores or publishing misleading quality results- ([#55](https://github.com/NVIDIA/SkillEvaluator/issues/55)).-- Tier 3 now normalizes host-configured `LLM_JUDGE_MODEL` and- `SKILL_EVAL_JUDGE_MODEL` overrides in Harbor's parent process and forwards- the selected value through its verifier-only job layer for standard grading.- This lets native separate-verifier placeholders resolve without injecting- either name into the evaluated agent's initial environment. Skill-authored- `runtime_env` and native task `[environment.env]` tables cannot set or alias- either operator-controlled override.- Native verifier declarations remain compatible, while the job-level value- takes precedence during standard grading. Tier 3 results now record the- configured judge provider, model, source, and whether a dedicated job-wide- override was applied, separately from agent models. A provider fallback may- still use a different model for an individual judge call.-- Quality scoring now uses boundary-aware lexical matching and CommonMark-parsed- structural links instead of hand-written Markdown parsing or regex inference- of author intent. Deterministic checks no longer infer MCP negation, temporal- intent, README guidance, or exclusivity from prose;- use `rubric-eval` for semantic documentation judgments and Tier 3 for- observed agent behavior.+- Quality scoring now uses boundary-aware and context-aware matching for XML tags,+ reserved names, MCP guidance, README references, time references, exclusivity+ language, instruction action verbs, and nested Markdown links, avoiding+ incidental-word score changes.- Tier 3 generated tasks now stage only an entry's declared `files`, preventingundeclared fixtures from the shared `evals/files/` directory from appearingin that task's `/workspace/input/`, while preserving copy-all behavior fordiff --git a/CITATION.cff b/CITATION.cffindex 3f918304..57f0e284 100644--- a/CITATION.cff+++ b/CITATION.cff@@ -2,7 +2,7 @@ cff-version: 1.2.0message: "If you use SkillEvaluator in academic or technical work, please cite this software."title: "SkillEvaluator"type: software-version: "0.2.0"+version: "0.2.1"authors:- name: "NVIDIA Corporation"license: Apache-2.0diff --git a/docs/developer-guide.mdx b/docs/developer-guide.mdxindex 92f1b39c..027f7a2f 100644--- a/docs/developer-guide.mdx+++ b/docs/developer-guide.mdx@@ -143,7 +143,7 @@ cff-version: 1.2.0message: "If you use SkillEvaluator in academic or technical work, please cite this software."title: "SkillEvaluator"type: software-version: "0.2.0"+version: "0.2.1"authors:- name: "NVIDIA Corporation"license: Apache-2.0diff --git a/docs/quickstart.mdx b/docs/quickstart.mdxindex 88c74b62..2c91c12d 100644--- a/docs/quickstart.mdx+++ b/docs/quickstart.mdx@@ -22,7 +22,7 @@ Get your first evaluation result in about two minutes, with no API key. This pagskillevaluator --version```- You should see `skillevaluator, version 0.2.0` (or newer).+ You should see `skillevaluator, version 0.2.1` (or newer).<Tip>If the shell can't find `skillevaluator`, run `uv tool update-shell` and open a new terminal — uv installs tools to `~/.local/bin`, which may not be on your PATH yet.diff --git a/pyproject.toml b/pyproject.tomlindex 302b43fa..578b5443 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -7,7 +7,7 @@ requires = ["setuptools >= 77", "wheel"][project]name = "skillevaluator"-version = "0.2.0"+version = "0.2.1"description = "Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation."authors = [{ name = "NVIDIA Corporation" }]readme = "README.md"diff --git a/src/skillevaluator/__init__.py b/src/skillevaluator/__init__.pyindex 06ec6fd2..c1d96d5a 100644--- a/src/skillevaluator/__init__.py+++ b/src/skillevaluator/__init__.py@@ -10,6 +10,6 @@ from importlib.metadata import PackageNotFoundError, versiontry:__version__ = version("skillevaluator")except PackageNotFoundError:- __version__ = "0.2.0"+ __version__ = "0.2.1"__all__ = ["__version__"]diff --git a/uv.lock b/uv.lockindex 174cb4ac..f6fb5c20 100644--- a/uv.lock+++ b/uv.lock@@ -1967,7 +1967,7 @@ wheels = [[[package]]name = "skillevaluator"-version = "0.2.0"+version = "0.2.1"source = { editable = "." }dependencies = [{ name = "click" },