Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

chore(release): prepare v0.2.1

Narendran Raghavan0691c64cd989
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3d96098c..169ca6a9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,8 @@ All notable changes to SkillEvaluator are documented in this file.
## Unreleased
+## 0.2.1 - 2026-08-24
+
### Added
- Added a public benchmark publication gate, regression coverage, and a
@@ -14,6 +16,12 @@ All notable changes to SkillEvaluator are documented in this file.
- Unified Tier 3 scoring around the canonical five dimensions, persisted an
immutable dataset-truth snapshot with provenance metadata, and redesigned
`BENCHMARK.md` as a decision-first publication card.
+- Updated public OpenAI / Anthropic / Bedrock chat defaults to pinned frontier
+ models (`gpt-5.6-sol`, `claude-opus-5`, `us.anthropic.claude-opus-5`),
+ centralized in `provider_config`, and documented `gpt-5.4-mini` as the
+ lower-cost OpenAI `SKILL_EVAL_LLM_MODEL` alternative. Raised
+ dimension/insights judge token budgets to 4096 and widened the gpt-5\*
+ temperature guard to bare model IDs.
### Fixed
@@ -25,6 +33,32 @@ All notable changes to SkillEvaluator are documented in this file.
- GitHub Actions pull request reports now link source targets to the checked-out
repository revision instead of the synthetic `<number>/merge` ref, preventing
broken or cross-repository links.
+- Tier 3 LLM insights now receive explicit labels and bounded expected-behavior
+ context for `expected_skill: null` negative controls, and the judge is
+ instructed not to flag unrelated successes without invocation or
+ failed-routing evidence.
+- Fixed Anthropic API-root normalization across evaluator and Claude Code
+ paths, and made required Tier 3 judge failures fail closed instead of
+ appearing as numeric zero scores or publishing misleading quality results
+ ([#55](https://github.com/NVIDIA/SkillEvaluator/issues/55)).
+- Tier 3 now normalizes host-configured `LLM_JUDGE_MODEL` and
+ `SKILL_EVAL_JUDGE_MODEL` overrides in Harbor's parent process and forwards
+ the selected value through its verifier-only job layer for standard grading.
+ This lets native separate-verifier placeholders resolve without injecting
+ either name into the evaluated agent's initial environment. Skill-authored
+ `runtime_env` and native task `[environment.env]` tables cannot set or alias
+ either operator-controlled override.
+ Native verifier declarations remain compatible, while the job-level value
+ takes precedence during standard grading. Tier 3 results now record the
+ configured judge provider, model, source, and whether a dedicated job-wide
+ override was applied, separately from agent models. A provider fallback may
+ still use a different model for an individual judge call.
+- Quality scoring now uses boundary-aware lexical matching and CommonMark-parsed
+ structural links instead of hand-written Markdown parsing or regex inference
+ of author intent. Deterministic checks no longer infer MCP negation, temporal
+ intent, README guidance, or exclusivity from prose;
+ use `rubric-eval` for semantic documentation judgments and Tier 3 for
+ observed agent behavior.
## 0.2.0 - 2026-08-18
@@ -38,12 +72,6 @@ All notable changes to SkillEvaluator are documented in this file.
### Changed
-- Updated public OpenAI / Anthropic / Bedrock chat defaults to pinned frontier
- models (`gpt-5.6-sol`, `claude-opus-5`, `us.anthropic.claude-opus-5`),
- centralized in `provider_config`, and documented `gpt-5.4-mini` as the
- lower-cost OpenAI `SKILL_EVAL_LLM_MODEL` alternative. Raised
- dimension/insights judge token budgets to 4096 and widened the gpt-5\*
- temperature guard to bare model IDs.
- Added explicit `--block-on-dedup` / `--no-block-on-dedup` and
`--block-on-agent-eval` / `--no-block-on-agent-eval` controls with
backward-compatible defaults, Tier 3 source preflight, and consistent gating
@@ -58,32 +86,10 @@ All notable changes to SkillEvaluator are documented in this file.
### Fixed
-- Tier 3 LLM insights now receive explicit labels and bounded expected-behavior
- context for `expected_skill: null` negative controls, and the judge is
- instructed not to flag unrelated successes without invocation or
- failed-routing evidence.
-- Fixed Anthropic API-root normalization across evaluator and Claude Code
- paths, and made required Tier 3 judge failures fail closed instead of
- appearing as numeric zero scores or publishing misleading quality results
- ([#55](https://github.com/NVIDIA/SkillEvaluator/issues/55)).
-- Tier 3 now normalizes host-configured `LLM_JUDGE_MODEL` and
- `SKILL_EVAL_JUDGE_MODEL` overrides in Harbor's parent process and forwards
- the selected value through its verifier-only job layer for standard grading.
- This lets native separate-verifier placeholders resolve without injecting
- either name into the evaluated agent's initial environment. Skill-authored
- `runtime_env` and native task `[environment.env]` tables cannot set or alias
- either operator-controlled override.
- Native verifier declarations remain compatible, while the job-level value
- takes precedence during standard grading. Tier 3 results now record the
- configured judge provider, model, source, and whether a dedicated job-wide
- override was applied, separately from agent models. A provider fallback may
- still use a different model for an individual judge call.
-- Quality scoring now uses boundary-aware lexical matching and CommonMark-parsed
- structural links instead of hand-written Markdown parsing or regex inference
- of author intent. Deterministic checks no longer infer MCP negation, temporal
- intent, README guidance, or exclusivity from prose;
- use `rubric-eval` for semantic documentation judgments and Tier 3 for
- observed agent behavior.
+- Quality scoring now uses boundary-aware and context-aware matching for XML tags,
+ reserved names, MCP guidance, README references, time references, exclusivity
+ language, instruction action verbs, and nested Markdown links, avoiding
+ incidental-word score changes.
- Tier 3 generated tasks now stage only an entry's declared `files`, preventing
undeclared fixtures from the shared `evals/files/` directory from appearing
in that task's `/workspace/input/`, while preserving copy-all behavior for
diff --git a/CITATION.cff b/CITATION.cff
index 3f918304..57f0e284 100644
--- a/CITATION.cff
+++ b/CITATION.cff
@@ -2,7 +2,7 @@ cff-version: 1.2.0
message: "If you use SkillEvaluator in academic or technical work, please cite this software."
title: "SkillEvaluator"
type: software
-version: "0.2.0"
+version: "0.2.1"
authors:
- name: "NVIDIA Corporation"
license: Apache-2.0
diff --git a/docs/developer-guide.mdx b/docs/developer-guide.mdx
index 92f1b39c..027f7a2f 100644
--- a/docs/developer-guide.mdx
+++ b/docs/developer-guide.mdx
@@ -143,7 +143,7 @@ cff-version: 1.2.0
message: "If you use SkillEvaluator in academic or technical work, please cite this software."
title: "SkillEvaluator"
type: software
-version: "0.2.0"
+version: "0.2.1"
authors:
- name: "NVIDIA Corporation"
license: Apache-2.0
diff --git a/docs/quickstart.mdx b/docs/quickstart.mdx
index 88c74b62..2c91c12d 100644
--- a/docs/quickstart.mdx
+++ b/docs/quickstart.mdx
@@ -22,7 +22,7 @@ Get your first evaluation result in about two minutes, with no API key. This pag
skillevaluator --version
```
- You should see `skillevaluator, version 0.2.0` (or newer).
+ You should see `skillevaluator, version 0.2.1` (or newer).
<Tip>
If the shell can't find `skillevaluator`, run `uv tool update-shell` and open a new terminal — uv installs tools to `~/.local/bin`, which may not be on your PATH yet.
diff --git a/pyproject.toml b/pyproject.toml
index 302b43fa..578b5443 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -7,7 +7,7 @@ requires = ["setuptools >= 77", "wheel"]
[project]
name = "skillevaluator"
-version = "0.2.0"
+version = "0.2.1"
description = "Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation."
authors = [{ name = "NVIDIA Corporation" }]
readme = "README.md"
diff --git a/src/skillevaluator/__init__.py b/src/skillevaluator/__init__.py
index 06ec6fd2..c1d96d5a 100644
--- a/src/skillevaluator/__init__.py
+++ b/src/skillevaluator/__init__.py
@@ -10,6 +10,6 @@ from importlib.metadata import PackageNotFoundError, version
try:
__version__ = version("skillevaluator")
except PackageNotFoundError:
- __version__ = "0.2.0"
+ __version__ = "0.2.1"
__all__ = ["__version__"]
diff --git a/uv.lock b/uv.lock
index 174cb4ac..f6fb5c20 100644
--- a/uv.lock
+++ b/uv.lock
@@ -1967,7 +1967,7 @@ wheels = [
[[package]]
name = "skillevaluator"
-version = "0.2.0"
+version = "0.2.1"
source = { editable = "." }
dependencies = [
{ name = "click" },