Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

Move SkillSpector's Anthropic and Bedrock default to Claude Opus 5.5 (#607)

Christopher Kevinf46946236940
diff --git a/.gitleaks.toml b/.gitleaks.toml
index 2c64835c..07a9323d 100644
--- a/.gitleaks.toml
+++ b/.gitleaks.toml
@@ -59,6 +59,13 @@ targetRules = ["generic-api-key"]
regexes = ['''^us\.anthropic\.claude-opus-5$''']
paths = ['''^tests/validators/test_security\.py$''']
+[[allowlists]]
+description = "Claude Opus 5.5 Bedrock model ID that follows a credential name in SkillSpector bridge tests"
+condition = "AND"
+targetRules = ["generic-api-key"]
+regexes = ['''^us\.anthropic\.claude-opus-5-5$''']
+paths = ['''^tests/validators/test_security\.py$''']
+
[[allowlists]]
description = "Bedrock model ID in the merge-leftover SkillSpector bridge test copy from commit e8aa35d6a"
condition = "AND"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c0198f83..3e9f0feb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -154,6 +154,10 @@ All notable changes to SkillEvaluator are documented in this file.
for the Super model, so a run with no `llm-model` set answered HTTP 404 and the
rubric and LLM security verification did not run. The same default applies to
SkillSpector, the Tier 3 judge and NVIDIA Build local agents.
+- SkillSpector's default model on Anthropic and Bedrock moves from Claude Opus 5
+ to Claude Opus 5.5 (`claude-opus-5-5`, `us.anthropic.claude-opus-5-5`), matching
+ the chat default. SkillSpector 2.13.4 returns structured output for Claude 5.5;
+ a scan on the native Anthropic API completed at full coverage.
- The HTTP 404 message for an unavailable LLM model now says the provider's default
may not be callable with the key and points to `SKILL_EVAL_LLM_MODEL` (`llm-model`
in the reusable workflow); `ci/README.md` notes the same.
diff --git a/docs/configuration.mdx b/docs/configuration.mdx
index d14f69b9..e307d4b4 100644
--- a/docs/configuration.mdx
+++ b/docs/configuration.mdx
@@ -322,7 +322,7 @@ The `--llm` security analysis runs [SkillSpector](https://github.com/NVIDIA/Skil
The subprocess environment is isolated. Only the selected provider settings and basic process environment variables (`PATH`, `HOME`, TLS bundles, temp dirs) pass through — other ambient credentials in your shell are never forwarded. Your SkillSpector model settings (`SKILLSPECTOR_MODEL_<SLOT>`, `SKILLSPECTOR_MODEL_REGISTRY`, `SKILLSPECTOR_REASONING_EFFORT`) pass through too; `SKILLSPECTOR_TEMPERATURE` does not.
-SkillSpector sends its own structured-output requests, so it has its own default model per provider instead of following the chat default. On Anthropic and Bedrock it stays on Claude Opus 5 while the chat default is Claude Opus 5.5. An explicit `SKILL_EVAL_LLM_MODEL` still reaches SkillSpector, and `SKILLSPECTOR_MODEL` overrides both. SkillSpector v2.13.3 does not return structured output for Claude 5.5 models: if `SKILL_EVAL_LLM_MODEL` names one, also set `SKILLSPECTOR_MODEL` to a supported model such as `claude-opus-5` (Bedrock: `us.anthropic.claude-opus-5`), or the `--llm` security stage is reported incomplete. See [SkillSpector bridge](/skills/skillevaluator/environment-variables#skillspector-bridge) for the defaults.
+SkillSpector sends its own structured-output requests, so it has its own default model per provider instead of following the chat default. On Anthropic and Bedrock the default is Claude Opus 5.5, the same as the chat default; SkillSpector 2.13.4 returns structured output for Claude 5.5 models. An explicit `SKILL_EVAL_LLM_MODEL` still reaches SkillSpector, and `SKILLSPECTOR_MODEL` overrides both. See [SkillSpector bridge](/skills/skillevaluator/environment-variables#skillspector-bridge) for the defaults.
Setting `SKILLSPECTOR_PROVIDER` yourself overrides the bridge, but it must name a supported public provider (`openai`, `anthropic`, or `bedrock` — or `nv_build` to force the Build mapping); anything else fails closed and the subprocess receives no credential.
diff --git a/docs/environment-variables.mdx b/docs/environment-variables.mdx
index 1ad45d65..fcf231cd 100644
--- a/docs/environment-variables.mdx
+++ b/docs/environment-variables.mdx
@@ -108,7 +108,7 @@ The `--llm` security analysis runs [SkillSpector](https://github.com/NVIDIA/Skil
| Variable | Default | Effect |
| --- | --- | --- |
| `SKILLSPECTOR_PROVIDER` | bridged from your provider | Overrides the provider SkillSpector uses. Must name a supported public provider (`openai`, `anthropic`, `bedrock`, or `nv_build`) — only that provider's credential variables are forwarded. Any other value fails closed: SkillSpector runs without a credential. |
-| `SKILLSPECTOR_MODEL` | `SKILL_EVAL_LLM_MODEL` when set, else the SkillSpector default for your provider | Model SkillSpector uses for its LLM analysis. SkillSpector defaults: OpenAI `gpt-6.1-sol`, Anthropic `claude-opus-5`, Bedrock `us.anthropic.claude-opus-5`, NVIDIA Build `nvidia/nemotron-3.5-lightning-30b-a3b`, OpenAI-compatible `nvidia/nvidia/nemotron-3-super-120b-long-ctx`. They track the models the pinned SkillSpector release supports, not the chat defaults. |
+| `SKILLSPECTOR_MODEL` | `SKILL_EVAL_LLM_MODEL` when set, else the SkillSpector default for your provider | Model SkillSpector uses for its LLM analysis. SkillSpector defaults: OpenAI `gpt-6.1-sol`, Anthropic `claude-opus-5-5`, Bedrock `us.anthropic.claude-opus-5-5`, NVIDIA Build `nvidia/nemotron-3.5-lightning-30b-a3b`, OpenAI-compatible `nvidia/nvidia/nemotron-3-super-120b-long-ctx`. They track the models the pinned SkillSpector release supports, not the chat defaults. |
| `SKILLSPECTOR_MODEL_<SLOT>` | unset | Per-analyzer model override, such as `SKILLSPECTOR_MODEL_META_ANALYZER`. Forwarded as set. |
| `SKILLSPECTOR_MODEL_REGISTRY` | unset | Path to a complete custom SkillSpector model registry, for model IDs the bundled registry does not list. Forwarded as set. |
| `SKILLSPECTOR_REASONING_EFFORT` | `medium` for GPT-6 models, else unset | Reasoning effort for every SkillSpector analyzer. Your value wins. SkillEvaluator sets it only when every SkillSpector model, including `SKILLSPECTOR_MODEL_<SLOT>` overrides, is a GPT-6 model, so never for Claude. |
diff --git a/src/skillevaluator/provider_config.py b/src/skillevaluator/provider_config.py
index e7f23036..68b7bbd4 100644
--- a/src/skillevaluator/provider_config.py
+++ b/src/skillevaluator/provider_config.py
@@ -69,8 +69,8 @@ CHAT_DEFAULT_MODELS = {
# An explicit ``SKILL_EVAL_LLM_MODEL`` still reaches SkillSpector.
SKILLSPECTOR_DEFAULT_MODELS = {
"openai": "gpt-6.1-sol",
- "anthropic": "claude-opus-5",
- "bedrock": "us.anthropic.claude-opus-5",
+ "anthropic": "claude-opus-5-5",
+ "bedrock": "us.anthropic.claude-opus-5-5",
"nv_build": "nvidia/nemotron-3.5-lightning-30b-a3b",
"openai-compatible": "nvidia/nvidia/nemotron-3-super-120b-long-ctx",
}
diff --git a/tests/test_ci_workflows.py b/tests/test_ci_workflows.py
index 7503eeec..701bf0d6 100644
--- a/tests/test_ci_workflows.py
+++ b/tests/test_ci_workflows.py
@@ -289,6 +289,10 @@ def test_gitleaks_single_value_allowlists_are_exactly_scoped() -> None:
r"^us\.anthropic\.claude-opus-5$",
r"^tests/validators/test_security\.py$",
),
+ "Claude Opus 5.5 Bedrock model ID that follows a credential name in SkillSpector bridge tests": (
+ r"^us\.anthropic\.claude-opus-5-5$",
+ r"^tests/validators/test_security\.py$",
+ ),
"Bedrock model ID in the merge-leftover SkillSpector bridge test copy from commit e8aa35d6a": (
r"^us\.anthropic\.claude-opus-5$",
r"^tests/validators/test_security\.py\.orig$",
diff --git a/tests/test_provider_config.py b/tests/test_provider_config.py
index 8c7bb7a0..9c746802 100644
--- a/tests/test_provider_config.py
+++ b/tests/test_provider_config.py
@@ -93,8 +93,8 @@ def test_skillspector_defaults_cover_every_provider_and_keep_supported_claude_mo
# Literals, re-validated against the pinned SkillSpector release before they change.
assert SKILLSPECTOR_DEFAULT_MODELS == {
"openai": "gpt-6.1-sol",
- "anthropic": "claude-opus-5",
- "bedrock": "us.anthropic.claude-opus-5",
+ "anthropic": "claude-opus-5-5",
+ "bedrock": "us.anthropic.claude-opus-5-5",
"nv_build": "nvidia/nemotron-3.5-lightning-30b-a3b",
"openai-compatible": "nvidia/nvidia/nemotron-3-super-120b-long-ctx",
}
diff --git a/tests/validators/test_security.py b/tests/validators/test_security.py
index c7959269..ed453c04 100644
--- a/tests/validators/test_security.py
+++ b/tests/validators/test_security.py
@@ -5377,8 +5377,8 @@ assert cap is not None and get_max_output_tokens(model) <= cap, (cap, get_max_ou
("provider", "credential", "model", "effort"),
(
("openai", "OPENAI_API_KEY", "gpt-6.1-sol", "medium"),
- ("anthropic", "ANTHROPIC_API_KEY", "claude-opus-5", None),
- ("bedrock", "AWS_BEARER_TOKEN_BEDROCK", "us.anthropic.claude-opus-5", None),
+ ("anthropic", "ANTHROPIC_API_KEY", "claude-opus-5-5", None),
+ ("bedrock", "AWS_BEARER_TOKEN_BEDROCK", "us.anthropic.claude-opus-5-5", None),
),
)
def test_skillspector_child_environment_maps_supported_public_provider_defaults(
@@ -5507,8 +5507,8 @@ assert cap is not None and get_max_output_tokens(model) <= cap, (cap, get_max_ou
("provider", "credentials", "expected"),
(
("openai", {"OPENAI_API_KEY": "provider-test-key"}, "gpt-6.1-sol"),
- ("anthropic", {"ANTHROPIC_API_KEY": "provider-test-key"}, "claude-opus-5"),
- ("bedrock", {"AWS_BEARER_TOKEN_BEDROCK": "provider-test-key"}, "us.anthropic.claude-opus-5"),
+ ("anthropic", {"ANTHROPIC_API_KEY": "provider-test-key"}, "claude-opus-5-5"),
+ ("bedrock", {"AWS_BEARER_TOKEN_BEDROCK": "provider-test-key"}, "us.anthropic.claude-opus-5-5"),
("nv_build", {"NVIDIA_API_KEY": "provider-test-key"}, "nvidia/nemotron-3.5-lightning-30b-a3b"),
(
"openai-compatible",