Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

Name Nemotron 3.5 Lightning as the nv_build default in the docs (#606)

Christopher Kevinca2d2758b286
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3c25c47e..c0198f83 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -229,6 +229,12 @@ All notable changes to SkillEvaluator are documented in this file.
- The README's Tier 1 row now covers harness compatibility and links to the
`harness` check.
+- The `nv_build` documentation names `nvidia/nemotron-3.5-lightning-30b-a3b`
+ (Nemotron 3.5 Lightning) as the default model in the configuration guide,
+ environment variables reference, CLI reference, agents and sandboxes guide
+ and Tier 3 live evaluation guide, and the runnable `--agent-model` examples
+ use it. This entry covers the documentation only; the runtime default change
+ ships separately.
- `THIRD_PARTY_NOTICES.md` lists Bandit under the base dependencies. It has
been installed with the base package since it moved out of the `security`
extra, but the notice still sat in the Security row, which now lists
diff --git a/docs/agents-and-sandboxes.mdx b/docs/agents-and-sandboxes.mdx
index 068951c2..cfe0514e 100644
--- a/docs/agents-and-sandboxes.mdx
+++ b/docs/agents-and-sandboxes.mdx
@@ -162,7 +162,7 @@ automatically valid for every agent. Override the model per agent with
```bash title="Per-agent model override"
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker \
- --agent-model codex=nvidia/nemotron-3-super-120b-a12b
+ --agent-model codex=nvidia/nemotron-3.5-lightning-30b-a3b
```
To make an override persistent, set it in `evals/config.yml` under
@@ -201,9 +201,9 @@ orchestration and server tools Build cannot execute — `WebFetch`,
### Models on the one-key path
With `nv_build` and no explicit model override, every supported agent defaults
-to `nvidia/nemotron-3-super-120b-a12b`. The provider-native agent selection is
+to `nvidia/nemotron-3.5-lightning-30b-a3b`. The provider-native agent selection is
OpenCode. Its model is automatically qualified with the `nvidia/` adapter
-namespace, so it renders as `nvidia/nvidia/nemotron-3-super-120b-a12b` in run
+namespace, so it renders as `nvidia/nvidia/nemotron-3.5-lightning-30b-a3b` in run
output — the doubled prefix is the adapter namespace plus the catalog publisher,
not a bug.
diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdx
index d45a90fd..47bb2499 100644
--- a/docs/cli-reference.mdx
+++ b/docs/cli-reference.mdx
@@ -663,8 +663,8 @@ Use `skillevaluator tier3 PATH` for the complete workflow with automatic missing
Without a `--model`/`--agent-model` override, agent execution uses
`gpt-6.1-sol` for OpenAI (OpenCode: `openai/gpt-5.6-sol` unless
`SKILL_EVAL_LLM_MODEL` is set), `claude-opus-5-5` for Anthropic, and
-`nvidia/nemotron-3-super-120b-a12b` for NVIDIA Build. OpenCode displays the
-last value as `nvidia/nvidia/nemotron-3-super-120b-a12b` because its adapter
+`nvidia/nemotron-3.5-lightning-30b-a3b` for NVIDIA Build. OpenCode displays the
+last value as `nvidia/nvidia/nemotron-3.5-lightning-30b-a3b` because its adapter
namespace precedes the NVIDIA catalog ID.
Flags marked "unset" fall back to their matching keys in `evals/config.yml` where one exists, then to the built-in defaults. The exception is `--results-dir`, which resolves independently of `evals/config.yml`.
diff --git a/docs/configuration.mdx b/docs/configuration.mdx
index 63ee8d4b..d14f69b9 100644
--- a/docs/configuration.mdx
+++ b/docs/configuration.mdx
@@ -27,9 +27,10 @@ No model override is needed. Defaults are pinned in the installed release,
not automatically selected as the latest available model. Set
`SKILL_EVAL_LLM_MODEL` only when you want a different evaluator model; the
standard judge inherits it unless a judge override is set. Embeddings use their
-own model, and Tier 3 agent selection is separate. With NVIDIA Build, OpenCode
-follows `SKILL_EVAL_LLM_MODEL`, while Codex and Claude Code keep the Super
-default unless you override their models.
+own model, and Tier 3 agent selection is separate. With NVIDIA Build, all three
+agents default to Nemotron 3.5 Lightning. OpenCode follows
+`SKILL_EVAL_LLM_MODEL`, while Codex and Claude Code keep the Lightning default
+unless you override their models.
Anthropic and Bedrock need a separate embedding provider for Tier 2; custom
OpenAI-compatible endpoints require an explicit URL and key, with optional model overrides.
@@ -115,7 +116,7 @@ Set `SKILL_EVAL_LLM_PROVIDER` to one of `openai`, `anthropic`, `nv_build`, `bedr
export NVIDIA_API_KEY='nvapi-...'
```
- Default model: `nvidia/nemotron-3-super-120b-a12b`, shared by evaluator chat, Tier 3's default OpenCode agent, and the standard judge. The endpoint is fixed to `https://integrate.api.nvidia.com/v1` — `SKILL_EVAL_LLM_BASE_URL` and `SKILL_EVAL_EMBEDDING_BASE_URL` do not override it. To point at a custom or self-hosted endpoint, use the `openai-compatible` provider instead.
+ Default model: `nvidia/nemotron-3.5-lightning-30b-a3b`, shared by evaluator chat, Tier 3's default OpenCode agent, and the standard judge. The endpoint is fixed to `https://integrate.api.nvidia.com/v1` — `SKILL_EVAL_LLM_BASE_URL` and `SKILL_EVAL_EMBEDDING_BASE_URL` do not override it. To point at a custom or self-hosted endpoint, use the `openai-compatible` provider instead.
The previous default, `nvidia/nemotron-3-nano-30b-a3b`, has a deprecated public endpoint. If you explicitly pinned that model in `SKILL_EVAL_LLM_MODEL`, remove the override to use the current default. Explicit model selections are preserved; SkillEvaluator does not silently switch models after a failed request.
diff --git a/docs/environment-variables.mdx b/docs/environment-variables.mdx
index 9c79373d..1ad45d65 100644
--- a/docs/environment-variables.mdx
+++ b/docs/environment-variables.mdx
@@ -26,7 +26,7 @@ These variables select and configure the provider used for LLM-backed checks and
| Variable | Default | Effect |
| --- | --- | --- |
| `SKILL_EVAL_LLM_PROVIDER` | auto-selected (see below) | Provider for chat and judging: `openai`, `anthropic`, `nv_build`, `bedrock`, or `openai-compatible`. |
-| `SKILL_EVAL_LLM_MODEL` | OpenAI: `gpt-6.1-sol`; Anthropic: `claude-opus-5-5`; NVIDIA Build: `nvidia/nemotron-3-super-120b-a12b`; OpenAI-compatible: `nvidia/nvidia/nemotron-3-super-120b-long-ctx`; Bedrock: `us.anthropic.claude-opus-5-5` | Chat model override. For gateways, use the exact catalog ID when it differs from the default. Setting it to an empty string is a configuration error. For a lower-cost OpenAI option, use `gpt-5.4-mini`. |
+| `SKILL_EVAL_LLM_MODEL` | OpenAI: `gpt-6.1-sol`; Anthropic: `claude-opus-5-5`; NVIDIA Build: `nvidia/nemotron-3.5-lightning-30b-a3b`; OpenAI-compatible: `nvidia/nvidia/nemotron-3-super-120b-long-ctx`; Bedrock: `us.anthropic.claude-opus-5-5` | Chat model override. For gateways, use the exact catalog ID when it differs from the default. Setting it to an empty string is a configuration error. For a lower-cost OpenAI option, use `gpt-5.4-mini`. |
| `SKILL_EVAL_LLM_BASE_URL` | provider default endpoint | Endpoint override; takes precedence over `OPENAI_BASE_URL` and `ANTHROPIC_BASE_URL`. Required for `openai-compatible`. Ignored by `nv_build`, whose endpoint is fixed to `https://integrate.api.nvidia.com/v1` — point a custom endpoint at the `openai-compatible` provider instead. |
| `SKILL_EVAL_LLM_API_KEY` | — | API key for the `openai-compatible` provider (local servers still require it to be set). |
| `SKILL_EVAL_MODEL_CATALOG_ALLOW_HTTP_HOSTS` | unset | Comma-separated hosts whose model catalog may be read over plain HTTP. Catalog reads otherwise require HTTPS unless the host is loopback, because the request carries a bearer token. Use this only when the transport is already encrypted and authenticated below HTTP — a WireGuard or comparable tunnel peer, for example — where the tool cannot see that the link is protected. Each entry matches one whole host as written, with no resolution, no suffix matching, and no wildcards. A plain-HTTP request to an accepted host also bypasses any inherited HTTP proxy, so the bearer token is never offered to an intermediary. The transport rechecks authorization before dispatch and rejects hosts that are no longer allowed. HTTPS routing is unchanged. |
@@ -108,7 +108,7 @@ The `--llm` security analysis runs [SkillSpector](https://github.com/NVIDIA/Skil
| Variable | Default | Effect |
| --- | --- | --- |
| `SKILLSPECTOR_PROVIDER` | bridged from your provider | Overrides the provider SkillSpector uses. Must name a supported public provider (`openai`, `anthropic`, `bedrock`, or `nv_build`) — only that provider's credential variables are forwarded. Any other value fails closed: SkillSpector runs without a credential. |
-| `SKILLSPECTOR_MODEL` | `SKILL_EVAL_LLM_MODEL` when set, else the SkillSpector default for your provider | Model SkillSpector uses for its LLM analysis. SkillSpector defaults: OpenAI `gpt-6.1-sol`, Anthropic `claude-opus-5`, Bedrock `us.anthropic.claude-opus-5`, NVIDIA Build `nvidia/nemotron-3-super-120b-a12b`, OpenAI-compatible `nvidia/nvidia/nemotron-3-super-120b-long-ctx`. They track the models the pinned SkillSpector release supports, not the chat defaults. |
+| `SKILLSPECTOR_MODEL` | `SKILL_EVAL_LLM_MODEL` when set, else the SkillSpector default for your provider | Model SkillSpector uses for its LLM analysis. SkillSpector defaults: OpenAI `gpt-6.1-sol`, Anthropic `claude-opus-5`, Bedrock `us.anthropic.claude-opus-5`, NVIDIA Build `nvidia/nemotron-3.5-lightning-30b-a3b`, OpenAI-compatible `nvidia/nvidia/nemotron-3-super-120b-long-ctx`. They track the models the pinned SkillSpector release supports, not the chat defaults. |
| `SKILLSPECTOR_MODEL_<SLOT>` | unset | Per-analyzer model override, such as `SKILLSPECTOR_MODEL_META_ANALYZER`. Forwarded as set. |
| `SKILLSPECTOR_MODEL_REGISTRY` | unset | Path to a complete custom SkillSpector model registry, for model IDs the bundled registry does not list. Forwarded as set. |
| `SKILLSPECTOR_REASONING_EFFORT` | `medium` for GPT-6 models, else unset | Reasoning effort for every SkillSpector analyzer. Your value wins. SkillEvaluator sets it only when every SkillSpector model, including `SKILLSPECTOR_MODEL_<SLOT>` overrides, is a GPT-6 model, so never for Claude. |
diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdx
index f02402d3..12bd0aa0 100644
--- a/docs/tier3-live-evaluation.mdx
+++ b/docs/tier3-live-evaluation.mdx
@@ -672,18 +672,18 @@ in `--env-mode docker`, `kata`, or `local`, a single `NVIDIA_API_KEY` powers the
evaluator and all three agents. Direct OpenCode talks to NVIDIA Build directly;
Codex and experimental Claude Code use an in-container (Docker and Kata) or local compatibility bridge.
Explicit model overrides are preserved exactly. Without an override, all three
-agents default to Nemotron Super (`nvidia/nemotron-3-super-120b-a12b`); direct
-OpenCode renders it as `nvidia/nvidia/nemotron-3-super-120b-a12b` because its
+agents default to Nemotron 3.5 Lightning (`nvidia/nemotron-3.5-lightning-30b-a3b`); direct
+OpenCode renders it as `nvidia/nvidia/nemotron-3.5-lightning-30b-a3b` because its
adapter namespace precedes the NVIDIA catalog ID. Override models explicitly
when you want to compare harnesses or models:
```bash title="NVIDIA Build model overrides"
skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker \
- --agent-model opencode=nvidia/nvidia/nemotron-3-super-120b-a12b
+ --agent-model opencode=nvidia/nvidia/nemotron-3.5-lightning-30b-a3b
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker \
- --agent-model codex=nvidia/nemotron-3-super-120b-a12b
+ --agent-model codex=nvidia/nemotron-3.5-lightning-30b-a3b
skillevaluator tier3 evaluate ./my-skill --agents claude-code --env-mode docker \
- --agent-model claude-code=nvidia/nemotron-3-super-120b-a12b
+ --agent-model claude-code=nvidia/nemotron-3.5-lightning-30b-a3b
skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker \
--agent-model opencode=nvidia/meta/llama-3.1-8b-instruct
```
diff --git a/tests/test_oss_packaging.py b/tests/test_oss_packaging.py
index eb10bd72..90b9d821 100644
--- a/tests/test_oss_packaging.py
+++ b/tests/test_oss_packaging.py
@@ -850,13 +850,13 @@ def test_public_docs_show_external_nvidia_build_harness_paths_only() -> None:
assert f"`{CHAT_DEFAULT_OPENAI}`" in public_docs
assert "gpt-5.4-mini" in public_docs
assert f"`{CHAT_DEFAULT_ANTHROPIC}`" in public_docs
- assert "nvidia/nemotron-3-super-120b-a12b" in public_docs
- assert "nvidia/nvidia/nemotron-3-super-120b-a12b" in public_docs
- assert "Nemotron Super" in public_docs
+ assert "nvidia/nemotron-3.5-lightning-30b-a3b" in public_docs
+ assert "nvidia/nvidia/nemotron-3.5-lightning-30b-a3b" in public_docs
+ assert "Nemotron 3.5 Lightning" in public_docs
assert "meta/llama-3.1-8b-instruct" in public_docs
- assert "--agent-model opencode=nvidia/nvidia/nemotron-3-super-120b-a12b" in public_docs
- assert "--agent-model codex=nvidia/nemotron-3-super-120b-a12b" in public_docs
- assert "--agent-model claude-code=nvidia/nemotron-3-super-120b-a12b" in public_docs
+ assert "--agent-model opencode=nvidia/nvidia/nemotron-3.5-lightning-30b-a3b" in public_docs
+ assert "--agent-model codex=nvidia/nemotron-3.5-lightning-30b-a3b" in public_docs
+ assert "--agent-model claude-code=nvidia/nemotron-3.5-lightning-30b-a3b" in public_docs
assert "--agent-model opencode=nvidia/meta/llama-3.1-8b-instruct" in public_docs
assert "skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker\n" in tier3
assert "skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker\n" in tier3