Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
---
title: "Environment Variables"
description: "The complete reference of every environment variable SkillEvaluator reads — providers, embeddings, validation, and live evaluation."
layout: reference
---
Every environment variable SkillEvaluator reads, grouped by feature area. Each group links to the guide that explains the workflow the variables belong to — this page is the inventory, not the tutorial.
Unless a variable's row specifies otherwise, boolean variables accept `1`, `true`, `yes`, or `on` (case-insensitive) as true, and `0`, `false`, `no`, or `off` as false. Any other value that is set, including an empty one, also counts as false.
Two historical SkillEvaluator prefixes coexist. `SKILL_EVAL_*` covers provider
and model configuration plus a small set of Tier 3 engine knobs;
`SKILLEVALUATOR_*` covers product-level validation, report paths, scanner
paths, and sandbox/runtime behavior. The prefixes are part of each variable's
literal public name and are not interchangeable. The separately documented
`SKILLSPECTOR_*` family belongs to SkillSpector, provider credential families
belong to their providers, and
[Harbor](https://github.com/harbor-framework/harbor), the open-source agent
evaluation framework used for Tier 3, owns the `HARBOR_*` in-container grader
contract. None belongs to either SkillEvaluator prefix.
## LLM provider
These variables select and configure the provider used for LLM-backed checks and judging. The setup walkthrough, including per-provider export blocks and default models, is in [Providers & Credentials](/skills/skillevaluator/configuration).
| Variable | Default | Effect |
| --- | --- | --- |
| `SKILL_EVAL_LLM_PROVIDER` | auto-selected (see below) | Provider for chat and judging: `openai`, `anthropic`, `nv_build`, `bedrock`, or `openai-compatible`. |
| `SKILL_EVAL_LLM_MODEL` | OpenAI: `gpt-6.1-sol`; Anthropic: `claude-opus-5-5`; NVIDIA Build: `nvidia/nemotron-3.5-lightning-30b-a3b`; OpenAI-compatible: `nvidia/nvidia/nemotron-3-super-120b-long-ctx`; Bedrock: `us.anthropic.claude-opus-5-5` | Chat model override. For gateways, use the exact catalog ID when it differs from the default. Setting it to an empty string is a configuration error. For a lower-cost OpenAI option, use `gpt-5.4-mini`. |
| `SKILL_EVAL_LLM_BASE_URL` | provider default endpoint | Endpoint override; takes precedence over `OPENAI_BASE_URL` and `ANTHROPIC_BASE_URL`. Required for `openai-compatible`. Ignored by `nv_build`, whose endpoint is fixed to `https://integrate.api.nvidia.com/v1` — point a custom endpoint at the `openai-compatible` provider instead. |
| `SKILL_EVAL_LLM_API_KEY` | — | API key for the `openai-compatible` provider (local servers still require it to be set). |
| `SKILL_EVAL_MODEL_CATALOG_ALLOW_HTTP_HOSTS` | unset | Comma-separated hosts whose model catalog may be read over plain HTTP. Catalog reads otherwise require HTTPS unless the host is loopback, because the request carries a bearer token. Use this only when the transport is already encrypted and authenticated below HTTP — a WireGuard or comparable tunnel peer, for example — where the tool cannot see that the link is protected. Each entry matches one whole host as written, with no resolution, no suffix matching, and no wildcards. A plain-HTTP request to an accepted host also bypasses any inherited HTTP proxy, so the bearer token is never offered to an intermediary. The transport rechecks authorization before dispatch and rejects hosts that are no longer allowed. HTTPS routing is unchanged. |
| `NVIDIA_API_KEY` | — | Credential for the `nv_build` provider (NVIDIA Build). |
| `OPENAI_API_KEY` | — | Credential for the `openai` provider. |
| `ANTHROPIC_API_KEY` | — | Credential for the `anthropic` provider. |
| `OPENAI_BASE_URL` | `https://api.openai.com/v1` | Alternate OpenAI endpoint; `SKILL_EVAL_LLM_BASE_URL` wins when both are set. |
| `ANTHROPIC_BASE_URL` | Anthropic SDK default | Absolute HTTP(S) Anthropic API root. Accepts an unversioned root or a legacy terminal `/v1`; `SKILL_EVAL_LLM_BASE_URL` wins when both are set. |
| `AWS_REGION` | `us-west-2` | Region for the `bedrock` provider. |
| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, `AWS_PROFILE`, ... | — | Standard AWS credential chain, used as-is by the `bedrock` provider. |
### LLM retries and judge time budget
SkillEvaluator retries transient LLM failures itself. The retry settings apply
on the host to every chat call made through the shared LLM client, including
Tier 1 `--llm` checks, Tier 2 LLM analysis, eval dataset generation, and Tier 3
report suggestions. They also apply in the Harbor verifier to the direct
provider calls made by the `accuracy`, `goal_accuracy`, and `behavior_check`
judges. Embedding requests and the canonical OpenAI RAGAS goal scorer use their
own client and scoring path, so these settings do not apply to them.
| Variable | Default | Effect |
| --- | --- | --- |
| `SKILL_EVAL_LLM_MAX_RETRIES` | `3` | Number of retries after the initial request. `0` disables retries. Non-integer or negative values use the default. |
| `SKILL_EVAL_LLM_RETRY_BASE_DELAY` | `1` second | Base for full-jitter exponential backoff. The wait before retry *n* is a random value from 0 up to `base × 2^(n-1)` seconds, capped at the maximum delay. Non-numeric, non-finite, or negative values use the default. |
| `SKILL_EVAL_LLM_RETRY_MAX_DELAY` | `30` seconds | Longest single wait between attempts. A value below the base delay is raised to the base delay. Non-numeric, non-finite, or negative values use the default. |
| `SKILL_EVAL_LLM_JUDGE_BUDGET_SEC` | `180` seconds | Wall-time budget for each required LLM judge in the Harbor verifier, covering all of its attempts and waits. Non-numeric, non-finite, zero, or negative values use the default. |
Retries cover HTTP 408, 429, and every 5xx response except 501 and 505,
including 520–524 proxy errors and 529 overload responses. Connection errors
and timeouts are also retried, but TLS certificate verification failures are
not: retrying cannot repair an untrusted or invalid certificate. The host LLM
client inspects the underlying HTTP status in wrapped SDK errors; an adapter's
generic 500 or 503 does not hide a terminal failure. In the verifier, Bedrock
throttling and service-unavailable errors are also retried. Other HTTP errors,
such as 400, 401, 403, or 404, are not retried. When a retryable response carries
a `Retry-After` header,
either in seconds or as an HTTP date, the next attempt waits that long plus 0.1
to 0.5 seconds of jitter instead of the computed backoff. An unparseable value
waits the base delay. If `Retry-After` asks for longer than
`SKILL_EVAL_LLM_RETRY_MAX_DELAY`, the call fails immediately with the provider's
error.
In the verifier, `SKILL_EVAL_LLM_JUDGE_BUDGET_SEC` also bounds retries. Each
request's timeout is cut to the judge's remaining time, and a retry whose wait
would pass the deadline ends the judge with a timeout. The RAGAS goal scorer
runs under the same budget. A judge that runs out of time is an infrastructure
failure, not a `0.0` score (see
[Tier 3 judge failure semantics](#tier-3-judge-failure-semantics)). The three
required judges run one after another inside the generated task's 600-second
verifier timeout, so the default leaves time to write a failed-run report.
Set these variables in the host environment that runs `skillevaluator`. When
one is set, Tier 3 stages it into the generated Harbor task's `[verifier.env]`,
so the verifier uses the same value. They are never added to the agent's
environment, and `evals/config.yml` `runtime_env` cannot set `SKILL_EVAL_*`
names.
### Anthropic endpoint roots
When `anthropic` is the selected evaluator provider, endpoint precedence is `SKILL_EVAL_LLM_BASE_URL`, then `ANTHROPIC_BASE_URL`, then the Anthropic default. A root such as `https://gateway.example.com` and a legacy terminal `/v1` root such as `https://gateway.example.com/team/v1` both produce request paths with exactly one `/v1/messages` suffix for the Anthropic SDK, the Tier 3 verifier, and a matching Claude Code agent. An independently credentialed Claude Code route applies the same normalization and validation to `ANTHROPIC_BASE_URL` when the evaluator uses another provider. Routing through path prefixes, ports, IP addresses, internationalized domain names, and ordinary safe percent escapes is preserved.
Anthropic configuration fails before Harbor starts for a full `/v1/messages` endpoint, embedded credentials, a query or fragment, whitespace or control characters, a backslash, an invalid authority, interior empty or dot path segments, or ambiguous multi-layer endpoint encoding. This normalization applies only to Anthropic. OpenAI-compatible endpoints commonly include `/v1`, which remains unchanged.
### Auto-selection
When `SKILL_EVAL_LLM_PROVIDER` is unset, auto-selection only happens when exactly one public credential is exported:
- `NVIDIA_API_KEY` → `nv_build`
- `OPENAI_API_KEY` → `openai`
- `ANTHROPIC_API_KEY` → `anthropic`
If none is set, LLM-backed commands fail with a configuration error. If more than one is set, that is also a configuration error — there is no silent preference order, so set `SKILL_EVAL_LLM_PROVIDER` explicitly when multiple keys are exported.
### SkillSpector bridge
The `--llm` security analysis runs [SkillSpector](https://github.com/NVIDIA/SkillSpector), which has its own provider environment. SkillEvaluator bridges your configured provider into that invocation with an isolated subprocess environment: only the selected provider's settings, the SkillSpector model and deadline settings below, and a small set of basic process variables are forwarded, so ambient credentials for other providers never reach SkillSpector. `SKILLSPECTOR_TEMPERATURE` is not forwarded, because current reasoning models reject a custom temperature. The `nv_build` provider is bridged as `SKILLSPECTOR_PROVIDER=nv_build`, so SkillSpector sizes requests from its NVIDIA Build model registry, while the key travels as `OPENAI_API_KEY` with `OPENAI_BASE_URL` pointing at the public NVIDIA Build endpoint.
| Variable | Default | Effect |
| --- | --- | --- |
| `SKILLSPECTOR_PROVIDER` | bridged from your provider | Overrides the provider SkillSpector uses. Must name a supported public provider (`openai`, `anthropic`, `bedrock`, or `nv_build`) — only that provider's credential variables are forwarded. Any other value fails closed: SkillSpector runs without a credential. |
| `SKILLSPECTOR_MODEL` | `SKILL_EVAL_LLM_MODEL` when set, else the SkillSpector default for your provider | Model SkillSpector uses for its LLM analysis. SkillSpector defaults: OpenAI `gpt-6.1-sol`, Anthropic `claude-opus-5-5`, Bedrock `us.anthropic.claude-opus-5-5`, NVIDIA Build `nvidia/nemotron-3.5-lightning-30b-a3b`, OpenAI-compatible `nvidia/nvidia/nemotron-3-super-120b-long-ctx`. They track the models the pinned SkillSpector release supports, not the chat defaults. |
| `SKILLSPECTOR_MODEL_<SLOT>` | unset | Per-analyzer model override, such as `SKILLSPECTOR_MODEL_META_ANALYZER`. Forwarded as set. |
| `SKILLSPECTOR_MODEL_REGISTRY` | unset | Path to a complete custom SkillSpector model registry, for model IDs the bundled registry does not list. Forwarded as set. |
| `SKILLSPECTOR_REASONING_EFFORT` | `medium` for GPT-6 models, else unset | Reasoning effort for every SkillSpector analyzer. Your value wins. SkillEvaluator sets it only when every SkillSpector model, including `SKILLSPECTOR_MODEL_<SLOT>` overrides, is a GPT-6 model, so never for Claude. |
| `SKILLSPECTOR_MAX_WORKFLOW_SECONDS` | SkillSpector default (1080 in v2.13.3) | SkillSpector's own deadline for one scan, in seconds. Forwarded as set to both the static and the LLM stage. When it expires, SkillSpector reports partial analysis and the security scan is `INCOMPLETE`. Keep it below `SKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDS`. |
## Embeddings
Tier 2 semantic checks use an OpenAI-compatible embeddings API. Workflow context lives in [Providers & Credentials](/skills/skillevaluator/configuration#embeddings) and [Tier 2: Deduplication](/skills/skillevaluator/tier2-deduplication).
| Variable | Default | Effect |
| --- | --- | --- |
| `SKILL_EVAL_EMBEDDING_PROVIDER` | falls back to `SKILL_EVAL_LLM_PROVIDER` | Embedding provider: `openai`, `nv_build`, or `openai-compatible`. `anthropic` and `bedrock` are rejected — they provide no embeddings API. |
| `SKILL_EVAL_EMBEDDING_MODEL` | OpenAI: `text-embedding-3-small`; NVIDIA Build: `nvidia/nemotron-3-embed-1b`; OpenAI-compatible: `nvidia/nvidia/nemotron-3-embed-1b` | Optional embedding model override using the exact provider catalog ID. |
| `SKILL_EVAL_EMBEDDING_BASE_URL` | provider default endpoint | Embedding endpoint override; for `openai-compatible` it falls back to `SKILL_EVAL_LLM_BASE_URL`. Ignored by `nv_build`, whose endpoint is fixed to `https://integrate.api.nvidia.com/v1` — use `openai-compatible` for a custom endpoint. |
| `SKILL_EVAL_EMBEDDING_API_KEY` | falls back to `SKILL_EVAL_LLM_API_KEY` | API key for `openai-compatible` embeddings. `openai` and `nv_build` use their own provider keys. |
## Validation (Tier 1)
These variables tune the static and security checks. The check-by-check guide is [Tier 1: Validation](/skills/skillevaluator/tier1-validation).
| Variable | Default | Effect |
| --- | --- | --- |
| `SKILLEVALUATOR_PROFILE` | `default` | Validation policy when no `--profile` flag is passed: `default` or `registry`. `external` remains a compatibility alias for `default`. The CLI flag always wins. |
| `SKILLEVALUATOR_BANDIT_PATH` | auto-discovered | Absolute path to the Bandit executable. |
| `SKILLEVALUATOR_SEMGREP_PATH` | auto-discovered | Absolute path to the Semgrep executable. |
| `SKILLEVALUATOR_SKILLSPECTOR_PATH` | auto-discovered | Absolute path to the SkillSpector executable. |
| `SKILLEVALUATOR_SKILLSPECTOR_TIMEOUT_SECONDS` | `1200` | Watchdog for each SkillSpector run, in seconds. The static stage and, with `--llm`, the LLM stage each get the full value. A run that exceeds it is stopped and the security scan is `INCOMPLETE`. Set a positive integer; values above `86400` are capped, and other values fall back to the default with a warning. Keep it above `SKILLSPECTOR_MAX_WORKFLOW_SECONDS` so SkillSpector's own deadline ends a slow scan first. The effective value is recorded as `skillspector_timeout_seconds` in the security result. |
| `SKILLEVALUATOR_RUBRIC_SUPPLEMENT_BYTES` | `1048576` (1 MiB) | Total bytes of supplementary content `rubric-eval` collects for the judge: first the complete linked Markdown that `SKILL.md` requires, then bounded optional previews of `scripts/` and `references/`. Required linked guidance is never truncated — a document that does not fit the remaining budget fails the evaluation with an `unsafe_skill_input` finding instead of reaching the judge in part. Set a positive integer; values above `8388608` (8 MiB) are capped, and other values fall back to the default with a warning. The budget bounds how many bytes are read, never which paths: every reference is still confined to the skill's own declared root. |
| `SKILLEVALUATOR_SCHEMA_ALLOWED_DIRS` | — | Extra allowed skill-root directory names (comma- or whitespace-separated), e.g. `data,fixtures`. Additive only — names extend the built-in allowlist and never replace it. |
| `SKILLEVALUATOR_SUBMITTER` | auto-detected | Explicit submitter identity for the home-path PII check. It is combined with `GITHUB_ACTOR`, `USER`, `LOGNAME`, `USERNAME`, the OS login name, and the skill's declared author — every identity that resolves is protected. If none resolves, the home-path check is disabled with a warning. |
| `SKILLEVALUATOR_PREVIOUS_VERSION` | — | The previously published skill version that the default version check compares against. Without this value or `--previous-version`, the check validates the current version's format but does not enforce an increase. |
<Warning>
The scanner path overrides fail closed: a value that is not an absolute, executable file is treated as a configuration error rather than silently falling back to another same-named binary on `PATH`. This prevents a planted executable from hijacking a security scan.
</Warning>
## Live evaluation (Tier 3)
These variables control where results land and how Harbor confines local mode.
The run workflow is [Tier 3: Live Agent Evaluation](/skills/skillevaluator/tier3-live-evaluation);
execution environments are covered in
[Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes).
| Variable | Default | Effect |
| --- | --- | --- |
| `SKILLEVALUATOR_RESULTS_DIR` | `<skill>/evals/results` | External root for run results. Precedence for writes: the `--results-dir` flag, then this variable, then the canonical in-skill location. Configured roots must remain outside the skill; `evals/results` is the only supported in-skill results root so generated output cannot change the skill's publication identity. Read commands (`view`, `compare`) honor the same order and also fall back to the legacy location so older runs stay visible. |
| `SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE` | macOS: `~/Library/Application Support/skillevaluator/output-provenance.key`; Linux: `$XDG_STATE_HOME/skillevaluator/output-provenance.key` (`~/.local/state` when `XDG_STATE_HOME` is unset); Windows: `%LOCALAPPDATA%\skillevaluator\output-provenance.key` | Local key that authenticates the markers SkillEvaluator writes into its generated output directories; a directory is replaced only when its marker verifies for that path. The key is created on first use. It must be outside every evaluated skill and generated output tree. On macOS and Linux, the file and its directory must be owned by the current user and not accessible to group or other users. |
| `SKILLEVALUATOR_LOCAL_SANDBOX` | `require` | Local-mode sandbox policy. `require` fails closed when no OS sandbox backend (Bubblewrap on Linux, Seatbelt on macOS) is usable; `prefer` degrades to advisory-only guardrails with a loud warning; `off` skips sandbox probing entirely — for skills you fully trust. Any other value is rejected. No value enables native Windows: local mode fails closed there before anything runs — use WSL2 or `--env-mode docker`. |
| `SKILL_EVAL_NEMO_GYM_BIN` | `gym` on `PATH` | Path to the compatible pinned Gym executable if it is not on `PATH`. Public `nemo-gym==0.7.0` is incompatible. See [NeMo Gym integration](/skills/skillevaluator/nemo-gym) for installation, tested agents and remaining qualification limits. |
| `SKILL_EVAL_NEMO_GYM_MODEL_URL` | provider base URL | OpenAI-compatible base URL (ending in `/v1`) of the policy model for `--backend nemo-gym`. When set, also choose a model it serves with `--model`, `--agent-model` or `harbor.agents.<agent>.model`. |
| `SKILL_EVAL_NEMO_GYM_MODEL_API_KEY` | provider key, only when the URL is also the provider's | API key for that policy model. Required when `SKILL_EVAL_NEMO_GYM_MODEL_URL` is set; use `EMPTY` for a keyless server. Passed to Gym through the environment, never on the command line. |
| `SKILL_EVAL_NEMO_GYM_HOST` | — | Address of this host that the Gym sandbox can reach (for example the Docker bridge gateway). Needed when the host name resolves to a loopback address. |
| `SKILL_EVAL_NEMO_GYM_SANDBOX` | `docker` | NeMo Gym sandbox provider for `--backend nemo-gym`. |
| `SKILL_EVAL_NEMO_GYM_TIMEOUT_S` | `21600` | Wall-clock limit, in seconds, for one `gym eval run` (one agent and arm); must be a finite number greater than 0. With Vertex AI ADC credentials it is also capped to the token's lifetime. On timeout, Ctrl-C, SIGTERM or SIGHUP, Gym's process group gets SIGTERM and 30 seconds to clean up its servers and sandboxes, then SIGKILL; a second Ctrl-C skips the wait. If the group still cannot be confirmed stopped after one last SIGKILL, no further arm starts. After a timeout, the run then finishes as failed with that error and keeps the scores of the arms that finished; after an interruption, the progress output reports it. |
| `SKILLEVALUATOR_LOCAL_ALLOW_NET` | `true` | Network egress for local-mode trials. Set to `0` to airgap a skill that must not reach the network. Incompatible with the `nv_build` provider — NVIDIA Build local agents require network access, so airgapped runs are rejected up front. |
| `SKILLEVALUATOR_LOCAL_STRICT_READS` | `false` | Tightens the sandbox's read-only view of the host system to a stricter path set. |
| `SKILLEVALUATOR_LOCAL_INHERIT_AGENT_KEYS` | `false` | When true, local-mode trials inherit the host's agent credential variables into their ambient environment. Off by default so a hostile skill command cannot read credentials — agents and verifiers receive credentials per-exec instead. |
| `SKILLEVALUATOR_RUNTIME_DIR` | `~/.local/share/skillevaluator/runtimes` | Where local mode looks for managed agent CLIs. Must be a dedicated subdirectory — not the home directory or one of its parents. |
| `SKILLEVALUATOR_DISABLE_CUSTOM_BACKENDS` | unset (custom backends enabled) | Set to `1`, `true`, or `yes` (case-insensitive, with surrounding whitespace ignored) to hide every installed custom execution backend and reject its mode. Other values leave discovery enabled. Built-in modes remain available. See [Custom execution backends](/skills/skillevaluator/agents-and-sandboxes#custom-execution-backends). |
| `SKILLEVALUATOR_ALLOWED_MCP_HOSTS` | unset | Comma-separated hostnames approved for MCP endpoints receiving environment-variable references or authentication headers from skill-authored MCP declarations. Matching uses the whole hostname, case-insensitively; supply hostnames, not URLs or wildcard patterns. |
| `SKILLEVALUATOR_ALLOWED_MCP_SECRETS` | unset | Comma-separated, case-sensitive environment-variable names approved for references such as `${DEDICATED_MCP_TOKEN}` in skill-authored MCP declarations and `harbor.runtime_env`. Export the dedicated non-LLM credential on the host; literal secret values remain prohibited. Remote declarations also require the endpoint in `SKILLEVALUATOR_ALLOWED_MCP_HOSTS`. This allowlist cannot authorize operator-owned agent or provider credentials. |
| `SKILL_EVAL_HARBOR_PREBUILD_TASK_ENVS` | unset | Set to exactly `1` to opt in to pre-building Harbor task environment images before trials start (other truthy spellings are not honored). Opt-in because concurrent trial cleanup can remove a shared prebuilt image that sibling trials still need. |
| `SKILL_EVAL_JUDGE_MODEL` | falls back to `SKILL_EVAL_LLM_MODEL` | Operator-controlled model override for the Tier 3 dimension judge in standard grading. `LLM_JUDGE_MODEL` is honored as a higher-precedence spelling; when neither is set, the judge uses `SKILL_EVAL_LLM_MODEL`, then the built-in judge default. Both spellings change only the model used with the configured `SKILL_EVAL_LLM_PROVIDER`; they do not select a separate judge provider, so the model must be supported by that provider. Standard grading makes both spellings resolve to the selected override in Harbor's parent process, then forwards them through its verifier-only job layer, which takes precedence over task and step verifier declarations. Neither name is injected into the evaluated agent's initial environment; attempts to set or reference one from skill-authored `runtime_env` or native `[environment.env]` tables are rejected. Native custom-only graders keep ownership of their verifier configuration. Shared agent/verifier containers are a process boundary, not a confidentiality boundary; use a separate verifier environment when the model name itself must be hidden from untrusted agent processes. `run_config.json` records the configured judge provider, model, source, `override_applied`, reasoning effort, and catalog-verification status separately from agent models; an allowed provider fallback may use a different model for an individual judge call. When `override_applied` is false for a native Harbor task, `effective_model_source` is `native_harbor_runtime` and catalog verification is explicitly inconclusive because an authored task or step verifier declaration may supersede the recorded fallback. |
| `LLM_JUDGE_FALLBACK_MODELS` | unset | Comma-separated fallback models for the Tier 3 standard judges in the Harbor verifier. When the provider rejects the judge model as not found, invalid, or not accessible to the key, the verifier tries each fallback in order with the same provider. Fallback applies to the `openai`, `nv_build`, and `openai-compatible` providers. A safety refusal or an empty reply from the judge model also moves to the next fallback, on every provider. The value is forwarded only to standard verifier jobs, never to the agent environment or `custom_only` runs. |
| `SKILL_EVAL_LLM_REFUSAL_FALLBACK_MODEL` | `claude-opus-4-8` | Model named as the server-side refusal fallback on requests to Claude 5.5 or later models on the native Claude API; on Bedrock, the Tier 3 judge's refusal retry uses it. On Bedrock, a bare name such as `claude-opus-4-8` gets the judge model's geography prefix (`us.anthropic.`, `global.anthropic.`), and a full Bedrock ID such as `global.anthropic.claude-opus-4-8` is used as given. Set `none`, `off`, `disabled`, `false` or `0` to send no fallback, for example when the API answers HTTP 400 `is not a valid fallback target` for your model. An empty value counts as unset, and a fallback equal to the request model is never sent. Forwarded to standard verifier jobs like `LLM_JUDGE_FALLBACK_MODELS`. The reusable SkillEvaluator CI workflows do not take this variable; their provider-backed jobs keep the default. |
| `LLM_JUDGE_FALLBACK_POLICY` | `append` | Opt-in limit on which models may answer a Tier 3 standard judge. `append` keeps the built-in refusal fallback, which sends a refused Claude 5.5 or later judge to `SKILL_EVAL_LLM_REFUSAL_FALLBACK_MODEL` (default `claude-opus-4-8`, off with `none`, never sent when it equals the judge model). `replace` turns it off, so only the judge model and the `LLM_JUDGE_FALLBACK_MODELS` entries answer. Values are case-insensitive; any other value stops a run with standard grading before trials start, and the verifier fails every standard judge on it before a request is sent. Forwarded like `LLM_JUDGE_FALLBACK_MODELS`, and recorded as `judge.fallback_policy` in `run_config.json` when set. See [Judge fallback and retry policies](/skills/skillevaluator/tier3-grading#judge-fallback-and-retry-policies). |
| `LLM_JUDGE_RETRY_POLICY` | `legacy` | Opt-in retry policy for the Tier 3 standard judges. `strict` tries at most three models (the judge model, then the first two `LLM_JUDGE_FALLBACK_MODELS` entries) with at most two requests each within the per-judge time budget, moves on only when a model is unavailable, ends the judge on a refusal or a quota, budget, safety, or security error, and scores only a first completed reply that is exactly one JSON object. `legacy` keeps the standard retries, fallbacks, format retry, and salvage. Values are case-insensitive; any other value stops a run with standard grading before trials start, and the verifier fails every standard judge on it before a request is sent. Forwarded like `LLM_JUDGE_FALLBACK_MODELS`, and recorded as `judge.retry_policy` in `run_config.json` when set. See [Judge fallback and retry policies](/skills/skillevaluator/tier3-grading#judge-fallback-and-retry-policies). |
| `SKILLEVALUATOR_HARBOR_DIAGNOSTIC_ARTIFACT_MAX_BYTES` | `5242880` (5 MiB) | Size cap per diagnostic artifact copied from a Harbor trial into the results directory. Values are bounded to 0–67108864 bytes (64 MiB); `0` skips non-empty artifacts. Oversized artifacts are skipped and recorded in the artifact manifest. |
| `SKILLEVALUATOR_GKE_CLUSTER`, `SKILLEVALUATOR_GKE_REGION`, `SKILLEVALUATOR_GKE_NAMESPACE`, `SKILLEVALUATOR_GKE_REGISTRY_LOCATION`, `SKILLEVALUATOR_GKE_REGISTRY_NAME` | — | Target cluster, region, namespace, and Artifact Registry coordinates for `--env-mode gke` when not supplied via `--ek`. |
| `SKILLEVALUATOR_GKE_ALLOW_WORKLOAD_IDENTITY` | `false` | Opt in (`1`, `true`, `yes`) to single-pod GKE Workload Identity (`CLAUDE_CODE_USE_VERTEX=1` or a GCP-bound Kubernetes ServiceAccount). Restricted to trusted skills because the pod ServiceAccount and GCE/GKE metadata server are shared with evaluated skill commands. |
| `SKILLEVALUATOR_GKE_METADATA_PROBE_IMAGE` | `python:3.12-slim` | Trusted container image used by `skillevaluator-metadata-probe-init` and `skillevaluator-metadata-probe` in unprivileged `--env-mode gke` pods to verify that `169.254.169.254` and `169.254.169.252:988` are unreachable. Override when your GKE cluster restricts Docker Hub pulls and requires an internal Artifact Registry mirror. |
| `SKILLEVALUATOR_GKE_AUTOPILOT` | auto-detected | Explicitly set `1`/`true` or `0`/`false` (or pass `--ek autopilot=<bool>`) to control GKE Autopilot ephemeral-storage adjustment (`10Gi` → `10176Mi` on non-accelerator `main` containers so adding the `64Mi` probe container stays within Autopilot's `10Gi` general-purpose pod ceiling) when node listing or `gcloud` inspection is unavailable. |
| `SKILLEVALUATOR_VERTEX_ADC_JOB_TIMEOUT_SEC` | `3300` seconds | Maximum wall-clock timeout per Harbor subprocess invocation when using short-lived Google Application Default Credentials (ADC) with a Vertex AI OpenAPI endpoint, capped to the remaining token lifetime minus a 60-second safety margin. |
### Harbor process environment
SkillEvaluator starts every Harbor process in an empty, evaluator-owned working
directory with `.env` loading and Harbor telemetry disabled:
- Harbor does not read `.env` or `.env.local` files from your project or
current directory. Export variables in the shell or CI job that runs
`skillevaluator` instead.
- An evaluation run passes Harbor only an allowlisted environment: basic
process, locale, and temporary-directory variables such as `PATH`, `HOME`,
`LANG`, and `TMPDIR`; proxy and CA-bundle variables; the selected backend's
own variables (see [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes)); the
evaluator provider's settings and credential, plus the `AWS_*` chain for
`bedrock`; each selected agent's credentials; validated `runtime_env`
values; and the Tier 3 judge, retry, and evidence-budget variables on this
page. Other variables in your shell do not reach Harbor.
- Relative paths in allowlisted path variables, such as `KUBECONFIG`,
`DOCKER_CONFIG`, `SSL_CERT_FILE`, or `AWS_SHARED_CREDENTIALS_FILE`, resolve
against the directory where you ran `skillevaluator`.
- On Windows, `PROGRAMFILES`, `PROGRAMW6432`, and `PROGRAMDATA` reach Harbor
so Docker Desktop can discover Compose. `PYTHONUTF8` and
`PYTHONIOENCODING` are also forwarded when set; use `PYTHONUTF8=1` and
`PYTHONIOENCODING=utf-8` for a non-UTF-8 locale. See the
[PowerShell setup](/skills/skillevaluator/agents-and-sandboxes#windows-hosts).
- Setting `HARBOR_TELEMETRY` or `PYTHON_DOTENV_DISABLED` yourself has no
effect; SkillEvaluator always overrides both.
- Tier 3 needs python-dotenv 1.2.0 or newer, which the `tier3` extra installs.
With an older version installed, the prerequisite check fails before Harbor
starts.
### Judge evidence budgets
`SKILL_EVAL_JUDGE_CHARACTER_FACTS=1` optionally appends Unicode character metadata
to standard Tier 3 judge prompts. It is off by default; `true`, `yes`, and `on`
also enable it (case-insensitive). This operator-owned setting is forwarded only
to standard verifier jobs, never to agent environments or `custom_only` jobs.
When configured, its value is recorded as `judge.character_facts` in
`run_config.json`. See [character-sensitive judging](/skills/skillevaluator/tier3-grading#character-sensitive-judging)
for scope, limits, and the effect on the goal-accuracy route.
Before the Harbor verifier calls the `accuracy`, `goal_accuracy`, and
`behavior_check` judges, it condenses the agent's trajectory into a separate
evidence excerpt for each judge. These budgets cap each excerpt in characters.
When a run produces more evidence than fits, lower-priority material is
dropped or truncated first so the judge still sees the agent's final response.
Larger budgets give judges more context and cost more prompt tokens.
| Variable | Default | Effect |
| --- | --- | --- |
| `SKILL_EVAL_ACCURACY_BUDGET` | `8000` characters | Evidence budget for the `accuracy` judge: the agent's final response, files it wrote, and up to 6 recent tool results. |
| `SKILL_EVAL_GOAL_ACCURACY_BUDGET` | `12000` characters | Evidence budget for the `goal_accuracy` judge, including the RAGAS goal scorer: the final response, end-state file changes, and up to 12 recent tool results. |
| `SKILL_EVAL_BEHAVIOR_CHECK_BUDGET` | `12000` characters | Evidence budget for the `behavior_check` judge: file changes, the final response, the user request, and a compact tool history. A value you set is used exactly as set, up or down. Only the default is reconciled upwards, to `SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT` plus 4000, so raising that limit alone still leaves room for the tool history. |
| `SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT` | `8000` characters | Maximum length of the agent's final response within the `behavior_check` evidence. It also caps the reserved USER REQUEST section, which is otherwise 2000 characters. Raising it above 8000 also raises the *default* behavior-check budget, but never a budget you set yourself. |
Setting `SKILL_EVAL_BEHAVIOR_CHECK_BUDGET` below `SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT`
plus 1600 is honoured, but logs a warning: the final response will not fit
beside any tool history, and `behavior_check` then reports incomplete evidence
instead of a score (see
[behavior evidence completeness](/skills/skillevaluator/tier3-grading#evidence-budgets)).
Lower the final-response limit alongside the budget to cap the judge prompt
without losing the answer.
Each value must be an integer. Values below `1` are treated as `1`, and
non-integer values such as `8k` or `1.5` use the default; neither is a budget
you set, so `SKILL_EVAL_BEHAVIOR_CHECK_BUDGET` falls back to its reconciled
default in both cases. Set these variables in
the host environment that runs `skillevaluator`. As with the
[retry settings](#llm-retries-and-judge-time-budget), Tier 3 stages any that
are set into the generated task's `[verifier.env]`, and never into the agent's
environment.
### Tier 3 judge failure semantics
In standard grading, a required judge transport, authentication, timeout, HTTP, or exhausted response-parsing error is an evaluation infrastructure failure, not a numeric `0.0`. A condition containing a failed or incomplete default trial publishes none of its scores, dimensions, pass@k, or quality findings, and no cross-condition lift is published. Redacted diagnostics and coverage information remain available. A genuine model score of `0.0` remains valid and is published normally.
`default_plus_custom` cannot use a custom score to rescue failed standard judging. `custom_only` is unchanged and does not invoke the standard judge.
## In-container grader contract
The `HARBOR_*` variables are read by your custom grader *inside* the task container — you never set them yourself, and each has a fixed in-container default. The full reward contract, with defaults and examples, lives in [Custom Graders & Tasks](/skills/skillevaluator/custom-graders).
| Variable | What it carries |
| --- | --- |
| `HARBOR_REWARD_JSON` | Path where your grader writes its `reward.json` score. |
| `HARBOR_REWARD_TXT` | Path for the compatibility `reward.txt` dual-write. |
| `HARBOR_ENTRY_JSON` | Path to the eval case entry being graded. |
| `HARBOR_ATIF_PATH` | Path to the agent trajectory to grade, serialized in [Agent Trajectory Interchange Format (ATIF)](https://www.harborframework.com/docs/agents/trajectory-format). |
| `HARBOR_LOGS_DIR` | Root of the in-container log tree; the agent trajectory lives under its `agent/` subdirectory. |
| `HARBOR_TESTS_DIR` | Directory holding the staged grader and test assets. |
| `HARBOR_GRADER` | Path to the staged `grader.py`. |
## Set at runtime, not committed
<Warning>
Keep credentials out of skill repositories and CI configuration files. Export provider keys as masked secrets at runtime. Agent credentials for Tier 3 come only from the host environment — `evals/config.yml` `runtime_env` carries non-credential values, and any entry that names or references an operator-owned credential (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `NVIDIA_API_KEY`, their base-URL variables, `CLAUDE_CODE_USE_BEDROCK`, and the `AWS_*` credential variables) is rejected at run time, as are names that control the host process (`PATH`, `DOCKER_*`, proxy variables, `SKILL_EVAL_*`, `SKILLEVALUATOR_*`, and similar). See the blocked-names table in [Providers & Credentials](/skills/skillevaluator/configuration#tier-3-credentials-two-roles). With NVIDIA Build, one exported `NVIDIA_API_KEY` covers both the evaluator and the agents in the `docker`, `kata`, and `local` env-modes.
</Warning>
## See also
- [Providers & Credentials](/skills/skillevaluator/configuration) — the setup walkthrough behind the provider variables
- [CLI Reference](/skills/skillevaluator/cli-reference) — the per-run flags that override many of these variables
- [Gate Your CI](/skills/skillevaluator/ci-integration) — supplying these variables as masked CI secrets
## GitLab all-tier admission
| Variable | Purpose |
| --- | --- |
| `SKILLEVALUATOR_GITLAB_READ_TOKEN` | Optional project Reporter/read_api credential for current-MR and default-branch admission; required on GitLab before 18.4, or when `CI_JOB_TOKEN` cannot read the default branch, scoped to the execution environment and removed before provider subprocesses. |
| `SKILLEVALUATOR_GITLAB_TOKEN` | Separate project-scoped publisher credential in the protected feedback environment; never used by evaluation jobs. |
Local onboarding can use `SKILLEVALUATOR_GITLAB_TOKEN` or `GITLAB_TOKEN` in its
own process. Keep that user credential local; the CI publisher uses its separate
project-scoped credential. The [GitLab setup guide](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/all-tiers/README.md#gitlab-checks-and-feedback)
describes version detection, variable scopes and the two-file consumer.