Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(tier3): support Claude Code on Agent Platform and add active preflight probe (#140)
Narendran Raghavan39df2948d99e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5f8aebec..e2412f69 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -22,10 +22,541 @@ All notable changes to SkillEvaluator are documented in this file.operator/provider credential references and literal secrets, while allowingoperator-approved MCP endpoints and non-LLM secrets via`SKILLEVALUATOR_ALLOWED_MCP_HOSTS` and `SKILLEVALUATOR_ALLOWED_MCP_SECRETS`.+- The HTML report's Tier 1 card now shows the SkillSpector version that ran+ the security scan (for example `SkillSpector v2.12.0`), and the embedded+ report JSON records it as `skillspector_versions`. Findings depend on the+ scanner release, so reviewers can tell which SkillSpector produced them+ without rerunning the scan. The version is shown even when the report omits+ its timestamp.+- `--env-mode kata` runs Tier 3 trials through the hardened Docker backend with+ every task service in a Kata Containers microVM. It requires a Linux host with+ `/dev/kvm` and a Kata runtime registered with Docker, and accepts only the+ `kata_runtime` and `kata_dns` environment kwargs. Image builds still run on+ the host, and tasks with Compose sidecars are rejected.+- `--env-mode runta` and `--env-mode mosaic` run Tier 3 trials on Harbor's Runta+ and Mosaic backends. Runta's `mode` and Mosaic's `volume`, `persist`,+ `enable_ssh`, `build_args`, and `build_target` kwargs are reserved because they+ bypass the task image or let state and access outlive one isolated trial.++### Changed++- SkillSpector now has its own default model per provider instead of+ following the chat default, so chat-default upgrades no longer reach+ SkillSpector before it supports them. An explicit `SKILL_EVAL_LLM_MODEL`+ still reaches it, and `SKILLSPECTOR_MODEL` still overrides both. The bridge+ forwards your `SKILLSPECTOR_MODEL_<SLOT>`, `SKILLSPECTOR_MODEL_REGISTRY` and+ `SKILLSPECTOR_REASONING_EFFORT` settings (never `SKILLSPECTOR_TEMPERATURE`),+ defaults the reasoning effort to `medium` for GPT-6 models, and records the+ requested `skillspector_model` and `skillspector_reasoning_effort` next to+ `skillspector_version` in the security result.+- Install instructions and the missing-scanner hint now pin SkillSpector to+ `v2.12.0` instead of its default branch.+- GPT-6 models get the GPT-5 request shape everywhere SkillEvaluator calls a+ model: no `temperature`, and `max_completion_tokens` on the native OpenAI+ endpoint. Gateway catalog IDs such as `aws/anthropic/bedrock-claude-opus-5`+ are recognized as the Claude model they name.+- Output-token limits are sized per model family. OpenAI reasoning models get+ at least 32,768 tokens; other models get 16,384 for `--llm-verify`, rubric+ evaluation, deduplication verdicts and judges, 8,192 for report suggestions+ and 16,000 for dataset generation. On the Anthropic API, Claude Opus 4 and+ 4.1 stay at the 8,192 tokens the Anthropic SDK allows without streaming.+- Reasoning effort is sent only to GPT-6 models and to Claude 5.5 or later on+ the Anthropic API: `high` for `--llm-verify`, `medium` for rubric evaluation,+ deduplication verdicts and judges.+- Rubric evaluation and deduplication verdicts request structured output. The+ verifier and deduplication prompts ask for `rationale` instead of+ `reasoning`; replies that still say `reasoning` are accepted.+- The `goal_accuracy` RAGAS scorer is skipped for GPT-5 and GPT-6 judges, which+ reject its sampling settings.+- `run_config.json` records the judge reasoning effort, and each standard+ judge's details record `judge_model_used`, `judge_reasoning_effort` and+ `judge_fallback_used`. Skill lift leaves out a case whose two conditions were+ judged by different models and counts it in `judge_mixed_cases`.+- Removed the unused `CONTENT_DEDUP_LLM_*` constants and `LLM_VERIFY_MODEL`; an+ LLM client built from explicit credentials uses the provider default model.+- Tier 3 pins the agent CLIs Harbor installs: Codex 0.161.0 and Claude Code+ 2.1.292, instead of the latest release. Claude Code runs Claude 5.5 and later+ models with `reasoning_effort=medium`; Codex keeps `high`. Local mode keeps+ your installed CLIs and fails a trial early when they are too old for the+ model: Claude 5.5 needs Claude Code 2.1.280 or newer, and GPT-6 needs Codex+ 0.159.1 or newer.+- OpenAI-compatible gateway agent defaults: Codex moves to+ `openai/openai/gpt-6.1-sol` and Claude Code to+ `aws/anthropic/bedrock-claude-opus-5-5`. OpenCode stays on+ `nvidia/nvidia/nemotron-3-super-120b-long-ctx`, no longer tied to the gateway+ chat default. Claude Code on the shared gateway key runs with+ `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1`, so gateways that forward to+ another provider do not reject its pre-release request fields.+- The OpenAI default model moves from `gpt-5.6-sol` to `gpt-6.1-sol` for chat,+ judging, Codex and SkillSpector, which runs it at `medium` reasoning effort,+ so the effort values above apply to the OpenAI defaults. OpenCode keeps+ `openai/gpt-5.6-sol` unless `SKILL_EVAL_LLM_MODEL` is set, because+ GPT-6.1 Sol supports tool calls only on the Responses API.+- The Anthropic and Bedrock default models move from Claude Opus 5 to Claude+ Opus 5.5 (`claude-opus-5-5` and `us.anthropic.claude-opus-5-5`) for chat,+ judging and Claude Code. On the Claude API they get the effort values above+ and server-side refusal fallback, and Claude Code runs them at+ `reasoning_effort=medium`.+ SkillSpector stays on `claude-opus-5` and `us.anthropic.claude-opus-5`, which+ SkillSpector v2.12.0 supports.+- Docker Claude Code and Codex containers no longer get the distribution+ `nodejs` and `npm` packages from agent setup, except on Alpine images. On+ Debian-based images, including the default eval image, those packages also+ provided `npx`, `gcc`, `g++`, and `make`. Claude Code containers now have no+ Node.js unless the image provides it, and Codex's own Node.js is on `PATH`+ only for the Codex process. Skills whose scripts, tasks, or stdio MCP servers+ need these tools must install them through `scripts/apt-packages.txt` or a+ custom `evals/environment/Dockerfile`.+- Exposed `kata`, `runta`, and `mosaic` as the 24th, 25th, and 26th Harbor 0.24+ backends alongside local mode. `cua-cloud`, `opensandbox`, `hf-sandbox`,+ `podman`, `prime`, and `smol` remain disabled.+- The public benchmark gate (`scripts/ci/check_public_benchmarks.py`) now+ requires a `PASS` card to record its `Source digest` and `Tier 3 run ID`.+ `PASS` cards rendered by earlier releases fail the scan until they are+ regenerated; earlier `FAIL`, `NEUTRAL`, and `INCOMPLETE` cards without those+ lines are still accepted, and the lines are validated whenever present.+- Report output must be outside the publication target. An explicit+ `--output-dir` inside the evaluated skill is now rejected; use an external+ directory. A default in-target `reports/` location relocates to an+ authenticated sibling `<skill>-reports` (or `<catalog>-reports`), refuses+ unowned collisions, and is excluded from Tier 3 runtime staging. Report roots+ inside the target repository or an included-skill root are excluded from+ linked or full-repository context only after they are reserved or+ authenticated as generated output, so an authored subtree is never silently+ omitted.+- A custom Tier 3 `--results-dir` inside the evaluated skill is now rejected;+ use the canonical `evals/results` path or an external results root.+- `validate --block-on-agent-eval` now passes the exit gate only for a completed+ Tier 3 `PASS` verdict with complete attempt coverage. `NEUTRAL` and `FAIL`+ verdicts exit 1; previously any completed, scored run passed.++### Fixed++- Tier 3 now uses one versioned, equal-weight five-dimension overall-score+ policy across Harbor rewards, pass@k, Skill Lift, reports, comparisons, and+ best-agent selection. Previously, some paths averaged all six evaluators+ directly while reports averaged five dimensions, which gave Effectiveness+ two votes and could reverse the reported lift direction. Current artifacts+ persist `skill-evaluator-dimension-mean-v1`; legacy and partial artifacts+ retain their historical semantics. Conditions keep their own recorded metric+ sets; overall lift and best-agent rankings are omitted across different+ standard metric sets while individual scores are preserved, and+ `tier3 compare` refuses standard agents with different policies or metric+ sets. Custom-only agents keep their grader's own overall and are ranked and+ compared against standard agents+ (Relates to [#61](https://github.com/NVIDIA/SkillEvaluator/issues/61)).+- Preserve explicit Claude and OpenCode read failure metadata in reconstructed+ ATIF trajectories. Failed isolated SKILL.md reads and failed native Skill+ calls lose credit; compound shell failures and shell reads whose path may+ expand to several files (braces, globs, `$` or backticks) remain ambiguous+ and retain prior credit in host and verifier grading. Unattributed errors do+ not cancel successful reads, and shell reads count dash-prefixed filenames+ after `--`.+- Tier 1 pyproject dependency audits now inspect static target declarations+ rather than the evaluator environment. Exact direct pins (including optional+ groups) are queried without installation or build-script execution;+ unresolved declarations and invalid scanner evidence are incomplete.+- Tier 1 PII scanning now includes `.mdc` rules and workflows.+- Tier 3 negative-case accuracy excludes inapplicable skill identification and+ records its scoring policy. Reports separately expose measured token and+ agent execution-time differences with complete matching-arm coverage, while+ efficiency continues to measure routing and productive tool use.+- Derive accuracy scores from complete criterion verdicts in both judge+ runtimes (passing applicable criteria divided by applicable criteria), so an+ inconsistent model-supplied total cannot distort Skill Lift. Correct the+ Harbor prompt example to score five passing criteria as 1.0.+- Include schema-valid ATIF message content parts in final answers, conversation+ summaries, and judge evidence references in the host and Harbor verifier.+ Image and audio parts use markers without exposing their source paths.+- Preserve scored Harbor trials when agent answers mention authentication errors+ and per-step token usage is present but aggregate ATIF token metrics are absent+ or zero. Explicit runtime exceptions and error events still invalidate trials.+- Safety-classifier refusals, content filters and output-token truncation are+ detected instead of being read as empty or malformed output. `--llm-verify`+ and rubric evaluation report the cause, and `--llm-verify` keeps the+ complete verdicts of a truncated reply. Claude 5.5 or later requests to the+ native Claude API use server-side fallback to Claude Opus 4.8. In the Harbor+ verifier, Bedrock retries a filtered Claude 5.5 reply on Claude Opus 4.8, and+ a refusal or empty judge reply moves on to the next+ `LLM_JUDGE_FALLBACK_MODELS` entry.+- The Harbor verifier's last-resort judge model follows the selected provider.+ With no judge or chat model configured, it used to request an OpenAI model+ from Anthropic, Bedrock, NVIDIA Build and gateways. The unused host-side+ `llm_judge.DEFAULT_JUDGE_MODEL` constant is removed.+- Fixed Docker Codex and Claude Code setup wasting its timeout installing+ distribution Node/npm before Harbor's own installers. Native, gateway Codex,+ and NVIDIA Build agents now provision Node/npm only on the installer branches+ that need them, while preserving installed-agent and version-pin checks.+- Preserved multiline skill descriptions when saving and loading Tier 2+ embedding catalogs. Descriptions can contain line breaks and tabs, while+ catalog identifiers and other control characters remain strictly validated.+- Brought the published documentation in line with 0.5.0. It now covers how+ standard grading builds judge evidence and rebuilds trajectories from+ OpenCode and Codex logs, structured judge output and retries, the evidence+ budget and judge time-budget variables, and the Harbor process environment.+ The CLI reference adds `validate --workers`, the full `--env-mode` list, and+ the exit codes the CLI actually returns (`0`, `1`, `2`).+- `BENCHMARK.md` publication verdicts now require completed Tier 1 and Tier 2+ execution evidence by default (Tier 1 must cover every default check that+ applies to the content type, so a run narrowed with `--checks` cannot+ certify publication), conservatively resolve conflicting peer+ policy and Tier 3 result metadata, bind every tier and policy claim to one+ versioned source-tree digest, preserve the Tier 3 run ID, persist an explicit+ publication status in JSON and HTML, and reject+ publication `PASS` cards whose decision evidence is missing, incomplete,+ linked, or hidden in raw HTML. Tier 3 evidence now cross-checks complete+ per-condition attempt and error records, and the public-card gate rejects+ ambiguous structural headings, browser-rendered decision-text aliases, and+ literal or entity-encoded Unicode control/format characters. The source+ identity is read without following links before and after each producer,+ follows the host filesystem's case aliases, and records a bounded+ source-change marker. Tier 3 runs that stage repository context, external+ reference or workspace skills, or task inputs with `.git`, `.venv`,+ `__pycache__`, or `node_modules` entries remain runnable but cannot certify+ publication, and neither can a target whose authored paths below the root+ use names that Tier 1 skips (such as `references/results/`).+- Clean and not-applicable License, Code Risk, Secrets, and Dependency Audit+ runs now record structured execution evidence, so legitimate built-in Tier 1+ results can satisfy publication without relying on findings or errors.+- Skip reasons are bounded, control-free, and escaped in reports and in the+ terminal, so oversized metadata, Rich markup, or terminal escape sequences+ cannot corrupt them. Reports show one line; the pipeline view keeps the+ reason's line breaks.+- SkillSpector issue paths with NUL bytes or lone surrogates now make the+ security scan incomplete instead of being accepted.+- The public benchmark gate rejects `PASS` cards with placeholder identities,+ malformed or future evaluation dates, a missing Tier 3 run ID, hostile+ benchmark-policy metadata, or environment-label substitutions in required+ proof fields.+- Hardened Tier 3 HTML and Markdown reporting against recursive or malformed+ display metadata, Markdown structure injection, and inline-JavaScript+ injection through untrusted agent or skill names.++## 0.5.0 - 2026-10-06++### Added++- Transparent HTTP 429 (rate-limiting), transient 5xx, and timeout recovery for+ LLM judges in both the Harbor container verifier (`eval.py`) and host runtime+ (`LLMClient`). Features zero-dependency full jitter exponential backoff,+ RFC-7231 `Retry-After` header parsing, finite-value environment overrides+ (`SKILL_EVAL_LLM_MAX_RETRIES`, `SKILL_EVAL_LLM_RETRY_BASE_DELAY`, and+ `SKILL_EVAL_LLM_RETRY_MAX_DELAY`), and+ automatic container forwarding via Harbor `task.toml`, and a per-judge+ verifier time budget that leaves room for failure artifacts, without altering+ benchmark metrics or scoring formulas.+- Provider-aware structured JSON schema enforcement (`response_format` for+ OpenAI-compatible / Gemini Vertex / NVIDIA NIM endpoints and `output_config`+ for Anthropic `/v1/messages`) across the custom `judge_accuracy`,+ `judge_goal_accuracy`, and `judge_behavior_check` paths, with automatic+ schema-specific `HTTP 400`/`422` downgrade and per-target memoization+ (`_SCHEMA_UNSUPPORTED_TARGETS`), boolean prompt alignment, and a guard for+ missing `message` fields on reasoning token exhaustion. The canonical OpenAI+ RAGAS goal scorer retains its separate scoring path.+- Configurable evidence bundle budgets (`SKILL_EVAL_ACCURACY_BUDGET`, `SKILL_EVAL_GOAL_ACCURACY_BUDGET`, `SKILL_EVAL_BEHAVIOR_CHECK_BUDGET`) and final response limit (`SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT`).+- Tier 3 log converters now rebuild ATIF trajectories from OpenCode JSON streams+ (`opencode.txt`) and structured Codex tee logs (`codex.txt`) when+ `trajectory.json` is missing or empty.++### Changed++- Upgraded the optional Tier 3 backend to Harbor 0.24.0 and its compatible+ LiteLLM 1.92-1.93 window. Existing SkillEvaluator agent and environment+ options now use Harbor's unified selectors; installed Codex adapters preserve+ merged user and MCP configuration; Docker and local execution stream redacted+ callbacks under a shared 16 MiB per-command output limit, while the parent+ Harbor orchestration process has a separate 16 MiB combined stdout/stderr+ limit; and Docker supports stdin plus isolated sidecar operations without+ exposing environment values on Compose argv. Generated schema 1.3 and+ unmodified native task schemas remain compatible, while collection accepts+ Harbor 0.24 job, trial, reward, and ATIF v1.8 artifacts. Multimodal tool+ output reaches judges and reports as text with `[image]`/`[audio]` markers.+ Native tasks' own test scripts must write numeric `reward.json` values, which+ Harbor 0.24 enforces; a failed job now names the trial or step exception that+ Harbor recorded.+- Exposed 23 Harbor 0.24 backends alongside local mode. `cua-cloud`,+ `opensandbox`, `hf-sandbox`, `podman`, `kata`, `runta`, `prime`, `mosaic`,+ and `smol` remain disabled until generated tasks can be projected through a+ trusted image or backend-native provisioning path. Non-secret backend+ constructor options can be supplied with repeatable, operator-only+ `--environment-kwarg` / `--ek` flags; skill-owned configuration,+ credentials, and sandbox-policy overrides remain outside that surface.+ Harbor's `stream` and `enable_environment_dir_upload` options are reserved,+ and Modal's `modal_sandbox_v2` option no longer exists.+- `--env-mode wandb` now runs Harbor's `cwsandbox` backend with W&B+ authentication, because Harbor 0.24 merged the two. Install+ `harbor[cwsandbox]==0.24.0` instead of the removed `harbor[wandb]` extra.+- Harbor now always starts in an empty, evaluator-owned working directory with+ `.env` loading and Harbor telemetry disabled. A `.env.local` or `.env` file+ near the operator's project can no longer add credentials or settings that+ SkillEvaluator's allowlisted environment removed. The `tier3` extra requires+ python-dotenv 1.2.0.+- Codex runs with `reasoning_effort=high`, the default under Harbor 0.22, so+ scores and cost stay comparable after Harbor stopped pinning it.+- Native tasks follow Harbor 0.24's separate-verifier precedence. A dedicated+ verifier image or build definition must ship its own test script and never+ receives SkillEvaluator's grader. Otherwise the verifier is built from+ `environment/`, whose Compose model must stay within the verifier's+ environment allowlist. `docker-compose.yml` verifier definitions, which+ Harbor ignores, are rejected.++### Fixed++- Report non-string YAML keys as validation errors in skills, rules, workflows,+ and plugin manifests instead of raising a `TypeError`, including frontmatter+ `metadata` keys. Show boolean, null, and date keys in a readable YAML form.+- Harbor ``result.json`` case ids now prefer canonical ``task_id.path`` metadata+ over repository-prefixed ``task_name`` values when resolving eval entries.+- Codex log synthesis maps ``web_search`` action payloads and ``collab_tool_call``+ thread items into ATIF, and error-recovery checks recognize ``status=failed`` /+ ``exit_code=`` terminal evidence emitted by Codex converters.+- Tier 3 Harbor dual-arm evaluation propagates arm suffixes (`-with-skill`,+ `-without-skill`) to `[task] name` in staged native `task.toml` files,+ normalizes external repository and namespace prefixes, and commutatively+ resolves canonical case IDs across attempt and arm suffix combinations+ while preserving expected case IDs.+- Tier 3 native-task collection now keeps Harbor's staged directory selector,+ logical dataset ID, and display name separate; ambiguous or unresolved+ persisted identities fail closed instead of trusting grader-authored IDs.+ Runner-owned attempt ordinals are carried structurally, so `attempt`-like+ text in authored selectors, logical IDs, or display names cannot corrupt+ pass@k or `stop_on_pass` accounting, including truncated aggregate names.+- Tier 3 Harbor subprocess, Docker, and local diagnostics now redact raw and+ percent-decoded URI/proxy userinfo components across streamed callbacks,+ nonzero exits, timeouts, output limits, and persisted launch errors.+- Tier 3 local OpenCode runs routed through NVIDIA Build now retain the rendered+ user instruction for ATIF conversion and fail on OpenCode error events, in+ parity with Harbor's upstream agent lifecycle. Remote MCP servers in that+ configuration disable OAuth, as Harbor's own OpenCode configuration does.+- Tier 3 reports now use the collector's logical attempt overall whenever a+ condition mixes standard and custom rewards, including across separate+ trials, and aggregate execution summaries preserve child-declared hidden+ error counts and truncation through launch-error overlays.+- Tier 3 now rejects non-finite, overflowing, and finite-but-unscalable timeout+ multipliers at YAML, programmatic, and Harbor command boundaries.+- Tier 3 now preserves paired baseline isolation by rejecting skill-owned+ pre-agent setup, native task/step healthchecks, native step workdir overlays,+ and Harbor task-shipped prior trajectories whenever the baseline arm is+ enabled. Native standard grading also fails closed on step test overlays,+ separate verifier contexts, post-agent collect hooks, and task-controlled+ verifier executable-path, shell, loader, proxy/TLS, provider, and judge+ environment controls; exact operator-staged provider/judge placeholders and+ unrelated verifier variables remain compatible. Operator-configured judge+ fallback models are forwarded only to standard verifier jobs, not agent or+ `custom_only` environments. Its Python payload now runs from a replaced+ evaluator-owned directory in isolated mode. `custom_only`+ retains Harbor-native collect hooks and Harbor's shared/separate+ step-test resolution, and fully authored native test paths are left untouched+ instead of replacing their unused `tests/skill_evaluator/` package.+ All native grading modes reject Windows agent or effective verifier+ environments until evaluator projection and verifier scripts are OS-aware.+- Tier 3 collection now fails closed on unsafe reward identities and malformed+ custom-metric contracts, publishes exact truncation metadata for bounded+ case and failure-detail samples, and keeps findings, attribution, and+ per-trial JSON inside the report loader's artifact envelope.+- Tier 3 progress output, diagnostics, and result summaries no longer redact+ counts such as `0/1 scored` or `1 errored` when a credential-named+ environment variable holds a short flag value such as `1`. Short credential+ fragments in proxy URLs are still redacted.+- `create-eval-dataset --refine` no longer requires the `tier3` extra to read+ case IDs from Harbor result files, and no longer fails on multimodal+ trajectory messages.++## 0.4.0 - 2026-09-30++### Fixed++- Render untrusted skill content, paths, tool messages, and LLM output literally+ in CLI reports and logs. Escape Rich markup and strip terminal control+ sequences to prevent rendering failures and misleading output, including+ catalog summaries and the compact validation view. Preserve Windows paths+ and literal emoji codes+ ([#173](https://github.com/NVIDIA/SkillEvaluator/pull/173),+ [#175](https://github.com/NVIDIA/SkillEvaluator/pull/175)).+- Make sensitive-assignment, JWT, and private-key redaction linear-time,+ preventing long adversarial text from stalling logs and reports. Apply the+ JWT fix to Tier 3 command output and the bundled Harbor verifier+ ([#172](https://github.com/NVIDIA/SkillEvaluator/pull/172),+ [#176](https://github.com/NVIDIA/SkillEvaluator/pull/176)).+- `--no-llm` full datasets include a negative bucket only when eval guidance+ supplies an off-skill prompt; template mode no longer guesses canned+ negatives from a fixed question list. CLI and docs now describe `--full` as+ up to four cases instead of always four.+- Treat `apply_patch` file headers (`*** Add File:`, `*** Update File:`, `*** Delete File:`,+ `*** Move to:`) as write targets in the Tier 3 security check. A patch that targets a shell+ profile, SSH, credential, or privileged config path, sent as an `apply_patch` tool call or a+ shell heredoc, is now a critical `sensitive_file_write` finding whose evidence names the+ protected path, not the patch. Every header in the patch is checked.+- Mask GitHub tokens (`ghp_`, `gho_`, `ghu_`, `ghs_`, `ghr_`, and `github_pat_`) in Tier 3+ evidence excerpts and Harbor verifier log output.+- Stop the PII scan reporting User-Agent product versions such as `Chrome/140.0.0.0`+ as public IP addresses. Chromium's reduced User-Agent gives every version this shape.+- Separate Tier 2 collection limits from the 256-file per-skill limit. Fresh+ similarity scans now allow 1,024 selected manifests and 128 million scalar+ comparisons by default, covering 343 skills with 2,048-dimensional embeddings.+ Add `--max-entries` and `--max-scalar-comparisons` for explicit scan budgets;+ exceeded limits fail with actionable errors and never truncate the collection.+ Fresh pairwise scans reject excessive scalar work after the first validated+ embedding response, before requesting more embeddings or saving a catalog.+ Collection discovery has its own 20,000-path ceiling, allowing the supported+ 5,000-entry maximum for minimal collections while retaining the 4,096-path+ per-skill ceiling. Invalid Python command API budgets raise option-specific+ errors before provider initialization. Similarity comparisons validate and+ normalize each vector once instead of once per pair, making large pairwise+ scans more than an order of magnitude faster. Equal nonzero embeddings score+ exactly 1.0, so `--threshold 1` reports exact duplicates.+- Keep headings and comments inside fenced code examples in their enclosing Markdown+ section during Tier 2 content chunking, preserving original source line numbers.+- Run the public Docker image as an unprivileged user, with writable default report and home directories.+ Document UID/GID overrides for host-owned output mounts.+- Pin the HTML report Chart.js dependency and verify its integrity before browser execution.++- Preserve full Codex gateway model IDs in cloud environments, including E2B+ and Daytona, while retaining local runtime setup and native provider routing.+- Keep Tier 2 execution diagnostics visible alongside duplicate findings in+ CLI, HTML, and Markdown reports, without repeating the findings as errors.+- Use `nvidia/nemotron-3-super-120b-a12b` as the shared NVIDIA Build default for+ evaluator chat, agent execution, and judging, preserving explicit model overrides.+ Request nonstreaming chat responses explicitly to match the response parser.+- Tier 2 LLM failures now identify the selected provider and model, HTTP status,+ safe error metadata, and failed-cluster count without exposing response bodies.+- Replace the retired NVIDIA embedding default with `nvidia/nemotron-3-embed-1b`+ and send the required passage input type for NVIDIA document comparisons.+ Existing catalogs and caches must be rebuilt when switching embedding models.+- Report Tier 2 embedding and LLM service failures as incomplete checks, retaining+ a nonzero exit without inventing duplicate-content findings. Provider error+ messages include recovery guidance without echoing raw response bodies.+- Tier 3 script execution credit now requires evidence that the expected script+ was invoked. `check_script_execution` previously treated the script name as a+ substring of an execution command, so reading, printing or searching the+ script, or running a similarly named file, scored a full `Executed <script>`.+ Credit is now given only for a recognised invocation: the script run directly,+ an interpreter given it as its script argument, a `source`, or a `sh -c`+ payload that does one of those, with `cd` tracked and script identity compared+ exactly. Interpreter and wrapper options come from grammars derived by running+ each option against a script that records whether it executed, so `--help`,+ `perl -c` and `bash -n` run no script while `python -Wignore` and+ `env FOO=1` still resolve to theirs. A command the walk cannot resolve keeps+ the existing 0.75 partial credit rather than being scored either way: a path+ built at run time, an option outside a grammar, a name only in heredoc data,+ inline code or a module that names the script, and anything reaching a command+ through standard input, including `xargs` and `parallel`, whose behaviour the+ command text never determines. Partial credit is only ever given for a+ command that names the script: an unresolved command that never mentions it+ scores zero, as before. Redirections standing before the script+ (`python3 < /dev/null run.py`), a script's own arguments that look like shell+ options (`bash run.sh -c '...'`), invocations inside `if`, `while`, `until`+ and `for` bodies, and versioned interpreter names (`perl5.38.2`, `python3.13`)+ resolve as the shell runs them. A loop over an empty list keeps the earlier+ binding and its body is not read; a binding made inside `( ... )` stays+ there; the last command of a pipeline keeps its bindings where the shell+ does (zsh, ksh) and not where it forks it (bash, dash, mksh), with an+ option change that could move it (`shopt -s lastpipe`, `emulate sh`)+ read as unresolved; a quoted or escaped word that would read as syntax+ (`'done'`, `printf "("`, `';|'`) is the ordinary word it is; groups and+ compound pipeline stages nest in either order, each compound tested for its+ own pipe; `((` closed by `))` is an arithmetic command where the shell has+ one; the positional+ parameters are empty unless the text gives some, so `for f; do` at the top+ level runs nothing and keeps the variable's value; an interpreter fed its+ program by a pipe (`cat run.py | python3`) is unresolved like+ `python3 < run.py`; a variable bound by `export` or `readonly`, or by+ `declare` and `typeset` where the shell has them, holds the value it had+ when the builtin ran, `unset` empties it, `+x` and `export -n` unexport it+ however the builtin is reached, and one bound from data the text does not+ carry (`read`, `printf -v`, `local` outside a function, `declare -u`, a name+ `eval` may bind, including from a program held in a variable) is+ unresolved, as is a later assignment to a name given `-i`, `-u` or `-n`,+ while `-l` lowercases it and an array is never exported; what is done to a+ `-n` name (`nameref` in ksh and mksh) leaves the name it refers to+ unresolved; `let`, `$((...))` and `$[...]` assign as `((...))` does; an+ assignment the shell rejects (a value that is not a number for an `-i`+ name) and a special builtin given an option the shell rejects end the+ credit where the shell stops there; an assignment written before a command,+ `env NAME=value` included, is that command's environment only, reaching+ neither its own words nor the commands after it, except before a special+ builtin in the POSIX shells; a `-c` payload's shell sees the exported names,+ its command's own prefix and what `env` adds and removes, and a `$` quoted+ or escaped from the outer shell is expanded there (one left unquoted is+ expanded here, a name never bound to nothing), with the payload's own+ heredoc bodies kept as data; text that `eval`, inline+ code or a shell reading a heredoc may expand again is unresolved when a+ variable in it holds the script; every heredoc declared on a line takes its+ body after the line, in order, and past the number a shell accepts on one+ line (16 in bash) the line and the rest are data; a here-string, which dash+ and busybox ash reject before running anything, leaves nothing credited+ under those shells, and so does a compound's opening word written after an+ assignment (`A=1 for ...`, `A=1 if ...`, `A=1 ( ... )`), which every+ modelled shell rejects and which no longer raises; `(((` is a subshell+ around `((` in bash, zsh and mksh; and `ksh`, `mksh` and `ash` are+ recognised shells.+ Applied to both the host checker and the bundled Harbor verifier.++- Added `scripts/script_invocation_differential.py`, a differential harness that+ executes each command for real against fixtures that record whether they ran,+ and compares the result against both implementations. `--baseline REF` also+ scores every command with the checker at an earlier ref and lists each score+ that moved, so a change that lowers a command that ran, or raises one that+ did not, is seen before it is pushed.++### Added++- Interactive top-level help now opens with a green SkillEvaluator wordmark,+ installed version, and tier overview. Narrow terminals use a compact header;+ redirected output and subcommands keep their existing output format.++- OpenAI-compatible gateways now have chat, embedding, and separate Codex,+ Claude Code, and OpenCode model defaults. Set the provider, URL, and key;+ override model IDs when the gateway uses different catalog names. Claude+ Code inherits the gateway route unless an explicit Anthropic route is set.+ Run reports identify harness defaults separately from CLI/config overrides.++- Run individual tiers directly with `skillevaluator tier1 PATH`, `tier2 PATH`,+ and `tier3 PATH`, while retaining the expert subcommands. Tier 1 includes+ dependency checks and enables LLM checks when configured; Tier 2 reports+ whether a catalog comparison ran; Tier 3 creates a missing starter dataset+ and preserves existing evaluation sources.++### Changed++- Show elapsed waiting time during autopilot dataset generation and identify+ deterministic starter datasets used after a provider failure. Fully unscored+ Tier 3 runs show an `INCOMPLETE` summary with coverage, consolidated execution+ errors, and recovery steps; completed comparisons highlight overall Skill+ Lift ([#152](https://github.com/NVIDIA/SkillEvaluator/pull/152)).+- Harden documentation publishing with restricted token permissions, pinned+ checkout and Fern versions, and disabled persisted checkout credentials.+ Apply the checkout credential restriction to DCO checks+ ([#158](https://github.com/NVIDIA/SkillEvaluator/pull/158)).+- Add the methodology paper as the preferred citation and link research,+ developer-blog, and livestream resources from the README+ ([#146](https://github.com/NVIDIA/SkillEvaluator/pull/146),+ [#159](https://github.com/NVIDIA/SkillEvaluator/pull/159)).+- `validate PATH` now runs all three tiers for skills by default. Tier 3+ autopilot reuses an existing evaluation source or creates one starter case+ when none exists. `--full` remains compatible but is unnecessary;+ `--tiers`, `--no-tier3`, and `--no-autopilot` provide explicit scope controls.+ Keyless static CI gates should select `--tiers 1`.+- Tier 3 now selects a provider-native agent when `--agents` is omitted:+ OpenCode for NVIDIA Build, Codex for OpenAI, and Claude Code for Anthropic.+ NVIDIA Build agent runs default to Nemotron Super; explicit agent and model+ overrides remain unchanged.+- Tier 3's extra agent runtime preflight is now disabled by default because it+ executes the first real task prompt and incurs agent runtime and model cost.+ Enable it explicitly with `--agent-runtime-preflight` on either `tier3` or+ `validate`, or with `harbor.agent_runtime_preflight: true` in `evals/config.yml`.+- Missing-provider and API-key errors now show concise, copyable setup steps+ and a link to advanced configuration. Tier 1 also explains `--no-llm`.+- `tier1 validate` now runs only Tier 1, matching `tier1 PATH`. The top-level+ `validate` command retains its combined pipeline behavior.+- README and getting-started guides lead with provider-plus-key setup for+ NVIDIA Build and OpenAI, explain inherited model defaults, and separate+ credentials from scanner and agent runtime requirements.## 0.3.0 - 2026-09-17### Added+- Catalog validation now writes `catalog-summary.json` at the reports root withper-skill pass/fail status, optional severity rollups from child JSON reports,and paths to per-skill report directories.diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.mdindex 8b43e365..a9953a32 100644--- a/THIRD_PARTY_NOTICES.md+++ b/THIRD_PARTY_NOTICES.md@@ -8,7 +8,7 @@ records the exact resolved dependency set used for this release.| --- | --- || Base | Click (BSD-3-Clause), IDNA (BSD-3-Clause), Jinja2 (BSD-3-Clause), Markdown-It-Py (MIT), Packaging (Apache-2.0 or BSD-2-Clause), Pydantic (MIT), PyYAML (MIT), Rich (MIT) || LLM | Anthropic (MIT), Boto3 (Apache-2.0), LiteLLM (MIT), OpenAI (Apache-2.0) |-| Tier 3 | Harbor (Apache-2.0), MCP (MIT), PyJWT (MIT), python-dotenv (BSD-3-Clause) |+| Tier 3 | Harbor (Apache-2.0), MCP (MIT), OAuthLib (BSD-3-Clause), PyJWT (MIT), python-dotenv (BSD-3-Clause) || Security | Bandit (Apache-2.0), pip-audit (Apache-2.0) |## Unicode security datadiff --git a/docs/agents-and-sandboxes.mdx b/docs/agents-and-sandboxes.mdxindex de2abc02..3a64e2a2 100644--- a/docs/agents-and-sandboxes.mdx+++ b/docs/agents-and-sandboxes.mdx@@ -364,6 +364,13 @@ Environment rules — size limits, auto-generated fallbacks, and sidecarcontainers — are part of the `evals/` contract in[Eval Datasets](eval-datasets.mdx).+Docker Compose project `.env` files are rejected before startup. Protected+main-container and sidecar operations disable Compose's implicit env-file+loading so the same definition cannot start with one interpolation value and+later run with another. Put non-secret interpolation values in+`harbor.runtime_env` in `evals/config.yml`; keep operator credentials in the+host environment as described above.+## GKE modeWhen running with `--env-mode gke`, SkillEvaluator enforces least-privilege pod@@ -393,6 +400,35 @@ identity by default unless you explicitly opt in for trusted skills viaclusters, GPU/TPU pods, and explicit storage requests above `10Gi` are notlowered.+## Kata mode++`--env-mode kata` uses the same hardened Docker Compose backend as Docker mode,+including its credential handoff and container containment, but Harbor adds a+final Compose overlay that runs every task service under a Kata Containers+runtime, so agent execution and verification happen inside KVM microVMs instead+of on the host kernel. It needs a Linux host with `/dev/kvm`, Docker Compose v2,+and a Kata runtime handler registered with Docker (for example with Kata's+`kata-manager.sh`). `skillevaluator doctor --env-mode kata` checks all three,+including that Docker registers the selected handler.++Two environment kwargs are accepted:++| Kwarg | Default | Rule |+| --- | --- | --- |+| `kata_runtime` | `kata` | A registered Docker runtime handler whose name contains `kata`, such as `kata-clh` or `kata-qemu`. Other handlers, such as `runc`, are rejected. |+| `kata_dns` | `1.1.1.1,8.8.8.8` | At least one IP address, written to each microVM's `resolv.conf`, as a list or comma-separated string. The default replaces the resolvers Docker's embedded DNS would forward to, so set your network's resolvers when public DNS is blocked or when a gateway, judge, or package endpoint uses an internal hostname. |++Harbor's Kata backend has limits that SkillEvaluator keeps:++- Image builds are not VM-isolated. Dockerfile `RUN` steps execute on the+ host's BuildKit builder; the microVM boundary covers the agent and verifier.+- Non-public network policies are rejected before start, because Kata VMs+ cannot join the egress-control sidecar's network namespace. Generated tasks+ use public networking.+- Compose service-name DNS does not work inside the microVM, so tasks whose+ Compose model declares services other than `main`, such as a database+ sidecar, are rejected before the run starts.+## Local mode<Warning>diff --git a/pyproject.toml b/pyproject.tomlindex c20bd028..e8651948 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -62,11 +62,13 @@ tier2 = [# Tier 3 live agent evaluation through Harbor's native environments.tier3 = ["skillevaluator[llm]",- "harbor[gke]==0.13.2",+ "harbor[gke]==0.24.0",# CVE-2026-59950, CVE-2026-52870, and CVE-2026-52869 fixed floor."mcp>=1.28.1,<2",# CVE-2026-48522 through CVE-2026-48526 fixed floor."pyjwt[crypto]>=2.13.0",+ # GHSA-hj66-6f7g-4r5v and GHSA-xpv3-w29h-x7cv fixed floor (via harbor[gke]).+ "oauthlib>=4.0.0",# PYTHON_DOTENV_DISABLED, which keeps Harbor from loading .env files, needs 1.2.0."python-dotenv>=1.2.0",]diff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.pyindex ee70929c..e9d901a7 100644--- a/src/skillevaluator/cli.py+++ b/src/skillevaluator/cli.py@@ -2556,16 +2556,147 @@ def dedup_scan(raise click.ClickException("dedup scan failed")-def _parse_environment_kwargs_cli(raw_kwargs: tuple[str, ...]) -> dict[str, str]:+def _parse_environment_kwargs_cli(+ raw_kwargs: tuple[str, ...],+ *,+ env_mode: str | None = None,+) -> dict[str, Any]:"""Parse and validate environment kwargs for CLI commands, raising Click BadParameter on errors."""- from skillevaluator.tier3.commands import parse_environment_kwargs+ from skillevaluator.tier3.evals_config import parse_environment_kwarg_overridestry:- return parse_environment_kwargs(raw_kwargs)+ return parse_environment_kwarg_overrides(raw_kwargs, env_mode=env_mode)except ValueError as exc:raise click.BadParameter(str(exc), param_hint=["--ek"]) from exc+def _workflow_report_options(func):+ func = _report_options(func)+ for param in func.__click_params__:+ if param.name == "report_formats":+ param.default = ("cli", "json", "html")+ param.help = "Report formats; repeat -r or separate values with commas."+ return func++[email protected](cls=cli.command_class, context_settings=CONTEXT_SETTINGS)[email protected]("skill_path", metavar="PATH", type=click.Path(exists=True, file_okay=False, path_type=Path))[email protected]("--checks", default=None, help="Select static checks; by default all checks include dependency auditing.")+ "--llm/--no-llm", default=None, help="Include rubric, LLM security, and verification [default: when configured]."+)[email protected]("--min-score", type=click.IntRange(0, 100), default=70, show_default=True)[email protected]("--profile", default=None, help="Validation policy profile [default: external].")[email protected]("--previous-version", default=None, help="Previous version for version checks.")+@_workflow_report_options+def _tier1_workflow(+ skill_path: Path,+ checks: str | None,+ llm: bool | None,+ min_score: int,+ profile: str | None,+ previous_version: str | None,+ report_formats: tuple[str, ...],+ output_dir: Path,+) -> None:+ """Run Tier 1 static checks and configured LLM checks for one skill."""+ from skillevaluator.tier_workflows import run_tier1_workflow+ from skillevaluator.validators.policy import resolve_policy++ try:+ policy = resolve_policy(profile=profile)+ except (FileNotFoundError, ValueError) as exc:+ raise click.ClickException(str(exc)) from exc+ results = run_tier1_workflow(+ skill_path,+ checks=checks,+ llm=llm,+ min_score=min_score,+ profile=policy,+ previous_version=previous_version,+ )+ # Resolved after the run so configuration errors still leave no output.+ output_dir = _resolve_file_report_output_location(skill_path.resolve(), output_dir, report_formats)+ if not emit_reports(+ results,+ report_formats=report_formats,+ output_dir=output_dir,+ basename=report_basename("tier1"),+ policy=policy,+ target_path=str(skill_path),+ ):+ raise click.ClickException("Tier 1 checks failed or were incomplete")++[email protected](cls=cli.command_class, context_settings=CONTEXT_SETTINGS)[email protected]("skill_path", metavar="PATH", type=click.Path(exists=True, file_okay=False, path_type=Path))+ "--catalog",+ type=click.Path(exists=True, dir_okay=False, path_type=Path),+ default=None,+ help="Saved JSON catalog for inter-skill comparison.",+)+ "--threshold",+ type=float,+ default=0.80,+ show_default=True,+ callback=_validate_similarity_threshold,+ help="Intra-skill similarity threshold.",+)+ "--similarity-threshold",+ type=float,+ default=0.75,+ show_default=True,+ callback=_validate_similarity_threshold,+ help="Inter-skill similarity threshold.",+)[email protected]("--full-body", is_flag=True, help="Compare full SKILL.md content against a full-body catalog.")+@_workflow_report_options+def _tier2_workflow(+ skill_path: Path,+ catalog: Path | None,+ threshold: float,+ similarity_threshold: float,+ full_body: bool,+ report_formats: tuple[str, ...],+ output_dir: Path,+) -> None:+ """Run intra-skill deduplication and optional inter-skill catalog comparison."""+ from skillevaluator.tier_workflows import run_tier2_workflow++ _reject_linked_tier2_root(skill_path)+ basename = report_basename("tier2")+ _reject_catalog_report_collisions(catalog, report_formats=report_formats, output_dir=output_dir, basename=basename)+ results = run_tier2_workflow(+ skill_path,+ catalog=catalog,+ threshold=threshold,+ similarity_threshold=similarity_threshold,+ full_body=full_body,+ )+ sanitize_tier2_results(results, skill_path, catalog)+ # Resolved after the run so configuration errors still leave no output; a+ # relocated directory gets the same catalog-collision check before writing.+ resolved_output_dir = _resolve_file_report_output_location(skill_path.resolve(), output_dir, report_formats)+ if resolved_output_dir != output_dir:+ _reject_catalog_report_collisions(+ catalog,+ report_formats=report_formats,+ output_dir=resolved_output_dir,+ basename=basename,+ )+ if not emit_reports(+ results,+ report_formats=report_formats,+ output_dir=resolved_output_dir,+ basename=basename,+ target_path=str(skill_path),+ ):+ raise click.ClickException("Tier 2 checks failed or were incomplete")++# Hidden top-level spelling of ``tier3 evaluate`` — kept working for scripts,# but the tier namespace is the advertised name to avoid a duplicate surface.@cli.command(hidden=True)@@ -2643,13 +2774,6 @@ def _parse_environment_kwargs_cli(raw_kwargs: tuple[str, ...]) -> dict[str, str]show_default=True,help="Tier 3 progress presentation (auto uses Rich on a TTY and plain lines otherwise).",)- "--ek",- "--environment-kwarg",- "environment_kwargs",- multiple=True,- help="Environment kwarg override in KEY=VALUE form (repeatable).",-)def evaluate(skill_path: Path,agents: str | None,@@ -2680,7 +2804,6 @@ def evaluate(evaluated_source_revision: str | None,evaluator_container_revision: str | None,progress: str,- environment_kwargs: tuple[str, ...] = (),) -> None:"""Run Tier 3 live agent evaluation."""from skillevaluator.evaluation import EvaluationOptions, EvaluationService@@ -2697,7 +2820,7 @@ def evaluate(if autopilot:_ensure_autopilot_dataset(skill_path, progress=progress)- parsed_environment_kwargs = _parse_environment_kwargs_cli(environment_kwargs)+ parsed_environment_kwargs = _parse_environment_kwargs_cli(environment_kwarg, env_mode=env_mode)options = EvaluationOptions(skill_path=skill_path,@@ -2724,6 +2847,7 @@ def evaluate(override_cpus=override_cpus,override_memory_mb=override_memory_mb,override_storage_mb=override_storage_mb,+ evaluated_source=evaluated_source,environment_kwargs=parsed_environment_kwargs,)try:@@ -2759,7 +2883,7 @@ def evaluate(console.print(Panel(Text(- "Intended for trusted skills and least-privilege service accounts. Harbor 0.13.2 single-pod "+ "Intended for trusted skills and least-privilege service accounts. Harbor 0.24.0 single-pod ""GKE execution shares the pod service account and metadata server with evaluated skill commands.",style="yellow",),@@ -2953,24 +3077,17 @@ def models_command(limit: int, as_json: bool) -> None:is_flag=True,help="Check resolved agent-model catalog reachability with a live credential-bearing request.",)- "--ek",- "--environment-kwarg",- "environment_kwargs",- multiple=True,- help="Environment kwarg override in KEY=VALUE form (repeatable).",-)def doctor(- agents: str,+ agents: str | None,env_mode: str,+ environment_kwarg: tuple[str, ...],agent_model: tuple[str, ...],verify_models: bool,- environment_kwargs: tuple[str, ...] = (),) -> None:"""Check live-evaluation runtime readiness."""from skillevaluator.tier3.commands import doctor as tier3_doctor- parsed_environment_kwargs = _parse_environment_kwargs_cli(environment_kwargs)+ parsed_environment_kwargs = _parse_environment_kwargs_cli(environment_kwarg, env_mode=env_mode)raise SystemExit(tier3_doctor(agents=agents,@@ -3001,13 +3118,15 @@ def health_check(agents: str | None, env_mode: str, environment_kwarg: tuple[str"""Quick readiness check for the CLI and selected live-eval backend."""from skillevaluator.tier3.commands import doctor as tier3_doctor+ parsed_environment_kwargs = _parse_environment_kwargs_cli(environment_kwarg, env_mode=env_mode)raise SystemExit(tier3_doctor(agents=agents,env_mode=env_mode,+ environment_kwarg=environment_kwarg,verify_models=False,agent_model=(),- environment_kwargs={},+ environment_kwargs=parsed_environment_kwargs,))diff --git a/src/skillevaluator/evaluation/options.py b/src/skillevaluator/evaluation/options.pyindex af21ec75..c3b02379 100644--- a/src/skillevaluator/evaluation/options.py+++ b/src/skillevaluator/evaluation/options.py@@ -46,6 +46,11 @@ class EvaluationOptions:override_cpus: int | None = Noneoverride_memory_mb: int | None = Noneoverride_storage_mb: int | None = None+ repo_context_exclude_paths: tuple[Path, ...] = ()+ environment_kwarg: tuple[str, ...] = ()+ # Supplied by the orchestration input, never inferred from repository state:+ # the tree that runs the evaluator is not the tree being evaluated.+ evaluated_source: dict[str, str] | None = Noneenvironment_kwargs: dict[str, str] = field(default_factory=dict)def engine_kwargs(self) -> dict[str, Any]:diff --git a/src/skillevaluator/inference/client.py b/src/skillevaluator/inference/client.pyindex 2374e477..66fd83db 100644--- a/src/skillevaluator/inference/client.py+++ b/src/skillevaluator/inference/client.py@@ -603,42 +603,41 @@ class LLMClient:call_kwargs["response_format"] = _build_openai_response_format(response_schema, schema_name)return call_kwargs- response = _call_with_schema_fallback(- client.chat.completions.create,- _openai_kwargs,- target_key=target_key,- use_schema=use_schema,- )- return _extract_choice_content(response)-- try:- response = client.chat.completions.create(**call_kwargs)- except Exception as exc:- from skillevaluator.provider_config import _get_google_access_token-- if config.credential_env == "ADC":- from openai import AuthenticationError-- if isinstance(exc, AuthenticationError) or getattr(exc, "status_code", None) == 401:- new_token = _get_google_access_token()- if new_token:- import dataclasses+ def _create_chat_completion() -> Any:+ return _call_with_schema_fallback(+ client.chat.completions.create,+ _openai_kwargs,+ target_key=target_key,+ use_schema=use_schema,+ )- client.api_key = new_token- self._provider_config = dataclasses.replace(config, api_key=new_token)- if self._api_key is not None:- self._api_key = new_token- response = client.chat.completions.create(**call_kwargs)+ try:+ response = _create_chat_completion()+ except Exception as exc:+ from skillevaluator.provider_config import _get_google_access_token++ if config.credential_env == "ADC":+ from openai import AuthenticationError++ if isinstance(exc, AuthenticationError) or getattr(exc, "status_code", None) == 401:+ new_token = _get_google_access_token()+ if new_token:+ import dataclasses++ client.api_key = new_token+ self._provider_config = dataclasses.replace(config, api_key=new_token)+ if self._api_key is not None:+ self._api_key = new_token+ response = _create_chat_completion()+ else:+ raiseelse:raiseelse:raise- else:- raise- content = response.choices[0].message.content- if not content:- raise EmptyLLMResponseError("LLM returned empty response content")- return content.strip()+ return _extract_choice_content(response)++ return retry_call_with_backoff(_invoke_provider, config=self._retry_config)def extract_json_from_response(self,diff --git a/src/skillevaluator/tier3/commands.py b/src/skillevaluator/tier3/commands.pyindex 6b86b9c4..f0e9a9d0 100644--- a/src/skillevaluator/tier3/commands.py+++ b/src/skillevaluator/tier3/commands.py@@ -403,26 +403,15 @@ def parse_agent_model_overrides(raw_overrides: tuple[str, ...]) -> dict[str, lisreturn overrides-def parse_environment_kwargs(raw_kwargs: tuple[str, ...]) -> dict[str, str]:- """Parse repeatable ``--ek key=value`` CLI pairs."""- from skillevaluator.tier3.harbor.runner import _SENSITIVE_EK_VALUE_PATTERNS, _is_sensitive_ek_key+def parse_environment_kwargs(+ raw_kwargs: tuple[str, ...] | list[str],+ *,+ env_mode: str | None = None,+) -> dict[str, Any]:+ """Parse repeatable ``--ek key=value`` CLI pairs via ``parse_environment_kwarg_overrides``."""+ from skillevaluator.tier3.evals_config import parse_environment_kwarg_overrides- parsed: dict[str, str] = {}- for raw in raw_kwargs:- if "=" not in raw:- raise ValueError(f"--ek/--environment-kwarg must be in KEY=VALUE form, got: {raw}")- key, value = raw.split("=", 1)- key = key.strip()- value = value.strip()- if not key:- raise ValueError(f"--ek/--environment-kwarg key cannot be empty, got: {raw}")- if _is_sensitive_ek_key(key) or any(pattern.search(value) for pattern in _SENSITIVE_EK_VALUE_PATTERNS):- raise ValueError(- f"Sensitive key or value detected in environment_kwargs: {key}. "- "Credentials must not be passed via CLI flags or process arguments."- )- parsed[key] = value- return parsed+ return parse_environment_kwarg_overrides(raw_kwargs, env_mode=env_mode)def validate_agents(agents: list[str]) -> list[str]:@@ -739,7 +728,8 @@ def evaluate(agent_models = parse_agent_model_overrides(agent_model)from skillevaluator.tier3.evals_config import parse_environment_kwarg_overrides- environment_kwargs = parse_environment_kwarg_overrides(environment_kwarg, env_mode=env_mode)+ parsed_cli_kwargs = parse_environment_kwarg_overrides(environment_kwarg, env_mode=env_mode)+ merged_environment_kwargs = {**dict(environment_kwargs or {}), **parsed_cli_kwargs}unknown_model_agents = sorted(set(agent_models) - set(agent_list))if unknown_model_agents:raise ValueError(@@ -769,7 +759,7 @@ def evaluate(agent_runtime_preflight=agent_runtime_preflight,env_mode=env_mode,env_mode_source="CLI",- environment_kwargs=environment_kwargs,+ environment_kwargs=merged_environment_kwargs,timeout_multiplier=timeout_multiplier,evaluated_source=evaluated_source,override_cpus=override_cpus,@@ -777,7 +767,6 @@ def evaluate(override_storage_mb=override_storage_mb,repo_context_exclude_paths=repo_context_exclude_paths,progress_reporter=reporter,- environment_kwargs=environment_kwargs,)except Exception as exc:if not engine_started:@@ -794,17 +783,27 @@ def doctor(environment_kwarg: tuple[str, ...] = (),verify_models: bool = False,agent_model: tuple[str, ...] = (),- environment_kwargs: Mapping[str, str] | None = None,+ environment_kwargs: Mapping[str, Any] | None = None,) -> int:"""Check whether live evaluation dependencies are available."""+ from skillevaluator.tier3.evals_config import parse_environment_kwarg_overrides+env_mode = _engine_env_mode(env_mode)agent_list = parse_agents(agents) if agents is not None else []rows: list[tuple[str, str, str]] = []rows.append(("CLI package", "pass", f"skillevaluator {__version__}"))+ kwarg_parse_error: str | None = None+ try:+ parsed_cli_kwargs = parse_environment_kwarg_overrides(environment_kwarg, env_mode=env_mode)+ except ValueError as exc:+ parsed_cli_kwargs = {}+ kwarg_parse_error = str(exc)++ merged_cli_kwargs = {**dict(environment_kwargs or {}), **parsed_cli_kwargs}resolved_env_kwargs = _resolve_environment_kwargs(env_mode,- cli_kwargs=environment_kwargs,+ cli_kwargs=merged_cli_kwargs,environ=os.environ,)provider = None@@ -879,12 +878,17 @@ def doctor(else:rows.append(("Harbor agents", "pass", ", ".join(agent_list)))- prereq_errors = _check_prerequisites(- env_mode=env_mode,- agents=agent_list,- environment_kwargs=resolved_env_kwargs,- verify_live_cluster=verify_models,- )+ if kwarg_parse_error is not None:+ prereq_errors = [kwarg_parse_error]+ else:+ prerequisite_subprocess_env = dict(next(iter(runtime_plans.values())).subprocess_env) if runtime_plans else None+ prereq_errors = _check_prerequisites(+ env_mode=env_mode,+ agents=agent_list,+ environment_kwargs=resolved_env_kwargs,+ subprocess_env=prerequisite_subprocess_env,+ verify_live_cluster=verify_models,+ )if prereq_errors:for error in prereq_errors:rows.append((f"{env_mode} prerequisite", "fail", error))@@ -900,9 +904,8 @@ def doctor()from skillevaluator.tier3.harbor.runtime_preflight import (CredentialProbeDisposition,- ModelCatalogFailureKind,- _is_vertex_openapi_endpoint,credential_probe_disposition,+ is_unverified_gke_vertex_auth_probe,probe_model,)@@ -915,15 +918,7 @@ def doctor(detail = probe.detailelif disposition == CredentialProbeDisposition.DEGRADED:status = "warn"- is_vertex = (- _is_vertex_openapi_endpoint(getattr(selected_provider, "base_url", None))- or getattr(selected_provider, "credential_env", None) == "CLAUDE_CODE_USE_VERTEX"- )- is_auth_failure = getattr(probe, "failure_kind", None) in {- ModelCatalogFailureKind.AUTHENTICATION,- ModelCatalogFailureKind.AUTHORIZATION,- }- if not probe.ok and env_mode == "gke" and is_vertex and is_auth_failure:+ if is_unverified_gke_vertex_auth_probe(selected_provider, probe, env_mode=env_mode):detail = GKE_HOST_UNVERIFIED_VERTEX_AUTH_DETAILelif probe.ok:detail = f"{probe.detail}; catalog access does not verify runtime credentials for this endpoint"diff --git a/src/skillevaluator/tier3/eval_core/secret_redaction.py b/src/skillevaluator/tier3/eval_core/secret_redaction.pyindex 80cb34eb..dbbfc8d3 100644--- a/src/skillevaluator/tier3/eval_core/secret_redaction.py+++ b/src/skillevaluator/tier3/eval_core/secret_redaction.py@@ -22,7 +22,24 @@ LOG_NVAPI_RE = re.compile(r"(?<![A-Za-z0-9_-])nvapi-[a-zA-Z0-9_-]{8,}|nvapi-" +LOG_CRSR_RE = re.compile(r"(?<![A-Za-z0-9_-])crsr_[a-f0-9]{16,}")OPENSHIFT_TOKEN_RE = re.compile(r"(?<![A-Za-z0-9_-])sha256~[A-Za-z0-9._~-]+")LOG_YA29_RE = re.compile(r"(?<![A-Za-z0-9_-])ya29\.[A-Za-z0-9_-]{20,}")-LOG_JWT_RE = re.compile(r"\beyJ[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{20,}\b")+# A JWT used to start at any ``\beyJ``, so in a run of JWT characters such as+# "eyJ-" * n every "-eyJ" was a start, and each start scanned to the end of the+# run looking for ".". Now a match starts only at the beginning of a run. The part+# of the run before its first ``\beyJ`` is captured as ``lead`` and written back+# unchanged, which keeps JWTs glued to a "-" (x-eyJ...) redacted. Later starts in+# the same run are never tried: their first segment reaches the same "." with+# fewer characters, so they could only fail where the first start failed. ``lead``+# stops at the first ``\beyJ`` and every earlier offset fails ``\beyJ``, so+# backtracking into it is cheap. No atomic groups or possessive quantifiers: the+# Harbor verifier copy runs on the task image's python3, which may predate 3.11.+LOG_JWT_RE = re.compile(+ r"(?<![A-Za-z0-9_-])(?P<lead>(?:(?!\beyJ)[A-Za-z0-9_-])*)"+ r"\beyJ[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{20,}\b"+)+# GitHub classic (ghp_/gho_/ghu_/ghs_/ghr_) and fine-grained (github_pat_) tokens.+# Single bounded character classes keep both patterns linear.+LOG_GITHUB_TOKEN_RE = re.compile(r"\bgh[pousr]_[A-Za-z0-9]{36,255}\b")+LOG_GITHUB_PAT_RE = re.compile(r"\bgithub_pat_[A-Za-z0-9_]{22,255}\b")def redact_secrets_in_log_line(@@ -41,4 +58,6 @@ def redact_secrets_in_log_line(line = LOG_GITHUB_PAT_RE.sub("github_pat_<redacted>", line)line = OPENSHIFT_TOKEN_RE.sub("sha256~<redacted>", line)line = LOG_YA29_RE.sub("ya29.<redacted>", line)- return LOG_JWT_RE.sub("jwt-<redacted>", line)+ if "eyJ" not in line: # every JWT match contains "eyJ"; skip the scan on ordinary lines+ return line+ return LOG_JWT_RE.sub(r"\g<lead>jwt-<redacted>", line)diff --git a/src/skillevaluator/tier3/evals_config.py b/src/skillevaluator/tier3/evals_config.pyindex 624b706c..c90b2613 100644--- a/src/skillevaluator/tier3/evals_config.py+++ b/src/skillevaluator/tier3/evals_config.py@@ -52,7 +52,6 @@ _HARBOR_KEYS = {"passthrough_env","setup_commands","agents",- "environment_kwargs",}HARBOR_TASK_SOURCES = {"auto", "evals_json", "native_harbor"}# Harbor 0.22 multiplies its 600-second default verifier/build timeout by@@ -79,12 +78,42 @@ _GKE_INFRASTRUCTURE_KWARGS: frozenset[str] = frozenset("memory_limit_multiplier",})-_SKILL_SAFE_ENVIRONMENT_KWARGS: frozenset[str] = frozenset(- {- "workload_profile",- "custom_setting",- }+_URI_AUTHORITY_RE = re.compile(r"[A-Za-z][A-Za-z0-9+.-]{0,31}://(?P<authority>[^\s/?#]*)")+_SCHEMELESS_CREDENTIAL_AUTHORITY_RE = re.compile(r"[^\s/:@]+:[^\s/@]+@[^\s/?#]+")+_SECRET_VALUE_PATTERNS = (+ re.compile(r"\bsk-[a-zA-Z0-9_-]{12,}\b"),+ re.compile(r"\bnvapi-[a-zA-Z0-9_-]{12,}\b"),+ re.compile(r"(?<![A-Za-z0-9._-])ya29\.[A-Za-z0-9._-]{16,}"),+ re.compile(r"(?<![A-Za-z0-9_-])eyJ[A-Za-z0-9_-]{8,}\.eyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}"),+ re.compile(r"\bcrsr_[a-fA-F0-9]{12,}\b"),+ re.compile(r"\bsha256~[A-Za-z0-9_-]{16,}\b"),+ re.compile(r"(?<![A-Za-z0-9_-])(?:AKIA|ASIA)[0-9A-Z]{12,}"),+ re.compile(r"-----BEGIN (?:[A-Z0-9_-]+ )?PRIVATE KEY-----"),+ re.compile(r"(?i)\bBearer\s+[A-Za-z0-9._~+/-]{15,}"),+ re.compile(r"(?i)\b(?:gh[pours]_[A-Za-z0-9._-]{36,}|github_pat_[A-Za-z0-9_]{20,})\b"),+ re.compile(r"\bAIza[A-Za-z0-9_-]{20,}\b"),+)+_NON_CREDENTIAL_TOKEN_KEY_RE = re.compile(+ r"^(?:max|min|num|total|count|prompt|completion|input|output|request)?[_-]?tokens$|"+ r"^(?:max|min|num|total|count|prompt|completion|input|output)[_-]token$|"+ r"(?:^|.*[_-])tokens?[_-](?:bucket|rate|count|limit|budget|window|usage|size|per[_-]\w+)$|"+ r"(?:^|.*[_-])(?:tokenizer|tokenizers|detokenize|detokenizer|tokenization|detokenization)(?:[_-].*)?$",+ re.IGNORECASE,)+_REFERENCE_KWARG_NAMES_BY_ENV_MODE = {+ "ack": frozenset({"image_pull_secret"}),+ "cwsandbox": frozenset({"secrets"}),+ "daytona": frozenset({"secrets"}),+ "modal": frozenset({"registry_secret", "secrets"}),+ "skypilot": frozenset({"secrets"}),+ "wandb": frozenset({"secrets"}),+}+_CWSANDBOX_SECRET_REFERENCE_KEYS = frozenset({"env_var", "field", "name", "store"})+_KUBERNETES_DNS_LABEL_RE = re.compile(r"^[a-z0-9](?:[-a-z0-9]*[a-z0-9])?$")+_MAX_ENVIRONMENT_KWARGS_JSON_BYTES = 64 * 1024+_MAX_ENVIRONMENT_KWARGS_DEPTH = 32+_MAX_ENVIRONMENT_KWARGS_NODES = 4096+_MAX_ENVIRONMENT_REFERENCE_BYTES = 512class EvalsConfigError(ValueError):@@ -220,8 +249,6 @@ def _validate_config(raw: dict[str, Any], config_path: Path) -> dict[str, Any]:harbor["pre_agent_setup"] = _pre_agent_setup(pre_agent_setup_value, config_path)if "agents" in harbor_raw:harbor["agents"] = _agents(harbor_raw["agents"], config_path)- if "environment_kwargs" in harbor_raw:- harbor["environment_kwargs"] = _environment_kwargs(harbor_raw["environment_kwargs"], config_path)out["harbor"] = harbor@@ -356,14 +383,22 @@ def _is_sensitive_environment_kwarg_name(name: str) -> bool:normalized = re.sub(r"([A-Z]+)([A-Z][a-z])", r"\1_\2", name)normalized = re.sub(r"([a-z0-9])([A-Z])", r"\1_\2", normalized)tokens = tuple(part for part in re.sub(r"[^A-Za-z0-9]+", "_", normalized).lower().split("_") if part)+ if _NON_CREDENTIAL_TOKEN_KEY_RE.search("_".join(tokens)):+ return Falseif any(tokenin {"auth","authorization","authentication",+ "bearer",+ "cert",+ "certificate",+ "cookie","credential","credentials",+ "oauth",+ "passphrase","passwd","password","secret",@@ -374,7 +409,8 @@ def _is_sensitive_environment_kwarg_name(name: str) -> bool:):return Trueif any(- pair in {("api", "key"), ("access", "key"), ("private", "key"), ("secret", "key")} for pair in pairwise(tokens)+ pair in {("api", "key"), ("access", "key"), ("private", "key"), ("secret", "key"), ("account", "key")}+ for pair in pairwise(tokens)):return Truecompact = "".join(tokens)@@ -384,11 +420,18 @@ def _is_sensitive_environment_kwarg_name(name: str) -> bool:"accesskey","privatekey","secretkey",+ "serviceaccountkey","auth","authorization","authentication",+ "bearer",+ "cert",+ "certificate",+ "cookie","credential","credentials",+ "oauth",+ "passphrase","passwd","password","secret",@@ -449,8 +492,13 @@ def _environment_kwarg_shape_error(value: Any) -> str | None:if isinstance(current, str):if _contains_credential_bearing_uri(current):return (- f"{_safe_environment_kwarg_path(path)} contains a credential-bearing URI; "- "pass credentials through the host environment instead"+ f"Sensitive key or value detected in {_safe_environment_kwarg_path(path)}: "+ "contains a credential-bearing URI; pass credentials through the host environment instead"+ )+ if any(pattern.search(current) for pattern in _SECRET_VALUE_PATTERNS):+ return (+ f"Sensitive key or value detected in {_safe_environment_kwarg_path(path)}: "+ "contains a credential value; pass credentials through the host environment instead")continueif current is None or isinstance(current, bool | int):@@ -519,7 +567,7 @@ def _reference_kwarg_error(env_mode: str | None, name: str, value: Any) -> str |elif error := _secret_reference_name_error(field_value, label=f"secrets {field_name} field"):return errorreturn None- return f"{name} is secret-bearing; pass credentials through the host environment instead"+ return f"Sensitive key or value detected in {name}: is secret-bearing; pass credentials through the host environment instead"def _environment_kwarg_secret_policy_error(value: dict[str, Any], *, env_mode: str | None) -> str | None:@@ -537,8 +585,8 @@ def _environment_kwarg_secret_policy_error(value: dict[str, Any], *, env_mode: scontinueif _is_sensitive_environment_kwarg_name(raw_key):return (- f"{_safe_environment_kwarg_path(item_path)} is secret-bearing; "- "pass credentials through the host environment instead"+ f"Sensitive key or value detected in {_safe_environment_kwarg_path(item_path)}: "+ "is secret-bearing; pass credentials through the host environment instead")if isinstance(item, dict | list):stack.append((item, item_path))@@ -549,13 +597,23 @@ def _environment_kwarg_secret_policy_error(value: dict[str, Any], *, env_mode: sreturn None-def validate_environment_kwargs(value: Any, *, env_mode: str | None = None) -> dict[str, Any]:+def validate_environment_kwargs(+ value: Any,+ *,+ env_mode: str | None = None,+ allow_gke_infrastructure_kwargs: bool = True,+) -> dict[str, Any]:"""Validate non-secret Harbor constructor kwargs for safe argv forwarding."""if not isinstance(value, dict):raise ValueError("must be a mapping")for raw_name in value:if not isinstance(raw_name, str) or not _ENV_NAME_RE.fullmatch(raw_name):raise ValueError("keys must be valid Python keyword names")+ if not allow_gke_infrastructure_kwargs and raw_name in _GKE_INFRASTRUCTURE_KWARGS:+ raise ValueError(+ f"GKE infrastructure setting '{raw_name}' is not allowed in skill config; "+ "pass via --environment-kwarg or host environment"+ )if error := _environment_kwarg_shape_error(value):raise ValueError(error)if error := _environment_kwarg_secret_policy_error(value, env_mode=env_mode):@@ -702,23 +760,14 @@ def _grading(value: Any, config_path: Path) -> dict[str, str]:return out-def _environment_kwargs(value: Any, config_path: Path) -> dict[str, str]:- """Validate Harbor environment kwargs mapping."""- field = "harbor.environment_kwargs"- if not isinstance(value, dict):- raise EvalsConfigError(f"{config_path}: {field} must be a mapping")+def load_allowed_runtime_env(skill_path: Path) -> dict[str, str]:+ """Load declared harbor.runtime_env mapping from a skill's evals config if present."""+ try:+ cfg, _ = load_evals_config(skill_path)+ env_from_cfg = cfg.get("harbor", {}).get("runtime_env", {})+ if isinstance(env_from_cfg, dict):+ return dict(env_from_cfg)+ except Exception:+ pass+ return {}- out: dict[str, str] = {}- for key, val in value.items():- if not isinstance(key, str) or not key.strip():- raise EvalsConfigError(f"{config_path}: {field} keys must be non-empty strings")- stripped_key = key.strip()- if stripped_key not in _SKILL_SAFE_ENVIRONMENT_KWARGS:- raise EvalsConfigError(- f"{config_path}: {field}.{stripped_key} cannot be configured in skill evals/config.yml. "- "Infrastructure, network, mount, endpoint, and credential settings must be provided via CLI flags (--ek) or host environment variables."- )- if not isinstance(val, str) or not val.strip():- raise EvalsConfigError(f"{config_path}: {field}.{stripped_key} must be a non-empty string")- out[stripped_key] = val.strip()- return outdiff --git a/src/skillevaluator/tier3/evals_spec.py b/src/skillevaluator/tier3/evals_spec.pyindex 6aa960f4..dc617479 100644--- a/src/skillevaluator/tier3/evals_spec.py+++ b/src/skillevaluator/tier3/evals_spec.py@@ -792,17 +792,9 @@ def _validate_mcp_toml(path: Path, results: list[CheckResult]) -> None:)return- allowed_runtime_env: dict[str, str] = {}- try:- from .evals_config import load_evals_config+ from .evals_config import load_allowed_runtime_env- skill_root = path.parent.parent.parent- cfg, _ = load_evals_config(skill_root)- env_from_cfg = cfg.get("harbor", {}).get("runtime_env", {})- if isinstance(env_from_cfg, dict):- allowed_runtime_env = env_from_cfg- except Exception:- pass+ allowed_runtime_env = load_allowed_runtime_env(path.parent.parent.parent)for i, s in enumerate(servers):if not isinstance(s, dict) or "name" not in s:diff --git a/src/skillevaluator/tier3/harbor/__init__.py b/src/skillevaluator/tier3/harbor/__init__.pyindex d4a768f1..a962bcff 100644--- a/src/skillevaluator/tier3/harbor/__init__.py+++ b/src/skillevaluator/tier3/harbor/__init__.py@@ -31,13 +31,17 @@ LOCAL_AGENT_IMPORT_PATHS = {"codex": "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorLocalCodex","opencode": "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorLocalOpenCode",}-from skillevaluator.tier3.harbor.local_agents import CONTAINER_AGENT_IMPORT_PATHS+CONTAINER_AGENT_IMPORT_PATHS = {+ "claude-code": "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorClaudeCode",+}# Docker wrappers retain Harbor's installers while avoiding redundant Node/npm-# packages on the branches that bootstrap their own runtime.+# packages on the branches that bootstrap their own runtime. Claude Code uses the+# container wrapper, which builds on that installer fix and adds Vertex AI+# routing and rich MCP server declarations.DOCKER_AGENT_IMPORT_PATHS = {"codex": "skillevaluator.tier3.harbor.installed_agents:SkillEvaluatorCodex",- "claude-code": "skillevaluator.tier3.harbor.installed_agents:SkillEvaluatorClaudeCode",+ "claude-code": CONTAINER_AGENT_IMPORT_PATHS["claude-code"],}HARBOR_AGENTS_SUPPORTED = frozenset(diff --git a/src/skillevaluator/tier3/harbor/adapter.py b/src/skillevaluator/tier3/harbor/adapter.pyindex 092f0129..8900d2dd 100644--- a/src/skillevaluator/tier3/harbor/adapter.py+++ b/src/skillevaluator/tier3/harbor/adapter.py@@ -243,7 +243,6 @@ _VERIFIER_PROVIDER_ENV_VARS = frozenset("SKILL_EVAL_LLM_MODEL","SKILL_EVAL_LLM_API_KEY","SKILL_EVAL_LLM_BASE_URL",- "SKILL_EVAL_LLM_CREDENTIAL_SOURCE","OPENAI_API_KEY","OPENAI_BASE_URL","ANTHROPIC_API_KEY",@@ -314,6 +313,7 @@ _VERIFIER_PROVIDER_ENV_VARS = frozenset("BOTOCORE_TCP_KEEPALIVE",*_VERIFIER_BUDGET_ENV_VARS,*_VERIFIER_RETRY_ENV_VARS,+ "SKILL_EVAL_LLM_CREDENTIAL_SOURCE",})_RUNTIME_PROCESS_CONTROL_ENV_NAMES = frozenset(@@ -2089,15 +2089,9 @@ def _load_mcp_servers() -> list[dict[str, Any]]:"""Load MCP server declarations from evals/environment/mcp_servers.toml."""if allowed_runtime_env is None:- try:- from skillevaluator.tier3.evals_config import load_evals_config+ from skillevaluator.tier3.evals_config import load_allowed_runtime_env- cfg, _ = load_evals_config(skill_path)- env_from_cfg = cfg.get("harbor", {}).get("runtime_env", {})- if isinstance(env_from_cfg, dict):- allowed_runtime_env = env_from_cfg- except Exception:- pass+ allowed_runtime_env = load_allowed_runtime_env(skill_path)evals_dir = skill_path / "evals"environment_dir = evals_dir / "environment"diff --git a/src/skillevaluator/tier3/harbor/gke_environment.py b/src/skillevaluator/tier3/harbor/gke_environment.pyindex 27337b98..9f288cf4 100644--- a/src/skillevaluator/tier3/harbor/gke_environment.py+++ b/src/skillevaluator/tier3/harbor/gke_environment.py@@ -7,6 +7,7 @@ from __future__ import annotationsimport asyncioimport contextlib+import contextvarsimport osimport reimport shlex@@ -16,7 +17,13 @@ import urllib.parsefrom collections.abc import Mappingfrom typing import TYPE_CHECKING, Any-from harbor.environments.gke import GKEEnvironment, stream+from harbor.environments.gke import (+ GKEEnvironment,+ GKEExecStreamClosedError,+ GKEPodNotFoundError,+ _is_pod_not_found_error,+ stream,+)from kubernetes import client as k8s_clientfrom skillevaluator.provider_config import (@@ -73,6 +80,7 @@ _DNS_PORT = 53_DNS_PROTOCOLS = frozenset({"UDP", "TCP"})_EXEC_QUERY_COMMAND_RE = re.compile(r"(?P<sep>[?&])command=[^\s\"']+")_GKE_CLOUD_DNS_METADATA_CIDR = "169.254.169.254/32"+_GKE_AUTOPILOT_LABEL_PREFIX = "autopilot.gke.io"_GKE_DIRECT_WIF_PRINCIPAL_ANNOTATION = "iam.gke.io/return-principal-id-as-email"_GKE_METADATA_PROBE_URLS = ("http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token",@@ -84,6 +92,8 @@ _KUBE_DNS_POD_LABEL_KEY = "k8s-app"_KUBE_DNS_POD_LABEL_VALUE = "kube-dns"_KUBE_SYSTEM_NAMESPACE_LABEL_KEY = "kubernetes.io/metadata.name"_KUBE_SYSTEM_NAMESPACE_LABEL_VALUE = "kube-system"+# The name segment of a qualified Kubernetes label key.+_LABEL_KEY_NAME_RE = re.compile(r"[A-Za-z0-9](?:[-A-Za-z0-9_.]{0,61}[A-Za-z0-9])?")_MAX_TRANSIENT_EXEC_ATTEMPTS = 4_METADATA_PROBE_HTTP_TIMEOUT_SEC = 2_METADATA_PROBE_ISOLATED_SENTINEL = "SKILLEVALUATOR_METADATA_ISOLATED_OK"@@ -258,6 +268,8 @@ def _extract_pod_missing_exec_detail(exc: BaseException) -> str | None:cur: BaseException | None = excwhile cur is not None and id(cur) not in seen:seen.add(id(cur))+ if isinstance(cur, GKEPodNotFoundError) or _is_pod_not_found_error(cur):+ return str(cur)if isinstance(cur, Exception) and _is_k8s_http_status(cur, 404, "not found"):return str(cur)text = str(cur)@@ -272,6 +284,33 @@ def _extract_pod_missing_exec_detail(exc: BaseException) -> str | None:return None+class _ExecDispatchRecord:+ """Count exec stream handshakes for one exec() attempt.++ The Kubernetes exec API starts the command only after the WebSocket+ handshake succeeds, so an attempt whose handshakes all failed never ran+ the command and is safe to retry.+ """++ __slots__ = ("attempted", "established")++ def __init__(self) -> None:+ self.attempted = 0+ self.established = 0++ @property+ def failed_before_dispatch(self) -> bool:+ return self.attempted > 0 and self.established == 0+++# A mutable record per exec() attempt; asyncio.to_thread copies the context, so+# the wrapped API call updates the same record object the caller holds.+_EXEC_DISPATCH_RECORD: contextvars.ContextVar[_ExecDispatchRecord | None] = contextvars.ContextVar(+ "skillevaluator_gke_exec_dispatch_record",+ default=None,+)++def _ensure_default_exec_container_routing(api: Any, *, default_container: str = "main") -> None:"""Ensure untargeted connect_get_namespaced_pod_exec calls default to the main task container."""orig = getattr(api, "connect_get_namespaced_pod_exec", None)@@ -281,17 +320,23 @@ def _ensure_default_exec_container_routing(api: Any, *, default_container: str =def _wrapped_exec(*args: Any, **kwargs: Any) -> Any:if kwargs.get("container") is None:kwargs["container"] = default_container+ record = _EXEC_DISPATCH_RECORD.get()+ if record is not None:+ record.attempted += 1try:- return orig(*args, **kwargs)+ response = orig(*args, **kwargs)except Exception as exc:missing_detail = _extract_pod_missing_exec_detail(exc)if missing_detail is not None:pod_name = str(args[0] if len(args) > 0 else kwargs.get("name", "unknown"))namespace = str(args[1] if len(args) > 1 else kwargs.get("namespace", "default"))- raise RuntimeError(+ raise GKEPodNotFoundError(f"Pod {pod_name} in namespace {namespace} no longer exists (deleted or preempted): {missing_detail}") from excraise+ if record is not None:+ record.established += 1+ return response_wrapped_exec._skillevaluator_default_container_wrapped = True # type: ignore[attr-defined]_wrapped_exec.__self__ = getattr(orig, "__self__", api) # type: ignore[attr-defined]@@ -391,9 +436,9 @@ def _redact_exec_stderr(stderr: str | None, env_values: Mapping[str, str] | Noneif secret in redacted:redacted = redacted.replace(secret, "<redacted>")redacted = _redact_truncated_secret_suffix(redacted, secret_variants)+ redacted = _GOOGLE_CREDENTIAL_RE.sub("<redacted>", redacted)redacted = redact_secrets_in_log_line(redacted, extra_secret_values=secret_variants)- redacted = redact_sensitive_text(redacted)- return _GOOGLE_CREDENTIAL_RE.sub("<redacted>", redacted)+ return redact_sensitive_text(redacted)def _call_k8s_with_timeout(@@ -679,6 +724,16 @@ def ensure_gke_metadata_network_policy(_validate_existing_metadata_network_policy(reread, namespace=namespace)+def _qualified_label_key_prefix(key: object) -> str | None:+ """Return the prefix of a well-formed ``prefix/name`` Kubernetes label key, or None."""+ if not isinstance(key, str):+ return None+ prefix, separator, name = key.partition("/")+ if not separator or not prefix or not _LABEL_KEY_NAME_RE.fullmatch(name):+ return None+ return prefix++def _is_autopilot_node(node: Any) -> bool:"""Return True if a Kubernetes V1Node exhibits GKE Autopilot naming or label conventions."""metadata = getattr(node, "metadata", None)@@ -687,7 +742,7 @@ def _is_autopilot_node(node: Any) -> bool:return Truelabels = getattr(metadata, "labels", None)if isinstance(labels, Mapping):- if any(str(k).startswith("autopilot.gke.io/") for k in labels):+ if any(_qualified_label_key_prefix(key) == _GKE_AUTOPILOT_LABEL_PREFIX for key in labels):return Trueif parse_optional_bool_flag(labels.get("cloud.google.com/gke-autopilot")) is True:return True@@ -697,6 +752,23 @@ def _is_autopilot_node(node: Any) -> bool:class SkillEvaluatorGKEEnvironment(GKEEnvironment):"""Enforce least-privilege pod identity, exec readiness, and ADC refresh for GKE evaluation pods."""+ @classmethod+ def preflight(cls) -> None:+ """Verify gcloud and Kubernetes credentials, supporting split KUBECONFIG paths."""+ if not shutil.which("gcloud"):+ raise SystemExit(+ "GKE requires the gcloud CLI to be installed. See https://cloud.google.com/sdk/docs/install"+ )+ from skillevaluator.tier3.harbor.kubeconfig import _resolve_single_kubeconfig++ if _resolve_single_kubeconfig() is None:+ raise SystemExit(+ "GKE requires Kubernetes credentials. Run "+ "'gcloud container clusters get-credentials <CLUSTER> "+ "--region <REGION>' to configure credentials, or set the "+ "KUBECONFIG environment variable."+ )+def __init__(self,*args: Any,@@ -712,6 +784,18 @@ class SkillEvaluatorGKEEnvironment(GKEEnvironment):self._detected_autopilot: bool | None = Nonesuper().__init__(*args, **kwargs)+ def _get_core_api(self) -> Any:+ """Return the initialized Kubernetes CoreV1Api client without raising on uninitialized property access."""+ instance_dict = getattr(self, "__dict__", {})+ if "_api" in instance_dict and instance_dict["_api"] is not None:+ return instance_dict["_api"]+ core_api = getattr(self, "_core_api", None)+ if core_api is not None:+ return core_api+ with contextlib.suppress(Exception):+ return getattr(self, "_api", None)+ return None+def _is_workload_identity_enabled(self) -> bool:"""Return True when the operator explicitly opted into GKE Workload Identity."""if getattr(self, "_allow_workload_identity", False):@@ -819,7 +903,7 @@ class SkillEvaluatorGKEEnvironment(GKEEnvironment):persistent = getattr(self, "_persistent_env", None)if isinstance(persistent, dict):persistent.setdefault("GCE_METADATA_HOST", _BLOCKED_GCE_METADATA_HOST)- api = getattr(self, "_api", None)+ api = self._get_core_api()namespace = getattr(self, "namespace", "default")if api is None:raise RuntimeError(@@ -872,7 +956,7 @@ class SkillEvaluatorGKEEnvironment(GKEEnvironment):async def _delete_unisolated_pod_best_effort(self) -> None:"""Delete the evaluation pod on metadata isolation verification failure."""- api = getattr(self, "_api", None)+ api = self._get_core_api()delete_pod = getattr(api, "delete_namespaced_pod", None)if callable(delete_pod):with contextlib.suppress(Exception):@@ -1050,8 +1134,8 @@ class SkillEvaluatorGKEEnvironment(GKEEnvironment):namespace=namespace,)except Exception as exc:- if _is_k8s_http_status(exc, 404, "not found"):- raise RuntimeError(+ if _is_pod_not_found_error(exc) or _is_k8s_http_status(exc, 404, "not found"):+ raise GKEPodNotFoundError(f"Pod {self.pod_name} in namespace {namespace} no longer exists (deleted or preempted) ""and cannot accept exec.") from exc@@ -1077,7 +1161,7 @@ class SkillEvaluatorGKEEnvironment(GKEEnvironment):async def _wait_for_container_exec_ready(self, max_attempts: int = 60) -> None:"""Wait until the GKE kubelet accepts exec streams and verify metadata server isolation."""if not getattr(self, "_compose_mode", False):- api = getattr(self, "_api", None)+ api = self._get_core_api()if api is not None:_ensure_default_exec_container_routing(api, default_container="main")@@ -1101,6 +1185,15 @@ class SkillEvaluatorGKEEnvironment(GKEEnvironment):if not self._is_workload_identity_enabled():await self._verify_in_pod_metadata_isolation()+ def _merge_env(self, env: dict[str, str] | None) -> dict[str, str] | None:+ """Merge environment variables while ensuring _exec_env_overlays exists for object.__new__ instances."""+ if not hasattr(self, "_exec_env_overlays"):+ self._exec_env_overlays = contextvars.ContextVar(+ f"exec_env_overlays_{id(self)}",+ default=(),+ )+ return super()._merge_env(env)+async def exec(self,command: str,@@ -1108,8 +1201,9 @@ class SkillEvaluatorGKEEnvironment(GKEEnvironment):env: dict[str, str] | None = None,timeout_sec: int | None = None,user: str | int | None = None,+ service: str | None = None,) -> ExecResult:- """Refresh ADC-sourced Vertex OpenAPI credentials and retry transient kubelet exec errors."""+ """Refresh ADC-sourced Vertex OpenAPI credentials and retry kubelet errors raised before dispatch."""merged = dict(self._merge_env(env) or {})fresh_token = await asyncio.to_thread(refresh_host_vertex_adc_environment,@@ -1138,14 +1232,38 @@ class SkillEvaluatorGKEEnvironment(GKEEnvironment):result: ExecResult | None = Nonefor attempt in range(_MAX_TRANSIENT_EXEC_ATTEMPTS):- result = await super().exec(- command=command,- cwd=cwd,- env=env,- timeout_sec=timeout_sec,- user=user,- )- if result.return_code == 0 or not _is_transient_kubelet_exec_error(result.stderr):+ dispatch = _ExecDispatchRecord()+ dispatch_token = _EXEC_DISPATCH_RECORD.set(dispatch)+ try:+ result = await super().exec(+ command=command,+ cwd=cwd,+ env=env,+ timeout_sec=timeout_sec,+ user=user,+ **({"service": service} if service is not None else {}),+ )+ except GKEExecStreamClosedError as exc:+ # The stream was open, so the command may have run before its+ # status was lost. Replaying it could repeat its side effects.+ redacted_msg = _redact_exec_stderr(str(exc), merged) or str(exc)+ raise GKEExecStreamClosedError(redacted_msg) from None+ finally:+ _EXEC_DISPATCH_RECORD.reset(dispatch_token)+ if (+ result.return_code != 0+ and result.stderr+ and _extract_pod_missing_exec_detail(RuntimeError(result.stderr)) is not None+ ):+ redacted_msg = _redact_exec_stderr(result.stderr, merged) or result.stderr+ raise GKEPodNotFoundError(redacted_msg)+ # Retry only a kubelet error from an attempt whose exec handshake+ # never completed; a dispatched command's own stderr can match too.+ if (+ result.return_code == 0+ or not dispatch.failed_before_dispatch+ or not _is_transient_kubelet_exec_error(result.stderr)+ ):breakif attempt < _MAX_TRANSIENT_EXEC_ATTEMPTS - 1:await asyncio.sleep(1.5 * (attempt + 1))diff --git a/src/skillevaluator/tier3/harbor/kubeconfig.py b/src/skillevaluator/tier3/harbor/kubeconfig.pynew file mode 100644index 00000000..1ee97b8f--- /dev/null+++ b/src/skillevaluator/tier3/harbor/kubeconfig.py@@ -0,0 +1,154 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Kubeconfig path resolution and multi-file merging for Harbor GKE mode."""++from __future__ import annotations++import logging+import os+from collections.abc import Sequence+from pathlib import Path+from typing import Any++logger = logging.getLogger(__name__)++_KUBECONFIG_FILE_PATH_KEYS = frozenset(+ {+ "certificate-authority",+ "client-certificate",+ "client-key",+ "tokenFile",+ "idp-certificate-authority",+ }+)++_MERGED_KUBECONFIG_CACHE: dict[str, Path] = {}+++def _resolve_relative_kubeconfig_path(+ raw_value: str,+ base_dir: Path | None,+ fallback_dirs: Sequence[Path] = (),+) -> str:+ """Resolve a relative kubeconfig path against its source file directory."""+ expanded = Path(raw_value.strip()).expanduser()+ if expanded.is_absolute():+ return str(expanded)+ if base_dir is not None:+ return str((base_dir / expanded).resolve())+ for directory in fallback_dirs:+ candidate = (directory / expanded).resolve()+ if candidate.exists():+ return str(candidate)+ if fallback_dirs:+ return str((fallback_dirs[0] / expanded).resolve())+ return raw_value+++def _unwrap_kubeconfig_node(+ obj: Any,+ source_path: str | None = None,+ fallback_dirs: Sequence[Path] = (),+) -> Any:+ """Recursively unwrap Kubernetes ConfigNode wrappers while absolutizing relative file paths."""+ node_path = getattr(obj, "path", None) or source_path+ if hasattr(obj, "value"):+ return _unwrap_kubeconfig_node(obj.value, source_path=node_path, fallback_dirs=fallback_dirs)+ if isinstance(obj, dict):+ out: dict[str, Any] = {}+ base_dir = Path(node_path).expanduser().resolve().parent if node_path else None+ for k, v in obj.items():+ unwrapped = _unwrap_kubeconfig_node(v, source_path=node_path, fallback_dirs=fallback_dirs)+ if (+ isinstance(unwrapped, str)+ and unwrapped.strip()+ and (k in _KUBECONFIG_FILE_PATH_KEYS or (k == "command" and ("/" in unwrapped or os.sep in unwrapped)))+ ):+ unwrapped = _resolve_relative_kubeconfig_path(unwrapped, base_dir, fallback_dirs)+ out[k] = unwrapped+ return out+ if isinstance(obj, list):+ return [_unwrap_kubeconfig_node(item, source_path=node_path, fallback_dirs=fallback_dirs) for item in obj]+ return obj+++def _resolve_single_kubeconfig(raw_kubeconfig: str | None = None) -> Path | None:+ """Resolve a single existing kubeconfig file path from KUBECONFIG or default location.++ If multiple path entries are specified, merge them into a single temporary kubeconfig+ file preserving all clusters, contexts, and credentials, restricted to permissions 0600.+ """+ raw = raw_kubeconfig if raw_kubeconfig is not None else os.environ.get("KUBECONFIG")+ if raw:+ valid_candidates: list[Path] = []+ for entry in raw.split(os.pathsep):+ cleaned = entry.strip()+ if cleaned:+ candidate = Path(cleaned).expanduser().resolve()+ if candidate.is_file():+ valid_candidates.append(candidate)+ if valid_candidates:+ if len(valid_candidates) == 1:+ return valid_candidates[0]+ paths_str = os.pathsep.join(str(p) for p in valid_candidates)+ cache_key = os.pathsep.join(f"{p}:{p.stat().st_mtime_ns}:{p.stat().st_size}" for p in valid_candidates)+ cached_path = _MERGED_KUBECONFIG_CACHE.get(cache_key)+ if cached_path is not None and cached_path.is_file():+ return cached_path+ try:+ import atexit+ import tempfile++ import yaml+ from kubernetes.config.kube_config import KubeConfigMerger++ with tempfile.TemporaryDirectory(prefix="kubeconfig_stage_") as stage_dir:+ staged_paths: list[str] = []+ for idx, cand in enumerate(valid_candidates):+ cand_data = yaml.safe_load(cand.read_text(encoding="utf-8"))+ if isinstance(cand_data, dict):+ cand_data = _unwrap_kubeconfig_node(cand_data, source_path=str(cand))+ staged_file = Path(stage_dir) / f"cand_{idx}.yaml"+ staged_file.write_text(+ yaml.safe_dump(cand_data, default_flow_style=False),+ encoding="utf-8",+ )+ staged_file.chmod(0o600)+ staged_paths.append(str(staged_file))+ else:+ staged_paths.append(str(cand))+ merger = KubeConfigMerger(os.pathsep.join(staged_paths))+ fallback_dirs = [p.parent for p in valid_candidates]+ merged_dict = _unwrap_kubeconfig_node(merger.config, fallback_dirs=fallback_dirs)+ if not merged_dict or not isinstance(merged_dict, dict):+ logger.error("KubeConfigMerger produced empty or invalid configuration from %s", paths_str)+ return None+ with tempfile.NamedTemporaryFile(+ mode="w",+ encoding="utf-8",+ prefix="merged_kubeconfig_",+ suffix=".yaml",+ delete=False,+ ) as handle:+ merged_path = Path(handle.name)+ yaml.safe_dump(merged_dict, handle, default_flow_style=False)+ merged_path.chmod(0o600)+ _MERGED_KUBECONFIG_CACHE[cache_key] = merged_path+ atexit.register(lambda: merged_path.unlink(missing_ok=True))+ logger.info(+ "Merged %d kubeconfig files into temporary file %s",+ len(valid_candidates),+ merged_path,+ )+ return merged_path+ except Exception as exc:+ logger.error(+ "Failed to merge multiple kubeconfigs (%s): %s",+ raw,+ exc,+ )+ return None+ return None+ default_path = (Path.home() / ".kube" / "config").resolve()+ return default_path if default_path.is_file() else Nonediff --git a/src/skillevaluator/tier3/harbor/local_agents.py b/src/skillevaluator/tier3/harbor/local_agents.pyindex 8ab871f5..4ff2e80f 100644--- a/src/skillevaluator/tier3/harbor/local_agents.py+++ b/src/skillevaluator/tier3/harbor/local_agents.py@@ -8,25 +8,28 @@ from __future__ import annotationsimport asyncioimport copyimport json+import loggingimport osimport reimport secretsimport shleximport tempfileimport uuid-from collections.abc import Awaitable, Callable+from collections.abc import Awaitable, Callable, Mappingfrom contextlib import suppressfrom pathlib import Path, PurePosixPathfrom typing import TYPE_CHECKING, Anyfrom urllib.parse import urlsplitfrom harbor.agents.installed.base import NonZeroAgentExitCodeError-from harbor.agents.installed.claude_code import ClaudeCodefrom harbor.agents.installed.opencode import OpenCode+from harbor.agents.model_connection import ResolvedModelConnection, without_inferred_endpointfrom harbor.models.trial.paths import EnvironmentPaths+from harbor.utils.env import parse_bool_env_valuefrom skillevaluator.provider_config import is_claude_5_5_or_later, is_gpt6_model-from skillevaluator.tier3.harbor.installed_agents import SkillEvaluatorClaudeCode, SkillEvaluatorCodex+from skillevaluator.tier3.harbor.installed_agents import SkillEvaluatorClaudeCode as _InstalledClaudeCode+from skillevaluator.tier3.harbor.installed_agents import SkillEvaluatorCodexfrom skillevaluator.tier3.harbor.nvidia_build_bridge import (MAX_REQUESTS_PER_BRIDGE,RunningBridge,@@ -55,6 +58,8 @@ _CLAUDE_CODE_MIN_VERSION_FOR_CLAUDE_5_5 = (2, 1, 280)_CODEX_MIN_VERSION_FOR_GPT_6 = (0, 159, 1)_CLI_VERSION_RE = re.compile(r"(\d+)\.(\d+)\.(\d+)")+logger = logging.getLogger(__name__)+async def _await_task_uninterruptibly(task: asyncio.Task[Any],@@ -130,8 +135,104 @@ def _rewrite_launcher_segment(command: str, rewrite: Callable[[str], str]) -> streturn f"{rewrite(launcher)}{separator}{prompt}"-class SkillEvaluatorClaudeCode(ClaudeCode):- """Wrap Claude Code to preserve rich MCP server declarations (transport, headers)."""+class SkillEvaluatorClaudeCode(_InstalledClaudeCode):+ """Wrap Claude Code to preserve rich MCP server declarations and Vertex AI routing.++ It extends the installed-agent wrapper, so Harbor's installer still skips+ the distribution Node/npm packages that the native installer does not need.+ """++ def _env_sources(self) -> tuple[Mapping[str, str], ...]:+ """Environment sources in runtime precedence order, safe for partially initialized instances."""+ sources: list[Mapping[str, str]] = []+ resolved = getattr(self, "_resolved_env_vars", None)+ if isinstance(resolved, Mapping):+ sources.append(resolved)+ extra = getattr(self, "_extra_env", None)+ if isinstance(extra, Mapping):+ sources.append(extra)+ sources.append(os.environ)+ return tuple(sources)++ def _uses_vertex_auth(self) -> bool:+ """Return whether Claude Code is configured to route through Vertex AI."""+ try:+ return bool(+ parse_bool_env_value(+ self._get_env("CLAUDE_CODE_USE_VERTEX"),+ name="CLAUDE_CODE_USE_VERTEX",+ default=False,+ )+ )+ except (ValueError, AttributeError):+ logger.debug("Ignoring invalid boolean environment value CLAUDE_CODE_USE_VERTEX")+ return False++ @property+ def model_connection(self) -> ResolvedModelConnection:+ """Suppress Anthropic API endpoint inference when Vertex AI routing is active."""+ access = super().model_connection+ if self._uses_vertex_auth():+ return without_inferred_endpoint(access)+ return access++ def _resolve_auth_env(self) -> dict[str, str]:+ """Resolve Claude Code authentication environment including Vertex AI variables."""+ if not self._uses_vertex_auth():+ return super()._resolve_auth_env()++ env: dict[str, str] = {"CLAUDE_CODE_USE_VERTEX": "1"}+ for vertex_var in (+ "CLOUD_ML_REGION",+ "ANTHROPIC_VERTEX_PROJECT_ID",+ "GOOGLE_CLOUD_PROJECT",+ "GCLOUD_PROJECT",+ "CLOUDSDK_CORE_PROJECT",+ "GOOGLE_APPLICATION_CREDENTIALS",+ "CLOUDSDK_CONFIG",+ "ANTHROPIC_DEFAULT_OPUS_MODEL",+ "ANTHROPIC_DEFAULT_SONNET_MODEL",+ "ANTHROPIC_DEFAULT_HAIKU_MODEL",+ "CLAUDE_CODE_SUBAGENT_MODEL",+ ):+ val = (self._get_env(vertex_var) or "").strip()+ if val:+ env[vertex_var] = val+ if (self._get_env("DISABLE_PROMPT_CACHING") or "").strip() == "1":+ env["DISABLE_PROMPT_CACHING"] = "1"+ return env++ async def exec_as_agent(+ self,+ environment: BaseEnvironment,+ command: str,+ env: dict[str, str] | None = None,+ cwd: str | None = None,+ timeout_sec: int | None = None,+ ):+ if self._uses_vertex_auth():+ command = (+ f"unset ANTHROPIC_API_KEY ANTHROPIC_AUTH_TOKEN ANTHROPIC_BASE_URL CLAUDE_CODE_OAUTH_TOKEN; {command}"+ )+ if env:+ env = {+ k: v+ for k, v in env.items()+ if k+ not in {+ "ANTHROPIC_API_KEY",+ "ANTHROPIC_AUTH_TOKEN",+ "ANTHROPIC_BASE_URL",+ "CLAUDE_CODE_OAUTH_TOKEN",+ }+ }+ return await super().exec_as_agent(+ environment,+ command=command,+ env=env,+ cwd=cwd,+ timeout_sec=timeout_sec,+ )def _resolve_task_path(self) -> Path | None:"""Resolve the task directory path from trial config.json."""@@ -179,11 +280,6 @@ class SkillEvaluatorClaudeCode(ClaudeCode):if not mcp_servers:return super()._build_register_mcp_servers_command()- from skillevaluator.tier3.harbor.adapter import validate_mcp_server_declarations-- runtime_env = self._resolve_task_runtime_env()- validate_mcp_server_declarations(mcp_servers, allowed_runtime_env=runtime_env)-servers: dict[str, dict[str, Any]] = {}for server in mcp_servers:name = server.get("name")@@ -977,11 +1073,10 @@ class SkillEvaluatorLocalNvidiaBuildClaudeCode("""Managed local Claude Code CLI routed through an authenticated host bridge."""-LOCAL_AGENT_IMPORT_PATHS = {- "claude-code": "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorLocalClaudeCode",- "codex": "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorLocalCodex",- "opencode": "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorLocalOpenCode",-}+from skillevaluator.tier3.harbor import ( # noqa: F401+ CONTAINER_AGENT_IMPORT_PATHS,+ LOCAL_AGENT_IMPORT_PATHS,+)NVIDIA_BUILD_AGENT_IMPORT_PATHS = {"claude-code": "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorNvidiaBuildClaudeCode",@@ -992,7 +1087,3 @@ NVIDIA_BUILD_LOCAL_AGENT_IMPORT_PATHS = {"claude-code": "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorLocalNvidiaBuildClaudeCode","codex": "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorLocalNvidiaBuildCodex",}--CONTAINER_AGENT_IMPORT_PATHS = {- "claude-code": "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorClaudeCode",-}diff --git a/src/skillevaluator/tier3/harbor/progress.py b/src/skillevaluator/tier3/harbor/progress.pyindex 4f583cec..470ce074 100644--- a/src/skillevaluator/tier3/harbor/progress.py+++ b/src/skillevaluator/tier3/harbor/progress.py@@ -35,7 +35,7 @@ _SECRET_ASSIGNMENT_RE = re.compile(rf"(?<![A-Za-z0-9]){_SECRET_KEY_WORDS}\b\s*=)\s*)"r"(?:\"[^\"]*\"|'[^']*'|[^\s,;]+)")-_SECRET_ENV_NAME_RE = re.compile(r"(?i)(?:api[_-]?key|access[_-]?key|auth|credential|password|secret|token)")+_SECRET_ENV_NAME_RE = re.compile(r"(?i)(?:api[_-]?key|access[_-]?key|auth|credential|password|secret|token(?!s))")# Whole environment values shorter than this are never treated as exact secrets;# shorter credential fragments come only from URI userinfo.MIN_EXACT_SECRET_CHARS = 4diff --git a/src/skillevaluator/tier3/harbor/runner.py b/src/skillevaluator/tier3/harbor/runner.pyindex 947925c9..d75392db 100644--- a/src/skillevaluator/tier3/harbor/runner.py+++ b/src/skillevaluator/tier3/harbor/runner.py@@ -24,9 +24,9 @@ import tempfileimport threadingimport timeimport tomllib-from collections.abc import Mapping, Sequence+from collections.abc import Iterable, Iterator, Mapping, Sequencefrom concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, as_completed, wait-from contextlib import ExitStack+from contextlib import ExitStack, contextmanager, suppressfrom dataclasses import dataclass, field, replacefrom datetime import UTC, datetimefrom functools import wraps@@ -42,10 +42,12 @@ from skillevaluator.provider_config import (ADC_DISCOVERY_ENV_VARS,CHAT_DEFAULT_ANTHROPIC,CHAT_DEFAULT_NVIDIA,+ CHAT_DEFAULT_OPENAI,CREDENTIAL_EXPIRY_ENV,CREDENTIAL_SOURCE_ADC,CREDENTIAL_SOURCE_ENV,GATEWAY_AGENT_DEFAULT_MODELS,+ OPENCODE_OPENAI_DEFAULT_MODEL,ProviderConfig,ProviderConfigurationError,_get_google_access_token,@@ -53,24 +55,30 @@ from skillevaluator.provider_config import (_normalize_anthropic_base_url,_normalize_expiry_epoch,compute_adc_job_timeout,+ effective_reasoning_effort,get_adc_token_expiry,+ is_claude_5_5_or_later,refresh_host_vertex_adc_environment,resolve_llm_provider,)-from skillevaluator.tier3.eval_core.secret_redaction import (- LOG_CRSR_RE,- LOG_JWT_RE,- LOG_NVAPI_RE,- LOG_SK_RE,- LOG_YA29_RE,- OPENSHIFT_TOKEN_RE,+from skillevaluator.publication_identity import (+ PUBLICATION_ANY_DEPTH_EXCLUDED_NAMES,+ publication_source_path_is_excluded,+ publication_target_from_path,)+from skillevaluator.source_identity import normalized_evaluated_source+from skillevaluator.tier3.case_ids import validate_case_ids+from skillevaluator.tier3.eval_core.llm_judge import JUDGE_REASONING_EFFORTfrom skillevaluator.tier3.evals_config import (_GKE_INFRASTRUCTURE_KWARGS,- _SKILL_SAFE_ENVIRONMENT_KWARGS,+ MAX_HARBOR_TIMEOUT_MULTIPLIER,EvalsConfigError,+ _is_sensitive_environment_kwarg_name,+ encode_environment_kwarg,load_evals_config,+ validate_environment_kwargs,)+from skillevaluator.tier3.harbor import CONTAINER_AGENT_IMPORT_PATHS, DOCKER_AGENT_IMPORT_PATHSfrom skillevaluator.tier3.harbor.adapter import (_RUNTIME_PROCESS_CONTROL_ENV_NAMES,_RUNTIME_PROCESS_CONTROL_ENV_PREFIXES,@@ -105,7 +113,8 @@ from skillevaluator.tier3.harbor.gke_environment import (inspect_bound_gcp_service_account,is_gke_workload_identity_allowed,)-from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS, score_definition+from skillevaluator.tier3.harbor.kubeconfig import _resolve_single_kubeconfig+from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS, score_policy_for_metricsfrom skillevaluator.tier3.harbor.progress import (MIN_EXACT_SECRET_CHARS,NullProgressReporter,@@ -147,8 +156,21 @@ from skillevaluator.tier3.output_provenance import (write_output_file_atomically,)from skillevaluator.tier3.results_location import publish_latest_results-from skillevaluator.tier3_environments import DEFAULT_ENV_MODE, ENV_MODE_LOCAL, HARBOR_ENV_MODES-from skillevaluator.utils.redaction import is_sensitive_key+from skillevaluator.tier3_environments import (+ DEFAULT_ENV_MODE,+ ENV_MODE_LOCAL,+ HARBOR_DOCKER_ENGINE_ENV_MODES,+ HARBOR_ENV_MODES,+ HARBOR_ENVIRONMENT_ALIAS_KWARGS,+ HARBOR_ENVIRONMENT_EXTRAS,+ HARBOR_ENVIRONMENT_KWARGS,+ HARBOR_VERSION,+ SKILLEVALUATOR_GKE_EXTRA_ENVIRONMENT_KWARGS,+ harbor_environment_type,+)+from skillevaluator.utils.path_security import matches_filesystem_name+from skillevaluator.utils.redaction import is_sensitive_key, redact_sensitive_text+from skillevaluator.utils.secure_fs import SecurePathError, SecureRootlogger = logging.getLogger(__name__)_MAX_REPO_CONTEXT_METADATA_BYTES = 16 * 1024 * 1024@@ -743,9 +765,14 @@ _NVIDIA_BUILD_FILE_SENTINEL = "skillevaluator-file-backed-nvidia-key"_NVIDIA_BUILD_KEY_FILE_ENV = "SKILLEVALUATOR_NVIDIA_API_KEY_FILE"_NVIDIA_BUILD_AGENT_DEFAULT_MODEL = CHAT_DEFAULT_NVIDIA_GATEWAY_CODEX_IMPORT_PATH = "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorGatewayCodex"-# The OpenAI-compatible gateway Codex wrapper only routes the stock agent to the-# operator's gateway, so it is the one custom agent allowed on Harbor-native backends.-_NATIVE_BACKEND_AGENT_IMPORT_PATHS = frozenset({_GATEWAY_CODEX_IMPORT_PATH})+# The OpenAI-compatible gateway Codex wrapper and container agent wrappers are+# allowed on Harbor-native backends (including GKE).+_NATIVE_BACKEND_AGENT_IMPORT_PATHS = frozenset(+ {+ _GATEWAY_CODEX_IMPORT_PATH,+ *CONTAINER_AGENT_IMPORT_PATHS.values(),+ }+)_HARBOR_RUN_OUTPUT_MAX_BYTES = MAX_COMMAND_OUTPUT_BYTES_HARBOR_RUN_DIAGNOSTIC_TAIL_CHARS = 16 * 1024_HARBOR_RUN_OUTPUT_READ_BYTES = 64 * 1024@@ -796,75 +823,37 @@ def _reserve_run_dir(results_root: Path, timestamp: str) -> Path:_TRUSTED_NETWORK_HOST_ENV_VARS = frozenset({- "COMSPEC",- "HOME",- "LANG",- "LC_ALL",- "LC_CTYPE",- "PATH",- "PATHEXT",- "SKILLEVALUATOR_ALLOWED_MCP_HOSTS",- "SKILLEVALUATOR_ALLOWED_MCP_SECRETS",- "SYSTEMROOT",- "TEMP",- "TMP",- "TMPDIR",- "WINDIR",- "XDG_RUNTIME_DIR",+ "ALL_PROXY",+ "CURL_CA_BUNDLE",+ "HTTPS_PROXY",+ "HTTP_PROXY",+ "NO_PROXY",+ "REQUESTS_CA_BUNDLE",+ "SSL_CERT_DIR",+ "SSL_CERT_FILE",+ "all_proxy",+ "http_proxy",+ "https_proxy",+ "no_proxy",})-_HARBOR_ENV_MODE_VARS = {- "docker": frozenset(- {- "DOCKER_API_VERSION",- "DOCKER_CERT_PATH",- "DOCKER_CONFIG",- "DOCKER_CONTEXT",- "DOCKER_HOST",- "DOCKER_TLS_VERIFY",- }- ),- "daytona": frozenset(- {- "DAYTONA_API_KEY",- "DAYTONA_API_URL",- "DAYTONA_JWT_TOKEN",- "DAYTONA_ORGANIZATION_ID",- "DAYTONA_TARGET",- }- ),- "e2b": frozenset({"E2B_API_KEY"}),- "modal": frozenset({"MODAL_ENVIRONMENT", "MODAL_TOKEN_ID", "MODAL_TOKEN_SECRET"}),- "runloop": frozenset({"RUNLOOP_API_KEY"}),- "langsmith": frozenset(- {- "LANGCHAIN_API_KEY",- "LANGSMITH_API_KEY",- "LANGSMITH_ENDPOINT",- "LANGSMITH_PROFILE",- "LANGSMITH_SANDBOX_API_URL",- }- ),- "gke": frozenset(- {- "CLOUDSDK_CONFIG",- "GCP_PROJECT",- "GOOGLE_APPLICATION_CREDENTIALS",- "GOOGLE_CLOUD_PROJECT",- "KUBECONFIG",- "SKILLEVALUATOR_GKE_ALLOW_WORKLOAD_IDENTITY",- "SKILLEVALUATOR_GKE_AUTOPILOT",- "SKILLEVALUATOR_GKE_METADATA_PROBE_IMAGE",- }- ),- "novita": frozenset({"NOVITA_API_KEY", "NOVITA_API_URL", "NOVITA_BASE_URL", "NOVITA_DOMAIN"}),- "islo": frozenset({"ISLO_API_KEY", "ISLO_API_URL", "ISLO_COMPUTE_URL"}),- "tensorlake": frozenset({"TENSORLAKE_API_KEY"}),- "cwsandbox": frozenset({"CWSANDBOX_API_KEY"}),- "wandb": frozenset({"WANDB_API_KEY", "WANDB_BASE_URL"}),- "use-computer": frozenset(- {"USE_COMPUTER_API_KEY", "USE_COMPUTER_HOST", "USE_COMPUTER_SNAPSHOT", "USE_COMPUTER_VERSION"}- ),+_HARBOR_BASE_ENV_VARS = _TRUSTED_NETWORK_HOST_ENV_VARS | {+ "COMSPEC",+ "HOME",+ "LANG",+ "LC_ALL",+ "LC_CTYPE",+ "PATH",+ "PATHEXT",+ "SKILLEVALUATOR_ALLOWED_MCP_HOSTS",+ "SKILLEVALUATOR_ALLOWED_MCP_SECRETS",+ "SYSTEMROOT",+ "TEMP",+ "TMP",+ "TMPDIR",+ "USERPROFILE",+ "WINDIR",+ "XDG_RUNTIME_DIR",}_AWS_HOST_ENV_VARS = frozenset({@@ -940,8 +929,32 @@ _VERTEX_HOST_ENV_VARS = frozenset(*_ADC_HOST_ENV_VARS,})-_RUNTIME_ENV_HOST_CONTROL_NAMES = (- frozenset(+_EC2_HOST_ENV_VARS = _AWS_HOST_ENV_VARS | {"AWS_ENDPOINT_URL_EC2", "SSH_AUTH_SOCK"}+_DOCKER_HOST_ENV_VARS = frozenset(+ {+ "COMPOSE_ANSI",+ "COMPOSE_HTTP_TIMEOUT",+ "COMPOSE_IGNORE_ORPHANS",+ "COMPOSE_PARALLEL_LIMIT",+ "COMPOSE_PROGRESS",+ "COMPOSE_STATUS_STDOUT",+ "DOCKER_API_VERSION",+ "DOCKER_AUTH_CONFIG",+ "DOCKER_CERT_PATH",+ "DOCKER_CONFIG",+ "DOCKER_CONTEXT",+ "DOCKER_CUSTOM_HEADERS",+ "DOCKER_DEFAULT_PLATFORM",+ "DOCKER_HOST",+ "DOCKER_TLS",+ "DOCKER_TLS_VERIFY",+ "SSH_AUTH_SOCK",+ }+)+_HARBOR_ENV_MODE_VARS = {+ "docker": _DOCKER_HOST_ENV_VARS,+ "kata": _DOCKER_HOST_ENV_VARS,+ "daytona": frozenset({"DAYTONA_API_KEY","DAYTONA_API_URL",@@ -951,27 +964,160 @@ _RUNTIME_ENV_HOST_CONTROL_NAMES = ("DAYTONA_SERVER_URL","DAYTONA_TARGET",}- )+ ),+ "e2b": frozenset({"E2B_API_KEY", "E2B_API_URL", "E2B_DOMAIN", "E2B_SANDBOX_URL"}),+ "modal": frozenset(+ {+ "MODAL_CONFIG_PATH",+ "MODAL_ENVIRONMENT",+ "MODAL_OVERRIDE_HEADERS",+ "MODAL_PROFILE",+ "MODAL_SERVER_URL",+ "MODAL_TOKEN_ID",+ "MODAL_TOKEN_SECRET",+ }+ ),+ "runloop": frozenset({"RUNLOOP_API_KEY", "RUNLOOP_BASE_URL", "RUNLOOP_CUSTOM_HEADERS"}),+ "langsmith": frozenset(+ {+ "LANGCHAIN_API_KEY",+ "LANGCHAIN_ENDPOINT",+ "LANGSMITH_API_KEY",+ "LANGSMITH_CONFIG_FILE",+ "LANGSMITH_ENDPOINT",+ "LANGSMITH_PROFILE",+ "LANGSMITH_SANDBOX_API_URL",+ "LANGSMITH_WORKSPACE_ID",+ }+ ),+ "ec2": _EC2_HOST_ENV_VARS,+ "gke": frozenset(+ {+ "CLOUDSDK_ACTIVE_CONFIG_NAME",+ "CLOUDSDK_AUTH_ACCESS_TOKEN",+ "CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE",+ "CLOUDSDK_AUTH_IMPERSONATE_SERVICE_ACCOUNT",+ "CLOUDSDK_CONFIG",+ "CLOUDSDK_CORE_ACCOUNT",+ "CLOUDSDK_CORE_PROJECT",+ "GCP_PROJECT",+ "GOOGLE_APPLICATION_CREDENTIALS",+ "GOOGLE_CLOUD_PROJECT",+ "GOOGLE_CLOUD_QUOTA_PROJECT",+ "KUBECONFIG",+ "SKILLEVALUATOR_GKE_ALLOW_WORKLOAD_IDENTITY",+ "SKILLEVALUATOR_GKE_AUTOPILOT",+ "SKILLEVALUATOR_GKE_METADATA_PROBE_IMAGE",+ }+ ),+ "ack": _DOCKER_HOST_ENV_VARS+ | {+ "KUBECONFIG",+ "KUBERNETES_SERVICE_HOST",+ "KUBERNETES_SERVICE_PORT",+ },+ "openshift": frozenset({"KUBECONFIG"}),+ "novita": frozenset(+ {+ "NOVITA_ACCESS_TOKEN",+ "NOVITA_API_KEY",+ "NOVITA_API_URL",+ "NOVITA_BASE_URL",+ "NOVITA_DOMAIN",+ "NOVITA_SANDBOX_URL",+ }+ ),+ "apple-container": frozenset(),+ "singularity": frozenset(+ {+ "APPTAINER_AUTHFILE",+ "APPTAINER_CONFIGDIR",+ "APPTAINER_DOCKER_PASSWORD",+ "APPTAINER_DOCKER_USERNAME",+ "SINGULARITY_AUTHFILE",+ "SINGULARITY_CONFIGDIR",+ "SINGULARITY_DOCKER_PASSWORD",+ "SINGULARITY_DOCKER_USERNAME",+ }+ ),+ "islo": frozenset({"ISLO_API_KEY", "ISLO_API_URL", "ISLO_COMPUTE_URL"}),+ "tensorlake": frozenset(+ {+ "TENSORLAKE_API_KEY",+ "TENSORLAKE_API_URL",+ "TENSORLAKE_ORGANIZATION_ID",+ "TENSORLAKE_PAT",+ "TENSORLAKE_PROJECT_ID",+ "TENSORLAKE_SANDBOX_PROXY_URL",+ }+ ),+ "cwsandbox": frozenset({"CWSANDBOX_API_KEY", "CWSANDBOX_BASE_URL"}),+ "wandb": frozenset({"NETRC", "WANDB_API_KEY", "WANDB_BASE_URL", "WANDB_ENTITY", "WANDB_PROJECT"}),+ "use-computer": frozenset(+ {"USE_COMPUTER_API_KEY", "USE_COMPUTER_HOST", "USE_COMPUTER_SNAPSHOT", "USE_COMPUTER_VERSION"}+ ),+ "blaxel": frozenset(+ {+ "BL_API_KEY",+ "BL_API_VERSION",+ "BL_CLIENT_CREDENTIALS",+ "BL_ENV",+ "BL_REGION",+ "BL_WORKSPACE",+ }+ ),+ "beam": frozenset(+ {+ "API_HOST",+ "API_PORT",+ "BEAM_TOKEN",+ "GATEWAY_HOST",+ "GATEWAY_PORT",+ "INTERNAL_API_HOST",+ "INTERNAL_API_PORT",+ "REALTIME_HOST",+ }+ ),+ "skypilot": _DOCKER_HOST_ENV_VARS+ | frozenset(+ {+ "HARBOR_SKYPILOT_REGISTRY",+ "SKYPILOT_API_SERVER_ENDPOINT",+ "SKYPILOT_GLOBAL_CONFIG",+ "SKYPILOT_PROJECT_CONFIG",+ "SKYPILOT_SERVICE_ACCOUNT_TOKEN",+ }+ ),+ "hyperbrowser": _DOCKER_HOST_ENV_VARS | frozenset({"HYPERBROWSER_API_KEY", "HYPERBROWSER_BASE_URL"}),+ "vercel": frozenset({"VERCEL_OIDC_TOKEN", "VERCEL_PROJECT_ID", "VERCEL_TEAM_ID", "VERCEL_TOKEN"}),+ "runta": frozenset({"RUNTA_CONFIG", "RUNTA_ENDPOINT", "RUNTA_TOKEN"}),+ # The Mosaic SDK still reads the MAR_* names its MOSAIC_* names replaced.+ "mosaic": frozenset(+ {+ "MAR_API_TOKEN",+ "MAR_CONFIG",+ "MAR_ENDPOINT",+ "MOSAIC_API_TOKEN",+ "MOSAIC_API_URL",+ "MOSAIC_CONFIG",+ "MOSAIC_REGISTRY_PASSWORD",+ "MOSAIC_REGISTRY_USERNAME",+ "MOSAIC_RETRIES",+ }+ ),+}+_BEDROCK_HOST_ENV_VARS = _AWS_HOST_ENV_VARS | {+ "AWS_BEARER_TOKEN_BEDROCK",+ "AWS_CSM_ENABLED",+ "AWS_CSM_PORT",+ "AWS_ENDPOINT_URL_BEDROCK",+ "AWS_ENDPOINT_URL_BEDROCK_RUNTIME",+}+_RUNTIME_ENV_HOST_CONTROL_NAMES = (+ _RUNTIME_PROCESS_CONTROL_ENV_NAMES| _BEDROCK_HOST_ENV_VARS| _VERTEX_HOST_ENV_VARS- | _VERIFIER_JUDGE_MODEL_ENV_VARS- | frozenset().union(*_HARBOR_ENV_MODE_VARS.values())-)-_RUNTIME_ENV_HOST_CONTROL_PREFIXES = (- "BASH_FUNC_",- "COMPOSE_",- "DOCKER_",- "DYLD_",- "GIT_",- "HARBOR_",- "LD_",- "NODE_",- "OTEL_",- "PIP_",- "PYTHON",- "SKILL_EVAL_",- "SKILLEVALUATOR_",- "UV_",+ | _VERIFIER_JUDGE_CONTROL_ENV_VARS)_RUNTIME_ENV_HOST_CONTROL_PREFIXES = _RUNTIME_PROCESS_CONTROL_ENV_PREFIXES_OPERATOR_OWNED_AGENT_ENV = frozenset(@@ -1049,82 +1195,6 @@ def is_gke_vertex_workload_identity_active()-_SENSITIVE_EK_KEY_RE = re.compile(- r"(?i)(?:^|.*[_-])(?:api[_-]?key|secret|password|credential|credentials|authorization|bearer|"- r"private[_-]?key|service[_-]?account[_-]?key|access[_-]?key|session[_-]?token|auth[_-]?token|"- r"cookie|session[_-]?cookie|client[_-]?certificate|client[_-]?cert|certificate|cert|passphrase|oauth|"- r"(?:access|refresh|id|auth|user|client|secret)[_-]?tokens?)(?:$|[_-].*)"-)-_SAFE_BACKEND_CONSTRUCTOR_KWARGS: dict[str, frozenset[str]] = {- "gke": frozenset(- {- "allow_workload_identity",- "autopilot",- "cluster_name",- "region",- "namespace",- "registry_location",- "registry_name",- "project_id",- "cloud_build_machine_type",- "cloud_build_disk_size_gb",- "memory_limit_multiplier",- "cpu_request",- "cpu_limit",- "memory_request",- "memory_limit",- "ephemeral_storage_request",- *_SKILL_SAFE_ENVIRONMENT_KWARGS,- }- ),- "docker": frozenset(- {- "image",- "container_name",- "network_name",- "pull_policy",- *_SKILL_SAFE_ENVIRONMENT_KWARGS,- }- ),- "daytona": frozenset(- {- "target",- "snapshot",- "workspace_id",- *_SKILL_SAFE_ENVIRONMENT_KWARGS,- }- ),-}-_NON_CREDENTIAL_TOKEN_KEY_RE = re.compile(- r"^(?:max|min|num|total|count|prompt|completion|input|output|request)?[_-]?tokens$|"- r"^(?:max|min|num|total|count|prompt|completion|input|output)[_-]token$|"- r"(?:^|.*[_-])tokens?[_-](?:bucket|rate|count|limit|budget|window|usage|size|per[_-]\w+)$|"- r"(?:^|.*[_-])(?:tokenizer|tokenizers|detokenize|detokenizer|tokenization|detokenization)(?:[_-].*)?$",- re.IGNORECASE,-)---def _is_sensitive_ek_key(key: str) -> bool:- """Determine whether an environment kwarg key name represents a sensitive credential."""- normalized_key = re.sub(r"([a-z0-9])([A-Z])", r"\1_\2", key).lower()- if _NON_CREDENTIAL_TOKEN_KEY_RE.search(normalized_key):- return False- return is_sensitive_key(key) or bool(_SENSITIVE_EK_KEY_RE.search(normalized_key))---_SENSITIVE_EK_VALUE_PATTERNS = (- LOG_SK_RE,- LOG_NVAPI_RE,- LOG_YA29_RE,- LOG_JWT_RE,- LOG_CRSR_RE,- OPENSHIFT_TOKEN_RE,- re.compile(r"(?<![A-Za-z0-9_-])(?:AKIA|ASIA)[0-9A-Z]{12,}"),- re.compile(r"-----BEGIN (?:[A-Z0-9_-]+ )?PRIVATE KEY-----"),- re.compile(r"(?i)\bBearer\s+[A-Za-z0-9._~+/-]{15,}"),-)--@dataclass(frozen=True)class AgentRuntimePlan:"""One agent's immutable model, credential, and Harbor environment plan."""@@ -1302,26 +1372,26 @@ _GKE_ENV_VAR_MAPPINGS: Mapping[str, str] = MappingProxyType(_GKE_REQUIRED_KWARGS = tuple(_GKE_ENV_VAR_MAPPINGS)-def _missing_gke_kwargs(kwargs: Mapping[str, str]) -> list[str]:+def _missing_gke_kwargs(kwargs: Mapping[str, Any]) -> list[str]:"""Return required GKE kwargs that are missing or empty."""- return [k for k in _GKE_REQUIRED_KWARGS if not str(kwargs.get(k, "")).strip()]+ return [k for k in _GKE_REQUIRED_KWARGS if not isinstance(kwargs.get(k), str) or not str(kwargs.get(k, "")).strip()]def _resolve_environment_kwargs(env_mode: str,*,- config_kwargs: Mapping[str, str] | None = None,- cli_kwargs: Mapping[str, str] | None = None,+ config_kwargs: Mapping[str, Any] | None = None,+ cli_kwargs: Mapping[str, Any] | None = None,environ: Mapping[str, str] | None = None,-) -> dict[str, str]:+) -> dict[str, Any]:"""Resolve environment kwargs with CLI > environment variable > config precedence."""- resolved: dict[str, str] = {}+ resolved: dict[str, Any] = {}env = environ if environ is not None else os.environif config_kwargs:for k, v in config_kwargs.items():if v is not None and k not in _GKE_INFRASTRUCTURE_KWARGS:- resolved[k] = str(v).strip()+ resolved[k] = v.strip() if isinstance(v, str) else vif env_mode == "gke":for kwarg, env_var in _GKE_ENV_VAR_MAPPINGS.items():@@ -1331,11 +1401,217 @@ def _resolve_environment_kwargs(if cli_kwargs:for k, v in cli_kwargs.items():if v is not None:- resolved[k] = str(v).strip()+ resolved[k] = v.strip() if isinstance(v, str) else vreturn resolved+_HARBOR_RUNTIME_POLICY_KWARGS = frozenset(+ {+ "context_id",+ "cpu_enforcement_policy",+ "delete",+ "enable_environment_dir_upload",+ "environment_dir",+ "environment_name",+ "extra_allowed_hosts",+ "extra_docker_compose",+ "force_build",+ "keep_containers",+ "logger",+ "memory_enforcement_policy",+ "mounts",+ "mounts_json",+ "network_policy",+ "override_cpus",+ "override_gpus",+ "override_memory_mb",+ "override_storage_mb",+ "override_tpu",+ "persistent_env",+ "pod_capabilities_add",+ "pod_capabilities_drop",+ "pod_overrides",+ "pod_privileged",+ "pod_run_as_group",+ "pod_run_as_user",+ "phase_network_policies",+ "session_id",+ "stream",+ "suppress_override_warnings",+ "extra_env",+ "extra_volume_mounts",+ "extra_volumes",+ "init_containers",+ "task_env_config",+ "trial_paths",+ }+)++_HARBOR_ENVIRONMENT_RUNTIME_POLICY_KWARGS: dict[str, frozenset[str]] = {+ "ack": frozenset(+ {+ "build_job_namespace",+ "buildkit_address",+ "dind_image",+ "memory_limit_multiplier",+ "pod_annotations",+ "pod_labels",+ "sandbox_env_vars",+ "service_account",+ "use_buildkit",+ }+ ),+ "blaxel": frozenset({"dind_extra_args"}),+ # Credential strategy selection; SkillEvaluator sets it only for the wandb alias.+ "cwsandbox": frozenset({"auth"}),+ "daytona": frozenset({"network_block_all"}),+ "ec2": frozenset({"iam_instance_profile", "strict_host_key_checking"}),+ "gke": frozenset({"memory_limit_multiplier"}),+ "modal": frozenset({"volumes"}),+ # Persistent volumes, persisted sandboxes, SSH access, and alternate build+ # inputs would let state or access outlive one isolated trial or skip+ # evaluator-staged Dockerfile layers.+ "mosaic": frozenset({"build_args", "build_target", "enable_ssh", "persist", "volume"}),+ "openshift": frozenset({"service_account_name"}),+ # Direct mode runs commands in Runta's base runtime and ignores the task+ # image; Harbor's automatic selection honors the task's Dockerfile or Compose.+ "runta": frozenset({"mode"}),+ "singularity": frozenset({"singularity_no_mount"}),+ "use-computer": frozenset({"resources"}),+ "vercel": frozenset({"ports"}),+ "wandb": frozenset({"auth"}),+}+++_KATA_RUNTIME_NAME_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}")+_KATA_DNS_SERVER_RE = re.compile(r"[0-9A-Fa-f:.]{2,45}")+++def _environment_kwarg_policy_error(env_mode: str, environment_kwargs: Mapping[str, Any]) -> str | None:+ if not environment_kwargs:+ return None+ if env_mode == "docker":+ return "Environment kwargs are not supported for SkillEvaluator Docker mode"+ if env_mode == ENV_MODE_LOCAL:+ return "Environment kwargs are not supported for SkillEvaluator local mode"+ reserved = _HARBOR_RUNTIME_POLICY_KWARGS | _HARBOR_ENVIRONMENT_RUNTIME_POLICY_KWARGS.get(env_mode, frozenset())+ if collisions := sorted(reserved & environment_kwargs.keys()):+ return "Environment kwarg(s) reserved for Harbor runtime policy: " + ", ".join(collisions)+ allowed_kwargs = HARBOR_ENVIRONMENT_KWARGS[env_mode]+ if env_mode == "gke":+ allowed_kwargs = allowed_kwargs | SKILLEVALUATOR_GKE_EXTRA_ENVIRONMENT_KWARGS+ if unknown := sorted(environment_kwargs.keys() - allowed_kwargs):+ return f"Harbor {HARBOR_VERSION} environment '{env_mode}' does not accept environment kwarg(s): " + ", ".join(+ unknown+ )+ if env_mode == "kata":+ return _kata_environment_kwarg_error(environment_kwargs)+ return None+++def _kata_environment_kwarg_error(environment_kwargs: Mapping[str, Any]) -> str | None:+ """Keep Kata's runtime overlay pointed at a Kata handler and literal DNS servers."""+ runtime = environment_kwargs.get("kata_runtime", "kata")+ # Harbor treats Docker runtimes whose names contain "kata" as Kata handlers;+ # any other handler, such as runc, would silently drop the microVM boundary.+ if not isinstance(runtime, str) or "kata" not in runtime or not _KATA_RUNTIME_NAME_RE.fullmatch(runtime):+ return "Harbor environment 'kata' requires kata_runtime to name a registered Kata runtime handler"+ # Harbor writes each server into the microVM's resolv.conf. Parse them the+ # way Harbor does (comma strings are split and trimmed) and allow only IP+ # literals: an IPv6 zone ID (%...) could otherwise carry resolver directives.+ if "kata_dns" not in environment_kwargs:+ return None+ dns = environment_kwargs["kata_dns"]+ if isinstance(dns, str):+ servers = [server.strip() for server in dns.split(",") if server.strip()]+ elif isinstance(dns, list) and all(isinstance(server, str) for server in dns):+ servers = dns+ else:+ return "Harbor environment 'kata' requires kata_dns to be a list of IP addresses"+ # Without a server Harbor keeps Docker's embedded resolver, which a Kata VM cannot reach.+ if not servers:+ return "Harbor environment 'kata' requires kata_dns to list at least one IP address"+ for server in servers:+ if not _KATA_DNS_SERVER_RE.fullmatch(server):+ return "Harbor environment 'kata' requires kata_dns to be a list of IP addresses"+ try:+ ipaddress.ip_address(server)+ except ValueError:+ return "Harbor environment 'kata' requires kata_dns to be a list of IP addresses"+ return None+++def _staged_task_environment_error(env_mode: str, task_dirs: Iterable[Path]) -> str | None:+ """Reject staged task environments the selected backend cannot run."""+ if env_mode == "mosaic":+ for task_dir in task_dirs:+ if (task_dir / "environment" / "docker-compose.yaml").exists():+ return (+ f"Harbor environment 'mosaic' runs one VM per trial and cannot run Docker Compose task "+ f"'{task_dir.name}'; use a Dockerfile-only environment or another backend."+ )+ return None+ if env_mode != "kata":+ return None+ import yaml++ for task_dir in task_dirs:+ compose_path = task_dir / "environment" / "docker-compose.yaml"+ if not compose_path.is_file():+ continue+ try:+ compose = yaml.safe_load(compose_path.read_text(encoding="utf-8"))+ except (OSError, UnicodeDecodeError, yaml.YAMLError):+ return f"Could not read the Docker Compose model of task '{task_dir.name}'"+ services = compose.get("services") if isinstance(compose, dict) else None+ if sidecars := sorted(str(name) for name in services or {} if name != "main"):+ # Kata replaces each service's resolv.conf, so Compose service-name DNS stops working.+ return (+ f"Harbor environment 'kata' cannot run task '{task_dir.name}' with Compose sidecar services "+ f"({', '.join(sidecars)}): services in Kata microVMs cannot resolve each other by name"+ )+ return None+++def _validated_timeout_multiplier(value: object) -> float:+ """Return one finite positive timeout scale accepted by every entry point."""+ if isinstance(value, bool) or not isinstance(value, int | float):+ raise ValueError("timeout_multiplier must be a finite number greater than 0")+ try:+ normalized = float(value)+ except OverflowError:+ raise ValueError("timeout_multiplier must be a finite number greater than 0") from None+ if not math.isfinite(normalized) or normalized <= 0:+ raise ValueError("timeout_multiplier must be a finite number greater than 0")+ if normalized > MAX_HARBOR_TIMEOUT_MULTIPLIER:+ raise ValueError("timeout_multiplier must yield finite Harbor timeouts")+ return normalized+++def _validated_pass_threshold(value: object) -> float:+ """Return one finite unit-interval attempt threshold."""+ if isinstance(value, bool) or not isinstance(value, int | float):+ raise ValueError("pass_threshold must be a finite number between 0.0 and 1.0")+ try:+ normalized = float(value)+ except OverflowError:+ raise ValueError("pass_threshold must be a finite number between 0.0 and 1.0") from None+ if not math.isfinite(normalized) or not 0.0 <= normalized <= 1.0:+ raise ValueError("pass_threshold must be a finite number between 0.0 and 1.0")+ return normalized+++def _harbor_agent_kwargs(agent: str, model: str | None, env_mode: str) -> dict[str, str]:+ """Return the Harbor agent kwargs SkillEvaluator pins for ``agent`` running ``model``."""+ kwargs = dict(_HARBOR_AGENT_KWARGS.get(agent, {}))+ if env_mode == ENV_MODE_LOCAL:+ kwargs.pop("version", None)+ if agent == "claude-code" and model and is_claude_5_5_or_later(model):+ kwargs["reasoning_effort"] = _CLAUDE_CODE_REASONING_EFFORT+ return kwargs++def build_harbor_run_command(*,dataset_path: str | Path,@@ -1354,13 +1630,25 @@ def build_harbor_run_command(override_storage_mb: int | None = None,agent_import_path: str | None = None,verifier_env: Mapping[str, str] | None = None,- environment_kwargs: Mapping[str, str] | None = None,+ environment_kwargs: Mapping[str, Any] | None = None,) -> list[str]:"""Build a Harbor invocation for a built-in environment type or local mode."""if env_mode not in HARBOR_ENV_MODES:raise ValueError(f"env_mode must be one of: {', '.join(sorted(HARBOR_ENV_MODES))}")- if agent_import_path and env_mode not in {"docker", ENV_MODE_LOCAL, "gke"}:- raise ValueError("agent_import_path is supported only with --env docker, local, or gke")+ timeout_multiplier = _validated_timeout_multiplier(timeout_multiplier)+ if (+ agent_import_path+ and env_mode not in HARBOR_DOCKER_ENGINE_ENV_MODES | {ENV_MODE_LOCAL}+ and agent_import_path not in _NATIVE_BACKEND_AGENT_IMPORT_PATHS+ ):+ raise ValueError("agent_import_path is supported only with --env docker, kata, or local")+ validated_environment_kwargs = validate_environment_kwargs(+ dict(environment_kwargs or {}),+ env_mode=env_mode,+ )+ if policy_error := _environment_kwarg_policy_error(env_mode, validated_environment_kwargs):+ raise ValueError(policy_error)+ validated_environment_kwargs = _absolute_host_path_kwargs(env_mode, validated_environment_kwargs)command = [_harbor_bin(),@@ -1409,40 +1697,29 @@ def build_harbor_run_command(f"inherit_agent_keys={str(local_sandbox.coerce_flag(None, env_var=local_sandbox.INHERIT_AGENT_KEYS_ENV)).lower()}",])- elif env_mode in {"docker", "gke"}:- from skillevaluator.tier3.harbor import CONTAINER_AGENT_IMPORT_PATHS-- agent_import_path = agent_import_path or CONTAINER_AGENT_IMPORT_PATHS.get(agent)- if agent_import_path:- command.extend(["--agent-import-path", agent_import_path])- else:- command.extend(["-a", agent])- if env_mode == "docker":- command.extend(["--environment-import-path", SECURE_DOCKER_ENV_IMPORT_PATH])+ elif env_mode in HARBOR_DOCKER_ENGINE_ENV_MODES or env_mode == "gke":+ default_import_paths = (+ DOCKER_AGENT_IMPORT_PATHS if env_mode in HARBOR_DOCKER_ENGINE_ENV_MODES else CONTAINER_AGENT_IMPORT_PATHS+ )+ command.extend(["--agent", agent_import_path or default_import_paths.get(agent, agent)])+ if env_mode == "gke":+ env_import_path = SECURE_GKE_ENV_IMPORT_PATH+ elif env_mode == "kata":+ env_import_path = SECURE_KATA_ENV_IMPORT_PATHelse:- command.extend(["--env", env_mode, "--environment-import-path", SECURE_GKE_ENV_IMPORT_PATH])+ env_import_path = SECURE_DOCKER_ENV_IMPORT_PATH+ command.extend(["--env", env_import_path])else:- command.extend(["-a", agent, "--env", env_mode])-- if env_mode != ENV_MODE_LOCAL:- safe_keys = _SAFE_BACKEND_CONSTRUCTOR_KWARGS.get(env_mode, _SKILL_SAFE_ENVIRONMENT_KWARGS)- for key, value in sorted((environment_kwargs or {}).items()):- str_val = str(value)- normalized_key = re.sub(r"([a-z0-9])([A-Z])", r"\1_\2", key).lower()- is_benign_token = bool(_NON_CREDENTIAL_TOKEN_KEY_RE.search(normalized_key))- if (- (key not in safe_keys and not is_benign_token)- or _is_sensitive_ek_key(key)- or any(pattern.search(str_val) for pattern in _SENSITIVE_EK_VALUE_PATTERNS)- ):- raise ValueError(- f"Sensitive key or value detected in environment_kwargs: {key}. "- "Credentials and disallowed settings must not be passed via CLI flags or process arguments."- )- if key == "allow_workload_identity":- continue- command.extend(["--ek", f"{key}={value}"])-+ command.extend(["--agent", agent_import_path or agent, "--env", harbor_environment_type(env_mode)])+ for name, value in sorted(_harbor_agent_kwargs(agent, model, env_mode).items()):+ command.extend(["--ak", encode_environment_kwarg(name, value)])+ for name, value in sorted(validated_environment_kwargs.items()):+ if name == "allow_workload_identity":+ continue+ command.extend(["--ek", encode_environment_kwarg(name, value)])+ # Alias kwargs come after operator kwargs, which may never set them.+ for name, value in sorted(HARBOR_ENVIRONMENT_ALIAS_KWARGS.get(env_mode, {}).items()):+ command.extend(["--ek", encode_environment_kwarg(name, value)])if jobs_dir is not None:command.extend(["--jobs-dir", str(Path(jobs_dir).absolute())])if disable_verification:@@ -1461,7 +1738,7 @@ def build_harbor_run_command(command.extend(["--override-storage-mb", str(override_storage_mb)])for name, value in sorted((verifier_env or {}).items()):str_val = str(value)- if (is_operator_owned_or_provider_secret(name) or _is_sensitive_ek_key(name)) and not re.fullmatch(+ if (is_operator_owned_or_provider_secret(name) or _is_sensitive_environment_kwarg_name(name)) and not re.fullmatch(r"\$\{[A-Za-z_][A-Za-z0-9_]*\}", str_val):raise ValueError(@@ -1612,7 +1889,7 @@ def _validate_agent_provider_credentials("AWS_BEARER_TOKEN_BEDROCK for the agent environment."]- if provider.provider in {"openai", "openai-compatible"} and "claude-code" in agents:+ if provider.provider == "openai" and "claude-code" in agents:has_anthropic_key = bool(agent_runtime_env.get("ANTHROPIC_API_KEY", "").strip())has_vertex = env_mode == "gke" and agent_runtime_env.get("CLAUDE_CODE_USE_VERTEX", "").strip() == "1"if not has_anthropic_key and not has_vertex:@@ -1722,160 +1999,293 @@ def _validate_agent_provider_credentials(return []-_KUBECONFIG_FILE_PATH_KEYS = frozenset(- {- "certificate-authority",- "client-certificate",- "client-key",- "tokenFile",- "idp-certificate-authority",- }+def _environment_kwarg_prerequisite_errors(+ env_mode: str,+ environment_kwargs: Mapping[str, Any] | None,+) -> list[str]:+ """Validate constructor requirements Harbor cannot check in ``preflight``."""+ try:+ kwargs = validate_environment_kwargs(+ dict(environment_kwargs or {}),+ env_mode=env_mode,+ )+ except (RecursionError, TypeError, ValueError) as exc:+ return [f"Invalid --environment-kwarg: {exc}"]+ if policy_error := _environment_kwarg_policy_error(env_mode, kwargs):+ return [policy_error]++ def invalid_strings(*names: str) -> list[str]:+ return [name for name in names if not isinstance(kwargs.get(name), str) or not kwargs[name].strip()]++ required: tuple[str, ...] = ()+ if env_mode == "gke":+ required = ("cluster_name", "region", "namespace", "registry_location", "registry_name")+ elif env_mode == "ack":+ required = ("namespace",)+ elif env_mode == "ec2":+ required = ("region",)+ if invalid := invalid_strings(*required):+ return [+ f"Harbor environment '{env_mode}' requires non-empty string --environment-kwarg for: " + ", ".join(invalid)+ ]+ if env_mode == "ack" and (+ invalid := invalid_strings(*(name for name in ("context", "kubeconfig") if name in kwargs))+ ):+ return ["Harbor environment 'ack' requires non-empty string --environment-kwarg for: " + ", ".join(invalid)]+ if env_mode == "ec2":+ launch_mode_value = kwargs.get("launch_mode", "ephemeral")+ if not isinstance(launch_mode_value, str):+ return ["Harbor environment 'ec2' requires launch_mode to be 'ephemeral' or 'attach'"]+ launch_mode = launch_mode_value+ if launch_mode not in {"ephemeral", "attach"}:+ return ["Harbor environment 'ec2' requires launch_mode to be 'ephemeral' or 'attach'"]+ conditional = "ami_id" if launch_mode == "ephemeral" else "instance_id"+ if invalid_strings(conditional):+ return [+ f"Harbor environment 'ec2' launch_mode={launch_mode!r} requires --environment-kwarg {conditional}=VALUE"+ ]+ if (ssh_key_path := kwargs.get("ssh_key_path")) is not None:+ try:+ ssh_key_exists = isinstance(ssh_key_path, str) and Path(ssh_key_path).expanduser().is_file()+ except (OSError, RuntimeError):+ ssh_key_exists = False+ if not ssh_key_exists:+ return ["Harbor environment 'ec2' requires ssh_key_path to name an existing regular file"]+ if launch_mode == "ephemeral" and kwargs.get("use_public_ip") is False and invalid_strings("subnet_id"):+ return ["Harbor environment 'ec2' use_public_ip=False requires a non-empty subnet_id"]+ return []+++# Backend SDK configuration files that operators may name in the host environment.+_BACKEND_CONFIG_FILE_ENV_VARS: Mapping[str, tuple[str, ...]] = MappingProxyType(+ {"mosaic": ("MOSAIC_CONFIG", "MAR_CONFIG"), "runta": ("RUNTA_CONFIG",)})-def _resolve_relative_kubeconfig_path(- raw_value: str,- base_dir: Path | None,- fallback_dirs: Sequence[Path] = (),-) -> str:- """Resolve a relative kubeconfig path against its source file directory."""- expanded = Path(raw_value.strip()).expanduser()- if expanded.is_absolute():- return str(expanded)- if base_dir is not None:- return str((base_dir / expanded).resolve())- for directory in fallback_dirs:- candidate = (directory / expanded).resolve()- if candidate.exists():- return str(candidate)- if fallback_dirs:- return str((fallback_dirs[0] / expanded).resolve())- return raw_value---def _unwrap_kubeconfig_node(- obj: Any,- source_path: str | None = None,- fallback_dirs: Sequence[Path] = (),-) -> Any:- """Recursively unwrap Kubernetes ConfigNode wrappers while absolutizing relative file paths."""- node_path = getattr(obj, "path", None) or source_path- if hasattr(obj, "value"):- return _unwrap_kubeconfig_node(obj.value, source_path=node_path, fallback_dirs=fallback_dirs)- if isinstance(obj, dict):- out: dict[str, Any] = {}- base_dir = Path(node_path).expanduser().resolve().parent if node_path else None- for k, v in obj.items():- unwrapped = _unwrap_kubeconfig_node(v, source_path=node_path, fallback_dirs=fallback_dirs)- if (- isinstance(unwrapped, str)- and unwrapped.strip()- and (k in _KUBECONFIG_FILE_PATH_KEYS or (k == "command" and ("/" in unwrapped or os.sep in unwrapped)))- ):- unwrapped = _resolve_relative_kubeconfig_path(unwrapped, base_dir, fallback_dirs)- out[k] = unwrapped- return out- if isinstance(obj, list):- return [_unwrap_kubeconfig_node(item, source_path=node_path, fallback_dirs=fallback_dirs) for item in obj]- return obj+def _backend_config_prerequisite_errors(env_mode: str) -> list[str]:+ """Check Runta and Mosaic credential inputs against what Harbor's child will see."""+ errors: list[str] = []+ for name in _BACKEND_CONFIG_FILE_ENV_VARS.get(env_mode, ()):+ raw = os.environ.get(name)+ if raw is None:+ continue+ try:+ is_file = bool(raw.strip()) and Path(raw).expanduser().is_file()+ except (OSError, RuntimeError):+ is_file = False+ if not is_file:+ errors.append(f"Harbor environment '{env_mode}' requires {name} to name an existing regular file.")+ if env_mode == "mosaic" and not errors:+ try:+ from mosaic_sandbox.credentials import resolve_token+ except ImportError:+ return errors # Harbor's preflight reports the missing extra.+ _token, source = resolve_token()+ # The SDK accepts a Mosaic token in E2B_API_KEY for E2B migrations, but+ # that variable belongs to the e2b backend and never reaches Mosaic.+ if source == "e2b_environment":+ errors.append(+ "Harbor environment 'mosaic' does not forward E2B_API_KEY; set MOSAIC_API_TOKEN or run `mos login`."+ )+ return errors-_MERGED_KUBECONFIG_CACHE: dict[str, Path] = {}+def _harbor_missing_dependency_summary(exc: ImportError) -> str:+ """Keep Harbor's diagnosis, without its unpinned install commands or cloud-bundle hint."""+ lines = str(exc).strip().splitlines()+ summary = lines[0] if lines else type(exc).__name__+ for suffix in (" Install it with:", " Install them with:"):+ summary = summary.removesuffix(suffix)+ return summary.strip().rstrip(".")-def _resolve_single_kubeconfig(raw_kubeconfig: str | None = None) -> Path | None:- """Resolve a single existing kubeconfig file path from KUBECONFIG or default location.+def _cwsandbox_prerequisite_errors(env_mode: str) -> list[str]:+ """Check what Harbor's cwsandbox backend needs; Harbor 0.24 no longer checks it."""+ if harbor_environment_type(env_mode) != "cwsandbox":+ return []+ required_modules = ("cwsandbox", "wandb") if env_mode == "wandb" else ("cwsandbox",)+ if missing := [name for name in required_modules if importlib.util.find_spec(name) is None]:+ return [+ f"Harbor environment '{env_mode}' needs optional dependencies: {', '.join(missing)}. "+ f"{_environment_extra_install_hint(env_mode)}"+ ]+ if env_mode == "wandb":+ netrc_paths = [os.environ.get("NETRC", ""), str(Path.home() / ".netrc"), str(Path.home() / "_netrc")]+ if not os.environ.get("WANDB_API_KEY", "").strip() and not any(+ path and Path(path).expanduser().is_file() for path in netrc_paths+ ):+ return ["Harbor environment 'wandb' requires WANDB_API_KEY or a netrc file from `wandb login`."]+ elif not os.environ.get("CWSANDBOX_API_KEY", "").strip():+ return ["Harbor environment 'cwsandbox' requires CWSANDBOX_API_KEY."]+ return []- If multiple path entries are specified, merge them into a single temporary kubeconfig- file preserving all clusters, contexts, and credentials, restricted to permissions 0600.- """- raw = raw_kubeconfig if raw_kubeconfig is not None else os.environ.get("KUBECONFIG")- if raw:- valid_candidates: list[Path] = []- for entry in raw.split(os.pathsep):- cleaned = entry.strip()- if cleaned:- candidate = Path(cleaned).expanduser().resolve()- if candidate.is_file():- valid_candidates.append(candidate)- if valid_candidates:- if len(valid_candidates) == 1:- return valid_candidates[0]- paths_str = os.pathsep.join(str(p) for p in valid_candidates)- cache_key = os.pathsep.join(f"{p}:{p.stat().st_mtime_ns}:{p.stat().st_size}" for p in valid_candidates)- cached_path = _MERGED_KUBECONFIG_CACHE.get(cache_key)- if cached_path is not None and cached_path.is_file():- return cached_path- try:- import atexit- import tempfile-- import yaml- from kubernetes.config.kube_config import KubeConfigMerger-- with tempfile.TemporaryDirectory(prefix="kubeconfig_stage_") as stage_dir:- staged_paths: list[str] = []- for idx, cand in enumerate(valid_candidates):- cand_data = yaml.safe_load(cand.read_text(encoding="utf-8"))- if isinstance(cand_data, dict):- cand_data = _unwrap_kubeconfig_node(cand_data, source_path=str(cand))- staged_file = Path(stage_dir) / f"cand_{idx}.yaml"- staged_file.write_text(- yaml.safe_dump(cand_data, default_flow_style=False),- encoding="utf-8",- )- staged_file.chmod(0o600)- staged_paths.append(str(staged_file))- else:- staged_paths.append(str(cand))- merger = KubeConfigMerger(os.pathsep.join(staged_paths))- fallback_dirs = [p.parent for p in valid_candidates]- merged_dict = _unwrap_kubeconfig_node(merger.config, fallback_dirs=fallback_dirs)- if not merged_dict or not isinstance(merged_dict, dict):- logger.error("KubeConfigMerger produced empty or invalid configuration from %s", paths_str)- return None- with tempfile.NamedTemporaryFile(- mode="w",- encoding="utf-8",- prefix="merged_kubeconfig_",- suffix=".yaml",- delete=False,- ) as handle:- merged_path = Path(handle.name)- yaml.safe_dump(merged_dict, handle, default_flow_style=False)- merged_path.chmod(0o600)- _MERGED_KUBECONFIG_CACHE[cache_key] = merged_path- atexit.register(lambda: merged_path.unlink(missing_ok=True))- logger.info(- "Merged %d kubeconfig files into temporary file %s",- len(valid_candidates),- merged_path,- )- return merged_path- except Exception as exc:- logger.error(- "Failed to merge multiple kubeconfigs (%s): %s",- raw,- exc,- )- return None++def _environment_extra_install_hint(env_mode: str) -> str:+ extra = HARBOR_ENVIRONMENT_EXTRAS.get(env_mode)+ if extra is not None:+ return f"Install 'harbor[{extra}]=={HARBOR_VERSION}'."+ system_hints = {+ "apple-container": "Install the Apple container CLI; Harbor has no Python extra for this backend.",+ "openshift": "Install the OpenShift oc CLI; Harbor has no Python extra for this backend.",+ "singularity": "Install the singularity CLI; Harbor has no Python extra for this backend.",+ }+ return system_hints.get(env_mode, "Reinstall SkillEvaluator with its Tier 3 extra.")+++def _check_ack_cluster_readiness(environment_kwargs: Mapping[str, Any]) -> None:+ """Load ACK credentials and run a bounded namespaced pod-list probe."""+ from kubernetes import client as k8s_client+ from kubernetes import config as k8s_config++ load_kwargs: dict[str, str] = {}+ if context := environment_kwargs.get("context"):+ load_kwargs["context"] = str(context)+ if kubeconfig := environment_kwargs.get("kubeconfig"):+ load_kwargs["config_file"] = str(kubeconfig)+ try:+ k8s_config.load_kube_config(**load_kwargs)+ except k8s_config.ConfigException:+ k8s_config.load_incluster_config()++ api_client = k8s_client.ApiClient()+ try:+ core_api = k8s_client.CoreV1Api(api_client)+ core_api.list_namespaced_pod(+ namespace=str(environment_kwargs["namespace"]),+ limit=1,+ _request_timeout=(5, 10),+ )+ finally:+ api_client.close()+++_ACK_CLUSTER_READINESS_SUBPROCESS_TIMEOUT_SECONDS = 20+_ACK_CLUSTER_READINESS_REAP_TIMEOUT_SECONDS = 5+_ACK_CLUSTER_READINESS_PROBE_CODE = """\+import json+import sys++from skillevaluator.tier3.harbor.runner import _check_ack_cluster_readiness++try:+ _check_ack_cluster_readiness(json.loads(sys.stdin.read()))+except Exception as exc:+ sys.stderr.write((str(exc) or type(exc).__name__)[:4096])+ raise SystemExit(1) from None+"""+++def _terminate_ack_readiness_process(process: subprocess.Popen[str]) -> None:+ """Kill the ACK probe and its exec-auth descendants, then reap it."""+ if os.name == "posix":+ with suppress(ProcessLookupError):+ os.killpg(process.pid, signal.SIGKILL)+ else:+ process.kill()+ try:+ process.wait(timeout=_ACK_CLUSTER_READINESS_REAP_TIMEOUT_SECONDS)+ except subprocess.TimeoutExpired:+ process.kill()+ process.wait(timeout=_ACK_CLUSTER_READINESS_REAP_TIMEOUT_SECONDS)+++def _check_ack_cluster_readiness_subprocess(+ environment_kwargs: Mapping[str, Any],+ *,+ subprocess_env: Mapping[str, str],+) -> None:+ """Run ACK's bounded pod-list probe under Harbor's exact child environment."""+ validated = validate_environment_kwargs(dict(environment_kwargs), env_mode="ack")+ if policy_error := _environment_kwarg_policy_error("ack", validated):+ raise ValueError(policy_error)+ validated = _absolute_host_path_kwargs("ack", validated)+ payload = {name: validated[name] for name in ("namespace", "context", "kubeconfig") if name in validated}+ encoded = json.dumps(payload, ensure_ascii=True, separators=(",", ":"), allow_nan=False)+ child_env = dict(subprocess_env)+ redacted_values = secret_values_from_environment(child_env)+ redacted_values.update(str(value) for value in payload.values() if isinstance(value, str) and value)+ process: subprocess.Popen[str] | None = None+ try:+ with _harbor_launch_cwd() as launch_cwd:+ # ``-P`` keeps the probe's working directory off ``sys.path``.+ process = subprocess.Popen(+ [sys.executable, "-P", "-c", _ACK_CLUSTER_READINESS_PROBE_CODE],+ stdin=subprocess.PIPE,+ stdout=subprocess.PIPE,+ stderr=subprocess.PIPE,+ text=True,+ cwd=str(launch_cwd),+ env=_harbor_launch_environment(child_env),+ start_new_session=os.name == "posix",+ )+ stdout, stderr = process.communicate(+ encoded,+ timeout=_ACK_CLUSTER_READINESS_SUBPROCESS_TIMEOUT_SECONDS,+ )+ except subprocess.TimeoutExpired:+ assert process is not None+ _terminate_ack_readiness_process(process)+ raise RuntimeError(+ "ACK namespaced pod-list readiness probe timed out after "+ f"{_ACK_CLUSTER_READINESS_SUBPROCESS_TIMEOUT_SECONDS} seconds"+ ) from None+ except OSError as exc:+ if process is not None and process.returncode is None:+ _terminate_ack_readiness_process(process)+ detail = redact_progress_detail(exc, secret_values=redacted_values) or type(exc).__name__+ raise RuntimeError(f"ACK namespaced pod-list readiness probe could not start: {detail}") from None+ if process.returncode == 0:+ return+ output = "\n".join(part for part in (stderr, stdout) if part).strip()+ detail = redact_progress_detail(output, secret_values=redacted_values) or f"probe exited {process.returncode}"+ raise RuntimeError(f"ACK namespaced pod-list readiness probe failed: {detail[-2000:]}")+++def _modal_custom_config_status() -> tuple[bool, str | None]:+ raw = os.environ.get("MODAL_CONFIG_PATH")+ if raw is None:+ return False, None+ if not raw.strip():+ return False, "MODAL_CONFIG_PATH must name an existing regular file."+ try:+ is_file = Path(raw).expanduser().is_file()+ except (OSError, RuntimeError):+ is_file = False+ if not is_file:+ return False, "MODAL_CONFIG_PATH must name an existing regular file."+ return True, None+++def _python_dotenv_prerequisite_error() -> str | None:+ """Require the python-dotenv release that honors ``PYTHON_DOTENV_DISABLED``."""+ from importlib.metadata import PackageNotFoundError, version++ try:+ installed = version("python-dotenv")+ except PackageNotFoundError:+ return None+ release = re.match(r"(\d+)\.(\d+)", installed)+ if release is None or (int(release[1]), int(release[2])) >= (1, 2):return None- default_path = (Path.home() / ".kube" / "config").resolve()- return default_path if default_path.is_file() else None+ return (+ f"python-dotenv {installed} cannot disable Harbor's .env loading; "+ "install skillevaluator[tier3] to upgrade it to 1.2.0 or newer."+ )def _check_prerequisites(env_mode: str = DEFAULT_ENV_MODE,agents: list[str] | None = None,- environment_kwargs: Mapping[str, str] | None = None,+ environment_kwargs: Mapping[str, Any] | None = None,verify_live_cluster: bool = False,+ subprocess_env: Mapping[str, str] | None = None,) -> list[str]:"""Check Harbor and the selected environment (built-in, GKE, or local mode)."""if env_mode not in HARBOR_ENV_MODES:return [f"Unsupported Harbor environment '{env_mode}'. Choose one of: {', '.join(sorted(HARBOR_ENV_MODES))}"]if dotenv_error := _python_dotenv_prerequisite_error():return [dotenv_error]- if kwarg_errors := _environment_kwarg_prerequisite_errors(env_mode, environment_kwargs):+ if env_mode != "gke" and (kwarg_errors := _environment_kwarg_prerequisite_errors(env_mode, environment_kwargs)):return kwarg_errorsif env_mode == ENV_MODE_LOCAL:from skillevaluator.tier3.harbor import local_sandbox@@ -1892,7 +2302,7 @@ def _check_prerequisites(]if env_mode == "gke":- gke_errors = []+ gke_errors = list(_environment_kwarg_prerequisite_errors(env_mode, environment_kwargs))if not shutil.which("gcloud"):gke_errors.append("GKE requires the gcloud CLI to be installed. See https://cloud.google.com/sdk/docs/install"@@ -1910,12 +2320,6 @@ def _check_prerequisites("--region <REGION>' to configure credentials, or set the ""KUBECONFIG environment variable.")- missing_kwargs = _missing_gke_kwargs(environment_kwargs or {})- if missing_kwargs:- gke_errors.append(- f"GKE mode requires environment kwargs: {', '.join(missing_kwargs)}. "- "Specify them via --ek key=value or SKILLEVALUATOR_GKE_* environment variables."- )if is_gke_vertex_workload_identity_active(env_mode,agents or ["claude-code"],@@ -1961,6 +2365,23 @@ def _check_prerequisites(gke_errors.append(f"GKE cluster probe failed: {exc}")return gke_errors+ if env_mode == "singularity" and shutil.which("singularity") is None:+ return ["Harbor environment 'singularity' requires the singularity CLI on PATH."]+ if env_mode == "islo" and not os.environ.get("ISLO_API_KEY", "").strip():+ return ["Harbor environment 'islo' requires a non-empty ISLO_API_KEY in the host environment."]+ if env_mode == "modal":+ _, modal_config_error = _modal_custom_config_status()+ if modal_config_error:+ return [modal_config_error]+ try:+ modal_spec = importlib.util.find_spec("modal")+ except (ImportError, ValueError):+ modal_spec = None+ if modal_spec is None:+ return [+ f"Harbor environment 'modal' needs optional dependencies. {_environment_extra_install_hint(env_mode)}"+ ]+if env_mode == ENV_MODE_LOCAL:# Local mode is a host sandbox, not a Harbor-native backend: verify the# OS sandbox is usable and the requested agent CLIs are installed.@@ -2066,15 +2487,21 @@ def _check_prerequisites(return []-def _is_operator_owned_runtime_name(name: str, *, env_mode: str) -> bool:+def _is_operator_owned_runtime_name(name: str, *, env_mode: str = DEFAULT_ENV_MODE) -> bool:normalized = name.upper()- return (- is_sensitive_key(name)- or normalized in _RUNTIME_ENV_HOST_CONTROL_NAMES+ if (+ normalized in _RUNTIME_ENV_HOST_CONTROL_NAMESor normalized in _HARBOR_ENV_MODE_VARS.get(env_mode, frozenset())or normalized in _OPERATOR_OWNED_AGENT_ENVor normalized.startswith(_RUNTIME_ENV_HOST_CONTROL_PREFIXES)- )+ or is_operator_owned_or_provider_secret(name)+ ):+ return True+ if not is_sensitive_key(name):+ return False+ from skillevaluator.tier3.harbor.adapter import McpSecurityPolicy++ return name.strip() not in McpSecurityPolicy.from_environ().allowed_secretsdef _resolve_runtime_env(@@ -2098,6 +2525,11 @@ def _resolve_runtime_env(}percent_references = set(re.findall(r"%([A-Za-z_][A-Za-z0-9_]*)%", template_value))references = dollar_references | percent_references+ if is_sensitive_key(name) and not references:+ errors.append(+ f"harbor.runtime_env.{name} must reference an environment variable rather than a literal secret"+ )+ continueowned_references = sorted(reference for reference in references if _is_operator_owned_runtime_name(reference, env_mode=env_mode))@@ -2229,7 +2661,12 @@ def _independent_anthropic_agent_credentials(return credentials-def _gateway_anthropic_agent_credentials(provider: ProviderConfig) -> dict[str, str]:+def _gateway_anthropic_agent_credentials(+ provider: ProviderConfig,+ *,+ env_mode: str = "docker",+ environment_kwargs: Mapping[str, str] | None = None,+) -> dict[str, str]:"""Use a shared gateway unless the operator selected a separate Claude route.A standalone Anthropic key keeps its native endpoint, so a native key is@@ -2239,8 +2676,8 @@ def _gateway_anthropic_agent_credentials(provider: ProviderConfig) -> dict[str,On the gateway route Claude Code stops sending pre-release request fields,which gateways that forward to another provider reject with HTTP 400."""- credentials = _independent_anthropic_agent_credentials()- if credentials.get("ANTHROPIC_API_KEY", "").strip():+ credentials = _independent_anthropic_agent_credentials(env_mode=env_mode, environment_kwargs=environment_kwargs)+ if credentials.get("ANTHROPIC_API_KEY", "").strip() or credentials.get("CLAUDE_CODE_USE_VERTEX", "").strip() == "1":return credentialsbase_url = credentials.get("ANTHROPIC_BASE_URL")if not base_url:@@ -3065,7 +3502,6 @@ def _run_harbor(expected_trials: int | None = None,expected_total_trials: int | None = None,include_task_names: list[str] | None = None,- environment_kwargs: Mapping[str, str] | None = None,) -> tuple[bool, str]:base_url = ((verifier_env or {}).get("OPENAI_BASE_URL")@@ -3103,6 +3539,11 @@ def _run_harbor(except ValueError as exc:return False, str(exc)+ # Preserve the historical exact-value protection for every selected child+ # value, and additionally protect detached credential URI/proxy userinfo+ # (including percent-decoded and schemeless proxy components).+ secret_values = {value for value in run_env.values() if len(value) >= MIN_EXACT_SECRET_CHARS}+ secret_values.update(secret_values_from_environment(run_env))command = build_harbor_run_command(dataset_path=dataset,agent=agent,@@ -3124,24 +3565,35 @@ def _run_harbor(try:handoff = _nvidia_build_key_handoff(run_env, env_mode=env_mode)# Harbor owns its phase deadlines, and native tasks may intentionally- # leave the agent unbounded. An outer deadline can preempt valid jobs,- # except when non-GKE jobs rely on short-lived Google ADC bearer tokens.- run_kwargs: dict[str, Any] = {}+ # leave the agent unbounded. Keep bounded streaming and process-tree+ # cleanup without imposing an outer orchestration deadline except when+ # non-GKE jobs rely on short-lived Google ADC bearer tokens.+ with _harbor_launch_cwd() as launch_cwd:+ result = _run_bounded_harbor_process(+ command,+ env=handoff.subprocess_env,+ cwd=launch_cwd,+ stdin_text=handoff.stdin_text,+ timeout_seconds=adc_timeout,+ max_output_bytes=_HARBOR_RUN_OUTPUT_MAX_BYTES,+ diagnostic_tail_chars=_HARBOR_RUN_DIAGNOSTIC_TAIL_CHARS,+ secret_values=secret_values,+ )+ except _HarborRunTimeoutError as exc:if adc_timeout is not None:- run_kwargs["timeout"] = adc_timeout- result = subprocess.run(- command,- capture_output=True,- text=True,- input=handoff.stdin_text,- env=handoff.subprocess_env,- check=False,- **run_kwargs,- )- except subprocess.TimeoutExpired:- return False, f"Harbor job exceeded Google ADC token lifetime ({adc_timeout:g}s) in {env_mode} mode"- except OSError as exc:- return False, str(exc)+ return False, f"Harbor job exceeded Google ADC token lifetime ({adc_timeout:g}s) in {env_mode} mode"+ return False, _redact_harbor_diagnostic(exc, secret_values=secret_values)+ except (OSError, RuntimeError, UnicodeError) as exc:+ detail = _redact_harbor_diagnostic(exc, secret_values=secret_values)+ if not detail:+ detail = _redact_harbor_diagnostic(type(exc).__name__, secret_values=secret_values)+ return False, detail+ if result.output_exceeded:+ safe_tail = redact_progress_detail(result.output_tail, secret_values=secret_values)+ detail = f"harbor run output exceeded the {_HARBOR_RUN_OUTPUT_MAX_BYTES}-byte safety limit"+ if safe_tail:+ detail += f". Last output: {safe_tail[-2000:]}"+ return False, _redact_harbor_diagnostic(detail, secret_values=secret_values)if result.returncode == 0:validation_ok, validation_detail = _validate_harbor_job_result(jobs_dir,@@ -3597,7 +4049,7 @@ def _run_stop_on_pass_variant(override_storage_mb: int | None,agent_import_path: str | None = None,verifier_env: Mapping[str, str] | None = None,- environment_kwargs: Mapping[str, str] | None = None,+ environment_kwargs: Mapping[str, Any] | None = None,) -> list[str]:"""Run each case one attempt at a time, stopping its attempts on first pass."""errors: list[str] = []@@ -3624,7 +4076,6 @@ def _run_stop_on_pass_variant(environment_kwargs=environment_kwargs,expected_trials=1,include_task_names=[task_name],- environment_kwargs=environment_kwargs,)job_dir = jobs_dir / job_nameattempt_job_dirs.append(job_dir)@@ -3659,7 +4110,7 @@ def _run_agent_pair(pass_threshold: float = 0.50,task_names: list[str] | None = None,verifier_env: Mapping[str, str] | None = None,- environment_kwargs: Mapping[str, str] | None = None,+ environment_kwargs: Mapping[str, Any] | None = None,) -> list[str]:jobs = [("with", with_skill)]if baseline is not None:@@ -3722,7 +4173,6 @@ def _run_agent_pair(verifier_env=verifier_env,environment_kwargs=environment_kwargs,expected_trials=expected_trials,- environment_kwargs=environment_kwargs,): variantfor (variant, dataset), condition_concurrency in zip(jobs, job_concurrency, strict=True)}@@ -3885,7 +4335,7 @@ def _run_harbor_eval_impl(override_storage_mb: int | None = None,evaluated_source: dict[str, str] | None = None,progress_reporter: ProgressReporter | None = None,- environment_kwargs: Mapping[str, str] | None = None,+ repo_context_exclude_paths: Sequence[Path] = (),_evaluator_skill_path: Path | None = None,_snapshot_publication_target: dict[str, str] | None = None,_monotonic_start: float | None = None,@@ -3959,9 +4409,14 @@ def _run_harbor_eval_impl(# ``reuse``/``rebuild`` opt into the shared pre-built eval base image.base_image_mode = harbor_config.get("base_image_mode", "disabled")task_source = harbor_config.get("task_source", "auto")+ resolved_raw_environment_kwargs = _resolve_environment_kwargs(+ env_mode,+ cli_kwargs=environment_kwargs,+ environ=os.environ,+ )try:effective_environment_kwargs = validate_environment_kwargs(- dict(environment_kwargs or {}),+ dict(resolved_raw_environment_kwargs),env_mode=env_mode,)except (RecursionError, TypeError, ValueError) as exc:@@ -4046,19 +4501,35 @@ def _run_harbor_eval_impl())- config_env_kwargs = harbor_config.get("environment_kwargs", {})- resolved_environment_kwargs = _resolve_environment_kwargs(- env_mode,- config_kwargs=config_env_kwargs,- cli_kwargs=environment_kwargs,- environ=os.environ,- )+ prerequisite_subprocess_env: dict[str, str] | None = None+ if env_mode == "ack" and agents:+ preflight_agent = agents[0]+ runtime_provider_env = {+ name: value for name, value in provider_env.items() if name not in _VERIFIER_JUDGE_CONTROL_ENV_VARS+ }+ prerequisite_subprocess_env = _harbor_subprocess_environment(+ env_mode=env_mode,+ provider=provider,+ configured_runtime_env=configured_runtime_env,+ provider_env=runtime_provider_env,+ agent=preflight_agent,+ agent_model=model_resolution[preflight_agent]["model"],+ )+ prerequisite_subprocess_env.update(+ _agent_credentials(+ provider=provider,+ agent=preflight_agent,+ env_mode=env_mode,+ )+ )+ prerequisite_subprocess_env.update(_job_judge_subprocess_env(provider_env, grading_mode))reporter.emit(ProgressEvent(stage="environment-preflight", state="running", detail=env_mode))prereq_errors = _check_prerequisites(env_mode=env_mode,agents=agents,- environment_kwargs=resolved_environment_kwargs,+ environment_kwargs=effective_environment_kwargs,+ subprocess_env=prerequisite_subprocess_env,)if prereq_errors:reporter.emit(ProgressEvent(stage="environment-preflight", state="failed", detail="; ".join(prereq_errors)))@@ -4094,7 +4565,7 @@ def _run_harbor_eval_impl(configured_runtime_env=configured_runtime_env,env_mode=env_mode,model_sources={agent: details["source"] for agent, details in model_resolution.items()},- environment_kwargs=resolved_environment_kwargs,+ environment_kwargs=effective_environment_kwargs,)except ValueError as exc:reporter.emit(ProgressEvent(stage="credential-validation", state="failed", detail=str(exc)))@@ -4113,9 +4584,8 @@ def _run_harbor_eval_impl(# this import must remain lazy to avoid a module cycle.from skillevaluator.tier3.harbor.runtime_preflight import (CredentialProbeDisposition,- ModelCatalogFailureKind,- _is_vertex_openapi_endpoint,credential_probe_disposition,+ is_unverified_gke_vertex_auth_probe,probe_model,)@@ -4215,22 +4685,11 @@ def _run_harbor_eval_impl(safe_detail = redact_progress_detail(probe.detail, secret_values=runtime_secret_values)disposition = credential_probe_disposition(selected_provider, probe, env_mode=env_mode)- is_vertex = (- _is_vertex_openapi_endpoint(getattr(selected_provider, "base_url", None))- or getattr(selected_provider, "credential_env", None) == "CLAUDE_CODE_USE_VERTEX"- )- is_auth_failure = getattr(probe, "failure_kind", None) in {- ModelCatalogFailureKind.AUTHENTICATION,- ModelCatalogFailureKind.AUTHORIZATION,- }if probe.ok and disposition == CredentialProbeDisposition.DEGRADED:safe_detail = "model catalog access does not verify runtime credentials for this endpoint"elif (- not probe.ok- and env_mode == "gke"- and is_vertex- and is_auth_failure- and disposition == CredentialProbeDisposition.DEGRADED+ disposition == CredentialProbeDisposition.DEGRADED+ and is_unverified_gke_vertex_auth_probe(selected_provider, probe, env_mode=env_mode)):safe_detail = GKE_HOST_UNVERIFIED_VERTEX_AUTH_DETAILcredential_validation_targets.append(@@ -4672,8 +5131,8 @@ def _run_harbor_eval_impl(override_cpus=override_cpus,override_memory_mb=override_memory_mb,override_storage_mb=override_storage_mb,- agent_import_path=nvidia_build_agent_import_paths.get(agent),- environment_kwargs=resolved_environment_kwargs,+ agent_import_path=agent_import_paths.get(agent),+ environment_kwargs=effective_environment_kwargs,)if not preflight.ok:preflight_errors.append(f"{agent} runtime preflight failed: {preflight.detail}")@@ -4724,7 +5183,7 @@ def _run_harbor_eval_impl(pass_threshold=float(pass_threshold),task_names=task_selectors,verifier_env=job_judge_verifier_env,- environment_kwargs=resolved_environment_kwargs,+ environment_kwargs=effective_environment_kwargs,)active_agents: set[str] = set()@@ -4888,10 +5347,12 @@ def _run_harbor_eval_impl(report_path, report_warning = _render_current_report()if report_path is not None:results["report_path"] = str(report_path)- else:- report_warning = "HTML report was not generated: report file is missing"- except Exception as exc:- report_warning = f"HTML report was not generated: {exc}"+ elif stale_report_path is not None:+ try:+ stale_report_path.resolve().relative_to(run_dir.resolve())+ stale_report_path.unlink(missing_ok=True)+ except (OSError, RuntimeError, ValueError):+ passif is_gke_vertex_workload_identity_active(env_mode, agents):results.setdefault("security_notices", []).append(GKE_WORKLOAD_IDENTITY_WARNING_MESSAGE)if report_warning:diff --git a/src/skillevaluator/tier3/harbor/runtime_preflight.py b/src/skillevaluator/tier3/harbor/runtime_preflight.pyindex f3b3b7a3..7c6b03e6 100644--- a/src/skillevaluator/tier3/harbor/runtime_preflight.py+++ b/src/skillevaluator/tier3/harbor/runtime_preflight.py@@ -755,6 +755,24 @@ from skillevaluator.provider_config import ()+def is_unverified_gke_vertex_auth_probe(+ provider: ProviderConfig,+ probe: ModelProbeResult,+ *,+ env_mode: str,+) -> bool:+ """Return whether a failed probe represents host-unverified GKE Workload Identity Vertex auth."""+ is_vertex = (+ _is_vertex_openapi_endpoint(getattr(provider, "base_url", None))+ or getattr(provider, "credential_env", None) == "CLAUDE_CODE_USE_VERTEX"+ )+ is_auth_failure = getattr(probe, "failure_kind", None) in {+ ModelCatalogFailureKind.AUTHENTICATION,+ ModelCatalogFailureKind.AUTHORIZATION,+ }+ return not probe.ok and env_mode == "gke" and is_vertex and is_auth_failure++def credential_probe_disposition(provider: ProviderConfig,probe: ModelProbeResult,@@ -1574,6 +1592,26 @@ def _probe_vertex_anthropic_model()http_timeout = max(0.1, effective_deadline - monotonic())+ return _execute_vertex_http_probe(+ provider,+ req,+ http_timeout=http_timeout,+ service_label="Vertex AI",+ success_detail=f"model {provider.model} is available on Vertex AI ({region})",+ not_found_model_label=f"{provider.model} ({model_id})",+ )+++def _execute_vertex_http_probe(+ provider: ProviderConfig,+ req: urllib.request.Request,+ *,+ http_timeout: float,+ service_label: str,+ success_detail: str,+ not_found_model_label: str,+) -> ModelProbeResult:+ """Execute a Vertex HTTP probe request and map status/exceptions to ModelProbeResult."""try:with _urlopen_without_redirects(req, timeout=http_timeout) as response:if HTTPStatus.OK <= response.status < HTTPStatus.MULTIPLE_CHOICES:@@ -1581,14 +1619,14 @@ def _probe_vertex_anthropic_model(True,provider.provider,provider.model,- f"model {provider.model} is available on Vertex AI ({region})",+ success_detail,catalog_authoritative=True,)return ModelProbeResult(False,provider.provider,provider.model,- f"Vertex AI probe returned unexpected HTTP status {response.status}",+ f"{service_label} probe returned unexpected HTTP status {response.status}",failure_kind=ModelCatalogFailureKind.OTHER_HTTP,http_status=response.status,)@@ -1601,7 +1639,7 @@ def _probe_vertex_anthropic_model(False,provider.provider,provider.model,- f"Vertex AI probe rejected unexpected redirect ({exc.code}): {body}",+ f"{service_label} probe rejected unexpected redirect ({exc.code}): {body}",failure_kind=ModelCatalogFailureKind.INVALID_CONFIGURATION,http_status=exc.code,)@@ -1610,7 +1648,7 @@ def _probe_vertex_anthropic_model(False,provider.provider,provider.model,- f"Vertex AI authentication failed (401): {body}",+ f"{service_label} authentication failed (401): {body}",failure_kind=ModelCatalogFailureKind.AUTHENTICATION,http_status=HTTPStatus.UNAUTHORIZED,)@@ -1619,7 +1657,7 @@ def _probe_vertex_anthropic_model(False,provider.provider,provider.model,- f"Vertex AI permission denied (403): {body}",+ f"{service_label} permission denied (403): {body}",failure_kind=ModelCatalogFailureKind.AUTHORIZATION,http_status=HTTPStatus.FORBIDDEN,)@@ -1628,7 +1666,7 @@ def _probe_vertex_anthropic_model(False,provider.provider,provider.model,- f"Vertex AI model {provider.model} ({model_id}) not found (404): {body}",+ f"{service_label} model {not_found_model_label} not found (404): {body}",failure_kind=ModelCatalogFailureKind.MODEL_NOT_FOUND,http_status=HTTPStatus.NOT_FOUND,)@@ -1637,7 +1675,7 @@ def _probe_vertex_anthropic_model(False,provider.provider,provider.model,- f"Vertex AI rate limit / quota exceeded (429): {body}",+ f"{service_label} rate limit / quota exceeded (429): {body}",failure_kind=ModelCatalogFailureKind.UNAVAILABLE,http_status=HTTPStatus.TOO_MANY_REQUESTS,)@@ -1645,7 +1683,7 @@ def _probe_vertex_anthropic_model(False,provider.provider,provider.model,- f"Vertex AI request failed with HTTP {exc.code}: {body}",+ f"{service_label} request failed with HTTP {exc.code}: {body}",failure_kind=ModelCatalogFailureKind.OTHER_HTTP,http_status=exc.code,)@@ -1654,7 +1692,7 @@ def _probe_vertex_anthropic_model(False,provider.provider,provider.model,- "Vertex AI model probe timed out",+ f"{service_label} model probe timed out",failure_kind=ModelCatalogFailureKind.UNAVAILABLE,)except Exception as exc:@@ -1662,17 +1700,21 @@ def _probe_vertex_anthropic_model(False,provider.provider,provider.model,- f"Vertex AI model probe failed: {type(exc).__name__}",+ f"{service_label} model probe failed: {type(exc).__name__}",failure_kind=ModelCatalogFailureKind.UNKNOWN,)-def _probe_vertex_anthropic_model_with_deadline(+def _run_vertex_probe_with_deadline(provider: ProviderConfig,*,timeout_seconds: float,+ slot: Any,+ thread_name: str,+ service_label: str,+ probe_fn: Any,) -> ModelProbeResult:- """Bound Vertex AI credential acquisition and rawPredict I/O within a single deadline."""+ """Bound Vertex credential acquisition and probe I/O within a single deadline."""if (not isinstance(timeout_seconds, (int, float))or isinstance(timeout_seconds, bool)@@ -1689,53 +1731,53 @@ def _probe_vertex_anthropic_model_with_deadline(result_queue: Queue[ModelProbeResult] = Queue(maxsize=1)deadline = monotonic() + timeout_seconds- if not _VERTEX_PROBE_SLOT.acquire(timeout=timeout_seconds):+ if not slot.acquire(timeout=timeout_seconds):return ModelProbeResult(False,provider.provider,provider.model,- "Vertex AI model probe timed out waiting for worker slot",+ f"{service_label} model probe timed out waiting for worker slot",failure_kind=ModelCatalogFailureKind.UNAVAILABLE,)def run_probe() -> None:try:- result = _probe_vertex_anthropic_model(provider, timeout_seconds=timeout_seconds, deadline=deadline)+ result = probe_fn(provider, timeout_seconds=timeout_seconds, deadline=deadline)except Exception as exc:result = ModelProbeResult(False,provider.provider,provider.model,- f"Vertex AI model probe failed: {type(exc).__name__}",+ f"{service_label} model probe failed: {type(exc).__name__}",failure_kind=ModelCatalogFailureKind.UNKNOWN,)finally:- _VERTEX_PROBE_SLOT.release()+ slot.release()result_queue.put_nowait(result)remaining = deadline - monotonic()if remaining <= 0:- _VERTEX_PROBE_SLOT.release()+ slot.release()return ModelProbeResult(False,provider.provider,provider.model,- "Vertex AI model probe timed out",+ f"{service_label} model probe timed out",failure_kind=ModelCatalogFailureKind.UNAVAILABLE,)try:- worker = Thread(target=run_probe, name="vertex-anthropic-model-probe", daemon=True)+ worker = Thread(target=run_probe, name=thread_name, daemon=True)worker.start()except BaseException as exc:- _VERTEX_PROBE_SLOT.release()+ slot.release()if not isinstance(exc, Exception):raisereturn ModelProbeResult(False,provider.provider,provider.model,- f"Vertex AI model probe failed: {type(exc).__name__}",+ f"{service_label} model probe failed: {type(exc).__name__}",failure_kind=ModelCatalogFailureKind.UNKNOWN,)try:@@ -1745,11 +1787,27 @@ def _probe_vertex_anthropic_model_with_deadline(False,provider.provider,provider.model,- "Vertex AI model probe timed out",+ f"{service_label} model probe timed out",failure_kind=ModelCatalogFailureKind.UNAVAILABLE,)+def _probe_vertex_anthropic_model_with_deadline(+ provider: ProviderConfig,+ *,+ timeout_seconds: float,+) -> ModelProbeResult:+ """Bound Vertex AI credential acquisition and rawPredict I/O within a single deadline."""+ return _run_vertex_probe_with_deadline(+ provider,+ timeout_seconds=timeout_seconds,+ slot=_VERTEX_PROBE_SLOT,+ thread_name="vertex-anthropic-model-probe",+ service_label="Vertex AI",+ probe_fn=_probe_vertex_anthropic_model,+ )++def _probe_vertex_openapi_model(provider: ProviderConfig,*,@@ -1804,98 +1862,15 @@ def _probe_vertex_openapi_model()http_timeout = max(0.1, effective_deadline - monotonic())- try:- with _urlopen_without_redirects(req, timeout=http_timeout) as response:- if HTTPStatus.OK <= response.status < HTTPStatus.MULTIPLE_CHOICES:- location_str = f" ({location})" if location else ""- return ModelProbeResult(- True,- provider.provider,- provider.model,- f"model {provider.model} is available on Vertex AI OpenAPI{location_str}",- catalog_authoritative=True,- )- return ModelProbeResult(- False,- provider.provider,- provider.model,- f"Vertex AI OpenAPI probe returned unexpected HTTP status {response.status}",- failure_kind=ModelCatalogFailureKind.OTHER_HTTP,- http_status=response.status,- )- except urllib.error.HTTPError as exc:- body = ""- with contextlib.suppress(Exception):- body = exc.read().decode("utf-8", errors="replace")[:300]- if HTTPStatus.MULTIPLE_CHOICES <= exc.code < HTTPStatus.BAD_REQUEST:- return ModelProbeResult(- False,- provider.provider,- provider.model,- f"Vertex AI OpenAPI probe rejected unexpected redirect ({exc.code}): {body}",- failure_kind=ModelCatalogFailureKind.INVALID_CONFIGURATION,- http_status=exc.code,- )- if exc.code == HTTPStatus.UNAUTHORIZED:- return ModelProbeResult(- False,- provider.provider,- provider.model,- f"Vertex AI OpenAPI authentication failed (401): {body}",- failure_kind=ModelCatalogFailureKind.AUTHENTICATION,- http_status=HTTPStatus.UNAUTHORIZED,- )- if exc.code == HTTPStatus.FORBIDDEN:- return ModelProbeResult(- False,- provider.provider,- provider.model,- f"Vertex AI OpenAPI permission denied (403): {body}",- failure_kind=ModelCatalogFailureKind.AUTHORIZATION,- http_status=HTTPStatus.FORBIDDEN,- )- if exc.code == HTTPStatus.NOT_FOUND:- return ModelProbeResult(- False,- provider.provider,- provider.model,- f"Vertex AI OpenAPI model {provider.model} not found (404): {body}",- failure_kind=ModelCatalogFailureKind.MODEL_NOT_FOUND,- http_status=HTTPStatus.NOT_FOUND,- )- if exc.code == HTTPStatus.TOO_MANY_REQUESTS:- return ModelProbeResult(- False,- provider.provider,- provider.model,- f"Vertex AI OpenAPI rate limit / quota exceeded (429): {body}",- failure_kind=ModelCatalogFailureKind.UNAVAILABLE,- http_status=HTTPStatus.TOO_MANY_REQUESTS,- )- return ModelProbeResult(- False,- provider.provider,- provider.model,- f"Vertex AI OpenAPI request failed with HTTP {exc.code}: {body}",- failure_kind=ModelCatalogFailureKind.OTHER_HTTP,- http_status=exc.code,- )- except TimeoutError:- return ModelProbeResult(- False,- provider.provider,- provider.model,- "Vertex AI OpenAPI model probe timed out",- failure_kind=ModelCatalogFailureKind.UNAVAILABLE,- )- except Exception as exc:- return ModelProbeResult(- False,- provider.provider,- provider.model,- f"Vertex AI OpenAPI model probe failed: {type(exc).__name__}",- failure_kind=ModelCatalogFailureKind.UNKNOWN,- )+ location_str = f" ({location})" if location else ""+ return _execute_vertex_http_probe(+ provider,+ req,+ http_timeout=http_timeout,+ service_label="Vertex AI OpenAPI",+ success_detail=f"model {provider.model} is available on Vertex AI OpenAPI{location_str}",+ not_found_model_label=provider.model,+ )def _probe_vertex_openapi_model_with_deadline(@@ -1904,81 +1879,14 @@ def _probe_vertex_openapi_model_with_deadline(timeout_seconds: float,) -> ModelProbeResult:"""Bound Vertex AI OpenAPI credential acquisition and chat/completions I/O within a single deadline."""- if (- not isinstance(timeout_seconds, (int, float))- or isinstance(timeout_seconds, bool)- or not math.isfinite(timeout_seconds)- or timeout_seconds <= 0- ):- return ModelProbeResult(- False,- provider.provider,- provider.model,- "model catalog timeout must be a positive number",- failure_kind=ModelCatalogFailureKind.INVALID_CONFIGURATION,- )-- result_queue: Queue[ModelProbeResult] = Queue(maxsize=1)- deadline = monotonic() + timeout_seconds- if not _VERTEX_OPENAPI_PROBE_SLOT.acquire(timeout=timeout_seconds):- return ModelProbeResult(- False,- provider.provider,- provider.model,- "Vertex AI OpenAPI model probe timed out waiting for worker slot",- failure_kind=ModelCatalogFailureKind.UNAVAILABLE,- )-- def run_probe() -> None:- try:- result = _probe_vertex_openapi_model(provider, timeout_seconds=timeout_seconds, deadline=deadline)- except Exception as exc:- result = ModelProbeResult(- False,- provider.provider,- provider.model,- f"Vertex AI OpenAPI model probe failed: {type(exc).__name__}",- failure_kind=ModelCatalogFailureKind.UNKNOWN,- )- finally:- _VERTEX_OPENAPI_PROBE_SLOT.release()- result_queue.put_nowait(result)-- remaining = deadline - monotonic()- if remaining <= 0:- _VERTEX_OPENAPI_PROBE_SLOT.release()- return ModelProbeResult(- False,- provider.provider,- provider.model,- "Vertex AI OpenAPI model probe timed out",- failure_kind=ModelCatalogFailureKind.UNAVAILABLE,- )-- try:- worker = Thread(target=run_probe, name="vertex-openapi-model-probe", daemon=True)- worker.start()- except BaseException as exc:- _VERTEX_OPENAPI_PROBE_SLOT.release()- if not isinstance(exc, Exception):- raise- return ModelProbeResult(- False,- provider.provider,- provider.model,- f"Vertex AI OpenAPI model probe failed: {type(exc).__name__}",- failure_kind=ModelCatalogFailureKind.UNKNOWN,- )- try:- return result_queue.get(timeout=remaining)- except Empty:- return ModelProbeResult(- False,- provider.provider,- provider.model,- "Vertex AI OpenAPI model probe timed out",- failure_kind=ModelCatalogFailureKind.UNAVAILABLE,- )+ return _run_vertex_probe_with_deadline(+ provider,+ timeout_seconds=timeout_seconds,+ slot=_VERTEX_OPENAPI_PROBE_SLOT,+ thread_name="vertex-openapi-model-probe",+ service_label="Vertex AI OpenAPI",+ probe_fn=_probe_vertex_openapi_model,+ )def probe_model(provider: ProviderConfig, *, timeout_seconds: float = 15.0) -> ModelProbeResult:@@ -2078,7 +1986,7 @@ def run_agent_runtime_preflight(override_memory_mb: int | None = None,override_storage_mb: int | None = None,agent_import_path: str | None = None,- environment_kwargs: Mapping[str, str] | None = None,+ environment_kwargs: Mapping[str, Any] | None = None,) -> PreflightResult:"""Start one real agent task and stop before the full A/B matrix."""task_name = _first_task_name(dataset)diff --git a/src/skillevaluator/tier3/harbor/secret_redaction.py b/src/skillevaluator/tier3/harbor/secret_redaction.pyindex 8d8ae80e..f2390b64 100644--- a/src/skillevaluator/tier3/harbor/secret_redaction.py+++ b/src/skillevaluator/tier3/harbor/secret_redaction.py@@ -12,6 +12,7 @@ from skillevaluator.tier3.eval_core.secret_redaction import (LOG_JWT_RE,LOG_NVAPI_RE,LOG_SK_RE,+ LOG_YA29_RE,OPENSHIFT_TOKEN_RE,redact_secrets_in_log_line,)@@ -23,6 +24,7 @@ __all__ = ["LOG_JWT_RE","LOG_NVAPI_RE","LOG_SK_RE",+ "LOG_YA29_RE","OPENSHIFT_TOKEN_RE","redact_secrets_in_log_line",]diff --git a/src/skillevaluator/tier3/harbor/secure_docker_environment.py b/src/skillevaluator/tier3/harbor/secure_docker_environment.pyindex b7811ca6..a06e7657 100644--- a/src/skillevaluator/tier3/harbor/secure_docker_environment.py+++ b/src/skillevaluator/tier3/harbor/secure_docker_environment.py@@ -55,6 +55,12 @@ from skillevaluator.provider_config import (refresh_host_vertex_adc_environment,sync_refreshed_adc_persistent_env,)+from skillevaluator.tier3.harbor.progress import secret_values_from_environment+from skillevaluator.tier3.harbor.secure_copy import (+ _absolute_lexical,+ copy_file_secure,+ copytree_secure,+)from skillevaluator.tier3.harbor.sensitive_stdin import (NVIDIA_BUILD_KEY_STDIN_ENV,NVIDIA_BUILD_STDIN_SENTINEL,@@ -3456,6 +3462,13 @@ class SkillEvaluatorSecureDockerEnvironment(SkillEvaluatorDockerEnvironment):getattr(self, "_persistent_env", None),fresh_token=merged.get("OPENAI_API_KEY"),)+ secret_values = set(_eligible_secret_values(merged.values()))+ secret_values.update(_sensitive_environment_values(merged))+ secret_values.update(_credential_uri_environment_values(merged))+ if secret_values:+ # Fail before writing or uploading a handoff if no structurally+ # safe output marker can represent these values.+ _collision_safe_redaction_marker(secret_values, include_short=True)remote_path = f"/tmp/.skillevaluator-exec-env-{uuid.uuid4().hex}.sh"handoff_scope = _SecureHandoffScope(environment_names=frozenset(merged),diff --git a/src/skillevaluator/tier3/harbor/templates/eval.py b/src/skillevaluator/tier3/harbor/templates/eval.pyindex 33bbd90f..2af1203c 100644--- a/src/skillevaluator/tier3/harbor/templates/eval.py+++ b/src/skillevaluator/tier3/harbor/templates/eval.py@@ -41,7 +41,7 @@ import urllib.errorimport urllib.requestfrom contextvars import ContextVarfrom fnmatch import fnmatchcase-from functools import lru_cache+from functools import lru_cache, partialfrom pathlib import Pathfrom typing import Any, NamedTuplefrom urllib.parse import quote, unquote_to_bytes, urlparse, urlsplit@@ -217,7 +217,24 @@ LOG_NVAPI_RE = re.compile(r"(?<![A-Za-z0-9_-])nvapi-[a-zA-Z0-9_-]{8,}|nvapi-" +LOG_CRSR_RE = re.compile(r"(?<![A-Za-z0-9_-])crsr_[a-f0-9]{16,}")OPENSHIFT_TOKEN_RE = re.compile(r"(?<![A-Za-z0-9_-])sha256~[A-Za-z0-9._~-]+")LOG_YA29_RE = re.compile(r"(?<![A-Za-z0-9_-])ya29\.[A-Za-z0-9_-]{20,}")-LOG_JWT_RE = re.compile(r"\beyJ[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{20,}\b")+# A JWT used to start at any ``\beyJ``, so in a run of JWT characters such as+# "eyJ-" * n every "-eyJ" was a start, and each start scanned to the end of the+# run looking for ".". Now a match starts only at the beginning of a run. The part+# of the run before its first ``\beyJ`` is captured as ``lead`` and written back+# unchanged, which keeps JWTs glued to a "-" (x-eyJ...) redacted. Later starts in+# the same run are never tried: their first segment reaches the same "." with+# fewer characters, so they could only fail where the first start failed. ``lead``+# stops at the first ``\beyJ`` and every earlier offset fails ``\beyJ``, so+# backtracking into it is cheap. No atomic groups or possessive quantifiers: the+# Harbor verifier copy runs on the task image's python3, which may predate 3.11.+LOG_JWT_RE = re.compile(+ r"(?<![A-Za-z0-9_-])(?P<lead>(?:(?!\beyJ)[A-Za-z0-9_-])*)"+ r"\beyJ[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{20,}\b"+)+# GitHub classic (ghp_/gho_/ghu_/ghs_/ghr_) and fine-grained (github_pat_) tokens.+# Single bounded character classes keep both patterns linear.+LOG_GITHUB_TOKEN_RE = re.compile(r"\bgh[pousr]_[A-Za-z0-9]{36,255}\b")+LOG_GITHUB_PAT_RE = re.compile(r"\bgithub_pat_[A-Za-z0-9_]{22,255}\b")def redact_secrets_in_log_line(line, *, extra_secret_values=None):@@ -232,7 +249,9 @@ def redact_secrets_in_log_line(line, *, extra_secret_values=None):line = LOG_GITHUB_PAT_RE.sub("github_pat_<redacted>", line)line = OPENSHIFT_TOKEN_RE.sub("sha256~<redacted>", line)line = LOG_YA29_RE.sub("ya29.<redacted>", line)- return LOG_JWT_RE.sub("jwt-<redacted>", line)+ if "eyJ" not in line: # every JWT match contains "eyJ"; skip the scan on ordinary lines+ return line+ return LOG_JWT_RE.sub(r"\g<lead>jwt-<redacted>", line)_DESTRUCTIVE_PATTERNS = [@@ -1663,7 +1682,36 @@ def _get_vertex_access_token(timeout_seconds=10.0):return None-def _chat_completion_payload(model, prompt, max_tokens, temperature, provider=None, request_url=None):+def _build_openai_response_format(schema, schema_name="judge_response"):+ return {+ "type": "json_schema",+ "json_schema": {+ "name": schema_name,+ "strict": True,+ "schema": schema,+ },+ }+++def _build_anthropic_output_config(schema):+ return {+ "format": {+ "type": "json_schema",+ "schema": schema,+ }+ }+++def _chat_completion_payload(+ model,+ prompt,+ max_tokens,+ temperature,+ provider=None,+ request_url=None,+ response_schema=None,+ schema_name="judge_response",+):resolved_provider = _public_provider() if provider is None else providerresolved_request_url = _resolve_url(resolved_provider) if request_url is None else request_urltoken_key = (@@ -2386,7 +2434,7 @@ def _call_openai_compatible(target_key = SchemaTargetKey(provider=provider, base_url=request_url, model=model)use_schema = response_schema is not None and target_key not in _SCHEMA_UNSUPPORTED_TARGETS- def _build_request(include_schema):+ def _build_request(include_schema, *, _key=api_key):payload = _chat_completion_payload(model,prompt,@@ -2400,16 +2448,40 @@ def _call_openai_compatible(return urllib.request.Request(request_url,data=json.dumps(payload).encode(),- headers={"Content-Type": "application/json", "Authorization": f"Bearer {api_key}"},+ headers={"Content-Type": "application/json", "Authorization": f"Bearer {_key}"},)# request_url was validated by _resolve_url() before this request.- raw_response = _urlopen_with_schema_fallback(- _build_request,- target_key=target_key,- use_schema=use_schema,- timeout=90,- )+ try:+ raw_response = _urlopen_with_schema_fallback(+ _build_request,+ target_key=target_key,+ use_schema=use_schema,+ timeout=90,+ )+ except urllib.error.HTTPError as http_err:+ if (+ http_err.code == 401+ and _is_vertex_openapi_url(request_url)+ and os.environ.get("SKILL_EVAL_LLM_CREDENTIAL_SOURCE", "").strip() == "ADC"+ ):+ logger.info("Vertex AI OpenAPI 401 received; attempting in-process ADC token refresh")+ refreshed_token = _get_vertex_access_token()+ if refreshed_token and refreshed_token != api_key:+ if "OPENAI_API_KEY" in os.environ:+ os.environ["OPENAI_API_KEY"] = refreshed_token+ if "SKILL_EVAL_LLM_API_KEY" in os.environ:+ os.environ["SKILL_EVAL_LLM_API_KEY"] = refreshed_token+ raw_response = _urlopen_with_schema_fallback(+ partial(_build_request, _key=refreshed_token),+ target_key=target_key,+ use_schema=use_schema,+ timeout=90,+ )+ else:+ raise+ else:+ raisebody = json.loads(raw_response)choices = body.get("choices") or [{}]first_choice = choices[0] if choices and isinstance(choices[0], dict) else {}@@ -2461,71 +2533,34 @@ def _call_public_llm_with_provenance(last_provenance = provenancetry:if provider == "anthropic":- content, error = _call_anthropic(prompt, candidate_model, max_tokens, temperature)- if error:- return None, _redact_configured_credentials(error), provenance- return content, None, provenance- if provider == "bedrock":- content, error = _call_bedrock(prompt, candidate_model, max_tokens, temperature)- if error:- return None, _redact_configured_credentials(error), provenance- return content, None, provenance-- api_key = (- os.environ.get("NVIDIA_API_KEY", "")- if provider == "nv_build"- else os.environ.get("SKILL_EVAL_LLM_API_KEY", "") or os.environ.get("OPENAI_API_KEY", "")- )- if not api_key:- return None, f"No API key configured for {provider}", provenance- request_url = _resolve_url(provider)- request = urllib.request.Request(- request_url,- data=json.dumps(- _chat_completion_payload(- candidate_model,- prompt,- max_tokens,- temperature,- provider=provider,- request_url=request_url,- )- ).encode(),- headers={"Content-Type": "application/json", "Authorization": f"Bearer {api_key}"},- )- # request_url was validated by _resolve_url() before this request.- try:- with urllib.request.urlopen(request, timeout=90) as response: # nosec B310- body = json.loads(response.read())- except urllib.error.HTTPError as http_err:- if (- http_err.code == 401- and _is_vertex_openapi_url(request_url)- and os.environ.get("SKILL_EVAL_LLM_CREDENTIAL_SOURCE", "").strip() == "ADC"- ):- logger.info("Vertex AI OpenAPI 401 received; attempting in-process ADC token refresh")- refreshed_token = _get_vertex_access_token()- if refreshed_token and refreshed_token != api_key:- api_key = refreshed_token- if "OPENAI_API_KEY" in os.environ:- os.environ["OPENAI_API_KEY"] = refreshed_token- if "SKILL_EVAL_LLM_API_KEY" in os.environ:- os.environ["SKILL_EVAL_LLM_API_KEY"] = refreshed_token- request.add_header("Authorization", f"Bearer {refreshed_token}")- with urllib.request.urlopen(request, timeout=90) as response: # nosec B310- body = json.loads(response.read())- else:- raise- else:- raise- content = body.get("choices", [{}])[0].get("message", {}).get("content", "")- if content is None:- content = ""- if candidate_model != requested_model:- logger.warning("LLM judge model %s failed; using fallback model %s", requested_model, candidate_model)- return content.strip(), None, provenance- except urllib.error.HTTPError as error:- detail, should_try_fallback = _format_http_error_with_fallback(error)+ content, error = _call_anthropic(+ prompt,+ candidate_model,+ max_tokens,+ temperature,+ response_schema=response_schema,+ provenance=provenance,+ )+ elif provider == "bedrock":+ content, error = _call_bedrock(prompt, candidate_model, max_tokens, temperature, provenance=provenance)+ else:+ content, error = _call_openai_compatible(+ prompt,+ provider,+ candidate_model,+ max_tokens,+ temperature,+ response_schema=response_schema,+ schema_name=schema_name,+ )+ except _JudgeRefusedError as refusal:+ # A classifier refusal is fallback-eligible like a missing model.+ errors.append(f"{candidate_model}: {refusal}")+ if not allow_model_fallback:+ return None, str(refusal), provenance+ continue+ except urllib.error.HTTPError as http_error:+ detail, should_try_fallback = _format_http_error_with_fallback(http_error)errors.append(f"{candidate_model}: {detail}")if not allow_model_fallback or not should_try_fallback:return None, detail, provenancediff --git a/src/skillevaluator/tier3/workflow.py b/src/skillevaluator/tier3/workflow.pyindex 46f6e83f..cb1573f9 100644--- a/src/skillevaluator/tier3/workflow.py+++ b/src/skillevaluator/tier3/workflow.py@@ -89,21 +89,31 @@ def _source_snapshot(skill_path: Path) -> Iterator[Path]:yield snapshot-def _preflight_options(skill_path: Path, params: dict[str, Any]) -> dict[str, str]:+def _resolve_workflow_environment_kwargs(params: dict[str, Any]) -> dict[str, Any]:+ """Parse CLI environment kwargs and resolve effective backend kwargs."""+ from skillevaluator.tier3.evals_config import parse_environment_kwarg_overrides+ from skillevaluator.tier3.harbor.runner import _resolve_environment_kwargs++ raw_cli_kwargs = params.get("environment_kwarg") or params.get("environment_kwargs") or ()+ parsed_environment_kwargs = parse_environment_kwarg_overrides(raw_cli_kwargs, env_mode=params["env_mode"])+ return _resolve_environment_kwargs(+ params["env_mode"],+ cli_kwargs=parsed_environment_kwargs,+ )+++def _preflight_options(skill_path: Path, params: dict[str, Any]) -> dict[str, Any]:"""Reject known configuration errors before generating a paid dataset."""from skillevaluator.cli import _evaluated_source_from_optionsfrom skillevaluator.provider_config import resolve_llm_provider- from skillevaluator.tier3.commands import (- parse_agent_model_overrides,- parse_environment_kwargs,- resolve_agents,- validate_agents,+ from skillevaluator.tier3.commands import parse_agent_model_overrides, resolve_agents, validate_agents+ from skillevaluator.tier3.evals_config import (+ _validate_config,+ load_evals_config,)- from skillevaluator.tier3.evals_config import _validate_config, load_evals_configfrom skillevaluator.tier3.harbor.runner import (_model_for_agent,_resolve_agent_runtime_plan,- _resolve_environment_kwargs,_resolve_runtime_env,_workspace_skills,)@@ -132,7 +142,7 @@ def _preflight_options(skill_path: Path, params: dict[str, Any]) -> dict[str, stif params.get(cli_name) is not None:effective.setdefault(group, {})["mode"] = params[cli_name]_validate_config(effective, config_path or skill_path / "evals" / "config.yml")- parse_environment_kwarg_overrides(params.get("environment_kwarg") or (), env_mode=params["env_mode"])+ resolved_environment_kwargs = _resolve_workflow_environment_kwargs(params)if harbor.get("stop_on_pass", False) and harbor.get("n_attempts", 1) == 1:raise ValueError("stop_on_pass requires n_attempts > 1")@@ -167,13 +177,7 @@ def _preflight_options(skill_path: Path, params: dict[str, Any]) -> dict[str, st)for agent in agents}- parsed_environment_kwargs = parse_environment_kwargs(params.get("environment_kwargs", ()))- resolved_environment_kwargs = _resolve_environment_kwargs(- params["env_mode"],- config_kwargs=harbor.get("environment_kwargs"),- cli_kwargs=parsed_environment_kwargs,- )- runtime_env, errors = _resolve_runtime_env(harbor.get("runtime_env"))+ runtime_env, errors = _resolve_runtime_env(harbor.get("runtime_env"), env_mode=params["env_mode"])if errors:raise ValueError("; ".join(errors))_resolve_agent_runtime_plan(@@ -183,6 +187,7 @@ def _preflight_options(skill_path: Path, params: dict[str, Any]) -> dict[str, stconfigured_runtime_env=runtime_env,env_mode=params["env_mode"],model_sources={agent: value[1] for agent, value in resolution.items()},+ environment_kwargs=resolved_environment_kwargs,)_evaluated_source_from_options(params["evaluated_source_repository"],@@ -195,24 +200,19 @@ def _preflight_options(skill_path: Path, params: dict[str, Any]) -> dict[str, stdef _preflight_environment(params: dict[str, Any],*,- environment_kwargs: dict[str, str] | None = None,+ environment_kwargs: dict[str, Any] | None = None,) -> None:"""Check installed runtimes and backend configuration without an agent run."""- from skillevaluator.tier3.commands import parse_agents, parse_environment_kwargs- from skillevaluator.tier3.harbor.runner import _check_prerequisites, _resolve_environment_kwargs+ from skillevaluator.tier3.commands import parse_agents+ from skillevaluator.tier3.harbor.runner import _check_prerequisites# These preflights contact a remote control plane or cluster. Leave those# probes in the execution engine instead of running them before generation.if params["env_mode"] in {"ack", "cwsandbox", "langsmith", "openshift", "wandb"}:return- if environment_kwargs is None:- parsed_environment_kwargs = parse_environment_kwargs(params.get("environment_kwargs", ()))- resolved_environment_kwargs = _resolve_environment_kwargs(- params["env_mode"],- cli_kwargs=parsed_environment_kwargs,- )- else:- resolved_environment_kwargs = environment_kwargs+ resolved_environment_kwargs = (+ _resolve_workflow_environment_kwargs(params) if environment_kwargs is None else environment_kwargs+ )errors = _check_prerequisites(env_mode=params["env_mode"],agents=parse_agents(params["agents"]),diff --git a/src/skillevaluator/tier3_environments.py b/src/skillevaluator/tier3_environments.pyindex 03212270..bf63048c 100644--- a/src/skillevaluator/tier3_environments.py+++ b/src/skillevaluator/tier3_environments.py@@ -215,5 +215,8 @@ for _alias, _harbor_type in HARBOR_ENVIRONMENT_TYPE_ALIASES.items():HARBOR_ENVIRONMENT_KWARGS[_alias] = (HARBOR_ENVIRONMENT_KWARGS[_harbor_type] - HARBOR_ENVIRONMENT_ALIAS_KWARGS[_alias].keys())+# SkillEvaluatorGKEEnvironment wrapper kwargs accepted on ``--env-mode gke`` in+# addition to stock Harbor's GKEEnvironment.__init__ parameters.+SKILLEVALUATOR_GKE_EXTRA_ENVIRONMENT_KWARGS = frozenset({"allow_workload_identity", "autopilot"})ENV_MODE_LOCAL = "local"DEFAULT_ENV_MODE = "docker"diff --git a/src/skillevaluator/utils/redaction.py b/src/skillevaluator/utils/redaction.pyindex 623c2408..bf20cb2d 100644--- a/src/skillevaluator/utils/redaction.py+++ b/src/skillevaluator/utils/redaction.py@@ -75,12 +75,17 @@ _TOKEN_COUNT_KEYS = {_SENSITIVE_KEY_CHAR = r"[a-z0-9_.-]"_SENSITIVE_KEY_WORD = (r"(?:api[_-]?key|secret|password|credential|authorization|bearer|token|"- r"access[_-]?key|session[_-]?token|private[_-]?key)"+ r"access[_-]?key|session[_-]?token|private[_-]?key|service[_-]?account[_-]?key|"+ r"cookie|session[_-]?cookie|client[_-]?certificate|certificate|cert|passphrase|oauth)")_SENSITIVE_KEY_PATTERN = (- r"[a-z0-9_.-]*(?:api[_-]?key|secret|password|credential|authorization|bearer|token|"- r"access[_-]?key|session[_-]?token|private[_-]?key|service[_-]?account[_-]?key|"- r"cookie|session[_-]?cookie|client[_-]?certificate|certificate|cert|passphrase|oauth)[a-z0-9_.-]*"+ rf"(?<!{_SENSITIVE_KEY_CHAR})"+ # ``lead`` is empty when the run starts at a word boundary. Otherwise the first+ # boundary follows the run's leading [.-] characters ("--api-key"), or its+ # leading [a-z0-9_] characters when the run is glued to a word character that+ # is not a key character ("é" in "étoken.secret").+ r"(?P<lead>\b|(?<=\w)[a-z0-9_]++(?=[.-])|(?<!\w)[.-]++(?=[a-z0-9_]))"+ rf"(?P<key>(?>{_SENSITIVE_KEY_CHAR}*?{_SENSITIVE_KEY_WORD}){_SENSITIVE_KEY_CHAR}*+)")_AUTH_HEADER_RE = re.compile(r"(?im)\b(?P<key>(?:proxy-)?authorization)\s*:\s*(?P<scheme>[A-Za-z]+)\s+[^\r\n]+")_CREDENTIAL_URI_USERINFO_RE = re.compile(r"(?i)(?P<scheme>[a-z][a-z0-9+.-]{0,31}://)(?P<userinfo>[^\s/?#]+@)")@@ -140,11 +145,25 @@ _REDACTIONS = ((_JWT_RE, r"\g<lead>jwt-<redacted>"),(re.compile(r"(?:AKIA|ASIA)[A-Z0-9]{16}"), "aws-access-key-<redacted>"),(re.compile(r"(?i)\bBearer\s+[A-Za-z0-9._~+/=-]{8,}"), "Bearer <redacted>"),- (re.compile(r"(?<![A-Za-z0-9_-])sk-[a-zA-Z0-9_-]{8,}"), "sk-<redacted>"),- (re.compile(r"(?<![A-Za-z0-9_-])nvapi-[a-zA-Z0-9_-]{8,}"), "nvapi-<redacted>"),- (re.compile(r"(?<![A-Za-z0-9_-])crsr_[a-f0-9]{16,}"), "crsr_<redacted>"),- (re.compile(r"(?<![A-Za-z0-9_-])sha256~[A-Za-z0-9._~-]+"), "sha256~<redacted>"),+ (re.compile(r"(?<![A-Za-z0-9])sk-[a-zA-Z0-9_-]{8,}"), "sk-<redacted>"),+ (re.compile(r"nvapi-[a-zA-Z0-9_-]{8,}"), "nvapi-<redacted>"),+ (re.compile(r"crsr_[a-f0-9]{16,}"), "crsr_<redacted>"),+ (re.compile(r"sha256~[A-Za-z0-9._~-]+"), "sha256~<redacted>"),(re.compile(r"(?<![A-Za-z0-9_-])ya29\.[A-Za-z0-9_-]{20,}"), "ya29.<redacted>"),+ # GitHub's p/o/u/r families retain the 36-character opaque body. The s+ # family also has a variable-length ``ghs_APPID_JWT`` stateless format.+ (+ re.compile(r"(?i)gh[pour]_[A-Za-z0-9]{36}"),+ "github-token-<redacted>",+ ),+ (+ re.compile(r"(?i)ghs_[A-Za-z0-9.\-_]{36,}"),+ "github-token-<redacted>",+ ),+ (re.compile(r"(?i)github_pat_[A-Za-z0-9_]{20,}"), "github-token-<redacted>"),+ (re.compile(r"(?i)xox[baprs]-[A-Za-z0-9-]{10,}"), "slack-token-<redacted>"),+ (re.compile(r"AIza[A-Za-z0-9_-]{20,}"), "google-api-key-<redacted>"),+ (re.compile(r"(?i)glpat-[A-Za-z0-9_-]{20,}"), "gitlab-token-<redacted>"),)diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.jsonindex e4e45fb8..b777f2ca 100644--- a/tests/golden/cli_surface.json+++ b/tests/golden/cli_surface.json@@ -314,16 +314,6 @@],"param_type": "option","type": "boolean"- },- {- "multiple": true,- "name": "environment_kwargs",- "opts": [- "--ek",- "--environment-kwarg"- ],- "param_type": "option",- "type": "text"}]},@@ -627,16 +617,6 @@"--progress"],"param_type": "option"- },- {- "multiple": true,- "name": "environment_kwargs",- "opts": [- "--ek",- "--environment-kwarg"- ],- "param_type": "option",- "type": "text"}]},@@ -2196,16 +2176,6 @@],"param_type": "option","type": "boolean"- },- {- "multiple": true,- "name": "environment_kwargs",- "opts": [- "--ek",- "--environment-kwarg"- ],- "param_type": "option",- "type": "text"}]},@@ -2509,16 +2479,6 @@"--progress"],"param_type": "option"- },- {- "multiple": true,- "name": "environment_kwargs",- "opts": [- "--ek",- "--environment-kwarg"- ],- "param_type": "option",- "type": "text"}]},diff --git a/tests/reporting/test_rich_markup_escaping.py b/tests/reporting/test_rich_markup_escaping.pyindex 586a7260..9739f9c9 100644--- a/tests/reporting/test_rich_markup_escaping.py+++ b/tests/reporting/test_rich_markup_escaping.py@@ -298,6 +298,7 @@ def test_cli_reporter_consoles_print_emoji_codes_literally(capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch) -> None:monkeypatch.setenv("COLUMNS", "250")+ monkeypatch.delenv("TERM", raising=False)results = [_rubric_result(EMOJI_TEXT),_finding_result(EMOJI_TEXT),diff --git a/tests/test_harbor_gke_environment.py b/tests/test_harbor_gke_environment.pyindex 6b78b21f..88ceaa24 100644--- a/tests/test_harbor_gke_environment.py+++ b/tests/test_harbor_gke_environment.py@@ -553,6 +553,31 @@ def test_gke_environment_create_pod_autopilot_storage_idempotency_and_custom_proassert probe_container.resources.limits["ephemeral-storage"] == "64Mi"+ ("node_name", "labels", "expected"),+ [+ ("gke-std-pool-1", {"autopilot.gke.io/selected-node-type": "general-purpose"}, True),+ ("gke-std-pool-1", {"cloud.google.com/gke-autopilot": "true"}, True),+ ("gk3-eval-pool-1", {}, True),+ ("gke-std-pool-1", {"evil.example.com/autopilot.gke.io/x": "1"}, False),+ ("gke-std-pool-1", {"autopilot.gke.io.evil.example/x": "1"}, False),+ ("gke-std-pool-1", {"x-autopilot.gke.io/y": "1"}, False),+ ("gke-std-pool-1", {"autopilot.gke.io/": "1"}, False),+ ("gke-std-pool-1", {"autopilot.gke.io/a/b": "1"}, False),+ ("gke-std-pool-1", {"team": "autopilot.gke.io/primary-container"}, False),+ ],+)+def test_is_autopilot_node_matches_the_autopilot_label_prefix_exactly(+ node_name: str,+ labels: dict[str, str],+ expected: bool,+) -> None:+ """Treat only a well-formed autopilot.gke.io/<name> label key as Autopilot, not that text elsewhere."""+ node = SimpleNamespace(metadata=SimpleNamespace(name=node_name, labels=labels))++ assert gke_env_mod._is_autopilot_node(node) is expected++@pytest.mark.parametrize(("init_exit_code", "init_waiting_reason", "pod_phase", "expected_error_snippet"),[@@ -988,34 +1013,53 @@ def test_gke_environment_seam_routes_untargeted_exec_to_main_and_probes_trusted_]+def _pre_dispatch_exec_api(handshake_errors: list[str | None]) -> SimpleNamespace:+ """Return a fake CoreV1Api whose routed exec handshake fails with each queued error before dispatch."""++ def connect_get_namespaced_pod_exec(*_args: object, **_kwargs: object) -> object:+ error = handshake_errors.pop(0) if handshake_errors else None+ if error is not None:+ raise RuntimeError(error)+ return object()++ api = SimpleNamespace(connect_get_namespaced_pod_exec=connect_get_namespaced_pod_exec)+ gke_env_mod._ensure_default_exec_container_routing(api)+ return api+++async def _harbor_style_exec(api: SimpleNamespace, result: ExecResult) -> ExecResult:+ """Open the exec stream the way Harbor does, reporting a failed handshake as an ExecResult."""+ try:+ await asyncio.to_thread(api.connect_get_namespaced_pod_exec, "pod-under-test", "skilleval", command=["sh"])+ except Exception as exc:+ return ExecResult(stdout=None, stderr=str(exc), return_code=1)+ return result++def test_gke_environment_exec_retries_transient_kubelet_errors(monkeypatch: pytest.MonkeyPatch,) -> None:- """Retry transient kubelet 10250 errors during exec(), but do not retry regular command failures."""+ """Retry transient kubelet 10250 errors raised before dispatch, but do not retry regular command failures."""async def fast_sleep(_sec: float) -> None:return Nonemonkeypatch.setattr(gke_env_mod.asyncio, "sleep", fast_sleep)+ transient_error = (+ "invalid literal for int() with base 10: "+ "'error sending request: Post \"https://10.128.0.47:10250/exec/skilleval/pod/main?command=sh&command=-c&command=OPENAI_API_KEY=ya29.super-secret\"'"+ )- # 1. Transient kubelet error recovers on second attempt+ # 1. Transient kubelet error before dispatch recovers on second attemptexec_calls = 0+ flaky_api = _pre_dispatch_exec_api([transient_error])async def flaky_super_exec(self: GKEEnvironment, command: str, *args: object, env: dict[str, str] | None = None, **kwargs: object) -> ExecResult:nonlocal exec_callsexec_calls += 1- if exec_calls == 1:- return ExecResult(- stdout=None,- stderr=(- "invalid literal for int() with base 10: "- "'error sending request: Post \"https://10.128.0.47:10250/exec/skilleval/pod/main?command=sh&command=-c&command=OPENAI_API_KEY=ya29.super-secret\"'"- ),- return_code=1,- )- return ExecResult(stdout="created", stderr="", return_code=0)+ return await _harbor_style_exec(flaky_api, ExecResult(stdout="created", stderr="", return_code=0))monkeypatch.setattr(GKEEnvironment, "exec", flaky_super_exec)env = object.__new__(SkillEvaluatorGKEEnvironment)@@ -1041,20 +1085,18 @@ def test_gke_environment_exec_retries_transient_kubelet_errors(# 3. Exhausted transient retries scrub secrets from stderr before returningexhausted_calls = 0+ exhausted_error = (+ "invalid literal for int() with base 10: "+ "'error sending request: Post \"https://10.128.0.47:10250/exec/skilleval/pod/main?command=sh&command=-c&command=OPENAI_API_KEY=ya29.super-secret-token\"'"+ )+ exhausted_api = _pre_dispatch_exec_api([exhausted_error] * 4)async def always_failing_super_exec(self: GKEEnvironment, command: str, *args: object, env: dict[str, str] | None = None, **kwargs: object) -> ExecResult:nonlocal exhausted_callsexhausted_calls += 1- return ExecResult(- stdout=None,- stderr=(- "invalid literal for int() with base 10: "- "'error sending request: Post \"https://10.128.0.47:10250/exec/skilleval/pod/main?command=sh&command=-c&command=OPENAI_API_KEY=ya29.super-secret-token\"'"- ),- return_code=1,- )+ return await _harbor_style_exec(exhausted_api, ExecResult(stdout="", stderr="", return_code=0))monkeypatch.setattr(GKEEnvironment, "exec", always_failing_super_exec)res_fail = asyncio.run(@@ -1070,6 +1112,112 @@ def test_gke_environment_exec_retries_transient_kubelet_errors(assert "?command=<redacted>" in (res_fail.stderr or "")+ ("scenario", "statuses", "command_stderr", "expected_handshakes", "expected_runs", "expect_stream_closed"),+ [+ # The command appends once, then its completion status is lost: never replay it ([P1]).+ ("status_lost_once", [None, 0], "", 1, 1, True),+ # Every status is lost: still exactly one dispatch before raising.+ ("status_always_lost", [None, None, None, None], "", 1, 1, True),+ # A dispatched command whose own stderr looks like a kubelet error is not replayed.+ ("dispatched_command_kubelet_like_stderr", [1, 0], "error sending request: upstream", 1, 1, False),+ # A handshake rejected before dispatch is retried, and the command runs once.+ ("pre_dispatch_no_agent", ["no-agent", 0], "", 2, 1, False),+ ],+)+def test_gke_environment_exec_never_replays_dispatched_command(+ monkeypatch: pytest.MonkeyPatch,+ make_gke_env: Callable[..., SkillEvaluatorGKEEnvironment],+ scenario: str,+ statuses: list[int | str | None],+ command_stderr: str,+ expected_handshakes: int,+ expected_runs: int,+ expect_stream_closed: bool,+) -> None:+ """Drive Harbor's real GKE exec so a lost completion status after the side effect raises after one dispatch."""+ from harbor.environments.gke import GKEExecStreamClosedError++ _ = scenario+ secret_val = "sk-ant-api03-replaytestvalue"+ handshakes: list[str | None] = []+ side_effects: list[str] = []+ queued_statuses = list(statuses)++ class FakeWSResponse:+ def __init__(self, returncode: int | None) -> None:+ self.returncode = returncode++ def is_open(self) -> bool:+ return False++ def run_forever(self, timeout: int | float | None = None) -> None:+ _ = timeout++ def close(self) -> None:+ return++ class FakeCoreV1Api:+ def __init__(self) -> None:+ self.api_client = object()++ def connect_get_namespaced_pod_exec(+ self, _name: str, _namespace: str, *, container: str | None = None, **_kwargs: object+ ) -> FakeWSResponse:+ handshakes.append(container)+ status = queued_statuses.pop(0)+ if status == "no-agent":+ raise ApiException(status=500, reason="Internal Server Error", http_resp=None)+ # The command runs once the stream is open, before any status is reported.+ side_effects.append("appended")+ return FakeWSResponse(returncode=status if isinstance(status, int) else None)++ def passthrough_stream(fn: Callable[..., FakeWSResponse], *args: object, **kwargs: object) -> FakeWSResponse:+ assert hasattr(getattr(fn, "__self__", None), "api_client")+ return fn(*args, **kwargs)++ async def direct_ensure_client(self: GKEEnvironment) -> None:+ return None++ async def fast_sleep(_sec: float) -> None:+ return None++ monkeypatch.setattr("harbor.environments.gke.stream", passthrough_stream)+ monkeypatch.setattr(GKEEnvironment, "_ensure_client", direct_ensure_client)+ monkeypatch.setattr(gke_env_mod.asyncio, "sleep", fast_sleep)++ fake_api = FakeCoreV1Api()+ if "no-agent" in statuses:+ # Harbor reports this 500 handshake rejection as "No agent available" when the body says so.+ original_connect = fake_api.connect_get_namespaced_pod_exec++ def no_agent_connect(*args: object, **kwargs: object) -> FakeWSResponse:+ try:+ return original_connect(*args, **kwargs)+ except ApiException as exc:+ exc.body = "No agent available"+ raise++ no_agent_connect.__self__ = fake_api # type: ignore[attr-defined]+ fake_api.connect_get_namespaced_pod_exec = no_agent_connect # type: ignore[method-assign]+ gke_env_mod._ensure_default_exec_container_routing(fake_api)+ env = make_gke_env(namespace="skilleval", fake_api=fake_api)+ env._persistent_env = {"ANTHROPIC_API_KEY": secret_val}+ env._read_exec_output = lambda _resp: ("", command_stderr) # type: ignore[method-assign]++ if expect_stream_closed:+ with pytest.raises(GKEExecStreamClosedError) as exc_info:+ asyncio.run(env.exec("echo line >> /logs/agent/out.txt"))+ assert secret_val not in str(exc_info.value)+ else:+ result = asyncio.run(env.exec("echo line >> /logs/agent/out.txt"))+ assert result.return_code == (1 if command_stderr else 0)++ assert len(handshakes) == expected_handshakes+ assert set(handshakes) == {"main"}+ assert side_effects == ["appended"] * expected_runs++@pytest.mark.parametrize(("stderr", "env_values", "must_not_contain", "must_contain"),[@@ -1341,3 +1489,74 @@ def test_wait_for_container_exec_ready_aborts_immediately_on_gke_warden_missing_asyncio.run(env_obj._wait_for_container_exec_ready(max_attempts=60))assert sleep_calls == []+++def test_gke_environment_preflight_normalizes_multipath_kubeconfig(+ monkeypatch: pytest.MonkeyPatch,+ tmp_path: Path,+) -> None:+ """Accept split KUBECONFIG in SkillEvaluatorGKEEnvironment.preflight() and raise SystemExit when missing."""+ monkeypatch.setattr("skillevaluator.tier3.harbor.gke_environment.shutil.which", lambda cmd: f"/usr/bin/{cmd}")++ valid_kc = tmp_path / "valid_kubeconfig"+ valid_kc.write_text("apiVersion: v1\nclusters: []\n", encoding="utf-8")+ missing_kc = tmp_path / "missing_kubeconfig"+ monkeypatch.setenv("KUBECONFIG", f"{missing_kc}{os.pathsep}{valid_kc}")++ # Split KUBECONFIG with one valid file succeeds where stock GKEEnvironment.preflight() would fail+ SkillEvaluatorGKEEnvironment.preflight()++ # All-missing KUBECONFIG raises SystemExit+ monkeypatch.setenv("KUBECONFIG", str(missing_kc))+ with pytest.raises(SystemExit, match="Kubernetes credentials"):+ SkillEvaluatorGKEEnvironment.preflight()+++def test_gke_environment_exec_handles_harbor_024_stream_closed_and_pod_not_found(+ monkeypatch: pytest.MonkeyPatch,+ make_gke_env: Callable[..., SkillEvaluatorGKEEnvironment],+) -> None:+ """Raise GKEExecStreamClosedError without replaying the command, redact it, and raise GKEPodNotFoundError on Warden missing pod."""+ from harbor.environments.base import ExecResult+ from harbor.environments.gke import GKEEnvironment, GKEExecStreamClosedError, GKEPodNotFoundError++ async def _no_sleep(_sec: float) -> None:+ return None++ monkeypatch.setattr("skillevaluator.tier3.harbor.gke_environment.asyncio.sleep", _no_sleep)+ secret_val = "sk-ant-api03-supersecretvalue123456"+ env_obj = make_gke_env(namespace="default")+ env_obj._persistent_env = {"ANTHROPIC_API_KEY": secret_val}++ # 1-2. A lost completion status is not replayed and raises with a redacted message+ attempts = 0++ async def _closed_exec(self_inner: object, **kwargs: object) -> ExecResult:+ nonlocal attempts+ attempts += 1+ if attempts == 1:+ raise GKEExecStreamClosedError(f"WebSocket closed: token={secret_val}")+ return ExecResult(stdout="ok", stderr=None, return_code=0)++ monkeypatch.setattr(GKEEnvironment, "exec", _closed_exec)+ with pytest.raises(GKEExecStreamClosedError) as exc_info:+ asyncio.run(env_obj.exec("echo hi"))+ assert attempts == 1+ assert exc_info.value.__cause__ is None+ assert secret_val not in str(exc_info.value)+ assert "<redacted>" in str(exc_info.value)++ # 3. Warden 400 missing-pod in stderr raises GKEPodNotFoundError with redacted message+ async def _warden_missing_exec(self_inner: object, **kwargs: object) -> ExecResult:+ return ExecResult(+ stdout="",+ stderr=f'Handshake status 400 Bad Request: {{"message":"Cannot connect to pod default/pod-1, not found."}} key={secret_val}',+ return_code=1,+ )++ monkeypatch.setattr(GKEEnvironment, "exec", _warden_missing_exec)+ with pytest.raises(GKEPodNotFoundError) as pod_exc_info:+ asyncio.run(env_obj.exec("echo hi"))+ assert secret_val not in str(pod_exc_info.value)+ assert "<redacted>" in str(pod_exc_info.value)+diff --git a/tests/test_harbor_gke_mode.py b/tests/test_harbor_gke_mode.pyindex 8a1b6b55..b3f0d750 100644--- a/tests/test_harbor_gke_mode.py+++ b/tests/test_harbor_gke_mode.py@@ -110,7 +110,10 @@ def test_resolve_environment_kwargs_strict_missing():def test_build_harbor_run_command_gke_includes_all_ek_flags():- """Command construction includes --env gke and sorted --ek key=value flags."""+ """Command construction includes --env SECURE_GKE_ENV_IMPORT_PATH and sorted JSON-encoded --ek flags."""+ from skillevaluator.tier3.evals_config import encode_environment_kwarg+ from skillevaluator.tier3.harbor.gke_environment import SECURE_GKE_ENV_IMPORT_PATH+command = build_harbor_run_command(dataset_path="/tmp/dataset",agent="claude-code",@@ -122,16 +125,16 @@ def test_build_harbor_run_command_gke_includes_all_ek_flags():assert "--env" in commandenv_idx = command.index("--env")- assert command[env_idx + 1] == "gke"+ assert command[env_idx + 1] == SECURE_GKE_ENV_IMPORT_PATH- # Verify all 5 kwargs are emitted as --ek key=value+ # Verify all 5 kwargs are emitted as JSON-encoded --ek key=valuefor key, val in COMPLETE_GKE_KWARGS.items():assert "--ek" in command- assert f"{key}={val}" in command+ assert encode_environment_kwarg(key, val) in commanddef test_build_harbor_run_command_gke_supports_agent_import_path():- """Support custom agent import path in GKE mode without emitting agent name flag."""+ """Support custom agent import path in GKE mode via Harbor 0.24.0 unified --agent flag."""custom_import = "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorClaudeCode"command = build_harbor_run_command(dataset_path="/tmp/dataset",@@ -143,11 +146,11 @@ def test_build_harbor_run_command_gke_supports_agent_import_path():environment_kwargs=COMPLETE_GKE_KWARGS,)- assert "--agent-import-path" in command- idx = command.index("--agent-import-path")+ assert "--agent" in command+ idx = command.index("--agent")assert command[idx + 1] == custom_importassert "-a" not in command- assert "--agent" not in command+ assert "--agent-import-path" not in commanddef test_build_harbor_run_command_gke_defaults_claude_code_import_path():@@ -161,11 +164,11 @@ def test_build_harbor_run_command_gke_defaults_claude_code_import_path():environment_kwargs=COMPLETE_GKE_KWARGS,)- assert "--agent-import-path" in command- idx = command.index("--agent-import-path")+ assert "--agent" in command+ idx = command.index("--agent")assert command[idx + 1] == "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorClaudeCode"assert "-a" not in command- assert "--agent" not in command+ assert "--agent-import-path" not in commanddef test_check_prerequisites_gke_reports_missing_gcloud(monkeypatch: pytest.MonkeyPatch):@@ -350,7 +353,7 @@ def test_gke_mode_rejects_claude_code_without_vertex_or_anthropic_key(monkeypatcmonkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)monkeypatch.delenv("CLAUDE_CODE_USE_VERTEX", raising=False)errors = _validate_agent_provider_credentials(- _provider("openai-compatible", "google/gemini-3.8-flash"),+ _provider("openai", "gpt-5.5"),["claude-code"],{},{"claude-code": "claude-sonnet-5"},@@ -392,7 +395,7 @@ def test_gke_mode_suggests_vertex_suffix_when_anthropic_key_missing(monkeypatch:monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)monkeypatch.delenv("CLAUDE_CODE_USE_VERTEX", raising=False)errors = _validate_agent_provider_credentials(- _provider("openai-compatible", "google/gemini-3.8-flash"),+ _provider("openai", "gpt-5.5"),["claude-code"],{},{"claude-code": "claude-sonnet-5"},@@ -407,7 +410,7 @@ def test_docker_mode_does_not_suggest_vertex_suffix(monkeypatch: pytest.MonkeyPamonkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)monkeypatch.delenv("CLAUDE_CODE_USE_VERTEX", raising=False)errors = _validate_agent_provider_credentials(- _provider("openai-compatible", "google/gemini-3.8-flash"),+ _provider("openai", "gpt-5.5"),["claude-code"],{},{"claude-code": "claude-sonnet-5"},@@ -448,13 +451,16 @@ def test_gke_runtime_preflight_forwards_environment_kwargs(monkeypatch: pytest.Menvironment_kwargs=COMPLETE_GKE_KWARGS,)+ from skillevaluator.tier3.evals_config import encode_environment_kwarg+ from skillevaluator.tier3.harbor.gke_environment import SECURE_GKE_ENV_IMPORT_PATH+assert result.ok is Truecommand = captured["command"]assert isinstance(command, list)assert "--env" in command- assert command[command.index("--env") + 1] == "gke"+ assert command[command.index("--env") + 1] == SECURE_GKE_ENV_IMPORT_PATHfor key, val in COMPLETE_GKE_KWARGS.items():- assert f"{key}={val}" in command+ assert encode_environment_kwarg(key, val) in commanddef test_gke_mode_rejects_vertex_ai_without_project_id(monkeypatch: pytest.MonkeyPatch):@@ -510,7 +516,7 @@ def test_build_harbor_run_command_rejects_sensitive_kwargs(sensitive_key: str):"""Reject environment kwargs containing credentials to protect the OS process table."""with pytest.raises(ValueError,- match=r"(?:Sensitive|disallowed) key or value detected in environment_kwargs",+ match=r"Sensitive key or value detected",):build_harbor_run_command(dataset_path="/tmp/dataset",@@ -525,7 +531,7 @@ def test_build_harbor_run_command_rejects_unlisted_backend_kwargs():"""Reject backend kwargs not in the safe constructor allowlist for the backend."""with pytest.raises(ValueError,- match=r"(?:Sensitive|disallowed) key or value detected in environment_kwargs",+ match=r"does not accept environment kwarg\(s\): unrecognized_constructor_param",):build_harbor_run_command(dataset_path="/tmp/dataset",@@ -548,6 +554,8 @@ def test_build_harbor_run_command_rejects_unlisted_backend_kwargs():)def test_build_harbor_run_command_permits_benign_substrings_in_values(key: str, benign_value: str) -> None:"""Permit valid non-sensitive infrastructure names containing words like turnkey, monkey, token."""+ from skillevaluator.tier3.evals_config import encode_environment_kwarg+cmd = build_harbor_run_command(dataset_path="/tmp/dataset",agent="claude-code",@@ -555,13 +563,13 @@ def test_build_harbor_run_command_permits_benign_substrings_in_values(key: str,env_mode="gke",environment_kwargs={**COMPLETE_GKE_KWARGS, key: benign_value},)- assert f"--ek={key}={benign_value}" in cmd or f"{key}={benign_value}" in " ".join(cmd)+ assert encode_environment_kwarg(key, benign_value) in cmddef test_build_harbor_run_command_rejects_actual_secret_values() -> None:"""Reject actual secret tokens passed in values of environment kwargs."""ya29_token = "ya29." + "a" * 25- with pytest.raises(ValueError, match=r"Sensitive key or value detected in environment_kwargs"):+ with pytest.raises(ValueError, match=r"Sensitive key or value detected"):build_harbor_run_command(dataset_path="/tmp/dataset",agent="claude-code",@@ -594,17 +602,13 @@ def test_check_prerequisites_gke_supports_colon_separated_kubeconfig(assert any("Kubernetes credentials" in err or "kubeconfig" in err.lower() for err in errors)[email protected]("benign_key", ["token-bucket", "token_bucket", "max_tokens", "prompt_tokens"])[email protected]("benign_key", ["token_bucket", "max_tokens", "prompt_tokens"])def test_build_harbor_run_command_permits_benign_token_keys(benign_key: str) -> None:- """Permit rate-limiting and token count keys in environment kwargs."""- cmd = build_harbor_run_command(- dataset_path="/tmp/dataset",- agent="claude-code",- job_name="gke-job",- env_mode="gke",- environment_kwargs={**COMPLETE_GKE_KWARGS, benign_key: "100"},- )- assert f"--ek={benign_key}=100" in cmd or f"{benign_key}=100" in " ".join(cmd)+ """Permit rate-limiting and token count keys in environment_kwargs secret validation."""+ from skillevaluator.tier3.evals_config import validate_environment_kwargs++ validated = validate_environment_kwargs({**COMPLETE_GKE_KWARGS, benign_key: "100"}, env_mode="gke")+ assert validated[benign_key] == "100"def test_check_prerequisites_gke_expands_user_in_kubeconfig(@@ -628,27 +632,27 @@ def test_check_prerequisites_gke_expands_user_in_kubeconfig(def test_parse_environment_kwargs_valid():- """Parse valid --ek key=value arguments with whitespace trimming."""+ """Parse valid --ek key=value arguments with whitespace trimming and JSON coercion."""assert parse_environment_kwargs(()) == {}assert parse_environment_kwargs(("setting=value",)) == {"setting": "value"}assert parse_environment_kwargs((" foo = bar ", "baz=123", "multi=a=b=c")) == {"foo": "bar",- "baz": "123",+ "baz": 123,"multi": "a=b=c",}def test_parse_environment_kwargs_missing_equals():"""Reject arguments missing the equals separator."""- with pytest.raises(ValueError, match=r"--ek/--environment-kwarg must be in KEY=VALUE form"):+ with pytest.raises(ValueError, match=r"Invalid --environment-kwarg entry 1: expected KEY=VALUE"):parse_environment_kwargs(("no_equals",))def test_parse_environment_kwargs_empty_key():"""Reject arguments with an empty key."""- with pytest.raises(ValueError, match=r"--ek/--environment-kwarg key cannot be empty"):+ with pytest.raises(ValueError, match=r"Invalid --environment-kwarg: keys must be valid Python keyword names"):parse_environment_kwargs(("=value",))- with pytest.raises(ValueError, match=r"--ek/--environment-kwarg key cannot be empty"):+ with pytest.raises(ValueError, match=r"Invalid --environment-kwarg: keys must be valid Python keyword names"):parse_environment_kwargs((" =value",))@@ -665,7 +669,7 @@ def test_parse_environment_kwargs_empty_key():)def test_build_harbor_run_command_rejects_camel_case_sensitive_keys(sensitive_camel: str) -> None:"""Reject camelCase credential keys in environment kwargs."""- with pytest.raises(ValueError, match=r"Sensitive key or value detected in environment_kwargs"):+ with pytest.raises(ValueError, match=r"Sensitive key or value detected"):build_harbor_run_command(dataset_path="/tmp/dataset",agent="claude-code",@@ -686,15 +690,11 @@ def test_build_harbor_run_command_rejects_camel_case_sensitive_keys(sensitive_ca],)def test_build_harbor_run_command_permits_camel_case_benign_token_keys(benign_camel: str) -> None:- """Permit camelCase rate-limiting and token count keys."""- cmd = build_harbor_run_command(- dataset_path="/tmp/dataset",- agent="claude-code",- job_name="gke-job",- env_mode="gke",- environment_kwargs={**COMPLETE_GKE_KWARGS, benign_camel: "100"},- )- assert f"--ek={benign_camel}=100" in cmd or f"{benign_camel}=100" in " ".join(cmd)+ """Permit camelCase rate-limiting and token count keys in environment_kwargs secret validation."""+ from skillevaluator.tier3.evals_config import validate_environment_kwargs++ validated = validate_environment_kwargs({**COMPLETE_GKE_KWARGS, benign_camel: "100"}, env_mode="gke")+ assert validated[benign_camel] == "100"@pytest.mark.parametrize(@@ -708,13 +708,13 @@ def test_build_harbor_run_command_permits_camel_case_benign_token_keys(benign_ca)def test_build_harbor_run_command_rejects_aws_and_private_key_values(secret_val: str) -> None:"""Reject AWS access keys and PEM private keys in environment kwarg values."""- with pytest.raises(ValueError, match=r"Sensitive key or value detected in environment_kwargs"):+ with pytest.raises(ValueError, match=r"Sensitive key or value detected"):build_harbor_run_command(dataset_path="/tmp/dataset",agent="claude-code",job_name="gke-job",env_mode="gke",- environment_kwargs={**COMPLETE_GKE_KWARGS, "custom_setting": secret_val},+ environment_kwargs={**COMPLETE_GKE_KWARGS, "cluster_name": secret_val},)@@ -739,7 +739,7 @@ def test_resolve_vertex_model_id_lowercases_unmapped_models() -> None:)def test_build_harbor_run_command_rejects_credential_keys(sensitive_key: str) -> None:"""Reject credential keys in environment kwargs."""- with pytest.raises(ValueError, match=r"Sensitive key or value detected in environment_kwargs"):+ with pytest.raises(ValueError, match=r"Sensitive key or value detected"):build_harbor_run_command(dataset_path="/tmp/dataset",agent="claude-code",@@ -765,13 +765,13 @@ def test_build_harbor_run_command_rejects_credential_keys(sensitive_key: str) ->)def test_build_harbor_run_command_rejects_token_values(sensitive_value: str) -> None:"""Reject token and credential values in environment kwargs."""- with pytest.raises(ValueError, match=r"Sensitive key or value detected in environment_kwargs"):+ with pytest.raises(ValueError, match=r"Sensitive key or value detected"):build_harbor_run_command(dataset_path="/tmp/dataset",agent="claude-code",job_name="gke-job",env_mode="gke",- environment_kwargs={**COMPLETE_GKE_KWARGS, "custom_key": sensitive_value},+ environment_kwargs={**COMPLETE_GKE_KWARGS, "cluster_name": sensitive_value},)@@ -788,14 +788,19 @@ def test_build_harbor_run_command_rejects_token_values(sensitive_value: str) ->)def test_build_harbor_run_command_permits_valid_rate_and_token_flags(flag_key: str, flag_val: str) -> None:"""Permit valid rate limit, token budget, and cluster configuration flags."""- cmd = build_harbor_run_command(- dataset_path="/tmp/dataset",- agent="claude-code",- job_name="gke-job",- env_mode="gke",- environment_kwargs={**COMPLETE_GKE_KWARGS, flag_key: flag_val},- )- assert f"{flag_key}={flag_val}" in cmd+ from skillevaluator.tier3.evals_config import encode_environment_kwarg, validate_environment_kwargs++ validated = validate_environment_kwargs({**COMPLETE_GKE_KWARGS, flag_key: flag_val}, env_mode="gke")+ assert validated[flag_key] == flag_val+ if flag_key == "cluster_name":+ cmd = build_harbor_run_command(+ dataset_path="/tmp/dataset",+ agent="claude-code",+ job_name="gke-job",+ env_mode="gke",+ environment_kwargs={**COMPLETE_GKE_KWARGS, flag_key: flag_val},+ )+ assert encode_environment_kwarg(flag_key, flag_val) in cmd@pytest.mark.parametrize(@@ -815,7 +820,10 @@ def test_build_harbor_run_command_permits_valid_rate_and_token_flags(flag_key: s)def test_parse_environment_kwargs_rejects_credential_keys(sensitive_key: str) -> None:"""Verify parse_environment_kwargs rejects sensitive credential keys with ValueError."""- with pytest.raises(ValueError, match=r"Sensitive key or value detected in environment_kwargs"):+ with pytest.raises(+ ValueError,+ match=r"Invalid --environment-kwarg: (Sensitive key or value detected|keys must be valid Python keyword names)",+ ):parse_environment_kwargs((f"{sensitive_key}=secret123",))@@ -837,7 +845,7 @@ def test_parse_environment_kwargs_rejects_credential_keys(sensitive_key: str) ->def test_parse_environment_kwargs_permits_benign_token_keys(benign_token_key: str) -> None:"""Verify parse_environment_kwargs permits benign token count, rate, and tokenizer keys."""parsed = parse_environment_kwargs((f"{benign_token_key}=1000",))- assert parsed == {benign_token_key: "1000"}+ assert parsed == {benign_token_key: 1000}def test_resolve_single_kubeconfig_multi_path(tmp_path: Path) -> None:@@ -1247,7 +1255,7 @@ def _write_mcp_skill_fixture(def test_build_harbor_run_command_gke_uses_secure_environment_import_path() -> None:- """Route GKE Harbor commands through SECURE_GKE_ENV_IMPORT_PATH instead of --env gke."""+ """Route GKE Harbor commands through SECURE_GKE_ENV_IMPORT_PATH via Harbor 0.24.0 unified --env flag."""from skillevaluator.tier3.harbor.gke_environment import SECURE_GKE_ENV_IMPORT_PATHcmd = build_harbor_run_command(@@ -1258,8 +1266,9 @@ def test_build_harbor_run_command_gke_uses_secure_environment_import_path() -> Nmodel="gpt-5.5",environment_kwargs=COMPLETE_GKE_KWARGS,)- assert "--environment-import-path" in cmd- assert cmd[cmd.index("--environment-import-path") + 1] == SECURE_GKE_ENV_IMPORT_PATH+ assert "--env" in cmd+ assert cmd[cmd.index("--env") + 1] == SECURE_GKE_ENV_IMPORT_PATH+ assert "--environment-import-path" not in cmddef test_check_prerequisites_gke_live_cluster_rejects_bound_service_account_without_opt_in(@@ -1635,3 +1644,62 @@ def test_mcp_placeholder_stripping_rejects_operator_refs_literals_and_malformed_with pytest.raises(ValueError, match=expected_error_match):validate_mcp_server_declarations([bad_declaration])+++def test_resolve_runtime_env_allows_approved_mcp_secret_and_blocks_operator_or_unapproved(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """Allow SKILLEVALUATOR_ALLOWED_MCP_SECRETS in harbor.runtime_env while blocking unapproved or operator secrets."""+ from skillevaluator.tier3.harbor.runner import _resolve_runtime_env++ monkeypatch.setenv("DEVELOPERKNOWLEDGE_API_KEY", "dk-test-value")+ monkeypatch.delenv("SKILLEVALUATOR_ALLOWED_MCP_SECRETS", raising=False)++ # Unapproved sensitive key is blocked+ resolved, errors = _resolve_runtime_env(+ {"DEVELOPERKNOWLEDGE_API_KEY": "${DEVELOPERKNOWLEDGE_API_KEY}"},+ env_mode="gke",+ )+ assert resolved == {}+ assert any("controls the host process and is not allowed" in err for err in errors)++ # Operator-approved dedicated MCP secret is allowed when referenced via ${VAR}+ monkeypatch.setenv("SKILLEVALUATOR_ALLOWED_MCP_HOSTS", "developerknowledge.googleapis.com")+ monkeypatch.setenv(+ "SKILLEVALUATOR_ALLOWED_MCP_SECRETS",+ "DEVELOPERKNOWLEDGE_API_KEY,ANTHROPIC_API_KEY",+ )+ resolved, errors = _resolve_runtime_env(+ {"DEVELOPERKNOWLEDGE_API_KEY": "${DEVELOPERKNOWLEDGE_API_KEY}"},+ env_mode="gke",+ )+ assert errors == []+ assert resolved == {"DEVELOPERKNOWLEDGE_API_KEY": "dk-test-value"}+ subprocess_env = _harbor_subprocess_environment(+ env_mode="gke",+ provider=_provider(),+ configured_runtime_env=resolved,+ provider_env={},+ agent="claude-code",+ agent_model="claude-sonnet-5",+ )+ assert subprocess_env.get("SKILLEVALUATOR_ALLOWED_MCP_HOSTS") == "developerknowledge.googleapis.com"+ assert subprocess_env.get("SKILLEVALUATOR_ALLOWED_MCP_SECRETS") == "DEVELOPERKNOWLEDGE_API_KEY,ANTHROPIC_API_KEY"+ assert subprocess_env.get("DEVELOPERKNOWLEDGE_API_KEY") == "dk-test-value"++ # Literal secret for an approved MCP secret key is rejected+ resolved, errors = _resolve_runtime_env(+ {"DEVELOPERKNOWLEDGE_API_KEY": "literal-dk-key"},+ env_mode="gke",+ )+ assert resolved == {}+ assert any("must reference an environment variable rather than a literal secret" in err for err in errors)++ # Operator-owned LLM secret remains blocked even when listed in SKILLEVALUATOR_ALLOWED_MCP_SECRETS+ monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-test")+ resolved, errors = _resolve_runtime_env(+ {"ANTHROPIC_API_KEY": "${ANTHROPIC_API_KEY}"},+ env_mode="gke",+ )+ assert resolved == {}+ assert any("controls the host process and is not allowed" in err for err in errors)diff --git a/tests/test_harbor_runner_environment.py b/tests/test_harbor_runner_environment.pyindex 8f922795..627dddb6 100644--- a/tests/test_harbor_runner_environment.py+++ b/tests/test_harbor_runner_environment.py@@ -1753,6 +1753,9 @@ def test_harbor_backend_environment_allowlist_covers_every_native_022_mode() ->"GOOGLE_CLOUD_PROJECT","GOOGLE_CLOUD_QUOTA_PROJECT","KUBECONFIG",+ "SKILLEVALUATOR_GKE_ALLOW_WORKLOAD_IDENTITY",+ "SKILLEVALUATOR_GKE_AUTOPILOT",+ "SKILLEVALUATOR_GKE_METADATA_PROBE_IMAGE",},"ack": {"COMPOSE_ANSI",@@ -2354,10 +2357,10 @@ def test_run_harbor_reissues_adc_token_for_bounded_jobs(monkeypatch: pytest.Monknonlocal captured_timeoutcaptured_command.extend(command)captured_env.update(kwargs.get("env") or {})- captured_timeout = kwargs.get("timeout")- return subprocess.CompletedProcess(command, returncode=0, stdout="", stderr="")+ captured_timeout = kwargs.get("timeout_seconds")+ return runner._BoundedHarborProcessResult(returncode=0, output_tail="", output_exceeded=False)- monkeypatch.setattr(runner.subprocess, "run", mock_run)+ monkeypatch.setattr(runner, "_run_bounded_harbor_process", mock_run)monkeypatch.setattr(runner, "_validate_harbor_job_result", lambda *_args, **_kwargs: (True, "success"))monkeypatch.setattr(runner, "_get_google_access_token", lambda **_kw: "fresh-reissued-token")@@ -2407,10 +2410,10 @@ def test_run_harbor_preserves_explicit_vertex_credential(def mock_run(command, *args, **kwargs):nonlocal captured_timeoutcaptured_env.update(kwargs.get("env") or {})- captured_timeout = kwargs.get("timeout")- return subprocess.CompletedProcess(command, returncode=0, stdout="", stderr="")+ captured_timeout = kwargs.get("timeout_seconds")+ return runner._BoundedHarborProcessResult(returncode=0, output_tail="", output_exceeded=False)- monkeypatch.setattr(runner.subprocess, "run", mock_run)+ monkeypatch.setattr(runner, "_run_bounded_harbor_process", mock_run)monkeypatch.setattr(runner, "_validate_harbor_job_result", lambda *_args, **_kwargs: (True, "success"))monkeypatch.setattr("skillevaluator.provider_config._get_google_access_token", lambda **_kw: "ambient-host-adc")@@ -2485,9 +2488,9 @@ def test_run_harbor_enforces_adc_job_timeout_and_stages_credential_source(assert "SKILL_EVAL_LLM_CREDENTIAL_SOURCE" in _verifier_env_vars(provider_env)def timeout_run(command, *args, **kwargs):- raise subprocess.TimeoutExpired(cmd=command, timeout=kwargs.get("timeout", 3300))+ raise runner._HarborRunTimeoutError(f"timed out after {kwargs.get('timeout_seconds', 3300)}s")- monkeypatch.setattr(runner.subprocess, "run", timeout_run)+ monkeypatch.setattr(runner, "_run_bounded_harbor_process", timeout_run)adc_env = {"OPENAI_API_KEY": "initial-adc-token","OPENAI_BASE_URL": base_url,@@ -2529,10 +2532,10 @@ def test_run_harbor_caps_configured_adc_timeout_and_rejects_near_expiry(captured_timeouts: list[float | None] = []def fake_run(command, *args, **kwargs):- captured_timeouts.append(kwargs.get("timeout"))- return subprocess.CompletedProcess(command, 0, stdout="", stderr="")+ captured_timeouts.append(kwargs.get("timeout_seconds"))+ return runner._BoundedHarborProcessResult(returncode=0, output_tail="", output_exceeded=False)- monkeypatch.setattr(runner.subprocess, "run", fake_run)+ monkeypatch.setattr(runner, "_run_bounded_harbor_process", fake_run)monkeypatch.setattr(runner, "_validate_harbor_job_result", lambda *_args, **_kwargs: (True, "success"))# 1. 600s token + SKILLEVALUATOR_VERTEX_ADC_JOB_TIMEOUT_SEC=5000 -> capped at 540s@@ -2680,7 +2683,7 @@ def test_harbor_subprocess_environment_forwards_gke_probe_and_autopilot_vars(tmp_path: Path,) -> None:"""Forward SKILLEVALUATOR_GKE_METADATA_PROBE_IMAGE and SKILLEVALUATOR_GKE_AUTOPILOT in gke mode and allow --ek autopilot."""- from skillevaluator.tier3.evals_config import _GKE_INFRASTRUCTURE_KWARGS+ from skillevaluator.tier3.evals_config import _GKE_INFRASTRUCTURE_KWARGS, encode_environment_kwargmonkeypatch.setenv("SKILLEVALUATOR_GKE_METADATA_PROBE_IMAGE", "us-docker.pkg.dev/my-proj/repo/python:3.12-alpine")monkeypatch.setenv("SKILLEVALUATOR_GKE_AUTOPILOT", "true")@@ -2718,4 +2721,4 @@ def test_harbor_subprocess_environment_forwards_gke_probe_and_autopilot_vars(env_mode="gke",environment_kwargs={"autopilot": "true", "cluster_name": "c"},)- assert "autopilot=true" in cmd+ assert encode_environment_kwarg("autopilot", "true") in cmddiff --git a/tests/test_oss_packaging.py b/tests/test_oss_packaging.pyindex b0214d3d..770a3dcf 100644--- a/tests/test_oss_packaging.py+++ b/tests/test_oss_packaging.py@@ -148,7 +148,7 @@ def test_harbor_dependency_contract_keeps_base_install_isolated() -> None:assert base_names.isdisjoint({"harbor", "litellm"})assert len(harbor_requirements) == 1- assert not harbor_requirements[0].extras+ assert harbor_requirements[0].extras == {"gke"}assert HARBOR_VERSION == "0.24.0"assert str(harbor_requirements[0].specifier) == f"=={HARBOR_VERSION}"assert len(litellm_requirements) == 1@@ -441,6 +441,7 @@ def test_tier3_direct_dependencies_have_complete_license_notices() -> None:expected_notices = {"harbor": "Harbor (Apache-2.0)","mcp": "MCP (MIT)",+ "oauthlib": "OAuthLib (BSD-3-Clause)","pyjwt": "PyJWT (MIT)","python-dotenv": "python-dotenv (BSD-3-Clause)",}@@ -482,6 +483,7 @@ def test_release_lock_enforces_nspect_remediation_floors_without_removed_telemetassert "mcp>=1.28.1,<2" in tier3assert "pyjwt[crypto]>=2.13.0" in tier3+ assert "oauthlib>=4.0.0" in tier3assert "telemetry" not in extrasassert all("protobuf" not in requirement.lower() for requirements in extras.values() for requirement in requirements@@ -491,6 +493,7 @@ def test_release_lock_enforces_nspect_remediation_floors_without_removed_telemet)assert all(version >= Version("1.28.1") for version in all_lock_versions["mcp"])assert all(version >= Version("2.13.0") for version in all_lock_versions["pyjwt"])+ assert all(version >= Version("4.0.0") for version in all_lock_versions["oauthlib"])assert "protobuf" not in all_lock_versionsassert not any(name.startswith("opentelemetry") for name in all_lock_versions)diff --git a/tests/test_provider_agent_routing.py b/tests/test_provider_agent_routing.pyindex 7a88ed2d..d669f526 100644--- a/tests/test_provider_agent_routing.py+++ b/tests/test_provider_agent_routing.py@@ -7,7 +7,12 @@ from __future__ import annotationsimport pytest-from skillevaluator.provider_config import ProviderConfig+from skillevaluator.provider_config import (+ CHAT_DEFAULT_OPENAI,+ OPENCODE_OPENAI_DEFAULT_MODEL,+ ProviderConfig,+ resolve_llm_provider,+)from skillevaluator.tier3.harbor.runner import (_OPERATOR_OWNED_AGENT_ENV,_agent_provider_config,diff --git a/tests/test_secure_docker_environment.py b/tests/test_secure_docker_environment.pyindex 4078a90f..8b9e2688 100644--- a/tests/test_secure_docker_environment.py+++ b/tests/test_secure_docker_environment.py@@ -607,6 +607,7 @@ def test_secure_docker_exec_refreshes_adc_token_per_trial_and_verifier_exec(from skillevaluator.tier3.harbor import secure_docker_environmentenvironment = object.__new__(secure_docker_environment.SkillEvaluatorSecureDockerEnvironment)+ _initialize_harbor_context(environment)environment._persistent_env = {}environment.default_user = "1000"environment.task_env_config = SimpleNamespace(workdir="/workspace")@@ -619,14 +620,13 @@ def test_secure_docker_exec_refreshes_adc_token_per_trial_and_verifier_exec(command: list[str],check: bool = True,timeout_sec: int | None = None,- *,- stdin_bytes: bytes | None = None,- redact_values: set[str] | None = None,- stop_main_on_interrupt: bool = False,+ stdin_data: bytes | None = None,+ on_output: object | None = None,+ **_kwargs,):- del check, timeout_sec, redact_values, stop_main_on_interrupt- if stdin_bytes is not None:- handoff_payloads.append(stdin_bytes.decode("utf-8"))+ del check, timeout_sec, on_output+ if stdin_data is not None:+ handoff_payloads.append(stdin_data.decode("utf-8"))docker_commands.append(command)return SimpleNamespace(stdout="ok", stderr=None, return_code=0)@@ -789,12 +789,12 @@ def test_short_expiry_adc_docker_handoff_and_failed_refresh_fails_closed(captured_timeout: float | None = Nonedef fake_harbor_run(command, *args, **kwargs):- del args+ del command, argsnonlocal captured_timeout- captured_timeout = kwargs.get("timeout")- return subprocess.CompletedProcess(command, returncode=0, stdout="", stderr="")+ captured_timeout = kwargs.get("timeout_seconds")+ return runner._BoundedHarborProcessResult(returncode=0, output_tail="", output_exceeded=False)- monkeypatch.setattr(runner.subprocess, "run", fake_harbor_run)+ monkeypatch.setattr(runner, "_run_bounded_harbor_process", fake_harbor_run)monkeypatch.setattr(runner, "_validate_harbor_job_result", lambda *_a, **_kw: (True, "ok"))ok, _detail = runner._run_harbor(diff --git a/tests/test_tier3_progress.py b/tests/test_tier3_progress.pyindex 96947311..54e71fca 100644--- a/tests/test_tier3_progress.py+++ b/tests/test_tier3_progress.py@@ -653,6 +653,15 @@ def test_redact_progress_detail_preserves_hyphenated_identifiers_while_redactingassert progress.redact_progress_detail(raw_text) == expected_text+def test_progress_redaction_scales_linearly_for_large_non_uri_text() -> None:+ progress = _progress_module()+ payload = "x" * (1024 * 1024)++ rendered = progress.redact_progress_detail(payload)++ assert rendered == payload++def test_plain_reporter_redacts_secrets_from_plan_values() -> None:progress = _progress_module()output = io.StringIO()diff --git a/tests/test_tier3_public_runtime.py b/tests/test_tier3_public_runtime.pyindex dd9b2f79..25ffa657 100644--- a/tests/test_tier3_public_runtime.py+++ b/tests/test_tier3_public_runtime.py@@ -697,7 +697,7 @@ def test_custom_agent_import_path_preserves_native_cloud_environment(env_mode: sdef test_custom_agent_import_path_is_rejected_for_native_cloud() -> None:- with pytest.raises(ValueError, match="agent_import_path is supported only with --env docker, local, or gke"):+ with pytest.raises(ValueError, match="agent_import_path is supported only with --env docker, kata, or local"):build_harbor_run_command(dataset_path="/tmp/dataset",agent="codex",@@ -2042,3 +2042,910 @@ def test_verifier_401_does_not_retry_non_vertex_endpoints(monkeypatch: pytest.Moassert error is not Noneassert "401" in error or "Unauthorized" in errorassert refresh_called is False+++ ("has_skill", "arm_suffix", "expected_task_name"),+ [+ (True, "", "nvidia/skillevaluator-case-001"),+ (True, "-with-skill", "nvidia/skillevaluator-case-001-with-skill"),+ (False, "-without-skill", "nvidia/skillevaluator-case-001-without-skill"),+ ],+)+def test_write_task_toml_dual_arm_suffix(+ tmp_path: Path,+ has_skill: bool,+ arm_suffix: str,+ expected_task_name: str,+) -> None:+ """Verify that arm suffix is appended only when provided for dual-arm runs."""+ case_dir = tmp_path / "case"+ case_dir.mkdir()+ _write_task_toml(+ case_dir,+ {"id": "case-001", "expected_skill": "demo"},+ has_skill=has_skill,+ arm_suffix=arm_suffix,+ )+ task = tomllib.loads((case_dir / "task.toml").read_text(encoding="utf-8"))+ assert task["task"]["name"] == expected_task_name+++def test_write_task_toml_type_safety(tmp_path: Path) -> None:+ """Verify that _write_task_toml rejects non-string arm_suffix values."""+ case_dir = tmp_path / "case"+ case_dir.mkdir()++ with pytest.raises(TypeError, match="arm_suffix must be a string"):+ _write_task_toml(case_dir, {"id": "case-001"}, has_skill=True, arm_suffix=123) # type: ignore[arg-type]+++ ("value", "expected"),+ [+ ("case-001-with-skill", "case-001"),+ ("case-001-without-skill", "case-001"),+ ("case-001-with", "case-001-with"),+ ("case-001-without", "case-001-without"),+ ("case-001", "case-001"),+ ],+)+def test_strip_arm_suffix(value: str, expected: str) -> None:+ """Verify _strip_arm_suffix trims exact dual-arm suffixes."""+ from skillevaluator.tier3.harbor.collector import _strip_arm_suffix++ assert _strip_arm_suffix(value) == expected+++ ("value", "expected"),+ [+ ("case-001-with-skill-attempt1", "case-001"),+ ("case-001-without-skill_attempt2", "case-001"),+ ("case-001-attempt1-with-skill", "case-001"),+ ("case-001_attempt2-without-skill", "case-001"),+ ("case-001-with-skill", "case-001"),+ ("case-001-without-skill", "case-001"),+ ("case-001-attempt1", "case-001"),+ ("case-001", "case-001"),+ ],+)+def test_strip_arm_and_attempt_suffixes(value: str, expected: str) -> None:+ """Verify _strip_arm_and_attempt_suffixes removes both suffixes regardless of ordering."""+ from skillevaluator.tier3.harbor.collector import _strip_arm_and_attempt_suffixes++ assert _strip_arm_and_attempt_suffixes(value) == expected+++ ("raw_input", "expected_ids", "expected_output"),+ [+ # Bare case IDs without expected_case_ids preserve authored prefixes/suffixes+ ("case-001", None, "case-001"),+ ("skillevaluator-case-001", None, "skillevaluator-case-001"),+ ("skillevaluator-case-001", {"case-001", "case-002"}, "case-001"),+ # Without expected_case_ids, authored -with-skill / -without-skill are preserved+ ("case-001-with-skill", None, "case-001-with-skill"),+ ("case-001-without-skill", None, "case-001-without-skill"),+ ("skillevaluator-case-001-with-skill", None, "skillevaluator-case-001-with-skill"),+ ("skillevaluator-case-001-without-skill", None, "skillevaluator-case-001-without-skill"),+ # Dual-arm suffixes with expected_case_ids matching+ ("case-001-with-skill", {"case-001", "case-002"}, "case-001"),+ ("case-001-without-skill", {"case-001", "case-002"}, "case-001"),+ ("skillevaluator-case-001-with-skill", {"case-001", "case-002"}, "case-001"),+ ("skillevaluator-case-001-without-skill", {"case-001", "case-002"}, "case-001"),+ # Namespaced task names (e.g., nvidia/...)+ ("nvidia/skillevaluator-case-001", None, "skillevaluator-case-001"),+ ("nvidia/skillevaluator-case-001", {"case-001"}, "case-001"),+ ("nvidia/skillevaluator-case-001-with-skill", None, "skillevaluator-case-001-with-skill"),+ ("nvidia/skillevaluator-case-001-with-skill", {"case-001"}, "case-001"),+ ("nvidia/skillevaluator-case-001-without-skill", None, "skillevaluator-case-001-without-skill"),+ ("nvidia/skillevaluator-case-001-without-skill", {"case-001"}, "case-001"),+ ("custom/repo/skillevaluator-case-002-with-skill", {"case-002"}, "case-002"),+ # Attempt suffixes combined with dual-arm suffixes in both orderings+ ("case-001-with-skill-attempt1", None, "case-001-with-skill"),+ ("case-001-with-skill-attempt1", {"case-001"}, "case-001"),+ ("case-001-without-skill_attempt2", None, "case-001-without-skill"),+ ("case-001-without-skill_attempt2", {"case-001"}, "case-001"),+ ("case-001-attempt1-with-skill", {"case-001"}, "case-001"),+ ("case-001_attempt2-without-skill", {"case-001"}, "case-001"),+ ("skillevaluator-case-001-with-skill-attempt1", None, "skillevaluator-case-001-with-skill"),+ ("skillevaluator-case-001-with-skill-attempt1", {"case-001"}, "case-001"),+ ("skillevaluator-case-001-attempt1-with-skill", {"case-001"}, "case-001"),+ ("nvidia/skillevaluator-case-001-with-skill-attempt3", {"case-001"}, "case-001"),+ ("nvidia/skillevaluator-case-001-attempt3-with-skill", {"case-001"}, "case-001"),+ ("nvidia/skillevaluator-case-001-without-skill_attempt4", None, "skillevaluator-case-001-without-skill"),+ # Legitimate case IDs ending with -with or -without preserved when in expected_case_ids+ ("case-with", {"case-with"}, "case-with"),+ ("case-without", {"case-without"}, "case-without"),+ ("nvidia/case-with", {"case-with"}, "case-with"),+ ("nvidia/case-without", {"case-without"}, "case-without"),+ ("skillevaluator-case-with", {"case-with"}, "case-with"),+ ("skillevaluator-case-without", {"case-without"}, "case-without"),+ ("case-with-attempt1", {"case-with"}, "case-with"),+ ("case-without-attempt2", {"case-without"}, "case-without"),+ # Dual-arm runs on legitimate -with / -without IDs+ ("case-with-with-skill", {"case-with"}, "case-with"),+ ("case-without-without-skill", {"case-without"}, "case-without"),+ ("skillevaluator-case-with-with-skill", {"case-with"}, "case-with"),+ ("skillevaluator-case-without-without-skill", {"case-without"}, "case-without"),+ ("nvidia/skillevaluator-case-with-with-skill", {"case-with"}, "case-with"),+ ("nvidia/skillevaluator-case-without-without-skill", {"case-without"}, "case-without"),+ ("case-with-with-skill-attempt1", {"case-with"}, "case-with"),+ ("case-with-attempt1-with-skill", {"case-with"}, "case-with"),+ ("case-without-without-skill_attempt2", {"case-without"}, "case-without"),+ ("case-without_attempt2-without-skill", {"case-without"}, "case-without"),+ # Case IDs that retain the skillevaluator- prefix in expected_case_ids+ ("skillevaluator-case-001-with-skill", {"skillevaluator-case-001"}, "skillevaluator-case-001"),+ ("skillevaluator-case-001-without-skill", {"skillevaluator-case-001"}, "skillevaluator-case-001"),+ ("nvidia/skillevaluator-case-001-with-skill", {"skillevaluator-case-001"}, "skillevaluator-case-001"),+ ("skillevaluator-case-001-with-skill-attempt1", {"skillevaluator-case-001"}, "skillevaluator-case-001"),+ ("skillevaluator-case-001-attempt1-with-skill", {"skillevaluator-case-001"}, "skillevaluator-case-001"),+ # Empty and blank strings+ ("", None, ""),+ (" ", None, ""),+ ("", {"case-001"}, ""),+ # Double arm suffixes must strip at most one arm suffix (never collapse authored -with-skill onto base case)+ ("case-1-with-skill-with-skill", {"case-1"}, "case-1-with-skill-with-skill"),+ ("case-1-with-skill-with-skill", {"case-1", "case-1-with-skill"}, "case-1-with-skill"),+ ("case-1-with-skill-with-skill_attempt2", {"case-1", "case-1-with-skill"}, "case-1-with-skill"),+ ("case-1-with-skill_attempt2-with-skill", {"case-1", "case-1-with-skill"}, "case-1-with-skill"),+ ],+)+def test_canonical_case_id_arm_stripping(+ raw_input: str,+ expected_ids: set[str] | None,+ expected_output: str,+) -> None:+ """Verify _canonical_case_id normalizes task identifiers across naming and attempt variants."""+ from skillevaluator.tier3.harbor.collector import _canonical_case_id++ assert _canonical_case_id(raw_input, expected_ids) == expected_output+++ ("arm_suffix", "expected_task_name"),+ [+ ("-with-skill", "nvidia/case-001-with-skill"),+ ("-without-skill", "nvidia/case-001-without-skill"),+ ],+)+def test_stage_native_harbor_tasks_dual_arm_suffix(+ tmp_path: Path,+ arm_suffix: str,+ expected_task_name: str,+) -> None:+ """Verify that stage_native_harbor_tasks appends arm suffix to native task.toml name."""+ skill_dir = tmp_path / "target-skill"+ skill_dir.mkdir()+ (skill_dir / "SKILL.md").write_text("# Target Skill\n", encoding="utf-8")+ evals_dir = skill_dir / "evals" / "harbor"+ task_dir = evals_dir / "case-001"+ task_dir.mkdir(parents=True)+ (task_dir / "instruction.md").write_text("Instruction\n", encoding="utf-8")+ (task_dir / "task.toml").write_text(+ 'schema_version = "1.3"\n\n[task]\nname = "nvidia/case-001"\n\n[environment]\n',+ encoding="utf-8",+ )+ tests_dir = task_dir / "tests"+ tests_dir.mkdir()+ (tests_dir / "test.sh").write_text("#!/bin/bash\nexit 0\n", encoding="utf-8")++ out_dir = tmp_path / f"out{arm_suffix}"+ staged = stage_native_harbor_tasks(+ skill_dir,+ out_dir,+ grading_mode="custom_only",+ arm_suffix=arm_suffix,+ )[0]+ task = tomllib.loads((staged / "task.toml").read_text(encoding="utf-8"))+ assert task["task"]["name"] == expected_task_name+++ ("task_toml_body", "expected_task_name", "expected_metadata_name"),+ [+ (+ 'schema_version = "1.3"\n\n[task]\nname = """nvidia/case-001"""\n\n[environment]\n',+ "nvidia/case-001-with-skill",+ None,+ ),+ (+ "schema_version = \"1.3\"\n\n[task]\nname = '''nvidia/case-001'''\n\n[environment]\n",+ "nvidia/case-001-with-skill",+ None,+ ),+ (+ 'schema_version = "1.3"\n\n[metadata]\nname = "keep-metadata-name"\n\n'+ '[task] # task identity\nauthors = [{ name = "Alice" }]\nname = "nvidia/case-001" # inline comment\n\n[environment]\n',+ "nvidia/case-001-with-skill",+ "keep-metadata-name",+ ),+ (+ 'schema_version = "1.3"\n"task.name" = "root-quoted-dot-key"\n\n'+ '[task.subtable]\nname = "subtable-name"\n\n'+ '[task]\nkeywords = [\n "alpha",\n "beta",\n]\n'+ 'description = """\n[metadata]\n[task]\nname = "fake-inside-multiline"\n"""\n'+ 'name = "nvidia/case-001"\n\n[environment]\n',+ "nvidia/case-001-with-skill",+ None,+ ),+ (+ 'schema_version = "1.3"\nmetadata = { name = "inline-metadata-name" }\n\n'+ '[task]\nname = "nvidia/case-001"\n\n[environment]\n',+ "nvidia/case-001-with-skill",+ "inline-metadata-name",+ ),+ (+ 'schema_version = "1.3"\nmetadata = {}\n\n'+ 'task = { authors = [{ name = "Alice" }], name = "nvidia/case-001", tags = ["a", "b"] }\n\n[environment]\n',+ "nvidia/case-001-with-skill",+ None,+ ),+ (+ 'schema_version = "1.3"\nmetadata.name = "dotted-metadata-name"\n'+ 'task = { subtable = { name = "subtable-name" }, name = "nvidia/case-001" }\n\n[environment]\n',+ "nvidia/case-001-with-skill",+ "dotted-metadata-name",+ ),+ (+ 'schema_version = "1.3"\n["\\u006detadata"]\nname = "escaped-metadata-name"\n\n'+ '["\\u0074ask"]\n"\\u006eame" = "nvidia/case-001"\n\n[environment]\n',+ "nvidia/case-001-with-skill",+ "escaped-metadata-name",+ ),+ (+ 'schema_version = "1.3"\n"\\u0074ask"."\\u006eame" = "nvidia/case-001"\n\n[environment]\n',+ "nvidia/case-001-with-skill",+ None,+ ),+ (+ 'schema_version = "1.3"\ntask.name = "nvidia/case-001"\n\n[environment]\n',+ "nvidia/case-001-with-skill",+ None,+ ),+ (+ 'schema_version = "1.3"\n\n[metadata.extra]\nfoo = "bar"\n\n[task]\nname = "nvidia/case-001"\n\n[environment]\n',+ "nvidia/case-001-with-skill",+ None,+ ),+ (+ 'schema_version = "1.3"\n\n[[metadata.items]]\nname = "item1"\n\n[task]\nname = "nvidia/case-001"\n\n[environment]\n',+ "nvidia/case-001-with-skill",+ None,+ ),+ ],+)+def test_stage_native_harbor_tasks_structural_toml_variants(+ tmp_path: Path,+ task_toml_body: str,+ expected_task_name: str,+ expected_metadata_name: str | None,+) -> None:+ """Verify structural [task].name update preserves valid TOML across inline tables, escaped keys, and [metadata]."""+ skill_dir = tmp_path / "target-skill"+ skill_dir.mkdir()+ (skill_dir / "SKILL.md").write_text("# Target Skill\n", encoding="utf-8")+ task_dir = skill_dir / "evals" / "harbor" / "case-001"+ task_dir.mkdir(parents=True)+ (task_dir / "instruction.md").write_text("Instruction\n", encoding="utf-8")+ (task_dir / "task.toml").write_text(task_toml_body, encoding="utf-8")+ tests_dir = task_dir / "tests"+ tests_dir.mkdir()+ (tests_dir / "test.sh").write_text("#!/bin/bash\nexit 0\n", encoding="utf-8")++ staged = stage_native_harbor_tasks(+ skill_dir,+ tmp_path / "out",+ grading_mode="custom_only",+ arm_suffix="-with-skill",+ )[0]+ parsed = tomllib.loads((staged / "task.toml").read_text(encoding="utf-8"))+ assert parsed["task"]["name"] == expected_task_name+ assert parsed["metadata"]["entry_id"] == "case-001"+ if expected_metadata_name is not None:+ assert parsed["metadata"]["name"] == expected_metadata_name+ if "task.name" in parsed:+ assert parsed["task.name"] == "root-quoted-dot-key"+ assert parsed["task"]["subtable"]["name"] == "subtable-name"+++def test_append_native_task_name_suffix_sad_paths(tmp_path: Path) -> None:+ """Verify _append_native_task_name_suffix safely handles missing/corrupt/non-string task names and fails closed when [metadata].entry_id cannot be written."""+ from skillevaluator.tier3.harbor.adapter import _append_native_task_name_suffix++ missing_dir = tmp_path / "missing"+ missing_dir.mkdir()+ _append_native_task_name_suffix(missing_dir, "-with-skill")+ assert not (missing_dir / "task.toml").exists()++ for idx, body in enumerate(+ (+ "[task\nname = 'unclosed'\n",+ 'schema_version = "1.3"\n[environment]\n',+ 'schema_version = "1.3"\n[task]\nname = 123\n',+ )+ ):+ case_dir = tmp_path / f"sad-{idx}"+ case_dir.mkdir()+ (case_dir / "task.toml").write_text(body, encoding="utf-8")+ _append_native_task_name_suffix(case_dir, "-with-skill")+ assert (case_dir / "task.toml").read_text(encoding="utf-8") == body++ preserved_dir = tmp_path / "preserved-entry"+ preserved_dir.mkdir()+ (preserved_dir / "task.toml").write_text(+ '[metadata]\nentry_id = "authored-entry"\n\n[task]\nname = "nvidia/case-1"\n',+ encoding="utf-8",+ )+ _append_native_task_name_suffix(preserved_dir, "")+ assert tomllib.loads((preserved_dir / "task.toml").read_text(encoding="utf-8"))["task"]["name"] == "nvidia/case-1"+ _append_native_task_name_suffix(preserved_dir, "-with-skill", entry_id="authored-entry")+ parsed = tomllib.loads((preserved_dir / "task.toml").read_text(encoding="utf-8"))+ assert parsed["task"]["name"] == "nvidia/case-1-with-skill"+ assert parsed["metadata"]["entry_id"] == "authored-entry"++ uninjectable_dir = tmp_path / "uninjectable-metadata"+ uninjectable_dir.mkdir()+ (uninjectable_dir / "task.toml").write_text(+ 'metadata = "not-a-table"\n\n[task]\nname = "nvidia/case-1"\n',+ encoding="utf-8",+ )+ with pytest.raises(ValueError, match="Updated native Harbor task config is invalid TOML"):+ _append_native_task_name_suffix(uninjectable_dir, "-with-skill")++ mismatched_dir = tmp_path / "mismatched-metadata"+ mismatched_dir.mkdir()+ (mismatched_dir / "task.toml").write_text(+ '[metadata]\nentry_id = "existing-id"\n\n[task]\nname = "nvidia/case-1"\n',+ encoding="utf-8",+ )+ with pytest.raises(ValueError, match=r"Failed to record \[metadata\]\.entry_id"):+ _append_native_task_name_suffix(mismatched_dir, "-with-skill", entry_id="conflicting-id")+++def test_dual_arm_native_tasks_with_authored_arm_suffix_do_not_collide_in_custom_only_collection(+ tmp_path: Path,+) -> None:+ """Verify dual-arm staging and custom-only collection keep 'case-1', 'case-1-with-skill', and 'retry-attempt1' distinct across inline and standard task.toml syntax."""+ from skillevaluator.tier3.harbor.collector import collect_harbor_results++ skill_dir = tmp_path / "target-skill"+ skill_dir.mkdir()+ (skill_dir / "SKILL.md").write_text("# Target Skill\n", encoding="utf-8")+ harbor_dir = skill_dir / "evals" / "harbor"+ task_toml_templates = {+ "case-1": 'schema_version = "1.3"\n\n["\\u0074ask"]\n"\\u006eame" = "nvidia/case-1"\n\n[environment]\n',+ "case-1-with-skill": 'schema_version = "1.3"\ntask = { name = "nvidia/case-1-with-skill" }\n\n[environment]\n',+ "retry-attempt1": 'schema_version = "1.3"\n[task]\nname = "nvidia/skillevaluator-retry-attempt1"\n\n[environment]\n',+ }+ for case_id, toml_body in task_toml_templates.items():+ task_dir = harbor_dir / case_id+ task_dir.mkdir(parents=True)+ (task_dir / "instruction.md").write_text("Instruction\n", encoding="utf-8")+ (task_dir / "task.toml").write_text(toml_body, encoding="utf-8")+ tests_dir = task_dir / "tests"+ tests_dir.mkdir()+ (tests_dir / "test.sh").write_text("#!/bin/bash\nexit 0\n", encoding="utf-8")++ # Stage into results-1/_harbor-tasks/opencode/{with,without} so idx=1 (expected_case_ids=None)+ # tests authoritative [metadata].entry_id lookup via _staged_task_entry_id_map, while idx=0+ # (expected_case_ids=[...], no _harbor-tasks in results-0) tests deferred attempt-suffix stripping+ # on raw result["task_name"] protected by expected_case_ids.+ staged_root = tmp_path / "results-1" / "_harbor-tasks" / "opencode"+ with_staged = {+ p.name: tomllib.loads((p / "task.toml").read_text(encoding="utf-8"))+ for p in stage_native_harbor_tasks(+ skill_dir,+ staged_root / "with",+ grading_mode="custom_only",+ arm_suffix="-with-skill",+ )+ }+ without_staged = {+ p.name: tomllib.loads((p / "task.toml").read_text(encoding="utf-8"))+ for p in stage_native_harbor_tasks(+ skill_dir,+ staged_root / "without",+ grading_mode="custom_only",+ with_skill=False,+ arm_suffix="-without-skill",+ )+ }++ assert with_staged["case-1"]["task"]["name"] == "nvidia/case-1-with-skill"+ assert with_staged["case-1-with-skill"]["task"]["name"] == "nvidia/case-1-with-skill-with-skill"+ assert with_staged["retry-attempt1"]["task"]["name"] == "nvidia/skillevaluator-retry-attempt1-with-skill"+ assert without_staged["case-1"]["task"]["name"] == "nvidia/case-1-without-skill"+ assert without_staged["case-1-with-skill"]["task"]["name"] == "nvidia/case-1-with-skill-without-skill"+ assert without_staged["retry-attempt1"]["task"]["name"] == "nvidia/skillevaluator-retry-attempt1-without-skill"++ jobs_dir = tmp_path / "jobs"+ for variant, staged_map, scores in (+ ("with", with_staged, {"case-1": 1.0, "case-1-with-skill": 0.9, "retry-attempt1": 0.8}),+ ("without", without_staged, {"case-1": 0.2, "case-1-with-skill": 0.3, "retry-attempt1": 0.4}),+ ):+ job_dir = jobs_dir / f"target-skill-opencode-{variant}"+ trial_names: list[str] = []+ for idx, (case_id, parsed_toml) in enumerate(staged_map.items(), start=1):+ trial_name = f"trial-{idx}__attempt1"+ trial_names.append(trial_name)+ trial_dir = job_dir / trial_name+ trial_dir.mkdir(parents=True)+ # Test both raw task_name (no _attempt1 suffix) and _attempt1 suffixed task_name+ task_name_val = (+ parsed_toml["task"]["name"]+ if case_id == "retry-attempt1"+ else f"{parsed_toml['task']['name']}_attempt1"+ )+ (trial_dir / "result.json").write_text(+ json.dumps(+ {+ "trial_name": trial_name,+ "task_name": task_name_val,+ "verifier_result": {"rewards": {"reward": scores[case_id]}},+ }+ ),+ encoding="utf-8",+ )+ (job_dir / "result.json").write_text(+ json.dumps(+ {+ "n_total_trials": len(trial_names),+ "stats": {+ "n_trials": len(trial_names),+ "n_errors": 0,+ "evals": {+ "opencode": {+ "n_trials": len(trial_names),+ "n_errors": 0,+ "reward_stats": {"reward": {"1.0": trial_names}},+ }+ },+ },+ }+ ),+ encoding="utf-8",+ )++ for idx, expected_ids in enumerate((["case-1", "case-1-with-skill", "retry-attempt1"], None)):+ result = collect_harbor_results(+ skill_name="target-skill",+ agents=["opencode"],+ output_dir=tmp_path / f"results-{idx}",+ jobs_dir=jobs_dir,+ n_attempts=1,+ expected_cases=3,+ expected_case_ids=expected_ids,+ )++ assert result["execution_status"] == "succeeded"+ agent_res = result["agents"]["opencode"]+ assert set(agent_res["pass_at_k"]["with_skill"]["cases"]) == {"case-1", "case-1-with-skill", "retry-attempt1"}+ assert set(agent_res["pass_at_k"]["without_skill"]["cases"]) == {+ "case-1",+ "case-1-with-skill",+ "retry-attempt1",+ }+ assert agent_res["pass_at_k"]["with_skill"]["cases"]["case-1"]["best_score"] == 1.0+ assert agent_res["pass_at_k"]["with_skill"]["cases"]["case-1-with-skill"]["best_score"] == 0.9+ assert agent_res["pass_at_k"]["with_skill"]["cases"]["retry-attempt1"]["best_score"] == 0.8+ assert agent_res["pass_at_k"]["without_skill"]["cases"]["case-1"]["best_score"] == 0.2+ assert agent_res["pass_at_k"]["without_skill"]["cases"]["case-1-with-skill"]["best_score"] == 0.3+ assert agent_res["pass_at_k"]["without_skill"]["cases"]["retry-attempt1"]["best_score"] == 0.4++ # Sad path: if only 'case-1' is expected, 'case-1-with-skill' must be flagged as unexpected, not collapsed onto 'case-1'+ partial_expected_result = collect_harbor_results(+ skill_name="target-skill",+ agents=["opencode"],+ output_dir=tmp_path / "results-unexpected",+ jobs_dir=jobs_dir,+ n_attempts=1,+ expected_cases=3,+ expected_case_ids=["case-1"],+ )+ assert partial_expected_result["execution_status"] == "failed"+ assert any("case-1-with-skill" in err for err in partial_expected_result["execution_errors"])+++def test_stage_native_harbor_tasks_validates_metadata_entry_ids_sad_paths(tmp_path: Path) -> None:+ """Verify native staging rejects unsafe or duplicate [metadata].entry_id values before mutating output_dir."""+ # 1. Unsafe metadata.entry_id (e.g. path traversal or boolean)+ for idx, bad_entry_id in enumerate(('"../escape"', "true", '""')):+ skill_dir = tmp_path / f"bad-skill-{idx}"+ skill_dir.mkdir()+ (skill_dir / "SKILL.md").write_text("# Bad Skill\n", encoding="utf-8")+ task_dir = skill_dir / "evals" / "harbor" / "physical-case"+ task_dir.mkdir(parents=True)+ (task_dir / "instruction.md").write_text("Instruction\n", encoding="utf-8")+ (task_dir / "task.toml").write_text(+ f'[metadata]\nentry_id = {bad_entry_id}\n\n[task]\nname = "nvidia/physical-case"\n',+ encoding="utf-8",+ )+ tests_dir = task_dir / "tests"+ tests_dir.mkdir()+ (tests_dir / "test.sh").write_text("#!/bin/bash\nexit 0\n", encoding="utf-8")++ out_dir = tmp_path / f"out-bad-{idx}"+ with pytest.raises(ValueError, match="case id"):+ stage_native_harbor_tasks(skill_dir, out_dir, grading_mode="custom_only")+ assert not out_dir.exists()++ # 2. Duplicate metadata.entry_id across two distinct physical task directories+ dup_skill = tmp_path / "dup-skill"+ dup_skill.mkdir()+ (dup_skill / "SKILL.md").write_text("# Dup Skill\n", encoding="utf-8")+ for folder in ("physical-a", "physical-b"):+ task_dir = dup_skill / "evals" / "harbor" / folder+ task_dir.mkdir(parents=True)+ (task_dir / "instruction.md").write_text("Instruction\n", encoding="utf-8")+ (task_dir / "task.toml").write_text(+ '[metadata]\nentry_id = "shared-authored-id"\n\n[task]\nname = "nvidia/task"\n',+ encoding="utf-8",+ )+ tests_dir = task_dir / "tests"+ tests_dir.mkdir()+ (tests_dir / "test.sh").write_text("#!/bin/bash\nexit 0\n", encoding="utf-8")++ dup_out = tmp_path / "out-dup"+ with pytest.raises(ValueError, match=r"duplicate or cross-platform colliding case id.*shared-authored-id"):+ stage_native_harbor_tasks(dup_skill, dup_out, grading_mode="custom_only")+ assert not dup_out.exists()+++def test_native_harbor_metadata_entry_id_seam_across_runner_collector_and_dataset_snapshot(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """Verify native custom-only tasks with [metadata].entry_id != directory name align across runner, collector, and dataset_snapshot.json."""+ from skillevaluator.provider_config import ProviderConfig+ from skillevaluator.tier3.harbor import runner, runtime_preflight++ skill_dir = tmp_path / "target-skill"+ skill_dir.mkdir()+ (skill_dir / "SKILL.md").write_text("---\nname: target-skill\ndescription: demo\n---\n# Body\n", encoding="utf-8")+ (skill_dir / "evals").mkdir()+ (skill_dir / "evals" / "config.yaml").write_text(+ "schema_version: 1\nharbor:\n task_source: native_harbor\ngrading:\n mode: custom_only\n",+ encoding="utf-8",+ )+ for folder, authored_entry_id in (("physical-case", '"authored-entry"'), ("zero-case", "0")):+ task_dir = skill_dir / "evals" / "harbor" / folder+ task_dir.mkdir(parents=True)+ (task_dir / "instruction.md").write_text("Solve the task.\n", encoding="utf-8")+ (task_dir / "task.toml").write_text(+ f'schema_version = "1.3"\n\n[metadata]\nentry_id = {authored_entry_id}\n\n'+ f'[task]\nname = "nvidia/{folder}"\n\n[environment]\n',+ encoding="utf-8",+ )+ tests_dir = task_dir / "tests"+ tests_dir.mkdir()+ (tests_dir / "test.sh").write_text("#!/bin/bash\nexit 0\n", encoding="utf-8")++ output_dir = tmp_path / "eval-out"+ provider = ProviderConfig(+ provider="nv_build",+ model="nvidia/nemotron-3-nano-30b-a3b",+ api_key="nvapi-test",+ base_url="https://integrate.api.nvidia.com/v1",+ litellm_model="nvidia_nim/nvidia/nemotron-3-nano-30b-a3b",+ )+ monkeypatch.setattr(runner, "resolve_llm_provider", lambda: provider)+ monkeypatch.setattr(runner, "_check_prerequisites", lambda **_kwargs: [])+ monkeypatch.setattr(+ runtime_preflight,+ "probe_model",+ lambda selected_provider: runtime_preflight.ModelProbeResult(+ True,+ selected_provider.provider,+ selected_provider.model,+ "ok",+ ),+ )++ seen_include_task_names: list[list[str] | None] = []++ def fake_run_harbor(+ *,+ dataset: Path,+ agent: str,+ job_name: str,+ env_mode: str,+ model: str,+ jobs_dir: Path,+ run_env: dict[str, str],+ n_attempts: int,+ n_concurrent: int,+ timeout_multiplier: float,+ override_cpus: int | None,+ override_memory_mb: int | None,+ override_storage_mb: int | None,+ expected_trials: int,+ agent_import_path: str | None = None,+ verifier_env: object = None,+ include_task_names: list[str] | None = None,+ environment_kwargs: object = None,+ **_kwargs: object,+ ) -> tuple[bool, str]:+ from datetime import UTC, datetime+ from uuid import uuid4++ from harbor.models.job.result import JobResult, JobStats+ from harbor.models.trial.result import TrialResult++ _ = (agent_import_path, verifier_env, environment_kwargs)+ seen_include_task_names.append(include_task_names)+ job_dir = jobs_dir / job_name+ is_with = "-with-" in job_name or job_name.endswith("-with")+ arm_suffix = "-with-skill" if is_with else "-without-skill"+ selected_folders = include_task_names or ["physical-case", "zero-case"]+ now = datetime(2026, 9, 12, tzinfo=UTC)+ trial_results: list[TrialResult] = []+ for idx, folder in enumerate(selected_folders, start=1):+ trial_name = f"trial-{idx}-{folder}__attempt1"+ trial_dir = job_dir / trial_name+ trial_dir.mkdir(parents=True, exist_ok=True)+ # Harbor persists the staged task path, while task_name carries the+ # suffixed display name; only the runner's selector map may turn the+ # staged directory into the authored [metadata].entry_id.+ task_path = str(dataset / folder)+ trial_result = TrialResult.model_validate(+ {+ "id": uuid4(),+ "task_name": f"nvidia/{folder}{arm_suffix}_attempt1",+ "trial_name": trial_name,+ "trial_uri": trial_dir.as_uri(),+ "task_id": {"path": task_path},+ "task_checksum": "native-entry-id-seam-fixture",+ "config": {"task": {"path": task_path}, "trial_name": trial_name, "trials_dir": str(job_dir)},+ "agent_info": {"name": agent, "version": "test", "model_info": {"name": "test-model"}},+ "agent_result": {},+ "verifier_result": {"rewards": {"reward": 1.0 if is_with else 0.25}},+ "started_at": now,+ "finished_at": now,+ }+ )+ (trial_dir / "result.json").write_text(trial_result.model_dump_json(indent=2), encoding="utf-8")+ (trial_dir / "config.json").write_text(trial_result.config.model_dump_json(indent=2), encoding="utf-8")+ trial_results.append(trial_result)+ job_result = JobResult(+ id=uuid4(),+ started_at=now,+ updated_at=now,+ finished_at=now,+ n_total_trials=len(trial_results),+ stats=JobStats.from_trial_results(trial_results, n_total_trials=len(trial_results)),+ trial_results=[],+ )+ (job_dir / "result.json").write_text(job_result.model_dump_json(indent=2), encoding="utf-8")+ return True, ""++ monkeypatch.setattr(runner, "_run_harbor", fake_run_harbor)++ results = runner.run_harbor_eval(+ skill_path=skill_dir,+ agents=["opencode"],+ output_dir=output_dir,+ env_mode="docker",+ n_attempts=2,+ stop_on_pass=True,+ agent_runtime_preflight=False,+ )++ assert results["execution_status"] == "succeeded"+ # stop_on_pass passes physical task directory names to Harbor CLI --include-task-name+ # With-skill passes on attempt 1 (1.0 >= 0.50), baseline runs 2 attempts per task (0.25 < 0.50)+ assert seen_include_task_names == [+ ["physical-case"],+ ["zero-case"],+ ["physical-case"],+ ["physical-case"],+ ["zero-case"],+ ["zero-case"],+ ]++ agent_res = results["agents"]["opencode"]+ assert set(agent_res["pass_at_k"]["with_skill"]["cases"]) == {"authored-entry", "0"}+ assert set(agent_res["pass_at_k"]["without_skill"]["cases"]) == {"authored-entry", "0"}+ assert agent_res["pass_at_k"]["with_skill"]["cases"]["authored-entry"]["best_score"] == 1.0+ assert agent_res["pass_at_k"]["without_skill"]["cases"]["authored-entry"]["best_score"] == 0.25++ run_dir = Path(results["run_dir"])+ snapshot = json.loads((run_dir / "dataset_snapshot.json").read_text(encoding="utf-8"))+ assert snapshot["dataset"] == [{"id": "authored-entry"}, {"id": "0"}]+++def test_stage_native_harbor_tasks_type_safety(tmp_path: Path) -> None:+ """Verify that native staging functions reject non-string arm_suffix values."""+ skill_dir = tmp_path / "target-skill"+ skill_dir.mkdir()++ with pytest.raises(TypeError, match="arm_suffix must be a string"):+ stage_native_harbor_tasks(skill_dir, tmp_path / "err", arm_suffix=123) # type: ignore[arg-type]++ with pytest.raises(TypeError, match="arm_suffix must be a string"):+ _stage_native_harbor_tasks_into(+ skill_dir,+ tmp_path / "err",+ evaluator_skill_path=skill_dir,+ arm_suffix=123, # type: ignore[arg-type]+ )+++def test_generate_harbor_tasks_dual_arm_suffix(tmp_path: Path) -> None:+ """Verify that generate_harbor_tasks propagates arm suffix into task.toml name."""+ skill_dir = tmp_path / "gen-skill"+ skill_dir.mkdir()+ (skill_dir / "SKILL.md").write_text("# Gen Skill\n", encoding="utf-8")+ evals_dir = skill_dir / "evals"+ evals_dir.mkdir()+ (evals_dir / "evals.json").write_text(+ json.dumps([{"id": "case-001", "prompt": "test prompt"}]),+ encoding="utf-8",+ )++ out = tmp_path / "gen_out"+ staged = generate_harbor_tasks(skill_dir, out, arm_suffix="-with-skill")[0]+ task = tomllib.loads((staged / "task.toml").read_text(encoding="utf-8"))+ assert task["task"]["name"] == "nvidia/skillevaluator-case-001-with-skill"++ with pytest.raises(TypeError, match="arm_suffix must be a string"):+ generate_harbor_tasks(skill_dir, tmp_path / "err", arm_suffix=123) # type: ignore[arg-type]++ with pytest.raises(TypeError, match="arm_suffix must be a string"):+ _generate_harbor_tasks_into(+ skill_dir,+ tmp_path / "err",+ evaluator_skill_path=skill_dir,+ arm_suffix=123, # type: ignore[arg-type]+ )+++def test_write_task_toml_forwards_retry_env(tmp_path: Path) -> None:+ """Verify that _write_task_toml stages LLM retry variables into [verifier.env]."""+ case_dir = tmp_path / "case"+ case_dir.mkdir()+ runtime_env = {+ "SKILL_EVAL_LLM_MAX_RETRIES": "5",+ "SKILL_EVAL_LLM_RETRY_BASE_DELAY": "2.0",+ "SKILL_EVAL_LLM_RETRY_MAX_DELAY": "40.0",+ "UNRELATED_CUSTOM_VAR": "secret",+ }+ _write_task_toml(+ case_dir,+ {"id": "case-001", "expected_skill": "demo"},+ has_skill=True,+ runtime_env=runtime_env,+ )+ task = tomllib.loads((case_dir / "task.toml").read_text(encoding="utf-8"))+ verifier_env = task["verifier"]["env"]+ assert verifier_env["SKILL_EVAL_LLM_MAX_RETRIES"] == "${SKILL_EVAL_LLM_MAX_RETRIES}"+ assert verifier_env["SKILL_EVAL_LLM_RETRY_BASE_DELAY"] == "${SKILL_EVAL_LLM_RETRY_BASE_DELAY}"+ assert verifier_env["SKILL_EVAL_LLM_RETRY_MAX_DELAY"] == "${SKILL_EVAL_LLM_RETRY_MAX_DELAY}"+ assert "UNRELATED_CUSTOM_VAR" not in verifier_env+++ ("env_mode", "agent_import_path"),+ [+ ("docker", None),+ ("docker", "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorNvidiaBuildCodex"),+ ("local", None),+ ("e2b", None),+ ("daytona", "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorGatewayCodex"),+ ],+)+def test_codex_runs_pin_harbor_022_reasoning_effort_and_cli_version(+ env_mode: str, agent_import_path: str | None+) -> None:+ command = build_harbor_run_command(+ dataset_path="/tmp/dataset",+ agent="codex",+ job_name="effort",+ env_mode=env_mode,+ agent_import_path=agent_import_path,+ )++ # Local mode runs the installed CLI, so only Harbor-installed runs pin a version.+ expected = (+ {"reasoning_effort": "high"} if env_mode == "local" else {"reasoning_effort": "high", "version": "0.161.0"}+ )+ assert _harbor_agent_kwargs(command) == expected+++def _harbor_agent_kwargs(command: list[str]) -> dict[str, object]:+ from harbor.cli.utils import parse_kwargs++ return parse_kwargs([command[index + 1] for index, value in enumerate(command) if value == "--ak"])+++ ("model", "expected"),+ [+ (None, {"version": "2.1.292"}),+ ("claude-opus-5", {"version": "2.1.292"}),+ ("us.anthropic.claude-opus-5", {"version": "2.1.292"}),+ ("aws/anthropic/bedrock-claude-opus-5", {"version": "2.1.292"}),+ ("claude-opus-5-5", {"reasoning_effort": "medium", "version": "2.1.292"}),+ ("us.anthropic.claude-opus-5-5", {"reasoning_effort": "medium", "version": "2.1.292"}),+ ("aws/anthropic/bedrock-claude-opus-5-5", {"reasoning_effort": "medium", "version": "2.1.292"}),+ ("anthropic/claude-sonnet-5-5-20261001", {"reasoning_effort": "medium", "version": "2.1.292"}),+ ],+)+def test_claude_code_runs_pin_cli_version_and_claude_5_5_effort(model: str | None, expected: dict[str, str]) -> None:+ command = build_harbor_run_command(+ dataset_path="/tmp/dataset", agent="claude-code", job_name="effort", env_mode="docker", model=model+ )++ assert _harbor_agent_kwargs(command) == expected+++def test_local_claude_code_runs_keep_effort_without_a_cli_version() -> None:+ command = build_harbor_run_command(+ dataset_path="/tmp/dataset", agent="claude-code", job_name="effort", env_mode="local", model="claude-opus-5-5"+ )++ assert _harbor_agent_kwargs(command) == {"reasoning_effort": "medium"}+++def test_documented_agent_cli_versions_match_the_pins() -> None:+ from skillevaluator.tier3.harbor import local_agents, runner++ docs = Path(__file__).resolve().parents[1] / "docs"+ pages = {+ page: " ".join((docs / page).read_text(encoding="utf-8").split())+ for page in ("agents-and-sandboxes.mdx", "tier3-live-evaluation.mdx")+ }+ for page, text in pages.items():+ assert f"Codex {runner._CODEX_VERSION}" in text, page+ assert f"Claude Code {runner._CLAUDE_CODE_VERSION}" in text, page+ for name, minimum in (+ ("Claude Code", local_agents._CLAUDE_CODE_MIN_VERSION_FOR_CLAUDE_5_5),+ ("Codex", local_agents._CODEX_MIN_VERSION_FOR_GPT_6),+ ):+ assert f"{name} {'.'.join(map(str, minimum))} or newer" in pages["agents-and-sandboxes.mdx"]+++def test_opencode_runs_pass_no_agent_kwargs() -> None:+ command = build_harbor_run_command(+ dataset_path="/tmp/dataset",+ agent="opencode",+ job_name="effort",+ env_mode="docker",+ model="anthropic/claude-opus-5-5",+ )++ assert "--ak" not in command+++def test_codex_agents_render_the_pinned_reasoning_effort(tmp_path: Path) -> None:+ from harbor.agents.installed.codex import Codex++ from skillevaluator.tier3.harbor import local_agents++ for agent_class in (+ Codex,+ local_agents.SkillEvaluatorGatewayCodex,+ local_agents.SkillEvaluatorLocalCodex,+ local_agents.SkillEvaluatorNvidiaBuildCodex,+ ):+ agent = agent_class(logs_dir=tmp_path, model_name="openai/gpt-5", reasoning_effort="high")+ assert "-c model_reasoning_effort=high" in agent.build_cli_flags(), agent_class.__name__+++def test_claude_code_agents_accept_the_pinned_version_and_effort(tmp_path: Path) -> None:+ from harbor.agents.installed.claude_code import ClaudeCode++ from skillevaluator.tier3.harbor import local_agents++ for agent_class in (+ ClaudeCode,+ local_agents.SkillEvaluatorLocalClaudeCode,+ local_agents.SkillEvaluatorNvidiaBuildClaudeCode,+ local_agents.SkillEvaluatorLocalNvidiaBuildClaudeCode,+ ):+ agent = agent_class(+ logs_dir=tmp_path, model_name="anthropic/claude-opus-5-5", reasoning_effort="medium", version="2.1.292"+ )+ assert "--effort medium" in agent.build_cli_flags(), agent_class.__name__+ assert agent.version() == "2.1.292", agent_class.__name__diff --git a/tests/tier3/test_evals_config.py b/tests/tier3/test_evals_config.pyindex b02201e7..51bb8692 100644--- a/tests/tier3/test_evals_config.py+++ b/tests/tier3/test_evals_config.py@@ -7,9 +7,10 @@ import pytestfrom skillevaluator.tier3.dataset_utils import load_dataset_entries_with_formatfrom skillevaluator.tier3.evals_config import (- _GKE_INFRASTRUCTURE_KWARGS,EvalsConfigError,load_evals_config,+ parse_environment_kwarg_overrides,+ validate_environment_kwargs,)from skillevaluator.tier3.evals_spec import validate_skillevaluators as validate_skill_evals@@ -319,130 +320,21 @@ harbor:load_evals_config(skill)-def test_load_evals_config_valid_environment_kwargs(tmp_path):- skill = tmp_path / "gke-skill"- (skill / "evals").mkdir(parents=True)- (skill / "evals" / "config.yml").write_text(- """\-schema_version: 1-harbor:- environment_kwargs:- custom_setting: custom_val- workload_profile: memory-optimized-""",- encoding="utf-8",- )-- config, path = load_evals_config(skill)-- assert path == skill / "evals" / "config.yml"- assert config["harbor"]["environment_kwargs"] == {- "custom_setting": "custom_val",- "workload_profile": "memory-optimized",- }---def test_load_evals_config_invalid_environment_kwargs(tmp_path):- skill = tmp_path / "invalid-skill"- (skill / "evals").mkdir(parents=True)-- # Not a mapping- (skill / "evals" / "config.yml").write_text(- """\-schema_version: 1-harbor:- environment_kwargs:- - custom_setting-""",- encoding="utf-8",- )- with pytest.raises(EvalsConfigError, match=r"harbor\.environment_kwargs must be a mapping"):- load_evals_config(skill)-- # Non-string value- (skill / "evals" / "config.yml").write_text(- """\-schema_version: 1-harbor:- environment_kwargs:- custom_setting: 12345-""",- encoding="utf-8",- )- with pytest.raises(- EvalsConfigError,- match=r"harbor\.environment_kwargs\.custom_setting must be a non-empty string",- ):- load_evals_config(skill)---def test_gke_infrastructure_kwargs_contains_expected_keys():- """Verify all 10 GKE infrastructure and security control kwargs are protected."""- expected = {- "allow_workload_identity",- "cluster_name",- "region",- "namespace",- "registry_location",- "registry_name",- "project_id",- "cloud_build_machine_type",- "cloud_build_disk_size_gb",- "memory_limit_multiplier",- }- assert expected == _GKE_INFRASTRUCTURE_KWARGS--[email protected]("blocked_key", sorted(_GKE_INFRASTRUCTURE_KWARGS))-def test_load_evals_config_blocked_infrastructure_kwargs(tmp_path, blocked_key):- """Untrusted skills must not configure infrastructure keys in evals/config.yml."""- skill = tmp_path / "blocked-skill"- (skill / "evals").mkdir(parents=True)- (skill / "evals" / "config.yml").write_text(- f"""\-schema_version: 1-harbor:- environment_kwargs:- {blocked_key}: forbidden-val-""",- encoding="utf-8",- )- with pytest.raises(- EvalsConfigError,- match=rf"harbor\.environment_kwargs\.{blocked_key} cannot be configured in skill evals/config\.yml",- ):- load_evals_config(skill)--@pytest.mark.parametrize(- "forbidden_key",+ ("entry", "secret"),[- "network_block_all",- "langsmith_endpoint",- "extra_docker_compose",- "arbitrary_backend_knob",- "cookie",- "oauth",+ ("custom_setting=sk-ant-api03-1234567890abcdefghijklmnop", "sk-ant-api03-1234567890abcdefghijklmnop"),+ ("custom_setting=ghp_1234567890abcdefghijklmnopqrstuvwxyz", "ghp_1234567890abcdefghijklmnopqrstuvwxyz"),+ ("custom_setting=AIzaSyA1234567890abcdefghijklmnopqrstuvw", "AIzaSyA1234567890abcdefghijklmnopqrstuvw"),+ ("custom_setting=-----BEGIN PRIVATE KEY-----", "-----BEGIN PRIVATE KEY-----"),+ ("service_account_key=my-secret-sa-key", "my-secret-sa-key"),],)-def test_load_evals_config_blocked_backend_security_controls(tmp_path, forbidden_key):- """Verify skills can only configure keys from the safe allowlist."""- skill = tmp_path / "forbidden-skill"- (skill / "evals").mkdir(parents=True)- (skill / "evals" / "config.yml").write_text(- f"""\-schema_version: 1-harbor:- environment_kwargs:- {forbidden_key}: some-value-""",- encoding="utf-8",- )- with pytest.raises(- EvalsConfigError,- match=rf"harbor\.environment_kwargs\.{forbidden_key} cannot be configured in skill evals/config\.yml",- ):- load_evals_config(skill)+def test_cli_environment_kwargs_reject_secret_value_patterns_without_echoing(entry, secret):+ with pytest.raises(ValueError, match="Sensitive key or value detected") as caught:+ parse_environment_kwarg_overrides((entry,))++ assert secret not in str(caught.value)def test_validate_skill_evals_does_not_warn_expected_script_for_guide_only_skill(tmp_path):@@ -580,3 +472,32 @@ def test_legacy_evals_json_is_accepted_with_deprecation_warning(tmp_path):assert not any(r.status == "error" for r in results)assert any("Deprecated eval dataset format" in r.message for r in results)+++ "infra_key",+ [+ "allow_workload_identity",+ "autopilot",+ "cluster_name",+ "region",+ "namespace",+ "registry_location",+ "registry_name",+ "project_id",+ "cloud_build_machine_type",+ "cloud_build_disk_size_gb",+ "memory_limit_multiplier",+ ],+)+def test_validate_environment_kwargs_gke_infrastructure_boundary(infra_key: str) -> None:+ """Reject GKE infrastructure keys when allow_gke_infrastructure_kwargs=False while allowing them on host CLI."""+ from skillevaluator.tier3.evals_config import validate_environment_kwargs++ with pytest.raises(ValueError, match=r"GKE infrastructure setting .* is not allowed in skill config"):+ validate_environment_kwargs({infra_key: "test-val"}, allow_gke_infrastructure_kwargs=False)++ assert validate_environment_kwargs({infra_key: "test-val"}, allow_gke_infrastructure_kwargs=True) == {+ infra_key: "test-val"+ }+diff --git a/tests/tier3/test_harbor_local_agents.py b/tests/tier3/test_harbor_local_agents.pyindex 28340129..f067104c 100644--- a/tests/tier3/test_harbor_local_agents.py+++ b/tests/tier3/test_harbor_local_agents.py@@ -2,9 +2,13 @@# SPDX-License-Identifier: Apache-2.0import asyncio+import contextlibimport json+import osimport shlex+import tomllibfrom pathlib import Path+from types import SimpleNamespaceimport pytest@@ -1030,3 +1034,81 @@ def test_claude_mcp_servers_preserves_task_runtime_env_args(tmp_path: Path) -> Nassert command is not Noneassert "db-server" in commandassert "${LOCAL_DB_PATH}" in command+++def test_claude_code_vertex_model_connection_and_auth_env(+ monkeypatch: pytest.MonkeyPatch,+ tmp_path: Path,+) -> None:+ """Suppress inferred Anthropic endpoint and forward Vertex AI env vars when CLAUDE_CODE_USE_VERTEX=1."""+ monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)+ monkeypatch.delenv("ANTHROPIC_BASE_URL", raising=False)+ agent = SkillEvaluatorClaudeCode(+ logs_dir=tmp_path,+ model_name="vertex_ai/claude-sonnet-4-6",+ extra_env={+ "CLAUDE_CODE_USE_VERTEX": "1",+ "CLOUD_ML_REGION": "global",+ "ANTHROPIC_VERTEX_PROJECT_ID": "my-gcp-project",+ "GOOGLE_CLOUD_PROJECT": "my-gcp-project",+ "GOOGLE_APPLICATION_CREDENTIALS": "/run/secrets/vertex-adc.json",+ "ANTHROPIC_DEFAULT_SONNET_MODEL": "claude-sonnet-4-6",+ "DISABLE_PROMPT_CACHING": "1",+ },+ )++ conn = agent.model_connection+ assert conn.provider is None+ assert conn.api_key is None+ assert conn.configured_base_url is None+ assert agent._resolved_model_name() == "claude-sonnet-4-6"++ auth_env = agent._resolve_auth_env()+ assert auth_env == {+ "CLAUDE_CODE_USE_VERTEX": "1",+ "CLOUD_ML_REGION": "global",+ "ANTHROPIC_VERTEX_PROJECT_ID": "my-gcp-project",+ "GOOGLE_CLOUD_PROJECT": "my-gcp-project",+ "GOOGLE_APPLICATION_CREDENTIALS": "/run/secrets/vertex-adc.json",+ "ANTHROPIC_DEFAULT_SONNET_MODEL": "claude-sonnet-4-6",+ "DISABLE_PROMPT_CACHING": "1",+ }+ assert "CLAUDE_CODE_USE_BEDROCK" not in auth_env+ assert "AWS_REGION" not in auth_env+ assert "ANTHROPIC_API_KEY" not in auth_env+ assert "ANTHROPIC_BASE_URL" not in auth_env+++def test_claude_code_vertex_exec_as_agent_unsets_anthropic_vars(+ monkeypatch: pytest.MonkeyPatch,+ tmp_path: Path,+) -> None:+ """Prepend defensive unset for Anthropic direct-API vars only when Vertex auth is active."""+ captured: list[str] = []++ async def fake_parent_exec(self, environment, command, **kwargs):+ captured.append(command)++ monkeypatch.setattr(+ "harbor.agents.installed.base.BaseInstalledAgent.exec_as_agent",+ fake_parent_exec,+ )++ vertex_agent = SkillEvaluatorClaudeCode(+ logs_dir=tmp_path,+ model_name="vertex_ai/claude-sonnet-4-6",+ extra_env={"CLAUDE_CODE_USE_VERTEX": "1"},+ )+ asyncio.run(vertex_agent.exec_as_agent(object(), "claude --version"))+ assert captured[-1] == (+ "unset ANTHROPIC_API_KEY ANTHROPIC_AUTH_TOKEN ANTHROPIC_BASE_URL CLAUDE_CODE_OAUTH_TOKEN; claude --version"+ )++ direct_agent = SkillEvaluatorClaudeCode(+ logs_dir=tmp_path,+ model_name="anthropic/claude-sonnet-4-6",+ extra_env={"ANTHROPIC_API_KEY": "sk-ant-test"},+ )+ asyncio.run(direct_agent.exec_as_agent(object(), "claude --version"))+ assert captured[-1] == "claude --version"+diff --git a/tests/tier3/test_harbor_template_secret_patterns.py b/tests/tier3/test_harbor_template_secret_patterns.pyindex fb46c3ff..ab0e02b1 100644--- a/tests/tier3/test_harbor_template_secret_patterns.py+++ b/tests/tier3/test_harbor_template_secret_patterns.py@@ -168,6 +168,7 @@ def test_security_constants_stay_in_sync_with_eval_core(name):"LOG_NVAPI_RE","LOG_CRSR_RE","OPENSHIFT_TOKEN_RE",+ "LOG_YA29_RE","LOG_JWT_RE","LOG_GITHUB_TOKEN_RE","LOG_GITHUB_PAT_RE",diff --git a/tests/tier3/test_installed_agent_bootstrap.py b/tests/tier3/test_installed_agent_bootstrap.pyindex 337e60a9..c11055ca 100644--- a/tests/tier3/test_installed_agent_bootstrap.py+++ b/tests/tier3/test_installed_agent_bootstrap.py@@ -18,6 +18,7 @@ from harbor.models.trial.config import AgentConfigfrom skillevaluator.tier3.harbor import DOCKER_AGENT_IMPORT_PATHSfrom skillevaluator.tier3.harbor.installed_agents import SkillEvaluatorClaudeCode, SkillEvaluatorCodex+from skillevaluator.tier3.harbor.local_agents import SkillEvaluatorClaudeCode as ContainerClaudeCodefrom skillevaluator.tier3.harbor.local_agents import (SkillEvaluatorGatewayCodex,SkillEvaluatorLocalClaudeCode,@@ -264,7 +265,7 @@ def test_native_bootstrap_preserves_unknown_dependency_validation(tmp_path: Path("agent_name", "agent_class", "model_name"),[("codex", SkillEvaluatorCodex, "openai/test-model"),- ("claude-code", SkillEvaluatorClaudeCode, "anthropic/test-model"),+ ("claude-code", ContainerClaudeCode, "anthropic/test-model"),],)@pytest.mark.parametrize("config_field", ["name", "import_path"])@@ -292,7 +293,12 @@ def test_docker_import_routes_resolve_and_preserve_report_identity(@pytest.mark.parametrize("agent_class",- [SkillEvaluatorGatewayCodex, SkillEvaluatorNvidiaBuildCodex, SkillEvaluatorNvidiaBuildClaudeCode],+ [+ SkillEvaluatorGatewayCodex,+ SkillEvaluatorNvidiaBuildCodex,+ SkillEvaluatorNvidiaBuildClaudeCode,+ ContainerClaudeCode,+ ],)def test_gateway_and_nvidia_wrappers_use_repaired_bootstrap(tmp_path: Path, agent_class: type) -> None:environment = _BootstrapEnvironment()diff --git a/tests/tier3/test_workflow.py b/tests/tier3/test_workflow.pyindex ce40c3b2..e328ff35 100644--- a/tests/tier3/test_workflow.py+++ b/tests/tier3/test_workflow.py@@ -615,16 +615,14 @@ def test_gke_workflow_resolves_env_vars_and_enforces_precedence_before_preflightconfigured: list,monkeypatch: pytest.MonkeyPatch,) -> None:- """Resolve SKILLEVALUATOR_GKE_* env vars with CLI > env > config precedence before workflow preflight."""+ """Resolve SKILLEVALUATOR_GKE_* env vars with CLI > env precedence before workflow preflight."""from skillevaluator.tier3.harbor import runner_dataset(skill)(skill / "evals" / "config.yml").write_text("schema_version: 1\n""harbor:\n"- " environment_kwargs:\n"- " workload_profile: 'standard'\n"- " custom_setting: 'from-config'\n"+ " n_attempts: 1\n")gke_env = {"SKILLEVALUATOR_GKE_CLUSTER": "env-cluster",@@ -661,11 +659,9 @@ def test_gke_workflow_resolves_env_vars_and_enforces_precedence_before_preflight"namespace": "env-ns","registry_location": "us-central1","registry_name": "env-registry",- "workload_profile": "standard",- "custom_setting": "from-config",}- # 2. CLI > env > config precedence+ # 2. CLI > env precedenceresult_override = CliRunner().invoke(_command(),[@@ -676,8 +672,6 @@ def test_gke_workflow_resolves_env_vars_and_enforces_precedence_before_preflight"opencode","--ek","namespace=cli-ns",- "--ek",- "custom_setting=from-cli","--progress","off",],@@ -686,8 +680,6 @@ def test_gke_workflow_resolves_env_vars_and_enforces_precedence_before_preflightassert len(captured_prereq_kwargs) == 2assert captured_prereq_kwargs[1]["namespace"] == "cli-ns"assert captured_prereq_kwargs[1]["cluster_name"] == "env-cluster"- assert captured_prereq_kwargs[1]["custom_setting"] == "from-cli"- assert captured_prereq_kwargs[1]["workload_profile"] == "standard"# 3. Skill-authored GKE infrastructure kwargs are rejected in config.yml and stripped by _resolve_environment_kwargsstripped = runner._resolve_environment_kwargs(@@ -711,5 +703,5 @@ def test_gke_workflow_resolves_env_vars_and_enforces_precedence_before_preflight[str(skill), "--env-mode", "gke", "--agents", "opencode", "--progress", "off"],)assert result_infra_in_config.exit_code != 0- assert "cluster_name cannot be configured in skill evals/config.yml" in result_infra_in_config.output+ assert "unknown harbor key(s): environment_kwargs" in result_infra_in_config.outputassert len(captured_prereq_kwargs) == 2diff --git a/tests/validators/test_security.py b/tests/validators/test_security.pyindex e29fd6a8..08b1edab 100644--- a/tests/validators/test_security.py+++ b/tests/validators/test_security.py@@ -4077,6 +4077,150 @@ Call us at 555-123-4567 or +1-555-987-6543assert child_env["OPENAI_API_KEY"] == "generic-openai-key"assert child_env["SKILL_EVAL_LLM_API_KEY"] == "generic-openai-key"+ @pytest.mark.parametrize(+ ("provider", "credentials", "expected"),+ (+ ("openai", {"OPENAI_API_KEY": "provider-test-key"}, "gpt-6.1-sol"),+ ("anthropic", {"ANTHROPIC_API_KEY": "provider-test-key"}, "claude-opus-5"),+ ("bedrock", {"AWS_BEARER_TOKEN_BEDROCK": "provider-test-key"}, "us.anthropic.claude-opus-5"),+ ("nv_build", {"NVIDIA_API_KEY": "provider-test-key"}, "nvidia/nemotron-3-super-120b-a12b"),+ (+ "openai-compatible",+ {+ "SKILL_EVAL_LLM_API_KEY": "provider-test-key",+ "SKILL_EVAL_LLM_BASE_URL": "https://llm.example.test/v1",+ },+ "nvidia/nvidia/nemotron-3-super-120b-long-ctx",+ ),+ ),+ )+ def test_skillspector_model_does_not_follow_a_changed_chat_default(+ self, monkeypatch, provider: str, credentials: dict[str, str], expected: str+ ) -> None:+ monkeypatch.setitem(CHAT_DEFAULT_MODELS, provider, "changed-chat-default")+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", provider)+ for name, value in credentials.items():+ monkeypatch.setenv(name, value)+ for name in ("SKILL_EVAL_LLM_MODEL", "SKILLSPECTOR_PROVIDER", "SKILLSPECTOR_MODEL"):+ monkeypatch.delenv(name, raising=False)++ assert _skillspector_child_env()["SKILLSPECTOR_MODEL"] == expected++ @pytest.mark.parametrize(+ ("skillspector_model", "expected"),+ ((None, "claude-opus-5-5"), ("claude-opus-5", "claude-opus-5")),+ )+ def test_explicit_models_reach_skillspector_in_precedence_order(+ self,+ monkeypatch,+ skillspector_model: str | None,+ expected: str,+ ) -> None:+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "anthropic")+ monkeypatch.setenv("ANTHROPIC_API_KEY", "provider-test-key")+ monkeypatch.setenv("SKILL_EVAL_LLM_MODEL", "claude-opus-5-5")+ monkeypatch.delenv("SKILLSPECTOR_PROVIDER", raising=False)+ if skillspector_model is None:+ monkeypatch.delenv("SKILLSPECTOR_MODEL", raising=False)+ else:+ monkeypatch.setenv("SKILLSPECTOR_MODEL", skillspector_model)++ assert _skillspector_child_env()["SKILLSPECTOR_MODEL"] == expected++ @pytest.mark.parametrize("skillspector_provider", (None, "openai"))+ def test_operator_skillspector_model_settings_are_forwarded_without_temperature(+ self,+ monkeypatch,+ skillspector_provider: str | None,+ ) -> None:+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai")+ monkeypatch.setenv("OPENAI_API_KEY", "provider-test-key")+ if skillspector_provider is None:+ monkeypatch.delenv("SKILLSPECTOR_PROVIDER", raising=False)+ else:+ monkeypatch.setenv("SKILLSPECTOR_PROVIDER", skillspector_provider)+ monkeypatch.setenv("SKILLSPECTOR_MODEL_META_ANALYZER", "gpt-5.6-sol")+ monkeypatch.setenv("SKILLSPECTOR_MODEL_REGISTRY", "/etc/skillspector/model_registry.yaml")+ monkeypatch.setenv("SKILLSPECTOR_REASONING_EFFORT", "high")+ monkeypatch.setenv("SKILLSPECTOR_TEMPERATURE", "0")++ child_env = _skillspector_child_env()++ assert child_env["SKILLSPECTOR_MODEL_META_ANALYZER"] == "gpt-5.6-sol"+ assert child_env["SKILLSPECTOR_MODEL_REGISTRY"] == "/etc/skillspector/model_registry.yaml"+ assert child_env["SKILLSPECTOR_REASONING_EFFORT"] == "high"+ assert "SKILLSPECTOR_TEMPERATURE" not in child_env++ @pytest.mark.parametrize(+ ("environment", "effort"),+ (+ ({}, None),+ ({"SKILL_EVAL_LLM_MODEL": "openai/openai/gpt-6.1-sol"}, "medium"),+ ({"SKILL_EVAL_LLM_MODEL": "aws/anthropic/bedrock-claude-opus-5-5"}, None),+ ({"SKILLSPECTOR_MODEL": "gpt-6.1-sol"}, "medium"),+ ({"SKILLSPECTOR_MODEL": "gpt-5.6-sol"}, None),+ ({"SKILLSPECTOR_MODEL": "gpt-6.1-sol", "SKILLSPECTOR_REASONING_EFFORT": "high"}, "high"),+ ({"SKILLSPECTOR_MODEL": "gpt-6.1-sol", "SKILLSPECTOR_MODEL_META_ANALYZER": "gpt-6.1-sol"}, "medium"),+ ({"SKILLSPECTOR_MODEL": "gpt-6.1-sol", "SKILLSPECTOR_MODEL_META_ANALYZER": "claude-opus-5-5"}, None),+ ({"SKILLSPECTOR_MODEL": "gpt-6.1-sol", "SKILLSPECTOR_MODEL_REGISTRY": "/registry.yaml"}, "medium"),+ ({"SKILLSPECTOR_PROVIDER": "openai", "SKILLSPECTOR_MODEL": "gpt-6.1-sol"}, "medium"),+ ({"SKILLSPECTOR_PROVIDER": "openai"}, None),+ ),+ )+ def test_skillspector_reasoning_effort_defaults_to_medium_only_for_gpt_6_models(+ self,+ monkeypatch,+ environment: dict[str, str],+ effort: str | None,+ ) -> None:+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai-compatible")+ monkeypatch.setenv("SKILL_EVAL_LLM_API_KEY", "compatible-test-key")+ monkeypatch.setenv("SKILL_EVAL_LLM_BASE_URL", "https://llm.example.test/v1")+ for name in (+ "SKILL_EVAL_LLM_MODEL",+ "SKILLSPECTOR_PROVIDER",+ "SKILLSPECTOR_MODEL",+ "SKILLSPECTOR_MODEL_META_ANALYZER",+ "SKILLSPECTOR_MODEL_REGISTRY",+ "SKILLSPECTOR_REASONING_EFFORT",+ ):+ monkeypatch.delenv(name, raising=False)+ for name, value in environment.items():+ monkeypatch.setenv(name, value)++ assert _skillspector_child_env().get("SKILLSPECTOR_REASONING_EFFORT") == effort++ def test_llm_stage_records_requested_skillspector_model_and_effort(+ self,+ monkeypatch,+ sample_skill_dir: Path,+ ) -> None:+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai")+ monkeypatch.setenv("OPENAI_API_KEY", "provider-test-key")+ monkeypatch.setenv("SKILL_EVAL_LLM_MODEL", "gpt-6.1-sol")+ for name in ("SKILLSPECTOR_PROVIDER", "SKILLSPECTOR_MODEL", "SKILLSPECTOR_REASONING_EFFORT"):+ monkeypatch.delenv(name, raising=False)+ reports = (_skillspector_json_report(), _skillspector_json_report(llm_requested=True, llm_available=True))++ with (+ patch.object(Tools.skillspector, "_path", "/usr/bin/skillspector"),+ patch.object(+ Tools.skillspector,+ "run",+ side_effect=[+ ToolResult(success=True, stdout=json.dumps(report), stderr="", exit_code=0) for report in reports+ ],+ ) as mock_run,+ ):+ result = SecurityValidator(use_llm=True).validate_security_only(sample_skill_dir)++ static_env, child_env = (call.kwargs["env"] for call in mock_run.call_args_list)+ assert "SKILLSPECTOR_REASONING_EFFORT" not in static_env+ assert child_env["SKILLSPECTOR_REASONING_EFFORT"] == "medium"+ assert result.metadata["skillspector_model"] == "gpt-6.1-sol"+ assert result.metadata["skillspector_reasoning_effort"] == "medium"+ assert result.metadata["skillspector_version"] == "2.10.0"+def test_partial_llm_verdicts_are_reported_as_partial(self, tmp_path: Path) -> None:"""A verifier response covering only some findings must not claim full confirmation."""result = ValidationResult()diff --git a/uv.lock b/uv.lockindex 92c612ab..1ffe7179 100644--- a/uv.lock+++ b/uv.lock@@ -1,5 +1,5 @@version = 1-revision = 3+revision = 5requires-python = ">=3.12, <3.14"resolution-markers = ["sys_platform == 'win32'",@@ -422,10 +422,10 @@ resolution-markers = ["sys_platform == 'linux'",]dependencies = [- { name = "license-expression" },- { name = "packageurl-python" },- { name = "py-serializable" },- { name = "sortedcontainers" },+ { name = "license-expression", marker = "sys_platform == 'linux'" },+ { name = "packageurl-python", marker = "sys_platform == 'linux'" },+ { name = "py-serializable", marker = "sys_platform == 'linux'" },+ { name = "sortedcontainers", marker = "sys_platform == 'linux'" },]sdist = { url = "https://files.pythonhosted.org/packages/66/fc/abaad5482f7b59c9a0a9d8f354ce4ce23346d582a0d85730b559562bbeb4/cyclonedx_python_lib-9.1.0.tar.gz", hash = "sha256:86935f2c88a7b47a529b93c724dbd3e903bc573f6f8bd977628a7ca1b5dadea1", size = 1048735, upload-time = "2025-02-27T17:23:40.367Z" }wheels = [@@ -442,11 +442,11 @@ resolution-markers = ["sys_platform != 'emscripten' and sys_platform != 'linux' and sys_platform != 'win32'",]dependencies = [- { name = "license-expression" },- { name = "packageurl-python" },- { name = "py-serializable" },- { name = "sortedcontainers" },- { name = "typing-extensions", marker = "python_full_version < '3.13'" },+ { name = "license-expression", marker = "sys_platform != 'linux'" },+ { name = "packageurl-python", marker = "sys_platform != 'linux'" },+ { name = "py-serializable", marker = "sys_platform != 'linux'" },+ { name = "sortedcontainers", marker = "sys_platform != 'linux'" },+ { name = "typing-extensions", marker = "python_full_version < '3.13' and sys_platform != 'linux'" },]sdist = { url = "https://files.pythonhosted.org/packages/75/c9/5d0ccdd19bc7d8ab803b90695c1706aa2ea8529685d18e682dc2524d2630/cyclonedx_python_lib-11.11.0.tar.gz", hash = "sha256:4b3194db72b613717f2912447e67ab618c75ff7dcac6c4af3c0e9e1ac617c102", size = 1442983, upload-time = "2026-06-17T11:57:49.055Z" }wheels = [@@ -1175,59 +1175,13 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/81/08/7036c080d7117f28a4af526d794aab6a84463126db031b007717c1a6676e/multidict-6.7.1-py3-none-any.whl", hash = "sha256:55d97cc6dae627efa6a6e548885712d4864b81110ac76fa4e534c03819fa4a56", size = 12319, upload-time = "2026-01-26T02:46:44.004Z" },]-[[package]]-name = "multiprocess"-version = "0.70.19"-source = { registry = "https://pypi.org/simple" }-dependencies = [- { name = "dill" },-]-sdist = { url = "https://files.pythonhosted.org/packages/a2/f2/e783ac7f2aeeed14e9e12801f22529cc7e6b7ab80928d6dcce4e9f00922d/multiprocess-0.70.19.tar.gz", hash = "sha256:952021e0e6c55a4a9fe4cd787895b86e239a40e76802a789d6305398d3975897", size = 2079989, upload-time = "2026-01-19T06:47:39.744Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/e3/45/8004d1e6b9185c1a444d6b55ac5682acf9d98035e54386d967366035a03a/multiprocess-0.70.19-py310-none-any.whl", hash = "sha256:97404393419dcb2a8385910864eedf47a3cadf82c66345b44f036420eb0b5d87", size = 134948, upload-time = "2026-01-19T06:47:32.325Z" },- { url = "https://files.pythonhosted.org/packages/86/c2/dec9722dc3474c164a0b6bcd9a7ed7da542c98af8cabce05374abab35edd/multiprocess-0.70.19-py311-none-any.whl", hash = "sha256:928851ae7973aea4ce0eaf330bbdafb2e01398a91518d5c8818802845564f45c", size = 144457, upload-time = "2026-01-19T06:47:33.711Z" },- { url = "https://files.pythonhosted.org/packages/71/70/38998b950a97ea279e6bd657575d22d1a2047256caf707d9a10fbce4f065/multiprocess-0.70.19-py312-none-any.whl", hash = "sha256:3a56c0e85dd5025161bac5ce138dcac1e49174c7d8e74596537e729fd5c53c28", size = 150281, upload-time = "2026-01-19T06:47:35.037Z" },- { url = "https://files.pythonhosted.org/packages/7f/74/d2c27e03cb84251dfe7249b8e82923643c6d48fa4883b9476b025e7dc7eb/multiprocess-0.70.19-py313-none-any.whl", hash = "sha256:8d5eb4ec5017ba2fab4e34a747c6d2c2b6fecfe9e7236e77988db91580ada952", size = 156414, upload-time = "2026-01-19T06:47:35.915Z" },- { url = "https://files.pythonhosted.org/packages/7e/82/69e539c4c2027f1e1697e09aaa2449243085a0edf81ae2c6341e84d769b6/multiprocess-0.70.19-py39-none-any.whl", hash = "sha256:0d4b4397ed669d371c81dcd1ef33fd384a44d6c3de1bd0ca7ac06d837720d3c5", size = 133477, upload-time = "2026-01-19T06:47:38.619Z" },-]--[[package]]-name = "numpy"-version = "2.5.1"-source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/22/fd/89965aa4ac08c74998539fcbf24fa3540f3e15237fbeb6bcf9c908f4aade/numpy-2.5.1.tar.gz", hash = "sha256:a48a113e6afea91f5608793bafa7ef2ad481fefbda87ec5069f483de61cb9fa3", size = 20755553, upload-time = "2026-07-04T17:08:00.933Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/62/7b/14687aa674250e5e546f616f486b0d56d3631cd5b2415739141ce40bdcea/numpy-2.5.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2c889b56fe48b1018f764b0eec8df59ab654e9148aa91faa12596043500de277", size = 16801574, upload-time = "2026-07-04T17:06:12.423Z" },- { url = "https://files.pythonhosted.org/packages/e1/19/cc5bb2a3f2913d27d6dbb2c78d25921fabaedc6741d4a5a615a11f3c5bf3/numpy-2.5.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ab451b59c5643c570974c43aef780703ef1d3b4965d2be07afd530615a9358d1", size = 11772250, upload-time = "2026-07-04T17:06:15.726Z" },- { url = "https://files.pythonhosted.org/packages/42/77/fdf34a71dd30f54979b18603bee915e0aaf825b07afe79acd60b04b691e2/numpy-2.5.1-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:78798bd5b9ad744056af8efa90e3b9ddaa53272a0848a483084a1cc0a13b2dc0", size = 5331516, upload-time = "2026-07-04T17:06:17.913Z" },- { url = "https://files.pythonhosted.org/packages/ce/e2/eb7efa015b4cce41e2517bf182a7fce0d7d5b9d9ed76a29bfa0f4fe4505c/numpy-2.5.1-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:2ae0ca40bcb22d6ba59c1dfd5446f49940b0f2d821fde133f10dda11f816b84e", size = 6664863, upload-time = "2026-07-04T17:06:20.02Z" },- { url = "https://files.pythonhosted.org/packages/a9/4b/a2b32dd94ee9ffbeecb28152240042a3949db33b1c834d44090b80e1b3b8/numpy-2.5.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:61ac47e772e6b8ea489e1d2f441a34c5c3ac17327e7ce294cbdf535795ad4e75", size = 15167977, upload-time = "2026-07-04T17:06:21.621Z" },- { url = "https://files.pythonhosted.org/packages/b8/a9/6e73d68500f80773f65f0654ea932019d6694329a0eb0ed0533de38df376/numpy-2.5.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:59fda5e192b570217ec2580c96f00e9a7e12ef6866a900eb089b62c1a32545ca", size = 16672469, upload-time = "2026-07-04T17:06:24.064Z" },- { url = "https://files.pythonhosted.org/packages/24/7d/ad3e59015135f5261c95fd4cafeff159c955febd83a99a1d9250c4233815/numpy-2.5.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:f7119ebff1a9829e9f431a4f9d28e703023bb6b9fe7c8f724467dbfc27c94ab3", size = 16527531, upload-time = "2026-07-04T17:06:26.69Z" },- { url = "https://files.pythonhosted.org/packages/83/d0/a39b2fbcde9cb17a1dac678f254b33a6336298af9df338824c685425d5e8/numpy-2.5.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e824c2acf8862052246be5a44c15da1777940c60d010dd2aab897824d9c430f9", size = 18431940, upload-time = "2026-07-04T17:06:29.521Z" },- { url = "https://files.pythonhosted.org/packages/04/12/cff070947791c1ed425ff76413189adbdc2fbe215eba7ce7fa454a03c7f8/numpy-2.5.1-cp312-cp312-win32.whl", hash = "sha256:08d60c810432eb83360958dea0999ac4cfb94531ea8efcbf0b7f277c2068aeb2", size = 6066764, upload-time = "2026-07-04T17:06:32.571Z" },- { url = "https://files.pythonhosted.org/packages/65/66/53f31807a48a750f9d748da273bc3fcedd12b27ff1f3e373bfec55ef2dc0/numpy-2.5.1-cp312-cp312-win_amd64.whl", hash = "sha256:f7d60026c0bdb1380e83bfa7a0419c4577ee4b9a08880afcb6dadeb74c649fa2", size = 12430966, upload-time = "2026-07-04T17:06:34.926Z" },- { url = "https://files.pythonhosted.org/packages/2b/2a/d1a88066b1c14186f5d3c0d18c94f17b064511982bab0578d49ee9d43c29/numpy-2.5.1-cp312-cp312-win_arm64.whl", hash = "sha256:17a25e09640602e10bc8de0e6fa2b3fd68eedd84ba6d7842dc8f32f9ab87bd0b", size = 10350488, upload-time = "2026-07-04T17:06:37.785Z" },- { url = "https://files.pythonhosted.org/packages/eb/07/ec2a3f0c91761581d4b7104a740791800025983f9a4dc4e73f91a99aeac4/numpy-2.5.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:0bfebd8695f9863592fe744be833a258120b14a9f39da255e8aa8fade2c0ddd1", size = 16796419, upload-time = "2026-07-04T17:06:40.37Z" },- { url = "https://files.pythonhosted.org/packages/ab/ab/ddb499fc4f8780354395face5b65c7fd107bcd6e1d667a5f07d046956f6f/numpy-2.5.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:30b44a6b53a7ae63c54c089a8726e5563ed302716c5b7ccc85afade40b0e7ff6", size = 11765832, upload-time = "2026-07-04T17:06:42.768Z" },- { url = "https://files.pythonhosted.org/packages/88/b3/3c28c558a09fc72100c646dac6d2fce8e834c471b0edca01a29996706117/numpy-2.5.1-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:6165343f81b56ef8f514f396989e529b61d9dc709b99421b07e9f3e698e2287d", size = 5325143, upload-time = "2026-07-04T17:06:45.466Z" },- { url = "https://files.pythonhosted.org/packages/5e/0e/ce19b985bb15c596f4f05954e76cccc77c845083b3b8f938a6c68e523128/numpy-2.5.1-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:4939237038ada79308dda3204ac6462df056b5672b2e25db1149cf873668b3e1", size = 6659749, upload-time = "2026-07-04T17:06:47.288Z" },- { url = "https://files.pythonhosted.org/packages/2e/20/1ee6614d64332a1bba6411f38e68cb79eec1b2459e20a623777c5c5492a2/numpy-2.5.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1c6759f538fb912fc46de0a6b1758ccf7b57bc7c7ebebc23974fdac3de8db0cd", size = 15164716, upload-time = "2026-07-04T17:06:49.494Z" },- { url = "https://files.pythonhosted.org/packages/ed/a7/2bcd3fdbb87804755c35b729bf8709d62025c5f4cfd7d5b2415997097515/numpy-2.5.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9726558e8db4a5bf7929a70ae50f63abda4daf0efe810e3bfbab95976f75fc1a", size = 16661440, upload-time = "2026-07-04T17:06:52.061Z" },- { url = "https://files.pythonhosted.org/packages/fc/d7/a41e3310c886fe457d36e670bbf24fae411aca8a7b6ad92a32afd924077c/numpy-2.5.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3935f3b419b244a02732676fa5317a9193cc596a4c0646db07e5b421229ac9f7", size = 16526305, upload-time = "2026-07-04T17:06:54.605Z" },- { url = "https://files.pythonhosted.org/packages/53/75/4333a9a707c1edd3a4e1a0c58eca52c0f31e55089fa80db02b5565b24df7/numpy-2.5.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:dc932a65ded7ce9013d120845a2514dcccb1a67bfc8deb8d37633762951904a6", size = 18423008, upload-time = "2026-07-04T17:06:57.54Z" },- { url = "https://files.pythonhosted.org/packages/ee/90/e314a32b1c11a2ffe818ddad3a57b50b4b6e1b6c487192eb50cdef0415d0/numpy-2.5.1-cp313-cp313-win32.whl", hash = "sha256:4b4ff1608417eb7a59da7b967bbb798cacfe071d2caf526a24281cd562072ed9", size = 6063885, upload-time = "2026-07-04T17:07:00.14Z" },- { url = "https://files.pythonhosted.org/packages/10/70/800b3fca480af32df9e8ea9f3d4a0c8feb4b32d7f195d174eabbda4829ad/numpy-2.5.1-cp313-cp313-win_amd64.whl", hash = "sha256:6c3fe51bc6a16453d452997053454f309e8e0ed7b42d6b361ce4ac8c32913d74", size = 12425674, upload-time = "2026-07-04T17:07:02.387Z" },- { url = "https://files.pythonhosted.org/packages/8b/0b/196350c122f50f6ca56846f2d71efd5e0d24b7b2e07355e019b2e2c7a11e/numpy-2.5.1-cp313-cp313-win_arm64.whl", hash = "sha256:f7feb014281029e628ba2d5a007407443b06e418b6fe451d1e2adcbc8eba0107", size = 10350256, upload-time = "2026-07-04T17:07:04.878Z" },-]-[[package]]name = "oauthlib"-version = "3.3.1"+version = "4.0.0"source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/0b/5f/19930f824ffeb0ad4372da4812c50edbd1434f678c90c2733e1188edfc63/oauthlib-3.3.1.tar.gz", hash = "sha256:0f0f8aa759826a193cf66c12ea1af1637f87b9b4622d46e866952bb022e538c9", size = 185918, upload-time = "2025-06-19T22:48:08.269Z" }+sdist = { url = "https://files.pythonhosted.org/packages/7a/d8/a1bcc8ba112a627f8ffbdc212a78ce18d3ac07e91a5ca65d27918eee25a1/oauthlib-4.0.0.tar.gz", hash = "sha256:efb274799819440f95b4ab3b818869f1ce9ae26c5beacba0201d1a1b76b54f86", size = 187232, upload-time = "2026-09-28T06:01:18.77Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/be/9c/92789c596b8df838baa98fa71844d84283302f7604ed565dafe5a6b5041a/oauthlib-3.3.1-py3-none-any.whl", hash = "sha256:88119c938d2b8fb88561af5f6ee0eec8cc8d552b7bb1f712743136eb7523b7a1", size = 160065, upload-time = "2025-06-19T22:48:06.508Z" },+ { url = "https://files.pythonhosted.org/packages/d9/f4/78229a1066068ca14fc60fb26cf7381cabe4382261392b90e5f9552722d4/oauthlib-4.0.0-py3-none-any.whl", hash = "sha256:624c28c13a0a59cabf9747dfa52af63be3e512a7f2714df16e91b5b3a145e6cd", size = 159715, upload-time = "2026-09-28T06:01:17.008Z" },][[package]]@@ -1947,6 +1901,7 @@ all = [{ name = "harbor", extra = ["gke"] },{ name = "litellm" },{ name = "mcp" },+ { name = "oauthlib" },{ name = "openai" },{ name = "pip-audit" },{ name = "pyjwt", extra = ["crypto"] },@@ -1982,6 +1937,7 @@ tier3 = [{ name = "harbor", extra = ["gke"] },{ name = "litellm" },{ name = "mcp" },+ { name = "oauthlib" },{ name = "openai" },{ name = "pyjwt", extra = ["crypto"] },{ name = "python-dotenv" },@@ -1994,13 +1950,14 @@ requires-dist = [{ name = "boto3", marker = "extra == 'llm'", specifier = ">=1.34.0" },{ name = "build", marker = "extra == 'dev'", specifier = ">=1.2" },{ name = "click", specifier = ">=8.3.3,<9" },- { name = "harbor", extras = ["gke"], marker = "extra == 'tier3'", specifier = "==0.13.2" },+ { name = "harbor", extras = ["gke"], marker = "extra == 'tier3'", specifier = "==0.24.0" },{ name = "httpx", marker = "extra == 'dev'", specifier = ">=0.27" },{ name = "idna", specifier = ">=3.10,<4" },{ name = "jinja2", specifier = ">=3.1" },{ name = "litellm", marker = "extra == 'llm'", specifier = ">=1.92.0,<1.94.0.dev0" },{ name = "markdown-it-py", specifier = ">=4.2,<5" },{ name = "mcp", marker = "extra == 'tier3'", specifier = ">=1.28.1,<2" },+ { name = "oauthlib", marker = "extra == 'tier3'", specifier = ">=4.0.0" },{ name = "openai", marker = "extra == 'llm'", specifier = ">=2.21.0" },{ name = "packaging", specifier = ">=24" },{ name = "pip-audit", marker = "extra == 'security'", specifier = ">=2.10.0" },