Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): isolate agent-visible evaluation fixtures
Christopher Kevin4fe6bcdbd4dd
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 8a89dd5b..ff23ffeb 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -77,7 +77,11 @@ All notable changes to SkillEvaluator are documented in this file.- Tier 3 generated tasks now stage only an entry's declared `files`, preventingundeclared fixtures from the shared `evals/files/` directory from appearingin that task's `/workspace/input/`, while preserving copy-all behavior for- legacy entries that omit the field.+ legacy entries that omit the field. Agent-visible target, reference, and+ workspace skill projections now omit evaluator-owned `evals/` directories+ from every staged skill package, including sanitized `--copy-repo` contexts,+ while graders, native tasks, custom+ environments, and declared inputs continue to load from the source dataset.- Public benchmark cards now omit policy profiles, redact absolute host paths,and normalize imported internal or retired metadata before publication.- Previous-version validation now rejects catalog-wide scalar reuse and removaldiff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex cbeae438..c7e834ea 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -94,6 +94,34 @@ declared paths are staged into that task's `/workspace/input/`. An explicitempty list or `null` stages no task input. For backward compatibility, omittingthe `files` field stages the entire shared `evals/files/` directory.+An `evals/` directory directly owned by a skill package is evaluator-only. Tier+3 reads datasets, graders, native Harbor tasks, and custom environments from the+source tree, but does not install those directories from the target or any+nested, reference, or workspace skill into agent-visible copies. `--copy-repo`+uses the same filtering and excludes generated result roots. Runtime assets that+the agent needs should live outside skill-owned `evals/`, or be declared as task+inputs with `files`.++`evals/environment/input/` is reserved and is not copied into generated tasks.+Directory-level custom Docker `COPY input/` remains buildable for an empty entry+using an empty context directory. Missing, dynamic, linked, hardlinked, aliased,+or escaping input sources fail closed. Native Harbor tasks and Compose input+contexts use the same omitted-versus-explicit selection rules.++The final evaluator layer replaces `/workspace/input`, `/workspace/repo`, and+supported agent skill-discovery roots. Custom Docker `preserve` mode also+requires a statically known final `USER` so the projection can be installed and+the original user restored. Custom Dockerfiles and base images remain trusted+executable environment inputs: these controls isolate evaluation fixtures and+supported discovery roots; they are not a security boundary against an image+that intentionally changes system binaries or startup behavior.++Evaluator-owned in-skill output uses a path-bound authenticity marker backed by+a local key. A copied, forged, or stale marker cannot authorize replacement at+another path. A custom key path set through+`SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE` must remain outside every evaluated+skill source.+### Run the comparison```bash title="With-skill vs. without-skill in Docker"diff --git a/src/skillevaluator/tier3/case_ids.py b/src/skillevaluator/tier3/case_ids.pyindex 458edf64..883d73f9 100644--- a/src/skillevaluator/tier3/case_ids.py+++ b/src/skillevaluator/tier3/case_ids.py@@ -39,8 +39,6 @@ def validate_case_id(value: object) -> str:raise ValueError("case id must not be an absolute path")if "/" in case_id or "\\" in case_id:raise ValueError("case id must be one path component and must not contain '/' or '\\'")- if ".." in case_id:- raise ValueError("case id must not contain '..'")if not _CASE_ID_PATTERN.fullmatch(case_id):raise ValueError("case id must start with an ASCII letter or digit and contain only ASCII letters, digits, '.', '_', or '-'"@@ -135,6 +133,11 @@ def _validate_directory_components(path: Path, *, case_id: str) -> None:raise ValueError(f"refusing case id {case_id!r}: output directory component is not a directory")+def validate_output_directory_path(path: Path) -> None:+ """Reject link/reparse components before creating an evaluator output tree."""+ _validate_directory_components(path, case_id="<output>")++def safe_child(base: Path, component: object) -> Path:"""Return a validated strict child of *base*, rejecting symlink escapes."""case_id = validate_case_id(component)diff --git a/src/skillevaluator/tier3/harbor/adapter.py b/src/skillevaluator/tier3/harbor/adapter.pyindex 8cbc951c..6e90a157 100644--- a/src/skillevaluator/tier3/harbor/adapter.py+++ b/src/skillevaluator/tier3/harbor/adapter.py@@ -23,15 +23,25 @@ import posixpathimport reimport shleximport shutil+import statimport subprocessimport tempfileimport tomllib+from collections.abc import Sequencefrom pathlib import Pathfrom typing import Anyfrom urllib.parse import unquote-from skillevaluator.tier3.case_ids import safe_child, validate_case_ids-from skillevaluator.tier3.harbor.secure_copy import copytree_secure+from skillevaluator.tier3.case_ids import safe_child, validate_case_ids, validate_output_directory_path+from skillevaluator.tier3.harbor.secure_copy import copy_file_secure, copytree_secure+from skillevaluator.tier3.output_provenance import (+ GENERATED_OUTPUT_MARKER,+ is_generated_output_root,+ output_provenance_key_path,+ validate_generated_output_replacement,+ validate_provenance_key_outside,+ write_generated_output_marker,+)from skillevaluator.tier3.toml_utils import toml_quotefrom skillevaluator.utils.process_environment import child_process_env@@ -220,6 +230,34 @@ def _find_repo_root(path: Path) -> Path | None:return None+def _repo_context_root(path: Path) -> Path:+ """Return the exact source root used by repo-context staging."""+ try:+ repo_root = _find_repo_root(path)+ resolved = path.resolve()+ return repo_root or resolved.parent+ except (OSError, RuntimeError) as exc:+ raise ValueError(f"Cannot resolve repository context root for: {path}") from exc+++def validate_output_provenance_key_location(+ skill_path: Path,+ output_dir: Path,+ *,+ reference_skills_dir: Path | None,+ workspace_skill_paths: Sequence[Path],+) -> None:+ """Keep the private marker key out of every tree that may reach an agent."""+ protected_roots = [skill_path, output_dir, *workspace_skill_paths]+ if reference_skills_dir is not None:+ protected_roots.append(reference_skills_dir)+ repo_root = _find_repo_root(skill_path)+ if repo_root is not None:+ protected_roots.append(repo_root)+ for protected_root in protected_roots:+ validate_provenance_key_outside(protected_root)++def _is_relative_to(path: Path, parent: Path) -> bool:try:path.resolve().relative_to(parent.resolve())@@ -228,6 +266,398 @@ def _is_relative_to(path: Path, parent: Path) -> bool:return False+_SKILL_MANIFEST_NAMES = ("SKILL.md", "skill.md")+++def _skill_manifest(path: Path) -> Path | None:+ for name in _SKILL_MANIFEST_NAMES:+ candidate = path / name+ if candidate.is_file():+ return candidate+ return None+++def _iter_skill_manifests(root: Path) -> list[Path]:+ manifests: set[Path] = set()+ for name in _SKILL_MANIFEST_NAMES:+ manifests.update(root.rglob(name))+ return sorted(manifests)+++def _find_target_manifest_payload(+ root: Path,+ target_skill: Path,+ *,+ runtime_projection: bool = False,+ excluded_roots: Sequence[Path] = (),+ ignored_parts: frozenset[str] = frozenset(),+) -> Path | None:+ """Find a renamed file containing the exact evaluated-skill instructions."""+ target_manifest = _skill_manifest(target_skill)+ if target_manifest is None:+ return None+ target_payload = target_manifest.read_bytes()+ runtime_ignore = _runtime_skill_copy_ignore(root, excluded_roots) if runtime_projection else None+ for candidate in root.rglob("*"):+ relative_parts = candidate.relative_to(root).parts+ if ignored_parts.intersection(relative_parts):+ continue+ if runtime_ignore is not None and _runtime_projection_path_is_ignored(candidate, root, runtime_ignore):+ continue+ try:+ metadata = candidate.lstat()+ except OSError:+ continue+ if stat.S_ISREG(metadata.st_mode) and metadata.st_nlink == 1 and metadata.st_size >= len(target_payload):+ overlap = max(0, len(target_payload) - 1)+ previous = b""+ try:+ with candidate.open("rb") as stream:+ while chunk := stream.read(64 * 1024):+ combined = previous + chunk+ if target_payload in combined:+ return candidate+ previous = combined[-overlap:] if overlap else b""+ except OSError:+ continue+ return None+++def _path_is_excluded(path: Path, excluded_roots: Sequence[Path]) -> bool:+ """Check lexical and resolved containment using cross-platform path casing."""++ def _casefold_contained(candidate: Path, root: Path) -> bool:+ candidate_parts = tuple(part.casefold() for part in candidate.parts)+ root_parts = tuple(part.casefold() for part in root.parts)+ return len(candidate_parts) >= len(root_parts) and candidate_parts[: len(root_parts)] == root_parts++ for excluded_root in excluded_roots:+ if _casefold_contained(path.absolute(), excluded_root.absolute()):+ return True+ try:+ resolved_path = path.resolve()+ resolved_root = excluded_root.resolve()+ except (OSError, RuntimeError):+ continue+ if _casefold_contained(resolved_path, resolved_root):+ return True+ return False+++def _paths_equivalent(first: Path, second: Path) -> bool:+ """Return whether two paths identify the same cross-platform path."""+ return _path_is_excluded(first, (second,)) and _path_is_excluded(second, (first,))+++def _path_is_link_or_reparse(path: Path, metadata: os.stat_result | None = None) -> bool:+ """Return whether a path is a symlink, junction, or Windows reparse point."""+ if metadata is None:+ try:+ metadata = path.lstat()+ except FileNotFoundError:+ return False+ reparse_flag = getattr(stat, "FILE_ATTRIBUTE_REPARSE_POINT", 0x400)+ is_junction = getattr(path, "is_junction", None)+ return (+ stat.S_ISLNK(metadata.st_mode)+ or bool(getattr(metadata, "st_file_attributes", 0) & reparse_flag)+ or (callable(is_junction) and is_junction())+ )+++def _path_is_canonically_contained(path: Path, root: Path) -> bool:+ """Require exact-case lexical and resolved containment for safety exceptions."""+ for candidate, canonical_root in (+ (path.absolute(), root.absolute()),+ (path.resolve(), root.resolve()),+ ):+ try:+ candidate.relative_to(canonical_root)+ except ValueError:+ return False+ return True+++def _inside_declared_generated_dataset(path: Path, declared_roots: Sequence[Path]) -> bool:+ """Return whether *path* is below a staged dataset marker in a declared output root."""+ for declared_root in declared_roots:+ if not _path_is_excluded(path, (declared_root,)):+ continue+ current = path if path.is_dir() else path.parent+ while _path_is_excluded(current, (declared_root,)):+ if is_generated_output_root(current):+ return True+ if _paths_equivalent(current, declared_root):+ break+ current = current.parent+ return False+++def _target_runtime_source_roots(+ skill_path: Path,+ declared_output_roots: Sequence[Path] = (),+) -> tuple[Path, ...]:+ """Return standard and nested-package runtime subtrees that output cannot replace."""+ roots = {skill_path / name for name in ("assets", "references", "scripts", "templates")}+ default_results_root = skill_path / "evals" / "results"+ for manifest in _iter_skill_manifests(skill_path):+ if _path_is_canonically_contained(manifest, default_results_root):+ continue+ owner = manifest.parent+ if not _paths_equivalent(owner, skill_path) and not _inside_declared_generated_dataset(+ manifest, declared_output_roots+ ):+ roots.add(owner)+ return tuple(sorted(roots))+++def _validate_output_roots_outside_evaluator_sources(+ skill_path: Path,+ output_roots: Sequence[Path],+) -> None:+ """Reject output roots that would be copied back as evaluator source data."""+ evals_dir = skill_path / "evals"+ results_dir = evals_dir / "results"+ for output_root in output_roots:+ if not _path_is_excluded(output_root, (skill_path,)):+ continue+ if _path_is_excluded(output_root, (evals_dir,)) and not _path_is_canonically_contained(+ output_root, results_dir+ ):+ raise ValueError(+ f"Generated output root must not be inside evaluator source directory '{evals_dir}': {output_root}"+ )+ if _paths_equivalent(output_root, skill_path):+ raise ValueError(f"Generated output root must not equal the runtime skill root: {skill_path}")+ for source_root in _target_runtime_source_roots(skill_path, output_roots):+ if _path_is_excluded(output_root, (source_root,)) or _path_is_excluded(source_root, (output_root,)):+ raise ValueError(+ f"Generated output root must not overlap runtime skill source '{source_root}': {output_root}"+ )+++def _validate_staging_output_location(+ skill_path: Path,+ output_dir: Path,+ *,+ reference_skills_dir: Path | None = None,+ workspace_skill_paths: Sequence[Path] = (),+ declared_output_roots: Sequence[Path] = (),+) -> None:+ """Reject a staging destination that could delete evaluator source data."""+ validate_output_directory_path(output_dir)+ evals_dir = skill_path / "evals"+ results_dir = evals_dir / "results"+ output_is_results = _path_is_canonically_contained(output_dir, results_dir)+ if not output_is_results and (+ _path_is_excluded(output_dir, (evals_dir,)) or _path_is_excluded(evals_dir, (output_dir,))+ ):+ raise ValueError(f"Staging output directory overlaps evaluator source directory '{evals_dir}': {output_dir}")+ for source_root in _target_runtime_source_roots(skill_path, declared_output_roots):+ if _path_is_excluded(output_dir, (source_root,)) or _path_is_excluded(source_root, (output_dir,)):+ raise ValueError(f"Staging output directory overlaps runtime skill source '{source_root}': {output_dir}")+ output_inside_skill = _path_is_excluded(output_dir, (skill_path,))+ output_contains_skill = _path_is_excluded(skill_path, (output_dir,))+ declared_in_skill = any(+ _path_is_excluded(output_dir, (declared_root,))+ and _path_is_excluded(declared_root, (skill_path,))+ and not _paths_equivalent(declared_root, skill_path)+ for declared_root in declared_output_roots+ )+ if output_contains_skill or (output_inside_skill and not output_is_results and not declared_in_skill):+ raise ValueError(f"Staging output directory overlaps runtime skill source '{skill_path}': {output_dir}")++ runtime_sources = [*(workspace_skill_paths or [])]+ if reference_skills_dir is not None:+ runtime_sources.append(reference_skills_dir)+ for source_root in runtime_sources:+ if _path_is_excluded(skill_path, (source_root,)) and _path_is_excluded(output_dir, (skill_path,)):+ continue+ if _path_is_excluded(output_dir, (source_root,)) or _path_is_excluded(source_root, (output_dir,)):+ raise ValueError(f"Staging output directory overlaps runtime skill source '{source_root}': {output_dir}")+++def validate_results_root_location(+ skill_path: Path,+ results_root: Path,+ *,+ reference_skills_dir: Path | None = None,+ workspace_skill_paths: Sequence[Path] = (),+) -> None:+ """Validate that a result root cannot become evaluator input or build context."""+ validate_output_directory_path(results_root)+ if _path_is_excluded(skill_path, (results_root,)):+ raise ValueError(f"Generated output root must not contain the runtime skill source: {results_root}")+ _validate_output_roots_outside_evaluator_sources(skill_path, (results_root,))+ runtime_sources = [*(workspace_skill_paths or [])]+ if reference_skills_dir is not None:+ runtime_sources.append(reference_skills_dir)+ for source_root in runtime_sources:+ if _path_is_excluded(skill_path, (source_root,)) and _path_is_excluded(results_root, (skill_path,)):+ continue+ if _path_is_excluded(results_root, (source_root,)):+ raise ValueError(+ f"Generated output root must not be inside runtime skill source '{source_root}': {results_root}"+ )+++def _skill_scoped_excluded_roots(skill_root: Path, excluded_roots: Sequence[Path]) -> tuple[Path, ...]:+ """Keep only generated-output roots located inside this skill tree."""+ scoped: list[Path] = []+ for excluded_root in excluded_roots:+ if _paths_equivalent(excluded_root, skill_root):+ raise ValueError(f"Generated output root must not equal the runtime skill root: {skill_root}")+ if _path_is_excluded(excluded_root, (skill_root,)):+ scoped.append(excluded_root)+ return tuple(scoped)+++def _is_skill_evals_path(path: Path, skill_root: Path) -> bool:+ """Return whether *path* is under ``evals/`` owned by any nested skill package."""+ for candidate, root in (+ (path.absolute(), skill_root.absolute()),+ (path.resolve(), skill_root.resolve()),+ ):+ try:+ rel = candidate.relative_to(root)+ except ValueError:+ continue+ for index, part in enumerate(rel.parts):+ if part.casefold() != "evals":+ continue+ owner = root.joinpath(*rel.parts[:index])+ if _skill_manifest(owner) is not None and (owner / part).is_dir():+ return True+ return False+++def _runtime_skill_copy_ignore(skill_root: Path, excluded_roots: Sequence[Path] = ()):+ """Build a root-aware ignore callback for an agent-visible skill copy."""+ resolved_root = skill_root.resolve()+ excluded_roots = _skill_scoped_excluded_roots(skill_root, excluded_roots)++ def _ignore(directory: str, contents: list[str]) -> list[str]:+ current = Path(directory)+ ignored = {name for name in contents if name in {"results", "__pycache__", ".git"}}+ if current.resolve() == resolved_root:+ ignored.update(name for name in contents if name.casefold() == "evals")+ for name in contents:+ candidate = current / name+ if _is_skill_evals_path(candidate, skill_root) or _path_is_excluded(candidate, excluded_roots):+ ignored.add(name)+ return sorted(ignored)++ return _ignore+++def _runtime_projection_path_is_ignored(path: Path, root: Path, ignore) -> bool:+ """Apply a copytree ignore callback to every component of one source path."""+ current = root+ for part in path.relative_to(root).parts:+ if part in ignore(str(current), [part]):+ return True+ current /= part+ return False+++def _runtime_skill_fingerprint(skill_root: Path, excluded_roots: Sequence[Path] = ()) -> str | None:+ """Hash the agent-visible runtime tree independent of its directory name."""+ if _skill_manifest(skill_root) is None:+ return None+ ignore = _runtime_skill_copy_ignore(skill_root, excluded_roots)+ digest = hashlib.sha256()+ pending = [skill_root]+ while pending:+ directory = pending.pop()+ entries = sorted(os.scandir(directory), key=lambda entry: entry.name)+ ignored = set(ignore(str(directory), [entry.name for entry in entries]))+ for entry in entries:+ if entry.name in ignored:+ continue+ path = Path(entry.path)+ metadata = entry.stat(follow_symlinks=False)+ if _path_is_link_or_reparse(path, metadata):+ return None+ relative = path.relative_to(skill_root).as_posix().encode("utf-8")+ if stat.S_ISDIR(metadata.st_mode):+ digest.update(b"D\\0" + relative + b"\\0")+ pending.append(path)+ elif stat.S_ISREG(metadata.st_mode):+ if metadata.st_nlink != 1:+ return None+ digest.update(b"F\\0" + relative + b"\\0")+ with path.open("rb") as handle:+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):+ digest.update(chunk)+ else:+ return None+ return digest.hexdigest()+++def _check_baseline_skill_candidates_do_not_alias_target(+ target_skill: Path,+ reference_skills_dir: Path | None,+ workspace_skill_paths: list[Path] | None,+ *,+ excluded_roots: Sequence[Path] = (),+) -> None:+ target_fingerprint = _runtime_skill_fingerprint(target_skill, excluded_roots)+ target_manifest = _skill_manifest(target_skill)+ if target_fingerprint is None or target_manifest is None:+ return+ target_manifest_digest = hashlib.sha256(target_manifest.read_bytes()).digest()+ candidates: list[Path] = []+ if reference_skills_dir and reference_skills_dir.exists():+ candidates.extend(+ candidate+ for candidate in reference_skills_dir.iterdir()+ if candidate.is_dir() and not candidate.name.startswith(".")+ )+ candidates.extend(workspace_skill_paths or [])+ seen: set[Path] = set()+ for candidate in candidates:+ if _paths_equivalent(candidate, target_skill):+ continue+ resolved = candidate.resolve()+ if resolved in seen or not candidate.is_dir():+ continue+ seen.add(resolved)+ candidate_manifest = _skill_manifest(candidate)+ manifest_matches = candidate_manifest is not None and (+ hashlib.sha256(candidate_manifest.read_bytes()).digest() == target_manifest_digest+ )+ payload_alias = _find_target_manifest_payload(+ candidate,+ target_skill,+ runtime_projection=True,+ excluded_roots=excluded_roots,+ )+ if (+ manifest_matches+ or payload_alias is not None+ or _runtime_skill_fingerprint(candidate, excluded_roots) == target_fingerprint+ ):+ raise ValueError(+ f"Baseline skill candidate '{candidate}' is an alias of target skill '{target_skill.name}'. "+ "A baseline must not contain the evaluated skill under another directory name."+ )+++def _check_staged_baseline_does_not_contain_target(env_dir: Path, target_skill: Path) -> None:+ """Fail closed if any agent-visible staged file is the evaluated instructions."""+ payload = _find_target_manifest_payload(env_dir, target_skill)+ if payload is not None:+ raise ValueError(+ f"Baseline environment contains the target skill instructions in {payload}. "+ "Remove copied or renamed target content from reference skills, task input, and repo context."+ )+++def _repo_path_is_skill_evals(path: Path, root: Path) -> bool:+ """Identify evaluator-only roots without dropping unrelated nested ``evals``."""+ return _is_skill_evals_path(path, root)++def _strip_link_target(raw_target: str) -> str:target = raw_target.strip()if not target:@@ -296,8 +726,21 @@ def _staged_relative_link_path(skill_name: str, raw_target: str) -> Path | None:if normalized in {"", "."} or normalized.startswith("../"):return Nonerel = Path(normalized)- # Do not let linked repo docs stage sibling skills as discoverable skills.- if rel.parts[:1] == ("skills",) and (len(rel.parts) < 2 or rel.parts[1] != skill_name):+ if rel.parts and rel.parts[0].casefold() in {"input", "output", "repo"}:+ raise ValueError(f"SKILL.md link resolves into reserved runtime path '/workspace/{rel.parts[0]}': {raw_target}")+ # Runtime skills and agent configuration are projected separately after+ # recursive evaluator-only filtering. Linked repository content must never+ # create an alternate project-level discovery/configuration namespace.+ if rel.parts and rel.parts[0].casefold() in {+ "skills",+ ".agents",+ ".claude",+ ".codex",+ ".config",+ ".cursor",+ ".gemini",+ ".opencode",+ }:return Nonereturn rel@@ -307,6 +750,10 @@ def _repo_context_ignore_file(path: Path, root: Path) -> bool:path.resolve().relative_to(root.resolve())except ValueError:return True+ if _paths_equivalent(path, output_provenance_key_path()):+ return True+ if _repo_path_is_skill_evals(path, root):+ return Truetry:rel = path.absolute().relative_to(root.absolute())except ValueError:@@ -331,7 +778,14 @@ def _repo_context_ignore_file(path: Path, root: Path) -> bool:return False-def _copy_repo_context_file(src: Path, dest_root: Path, rel: Path, *, total_bytes: list[int]) -> bool:+def _copy_repo_context_file(+ src: Path,+ dest_root: Path,+ rel: Path,+ *,+ total_bytes: list[int],+ source_root: Path,+) -> bool:if not src.is_file():return Falsetry:@@ -346,7 +800,7 @@ def _copy_repo_context_file(src: Path, dest_root: Path, rel: Path, *, total_bytereturn Falsedest = dest_root / reldest.parent.mkdir(parents=True, exist_ok=True)- shutil.copy2(src, dest)+ copy_file_secure(src, dest, allowed_root=source_root)total_bytes[0] += sizereturn True@@ -380,16 +834,28 @@ def _stage_repo_context(source_skill_path: Path | None,mode: str,exclude_source_skill: bool = False,+ excluded_roots: Sequence[Path] = (),) -> dict[str, Any]:"""Stage repo files referenced by SKILL.md, or the full repo when requested."""if not source_skill_path:return {"mode": "none", "files": []}source_skill_path = source_skill_path.resolve()- skill_md = source_skill_path / "SKILL.md"- repo_root = _find_repo_root(source_skill_path)- if repo_root is None:- repo_root = source_skill_path.parent.resolve()+ skill_md = _skill_manifest(source_skill_path)+ repo_root = _repo_context_root(source_skill_path)++ resolved_excluded_roots: list[Path] = []+ for excluded_root in excluded_roots:+ candidate = excluded_root.resolve()+ if mode == "full" and _paths_equivalent(candidate, repo_root):+ raise ValueError(f"Repo context output root must not contain the repository root: {candidate}")+ if _path_is_excluded(repo_root, (candidate,)):+ continue+ if _path_is_excluded(candidate, (repo_root,)):+ resolved_excluded_roots.append(candidate)++ def _is_excluded_output(path: Path) -> bool:+ return _path_is_excluded(path, resolved_excluded_roots)repo_dest = env_dir / "repo"linked_root_dest = env_dir / "repo-linked-root"@@ -398,6 +864,8 @@ def _stage_repo_context(if mode == "full":for src in _iter_repo_context_files(repo_root):+ if _is_excluded_output(src):+ continueif _repo_context_ignore_file(src, repo_root):continueif exclude_source_skill and _is_relative_to(src, source_skill_path):@@ -405,25 +873,37 @@ def _stage_repo_context(rel = _safe_repo_context_rel(src, repo_root)if rel is None:continue- if _copy_repo_context_file(src, repo_dest, rel, total_bytes=total_bytes):+ if _copy_repo_context_file(+ src,+ repo_dest,+ rel,+ total_bytes=total_bytes,+ source_root=repo_root,+ ):staged.append({"source": str(src), "container": f"/workspace/repo/{rel.as_posix()}"})metadata = {"mode": "full", "repo_root": str(repo_root), "files": staged}if staged:(env_dir / "repo-context.json").write_text(json.dumps(metadata, indent=2), encoding="utf-8")return metadata- if mode != "linked" or not skill_md.exists():+ if mode != "linked" or skill_md is None:return {"mode": mode, "repo_root": str(repo_root), "files": []}for raw_target in _discover_skill_link_targets(skill_md):target = _strip_link_target(raw_target)- target_path = (source_skill_path / target).resolve() if not Path(target).is_absolute() else Path(target)+ lexical_target_path = source_skill_path / target if not Path(target).is_absolute() else Path(target)+ target_path = lexical_target_path.resolve()if not target_path.is_file():continueif not _is_relative_to(target_path, repo_root):logger.warning("Skipping SKILL.md link outside repo root: %s", raw_target)continue- if _repo_context_ignore_file(target_path, repo_root):+ if _is_excluded_output(target_path) or _path_is_excluded(lexical_target_path, resolved_excluded_roots):+ logger.warning("Skipping SKILL.md link into generated output: %s", raw_target)+ continue+ if _repo_context_ignore_file(lexical_target_path, repo_root) or _repo_context_ignore_file(+ target_path, repo_root+ ):logger.warning("Skipping ignored SKILL.md repo link: %s", raw_target)continueif _is_relative_to(target_path, source_skill_path):@@ -431,10 +911,22 @@ def _stage_repo_context(rel = _safe_repo_context_rel(target_path, repo_root)if rel is None:continue- copied = _copy_repo_context_file(target_path, repo_dest, rel, total_bytes=total_bytes)+ copied = _copy_repo_context_file(+ target_path,+ repo_dest,+ rel,+ total_bytes=total_bytes,+ source_root=repo_root,+ )compat_rel = _staged_relative_link_path(source_skill_path.name, raw_target)if compat_rel is not None:- _copy_repo_context_file(target_path, linked_root_dest, compat_rel, total_bytes=total_bytes)+ _copy_repo_context_file(+ target_path,+ linked_root_dest,+ compat_rel,+ total_bytes=total_bytes,+ source_root=repo_root,+ )if copied:staged.append({@@ -454,6 +946,8 @@ def _collect_all_skill_deps(skill_path: Path | None,reference_skills_dir: Path | None,workspace_skill_paths: list[Path] | None = None,+ *,+ excluded_roots: Sequence[Path] = (),) -> tuple[list[str], list[str]]:"""Collect pip and apt deps from target skill + staged workspace skills."""all_pip: set[str] = set()@@ -471,12 +965,31 @@ def _collect_all_skill_deps(skill_dirs.append(workspace_skill)for sd in skill_dirs:+ runtime_ignore = _runtime_skill_copy_ignore(sd, excluded_roots)for req_file in sd.rglob("requirements.txt"):+ if _runtime_projection_path_is_ignored(req_file, sd, runtime_ignore):+ continue+ metadata = req_file.lstat()+ if (+ _path_is_link_or_reparse(req_file, metadata)+ or not stat.S_ISREG(metadata.st_mode)+ or metadata.st_nlink != 1+ ):+ raise ValueError(f"Runtime dependency manifest must be a regular non-linked file: {req_file}")for line in req_file.read_text(encoding="utf-8").splitlines():line = line.strip()if line and not line.startswith("#"):all_pip.add(line)for apt_file in sd.rglob("apt-packages.txt"):+ if _runtime_projection_path_is_ignored(apt_file, sd, runtime_ignore):+ continue+ metadata = apt_file.lstat()+ if (+ _path_is_link_or_reparse(apt_file, metadata)+ or not stat.S_ISREG(metadata.st_mode)+ or metadata.st_nlink != 1+ ):+ raise ValueError(f"Runtime dependency manifest must be a regular non-linked file: {apt_file}")for line in apt_file.read_text(encoding="utf-8").splitlines():line = line.strip()if line and not line.startswith("#"):@@ -490,6 +1003,7 @@ def build_eval_base_image(reference_skills_dir: Path | None = None,*,workspace_skill_paths: list[Path] | None = None,+ excluded_roots: Sequence[Path] = (),force_rebuild: bool = False,action_out: list[str] | None = None,) -> str:@@ -513,9 +1027,19 @@ def build_eval_base_image()if has_custom_dockerfile:- extra_pip, extra_apt = _collect_all_skill_deps(None, reference_skills_dir, workspace_skill_paths)+ extra_pip, extra_apt = _collect_all_skill_deps(+ None,+ reference_skills_dir,+ workspace_skill_paths,+ excluded_roots=excluded_roots,+ )else:- extra_pip, extra_apt = _collect_all_skill_deps(skill_path, reference_skills_dir, workspace_skill_paths)+ extra_pip, extra_apt = _collect_all_skill_deps(+ skill_path,+ reference_skills_dir,+ workspace_skill_paths,+ excluded_roots=excluded_roots,+ )lines = ["FROM python:3.12-slim",@@ -529,20 +1053,7 @@ def build_eval_base_image(lines.append(" " + " ".join(extra_apt) + " \\")lines.append(" && rm -rf /var/lib/apt/lists/*")- lines.extend(- [- "",- "RUN pip install --no-cache-dir \\",- " ragas~=0.4.0 \\",- " openai>=1.0 \\",- " anthropic>=0.40 \\",- " boto3>=1.34",- ]- )-- if extra_pip:- escaped = " ".join(f'"{r}"' for r in extra_pip)- lines.append(f"RUN pip install --no-cache-dir {escaped}")+ lines.extend(["", *_verifier_install_lines(extra_pip)])lines.extend([@@ -628,12 +1139,32 @@ def _set_task_docker_image(task_dir: Path, image_tag: str) -> None:toml_path.write_text(replaced, encoding="utf-8")+def _task_environment_context_hash(environment_dir: Path) -> str:+ """Hash the Docker context shape, modes, paths, and file contents."""+ digest = hashlib.sha256()+ for path in sorted(+ environment_dir.rglob("*"), key=lambda candidate: candidate.relative_to(environment_dir).as_posix()+ ):+ relative = path.relative_to(environment_dir).as_posix()+ metadata = path.lstat()+ mode = stat.S_IMODE(metadata.st_mode)+ if stat.S_ISDIR(metadata.st_mode):+ digest.update(f"D\0{relative}\0{mode:o}\0".encode())+ elif stat.S_ISREG(metadata.st_mode):+ digest.update(f"F\0{relative}\0{mode:o}\0".encode())+ digest.update(path.read_bytes())+ digest.update(b"\0")+ else:+ raise ValueError(f"Task Docker context contains an unsupported path: {path}")+ return digest.hexdigest()++def prebuild_task_environments(dataset_dirs: list[Path]) -> int:"""Pre-build Docker images for generated task environments.After ``generate_harbor_tasks`` has written all task directories, this- function builds each unique environment once, tags it, and sets- ``docker_image`` in every task's ``task.toml`` so Harbor uses the prebuilt+ function builds each task's exact environment, tags it independently, and+ sets ``docker_image`` only in that task's ``task.toml`` so Harbor uses the prebuiltimage directly (skipping ``docker compose build`` entirely).Returns the number of tasks configured to use prebuilt images.@@ -662,71 +1193,161 @@ def prebuild_task_environments(dataset_dirs: list[Path]) -> int:if not task_dirs:continue- first_env = task_dirs[0] / "environment"-- ctx_hash = hashlib.sha256()- for f in sorted(first_env.rglob("*")):- if f.is_file():- ctx_hash.update(str(f.relative_to(first_env)).encode())- ctx_hash.update(f.read_bytes())- tag = f"skillevaluator-env:{ctx_hash.hexdigest()[:12]}"-- try:- check = subprocess.run(- ["docker", "image", "inspect", tag],- capture_output=True,- timeout=10,- env=child_process_env(),- )- if check.returncode == 0:- logger.debug("Pre-built env %s exists, reusing for %d tasks", tag, len(task_dirs))- for td in task_dirs:- _set_task_docker_image(td, tag)- rewritten += len(task_dirs)+ for task_dir in task_dirs:+ environment_dir = task_dir / "environment"+ if any((environment_dir / name).exists() for name in ("docker-compose.yaml", "docker-compose.yml")):+ logger.warning(+ "Skipping task environment pre-build for %s because Compose build overrides require native Compose semantics",+ task_dir,+ )+ continue+ try:+ context_hash = _task_environment_context_hash(environment_dir)+ except (OSError, ValueError) as error:+ logger.warning("Skipping unsafe task environment pre-build for %s: %s", task_dir, error)+ continue+ task_identity = hashlib.sha256(f"{dataset_dir.resolve()}::{task_dir.name}".encode()).hexdigest()[:8]+ tag = f"skillevaluator-env:{context_hash[:12]}-{task_identity}"++ try:+ check = subprocess.run(+ ["docker", "image", "inspect", tag],+ capture_output=True,+ timeout=10,+ env=child_process_env(),+ )+ if check.returncode == 0:+ logger.debug("Pre-built env %s exists, reusing for task %s", tag, task_dir.name)+ _set_task_docker_image(task_dir, tag)+ rewritten += 1+ continue+ except (subprocess.TimeoutExpired, OSError):+ pass++ logger.debug("Pre-building task environment as %s ...", tag)+ try:+ result = subprocess.run(+ ["docker", "build", "-t", tag, str(environment_dir)],+ capture_output=True,+ text=True,+ timeout=600,+ env=child_process_env(),+ )+ except subprocess.TimeoutExpired:+ logger.warning("Environment pre-build timed out for %s", tag)continue- except (subprocess.TimeoutExpired, OSError):- pass-- logger.debug("Pre-building task environment as %s ...", tag)- try:- result = subprocess.run(- ["docker", "build", "-t", tag, str(first_env)],- capture_output=True,- text=True,- timeout=600,- env=child_process_env(),- )- except subprocess.TimeoutExpired:- logger.warning("Environment pre-build timed out for %s", tag)- continue- if result.returncode != 0:- stderr_tail = (result.stderr or "")[-500:]- logger.warning("Environment pre-build failed for %s: %s", tag, stderr_tail)- continue+ if result.returncode != 0:+ stderr_tail = (result.stderr or "")[-500:]+ logger.warning("Environment pre-build failed for %s: %s", tag, stderr_tail)+ continue- for td in task_dirs:- _set_task_docker_image(td, tag)- rewritten += len(task_dirs)- logger.debug("Pre-built env %s, configured %d tasks", tag, len(task_dirs))+ _set_task_docker_image(task_dir, tag)+ rewritten += 1+ logger.debug("Pre-built env %s for task %s", tag, task_dir.name)return rewrittendef _load_evals(evals_path: Path) -> list[dict[str, Any]]:"""Load normalized dataset entries from evals.json/jsonl/yaml."""- from skillevaluator.tier3.dataset_utils import load_dataset_entries+ from skillevaluator.tier3.dataset_utils import normalize_dataset_entries++ payload = _read_regular_evals_file(evals_path).decode("utf-8")+ suffix = evals_path.suffix.casefold()+ if suffix == ".jsonl":+ data: Any = [json.loads(line) for raw_line in payload.splitlines() if (line := raw_line.strip())]+ elif suffix in {".yaml", ".yml"}:+ import yaml++ data = yaml.safe_load(payload)+ elif suffix == ".json":+ data = json.loads(payload)+ else:+ raise ValueError(f"Unsupported dataset format: {suffix}")+ return normalize_dataset_entries(data)- return load_dataset_entries(evals_path)++def _read_regular_evals_file(path: Path) -> bytes:+ """Read one evaluator file without following links or accepting swaps."""+ try:+ before = path.lstat()+ except OSError as exc:+ raise ValueError(f"Cannot inspect eval dataset file: {path}") from exc+ if _path_is_link_or_reparse(path, before) or not stat.S_ISREG(before.st_mode) or before.st_nlink != 1:+ raise ValueError(f"Eval dataset must be a regular non-linked file: {path}")+ if before.st_size > 64 * 1024 * 1024:+ raise ValueError(f"Eval dataset exceeds the 64 MiB safety limit: {path}")+ flags = os.O_RDONLY | getattr(os, "O_CLOEXEC", 0) | getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_NONBLOCK", 0)+ try:+ descriptor = os.open(path, flags)+ except OSError as exc:+ raise ValueError(f"Cannot safely open eval dataset file: {path}") from exc+ try:+ opened = os.fstat(descriptor)+ before_fingerprint = (before.st_dev, before.st_ino, before.st_mode, before.st_nlink, before.st_size)+ opened_fingerprint = (opened.st_dev, opened.st_ino, opened.st_mode, opened.st_nlink, opened.st_size)+ if (+ _path_is_link_or_reparse(path, opened)+ or not stat.S_ISREG(opened.st_mode)+ or opened.st_nlink != 1+ or opened_fingerprint != before_fingerprint+ ):+ raise ValueError(f"Eval dataset changed while it was opened: {path}")+ chunks: list[bytes] = []+ remaining = opened.st_size + 1+ while remaining:+ chunk = os.read(descriptor, min(1024 * 1024, remaining))+ if not chunk:+ break+ chunks.append(chunk)+ remaining -= len(chunk)+ payload = b"".join(chunks)+ after = os.fstat(descriptor)+ after_fingerprint = (after.st_dev, after.st_ino, after.st_mode, after.st_nlink, after.st_size)+ if after_fingerprint != opened_fingerprint or len(payload) != opened.st_size:+ raise ValueError(f"Eval dataset changed while it was read: {path}")+ return payload+ finally:+ os.close(descriptor)+++def _validate_evals_source_directory(skill_path: Path) -> None:+ """Reject a link/reparse-point evaluator root before loading any assets."""+ evals_dir = skill_path / "evals"+ try:+ metadata = evals_dir.lstat()+ except FileNotFoundError:+ return+ except OSError as exc:+ raise ValueError(f"Cannot inspect evaluator source directory: {evals_dir}") from exc+ if _path_is_link_or_reparse(evals_dir, metadata):+ raise ValueError(+ f"evals directory must be a real directory, not a symlink, reparse point, or junction: {evals_dir}"+ )+ if not stat.S_ISDIR(metadata.st_mode):+ raise ValueError(f"Evaluator source path must be a directory: {evals_dir}")def find_evals_file(skill_path: Path) -> Path | None:"""Return the first supported SkillEvaluator eval dataset for a skill, if present."""evals_dir = skill_path / "evals"+ _validate_evals_source_directory(skill_path)for name in ("evals.json", "evals.jsonl", "evals.yaml", "evals.yml", "dataset.json", "dataset.jsonl"):candidate = evals_dir / name- if candidate.exists():- return candidate+ try:+ metadata = candidate.lstat()+ except FileNotFoundError:+ continue+ except OSError as exc:+ raise ValueError(f"Cannot inspect eval dataset file: {candidate}") from exc+ if (+ _path_is_link_or_reparse(candidate, metadata)+ or not stat.S_ISREG(metadata.st_mode)+ or metadata.st_nlink != 1+ ):+ raise ValueError(f"Eval dataset must be a regular non-linked file: {candidate}")+ return candidatereturn None@@ -869,8 +1490,7 @@ def _toml_value(value: Any) -> str:def _runtime_env_toml_block(runtime_env: dict[str, str] | None) -> str:- if not runtime_env:- return ""+ runtime_env = {**(runtime_env or {}), **_EVALUATOR_MANAGED_RUNTIME_ENV}lines = ["", "[environment.env]"]for key in sorted(runtime_env):lines.append(f"{_toml_quote(key)} = {_toml_quote(runtime_env[key])}")@@ -960,7 +1580,7 @@ def _has_symlink_component(path: Path, root: Path) -> bool:current = rootfor part in path.relative_to(root).parts:current /= part- if current.is_symlink():+ if _path_is_link_or_reparse(current):return Truereturn False@@ -1029,14 +1649,780 @@ def _validate_custom_dockerfile(path: Path) -> str | None:return None+_DOCKER_HEREDOC_RE = re.compile(+ r"<<(?P<strip>-?)(?:(?P<quote>['\"])(?P<quoted>[^'\"\s]+)(?P=quote)|(?P<bare>[^\s'\"<>]+))(?=\s|$)"+)+++def _dockerfile_heredocs(line: str) -> list[tuple[str, bool]]:+ """Return heredoc delimiters introduced by one Dockerfile instruction."""+ heredocs: list[tuple[str, bool]] = []+ quote: str | None = None+ index = 0+ while index < len(line):+ character = line[index]+ if quote is not None:+ if character == quote:+ quote = None+ elif character == "\\" and quote == '"':+ index += 1+ index += 1+ continue+ if character in {"'", '"'}:+ quote = character+ index += 1+ continue+ if character == "\\":+ index += 2+ continue+ if character == "<" and (index == 0 or line[index - 1].isspace()):+ match = _DOCKER_HEREDOC_RE.match(line, index)+ if match is not None:+ delimiter = match.group("quoted") or match.group("bare")+ heredocs.append((delimiter, match.group("strip") == "-"))+ index = match.end()+ continue+ index += 1+ return heredocs+++def _dockerfile_logical_lines_from_content(content: str, *, source: str = "Dockerfile") -> list[str]:+ """Return Dockerfile instructions after applying its escape directive."""+ raw_lines = content.splitlines()+ escape_char = "\\"+ for raw_line in raw_lines:+ directive = re.match(r"^\s*#\s*escape\s*=\s*([`\\])\s*$", raw_line, flags=re.IGNORECASE)+ if directive:+ escape_char = directive.group(1)+ break+ if raw_line.strip() and not raw_line.lstrip().startswith("#"):+ break++ logical_lines: list[str] = []+ pending = ""+ active_heredocs: list[tuple[str, bool]] = []+ for raw_line in raw_lines:+ if active_heredocs:+ delimiter, strip_tabs = active_heredocs[0]+ candidate = raw_line.lstrip("\t") if strip_tabs else raw_line+ if candidate == delimiter:+ active_heredocs.pop(0)+ continue+ line = raw_line.strip()+ if not line or (line.startswith("#") and not pending):+ continue+ pending = f"{pending} {line}".strip()+ trailing_escapes = len(pending) - len(pending.rstrip(escape_char))+ if trailing_escapes % 2 == 1:+ pending = pending[:-1].rstrip()+ continue+ logical_lines.append(pending)+ active_heredocs.extend(_dockerfile_heredocs(pending))+ pending = ""+ if pending:+ logical_lines.append(pending)+ active_heredocs.extend(_dockerfile_heredocs(pending))+ if active_heredocs:+ raise ValueError(f"Cannot safely parse unterminated Dockerfile heredoc in {source}")+ return logical_lines+++def _dockerfile_logical_lines(path: Path) -> list[str]:+ return _dockerfile_logical_lines_from_content(path.read_text(encoding="utf-8"), source=str(path))+++def _dockerfile_final_explicit_user(content: str) -> str | None:+ """Return the statically known USER inherited by the final stage."""+ user: str | None = None+ stage_users: dict[str, str | None] = {}+ current_stage_name: str | None = None+ current_stage_index = -1++ def _save_current_stage() -> None:+ if current_stage_index < 0:+ return+ stage_users[str(current_stage_index)] = user+ if current_stage_name is not None:+ stage_users[current_stage_name.casefold()] = user++ for line in _dockerfile_logical_lines_from_content(content):+ parts = line.split(None, 1)+ if not parts:+ continue+ instruction = parts[0].upper()+ if instruction == "FROM" and len(parts) == 2:+ _save_current_stage()+ try:+ from_parts = shlex.split(parts[1])+ except ValueError:+ from_parts = parts[1].split()+ non_options = [part for part in from_parts if not part.startswith("--")]+ base = non_options[0] if non_options else ""+ alias = next(+ (non_options[index + 1] for index, value in enumerate(non_options[:-1]) if value.upper() == "AS"),+ None,+ )+ user = stage_users.get(base.casefold())+ current_stage_index += 1+ current_stage_name = alias+ elif instruction == "USER" and len(parts) == 2:+ user = parts[1].strip()+ return user+++def _replace_single_from_image(content: str, base_image: str) -> tuple[str, str]:+ """Replace one Dockerfile stage image while preserving options and its alias."""+ from_lines = [+ line for line in _dockerfile_logical_lines_from_content(content) if line.split(None, 1)[0].upper() == "FROM"+ ]+ if len(from_lines) != 1:+ raise ValueError(+ "Custom Dockerfile rebase mode requires exactly one FROM instruction; "+ "use preserve mode for multi-stage Dockerfiles"+ )+ match = re.search(r"^(?P<prefix>[ \t]*FROM[ \t]+)(?P<body>[^\r\n]+)", content, flags=re.IGNORECASE | re.MULTILINE)+ if match is None:+ raise ValueError("Custom Dockerfile rebase mode could not locate its FROM instruction safely")+ body = match.group("body")+ if body.rstrip().endswith(("\\", "`")):+ raise ValueError(+ "Custom Dockerfile rebase mode does not support a continued FROM instruction; use preserve mode"+ )+ tokens = list(re.finditer(r'"[^"\r\n]*"|\'[^\'\r\n]*\'|\S+', body))+ image_token = next((token for token in tokens if not token.group(0).startswith("--")), None)+ if image_token is None:+ raise ValueError("Custom Dockerfile FROM instruction does not contain an image")+ body_start = match.start("body")+ image_start = body_start + image_token.start()+ image_end = body_start + image_token.end()+ return content[:image_start] + base_image + content[image_end:], from_lines[0]+++def _dockerfile_instruction_sources(line: str) -> list[str]:+ """Return build-context sources for one COPY/ADD instruction."""+ parts = line.split(None, 1)+ if len(parts) != 2 or parts[0].upper() not in {"COPY", "ADD"}:+ return []+ payload = parts[1].strip()+ from_stage = False+ while payload.startswith("--"):+ option_parts = payload.split(None, 1)+ option = option_parts[0]+ if len(option_parts) == 1:+ raise ValueError(f"Cannot safely parse custom Dockerfile COPY/ADD options: {line}")+ payload = option_parts[1].lstrip()+ if option == "--from":+ stage_parts = payload.split(None, 1)+ from_stage = True+ if len(stage_parts) == 1:+ raise ValueError(f"Cannot safely parse custom Dockerfile COPY/ADD --from option: {line}")+ payload = stage_parts[1].lstrip()+ elif option.startswith("--from="):+ from_stage = True+ if from_stage or not payload:+ return []+ if _docker_expression_has_literal_dollar(payload):+ raise ValueError(f"Cannot safely parse quoted or escaped Dockerfile COPY/ADD variable: {line}")+ if re.match(r'^\[\s*"', payload):+ try:+ values = json.loads(payload)+ except json.JSONDecodeError as exc:+ raise ValueError(f"Cannot parse custom Dockerfile COPY/ADD JSON form: {line}") from exc+ if isinstance(values, list) and len(values) >= 2 and all(isinstance(value, str) for value in values):+ return values[:-1]+ raise ValueError(f"Cannot safely parse custom Dockerfile COPY/ADD JSON form: {line}")+ try:+ values = shlex.split(payload)+ except ValueError as exc:+ raise ValueError(f"Cannot safely parse custom Dockerfile COPY/ADD form: {line}") from exc+ if len(values) < 2:+ raise ValueError(f"Cannot safely parse custom Dockerfile COPY/ADD form: {line}")+ sources = values[:-1]+ heredoc_sources = [source for source in sources if source.startswith("<<")]+ malformed_heredocs = [source for source in heredoc_sources if _DOCKER_HEREDOC_RE.fullmatch(source) is None]+ if malformed_heredocs:+ raise ValueError(f"Cannot safely parse custom Dockerfile COPY/ADD heredoc: {line}")+ return [source for source in sources if source not in heredoc_sources]+++def _dockerfile_named_context_sources(line: str) -> list[tuple[str, str]]:+ """Return ``(--from context, source)`` pairs for one COPY/ADD."""+ parts = line.split(None, 1)+ if len(parts) != 2 or parts[0].upper() not in {"COPY", "ADD"}:+ return []+ payload = parts[1].strip()+ from_context: str | None = None+ while payload.startswith("--"):+ option_parts = payload.split(None, 1)+ option = option_parts[0]+ if len(option_parts) == 1:+ raise ValueError(f"Cannot safely parse custom Dockerfile COPY/ADD options: {line}")+ payload = option_parts[1].lstrip()+ if option == "--from":+ context_parts = payload.split(None, 1)+ if len(context_parts) == 1:+ raise ValueError(f"Cannot safely parse custom Dockerfile COPY/ADD --from option: {line}")+ from_context = context_parts[0]+ payload = context_parts[1].lstrip()+ elif option.startswith("--from="):+ from_context = option.partition("=")[2]+ if from_context is None or not payload:+ return []+ if "$" in from_context or _docker_expression_has_literal_dollar(payload):+ raise ValueError(f"Cannot safely parse dynamic Dockerfile named-context source: {line}")+ if re.match(r'^\[\s*"', payload):+ try:+ values = json.loads(payload)+ except json.JSONDecodeError as exc:+ raise ValueError(f"Cannot parse custom Dockerfile COPY/ADD JSON form: {line}") from exc+ if not (isinstance(values, list) and len(values) >= 2 and all(isinstance(value, str) for value in values)):+ raise ValueError(f"Cannot safely parse custom Dockerfile COPY/ADD JSON form: {line}")+ sources = values[:-1]+ else:+ try:+ values = shlex.split(payload)+ except ValueError as exc:+ raise ValueError(f"Cannot safely parse custom Dockerfile COPY/ADD form: {line}") from exc+ if len(values) < 2:+ raise ValueError(f"Cannot safely parse custom Dockerfile COPY/ADD form: {line}")+ sources = values[:-1]+ heredoc_sources = [source for source in sources if source.startswith("<<")]+ malformed_heredocs = [source for source in heredoc_sources if _DOCKER_HEREDOC_RE.fullmatch(source) is None]+ if malformed_heredocs:+ raise ValueError(f"Cannot safely parse custom Dockerfile COPY/ADD heredoc: {line}")+ return [(from_context, source) for source in sources if source not in heredoc_sources]+++def _dockerfile_named_build_context_sources(path: Path) -> list[tuple[str, str]]:+ """Return statically named build-context sources from a Dockerfile."""+ sources: list[tuple[str, str]] = []+ for line in _dockerfile_logical_lines(path):+ parts = line.split(None, 1)+ effective_line = parts[1].strip() if len(parts) == 2 and parts[0].upper() == "ONBUILD" else line+ sources.extend(_dockerfile_named_context_sources(effective_line))+ return sources+++def _dockerfile_build_context_sources(path: Path) -> list[str]:+ """Return COPY/ADD build-context sources from a Dockerfile."""+ sources: list[str] = []+ for line in _dockerfile_logical_lines(path):+ parts = line.split(None, 1)+ effective_line = parts[1].strip() if len(parts) == 2 and parts[0].upper() == "ONBUILD" else line+ sources.extend(_dockerfile_instruction_sources(effective_line))+ return sources+++def _normalize_docker_context_source(source: str) -> str:+ """Normalize a COPY/ADD source using Docker build-context traversal rules."""+ normalized = posixpath.normpath(source.replace("\\", "/")).lstrip("/")+ while normalized.startswith("../"):+ normalized = normalized[3:]+ if normalized == "..":+ return "."+ while normalized.startswith("./"):+ normalized = normalized[2:]+ return normalized+++_DOCKER_VARIABLE_RE = re.compile(r"\$(?:\{(?P<braced>[A-Za-z_][A-Za-z0-9_]*)\}|(?P<plain>[A-Za-z_][A-Za-z0-9_]*))")+++def _docker_expression_has_literal_dollar(value: str) -> bool:+ """Return whether Docker keeps a dollar literal that shlex would erase."""+ return "$$" in value or re.search(r"\\+\$", value) is not None or re.search(r"'[^']*\$[^']*'", value) is not None+++def _resolve_docker_source_variables(source: str, defaults: dict[str, str | None]) -> tuple[str, bool]:+ """Resolve known static Docker variables and report any unresolved reference."""+ resolved = source+ for _ in range(len(defaults) + 1):+ changed = False++ def _replace(match: re.Match[str]) -> str:+ nonlocal changed+ name = match.group("braced") or match.group("plain")+ if name not in defaults or defaults[name] is None:+ return match.group(0)+ changed = True+ return defaults[name]++ updated = _DOCKER_VARIABLE_RE.sub(_replace, resolved)+ resolved = updated+ if not changed:+ break+ return resolved, "$" in resolved+++def _dockerfile_resolved_build_context_sources(+ path: Path,+ *,+ build_arg_overrides: dict[str, str | None] | None = None,+) -> list[tuple[str, str, bool]]:+ """Resolve COPY/ADD sources using ARG/ENV values visible at each instruction."""+ global_args: dict[str, str | None] = {}+ stage_args: dict[str, str | None] = {}+ stage_env: dict[str, str | None] = {}+ stage_onbuild: list[str] = []+ saved_stages: dict[str, tuple[dict[str, str | None], dict[str, str | None], list[str]]] = {}+ current_stage_name: str | None = None+ current_stage_index = -1+ in_stage = False+ resolved_sources: list[tuple[str, str, bool]] = []+ build_arg_overrides = build_arg_overrides or {}++ def _apply_arg(payload: str, *, stage_scope: bool) -> None:+ name, separator, value = payload.partition("=")+ name = name.strip()+ if re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", name) is None:+ raise ValueError(f"Cannot safely parse custom Dockerfile ARG: {payload}")+ target = stage_args if stage_scope else global_args+ visible = {**stage_args, **stage_env} if stage_scope else dict(global_args)+ if name in build_arg_overrides:+ target[name] = build_arg_overrides[name]+ elif separator:+ raw_value = value.strip()+ if _docker_expression_has_literal_dollar(raw_value):+ raise ValueError(f"Cannot safely parse quoted or escaped Dockerfile ARG variable: {payload}")+ resolved, unresolved = _resolve_docker_source_variables(raw_value.strip("\"'"), visible)+ target[name] = None if unresolved else resolved+ elif stage_scope and name in target:+ return+ elif stage_scope and name in global_args:+ target[name] = global_args[name]+ else:+ target[name] = None++ def _apply_env(payload: str) -> None:+ if _docker_expression_has_literal_dollar(payload):+ raise ValueError(f"Cannot safely parse quoted or escaped Dockerfile ENV variable: {payload}")+ try:+ assignments = shlex.split(payload)+ except ValueError as exc:+ raise ValueError(f"Cannot safely parse custom Dockerfile ENV: {payload}") from exc+ pairs: list[tuple[str, str]] = []+ if assignments and all("=" in assignment for assignment in assignments):+ pairs = [tuple(assignment.split("=", 1)) for assignment in assignments]+ elif len(assignments) >= 2:+ pairs = [(assignments[0], " ".join(assignments[1:]))]+ if not pairs:+ raise ValueError(f"Cannot safely parse custom Dockerfile ENV: {payload}")+ visible = {**stage_args, **stage_env}+ updates: dict[str, str | None] = {}+ for name, value in pairs:+ if re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", name) is None:+ raise ValueError(f"Cannot safely parse custom Dockerfile ENV: {payload}")+ resolved, unresolved = _resolve_docker_source_variables(value, visible)+ updates[name] = None if unresolved else resolved+ stage_env.update(updates)++ def _append_context_sources(line: str) -> None:+ visible = {**stage_args, **stage_env}+ for source in _dockerfile_instruction_sources(line):+ resolved, unresolved = _resolve_docker_source_variables(source, visible)+ resolved_sources.append((source, resolved, unresolved))++ def _run_onbuild_trigger(trigger: str) -> None:+ trigger_parts = trigger.split(None, 1)+ if len(trigger_parts) != 2:+ raise ValueError(f"Cannot safely parse custom Dockerfile ONBUILD trigger: {trigger}")+ trigger_instruction, trigger_payload = trigger_parts[0].upper(), trigger_parts[1].strip()+ if trigger_instruction == "ARG":+ _apply_arg(trigger_payload, stage_scope=True)+ elif trigger_instruction == "ENV":+ _apply_env(trigger_payload)+ elif trigger_instruction in {"COPY", "ADD"}:+ _append_context_sources(trigger)+ elif trigger_instruction in {"FROM", "MAINTAINER", "ONBUILD"}:+ raise ValueError(f"Cannot safely parse unsupported Dockerfile ONBUILD trigger: {trigger}")++ for line in _dockerfile_logical_lines(path):+ parts = line.split(None, 1)+ if len(parts) != 2:+ continue+ instruction, payload = parts[0].upper(), parts[1].strip()++ if instruction == "FROM":+ if in_stage:+ saved_stage = (dict(stage_args), dict(stage_env), list(stage_onbuild))+ saved_stages[str(current_stage_index)] = saved_stage+ if current_stage_name is not None:+ saved_stages[current_stage_name.casefold()] = saved_stage+ try:+ from_parts = shlex.split(payload)+ except ValueError:+ from_parts = payload.split()+ non_options = [part for part in from_parts if not part.startswith("--")]+ raw_base = non_options[0] if non_options else ""+ base, base_unresolved = _resolve_docker_source_variables(raw_base, global_args)+ alias = next(+ (non_options[index + 1] for index, value in enumerate(non_options[:-1]) if value.upper() == "AS"),+ None,+ )+ inherited_args, inherited_env, inherited_onbuild = (+ saved_stages.get(base.casefold(), ({}, {}, [])) if not base_unresolved else ({}, {}, [])+ )+ stage_args = dict(inherited_args)+ stage_env = dict(inherited_env)+ stage_onbuild = []+ current_stage_name = alias+ current_stage_index += 1+ in_stage = True+ for trigger in inherited_onbuild:+ _run_onbuild_trigger(trigger)+ continue++ if instruction == "ARG":+ _apply_arg(payload, stage_scope=in_stage)+ continue++ if instruction == "ENV" and in_stage:+ _apply_env(payload)+ continue++ if instruction == "ONBUILD" and in_stage:+ stage_onbuild.append(payload)+ continue++ if instruction not in {"COPY", "ADD"}:+ continue+ _append_context_sources(line)++ return resolved_sources+++def _compose_build_arg_overrides(env_dir: Path, dockerfile: Path) -> dict[str, str | None]:+ """Return literal build-arg overrides for Compose services that build *dockerfile*."""+ import yaml++ compose_files = [env_dir / name for name in ("docker-compose.yaml", "docker-compose.yml")]+ compose_files = [compose for compose in compose_files if compose.is_file()]+ if not compose_files:+ return {}++ overrides: dict[str, str | None] = {}+ for compose in compose_files:+ try:+ content = yaml.safe_load(compose.read_text(encoding="utf-8"))+ except (OSError, yaml.YAMLError) as exc:+ raise ValueError(f"Cannot safely resolve custom Docker Compose build args: {exc}") from exc+ services = content.get("services", {}) if isinstance(content, dict) else {}+ if not isinstance(services, dict):+ continue+ for service_name, service in services.items():+ if service_name != "main" or not isinstance(service, dict):+ continue+ build = service.get("build")+ if not isinstance(build, dict):+ continue+ context = build.get("context", ".")+ dockerfile_name = build.get("dockerfile", "Dockerfile")+ if not isinstance(context, str) or not isinstance(dockerfile_name, str):+ continue+ candidate = (compose.parent / context / dockerfile_name).resolve()+ if candidate != dockerfile.resolve():+ continue+ args = build.get("args", {})+ if isinstance(args, list):+ items = [item.partition("=") if isinstance(item, str) else ("", "", "") for item in args]+ parsed = {name: value if separator else None for name, separator, value in items if name}+ elif isinstance(args, dict):+ parsed = {str(name): None if value is None else str(value) for name, value in args.items()}+ else:+ raise ValueError("Cannot safely resolve custom Docker Compose build args")+ for name, value in parsed.items():+ if value is not None and "$" in value:+ value = None+ if name in overrides and overrides[name] != value:+ overrides[name] = None+ else:+ overrides[name] = value+ return overrides+++def _ensure_empty_docker_input_compatibility(+ context_dir: Path,+ dockerfile: Path,+ *,+ has_input: bool,+ context_is_reserved_input: bool = False,+ build_arg_overrides: dict[str, str | None] | None = None,+) -> None:+ """Keep directory-level ``COPY input/`` valid for one Docker build context."""+ resolved_sources: list[str] = []+ unresolved_sources: list[str] = []+ for source, resolved, unresolved in _dockerfile_resolved_build_context_sources(+ dockerfile,+ build_arg_overrides=build_arg_overrides or {},+ ):+ if unresolved:+ unresolved_sources.append(source)+ else:+ resolved_sources.append(resolved)+ for source in _dockerfile_main_mount_context_sources(dockerfile):+ if "$" in source:+ unresolved_sources.append(source)+ else:+ resolved_sources.append(source)+ normalized_sources = [_normalize_docker_context_source(source) for source in resolved_sources]+ for source in normalized_sources:+ first_component = source.split("/", 1)[0]+ if first_component.casefold() == "input" and first_component != "input":+ raise ValueError("Custom Dockerfile input sources must use the canonical lowercase path 'input/'")+ if has_input:+ return+ if unresolved_sources:+ raise ValueError(+ "Custom Dockerfile uses an ambiguous input source for an eval entry with no staged input files; "+ "use a literal path or a static ARG/ENV default"+ )+ if any(source.casefold().startswith("input/") for source in normalized_sources):+ raise ValueError("Custom Dockerfile copies a specific input path, but this eval entry stages no input files")+ if context_is_reserved_input:+ if any(source not in {"", "."} for source in normalized_sources):+ raise ValueError(+ "Custom Dockerfile copies a specific path from the input build context, "+ "but this eval entry stages no input files"+ )+ return+ # External base images can carry ONBUILD COPY input/ triggers that are not+ # visible in the authored Dockerfile. An empty reserved directory keeps+ # those builds valid without exposing any fixture bytes.+ (context_dir / "input").mkdir(parents=True, exist_ok=True)+++def _ensure_empty_custom_docker_input_compatibility(env_dir: Path, dockerfile: Path, *, has_input: bool) -> None:+ """Keep directory-level ``COPY input/`` valid for the managed main build."""+ _ensure_empty_docker_input_compatibility(+ env_dir,+ dockerfile,+ has_input=has_input,+ build_arg_overrides=_compose_build_arg_overrides(env_dir, dockerfile),+ )+++def _literal_compose_build_args(build: dict[str, Any]) -> dict[str, str | None]:+ raw_args = build.get("args", {})+ if isinstance(raw_args, list):+ items = [item.partition("=") if isinstance(item, str) else ("", "", "") for item in raw_args]+ parsed = {name: value if separator else None for name, separator, value in items if name}+ elif isinstance(raw_args, dict):+ parsed = {str(name): None if value is None else str(value) for name, value in raw_args.items()}+ else:+ raise ValueError("Cannot safely resolve custom Docker Compose build args")+ return {name: None if value is not None and "$" in value else value for name, value in parsed.items()}+++def _compose_build_context_values(build: str | dict[str, Any]) -> list[str]:+ """Return validated primary and named build-context paths."""+ if isinstance(build, str):+ return [build]+ values = [str(build.get("context", "."))]+ additional = build.get("additional_contexts", {})+ if isinstance(additional, list):+ values.extend(item.split("=", 1)[1] for item in additional if isinstance(item, str) and "=" in item)+ elif isinstance(additional, dict):+ values.extend(str(value) for value in additional.values())+ return values+++def _compose_additional_build_contexts(build: str | dict[str, Any]) -> dict[str, str]:+ """Return named Compose build contexts after structural validation."""+ if not isinstance(build, dict):+ return {}+ additional = build.get("additional_contexts", {})+ if isinstance(additional, list):+ return {+ name: value+ for item in additional+ if isinstance(item, str) and "=" in item+ for name, _separator, value in [item.partition("=")]+ if name+ }+ if isinstance(additional, dict):+ return {str(name): str(value) for name, value in additional.items()}+ return {}+++def _dockerfile_mount_context_sources(path: Path) -> list[tuple[str | None, str]]:+ """Return BuildKit RUN-mount contexts and their source paths."""+ sources: list[tuple[str | None, str]] = []+ for line in _dockerfile_logical_lines(path):+ parts = line.split(None, 1)+ effective_line = parts[1].strip() if len(parts) == 2 and parts[0].upper() == "ONBUILD" else line+ effective_parts = effective_line.split(None, 1)+ if len(effective_parts) != 2 or effective_parts[0].upper() != "RUN":+ continue+ try:+ tokens = shlex.split(effective_parts[1])+ except ValueError as exc:+ raise ValueError(f"Cannot safely parse custom Dockerfile RUN options: {effective_line}") from exc+ for token in tokens:+ if not token.startswith("--"):+ break+ if not token.startswith("--mount="):+ continue+ mount = token.partition("=")[2]+ options = dict(item.partition("=")[::2] for item in mount.split(",") if "=" in item)+ source = options.get("source", options.get("src", "."))+ sources.append((options.get("from"), source))+ return sources+++def _dockerfile_main_mount_context_sources(path: Path) -> list[str]:+ """Return source paths for BuildKit mounts from the primary context."""+ return [source for context, source in _dockerfile_mount_context_sources(path) if context is None]+++def _dockerfile_named_mount_context_sources(path: Path) -> list[tuple[str, str]]:+ """Return source paths for BuildKit mounts from named contexts."""+ return [(context, source) for context, source in _dockerfile_mount_context_sources(path) if context is not None]+++def _validate_empty_named_input_context_sources(+ dockerfile: Path,+ reserved_named_contexts: set[str],+ has_input: bool,+) -> None:+ """Reject specific reads from an explicitly empty named input context."""+ if has_input or not reserved_named_contexts:+ return+ named_sources = [+ *_dockerfile_named_build_context_sources(dockerfile),+ *_dockerfile_named_mount_context_sources(dockerfile),+ ]+ for context, source in named_sources:+ if context not in reserved_named_contexts:+ continue+ if "$" in source or _normalize_docker_context_source(source) not in {"", "."}:+ raise ValueError(+ "Custom Dockerfile reads a specific path from an input context, "+ "but this eval entry stages no input files"+ )+++def _ensure_empty_compose_input_compatibility(env_dir: Path, *, has_input: bool) -> None:+ """Apply explicit-empty input compatibility to every validated Compose build."""+ compose_path = _custom_compose_path(env_dir)+ if compose_path is None:+ return+ import yaml++ content = yaml.safe_load(compose_path.read_text(encoding="utf-8"))+ services = content.get("services", {}) if isinstance(content, dict) else {}+ recipes: list[tuple[Path, Path | None, str | None, dict[str, str | None], bool, set[str]]] = []+ for service in services.values():+ if not isinstance(service, dict) or "build" not in service:+ continue+ build = service["build"]+ if not has_input:+ for value in _compose_build_context_values(build):+ normalized = _normalize_docker_context_source(value)+ first_component = normalized.split("/", 1)[0]+ if first_component.casefold() == "input" and first_component != "input":+ raise ValueError(+ "Custom Docker Compose input contexts must use the canonical lowercase path 'input'"+ )+ if normalized.casefold() == "input":+ (env_dir / "input").mkdir(parents=True, exist_ok=True)+ elif normalized.casefold().startswith("input/"):+ raise ValueError(+ "Custom Docker Compose references a specific input context, but this eval entry stages no input files"+ )+ if isinstance(build, str):+ context_dir = (env_dir / build).resolve()+ context_is_reserved_input = _normalize_docker_context_source(build).casefold() == "input"+ recipes.append((context_dir, context_dir / "Dockerfile", None, {}, context_is_reserved_input, set()))+ continue+ if not isinstance(build, dict):+ continue+ context_dir = (env_dir / str(build.get("context", "."))).resolve()+ inline = build.get("dockerfile_inline")+ dockerfile = None if inline is not None else context_dir / str(build.get("dockerfile", "Dockerfile"))+ context_is_reserved_input = (+ _normalize_docker_context_source(str(build.get("context", "."))).casefold() == "input"+ )+ reserved_named_contexts = {+ name+ for name, value in _compose_additional_build_contexts(build).items()+ if _normalize_docker_context_source(value).casefold() == "input"+ }+ recipes.append(+ (+ context_dir,+ dockerfile,+ inline,+ _literal_compose_build_args(build),+ context_is_reserved_input,+ reserved_named_contexts,+ )+ )++ initial_input = {+ context_dir: (+ has_input+ if _paths_equivalent(context_dir, env_dir) or context_is_reserved_input+ else (context_dir / "input").is_dir()+ )+ for context_dir, _dockerfile, _inline, _args, context_is_reserved_input, _named in recipes+ }+ for context_dir, dockerfile, inline, build_args, context_is_reserved_input, reserved_named_contexts in recipes:+ if (+ dockerfile is not None+ and _paths_equivalent(context_dir, env_dir)+ and _paths_equivalent(dockerfile, env_dir / "Dockerfile")+ ):+ continue+ if inline is not None:+ with tempfile.TemporaryDirectory(prefix="skillevaluator-compose-inline-") as temporary:+ parsed_dockerfile = Path(temporary) / "Dockerfile"+ parsed_dockerfile.write_text(str(inline).replace("$$", "$"), encoding="utf-8")+ _ensure_empty_docker_input_compatibility(+ context_dir,+ parsed_dockerfile,+ has_input=initial_input[context_dir],+ context_is_reserved_input=context_is_reserved_input,+ build_arg_overrides=build_args,+ )+ _validate_empty_named_input_context_sources(parsed_dockerfile, reserved_named_contexts, has_input)+ continue+ if dockerfile is None or not dockerfile.is_file():+ raise ValueError(f"Custom Docker Compose build Dockerfile does not exist: {dockerfile}")+ _ensure_empty_docker_input_compatibility(+ context_dir,+ dockerfile,+ has_input=initial_input[context_dir],+ context_is_reserved_input=context_is_reserved_input,+ build_arg_overrides=build_args,+ )+ _validate_empty_named_input_context_sources(dockerfile, reserved_named_contexts, has_input)++def _compose_strings(value: Any) -> list[str]:- if isinstance(value, str):- return [value]- if isinstance(value, dict):- return [item for child in value.values() for item in _compose_strings(child)]- if isinstance(value, list):- return [item for child in value for item in _compose_strings(child)]- return []+ strings: list[str] = []+ seen_containers: set[int] = set()+ nodes = 0++ def _visit(node: Any, depth: int) -> None:+ nonlocal nodes+ nodes += 1+ if depth > 64 or nodes > 10_000:+ raise ValueError("Custom Docker Compose structure exceeds safe traversal limits")+ if isinstance(node, str):+ strings.append(node)+ return+ if not isinstance(node, (dict, list)):+ return+ identity = id(node)+ if identity in seen_containers:+ raise ValueError("Custom Docker Compose aliases or cycles are not supported")+ seen_containers.add(identity)+ children = node.values() if isinstance(node, dict) else node+ for child in children:+ _visit(child, depth + 1)++ _visit(value, 0)+ return stringsdef _validate_compose_interpolation(content: dict[str, Any], allowed_env: set[str]) -> None:@@ -1090,6 +2476,11 @@ def _validate_compose_host_passthrough(value: object, *, allowed_env: set[str],)+def _require_string_mapping_keys(value: dict[object, object], *, field: str) -> None:+ if any(not isinstance(key, str) for key in value):+ raise ValueError(f"Custom Docker Compose {field} keys must be strings")++def _validate_compose_build(service_name: str,build: object,@@ -1103,6 +2494,7 @@ def _validate_compose_build(returnif not isinstance(build, dict):raise ValueError(f"Custom Docker Compose service '{service_name}' build must be a path or mapping")+ _require_string_mapping_keys(build, field=f"service '{service_name}' build")unsupported_keys = sorted(set(build) - _COMPOSE_ALLOWED_BUILD_KEYS)if unsupported_keys:@@ -1116,8 +2508,15 @@ def _validate_compose_build(context_dir = environment_dir / str(context)dockerfile = build.get("dockerfile")+ inline = build.get("dockerfile_inline")+ if dockerfile is not None and inline is not None:+ raise ValueError(f"Custom Docker Compose service '{service_name}' cannot set both dockerfile forms")if dockerfile is not None and not _is_relative_compose_path(dockerfile, context_dir):raise ValueError(f"Custom Docker Compose service '{service_name}' has an unsafe build dockerfile")+ if inline is not None and (not isinstance(inline, str) or not inline.strip() or len(inline) > 20_000):+ raise ValueError(f"Custom Docker Compose service '{service_name}' has an invalid inline Dockerfile")+ if service_name == "main" and "target" in build:+ raise ValueError("Custom Docker Compose service 'main' cannot select an authored build target")additional_contexts = build.get("additional_contexts", {})if isinstance(additional_contexts, list):@@ -1125,6 +2524,7 @@ def _validate_compose_build(item.split("=", 1)[1] if isinstance(item, str) and "=" in item else None for item in additional_contexts]elif isinstance(additional_contexts, dict):+ _require_string_mapping_keys(additional_contexts, field=f"service '{service_name}' build additional_contexts")values = list(additional_contexts.values())else:raise ValueError(@@ -1185,6 +2585,7 @@ def _validate_and_sanitize_custom_compose(raise ValueError(f"Custom Docker Compose file cannot be read safely: {exc}") from excif not isinstance(content, dict):raise ValueError("Custom Docker Compose file must contain a top-level mapping")+ _require_string_mapping_keys(content, field="top-level")unsupported_top_level = sorted(set(content) - _COMPOSE_ALLOWED_TOP_LEVEL_KEYS)if unsupported_top_level:@@ -1198,6 +2599,9 @@ def _validate_and_sanitize_custom_compose(for svc_name, service in services.items():if not isinstance(svc_name, str) or not isinstance(service, dict):raise ValueError("Custom Docker Compose services must map names to service mappings")+ _require_string_mapping_keys(service, field=f"service '{svc_name}'")+ if "build" in service and "image" in service:+ raise ValueError(f"Custom Docker Compose service '{svc_name}' cannot set image together with build")volumes = service.get("volumes", [])if not isinstance(volumes, list):@@ -1236,10 +2640,13 @@ def _validate_and_sanitize_custom_compose(if not isinstance(volumes, dict):raise ValueError("Custom Docker Compose top-level volumes must be a mapping")for volume_name, volume in volumes.items():+ if not isinstance(volume_name, str):+ raise ValueError("Custom Docker Compose volume names must be strings")if volume is None:continueif not isinstance(volume, dict):raise ValueError(f"Custom Docker Compose volume '{volume_name}' must be a mapping")+ _require_string_mapping_keys(volume, field=f"volume '{volume_name}'")unsupported_keys = sorted(set(volume) - _COMPOSE_ALLOWED_VOLUME_KEYS)if unsupported_keys:raise ValueError(f"Custom Docker Compose volume '{volume_name}' cannot set: {', '.join(unsupported_keys)}")@@ -1248,10 +2655,13 @@ def _validate_and_sanitize_custom_compose(if not isinstance(networks, dict):raise ValueError("Custom Docker Compose top-level networks must be a mapping")for network_name, network in networks.items():+ if not isinstance(network_name, str):+ raise ValueError("Custom Docker Compose network names must be strings")if network is None:continueif not isinstance(network, dict):raise ValueError(f"Custom Docker Compose network '{network_name}' must be a mapping")+ _require_string_mapping_keys(network, field=f"network '{network_name}'")unsupported_keys = sorted(set(network) - _COMPOSE_ALLOWED_NETWORK_KEYS)if unsupported_keys:raise ValueError(@@ -1264,7 +2674,40 @@ def _validate_and_sanitize_custom_compose(compose_path.write_text(yaml.dump(content, default_flow_style=False), encoding="utf-8")-_VERIFIER_DEPS = "ragas~=0.4.0 openai>=1.0 anthropic>=0.40 boto3>=1.34"+def _custom_compose_path(environment_dir: Path) -> Path | None:+ candidates = [+ environment_dir / name+ for name in ("docker-compose.yaml", "docker-compose.yml")+ if (environment_dir / name).exists()+ ]+ if len(candidates) > 1:+ raise ValueError("Custom environment must not contain both docker-compose.yaml and docker-compose.yml")+ return candidates[0] if candidates else None+++_VERIFIER_DEPS = "ragas~=0.4.0 langchain-community<0.4.2 openai>=1.0 anthropic>=0.40 boto3>=1.34"+_VERIFIER_IMPORT_SMOKE = (+ "from ragas import SingleTurnSample; "+ "from ragas.llms.base import llm_factory; "+ "from ragas.messages import AIMessage, HumanMessage; "+ "from ragas.metrics.collections import AgentGoalAccuracyWithReference"+)+_MANAGED_PATH_SHELL = 'PATH="${PATH:+$PATH:}/usr/local/bin:/usr/bin:/bin"; export PATH; exec "$@"'+++def _managed_run(command: str, *args: str) -> str:+ """Build an exec-form RUN that remains usable after an authored PATH override."""+ return "RUN " + json.dumps(["/bin/sh", "-c", _MANAGED_PATH_SHELL, "skillevaluator", command, *args])+++def _verifier_install_lines(extra_requirements: Sequence[str] = ()) -> list[str]:+ """Install verifier and skill requirements in one resolver transaction."""+ requirements = [*_VERIFIER_DEPS.split(), *extra_requirements]+ return [+ _managed_run("python", "-m", "pip", "install", "--no-cache-dir", *requirements),+ _managed_run("python", "-c", _VERIFIER_IMPORT_SMOKE),+ ]+_WORKSPACE_SKILL_PATH = "COPY skills/ /workspace/skills/"_AGENT_SKILL_PATHS = [@@ -1272,6 +2715,242 @@ _AGENT_SKILL_PATHS = ["COPY skills/ /root/.agents/skills/","COPY skills/ /root/.config/opencode/skills/",]+_RUNTIME_SKILL_DIRS = (+ "/etc/codex/skills",+ "/workspace/skills",+ "/workspace/.agents/skills",+ "/workspace/.claude/commands",+ "/workspace/.claude/skills",+ "/workspace/.codex/skills",+ "/workspace/.config/opencode/skills",+ "/workspace/.cursor/skills",+ "/workspace/.gemini/skills",+ "/workspace/.gemini/extensions",+ "/workspace/.opencode/skills",+ "/root/.claude/commands",+ "/root/.claude/skills",+ "/root/.agents/skills",+ "/root/.codex/skills",+ "/root/.config/opencode/skills",+ "/root/.gemini/extensions",+ "/logs/agent/sessions/skills",+ "/tmp/agent-home/sessions/skills",+)+_PROJECT_SKILL_RELATIVE_DIRS = (+ ".agents/skills",+ ".claude/commands",+ ".claude/skills",+ ".codex/skills",+ ".config/opencode/skills",+ ".cursor/skills",+ ".gemini/skills",+ ".gemini/extensions",+ ".opencode/skills",+)+_RUNTIME_DISCOVERY_ENV_NAMES = frozenset(+ {+ "CLAUDE_CODE_DISABLE_POLICY_SKILLS",+ "CLAUDE_CONFIG_DIR",+ "CODEX_HOME",+ "GEMINI_CLI_HOME",+ "HOME",+ "OPENCODE_CONFIG_DIR",+ "USERPROFILE",+ "XDG_CONFIG_HOME",+ }+)+_RUNTIME_LOADER_ENV_NAMES = frozenset(+ {+ "BASH_ENV",+ "CLASSPATH",+ "DYLD_INSERT_LIBRARIES",+ "ENV",+ "GCONV_PATH",+ "JAVA_TOOL_OPTIONS",+ "LD_AUDIT",+ "LD_LIBRARY_PATH",+ "LD_PRELOAD",+ "LOCPATH",+ "LUA_CPATH",+ "LUA_INIT",+ "LUA_PATH",+ "NLSPATH",+ "NODE_OPTIONS",+ "PERL5LIB",+ "PERL5OPT",+ "PYTHONHOME",+ "PYTHONINSPECT",+ "PYTHONPATH",+ "PYTHONSTARTUP",+ "RUBYLIB",+ "RUBYOPT",+ "ZDOTDIR",+ "_JAVA_OPTIONS",+ }+)+_RUNTIME_LOADER_ENV_PREFIXES = ("BASH_FUNC_",)+_RUNTIME_LOADER_ENV_RESET = "ENV " + " ".join(f'{name}=""' for name in sorted(_RUNTIME_LOADER_ENV_NAMES))+_RUNTIME_POLICY_ENV_RESET = 'ENV CLAUDE_CODE_DISABLE_POLICY_SKILLS="1"'+_EVALUATOR_MANAGED_RUNTIME_ENV = {+ **dict.fromkeys(_RUNTIME_LOADER_ENV_NAMES, ""),+ "CLAUDE_CODE_DISABLE_POLICY_SKILLS": "1",+}+_CLEAR_SKILL_ROOT_FUNCTION = (+ "clear_skill_root() { target=$1; "+ 'if [ -d "$target" ] && [ ! -L "$target" ]; then '+ 'for child in "$target"/* "$target"/.[!.]* "$target"/..?*; do '+ '[ -e "$child" ] || [ -L "$child" ] || continue; /bin/rm -rf -- "$child"; done; '+ 'else /bin/rm -rf -- "$target"; fi; }; '+)+_PROJECT_SKILL_ANCESTOR_RESET = (+ "set -eu; current=$(/bin/pwd -P); while :; do "+ "for rel in "+ + " ".join(_PROJECT_SKILL_RELATIVE_DIRS)+ + '; do /bin/rm -rf -- "$current/$rel"; /bin/mkdir -p -- "$current/$rel"; done; '+ '[ "$current" = / ] && break; current=${current%/*}; [ -n "$current" ] || current=/; done'+)+_HOME_SKILL_RESET = (+ "set -eu; " + _CLEAR_SKILL_ROOT_FUNCTION + 'home=${HOME:-/root}; case "$home" in /*) ;; *) '+ "echo 'SkillEvaluator requires an absolute HOME for runtime skill isolation' >&2; exit 78;; esac; "+ "for rel in " + " ".join(_PROJECT_SKILL_RELATIVE_DIRS) + '; do clear_skill_root "$home/$rel"; done'+)+_PASSWD_HOME_SKILL_RESET = (+ "set -eu; " + _CLEAR_SKILL_ROOT_FUNCTION + "[ -r /etc/passwd ] || exit 0; "+ 'while IFS=: read -r _ _ _ _ _ home _; do case "$home" in /*) ;; *) continue;; esac; '+ '[ -d "$home" ] || continue; '+ "for rel in "+ + " ".join(_PROJECT_SKILL_RELATIVE_DIRS)+ + '; do clear_skill_root "$home/$rel"; done; done < /etc/passwd'+)+_DISCOVERY_ENV_SKILL_RESET = (+ "set -eu; "+ + _CLEAR_SKILL_ROOT_FUNCTION+ + 'if [ -n "${CLAUDE_CONFIG_DIR:-}" ]; then case "$CLAUDE_CONFIG_DIR" in /*) ;; *) exit 78;; esac; '+ 'clear_skill_root "$CLAUDE_CONFIG_DIR/skills"; clear_skill_root "$CLAUDE_CONFIG_DIR/commands"; fi; '+ 'if [ -n "${CODEX_HOME:-}" ]; then case "$CODEX_HOME" in /*) ;; *) exit 78;; esac; '+ 'clear_skill_root "$CODEX_HOME/skills"; fi; '+ 'if [ -n "${GEMINI_CLI_HOME:-}" ]; then case "$GEMINI_CLI_HOME" in /*) ;; *) exit 78;; esac; '+ 'clear_skill_root "$GEMINI_CLI_HOME/.gemini/skills"; '+ 'clear_skill_root "$GEMINI_CLI_HOME/.gemini/extensions"; fi; '+ 'if [ -n "${OPENCODE_CONFIG_DIR:-}" ]; then case "$OPENCODE_CONFIG_DIR" in /*) ;; *) exit 78;; esac; '+ 'clear_skill_root "$OPENCODE_CONFIG_DIR/skills"; fi; '+ 'if [ -n "${XDG_CONFIG_HOME:-}" ]; then case "$XDG_CONFIG_HOME" in /*) ;; *) exit 78;; esac; '+ 'clear_skill_root "$XDG_CONFIG_HOME/opencode/skills"; fi'+)+_RUNTIME_PROJECTION_OVERLAP_PREFLIGHT = (+ 'set -eu; check_path() { value=$1; case "$value" in '+ "/workspace/input|/workspace/input/*|/workspace/repo|/workspace/repo/*|"+ "/workspace/skills|/workspace/skills/*|"+ "*/.agents/skills|*/.agents/skills/*|*/.claude/commands|*/.claude/commands/*|"+ "*/.claude/skills|*/.claude/skills/*|"+ "*/.codex/skills|*/.codex/skills/*|*/.config/opencode/skills|*/.config/opencode/skills/*|"+ "*/.cursor/skills|*/.cursor/skills/*|*/.gemini/skills|*/.gemini/skills/*|"+ "*/.gemini/extensions|*/.gemini/extensions/*|"+ "*/.opencode/skills|*/.opencode/skills/*) "+ "echo 'SkillEvaluator agent workdir/home overlaps an evaluator-controlled runtime projection' >&2; "+ 'exit 78;; esac; }; check_control_path() { value=$1; check_path "$value"; '+ 'if [ -d "$value" ]; then canonical=$(CDPATH= cd -P -- "$value" 2>/dev/null && /bin/pwd -P) || exit 78; '+ 'check_path "$canonical"; elif [ -e "$value" ] || [ -L "$value" ]; then '+ "echo 'SkillEvaluator discovery home must resolve to a directory' >&2; exit 78; fi; }; "+ 'current=$(/bin/pwd -P); check_path "$current"; '+ 'home=${HOME:-/root}; case "$home" in /*) check_control_path "$home";; *) '+ "echo 'SkillEvaluator requires an absolute HOME for runtime skill isolation' >&2; exit 78;; esac; "+ 'for controlled_home in "${CLAUDE_CONFIG_DIR:-}" "${CODEX_HOME:-}" "${GEMINI_CLI_HOME:-}" '+ '"${OPENCODE_CONFIG_DIR:-}" '+ '"${XDG_CONFIG_HOME:-}"; do [ -z "$controlled_home" ] || check_control_path "$controlled_home"; done; '+ "[ -r /etc/passwd ] || exit 0; while IFS=: read -r _ _ _ _ _ passwd_home _; do "+ 'case "$passwd_home" in /*) '+ '[ -e "$passwd_home" ] && [ ! -d "$passwd_home" ] && continue; '+ 'check_control_path "$passwd_home";; esac; done < /etc/passwd'+)+_RUNTIME_REPO_RESET_LINES = [+ 'RUN ["/bin/rm", "-rf", "/workspace/repo"]',+ 'RUN ["/bin/mkdir", "-p", "/workspace/repo"]',+]+_RUNTIME_ROOT_PREFLIGHT = "RUN " + json.dumps(+ [+ "/bin/sh",+ "-c",+ 'PATH="${PATH:+$PATH:}/usr/local/bin:/usr/bin:/bin"; export PATH; '+ 'if [ "$(id -u)" -ne 0 ]; then echo \'SkillEvaluator final runtime projection requires root; '+ "declare the Dockerfile final USER explicitly so it can be restored' >&2; exit 78; fi",+ ]+)+++def _validated_agent_workdir(agent_workdir: str | None) -> str | None:+ """Return a normalized absolute container workdir or fail closed."""+ if agent_workdir is None:+ return None+ if not isinstance(agent_workdir, str) or not agent_workdir:+ raise ValueError("agent_workdir must be a non-empty absolute container path")+ normalized = posixpath.normpath(agent_workdir)+ typed_normalized = agent_workdir.rstrip("/") or "/"+ if (+ not normalized.startswith("/")+ or normalized != typed_normalized+ or re.fullmatch(r"/[A-Za-z0-9._/-]*", normalized) is None+ ):+ raise ValueError("agent_workdir must be a normalized absolute POSIX path")+ reserved_roots = ("/workspace/input", "/workspace/repo", "/workspace/skills")+ if any(normalized == root or normalized.startswith(f"{root}/") for root in reserved_roots):+ raise ValueError("agent_workdir must not be inside an evaluator-controlled runtime projection")+ parts = tuple(part for part in normalized.split("/") if part)+ discovery_parts = (+ (".agents", "skills"),+ (".claude", "skills"),+ (".codex", "skills"),+ (".config", "opencode", "skills"),+ (".cursor", "skills"),+ (".gemini", "skills"),+ (".opencode", "skills"),+ )+ if any(+ parts[index : index + len(candidate)] == candidate+ for candidate in discovery_parts+ for index in range(len(parts) - len(candidate) + 1)+ ):+ raise ValueError("agent_workdir must not be inside an agent skill discovery directory")+ return normalized+++def _validate_runtime_discovery_env(runtime_env: dict[str, str] | None) -> None:+ collisions = sorted(name for name in (runtime_env or {}) if name.upper() in _RUNTIME_DISCOVERY_ENV_NAMES)+ if collisions:+ raise ValueError(+ "harbor.runtime_env cannot override agent skill discovery location(s): " + ", ".join(collisions)+ )+++def _validate_runtime_loader_env(runtime_env: dict[str, str] | None) -> None:+ collisions = sorted(+ name+ for name in (runtime_env or {})+ if name.upper() in _RUNTIME_LOADER_ENV_NAMES or name.upper().startswith(_RUNTIME_LOADER_ENV_PREFIXES)+ )+ if collisions:+ raise ValueError("harbor.runtime_env cannot override process loader environment: " + ", ".join(collisions))+++def _runtime_skill_reset_lines(agent_workdir: str | None = None) -> list[str]:+ """Replace fixed, effective-home, and project-ancestor skill roots."""+ lines = [+ "RUN " + json.dumps(["/bin/rm", "-rf", *_RUNTIME_SKILL_DIRS]),+ "RUN " + json.dumps(["/bin/mkdir", "-p", *_RUNTIME_SKILL_DIRS]),+ "RUN " + json.dumps(["/bin/sh", "-c", _PROJECT_SKILL_ANCESTOR_RESET]),+ "RUN " + json.dumps(["/bin/sh", "-c", _HOME_SKILL_RESET]),+ "RUN " + json.dumps(["/bin/sh", "-c", _PASSWD_HOME_SKILL_RESET]),+ "RUN " + json.dumps(["/bin/sh", "-c", _DISCOVERY_ENV_SKILL_RESET]),+ ]+ validated_workdir = _validated_agent_workdir(agent_workdir)+ if validated_workdir is not None:+ lines.extend(+ [+ f"WORKDIR {validated_workdir}",+ "RUN " + json.dumps(["/bin/sh", "-c", _PROJECT_SKILL_ANCESTOR_RESET]),+ ]+ )+ return linesdef _runtime_copy_lines(@@ -1279,10 +2958,17 @@ def _runtime_copy_lines(include_input: bool,include_repo: bool = False,include_repo_linked_root: bool = False,+ agent_workdir: str | None = None,) -> list[str]:- lines = [_WORKSPACE_SKILL_PATH, *_AGENT_SKILL_PATHS, *agent_config_lines]- if include_input:- lines.append("COPY input/ /workspace/input/")+ _ = include_input+ lines = [+ _RUNTIME_ROOT_PREFLIGHT,+ *_runtime_skill_reset_lines(agent_workdir),+ *_RUNTIME_REPO_RESET_LINES,+ _WORKSPACE_SKILL_PATH,+ *_AGENT_SKILL_PATHS,+ *agent_config_lines,+ ]if include_repo:lines.append("COPY repo/ /workspace/repo/")if include_repo_linked_root:@@ -1290,19 +2976,72 @@ def _runtime_copy_lines(return lines-def _append_missing_lines(content: str, lines: list[str]) -> str:- additions = ""- for line in lines:- if not line:- if additions and not additions.endswith("\n\n"):- additions += "\n"- continue- if line not in content:- additions += f"{line}\n"- if not additions:- return content+def _append_task_input_projection(+ content: str,+ *,+ include_input: bool,+ agent_config_lines: list[str],+ restore_user: str | None = None,+) -> str:+ """Append the final evaluator-owned replacement of ``/workspace/input``."""+ separator = "" if content.endswith("\n") else "\n"+ lines = [+ "",+ "# SkillEvaluator: replace task input after all authored/base-image layers",+ 'RUN ["/bin/rm", "-rf", "/workspace/input"]',+ 'RUN ["/bin/mkdir", "-p", "/workspace/input"]',+ ]+ if include_input:+ lines.append("COPY input/ /workspace/input/")+ lines.extend(+ [+ "RUN " + json.dumps(["/bin/sh", "-c", _RUNTIME_PROJECTION_OVERLAP_PREFLIGHT]),+ "RUN " + json.dumps(["/bin/sh", "-c", _PROJECT_SKILL_ANCESTOR_RESET]),+ _WORKSPACE_SKILL_PATH,+ *_AGENT_SKILL_PATHS,+ *agent_config_lines,+ _RUNTIME_LOADER_ENV_RESET,+ _RUNTIME_POLICY_ENV_RESET,+ "ENTRYPOINT []",+ "HEALTHCHECK NONE",+ ]+ )+ if restore_user is not None and restore_user.casefold() != "root":+ lines.extend(["", f"USER {restore_user}"])+ return content + separator + "\n".join(lines) + "\n"+++def _extend_task_input_projection(lines: list[str], *, include_input: bool, agent_config_lines: list[str]) -> None:+ """Add the final evaluator-owned input replacement to generated Dockerfiles."""+ lines.extend(+ [+ "",+ 'RUN ["/bin/rm", "-rf", "/workspace/input"]',+ 'RUN ["/bin/mkdir", "-p", "/workspace/input"]',+ ]+ )+ if include_input:+ lines.append("COPY input/ /workspace/input/")+ lines.extend(+ [+ "RUN " + json.dumps(["/bin/sh", "-c", _RUNTIME_PROJECTION_OVERLAP_PREFLIGHT]),+ "RUN " + json.dumps(["/bin/sh", "-c", _PROJECT_SKILL_ANCESTOR_RESET]),+ _WORKSPACE_SKILL_PATH,+ *_AGENT_SKILL_PATHS,+ *agent_config_lines,+ _RUNTIME_LOADER_ENV_RESET,+ _RUNTIME_POLICY_ENV_RESET,+ "ENTRYPOINT []",+ "HEALTHCHECK NONE",+ ]+ )+++def _append_evaluator_runtime_lines(content: str, lines: list[str], *, elevate: bool = False) -> str:+ """Append evaluator-owned instructions without trusting authored text matches."""separator = "" if content.endswith("\n") else "\n"- return content + separator + additions+ user_line = "USER root\n" if elevate else ""+ return content + separator + "\n# SkillEvaluator: final runtime projection\n" + user_line + "\n".join(lines) + "\n"def _write_agent_configs(env_dir: Path) -> list[str]:@@ -1319,7 +3058,8 @@ def _rebase_custom_dockerfile_content(include_input: bool,include_repo: bool = False,include_repo_linked_root: bool = False,-) -> tuple[str, str] | None:+ agent_workdir: str | None = None,+) -> tuple[str, str]:"""Return custom Dockerfile content layered on top of the eval base image.The base image already contains: python:3.12-slim, system packages@@ -1329,34 +3069,30 @@ def _rebase_custom_dockerfile_content(The custom Dockerfile's FROM line is replaced so the skill author'sadditions (extra apt/pip packages, COPY, RUN) layer on top. Multi-agentskill discovery paths are appended if not already present.- Returns the rebased content and original ``FROM`` instruction, or ``None``- when the content has no ``FROM`` instruction.+ Returns the rebased content and original ``FROM`` instruction. Rebase mode+ fails closed for missing or multi-stage ``FROM`` instructions."""- lines = content.splitlines(keepends=True)- rebased: list[str] = []- from_replaced = False- original_from = ""- for line in lines:- stripped = line.strip()- if not from_replaced and stripped.upper().startswith("FROM "):- rebased.append(f"FROM {base_image}\n")- rebased.append(f"# SkillEvaluator: original base was {stripped}\n")- from_replaced = True- original_from = stripped- else:- rebased.append(line)-- if not from_replaced:- return None-- rebased_content = _append_missing_lines(- "".join(rebased),- _runtime_copy_lines(- agent_config_lines,- include_input,- include_repo=include_repo,- include_repo_linked_root=include_repo_linked_root,- ),+ restore_user = _dockerfile_final_explicit_user(content)+ rebased_source, original_from = _replace_single_from_image(content, base_image)+ rebased_content = _append_evaluator_runtime_lines(+ rebased_source + f"\n# SkillEvaluator: original base was {original_from}\n",+ [+ *_verifier_install_lines(),+ *_runtime_copy_lines(+ agent_config_lines,+ include_input,+ include_repo=include_repo,+ include_repo_linked_root=include_repo_linked_root,+ agent_workdir=agent_workdir,+ ),+ ],+ elevate=True,+ )+ rebased_content = _append_task_input_projection(+ rebased_content,+ include_input=include_input,+ agent_config_lines=agent_config_lines,+ restore_user=restore_user,)return rebased_content, original_from@@ -1368,23 +3104,29 @@ def _ensure_verifier_deps(include_input: bool,include_repo: bool = False,include_repo_linked_root: bool = False,+ agent_workdir: str | None = None,) -> None:"""Append verifier deps and runtime COPY lines to a custom Dockerfile."""content = dockerfile_path.read_text(encoding="utf-8")- additions = ""- if not ("ragas" in content and "anthropic" in content):- additions += f"\nRUN pip install --no-cache-dir {_VERIFIER_DEPS}\n"- updated = _append_missing_lines(+ restore_user = _dockerfile_final_explicit_user(content)+ user_prefix = "USER root\n" if restore_user is not None else ""+ additions = f"\n{user_prefix}" + "\n".join(_verifier_install_lines()) + "\n"+ updated = _append_evaluator_runtime_lines(content + additions,_runtime_copy_lines(agent_config_lines,include_input,include_repo=include_repo,include_repo_linked_root=include_repo_linked_root,+ agent_workdir=agent_workdir,),)- if updated == content:- return+ updated = _append_task_input_projection(+ updated,+ include_input=include_input,+ agent_config_lines=agent_config_lines,+ restore_user=restore_user,+ )dockerfile_path.write_text(updated, encoding="utf-8")logger.debug("Appended verifier deps + agent paths to %s", dockerfile_path)@@ -1423,6 +3165,8 @@ def _resolve_entry_file_ref(ref_path = Path(ref)if ref_path.is_absolute():raise ValueError(f"evals.json files entry must be relative to evals/: {ref}")+ if ".." in ref_path.parts:+ raise ValueError(f"evals.json files entry cannot traverse parent directories: {ref}")if _LOCAL_LINK_SCHEME_RE.match(ref):raise ValueError(f"evals.json files entry uses unsupported URI scheme: {ref}")@@ -1431,13 +3175,16 @@ def _resolve_entry_file_ref(else:candidates = [evals_dir / ref_path, skill_path / ref_path]- source = next((candidate.resolve() for candidate in candidates if candidate.exists()), None)- if source is None:+ source_path = next((candidate.absolute() for candidate in candidates if candidate.exists()), None)+ if source_path is None:raise FileNotFoundError(f"evals.json files entry does not exist: {ref}")+ source = source_path.resolve()resolved_evals_dir = evals_dir.resolve()if source == resolved_evals_dir or not _is_relative_to(source, resolved_evals_dir):raise ValueError(f"evals.json files entry resolves outside evals/: {ref}")+ if _has_symlink_component(source_path, evals_dir.absolute()):+ raise ValueError(f"evals.json files entry must not contain symlinks: {ref}")resolved_input_files_dir = input_files_dir.resolve() if input_files_dir and input_files_dir.exists() else Noneif resolved_input_files_dir and _is_relative_to(source, resolved_input_files_dir):@@ -1445,19 +3192,19 @@ def _resolve_entry_file_ref(else:rel = source.relative_to(resolved_evals_dir)- return source, rel+ return source_path, rel-def _copy_input_ref(source: Path, input_dir: Path, rel: Path) -> None:+def _copy_input_ref(source: Path, input_dir: Path, rel: Path, *, allowed_root: Path) -> None:dest = input_dir if rel == Path() else input_dir / relif source.is_dir():if dest.exists() and dest.is_file():dest.unlink()- copytree_secure(source, dest, dirs_exist_ok=True)+ copytree_secure(source, dest, dirs_exist_ok=True, allowed_root=allowed_root)returndest.parent.mkdir(parents=True, exist_ok=True)- shutil.copy2(source, dest)+ copy_file_secure(source, dest, allowed_root=allowed_root)def _stage_task_inputs(@@ -1469,13 +3216,13 @@ def _stage_task_inputs(evals_dir: Path,) -> bool:input_dir = env_dir / "input"- if os.path.lexists(input_dir) and (input_dir.is_symlink() or not input_dir.is_dir()):+ if os.path.lexists(input_dir) and (_path_is_link_or_reparse(input_dir) or not input_dir.is_dir()):input_dir.unlink()elif input_dir.exists():shutil.rmtree(input_dir)if "files" not in entry and input_files_dir and input_files_dir.exists():- copytree_secure(input_files_dir, input_dir, dirs_exist_ok=True)+ copytree_secure(input_files_dir, input_dir, dirs_exist_ok=True, allowed_root=evals_dir)for ref in _entry_file_refs(entry):source, rel = _resolve_entry_file_ref(@@ -1485,11 +3232,38 @@ def _stage_task_inputs(input_files_dir=input_files_dir,)input_dir.mkdir(parents=True, exist_ok=True)- _copy_input_ref(source, input_dir, rel)+ _copy_input_ref(source, input_dir, rel, allowed_root=evals_dir)return input_dir.exists()+def _entry_declares_task_input(entry: dict[str, Any], input_files_dir: Path | None) -> bool:+ """Return whether eval metadata should replace a native task's own input."""+ return "files" in entry or bool(input_files_dir is not None and input_files_dir.exists())+++def _remove_staged_path(path: Path) -> None:+ if _path_is_link_or_reparse(path) or not path.is_dir():+ path.unlink()+ else:+ shutil.rmtree(path)+++def _sanitize_staged_runtime_skills(skills_dir: Path) -> None:+ """Remove evaluator-only roots from authored or evaluator-copied skill bundles."""+ if not skills_dir.exists():+ return+ eval_roots: set[Path] = set()+ for manifest in _iter_skill_manifests(skills_dir):+ skill_root = manifest.parent+ if not skill_root.is_dir():+ continue+ eval_roots.update(child for child in skill_root.iterdir() if child.name.casefold() == "evals")+ for eval_root in sorted(eval_roots, key=lambda path: len(path.parts), reverse=True):+ if os.path.lexists(eval_root):+ _remove_staged_path(eval_root)++def _write_dockerfile(task_dir: Path,skill_path: Path | None,@@ -1505,6 +3279,8 @@ def _write_dockerfile(repo_context_skill_path: Path | None = None,repo_context_mode: str = "linked",compose_env_names: set[str] | None = None,+ repo_context_exclude_paths: Sequence[Path] = (),+ agent_workdir: str | None = None,) -> None:"""Generate a Dockerfile that installs skills into the container.@@ -1517,17 +3293,27 @@ def _write_dockerfile(env_dir = task_dir / "environment"env_dir.mkdir(parents=True, exist_ok=True)+ if not has_skill and skill_path is not None:+ _check_baseline_skill_candidates_do_not_alias_target(+ skill_path,+ reference_skills_dir,+ workspace_skill_paths,+ excluded_roots=repo_context_exclude_paths,+ )+skills_dir = env_dir / "skills"skills_dir.mkdir(parents=True, exist_ok=True)- def _ignore_results(directory, contents):- return [c for c in contents if c in ("results", "__pycache__", ".git")]-if has_skill and skill_path and skill_path.exists():dest = skills_dir / skill_path.nameif dest.exists():shutil.rmtree(dest)- copytree_secure(skill_path, dest, dirs_exist_ok=True, ignore=_ignore_results)+ copytree_secure(+ skill_path,+ dest,+ dirs_exist_ok=True,+ ignore=_runtime_skill_copy_ignore(skill_path, repo_context_exclude_paths),+ )if reference_skills_dir and reference_skills_dir.exists():for ref_skill in reference_skills_dir.iterdir():@@ -1536,7 +3322,12 @@ def _write_dockerfile(continuedest = skills_dir / ref_skill.nameif not dest.exists():- copytree_secure(ref_skill, dest, dirs_exist_ok=True, ignore=_ignore_results)+ copytree_secure(+ ref_skill,+ dest,+ dirs_exist_ok=True,+ ignore=_runtime_skill_copy_ignore(ref_skill, repo_context_exclude_paths),+ )for workspace_skill in workspace_skill_paths or []:if not workspace_skill.exists() or not workspace_skill.is_dir():@@ -1546,7 +3337,14 @@ def _write_dockerfile(dest = skills_dir / workspace_skill.nameif dest.exists():continue- copytree_secure(workspace_skill, dest, dirs_exist_ok=True, ignore=_ignore_results)+ copytree_secure(+ workspace_skill,+ dest,+ dirs_exist_ok=True,+ ignore=_runtime_skill_copy_ignore(workspace_skill, repo_context_exclude_paths),+ )++ _sanitize_staged_runtime_skills(skills_dir)include_input = Falseif entry is not None and skill_path is not None and evals_dir is not None:@@ -1564,7 +3362,10 @@ def _write_dockerfile(source_skill_path=repo_context_skill_path,mode=effective_repo_context_mode,exclude_source_skill=repo_context_mode == "full" and not has_skill,+ excluded_roots=repo_context_exclude_paths,)+ if not has_skill and skill_path is not None:+ _check_staged_baseline_does_not_contain_target(env_dir, skill_path)agent_config_lines = _write_agent_configs(env_dir)include_repo = (env_dir / "repo").exists()@@ -1577,7 +3378,7 @@ def _write_dockerfile(copytree_secure(custom_env_dir, staged_custom_env, allowed_root=skill_path)try:if not has_skill and skill_path:- _check_custom_environment_does_not_stage_target(staged_custom_env, skill_path.name)+ _check_custom_environment_does_not_stage_target(staged_custom_env, skill_path)custom_dockerfile = staged_custom_env / "Dockerfile"custom_dockerfile_accepted = False@@ -1590,10 +3391,8 @@ def _write_dockerfile(custom_dockerfile_accepted = Truelogger.debug("Using custom Dockerfile from %s", custom_env_dir / "Dockerfile")- compose_file = staged_custom_env / "docker-compose.yaml"- if not compose_file.exists():- compose_file = staged_custom_env / "docker-compose.yml"- if compose_file.exists():+ compose_file = _custom_compose_path(staged_custom_env)+ if compose_file is not None:shutil.copy2(compose_file, env_dir / "docker-compose.yaml")_validate_and_sanitize_custom_compose(env_dir / "docker-compose.yaml",@@ -1602,13 +3401,20 @@ def _write_dockerfile(logger.debug("Copied docker-compose.yaml from %s", custom_env_dir / compose_file.name)for subdir in staged_custom_env.iterdir():- if subdir.is_dir() and subdir.name not in ("__pycache__", ".git"):+ if subdir.is_dir() and subdir.name not in ("__pycache__", ".git") and subdir.name.casefold() != "input":dest = env_dir / subdir.nameif not dest.exists():copytree_secure(subdir, dest, allowed_root=staged_custom_env)logger.debug("Copied sidecar dir %s", subdir.name)+ _ensure_empty_compose_input_compatibility(env_dir, has_input=include_input)+if custom_dockerfile_accepted:+ _ensure_empty_custom_docker_input_compatibility(+ env_dir,+ env_dir / "Dockerfile",+ has_input=include_input,+ )if base_image and custom_dockerfile_mode == "rebase":dockerfile_path = env_dir / "Dockerfile"rebased = _rebase_custom_dockerfile_content(@@ -1618,6 +3424,7 @@ def _write_dockerfile(include_input=include_input,include_repo=include_repo,include_repo_linked_root=include_repo_linked_root,+ agent_workdir=agent_workdir,)if rebased is not None:rebased_content, original_from = rebased@@ -1634,6 +3441,7 @@ def _write_dockerfile(include_input=include_input,include_repo=include_repo,include_repo_linked_root=include_repo_linked_root,+ agent_workdir=agent_workdir,)returnfinally:@@ -1643,20 +3451,19 @@ def _write_dockerfile(dockerfile_lines = [f"FROM {base_image}","",- "COPY skills/ /workspace/skills/",- "",- "COPY skills/ /root/.claude/skills/",- "COPY skills/ /root/.agents/skills/",- "COPY skills/ /root/.config/opencode/skills/",+ *_runtime_copy_lines(+ agent_config_lines,+ include_input,+ include_repo=include_repo,+ include_repo_linked_root=include_repo_linked_root,+ agent_workdir=agent_workdir,+ ),]- dockerfile_lines.extend(agent_config_lines)-- if include_input:- dockerfile_lines.append("COPY input/ /workspace/input/")- if include_repo:- dockerfile_lines.append("COPY repo/ /workspace/repo/")- if include_repo_linked_root:- dockerfile_lines.append("COPY repo-linked-root/ /workspace/")+ _extend_task_input_projection(+ dockerfile_lines,+ include_input=include_input,+ agent_config_lines=agent_config_lines,+ )(env_dir / "Dockerfile").write_text("\n".join(dockerfile_lines) + "\n", encoding="utf-8")return@@ -1676,20 +3483,7 @@ def _write_dockerfile(dockerfile_lines.append(" " + " ".join(apt_pkgs) + " \\")dockerfile_lines.append(" && rm -rf /var/lib/apt/lists/*")- dockerfile_lines.extend(- [- "",- "RUN pip install --no-cache-dir \\",- " ragas~=0.4.0 \\",- " openai>=1.0 \\",- " anthropic>=0.40 \\",- " boto3>=1.34",- ]- )-- if pip_reqs:- escaped = " ".join(f'"{r}"' for r in pip_reqs)- dockerfile_lines.append(f"RUN pip install --no-cache-dir {escaped}")+ dockerfile_lines.extend(["", *_verifier_install_lines(pip_reqs)])dockerfile_lines.extend([@@ -1697,22 +3491,20 @@ def _write_dockerfile("RUN mkdir -p /workspace/skills /workspace/input /workspace/output \\"," /logs/verifier /logs/agent","",- "COPY skills/ /workspace/skills/",- "",- "# Multi-agent skill discovery paths",- "COPY skills/ /root/.claude/skills/",- "COPY skills/ /root/.agents/skills/",- "COPY skills/ /root/.config/opencode/skills/",+ *_runtime_copy_lines(+ agent_config_lines,+ include_input,+ include_repo=include_repo,+ include_repo_linked_root=include_repo_linked_root,+ agent_workdir=agent_workdir,+ ),])- dockerfile_lines.extend(agent_config_lines)-- if include_input:- dockerfile_lines.append("COPY input/ /workspace/input/")- if include_repo:- dockerfile_lines.append("COPY repo/ /workspace/repo/")- if include_repo_linked_root:- dockerfile_lines.append("COPY repo-linked-root/ /workspace/")+ _extend_task_input_projection(+ dockerfile_lines,+ include_input=include_input,+ agent_config_lines=agent_config_lines,+ )dockerfile_lines.extend([@@ -1732,19 +3524,22 @@ def _copy_skill_dirs(workspace_skill_paths: list[Path] | None,has_skill: bool,exclude_skill_name: str | None,+ excluded_roots: Sequence[Path] = (),) -> None:"""Stage target/workspace skills into a task environment."""skills_dir = env_dir / "skills"skills_dir.mkdir(parents=True, exist_ok=True)- def _ignore_results(directory, contents):- return [c for c in contents if c in ("results", "__pycache__", ".git")]-if has_skill and skill_path and skill_path.exists():dest = skills_dir / skill_path.nameif dest.exists():shutil.rmtree(dest)- copytree_secure(skill_path, dest, dirs_exist_ok=True, ignore=_ignore_results)+ copytree_secure(+ skill_path,+ dest,+ dirs_exist_ok=True,+ ignore=_runtime_skill_copy_ignore(skill_path, excluded_roots),+ )if reference_skills_dir and reference_skills_dir.exists():for ref_skill in reference_skills_dir.iterdir():@@ -1753,7 +3548,12 @@ def _copy_skill_dirs(continuedest = skills_dir / ref_skill.nameif not dest.exists():- copytree_secure(ref_skill, dest, dirs_exist_ok=True, ignore=_ignore_results)+ copytree_secure(+ ref_skill,+ dest,+ dirs_exist_ok=True,+ ignore=_runtime_skill_copy_ignore(ref_skill, excluded_roots),+ )for workspace_skill in workspace_skill_paths or []:if not workspace_skill.exists() or not workspace_skill.is_dir():@@ -1763,7 +3563,14 @@ def _copy_skill_dirs(dest = skills_dir / workspace_skill.nameif dest.exists():continue- copytree_secure(workspace_skill, dest, dirs_exist_ok=True, ignore=_ignore_results)+ copytree_secure(+ workspace_skill,+ dest,+ dirs_exist_ok=True,+ ignore=_runtime_skill_copy_ignore(workspace_skill, excluded_roots),+ )++ _sanitize_staged_runtime_skills(skills_dir)def _write_default_environment_dockerfile(@@ -1775,24 +3582,25 @@ def _write_default_environment_dockerfile(include_repo: bool = False,include_repo_linked_root: bool = False,include_verifier_deps: bool = True,+ agent_workdir: str | None = None,) -> None:if base_image:dockerfile_lines = [f"FROM {base_image}","",- "COPY skills/ /workspace/skills/",- "",- "COPY skills/ /root/.claude/skills/",- "COPY skills/ /root/.agents/skills/",- "COPY skills/ /root/.config/opencode/skills/",+ *_runtime_copy_lines(+ agent_config_lines,+ include_input,+ include_repo=include_repo,+ include_repo_linked_root=include_repo_linked_root,+ agent_workdir=agent_workdir,+ ),]- dockerfile_lines.extend(agent_config_lines)- if include_input:- dockerfile_lines.append("COPY input/ /workspace/input/")- if include_repo:- dockerfile_lines.append("COPY repo/ /workspace/repo/")- if include_repo_linked_root:- dockerfile_lines.append("COPY repo-linked-root/ /workspace/")+ _extend_task_input_projection(+ dockerfile_lines,+ include_input=include_input,+ agent_config_lines=agent_config_lines,+ )(env_dir / "Dockerfile").write_text("\n".join(dockerfile_lines) + "\n", encoding="utf-8")return@@ -1811,40 +3619,29 @@ def _write_default_environment_dockerfile(dockerfile_lines.append(" " + " ".join(apt_pkgs) + " \\")dockerfile_lines.append(" && rm -rf /var/lib/apt/lists/*")if include_verifier_deps:- dockerfile_lines.extend(- [- "",- "RUN pip install --no-cache-dir \\",- " ragas~=0.4.0 \\",- " openai>=1.0 \\",- " anthropic>=0.40 \\",- " boto3>=1.34",- ]- )- if pip_reqs:- escaped = " ".join(f'"{r}"' for r in pip_reqs)- dockerfile_lines.append(f"RUN pip install --no-cache-dir {escaped}")+ dockerfile_lines.extend(["", *_verifier_install_lines(pip_reqs)])+ elif pip_reqs:+ dockerfile_lines.append(_managed_run("python", "-m", "pip", "install", "--no-cache-dir", *pip_reqs))dockerfile_lines.extend(["","RUN mkdir -p /workspace/skills /workspace/input /workspace/output \\"," /logs/verifier /logs/agent","",- "COPY skills/ /workspace/skills/",- "",- "# Multi-agent skill discovery paths",- "COPY skills/ /root/.claude/skills/",- "COPY skills/ /root/.agents/skills/",- "COPY skills/ /root/.config/opencode/skills/",+ *_runtime_copy_lines(+ agent_config_lines,+ include_input,+ include_repo=include_repo,+ include_repo_linked_root=include_repo_linked_root,+ agent_workdir=agent_workdir,+ ),])- dockerfile_lines.extend(agent_config_lines)- if include_input:- dockerfile_lines.append("COPY input/ /workspace/input/")- if include_repo:- dockerfile_lines.append("COPY repo/ /workspace/repo/")- if include_repo_linked_root:- dockerfile_lines.append("COPY repo-linked-root/ /workspace/")+ _extend_task_input_projection(+ dockerfile_lines,+ include_input=include_input,+ agent_config_lines=agent_config_lines,+ )dockerfile_lines.extend(["", "WORKDIR /workspace"])(env_dir / "Dockerfile").write_text("\n".join(dockerfile_lines) + "\n", encoding="utf-8")@@ -1861,10 +3658,19 @@ def _prepare_native_environment(grading_mode: str,repo_context_mode: str = "linked",compose_env_names: set[str] | None = None,+ repo_context_exclude_paths: Sequence[Path] = (),+ agent_workdir: str | None = None,) -> None:"""Stage SkillEvaluator runtime additions into a copied native Harbor task."""env_dir = task_dir / "environment"env_dir.mkdir(parents=True, exist_ok=True)+ if not has_skill:+ _check_baseline_skill_candidates_do_not_alias_target(+ skill_path,+ reference_skills_dir,+ workspace_skill_paths,+ excluded_roots=repo_context_exclude_paths,+ )_copy_skill_dirs(env_dir=env_dir,skill_path=skill_path if has_skill else None,@@ -1872,6 +3678,7 @@ def _prepare_native_environment(workspace_skill_paths=workspace_skill_paths,has_skill=has_skill,exclude_skill_name=skill_path.name,+ excluded_roots=repo_context_exclude_paths,)agent_config_lines = _write_agent_configs(env_dir)effective_repo_context_mode = "full" if repo_context_mode == "full" else ("linked" if has_skill else "none")@@ -1880,19 +3687,30 @@ def _prepare_native_environment(source_skill_path=skill_path,mode=effective_repo_context_mode,exclude_source_skill=repo_context_mode == "full" and not has_skill,+ excluded_roots=repo_context_exclude_paths,)+ if not has_skill:+ _check_staged_baseline_does_not_contain_target(env_dir, skill_path)include_input = (env_dir / "input").exists()include_repo = (env_dir / "repo").exists()include_repo_linked_root = (env_dir / "repo-linked-root").exists()dockerfile_path = env_dir / "Dockerfile"- compose_path = env_dir / "docker-compose.yaml"- if compose_path.exists():+ compose_path = _custom_compose_path(env_dir)+ if compose_path is not None:_validate_and_sanitize_custom_compose(compose_path, allowed_env=compose_env_names or set())+ if compose_path.name == "docker-compose.yml":+ compose_path = compose_path.replace(env_dir / "docker-compose.yaml")+ _ensure_empty_compose_input_compatibility(env_dir, has_input=include_input)if dockerfile_path.exists():err = _validate_custom_dockerfile(dockerfile_path)if err:raise ValueError(f"{dockerfile_path}: {err}")+ _ensure_empty_custom_docker_input_compatibility(+ env_dir,+ dockerfile_path,+ has_input=include_input,+ )if base_image and custom_dockerfile_mode == "rebase":rebased = _rebase_custom_dockerfile_content(dockerfile_path.read_text(encoding="utf-8"),@@ -1901,6 +3719,7 @@ def _prepare_native_environment(include_input=include_input,include_repo=include_repo,include_repo_linked_root=include_repo_linked_root,+ agent_workdir=agent_workdir,)if rebased is not None:rebased_content, original_from = rebased@@ -1912,17 +3731,25 @@ def _prepare_native_environment()elif grading_mode == "custom_only":content = dockerfile_path.read_text(encoding="utf-8")- updated = _append_missing_lines(+ restore_user = _dockerfile_final_explicit_user(content)+ updated = _append_evaluator_runtime_lines(content,_runtime_copy_lines(agent_config_lines,include_input,include_repo=include_repo,include_repo_linked_root=include_repo_linked_root,+ agent_workdir=agent_workdir,),+ elevate=restore_user is not None,+ )+ updated = _append_task_input_projection(+ updated,+ include_input=include_input,+ agent_config_lines=agent_config_lines,+ restore_user=restore_user,)- if updated != content:- dockerfile_path.write_text(updated, encoding="utf-8")+ dockerfile_path.write_text(updated, encoding="utf-8")else:_ensure_verifier_deps(dockerfile_path,@@ -1930,6 +3757,7 @@ def _prepare_native_environment(include_input=include_input,include_repo=include_repo,include_repo_linked_root=include_repo_linked_root,+ agent_workdir=agent_workdir,)return@@ -1941,6 +3769,7 @@ def _prepare_native_environment(include_repo=include_repo,include_repo_linked_root=include_repo_linked_root,include_verifier_deps=grading_mode != "custom_only",+ agent_workdir=agent_workdir,)@@ -1967,6 +3796,60 @@ def _native_entry_id(task_dir: Path) -> str:return task_dir.name+def _native_task_workdir(task_dir: Path, *, allow_docker_image: bool = False) -> str | None:+ """Read and validate the workdir Harbor will use for a native task."""+ try:+ import tomllib++ data = tomllib.loads((task_dir / "task.toml").read_text(encoding="utf-8"))+ except Exception as exc:+ raise ValueError(f"Cannot parse native Harbor task config: {task_dir / 'task.toml'}") from exc+ environment = data.get("environment", {}) if isinstance(data, dict) else {}+ if not isinstance(environment, dict):+ raise ValueError(f"Native Harbor task [environment] must be a table: {task_dir / 'task.toml'}")+ environment_env = environment.get("env", {})+ if not isinstance(environment_env, dict):+ raise ValueError(f"Native Harbor task [environment.env] must be a table: {task_dir / 'task.toml'}")+ _validate_runtime_discovery_env(environment_env)+ _validate_runtime_loader_env(environment_env)+ skills_dir = environment.get("skills_dir")+ if skills_dir not in (None, "/workspace/skills"):+ raise ValueError(+ "Native Harbor task skills_dir must use the evaluator-controlled /workspace/skills projection: "+ f"{task_dir / 'task.toml'}"+ )+ docker_image = environment.get("docker_image")+ if docker_image not in (None, "") and not allow_docker_image:+ raise ValueError(+ "Native Harbor task docker_image bypasses the evaluator-controlled runtime projection; "+ "declare the image in environment/Dockerfile instead: "+ f"{task_dir / 'task.toml'}"+ )+ workdir = environment.get("workdir")+ if workdir is not None and not isinstance(workdir, str):+ raise ValueError(f"Native Harbor task workdir must be a string: {task_dir / 'task.toml'}")+ return _validated_agent_workdir(workdir)+++def _ensure_native_skills_dir(task_dir: Path) -> None:+ """Pin native tasks to the evaluator-owned runtime skill projection."""+ task_toml = task_dir / "task.toml"+ content = task_toml.read_text(encoding="utf-8")+ lines = content.splitlines()+ if "[environment]" in lines:+ environment_index = lines.index("[environment]")+ section_end = next(+ (index for index in range(environment_index + 1, len(lines)) if lines[index].strip().startswith("[")),+ len(lines),+ )+ if any(line.strip().startswith("skills_dir") for line in lines[environment_index + 1 : section_end]):+ return+ lines.insert(environment_index + 1, 'skills_dir = "/workspace/skills"')+ else:+ lines.extend(["", "[environment]", 'skills_dir = "/workspace/skills"'])+ task_toml.write_text("\n".join(lines) + "\n", encoding="utf-8")++def _ensure_skill_evaluator_verifier_env(task_dir: Path, *, verifier_env: dict[str, str] | None) -> None:"""Ensure staged native tasks forward configured public provider variables."""task_toml = task_dir / "task.toml"@@ -2019,8 +3902,7 @@ def _insert_table_block(lines: list[str], anchor: str, block: list[str]) -> Nonedef _ensure_environment_env(task_dir: Path, runtime_env: dict[str, str]) -> None:- if not runtime_env:- return+ runtime_env = {**runtime_env, **_EVALUATOR_MANAGED_RUNTIME_ENV}task_toml = task_dir / "task.toml"lines = task_toml.read_text(encoding="utf-8").splitlines()@@ -2122,13 +4004,20 @@ def _dockerfile_copy_or_add_mentions_skill(line: str, target_skill_name: str) ->return False-def _check_custom_environment_does_not_stage_target(custom_env_dir: Path, target_skill_name: str) -> None:- for skill_md in custom_env_dir.rglob("SKILL.md"):- if skill_md.parent.name == target_skill_name:- raise ValueError(- f"Custom eval environment already stages target skill '{target_skill_name}' at {skill_md}. "- "SkillEvaluator must control with-skill and baseline staging so Skill Lift stays uncontaminated."- )+def _check_custom_environment_does_not_stage_target(custom_env_dir: Path, target_skill: Path) -> None:+ target_skill_name = target_skill.name+ manifests = _iter_skill_manifests(custom_env_dir)+ if manifests:+ raise ValueError(+ f"Baseline custom eval environment contains an unmanaged skill package at {manifests[0]}. "+ "Configure reference/workspace skills through SkillEvaluator instead so baseline staging stays uncontaminated."+ )+ payload = _find_target_manifest_payload(custom_env_dir, target_skill)+ if payload is not None:+ raise ValueError(+ f"Baseline custom eval environment contains the target skill instructions in {payload}. "+ "The evaluated skill must not be embedded under an alias."+ )for dockerfile in custom_env_dir.rglob("Dockerfile"):try:lines = dockerfile.read_text(encoding="utf-8").splitlines()@@ -2142,14 +4031,50 @@ def _check_custom_environment_does_not_stage_target(custom_env_dir: Path, target)-def _check_native_source_does_not_stage_target(native_dir: Path, target_skill_name: str) -> None:- for skill_md in native_dir.rglob("SKILL.md"):- if skill_md.parent.name == target_skill_name and "environment" in skill_md.parts:+def _check_native_source_does_not_stage_target(+ native_dir: Path,+ target_skill: Path,+ *,+ with_skill: bool,+) -> None:+ ignored_parts = frozenset({"results", "__pycache__", ".git"})++ def _is_ignored(path: Path) -> bool:+ return bool(ignored_parts.intersection(path.relative_to(native_dir).parts))++ target_skill_name = target_skill.name+ for skill_md in _iter_skill_manifests(native_dir):+ if _is_ignored(skill_md):+ continue+ if "environment" not in skill_md.relative_to(native_dir).parts:+ continue+ if not with_skill:+ raise ValueError(+ f"Baseline native environment contains an unmanaged skill package at {skill_md}. "+ "Configure reference/workspace skills through SkillEvaluator instead."+ )+ if skill_md.parent.name == target_skill_name:raise ValueError(f"BYOT source already stages target skill '{target_skill_name}' at {skill_md}. ""SkillEvaluator must control with-skill and baseline staging.")+ if not with_skill:+ for environment_dir in native_dir.rglob("environment"):+ if not environment_dir.is_dir() or _is_ignored(environment_dir):+ continue+ payload = _find_target_manifest_payload(+ environment_dir,+ target_skill,+ ignored_parts=ignored_parts,+ )+ if payload is not None:+ raise ValueError(+ f"Baseline native environment contains the target skill instructions in {payload}. "+ "The evaluated skill must not be embedded under an alias."+ )for dockerfile in native_dir.rglob("Dockerfile"):+ if _is_ignored(dockerfile):+ continuetry:lines = dockerfile.read_text(encoding="utf-8").splitlines()except OSError:@@ -2174,6 +4099,7 @@ def _stage_native_harbor_tasks_into(base_image: str = "",custom_dockerfile_mode: str = "rebase",copy_repo: bool = False,+ repo_context_exclude_paths: Sequence[Path] = (),runtime_env: dict[str, str] | None = None,verifier_env: dict[str, str] | None = None,pre_agent_setup: list[str] | None = None,@@ -2185,20 +4111,38 @@ def _stage_native_harbor_tasks_into(The source tree is copied first and all SkillEvaluator injections happen only in thestaged result directory."""+ _validate_runtime_discovery_env(runtime_env)+ _validate_runtime_loader_env(runtime_env)native_dir = skill_path / "evals" / "harbor"if not native_dir.exists():raise FileNotFoundError(f"No native Harbor task source found at {native_dir}")- _check_native_source_does_not_stage_target(native_dir, skill_path.name)+ _validate_staging_output_location(+ skill_path,+ output_dir,+ reference_skills_dir=reference_skills_dir,+ workspace_skill_paths=workspace_skill_paths or (),+ )+ _validate_output_roots_outside_evaluator_sources(skill_path, repo_context_exclude_paths)+ _check_native_source_does_not_stage_target(native_dir, skill_path, with_skill=with_skill)+ entries_by_id = _load_entries_by_id(skill_path)+ source_task_dirs = _native_task_dirs(native_dir)+ if not source_task_dirs:+ raise ValueError(f"No Harbor task directories with task.toml found in {native_dir}")+ validate_case_ids(path.name for path in source_task_dirs)+ for source_task_dir in source_task_dirs:+ _native_task_workdir(source_task_dir)_ = task_resources_ = agent_workdir- entries_by_id = _load_entries_by_id(skill_path)if output_dir.exists():shutil.rmtree(output_dir)- copytree_secure(native_dir, output_dir, ignore=shutil.ignore_patterns("results", "__pycache__", ".git"))+ copytree_secure(+ native_dir,+ output_dir,+ ignore=shutil.ignore_patterns("results", "__pycache__", ".git", GENERATED_OUTPUT_MARKER),+ )+ output_dir.mkdir(parents=True, exist_ok=True)task_dirs = _native_task_dirs(output_dir)- if not task_dirs:- raise ValueError(f"No Harbor task directories with task.toml found in {native_dir}")if grading_mode in ("default", "default_plus_custom") and not entries_by_id:raise FileNotFoundError(@@ -2208,6 +4152,8 @@ def _stage_native_harbor_tasks_into(workspace_skill_names = sorted({p.name for p in workspace_skill_paths or []})for task_dir in task_dirs:entry_id = _native_entry_id(task_dir)+ native_agent_workdir = _native_task_workdir(task_dir)+ _ensure_native_skills_dir(task_dir)entry = entries_by_id.get(entry_id)if grading_mode in ("default", "default_plus_custom") and entry is None:raise ValueError(f"Native Harbor task '{entry_id}' has no matching entry in evals/evals.json")@@ -2260,6 +4206,14 @@ def _stage_native_harbor_tasks_into(f"custom_only native Harbor task '{entry_id}' requires tests/grader.py or tests/test.sh")+ if entry is not None and _entry_declares_task_input(entry, skill_path / "evals" / "files"):+ _stage_task_inputs(+ task_dir / "environment",+ input_files_dir=skill_path / "evals" / "files",+ entry=entry,+ source_skill_path=skill_path,+ evals_dir=skill_path / "evals",+ )_prepare_native_environment(task_dir,skill_path=skill_path,@@ -2271,6 +4225,8 @@ def _stage_native_harbor_tasks_into(grading_mode=grading_mode,repo_context_mode="full" if copy_repo else "linked",compose_env_names=set(runtime_env or {}),+ repo_context_exclude_paths=repo_context_exclude_paths,+ agent_workdir=native_agent_workdir,)if not (output_dir / "dataset.toml").exists():@@ -2291,6 +4247,7 @@ def stage_native_harbor_tasks(base_image: str = "",custom_dockerfile_mode: str = "rebase",copy_repo: bool = False,+ repo_context_exclude_paths: Sequence[Path] = (),runtime_env: dict[str, str] | None = None,verifier_env: dict[str, str] | None = None,pre_agent_setup: list[str] | None = None,@@ -2299,6 +4256,23 @@ def stage_native_harbor_tasks() -> list[Path]:"""Stage native tasks privately, then publish one exact output snapshot."""+ validate_output_provenance_key_location(+ skill_path,+ output_dir,+ reference_skills_dir=reference_skills_dir,+ workspace_skill_paths=workspace_skill_paths or (),+ )+ _validate_staging_output_location(+ skill_path,+ output_dir,+ reference_skills_dir=reference_skills_dir,+ workspace_skill_paths=workspace_skill_paths or (),+ declared_output_roots=repo_context_exclude_paths,+ )+ output_requires_provenance = _path_is_excluded(output_dir, (skill_path,))+ if output_requires_provenance:+ validate_generated_output_replacement(output_dir)+with tempfile.TemporaryDirectory(prefix="skillevaluator-native-tasks-") as temporary:private_root = Path(temporary).resolve(strict=True)private_output = private_root / "dataset"@@ -2313,6 +4287,7 @@ def stage_native_harbor_tasks(base_image=base_image,custom_dockerfile_mode=custom_dockerfile_mode,copy_repo=copy_repo,+ repo_context_exclude_paths=(*repo_context_exclude_paths, output_dir),runtime_env=runtime_env,verifier_env=verifier_env,pre_agent_setup=pre_agent_setup,@@ -2320,6 +4295,9 @@ def stage_native_harbor_tasks(agent_workdir=agent_workdir,)relative_tasks = [task.relative_to(private_output) for task in private_tasks]+ if output_requires_provenance:+ write_generated_output_marker(private_output, destination=output_dir)+ validate_generated_output_replacement(output_dir)copytree_secure(private_output,output_dir,@@ -2364,6 +4342,7 @@ def _generate_harbor_tasks_into(base_image: str = "",custom_dockerfile_mode: str = "rebase",copy_repo: bool = False,+ repo_context_exclude_paths: Sequence[Path] = (),runtime_env: dict[str, str] | None = None,verifier_env: dict[str, str] | None = None,pre_agent_setup: list[str] | None = None,@@ -2401,8 +4380,18 @@ def _generate_harbor_tasks_into(Returns:List of generated task directory paths."""+ _validate_runtime_discovery_env(runtime_env)+ _validate_runtime_loader_env(runtime_env)+ agent_workdir = _validated_agent_workdir(agent_workdir)evals_dir = skill_path / "evals"evals_file = find_evals_file(skill_path)+ _validate_staging_output_location(+ skill_path,+ output_dir,+ reference_skills_dir=reference_skills_dir,+ workspace_skill_paths=workspace_skill_paths or (),+ )+ _validate_output_roots_outside_evaluator_sources(skill_path, repo_context_exclude_paths)if not evals_file:raise FileNotFoundError(f"No evals dataset found in {evals_dir}")@@ -2475,6 +4464,8 @@ def _generate_harbor_tasks_into(repo_context_skill_path=skill_path,repo_context_mode="full" if copy_repo else "linked",compose_env_names=set(runtime_env or {}),+ repo_context_exclude_paths=repo_context_exclude_paths,+ agent_workdir=agent_workdir,)task_dirs.append(case_id)@@ -2505,6 +4496,7 @@ def generate_harbor_tasks(base_image: str = "",custom_dockerfile_mode: str = "rebase",copy_repo: bool = False,+ repo_context_exclude_paths: Sequence[Path] = (),runtime_env: dict[str, str] | None = None,verifier_env: dict[str, str] | None = None,pre_agent_setup: list[str] | None = None,@@ -2513,6 +4505,23 @@ def generate_harbor_tasks() -> list[Path]:"""Generate tasks privately, then publish one exact output snapshot."""+ validate_output_provenance_key_location(+ skill_path,+ output_dir,+ reference_skills_dir=reference_skills_dir,+ workspace_skill_paths=workspace_skill_paths or (),+ )+ _validate_staging_output_location(+ skill_path,+ output_dir,+ reference_skills_dir=reference_skills_dir,+ workspace_skill_paths=workspace_skill_paths or (),+ declared_output_roots=repo_context_exclude_paths,+ )+ output_requires_provenance = _path_is_excluded(output_dir, (skill_path,))+ if output_requires_provenance:+ validate_generated_output_replacement(output_dir)+with tempfile.TemporaryDirectory(prefix="skillevaluator-generated-tasks-") as temporary:private_root = Path(temporary).resolve(strict=True)private_output = private_root / "dataset"@@ -2527,6 +4536,7 @@ def generate_harbor_tasks(base_image=base_image,custom_dockerfile_mode=custom_dockerfile_mode,copy_repo=copy_repo,+ repo_context_exclude_paths=(*repo_context_exclude_paths, output_dir),runtime_env=runtime_env,verifier_env=verifier_env,pre_agent_setup=pre_agent_setup,@@ -2534,6 +4544,9 @@ def generate_harbor_tasks(agent_workdir=agent_workdir,)relative_tasks = [task.relative_to(private_output) for task in private_tasks]+ if output_requires_provenance:+ write_generated_output_marker(private_output, destination=output_dir)+ validate_generated_output_replacement(output_dir)copytree_secure(private_output,output_dir,diff --git a/src/skillevaluator/tier3/harbor/local_environment.py b/src/skillevaluator/tier3/harbor/local_environment.pyindex 958ff3a2..d82a5c1c 100644--- a/src/skillevaluator/tier3/harbor/local_environment.py+++ b/src/skillevaluator/tier3/harbor/local_environment.py@@ -33,6 +33,7 @@ from skillevaluator.tier3.harbor.local_runtime import ()from skillevaluator.tier3.harbor.secret_redaction import redact_secrets_in_log_linefrom skillevaluator.tier3.harbor.secure_copy import copytree_secure+from skillevaluator.tier3.output_provenance import output_provenance_key_path_SAFE_HOST_ENV = frozenset({@@ -534,6 +535,7 @@ class SkillEvaluatorLocalEnvironment(BaseEnvironment):allow_net=self._allow_net,extra_ro=self._runtime_ro_binds(),strict_reads=self._strict_reads,+ deny_reads=(output_provenance_key_path(),),)creation = asyncio.create_task(asyncio.create_subprocess_exec(diff --git a/src/skillevaluator/tier3/harbor/local_sandbox.py b/src/skillevaluator/tier3/harbor/local_sandbox.pyindex eb9e0a88..18013636 100644--- a/src/skillevaluator/tier3/harbor/local_sandbox.py+++ b/src/skillevaluator/tier3/harbor/local_sandbox.py@@ -189,6 +189,7 @@ class Sandbox:allow_net: bool,extra_ro: list[Path],strict_reads: bool = False,+ deny_reads: Iterable[Path] = (),) -> list[str]:if self.plan.backend == "bubblewrap":return _bwrap_argv(@@ -200,6 +201,7 @@ class Sandbox:allow_net=allow_net,extra_ro=extra_ro,strict_reads=strict_reads,+ deny_reads=deny_reads,)if self.plan.backend == "seatbelt":return _seatbelt_argv(@@ -210,6 +212,7 @@ class Sandbox:extra_ro=extra_ro,allow_net=allow_net,strict_reads=strict_reads,+ deny_reads=deny_reads,)return list(argv)@@ -484,6 +487,7 @@ def _bwrap_argv(allow_net: bool,extra_ro: list[Path],strict_reads: bool = False,+ deny_reads: Iterable[Path] = (),) -> list[str]:"""Build the bubblewrap launcher argv.@@ -543,6 +547,7 @@ def _bwrap_argv(wrapper += ["--bind", str(root), str(root)]read_roots = _validated_strict_read_roots(extra_ro) if strict_reads else extra_ro+ published_read_roots: list[Path] = []for ro_path in read_roots:raw = Path(ro_path).expanduser().absolute()try:@@ -563,6 +568,7 @@ def _bwrap_argv(# --ro-bind would re-mount that subtree read-only inside the writable# tree, so a skill writing there would get EROFS.if resolved.exists() and not any(_is_within(resolved, root) for root in resolved_write_roots):+ published_read_roots.extend((visible, resolved))ensure_bind_parents(visible)if visible.is_symlink():# Intermediate aliases are not published, so point the final@@ -571,6 +577,19 @@ def _bwrap_argv(else:wrapper += ["--ro-bind", str(resolved), str(visible)]+ published_roots = (*system_mount_roots, *resolved_write_roots, *published_read_roots)+ for denied_path in deny_reads:+ try:+ raw_denied = Path(denied_path).expanduser().absolute()+ denied_variants = {raw_denied, raw_denied.resolve(strict=True)}+ except (OSError, RuntimeError):+ continue+ for denied in denied_variants:+ if not any(_is_within(denied, root) for root in published_roots):+ continue+ ensure_bind_parents(denied)+ wrapper += ["--ro-bind", "/dev/null", str(denied)]+wrapper += ["--setenv","HOME",@@ -749,6 +768,7 @@ def _seatbelt_argv(extra_ro: list[Path],allow_net: bool,strict_reads: bool = False,+ deny_reads: Iterable[Path] = (),) -> list[str]:"""Build the macOS Seatbelt launcher argv.@@ -830,6 +850,14 @@ def _seatbelt_argv()read_policy = f"{home_read_rule}{absolute_read_denies}\n"+ explicit_read_denies = "\n".join(+ f'(deny file-read* (literal "{_sbpl_quote(path)}"))'+ for candidate in deny_reads+ for path in {Path(candidate).expanduser().absolute(), Path(candidate).expanduser().resolve(strict=False)}+ )+ if explicit_read_denies:+ read_policy = f"{read_policy}{explicit_read_denies}\n"+if allow_net:# Keep the host IPC boundary while permitting model-serving traffic.# macOS DNS uses this one system Unix socket; all other Unix sockets,diff --git a/src/skillevaluator/tier3/harbor/runner.py b/src/skillevaluator/tier3/harbor/runner.pyindex 420240a9..068e7e77 100644--- a/src/skillevaluator/tier3/harbor/runner.py+++ b/src/skillevaluator/tier3/harbor/runner.py@@ -17,7 +17,7 @@ import timeimport tomllibfrom collections.abc import Iterator, Mappingfrom concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, as_completed, wait-from contextlib import contextmanager+from contextlib import contextmanager, suppressfrom dataclasses import dataclassfrom datetime import UTC, datetimefrom functools import wraps@@ -25,6 +25,7 @@ from pathlib import Pathfrom queue import Empty, SimpleQueuefrom types import MappingProxyTypefrom typing import Any+from uuid import uuid4from skillevaluator.evaluation.tier3_report import render_agent_eval_html_reportfrom skillevaluator.provider_config import ProviderConfig, ProviderConfigurationError, resolve_llm_provider@@ -34,6 +35,8 @@ from skillevaluator.tier3.harbor.adapter import (find_evals_file,generate_harbor_tasks,stage_native_harbor_tasks,+ validate_output_provenance_key_location,+ validate_results_root_location,)from skillevaluator.tier3.harbor.artifact_retention import HarborArtifactLifecycle, RetentionOutcomefrom skillevaluator.tier3.harbor.collector import collect_harbor_results, validate_harbor_job_result@@ -49,6 +52,7 @@ from skillevaluator.tier3.harbor.progress import ()from skillevaluator.tier3.harbor.secure_copy import copytree_securefrom skillevaluator.tier3.harbor.secure_docker_environment import SECURE_DOCKER_ENV_IMPORT_PATH+from skillevaluator.tier3.output_provenance import mark_generated_output_rootfrom skillevaluator.tier3_environments import DEFAULT_ENV_MODE, ENV_MODE_LOCAL, HARBOR_ENV_MODESlogger = logging.getLogger(__name__)@@ -57,6 +61,27 @@ _NVIDIA_BUILD_FILE_SENTINEL = "skillevaluator-file-backed-nvidia-key"_NVIDIA_BUILD_KEY_FILE_ENV = "SKILLEVALUATOR_NVIDIA_API_KEY_FILE"_NVIDIA_BUILD_BRIDGED_AGENT_DEFAULT_MODEL = "nvidia/nemotron-3-super-120b-a12b"++def _reserve_run_dir(results_root: Path, timestamp: str) -> Path:+ """Atomically reserve and authenticate a unique run directory."""+ results_root.mkdir(parents=True, exist_ok=True)+ for _ in range(100):+ run_id = f"{timestamp}_{os.getpid()}_{uuid4().hex[:12]}"+ run_dir = results_root / run_id+ try:+ run_dir.mkdir()+ except FileExistsError:+ continue+ try:+ mark_generated_output_root(run_dir)+ except Exception:+ with suppress(OSError):+ run_dir.rmdir()+ raise+ return run_dir+ raise RuntimeError("Could not reserve a unique Tier 3 run directory")++_HARBOR_BASE_ENV_VARS = frozenset({"COMSPEC",@@ -145,10 +170,14 @@ _RUNTIME_ENV_HOST_CONTROL_NAMES = ("BASHOPTS","BASH_ENV","CDPATH",+ "CLAUDE_CODE_DISABLE_POLICY_SKILLS",+ "CLAUDE_CONFIG_DIR","CLASSPATH","COMSPEC",+ "CODEX_HOME","ENV","GCONV_PATH",+ "GEMINI_CLI_HOME","HOME","HOSTALIASES","HTTPS_PROXY",@@ -156,8 +185,12 @@ _RUNTIME_ENV_HOST_CONTROL_NAMES = ("IFS","JAVA_TOOL_OPTIONS","LOCPATH",+ "LUA_CPATH",+ "LUA_INIT",+ "LUA_PATH","NLSPATH","NO_PROXY",+ "OPENCODE_CONFIG_DIR","PATHEXT","PATH","PERL5LIB",@@ -165,6 +198,7 @@ _RUNTIME_ENV_HOST_CONTROL_NAMES = ("REQUESTS_CA_BUNDLE","RES_OPTIONS","RUBYOPT",+ "RUBYLIB","SHELLOPTS","SSLKEYLOGFILE","SSL_CERT_DIR",@@ -176,7 +210,9 @@ _RUNTIME_ENV_HOST_CONTROL_NAMES = ("TMPDIR","USERPROFILE","WINDIR",+ "XDG_CONFIG_HOME","XDG_RUNTIME_DIR",+ "ZDOTDIR","_JAVA_OPTIONS",})@@ -184,6 +220,7 @@ _RUNTIME_ENV_HOST_CONTROL_NAMES = (| frozenset().union(*_HARBOR_ENV_MODE_VARS.values()))_RUNTIME_ENV_HOST_CONTROL_PREFIXES = (+ "BASH_FUNC_","COMPOSE_","DOCKER_","DYLD_",@@ -1716,9 +1753,30 @@ def _run_harbor_eval_impl(reporter.emit(ProgressEvent(stage="with-skill-tasks", state="failed", detail="invalid task source"))return {"error": ["harbor.task_source must be auto, evals_json, or native_harbor"]}- root = output_dir or (skill_path / "evals" / "results")- run_id = datetime.now(UTC).strftime("%Y%m%d_%H%M%S")- run_dir = root / run_id+ root = Path(output_dir) if output_dir is not None else skill_path / "evals" / "results"+ try:+ validate_output_provenance_key_location(+ skill_path,+ root,+ reference_skills_dir=reference_skills_dir,+ workspace_skill_paths=workspace_skills,+ )+ validate_results_root_location(+ skill_path,+ root,+ reference_skills_dir=reference_skills_dir,+ workspace_skill_paths=workspace_skills,+ )+ except ValueError as exc:+ reporter.emit(ProgressEvent(stage="with-skill-tasks", state="failed", detail=str(exc)))+ return {"error": [str(exc)]}+ timestamp = datetime.now(UTC).strftime("%Y%m%d_%H%M%S")+ try:+ run_dir = _reserve_run_dir(root, timestamp)+ except (OSError, RuntimeError, ValueError) as exc:+ reporter.emit(ProgressEvent(stage="with-skill-tasks", state="failed", detail=str(exc)))+ return {"error": [str(exc)]}+ run_id = run_dir.namejobs_dir = run_dir / "_harbor-jobs"tasks_dir = run_dir / "_harbor-tasks"result_path = run_dir / "result.json"@@ -1753,6 +1811,7 @@ def _run_harbor_eval_impl(skill_path.resolve(),reference_skills_dir,workspace_skill_paths=workspace_skills,+ excluded_roots=(root,),force_rebuild=base_image_mode == "rebuild",)if base_image:@@ -1792,6 +1851,7 @@ def _run_harbor_eval_impl(base_image=base_image,custom_dockerfile_mode=dockerfile_mode,copy_repo=copy_repo,+ repo_context_exclude_paths=(root,),runtime_env=dict(runtime_plans[agent].staged_env),verifier_env=staged_verifier_env,pre_agent_setup=harbor_config.get("pre_agent_setup", []),@@ -1821,6 +1881,7 @@ def _run_harbor_eval_impl(base_image=base_image,custom_dockerfile_mode=dockerfile_mode,copy_repo=copy_repo,+ repo_context_exclude_paths=(root,),runtime_env=dict(runtime_plans[agent].staged_env),verifier_env=staged_verifier_env,pre_agent_setup=harbor_config.get("pre_agent_setup", []),diff --git a/src/skillevaluator/tier3/output_provenance.py b/src/skillevaluator/tier3/output_provenance.pynew file mode 100644index 00000000..4949abb1--- /dev/null+++ b/src/skillevaluator/tier3/output_provenance.py@@ -0,0 +1,434 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Path-bound provenance for evaluator-owned generated output trees."""++from __future__ import annotations++import base64+import contextlib+import hmac+import os+import secrets+import stat+import sys+import time+from pathlib import Path++from skillevaluator.tier3.case_ids import validate_output_directory_path++GENERATED_OUTPUT_MARKER = ".skillevaluator-generated-output"+OUTPUT_PROVENANCE_KEY_ENV = "SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE"++_KEY_BYTES = 32+_MAX_STORED_KEY_BYTES = 4096+_KEY_TEMP_PREFIX = ".output-provenance.key.tmp-"+_MARKER_PREFIX = b"SkillEvaluator generated output v2\n"+_MARKER_CONTEXT = b"skillevaluator.generated-output.v2\0"+_MARKER_SIZE = len(_MARKER_PREFIX) + len(base64.urlsafe_b64encode(bytes(_KEY_BYTES)).rstrip(b"=")) + 1+++def _is_link_or_reparse(metadata: os.stat_result) -> bool:+ reparse_flag = getattr(stat, "FILE_ATTRIBUTE_REPARSE_POINT", 0x400)+ return stat.S_ISLNK(metadata.st_mode) or bool(getattr(metadata, "st_file_attributes", 0) & reparse_flag)+++def _node_fingerprint(metadata: os.stat_result) -> tuple[int, int, int, int, int]:+ return metadata.st_dev, metadata.st_ino, metadata.st_mode, metadata.st_nlink, metadata.st_size+++def _protect_key_for_storage(key: bytes) -> bytes:+ """Bind key bytes to the current Windows user with DPAPI."""+ if os.name != "nt":+ return key+ import ctypes+ from ctypes import wintypes++ class _DataBlob(ctypes.Structure):+ _fields_ = [("cbData", wintypes.DWORD), ("pbData", ctypes.POINTER(ctypes.c_ubyte))]++ source = ctypes.create_string_buffer(key)+ source_blob = _DataBlob(len(key), ctypes.cast(source, ctypes.POINTER(ctypes.c_ubyte)))+ protected_blob = _DataBlob()+ crypt32 = ctypes.WinDLL("crypt32", use_last_error=True)+ kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)+ crypt32.CryptProtectData.restype = wintypes.BOOL+ kernel32.LocalFree.argtypes = [wintypes.HLOCAL]+ kernel32.LocalFree.restype = wintypes.HLOCAL+ if not crypt32.CryptProtectData(+ ctypes.byref(source_blob),+ None,+ None,+ None,+ None,+ 0x1,+ ctypes.byref(protected_blob),+ ):+ raise ValueError("Cannot protect output provenance key for the current Windows user") from ctypes.WinError(+ ctypes.get_last_error()+ )+ try:+ return ctypes.string_at(protected_blob.pbData, protected_blob.cbData)+ finally:+ kernel32.LocalFree(ctypes.cast(protected_blob.pbData, wintypes.HLOCAL))+++def _unprotect_stored_key(payload: bytes) -> bytes:+ """Decode DPAPI-protected Windows bytes or return POSIX bytes unchanged."""+ if os.name != "nt":+ return payload+ import ctypes+ from ctypes import wintypes++ class _DataBlob(ctypes.Structure):+ _fields_ = [("cbData", wintypes.DWORD), ("pbData", ctypes.POINTER(ctypes.c_ubyte))]++ source = ctypes.create_string_buffer(payload)+ source_blob = _DataBlob(len(payload), ctypes.cast(source, ctypes.POINTER(ctypes.c_ubyte)))+ key_blob = _DataBlob()+ crypt32 = ctypes.WinDLL("crypt32", use_last_error=True)+ kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)+ crypt32.CryptUnprotectData.restype = wintypes.BOOL+ kernel32.LocalFree.argtypes = [wintypes.HLOCAL]+ kernel32.LocalFree.restype = wintypes.HLOCAL+ if not crypt32.CryptUnprotectData(+ ctypes.byref(source_blob),+ None,+ None,+ None,+ None,+ 0x1,+ ctypes.byref(key_blob),+ ):+ raise ValueError("Output provenance key is not protected for the current Windows user") from ctypes.WinError(+ ctypes.get_last_error()+ )+ try:+ key = ctypes.string_at(key_blob.pbData, key_blob.cbData)+ finally:+ kernel32.LocalFree(ctypes.cast(key_blob.pbData, wintypes.HLOCAL))+ if len(key) != _KEY_BYTES:+ raise ValueError("Output provenance key has an invalid decrypted size")+ return key+++def _stored_key_size_is_valid(size: int) -> bool:+ return 0 < size <= _MAX_STORED_KEY_BYTES if os.name == "nt" else size == _KEY_BYTES+++def _default_key_path() -> Path:+ override = os.environ.get(OUTPUT_PROVENANCE_KEY_ENV)+ if override:+ return Path(override).expanduser().absolute()+ if os.name == "nt":+ base = Path(os.environ.get("LOCALAPPDATA", Path.home() / "AppData" / "Local"))+ elif sys.platform == "darwin":+ base = Path.home() / "Library" / "Application Support"+ else:+ base = Path(os.environ.get("XDG_STATE_HOME", Path.home() / ".local" / "state"))+ return base / "skillevaluator" / "output-provenance.key"+++def output_provenance_key_path() -> Path:+ """Return the configured private-key path without creating the key."""+ return _default_key_path()+++def _validate_key_metadata(path: Path, metadata: os.stat_result) -> None:+ if _is_link_or_reparse(metadata) or not stat.S_ISREG(metadata.st_mode) or metadata.st_nlink != 1:+ raise ValueError(f"Output provenance key must be a single-link regular file: {path}")+ if not _stored_key_size_is_valid(metadata.st_size):+ raise ValueError(f"Output provenance key has an invalid size: {path}")+ if os.name == "posix":+ if metadata.st_uid != os.getuid():+ raise ValueError(f"Output provenance key is not owned by the current user: {path}")+ if stat.S_IMODE(metadata.st_mode) & 0o077:+ raise ValueError(f"Output provenance key must not be accessible by group or other users: {path}")+++def _read_key(path: Path) -> bytes:+ before = path.lstat()+ _validate_key_metadata(path, before)+ flags = os.O_RDONLY | getattr(os, "O_CLOEXEC", 0) | getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_NONBLOCK", 0)+ descriptor = os.open(path, flags)+ try:+ opened = os.fstat(descriptor)+ _validate_key_metadata(path, opened)+ if _node_fingerprint(opened) != _node_fingerprint(before):+ raise ValueError(f"Output provenance key changed while it was opened: {path}")+ payload = os.read(descriptor, before.st_size + 1)+ after = os.fstat(descriptor)+ _validate_key_metadata(path, after)+ if _node_fingerprint(after) != _node_fingerprint(before):+ raise ValueError(f"Output provenance key changed while it was read: {path}")+ finally:+ os.close(descriptor)+ if len(payload) != before.st_size:+ raise ValueError(f"Output provenance key has an invalid size: {path}")+ return _unprotect_stored_key(payload)+++def _validate_key_directory(path: Path, metadata: os.stat_result) -> None:+ if _is_link_or_reparse(metadata) or not stat.S_ISDIR(metadata.st_mode):+ raise ValueError(f"Output provenance key directory must be a real directory: {path}")+ if os.name == "posix":+ if metadata.st_uid != os.getuid():+ raise ValueError(f"Output provenance key directory is not owned by the current user: {path}")+ if stat.S_IMODE(metadata.st_mode) & 0o077:+ raise ValueError(f"Output provenance key directory must not be accessible by group or other users: {path}")+++def _load_existing_key() -> bytes | None:+ path = _default_key_path()+ try:+ parent_metadata = path.parent.lstat()+ except FileNotFoundError:+ return None+ except OSError as exc:+ raise ValueError(f"Cannot inspect output provenance key directory: {path.parent}") from exc+ _validate_key_directory(path.parent, parent_metadata)+ try:+ return _read_key(path)+ except FileNotFoundError:+ return None+ except OSError as exc:+ raise ValueError(f"Cannot safely read output provenance key: {path}") from exc+++def _recover_interrupted_key_publish(path: Path) -> bool:+ """Remove only same-inode private temp links left by an interrupted publish."""+ try:+ target = path.lstat()+ except OSError:+ return False+ if _is_link_or_reparse(target) or not stat.S_ISREG(target.st_mode) or not _stored_key_size_is_valid(target.st_size):+ return False+ if os.name == "posix" and target.st_uid != os.getuid():+ return False+ recovered = False+ for candidate in path.parent.iterdir():+ if not candidate.name.startswith(_KEY_TEMP_PREFIX):+ continue+ try:+ metadata = candidate.lstat()+ if (+ not _is_link_or_reparse(metadata)+ and stat.S_ISREG(metadata.st_mode)+ and metadata.st_dev == target.st_dev+ and metadata.st_ino == target.st_ino+ ):+ candidate.unlink()+ recovered = True+ except OSError:+ continue+ return recovered+++def _read_key_after_concurrent_publish(path: Path) -> bytes:+ last_error: OSError | ValueError | None = None+ for _ in range(50):+ try:+ return _read_key(path)+ except (OSError, ValueError) as exc:+ last_error = exc+ _recover_interrupted_key_publish(path)+ time.sleep(0.002)+ if last_error is not None:+ raise last_error+ raise ValueError(f"Cannot safely read concurrently published output provenance key: {path}")+++def _load_or_create_key() -> bytes:+ path = _default_key_path()+ path.parent.mkdir(parents=True, mode=0o700, exist_ok=True)+ try:+ parent_metadata = path.parent.lstat()+ except OSError as exc:+ raise ValueError(f"Cannot inspect output provenance key directory: {path.parent}") from exc+ _validate_key_directory(path.parent, parent_metadata)+ try:+ return _read_key(path)+ except FileNotFoundError:+ pass+ except ValueError:+ return _read_key_after_concurrent_publish(path)+ except OSError as exc:+ raise ValueError(f"Cannot safely read output provenance key: {path}") from exc++ key = secrets.token_bytes(_KEY_BYTES)+ stored_key = _protect_key_for_storage(key)+ temporary = path.parent / f"{_KEY_TEMP_PREFIX}{os.getpid()}-{secrets.token_hex(8)}"+ flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, "O_CLOEXEC", 0) | getattr(os, "O_NOFOLLOW", 0)+ try:+ descriptor = os.open(temporary, flags, 0o600)+ with os.fdopen(descriptor, "wb", closefd=True) as handle:+ handle.write(stored_key)+ handle.flush()+ os.fsync(handle.fileno())+ try:+ os.link(temporary, path)+ except FileExistsError:+ return _read_key_after_concurrent_publish(path)+ with contextlib.suppress(FileNotFoundError):+ temporary.unlink()+ try:+ directory_descriptor = os.open(path.parent, os.O_RDONLY | getattr(os, "O_DIRECTORY", 0))+ except OSError:+ directory_descriptor = None+ if directory_descriptor is not None:+ try:+ os.fsync(directory_descriptor)+ finally:+ os.close(directory_descriptor)+ finally:+ with contextlib.suppress(FileNotFoundError):+ temporary.unlink()+ return _read_key(path)+++def _canonical_output_binding(path: Path) -> bytes:+ try:+ resolved = path.expanduser().resolve(strict=False)+ except (OSError, RuntimeError) as exc:+ raise ValueError(f"Cannot resolve generated output path for provenance: {path}") from exc+ normalized = os.path.normcase(os.path.normpath(os.fspath(resolved)))+ return os.fsencode(normalized)+++def _path_is_contained(path: Path, root: Path) -> bool:+ for candidate, candidate_root in (+ (path.expanduser().absolute(), root.expanduser().absolute()),+ (path.expanduser().resolve(strict=False), root.expanduser().resolve(strict=False)),+ ):+ normalized = Path(os.path.normcase(os.path.normpath(os.fspath(candidate))))+ normalized_root = Path(os.path.normcase(os.path.normpath(os.fspath(candidate_root))))+ try:+ normalized.relative_to(normalized_root)+ except ValueError:+ continue+ return True+ return False+++def validate_provenance_key_outside(workspace_root: Path) -> None:+ """Reject a private-key location that could enter an evaluated or generated tree."""+ try:+ inside_workspace = _path_is_contained(_default_key_path(), workspace_root)+ except (OSError, RuntimeError) as exc:+ raise ValueError(f"Cannot validate output provenance key location against: {workspace_root}") from exc+ if inside_workspace:+ raise ValueError(f"Output provenance key must be outside evaluated and generated trees: {workspace_root}")+++def _marker_payload(destination: Path, key: bytes) -> bytes:+ digest = hmac.digest(key, _MARKER_CONTEXT + _canonical_output_binding(destination), "sha256")+ signature = base64.urlsafe_b64encode(digest).rstrip(b"=")+ return _MARKER_PREFIX + signature + b"\n"+++def generated_output_marker_payload(destination: Path) -> bytes:+ """Return marker bytes signed for one canonical public destination."""+ validate_provenance_key_outside(destination)+ return _marker_payload(destination, _load_or_create_key())+++def _read_marker(path: Path, expected_size: int) -> bytes | None:+ try:+ before = path.lstat()+ except OSError:+ return None+ if (+ _is_link_or_reparse(before)+ or not stat.S_ISREG(before.st_mode)+ or before.st_nlink != 1+ or before.st_size != expected_size+ ):+ return None+ flags = os.O_RDONLY | getattr(os, "O_CLOEXEC", 0) | getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_NONBLOCK", 0)+ try:+ descriptor = os.open(path, flags)+ except OSError:+ return None+ try:+ opened = os.fstat(descriptor)+ if (+ _is_link_or_reparse(opened)+ or not stat.S_ISREG(opened.st_mode)+ or opened.st_nlink != 1+ or opened.st_size != expected_size+ or _node_fingerprint(opened) != _node_fingerprint(before)+ ):+ return None+ payload = os.read(descriptor, expected_size + 1)+ after = os.fstat(descriptor)+ if _node_fingerprint(after) != _node_fingerprint(before):+ return None+ return payload+ except OSError:+ return None+ finally:+ os.close(descriptor)+++def is_generated_output_root(path: Path) -> bool:+ """Return whether *path* has a valid marker bound to its canonical location."""+ observed = _read_marker(path / GENERATED_OUTPUT_MARKER, _MARKER_SIZE)+ if observed is None or not observed.startswith(_MARKER_PREFIX):+ return False+ validate_provenance_key_outside(path)+ key = _load_existing_key()+ if key is None:+ return False+ return hmac.compare_digest(observed, _marker_payload(path, key))+++def validate_generated_output_replacement(path: Path) -> None:+ """Allow replacement only for absent, empty, or authentically owned roots."""+ validate_output_directory_path(path)+ if not os.path.lexists(path):+ return+ try:+ metadata = path.lstat()+ except OSError as exc:+ raise ValueError(f"Cannot inspect generated output root before replacement: {path}") from exc+ if not stat.S_ISDIR(metadata.st_mode):+ raise ValueError(f"Generated output root must be a real directory: {path}")+ try:+ has_contents = next(path.iterdir(), None) is not None+ except OSError as exc:+ raise ValueError(f"Cannot inspect generated output root before replacement: {path}") from exc+ if has_contents and not is_generated_output_root(path):+ raise ValueError(f"Generated output marker is missing, invalid, or bound to another path: {path}")+++def write_generated_output_marker(root: Path, *, destination: Path | None = None) -> None:+ """Write provenance into a caller-owned tree, optionally for a later public path."""+ validate_provenance_key_outside(root)+ validate_output_directory_path(root)+ root.mkdir(parents=True, exist_ok=True)+ validate_output_directory_path(root)+ expected = generated_output_marker_payload(destination or root)+ marker = root / GENERATED_OUTPUT_MARKER+ try:+ descriptor = os.open(+ marker,+ os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, "O_CLOEXEC", 0) | getattr(os, "O_NOFOLLOW", 0),+ 0o600,+ )+ except FileExistsError:+ observed = _read_marker(marker, len(expected))+ if observed is None or not hmac.compare_digest(observed, expected):+ raise ValueError(f"Generated output marker is invalid or unsafe: {marker}") from None+ return+ with os.fdopen(descriptor, "wb", closefd=True) as handle:+ handle.write(expected)+ handle.flush()+ os.fsync(handle.fileno())+++def mark_generated_output_root(path: Path) -> None:+ """Claim or validate an output root before any generated child is replaced."""+ validate_generated_output_replacement(path)+ path.mkdir(parents=True, exist_ok=True)+ validate_output_directory_path(path)+ write_generated_output_marker(path)diff --git a/src/skillevaluator/tier3/results_location.py b/src/skillevaluator/tier3/results_location.pyindex 5f532b44..c6c21bb3 100644--- a/src/skillevaluator/tier3/results_location.py+++ b/src/skillevaluator/tier3/results_location.py@@ -12,7 +12,7 @@ from datetime import datetimefrom pathlib import PathENV_RESULTS_DIR = "SKILLEVALUATOR_RESULTS_DIR"-_RUN_TIMESTAMP_FORMATS = ("%Y%m%d_%H%M%S", "%Y-%m-%d_%H%M%S")+_RUN_TIMESTAMP_FORMATS = (("%Y%m%d_%H%M%S", 15), ("%Y-%m-%d_%H%M%S", 17))def _expand(path: str | Path) -> Path:@@ -91,9 +91,13 @@ def iter_candidate_results_roots(def _run_timestamp(name: str) -> datetime | None:- for timestamp_format in _RUN_TIMESTAMP_FORMATS:+ for timestamp_format, prefix_length in _RUN_TIMESTAMP_FORMATS:+ prefix = name[:prefix_length]+ suffix = name[prefix_length:]+ if suffix and not suffix.startswith("_"):+ continuetry:- return datetime.strptime(name, timestamp_format) # noqa: DTZ007 -- directory names have no timezone+ return datetime.strptime(prefix, timestamp_format) # noqa: DTZ007 -- directory names have no timezoneexcept ValueError:continuereturn None@@ -106,7 +110,7 @@ def _newest_completed_run(root: Path) -> Path | None:except OSError:return None- completed: list[tuple[datetime, Path]] = []+ completed: list[tuple[datetime, int, str, Path]] = []try:for candidate in children:timestamp = _run_timestamp(candidate.name)@@ -115,14 +119,16 @@ def _newest_completed_run(root: Path) -> Path | None:try:if not candidate.is_dir():continue- result = json.loads((candidate / "result.json").read_text(encoding="utf-8"))+ result_path = candidate / "result.json"+ result = json.loads(result_path.read_text(encoding="utf-8"))+ completion_mtime = result_path.stat().st_mtime_nsexcept (FileNotFoundError, json.JSONDecodeError, OSError):continueif isinstance(result, dict) and result.get("run_id") == candidate.name:- completed.append((timestamp, candidate))+ completed.append((timestamp, completion_mtime, candidate.name, candidate))except OSError:return None- return max(completed, default=(None, None), key=lambda item: item[0])[1]+ return max(completed, default=(None, -1, "", None), key=lambda item: (item[0], item[1], item[2]))[3]def resolve_latest_results(diff --git a/tests/conftest.py b/tests/conftest.pyindex b1efd948..0f184e6f 100644--- a/tests/conftest.py+++ b/tests/conftest.py@@ -10,6 +10,16 @@ from pathlib import Pathimport pytest+[email protected](autouse=True)+def _isolate_output_provenance_key(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:+ """Keep every test away from the developer's persistent provenance key."""+ monkeypatch.setenv(+ "SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE",+ str(tmp_path / ".skillevaluator-state" / "output-provenance.key"),+ )++# =============================================================================# SKILLS FIXTURES# =============================================================================diff --git a/tests/test_harbor_adapter_safety.py b/tests/test_harbor_adapter_safety.pyindex c61d899a..753fabd5 100644--- a/tests/test_harbor_adapter_safety.py+++ b/tests/test_harbor_adapter_safety.py@@ -227,10 +227,10 @@ def test_generated_task_rebases_custom_dockerfile_content(tmp_path: Path) -> Non)[0]dockerfile = (task / "environment" / "Dockerfile").read_text(encoding="utf-8")- assert dockerfile.startswith(- "FROM registry.example/eval-base:verified\n# SkillEvaluator: original base was FROM python:3.11-slim\n"+ assert dockerfile.startswith("FROM registry.example/eval-base:verified\n")+ assert dockerfile.index("RUN echo generated-custom-layer\n") < dockerfile.index(+ "# SkillEvaluator: original base was FROM python:3.11-slim\n")- assert "RUN echo generated-custom-layer\n" in dockerfiledef test_native_task_rebases_custom_dockerfile_content(tmp_path: Path) -> None:@@ -254,24 +254,22 @@ def test_native_task_rebases_custom_dockerfile_content(tmp_path: Path) -> None:)[0]dockerfile = (task / "environment" / "Dockerfile").read_text(encoding="utf-8")- assert dockerfile.startswith(- "FROM registry.example/eval-base:verified\n# SkillEvaluator: original base was FROM ubuntu:24.04\n"+ assert dockerfile.startswith("FROM registry.example/eval-base:verified\n")+ assert dockerfile.index("RUN echo native-custom-layer\n") < dockerfile.index(+ "# SkillEvaluator: original base was FROM ubuntu:24.04\n")- assert "RUN echo native-custom-layer\n" in dockerfile-def test_rebase_custom_dockerfile_content_without_from_is_unchanged() -> None:+def test_rebase_custom_dockerfile_content_without_from_fails_closed() -> None:content = "# comment only\nRUN echo unchanged\n"- assert (+ with pytest.raises(ValueError, match="exactly one FROM instruction"):_rebase_custom_dockerfile_content(content,"registry.example/eval-base:verified",agent_config_lines=[],include_input=False,)- is None- )@pytest.mark.parametrize("candidate", _GRADER_CANDIDATES, ids=str)diff --git a/tests/test_harbor_case_id_safety.py b/tests/test_harbor_case_id_safety.pyindex bbee4a1f..eb352f1f 100644--- a/tests/test_harbor_case_id_safety.py+++ b/tests/test_harbor_case_id_safety.py@@ -64,7 +64,6 @@ def _write_skill(tmp_path: Path, case_ids: list[object]) -> Path:pytest.param('case"next', id="quote"),pytest.param(".hidden", id="leading-dot"),pytest.param("case.", id="trailing-dot"),- pytest.param("case..part", id="consecutive-dots"),pytest.param("x" * 129, id="overlong"),pytest.param(None, id="null"),pytest.param(True, id="boolean"),@@ -127,6 +126,16 @@ def test_numeric_agentskills_id_is_canonicalized_for_harbor(tmp_path: Path) -> Nassert entry["id"] == "1"+def test_consecutive_dots_inside_case_id_are_safe(tmp_path: Path) -> None:+ skill_path = _write_skill(tmp_path, ["case..part"])+ output_dir = tmp_path / "generated"++ task_paths = generate_harbor_tasks(skill_path, output_dir)++ assert task_paths == [output_dir / "case..part"]+ assert (task_paths[0] / "task.toml").is_file()++def test_validate_evals_reports_unsafe_case_ids(tmp_path: Path) -> None:skill_path = _write_skill(tmp_path, ["safe-case", ".."])@@ -314,7 +323,7 @@ def test_all_dynamic_task_toml_strings_and_keys_are_quoted_for_real_harbor(tmp_poutput_dir,runtime_env={"SAFE_ENV": hostile},pre_agent_setup=[hostile],- agent_workdir=hostile,+ agent_workdir="/workspace/safe",)_write_task_toml(task_paths[0],diff --git a/tests/test_harbor_input_staging.py b/tests/test_harbor_input_staging.pyindex 0c2de145..29f202cb 100644--- a/tests/test_harbor_input_staging.py+++ b/tests/test_harbor_input_staging.py@@ -178,6 +178,76 @@ class TestStageTaskInputs:)assert staged is False+ def test_declared_file_symlink_to_undeclared_fixture_is_rejected(self, tmp_path: Path):+ skill, evals, env_dir = _make_skill(tmp_path)+ alias = evals / "files" / "alias.txt"+ try:+ alias.symlink_to("unrelated.txt")+ except OSError as exc: # pragma: no cover - host policy, primarily native Windows+ pytest.skip(f"symlinks unavailable on this host: {exc}")++ with pytest.raises(ValueError, match="symlink"):+ _stage_task_inputs(+ env_dir,+ input_files_dir=evals / "files",+ entry={"id": "t1", "files": ["files/alias.txt"]},+ source_skill_path=skill,+ evals_dir=evals,+ )++ def test_declared_directory_with_nested_symlink_is_rejected(self, tmp_path: Path):+ skill, evals, env_dir = _make_skill(tmp_path)+ declared = evals / "data" / "declared"+ declared.mkdir()+ alias = declared / "alias.txt"+ try:+ alias.symlink_to("../../files/unrelated.txt")+ except OSError as exc: # pragma: no cover - host policy, primarily native Windows+ pytest.skip(f"symlinks unavailable on this host: {exc}")++ with pytest.raises(ValueError, match="symlink"):+ _stage_task_inputs(+ env_dir,+ input_files_dir=evals / "files",+ entry={"id": "t1", "files": ["data/declared"]},+ source_skill_path=skill,+ evals_dir=evals,+ )++ def test_legacy_shared_directory_with_nested_symlink_is_rejected(self, tmp_path: Path):+ skill, evals, env_dir = _make_skill(tmp_path)+ alias = evals / "files" / "alias.txt"+ try:+ alias.symlink_to("unrelated.txt")+ except OSError as exc: # pragma: no cover - host policy, primarily native Windows+ pytest.skip(f"symlinks unavailable on this host: {exc}")++ with pytest.raises(ValueError, match="symlink"):+ _stage_task_inputs(+ env_dir,+ input_files_dir=evals / "files",+ entry={"id": "t1"},+ source_skill_path=skill,+ evals_dir=evals,+ )++ def test_declared_hardlink_is_rejected(self, tmp_path: Path):+ skill, evals, env_dir = _make_skill(tmp_path)+ alias = evals / "data" / "alias.txt"+ try:+ os.link(evals / "files" / "unrelated.txt", alias)+ except OSError as exc: # pragma: no cover - filesystem policy+ pytest.skip(f"hardlinks unavailable on this host: {exc}")++ with pytest.raises(ValueError, match=r"hardlink|hard link"):+ _stage_task_inputs(+ env_dir,+ input_files_dir=evals / "files",+ entry={"id": "t1", "files": ["data/alias.txt"]},+ source_skill_path=skill,+ evals_dir=evals,+ )+def test_generated_tasks_apply_per_entry_input_isolation(tmp_path: Path):skill, evals, _ = _make_skill(tmp_path)diff --git a/tests/test_harbor_local_mode.py b/tests/test_harbor_local_mode.pyindex f1629bda..46a4becf 100644--- a/tests/test_harbor_local_mode.py+++ b/tests/test_harbor_local_mode.py@@ -2627,6 +2627,7 @@ def test_strict_exec_filters_broad_prefixes_and_publishes_selected_npm_runtime(@staticmethoddef wrap(argv: list[str], **kwargs: object) -> list[str]:captured["extra_ro"] = kwargs["extra_ro"]+ captured["deny_reads"] = kwargs["deny_reads"]captured["wrapped"] = real_sandbox.wrap(argv, **kwargs) # type: ignore[arg-type]return argv@@ -2640,6 +2641,7 @@ def test_strict_exec_filters_broad_prefixes_and_publishes_selected_npm_runtime(extra_ro = captured["extra_ro"]assert isinstance(wrapped, list)assert isinstance(extra_ro, list)+ assert captured["deny_reads"] == (Path(os.environ["SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE"]),)ro_binds = {tuple(wrapped[index + 1 : index + 3]) for index, value in enumerate(wrapped) if value == "--ro-bind"}symlinks = {tuple(wrapped[index + 1 : index + 3]) for index, value in enumerate(wrapped) if value == "--symlink"}broad_roots = {"/", "/usr", "/usr/local", "/opt", "/tmp", "/private/tmp", "/var/tmp", str(Path.home())}diff --git a/tests/test_harbor_local_sandbox.py b/tests/test_harbor_local_sandbox.pyindex 176e6f61..560a294b 100644--- a/tests/test_harbor_local_sandbox.py+++ b/tests/test_harbor_local_sandbox.py@@ -1505,3 +1505,87 @@ else:finally:runtime_write_probe.unlink(missing_ok=True)shutil.rmtree(probe_dir, ignore_errors=True)+++def test_bwrap_masks_explicit_denied_key_inside_broad_read_root(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ broad = tmp_path / "broad"+ key = broad / "state" / "output-provenance.key"+ key.parent.mkdir(parents=True)+ key.write_text("secret", encoding="utf-8")+ run_root = tmp_path / "run"+ home = run_root / "home"+ sandbox_tmp = run_root / "tmp"+ home.mkdir(parents=True)+ sandbox_tmp.mkdir()+ monkeypatch.setattr(local_sandbox, "_SYSTEM_RO_PATHS", (str(broad),))++ argv = local_sandbox._bwrap_argv(+ ["/usr/bin/true"],+ workdir=run_root,+ write_roots=[run_root],+ home=home,+ tmp=sandbox_tmp,+ allow_net=False,+ extra_ro=[],+ deny_reads=[key],+ )++ ro_binds = [argv[index : index + 3] for index, value in enumerate(argv) if value == "--ro-bind"]+ assert ["--ro-bind", "/dev/null", str(key.resolve())] in ro_binds+++def test_seatbelt_profile_explicitly_denies_provenance_key(tmp_path: Path) -> None:+ key = tmp_path / "private" / "output-provenance.key"+ key.parent.mkdir()+ key.write_text("secret", encoding="utf-8")+ run_root = tmp_path / "run"+ home = run_root / "home"+ sandbox_tmp = run_root / "tmp"+ home.mkdir(parents=True)+ sandbox_tmp.mkdir()++ argv = local_sandbox._seatbelt_argv(+ ["/usr/bin/true"],+ write_roots=[run_root],+ tmp=sandbox_tmp,+ home=home,+ extra_ro=[key.parent],+ allow_net=False,+ deny_reads=[key],+ )++ profile = argv[2]+ assert f'(deny file-read* (literal "{local_sandbox._sbpl_quote(key.absolute())}"))' in profile+ assert f'(deny file-read* (literal "{local_sandbox._sbpl_quote(key.resolve())}"))' in profile+++ platform.system() != "Darwin" or not Path("/usr/bin/sandbox-exec").exists(),+ reason="requires macOS sandbox-exec",+)+def test_actual_seatbelt_blocks_provenance_key_read(tmp_path: Path) -> None:+ key = tmp_path / "private" / "output-provenance.key"+ key.parent.mkdir()+ key.write_text("secret", encoding="utf-8")+ run_root = tmp_path / "run"+ home = run_root / "home"+ sandbox_tmp = run_root / "tmp"+ home.mkdir(parents=True)+ sandbox_tmp.mkdir()+ argv = local_sandbox._seatbelt_argv(+ ["/bin/cat", str(key)],+ write_roots=[run_root],+ tmp=sandbox_tmp,+ home=home,+ extra_ro=[key.parent],+ allow_net=False,+ deny_reads=[key],+ )++ result = subprocess.run(argv, capture_output=True, text=True, check=False)++ assert result.returncode != 0+ assert "secret" not in result.stdoutdiff --git a/tests/test_harbor_output_provenance.py b/tests/test_harbor_output_provenance.pynew file mode 100644index 00000000..018faa5d--- /dev/null+++ b/tests/test_harbor_output_provenance.py@@ -0,0 +1,262 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Regression coverage for path-bound generated-output provenance."""++from __future__ import annotations++import json+import os+import shutil+import threading+from concurrent.futures import ThreadPoolExecutor+from pathlib import Path++import pytest++from skillevaluator.tier3 import output_provenance+from skillevaluator.tier3.harbor import secure_copy+from skillevaluator.tier3.harbor.adapter import generate_harbor_tasks, stage_native_harbor_tasks+from skillevaluator.tier3.output_provenance import (+ GENERATED_OUTPUT_MARKER,+ is_generated_output_root,+ mark_generated_output_root,+)++[email protected](autouse=True)+def _isolated_output_provenance_key(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:+ monkeypatch.setenv(+ "SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE",+ str(tmp_path / ".skillevaluator-state" / "output-provenance.key"),+ )+++def _write_skill(tmp_path: Path, *, native: bool = False) -> Path:+ skill = tmp_path / "skill"+ (skill / "evals").mkdir(parents=True)+ (skill / "SKILL.md").write_text("# Test skill\n", encoding="utf-8")+ (skill / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "Complete the task.", "files": []}]),+ encoding="utf-8",+ )+ if native:+ task = skill / "evals" / "harbor" / "case-001"+ task.mkdir(parents=True)+ (task / "instruction.md").write_text("Complete the native task.\n", encoding="utf-8")+ (task / "task.toml").write_text(+ 'schema_version = "1.3"\n\n[task]\nname = "nvidia/case-001"\n\n'+ '[metadata]\nentry_id = "case-001"\n\n[environment]\n',+ encoding="utf-8",+ )+ return skill+++def _in_skill_output(skill: Path, name: str) -> tuple[Path, Path]:+ declared_root = skill / name+ return declared_root, declared_root / "dataset"+++def test_concurrent_first_key_creation_publishes_one_complete_key(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ destination = tmp_path / "generated"+ barrier = threading.Barrier(2)+ original_link = output_provenance.os.link++ def synchronized_link(source: Path, target: Path) -> None:+ barrier.wait(timeout=5)+ original_link(source, target)++ monkeypatch.setattr(output_provenance.os, "link", synchronized_link)+ with ThreadPoolExecutor(max_workers=2) as executor:+ payloads = list(+ executor.map(+ lambda _index: output_provenance.generated_output_marker_payload(destination),+ range(2),+ )+ )++ key_path = Path(os.environ["SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE"])+ assert payloads[0] == payloads[1]+ if os.name == "nt":+ assert 32 < key_path.stat().st_size <= 4096+ else:+ assert key_path.stat().st_size == 32+ assert key_path.stat().st_nlink == 1+ assert not list(key_path.parent.glob(".output-provenance.key.tmp-*"))+++def test_interrupted_hardlink_publish_is_recovered(tmp_path: Path) -> None:+ key_path = Path(os.environ["SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE"])+ output_provenance.generated_output_marker_payload(tmp_path / "seed")+ temporary = key_path.parent / ".output-provenance.key.tmp-interrupted"+ os.link(key_path, temporary)++ payload = output_provenance.generated_output_marker_payload(tmp_path / "generated")++ assert payload.startswith(b"SkillEvaluator generated output v2\n")+ assert not temporary.exists()+ assert key_path.stat().st_nlink == 1+++def test_fixed_marker_cannot_authorize_authored_source_replacement(tmp_path: Path) -> None:+ skill = _write_skill(tmp_path)+ declared_root, output = _in_skill_output(skill, "fixed-marker-results")+ authored_task = output / "case-001"+ authored_task.mkdir(parents=True)+ (output / GENERATED_OUTPUT_MARKER).write_bytes(b"SkillEvaluator generated output v1\n")+ (authored_task / "SKILL.md").write_text("# Authored source\n", encoding="utf-8")+ sentinel = authored_task / "keep.txt"+ sentinel.write_text("preserve\n", encoding="utf-8")++ with pytest.raises(ValueError, match=r"runtime skill source|marker"):+ generate_harbor_tasks(skill, output, repo_context_exclude_paths=(declared_root,))++ assert sentinel.read_text(encoding="utf-8") == "preserve\n"+++def test_generated_marker_is_bound_to_its_original_destination(tmp_path: Path) -> None:+ skill = _write_skill(tmp_path)+ original_root, original = _in_skill_output(skill, "original-results")+ copied_root, copied = _in_skill_output(skill, "copied-results")+ generate_harbor_tasks(skill, original, repo_context_exclude_paths=(original_root,))+ copied.mkdir(parents=True)+ shutil.copy2(original / GENERATED_OUTPUT_MARKER, copied / GENERATED_OUTPUT_MARKER)+ authored_task = copied / "case-001"+ authored_task.mkdir()+ (authored_task / "SKILL.md").write_text("# Authored source\n", encoding="utf-8")+ sentinel = authored_task / "keep.txt"+ sentinel.write_text("preserve\n", encoding="utf-8")++ with pytest.raises(ValueError, match=r"runtime skill source|marker"):+ generate_harbor_tasks(skill, copied, repo_context_exclude_paths=(copied_root,))++ assert sentinel.read_text(encoding="utf-8") == "preserve\n"++[email protected]("native", [False, True])+def test_invalid_marker_blocks_in_skill_atomic_replacement(tmp_path: Path, native: bool) -> None:+ skill = _write_skill(tmp_path, native=native)+ declared_root, output = _in_skill_output(skill, "invalid-marker-results")+ output.mkdir(parents=True)+ (output / GENERATED_OUTPUT_MARKER).write_text("invalid\n", encoding="utf-8")+ sentinel = output / "keep.txt"+ sentinel.write_text("preserve\n", encoding="utf-8")+ stager = stage_native_harbor_tasks if native else generate_harbor_tasks++ with pytest.raises(ValueError, match="marker"):+ stager(skill, output, repo_context_exclude_paths=(declared_root,))++ assert sentinel.read_text(encoding="utf-8") == "preserve\n"+++def test_partial_atomic_publication_preserves_signed_output_and_reruns(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ skill = _write_skill(tmp_path)+ declared_root, output = _in_skill_output(skill, "partial-results")+ generate_harbor_tasks(skill, output, repo_context_exclude_paths=(declared_root,))+ old_dataset = (output / "dataset.toml").read_bytes()+ original_rename = secure_copy._rename_no_replace+ rename_calls = 0++ def _fail_second_rename(*args: object, **kwargs: object) -> None:+ nonlocal rename_calls+ rename_calls += 1+ if rename_calls == 2:+ raise OSError("injected generated-output publish failure after old destination moved")+ original_rename(*args, **kwargs)++ monkeypatch.setattr(secure_copy, "_rename_no_replace", _fail_second_rename)+ with pytest.raises(OSError, match="generated-output publish"):+ generate_harbor_tasks(skill, output, repo_context_exclude_paths=(declared_root,))++ assert (output / "dataset.toml").read_bytes() == old_dataset+ assert is_generated_output_root(output)++ monkeypatch.setattr(secure_copy, "_rename_no_replace", original_rename)+ tasks = generate_harbor_tasks(skill, output, repo_context_exclude_paths=(declared_root,))+ assert tasks+ assert is_generated_output_root(output)+++def test_external_output_retains_unmarked_overwrite_behavior(tmp_path: Path) -> None:+ skill = _write_skill(tmp_path)+ output = tmp_path / "external-output"+ output.mkdir()+ (output / GENERATED_OUTPUT_MARKER).write_text("invalid\n", encoding="utf-8")+ sentinel = output / "keep.txt"+ sentinel.write_text("replace\n", encoding="utf-8")++ tasks = generate_harbor_tasks(skill, output)++ assert tasks+ assert not sentinel.exists()+ assert not (output / GENERATED_OUTPUT_MARKER).exists()+++def test_unsigned_v2_marker_does_not_create_private_key(tmp_path: Path) -> None:+ skill = _write_skill(tmp_path)+ declared_root, output = _in_skill_output(skill, "unsigned-results")+ output.mkdir(parents=True)+ (output / GENERATED_OUTPUT_MARKER).write_bytes(b"SkillEvaluator generated output v2\n" + (b"A" * 43) + b"\n")+ sentinel = output / "keep.txt"+ sentinel.write_text("preserve\n", encoding="utf-8")+ key_path = Path(os.environ["SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE"])++ with pytest.raises(ValueError, match="marker"):+ generate_harbor_tasks(skill, output, repo_context_exclude_paths=(declared_root,))++ assert sentinel.read_text(encoding="utf-8") == "preserve\n"+ assert not key_path.exists()+++def test_key_override_inside_skill_is_rejected_before_mutation(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ skill = _write_skill(tmp_path)+ declared_root, output = _in_skill_output(skill, "key-location-results")+ key_path = skill / "private-output-key"+ monkeypatch.setenv("SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE", str(key_path))++ with pytest.raises(ValueError, match="outside evaluated and generated trees"):+ generate_harbor_tasks(skill, output, repo_context_exclude_paths=(declared_root,))++ assert not key_path.exists()+ assert not output.exists()++[email protected](not hasattr(os, "mkfifo"), reason="requires POSIX FIFO support")+def test_fifo_marker_and_key_are_rejected_without_blocking(tmp_path: Path) -> None:+ skill = _write_skill(tmp_path)+ declared_root, output = _in_skill_output(skill, "fifo-results")+ output.mkdir(parents=True)+ os.mkfifo(output / GENERATED_OUTPUT_MARKER)+ sentinel = output / "keep.txt"+ sentinel.write_text("preserve\n", encoding="utf-8")++ with pytest.raises(ValueError, match="marker"):+ generate_harbor_tasks(skill, output, repo_context_exclude_paths=(declared_root,))+ assert sentinel.read_text(encoding="utf-8") == "preserve\n"++ key_path = Path(os.environ["SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE"])+ key_path.parent.mkdir(parents=True, mode=0o700)+ os.mkfifo(key_path)+ with pytest.raises(ValueError, match="single-link regular file"):+ mark_generated_output_root(tmp_path / "key-fifo-output")+++def test_symlinked_private_marker_root_is_rejected(tmp_path: Path) -> None:+ target = tmp_path / "marker-target"+ target.mkdir()+ linked_root = tmp_path / "marker-link"+ linked_root.symlink_to(target, target_is_directory=True)++ with pytest.raises(ValueError, match=r"symlink|reparse|junction"):+ output_provenance.write_generated_output_marker(linked_root, destination=tmp_path / "destination")++ assert list(target.iterdir()) == []diff --git a/tests/test_harbor_runner_environment.py b/tests/test_harbor_runner_environment.pyindex 7d77e390..13549ac8 100644--- a/tests/test_harbor_runner_environment.py+++ b/tests/test_harbor_runner_environment.py@@ -454,6 +454,37 @@ def test_run_harbor_eval_stages_per_agent_credential_trees(assert launched["claude-code"][2]["NVIDIA_API_KEY"] == "provider-key"+def test_run_harbor_eval_rejects_provenance_key_inside_skill_before_creation(+ monkeypatch: pytest.MonkeyPatch,+ tmp_path,+) -> None:+ skill = tmp_path / "demo"+ skill.mkdir()+ key_path = skill / "private-output-provenance.key"+ monkeypatch.setenv("SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE", str(key_path))+ monkeypatch.setattr(runner, "resolve_llm_provider", lambda: _provider("nv_build"))+ monkeypatch.setattr(+ runner,+ "load_evals_config",+ lambda _path: ({"harbor": {"task_source": "evals_json"}}, None),+ )+ monkeypatch.setattr(runner, "find_evals_file", lambda _path: skill / "evals" / "evals.json")+ monkeypatch.setattr(runner, "_check_prerequisites", lambda **_kwargs: [])++ result = runner.run_harbor_eval(+ skill,+ ["opencode"],+ output_dir=tmp_path / "results",+ env_mode="docker",+ agent_runtime_preflight=False,+ )++ assert "error" in result+ assert "provenance key must be outside" in result["error"][0]+ assert not key_path.exists()+ assert not (tmp_path / "results").exists()++def test_harbor_subprocess_environment_excludes_arbitrary_host_secrets(monkeypatch: pytest.MonkeyPatch,tmp_path,@@ -573,9 +604,18 @@ def test_runtime_env_rejects_host_process_control_names() -> None:unsafe_names = ("PATH","PYTHONPATH",+ "HOME",+ "USERPROFILE",+ "XDG_CONFIG_HOME",+ "CLAUDE_CONFIG_DIR",+ "CLAUDE_CODE_DISABLE_POLICY_SKILLS",+ "CODEX_HOME",+ "GEMINI_CLI_HOME",+ "OPENCODE_CONFIG_DIR","LD_PRELOAD","DYLD_INSERT_LIBRARIES","BASH_ENV",+ "BASH_FUNC_hidden%%","NODE_OPTIONS","DOCKER_HOST","COMPOSE_FILE",diff --git a/tests/test_harbor_runtime_skill_isolation.py b/tests/test_harbor_runtime_skill_isolation.pynew file mode 100644index 00000000..0c0c3fac--- /dev/null+++ b/tests/test_harbor_runtime_skill_isolation.py@@ -0,0 +1,2367 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Regression coverage for agent-visible Tier 3 skill projection."""++from __future__ import annotations++import json+import os+import shutil+import subprocess+from pathlib import Path++import pytest++from skillevaluator.tier3.harbor.adapter import (+ _collect_all_skill_deps,+ _dockerfile_resolved_build_context_sources,+ _ensure_empty_custom_docker_input_compatibility,+ _path_is_excluded,+ generate_harbor_tasks,+ prebuild_task_environments,+ stage_native_harbor_tasks,+ validate_results_root_location,+)+from skillevaluator.tier3.output_provenance import mark_generated_output_root+++def _write_runtime_skill(path: Path, package: str) -> Path:+ path.mkdir(parents=True)+ (path / "SKILL.md").write_text(f"# {path.name}\n", encoding="utf-8")+ scripts = path / "scripts"+ scripts.mkdir()+ (scripts / "run.py").write_text("print('runtime')\n", encoding="utf-8")+ (scripts / "requirements.txt").write_text(f"{package}-runtime==1\n", encoding="utf-8")+ (scripts / "apt-packages.txt").write_text(f"{package}-runtime-tool\n", encoding="utf-8")++ nested_evals = path / "references" / "evals"+ nested_evals.mkdir(parents=True)+ (nested_evals / "keep.md").write_text("legitimate runtime reference\n", encoding="utf-8")+ (nested_evals / "requirements.txt").write_text(f"{package}-nested-runtime==1\n", encoding="utf-8")+ (nested_evals / "apt-packages.txt").write_text(f"{package}-nested-runtime-tool\n", encoding="utf-8")++ evals = path / "evals"+ evals.mkdir()+ (evals / "evals.json").write_text("[]\n", encoding="utf-8")+ (evals / "requirements.txt").write_text(f"{package}-oracle==9\n", encoding="utf-8")+ (evals / "apt-packages.txt").write_text(f"{package}-oracle-tool\n", encoding="utf-8")+ return path+++def _write_nested_runtime_skill(wrapper: Path, package: str) -> Path:+ nested = wrapper / "references" / "nested-skill"+ nested.mkdir(parents=True)+ (nested / "SKILL.md").write_text(f"# Nested {package}\n", encoding="utf-8")+ scripts = nested / "scripts"+ scripts.mkdir()+ (scripts / "requirements.txt").write_text(f"{package}-nested-skill-runtime==1\n", encoding="utf-8")+ (scripts / "apt-packages.txt").write_text(f"{package}-nested-skill-runtime-tool\n", encoding="utf-8")+ evals = nested / "evals"+ evals.mkdir()+ (evals / "evals.json").write_text('[{"ground_truth": "NESTED-ORACLE"}]\n', encoding="utf-8")+ (evals / "requirements.txt").write_text(f"{package}-nested-skill-oracle==9\n", encoding="utf-8")+ (evals / "apt-packages.txt").write_text(f"{package}-nested-skill-oracle-tool\n", encoding="utf-8")+ preserved = nested / "references" / "evals"+ preserved.mkdir(parents=True)+ (preserved / "keep.md").write_text("nested runtime reference\n", encoding="utf-8")+ return nested+++def _write_minimal_native_task(target: Path) -> None:+ native_task = target / "evals" / "harbor" / "case-001"+ native_task.mkdir(parents=True)+ (native_task / "instruction.md").write_text("Run the native case.\n", encoding="utf-8")+ (native_task / "task.toml").write_text(+ 'schema_version = "1.3"\n\n[task]\nname = "nvidia/case-001"\n\n'+ '[metadata]\nentry_id = "case-001"\n\n[environment]\n',+ encoding="utf-8",+ )+++def _write_projection_fixture(tmp_path: Path) -> tuple[Path, Path, Path, Path]:+ repo = tmp_path / "repo"+ target = _write_runtime_skill(repo / "target-skill", "target")+ references_dir = repo / "reference-skills"+ _write_runtime_skill(references_dir / "reference-skill", "reference")+ workspace = _write_runtime_skill(repo / "workspace-skill", "workspace")++ docs_evals = repo / "docs" / "evals"+ docs_evals.mkdir(parents=True)+ (docs_evals / "keep.txt").write_text("not a skill dataset\n", encoding="utf-8")++ evals = target / "evals"+ files = evals / "files"+ files.mkdir()+ (files / "selected.txt").write_text("SELECTED-FIXTURE\n", encoding="utf-8")+ (files / "hidden.txt").write_text("UNDECLARED-ORACLE\n", encoding="utf-8")+ (evals / "evals.json").write_text(+ json.dumps(+ [+ {+ "id": "case-001",+ "question": "Use only the selected fixture.",+ "files": ["evals/files/selected.txt"],+ "ground_truth": "GROUND-TRUTH-SECRET",+ }+ ]+ ),+ encoding="utf-8",+ )+ (evals / "grader.py").write_text("def grade(*args, **kwargs):\n return 1\n", encoding="utf-8")+ sidecar = evals / "environment" / "sidecar"+ sidecar.mkdir(parents=True)+ (sidecar / "seed.txt").write_text("custom environment asset\n", encoding="utf-8")++ return repo, target, references_dir, workspace+++def _assert_runtime_projection(task: Path, target: Path, workspace: Path) -> None:+ env = task / "environment"+ staged_skills = env / "skills"+ for name in (target.name, "reference-skill", workspace.name):+ staged = staged_skills / name+ assert (staged / "SKILL.md").is_file()+ assert (staged / "scripts" / "run.py").is_file()+ assert not (staged / "evals").exists()+ assert (staged / "references" / "evals" / "keep.md").is_file()++ staged_repo = env / "repo"+ assert not (staged_repo / target.name / "evals").exists()+ assert not (staged_repo / "reference-skills" / "reference-skill" / "evals").exists()+ assert not (staged_repo / workspace.name / "evals").exists()+ assert (staged_repo / target.name / "references" / "evals" / "keep.md").is_file()+ assert (staged_repo / "docs" / "evals" / "keep.txt").is_file()++ readable_environment = "\n".join(+ path.read_text(encoding="utf-8", errors="ignore") for path in env.rglob("*") if path.is_file()+ )+ assert "UNDECLARED-ORACLE" not in readable_environment+ assert "GROUND-TRUTH-SECRET" not in readable_environment+ assert (target / "evals" / "evals.json").is_file(), "source evaluation assets must remain intact"+++def test_generated_tasks_project_runtime_skills_without_root_evals(tmp_path: Path) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)++ task = generate_harbor_tasks(+ target,+ tmp_path / "generated",+ with_skill=True,+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ copy_repo=True,+ )[0]++ _assert_runtime_projection(task, target, workspace)+ assert (task / "environment" / "input" / "selected.txt").read_text(encoding="utf-8") == "SELECTED-FIXTURE\n"+ assert not (task / "environment" / "input" / "hidden.txt").exists()+ assert (task / "tests" / "grader.py").is_file()+ assert "GROUND-TRUTH-SECRET" in (task / "tests" / "entry.json").read_text(encoding="utf-8")+ assert (task / "environment" / "sidecar" / "seed.txt").is_file()+ dockerfile = (task / "environment" / "Dockerfile").read_text(encoding="utf-8")+ assert "target-runtime==1" in dockerfile+ assert "target-oracle==9" not in dockerfile+++def test_native_tasks_project_runtime_skills_without_root_evals(tmp_path: Path) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ native_task = target / "evals" / "harbor" / "case-001"+ native_environment = native_task / "environment"+ native_environment.mkdir(parents=True)+ (native_environment / "native.txt").write_text("native environment asset\n", encoding="utf-8")+ prebundled = native_environment / "skills" / "prebundled-skill"+ (prebundled / "evals").mkdir(parents=True)+ (prebundled / "skill.md").write_text("# Prebundled\n", encoding="utf-8")+ (prebundled / "evals" / "secret.txt").write_text("PREBUNDLED-ORACLE\n", encoding="utf-8")+ (prebundled / "references" / "evals").mkdir(parents=True)+ (prebundled / "references" / "evals" / "keep.txt").write_text("keep\n", encoding="utf-8")+ nested_prebundled = _write_nested_runtime_skill(prebundled, "prebundled")+ (native_task / "instruction.md").write_text("Run the native case.\n", encoding="utf-8")+ (native_task / "task.toml").write_text(+ 'schema_version = "1.3"\n\n[task]\nname = "nvidia/case-001"\n\n'+ '[metadata]\nentry_id = "case-001"\n\n[environment]\n',+ encoding="utf-8",+ )++ task = stage_native_harbor_tasks(+ target,+ tmp_path / "native",+ with_skill=True,+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ copy_repo=True,+ )[0]++ _assert_runtime_projection(task, target, workspace)+ assert (task / "environment" / "input" / "selected.txt").read_text(encoding="utf-8") == "SELECTED-FIXTURE\n"+ assert not (task / "environment" / "input" / "hidden.txt").exists()+ assert (task / "instruction.md").read_text(encoding="utf-8") == "Run the native case.\n"+ assert (task / "environment" / "native.txt").is_file()+ assert not (task / "environment" / "skills" / "prebundled-skill" / "evals").exists()+ assert (task / "environment" / "skills" / "prebundled-skill" / "references" / "evals" / "keep.txt").is_file()+ staged_nested_prebundled = (+ task / "environment" / "skills" / "prebundled-skill" / nested_prebundled.relative_to(prebundled)+ )+ assert (staged_nested_prebundled / "SKILL.md").is_file()+ assert not (staged_nested_prebundled / "evals").exists()+ assert (staged_nested_prebundled / "references" / "evals" / "keep.md").is_file()+ assert (task / "tests" / "entry.json").is_file()+ assert (task / "tests" / "grader.py").is_file()+++def test_base_image_dependency_collection_ignores_only_root_evals(tmp_path: Path) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)++ pip_reqs, apt_packages = _collect_all_skill_deps(target, references_dir, [workspace])++ assert pip_reqs == [+ "reference-nested-runtime==1",+ "reference-runtime==1",+ "target-nested-runtime==1",+ "target-runtime==1",+ "workspace-nested-runtime==1",+ "workspace-runtime==1",+ ]+ assert apt_packages == [+ "reference-nested-runtime-tool",+ "reference-runtime-tool",+ "target-nested-runtime-tool",+ "target-runtime-tool",+ "workspace-nested-runtime-tool",+ "workspace-runtime-tool",+ ]+++def test_runtime_projection_excludes_nested_skill_evals_and_dependencies(tmp_path: Path) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ wrappers = [target, references_dir / "reference-skill", workspace]+ nested_skills = [+ _write_nested_runtime_skill(wrapper, package)+ for wrapper, package in zip(wrappers, ("target", "reference", "workspace"), strict=True)+ ]++ task = generate_harbor_tasks(+ target,+ tmp_path / "nested-skill-evals",+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ )[0]++ staged_wrappers = [+ task / "environment" / "skills" / target.name,+ task / "environment" / "skills" / "reference-skill",+ task / "environment" / "skills" / workspace.name,+ ]+ for nested, wrapper, staged_wrapper in zip(nested_skills, wrappers, staged_wrappers, strict=True):+ staged_nested = staged_wrapper / nested.relative_to(wrapper)+ assert (staged_nested / "SKILL.md").is_file()+ assert not (staged_nested / "evals").exists()+ assert (staged_nested / "references" / "evals" / "keep.md").is_file()++ pip_reqs, apt_packages = _collect_all_skill_deps(target, references_dir, [workspace])+ for package in ("target", "reference", "workspace"):+ assert f"{package}-nested-skill-runtime==1" in pip_reqs+ assert f"{package}-nested-skill-runtime-tool" in apt_packages+ assert f"{package}-nested-skill-oracle==9" not in pip_reqs+ assert f"{package}-nested-skill-oracle-tool" not in apt_packages+++def test_base_image_dependency_collection_does_not_follow_alias_into_root_evals(tmp_path: Path) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ alias_dir = target / "scripts" / "eval-alias"+ alias_dir.mkdir()+ try:+ (alias_dir / "requirements.txt").symlink_to("../../evals/requirements.txt")+ (alias_dir / "apt-packages.txt").symlink_to("../../evals/apt-packages.txt")+ except OSError as exc: # pragma: no cover - host policy, primarily native Windows+ pytest.skip(f"symlinks unavailable on this host: {exc}")++ pip_reqs, apt_packages = _collect_all_skill_deps(target, references_dir, [workspace])++ assert "target-oracle==9" not in pip_reqs+ assert "target-oracle-tool" not in apt_packages+++def test_base_image_dependency_collection_matches_ignored_runtime_projection(tmp_path: Path) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ archived = target / "results"+ archived.mkdir()+ (archived / "requirements.txt").write_text("archived-oracle==9\n", encoding="utf-8")+ (archived / "apt-packages.txt").write_text("archived-oracle-tool\n", encoding="utf-8")++ pip_reqs, apt_packages = _collect_all_skill_deps(target, references_dir, [workspace])++ assert "archived-oracle==9" not in pip_reqs+ assert "archived-oracle-tool" not in apt_packages++[email protected]("link_kind", ["symlink", "hardlink"])+def test_runtime_skill_projection_rejects_linked_host_file(tmp_path: Path, link_kind: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ outside = tmp_path / "host-secret.txt"+ outside.write_text("HOST-SECRET\n", encoding="utf-8")+ alias = target / "scripts" / "host-secret.txt"+ try:+ if link_kind == "symlink":+ alias.symlink_to(outside)+ else:+ alias.hardlink_to(outside)+ except OSError as exc: # pragma: no cover - filesystem policy+ pytest.skip(f"{link_kind}s unavailable on this host: {exc}")++ with pytest.raises(ValueError, match=r"symlink|hard.?link"):+ generate_harbor_tasks(target, tmp_path / f"runtime-{link_kind}")+++def test_copy_repo_does_not_follow_alias_into_skill_root_evals(tmp_path: Path) -> None:+ repo, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ alias = repo / "oracle-alias.txt"+ try:+ alias.symlink_to(target / "evals" / "files" / "hidden.txt")+ except OSError as exc: # pragma: no cover - host policy, primarily native Windows+ pytest.skip(f"symlinks unavailable on this host: {exc}")++ task = generate_harbor_tasks(+ target,+ tmp_path / "generated-alias",+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ copy_repo=True,+ )[0]++ assert not (task / "environment" / "repo" / alias.name).exists()+++def test_copy_repo_ignores_lexical_skill_evals_alias_to_repo_file(tmp_path: Path) -> None:+ repo, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ secret = repo / ".env"+ secret.write_text("TOKEN=repo-secret\n", encoding="utf-8")+ alias = target / "evals" / "repo-secret-alias"+ try:+ alias.symlink_to(secret)+ except OSError as exc: # pragma: no cover - host policy, primarily native Windows+ pytest.skip(f"symlinks unavailable on this host: {exc}")++ task = generate_harbor_tasks(+ target,+ tmp_path / "generated-lexical-alias",+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ copy_repo=True,+ )[0]++ assert not (task / "environment" / "repo" / target.name / "evals" / alias.name).exists()+++def test_copy_repo_rejects_hardlink_alias(tmp_path: Path) -> None:+ repo, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ alias = repo / "oracle-hardlink.txt"+ try:+ alias.hardlink_to(target / "evals" / "files" / "hidden.txt")+ except OSError as exc: # pragma: no cover - filesystem policy+ pytest.skip(f"hardlinks unavailable on this host: {exc}")++ with pytest.raises(ValueError, match=r"hardlink|hard link"):+ generate_harbor_tasks(+ target,+ tmp_path / "generated-hardlink",+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ copy_repo=True,+ )+++def test_linked_repo_ignores_lexical_skill_evals_alias(tmp_path: Path) -> None:+ repo, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ secret = repo / "linked-secret.txt"+ secret.write_text("LINKED-REPO-SECRET\n", encoding="utf-8")+ alias = target / "evals" / "linked-secret-alias"+ try:+ alias.symlink_to(secret)+ except OSError as exc: # pragma: no cover - host policy, primarily native Windows+ pytest.skip(f"symlinks unavailable on this host: {exc}")+ manifest = target / "SKILL.md"+ manifest.write_text(+ manifest.read_text(encoding="utf-8") + "\n[hidden](evals/linked-secret-alias)\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(+ target,+ tmp_path / "linked-lexical-alias",+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ copy_repo=False,+ )[0]++ environment = task / "environment"+ assert "LINKED-REPO-SECRET" not in "\n".join(+ path.read_text(encoding="utf-8", errors="ignore") for path in environment.rglob("*") if path.is_file()+ )+++def test_copy_repo_filters_lowercase_skill_manifest_evals(tmp_path: Path) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ lowercase_skill = target.parent / "lowercase-skill"+ (lowercase_skill / "evals").mkdir(parents=True)+ (lowercase_skill / "skill.md").write_text("# Lowercase\n", encoding="utf-8")+ (lowercase_skill / "evals" / "secret.txt").write_text("LOWERCASE-ORACLE\n", encoding="utf-8")++ task = generate_harbor_tasks(+ target,+ tmp_path / "lowercase-filter",+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ copy_repo=True,+ )[0]++ assert not (task / "environment" / "repo" / lowercase_skill.name / "evals").exists()+++def test_copy_repo_excludes_configured_in_repo_output_root(tmp_path: Path) -> None:+ repo, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ output_dir = repo / "custom-results"+ leaked_entry = output_dir / "prior-task" / "tests" / "entry.json"+ leaked_entry.parent.mkdir(parents=True)+ leaked_entry.write_text('{"ground_truth": "PRIOR-RUN-SECRET"}\n', encoding="utf-8")++ task = generate_harbor_tasks(+ target,+ output_dir,+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ copy_repo=True,+ )[0]++ assert not (task / "environment" / "repo" / output_dir.name).exists()+++def test_runtime_skill_projection_excludes_shared_in_skill_output_root(tmp_path: Path) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ output_root = target / "Custom-Results"+ configured_output_root = target / "custom-results"+ first_output = output_root / "agent-a"+ generate_harbor_tasks(+ target,+ first_output,+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ repo_context_exclude_paths=(configured_output_root,),+ )++ second = generate_harbor_tasks(+ target,+ output_root / "agent-b",+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ repo_context_exclude_paths=(configured_output_root,),+ )[0]++ assert not (second / "environment" / "skills" / target.name / output_root.name).exists()+ pip_reqs, apt_packages = _collect_all_skill_deps(+ target,+ references_dir,+ [workspace],+ excluded_roots=(configured_output_root,),+ )+ assert all("custom-results" not in dep.casefold() for dep in [*pip_reqs, *apt_packages])+++def test_copy_repo_excludes_case_only_output_alias(tmp_path: Path) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ prior_output = target / "Custom-Results"+ prior_output.mkdir()+ (prior_output / "prior.json").write_text("PRIOR-RESULT-ORACLE\n", encoding="utf-8")++ task = generate_harbor_tasks(+ target,+ tmp_path / "casefold-copy-repo",+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ copy_repo=True,+ repo_context_exclude_paths=(target / "custom-results",),+ )[0]++ environment = task / "environment"+ assert not (environment / "skills" / target.name / "Custom-Results").exists()+ assert not (environment / "repo" / target.name / "Custom-Results").exists()++[email protected]("stager", ["generated", "native"])+def test_declared_in_skill_output_can_be_regenerated(tmp_path: Path, stager: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ if stager == "native":+ _write_minimal_native_task(target)+ output_root = target / "custom-results"+ output_dir = output_root / "dataset"+ stager_fn = generate_harbor_tasks if stager == "generated" else stage_native_harbor_tasks++ first = stager_fn(target, output_dir, repo_context_exclude_paths=(output_root,))+ second = stager_fn(target, output_dir, repo_context_exclude_paths=(output_root,))++ assert first and second+ assert (target / "SKILL.md").is_file()+ assert (output_dir / "dataset.toml").is_file()+++def test_broad_output_ancestor_does_not_empty_runtime_skill(tmp_path: Path) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)++ task = generate_harbor_tasks(+ target,+ tmp_path / "ancestor-output",+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ repo_context_exclude_paths=(tmp_path,),+ )[0]++ staged_target = task / "environment" / "skills" / target.name+ assert (staged_target / "SKILL.md").is_file()+ assert (staged_target / "scripts" / "run.py").is_file()+ pip_reqs, _ = _collect_all_skill_deps(target, references_dir, [workspace], excluded_roots=(tmp_path,))+ assert "target-runtime==1" in pip_reqs++[email protected]("source_name", ["files", "environment"])+def test_generated_rejects_output_root_inside_evaluator_source(tmp_path: Path, source_name: str) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ unsafe_root = target / "evals" / source_name / "configured-results"++ with pytest.raises(ValueError, match="output root must not be inside evaluator source"):+ generate_harbor_tasks(+ target,+ tmp_path / f"unsafe-generated-{source_name}",+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ repo_context_exclude_paths=(unsafe_root,),+ )+++def test_native_rejects_output_root_inside_evaluator_source(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_minimal_native_task(target)+ unsafe_root = target / "evals" / "harbor" / "configured-results"++ with pytest.raises(ValueError, match="output root must not be inside evaluator source"):+ stage_native_harbor_tasks(+ target,+ tmp_path / "unsafe-native-output",+ repo_context_exclude_paths=(unsafe_root,),+ )+++def test_default_results_root_remains_supported(tmp_path: Path) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)++ tasks = generate_harbor_tasks(+ target,+ tmp_path / "safe-default-results",+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ repo_context_exclude_paths=(target / "evals" / "results",),+ )++ assert tasks+++def test_declared_custom_in_skill_results_root_remains_supported(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)++ validate_results_root_location(target, target / "custom-results")+ with pytest.raises(ValueError, match="runtime skill source"):+ validate_results_root_location(target, target / "scripts" / "results")++[email protected]("source_alias", ["FILES", "Environment", "HARBOR"])+def test_results_root_rejects_case_alias_of_evaluator_source(tmp_path: Path, source_alias: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)++ with pytest.raises(ValueError, match="output root must not be inside evaluator source"):+ validate_results_root_location(target, target / "evals" / source_alias / "prior-results")+++def test_excluded_root_matching_is_case_insensitive(tmp_path: Path) -> None:+ actual = tmp_path / "Custom-Results" / "agent-a"+ typed_alias = tmp_path / "custom-results"++ assert _path_is_excluded(actual, (typed_alias,))+++def test_prebuild_uses_each_tasks_exact_environment(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ dataset = tmp_path / "dataset"+ build_calls: list[tuple[str, Path]] = []+ for case_id, fixture in (("case-a", "ALPHA"), ("case-b", "BETA")):+ task = dataset / case_id+ environment = task / "environment"+ (environment / "input").mkdir(parents=True)+ (environment / "Dockerfile").write_text(+ "FROM python:3.12-slim\nCOPY input/ /workspace/input/\n",+ encoding="utf-8",+ )+ (environment / "input" / "fixture.txt").write_text(fixture, encoding="utf-8")+ (task / "task.toml").write_text("[environment]\n", encoding="utf-8")++ class _Result:+ def __init__(self, returncode: int) -> None:+ self.returncode = returncode+ self.stderr = ""+ self.stdout = ""++ def _fake_run(command: list[str], **_kwargs: object) -> _Result:+ if command[:3] == ["docker", "image", "inspect"]:+ return _Result(1)+ assert command[:3] == ["docker", "build", "-t"]+ build_calls.append((command[3], Path(command[4])))+ return _Result(0)++ monkeypatch.setenv("SKILL_EVAL_HARBOR_PREBUILD_TASK_ENVS", "1")+ monkeypatch.setattr("skillevaluator.tier3.harbor.adapter.subprocess.run", _fake_run)++ assert prebuild_task_environments([dataset]) == 2+ assert {path.name for _, path in build_calls} == {"environment"}+ assert {path.parent.name for _, path in build_calls} == {"case-a", "case-b"}+ assert len({tag for tag, _ in build_calls}) == 2+ task_tags = {+ task.name: next(+ line for line in (task / "task.toml").read_text(encoding="utf-8").splitlines() if "docker_image" in line+ )+ for task in sorted(dataset.iterdir())+ }+ assert task_tags["case-a"] != task_tags["case-b"]++ first_build_tags = {path.parent.name: tag for tag, path in build_calls}+ (dataset / "case-a" / "environment" / "input" / "fixture.txt").chmod(0o755)+ for task in dataset.iterdir():+ (task / "task.toml").write_text("[environment]\n", encoding="utf-8")+ build_calls.clear()++ assert prebuild_task_environments([dataset]) == 2+ second_build_tags = {path.parent.name: tag for tag, path in build_calls}+ assert second_build_tags["case-a"] != first_build_tags["case-a"]+ assert second_build_tags["case-b"] == first_build_tags["case-b"]+++def test_prebuild_skips_task_with_compose_build_override(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:+ monkeypatch.setenv("SKILL_EVAL_HARBOR_PREBUILD_TASK_ENVS", "1")+ dataset = tmp_path / "dataset"+ task = dataset / "case-a"+ environment = task / "environment"+ environment.mkdir(parents=True)+ (environment / "Dockerfile").write_text("FROM python:3.12-slim\n", encoding="utf-8")+ (environment / "docker-compose.yaml").write_text(+ "services:\n main:\n build:\n context: ./alternate\n",+ encoding="utf-8",+ )+ (task / "task.toml").write_text("[environment]\n", encoding="utf-8")+ calls: list[list[str]] = []++ def _unexpected_run(command: list[str], **_kwargs: object) -> subprocess.CompletedProcess[str]:+ calls.append(command)+ return subprocess.CompletedProcess(command, 0, "", "")++ monkeypatch.setattr("skillevaluator.tier3.harbor.adapter.subprocess.run", _unexpected_run)++ assert prebuild_task_environments([dataset]) == 0+ assert calls == []+ assert "docker_image" not in (task / "task.toml").read_text(encoding="utf-8")++[email protected]("stager", ["generated", "native"])[email protected]("output_kind", ["evals", "skill"])+def test_staging_rejects_output_that_contains_evaluator_sources_before_mutation(+ tmp_path: Path,+ stager: str,+ output_kind: str,+) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ if stager == "native":+ _write_minimal_native_task(target)+ output_dir = target / "evals" if output_kind == "evals" else target++ with pytest.raises(ValueError, match="Staging output directory overlaps evaluator source"):+ if stager == "generated":+ generate_harbor_tasks(target, output_dir)+ else:+ stage_native_harbor_tasks(target, output_dir)++ assert (target / "SKILL.md").is_file()+ assert (target / "evals" / "evals.json").is_file()++[email protected]("stager", ["generated", "native"])[email protected]("eval_subdir", ["data", "tests"])+def test_staging_rejects_output_inside_any_evaluator_source_subtree(+ tmp_path: Path,+ stager: str,+ eval_subdir: str,+) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ if stager == "native":+ _write_minimal_native_task(target)+ output_dir = target / "evals" / eval_subdir+ output_dir.mkdir(exist_ok=True)+ sentinel = output_dir / "keep.txt"+ sentinel.write_text("preserve\n", encoding="utf-8")++ with pytest.raises(ValueError, match="Staging output directory overlaps evaluator source"):+ if stager == "generated":+ generate_harbor_tasks(target, output_dir)+ else:+ stage_native_harbor_tasks(target, output_dir)++ assert sentinel.read_text(encoding="utf-8") == "preserve\n"++[email protected]("stager", ["generated", "native"])[email protected]("alias", ["Results", "RESULTS"])+def test_staging_rejects_case_alias_of_reserved_results_directory(+ tmp_path: Path,+ stager: str,+ alias: str,+) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ if stager == "native":+ _write_minimal_native_task(target)+ output_dir = target / "evals" / alias+ output_dir.mkdir()+ sentinel = output_dir / "keep.txt"+ sentinel.write_text("preserve\n", encoding="utf-8")++ with pytest.raises(ValueError, match="evaluator source"):+ stager_fn = generate_harbor_tasks if stager == "generated" else stage_native_harbor_tasks+ stager_fn(target, output_dir)++ assert sentinel.read_text(encoding="utf-8") == "preserve\n"++[email protected]("stager", ["generated", "native", "results-root"])+def test_output_path_rejects_parent_traversal_before_evaluator_source_mutation(tmp_path: Path, stager: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ if stager == "native":+ _write_minimal_native_task(target)+ sentinel = target / "evals" / "files" / "selected.txt"+ output_dir = target / "evals" / "results" / ".." / "files"++ with pytest.raises(ValueError, match="parent traversal"):+ if stager == "generated":+ generate_harbor_tasks(target, output_dir)+ elif stager == "native":+ stage_native_harbor_tasks(target, output_dir)+ else:+ validate_results_root_location(target, output_dir)++ assert sentinel.read_text(encoding="utf-8") == "SELECTED-FIXTURE\n"++[email protected]("stager", ["generated", "native"])[email protected]("source_kind", ["reference", "reference-parent", "workspace", "workspace-parent"])+def test_staging_rejects_output_overlapping_supplied_skill_sources(+ tmp_path: Path,+ stager: str,+ source_kind: str,+) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ if stager == "native":+ _write_minimal_native_task(target)+ reference_parent = tmp_path / "external-reference-source"+ references = reference_parent / "references"+ reference = _write_runtime_skill(references / "reference", "reference-external")+ workspace_parent = tmp_path / "external-workspace-source"+ workspace = _write_runtime_skill(workspace_parent / "workspace", "workspace-external")+ output_dir = {+ "reference": reference,+ "reference-parent": reference_parent,+ "workspace": workspace,+ "workspace-parent": workspace_parent,+ }[source_kind]+ marker = output_dir / "source-marker.txt"+ marker.write_text("preserve\n", encoding="utf-8")++ with pytest.raises(ValueError, match="Staging output directory overlaps runtime skill source"):+ stager_fn = generate_harbor_tasks if stager == "generated" else stage_native_harbor_tasks+ stager_fn(+ target,+ output_dir,+ reference_skills_dir=references,+ workspace_skill_paths=[workspace],+ )++ assert marker.read_text(encoding="utf-8") == "preserve\n"++[email protected]("stager", ["generated", "native"])+def test_staging_rejects_undeclared_output_inside_target_runtime_tree(tmp_path: Path, stager: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ runtime_source = target / "src"+ runtime_source.mkdir()+ marker = runtime_source / "runtime.py"+ marker.write_text("print('preserve')\n", encoding="utf-8")+ if stager == "native":+ _write_minimal_native_task(target)++ with pytest.raises(ValueError, match="Staging output directory overlaps runtime skill source"):+ stager_fn = generate_harbor_tasks if stager == "generated" else stage_native_harbor_tasks+ stager_fn(target, runtime_source)++ assert marker.read_text(encoding="utf-8") == "print('preserve')\n"++[email protected]("stager", ["generated", "native"])+def test_declared_output_cannot_replace_standard_runtime_source(tmp_path: Path, stager: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ scripts = target / "scripts"+ marker = scripts / "run.py"+ if stager == "native":+ _write_minimal_native_task(target)++ with pytest.raises(ValueError, match="runtime skill source"):+ stager_fn = generate_harbor_tasks if stager == "generated" else stage_native_harbor_tasks+ stager_fn(target, scripts, repo_context_exclude_paths=(scripts,))++ assert marker.read_text(encoding="utf-8") == "print('runtime')\n"++[email protected]("stager", ["generated", "native"])+def test_declared_output_root_cannot_contain_authored_nested_skill(tmp_path: Path, stager: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ declared_root = target / "addons"+ nested_skill = declared_root / "helper"+ nested_skill.mkdir(parents=True)+ (nested_skill / "SKILL.md").write_text("# Helper\n", encoding="utf-8")+ marker = nested_skill / "runtime.txt"+ marker.write_text("preserve\n", encoding="utf-8")+ if stager == "native":+ _write_minimal_native_task(target)++ with pytest.raises(ValueError, match="output root must not overlap runtime skill source"):+ stager_fn = generate_harbor_tasks if stager == "generated" else stage_native_harbor_tasks+ stager_fn(+ target,+ declared_root / "generated" / "dataset",+ repo_context_exclude_paths=(declared_root,),+ )++ assert marker.read_text(encoding="utf-8") == "preserve\n"+++def test_declared_output_does_not_trust_authored_dataset_manifest(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ declared_root = target / "addons"+ nested_skill = declared_root / "helper"+ nested_skill.mkdir(parents=True)+ (declared_root / "dataset.toml").write_text('[dataset]\nname = "authored"\n', encoding="utf-8")+ (nested_skill / "SKILL.md").write_text("# Authored helper\n", encoding="utf-8")+ sentinel = nested_skill / "runtime.txt"+ sentinel.write_text("preserve\n", encoding="utf-8")+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "helper", "question": "Do not replace authored content."}]),+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="overlaps runtime skill source"):+ generate_harbor_tasks(target, declared_root, repo_context_exclude_paths=(declared_root,))++ assert sentinel.read_text(encoding="utf-8") == "preserve\n"+++def test_marked_prior_run_does_not_block_declared_results_root(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ results_root = target / "custom-results"+ prior_run = results_root / "run-old"+ mark_generated_output_root(prior_run)+ retained_skill = prior_run / "jobs" / "case-001" / "workspace" / "skills" / target.name+ retained_skill.mkdir(parents=True)+ (retained_skill / "SKILL.md").write_text("# Retained generated copy\n", encoding="utf-8")++ validate_results_root_location(target, results_root)+++def test_declared_in_skill_output_recovers_after_partial_generation_failure(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps(+ [+ {"id": "case-a", "question": "Stage existing.", "files": ["evals/files/selected.txt"]},+ {"id": "case-b", "question": "Stage missing.", "files": ["evals/files/later.txt"]},+ ]+ ),+ encoding="utf-8",+ )+ declared_root = target / "custom-results"+ output_dir = declared_root / "dataset"++ with pytest.raises(FileNotFoundError, match="does not exist"):+ generate_harbor_tasks(target, output_dir, repo_context_exclude_paths=(declared_root,))++ (target / "evals" / "files" / "later.txt").write_text("LATER\n", encoding="utf-8")+ tasks = generate_harbor_tasks(target, output_dir, repo_context_exclude_paths=(declared_root,))++ assert {task.name for task in tasks} == {"case-a", "case-b"}+ assert (output_dir / ".skillevaluator-generated-output").is_file()++[email protected]("stager", ["generated", "native"])+def test_staging_rejects_symlinked_output_parent_without_replacing_target(tmp_path: Path, stager: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ if stager == "native":+ _write_minimal_native_task(target)+ victim_parent = tmp_path / "victim-parent"+ victim_output = victim_parent / "dataset"+ victim_output.mkdir(parents=True)+ sentinel = victim_output / "keep.txt"+ sentinel.write_text("preserve\n", encoding="utf-8")+ alias = tmp_path / "output-alias"+ alias.symlink_to(victim_parent, target_is_directory=True)++ with pytest.raises(ValueError, match=r"symlink|reparse|junction"):+ stager_fn = generate_harbor_tasks if stager == "generated" else stage_native_harbor_tasks+ stager_fn(target, alias / "dataset")++ assert sentinel.read_text(encoding="utf-8") == "preserve\n"+++def test_legacy_input_rejects_symlinked_evals_ancestor(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ outside_evals = tmp_path / "outside-evals"+ (target / "evals").rename(outside_evals)+ (outside_evals / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "Legacy shared inputs."}]),+ encoding="utf-8",+ )+ (target / "evals").symlink_to(outside_evals, target_is_directory=True)++ with pytest.raises(ValueError, match=r"evals directory must be a real directory.*symlink"):+ generate_harbor_tasks(target, tmp_path / "symlinked-evals-output")++ assert not (tmp_path / "symlinked-evals-output").exists()++[email protected]("invalid_entries", [[{"id": ".."}], [{"id": "case-001"}, {"id": "Case-001"}]])+def test_native_rejects_invalid_or_colliding_case_ids_before_output_mutation(+ tmp_path: Path,+ invalid_entries: list[dict[str, str]],+) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_minimal_native_task(target)+ (target / "evals" / "evals.json").write_text(json.dumps(invalid_entries), encoding="utf-8")+ output_dir = tmp_path / "native-existing-output"+ output_dir.mkdir()+ sentinel = output_dir / "keep.txt"+ sentinel.write_text("preserve\n", encoding="utf-8")++ with pytest.raises(ValueError, match=r"invalid case id|cross-platform colliding case id"):+ stage_native_harbor_tasks(target, output_dir)++ assert sentinel.read_text(encoding="utf-8") == "preserve\n"+++def test_native_rejects_unsafe_task_directory_name_before_output_mutation(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_minimal_native_task(target)+ native_root = target / "evals" / "harbor"+ (native_root / "case-001").rename(native_root / 'bad"name')+ output_dir = tmp_path / "native-invalid-directory"++ with pytest.raises(ValueError, match="invalid case id"):+ stage_native_harbor_tasks(target, output_dir)++ assert not output_dir.exists()+++def test_results_root_rejects_supplied_skill_source(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ references = tmp_path / "external-references"+ reference = _write_runtime_skill(references / "reference", "reference-output")+ workspace = _write_runtime_skill(tmp_path / "external-workspace", "workspace-output")++ for unsafe_root in (reference / "evals" / "files" / "results", workspace / "scripts" / "results"):+ with pytest.raises(ValueError, match="output root must not be inside runtime skill source"):+ validate_results_root_location(+ target,+ unsafe_root,+ reference_skills_dir=references,+ workspace_skill_paths=[workspace],+ )+++def test_baseline_rejects_target_copied_under_reference_alias(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ references_dir = tmp_path / "aliased-references"+ aliased_target = references_dir / "alias-target"+ shutil.copytree(target, aliased_target)+ (aliased_target / "extra.txt").write_text("tree changed but instructions are identical\n", encoding="utf-8")++ with pytest.raises(ValueError, match="alias of target skill"):+ generate_harbor_tasks(+ target,+ tmp_path / "baseline-reference-alias",+ with_skill=False,+ reference_skills_dir=references_dir,+ )++[email protected]("stager", ["generated", "native"])+def test_baseline_reference_parent_skips_real_target_candidate(tmp_path: Path, stager: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ if stager == "native":+ _write_minimal_native_task(target)+ tasks = stage_native_harbor_tasks(+ target,+ tmp_path / "native-parent-references",+ with_skill=False,+ reference_skills_dir=target.parent,+ )+ else:+ tasks = generate_harbor_tasks(+ target,+ tmp_path / "generated-parent-references",+ with_skill=False,+ reference_skills_dir=target.parent,+ )++ assert tasks+++def test_baseline_alias_scan_ignores_unstaged_candidate_results(tmp_path: Path) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ archived = references_dir / "reference-skill" / "results" / "archived.txt"+ archived.parent.mkdir()+ archived.write_bytes((target / "SKILL.md").read_bytes())++ task = generate_harbor_tasks(+ target,+ tmp_path / "ignored-reference-results",+ with_skill=False,+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ )[0]++ assert not (task / "environment" / "skills" / "reference-skill" / "results").exists()++[email protected]("stager", ["generated", "native"])[email protected]("candidate_kind", ["reference", "workspace"])+def test_baseline_rejects_target_nested_inside_skill_candidate(+ tmp_path: Path,+ stager: str,+ candidate_kind: str,+) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ candidate = references_dir / "reference-skill" if candidate_kind == "reference" else workspace+ nested_alias = candidate / "references" / "target-copy"+ nested_alias.mkdir(parents=True)+ (nested_alias / "SKILL.md").write_bytes((target / "SKILL.md").read_bytes())++ kwargs = {+ "with_skill": False,+ "reference_skills_dir": references_dir,+ "workspace_skill_paths": [workspace],+ }+ with pytest.raises(ValueError, match="alias of target skill"):+ if stager == "generated":+ generate_harbor_tasks(target, tmp_path / f"nested-alias-{candidate_kind}", **kwargs)+ else:+ _write_minimal_native_task(target)+ stage_native_harbor_tasks(target, tmp_path / f"native-nested-alias-{candidate_kind}", **kwargs)++[email protected]("stager", ["generated", "native"])+def test_copy_repo_baseline_rejects_renamed_target_payload(tmp_path: Path, stager: str) -> None:+ repo, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ archived = repo / "archive" / "renamed-target.md"+ archived.parent.mkdir()+ archived.write_bytes((target / "SKILL.md").read_bytes())+ kwargs = {+ "with_skill": False,+ "reference_skills_dir": references_dir,+ "workspace_skill_paths": [workspace],+ "copy_repo": True,+ }++ with pytest.raises(ValueError, match="target skill instructions"):+ if stager == "generated":+ generate_harbor_tasks(target, tmp_path / "repo-payload-alias", **kwargs)+ else:+ _write_minimal_native_task(target)+ stage_native_harbor_tasks(target, tmp_path / "native-repo-payload-alias", **kwargs)+++def test_baseline_projects_reference_and_workspace_skills_without_root_evals(tmp_path: Path) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)++ task = generate_harbor_tasks(+ target,+ tmp_path / "baseline",+ with_skill=False,+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ copy_repo=True,+ )[0]++ environment = task / "environment"+ assert not (environment / "skills" / target.name).exists()+ for name in ("reference-skill", workspace.name):+ assert (environment / "skills" / name / "SKILL.md").is_file()+ assert not (environment / "skills" / name / "evals").exists()+ assert (environment / "skills" / name / "references" / "evals" / "keep.md").is_file()+ assert not (environment / "repo" / target.name).exists()+ assert not (environment / "repo" / "reference-skills" / "reference-skill" / "evals").exists()+ assert not (environment / "repo" / workspace.name / "evals").exists()+ assert (environment / "repo" / "docs" / "evals" / "keep.txt").is_file()+ assert (environment / "input" / "selected.txt").is_file()+ assert "GROUND-TRUTH-SECRET" not in "\n".join(+ path.read_text(encoding="utf-8", errors="ignore") for path in environment.rglob("*") if path.is_file()+ )+++def test_linked_repo_context_excludes_skill_evals_but_keeps_unrelated_evals(tmp_path: Path) -> None:+ _, target, references_dir, workspace = _write_projection_fixture(tmp_path)+ (target / "SKILL.md").write_text(+ "# Target\n\n"+ "[Hidden evaluator data](../reference-skills/reference-skill/evals/evals.json)\n"+ "[Runtime documentation](../docs/evals/keep.txt)\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(+ target,+ tmp_path / "linked",+ reference_skills_dir=references_dir,+ workspace_skill_paths=[workspace],+ copy_repo=False,+ )[0]++ staged_repo = task / "environment" / "repo"+ assert not (staged_repo / "reference-skills" / "reference-skill" / "evals" / "evals.json").exists()+ assert (staged_repo / "docs" / "evals" / "keep.txt").is_file()++[email protected]("files_value", [[], ["evals/files/selected.txt"]])[email protected]("source", ["INPUT/", "Input/fixture.txt"])+def test_custom_dockerfile_rejects_noncanonical_input_case(+ tmp_path: Path,+ files_value: list[str],+ source: str,+) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "Use fixtures.", "files": files_value}]),+ encoding="utf-8",+ )+ custom_environment = target / "evals" / "environment"+ authored_input = custom_environment / "INPUT"+ authored_input.mkdir()+ (authored_input / "fixture.txt").write_text("UNDECLARED-UPPERCASE-INPUT\n", encoding="utf-8")+ (custom_environment / "Dockerfile").write_text(+ f"FROM python:3.12-slim\nCOPY {source} /workspace/input/\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="canonical lowercase path"):+ generate_harbor_tasks(target, tmp_path / "uppercase-input")++[email protected]("custom_dockerfile_mode", ["preserve", "rebase"])+ "copy_instruction",+ [+ "COPY input/ /workspace/input/",+ "COPY input/. /workspace/input/",+ "COPY ../input/ /workspace/input/",+ "COPY --chown=0:0\tinput/\t/workspace/input/",+ 'COPY --chown=0:0 ["input/", "/workspace/input/"]',+ 'COPY ["../input/", "/workspace/input/"]',+ ],+)+def test_explicit_empty_input_keeps_authored_copy_input_buildable(+ tmp_path: Path,+ custom_dockerfile_mode: str,+ copy_instruction: str,+) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ custom_environment = target / "evals" / "environment"+ authored_input = custom_environment / "input"+ authored_input.mkdir()+ (authored_input / "hidden.txt").write_text("UNDECLARED-CUSTOM-INPUT\n", encoding="utf-8")+ (custom_environment / "Dockerfile").write_text(+ f"FROM python:3.12-slim\n{copy_instruction}\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(+ target,+ tmp_path / f"custom-empty-{custom_dockerfile_mode}",+ custom_dockerfile_mode=custom_dockerfile_mode,+ base_image="python:3.12-slim" if custom_dockerfile_mode == "rebase" else "",+ )[0]++ input_dir = task / "environment" / "input"+ assert input_dir.is_dir()+ assert list(input_dir.iterdir()) == []+ assert copy_instruction in (task / "environment" / "Dockerfile").read_text(encoding="utf-8")+++def test_native_copy_input_without_fixtures_gets_empty_build_context_directory(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ native_task = target / "evals" / "harbor" / "case-001"+ environment = native_task / "environment"+ environment.mkdir(parents=True)+ (environment / "Dockerfile").write_text(+ "FROM python:3.12-slim\nCOPY input/ /workspace/input/\n",+ encoding="utf-8",+ )+ (native_task / "instruction.md").write_text("Run the native case.\n", encoding="utf-8")+ (native_task / "task.toml").write_text(+ 'schema_version = "1.3"\n\n[task]\nname = "nvidia/case-001"\n\n'+ '[metadata]\nentry_id = "case-001"\n\n[environment]\n',+ encoding="utf-8",+ )++ task = stage_native_harbor_tasks(target, tmp_path / "native-empty-input")[0]++ input_dir = task / "environment" / "input"+ assert input_dir.is_dir()+ assert list(input_dir.iterdir()) == []+++def test_native_input_is_preserved_without_eval_fixture_declaration(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ shutil.rmtree(target / "evals" / "files")+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "Use native input."}]),+ encoding="utf-8",+ )+ native_task = target / "evals" / "harbor" / "case-001"+ environment = native_task / "environment"+ native_input = environment / "input"+ native_input.mkdir(parents=True)+ (native_input / "native-only.txt").write_text("NATIVE-ONLY\n", encoding="utf-8")+ (environment / "Dockerfile").write_text("FROM python:3.12-slim\n", encoding="utf-8")+ (native_task / "instruction.md").write_text("Run the native case.\n", encoding="utf-8")+ (native_task / "task.toml").write_text(+ 'schema_version = "1.3"\n\n[task]\nname = "nvidia/case-001"\n\n'+ '[metadata]\nentry_id = "case-001"\n\n[environment]\n',+ encoding="utf-8",+ )++ task = stage_native_harbor_tasks(target, tmp_path / "native-preserved-input")[0]++ assert (task / "environment" / "input" / "native-only.txt").read_text(encoding="utf-8") == "NATIVE-ONLY\n"+++def test_generated_baseline_rejects_aliased_authored_skill(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ custom_environment = target / "evals" / "environment"+ alias = custom_environment / "alias"+ alias.mkdir()+ (alias / "SKILL.md").write_text((target / "SKILL.md").read_text(encoding="utf-8"), encoding="utf-8")+ (custom_environment / "Dockerfile").write_text(+ "FROM python:3.12-slim\nCOPY alias/ /root/.agents/skills/alias/\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="unmanaged skill"):+ generate_harbor_tasks(target, tmp_path / "baseline-alias", with_skill=False)+++def test_generated_baseline_rejects_renamed_target_manifest_payload(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ custom_environment = target / "evals" / "environment"+ (custom_environment / "payload.txt").write_bytes((target / "SKILL.md").read_bytes())+ (custom_environment / "Dockerfile").write_text(+ "FROM python:3.12-slim\nCOPY payload.txt /root/.agents/skills/alias/SKILL.md\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="target skill instructions"):+ generate_harbor_tasks(target, tmp_path / "baseline-renamed-payload", with_skill=False)+++ "project_skill_root",+ [+ "/workspace/.agents/skills",+ "/workspace/.claude/skills",+ "/workspace/.codex/skills",+ "/workspace/.config/opencode/skills",+ "/workspace/.cursor/skills",+ "/workspace/.gemini/skills",+ "/workspace/.opencode/skills",+ ],+)+def test_final_projection_clears_authored_project_skill_root(+ tmp_path: Path,+ project_skill_root: str,+) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ environment = target / "evals" / "environment"+ hidden_manifest = f"{project_skill_root}/hidden/SKILL.md"+ (environment / "Dockerfile").write_text(+ "FROM python:3.12-slim\n"+ f"RUN mkdir -p {project_skill_root}/hidden && "+ f"printf 'PROJECT-ROOT-ORACLE\\n' > {hidden_manifest}\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "project-skill-reset", with_skill=False)[0]++ dockerfile = (task / "environment" / "Dockerfile").read_text(encoding="utf-8")+ authored_index = dockerfile.index("PROJECT-ROOT-ORACLE")+ reset_line = next(+ line+ for line in dockerfile.splitlines()+ if line.startswith('RUN ["/bin/rm", "-rf"') and f'"{project_skill_root}"' in line+ )+ assert dockerfile.index(reset_line) > authored_index+++def test_final_projection_clears_effective_workdir_ancestors_and_home(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text(+ "FROM python:3.12-slim\n"+ "ENV HOME=/home/agent\n"+ "WORKDIR /app\n"+ "RUN mkdir -p /app/.claude/skills/hidden /home/agent/.claude/skills/hidden && "+ "printf 'AUTHORED-CWD-ORACLE\\n' > /app/.claude/skills/hidden/SKILL.md && "+ "printf 'AUTHORED-HOME-ORACLE\\n' > /home/agent/.claude/skills/hidden/SKILL.md\n"+ "USER 12345\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(+ target,+ tmp_path / "dynamic-skill-root-reset",+ with_skill=False,+ grading_mode="custom_only",+ agent_workdir="/opt/task/subdir",+ )[0]++ dockerfile = (task / "environment" / "Dockerfile").read_text(encoding="utf-8")+ final_projection = dockerfile.index("# SkillEvaluator: final runtime projection")+ assert dockerfile.index("AUTHORED-CWD-ORACLE") < final_projection+ assert dockerfile.index("AUTHORED-HOME-ORACLE") < final_projection+ assert dockerfile.count("current=$(/bin/pwd -P)") >= 2+ assert "$home/$rel" in dockerfile+ assert "done < /etc/passwd" in dockerfile+ assert r"[ -d \"$home\" ] || continue" in dockerfile+ assert r"[ -e \"$passwd_home\" ] && [ ! -d \"$passwd_home\" ] && continue" in dockerfile+ assert "${CODEX_HOME:-}" in dockerfile+ assert "${OPENCODE_CONFIG_DIR:-}" in dockerfile+ assert "canonical=$(CDPATH= cd -P" in dockerfile+ assert "ENTRYPOINT []" in dockerfile+ assert "HEALTHCHECK NONE" in dockerfile+ assert 'BASH_ENV=""' in dockerfile+ assert 'CLAUDE_CODE_DISABLE_POLICY_SKILLS="1"' in dockerfile+ assert "/etc/codex/skills" in dockerfile+ assert "WORKDIR /opt/task/subdir" in dockerfile[final_projection:]+ assert dockerfile.rindex("current=$(/bin/pwd -P)") > dockerfile.rindex("COPY input/ /workspace/input/")+ task_toml = (task / "task.toml").read_text(encoding="utf-8")+ assert 'workdir = "/opt/task/subdir"' in task_toml+ assert '"BASH_ENV" = ""' in task_toml+ assert '"CLAUDE_CODE_DISABLE_POLICY_SKILLS" = "1"' in task_toml++[email protected]("agent_workdir", ["relative", "/opt/../tmp", "/opt/task path"])+def test_generated_tasks_reject_unsafe_agent_workdir(tmp_path: Path, agent_workdir: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)++ with pytest.raises(ValueError, match="normalized absolute POSIX path"):+ generate_harbor_tasks(target, tmp_path / "unsafe-agent-workdir", agent_workdir=agent_workdir)+++ "agent_workdir",+ [+ "/workspace/input",+ "/workspace/input/subdir",+ "/workspace/repo/subdir",+ "/workspace/skills",+ "/opt/project/.claude/skills/hidden",+ ],+)+def test_generated_tasks_reject_workdir_inside_runtime_projection(tmp_path: Path, agent_workdir: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)++ with pytest.raises(ValueError, match=r"runtime projection|skill discovery"):+ generate_harbor_tasks(target, tmp_path / "overlapping-agent-workdir", agent_workdir=agent_workdir)+++ "name",+ [+ "HOME",+ "USERPROFILE",+ "XDG_CONFIG_HOME",+ "CLAUDE_CONFIG_DIR",+ "CLAUDE_CODE_DISABLE_POLICY_SKILLS",+ "CODEX_HOME",+ "GEMINI_CLI_HOME",+ "OPENCODE_CONFIG_DIR",+ ],+)+def test_generated_tasks_reject_runtime_skill_discovery_environment(tmp_path: Path, name: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)++ with pytest.raises(ValueError, match="skill discovery"):+ generate_harbor_tasks(+ target,+ tmp_path / "runtime-discovery-env",+ runtime_env={name: "/runtime/hidden"},+ )+++ ("environment_toml", "message"),+ [+ ('[environment.env]\nHOME = "/runtime/hidden"\n', "skill discovery"),+ ('[environment.env]\nBASH_ENV = "/runtime/seed.sh"\n', "process loader"),+ ('[environment.env]\n"BASH_FUNC_hidden%%" = "/runtime/seed.sh"\n', "process loader"),+ ('[environment.env]\nCLAUDE_CODE_DISABLE_POLICY_SKILLS = "0"\n', "skill discovery"),+ ('skills_dir = "/workspace/input"\n', "skills_dir"),+ ('docker_image = "prebuilt:latest"\n', "docker_image"),+ ],+)+def test_native_tasks_reject_runtime_projection_bypasses(tmp_path: Path, environment_toml: str, message: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ native_task = target / "evals" / "harbor" / "case-001"+ native_task.mkdir(parents=True)+ (native_task / "task.toml").write_text(+ 'schema_version = "1.3"\n\n[task]\nname = "nvidia/case-001"\n\n'+ '[metadata]\nentry_id = "case-001"\n\n[environment]\n' + environment_toml,+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match=message):+ stage_native_harbor_tasks(target, tmp_path / "native-projection-bypass", grading_mode="custom_only")+++def test_native_tasks_pin_skills_dir_to_controlled_projection(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ native_task = target / "evals" / "harbor" / "case-001"+ (native_task / "environment").mkdir(parents=True)+ (native_task / "tests").mkdir()+ (native_task / "task.toml").write_text(+ 'schema_version = "1.3"\n\n[task]\nname = "nvidia/case-001"\n\n'+ '[metadata]\nentry_id = "case-001"\n\n[environment]\n',+ encoding="utf-8",+ )+ (native_task / "environment" / "Dockerfile").write_text("FROM python:3.12-slim\n", encoding="utf-8")+ (native_task / "tests" / "test.sh").write_text("#!/bin/sh\nexit 0\n", encoding="utf-8")++ staged = stage_native_harbor_tasks(+ target,+ tmp_path / "native-controlled-skills-dir",+ grading_mode="custom_only",+ )[0]++ staged_toml = (staged / "task.toml").read_text(encoding="utf-8")+ assert 'skills_dir = "/workspace/skills"' in staged_toml+ assert '"BASH_ENV" = ""' in staged_toml+ assert '"CLAUDE_CODE_DISABLE_POLICY_SKILLS" = "1"' in staged_toml++[email protected]("name", ["BASH_ENV", "BASH_FUNC_hidden%%"])+def test_generated_tasks_reject_runtime_process_loader_environment(tmp_path: Path, name: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)++ with pytest.raises(ValueError, match="process loader"):+ generate_harbor_tasks(+ target,+ tmp_path / "runtime-loader-env",+ runtime_env={name: "/runtime/seed.sh"},+ )+++def test_native_baseline_rejects_aliased_authored_skill(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ native_task = target / "evals" / "harbor" / "case-001"+ environment = native_task / "environment"+ alias = environment / "skills" / "alias"+ alias.mkdir(parents=True)+ (alias / "SKILL.md").write_text((target / "SKILL.md").read_text(encoding="utf-8"), encoding="utf-8")+ (native_task / "instruction.md").write_text("Run the native case.\n", encoding="utf-8")+ (native_task / "task.toml").write_text(+ 'schema_version = "1.3"\n\n[task]\nname = "nvidia/case-001"\n\n'+ '[metadata]\nentry_id = "case-001"\n\n[environment]\n',+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="unmanaged skill"):+ stage_native_harbor_tasks(target, tmp_path / "native-baseline-alias", with_skill=False)+++def test_native_baseline_rejects_renamed_target_manifest_payload(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ native_task = target / "evals" / "harbor" / "case-001"+ environment = native_task / "environment"+ environment.mkdir(parents=True)+ (environment / "payload.txt").write_bytes((target / "SKILL.md").read_bytes())+ (environment / "Dockerfile").write_text(+ "FROM python:3.12-slim\nCOPY payload.txt /root/.agents/skills/alias/SKILL.md\n",+ encoding="utf-8",+ )+ (native_task / "instruction.md").write_text("Run the native case.\n", encoding="utf-8")+ (native_task / "task.toml").write_text(+ 'schema_version = "1.3"\n\n[task]\nname = "nvidia/case-001"\n\n'+ '[metadata]\nentry_id = "case-001"\n\n[environment]\n',+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="target skill instructions"):+ stage_native_harbor_tasks(target, tmp_path / "native-renamed-payload", with_skill=False)++[email protected]("payload_kind", ["manifest", "renamed"])+def test_native_baseline_ignores_target_payload_in_unstaged_results(tmp_path: Path, payload_kind: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_minimal_native_task(target)+ results = target / "evals" / "harbor" / "case-001" / "environment" / "results"+ payload = (target / "SKILL.md").read_bytes()+ if payload_kind == "manifest":+ alias = results / "alias"+ alias.mkdir(parents=True)+ (alias / "SKILL.md").write_bytes(payload)+ else:+ results.mkdir(parents=True)+ (results / "archived.txt").write_bytes(payload)++ task = stage_native_harbor_tasks(+ target,+ tmp_path / f"native-ignored-results-{payload_kind}",+ with_skill=False,+ )[0]++ assert not (task / "environment" / "results").exists()+++ "copy_instruction",+ [+ "COPY input/missing.txt /workspace/input/missing.txt",+ "COPY ../input/missing.txt /workspace/input/missing.txt",+ 'COPY ["../input/missing.txt", "/workspace/input/missing.txt"]',+ "RUN --mount=type=bind,source=input/missing.txt,target=/payload true",+ ],+)+def test_explicit_empty_rejects_specific_custom_docker_input_child(+ tmp_path: Path,+ copy_instruction: str,+) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ (target / "evals" / "environment" / "Dockerfile").write_text(+ f"FROM python:3.12-slim\n{copy_instruction}\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="specific input path"):+ generate_harbor_tasks(target, tmp_path / "specific-empty-input")+++def test_explicit_empty_rejects_dynamic_custom_docker_input_source(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ (target / "evals" / "environment" / "Dockerfile").write_text(+ "FROM python:3.12-slim\nCOPY ${SRC}/../input/ /workspace/input/\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="ambiguous input source"):+ generate_harbor_tasks(target, tmp_path / "ambiguous-empty-input")+++def test_explicit_empty_rejects_unresolved_docker_parameter_expansion(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ (target / "evals" / "environment" / "Dockerfile").write_text(+ "FROM python:3.12-slim\nARG INPUT_DIR\nCOPY ${INPUT_DIR:-input}/ /workspace/input/\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="ambiguous input source"):+ generate_harbor_tasks(target, tmp_path / "parameter-expansion-empty-input")++[email protected]("declaration", ["ARG INPUT_DIR=input", "ENV INPUT_DIR=input"])+def test_explicit_empty_resolves_static_dynamic_input_directory(tmp_path: Path, declaration: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ (target / "evals" / "environment" / "Dockerfile").write_text(+ f"FROM python:3.12-slim\n{declaration}\nCOPY ${{INPUT_DIR}}/ /workspace/input/\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "static-dynamic-empty-input")[0]++ assert (task / "environment" / "input").is_dir()+++def test_explicit_empty_resolves_docker_variables_at_each_instruction(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ fixtures = target / "evals" / "environment" / "fixtures"+ fixtures.mkdir()+ (fixtures / "runtime.txt").write_text("runtime\n", encoding="utf-8")+ (target / "evals" / "environment" / "Dockerfile").write_text(+ "FROM python:3.12-slim AS first\n"+ "ARG SRC=input\n"+ "COPY ${SRC}/ /workspace/input/\n"+ "FROM first AS final\n"+ "ARG SRC=fixtures\n"+ "COPY ${SRC}/ /app/\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "instruction-scoped-dynamic-input")[0]++ assert (task / "environment" / "input").is_dir()+++def test_explicit_empty_does_not_expose_global_arg_without_stage_redeclaration(tmp_path: Path) -> None:+ dockerfile = tmp_path / "Dockerfile"+ dockerfile.write_text(+ "ARG SRC=input\nFROM python:3.12-slim\nCOPY ${SRC}/ /workspace/input/\n",+ encoding="utf-8",+ )++ assert _dockerfile_resolved_build_context_sources(dockerfile) == [("${SRC}/", "${SRC}/", True)]+++def test_explicit_empty_inherits_arg_and_env_from_parent_stage(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ (target / "evals" / "environment" / "Dockerfile").write_text(+ "FROM python:3.12-slim AS base\n"+ "ARG ROOT=workspace\n"+ "ENV SRC=input\n"+ "FROM base AS final\n"+ "COPY ${SRC}/ /${ROOT}/input/\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "parent-stage-scope")++ assert (task[0] / "environment" / "input").is_dir()+++def test_explicit_empty_uses_pre_instruction_env_snapshot(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ environment = target / "evals" / "environment"+ fixtures = environment / "fixtures"+ fixtures.mkdir()+ (fixtures / "runtime.txt").write_text("runtime\n", encoding="utf-8")+ (environment / "Dockerfile").write_text(+ "FROM python:3.12-slim\nENV SRC=fixtures\nENV SRC=input COPY_ROOT=$SRC\nCOPY ${COPY_ROOT}/ /app/\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "env-snapshot")++ assert (task[0] / "environment" / "input").is_dir()+ assert list((task[0] / "environment" / "input").iterdir()) == []+++def test_explicit_empty_does_not_retroactively_resolve_unknown_env(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ (target / "evals" / "environment" / "Dockerfile").write_text(+ "FROM python:3.12-slim\nENV COPY_ROOT=$SRC\nENV SRC=input\nCOPY ${COPY_ROOT}/ /app/\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="ambiguous input source"):+ generate_harbor_tasks(target, tmp_path / "unknown-env-snapshot")+++def test_explicit_empty_ignores_docker_heredoc_body_in_variable_scope(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ (target / "evals" / "environment" / "Dockerfile").write_text(+ "FROM python:3.12-slim\n"+ "ARG SRC=input\n"+ "COPY <<EOF /fake-script\n"+ "ARG SRC=fixtures\n"+ "COPY fixtures/ /not-an-instruction/\n"+ "EOF\n"+ "COPY ${SRC}/ /workspace/input/\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "heredoc-scope")++ assert (task[0] / "environment" / "input").is_dir()+++def test_punctuation_heredoc_body_cannot_change_restored_runtime_user(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text(+ "FROM python:3.12-slim\nCOPY <<'E!' /payload\nUSER 12345\nE!\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "punctuation-heredoc-user")[0]++ dockerfile = (task / "environment" / "Dockerfile").read_text(encoding="utf-8")+ assert not dockerfile.rstrip().endswith("USER 12345")++[email protected]("source", ["'$SRC'/", r"\$SRC/", "$$SRC/"])+def test_explicit_empty_rejects_literal_or_escaped_docker_dollar_source(tmp_path: Path, source: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ (target / "evals" / "environment" / "Dockerfile").write_text(+ f"FROM python:3.12-slim\nARG SRC=input\nCOPY {source} /workspace/input/\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match=r"safely parse|ambiguous"):+ generate_harbor_tasks(target, tmp_path / "literal-dollar-source")++[email protected]("declaration", ["ARG COPY_ROOT='$SRC'", "ENV COPY_ROOT='$SRC'", r"ENV COPY_ROOT=\$SRC"])+def test_explicit_empty_rejects_literal_dollar_in_docker_variable_declaration(+ tmp_path: Path,+ declaration: str,+) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ (target / "evals" / "environment" / "Dockerfile").write_text(+ f"FROM python:3.12-slim\nARG SRC=input\n{declaration}\nCOPY ${{COPY_ROOT}}/ /workspace/input/\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match=r"quoted or escaped Dockerfile"):+ generate_harbor_tasks(target, tmp_path / "literal-dollar-declaration")+++def test_explicit_empty_honors_deferred_onbuild_copy(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ (target / "evals" / "environment" / "Dockerfile").write_text(+ "FROM python:3.12-slim AS base\n"+ "ONBUILD COPY input/ /workspace/input/\n"+ "FROM base AS final\n"+ "RUN test -d /workspace/input\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "onbuild-input")++ assert (task[0] / "environment" / "input").is_dir()+++def test_quoted_shell_text_is_not_treated_as_docker_heredoc(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ (target / "evals" / "environment" / "Dockerfile").write_text(+ "FROM python:3.12-slim\nRUN echo 'x <<EOF y'\nCOPY input/ /workspace/input/\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "quoted-heredoc-text")++ assert (task[0] / "environment" / "input").is_dir()+++def test_explicit_empty_reads_compose_yml_build_arg_override(tmp_path: Path) -> None:+ environment = tmp_path / "environment"+ environment.mkdir()+ dockerfile = environment / "Dockerfile"+ dockerfile.write_text(+ "FROM python:3.12-slim\nARG SRC=fixtures\nCOPY ${SRC}/ /workspace/input/\n",+ encoding="utf-8",+ )+ (environment / "docker-compose.yml").write_text(+ "services:\n main:\n build:\n context: .\n args:\n SRC: input\n",+ encoding="utf-8",+ )++ _ensure_empty_custom_docker_input_compatibility(environment, dockerfile, has_input=False)++ assert (environment / "input").is_dir()+++def test_explicit_empty_allows_broad_custom_docker_wildcard(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ (target / "evals" / "environment" / "app.txt").write_text("runtime asset\n", encoding="utf-8")+ (target / "evals" / "environment" / "Dockerfile").write_text(+ "FROM python:3.12-slim\nCOPY * /app/\n",+ encoding="utf-8",+ )++ assert generate_harbor_tasks(target, tmp_path / "broad-wildcard")+++def test_explicit_empty_allows_unrelated_nested_input_glob(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ fixtures = target / "evals" / "environment" / "fixtures"+ fixtures.mkdir()+ (fixtures / "myinput-data.txt").write_text("runtime asset\n", encoding="utf-8")+ (target / "evals" / "environment" / "Dockerfile").write_text(+ "FROM python:3.12-slim\nCOPY fixtures/myinput*.txt /app/\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "unrelated-input-glob")+ assert task+++ ("declaration", "source", "source_dir"),+ [+ ("ARG APP_DIR=fixtures", "${APP_DIR}/", "fixtures"),+ ("ARG ARCH=amd64", "fixtures/${ARCH}/", "fixtures/amd64"),+ ],+)+def test_explicit_empty_allows_unrelated_dynamic_custom_docker_source(+ tmp_path: Path,+ declaration: str,+ source: str,+ source_dir: str,+) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ runtime_source = target / "evals" / "environment" / source_dir+ runtime_source.mkdir(parents=True)+ (runtime_source / "runtime.txt").write_text("runtime asset\n", encoding="utf-8")+ (target / "evals" / "environment" / "Dockerfile").write_text(+ f"FROM python:3.12-slim\n{declaration}\nCOPY {source} /app/\n",+ encoding="utf-8",+ )++ assert generate_harbor_tasks(target, tmp_path / "unrelated-dynamic-source")++[email protected]("escape_char", ["\\", "`"])+def test_explicit_empty_honors_custom_docker_escape_directive(tmp_path: Path, escape_char: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+ (target / "evals" / "environment" / "Dockerfile").write_text(+ f"# escape={escape_char}\n"+ "FROM python:3.12-slim\n"+ f"COPY {escape_char}\n"+ f" input/ {escape_char}\n"+ " /workspace/input/\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / f"escape-{ord(escape_char)}")[0]++ assert (task / "environment" / "input").is_dir()+++def _write_explicit_empty_entry(target: Path) -> None:+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No fixture.", "files": []}]),+ encoding="utf-8",+ )+++def test_explicit_empty_materializes_compose_additional_input_context(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_explicit_empty_entry(target)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text("FROM python:3.12-slim\n", encoding="utf-8")+ (environment / "docker-compose.yaml").write_text(+ "services:\n"+ " helper:\n"+ " build:\n"+ " context: .\n"+ " dockerfile_inline: |\n"+ " FROM scratch\n"+ " COPY --from=taskinput . /payload/\n"+ " additional_contexts:\n"+ " taskinput: input\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "compose-additional-input")[0]++ assert (task / "environment" / "input").is_dir()+ assert list((task / "environment" / "input").iterdir()) == []+++def test_explicit_empty_materializes_compose_primary_input_context(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_explicit_empty_entry(target)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text("FROM python:3.12-slim\n", encoding="utf-8")+ (environment / "docker-compose.yaml").write_text(+ "services:\n"+ " helper:\n"+ " build:\n"+ " context: input\n"+ " dockerfile_inline: |\n"+ " FROM scratch\n"+ " COPY . /payload/\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "compose-primary-input")[0]++ assert (task / "environment" / "input").is_dir()+ assert list((task / "environment" / "input").iterdir()) == []+++def test_explicit_empty_rejects_specific_source_from_compose_primary_input_context(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_explicit_empty_entry(target)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text("FROM python:3.12-slim\n", encoding="utf-8")+ (environment / "docker-compose.yaml").write_text(+ "services:\n"+ " helper:\n"+ " build:\n"+ " context: input\n"+ " dockerfile_inline: |\n"+ " FROM scratch\n"+ " COPY missing.txt /payload/\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="specific path from the input build context"):+ generate_harbor_tasks(target, tmp_path / "compose-primary-specific-source")+++ "instruction",+ [+ "COPY --from=taskinput missing.txt /payload/",+ "RUN --mount=type=bind,from=taskinput,source=missing.txt,target=/payload true",+ ],+)+def test_explicit_empty_rejects_specific_source_from_named_input_context(+ tmp_path: Path,+ instruction: str,+) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_explicit_empty_entry(target)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text("FROM python:3.12-slim\n", encoding="utf-8")+ (environment / "docker-compose.yaml").write_text(+ "services:\n"+ " helper:\n"+ " build:\n"+ " context: .\n"+ " dockerfile_inline: |\n"+ " FROM scratch\n"+ f" {instruction}\n"+ " additional_contexts:\n"+ " taskinput: input\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="specific path from an input context"):+ generate_harbor_tasks(target, tmp_path / "compose-named-specific-source")+++def test_explicit_empty_resolves_compose_escaped_inline_arg(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_explicit_empty_entry(target)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text("FROM python:3.12-slim\n", encoding="utf-8")+ (environment / "docker-compose.yaml").write_text(+ "services:\n"+ " helper:\n"+ " build:\n"+ " context: .\n"+ " args:\n"+ " SRC: input\n"+ " dockerfile_inline: |\n"+ " FROM scratch\n"+ " ARG SRC\n"+ " COPY $${SRC}/ /payload/\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "compose-inline-arg")[0]++ assert (task / "environment" / "input").is_dir()+++def test_explicit_empty_rejects_specific_compose_input_context(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_explicit_empty_entry(target)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text("FROM python:3.12-slim\n", encoding="utf-8")+ (environment / "docker-compose.yaml").write_text(+ "services:\n"+ " helper:\n"+ " build:\n"+ " context: .\n"+ " dockerfile_inline: 'FROM scratch'\n"+ " additional_contexts:\n"+ " taskinput: input/missing\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="specific input context"):+ generate_harbor_tasks(target, tmp_path / "compose-specific-input")+++def test_compose_rejects_build_with_global_image_tag(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text("FROM python:3.12-slim\n", encoding="utf-8")+ (environment / "docker-compose.yaml").write_text(+ "services:\n helper:\n image: trusted-global-name:latest\n build:\n context: sidecar\n",+ encoding="utf-8",+ )+ (environment / "sidecar" / "Dockerfile").write_text("FROM scratch\n", encoding="utf-8")++ with pytest.raises(ValueError, match="image together with build"):+ generate_harbor_tasks(target, tmp_path / "compose-image-poisoning")+++def test_compose_rejects_non_string_mapping_keys(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text("FROM python:3.12-slim\n", encoding="utf-8")+ (environment / "docker-compose.yaml").write_text(+ "1: unsafe\nservices:\n helper:\n image: alpine:3.20\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="keys must be strings"):+ generate_harbor_tasks(target, tmp_path / "compose-non-string-key")+++def test_rebase_accepts_tab_from_and_preserves_alias_and_runtime_user(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text(+ "FROM\tpython:3.12-slim AS app\nUSER 12345\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(+ target,+ tmp_path / "rebase-tab-from",+ base_image="skillevaluator-base:test",+ custom_dockerfile_mode="rebase",+ )[0]++ dockerfile = (task / "environment" / "Dockerfile").read_text(encoding="utf-8")+ assert dockerfile.startswith("FROM\tskillevaluator-base:test AS app\n")+ assert dockerfile.rstrip().endswith("USER 12345")+++def test_rebase_rejects_multi_stage_dockerfile(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "environment" / "Dockerfile").write_text(+ "FROM python:3.12-slim AS build\nFROM build AS final\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="exactly one FROM"):+ generate_harbor_tasks(+ target,+ tmp_path / "rebase-multi-stage",+ base_image="skillevaluator-base:test",+ custom_dockerfile_mode="rebase",+ )+++def test_rebase_rejects_continued_from_instruction(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "evals" / "environment" / "Dockerfile").write_text(+ "FROM \\\n python:3.12-slim\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="continued FROM"):+ generate_harbor_tasks(+ target,+ tmp_path / "rebase-continued-from",+ base_image="skillevaluator-base:test",+ custom_dockerfile_mode="rebase",+ )+++def test_custom_projection_uses_exec_runs_and_ignores_comment_decoys(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text(+ "FROM python:3.12-slim\n"+ 'SHELL ["python", "-c"]\n'+ "# ragas langchain anthropic\n"+ "# COPY skills/ /root/.agents/skills/\n"+ "# RUN rm -rf /workspace/input && mkdir -p /workspace/input\n"+ "USER 12345\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "custom-exec-projection")[0]++ dockerfile = (task / "environment" / "Dockerfile").read_text(encoding="utf-8")+ assert 'RUN ["/bin/sh", "-c"' in dockerfile+ assert '"python", "-m", "pip", "install"' in dockerfile+ assert 'RUN ["/bin/rm", "-rf", "/workspace/input"]' in dockerfile+ assert dockerfile.rstrip().endswith("USER 12345")+++def test_preserve_projection_restores_user_inherited_from_authored_stage(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text(+ "FROM python:3.12-slim AS base\nUSER 12345\nFROM base AS final\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "inherited-user-projection")[0]++ dockerfile = (task / "environment" / "Dockerfile").read_text(encoding="utf-8")+ assert "USER root\n" in dockerfile+ assert dockerfile.rstrip().endswith("USER 12345")+++def test_managed_projection_survives_authored_path_override(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text(+ "FROM python:3.12-slim\nENV PATH=/custom\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "managed-path-projection")[0]++ dockerfile = (task / "environment" / "Dockerfile").read_text(encoding="utf-8")+ assert 'PATH=\\"${PATH:+$PATH:}/usr/local/bin:/usr/bin:/bin\\"' in dockerfile+ assert 'RUN ["/bin/rm", "-rf", "/workspace/input"]' in dockerfile+++def test_verifier_constraint_and_skill_requirements_share_resolver_transaction(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ (target / "scripts" / "requirements.txt").write_text(+ "langchain-community>=0.4.2\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(target, tmp_path / "verifier-requirement-conflict")[0]++ dockerfile = (task / "environment" / "Dockerfile").read_text(encoding="utf-8")+ install_line = next(line for line in dockerfile.splitlines() if '"pip", "install"' in line)+ assert "langchain-community<0.4.2" in install_line+ assert "langchain-community>=0.4.2" in install_line+ assert dockerfile.count('"pip", "install"') == 1+ assert "from ragas.llms.base import llm_factory" in dockerfile+++def test_rebase_reasserts_verifier_constraint_after_authored_dependencies(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ environment = target / "evals" / "environment"+ (environment / "Dockerfile").write_text(+ "FROM python:3.12-slim\nRUN python -m pip install 'langchain-community>=0.4.2'\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(+ target,+ tmp_path / "rebase-verifier-constraint",+ base_image="skillevaluator-base:test",+ custom_dockerfile_mode="rebase",+ )[0]++ dockerfile = (task / "environment" / "Dockerfile").read_text(encoding="utf-8")+ assert dockerfile.rfind("langchain-community<0.4.2") > dockerfile.find("langchain-community>=0.4.2")+ assert dockerfile.rfind("from ragas.llms.base import llm_factory") > dockerfile.find("langchain-community>=0.4.2")+++def test_baseline_rejects_target_manifest_payload_with_appended_comment(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ environment = target / "evals" / "environment"+ payload = (target / "SKILL.md").read_text(encoding="utf-8") + "\n# harmless comment\n"+ (environment / "payload.txt").write_text(payload, encoding="utf-8")+ (environment / "Dockerfile").write_text(+ "FROM python:3.12-slim\nCOPY payload.txt /root/.agents/skills/alias/SKILL.md\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match="target skill instructions"):+ generate_harbor_tasks(target, tmp_path / "baseline-appended-payload", with_skill=False)+++def test_linked_repo_projection_rejects_reserved_workspace_input(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ monkeypatch.setenv(+ "SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE",+ str(tmp_path.parent / f"{tmp_path.name}-state" / "output-provenance.key"),+ )+ repo, target, _, _ = _write_projection_fixture(tmp_path)+ (tmp_path / ".git").mkdir()+ outside_input = tmp_path / "input"+ outside_input.mkdir()+ (outside_input / "hidden.txt").write_text("RESERVED-INPUT-ORACLE\n", encoding="utf-8")+ manifest = target / "SKILL.md"+ manifest.write_text(+ manifest.read_text(encoding="utf-8") + "\n[hidden](../../input/hidden.txt)\n",+ encoding="utf-8",+ )+ assert repo.is_dir()++ with pytest.raises(ValueError, match="reserved runtime path '/workspace/input'"):+ generate_harbor_tasks(target, tmp_path / "reserved-linked-input", copy_repo=False)+++ ("linked_path", "linked_target"),+ [+ ("skills/target-skill/SKILL.md", "../../skills/target-skill/SKILL.md"),+ (".claude/skills/hidden/SKILL.md", "../../.claude/skills/hidden/SKILL.md"),+ ],+)+def test_linked_repo_cannot_overlay_agent_discovery_roots(+ tmp_path: Path,+ linked_path: str,+ linked_target: str,+) -> None:+ repo, original_target, references, workspace = _write_projection_fixture(tmp_path)+ target = repo / "packages" / original_target.name+ target.parent.mkdir()+ shutil.move(original_target, target)+ (repo / ".git").mkdir()+ hidden_manifest = repo / linked_path+ hidden_manifest.parent.mkdir(parents=True, exist_ok=True)+ hidden_manifest.write_text("HIDDEN-DISCOVERY-ORACLE\n", encoding="utf-8")+ manifest = target / "SKILL.md"+ manifest.write_text(+ manifest.read_text(encoding="utf-8") + f"\n[hidden]({linked_target})\n",+ encoding="utf-8",+ )++ task = generate_harbor_tasks(+ target,+ tmp_path / f"linked-discovery-{hidden_manifest.parent.name}",+ reference_skills_dir=references,+ workspace_skill_paths=[workspace],+ copy_repo=False,+ )[0]++ linked_root = task / "environment" / "repo-linked-root"+ staged_text = (+ "\n".join(+ path.read_text(encoding="utf-8", errors="ignore") for path in linked_root.rglob("*") if path.is_file()+ )+ if linked_root.exists()+ else ""+ )+ assert "HIDDEN-DISCOVERY-ORACLE" not in staged_text++[email protected](os.name != "nt", reason="requires Windows junctions")[email protected]("location", ["runtime", "input"])+def test_windows_junction_sources_are_rejected(tmp_path: Path, location: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ outside = tmp_path / "outside-junction"+ outside.mkdir()+ (outside / "hidden.txt").write_text("JUNCTION-ORACLE\n", encoding="utf-8")+ if location == "runtime":+ junction = target / "references" / "junction"+ else:+ junction = target / "evals" / "files" / "junction"+ (target / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "question": "No explicit files."}]),+ encoding="utf-8",+ )+ subprocess.run(+ ["cmd", "/c", "mklink", "/J", str(junction), str(outside)],+ check=True,+ capture_output=True,+ text=True,+ )++ with pytest.raises(ValueError, match=r"symlink|reparse|junction"):+ generate_harbor_tasks(target, tmp_path / f"windows-junction-{location}")++[email protected]("kind", ["symlink", "hardlink", "fifo"])+def test_eval_dataset_file_must_be_regular_and_unlinked(tmp_path: Path, kind: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ dataset = target / "evals" / "evals.json"+ dataset.unlink()+ outside = tmp_path / "outside-evals.json"+ outside.write_text('[{"id":"case-001","question":"ORACLE"}]\n', encoding="utf-8")+ try:+ if kind == "symlink":+ dataset.symlink_to(outside)+ elif kind == "hardlink":+ dataset.hardlink_to(outside)+ elif hasattr(os, "mkfifo"):+ os.mkfifo(dataset)+ else:+ pytest.skip("FIFO creation is unavailable")+ except OSError as exc:+ pytest.skip(f"{kind} creation is unavailable: {exc}")++ with pytest.raises(ValueError, match="regular non-linked file"):+ generate_harbor_tasks(target, tmp_path / f"unsafe-dataset-{kind}")diff --git a/tests/test_results_location.py b/tests/test_results_location.pyindex cc073637..29beabb1 100644--- a/tests/test_results_location.py+++ b/tests/test_results_location.py@@ -6,6 +6,7 @@from __future__ import annotationsimport json+import osfrom pathlib import Pathfrom skillevaluator.tier3.results_location import external_results_root, resolve_latest_results@@ -35,6 +36,34 @@ def test_latest_results_falls_back_to_newest_completed_run_without_symlink(tmp_passert resolved == newest+def test_latest_results_fallback_accepts_unique_suffixed_run_ids(tmp_path: Path) -> None:+ skill_path = tmp_path / "demo"+ skill_path.mkdir()+ cli_results_dir = tmp_path / "results"+ root = external_results_root(cli_results_dir, skill_path)+ _write_completed_run(root, "20260705_120000_111_aaaaaaaaaaaa")+ newest = _write_completed_run(root, "20260705_130000_222_bbbbbbbbbbbb")++ resolved = resolve_latest_results(skill_path, cli_results_dir, environ={})++ assert resolved == newest+++def test_latest_results_same_timestamp_uses_completion_mtime(tmp_path: Path) -> None:+ skill_path = tmp_path / "demo"+ skill_path.mkdir()+ cli_results_dir = tmp_path / "results"+ root = external_results_root(cli_results_dir, skill_path)+ lexically_later = _write_completed_run(root, "20260705_130000_999_ffffffffffff")+ completed_later = _write_completed_run(root, "20260705_130000_111_aaaaaaaaaaaa")+ os.utime(lexically_later / "result.json", ns=(100, 100))+ os.utime(completed_later / "result.json", ns=(200, 200))++ resolved = resolve_latest_results(skill_path, cli_results_dir, environ={})++ assert resolved == completed_later++def test_latest_results_fallback_ignores_hidden_partial_and_malformed_directories(tmp_path: Path) -> None:skill_path = tmp_path / "demo"skill_path.mkdir()diff --git a/tests/test_tier3_progress.py b/tests/test_tier3_progress.pyindex e03f164c..94e62d32 100644--- a/tests/test_tier3_progress.py+++ b/tests/test_tier3_progress.py@@ -10,6 +10,7 @@ import ioimport jsonimport subprocessimport threading+from concurrent.futures import ThreadPoolExecutorfrom pathlib import Pathfrom types import SimpleNamespacefrom typing import Any, ClassVar@@ -688,6 +689,40 @@ def test_default_run_cleans_transient_harbor_artifacts(assert persisted["run_config"]["harbor"]["jobs_retained"] is False+def test_runner_rejects_preexisting_run_symlink_before_child_creation(+ monkeypatch: pytest.MonkeyPatch,+ tmp_path: Path,+) -> None:+ runner, skill = _stub_runner(monkeypatch, tmp_path)+ timestamp = "20260804_123456"+ fixed_now = SimpleNamespace(strftime=lambda _format: timestamp)+ monkeypatch.setattr(runner, "datetime", SimpleNamespace(now=lambda _timezone: fixed_now))+ monkeypatch.setattr(runner.os, "getpid", lambda: 12345)+ monkeypatch.setattr(runner, "uuid4", lambda: SimpleNamespace(hex="abcdef012345"))+ run_id = f"{timestamp}_12345_abcdef012345"+ results_root = tmp_path / "results"+ victim = tmp_path / "victim"+ results_root.mkdir()+ victim.mkdir()+ (results_root / run_id).symlink_to(victim, target_is_directory=True)++ result = runner.run_harbor_eval(skill, ["codex"], output_dir=results_root)++ assert "unique Tier 3 run directory" in str(result["error"][0])+ assert not (victim / "_harbor-jobs").exists()+++def test_runner_reserves_unique_run_directories_concurrently(tmp_path: Path) -> None:+ from skillevaluator.tier3.harbor import runner++ results_root = tmp_path / "results"+ with ThreadPoolExecutor(max_workers=8) as executor:+ run_dirs = list(executor.map(lambda _index: runner._reserve_run_dir(results_root, "20260804_123456"), range(8)))++ assert len(set(run_dirs)) == 8+ assert all((run_dir / ".skillevaluator-generated-output").is_file() for run_dir in run_dirs)++def test_runner_persists_compact_feedback_to_result_json(monkeypatch: pytest.MonkeyPatch,tmp_path: Path,diff --git a/tests/test_tier3_public_runtime.py b/tests/test_tier3_public_runtime.pyindex 962f25e6..c4293fe0 100644--- a/tests/test_tier3_public_runtime.py+++ b/tests/test_tier3_public_runtime.py@@ -18,7 +18,7 @@ from skillevaluator.cli import clifrom skillevaluator.provider_config import ProviderConfigfrom skillevaluator.tier3 import commands as tier3_commandsfrom skillevaluator.tier3.evals_config import EvalsConfigError, load_evals_config-from skillevaluator.tier3.harbor.adapter import _write_task_toml+from skillevaluator.tier3.harbor.adapter import _EVALUATOR_MANAGED_RUNTIME_ENV, _write_task_tomlfrom skillevaluator.tier3.harbor.runner import (_model_for_agent,_nvidia_build_agent_import_path,@@ -366,7 +366,10 @@ def test_generated_task_keeps_evaluator_provider_variables_out_of_agent_environm"NVIDIA_API_KEY": "${NVIDIA_API_KEY}","SKILL_EVAL_LLM_PROVIDER": "${SKILL_EVAL_LLM_PROVIDER}",}- assert task["environment"]["env"] == {"SERVICE_API_TOKEN": "${SERVICE_API_TOKEN}"}+ assert task["environment"]["env"] == {+ **_EVALUATOR_MANAGED_RUNTIME_ENV,+ "SERVICE_API_TOKEN": "${SERVICE_API_TOKEN}",+ }def test_nvidia_build_provider_mapping_does_not_supply_an_openai_agent_credential() -> None: