Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.
Python agentaichatgptdeepseekfine-tuninggemmaimage-generationllamallmllmsopenaipythonqwenreinforcement-learningself-hostedstable-diffusiontext-to-speechttsuiunsloth
Look for nvidia-smi everywhere it actually is (#11119)
Daniel Han2f3cc594df39
diff --git a/.github/workflows/cross-platform-parity-ci.yml b/.github/workflows/cross-platform-parity-ci.ymlindex b7c086a59aa..971fa05fd0c 100644--- a/.github/workflows/cross-platform-parity-ci.yml+++ b/.github/workflows/cross-platform-parity-ci.yml@@ -97,6 +97,7 @@ on:- 'tests/studio/test_amd_venv_repair_loop.ps1'- 'tests/studio/test_amd_name_arch_r9700.ps1'- 'tests/studio/test_nvidia_library_inventory.ps1'+ - 'tests/studio/test_nvidia_smi_discovery.ps1'- 'tests/studio/test_application_control_cli_fallback.ps1'# test_application_control_cli_fallback.ps1 pins install.ps1's trampoline against# process.rs's, so a change to either spelling has to run here.@@ -176,6 +177,7 @@ on:- 'tests/studio/test_amd_venv_repair_loop.ps1'- 'tests/studio/test_amd_name_arch_r9700.ps1'- 'tests/studio/test_nvidia_library_inventory.ps1'+ - 'tests/studio/test_nvidia_smi_discovery.ps1'- 'tests/studio/test_application_control_cli_fallback.ps1'# test_application_control_cli_fallback.ps1 pins install.ps1's trampoline against# process.rs's, so a change to either spelling has to run here.@@ -308,6 +310,15 @@ jobs:if: runner.os == 'Windows'shell: powershellrun: powershell -NoProfile -File tests\studio\test_nvidia_library_inventory.ps1+ # Both legs, and both matter: the candidate list is what decides whether a real GPU is+ # seen at all, and 5.1 is the host the Desktop app launches the installer in.+ - name: nvidia-smi discovery tests+ shell: pwsh+ run: pwsh -NoProfile -File tests/studio/test_nvidia_smi_discovery.ps1+ - name: nvidia-smi discovery tests (Windows PowerShell 5.1)+ if: runner.os == 'Windows'+ shell: powershell+ run: powershell -NoProfile -File tests\studio\test_nvidia_smi_discovery.ps1# AST-extracted helpers plus a real .cmd written to a temp dir, so no venv and no# Windows needed. The 1260 classification is exercised against crafted exceptions:# a hosted runner has no Application Control policy to be denied by. The enforceddiff --git a/install.ps1 b/install.ps1index 3f9889e9b15..0a236892a65 100644--- a/install.ps1+++ b/install.ps1@@ -1134,18 +1134,149 @@ function Install-UnslothStudio {}}+ # ── BEGIN SHARED WITH studio/setup.ps1 (Get-NvidiaSmiCandidatePaths) ──+ # Every directory nvidia-smi might be in, in the order worth trying, filtered to the ones that+ # actually hold the binary. Callers still run their own Test-NvidiaSmiHasGpu over the result:+ # existing is not the same as answering.+ #+ # Only two locations were searched before, and each addition below is a host where a real NVIDIA+ # GPU was reported absent, which sends the installer to CPU-only PyTorch:+ #+ # SysNative From a 32-bit process "System32" is redirected to SysWOW64, which has no+ # nvidia-smi. SysNative is the alias that reaches the real System32.+ # ProgramW64 In that same process $env:ProgramFiles is "Program Files (x86)".+ # ProgramW6432 is the 64-bit Program Files whatever the process bitness.+ # x86 Not the driver's own location, but a machine that once had a 32-bit+ # toolkit can carry a working copy there.+ # DriverStore Where the driver package itself lives. Present on hosts where the copy+ # into System32 did not happen, which is the #9255 population.+ #+ # Nothing is removed. One ordering does change, and deliberately: the two call sites disagreed+ # with each other before, one trying System32 first and the other the NVSMI directory first, and+ # a single list cannot keep both. System32 wins, because that is where the current driver puts+ # its copy; NVSMI is the legacy location and a machine carrying both can have an older binary+ # there reporting an older CUDA version. This only affects a host that has both, and only in+ # which of two working binaries answers.+ function Get-NvidiaSmiCandidatePaths {+ $dirs = @()+ # Current driver locations first, in BOTH process bitnesses, before any legacy one.+ # Both bitness spellings before $env:ProgramFiles, which in a 32-bit process is the x86 tree+ # where a stale toolkit copy can sit and answer first.+ if ($env:SystemRoot) { $dirs += (Join-Path $env:SystemRoot "System32") }+ if ($env:SystemRoot) { $dirs += (Join-Path $env:SystemRoot "SysNative") }+ if ($env:ProgramW6432) { $dirs += (Join-Path $env:ProgramW6432 "NVIDIA Corporation\NVSMI") }+ if ($env:ProgramFiles) { $dirs += (Join-Path $env:ProgramFiles "NVIDIA Corporation\NVSMI") }+ $pfx86 = ${env:ProgramFiles(x86)}+ if ($pfx86) { $dirs += (Join-Path $pfx86 "NVIDIA Corporation\NVSMI") }+ # Bounded on purpose. FileRepository holds every driver package the machine has ever had.+ #+ # The bound is applied to directories that ACTUALLY HOLD the binary, not to the nv* matches.+ # NVIDIA ships several packages whose names start nv (HD audio, the virtual audio device,+ # the network service), so a host with eight of those newer than the display driver would+ # otherwise spend the whole allowance on directories with no nvidia-smi in them and report+ # the machine as having none. Test-Path is a file existence check, not a process spawn; the+ # bound that matters is on the candidates handed back, since each of those costs a probe.+ try {+ $repos = @()+ if ($env:SystemRoot) {+ # Both spellings, same WOW64 reason as above. At most one resolves in any given process, so+ # this is not a doubled scan.+ $repos += (Join-Path $env:SystemRoot "System32\DriverStore\FileRepository")+ $repos += (Join-Path $env:SystemRoot "SysNative\DriverStore\FileRepository")+ }+ $packages = @()+ foreach ($repo in $repos) {+ if (-not (Test-Path -LiteralPath $repo -PathType Container)) { continue }+ $packages += @(Get-ChildItem -LiteralPath $repo -Directory -Filter "nv*" -ErrorAction SilentlyContinue |+ Where-Object { Test-Path -LiteralPath (Join-Path $_.FullName "nvidia-smi.exe") -PathType Leaf })+ }+ # The bound is on the whole DriverStore contribution rather than per root, so adding+ # the second spelling cannot double the number of probes this hands back.+ foreach ($dir in @($packages | Sort-Object LastWriteTime -Descending | Select-Object -First 8)) {+ $dirs += $dir.FullName+ }+ } catch {}+ $paths = @()+ $seen = @{}+ foreach ($dir in $dirs) {+ if ([string]::IsNullOrWhiteSpace($dir)) { continue }+ $candidate = Join-Path $dir "nvidia-smi.exe"+ # Case-insensitive, because the same directory reached two ways must not be probed twice:+ # each probe is a bounded process spawn with a wall-clock timeout behind it.+ $key = $candidate.ToLowerInvariant()+ if ($seen.ContainsKey($key)) { continue }+ $seen[$key] = $true+ if (Test-Path -LiteralPath $candidate -PathType Leaf) { $paths += $candidate }+ }+ return $paths+ }+ # ── END SHARED WITH studio/setup.ps1 (Get-NvidiaSmiCandidatePaths) ──+# One list, because the presence probe and the driver-version probe must agree: a host only the first finds reports a GPU with no driver version, and the CUDA-major guard is then skipped.+ # Reset per run: under `irm | iex` the script scope IS the caller's session, so a second+ # invocation in one shell would otherwise reuse the first run's answer.+ $script:WoaNvidiaSmiProbed = $false+ $script:WoaNvidiaSmiPath = $null+function Get-WoaNvidiaSmiPath {+ # Memoised because the two callers each probe, and without this the whole candidate list+ # would be walked twice with a bounded process spawn per entry.+ if ($script:WoaNvidiaSmiProbed) { return $script:WoaNvidiaSmiPath }+ $script:WoaNvidiaSmiProbed = $true$exe = $nulltry { $exe = (Get-Command nvidia-smi -ErrorAction SilentlyContinue).Source } catch { $exe = $null }- if ($exe) { return $exe }- foreach ($candidate in @(- "$env:SystemRoot\System32\nvidia-smi.exe",- "$env:ProgramFiles\NVIDIA Corporation\NVSMI\nvidia-smi.exe"- )) {- if (Test-Path -LiteralPath $candidate -PathType Leaf) { return $candidate }+ if ($exe) { $script:WoaNvidiaSmiPath = $exe; return $exe }+ $found = @(Get-NvidiaSmiCandidatePaths)+ if ($found.Count -eq 0) { return $null }+ # The first candidate that ANSWERS, not the first that exists.+ #+ # Both callers immediately probe what this returns, and treat a non-answer as "no NVIDIA+ # here": Test-WoaNvidiaPresent returns false and Get-WoaDriverCudaVersion returns null,+ # which between them decide whether the native ARM64 CUDA stack is installed at all. So+ # handing back a broken copy does not cost a retry, it declines the whole route.+ #+ # That was survivable when two locations were searched and is not now that the list is+ # longer: every location added is another chance that the first entry is a stale copy+ # sitting in front of a working one. Same reason the main detection loop stopped taking+ # the first listing candidate.+ # Same two-budget shape as the main detection loop, and for the same reasons. Each probe+ # below is a bounded process spawn, and the list can now be eleven entries, so a wedged+ # driver would otherwise hold Initialize-WoaNativeCudaTorch for minutes before the+ # install even starts. The soft budget only applies once something has answered; only the+ # hard one may end the scan with nothing, because that decides the whole ARM64 route.+ $woaDeadline = (Get-Date).AddSeconds(30)+ $woaHardDeadline = (Get-Date).AddSeconds(60)+ $woaListing = $null+ foreach ($p in $found) {+ if ($null -ne $woaListing -and (Get-Date) -gt $woaDeadline) { break }+ if ((Get-Date) -gt $woaHardDeadline) { break }+ try {+ $listing = Invoke-NvidiaSmiBounded $p @('-L')+ if (-not ($LASTEXITCODE -eq 0 -and $listing -match '(?m)^\s*GPU\s+\d+')) { continue }+ if (-not $woaListing) { $woaListing = $p }+ # Two passes, because listing a GPU is not enough here. Get-WoaDriverCudaVersion+ # asks this same binary for the CUDA banner, and a null answer does not merely+ # lose a version: it skips the CUDA-major compatibility guard in+ # Initialize-WoaNativeCudaTorch, which is what stops a native wheel newer than+ # the installed driver being chosen. So prefer a candidate that answers BOTH.+ if ((Get-Date) -gt $woaDeadline) { break }+ $banner = Invoke-NvidiaSmiBounded $p+ if ($banner -match 'CUDA(?: UMD)? Version:\s+\d+\.\d+') {+ $script:WoaNvidiaSmiPath = $p+ return $p+ }+ } catch {}}- return $null+ # One that listed a GPU but never named a version is still better than one that did not+ # answer at all: the callers' presence check works, and the version check declines.+ if ($woaListing) {+ $script:WoaNvidiaSmiPath = $woaListing+ return $woaListing+ }+ # Nothing answered. Hand back the first that exists, which is what this did before, so a+ # host where the probe cannot run is no worse off than it was.+ $script:WoaNvidiaSmiPath = $found[0]+ return $found[0]}function Test-WoaNvidiaPresent {@@ -7521,15 +7652,79 @@ exit 0}} catch {}if (-not $HasNvidiaSmi) {- foreach ($p in @(- "$env:ProgramFiles\NVIDIA Corporation\NVSMI\nvidia-smi.exe",- "$env:SystemRoot\System32\nvidia-smi.exe"- )) {- if (Test-Path $p) {- try {- if (Test-NvidiaSmiHasGpu $p) { $HasNvidiaSmi = $true; $NvidiaSmiExe = $p; break }- } catch {}- }+ # Two passes, not one, and the reason is the ordering this change introduces. Listing a+ # GPU is not the same as being able to report a CUDA version: a partially installed or+ # stale utility can answer -L and still print a banner Get-TorchIndexUrl cannot parse,+ # and that path falls back to cu126. Taking the first binary that merely lists a GPU+ # could therefore hand a cu128-capable host cu126 purely because of which directory is+ # searched first. So prefer a candidate that answers BOTH questions, and only settle for+ # one that just lists a GPU if none does.+ #+ # One deadline across the whole loop, not just the per-call one. Each candidate gets its own+ # bounded runner, and on a host with a wedged driver every one of them waits out that bound+ # before failing. The list is longer than it was, so what used to be two stalls could now be+ # ten, and all of it is spent in front of the driver-library fallback that would have answered+ # in milliseconds. The bound is on TIME rather than on the number of candidates because the+ # cost being controlled is time: a machine where every probe answers at once still gets to+ # try them all, and one where they hang stops early.+ #+ # Get-Date rather than a Stopwatch: Constrained Language Mode refuses the method calls, and+ # this runs on hosts that enforce it.+ # Enumerated BEFORE the clock starts. Discovery is Test-Path calls, but on a machine with a+ # filesystem filter driver or slow storage it is not free, and charging it to the probe+ # budget is how a slow disk turns into no GPU.+ $smiCandidates = @(Get-NvidiaSmiCandidatePaths)+ # Two budgets, because "stop early" means two different things here.+ #+ # The soft one applies ONLY once a usable answer is already in hand: there is a working+ # nvidia-smi, and continuing just looks for a better CUDA banner. Giving that up costs at+ # most a wheel family.+ #+ # The hard one is the only thing allowed to end the search with NOTHING found, because that+ # outcome is CPU-only PyTorch on a machine with a GPU. The first version of this had a single+ # budget and broke out of the loop after one slow failure, so a host whose System32 copy is+ # broken and whose legacy NVSMI copy works lost its GPU entirely. That is strictly worse than+ # the stall it was added to prevent.+ $probeDeadline = (Get-Date).AddSeconds(30)+ $probeHardDeadline = (Get-Date).AddSeconds(60)+ $firstListing = $null+ # Captured beside the path, because $script:NvidiaSmiWedged describes the LAST binary probed+ # and the one this loop settles on can be an earlier one. Captured rather than assumed to be+ # false: it is false today because a candidate only becomes $firstListing when its -L probe+ # succeeded, and that is a property of Test-NvidiaSmiHasGpu rather than of this loop.+ $firstListingWedged = $false+ foreach ($p in $smiCandidates) {+ # Only give up early when there is already something to fall back on.+ if ($null -ne $firstListing -and (Get-Date) -gt $probeDeadline) { break }+ # With nothing found, keep going to the hard bound. Reporting no GPU is the expensive+ # answer, so it has to be the one that costs the most before it is reached.+ if ((Get-Date) -gt $probeHardDeadline) { break }+ try {+ if (-not (Test-NvidiaSmiHasGpu $p)) { continue }+ if (-not $firstListing) {+ $firstListing = $p+ $firstListingWedged = $script:NvidiaSmiWedged+ }+ # Rechecked here, not only at the top. The listing probe that just returned can have+ # spent most of its own bound, and the banner probe below has a full bound of its+ # own, so a candidate starting just inside the deadline could add nearly twice the+ # per-probe timeout after it. There is already a usable answer in $firstListing at+ # this point, so stopping costs at most a wheel family.+ if ((Get-Date) -gt $probeDeadline) { break }+ $banner = Invoke-NvidiaSmiBounded $p+ if ($banner -match 'CUDA(?: UMD)? Version:\s+\d+\.\d+') {+ $HasNvidiaSmi = $true; $NvidiaSmiExe = $p; break+ }+ } catch {}+ }+ if (-not $HasNvidiaSmi -and $firstListing) {+ $HasNvidiaSmi = $true; $NvidiaSmiExe = $firstListing+ # Restored for the binary actually selected. A later candidate that timed out leaves the+ # flag set for ITS path, and the guard downstream reads the flag to decide whether to ask+ # the selected binary for the name, the compute capability and the driver version. Left+ # alone, a working nvidia-smi would be treated as wedged and all three queries skipped+ # because a different binary elsewhere on the machine hung.+ $script:NvidiaSmiWedged = $firstListingWedged}}if (-not $HasNvidiaSmi -and (Get-NvidiaLibraryInventory)) {diff --git a/studio/setup.ps1 b/studio/setup.ps1index 1f77756722e..7c90a7ec987 100644--- a/studio/setup.ps1+++ b/studio/setup.ps1@@ -2647,6 +2647,84 @@ function Invoke-NvidiaSmiBounded {}}+# ── BEGIN SHARED WITH install.ps1 (Get-NvidiaSmiCandidatePaths) ──+# Every directory nvidia-smi might be in, in the order worth trying, filtered to the ones that+# actually hold the binary. Callers still run their own Test-NvidiaSmiHasGpu over the result:+# existing is not the same as answering.+#+# Only two locations were searched before, and each addition below is a host where a real NVIDIA+# GPU was reported absent, which sends the installer to CPU-only PyTorch:+#+# SysNative From a 32-bit process "System32" is redirected to SysWOW64, which has no+# nvidia-smi. SysNative is the alias that reaches the real System32.+# ProgramW64 In that same process $env:ProgramFiles is "Program Files (x86)".+# ProgramW6432 is the 64-bit Program Files whatever the process bitness.+# x86 Not the driver's own location, but a machine that once had a 32-bit+# toolkit can carry a working copy there.+# DriverStore Where the driver package itself lives. Present on hosts where the copy+# into System32 did not happen, which is the #9255 population.+#+# Nothing is removed. One ordering does change, and deliberately: the two call sites disagreed+# with each other before, one trying System32 first and the other the NVSMI directory first, and+# a single list cannot keep both. System32 wins, because that is where the current driver puts+# its copy; NVSMI is the legacy location and a machine carrying both can have an older binary+# there reporting an older CUDA version. This only affects a host that has both, and only in+# which of two working binaries answers.+function Get-NvidiaSmiCandidatePaths {+ $dirs = @()+ # Current driver locations first, in BOTH process bitnesses, before any legacy one.+ # Both bitness spellings before $env:ProgramFiles, which in a 32-bit process is the x86 tree+ # where a stale toolkit copy can sit and answer first.+ if ($env:SystemRoot) { $dirs += (Join-Path $env:SystemRoot "System32") }+ if ($env:SystemRoot) { $dirs += (Join-Path $env:SystemRoot "SysNative") }+ if ($env:ProgramW6432) { $dirs += (Join-Path $env:ProgramW6432 "NVIDIA Corporation\NVSMI") }+ if ($env:ProgramFiles) { $dirs += (Join-Path $env:ProgramFiles "NVIDIA Corporation\NVSMI") }+ $pfx86 = ${env:ProgramFiles(x86)}+ if ($pfx86) { $dirs += (Join-Path $pfx86 "NVIDIA Corporation\NVSMI") }+ # Bounded on purpose. FileRepository holds every driver package the machine has ever had.+ #+ # The bound is applied to directories that ACTUALLY HOLD the binary, not to the nv* matches.+ # NVIDIA ships several packages whose names start nv (HD audio, the virtual audio device,+ # the network service), so a host with eight of those newer than the display driver would+ # otherwise spend the whole allowance on directories with no nvidia-smi in them and report+ # the machine as having none. Test-Path is a file existence check, not a process spawn; the+ # bound that matters is on the candidates handed back, since each of those costs a probe.+ try {+ $repos = @()+ if ($env:SystemRoot) {+ # Both spellings, same WOW64 reason as above. At most one resolves in any given process, so+ # this is not a doubled scan.+ $repos += (Join-Path $env:SystemRoot "System32\DriverStore\FileRepository")+ $repos += (Join-Path $env:SystemRoot "SysNative\DriverStore\FileRepository")+ }+ $packages = @()+ foreach ($repo in $repos) {+ if (-not (Test-Path -LiteralPath $repo -PathType Container)) { continue }+ $packages += @(Get-ChildItem -LiteralPath $repo -Directory -Filter "nv*" -ErrorAction SilentlyContinue |+ Where-Object { Test-Path -LiteralPath (Join-Path $_.FullName "nvidia-smi.exe") -PathType Leaf })+ }+ # The bound is on the whole DriverStore contribution rather than per root, so adding+ # the second spelling cannot double the number of probes this hands back.+ foreach ($dir in @($packages | Sort-Object LastWriteTime -Descending | Select-Object -First 8)) {+ $dirs += $dir.FullName+ }+ } catch {}+ $paths = @()+ $seen = @{}+ foreach ($dir in $dirs) {+ if ([string]::IsNullOrWhiteSpace($dir)) { continue }+ $candidate = Join-Path $dir "nvidia-smi.exe"+ # Case-insensitive, because the same directory reached two ways must not be probed twice:+ # each probe is a bounded process spawn with a wall-clock timeout behind it.+ $key = $candidate.ToLowerInvariant()+ if ($seen.ContainsKey($key)) { continue }+ $seen[$key] = $true+ if (Test-Path -LiteralPath $candidate -PathType Leaf) { $paths += $candidate }+ }+ return $paths+}+# ── END SHARED WITH install.ps1 (Get-NvidiaSmiCandidatePaths) ──+# A driverless nvidia-smi exits 0 listing no GPU, so require a "GPU <n>:" row.# 124 is what Invoke-NvidiaSmiBounded reports when it had to kill the probe. Recorded so# the banner below can skip a second query: detection already waited out the full bound on@@ -2672,21 +2750,83 @@ try {}} catch {}if (-not $HasNvidiaSmi) {- $nvSmiDefaults = @(- "$env:ProgramFiles\NVIDIA Corporation\NVSMI\nvidia-smi.exe",- "$env:SystemRoot\System32\nvidia-smi.exe"- )- foreach ($p in $nvSmiDefaults) {- if (Test-Path $p) {- try {- if (Test-NvidiaSmiHasGpu $p) {- $HasNvidiaSmi = $true- $NvidiaSmiExe = $p- Write-StudioLine " Found nvidia-smi at $(Split-Path $p -Parent)" -ForegroundColor Gray- break- }- } catch {}- }+ # Two passes, not one, and the reason is the ordering this change introduces. Listing a GPU is+ # not the same as being able to report a CUDA version: a partially installed or stale utility+ # can answer -L and still print a banner the version ladder cannot parse. Taking the first+ # binary that merely lists a GPU could hand a newer-capable host an older CUDA family purely+ # because of which directory is searched first. Prefer a candidate that answers BOTH, and+ # settle for one that just lists a GPU only if none does.+ #+ # One deadline across the whole loop, not just the per-call one. Each candidate gets its own+ # bounded runner, and on a host with a wedged driver every one of them waits out that bound+ # before failing. The list is longer than it was, so what used to be two stalls could now be+ # ten, and all of it is spent in front of the driver-library fallback that would have answered+ # in milliseconds. The bound is on TIME rather than on the number of candidates because the+ # cost being controlled is time: a machine where every probe answers at once still gets to+ # try them all, and one where they hang stops early.+ #+ # Get-Date rather than a Stopwatch: Constrained Language Mode refuses the method calls, and+ # this runs on hosts that enforce it.+ # Enumerated BEFORE the clock starts. Discovery is Test-Path calls, but on a machine with a+ # filesystem filter driver or slow storage it is not free, and charging it to the probe+ # budget is how a slow disk turns into no GPU.+ $smiCandidates = @(Get-NvidiaSmiCandidatePaths)+ # Two budgets, because "stop early" means two different things here.+ #+ # The soft one applies ONLY once a usable answer is already in hand: there is a working+ # nvidia-smi, and continuing just looks for a better CUDA banner. Giving that up costs at+ # most a wheel family.+ #+ # The hard one is the only thing allowed to end the search with NOTHING found, because that+ # outcome is CPU-only PyTorch on a machine with a GPU. The first version of this had a single+ # budget and broke out of the loop after one slow failure, so a host whose System32 copy is+ # broken and whose legacy NVSMI copy works lost its GPU entirely. That is strictly worse than+ # the stall it was added to prevent.+ $probeDeadline = (Get-Date).AddSeconds(30)+ $probeHardDeadline = (Get-Date).AddSeconds(60)+ $firstListing = $null+ # Captured beside the path, because $script:NvidiaSmiWedged describes the LAST binary probed+ # and the one this loop settles on can be an earlier one. Captured rather than assumed to be+ # false: it is false today because a candidate only becomes $firstListing when its -L probe+ # succeeded, and that is a property of Test-NvidiaSmiHasGpu rather than of this loop.+ $firstListingWedged = $false+ foreach ($p in $smiCandidates) {+ # Only give up early when there is already something to fall back on.+ if ($null -ne $firstListing -and (Get-Date) -gt $probeDeadline) { break }+ # With nothing found, keep going to the hard bound. Reporting no GPU is the expensive+ # answer, so it has to be the one that costs the most before it is reached.+ if ((Get-Date) -gt $probeHardDeadline) { break }+ try {+ if (-not (Test-NvidiaSmiHasGpu $p)) { continue }+ if (-not $firstListing) {+ $firstListing = $p+ $firstListingWedged = $script:NvidiaSmiWedged+ }+ # Rechecked here, not only at the top. The listing probe that just returned can have+ # spent most of its own bound, and the banner probe below has a full bound of its+ # own, so a candidate starting just inside the deadline could add nearly twice the+ # per-probe timeout after it. There is already a usable answer in $firstListing at+ # this point, so stopping costs at most a wheel family.+ if ((Get-Date) -gt $probeDeadline) { break }+ $banner = Invoke-NvidiaSmiBounded $p+ if ($banner -match 'CUDA(?: UMD)? Version:\s+\d+\.\d+') {+ $HasNvidiaSmi = $true+ $NvidiaSmiExe = $p+ Write-StudioLine " Found nvidia-smi at $(Split-Path $p -Parent)" -ForegroundColor Gray+ break+ }+ } catch {}+ }+ if (-not $HasNvidiaSmi -and $firstListing) {+ $HasNvidiaSmi = $true+ $NvidiaSmiExe = $firstListing+ # Restored for the binary actually selected. A later candidate that timed out leaves the+ # flag set for ITS path, and the guard downstream reads the flag to decide whether to ask+ # the selected binary for the name, the compute capability and the driver version. Left+ # alone, a working nvidia-smi would be treated as wedged and all three queries skipped+ # because a different binary elsewhere on the machine hung.+ $script:NvidiaSmiWedged = $firstListingWedged+ Write-StudioLine " Found nvidia-smi at $(Split-Path $firstListing -Parent)" -ForegroundColor Gray}}if (-not $HasNvidiaSmi -and (Get-NvidiaLibraryInventory)) {diff --git a/tests/studio/install/test_woa_torch_index_persistence.py b/tests/studio/install/test_woa_torch_index_persistence.pyindex 7f600010ae7..dbbaa73ecb9 100644--- a/tests/studio/install/test_woa_torch_index_persistence.py+++ b/tests/studio/install/test_woa_torch_index_persistence.py@@ -4264,6 +4264,7 @@ class TestTheEarlyNvidiaProbesAreBounded:_function_source(INSTALL_SRC, "Invoke-NvidiaSmiBounded"),# The probe resolves its executable through this; without it the call is# unresolved and the probe answers False for a reason the test is not about.+ _function_source(INSTALL_SRC, "Get-NvidiaSmiCandidatePaths"),_function_source(INSTALL_SRC, "Get-WoaNvidiaSmiPath"),_function_source(INSTALL_SRC, "Test-WoaNvidiaPresent"),f"$env:PATH = '{tmp_path}' + [System.IO.Path]::PathSeparator + $env:PATH",@@ -4283,6 +4284,7 @@ class TestTheEarlyNvidiaProbesAreBounded:# Required, and easy to miss: an unresolved lookup makes Get-WoaDriverCudaVersion# return $null before it ever calls nvidia-smi, so the "[]" below would pass# without the timeout this test exists to bound ever being exercised.+ _function_source(INSTALL_SRC, "Get-NvidiaSmiCandidatePaths"),_function_source(INSTALL_SRC, "Get-WoaNvidiaSmiPath"),_function_source(INSTALL_SRC, "Get-WoaDriverCudaVersion"),f"$env:PATH = '{tmp_path}' + [System.IO.Path]::PathSeparator + $env:PATH",@@ -5736,8 +5738,22 @@ class TestBothNvidiaSmiProbesSearchTheSameLocations:def test_the_shared_helper_lists_every_supported_location(self):body = _function_source(INSTALL_SRC, "Get-WoaNvidiaSmiPath")assert "Get-Command nvidia-smi" in body+ # The candidate list moved into Get-NvidiaSmiCandidatePaths, which install.ps1 and+ # studio/setup.ps1 share, so the WoA probe and the ordinary Windows probe cannot drift+ # apart either. What this class is about is unchanged: one list, and both probes on it.+ assert "Get-NvidiaSmiCandidatePaths" in body, "the WoA probe no longer uses the shared list"+ candidates = _function_source(INSTALL_SRC, "Get-NvidiaSmiCandidatePaths")+ assert '"nvidia-smi.exe"' in candidates+ # The list builds each directory with Join-Path, so a whole path literal never appears in+ # it. Check the parts, which is what it actually promises. Behaviour, including the+ # locations added beyond these two, is driven for real against a planted filesystem in+ # tests/studio/test_nvidia_smi_discovery.ps1.for location in self.LOCATIONS:- assert location in body, location+ directory, _, leaf = location.rpartition("\\")+ assert leaf == "nvidia-smi.exe", location+ root, _, rest = directory.partition("\\")+ assert root in candidates, location+ assert f'"{rest}"' in candidates, location@pytest.mark.parametrize("name", ("Test-WoaNvidiaPresent", "Get-WoaDriverCudaVersion"))def test_neither_probe_keeps_its_own_candidate_list(self, name):@@ -5765,16 +5781,25 @@ class TestBothNvidiaSmiProbesSearchTheSameLocations:if "Get-WoaNvidiaSmiPath" in _function_source(INSTALL_SRC, name)]assert callers, "neither probe routes through the shared lookup any more"+ # Follow the chain rather than naming one helper: Get-WoaNvidiaSmiPath now calls+ # Get-NvidiaSmiCandidatePaths, and a composition missing the second one fails in exactly+ # the silent way this test exists to catch.+ needed = ["Get-WoaNvidiaSmiPath"]+ for helper in needed:+ for callee in ("Get-NvidiaSmiCandidatePaths",):+ if callee in _function_source(INSTALL_SRC, helper) and callee not in needed:+ needed.append(callee)own = pathlib.Path(__file__).read_text(encoding = "utf-8")for name in callers:for match in re.finditer(rf'_function_source\(INSTALL_SRC, "{re.escape(name)}"\)', own):# The _script(...) call this appears in, back to its opening paren.start = own.rindex("_script(", 0, match.start())block = own[start : own.index("\n )", match.end())]- assert '_function_source(INSTALL_SRC, "Get-WoaNvidiaSmiPath")' in block, (- f"a composed script injects {name}, which calls Get-WoaNvidiaSmiPath, "- "without injecting that helper"- )+ for helper in needed:+ assert f'_function_source(INSTALL_SRC, "{helper}")' in block, (+ f"a composed script injects {name}, which reaches {helper}, "+ "without injecting that helper"+ )def test_the_guard_this_protects_is_still_there(self):# If the CUDA-major check ever goes away, this whole class is pointless; say so loudly.diff --git a/tests/studio/test_nvidia_smi_discovery.ps1 b/tests/studio/test_nvidia_smi_discovery.ps1new file mode 100644index 00000000000..3b258ef95dc--- /dev/null+++ b/tests/studio/test_nvidia_smi_discovery.ps1@@ -0,0 +1,525 @@+#!/usr/bin/env pwsh+# SPDX-License-Identifier: AGPL-3.0-only+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0+# Where the installer looks for nvidia-smi.+#+# $HasNvidiaSmi does not mean "nvidia-smi exists". It means "an NVIDIA GPU is present", and the+# whole wheel-index decision hangs off it: a host where nvidia-smi is not found gets CPU-only+# PyTorch with no warning. Two directories were searched, and real hosts have a real GPU in+# neither of them (#9255).+#+# Driven against a fake filesystem rather than asserted by reading the source. Each check plants+# exactly one copy, so a location that stops being searched fails on its own row.+# Run: pwsh -NoProfile -File tests/studio/test_nvidia_smi_discovery.ps1++$ErrorActionPreference = "Stop"+$root = (Resolve-Path ([System.IO.Path]::Combine($PSScriptRoot, "..", ".."))).Path+$installPs1 = Join-Path $root "install.ps1"+$setupPs1 = Join-Path $root "studio/setup.ps1"++$failures = 0+function Check($name, $cond) {+ if ($cond) { Write-Host " PASS $name" }+ else { Write-Host " FAIL $name" -ForegroundColor Red; $script:failures++ }+}++function Get-FunctionText($file, $name) {+ $errors = $null+ $ast = [System.Management.Automation.Language.Parser]::ParseFile($file, [ref]$null, [ref]$errors)+ if ($errors) { $errors | ForEach-Object { $_.ToString() }; throw "$file has parse errors" }+ $fn = $ast.FindAll({ param($n)+ $n -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $n.Name -eq $name+ }, $true)+ if ($fn.Count -lt 1) { throw "expected $name in $file, found none" }+ return $fn[0].Extent.Text+}++# The two copies have to stay the same helper. install.ps1 nests one level deeper, so compare+# without indentation, the same way test_nvidia_library_inventory.ps1 does.+$installText = Get-FunctionText $installPs1 "Get-NvidiaSmiCandidatePaths"+$setupText = Get-FunctionText $setupPs1 "Get-NvidiaSmiCandidatePaths"+$strip = { param($t) (($t -split "`n") | ForEach-Object { $_.TrimStart() }) -join "`n" }+Check "install.ps1 and setup.ps1 carry the same Get-NvidiaSmiCandidatePaths" (+ (& $strip $installText) -eq (& $strip $setupText))++Invoke-Expression $setupText++$tmp = Join-Path ([System.IO.Path]::GetTempPath()) ("nvsmi-" + [guid]::NewGuid().ToString("N"))+$saved = @{+ SystemRoot = $env:SystemRoot+ ProgramFiles = $env:ProgramFiles+ ProgramW6432 = $env:ProgramW6432+ ProgramFilesX86 = ${env:ProgramFiles(x86)}+}++function Reset-FakeMachine {+ if (Test-Path -LiteralPath $tmp) { Remove-Item -LiteralPath $tmp -Recurse -Force }+ $env:SystemRoot = Join-Path $tmp "Windows"+ $env:ProgramFiles = Join-Path $tmp "Program Files"+ $env:ProgramW6432 = Join-Path $tmp "Program Files"+ ${env:ProgramFiles(x86)} = Join-Path $tmp "Program Files (x86)"+}++function Add-FakeSmi($relative) {+ $dir = Join-Path $tmp $relative+ New-Item -ItemType Directory -Force -Path $dir | Out-Null+ $exe = Join-Path $dir "nvidia-smi.exe"+ [System.IO.File]::WriteAllText($exe, "")+ return $exe+}++try {+ # Each location on its own. A single planted copy has to be the single answer, so a rung+ # dropped from the list fails here and nowhere else.+ foreach ($case in @(+ @{ Name = "System32, where a current driver puts it"; Dir = "Windows/System32" },+ @{ Name = "the legacy NVSMI directory"; Dir = "Program Files/NVIDIA Corporation/NVSMI" },+ @{ Name = "SysNative, which is System32 seen from a 32-bit process"; Dir = "Windows/SysNative" },+ @{ Name = "Program Files (x86), left by an old 32-bit toolkit"; Dir = "Program Files (x86)/NVIDIA Corporation/NVSMI" },+ @{ Name = "the DriverStore package the driver ships in"; Dir = "Windows/System32/DriverStore/FileRepository/nvmii.inf_amd64_1234/" }+ )) {+ Reset-FakeMachine+ $planted = Add-FakeSmi $case.Dir+ $found = @(Get-NvidiaSmiCandidatePaths)+ Check "found in $($case.Name)" ($found.Count -eq 1 -and $found[0] -eq $planted)+ }++ # ProgramW6432 is the 64-bit Program Files whatever the process bitness. Model the 32-bit+ # host: $env:ProgramFiles is the x86 tree, and the driver is only in the 64-bit one.+ Reset-FakeMachine+ $env:ProgramFiles = Join-Path $tmp "Program Files (x86)"+ $planted = Add-FakeSmi "Program Files/NVIDIA Corporation/NVSMI"+ $found = @(Get-NvidiaSmiCandidatePaths)+ Check "found through ProgramW6432 when ProgramFiles points at the x86 tree" (+ $found.Count -eq 1 -and $found[0] -eq $planted)++ # Bites control. A host with no copy anywhere must come back empty, or every check above+ # would pass on a helper that simply returns a fixed list without testing for the file.+ Reset-FakeMachine+ New-Item -ItemType Directory -Force -Path (Join-Path $tmp "Windows/System32") | Out-Null+ Check "control: nothing planted, nothing found" ((@(Get-NvidiaSmiCandidatePaths)).Count -eq 0)++ # A directory named but absent must not throw, since this runs before anything is installed.+ Reset-FakeMachine+ Check "a machine missing every directory is handled" ((@(Get-NvidiaSmiCandidatePaths)).Count -eq 0)++ # The current driver's copy wins over the legacy one. Both work; the System32 copy belongs to+ # the driver that is actually loaded, and the NVSMI one can be older and report an older CUDA.+ Reset-FakeMachine+ $current = Add-FakeSmi "Windows/System32"+ $null = Add-FakeSmi "Program Files/NVIDIA Corporation/NVSMI"+ $found = @(Get-NvidiaSmiCandidatePaths)+ Check "System32 is offered before the legacy NVSMI directory" (+ $found.Count -eq 2 -and $found[0] -eq $current)++ # Each probe is a process spawn with a wall-clock bound behind it, so the same file reached+ # two ways must be offered once. ProgramFiles and ProgramW6432 are equal on a 64-bit host,+ # which is the ordinary case, not a contrived one.+ Reset-FakeMachine+ $planted = Add-FakeSmi "Program Files/NVIDIA Corporation/NVSMI"+ $found = @(Get-NvidiaSmiCandidatePaths)+ Check "a directory reached twice is probed once" ($found.Count -eq 1 -and $found[0] -eq $planted)++ # DriverStore holds every package the machine has ever had. Walking it whole on a host with a+ # long driver history is the kind of thing that turns an install into a hang.+ Reset-FakeMachine+ for ($i = 0; $i -lt 40; $i++) { $null = Add-FakeSmi "Windows/System32/DriverStore/FileRepository/nv$i.inf_amd64_$i/" }+ $null = Add-FakeSmi "Windows/System32/DriverStore/FileRepository/unrelated.inf_amd64_9/"+ $found = @(Get-NvidiaSmiCandidatePaths)+ Check "the DriverStore scan is bounded and filtered to NVIDIA's own packages" (+ $found.Count -le 8 -and -not ($found -match "unrelated"))++ # Same WOW64 reason SysNative is in the direct list: in a 32-bit process "System32" redirects+ # to SysWOW64, which carries no DriverStore at all, so a host whose ONLY nvidia-smi is the+ # driver package was reported as having no NVIDIA GPU. That is the population this recovers.+ Reset-FakeMachine+ $planted = Add-FakeSmi "Windows/SysNative/DriverStore/FileRepository/nvmii.inf_amd64_0001/"+ $found = @(Get-NvidiaSmiCandidatePaths)+ Check "the DriverStore is reached through SysNative as well as System32" (+ $found.Count -eq 1 -and $found[0] -eq $planted)++ # The two spellings name one directory on a real host, so the bound is on the whole+ # DriverStore contribution rather than per root: adding SysNative must not double the probes.+ Reset-FakeMachine+ for ($i = 0; $i -lt 12; $i++) { $null = Add-FakeSmi "Windows/System32/DriverStore/FileRepository/nva$i.inf_amd64_$i/" }+ for ($i = 0; $i -lt 12; $i++) { $null = Add-FakeSmi "Windows/SysNative/DriverStore/FileRepository/nvb$i.inf_amd64_$i/" }+ $found = @(Get-NvidiaSmiCandidatePaths)+ Check "both DriverStore spellings still share one bound of eight" ($found.Count -le 8)++ # The bound applies to directories that actually HOLD the binary, not to the nv* name matches.+ # NVIDIA ships nv-something packages with no nvidia-smi in them (HD audio, virtual audio,+ # network service); installed after the display driver they are newer by write time, so a+ # bound on the name match spends the whole allowance on empty directories and reports no+ # nvidia-smi. Plant the real one FIRST so it is the oldest, then bury it.+ Reset-FakeMachine+ $repo = "Windows/System32/DriverStore/FileRepository"+ $planted = Add-FakeSmi "$repo/nvmii.inf_amd64_0001/"+ for ($i = 0; $i -lt 12; $i++) {+ $dir = Join-Path $tmp "$repo/nvhda.inf_amd64_10$i"+ New-Item -ItemType Directory -Force -Path $dir | Out-Null+ [System.IO.File]::WriteAllText((Join-Path $dir "nvhda.sys"), "")+ }+ $found = @(Get-NvidiaSmiCandidatePaths)+ Check "a display driver buried under newer nv* packages that ship no nvidia-smi is still found" (+ $found.Count -eq 1 -and $found[0] -eq $planted)++ # The 32-bit process. $env:ProgramFiles is the x86 tree, so a stale copy from an old 32-bit+ # toolkit sits in the FIRST Program Files rung unless the 64-bit locations come first. It is a+ # real binary reporting its own older CUDA version, so only the order rescues this.+ Reset-FakeMachine+ $env:ProgramFiles = Join-Path $tmp "Program Files (x86)"+ $env:ProgramW6432 = Join-Path $tmp "Program Files"+ $stale = Add-FakeSmi "Program Files (x86)/NVIDIA Corporation/NVSMI"+ $current = Add-FakeSmi "Windows/SysNative"+ $found = @(Get-NvidiaSmiCandidatePaths)+ Check "in a 32-bit process SysNative is probed before the stale x86 toolkit copy" (+ $found.Count -eq 2 -and $found[0] -eq $current -and $found[1] -eq $stale)++ # Same host, driver in the 64-bit Program Files rather than SysNative.+ Reset-FakeMachine+ $env:ProgramFiles = Join-Path $tmp "Program Files (x86)"+ $env:ProgramW6432 = Join-Path $tmp "Program Files"+ $stale = Add-FakeSmi "Program Files (x86)/NVIDIA Corporation/NVSMI"+ $current = Add-FakeSmi "Program Files/NVIDIA Corporation/NVSMI"+ $found = @(Get-NvidiaSmiCandidatePaths)+ Check "in a 32-bit process the 64-bit NVSMI copy is probed before the x86 one" (+ $found.Count -eq 2 -and $found[0] -eq $current -and $found[1] -eq $stale)+} finally {+ $env:SystemRoot = $saved.SystemRoot+ $env:ProgramFiles = $saved.ProgramFiles+ if ($null -eq $saved.ProgramW6432) { Remove-Item Env:ProgramW6432 -ErrorAction SilentlyContinue }+ else { $env:ProgramW6432 = $saved.ProgramW6432 }+ if ($null -eq $saved.ProgramFilesX86) { Remove-Item "Env:ProgramFiles(x86)" -ErrorAction SilentlyContinue }+ else { ${env:ProgramFiles(x86)} = $saved.ProgramFilesX86 }+ if (Test-Path -LiteralPath $tmp) { Remove-Item -LiteralPath $tmp -Recurse -Force -ErrorAction SilentlyContinue }+}++# ------------------------------------------------- which candidate the detection loop settles on+#+# The scenario this reordering could have caused. Two binaries both answer -L; the first prints a+# banner with no parseable CUDA version, the second reports 12.8. Taking the first would send a+# cu128-capable host to cu126 purely because of which directory is searched first.+$setupAll = Get-Content -Raw -LiteralPath $setupPs1+$start = $setupAll.IndexOf('$smiCandidates = @(Get-NvidiaSmiCandidatePaths)')+if ($start -lt 0) { Check "the detection loop still makes two passes" $false }+else {+ # Bounded by the next statement rather than by a character count. A fixed 1400 stopped+ # short of the fallback the moment a comment was added above it, and the check failed+ # for a reason that had nothing to do with the loop.+ $loopEndForRead = $setupAll.IndexOf('if (-not $HasNvidiaSmi -and (Get-NvidiaLibraryInventory))', $start)+ $loop = $setupAll.Substring($start, $loopEndForRead - $start)+ Check "the loop prefers a candidate that also reports a CUDA version" (+ $loop -match "CUDA\(\?: UMD\)\? Version:")+ Check "and still settles for one that only lists a GPU when none reports a version" (+ $loop -match '\$firstListing' -and $loop -match 'if \(-not \$HasNvidiaSmi -and \$firstListing\)')+}++# Driven, not read. Both candidates list a GPU; only the second names a version.+$script:SmiCalls = @()+function Test-NvidiaSmiHasGpu { param([string]$Exe) $script:SmiCalls += "L:$Exe"; return $true }+function Invoke-NvidiaSmiBounded {+ param($Exe, $Arguments)+ $script:SmiCalls += "B:$Exe"+ if ($Exe -like "*second*") { return "CUDA Version: 12.8" }+ return "NVIDIA-SMI has failed because it couldn't communicate with the driver"+}+function Write-StudioLine { param([string]$Line, [string]$ForegroundColor = "") }+function Get-NvidiaSmiCandidatePaths { return @("C:\first\nvidia-smi.exe", "C:\second\nvidia-smi.exe") }+# Anchored on the deadline, the first statement of the loop's setup. Anchoring on $firstListing+# left $probeDeadline undefined inside the slice, and comparing against $null broke out on the+# first iteration, so every driven check below passed or failed for the wrong reason.+$loopStart = $setupAll.IndexOf('$smiCandidates = @(Get-NvidiaSmiCandidatePaths)')+$loopEnd = $setupAll.IndexOf('if (-not $HasNvidiaSmi -and (Get-NvidiaLibraryInventory))', $loopStart)+# The slice starts inside setup.ps1's "if (-not $HasNvidiaSmi) {" block, so it carries that+# block's closing brace and is short one opening brace. Supply the opener rather than trimming a+# brace off the end, which would silently drop the last statement.+$loopSrc = 'if ($true) {' + [System.Environment]::NewLine + $setupAll.Substring($loopStart, $loopEnd - $loopStart)+$HasNvidiaSmi = $false+$NvidiaSmiExe = $null+Invoke-Expression $loopSrc+Check "the candidate that reports a CUDA version wins over the one searched first" (+ $HasNvidiaSmi -eq $true -and $NvidiaSmiExe -eq "C:\second\nvidia-smi.exe")++# And when NOTHING reports a version, the first that lists a GPU is still used: this must not+# turn a detected GPU into no GPU at all.+function Invoke-NvidiaSmiBounded { param($Exe, $Arguments) return "no version here" }+$HasNvidiaSmi = $false+$NvidiaSmiExe = $null+Invoke-Expression $loopSrc+Check "with no version anywhere, the first listing candidate is still taken" (+ $HasNvidiaSmi -eq $true -and $NvidiaSmiExe -eq "C:\first\nvidia-smi.exe")++# ----------------------------------------------------- and it gives up before setup does+#+# Each candidate gets its own bounded runner, and on a wedged driver every one waits out that+# bound. A longer list turns two stalls into ten, all in front of a driver-library fallback that+# answers in milliseconds. Driven with probes that really sleep: the thing under test is elapsed+# time, and a stubbed clock would only prove the arithmetic.+#+# The deadline in the shipped file is 30 seconds, which no test should sit through. The slice is+# rewritten to 2 before it is invoked, and the real value is asserted separately just below so+# that rewrite cannot quietly become the thing being tested.+Check "the shipped soft deadline is 30 seconds" ($setupAll -match '\$probeDeadline = \(Get-Date\)\.AddSeconds\(30\)')+Check "the shipped hard deadline is 60 seconds" ($setupAll -match '\$probeHardDeadline = \(Get-Date\)\.AddSeconds\(60\)')+$fastLoop = ($loopSrc -replace 'AddSeconds\(30\)', 'AddSeconds(2)') -replace 'AddSeconds\(60\)', 'AddSeconds(4)'+Check "the slice really carries both shortened deadlines (bites)" (+ $fastLoop -match 'AddSeconds\(2\)' -and $fastLoop -match 'AddSeconds\(4\)')++$script:Probed = 0+function Test-NvidiaSmiHasGpu { param([string]$Exe) $script:Probed++; Start-Sleep -Milliseconds 700; return $true }+function Invoke-NvidiaSmiBounded { param($Exe, $Arguments) return "no version here" }+function Get-NvidiaSmiCandidatePaths {+ return @(1..10 | ForEach-Object { "C:\wedged$_\nvidia-smi.exe" })+}+$HasNvidiaSmi = $false+$NvidiaSmiExe = $null+$started = Get-Date+Invoke-Expression $fastLoop+$elapsed = ((Get-Date) - $started).TotalSeconds++Check "a wedged host stops probing instead of walking every candidate" ($script:Probed -lt 10)+Check "it probed at least once rather than skipping the loop outright" ($script:Probed -ge 1)+Check "the whole loop is bounded near its deadline, not by the candidate count" ($elapsed -lt 6)+# The bound must not cost a detection. A host that answers is still detected, and it is the FIRST+# listing candidate, exactly as it was without the deadline.+Check "giving up early still reports the GPU it did find" (+ $HasNvidiaSmi -eq $true -and $NvidiaSmiExe -eq "C:\wedged1\nvidia-smi.exe")++# Bites control: the same loop on a host where nothing hangs walks the entire list, so the check+# above is about elapsed time and not about the loop simply stopping after one candidate.+$script:Probed = 0+function Test-NvidiaSmiHasGpu { param([string]$Exe) $script:Probed++; return $false }+$HasNvidiaSmi = $false+$NvidiaSmiExe = $null+Invoke-Expression $fastLoop+Check "control: with fast probes every candidate is still tried" ($script:Probed -eq 10)++# ------------------------------------------- the wedge flag must describe the binary we chose+#+# Test-NvidiaSmiHasGpu records whether the binary it just probed timed out, and its own comment+# says what matters is "whether the binary it settled on answered". The two-pass loop broke that:+# it can keep probing after the candidate it will settle on, so the flag ends up describing a+# LATER binary. Downstream that flag decides whether to ask the selected nvidia-smi for the GPU+# name, the compute capability and the driver version, so a working binary would have all three+# skipped because a different copy elsewhere on the machine hung.+$script:NvidiaSmiWedged = $false+function Test-NvidiaSmiHasGpu {+ param([string]$Exe)+ # The first candidate answers and is the one the loop settles on. The second hangs.+ if ($Exe -like "*good*") { $script:NvidiaSmiWedged = $false; return $true }+ $script:NvidiaSmiWedged = $true+ return $false+}+function Invoke-NvidiaSmiBounded { param($Exe, $Arguments) return "no version here" }+function Get-NvidiaSmiCandidatePaths {+ return @("C:\good\nvidia-smi.exe", "C:\hung\nvidia-smi.exe")+}+$HasNvidiaSmi = $false+$NvidiaSmiExe = $null+Invoke-Expression $loopSrc+Check "the working candidate is the one selected" (+ $HasNvidiaSmi -eq $true -and $NvidiaSmiExe -eq "C:\good\nvidia-smi.exe")+Check "a later candidate hanging does not mark the selected binary wedged" (+ $script:NvidiaSmiWedged -eq $false)++# Bites control: when the binary the loop settles on is ITSELF the one that hung, the flag must+# stay set, or the guard downstream would query a hung nvidia-smi and wait out the bound again.+function Test-NvidiaSmiHasGpu {+ param([string]$Exe)+ $script:NvidiaSmiWedged = $true+ return $true+}+$script:NvidiaSmiWedged = $false+$HasNvidiaSmi = $false+$NvidiaSmiExe = $null+Invoke-Expression $loopSrc+Check "control: a selected binary that did hang is still reported wedged" (+ $script:NvidiaSmiWedged -eq $true)++# ------------------------------------- giving up must never cost a GPU that was there to find+#+# The regression the first version of the deadline introduced. One budget, and a break in the+# failure branch, meant a single slow FAILING probe ended the whole search: a machine whose+# System32 copy is broken and whose legacy NVSMI copy works lost its GPU entirely and went to+# CPU-only PyTorch, strictly worse than the stall the deadline was added to prevent.+#+# Two budgets now: the soft one only applies once there is already a usable answer, and only the+# hard one may end the search empty-handed.+$script:Probed = 0+function Test-NvidiaSmiHasGpu {+ param([string]$Exe)+ $script:Probed+++ if ($Exe -like "*broken*") { Start-Sleep -Milliseconds 2500; return $false } # slow AND failing+ return $true+}+function Invoke-NvidiaSmiBounded { param($Exe, $Arguments) return "CUDA Version: 12.8" }+function Get-NvidiaSmiCandidatePaths {+ return @("C:\broken\nvidia-smi.exe", "C:\legacy\nvidia-smi.exe")+}+$script:NvidiaSmiWedged = $false+$HasNvidiaSmi = $false+$NvidiaSmiExe = $null+Invoke-Expression $fastLoop+Check "a slow FAILING first probe does not end the search" ($script:Probed -ge 2)+Check "the working copy behind it is still found" (+ $HasNvidiaSmi -eq $true -and $NvidiaSmiExe -eq "C:\legacy\nvidia-smi.exe")++# The hard bound still exists: with nothing working anywhere, the loop stops rather than walking+# an unbounded list. This is the property the deadline was added for, and it must survive the fix.+$script:Probed = 0+function Test-NvidiaSmiHasGpu { param([string]$Exe) $script:Probed++; Start-Sleep -Milliseconds 700; return $false }+function Get-NvidiaSmiCandidatePaths { return @(1..20 | ForEach-Object { "C:\wedged$_\nvidia-smi.exe" }) }+$HasNvidiaSmi = $false+$NvidiaSmiExe = $null+$t0 = Get-Date+Invoke-Expression $fastLoop+$spent = ((Get-Date) - $t0).TotalSeconds+Check "with nothing working the hard bound still stops the walk" ($script:Probed -lt 20)+Check "and it spends the HARD budget, not the soft one, before giving up" ($spent -ge 2)+Check "control: giving up empty-handed really reports no GPU" ($HasNvidiaSmi -eq $false)++# The banner probe has to respect the deadline too.+#+# Checking only at the top of the loop is not enough: a candidate starting just inside the+# deadline can add nearly twice the per-probe timeout after it, since the listing probe may have+# spent most of its bound and the banner probe has a full one. The earlier elapsed-time check+# missed this because its banner stub returned instantly.+# Driven with a banner probe that is the slow one.+$script:Listings = 0+$script:Banners = 0+function Test-NvidiaSmiHasGpu { param([string]$Exe) $script:Listings++; Start-Sleep -Milliseconds 900; return $true }+function Invoke-NvidiaSmiBounded {+ param($Exe, $Arguments)+ $script:Banners+++ Start-Sleep -Milliseconds 900+ return "no version here"+}+function Get-NvidiaSmiCandidatePaths { return @(1..10 | ForEach-Object { "C:\slow$_\nvidia-smi.exe" }) }+$script:NvidiaSmiWedged = $false+$HasNvidiaSmi = $false+$NvidiaSmiExe = $null+$t0 = Get-Date+Invoke-Expression $fastLoop+$spent = ((Get-Date) - $t0).TotalSeconds+# Counted, not timed. "Fewer than 10 banners" and "under 8 seconds" BOTH held with the fix+# reverted, since at 1.8 s per candidate the top-of-loop check stops one candidate later either+# way. With the recheck exactly one banner runs; without it candidate 2 gets one too.+Check "a slow banner probe does not start after the deadline" ($script:Banners -eq 1)+Check "control: a banner probe did run for the candidate before it" ($script:Listings -ge 2)+# Loose sanity only; the counter above is the check that bites.+Check "the loop does not spend two probe bounds per candidate after expiry" ($spent -lt 8)+# And the GPU is still reported: everything here listed, so the fallback must take the first.+Check "control: the GPU found before the deadline is still reported" (+ $HasNvidiaSmi -eq $true -and $NvidiaSmiExe -eq "C:\slow1\nvidia-smi.exe")++# ------------------------------- the Windows-on-ARM picker must answer with a WORKING binary+#+# Get-WoaNvidiaSmiPath fed the first candidate that EXISTS to Test-WoaNvidiaPresent and+# Get-WoaDriverCudaVersion, both of which treat a non-answer as "no NVIDIA on this machine", so a+# broken copy declines the native ARM64 CUDA stack outright rather than costing a retry. A longer+# list makes that likely rather than unlikely.+$woaFn = Get-FunctionText $installPs1 "Get-WoaNvidiaSmiPath"+$script:WoaNvidiaSmiProbed = $false+$script:WoaNvidiaSmiPath = $null+$script:WoaProbes = @()+function Get-Command { param($Name, $ErrorAction) return $null } # nothing on PATH+function Invoke-NvidiaSmiBounded {+ param($Exe, $Arguments)+ $script:WoaProbes += $Exe+ if ($Exe -like "*working*") {+ $global:LASTEXITCODE = 0+ if ($Arguments -contains '-L') { return "GPU 0: NVIDIA GeForce RTX 4090" }+ return "CUDA Version: 12.8"+ }+ $global:LASTEXITCODE = 1+ return ""+}+function Get-NvidiaSmiCandidatePaths {+ return @("C:\stale\nvidia-smi.exe", "C:\working\nvidia-smi.exe")+}+Invoke-Expression $woaFn+$woaAnswer = Get-WoaNvidiaSmiPath+Check "the ARM64 picker skips a stale copy and returns one that answers" (+ $woaAnswer -eq "C:\working\nvidia-smi.exe")+Check "it really probed rather than pattern-matching the name" ($script:WoaProbes.Count -ge 2)++# Memoised: the two callers each probe, and walking the list twice costs a process spawn per+# entry. A second call must not re-probe.+$before = $script:WoaProbes.Count+$null = Get-WoaNvidiaSmiPath+Check "a second call is served from the memo" ($script:WoaProbes.Count -eq $before)++# And when NOTHING answers it still hands back the first that exists, which is what it did+# before this change, so a host where the probe cannot run is no worse off than it was.+$script:WoaNvidiaSmiProbed = $false+$script:WoaNvidiaSmiPath = $null+function Invoke-NvidiaSmiBounded { param($Exe, $Arguments) $global:LASTEXITCODE = 1; return "" }+Check "with nothing answering it falls back to the first existing copy" (+ (Get-WoaNvidiaSmiPath) -eq "C:\stale\nvidia-smi.exe")++# The ARM64 picker needs BOTH probes, not just the listing.+#+# A null banner from this same binary does not merely lose a version: it skips the CUDA-major+# guard in Initialize-WoaNativeCudaTorch that stops a native wheel newer than the installed driver+# being selected. So a listing-only candidate must not win over one that names a version.+$script:WoaNvidiaSmiProbed = $false+$script:WoaNvidiaSmiPath = $null+function Invoke-NvidiaSmiBounded {+ param($Exe, $Arguments)+ $global:LASTEXITCODE = 0+ if ($Arguments -contains '-L') { return "GPU 0: NVIDIA GeForce RTX 4090" }+ if ($Exe -like "*versioned*") { return "CUDA Version: 12.8" }+ return "NVIDIA-SMI has failed because it couldn't communicate with the driver"+}+function Get-NvidiaSmiCandidatePaths {+ return @("C:\listingonly\nvidia-smi.exe", "C:\versioned\nvidia-smi.exe")+}+Check "a candidate that also names a CUDA version wins over a listing-only one" (+ (Get-WoaNvidiaSmiPath) -eq "C:\versioned\nvidia-smi.exe")++# And when none names a version, the listing-only candidate is still used: the presence check+# must keep working, and only the version check declines.+$script:WoaNvidiaSmiProbed = $false+$script:WoaNvidiaSmiPath = $null+function Invoke-NvidiaSmiBounded {+ param($Exe, $Arguments)+ $global:LASTEXITCODE = 0+ if ($Arguments -contains '-L') { return "GPU 0: NVIDIA GeForce RTX 4090" }+ return "no version here"+}+Check "with no version anywhere the first listing candidate is still used" (+ (Get-WoaNvidiaSmiPath) -eq "C:\listingonly\nvidia-smi.exe")++# The scan is bounded as a whole. Without this a wedged driver plus eight retained DriverStore+# copies held Initialize-WoaNativeCudaTorch for minutes before the install started.+Check "the ARM64 scan carries a soft and a hard deadline" (+ $woaFn -match '\$woaDeadline = \(Get-Date\)\.AddSeconds\(30\)' -and+ $woaFn -match '\$woaHardDeadline = \(Get-Date\)\.AddSeconds\(60\)')++$fastWoa = ($woaFn -replace 'AddSeconds\(30\)', 'AddSeconds(2)') -replace 'AddSeconds\(60\)', 'AddSeconds(4)'+Check "the shortened ARM64 slice really carries both (bites)" (+ $fastWoa -match 'AddSeconds\(2\)' -and $fastWoa -match 'AddSeconds\(4\)')+Invoke-Expression $fastWoa+$script:WoaNvidiaSmiProbed = $false+$script:WoaNvidiaSmiPath = $null+$script:WoaSlowProbes = 0+function Invoke-NvidiaSmiBounded {+ param($Exe, $Arguments)+ $script:WoaSlowProbes+++ Start-Sleep -Milliseconds 700+ $global:LASTEXITCODE = 1+ return ""+}+function Get-NvidiaSmiCandidatePaths { return @(1..20 | ForEach-Object { "C:\wedged$_\nvidia-smi.exe" }) }+$null = Get-WoaNvidiaSmiPath+Check "a wedged ARM64 host stops scanning instead of walking every candidate" (+ $script:WoaSlowProbes -lt 20)+Check "control: it probed rather than declining immediately" ($script:WoaSlowProbes -ge 2)+Invoke-Expression $woaFn+Remove-Item Function:\Get-Command -ErrorAction SilentlyContinue++if ($failures -gt 0) {+ Write-Host "$failures check(s) failed" -ForegroundColor Red+ exit 1+}+Write-Host "all checks passed" -ForegroundColor Green