Skip to content

Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.

Look for nvidia-smi everywhere it actually is (#11119)

Daniel Han2f3cc594df39
diff --git a/.github/workflows/cross-platform-parity-ci.yml b/.github/workflows/cross-platform-parity-ci.yml
index b7c086a59aa..971fa05fd0c 100644
--- a/.github/workflows/cross-platform-parity-ci.yml
+++ b/.github/workflows/cross-platform-parity-ci.yml
@@ -97,6 +97,7 @@ on:
- 'tests/studio/test_amd_venv_repair_loop.ps1'
- 'tests/studio/test_amd_name_arch_r9700.ps1'
- 'tests/studio/test_nvidia_library_inventory.ps1'
+ - 'tests/studio/test_nvidia_smi_discovery.ps1'
- 'tests/studio/test_application_control_cli_fallback.ps1'
# test_application_control_cli_fallback.ps1 pins install.ps1's trampoline against
# process.rs's, so a change to either spelling has to run here.
@@ -176,6 +177,7 @@ on:
- 'tests/studio/test_amd_venv_repair_loop.ps1'
- 'tests/studio/test_amd_name_arch_r9700.ps1'
- 'tests/studio/test_nvidia_library_inventory.ps1'
+ - 'tests/studio/test_nvidia_smi_discovery.ps1'
- 'tests/studio/test_application_control_cli_fallback.ps1'
# test_application_control_cli_fallback.ps1 pins install.ps1's trampoline against
# process.rs's, so a change to either spelling has to run here.
@@ -308,6 +310,15 @@ jobs:
if: runner.os == 'Windows'
shell: powershell
run: powershell -NoProfile -File tests\studio\test_nvidia_library_inventory.ps1
+ # Both legs, and both matter: the candidate list is what decides whether a real GPU is
+ # seen at all, and 5.1 is the host the Desktop app launches the installer in.
+ - name: nvidia-smi discovery tests
+ shell: pwsh
+ run: pwsh -NoProfile -File tests/studio/test_nvidia_smi_discovery.ps1
+ - name: nvidia-smi discovery tests (Windows PowerShell 5.1)
+ if: runner.os == 'Windows'
+ shell: powershell
+ run: powershell -NoProfile -File tests\studio\test_nvidia_smi_discovery.ps1
# AST-extracted helpers plus a real .cmd written to a temp dir, so no venv and no
# Windows needed. The 1260 classification is exercised against crafted exceptions:
# a hosted runner has no Application Control policy to be denied by. The enforced
diff --git a/install.ps1 b/install.ps1
index 3f9889e9b15..0a236892a65 100644
--- a/install.ps1
+++ b/install.ps1
@@ -1134,18 +1134,149 @@ function Install-UnslothStudio {
}
}
+ # ── BEGIN SHARED WITH studio/setup.ps1 (Get-NvidiaSmiCandidatePaths) ──
+ # Every directory nvidia-smi might be in, in the order worth trying, filtered to the ones that
+ # actually hold the binary. Callers still run their own Test-NvidiaSmiHasGpu over the result:
+ # existing is not the same as answering.
+ #
+ # Only two locations were searched before, and each addition below is a host where a real NVIDIA
+ # GPU was reported absent, which sends the installer to CPU-only PyTorch:
+ #
+ # SysNative From a 32-bit process "System32" is redirected to SysWOW64, which has no
+ # nvidia-smi. SysNative is the alias that reaches the real System32.
+ # ProgramW64 In that same process $env:ProgramFiles is "Program Files (x86)".
+ # ProgramW6432 is the 64-bit Program Files whatever the process bitness.
+ # x86 Not the driver's own location, but a machine that once had a 32-bit
+ # toolkit can carry a working copy there.
+ # DriverStore Where the driver package itself lives. Present on hosts where the copy
+ # into System32 did not happen, which is the #9255 population.
+ #
+ # Nothing is removed. One ordering does change, and deliberately: the two call sites disagreed
+ # with each other before, one trying System32 first and the other the NVSMI directory first, and
+ # a single list cannot keep both. System32 wins, because that is where the current driver puts
+ # its copy; NVSMI is the legacy location and a machine carrying both can have an older binary
+ # there reporting an older CUDA version. This only affects a host that has both, and only in
+ # which of two working binaries answers.
+ function Get-NvidiaSmiCandidatePaths {
+ $dirs = @()
+ # Current driver locations first, in BOTH process bitnesses, before any legacy one.
+ # Both bitness spellings before $env:ProgramFiles, which in a 32-bit process is the x86 tree
+ # where a stale toolkit copy can sit and answer first.
+ if ($env:SystemRoot) { $dirs += (Join-Path $env:SystemRoot "System32") }
+ if ($env:SystemRoot) { $dirs += (Join-Path $env:SystemRoot "SysNative") }
+ if ($env:ProgramW6432) { $dirs += (Join-Path $env:ProgramW6432 "NVIDIA Corporation\NVSMI") }
+ if ($env:ProgramFiles) { $dirs += (Join-Path $env:ProgramFiles "NVIDIA Corporation\NVSMI") }
+ $pfx86 = ${env:ProgramFiles(x86)}
+ if ($pfx86) { $dirs += (Join-Path $pfx86 "NVIDIA Corporation\NVSMI") }
+ # Bounded on purpose. FileRepository holds every driver package the machine has ever had.
+ #
+ # The bound is applied to directories that ACTUALLY HOLD the binary, not to the nv* matches.
+ # NVIDIA ships several packages whose names start nv (HD audio, the virtual audio device,
+ # the network service), so a host with eight of those newer than the display driver would
+ # otherwise spend the whole allowance on directories with no nvidia-smi in them and report
+ # the machine as having none. Test-Path is a file existence check, not a process spawn; the
+ # bound that matters is on the candidates handed back, since each of those costs a probe.
+ try {
+ $repos = @()
+ if ($env:SystemRoot) {
+ # Both spellings, same WOW64 reason as above. At most one resolves in any given process, so
+ # this is not a doubled scan.
+ $repos += (Join-Path $env:SystemRoot "System32\DriverStore\FileRepository")
+ $repos += (Join-Path $env:SystemRoot "SysNative\DriverStore\FileRepository")
+ }
+ $packages = @()
+ foreach ($repo in $repos) {
+ if (-not (Test-Path -LiteralPath $repo -PathType Container)) { continue }
+ $packages += @(Get-ChildItem -LiteralPath $repo -Directory -Filter "nv*" -ErrorAction SilentlyContinue |
+ Where-Object { Test-Path -LiteralPath (Join-Path $_.FullName "nvidia-smi.exe") -PathType Leaf })
+ }
+ # The bound is on the whole DriverStore contribution rather than per root, so adding
+ # the second spelling cannot double the number of probes this hands back.
+ foreach ($dir in @($packages | Sort-Object LastWriteTime -Descending | Select-Object -First 8)) {
+ $dirs += $dir.FullName
+ }
+ } catch {}
+ $paths = @()
+ $seen = @{}
+ foreach ($dir in $dirs) {
+ if ([string]::IsNullOrWhiteSpace($dir)) { continue }
+ $candidate = Join-Path $dir "nvidia-smi.exe"
+ # Case-insensitive, because the same directory reached two ways must not be probed twice:
+ # each probe is a bounded process spawn with a wall-clock timeout behind it.
+ $key = $candidate.ToLowerInvariant()
+ if ($seen.ContainsKey($key)) { continue }
+ $seen[$key] = $true
+ if (Test-Path -LiteralPath $candidate -PathType Leaf) { $paths += $candidate }
+ }
+ return $paths
+ }
+ # ── END SHARED WITH studio/setup.ps1 (Get-NvidiaSmiCandidatePaths) ──
+
# One list, because the presence probe and the driver-version probe must agree: a host only the first finds reports a GPU with no driver version, and the CUDA-major guard is then skipped.
+ # Reset per run: under `irm | iex` the script scope IS the caller's session, so a second
+ # invocation in one shell would otherwise reuse the first run's answer.
+ $script:WoaNvidiaSmiProbed = $false
+ $script:WoaNvidiaSmiPath = $null
+
function Get-WoaNvidiaSmiPath {
+ # Memoised because the two callers each probe, and without this the whole candidate list
+ # would be walked twice with a bounded process spawn per entry.
+ if ($script:WoaNvidiaSmiProbed) { return $script:WoaNvidiaSmiPath }
+ $script:WoaNvidiaSmiProbed = $true
$exe = $null
try { $exe = (Get-Command nvidia-smi -ErrorAction SilentlyContinue).Source } catch { $exe = $null }
- if ($exe) { return $exe }
- foreach ($candidate in @(
- "$env:SystemRoot\System32\nvidia-smi.exe",
- "$env:ProgramFiles\NVIDIA Corporation\NVSMI\nvidia-smi.exe"
- )) {
- if (Test-Path -LiteralPath $candidate -PathType Leaf) { return $candidate }
+ if ($exe) { $script:WoaNvidiaSmiPath = $exe; return $exe }
+ $found = @(Get-NvidiaSmiCandidatePaths)
+ if ($found.Count -eq 0) { return $null }
+ # The first candidate that ANSWERS, not the first that exists.
+ #
+ # Both callers immediately probe what this returns, and treat a non-answer as "no NVIDIA
+ # here": Test-WoaNvidiaPresent returns false and Get-WoaDriverCudaVersion returns null,
+ # which between them decide whether the native ARM64 CUDA stack is installed at all. So
+ # handing back a broken copy does not cost a retry, it declines the whole route.
+ #
+ # That was survivable when two locations were searched and is not now that the list is
+ # longer: every location added is another chance that the first entry is a stale copy
+ # sitting in front of a working one. Same reason the main detection loop stopped taking
+ # the first listing candidate.
+ # Same two-budget shape as the main detection loop, and for the same reasons. Each probe
+ # below is a bounded process spawn, and the list can now be eleven entries, so a wedged
+ # driver would otherwise hold Initialize-WoaNativeCudaTorch for minutes before the
+ # install even starts. The soft budget only applies once something has answered; only the
+ # hard one may end the scan with nothing, because that decides the whole ARM64 route.
+ $woaDeadline = (Get-Date).AddSeconds(30)
+ $woaHardDeadline = (Get-Date).AddSeconds(60)
+ $woaListing = $null
+ foreach ($p in $found) {
+ if ($null -ne $woaListing -and (Get-Date) -gt $woaDeadline) { break }
+ if ((Get-Date) -gt $woaHardDeadline) { break }
+ try {
+ $listing = Invoke-NvidiaSmiBounded $p @('-L')
+ if (-not ($LASTEXITCODE -eq 0 -and $listing -match '(?m)^\s*GPU\s+\d+')) { continue }
+ if (-not $woaListing) { $woaListing = $p }
+ # Two passes, because listing a GPU is not enough here. Get-WoaDriverCudaVersion
+ # asks this same binary for the CUDA banner, and a null answer does not merely
+ # lose a version: it skips the CUDA-major compatibility guard in
+ # Initialize-WoaNativeCudaTorch, which is what stops a native wheel newer than
+ # the installed driver being chosen. So prefer a candidate that answers BOTH.
+ if ((Get-Date) -gt $woaDeadline) { break }
+ $banner = Invoke-NvidiaSmiBounded $p
+ if ($banner -match 'CUDA(?: UMD)? Version:\s+\d+\.\d+') {
+ $script:WoaNvidiaSmiPath = $p
+ return $p
+ }
+ } catch {}
}
- return $null
+ # One that listed a GPU but never named a version is still better than one that did not
+ # answer at all: the callers' presence check works, and the version check declines.
+ if ($woaListing) {
+ $script:WoaNvidiaSmiPath = $woaListing
+ return $woaListing
+ }
+ # Nothing answered. Hand back the first that exists, which is what this did before, so a
+ # host where the probe cannot run is no worse off than it was.
+ $script:WoaNvidiaSmiPath = $found[0]
+ return $found[0]
}
function Test-WoaNvidiaPresent {
@@ -7521,15 +7652,79 @@ exit 0
}
} catch {}
if (-not $HasNvidiaSmi) {
- foreach ($p in @(
- "$env:ProgramFiles\NVIDIA Corporation\NVSMI\nvidia-smi.exe",
- "$env:SystemRoot\System32\nvidia-smi.exe"
- )) {
- if (Test-Path $p) {
- try {
- if (Test-NvidiaSmiHasGpu $p) { $HasNvidiaSmi = $true; $NvidiaSmiExe = $p; break }
- } catch {}
- }
+ # Two passes, not one, and the reason is the ordering this change introduces. Listing a
+ # GPU is not the same as being able to report a CUDA version: a partially installed or
+ # stale utility can answer -L and still print a banner Get-TorchIndexUrl cannot parse,
+ # and that path falls back to cu126. Taking the first binary that merely lists a GPU
+ # could therefore hand a cu128-capable host cu126 purely because of which directory is
+ # searched first. So prefer a candidate that answers BOTH questions, and only settle for
+ # one that just lists a GPU if none does.
+ #
+ # One deadline across the whole loop, not just the per-call one. Each candidate gets its own
+ # bounded runner, and on a host with a wedged driver every one of them waits out that bound
+ # before failing. The list is longer than it was, so what used to be two stalls could now be
+ # ten, and all of it is spent in front of the driver-library fallback that would have answered
+ # in milliseconds. The bound is on TIME rather than on the number of candidates because the
+ # cost being controlled is time: a machine where every probe answers at once still gets to
+ # try them all, and one where they hang stops early.
+ #
+ # Get-Date rather than a Stopwatch: Constrained Language Mode refuses the method calls, and
+ # this runs on hosts that enforce it.
+ # Enumerated BEFORE the clock starts. Discovery is Test-Path calls, but on a machine with a
+ # filesystem filter driver or slow storage it is not free, and charging it to the probe
+ # budget is how a slow disk turns into no GPU.
+ $smiCandidates = @(Get-NvidiaSmiCandidatePaths)
+ # Two budgets, because "stop early" means two different things here.
+ #
+ # The soft one applies ONLY once a usable answer is already in hand: there is a working
+ # nvidia-smi, and continuing just looks for a better CUDA banner. Giving that up costs at
+ # most a wheel family.
+ #
+ # The hard one is the only thing allowed to end the search with NOTHING found, because that
+ # outcome is CPU-only PyTorch on a machine with a GPU. The first version of this had a single
+ # budget and broke out of the loop after one slow failure, so a host whose System32 copy is
+ # broken and whose legacy NVSMI copy works lost its GPU entirely. That is strictly worse than
+ # the stall it was added to prevent.
+ $probeDeadline = (Get-Date).AddSeconds(30)
+ $probeHardDeadline = (Get-Date).AddSeconds(60)
+ $firstListing = $null
+ # Captured beside the path, because $script:NvidiaSmiWedged describes the LAST binary probed
+ # and the one this loop settles on can be an earlier one. Captured rather than assumed to be
+ # false: it is false today because a candidate only becomes $firstListing when its -L probe
+ # succeeded, and that is a property of Test-NvidiaSmiHasGpu rather than of this loop.
+ $firstListingWedged = $false
+ foreach ($p in $smiCandidates) {
+ # Only give up early when there is already something to fall back on.
+ if ($null -ne $firstListing -and (Get-Date) -gt $probeDeadline) { break }
+ # With nothing found, keep going to the hard bound. Reporting no GPU is the expensive
+ # answer, so it has to be the one that costs the most before it is reached.
+ if ((Get-Date) -gt $probeHardDeadline) { break }
+ try {
+ if (-not (Test-NvidiaSmiHasGpu $p)) { continue }
+ if (-not $firstListing) {
+ $firstListing = $p
+ $firstListingWedged = $script:NvidiaSmiWedged
+ }
+ # Rechecked here, not only at the top. The listing probe that just returned can have
+ # spent most of its own bound, and the banner probe below has a full bound of its
+ # own, so a candidate starting just inside the deadline could add nearly twice the
+ # per-probe timeout after it. There is already a usable answer in $firstListing at
+ # this point, so stopping costs at most a wheel family.
+ if ((Get-Date) -gt $probeDeadline) { break }
+ $banner = Invoke-NvidiaSmiBounded $p
+ if ($banner -match 'CUDA(?: UMD)? Version:\s+\d+\.\d+') {
+ $HasNvidiaSmi = $true; $NvidiaSmiExe = $p; break
+ }
+ } catch {}
+ }
+ if (-not $HasNvidiaSmi -and $firstListing) {
+ $HasNvidiaSmi = $true; $NvidiaSmiExe = $firstListing
+ # Restored for the binary actually selected. A later candidate that timed out leaves the
+ # flag set for ITS path, and the guard downstream reads the flag to decide whether to ask
+ # the selected binary for the name, the compute capability and the driver version. Left
+ # alone, a working nvidia-smi would be treated as wedged and all three queries skipped
+ # because a different binary elsewhere on the machine hung.
+ $script:NvidiaSmiWedged = $firstListingWedged
}
}
if (-not $HasNvidiaSmi -and (Get-NvidiaLibraryInventory)) {
diff --git a/studio/setup.ps1 b/studio/setup.ps1
index 1f77756722e..7c90a7ec987 100644
--- a/studio/setup.ps1
+++ b/studio/setup.ps1
@@ -2647,6 +2647,84 @@ function Invoke-NvidiaSmiBounded {
}
}
+# ── BEGIN SHARED WITH install.ps1 (Get-NvidiaSmiCandidatePaths) ──
+# Every directory nvidia-smi might be in, in the order worth trying, filtered to the ones that
+# actually hold the binary. Callers still run their own Test-NvidiaSmiHasGpu over the result:
+# existing is not the same as answering.
+#
+# Only two locations were searched before, and each addition below is a host where a real NVIDIA
+# GPU was reported absent, which sends the installer to CPU-only PyTorch:
+#
+# SysNative From a 32-bit process "System32" is redirected to SysWOW64, which has no
+# nvidia-smi. SysNative is the alias that reaches the real System32.
+# ProgramW64 In that same process $env:ProgramFiles is "Program Files (x86)".
+# ProgramW6432 is the 64-bit Program Files whatever the process bitness.
+# x86 Not the driver's own location, but a machine that once had a 32-bit
+# toolkit can carry a working copy there.
+# DriverStore Where the driver package itself lives. Present on hosts where the copy
+# into System32 did not happen, which is the #9255 population.
+#
+# Nothing is removed. One ordering does change, and deliberately: the two call sites disagreed
+# with each other before, one trying System32 first and the other the NVSMI directory first, and
+# a single list cannot keep both. System32 wins, because that is where the current driver puts
+# its copy; NVSMI is the legacy location and a machine carrying both can have an older binary
+# there reporting an older CUDA version. This only affects a host that has both, and only in
+# which of two working binaries answers.
+function Get-NvidiaSmiCandidatePaths {
+ $dirs = @()
+ # Current driver locations first, in BOTH process bitnesses, before any legacy one.
+ # Both bitness spellings before $env:ProgramFiles, which in a 32-bit process is the x86 tree
+ # where a stale toolkit copy can sit and answer first.
+ if ($env:SystemRoot) { $dirs += (Join-Path $env:SystemRoot "System32") }
+ if ($env:SystemRoot) { $dirs += (Join-Path $env:SystemRoot "SysNative") }
+ if ($env:ProgramW6432) { $dirs += (Join-Path $env:ProgramW6432 "NVIDIA Corporation\NVSMI") }
+ if ($env:ProgramFiles) { $dirs += (Join-Path $env:ProgramFiles "NVIDIA Corporation\NVSMI") }
+ $pfx86 = ${env:ProgramFiles(x86)}
+ if ($pfx86) { $dirs += (Join-Path $pfx86 "NVIDIA Corporation\NVSMI") }
+ # Bounded on purpose. FileRepository holds every driver package the machine has ever had.
+ #
+ # The bound is applied to directories that ACTUALLY HOLD the binary, not to the nv* matches.
+ # NVIDIA ships several packages whose names start nv (HD audio, the virtual audio device,
+ # the network service), so a host with eight of those newer than the display driver would
+ # otherwise spend the whole allowance on directories with no nvidia-smi in them and report
+ # the machine as having none. Test-Path is a file existence check, not a process spawn; the
+ # bound that matters is on the candidates handed back, since each of those costs a probe.
+ try {
+ $repos = @()
+ if ($env:SystemRoot) {
+ # Both spellings, same WOW64 reason as above. At most one resolves in any given process, so
+ # this is not a doubled scan.
+ $repos += (Join-Path $env:SystemRoot "System32\DriverStore\FileRepository")
+ $repos += (Join-Path $env:SystemRoot "SysNative\DriverStore\FileRepository")
+ }
+ $packages = @()
+ foreach ($repo in $repos) {
+ if (-not (Test-Path -LiteralPath $repo -PathType Container)) { continue }
+ $packages += @(Get-ChildItem -LiteralPath $repo -Directory -Filter "nv*" -ErrorAction SilentlyContinue |
+ Where-Object { Test-Path -LiteralPath (Join-Path $_.FullName "nvidia-smi.exe") -PathType Leaf })
+ }
+ # The bound is on the whole DriverStore contribution rather than per root, so adding
+ # the second spelling cannot double the number of probes this hands back.
+ foreach ($dir in @($packages | Sort-Object LastWriteTime -Descending | Select-Object -First 8)) {
+ $dirs += $dir.FullName
+ }
+ } catch {}
+ $paths = @()
+ $seen = @{}
+ foreach ($dir in $dirs) {
+ if ([string]::IsNullOrWhiteSpace($dir)) { continue }
+ $candidate = Join-Path $dir "nvidia-smi.exe"
+ # Case-insensitive, because the same directory reached two ways must not be probed twice:
+ # each probe is a bounded process spawn with a wall-clock timeout behind it.
+ $key = $candidate.ToLowerInvariant()
+ if ($seen.ContainsKey($key)) { continue }
+ $seen[$key] = $true
+ if (Test-Path -LiteralPath $candidate -PathType Leaf) { $paths += $candidate }
+ }
+ return $paths
+}
+# ── END SHARED WITH install.ps1 (Get-NvidiaSmiCandidatePaths) ──
+
# A driverless nvidia-smi exits 0 listing no GPU, so require a "GPU <n>:" row.
# 124 is what Invoke-NvidiaSmiBounded reports when it had to kill the probe. Recorded so
# the banner below can skip a second query: detection already waited out the full bound on
@@ -2672,21 +2750,83 @@ try {
}
} catch {}
if (-not $HasNvidiaSmi) {
- $nvSmiDefaults = @(
- "$env:ProgramFiles\NVIDIA Corporation\NVSMI\nvidia-smi.exe",
- "$env:SystemRoot\System32\nvidia-smi.exe"
- )
- foreach ($p in $nvSmiDefaults) {
- if (Test-Path $p) {
- try {
- if (Test-NvidiaSmiHasGpu $p) {
- $HasNvidiaSmi = $true
- $NvidiaSmiExe = $p
- Write-StudioLine " Found nvidia-smi at $(Split-Path $p -Parent)" -ForegroundColor Gray
- break
- }
- } catch {}
- }
+ # Two passes, not one, and the reason is the ordering this change introduces. Listing a GPU is
+ # not the same as being able to report a CUDA version: a partially installed or stale utility
+ # can answer -L and still print a banner the version ladder cannot parse. Taking the first
+ # binary that merely lists a GPU could hand a newer-capable host an older CUDA family purely
+ # because of which directory is searched first. Prefer a candidate that answers BOTH, and
+ # settle for one that just lists a GPU only if none does.
+ #
+ # One deadline across the whole loop, not just the per-call one. Each candidate gets its own
+ # bounded runner, and on a host with a wedged driver every one of them waits out that bound
+ # before failing. The list is longer than it was, so what used to be two stalls could now be
+ # ten, and all of it is spent in front of the driver-library fallback that would have answered
+ # in milliseconds. The bound is on TIME rather than on the number of candidates because the
+ # cost being controlled is time: a machine where every probe answers at once still gets to
+ # try them all, and one where they hang stops early.
+ #
+ # Get-Date rather than a Stopwatch: Constrained Language Mode refuses the method calls, and
+ # this runs on hosts that enforce it.
+ # Enumerated BEFORE the clock starts. Discovery is Test-Path calls, but on a machine with a
+ # filesystem filter driver or slow storage it is not free, and charging it to the probe
+ # budget is how a slow disk turns into no GPU.
+ $smiCandidates = @(Get-NvidiaSmiCandidatePaths)
+ # Two budgets, because "stop early" means two different things here.
+ #
+ # The soft one applies ONLY once a usable answer is already in hand: there is a working
+ # nvidia-smi, and continuing just looks for a better CUDA banner. Giving that up costs at
+ # most a wheel family.
+ #
+ # The hard one is the only thing allowed to end the search with NOTHING found, because that
+ # outcome is CPU-only PyTorch on a machine with a GPU. The first version of this had a single
+ # budget and broke out of the loop after one slow failure, so a host whose System32 copy is
+ # broken and whose legacy NVSMI copy works lost its GPU entirely. That is strictly worse than
+ # the stall it was added to prevent.
+ $probeDeadline = (Get-Date).AddSeconds(30)
+ $probeHardDeadline = (Get-Date).AddSeconds(60)
+ $firstListing = $null
+ # Captured beside the path, because $script:NvidiaSmiWedged describes the LAST binary probed
+ # and the one this loop settles on can be an earlier one. Captured rather than assumed to be
+ # false: it is false today because a candidate only becomes $firstListing when its -L probe
+ # succeeded, and that is a property of Test-NvidiaSmiHasGpu rather than of this loop.
+ $firstListingWedged = $false
+ foreach ($p in $smiCandidates) {
+ # Only give up early when there is already something to fall back on.
+ if ($null -ne $firstListing -and (Get-Date) -gt $probeDeadline) { break }
+ # With nothing found, keep going to the hard bound. Reporting no GPU is the expensive
+ # answer, so it has to be the one that costs the most before it is reached.
+ if ((Get-Date) -gt $probeHardDeadline) { break }
+ try {
+ if (-not (Test-NvidiaSmiHasGpu $p)) { continue }
+ if (-not $firstListing) {
+ $firstListing = $p
+ $firstListingWedged = $script:NvidiaSmiWedged
+ }
+ # Rechecked here, not only at the top. The listing probe that just returned can have
+ # spent most of its own bound, and the banner probe below has a full bound of its
+ # own, so a candidate starting just inside the deadline could add nearly twice the
+ # per-probe timeout after it. There is already a usable answer in $firstListing at
+ # this point, so stopping costs at most a wheel family.
+ if ((Get-Date) -gt $probeDeadline) { break }
+ $banner = Invoke-NvidiaSmiBounded $p
+ if ($banner -match 'CUDA(?: UMD)? Version:\s+\d+\.\d+') {
+ $HasNvidiaSmi = $true
+ $NvidiaSmiExe = $p
+ Write-StudioLine " Found nvidia-smi at $(Split-Path $p -Parent)" -ForegroundColor Gray
+ break
+ }
+ } catch {}
+ }
+ if (-not $HasNvidiaSmi -and $firstListing) {
+ $HasNvidiaSmi = $true
+ $NvidiaSmiExe = $firstListing
+ # Restored for the binary actually selected. A later candidate that timed out leaves the
+ # flag set for ITS path, and the guard downstream reads the flag to decide whether to ask
+ # the selected binary for the name, the compute capability and the driver version. Left
+ # alone, a working nvidia-smi would be treated as wedged and all three queries skipped
+ # because a different binary elsewhere on the machine hung.
+ $script:NvidiaSmiWedged = $firstListingWedged
+ Write-StudioLine " Found nvidia-smi at $(Split-Path $firstListing -Parent)" -ForegroundColor Gray
}
}
if (-not $HasNvidiaSmi -and (Get-NvidiaLibraryInventory)) {
diff --git a/tests/studio/install/test_woa_torch_index_persistence.py b/tests/studio/install/test_woa_torch_index_persistence.py
index 7f600010ae7..dbbaa73ecb9 100644
--- a/tests/studio/install/test_woa_torch_index_persistence.py
+++ b/tests/studio/install/test_woa_torch_index_persistence.py
@@ -4264,6 +4264,7 @@ class TestTheEarlyNvidiaProbesAreBounded:
_function_source(INSTALL_SRC, "Invoke-NvidiaSmiBounded"),
# The probe resolves its executable through this; without it the call is
# unresolved and the probe answers False for a reason the test is not about.
+ _function_source(INSTALL_SRC, "Get-NvidiaSmiCandidatePaths"),
_function_source(INSTALL_SRC, "Get-WoaNvidiaSmiPath"),
_function_source(INSTALL_SRC, "Test-WoaNvidiaPresent"),
f"$env:PATH = '{tmp_path}' + [System.IO.Path]::PathSeparator + $env:PATH",
@@ -4283,6 +4284,7 @@ class TestTheEarlyNvidiaProbesAreBounded:
# Required, and easy to miss: an unresolved lookup makes Get-WoaDriverCudaVersion
# return $null before it ever calls nvidia-smi, so the "[]" below would pass
# without the timeout this test exists to bound ever being exercised.
+ _function_source(INSTALL_SRC, "Get-NvidiaSmiCandidatePaths"),
_function_source(INSTALL_SRC, "Get-WoaNvidiaSmiPath"),
_function_source(INSTALL_SRC, "Get-WoaDriverCudaVersion"),
f"$env:PATH = '{tmp_path}' + [System.IO.Path]::PathSeparator + $env:PATH",
@@ -5736,8 +5738,22 @@ class TestBothNvidiaSmiProbesSearchTheSameLocations:
def test_the_shared_helper_lists_every_supported_location(self):
body = _function_source(INSTALL_SRC, "Get-WoaNvidiaSmiPath")
assert "Get-Command nvidia-smi" in body
+ # The candidate list moved into Get-NvidiaSmiCandidatePaths, which install.ps1 and
+ # studio/setup.ps1 share, so the WoA probe and the ordinary Windows probe cannot drift
+ # apart either. What this class is about is unchanged: one list, and both probes on it.
+ assert "Get-NvidiaSmiCandidatePaths" in body, "the WoA probe no longer uses the shared list"
+ candidates = _function_source(INSTALL_SRC, "Get-NvidiaSmiCandidatePaths")
+ assert '"nvidia-smi.exe"' in candidates
+ # The list builds each directory with Join-Path, so a whole path literal never appears in
+ # it. Check the parts, which is what it actually promises. Behaviour, including the
+ # locations added beyond these two, is driven for real against a planted filesystem in
+ # tests/studio/test_nvidia_smi_discovery.ps1.
for location in self.LOCATIONS:
- assert location in body, location
+ directory, _, leaf = location.rpartition("\\")
+ assert leaf == "nvidia-smi.exe", location
+ root, _, rest = directory.partition("\\")
+ assert root in candidates, location
+ assert f'"{rest}"' in candidates, location
@pytest.mark.parametrize("name", ("Test-WoaNvidiaPresent", "Get-WoaDriverCudaVersion"))
def test_neither_probe_keeps_its_own_candidate_list(self, name):
@@ -5765,16 +5781,25 @@ class TestBothNvidiaSmiProbesSearchTheSameLocations:
if "Get-WoaNvidiaSmiPath" in _function_source(INSTALL_SRC, name)
]
assert callers, "neither probe routes through the shared lookup any more"
+ # Follow the chain rather than naming one helper: Get-WoaNvidiaSmiPath now calls
+ # Get-NvidiaSmiCandidatePaths, and a composition missing the second one fails in exactly
+ # the silent way this test exists to catch.
+ needed = ["Get-WoaNvidiaSmiPath"]
+ for helper in needed:
+ for callee in ("Get-NvidiaSmiCandidatePaths",):
+ if callee in _function_source(INSTALL_SRC, helper) and callee not in needed:
+ needed.append(callee)
own = pathlib.Path(__file__).read_text(encoding = "utf-8")
for name in callers:
for match in re.finditer(rf'_function_source\(INSTALL_SRC, "{re.escape(name)}"\)', own):
# The _script(...) call this appears in, back to its opening paren.
start = own.rindex("_script(", 0, match.start())
block = own[start : own.index("\n )", match.end())]
- assert '_function_source(INSTALL_SRC, "Get-WoaNvidiaSmiPath")' in block, (
- f"a composed script injects {name}, which calls Get-WoaNvidiaSmiPath, "
- "without injecting that helper"
- )
+ for helper in needed:
+ assert f'_function_source(INSTALL_SRC, "{helper}")' in block, (
+ f"a composed script injects {name}, which reaches {helper}, "
+ "without injecting that helper"
+ )
def test_the_guard_this_protects_is_still_there(self):
# If the CUDA-major check ever goes away, this whole class is pointless; say so loudly.
diff --git a/tests/studio/test_nvidia_smi_discovery.ps1 b/tests/studio/test_nvidia_smi_discovery.ps1
new file mode 100644
index 00000000000..3b258ef95dc
--- /dev/null
+++ b/tests/studio/test_nvidia_smi_discovery.ps1
@@ -0,0 +1,525 @@
+#!/usr/bin/env pwsh
+# SPDX-License-Identifier: AGPL-3.0-only
+# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
+# Where the installer looks for nvidia-smi.
+#
+# $HasNvidiaSmi does not mean "nvidia-smi exists". It means "an NVIDIA GPU is present", and the
+# whole wheel-index decision hangs off it: a host where nvidia-smi is not found gets CPU-only
+# PyTorch with no warning. Two directories were searched, and real hosts have a real GPU in
+# neither of them (#9255).
+#
+# Driven against a fake filesystem rather than asserted by reading the source. Each check plants
+# exactly one copy, so a location that stops being searched fails on its own row.
+# Run: pwsh -NoProfile -File tests/studio/test_nvidia_smi_discovery.ps1
+
+$ErrorActionPreference = "Stop"
+$root = (Resolve-Path ([System.IO.Path]::Combine($PSScriptRoot, "..", ".."))).Path
+$installPs1 = Join-Path $root "install.ps1"
+$setupPs1 = Join-Path $root "studio/setup.ps1"
+
+$failures = 0
+function Check($name, $cond) {
+ if ($cond) { Write-Host " PASS $name" }
+ else { Write-Host " FAIL $name" -ForegroundColor Red; $script:failures++ }
+}
+
+function Get-FunctionText($file, $name) {
+ $errors = $null
+ $ast = [System.Management.Automation.Language.Parser]::ParseFile($file, [ref]$null, [ref]$errors)
+ if ($errors) { $errors | ForEach-Object { $_.ToString() }; throw "$file has parse errors" }
+ $fn = $ast.FindAll({ param($n)
+ $n -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $n.Name -eq $name
+ }, $true)
+ if ($fn.Count -lt 1) { throw "expected $name in $file, found none" }
+ return $fn[0].Extent.Text
+}
+
+# The two copies have to stay the same helper. install.ps1 nests one level deeper, so compare
+# without indentation, the same way test_nvidia_library_inventory.ps1 does.
+$installText = Get-FunctionText $installPs1 "Get-NvidiaSmiCandidatePaths"
+$setupText = Get-FunctionText $setupPs1 "Get-NvidiaSmiCandidatePaths"
+$strip = { param($t) (($t -split "`n") | ForEach-Object { $_.TrimStart() }) -join "`n" }
+Check "install.ps1 and setup.ps1 carry the same Get-NvidiaSmiCandidatePaths" (
+ (& $strip $installText) -eq (& $strip $setupText))
+
+Invoke-Expression $setupText
+
+$tmp = Join-Path ([System.IO.Path]::GetTempPath()) ("nvsmi-" + [guid]::NewGuid().ToString("N"))
+$saved = @{
+ SystemRoot = $env:SystemRoot
+ ProgramFiles = $env:ProgramFiles
+ ProgramW6432 = $env:ProgramW6432
+ ProgramFilesX86 = ${env:ProgramFiles(x86)}
+}
+
+function Reset-FakeMachine {
+ if (Test-Path -LiteralPath $tmp) { Remove-Item -LiteralPath $tmp -Recurse -Force }
+ $env:SystemRoot = Join-Path $tmp "Windows"
+ $env:ProgramFiles = Join-Path $tmp "Program Files"
+ $env:ProgramW6432 = Join-Path $tmp "Program Files"
+ ${env:ProgramFiles(x86)} = Join-Path $tmp "Program Files (x86)"
+}
+
+function Add-FakeSmi($relative) {
+ $dir = Join-Path $tmp $relative
+ New-Item -ItemType Directory -Force -Path $dir | Out-Null
+ $exe = Join-Path $dir "nvidia-smi.exe"
+ [System.IO.File]::WriteAllText($exe, "")
+ return $exe
+}
+
+try {
+ # Each location on its own. A single planted copy has to be the single answer, so a rung
+ # dropped from the list fails here and nowhere else.
+ foreach ($case in @(
+ @{ Name = "System32, where a current driver puts it"; Dir = "Windows/System32" },
+ @{ Name = "the legacy NVSMI directory"; Dir = "Program Files/NVIDIA Corporation/NVSMI" },
+ @{ Name = "SysNative, which is System32 seen from a 32-bit process"; Dir = "Windows/SysNative" },
+ @{ Name = "Program Files (x86), left by an old 32-bit toolkit"; Dir = "Program Files (x86)/NVIDIA Corporation/NVSMI" },
+ @{ Name = "the DriverStore package the driver ships in"; Dir = "Windows/System32/DriverStore/FileRepository/nvmii.inf_amd64_1234/" }
+ )) {
+ Reset-FakeMachine
+ $planted = Add-FakeSmi $case.Dir
+ $found = @(Get-NvidiaSmiCandidatePaths)
+ Check "found in $($case.Name)" ($found.Count -eq 1 -and $found[0] -eq $planted)
+ }
+
+ # ProgramW6432 is the 64-bit Program Files whatever the process bitness. Model the 32-bit
+ # host: $env:ProgramFiles is the x86 tree, and the driver is only in the 64-bit one.
+ Reset-FakeMachine
+ $env:ProgramFiles = Join-Path $tmp "Program Files (x86)"
+ $planted = Add-FakeSmi "Program Files/NVIDIA Corporation/NVSMI"
+ $found = @(Get-NvidiaSmiCandidatePaths)
+ Check "found through ProgramW6432 when ProgramFiles points at the x86 tree" (
+ $found.Count -eq 1 -and $found[0] -eq $planted)
+
+ # Bites control. A host with no copy anywhere must come back empty, or every check above
+ # would pass on a helper that simply returns a fixed list without testing for the file.
+ Reset-FakeMachine
+ New-Item -ItemType Directory -Force -Path (Join-Path $tmp "Windows/System32") | Out-Null
+ Check "control: nothing planted, nothing found" ((@(Get-NvidiaSmiCandidatePaths)).Count -eq 0)
+
+ # A directory named but absent must not throw, since this runs before anything is installed.
+ Reset-FakeMachine
+ Check "a machine missing every directory is handled" ((@(Get-NvidiaSmiCandidatePaths)).Count -eq 0)
+
+ # The current driver's copy wins over the legacy one. Both work; the System32 copy belongs to
+ # the driver that is actually loaded, and the NVSMI one can be older and report an older CUDA.
+ Reset-FakeMachine
+ $current = Add-FakeSmi "Windows/System32"
+ $null = Add-FakeSmi "Program Files/NVIDIA Corporation/NVSMI"
+ $found = @(Get-NvidiaSmiCandidatePaths)
+ Check "System32 is offered before the legacy NVSMI directory" (
+ $found.Count -eq 2 -and $found[0] -eq $current)
+
+ # Each probe is a process spawn with a wall-clock bound behind it, so the same file reached
+ # two ways must be offered once. ProgramFiles and ProgramW6432 are equal on a 64-bit host,
+ # which is the ordinary case, not a contrived one.
+ Reset-FakeMachine
+ $planted = Add-FakeSmi "Program Files/NVIDIA Corporation/NVSMI"
+ $found = @(Get-NvidiaSmiCandidatePaths)
+ Check "a directory reached twice is probed once" ($found.Count -eq 1 -and $found[0] -eq $planted)
+
+ # DriverStore holds every package the machine has ever had. Walking it whole on a host with a
+ # long driver history is the kind of thing that turns an install into a hang.
+ Reset-FakeMachine
+ for ($i = 0; $i -lt 40; $i++) { $null = Add-FakeSmi "Windows/System32/DriverStore/FileRepository/nv$i.inf_amd64_$i/" }
+ $null = Add-FakeSmi "Windows/System32/DriverStore/FileRepository/unrelated.inf_amd64_9/"
+ $found = @(Get-NvidiaSmiCandidatePaths)
+ Check "the DriverStore scan is bounded and filtered to NVIDIA's own packages" (
+ $found.Count -le 8 -and -not ($found -match "unrelated"))
+
+ # Same WOW64 reason SysNative is in the direct list: in a 32-bit process "System32" redirects
+ # to SysWOW64, which carries no DriverStore at all, so a host whose ONLY nvidia-smi is the
+ # driver package was reported as having no NVIDIA GPU. That is the population this recovers.
+ Reset-FakeMachine
+ $planted = Add-FakeSmi "Windows/SysNative/DriverStore/FileRepository/nvmii.inf_amd64_0001/"
+ $found = @(Get-NvidiaSmiCandidatePaths)
+ Check "the DriverStore is reached through SysNative as well as System32" (
+ $found.Count -eq 1 -and $found[0] -eq $planted)
+
+ # The two spellings name one directory on a real host, so the bound is on the whole
+ # DriverStore contribution rather than per root: adding SysNative must not double the probes.
+ Reset-FakeMachine
+ for ($i = 0; $i -lt 12; $i++) { $null = Add-FakeSmi "Windows/System32/DriverStore/FileRepository/nva$i.inf_amd64_$i/" }
+ for ($i = 0; $i -lt 12; $i++) { $null = Add-FakeSmi "Windows/SysNative/DriverStore/FileRepository/nvb$i.inf_amd64_$i/" }
+ $found = @(Get-NvidiaSmiCandidatePaths)
+ Check "both DriverStore spellings still share one bound of eight" ($found.Count -le 8)
+
+ # The bound applies to directories that actually HOLD the binary, not to the nv* name matches.
+ # NVIDIA ships nv-something packages with no nvidia-smi in them (HD audio, virtual audio,
+ # network service); installed after the display driver they are newer by write time, so a
+ # bound on the name match spends the whole allowance on empty directories and reports no
+ # nvidia-smi. Plant the real one FIRST so it is the oldest, then bury it.
+ Reset-FakeMachine
+ $repo = "Windows/System32/DriverStore/FileRepository"
+ $planted = Add-FakeSmi "$repo/nvmii.inf_amd64_0001/"
+ for ($i = 0; $i -lt 12; $i++) {
+ $dir = Join-Path $tmp "$repo/nvhda.inf_amd64_10$i"
+ New-Item -ItemType Directory -Force -Path $dir | Out-Null
+ [System.IO.File]::WriteAllText((Join-Path $dir "nvhda.sys"), "")
+ }
+ $found = @(Get-NvidiaSmiCandidatePaths)
+ Check "a display driver buried under newer nv* packages that ship no nvidia-smi is still found" (
+ $found.Count -eq 1 -and $found[0] -eq $planted)
+
+ # The 32-bit process. $env:ProgramFiles is the x86 tree, so a stale copy from an old 32-bit
+ # toolkit sits in the FIRST Program Files rung unless the 64-bit locations come first. It is a
+ # real binary reporting its own older CUDA version, so only the order rescues this.
+ Reset-FakeMachine
+ $env:ProgramFiles = Join-Path $tmp "Program Files (x86)"
+ $env:ProgramW6432 = Join-Path $tmp "Program Files"
+ $stale = Add-FakeSmi "Program Files (x86)/NVIDIA Corporation/NVSMI"
+ $current = Add-FakeSmi "Windows/SysNative"
+ $found = @(Get-NvidiaSmiCandidatePaths)
+ Check "in a 32-bit process SysNative is probed before the stale x86 toolkit copy" (
+ $found.Count -eq 2 -and $found[0] -eq $current -and $found[1] -eq $stale)
+
+ # Same host, driver in the 64-bit Program Files rather than SysNative.
+ Reset-FakeMachine
+ $env:ProgramFiles = Join-Path $tmp "Program Files (x86)"
+ $env:ProgramW6432 = Join-Path $tmp "Program Files"
+ $stale = Add-FakeSmi "Program Files (x86)/NVIDIA Corporation/NVSMI"
+ $current = Add-FakeSmi "Program Files/NVIDIA Corporation/NVSMI"
+ $found = @(Get-NvidiaSmiCandidatePaths)
+ Check "in a 32-bit process the 64-bit NVSMI copy is probed before the x86 one" (
+ $found.Count -eq 2 -and $found[0] -eq $current -and $found[1] -eq $stale)
+} finally {
+ $env:SystemRoot = $saved.SystemRoot
+ $env:ProgramFiles = $saved.ProgramFiles
+ if ($null -eq $saved.ProgramW6432) { Remove-Item Env:ProgramW6432 -ErrorAction SilentlyContinue }
+ else { $env:ProgramW6432 = $saved.ProgramW6432 }
+ if ($null -eq $saved.ProgramFilesX86) { Remove-Item "Env:ProgramFiles(x86)" -ErrorAction SilentlyContinue }
+ else { ${env:ProgramFiles(x86)} = $saved.ProgramFilesX86 }
+ if (Test-Path -LiteralPath $tmp) { Remove-Item -LiteralPath $tmp -Recurse -Force -ErrorAction SilentlyContinue }
+}
+
+# ------------------------------------------------- which candidate the detection loop settles on
+#
+# The scenario this reordering could have caused. Two binaries both answer -L; the first prints a
+# banner with no parseable CUDA version, the second reports 12.8. Taking the first would send a
+# cu128-capable host to cu126 purely because of which directory is searched first.
+$setupAll = Get-Content -Raw -LiteralPath $setupPs1
+$start = $setupAll.IndexOf('$smiCandidates = @(Get-NvidiaSmiCandidatePaths)')
+if ($start -lt 0) { Check "the detection loop still makes two passes" $false }
+else {
+ # Bounded by the next statement rather than by a character count. A fixed 1400 stopped
+ # short of the fallback the moment a comment was added above it, and the check failed
+ # for a reason that had nothing to do with the loop.
+ $loopEndForRead = $setupAll.IndexOf('if (-not $HasNvidiaSmi -and (Get-NvidiaLibraryInventory))', $start)
+ $loop = $setupAll.Substring($start, $loopEndForRead - $start)
+ Check "the loop prefers a candidate that also reports a CUDA version" (
+ $loop -match "CUDA\(\?: UMD\)\? Version:")
+ Check "and still settles for one that only lists a GPU when none reports a version" (
+ $loop -match '\$firstListing' -and $loop -match 'if \(-not \$HasNvidiaSmi -and \$firstListing\)')
+}
+
+# Driven, not read. Both candidates list a GPU; only the second names a version.
+$script:SmiCalls = @()
+function Test-NvidiaSmiHasGpu { param([string]$Exe) $script:SmiCalls += "L:$Exe"; return $true }
+function Invoke-NvidiaSmiBounded {
+ param($Exe, $Arguments)
+ $script:SmiCalls += "B:$Exe"
+ if ($Exe -like "*second*") { return "CUDA Version: 12.8" }
+ return "NVIDIA-SMI has failed because it couldn't communicate with the driver"
+}
+function Write-StudioLine { param([string]$Line, [string]$ForegroundColor = "") }
+function Get-NvidiaSmiCandidatePaths { return @("C:\first\nvidia-smi.exe", "C:\second\nvidia-smi.exe") }
+# Anchored on the deadline, the first statement of the loop's setup. Anchoring on $firstListing
+# left $probeDeadline undefined inside the slice, and comparing against $null broke out on the
+# first iteration, so every driven check below passed or failed for the wrong reason.
+$loopStart = $setupAll.IndexOf('$smiCandidates = @(Get-NvidiaSmiCandidatePaths)')
+$loopEnd = $setupAll.IndexOf('if (-not $HasNvidiaSmi -and (Get-NvidiaLibraryInventory))', $loopStart)
+# The slice starts inside setup.ps1's "if (-not $HasNvidiaSmi) {" block, so it carries that
+# block's closing brace and is short one opening brace. Supply the opener rather than trimming a
+# brace off the end, which would silently drop the last statement.
+$loopSrc = 'if ($true) {' + [System.Environment]::NewLine + $setupAll.Substring($loopStart, $loopEnd - $loopStart)
+$HasNvidiaSmi = $false
+$NvidiaSmiExe = $null
+Invoke-Expression $loopSrc
+Check "the candidate that reports a CUDA version wins over the one searched first" (
+ $HasNvidiaSmi -eq $true -and $NvidiaSmiExe -eq "C:\second\nvidia-smi.exe")
+
+# And when NOTHING reports a version, the first that lists a GPU is still used: this must not
+# turn a detected GPU into no GPU at all.
+function Invoke-NvidiaSmiBounded { param($Exe, $Arguments) return "no version here" }
+$HasNvidiaSmi = $false
+$NvidiaSmiExe = $null
+Invoke-Expression $loopSrc
+Check "with no version anywhere, the first listing candidate is still taken" (
+ $HasNvidiaSmi -eq $true -and $NvidiaSmiExe -eq "C:\first\nvidia-smi.exe")
+
+# ----------------------------------------------------- and it gives up before setup does
+#
+# Each candidate gets its own bounded runner, and on a wedged driver every one waits out that
+# bound. A longer list turns two stalls into ten, all in front of a driver-library fallback that
+# answers in milliseconds. Driven with probes that really sleep: the thing under test is elapsed
+# time, and a stubbed clock would only prove the arithmetic.
+#
+# The deadline in the shipped file is 30 seconds, which no test should sit through. The slice is
+# rewritten to 2 before it is invoked, and the real value is asserted separately just below so
+# that rewrite cannot quietly become the thing being tested.
+Check "the shipped soft deadline is 30 seconds" ($setupAll -match '\$probeDeadline = \(Get-Date\)\.AddSeconds\(30\)')
+Check "the shipped hard deadline is 60 seconds" ($setupAll -match '\$probeHardDeadline = \(Get-Date\)\.AddSeconds\(60\)')
+$fastLoop = ($loopSrc -replace 'AddSeconds\(30\)', 'AddSeconds(2)') -replace 'AddSeconds\(60\)', 'AddSeconds(4)'
+Check "the slice really carries both shortened deadlines (bites)" (
+ $fastLoop -match 'AddSeconds\(2\)' -and $fastLoop -match 'AddSeconds\(4\)')
+
+$script:Probed = 0
+function Test-NvidiaSmiHasGpu { param([string]$Exe) $script:Probed++; Start-Sleep -Milliseconds 700; return $true }
+function Invoke-NvidiaSmiBounded { param($Exe, $Arguments) return "no version here" }
+function Get-NvidiaSmiCandidatePaths {
+ return @(1..10 | ForEach-Object { "C:\wedged$_\nvidia-smi.exe" })
+}
+$HasNvidiaSmi = $false
+$NvidiaSmiExe = $null
+$started = Get-Date
+Invoke-Expression $fastLoop
+$elapsed = ((Get-Date) - $started).TotalSeconds
+
+Check "a wedged host stops probing instead of walking every candidate" ($script:Probed -lt 10)
+Check "it probed at least once rather than skipping the loop outright" ($script:Probed -ge 1)
+Check "the whole loop is bounded near its deadline, not by the candidate count" ($elapsed -lt 6)
+# The bound must not cost a detection. A host that answers is still detected, and it is the FIRST
+# listing candidate, exactly as it was without the deadline.
+Check "giving up early still reports the GPU it did find" (
+ $HasNvidiaSmi -eq $true -and $NvidiaSmiExe -eq "C:\wedged1\nvidia-smi.exe")
+
+# Bites control: the same loop on a host where nothing hangs walks the entire list, so the check
+# above is about elapsed time and not about the loop simply stopping after one candidate.
+$script:Probed = 0
+function Test-NvidiaSmiHasGpu { param([string]$Exe) $script:Probed++; return $false }
+$HasNvidiaSmi = $false
+$NvidiaSmiExe = $null
+Invoke-Expression $fastLoop
+Check "control: with fast probes every candidate is still tried" ($script:Probed -eq 10)
+
+# ------------------------------------------- the wedge flag must describe the binary we chose
+#
+# Test-NvidiaSmiHasGpu records whether the binary it just probed timed out, and its own comment
+# says what matters is "whether the binary it settled on answered". The two-pass loop broke that:
+# it can keep probing after the candidate it will settle on, so the flag ends up describing a
+# LATER binary. Downstream that flag decides whether to ask the selected nvidia-smi for the GPU
+# name, the compute capability and the driver version, so a working binary would have all three
+# skipped because a different copy elsewhere on the machine hung.
+$script:NvidiaSmiWedged = $false
+function Test-NvidiaSmiHasGpu {
+ param([string]$Exe)
+ # The first candidate answers and is the one the loop settles on. The second hangs.
+ if ($Exe -like "*good*") { $script:NvidiaSmiWedged = $false; return $true }
+ $script:NvidiaSmiWedged = $true
+ return $false
+}
+function Invoke-NvidiaSmiBounded { param($Exe, $Arguments) return "no version here" }
+function Get-NvidiaSmiCandidatePaths {
+ return @("C:\good\nvidia-smi.exe", "C:\hung\nvidia-smi.exe")
+}
+$HasNvidiaSmi = $false
+$NvidiaSmiExe = $null
+Invoke-Expression $loopSrc
+Check "the working candidate is the one selected" (
+ $HasNvidiaSmi -eq $true -and $NvidiaSmiExe -eq "C:\good\nvidia-smi.exe")
+Check "a later candidate hanging does not mark the selected binary wedged" (
+ $script:NvidiaSmiWedged -eq $false)
+
+# Bites control: when the binary the loop settles on is ITSELF the one that hung, the flag must
+# stay set, or the guard downstream would query a hung nvidia-smi and wait out the bound again.
+function Test-NvidiaSmiHasGpu {
+ param([string]$Exe)
+ $script:NvidiaSmiWedged = $true
+ return $true
+}
+$script:NvidiaSmiWedged = $false
+$HasNvidiaSmi = $false
+$NvidiaSmiExe = $null
+Invoke-Expression $loopSrc
+Check "control: a selected binary that did hang is still reported wedged" (
+ $script:NvidiaSmiWedged -eq $true)
+
+# ------------------------------------- giving up must never cost a GPU that was there to find
+#
+# The regression the first version of the deadline introduced. One budget, and a break in the
+# failure branch, meant a single slow FAILING probe ended the whole search: a machine whose
+# System32 copy is broken and whose legacy NVSMI copy works lost its GPU entirely and went to
+# CPU-only PyTorch, strictly worse than the stall the deadline was added to prevent.
+#
+# Two budgets now: the soft one only applies once there is already a usable answer, and only the
+# hard one may end the search empty-handed.
+$script:Probed = 0
+function Test-NvidiaSmiHasGpu {
+ param([string]$Exe)
+ $script:Probed++
+ if ($Exe -like "*broken*") { Start-Sleep -Milliseconds 2500; return $false } # slow AND failing
+ return $true
+}
+function Invoke-NvidiaSmiBounded { param($Exe, $Arguments) return "CUDA Version: 12.8" }
+function Get-NvidiaSmiCandidatePaths {
+ return @("C:\broken\nvidia-smi.exe", "C:\legacy\nvidia-smi.exe")
+}
+$script:NvidiaSmiWedged = $false
+$HasNvidiaSmi = $false
+$NvidiaSmiExe = $null
+Invoke-Expression $fastLoop
+Check "a slow FAILING first probe does not end the search" ($script:Probed -ge 2)
+Check "the working copy behind it is still found" (
+ $HasNvidiaSmi -eq $true -and $NvidiaSmiExe -eq "C:\legacy\nvidia-smi.exe")
+
+# The hard bound still exists: with nothing working anywhere, the loop stops rather than walking
+# an unbounded list. This is the property the deadline was added for, and it must survive the fix.
+$script:Probed = 0
+function Test-NvidiaSmiHasGpu { param([string]$Exe) $script:Probed++; Start-Sleep -Milliseconds 700; return $false }
+function Get-NvidiaSmiCandidatePaths { return @(1..20 | ForEach-Object { "C:\wedged$_\nvidia-smi.exe" }) }
+$HasNvidiaSmi = $false
+$NvidiaSmiExe = $null
+$t0 = Get-Date
+Invoke-Expression $fastLoop
+$spent = ((Get-Date) - $t0).TotalSeconds
+Check "with nothing working the hard bound still stops the walk" ($script:Probed -lt 20)
+Check "and it spends the HARD budget, not the soft one, before giving up" ($spent -ge 2)
+Check "control: giving up empty-handed really reports no GPU" ($HasNvidiaSmi -eq $false)
+
+# The banner probe has to respect the deadline too.
+#
+# Checking only at the top of the loop is not enough: a candidate starting just inside the
+# deadline can add nearly twice the per-probe timeout after it, since the listing probe may have
+# spent most of its bound and the banner probe has a full one. The earlier elapsed-time check
+# missed this because its banner stub returned instantly.
+# Driven with a banner probe that is the slow one.
+$script:Listings = 0
+$script:Banners = 0
+function Test-NvidiaSmiHasGpu { param([string]$Exe) $script:Listings++; Start-Sleep -Milliseconds 900; return $true }
+function Invoke-NvidiaSmiBounded {
+ param($Exe, $Arguments)
+ $script:Banners++
+ Start-Sleep -Milliseconds 900
+ return "no version here"
+}
+function Get-NvidiaSmiCandidatePaths { return @(1..10 | ForEach-Object { "C:\slow$_\nvidia-smi.exe" }) }
+$script:NvidiaSmiWedged = $false
+$HasNvidiaSmi = $false
+$NvidiaSmiExe = $null
+$t0 = Get-Date
+Invoke-Expression $fastLoop
+$spent = ((Get-Date) - $t0).TotalSeconds
+# Counted, not timed. "Fewer than 10 banners" and "under 8 seconds" BOTH held with the fix
+# reverted, since at 1.8 s per candidate the top-of-loop check stops one candidate later either
+# way. With the recheck exactly one banner runs; without it candidate 2 gets one too.
+Check "a slow banner probe does not start after the deadline" ($script:Banners -eq 1)
+Check "control: a banner probe did run for the candidate before it" ($script:Listings -ge 2)
+# Loose sanity only; the counter above is the check that bites.
+Check "the loop does not spend two probe bounds per candidate after expiry" ($spent -lt 8)
+# And the GPU is still reported: everything here listed, so the fallback must take the first.
+Check "control: the GPU found before the deadline is still reported" (
+ $HasNvidiaSmi -eq $true -and $NvidiaSmiExe -eq "C:\slow1\nvidia-smi.exe")
+
+# ------------------------------- the Windows-on-ARM picker must answer with a WORKING binary
+#
+# Get-WoaNvidiaSmiPath fed the first candidate that EXISTS to Test-WoaNvidiaPresent and
+# Get-WoaDriverCudaVersion, both of which treat a non-answer as "no NVIDIA on this machine", so a
+# broken copy declines the native ARM64 CUDA stack outright rather than costing a retry. A longer
+# list makes that likely rather than unlikely.
+$woaFn = Get-FunctionText $installPs1 "Get-WoaNvidiaSmiPath"
+$script:WoaNvidiaSmiProbed = $false
+$script:WoaNvidiaSmiPath = $null
+$script:WoaProbes = @()
+function Get-Command { param($Name, $ErrorAction) return $null } # nothing on PATH
+function Invoke-NvidiaSmiBounded {
+ param($Exe, $Arguments)
+ $script:WoaProbes += $Exe
+ if ($Exe -like "*working*") {
+ $global:LASTEXITCODE = 0
+ if ($Arguments -contains '-L') { return "GPU 0: NVIDIA GeForce RTX 4090" }
+ return "CUDA Version: 12.8"
+ }
+ $global:LASTEXITCODE = 1
+ return ""
+}
+function Get-NvidiaSmiCandidatePaths {
+ return @("C:\stale\nvidia-smi.exe", "C:\working\nvidia-smi.exe")
+}
+Invoke-Expression $woaFn
+$woaAnswer = Get-WoaNvidiaSmiPath
+Check "the ARM64 picker skips a stale copy and returns one that answers" (
+ $woaAnswer -eq "C:\working\nvidia-smi.exe")
+Check "it really probed rather than pattern-matching the name" ($script:WoaProbes.Count -ge 2)
+
+# Memoised: the two callers each probe, and walking the list twice costs a process spawn per
+# entry. A second call must not re-probe.
+$before = $script:WoaProbes.Count
+$null = Get-WoaNvidiaSmiPath
+Check "a second call is served from the memo" ($script:WoaProbes.Count -eq $before)
+
+# And when NOTHING answers it still hands back the first that exists, which is what it did
+# before this change, so a host where the probe cannot run is no worse off than it was.
+$script:WoaNvidiaSmiProbed = $false
+$script:WoaNvidiaSmiPath = $null
+function Invoke-NvidiaSmiBounded { param($Exe, $Arguments) $global:LASTEXITCODE = 1; return "" }
+Check "with nothing answering it falls back to the first existing copy" (
+ (Get-WoaNvidiaSmiPath) -eq "C:\stale\nvidia-smi.exe")
+
+# The ARM64 picker needs BOTH probes, not just the listing.
+#
+# A null banner from this same binary does not merely lose a version: it skips the CUDA-major
+# guard in Initialize-WoaNativeCudaTorch that stops a native wheel newer than the installed driver
+# being selected. So a listing-only candidate must not win over one that names a version.
+$script:WoaNvidiaSmiProbed = $false
+$script:WoaNvidiaSmiPath = $null
+function Invoke-NvidiaSmiBounded {
+ param($Exe, $Arguments)
+ $global:LASTEXITCODE = 0
+ if ($Arguments -contains '-L') { return "GPU 0: NVIDIA GeForce RTX 4090" }
+ if ($Exe -like "*versioned*") { return "CUDA Version: 12.8" }
+ return "NVIDIA-SMI has failed because it couldn't communicate with the driver"
+}
+function Get-NvidiaSmiCandidatePaths {
+ return @("C:\listingonly\nvidia-smi.exe", "C:\versioned\nvidia-smi.exe")
+}
+Check "a candidate that also names a CUDA version wins over a listing-only one" (
+ (Get-WoaNvidiaSmiPath) -eq "C:\versioned\nvidia-smi.exe")
+
+# And when none names a version, the listing-only candidate is still used: the presence check
+# must keep working, and only the version check declines.
+$script:WoaNvidiaSmiProbed = $false
+$script:WoaNvidiaSmiPath = $null
+function Invoke-NvidiaSmiBounded {
+ param($Exe, $Arguments)
+ $global:LASTEXITCODE = 0
+ if ($Arguments -contains '-L') { return "GPU 0: NVIDIA GeForce RTX 4090" }
+ return "no version here"
+}
+Check "with no version anywhere the first listing candidate is still used" (
+ (Get-WoaNvidiaSmiPath) -eq "C:\listingonly\nvidia-smi.exe")
+
+# The scan is bounded as a whole. Without this a wedged driver plus eight retained DriverStore
+# copies held Initialize-WoaNativeCudaTorch for minutes before the install started.
+Check "the ARM64 scan carries a soft and a hard deadline" (
+ $woaFn -match '\$woaDeadline = \(Get-Date\)\.AddSeconds\(30\)' -and
+ $woaFn -match '\$woaHardDeadline = \(Get-Date\)\.AddSeconds\(60\)')
+
+$fastWoa = ($woaFn -replace 'AddSeconds\(30\)', 'AddSeconds(2)') -replace 'AddSeconds\(60\)', 'AddSeconds(4)'
+Check "the shortened ARM64 slice really carries both (bites)" (
+ $fastWoa -match 'AddSeconds\(2\)' -and $fastWoa -match 'AddSeconds\(4\)')
+Invoke-Expression $fastWoa
+$script:WoaNvidiaSmiProbed = $false
+$script:WoaNvidiaSmiPath = $null
+$script:WoaSlowProbes = 0
+function Invoke-NvidiaSmiBounded {
+ param($Exe, $Arguments)
+ $script:WoaSlowProbes++
+ Start-Sleep -Milliseconds 700
+ $global:LASTEXITCODE = 1
+ return ""
+}
+function Get-NvidiaSmiCandidatePaths { return @(1..20 | ForEach-Object { "C:\wedged$_\nvidia-smi.exe" }) }
+$null = Get-WoaNvidiaSmiPath
+Check "a wedged ARM64 host stops scanning instead of walking every candidate" (
+ $script:WoaSlowProbes -lt 20)
+Check "control: it probed rather than declining immediately" ($script:WoaSlowProbes -ge 2)
+Invoke-Expression $woaFn
+Remove-Item Function:\Get-Command -ErrorAction SilentlyContinue
+
+if ($failures -gt 0) {
+ Write-Host "$failures check(s) failed" -ForegroundColor Red
+ exit 1
+}
+Write-Host "all checks passed" -ForegroundColor Green