Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

feat(os): check NVIDIA readiness without interrupting work (#835)

Deee81e1004ac83
diff --git a/.github/workflows/os-nvidia.yml b/.github/workflows/os-nvidia.yml
index f1b4c8e6d..78dcdd1a4 100644
--- a/.github/workflows/os-nvidia.yml
+++ b/.github/workflows/os-nvidia.yml
@@ -3,6 +3,11 @@ name: Harness OS offline NVIDIA installation
on:
workflow_dispatch:
inputs:
+ probe:
+ description: 'Offline driver installation or private GPU diagnostic integration'
+ type: choice
+ options: [install, health, ptx]
+ default: install
image_run_id:
description: 'Verified reference image for focused candidate testing'
type: string
@@ -14,11 +19,12 @@ permissions:
actions: read
concurrency:
- group: harness-nvidia-install-${{ github.ref }}
+ group: harness-nvidia-${{ inputs.probe || 'install' }}-${{ github.ref }}
cancel-in-progress: false
jobs:
nvidia:
+ if: ${{ !inputs.probe || inputs.probe == 'install' }}
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
@@ -79,3 +85,59 @@ jobs:
os/dist/nvidia-bundle-manifest.json
retention-days: 14
compression-level: 0
+
+ ptx:
+ if: ${{ inputs.probe == 'ptx' }}
+ runs-on: ubuntu-24.04
+ timeout-minutes: 5
+ steps:
+ - uses: actions/checkout@v7
+ - name: Assemble the exact probe with the pinned NVIDIA compiler
+ run: python3 os/tests/gpu_ptx_native.py --output os/test-results/gpu-ptx
+ - name: Retain compiler evidence without the tool archive
+ if: always()
+ uses: actions/upload-artifact@v6
+ with:
+ name: harness-os-gpu-ptx
+ path: os/test-results/gpu-ptx/
+ retention-days: 14
+ compression-level: 0
+
+ health:
+ if: ${{ inputs.probe == 'health' }}
+ runs-on: ubuntu-24.04
+ timeout-minutes: 20
+ steps:
+ - uses: actions/checkout@v7
+ - uses: actions/download-artifact@v6
+ with:
+ name: programmer-os-x86_64
+ path: os/dist
+ run-id: ${{ inputs.image_run_id }}
+ github-token: ${{ github.token }}
+ - name: Prepare native ABI and VM tools
+ run: |
+ sudo apt-get update -qq
+ sudo apt-get install -y --no-install-recommends build-essential libegl1 libegl-mesa0 libgles2 mesa-vulkan-drivers qemu-system-x86 qemu-utils ovmf python3-pil tesseract-ocr
+ sudo chmod a+rw /dev/kvm
+ - name: Check OS contracts and real EGL ABI
+ env:
+ LIBGL_ALWAYS_SOFTWARE: 'true'
+ run: |
+ make -C os check
+ go run github.com/rhysd/actionlint/cmd/[email protected] .github/workflows/os-nvidia.yml
+ mkdir -p os/test-results/gpu-abi
+ python3 os/tests/gpu_egl_native.py > os/test-results/gpu-abi/mesa.json
+ - name: Check automatic diagnostics across encrypted cold boots
+ run: python3 os/tests/gpu_health_vm.py --iso os/dist/*.iso --output os/test-results/gpu-health
+ - name: Retain evidence without virtual disks
+ if: always()
+ uses: actions/upload-artifact@v6
+ with:
+ name: harness-os-gpu-health
+ path: |
+ os/test-results/gpu-abi/
+ os/test-results/gpu-health/
+ !os/test-results/gpu-health/*.qcow2
+ !os/test-results/gpu-health/OVMF_VARS.fd
+ retention-days: 14
diff --git a/os/DEVELOPMENT.md b/os/DEVELOPMENT.md
index 7d88d1508..0e5f23c26 100644
--- a/os/DEVELOPMENT.md
+++ b/os/DEVELOPMENT.md
@@ -13,6 +13,47 @@ display/SSH interaction is still unverified.
## Next release priorities
+### NVIDIA readiness
+
+The installed PC session starts a one-shot user check 15 seconds after the
+workspace starts. It is inert on the installer and on computers without NVIDIA
+GPUs. A per-user lock and boot/kernel/driver/device/probe fingerprint prevent
+screen restarts from repeating it. Each new boot or relevant change rechecks;
+`harness hardware --check-gpu` explicitly repeats it. Reports and the last fully
+working result are atomic local JSON in `~/.local/state/harness-os/gpu/`.
+They are user diagnostics, never privileged authorization or proof for a release.
+
+Each owned card is addressed by PCI bus ID. The CUDA Driver API checks device
+visibility, a 4 KiB memory upload/readback, and 1,024 integer results from a tiny
+PTX kernel. A separate EGL device context renders and reads three colors using
+the NVIDIA renderer. No CUDA toolkit, compiler, model or inference runtime enters
+the image. Normal user permissions apply. Reserved devices are skipped, and
+headless graphics failures remain explicitly unverified rather than labelling
+working compute broken. Offscreen EGL does not establish compositor, browser,
+physical display or suspend behavior; the memory sample is not a stress test.
+
+Each subprocess has a 20-second deadline within a 120-second total budget; hangs,
+crashes, incomplete responses and wrong readbacks are failures with the last
+operation recorded. All contexts are private and released. The check never
+resets a GPU, unloads modules, changes packages, restarts an agent or reboots.
+Only actionable failure gets a short existing Harness status message. Package
+replacement/restart-pending state defers verification until reboot. Existing full
+snapshot updates keep the kernel and driver together and preserve the original
+root/boot checkpoint; failures retain that recovery path, without an automatic
+rollback that might interrupt work. Local-model provisioning remains the TUI's
+responsibility.
+
+The API sources are NVIDIA's [Driver API](https://docs.nvidia.com/cuda/cuda-programming-guide/03-advanced/driver-api.html),
+[stable context ABI](https://docs.nvidia.com/cuda/archive/12.9.1/cuda-driver-api/group__CUDA__CTX.html),
+and Khronos [EGL device displays](https://registry.khronos.org/EGL/extensions/EXT/EGL_EXT_platform_device.txt),
+[render-node mapping](https://registry.khronos.org/EGL/extensions/EXT/EGL_EXT_device_drm_render_node.txt),
+and [CUDA device mapping](https://registry.khronos.org/EGL/extensions/NV/EGL_NV_device_cuda.txt).
+Portable tests include a compiled C ABI fixture with corrupt-memory, wrong-answer,
+async-error and wrong-device cases. Such fixtures establish error handling, not
+physical NVIDIA acceptance. Real cards must still pass these checks and actual
+display, browser, update/recovery and sleep/wake testing before a family is called
+supported.
+
The next preview improves the complete experience before expanding the interface.
Keep the existing terminal, agents and optional browser. Hardware integration must
not introduce a desktop, control panel or extra launcher. Update checks use a
diff --git a/os/gpu_health.py b/os/gpu_health.py
new file mode 100644
index 000000000..c52f50e93
--- /dev/null
+++ b/os/gpu_health.py
@@ -0,0 +1,260 @@
+#!/usr/bin/env python3
+"""Quiet, bounded GPU verification for an installed Harness session."""
+import argparse
+from datetime import datetime, timezone
+import fcntl
+import hashlib
+import importlib.util
+import json
+import os
+from pathlib import Path
+import re
+import subprocess
+import sys
+import tempfile
+import time
+
+STATE = Path.home() / '.local/state/harness-os/gpu'
+SYS = Path('/sys')
+PROC = Path('/proc')
+SYSTEM_STATE = Path('/var/lib/harness-os')
+PROBE = Path(__file__).with_name('gpu_probe.py')
+VERSION = 1
+PROBE_SECONDS = 20
+TOTAL_SECONDS = 120
+
+
+def read_json(path):
+ try:
+ value = json.loads(path.read_text())
+ return value if isinstance(value, dict) else {}
+ except (OSError, ValueError):
+ return {}
+
+
+def text(path):
+ try:
+ return path.read_text().strip()
+ except (OSError, UnicodeError):
+ return ''
+
+
+def hardware():
+ spec = importlib.util.spec_from_file_location('harness_gpu_hardware', Path(__file__).with_name('hardware.py'))
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+def write_json(path, value):
+ # Same-directory atomic publication; no half-written success after a crash.
+ with tempfile.NamedTemporaryFile(dir=path.parent, prefix='.gpu-', mode='w', delete=False) as handle:
+ pending = Path(handle.name)
+ try:
+ json.dump(value, handle, indent=2)
+ handle.write('\n')
+ handle.flush()
+ os.fsync(handle.fileno())
+ pending.replace(path)
+ finally:
+ pending.unlink(missing_ok=True)
+
+
+def library_identity(path):
+ try:
+ resolved, info = path.resolve(strict=True), path.stat()
+ return {'path': str(resolved), 'bytes': info.st_size, 'mtime_ns': info.st_mtime_ns}
+ except OSError:
+ return None
+
+
+def inventory():
+ cards = [card for card in hardware().pci_devices(SYS)
+ if card['id'].startswith('10de:') and card['class'] in {'030000', '030200'}]
+ return {'schema': VERSION, 'boot_id': text(PROC / 'sys/kernel/random/boot_id'),
+ 'kernel': os.uname().release, 'driver_version': text(SYS / 'module/nvidia/version'),
+ 'libraries': {name: library_identity(Path('/usr/lib') / name)
+ for name in ['libcuda.so.1', 'libEGL_nvidia.so.0', 'libnvidia-eglcore.so.' + text(SYS / 'module/nvidia/version')]}
+ if cards else {}, 'cards': cards,
+ 'probe_sha256': hashlib.sha256(PROBE.read_bytes()).hexdigest(),
+ 'controller_sha256': hashlib.sha256(Path(__file__).read_bytes()).hexdigest()}
+
+
+def fingerprint(value):
+ return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()
+
+
+def cached_report():
+ report = read_json(STATE / 'health.json')
+ if not report:
+ return {'status': 'not-tested', 'reason': 'GPU verification has not run for this user.'}
+ report['stale'] = report.get('fingerprint') != fingerprint(inventory())
+ return report
+
+
+def probe(kind, address, timeout):
+ # Ignore a task's CUDA selection/debug injection in this OS diagnostic. Never
+ # change the environment of existing agents, their processes or GPU contexts.
+ env = {key: value for key, value in os.environ.items() if not key.startswith(
+ ('CUDA_', 'NVIDIA_', '__NV_', '__EGL_', 'LD_'))}
+ env.update(CUDA_CACHE_DISABLE='1', CUDA_MODULE_LOADING='EAGER')
+ command = [sys.executable, str(PROBE), kind, address]
+ failure = None
+ try:
+ result = subprocess.run(command, capture_output=True, text=True, timeout=timeout, env=env)
+ output, errors = result.stdout, result.stderr
+ if result.returncode:
+ failure = f'Probe exited with status {result.returncode}'
+ except subprocess.TimeoutExpired as error:
+ output, errors = error.stdout or b'', error.stderr or b''
+ failure = f'Probe exceeded {timeout:.1f} seconds'
+ except OSError as error:
+ output, errors, failure = '', '', str(error)
+ if isinstance(output, bytes):
+ output = output.decode(errors='replace')
+ if isinstance(errors, bytes):
+ errors = errors.decode(errors='replace')
+ events = []
+ for line in output.splitlines():
+ try:
+ item = json.loads(line)
+ if isinstance(item, dict):
+ events.append(item)
+ except ValueError:
+ pass
+ final = events[-1].get('result') if events else None
+ if not failure and isinstance(final, dict) and final.get('address') == address and final.get('probe') == kind:
+ checks = final.get('checks')
+ required = {'cuda.device', 'cuda.memory', 'cuda.compute'} if kind == 'cuda' else {'graphics.render'}
+ if (isinstance(checks, list) and all(isinstance(c, dict) and c.get('status') in {'passed', 'failed'} for c in checks)
+ and (final.get('status') == 'failed' and any(c['status'] == 'failed' for c in checks)
+ or final.get('status') == 'passed' and all(c['status'] == 'passed' for c in checks)
+ and required <= {c.get('test') for c in checks})):
+ final['stderr'] = errors[-2048:]
+ return final
+ last = {key: next((e[key] for e in reversed(events) if key in e), None) for key in ['stage', 'operation']}
+ return {'address': address, 'probe': kind, 'status': 'failed', 'checks': [
+ {'test': kind, 'status': 'failed', 'detail': failure or 'Incomplete probe result', **last}], 'stderr': errors[-2048:]}
+
+
+def connections(address):
+ result = {}
+ for card in (SYS / 'class/drm').glob('card[0-9]*'):
+ if '-' not in card.name and (card / 'device').resolve().name == address:
+ for connector in (SYS / 'class/drm').glob(card.name + '-*'):
+ result[connector.name] = text(connector / 'status')
+ return result
+
+
+def recovery_context():
+ system = read_json(SYSTEM_STATE / 'update.json')
+ pointer = read_json(SYSTEM_STATE / 'runtime-updates/latest.json').get('id', '')
+ # The root-owned pointer can still be absent or malformed after an interrupted
+ # update. A diagnostic never follows an arbitrary path from that JSON.
+ runtime = (read_json(SYSTEM_STATE / 'runtime-updates' / pointer / 'receipt.json')
+ if isinstance(pointer, str) and re.fullmatch(r'[a-zA-Z0-9_-]{1,100}', pointer) else {})
+ return {'system_update': {key: system.get(key) for key in ['snapshot', 'checkpoint', 'exit_status']},
+ 'harness_update': {key: runtime.get(key) for key in ['checkpoint', 'status']},
+ 'method': 'Offline checkpoint recovery preserves the home subvolume. No automatic rollback or reboot.'}
+
+
+def evaluate(current):
+ started = time.monotonic()
+ installed = read_json(SYSTEM_STATE / 'hardware.json').get('nvidia', {})
+ report = {'schema': VERSION, 'fingerprint': fingerprint(current), 'inventory': current,
+ 'checked_at': datetime.now(timezone.utc).isoformat(), 'devices': [],
+ 'recovery': recovery_context(), 'limitations': [
+ 'A small memory sample and computation, not a full VRAM stress test.',
+ 'Offscreen EGL rendering does not verify display scanout, browser acceleration, or sleep/wake.',
+ 'No model runtime, GPU reset, driver unload, package change or automatic reboot.']}
+ # An intentional assignment must remain untouched, even when this GPU was
+ # installed by Harness previously. Missing drivers on managed cards are errors.
+ for card in current['cards']:
+ record = dict(card, display_connections=connections(card['address']), checks=[])
+ report['devices'].append(record)
+ if (card.get('driver_override') not in {None, '', 'nvidia'} or card['driver'] not in {None, 'nvidia', 'nouveau'}):
+ record.update(status='skipped', reason='GPU is assigned to another driver; its assignment is preserved.')
+ continue
+ if card['driver'] != 'nvidia':
+ managed = installed.get('status') == 'installed' and card['id'] in installed.get('devices', [])
+ record.update(status='failed' if managed else 'not-tested', reason='NVIDIA driver is not bound to this GPU.')
+ record['checks'].append({'test': 'driver.binding', 'status': record['status'], 'actual': card['driver']})
+ continue
+ record['checks'].append({'test': 'driver.binding', 'status': 'passed', 'actual': card['driver']})
+ for kind in ['cuda', 'graphics']:
+ remaining = TOTAL_SECONDS - (time.monotonic() - started)
+ if remaining <= 0:
+ record['checks'].append({'test': kind, 'status': 'failed', 'detail': 'Overall diagnostic deadline reached; not tested.'})
+ continue
+ result = probe(kind, card['address'], min(PROBE_SECONDS, remaining))
+ for check in result['checks']:
+ # A compute-only/headless configuration may deliberately lack
+ # graphics. Keep that evidence, without calling CUDA broken.
+ unavailable = check.get('operation') in {'load_driver', 'EGL extensions', 'eglQueryDevicesEXT',
+ 'graphics.device', 'eglGetPlatformDisplayEXT', 'eglInitialize', 'eglChooseConfig'}
+ if kind == 'graphics' and check['status'] == 'failed' and unavailable and 'connected' not in record['display_connections'].values():
+ check = dict(check, status='not-tested', reason='No connected display on this card; graphics remains unverified.')
+ record['checks'].append(check)
+ if result.get('stderr'):
+ record.setdefault('probe_stderr', {})[kind] = result['stderr']
+ record['status'] = ('failed' if any(c['status'] == 'failed' for c in record['checks']) else
+ 'partial' if any(c['status'] == 'not-tested' for c in record['checks']) else 'passed')
+ states = {row['status'] for row in report['devices']}
+ report['status'] = ('failed' if 'failed' in states else 'partial' if states & {'partial', 'not-tested'} else
+ 'passed' if 'passed' in states else 'skipped')
+ report['duration_seconds'] = round(time.monotonic() - started, 3)
+ return report
+
+
+def notify(report):
+ if report['status'] != 'failed':
+ return False
+ try:
+ subprocess.run(['/usr/bin/hn', 'display-message', '-d', '10000',
+ 'GPU needs attention. Ask your agent to check harness hardware.'],
+ check=True, capture_output=True, timeout=3)
+ return True
+ except (OSError, subprocess.SubprocessError):
+ return False # The durable report remains available without a UI.
+
+
+def check(force=False, announce=False):
+ STATE.mkdir(parents=True, exist_ok=True, mode=0o700)
+ with (STATE / 'check.lock').open('a') as lock:
+ try:
+ fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
+ except BlockingIOError:
+ return {'status': 'running', 'reason': 'GPU verification is already running.'}
+ current = inventory()
+ previous = read_json(STATE / 'health.json')
+ # Package replacement can mix a loaded old driver with new userspace.
+ # Never wake/check GPUs mid-update, nor report that as a failed new boot.
+ if Path('/var/lib/pacman/db.lck').exists() or Path('/run/harness-os-restart-required').exists():
+ report = {'schema': VERSION, 'status': 'deferred', 'reason': 'GPU verification will run after the update and restart.',
+ 'inventory': current, 'devices': [], 'recovery': recovery_context()}
+ else:
+ if not force and previous.get('fingerprint') == fingerprint(current):
+ return previous
+ report = evaluate(current)
+ if fingerprint(inventory()) != report['fingerprint'] or Path('/var/lib/pacman/db.lck').exists():
+ report['status'] = 'deferred'
+ report.pop('fingerprint', None)
+ report['reason'] = 'Hardware or driver changed during verification; run again after restart.'
+ write_json(STATE / 'health.json', report)
+ if report['status'] == 'passed':
+ write_json(STATE / 'last-working.json', report)
+ if announce:
+ notify(report)
+ return report
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument('--force', action='store_true', help='Repeat checks on this boot')
+ parser.add_argument('--notify', action='store_true', help='Show only actionable failure in the current Harness session')
+ args = parser.parse_args()
+ print(json.dumps(check(args.force, args.notify), indent=2))
+
+
+if __name__ == '__main__':
+ main()
diff --git a/os/gpu_probe.py b/os/gpu_probe.py
new file mode 100644
index 000000000..61247c2e0
--- /dev/null
+++ b/os/gpu_probe.py
@@ -0,0 +1,273 @@
+#!/usr/bin/env python3
+"""Short-lived, unprivileged GPU probes. The supervisor owns all time limits.
+
+CUDA uses the stable, explicitly versioned Driver API, not a toolkit/runtime.
+EGL uses EXT_platform_device and EXT_device_drm_render_node for exact PCI mapping.
+See os/DEVELOPMENT.md for API references and what these checks do not prove.
+"""
+import argparse
+import ctypes as C
+import json
+from pathlib import Path
+import re
+
+I, U, P = C.c_int, C.c_uint, C.c_void_p
+N = 1024
+PTX = b'''
+.version 6.0
+.target sm_50
+.address_size 64
+.visible .entry harness_check(.param .u64 data) {
+ .reg .b32 i, b, n, v;
+ .reg .b64 p, offset, addr;
+ ld.param.u64 p, [data];
+ mov.u32 i, %tid.x;
+ mov.u32 b, %ctaid.x;
+ mov.u32 n, %ntid.x;
+ mad.lo.u32 i, b, n, i;
+ mul.wide.u32 offset, i, 4;
+ add.u64 addr, p, offset;
+ ld.global.u32 v, [addr];
+ mad.lo.u32 v, v, 3, 7;
+ st.global.u32 [addr], v;
+ ret;
+}
+'''
+
+
+def event(**value):
+ print(json.dumps(value), flush=True)
+
+
+def function(library, name, result, *arguments):
+ fn = getattr(library, name)
+ fn.restype, fn.argtypes = result, list(arguments)
+ return fn
+
+
+class ProbeError(Exception):
+ def __init__(self, operation, code=None, detail=''):
+ super().__init__(detail or operation)
+ self.operation, self.code, self.detail = operation, code, detail
+
+
+def same_pci(first, second):
+ def parts(value):
+ match = re.fullmatch(r'([0-9a-f]{4,8}):([0-9a-f]{2}):([0-9a-f]{2})\.([0-7])', value.lower())
+ return tuple(int(part, 16) for part in match.groups()) if match else None
+ return parts(first) is not None and parts(first) == parts(second)
+
+
+class CUDA:
+ def __init__(self):
+ self.lib = C.CDLL('libcuda.so.1')
+ self.error_name = function(self.lib, 'cuGetErrorName', I, I, C.POINTER(C.c_char_p))
+
+ def call(self, name, types, *values):
+ event(operation=name)
+ code = function(self.lib, name, I, *types)(*values)
+ if code:
+ label = C.c_char_p()
+ self.error_name(code, C.byref(label))
+ raise ProbeError(name, code, label.value.decode() if label.value else 'CUDA error')
+
+ def pci(self, device):
+ address = C.create_string_buffer(32)
+ self.call('cuDeviceGetPCIBusId', [P, I, I], address, len(address), device)
+ return address.value.decode().lower()
+
+
+def compute(address, checks):
+ event(stage='cuda.device')
+ api = CUDA()
+ api.call('cuInit', [U], 0)
+ device, driver = I(), I()
+ api.call('cuDeviceGetByPCIBusId', [P, C.c_char_p], C.byref(device), address.encode())
+ actual = api.pci(device)
+ if not same_pci(actual, address):
+ raise ProbeError('cuDeviceGetPCIBusId', detail=f'Expected {address}, got {actual}')
+ name, total = C.create_string_buffer(256), C.c_size_t()
+ api.call('cuDeviceGetName', [P, I, I], name, len(name), device)
+ api.call('cuDeviceTotalMem_v2', [P, I], C.byref(total), device)
+ api.call('cuDriverGetVersion', [P], C.byref(driver))
+ checks.append({'test': 'cuda.device', 'status': 'passed', 'pci_address': actual,
+ 'name': name.value.decode(errors='replace'), 'total_memory_bytes': total.value,
+ 'driver_api_version': driver.value})
+ # A private context cannot reset, unload or synchronize another application's
+ # context. Explicit _v2 symbols avoid the incompatible cuCtxCreate_v4 ABI.
+ context, module, memory = P(), P(), C.c_uint64()
+ try:
+ event(stage='cuda.context')
+ api.call('cuCtxCreate_v2', [P, U, I], C.byref(context), 4, device)
+ event(stage='cuda.memory')
+ original = [(i * 17 + 3) for i in range(N)]
+ source, output = (U * N)(*original), (U * N)()
+ size = C.sizeof(source)
+ api.call('cuMemAlloc_v2', [P, C.c_size_t], C.byref(memory), size)
+ api.call('cuMemcpyHtoD_v2', [C.c_uint64, P, C.c_size_t], memory, source, size)
+ api.call('cuMemcpyDtoH_v2', [P, C.c_uint64, C.c_size_t], output, memory, size)
+ if list(output) != original:
+ raise ProbeError('memory_round_trip', detail='GPU memory readback differs from the uploaded pattern')
+ checks.append({'test': 'cuda.memory', 'status': 'passed', 'bytes': size})
+ event(stage='cuda.compute')
+ api.call('cuModuleLoadData', [P, C.c_char_p], C.byref(module), PTX)
+ kernel = P()
+ api.call('cuModuleGetFunction', [P, P, C.c_char_p], C.byref(kernel), module, b'harness_check')
+ parameters = (P * 1)(C.cast(C.byref(memory), P))
+ api.call('cuLaunchKernel', [P, U, U, U, U, U, U, U, P, P, P],
+ kernel, N // 256, 1, 1, 256, 1, 1, 0, None, parameters, None)
+ api.call('cuCtxSynchronize', [])
+ api.call('cuMemcpyDtoH_v2', [P, C.c_uint64, C.c_size_t], output, memory, size)
+ expected = [value * 3 + 7 for value in original]
+ mismatch = next((i for i in range(N) if output[i] != expected[i]), None)
+ if mismatch is not None:
+ raise ProbeError('integer_vector', detail=f'Element {mismatch}: expected {expected[mismatch]}, got {output[mismatch]}')
+ checks.append({'test': 'cuda.compute', 'status': 'passed', 'elements': N, 'operation': '3*x+7'})
+ finally:
+ # A context owns these allocations. Even if a cleanup call fails, always
+ # attempt the remaining releases; the worker then exits entirely.
+ for name, types, value in [('cuMemFree_v2', [C.c_uint64], memory),
+ ('cuModuleUnload', [P], module), ('cuCtxDestroy_v2', [P], context)]:
+ if value.value:
+ try:
+ api.call(name, types, value)
+ except ProbeError as error:
+ checks.append({'test': 'cuda.cleanup', 'status': 'failed',
+ 'operation': error.operation, 'code': error.code, 'detail': error.detail})
+
+
+def extensions(value):
+ return set(value.decode().split()) if value else set()
+
+
+def device_matches(device, device_string, extension, address):
+ ext = extensions(device_string(device, 0x3055))
+ if 'EGL_EXT_device_drm_render_node' in ext:
+ node = device_string(device, 0x3377)
+ if node and same_pci((Path('/sys/class/drm') / Path(node.decode()).name / 'device').resolve().name, address):
+ return True
+ # Headless cards need not expose a render node. The official EGL/CUDA mapping
+ # still identifies the exact PCI GPU; an EGL ordinal alone is not its identity.
+ if 'EGL_NV_device_cuda' in ext:
+ ordinal = C.c_ssize_t()
+ device_attribute = extension('eglQueryDeviceAttribEXT', U, P, I, P)
+ if device_attribute(device, 0x323A, C.byref(ordinal)):
+ api = CUDA()
+ api.call('cuInit', [U], 0)
+ return same_pci(api.pci(ordinal.value), address)
+ return False
+
+
+def accelerated_renderer(vendor, renderer):
+ return bool('nvidia' in vendor.lower() and renderer and not any(
+ word in renderer.lower() for word in ['llvmpipe', 'softpipe', 'software']))
+
+
+def graphics(address, checks):
+ event(stage='graphics.device')
+ egl = C.CDLL('libEGL.so.1')
+ get_proc = function(egl, 'eglGetProcAddress', P, C.c_char_p)
+ get_error = function(egl, 'eglGetError', I)
+
+ def extension(name, result, *args):
+ pointer = get_proc(name.encode())
+ if not pointer:
+ raise ProbeError(name, detail='EGL entry point is unavailable')
+ return C.CFUNCTYPE(result, *args)(pointer)
+
+ def call(name, result, types, *args):
+ event(operation=name)
+ value = function(egl, name, result, *types)(*args)
+ if not value:
+ raise ProbeError(name, get_error(), 'EGL operation failed')
+ return value
+
+ query = function(egl, 'eglQueryString', C.c_char_p, P, I)
+ client = extensions(query(None, 0x3055)) # EGL_EXTENSIONS
+ if not {'EGL_EXT_device_enumeration', 'EGL_EXT_device_query', 'EGL_EXT_platform_device'} <= client:
+ raise ProbeError('EGL extensions', detail='EGL device selection is unavailable')
+ enumerate_devices = extension('eglQueryDevicesEXT', U, I, P, P)
+ device_string = extension('eglQueryDeviceStringEXT', C.c_char_p, P, I)
+ devices, count = (P * 64)(), I()
+ if not enumerate_devices(len(devices), devices, C.byref(count)) or not 0 < count.value <= len(devices):
+ raise ProbeError('eglQueryDevicesEXT', get_error(), 'No usable EGL device inventory')
+ selected = None
+ for device in devices[:count.value]:
+ if device_matches(device, device_string, extension, address):
+ selected = device
+ break
+ if selected is None:
+ raise ProbeError('graphics.device', detail='No EGL device maps to this PCI GPU')
+ platform = extension('eglGetPlatformDisplayEXT', P, U, P, P)
+ display = platform(0x313F, selected, None) # EGL_PLATFORM_DEVICE_EXT
+ if not display:
+ raise ProbeError('eglGetPlatformDisplayEXT', get_error(), 'Device display is unavailable')
+ context = surface = None
+ initialized = False
+ try:
+ major, minor = I(), I()
+ call('eglInitialize', U, [P, P, P], display, C.byref(major), C.byref(minor))
+ initialized = True
+ call('eglBindAPI', U, [U], 0x30A0) # EGL_OPENGL_ES_API
+ attributes = (I * 13)(0x3033, 1, 0x3040, 4, 0x3024, 8, 0x3023, 8, 0x3022, 8, 0x3021, 8, 0x3038)
+ config, found = P(), I()
+ call('eglChooseConfig', U, [P, P, P, I, P], display, attributes, C.byref(config), 1, C.byref(found))
+ if found.value != 1:
+ raise ProbeError('eglChooseConfig', detail='RGBA8 ES2 pbuffer is unavailable')
+ surface = call('eglCreatePbufferSurface', P, [P, P, P], display, config,
+ (I * 5)(0x3057, 8, 0x3056, 8, 0x3038))
+ context = call('eglCreateContext', P, [P, P, P, P], display, config, None,
+ (I * 3)(0x3098, 2, 0x3038))
+ call('eglMakeCurrent', U, [P, P, P, P], display, surface, surface, context)
+ gl = C.CDLL('libGLESv2.so.2')
+ string = function(gl, 'glGetString', C.c_char_p, U)
+ vendor, renderer = (string(token) for token in [0x1F00, 0x1F01])
+ vendor = vendor.decode(errors='replace') if vendor else ''
+ renderer = renderer.decode(errors='replace') if renderer else ''
+ if not accelerated_renderer(vendor, renderer):
+ raise ProbeError('graphics.renderer', detail=f'Expected NVIDIA acceleration, got {vendor}: {renderer}')
+ event(stage='graphics.render')
+ clear_color = function(gl, 'glClearColor', None, *[C.c_float] * 4)
+ clear = function(gl, 'glClear', None, U)
+ read_pixels = function(gl, 'glReadPixels', None, I, I, I, I, U, U, P)
+ gl_error = function(gl, 'glGetError', U)
+ pixel = (C.c_ubyte * 4)()
+ for color in [(255, 0, 0, 255), (0, 255, 0, 255), (0, 0, 255, 255)]:
+ clear_color(*(value / 255 for value in color))
+ clear(0x4000) # GL_COLOR_BUFFER_BIT
+ read_pixels(0, 0, 1, 1, 0x1908, 0x1401, pixel) # RGBA, UNSIGNED_BYTE
+ code = gl_error()
+ if code or tuple(pixel) != color:
+ raise ProbeError('graphics.readback', code, f'Expected {color}, got {tuple(pixel)}')
+ checks.append({'test': 'graphics.render', 'status': 'passed', 'vendor': vendor,
+ 'renderer': renderer, 'surface': '8x8 offscreen EGL pbuffer', 'colors': 3})
+ finally:
+ if initialized:
+ function(egl, 'eglMakeCurrent', U, P, P, P, P)(display, None, None, None)
+ if context:
+ function(egl, 'eglDestroyContext', U, P, P)(display, context)
+ if surface:
+ function(egl, 'eglDestroySurface', U, P, P)(display, surface)
+ function(egl, 'eglTerminate', U, P)(display)
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument('probe', choices=['cuda', 'graphics'])
+ parser.add_argument('address')
+ args = parser.parse_args()
+ if not re.fullmatch(r'[0-9a-f]{4,8}:[0-9a-f]{2}:[0-9a-f]{2}\.[0-7]', args.address):
+ parser.error('Expected a PCI address.')
+ checks = []
+ try:
+ (compute if args.probe == 'cuda' else graphics)(args.address, checks)
+ except (ProbeError, OSError, AttributeError) as error:
+ checks.append({'test': args.probe, 'status': 'failed',
+ 'operation': getattr(error, 'operation', 'load_driver'),
+ 'code': getattr(error, 'code', None), 'detail': str(error)})
+ event(result={'address': args.address, 'probe': args.probe, 'checks': checks,
+ 'status': 'failed' if any(row['status'] == 'failed' for row in checks) else 'passed'})
+
+
+if __name__ == '__main__':
+ main()
diff --git a/os/hardware.py b/os/hardware.py
index 1be5555ea..d97e3ae50 100644
--- a/os/hardware.py
+++ b/os/hardware.py
@@ -3,6 +3,7 @@
import argparse
import fcntl
import hashlib
+import importlib.util
import json
import os
from pathlib import Path
@@ -109,7 +110,15 @@ def report(sysfs=Path('/sys'), proc=Path('/proc')):
'installation_blocker': installation_blocker(sysfs),
'backlights': [p.name for p in sorted((sysfs / 'class/backlight').glob('*'))],
'broadcom_bundle_available': (BUNDLE / 'manifest.json').is_file(),
- 'nvidia_bundle_available': (NVIDIA_BUNDLE / 'manifest.json').is_file()}
+ 'nvidia_bundle_available': (NVIDIA_BUNDLE / 'manifest.json').is_file(),
+ 'gpu_health': gpu_health().cached_report() if Path(__file__).with_name('gpu_health.py').is_file() else None}
+
+
+def gpu_health():
+ spec = importlib.util.spec_from_file_location('harness_gpu_health', Path(__file__).with_name('gpu_health.py'))
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
def nvidia_bundle_manifest(folder, all_files=False):
@@ -331,7 +340,8 @@ def configure_install(target, devices=None, bundle=BUNDLE):
def main():
parser = argparse.ArgumentParser(description=__doc__)
sub = parser.add_subparsers(dest='command')
- sub.add_parser('report')
+ reporting = sub.add_parser('report')
+ reporting.add_argument('--check-gpu', action='store_true', help='Repeat GPU verification as the current user')
activation = sub.add_parser('activate')
activation.add_argument('address')
installation = sub.add_parser('configure-install')
@@ -351,6 +361,10 @@ def main():
elif args.command == 'configure-install':
result = configure_install(args.target)
else:
+ if getattr(args, 'check_gpu', False):
+ if not Path(__file__).with_name('gpu_health.py').is_file():
+ parser.error('GPU verification is not available in this system profile.')
+ gpu_health().check(force=True)
result = report()
print(json.dumps(result, indent=2))
diff --git a/os/root/usr/lib/harness-os/session b/os/root/usr/lib/harness-os/session
index b139146f1..f4a0797ca 100644
--- a/os/root/usr/lib/harness-os/session
+++ b/os/root/usr/lib/harness-os/session
@@ -46,6 +46,9 @@ if [ -f /etc/harness-live ]; then
exec sudo /usr/bin/python3 /usr/lib/harness-os/install.py --boot
fi
systemctl --user start harness-daemon.service || true
+if [ -f /usr/lib/harness-os/gpu_health.py ]; then
+ systemctl --user start harness-gpu-check.timer || true
+fi
printf '%s\n' 'Graphics unavailable; continuing in hn on the console.'
cd "$HOME/projects"
while :; do
diff --git a/os/root/usr/lib/systemd/user/harness-gpu-check.service b/os/root/usr/lib/systemd/user/harness-gpu-check.service
new file mode 100644
index 000000000..cad25fdcb
--- /dev/null
+++ b/os/root/usr/lib/systemd/user/harness-gpu-check.service
@@ -0,0 +1,16 @@
+[Unit]
+Description=Verify GPU readiness without interrupting work
+ConditionPathExists=!/etc/harness-live
+PartOf=harness-os.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/python3 /usr/lib/harness-os/gpu_health.py --notify
+TimeoutStartSec=150
+TimeoutStopSec=3
+Nice=10
+LimitCORE=0
+# This is the active user's GPU access, not a privileged diagnostic. Keep normal
+# device permissions and avoid any reset, module unloading or session restart.
+# Leave the driver's standard nvidia-modprobe helper usable, just as it is for
+# an ordinary user process; this check adds no privileged helper of its own.
diff --git a/os/root/usr/lib/systemd/user/harness-gpu-check.timer b/os/root/usr/lib/systemd/user/harness-gpu-check.timer
new file mode 100644
index 000000000..b602ef526
--- /dev/null
+++ b/os/root/usr/lib/systemd/user/harness-gpu-check.timer
@@ -0,0 +1,9 @@
+[Unit]
+Description=Check GPU readiness after the workspace starts
+ConditionPathExists=!/etc/harness-live
+PartOf=harness-os.target
+
+[Timer]
+OnActiveSec=15
+AccuracySec=1
+Unit=harness-gpu-check.service
diff --git a/os/root/usr/lib/systemd/user/harness-os.target b/os/root/usr/lib/systemd/user/harness-os.target
index 7fe9b7ac9..96856b3b9 100644
--- a/os/root/usr/lib/systemd/user/harness-os.target
+++ b/os/root/usr/lib/systemd/user/harness-os.target
@@ -1,4 +1,4 @@
[Unit]
Description=The hn screen
-Wants=hn-screen.service harness-install.service harness-idle.service harness-update.timer
+Wants=hn-screen.service harness-install.service harness-idle.service harness-update.timer harness-gpu-check.timer
After=graphical-session-pre.target
diff --git a/os/root/usr/share/harness-os/AGENTS.md b/os/root/usr/share/harness-os/AGENTS.md
index 87e567621..d9ae95338 100644
--- a/os/root/usr/share/harness-os/AGENTS.md
+++ b/os/root/usr/share/harness-os/AGENTS.md
@@ -49,6 +49,24 @@ is required.
## Network and hardware
+- GPU verification runs briefly after the installed workspace starts. Success
+ is quiet. Read `harness hardware` → `gpu_health` before changing NVIDIA drivers;
+ `harness hardware --check-gpu` repeats the checks as the current user. Do not run
+ it with sudo: ordinary session device access is part of the check. Each PCI GPU
+ has separate binding, memory, computation and offscreen graphics results, with
+ the exact failed API and error. `stale: true` means the saved result no longer
+ describes this boot/driver. Unavailable or skipped checks are not passes.
+- These are small readiness checks, not full VRAM, display, browser, sleep or
+ model-workload validation. A headless GPU can pass compute while graphics stays
+ explicitly unverified. Passthrough devices are never touched. Reports stay in
+ `~/.local/state/harness-os/gpu/`; no hardware data is uploaded.
+- Updates retain the original Btrfs root and matching boot checkpoint. GPU
+ diagnostics run again after reboot, never by resetting a GPU used by agents.
+ A failure does not automatically roll back or reboot. Use the reported update
+ checkpoint with the existing offline recovery procedure when appropriate;
+ projects remain in the separate home subvolume. Local models belong to the
+ TUI's existing local-model workflow, not this hardware check.
+
- `harness hardware` reports the model, CPU baseline, PCI devices, bound drivers
and backlights. It does not collect serial numbers, Wi-Fi names, MAC addresses
or passwords. Use actual device IDs when diagnosing hardware.
diff --git a/os/tests/gpu_cuda_fixture.c b/os/tests/gpu_cuda_fixture.c
new file mode 100644
index 000000000..e6475242b
--- /dev/null
+++ b/os/tests/gpu_cuda_fixture.c
@@ -0,0 +1,69 @@
+/* ABI test double, never a GPU acceptance result or product payload. */
+#include <stdint.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdio.h>
+static unsigned int values[1024];
+static int mode, allocations, contexts, modules;
+void fixture_mode(int value) { mode = value; }
+int fixture_live(void) { return allocations + contexts + modules; }
+int cuGetErrorName(int code, const char **name) { (void)code; *name = "CUDA_FIXTURE_ERROR"; return 0; }
+int cuInit(unsigned int flags) { return flags ? 1 : 0; }
+int cuDeviceGetByPCIBusId(int *device, const char *address) {
+ if (strcmp(address, "0000:01:00.0")) return 101;
+ *device = 3; return 0;
+}
+int cuDeviceGetPCIBusId(char *address, int size, int device) {
+ if (device != 3) return 101;
+ snprintf(address, size, "%s", mode == 4 ? "0000:02:00.0" : "0000:01:00.0"); return 0;
+}
+int cuDeviceGetName(char *name, int size, int device) { (void)device; snprintf(name, size, "Fixture only"); return 0; }
+int cuDeviceTotalMem_v2(size_t *bytes, int device) { (void)device; *bytes = (size_t)24 << 30; return 0; }
+int cuDriverGetVersion(int *version) { *version = 13040; return 0; }
+int cuCtxCreate_v2(void **context, unsigned int flags, int device) {
+ if (flags != 4 || device != 3) return 1;
+ *context = (void *)(uintptr_t)0x100000004ULL; ++contexts; return 0;
+}
+int cuMemAlloc_v2(uint64_t *memory, size_t bytes) {
+ if (bytes != sizeof(values)) return 1;
+ *memory = (uint64_t)(uintptr_t)values; ++allocations; return 0;
+}
+int cuMemcpyHtoD_v2(uint64_t device, const void *host, size_t bytes) {
+ if (device != (uint64_t)(uintptr_t)values || bytes != sizeof(values)) return 1;
+ memcpy(values, host, bytes); return 0;
+}
+int cuMemcpyDtoH_v2(void *host, uint64_t device, size_t bytes) {
+ if (device != (uint64_t)(uintptr_t)values || bytes != sizeof(values)) return 1;
+ memcpy(host, values, bytes); if (mode == 1) ((unsigned int *)host)[17] ^= 1; return 0;
+}
+int cuModuleLoadData(void **module, const void *ptx) {
+ if (!strstr(ptx, ".visible .entry harness_check")) return 218;
+ *module = (void *)(uintptr_t)0x100000008ULL; ++modules; return 0;
+}
+int cuModuleGetFunction(void **function, void *module, const char *name) {
+ if (module != (void *)(uintptr_t)0x100000008ULL || strcmp(name, "harness_check")) return 1;
+ *function = (void *)(uintptr_t)0x10000000cULL; return 0;
+}
+int cuLaunchKernel(void *function, unsigned int gx, unsigned int gy, unsigned int gz,
+ unsigned int bx, unsigned int by, unsigned int bz, unsigned int shared,
+ void *stream, void **parameters, void **extra) {
+ if (function != (void *)(uintptr_t)0x10000000cULL || gx != 4 || gy != 1 || gz != 1 ||
+ bx != 256 || by != 1 || bz != 1 || shared || stream || extra ||
+ *(uint64_t *)parameters[0] != (uint64_t)(uintptr_t)values) return 1;
+ for (int i = 0; i < 1024; ++i) values[i] = values[i] + values[i] + values[i] + 7;
+ if (mode == 2) values[513] += 1;
+ return 0;
+}
+int cuCtxSynchronize(void) { return mode == 3 ? 700 : 0; }
+int cuMemFree_v2(uint64_t memory) {
+ if (memory != (uint64_t)(uintptr_t)values) return 1;
+ --allocations; return 0;
+}
+int cuModuleUnload(void *module) {
+ if (module != (void *)(uintptr_t)0x100000008ULL) return 1;
+ --modules; return 0;
+}
+int cuCtxDestroy_v2(void *context) {
+ if (context != (void *)(uintptr_t)0x100000004ULL) return 1;
+ --contexts; return 0;
+}
diff --git a/os/tests/gpu_egl_native.py b/os/tests/gpu_egl_native.py
new file mode 100644
index 000000000..16052fe3d
--- /dev/null
+++ b/os/tests/gpu_egl_native.py
@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+"""Real Mesa EGL ABI exercise; never evidence for NVIDIA hardware acceleration."""
+import contextlib
+import importlib.util
+import io
+import json
+from pathlib import Path
+from unittest.mock import patch
+
+spec = importlib.util.spec_from_file_location('probe', Path(__file__).parents[1] / 'gpu_probe.py')
+probe = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(probe)
+
+
+def main():
+ records = []
+ # Hosted Linux has Mesa's headless software EGL device. Select it explicitly
+ # for an API/cleanup/readback check; keep its true vendor and renderer in the
+ # receipt. This does not add a software fallback to the product.
+ with patch.object(probe, 'device_matches', return_value=True), contextlib.redirect_stdout(io.StringIO()):
+ try:
+ probe.graphics('0000:00:00.0', [])
+ except probe.ProbeError as error:
+ assert error.operation == 'graphics.renderer', str(error)
+ records.append({'check': 'software-renderer-rejected', 'detail': error.detail})
+ else:
+ raise AssertionError('Software graphics incorrectly passed NVIDIA verification')
+ results = []
+ with patch.object(probe, 'accelerated_renderer', return_value=True):
+ probe.graphics('0000:00:00.0', results)
+ assert len(results) == 1 and results[0]['status'] == 'passed'
+ assert not probe.accelerated_renderer(results[0]['vendor'], results[0]['renderer'])
+ records.append({'check': 'real-egl-api-and-color-readback', 'device': results[0]})
+ print(json.dumps({'status': 'passed', 'checks': records,
+ 'limits': 'Software Mesa with explicit test-only device selection; NVIDIA hardware is unverified.'}, indent=2))
+
+
+if __name__ == '__main__':
+ main()
diff --git a/os/tests/gpu_health_vm.py b/os/tests/gpu_health_vm.py
new file mode 100644
index 000000000..05e60b36b
--- /dev/null
+++ b/os/tests/gpu_health_vm.py
@@ -0,0 +1,162 @@
+#!/usr/bin/env python3
+"""Installed OS diagnostic integration; virtual display, no physical NVIDIA GPU."""
+import argparse
+import hashlib
+import json
+import os
+from pathlib import Path
+import re
+import shlex
+import subprocess
+import time
+
+from footprint_vm import copy_file
+from hardware_update_vm import graceful_stop, verify_image
+from vm import VM, check_graphical_keyboard
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument('--iso', type=Path, required=True)
+ parser.add_argument('--output', type=Path, required=True)
+ args = parser.parse_args()
+ iso, folder = args.iso.resolve(), args.output.absolute()
+ image = verify_image(iso, json.loads(Path(__file__).with_name('hardware-update.lock.json').read_text()))
+ if not os.access('/dev/kvm', os.R_OK | os.W_OK):
+ parser.error('Native x86 KVM is required.')
+ source = subprocess.check_output(['git', 'rev-parse', 'HEAD'], text=True).strip()
+ if subprocess.check_output(['git', 'status', '--porcelain'], text=True).strip():
+ parser.error('Commit the candidate before native acceptance.')
+ folder.mkdir(parents=True, exist_ok=False)
+ root = Path(__file__).resolve().parents[1]
+ files = {name: 'usr/lib/harness-os/' + name for name in ['gpu_health.py', 'gpu_probe.py', 'hardware.py']}
+ files.update({'root/' + name: name for name in ['usr/lib/harness-os/session',
+ *['usr/lib/systemd/user/' + unit for unit in ['harness-os.target', 'harness-gpu-check.service', 'harness-gpu-check.timer']]]})
+ record = dict(status='running', started_at=time.time(), source_commit=source, image=image['iso'],
+ image_source=image['source_commit'], files={name: hashlib.sha256((root / name).read_bytes()).hexdigest() for name in files},
+ checks=[], limits=['Candidate OS overlay on original preview14, not a newly built image.',
+ 'Virtual display with no NVIDIA device. This proves service/reporting behavior, not GPU hardware support.'])
+ vm = VM(folder, iso, 'uefi', 2048, cpu='Nehalem')
+ config = dict(disk='/dev/vda', expected_serial='HN_OS_TEST', confirm_erase='/dev/vda',
+ username='me', hostname='harness', password='test-password-123', encrypt=True, serial_console=True)
+
+ def auth():
+ vm.command('printf %s ' + shlex.quote(config['password'] + '\n') + ' | sudo -S -v')
+
+ try:
+ vm.start(live=True)
+ vm.monitor('set_link', name='hnnet', up=False)
+ vm.wait(r'root@[^\r\n]*[#] ')
+ vm.shell_ready = True
+ vm.command('stty -echo; nmcli networking off')
+ copy_file(vm, json.dumps(config).encode(), '/tmp/install.json')
+ output, _ = vm.command('harness install --config /tmp/install.json --yes-erase-disk', timeout=360)
+ (folder / 'install.log').write_text(output)
+ graceful_stop(vm, False)
+ vm.start(live=False)
+ vm.login_installed(config)
+ auth()
+ vm.command('systemctl --user mask --now harness-update.timer harness-update.service')
+ for index, (local, target) in enumerate(files.items()):
+ data = (root / local).read_bytes()
+ temporary = '/tmp/gpu-overlay-' + str(index)
+ copy_file(vm, data, temporary)
+ vm.command('printf %s ' + shlex.quote(record['files'][local] + ' ' + temporary + '\n') + ' | sha256sum -c -')
+ vm.command('sudo -n install -m ' + ('755' if target.endswith('/session') else '644') + ' ' + temporary + ' /' + target)
+ # No GPU readiness check belongs in the boot critical path. Activate the
+ # package's real target by cold boot, not by calling its Python entrypoint.
+ graceful_stop(vm, True)
+ vm.start(live=False)
+ vm.login_installed(config)
+ path = '/home/me/.local/state/harness-os/gpu/health.json'
+ vm.command('timeout 40 sh -c ' + shlex.quote('until test -f ' + path + '; do sleep .2; done'), timeout=45)
+ before = json.loads(vm.read_file(path))
+ assert before['status'] == 'skipped' and before['devices'] == []
+ assert before['duration_seconds'] < 3, 'No-GPU check performed unnecessary work'
+ assert before['inventory']['boot_id'] == vm.read_file('/proc/sys/kernel/random/boot_id').decode().strip()
+ (folder / 'first-check.json').write_text(json.dumps(before, indent=2) + '\n')
+ check_graphical_keyboard(vm, 'gpu-health-keyboard')
+ vm.command('hn display-message -p "#{pane_id}" > /tmp/gpu-original-pane')
+ original_pane = vm.read_file('/tmp/gpu-original-pane').decode().strip()
+ assert re.fullmatch(r'%\d+', original_pane), 'Expected the existing shell pane'
+ # The reference client restores terminal shells after a cold boot; it
+ # does not replay an arbitrary prior foreground command. Start a real
+ # bundled agent explicitly before asserting that checks preserve it.
+ vm.command("mkdir -p ~/projects/gpu-health-proof && hn new-window -n gpu-proof " +
+ shlex.quote('cd "$HOME/projects/gpu-health-proof" && exec /usr/bin/opencode'))
+ vm.command('timeout 40 sh -c ' + shlex.quote('until pgrep -u 1000 -x opencode >/dev/null; do sleep .2; done'), timeout=45)
+ deadline = time.monotonic() + 40
+ while True:
+ output, _ = vm.command('hn capture-pane -p')
+ if 'opencode' in output.lower() and 'Ask anything' in output:
+ break
+ if time.monotonic() > deadline:
+ raise TimeoutError('The bundled OpenCode did not become interactive')
+ time.sleep(.5)
+ (folder / 'agent-before-check.txt').write_text(output)
+ vm.screenshot('agent-before-check')
+ # Existing work must remain alive while checks and reports run.
+ vm.command('pgrep -x opencode > /tmp/gpu-agent-pids && systemctl --user show harness-daemon -p MainPID --value > /tmp/gpu-daemon-pid')
+ vm.command('systemctl --user start harness-gpu-check.service')
+ assert json.loads(vm.read_file(path)) == before, 'A second start repeated the same-boot check'
+ vm.command('harness hardware > /tmp/gpu-hardware.json')
+ report = json.loads(vm.read_file('/tmp/gpu-hardware.json'))
+ assert report['gpu_health']['stale'] is False and report['gpu_health']['status'] == 'skipped'
+ # Test failure-message routing with an explicit diagnostic fixture, not
+ # a fabricated GPU pass. The product source and real report stay intact.
+ notice = ('import sys; sys.path.insert(0,"/usr/lib/harness-os"); import gpu_health; '
+ 'assert gpu_health.notify({"status":"failed"})')
+ vm.command('python3 -c ' + shlex.quote(notice))
+ vm.screenshot('failure-message')
+ output, _ = vm.command('hn show-messages')
+ (folder / 'notification.log').write_text(output)
+ assert 'GPU needs attention' in output
+ vm.command('pgrep -x opencode | diff /tmp/gpu-agent-pids - && systemctl --user show harness-daemon -p MainPID --value | diff /tmp/gpu-daemon-pid -')
+ # The shared keyboard fixture waits for a visible shell prompt. OpenCode
+ # is now active in its own tab, so select the retained shell first.
+ vm.command('hn select-window -t ' + original_pane + ' && hn select-pane -t ' + original_pane)
+ check_graphical_keyboard(vm, 'gpu-health-after-check')
+ record['checks'].append('Packaged timer cold-boots, skips absent hardware quickly, caches within the boot, exposes a fresh JSON report, and routes a fixture failure without replacing the agent or daemon.')
+ auth()
+ graceful_stop(vm, True)
+ vm.start(live=False)
+ vm.login_installed(config)
+ vm.command('timeout 40 sh -c ' + shlex.quote('until systemctl --user show harness-gpu-check.service -p Result --value | grep -qx success && test -f ' + path + '; do sleep .2; done'), timeout=45)
+ # Result=success is also the service's initial state: wait for the actual
+ # report to carry this boot ID before claiming the second execution.
+ expected_boot = vm.read_file('/proc/sys/kernel/random/boot_id').decode().strip()
+ deadline = time.monotonic() + 35
+ while True:
+ after = json.loads(vm.read_file(path))
+ if after['inventory']['boot_id'] == expected_boot:
+ break
+ if time.monotonic() > deadline:
+ raise TimeoutError('New boot did not run GPU verification')
+ time.sleep(.5)
+ assert after['inventory']['boot_id'] != before['inventory']['boot_id'] and after['status'] == 'skipped'
+ (folder / 'second-check.json').write_text(json.dumps(after, indent=2) + '\n')
+ record['checks'].append('A second real encrypted cold boot rechecks automatically with a new boot identity.')
+ auth()
+ output, _ = vm.command('journalctl --user -u harness-gpu-check.service --no-pager')
+ (folder / 'service.log').write_text(output)
+ graceful_stop(vm, True)
+ record['status'] = 'passed'
+ except Exception as error:
+ record.update(status='failed', error=repr(error))
+ if vm.process and vm.process.poll() is None:
+ try:
+ vm.screenshot('failure')
+ except Exception as screenshot_error:
+ record['screenshot_error'] = repr(screenshot_error)
+ raise
+ finally:
+ vm.stop()
+ vm.log.close()
+ vm.stderr.close()
+ vm.control.cleanup()
+ record['duration_seconds'] = round(time.time() - record['started_at'], 3)
+ (folder / 'receipt.json').write_text(json.dumps(record, indent=2) + '\n')
+
+
+if __name__ == '__main__':
+ main()
diff --git a/os/tests/gpu_ptx_native.py b/os/tests/gpu_ptx_native.py
new file mode 100644
index 000000000..6132369d8
--- /dev/null
+++ b/os/tests/gpu_ptx_native.py
@@ -0,0 +1,80 @@
+#!/usr/bin/env python3
+"""Assemble the shipped PTX for Turing, Ada and Blackwell; no GPU execution claim."""
+import argparse
+import hashlib
+import importlib.util
+import json
+from pathlib import Path
+import subprocess
+import time
+from urllib.request import urlopen
+import zipfile
+
+URL = ('https://files.pythonhosted.org/packages/25/48/b54a06168a2190572a312bfe4ce443687773eb61367ced31e064953dd2f7/'
+ 'nvidia_cuda_nvcc_cu12-12.9.86-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl')
+SHA256 = '5d6a0d32fdc7ea39917c20065614ae93add6f577d840233237ff08e9a38f58f0'
+BYTES = 40546229
+
+
+def digest(path):
+ with path.open('rb') as handle:
+ return hashlib.file_digest(handle, 'sha256').hexdigest()
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument('--output', type=Path, required=True)
+ args = parser.parse_args()
+ output = args.output.absolute()
+ output.mkdir(parents=True, exist_ok=False)
+ # Test-host tooling only. Neither the compiler nor its outputs are included
+ # in the OS payload. Select one exact archive member, never extract a wheel.
+ tools = output.parent / ('ptx-tools-' + str(time.time_ns()))
+ tools.mkdir()
+ wheel = tools / 'nvcc.whl'
+ with urlopen(URL, timeout=30) as response, wheel.open('wb') as handle:
+ count = 0
+ while chunk := response.read(128 * 1024):
+ count += len(chunk)
+ if count > BYTES:
+ raise ValueError('Compiler wheel exceeds its pinned size')
+ handle.write(chunk)
+ if wheel.stat().st_size != BYTES or digest(wheel) != SHA256:
+ raise ValueError('Compiler wheel checksum mismatch')
+ with zipfile.ZipFile(wheel) as archive:
+ compiler = tools / 'ptxas'
+ compiler.write_bytes(archive.read('nvidia/cuda_nvcc/bin/ptxas'))
+ compiler.chmod(0o755)
+ source = Path(__file__).parents[1] / 'gpu_probe.py'
+ spec = importlib.util.spec_from_file_location('probe', source)
+ probe = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(probe)
+ ptx = output / 'harness.ptx'
+ ptx.write_bytes(probe.PTX)
+ version = subprocess.check_output([str(compiler), '--version'], text=True, timeout=10)
+ record = dict(status='running', wheel={'url': URL, 'bytes': BYTES, 'sha256': SHA256},
+ compiler={'version': version, 'sha256': digest(compiler)},
+ probe_sha256=digest(source), ptx_sha256=digest(ptx), targets=[],
+ limits='Offline NVIDIA assembler only; device execution and driver JIT remain unverified.')
+ for target in ['sm_75', 'sm_89', 'sm_120']:
+ binary = output / (target + '.cubin')
+ result = subprocess.run([str(compiler), '--gpu-name=' + target, str(ptx), '-o', str(binary)],
+ text=True, capture_output=True, timeout=30)
+ (output / (target + '.log')).write_text(result.stdout + result.stderr)
+ result.check_returncode()
+ if binary.read_bytes()[:4] != b'\x7fELF':
+ raise ValueError('Assembler did not produce a device binary')
+ record['targets'].append({'target': target, 'bytes': binary.stat().st_size, 'sha256': digest(binary)})
+ invalid = tools / 'invalid.ptx'
+ invalid.write_bytes(probe.PTX.replace(b'mad.lo.u32 v, v, 3, 7;', b'not_a_ptx_instruction;'))
+ rejected = subprocess.run([str(compiler), '--gpu-name=sm_89', str(invalid), '-o', str(tools / 'invalid.cubin')],
+ capture_output=True, text=True, timeout=30)
+ if rejected.returncode == 0:
+ raise ValueError('The compiler did not reject intentionally invalid PTX')
+ record.update(status='passed', negative_control={'returncode': rejected.returncode, 'stderr': rejected.stderr})
+ (output / 'receipt.json').write_text(json.dumps(record, indent=2) + '\n')
+ print(json.dumps(record, indent=2))
+
+
+if __name__ == '__main__':
+ main()
diff --git a/os/tests/test_gpu_health.py b/os/tests/test_gpu_health.py
new file mode 100644
index 000000000..398362967
--- /dev/null
+++ b/os/tests/test_gpu_health.py
@@ -0,0 +1,160 @@
+"""GPU health failures must be visible and cannot masquerade as acceptance."""
+import copy
+import importlib.util
+import json
+from pathlib import Path
+import subprocess
+import tempfile
+import unittest
+from unittest.mock import patch
+
+spec = importlib.util.spec_from_file_location('gpu_health', Path(__file__).parents[1] / 'gpu_health.py')
+health = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(health)
+
+
+def card(address='0000:01:00.0', driver='nvidia', override=None):
+ return dict(address=address, id='10de:2684', driver=driver, driver_override=override, **{'class': '030000'})
+
+
+def inventory(cards=None):
+ return dict(schema=1, boot_id='first', kernel='fixture', driver_version='615.71.09',
+ cards=[card()] if cards is None else cards, libraries={})
+
+
+def result(kind, address, timeout):
+ names = ['cuda.device', 'cuda.memory', 'cuda.compute'] if kind == 'cuda' else ['graphics.render']
+ return dict(probe=kind, address=address, status='passed', checks=[dict(test=name, status='passed') for name in names])
+
+
+class HealthTests(unittest.TestCase):
+ def setUp(self):
+ self.temp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.temp.cleanup)
+ self.root = Path(self.temp.name)
+ for name, value in [('STATE', self.root / 'state'), ('SYSTEM_STATE', self.root / 'system')]:
+ value.mkdir()
+ patcher = patch.object(health, name, value)
+ patcher.start()
+ self.addCleanup(patcher.stop)
+ self.connected = patch.object(health, 'connections', return_value={'card0-DP-1': 'connected'})
+ self.connected.start()
+ self.addCleanup(self.connected.stop)
+
+ def test_no_gpu_and_passthrough_never_load_cuda_or_notify(self):
+ for cards in [[], [card(driver='vfio-pci')], [card(driver=None, override='vfio-pci')],
+ [card(driver=None, override='pci-stub')]]:
+ with self.subTest(cards=cards), patch.object(health, 'inventory', return_value=inventory(cards)), \
+ patch.object(health, 'probe', side_effect=AssertionError('GPU was touched')):
+ report = health.check(force=True)
+ self.assertEqual(report['status'], 'skipped')
+ with patch.object(health.subprocess, 'run') as run:
+ self.assertFalse(health.notify(report))
+ run.assert_not_called()
+
+ def test_every_gpu_is_checked_and_one_bad_card_fails_the_report(self):
+ second = '0000:02:00.0'
+ def broken(kind, address, timeout):
+ item = result(kind, address, timeout)
+ if address == second and kind == 'cuda':
+ item['status'] = 'failed'
+ item['checks'][-1] = dict(test='cuda.compute', status='failed', detail='wrong answer')
+ return item
+ with patch.object(health, 'probe', side_effect=broken) as probe:
+ report = health.evaluate(inventory([card(), card(second)]))
+ self.assertEqual(probe.call_count, 4)
+ self.assertEqual([row['status'] for row in report['devices']], ['passed', 'failed'])
+ self.assertEqual(report['status'], 'failed')
+ self.assertEqual(report['devices'][1]['checks'][3]['detail'], 'wrong answer')
+
+ def test_headless_graphics_unavailable_is_explicitly_partial_not_broken_compute(self):
+ def headless(kind, address, timeout):
+ if kind == 'graphics':
+ return dict(status='failed', checks=[dict(test='graphics.device', status='failed', operation='graphics.device', detail='No EGL device')])
+ return result(kind, address, timeout)
+ with patch.object(health, 'connections', return_value={}), patch.object(health, 'probe', side_effect=headless):
+ report = health.evaluate(inventory())
+ self.assertEqual(report['status'], 'partial')
+ self.assertEqual(report['devices'][0]['checks'][-1]['status'], 'not-tested')
+
+ def test_bad_graphics_readback_fails_even_without_a_monitor(self):
+ def bad_color(kind, address, timeout):
+ if kind == 'graphics':
+ return dict(status='failed', checks=[dict(test='graphics.render', status='failed', operation='graphics.readback', detail='Wrong pixel')])
+ return result(kind, address, timeout)
+ with patch.object(health, 'connections', return_value={}), patch.object(health, 'probe', side_effect=bad_color):
+ self.assertEqual(health.evaluate(inventory())['status'], 'failed')
+
+ def test_missing_managed_driver_is_failure_but_existing_unsupported_gpu_is_not_claimed(self):
+ current = inventory([card(driver='nouveau')])
+ with patch.object(health, 'probe') as probe:
+ self.assertEqual(health.evaluate(current)['status'], 'partial')
+ (health.SYSTEM_STATE / 'hardware.json').write_text(json.dumps({'nvidia': {'status': 'installed', 'devices': ['10de:2684']}}))
+ report = health.evaluate(current)
+ self.assertEqual(report['status'], 'failed')
+ self.assertEqual(report['devices'][0]['checks'][0]['test'], 'driver.binding')
+ probe.assert_not_called()
+
+ def test_cache_survives_screen_restart_but_boot_driver_device_or_test_change_rechecks(self):
+ current = inventory()
+ with patch.object(health, 'inventory', side_effect=lambda: copy.deepcopy(current)), patch.object(health, 'probe', side_effect=result) as probe:
+ original = health.check()
+ self.assertEqual(health.check(), original)
+ self.assertEqual(probe.call_count, 2)
+ for field, value in [('boot_id', 'second'), ('driver_version', '616.0'), ('kernel', 'next'), ('cards', [card('0000:03:00.0')]), ('schema', 2)]:
+ current[field] = value
+ health.check()
+ self.assertEqual(probe.call_count, 12)
+ self.assertFalse(health.cached_report()['stale'])
+ current['driver_version'] = 'new'
+ self.assertTrue(health.cached_report()['stale'])
+
+ def test_last_working_result_is_retained_after_failure(self):
+ with patch.object(health, 'inventory', return_value=inventory()), patch.object(health, 'probe', side_effect=result):
+ health.check()
+ good = (health.STATE / 'last-working.json').read_bytes()
+ with patch.object(health, 'inventory', return_value=inventory()), patch.object(health, 'probe', return_value={
+ 'status': 'failed', 'checks': [dict(test='cuda', status='failed', detail='driver mismatch')]}):
+ self.assertEqual(health.check(force=True)['status'], 'failed')
+ self.assertEqual((health.STATE / 'last-working.json').read_bytes(), good)
+
+ def test_running_update_defers_without_replacing_good_evidence_or_claiming_failure(self):
+ original_exists = Path.exists
+ def exists(path):
+ return str(path) == '/run/harness-os-restart-required' or original_exists(path)
+ with patch.object(health, 'inventory', return_value=inventory()), patch.object(Path, 'exists', exists), patch.object(health, 'probe') as probe:
+ report = health.check()
+ self.assertEqual(report['status'], 'deferred')
+ self.assertNotIn('fingerprint', report)
+ probe.assert_not_called()
+
+ def test_changed_driver_during_check_cannot_be_recorded_as_working(self):
+ with patch.object(health, 'inventory', side_effect=[inventory(), dict(inventory(), driver_version='new')]), patch.object(health, 'probe', side_effect=result):
+ report = health.check()
+ self.assertEqual(report['status'], 'deferred')
+ self.assertFalse((health.STATE / 'last-working.json').exists())
+
+ def test_timeout_crash_incomplete_or_wrong_device_result_never_passes(self):
+ cases = [subprocess.TimeoutExpired(['probe'], 20, output=b'{"stage":"cuda.compute"}\n{"operation":"cuCtxSynchronize"}\n'),
+ subprocess.CompletedProcess([], -11, '{"stage":"cuda.device"}', 'crash'),
+ subprocess.CompletedProcess([], 0, json.dumps({'result': dict(result('cuda', '0000:01:00.0', 1), checks=[])}), ''),
+ subprocess.CompletedProcess([], 0, json.dumps({'result': result('cuda', '0000:02:00.0', 1)}), '')]
+ for case in cases:
+ with self.subTest(case=case), patch.object(health.subprocess, 'run', side_effect=case if isinstance(case, Exception) else None,
+ return_value=case):
+ report = health.probe('cuda', '0000:01:00.0', 20)
+ self.assertEqual(report['status'], 'failed')
+ if isinstance(case, Exception):
+ self.assertEqual(report['checks'][0]['operation'], 'cuCtxSynchronize')
+
+ def test_probe_ignores_agent_gpu_filters_and_has_finite_timeout(self):
+ with patch.dict(health.os.environ, CUDA_VISIBLE_DEVICES='1', LD_PRELOAD='/tmp/injected.so'), patch.object(
+ health.subprocess, 'run', return_value=subprocess.CompletedProcess([], 0, json.dumps({'result': result('cuda', '0000:01:00.0', 1)}), '')) as run:
+ self.assertEqual(health.probe('cuda', '0000:01:00.0', 20)['status'], 'passed')
+ self.assertNotIn('CUDA_VISIBLE_DEVICES', run.call_args.kwargs['env'])
+ self.assertNotIn('LD_PRELOAD', run.call_args.kwargs['env'])
+ self.assertEqual(run.call_args.kwargs['timeout'], 20)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/os/tests/test_gpu_probe.py b/os/tests/test_gpu_probe.py
new file mode 100644
index 000000000..7114d6c57
--- /dev/null
+++ b/os/tests/test_gpu_probe.py
@@ -0,0 +1,58 @@
+"""Exercise the real ctypes ABI and error/readback handling without claiming a GPU."""
+import contextlib
+import ctypes
+import importlib.util
+import io
+from pathlib import Path
+import shutil
+import subprocess
+import sys
+import tempfile
+import unittest
+from unittest.mock import patch
+
+spec = importlib.util.spec_from_file_location('gpu_probe', Path(__file__).parents[1] / 'gpu_probe.py')
+probe = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(probe)
+
+
[email protected](shutil.which('cc'), 'A C compiler is required for the native ABI fixture')
+class DriverABITests(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls):
+ cls.temp = tempfile.TemporaryDirectory()
+ cls.addClassCleanup(cls.temp.cleanup)
+ output = Path(cls.temp.name) / 'fixture.so'
+ subprocess.run(['cc', '-dynamiclib' if sys.platform == 'darwin' else '-shared', '-fPIC',
+ '-Wall', '-Wextra', '-Werror', str(Path(__file__).with_name('gpu_cuda_fixture.c')),
+ '-o', str(output)], check=True, capture_output=True, timeout=30)
+ cls.library = ctypes.CDLL(str(output))
+ cls.library.fixture_mode.argtypes = [ctypes.c_int]
+ cls.library.fixture_live.restype = ctypes.c_int
+
+ def run_probe(self, mode):
+ self.library.fixture_mode(mode)
+ checks = []
+ with patch.object(probe.C, 'CDLL', return_value=self.library), contextlib.redirect_stdout(io.StringIO()):
+ try:
+ probe.compute('0000:01:00.0', checks)
+ finally:
+ self.assertEqual(self.library.fixture_live(), 0, 'The probe leaked a context, module or allocation')
+ return checks
+
+ def test_stable_64_bit_handles_and_kernel_parameter_pointer_survive_the_c_abi(self):
+ checks = self.run_probe(0)
+ self.assertEqual([row['test'] for row in checks], ['cuda.device', 'cuda.memory', 'cuda.compute'])
+ self.assertEqual(checks[0]['total_memory_bytes'], 24 << 30)
+ self.assertEqual(checks[1]['bytes'], 4096)
+ self.assertEqual(checks[2]['elements'], 1024)
+
+ def test_corrupt_copy_wrong_computation_async_error_and_wrong_gpu_are_rejected(self):
+ for mode, operation in [(1, 'memory_round_trip'), (2, 'integer_vector'), (3, 'cuCtxSynchronize'), (4, 'cuDeviceGetPCIBusId')]:
+ with self.subTest(mode=mode), self.assertRaises(probe.ProbeError) as caught:
+ self.run_probe(mode)
+ self.assertEqual(caught.exception.operation, operation)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/os/tools/build-package.py b/os/tools/build-package.py
index c0b987f30..7dcb6d8d7 100644
--- a/os/tools/build-package.py
+++ b/os/tools/build-package.py
@@ -51,6 +51,8 @@ def stage(source, runtime, destination, commit):
'os/live_update.py': 'usr/lib/harness-os/live_update.py',
'os/release_update.py': 'usr/lib/harness-os/release_update.py',
'os/hardware.py': 'usr/lib/harness-os/hardware.py',
+ 'os/gpu_health.py': 'usr/lib/harness-os/gpu_health.py',
+ 'os/gpu_probe.py': 'usr/lib/harness-os/gpu_probe.py',
'os/tools/hn-os': 'usr/bin/hn-os',
'os/lock.json': 'usr/share/harness-os/lock.json',
'tui/README.md': 'usr/share/harness-os/guide/tui.md',
diff --git a/os/tools/inspect_image.py b/os/tools/inspect_image.py
index 7ce4b836a..e48142891 100644
--- a/os/tools/inspect_image.py
+++ b/os/tools/inspect_image.py
@@ -40,6 +40,8 @@ def inspect(iso):
'usr/lib/harness-os/live_update.py': source / 'live_update.py',
'usr/lib/harness-os/release_update.py': source / 'release_update.py',
'usr/lib/harness-os/hardware.py': source / 'hardware.py',
+ 'usr/lib/harness-os/gpu_health.py': source / 'gpu_health.py',
+ 'usr/lib/harness-os/gpu_probe.py': source / 'gpu_probe.py',
'usr/bin/hn-os': source / 'tools/hn-os',
'usr/share/harness-os/lock.json': source / 'lock.json',
'usr/share/harness-os/guide/tui.md': source.parent / 'tui/README.md',