The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
feat(os): check NVIDIA readiness without interrupting work (#835)
Deee81e1004ac83
diff --git a/.github/workflows/os-nvidia.yml b/.github/workflows/os-nvidia.ymlindex f1b4c8e6d..78dcdd1a4 100644--- a/.github/workflows/os-nvidia.yml+++ b/.github/workflows/os-nvidia.yml@@ -3,6 +3,11 @@ name: Harness OS offline NVIDIA installationon:workflow_dispatch:inputs:+ probe:+ description: 'Offline driver installation or private GPU diagnostic integration'+ type: choice+ options: [install, health, ptx]+ default: installimage_run_id:description: 'Verified reference image for focused candidate testing'type: string@@ -14,11 +19,12 @@ permissions:actions: readconcurrency:- group: harness-nvidia-install-${{ github.ref }}+ group: harness-nvidia-${{ inputs.probe || 'install' }}-${{ github.ref }}cancel-in-progress: falsejobs:nvidia:+ if: ${{ !inputs.probe || inputs.probe == 'install' }}runs-on: ubuntu-24.04timeout-minutes: 30steps:@@ -79,3 +85,59 @@ jobs:os/dist/nvidia-bundle-manifest.jsonretention-days: 14compression-level: 0++ ptx:+ if: ${{ inputs.probe == 'ptx' }}+ runs-on: ubuntu-24.04+ timeout-minutes: 5+ steps:+ - uses: actions/checkout@v7+ - name: Assemble the exact probe with the pinned NVIDIA compiler+ run: python3 os/tests/gpu_ptx_native.py --output os/test-results/gpu-ptx+ - name: Retain compiler evidence without the tool archive+ if: always()+ uses: actions/upload-artifact@v6+ with:+ name: harness-os-gpu-ptx+ path: os/test-results/gpu-ptx/+ retention-days: 14+ compression-level: 0++ health:+ if: ${{ inputs.probe == 'health' }}+ runs-on: ubuntu-24.04+ timeout-minutes: 20+ steps:+ - uses: actions/checkout@v7+ - uses: actions/download-artifact@v6+ with:+ name: programmer-os-x86_64+ path: os/dist+ run-id: ${{ inputs.image_run_id }}+ github-token: ${{ github.token }}+ - name: Prepare native ABI and VM tools+ run: |+ sudo apt-get update -qq+ sudo apt-get install -y --no-install-recommends build-essential libegl1 libegl-mesa0 libgles2 mesa-vulkan-drivers qemu-system-x86 qemu-utils ovmf python3-pil tesseract-ocr+ sudo chmod a+rw /dev/kvm+ - name: Check OS contracts and real EGL ABI+ env:+ LIBGL_ALWAYS_SOFTWARE: 'true'+ run: |+ make -C os check+ go run github.com/rhysd/actionlint/cmd/[email protected] .github/workflows/os-nvidia.yml+ mkdir -p os/test-results/gpu-abi+ python3 os/tests/gpu_egl_native.py > os/test-results/gpu-abi/mesa.json+ - name: Check automatic diagnostics across encrypted cold boots+ run: python3 os/tests/gpu_health_vm.py --iso os/dist/*.iso --output os/test-results/gpu-health+ - name: Retain evidence without virtual disks+ if: always()+ uses: actions/upload-artifact@v6+ with:+ name: harness-os-gpu-health+ path: |+ os/test-results/gpu-abi/+ os/test-results/gpu-health/+ !os/test-results/gpu-health/*.qcow2+ !os/test-results/gpu-health/OVMF_VARS.fd+ retention-days: 14diff --git a/os/DEVELOPMENT.md b/os/DEVELOPMENT.mdindex 7d88d1508..0e5f23c26 100644--- a/os/DEVELOPMENT.md+++ b/os/DEVELOPMENT.md@@ -13,6 +13,47 @@ display/SSH interaction is still unverified.## Next release priorities+### NVIDIA readiness++The installed PC session starts a one-shot user check 15 seconds after the+workspace starts. It is inert on the installer and on computers without NVIDIA+GPUs. A per-user lock and boot/kernel/driver/device/probe fingerprint prevent+screen restarts from repeating it. Each new boot or relevant change rechecks;+`harness hardware --check-gpu` explicitly repeats it. Reports and the last fully+working result are atomic local JSON in `~/.local/state/harness-os/gpu/`.+They are user diagnostics, never privileged authorization or proof for a release.++Each owned card is addressed by PCI bus ID. The CUDA Driver API checks device+visibility, a 4 KiB memory upload/readback, and 1,024 integer results from a tiny+PTX kernel. A separate EGL device context renders and reads three colors using+the NVIDIA renderer. No CUDA toolkit, compiler, model or inference runtime enters+the image. Normal user permissions apply. Reserved devices are skipped, and+headless graphics failures remain explicitly unverified rather than labelling+working compute broken. Offscreen EGL does not establish compositor, browser,+physical display or suspend behavior; the memory sample is not a stress test.++Each subprocess has a 20-second deadline within a 120-second total budget; hangs,+crashes, incomplete responses and wrong readbacks are failures with the last+operation recorded. All contexts are private and released. The check never+resets a GPU, unloads modules, changes packages, restarts an agent or reboots.+Only actionable failure gets a short existing Harness status message. Package+replacement/restart-pending state defers verification until reboot. Existing full+snapshot updates keep the kernel and driver together and preserve the original+root/boot checkpoint; failures retain that recovery path, without an automatic+rollback that might interrupt work. Local-model provisioning remains the TUI's+responsibility.++The API sources are NVIDIA's [Driver API](https://docs.nvidia.com/cuda/cuda-programming-guide/03-advanced/driver-api.html),+[stable context ABI](https://docs.nvidia.com/cuda/archive/12.9.1/cuda-driver-api/group__CUDA__CTX.html),+and Khronos [EGL device displays](https://registry.khronos.org/EGL/extensions/EXT/EGL_EXT_platform_device.txt),+[render-node mapping](https://registry.khronos.org/EGL/extensions/EXT/EGL_EXT_device_drm_render_node.txt),+and [CUDA device mapping](https://registry.khronos.org/EGL/extensions/NV/EGL_NV_device_cuda.txt).+Portable tests include a compiled C ABI fixture with corrupt-memory, wrong-answer,+async-error and wrong-device cases. Such fixtures establish error handling, not+physical NVIDIA acceptance. Real cards must still pass these checks and actual+display, browser, update/recovery and sleep/wake testing before a family is called+supported.+The next preview improves the complete experience before expanding the interface.Keep the existing terminal, agents and optional browser. Hardware integration mustnot introduce a desktop, control panel or extra launcher. Update checks use adiff --git a/os/gpu_health.py b/os/gpu_health.pynew file mode 100644index 000000000..c52f50e93--- /dev/null+++ b/os/gpu_health.py@@ -0,0 +1,260 @@+#!/usr/bin/env python3+"""Quiet, bounded GPU verification for an installed Harness session."""+import argparse+from datetime import datetime, timezone+import fcntl+import hashlib+import importlib.util+import json+import os+from pathlib import Path+import re+import subprocess+import sys+import tempfile+import time++STATE = Path.home() / '.local/state/harness-os/gpu'+SYS = Path('/sys')+PROC = Path('/proc')+SYSTEM_STATE = Path('/var/lib/harness-os')+PROBE = Path(__file__).with_name('gpu_probe.py')+VERSION = 1+PROBE_SECONDS = 20+TOTAL_SECONDS = 120+++def read_json(path):+ try:+ value = json.loads(path.read_text())+ return value if isinstance(value, dict) else {}+ except (OSError, ValueError):+ return {}+++def text(path):+ try:+ return path.read_text().strip()+ except (OSError, UnicodeError):+ return ''+++def hardware():+ spec = importlib.util.spec_from_file_location('harness_gpu_hardware', Path(__file__).with_name('hardware.py'))+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return module+++def write_json(path, value):+ # Same-directory atomic publication; no half-written success after a crash.+ with tempfile.NamedTemporaryFile(dir=path.parent, prefix='.gpu-', mode='w', delete=False) as handle:+ pending = Path(handle.name)+ try:+ json.dump(value, handle, indent=2)+ handle.write('\n')+ handle.flush()+ os.fsync(handle.fileno())+ pending.replace(path)+ finally:+ pending.unlink(missing_ok=True)+++def library_identity(path):+ try:+ resolved, info = path.resolve(strict=True), path.stat()+ return {'path': str(resolved), 'bytes': info.st_size, 'mtime_ns': info.st_mtime_ns}+ except OSError:+ return None+++def inventory():+ cards = [card for card in hardware().pci_devices(SYS)+ if card['id'].startswith('10de:') and card['class'] in {'030000', '030200'}]+ return {'schema': VERSION, 'boot_id': text(PROC / 'sys/kernel/random/boot_id'),+ 'kernel': os.uname().release, 'driver_version': text(SYS / 'module/nvidia/version'),+ 'libraries': {name: library_identity(Path('/usr/lib') / name)+ for name in ['libcuda.so.1', 'libEGL_nvidia.so.0', 'libnvidia-eglcore.so.' + text(SYS / 'module/nvidia/version')]}+ if cards else {}, 'cards': cards,+ 'probe_sha256': hashlib.sha256(PROBE.read_bytes()).hexdigest(),+ 'controller_sha256': hashlib.sha256(Path(__file__).read_bytes()).hexdigest()}+++def fingerprint(value):+ return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()+++def cached_report():+ report = read_json(STATE / 'health.json')+ if not report:+ return {'status': 'not-tested', 'reason': 'GPU verification has not run for this user.'}+ report['stale'] = report.get('fingerprint') != fingerprint(inventory())+ return report+++def probe(kind, address, timeout):+ # Ignore a task's CUDA selection/debug injection in this OS diagnostic. Never+ # change the environment of existing agents, their processes or GPU contexts.+ env = {key: value for key, value in os.environ.items() if not key.startswith(+ ('CUDA_', 'NVIDIA_', '__NV_', '__EGL_', 'LD_'))}+ env.update(CUDA_CACHE_DISABLE='1', CUDA_MODULE_LOADING='EAGER')+ command = [sys.executable, str(PROBE), kind, address]+ failure = None+ try:+ result = subprocess.run(command, capture_output=True, text=True, timeout=timeout, env=env)+ output, errors = result.stdout, result.stderr+ if result.returncode:+ failure = f'Probe exited with status {result.returncode}'+ except subprocess.TimeoutExpired as error:+ output, errors = error.stdout or b'', error.stderr or b''+ failure = f'Probe exceeded {timeout:.1f} seconds'+ except OSError as error:+ output, errors, failure = '', '', str(error)+ if isinstance(output, bytes):+ output = output.decode(errors='replace')+ if isinstance(errors, bytes):+ errors = errors.decode(errors='replace')+ events = []+ for line in output.splitlines():+ try:+ item = json.loads(line)+ if isinstance(item, dict):+ events.append(item)+ except ValueError:+ pass+ final = events[-1].get('result') if events else None+ if not failure and isinstance(final, dict) and final.get('address') == address and final.get('probe') == kind:+ checks = final.get('checks')+ required = {'cuda.device', 'cuda.memory', 'cuda.compute'} if kind == 'cuda' else {'graphics.render'}+ if (isinstance(checks, list) and all(isinstance(c, dict) and c.get('status') in {'passed', 'failed'} for c in checks)+ and (final.get('status') == 'failed' and any(c['status'] == 'failed' for c in checks)+ or final.get('status') == 'passed' and all(c['status'] == 'passed' for c in checks)+ and required <= {c.get('test') for c in checks})):+ final['stderr'] = errors[-2048:]+ return final+ last = {key: next((e[key] for e in reversed(events) if key in e), None) for key in ['stage', 'operation']}+ return {'address': address, 'probe': kind, 'status': 'failed', 'checks': [+ {'test': kind, 'status': 'failed', 'detail': failure or 'Incomplete probe result', **last}], 'stderr': errors[-2048:]}+++def connections(address):+ result = {}+ for card in (SYS / 'class/drm').glob('card[0-9]*'):+ if '-' not in card.name and (card / 'device').resolve().name == address:+ for connector in (SYS / 'class/drm').glob(card.name + '-*'):+ result[connector.name] = text(connector / 'status')+ return result+++def recovery_context():+ system = read_json(SYSTEM_STATE / 'update.json')+ pointer = read_json(SYSTEM_STATE / 'runtime-updates/latest.json').get('id', '')+ # The root-owned pointer can still be absent or malformed after an interrupted+ # update. A diagnostic never follows an arbitrary path from that JSON.+ runtime = (read_json(SYSTEM_STATE / 'runtime-updates' / pointer / 'receipt.json')+ if isinstance(pointer, str) and re.fullmatch(r'[a-zA-Z0-9_-]{1,100}', pointer) else {})+ return {'system_update': {key: system.get(key) for key in ['snapshot', 'checkpoint', 'exit_status']},+ 'harness_update': {key: runtime.get(key) for key in ['checkpoint', 'status']},+ 'method': 'Offline checkpoint recovery preserves the home subvolume. No automatic rollback or reboot.'}+++def evaluate(current):+ started = time.monotonic()+ installed = read_json(SYSTEM_STATE / 'hardware.json').get('nvidia', {})+ report = {'schema': VERSION, 'fingerprint': fingerprint(current), 'inventory': current,+ 'checked_at': datetime.now(timezone.utc).isoformat(), 'devices': [],+ 'recovery': recovery_context(), 'limitations': [+ 'A small memory sample and computation, not a full VRAM stress test.',+ 'Offscreen EGL rendering does not verify display scanout, browser acceleration, or sleep/wake.',+ 'No model runtime, GPU reset, driver unload, package change or automatic reboot.']}+ # An intentional assignment must remain untouched, even when this GPU was+ # installed by Harness previously. Missing drivers on managed cards are errors.+ for card in current['cards']:+ record = dict(card, display_connections=connections(card['address']), checks=[])+ report['devices'].append(record)+ if (card.get('driver_override') not in {None, '', 'nvidia'} or card['driver'] not in {None, 'nvidia', 'nouveau'}):+ record.update(status='skipped', reason='GPU is assigned to another driver; its assignment is preserved.')+ continue+ if card['driver'] != 'nvidia':+ managed = installed.get('status') == 'installed' and card['id'] in installed.get('devices', [])+ record.update(status='failed' if managed else 'not-tested', reason='NVIDIA driver is not bound to this GPU.')+ record['checks'].append({'test': 'driver.binding', 'status': record['status'], 'actual': card['driver']})+ continue+ record['checks'].append({'test': 'driver.binding', 'status': 'passed', 'actual': card['driver']})+ for kind in ['cuda', 'graphics']:+ remaining = TOTAL_SECONDS - (time.monotonic() - started)+ if remaining <= 0:+ record['checks'].append({'test': kind, 'status': 'failed', 'detail': 'Overall diagnostic deadline reached; not tested.'})+ continue+ result = probe(kind, card['address'], min(PROBE_SECONDS, remaining))+ for check in result['checks']:+ # A compute-only/headless configuration may deliberately lack+ # graphics. Keep that evidence, without calling CUDA broken.+ unavailable = check.get('operation') in {'load_driver', 'EGL extensions', 'eglQueryDevicesEXT',+ 'graphics.device', 'eglGetPlatformDisplayEXT', 'eglInitialize', 'eglChooseConfig'}+ if kind == 'graphics' and check['status'] == 'failed' and unavailable and 'connected' not in record['display_connections'].values():+ check = dict(check, status='not-tested', reason='No connected display on this card; graphics remains unverified.')+ record['checks'].append(check)+ if result.get('stderr'):+ record.setdefault('probe_stderr', {})[kind] = result['stderr']+ record['status'] = ('failed' if any(c['status'] == 'failed' for c in record['checks']) else+ 'partial' if any(c['status'] == 'not-tested' for c in record['checks']) else 'passed')+ states = {row['status'] for row in report['devices']}+ report['status'] = ('failed' if 'failed' in states else 'partial' if states & {'partial', 'not-tested'} else+ 'passed' if 'passed' in states else 'skipped')+ report['duration_seconds'] = round(time.monotonic() - started, 3)+ return report+++def notify(report):+ if report['status'] != 'failed':+ return False+ try:+ subprocess.run(['/usr/bin/hn', 'display-message', '-d', '10000',+ 'GPU needs attention. Ask your agent to check harness hardware.'],+ check=True, capture_output=True, timeout=3)+ return True+ except (OSError, subprocess.SubprocessError):+ return False # The durable report remains available without a UI.+++def check(force=False, announce=False):+ STATE.mkdir(parents=True, exist_ok=True, mode=0o700)+ with (STATE / 'check.lock').open('a') as lock:+ try:+ fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)+ except BlockingIOError:+ return {'status': 'running', 'reason': 'GPU verification is already running.'}+ current = inventory()+ previous = read_json(STATE / 'health.json')+ # Package replacement can mix a loaded old driver with new userspace.+ # Never wake/check GPUs mid-update, nor report that as a failed new boot.+ if Path('/var/lib/pacman/db.lck').exists() or Path('/run/harness-os-restart-required').exists():+ report = {'schema': VERSION, 'status': 'deferred', 'reason': 'GPU verification will run after the update and restart.',+ 'inventory': current, 'devices': [], 'recovery': recovery_context()}+ else:+ if not force and previous.get('fingerprint') == fingerprint(current):+ return previous+ report = evaluate(current)+ if fingerprint(inventory()) != report['fingerprint'] or Path('/var/lib/pacman/db.lck').exists():+ report['status'] = 'deferred'+ report.pop('fingerprint', None)+ report['reason'] = 'Hardware or driver changed during verification; run again after restart.'+ write_json(STATE / 'health.json', report)+ if report['status'] == 'passed':+ write_json(STATE / 'last-working.json', report)+ if announce:+ notify(report)+ return report+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument('--force', action='store_true', help='Repeat checks on this boot')+ parser.add_argument('--notify', action='store_true', help='Show only actionable failure in the current Harness session')+ args = parser.parse_args()+ print(json.dumps(check(args.force, args.notify), indent=2))+++if __name__ == '__main__':+ main()diff --git a/os/gpu_probe.py b/os/gpu_probe.pynew file mode 100644index 000000000..61247c2e0--- /dev/null+++ b/os/gpu_probe.py@@ -0,0 +1,273 @@+#!/usr/bin/env python3+"""Short-lived, unprivileged GPU probes. The supervisor owns all time limits.++CUDA uses the stable, explicitly versioned Driver API, not a toolkit/runtime.+EGL uses EXT_platform_device and EXT_device_drm_render_node for exact PCI mapping.+See os/DEVELOPMENT.md for API references and what these checks do not prove.+"""+import argparse+import ctypes as C+import json+from pathlib import Path+import re++I, U, P = C.c_int, C.c_uint, C.c_void_p+N = 1024+PTX = b'''+.version 6.0+.target sm_50+.address_size 64+.visible .entry harness_check(.param .u64 data) {+ .reg .b32 i, b, n, v;+ .reg .b64 p, offset, addr;+ ld.param.u64 p, [data];+ mov.u32 i, %tid.x;+ mov.u32 b, %ctaid.x;+ mov.u32 n, %ntid.x;+ mad.lo.u32 i, b, n, i;+ mul.wide.u32 offset, i, 4;+ add.u64 addr, p, offset;+ ld.global.u32 v, [addr];+ mad.lo.u32 v, v, 3, 7;+ st.global.u32 [addr], v;+ ret;+}+'''+++def event(**value):+ print(json.dumps(value), flush=True)+++def function(library, name, result, *arguments):+ fn = getattr(library, name)+ fn.restype, fn.argtypes = result, list(arguments)+ return fn+++class ProbeError(Exception):+ def __init__(self, operation, code=None, detail=''):+ super().__init__(detail or operation)+ self.operation, self.code, self.detail = operation, code, detail+++def same_pci(first, second):+ def parts(value):+ match = re.fullmatch(r'([0-9a-f]{4,8}):([0-9a-f]{2}):([0-9a-f]{2})\.([0-7])', value.lower())+ return tuple(int(part, 16) for part in match.groups()) if match else None+ return parts(first) is not None and parts(first) == parts(second)+++class CUDA:+ def __init__(self):+ self.lib = C.CDLL('libcuda.so.1')+ self.error_name = function(self.lib, 'cuGetErrorName', I, I, C.POINTER(C.c_char_p))++ def call(self, name, types, *values):+ event(operation=name)+ code = function(self.lib, name, I, *types)(*values)+ if code:+ label = C.c_char_p()+ self.error_name(code, C.byref(label))+ raise ProbeError(name, code, label.value.decode() if label.value else 'CUDA error')++ def pci(self, device):+ address = C.create_string_buffer(32)+ self.call('cuDeviceGetPCIBusId', [P, I, I], address, len(address), device)+ return address.value.decode().lower()+++def compute(address, checks):+ event(stage='cuda.device')+ api = CUDA()+ api.call('cuInit', [U], 0)+ device, driver = I(), I()+ api.call('cuDeviceGetByPCIBusId', [P, C.c_char_p], C.byref(device), address.encode())+ actual = api.pci(device)+ if not same_pci(actual, address):+ raise ProbeError('cuDeviceGetPCIBusId', detail=f'Expected {address}, got {actual}')+ name, total = C.create_string_buffer(256), C.c_size_t()+ api.call('cuDeviceGetName', [P, I, I], name, len(name), device)+ api.call('cuDeviceTotalMem_v2', [P, I], C.byref(total), device)+ api.call('cuDriverGetVersion', [P], C.byref(driver))+ checks.append({'test': 'cuda.device', 'status': 'passed', 'pci_address': actual,+ 'name': name.value.decode(errors='replace'), 'total_memory_bytes': total.value,+ 'driver_api_version': driver.value})+ # A private context cannot reset, unload or synchronize another application's+ # context. Explicit _v2 symbols avoid the incompatible cuCtxCreate_v4 ABI.+ context, module, memory = P(), P(), C.c_uint64()+ try:+ event(stage='cuda.context')+ api.call('cuCtxCreate_v2', [P, U, I], C.byref(context), 4, device)+ event(stage='cuda.memory')+ original = [(i * 17 + 3) for i in range(N)]+ source, output = (U * N)(*original), (U * N)()+ size = C.sizeof(source)+ api.call('cuMemAlloc_v2', [P, C.c_size_t], C.byref(memory), size)+ api.call('cuMemcpyHtoD_v2', [C.c_uint64, P, C.c_size_t], memory, source, size)+ api.call('cuMemcpyDtoH_v2', [P, C.c_uint64, C.c_size_t], output, memory, size)+ if list(output) != original:+ raise ProbeError('memory_round_trip', detail='GPU memory readback differs from the uploaded pattern')+ checks.append({'test': 'cuda.memory', 'status': 'passed', 'bytes': size})+ event(stage='cuda.compute')+ api.call('cuModuleLoadData', [P, C.c_char_p], C.byref(module), PTX)+ kernel = P()+ api.call('cuModuleGetFunction', [P, P, C.c_char_p], C.byref(kernel), module, b'harness_check')+ parameters = (P * 1)(C.cast(C.byref(memory), P))+ api.call('cuLaunchKernel', [P, U, U, U, U, U, U, U, P, P, P],+ kernel, N // 256, 1, 1, 256, 1, 1, 0, None, parameters, None)+ api.call('cuCtxSynchronize', [])+ api.call('cuMemcpyDtoH_v2', [P, C.c_uint64, C.c_size_t], output, memory, size)+ expected = [value * 3 + 7 for value in original]+ mismatch = next((i for i in range(N) if output[i] != expected[i]), None)+ if mismatch is not None:+ raise ProbeError('integer_vector', detail=f'Element {mismatch}: expected {expected[mismatch]}, got {output[mismatch]}')+ checks.append({'test': 'cuda.compute', 'status': 'passed', 'elements': N, 'operation': '3*x+7'})+ finally:+ # A context owns these allocations. Even if a cleanup call fails, always+ # attempt the remaining releases; the worker then exits entirely.+ for name, types, value in [('cuMemFree_v2', [C.c_uint64], memory),+ ('cuModuleUnload', [P], module), ('cuCtxDestroy_v2', [P], context)]:+ if value.value:+ try:+ api.call(name, types, value)+ except ProbeError as error:+ checks.append({'test': 'cuda.cleanup', 'status': 'failed',+ 'operation': error.operation, 'code': error.code, 'detail': error.detail})+++def extensions(value):+ return set(value.decode().split()) if value else set()+++def device_matches(device, device_string, extension, address):+ ext = extensions(device_string(device, 0x3055))+ if 'EGL_EXT_device_drm_render_node' in ext:+ node = device_string(device, 0x3377)+ if node and same_pci((Path('/sys/class/drm') / Path(node.decode()).name / 'device').resolve().name, address):+ return True+ # Headless cards need not expose a render node. The official EGL/CUDA mapping+ # still identifies the exact PCI GPU; an EGL ordinal alone is not its identity.+ if 'EGL_NV_device_cuda' in ext:+ ordinal = C.c_ssize_t()+ device_attribute = extension('eglQueryDeviceAttribEXT', U, P, I, P)+ if device_attribute(device, 0x323A, C.byref(ordinal)):+ api = CUDA()+ api.call('cuInit', [U], 0)+ return same_pci(api.pci(ordinal.value), address)+ return False+++def accelerated_renderer(vendor, renderer):+ return bool('nvidia' in vendor.lower() and renderer and not any(+ word in renderer.lower() for word in ['llvmpipe', 'softpipe', 'software']))+++def graphics(address, checks):+ event(stage='graphics.device')+ egl = C.CDLL('libEGL.so.1')+ get_proc = function(egl, 'eglGetProcAddress', P, C.c_char_p)+ get_error = function(egl, 'eglGetError', I)++ def extension(name, result, *args):+ pointer = get_proc(name.encode())+ if not pointer:+ raise ProbeError(name, detail='EGL entry point is unavailable')+ return C.CFUNCTYPE(result, *args)(pointer)++ def call(name, result, types, *args):+ event(operation=name)+ value = function(egl, name, result, *types)(*args)+ if not value:+ raise ProbeError(name, get_error(), 'EGL operation failed')+ return value++ query = function(egl, 'eglQueryString', C.c_char_p, P, I)+ client = extensions(query(None, 0x3055)) # EGL_EXTENSIONS+ if not {'EGL_EXT_device_enumeration', 'EGL_EXT_device_query', 'EGL_EXT_platform_device'} <= client:+ raise ProbeError('EGL extensions', detail='EGL device selection is unavailable')+ enumerate_devices = extension('eglQueryDevicesEXT', U, I, P, P)+ device_string = extension('eglQueryDeviceStringEXT', C.c_char_p, P, I)+ devices, count = (P * 64)(), I()+ if not enumerate_devices(len(devices), devices, C.byref(count)) or not 0 < count.value <= len(devices):+ raise ProbeError('eglQueryDevicesEXT', get_error(), 'No usable EGL device inventory')+ selected = None+ for device in devices[:count.value]:+ if device_matches(device, device_string, extension, address):+ selected = device+ break+ if selected is None:+ raise ProbeError('graphics.device', detail='No EGL device maps to this PCI GPU')+ platform = extension('eglGetPlatformDisplayEXT', P, U, P, P)+ display = platform(0x313F, selected, None) # EGL_PLATFORM_DEVICE_EXT+ if not display:+ raise ProbeError('eglGetPlatformDisplayEXT', get_error(), 'Device display is unavailable')+ context = surface = None+ initialized = False+ try:+ major, minor = I(), I()+ call('eglInitialize', U, [P, P, P], display, C.byref(major), C.byref(minor))+ initialized = True+ call('eglBindAPI', U, [U], 0x30A0) # EGL_OPENGL_ES_API+ attributes = (I * 13)(0x3033, 1, 0x3040, 4, 0x3024, 8, 0x3023, 8, 0x3022, 8, 0x3021, 8, 0x3038)+ config, found = P(), I()+ call('eglChooseConfig', U, [P, P, P, I, P], display, attributes, C.byref(config), 1, C.byref(found))+ if found.value != 1:+ raise ProbeError('eglChooseConfig', detail='RGBA8 ES2 pbuffer is unavailable')+ surface = call('eglCreatePbufferSurface', P, [P, P, P], display, config,+ (I * 5)(0x3057, 8, 0x3056, 8, 0x3038))+ context = call('eglCreateContext', P, [P, P, P, P], display, config, None,+ (I * 3)(0x3098, 2, 0x3038))+ call('eglMakeCurrent', U, [P, P, P, P], display, surface, surface, context)+ gl = C.CDLL('libGLESv2.so.2')+ string = function(gl, 'glGetString', C.c_char_p, U)+ vendor, renderer = (string(token) for token in [0x1F00, 0x1F01])+ vendor = vendor.decode(errors='replace') if vendor else ''+ renderer = renderer.decode(errors='replace') if renderer else ''+ if not accelerated_renderer(vendor, renderer):+ raise ProbeError('graphics.renderer', detail=f'Expected NVIDIA acceleration, got {vendor}: {renderer}')+ event(stage='graphics.render')+ clear_color = function(gl, 'glClearColor', None, *[C.c_float] * 4)+ clear = function(gl, 'glClear', None, U)+ read_pixels = function(gl, 'glReadPixels', None, I, I, I, I, U, U, P)+ gl_error = function(gl, 'glGetError', U)+ pixel = (C.c_ubyte * 4)()+ for color in [(255, 0, 0, 255), (0, 255, 0, 255), (0, 0, 255, 255)]:+ clear_color(*(value / 255 for value in color))+ clear(0x4000) # GL_COLOR_BUFFER_BIT+ read_pixels(0, 0, 1, 1, 0x1908, 0x1401, pixel) # RGBA, UNSIGNED_BYTE+ code = gl_error()+ if code or tuple(pixel) != color:+ raise ProbeError('graphics.readback', code, f'Expected {color}, got {tuple(pixel)}')+ checks.append({'test': 'graphics.render', 'status': 'passed', 'vendor': vendor,+ 'renderer': renderer, 'surface': '8x8 offscreen EGL pbuffer', 'colors': 3})+ finally:+ if initialized:+ function(egl, 'eglMakeCurrent', U, P, P, P, P)(display, None, None, None)+ if context:+ function(egl, 'eglDestroyContext', U, P, P)(display, context)+ if surface:+ function(egl, 'eglDestroySurface', U, P, P)(display, surface)+ function(egl, 'eglTerminate', U, P)(display)+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument('probe', choices=['cuda', 'graphics'])+ parser.add_argument('address')+ args = parser.parse_args()+ if not re.fullmatch(r'[0-9a-f]{4,8}:[0-9a-f]{2}:[0-9a-f]{2}\.[0-7]', args.address):+ parser.error('Expected a PCI address.')+ checks = []+ try:+ (compute if args.probe == 'cuda' else graphics)(args.address, checks)+ except (ProbeError, OSError, AttributeError) as error:+ checks.append({'test': args.probe, 'status': 'failed',+ 'operation': getattr(error, 'operation', 'load_driver'),+ 'code': getattr(error, 'code', None), 'detail': str(error)})+ event(result={'address': args.address, 'probe': args.probe, 'checks': checks,+ 'status': 'failed' if any(row['status'] == 'failed' for row in checks) else 'passed'})+++if __name__ == '__main__':+ main()diff --git a/os/hardware.py b/os/hardware.pyindex 1be5555ea..d97e3ae50 100644--- a/os/hardware.py+++ b/os/hardware.py@@ -3,6 +3,7 @@import argparseimport fcntlimport hashlib+import importlib.utilimport jsonimport osfrom pathlib import Path@@ -109,7 +110,15 @@ def report(sysfs=Path('/sys'), proc=Path('/proc')):'installation_blocker': installation_blocker(sysfs),'backlights': [p.name for p in sorted((sysfs / 'class/backlight').glob('*'))],'broadcom_bundle_available': (BUNDLE / 'manifest.json').is_file(),- 'nvidia_bundle_available': (NVIDIA_BUNDLE / 'manifest.json').is_file()}+ 'nvidia_bundle_available': (NVIDIA_BUNDLE / 'manifest.json').is_file(),+ 'gpu_health': gpu_health().cached_report() if Path(__file__).with_name('gpu_health.py').is_file() else None}+++def gpu_health():+ spec = importlib.util.spec_from_file_location('harness_gpu_health', Path(__file__).with_name('gpu_health.py'))+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return moduledef nvidia_bundle_manifest(folder, all_files=False):@@ -331,7 +340,8 @@ def configure_install(target, devices=None, bundle=BUNDLE):def main():parser = argparse.ArgumentParser(description=__doc__)sub = parser.add_subparsers(dest='command')- sub.add_parser('report')+ reporting = sub.add_parser('report')+ reporting.add_argument('--check-gpu', action='store_true', help='Repeat GPU verification as the current user')activation = sub.add_parser('activate')activation.add_argument('address')installation = sub.add_parser('configure-install')@@ -351,6 +361,10 @@ def main():elif args.command == 'configure-install':result = configure_install(args.target)else:+ if getattr(args, 'check_gpu', False):+ if not Path(__file__).with_name('gpu_health.py').is_file():+ parser.error('GPU verification is not available in this system profile.')+ gpu_health().check(force=True)result = report()print(json.dumps(result, indent=2))diff --git a/os/root/usr/lib/harness-os/session b/os/root/usr/lib/harness-os/sessionindex b139146f1..f4a0797ca 100644--- a/os/root/usr/lib/harness-os/session+++ b/os/root/usr/lib/harness-os/session@@ -46,6 +46,9 @@ if [ -f /etc/harness-live ]; thenexec sudo /usr/bin/python3 /usr/lib/harness-os/install.py --bootfisystemctl --user start harness-daemon.service || true+if [ -f /usr/lib/harness-os/gpu_health.py ]; then+ systemctl --user start harness-gpu-check.timer || true+fiprintf '%s\n' 'Graphics unavailable; continuing in hn on the console.'cd "$HOME/projects"while :; dodiff --git a/os/root/usr/lib/systemd/user/harness-gpu-check.service b/os/root/usr/lib/systemd/user/harness-gpu-check.servicenew file mode 100644index 000000000..cad25fdcb--- /dev/null+++ b/os/root/usr/lib/systemd/user/harness-gpu-check.service@@ -0,0 +1,16 @@+[Unit]+Description=Verify GPU readiness without interrupting work+ConditionPathExists=!/etc/harness-live+PartOf=harness-os.target++[Service]+Type=oneshot+ExecStart=/usr/bin/python3 /usr/lib/harness-os/gpu_health.py --notify+TimeoutStartSec=150+TimeoutStopSec=3+Nice=10+LimitCORE=0+# This is the active user's GPU access, not a privileged diagnostic. Keep normal+# device permissions and avoid any reset, module unloading or session restart.+# Leave the driver's standard nvidia-modprobe helper usable, just as it is for+# an ordinary user process; this check adds no privileged helper of its own.diff --git a/os/root/usr/lib/systemd/user/harness-gpu-check.timer b/os/root/usr/lib/systemd/user/harness-gpu-check.timernew file mode 100644index 000000000..b602ef526--- /dev/null+++ b/os/root/usr/lib/systemd/user/harness-gpu-check.timer@@ -0,0 +1,9 @@+[Unit]+Description=Check GPU readiness after the workspace starts+ConditionPathExists=!/etc/harness-live+PartOf=harness-os.target++[Timer]+OnActiveSec=15+AccuracySec=1+Unit=harness-gpu-check.servicediff --git a/os/root/usr/lib/systemd/user/harness-os.target b/os/root/usr/lib/systemd/user/harness-os.targetindex 7fe9b7ac9..96856b3b9 100644--- a/os/root/usr/lib/systemd/user/harness-os.target+++ b/os/root/usr/lib/systemd/user/harness-os.target@@ -1,4 +1,4 @@[Unit]Description=The hn screen-Wants=hn-screen.service harness-install.service harness-idle.service harness-update.timer+Wants=hn-screen.service harness-install.service harness-idle.service harness-update.timer harness-gpu-check.timerAfter=graphical-session-pre.targetdiff --git a/os/root/usr/share/harness-os/AGENTS.md b/os/root/usr/share/harness-os/AGENTS.mdindex 87e567621..d9ae95338 100644--- a/os/root/usr/share/harness-os/AGENTS.md+++ b/os/root/usr/share/harness-os/AGENTS.md@@ -49,6 +49,24 @@ is required.## Network and hardware+- GPU verification runs briefly after the installed workspace starts. Success+ is quiet. Read `harness hardware` → `gpu_health` before changing NVIDIA drivers;+ `harness hardware --check-gpu` repeats the checks as the current user. Do not run+ it with sudo: ordinary session device access is part of the check. Each PCI GPU+ has separate binding, memory, computation and offscreen graphics results, with+ the exact failed API and error. `stale: true` means the saved result no longer+ describes this boot/driver. Unavailable or skipped checks are not passes.+- These are small readiness checks, not full VRAM, display, browser, sleep or+ model-workload validation. A headless GPU can pass compute while graphics stays+ explicitly unverified. Passthrough devices are never touched. Reports stay in+ `~/.local/state/harness-os/gpu/`; no hardware data is uploaded.+- Updates retain the original Btrfs root and matching boot checkpoint. GPU+ diagnostics run again after reboot, never by resetting a GPU used by agents.+ A failure does not automatically roll back or reboot. Use the reported update+ checkpoint with the existing offline recovery procedure when appropriate;+ projects remain in the separate home subvolume. Local models belong to the+ TUI's existing local-model workflow, not this hardware check.+- `harness hardware` reports the model, CPU baseline, PCI devices, bound driversand backlights. It does not collect serial numbers, Wi-Fi names, MAC addressesor passwords. Use actual device IDs when diagnosing hardware.diff --git a/os/tests/gpu_cuda_fixture.c b/os/tests/gpu_cuda_fixture.cnew file mode 100644index 000000000..e6475242b--- /dev/null+++ b/os/tests/gpu_cuda_fixture.c@@ -0,0 +1,69 @@+/* ABI test double, never a GPU acceptance result or product payload. */+#include <stdint.h>+#include <stdlib.h>+#include <string.h>+#include <stdio.h>+static unsigned int values[1024];+static int mode, allocations, contexts, modules;+void fixture_mode(int value) { mode = value; }+int fixture_live(void) { return allocations + contexts + modules; }+int cuGetErrorName(int code, const char **name) { (void)code; *name = "CUDA_FIXTURE_ERROR"; return 0; }+int cuInit(unsigned int flags) { return flags ? 1 : 0; }+int cuDeviceGetByPCIBusId(int *device, const char *address) {+ if (strcmp(address, "0000:01:00.0")) return 101;+ *device = 3; return 0;+}+int cuDeviceGetPCIBusId(char *address, int size, int device) {+ if (device != 3) return 101;+ snprintf(address, size, "%s", mode == 4 ? "0000:02:00.0" : "0000:01:00.0"); return 0;+}+int cuDeviceGetName(char *name, int size, int device) { (void)device; snprintf(name, size, "Fixture only"); return 0; }+int cuDeviceTotalMem_v2(size_t *bytes, int device) { (void)device; *bytes = (size_t)24 << 30; return 0; }+int cuDriverGetVersion(int *version) { *version = 13040; return 0; }+int cuCtxCreate_v2(void **context, unsigned int flags, int device) {+ if (flags != 4 || device != 3) return 1;+ *context = (void *)(uintptr_t)0x100000004ULL; ++contexts; return 0;+}+int cuMemAlloc_v2(uint64_t *memory, size_t bytes) {+ if (bytes != sizeof(values)) return 1;+ *memory = (uint64_t)(uintptr_t)values; ++allocations; return 0;+}+int cuMemcpyHtoD_v2(uint64_t device, const void *host, size_t bytes) {+ if (device != (uint64_t)(uintptr_t)values || bytes != sizeof(values)) return 1;+ memcpy(values, host, bytes); return 0;+}+int cuMemcpyDtoH_v2(void *host, uint64_t device, size_t bytes) {+ if (device != (uint64_t)(uintptr_t)values || bytes != sizeof(values)) return 1;+ memcpy(host, values, bytes); if (mode == 1) ((unsigned int *)host)[17] ^= 1; return 0;+}+int cuModuleLoadData(void **module, const void *ptx) {+ if (!strstr(ptx, ".visible .entry harness_check")) return 218;+ *module = (void *)(uintptr_t)0x100000008ULL; ++modules; return 0;+}+int cuModuleGetFunction(void **function, void *module, const char *name) {+ if (module != (void *)(uintptr_t)0x100000008ULL || strcmp(name, "harness_check")) return 1;+ *function = (void *)(uintptr_t)0x10000000cULL; return 0;+}+int cuLaunchKernel(void *function, unsigned int gx, unsigned int gy, unsigned int gz,+ unsigned int bx, unsigned int by, unsigned int bz, unsigned int shared,+ void *stream, void **parameters, void **extra) {+ if (function != (void *)(uintptr_t)0x10000000cULL || gx != 4 || gy != 1 || gz != 1 ||+ bx != 256 || by != 1 || bz != 1 || shared || stream || extra ||+ *(uint64_t *)parameters[0] != (uint64_t)(uintptr_t)values) return 1;+ for (int i = 0; i < 1024; ++i) values[i] = values[i] + values[i] + values[i] + 7;+ if (mode == 2) values[513] += 1;+ return 0;+}+int cuCtxSynchronize(void) { return mode == 3 ? 700 : 0; }+int cuMemFree_v2(uint64_t memory) {+ if (memory != (uint64_t)(uintptr_t)values) return 1;+ --allocations; return 0;+}+int cuModuleUnload(void *module) {+ if (module != (void *)(uintptr_t)0x100000008ULL) return 1;+ --modules; return 0;+}+int cuCtxDestroy_v2(void *context) {+ if (context != (void *)(uintptr_t)0x100000004ULL) return 1;+ --contexts; return 0;+}diff --git a/os/tests/gpu_egl_native.py b/os/tests/gpu_egl_native.pynew file mode 100644index 000000000..16052fe3d--- /dev/null+++ b/os/tests/gpu_egl_native.py@@ -0,0 +1,39 @@+#!/usr/bin/env python3+"""Real Mesa EGL ABI exercise; never evidence for NVIDIA hardware acceleration."""+import contextlib+import importlib.util+import io+import json+from pathlib import Path+from unittest.mock import patch++spec = importlib.util.spec_from_file_location('probe', Path(__file__).parents[1] / 'gpu_probe.py')+probe = importlib.util.module_from_spec(spec)+spec.loader.exec_module(probe)+++def main():+ records = []+ # Hosted Linux has Mesa's headless software EGL device. Select it explicitly+ # for an API/cleanup/readback check; keep its true vendor and renderer in the+ # receipt. This does not add a software fallback to the product.+ with patch.object(probe, 'device_matches', return_value=True), contextlib.redirect_stdout(io.StringIO()):+ try:+ probe.graphics('0000:00:00.0', [])+ except probe.ProbeError as error:+ assert error.operation == 'graphics.renderer', str(error)+ records.append({'check': 'software-renderer-rejected', 'detail': error.detail})+ else:+ raise AssertionError('Software graphics incorrectly passed NVIDIA verification')+ results = []+ with patch.object(probe, 'accelerated_renderer', return_value=True):+ probe.graphics('0000:00:00.0', results)+ assert len(results) == 1 and results[0]['status'] == 'passed'+ assert not probe.accelerated_renderer(results[0]['vendor'], results[0]['renderer'])+ records.append({'check': 'real-egl-api-and-color-readback', 'device': results[0]})+ print(json.dumps({'status': 'passed', 'checks': records,+ 'limits': 'Software Mesa with explicit test-only device selection; NVIDIA hardware is unverified.'}, indent=2))+++if __name__ == '__main__':+ main()diff --git a/os/tests/gpu_health_vm.py b/os/tests/gpu_health_vm.pynew file mode 100644index 000000000..05e60b36b--- /dev/null+++ b/os/tests/gpu_health_vm.py@@ -0,0 +1,162 @@+#!/usr/bin/env python3+"""Installed OS diagnostic integration; virtual display, no physical NVIDIA GPU."""+import argparse+import hashlib+import json+import os+from pathlib import Path+import re+import shlex+import subprocess+import time++from footprint_vm import copy_file+from hardware_update_vm import graceful_stop, verify_image+from vm import VM, check_graphical_keyboard+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument('--iso', type=Path, required=True)+ parser.add_argument('--output', type=Path, required=True)+ args = parser.parse_args()+ iso, folder = args.iso.resolve(), args.output.absolute()+ image = verify_image(iso, json.loads(Path(__file__).with_name('hardware-update.lock.json').read_text()))+ if not os.access('/dev/kvm', os.R_OK | os.W_OK):+ parser.error('Native x86 KVM is required.')+ source = subprocess.check_output(['git', 'rev-parse', 'HEAD'], text=True).strip()+ if subprocess.check_output(['git', 'status', '--porcelain'], text=True).strip():+ parser.error('Commit the candidate before native acceptance.')+ folder.mkdir(parents=True, exist_ok=False)+ root = Path(__file__).resolve().parents[1]+ files = {name: 'usr/lib/harness-os/' + name for name in ['gpu_health.py', 'gpu_probe.py', 'hardware.py']}+ files.update({'root/' + name: name for name in ['usr/lib/harness-os/session',+ *['usr/lib/systemd/user/' + unit for unit in ['harness-os.target', 'harness-gpu-check.service', 'harness-gpu-check.timer']]]})+ record = dict(status='running', started_at=time.time(), source_commit=source, image=image['iso'],+ image_source=image['source_commit'], files={name: hashlib.sha256((root / name).read_bytes()).hexdigest() for name in files},+ checks=[], limits=['Candidate OS overlay on original preview14, not a newly built image.',+ 'Virtual display with no NVIDIA device. This proves service/reporting behavior, not GPU hardware support.'])+ vm = VM(folder, iso, 'uefi', 2048, cpu='Nehalem')+ config = dict(disk='/dev/vda', expected_serial='HN_OS_TEST', confirm_erase='/dev/vda',+ username='me', hostname='harness', password='test-password-123', encrypt=True, serial_console=True)++ def auth():+ vm.command('printf %s ' + shlex.quote(config['password'] + '\n') + ' | sudo -S -v')++ try:+ vm.start(live=True)+ vm.monitor('set_link', name='hnnet', up=False)+ vm.wait(r'root@[^\r\n]*[#] ')+ vm.shell_ready = True+ vm.command('stty -echo; nmcli networking off')+ copy_file(vm, json.dumps(config).encode(), '/tmp/install.json')+ output, _ = vm.command('harness install --config /tmp/install.json --yes-erase-disk', timeout=360)+ (folder / 'install.log').write_text(output)+ graceful_stop(vm, False)+ vm.start(live=False)+ vm.login_installed(config)+ auth()+ vm.command('systemctl --user mask --now harness-update.timer harness-update.service')+ for index, (local, target) in enumerate(files.items()):+ data = (root / local).read_bytes()+ temporary = '/tmp/gpu-overlay-' + str(index)+ copy_file(vm, data, temporary)+ vm.command('printf %s ' + shlex.quote(record['files'][local] + ' ' + temporary + '\n') + ' | sha256sum -c -')+ vm.command('sudo -n install -m ' + ('755' if target.endswith('/session') else '644') + ' ' + temporary + ' /' + target)+ # No GPU readiness check belongs in the boot critical path. Activate the+ # package's real target by cold boot, not by calling its Python entrypoint.+ graceful_stop(vm, True)+ vm.start(live=False)+ vm.login_installed(config)+ path = '/home/me/.local/state/harness-os/gpu/health.json'+ vm.command('timeout 40 sh -c ' + shlex.quote('until test -f ' + path + '; do sleep .2; done'), timeout=45)+ before = json.loads(vm.read_file(path))+ assert before['status'] == 'skipped' and before['devices'] == []+ assert before['duration_seconds'] < 3, 'No-GPU check performed unnecessary work'+ assert before['inventory']['boot_id'] == vm.read_file('/proc/sys/kernel/random/boot_id').decode().strip()+ (folder / 'first-check.json').write_text(json.dumps(before, indent=2) + '\n')+ check_graphical_keyboard(vm, 'gpu-health-keyboard')+ vm.command('hn display-message -p "#{pane_id}" > /tmp/gpu-original-pane')+ original_pane = vm.read_file('/tmp/gpu-original-pane').decode().strip()+ assert re.fullmatch(r'%\d+', original_pane), 'Expected the existing shell pane'+ # The reference client restores terminal shells after a cold boot; it+ # does not replay an arbitrary prior foreground command. Start a real+ # bundled agent explicitly before asserting that checks preserve it.+ vm.command("mkdir -p ~/projects/gpu-health-proof && hn new-window -n gpu-proof " ++ shlex.quote('cd "$HOME/projects/gpu-health-proof" && exec /usr/bin/opencode'))+ vm.command('timeout 40 sh -c ' + shlex.quote('until pgrep -u 1000 -x opencode >/dev/null; do sleep .2; done'), timeout=45)+ deadline = time.monotonic() + 40+ while True:+ output, _ = vm.command('hn capture-pane -p')+ if 'opencode' in output.lower() and 'Ask anything' in output:+ break+ if time.monotonic() > deadline:+ raise TimeoutError('The bundled OpenCode did not become interactive')+ time.sleep(.5)+ (folder / 'agent-before-check.txt').write_text(output)+ vm.screenshot('agent-before-check')+ # Existing work must remain alive while checks and reports run.+ vm.command('pgrep -x opencode > /tmp/gpu-agent-pids && systemctl --user show harness-daemon -p MainPID --value > /tmp/gpu-daemon-pid')+ vm.command('systemctl --user start harness-gpu-check.service')+ assert json.loads(vm.read_file(path)) == before, 'A second start repeated the same-boot check'+ vm.command('harness hardware > /tmp/gpu-hardware.json')+ report = json.loads(vm.read_file('/tmp/gpu-hardware.json'))+ assert report['gpu_health']['stale'] is False and report['gpu_health']['status'] == 'skipped'+ # Test failure-message routing with an explicit diagnostic fixture, not+ # a fabricated GPU pass. The product source and real report stay intact.+ notice = ('import sys; sys.path.insert(0,"/usr/lib/harness-os"); import gpu_health; '+ 'assert gpu_health.notify({"status":"failed"})')+ vm.command('python3 -c ' + shlex.quote(notice))+ vm.screenshot('failure-message')+ output, _ = vm.command('hn show-messages')+ (folder / 'notification.log').write_text(output)+ assert 'GPU needs attention' in output+ vm.command('pgrep -x opencode | diff /tmp/gpu-agent-pids - && systemctl --user show harness-daemon -p MainPID --value | diff /tmp/gpu-daemon-pid -')+ # The shared keyboard fixture waits for a visible shell prompt. OpenCode+ # is now active in its own tab, so select the retained shell first.+ vm.command('hn select-window -t ' + original_pane + ' && hn select-pane -t ' + original_pane)+ check_graphical_keyboard(vm, 'gpu-health-after-check')+ record['checks'].append('Packaged timer cold-boots, skips absent hardware quickly, caches within the boot, exposes a fresh JSON report, and routes a fixture failure without replacing the agent or daemon.')+ auth()+ graceful_stop(vm, True)+ vm.start(live=False)+ vm.login_installed(config)+ vm.command('timeout 40 sh -c ' + shlex.quote('until systemctl --user show harness-gpu-check.service -p Result --value | grep -qx success && test -f ' + path + '; do sleep .2; done'), timeout=45)+ # Result=success is also the service's initial state: wait for the actual+ # report to carry this boot ID before claiming the second execution.+ expected_boot = vm.read_file('/proc/sys/kernel/random/boot_id').decode().strip()+ deadline = time.monotonic() + 35+ while True:+ after = json.loads(vm.read_file(path))+ if after['inventory']['boot_id'] == expected_boot:+ break+ if time.monotonic() > deadline:+ raise TimeoutError('New boot did not run GPU verification')+ time.sleep(.5)+ assert after['inventory']['boot_id'] != before['inventory']['boot_id'] and after['status'] == 'skipped'+ (folder / 'second-check.json').write_text(json.dumps(after, indent=2) + '\n')+ record['checks'].append('A second real encrypted cold boot rechecks automatically with a new boot identity.')+ auth()+ output, _ = vm.command('journalctl --user -u harness-gpu-check.service --no-pager')+ (folder / 'service.log').write_text(output)+ graceful_stop(vm, True)+ record['status'] = 'passed'+ except Exception as error:+ record.update(status='failed', error=repr(error))+ if vm.process and vm.process.poll() is None:+ try:+ vm.screenshot('failure')+ except Exception as screenshot_error:+ record['screenshot_error'] = repr(screenshot_error)+ raise+ finally:+ vm.stop()+ vm.log.close()+ vm.stderr.close()+ vm.control.cleanup()+ record['duration_seconds'] = round(time.time() - record['started_at'], 3)+ (folder / 'receipt.json').write_text(json.dumps(record, indent=2) + '\n')+++if __name__ == '__main__':+ main()diff --git a/os/tests/gpu_ptx_native.py b/os/tests/gpu_ptx_native.pynew file mode 100644index 000000000..6132369d8--- /dev/null+++ b/os/tests/gpu_ptx_native.py@@ -0,0 +1,80 @@+#!/usr/bin/env python3+"""Assemble the shipped PTX for Turing, Ada and Blackwell; no GPU execution claim."""+import argparse+import hashlib+import importlib.util+import json+from pathlib import Path+import subprocess+import time+from urllib.request import urlopen+import zipfile++URL = ('https://files.pythonhosted.org/packages/25/48/b54a06168a2190572a312bfe4ce443687773eb61367ced31e064953dd2f7/'+ 'nvidia_cuda_nvcc_cu12-12.9.86-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl')+SHA256 = '5d6a0d32fdc7ea39917c20065614ae93add6f577d840233237ff08e9a38f58f0'+BYTES = 40546229+++def digest(path):+ with path.open('rb') as handle:+ return hashlib.file_digest(handle, 'sha256').hexdigest()+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument('--output', type=Path, required=True)+ args = parser.parse_args()+ output = args.output.absolute()+ output.mkdir(parents=True, exist_ok=False)+ # Test-host tooling only. Neither the compiler nor its outputs are included+ # in the OS payload. Select one exact archive member, never extract a wheel.+ tools = output.parent / ('ptx-tools-' + str(time.time_ns()))+ tools.mkdir()+ wheel = tools / 'nvcc.whl'+ with urlopen(URL, timeout=30) as response, wheel.open('wb') as handle:+ count = 0+ while chunk := response.read(128 * 1024):+ count += len(chunk)+ if count > BYTES:+ raise ValueError('Compiler wheel exceeds its pinned size')+ handle.write(chunk)+ if wheel.stat().st_size != BYTES or digest(wheel) != SHA256:+ raise ValueError('Compiler wheel checksum mismatch')+ with zipfile.ZipFile(wheel) as archive:+ compiler = tools / 'ptxas'+ compiler.write_bytes(archive.read('nvidia/cuda_nvcc/bin/ptxas'))+ compiler.chmod(0o755)+ source = Path(__file__).parents[1] / 'gpu_probe.py'+ spec = importlib.util.spec_from_file_location('probe', source)+ probe = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(probe)+ ptx = output / 'harness.ptx'+ ptx.write_bytes(probe.PTX)+ version = subprocess.check_output([str(compiler), '--version'], text=True, timeout=10)+ record = dict(status='running', wheel={'url': URL, 'bytes': BYTES, 'sha256': SHA256},+ compiler={'version': version, 'sha256': digest(compiler)},+ probe_sha256=digest(source), ptx_sha256=digest(ptx), targets=[],+ limits='Offline NVIDIA assembler only; device execution and driver JIT remain unverified.')+ for target in ['sm_75', 'sm_89', 'sm_120']:+ binary = output / (target + '.cubin')+ result = subprocess.run([str(compiler), '--gpu-name=' + target, str(ptx), '-o', str(binary)],+ text=True, capture_output=True, timeout=30)+ (output / (target + '.log')).write_text(result.stdout + result.stderr)+ result.check_returncode()+ if binary.read_bytes()[:4] != b'\x7fELF':+ raise ValueError('Assembler did not produce a device binary')+ record['targets'].append({'target': target, 'bytes': binary.stat().st_size, 'sha256': digest(binary)})+ invalid = tools / 'invalid.ptx'+ invalid.write_bytes(probe.PTX.replace(b'mad.lo.u32 v, v, 3, 7;', b'not_a_ptx_instruction;'))+ rejected = subprocess.run([str(compiler), '--gpu-name=sm_89', str(invalid), '-o', str(tools / 'invalid.cubin')],+ capture_output=True, text=True, timeout=30)+ if rejected.returncode == 0:+ raise ValueError('The compiler did not reject intentionally invalid PTX')+ record.update(status='passed', negative_control={'returncode': rejected.returncode, 'stderr': rejected.stderr})+ (output / 'receipt.json').write_text(json.dumps(record, indent=2) + '\n')+ print(json.dumps(record, indent=2))+++if __name__ == '__main__':+ main()diff --git a/os/tests/test_gpu_health.py b/os/tests/test_gpu_health.pynew file mode 100644index 000000000..398362967--- /dev/null+++ b/os/tests/test_gpu_health.py@@ -0,0 +1,160 @@+"""GPU health failures must be visible and cannot masquerade as acceptance."""+import copy+import importlib.util+import json+from pathlib import Path+import subprocess+import tempfile+import unittest+from unittest.mock import patch++spec = importlib.util.spec_from_file_location('gpu_health', Path(__file__).parents[1] / 'gpu_health.py')+health = importlib.util.module_from_spec(spec)+spec.loader.exec_module(health)+++def card(address='0000:01:00.0', driver='nvidia', override=None):+ return dict(address=address, id='10de:2684', driver=driver, driver_override=override, **{'class': '030000'})+++def inventory(cards=None):+ return dict(schema=1, boot_id='first', kernel='fixture', driver_version='615.71.09',+ cards=[card()] if cards is None else cards, libraries={})+++def result(kind, address, timeout):+ names = ['cuda.device', 'cuda.memory', 'cuda.compute'] if kind == 'cuda' else ['graphics.render']+ return dict(probe=kind, address=address, status='passed', checks=[dict(test=name, status='passed') for name in names])+++class HealthTests(unittest.TestCase):+ def setUp(self):+ self.temp = tempfile.TemporaryDirectory()+ self.addCleanup(self.temp.cleanup)+ self.root = Path(self.temp.name)+ for name, value in [('STATE', self.root / 'state'), ('SYSTEM_STATE', self.root / 'system')]:+ value.mkdir()+ patcher = patch.object(health, name, value)+ patcher.start()+ self.addCleanup(patcher.stop)+ self.connected = patch.object(health, 'connections', return_value={'card0-DP-1': 'connected'})+ self.connected.start()+ self.addCleanup(self.connected.stop)++ def test_no_gpu_and_passthrough_never_load_cuda_or_notify(self):+ for cards in [[], [card(driver='vfio-pci')], [card(driver=None, override='vfio-pci')],+ [card(driver=None, override='pci-stub')]]:+ with self.subTest(cards=cards), patch.object(health, 'inventory', return_value=inventory(cards)), \+ patch.object(health, 'probe', side_effect=AssertionError('GPU was touched')):+ report = health.check(force=True)+ self.assertEqual(report['status'], 'skipped')+ with patch.object(health.subprocess, 'run') as run:+ self.assertFalse(health.notify(report))+ run.assert_not_called()++ def test_every_gpu_is_checked_and_one_bad_card_fails_the_report(self):+ second = '0000:02:00.0'+ def broken(kind, address, timeout):+ item = result(kind, address, timeout)+ if address == second and kind == 'cuda':+ item['status'] = 'failed'+ item['checks'][-1] = dict(test='cuda.compute', status='failed', detail='wrong answer')+ return item+ with patch.object(health, 'probe', side_effect=broken) as probe:+ report = health.evaluate(inventory([card(), card(second)]))+ self.assertEqual(probe.call_count, 4)+ self.assertEqual([row['status'] for row in report['devices']], ['passed', 'failed'])+ self.assertEqual(report['status'], 'failed')+ self.assertEqual(report['devices'][1]['checks'][3]['detail'], 'wrong answer')++ def test_headless_graphics_unavailable_is_explicitly_partial_not_broken_compute(self):+ def headless(kind, address, timeout):+ if kind == 'graphics':+ return dict(status='failed', checks=[dict(test='graphics.device', status='failed', operation='graphics.device', detail='No EGL device')])+ return result(kind, address, timeout)+ with patch.object(health, 'connections', return_value={}), patch.object(health, 'probe', side_effect=headless):+ report = health.evaluate(inventory())+ self.assertEqual(report['status'], 'partial')+ self.assertEqual(report['devices'][0]['checks'][-1]['status'], 'not-tested')++ def test_bad_graphics_readback_fails_even_without_a_monitor(self):+ def bad_color(kind, address, timeout):+ if kind == 'graphics':+ return dict(status='failed', checks=[dict(test='graphics.render', status='failed', operation='graphics.readback', detail='Wrong pixel')])+ return result(kind, address, timeout)+ with patch.object(health, 'connections', return_value={}), patch.object(health, 'probe', side_effect=bad_color):+ self.assertEqual(health.evaluate(inventory())['status'], 'failed')++ def test_missing_managed_driver_is_failure_but_existing_unsupported_gpu_is_not_claimed(self):+ current = inventory([card(driver='nouveau')])+ with patch.object(health, 'probe') as probe:+ self.assertEqual(health.evaluate(current)['status'], 'partial')+ (health.SYSTEM_STATE / 'hardware.json').write_text(json.dumps({'nvidia': {'status': 'installed', 'devices': ['10de:2684']}}))+ report = health.evaluate(current)+ self.assertEqual(report['status'], 'failed')+ self.assertEqual(report['devices'][0]['checks'][0]['test'], 'driver.binding')+ probe.assert_not_called()++ def test_cache_survives_screen_restart_but_boot_driver_device_or_test_change_rechecks(self):+ current = inventory()+ with patch.object(health, 'inventory', side_effect=lambda: copy.deepcopy(current)), patch.object(health, 'probe', side_effect=result) as probe:+ original = health.check()+ self.assertEqual(health.check(), original)+ self.assertEqual(probe.call_count, 2)+ for field, value in [('boot_id', 'second'), ('driver_version', '616.0'), ('kernel', 'next'), ('cards', [card('0000:03:00.0')]), ('schema', 2)]:+ current[field] = value+ health.check()+ self.assertEqual(probe.call_count, 12)+ self.assertFalse(health.cached_report()['stale'])+ current['driver_version'] = 'new'+ self.assertTrue(health.cached_report()['stale'])++ def test_last_working_result_is_retained_after_failure(self):+ with patch.object(health, 'inventory', return_value=inventory()), patch.object(health, 'probe', side_effect=result):+ health.check()+ good = (health.STATE / 'last-working.json').read_bytes()+ with patch.object(health, 'inventory', return_value=inventory()), patch.object(health, 'probe', return_value={+ 'status': 'failed', 'checks': [dict(test='cuda', status='failed', detail='driver mismatch')]}):+ self.assertEqual(health.check(force=True)['status'], 'failed')+ self.assertEqual((health.STATE / 'last-working.json').read_bytes(), good)++ def test_running_update_defers_without_replacing_good_evidence_or_claiming_failure(self):+ original_exists = Path.exists+ def exists(path):+ return str(path) == '/run/harness-os-restart-required' or original_exists(path)+ with patch.object(health, 'inventory', return_value=inventory()), patch.object(Path, 'exists', exists), patch.object(health, 'probe') as probe:+ report = health.check()+ self.assertEqual(report['status'], 'deferred')+ self.assertNotIn('fingerprint', report)+ probe.assert_not_called()++ def test_changed_driver_during_check_cannot_be_recorded_as_working(self):+ with patch.object(health, 'inventory', side_effect=[inventory(), dict(inventory(), driver_version='new')]), patch.object(health, 'probe', side_effect=result):+ report = health.check()+ self.assertEqual(report['status'], 'deferred')+ self.assertFalse((health.STATE / 'last-working.json').exists())++ def test_timeout_crash_incomplete_or_wrong_device_result_never_passes(self):+ cases = [subprocess.TimeoutExpired(['probe'], 20, output=b'{"stage":"cuda.compute"}\n{"operation":"cuCtxSynchronize"}\n'),+ subprocess.CompletedProcess([], -11, '{"stage":"cuda.device"}', 'crash'),+ subprocess.CompletedProcess([], 0, json.dumps({'result': dict(result('cuda', '0000:01:00.0', 1), checks=[])}), ''),+ subprocess.CompletedProcess([], 0, json.dumps({'result': result('cuda', '0000:02:00.0', 1)}), '')]+ for case in cases:+ with self.subTest(case=case), patch.object(health.subprocess, 'run', side_effect=case if isinstance(case, Exception) else None,+ return_value=case):+ report = health.probe('cuda', '0000:01:00.0', 20)+ self.assertEqual(report['status'], 'failed')+ if isinstance(case, Exception):+ self.assertEqual(report['checks'][0]['operation'], 'cuCtxSynchronize')++ def test_probe_ignores_agent_gpu_filters_and_has_finite_timeout(self):+ with patch.dict(health.os.environ, CUDA_VISIBLE_DEVICES='1', LD_PRELOAD='/tmp/injected.so'), patch.object(+ health.subprocess, 'run', return_value=subprocess.CompletedProcess([], 0, json.dumps({'result': result('cuda', '0000:01:00.0', 1)}), '')) as run:+ self.assertEqual(health.probe('cuda', '0000:01:00.0', 20)['status'], 'passed')+ self.assertNotIn('CUDA_VISIBLE_DEVICES', run.call_args.kwargs['env'])+ self.assertNotIn('LD_PRELOAD', run.call_args.kwargs['env'])+ self.assertEqual(run.call_args.kwargs['timeout'], 20)+++if __name__ == '__main__':+ unittest.main()diff --git a/os/tests/test_gpu_probe.py b/os/tests/test_gpu_probe.pynew file mode 100644index 000000000..7114d6c57--- /dev/null+++ b/os/tests/test_gpu_probe.py@@ -0,0 +1,58 @@+"""Exercise the real ctypes ABI and error/readback handling without claiming a GPU."""+import contextlib+import ctypes+import importlib.util+import io+from pathlib import Path+import shutil+import subprocess+import sys+import tempfile+import unittest+from unittest.mock import patch++spec = importlib.util.spec_from_file_location('gpu_probe', Path(__file__).parents[1] / 'gpu_probe.py')+probe = importlib.util.module_from_spec(spec)+spec.loader.exec_module(probe)++[email protected](shutil.which('cc'), 'A C compiler is required for the native ABI fixture')+class DriverABITests(unittest.TestCase):+ @classmethod+ def setUpClass(cls):+ cls.temp = tempfile.TemporaryDirectory()+ cls.addClassCleanup(cls.temp.cleanup)+ output = Path(cls.temp.name) / 'fixture.so'+ subprocess.run(['cc', '-dynamiclib' if sys.platform == 'darwin' else '-shared', '-fPIC',+ '-Wall', '-Wextra', '-Werror', str(Path(__file__).with_name('gpu_cuda_fixture.c')),+ '-o', str(output)], check=True, capture_output=True, timeout=30)+ cls.library = ctypes.CDLL(str(output))+ cls.library.fixture_mode.argtypes = [ctypes.c_int]+ cls.library.fixture_live.restype = ctypes.c_int++ def run_probe(self, mode):+ self.library.fixture_mode(mode)+ checks = []+ with patch.object(probe.C, 'CDLL', return_value=self.library), contextlib.redirect_stdout(io.StringIO()):+ try:+ probe.compute('0000:01:00.0', checks)+ finally:+ self.assertEqual(self.library.fixture_live(), 0, 'The probe leaked a context, module or allocation')+ return checks++ def test_stable_64_bit_handles_and_kernel_parameter_pointer_survive_the_c_abi(self):+ checks = self.run_probe(0)+ self.assertEqual([row['test'] for row in checks], ['cuda.device', 'cuda.memory', 'cuda.compute'])+ self.assertEqual(checks[0]['total_memory_bytes'], 24 << 30)+ self.assertEqual(checks[1]['bytes'], 4096)+ self.assertEqual(checks[2]['elements'], 1024)++ def test_corrupt_copy_wrong_computation_async_error_and_wrong_gpu_are_rejected(self):+ for mode, operation in [(1, 'memory_round_trip'), (2, 'integer_vector'), (3, 'cuCtxSynchronize'), (4, 'cuDeviceGetPCIBusId')]:+ with self.subTest(mode=mode), self.assertRaises(probe.ProbeError) as caught:+ self.run_probe(mode)+ self.assertEqual(caught.exception.operation, operation)+++if __name__ == '__main__':+ unittest.main()diff --git a/os/tools/build-package.py b/os/tools/build-package.pyindex c0b987f30..7dcb6d8d7 100644--- a/os/tools/build-package.py+++ b/os/tools/build-package.py@@ -51,6 +51,8 @@ def stage(source, runtime, destination, commit):'os/live_update.py': 'usr/lib/harness-os/live_update.py','os/release_update.py': 'usr/lib/harness-os/release_update.py','os/hardware.py': 'usr/lib/harness-os/hardware.py',+ 'os/gpu_health.py': 'usr/lib/harness-os/gpu_health.py',+ 'os/gpu_probe.py': 'usr/lib/harness-os/gpu_probe.py','os/tools/hn-os': 'usr/bin/hn-os','os/lock.json': 'usr/share/harness-os/lock.json','tui/README.md': 'usr/share/harness-os/guide/tui.md',diff --git a/os/tools/inspect_image.py b/os/tools/inspect_image.pyindex 7ce4b836a..e48142891 100644--- a/os/tools/inspect_image.py+++ b/os/tools/inspect_image.py@@ -40,6 +40,8 @@ def inspect(iso):'usr/lib/harness-os/live_update.py': source / 'live_update.py','usr/lib/harness-os/release_update.py': source / 'release_update.py','usr/lib/harness-os/hardware.py': source / 'hardware.py',+ 'usr/lib/harness-os/gpu_health.py': source / 'gpu_health.py',+ 'usr/lib/harness-os/gpu_probe.py': source / 'gpu_probe.py','usr/bin/hn-os': source / 'tools/hn-os','usr/share/harness-os/lock.json': source / 'lock.json','usr/share/harness-os/guide/tui.md': source.parent / 'tui/README.md',