The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
Handle busy encrypted-device cleanup after installation (#708)
Deefaa29f5d5059
diff --git a/os/INSTALL.md b/os/INSTALL.mdindex 5e5a0eb3f..93b0f7b5c 100644--- a/os/INSTALL.md+++ b/os/INSTALL.md@@ -1,6 +1,6 @@# Install Harness on a ThinkPad-These instructions are for **0.1.0-preview.8**, using a Mac to prepare the USB.+These instructions are for **0.1.0-preview.9**, using a Mac to prepare the USB.The USB starts a live session. Installation begins only when you choose **Install**in the installer; it erases the entire selected disk.@@ -21,20 +21,20 @@ compute still need testing on the actual hardware.## 2. Download and verify on the Mac-From the [preview 8 release](https://github.com/autonomous-ai/openharness/releases/tag/os-v0.1.0-preview.8),+From the [preview 9 release](https://github.com/autonomous-ai/openharness/releases/tag/os-v0.1.0-preview.9),download both files into the same folder:-- `harness-0.1.0-preview.8-x86_64.iso`-- `harness-0.1.0-preview.8-x86_64.iso.sha256`+- `harness-0.1.0-preview.9-x86_64.iso`+- `harness-0.1.0-preview.9-x86_64.iso.sha256`If they are in Downloads, open Terminal and run:```shcd ~/Downloads-shasum -a 256 -c harness-0.1.0-preview.8-x86_64.iso.sha256+shasum -a 256 -c harness-0.1.0-preview.9-x86_64.iso.sha256```-The result must say `harness-0.1.0-preview.8-x86_64.iso: OK`.+The result must say `harness-0.1.0-preview.9-x86_64.iso: OK`.If it does not, download the files again before flashing.## 3. Flash the USB@@ -183,8 +183,9 @@ They retain a recovery checkpoint and rebuild the boot image. When the screenoffers **Restart now**, save your work and press Enter when ready. Downloads donot restart the computer, and routine updates do not require another USB flash.-Preview 8 reduces the USB download size; it does not require an update or reflash-on an installed preview 7 computer. The installed-system feed remains on preview 7.+Preview 9 fixes a transient encrypted-disk cleanup failure after installation.+An installed computer that boots successfully needs no update or reflash for this+fix. The installed-system feed remains on preview 7.Preview 4 needs the 7.2 MB [preview 7 bootstrap bundle](https://github.com/autonomous-ai/openharness/releases/download/os-v0.1.0-preview.7/harness-update-0.1.0-preview.7-47872670a-x86_64.zip)once. Verify and extract that bundle, open a terminal in its folder, and run:@@ -220,6 +221,14 @@ The [OS README](https://github.com/autonomous-ai/openharness/blob/main/os/README## Older USB images+Preview 8 can finish writing the system, run `sync`, and successfully unmount+`/mnt/harness-os`, then report `cryptsetup close ... returned non-zero exit status 5`+with `Device ... is still in use`. Check `/var/log/harness-install.log`: if those+are the final steps, shut down normally with `sudo systemctl poweroff`, remove the+USB once off, and try the installed disk. Reinstallation is usually unnecessary.+If an earlier installation step failed, that error still needs diagnosis; reaching+the cleanup command alone does not prove that installation finished.+Preview 2 may report `Live system payload is missing` after copying the image intoRAM. If `/run/archiso/copytoram/airootfs.sfs` exists, its workaround is:diff --git a/os/README.md b/os/README.mdindex d5f6d41de..f4d113e79 100644--- a/os/README.md+++ b/os/README.md@@ -11,7 +11,7 @@ audience, not its name.Product names and interface copy follow the [Naming System](../docs/naming-system.md).-**Preview 8:** the USB opens a short network step+**Preview 9:** the USB opens a short network stepor goes straight into bundled OpenCode when already connected. **Install withoutconnecting** opens the same native installer offline. The agent works beside**New Harness** and **Connect a computer** starter panels; a persistent install@@ -25,12 +25,17 @@ Both the live and installed system use `me@harness`.`Super+u` opens Updates. Frequent hn/CLI releases download in the background;you choose when to reconnect the screen. Running agents and terminals stay alive.-[Download preview 8](https://github.com/autonomous-ai/openharness/releases/tag/os-v0.1.0-preview.8)+[Download preview 9](https://github.com/autonomous-ai/openharness/releases/tag/os-v0.1.0-preview.9)· [Mac → USB → ThinkPad installation guide](INSTALL.md)· [Standalone HTML/CSS landing page](../website/public/os/README.md)· [Development feedback loop and Mac support targets](DEVELOPMENT.md)-Preview 8 uses stronger USB payload compression and fixes a startup race: opening+Preview 9 briefly retries a busy encrypted-device close after the installed disk+has been unmounted. A persistent holder still reports an error; cleanup failures+also preserve any earlier installation error. A ThinkPad running preview 8 reached+Harness after normal shutdown and USB removal despite this final cleanup error.++Preview 8 introduced stronger USB payload compression and fixed a startup race: openinga terminal immediately can no longer send the USB welcome into that new window.Encrypted installation also budgets key derivation against available RAM,without treating RAM-backed swap as extra capacity after the agent trial.diff --git a/os/installer.py b/os/installer.pyindex fb416c7e1..d74c0fcff 100644--- a/os/installer.py+++ b/os/installer.py@@ -65,16 +65,49 @@ def command_log(path):os.close(fd)-def run(*args, input=None, capture=False):+def run(*args, input=None, capture=False, timeout=None):if COMMAND_LOG:COMMAND_LOG.write('\n$ ' + shlex.join(str(a) for a in args) + '\n')COMMAND_LOG.flush()- result = subprocess.run([str(a) for a in args], input=input, check=True,+ result = subprocess.run([str(a) for a in args], input=input, check=True, timeout=timeout,stdout=subprocess.PIPE if capture else COMMAND_LOG,stderr=COMMAND_LOG if capture else subprocess.STDOUT if COMMAND_LOG else None)return result.stdout.decode().strip() if capture else None+def log_diagnostic(message):+ if COMMAND_LOG:+ try:+ COMMAND_LOG.write(message + '\n')+ COMMAND_LOG.flush()+ except OSError:+ # A failed diagnostic write must not replace the installation error.+ pass+++def close_install_mapping(mapper, timeout=10):+ """Wait briefly for device probes; never force removal of an active mapping."""+ deadline = time.monotonic() + timeout+ while True:+ try:+ run('cryptsetup', 'close', mapper, timeout=max(0.01, deadline - time.monotonic()))+ return+ except subprocess.CalledProcessError as error:+ # cryptsetup documents 5 as a busy/existing device. Other failures+ # are not transient and must retain their original diagnostics.+ if error.returncode != 5 or time.monotonic() >= deadline:+ raise+ # udev may still be inspecting the filesystem after unmount. Wait+ # for queued probes without allowing an unrelated stuck event to+ # hold up shutdown forever. A persistent holder remains an error.+ try:+ run('udevadm', 'settle', '--timeout=1',+ timeout=max(0.01, min(2, deadline - time.monotonic())))+ except (subprocess.CalledProcessError, subprocess.TimeoutExpired):+ pass+ time.sleep(max(0, min(0.25, deadline - time.monotonic())))++def partitions(disk):separator = 'p' if disk[-1].isdigit() else ''return [f'{disk}{separator}{n}' for n in (1, 2, 3)]@@ -395,13 +428,26 @@ def install(config, source, target, progress=None):write(target, '/var/lib/harness-os/install.json', json.dumps(receipt, indent=2) + '\n')report('Finishing installation…')run('sync')- if progress is None:- print(f"Installed in {receipt['duration_seconds']:.1f}s. Shut down, remove the USB, and boot the disk.", flush=True)finally:- if mounted:- run('umount', '-R', target)- if opened:- run('cryptsetup', 'close', mapper)+ original_error = sys.exception()+ if original_error is not None:+ log_diagnostic(f'\nInstallation failed: {original_error}')+ try:+ if mounted:+ run('umount', '-R', target)+ if opened:+ close_install_mapping(mapper)+ except (OSError, subprocess.SubprocessError) as cleanup_error:+ if original_error is None:+ cleanup_error.add_note('Harness was written and synced. Shut down normally before removing the USB, then try booting the installed disk.')+ raise+ # Keep the actual installation failure visible. Cleanup is still+ # attempted and its failure is retained as additional information.+ detail = f'Disk cleanup also failed: {cleanup_error}'+ original_error.add_note(detail)+ log_diagnostic(detail)+ if progress is None:+ print(f"Installed in {receipt['duration_seconds']:.1f}s. Shut down, remove the USB, and boot the disk.", flush=True)def selected_size(config):@@ -731,8 +777,10 @@ def entrypoint():main(args)except KeyboardInterrupt:return 130- except (ValueError, OSError, curses.error, subprocess.CalledProcessError) as error:+ except (ValueError, OSError, curses.error, subprocess.SubprocessError) as error:print(f'Installation stopped: {error}', file=sys.stderr, flush=True)+ for note in getattr(error, '__notes__', ()):+ print(note, file=sys.stderr, flush=True)if not args.config and LAST_LOG is not None:print(f'Details: {LAST_LOG}', file=sys.stderr, flush=True)# An installer launched from the USB welcome owns its pane. Retain thediff --git a/os/lock.json b/os/lock.jsonindex eecc5532b..116ff6c6e 100644--- a/os/lock.json+++ b/os/lock.json@@ -1,5 +1,5 @@{- "version": "0.1.0-preview.8",+ "version": "0.1.0-preview.9","arch_snapshot": "2026/10/01","architecture": "x86_64","upgrades_from": [diff --git a/os/tests/installer-cleanup-native.py b/os/tests/installer-cleanup-native.pynew file mode 100644index 000000000..73da35fe4--- /dev/null+++ b/os/tests/installer-cleanup-native.py@@ -0,0 +1,136 @@+#!/usr/bin/env python3+"""Exercise cleanup with real dm-crypt holders on an owned temporary loop disk."""+import argparse+import hashlib+import importlib.util+import json+import os+from pathlib import Path+import re+import subprocess+import tempfile+import threading+import time+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument('--installer', type=Path, default=Path(__file__).resolve().parents[1] / 'installer.py')+ parser.add_argument('--output', type=Path, required=True)+ args = parser.parse_args()+ if os.uname().sysname != 'Linux' or os.geteuid() != 0:+ parser.error('Run as root in a disposable Linux VM with cryptsetup, Btrfs and loop devices.')+ output = args.output.resolve()+ output.mkdir(parents=True, exist_ok=False)+ spec = importlib.util.spec_from_file_location('installer', args.installer)+ installer = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(installer)+ started = time.monotonic()+ result = {'status': 'running', 'architecture': os.uname().machine,+ 'kernel': os.uname().release, 'checks': [],+ 'installer_sha256': hashlib.sha256(args.installer.read_bytes()).hexdigest()}+ mapper = f'hn-cleanup-test-{os.getpid()}'+ mapped = Path('/dev/mapper') / mapper+ assert not mapped.exists()+ loop, holder, timer = None, None, None+ try:+ with tempfile.TemporaryDirectory(prefix='harness-cleanup-') as temporary:+ root = Path(temporary)+ disk, mount = root / 'disk.raw', root / 'root'+ mount.mkdir()+ with disk.open('wb') as image:+ image.truncate(256 * 1024**2)++ def run(*command, **kwargs):+ return subprocess.run(list(map(str, command)), check=True, timeout=15,+ stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs)++ def open_mapping():+ # The fixture tests dm-crypt close behavior, not passphrase KDFs.+ # Its random, throwaway key is supplied only over stdin.+ run('cryptsetup', 'open', '--type', 'plain', '--cipher', 'aes-xts-plain64',+ '--key-size', '256', '--key-file=-', loop, mapper, input=key)++ key = os.urandom(32)+ loop = run('losetup', '--find', '--show', disk).stdout.decode().strip()+ assert re.fullmatch(r'/dev/loop\d+', loop)+ result['cryptsetup'] = run('cryptsetup', '--version').stdout.decode().strip()+ try:+ open_mapping()+ run('mkfs.btrfs', '-f', mapped)+ run('mount', mapped, mount)+ run('btrfs', 'subvolume', 'create', mount / 'home')+ run('mount', '-o', 'subvol=home', mapped, mount / 'home')+ (mount / 'marker').write_text('keep the completed installation\n')+ (mount / 'home' / 'project').write_text('keep the trial project\n')+ run('sync', '-f', mount)+ # A raw-device reader survives filesystem unmount, just like+ # a probe which is still inspecting the newly written device.+ holder = os.open(mapped, os.O_RDONLY)+ run('umount', '-R', mount)+ baseline = subprocess.run(['cryptsetup', 'close', mapper],+ capture_output=True, text=True, timeout=15)+ (output / 'baseline-close.txt').write_text(baseline.stdout + baseline.stderr)+ assert baseline.returncode == 5, baseline+ assert mapped.exists()+ result['checks'].append('Unmodified close returns 5 with the filesystem unmounted and a device reader open')++ timer = threading.Timer(1.5, os.close, args=(holder,))+ timer.start()+ holder = None # The joined timer owns this descriptor now.+ retried = time.monotonic()+ with installer.command_log(output / 'transient-close.log'):+ installer.close_install_mapping(mapper)+ timer.join()+ timer = None+ assert not mapped.exists()+ result['transient_close_seconds'] = round(time.monotonic() - retried, 3)+ result['checks'].append('Cleanup waits for the temporary reader and removes the mapping normally')++ open_mapping()+ holder = os.open(mapped, os.O_RDONLY)+ persistent = time.monotonic()+ try:+ with installer.command_log(output / 'persistent-close.log'):+ installer.close_install_mapping(mapper, timeout=1)+ except (subprocess.CalledProcessError, subprocess.TimeoutExpired):+ pass+ else:+ raise AssertionError('A persistent holder must not become success')+ assert mapped.exists()+ result['persistent_close_seconds'] = round(time.monotonic() - persistent, 3)+ assert result['persistent_close_seconds'] < 3+ result['checks'].append('A persistent reader remains an error within the deadline and is never forcibly removed')+ os.close(holder)+ holder = None+ # Reopen the same filesystem and verify the completed data was+ # not modified by either failed or retried close operation.+ run('mount', mapped, mount)+ assert (mount / 'marker').read_text() == 'keep the completed installation\n'+ assert (mount / 'home' / 'project').read_text() == 'keep the trial project\n'+ run('umount', '-R', mount)+ installer.close_install_mapping(mapper)+ result['checks'].append('The filesystem remains intact after the failed close and closes after its reader leaves')+ finally:+ if timer is not None:+ timer.join(timeout=5)+ if holder is not None:+ os.close(holder)+ if mount.is_mount():+ run('umount', '-R', mount)+ if mapped.exists():+ installer.close_install_mapping(mapper)+ if loop is not None:+ run('losetup', '--detach', loop)+ result['status'] = 'passed'+ except BaseException as error:+ result.update(status='failed', error=repr(error))+ raise+ finally:+ result['duration_seconds'] = round(time.monotonic() - started, 3)+ (output / 'receipt.json').write_text(json.dumps(result, indent=2) + '\n')+ print(json.dumps(result, indent=2), flush=True)+++if __name__ == '__main__':+ main()diff --git a/os/tests/test_installer.py b/os/tests/test_installer.pyindex 2c60c687c..18f382ed2 100644--- a/os/tests/test_installer.py+++ b/os/tests/test_installer.py@@ -199,6 +199,85 @@ class EncryptionMemory(unittest.TestCase):budget.assert_not_called()+class InstallerCleanup(unittest.TestCase):+ def test_transient_reader_is_retried_without_force_or_deferred_removal(self):+ attempts = []+ def command(*args, **kwargs):+ if args[:2] == ('cryptsetup', 'close'):+ attempts.append(args)+ if len(attempts) < 3:+ raise subprocess.CalledProcessError(5, args)+ elif args[0] == 'udevadm':+ # A queued unrelated event can outlast settle's brief budget.+ raise subprocess.CalledProcessError(1, args)+ with patch.object(installer, 'run', side_effect=command), patch.object(installer.time, 'sleep'):+ installer.close_install_mapping('hn-install-test')+ self.assertEqual(attempts, [('cryptsetup', 'close', 'hn-install-test')] * 3)++ def test_persistent_holder_remains_an_error_with_a_bounded_wait(self):+ error = subprocess.CalledProcessError(5, ['cryptsetup', 'close', 'hn-install-test'])+ def command(*args, **kwargs):+ if args[0] == 'cryptsetup':+ raise error+ started = installer.time.monotonic()+ with patch.object(installer, 'run', side_effect=command):+ with self.assertRaises(subprocess.CalledProcessError) as caught:+ installer.close_install_mapping('hn-install-test', timeout=0.02)+ self.assertIs(caught.exception, error)+ self.assertLess(installer.time.monotonic() - started, 1)++ def test_unexpected_close_error_and_timeout_are_not_retried(self):+ for error in [subprocess.CalledProcessError(4, ['cryptsetup', 'close']),+ subprocess.TimeoutExpired(['cryptsetup', 'close'], 10)]:+ with self.subTest(error=error), patch.object(installer, 'run', side_effect=error) as command:+ with self.assertRaises(type(error)) as caught:+ installer.close_install_mapping('hn-install-test')+ self.assertIs(caught.exception, error)+ self.assertEqual(command.call_count, 1)++ def failed_install(self, unmount_failure=False, log_failure=False):+ config = dict(username='me', hostname='harness', password='private-password',+ encrypt=True, disk='/dev/vda', confirm_erase='/dev/vda')+ original = subprocess.CalledProcessError(1, ['unsquashfs', 'payload.sfs'])+ cleanup = subprocess.CalledProcessError(5, ['cryptsetup', 'close', 'hn-install-test'])+ def command(*args, **kwargs):+ if args[0] == 'unsquashfs':+ raise original+ if unmount_failure and args[:2] == ('umount', '-R'):+ raise OSError('target is still mounted')+ return 'test-uuid'+ with tempfile.TemporaryDirectory() as temp:+ source, target, log = (Path(temp) / name for name in ('payload', 'target', 'install.log'))+ source.touch()+ with patch.object(installer, 'selected_disk'), \+ patch.object(installer, 'preflight'), \+ patch.object(installer, 'trial_source', return_value=None), \+ patch.object(installer, 'encryption_memory', return_value=128 * 1024), \+ patch.object(installer, 'run', side_effect=command), \+ patch.object(installer, 'close_install_mapping', side_effect=cleanup) as close, \+ installer.command_log(log), redirect_stdout(io.StringIO()) as output, ExitStack() as stack:+ if log_failure:+ stack.enter_context(patch.object(installer.COMMAND_LOG, 'write', side_effect=OSError('log is full')))+ with self.assertRaises(subprocess.CalledProcessError) as caught:+ installer.install(config, source, target)+ self.assertIs(caught.exception, original)+ self.assertIn('Disk cleanup also failed', original.__notes__[0])+ if not log_failure:+ self.assertIn('unsquashfs', log.read_text())+ self.assertNotIn('private-password', log.read_text())+ self.assertNotIn('Installed in', output.getvalue())+ self.assertEqual(close.call_count, 0 if unmount_failure else 1)++ def test_close_failure_never_hides_the_original_installation_error(self):+ self.failed_install()++ def test_failed_unmount_never_closes_a_still_mounted_mapping_or_hides_the_error(self):+ self.failed_install(unmount_failure=True)++ def test_failed_diagnostic_write_preserves_the_error_and_still_attempts_cleanup(self):+ self.failed_install(log_failure=True)++class Screen:"""Capture drawn text and supply keys; never expose a real disk or terminal."""def __init__(self):@@ -474,6 +553,15 @@ class InstallerExit(unittest.TestCase):self.assertIn('fixture read failure', error)acknowledge.assert_not_called()+ def test_cleanup_context_and_command_timeout_remain_visible(self):+ failure = subprocess.TimeoutExpired(['cryptsetup', 'close', 'hn-install-test'], 10)+ failure.add_note('Harness was written and synced. Shut down normally.')+ status, acknowledge, error = self.exercise(failure)+ self.assertEqual(status, 1)+ self.assertIn('cryptsetup', error)+ self.assertIn('Harness was written and synced. Shut down normally.', error)+ acknowledge.assert_called_once()+def test_deliberate_cancel_does_not_open_an_error_prompt(self):status, acknowledge, error = self.exercise(KeyboardInterrupt())self.assertEqual(status, 130)diff --git a/os/tests/vm.py b/os/tests/vm.pyindex 2002bb559..b34b47bb1 100644--- a/os/tests/vm.py+++ b/os/tests/vm.py@@ -19,6 +19,7 @@ import shleximport shutilimport socketimport subprocess+import sysimport tempfileimport tarfileimport time@@ -675,6 +676,33 @@ if status:assert config['password'] not in output, 'Installation diagnostics must not contain the password'+def check_installer_cleanup(vm, folder):+ """Exercise the actual packaged cleanup against a real temporary dm device."""+ encoded = base64.b64encode(Path(__file__).with_name('installer-cleanup-native.py').read_bytes()).decode()+ vm.command('install -m 600 /dev/null /run/hn-cleanup-native.b64')+ for offset in range(0, len(encoded), 3000):+ vm.command('printf %s ' + encoded[offset:offset + 3000] + ' >> /run/hn-cleanup-native.b64')+ vm.command('base64 -d /run/hn-cleanup-native.b64 > /run/hn-cleanup-native.py')+ try:+ output, _ = vm.command('python3 /run/hn-cleanup-native.py --installer /usr/lib/harness-os/install.py '+ '--output /run/hn-cleanup-result', timeout=90)+ (folder / 'cleanup-native.log').write_text(output)+ finally:+ failure = sys.exception()+ for name in ['receipt.json', 'baseline-close.txt', 'transient-close.log', 'persistent-close.log']:+ try:+ (folder / ('cleanup-' + name)).write_bytes(vm.read_file('/run/hn-cleanup-result/' + name, timeout=10))+ except Exception as error:+ if failure is None:+ raise+ failure.add_note(f'Could not retain cleanup {name}: {error}')+ receipt = json.loads((folder / 'cleanup-receipt.json').read_text())+ assert receipt['status'] == 'passed'+ expected = hashlib.sha256(Path(__file__).resolve().parents[1].joinpath('installer.py').read_bytes()).hexdigest()+ assert receipt['installer_sha256'] == expected, 'Cleanup evidence must exercise this exact installer'+ return receipt++def main():parser = argparse.ArgumentParser(description=__doc__)parser.add_argument('--iso', required=True, type=Path)@@ -809,6 +837,9 @@ assert str(i.live_payload()) == '/run/archiso/copytoram/airootfs.sfs'if args.live_only:result['status'] = 'passed'return+ if args.encrypt:+ result['installer_cleanup'] = check_installer_cleanup(vm, folder)+ result['checks'].append('Packaged installer closes a transiently busy encrypted mapping, preserves its filesystem and rejects a persistent holder')config = dict(disk='/dev/vda', expected_serial='HN_OS_TEST', confirm_erase='/dev/vda',username='me', hostname='harness', password='test-password-123',encrypt=args.encrypt, serial_console=True)