The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
os: resume encrypted Asahi payload installation (#949)
Deebf99e1557802
diff --git a/os/platforms/apple-silicon/README.md b/os/platforms/apple-silicon/README.mdindex c3d118c70..256e26197 100644--- a/os/platforms/apple-silicon/README.md+++ b/os/platforms/apple-silicon/README.md@@ -140,3 +140,45 @@ metadata. QEMU injects the firmware ESP identity; its protected partitions contasentinels, not macOS filesystems. These checks do not establish physical Applesupport or recovery from a torn GPT write. Fresh encryption, payload copying,boot configuration and installer-media integration remain the next stages.++## Private encrypted payload copy++`storage.py` continues from a completed target plan. It verifies the entire+read-only raw source image by SHA-256 before writing the destination, then checks+the image's Harness provenance, pristine account state and Fedora/Asahi layout.+The disk lock spans enrollment and copying; kernel device extents must match the+saved GPT plan before a filesystem can be created.++Every installation receives fresh LUKS2, Btrfs and ext4 identities. Cryptsetup+creates the volume key and an Argon2id password slot. The password is passed only+through stdin, never process arguments, temporary files or the EFI progress record.+The record is committed before formatting. A retry recognizes only its own+encryption/filesystem identities; a wrong password or foreign filesystem stops+the stage. It does not format a missing filesystem after copying has begun.++Root, home and boot files are copied with Unix ownership, hard links, ACLs and+extended attributes preserved. An interrupted copy can resume from the same+verified image. Once the copy is complete, a retry does not copy or format again,+so later work stays intact. Existing Asahi EFI files and vendor firmware are left+in place. Cleanup unmounts only owned paths and never recursively deletes a mount+directory.++```sh+python3 os/tests/asahi_storage_vm.py \+ --image /path/to/harness-asahi-private.raw --sha256 IMAGE_SHA256 \+ --image-source FULL_IMAGE_COMMIT \+ --fixture /path/to/verified-arm-fixture --fixture-source FULL_FIXTURE_COMMIT \+ --output os/test-results/asahi-storage+```++The native test repeats partition interruption/reboot acceptance, exits after the+LUKS format but before its progress update, rejects a wrong password on retry,+kills a real rsync during its copy, and resumes offline after another reboot.+It compares every copied tree, checks the exact frozen runtime hashes and unchanged+LUKS header, then verifies that a completed retry preserves a new project. Its+source image is attached read-only and checked again afterwards.++This remains a private construction stage. Boot configuration, account handoff,+installer UI/media and physical Apple acceptance are still required before this+is a bootable Harness installation. The native test uses a public fixture password;+its resulting disk must never be released.diff --git a/os/platforms/apple-silicon/storage.py b/os/platforms/apple-silicon/storage.pynew file mode 100644index 000000000..203f4c640--- /dev/null+++ b/os/platforms/apple-silicon/storage.py@@ -0,0 +1,334 @@+"""Private Asahi encrypted storage enrollment and verified image copy.++Only prepared target.py partitions can be written. This stage does not configure+boot or make an installation live; completed copies are never recopied on retry.+"""+from contextlib import contextmanager, ExitStack+import hashlib+import importlib.util+import json+import os+from pathlib import Path+import re+import shutil+import stat+import subprocess+import tempfile+import uuid++spec = importlib.util.spec_from_file_location('asahi_target', Path(__file__).with_name('target.py'))+target = importlib.util.module_from_spec(spec)+spec.loader.exec_module(target)+PHASES = ('planned', 'encrypted', 'filesystems', 'copying', 'copied')+++class StorageError(ValueError):+ pass+++def run(*args, secret=None, timeout=120):+ result = subprocess.run(list(map(str, args)), input=secret, stdout=subprocess.PIPE,+ stderr=subprocess.PIPE, timeout=timeout, env={**os.environ, 'LC_ALL': 'C'})+ if result.returncode:+ # Passwords go only through stdin, never arguments, state or diagnostics.+ raise StorageError(f'{args[0]} failed ({result.returncode}): ' + result.stderr.decode(errors='replace').strip())+ return result.stdout.decode().strip()+++def fingerprint(value):+ return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(',', ':')).encode()).hexdigest()+++def validate_identity(digest, source):+ if not re.fullmatch('[a-f0-9]{64}', str(digest)) or not re.fullmatch('[a-f0-9]{40}', str(source)):+ raise StorageError('Use the verified image SHA-256 and full source commit.')+++def new_state(plan, image_sha256, source_commit):+ target.validate_plan(plan)+ validate_identity(image_sha256, source_commit)+ return {'schema': 1, 'kind': 'harness-asahi-storage', 'plan_sha256': fingerprint(plan),+ 'image_sha256': image_sha256, 'source_commit': source_commit,+ 'luks_uuid': str(uuid.uuid4()), 'root_uuid': str(uuid.uuid4()),+ 'boot_uuid': str(uuid.uuid4()), 'phase': 'planned'}+++def validate_state(state, plan, image_sha256, source_commit):+ fields = {'schema', 'kind', 'plan_sha256', 'image_sha256', 'source_commit',+ 'luks_uuid', 'root_uuid', 'boot_uuid', 'phase'}+ if (not isinstance(state, dict) or set(state) != fields or type(state['schema']) is not int or state['schema'] != 1 or+ state['kind'] != 'harness-asahi-storage' or state['plan_sha256'] != fingerprint(plan) or+ state['image_sha256'] != image_sha256 or state['source_commit'] != source_commit or+ state['phase'] not in PHASES):+ raise StorageError('The saved copy belongs to a different installation or image.')+ ids = [target.identifier(state[name]) for name in ('luks_uuid', 'root_uuid', 'boot_uuid')]+ if len(set(ids)) != 3 or ids != [state[name] for name in ('luks_uuid', 'root_uuid', 'boot_uuid')]:+ raise StorageError('The installation filesystem identities are invalid.')+ return state+++def read_state(path, plan, digest, source):+ directory = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW)+ fd = None+ try:+ target.private_directory(directory)+ try:+ fd = os.open(path.name, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=directory)+ except FileNotFoundError:+ return None+ target.private_file(fd)+ with os.fdopen(fd, 'r') as stream:+ fd = None+ try:+ state = json.loads(stream.read(128 * 1024))+ except (UnicodeError, json.JSONDecodeError) as error:+ raise StorageError('The saved storage record is incomplete. Keep it for recovery.') from error+ return validate_state(state, plan, digest, source)+ finally:+ if fd is not None:+ os.close(fd)+ os.close(directory)+++def save_state(path, state):+ """Caller holds the verified target disk lock; publish only a fully synced record."""+ directory = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW)+ temporary = None+ try:+ target.private_directory(directory)+ if os.path.lexists(path):+ fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)+ try:+ target.private_file(fd)+ finally:+ os.close(fd)+ fd, temporary = tempfile.mkstemp(prefix='.storage-', dir=path.parent)+ with os.fdopen(fd, 'w') as stream:+ json.dump(state, stream, sort_keys=True, indent=2)+ stream.write('\n')+ stream.flush()+ os.fsync(stream.fileno())+ os.replace(temporary, path)+ temporary = None+ os.fsync(directory)+ finally:+ if temporary is not None:+ os.unlink(temporary)+ os.close(directory)+++def advance(path, state, phase):+ if PHASES.index(state['phase']) < PHASES.index(phase):+ updated = {**state, 'phase': phase}+ save_state(path, updated)+ state.update(updated)+++def probe(device):+ result = subprocess.run(['blkid', '-p', '-o', 'export', str(device)], stdout=subprocess.PIPE,+ stderr=subprocess.PIPE, timeout=30)+ if result.returncode not in (0, 2):+ raise StorageError('Cannot identify the installation filesystem.')+ values = dict(line.split('=', 1) for line in result.stdout.decode().splitlines() if '=' in line)+ if result.returncode == 2 and values:+ raise StorageError('The filesystem probe is incomplete.')+ return values+++def require_filesystem(values, kind, identity):+ if values.get('TYPE') != kind or values.get('UUID') != identity:+ raise StorageError('An installation filesystem has changed. Keep its data untouched.')+++@contextmanager+def work_directory():+ root = Path(tempfile.mkdtemp(prefix='harness-asahi-', dir='/run'))+ try:+ yield root+ finally:+ # Never recursively delete a work directory: an unsuccessful unmount+ # could otherwise turn cleanup into deletion of the installed OS.+ for child in root.iterdir():+ child.rmdir()+ root.rmdir()+++@contextmanager+def mounted(device, path, options):+ path.mkdir()+ try:+ run('mount', '-o', options, device, path)+ yield path+ finally:+ if os.path.ismount(path):+ run('umount', path)+++class Payload:+ """A SHA-256-verified raw Asahi image exposed as a read-only block device."""+ def __init__(self, device, image_sha256, source_commit):+ validate_identity(image_sha256, source_commit)+ self.device = str(device)+ self.sha256, self.source_commit = image_sha256, source_commit+ self.root = self.home = self.boot = None++ @contextmanager+ def open(self):+ info = os.stat(self.device, follow_symlinks=False)+ if not stat.S_ISBLK(info.st_mode) or run('blockdev', '--getro', self.device) != '1':+ raise StorageError('The verified source image must be a read-only block device.')+ if run('blockdev', '--getss', self.device) != '4096':+ raise StorageError('Use the Asahi 4096-byte-sector source image.')+ with open(self.device, 'rb', buffering=0) as stream:+ if hashlib.file_digest(stream, 'sha256').hexdigest() != self.sha256:+ raise StorageError('The source image checksum does not match. Nothing was installed.')+ table = target.normalize(target.read_table(self.device))+ target.verify_gpt(stream.fileno(), int(run('blockdev', '--getsize64', self.device)), table)+ separator = 'p' if self.device[-1].isdigit() else ''+ parts = [self.device + separator + str(n) for n in (1, 2, 3)]+ if [p['node'] for p in table['partitions']] != parts:+ raise StorageError('The source image partition layout has changed.')+ for device, kind in zip(parts, ('vfat', 'ext4', 'btrfs')):+ if probe(device).get('TYPE') != kind:+ raise StorageError('The source image filesystems have changed.')+ with work_directory() as root, ExitStack() as mounts:+ top = mounts.enter_context(mounted(parts[2], root / 'top', 'ro,rescue=nologreplay,subvolid=5'))+ boot = mounts.enter_context(mounted(parts[1], root / 'boot', 'ro,noload'))+ self.root, self.home, self.boot = top / 'root', top / 'home', boot+ try:+ self.verify_pristine()+ yield self+ finally:+ self.root = self.home = self.boot = None++ def verify_pristine(self):+ image = json.loads((self.root / 'usr/share/harness-os/image.json').read_text())+ if (image.get('kind') != 'harness-asahi-image-construction' or image.get('profile') != 'Harness' or+ image.get('release_ready') is not False or image.get('source_commit') != self.source_commit):+ raise StorageError('Use the verified private Harness Asahi image.')+ accounts = [row.split(':') for row in (self.root / 'etc/passwd').read_text().splitlines()]+ if any(1000 <= int(row[2]) < 65534 for row in accounts) or any(self.home.iterdir()):+ raise StorageError('The source image already contains a user or home data.')+ root = next(row.split(':') for row in (self.root / 'etc/shadow').read_text().splitlines() if row.startswith('root:'))+ if not root[1].startswith(('!', '*')) or (self.root / 'var/lib/harness-os/firstboot.json').exists():+ raise StorageError('The source image has already been configured.')+++@contextmanager+def encrypted_root(device, state, secret):+ name = 'harness-stage-' + state['luks_uuid'].replace('-', '')+ path = Path('/dev/mapper') / name+ if path.exists():+ identity = run('dmsetup', 'info', '--columns', '--noheadings', '--options', 'uuid', name)+ expected = 'CRYPT-LUKS2-' + state['luks_uuid'].replace('-', '') + '-' + name+ slaves = Path('/sys/dev/block') / f'{os.major(path.stat().st_rdev)}:{os.minor(path.stat().st_rdev)}' / 'slaves'+ if (identity != expected or len(list(slaves.iterdir())) != 1 or+ next(slaves.iterdir()).name != Path(device).name or+ run('dmsetup', 'info', '--columns', '--noheadings', '--options', 'open', name) != '0'):+ raise StorageError('An existing encryption mapping is in use or belongs to another installation.')+ run('cryptsetup', 'open', '--test-passphrase', '--key-file=-', device, secret=secret)+ else:+ run('cryptsetup', 'open', '--key-file=-', device, name, secret=secret)+ try:+ yield path+ finally:+ run('cryptsetup', 'close', name, timeout=30)+++def ensure_filesystem(device, kind, identity, state, *mkfs):+ values = probe(device)+ if not values.get('TYPE'):+ if PHASES.index(state['phase']) >= PHASES.index('filesystems') or values.get('PTTYPE'):+ raise StorageError('A previously created filesystem is missing. Keep this target untouched.')+ run(*mkfs, device)+ values = probe(device)+ require_filesystem(values, kind, identity)+++def subvolumes(top):+ if any(path.name not in ('root', 'home') for path in top.iterdir()):+ raise StorageError('The installation root contains another layout.')+ for name in ('root', 'home'):+ path = top / name+ if not path.exists():+ run('btrfs', 'subvolume', 'create', path)+ if path.is_symlink() or not path.is_dir():+ raise StorageError('An installation subvolume has changed.')+ run('btrfs', 'subvolume', 'show', path)+++def copy_tree(source, destination, excludes=()):+ run('rsync', '-aHAX', '--numeric-ids', '--one-file-system', '--delete', '--checksum',+ *('--exclude=' + value for value in excludes), str(source) + '/', str(destination) + '/', timeout=300)+++def mountpoint(source, destination):+ if os.path.lexists(destination) and (destination.is_symlink() or not destination.is_dir()):+ raise StorageError('An installation mountpoint has changed.')+ destination.mkdir(exist_ok=True)+ shutil.copystat(source, destination)+++def install(plan_path, payload, password, progress=None):+ """Enroll unique encrypted storage, then resume only its unfinished OS copy."""+ if not isinstance(password, str) or not password or any(c in password for c in '\0\n\r'):+ raise StorageError('Enter a password without line breaks.')+ report = progress or (lambda message: None)+ path = Path(plan_path).with_name('storage.json')+ # Authenticate the immutable payload before changing the destination.+ report('Checking installation files…')+ with payload.open(), target.locked_plan(plan_path) as (plan, disk_fd):+ boot, encrypted = [part['node'] for part in plan['additions']]+ state = read_state(path, plan, payload.sha256, payload.source_commit)+ if state is None:+ for device in (boot, encrypted):+ existing = probe(device)+ if existing.get('TYPE') or existing.get('PTTYPE'):+ raise StorageError('The reserved partitions already contain a filesystem. Keep it untouched.')+ state = new_state(plan, payload.sha256, payload.source_commit)+ save_state(path, state) # Durable identities before the first format.+ report('Setting up encryption…')+ values = probe(encrypted)+ if not values.get('TYPE'):+ if state['phase'] != 'planned' or values.get('PTTYPE'):+ raise StorageError('The encrypted filesystem is missing. Keep this target untouched.')+ run('cryptsetup', 'luksFormat', '--type', 'luks2', '--batch-mode', '--uuid', state['luks_uuid'],+ '--pbkdf', 'argon2id', '--pbkdf-memory', '262144', '--key-file=-', encrypted,+ secret=password.encode())+ values = probe(encrypted)+ require_filesystem(values, 'crypto_LUKS', state['luks_uuid'])+ metadata = json.loads(run('cryptsetup', 'luksDump', '--dump-json-metadata', encrypted))+ if not metadata.get('keyslots') or any(slot['kdf']['type'] != 'argon2id' for slot in metadata['keyslots'].values()):+ raise StorageError('The encryption configuration has changed.')+ with encrypted_root(encrypted, state, password.encode()) as mapper:+ advance(path, state, 'encrypted')+ ensure_filesystem(mapper, 'btrfs', state['root_uuid'], state,+ 'mkfs.btrfs', '--uuid', state['root_uuid'], '--label', 'Harness')+ ensure_filesystem(boot, 'ext4', state['boot_uuid'], state,+ 'mkfs.ext4', '-U', state['boot_uuid'], '-L', 'Harness boot',+ '-E', 'lazy_itable_init=0,lazy_journal_init=0')+ if state['phase'] == 'copied':+ return state # Never overwrite work in a completed installation.+ with work_directory() as root, ExitStack() as mounts:+ top = mounts.enter_context(mounted(mapper, root / 'top', 'subvolid=5,compress=zstd:1,noatime'))+ boot_path = mounts.enter_context(mounted(boot, root / 'boot', 'noatime'))+ subvolumes(top)+ advance(path, state, 'filesystems')+ advance(path, state, 'copying')+ report('Copying Harness…')+ # Mountpoints remain present even when rsync excludes their contents.+ for name in ('boot', 'home'):+ mountpoint(payload.root / name, top / 'root' / name)+ mountpoint(payload.boot / 'efi', boot_path / 'efi')+ copy_tree(payload.root, top / 'root', excludes=('/boot/***', '/home/***'))+ copy_tree(payload.home, top / 'home')+ copy_tree(payload.boot, boot_path, excludes=('/efi/***',))+ run('sync', '-f', top)+ run('sync', '-f', boot_path)+ current = target.read_table(plan['original']['device'])+ target.verify_gpt(disk_fd, int(run('blockdev', '--getsize64', plan['original']['device'])), target.normalize(current))+ if target.remaining(plan, current):+ raise StorageError('The target partition layout changed during copying.')+ advance(path, state, 'copied')+ report('Harness files are ready.')+ return statediff --git a/os/platforms/apple-silicon/target.py b/os/platforms/apple-silicon/target.pyindex 89a30f49c..826265781 100644--- a/os/platforms/apple-silicon/target.py+++ b/os/platforms/apple-silicon/target.py@@ -6,6 +6,7 @@ partitions are never removed, moved, resized or formatted by this module."""from __future__ import annotations+from contextlib import contextmanagerimport hashlibimport fcntlimport json@@ -337,13 +338,23 @@ def lock_disk(fd, timeout=5):time.sleep(.05)-def apply_plan(path):- """Append only the persisted plan's missing entries; safe to resume after one write.-- Re-read firmware ownership on every attempt. The plan must be persisted on- that ESP, mounted with private root-only permissions. This function does not- create filesystems or update boot files.- """+def verify_partition_devices(plan):+ """The kernel's writable device nodes must describe the same planned extents."""+ disk = os.stat(plan['original']['device']).st_rdev+ for part in plan['additions']:+ node = os.stat(part['node'], follow_symlinks=False)+ if not stat.S_ISBLK(node.st_mode):+ raise TargetError('An installation partition is not a block device.')+ block = (Path('/sys/dev/block') / f'{os.major(node.st_rdev)}:{os.minor(node.st_rdev)}').resolve(strict=True)+ if (int((block / 'start').read_text()) * 512 != part['start'] * 4096 or+ int((block / 'size').read_text()) * 512 != part['size'] * 4096 or+ (block.parent / 'dev').read_text().strip() != f'{os.major(disk)}:{os.minor(disk)}'):+ raise TargetError('Kernel partition devices differ from the installation plan. Restart before continuing.')+++@contextmanager+def locked_plan(path, *, allow_missing=False):+ """Revalidate ownership and hold the disk lock through an installation stage."""if platform.system() != 'Linux' or os.geteuid() != 0:raise TargetError('Partition preparation requires the Linux installer.')if (os.readlink('/proc/self/ns/mnt') == os.readlink('/proc/1/ns/mnt') or@@ -364,6 +375,27 @@ def apply_plan(path):or command('blockdev', '--getro', device) != '0'):raise TargetError('The planned disk is not a writable 4096-byte-sector block device.')lock_disk(fd)+ snapshot = read_table(device)+ verify_gpt(fd, int(command('blockdev', '--getsize64', device)), normalize(snapshot))+ missing = remaining(plan, snapshot)+ if missing and not allow_missing:+ raise TargetError('Prepare the planned Linux partitions before copying Harness.')+ if not allow_missing:+ verify_partition_devices(plan)+ yield plan, fd+ finally:+ os.close(fd)+++def apply_plan(path):+ """Append only the persisted plan's missing entries; safe to resume after one write.++ Re-read firmware ownership on every attempt. The plan must be persisted on+ that ESP, mounted with private root-only permissions. This function does not+ create filesystems or update boot files.+ """+ with locked_plan(path, allow_missing=True) as (plan, fd):+ device = plan['original']['device']# Revalidate each time. Foreign changes must never be mistaken for# resumable progress. The lock also excludes cooperating disk tools.for part in plan['additions']:@@ -379,11 +411,10 @@ def apply_plan(path):verify_gpt(fd, int(command('blockdev', '--getsize64', device)), normalize(snapshot))if remaining(plan, snapshot):raise TargetError('The planned Linux partitions were not fully created.')- finally:- os.close(fd)# An ESP mounted for the journal can prevent a whole-table kernel reread.# Update only our two entries, never remove/re-add existing partitions.for part in plan['additions']:command('partx', '--update', '--nr', partition_number(device, part['node']), device)command('udevadm', 'settle', '--timeout=10')+ verify_partition_devices(plan)return plandiff --git a/os/tests/asahi_encryption_vm.py b/os/tests/asahi_encryption_vm.pyindex 12980e084..dbb867801 100644--- a/os/tests/asahi_encryption_vm.py+++ b/os/tests/asahi_encryption_vm.py@@ -22,11 +22,14 @@ from session_vm import putclass MaintenanceVM(SessionVM):- def __init__(self, folder, disk, kernel, target):+ def __init__(self, folder, disk, kernel, target, source=None):super().__init__(folder, disk, kernel)if not target.is_file() or target.is_symlink():raise ValueError('The encryption target must be a cloned regular file.')self.target = target+ if source is not None and (not source.is_file() or source.is_symlink()):+ raise ValueError('The read-only payload must be a verified regular image file.')+ self.source = sourcedef start(self, offline=False):self.started = time.monotonic()@@ -41,6 +44,9 @@ class MaintenanceVM(SessionVM):'-netdev', 'user,id=net', '-device', 'virtio-net-pci,netdev=net,id=hnnet,romfile=','-serial', f'unix:{self.control_path / "serial.sock"},server=on,wait=off','-qmp', f'unix:{self.control_path / "qmp.sock"},server=on,wait=off']+ if self.source is not None:+ args.extend(['-drive', f'file={self.source},format=raw,if=none,id=payload,readonly=on',+ '-device', 'virtio-blk-pci,drive=payload,serial=HARNESS_PAYLOAD,logical_block_size=4096,physical_block_size=4096'])(self.folder / 'command.json').write_text(json.dumps(args, indent=2) + '\n')self.process = subprocess.Popen(args, stdout=self.stderr, stderr=self.stderr)self.serial = self.connect('serial.sock')diff --git a/os/tests/asahi_storage_guest.py b/os/tests/asahi_storage_guest.pynew file mode 100644index 000000000..197efd723--- /dev/null+++ b/os/tests/asahi_storage_guest.py@@ -0,0 +1,193 @@+#!/usr/bin/env python3+"""Private encrypted-copy acceptance on the owned Asahi target VM fixture."""+from contextlib import ExitStack+import hashlib+import importlib.util+import json+import os+from pathlib import Path+import subprocess+import sys+import time+++def load(name, path):+ spec = importlib.util.spec_from_file_location(name, path)+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return module+++fixture = load('fixture', '/var/tmp/harness-target-guest.py')+storage = load('storage', '/var/tmp/storage.py')+PASSWORD = 'firstboot-local-42'+STATE = fixture.PLAN.with_name('storage.json')+INPUT = Path('/var/tmp/harness-storage-input.json')+EVIDENCE = Path('/var/tmp/harness-storage-evidence.json')+RESULT = Path('/var/tmp/harness-storage-result.json')+ENCRYPTED = '/dev/vdb6'+++def payload():+ inputs = json.loads(INPUT.read_text())+ return storage.Payload('/dev/vdc', inputs['image_sha256'], inputs['source_commit'])+++def guard():+ fixture.guard()+ storage.target.platform_esp = lambda: fixture.ESP+ serial = fixture.run('lsblk', '-ndo', 'SERIAL', '/dev/vdc')+ assert serial == 'HARNESS_PAYLOAD', repr(serial)+ assert fixture.run('blockdev', '--getro', '/dev/vdc') == '1'+ fixture.mount()+++def header():+ metadata = json.loads(storage.run('cryptsetup', 'luksDump', '--dump-json-metadata', ENCRYPTED))+ offset = int(metadata['segments']['0']['offset'])+ with open(ENCRYPTED, 'rb', buffering=0) as handle:+ content = os.pread(handle.fileno(), offset, 0)+ return {'sha256': hashlib.sha256(content).hexdigest(), 'bytes': offset,+ 'uuid': storage.run('cryptsetup', 'luksUUID', ENCRYPTED),+ 'kdf': metadata['keyslots']['0']['kdf']['type']}+++def secret_absent():+ for path in (STATE, fixture.PLAN):+ assert PASSWORD not in path.read_text()+ state = json.loads(STATE.read_text())+ assert all('pass' not in key and 'key' not in key for key in state)+++def interrupt_encryption():+ command = storage.run+ def interrupted(*args, **kwargs):+ result = command(*args, **kwargs)+ if args[:2] == ('cryptsetup', 'luksFormat'):+ evidence = {'header': header(), 'phase_after_format': json.loads(STATE.read_text())['phase']}+ assert evidence['phase_after_format'] == 'planned'+ secret_absent()+ EVIDENCE.write_text(json.dumps(evidence, indent=2) + '\n')+ os._exit(76)+ return result+ storage.run = interrupted+ storage.install(fixture.PLAN, payload(), PASSWORD)+ raise AssertionError('The encryption interruption was not exercised')+++def interrupt_copy():+ evidence = json.loads(EVIDENCE.read_text())+ assert header() == evidence['header']+ record = STATE.read_bytes()+ try:+ storage.install(fixture.PLAN, payload(), 'incorrect-password')+ raise AssertionError('The retry accepted an incorrect disk password')+ except storage.StorageError as error:+ assert 'cryptsetup failed' in str(error)+ assert STATE.read_bytes() == record and header() == evidence['header']+ assert not storage.probe('/dev/vdb5').get('TYPE')+ evidence['incorrect_password_rejected_before_formatting'] = True+ original = storage.run+ def stop_during_copy(*args, **kwargs):+ if args[0] != 'rsync':+ return original(*args, **kwargs)+ destination = Path(args[-1])+ with open('/var/tmp/interrupted-rsync.log', 'wb') as log:+ process = subprocess.Popen(list(map(str, args)), stdout=log, stderr=log)+ deadline = time.monotonic() + 60+ marker = destination / 'etc/passwd'+ while not marker.is_file():+ if process.poll() is not None or time.monotonic() >= deadline:+ if process.poll() is None:+ process.kill()+ process.wait(timeout=15)+ raise AssertionError('rsync did not reach a live partial copy')+ time.sleep(.02)+ assert process.poll() is None+ process.kill()+ assert process.wait(timeout=15) == -9+ state = json.loads(STATE.read_text())+ assert state['phase'] == 'copying'+ assert header() == evidence['header']+ evidence.update(partial_copy=True, partial_state=state, partial_passwd_sha256=fixture.digest(marker))+ EVIDENCE.write_text(json.dumps(evidence, indent=2) + '\n')+ secret_absent()+ os._exit(77)+ storage.run = stop_during_copy+ storage.install(fixture.PLAN, payload(), PASSWORD)+ raise AssertionError('The real rsync interruption was not exercised')+++def assert_protected():+ baseline = json.loads(fixture.BASELINE.read_text())+ assert fixture.protected() == {k: v for k, v in baseline.items() if k not in ('table', 'started_at')}+++def verify_copy(state):+ with payload().open() as source, storage.encrypted_root(ENCRYPTED, state, PASSWORD.encode()) as mapper:+ with storage.work_directory() as root, ExitStack() as mounts:+ top = mounts.enter_context(storage.mounted(mapper, root / 'top', 'ro,rescue=nologreplay,subvolid=5'))+ boot = mounts.enter_context(storage.mounted('/dev/vdb5', root / 'boot', 'ro,noload'))+ for src, dst, excludes in [(source.root, top / 'root', ['/boot/***', '/home/***']),+ (source.home, top / 'home', []), (source.boot, boot, ['/efi/***'])]:+ changes = storage.run('rsync', '-aHAXnci', '--numeric-ids', '--one-file-system', '--delete',+ *('--exclude=' + value for value in excludes), str(src) + '/', str(dst) + '/', timeout=180)+ assert not changes, changes+ image = json.loads((top / 'root/usr/share/harness-os/image.json').read_text())+ manifest = image['session_package']['files']+ for name, expected in manifest.items():+ assert fixture.digest(top / 'root' / name) == expected, name+ for name, expected in image['first_boot']['files'].items():+ assert fixture.digest(top / 'root' / name) == expected['sha256'], name+ return {'runtime_files_checked': len(manifest), 'source_commit': image['source_commit'],+ 'root_uuid': storage.probe(mapper)['UUID'], 'boot_uuid': storage.probe('/dev/vdb5')['UUID']}+++def preserved_project(state, *, create=False):+ with storage.encrypted_root(ENCRYPTED, state, PASSWORD.encode()) as mapper, storage.work_directory() as root:+ with storage.mounted(mapper, root / 'home', 'subvol=home,noatime') as home:+ project = home / 'me/projects/keep/result.txt'+ if create:+ project.parent.mkdir(parents=True)+ project.write_text('Keep work after installation is copied.\n')+ storage.run('sync', '-f', home)+ return fixture.digest(project)+++def finish():+ evidence = json.loads(EVIDENCE.read_text())+ assert json.loads(STATE.read_text())['phase'] == 'copying'+ assert header() == evidence['header']+ state = storage.install(fixture.PLAN, payload(), PASSWORD)+ assert state['phase'] == 'copied' and header() == evidence['header']+ evidence['copy_verification'] = verify_copy(state)+ assert evidence['copy_verification']['root_uuid'] == state['root_uuid']+ assert evidence['copy_verification']['boot_uuid'] == state['boot_uuid']+ assert_protected()+ project = preserved_project(state, create=True)+ record = STATE.read_bytes()+ commands = []+ command = storage.run+ def observed(*args, **kwargs):+ commands.append(args)+ return command(*args, **kwargs)+ storage.run = observed+ assert storage.install(fixture.PLAN, payload(), PASSWORD) == state+ storage.run = command+ assert not any(args[0] in ('rsync', 'mkfs.btrfs', 'mkfs.ext4') or args[:2] == ('cryptsetup', 'luksFormat') for args in commands)+ assert STATE.read_bytes() == record+ assert preserved_project(state) == project and header() == evidence['header']+ assert_protected()+ secret_absent()+ fixture.run('umount', fixture.ROOT)+ assert not Path('/dev/mapper/harness-stage-' + state['luks_uuid'].replace('-', '')).exists()+ evidence.update(status='passed', state=state, project_sha256=project,+ completed_retry_preserves_work=True, protected_data_unchanged=True,+ header_unchanged=True, no_secret_in_records=True, mapping_closed=True)+ RESULT.write_text(json.dumps(evidence, indent=2) + '\n')+ print('Fresh encrypted copy resumed; complete file comparison, runtime hashes and project preservation passed.', flush=True)+++if __name__ == '__main__':+ guard()+ {'interrupt-encryption': interrupt_encryption, 'interrupt-copy': interrupt_copy, 'finish': finish}[sys.argv[1]]()diff --git a/os/tests/asahi_storage_vm.py b/os/tests/asahi_storage_vm.pynew file mode 100644index 000000000..7e8384856--- /dev/null+++ b/os/tests/asahi_storage_vm.py@@ -0,0 +1,117 @@+#!/usr/bin/env python3+"""Fresh encrypted Asahi payload install with real interruption and offline retries."""+import argparse+import json+from pathlib import Path+import platform+import re+import subprocess+import time++from arm_boot import digest+from arm_session import fixture_identity+from asahi_encryption_vm import MaintenanceVM+from asahi_target_vm import guest as target_guest+from session_vm import put++ROOT = Path(__file__).resolve().parents[1]+++def storage_guest(vm, action, image_hash, source):+ for name in ('storage.py', 'target.py'):+ put(vm, '/var/tmp/' + name, (ROOT / 'platforms/apple-silicon' / name).read_text())+ put(vm, '/var/tmp/harness-storage-guest.py', Path(__file__).with_name('asahi_storage_guest.py').read_text())+ put(vm, '/var/tmp/harness-storage-input.json', json.dumps({'image_sha256': image_hash, 'source_commit': source}))+ output, code = vm.command('unshare --mount --propagation private python3 /var/tmp/harness-storage-guest.py ' + action,+ timeout=420, check=False)+ expected = {'interrupt-encryption': 76, 'interrupt-copy': 77, 'finish': 0}[action]+ if code != expected:+ raise RuntimeError(f'{action} exited {code}, expected {expected}: {output[-6000:]}')+ if code:+ return {'exit_code': code, 'phase': action}+ raw = vm.read_file('/var/tmp/harness-storage-result.json')+ (vm.folder / 'result.json').write_bytes(raw)+ return json.loads(raw)+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument('--fixture', type=Path, required=True)+ parser.add_argument('--fixture-source', required=True)+ parser.add_argument('--image', type=Path, required=True)+ parser.add_argument('--sha256', required=True)+ parser.add_argument('--image-source', required=True)+ parser.add_argument('--output', type=Path, required=True)+ args = parser.parse_args()+ if (platform.system(), platform.machine()) != ('Darwin', 'arm64'):+ parser.error('Use the native Apple Silicon host observer.')+ if not re.fullmatch('[a-f0-9]{40}', args.image_source) or not re.fullmatch('[a-f0-9]{64}', args.sha256):+ parser.error('Use full source and image SHA-256 identities.')+ source = args.image.resolve()+ if args.image.is_symlink() or not source.is_file() or digest(source) != args.sha256:+ parser.error('Use the regular, SHA-256-verified private image.')+ info = fixture_identity(args.fixture, args.fixture_source)+ output = args.output.resolve()+ output.mkdir(parents=True, exist_ok=False)+ maintenance, disk = output / 'maintenance.raw', output / 'target.raw'+ with disk.open('xb') as handle:+ handle.truncate(24 * 1024**3)+ subprocess.run(['zstd', '-d', '--sparse', args.fixture / 'guest.raw.zst', '-o', maintenance], check=True, timeout=120)+ assert digest(maintenance) == info['raw_disk']['sha256']+ paths = [ROOT / 'platforms/apple-silicon' / name for name in ('target.py', 'storage.py')]+ paths += [ROOT / 'tools/fedora_payload.py']+ paths += [Path(__file__).with_name(name) for name in+ ('asahi_storage_vm.py', 'asahi_storage_guest.py', 'asahi_target_vm.py', 'asahi_target_guest.py',+ 'asahi_encryption_vm.py', 'asahi_encryption_guest.py', 'asahi_firstboot_vm.py',+ 'arm_boot.py', 'arm_session.py', 'fedora_session_vm.py', 'session_vm.py', 'vm.py')]+ receipt = {'status': 'running', 'started_at': time.time(), 'publication': False,+ 'image_sha256': args.sha256, 'image_source': args.image_source,+ 'maintenance_source': args.fixture_source, 'fixture_manifest_sha256': digest(args.fixture / 'manifest.json'),+ 'inputs': {str(path.relative_to(ROOT)): digest(path) for path in paths},+ 'limitations': ['QEMU supplies the firmware ESP UUID; protected macOS/recovery partitions contain sentinels.',+ 'No physical Apple or complete bootable installer claim.',+ 'Known public test password: never publish the resulting disk.'], 'shutdowns': []}+ vm = None+ try:+ vm = MaintenanceVM(output / 'prepare', maintenance, args.fixture / 'Image', disk, source=source)+ vm.start()+ vm.command('dnf5 install -y --setopt=install_weak_deps=False --setopt=gpgcheck=True gdisk dosfstools cryptsetup btrfs-progs e2fsprogs rsync', timeout=180)+ receipt['initialize'] = target_guest(vm, 'initialize', output)+ receipt['partition_interruption'] = target_guest(vm, 'interrupt', output)+ receipt['shutdowns'].append(vm.poweroff())+ vm.close()+ vm = MaintenanceVM(output / 'encryption', maintenance, args.fixture / 'Image', disk, source=source)+ vm.start(offline=True)+ receipt['target_acceptance'] = target_guest(vm, 'resume', output)+ receipt['encryption_interruption'] = storage_guest(vm, 'interrupt-encryption', args.sha256, args.image_source)+ receipt['shutdowns'].append(vm.poweroff())+ vm.close()+ vm = MaintenanceVM(output / 'partial-copy', maintenance, args.fixture / 'Image', disk, source=source)+ vm.start(offline=True)+ receipt['copy_interruption'] = storage_guest(vm, 'interrupt-copy', args.sha256, args.image_source)+ receipt['shutdowns'].append(vm.poweroff())+ vm.close()+ vm = MaintenanceVM(output / 'completed-copy', maintenance, args.fixture / 'Image', disk, source=source)+ vm.start(offline=True)+ receipt['result'] = storage_guest(vm, 'finish', args.sha256, args.image_source)+ receipt['shutdowns'].append(vm.poweroff())+ receipt['status'] = 'passed'+ print('Fresh LUKS2 enrollment, interrupted rsync, offline reboot/resume and preserved work passed.', flush=True)+ except BaseException as error:+ receipt.update(status='failed', error=str(error))+ raise+ finally:+ try:+ if vm:+ vm.close()+ finally:+ receipt.update(finished_at=time.time(), original_source_unchanged=digest(source) == args.sha256)+ if not receipt['original_source_unchanged']:+ receipt.update(status='failed', error='The source image changed.')+ (output / 'receipt.json').write_text(json.dumps(receipt, indent=2) + '\n')+ if receipt['status'] != 'passed':+ raise SystemExit(1)+++if __name__ == '__main__':+ main()diff --git a/os/tests/test_asahi_storage.py b/os/tests/test_asahi_storage.pynew file mode 100644index 000000000..ef469b496--- /dev/null+++ b/os/tests/test_asahi_storage.py@@ -0,0 +1,137 @@+"""State ownership and non-destructive retry checks for private Asahi storage."""+import copy+import importlib.util+import json+import os+from pathlib import Path+import tempfile+import unittest+from unittest.mock import patch++from test_asahi_target import fixture, ESP+spec = importlib.util.spec_from_file_location('asahi_storage', Path(__file__).resolve().parents[1] / 'platforms/apple-silicon/storage.py')+storage = importlib.util.module_from_spec(spec)+spec.loader.exec_module(storage)+++class StorageState(unittest.TestCase):+ def setUp(self):+ temporary = tempfile.TemporaryDirectory()+ self.addCleanup(temporary.cleanup)+ self.root = Path(temporary.name)+ self.path = self.root / 'storage.json'+ self.plan = storage.target.new_plan(fixture(), ESP)+ self.state = storage.new_state(self.plan, 'a' * 64, 'b' * 40)++ def read(self):+ return storage.read_state(self.path, self.plan, 'a' * 64, 'b' * 40)++ def test_new_installations_have_distinct_persisted_identities_and_no_password(self):+ another = storage.new_state(self.plan, 'a' * 64, 'b' * 40)+ self.assertTrue(set(self.state[name] for name in ('luks_uuid', 'root_uuid', 'boot_uuid')).isdisjoint(+ another[name] for name in ('luks_uuid', 'root_uuid', 'boot_uuid')))+ storage.save_state(self.path, self.state)+ self.assertEqual(self.read(), self.state)+ self.assertEqual(self.path.stat().st_mode & 0o777, 0o600)+ self.assertNotIn('password', self.path.read_text())++ def test_record_binds_exact_target_and_payload(self):+ storage.save_state(self.path, self.state)+ cases = [(storage.target.new_plan(fixture(), ESP), 'a' * 64, 'b' * 40),+ (self.plan, 'c' * 64, 'b' * 40), (self.plan, 'a' * 64, 'c' * 40)]+ before = self.path.read_bytes()+ for plan, digest, source in cases:+ with self.assertRaises(storage.StorageError):+ storage.read_state(self.path, plan, digest, source)+ self.assertEqual(self.path.read_bytes(), before)++ def test_corrupt_or_foreign_record_never_falls_back_to_fresh_format(self):+ storage.save_state(self.path, self.state)+ for content in ['{', json.dumps({**self.state, 'phase': 'unknown'}),+ json.dumps({**self.state, 'schema': True}),+ json.dumps({**self.state, 'boot_uuid': self.state['root_uuid']}),+ json.dumps({**self.state, 'password': 'must-not-be-saved'})]:+ self.path.write_text(content)+ with self.subTest(content=content), self.assertRaises(ValueError):+ self.read()+ self.path.unlink()+ self.assertIsNone(self.read())++ def test_interrupted_record_replacement_preserves_previous_phase(self):+ storage.save_state(self.path, self.state)+ with patch.object(storage.os, 'fsync', side_effect=OSError('interrupted')):+ with self.assertRaises(OSError):+ storage.advance(self.path, self.state, 'copying')+ self.assertEqual(self.read()['phase'], 'planned')+ self.assertEqual(self.state['phase'], 'planned')+ self.assertFalse(list(self.root.glob('.storage-*')))++ def test_completed_phase_cannot_be_downgraded(self):+ storage.save_state(self.path, self.state)+ storage.advance(self.path, self.state, 'copied')+ before = self.path.read_bytes()+ for phase in storage.PHASES:+ storage.advance(self.path, self.state, phase)+ self.assertEqual(self.path.read_bytes(), before)+ self.assertEqual(self.read()['phase'], 'copied')++ def test_loose_parent_symlink_and_fifo_are_refused(self):+ storage.save_state(self.path, self.state)+ self.root.chmod(0o777)+ with self.assertRaises(storage.target.TargetError):+ self.read()+ self.root.chmod(0o700)+ original = self.root / 'original'+ self.path.rename(original)+ self.path.symlink_to(original)+ with self.assertRaises(OSError):+ self.read()+ self.path.unlink()+ os.mkfifo(self.path, 0o600)+ with self.assertRaises(storage.target.TargetError):+ self.read()++ def test_owned_existing_filesystem_is_not_reformatted(self):+ with patch.object(storage, 'probe', return_value={'TYPE': 'btrfs', 'UUID': self.state['root_uuid']}), patch.object(storage, 'run') as run:+ storage.ensure_filesystem('/device', 'btrfs', self.state['root_uuid'], self.state, 'mkfs.btrfs')+ run.assert_not_called()++ def test_foreign_or_disappeared_completed_filesystem_is_never_formatted(self):+ cases = [({'TYPE': 'ext4', 'UUID': self.state['root_uuid']}, 'planned'),+ ({'TYPE': 'btrfs', 'UUID': self.state['boot_uuid']}, 'planned'),+ ({'PTTYPE': 'gpt'}, 'planned'), ({}, 'filesystems'), ({}, 'copying'), ({}, 'copied')]+ for values, phase in cases:+ state = {**self.state, 'phase': phase}+ with self.subTest(values=values, phase=phase), patch.object(storage, 'probe', return_value=values), patch.object(storage, 'run') as run:+ with self.assertRaises(storage.StorageError):+ storage.ensure_filesystem('/device', 'btrfs', self.state['root_uuid'], state, 'mkfs.btrfs')+ run.assert_not_called()++ def test_invalid_password_does_not_touch_payload_or_target(self):+ for password in ('', None, b'secret', 'line\nbreak', 'null\0byte'):+ with self.assertRaises(storage.StorageError):+ storage.install(self.root / 'target.json', None, password)++ def test_cleanup_never_recursively_removes_a_populated_mountpoint(self):+ area = self.root / 'work'+ area.mkdir()+ data = area / 'mount' / 'keep'+ data.parent.mkdir()+ data.write_text('Keep this installed file')+ with patch.object(storage.tempfile, 'mkdtemp', return_value=str(area)):+ with self.assertRaises(OSError):+ with storage.work_directory():+ pass+ self.assertEqual(data.read_text(), 'Keep this installed file')++ def test_mountpoint_does_not_follow_an_existing_symlink(self):+ source = self.root / 'source'+ source.mkdir()+ destination = self.root / 'destination'+ destination.symlink_to(source)+ with self.assertRaises(storage.StorageError):+ storage.mountpoint(source, destination)+++if __name__ == '__main__':+ unittest.main()