Run a virtual iPhone on an Apple Silicon Mac.
Swift
Record the VM disk-space study in Research
Jackyd2f85f49f23a
diff --git a/Research/Host/content_addressed_block_store.md b/Research/Host/content_addressed_block_store.mdnew file mode 100644index 00000000..bbd5b64f--- /dev/null+++ b/Research/Host/content_addressed_block_store.md@@ -0,0 +1,254 @@+# Content-addressed block store over NBD: why it was not done++Whether serving machine disks from a deduplicating, compressing block store+through Virtualization's NBD client would save enough over templates to be+worth a storage subsystem. Studied on 2026-10-07 by measuring two stopped+images, probing Virtualization's NBD client with throwaway VMs and a+prototype server outside the repository; no vphone machine was booted on it.+Part of the [disk-space study](disk_space.md).++## Summary++Not now. Templates with new-identity clones already take most of the saving+(five machines of one build: about 102 GB → 20 GB). On top of that, a block+store would save roughly another 30–40%, almost all of it from two sources:++- compression (about 30%), which only a store can do for live disks;+- deduplication across builds (about 17% at 4 KiB), which matters only while+ several builds are kept.++That is 7–8 GB per build with one build, 25–50 GB with three builds of 5–10+machines each. The costs, several of them set by Virtualization and not by+the prototype:++1. The NBD client hands the guest a write-through disk and never sends FLUSH+ or FUA, so a server must make every write durable before replying or+ accept losing acknowledged writes on a host crash.+2. The host cannot attach an NBD disk. `cfw install`, `cfw+ update-environment`, `cfw update-kernel`, the offline trim, `vm rebase`,+ snapshots, clones, export, import and the running-machine check all treat+ `Disk.img` as a file. Every offline operation would have to materialize a+ 20 GB raw image and write it back.+3. If the server is gone for more than about 30 s, the client gives up for+ good and every later guest I/O fails.+4. A garbage collector, index and map memory, log replay at start, and one+ more user-space hop per I/O.++## Potential, measured++Two `vm clone` copies of stopped machines (iPhone99,11 iOS 27.0 24A435 and an+iPad on iOS 26.6.2 23G90, both booted before) were attached read-only, their+System (with the `orig-fs` snapshot), Preboot, xART, Hardware and Update+volumes mounted read-only, and every file's blocks (resource forks of+decmpfs files included) mapped with `F_LOG2PHYS_EXT`; 343 sampled files read+back identical through their physical addresses. Unmapped non-zero blocks are+Data, User and container metadata. Fixed-size chunks were hashed (SHA-256,+zero chunks skipped) and 1 in 32 first occurrences compressed with+libcompression.++Both images together, 42 GB of non-zero data:++| Chunk | After dedup | Ratio | LZ4 | LZFSE | ZLIB | Index entries |+| --- | --- | --- | --- | --- | --- | --- |+| 4 KiB | 35.9 GB | ×1.17 | 27.4 GB | 24.9 GB | 24.1 GB | 8.76 M |+| 16 KiB | 39.8 GB | ×1.07 | 29.3 GB | 26.5 GB | 26.1 GB | 2.43 M |+| 64 KiB | 41.8 GB | ×1.03 | 29.8 GB | 26.8 GB | 26.6 GB | 0.64 M |++| Region | iPhone 27.0 | iPad 26.6.2 | iPad found in iPhone (4K / 16K / 64K) | 4 KiB compression (LZ4 / LZFSE / ZLIB) |+| --- | --- | --- | --- | --- |+| System, with snapshot | 15.67 GB | 13.73 GB | 3.00 / 0.64 / 0.13 GB | 0.76 / 0.69 / 0.67 |+| Data, User, metadata | 4.72 GB | 7.95 GB | 0.72 / 0.29 / 0.16 GB | 0.77 / 0.69 / 0.66 |+| Preboot | 0.06 GB | 0.05 GB | 0.04 GB | 0.74 / 0.66 / 0.63 |++- Cross-build dedup is about 22% of System and 9% of Data at 4 KiB, and+ nearly vanishes at 16 KiB and above: identical files sit at random phases+ relative to the chunk grid. Aligning the grid to the container start+ (20,480 bytes) gave the same figures. A store would need 4 KiB chunks (a+ large index) or content-defined chunking.+- Identical data at the same offset between the two builds: 0.00 GB. `vm+ rebase` cannot share anything between templates of different builds; it+ helps only separately restored machines of one IPSW (about 8.7 GB a pair),+ so it does not compete with a store.+- Within one image: the iPhone's Data/User region has 1.88 GB (40% of the+ region) duplicating other chunks in the same region, probably copies+ between the Data and User volumes (unverified); System has 0.17 GB.+- Compression: System 4 KiB chunks compress to about 0.69 with LZFSE (many+ root file system files are already decmpfs-compressed; the dyld cache is+ not). Larger chunks compress slightly better (0.66 at 64 KiB) at the cost of+ read amplification.++### Data and User blocks++Data and User report FileVault locked on the host, and were expected to be+incompressible ciphertext. They are not uniformly so. On the iPhone image+about 2.0 GB of the region is compressible like plaintext (LZ4 0.55); chunks+begin with `SQLite format 3` (429), `bplist00` (4,006), a PNG signature (874)+and `caff` (711); 2.6 GB is high-entropy. On the iPad image, 3.5 GB+compressible and 4.2 GB high-entropy; 9% of the iPad's Data chunks also occur+in the iPhone's. This suggests guest file contents on virtio storage reach the+disk image unencrypted while metadata and keys are protected, but it was not+verified which files these are or why. See+[the open question](disk_space.md#observation-to-verify-plaintext-in-data-and-user-blocks).++### Projected host bytes++Template 18.6 GB never booted; 0.35 GB per clone after its first boots (3 GB+for "used for a while"); 20.5 GB per separately restored machine. Store at+4 KiB with LZFSE: first template 18.6 × 0.894 (dedup within the image) × 0.69+≈ 11.5 GB, each further build's template 18.6 × 0.81 × 0.69 ≈ 10.4 GB (likely+less between adjacent builds of one device, unmeasured), each clone's+divergence × 0.7 (compression only). IPSW caches not counted.++| N per build, K builds | Separate restores | Template + clones (+ rebase) | Store | Store saves |+| --- | --- | --- | --- | --- |+| 5, 1 | 102 | 20.4 | 12.7 | 7.7 |+| 10, 1 | 205 | 22.1 | 14.0 | 8.1 |+| 5, 3 | 308 | 61.1 | 36.0 | 25 |+| 10, 3 | 615 | 66.3 | 39.7 | 27 |+| 5, 3, 3 GB per clone | 324 | 101 | 64 | 37 |+| 10, 3, 3 GB per clone | 648 | 146 | 95 | 51 |++Without compression, dedup alone adds only the cross-build ~17% (about 2 GB+with one build): the store's gain is mostly compression. A store with 4 KiB+chunks also avoids the up to fourfold amplification of a clone diverging in+16 KiB pages ([disk rebase](disk_rebase.md#the-unit-is-the-16-kib-page-not-the-4-kib-block)),+but a clone's first-boot divergence is only 0.35 GB.++## Virtualization's NBD client++From the SDK headers, strings of `com.apple.Virtualization.VirtualMachine.xpc`,+an EFI probe VM with a read-only disk to watch the handshake, and a Linux+guest to watch real I/O. Both probes were ad hoc signed with only the public+`com.apple.security.virtualization` entitlement.++| Question | Finding |+| --- | --- |+| URLs | `nbd://host[:port]/export`, `nbd://[::1]/…` and `nbd+unix:///export?socket=/path` work (validated and connected). `nbds://`, `nbds+unix://`, `nbd+tcp://` and `nbd+vsock://` are "not supported": no TLS, no authentication. Unix socket paths are limited by `sun_path` (104 bytes) |+| Handshake | Fixed newstyle only ("Old style negotiation is not supported", "Non-fixed new style negotiation is not supported"). Client flags `FIXED_NEWSTYLE \| NO_ZEROES`; `NBD_OPT_INFO`, then `NBD_OPT_GO`, both asking for `NBD_INFO_BLOCK_SIZE`. No structured replies, extended headers, meta contexts or STARTTLS, so no `BLOCK_STATUS`. Export size must be a multiple of 512 |+| Block size | The server's advertised minimum block size becomes the guest's logical block size (4096 made EFI read 4K; 16384 was accepted). vphone images have a GPT with 512-byte LBAs, so a server must advertise a minimum of 512 or less |+| Commands | READ, WRITE and TRIM seen; with `SEND_TRIM`/`SEND_WRITE_ZEROES` advertised, the guest saw discard and write-zeroes limits equal to the disk size. **No FLUSH, and FUA never set**, with synchronization mode full or none. The Linux guest saw `write_cache = write through`; on a raw `VZDiskImageStorageDeviceAttachment` the same kernel saw `write back`. The strings mention an "NBD Flush request timer" that never fired |+| Concurrency | One connection with up to 8 requests in flight (Linux, one queue, 8 parallel writers); requests up to 4 MiB, all 4K-aligned |+| Who connects | `LOCAL_PEERPID` is the app process, while the client code (`Storage::NetworkBlockDeviceConnection`) is in the XPC service, so the descriptor is presumably passed over. An unsandboxed process connects over TCP and Unix sockets without `com.apple.security.network.client`; `vphone-vm` is not sandboxed |+| Server down at start | `start` fails: "The storage device attachment is invalid." No retry |+| Server down while running | 6 s: guest I/O stalls, then resumes transparently, while the delegate receives `Code=20001 Failed to negotiate with server` three times (the header calls such errors unrecoverable). 45 s: after about 33 s the client gives up; every later guest I/O is an `I/O error`, and it does not reconnect when the server returns |+| Server returns EIO | Passed straight to the guest as an I/O error, no retry |+| Boot disk of a PV=3 research VM | Not tested. The storage path is the same `VZVirtioBlockDeviceConfiguration` and iBoot reads Preboot through virtio-blk, so it should work with a 512-byte minimum block |++Guest throughput (Linux, 256 MiB disk; the NBD server kept data in the page+cache, so these rows pay no durability cost and are not comparable to raw):++| | 4K O_DIRECT write | 4K O_DIRECT read | 1 MiB write | 1 MiB read |+| --- | --- | --- | --- | --- |+| Raw `VZDiskImage`, sync full | 74 MB/s | 61 MB/s | 1.0 GB/s | 3.2 GB/s |+| NBD over a Unix socket | 117 MB/s | 130 MB/s | 5.0 GB/s | 6.9 GB/s |+| NBD over TCP | 33 MB/s | 72 MB/s | 2.8 GB/s | 3.0 GB/s |++## Prototype and benchmarks++The prototype, in C with system libraries only and outside the repository:+an NBD server (fixed newstyle; GO, INFO, EXPORT_NAME, LIST, ABORT; READ,+WRITE, FLUSH, TRIM, WRITE_ZEROES, DISC, FUA honoured; one reader thread per+connection, requests on a concurrent queue, FLUSH and TRIM as barriers) over+either a raw file (TRIM as `F_PUNCHHOLE`) or a log-structured store: one+append-only log of data records (SHA-256 truncated to 128 bits, LZ4 or LZFSE,+stored raw unless it saves 10%) and map records (one per request, listing+chunk → entry), with the hash index and map rebuilt by replaying the log.+Replay stops at the first record that fails its checksum and truncates there;+a map record may only reference verified data records, so recovery yields a+prefix of acknowledged writes. A read-only base file passes through, so a+clone is a map rather than a full import.++Import of 2 GiB sampled across an iPhone image (32 × 64 MiB): 4K + LZ4+1.55 GB at 540–800 MB/s, 4K + LZFSE 1.44 GB, 4K uncompressed 1.93 GB, 16K or+64K + LZ4 1.56 GB at 1.6–2.2 GB/s. Two writers racing to store the same new+content left about 0.07% duplicate records.++Loopback, data in the page cache, 2 GiB volume:++| | 4K rand read QD1 | 4K rand read QD32 | 16K rand read QD32 | 4K rand write QD1 | 4K rand write QD32 | 1 MiB seq write | Flush after random writes |+| --- | --- | --- | --- | --- | --- | --- | --- |+| Direct `pread`/`pwrite` | 0.8 µs | 0.9 M IOPS | 14.7 GB/s | 6.8 µs | 199 k | 3.2 GB/s | 0.33–0.6 s |+| NBD Unix, raw file | 10.9 µs | 229 k | 3.2 GB/s | 14.6 µs | 147 k | 0.83 GB/s | 0.8–1.3 s |+| NBD TCP, raw file | 25.6 µs | 108 k | 1.4 GB/s | 37.5 µs | 83 k | 0.75 GB/s | 0.6–0.75 s |+| NBD Unix, store 4K | 10.0 µs | 223 k | 2.9 GB/s | 21 µs | 133 k | 0.79 GB/s | 8–14 ms |+| NBD Unix, store 16K | 12.2 µs | 210 k | 3.3 GB/s | 39 µs (read-modify-write) | 104 k | 0.79 GB/s | 6–10 ms |++Durable before each reply:++| | 4K write QD1 | 4K write QD32 | 1 MiB seq write |+| --- | --- | --- | --- |+| Store, `F_BARRIERFSYNC` | 977 IOPS (1.0 ms) | 8.5 k | 169 MB/s |+| Store, `F_FULLFSYNC` | 271 IOPS (3.7 ms) | 484 | 226 MB/s |+| Raw file, `F_BARRIERFSYNC` | 883 | 7.9 k | 725 MB/s |+| Raw file, `F_FULLFSYNC` | 255 | 621 | 207 MB/s |++- The protocol and thread hops add 10–25 µs per operation. Reads from the+ store are as fast as raw (LZ4 decode is cheap); writes pay hashing and+ compression, 10–40% slower, twice that for 4K writes into 16K chunks.+- The log turns random writes into appends, so a flush takes milliseconds+ where a raw file takes about a second. That advantage disappears once every+ write must be durable, which a write-through client demands: a raw image's+ guest pays one `F_FULLFSYNC` per APFS checkpoint, a store about 1000 IOPS at+ QD1. Group commit (one barrier per ~1 ms batch) helps only at QD > 1.+- The log grows: about a million 4K random overwrites on a 2 GiB volume made+ an 8.7 GB log. Garbage collection is required, not optional.+- The first version wrote map records only on FLUSH. A guest wrote 1 MiB and+ synced, the server was killed and restarted, the client reconnected, and the+ guest read back zeros: acknowledged writes silently lost, because the client+ never flushes. Writing a map record before each reply fixed it. Host power+ loss was not tested.++A real vphone boot from NBD was not attempted: it needs a `vphone-vm` build+with a changed attachment and therefore a new cdhash admitted by the AMFI+allowlist. The Linux guest covered reads, writes, TRIM, concurrency,+reconnection, the timeout and EIO on the real client.++## If it were built++- **Server:** not inside `vphone-vm` (a VM crash would take its storage with+ it, and one store per process cannot deduplicate across machines). One+ per-user helper in the bundle, started on demand, one Unix socket per+ machine with a path under 104 bytes. `vm launch` must see the socket+ listening first. The ~30 s client timeout bounds helper restarts, upgrades+ and GC pauses.+- **Consistency:** log-structured with replay, as the prototype; map+ checkpoints to bound replay; group commit; a stated contract ("after a host+ crash the disk returns to a consistent state a few ms old", or per-write+ durability at about 1/100 of the speed). APFS tolerates a consistent prefix,+ as it does a raw image with synchronization off.+- **GC:** mark from all maps and snapshot maps, copy live records into new+ segments, segment by segment. TRIM only clears map entries; space returns+ at GC, unlike a raw file where TRIM punches at once.+- **Snapshots and clones:** a copy of the map (125 MB dense for 64 GB at 4K,+ much less sparse). Attractive, but `clonefile` already does it for free.+- **Host maintenance:** the largest cost. Every offline verb becomes+ materialize, modify, write back, needing about 20 GB of scratch each time on+ hosts that are often short of space.+- **Failure modes:** a helper down for more than ~30 s fails every machine's+ disk; log corruption truncates to the last verified prefix (data goes back+ in time); a full host disk returns ENOSPC to the guest; one storage bug hits+ every machine at once.+- **Upkeep:** a versioned on-disk format with migrations, helper lifecycle and+ permissions, GC scheduling, Launchpad's size accounting, crash-injection+ tests, and a new minor series because Launchpad would depend on it.++## Cheaper alternatives++1. Templates, the offline trim, and deleting IPSWs (13.5 GB per build) once a+ template exists: all in place.+2. Compress cold templates: archive the `Disk.img` of a template no one clones+ any more (LZFSE, about 30–35% smaller by the figures above) and expand it+ back before the next clone. That takes the store's largest gain, compression,+ without touching the I/O path, host mounts or the CFW tools. Only "three or+ more builds all in use" misses out. Not built.+3. Cross-build dedup (~17%) needs a store or host block-range cloning, which+ macOS does not offer publicly. Left alone.++## When to revisit++- Disk space is still the limit with templates, trim and IPSW deletion in+ place, and three or more builds of ten or more machines each are kept in+ use at once; or+- Apple provides a host-mountable user-space block device (a DiskImages2+ plug-in, or NBD the host can attach); or+- Virtualization's NBD client gains write-back with FLUSH.diff --git a/Research/Host/deterministic_disk_layout.md b/Research/Host/deterministic_disk_layout.mdnew file mode 100644index 00000000..e0624db7--- /dev/null+++ b/Research/Host/deterministic_disk_layout.md@@ -0,0 +1,178 @@+# Deterministic disk layout: why it was not done++Two machines restored separately from one IPSW hold mostly the same bytes,+but only 44% of them at the same offset in `Disk.img`, and `vm rebase` can+share only same-offset blocks ([disk rebase](disk_rebase.md)). This note+records what the shifted bytes are, why they shift, what it would take to+pin them, and why that was not worth it once templates existed. Studied on+2026-10-07 by measurement, reading and offline allocation experiments, with+no restore run. Part of the [disk-space study](disk_space.md).++## Summary++- The shifted region is mainly the OS and App cryptexes that `cfw install`+ copies onto the System volume on the host: 7.35 GB (dyld cache 7.18 GB) of+ the 9.9 GB that differs only in position. The other ~2.5 GB is Data, User+ and some Preboot content the guest writes during the restore's `fixup_var`+ and the first boot.+- Restore writes the same large data in both machines. What differs are+ per-device small writes (the APTicket is 5805 bytes on one and 5806 on the+ other, ECID-derived files, keybags, log timestamps) and the guest's own+ concurrency during restore.+- The host's APFS allocator is deterministic but chaotic: a few blocks of+ difference before a large copy move all of it by hundreds of MB.+- The only way to pin the cryptexes is to merge them into the root file+ system before the restore, so ASR writes them block for block. That would+ raise same-offset sharing to about 81% (15.8 of 19.4 GB), not more.+- Not done: with templates, separately restored machines are the exception,+ and the cost is a rebuilt, resealed root file system per IPSW.++## What shifts++Method: two booted machines, iPhone99,11 iOS 27.0 (24A435) + cloudOS 26.4,+standard preset, each attached read-only with System and Preboot mounted+read-only (Data and User are locked). Each file's physical extents+(`F_LOG2PHYS_EXT`, plus the partition offset of 20,480 bytes) gave its offset+in the image; a sample was checked byte for byte. The 217,000+decmpfs-compressed root file system files cannot be mapped this way; they sit+in the ASR region, at the same offset in both.++Layout of one image:++| Image offset | Content |+| --- | --- |+| 0 – 1.07 GB | APFS metadata, Preboot, guest-written Data |+| 1.07 – 9.93 GB | The root file system ASR wrote and `apfs_invert` turned into the System volume: the same offset in both machines |+| Small holes inside that | Small cryptex files (even-numbered dyld subcaches of 0.2–0.5 MB, the atlas) filling free extents inside the ASR image |+| 9.93 – 10.63 GB | About 600 MB of unmapped plaintext, guest-written Data/User |+| 10.63 – 16.50 GB | dyld subcaches `.21`–`.77`, `.dyldlinkedit` and `.symbols` (1.22 GB): 30 files, 5.85 GB, one extent each |+| 16.50 – 19.9 GB | About 3.4 GB unmapped: first-boot Data/User |+| 62.0 – 64.0 GB | Odd subcaches `.01`–`.19` (about 1.3 GB), the DriverKit cache, a QuartzCore metallib and a few CFW-modified System files. In the guest this is the APFS metazone (`metazone … encrypted: 15136710-15380850` in the restore log); the host's mount does not reserve it, so the first large files copied land there |++The two machines' 81 dyld cache files had identical SHA-1s (CFW from two+different bundle builds patched them identically), yet moved:++| Group | Size | Shift (B − A) |+| --- | --- | --- |+| Subcaches `.21`–`.77`, `.symbols` | 5.85 GB | −6415 blocks (−26.3 MB) |+| Subcaches `.07`–`.19` (metazone tail) | 0.92 GB | −36 blocks |+| `.03`, `.05` | 0.26 GB | −1 block |+| `.01` | 0.13 GB | −219 blocks |+| 224 small cryptex files | 8 MB | −1358 blocks |+| Uncompressed root file system files | 4.55 GB | 4.49 GB at the same offset |++A content-hash comparison at any offset gave: of B's 19.4 GB non-zero data,+8.45 GB at the same offset as A (44%), 9.89 GB present in A elsewhere, 1.04 GB+unique to B. By region of B:++| Region of B | Same offset | Shifted | Unique |+| --- | --- | --- | --- |+| 0 – 1.07 GB | 0.00 | 0.38 | 0.24 |+| 1.07 – 9.93 GB (ASR) | 8.43 | 0.26 | 0.01 |+| 9.93 – 16.37 GB (cryptex bulk) | 0.02 | 6.12 | 0.09 |+| 16.37 – 19 GB (Data/User) | 0.00 | 1.50 | 0.66 |+| 62 – 64 GB (cryptex tail) | 0.00 | 1.63 | 0.04 |++The 1.5 GB of shifted Data/User content in B lies in A at 0.6–1.08 GB,+3.4 GB, 10.3–10.6 GB and elsewhere, displaced by tens of MB to 16 GB: the+guest's allocator put identical content into different holes.++## Why it shifts++The order of writes: restore partitions and creates the container,+preallocates a 10.1 GB `/mnt1/apfs_invert_asr_img`, streams ASR into it and+inverts it into the System volume; then installs the kernelcache, device tree,+root hash and ticket into Preboot, runs `fixup_var` on Data, the recovery OS+restore, `seal_system_volume`, `create_system_snapshot` and+`split_data_volume`. Afterwards `cfw install` on the host copies the cryptexes+(`installCryptexes`: `VPhoneConfinedDirectory.copyTree`, names sorted,+depth-first, `fcopyfile(COPYFILE_DATA)` per file). The dyld cache patches+after that write in place and allocate nothing. So where the cryptexes land+depends only on the container's allocation state when the restore ends and on+the host allocator.++The two restores' space-manager statistics, from their logs:++| Point | A | B |+| --- | --- | --- |+| Container created | 15,595,537 free / 10 extents | same |+| After invert | 13,078,288 free / 13 extents | same |+| Around seal | 13,265,750 / 4625 extents → 13,157,004 | 13,265,759 / 4624 → 13,157,013 |+| `split_data_volume` | 13,156,553 / 4538 extents | 13,156,562 / 4530 extents |+| Restore end | 13,251,848 | 13,251,857 |++Nine blocks apart. Step durations differed widely (`fixup_var` 6 s against+22 s, `seal_system_volume` 25 s against 94 s), which changes which small file+lands in which hole; Preboot showed the same firmware file at different+places in the two.++### The host allocator, offline++On a scratch 20 GB raw APFS image (`diskutil image create blank --format RAW+--fs APFS`), each trial cloned an identical copy, mounted it and copied A's+`System/Cryptexes/{OS,App}` (7.35 GB, 3345 extents) with a C reimplementation+of `copyTree`, then mapped the result:++| Trial | Same offset as the first run | Note |+| --- | --- | --- |+| The same copy twice more (7 s and 17 s) | 100% | Same state and order: fully deterministic, timing does not matter |+| `F_PREALLOCATE` every file first | 100% | Preallocation does not change placement |+| An 80 MB file written first | 1.4% | Most moved +1 to +5 MB |+| Three small files (1 + 2 + 3 blocks, 24 KB) first | 0.6% | 7.2 GB moved +268.4 MB (65,536 blocks: the start of the 3-block allocation zone) |+| Eight small files (144 KB) first | 0% | Moved +402.7 MB (98,304 blocks) |+| Container padded to a fixed 1000 MB used first | 99.4% | Works only if the earlier difference was one contiguous file |+| Padding plus three small files | 0% | Moved +309 MB |++Placement follows the free-extent table and the per-size allocation-zone+cursors. A few blocks of difference shift a large file by hundreds of MB.+The real machines differ by one to a few thousand blocks, so their shifts are+small, but for same-offset sharing one block is as bad as a gigabyte.++## Ways to pin it++| Approach | Result |+| --- | --- |+| Fixed copy order on the host | Already fixed (sorted). The start state is the problem |+| `F_PREALLOCATE` | Changes nothing (trial above) |+| Copy before the variable writes | Those happen inside the guest's restore, before `cfw install` can run |+| Pad the container to a fixed usage, or fill below a boundary | Defeated by a few small files, which is exactly the real difference |+| Write blocks to fixed offsets on the host and fix up APFS metadata | Means writing an APFS writer (fs tree, extent references, space manager, checkpoints) |+| Clone ranges between files on the host | macOS has no public range-clone call; `clonefile` is whole-file |+| **Merge the cryptexes into the root file system before the restore** | Works in principle: the ASR region already matches block for block, and both containers had identical free-extent statistics after the invert. The `.less` variant's `CryptexFilesystemPatcher` does this merge (mount OS, SystemOS and AppOS, copy, rebuild trust cache, mtree, digest and root hash, `Ap,SystemVolumeCanonicalMetadata`), but it runs only for `.less` (`FirmwarePipelineComponents.swift`). After the merge `installCryptexes` finds both directories populated and only creates the links, and the in-place dyld cache patches keep identical bytes identical |+| The cryptexes as a second virtual disk per IPSW, cloned per machine | Would share 100% regardless of layout, but dyld needs the cache before launchd, so the guest would have to mount it at early boot: a redesign of how CFW wires in the cryptexes. Untested, highest risk |++The ASR merge was not prototyped. It needs, per IPSW, decrypting and+rebuilding a 16–17 GB root file system and a real restore to check boot, and+these questions answered first:++- whether TSS signs a changed `OS`, `SystemVolume` and+ `Ap,SystemVolumeCanonicalMetadata` on the JB path (the JB kernel already+ bypasses the root snapshot seal check, `authapfs_seal_is_broken` and the+ graft's root hash check, and iBoot's image4 callback is bypassed);+- the preallocated invert image growing from 10.1 GB to about 17.5 GB, and ASR+ transfer from 62–95 s to roughly 2–3 minutes;+- how much of the merged cryptex content ends up compressed (System volume+ size only; it does not affect determinism).++## Decision++- Reachable same-offset sharing: about 81% (8.45 + 7.35 ≈ 15.8 of 19.4 GB),+ not the 96% that is identical at some offset. The ~2.5 GB the guest writes+ is placed by the guest's allocator and spreads further with every boot.+- Cost: a decrypted, merged and resealed root file system per IPSW (minutes+ of work and a 16–17 GB artifact to cache or rebuild), about 80% more restore+ transfer, the `.less` merge path revived and maintained for JB, and+ unverified TSS and boot-chain risk.+- Gain: about 7.35 GB more shared per separately restored machine, and only+ after `vm rebase`. Template clones already share nearly everything; separate+ restores remain for machines that need their own SEP secret+ (`--no-template`).++Not done. If separate restores become the norm, start from the ASR merge and+verify TSS and boot on one test machine first. Until then `vm rebase` between+separate restores recovers what happens to line up: 8.7 GB (about 43%) for+the pair measured here. One later rebase of a separately restored machine+onto a template of the same build shared 13.7 GB of about 18.2 GB+([disk rebase](disk_rebase.md#measurements)), which suggests the cryptex copy+landed at the same offsets there, as it would if the two restores happened to+end in the same allocation state; that case was not analysed.diff --git a/Research/Host/disk_rebase.md b/Research/Host/disk_rebase.mdindex 5122b965..62cc0078 100644--- a/Research/Host/disk_rebase.md+++ b/Research/Host/disk_rebase.md@@ -16,6 +16,11 @@ restored from iOS 27.0 (24A435) on cloudOS 26.4 (23E5207q): of about 20 GBof data in each 64 GB sparse `Disk.img`, 8.6 GB is byte-identical at the sameoffset, mostly the ASR-restored system volume between roughly 1 GB and 9.7 GBinto the image. Physical sharing between them, by `F_LOG2PHYS_EXT`, was 0.+Most of the rest is identical too, but at other offsets: chiefly the cryptexes+`cfw install` copies on the host, which a few blocks of difference in the+restore's end state move as a whole (see+[deterministic disk layout](deterministic_disk_layout.md)). A rebase cannot+share those.## Method@@ -213,3 +218,25 @@ note: the rebase would free almost nothing: rb-c already shares its identical bl2,097,212 KiB (2.15 GB); a dry run of it afterwards reported 0.00 GB newlyand 2.15 GB already shared. Mapping both images took well under a second.++A separately restored machine onto a template, 2026-10-09: a `--no-template`+machine (no trim, no setup boot) and a trimmed template, both built from iOS+27.0 (24A435) / cloudOS 26.4 by the same bundle build, the machine stopped.+`vm rebase <machine> --onto <template id> --dry-run` took 4 s and reported+13.70 GB newly shared, 0.03 GB already shared, 4.46 GB written and 0.11 GB+punched; the rebase itself took 10 s and printed the same figures. `df` rose+by 13,445,384 KiB (13.77 GB); the newest Time Machine local snapshot predated+the machine, so none held the old blocks. The template's `Template.plist`,+`Disk.img` size and modification times and its `vm template show` output were+unchanged, and it listed only its own clones as users. The machine got no+`TemplateSource.plist`, kept its `SEPStorage`, `nvram.bin` and `config.plist`+untouched, booted with its own UDID with vphoned answering in 5 s and no+panic. A dry run afterwards reported 0.00 GB newly and 13.72 GB already+shared, with the "free almost nothing" note; so did one onto a clone of the+template.++13.70 GB is well above the 8.7 GB two separately restored machines shared:+in this pair the host's cryptex copy probably landed at the same offsets+too, which the layout study found depends on the two restores ending in the+same allocation state ([deterministic disk layout](deterministic_disk_layout.md)).+Why it did here was not analysed, so expect anything between the two figures.diff --git a/Research/Host/disk_space.md b/Research/Host/disk_space.mdnew file mode 100644index 00000000..77c2a3a5--- /dev/null+++ b/Research/Host/disk_space.md@@ -0,0 +1,293 @@+# Where a VM's disk space goes++The disk-space study of 2026-10-06 to 2026-10-09: what a machine's ~20 GB is+made of, what each way of saving it was measured to save, and why one+template per firmware with new-identity clones became the default. The+mechanisms have notes of their own, linked below; this note keeps the+composition, the comparison and the end-to-end numbers.++Unless stated otherwise the guests are iPhone17,3 (or iPhone99,11) on iOS 27.0+(24A435) with cloudOS 26.4 (23E5207q), standard preset, a 64 GB sparse+`Disk.img`, on an APFS host volume. Block sharing was measured with+`fcntl(F_LOG2PHYS_EXT)` on both files ("physically shared"), not with `du`.++## Summary++- A restored machine that has booted once allocates about 20.5 GB. About+ 8.6 GB of it is the IPSW's root file system, 7.2 GB the OS and App+ cryptexes `cfw install` copies onto the System volume, and 2–5 GB the Data+ and User volumes' first-boot state. Building one also needs the ~11 GB+ restore tree and 13.5 GB of IPSWs.+- Space the guest frees comes back to the host by itself, within about 15 s+ (guest TRIM). There is nothing worth compacting offline+ ([guest TRIM](guest_discard.md)).+- The saving that matters is sharing. A template per firmware, and every+ machine a `clonefile` copy of it with a new ECID, costs each machine about+ 0.35 GB after its first boots instead of 20.5 GB+ ([templates](machine_templates.md),+ [identity and clone](machine_identity_and_clone.md)).+- Everything else is small next to it and counts once per template, not per+ machine: the offline file trim 1.2 GB, the orig-fs snapshot itself about+ 21 MB, deleting the IPSWs 13.5 GB per build. `vm rebase` gets back about+ 8.7 GB per pair of machines restored separately ([disk rebase](disk_rebase.md)).+- Two further approaches were studied and not built: making the restore's+ disk layout deterministic ([deterministic layout](deterministic_disk_layout.md))+ and a content-addressed block store behind Virtualization's NBD client+ ([block store over NBD](content_addressed_block_store.md)).++## What a machine holds++### The machine folder++| Item | Size | Note |+| --- | --- | --- |+| `Disk.img` | 64 GB logical | Sparse. About 18.6 GB allocated after `cfw install` and before any boot; about 20.5 GB after the first boot (2026-10-07, two machines) |+| `nvram.bin` | 33.6 MB | Boot firmware and NVRAM; cloned, diverges by a few blocks |+| `SEPStorage` | 512 KB | |+| `.vphoned.signed` | 9.3 MB | The staged vphoned |+| `iPhone*_Restore/` | about 11 GB | Only while a machine is built; removed afterwards unless `--keep-artifacts` with `--no-template` |+| IPSW cache (outside the folder) | 12.3 GB iPhone + 1.2 GB cloudOS | Per build, shared by every machine of it |++### Inside `Disk.img`++Attached read-only (`diskutil image attach --readOnly --noMount`), a booted+machine's container showed:++| Volume | In use | Content |+| --- | --- | --- |+| System | 16.2 GB (Sealed: Broken) | Below |+| Data | 1.2–2.5 GB | FileVault, locked on the host |+| User | 0.5–2.7 GB | FileVault, locked on the host |+| Preboot | 61 MB | kernelcache 43 MB, SEP and other firmware 17 MB; `Preboot/Cryptexes/{OS,App}` empty |+| xART, Hardware | 10.5 MB, 6.3 MB | Gigalocker; activation UUID and ticket |+| Update | | Not mounted |++The container's "In Use" (21.0 GB) matched the image's allocated size: the+blocks `cfw install` freed on the host had been punched out of the file, so a+fresh machine carries no dead space.++The System volume is two layers:++- The sealed root file system restore wrote (ASR, then `apfs_invert`). CFW+ renames its snapshot to `orig-fs.disabled.rn-<hash>`, and the patched kernel+ boots the live volume. The snapshot still references the original 8.59 GB;+ of the CFW-modified files, only 35 MB of old versions are held by nothing+ else (21.5 MB measured when the snapshot was deleted, see+ [snapshot deletion](../Guest/template_snapshot_deletion.md)).+- The OS and App cryptexes (7.2 GB on the live volume), which `cfw install`+ copies from the restore tree into `System/Cryptexes/{OS,App}` and links in+ (`System/Library/Caches/com.apple.dyld` points into+ `System/Cryptexes/OS/…`). vphone boots without the cryptex graft+ (`cryptex1 sniff: ignition failed: 8` on every boot), so the System volume+ copy is the only one dyld finds. The snapshot does not hold these blocks.++The live volume by kind, deduplicated by inode, `st_blocks` (2026-10-07):++| Kind | Size | Layer |+| --- | --- | --- |+| dyld shared cache: 71 subcaches, data, readonly, linkedit, atlas | 5,955 MB | cryptex |+| dyld shared cache `.symbols` | 1,223 MB | cryptex |+| PrivateFrameworks (without localizations and models) | 2,989 MB | root fs |+| ML models (`.mlmodelc`, `.espresso`, …) | 1,021 MB | root fs |+| Other CoreServices, ExtensionKit, PreferenceBundles and the like | 902 MB | root fs |+| `LinguisticData` | 558 MB (418 MB outside en and zh-Hans) | root fs |+| `.loctable` (all languages in one file) | 434 MB | root fs |+| `PreinstalledAssetsV2/RequiredByOs` | 424 MB (SharingDeviceAssets 232 MB) | root fs |+| Frameworks, the rest | 400 MB | root fs |+| `usr/standalone` | 375 MB (SU ramdisk 243 MB, baseband firmware 114 MB) | root fs |+| Fonts | 373 MB (AppleColorEmoji 139 MB) | root fs |+| `usr`, the rest | 361 MB | root fs |+| Health assets and NanoTimeKit | 350 MB (watch faces 144 MB) | root fs |+| `/Applications` | 257 MB, 219 apps | root fs |+| `.lproj` of languages other than en, Base, zh-Hans | 220 MB | root fs |+| TTS and voice, audio tunings, ringtones, app placeholders, wallpapers | 530 MB | root fs |++The dyld cache is 45% of the System volume and can only be kept or replaced+whole. Most localizations live in `.loctable` files, all languages in one+file, so trimming languages from `.lproj` folders would save about 0.3 GB at+most.++### Data and User++The host cannot mount these volumes (their keys are wrapped by the guest's+SEP), so their content was only classified by block. After the first boot+they hold 2–5 GB: the restore's `fixup_var`, the first boot's data migration,+caches, logs and the expanded removable system apps. This is the part each+clone writes for itself; see the per-clone numbers below.++## What each approach saves++| Approach | Saving | Counts | Status |+| --- | --- | --- | --- |+| Guest TRIM | Deleted guest data returns within ~15 s; ~17 MB per machine left unreclaimed | Per machine | Works as shipped ([guest TRIM](guest_discard.md)) |+| Template + new-identity clone | 20.5 GB → about 0.35 GB per machine after its first boots | Per machine | Default for `vm create` and Launchpad 2.9 |+| Setup boot in the template | A clone's first 100 s write 1,040 → 372 MiB; private bytes after 7 min 3.2 → 0.6 GB | Per machine | Part of every template build |+| Offline trim, standard tier | 1.17 GB (conservative 0.36 GB) | Per template | Default tier ([templates](machine_templates.md#offline-trim)) |+| orig-fs snapshot deletion | About 21 MB itself; makes the trim free anything | Per template | Setup boot step 0 |+| Deleting the IPSWs | 13.5 GB | Per build | Offered by Launchpad once a template exists |+| `vm rebase` of separate restores | About 8.7 GB per pair (≈ 43%); 13.7 GB measured onto a template | Per machine | `vm rebase` |+| Service trimming | Idle writes −60% (27.0), −94% (26.6.2); first-boot writes about −35% | Per machine, writes | `trimmed` profile ([service trimming](../Guest/service_trimming.md)) |+| Deterministic disk layout | Same-offset sharing of separate restores from 44% to at most ~81% | Per machine | Not done |+| Content-addressed store over NBD | A further 30–40% over templates, mostly from compression | Whole library | Not done |++Projected host bytes for N machines per build and K builds (IPSW caches not+counted; template 18.6 GB, 0.35 GB per clone after first boots, 20.5 GB per+separately restored machine; the CAS column is 4 KiB blocks with LZFSE):++| N × K | Separate restores | Template + clones | CAS |+| --- | --- | --- | --- |+| 5 × 1 | 102 GB | 20.4 GB | 12.7 GB |+| 10 × 1 | 205 GB | 22.1 GB | 14.0 GB |+| 5 × 3 | 308 GB | 61.1 GB | 36.0 GB |+| 10 × 3 | 615 GB | 66.3 GB | 39.7 GB |+| 10 × 3, 3 GB written per clone | 648 GB | 146 GB | 95 GB |++`vm rebase` does not change the template column: templates of different+builds share no block at the same offset (0.00 GB between an iPhone 27.0 and+an iPad 26.6.2 image), and clones already share their template's.++## Why templates won++A template is the only approach whose saving grows with the number of+machines and costs nothing on the I/O path: a clone is a `clonefile`, the disk+stays a raw file the host can attach for `cfw install`, `cfw+update-environment`, the trim, `vm rebase` and snapshots, and Virtualization+keeps its write-back cache. Its costs are known and documented: every clone+shares the template's SEP root secret, gigalocker and volume keys+([shared secrets](machine_templates.md#shared-secrets); `--no-template` makes+a machine with its own), a template must never boot once frozen, and the disk+size is part of the key because no code grows a container after the restore+(a larger size needs another template).++Three places to trim the System volume were considered:++- Before the restore, in the IPSW's root file system image. The repository+ has the machinery (the `.less` variant's `CryptexFilesystemPatcher`), but+ it means decrypting and resealing an 8–9 GB image per IPSW, and whether TSS+ signs a changed `SystemVolume`/`Ap,SystemVolumeCanonicalMetadata` digest on+ the JB path was never tested. CFW then rewrites the result anyway.+- After `cfw install`, on the host, with the System volume mounted: chosen.+ No guest involved, repeatable, and a raw image is hole-punched when the+ volume is unmounted (a 1 GB file written and deleted on a test image left+ 5.6 MB allocated after the unmount, 982 MB before it).+- Inside the running guest through vphoned: rejected when templates were+ still meant never to boot. The setup boot came later, for Setup and app+ removal, but the host-side trim stayed the simpler choice: it needs no+ writable System volume in the guest and no destructive RPC.++The catch was the snapshot: deleting root file system files on the host+frees nothing while `orig-fs.disabled.rn-*` exists, and the host cannot+delete that snapshot (SIP, `-69863` even as root). The guest can, so the+setup boot deletes it first ([snapshot deletion](../Guest/template_snapshot_deletion.md)).++Never trimmed: the dyld cache and its `.symbols` (the CFW cache patcher's+`DyldSharedCacheSymbolResolver` reads the local symbols, so a later+`cfw update-environment` on a clone would fail to find its patch sites), ML+models (Photos analysis, Spotlight, Live Text and the camera pipeline load+them; a missing model makes daemons crash or retry), and `/Applications`+(system UI and view services). The `aggressive` tier (other `.lproj`, Health+assets, some fonts; about 1.7 GB with the standard items) stays reserved because it was never+booted.++## Measured end to end++Template builds from scratch, Launchpad 2.9 driving bundle 2.9 builds, IPSWs+cached, seconds per step (2026-10-09). The setup boot is mostly the guest's+first-boot data migration ([Setup Assistant skip](../Guest/setup_assistant_skip.md#the-first-boot-after-a-restore)).++| iOS | prepare | patch | restore | stop DFU | `cfw install` | trim | setup boot | clone + first boot | total |+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |+| 27.0 | 6 | 22 | 83 | 8 | 67 | 2 | 79 | 5 | 4 min 39 s |+| 27.0 | 5 | 23 | 86 | 9 | 62 | 2 | 105 | 5 | 5 min 04 s |+| 27.0 | 5 | 23 | 114 | 8 | 58 | 2 | 89 | 7 | 5 min 14 s |+| 26.6.2 | 4 | 19 | 80 | 8 | 199 (about 160 s of it an administrator prompt) | 2 | 95 | 5 | 6 min 59 s |++The third 27.0 build's setup boot: connect 4 s, setup-skip 0 s, settle 63 s,+remove apps 5 s, service profile 0 s, reboot 10 s, verify 1 s, crash reports+1 s, stop 5 s. The first two ran the earlier `setup.skip`, which retried for+45–98 s while settle then took about 10 s+([setup boot](machine_templates.md#setup-boot)); the setup boot took about as+long either way, because the wait is the migration.++Machines from an existing template:++| Path | Time |+| --- | --- |+| `vphone-cli vm create --template <id> --skip-first-boot` | under 1 s (the clone itself 0.06 s) |+| Launchpad New Machine (find, clone, first boot until vphoned answers) | 6–9 s |+| Launchpad, IPSWs deleted (template found by its recorded sources) | 6 s |+| Before templates, a full restore per machine | 3.5–5 min iPhone, about 7.5 min iPad |++Sizes:++| Object | Allocated | Note |+| --- | --- | --- |+| Template, standard trim, setup boot | 17.4–17.7 GB | `Disk.img` 17.55 GB plus 43 MB of other files |+| Template, setup boot, no trim | 19.3 GB | |+| Template adopted after a plain first boot, no trim | 18.9 GB | Clones then reach 3.2 GB private after 7 min |+| Clone, 35 s after its first start | 0.34–0.37 GB private | |+| Clone, after 7 minutes | 0.59–0.61 GB private | The same with and without the trim: the trim saves per template |+| Clone of a stopped machine, two boots | 0.33–0.52 GB private | `vm clone --new-identity` |++The template's own private bytes stay at about 0.1 GB, the blocks every clone+has rewritten. Booting a template after clones exist raised one clone's+private bytes from 0.42 to 0.80 GB, which is why a frozen template never boots+([templates](machine_templates.md#never-booting-once-frozen)).++## Measuring it++- `du`, Finder and a folder's allocated size count every clone at its full+ ~18 GB. `df` before and after an operation is the honest total.+- Per object, compare physical extents (`F_LOG2PHYS_EXT`), as Launchpad's+ Exclusive column does. APFS's private size (`ATTR_CMNEXT_PRIVATESIZE`) reads+ 0 for every object after each hourly Time Machine local snapshot, until it+ writes again ([disk use](machine_templates.md#disk-use)).+- `st_blocks` of a running machine's image is not monotonic: guest TRIM pulls+ it back. Writes by the VM are better read from the Virtualization process's+ `ri_diskio_byteswritten` (`proc_pid_rusage`).+- Deleting a template frees only its own blocks while clones exist; deleting a+ source frees nothing a clone still shares.++## Observation to verify: plaintext in Data and User blocks++The block-level survey for the block-store study+([content-addressed store](content_addressed_block_store.md#data-and-user-blocks))+found that the image region not mapped to System, Preboot, xART, Hardware or+Update files (Data, User and container metadata), although both volumes show+as FileVault locked on the host, is not uniformly ciphertext. On the iPhone+27.0 image about 2.0 GB of it compressed like plaintext (LZ4 ratio 0.55), and+blocks began with `SQLite format 3` (429 blocks), `bplist00` (4,006), a PNG+signature (874) and `caff` (711); 2.6 GB looked random. The iPad 26.6.2 image+showed 3.5 GB compressible and 4.2 GB random.++Read as it stands, someone holding a `Disk.img` could carve some guest file+contents out of it without the SEP. It has not been checked which files these+are, which protection classes they carry, or whether the content-protection+layer behaves differently on virtio storage than on Apple NAND. Treat it as an+open question, not a finding about iOS.++## Open items++- **Stop the DFU VM at the iBoot banner.** Every restore's post-restore reboot+ panics because the System volume has no dyld cache before `cfw install`,+ and `vm create` and Launchpad wait for that panic; its report then lands in+ the guest. Stopping at the banner would avoid both for every machine+ ([setup boot](machine_templates.md#setup-boot), last paragraphs on crash+ reports).+- **Cold template archives.** Compressing the `Disk.img` of a template no one+ clones any more (LZFSE, about 30–35% smaller by the block survey) and+ expanding it back before the next clone would take most of the block+ store's remaining gain without touching the I/O path. Not built.+- **ASIF images.** Virtualization's XPC service knows `.asif`, but whether+ discard frees blocks in an ASIF image was not tested. Raw sparse files+ already return freed space, `VPhoneAPFSSnapshot` reads and patches the image+ at raw byte offsets, and `vm rebase` takes file offsets for disk offsets;+ ASIF would only help where the host volume cannot hold a+ sparse file.+- **Growing a disk.** No code resizes a container after the restore, so a+ larger disk means another template or `--no-template`.+- **Revisit the block store** only if disk space is still the limit with+ templates, trim and IPSW deletion in place and three or more builds kept hot+ at once, or if Apple provides a host-mountable user-space block device or an+ NBD client with write-back and FLUSH+ ([content-addressed store](content_addressed_block_store.md#when-to-revisit)).diff --git a/Research/Host/guest_discard.md b/Research/Host/guest_discard.mdnew file mode 100644index 00000000..28d44d09--- /dev/null+++ b/Research/Host/guest_discard.md@@ -0,0 +1,136 @@+# Guest TRIM and offline compaction++Whether space the guest frees inside `Disk.img` returns to the host, and+whether an offline `vm compact` would be worth having. Measured on 2026-10-06+on two `vm clone` copies of a stopped iOS 27.0 (24A435) / cloudOS 26.4 machine+with a 32 GB image: one written to, deleted from and hole-punched, the other+left alone as a control. Part of the [disk-space study](disk_space.md).++## Summary++It does, and no change is needed. With the attachment vphone uses,+`VZDiskImageStorageDeviceAttachment(url:readOnly: false)` on a+`VZVirtioBlockDeviceConfiguration` (default caching and synchronization+modes, raw sparse file, `VPhoneVirtualMachine.swift`), the guest's discards+reach the host and deallocate the file's blocks:++- 2.1 GB written into the guest and deleted came back to the host within+ 15 s, to within 9 MB of the baseline.+- Every APFS mount in the guest also trims all free space.+- After a clean shutdown, APFS-free but host-allocated space was 16.9 MB, the+ 1–2 block extents APFS declines to trim. Punching those out offline changed+ nothing observable in the guest.++So there is no guest kernel patch to write and no `vm compact` worth adding.+What occupies the host is what the guest really holds, and blocks a clone has+stopped sharing.++## The guest driver negotiates discard++From the vphone600 research kernelcache:++- `com.apple.driver.AppleVirtIO` matches standard virtio-blk+ (`IOVirtIOPrimaryMatch` `0x00021af4`) as `AppleVirtIOBlockStorageDevice`.+ Its feature mask is `0x3a66`, which includes bit 13,+ `VIRTIO_BLK_F_DISCARD`, and not bit 14, `WRITE_ZEROES`. When discard is+ negotiated it reads `max_discard_sectors` and `max_discard_seg` from the+ device config (offsets 0x24 and 0x28; `discard_sector_alignment` is not+ read), and its `doUnmap` override splits extents by those limits and sends+ DISCARD requests (type 0xb, segments `{u64 sector, u32 sectors, u32 flags}`).+ Without the feature, or with either limit 0, it returns+ `kIOReturnUnsupported` like the `IOBlockStorageDevice` base. With it, the+ device publishes `IOStorageFeatures` `{Unmap = true}`.+- The root disk actually attaches through `com.apple.iokit.AppleVirtIOStorage`+ (`…/AppleVirtIOStorageDevice/AppleVirtIODiskStorageDevice@1/IOBlockStorageDriver`+ in the boot log), which overrides `doUnmap` too. Either path supports unmap.+- APFS has both paths: a full free-space trim at mount+ (`nx_mount_initiate_free_space_trims`, `spaceman_scan_free_blocks`) and+ unmaps at run time (`spaceman_trim_free_extent_callback`).++The host side was not disassembled, so the exact call Virtualization makes is+not confirmed. That the file's `st_blocks` drops after a guest delete shows+the blocks are deallocated (a punched hole) rather than written with zeros,+which would leave `st_blocks` unchanged. Behaviour under non-default caching+or synchronization modes was not tested.++## Measured++Guest side, the APFS mount log of the first boot:++```text+spaceman_scan_free_blocks:4431: disk1 scan took 0.567917 s, trims took 0.567313 s+spaceman_scan_free_blocks:4444: disk1 2950981 blocks trimmed in 6916 extents (82 us/trim, 12190 trims/s)+spaceman_scan_free_blocks:4455: disk1 trims dropped: 3068 blocks 2500 extents, avg 1.22+```++"trimmed" rather than an error is itself evidence that discard was+negotiated; with `doUnmap` unsupported APFS would not count them.+(`device.ioreg` returns class and name only and stops at 300 entries, so it+could not show `IOStorageFeatures`.)++Host side, `stat -f %b Disk.img` in 512-byte blocks:++| Event | Blocks | GB |+| --- | --- | --- |+| Cloned, not booted | 38,834,784 | 19.884 |+| First boot (its mount trim freed 117 MB) | 38,605,416 | 19.766 |+| Baseline before writing | 39,051,656 | 19.994 |+| `files.copy` of 16 dyld subcaches (~131 MB each) into `/var/mobile` | 43,150,728 | 22.093 |+| One minute later | 43,153,744 | 22.095 |+| `files.remove {recursive}` returned | 40,861,912 | 20.921 |+| 15 s later | 39,068,848 | 20.003 |+| 2 minutes later | 39,069,520 | 20.004 |+| After `system.reboot` (mount trim freed ~106 MB more) | 38,864,064 | 19.898 |+| After `vm stop` | 39,285,120 | 20.114 |++2.099 GB written, 2.092 GB returned within 15 s; the difference is the+guest's own writes meanwhile. The reboot's mount trim reported `2922515 blocks+trimmed in 7101 extents` and `trims dropped: 3959 blocks 2646 extents`. APFS's+used space at that point was (7,812,490 − 2,922,515) × 4096 = 20.03 GB, against+19.90 GB the host had allocated: the host held less than APFS counted as used+(some allocated APFS blocks were never written), so almost nothing was left to+reclaim.++## Offline analysis++A standalone prototype (not in the repository) read a stopped image without+keys and with no mount: GPT entry → container superblock → the checkpoint+descriptor area's newest valid NXSB (Fletcher-64) → the checkpoint map entry+for the space manager → its chunk-info and chunk-info-address blocks → the+allocation bitmaps. Free runs were intersected with the file's data ranges+(`SEEK_DATA`/`SEEK_HOLE`).++```text+block size 4096, blocks 7812490, latest xid 26277, spaceman oid 1024+spaceman@14538: blocks 7812490 chunks 239 cibs 2 cabs 0 free 2895575 (11860 MB) freeQueues [19, 9, 0]+host: size 32000000000, st_blocks*512 20113981440, data extents 6485+reclaimable (APFS-free but host-allocated): 16932864 bytes in 963 ranges+```++The bitmap's free count equalled the space manager's `sm_free_count`+exactly. Punching the 963 ranges with `F_PUNCHHOLE` freed exactly+16,932,864 bytes, and a second run found nothing. The punched clone booted,+its mount trim reported the same free count, vphoned answered, and+`diagnostics.self_test` gave the same 7 passes and 3 failures as the untouched+control (accessibility frontmost check, OCR, keychain persistence: properties+of the source machine).++Rules for doing this safely, if it is ever wanted:++- Space manager, checkpoints and bitmaps are unencrypted container metadata;+ no keys are needed to know which blocks are free.+- Only blocks the bitmap marks free may go. Keybags, checkpoint areas, the+ object map, the space manager itself and blocks a snapshot references are+ all marked allocated. Blocks in the free queues are still marked allocated+ and are skipped.+- Only a cleanly stopped image. Punching blocks free in the newest checkpoint+ but used by an older one removes APFS's ability to fall back to the older+ checkpoint, which a crashed image may need.+- macOS `fsck_apfs -n` verifies only the container: it stops at the container+ keybag, which the guest's SEP wraps (the control image fails identically).+ The only check of the volumes is booting the guest.+- On a clone, punching frees only blocks the file does not share.++`vm compact` would save 10–20 MB per machine. The one case where it would+help, an image written and emptied before any trim ran, is handled just as+well by booting it once and letting the mount trim run.diff --git a/Research/README.md b/Research/README.mdindex 80173b65..85724bb4 100644--- a/Research/README.md+++ b/Research/README.md@@ -19,7 +19,8 @@ This directory keeps the evidence behind firmware patches, restore and the self-| Kernel | [CFW overview](Kernel/kernel_jb_patch_notes.md), [patcher verification](Kernel/kernel_patcher_verification.md), [FairPlay kexts](Kernel/kernel_fairplay_kexts.md), [base validation 1–5](Kernel/kernel_patch_base_first5_validation.md), [11–15](Kernel/kernel_patch_base_11_15_validation.md), [16–20](Kernel/kernel_patch_base_16_20_validation.md), [sandbox hooks](Kernel/kernel_patch_sandbox_hooks_17_26_validation.md), [individual patch notes](KernelCustomFirmwarePatches/README.md) || Other patches and captures | [Launchd jetsam](Patches/cfw_patch_launchd_jetsam.md), [user-mode hypervisor references](Patches/hv_vmm_present_usermode_xrefs.md), [reference capture](Patches/patch_reference_capture.md) || Restore | [DFU probe](Restore/virtual_dfu_probe.md), [in-process restore](Restore/native_restore_architecture.md) |-| Host and archives | [Binary split](Host/host_binary_split.md), [runtime dependency tiers](Host/runtime_dependency_tiers.md), [archive extraction contracts](Host/archive_extraction_contracts.md), [libarchive validation](Host/libarchive_xcframework_validation.md), [guest MAC, fixed address, port forwarding and names](Host/network_fixed_address.md), [machine identity and `vm clone --new-identity`](Host/machine_identity_and_clone.md), [sharing identical disk blocks with `vm rebase`](Host/disk_rebase.md), [machine templates](Host/machine_templates.md) |+| Host and archives | [Binary split](Host/host_binary_split.md), [runtime dependency tiers](Host/runtime_dependency_tiers.md), [archive extraction contracts](Host/archive_extraction_contracts.md), [libarchive validation](Host/libarchive_xcframework_validation.md), [guest MAC, fixed address, port forwarding and names](Host/network_fixed_address.md), [machine identity and `vm clone --new-identity`](Host/machine_identity_and_clone.md) |+| VM disk space | [Where a VM's disk space goes](Host/disk_space.md) (overview and measured savings), [machine templates](Host/machine_templates.md), [sharing identical disk blocks with `vm rebase`](Host/disk_rebase.md), [guest TRIM and offline compaction](Host/guest_discard.md), [why the disk layout was not made deterministic](Host/deterministic_disk_layout.md), [why there is no content-addressed block store over NBD](Host/content_addressed_block_store.md) || Guest interaction and VM identity | [DevMode XPC](Guest/devmode_xpc_protocol.md), [keyboard events](Guest/keyboard_event_pipeline.md), [machine identifier](Guest/machine_identifier_storage_analysis.md), [Auto-Lock on the Lock Screen](Guest/lock_screen_idle_timer.md), [display refresh rate](Guest/display_refresh_rate.md), [GPU acceleration](Guest/gpu_acceleration.md), [RootHide bootstrap base](Guest/roothide_bootstrap_base.md), [the Mac's .local name](Guest/mac_name_resolution.md), [locationd startup deadlock](Guest/locationd_startup_deadlock.md), [trimming system services](Guest/service_trimming.md), [deleting the orig-fs snapshot for templates](Guest/template_snapshot_deletion.md) || Historical project records | [Migration ledger](History/intg_update_status.md), [manifest refactoring summary](History/manifest_and_refactoring_summary.md) |