Skip to content

Run a virtual iPhone on an Apple Silicon Mac.

Swift

Record the VM disk-space study in Research

Jackyd2f85f49f23a
diff --git a/Research/Host/content_addressed_block_store.md b/Research/Host/content_addressed_block_store.md
new file mode 100644
index 00000000..bbd5b64f
--- /dev/null
+++ b/Research/Host/content_addressed_block_store.md
@@ -0,0 +1,254 @@
+# Content-addressed block store over NBD: why it was not done
+
+Whether serving machine disks from a deduplicating, compressing block store
+through Virtualization's NBD client would save enough over templates to be
+worth a storage subsystem. Studied on 2026-10-07 by measuring two stopped
+images, probing Virtualization's NBD client with throwaway VMs and a
+prototype server outside the repository; no vphone machine was booted on it.
+Part of the [disk-space study](disk_space.md).
+
+## Summary
+
+Not now. Templates with new-identity clones already take most of the saving
+(five machines of one build: about 102 GB → 20 GB). On top of that, a block
+store would save roughly another 30–40%, almost all of it from two sources:
+
+- compression (about 30%), which only a store can do for live disks;
+- deduplication across builds (about 17% at 4 KiB), which matters only while
+ several builds are kept.
+
+That is 7–8 GB per build with one build, 25–50 GB with three builds of 5–10
+machines each. The costs, several of them set by Virtualization and not by
+the prototype:
+
+1. The NBD client hands the guest a write-through disk and never sends FLUSH
+ or FUA, so a server must make every write durable before replying or
+ accept losing acknowledged writes on a host crash.
+2. The host cannot attach an NBD disk. `cfw install`, `cfw
+ update-environment`, `cfw update-kernel`, the offline trim, `vm rebase`,
+ snapshots, clones, export, import and the running-machine check all treat
+ `Disk.img` as a file. Every offline operation would have to materialize a
+ 20 GB raw image and write it back.
+3. If the server is gone for more than about 30 s, the client gives up for
+ good and every later guest I/O fails.
+4. A garbage collector, index and map memory, log replay at start, and one
+ more user-space hop per I/O.
+
+## Potential, measured
+
+Two `vm clone` copies of stopped machines (iPhone99,11 iOS 27.0 24A435 and an
+iPad on iOS 26.6.2 23G90, both booted before) were attached read-only, their
+System (with the `orig-fs` snapshot), Preboot, xART, Hardware and Update
+volumes mounted read-only, and every file's blocks (resource forks of
+decmpfs files included) mapped with `F_LOG2PHYS_EXT`; 343 sampled files read
+back identical through their physical addresses. Unmapped non-zero blocks are
+Data, User and container metadata. Fixed-size chunks were hashed (SHA-256,
+zero chunks skipped) and 1 in 32 first occurrences compressed with
+libcompression.
+
+Both images together, 42 GB of non-zero data:
+
+| Chunk | After dedup | Ratio | LZ4 | LZFSE | ZLIB | Index entries |
+| --- | --- | --- | --- | --- | --- | --- |
+| 4 KiB | 35.9 GB | ×1.17 | 27.4 GB | 24.9 GB | 24.1 GB | 8.76 M |
+| 16 KiB | 39.8 GB | ×1.07 | 29.3 GB | 26.5 GB | 26.1 GB | 2.43 M |
+| 64 KiB | 41.8 GB | ×1.03 | 29.8 GB | 26.8 GB | 26.6 GB | 0.64 M |
+
+| Region | iPhone 27.0 | iPad 26.6.2 | iPad found in iPhone (4K / 16K / 64K) | 4 KiB compression (LZ4 / LZFSE / ZLIB) |
+| --- | --- | --- | --- | --- |
+| System, with snapshot | 15.67 GB | 13.73 GB | 3.00 / 0.64 / 0.13 GB | 0.76 / 0.69 / 0.67 |
+| Data, User, metadata | 4.72 GB | 7.95 GB | 0.72 / 0.29 / 0.16 GB | 0.77 / 0.69 / 0.66 |
+| Preboot | 0.06 GB | 0.05 GB | 0.04 GB | 0.74 / 0.66 / 0.63 |
+
+- Cross-build dedup is about 22% of System and 9% of Data at 4 KiB, and
+ nearly vanishes at 16 KiB and above: identical files sit at random phases
+ relative to the chunk grid. Aligning the grid to the container start
+ (20,480 bytes) gave the same figures. A store would need 4 KiB chunks (a
+ large index) or content-defined chunking.
+- Identical data at the same offset between the two builds: 0.00 GB. `vm
+ rebase` cannot share anything between templates of different builds; it
+ helps only separately restored machines of one IPSW (about 8.7 GB a pair),
+ so it does not compete with a store.
+- Within one image: the iPhone's Data/User region has 1.88 GB (40% of the
+ region) duplicating other chunks in the same region, probably copies
+ between the Data and User volumes (unverified); System has 0.17 GB.
+- Compression: System 4 KiB chunks compress to about 0.69 with LZFSE (many
+ root file system files are already decmpfs-compressed; the dyld cache is
+ not). Larger chunks compress slightly better (0.66 at 64 KiB) at the cost of
+ read amplification.
+
+### Data and User blocks
+
+Data and User report FileVault locked on the host, and were expected to be
+incompressible ciphertext. They are not uniformly so. On the iPhone image
+about 2.0 GB of the region is compressible like plaintext (LZ4 0.55); chunks
+begin with `SQLite format 3` (429), `bplist00` (4,006), a PNG signature (874)
+and `caff` (711); 2.6 GB is high-entropy. On the iPad image, 3.5 GB
+compressible and 4.2 GB high-entropy; 9% of the iPad's Data chunks also occur
+in the iPhone's. This suggests guest file contents on virtio storage reach the
+disk image unencrypted while metadata and keys are protected, but it was not
+verified which files these are or why. See
+[the open question](disk_space.md#observation-to-verify-plaintext-in-data-and-user-blocks).
+
+### Projected host bytes
+
+Template 18.6 GB never booted; 0.35 GB per clone after its first boots (3 GB
+for "used for a while"); 20.5 GB per separately restored machine. Store at
+4 KiB with LZFSE: first template 18.6 × 0.894 (dedup within the image) × 0.69
+≈ 11.5 GB, each further build's template 18.6 × 0.81 × 0.69 ≈ 10.4 GB (likely
+less between adjacent builds of one device, unmeasured), each clone's
+divergence × 0.7 (compression only). IPSW caches not counted.
+
+| N per build, K builds | Separate restores | Template + clones (+ rebase) | Store | Store saves |
+| --- | --- | --- | --- | --- |
+| 5, 1 | 102 | 20.4 | 12.7 | 7.7 |
+| 10, 1 | 205 | 22.1 | 14.0 | 8.1 |
+| 5, 3 | 308 | 61.1 | 36.0 | 25 |
+| 10, 3 | 615 | 66.3 | 39.7 | 27 |
+| 5, 3, 3 GB per clone | 324 | 101 | 64 | 37 |
+| 10, 3, 3 GB per clone | 648 | 146 | 95 | 51 |
+
+Without compression, dedup alone adds only the cross-build ~17% (about 2 GB
+with one build): the store's gain is mostly compression. A store with 4 KiB
+chunks also avoids the up to fourfold amplification of a clone diverging in
+16 KiB pages ([disk rebase](disk_rebase.md#the-unit-is-the-16-kib-page-not-the-4-kib-block)),
+but a clone's first-boot divergence is only 0.35 GB.
+
+## Virtualization's NBD client
+
+From the SDK headers, strings of `com.apple.Virtualization.VirtualMachine.xpc`,
+an EFI probe VM with a read-only disk to watch the handshake, and a Linux
+guest to watch real I/O. Both probes were ad hoc signed with only the public
+`com.apple.security.virtualization` entitlement.
+
+| Question | Finding |
+| --- | --- |
+| URLs | `nbd://host[:port]/export`, `nbd://[::1]/…` and `nbd+unix:///export?socket=/path` work (validated and connected). `nbds://`, `nbds+unix://`, `nbd+tcp://` and `nbd+vsock://` are "not supported": no TLS, no authentication. Unix socket paths are limited by `sun_path` (104 bytes) |
+| Handshake | Fixed newstyle only ("Old style negotiation is not supported", "Non-fixed new style negotiation is not supported"). Client flags `FIXED_NEWSTYLE \| NO_ZEROES`; `NBD_OPT_INFO`, then `NBD_OPT_GO`, both asking for `NBD_INFO_BLOCK_SIZE`. No structured replies, extended headers, meta contexts or STARTTLS, so no `BLOCK_STATUS`. Export size must be a multiple of 512 |
+| Block size | The server's advertised minimum block size becomes the guest's logical block size (4096 made EFI read 4K; 16384 was accepted). vphone images have a GPT with 512-byte LBAs, so a server must advertise a minimum of 512 or less |
+| Commands | READ, WRITE and TRIM seen; with `SEND_TRIM`/`SEND_WRITE_ZEROES` advertised, the guest saw discard and write-zeroes limits equal to the disk size. **No FLUSH, and FUA never set**, with synchronization mode full or none. The Linux guest saw `write_cache = write through`; on a raw `VZDiskImageStorageDeviceAttachment` the same kernel saw `write back`. The strings mention an "NBD Flush request timer" that never fired |
+| Concurrency | One connection with up to 8 requests in flight (Linux, one queue, 8 parallel writers); requests up to 4 MiB, all 4K-aligned |
+| Who connects | `LOCAL_PEERPID` is the app process, while the client code (`Storage::NetworkBlockDeviceConnection`) is in the XPC service, so the descriptor is presumably passed over. An unsandboxed process connects over TCP and Unix sockets without `com.apple.security.network.client`; `vphone-vm` is not sandboxed |
+| Server down at start | `start` fails: "The storage device attachment is invalid." No retry |
+| Server down while running | 6 s: guest I/O stalls, then resumes transparently, while the delegate receives `Code=20001 Failed to negotiate with server` three times (the header calls such errors unrecoverable). 45 s: after about 33 s the client gives up; every later guest I/O is an `I/O error`, and it does not reconnect when the server returns |
+| Server returns EIO | Passed straight to the guest as an I/O error, no retry |
+| Boot disk of a PV=3 research VM | Not tested. The storage path is the same `VZVirtioBlockDeviceConfiguration` and iBoot reads Preboot through virtio-blk, so it should work with a 512-byte minimum block |
+
+Guest throughput (Linux, 256 MiB disk; the NBD server kept data in the page
+cache, so these rows pay no durability cost and are not comparable to raw):
+
+| | 4K O_DIRECT write | 4K O_DIRECT read | 1 MiB write | 1 MiB read |
+| --- | --- | --- | --- | --- |
+| Raw `VZDiskImage`, sync full | 74 MB/s | 61 MB/s | 1.0 GB/s | 3.2 GB/s |
+| NBD over a Unix socket | 117 MB/s | 130 MB/s | 5.0 GB/s | 6.9 GB/s |
+| NBD over TCP | 33 MB/s | 72 MB/s | 2.8 GB/s | 3.0 GB/s |
+
+## Prototype and benchmarks
+
+The prototype, in C with system libraries only and outside the repository:
+an NBD server (fixed newstyle; GO, INFO, EXPORT_NAME, LIST, ABORT; READ,
+WRITE, FLUSH, TRIM, WRITE_ZEROES, DISC, FUA honoured; one reader thread per
+connection, requests on a concurrent queue, FLUSH and TRIM as barriers) over
+either a raw file (TRIM as `F_PUNCHHOLE`) or a log-structured store: one
+append-only log of data records (SHA-256 truncated to 128 bits, LZ4 or LZFSE,
+stored raw unless it saves 10%) and map records (one per request, listing
+chunk → entry), with the hash index and map rebuilt by replaying the log.
+Replay stops at the first record that fails its checksum and truncates there;
+a map record may only reference verified data records, so recovery yields a
+prefix of acknowledged writes. A read-only base file passes through, so a
+clone is a map rather than a full import.
+
+Import of 2 GiB sampled across an iPhone image (32 × 64 MiB): 4K + LZ4
+1.55 GB at 540–800 MB/s, 4K + LZFSE 1.44 GB, 4K uncompressed 1.93 GB, 16K or
+64K + LZ4 1.56 GB at 1.6–2.2 GB/s. Two writers racing to store the same new
+content left about 0.07% duplicate records.
+
+Loopback, data in the page cache, 2 GiB volume:
+
+| | 4K rand read QD1 | 4K rand read QD32 | 16K rand read QD32 | 4K rand write QD1 | 4K rand write QD32 | 1 MiB seq write | Flush after random writes |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| Direct `pread`/`pwrite` | 0.8 µs | 0.9 M IOPS | 14.7 GB/s | 6.8 µs | 199 k | 3.2 GB/s | 0.33–0.6 s |
+| NBD Unix, raw file | 10.9 µs | 229 k | 3.2 GB/s | 14.6 µs | 147 k | 0.83 GB/s | 0.8–1.3 s |
+| NBD TCP, raw file | 25.6 µs | 108 k | 1.4 GB/s | 37.5 µs | 83 k | 0.75 GB/s | 0.6–0.75 s |
+| NBD Unix, store 4K | 10.0 µs | 223 k | 2.9 GB/s | 21 µs | 133 k | 0.79 GB/s | 8–14 ms |
+| NBD Unix, store 16K | 12.2 µs | 210 k | 3.3 GB/s | 39 µs (read-modify-write) | 104 k | 0.79 GB/s | 6–10 ms |
+
+Durable before each reply:
+
+| | 4K write QD1 | 4K write QD32 | 1 MiB seq write |
+| --- | --- | --- | --- |
+| Store, `F_BARRIERFSYNC` | 977 IOPS (1.0 ms) | 8.5 k | 169 MB/s |
+| Store, `F_FULLFSYNC` | 271 IOPS (3.7 ms) | 484 | 226 MB/s |
+| Raw file, `F_BARRIERFSYNC` | 883 | 7.9 k | 725 MB/s |
+| Raw file, `F_FULLFSYNC` | 255 | 621 | 207 MB/s |
+
+- The protocol and thread hops add 10–25 µs per operation. Reads from the
+ store are as fast as raw (LZ4 decode is cheap); writes pay hashing and
+ compression, 10–40% slower, twice that for 4K writes into 16K chunks.
+- The log turns random writes into appends, so a flush takes milliseconds
+ where a raw file takes about a second. That advantage disappears once every
+ write must be durable, which a write-through client demands: a raw image's
+ guest pays one `F_FULLFSYNC` per APFS checkpoint, a store about 1000 IOPS at
+ QD1. Group commit (one barrier per ~1 ms batch) helps only at QD > 1.
+- The log grows: about a million 4K random overwrites on a 2 GiB volume made
+ an 8.7 GB log. Garbage collection is required, not optional.
+- The first version wrote map records only on FLUSH. A guest wrote 1 MiB and
+ synced, the server was killed and restarted, the client reconnected, and the
+ guest read back zeros: acknowledged writes silently lost, because the client
+ never flushes. Writing a map record before each reply fixed it. Host power
+ loss was not tested.
+
+A real vphone boot from NBD was not attempted: it needs a `vphone-vm` build
+with a changed attachment and therefore a new cdhash admitted by the AMFI
+allowlist. The Linux guest covered reads, writes, TRIM, concurrency,
+reconnection, the timeout and EIO on the real client.
+
+## If it were built
+
+- **Server:** not inside `vphone-vm` (a VM crash would take its storage with
+ it, and one store per process cannot deduplicate across machines). One
+ per-user helper in the bundle, started on demand, one Unix socket per
+ machine with a path under 104 bytes. `vm launch` must see the socket
+ listening first. The ~30 s client timeout bounds helper restarts, upgrades
+ and GC pauses.
+- **Consistency:** log-structured with replay, as the prototype; map
+ checkpoints to bound replay; group commit; a stated contract ("after a host
+ crash the disk returns to a consistent state a few ms old", or per-write
+ durability at about 1/100 of the speed). APFS tolerates a consistent prefix,
+ as it does a raw image with synchronization off.
+- **GC:** mark from all maps and snapshot maps, copy live records into new
+ segments, segment by segment. TRIM only clears map entries; space returns
+ at GC, unlike a raw file where TRIM punches at once.
+- **Snapshots and clones:** a copy of the map (125 MB dense for 64 GB at 4K,
+ much less sparse). Attractive, but `clonefile` already does it for free.
+- **Host maintenance:** the largest cost. Every offline verb becomes
+ materialize, modify, write back, needing about 20 GB of scratch each time on
+ hosts that are often short of space.
+- **Failure modes:** a helper down for more than ~30 s fails every machine's
+ disk; log corruption truncates to the last verified prefix (data goes back
+ in time); a full host disk returns ENOSPC to the guest; one storage bug hits
+ every machine at once.
+- **Upkeep:** a versioned on-disk format with migrations, helper lifecycle and
+ permissions, GC scheduling, Launchpad's size accounting, crash-injection
+ tests, and a new minor series because Launchpad would depend on it.
+
+## Cheaper alternatives
+
+1. Templates, the offline trim, and deleting IPSWs (13.5 GB per build) once a
+ template exists: all in place.
+2. Compress cold templates: archive the `Disk.img` of a template no one clones
+ any more (LZFSE, about 30–35% smaller by the figures above) and expand it
+ back before the next clone. That takes the store's largest gain, compression,
+ without touching the I/O path, host mounts or the CFW tools. Only "three or
+ more builds all in use" misses out. Not built.
+3. Cross-build dedup (~17%) needs a store or host block-range cloning, which
+ macOS does not offer publicly. Left alone.
+
+## When to revisit
+
+- Disk space is still the limit with templates, trim and IPSW deletion in
+ place, and three or more builds of ten or more machines each are kept in
+ use at once; or
+- Apple provides a host-mountable user-space block device (a DiskImages2
+ plug-in, or NBD the host can attach); or
+- Virtualization's NBD client gains write-back with FLUSH.
diff --git a/Research/Host/deterministic_disk_layout.md b/Research/Host/deterministic_disk_layout.md
new file mode 100644
index 00000000..e0624db7
--- /dev/null
+++ b/Research/Host/deterministic_disk_layout.md
@@ -0,0 +1,178 @@
+# Deterministic disk layout: why it was not done
+
+Two machines restored separately from one IPSW hold mostly the same bytes,
+but only 44% of them at the same offset in `Disk.img`, and `vm rebase` can
+share only same-offset blocks ([disk rebase](disk_rebase.md)). This note
+records what the shifted bytes are, why they shift, what it would take to
+pin them, and why that was not worth it once templates existed. Studied on
+2026-10-07 by measurement, reading and offline allocation experiments, with
+no restore run. Part of the [disk-space study](disk_space.md).
+
+## Summary
+
+- The shifted region is mainly the OS and App cryptexes that `cfw install`
+ copies onto the System volume on the host: 7.35 GB (dyld cache 7.18 GB) of
+ the 9.9 GB that differs only in position. The other ~2.5 GB is Data, User
+ and some Preboot content the guest writes during the restore's `fixup_var`
+ and the first boot.
+- Restore writes the same large data in both machines. What differs are
+ per-device small writes (the APTicket is 5805 bytes on one and 5806 on the
+ other, ECID-derived files, keybags, log timestamps) and the guest's own
+ concurrency during restore.
+- The host's APFS allocator is deterministic but chaotic: a few blocks of
+ difference before a large copy move all of it by hundreds of MB.
+- The only way to pin the cryptexes is to merge them into the root file
+ system before the restore, so ASR writes them block for block. That would
+ raise same-offset sharing to about 81% (15.8 of 19.4 GB), not more.
+- Not done: with templates, separately restored machines are the exception,
+ and the cost is a rebuilt, resealed root file system per IPSW.
+
+## What shifts
+
+Method: two booted machines, iPhone99,11 iOS 27.0 (24A435) + cloudOS 26.4,
+standard preset, each attached read-only with System and Preboot mounted
+read-only (Data and User are locked). Each file's physical extents
+(`F_LOG2PHYS_EXT`, plus the partition offset of 20,480 bytes) gave its offset
+in the image; a sample was checked byte for byte. The 217,000
+decmpfs-compressed root file system files cannot be mapped this way; they sit
+in the ASR region, at the same offset in both.
+
+Layout of one image:
+
+| Image offset | Content |
+| --- | --- |
+| 0 – 1.07 GB | APFS metadata, Preboot, guest-written Data |
+| 1.07 – 9.93 GB | The root file system ASR wrote and `apfs_invert` turned into the System volume: the same offset in both machines |
+| Small holes inside that | Small cryptex files (even-numbered dyld subcaches of 0.2–0.5 MB, the atlas) filling free extents inside the ASR image |
+| 9.93 – 10.63 GB | About 600 MB of unmapped plaintext, guest-written Data/User |
+| 10.63 – 16.50 GB | dyld subcaches `.21`–`.77`, `.dyldlinkedit` and `.symbols` (1.22 GB): 30 files, 5.85 GB, one extent each |
+| 16.50 – 19.9 GB | About 3.4 GB unmapped: first-boot Data/User |
+| 62.0 – 64.0 GB | Odd subcaches `.01`–`.19` (about 1.3 GB), the DriverKit cache, a QuartzCore metallib and a few CFW-modified System files. In the guest this is the APFS metazone (`metazone … encrypted: 15136710-15380850` in the restore log); the host's mount does not reserve it, so the first large files copied land there |
+
+The two machines' 81 dyld cache files had identical SHA-1s (CFW from two
+different bundle builds patched them identically), yet moved:
+
+| Group | Size | Shift (B − A) |
+| --- | --- | --- |
+| Subcaches `.21`–`.77`, `.symbols` | 5.85 GB | −6415 blocks (−26.3 MB) |
+| Subcaches `.07`–`.19` (metazone tail) | 0.92 GB | −36 blocks |
+| `.03`, `.05` | 0.26 GB | −1 block |
+| `.01` | 0.13 GB | −219 blocks |
+| 224 small cryptex files | 8 MB | −1358 blocks |
+| Uncompressed root file system files | 4.55 GB | 4.49 GB at the same offset |
+
+A content-hash comparison at any offset gave: of B's 19.4 GB non-zero data,
+8.45 GB at the same offset as A (44%), 9.89 GB present in A elsewhere, 1.04 GB
+unique to B. By region of B:
+
+| Region of B | Same offset | Shifted | Unique |
+| --- | --- | --- | --- |
+| 0 – 1.07 GB | 0.00 | 0.38 | 0.24 |
+| 1.07 – 9.93 GB (ASR) | 8.43 | 0.26 | 0.01 |
+| 9.93 – 16.37 GB (cryptex bulk) | 0.02 | 6.12 | 0.09 |
+| 16.37 – 19 GB (Data/User) | 0.00 | 1.50 | 0.66 |
+| 62 – 64 GB (cryptex tail) | 0.00 | 1.63 | 0.04 |
+
+The 1.5 GB of shifted Data/User content in B lies in A at 0.6–1.08 GB,
+3.4 GB, 10.3–10.6 GB and elsewhere, displaced by tens of MB to 16 GB: the
+guest's allocator put identical content into different holes.
+
+## Why it shifts
+
+The order of writes: restore partitions and creates the container,
+preallocates a 10.1 GB `/mnt1/apfs_invert_asr_img`, streams ASR into it and
+inverts it into the System volume; then installs the kernelcache, device tree,
+root hash and ticket into Preboot, runs `fixup_var` on Data, the recovery OS
+restore, `seal_system_volume`, `create_system_snapshot` and
+`split_data_volume`. Afterwards `cfw install` on the host copies the cryptexes
+(`installCryptexes`: `VPhoneConfinedDirectory.copyTree`, names sorted,
+depth-first, `fcopyfile(COPYFILE_DATA)` per file). The dyld cache patches
+after that write in place and allocate nothing. So where the cryptexes land
+depends only on the container's allocation state when the restore ends and on
+the host allocator.
+
+The two restores' space-manager statistics, from their logs:
+
+| Point | A | B |
+| --- | --- | --- |
+| Container created | 15,595,537 free / 10 extents | same |
+| After invert | 13,078,288 free / 13 extents | same |
+| Around seal | 13,265,750 / 4625 extents → 13,157,004 | 13,265,759 / 4624 → 13,157,013 |
+| `split_data_volume` | 13,156,553 / 4538 extents | 13,156,562 / 4530 extents |
+| Restore end | 13,251,848 | 13,251,857 |
+
+Nine blocks apart. Step durations differed widely (`fixup_var` 6 s against
+22 s, `seal_system_volume` 25 s against 94 s), which changes which small file
+lands in which hole; Preboot showed the same firmware file at different
+places in the two.
+
+### The host allocator, offline
+
+On a scratch 20 GB raw APFS image (`diskutil image create blank --format RAW
+--fs APFS`), each trial cloned an identical copy, mounted it and copied A's
+`System/Cryptexes/{OS,App}` (7.35 GB, 3345 extents) with a C reimplementation
+of `copyTree`, then mapped the result:
+
+| Trial | Same offset as the first run | Note |
+| --- | --- | --- |
+| The same copy twice more (7 s and 17 s) | 100% | Same state and order: fully deterministic, timing does not matter |
+| `F_PREALLOCATE` every file first | 100% | Preallocation does not change placement |
+| An 80 MB file written first | 1.4% | Most moved +1 to +5 MB |
+| Three small files (1 + 2 + 3 blocks, 24 KB) first | 0.6% | 7.2 GB moved +268.4 MB (65,536 blocks: the start of the 3-block allocation zone) |
+| Eight small files (144 KB) first | 0% | Moved +402.7 MB (98,304 blocks) |
+| Container padded to a fixed 1000 MB used first | 99.4% | Works only if the earlier difference was one contiguous file |
+| Padding plus three small files | 0% | Moved +309 MB |
+
+Placement follows the free-extent table and the per-size allocation-zone
+cursors. A few blocks of difference shift a large file by hundreds of MB.
+The real machines differ by one to a few thousand blocks, so their shifts are
+small, but for same-offset sharing one block is as bad as a gigabyte.
+
+## Ways to pin it
+
+| Approach | Result |
+| --- | --- |
+| Fixed copy order on the host | Already fixed (sorted). The start state is the problem |
+| `F_PREALLOCATE` | Changes nothing (trial above) |
+| Copy before the variable writes | Those happen inside the guest's restore, before `cfw install` can run |
+| Pad the container to a fixed usage, or fill below a boundary | Defeated by a few small files, which is exactly the real difference |
+| Write blocks to fixed offsets on the host and fix up APFS metadata | Means writing an APFS writer (fs tree, extent references, space manager, checkpoints) |
+| Clone ranges between files on the host | macOS has no public range-clone call; `clonefile` is whole-file |
+| **Merge the cryptexes into the root file system before the restore** | Works in principle: the ASR region already matches block for block, and both containers had identical free-extent statistics after the invert. The `.less` variant's `CryptexFilesystemPatcher` does this merge (mount OS, SystemOS and AppOS, copy, rebuild trust cache, mtree, digest and root hash, `Ap,SystemVolumeCanonicalMetadata`), but it runs only for `.less` (`FirmwarePipelineComponents.swift`). After the merge `installCryptexes` finds both directories populated and only creates the links, and the in-place dyld cache patches keep identical bytes identical |
+| The cryptexes as a second virtual disk per IPSW, cloned per machine | Would share 100% regardless of layout, but dyld needs the cache before launchd, so the guest would have to mount it at early boot: a redesign of how CFW wires in the cryptexes. Untested, highest risk |
+
+The ASR merge was not prototyped. It needs, per IPSW, decrypting and
+rebuilding a 16–17 GB root file system and a real restore to check boot, and
+these questions answered first:
+
+- whether TSS signs a changed `OS`, `SystemVolume` and
+ `Ap,SystemVolumeCanonicalMetadata` on the JB path (the JB kernel already
+ bypasses the root snapshot seal check, `authapfs_seal_is_broken` and the
+ graft's root hash check, and iBoot's image4 callback is bypassed);
+- the preallocated invert image growing from 10.1 GB to about 17.5 GB, and ASR
+ transfer from 62–95 s to roughly 2–3 minutes;
+- how much of the merged cryptex content ends up compressed (System volume
+ size only; it does not affect determinism).
+
+## Decision
+
+- Reachable same-offset sharing: about 81% (8.45 + 7.35 ≈ 15.8 of 19.4 GB),
+ not the 96% that is identical at some offset. The ~2.5 GB the guest writes
+ is placed by the guest's allocator and spreads further with every boot.
+- Cost: a decrypted, merged and resealed root file system per IPSW (minutes
+ of work and a 16–17 GB artifact to cache or rebuild), about 80% more restore
+ transfer, the `.less` merge path revived and maintained for JB, and
+ unverified TSS and boot-chain risk.
+- Gain: about 7.35 GB more shared per separately restored machine, and only
+ after `vm rebase`. Template clones already share nearly everything; separate
+ restores remain for machines that need their own SEP secret
+ (`--no-template`).
+
+Not done. If separate restores become the norm, start from the ASR merge and
+verify TSS and boot on one test machine first. Until then `vm rebase` between
+separate restores recovers what happens to line up: 8.7 GB (about 43%) for
+the pair measured here. One later rebase of a separately restored machine
+onto a template of the same build shared 13.7 GB of about 18.2 GB
+([disk rebase](disk_rebase.md#measurements)), which suggests the cryptex copy
+landed at the same offsets there, as it would if the two restores happened to
+end in the same allocation state; that case was not analysed.
diff --git a/Research/Host/disk_rebase.md b/Research/Host/disk_rebase.md
index 5122b965..62cc0078 100644
--- a/Research/Host/disk_rebase.md
+++ b/Research/Host/disk_rebase.md
@@ -16,6 +16,11 @@ restored from iOS 27.0 (24A435) on cloudOS 26.4 (23E5207q): of about 20 GB
of data in each 64 GB sparse `Disk.img`, 8.6 GB is byte-identical at the same
offset, mostly the ASR-restored system volume between roughly 1 GB and 9.7 GB
into the image. Physical sharing between them, by `F_LOG2PHYS_EXT`, was 0.
+Most of the rest is identical too, but at other offsets: chiefly the cryptexes
+`cfw install` copies on the host, which a few blocks of difference in the
+restore's end state move as a whole (see
+[deterministic disk layout](deterministic_disk_layout.md)). A rebase cannot
+share those.
## Method
@@ -213,3 +218,25 @@ note: the rebase would free almost nothing: rb-c already shares its identical bl
2,097,212 KiB (2.15 GB); a dry run of it afterwards reported 0.00 GB newly
and 2.15 GB already shared. Mapping both images took well under a second.
+
+A separately restored machine onto a template, 2026-10-09: a `--no-template`
+machine (no trim, no setup boot) and a trimmed template, both built from iOS
+27.0 (24A435) / cloudOS 26.4 by the same bundle build, the machine stopped.
+`vm rebase <machine> --onto <template id> --dry-run` took 4 s and reported
+13.70 GB newly shared, 0.03 GB already shared, 4.46 GB written and 0.11 GB
+punched; the rebase itself took 10 s and printed the same figures. `df` rose
+by 13,445,384 KiB (13.77 GB); the newest Time Machine local snapshot predated
+the machine, so none held the old blocks. The template's `Template.plist`,
+`Disk.img` size and modification times and its `vm template show` output were
+unchanged, and it listed only its own clones as users. The machine got no
+`TemplateSource.plist`, kept its `SEPStorage`, `nvram.bin` and `config.plist`
+untouched, booted with its own UDID with vphoned answering in 5 s and no
+panic. A dry run afterwards reported 0.00 GB newly and 13.72 GB already
+shared, with the "free almost nothing" note; so did one onto a clone of the
+template.
+
+13.70 GB is well above the 8.7 GB two separately restored machines shared:
+in this pair the host's cryptex copy probably landed at the same offsets
+too, which the layout study found depends on the two restores ending in the
+same allocation state ([deterministic disk layout](deterministic_disk_layout.md)).
+Why it did here was not analysed, so expect anything between the two figures.
diff --git a/Research/Host/disk_space.md b/Research/Host/disk_space.md
new file mode 100644
index 00000000..77c2a3a5
--- /dev/null
+++ b/Research/Host/disk_space.md
@@ -0,0 +1,293 @@
+# Where a VM's disk space goes
+
+The disk-space study of 2026-10-06 to 2026-10-09: what a machine's ~20 GB is
+made of, what each way of saving it was measured to save, and why one
+template per firmware with new-identity clones became the default. The
+mechanisms have notes of their own, linked below; this note keeps the
+composition, the comparison and the end-to-end numbers.
+
+Unless stated otherwise the guests are iPhone17,3 (or iPhone99,11) on iOS 27.0
+(24A435) with cloudOS 26.4 (23E5207q), standard preset, a 64 GB sparse
+`Disk.img`, on an APFS host volume. Block sharing was measured with
+`fcntl(F_LOG2PHYS_EXT)` on both files ("physically shared"), not with `du`.
+
+## Summary
+
+- A restored machine that has booted once allocates about 20.5 GB. About
+ 8.6 GB of it is the IPSW's root file system, 7.2 GB the OS and App
+ cryptexes `cfw install` copies onto the System volume, and 2–5 GB the Data
+ and User volumes' first-boot state. Building one also needs the ~11 GB
+ restore tree and 13.5 GB of IPSWs.
+- Space the guest frees comes back to the host by itself, within about 15 s
+ (guest TRIM). There is nothing worth compacting offline
+ ([guest TRIM](guest_discard.md)).
+- The saving that matters is sharing. A template per firmware, and every
+ machine a `clonefile` copy of it with a new ECID, costs each machine about
+ 0.35 GB after its first boots instead of 20.5 GB
+ ([templates](machine_templates.md),
+ [identity and clone](machine_identity_and_clone.md)).
+- Everything else is small next to it and counts once per template, not per
+ machine: the offline file trim 1.2 GB, the orig-fs snapshot itself about
+ 21 MB, deleting the IPSWs 13.5 GB per build. `vm rebase` gets back about
+ 8.7 GB per pair of machines restored separately ([disk rebase](disk_rebase.md)).
+- Two further approaches were studied and not built: making the restore's
+ disk layout deterministic ([deterministic layout](deterministic_disk_layout.md))
+ and a content-addressed block store behind Virtualization's NBD client
+ ([block store over NBD](content_addressed_block_store.md)).
+
+## What a machine holds
+
+### The machine folder
+
+| Item | Size | Note |
+| --- | --- | --- |
+| `Disk.img` | 64 GB logical | Sparse. About 18.6 GB allocated after `cfw install` and before any boot; about 20.5 GB after the first boot (2026-10-07, two machines) |
+| `nvram.bin` | 33.6 MB | Boot firmware and NVRAM; cloned, diverges by a few blocks |
+| `SEPStorage` | 512 KB | |
+| `.vphoned.signed` | 9.3 MB | The staged vphoned |
+| `iPhone*_Restore/` | about 11 GB | Only while a machine is built; removed afterwards unless `--keep-artifacts` with `--no-template` |
+| IPSW cache (outside the folder) | 12.3 GB iPhone + 1.2 GB cloudOS | Per build, shared by every machine of it |
+
+### Inside `Disk.img`
+
+Attached read-only (`diskutil image attach --readOnly --noMount`), a booted
+machine's container showed:
+
+| Volume | In use | Content |
+| --- | --- | --- |
+| System | 16.2 GB (Sealed: Broken) | Below |
+| Data | 1.2–2.5 GB | FileVault, locked on the host |
+| User | 0.5–2.7 GB | FileVault, locked on the host |
+| Preboot | 61 MB | kernelcache 43 MB, SEP and other firmware 17 MB; `Preboot/Cryptexes/{OS,App}` empty |
+| xART, Hardware | 10.5 MB, 6.3 MB | Gigalocker; activation UUID and ticket |
+| Update | | Not mounted |
+
+The container's "In Use" (21.0 GB) matched the image's allocated size: the
+blocks `cfw install` freed on the host had been punched out of the file, so a
+fresh machine carries no dead space.
+
+The System volume is two layers:
+
+- The sealed root file system restore wrote (ASR, then `apfs_invert`). CFW
+ renames its snapshot to `orig-fs.disabled.rn-<hash>`, and the patched kernel
+ boots the live volume. The snapshot still references the original 8.59 GB;
+ of the CFW-modified files, only 35 MB of old versions are held by nothing
+ else (21.5 MB measured when the snapshot was deleted, see
+ [snapshot deletion](../Guest/template_snapshot_deletion.md)).
+- The OS and App cryptexes (7.2 GB on the live volume), which `cfw install`
+ copies from the restore tree into `System/Cryptexes/{OS,App}` and links in
+ (`System/Library/Caches/com.apple.dyld` points into
+ `System/Cryptexes/OS/…`). vphone boots without the cryptex graft
+ (`cryptex1 sniff: ignition failed: 8` on every boot), so the System volume
+ copy is the only one dyld finds. The snapshot does not hold these blocks.
+
+The live volume by kind, deduplicated by inode, `st_blocks` (2026-10-07):
+
+| Kind | Size | Layer |
+| --- | --- | --- |
+| dyld shared cache: 71 subcaches, data, readonly, linkedit, atlas | 5,955 MB | cryptex |
+| dyld shared cache `.symbols` | 1,223 MB | cryptex |
+| PrivateFrameworks (without localizations and models) | 2,989 MB | root fs |
+| ML models (`.mlmodelc`, `.espresso`, …) | 1,021 MB | root fs |
+| Other CoreServices, ExtensionKit, PreferenceBundles and the like | 902 MB | root fs |
+| `LinguisticData` | 558 MB (418 MB outside en and zh-Hans) | root fs |
+| `.loctable` (all languages in one file) | 434 MB | root fs |
+| `PreinstalledAssetsV2/RequiredByOs` | 424 MB (SharingDeviceAssets 232 MB) | root fs |
+| Frameworks, the rest | 400 MB | root fs |
+| `usr/standalone` | 375 MB (SU ramdisk 243 MB, baseband firmware 114 MB) | root fs |
+| Fonts | 373 MB (AppleColorEmoji 139 MB) | root fs |
+| `usr`, the rest | 361 MB | root fs |
+| Health assets and NanoTimeKit | 350 MB (watch faces 144 MB) | root fs |
+| `/Applications` | 257 MB, 219 apps | root fs |
+| `.lproj` of languages other than en, Base, zh-Hans | 220 MB | root fs |
+| TTS and voice, audio tunings, ringtones, app placeholders, wallpapers | 530 MB | root fs |
+
+The dyld cache is 45% of the System volume and can only be kept or replaced
+whole. Most localizations live in `.loctable` files, all languages in one
+file, so trimming languages from `.lproj` folders would save about 0.3 GB at
+most.
+
+### Data and User
+
+The host cannot mount these volumes (their keys are wrapped by the guest's
+SEP), so their content was only classified by block. After the first boot
+they hold 2–5 GB: the restore's `fixup_var`, the first boot's data migration,
+caches, logs and the expanded removable system apps. This is the part each
+clone writes for itself; see the per-clone numbers below.
+
+## What each approach saves
+
+| Approach | Saving | Counts | Status |
+| --- | --- | --- | --- |
+| Guest TRIM | Deleted guest data returns within ~15 s; ~17 MB per machine left unreclaimed | Per machine | Works as shipped ([guest TRIM](guest_discard.md)) |
+| Template + new-identity clone | 20.5 GB → about 0.35 GB per machine after its first boots | Per machine | Default for `vm create` and Launchpad 2.9 |
+| Setup boot in the template | A clone's first 100 s write 1,040 → 372 MiB; private bytes after 7 min 3.2 → 0.6 GB | Per machine | Part of every template build |
+| Offline trim, standard tier | 1.17 GB (conservative 0.36 GB) | Per template | Default tier ([templates](machine_templates.md#offline-trim)) |
+| orig-fs snapshot deletion | About 21 MB itself; makes the trim free anything | Per template | Setup boot step 0 |
+| Deleting the IPSWs | 13.5 GB | Per build | Offered by Launchpad once a template exists |
+| `vm rebase` of separate restores | About 8.7 GB per pair (≈ 43%); 13.7 GB measured onto a template | Per machine | `vm rebase` |
+| Service trimming | Idle writes −60% (27.0), −94% (26.6.2); first-boot writes about −35% | Per machine, writes | `trimmed` profile ([service trimming](../Guest/service_trimming.md)) |
+| Deterministic disk layout | Same-offset sharing of separate restores from 44% to at most ~81% | Per machine | Not done |
+| Content-addressed store over NBD | A further 30–40% over templates, mostly from compression | Whole library | Not done |
+
+Projected host bytes for N machines per build and K builds (IPSW caches not
+counted; template 18.6 GB, 0.35 GB per clone after first boots, 20.5 GB per
+separately restored machine; the CAS column is 4 KiB blocks with LZFSE):
+
+| N × K | Separate restores | Template + clones | CAS |
+| --- | --- | --- | --- |
+| 5 × 1 | 102 GB | 20.4 GB | 12.7 GB |
+| 10 × 1 | 205 GB | 22.1 GB | 14.0 GB |
+| 5 × 3 | 308 GB | 61.1 GB | 36.0 GB |
+| 10 × 3 | 615 GB | 66.3 GB | 39.7 GB |
+| 10 × 3, 3 GB written per clone | 648 GB | 146 GB | 95 GB |
+
+`vm rebase` does not change the template column: templates of different
+builds share no block at the same offset (0.00 GB between an iPhone 27.0 and
+an iPad 26.6.2 image), and clones already share their template's.
+
+## Why templates won
+
+A template is the only approach whose saving grows with the number of
+machines and costs nothing on the I/O path: a clone is a `clonefile`, the disk
+stays a raw file the host can attach for `cfw install`, `cfw
+update-environment`, the trim, `vm rebase` and snapshots, and Virtualization
+keeps its write-back cache. Its costs are known and documented: every clone
+shares the template's SEP root secret, gigalocker and volume keys
+([shared secrets](machine_templates.md#shared-secrets); `--no-template` makes
+a machine with its own), a template must never boot once frozen, and the disk
+size is part of the key because no code grows a container after the restore
+(a larger size needs another template).
+
+Three places to trim the System volume were considered:
+
+- Before the restore, in the IPSW's root file system image. The repository
+ has the machinery (the `.less` variant's `CryptexFilesystemPatcher`), but
+ it means decrypting and resealing an 8–9 GB image per IPSW, and whether TSS
+ signs a changed `SystemVolume`/`Ap,SystemVolumeCanonicalMetadata` digest on
+ the JB path was never tested. CFW then rewrites the result anyway.
+- After `cfw install`, on the host, with the System volume mounted: chosen.
+ No guest involved, repeatable, and a raw image is hole-punched when the
+ volume is unmounted (a 1 GB file written and deleted on a test image left
+ 5.6 MB allocated after the unmount, 982 MB before it).
+- Inside the running guest through vphoned: rejected when templates were
+ still meant never to boot. The setup boot came later, for Setup and app
+ removal, but the host-side trim stayed the simpler choice: it needs no
+ writable System volume in the guest and no destructive RPC.
+
+The catch was the snapshot: deleting root file system files on the host
+frees nothing while `orig-fs.disabled.rn-*` exists, and the host cannot
+delete that snapshot (SIP, `-69863` even as root). The guest can, so the
+setup boot deletes it first ([snapshot deletion](../Guest/template_snapshot_deletion.md)).
+
+Never trimmed: the dyld cache and its `.symbols` (the CFW cache patcher's
+`DyldSharedCacheSymbolResolver` reads the local symbols, so a later
+`cfw update-environment` on a clone would fail to find its patch sites), ML
+models (Photos analysis, Spotlight, Live Text and the camera pipeline load
+them; a missing model makes daemons crash or retry), and `/Applications`
+(system UI and view services). The `aggressive` tier (other `.lproj`, Health
+assets, some fonts; about 1.7 GB with the standard items) stays reserved because it was never
+booted.
+
+## Measured end to end
+
+Template builds from scratch, Launchpad 2.9 driving bundle 2.9 builds, IPSWs
+cached, seconds per step (2026-10-09). The setup boot is mostly the guest's
+first-boot data migration ([Setup Assistant skip](../Guest/setup_assistant_skip.md#the-first-boot-after-a-restore)).
+
+| iOS | prepare | patch | restore | stop DFU | `cfw install` | trim | setup boot | clone + first boot | total |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| 27.0 | 6 | 22 | 83 | 8 | 67 | 2 | 79 | 5 | 4 min 39 s |
+| 27.0 | 5 | 23 | 86 | 9 | 62 | 2 | 105 | 5 | 5 min 04 s |
+| 27.0 | 5 | 23 | 114 | 8 | 58 | 2 | 89 | 7 | 5 min 14 s |
+| 26.6.2 | 4 | 19 | 80 | 8 | 199 (about 160 s of it an administrator prompt) | 2 | 95 | 5 | 6 min 59 s |
+
+The third 27.0 build's setup boot: connect 4 s, setup-skip 0 s, settle 63 s,
+remove apps 5 s, service profile 0 s, reboot 10 s, verify 1 s, crash reports
+1 s, stop 5 s. The first two ran the earlier `setup.skip`, which retried for
+45–98 s while settle then took about 10 s
+([setup boot](machine_templates.md#setup-boot)); the setup boot took about as
+long either way, because the wait is the migration.
+
+Machines from an existing template:
+
+| Path | Time |
+| --- | --- |
+| `vphone-cli vm create --template <id> --skip-first-boot` | under 1 s (the clone itself 0.06 s) |
+| Launchpad New Machine (find, clone, first boot until vphoned answers) | 6–9 s |
+| Launchpad, IPSWs deleted (template found by its recorded sources) | 6 s |
+| Before templates, a full restore per machine | 3.5–5 min iPhone, about 7.5 min iPad |
+
+Sizes:
+
+| Object | Allocated | Note |
+| --- | --- | --- |
+| Template, standard trim, setup boot | 17.4–17.7 GB | `Disk.img` 17.55 GB plus 43 MB of other files |
+| Template, setup boot, no trim | 19.3 GB | |
+| Template adopted after a plain first boot, no trim | 18.9 GB | Clones then reach 3.2 GB private after 7 min |
+| Clone, 35 s after its first start | 0.34–0.37 GB private | |
+| Clone, after 7 minutes | 0.59–0.61 GB private | The same with and without the trim: the trim saves per template |
+| Clone of a stopped machine, two boots | 0.33–0.52 GB private | `vm clone --new-identity` |
+
+The template's own private bytes stay at about 0.1 GB, the blocks every clone
+has rewritten. Booting a template after clones exist raised one clone's
+private bytes from 0.42 to 0.80 GB, which is why a frozen template never boots
+([templates](machine_templates.md#never-booting-once-frozen)).
+
+## Measuring it
+
+- `du`, Finder and a folder's allocated size count every clone at its full
+ ~18 GB. `df` before and after an operation is the honest total.
+- Per object, compare physical extents (`F_LOG2PHYS_EXT`), as Launchpad's
+ Exclusive column does. APFS's private size (`ATTR_CMNEXT_PRIVATESIZE`) reads
+ 0 for every object after each hourly Time Machine local snapshot, until it
+ writes again ([disk use](machine_templates.md#disk-use)).
+- `st_blocks` of a running machine's image is not monotonic: guest TRIM pulls
+ it back. Writes by the VM are better read from the Virtualization process's
+ `ri_diskio_byteswritten` (`proc_pid_rusage`).
+- Deleting a template frees only its own blocks while clones exist; deleting a
+ source frees nothing a clone still shares.
+
+## Observation to verify: plaintext in Data and User blocks
+
+The block-level survey for the block-store study
+([content-addressed store](content_addressed_block_store.md#data-and-user-blocks))
+found that the image region not mapped to System, Preboot, xART, Hardware or
+Update files (Data, User and container metadata), although both volumes show
+as FileVault locked on the host, is not uniformly ciphertext. On the iPhone
+27.0 image about 2.0 GB of it compressed like plaintext (LZ4 ratio 0.55), and
+blocks began with `SQLite format 3` (429 blocks), `bplist00` (4,006), a PNG
+signature (874) and `caff` (711); 2.6 GB looked random. The iPad 26.6.2 image
+showed 3.5 GB compressible and 4.2 GB random.
+
+Read as it stands, someone holding a `Disk.img` could carve some guest file
+contents out of it without the SEP. It has not been checked which files these
+are, which protection classes they carry, or whether the content-protection
+layer behaves differently on virtio storage than on Apple NAND. Treat it as an
+open question, not a finding about iOS.
+
+## Open items
+
+- **Stop the DFU VM at the iBoot banner.** Every restore's post-restore reboot
+ panics because the System volume has no dyld cache before `cfw install`,
+ and `vm create` and Launchpad wait for that panic; its report then lands in
+ the guest. Stopping at the banner would avoid both for every machine
+ ([setup boot](machine_templates.md#setup-boot), last paragraphs on crash
+ reports).
+- **Cold template archives.** Compressing the `Disk.img` of a template no one
+ clones any more (LZFSE, about 30–35% smaller by the block survey) and
+ expanding it back before the next clone would take most of the block
+ store's remaining gain without touching the I/O path. Not built.
+- **ASIF images.** Virtualization's XPC service knows `.asif`, but whether
+ discard frees blocks in an ASIF image was not tested. Raw sparse files
+ already return freed space, `VPhoneAPFSSnapshot` reads and patches the image
+ at raw byte offsets, and `vm rebase` takes file offsets for disk offsets;
+ ASIF would only help where the host volume cannot hold a
+ sparse file.
+- **Growing a disk.** No code resizes a container after the restore, so a
+ larger disk means another template or `--no-template`.
+- **Revisit the block store** only if disk space is still the limit with
+ templates, trim and IPSW deletion in place and three or more builds kept hot
+ at once, or if Apple provides a host-mountable user-space block device or an
+ NBD client with write-back and FLUSH
+ ([content-addressed store](content_addressed_block_store.md#when-to-revisit)).
diff --git a/Research/Host/guest_discard.md b/Research/Host/guest_discard.md
new file mode 100644
index 00000000..28d44d09
--- /dev/null
+++ b/Research/Host/guest_discard.md
@@ -0,0 +1,136 @@
+# Guest TRIM and offline compaction
+
+Whether space the guest frees inside `Disk.img` returns to the host, and
+whether an offline `vm compact` would be worth having. Measured on 2026-10-06
+on two `vm clone` copies of a stopped iOS 27.0 (24A435) / cloudOS 26.4 machine
+with a 32 GB image: one written to, deleted from and hole-punched, the other
+left alone as a control. Part of the [disk-space study](disk_space.md).
+
+## Summary
+
+It does, and no change is needed. With the attachment vphone uses,
+`VZDiskImageStorageDeviceAttachment(url:readOnly: false)` on a
+`VZVirtioBlockDeviceConfiguration` (default caching and synchronization
+modes, raw sparse file, `VPhoneVirtualMachine.swift`), the guest's discards
+reach the host and deallocate the file's blocks:
+
+- 2.1 GB written into the guest and deleted came back to the host within
+ 15 s, to within 9 MB of the baseline.
+- Every APFS mount in the guest also trims all free space.
+- After a clean shutdown, APFS-free but host-allocated space was 16.9 MB, the
+ 1–2 block extents APFS declines to trim. Punching those out offline changed
+ nothing observable in the guest.
+
+So there is no guest kernel patch to write and no `vm compact` worth adding.
+What occupies the host is what the guest really holds, and blocks a clone has
+stopped sharing.
+
+## The guest driver negotiates discard
+
+From the vphone600 research kernelcache:
+
+- `com.apple.driver.AppleVirtIO` matches standard virtio-blk
+ (`IOVirtIOPrimaryMatch` `0x00021af4`) as `AppleVirtIOBlockStorageDevice`.
+ Its feature mask is `0x3a66`, which includes bit 13,
+ `VIRTIO_BLK_F_DISCARD`, and not bit 14, `WRITE_ZEROES`. When discard is
+ negotiated it reads `max_discard_sectors` and `max_discard_seg` from the
+ device config (offsets 0x24 and 0x28; `discard_sector_alignment` is not
+ read), and its `doUnmap` override splits extents by those limits and sends
+ DISCARD requests (type 0xb, segments `{u64 sector, u32 sectors, u32 flags}`).
+ Without the feature, or with either limit 0, it returns
+ `kIOReturnUnsupported` like the `IOBlockStorageDevice` base. With it, the
+ device publishes `IOStorageFeatures` `{Unmap = true}`.
+- The root disk actually attaches through `com.apple.iokit.AppleVirtIOStorage`
+ (`…/AppleVirtIOStorageDevice/AppleVirtIODiskStorageDevice@1/IOBlockStorageDriver`
+ in the boot log), which overrides `doUnmap` too. Either path supports unmap.
+- APFS has both paths: a full free-space trim at mount
+ (`nx_mount_initiate_free_space_trims`, `spaceman_scan_free_blocks`) and
+ unmaps at run time (`spaceman_trim_free_extent_callback`).
+
+The host side was not disassembled, so the exact call Virtualization makes is
+not confirmed. That the file's `st_blocks` drops after a guest delete shows
+the blocks are deallocated (a punched hole) rather than written with zeros,
+which would leave `st_blocks` unchanged. Behaviour under non-default caching
+or synchronization modes was not tested.
+
+## Measured
+
+Guest side, the APFS mount log of the first boot:
+
+```text
+spaceman_scan_free_blocks:4431: disk1 scan took 0.567917 s, trims took 0.567313 s
+spaceman_scan_free_blocks:4444: disk1 2950981 blocks trimmed in 6916 extents (82 us/trim, 12190 trims/s)
+spaceman_scan_free_blocks:4455: disk1 trims dropped: 3068 blocks 2500 extents, avg 1.22
+```
+
+"trimmed" rather than an error is itself evidence that discard was
+negotiated; with `doUnmap` unsupported APFS would not count them.
+(`device.ioreg` returns class and name only and stops at 300 entries, so it
+could not show `IOStorageFeatures`.)
+
+Host side, `stat -f %b Disk.img` in 512-byte blocks:
+
+| Event | Blocks | GB |
+| --- | --- | --- |
+| Cloned, not booted | 38,834,784 | 19.884 |
+| First boot (its mount trim freed 117 MB) | 38,605,416 | 19.766 |
+| Baseline before writing | 39,051,656 | 19.994 |
+| `files.copy` of 16 dyld subcaches (~131 MB each) into `/var/mobile` | 43,150,728 | 22.093 |
+| One minute later | 43,153,744 | 22.095 |
+| `files.remove {recursive}` returned | 40,861,912 | 20.921 |
+| 15 s later | 39,068,848 | 20.003 |
+| 2 minutes later | 39,069,520 | 20.004 |
+| After `system.reboot` (mount trim freed ~106 MB more) | 38,864,064 | 19.898 |
+| After `vm stop` | 39,285,120 | 20.114 |
+
+2.099 GB written, 2.092 GB returned within 15 s; the difference is the
+guest's own writes meanwhile. The reboot's mount trim reported `2922515 blocks
+trimmed in 7101 extents` and `trims dropped: 3959 blocks 2646 extents`. APFS's
+used space at that point was (7,812,490 − 2,922,515) × 4096 = 20.03 GB, against
+19.90 GB the host had allocated: the host held less than APFS counted as used
+(some allocated APFS blocks were never written), so almost nothing was left to
+reclaim.
+
+## Offline analysis
+
+A standalone prototype (not in the repository) read a stopped image without
+keys and with no mount: GPT entry → container superblock → the checkpoint
+descriptor area's newest valid NXSB (Fletcher-64) → the checkpoint map entry
+for the space manager → its chunk-info and chunk-info-address blocks → the
+allocation bitmaps. Free runs were intersected with the file's data ranges
+(`SEEK_DATA`/`SEEK_HOLE`).
+
+```text
+block size 4096, blocks 7812490, latest xid 26277, spaceman oid 1024
+spaceman@14538: blocks 7812490 chunks 239 cibs 2 cabs 0 free 2895575 (11860 MB) freeQueues [19, 9, 0]
+host: size 32000000000, st_blocks*512 20113981440, data extents 6485
+reclaimable (APFS-free but host-allocated): 16932864 bytes in 963 ranges
+```
+
+The bitmap's free count equalled the space manager's `sm_free_count`
+exactly. Punching the 963 ranges with `F_PUNCHHOLE` freed exactly
+16,932,864 bytes, and a second run found nothing. The punched clone booted,
+its mount trim reported the same free count, vphoned answered, and
+`diagnostics.self_test` gave the same 7 passes and 3 failures as the untouched
+control (accessibility frontmost check, OCR, keychain persistence: properties
+of the source machine).
+
+Rules for doing this safely, if it is ever wanted:
+
+- Space manager, checkpoints and bitmaps are unencrypted container metadata;
+ no keys are needed to know which blocks are free.
+- Only blocks the bitmap marks free may go. Keybags, checkpoint areas, the
+ object map, the space manager itself and blocks a snapshot references are
+ all marked allocated. Blocks in the free queues are still marked allocated
+ and are skipped.
+- Only a cleanly stopped image. Punching blocks free in the newest checkpoint
+ but used by an older one removes APFS's ability to fall back to the older
+ checkpoint, which a crashed image may need.
+- macOS `fsck_apfs -n` verifies only the container: it stops at the container
+ keybag, which the guest's SEP wraps (the control image fails identically).
+ The only check of the volumes is booting the guest.
+- On a clone, punching frees only blocks the file does not share.
+
+`vm compact` would save 10–20 MB per machine. The one case where it would
+help, an image written and emptied before any trim ran, is handled just as
+well by booting it once and letting the mount trim run.
diff --git a/Research/README.md b/Research/README.md
index 80173b65..85724bb4 100644
--- a/Research/README.md
+++ b/Research/README.md
@@ -19,7 +19,8 @@ This directory keeps the evidence behind firmware patches, restore and the self-
| Kernel | [CFW overview](Kernel/kernel_jb_patch_notes.md), [patcher verification](Kernel/kernel_patcher_verification.md), [FairPlay kexts](Kernel/kernel_fairplay_kexts.md), [base validation 1–5](Kernel/kernel_patch_base_first5_validation.md), [11–15](Kernel/kernel_patch_base_11_15_validation.md), [16–20](Kernel/kernel_patch_base_16_20_validation.md), [sandbox hooks](Kernel/kernel_patch_sandbox_hooks_17_26_validation.md), [individual patch notes](KernelCustomFirmwarePatches/README.md) |
| Other patches and captures | [Launchd jetsam](Patches/cfw_patch_launchd_jetsam.md), [user-mode hypervisor references](Patches/hv_vmm_present_usermode_xrefs.md), [reference capture](Patches/patch_reference_capture.md) |
| Restore | [DFU probe](Restore/virtual_dfu_probe.md), [in-process restore](Restore/native_restore_architecture.md) |
-| Host and archives | [Binary split](Host/host_binary_split.md), [runtime dependency tiers](Host/runtime_dependency_tiers.md), [archive extraction contracts](Host/archive_extraction_contracts.md), [libarchive validation](Host/libarchive_xcframework_validation.md), [guest MAC, fixed address, port forwarding and names](Host/network_fixed_address.md), [machine identity and `vm clone --new-identity`](Host/machine_identity_and_clone.md), [sharing identical disk blocks with `vm rebase`](Host/disk_rebase.md), [machine templates](Host/machine_templates.md) |
+| Host and archives | [Binary split](Host/host_binary_split.md), [runtime dependency tiers](Host/runtime_dependency_tiers.md), [archive extraction contracts](Host/archive_extraction_contracts.md), [libarchive validation](Host/libarchive_xcframework_validation.md), [guest MAC, fixed address, port forwarding and names](Host/network_fixed_address.md), [machine identity and `vm clone --new-identity`](Host/machine_identity_and_clone.md) |
+| VM disk space | [Where a VM's disk space goes](Host/disk_space.md) (overview and measured savings), [machine templates](Host/machine_templates.md), [sharing identical disk blocks with `vm rebase`](Host/disk_rebase.md), [guest TRIM and offline compaction](Host/guest_discard.md), [why the disk layout was not made deterministic](Host/deterministic_disk_layout.md), [why there is no content-addressed block store over NBD](Host/content_addressed_block_store.md) |
| Guest interaction and VM identity | [DevMode XPC](Guest/devmode_xpc_protocol.md), [keyboard events](Guest/keyboard_event_pipeline.md), [machine identifier](Guest/machine_identifier_storage_analysis.md), [Auto-Lock on the Lock Screen](Guest/lock_screen_idle_timer.md), [display refresh rate](Guest/display_refresh_rate.md), [GPU acceleration](Guest/gpu_acceleration.md), [RootHide bootstrap base](Guest/roothide_bootstrap_base.md), [the Mac's .local name](Guest/mac_name_resolution.md), [locationd startup deadlock](Guest/locationd_startup_deadlock.md), [trimming system services](Guest/service_trimming.md), [deleting the orig-fs snapshot for templates](Guest/template_snapshot_deletion.md) |
| Historical project records | [Migration ledger](History/intg_update_status.md), [manifest refactoring summary](History/manifest_and_refactoring_summary.md) |