Run a virtual iPhone on an Apple Silicon Mac.
Swift
Report only newly shared blocks as a rebase's saving
Jackyeec7350eb71d
diff --git a/Documents/Guides/create-and-run.md b/Documents/Guides/create-and-run.mdindex 7d0e7cce..38ac1568 100644--- a/Documents/Guides/create-and-run.md+++ b/Documents/Guides/create-and-run.md@@ -211,16 +211,18 @@ Run resource-heavy creations **one at a time**. Both the IPSWs and temporary res### Sharing disk space between VMs-Two VMs restored separately from the same IPSW share nothing on disk, although much of their disk images is the same: two iOS 27.0 iPhones hold about 8.6 GB of identical bytes at the same places in their roughly 20 GB of data, mostly the system volume. `vm rebase` makes a stopped VM store those blocks once with another VM's:+Two VMs restored separately from the same IPSW share nothing on disk, although much of their disk images is the same: two iOS 27.0 iPhones hold about 8.6 GB of identical bytes at the same places in their roughly 20 GB of data, mostly the system volume. `vm rebase` makes a stopped VM store those blocks once with another VM's, or with a template's:```sh-vphone-cli vm rebase second --onto template --dry-run-vphone-cli vm rebase second --onto template+vphone-cli vm rebase second --onto first --dry-run+vphone-cli vm rebase second --onto 3f2a91c0d4e7```-It rebuilds `second`'s disk image from an APFS clone of `template`'s and writes in only the blocks that differ. Every byte is compared with the original before the image is replaced, so the guest sees exactly the same disk. Only the disk image changes: `SEPStorage`, `nvram.bin`, `config.plist` and the device identity stay as they are. Both VMs must be stopped and on the same APFS volume, and the volume needs free space for the blocks that are written until the old image is released. `--dry-run` reports what would be shared and written without writing anything.+It rebuilds `second`'s disk image from an APFS clone of the base's and writes in only the blocks that differ. `--onto` takes a VM name or a template identifier (or a unique prefix of one) from `vm template list`; a VM wins when a name matches both. Every byte is compared with the original before the image is replaced, so the guest sees exactly the same disk. Only the disk image changes: `SEPStorage`, `nvram.bin`, `config.plist` and the device identity stay as they are. Rebasing onto a template leaves the template as it is, and the VM does not become one of its clones: `vm template show` does not list it. Both must be stopped and on the same APFS volume, and the volume needs free space for the blocks that are written until the old image is released. `--dry-run` reports what would be shared and written without writing anything.++The saving is the **newly shared** figure. Bytes that are identical but already the same blocks on disk are reported as **already shared**, and rebasing them frees nothing. VMs made by `vm create` from one template already share its blocks, so rebasing one onto another, or onto that template, frees little; the command says so before it compares and again when the saving is under 0.1 GB. The VMs worth rebasing are ones restored without a template (`--no-template`) or imported.- The space comes back only when nothing else holds the old image's blocks. Snapshots of the VM, and clones made from it, keep them; delete those first or expect no saving.- `du` and Finder still count each image at its full size. Free space on the volume (`df`) is the honest measure.-- The two images drift apart again as either VM writes to its disk. A base that is never booted, such as a freshly restored VM kept as a template, stays the best base, and rebasing again later recovers what drifted.+- The two images drift apart again as either VM writes to its disk. A base that is never booted, such as a template, stays the best base, and rebasing again later recovers what drifted.- A rebase that is interrupted leaves the VM unchanged and may leave a hidden `.rebase-*` folder in it; delete that folder.diff --git a/Research/Host/disk_rebase.md b/Research/Host/disk_rebase.mdindex 4f366d36..5122b965 100644--- a/Research/Host/disk_rebase.md+++ b/Research/Host/disk_rebase.md@@ -77,6 +77,59 @@ filled while a 23 MiB one was kept. Holes punched with `F_PUNCHHOLE` stayed.This only matters for the unit tests, which use 64 MiB images with largegaps; a 64 GB `Disk.img` keeps its holes.+### Already shared blocks++Byte-identical is not the same as a saving. Two machines cloned from one+template (`vm create` clones `<library>/.templates/<id>/` with `clonefile`)+read the same bytes over most of their images because they are the same+blocks. Until 2026-10-09 every identical unit counted as shared, so a dry run+of one such clone onto another (live test of PR #633, R1) reported "shared+17.26 GB, written 0.36 GB" although the target held only about 0.35 GB of+blocks of its own (by `F_LOG2PHYS_EXT`): the rebase would have freed nothing.++Before comparing, `plan` and `rebase` now map both images with+`F_LOG2PHYS_EXT`, one call per contiguous extent over the `SEEK_DATA` ranges+(`ImageFile.physicalRuns`), and intersect the two maps+(`VPhoneDiskRebase.sharedRanges`): a file offset where both files point at+the same device offset is already shared. An identical unit then splits+into:++- **newly shared** (`newlySharedBytes`): the target's data there that is not+ already the base's. These are the blocks the rebase gives back, so the+ headline of both outputs is this figure.+- **already shared** (`alreadySharedBytes`, the rest of `sharedBytes`):+ blocks that already are the base's, or a hole in the target. Sharing them+ again frees nothing.++The map is taken from the two original images, before the staging clone+exists, so a dry run and the rebase report the same figures. Nothing else+changes: the staging file is a clone of the base, so an already shared unit+is neither read differently nor written. A volume that does not map files+gives no ranges, and every identical byte of target data counts as newly+shared, as before. Unlike APFS's private size, a Time Machine local snapshot+does not make blocks look shared here: only the two files are compared (see+the disk-use section of [machine templates](machine_templates.md)).++When the newly shared bytes stay under 0.1 GB the command adds "the rebase+would free almost nothing" and says why: the identical blocks are already+shared, or few are identical. Before comparing it also notes when the+records say both sides come from one template build: the target's+`TemplateSource.plist` names the template given as `--onto`+(`VPhoneMachineTemplateSource.isClone(of:)`), or both machines' records name+the same template and `Build` (`VPhoneDiskRebaseBase.commonTemplate`).++### A template as the base++`--onto` takes a machine name or a template identifier (or unique prefix,+as `vm template` takes it), resolved by `VPhoneDiskRebaseBase.resolve`; a+machine whose name matches wins. A frozen template is never booted, so its+image never drifts, which makes it the base the caveats below ask for. The+rebase only reads and clones the template's image. The rebased machine gets+no `TemplateSource.plist`: it keeps its own `SEPStorage`, NVRAM and+identity and only shares disk blocks, so `vm template list/show`, Launchpad+and `vm delete`'s "no longer used" note do not count it as a clone. A+machine that already had a `TemplateSource.plist` keeps it.+## Caveats- The space comes back only when the old image's blocks have no other owner.@@ -84,7 +137,7 @@ gaps; a 64 GB `Disk.img` keeps its holes.rebase then costs the written bytes instead of saving the shared ones.- `du` and Finder count each image at its full size; `df` is the measure.- Both images diverge again as either machine writes. A base that is never- booted makes the best one.+ booted, a template, makes the best one.## Measurements@@ -104,6 +157,8 @@ same), so that the old image's blocks had no other owner.| Free space (`df`, 1K blocks) | 66,360,908 | 75,132,760 (+8.77 GB) || SHA-256 of `rbtest-a/Disk.img` (64 GB) | `c14ced17…c27810d` | `c14ced17…c27810d` |+In the output format of the time, before shared bytes were split:+```text$ vphone-cli vm rebase rbtest-a --onto rbtest-brebased rbtest-a onto rbtest-b in 00:17@@ -132,3 +187,29 @@ rebased rbtest-a onto rbtest-b in 00:17- Deleting the three test machines afterwards returned 12.3 GB: the 11.95 GBwritten plus what `rbtest-boot` wrote while it ran.+Already-shared accounting, 2026-10-09, Release `vphone-cli` on a scratch+library (no real machine): `rb-a` with 2 GiB of random data at 1 GiB in a+64 GB sparse image, `rb-b` and `rb-c` `clonefile` copies of it, `rb-b` then+given 256 MiB of new bytes and 256 MiB rewritten with the same bytes, and+`rb-d` the same 2 GiB written into a separate file.++```text+$ vphone-cli vm rebase rb-b --onto rb-c --dry-run+note: rb-b and rb-c were both cloned from template 3f2a91c0d4e7 and already share its blocks; expect little to be newly shared+dry run: 0.27 GB would be newly shared with rb-c; compared rb-b in 00:00, nothing was changed+ disk image 64.00 GB logical size; 2.15 GB holds data in either image+ newly shared 0.27 GB identical to rb-c at the same offset, would be shared+ already shared 1.61 GB identical, and already the same blocks on disk (or a hole): frees nothing+ written 0.27 GB differs from rb-c, would be written+ punched 0.00 GB zeros where rb-c has data, would become holes++$ vphone-cli vm rebase rb-c --onto rb-a --dry-run+dry run: 0.00 GB would be newly shared with rb-a; …+note: the rebase would free almost nothing: rb-c already shares its identical blocks with rb-a on disk+```++(The template note came from hand-written `TemplateSource.plist` files.)+`vm rebase rb-d --onto rb-a` reported 2.15 GB newly shared and `df` rose by+2,097,212 KiB (2.15 GB); a dry run of it afterwards reported 0.00 GB newly+and 2.15 GB already shared. Mapping both images took well under a second.+diff --git a/Research/Host/machine_templates.md b/Research/Host/machine_templates.mdindex 4a707886..b64516fd 100644--- a/Research/Host/machine_templates.md+++ b/Research/Host/machine_templates.md@@ -128,6 +128,14 @@ re-resolves with the template's own `PatchSelection.plist`.so it no longer says "its space is free now" (retest B8). `--json` (with`--force`) prints `deleted`, `path`, `machines`, `blocksFreed` and thesame `note`.+- `vm rebase <vm> --onto <id>` takes a template as the base+ (`VPhoneDiskRebaseBase.resolve`; a machine with that name wins). It only+ reads and clones the template's image, and writes no `TemplateSource.plist`+ into the rebased machine: that machine shares disk blocks but keeps its own+ SEP storage and identity, so `usage` does not count it. Rebasing a clone of+ a template onto that template, or onto another clone of the same `Build`,+ frees nearly nothing; the command notes it and reports such blocks as+ already shared (`F_LOG2PHYS_EXT`, see [disk rebase](disk_rebase.md)).- A build happens in `.building-<id>-<uuid>/<id>/` and is frozen by writing`Frozen = true` and one `renamex_np(RENAME_EXCL)` to `.templates/<id>`. Alisted template is always complete; a race with another build of the samediff --git a/VPhoneExecutable/VPhoneCommand/VPhoneCommand/VirtualMachine/VPhoneVirtualMachineRebaseCommand.swift b/VPhoneExecutable/VPhoneCommand/VPhoneCommand/VirtualMachine/VPhoneVirtualMachineRebaseCommand.swiftindex 447fe023..69aa2490 100644--- a/VPhoneExecutable/VPhoneCommand/VPhoneCommand/VirtualMachine/VPhoneVirtualMachineRebaseCommand.swift+++ b/VPhoneExecutable/VPhoneCommand/VPhoneCommand/VirtualMachine/VPhoneVirtualMachineRebaseCommand.swift@@ -8,23 +8,30 @@ import VPhoneCoreKitstruct VPhoneVirtualMachineRebaseCommand: ParsableCommand {static let configuration = CommandConfiguration(commandName: "rebase",- abstract: "Share a stopped VM's identical disk blocks with another VM",+ abstract: "Share a stopped VM's identical disk blocks with another VM or a template",discussion: """- Compares the disk images of two stopped VMs and rebuilds the first one's from an \+ Compares the disk images of a stopped VM and a base and rebuilds the VM's from an \APFS clone of the base's, writing only the blocks that differ. Wherever both images \hold the same bytes at the same offset (for VMs restored from the same IPSW, most of \the system volume), the blocks are then stored once for both. Every byte is checked \against the original before it is replaced, so the guest sees exactly the same disk; \SEPStorage, nvram.bin, config.plist and the device identity are not touched.- Both VMs must be stopped and on one APFS volume. The rebase needs free space for the \- blocks it writes until the old image is released.+ The base is another stopped VM or a template from `vm template list`, by identifier \+ or unique prefix; a VM wins when a name matches both. A template is never booted, so \+ it makes the best base: its image does not drift. The rebase only reads it, and the \+ rebased VM does not become one of its clones (it keeps its own SEP storage and \+ identity, and `vm template show` does not list it).- The space comes back only if nothing else holds the old image's blocks: snapshots of \- the VM, and clones made from it, keep them. du and Finder still report the full size \- of each image; free space on the volume is the honest measure. The two images drift \- apart again as either VM writes to its disk, so a base that is never booted, such as \- a freshly restored template, makes the best base.+ Identical bytes that are already the same blocks on disk, as in two VMs cloned from \+ one template, are reported as already shared: rebasing them frees nothing. The \+ saving is the newly shared bytes.++ Both must be on one APFS volume. The rebase needs free space for the blocks it \+ writes until the old image is released. The space comes back only if nothing else \+ holds the old image's blocks: snapshots of the VM, and clones made from it, keep \+ them. du and Finder still report the full size of each image; free space on the \+ volume is the honest measure. The two images drift apart again as either side writes.--dry-run only compares the images and reports what a rebase would share and write.""",@@ -32,13 +39,17 @@ struct VPhoneVirtualMachineRebaseCommand: ParsableCommand {@OptionGroup var lib: VPhoneLibraryOption@Argument(help: "VM whose disk image is rebased") var name: String?- @Option(name: .long, help: "VM whose disk blocks it shares") var onto: String+ @Option(name: .long, help: "VM or template identifier whose disk blocks it shares") var onto: String@Flag(help: "only report what would be shared and written") var dryRun = falsefunc run() throws {let name = try VPhoneVirtualMachineSelection.resolveExisting(name, in: lib.library)let target = try lib.library.bundle(named: name)- let base = try lib.library.bundle(named: onto)+ let base = try VPhoneDiskRebaseBase.resolve(onto, in: lib.library)+ if let template = base.commonTemplate(with: target, in: lib.library) {+ let both = base.template == nil ? "\(target.name) and \(base.label) were both cloned from" : "\(target.name) was cloned from"+ print("note: \(both) template \(template) and already share its blocks; expect little to be newly shared")+ }let clock = ContinuousClock()let start = clock.now@@ -53,21 +64,25 @@ struct VPhoneVirtualMachineRebaseCommand: ParsableCommand {bar?.update(done: done, total: total)}let report = dryRun- ? try VPhoneDiskRebase.plan(target, onto: base, progress: progress)- : try VPhoneDiskRebase.rebase(target, onto: base, progress: progress)+ ? try VPhoneDiskRebase.plan(target, onto: base.bundle, progress: progress)+ : try VPhoneDiskRebase.rebase(target, onto: base.bundle, progress: progress)bar?.finish()let elapsed = clock.now - startlet seconds = Double(elapsed.components.seconds) + Double(elapsed.components.attoseconds) / 1e18let elapsedText = VPhoneProgressBar.clock(seconds)+ let saving = Self.gigabytes(report.newlySharedBytes)if dryRun {- print("dry run: compared \(target.name) with \(base.name) in \(elapsedText); nothing was changed")+ print("dry run: \(saving) would be newly shared with \(base.label); compared \(target.name) in \(elapsedText), nothing was changed")} else {- print("rebased \(target.name) onto \(base.name) in \(elapsedText)")+ print("rebased \(target.name) onto \(base.label) in \(elapsedText); \(saving) newly shared")}- for line in Self.describe(report, base: base.name, dryRun: dryRun) {+ for line in Self.describe(report, base: base.label, dryRun: dryRun) {print(" " + line)}+ if let note = Self.savingNote(report, target: target.name, base: base.label, dryRun: dryRun) {+ print("note: " + note)+ }if !dryRun, let snapshots = try? VPhoneMachineSnapshots.list(of: target), !snapshots.isEmpty {print("note: \(target.name) has \(snapshots.count) snapshot(s), which keep the old image's blocks until deleted")}@@ -77,20 +92,34 @@ struct VPhoneVirtualMachineRebaseCommand: ParsableCommand {static func describe(_ report: VPhoneDiskRebaseReport, base: String, dryRun: Bool) -> [String] {let rows: [(String, Int64, String)] = [("disk image", report.logicalSize, "logical size; \(gigabytes(report.comparedBytes)) holds data in either image"),- ("shared", report.sharedBytes, dryRun+ ("newly shared", report.newlySharedBytes, dryRun? "identical to \(base) at the same offset, would be shared": "identical to \(base) at the same offset, now shared"),+ ("already shared", report.alreadySharedBytes, "identical, and already the same blocks on disk (or a hole): frees nothing"),("written", report.writtenBytes, dryRun ? "differs from \(base), would be written" : "differs from \(base), written"),("punched", report.punchedBytes, dryRun ? "zeros where \(base) has data, would become holes" : "zeros where \(base) has data, now holes"),]let sizes = rows.map { gigabytes($0.1) }let width = sizes.map(\.count).max() ?? 0return zip(rows, sizes).map { row, size in- row.0.padding(toLength: 12, withPad: " ", startingAt: 0)+ row.0.padding(toLength: 16, withPad: " ", startingAt: 0)+ String(repeating: " ", count: width - size.count) + size + " " + row.2}}+ /// Below this, the rebase is reported as freeing almost nothing.+ static let negligibleSaving: Int64 = 100_000_000++ /// Why a rebase that newly shares almost nothing is not worth it.+ static func savingNote(_ report: VPhoneDiskRebaseReport, target: String, base: String, dryRun: Bool) -> String? {+ guard report.newlySharedBytes < negligibleSaving else { return nil }+ let frees = dryRun ? "the rebase would free almost nothing" : "the rebase freed almost nothing"+ if report.alreadySharedBytes > report.newlySharedBytes {+ return "\(frees): \(target) already shares its identical blocks with \(base) on disk"+ }+ return "\(frees): \(target) has few blocks identical to \(base)"+ }+static func gigabytes(_ bytes: Int64) -> String {String(format: "%.2f GB", Double(bytes) / 1e9)}diff --git a/VPhoneKit/VPhoneCoreKit/Bundle/VPhoneDiskRebase.swift b/VPhoneKit/VPhoneCoreKit/Bundle/VPhoneDiskRebase.swiftindex 7bd8113a..394d934d 100644--- a/VPhoneKit/VPhoneCoreKit/Bundle/VPhoneDiskRebase.swift+++ b/VPhoneKit/VPhoneCoreKit/Bundle/VPhoneDiskRebase.swift@@ -16,18 +16,92 @@ public struct VPhoneDiskRebaseReport: Equatable, Sendable {/// stored once for both machines. Counted in whole units of/// `VPhoneDiskRebase.blockSize`.public var sharedBytes: Int64 = 0+ /// The part of `sharedBytes` the target stores in blocks of its own+ /// before the rebase: what the rebase saves, once nothing else holds+ /// those blocks. The rest costs the target nothing already: there it+ /// holds no data, or its blocks are the base's, at the same device+ /// offset by `F_LOG2PHYS_EXT`, as in two clones of one template.+ public var newlySharedBytes: Int64 = 0/// Bytes that differ from the base and were written from the target.public var writtenBytes: Int64 = 0/// Bytes where the base holds data and the target reads as zeros, a hole/// or zero-filled blocks. They become holes in the rebased image.public var punchedBytes: Int64 = 0+ /// Identical bytes that already cost the target nothing: sharing them+ /// again frees nothing.+ public var alreadySharedBytes: Int64 {+ sharedBytes - newlySharedBytes+ }+public init() {}}+// MARK: - Base++/// What a machine is rebased onto: another machine of the library, or a+/// frozen template in its `.templates`. A template is never booted, so its+/// image does not drift; the rebase only clones it and leaves it as it is.+public struct VPhoneDiskRebaseBase: Sendable {+ public let bundle: VPhoneBundle+ /// Set when the base is a template.+ public let template: VPhoneMachineTemplate?++ public init(bundle: VPhoneBundle, template: VPhoneMachineTemplate? = nil) {+ self.bundle = bundle+ self.template = template+ }++ /// How output names it.+ public var label: String {+ template.map { "template \($0.identifier)" } ?? bundle.name+ }++ /// The machine named `name`, or else the template with that identifier+ /// or unique prefix. A machine wins when both match.+ public static func resolve(_ name: String, in library: VPhoneLibrary) throws -> Self {+ do {+ return try Self(bundle: library.bundle(named: name))+ } catch VPhoneLibraryError.notFound {+ let template: VPhoneMachineTemplate+ do {+ template = try VPhoneMachineTemplates.template(name, in: library)+ } catch VPhoneMachineTemplateError.notFound, VPhoneMachineTemplateError.invalidIdentifier {+ throw VPhoneDiskRebaseError.baseNotFound(name: name)+ }+ return try Self(bundle: template.bundle(), template: template)+ }+ }++ /// The template that `target` and this base both come from, by their+ /// records: `target` was cloned from this template, or both machines+ /// were cloned from the same build of one. Their images then already+ /// share its blocks wherever neither has written since, and a rebase+ /// frees little. Nil when the records do not show it.+ public func commonTemplate(with target: VPhoneBundle, in library: VPhoneLibrary) -> String? {+ guard let source = VPhoneMachineTemplates.readSource(inBundle: target.url) else { return nil }+ if let template {+ return source.isClone(of: template.record) ? template.identifier : nil+ }+ guard let other = VPhoneMachineTemplates.readSource(inBundle: bundle.url),+ other.identifier == source.identifier+ else { return nil }+ if let build = source.build, let otherBuild = other.build {+ return build == otherBuild ? source.identifier : nil+ }+ // A record without a build: only the template that exists now, by+ // the dates `isClone` compares, can tell.+ guard let template = try? VPhoneMachineTemplates.template(source.identifier, in: library),+ source.isClone(of: template.record), other.isClone(of: template.record)+ else { return nil }+ return source.identifier+ }+}+// MARK: - Rebase-/// Re-shares a stopped machine's disk image with another machine's, offline.+/// Re-shares a stopped machine's disk image with another machine's, or a+/// template's, offline.////// Two machines restored separately from one IPSW share nothing on disk, yet/// most of their system volume is byte-identical at the same offset in both@@ -112,6 +186,7 @@ public enum VPhoneDiskRebase {target: targetFile,reference: baseFile,referenceData: baseFile.dataRanges(upTo: min(size, baseFile.size)),+ alreadyShared: physicallyShared(targetFile, baseFile),size: size,chunkSize: options.chunkSize,)@@ -143,6 +218,16 @@ public enum VPhoneDiskRebase {try VPhoneBundleActivity.requireStopped(target)try VPhoneBundleActivity.requireStopped(base)let original = try FileState(path: targetImage.path)+ // Mapped before the clone exists, from the two images as they are: a+ // block of the target that is the base's already costs nothing now+ // and frees nothing later.+ let alreadyShared = try {+ let targetFile = try ImageFile(path: targetImage.path, writable: false)+ defer { targetFile.close() }+ let baseFile = try ImageFile(path: baseImage.path, writable: false)+ defer { baseFile.close() }+ return try physicallyShared(targetFile, baseFile)+ }()let fm = FileManager.defaultlet staging = target.url.appendingPathComponent(stagingPrefix + UUID().uuidString, isDirectory: true)@@ -173,6 +258,7 @@ public enum VPhoneDiskRebase {target: targetFile,reference: stagedFile,referenceData: stagedFile.dataRanges(upTo: size),+ alreadyShared: alreadyShared,size: size,chunkSize: options.chunkSize,)@@ -295,6 +381,8 @@ public enum VPhoneDiskRebase {let reference: ImageFile/// The reference's data ranges, sorted, within the target's size.let referenceData: [Range<Int64>]+ /// Where the target already is the base's blocks, sorted.+ let alreadyShared: [Range<Int64>]let size: Int64let chunkSize: Int@@ -305,12 +393,15 @@ public enum VPhoneDiskRebase {var report = VPhoneDiskRebaseReport()report.logicalSize = sizelet block = VPhoneDiskRebase.blockSize- let spans = try alignedUnion(target.dataRanges(upTo: size), referenceData, block: Int64(block), size: size)+ let targetData = try target.dataRanges(upTo: size)+ let spans = alignedUnion(targetData, referenceData, block: Int64(block), size: size)let total = spans.reduce(Int64(0)) { $0 + Int64($1.count) }let ours = AlignedBuffer(size: chunkSize)let theirs = AlignedBuffer(size: chunkSize)let zeros = AlignedBuffer(size: block)var cursor = 0 // into referenceData; blocks are visited in order+ var ownData = Overlap(ranges: targetData)+ var baseBlocks = Overlap(ranges: alreadyShared)var done: Int64 = 0progress(.comparing, 0, total)@@ -351,6 +442,11 @@ public enum VPhoneDiskRebase {try flush()if referenceHasData {report.sharedBytes += Int64(count)+ // Only the target's own blocks are a saving:+ // not its holes, nor blocks that already are+ // the base's (a subset of its data).+ let unit = at ..< at + Int64(count)+ report.newlySharedBytes += ownData.bytes(in: unit) - baseBlocks.bytes(in: unit)}} else {// Zeros where the reference has other bytes become@@ -383,6 +479,30 @@ public enum VPhoneDiskRebase {}}+ /// How many bytes of sorted, disjoint ranges lie in each of a series of+ /// windows that only move forward.+ private struct Overlap {+ let ranges: [Range<Int64>]+ private var index = 0++ init(ranges: [Range<Int64>]) {+ self.ranges = ranges+ }++ mutating func bytes(in window: Range<Int64>) -> Int64 {+ while index < ranges.count, ranges[index].upperBound <= window.lowerBound {+ index += 1+ }+ var total: Int64 = 0+ var next = index+ while next < ranges.count, ranges[next].lowerBound < window.upperBound {+ total += min(window.upperBound, ranges[next].upperBound) - max(window.lowerBound, ranges[next].lowerBound)+ next += 1+ }+ return total+ }+ }+/// Calls `body` for each piece of `span`, split at multiples of/// `chunkSize` so pieces of different spans never overlap a boundary.private static func forEachChunk(@@ -400,6 +520,57 @@ public enum VPhoneDiskRebase {}}+ // MARK: - Physical sharing++ /// A run of a file's data that lies contiguously on the device: the+ /// byte at `logical.lowerBound + n` is at device offset `physical + n`.+ struct PhysicalRun: Equatable {+ let logical: Range<Int64>+ let physical: Int64+ }++ /// Where the target's data, below its size and the base's, is already+ /// the base's blocks: the same device offset for the same file offset.+ /// Two clones of one image, or a clone and its source, share these until+ /// either writes them. A volume that does not map files (not APFS, a+ /// compressed file) gives no ranges, and every identical byte the target+ /// holds data for then counts as newly shared.+ private static func physicallyShared(_ target: ImageFile, _ base: ImageFile) throws -> [Range<Int64>] {+ guard target.device == base.device else { return [] }+ let limit = min(target.size, base.size)+ guard let ours = try target.physicalRuns(in: target.dataRanges(upTo: limit)),+ let theirs = try base.physicalRuns(in: base.dataRanges(upTo: limit))+ else { return [] }+ return sharedRanges(ours, theirs)+ }++ /// The file offsets where two files' runs, each sorted and disjoint,+ /// point at the same device offsets, merged.+ static func sharedRanges(_ a: [PhysicalRun], _ b: [PhysicalRun]) -> [Range<Int64>] {+ var shared: [Range<Int64>] = []+ var i = 0+ var j = 0+ while i < a.count, j < b.count {+ let lower = max(a[i].logical.lowerBound, b[j].logical.lowerBound)+ let upper = min(a[i].logical.upperBound, b[j].logical.upperBound)+ // Same device offset at one file offset of the overlap means the+ // same at all of them: both runs are contiguous.+ if lower < upper, a[i].physical - a[i].logical.lowerBound == b[j].physical - b[j].logical.lowerBound {+ if let last = shared.last, last.upperBound == lower {+ shared[shared.count - 1] = last.lowerBound ..< upper+ } else {+ shared.append(lower ..< upper)+ }+ }+ if a[i].logical.upperBound <= b[j].logical.upperBound {+ i += 1+ } else {+ j += 1+ }+ }+ return shared+ }+/// The union of two sorted range lists, widened to whole blocks (the last/// one ends at `size`) and merged.static func alignedUnion(_ a: [Range<Int64>], _ b: [Range<Int64>], block: Int64, size: Int64) -> [Range<Int64>] {@@ -488,6 +659,7 @@ private final class AlignedBuffer {private final class ImageFile {let path: Stringlet descriptor: Int32+ let device: dev_tlet size: Int64private var isOpen = true@@ -504,6 +676,7 @@ private final class ImageFile {Darwin.close(descriptor)throw VPhoneDiskRebaseError.failed(path: path, reason: reason)}+ device = info.st_devsize = Int64(info.st_size)}@@ -544,6 +717,39 @@ private final class ImageFile {return ranges}+ /// Where `ranges` of the file lie on the device, by `F_LOG2PHYS_EXT`, one+ /// call per contiguous extent: a 20 GB image of about 120,000 extents+ /// maps in a tenth of a second. Nil when the file system does not map+ /// files; a stretch without a device offset (not yet allocated) is left+ /// out.+ func physicalRuns(in ranges: [Range<Int64>]) -> [VPhoneDiskRebase.PhysicalRun]? {+ var runs: [VPhoneDiskRebase.PhysicalRun] = []+ for range in ranges {+ var position = range.lowerBound+ while position < range.upperBound {+ var request = log2phys()+ request.l2p_devoffset = off_t(position)+ request.l2p_contigbytes = off_t(range.upperBound - position)+ guard fcntl(descriptor, F_LOG2PHYS_EXT, &request) == 0, request.l2p_contigbytes > 0 else {+ return nil+ }+ let length = min(Int64(request.l2p_contigbytes), range.upperBound - position)+ let physical = Int64(request.l2p_devoffset)+ if physical >= 0 {+ if let last = runs.last, last.logical.upperBound == position,+ last.physical + Int64(last.logical.count) == physical+ {+ runs[runs.count - 1] = .init(logical: last.logical.lowerBound ..< position + length, physical: last.physical)+ } else {+ runs.append(.init(logical: position ..< position + length, physical: physical))+ }+ }+ position += length+ }+ }+ return runs+ }+/// Fills `length` bytes from `offset`; past the end of the file they/// read as zeros, as they do in a hole.func read(into buffer: AlignedBuffer, length: Int, at offset: Int64) throws {@@ -621,6 +827,7 @@ private final class ImageFile {public enum VPhoneDiskRebaseError: Error, Equatable {case sameMachine(name: String)+ case baseNotFound(name: String)case missingImage(machine: String, path: String)case notCloneable(path: String)case insufficientSpace(path: String, available: Int64, reserve: Int64)@@ -635,6 +842,8 @@ extension VPhoneDiskRebaseError: CustomStringConvertible, LocalizedError {switch self {case let .sameMachine(name):"Cannot rebase VM '\(name)' onto itself. Name a different VM as the base."+ case let .baseNotFound(name):+ "No VM or template named '\(name)'. Name a VM from `vphone-cli vm list` or a template from `vphone-cli vm template list`."case let .missingImage(machine, path):"VM '\(machine)' has no disk image at \(path)."case let .notCloneable(path):diff --git a/VPhoneKit/VPhoneCoreKitTests/Bundle/DiskRebaseTests.swift b/VPhoneKit/VPhoneCoreKitTests/Bundle/DiskRebaseTests.swiftindex 9419c0a3..af5205d5 100644--- a/VPhoneKit/VPhoneCoreKitTests/Bundle/DiskRebaseTests.swift+++ b/VPhoneKit/VPhoneCoreKitTests/Bundle/DiskRebaseTests.swift@@ -518,6 +518,199 @@ struct DiskRebaseTests {#expect(try modified(of: machines.targetImage) == 1_000_000_000 * 1_000_000_000)}+ // MARK: - Physical sharing++ /// Replaces the target's image with a clone of the base's: every block+ /// the same, as for two machines cloned from one template.+ private func cloneBase(_ machines: Machines) throws {+ unlink(machines.targetImage.path)+ try #require(clonefile(machines.baseImage.path, machines.targetImage.path, 0) == 0)+ }++ /// Writes `bytes` over units of an image in place and flushes them, so+ /// a clone takes new blocks there.+ private func overwrite(_ url: URL, units: Range<Int>, with bytes: [UInt8]) throws {+ let fd = open(url.path, O_RDWR)+ try #require(fd >= 0)+ defer { close(fd) }+ let unit = Self.unit+ let put = bytes[units.lowerBound * unit ..< units.upperBound * unit].withUnsafeBytes {+ pwrite(fd, $0.baseAddress, $0.count, off_t(units.lowerBound * unit))+ }+ try #require(put == units.count * unit)+ try #require(fcntl(fd, F_FULLFSYNC) == 0)+ }++ @Test func `identical bytes that already are the base's blocks are not newly shared`() throws {+ let machines = try makeMachines()+ defer { try? FileManager.default.removeItem(at: machines.root) }+ let unit = Self.unit+ let data = Self.pattern(seed: 81, count: 2 << 20)+ try writeImage(machines.baseImage, [Extent(0, data)])+ try cloneBase(machines)++ // A plain clone: everything identical is already shared.+ let clone = try VPhoneDiskRebase.plan(machines.target, onto: machines.base, options: options(), progress: { _, _, _ in })+ #expect(clone.sharedBytes == Int64(data.count))+ #expect(clone.alreadySharedBytes == clone.sharedBytes)+ #expect(clone.newlySharedBytes == 0)++ // The same bytes written again take blocks of the target's own, which+ // the rebase gives back; changed bytes are written as before.+ try overwrite(machines.targetImage, units: 10 ..< 20, with: data)+ var changed = data+ for index in 40 * unit ..< 45 * unit {+ changed[index] ^= 0xFF+ }+ try overwrite(machines.targetImage, units: 40 ..< 45, with: changed)+ #expect(try !shares(machines, at: 10 * unit))+ #expect(try shares(machines, at: 30 * unit))++ let report = try rebase(machines)+ #expect(report.writtenBytes == Int64(5 * unit))+ #expect(report.sharedBytes == Int64(data.count - 5 * unit))+ #expect(report.newlySharedBytes == Int64(10 * unit))+ #expect(report.alreadySharedBytes == Int64(data.count - 15 * unit))+ for index in 0 ..< data.count / unit {+ #expect(try shares(machines, at: index * unit) == !(40 ..< 45).contains(index), "unit \(index)")+ }++ // Rebased: nothing left to share.+ let again = try VPhoneDiskRebase.plan(machines.target, onto: machines.base, options: options(), progress: { _, _, _ in })+ #expect(again.newlySharedBytes == 0)+ #expect(again.alreadySharedBytes == again.sharedBytes)+ }++ @Test func `images written apart share nothing yet, and a hole in the target is no saving`() throws {+ let machines = try makeMachines()+ defer { try? FileManager.default.removeItem(at: machines.root) }+ let unit = Self.unit+ let data = Self.pattern(seed: 91, count: 1 << 20)+ // The base also holds zeros where the target has a hole: identical,+ // but the target stores nothing there to give back.+ let zeros = [UInt8](repeating: 0, count: 8 * unit)+ try writeImage(machines.baseImage, [Extent(0, data), Extent(40 << 20, zeros)])+ try writeImage(machines.targetImage, [Extent(0, data)])++ let baseData = try dataBytes(of: machines.baseImage)+ let targetData = try dataBytes(of: machines.targetImage)+ let report = try rebase(machines)+ // APFS may fill a smaller hole of either image with zeros, which+ // reads the same; only the target's data is its own.+ #expect(report.sharedBytes == Int64(baseData))+ #expect(report.newlySharedBytes == Int64(targetData))+ #expect(report.alreadySharedBytes >= Int64(zeros.count))+ }++ @Test func `runs at the same device offset for the same file offset are shared`() {+ typealias Run = VPhoneDiskRebase.PhysicalRun+ let target = [+ Run(logical: 0 ..< 100, physical: 1000), // the base's at 0..<60+ Run(logical: 100 ..< 200, physical: 5000), // its own+ Run(logical: 300 ..< 400, physical: 1300), // the base's at 320..<400+ ]+ let base = [+ Run(logical: 0 ..< 60, physical: 1000),+ Run(logical: 60 ..< 100, physical: 9000),+ Run(logical: 100 ..< 200, physical: 6000),+ Run(logical: 320 ..< 500, physical: 1320),+ ]+ #expect(VPhoneDiskRebase.sharedRanges(target, base) == [0 ..< 60, 320 ..< 400])+ #expect(VPhoneDiskRebase.sharedRanges(target, target) == [0 ..< 200, 300 ..< 400])+ #expect(VPhoneDiskRebase.sharedRanges([], base).isEmpty)+ }++ // MARK: - Templates++ private func adoptTemplate(_ name: String, in library: VPhoneLibrary) throws -> VPhoneMachineTemplate {+ try VPhoneMachineTemplates.adopt(+ machineNamed: name,+ in: library,+ record: VPhoneMachineTemplateRecord(key: MachineTemplateKeyTests.key(), sourceMachine: name),+ )+ }++ @Test func `a template is a base by identifier or prefix, and a machine with its name wins`() throws {+ let machines = try makeMachines()+ defer { try? FileManager.default.removeItem(at: machines.root) }+ let library = VPhoneLibrary(root: machines.root)+ let template = try adoptTemplate("base", in: library)+ let id = template.identifier++ let byIdentifier = try VPhoneDiskRebaseBase.resolve(id, in: library)+ #expect(byIdentifier.template?.identifier == id)+ #expect(byIdentifier.bundle.url.standardizedFileURL == template.url.standardizedFileURL)+ #expect(byIdentifier.label == "template \(id)")+ #expect(try VPhoneDiskRebaseBase.resolve(String(id.prefix(6)), in: library).template?.identifier == id)+ #expect(try VPhoneDiskRebaseBase.resolve("target", in: library).template == nil)+ #expect(throws: VPhoneDiskRebaseError.baseNotFound(name: "nothing")) {+ try VPhoneDiskRebaseBase.resolve("nothing", in: library)+ }++ try VPhoneBundleOperations.clone(bundleNamed: "target", to: id, in: library)+ let machine = try VPhoneDiskRebaseBase.resolve(id, in: library)+ #expect(machine.template == nil)+ #expect(machine.bundle.url.standardizedFileURL == library.url(forName: id).standardizedFileURL)+ }++ @Test func `rebasing onto a template leaves it as it is and makes no clone of it`() throws {+ let machines = try makeMachines()+ defer { try? FileManager.default.removeItem(at: machines.root) }+ let library = VPhoneLibrary(root: machines.root)+ let extents = [Extent(0, seed: 101, count: 1 << 20), Extent(40 << 20, seed: 102, count: 256 << 10)]+ try writeImage(machines.baseImage, extents)+ try writeImage(machines.targetImage, [extents[0], Extent(40 << 20, seed: 103, count: 256 << 10)])+ let template = try adoptTemplate("base", in: library)+ let base = try VPhoneDiskRebaseBase.resolve(template.identifier, in: library)+ let templateImage = template.url.appendingPathComponent("Disk.img")+ let templateBefore = machines.root.appendingPathComponent("template-before")+ try #require(clonefile(templateImage.path, templateBefore.path, 0) == 0)+ let modified = try modified(of: templateImage)+ let record = try VPhoneMachineTemplates.readRecord(inBundle: template.url)++ let report = try VPhoneDiskRebase.rebase(machines.target, onto: base.bundle, options: options(), progress: { _, _, _ in })+ #expect(report.sharedBytes >= Int64(extents[0].bytes.count))+ #expect(report.alreadySharedBytes == 0)+ for offset in stride(from: 0, to: extents[0].bytes.count, by: Self.unit) {+ #expect(try physical(machines.targetImage, at: offset) == physical(templateImage, at: offset))+ }+ #expect(try sameBytes(templateImage, templateBefore))+ #expect(try self.modified(of: templateImage) == modified)+ #expect(try VPhoneMachineTemplates.readRecord(inBundle: template.url) == record)+ // It shares blocks, but it is not a clone of the template.+ #expect(VPhoneMachineTemplates.readSource(inBundle: machines.target.url) == nil)+ #expect(VPhoneMachineTemplates.usage(in: library).isEmpty)+ #expect(base.commonTemplate(with: machines.target, in: library) == nil)+ }++ @Test func `machines cloned from one build of a template are noted as already sharing it`() throws {+ let machines = try makeMachines()+ defer { try? FileManager.default.removeItem(at: machines.root) }+ let library = VPhoneLibrary(root: machines.root)+ try writeImage(machines.baseImage, [Extent(0, seed: 111, count: 1 << 20)])+ let template = try adoptTemplate("base", in: library)+ let a = try VPhoneMachineTemplates.cloneMachine(from: template, to: "a", in: library)+ let b = try VPhoneMachineTemplates.cloneMachine(from: template, to: "b", in: library)++ let onTemplate = try VPhoneDiskRebaseBase.resolve(template.identifier, in: library)+ let onB = try VPhoneDiskRebaseBase.resolve("b", in: library)+ #expect(onTemplate.commonTemplate(with: a, in: library) == template.identifier)+ #expect(onB.commonTemplate(with: a, in: library) == template.identifier)+ #expect(try VPhoneDiskRebaseBase.resolve("target", in: library).commonTemplate(with: a, in: library) == nil)+ #expect(onB.commonTemplate(with: machines.target, in: library) == nil)++ let report = try VPhoneDiskRebase.plan(a, onto: b, options: options(), progress: { _, _, _ in })+ #expect(report.sharedBytes >= Int64(1 << 20))+ #expect(report.newlySharedBytes == 0)++ // Another build of the template, under the same identifier: b no+ // longer shares a's blocks, whatever the identifier says.+ var source = try #require(VPhoneMachineTemplates.readSource(inBundle: b.url))+ source.build = UUID().uuidString+ try VPhoneMachineTemplates.writeSource(source, inBundle: b.url)+ #expect(onB.commonTemplate(with: a, in: library) == nil)+ }+// MARK: - Ranges and export@Test func `the union of data ranges is widened to units and merged`() {