Skip to content

Run a virtual iPhone on an Apple Silicon Mac.

Swift

Reboot a stalled first boot once in the setup boot's settle step

Jacky1ec349bb3412
diff --git a/Research/Host/machine_templates.md b/Research/Host/machine_templates.md
index 4474cbd4..d09307d3 100644
--- a/Research/Host/machine_templates.md
+++ b/Research/Host/machine_templates.md
@@ -525,7 +525,7 @@ setup boot is the guest's first boot). It talks to vphoned through
| connect | `ping` every second | 300 s | vphoned answers |
| 0 snapshot | `apfs.snapshot.delete {force}` | 120 s | no `orig-fs.disabled.rn-*` in `remaining`/`after` (none to begin with is fine) |
| a skip | `setup.skip {force}`, any refusal retried | 600 s | `setup_done` |
-| b settle | `setup.settle {timeout_s ≤ 30}` repeated, a progress line after each | 600 s overall | `settled` (system apps expanded and data migration finished) |
+| b settle | `setup.settle {timeout_s ≤ 30}` repeated, a progress line after each; after 180 s without progress, a capture and one reboot | 600 s overall, again from that reboot | `settled` (system apps expanded and data migration finished) |
| c apps | `apps.remove_system {bundle_ids, force}` | 300 s | every app `removed`, `absent` or `unregistered_stale` |
| d/e profile | `services.profile.apply {profile, groups, force}`, `services.profile` | 180 s | no `failed`; the record holds `signin_followup` and the extra groups; followupd and appleidsetupd disabled |
| f reboot | `processes.list` (launchd's `start_time`), `system.reboot {force}`, then `ping` and `processes.list` (10 s timeout) every 3 s | 300 s | vphoned answers with another launchd start time |
@@ -582,6 +582,39 @@ gets through, and its refusals print `waiting for the guest's first-boot data
migration before SpringBoard can restart (N s)` at most every 30 s instead of
one error line each.
+A first boot can also stall rather than run slowly. On a Launchpad template
+build of iPhone17,3 27.0.1 (24A446) on 2026-10-15, the app expansion stopped
+after four apps (216 → 220 registered). `staged_system_apps` then kept its 46
+entries, installd used no CPU and DataMigrator kept running, all for the full
+600 s. `vm template setup` run again on the same machine, which is a second
+boot, settled in 45 s. The cause is not known. It looks like the iOS 27
+userland on the 26.4 kernel losing a dispatch wake-up
+(`Research/Guest/locationd_startup_deadlock.md`), but nothing from that boot
+survived to show it. The settle step therefore watches the `signals` of each
+`setup.settle` answer (`SettleStallWatch`). Progress is any change in the
+staged count, the app count, installd's pid or the migration's state, or
+installd using 0.2 s of CPU. When data migration has not finished and
+nothing has progressed for `settleStall` (180 s; the quietest healthy
+stretch, `SpringBoard.migrator`, took 26–67 s), the step does three things:
+
+1. It captures the stall into the build log, best effort, within 120 s:
+ installd's and DataMigrator's `processes.list` rows, installd's launchd
+ record (`state`, `active count`, `runs`, `pid`, `immediate reason`,
+ `last exit code`), and a 5 s whole-system spindump. vphoned cannot run a
+ program, so the spindump runs as a launchd job
+ (`com.vphone.settle-stall.spindump`, plist and report in
+ `/private/var/tmp`). Its `installd` and `com.apple.datamigrator` sections
+ are logged without their binary images, at most 250 lines each. The job
+ and the report are removed afterwards, so a template frozen from the boot
+ does not carry them.
+2. It reboots the guest the way step f does.
+3. It waits `settle` again, from the reboot.
+
+A stall after that reboot is not rebooted again: the step fails at the new
+deadline with "… within 600 s of the reboot that was to resume it". The
+outcome records `settleReboots` and a warning, which `vm template setup`
+prints in its summary.
+
The migration's length varies between runs, and so does the boot. Measured
from the end of migration, the old and the new flow both reach remove-apps
about 15 s later. The reboot step took either about 10 s or about 33 s (4 of 6
diff --git a/VPhoneKit/VPhoneCoreKit/Bundle/VPhoneTemplateSetupBoot.swift b/VPhoneKit/VPhoneCoreKit/Bundle/VPhoneTemplateSetupBoot.swift
index 9ca57f9e..841586b5 100644
--- a/VPhoneKit/VPhoneCoreKit/Bundle/VPhoneTemplateSetupBoot.swift
+++ b/VPhoneKit/VPhoneCoreKit/Bundle/VPhoneTemplateSetupBoot.swift
@@ -105,6 +105,16 @@ public struct VPhoneTemplateSetupTimeouts: Equatable, Sendable {
/// seconds each. The first boot's data migration ended 60–180 s after
/// the VM started on a fast Mac (2026-10-09); a slow host takes longer.
public var settle: TimeInterval = 600
+ /// No progress for this long while data migration has not finished is a
+ /// stall: the settle step captures what installd and DataMigrator are
+ /// doing, reboots the guest once and waits `settle` again. On a 27.0.1
+ /// setup boot (2026-10-15) the app expansion stopped with 46 apps staged
+ /// and installd idle for all 600 s; the next boot finished it in 45 s.
+ /// A healthy boot's longest quiet stretch, `SpringBoard.migrator` after
+ /// the expansion, took 26–67 s.
+ public var settleStall: TimeInterval = 180
+ /// For capturing a stall before the reboot, spindump included.
+ public var stallCapture: TimeInterval = 120
/// How often a waiting step prints its progress.
public var progressInterval: TimeInterval = 30
public var removeApps: TimeInterval = 300
@@ -241,6 +251,8 @@ public struct VPhoneTemplateSetupOutcome: Equatable, Sendable {
public var snapshotsGone = false
public var setupSkipped = false
public var settled = false
+ /// Reboots the settle step made to resume a stalled first boot (0 or 1).
+ public var settleReboots = 0
public var apps: [VPhoneTemplateSetupAppResult] = []
public var serviceProfile = "none"
public var serviceGroups: [String] = []
@@ -493,15 +505,19 @@ public final class VPhoneTemplateSetupBoot {
/// it is called again until the overall deadline, printing what it waits
/// for in between. It settles once the system apps are expanded and data
/// migration has ended, which is also when SpringBoard reads the keys
- /// `setup.skip` wrote.
+ /// `setup.skip` wrote. A first boot that stops making progress
+ /// (`SettleStallWatch`) is captured and rebooted once; data migration
+ /// carries on in the next boot, and the deadline starts again.
private func settle() throws {
let start = clock.now()
- let end = deadline(timeouts.settle)
+ var end = deadline(timeouts.settle)
+ var watch = SettleStallWatch(window: timeouts.settleStall)
var reasons: [String] = []
while true {
let remaining = end.timeIntervalSince(clock.now())
guard remaining > 0 else {
throw StepError("first-boot work did not settle within \(Int(timeouts.settle)) s"
+ + (outcome.settleReboots > 0 ? " of the reboot that was to resume it" : "")
+ (reasons.isEmpty ? "" : ": \(reasons.joined(separator: "; "))"))
}
let wait = Int(min(Double(Self.settleCallLimit), max(10, remaining.rounded(.up))))
@@ -514,7 +530,275 @@ public final class VPhoneTemplateSetupBoot {
}
reasons = strings(result["reasons"])
log(Self.settleProgress(reasons, elapsed: elapsed))
+
+ let signals = SettleSignals(result["signals"] as? [String: Any])
+ guard let quiet = watch.observe(signals, at: clock.now()), outcome.settleReboots == 0 else { continue }
+ let summary = signals?.summary ?? "no signals"
+ log(" first-boot work made no progress for \(Int(quiet)) s (\(summary)); data migration has not finished")
+ captureStall()
+ log(" rebooting the guest once to resume first-boot work")
+ try restartGuest(deadline: deadline(timeouts.reboot))
+ outcome.settleReboots += 1
+ outcome.warnings.append("first-boot work stalled at \(elapsed) s (\(summary)); the guest was rebooted once to resume it")
+ end = deadline(timeouts.settle)
+ watch = SettleStallWatch(window: timeouts.settleStall)
+ }
+ }
+
+ /// What one `setup.settle` answer's `signals` say about progress.
+ struct SettleSignals: Equatable {
+ var stagedSystemApps: Int?
+ /// The app counts of the answer's last polls, oldest first.
+ var appCounts: [Int?]
+ var installdPID: Int?
+ var installdCPUSeconds: Double?
+ /// installd's CPU between the answer's last two polls.
+ var installdCPUDelta: Double?
+ var dataMigrationDone: Bool?
+
+ init(
+ stagedSystemApps: Int? = nil, appCounts: [Int?] = [], installdPID: Int? = nil,
+ installdCPUSeconds: Double? = nil, installdCPUDelta: Double? = nil, dataMigrationDone: Bool? = nil,
+ ) {
+ self.stagedSystemApps = stagedSystemApps
+ self.appCounts = appCounts
+ self.installdPID = installdPID
+ self.installdCPUSeconds = installdCPUSeconds
+ self.installdCPUDelta = installdCPUDelta
+ self.dataMigrationDone = dataMigrationDone
+ }
+
+ /// Nil without `signals` (a vphoned that does not send them).
+ init?(_ signals: [String: Any]?) {
+ guard let signals else { return nil }
+ func int(_ key: String) -> Int? {
+ VPhoneTemplateSetupBoot.number(signals[key]).flatMap { Int(exactly: $0.rounded()) }
+ }
+ self.init(
+ stagedSystemApps: int("staged_system_apps"),
+ appCounts: (signals["app_counts"] as? [Any] ?? []).map { VPhoneTemplateSetupBoot.number($0).map { Int($0) } },
+ installdPID: int("installd_pid"),
+ installdCPUSeconds: VPhoneTemplateSetupBoot.number(signals["installd_cpu_seconds"]),
+ installdCPUDelta: VPhoneTemplateSetupBoot.number(signals["installd_cpu_delta"]),
+ dataMigrationDone: signals["data_migration_done"] as? Bool,
+ )
+ }
+
+ var appCount: Int? {
+ appCounts.last ?? nil
+ }
+
+ /// `46 apps staged, 220 apps, installd 312 idle`.
+ var summary: String {
+ var parts = ["\(stagedSystemApps.map(String.init) ?? "?") apps staged"]
+ if let appCount {
+ parts.append("\(appCount) apps")
+ }
+ parts.append(installdPID.map { "installd \($0)" } ?? "installd not running")
+ return parts.joined(separator: ", ")
+ }
+ }
+
+ /// Tells a slow first boot from a stalled one across `setup.settle`
+ /// answers. Progress is any change in the staged apps, the app count,
+ /// installd's pid or the migration's state, or installd using CPU
+ /// (vphoned's idle threshold, 0.2 s between polls). Stalled: no progress
+ /// for `window` while data migration has not finished; an unknown
+ /// migration state never stalls.
+ struct SettleStallWatch {
+ static let installdBusyCPUSeconds = 0.2
+
+ let window: TimeInterval
+ private var last: SettleSignals?
+ private var lastProgress: Date?
+
+ init(window: TimeInterval) {
+ self.window = window
+ }
+
+ /// The seconds without progress once that reaches `window`, else nil.
+ mutating func observe(_ signals: SettleSignals?, at now: Date) -> TimeInterval? {
+ guard let signals else { return nil }
+ if let last, lastProgress != nil, !Self.progressed(from: last, to: signals) {
+ // No progress: keep the time it was last made.
+ } else {
+ lastProgress = now
+ }
+ last = signals
+ guard signals.dataMigrationDone == false, let lastProgress else { return nil }
+ let quiet = now.timeIntervalSince(lastProgress)
+ return quiet >= window ? quiet : nil
+ }
+
+ static func progressed(from earlier: SettleSignals, to later: SettleSignals) -> Bool {
+ if earlier.stagedSystemApps != later.stagedSystemApps || earlier.installdPID != later.installdPID
+ || earlier.dataMigrationDone != later.dataMigrationDone || earlier.appCount != later.appCount
+ {
+ return true
+ }
+ if Set(later.appCounts.compactMap(\.self)).count > 1 {
+ return true
+ }
+ if (later.installdCPUDelta ?? 0) >= installdBusyCPUSeconds {
+ return true
+ }
+ if let before = earlier.installdCPUSeconds, let after = later.installdCPUSeconds,
+ after - before >= installdBusyCPUSeconds
+ {
+ return true
+ }
+ return false
+ }
+ }
+
+ // MARK: Stall capture
+
+ /// A launchd job that spindumps every process for `stallSpindumpSeconds`.
+ static let stallSpindumpLabel = "com.vphone.settle-stall.spindump"
+ static let stallSpindumpJobPath = "/private/var/tmp/com.vphone.settle-stall.spindump.plist"
+ static let stallSpindumpReportPath = "/private/var/tmp/com.vphone.settle-stall.spindump.txt"
+ static let stallSpindumpSeconds = 5
+ /// At most this many lines of each process from the report.
+ static let stallSpindumpLinesPerProcess = 250
+
+ /// The first-boot processes a stall capture reports: installd, which
+ /// expands the staged system apps, and DataMigrator's XPC service
+ /// (`com.apple.datamigrator`; the kernel cuts its name to `com.apple.datami`).
+ static func isStallProcess(name: String, executable: String) -> Bool {
+ executable.hasSuffix("/installd") || name == "installd"
+ || executable.hasSuffix("/com.apple.datamigrator") || name.hasPrefix("com.apple.datami")
+ }
+
+ /// Best effort, within `timeouts.stallCapture`: what installd and
+ /// DataMigrator are doing, into the log, before the reboot ends them.
+ /// Each part that fails says so and the capture goes on.
+ private func captureStall() {
+ let end = deadline(timeouts.stallCapture)
+ log(" capturing the stall:")
+ do {
+ let rows = try call("processes.list", [:], deadline: end)["processes"] as? [[String: Any]] ?? []
+ let picked = rows.filter {
+ Self.isStallProcess(name: $0["name"] as? String ?? "", executable: $0["executable"] as? String ?? "")
+ }
+ if picked.isEmpty {
+ log(" neither installd nor DataMigrator is running")
+ }
+ for row in picked {
+ let cpu = Self.number(row["cpu_seconds"]).map { String(format: "%.2f s CPU", $0) } ?? "CPU ?"
+ let started = Self.number(row["start_time"])
+ .map { "started \(Int(clock.now().timeIntervalSince1970 - $0)) s ago" } ?? "start ?"
+ log(" \(row["executable"] as? String ?? row["name"] as? String ?? "?") pid \(row["pid"] ?? "?"): \(cpu), \(started)")
+ }
+ } catch {
+ log(" processes.list: \(error)")
+ }
+ do {
+ let text = try call("services.print", ["label": "com.apple.mobile.installd"], deadline: end)["description"] as? String ?? ""
+ log(" installd in launchd: \(Self.launchdSummary(text))")
+ } catch {
+ log(" services.print com.apple.mobile.installd: \(error)")
+ }
+ do {
+ let report = try stallSpindump(deadline: end)
+ let excerpt = Self.spindumpExcerpt(report, maximumLines: Self.stallSpindumpLinesPerProcess) {
+ Self.isStallProcess(name: $0, executable: "")
+ }
+ log(excerpt.isEmpty
+ ? " spindump (\(report.utf8.count) bytes) has neither installd nor DataMigrator"
+ : " spindump, \(Self.stallSpindumpSeconds) s:")
+ for line in excerpt {
+ log(" | \(line)")
+ }
+ } catch {
+ log(" spindump: \(error)")
+ }
+ }
+
+ /// The few lines of a `launchctl print` that say whether a job is stuck:
+ /// `state = running; active count = 2; runs = 1; …`.
+ static func launchdSummary(_ description: String) -> String {
+ let keys = ["state", "active count", "runs", "pid", "immediate reason", "last exit code"]
+ var found: [String: String] = [:]
+ for line in description.split(separator: "\n") {
+ // The job's own keys are one tab deep; nested blocks are deeper.
+ guard line.hasPrefix("\t"), !line.hasPrefix("\t\t") else { continue }
+ let parts = line.dropFirst().split(separator: "=", maxSplits: 1).map { $0.trimmingCharacters(in: .whitespaces) }
+ guard parts.count == 2, keys.contains(parts[0]), found[parts[0]] == nil else { continue }
+ found[parts[0]] = parts[1]
+ }
+ let summary = keys.compactMap { key in found[key].map { "\(key) = \($0)" } }
+ return summary.isEmpty ? "no record" : summary.joined(separator: "; ")
+ }
+
+ /// The sections of a spindump report whose `Process:` line names a
+ /// process `matches` accepts, each without its `Binary Images` list and
+ /// cut to `maximumLines`.
+ static func spindumpExcerpt(_ report: String, maximumLines: Int, matches: (String) -> Bool) -> [String] {
+ var lines: [String] = []
+ var taken = 0
+ var inSection = false
+ var inImages = false
+ for raw in report.split(separator: "\n", omittingEmptySubsequences: false) {
+ let line = String(raw)
+ if line.hasPrefix("Process:") {
+ // `Process: installd [312]`
+ let name = line.dropFirst("Process:".count).trimmingCharacters(in: .whitespaces)
+ .components(separatedBy: " [").first ?? ""
+ inSection = matches(name)
+ inImages = false
+ taken = 0
+ } else if line.hasPrefix("Binary Images:") {
+ inImages = true
+ }
+ guard inSection, !inImages else { continue }
+ if taken == maximumLines {
+ lines.append("… (cut at \(maximumLines) lines)")
+ }
+ if taken < maximumLines {
+ lines.append(line)
+ }
+ taken += 1
}
+ return lines
+ }
+
+ /// The guest spindumps itself: vphoned cannot run a program, so a launchd
+ /// job does, and the report is read back. The job and the report are
+ /// removed afterwards, whatever happened.
+ private func stallSpindump(deadline end: Date) throws -> String {
+ let job: [String: Any] = [
+ "Label": Self.stallSpindumpLabel,
+ "ProgramArguments": [
+ "/usr/sbin/spindump", "-notarget", "\(Self.stallSpindumpSeconds)", "10",
+ "-file", Self.stallSpindumpReportPath,
+ ],
+ "RunAtLoad": true,
+ ]
+ let data = try PropertyListSerialization.data(fromPropertyList: job, format: .xml, options: 0)
+ defer {
+ _ = try? machine.call("services.unload", params: ["paths": [Self.stallSpindumpJobPath], "force": true], timeout: 10)
+ _ = try? machine.call("files.remove", params: ["path": Self.stallSpindumpJobPath], timeout: 10)
+ _ = try? machine.call("files.remove", params: ["path": Self.stallSpindumpReportPath], timeout: 10)
+ }
+ _ = try? machine.call("files.remove", params: ["path": Self.stallSpindumpReportPath], timeout: 10)
+ _ = try call("files.write", ["path": Self.stallSpindumpJobPath, "content": String(decoding: data, as: UTF8.self)], deadline: end)
+ _ = try call("services.load", ["paths": [Self.stallSpindumpJobPath]], deadline: end)
+ clock.sleep(TimeInterval(Self.stallSpindumpSeconds))
+ while true {
+ let status = try call("services.status", ["label": Self.stallSpindumpLabel], deadline: end)
+ if status["running"] as? Bool != true {
+ break
+ }
+ guard clock.now().addingTimeInterval(timeouts.retryInterval) < end else {
+ throw StepError("did not finish within \(Int(timeouts.stallCapture)) s")
+ }
+ clock.sleep(timeouts.retryInterval)
+ }
+ let read = try call("files.read", ["path": Self.stallSpindumpReportPath, "limit": 24 << 20], deadline: end, timeout: 60)
+ guard let content = read["content"] as? String else {
+ throw StepError("no report at \(Self.stallSpindumpReportPath)")
+ }
+ return content
}
static let migrationReason = "data migration has not finished"
@@ -643,7 +927,10 @@ public final class VPhoneTemplateSetupBoot {
/// f. Reboots the guest and waits for vphoned of the next boot: launchd's
/// start time (pid 1) tells the boots apart.
private func reboot() throws {
- let end = deadline(timeouts.reboot)
+ try restartGuest(deadline: deadline(timeouts.reboot))
+ }
+
+ private func restartGuest(deadline end: Date) throws {
let before = try bootMarker(deadline: end)
do {
_ = try machine.call("system.reboot", params: ["force": true], timeout: 30)
@@ -679,7 +966,7 @@ public final class VPhoneTemplateSetupBoot {
}
/// A number from JSON (NSNumber) or from a dictionary built in Swift.
- private static func number(_ value: Any?) -> Double? {
+ static func number(_ value: Any?) -> Double? {
switch value {
case let value as Int: Double(value)
case let value as Double: value
diff --git a/VPhoneKit/VPhoneCoreKitTests/Bundle/TemplateSetupBootTests.swift b/VPhoneKit/VPhoneCoreKitTests/Bundle/TemplateSetupBootTests.swift
index 63bf0df6..7e601ec7 100644
--- a/VPhoneKit/VPhoneCoreKitTests/Bundle/TemplateSetupBootTests.swift
+++ b/VPhoneKit/VPhoneCoreKitTests/Bundle/TemplateSetupBootTests.swift
@@ -73,6 +73,40 @@ private final class FakeGuest: VPhoneTemplateSetupMachine {
]
var undeletableReports: Set<String> = []
var crashListRefused = false
+ /// The first boot stalls: `setup.settle` answers unsettled with
+ /// `stallSignals` until a reboot (or for good, `stallSurvivesReboot`).
+ var stalledBoot = false
+ var stallSurvivesReboot = false
+ /// Each stalled answer has one app fewer staged: slow, not stalled.
+ var stallProgresses = false
+ var stallSignals: [String: Any] = [
+ "staged_system_apps": 46, "app_count": 220, "app_counts": [220, 220, 220],
+ "installd_pid": 312, "installd_cpu_seconds": 1.25, "installd_cpu_delta": 0.0,
+ "setup_pending": true, "data_migration_done": false,
+ ]
+ /// The guest refuses the spindump job (an older vphoned, a guest without
+ /// spindump).
+ var spindumpRefused = false
+ var spindumpJobRunning = false
+ var spindumpReport = """
+ Date/Time: 2026-10-15 16:40:01 +0000
+ Process: SpringBoard [40]
+ Thread 0x1a 1000 samples
+ 1000 start + 6000 (dyld + 0)
+ Binary Images:
+ 0x100000000 SpringBoard
+
+ Process: installd [312]
+ Path: /usr/libexec/installd
+ Thread 0x2b DispatchQueue "com.apple.installd.staged" 1000 samples
+ 1000 __workq_kernreturn + 8 (libsystem_kernel.dylib + 1234)
+ Binary Images:
+ 0x100000000 installd
+
+ Process: com.apple.datamigrator [205]
+ Thread 0x3c 1000 samples
+ 1000 mach_msg2_trap + 8 (libsystem_kernel.dylib + 4567)
+ """
init(clock: FakeClock) {
self.clock = clock
@@ -145,6 +179,16 @@ private final class FakeGuest: VPhoneTemplateSetupMachine {
case "setup.settle":
let wait = params["timeout_s"] as? Int ?? 90
#expect(wait <= 30)
+ if stalledBoot {
+ clock.sleep(TimeInterval(wait))
+ if stallProgresses, let staged = stallSignals["staged_system_apps"] as? Int {
+ stallSignals["staged_system_apps"] = max(staged - 1, 1)
+ }
+ return [
+ "settled": false, "elapsed_s": wait, "signals": stallSignals,
+ "reasons": ["staged_system_apps has 46 entries", "data migration has not finished"],
+ ]
+ }
if unsettledAnswers > 0 {
unsettledAnswers -= 1
clock.sleep(TimeInterval(wait))
@@ -191,9 +235,56 @@ private final class FakeGuest: VPhoneTemplateSetupMachine {
let record: Any = profile == "none" ? NSNull() : ["profile": profile, "groups": groups, "labels": Array(disabled)]
return ["profile": profile, "record": record, "disabled": Array(disabled), "running": running]
case "processes.list":
- return ["processes": [["pid": 1, "start_time": bootTime], ["pid": 53, "start_time": bootTime + 4]], "count": 2]
+ return ["processes": [
+ ["pid": 1, "start_time": bootTime],
+ ["pid": 53, "start_time": bootTime + 4],
+ ["pid": 312, "name": "installd", "executable": "/usr/libexec/installd", "cpu_seconds": 1.25, "start_time": bootTime + 5],
+ ["pid": 205, "name": "com.apple.datami", "cpu_seconds": 0.5, "start_time": bootTime + 6,
+ "executable": "/System/Library/PrivateFrameworks/DataMigration.framework/XPCServices/com.apple.datamigrator.xpc/com.apple.datamigrator"],
+ ], "count": 4]
+ case "services.print":
+ return ["label": params["label"] ?? "", "domain": "user", "description": """
+ user/501/com.apple.mobile.installd = {
+ \tactive count = 2
+ \tstate = running
+ \tendpoints = {
+ \t\t"com.apple.mobile.installd" = {
+ \t\t\tactive = 1
+ \t\t}
+ \t}
+ \truns = 1
+ \tpid = 312
+ \timmediate reason = ipc (mach)
+ \tlast exit code = (never exited)
+ }
+ """]
+ case "files.write":
+ #expect(params["path"] as? String == VPhoneTemplateSetupBoot.stallSpindumpJobPath)
+ let content = params["content"] as? String ?? ""
+ #expect(content.contains("/usr/sbin/spindump") && content.contains(VPhoneTemplateSetupBoot.stallSpindumpReportPath))
+ return ["path": params["path"] ?? "", "bytes": content.utf8.count]
+ case "services.load":
+ if spindumpRefused {
+ throw refused("launchctl bootstrap failed: 5", ["code": "failed"])
+ }
+ spindumpJobRunning = true
+ return ["loaded": params["paths"] ?? []]
+ case "services.status":
+ // Running for one more poll, then done.
+ let running = spindumpJobRunning
+ spindumpJobRunning = false
+ return ["label": params["label"] ?? "", "running": running, "pid": running ? 900 : 0]
+ case "files.read":
+ #expect(params["path"] as? String == VPhoneTemplateSetupBoot.stallSpindumpReportPath)
+ return ["path": params["path"] ?? "", "content": spindumpReport, "encoding": "utf8", "truncated": false]
+ case "services.unload":
+ #expect(params["force"] as? Bool == true)
+ return ["unloaded": params["paths"] ?? []]
case "system.reboot":
bootTime += 100
+ if !stallSurvivesReboot {
+ stalledBoot = false
+ }
downCalls = 3
dyingVphoned = dyingVphonedTakesAList
if !servicesKeepRunning {
@@ -451,6 +542,185 @@ struct TemplateSetupBootTests {
== " waiting for first-boot work (70 s): app count changed: 250 → 258")
}
+ // MARK: Settle stalls
+
+ private func stallRun(_ guest: FakeGuest) -> (Result<VPhoneTemplateSetupOutcome, VPhoneTemplateSetupFailure>, [String]) {
+ var lines: [String] = []
+ let boot = VPhoneTemplateSetupBoot(
+ machine: guest,
+ plan: VPhoneTemplateSetupPlan(slimming: VPhoneTemplateSlimmingRequest.defaultSlimming, requiresEveryApp: true),
+ clock: guest.clock,
+ log: { lines.append($0) },
+ )
+ do {
+ return try (.success(boot.run()), lines)
+ } catch let failure as VPhoneTemplateSetupFailure {
+ return (.failure(failure), lines)
+ } catch {
+ Issue.record("unexpected \(error)")
+ return (.failure(VPhoneTemplateSetupFailure(step: .boot, reason: "\(error)", outcome: .init())), lines)
+ }
+ }
+
+ /// The 27.0.1 setup boot of 2026-10-15: 46 apps stayed staged and
+ /// installd idle for 600 s; a reboot finished the migration in 45 s.
+ @Test func `a stalled first boot is captured, rebooted once and settles`() throws {
+ let guest = FakeGuest(clock: FakeClock())
+ guest.migrationPending = true
+ guest.stalledBoot = true
+ let (result, lines) = stallRun(guest)
+ let outcome = try result.get()
+
+ #expect(outcome.isComplete)
+ #expect(outcome.settleReboots == 1)
+ #expect(outcome.warnings.contains { $0.contains("first-boot work stalled") && $0.contains("46 apps staged") })
+ // Capture, reboot, then settle again before anything is slimmed.
+ #expect(inOrder(guest.calls, [
+ "setup.settle", "processes.list", "services.print", "files.write", "services.load",
+ "services.status", "files.read", "services.unload", "files.remove", "files.remove",
+ "processes.list", "system.reboot", "processes.list", "setup.settle", "apps.remove_system",
+ ]))
+ #expect(guest.calls.count(where: { $0 == "system.reboot" }) == 2)
+ // Detected after the stall window, not at the 600 s deadline.
+ let settle = try #require(outcome.durations[.settle])
+ #expect(settle > 180 && settle < 400)
+
+ #expect(lines.contains { $0.contains("made no progress for") && $0.contains("46 apps staged, 220 apps, installd 312") })
+ #expect(lines.contains { $0.hasPrefix(" /usr/libexec/installd pid 312: 1.25 s CPU, started ") })
+ #expect(lines.contains { $0.contains("/com.apple.datamigrator pid 205: 0.50 s CPU, started ") })
+ #expect(lines.contains(" installd in launchd: state = running; active count = 2; runs = 1; pid = 312; immediate reason = ipc (mach); last exit code = (never exited)"))
+ #expect(lines.contains(" | Process: installd [312]"))
+ #expect(lines.contains { $0.contains("__workq_kernreturn") })
+ #expect(lines.contains(" | Process: com.apple.datamigrator [205]"))
+ #expect(!lines.contains { $0.contains("SpringBoard [40]") || $0.contains("0x100000000") })
+ #expect(lines.contains(" rebooting the guest once to resume first-boot work"))
+ #expect(lines.contains { $0.hasPrefix(" settled after ") })
+ }
+
+ @Test func `a stall that outlasts the reboot fails at the new deadline after one reboot`() throws {
+ let guest = FakeGuest(clock: FakeClock())
+ guest.stalledBoot = true
+ guest.stallSurvivesReboot = true
+ let (result, _) = stallRun(guest)
+ let failure = try #require(result.failed)
+ #expect(failure.step == .settle)
+ #expect(failure.reason.contains("of the reboot that was to resume it"))
+ #expect(failure.outcome.settleReboots == 1)
+ #expect(guest.calls.count(where: { $0 == "system.reboot" }) == 1)
+ #expect(guest.killed)
+ }
+
+ @Test func `a capture that fails does not hold the reboot back`() throws {
+ let guest = FakeGuest(clock: FakeClock())
+ guest.stalledBoot = true
+ guest.spindumpRefused = true
+ let (result, lines) = stallRun(guest)
+ let outcome = try result.get()
+ #expect(outcome.settleReboots == 1)
+ #expect(lines.contains { $0.hasPrefix(" spindump: services.load: launchctl bootstrap failed") })
+ // The job is removed even though it never loaded.
+ #expect(inOrder(guest.calls, ["services.load", "services.unload", "files.remove", "files.remove", "system.reboot"]))
+ }
+
+ @Test func `a first boot that keeps progressing is never rebooted`() throws {
+ let guest = FakeGuest(clock: FakeClock())
+ // Slow to the point of failing at the deadline, but every answer
+ // shows one more app expanded.
+ guest.stalledBoot = true
+ guest.stallSurvivesReboot = true
+ guest.stallProgresses = true
+ let (result, _) = stallRun(guest)
+ let failure = try #require(result.failed)
+ #expect(failure.step == .settle)
+ #expect(failure.outcome.settleReboots == 0)
+ #expect(!guest.calls.contains("system.reboot"))
+ #expect(!failure.reason.contains("of the reboot"))
+ }
+
+ @Test func `the stall watch counts any change or installd work as progress`() {
+ typealias Signals = VPhoneTemplateSetupBoot.SettleSignals
+ let start = Date(timeIntervalSince1970: 0)
+ let stuck = Signals(stagedSystemApps: 46, appCounts: [220, 220], installdPID: 312, installdCPUSeconds: 1, installdCPUDelta: 0, dataMigrationDone: false)
+ var watch = VPhoneTemplateSetupBoot.SettleStallWatch(window: 180)
+ #expect(watch.observe(stuck, at: start) == nil)
+ #expect(watch.observe(stuck, at: start + 179) == nil)
+ #expect(watch.observe(stuck, at: start + 180) == 180)
+
+ func progressed(_ change: (inout Signals) -> Void) -> Bool {
+ var later = stuck
+ change(&later)
+ return VPhoneTemplateSetupBoot.SettleStallWatch.progressed(from: stuck, to: later)
+ }
+ #expect(!progressed { _ in })
+ #expect(progressed { $0.stagedSystemApps = 45 })
+ #expect(progressed { $0.appCounts = [220, 221] })
+ #expect(progressed { $0.appCounts = [221, 221] })
+ #expect(progressed { $0.installdPID = 400 })
+ #expect(progressed { $0.installdCPUSeconds = 1.3 })
+ #expect(!progressed { $0.installdCPUSeconds = 1.1 })
+ #expect(progressed { $0.installdCPUDelta = 0.25 })
+ #expect(progressed { $0.dataMigrationDone = true })
+
+ // Progress restarts the window.
+ var moving = VPhoneTemplateSetupBoot.SettleStallWatch(window: 180)
+ _ = moving.observe(stuck, at: start)
+ var next = stuck
+ next.stagedSystemApps = 10
+ #expect(moving.observe(next, at: start + 170) == nil)
+ #expect(moving.observe(next, at: start + 340) == nil)
+ #expect(moving.observe(next, at: start + 350) == 180)
+
+ // An unknown or finished migration never stalls; no signals say nothing.
+ var unknown = VPhoneTemplateSetupBoot.SettleStallWatch(window: 180)
+ var quiet = stuck
+ quiet.dataMigrationDone = nil
+ _ = unknown.observe(quiet, at: start)
+ #expect(unknown.observe(quiet, at: start + 1000) == nil)
+ #expect(unknown.observe(nil, at: start + 2000) == nil)
+ }
+
+ @Test func `settle signals are read from vphoned's answer`() throws {
+ let signals = try #require(VPhoneTemplateSetupBoot.SettleSignals([
+ "staged_system_apps": NSNumber(value: 46), "app_count": NSNumber(value: 220),
+ "app_counts": [NSNumber(value: 216), NSNull(), NSNumber(value: 220)],
+ "installd_pid": NSNumber(value: 312), "installd_cpu_seconds": NSNumber(value: 1.5),
+ "installd_cpu_delta": NSNull(), "data_migration_done": NSNumber(value: false),
+ ]))
+ #expect(signals == VPhoneTemplateSetupBoot.SettleSignals(
+ stagedSystemApps: 46, appCounts: [216, nil, 220], installdPID: 312,
+ installdCPUSeconds: 1.5, installdCPUDelta: nil, dataMigrationDone: false,
+ ))
+ #expect(signals.summary == "46 apps staged, 220 apps, installd 312")
+ #expect(VPhoneTemplateSetupBoot.SettleSignals(nil) == nil)
+ #expect(VPhoneTemplateSetupBoot.SettleSignals([:])?.summary == "? apps staged, installd not running")
+ }
+
+ @Test func `a spindump excerpt keeps the named processes without their images, cut to a limit`() {
+ let report = """
+ Process: installd [312]
+ Thread 0x1
+ a
+ b
+ c
+ Binary Images:
+ 0x1 installd
+ Process: locationd [90]
+ Thread 0x9
+ """
+ let lines = VPhoneTemplateSetupBoot.spindumpExcerpt(report, maximumLines: 3) { $0 == "installd" }
+ #expect(lines == ["Process: installd [312]", " Thread 0x1", " a", "… (cut at 3 lines)"])
+ #expect(VPhoneTemplateSetupBoot.spindumpExcerpt(report, maximumLines: 10) { $0 == "nothing" }.isEmpty)
+ #expect(VPhoneTemplateSetupBoot.isStallProcess(name: "com.apple.datami", executable: ""))
+ #expect(VPhoneTemplateSetupBoot.isStallProcess(name: "com.apple.datamigrator", executable: ""))
+ #expect(!VPhoneTemplateSetupBoot.isStallProcess(name: "installcoordinationd", executable: "/usr/libexec/installcoordinationd"))
+ }
+
+ @Test func `launchd's record is cut to the lines that show a stuck job`() {
+ #expect(VPhoneTemplateSetupBoot.launchdSummary("") == "no record")
+ #expect(VPhoneTemplateSetupBoot.launchdSummary("x = {\n\tstate = waiting\n\t\tstate = active\n\truns = 3\n}")
+ == "state = waiting; runs = 3")
+ }
+
/// 2.9.0's vphoned restarts SpringBoard in every skip and is refused
/// until migration ends: 14 refusals (about 130 s) failed the step on the
/// MacBook Air. Now the step outlasts them and reports the wait every 30 s