Skip to content

Run a virtual iPhone on an Apple Silicon Mac.

Swift

Duo: wake the panel from black, not from its last frame

Jackybde6440c410b
diff --git a/Research/Guest/iphone_duo_bringup.md b/Research/Guest/iphone_duo_bringup.md
index 2a0d2424..5f274c2b 100644
--- a/Research/Guest/iphone_duo_bringup.md
+++ b/Research/Guest/iphone_duo_bringup.md
@@ -5090,3 +5090,18 @@ DeviceHub's slider as it animates, and taking every read reshaped the window
at each step of a sweep. The cover drawn in the window's open shape,
sideways, now lasts a fraction of the half second it did. A fold while the screen sleeps lights it, and the
panel's first 0.2 s are its last frame from before sleep.
+
+The fade to black stays for a fold that starts with the panel off or lit
+less than a second ago. A fold made while the screen sleeps (DeviceHub, a
+guest-side request) runs only as the screen comes on, since SpringBoard's
+pose provider commits on display frames; without the fade, the effect's
+carried-over inner content (its Home Screen, sideways on the cover) showed
+for about half a second after the wake. And coming on, the panel first shows
+its last frame from before it went off (the unlocked Home Screen on a plain
+wake), which nothing in the guest replaces: SpringBoard stops compositing
+before it turns the panel off, and neither `-[CAWindowServerDisplay
+presentSurface:withOptions:]` nor `acquireFrozenSurface` reached the shown
+surface (`willUnblank` never runs: the panel goes off by display state). So
+`display.orientation` reports `screen_off`, and the VM window covers the
+panel with black while it is off and for 0.3 s after it comes on. Both wakes
+now go from black to the lock screen (60 fps capture).
diff --git a/Research/vphoned_http_api.md b/Research/vphoned_http_api.md
index 7483bcfd..7a027121 100644
--- a/Research/vphoned_http_api.md
+++ b/Research/vphoned_http_api.md
@@ -497,7 +497,7 @@ request carries `"force": true`.
| Area | Methods |
| --- | --- |
| Device | `device.snapshot`, `device.info` (snapshot plus network, screen, rotation, brightness, volume, low power, Developer Mode, agent, `device_name` `{name, own_name}`: the pinned name and the guest's own ComputerName), `device.screen`, `device.network`, `device.displays` (a flat IOService probe of every display-ish object with its small properties — what the display stack bound; built for the foldable display work: `AppleParavirtGPU` `DisplayPortCount`, and each `AppleParavirtDisplay` width/height/`primary`), `device.ioreg {plane}`, `device.environment`, `device.basebin {archive?}` |
-| Display, audio | `display.pose` (read-only SpringBoard pose snapshot: orientation, device_orientation, hinge, tabletop; reports unavailable for an unsupported ABI), `display.modes` (read-only CoreAnimation display bounds, native size, current/available modes, and virtual-mode selector availability; availability does not prove a mode change is supported), `display.brightness {value?}`, `display.rotation {orientation?}`, `display.posture {hinge, orientation?, tabletop?}` (a foldable guest's pose through SpringBoard's test-automation override, see below; capability `device_pose`), `display.orientation` (`{degrees, source, fold_angle?}`: SpringBoard's interface orientation without a screen capture, for polling; `degrees` is null while SpringBoard has no answer; on a foldable `fold_angle` is the virtual hinge's settled angle, which every fold leaves it at, so a host can follow folds it did not ask for; capability `display_orientation`), `display.rotation_lock {locked}`, `display.auto_lock` (`{auto_lock_seconds, never, lock_screen_minimum_seconds}`), `display.auto_lock.set {seconds}` (writes the mobile UserSettings input through cfprefsd, posts profiled's effective-settings notification, and returns an immediate readback; the caller must verify the effective value), `screen.unlock {passcode?, timeout?}` (`{locked, screen_off, was_locked, was_screen_off}`: the display on and the Lock Screen passed, see below; capability `screen_unlock`), `audio.volume {value?, category?}`, `audio.state` |
+| Display, audio | `display.pose` (read-only SpringBoard pose snapshot: orientation, device_orientation, hinge, tabletop; reports unavailable for an unsupported ABI), `display.modes` (read-only CoreAnimation display bounds, native size, current/available modes, and virtual-mode selector availability; availability does not prove a mode change is supported), `display.brightness {value?}`, `display.rotation {orientation?}`, `display.posture {hinge, orientation?, tabletop?}` (a foldable guest's pose through SpringBoard's test-automation override, see below; capability `device_pose`), `display.orientation` (`{degrees, source, fold_angle?, screen_off?}`: SpringBoard's interface orientation without a screen capture, for polling; `degrees` is null while SpringBoard has no answer; on a foldable `fold_angle` is the virtual hinge's angle, so a host can follow folds it did not ask for, and `screen_off` whether SpringBoard has the screen off, so a host can keep the panel's frame from before off its window as it comes on; capability `display_orientation`), `display.rotation_lock {locked}`, `display.auto_lock` (`{auto_lock_seconds, never, lock_screen_minimum_seconds}`), `display.auto_lock.set {seconds}` (writes the mobile UserSettings input through cfprefsd, posts profiled's effective-settings notification, and returns an immediate readback; the caller must verify the effective value), `screen.unlock {passcode?, timeout?}` (`{locked, screen_off, was_locked, was_screen_off}`: the display on and the Lock Screen passed, see below; capability `screen_unlock`), `audio.volume {value?, category?}`, `audio.state` |
| Input | `input.touch`, `input.hid {page, usage, down?, long_press?}`, `input.button {name}`, `input.key {name}`, `input.type {text, delay_ms?}`, `input.paste {text}`, `input.tap`, `input.double_tap`, `input.long_press`, `input.swipe`, `input.drag {points}`, `input.touch_sequence {events}` — gesture coordinates are screen points; `input.pointer {x, y, seq?, buttons?}`, `input.scroll {phase, dx?, dy?, x?, y?}` (capabilities `pointer_scroll` and `pointer_buttons`, see below) |
| UI | `ui.tree` (alias `accessibility.tree`), `ui.element_at`, `ui.tap_element`, `ui.wait`, `ui.wait_gone`, `ui.ocr {languages?, min_confidence?}`, `ui.describe`, `screen.screenshot` |
| Processes | `processes.list {filter?}`, `processes.kill {pid, signal?}` **force**, `process.spawn {executable, arguments?, timeout_seconds?}`, `memory.jetsam`, `memory.pressure` (only the three kernel memory sysctls, for polling) |
diff --git a/VPhoneDaemon/Daemon/GuestAPI+Device.swift b/VPhoneDaemon/Daemon/GuestAPI+Device.swift
index 2bbbc438..883f3598 100644
--- a/VPhoneDaemon/Daemon/GuestAPI+Device.swift
+++ b/VPhoneDaemon/Daemon/GuestAPI+Device.swift
@@ -176,7 +176,12 @@ extension GuestAPI {
"degrees": interfaceRotationDegrees().map { $0 as Any } ?? NSNull(),
"source": "springboard",
]
- if let angle = settledFoldAngle() { reply["fold_angle"] = angle }
+ if let angle = settledFoldAngle() {
+ reply["fold_angle"] = angle
+ // The window covers the panel while it is off: coming on, it
+ // shows its last frame from before until the guest draws again.
+ reply["screen_off"] = GuestScreenUnlock.screenIsOff
+ }
return reply
}
diff --git a/VPhoneDaemon/Daemon/GuestScreenUnlock.swift b/VPhoneDaemon/Daemon/GuestScreenUnlock.swift
index c7acaab3..db8d9231 100644
--- a/VPhoneDaemon/Daemon/GuestScreenUnlock.swift
+++ b/VPhoneDaemon/Daemon/GuestScreenUnlock.swift
@@ -173,6 +173,9 @@ enum GuestScreenUnlock {
// MARK: - State
+ /// Whether SpringBoard has the screen off.
+ static var screenIsOff: Bool { state().screenOff }
+
private struct State {
let locked: Bool
let screenOff: Bool
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneVirtualMachineWindowController.swift b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneVirtualMachineWindowController.swift
index e8a83206..06b1738b 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneVirtualMachineWindowController.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneVirtualMachineWindowController.swift
@@ -287,6 +287,28 @@ class VPhoneVirtualMachineWindowController: NSObject, NSWindowDelegate {
private var minimumDuoPanelSize: CGSize?
private var orientationPollInFlight = false
private var lastPolledFoldAngle: Double?
+ private var screenRevealWork: DispatchWorkItem?
+
+ /// Covers a foldable guest's panel while its screen is off, and for a
+ /// moment after it comes on: its display first shows the frame it showed
+ /// before going off (after a fold made asleep, the other panel's picture,
+ /// sideways), until the guest draws again.
+ private func followGuestScreen(off: Bool) {
+ guard let container = displayContainer else { return }
+ if off {
+ screenRevealWork?.cancel()
+ screenRevealWork = nil
+ container.isScreenDark = true
+ return
+ }
+ guard container.isScreenDark, screenRevealWork == nil else { return }
+ let work = DispatchWorkItem { [weak self] in
+ self?.displayContainer?.isScreenDark = false
+ self?.screenRevealWork = nil
+ }
+ screenRevealWork = work
+ DispatchQueue.main.asyncAfter(deadline: .now() + 0.3, execute: work)
+ }
/// Asks vphoned for the interface orientation once a second (a foldable
/// guest four times a second). Guests without `display.orientation` stay
@@ -308,6 +330,9 @@ class VPhoneVirtualMachineWindowController: NSObject, NSWindowDelegate {
if foldAngle == lastPolledFoldAngle { control.followGuestFold(angle: foldAngle) }
lastPolledFoldAngle = foldAngle
}
+ if let screenOff = result["screen_off"] as? Bool {
+ followGuestScreen(off: screenOff)
+ }
guard !control.isChangingOrientation,
let degrees = (result["degrees"] as? NSNumber)?.intValue,
let orientation = VPhoneDisplayOrientation(degrees: degrees)
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneDisplayContainerView.swift b/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneDisplayContainerView.swift
index 06cb5aa9..fa6d2418 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneDisplayContainerView.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneDisplayContainerView.swift
@@ -28,6 +28,23 @@ final class VPhoneDisplayContainerView: NSView {
private static let turnDuration: CFTimeInterval = 0.35
+ /// Black over the panel, above the VM view's layer rather than in its
+ /// place, so clicks still reach the VM view (a tap wakes the guest).
+ private let darkLayer = CALayer()
+
+ /// Covers the panel while the guest's screen is off: coming on, the
+ /// guest's display shows its last frame from before until it draws again,
+ /// which after a fold made asleep was the other panel's picture.
+ var isScreenDark = false {
+ didSet {
+ guard isScreenDark != oldValue else { return }
+ CATransaction.begin()
+ CATransaction.setDisableActions(true)
+ darkLayer.opacity = isScreenDark ? 1 : 0
+ CATransaction.commit()
+ }
+ }
+
/// One animated turn: the panel's angle and the window frame move
/// together, so the window reshapes as the panel turns.
private struct Turn {
@@ -48,6 +65,10 @@ final class VPhoneDisplayContainerView: NSView {
viewportView.layer?.masksToBounds = true
addSubview(viewportView)
viewportView.addSubview(displayView)
+ darkLayer.backgroundColor = NSColor.black.cgColor
+ darkLayer.opacity = 0
+ darkLayer.zPosition = 1000
+ viewportView.layer?.addSublayer(darkLayer)
installFoldChrome()
}
@@ -249,6 +270,10 @@ let backing = backingPanelSize.width > 0 ? backingPanelSize : panelSize
let full = NSSize(width: backing.width * scale, height: backing.height * scale)
displayView.frameCenterRotation = 0
displayView.frame = NSRect(x: 0, y: size.height - full.height, width: full.width, height: full.height)
+ CATransaction.begin()
+ CATransaction.setDisableActions(true)
+ darkLayer.frame = viewportView.bounds
+ CATransaction.commit()
viewportView.frameCenterRotation = angle
if let vmView = displayView as? VPhoneVirtualMachineView {
vmView.enclosingDisplayRotation = angle
diff --git a/VPhoneGuestComponents/FoldDisplay/FoldDisplayPanel.m b/VPhoneGuestComponents/FoldDisplay/FoldDisplayPanel.m
index 2878952e..1518653d 100644
--- a/VPhoneGuestComponents/FoldDisplay/FoldDisplayPanel.m
+++ b/VPhoneGuestComponents/FoldDisplay/FoldDisplayPanel.m
@@ -39,8 +39,12 @@ static unsigned int vpStateControlDisplayId(id control) {
return *(unsigned int *)((uint8_t *)(__bridge void *)control + ivar_getOffset(ivar));
}
+// When the panel last came on (system uptime), for vpFollowContentModeTransition.
+static NSTimeInterval vpPanelOnAt;
+
static void vpApplyPanelState(id control, long long state, long long *applied, id completion) {
if (!control || (state == *applied && !completion)) return;
+ if (control == vpPrimaryStateControl && state && !*applied) vpPanelOnAt = NSProcessInfo.processInfo.systemUptime;
*applied = state;
((void (*)(id, SEL, long long, id))vpOriginalTransitionDisplayState)(control,
sel_registerName("transitionToDisplayState:withCompletion:"), state, completion);
@@ -122,7 +126,22 @@ static void vpFollowPanelOwner(void) {
dispatch_async(dispatch_get_main_queue(), ^{ vpApplyPanelOwner(); });
}
+// SpringBoard's fold effect fades the panel being folded away to black and
+// the arriving one in from black: two screens on hardware, about 0.12 s of
+// black on this one panel each time it opens. Without the fade the cover dims
+// and blurs and the inner display takes over at the swap. A fold made while
+// the screen is dark runs only as it comes on (the pose provider commits on
+// display frames), and there the fade hides what the effect carries over from
+// the inner display (its Home Screen, sideways on the cover), so it stays for
+// a fold that starts with the panel off or just lit.
+static void vpSetFoldFadeToBlack(BOOL enabled) {
+ id settings = ((id (*)(id, SEL))objc_msgSend)(objc_getClass("SBDeviceContinuousFoldInteractionDomain"), sel_registerName("rootSettings"));
+ SEL setFade = sel_registerName("setFadeToBlackEnabled:");
+ if ([settings respondsToSelector:setFade]) ((void (*)(id, SEL, BOOL))objc_msgSend)(settings, setFade, enabled);
+}
+
static void vpFollowContentModeTransition(id self, SEL selector, id transitionCoordinator) {
+ vpSetFoldFadeToBlack(!vpPrimaryAppliedState || NSProcessInfo.processInfo.systemUptime - vpPanelOnAt < 1);
((void (*)(id, SEL, id))vpOriginalPerformContentModeTransition)(self, selector, transitionCoordinator);
vpFollowPanelOwner();
}
@@ -164,14 +183,7 @@ void vpInstallSharedPanel(void) {
vpOriginalTransitionDisplayState = method_setImplementation(transition, (IMP)vpSharedScanoutTransition);
vpInstallContentModeOwnership();
vpFollowInnerPages();
- // SpringBoard's fold effect fades the panel being folded away to black and
- // the arriving one in from black: two screens on hardware, about 0.12 s of
- // black on this one panel each time it opens. Without the fade the cover
- // dims and blurs and the inner display takes over at the swap.
- id settings =((id (*)(id, SEL))objc_msgSend)(objc_getClass("SBDeviceContinuousFoldInteractionDomain"), sel_registerName("rootSettings"));
- SEL setFade = sel_registerName("setFadeToBlackEnabled:");
- if ([settings respondsToSelector:setFade]) ((void (*)(id, SEL, BOOL))objc_msgSend)(settings, setFade, NO);
- vpFoldDisplayLog([NSString stringWithFormat:@"inner display: fold fade to black off=%d", [settings respondsToSelector:setFade]]);
+ vpSetFoldFadeToBlack(NO);
vpFoldDisplayLog(@"inner display: shared scanout installed");
}